2024-03-06T07:10:41.307 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:10:41.308 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --verbose --owner scheduled_leonidus@teuthology --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583067 -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583067/orig.config.yaml 2024-03-06T07:10:41.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:10:41.433 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583067 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583067' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: syntax: v2 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ54/aN33i6HtJNgf9pJmoUcHsB2S7lNYez9DNZQ1tlc71X7eIK+eZWdJb1+0cdpm8yd0reja+HvCardZYjPXb8= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKGfW1Jlke62EgM/9jUGOrGX2abc4En0/A3grQO8zHtWE/0R93RfUnIhGf9EFalS/rWao6ByiVrAHEI1uP/MYNM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:10:41.433 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:10:41.434 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:10:41.434 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:10:41.435 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:10:41.459 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:10:41.459 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:10:41.459 INFO:teuthology.packaging:ref: None 2024-03-06T07:10:41.459 INFO:teuthology.packaging:tag: None 2024-03-06T07:10:41.459 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:10:41.459 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:10:41.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:10:41.612 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:10:41.613 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:10:41.619 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:10:41.619 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:10:41.666 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:10:41.683 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:10:41.700 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:10:41.724 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:43.201928', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ54/aN33i6HtJNgf9pJmoUcHsB2S7lNYez9DNZQ1tlc71X7eIK+eZWdJb1+0cdpm8yd0reja+HvCardZYjPXb8='} 2024-03-06T07:10:41.748 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583067', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:43.200617', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKGfW1Jlke62EgM/9jUGOrGX2abc4En0/A3grQO8zHtWE/0R93RfUnIhGf9EFalS/rWao6ByiVrAHEI1uP/MYNM='} 2024-03-06T07:10:41.748 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:10:41.753 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:10:41.753 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:10:41.753 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:10:41.847 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f68b9bb0f70>, signals=[15]) 2024-03-06T07:10:41.848 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:10:41.853 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:10:41.853 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:10:41.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:41.944 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:10:41.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:42.031 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:10:42.036 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-03-06T07:10:42.053 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-03-06T07:10:42.053 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:42.110 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:42.111 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:42.111 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:42.111 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:42.111 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-03-06T07:10:42.132 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-03-06T07:10:42.149 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-03-06T07:10:42.149 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-06T07:10:42.207 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-06T07:10:42.207 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-06T07:10:42.207 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:42.208 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:42.209 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-06T07:10:42.232 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:10:42.241 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:10:42.267 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:10:42.267 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:42.269 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:42.286 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:10:42.301 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:10:42.301 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:42.326 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:42.442 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:10:42.495 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:10:42.510 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-06T07:10:42.511 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-03-06T07:10:42.511 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-03-06T07:10:42.511 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:42.512 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:42.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:10:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:42.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:10:42.784 DEBUG:teuthology.task.kernel:sha1 for client.1 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:42.785 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:10:42.785 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-06T07:10:42.802 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:42.802 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:42.803 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:10:42.803 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum upgrade -y linux-firmware 2024-03-06T07:10:42.891 DEBUG:teuthology.task.kernel:sha1 for client.0 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:42.891 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:10:42.892 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-06T07:10:42.908 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:42.909 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:42.909 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:10:42.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum upgrade -y linux-firmware 2024-03-06T07:10:44.885 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-06T07:10:45.207 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-06T07:10:46.560 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-06T07:10:46.591 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-06T07:10:57.199 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 798 kB/s | 8.0 MB 00:10 2024-03-06T07:10:57.755 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 759 kB/s | 8.0 MB 00:10 2024-03-06T07:11:24.921 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 722 kB/s | 19 MB 00:26 2024-03-06T07:11:32.022 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-06T07:11:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 574 kB/s | 19 MB 00:33 2024-03-06T07:11:33.872 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2024-03-06T07:11:37.678 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-06T07:11:39.229 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-06T07:11:40.746 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-06T07:11:45.546 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-03-06T07:11:47.850 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-06T07:11:47.860 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-06T07:11:47.860 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-06T07:11:47.860 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-06T07:11:47.860 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 1 Package 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 375 M 2024-03-06T07:11:47.861 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-06T07:12:22.544 INFO:teuthology.orchestra.run.smithi114.stdout:linux-firmware-20240219-143.el9.noarch.rpm 11 MB/s | 375 MB 00:34 2024-03-06T07:12:22.545 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:12:22.546 INFO:teuthology.orchestra.run.smithi114.stdout:Total 11 MB/s | 375 MB 00:34 2024-03-06T07:12:22.546 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-06T07:12:22.633 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-06T07:12:22.634 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-06T07:12:25.024 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 411 kB/s | 20 MB 00:51 2024-03-06T07:12:25.107 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-06T07:12:25.107 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-06T07:12:29.898 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-06T07:12:31.322 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-06T07:12:31.594 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:31.677 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:32.169 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-06T07:12:32.181 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-06T07:12:32.181 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-06T07:12:32.181 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-06T07:12:32.181 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-03-06T07:12:32.181 INFO:teuthology.orchestra.run.smithi177.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:12:32.181 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:12:32.182 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-06T07:12:32.182 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-06T07:12:32.182 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 1 Package 2024-03-06T07:12:32.182 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:12:32.182 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 375 M 2024-03-06T07:12:32.182 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-06T07:12:32.582 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:32.583 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:32.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:32.950 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:12:32.950 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-03-06T07:12:32.951 INFO:teuthology.orchestra.run.smithi114.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:12:32.951 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:12:32.951 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-06T07:12:33.255 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:12:33.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:12:33.415 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:12:33.415 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:12:33.418 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi114.front.sepia.ceph.com to complete... 2024-03-06T07:12:44.206 INFO:teuthology.orchestra.run.smithi177.stdout:linux-firmware-20240219-143.el9.noarch.rpm 31 MB/s | 375 MB 00:12 2024-03-06T07:12:44.207 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:12:44.208 INFO:teuthology.orchestra.run.smithi177.stdout:Total 31 MB/s | 375 MB 00:12 2024-03-06T07:12:44.208 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-06T07:12:44.304 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-06T07:12:44.304 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-06T07:12:46.785 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-06T07:12:46.785 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-06T07:12:53.257 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-06T07:12:53.557 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:53.633 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:54.939 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:54.939 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:55.303 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:55.303 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:12:55.303 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-03-06T07:12:55.303 INFO:teuthology.orchestra.run.smithi177.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:12:55.303 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:12:55.303 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-06T07:12:55.584 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:12:55.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:12:55.814 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:12:55.814 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:12:55.820 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi177.front.sepia.ceph.com to complete... 2024-03-06T07:15:05.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06 07:15:05 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:15:05.322 INFO:teuthology.orchestra.run.smithi114.stderr:FINISHED --2024-03-06 07:15:05-- 2024-03-06T07:15:05.322 INFO:teuthology.orchestra.run.smithi114.stderr:Total wall clock time: 2m 32s 2024-03-06T07:15:05.322 INFO:teuthology.orchestra.run.smithi114.stderr:Downloaded: 1 files, 320M in 2m 31s (2.12 MB/s) 2024-03-06T07:15:05.326 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:15:05.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:15:07.259 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying... ######################################## 2024-03-06T07:15:07.401 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing... ######################################## 2024-03-06T07:15:07.481 INFO:teuthology.orchestra.run.smithi114.stdout:Updating / installing... 2024-03-06T07:15:15.001 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:15:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:15:31.152 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06 07:15:31 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:15:31.152 INFO:teuthology.orchestra.run.smithi177.stderr:FINISHED --2024-03-06 07:15:31-- 2024-03-06T07:15:31.152 INFO:teuthology.orchestra.run.smithi177.stderr:Total wall clock time: 2m 35s 2024-03-06T07:15:31.152 INFO:teuthology.orchestra.run.smithi177.stderr:Downloaded: 1 files, 320M in 2m 35s (2.07 MB/s) 2024-03-06T07:15:31.156 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:15:31.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:15:33.089 INFO:teuthology.orchestra.run.smithi177.stdout:Verifying... ######################################## 2024-03-06T07:15:33.152 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing... ######################################## 2024-03-06T07:15:33.232 INFO:teuthology.orchestra.run.smithi177.stdout:Updating / installing... 2024-03-06T07:15:40.764 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:15:51.892 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:15:51.892 DEBUG:teuthology.orchestra.run.smithi114:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:51.968 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:51.968 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:15:51.969 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:15:51.970 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:15:51.971 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:15:51.972 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:15:51.973 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:15:51.974 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:15:51.975 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:15:51.976 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:15:51.977 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:15:51.977 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:15:51.977 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:15:51.977 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:15:51.977 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:15:51.977 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:15:51.977 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:15:51.978 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:15:51.979 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:15:51.980 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:15:51.981 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:15:51.982 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:15:51.983 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:15:51.984 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:15:51.985 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:15:51.986 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:15:51.987 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:15:51.988 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:15:51.989 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:15:51.990 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:15:51.991 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:15:51.992 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:15:51.993 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:15:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:15:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:15:51.996 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:15:51.997 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:15:51.998 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:15:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:15:52.000 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:15:52.001 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:15:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:15:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:15:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:15:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:15:52.006 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:15:52.007 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:15:52.008 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:15:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:15:52.010 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:15:52.011 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:15:52.012 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:15:52.013 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:15:52.014 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:15:52.015 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:15:52.016 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:15:52.017 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:15:52.018 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:15:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:15:52.020 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:15:52.021 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:15:52.022 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:15:52.023 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:15:52.024 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:15:52.025 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:15:52.026 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:15:52.027 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:15:52.028 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:15:52.029 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:15:52.030 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:15:52.031 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:15:52.032 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:15:52.033 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:15:52.034 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:15:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:15:52.036 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:15:52.037 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:15:52.038 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:15:52.039 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:15:52.040 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:15:52.041 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:15:52.042 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:15:52.043 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:15:52.044 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:15:52.045 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:15:52.046 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:15:52.047 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:15:52.048 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:15:52.049 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:15:52.050 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:15:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:15:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:15:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:15:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:15:52.055 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:15:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:15:52.057 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:15:52.058 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:15:52.059 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:15:52.060 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:15:52.061 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:15:52.062 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:15:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:15:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:15:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:15:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:15:52.067 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:15:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:15:52.069 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:15:52.070 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:15:52.071 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:15:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:15:52.073 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:15:52.074 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:15:52.075 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:15:52.076 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:15:52.077 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:15:52.078 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:15:52.079 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:15:52.080 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:15:52.081 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:15:52.082 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:15:52.083 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:15:52.084 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:15:52.085 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:15:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:15:52.087 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:15:52.088 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:15:52.089 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:15:52.090 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:15:52.091 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:15:52.092 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:15:52.093 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:15:52.094 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:15:52.095 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:15:52.096 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:15:52.097 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:15:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:15:52.100 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:15:52.101 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:15:52.102 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:15:52.103 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:15:52.104 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:15:52.105 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:15:52.106 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:52.110 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:52.111 DEBUG:teuthology.orchestra.run.smithi114:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:52.132 INFO:teuthology.orchestra.run.smithi114.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:15:52.132 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:52.132 INFO:teuthology.orchestra.run.smithi114.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:52.132 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-03-06T07:15:52.132 INFO:teuthology.orchestra.run.smithi114.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:15:52.132 INFO:teuthology.orchestra.run.smithi114.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout: fi 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:done 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-03-06T07:15:52.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-03-06T07:15:52.209 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-03-06T07:15:52.209 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-03-06T07:15:52.209 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-03-06T07:15:52.209 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-03-06T07:15:52.210 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:15:52.211 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:15:52.212 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:15:52.213 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:52.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:15:52.781 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-03-06T07:15:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:15:54.208 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-03-06T07:15:54.210 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-06T07:15:54.226 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.u5DPfKImou 2024-03-06T07:15:54.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.u5DPfKImou 2024-03-06T07:15:54.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.u5DPfKImou 2024-03-06T07:15:54.430 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.u5DPfKImou is 6KB 2024-03-06T07:15:54.481 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.u5DPfKImou 2024-03-06T07:15:54.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:15:54.566 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:15:54.566 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:15:54.566 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:15:54.567 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:15:54.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:54.694 INFO:teuthology.orchestra.run.smithi177.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:15:54.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-03-06T07:16:17.187 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:16:17.187 DEBUG:teuthology.orchestra.run.smithi177:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:16:17.261 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:16:17.262 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:16:17.263 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:16:17.264 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:16:17.265 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:16:17.266 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:16:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:16:17.268 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:16:17.269 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:16:17.270 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:16:17.271 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:16:17.272 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:16:17.273 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:16:17.274 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:16:17.275 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:16:17.276 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:16:17.277 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:16:17.278 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:16:17.279 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:16:17.280 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:16:17.281 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:16:17.282 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:16:17.283 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:16:17.284 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:16:17.285 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:16:17.286 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:16:17.287 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:16:17.288 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:16:17.289 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:16:17.290 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:16:17.291 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:16:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:16:17.293 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:16:17.295 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:16:17.295 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:16:17.296 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:16:17.297 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:16:17.298 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:16:17.299 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:16:17.300 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:16:17.301 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:16:17.302 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:16:17.303 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:16:17.304 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:16:17.305 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:16:17.306 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:16:17.307 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:16:17.308 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:16:17.309 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:16:17.310 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:16:17.311 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:16:17.312 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:16:17.313 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:16:17.314 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:16:17.315 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:16:17.316 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:16:17.317 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:16:17.318 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:16:17.319 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:16:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:16:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:16:17.322 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:16:17.323 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:16:17.324 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:16:17.325 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:16:17.326 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:16:17.327 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:16:17.328 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:16:17.329 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:16:17.330 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:16:17.331 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:16:17.332 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:16:17.333 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:16:17.334 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:16:17.335 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:16:17.336 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:16:17.337 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:16:17.338 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:16:17.339 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:16:17.340 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:16:17.341 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:16:17.342 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:16:17.343 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:16:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:16:17.345 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:16:17.346 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:16:17.347 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:16:17.348 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:16:17.349 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:16:17.350 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:16:17.351 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:16:17.352 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:16:17.353 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:16:17.354 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:16:17.355 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:16:17.356 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:16:17.357 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:16:17.358 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:16:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:16:17.360 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:16:17.361 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:16:17.362 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:16:17.363 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:16:17.364 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:16:17.365 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:16:17.366 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:16:17.367 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:16:17.368 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:16:17.369 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:16:17.370 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:16:17.371 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:16:17.372 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:16:17.373 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:16:17.374 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:16:17.375 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:16:17.376 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:16:17.377 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:16:17.378 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:16:17.379 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:16:17.380 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:16:17.381 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:16:17.382 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:16:17.383 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:16:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:16:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:16:17.386 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:16:17.387 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:16:17.388 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:16:17.389 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:16:17.390 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:16:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:16:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:16:17.393 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:16:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:16:17.395 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:16:17.396 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:16:17.397 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:16:17.398 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:16:17.399 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:16:17.400 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:16:17.401 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:16:17.402 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:16:17.403 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:16:17.403 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:16:17.403 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:16:17.403 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:16:17.403 INFO:teuthology.orchestra.run.smithi177.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:16:17.407 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:17.407 DEBUG:teuthology.orchestra.run.smithi177:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout:fi 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:16:17.429 INFO:teuthology.orchestra.run.smithi177.stdout: fi 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:done 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:fi 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:17.430 INFO:teuthology.orchestra.run.smithi177.stdout:fi 2024-03-06T07:16:17.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:16:17.506 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:16:17.507 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:16:17.508 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:16:17.508 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:17.509 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:16:18.163 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-03-06T07:16:19.508 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:16:19.535 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-03-06T07:16:19.540 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-03-06T07:16:19.555 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.Z44ycIdVZ1 2024-03-06T07:16:19.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z44ycIdVZ1 2024-03-06T07:16:19.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.Z44ycIdVZ1 2024-03-06T07:16:19.771 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.Z44ycIdVZ1 is 6KB 2024-03-06T07:16:19.819 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.Z44ycIdVZ1 2024-03-06T07:16:19.833 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:16:19.898 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:16:19.898 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:16:19.898 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:16:19.899 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:16:19.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:16:20.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-03-06T07:16:20.159 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-06T07:16:24.724 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-06T07:16:24.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-06T07:16:24.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:16:50.080 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-06T07:16:50.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-06T07:16:50.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:24.763 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:17:33.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-06T07:17:33.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:37.152 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-06T07:17:37.752 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-06T07:17:37.752 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:17:37.752 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-06T07:17:37.801 INFO:teuthology.orchestra.run.smithi114.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:37.801 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:37.801 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:17:37.801 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:17:38.805 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:50.083 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:17:59.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-06T07:17:59.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:06.573 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-06T07:18:07.159 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-06T07:18:07.159 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:18:07.160 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-06T07:18:07.198 INFO:teuthology.orchestra.run.smithi177.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:18:07.198 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:18:07.198 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:18:07.198 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:18:08.201 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:08.202 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:18:08.209 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:18:08.209 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:18:08.218 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:18:08.269 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:18:08.276 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:18:08.309 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:18:08.309 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:18:08.316 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:18:08.383 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:18:08.390 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:18:08.390 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:18:08.397 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:18:08.533 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:08.561 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:08.606 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:08.630 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:18:08.635 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:18:08.643 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:18:08.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:18:08.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:18:08.737 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:18:08.746 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:18:08.746 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:18:08.751 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:18:08.798 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:08.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:08.997 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:09.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:09.183 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:18:09.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:18:09.278 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:09.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:18:09.468 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:09.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:18:09.643 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:18:09.643 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:18:09.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-03-06T07:18:09.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-03-06T07:18:09.865 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:18:09.884 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:18:10.348 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:18:10.356 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:18:10.356 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:18:10.387 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:18:10.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-03-06T07:18:10.558 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-03-06T07:18:10.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-03-06T07:18:10.687 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-03-06T07:18:10.691 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:18:10.691 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-03-06T07:18:10.751 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-03-06T07:18:10.798 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-03-06T07:18:10.798 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:18:10.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:18:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:18:10.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:18:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:18:10.957 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:18:10.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:18:11.048 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:18:11.128 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:18:11.139 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:18:11.156 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:18:11.157 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:24:05.111 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-03-06T07:24:05.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-06T07:24:05.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:24:05.267 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-06T07:24:05.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-06T07:24:05.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-06T07:24:05.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:24:05.562 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-06T07:24:05.713 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-06T07:24:05.714 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:24:05.730 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:24:05.731 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:24:05.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:24:05.739 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:24:05.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:24:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:24:05.838 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:24:05.886 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:24:05.890 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:24:05.977 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-03-06T07:24:05.980 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-03-06T07:24:06.013 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-03-06T07:24:06.020 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-03-06T07:24:06.065 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:24:06.071 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:24:06.116 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:24:06.125 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:24:06.245 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:24:06.254 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:24:06.254 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:24:06.254 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-06T07:24:06.254 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.254 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.254 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.254 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.262 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:24:06.262 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-06T07:24:06.262 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.262 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.262 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.262 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:24:06.263 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:24:06.271 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:24:06.271 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:24:06.272 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:24:06.272 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:24:06.277 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:24:06.277 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:24:06.278 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:24:06.278 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:24:06.280 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:24:06.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:24:06.609 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:24:06.609 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:24:06.633 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:24:06.634 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:24:06.819 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:24:06.819 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:24:06.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:24:06.868 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:24:06.868 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:24:06.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:24:06.919 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:24:06.919 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:24:06.968 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:24:06.968 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:24:07.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:24:07.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:24:07.191 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:24:07.240 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:24:07.281 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-03-06T07:24:07.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-03-06T07:24:07.328 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-03-06T07:24:07.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-03-06T07:24:07.631 INFO:teuthology.orchestra.run.smithi114.stdout:74 files removed 2024-03-06T07:24:07.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:24:07.683 INFO:teuthology.orchestra.run.smithi177.stdout:74 files removed 2024-03-06T07:24:07.723 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:24:08.621 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 186 kB/s | 83 kB 00:00 2024-03-06T07:24:08.828 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 138 kB/s | 83 kB 00:00 2024-03-06T07:24:09.230 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-06T07:24:09.370 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-06T07:24:09.839 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-06T07:24:09.971 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-03-06T07:24:10.579 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-06T07:24:10.749 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-06T07:24:12.560 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-06T07:24:12.767 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-03-06T07:24:13.572 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-06T07:24:13.988 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-06T07:24:21.807 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:07 2024-03-06T07:24:26.099 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-06T07:24:27.967 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-06T07:24:30.797 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-06T07:24:37.589 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-06T07:24:40.564 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 9.4 kB/s | 15 kB 00:01 2024-03-06T07:24:41.608 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-06T07:24:47.316 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-06T07:24:50.069 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:50.071 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:50.094 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:24:50.095 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:24:50.096 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:24:50.098 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:50.101 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:50.105 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:24:50.106 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:24:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-06T07:24:50.218 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-06T07:24:50.218 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:24:50.219 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:24:50.220 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-03-06T07:24:50.221 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:24:50.222 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:24:50.223 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:24:50.224 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:24:50.225 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:24:50.226 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:24:50.227 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:24:50.228 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:24:50.229 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:24:50.230 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout:Install 150 Packages 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 26 Packages 2024-03-06T07:24:50.231 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:24:50.232 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 224 M 2024-03-06T07:24:50.232 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-06T07:24:51.376 INFO:teuthology.orchestra.run.smithi177.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-06T07:24:52.388 INFO:teuthology.orchestra.run.smithi177.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 879 kB/s | 888 kB 00:01 2024-03-06T07:24:52.601 INFO:teuthology.orchestra.run.smithi177.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 686 kB/s | 145 kB 00:00 2024-03-06T07:24:53.126 INFO:teuthology.orchestra.run.smithi177.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 2.4 MB/s | 5.3 MB 00:02 2024-03-06T07:24:53.621 INFO:teuthology.orchestra.run.smithi177.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-03-06T07:24:54.031 INFO:teuthology.orchestra.run.smithi177.stdout:(6/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.6 MB/s | 2.3 MB 00:01 2024-03-06T07:24:54.910 INFO:teuthology.orchestra.run.smithi177.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 3.6 MB/s | 4.7 MB 00:01 2024-03-06T07:24:56.511 INFO:teuthology.orchestra.run.smithi177.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 3.6 MB/s | 20 MB 00:05 2024-03-06T07:24:56.577 INFO:teuthology.orchestra.run.smithi177.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 385 kB/s | 25 kB 00:00 2024-03-06T07:24:56.869 INFO:teuthology.orchestra.run.smithi177.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 5.0 MB/s | 9.8 MB 00:01 2024-03-06T07:24:56.953 INFO:teuthology.orchestra.run.smithi177.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 2.6 MB/s | 220 kB 00:00 2024-03-06T07:24:57.036 INFO:teuthology.orchestra.run.smithi177.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 329 kB/s | 27 kB 00:00 2024-03-06T07:24:57.120 INFO:teuthology.orchestra.run.smithi177.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 378 kB/s | 31 kB 00:00 2024-03-06T07:24:57.345 INFO:teuthology.orchestra.run.smithi177.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 3.3 MB/s | 747 kB 00:00 2024-03-06T07:24:57.445 INFO:teuthology.orchestra.run.smithi177.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.6 MB/s | 163 kB 00:00 2024-03-06T07:24:57.529 INFO:teuthology.orchestra.run.smithi177.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.5 MB/s | 127 kB 00:00 2024-03-06T07:24:57.727 INFO:teuthology.orchestra.run.smithi177.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 2.5 MB/s | 501 kB 00:00 2024-03-06T07:24:59.021 INFO:teuthology.orchestra.run.smithi177.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 4.0 MB/s | 5.1 MB 00:01 2024-03-06T07:24:59.104 INFO:teuthology.orchestra.run.smithi177.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 548 kB/s | 45 kB 00:00 2024-03-06T07:24:59.370 INFO:teuthology.orchestra.run.smithi177.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 496 kB/s | 131 kB 00:00 2024-03-06T07:24:59.705 INFO:teuthology.orchestra.run.smithi177.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 484 kB/s | 162 kB 00:00 2024-03-06T07:25:00.122 INFO:teuthology.orchestra.run.smithi177.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 771 kB/s | 321 kB 00:00 2024-03-06T07:25:00.606 INFO:teuthology.orchestra.run.smithi177.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 617 kB/s | 298 kB 00:00 2024-03-06T07:25:00.731 INFO:teuthology.orchestra.run.smithi177.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 799 kB/s | 99 kB 00:00 2024-03-06T07:25:00.856 INFO:teuthology.orchestra.run.smithi177.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 684 kB/s | 85 kB 00:00 2024-03-06T07:25:01.440 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 630 kB/s | 20 MB 00:33 2024-03-06T07:25:03.548 INFO:teuthology.orchestra.run.smithi177.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 3.1 MB 00:02 2024-03-06T07:25:03.715 INFO:teuthology.orchestra.run.smithi177.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 1.0 MB/s | 171 kB 00:00 2024-03-06T07:25:03.823 INFO:teuthology.orchestra.run.smithi177.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 291 kB/s | 31 kB 00:00 2024-03-06T07:25:04.007 INFO:teuthology.orchestra.run.smithi177.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 756 kB/s | 138 kB 00:00 2024-03-06T07:25:05.316 INFO:teuthology.orchestra.run.smithi177.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.4 MB/s | 1.8 MB 00:01 2024-03-06T07:25:07.215 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-06T07:25:07.875 INFO:teuthology.orchestra.run.smithi177.stdout:(31/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 1.2 MB/s | 17 MB 00:13 2024-03-06T07:25:08.141 INFO:teuthology.orchestra.run.smithi177.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 938 kB/s | 248 kB 00:00 2024-03-06T07:25:08.241 INFO:teuthology.orchestra.run.smithi177.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 498 kB/s | 49 kB 00:00 2024-03-06T07:25:08.363 INFO:teuthology.orchestra.run.smithi177.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 127 kB/s | 15 kB 00:00 2024-03-06T07:25:08.625 INFO:teuthology.orchestra.run.smithi177.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.0 MB/s | 264 kB 00:00 2024-03-06T07:25:09.193 INFO:teuthology.orchestra.run.smithi177.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.3 MB/s | 748 kB 00:00 2024-03-06T07:25:09.401 INFO:teuthology.orchestra.run.smithi177.stdout:(37/176): ceph-mgr-diskprediction-local-19.0.0- 1.8 MB/s | 7.4 MB 00:04 2024-03-06T07:25:09.418 INFO:teuthology.orchestra.run.smithi177.stdout:(38/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 81 kB/s | 18 kB 00:00 2024-03-06T07:25:09.643 INFO:teuthology.orchestra.run.smithi177.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-06T07:25:09.685 INFO:teuthology.orchestra.run.smithi177.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-06T07:25:09.710 INFO:teuthology.orchestra.run.smithi177.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-06T07:25:09.735 INFO:teuthology.orchestra.run.smithi177.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 420 kB/s | 20 kB 00:00 2024-03-06T07:25:09.769 INFO:teuthology.orchestra.run.smithi177.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-06T07:25:09.794 INFO:teuthology.orchestra.run.smithi177.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 571 kB/s | 32 kB 00:00 2024-03-06T07:25:09.911 INFO:teuthology.orchestra.run.smithi177.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-06T07:25:09.936 INFO:teuthology.orchestra.run.smithi177.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 17 kB 00:00 2024-03-06T07:25:09.969 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:25:09.970 INFO:teuthology.orchestra.run.smithi177.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 313 kB/s | 10 kB 00:00 2024-03-06T07:25:09.971 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:25:09.987 INFO:teuthology.orchestra.run.smithi177.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-03-06T07:25:09.993 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:25:09.995 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:25:09.996 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:25:09.998 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:25:10.003 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:25:10.004 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:25:10.005 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:25:10.045 INFO:teuthology.orchestra.run.smithi177.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 44 kB 00:00 2024-03-06T07:25:10.094 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-06T07:25:10.096 INFO:teuthology.orchestra.run.smithi177.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:25:10.120 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:25:10.121 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:25:10.122 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:25:10.123 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:25:10.124 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:25:10.125 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:25:10.126 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:25:10.127 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:25:10.128 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:25:10.129 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:25:10.130 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:25:10.131 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:25:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout:Install 150 Packages 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 26 Packages 2024-03-06T07:25:10.133 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:25:10.134 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 224 M 2024-03-06T07:25:10.134 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-06T07:25:10.137 INFO:teuthology.orchestra.run.smithi177.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2024-03-06T07:25:10.163 INFO:teuthology.orchestra.run.smithi177.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 397 kB/s | 24 kB 00:00 2024-03-06T07:25:10.188 INFO:teuthology.orchestra.run.smithi177.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 46 kB 00:00 2024-03-06T07:25:10.230 INFO:teuthology.orchestra.run.smithi177.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-06T07:25:10.264 INFO:teuthology.orchestra.run.smithi177.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-03-06T07:25:10.289 INFO:teuthology.orchestra.run.smithi177.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 605 kB/s | 34 kB 00:00 2024-03-06T07:25:10.465 INFO:teuthology.orchestra.run.smithi177.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-06T07:25:10.565 INFO:teuthology.orchestra.run.smithi177.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-06T07:25:10.615 INFO:teuthology.orchestra.run.smithi177.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-06T07:25:10.657 INFO:teuthology.orchestra.run.smithi177.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 8.7 MB/s | 811 kB 00:00 2024-03-06T07:25:10.682 INFO:teuthology.orchestra.run.smithi177.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-03-06T07:25:10.824 INFO:teuthology.orchestra.run.smithi177.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-06T07:25:10.883 INFO:teuthology.orchestra.run.smithi177.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-06T07:25:10.933 INFO:teuthology.orchestra.run.smithi177.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-06T07:25:11.033 INFO:teuthology.orchestra.run.smithi114.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-06T07:25:11.183 INFO:teuthology.orchestra.run.smithi177.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi177.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 875 kB/s | 227 kB 00:00 2024-03-06T07:25:11.267 INFO:teuthology.orchestra.run.smithi177.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 424 kB/s | 24 kB 00:00 2024-03-06T07:25:11.292 INFO:teuthology.orchestra.run.smithi177.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 220 kB/s | 18 kB 00:00 2024-03-06T07:25:11.342 INFO:teuthology.orchestra.run.smithi177.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-06T07:25:11.367 INFO:teuthology.orchestra.run.smithi177.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 428 kB/s | 32 kB 00:00 2024-03-06T07:25:11.484 INFO:teuthology.orchestra.run.smithi177.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.8 MB/s | 217 kB 00:00 2024-03-06T07:25:11.626 INFO:teuthology.orchestra.run.smithi177.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-06T07:25:11.651 INFO:teuthology.orchestra.run.smithi177.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-06T07:25:11.694 INFO:teuthology.orchestra.run.smithi177.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-06T07:25:11.710 INFO:teuthology.orchestra.run.smithi177.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 717 kB/s | 41 kB 00:00 2024-03-06T07:25:11.744 INFO:teuthology.orchestra.run.smithi177.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-06T07:25:11.769 INFO:teuthology.orchestra.run.smithi177.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-06T07:25:11.803 INFO:teuthology.orchestra.run.smithi177.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-06T07:25:11.828 INFO:teuthology.orchestra.run.smithi177.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 844 kB/s | 47 kB 00:00 2024-03-06T07:25:11.850 INFO:teuthology.orchestra.run.smithi114.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 1.1 MB/s | 888 kB 00:00 2024-03-06T07:25:11.853 INFO:teuthology.orchestra.run.smithi177.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 876 kB/s | 43 kB 00:00 2024-03-06T07:25:12.008 INFO:teuthology.orchestra.run.smithi114.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 921 kB/s | 145 kB 00:00 2024-03-06T07:25:12.221 INFO:teuthology.orchestra.run.smithi177.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 107 kB/s | 39 kB 00:00 2024-03-06T07:25:12.471 INFO:teuthology.orchestra.run.smithi177.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-03-06T07:25:12.538 INFO:teuthology.orchestra.run.smithi177.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 136 kB/s | 41 kB 00:00 2024-03-06T07:25:12.580 INFO:teuthology.orchestra.run.smithi177.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 5.2 MB/s | 215 kB 00:00 2024-03-06T07:25:12.605 INFO:teuthology.orchestra.run.smithi177.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 517 kB/s | 34 kB 00:00 2024-03-06T07:25:12.647 INFO:teuthology.orchestra.run.smithi177.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-06T07:25:12.672 INFO:teuthology.orchestra.run.smithi177.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-06T07:25:12.697 INFO:teuthology.orchestra.run.smithi177.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-06T07:25:12.723 INFO:teuthology.orchestra.run.smithi177.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 87 kB 00:00 2024-03-06T07:25:12.748 INFO:teuthology.orchestra.run.smithi177.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-06T07:25:12.773 INFO:teuthology.orchestra.run.smithi177.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 902 kB/s | 44 kB 00:00 2024-03-06T07:25:12.798 INFO:teuthology.orchestra.run.smithi177.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-06T07:25:12.832 INFO:teuthology.orchestra.run.smithi177.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-03-06T07:25:12.875 INFO:teuthology.orchestra.run.smithi114.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 2.7 MB/s | 2.3 MB 00:00 2024-03-06T07:25:12.927 INFO:teuthology.orchestra.run.smithi177.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 6.1 MB/s | 802 kB 00:00 2024-03-06T07:25:12.957 INFO:teuthology.orchestra.run.smithi177.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 194 kB 00:00 2024-03-06T07:25:12.991 INFO:teuthology.orchestra.run.smithi177.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.5 MB/s | 217 kB 00:00 2024-03-06T07:25:13.016 INFO:teuthology.orchestra.run.smithi177.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 241 kB 00:00 2024-03-06T07:25:13.075 INFO:teuthology.orchestra.run.smithi177.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-06T07:25:13.100 INFO:teuthology.orchestra.run.smithi177.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-06T07:25:13.126 INFO:teuthology.orchestra.run.smithi177.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.5 MB/s | 124 kB 00:00 2024-03-06T07:25:13.151 INFO:teuthology.orchestra.run.smithi177.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.3 MB/s | 113 kB 00:00 2024-03-06T07:25:13.176 INFO:teuthology.orchestra.run.smithi177.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.8 MB/s | 187 kB 00:00 2024-03-06T07:25:13.227 INFO:teuthology.orchestra.run.smithi177.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 455 kB/s | 22 kB 00:00 2024-03-06T07:25:13.336 INFO:teuthology.orchestra.run.smithi114.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 3.3 MB/s | 1.5 MB 00:00 2024-03-06T07:25:13.360 INFO:teuthology.orchestra.run.smithi177.stdout:(104/176): protobuf-compiler-3.14.0-13.el9.x86_ 6.4 MB/s | 863 kB 00:00 2024-03-06T07:25:14.203 INFO:teuthology.orchestra.run.smithi114.stdout:(6/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 5.4 MB/s | 4.7 MB 00:00 2024-03-06T07:25:14.235 INFO:teuthology.orchestra.run.smithi177.stdout:(105/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 2.7 MB/s | 47 MB 00:17 2024-03-06T07:25:14.269 INFO:teuthology.orchestra.run.smithi177.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 604 kB/s | 548 kB 00:00 2024-03-06T07:25:14.361 INFO:teuthology.orchestra.run.smithi177.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 213 kB/s | 19 kB 00:00 2024-03-06T07:25:14.581 INFO:teuthology.orchestra.run.smithi177.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 898 kB/s | 309 kB 00:00 2024-03-06T07:25:14.637 INFO:teuthology.orchestra.run.smithi177.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 473 kB/s | 26 kB 00:00 2024-03-06T07:25:14.670 INFO:teuthology.orchestra.run.smithi177.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-03-06T07:25:14.737 INFO:teuthology.orchestra.run.smithi177.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-06T07:25:14.812 INFO:teuthology.orchestra.run.smithi177.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-06T07:25:14.863 INFO:teuthology.orchestra.run.smithi177.stdout:(113/176): python3-devel-3.9.17-2.el9.x86_64.rp 144 kB/s | 245 kB 00:01 2024-03-06T07:25:15.163 INFO:teuthology.orchestra.run.smithi177.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-03-06T07:25:15.196 INFO:teuthology.orchestra.run.smithi177.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-03-06T07:25:15.239 INFO:teuthology.orchestra.run.smithi177.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 779 kB/s | 32 kB 00:00 2024-03-06T07:25:15.272 INFO:teuthology.orchestra.run.smithi177.stdout:(117/176): python3-asyncssh-2.13.2-4.el9.noarch 1.3 MB/s | 548 kB 00:00 2024-03-06T07:25:15.289 INFO:teuthology.orchestra.run.smithi177.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 303 kB/s | 15 kB 00:00 2024-03-06T07:25:15.303 INFO:teuthology.orchestra.run.smithi114.stdout:(7/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.1 MB/s | 5.3 MB 00:04 2024-03-06T07:25:15.356 INFO:teuthology.orchestra.run.smithi177.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-06T07:25:15.456 INFO:teuthology.orchestra.run.smithi177.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-03-06T07:25:15.558 INFO:teuthology.orchestra.run.smithi177.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 866 kB/s | 35 kB 00:00 2024-03-06T07:25:15.558 INFO:teuthology.orchestra.run.smithi177.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.7 MB/s | 303 kB 00:00 2024-03-06T07:25:15.657 INFO:teuthology.orchestra.run.smithi177.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 1.3 MB/s | 203 kB 00:00 2024-03-06T07:25:15.766 INFO:teuthology.orchestra.run.smithi177.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-06T07:25:15.833 INFO:teuthology.orchestra.run.smithi177.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 162 kB/s | 11 kB 00:00 2024-03-06T07:25:15.874 INFO:teuthology.orchestra.run.smithi177.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 434 kB/s | 18 kB 00:00 2024-03-06T07:25:15.916 INFO:teuthology.orchestra.run.smithi177.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 567 kB/s | 23 kB 00:00 2024-03-06T07:25:15.958 INFO:teuthology.orchestra.run.smithi177.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 477 kB/s | 19 kB 00:00 2024-03-06T07:25:16.000 INFO:teuthology.orchestra.run.smithi177.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 482 kB/s | 20 kB 00:00 2024-03-06T07:25:16.167 INFO:teuthology.orchestra.run.smithi177.stdout:(130/176): libarrow-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-03-06T07:25:16.226 INFO:teuthology.orchestra.run.smithi177.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 805 kB/s | 46 kB 00:00 2024-03-06T07:25:16.293 INFO:teuthology.orchestra.run.smithi177.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi177.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-06T07:25:16.460 INFO:teuthology.orchestra.run.smithi177.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 2.2 MB/s | 1.0 MB 00:00 2024-03-06T07:25:16.503 INFO:teuthology.orchestra.run.smithi114.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 3.5 MB/s | 20 MB 00:05 2024-03-06T07:25:16.510 INFO:teuthology.orchestra.run.smithi177.stdout:(135/176): python3-grpcio-1.46.7-10.el9.x86_64. 2.1 MB/s | 2.0 MB 00:00 2024-03-06T07:25:16.536 INFO:teuthology.orchestra.run.smithi177.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 221 kB/s | 16 kB 00:00 2024-03-06T07:25:16.561 INFO:teuthology.orchestra.run.smithi177.stdout:(137/176): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-06T07:25:16.586 INFO:teuthology.orchestra.run.smithi177.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 572 kB/s | 28 kB 00:00 2024-03-06T07:25:16.629 INFO:teuthology.orchestra.run.smithi177.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-03-06T07:25:16.662 INFO:teuthology.orchestra.run.smithi177.stdout:(140/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 600 kB/s | 90 kB 00:00 2024-03-06T07:25:16.704 INFO:teuthology.orchestra.run.smithi177.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 805 kB/s | 59 kB 00:00 2024-03-06T07:25:16.729 INFO:teuthology.orchestra.run.smithi114.stdout:(9/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 6.7 MB/s | 17 MB 00:02 2024-03-06T07:25:16.729 INFO:teuthology.orchestra.run.smithi177.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 541 kB/s | 36 kB 00:00 2024-03-06T07:25:16.745 INFO:teuthology.orchestra.run.smithi114.stdout:(10/176): ceph-selinux-19.0.0-1739.g1bc250c7.el 104 kB/s | 25 kB 00:00 2024-03-06T07:25:16.754 INFO:teuthology.orchestra.run.smithi177.stdout:(143/176): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-03-06T07:25:16.779 INFO:teuthology.orchestra.run.smithi177.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-03-06T07:25:16.838 INFO:teuthology.orchestra.run.smithi177.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-06T07:25:16.880 INFO:teuthology.orchestra.run.smithi177.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 487 kB/s | 20 kB 00:00 2024-03-06T07:25:16.887 INFO:teuthology.orchestra.run.smithi114.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.5 MB/s | 220 kB 00:00 2024-03-06T07:25:16.922 INFO:teuthology.orchestra.run.smithi177.stdout:(147/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-06T07:25:16.946 INFO:teuthology.orchestra.run.smithi114.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 473 kB/s | 27 kB 00:00 2024-03-06T07:25:17.014 INFO:teuthology.orchestra.run.smithi114.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 474 kB/s | 31 kB 00:00 2024-03-06T07:25:17.040 INFO:teuthology.orchestra.run.smithi177.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-06T07:25:17.157 INFO:teuthology.orchestra.run.smithi177.stdout:(149/176): python3-werkzeug-2.0.3-3.el9.1.noarc 1.1 MB/s | 427 kB 00:00 2024-03-06T07:25:17.255 INFO:teuthology.orchestra.run.smithi114.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 3.0 MB/s | 747 kB 00:00 2024-03-06T07:25:17.355 INFO:teuthology.orchestra.run.smithi114.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.6 MB/s | 163 kB 00:00 2024-03-06T07:25:17.456 INFO:teuthology.orchestra.run.smithi114.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.2 MB/s | 127 kB 00:00 2024-03-06T07:25:17.774 INFO:teuthology.orchestra.run.smithi177.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-03-06T07:25:17.807 INFO:teuthology.orchestra.run.smithi177.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:25:17.916 INFO:teuthology.orchestra.run.smithi177.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:25:17.964 INFO:teuthology.orchestra.run.smithi114.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 987 kB/s | 501 kB 00:00 2024-03-06T07:25:18.025 INFO:teuthology.orchestra.run.smithi177.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-06T07:25:18.059 INFO:teuthology.orchestra.run.smithi177.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:25:18.101 INFO:teuthology.orchestra.run.smithi177.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-03-06T07:25:18.134 INFO:teuthology.orchestra.run.smithi177.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.1 MB/s | 69 kB 00:00 2024-03-06T07:25:18.168 INFO:teuthology.orchestra.run.smithi177.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-03-06T07:25:18.210 INFO:teuthology.orchestra.run.smithi177.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-03-06T07:25:18.274 INFO:teuthology.orchestra.run.smithi177.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-03-06T07:25:18.335 INFO:teuthology.orchestra.run.smithi177.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:25:18.386 INFO:teuthology.orchestra.run.smithi177.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.6 MB/s | 433 kB 00:00 2024-03-06T07:25:18.419 INFO:teuthology.orchestra.run.smithi177.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 676 kB/s | 22 kB 00:00 2024-03-06T07:25:18.461 INFO:teuthology.orchestra.run.smithi177.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 107 kB 00:00 2024-03-06T07:25:18.503 INFO:teuthology.orchestra.run.smithi177.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.7 MB/s | 154 kB 00:00 2024-03-06T07:25:18.545 INFO:teuthology.orchestra.run.smithi177.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-03-06T07:25:18.579 INFO:teuthology.orchestra.run.smithi177.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.9 MB/s | 63 kB 00:00 2024-03-06T07:25:18.612 INFO:teuthology.orchestra.run.smithi177.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 845 kB/s | 28 kB 00:00 2024-03-06T07:25:18.729 INFO:teuthology.orchestra.run.smithi177.stdout:(168/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 2.0 MB/s | 3.1 MB 00:01 2024-03-06T07:25:18.780 INFO:teuthology.orchestra.run.smithi177.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-03-06T07:25:18.865 INFO:teuthology.orchestra.run.smithi177.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-06T07:25:18.947 INFO:teuthology.orchestra.run.smithi177.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-06T07:25:18.980 INFO:teuthology.orchestra.run.smithi177.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 4.1 MB/s | 438 kB 00:00 2024-03-06T07:25:19.005 INFO:teuthology.orchestra.run.smithi177.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 557 kB/s | 30 kB 00:00 2024-03-06T07:25:19.039 INFO:teuthology.orchestra.run.smithi177.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2024-03-06T07:25:19.114 INFO:teuthology.orchestra.run.smithi177.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 10 MB/s | 1.1 MB 00:00 2024-03-06T07:25:19.315 INFO:teuthology.orchestra.run.smithi114.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 3.8 MB/s | 5.1 MB 00:01 2024-03-06T07:25:19.406 INFO:teuthology.orchestra.run.smithi114.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 498 kB/s | 45 kB 00:00 2024-03-06T07:25:19.490 INFO:teuthology.orchestra.run.smithi114.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.6 MB/s | 131 kB 00:00 2024-03-06T07:25:19.582 INFO:teuthology.orchestra.run.smithi114.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.7 MB/s | 162 kB 00:00 2024-03-06T07:25:19.664 INFO:teuthology.orchestra.run.smithi177.stdout:(176/176): librados2-19.0.0-1739.g1bc250c7.el9. 1.3 MB/s | 3.4 MB 00:02 2024-03-06T07:25:19.679 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:25:19.679 INFO:teuthology.orchestra.run.smithi177.stdout:Total 7.6 MB/s | 224 MB 00:29 2024-03-06T07:25:19.732 INFO:teuthology.orchestra.run.smithi114.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 2.1 MB/s | 321 kB 00:00 2024-03-06T07:25:19.810 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-06T07:25:19.874 INFO:teuthology.orchestra.run.smithi114.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 2.1 MB/s | 298 kB 00:00 2024-03-06T07:25:19.958 INFO:teuthology.orchestra.run.smithi114.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.2 MB/s | 99 kB 00:00 2024-03-06T07:25:20.058 INFO:teuthology.orchestra.run.smithi114.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 858 kB/s | 85 kB 00:00 2024-03-06T07:25:20.183 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-06T07:25:20.183 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-06T07:25:20.900 INFO:teuthology.orchestra.run.smithi114.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 3.7 MB/s | 3.1 MB 00:00 2024-03-06T07:25:21.000 INFO:teuthology.orchestra.run.smithi114.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 1.7 MB/s | 171 kB 00:00 2024-03-06T07:25:21.101 INFO:teuthology.orchestra.run.smithi114.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 317 kB/s | 31 kB 00:00 2024-03-06T07:25:21.201 INFO:teuthology.orchestra.run.smithi114.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.4 MB/s | 138 kB 00:00 2024-03-06T07:25:21.618 INFO:teuthology.orchestra.run.smithi114.stdout:(30/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 1.5 MB/s | 9.8 MB 00:06 2024-03-06T07:25:21.835 INFO:teuthology.orchestra.run.smithi114.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 2.8 MB/s | 1.8 MB 00:00 2024-03-06T07:25:22.010 INFO:teuthology.orchestra.run.smithi114.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.4 MB/s | 248 kB 00:00 2024-03-06T07:25:22.102 INFO:teuthology.orchestra.run.smithi114.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 543 kB/s | 49 kB 00:00 2024-03-06T07:25:22.186 INFO:teuthology.orchestra.run.smithi114.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 187 kB/s | 15 kB 00:00 2024-03-06T07:25:22.303 INFO:teuthology.orchestra.run.smithi114.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 2.2 MB/s | 264 kB 00:00 2024-03-06T07:25:22.423 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-06T07:25:22.423 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-06T07:25:22.662 INFO:teuthology.orchestra.run.smithi114.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 2.0 MB/s | 748 kB 00:00 2024-03-06T07:25:22.745 INFO:teuthology.orchestra.run.smithi114.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 221 kB/s | 18 kB 00:00 2024-03-06T07:25:22.937 INFO:teuthology.orchestra.run.smithi114.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-06T07:25:22.996 INFO:teuthology.orchestra.run.smithi114.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 920 kB 00:00 2024-03-06T07:25:23.038 INFO:teuthology.orchestra.run.smithi114.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-06T07:25:23.071 INFO:teuthology.orchestra.run.smithi114.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 630 kB/s | 20 kB 00:00 2024-03-06T07:25:23.113 INFO:teuthology.orchestra.run.smithi114.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2024-03-06T07:25:23.606 INFO:teuthology.orchestra.run.smithi114.stdout:(43/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 6.9 MB/s | 47 MB 00:06 2024-03-06T07:25:23.631 INFO:teuthology.orchestra.run.smithi114.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 61 kB/s | 32 kB 00:00 2024-03-06T07:25:23.739 INFO:teuthology.orchestra.run.smithi114.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2024-03-06T07:25:23.764 INFO:teuthology.orchestra.run.smithi114.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 147 kB/s | 17 kB 00:00 2024-03-06T07:25:23.806 INFO:teuthology.orchestra.run.smithi114.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.9 MB/s | 159 kB 00:00 2024-03-06T07:25:23.823 INFO:teuthology.orchestra.run.smithi114.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 175 kB/s | 10 kB 00:00 2024-03-06T07:25:23.840 INFO:teuthology.orchestra.run.smithi114.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:25:23.891 INFO:teuthology.orchestra.run.smithi114.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-06T07:25:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2024-03-06T07:25:23.950 INFO:teuthology.orchestra.run.smithi114.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 460 kB/s | 24 kB 00:00 2024-03-06T07:25:23.967 INFO:teuthology.orchestra.run.smithi114.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-06T07:25:24.000 INFO:teuthology.orchestra.run.smithi114.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 282 kB 00:00 2024-03-06T07:25:24.025 INFO:teuthology.orchestra.run.smithi114.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.3 MB/s | 187 kB 00:00 2024-03-06T07:25:24.042 INFO:teuthology.orchestra.run.smithi114.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 859 kB/s | 34 kB 00:00 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi114.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-06T07:25:24.287 INFO:teuthology.orchestra.run.smithi114.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-06T07:25:24.352 INFO:teuthology.orchestra.run.smithi114.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 9.8 MB/s | 1.5 MB 00:00 2024-03-06T07:25:24.428 INFO:teuthology.orchestra.run.smithi114.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 6.2 MB/s | 811 kB 00:00 2024-03-06T07:25:24.453 INFO:teuthology.orchestra.run.smithi114.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-06T07:25:24.603 INFO:teuthology.orchestra.run.smithi114.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-03-06T07:25:24.645 INFO:teuthology.orchestra.run.smithi114.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-06T07:25:24.720 INFO:teuthology.orchestra.run.smithi114.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 78 kB 00:00 2024-03-06T07:25:24.812 INFO:teuthology.orchestra.run.smithi114.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-06T07:25:24.845 INFO:teuthology.orchestra.run.smithi114.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 227 kB 00:00 2024-03-06T07:25:24.862 INFO:teuthology.orchestra.run.smithi114.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 24 kB 00:00 2024-03-06T07:25:24.880 INFO:teuthology.orchestra.run.smithi114.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 549 kB/s | 18 kB 00:00 2024-03-06T07:25:24.905 INFO:teuthology.orchestra.run.smithi114.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-06T07:25:24.930 INFO:teuthology.orchestra.run.smithi114.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 655 kB/s | 32 kB 00:00 2024-03-06T07:25:25.056 INFO:teuthology.orchestra.run.smithi114.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi114.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-06T07:25:25.122 INFO:teuthology.orchestra.run.smithi114.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-06T07:25:25.148 INFO:teuthology.orchestra.run.smithi114.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-06T07:25:25.190 INFO:teuthology.orchestra.run.smithi114.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2024-03-06T07:25:25.215 INFO:teuthology.orchestra.run.smithi114.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-06T07:25:25.257 INFO:teuthology.orchestra.run.smithi114.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-03-06T07:25:25.282 INFO:teuthology.orchestra.run.smithi114.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-06T07:25:25.316 INFO:teuthology.orchestra.run.smithi114.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-06T07:25:25.341 INFO:teuthology.orchestra.run.smithi114.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 748 kB/s | 43 kB 00:00 2024-03-06T07:25:25.400 INFO:teuthology.orchestra.run.smithi114.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 678 kB/s | 39 kB 00:00 2024-03-06T07:25:25.725 INFO:teuthology.orchestra.run.smithi114.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-06T07:25:25.750 INFO:teuthology.orchestra.run.smithi114.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 122 kB/s | 41 kB 00:00 2024-03-06T07:25:25.792 INFO:teuthology.orchestra.run.smithi114.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 5.2 MB/s | 215 kB 00:00 2024-03-06T07:25:25.809 INFO:teuthology.orchestra.run.smithi114.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 594 kB/s | 34 kB 00:00 2024-03-06T07:25:25.842 INFO:teuthology.orchestra.run.smithi114.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:25:25.876 INFO:teuthology.orchestra.run.smithi114.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi114.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.2 MB/s | 56 kB 00:00 2024-03-06T07:25:25.919 INFO:teuthology.orchestra.run.smithi114.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.1 MB/s | 87 kB 00:00 2024-03-06T07:25:25.936 INFO:teuthology.orchestra.run.smithi114.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-06T07:25:25.961 INFO:teuthology.orchestra.run.smithi114.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-06T07:25:25.986 INFO:teuthology.orchestra.run.smithi114.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-06T07:25:26.020 INFO:teuthology.orchestra.run.smithi114.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-03-06T07:25:26.062 INFO:teuthology.orchestra.run.smithi114.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-03-06T07:25:26.095 INFO:teuthology.orchestra.run.smithi114.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 194 kB 00:00 2024-03-06T07:25:26.130 INFO:teuthology.orchestra.run.smithi114.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-06T07:25:26.163 INFO:teuthology.orchestra.run.smithi114.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 241 kB 00:00 2024-03-06T07:25:26.213 INFO:teuthology.orchestra.run.smithi114.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-06T07:25:26.238 INFO:teuthology.orchestra.run.smithi114.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-06T07:25:26.264 INFO:teuthology.orchestra.run.smithi114.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.5 MB/s | 124 kB 00:00 2024-03-06T07:25:26.289 INFO:teuthology.orchestra.run.smithi114.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.3 MB/s | 113 kB 00:00 2024-03-06T07:25:26.314 INFO:teuthology.orchestra.run.smithi114.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.8 MB/s | 187 kB 00:00 2024-03-06T07:25:26.481 INFO:teuthology.orchestra.run.smithi114.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 135 kB/s | 22 kB 00:00 2024-03-06T07:25:26.631 INFO:teuthology.orchestra.run.smithi114.stdout:(104/176): ceph-mgr-diskprediction-local-19.0.0 1.5 MB/s | 7.4 MB 00:05 2024-03-06T07:25:26.665 INFO:teuthology.orchestra.run.smithi114.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 657 kB/s | 245 kB 00:00 2024-03-06T07:25:26.841 INFO:teuthology.orchestra.run.smithi114.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-03-06T07:25:27.116 INFO:teuthology.orchestra.run.smithi114.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-06T07:25:27.174 INFO:teuthology.orchestra.run.smithi114.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-06T07:25:27.208 INFO:teuthology.orchestra.run.smithi114.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 570 kB/s | 309 kB 00:00 2024-03-06T07:25:27.250 INFO:teuthology.orchestra.run.smithi114.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 348 kB/s | 26 kB 00:00 2024-03-06T07:25:27.284 INFO:teuthology.orchestra.run.smithi114.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 651 kB/s | 49 kB 00:00 2024-03-06T07:25:27.350 INFO:teuthology.orchestra.run.smithi114.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 677 kB/s | 67 kB 00:00 2024-03-06T07:25:27.400 INFO:teuthology.orchestra.run.smithi114.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-06T07:25:27.492 INFO:teuthology.orchestra.run.smithi114.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-06T07:25:27.568 INFO:teuthology.orchestra.run.smithi114.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-03-06T07:25:27.602 INFO:teuthology.orchestra.run.smithi114.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-03-06T07:25:27.711 INFO:teuthology.orchestra.run.smithi114.stdout:(117/176): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-03-06T07:25:27.736 INFO:teuthology.orchestra.run.smithi114.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 112 kB/s | 15 kB 00:00 2024-03-06T07:25:27.761 INFO:teuthology.orchestra.run.smithi114.stdout:(119/176): python3-cachetools-4.2.4-1.el9.noarc 168 kB/s | 32 kB 00:00 2024-03-06T07:25:27.794 INFO:teuthology.orchestra.run.smithi114.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-06T07:25:27.828 INFO:teuthology.orchestra.run.smithi114.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-06T07:25:27.862 INFO:teuthology.orchestra.run.smithi114.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-06T07:25:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:(123/176): python3-colorama-0.4.4-12.el9.noarch 385 kB/s | 35 kB 00:00 2024-03-06T07:25:27.921 INFO:teuthology.orchestra.run.smithi114.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-06T07:25:27.962 INFO:teuthology.orchestra.run.smithi114.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-06T07:25:27.996 INFO:teuthology.orchestra.run.smithi114.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-03-06T07:25:28.038 INFO:teuthology.orchestra.run.smithi114.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2024-03-06T07:25:28.105 INFO:teuthology.orchestra.run.smithi114.stdout:(128/176): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-03-06T07:25:28.131 INFO:teuthology.orchestra.run.smithi114.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 174 kB/s | 23 kB 00:00 2024-03-06T07:25:28.156 INFO:teuthology.orchestra.run.smithi114.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 167 kB/s | 19 kB 00:00 2024-03-06T07:25:28.181 INFO:teuthology.orchestra.run.smithi114.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 264 kB/s | 20 kB 00:00 2024-03-06T07:25:28.231 INFO:teuthology.orchestra.run.smithi114.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 626 kB/s | 46 kB 00:00 2024-03-06T07:25:28.256 INFO:teuthology.orchestra.run.smithi114.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-06T07:25:28.315 INFO:teuthology.orchestra.run.smithi114.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 6.5 MB/s | 1.0 MB 00:00 2024-03-06T07:25:28.340 INFO:teuthology.orchestra.run.smithi114.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 535 kB/s | 58 kB 00:00 2024-03-06T07:25:28.382 INFO:teuthology.orchestra.run.smithi114.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-06T07:25:28.424 INFO:teuthology.orchestra.run.smithi114.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 152 kB/s | 16 kB 00:00 2024-03-06T07:25:28.449 INFO:teuthology.orchestra.run.smithi114.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 833 kB/s | 90 kB 00:00 2024-03-06T07:25:28.474 INFO:teuthology.orchestra.run.smithi114.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 308 kB/s | 28 kB 00:00 2024-03-06T07:25:28.500 INFO:teuthology.orchestra.run.smithi114.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 411 kB/s | 31 kB 00:00 2024-03-06T07:25:28.525 INFO:teuthology.orchestra.run.smithi114.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-06T07:25:28.551 INFO:teuthology.orchestra.run.smithi114.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-03-06T07:25:28.576 INFO:teuthology.orchestra.run.smithi114.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 480 kB/s | 36 kB 00:00 2024-03-06T07:25:28.601 INFO:teuthology.orchestra.run.smithi114.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-06T07:25:28.635 INFO:teuthology.orchestra.run.smithi114.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-06T07:25:28.660 INFO:teuthology.orchestra.run.smithi114.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-06T07:25:28.694 INFO:teuthology.orchestra.run.smithi114.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-06T07:25:28.711 INFO:teuthology.orchestra.run.smithi114.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-06T07:25:28.761 INFO:teuthology.orchestra.run.smithi114.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-06T07:25:28.837 INFO:teuthology.orchestra.run.smithi114.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-06T07:25:28.870 INFO:teuthology.orchestra.run.smithi114.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:25:28.971 INFO:teuthology.orchestra.run.smithi114.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-03-06T07:25:29.054 INFO:teuthology.orchestra.run.smithi114.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-06T07:25:29.088 INFO:teuthology.orchestra.run.smithi114.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:25:29.128 INFO:teuthology.orchestra.run.smithi114.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 6.2 MB/s | 247 kB 00:00 2024-03-06T07:25:29.170 INFO:teuthology.orchestra.run.smithi114.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 69 kB 00:00 2024-03-06T07:25:29.203 INFO:teuthology.orchestra.run.smithi114.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-03-06T07:25:29.246 INFO:teuthology.orchestra.run.smithi114.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:25:29.305 INFO:teuthology.orchestra.run.smithi114.stdout:(159/176): librados2-19.0.0-1739.g1bc250c7.el9. 5.7 MB/s | 3.4 MB 00:00 2024-03-06T07:25:29.338 INFO:teuthology.orchestra.run.smithi114.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 254 kB 00:00 2024-03-06T07:25:29.380 INFO:teuthology.orchestra.run.smithi114.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-03-06T07:25:29.414 INFO:teuthology.orchestra.run.smithi114.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 6.2 MB/s | 433 kB 00:00 2024-03-06T07:25:29.431 INFO:teuthology.orchestra.run.smithi114.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 479 kB/s | 22 kB 00:00 2024-03-06T07:25:29.456 INFO:teuthology.orchestra.run.smithi114.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 107 kB 00:00 2024-03-06T07:25:29.481 INFO:teuthology.orchestra.run.smithi114.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 154 kB 00:00 2024-03-06T07:25:29.507 INFO:teuthology.orchestra.run.smithi114.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.7 MB/s | 135 kB 00:00 2024-03-06T07:25:29.532 INFO:teuthology.orchestra.run.smithi114.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.3 MB/s | 63 kB 00:00 2024-03-06T07:25:29.557 INFO:teuthology.orchestra.run.smithi114.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 564 kB/s | 28 kB 00:00 2024-03-06T07:25:29.616 INFO:teuthology.orchestra.run.smithi114.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi114.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-06T07:25:29.775 INFO:teuthology.orchestra.run.smithi114.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-06T07:25:29.808 INFO:teuthology.orchestra.run.smithi114.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 438 kB 00:00 2024-03-06T07:25:29.859 INFO:teuthology.orchestra.run.smithi114.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 380 kB/s | 30 kB 00:00 2024-03-06T07:25:29.892 INFO:teuthology.orchestra.run.smithi114.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 3.8 MB/s | 321 kB 00:00 2024-03-06T07:25:29.934 INFO:teuthology.orchestra.run.smithi114.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 15 MB/s | 1.1 MB 00:00 2024-03-06T07:25:30.183 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-06T07:25:30.318 INFO:teuthology.orchestra.run.smithi114.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 2.0 MB/s | 3.1 MB 00:01 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:25:30.332 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:25:30.332 INFO:teuthology.orchestra.run.smithi114.stdout:Total 11 MB/s | 224 MB 00:20 2024-03-06T07:25:30.369 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:25:30.408 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:30.459 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-06T07:25:30.489 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:30.559 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:25:30.643 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:25:30.773 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:25:30.831 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-06T07:25:30.831 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-06T07:25:30.834 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:25:30.874 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:25:30.963 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:25:31.375 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:25:31.413 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:31.589 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:31.614 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:31.734 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:31.779 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:25:32.022 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:25:32.085 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:25:32.110 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:32.258 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:32.286 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:32.366 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:32.420 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:25:32.446 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:32.518 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:32.845 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:32.895 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:25:33.084 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-06T07:25:33.085 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-06T07:25:33.299 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:25:33.503 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:33.592 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:33.802 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:33.856 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:34.232 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:34.423 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:25:34.549 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:25:34.586 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:25:34.685 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:25:34.757 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:25:34.807 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:25:34.867 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:25:34.920 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:25:35.034 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:25:35.077 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:25:35.142 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:25:35.215 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:25:35.263 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:25:35.386 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:25:35.424 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:25:35.480 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:25:35.514 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:25:35.632 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:25:35.693 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:25:35.790 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:25:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:25:36.044 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:25:36.743 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:25:36.861 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:25:36.981 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:25:37.034 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:37.058 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:37.111 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:25:37.161 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:25:37.238 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:25:37.446 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:25:37.505 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:25:37.876 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:25:38.078 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:25:38.174 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:25:38.229 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:25:38.445 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:25:38.757 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:25:38.836 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:25:38.923 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:25:38.980 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:25:39.044 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:25:39.113 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:25:39.153 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:25:39.224 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:25:39.299 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:25:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:25:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:25:39.571 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:25:39.621 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:25:39.751 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:25:39.809 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:25:39.859 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:25:39.903 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:25:39.936 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:25:40.109 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:25:40.288 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:25:40.430 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:25:40.479 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:25:40.549 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:25:40.621 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-06T07:25:40.622 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:25:40.737 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:25:40.774 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:25:40.803 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:40.876 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:25:40.892 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:40.933 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:25:40.947 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:25:41.007 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:25:41.100 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:25:41.107 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:25:41.160 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:25:41.163 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:25:41.194 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:25:41.204 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:25:41.246 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:25:41.288 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:25:41.302 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:25:41.346 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:25:41.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:25:41.474 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:25:41.555 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:25:41.608 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:25:41.665 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:25:41.686 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:25:41.736 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:41.795 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:25:41.866 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:25:41.913 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:25:41.918 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:41.946 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:41.978 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:25:42.023 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:25:42.065 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:42.105 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:25:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:25:42.286 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:25:42.351 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:25:42.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:25:42.424 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:25:42.448 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:25:42.450 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:42.594 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:25:42.610 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:42.643 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:42.652 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:25:42.732 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:42.786 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:25:42.811 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:42.872 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:43.187 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:43.244 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:43.286 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:25:43.617 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:25:43.623 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:25:43.668 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:25:43.701 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:25:43.757 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:25:43.795 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:43.868 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:25:43.882 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:43.927 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:25:44.083 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:44.155 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:25:44.183 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:25:44.284 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:25:44.474 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:44.659 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:25:44.791 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:25:44.828 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:25:44.928 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:25:44.995 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:25:45.032 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:25:45.094 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:25:45.119 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:25:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:25:45.228 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:25:45.270 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:25:45.318 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:25:45.382 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:25:45.430 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:25:45.462 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:25:45.559 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:25:45.598 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:25:45.647 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:25:45.680 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:25:45.792 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:25:45.943 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:25:46.122 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:25:46.212 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:25:46.750 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:25:46.830 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:25:46.904 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:25:47.002 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:25:47.111 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:25:47.169 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:47.225 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:25:47.267 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:47.268 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:25:47.372 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:25:47.386 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:47.415 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:25:47.471 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:25:47.565 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:25:47.615 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:25:47.970 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:25:47.986 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:25:48.012 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:48.094 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:48.122 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:25:48.228 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:25:48.319 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:25:48.402 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:25:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:25:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:25:48.758 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:25:48.850 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:25:48.897 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:25:48.937 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:25:49.004 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:25:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:25:49.118 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:25:49.184 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:25:49.376 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:25:49.448 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:25:49.491 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:25:49.550 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:25:49.692 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:25:49.731 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:25:49.776 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:25:49.822 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:25:49.855 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:25:50.016 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:25:50.198 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:25:50.330 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:25:50.373 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:25:50.435 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:25:50.509 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:25:50.760 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:50.762 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:25:50.803 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:50.819 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:25:50.987 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:25:51.040 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:25:51.082 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:25:51.131 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:25:51.171 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:25:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:25:51.257 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:25:51.305 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:25:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:25:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:25:51.427 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:25:51.473 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:51.594 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:25:51.625 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:51.625 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:25:51.625 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:25:51.625 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:25:51.673 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:25:51.682 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:51.726 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:25:51.768 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:25:51.897 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:25:52.008 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:25:52.070 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:25:52.107 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:25:52.163 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:25:52.309 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:25:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:25:53.273 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:25:53.890 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:25:53.944 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:25:54.037 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:25:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:25:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:25:54.496 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:25:54.598 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:25:55.465 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:25:55.601 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:25:57.157 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:25:57.235 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:25:57.269 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:25:57.620 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:25:57.655 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:57.774 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:57.803 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:25:57.857 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:25:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:25:58.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:58.473 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:58.502 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:26:01.049 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:26:01.098 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:26:01.718 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:26:01.760 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:26:01.911 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:26:01.912 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:26:01.912 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:26:01.912 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:01.953 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:26:04.805 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:26:04.805 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-03-06T07:26:04.805 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-03-06T07:26:04.805 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-03-06T07:26:04.805 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-03-06T07:26:04.805 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-03-06T07:26:04.806 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-03-06T07:26:04.806 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-03-06T07:26:04.806 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:05.044 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:05.091 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:05.091 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:05.092 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:26:05.092 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:06.762 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:06.840 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:06.840 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:06.840 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:26:06.840 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:06.841 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:06.841 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:07.089 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:07.156 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:07.156 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:07.156 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:26:07.156 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:07.156 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:07.157 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:07.639 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:07.707 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:07.707 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:07.708 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:26:07.708 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:07.708 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:07.708 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:07.825 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:26:08.432 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:08.475 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:08.525 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:26:08.630 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:26:08.662 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:09.005 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:10.093 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:10.138 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:10.289 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:10.468 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:26:10.507 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:10.575 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:10.575 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:10.575 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:26:10.575 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:10.575 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:10.575 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:10.651 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:10.688 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:10.780 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:26:11.763 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:26:11.801 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:11.869 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:11.869 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:11.870 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:26:11.870 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:11.870 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:11.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:11.936 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:12.007 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:12.007 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:12.008 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:26:12.008 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:12.095 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:12.172 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:12.172 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:12.172 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:26:12.173 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:12.173 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:12.173 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:12.562 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:12.635 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:12.635 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:12.635 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:26:12.635 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:12.635 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:12.635 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-03-06T07:26:14.810 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-03-06T07:26:14.811 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:15.023 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:15.069 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:26:15.070 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:15.070 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:26:15.070 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:16.719 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:16.799 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:26:16.799 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:16.799 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:26:16.799 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:16.799 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:26:16.799 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:17.112 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:26:17.112 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:17.112 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:26:17.112 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:17.112 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:26:17.112 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:17.443 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:26:17.496 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:26:17.532 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:26:17.571 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:26:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:17.678 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:26:17.678 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:17.678 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:26:17.678 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:17.678 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:26:17.678 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:17.789 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:26:17.857 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:26:17.990 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:26:18.042 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:26:18.096 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:26:18.158 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:26:18.296 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:26:18.335 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:26:18.357 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:18.373 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:26:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:26:18.409 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:26:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:26:18.469 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:26:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:26:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:18.594 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:26:18.687 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:26:18.736 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:26:18.908 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:26:18.944 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:19.637 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:19.683 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:19.734 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:20.062 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:26:20.086 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:20.229 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:26:20.399 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:26:20.430 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:20.498 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:26:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:26:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:20.498 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:26:20.498 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:20.555 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:20.592 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:26:20.652 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:26:21.067 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:26:21.145 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:26:21.198 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:26:21.332 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:26:21.373 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:26:21.441 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:26:21.507 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:26:21.508 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:21.584 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:26:21.616 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:21.640 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:21.690 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:26:21.691 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:26:21.691 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:21.691 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:26:21.691 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:21.691 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:26:21.691 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:21.745 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:26:21.745 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:21.758 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:21.828 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:26:21.828 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:21.828 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:26:21.828 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:21.892 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:21.900 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:21.953 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:26:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:26:22.099 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:26:22.149 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:26:22.225 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:26:22.329 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:26:22.335 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:22.371 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:26:22.404 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:26:22.404 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:22.405 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:26:22.405 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:26:22.405 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:22.405 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:26:22.405 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:22.438 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:26:22.480 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:26:22.549 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:22.549 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:22.880 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:22.881 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:23.019 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:23.072 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:23.130 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:26:23.166 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:26:23.203 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:26:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:23.592 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:23.592 INFO:teuthology.orchestra.run.smithi177.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:26:23.592 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:27.182 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:26:27.226 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:26:27.271 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:26:27.309 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:26:27.584 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:26:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:26:27.764 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:26:27.821 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:26:27.881 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:26:27.988 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:26:28.032 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:26:28.078 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:26:28.107 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:26:28.158 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:26:28.280 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:26:28.327 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:26:28.327 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:28.328 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:26:28.329 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:26:28.330 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:26:28.346 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:26:28.347 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:26:28.348 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:26:28.349 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:26:28.350 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:26:28.351 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:26:28.352 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:26:28.353 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:26:28.354 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:26:28.355 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:26:28.356 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:26:28.357 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:26:28.358 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:28.359 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:26:28.360 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:26:28.360 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:26:28.379 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:26:28.416 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:26:29.274 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:29.321 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:29.377 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:30.717 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:26:30.793 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:26:30.839 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:26:30.952 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:26:30.973 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:26:30.973 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:30.973 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-03-06T07:26:30.973 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:30.973 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:30.973 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.974 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:26:30.975 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.976 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:26:30.977 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:26:30.978 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:26:30.980 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:26:30.981 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:26:30.982 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.983 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:26:30.984 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:26:30.985 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:26:30.986 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:26:30.986 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:30.986 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-06T07:26:30.987 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:26:31.020 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:26:31.046 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:26:31.086 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:26:31.086 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:31.271 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:26:31.317 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:26:31.317 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:31.447 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:31.500 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:26:31.548 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:26:31.610 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:26:31.679 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:26:31.701 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:31.721 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:26:31.793 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:26:31.876 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:26:31.926 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:26:31.958 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:26:31.993 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:26:32.026 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:32.059 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:26:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:32.435 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:32.435 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:32.557 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:32.609 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:32.646 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:26:32.712 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:26:32.749 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:26:32.874 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:33.121 INFO:teuthology.orchestra.run.smithi114.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:26:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:37.976 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:26:37.976 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:26:37.979 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:26:37.980 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:26:37.983 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:26:37.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:26:37.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:26:37.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:26:37.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:26:37.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:26:37.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:26:37.988 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:26:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:26:37.990 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:26:37.991 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:26:37.992 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:37.993 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:26:37.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:26:37.994 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:26:40.478 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:40.479 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.480 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.481 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:26:40.482 INFO:teuthology.orchestra.run.smithi114.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:26:40.483 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:26:40.484 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:26:40.485 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:26:40.486 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:26:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:26:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:26:40.489 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:26:40.490 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:26:40.491 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:40.491 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-06T07:26:41.096 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:41.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:26:41.316 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:41.378 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:41.378 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:26:41.380 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:26:41.523 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:26:41.595 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:41.595 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:41.595 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:26:41.597 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:26:41.597 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:41.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:26:41.677 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:41.677 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:26:41.779 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:26:41.779 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:41.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:26:41.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:26:41.975 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:41.975 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:26:42.075 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:26:42.170 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:26:42.170 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:42.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:26:42.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:26:42.365 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:42.365 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:26:42.461 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:26:42.600 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:26:42.600 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:42.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:26:42.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:26:42.794 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:42.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:26:42.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:26:42.977 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:26:43.058 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:26:43.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2024-03-06T07:26:43.063 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-03-06T07:26:43.160 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:26:43.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:26:43.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:26:43.257 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:26:43.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:26:43.261 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:26:43.357 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:26:43.357 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:26:43.357 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:26:43.357 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:43.357 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:26:43.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:43.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:43.668 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:43.668 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:26:43.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:43.992 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:44.097 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:26:44.097 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:26:44.097 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583067', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583067', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'kclient': {'syntax': 'v2'}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ54/aN33i6HtJNgf9pJmoUcHsB2S7lNYez9DNZQ1tlc71X7eIK+eZWdJb1+0cdpm8yd0reja+HvCardZYjPXb8=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKGfW1Jlke62EgM/9jUGOrGX2abc4En0/A3grQO8zHtWE/0R93RfUnIhGf9EFalS/rWao6ByiVrAHEI1uP/MYNM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:26:44.098 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:26:44.102 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:26:44.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:26:44.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:26:44.288 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:44.288 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:26:44.325 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:26:44.325 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-06 07:26:33.559731890 +0000 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-06 07:23:33.106569767 +0000 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-06 07:23:33.106569767 +0000 2024-03-06T07:26:44.403 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-06 07:23:33.106569767 +0000 2024-03-06T07:26:44.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:26:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-06T07:26:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-06T07:26:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000312649 s, 1.6 MB/s 2024-03-06T07:26:44.509 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:26:44.550 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-06 07:26:33.559731890 +0000 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-06 07:23:33.676557949 +0000 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-06 07:23:33.676557949 +0000 2024-03-06T07:26:44.627 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-06 07:23:33.676557949 +0000 2024-03-06T07:26:44.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:26:44.733 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-06T07:26:44.733 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-06T07:26:44.733 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000273139 s, 1.9 MB/s 2024-03-06T07:26:44.736 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:26:44.775 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-03-06T07:26:44.853 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-06 07:26:33.560731869 +0000 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-06 07:23:34.245546155 +0000 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-06 07:23:34.245546155 +0000 2024-03-06T07:26:44.854 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-06 07:23:34.245546155 +0000 2024-03-06T07:26:44.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:26:44.957 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-06T07:26:44.957 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-06T07:26:44.957 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000264952 s, 1.9 MB/s 2024-03-06T07:26:44.961 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:26:45.001 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-03-06T07:26:45.080 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:26:45.080 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:45.080 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-06T07:26:45.080 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:45.080 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:45.080 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-06 07:26:33.561731849 +0000 2024-03-06T07:26:45.080 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-06 07:23:34.805534549 +0000 2024-03-06T07:26:45.081 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-06 07:23:34.805534549 +0000 2024-03-06T07:26:45.081 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-06 07:23:34.805534549 +0000 2024-03-06T07:26:45.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:26:45.188 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-06T07:26:45.188 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-06T07:26:45.188 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000268223 s, 1.9 MB/s 2024-03-06T07:26:45.192 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:26:45.230 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:45.231 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:45.231 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:26:45.285 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:26:45.286 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-03-06T07:26:45.337 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 639 Links: 1 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-06 07:26:24.022834108 +0000 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-06 07:23:33.189290376 +0000 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-06 07:23:33.189290376 +0000 2024-03-06T07:26:45.338 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-06 07:23:33.189290376 +0000 2024-03-06T07:26:45.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:26:45.418 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-06T07:26:45.418 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-06T07:26:45.419 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.00034518 s, 1.5 MB/s 2024-03-06T07:26:45.422 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:26:45.475 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-03-06T07:26:45.525 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:26:45.525 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:45.525 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 656 Links: 1 2024-03-06T07:26:45.526 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:45.526 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:45.526 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-06 07:26:24.022834108 +0000 2024-03-06T07:26:45.526 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-06 07:23:33.770274740 +0000 2024-03-06T07:26:45.526 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-06 07:23:33.770274740 +0000 2024-03-06T07:26:45.526 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-06 07:23:33.770274740 +0000 2024-03-06T07:26:45.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-06T07:26:45.611 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000383346 s, 1.3 MB/s 2024-03-06T07:26:45.614 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:26:45.667 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-03-06T07:26:45.716 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:26:45.716 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:45.716 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 677 Links: 1 2024-03-06T07:26:45.716 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:45.716 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:45.716 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-06 07:26:24.023834080 +0000 2024-03-06T07:26:45.716 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-06 07:23:34.324259591 +0000 2024-03-06T07:26:45.717 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-06 07:23:34.324259591 +0000 2024-03-06T07:26:45.717 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-06 07:23:34.324259591 +0000 2024-03-06T07:26:45.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:26:45.802 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-06T07:26:45.802 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-06T07:26:45.802 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000326037 s, 1.6 MB/s 2024-03-06T07:26:45.806 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:26:45.863 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-03-06T07:26:45.911 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:26:45.911 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:45.911 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 694 Links: 1 2024-03-06T07:26:45.911 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:45.911 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:45.911 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-06 07:26:24.024834053 +0000 2024-03-06T07:26:45.912 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-06 07:23:34.894243987 +0000 2024-03-06T07:26:45.912 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-06 07:23:34.894243987 +0000 2024-03-06T07:26:45.912 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-06 07:23:34.894243987 +0000 2024-03-06T07:26:45.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:26:45.996 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-06T07:26:45.996 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-06T07:26:45.996 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000335932 s, 1.5 MB/s 2024-03-06T07:26:46.000 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:26:46.051 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:46.051 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:26:46.051 INFO:tasks.ceph:Generating config... 2024-03-06T07:26:46.053 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:26:46.053 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:26:46.053 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:26:46.053 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:26:46.053 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:26:46.053 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:26:46.053 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-06T07:26:46.053 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:26:46.053 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:26:46.054 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:26:46.055 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:26:46.056 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:26:46.056 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:26:46.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:26:46.188 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:26:46.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:26:46.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:46.420 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.114'), ('mon.b', '172.21.15.177'), ('mon.c', '[v2:172.21.15.177:3301,v1:172.21.15.177:6790]')] 2024-03-06T07:26:46.420 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:26:46.421 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:46.421 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:26:46.473 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.114 --add b 172.21.15.177 --addv c '[v2:172.21.15.177:3301,v1:172.21.15.177:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:46.624 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:46.624 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: generated fsid 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:26:46.624 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-03-06T07:26:46.624 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 0 2024-03-06T07:26:46.624 INFO:teuthology.orchestra.run.smithi114.stdout:fsid 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:last_changed 2024-03-06T07:26:46.623450+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-03-06T07:26:46.623450+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:min_mon_release 17 (quincy) 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:election_strategy: 1 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:2: [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] mon.c 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:26:46.628 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:26:46.671 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 149c34a4-b335-4a4b-8a41-0fe21b0db808... 2024-03-06T07:26:46.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:46.719 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.879 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:46.880 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-06T07:26:46.881 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:46.882 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:46.883 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-03-06T07:26:46.884 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-06T07:26:46.885 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:26:46.886 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:46.887 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-03-06T07:26:46.888 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-03-06T07:26:46.889 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:26:46.890 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:26:46.893 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-06T07:26:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-06T07:26:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on skipped message = False 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-06T07:26:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-06T07:26:46.903 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:26:46.939 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:26:46.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:26:47.079 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:26:47.079 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:47.079 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:26:47.127 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:47.128 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:26:47.223 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:26:47.223 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:47.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:26:47.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:47.361 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:26:47.361 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:47.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:26:47.459 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:47.459 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:26:47.459 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:47.609 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:26:47.609 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:47.704 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:26:47.704 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:26:47.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:26:47.887 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:26:47.897 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:26:48.108 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:26:48.120 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:26:48.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:26:48.302 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:26:48.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:26:48.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:26:48.572 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:26:48.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:26:48.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:26:48.860 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:26:48.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:26:48.962 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:26:49.136 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:26:49.143 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:26:49.229 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:26:49.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:26:49.356 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:26:49.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:26:49.543 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:26:49.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:26:49.704 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:26:49.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:26:49.861 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:26:49.911 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:26:49.911 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:26:49.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:50.010 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:50.010 INFO:tasks.ceph:role: osd.0 2024-03-06T07:26:50.010 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:26:50.010 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:26:50.114 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:50.114 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:50.114 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:50.114 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:50.115 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:50.115 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:50.115 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:50.115 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:50.115 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:50.115 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:50.645 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-06T07:26:50.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-06T07:26:50.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:50.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:50.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:51.022 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:51.022 INFO:tasks.ceph:role: osd.1 2024-03-06T07:26:51.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:26:51.022 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:26:51.118 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:51.118 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:51.118 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:51.119 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:51.119 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:51.119 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:51.119 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:51.119 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:51.119 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:51.119 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:51.599 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-06T07:26:51.605 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-06T07:26:51.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:51.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:51.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:51.919 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:51.919 INFO:tasks.ceph:role: osd.2 2024-03-06T07:26:51.919 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:26:51.919 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:52.553 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-06T07:26:52.559 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-06T07:26:52.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:52.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:52.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:52.927 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:52.927 INFO:tasks.ceph:role: osd.3 2024-03-06T07:26:52.927 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:26:52.927 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:53.475 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-06T07:26:53.482 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-06T07:26:53.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:53.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:53.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:53.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:53.860+0000 7f6e97668640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:26:53.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:53.862+0000 7f6e97668640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:26:53.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:53.862+0000 7f6e97668640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:53.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:53.862+0000 7f6e97668640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:53.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:53.862+0000 7f6e97668640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:53.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:53.867+0000 7f6e97668640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:26:57.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:57.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:57.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:57.261+0000 7ff65d393640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:26:57.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:57.261+0000 7ff65d393640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:26:57.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:57.261+0000 7ff65d393640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:57.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:57.262+0000 7ff65d393640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:57.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:57.262+0000 7ff65d393640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:57.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:26:57.264+0000 7ff65d393640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:27:00.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:27:00.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:00.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:00.656+0000 7f9692248640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:27:00.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:00.656+0000 7f9692248640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:27:00.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:00.656+0000 7f9692248640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:27:00.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:00.657+0000 7f9692248640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:27:00.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:00.657+0000 7f9692248640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:27:00.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:00.659+0000 7f9692248640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:27:03.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:27:03.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:04.080+0000 7fda55611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:04.081+0000 7fda55611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:04.081+0000 7fda55611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:04.081+0000 7fda55611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:27:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:04.081+0000 7fda55611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:27:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:04.083+0000 7fda55611640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:27:07.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:27:07.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:27:07.239 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:07.239 INFO:tasks.ceph:role: osd.4 2024-03-06T07:27:07.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:27:07.239 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:27:07.338 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:07.339 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:07.849 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-06T07:27:07.855 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-06T07:27:07.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:27:08.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:27:08.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:08.226 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:08.226 INFO:tasks.ceph:role: osd.5 2024-03-06T07:27:08.226 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:27:08.227 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:27:08.316 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:08.316 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:08.316 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:08.316 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:08.317 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:08.800 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-06T07:27:08.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-06T07:27:08.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:08.930 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:09.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:09.105 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:09.105 INFO:tasks.ceph:role: osd.6 2024-03-06T07:27:09.105 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:27:09.105 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:27:09.202 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:09.203 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:09.715 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-06T07:27:09.721 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-06T07:27:09.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:09.909 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:10.005 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:10.094 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:27:10.094 INFO:tasks.ceph:role: osd.7 2024-03-06T07:27:10.094 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:27:10.094 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:27:10.191 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:27:10.192 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:27:10.192 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-03-06T07:27:10.654 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-03-06T07:27:10.655 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:10.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:10.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:11.059 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:11.057+0000 7f464f011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:27:11.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:11.063+0000 7f464f011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:27:11.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:11.063+0000 7f464f011640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:27:11.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:11.064+0000 7f464f011640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:27:11.065 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:11.064+0000 7f464f011640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:27:11.069 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:11.067+0000 7f464f011640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:27:14.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:14.107 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:14.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:27:14.437 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:14.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:14.570+0000 7f0d27771640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:27:14.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:14.571+0000 7f0d27771640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:27:14.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:14.571+0000 7f0d27771640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:27:14.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:14.571+0000 7f0d27771640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:27:14.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:14.571+0000 7f0d27771640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:27:14.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:14.573+0000 7f0d27771640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:27:17.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:27:17.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:17.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:17.974+0000 7f921120d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:27:17.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:17.975+0000 7f921120d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:27:17.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:17.975+0000 7f921120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:27:17.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:17.975+0000 7f921120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:27:17.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:17.975+0000 7f921120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:27:17.978 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:17.977+0000 7f921120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:27:20.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:27:20.999 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:21.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:21.132+0000 7f334520e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:27:21.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:21.133+0000 7f334520e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:27:21.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:21.133+0000 7f334520e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:27:21.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:21.133+0000 7f334520e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:27:21.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:21.133+0000 7f334520e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:27:21.137 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:21.135+0000 7f334520e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:27:24.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:27:24.156 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:27:24.156 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:24.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:27:24.252 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:24.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:27:24.346 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:24.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:27:24.486 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:24.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:27:24.577 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:24.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:27:24.673 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:24.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:27:24.768 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:24.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:27:24.860 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:24.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:27:24.958 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:24.958 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:27:25.049 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:25.049 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:27:25.139 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:25.139 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:27:25.264 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:25.265 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:27:25.352 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:25.352 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:27:25.488 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:25.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:27:25.581 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:25.581 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:27:25.637 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:25.637 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:27:25.686 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:25.686 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:27:25.741 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:25.741 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:27:25.789 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:27:25.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:27:25.796 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.y] 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA== 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg== 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w== 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ== 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg== 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.6] 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw== 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.7] 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A== 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout:[client.2] 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q== 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw== 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout:[client.3] 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w== 2024-03-06T07:27:25.920 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.y] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA== 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg== 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ== 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw== 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw== 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA== 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww== 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-03-06T07:27:25.921 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.6] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.7] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout:[client.2] 2024-03-06T07:27:25.922 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q== 2024-03-06T07:27:25.923 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2024-03-06T07:27:25.923 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw== 2024-03-06T07:27:25.923 INFO:teuthology.orchestra.run.smithi177.stdout:[client.3] 2024-03-06T07:27:25.923 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w== 2024-03-06T07:27:25.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:25.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:26.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:26.080 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:26.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:26.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:26.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.676 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:26.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:26.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:27.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:27.117 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:27.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:27.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:27.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.413 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.701 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:27.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:27.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:28.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:28.143 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:28.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:28.289 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:28.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:28.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:28.574 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:27:28.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:27:28.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:28.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:27:28.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:27:29.002 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:29.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:27:29.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:27:29.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:29.475 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:27:29.558 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:29.565 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:29.615 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:27:29.616 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:27:29.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:27:29.621 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:27:29.621 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:27:29.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:27:29.626 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:27:29.627 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:27:29.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:27:29.631 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:27:29.631 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:27:29.631 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:27:29.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:27:29.636 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:27:29.636 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:27:29.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:27:29.641 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:27:29.641 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:27:29.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:27:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.816+0000 7f2246400640 1 Processor -- start 2024-03-06T07:27:29.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.817+0000 7f2246400640 1 -- start start 2024-03-06T07:27:29.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2246400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2246400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2246400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2240071440 con 0x7f224015fc10 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2246400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22400b46c0 con 0x7f22400b3f70 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2246400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22400b49a0 con 0x7f2240070ff0 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2236a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2236a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47552/0 (socket says 172.21.15.114:47552) 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2236a00640 1 -- 172.21.15.114:0/1698945305 learned_addr learned my addr 172.21.15.114:0/1698945305 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2236a00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 msgr2=0x7f2240160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2236a00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 msgr2=0x7f2240160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2236a00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f2236a00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.818+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.819+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 msgr2=0x7f22400ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:27:29.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.819+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 msgr2=0x7f22400ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:29.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.819+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:29.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:29.819+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:27:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.019+0000 7f2236a00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.020+0000 7f2236a00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 msgr2=0x7f2240160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:27:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.020+0000 7f2236a00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 msgr2=0x7f2240160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.020+0000 7f2236a00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:30.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.020+0000 7f2236a00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:27:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.020+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.021+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 msgr2=0x7f22400ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:27:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.021+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 msgr2=0x7f22400ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.021+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.021+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:27:30.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.059+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034789472 0 0) 0x7f2240071440 con 0x7f224015fc10 2024-03-06T07:27:30.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.059+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2224003520 con 0x7f224015fc10 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 509560539 0 0) 0x7f2224003520 con 0x7f224015fc10 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 msgr2=0x7f22400ce4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.177:3300/0 conn(0x7f2240070ff0 0x7f22400ce4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 msgr2=0x7f2240160700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 --2- 172.21.15.114:0/1698945305 >> v2:172.21.15.114:3300/0 conn(0x7f22400b3f70 0x7f2240160700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2240072c90 con 0x7f224015fc10 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2230001670 con 0x7f224015fc10 2024-03-06T07:27:30.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 >> v1:172.21.15.114:6789/0 conn(0x7f224015fc10 legacy=0x7f224015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.060+0000 7f223ea00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22240039b0 0x7f2224003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.061+0000 7f223ea00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2224004800 0x7f2224004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.061+0000 7f223ea00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f222400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.061+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b46c0 con 0x7f22240051c0 2024-03-06T07:27:30.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.061+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b49a0 con 0x7f22240039b0 2024-03-06T07:27:30.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.061+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222400c060 con 0x7f2224004800 2024-03-06T07:27:30.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.062+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f222400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.062+0000 7f223f400640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22240039b0 0x7f2224003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.062+0000 7f2236a00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2224004800 0x7f2224004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.063+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2224004800 msgr2=0x7f2224004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.063+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2224004800 0x7f2224004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.063+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22240039b0 msgr2=0x7f2224003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.063+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22240039b0 0x7f2224003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.063+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2240072c90 con 0x7f22240051c0 2024-03-06T07:27:30.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.063+0000 7f2236a00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2224004800 0x7f2224004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:30.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.065+0000 7f223fe00640 1 -- 172.21.15.114:0/1698945305 reap_dead start 2024-03-06T07:27:30.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.065+0000 7f223fe00640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f222400b9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2238009610 tx=0x7f2224005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d059c44e296a0ec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.066+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238038070 con 0x7f22240051c0 2024-03-06T07:27:30.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.066+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2238033640 con 0x7f22240051c0 2024-03-06T07:27:30.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.066+0000 7f223ea00640 1 -- 172.21.15.114:0/1698945305 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238004040 con 0x7f22240051c0 2024-03-06T07:27:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.067+0000 7f2246400640 1 -- 172.21.15.114:0/1698945305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 msgr2=0x7f222400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.067+0000 7f2246400640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f222400b9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2238009610 tx=0x7f2224005830 comp rx=0 tx=0).stop 2024-03-06T07:27:30.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.068+0000 7f2246400640 1 -- 172.21.15.114:0/1698945305 shutdown_connections 2024-03-06T07:27:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.068+0000 7f2246400640 1 --2- 172.21.15.114:0/1698945305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f222400b9c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.068+0000 7f2246400640 1 -- 172.21.15.114:0/1698945305 >> 172.21.15.114:0/1698945305 conn(0x7f22400b1940 msgr2=0x7f22400b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.068+0000 7f2246400640 1 -- 172.21.15.114:0/1698945305 shutdown_connections 2024-03-06T07:27:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.068+0000 7f2246400640 1 -- 172.21.15.114:0/1698945305 wait complete. 2024-03-06T07:27:30.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.071+0000 7f2246400640 1 Processor -- start 2024-03-06T07:27:30.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.071+0000 7f2246400640 1 -- start start 2024-03-06T07:27:30.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f2246400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f224014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f2246400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22400cc690 0x7f22400ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f2246400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22400c9ee0 0x7f22400ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f2246400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240072a30 con 0x7f22240051c0 2024-03-06T07:27:30.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f2246400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400728b0 con 0x7f22400c9ee0 2024-03-06T07:27:30.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f2246400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224015c2e0 con 0x7f22400cc690 2024-03-06T07:27:30.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f223f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f224014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f223f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f224014e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47584/0 (socket says 172.21.15.114:47584) 2024-03-06T07:27:30.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.073+0000 7f223f400640 1 -- 172.21.15.114:0/3813536030 learned_addr learned my addr 172.21.15.114:0/3813536030 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:30.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f223fe00640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22400c9ee0 0x7f22400ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f2236a00640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22400cc690 0x7f22400ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f223f400640 1 -- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22400cc690 msgr2=0x7f22400ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f223f400640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22400cc690 0x7f22400ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f223f400640 1 -- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22400c9ee0 msgr2=0x7f22400ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f223f400640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22400c9ee0 0x7f22400ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f223f400640 1 -- 172.21.15.114:0/3813536030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400ca8a0 con 0x7f22240051c0 2024-03-06T07:27:30.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.074+0000 7f223f400640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f224014e770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f223000fb70 tx=0x7f223000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.075+0000 7f2237e00640 1 -- 172.21.15.114:0/3813536030 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223000fe70 con 0x7f22240051c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.075+0000 7f2237e00640 1 -- 172.21.15.114:0/3813536030 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2230018070 con 0x7f22240051c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.075+0000 7f2237e00640 1 -- 172.21.15.114:0/3813536030 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223000de30 con 0x7f22240051c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.075+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224009a9f0 con 0x7f22240051c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.075+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224009ac90 con 0x7f22240051c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.075+0000 7f2237e00640 1 -- 172.21.15.114:0/3813536030 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f22300050d0 con 0x7f22240051c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.075+0000 7f2237e00640 1 -- 172.21.15.114:0/3813536030 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f223001d610 con 0x7f22240051c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.076+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22400cd030 con 0x7f22240051c0 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.080+0000 7f2237e00640 1 -- 172.21.15.114:0/3813536030 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f223001d810 con 0x7f22240051c0 2024-03-06T07:27:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.154+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f224004f330 con 0x7f22240051c0 2024-03-06T07:27:30.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.157+0000 7f2237e00640 1 -- 172.21.15.114:0/3813536030 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f223001d9f0 con 0x7f22240051c0 2024-03-06T07:27:30.159 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-03-06T07:27:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.159+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 msgr2=0x7f224014e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.159+0000 7f2246400640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f224014e770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f223000fb70 tx=0x7f223000d520 comp rx=0 tx=0).stop 2024-03-06T07:27:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.160+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 shutdown_connections 2024-03-06T07:27:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.160+0000 7f2246400640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22400c9ee0 0x7f22400ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.160+0000 7f2246400640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22400cc690 0x7f22400ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.160+0000 7f2246400640 1 --2- 172.21.15.114:0/3813536030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22240051c0 0x7f224014e770 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.160+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 >> 172.21.15.114:0/3813536030 conn(0x7f22400b1940 msgr2=0x7f22400b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.160+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 shutdown_connections 2024-03-06T07:27:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:30.160+0000 7f2246400640 1 -- 172.21.15.114:0/3813536030 wait complete. 2024-03-06T07:27:30.178 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:27:30.178 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:27:30.178 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:30.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:27:30.256 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:30.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:27:30.389 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:30.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:27:30.475 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:27:30.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:27:30.603 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:30.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:27:30.647 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:30.646+0000 7f4b2fcb6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:27:30.683 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:30.681+0000 7f303142e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:27:30.690 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:30.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:27:30.713 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:30.711+0000 7f4b2fcb6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:27:30.750 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:30.749+0000 7f303142e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:27:30.797 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:30.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:27:30.921 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:27:30.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:27:31.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 24fea251-8ceb-4e5a-8d7a-379c6e928d1e 0 2024-03-06T07:27:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.268+0000 7ffbfa200640 1 Processor -- start 2024-03-06T07:27:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.269+0000 7ffbfa200640 1 -- start start 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbfa200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffbf40d0e60 0x7ffbf40b3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbfa200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffbf40b3120 0x7ffbf40b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbfa200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf40d4ce0 con 0x7ffbf415e100 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbfa200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf40d4a40 con 0x7ffbf40b3120 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbfa200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf415e4e0 con 0x7ffbf40d0e60 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbf3e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffbf40d0e60 0x7ffbf40b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbf3e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffbf40d0e60 0x7ffbf40b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:48894/0 (socket says 172.21.15.177:48894) 2024-03-06T07:27:31.272 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.270+0000 7ffbf3e00640 1 -- 172.21.15.177:0/2050854491 learned_addr learned my addr 172.21.15.177:0/2050854491 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.271+0000 7ffbf9800640 1 --2- 172.21.15.177:0/2050854491 >> v2:172.21.15.114:3300/0 conn(0x7ffbf40b3120 0x7ffbf40b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.271+0000 7ffbf3e00640 1 -- 172.21.15.177:0/2050854491 >> v2:172.21.15.114:3300/0 conn(0x7ffbf40b3120 msgr2=0x7ffbf40b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.271+0000 7ffbf3e00640 1 --2- 172.21.15.177:0/2050854491 >> v2:172.21.15.114:3300/0 conn(0x7ffbf40b3120 0x7ffbf40b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.271+0000 7ffbf3e00640 1 -- 172.21.15.177:0/2050854491 >> v1:172.21.15.177:6789/0 conn(0x7ffbf415e100 legacy=0x7ffbf40d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.271+0000 7ffbf3e00640 1 -- 172.21.15.177:0/2050854491 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf40d2470 con 0x7ffbf40d0e60 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.271+0000 7ffbf3400640 1 -- 172.21.15.177:0/2050854491 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356105637 0 0) 0x7ffbf40d4ce0 con 0x7ffbf415e100 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.272+0000 7ffbf3e00640 1 --2- 172.21.15.177:0/2050854491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40b3770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffbf408c5c0 tx=0x7ffbe002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c71d550b911e44f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.272+0000 7ffbf3400640 1 -- 172.21.15.177:0/2050854491 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0034070 con 0x7ffbf40d0e60 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.272+0000 7ffbf3400640 1 -- 172.21.15.177:0/2050854491 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe0030860 con 0x7ffbf40d0e60 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.272+0000 7ffbf3400640 1 -- 172.21.15.177:0/2050854491 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0030c30 con 0x7ffbf40d0e60 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 -- 172.21.15.177:0/2050854491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40d0e60 msgr2=0x7ffbf40b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 --2- 172.21.15.177:0/2050854491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40b3770 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffbf408c5c0 tx=0x7ffbe002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 -- 172.21.15.177:0/2050854491 shutdown_connections 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 --2- 172.21.15.177:0/2050854491 >> v2:172.21.15.114:3300/0 conn(0x7ffbf40b3120 0x7ffbf40b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 --2- 172.21.15.177:0/2050854491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40b3770 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 -- 172.21.15.177:0/2050854491 >> 172.21.15.177:0/2050854491 conn(0x7ffbf40d3d80 msgr2=0x7ffbf40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 -- 172.21.15.177:0/2050854491 shutdown_connections 2024-03-06T07:27:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.273+0000 7ffbfa200640 1 -- 172.21.15.177:0/2050854491 wait complete. 2024-03-06T07:27:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.275+0000 7ffbfa200640 1 Processor -- start 2024-03-06T07:27:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.275+0000 7ffbfa200640 1 -- start start 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbfa200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40b3120 0x7ffbf41f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbfa200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbfa200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbf415e100 0x7ffbf40b8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbfa200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf40cfbd0 con 0x7ffbf40d0e60 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbfa200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf40d2a30 con 0x7ffbf40b3120 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbfa200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf40b2bd0 con 0x7ffbf415e100 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40b3120 0x7ffbf41f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf9800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbf415e100 0x7ffbf40b8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40b3120 0x7ffbf41f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:48904/0 (socket says 172.21.15.177:48904) 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 -- 172.21.15.177:0/1101448455 learned_addr learned my addr 172.21.15.177:0/1101448455 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf3e00640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 -- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbf415e100 msgr2=0x7ffbf40b8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbf415e100 0x7ffbf40b8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 -- 172.21.15.177:0/1101448455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbf40d0e60 msgr2=0x7ffbf40a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf8e00640 1 -- 172.21.15.177:0/1101448455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf41f4a10 con 0x7ffbf40b3120 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf3e00640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.276+0000 7ffbf9800640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbf415e100 0x7ffbf40b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.277+0000 7ffbf8e00640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40b3120 0x7ffbf41f3a60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800c3e0 tx=0x7ffbe800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.277+0000 7ffbf0c00640 1 -- 172.21.15.177:0/1101448455 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe8017070 con 0x7ffbf40b3120 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.277+0000 7ffbf0c00640 1 -- 172.21.15.177:0/1101448455 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe80052d0 con 0x7ffbf40b3120 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.277+0000 7ffbf0c00640 1 -- 172.21.15.177:0/1101448455 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe8003830 con 0x7ffbf40b3120 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.277+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf41f4170 con 0x7ffbf40b3120 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.277+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf41f4730 con 0x7ffbf40b3120 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.277+0000 7ffbf0c00640 1 -- 172.21.15.177:0/1101448455 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ffbe801a070 con 0x7ffbf40b3120 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.278+0000 7ffbf0c00640 1 -- 172.21.15.177:0/1101448455 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ffbe801c440 con 0x7ffbf40b3120 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.278+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf40753e0 con 0x7ffbf40b3120 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.281+0000 7ffbf0c00640 1 -- 172.21.15.177:0/1101448455 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ffbe8021020 con 0x7ffbf40b3120 2024-03-06T07:27:31.337 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.335+0000 7f4b2fcb6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:27:31.356 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.355+0000 7f303142e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:27:31.362 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.359+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "24fea251-8ceb-4e5a-8d7a-379c6e928d1e", "id": 0} v 0) v1 -- 0x7ffbf40d1ea0 con 0x7ffbf40b3120 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.365+0000 7ffbf0c00640 1 -- 172.21.15.177:0/1101448455 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24fea251-8ceb-4e5a-8d7a-379c6e928d1e", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffbe8011590 con 0x7ffbf40b3120 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40b3120 msgr2=0x7ffbf41f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40b3120 0x7ffbf41f3a60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800c3e0 tx=0x7ffbe800c8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 shutdown_connections 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbf415e100 0x7ffbf40b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbf40d0e60 0x7ffbf40a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 --2- 172.21.15.177:0/1101448455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbf40b3120 0x7ffbf41f3a60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 >> 172.21.15.177:0/1101448455 conn(0x7ffbf40d3d80 msgr2=0x7ffbf40cf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 shutdown_connections 2024-03-06T07:27:31.369 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.366+0000 7ffbfa200640 1 -- 172.21.15.177:0/1101448455 wait complete. 2024-03-06T07:27:31.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 8c89fbef-928a-4868-b4fc-da93fa4f0b9c 1 2024-03-06T07:27:31.404 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.402+0000 7f4b2fcb6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:27:31.421 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.419+0000 7f303142e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:27:31.459 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.456+0000 7f4b2fcb6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:27:31.471 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.469+0000 7f303142e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:27:31.524 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.522+0000 7f4b2fcb6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:27:31.533 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.532+0000 7f303142e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:27:31.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.565+0000 7f5903000640 1 Processor -- start 2024-03-06T07:27:31.567 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.565+0000 7f5903000640 1 -- start start 2024-03-06T07:27:31.570 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.567+0000 7f5903000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58fc070050 0x7f58fc0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.570 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.567+0000 7f5903000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc06faa0 con 0x7f58fc15de30 2024-03-06T07:27:31.570 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.567+0000 7f5903000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc0ae410 con 0x7f58fc06b6d0 2024-03-06T07:27:31.570 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.567+0000 7f5903000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58fc0aea60 con 0x7f58fc070050 2024-03-06T07:27:31.570 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.567+0000 7f5901200640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f58fc06b6d0 0x7f58fc06d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:39556/0 (socket says 172.21.15.177:39556) 2024-03-06T07:27:31.570 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.567+0000 7f5901200640 1 -- 172.21.15.177:0/1874841234 learned_addr learned my addr 172.21.15.177:0/1874841234 (peer_addr_for_me v1:172.21.15.177:0/0) 2024-03-06T07:27:31.570 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.567+0000 7f5902600640 1 --2- 172.21.15.177:0/1874841234 >> v2:172.21.15.114:3300/0 conn(0x7f58fc070050 0x7f58fc0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.568+0000 7f58fbe00640 1 -- 172.21.15.177:0/1874841234 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896197903 0 0) 0x7f58fc0ae410 con 0x7f58fc06b6d0 2024-03-06T07:27:31.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.568+0000 7f58fbe00640 1 -- 172.21.15.177:0/1874841234 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e4003520 con 0x7f58fc06b6d0 2024-03-06T07:27:31.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.568+0000 7f5902600640 1 -- 172.21.15.177:0/1874841234 >> v1:172.21.15.177:6789/0 conn(0x7f58fc06b6d0 legacy=0x7f58fc06d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.568+0000 7f5902600640 1 -- 172.21.15.177:0/1874841234 >> v1:172.21.15.114:6789/0 conn(0x7f58fc15de30 legacy=0x7f58fc06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.568+0000 7f5902600640 1 -- 172.21.15.177:0/1874841234 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc0ae6f0 con 0x7f58fc070050 2024-03-06T07:27:31.572 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.570+0000 7f5902600640 1 --2- 172.21.15.177:0/1874841234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 0x7f58fc0b0000 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009d90 tx=0x7f58f402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2d1d72db1f1a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.570+0000 7f58fbe00640 1 -- 172.21.15.177:0/1874841234 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4039070 con 0x7f58fc070050 2024-03-06T07:27:31.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.570+0000 7f58fbe00640 1 -- 172.21.15.177:0/1874841234 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f402cd00 con 0x7f58fc070050 2024-03-06T07:27:31.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.570+0000 7f58fbe00640 1 -- 172.21.15.177:0/1874841234 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4034640 con 0x7f58fc070050 2024-03-06T07:27:31.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.571+0000 7f5903000640 1 -- 172.21.15.177:0/1874841234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 msgr2=0x7f58fc0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.571+0000 7f5903000640 1 --2- 172.21.15.177:0/1874841234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 0x7f58fc0b0000 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009d90 tx=0x7f58f402bf50 comp rx=0 tx=0).stop 2024-03-06T07:27:31.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.571+0000 7f5903000640 1 -- 172.21.15.177:0/1874841234 shutdown_connections 2024-03-06T07:27:31.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.571+0000 7f5903000640 1 --2- 172.21.15.177:0/1874841234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 0x7f58fc0b0000 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.571+0000 7f5903000640 1 -- 172.21.15.177:0/1874841234 >> 172.21.15.177:0/1874841234 conn(0x7f58fc06c0d0 msgr2=0x7f58fc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.571+0000 7f5903000640 1 -- 172.21.15.177:0/1874841234 shutdown_connections 2024-03-06T07:27:31.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.571+0000 7f5903000640 1 -- 172.21.15.177:0/1874841234 wait complete. 2024-03-06T07:27:31.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.573+0000 7f5903000640 1 Processor -- start 2024-03-06T07:27:31.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.573+0000 7f5903000640 1 -- start start 2024-03-06T07:27:31.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5903000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc06b6d0 0x7f58fc149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5903000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 0x7f58fc160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5903000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc15de30 0x7f58fc06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5903000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc06bf20 con 0x7f58fc070050 2024-03-06T07:27:31.577 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5903000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc06bda0 con 0x7f58fc06b6d0 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5903000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc06e030 con 0x7f58fc15de30 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5901c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc06b6d0 0x7f58fc149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5901c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc06b6d0 0x7f58fc149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:48908/0 (socket says 172.21.15.177:48908) 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5901c00640 1 -- 172.21.15.177:0/793377402 learned_addr learned my addr 172.21.15.177:0/793377402 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5902600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc15de30 0x7f58fc06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.574+0000 7f5901200640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 0x7f58fc160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5901c00640 1 -- 172.21.15.177:0/793377402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc15de30 msgr2=0x7f58fc06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5901c00640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc15de30 0x7f58fc06c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5901c00640 1 -- 172.21.15.177:0/793377402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 msgr2=0x7f58fc160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5901c00640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 0x7f58fc160e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5901c00640 1 -- 172.21.15.177:0/793377402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc1618e0 con 0x7f58fc06b6d0 2024-03-06T07:27:31.578 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5901c00640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc06b6d0 0x7f58fc149580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c400 tx=0x7f58f000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.579 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f58f9600640 1 -- 172.21.15.177:0/793377402 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0017070 con 0x7f58fc06b6d0 2024-03-06T07:27:31.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f58f9600640 1 -- 172.21.15.177:0/793377402 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f00052d0 con 0x7f58fc06b6d0 2024-03-06T07:27:31.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f58f9600640 1 -- 172.21.15.177:0/793377402 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0003830 con 0x7f58fc06b6d0 2024-03-06T07:27:31.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc161bc0 con 0x7f58fc06b6d0 2024-03-06T07:27:31.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.575+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc162180 con 0x7f58fc06b6d0 2024-03-06T07:27:31.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.576+0000 7f58f9600640 1 -- 172.21.15.177:0/793377402 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f58f001a070 con 0x7f58fc06b6d0 2024-03-06T07:27:31.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.576+0000 7f58f9600640 1 -- 172.21.15.177:0/793377402 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f58f0011c10 con 0x7f58fc06b6d0 2024-03-06T07:27:31.580 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.576+0000 7f4b2fcb6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:27:31.580 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.576+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc0753e0 con 0x7f58fc06b6d0 2024-03-06T07:27:31.581 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.579+0000 7f58f9600640 1 -- 172.21.15.177:0/793377402 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f58f0021020 con 0x7f58fc06b6d0 2024-03-06T07:27:31.584 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.583+0000 7f303142e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:27:31.660 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.657+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8c89fbef-928a-4868-b4fc-da93fa4f0b9c", "id": 1} v 0) v1 -- 0x7f58fc04f850 con 0x7f58fc06b6d0 2024-03-06T07:27:31.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.664+0000 7f58f9600640 1 -- 172.21.15.177:0/793377402 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8c89fbef-928a-4868-b4fc-da93fa4f0b9c", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f58f0011e10 con 0x7f58fc06b6d0 2024-03-06T07:27:31.668 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-06T07:27:31.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.665+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc06b6d0 msgr2=0x7f58fc149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc06b6d0 0x7f58fc149580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c400 tx=0x7f58f000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 shutdown_connections 2024-03-06T07:27:31.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc15de30 0x7f58fc06c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc070050 0x7f58fc160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 --2- 172.21.15.177:0/793377402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc06b6d0 0x7f58fc149580 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 >> 172.21.15.177:0/793377402 conn(0x7f58fc06c0d0 msgr2=0x7f58fc15c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 shutdown_connections 2024-03-06T07:27:31.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.666+0000 7f5903000640 1 -- 172.21.15.177:0/793377402 wait complete. 2024-03-06T07:27:31.686 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 4292fc64-f253-4fee-97b3-9440ad950e3a 2 2024-03-06T07:27:31.715 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.713+0000 7f4b2fcb6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:27:31.716 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.714+0000 7f303142e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:27:31.795 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.794+0000 7f303142e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:31.800 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.797+0000 7f4b2fcb6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:31.848 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.846+0000 7f303142e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:27:31.856 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.853+0000 7f4b2fcb6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:27:31.908 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.905+0000 7f8de5000640 1 Processor -- start 2024-03-06T07:27:31.908 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.905+0000 7f8de5000640 1 -- start start 2024-03-06T07:27:31.908 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.907+0000 7f303142e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:27:31.910 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8de5000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.910 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8de5000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8de00ced50 0x7f8de00cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.910 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8de5000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de00d4c40 con 0x7f8de015f490 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8de5000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de00d49f0 con 0x7f8de00ced50 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8de5000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00d24b0 con 0x7f8de00b4de0 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8dd7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8dd7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00d0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:42916/0 (socket says 172.21.15.177:42916) 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8dd7e00640 1 -- 172.21.15.177:0/3277180331 learned_addr learned my addr 172.21.15.177:0/3277180331 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.907+0000 7f8dde800640 1 --2- 172.21.15.177:0/3277180331 >> v2:172.21.15.114:3300/0 conn(0x7f8de00ced50 0x7f8de00cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.908+0000 7f8dd7e00640 1 -- 172.21.15.177:0/3277180331 >> v2:172.21.15.114:3300/0 conn(0x7f8de00ced50 msgr2=0x7f8de00cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.908+0000 7f8dd7e00640 1 --2- 172.21.15.177:0/3277180331 >> v2:172.21.15.114:3300/0 conn(0x7f8de00ced50 0x7f8de00cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.908+0000 7f8dd7e00640 1 -- 172.21.15.177:0/3277180331 >> v1:172.21.15.114:6789/0 conn(0x7f8de015f490 legacy=0x7f8de00d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.911 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.908+0000 7f8dd7e00640 1 -- 172.21.15.177:0/3277180331 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00d2700 con 0x7f8de00b4de0 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.909+0000 7f8dd7e00640 1 --2- 172.21.15.177:0/3277180331 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00d0fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009870 tx=0x7f8dc802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c198b2b59999e009 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.909+0000 7f8ddd400640 1 -- 172.21.15.177:0/3277180331 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8035020 con 0x7f8de00b4de0 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.909+0000 7f8ddd400640 1 -- 172.21.15.177:0/3277180331 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8031640 con 0x7f8de00b4de0 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.909+0000 7f8ddd400640 1 -- 172.21.15.177:0/3277180331 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc802d040 con 0x7f8de00b4de0 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 -- 172.21.15.177:0/3277180331 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 msgr2=0x7f8de00d0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 --2- 172.21.15.177:0/3277180331 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00d0fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009870 tx=0x7f8dc802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 -- 172.21.15.177:0/3277180331 shutdown_connections 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 --2- 172.21.15.177:0/3277180331 >> v2:172.21.15.114:3300/0 conn(0x7f8de00ced50 0x7f8de00cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 --2- 172.21.15.177:0/3277180331 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00d0fb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 -- 172.21.15.177:0/3277180331 >> 172.21.15.177:0/3277180331 conn(0x7f8de00d3ed0 msgr2=0x7f8de00d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 -- 172.21.15.177:0/3277180331 shutdown_connections 2024-03-06T07:27:31.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.910+0000 7f8de5000640 1 -- 172.21.15.177:0/3277180331 wait complete. 2024-03-06T07:27:31.914 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.912+0000 7f8de5000640 1 Processor -- start 2024-03-06T07:27:31.914 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.912+0000 7f8de5000640 1 -- start start 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8de5000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8de5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8de00ced50 0x7f8de014ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8de5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8de015f490 0x7f8de014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8de5000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de015fd50 con 0x7f8de00ced50 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8de5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de00d2cc0 con 0x7f8de015f490 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8de5000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de015f870 con 0x7f8de00b4de0 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00b0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:42920/0 (socket says 172.21.15.177:42920) 2024-03-06T07:27:31.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 -- 172.21.15.177:0/3972987779 learned_addr learned my addr 172.21.15.177:0/3972987779 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8dde800640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8de015f490 0x7f8de014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8dd7e00640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8de00ced50 0x7f8de014ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 -- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8de015f490 msgr2=0x7f8de014f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8de015f490 0x7f8de014f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 -- 172.21.15.177:0/3972987779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8de00ced50 msgr2=0x7f8de014ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8de00ced50 0x7f8de014ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.913+0000 7f8ddde00640 1 -- 172.21.15.177:0/3972987779 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00c9170 con 0x7f8de00b4de0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8ddde00640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00b0a90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000c400 tx=0x7f8dd000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8dd6000640 1 -- 172.21.15.177:0/3972987779 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00052d0 con 0x7f8de00b4de0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8dd6000640 1 -- 172.21.15.177:0/3972987779 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd0005490 con 0x7f8de00b4de0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8dd6000640 1 -- 172.21.15.177:0/3972987779 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0007260 con 0x7f8de00b4de0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de00c9450 con 0x7f8de00b4de0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de00c9a10 con 0x7f8de00b4de0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8dd6000640 1 -- 172.21.15.177:0/3972987779 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8dd0007400 con 0x7f8de00b4de0 2024-03-06T07:27:31.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.914+0000 7f8dd6000640 1 -- 172.21.15.177:0/3972987779 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f8dd0011590 con 0x7f8de00b4de0 2024-03-06T07:27:31.919 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.915+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de004f3a0 con 0x7f8de00b4de0 2024-03-06T07:27:31.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.918+0000 7f8dd6000640 1 -- 172.21.15.177:0/3972987779 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8dd00035f0 con 0x7f8de00b4de0 2024-03-06T07:27:31.921 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.918+0000 7f4b2fcb6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:27:31.964 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:31.962+0000 7f303142e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:27:31.978 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:31.975+0000 7f4b2fcb6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:27:31.999 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:31.997+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4292fc64-f253-4fee-97b3-9440ad950e3a", "id": 2} v 0) v1 -- 0x7f8de008d950 con 0x7f8de00b4de0 2024-03-06T07:27:32.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.004+0000 7f8dd6000640 1 -- 172.21.15.177:0/3972987779 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4292fc64-f253-4fee-97b3-9440ad950e3a", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8dd0003a90 con 0x7f8de00b4de0 2024-03-06T07:27:32.007 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-03-06T07:27:32.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.005+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 msgr2=0x7f8de00b0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.005+0000 7f8de5000640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00b0a90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000c400 tx=0x7f8dd000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.005+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 shutdown_connections 2024-03-06T07:27:32.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.005+0000 7f8de5000640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8de015f490 0x7f8de014f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.005+0000 7f8de5000640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8de00ced50 0x7f8de014ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.009 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.005+0000 7f8de5000640 1 --2- 172.21.15.177:0/3972987779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8de00b4de0 0x7f8de00b0a90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.009 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.005+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 >> 172.21.15.177:0/3972987779 conn(0x7f8de00d3ed0 msgr2=0x7f8de00b8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.009 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.006+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 shutdown_connections 2024-03-06T07:27:32.009 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.006+0000 7f8de5000640 1 -- 172.21.15.177:0/3972987779 wait complete. 2024-03-06T07:27:32.025 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 56b8f5df-65df-4f34-80cd-d7dae655a279 3 2024-03-06T07:27:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.206+0000 7ff13aa00640 1 Processor -- start 2024-03-06T07:27:32.209 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.206+0000 7ff13aa00640 1 -- start start 2024-03-06T07:27:32.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff13aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff1340b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff13aa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1340d4ce0 con 0x7ff13415e100 2024-03-06T07:27:32.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff13aa00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1340d4a40 con 0x7ff1340d0e60 2024-03-06T07:27:32.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff13aa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13415e4e0 con 0x7ff1340b3120 2024-03-06T07:27:32.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff13a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff1340b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff13a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff1340b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:42934/0 (socket says 172.21.15.177:42934) 2024-03-06T07:27:32.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff13a000640 1 -- 172.21.15.177:0/3265245262 learned_addr learned my addr 172.21.15.177:0/3265245262 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:32.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.208+0000 7ff138c00640 1 --1- 172.21.15.177:0/3265245262 >> v1:172.21.15.177:6789/0 conn(0x7ff1340d0e60 0x7ff1340b3770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:46222/0 (socket says 172.21.15.177:46222) 2024-03-06T07:27:32.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.209+0000 7ff13a000640 1 -- 172.21.15.177:0/3265245262 >> v1:172.21.15.177:6789/0 conn(0x7ff1340d0e60 legacy=0x7ff1340b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.209+0000 7ff13a000640 1 -- 172.21.15.177:0/3265245262 >> v1:172.21.15.114:6789/0 conn(0x7ff13415e100 legacy=0x7ff1340d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.209+0000 7ff133e00640 1 -- 172.21.15.177:0/3265245262 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832959105 0 0) 0x7ff1340d4a40 con 0x7ff1340d0e60 2024-03-06T07:27:32.212 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.209+0000 7ff13a000640 1 -- 172.21.15.177:0/3265245262 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1340d2470 con 0x7ff1340b3120 2024-03-06T07:27:32.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.210+0000 7ff13a000640 1 --2- 172.21.15.177:0/3265245262 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff1340b3e80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009e80 tx=0x7ff12c02bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699771a7051d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.210+0000 7ff133e00640 1 -- 172.21.15.177:0/3265245262 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c02cb70 con 0x7ff1340b3120 2024-03-06T07:27:32.213 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.210+0000 7ff133e00640 1 -- 172.21.15.177:0/3265245262 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c02cd10 con 0x7ff1340b3120 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.210+0000 7ff133e00640 1 -- 172.21.15.177:0/3265245262 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c034610 con 0x7ff1340b3120 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.211+0000 7ff13aa00640 1 -- 172.21.15.177:0/3265245262 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 msgr2=0x7ff1340b3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.211+0000 7ff13aa00640 1 --2- 172.21.15.177:0/3265245262 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff1340b3e80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009e80 tx=0x7ff12c02bfd0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.211+0000 7ff13aa00640 1 -- 172.21.15.177:0/3265245262 shutdown_connections 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.211+0000 7ff13aa00640 1 --2- 172.21.15.177:0/3265245262 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff1340b3e80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.211+0000 7ff13aa00640 1 -- 172.21.15.177:0/3265245262 >> 172.21.15.177:0/3265245262 conn(0x7ff1340d3d80 msgr2=0x7ff1340d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.211+0000 7ff13aa00640 1 -- 172.21.15.177:0/3265245262 shutdown_connections 2024-03-06T07:27:32.214 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.211+0000 7ff13aa00640 1 -- 172.21.15.177:0/3265245262 wait complete. 2024-03-06T07:27:32.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.212+0000 7ff13aa00640 1 Processor -- start 2024-03-06T07:27:32.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.213+0000 7ff13aa00640 1 -- start start 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.213+0000 7ff13aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff13415c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.213+0000 7ff13aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1340d0e60 0x7ff1340b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.213+0000 7ff13aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff13415e100 0x7ff1340cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.213+0000 7ff13aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1340cfbd0 con 0x7ff13415e100 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.213+0000 7ff13aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1340d2a30 con 0x7ff1340d0e60 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.213+0000 7ff13aa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1340b2bd0 con 0x7ff1340b3120 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff13415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff13415c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:42950/0 (socket says 172.21.15.177:42950) 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 -- 172.21.15.177:0/4235910897 learned_addr learned my addr 172.21.15.177:0/4235910897 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff138c00640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1340d0e60 0x7ff1340b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.218 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff13a000640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff13415e100 0x7ff1340cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 -- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1340d0e60 msgr2=0x7ff1340b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1340d0e60 0x7ff1340b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 -- 172.21.15.177:0/4235910897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff13415e100 msgr2=0x7ff1340cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff13415e100 0x7ff1340cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 -- 172.21.15.177:0/4235910897 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1341f3d50 con 0x7ff1340b3120 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.214+0000 7ff139600640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff13415c180 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff12800c400 tx=0x7ff12800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.215+0000 7ff131600640 1 -- 172.21.15.177:0/4235910897 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128017070 con 0x7ff1340b3120 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.215+0000 7ff131600640 1 -- 172.21.15.177:0/4235910897 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1280052d0 con 0x7ff1340b3120 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.215+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1340b8900 con 0x7ff1340b3120 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.215+0000 7ff131600640 1 -- 172.21.15.177:0/4235910897 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128003830 con 0x7ff1340b3120 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.215+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1340b8ec0 con 0x7ff1340b3120 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.215+0000 7ff131600640 1 -- 172.21.15.177:0/4235910897 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff12801a070 con 0x7ff1340b3120 2024-03-06T07:27:32.219 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.215+0000 7ff131600640 1 -- 172.21.15.177:0/4235910897 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7ff128011590 con 0x7ff1340b3120 2024-03-06T07:27:32.220 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.216+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1340753e0 con 0x7ff1340b3120 2024-03-06T07:27:32.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.218+0000 7ff131600640 1 -- 172.21.15.177:0/4235910897 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff128021070 con 0x7ff1340b3120 2024-03-06T07:27:32.299 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.296+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd new", "uuid": "56b8f5df-65df-4f34-80cd-d7dae655a279", "id": 3} v 0) v1 -- 0x7ff134068490 con 0x7ff1340b3120 2024-03-06T07:27:32.306 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.303+0000 7ff131600640 1 -- 172.21.15.177:0/4235910897 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "56b8f5df-65df-4f34-80cd-d7dae655a279", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff128011b80 con 0x7ff1340b3120 2024-03-06T07:27:32.306 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-03-06T07:27:32.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.304+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 msgr2=0x7ff13415c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.307 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.304+0000 7ff13aa00640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff13415c180 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff12800c400 tx=0x7ff12800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.304+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 shutdown_connections 2024-03-06T07:27:32.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.304+0000 7ff13aa00640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff13415e100 0x7ff1340cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.304+0000 7ff13aa00640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1340d0e60 0x7ff1340b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.304+0000 7ff13aa00640 1 --2- 172.21.15.177:0/4235910897 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1340b3120 0x7ff13415c180 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.304+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 >> 172.21.15.177:0/4235910897 conn(0x7ff1340d3d80 msgr2=0x7ff1340b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.305+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 shutdown_connections 2024-03-06T07:27:32.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.305+0000 7ff13aa00640 1 -- 172.21.15.177:0/4235910897 wait complete. 2024-03-06T07:27:32.323 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 03440662-d24e-4004-bb4c-2edb870e3000 4 2024-03-06T07:27:32.414 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:32.412+0000 7f303142e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:27:32.455 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:32.453+0000 7f4b2fcb6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:27:32.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.508+0000 7f529f200640 1 Processor -- start 2024-03-06T07:27:32.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.508+0000 7f529f200640 1 -- start start 2024-03-06T07:27:32.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.509+0000 7f529f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a00cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.509+0000 7f529f200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f52a00cff80 0x7f52a00af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.509+0000 7f529f200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a00cefa0 con 0x7f52a015bc60 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.509+0000 7f529f200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a015c040 con 0x7f52a00cff80 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00cd5a0 con 0x7f52a006b9d0 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a00cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a00cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:42962/0 (socket says 172.21.15.177:42962) 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529d400640 1 -- 172.21.15.177:0/3589390010 learned_addr learned my addr 172.21.15.177:0/3589390010 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:32.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529e800640 1 --2- 172.21.15.177:0/3589390010 >> v2:172.21.15.177:3300/0 conn(0x7f52a00cff80 0x7f52a00af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529d400640 1 -- 172.21.15.177:0/3589390010 >> v2:172.21.15.177:3300/0 conn(0x7f52a00cff80 msgr2=0x7f52a00af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529d400640 1 --2- 172.21.15.177:0/3589390010 >> v2:172.21.15.177:3300/0 conn(0x7f52a00cff80 0x7f52a00af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529d400640 1 -- 172.21.15.177:0/3589390010 >> v1:172.21.15.114:6789/0 conn(0x7f52a015bc60 legacy=0x7f52a006d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529d400640 1 -- 172.21.15.177:0/3589390010 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a00cf240 con 0x7f52a006b9d0 2024-03-06T07:27:32.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.510+0000 7f529e800640 1 --2- 172.21.15.177:0/3589390010 >> v2:172.21.15.177:3300/0 conn(0x7f52a00cff80 0x7f52a00af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:32.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.512+0000 7f529d400640 1 --2- 172.21.15.177:0/3589390010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a00cfa40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5288009870 tx=0x7f528802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c08788cf04adc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.512+0000 7f529ca00640 1 -- 172.21.15.177:0/3589390010 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5288034070 con 0x7f52a006b9d0 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.512+0000 7f529ca00640 1 -- 172.21.15.177:0/3589390010 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5288030840 con 0x7f52a006b9d0 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.512+0000 7f529ca00640 1 -- 172.21.15.177:0/3589390010 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5288030c10 con 0x7f52a006b9d0 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 -- 172.21.15.177:0/3589390010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 msgr2=0x7f52a00cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 --2- 172.21.15.177:0/3589390010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a00cfa40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5288009870 tx=0x7f528802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 -- 172.21.15.177:0/3589390010 shutdown_connections 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 --2- 172.21.15.177:0/3589390010 >> v2:172.21.15.177:3300/0 conn(0x7f52a00cff80 0x7f52a00af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 --2- 172.21.15.177:0/3589390010 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a00cfa40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 -- 172.21.15.177:0/3589390010 >> 172.21.15.177:0/3589390010 conn(0x7f52a00d0940 msgr2=0x7f52a00d0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 -- 172.21.15.177:0/3589390010 shutdown_connections 2024-03-06T07:27:32.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.513+0000 7f529f200640 1 -- 172.21.15.177:0/3589390010 wait complete. 2024-03-06T07:27:32.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.514+0000 7f529f200640 1 Processor -- start 2024-03-06T07:27:32.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.514+0000 7f529f200640 1 -- start start 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.515+0000 7f529f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.515+0000 7f529f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a00cff80 0x7f52a0146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.515+0000 7f529f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52a015bc60 0x7f52a0146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.515+0000 7f529f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00cd880 con 0x7f52a00cff80 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.515+0000 7f529f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a006e0b0 con 0x7f52a015bc60 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.515+0000 7f529f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00cda00 con 0x7f52a006b9d0 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52a015bc60 0x7f52a0146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.520 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a015e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:42976/0 (socket says 172.21.15.177:42976) 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 -- 172.21.15.177:0/4176514169 learned_addr learned my addr 172.21.15.177:0/4176514169 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529d400640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a00cff80 0x7f52a0146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 -- 172.21.15.177:0/4176514169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52a015bc60 msgr2=0x7f52a0146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52a015bc60 0x7f52a0146560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 -- 172.21.15.177:0/4176514169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a00cff80 msgr2=0x7f52a0146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a00cff80 0x7f52a0146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 -- 172.21.15.177:0/4176514169 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0146aa0 con 0x7f52a006b9d0 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f529de00640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a015e940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f529800c730 tx=0x7f529800cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.516+0000 7f5296000640 1 -- 172.21.15.177:0/4176514169 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298003b70 con 0x7f52a006b9d0 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.517+0000 7f5296000640 1 -- 172.21.15.177:0/4176514169 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298003d30 con 0x7f52a006b9d0 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.517+0000 7f5296000640 1 -- 172.21.15.177:0/4176514169 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298005310 con 0x7f52a006b9d0 2024-03-06T07:27:32.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.517+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a0146d30 con 0x7f52a006b9d0 2024-03-06T07:27:32.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.517+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a015d720 con 0x7f52a006b9d0 2024-03-06T07:27:32.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.517+0000 7f5296000640 1 -- 172.21.15.177:0/4176514169 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f52980054b0 con 0x7f52a006b9d0 2024-03-06T07:27:32.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.517+0000 7f5296000640 1 -- 172.21.15.177:0/4176514169 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f52980057a0 con 0x7f52a006b9d0 2024-03-06T07:27:32.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.518+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a00753e0 con 0x7f52a006b9d0 2024-03-06T07:27:32.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.520+0000 7f5296000640 1 -- 172.21.15.177:0/4176514169 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f529800a640 con 0x7f52a006b9d0 2024-03-06T07:27:32.574 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:32.573+0000 7f303142e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:27:32.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.599+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd new", "uuid": "03440662-d24e-4004-bb4c-2edb870e3000", "id": 4} v 0) v1 -- 0x7f52a00d0720 con 0x7f52a006b9d0 2024-03-06T07:27:32.608 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.605+0000 7f5296000640 1 -- 172.21.15.177:0/4176514169 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "03440662-d24e-4004-bb4c-2edb870e3000", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f529800a820 con 0x7f52a006b9d0 2024-03-06T07:27:32.608 INFO:teuthology.orchestra.run.smithi177.stdout:4 2024-03-06T07:27:32.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 msgr2=0x7f52a015e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a015e940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f529800c730 tx=0x7f529800cb30 comp rx=0 tx=0).stop 2024-03-06T07:27:32.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 shutdown_connections 2024-03-06T07:27:32.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52a015bc60 0x7f52a0146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a00cff80 0x7f52a0146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.610 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 --2- 172.21.15.177:0/4176514169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a006b9d0 0x7f52a015e940 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 >> 172.21.15.177:0/4176514169 conn(0x7f52a00d0940 msgr2=0x7f52a014e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 shutdown_connections 2024-03-06T07:27:32.611 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.607+0000 7f529f200640 1 -- 172.21.15.177:0/4176514169 wait complete. 2024-03-06T07:27:32.632 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new e8265969-0287-457e-968c-d7ec7047e491 5 2024-03-06T07:27:32.640 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:32.631+0000 7f4b2fcb6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:27:32.711 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:32.710+0000 7f303142e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:27:32.765 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:32.763+0000 7f303142e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:27:32.765 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:32.763+0000 7f4b2fcb6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:27:32.822 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:32.819+0000 7f4b2fcb6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:27:32.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.820+0000 7f7257c00640 1 Processor -- start 2024-03-06T07:27:32.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.820+0000 7f7257c00640 1 -- start start 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.821+0000 7f7257c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f725015e0d0 0x7f72500b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.821+0000 7f7257c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f725015e4b0 0x7f725006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7257c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72500b0070 con 0x7f7250070370 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7257c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72500b3c80 con 0x7f725015e4b0 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7257c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72500b0310 con 0x7f725015e0d0 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f725015e0d0 0x7f72500b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f725015e0d0 0x7f72500b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:40750/0 (socket says 172.21.15.177:40750) 2024-03-06T07:27:32.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255c00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f7250070370 0x7f725006d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.177:46236/0 (socket says 172.21.15.177:46236) 2024-03-06T07:27:32.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255200640 1 -- 172.21.15.177:0/3367094227 learned_addr learned my addr 172.21.15.177:0/3367094227 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:32.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7247e00640 1 --2- 172.21.15.177:0/3367094227 >> v2:172.21.15.114:3300/0 conn(0x7f725015e4b0 0x7f725006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255200640 1 -- 172.21.15.177:0/3367094227 >> v2:172.21.15.114:3300/0 conn(0x7f725015e4b0 msgr2=0x7f725006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255200640 1 --2- 172.21.15.177:0/3367094227 >> v2:172.21.15.114:3300/0 conn(0x7f725015e4b0 0x7f725006afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255200640 1 -- 172.21.15.177:0/3367094227 >> v1:172.21.15.177:6789/0 conn(0x7f7250070370 legacy=0x7f725006d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.822+0000 7f7255200640 1 -- 172.21.15.177:0/3367094227 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72500b3f20 con 0x7f725015e0d0 2024-03-06T07:27:32.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.823+0000 7f7255200640 1 --2- 172.21.15.177:0/3367094227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f725015e0d0 0x7f72500b4440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f725006af20 tx=0x7f724002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b6af238c6dea442 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f724fe00640 1 -- 172.21.15.177:0/3367094227 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240034070 con 0x7f725015e0d0 2024-03-06T07:27:32.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f724fe00640 1 -- 172.21.15.177:0/3367094227 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7240030860 con 0x7f725015e0d0 2024-03-06T07:27:32.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f724fe00640 1 -- 172.21.15.177:0/3367094227 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240030c30 con 0x7f725015e0d0 2024-03-06T07:27:32.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f7257c00640 1 -- 172.21.15.177:0/3367094227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f725015e0d0 msgr2=0x7f72500b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f7257c00640 1 --2- 172.21.15.177:0/3367094227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f725015e0d0 0x7f72500b4440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f725006af20 tx=0x7f724002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.827 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f7257c00640 1 -- 172.21.15.177:0/3367094227 shutdown_connections 2024-03-06T07:27:32.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f7257c00640 1 --2- 172.21.15.177:0/3367094227 >> v2:172.21.15.114:3300/0 conn(0x7f725015e4b0 0x7f725006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f7257c00640 1 --2- 172.21.15.177:0/3367094227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f725015e0d0 0x7f72500b4440 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f7257c00640 1 -- 172.21.15.177:0/3367094227 >> 172.21.15.177:0/3367094227 conn(0x7f72500d0bf0 msgr2=0x7f72500d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.824+0000 7f7257c00640 1 -- 172.21.15.177:0/3367094227 shutdown_connections 2024-03-06T07:27:32.828 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.825+0000 7f7257c00640 1 -- 172.21.15.177:0/3367094227 wait complete. 2024-03-06T07:27:32.829 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.826+0000 7f7257c00640 1 Processor -- start 2024-03-06T07:27:32.829 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.826+0000 7f7257c00640 1 -- start start 2024-03-06T07:27:32.831 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7257c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250070370 0x7f7250150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7257c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f725015e0d0 0x7f7250157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7257c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f725015e4b0 0x7f7250154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7257c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72500d1890 con 0x7f725015e4b0 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7257c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72500d1710 con 0x7f7250070370 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7257c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72500b4980 con 0x7f725015e0d0 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7255200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250070370 0x7f7250150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7255200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250070370 0x7f7250150340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:40758/0 (socket says 172.21.15.177:40758) 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7255200640 1 -- 172.21.15.177:0/3596813582 learned_addr learned my addr 172.21.15.177:0/3596813582 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.827+0000 7f7247e00640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f725015e0d0 0x7f7250157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255c00640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f725015e4b0 0x7f7250154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255200640 1 -- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f725015e0d0 msgr2=0x7f7250157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255200640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f725015e0d0 0x7f7250157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.832 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255200640 1 -- 172.21.15.177:0/3596813582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f725015e4b0 msgr2=0x7f7250154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255200640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f725015e4b0 0x7f7250154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255200640 1 -- 172.21.15.177:0/3596813582 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725015ee00 con 0x7f7250070370 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255c00640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f725015e4b0 0x7f7250154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7255200640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250070370 0x7f7250150340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f724002b9e0 tx=0x7f7240004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f724d600640 1 -- 172.21.15.177:0/3596813582 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240048070 con 0x7f7250070370 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f724d600640 1 -- 172.21.15.177:0/3596813582 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7240030660 con 0x7f7250070370 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f7257c00640 1 -- 172.21.15.177:0/3596813582 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725015f0e0 con 0x7f7250070370 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.828+0000 7f724d600640 1 -- 172.21.15.177:0/3596813582 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240034040 con 0x7f7250070370 2024-03-06T07:27:32.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.829+0000 7f7257c00640 1 -- 172.21.15.177:0/3596813582 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725015f6a0 con 0x7f7250070370 2024-03-06T07:27:32.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.830+0000 7f724d600640 1 -- 172.21.15.177:0/3596813582 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f724004b070 con 0x7f7250070370 2024-03-06T07:27:32.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.830+0000 7f724d600640 1 -- 172.21.15.177:0/3596813582 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f7240043990 con 0x7f7250070370 2024-03-06T07:27:32.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.830+0000 7f7246000640 1 -- 172.21.15.177:0/3596813582 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725004f3a0 con 0x7f7250070370 2024-03-06T07:27:32.834 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.832+0000 7f724d600640 1 -- 172.21.15.177:0/3596813582 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7240030810 con 0x7f7250070370 2024-03-06T07:27:32.891 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:32.889+0000 7f303142e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:32.913 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.910+0000 7f7246000640 1 -- 172.21.15.177:0/3596813582 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e8265969-0287-457e-968c-d7ec7047e491", "id": 5} v 0) v1 -- 0x7f725015abc0 con 0x7f7250070370 2024-03-06T07:27:32.920 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.918+0000 7f724d600640 1 -- 172.21.15.177:0/3596813582 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8265969-0287-457e-968c-d7ec7047e491", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7240043b90 con 0x7f7250070370 2024-03-06T07:27:32.921 INFO:teuthology.orchestra.run.smithi177.stdout:5 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 -- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250070370 msgr2=0x7f7250150340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250070370 0x7f7250150340 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f724002b9e0 tx=0x7f7240004730 comp rx=0 tx=0).stop 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 -- 172.21.15.177:0/3596813582 shutdown_connections 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f725015e4b0 0x7f7250154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f725015e0d0 0x7f7250157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 --2- 172.21.15.177:0/3596813582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7250070370 0x7f7250150340 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 -- 172.21.15.177:0/3596813582 >> 172.21.15.177:0/3596813582 conn(0x7f72500d0bf0 msgr2=0x7f72500b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 -- 172.21.15.177:0/3596813582 shutdown_connections 2024-03-06T07:27:32.923 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:32.919+0000 7f7246000640 1 -- 172.21.15.177:0/3596813582 wait complete. 2024-03-06T07:27:32.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 4182701d-5859-4c42-bf7f-5b92a73e6ec4 6 2024-03-06T07:27:32.957 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:32.955+0000 7f4b2fcb6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:33.002 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:33.001+0000 7f303142e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:27:33.076 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:33.074+0000 7f4b2fcb6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:27:33.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.120+0000 7f3e2ea00640 1 Processor -- start 2024-03-06T07:27:33.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.120+0000 7f3e2ea00640 1 -- start start 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.121+0000 7f3e2ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e280b3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.121+0000 7f3e2ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3e280b3120 0x7f3e280b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.121+0000 7f3e2ea00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e280d4ce0 con 0x7f3e2815e100 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.121+0000 7f3e2ea00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e280d4a40 con 0x7f3e280b3120 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.121+0000 7f3e2ea00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2815e4e0 con 0x7f3e280d0e60 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e280b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e280b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:43000/0 (socket says 172.21.15.177:43000) 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2cc00640 1 -- 172.21.15.177:0/1995325530 learned_addr learned my addr 172.21.15.177:0/1995325530 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2e000640 1 --2- 172.21.15.177:0/1995325530 >> v2:172.21.15.114:3300/0 conn(0x7f3e280b3120 0x7f3e280b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.125 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2cc00640 1 -- 172.21.15.177:0/1995325530 >> v2:172.21.15.114:3300/0 conn(0x7f3e280b3120 msgr2=0x7f3e280b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.126 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2cc00640 1 --2- 172.21.15.177:0/1995325530 >> v2:172.21.15.114:3300/0 conn(0x7f3e280b3120 0x7f3e280b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.126 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2cc00640 1 -- 172.21.15.177:0/1995325530 >> v1:172.21.15.114:6789/0 conn(0x7f3e2815e100 legacy=0x7f3e280d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.126 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.122+0000 7f3e2cc00640 1 -- 172.21.15.177:0/1995325530 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e280d2470 con 0x7f3e280d0e60 2024-03-06T07:27:33.126 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.123+0000 7f3e2cc00640 1 --2- 172.21.15.177:0/1995325530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e280b3770 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009870 tx=0x7f3e1402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38240d42ad35db41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.126 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e27e00640 1 -- 172.21.15.177:0/1995325530 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14034070 con 0x7f3e280d0e60 2024-03-06T07:27:33.126 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e27e00640 1 -- 172.21.15.177:0/1995325530 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e14030840 con 0x7f3e280d0e60 2024-03-06T07:27:33.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e27e00640 1 -- 172.21.15.177:0/1995325530 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14030c10 con 0x7f3e280d0e60 2024-03-06T07:27:33.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e2ea00640 1 -- 172.21.15.177:0/1995325530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 msgr2=0x7f3e280b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e2ea00640 1 --2- 172.21.15.177:0/1995325530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e280b3770 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009870 tx=0x7f3e1402b980 comp rx=0 tx=0).stop 2024-03-06T07:27:33.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e2ea00640 1 -- 172.21.15.177:0/1995325530 shutdown_connections 2024-03-06T07:27:33.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e2ea00640 1 --2- 172.21.15.177:0/1995325530 >> v2:172.21.15.114:3300/0 conn(0x7f3e280b3120 0x7f3e280b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.128 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e2ea00640 1 --2- 172.21.15.177:0/1995325530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e280b3770 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.128 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.124+0000 7f3e2ea00640 1 -- 172.21.15.177:0/1995325530 >> 172.21.15.177:0/1995325530 conn(0x7f3e280d3d80 msgr2=0x7f3e280d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.128 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.125+0000 7f3e2ea00640 1 -- 172.21.15.177:0/1995325530 shutdown_connections 2024-03-06T07:27:33.128 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.125+0000 7f3e2ea00640 1 -- 172.21.15.177:0/1995325530 wait complete. 2024-03-06T07:27:33.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.126+0000 7f3e2ea00640 1 Processor -- start 2024-03-06T07:27:33.129 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.126+0000 7f3e2ea00640 1 -- start start 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e280b3120 0x7f3e28160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e28152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2815e100 0x7f3e2815bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2ea00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e280cfbd0 con 0x7f3e2815e100 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2ea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e280d2a30 con 0x7f3e280b3120 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2ea00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e280b2bd0 con 0x7f3e280d0e60 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e28152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e280b3120 0x7f3e28160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e28152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.177:43008/0 (socket says 172.21.15.177:43008) 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.127+0000 7f3e2cc00640 1 -- 172.21.15.177:0/3008948712 learned_addr learned my addr 172.21.15.177:0/3008948712 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.128+0000 7f3e2e000640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2815e100 0x7f3e2815bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.128+0000 7f3e2cc00640 1 -- 172.21.15.177:0/3008948712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e280b3120 msgr2=0x7f3e28160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.131 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.128+0000 7f3e2cc00640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e280b3120 0x7f3e28160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.128+0000 7f3e2cc00640 1 -- 172.21.15.177:0/3008948712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2815e100 msgr2=0x7f3e2815bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.128+0000 7f3e2cc00640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2815e100 0x7f3e2815bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.128+0000 7f3e2cc00640 1 -- 172.21.15.177:0/3008948712 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28160b00 con 0x7f3e280d0e60 2024-03-06T07:27:33.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.128+0000 7f3e2cc00640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e28152200 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3e14000c00 tx=0x7f3e14004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.129+0000 7f3e25600640 1 -- 172.21.15.177:0/3008948712 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14048070 con 0x7f3e280d0e60 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.129+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e280c79e0 con 0x7f3e280d0e60 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.129+0000 7f3e25600640 1 -- 172.21.15.177:0/3008948712 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e14030640 con 0x7f3e280d0e60 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.129+0000 7f3e25600640 1 -- 172.21.15.177:0/3008948712 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14034040 con 0x7f3e280d0e60 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.129+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e280c7c80 con 0x7f3e280d0e60 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.129+0000 7f3e25600640 1 -- 172.21.15.177:0/3008948712 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3e14037070 con 0x7f3e280d0e60 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.129+0000 7f3e25600640 1 -- 172.21.15.177:0/3008948712 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f3e14043cd0 con 0x7f3e280d0e60 2024-03-06T07:27:33.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.130+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e280d37c0 con 0x7f3e280d0e60 2024-03-06T07:27:33.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.134+0000 7f3e25600640 1 -- 172.21.15.177:0/3008948712 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3e14051020 con 0x7f3e280d0e60 2024-03-06T07:27:33.179 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:33.177+0000 7f303142e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:33.215 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.213+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4182701d-5859-4c42-bf7f-5b92a73e6ec4", "id": 6} v 0) v1 -- 0x7f3e28152740 con 0x7f3e280d0e60 2024-03-06T07:27:33.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.220+0000 7f3e25600640 1 -- 172.21.15.177:0/3008948712 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4182701d-5859-4c42-bf7f-5b92a73e6ec4", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e140307f0 con 0x7f3e280d0e60 2024-03-06T07:27:33.223 INFO:teuthology.orchestra.run.smithi177.stdout:6 2024-03-06T07:27:33.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.221+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 msgr2=0x7f3e28152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.221+0000 7f3e2ea00640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e28152200 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3e14000c00 tx=0x7f3e14004730 comp rx=0 tx=0).stop 2024-03-06T07:27:33.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.221+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 shutdown_connections 2024-03-06T07:27:33.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.221+0000 7f3e2ea00640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2815e100 0x7f3e2815bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.221+0000 7f3e2ea00640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e280d0e60 0x7f3e28152200 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.221+0000 7f3e2ea00640 1 --2- 172.21.15.177:0/3008948712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e280b3120 0x7f3e28160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.221+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 >> 172.21.15.177:0/3008948712 conn(0x7f3e280d3d80 msgr2=0x7f3e280d19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.222+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 shutdown_connections 2024-03-06T07:27:33.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.222+0000 7f3e2ea00640 1 -- 172.21.15.177:0/3008948712 wait complete. 2024-03-06T07:27:33.229 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:33.227+0000 7f303142e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:33.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 13556954-d129-4900-ac0b-bc34c9368af3 7 2024-03-06T07:27:33.265 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:33.262+0000 7f4b2fcb6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:33.317 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:33.314+0000 7f4b2fcb6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:33.401 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:33.399+0000 7f303142e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:27:33.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.452+0000 7efd63200640 1 Processor -- start 2024-03-06T07:27:33.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.452+0000 7efd63200640 1 -- start start 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.453+0000 7efd63200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd6415e100 0x7efd640d0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.453+0000 7efd63200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efd640b4780 0x7efd640d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.453+0000 7efd63200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640d4ce0 con 0x7efd640b3120 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.453+0000 7efd63200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd640d4a40 con 0x7efd640b4780 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.453+0000 7efd63200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6415e4e0 con 0x7efd6415e100 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd61e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd6415e100 0x7efd640d0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd61e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efd6415e100 0x7efd640d0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:40780/0 (socket says 172.21.15.177:40780) 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd61e00640 1 -- 172.21.15.177:0/4004049735 learned_addr learned my addr 172.21.15.177:0/4004049735 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd62800640 1 --2- 172.21.15.177:0/4004049735 >> v2:172.21.15.114:3300/0 conn(0x7efd640b4780 0x7efd640d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd61e00640 1 -- 172.21.15.177:0/4004049735 >> v2:172.21.15.114:3300/0 conn(0x7efd640b4780 msgr2=0x7efd640d0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd61e00640 1 --2- 172.21.15.177:0/4004049735 >> v2:172.21.15.114:3300/0 conn(0x7efd640b4780 0x7efd640d0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd61e00640 1 -- 172.21.15.177:0/4004049735 >> v1:172.21.15.114:6789/0 conn(0x7efd640b3120 legacy=0x7efd640ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd61e00640 1 -- 172.21.15.177:0/4004049735 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640d2470 con 0x7efd6415e100 2024-03-06T07:27:33.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.454+0000 7efd62800640 1 --2- 172.21.15.177:0/4004049735 >> v2:172.21.15.114:3300/0 conn(0x7efd640b4780 0x7efd640d0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.455+0000 7efd61e00640 1 --2- 172.21.15.177:0/4004049735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd6415e100 0x7efd640d0750 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd6408d730 tx=0x7efd5402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b9153b2b5d8e3cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.455+0000 7efd60a00640 1 -- 172.21.15.177:0/4004049735 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54035020 con 0x7efd6415e100 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.455+0000 7efd60a00640 1 -- 172.21.15.177:0/4004049735 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54031870 con 0x7efd6415e100 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.455+0000 7efd60a00640 1 -- 172.21.15.177:0/4004049735 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54031c40 con 0x7efd6415e100 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 -- 172.21.15.177:0/4004049735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd6415e100 msgr2=0x7efd640d0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 --2- 172.21.15.177:0/4004049735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd6415e100 0x7efd640d0750 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd6408d730 tx=0x7efd5402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 -- 172.21.15.177:0/4004049735 shutdown_connections 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 --2- 172.21.15.177:0/4004049735 >> v2:172.21.15.114:3300/0 conn(0x7efd640b4780 0x7efd640d0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.459 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 --2- 172.21.15.177:0/4004049735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd6415e100 0x7efd640d0750 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 -- 172.21.15.177:0/4004049735 >> 172.21.15.177:0/4004049735 conn(0x7efd640d3d80 msgr2=0x7efd640d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 -- 172.21.15.177:0/4004049735 shutdown_connections 2024-03-06T07:27:33.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.456+0000 7efd63200640 1 -- 172.21.15.177:0/4004049735 wait complete. 2024-03-06T07:27:33.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.459+0000 7efd63200640 1 Processor -- start 2024-03-06T07:27:33.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.459+0000 7efd63200640 1 -- start start 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd63200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd640b3120 0x7efd6415bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd63200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd640b4780 0x7efd640cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd63200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd6415e100 0x7efd640cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd63200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd640cfbd0 con 0x7efd640b4780 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd63200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd640d2a30 con 0x7efd640b3120 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd63200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd640b2bd0 con 0x7efd6415e100 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd640b3120 0x7efd6415bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd640b3120 0x7efd6415bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:40786/0 (socket says 172.21.15.177:40786) 2024-03-06T07:27:33.465 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 -- 172.21.15.177:0/1788416754 learned_addr learned my addr 172.21.15.177:0/1788416754 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd62800640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd6415e100 0x7efd640cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61400640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd640b4780 0x7efd640cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 -- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd6415e100 msgr2=0x7efd640cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd6415e100 0x7efd640cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 -- 172.21.15.177:0/1788416754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd640b4780 msgr2=0x7efd640cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd640b4780 0x7efd640cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.461+0000 7efd61e00640 1 -- 172.21.15.177:0/1788416754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640cb8f0 con 0x7efd640b3120 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.462+0000 7efd61e00640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd640b3120 0x7efd6415bec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efd54000c00 tx=0x7efd540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.462+0000 7efd5a000640 1 -- 172.21.15.177:0/1788416754 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54040070 con 0x7efd640b3120 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.462+0000 7efd5a000640 1 -- 172.21.15.177:0/1788416754 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5403a420 con 0x7efd640b3120 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.462+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd640cbbd0 con 0x7efd640b3120 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.462+0000 7efd5a000640 1 -- 172.21.15.177:0/1788416754 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54035040 con 0x7efd640b3120 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.462+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd640cc190 con 0x7efd640b3120 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.462+0000 7efd5a000640 1 -- 172.21.15.177:0/1788416754 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7efd5404b060 con 0x7efd640b3120 2024-03-06T07:27:33.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.463+0000 7efd5a000640 1 -- 172.21.15.177:0/1788416754 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7efd54031730 con 0x7efd640b3120 2024-03-06T07:27:33.467 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.463+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd640cc970 con 0x7efd640b3120 2024-03-06T07:27:33.468 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.465+0000 7efd5a000640 1 -- 172.21.15.177:0/1788416754 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7efd640cc970 con 0x7efd640b3120 2024-03-06T07:27:33.502 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:33.499+0000 7f4b2fcb6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:27:33.547 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.545+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "13556954-d129-4900-ac0b-bc34c9368af3", "id": 7} v 0) v1 -- 0x7efd640753e0 con 0x7efd640b3120 2024-03-06T07:27:33.554 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.551+0000 7efd5a000640 1 -- 172.21.15.177:0/1788416754 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "13556954-d129-4900-ac0b-bc34c9368af3", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7efd5403aba0 con 0x7efd640b3120 2024-03-06T07:27:33.554 INFO:teuthology.orchestra.run.smithi177.stdout:7 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd640b3120 msgr2=0x7efd6415bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd640b3120 0x7efd6415bec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efd54000c00 tx=0x7efd540046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 shutdown_connections 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd6415e100 0x7efd640cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd640b4780 0x7efd640cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 --2- 172.21.15.177:0/1788416754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd640b3120 0x7efd6415bec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 >> 172.21.15.177:0/1788416754 conn(0x7efd640d3d80 msgr2=0x7efd640d2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 shutdown_connections 2024-03-06T07:27:33.556 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-06T07:27:33.553+0000 7efd63200640 1 -- 172.21.15.177:0/1788416754 wait complete. 2024-03-06T07:27:33.571 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:27:33.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:27:33.579 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:27:33.579 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:27:33.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:27:33.586 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:27:33.586 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:27:33.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:27:33.590 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:27:33.590 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:27:33.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:27:33.595 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:27:33.595 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:27:33.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:27:33.602 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:27:33.602 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:27:33.602 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:27:33.608 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:27:33.608 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:27:33.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:27:33.614 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:27:33.614 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:27:33.614 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:27:33.619 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:27:33.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.843+0000 7f04a4000640 1 Processor -- start 2024-03-06T07:27:33.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.844+0000 7f04a4000640 1 -- start start 2024-03-06T07:27:33.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.845+0000 7f04a4000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f049c0cff20 0x7f049c0b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.845+0000 7f04a4000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0b73e0 0x7f049c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.845+0000 7f04a4000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c0b5e00 con 0x7f049c0b7860 2024-03-06T07:27:33.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.845+0000 7f04a4000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c0b6a10 con 0x7f049c0cff20 2024-03-06T07:27:33.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.845+0000 7f04a4000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0b60e0 con 0x7f049c0b73e0 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a2000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f049c0b7860 0x7f049c0d0840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37686/0 (socket says 172.21.15.114:37686) 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a2000640 1 -- 172.21.15.114:0/4162768118 learned_addr learned my addr 172.21.15.114:0/4162768118 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a0c00640 1 --2- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0b73e0 0x7f049c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a1600640 1 --2- 172.21.15.114:0/4162768118 >> v2:172.21.15.177:3300/0 conn(0x7f049c0cff20 0x7f049c0b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f049be00640 1 -- 172.21.15.114:0/4162768118 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475119305 0 0) 0x7f049c0b5e00 con 0x7f049c0b7860 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f049be00640 1 -- 172.21.15.114:0/4162768118 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0484003520 con 0x7f049c0b7860 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a1600640 1 -- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0b73e0 msgr2=0x7f049c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a1600640 1 --2- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0b73e0 0x7f049c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a1600640 1 -- 172.21.15.114:0/4162768118 >> v1:172.21.15.114:6789/0 conn(0x7f049c0b7860 legacy=0x7f049c0d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.846+0000 7f04a1600640 1 -- 172.21.15.114:0/4162768118 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c0b63c0 con 0x7f049c0cff20 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.847+0000 7f04a1600640 1 --2- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0cff20 0x7f049c0b6ea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f04900097d0 tx=0x7f049002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29630ecba7351713 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.848+0000 7f049be00640 1 -- 172.21.15.114:0/4162768118 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490035020 con 0x7f049c0cff20 2024-03-06T07:27:33.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.848+0000 7f049be00640 1 -- 172.21.15.114:0/4162768118 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0490031800 con 0x7f049c0cff20 2024-03-06T07:27:33.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.848+0000 7f049be00640 1 -- 172.21.15.114:0/4162768118 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490031bb0 con 0x7f049c0cff20 2024-03-06T07:27:33.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 -- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0cff20 msgr2=0x7f049c0b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 --2- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0cff20 0x7f049c0b6ea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f04900097d0 tx=0x7f049002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 -- 172.21.15.114:0/4162768118 shutdown_connections 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 --2- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0b73e0 0x7f049c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 --2- 172.21.15.114:0/4162768118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0cff20 0x7f049c0b6ea0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 -- 172.21.15.114:0/4162768118 >> 172.21.15.114:0/4162768118 conn(0x7f049c0d1c90 msgr2=0x7f049c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 -- 172.21.15.114:0/4162768118 shutdown_connections 2024-03-06T07:27:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.849+0000 7f04a4000640 1 -- 172.21.15.114:0/4162768118 wait complete. 2024-03-06T07:27:33.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.850+0000 7f04a4000640 1 Processor -- start 2024-03-06T07:27:33.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.851+0000 7f04a4000640 1 -- start start 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a4000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c0b73e0 0x7f049c14f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a4000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0b7860 0x7f049c09ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a4000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0cff20 0x7f049c09f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a4000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c161a50 con 0x7f049c0b73e0 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a4000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0b5af0 con 0x7f049c0b7860 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a4000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c0d0f50 con 0x7f049c0cff20 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c0b73e0 0x7f049c14f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c0b73e0 0x7f049c14f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47604/0 (socket says 172.21.15.114:47604) 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 -- 172.21.15.114:0/2625956526 learned_addr learned my addr 172.21.15.114:0/2625956526 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a2000640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0cff20 0x7f049c09f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a0c00640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0b7860 0x7f049c09ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 -- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0cff20 msgr2=0x7f049c09f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0cff20 0x7f049c09f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 -- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0b7860 msgr2=0x7f049c09ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0b7860 0x7f049c09ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.852+0000 7f04a1600640 1 -- 172.21.15.114:0/2625956526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c152e30 con 0x7f049c0b73e0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f04a1600640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c0b73e0 0x7f049c14f820 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f04900098f0 tx=0x7f04900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f0499600640 1 -- 172.21.15.114:0/2625956526 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490041070 con 0x7f049c0b73e0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f0499600640 1 -- 172.21.15.114:0/2625956526 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0490031e30 con 0x7f049c0b73e0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c153110 con 0x7f049c0b73e0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f0499600640 1 -- 172.21.15.114:0/2625956526 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490035040 con 0x7f049c0b73e0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c150540 con 0x7f049c0b73e0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f0499600640 1 -- 172.21.15.114:0/2625956526 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f049004c060 con 0x7f049c0b73e0 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.853+0000 7f0499600640 1 -- 172.21.15.114:0/2625956526 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0490045400 con 0x7f049c0b73e0 2024-03-06T07:27:33.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.854+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c04f330 con 0x7f049c0b73e0 2024-03-06T07:27:33.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.856+0000 7f0499600640 1 -- 172.21.15.114:0/2625956526 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f049003b7f0 con 0x7f049c0b73e0 2024-03-06T07:27:33.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.923+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f049c0684a0 con 0x7f049c0b73e0 2024-03-06T07:27:33.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.924+0000 7f0499600640 1 -- 172.21.15.114:0/2625956526 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f049003b9d0 con 0x7f049c0b73e0 2024-03-06T07:27:33.925 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:33.926 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":10,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:33.548233+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:33.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c0b73e0 msgr2=0x7f049c14f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c0b73e0 0x7f049c14f820 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f04900098f0 tx=0x7f04900046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 shutdown_connections 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f049c0cff20 0x7f049c09f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f049c0b7860 0x7f049c09ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 --2- 172.21.15.114:0/2625956526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f049c0b73e0 0x7f049c14f820 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 >> 172.21.15.114:0/2625956526 conn(0x7f049c0d1c90 msgr2=0x7f049c0c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 shutdown_connections 2024-03-06T07:27:33.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:33.925+0000 7f04a4000640 1 -- 172.21.15.114:0/2625956526 wait complete. 2024-03-06T07:27:33.943 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:27:33.943 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:27:34.285 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-03-06T07:27:34.283+0000 7f0683e1c640 -1 Falling back to public interface 2024-03-06T07:27:34.296 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-03-06T07:27:34.294+0000 7f0b9e611640 -1 Falling back to public interface 2024-03-06T07:27:34.298 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-06T07:27:34.297+0000 7fc5bf6bf640 -1 Falling back to public interface 2024-03-06T07:27:34.302 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-06T07:27:34.301+0000 7f9332d9a640 -1 Falling back to public interface 2024-03-06T07:27:34.307 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-06T07:27:34.305+0000 7ff73f00d640 -1 Falling back to public interface 2024-03-06T07:27:34.310 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-06T07:27:34.308+0000 7fe6483e1640 -1 Falling back to public interface 2024-03-06T07:27:34.313 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-03-06T07:27:34.311+0000 7fcba52b3640 -1 Falling back to public interface 2024-03-06T07:27:34.333 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-03-06T07:27:34.331+0000 7f8c8ce0f640 -1 Falling back to public interface 2024-03-06T07:27:34.559 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:34.557+0000 7f303142e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:27:34.614 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:34.611+0000 7f4b2fcb6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:27:34.748 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:34.875 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:34.874+0000 7f303142e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:27:34.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.911+0000 7fc6e0200640 1 Processor -- start 2024-03-06T07:27:34.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.911+0000 7fc6e0200640 1 -- start start 2024-03-06T07:27:34.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.912+0000 7fc6e0200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc6d80cf240 0x7fc6d8162bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.912+0000 7fc6e0200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d806cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.912+0000 7fc6e0200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d8071530 con 0x7fc6d806cff0 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.912+0000 7fc6e0200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d8072140 con 0x7fc6d80cf240 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.912+0000 7fc6e0200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d8071810 con 0x7fc6d8070d50 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dd800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc6d80cf240 0x7fc6d8162bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d806cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dd800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc6d80cf240 0x7fc6d8162bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:53540/0 (socket says 172.21.15.114:53540) 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6de200640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fc6d806cff0 0x7fc6d80b29b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:57062/0 (socket says 172.21.15.114:57062) 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dd800640 1 -- 172.21.15.114:0/797517471 learned_addr learned my addr 172.21.15.114:0/797517471 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d806cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:44024/0 (socket says 172.21.15.114:44024) 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dd800640 1 -- 172.21.15.114:0/797517471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 msgr2=0x7fc6d806cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.916 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dd800640 1 --2- 172.21.15.114:0/797517471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d806cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dd800640 1 -- 172.21.15.114:0/797517471 >> v1:172.21.15.177:6789/0 conn(0x7fc6d806cff0 legacy=0x7fc6d80b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.917 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.913+0000 7fc6dd800640 1 -- 172.21.15.114:0/797517471 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d8071af0 con 0x7fc6d80cf240 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.914+0000 7fc6dd800640 1 --2- 172.21.15.114:0/797517471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d80cf240 0x7fc6d8162bb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3c943f8b5f97ab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6d7e00640 1 -- 172.21.15.114:0/797517471 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc034070 con 0x7fc6d80cf240 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6d7e00640 1 -- 172.21.15.114:0/797517471 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc030840 con 0x7fc6d80cf240 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6d7e00640 1 -- 172.21.15.114:0/797517471 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc030c10 con 0x7fc6d80cf240 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 -- 172.21.15.114:0/797517471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d80cf240 msgr2=0x7fc6d8162bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 --2- 172.21.15.114:0/797517471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d80cf240 0x7fc6d8162bb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 -- 172.21.15.114:0/797517471 shutdown_connections 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 --2- 172.21.15.114:0/797517471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d806cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 --2- 172.21.15.114:0/797517471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d80cf240 0x7fc6d8162bb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 -- 172.21.15.114:0/797517471 >> 172.21.15.114:0/797517471 conn(0x7fc6d80b16b0 msgr2=0x7fc6d80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 -- 172.21.15.114:0/797517471 shutdown_connections 2024-03-06T07:27:34.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.915+0000 7fc6e0200640 1 -- 172.21.15.114:0/797517471 wait complete. 2024-03-06T07:27:34.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.917+0000 7fc6e0200640 1 Processor -- start 2024-03-06T07:27:34.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.917+0000 7fc6e0200640 1 -- start start 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6e0200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d806cff0 0x7fc6d8161850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6e0200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d8147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6e0200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d80cf240 0x7fc6d815b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6e0200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80711d0 con 0x7fc6d80cf240 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6e0200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80725d0 con 0x7fc6d806cff0 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6e0200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d80b30c0 con 0x7fc6d8070d50 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6de200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d80cf240 0x7fc6d815b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6dd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d806cff0 0x7fc6d8161850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6dce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d8147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6dd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d806cff0 0x7fc6d8161850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:53556/0 (socket says 172.21.15.114:53556) 2024-03-06T07:27:34.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.918+0000 7fc6dd800640 1 -- 172.21.15.114:0/782343549 learned_addr learned my addr 172.21.15.114:0/782343549 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:34.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6dd800640 1 -- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 msgr2=0x7fc6d8147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6dd800640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d8147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6dd800640 1 -- 172.21.15.114:0/782343549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d80cf240 msgr2=0x7fc6d815b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6dd800640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d80cf240 0x7fc6d815b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6dd800640 1 -- 172.21.15.114:0/782343549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d81f8ca0 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6de200640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d80cf240 0x7fc6d815b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6dce00640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d8147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6dd800640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d806cff0 0x7fc6d8161850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc000c00 tx=0x7fc6cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6d5600640 1 -- 172.21.15.114:0/782343549 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc034040 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6d5600640 1 -- 172.21.15.114:0/782343549 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc039d90 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d8146830 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.919+0000 7fc6d5600640 1 -- 172.21.15.114:0/782343549 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc043630 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.920+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d8161d90 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.920+0000 7fc6d5600640 1 -- 172.21.15.114:0/782343549 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc6cc04a060 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.920+0000 7fc6d5600640 1 -- 172.21.15.114:0/782343549 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc6cc0437d0 con 0x7fc6d806cff0 2024-03-06T07:27:34.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.921+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d804f330 con 0x7fc6d806cff0 2024-03-06T07:27:34.925 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.924+0000 7fc6d5600640 1 -- 172.21.15.114:0/782343549 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc6cc03e070 con 0x7fc6d806cff0 2024-03-06T07:27:34.940 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:34.939+0000 7f303142e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:27:34.948 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:34.946+0000 7f4b2fcb6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:27:34.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.990+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6d8146600 con 0x7fc6d806cff0 2024-03-06T07:27:34.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.992+0000 7fc6d5600640 1 -- 172.21.15.114:0/782343549 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc6cc043ab0 con 0x7fc6d806cff0 2024-03-06T07:27:34.994 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-06T07:27:34.994 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:33.548233+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:34.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d806cff0 msgr2=0x7fc6d8161850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d806cff0 0x7fc6d8161850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc000c00 tx=0x7fc6cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 shutdown_connections 2024-03-06T07:27:34.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6d80cf240 0x7fc6d815b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6d8070d50 0x7fc6d8147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 --2- 172.21.15.114:0/782343549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d806cff0 0x7fc6d8161850 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 >> 172.21.15.114:0/782343549 conn(0x7fc6d80b16b0 msgr2=0x7fc6d81607d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 shutdown_connections 2024-03-06T07:27:34.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:34.994+0000 7fc6e0200640 1 -- 172.21.15.114:0/782343549 wait complete. 2024-03-06T07:27:35.007 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:35.004+0000 7f4b2fcb6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:27:35.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:27:35.074 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:35.073+0000 7f303142e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:27:35.148 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-03-06T07:27:35.146+0000 7f4b2fcb6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:27:36.114 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:36.112+0000 7f2fd1600640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:27:36.121 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-06T07:27:36.120+0000 7f2fd1600640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:27:37.803 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-03-06T07:27:37.800+0000 7f0683e1c640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:27:37.863 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-06T07:27:37.862+0000 7fc5bf6bf640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:27:37.865 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-06T07:27:37.864+0000 7fe6483e1640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:27:37.867 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-03-06T07:27:37.865+0000 7f0b9e611640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:27:37.876 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-06T07:27:37.874+0000 7f9332d9a640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:27:37.876 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-03-06T07:27:37.874+0000 7fcba52b3640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:27:37.887 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-03-06T07:27:37.885+0000 7f8c8ce0f640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:27:37.894 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-06T07:27:37.892+0000 7ff73f00d640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:27:41.347 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-06T07:27:41.346+0000 7ff73ac00640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:27:41.352 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-06T07:27:41.350+0000 7ff72f800640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.394 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-06T07:27:41.393+0000 7f932f000640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:27:41.405 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-06T07:27:41.404+0000 7f9323c00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.410 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-03-06T07:27:41.408+0000 7f8c87200640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:27:41.414 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-06T07:27:41.413+0000 7fc5bba00640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:27:41.416 INFO:tasks.ceph.osd.7.smithi177.stderr:2024-03-06T07:27:41.413+0000 7f8c7dc00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.419 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-06T07:27:41.418+0000 7fc5b0600640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.435 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-03-06T07:27:41.433+0000 7f067dc00640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:27:41.444 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-03-06T07:27:41.441+0000 7f0674600640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.445 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-06T07:27:41.444+0000 7fe642800640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:27:41.452 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-03-06T07:27:41.449+0000 7fcba1600640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:27:41.461 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-06T07:27:41.459+0000 7fe639200640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.461 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-03-06T07:27:41.459+0000 7f0b98400640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:27:41.464 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-03-06T07:27:41.461+0000 7fcb96200640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.480 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-03-06T07:27:41.478+0000 7f0b8ee00640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:41.815 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:41.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.994+0000 7f44e2a00640 1 Processor -- start 2024-03-06T07:27:41.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.994+0000 7f44e2a00640 1 -- start start 2024-03-06T07:27:41.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.996+0000 7f44e2a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f44dc15fbf0 0x7f44dc160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.996+0000 7f44e2a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc0712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.996+0000 7f44e2a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc0cedd0 con 0x7f44dc13eeb0 2024-03-06T07:27:41.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.996+0000 7f44e2a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44dc0cf0b0 con 0x7f44dc15fbf0 2024-03-06T07:27:41.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.996+0000 7f44e2a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc0cf390 con 0x7f44dc070ec0 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44e0a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc0712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44dbe00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f44dc13eeb0 0x7f44dc0ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:41218/0 (socket says 172.21.15.114:41218) 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44dbe00640 1 -- 172.21.15.114:0/1252990387 learned_addr learned my addr 172.21.15.114:0/1252990387 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44d3400640 1 --2- 172.21.15.114:0/1252990387 >> v2:172.21.15.177:3300/0 conn(0x7f44dc15fbf0 0x7f44dc160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44db400640 1 -- 172.21.15.114:0/1252990387 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017584992 0 0) 0x7f44dc0cedd0 con 0x7f44dc13eeb0 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44db400640 1 -- 172.21.15.114:0/1252990387 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c0003520 con 0x7f44dc13eeb0 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44e0a00640 1 -- 172.21.15.114:0/1252990387 >> v2:172.21.15.177:3300/0 conn(0x7f44dc15fbf0 msgr2=0x7f44dc160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44e0a00640 1 --2- 172.21.15.114:0/1252990387 >> v2:172.21.15.177:3300/0 conn(0x7f44dc15fbf0 0x7f44dc160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44e0a00640 1 -- 172.21.15.114:0/1252990387 >> v1:172.21.15.177:6789/0 conn(0x7f44dc13eeb0 legacy=0x7f44dc0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44e0a00640 1 -- 172.21.15.114:0/1252990387 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc0cf670 con 0x7f44dc070ec0 2024-03-06T07:27:42.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.997+0000 7f44d3400640 1 --2- 172.21.15.114:0/1252990387 >> v2:172.21.15.177:3300/0 conn(0x7f44dc15fbf0 0x7f44dc160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:42.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.999+0000 7f44e0a00640 1 --2- 172.21.15.114:0/1252990387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc0712f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009d90 tx=0x7f44d402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e939ffd298b06bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.999+0000 7f44db400640 1 -- 172.21.15.114:0/1252990387 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4039070 con 0x7f44dc070ec0 2024-03-06T07:27:42.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.999+0000 7f44db400640 1 -- 172.21.15.114:0/1252990387 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d402cbc0 con 0x7f44dc070ec0 2024-03-06T07:27:42.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:41.999+0000 7f44db400640 1 -- 172.21.15.114:0/1252990387 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4034440 con 0x7f44dc070ec0 2024-03-06T07:27:42.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 -- 172.21.15.114:0/1252990387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 msgr2=0x7f44dc0712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1252990387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc0712f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009d90 tx=0x7f44d402be10 comp rx=0 tx=0).stop 2024-03-06T07:27:42.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 -- 172.21.15.114:0/1252990387 shutdown_connections 2024-03-06T07:27:42.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1252990387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc0712f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1252990387 >> v2:172.21.15.177:3300/0 conn(0x7f44dc15fbf0 0x7f44dc160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 -- 172.21.15.114:0/1252990387 >> 172.21.15.114:0/1252990387 conn(0x7f44dc0b16b0 msgr2=0x7f44dc0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 -- 172.21.15.114:0/1252990387 shutdown_connections 2024-03-06T07:27:42.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.000+0000 7f44e2a00640 1 -- 172.21.15.114:0/1252990387 wait complete. 2024-03-06T07:27:42.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.002+0000 7f44e2a00640 1 Processor -- start 2024-03-06T07:27:42.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.002+0000 7f44e2a00640 1 -- start start 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.003+0000 7f44e2a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.003+0000 7f44e2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc13eeb0 0x7f44dc0b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.003+0000 7f44e2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44dc15fbf0 0x7f44dc0b4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.003+0000 7f44e2a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc160a30 con 0x7f44dc13eeb0 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.003+0000 7f44e2a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc1608b0 con 0x7f44dc15fbf0 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.003+0000 7f44e2a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc0ce990 con 0x7f44dc070ec0 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc13eeb0 0x7f44dc0b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc13eeb0 0x7f44dc0b4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43728/0 (socket says 172.21.15.114:43728) 2024-03-06T07:27:42.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 -- 172.21.15.114:0/1695205062 learned_addr learned my addr 172.21.15.114:0/1695205062 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44dbe00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44e0a00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44dc15fbf0 0x7f44dc0b4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 -- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 msgr2=0x7f44dc15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 -- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44dc15fbf0 msgr2=0x7f44dc0b4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44dc15fbf0 0x7f44dc0b4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 -- 172.21.15.114:0/1695205062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc0b2c50 con 0x7f44dc13eeb0 2024-03-06T07:27:42.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.004+0000 7f44d3400640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc13eeb0 0x7f44dc0b4100 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44c400e960 tx=0x7f44c400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.005+0000 7f44d8c00640 1 -- 172.21.15.114:0/1695205062 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4017070 con 0x7f44dc13eeb0 2024-03-06T07:27:42.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.005+0000 7f44d8c00640 1 -- 172.21.15.114:0/1695205062 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c400ca00 con 0x7f44dc13eeb0 2024-03-06T07:27:42.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.005+0000 7f44d8c00640 1 -- 172.21.15.114:0/1695205062 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c400cdd0 con 0x7f44dc13eeb0 2024-03-06T07:27:42.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.005+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44dc0b2f30 con 0x7f44dc13eeb0 2024-03-06T07:27:42.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.005+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44dc0b34f0 con 0x7f44dc13eeb0 2024-03-06T07:27:42.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.006+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44dc04f330 con 0x7f44dc13eeb0 2024-03-06T07:27:42.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.007+0000 7f44d8c00640 1 -- 172.21.15.114:0/1695205062 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f44c401a070 con 0x7f44dc13eeb0 2024-03-06T07:27:42.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.008+0000 7f44d8c00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44b806b440 0x7f44b806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.008+0000 7f44d8c00640 1 -- 172.21.15.114:0/1695205062 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f44c4089660 con 0x7f44dc13eeb0 2024-03-06T07:27:42.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.008+0000 7f44dbe00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44b806b440 0x7f44b806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.009+0000 7f44dbe00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44b806b440 0x7f44b806d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44cc001200 tx=0x7f44cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.014+0000 7f44d8c00640 1 -- 172.21.15.114:0/1695205062 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44c40592b0 con 0x7f44dc13eeb0 2024-03-06T07:27:42.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.207+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44dc071990 con 0x7f44dc13eeb0 2024-03-06T07:27:42.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.208+0000 7f44d8c00640 1 -- 172.21.15.114:0/1695205062 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f44c4058bb0 con 0x7f44dc13eeb0 2024-03-06T07:27:42.209 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-06T07:27:42.209 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":12,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:39.119889+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44b806b440 msgr2=0x7f44b806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44b806b440 0x7f44b806d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f44cc001200 tx=0x7f44cc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc13eeb0 msgr2=0x7f44dc0b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc13eeb0 0x7f44dc0b4100 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44c400e960 tx=0x7f44c400ee20 comp rx=0 tx=0).stop 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 shutdown_connections 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44dc15fbf0 0x7f44dc0b4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44b806b440 0x7f44b806d900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc13eeb0 0x7f44dc0b4100 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 --2- 172.21.15.114:0/1695205062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44dc070ec0 0x7f44dc15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 >> 172.21.15.114:0/1695205062 conn(0x7f44dc0b16b0 msgr2=0x7f44dc15d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 shutdown_connections 2024-03-06T07:27:42.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:42.210+0000 7f44e2a00640 1 -- 172.21.15.114:0/1695205062 wait complete. 2024-03-06T07:27:42.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:27:44.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:44.310 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:49.031 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:49.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.211+0000 7f91dcc00640 1 Processor -- start 2024-03-06T07:27:49.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.212+0000 7f91dcc00640 1 -- start start 2024-03-06T07:27:49.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.213+0000 7f91dcc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f91d80cf240 0x7f91d8162bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.213+0000 7f91dcc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d806d1c0 0x7f91d816c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.213+0000 7f91dcc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d8071530 con 0x7f91d8070d50 2024-03-06T07:27:49.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.213+0000 7f91dcc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d8072140 con 0x7f91d80cf240 2024-03-06T07:27:49.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.213+0000 7f91dcc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8071810 con 0x7f91d806d1c0 2024-03-06T07:27:49.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.214+0000 7f91d6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d806d1c0 0x7f91d816c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.214+0000 7f91d6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d806d1c0 0x7f91d816c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:54566/0 (socket says 172.21.15.114:54566) 2024-03-06T07:27:49.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.214+0000 7f91d6800640 1 -- 172.21.15.114:0/557960559 learned_addr learned my addr 172.21.15.114:0/557960559 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:49.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.214+0000 7f91d5e00640 1 --2- 172.21.15.114:0/557960559 >> v2:172.21.15.177:3300/0 conn(0x7f91d80cf240 0x7f91d8162bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.215+0000 7f91d6800640 1 -- 172.21.15.114:0/557960559 >> v2:172.21.15.177:3300/0 conn(0x7f91d80cf240 msgr2=0x7f91d8162bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.215+0000 7f91d6800640 1 --2- 172.21.15.114:0/557960559 >> v2:172.21.15.177:3300/0 conn(0x7f91d80cf240 0x7f91d8162bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.215+0000 7f91d6800640 1 -- 172.21.15.114:0/557960559 >> v1:172.21.15.177:6789/0 conn(0x7f91d8070d50 legacy=0x7f91d806cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.215+0000 7f91d6800640 1 -- 172.21.15.114:0/557960559 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d8071af0 con 0x7f91d806d1c0 2024-03-06T07:27:49.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.216+0000 7f91d6800640 1 --2- 172.21.15.114:0/557960559 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d806d1c0 0x7f91d816c1d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009e50 tx=0x7f91d002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a85d652f75c9481f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.216+0000 7f91d5400640 1 -- 172.21.15.114:0/557960559 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d002cb00 con 0x7f91d806d1c0 2024-03-06T07:27:49.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91d5400640 1 -- 172.21.15.114:0/557960559 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d002cca0 con 0x7f91d806d1c0 2024-03-06T07:27:49.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91d5400640 1 -- 172.21.15.114:0/557960559 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0034610 con 0x7f91d806d1c0 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 -- 172.21.15.114:0/557960559 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d806d1c0 msgr2=0x7f91d816c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 --2- 172.21.15.114:0/557960559 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d806d1c0 0x7f91d816c1d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009e50 tx=0x7f91d002bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 -- 172.21.15.114:0/557960559 shutdown_connections 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 --2- 172.21.15.114:0/557960559 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d806d1c0 0x7f91d816c1d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 --2- 172.21.15.114:0/557960559 >> v2:172.21.15.177:3300/0 conn(0x7f91d80cf240 0x7f91d8162bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 -- 172.21.15.114:0/557960559 >> 172.21.15.114:0/557960559 conn(0x7f91d80b16b0 msgr2=0x7f91d80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 -- 172.21.15.114:0/557960559 shutdown_connections 2024-03-06T07:27:49.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.217+0000 7f91dcc00640 1 -- 172.21.15.114:0/557960559 wait complete. 2024-03-06T07:27:49.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.219+0000 7f91dcc00640 1 Processor -- start 2024-03-06T07:27:49.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.219+0000 7f91dcc00640 1 -- start start 2024-03-06T07:27:49.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91dcc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91d806d1c0 0x7f91d815ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91dcc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d8070d50 0x7f91d80b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91d80cf240 0x7f91d80b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91dcc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8071200 con 0x7f91d806d1c0 2024-03-06T07:27:49.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91dcc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80725d0 con 0x7f91d80cf240 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91dcc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80b39c0 con 0x7f91d8070d50 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91d5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91d806d1c0 0x7f91d815ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91d5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91d806d1c0 0x7f91d815ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46370/0 (socket says 172.21.15.114:46370) 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.220+0000 7f91d5e00640 1 -- 172.21.15.114:0/284427041 learned_addr learned my addr 172.21.15.114:0/284427041 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91d5e00640 1 -- 172.21.15.114:0/284427041 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d8070d50 msgr2=0x7f91d80b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91d5e00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d8070d50 0x7f91d80b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91d5e00640 1 -- 172.21.15.114:0/284427041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91d80cf240 msgr2=0x7f91d80b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91d5e00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91d80cf240 0x7f91d80b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91d5e00640 1 -- 172.21.15.114:0/284427041 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d8147480 con 0x7f91d806d1c0 2024-03-06T07:27:49.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91d5e00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91d806d1c0 0x7f91d815ba20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f91c800e870 tx=0x7f91c800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91ce000640 1 -- 172.21.15.114:0/284427041 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8017070 con 0x7f91d806d1c0 2024-03-06T07:27:49.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91ce000640 1 -- 172.21.15.114:0/284427041 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c800c940 con 0x7f91d806d1c0 2024-03-06T07:27:49.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91ce000640 1 -- 172.21.15.114:0/284427041 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c800cd30 con 0x7f91d806d1c0 2024-03-06T07:27:49.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d8147760 con 0x7f91d806d1c0 2024-03-06T07:27:49.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.221+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d8144150 con 0x7f91d806d1c0 2024-03-06T07:27:49.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.223+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d804f330 con 0x7f91d806d1c0 2024-03-06T07:27:49.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.224+0000 7f91ce000640 1 -- 172.21.15.114:0/284427041 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91c801a070 con 0x7f91d806d1c0 2024-03-06T07:27:49.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.224+0000 7f91ce000640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91b406b4d0 0x7f91b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.225+0000 7f91ce000640 1 -- 172.21.15.114:0/284427041 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f91c808a1f0 con 0x7f91d806d1c0 2024-03-06T07:27:49.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.225+0000 7f91cfe00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91b406b4d0 0x7f91b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.225+0000 7f91cfe00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91b406b4d0 0x7f91b406d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91c0000f60 tx=0x7f91c00079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.230+0000 7f91ce000640 1 -- 172.21.15.114:0/284427041 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91c8059320 con 0x7f91d806d1c0 2024-03-06T07:27:49.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.425+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91d806cca0 con 0x7f91d806d1c0 2024-03-06T07:27:49.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.427+0000 7f91ce000640 1 -- 172.21.15.114:0/284427041 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12932 (secure 0 0 0) 0x7f91c8058c20 con 0x7f91d806d1c0 2024-03-06T07:27:49.428 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-06T07:27:49.429 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":16,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:45.393016+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91b406b4d0 msgr2=0x7f91b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91b406b4d0 0x7f91b406d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f91c0000f60 tx=0x7f91c00079d0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91d806d1c0 msgr2=0x7f91d815ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91d806d1c0 0x7f91d815ba20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f91c800e870 tx=0x7f91c800ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 shutdown_connections 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91d80cf240 0x7f91d80b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91d8070d50 0x7f91d80b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91b406b4d0 0x7f91b406d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 --2- 172.21.15.114:0/284427041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91d806d1c0 0x7f91d815ba20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 >> 172.21.15.114:0/284427041 conn(0x7f91d80b16b0 msgr2=0x7f91d80af520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 shutdown_connections 2024-03-06T07:27:49.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-06T07:27:49.429+0000 7f91dcc00640 1 -- 172.21.15.114:0/284427041 wait complete. 2024-03-06T07:27:49.445 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:27:49.446 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:27:49.446 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:49.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:49.450 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:49.451 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:49.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:49.457 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:49.457 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:49.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:49.465 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:49.465 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:49.465 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:49.472 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:49.473 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:27:49.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:49.602 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:27:49.604 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:27:49.623 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:27:49.630 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:27:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.656+0000 7f5497e00640 1 Processor -- start 2024-03-06T07:27:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.656+0000 7f5497e00640 1 -- start start 2024-03-06T07:27:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.657+0000 7f5497e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5498161580 0x7f54980d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.657+0000 7f5497e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54980d2a70 0x7f54980b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.657+0000 7f5497e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5498161960 con 0x7f54980d1e90 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.657+0000 7f5497e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54980cf6f0 con 0x7f54980d2a70 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.657+0000 7f5497e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54980cf9d0 con 0x7f5498161580 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.657+0000 7f5497400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54980d2a70 0x7f54980b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.657+0000 7f5497400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f54980d2a70 0x7f54980b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46408/0 (socket says 172.21.15.114:46408) 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.658+0000 7f5497400640 1 -- 172.21.15.114:0/172851566 learned_addr learned my addr 172.21.15.114:0/172851566 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.658+0000 7f5496a00640 1 --2- 172.21.15.114:0/172851566 >> v2:172.21.15.177:3300/0 conn(0x7f5498161580 0x7f54980d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.658+0000 7f5497400640 1 -- 172.21.15.114:0/172851566 >> v2:172.21.15.177:3300/0 conn(0x7f5498161580 msgr2=0x7f54980d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.658+0000 7f5497400640 1 --2- 172.21.15.114:0/172851566 >> v2:172.21.15.177:3300/0 conn(0x7f5498161580 0x7f54980d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.658+0000 7f5497400640 1 -- 172.21.15.114:0/172851566 >> v1:172.21.15.177:6789/0 conn(0x7f54980d1e90 legacy=0x7f54980d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.658+0000 7f5497400640 1 -- 172.21.15.114:0/172851566 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980cfcb0 con 0x7f54980d2a70 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.659+0000 7f5497400640 1 --2- 172.21.15.114:0/172851566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980b3ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5498073740 tx=0x7f549002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdb41a8e82778 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.659+0000 7f5495600640 1 -- 172.21.15.114:0/172851566 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549002cb20 con 0x7f54980d2a70 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.659+0000 7f5495600640 1 -- 172.21.15.114:0/172851566 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549002ccc0 con 0x7f54980d2a70 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.659+0000 7f5495600640 1 -- 172.21.15.114:0/172851566 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490034610 con 0x7f54980d2a70 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 -- 172.21.15.114:0/172851566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 msgr2=0x7f54980b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 --2- 172.21.15.114:0/172851566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980b3ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5498073740 tx=0x7f549002bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 -- 172.21.15.114:0/172851566 shutdown_connections 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 --2- 172.21.15.114:0/172851566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980b3ec0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 --2- 172.21.15.114:0/172851566 >> v2:172.21.15.177:3300/0 conn(0x7f5498161580 0x7f54980d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 -- 172.21.15.114:0/172851566 >> 172.21.15.114:0/172851566 conn(0x7f54980d3d80 msgr2=0x7f54980d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 -- 172.21.15.114:0/172851566 shutdown_connections 2024-03-06T07:27:49.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.660+0000 7f5497e00640 1 -- 172.21.15.114:0/172851566 wait complete. 2024-03-06T07:27:49.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.662+0000 7f5497e00640 1 Processor -- start 2024-03-06T07:27:49.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.662+0000 7f5497e00640 1 -- start start 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5497e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54980d1e90 0x7f5498150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5497e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5497e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5498161580 0x7f54980b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5497e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54980cf3a0 con 0x7f54980d2a70 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5497e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54980b39a0 con 0x7f5498161580 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5497e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54980b4400 con 0x7f54980d1e90 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46422/0 (socket says 172.21.15.114:46422) 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 -- 172.21.15.114:0/1507287689 learned_addr learned my addr 172.21.15.114:0/1507287689 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496a00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54980d1e90 0x7f5498150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5497400640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5498161580 0x7f54980b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 -- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54980d1e90 msgr2=0x7f5498150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54980d1e90 0x7f5498150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 -- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5498161580 msgr2=0x7f54980b8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5498161580 0x7f54980b8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496000640 1 -- 172.21.15.114:0/1507287689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980b6a10 con 0x7f54980d2a70 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.663+0000 7f5496a00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54980d1e90 0x7f5498150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.664+0000 7f5496000640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980cbfd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f548400c430 tx=0x7f548400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.664+0000 7f548ea00640 1 -- 172.21.15.114:0/1507287689 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54840052d0 con 0x7f54980d2a70 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.664+0000 7f548ea00640 1 -- 172.21.15.114:0/1507287689 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5484005490 con 0x7f54980d2a70 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.664+0000 7f548ea00640 1 -- 172.21.15.114:0/1507287689 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484007260 con 0x7f54980d2a70 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.664+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54980b6c80 con 0x7f54980d2a70 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.664+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5498150690 con 0x7f54980d2a70 2024-03-06T07:27:49.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.665+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549808dde0 con 0x7f54980d2a70 2024-03-06T07:27:49.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.666+0000 7f548ea00640 1 -- 172.21.15.114:0/1507287689 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5484007400 con 0x7f54980d2a70 2024-03-06T07:27:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.667+0000 7f548ea00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f547806b4d0 0x7f547806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.668+0000 7f548ea00640 1 -- 172.21.15.114:0/1507287689 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54840892f0 con 0x7f54980d2a70 2024-03-06T07:27:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.668+0000 7f5496a00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f547806b4d0 0x7f547806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.668+0000 7f5496a00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f547806b4d0 0x7f547806d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5488009af0 tx=0x7f5488008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.673+0000 7f548ea00640 1 -- 172.21.15.114:0/1507287689 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f549808dde0 con 0x7f54980d2a70 2024-03-06T07:27:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.867+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f549804f330 con 0x7f54980d2a70 2024-03-06T07:27:49.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.869+0000 7f548ea00640 1 -- 172.21.15.114:0/1507287689 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12932 (secure 0 0 0) 0x7f5484057e70 con 0x7f54980d2a70 2024-03-06T07:27:49.871 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:49.871 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":16,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:45.393016+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f547806b4d0 msgr2=0x7f547806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f547806b4d0 0x7f547806d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5488009af0 tx=0x7f5488008040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 msgr2=0x7f54980cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980cbfd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f548400c430 tx=0x7f548400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 shutdown_connections 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5498161580 0x7f54980b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f547806b4d0 0x7f547806d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54980d2a70 0x7f54980cbfd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 --2- 172.21.15.114:0/1507287689 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f54980d1e90 0x7f5498150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 >> 172.21.15.114:0/1507287689 conn(0x7f54980d3d80 msgr2=0x7f54980cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 shutdown_connections 2024-03-06T07:27:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:49.871+0000 7f5497e00640 1 -- 172.21.15.114:0/1507287689 wait complete. 2024-03-06T07:27:49.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:49.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:50.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.087+0000 7f8868000640 1 Processor -- start 2024-03-06T07:27:50.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.087+0000 7f8868000640 1 -- start start 2024-03-06T07:27:50.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.088+0000 7f8868000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f88600b3530 0x7f886015b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.088+0000 7f8868000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f88600acf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8868000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886015bcc0 con 0x7f88600ad4a0 2024-03-06T07:27:50.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8868000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88600ce690 con 0x7f88600b3530 2024-03-06T07:27:50.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8868000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88600ce970 con 0x7f88600b3910 2024-03-06T07:27:50.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8864c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f88600acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8865600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f88600b3530 0x7f886015b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8864c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f88600acf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:54622/0 (socket says 172.21.15.114:54622) 2024-03-06T07:27:50.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8866000640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f88600ad4a0 0x7f88600ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:53184/0 (socket says 172.21.15.114:53184) 2024-03-06T07:27:50.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.089+0000 7f8864c00640 1 -- 172.21.15.114:0/159539080 learned_addr learned my addr 172.21.15.114:0/159539080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:50.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.090+0000 7f8865600640 1 -- 172.21.15.114:0/159539080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 msgr2=0x7f88600acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.090+0000 7f8865600640 1 --2- 172.21.15.114:0/159539080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f88600acf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.090+0000 7f8865600640 1 -- 172.21.15.114:0/159539080 >> v1:172.21.15.177:6789/0 conn(0x7f88600ad4a0 legacy=0x7f88600ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.090+0000 7f8865600640 1 -- 172.21.15.114:0/159539080 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88600cec50 con 0x7f88600b3530 2024-03-06T07:27:50.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.091+0000 7f8865600640 1 --2- 172.21.15.114:0/159539080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600b3530 0x7f886015b780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88540097d0 tx=0x7f885402b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e2af53413e5d3cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.092+0000 7f885fe00640 1 -- 172.21.15.114:0/159539080 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854035020 con 0x7f88600b3530 2024-03-06T07:27:50.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.092+0000 7f885fe00640 1 -- 172.21.15.114:0/159539080 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8854031800 con 0x7f88600b3530 2024-03-06T07:27:50.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.092+0000 7f885fe00640 1 -- 172.21.15.114:0/159539080 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854031bd0 con 0x7f88600b3530 2024-03-06T07:27:50.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.092+0000 7f8868000640 1 -- 172.21.15.114:0/159539080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600b3530 msgr2=0x7f886015b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.092+0000 7f8868000640 1 --2- 172.21.15.114:0/159539080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600b3530 0x7f886015b780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88540097d0 tx=0x7f885402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.092+0000 7f8868000640 1 -- 172.21.15.114:0/159539080 shutdown_connections 2024-03-06T07:27:50.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.093+0000 7f8868000640 1 --2- 172.21.15.114:0/159539080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f88600acf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.093+0000 7f8868000640 1 --2- 172.21.15.114:0/159539080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600b3530 0x7f886015b780 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.093+0000 7f8868000640 1 -- 172.21.15.114:0/159539080 >> 172.21.15.114:0/159539080 conn(0x7f88600b1940 msgr2=0x7f88600b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.093+0000 7f8868000640 1 -- 172.21.15.114:0/159539080 shutdown_connections 2024-03-06T07:27:50.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.093+0000 7f8868000640 1 -- 172.21.15.114:0/159539080 wait complete. 2024-03-06T07:27:50.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.094+0000 7f8868000640 1 Processor -- start 2024-03-06T07:27:50.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.094+0000 7f8868000640 1 -- start start 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8868000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600ad4a0 0x7f8860151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8868000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88600b3530 0x7f886009b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8868000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f886009d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8868000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88600b3d90 con 0x7f88600b3530 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8868000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88600cf210 con 0x7f88600ad4a0 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8868000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886015aa30 con 0x7f88600b3910 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8864c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88600b3530 0x7f886009b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8864c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88600b3530 0x7f886009b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46438/0 (socket says 172.21.15.114:46438) 2024-03-06T07:27:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.095+0000 7f8864c00640 1 -- 172.21.15.114:0/312783760 learned_addr learned my addr 172.21.15.114:0/312783760 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:50.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8864c00640 1 -- 172.21.15.114:0/312783760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 msgr2=0x7f886009d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8864c00640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f886009d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8864c00640 1 -- 172.21.15.114:0/312783760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600ad4a0 msgr2=0x7f8860151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8865600640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600ad4a0 0x7f8860151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8864c00640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600ad4a0 0x7f8860151930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8864c00640 1 -- 172.21.15.114:0/312783760 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886009d020 con 0x7f88600b3530 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8864c00640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88600b3530 0x7f886009b8a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e8e0 tx=0x7f884c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f885d600640 1 -- 172.21.15.114:0/312783760 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c016070 con 0x7f88600b3530 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f885d600640 1 -- 172.21.15.114:0/312783760 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c00cc40 con 0x7f88600b3530 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f885d600640 1 -- 172.21.15.114:0/312783760 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c00b040 con 0x7f88600b3530 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.096+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886009bf80 con 0x7f88600b3530 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.097+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886009c540 con 0x7f88600b3530 2024-03-06T07:27:50.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.098+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886004f330 con 0x7f88600b3530 2024-03-06T07:27:50.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.099+0000 7f885d600640 1 -- 172.21.15.114:0/312783760 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f884c019070 con 0x7f88600b3530 2024-03-06T07:27:50.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.100+0000 7f885d600640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f884006b500 0x7f884006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.100+0000 7f885d600640 1 -- 172.21.15.114:0/312783760 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f884c089a80 con 0x7f88600b3530 2024-03-06T07:27:50.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.100+0000 7f8865600640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f884006b500 0x7f884006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.101+0000 7f8865600640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f884006b500 0x7f884006d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8854036000 tx=0x7f8854008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.105+0000 7f885d600640 1 -- 172.21.15.114:0/312783760 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f884c08be20 con 0x7f88600b3530 2024-03-06T07:27:50.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.286+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88600b1050 con 0x7f88600b3530 2024-03-06T07:27:50.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.287+0000 7f885d600640 1 -- 172.21.15.114:0/312783760 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f884c058c40 con 0x7f88600b3530 2024-03-06T07:27:50.288 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:27:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f884006b500 msgr2=0x7f884006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f884006b500 0x7f884006d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8854036000 tx=0x7f8854008030 comp rx=0 tx=0).stop 2024-03-06T07:27:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88600b3530 msgr2=0x7f886009b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88600b3530 0x7f886009b8a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e8e0 tx=0x7f884c00eda0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 shutdown_connections 2024-03-06T07:27:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88600b3910 0x7f886009d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f884006b500 0x7f884006d9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88600b3530 0x7f886009b8a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 --2- 172.21.15.114:0/312783760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88600ad4a0 0x7f8860151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 >> 172.21.15.114:0/312783760 conn(0x7f88600b1940 msgr2=0x7f88600b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 shutdown_connections 2024-03-06T07:27:50.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.289+0000 7f8868000640 1 -- 172.21.15.114:0/312783760 wait complete. 2024-03-06T07:27:50.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:50.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.511+0000 7f9d82400640 1 Processor -- start 2024-03-06T07:27:50.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.511+0000 7f9d82400640 1 -- start start 2024-03-06T07:27:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.513+0000 7f9d82400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9d7c15fe60 0x7f9d7c071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.513+0000 7f9d82400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7c06cef0 con 0x7f9d7c144c00 2024-03-06T07:27:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.513+0000 7f9d82400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7c06d1d0 con 0x7f9d7c15f9d0 2024-03-06T07:27:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.513+0000 7f9d82400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c06d4b0 con 0x7f9d7c15fe60 2024-03-06T07:27:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.513+0000 7f9d7b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9d7c144c00 0x7f9d7c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54504/0 (socket says 172.21.15.114:54504) 2024-03-06T07:27:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.513+0000 7f9d7b400640 1 -- 172.21.15.114:0/2486791588 learned_addr learned my addr 172.21.15.114:0/2486791588 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:50.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.513+0000 7f9d7be00640 1 --2- 172.21.15.114:0/2486791588 >> v2:172.21.15.114:3300/0 conn(0x7f9d7c15fe60 0x7f9d7c071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.514+0000 7f9d7be00640 1 -- 172.21.15.114:0/2486791588 >> v1:172.21.15.177:6789/0 conn(0x7f9d7c15f9d0 legacy=0x7f9d7c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.514+0000 7f9d7be00640 1 -- 172.21.15.114:0/2486791588 >> v1:172.21.15.114:6789/0 conn(0x7f9d7c144c00 legacy=0x7f9d7c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.514+0000 7f9d7be00640 1 -- 172.21.15.114:0/2486791588 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c06d790 con 0x7f9d7c15fe60 2024-03-06T07:27:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.515+0000 7f9d7be00640 1 --2- 172.21.15.114:0/2486791588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c15fe60 0x7f9d7c071220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009d90 tx=0x7f9d7402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe109951ef50a8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.515+0000 7f9d7aa00640 1 -- 172.21.15.114:0/2486791588 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74039070 con 0x7f9d7c15fe60 2024-03-06T07:27:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.515+0000 7f9d7aa00640 1 -- 172.21.15.114:0/2486791588 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7402ccf0 con 0x7f9d7c15fe60 2024-03-06T07:27:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.515+0000 7f9d7aa00640 1 -- 172.21.15.114:0/2486791588 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74034660 con 0x7f9d7c15fe60 2024-03-06T07:27:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.516+0000 7f9d82400640 1 -- 172.21.15.114:0/2486791588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c15fe60 msgr2=0x7f9d7c071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.516+0000 7f9d82400640 1 --2- 172.21.15.114:0/2486791588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c15fe60 0x7f9d7c071220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009d90 tx=0x7f9d7402bf40 comp rx=0 tx=0).stop 2024-03-06T07:27:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.516+0000 7f9d82400640 1 -- 172.21.15.114:0/2486791588 shutdown_connections 2024-03-06T07:27:50.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.516+0000 7f9d82400640 1 --2- 172.21.15.114:0/2486791588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c15fe60 0x7f9d7c071220 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.516+0000 7f9d82400640 1 -- 172.21.15.114:0/2486791588 >> 172.21.15.114:0/2486791588 conn(0x7f9d7c0b1730 msgr2=0x7f9d7c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.516+0000 7f9d82400640 1 -- 172.21.15.114:0/2486791588 shutdown_connections 2024-03-06T07:27:50.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.516+0000 7f9d82400640 1 -- 172.21.15.114:0/2486791588 wait complete. 2024-03-06T07:27:50.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.517+0000 7f9d82400640 1 Processor -- start 2024-03-06T07:27:50.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.518+0000 7f9d82400640 1 -- start start 2024-03-06T07:27:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d82400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c144c00 0x7f9d7c15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d82400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d7c15f9d0 0x7f9d7c1422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d82400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d7c15fe60 0x7f9d7c0cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d82400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c06cb50 con 0x7f9d7c144c00 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d82400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c160730 con 0x7f9d7c15f9d0 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d82400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c06ccd0 con 0x7f9d7c15fe60 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c144c00 0x7f9d7c15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c144c00 0x7f9d7c15b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46456/0 (socket says 172.21.15.114:46456) 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 -- 172.21.15.114:0/3899448835 learned_addr learned my addr 172.21.15.114:0/3899448835 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d72a00640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d7c15f9d0 0x7f9d7c1422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 -- 172.21.15.114:0/3899448835 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d7c15fe60 msgr2=0x7f9d7c0cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d7c15fe60 0x7f9d7c0cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 -- 172.21.15.114:0/3899448835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d7c15f9d0 msgr2=0x7f9d7c1422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d7c15f9d0 0x7f9d7c1422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.519+0000 7f9d7b400640 1 -- 172.21.15.114:0/3899448835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c15b910 con 0x7f9d7c144c00 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.520+0000 7f9d7b400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c144c00 0x7f9d7c15b3d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c4b0 tx=0x7f9d6c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.520+0000 7f9d73e00640 1 -- 172.21.15.114:0/3899448835 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c017070 con 0x7f9d7c144c00 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.520+0000 7f9d73e00640 1 -- 172.21.15.114:0/3899448835 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c005ce0 con 0x7f9d7c144c00 2024-03-06T07:27:50.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.520+0000 7f9d73e00640 1 -- 172.21.15.114:0/3899448835 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c007270 con 0x7f9d7c144c00 2024-03-06T07:27:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.520+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c13d2d0 con 0x7f9d7c144c00 2024-03-06T07:27:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.520+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c13d590 con 0x7f9d7c144c00 2024-03-06T07:27:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.521+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c04f330 con 0x7f9d7c144c00 2024-03-06T07:27:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.522+0000 7f9d73e00640 1 -- 172.21.15.114:0/3899448835 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d6c01a070 con 0x7f9d7c144c00 2024-03-06T07:27:50.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.523+0000 7f9d73e00640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d5806b500 0x7f9d5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.523+0000 7f9d73e00640 1 -- 172.21.15.114:0/3899448835 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9d6c089080 con 0x7f9d7c144c00 2024-03-06T07:27:50.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.523+0000 7f9d72a00640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d5806b500 0x7f9d5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.524+0000 7f9d72a00640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d5806b500 0x7f9d5806d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d64007fd0 tx=0x7f9d64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.528+0000 7f9d73e00640 1 -- 172.21.15.114:0/3899448835 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d6c057bf0 con 0x7f9d7c144c00 2024-03-06T07:27:50.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.761+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d7c0ce200 con 0x7f9d7c144c00 2024-03-06T07:27:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.762+0000 7f9d73e00640 1 -- 172.21.15.114:0/3899448835 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3706 (secure 0 0 0) 0x7f9d6c05bc90 con 0x7f9d7c144c00 2024-03-06T07:27:50.763 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:50.763 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:27:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 3 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d5806b500 msgr2=0x7f9d5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d5806b500 0x7f9d5806d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d64007fd0 tx=0x7f9d64009040 comp rx=0 tx=0).stop 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c144c00 msgr2=0x7f9d7c15b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c144c00 0x7f9d7c15b3d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c4b0 tx=0x7f9d6c00c970 comp rx=0 tx=0).stop 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 shutdown_connections 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d7c15fe60 0x7f9d7c0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d7c15f9d0 0x7f9d7c1422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d5806b500 0x7f9d5806d9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 --2- 172.21.15.114:0/3899448835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d7c144c00 0x7f9d7c15b3d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 >> 172.21.15.114:0/3899448835 conn(0x7f9d7c0b1730 msgr2=0x7f9d7c0717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 shutdown_connections 2024-03-06T07:27:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.764+0000 7f9d82400640 1 -- 172.21.15.114:0/3899448835 wait complete. 2024-03-06T07:27:50.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:50.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.981+0000 7fc378c00640 1 Processor -- start 2024-03-06T07:27:50.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.982+0000 7fc378c00640 1 -- start start 2024-03-06T07:27:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.983+0000 7fc378c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 0x7fc37406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.983+0000 7fc378c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc37415f9d0 0x7fc37415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.983+0000 7fc378c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37406ee70 con 0x7fc3740ce0a0 2024-03-06T07:27:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.983+0000 7fc378c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc374160370 con 0x7fc37415f9d0 2024-03-06T07:27:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.983+0000 7fc378c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406cef0 con 0x7fc374144c00 2024-03-06T07:27:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc372800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc3740ce0a0 0x7fc3740ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54514/0 (socket says 172.21.15.114:54514) 2024-03-06T07:27:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc372800640 1 -- 172.21.15.114:0/3819625877 learned_addr learned my addr 172.21.15.114:0/3819625877 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc371e00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 0x7fc37406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc371400640 1 --2- 172.21.15.114:0/3819625877 >> v2:172.21.15.177:3300/0 conn(0x7fc37415f9d0 0x7fc37415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239710553 0 0) 0x7fc36c001110 con 0x7fc3740ce0a0 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc354003520 con 0x7fc3740ce0a0 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090471395 0 0) 0x7fc354003520 con 0x7fc3740ce0a0 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 msgr2=0x7fc37406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 0x7fc37406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 >> v2:172.21.15.177:3300/0 conn(0x7fc37415f9d0 msgr2=0x7fc37415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 --2- 172.21.15.114:0/3819625877 >> v2:172.21.15.177:3300/0 conn(0x7fc37415f9d0 0x7fc37415fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37406d1d0 con 0x7fc3740ce0a0 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fc36c001840 con 0x7fc3740ce0a0 2024-03-06T07:27:50.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.984+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 >> v1:172.21.15.114:6789/0 conn(0x7fc3740ce0a0 legacy=0x7fc3740ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc370a00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3540039b0 0x7fc354003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc370a00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc354004800 0x7fc354004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc370a00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 0x7fc3540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406cef0 con 0x7fc3540039b0 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400b9c0 con 0x7fc354004800 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400bb40 con 0x7fc374144c00 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371e00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3540039b0 0x7fc354003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371400640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc354004800 0x7fc354004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc372800640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 0x7fc3540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371e00640 1 -- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 msgr2=0x7fc3540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371e00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 0x7fc3540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371e00640 1 -- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc354004800 msgr2=0x7fc354004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371e00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc354004800 0x7fc354004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371e00640 1 -- 172.21.15.114:0/3819625877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354003520 con 0x7fc3540039b0 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc371400640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc354004800 0x7fc354004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:50.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.985+0000 7fc372800640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc374144c00 0x7fc3540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:50.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.986+0000 7fc371e00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3540039b0 0x7fc354003dd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc364009ac0 tx=0x7fc354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38740f53b597edf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364039070 con 0x7fc3540039b0 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36402c6b0 con 0x7fc3540039b0 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc370a00640 1 -- 172.21.15.114:0/3819625877 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364009690 con 0x7fc3540039b0 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc378c00640 1 -- 172.21.15.114:0/3819625877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3540039b0 msgr2=0x7fc354003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc378c00640 1 --2- 172.21.15.114:0/3819625877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3540039b0 0x7fc354003dd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc364009ac0 tx=0x7fc354005830 comp rx=0 tx=0).stop 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc372800640 1 -- 172.21.15.114:0/3819625877 reap_dead start 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc378c00640 1 -- 172.21.15.114:0/3819625877 shutdown_connections 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc378c00640 1 -- 172.21.15.114:0/3819625877 >> 172.21.15.114:0/3819625877 conn(0x7fc3740b1730 msgr2=0x7fc3740b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc378c00640 1 -- 172.21.15.114:0/3819625877 shutdown_connections 2024-03-06T07:27:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.987+0000 7fc378c00640 1 -- 172.21.15.114:0/3819625877 wait complete. 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.989+0000 7fc378c00640 1 Processor -- start 2024-03-06T07:27:50.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.989+0000 7fc378c00640 1 -- start start 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc378c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc3540039b0 0x7fc3740b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc378c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3740b4060 0x7fc374137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc378c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc37413a930 0x7fc374137b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc378c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374160730 con 0x7fc3740b4060 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc378c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406d790 con 0x7fc37413a930 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc378c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406cb50 con 0x7fc3540039b0 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3740b4060 0x7fc374137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3740b4060 0x7fc374137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46476/0 (socket says 172.21.15.114:46476) 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 -- 172.21.15.114:0/1882477404 learned_addr learned my addr 172.21.15.114:0/1882477404 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371e00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc3540039b0 0x7fc3740b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 -- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc3540039b0 msgr2=0x7fc3740b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc3540039b0 0x7fc3740b29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 -- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc37413a930 msgr2=0x7fc374137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc37413a930 0x7fc374137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371400640 1 -- 172.21.15.114:0/1882477404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740b2fc0 con 0x7fc3740b4060 2024-03-06T07:27:50.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.990+0000 7fc371e00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc3540039b0 0x7fc3740b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.991+0000 7fc371400640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3740b4060 0x7fc374137000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007560 tx=0x7fc35c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.991+0000 7fc36a000640 1 -- 172.21.15.114:0/1882477404 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c01e020 con 0x7fc3740b4060 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.991+0000 7fc36a000640 1 -- 172.21.15.114:0/1882477404 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c003990 con 0x7fc3740b4060 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.991+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3740b32a0 con 0x7fc3740b4060 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.991+0000 7fc36a000640 1 -- 172.21.15.114:0/1882477404 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c003d40 con 0x7fc3740b4060 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.991+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3740b3860 con 0x7fc3740b4060 2024-03-06T07:27:50.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.992+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37404f330 con 0x7fc3740b4060 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.993+0000 7fc36a000640 1 -- 172.21.15.114:0/1882477404 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc35c021070 con 0x7fc3740b4060 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.994+0000 7fc36a000640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc35006b4d0 0x7fc35006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.994+0000 7fc36a000640 1 -- 172.21.15.114:0/1882477404 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc35c08e170 con 0x7fc3740b4060 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.994+0000 7fc371e00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc35006b4d0 0x7fc35006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:50.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.995+0000 7fc371e00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc35006b4d0 0x7fc35006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc364036000 tx=0x7fc364030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:50.998+0000 7fc36a000640 1 -- 172.21.15.114:0/1882477404 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc35c05d330 con 0x7fc3740b4060 2024-03-06T07:27:51.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.194+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc374072890 con 0x7fc3740b4060 2024-03-06T07:27:51.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.196+0000 7fc36a000640 1 -- 172.21.15.114:0/1882477404 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12932 (secure 0 0 0) 0x7fc35c05cc30 con 0x7fc3740b4060 2024-03-06T07:27:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:51.198 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":16,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:45.393016+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:51.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.198+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc35006b4d0 msgr2=0x7fc35006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.198+0000 7fc378c00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc35006b4d0 0x7fc35006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc364036000 tx=0x7fc364030000 comp rx=0 tx=0).stop 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3740b4060 msgr2=0x7fc374137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3740b4060 0x7fc374137000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc35c007560 tx=0x7fc35c007a20 comp rx=0 tx=0).stop 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 shutdown_connections 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc37413a930 0x7fc374137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc35006b4d0 0x7fc35006d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3740b4060 0x7fc374137000 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 --2- 172.21.15.114:0/1882477404 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc3540039b0 0x7fc3740b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 >> 172.21.15.114:0/1882477404 conn(0x7fc3740b1730 msgr2=0x7fc3740ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 shutdown_connections 2024-03-06T07:27:51.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.199+0000 7fc378c00640 1 -- 172.21.15.114:0/1882477404 wait complete. 2024-03-06T07:27:51.218 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:51.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.417+0000 7fd9c2000640 1 Processor -- start 2024-03-06T07:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.418+0000 7fd9c2000640 1 -- start start 2024-03-06T07:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c2000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd9bc0d1ea0 0x7fd9bc0d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc0c7970 0x7fd9bc0c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c2000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9bc0d13e0 con 0x7fd9bc15e130 2024-03-06T07:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c2000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc0d2840 con 0x7fd9bc0d1ea0 2024-03-06T07:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c2000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cf120 con 0x7fd9bc0c7970 2024-03-06T07:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c1600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc0c7970 0x7fd9bc0c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9bbe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd9bc0d1ea0 0x7fd9bc0d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c0c00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fd9bc15e130 0x7fd9bc0cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:53206/0 (socket says 172.21.15.114:53206) 2024-03-06T07:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.419+0000 7fd9c0c00640 1 -- 172.21.15.114:0/647873993 learned_addr learned my addr 172.21.15.114:0/647873993 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.420+0000 7fd9bbe00640 1 -- 172.21.15.114:0/647873993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc0c7970 msgr2=0x7fd9bc0c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.420+0000 7fd9bbe00640 1 --2- 172.21.15.114:0/647873993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc0c7970 0x7fd9bc0c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.420+0000 7fd9bbe00640 1 -- 172.21.15.114:0/647873993 >> v1:172.21.15.177:6789/0 conn(0x7fd9bc15e130 legacy=0x7fd9bc0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.420+0000 7fd9bbe00640 1 -- 172.21.15.114:0/647873993 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc0cead0 con 0x7fd9bc0d1ea0 2024-03-06T07:27:51.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.421+0000 7fd9bbe00640 1 --2- 172.21.15.114:0/647873993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0d2300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40097a0 tx=0x7fd9a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=534642a006687e44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.422+0000 7fd9bb400640 1 -- 172.21.15.114:0/647873993 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4035020 con 0x7fd9bc0d1ea0 2024-03-06T07:27:51.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.422+0000 7fd9bb400640 1 -- 172.21.15.114:0/647873993 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a4031830 con 0x7fd9bc0d1ea0 2024-03-06T07:27:51.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.422+0000 7fd9bb400640 1 -- 172.21.15.114:0/647873993 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4031c00 con 0x7fd9bc0d1ea0 2024-03-06T07:27:51.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.422+0000 7fd9c2000640 1 -- 172.21.15.114:0/647873993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 msgr2=0x7fd9bc0d2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.422+0000 7fd9c2000640 1 --2- 172.21.15.114:0/647873993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0d2300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40097a0 tx=0x7fd9a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.423+0000 7fd9c2000640 1 -- 172.21.15.114:0/647873993 shutdown_connections 2024-03-06T07:27:51.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.423+0000 7fd9c2000640 1 --2- 172.21.15.114:0/647873993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc0c7970 0x7fd9bc0c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.423+0000 7fd9c2000640 1 --2- 172.21.15.114:0/647873993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0d2300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.423+0000 7fd9c2000640 1 -- 172.21.15.114:0/647873993 >> 172.21.15.114:0/647873993 conn(0x7fd9bc09d070 msgr2=0x7fd9bc0d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.423+0000 7fd9c2000640 1 -- 172.21.15.114:0/647873993 shutdown_connections 2024-03-06T07:27:51.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.423+0000 7fd9c2000640 1 -- 172.21.15.114:0/647873993 wait complete. 2024-03-06T07:27:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.424+0000 7fd9c2000640 1 Processor -- start 2024-03-06T07:27:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.424+0000 7fd9c2000640 1 -- start start 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9bc0c7970 0x7fd9bc15c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc15e130 0x7fd9bc0b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c2000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cfcc0 con 0x7fd9bc0c7970 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c2000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0c8340 con 0x7fd9bc0d1ea0 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c2000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0d16c0 con 0x7fd9bc15e130 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9bc0c7970 0x7fd9bc15c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9bc0c7970 0x7fd9bc15c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46486/0 (socket says 172.21.15.114:46486) 2024-03-06T07:27:51.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.425+0000 7fd9c0c00640 1 -- 172.21.15.114:0/1002751130 learned_addr learned my addr 172.21.15.114:0/1002751130 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c1600640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc15e130 0x7fd9bc0b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9bbe00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c0c00640 1 -- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc15e130 msgr2=0x7fd9bc0b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c0c00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc15e130 0x7fd9bc0b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c0c00640 1 -- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 msgr2=0x7fd9bc0b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c0c00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c0c00640 1 -- 172.21.15.114:0/1002751130 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc15caa0 con 0x7fd9bc0c7970 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9bbe00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c0c00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9bc0c7970 0x7fd9bc15c560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c400 tx=0x7fd9ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9b8c00640 1 -- 172.21.15.114:0/1002751130 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac017070 con 0x7fd9bc0c7970 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9b8c00640 1 -- 172.21.15.114:0/1002751130 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ac005ce0 con 0x7fd9bc0c7970 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc0b8300 con 0x7fd9bc0c7970 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9b8c00640 1 -- 172.21.15.114:0/1002751130 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac0072f0 con 0x7fd9bc0c7970 2024-03-06T07:27:51.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.426+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc0b6f70 con 0x7fd9bc0c7970 2024-03-06T07:27:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.427+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc0b6a80 con 0x7fd9bc0c7970 2024-03-06T07:27:51.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.429+0000 7fd9b8c00640 1 -- 172.21.15.114:0/1002751130 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9ac01a070 con 0x7fd9bc0c7970 2024-03-06T07:27:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.429+0000 7fd9b8c00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd99806b4d0 0x7fd99806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.429+0000 7fd9b8c00640 1 -- 172.21.15.114:0/1002751130 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd9ac089240 con 0x7fd9bc0c7970 2024-03-06T07:27:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.430+0000 7fd9bbe00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd99806b4d0 0x7fd99806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.430+0000 7fd9bbe00640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd99806b4d0 0x7fd99806d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4036000 tx=0x7fd9a4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.434+0000 7fd9b8c00640 1 -- 172.21.15.114:0/1002751130 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9ac0584a0 con 0x7fd9bc0c7970 2024-03-06T07:27:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.612+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9bc04f330 con 0x7fd9bc0c7970 2024-03-06T07:27:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.613+0000 7fd9b8c00640 1 -- 172.21.15.114:0/1002751130 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9ac057da0 con 0x7fd9bc0c7970 2024-03-06T07:27:51.614 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:27:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd99806b4d0 msgr2=0x7fd99806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd99806b4d0 0x7fd99806d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4036000 tx=0x7fd9a4008030 comp rx=0 tx=0).stop 2024-03-06T07:27:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9bc0c7970 msgr2=0x7fd9bc15c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9bc0c7970 0x7fd9bc15c560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c400 tx=0x7fd9ac00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 shutdown_connections 2024-03-06T07:27:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9bc15e130 0x7fd9bc0b94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9bc0d1ea0 0x7fd9bc0b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd99806b4d0 0x7fd99806d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 --2- 172.21.15.114:0/1002751130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9bc0c7970 0x7fd9bc15c560 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 >> 172.21.15.114:0/1002751130 conn(0x7fd9bc09d070 msgr2=0x7fd9bc162310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.615+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 shutdown_connections 2024-03-06T07:27:51.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.616+0000 7fd9c2000640 1 -- 172.21.15.114:0/1002751130 wait complete. 2024-03-06T07:27:51.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.833+0000 7f3b69000640 1 Processor -- start 2024-03-06T07:27:51.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.833+0000 7f3b69000640 1 -- start start 2024-03-06T07:27:51.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.834+0000 7f3b69000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3b64144c00 0x7f3b6406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b69000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3b6415fe60 0x7f3b640ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b69000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6406ee70 con 0x7f3b6415f9d0 2024-03-06T07:27:51.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b69000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6406cef0 con 0x7f3b64144c00 2024-03-06T07:27:51.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b69000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6406d1d0 con 0x7f3b6415fe60 2024-03-06T07:27:51.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b61e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3b64144c00 0x7f3b6406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b61e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3b64144c00 0x7f3b6406e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46488/0 (socket says 172.21.15.114:46488) 2024-03-06T07:27:51.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b61e00640 1 -- 172.21.15.114:0/3658199275 learned_addr learned my addr 172.21.15.114:0/3658199275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b61e00640 1 -- 172.21.15.114:0/3658199275 >> v2:172.21.15.177:3300/0 conn(0x7f3b6415fe60 msgr2=0x7f3b640ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b62800640 1 --2- 172.21.15.114:0/3658199275 >> v2:172.21.15.177:3300/0 conn(0x7f3b6415fe60 0x7f3b640ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b61e00640 1 --2- 172.21.15.114:0/3658199275 >> v2:172.21.15.177:3300/0 conn(0x7f3b6415fe60 0x7f3b640ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b61e00640 1 -- 172.21.15.114:0/3658199275 >> v1:172.21.15.177:6789/0 conn(0x7f3b6415f9d0 legacy=0x7f3b640ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.835+0000 7f3b61e00640 1 -- 172.21.15.114:0/3658199275 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6406d4b0 con 0x7f3b64144c00 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.836+0000 7f3b61e00640 1 --2- 172.21.15.114:0/3658199275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b64144c00 0x7f3b6406e930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b6406e500 tx=0x7f3b5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1a29bea0f0a605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b61400640 1 -- 172.21.15.114:0/3658199275 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54035020 con 0x7f3b64144c00 2024-03-06T07:27:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b61400640 1 -- 172.21.15.114:0/3658199275 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b54031870 con 0x7f3b64144c00 2024-03-06T07:27:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b61400640 1 -- 172.21.15.114:0/3658199275 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54031c40 con 0x7f3b64144c00 2024-03-06T07:27:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b69000640 1 -- 172.21.15.114:0/3658199275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b64144c00 msgr2=0x7f3b6406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b69000640 1 --2- 172.21.15.114:0/3658199275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b64144c00 0x7f3b6406e930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b6406e500 tx=0x7f3b5402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b69000640 1 -- 172.21.15.114:0/3658199275 shutdown_connections 2024-03-06T07:27:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b69000640 1 --2- 172.21.15.114:0/3658199275 >> v2:172.21.15.177:3300/0 conn(0x7f3b6415fe60 0x7f3b640ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b69000640 1 --2- 172.21.15.114:0/3658199275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b64144c00 0x7f3b6406e930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.837+0000 7f3b69000640 1 -- 172.21.15.114:0/3658199275 >> 172.21.15.114:0/3658199275 conn(0x7f3b640b1730 msgr2=0x7f3b640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.838+0000 7f3b69000640 1 -- 172.21.15.114:0/3658199275 shutdown_connections 2024-03-06T07:27:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.838+0000 7f3b69000640 1 -- 172.21.15.114:0/3658199275 wait complete. 2024-03-06T07:27:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.840+0000 7f3b69000640 1 Processor -- start 2024-03-06T07:27:51.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.840+0000 7f3b69000640 1 -- start start 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b69000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b64144c00 0x7f3b64137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b69000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b6415f9d0 0x7f3b640b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b6415fe60 0x7f3b640b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b69000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64160730 con 0x7f3b6415f9d0 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b69000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64160270 con 0x7f3b6415fe60 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b69000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6406cb50 con 0x7f3b64144c00 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b5be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b6415f9d0 0x7f3b640b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b5be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b6415f9d0 0x7f3b640b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46494/0 (socket says 172.21.15.114:46494) 2024-03-06T07:27:51.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b5be00640 1 -- 172.21.15.114:0/3282113970 learned_addr learned my addr 172.21.15.114:0/3282113970 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.841+0000 7f3b61e00640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b64144c00 0x7f3b64137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b62800640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b6415fe60 0x7f3b640b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5be00640 1 -- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b64144c00 msgr2=0x7f3b64137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5be00640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b64144c00 0x7f3b64137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5be00640 1 -- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b6415fe60 msgr2=0x7f3b640b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5be00640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b6415fe60 0x7f3b640b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5be00640 1 -- 172.21.15.114:0/3282113970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6415f000 con 0x7f3b6415f9d0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5be00640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b6415f9d0 0x7f3b640b2740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00c430 tx=0x7f3b4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5a000640 1 -- 172.21.15.114:0/3282113970 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c011790 con 0x7f3b6415f9d0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5a000640 1 -- 172.21.15.114:0/3282113970 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c017070 con 0x7f3b6415f9d0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b5a000640 1 -- 172.21.15.114:0/3282113970 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c011bb0 con 0x7f3b6415f9d0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6415f2e0 con 0x7f3b6415f9d0 2024-03-06T07:27:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.842+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b64139860 con 0x7f3b6415f9d0 2024-03-06T07:27:51.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.844+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6404f330 con 0x7f3b6415f9d0 2024-03-06T07:27:51.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.844+0000 7f3b5a000640 1 -- 172.21.15.114:0/3282113970 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b4c01c460 con 0x7f3b6415f9d0 2024-03-06T07:27:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.845+0000 7f3b5a000640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b4006b4d0 0x7f3b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.846+0000 7f3b5a000640 1 -- 172.21.15.114:0/3282113970 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b4c089dc0 con 0x7f3b6415f9d0 2024-03-06T07:27:51.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.846+0000 7f3b61e00640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b4006b4d0 0x7f3b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.846+0000 7f3b61e00640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b4006b4d0 0x7f3b4006d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3b54036000 tx=0x7f3b5403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:51.850+0000 7f3b5a000640 1 -- 172.21.15.114:0/3282113970 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b4c058f00 con 0x7f3b6415f9d0 2024-03-06T07:27:52.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.051+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3b641458a0 con 0x7f3b6415f9d0 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.052+0000 7f3b5a000640 1 -- 172.21.15.114:0/3282113970 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3b4c058800 con 0x7f3b6415f9d0 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:27:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b4006b4d0 msgr2=0x7f3b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b4006b4d0 0x7f3b4006d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3b54036000 tx=0x7f3b5403c040 comp rx=0 tx=0).stop 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b6415f9d0 msgr2=0x7f3b640b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b6415f9d0 0x7f3b640b2740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00c430 tx=0x7f3b4c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 shutdown_connections 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b6415fe60 0x7f3b640b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b4006b4d0 0x7f3b4006d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b6415f9d0 0x7f3b640b2740 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 --2- 172.21.15.114:0/3282113970 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b64144c00 0x7f3b64137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 >> 172.21.15.114:0/3282113970 conn(0x7f3b640b1730 msgr2=0x7f3b640b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 shutdown_connections 2024-03-06T07:27:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.055+0000 7f3b69000640 1 -- 172.21.15.114:0/3282113970 wait complete. 2024-03-06T07:27:52.075 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:52.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.283+0000 7fa31cc00640 1 Processor -- start 2024-03-06T07:27:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.283+0000 7fa31cc00640 1 -- start start 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa31cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa318145980 0x7fa31815fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa31cc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa3180ced20 0x7fa31815ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa31cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3180712d0 0x7fa318071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa31cc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa318070e40 con 0x7fa318145980 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa31cc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3180cf1e0 con 0x7fa3180ced20 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa31cc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3180cf4c0 con 0x7fa3180712d0 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa318145980 0x7fa31815fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa318145980 0x7fa31815fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46500/0 (socket says 172.21.15.114:46500) 2024-03-06T07:27:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 -- 172.21.15.114:0/4043841459 learned_addr learned my addr 172.21.15.114:0/4043841459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 -- 172.21.15.114:0/4043841459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3180712d0 msgr2=0x7fa318071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 --2- 172.21.15.114:0/4043841459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3180712d0 0x7fa318071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 -- 172.21.15.114:0/4043841459 >> v2:172.21.15.177:3300/0 conn(0x7fa3180ced20 msgr2=0x7fa31815ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 --2- 172.21.15.114:0/4043841459 >> v2:172.21.15.177:3300/0 conn(0x7fa3180ced20 0x7fa31815ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.285+0000 7fa315e00640 1 -- 172.21.15.114:0/4043841459 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3180cf7a0 con 0x7fa318145980 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.286+0000 7fa315e00640 1 --2- 172.21.15.114:0/4043841459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa318145980 0x7fa31815fa00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa318070a60 tx=0x7fa30802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63475b223677740 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.287+0000 7fa315400640 1 -- 172.21.15.114:0/4043841459 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308035020 con 0x7fa318145980 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.287+0000 7fa315400640 1 -- 172.21.15.114:0/4043841459 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa308031870 con 0x7fa318145980 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.287+0000 7fa315400640 1 -- 172.21.15.114:0/4043841459 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308031c40 con 0x7fa318145980 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.287+0000 7fa31cc00640 1 -- 172.21.15.114:0/4043841459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa318145980 msgr2=0x7fa31815fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.287+0000 7fa31cc00640 1 --2- 172.21.15.114:0/4043841459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa318145980 0x7fa31815fa00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa318070a60 tx=0x7fa30802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.287+0000 7fa31cc00640 1 -- 172.21.15.114:0/4043841459 shutdown_connections 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.288+0000 7fa31cc00640 1 --2- 172.21.15.114:0/4043841459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3180712d0 0x7fa318071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.288+0000 7fa31cc00640 1 --2- 172.21.15.114:0/4043841459 >> v2:172.21.15.177:3300/0 conn(0x7fa3180ced20 0x7fa31815ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.288+0000 7fa31cc00640 1 --2- 172.21.15.114:0/4043841459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa318145980 0x7fa31815fa00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.288+0000 7fa31cc00640 1 -- 172.21.15.114:0/4043841459 >> 172.21.15.114:0/4043841459 conn(0x7fa3180cfa80 msgr2=0x7fa3180cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.288+0000 7fa31cc00640 1 -- 172.21.15.114:0/4043841459 shutdown_connections 2024-03-06T07:27:52.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.288+0000 7fa31cc00640 1 -- 172.21.15.114:0/4043841459 wait complete. 2024-03-06T07:27:52.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.289+0000 7fa31cc00640 1 Processor -- start 2024-03-06T07:27:52.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.290+0000 7fa31cc00640 1 -- start start 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa31cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3180712d0 0x7fa3181f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa31cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3180ced20 0x7fa3181425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa31cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa318145980 0x7fa318153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa31cc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3181608c0 con 0x7fa3180ced20 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa31cc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318160740 con 0x7fa3180712d0 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa31cc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa318071c40 con 0x7fa318145980 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3180ced20 0x7fa3181425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3180ced20 0x7fa3181425b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46514/0 (socket says 172.21.15.114:46514) 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 -- 172.21.15.114:0/2623515717 learned_addr learned my addr 172.21.15.114:0/2623515717 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 -- 172.21.15.114:0/2623515717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa318145980 msgr2=0x7fa318153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa318145980 0x7fa318153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 -- 172.21.15.114:0/2623515717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3180712d0 msgr2=0x7fa3181f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3180712d0 0x7fa3181f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.291+0000 7fa30d400640 1 -- 172.21.15.114:0/2623515717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318142e90 con 0x7fa3180ced20 2024-03-06T07:27:52.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.292+0000 7fa30d400640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3180ced20 0x7fa3181425b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa30000b4a0 tx=0x7fa30000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.292+0000 7fa30ea00640 1 -- 172.21.15.114:0/2623515717 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300014070 con 0x7fa3180ced20 2024-03-06T07:27:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.292+0000 7fa30ea00640 1 -- 172.21.15.114:0/2623515717 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300010610 con 0x7fa3180ced20 2024-03-06T07:27:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.292+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa318143170 con 0x7fa3180ced20 2024-03-06T07:27:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.292+0000 7fa30ea00640 1 -- 172.21.15.114:0/2623515717 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300010a00 con 0x7fa3180ced20 2024-03-06T07:27:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.292+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3180b3660 con 0x7fa3180ced20 2024-03-06T07:27:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.293+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31804f330 con 0x7fa3180ced20 2024-03-06T07:27:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.294+0000 7fa30ea00640 1 -- 172.21.15.114:0/2623515717 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa300017070 con 0x7fa3180ced20 2024-03-06T07:27:52.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.295+0000 7fa30ea00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.295+0000 7fa30ea00640 1 -- 172.21.15.114:0/2623515717 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa300089a20 con 0x7fa3180ced20 2024-03-06T07:27:52.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.295+0000 7fa315e00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.296+0000 7fa315e00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2f406b4d0 0x7fa2f406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa308036000 tx=0x7fa308030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.299+0000 7fa30ea00640 1 -- 172.21.15.114:0/2623515717 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa300058ac0 con 0x7fa3180ced20 2024-03-06T07:27:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.478+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa3180aff90 con 0x7fa3180ced20 2024-03-06T07:27:52.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.967+0000 7fa30ea00640 1 -- 172.21.15.114:0/2623515717 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa3000583c0 con 0x7fa3180ced20 2024-03-06T07:27:52.969 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.971+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2f406b4d0 msgr2=0x7fa2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.971+0000 7fa31cc00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2f406b4d0 0x7fa2f406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa308036000 tx=0x7fa308030000 comp rx=0 tx=0).stop 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.971+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3180ced20 msgr2=0x7fa3181425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.971+0000 7fa31cc00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3180ced20 0x7fa3181425b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa30000b4a0 tx=0x7fa30000b960 comp rx=0 tx=0).stop 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.973+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 shutdown_connections 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.973+0000 7fa31cc00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa318145980 0x7fa318153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.973+0000 7fa31cc00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.973+0000 7fa31cc00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3180ced20 0x7fa3181425b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.973+0000 7fa31cc00640 1 --2- 172.21.15.114:0/2623515717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3180712d0 0x7fa3181f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.973+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 >> 172.21.15.114:0/2623515717 conn(0x7fa3180cfa80 msgr2=0x7fa31815eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.974+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 shutdown_connections 2024-03-06T07:27:52.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:52.975+0000 7fa31cc00640 1 -- 172.21.15.114:0/2623515717 wait complete. 2024-03-06T07:27:53.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:53.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.193+0000 7f879e600640 1 Processor -- start 2024-03-06T07:27:53.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.193+0000 7f879e600640 1 -- start start 2024-03-06T07:27:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f879e600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f87980cd2b0 0x7f87980cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f879e600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8798066d00 0x7f879806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f879e600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980b0f40 con 0x7f8798067140 2024-03-06T07:27:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f879e600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980b5020 con 0x7f8798066d00 2024-03-06T07:27:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f879e600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980b5300 con 0x7f87980cd2b0 2024-03-06T07:27:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f878ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8798066d00 0x7f879806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f878ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8798066d00 0x7f879806afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46520/0 (socket says 172.21.15.114:46520) 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f878ea00640 1 -- 172.21.15.114:0/460824324 learned_addr learned my addr 172.21.15.114:0/460824324 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f8797e00640 1 --1- 172.21.15.114:0/460824324 >> v1:172.21.15.114:6789/0 conn(0x7f8798067140 0x7f87980b5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54530/0 (socket says 172.21.15.114:54530) 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.195+0000 7f8797400640 1 --2- 172.21.15.114:0/460824324 >> v2:172.21.15.177:3300/0 conn(0x7f87980cd2b0 0x7f87980cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.196+0000 7f878ea00640 1 -- 172.21.15.114:0/460824324 >> v2:172.21.15.177:3300/0 conn(0x7f87980cd2b0 msgr2=0x7f87980cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.196+0000 7f878ea00640 1 --2- 172.21.15.114:0/460824324 >> v2:172.21.15.177:3300/0 conn(0x7f87980cd2b0 0x7f87980cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.196+0000 7f878ea00640 1 -- 172.21.15.114:0/460824324 >> v1:172.21.15.114:6789/0 conn(0x7f8798067140 legacy=0x7f87980b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.196+0000 7f8796a00640 1 -- 172.21.15.114:0/460824324 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703402129 0 0) 0x7f87980b0f40 con 0x7f8798067140 2024-03-06T07:27:53.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.196+0000 7f878ea00640 1 -- 172.21.15.114:0/460824324 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980b1220 con 0x7f8798066d00 2024-03-06T07:27:53.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.197+0000 7f878ea00640 1 --2- 172.21.15.114:0/460824324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798066d00 0x7f879806afc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f87980d1420 tx=0x7f878002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7974c1705abe40d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.197+0000 7f8796a00640 1 -- 172.21.15.114:0/460824324 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878002f850 con 0x7f8798066d00 2024-03-06T07:27:53.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.197+0000 7f8796a00640 1 -- 172.21.15.114:0/460824324 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878002f9f0 con 0x7f8798066d00 2024-03-06T07:27:53.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.197+0000 7f8796a00640 1 -- 172.21.15.114:0/460824324 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878002fdc0 con 0x7f8798066d00 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 -- 172.21.15.114:0/460824324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798066d00 msgr2=0x7f879806afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 --2- 172.21.15.114:0/460824324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798066d00 0x7f879806afc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f87980d1420 tx=0x7f878002ba30 comp rx=0 tx=0).stop 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 -- 172.21.15.114:0/460824324 shutdown_connections 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 --2- 172.21.15.114:0/460824324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798066d00 0x7f879806afc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 --2- 172.21.15.114:0/460824324 >> v2:172.21.15.177:3300/0 conn(0x7f87980cd2b0 0x7f87980cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 -- 172.21.15.114:0/460824324 >> 172.21.15.114:0/460824324 conn(0x7f87980b1e80 msgr2=0x7f87980b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 -- 172.21.15.114:0/460824324 shutdown_connections 2024-03-06T07:27:53.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.198+0000 7f879e600640 1 -- 172.21.15.114:0/460824324 wait complete. 2024-03-06T07:27:53.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.200+0000 7f879e600640 1 Processor -- start 2024-03-06T07:27:53.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.200+0000 7f879e600640 1 -- start start 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f879e600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8798066d00 0x7f879814f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f879e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798067140 0x7f87981f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f879e600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87980cd2b0 0x7f87980c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f879e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87980cd690 con 0x7f8798067140 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f879e600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87980ce460 con 0x7f87980cd2b0 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f879e600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87980b2b30 con 0x7f8798066d00 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798067140 0x7f87981f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798067140 0x7f87981f7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46528/0 (socket says 172.21.15.114:46528) 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 -- 172.21.15.114:0/2791981598 learned_addr learned my addr 172.21.15.114:0/2791981598 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f8797400640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8798066d00 0x7f879814f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f8797e00640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87980cd2b0 0x7f87980c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 -- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8798066d00 msgr2=0x7f879814f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8798066d00 0x7f879814f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 -- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87980cd2b0 msgr2=0x7f87980c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87980cd2b0 0x7f87980c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f878ea00640 1 -- 172.21.15.114:0/2791981598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87981f8220 con 0x7f8798067140 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f8797e00640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87980cd2b0 0x7f87980c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.201+0000 7f8797400640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8798066d00 0x7f879814f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.202+0000 7f878ea00640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798067140 0x7f87981f7ce0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87800090b0 tx=0x7f8780004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.202+0000 7f878fe00640 1 -- 172.21.15.114:0/2791981598 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780047070 con 0x7f8798067140 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.202+0000 7f878fe00640 1 -- 172.21.15.114:0/2791981598 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8780037c40 con 0x7f8798067140 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.202+0000 7f878fe00640 1 -- 172.21.15.114:0/2791981598 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780004030 con 0x7f8798067140 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.202+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87981f40f0 con 0x7f8798067140 2024-03-06T07:27:53.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.202+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87981f46b0 con 0x7f8798067140 2024-03-06T07:27:53.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.203+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879804f330 con 0x7f8798067140 2024-03-06T07:27:53.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.204+0000 7f878fe00640 1 -- 172.21.15.114:0/2791981598 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f878004a070 con 0x7f8798067140 2024-03-06T07:27:53.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.205+0000 7f878fe00640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f877406b4d0 0x7f877406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.205+0000 7f878fe00640 1 -- 172.21.15.114:0/2791981598 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f87800ac1a0 con 0x7f8798067140 2024-03-06T07:27:53.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.205+0000 7f8797400640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f877406b4d0 0x7f877406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.206+0000 7f8797400640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f877406b4d0 0x7f877406d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8788006fd0 tx=0x7f8788008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.210+0000 7f878fe00640 1 -- 172.21.15.114:0/2791981598 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87981f8220 con 0x7f8798067140 2024-03-06T07:27:53.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.404+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f87980cc680 con 0x7f8798067140 2024-03-06T07:27:53.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.982+0000 7f878fe00640 1 -- 172.21.15.114:0/2791981598 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8780033030 con 0x7f8798067140 2024-03-06T07:27:54.036 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f877406b4d0 msgr2=0x7f877406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f877406b4d0 0x7f877406d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8788006fd0 tx=0x7f8788008040 comp rx=0 tx=0).stop 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798067140 msgr2=0x7f87981f7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798067140 0x7f87981f7ce0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87800090b0 tx=0x7f8780004730 comp rx=0 tx=0).stop 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 shutdown_connections 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87980cd2b0 0x7f87980c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f877406b4d0 0x7f877406d990 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8788006fd0 tx=0x7f8788008040 comp rx=0 tx=0).stop 2024-03-06T07:27:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8798067140 0x7f87981f7ce0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 --2- 172.21.15.114:0/2791981598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8798066d00 0x7f879814f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.986+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 >> 172.21.15.114:0/2791981598 conn(0x7f87980b1e80 msgr2=0x7f8798072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.987+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 shutdown_connections 2024-03-06T07:27:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:53.987+0000 7f879e600640 1 -- 172.21.15.114:0/2791981598 wait complete. 2024-03-06T07:27:54.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:54.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.219+0000 7f5001e00640 1 Processor -- start 2024-03-06T07:27:54.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.219+0000 7f5001e00640 1 -- start start 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.220+0000 7f5001e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4ffc15fc10 0x7f4ffc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.220+0000 7f5001e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4ffc15fff0 0x7f4ffc160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.220+0000 7f5001e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc0b46c0 con 0x7f4ffc0ce4a0 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.220+0000 7f5001e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0b49a0 con 0x7f4ffc15fff0 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.220+0000 7f5001e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc072c90 con 0x7f4ffc15fc10 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ff2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4ffc15fff0 0x7f4ffc160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ff2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4ffc15fff0 0x7f4ffc160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46532/0 (socket says 172.21.15.114:46532) 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ff2000640 1 -- 172.21.15.114:0/1523206398 learned_addr learned my addr 172.21.15.114:0/1523206398 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ffaa00640 1 --2- 172.21.15.114:0/1523206398 >> v2:172.21.15.177:3300/0 conn(0x7f4ffc15fc10 0x7f4ffc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ff2000640 1 -- 172.21.15.114:0/1523206398 >> v2:172.21.15.177:3300/0 conn(0x7f4ffc15fc10 msgr2=0x7f4ffc070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ff2000640 1 --2- 172.21.15.114:0/1523206398 >> v2:172.21.15.177:3300/0 conn(0x7f4ffc15fc10 0x7f4ffc070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ff2000640 1 -- 172.21.15.114:0/1523206398 >> v1:172.21.15.177:6789/0 conn(0x7f4ffc0ce4a0 legacy=0x7f4ffc0ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:54.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.221+0000 7f4ff2000640 1 -- 172.21.15.114:0/1523206398 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc160990 con 0x7f4ffc15fff0 2024-03-06T07:27:54.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.222+0000 7f4ff2000640 1 --2- 172.21.15.114:0/1523206398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc160450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0cf770 tx=0x7f4fe402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949d05ca4f5cb005 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f4ffa000640 1 -- 172.21.15.114:0/1523206398 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4035020 con 0x7f4ffc15fff0 2024-03-06T07:27:54.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f4ffa000640 1 -- 172.21.15.114:0/1523206398 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4031870 con 0x7f4ffc15fff0 2024-03-06T07:27:54.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f4ffa000640 1 -- 172.21.15.114:0/1523206398 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4031c40 con 0x7f4ffc15fff0 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f5001e00640 1 -- 172.21.15.114:0/1523206398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 msgr2=0x7f4ffc160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f5001e00640 1 --2- 172.21.15.114:0/1523206398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc160450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0cf770 tx=0x7f4fe402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f5001e00640 1 -- 172.21.15.114:0/1523206398 shutdown_connections 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f5001e00640 1 --2- 172.21.15.114:0/1523206398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc160450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f5001e00640 1 --2- 172.21.15.114:0/1523206398 >> v2:172.21.15.177:3300/0 conn(0x7f4ffc15fc10 0x7f4ffc070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.223+0000 7f5001e00640 1 -- 172.21.15.114:0/1523206398 >> 172.21.15.114:0/1523206398 conn(0x7f4ffc0b1940 msgr2=0x7f4ffc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.224+0000 7f5001e00640 1 -- 172.21.15.114:0/1523206398 shutdown_connections 2024-03-06T07:27:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.224+0000 7f5001e00640 1 -- 172.21.15.114:0/1523206398 wait complete. 2024-03-06T07:27:54.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.225+0000 7f5001e00640 1 Processor -- start 2024-03-06T07:27:54.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.225+0000 7f5001e00640 1 -- start start 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.226+0000 7f5001e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ffc0ce4a0 0x7f4ffc06e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.226+0000 7f5001e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ffc15fc10 0x7f4ffc09bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.226+0000 7f5001e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc09c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.226+0000 7f5001e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc072a30 con 0x7f4ffc15fff0 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.226+0000 7f5001e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0728b0 con 0x7f4ffc15fc10 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.226+0000 7f5001e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc15c2e0 con 0x7f4ffc0ce4a0 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc09c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc09c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46548/0 (socket says 172.21.15.114:46548) 2024-03-06T07:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 -- 172.21.15.114:0/3332575736 learned_addr learned my addr 172.21.15.114:0/3332575736 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 -- 172.21.15.114:0/3332575736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ffc0ce4a0 msgr2=0x7f4ffc06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ffc0ce4a0 0x7f4ffc06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 -- 172.21.15.114:0/3332575736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ffc15fc10 msgr2=0x7f4ffc09bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ff2000640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ffc15fc10 0x7f4ffc09bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ffc15fc10 0x7f4ffc09bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 -- 172.21.15.114:0/3332575736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc154160 con 0x7f4ffc15fff0 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.227+0000 7f4ffb400640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc09c000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400c7b0 tx=0x7f4ff400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.228+0000 7f4ff3e00640 1 -- 172.21.15.114:0/3332575736 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4004b60 con 0x7f4ffc15fff0 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.228+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc06d2f0 con 0x7f4ffc15fff0 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.228+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc06d8b0 con 0x7f4ffc15fff0 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.228+0000 7f4ff3e00640 1 -- 172.21.15.114:0/3332575736 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff4004d00 con 0x7f4ffc15fff0 2024-03-06T07:27:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.228+0000 7f4ff3e00640 1 -- 172.21.15.114:0/3332575736 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400aa50 con 0x7f4ffc15fff0 2024-03-06T07:27:54.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.229+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc04f330 con 0x7f4ffc15fff0 2024-03-06T07:27:54.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.230+0000 7f4ff3e00640 1 -- 172.21.15.114:0/3332575736 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ff400abf0 con 0x7f4ffc15fff0 2024-03-06T07:27:54.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.231+0000 7f4ff3e00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fd806b4d0 0x7f4fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:54.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.231+0000 7f4ff3e00640 1 -- 172.21.15.114:0/3332575736 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4ff408a4c0 con 0x7f4ffc15fff0 2024-03-06T07:27:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.231+0000 7f4ffaa00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fd806b4d0 0x7f4fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.232+0000 7f4ffaa00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fd806b4d0 0x7f4fd806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0098f0 tx=0x7f4fec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:54.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.236+0000 7f4ff3e00640 1 -- 172.21.15.114:0/3332575736 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ff4020070 con 0x7f4ffc15fff0 2024-03-06T07:27:54.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:54.462+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4ffc06ef30 con 0x7f4ffc15fff0 2024-03-06T07:27:55.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.004+0000 7f4ff3e00640 1 -- 172.21.15.114:0/3332575736 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4ff4059310 con 0x7f4ffc15fff0 2024-03-06T07:27:55.007 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fd806b4d0 msgr2=0x7f4fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fd806b4d0 0x7f4fd806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0098f0 tx=0x7f4fec008040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 msgr2=0x7f4ffc09c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc09c000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400c7b0 tx=0x7f4ff400cc70 comp rx=0 tx=0).stop 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 shutdown_connections 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fd806b4d0 0x7f4fd806d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ffc15fff0 0x7f4ffc09c000 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ffc15fc10 0x7f4ffc09bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 --2- 172.21.15.114:0/3332575736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ffc0ce4a0 0x7f4ffc06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.010+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 >> 172.21.15.114:0/3332575736 conn(0x7f4ffc0b1940 msgr2=0x7f4ffc0b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.011+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 shutdown_connections 2024-03-06T07:27:55.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.012+0000 7f5001e00640 1 -- 172.21.15.114:0/3332575736 wait complete. 2024-03-06T07:27:55.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:55.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.221+0000 7fa397800640 1 Processor -- start 2024-03-06T07:27:55.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.221+0000 7fa397800640 1 -- start start 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa397800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3900cf430 0x7fa390070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa397800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa39006d260 0x7fa3900b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa397800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3900717c0 con 0x7fa390071260 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa397800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa390071aa0 con 0x7fa39006d260 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa397800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa390071d80 con 0x7fa3900cf430 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa387e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa390071260 0x7fa39006cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54546/0 (socket says 172.21.15.114:54546) 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa387e00640 1 -- 172.21.15.114:0/3325780662 learned_addr learned my addr 172.21.15.114:0/3325780662 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:55.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.223+0000 7fa395800640 1 --2- 172.21.15.114:0/3325780662 >> v2:172.21.15.114:3300/0 conn(0x7fa39006d260 0x7fa3900b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.224+0000 7fa394e00640 1 --2- 172.21.15.114:0/3325780662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3900cf430 0x7fa390070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.224+0000 7fa38fe00640 1 -- 172.21.15.114:0/3325780662 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210234098 0 0) 0x7fa3900717c0 con 0x7fa390071260 2024-03-06T07:27:55.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.224+0000 7fa38fe00640 1 -- 172.21.15.114:0/3325780662 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa370003520 con 0x7fa390071260 2024-03-06T07:27:55.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.224+0000 7fa395800640 1 -- 172.21.15.114:0/3325780662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3900cf430 msgr2=0x7fa390070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.224+0000 7fa395800640 1 --2- 172.21.15.114:0/3325780662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3900cf430 0x7fa390070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.224+0000 7fa395800640 1 -- 172.21.15.114:0/3325780662 >> v1:172.21.15.114:6789/0 conn(0x7fa390071260 legacy=0x7fa39006cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.224+0000 7fa395800640 1 -- 172.21.15.114:0/3325780662 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa390072060 con 0x7fa39006d260 2024-03-06T07:27:55.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.225+0000 7fa395800640 1 --2- 172.21.15.114:0/3325780662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa39006d260 0x7fa3900b2d10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa388009d90 tx=0x7fa38802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32547731c59e950 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.225+0000 7fa38fe00640 1 -- 172.21.15.114:0/3325780662 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388039070 con 0x7fa39006d260 2024-03-06T07:27:55.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.225+0000 7fa38fe00640 1 -- 172.21.15.114:0/3325780662 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38802cc50 con 0x7fa39006d260 2024-03-06T07:27:55.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.225+0000 7fa38fe00640 1 -- 172.21.15.114:0/3325780662 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388034440 con 0x7fa39006d260 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 -- 172.21.15.114:0/3325780662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa39006d260 msgr2=0x7fa3900b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 --2- 172.21.15.114:0/3325780662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa39006d260 0x7fa3900b2d10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa388009d90 tx=0x7fa38802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 -- 172.21.15.114:0/3325780662 shutdown_connections 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 --2- 172.21.15.114:0/3325780662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa39006d260 0x7fa3900b2d10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 --2- 172.21.15.114:0/3325780662 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3900cf430 0x7fa390070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 -- 172.21.15.114:0/3325780662 >> 172.21.15.114:0/3325780662 conn(0x7fa3900b1730 msgr2=0x7fa3900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 -- 172.21.15.114:0/3325780662 shutdown_connections 2024-03-06T07:27:55.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.226+0000 7fa397800640 1 -- 172.21.15.114:0/3325780662 wait complete. 2024-03-06T07:27:55.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.228+0000 7fa397800640 1 Processor -- start 2024-03-06T07:27:55.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.228+0000 7fa397800640 1 -- start start 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa397800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa39006d260 0x7fa3901662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa397800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa390071260 0x7fa390144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa397800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3900cf430 0x7fa390156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa397800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa390162cf0 con 0x7fa3900cf430 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa397800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa390162b70 con 0x7fa39006d260 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa397800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3900b3250 con 0x7fa390071260 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa395800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3900cf430 0x7fa390156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa395800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3900cf430 0x7fa390156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46554/0 (socket says 172.21.15.114:46554) 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa395800640 1 -- 172.21.15.114:0/3469642782 learned_addr learned my addr 172.21.15.114:0/3469642782 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.229+0000 7fa394e00640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa39006d260 0x7fa3901662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa387e00640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa390071260 0x7fa390144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa395800640 1 -- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa390071260 msgr2=0x7fa390144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa395800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa390071260 0x7fa390144c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa395800640 1 -- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa39006d260 msgr2=0x7fa3901662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa395800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa39006d260 0x7fa3901662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa395800640 1 -- 172.21.15.114:0/3469642782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39014fa90 con 0x7fa3900cf430 2024-03-06T07:27:55.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa395800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3900cf430 0x7fa390156d50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa388009d90 tx=0x7fa388004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa38d600640 1 -- 172.21.15.114:0/3469642782 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388039050 con 0x7fa3900cf430 2024-03-06T07:27:55.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa38d600640 1 -- 172.21.15.114:0/3469642782 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa388045070 con 0x7fa3900cf430 2024-03-06T07:27:55.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39014fd70 con 0x7fa3900cf430 2024-03-06T07:27:55.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa38d600640 1 -- 172.21.15.114:0/3469642782 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38803e610 con 0x7fa3900cf430 2024-03-06T07:27:55.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.230+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa390150330 con 0x7fa3900cf430 2024-03-06T07:27:55.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.231+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3900b2fc0 con 0x7fa3900cf430 2024-03-06T07:27:55.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.232+0000 7fa38d600640 1 -- 172.21.15.114:0/3469642782 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa38803e7b0 con 0x7fa3900cf430 2024-03-06T07:27:55.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.233+0000 7fa38d600640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa36c06b4d0 0x7fa36c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.233+0000 7fa38d600640 1 -- 172.21.15.114:0/3469642782 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3880acd40 con 0x7fa3900cf430 2024-03-06T07:27:55.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.234+0000 7fa394e00640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa36c06b4d0 0x7fa36c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.234+0000 7fa394e00640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa36c06b4d0 0x7fa36c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa378006fd0 tx=0x7fa378008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.238+0000 7fa38d600640 1 -- 172.21.15.114:0/3469642782 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3900b2fc0 con 0x7fa3900cf430 2024-03-06T07:27:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.430+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa39004f330 con 0x7fa3900cf430 2024-03-06T07:27:55.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.432+0000 7fa38d600640 1 -- 172.21.15.114:0/3469642782 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16666 (secure 0 0 0) 0x7fa38807bbb0 con 0x7fa3900cf430 2024-03-06T07:27:55.434 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:55.434 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":19,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:54.463387+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa36c06b4d0 msgr2=0x7fa36c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa36c06b4d0 0x7fa36c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa378006fd0 tx=0x7fa378008040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3900cf430 msgr2=0x7fa390156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3900cf430 0x7fa390156d50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa388009d90 tx=0x7fa388004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 shutdown_connections 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa36c06b4d0 0x7fa36c06d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3900cf430 0x7fa390156d50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa390071260 0x7fa390144c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 --2- 172.21.15.114:0/3469642782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa39006d260 0x7fa3901662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 >> 172.21.15.114:0/3469642782 conn(0x7fa3900b1730 msgr2=0x7fa390159030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 shutdown_connections 2024-03-06T07:27:55.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.435+0000 7fa397800640 1 -- 172.21.15.114:0/3469642782 wait complete. 2024-03-06T07:27:55.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:55.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.653+0000 7f0fbb200640 1 Processor -- start 2024-03-06T07:27:55.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.653+0000 7f0fbb200640 1 -- start start 2024-03-06T07:27:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fbb200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0fb415f9d0 0x7f0fb415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fbb200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0fb4070b10 0x7f0fb4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fbb200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb406cef0 con 0x7f0fb4144c00 2024-03-06T07:27:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fbb200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb406d1d0 con 0x7f0fb4070b10 2024-03-06T07:27:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fbb200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb406d4b0 con 0x7f0fb415f9d0 2024-03-06T07:27:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fb3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0fb4144c00 0x7f0fb406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54552/0 (socket says 172.21.15.114:54552) 2024-03-06T07:27:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fb3e00640 1 -- 172.21.15.114:0/3844521758 learned_addr learned my addr 172.21.15.114:0/3844521758 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fb9200640 1 --2- 172.21.15.114:0/3844521758 >> v2:172.21.15.114:3300/0 conn(0x7f0fb4070b10 0x7f0fb4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fb3400640 1 --2- 172.21.15.114:0/3844521758 >> v2:172.21.15.177:3300/0 conn(0x7f0fb415f9d0 0x7f0fb415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fb2a00640 1 -- 172.21.15.114:0/3844521758 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086510432 0 0) 0x7f0fb406cef0 con 0x7f0fb4144c00 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.655+0000 7f0fb2a00640 1 -- 172.21.15.114:0/3844521758 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f98003520 con 0x7f0fb4144c00 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.656+0000 7f0fb9200640 1 -- 172.21.15.114:0/3844521758 >> v2:172.21.15.177:3300/0 conn(0x7f0fb415f9d0 msgr2=0x7f0fb415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.656+0000 7f0fb9200640 1 --2- 172.21.15.114:0/3844521758 >> v2:172.21.15.177:3300/0 conn(0x7f0fb415f9d0 0x7f0fb415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.656+0000 7f0fb9200640 1 -- 172.21.15.114:0/3844521758 >> v1:172.21.15.114:6789/0 conn(0x7f0fb4144c00 legacy=0x7f0fb406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.656+0000 7f0fb9200640 1 -- 172.21.15.114:0/3844521758 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb406d790 con 0x7f0fb4070b10 2024-03-06T07:27:55.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.657+0000 7f0fb9200640 1 --2- 172.21.15.114:0/3844521758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb4070b10 0x7f0fb4070f20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40b26f0 tx=0x7f0fac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0cdb739403bd39c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.657+0000 7f0fb2a00640 1 -- 172.21.15.114:0/3844521758 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac039070 con 0x7f0fb4070b10 2024-03-06T07:27:55.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.657+0000 7f0fb2a00640 1 -- 172.21.15.114:0/3844521758 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac02cc80 con 0x7f0fb4070b10 2024-03-06T07:27:55.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.657+0000 7f0fb2a00640 1 -- 172.21.15.114:0/3844521758 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac034660 con 0x7f0fb4070b10 2024-03-06T07:27:55.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.657+0000 7f0fbb200640 1 -- 172.21.15.114:0/3844521758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb4070b10 msgr2=0x7f0fb4070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.657+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3844521758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb4070b10 0x7f0fb4070f20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40b26f0 tx=0x7f0fac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.658+0000 7f0fbb200640 1 -- 172.21.15.114:0/3844521758 shutdown_connections 2024-03-06T07:27:55.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.658+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3844521758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb4070b10 0x7f0fb4070f20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.658+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3844521758 >> v2:172.21.15.177:3300/0 conn(0x7f0fb415f9d0 0x7f0fb415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.658+0000 7f0fbb200640 1 -- 172.21.15.114:0/3844521758 >> 172.21.15.114:0/3844521758 conn(0x7f0fb40b1730 msgr2=0x7f0fb40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.658+0000 7f0fbb200640 1 -- 172.21.15.114:0/3844521758 shutdown_connections 2024-03-06T07:27:55.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.658+0000 7f0fbb200640 1 -- 172.21.15.114:0/3844521758 wait complete. 2024-03-06T07:27:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.659+0000 7f0fbb200640 1 Processor -- start 2024-03-06T07:27:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.660+0000 7f0fbb200640 1 -- start start 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fbb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fb4070b10 0x7f0fb415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fbb200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fb4144c00 0x7f0fb4140630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fbb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb415f9d0 0x7f0fb4140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fbb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb406ccd0 con 0x7f0fb415f9d0 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fbb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb406cb50 con 0x7f0fb4070b10 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fbb200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb406df90 con 0x7f0fb4144c00 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb415f9d0 0x7f0fb4140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb415f9d0 0x7f0fb4140b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46572/0 (socket says 172.21.15.114:46572) 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 -- 172.21.15.114:0/3472447298 learned_addr learned my addr 172.21.15.114:0/3472447298 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:55.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb3e00640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fb4070b10 0x7f0fb415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 -- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fb4144c00 msgr2=0x7f0fb4140630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb3400640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fb4144c00 0x7f0fb4140630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fb4144c00 0x7f0fb4140630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 -- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fb4070b10 msgr2=0x7f0fb415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fb4070b10 0x7f0fb415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb9200640 1 -- 172.21.15.114:0/3472447298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb415de40 con 0x7f0fb415f9d0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.661+0000 7f0fb3e00640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fb4070b10 0x7f0fb415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.662+0000 7f0fb9200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb415f9d0 0x7f0fb4140b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009e80 tx=0x7f0fac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.662+0000 7f0fabe00640 1 -- 172.21.15.114:0/3472447298 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac039050 con 0x7f0fb415f9d0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.662+0000 7f0fabe00640 1 -- 172.21.15.114:0/3472447298 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac045070 con 0x7f0fb415f9d0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.662+0000 7f0fabe00640 1 -- 172.21.15.114:0/3472447298 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac009450 con 0x7f0fb415f9d0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.662+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb41411f0 con 0x7f0fb415f9d0 2024-03-06T07:27:55.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.662+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb41417b0 con 0x7f0fb415f9d0 2024-03-06T07:27:55.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.663+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb404f330 con 0x7f0fb415f9d0 2024-03-06T07:27:55.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.664+0000 7f0fabe00640 1 -- 172.21.15.114:0/3472447298 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fac03e850 con 0x7f0fb415f9d0 2024-03-06T07:27:55.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.665+0000 7f0fabe00640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f9006b4d0 0x7f0f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.665+0000 7f0fabe00640 1 -- 172.21.15.114:0/3472447298 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fac0ace90 con 0x7f0fb415f9d0 2024-03-06T07:27:55.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.665+0000 7f0fb3e00640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f9006b4d0 0x7f0f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.666+0000 7f0fb3e00640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f9006b4d0 0x7f0f9006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4007fd0 tx=0x7f0fa4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.670+0000 7f0fabe00640 1 -- 172.21.15.114:0/3472447298 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fac07be20 con 0x7f0fb415f9d0 2024-03-06T07:27:55.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.864+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fb4072890 con 0x7f0fb415f9d0 2024-03-06T07:27:55.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.866+0000 7f0fabe00640 1 -- 172.21.15.114:0/3472447298 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f0fac0ae090 con 0x7f0fb415f9d0 2024-03-06T07:27:55.868 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:55.868 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":19,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:54.463387+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f9006b4d0 msgr2=0x7f0f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f9006b4d0 0x7f0f9006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4007fd0 tx=0x7f0fa4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb415f9d0 msgr2=0x7f0fb4140b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb415f9d0 0x7f0fb4140b70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009e80 tx=0x7f0fac004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 shutdown_connections 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f9006b4d0 0x7f0f9006d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fb415f9d0 0x7f0fb4140b70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fb4144c00 0x7f0fb4140630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 --2- 172.21.15.114:0/3472447298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fb4070b10 0x7f0fb415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.868+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 >> 172.21.15.114:0/3472447298 conn(0x7f0fb40b1730 msgr2=0x7f0fb4145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.869+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 shutdown_connections 2024-03-06T07:27:55.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:55.869+0000 7f0fbb200640 1 -- 172.21.15.114:0/3472447298 wait complete. 2024-03-06T07:27:55.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:56.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.093+0000 7fc574a00640 1 Processor -- start 2024-03-06T07:27:56.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.093+0000 7fc574a00640 1 -- start start 2024-03-06T07:27:56.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.094+0000 7fc574a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc5700c8080 0x7fc5700d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc574a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5700d13e0 con 0x7fc57015e130 2024-03-06T07:27:56.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc574a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5700cead0 con 0x7fc5700d1ea0 2024-03-06T07:27:56.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc574a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5700d2840 con 0x7fc5700c8080 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56e800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc57015e130 0x7fc5700cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54566/0 (socket says 172.21.15.114:54566) 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56e800640 1 -- 172.21.15.114:0/1020433999 learned_addr learned my addr 172.21.15.114:0/1020433999 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45057115 0 0) 0x7fc5700d13e0 con 0x7fc57015e130 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc554003520 con 0x7fc57015e130 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56f200640 1 --2- 172.21.15.114:0/1020433999 >> v2:172.21.15.177:3300/0 conn(0x7fc5700c8080 0x7fc5700d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3725091611 0 0) 0x7fc554003520 con 0x7fc57015e130 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 >> v2:172.21.15.177:3300/0 conn(0x7fc5700c8080 msgr2=0x7fc5700d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 --2- 172.21.15.114:0/1020433999 >> v2:172.21.15.177:3300/0 conn(0x7fc5700c8080 0x7fc5700d2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 >> v1:172.21.15.177:6789/0 conn(0x7fc5700d1ea0 legacy=0x7fc5700c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700cedb0 con 0x7fc57015e130 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fc560001670 con 0x7fc57015e130 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.095+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 >> v1:172.21.15.114:6789/0 conn(0x7fc57015e130 legacy=0x7fc5700cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56de00640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540039b0 0x7fc554003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56de00640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc554004800 0x7fc554004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56de00640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540051c0 0x7fc55400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55400c0c0 con 0x7fc5540051c0 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55400c270 con 0x7fc5540039b0 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55400c470 con 0x7fc554004800 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56f200640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540051c0 0x7fc55400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56e800640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540039b0 0x7fc554003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc567e00640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc554004800 0x7fc554004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56f200640 1 -- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc554004800 msgr2=0x7fc554004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56f200640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc554004800 0x7fc554004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56f200640 1 -- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540039b0 msgr2=0x7fc554003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56f200640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540039b0 0x7fc554003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc56f200640 1 -- 172.21.15.114:0/1020433999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554003520 con 0x7fc5540051c0 2024-03-06T07:27:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.096+0000 7fc567e00640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc554004800 0x7fc554004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.097+0000 7fc56f200640 1 -- 172.21.15.114:0/1020433999 reap_dead start 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.097+0000 7fc56f200640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540051c0 0x7fc55400b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc568007330 tx=0x7fc554005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145ad14c5b5afc25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568009be0 con 0x7fc5540051c0 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56802ccd0 con 0x7fc5540051c0 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc56de00640 1 -- 172.21.15.114:0/1020433999 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568003bd0 con 0x7fc5540051c0 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc574a00640 1 -- 172.21.15.114:0/1020433999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540051c0 msgr2=0x7fc55400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc574a00640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540051c0 0x7fc55400b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc568007330 tx=0x7fc554005830 comp rx=0 tx=0).stop 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc574a00640 1 -- 172.21.15.114:0/1020433999 shutdown_connections 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc574a00640 1 --2- 172.21.15.114:0/1020433999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5540051c0 0x7fc55400b9c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc574a00640 1 -- 172.21.15.114:0/1020433999 >> 172.21.15.114:0/1020433999 conn(0x7fc57009d070 msgr2=0x7fc5700d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc574a00640 1 -- 172.21.15.114:0/1020433999 shutdown_connections 2024-03-06T07:27:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.098+0000 7fc574a00640 1 -- 172.21.15.114:0/1020433999 wait complete. 2024-03-06T07:27:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.100+0000 7fc574a00640 1 Processor -- start 2024-03-06T07:27:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.100+0000 7fc574a00640 1 -- start start 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc574a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540051c0 0x7fc5700cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc574a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5700cbf40 0x7fc5700a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc574a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5701f0da0 0x7fc57009dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc574a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700d2300 con 0x7fc5700cbf40 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc574a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700cf120 con 0x7fc5540051c0 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc574a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700d2480 con 0x7fc5701f0da0 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc567e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5700cbf40 0x7fc5700a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc567e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5700cbf40 0x7fc5700a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46580/0 (socket says 172.21.15.114:46580) 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.101+0000 7fc567e00640 1 -- 172.21.15.114:0/31539461 learned_addr learned my addr 172.21.15.114:0/31539461 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc56e800640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540051c0 0x7fc5700cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc567e00640 1 -- 172.21.15.114:0/31539461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5701f0da0 msgr2=0x7fc57009dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc567e00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5701f0da0 0x7fc57009dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc567e00640 1 -- 172.21.15.114:0/31539461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540051c0 msgr2=0x7fc5700cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc567e00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540051c0 0x7fc5700cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc567e00640 1 -- 172.21.15.114:0/31539461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700cc440 con 0x7fc5700cbf40 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc567e00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5700cbf40 0x7fc5700a0d80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc55800d590 tx=0x7fc55800da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc566a00640 1 -- 172.21.15.114:0/31539461 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558016070 con 0x7fc5700cbf40 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc566a00640 1 -- 172.21.15.114:0/31539461 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558007240 con 0x7fc5700cbf40 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc566a00640 1 -- 172.21.15.114:0/31539461 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558007630 con 0x7fc5700cbf40 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5700cc720 con 0x7fc5700cbf40 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.102+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5700ccce0 con 0x7fc5700cbf40 2024-03-06T07:27:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.103+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57004f330 con 0x7fc5700cbf40 2024-03-06T07:27:56.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.104+0000 7fc566a00640 1 -- 172.21.15.114:0/31539461 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5580067d0 con 0x7fc5700cbf40 2024-03-06T07:27:56.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.105+0000 7fc566a00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc54c06b4d0 0x7fc54c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.105+0000 7fc566a00640 1 -- 172.21.15.114:0/31539461 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc55808a090 con 0x7fc5700cbf40 2024-03-06T07:27:56.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.105+0000 7fc56e800640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc54c06b4d0 0x7fc54c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.106+0000 7fc56e800640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc54c06b4d0 0x7fc54c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc560001200 tx=0x7fc560000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.114+0000 7fc566a00640 1 -- 172.21.15.114:0/31539461 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc558058f00 con 0x7fc5700cbf40 2024-03-06T07:27:56.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:56.349+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc57009d690 con 0x7fc5700cbf40 2024-03-06T07:27:57.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.026+0000 7fc566a00640 1 -- 172.21.15.114:0/31539461 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc558058800 con 0x7fc5700cbf40 2024-03-06T07:27:57.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc54c06b4d0 msgr2=0x7fc54c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc54c06b4d0 0x7fc54c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc560001200 tx=0x7fc560000ed0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5700cbf40 msgr2=0x7fc5700a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5700cbf40 0x7fc5700a0d80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc55800d590 tx=0x7fc55800da50 comp rx=0 tx=0).stop 2024-03-06T07:27:57.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 shutdown_connections 2024-03-06T07:27:57.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5701f0da0 0x7fc57009dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc54c06b4d0 0x7fc54c06d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5700cbf40 0x7fc5700a0d80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 --2- 172.21.15.114:0/31539461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5540051c0 0x7fc5700cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 >> 172.21.15.114:0/31539461 conn(0x7fc57009d070 msgr2=0x7fc5700b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 shutdown_connections 2024-03-06T07:27:57.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.031+0000 7fc574a00640 1 -- 172.21.15.114:0/31539461 wait complete. 2024-03-06T07:27:57.059 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:57.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:57.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.267+0000 7fce39400640 1 Processor -- start 2024-03-06T07:27:57.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.267+0000 7fce39400640 1 -- start start 2024-03-06T07:27:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce39400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce39400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fce341605b0 0x7fce340cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce39400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340ce1e0 con 0x7fce340b43b0 2024-03-06T07:27:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce39400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce340cedf0 con 0x7fce341605b0 2024-03-06T07:27:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce39400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce340ce4c0 con 0x7fce3415c330 2024-03-06T07:27:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce33200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fce341605b0 0x7fce340cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce33200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fce341605b0 0x7fce340cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46586/0 (socket says 172.21.15.114:46586) 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce29e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fce340b43b0 0x7fce3415fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54572/0 (socket says 172.21.15.114:54572) 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce33200640 1 -- 172.21.15.114:0/4088891273 learned_addr learned my addr 172.21.15.114:0/4088891273 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.269+0000 7fce32800640 1 --2- 172.21.15.114:0/4088891273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.270+0000 7fce33200640 1 -- 172.21.15.114:0/4088891273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 msgr2=0x7fce340d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.270+0000 7fce33200640 1 --2- 172.21.15.114:0/4088891273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.270+0000 7fce33200640 1 -- 172.21.15.114:0/4088891273 >> v1:172.21.15.114:6789/0 conn(0x7fce340b43b0 legacy=0x7fce3415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.270+0000 7fce31e00640 1 -- 172.21.15.114:0/4088891273 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871376121 0 0) 0x7fce340ce1e0 con 0x7fce340b43b0 2024-03-06T07:27:57.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.270+0000 7fce33200640 1 -- 172.21.15.114:0/4088891273 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340ce7a0 con 0x7fce341605b0 2024-03-06T07:27:57.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.271+0000 7fce33200640 1 --2- 172.21.15.114:0/4088891273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cc870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fce2c009e50 tx=0x7fce2c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15116fac035d8a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.271+0000 7fce31e00640 1 -- 172.21.15.114:0/4088891273 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c02cb00 con 0x7fce341605b0 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.271+0000 7fce31e00640 1 -- 172.21.15.114:0/4088891273 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2c02cca0 con 0x7fce341605b0 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.271+0000 7fce31e00640 1 -- 172.21.15.114:0/4088891273 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c034610 con 0x7fce341605b0 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 -- 172.21.15.114:0/4088891273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 msgr2=0x7fce340cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 --2- 172.21.15.114:0/4088891273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cc870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fce2c009e50 tx=0x7fce2c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 -- 172.21.15.114:0/4088891273 shutdown_connections 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 --2- 172.21.15.114:0/4088891273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cc870 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 --2- 172.21.15.114:0/4088891273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 -- 172.21.15.114:0/4088891273 >> 172.21.15.114:0/4088891273 conn(0x7fce340b3790 msgr2=0x7fce340b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 -- 172.21.15.114:0/4088891273 shutdown_connections 2024-03-06T07:27:57.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.272+0000 7fce39400640 1 -- 172.21.15.114:0/4088891273 wait complete. 2024-03-06T07:27:57.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.274+0000 7fce39400640 1 Processor -- start 2024-03-06T07:27:57.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.274+0000 7fce39400640 1 -- start start 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce39400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce340b43b0 0x7fce341f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce39400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce39400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce39400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce340cf280 con 0x7fce341605b0 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce39400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3415d720 con 0x7fce340b43b0 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce39400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce341f24f0 con 0x7fce3415c330 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce33200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce33200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46596/0 (socket says 172.21.15.114:46596) 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce33200640 1 -- 172.21.15.114:0/1001559125 learned_addr learned my addr 172.21.15.114:0/1001559125 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce32800640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce340b43b0 0x7fce341f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce29e00640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.275+0000 7fce33200640 1 -- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 msgr2=0x7fce340cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce33200640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce33200640 1 -- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce340b43b0 msgr2=0x7fce341f1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce33200640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce340b43b0 0x7fce341f1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce33200640 1 -- 172.21.15.114:0/1001559125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340ca1f0 con 0x7fce341605b0 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce29e00640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce33200640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cb790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce2c036040 tx=0x7fce2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce2b400640 1 -- 172.21.15.114:0/1001559125 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c044030 con 0x7fce341605b0 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce2b400640 1 -- 172.21.15.114:0/1001559125 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2c02ce50 con 0x7fce341605b0 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce2b400640 1 -- 172.21.15.114:0/1001559125 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c009510 con 0x7fce341605b0 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce340ca770 con 0x7fce341605b0 2024-03-06T07:27:57.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.276+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce340cad30 con 0x7fce341605b0 2024-03-06T07:27:57.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.278+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3406e2e0 con 0x7fce341605b0 2024-03-06T07:27:57.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.279+0000 7fce2b400640 1 -- 172.21.15.114:0/1001559125 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce2c047070 con 0x7fce341605b0 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.280+0000 7fce2b400640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fce1006b4d0 0x7fce1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.280+0000 7fce2b400640 1 -- 172.21.15.114:0/1001559125 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce2c0ac770 con 0x7fce341605b0 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.281+0000 7fce32800640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fce1006b4d0 0x7fce1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:57.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.281+0000 7fce32800640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fce1006b4d0 0x7fce1006d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fce1c006fd0 tx=0x7fce1c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:57.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.285+0000 7fce2b400640 1 -- 172.21.15.114:0/1001559125 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce3406e2e0 con 0x7fce341605b0 2024-03-06T07:27:57.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:57.517+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fce3404f330 con 0x7fce341605b0 2024-03-06T07:27:58.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.035+0000 7fce2b400640 1 -- 172.21.15.114:0/1001559125 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fce2c04e070 con 0x7fce341605b0 2024-03-06T07:27:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fce1006b4d0 msgr2=0x7fce1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fce1006b4d0 0x7fce1006d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fce1c006fd0 tx=0x7fce1c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 msgr2=0x7fce340cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cb790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce2c036040 tx=0x7fce2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 shutdown_connections 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fce1006b4d0 0x7fce1006d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce341605b0 0x7fce340cb790 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.039+0000 7fce39400640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce3415c330 0x7fce340cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.040+0000 7fce39400640 1 --2- 172.21.15.114:0/1001559125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce340b43b0 0x7fce341f1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.040+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 >> 172.21.15.114:0/1001559125 conn(0x7fce340b3790 msgr2=0x7fce340b0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.040+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 shutdown_connections 2024-03-06T07:27:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.040+0000 7fce39400640 1 -- 172.21.15.114:0/1001559125 wait complete. 2024-03-06T07:27:58.062 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:58.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:58.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.257+0000 7fea78200640 1 Processor -- start 2024-03-06T07:27:58.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.257+0000 7fea78200640 1 -- start start 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.258+0000 7fea78200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 0x7fea7015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.258+0000 7fea78200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7006cef0 con 0x7fea70070b10 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.258+0000 7fea78200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7006d1d0 con 0x7fea70144c00 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.258+0000 7fea78200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea70160370 con 0x7fea7015f9d0 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.259+0000 7fea76200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fea70070b10 0x7fea70070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35932/0 (socket says 172.21.15.114:35932) 2024-03-06T07:27:58.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.259+0000 7fea76200640 1 -- 172.21.15.114:0/3735999063 learned_addr learned my addr 172.21.15.114:0/3735999063 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:58.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.259+0000 7fea74e00640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 0x7fea7015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.259+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214788236 0 0) 0x7fea7006cef0 con 0x7fea70070b10 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.259+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea58003520 con 0x7fea70070b10 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191976271 0 0) 0x7fea58003520 con 0x7fea70070b10 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 msgr2=0x7fea7015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 0x7fea7015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 >> v1:172.21.15.177:6789/0 conn(0x7fea70144c00 legacy=0x7fea7006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7006d4b0 con 0x7fea70070b10 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fea680019f0 con 0x7fea70070b10 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 >> v1:172.21.15.114:6789/0 conn(0x7fea70070b10 legacy=0x7fea70070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea58003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 0x7fea58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea58004d40 0x7fea580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5800ba70 con 0x7fea580039e0 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5800bc20 con 0x7fea58004d40 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5800be20 con 0x7fea7015f9d0 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea75800640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea58003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea74e00640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 0x7fea58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.260+0000 7fea76200640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea58004d40 0x7fea580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.261+0000 7fea75800640 1 -- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 msgr2=0x7fea58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.261+0000 7fea75800640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea7015f9d0 0x7fea58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.261+0000 7fea75800640 1 -- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea58004d40 msgr2=0x7fea580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.261+0000 7fea75800640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea58004d40 0x7fea580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.261+0000 7fea75800640 1 -- 172.21.15.114:0/3735999063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea58003520 con 0x7fea580039e0 2024-03-06T07:27:58.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.262+0000 7fea75800640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea58003e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fea64001720 tx=0x7fea58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4979fd46aa1096c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64001370 con 0x7fea580039e0 2024-03-06T07:27:58.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea640052d0 con 0x7fea580039e0 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea6fe00640 1 -- 172.21.15.114:0/3735999063 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64003830 con 0x7fea580039e0 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea78200640 1 -- 172.21.15.114:0/3735999063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 msgr2=0x7fea58003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea78200640 1 --2- 172.21.15.114:0/3735999063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea58003e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fea64001720 tx=0x7fea58005830 comp rx=0 tx=0).stop 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea76200640 1 -- 172.21.15.114:0/3735999063 reap_dead start 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea78200640 1 -- 172.21.15.114:0/3735999063 shutdown_connections 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.263+0000 7fea78200640 1 -- 172.21.15.114:0/3735999063 >> 172.21.15.114:0/3735999063 conn(0x7fea700b1730 msgr2=0x7fea700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.264+0000 7fea78200640 1 -- 172.21.15.114:0/3735999063 shutdown_connections 2024-03-06T07:27:58.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.264+0000 7fea78200640 1 -- 172.21.15.114:0/3735999063 wait complete. 2024-03-06T07:27:58.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.265+0000 7fea78200640 1 Processor -- start 2024-03-06T07:27:58.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.265+0000 7fea78200640 1 -- start start 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.266+0000 7fea78200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea70133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.266+0000 7fea78200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea70134210 0x7fea701f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.266+0000 7fea78200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea701f4630 0x7fea700ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.266+0000 7fea78200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7006cb50 con 0x7fea580039e0 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.266+0000 7fea78200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea70160730 con 0x7fea70134210 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.266+0000 7fea78200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7006ccd0 con 0x7fea701f4630 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.266+0000 7fea75800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea70133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea70133cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55636/0 (socket says 172.21.15.114:55636) 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 -- 172.21.15.114:0/3817053818 learned_addr learned my addr 172.21.15.114:0/3817053818 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea76200640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea701f4630 0x7fea700ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea74e00640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea70134210 0x7fea701f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 -- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea701f4630 msgr2=0x7fea700ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea701f4630 0x7fea700ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 -- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea70134210 msgr2=0x7fea701f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea70134210 0x7fea701f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 -- 172.21.15.114:0/3817053818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea701368a0 con 0x7fea580039e0 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea74e00640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea70134210 0x7fea701f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.267+0000 7fea75800640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea70133cd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea64008fd0 tx=0x7fea64001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.268+0000 7fea6d600640 1 -- 172.21.15.114:0/3817053818 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64041070 con 0x7fea580039e0 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.268+0000 7fea6d600640 1 -- 172.21.15.114:0/3817053818 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea64005480 con 0x7fea580039e0 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.268+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea70136b80 con 0x7fea580039e0 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.268+0000 7fea6d600640 1 -- 172.21.15.114:0/3817053818 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64007a30 con 0x7fea580039e0 2024-03-06T07:27:58.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.268+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea70137140 con 0x7fea580039e0 2024-03-06T07:27:58.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.270+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7004f330 con 0x7fea580039e0 2024-03-06T07:27:58.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.270+0000 7fea6d600640 1 -- 172.21.15.114:0/3817053818 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea64044070 con 0x7fea580039e0 2024-03-06T07:27:58.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.271+0000 7fea6d600640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea5006b500 0x7fea5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.271+0000 7fea6d600640 1 -- 172.21.15.114:0/3817053818 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea640abe40 con 0x7fea580039e0 2024-03-06T07:27:58.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.271+0000 7fea74e00640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea5006b500 0x7fea5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.272+0000 7fea74e00640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea5006b500 0x7fea5006d9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006ce0 tx=0x7fea5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.275+0000 7fea6d600640 1 -- 172.21.15.114:0/3817053818 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea6407ac20 con 0x7fea580039e0 2024-03-06T07:27:58.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.506+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea700ad9e0 con 0x7fea580039e0 2024-03-06T07:27:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.507+0000 7fea6d600640 1 -- 172.21.15.114:0/3817053818 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7fea6407a520 con 0x7fea580039e0 2024-03-06T07:27:58.510 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:27:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea5006b500 msgr2=0x7fea5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea5006b500 0x7fea5006d9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006ce0 tx=0x7fea5c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 msgr2=0x7fea70133cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea70133cd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea64008fd0 tx=0x7fea64001120 comp rx=0 tx=0).stop 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 shutdown_connections 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea701f4630 0x7fea700ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea70134210 0x7fea701f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea5006b500 0x7fea5006d9c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 --2- 172.21.15.114:0/3817053818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea580039e0 0x7fea70133cd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 >> 172.21.15.114:0/3817053818 conn(0x7fea700b1730 msgr2=0x7fea700ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 shutdown_connections 2024-03-06T07:27:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.510+0000 7fea78200640 1 -- 172.21.15.114:0/3817053818 wait complete. 2024-03-06T07:27:58.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:58.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.728+0000 7f1acb800640 1 Processor -- start 2024-03-06T07:27:58.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.728+0000 7f1acb800640 1 -- start start 2024-03-06T07:27:58.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.729+0000 7f1acb800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 0x7f1ac40cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.729+0000 7f1acb800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1ac40d1c20 0x7f1ac40d2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.729+0000 7f1acb800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40cf140 con 0x7f1ac415f110 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.729+0000 7f1acb800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40cfd50 con 0x7f1ac40d1c20 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.729+0000 7f1acb800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40cf420 con 0x7f1ac40d07b0 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1ac415f110 0x7f1ac40d0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35946/0 (socket says 172.21.15.114:35946) 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3e00640 1 -- 172.21.15.114:0/883851147 learned_addr learned my addr 172.21.15.114:0/883851147 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac8e00640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 0x7f1ac40cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac9800640 1 --2- 172.21.15.114:0/883851147 >> v2:172.21.15.177:3300/0 conn(0x7f1ac40d1c20 0x7f1ac40d2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958169359 0 0) 0x7f1ac40cf140 con 0x7f1ac415f110 2024-03-06T07:27:58.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aac003520 con 0x7f1ac415f110 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162581039 0 0) 0x7f1aac003520 con 0x7f1ac415f110 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 msgr2=0x7f1ac40cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3400640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 0x7f1ac40cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 >> v2:172.21.15.177:3300/0 conn(0x7f1ac40d1c20 msgr2=0x7f1ac40d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.730+0000 7f1ac3400640 1 --2- 172.21.15.114:0/883851147 >> v2:172.21.15.177:3300/0 conn(0x7f1ac40d1c20 0x7f1ac40d2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40cf700 con 0x7f1ac415f110 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f1ab0001670 con 0x7f1ac415f110 2024-03-06T07:27:58.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 >> v1:172.21.15.114:6789/0 conn(0x7f1ac415f110 legacy=0x7f1ac40d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 0x7f1aac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aac003f80 0x7f1aac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aac004d40 0x7f1aac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40cfd50 con 0x7f1aac004d40 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40cf420 con 0x7f1aac003f80 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac00b9c0 con 0x7f1ac40d07b0 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac9800640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aac004d40 0x7f1aac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac8e00640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 0x7f1aac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac3e00640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aac003f80 0x7f1aac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac9800640 1 -- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 msgr2=0x7f1aac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac9800640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac40d07b0 0x7f1aac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac9800640 1 -- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aac003f80 msgr2=0x7f1aac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac9800640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1aac003f80 0x7f1aac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.731+0000 7f1ac9800640 1 -- 172.21.15.114:0/883851147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac003520 con 0x7f1aac004d40 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.733+0000 7f1ac9800640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aac004d40 0x7f1aac0051c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00e170 tx=0x7f1aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f495c5c3d78243ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.733+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc039070 con 0x7f1aac004d40 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.733+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc02cc20 con 0x7f1aac004d40 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.733+0000 7f1ac3400640 1 -- 172.21.15.114:0/883851147 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc009c10 con 0x7f1aac004d40 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.734+0000 7f1acb800640 1 -- 172.21.15.114:0/883851147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aac004d40 msgr2=0x7f1aac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.734+0000 7f1acb800640 1 --2- 172.21.15.114:0/883851147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aac004d40 0x7f1aac0051c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00e170 tx=0x7f1aac005830 comp rx=0 tx=0).stop 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.734+0000 7f1ac9800640 1 -- 172.21.15.114:0/883851147 reap_dead start 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.734+0000 7f1acb800640 1 -- 172.21.15.114:0/883851147 shutdown_connections 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.734+0000 7f1acb800640 1 -- 172.21.15.114:0/883851147 >> 172.21.15.114:0/883851147 conn(0x7f1ac40d3ee0 msgr2=0x7f1ac40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.734+0000 7f1acb800640 1 -- 172.21.15.114:0/883851147 shutdown_connections 2024-03-06T07:27:58.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.734+0000 7f1acb800640 1 -- 172.21.15.114:0/883851147 wait complete. 2024-03-06T07:27:58.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.737+0000 7f1acb800640 1 Processor -- start 2024-03-06T07:27:58.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.737+0000 7f1acb800640 1 -- start start 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1acb800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac41f3a60 0x7f1ac40c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1acb800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac40c8f00 0x7f1ac40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1acb800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac41f3e40 0x7f1ac40cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1acb800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac415e390 con 0x7f1ac41f3e40 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1acb800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac415f570 con 0x7f1ac40c8f00 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1acb800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac415e510 con 0x7f1ac41f3a60 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac41f3e40 0x7f1ac40cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac41f3e40 0x7f1ac40cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55650/0 (socket says 172.21.15.114:55650) 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 -- 172.21.15.114:0/408395635 learned_addr learned my addr 172.21.15.114:0/408395635 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac3e00640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac40c8f00 0x7f1ac40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 -- 172.21.15.114:0/408395635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac41f3a60 msgr2=0x7f1ac40c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac41f3a60 0x7f1ac40c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 -- 172.21.15.114:0/408395635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac40c8f00 msgr2=0x7f1ac40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac40c8f00 0x7f1ac40ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.739+0000 7f1ac9800640 1 -- 172.21.15.114:0/408395635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40c93a0 con 0x7f1ac41f3e40 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.740+0000 7f1ac9800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac41f3e40 0x7f1ac40cd500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1abc037040 tx=0x7f1abc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.740+0000 7f1ac0c00640 1 -- 172.21.15.114:0/408395635 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc039030 con 0x7f1ac41f3e40 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.740+0000 7f1ac0c00640 1 -- 172.21.15.114:0/408395635 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc043070 con 0x7f1ac41f3e40 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.740+0000 7f1ac0c00640 1 -- 172.21.15.114:0/408395635 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc009c10 con 0x7f1ac41f3e40 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.740+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac40c9680 con 0x7f1ac41f3e40 2024-03-06T07:27:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.740+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac415a4f0 con 0x7f1ac41f3e40 2024-03-06T07:27:58.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.741+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac404f330 con 0x7f1ac41f3e40 2024-03-06T07:27:58.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.742+0000 7f1ac0c00640 1 -- 172.21.15.114:0/408395635 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1abc005d10 con 0x7f1ac41f3e40 2024-03-06T07:27:58.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.744+0000 7f1ac0c00640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1aa406b500 0x7f1aa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.744+0000 7f1ac0c00640 1 -- 172.21.15.114:0/408395635 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1abc0ac370 con 0x7f1ac41f3e40 2024-03-06T07:27:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.744+0000 7f1ac8e00640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1aa406b500 0x7f1aa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.745+0000 7f1ac8e00640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1aa406b500 0x7f1aa406d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009910 tx=0x7f1ab8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.747+0000 7f1ac0c00640 1 -- 172.21.15.114:0/408395635 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1abc07b1e0 con 0x7f1ac41f3e40 2024-03-06T07:27:58.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.946+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ac408d940 con 0x7f1ac41f3e40 2024-03-06T07:27:58.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.947+0000 7f1ac0c00640 1 -- 172.21.15.114:0/408395635 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f1abc07aae0 con 0x7f1ac41f3e40 2024-03-06T07:27:58.950 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:58.950 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:58.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.949+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1aa406b500 msgr2=0x7f1aa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.949+0000 7f1acb800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1aa406b500 0x7f1aa406d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009910 tx=0x7f1ab8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:58.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac41f3e40 msgr2=0x7f1ac40cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac41f3e40 0x7f1ac40cd500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1abc037040 tx=0x7f1abc009750 comp rx=0 tx=0).stop 2024-03-06T07:27:58.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 shutdown_connections 2024-03-06T07:27:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1aa406b500 0x7f1aa406d9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ac41f3e40 0x7f1ac40cd500 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac40c8f00 0x7f1ac40ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 --2- 172.21.15.114:0/408395635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ac41f3a60 0x7f1ac40c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 >> 172.21.15.114:0/408395635 conn(0x7f1ac40d3ee0 msgr2=0x7f1ac4159830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 shutdown_connections 2024-03-06T07:27:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:58.950+0000 7f1acb800640 1 -- 172.21.15.114:0/408395635 wait complete. 2024-03-06T07:27:58.971 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:58.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:59.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.173+0000 7f9e67e00640 1 Processor -- start 2024-03-06T07:27:59.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.173+0000 7f9e67e00640 1 -- start start 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.174+0000 7f9e67e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e60144c00 0x7f9e6006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.174+0000 7f9e67e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9e6015f9d0 0x7f9e6015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.174+0000 7f9e67e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6006ee70 con 0x7f9e600ce0a0 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.174+0000 7f9e67e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6006cef0 con 0x7f9e60144c00 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.174+0000 7f9e67e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6006d1d0 con 0x7f9e6015f9d0 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.174+0000 7f9e65400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e60144c00 0x7f9e6006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e65400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e60144c00 0x7f9e6006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55662/0 (socket says 172.21.15.114:55662) 2024-03-06T07:27:59.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e65400640 1 -- 172.21.15.114:0/2715222654 learned_addr learned my addr 172.21.15.114:0/2715222654 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:59.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e65e00640 1 --1- 172.21.15.114:0/2715222654 >> v1:172.21.15.114:6789/0 conn(0x7f9e600ce0a0 0x7f9e600ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35952/0 (socket says 172.21.15.114:35952) 2024-03-06T07:27:59.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e64a00640 1 --2- 172.21.15.114:0/2715222654 >> v2:172.21.15.177:3300/0 conn(0x7f9e6015f9d0 0x7f9e6015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e65400640 1 -- 172.21.15.114:0/2715222654 >> v2:172.21.15.177:3300/0 conn(0x7f9e6015f9d0 msgr2=0x7f9e6015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e65400640 1 --2- 172.21.15.114:0/2715222654 >> v2:172.21.15.177:3300/0 conn(0x7f9e6015f9d0 0x7f9e6015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e65400640 1 -- 172.21.15.114:0/2715222654 >> v1:172.21.15.114:6789/0 conn(0x7f9e600ce0a0 legacy=0x7f9e600ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e65400640 1 -- 172.21.15.114:0/2715222654 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6006d4b0 con 0x7f9e60144c00 2024-03-06T07:27:59.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.175+0000 7f9e64a00640 1 --2- 172.21.15.114:0/2715222654 >> v2:172.21.15.177:3300/0 conn(0x7f9e6015f9d0 0x7f9e6015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:59.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.176+0000 7f9e65400640 1 --2- 172.21.15.114:0/2715222654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6006e930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9e600b26f0 tx=0x7f9e5002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe432f07e0808742 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.176+0000 7f9e5fe00640 1 -- 172.21.15.114:0/2715222654 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5002cb10 con 0x7f9e60144c00 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.176+0000 7f9e5fe00640 1 -- 172.21.15.114:0/2715222654 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5002ccb0 con 0x7f9e60144c00 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.176+0000 7f9e5fe00640 1 -- 172.21.15.114:0/2715222654 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5002ce60 con 0x7f9e60144c00 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 -- 172.21.15.114:0/2715222654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 msgr2=0x7f9e6006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2715222654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6006e930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9e600b26f0 tx=0x7f9e5002bf70 comp rx=0 tx=0).stop 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 -- 172.21.15.114:0/2715222654 shutdown_connections 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2715222654 >> v2:172.21.15.177:3300/0 conn(0x7f9e6015f9d0 0x7f9e6015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2715222654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6006e930 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 -- 172.21.15.114:0/2715222654 >> 172.21.15.114:0/2715222654 conn(0x7f9e600b1730 msgr2=0x7f9e600b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 -- 172.21.15.114:0/2715222654 shutdown_connections 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.177+0000 7f9e67e00640 1 -- 172.21.15.114:0/2715222654 wait complete. 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.179+0000 7f9e67e00640 1 Processor -- start 2024-03-06T07:27:59.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.179+0000 7f9e67e00640 1 -- start start 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.180+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e600ce0a0 0x7f9e601e7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6015e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6015f9d0 0x7f9e6015ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e67e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6006cb50 con 0x7f9e60144c00 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e67e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60160730 con 0x7f9e600ce0a0 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e67e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6006ccd0 con 0x7f9e6015f9d0 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6015e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6015e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55676/0 (socket says 172.21.15.114:55676) 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 -- 172.21.15.114:0/2185463992 learned_addr learned my addr 172.21.15.114:0/2185463992 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:59.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e65400640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e600ce0a0 0x7f9e601e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 -- 172.21.15.114:0/2185463992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6015f9d0 msgr2=0x7f9e6015ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6015f9d0 0x7f9e6015ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 -- 172.21.15.114:0/2185463992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e600ce0a0 msgr2=0x7f9e601e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e600ce0a0 0x7f9e601e7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.181+0000 7f9e64a00640 1 -- 172.21.15.114:0/2185463992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e601f3bf0 con 0x7f9e60144c00 2024-03-06T07:27:59.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.182+0000 7f9e64a00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6015e400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c7f0 tx=0x7f9e4c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.182+0000 7f9e5d600640 1 -- 172.21.15.114:0/2185463992 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c015070 con 0x7f9e60144c00 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.182+0000 7f9e5d600640 1 -- 172.21.15.114:0/2185463992 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c007a30 con 0x7f9e60144c00 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.182+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e601f3ed0 con 0x7f9e60144c00 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.182+0000 7f9e5d600640 1 -- 172.21.15.114:0/2185463992 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c007e00 con 0x7f9e60144c00 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.182+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e601f4490 con 0x7f9e60144c00 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.183+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6004f330 con 0x7f9e60144c00 2024-03-06T07:27:59.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.184+0000 7f9e5d600640 1 -- 172.21.15.114:0/2185463992 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e4c018070 con 0x7f9e60144c00 2024-03-06T07:27:59.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.185+0000 7f9e5d600640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4006b4d0 0x7f9e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.185+0000 7f9e5d600640 1 -- 172.21.15.114:0/2185463992 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e4c08a250 con 0x7f9e60144c00 2024-03-06T07:27:59.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.185+0000 7f9e65400640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4006b4d0 0x7f9e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.186+0000 7f9e65400640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4006b4d0 0x7f9e4006d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9e50035000 tx=0x7f9e50030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.189+0000 7f9e5d600640 1 -- 172.21.15.114:0/2185463992 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e4c059030 con 0x7f9e60144c00 2024-03-06T07:27:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.381+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e60070e50 con 0x7f9e4006b4d0 2024-03-06T07:27:59.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.393+0000 7f9e5d600640 1 -- 172.21.15.114:0/2185463992 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251950 (secure 0 0 0) 0x7f9e60070e50 con 0x7f9e4006b4d0 2024-03-06T07:27:59.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:59.401 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:27:59.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4006b4d0 msgr2=0x7f9e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4006b4d0 0x7f9e4006d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9e50035000 tx=0x7f9e50030000 comp rx=0 tx=0).stop 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 msgr2=0x7f9e6015e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6015e400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c7f0 tx=0x7f9e4c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 shutdown_connections 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6015f9d0 0x7f9e6015ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4006b4d0 0x7f9e4006d990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e60144c00 0x7f9e6015e400 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 --2- 172.21.15.114:0/2185463992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e600ce0a0 0x7f9e601e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.398+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 >> 172.21.15.114:0/2185463992 conn(0x7f9e600b1730 msgr2=0x7f9e600b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.399+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 shutdown_connections 2024-03-06T07:27:59.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.399+0000 7f9e67e00640 1 -- 172.21.15.114:0/2185463992 wait complete. 2024-03-06T07:27:59.422 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:27:59.100320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001263"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517399+0000","last_change":"2024-03-06T07:27:55.014476+0000","last_active":"2024-03-06T07:27:56.517399+0000","last_peered":"2024-03-06T07:27:56.517399+0000","last_clean":"2024-03-06T07:27:56.517399+0000","last_became_active":"2024-03-06T07:27:55.014169+0000","last_became_peered":"2024-03-06T07:27:55.014169+0000","last_unstale":"2024-03-06T07:27:56.517399+0000","last_undegraded":"2024-03-06T07:27:56.517399+0000","last_fullsized":"2024-03-06T07:27:56.517399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:20.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020046+0000","last_change":"2024-03-06T07:27:55.025640+0000","last_active":"2024-03-06T07:27:56.020046+0000","last_peered":"2024-03-06T07:27:56.020046+0000","last_clean":"2024-03-06T07:27:56.020046+0000","last_became_active":"2024-03-06T07:27:55.025215+0000","last_became_peered":"2024-03-06T07:27:55.025215+0000","last_unstale":"2024-03-06T07:27:56.020046+0000","last_undegraded":"2024-03-06T07:27:56.020046+0000","last_fullsized":"2024-03-06T07:27:56.020046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514320+0000","last_change":"2024-03-06T07:27:55.015505+0000","last_active":"2024-03-06T07:27:56.514320+0000","last_peered":"2024-03-06T07:27:56.514320+0000","last_clean":"2024-03-06T07:27:56.514320+0000","last_became_active":"2024-03-06T07:27:55.015199+0000","last_became_peered":"2024-03-06T07:27:55.015199+0000","last_unstale":"2024-03-06T07:27:56.514320+0000","last_undegraded":"2024-03-06T07:27:56.514320+0000","last_fullsized":"2024-03-06T07:27:56.514320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518415+0000","last_change":"2024-03-06T07:27:55.033897+0000","last_active":"2024-03-06T07:27:56.518415+0000","last_peered":"2024-03-06T07:27:56.518415+0000","last_clean":"2024-03-06T07:27:56.518415+0000","last_became_active":"2024-03-06T07:27:55.032987+0000","last_became_peered":"2024-03-06T07:27:55.032987+0000","last_unstale":"2024-03-06T07:27:56.518415+0000","last_undegraded":"2024-03-06T07:27:56.518415+0000","last_fullsized":"2024-03-06T07:27:56.518415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515678+0000","last_change":"2024-03-06T07:27:55.014597+0000","last_active":"2024-03-06T07:27:56.515678+0000","last_peered":"2024-03-06T07:27:56.515678+0000","last_clean":"2024-03-06T07:27:56.515678+0000","last_became_active":"2024-03-06T07:27:55.014206+0000","last_became_peered":"2024-03-06T07:27:55.014206+0000","last_unstale":"2024-03-06T07:27:56.515678+0000","last_undegraded":"2024-03-06T07:27:56.515678+0000","last_fullsized":"2024-03-06T07:27:56.515678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516836+0000","last_change":"2024-03-06T07:27:55.028575+0000","last_active":"2024-03-06T07:27:56.516836+0000","last_peered":"2024-03-06T07:27:56.516836+0000","last_clean":"2024-03-06T07:27:56.516836+0000","last_became_active":"2024-03-06T07:27:55.027801+0000","last_became_peered":"2024-03-06T07:27:55.027801+0000","last_unstale":"2024-03-06T07:27:56.516836+0000","last_undegraded":"2024-03-06T07:27:56.516836+0000","last_fullsized":"2024-03-06T07:27:56.516836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514995+0000","last_change":"2024-03-06T07:27:55.028417+0000","last_active":"2024-03-06T07:27:56.514995+0000","last_peered":"2024-03-06T07:27:56.514995+0000","last_clean":"2024-03-06T07:27:56.514995+0000","last_became_active":"2024-03-06T07:27:55.028108+0000","last_became_peered":"2024-03-06T07:27:55.028108+0000","last_unstale":"2024-03-06T07:27:56.514995+0000","last_undegraded":"2024-03-06T07:27:56.514995+0000","last_fullsized":"2024-03-06T07:27:56.514995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021025+0000","last_change":"2024-03-06T07:27:55.024045+0000","last_active":"2024-03-06T07:27:56.021025+0000","last_peered":"2024-03-06T07:27:56.021025+0000","last_clean":"2024-03-06T07:27:56.021025+0000","last_became_active":"2024-03-06T07:27:55.023528+0000","last_became_peered":"2024-03-06T07:27:55.023528+0000","last_unstale":"2024-03-06T07:27:56.021025+0000","last_undegraded":"2024-03-06T07:27:56.021025+0000","last_fullsized":"2024-03-06T07:27:56.021025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517632+0000","last_change":"2024-03-06T07:27:55.021303+0000","last_active":"2024-03-06T07:27:56.517632+0000","last_peered":"2024-03-06T07:27:56.517632+0000","last_clean":"2024-03-06T07:27:56.517632+0000","last_became_active":"2024-03-06T07:27:55.020953+0000","last_became_peered":"2024-03-06T07:27:55.020953+0000","last_unstale":"2024-03-06T07:27:56.517632+0000","last_undegraded":"2024-03-06T07:27:56.517632+0000","last_fullsized":"2024-03-06T07:27:56.517632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:30.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027586+0000","last_change":"2024-03-06T07:27:55.019249+0000","last_active":"2024-03-06T07:27:56.027586+0000","last_peered":"2024-03-06T07:27:56.027586+0000","last_clean":"2024-03-06T07:27:56.027586+0000","last_became_active":"2024-03-06T07:27:55.018947+0000","last_became_peered":"2024-03-06T07:27:55.018947+0000","last_unstale":"2024-03-06T07:27:56.027586+0000","last_undegraded":"2024-03-06T07:27:56.027586+0000","last_fullsized":"2024-03-06T07:27:56.027586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027423+0000","last_change":"2024-03-06T07:27:55.023288+0000","last_active":"2024-03-06T07:27:56.027423+0000","last_peered":"2024-03-06T07:27:56.027423+0000","last_clean":"2024-03-06T07:27:56.027423+0000","last_became_active":"2024-03-06T07:27:55.023020+0000","last_became_peered":"2024-03-06T07:27:55.023020+0000","last_unstale":"2024-03-06T07:27:56.027423+0000","last_undegraded":"2024-03-06T07:27:56.027423+0000","last_fullsized":"2024-03-06T07:27:56.027423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029290+0000","last_change":"2024-03-06T07:27:55.014888+0000","last_active":"2024-03-06T07:27:56.029290+0000","last_peered":"2024-03-06T07:27:56.029290+0000","last_clean":"2024-03-06T07:27:56.029290+0000","last_became_active":"2024-03-06T07:27:55.014394+0000","last_became_peered":"2024-03-06T07:27:55.014394+0000","last_unstale":"2024-03-06T07:27:56.029290+0000","last_undegraded":"2024-03-06T07:27:56.029290+0000","last_fullsized":"2024-03-06T07:27:56.029290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:58.566589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021843+0000","last_change":"2024-03-06T07:27:55.014936+0000","last_active":"2024-03-06T07:27:56.021843+0000","last_peered":"2024-03-06T07:27:56.021843+0000","last_clean":"2024-03-06T07:27:56.021843+0000","last_became_active":"2024-03-06T07:27:55.012277+0000","last_became_peered":"2024-03-06T07:27:55.012277+0000","last_unstale":"2024-03-06T07:27:56.021843+0000","last_undegraded":"2024-03-06T07:27:56.021843+0000","last_fullsized":"2024-03-06T07:27:56.021843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.012205+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.011928+0000","last_became_peered":"2024-03-06T07:27:55.011928+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019838+0000","last_change":"2024-03-06T07:27:55.030004+0000","last_active":"2024-03-06T07:27:56.019838+0000","last_peered":"2024-03-06T07:27:56.019838+0000","last_clean":"2024-03-06T07:27:56.019838+0000","last_became_active":"2024-03-06T07:27:55.029632+0000","last_became_peered":"2024-03-06T07:27:55.029632+0000","last_unstale":"2024-03-06T07:27:56.019838+0000","last_undegraded":"2024-03-06T07:27:56.019838+0000","last_fullsized":"2024-03-06T07:27:56.019838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515777+0000","last_change":"2024-03-06T07:27:55.028212+0000","last_active":"2024-03-06T07:27:56.515777+0000","last_peered":"2024-03-06T07:27:56.515777+0000","last_clean":"2024-03-06T07:27:56.515777+0000","last_became_active":"2024-03-06T07:27:55.027899+0000","last_became_peered":"2024-03-06T07:27:55.027899+0000","last_unstale":"2024-03-06T07:27:56.515777+0000","last_undegraded":"2024-03-06T07:27:56.515777+0000","last_fullsized":"2024-03-06T07:27:56.515777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519519+0000","last_change":"2024-03-06T07:27:55.024791+0000","last_active":"2024-03-06T07:27:56.519519+0000","last_peered":"2024-03-06T07:27:56.519519+0000","last_clean":"2024-03-06T07:27:56.519519+0000","last_became_active":"2024-03-06T07:27:55.023356+0000","last_became_peered":"2024-03-06T07:27:55.023356+0000","last_unstale":"2024-03-06T07:27:56.519519+0000","last_undegraded":"2024-03-06T07:27:56.519519+0000","last_fullsized":"2024-03-06T07:27:56.519519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514884+0000","last_change":"2024-03-06T07:27:54.994313+0000","last_active":"2024-03-06T07:27:56.514884+0000","last_peered":"2024-03-06T07:27:56.514884+0000","last_clean":"2024-03-06T07:27:56.514884+0000","last_became_active":"2024-03-06T07:27:54.993537+0000","last_became_peered":"2024-03-06T07:27:54.993537+0000","last_unstale":"2024-03-06T07:27:56.514884+0000","last_undegraded":"2024-03-06T07:27:56.514884+0000","last_fullsized":"2024-03-06T07:27:56.514884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:34.222639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516724+0000","last_change":"2024-03-06T07:27:55.016027+0000","last_active":"2024-03-06T07:27:56.516724+0000","last_peered":"2024-03-06T07:27:56.516724+0000","last_clean":"2024-03-06T07:27:56.516724+0000","last_became_active":"2024-03-06T07:27:55.015462+0000","last_became_peered":"2024-03-06T07:27:55.015462+0000","last_unstale":"2024-03-06T07:27:56.516724+0000","last_undegraded":"2024-03-06T07:27:56.516724+0000","last_fullsized":"2024-03-06T07:27:56.516724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027150+0000","last_change":"2024-03-06T07:27:55.015671+0000","last_active":"2024-03-06T07:27:56.027150+0000","last_peered":"2024-03-06T07:27:56.027150+0000","last_clean":"2024-03-06T07:27:56.027150+0000","last_became_active":"2024-03-06T07:27:55.015352+0000","last_became_peered":"2024-03-06T07:27:55.015352+0000","last_unstale":"2024-03-06T07:27:56.027150+0000","last_undegraded":"2024-03-06T07:27:56.027150+0000","last_fullsized":"2024-03-06T07:27:56.027150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513815+0000","last_change":"2024-03-06T07:27:55.026752+0000","last_active":"2024-03-06T07:27:56.513815+0000","last_peered":"2024-03-06T07:27:56.513815+0000","last_clean":"2024-03-06T07:27:56.513815+0000","last_became_active":"2024-03-06T07:27:55.026382+0000","last_became_peered":"2024-03-06T07:27:55.026382+0000","last_unstale":"2024-03-06T07:27:56.513815+0000","last_undegraded":"2024-03-06T07:27:56.513815+0000","last_fullsized":"2024-03-06T07:27:56.513815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517269+0000","last_change":"2024-03-06T07:27:55.024122+0000","last_active":"2024-03-06T07:27:56.517269+0000","last_peered":"2024-03-06T07:27:56.517269+0000","last_clean":"2024-03-06T07:27:56.517269+0000","last_became_active":"2024-03-06T07:27:55.023796+0000","last_became_peered":"2024-03-06T07:27:55.023796+0000","last_unstale":"2024-03-06T07:27:56.517269+0000","last_undegraded":"2024-03-06T07:27:56.517269+0000","last_fullsized":"2024-03-06T07:27:56.517269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021097+0000","last_change":"2024-03-06T07:27:55.020822+0000","last_active":"2024-03-06T07:27:56.021097+0000","last_peered":"2024-03-06T07:27:56.021097+0000","last_clean":"2024-03-06T07:27:56.021097+0000","last_became_active":"2024-03-06T07:27:55.020406+0000","last_became_peered":"2024-03-06T07:27:55.020406+0000","last_unstale":"2024-03-06T07:27:56.021097+0000","last_undegraded":"2024-03-06T07:27:56.021097+0000","last_fullsized":"2024-03-06T07:27:56.021097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516215+0000","last_change":"2024-03-06T07:27:55.016257+0000","last_active":"2024-03-06T07:27:56.516215+0000","last_peered":"2024-03-06T07:27:56.516215+0000","last_clean":"2024-03-06T07:27:56.516215+0000","last_became_active":"2024-03-06T07:27:55.013454+0000","last_became_peered":"2024-03-06T07:27:55.013454+0000","last_unstale":"2024-03-06T07:27:56.516215+0000","last_undegraded":"2024-03-06T07:27:56.516215+0000","last_fullsized":"2024-03-06T07:27:56.516215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021692+0000","last_change":"2024-03-06T07:27:55.014609+0000","last_active":"2024-03-06T07:27:56.021692+0000","last_peered":"2024-03-06T07:27:56.021692+0000","last_clean":"2024-03-06T07:27:56.021692+0000","last_became_active":"2024-03-06T07:27:55.014270+0000","last_became_peered":"2024-03-06T07:27:55.014270+0000","last_unstale":"2024-03-06T07:27:56.021692+0000","last_undegraded":"2024-03-06T07:27:56.021692+0000","last_fullsized":"2024-03-06T07:27:56.021692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027608+0000","last_change":"2024-03-06T07:27:55.012622+0000","last_active":"2024-03-06T07:27:56.027608+0000","last_peered":"2024-03-06T07:27:56.027608+0000","last_clean":"2024-03-06T07:27:56.027608+0000","last_became_active":"2024-03-06T07:27:55.012268+0000","last_became_peered":"2024-03-06T07:27:55.012268+0000","last_unstale":"2024-03-06T07:27:56.027608+0000","last_undegraded":"2024-03-06T07:27:56.027608+0000","last_fullsized":"2024-03-06T07:27:56.027608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019641+0000","last_change":"2024-03-06T07:27:55.025182+0000","last_active":"2024-03-06T07:27:56.019641+0000","last_peered":"2024-03-06T07:27:56.019641+0000","last_clean":"2024-03-06T07:27:56.019641+0000","last_became_active":"2024-03-06T07:27:55.024874+0000","last_became_peered":"2024-03-06T07:27:55.024874+0000","last_unstale":"2024-03-06T07:27:56.019641+0000","last_undegraded":"2024-03-06T07:27:56.019641+0000","last_fullsized":"2024-03-06T07:27:56.019641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.022062+0000","last_change":"2024-03-06T07:27:55.031727+0000","last_active":"2024-03-06T07:27:56.022062+0000","last_peered":"2024-03-06T07:27:56.022062+0000","last_clean":"2024-03-06T07:27:56.022062+0000","last_became_active":"2024-03-06T07:27:55.029328+0000","last_became_peered":"2024-03-06T07:27:55.029328+0000","last_unstale":"2024-03-06T07:27:56.022062+0000","last_undegraded":"2024-03-06T07:27:56.022062+0000","last_fullsized":"2024-03-06T07:27:56.022062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513894+0000","last_change":"2024-03-06T07:27:55.011251+0000","last_active":"2024-03-06T07:27:56.513894+0000","last_peered":"2024-03-06T07:27:56.513894+0000","last_clean":"2024-03-06T07:27:56.513894+0000","last_became_active":"2024-03-06T07:27:55.010892+0000","last_became_peered":"2024-03-06T07:27:55.010892+0000","last_unstale":"2024-03-06T07:27:56.513894+0000","last_undegraded":"2024-03-06T07:27:56.513894+0000","last_fullsized":"2024-03-06T07:27:56.513894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517664+0000","last_change":"2024-03-06T07:27:55.036627+0000","last_active":"2024-03-06T07:27:56.517664+0000","last_peered":"2024-03-06T07:27:56.517664+0000","last_clean":"2024-03-06T07:27:56.517664+0000","last_became_active":"2024-03-06T07:27:55.036245+0000","last_became_peered":"2024-03-06T07:27:55.036245+0000","last_unstale":"2024-03-06T07:27:56.517664+0000","last_undegraded":"2024-03-06T07:27:56.517664+0000","last_fullsized":"2024-03-06T07:27:56.517664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514454+0000","last_change":"2024-03-06T07:27:55.018254+0000","last_active":"2024-03-06T07:27:56.514454+0000","last_peered":"2024-03-06T07:27:56.514454+0000","last_clean":"2024-03-06T07:27:56.514454+0000","last_became_active":"2024-03-06T07:27:55.017832+0000","last_became_peered":"2024-03-06T07:27:55.017832+0000","last_unstale":"2024-03-06T07:27:56.514454+0000","last_undegraded":"2024-03-06T07:27:56.514454+0000","last_fullsized":"2024-03-06T07:27:56.514454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:11.749502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026113+0000","last_change":"2024-03-06T07:27:55.024465+0000","last_active":"2024-03-06T07:27:56.026113+0000","last_peered":"2024-03-06T07:27:56.026113+0000","last_clean":"2024-03-06T07:27:56.026113+0000","last_became_active":"2024-03-06T07:27:55.024173+0000","last_became_peered":"2024-03-06T07:27:55.024173+0000","last_unstale":"2024-03-06T07:27:56.026113+0000","last_undegraded":"2024-03-06T07:27:56.026113+0000","last_fullsized":"2024-03-06T07:27:56.026113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028167+0000","last_change":"2024-03-06T07:27:55.013597+0000","last_active":"2024-03-06T07:27:56.028167+0000","last_peered":"2024-03-06T07:27:56.028167+0000","last_clean":"2024-03-06T07:27:56.028167+0000","last_became_active":"2024-03-06T07:27:55.013199+0000","last_became_peered":"2024-03-06T07:27:55.013199+0000","last_unstale":"2024-03-06T07:27:56.028167+0000","last_undegraded":"2024-03-06T07:27:56.028167+0000","last_fullsized":"2024-03-06T07:27:56.028167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:13.574066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021899+0000","last_change":"2024-03-06T07:27:54.014875+0000","last_active":"2024-03-06T07:27:56.021899+0000","last_peered":"2024-03-06T07:27:56.021899+0000","last_clean":"2024-03-06T07:27:56.021899+0000","last_became_active":"2024-03-06T07:27:54.014526+0000","last_became_peered":"2024-03-06T07:27:54.014526+0000","last_unstale":"2024-03-06T07:27:56.021899+0000","last_undegraded":"2024-03-06T07:27:56.021899+0000","last_fullsized":"2024-03-06T07:27:56.021899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514018+0000","last_change":"2024-03-06T07:27:55.017432+0000","last_active":"2024-03-06T07:27:56.514018+0000","last_peered":"2024-03-06T07:27:56.514018+0000","last_clean":"2024-03-06T07:27:56.514018+0000","last_became_active":"2024-03-06T07:27:55.015135+0000","last_became_peered":"2024-03-06T07:27:55.015135+0000","last_unstale":"2024-03-06T07:27:56.514018+0000","last_undegraded":"2024-03-06T07:27:56.514018+0000","last_fullsized":"2024-03-06T07:27:56.514018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026039+0000","last_change":"2024-03-06T07:27:54.023420+0000","last_active":"2024-03-06T07:27:56.026039+0000","last_peered":"2024-03-06T07:27:56.026039+0000","last_clean":"2024-03-06T07:27:56.026039+0000","last_became_active":"2024-03-06T07:27:54.022665+0000","last_became_peered":"2024-03-06T07:27:54.022665+0000","last_unstale":"2024-03-06T07:27:56.026039+0000","last_undegraded":"2024-03-06T07:27:56.026039+0000","last_fullsized":"2024-03-06T07:27:56.026039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021422+0000","last_change":"2024-03-06T07:27:54.015366+0000","last_active":"2024-03-06T07:27:56.021422+0000","last_peered":"2024-03-06T07:27:56.021422+0000","last_clean":"2024-03-06T07:27:56.021422+0000","last_became_active":"2024-03-06T07:27:54.015155+0000","last_became_peered":"2024-03-06T07:27:54.015155+0000","last_unstale":"2024-03-06T07:27:56.021422+0000","last_undegraded":"2024-03-06T07:27:56.021422+0000","last_fullsized":"2024-03-06T07:27:56.021422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:04.758569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021171+0000","last_change":"2024-03-06T07:27:55.029914+0000","last_active":"2024-03-06T07:27:56.021171+0000","last_peered":"2024-03-06T07:27:56.021171+0000","last_clean":"2024-03-06T07:27:56.021171+0000","last_became_active":"2024-03-06T07:27:55.029599+0000","last_became_peered":"2024-03-06T07:27:55.029599+0000","last_unstale":"2024-03-06T07:27:56.021171+0000","last_undegraded":"2024-03-06T07:27:56.021171+0000","last_fullsized":"2024-03-06T07:27:56.021171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516973+0000","last_change":"2024-03-06T07:27:54.023284+0000","last_active":"2024-03-06T07:27:56.516973+0000","last_peered":"2024-03-06T07:27:56.516973+0000","last_clean":"2024-03-06T07:27:56.516973+0000","last_became_active":"2024-03-06T07:27:54.022909+0000","last_became_peered":"2024-03-06T07:27:54.022909+0000","last_unstale":"2024-03-06T07:27:56.516973+0000","last_undegraded":"2024-03-06T07:27:56.516973+0000","last_fullsized":"2024-03-06T07:27:56.516973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.016848+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.016590+0000","last_became_peered":"2024-03-06T07:27:55.016590+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028999+0000","last_change":"2024-03-06T07:27:54.025189+0000","last_active":"2024-03-06T07:27:56.028999+0000","last_peered":"2024-03-06T07:27:56.028999+0000","last_clean":"2024-03-06T07:27:56.028999+0000","last_became_active":"2024-03-06T07:27:54.024875+0000","last_became_peered":"2024-03-06T07:27:54.024875+0000","last_unstale":"2024-03-06T07:27:56.028999+0000","last_undegraded":"2024-03-06T07:27:56.028999+0000","last_fullsized":"2024-03-06T07:27:56.028999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519254+0000","last_change":"2024-03-06T07:27:55.018921+0000","last_active":"2024-03-06T07:27:56.519254+0000","last_peered":"2024-03-06T07:27:56.519254+0000","last_clean":"2024-03-06T07:27:56.519254+0000","last_became_active":"2024-03-06T07:27:55.018597+0000","last_became_peered":"2024-03-06T07:27:55.018597+0000","last_unstale":"2024-03-06T07:27:56.519254+0000","last_undegraded":"2024-03-06T07:27:56.519254+0000","last_fullsized":"2024-03-06T07:27:56.519254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519090+0000","last_change":"2024-03-06T07:27:54.025386+0000","last_active":"2024-03-06T07:27:56.519090+0000","last_peered":"2024-03-06T07:27:56.519090+0000","last_clean":"2024-03-06T07:27:56.519090+0000","last_became_active":"2024-03-06T07:27:54.020646+0000","last_became_peered":"2024-03-06T07:27:54.020646+0000","last_unstale":"2024-03-06T07:27:56.519090+0000","last_undegraded":"2024-03-06T07:27:56.519090+0000","last_fullsized":"2024-03-06T07:27:56.519090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514919+0000","last_change":"2024-03-06T07:27:55.022924+0000","last_active":"2024-03-06T07:27:56.514919+0000","last_peered":"2024-03-06T07:27:56.514919+0000","last_clean":"2024-03-06T07:27:56.514919+0000","last_became_active":"2024-03-06T07:27:55.022605+0000","last_became_peered":"2024-03-06T07:27:55.022605+0000","last_unstale":"2024-03-06T07:27:56.514919+0000","last_undegraded":"2024-03-06T07:27:56.514919+0000","last_fullsized":"2024-03-06T07:27:56.514919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019719+0000","last_change":"2024-03-06T07:27:54.012324+0000","last_active":"2024-03-06T07:27:56.019719+0000","last_peered":"2024-03-06T07:27:56.019719+0000","last_clean":"2024-03-06T07:27:56.019719+0000","last_became_active":"2024-03-06T07:27:54.012025+0000","last_became_peered":"2024-03-06T07:27:54.012025+0000","last_unstale":"2024-03-06T07:27:56.019719+0000","last_undegraded":"2024-03-06T07:27:56.019719+0000","last_fullsized":"2024-03-06T07:27:56.019719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029716+0000","last_change":"2024-03-06T07:27:55.028572+0000","last_active":"2024-03-06T07:27:56.029716+0000","last_peered":"2024-03-06T07:27:56.029716+0000","last_clean":"2024-03-06T07:27:56.029716+0000","last_became_active":"2024-03-06T07:27:55.026783+0000","last_became_peered":"2024-03-06T07:27:55.026783+0000","last_unstale":"2024-03-06T07:27:56.029716+0000","last_undegraded":"2024-03-06T07:27:56.029716+0000","last_fullsized":"2024-03-06T07:27:56.029716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:43.960764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027082+0000","last_change":"2024-03-06T07:27:54.011288+0000","last_active":"2024-03-06T07:27:56.027082+0000","last_peered":"2024-03-06T07:27:56.027082+0000","last_clean":"2024-03-06T07:27:56.027082+0000","last_became_active":"2024-03-06T07:27:54.010900+0000","last_became_peered":"2024-03-06T07:27:54.010900+0000","last_unstale":"2024-03-06T07:27:56.027082+0000","last_undegraded":"2024-03-06T07:27:56.027082+0000","last_fullsized":"2024-03-06T07:27:56.027082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515920+0000","last_change":"2024-03-06T07:27:55.035258+0000","last_active":"2024-03-06T07:27:56.515920+0000","last_peered":"2024-03-06T07:27:56.515920+0000","last_clean":"2024-03-06T07:27:56.515920+0000","last_became_active":"2024-03-06T07:27:55.034985+0000","last_became_peered":"2024-03-06T07:27:55.034985+0000","last_unstale":"2024-03-06T07:27:56.515920+0000","last_undegraded":"2024-03-06T07:27:56.515920+0000","last_fullsized":"2024-03-06T07:27:56.515920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513956+0000","last_change":"2024-03-06T07:27:54.019343+0000","last_active":"2024-03-06T07:27:56.513956+0000","last_peered":"2024-03-06T07:27:56.513956+0000","last_clean":"2024-03-06T07:27:56.513956+0000","last_became_active":"2024-03-06T07:27:54.019083+0000","last_became_peered":"2024-03-06T07:27:54.019083+0000","last_unstale":"2024-03-06T07:27:56.513956+0000","last_undegraded":"2024-03-06T07:27:56.513956+0000","last_fullsized":"2024-03-06T07:27:56.513956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:12.617582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.028252+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.027919+0000","last_became_peered":"2024-03-06T07:27:55.027919+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:57.800588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517776+0000","last_change":"2024-03-06T07:27:54.015808+0000","last_active":"2024-03-06T07:27:56.517776+0000","last_peered":"2024-03-06T07:27:56.517776+0000","last_clean":"2024-03-06T07:27:56.517776+0000","last_became_active":"2024-03-06T07:27:54.015279+0000","last_became_peered":"2024-03-06T07:27:54.015279+0000","last_unstale":"2024-03-06T07:27:56.517776+0000","last_undegraded":"2024-03-06T07:27:56.517776+0000","last_fullsized":"2024-03-06T07:27:56.517776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516046+0000","last_change":"2024-03-06T07:27:55.015432+0000","last_active":"2024-03-06T07:27:56.516046+0000","last_peered":"2024-03-06T07:27:56.516046+0000","last_clean":"2024-03-06T07:27:56.516046+0000","last_became_active":"2024-03-06T07:27:55.015132+0000","last_became_peered":"2024-03-06T07:27:55.015132+0000","last_unstale":"2024-03-06T07:27:56.516046+0000","last_undegraded":"2024-03-06T07:27:56.516046+0000","last_fullsized":"2024-03-06T07:27:56.516046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026103+0000","last_change":"2024-03-06T07:27:54.020993+0000","last_active":"2024-03-06T07:27:56.026103+0000","last_peered":"2024-03-06T07:27:56.026103+0000","last_clean":"2024-03-06T07:27:56.026103+0000","last_became_active":"2024-03-06T07:27:54.020618+0000","last_became_peered":"2024-03-06T07:27:54.020618+0000","last_unstale":"2024-03-06T07:27:56.026103+0000","last_undegraded":"2024-03-06T07:27:56.026103+0000","last_fullsized":"2024-03-06T07:27:56.026103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026185+0000","last_change":"2024-03-06T07:27:55.022089+0000","last_active":"2024-03-06T07:27:56.026185+0000","last_peered":"2024-03-06T07:27:56.026185+0000","last_clean":"2024-03-06T07:27:56.026185+0000","last_became_active":"2024-03-06T07:27:55.021113+0000","last_became_peered":"2024-03-06T07:27:55.021113+0000","last_unstale":"2024-03-06T07:27:56.026185+0000","last_undegraded":"2024-03-06T07:27:56.026185+0000","last_fullsized":"2024-03-06T07:27:56.026185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514788+0000","last_change":"2024-03-06T07:27:54.002389+0000","last_active":"2024-03-06T07:27:56.514788+0000","last_peered":"2024-03-06T07:27:56.514788+0000","last_clean":"2024-03-06T07:27:56.514788+0000","last_became_active":"2024-03-06T07:27:54.001928+0000","last_became_peered":"2024-03-06T07:27:54.001928+0000","last_unstale":"2024-03-06T07:27:56.514788+0000","last_undegraded":"2024-03-06T07:27:56.514788+0000","last_fullsized":"2024-03-06T07:27:56.514788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515180+0000","last_change":"2024-03-06T07:27:55.018703+0000","last_active":"2024-03-06T07:27:56.515180+0000","last_peered":"2024-03-06T07:27:56.515180+0000","last_clean":"2024-03-06T07:27:56.515180+0000","last_became_active":"2024-03-06T07:27:55.018315+0000","last_became_peered":"2024-03-06T07:27:55.018315+0000","last_unstale":"2024-03-06T07:27:56.515180+0000","last_undegraded":"2024-03-06T07:27:56.515180+0000","last_fullsized":"2024-03-06T07:27:56.515180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029303+0000","last_change":"2024-03-06T07:27:54.015632+0000","last_active":"2024-03-06T07:27:56.029303+0000","last_peered":"2024-03-06T07:27:56.029303+0000","last_clean":"2024-03-06T07:27:56.029303+0000","last_became_active":"2024-03-06T07:27:54.014810+0000","last_became_peered":"2024-03-06T07:27:54.014810+0000","last_unstale":"2024-03-06T07:27:56.029303+0000","last_undegraded":"2024-03-06T07:27:56.029303+0000","last_fullsized":"2024-03-06T07:27:56.029303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:09.001445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516849+0000","last_change":"2024-03-06T07:27:55.022324+0000","last_active":"2024-03-06T07:27:56.516849+0000","last_peered":"2024-03-06T07:27:56.516849+0000","last_clean":"2024-03-06T07:27:56.516849+0000","last_became_active":"2024-03-06T07:27:55.022013+0000","last_became_peered":"2024-03-06T07:27:55.022013+0000","last_unstale":"2024-03-06T07:27:56.516849+0000","last_undegraded":"2024-03-06T07:27:56.516849+0000","last_fullsized":"2024-03-06T07:27:56.516849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515058+0000","last_change":"2024-03-06T07:27:54.005847+0000","last_active":"2024-03-06T07:27:56.515058+0000","last_peered":"2024-03-06T07:27:56.515058+0000","last_clean":"2024-03-06T07:27:56.515058+0000","last_became_active":"2024-03-06T07:27:54.005460+0000","last_became_peered":"2024-03-06T07:27:54.005460+0000","last_unstale":"2024-03-06T07:27:56.515058+0000","last_undegraded":"2024-03-06T07:27:56.515058+0000","last_fullsized":"2024-03-06T07:27:56.515058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517605+0000","last_change":"2024-03-06T07:27:55.021338+0000","last_active":"2024-03-06T07:27:56.517605+0000","last_peered":"2024-03-06T07:27:56.517605+0000","last_clean":"2024-03-06T07:27:56.517605+0000","last_became_active":"2024-03-06T07:27:55.021017+0000","last_became_peered":"2024-03-06T07:27:55.021017+0000","last_unstale":"2024-03-06T07:27:56.517605+0000","last_undegraded":"2024-03-06T07:27:56.517605+0000","last_fullsized":"2024-03-06T07:27:56.517605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:47.579731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019961+0000","last_change":"2024-03-06T07:27:54.022289+0000","last_active":"2024-03-06T07:27:56.019961+0000","last_peered":"2024-03-06T07:27:56.019961+0000","last_clean":"2024-03-06T07:27:56.019961+0000","last_became_active":"2024-03-06T07:27:54.022098+0000","last_became_peered":"2024-03-06T07:27:54.022098+0000","last_unstale":"2024-03-06T07:27:56.019961+0000","last_undegraded":"2024-03-06T07:27:56.019961+0000","last_fullsized":"2024-03-06T07:27:56.019961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518836+0000","last_change":"2024-03-06T07:27:55.018543+0000","last_active":"2024-03-06T07:27:56.518836+0000","last_peered":"2024-03-06T07:27:56.518836+0000","last_clean":"2024-03-06T07:27:56.518836+0000","last_became_active":"2024-03-06T07:27:55.017787+0000","last_became_peered":"2024-03-06T07:27:55.017787+0000","last_unstale":"2024-03-06T07:27:56.518836+0000","last_undegraded":"2024-03-06T07:27:56.518836+0000","last_fullsized":"2024-03-06T07:27:56.518836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515131+0000","last_change":"2024-03-06T07:27:54.024731+0000","last_active":"2024-03-06T07:27:56.515131+0000","last_peered":"2024-03-06T07:27:56.515131+0000","last_clean":"2024-03-06T07:27:56.515131+0000","last_became_active":"2024-03-06T07:27:54.024389+0000","last_became_peered":"2024-03-06T07:27:54.024389+0000","last_unstale":"2024-03-06T07:27:56.515131+0000","last_undegraded":"2024-03-06T07:27:56.515131+0000","last_fullsized":"2024-03-06T07:27:56.515131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515569+0000","last_change":"2024-03-06T07:27:55.019440+0000","last_active":"2024-03-06T07:27:56.515569+0000","last_peered":"2024-03-06T07:27:56.515569+0000","last_clean":"2024-03-06T07:27:56.515569+0000","last_became_active":"2024-03-06T07:27:55.019018+0000","last_became_peered":"2024-03-06T07:27:55.019018+0000","last_unstale":"2024-03-06T07:27:56.515569+0000","last_undegraded":"2024-03-06T07:27:56.515569+0000","last_fullsized":"2024-03-06T07:27:56.515569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514828+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:27:56.514828+0000","last_peered":"2024-03-06T07:27:56.514828+0000","last_clean":"2024-03-06T07:27:56.514828+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:27:56.514828+0000","last_undegraded":"2024-03-06T07:27:56.514828+0000","last_fullsized":"2024-03-06T07:27:56.514828+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515546+0000","last_change":"2024-03-06T07:27:54.013591+0000","last_active":"2024-03-06T07:27:56.515546+0000","last_peered":"2024-03-06T07:27:56.515546+0000","last_clean":"2024-03-06T07:27:56.515546+0000","last_became_active":"2024-03-06T07:27:54.012154+0000","last_became_peered":"2024-03-06T07:27:54.012154+0000","last_unstale":"2024-03-06T07:27:56.515546+0000","last_undegraded":"2024-03-06T07:27:56.515546+0000","last_fullsized":"2024-03-06T07:27:56.515546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021505+0000","last_change":"2024-03-06T07:27:55.029404+0000","last_active":"2024-03-06T07:27:56.021505+0000","last_peered":"2024-03-06T07:27:56.021505+0000","last_clean":"2024-03-06T07:27:56.021505+0000","last_became_active":"2024-03-06T07:27:55.028890+0000","last_became_peered":"2024-03-06T07:27:55.028890+0000","last_unstale":"2024-03-06T07:27:56.021505+0000","last_undegraded":"2024-03-06T07:27:56.021505+0000","last_fullsized":"2024-03-06T07:27:56.021505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514470+0000","last_change":"2024-03-06T07:27:54.025195+0000","last_active":"2024-03-06T07:27:56.514470+0000","last_peered":"2024-03-06T07:27:56.514470+0000","last_clean":"2024-03-06T07:27:56.514470+0000","last_became_active":"2024-03-06T07:27:54.024942+0000","last_became_peered":"2024-03-06T07:27:54.024942+0000","last_unstale":"2024-03-06T07:27:56.514470+0000","last_undegraded":"2024-03-06T07:27:56.514470+0000","last_fullsized":"2024-03-06T07:27:56.514470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513738+0000","last_change":"2024-03-06T07:27:55.022642+0000","last_active":"2024-03-06T07:27:56.513738+0000","last_peered":"2024-03-06T07:27:56.513738+0000","last_clean":"2024-03-06T07:27:56.513738+0000","last_became_active":"2024-03-06T07:27:55.021096+0000","last_became_peered":"2024-03-06T07:27:55.021096+0000","last_unstale":"2024-03-06T07:27:56.513738+0000","last_undegraded":"2024-03-06T07:27:56.513738+0000","last_fullsized":"2024-03-06T07:27:56.513738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514360+0000","last_change":"2024-03-06T07:27:54.011028+0000","last_active":"2024-03-06T07:27:56.514360+0000","last_peered":"2024-03-06T07:27:56.514360+0000","last_clean":"2024-03-06T07:27:56.514360+0000","last_became_active":"2024-03-06T07:27:54.010767+0000","last_became_peered":"2024-03-06T07:27:54.010767+0000","last_unstale":"2024-03-06T07:27:56.514360+0000","last_undegraded":"2024-03-06T07:27:56.514360+0000","last_fullsized":"2024-03-06T07:27:56.514360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019932+0000","last_change":"2024-03-06T07:27:55.014190+0000","last_active":"2024-03-06T07:27:56.019932+0000","last_peered":"2024-03-06T07:27:56.019932+0000","last_clean":"2024-03-06T07:27:56.019932+0000","last_became_active":"2024-03-06T07:27:55.013895+0000","last_became_peered":"2024-03-06T07:27:55.013895+0000","last_unstale":"2024-03-06T07:27:56.019932+0000","last_undegraded":"2024-03-06T07:27:56.019932+0000","last_fullsized":"2024-03-06T07:27:56.019932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517915+0000","last_change":"2024-03-06T07:27:54.015537+0000","last_active":"2024-03-06T07:27:56.517915+0000","last_peered":"2024-03-06T07:27:56.517915+0000","last_clean":"2024-03-06T07:27:56.517915+0000","last_became_active":"2024-03-06T07:27:54.014754+0000","last_became_peered":"2024-03-06T07:27:54.014754+0000","last_unstale":"2024-03-06T07:27:56.517915+0000","last_undegraded":"2024-03-06T07:27:56.517915+0000","last_fullsized":"2024-03-06T07:27:56.517915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028699+0000","last_change":"2024-03-06T07:27:55.015839+0000","last_active":"2024-03-06T07:27:56.028699+0000","last_peered":"2024-03-06T07:27:56.028699+0000","last_clean":"2024-03-06T07:27:56.028699+0000","last_became_active":"2024-03-06T07:27:55.015468+0000","last_became_peered":"2024-03-06T07:27:55.015468+0000","last_unstale":"2024-03-06T07:27:56.028699+0000","last_undegraded":"2024-03-06T07:27:56.028699+0000","last_fullsized":"2024-03-06T07:27:56.028699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:41.571023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019749+0000","last_change":"2024-03-06T07:27:54.003778+0000","last_active":"2024-03-06T07:27:56.019749+0000","last_peered":"2024-03-06T07:27:56.019749+0000","last_clean":"2024-03-06T07:27:56.019749+0000","last_became_active":"2024-03-06T07:27:54.003482+0000","last_became_peered":"2024-03-06T07:27:54.003482+0000","last_unstale":"2024-03-06T07:27:56.019749+0000","last_undegraded":"2024-03-06T07:27:56.019749+0000","last_fullsized":"2024-03-06T07:27:56.019749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517164+0000","last_change":"2024-03-06T07:27:55.036670+0000","last_active":"2024-03-06T07:27:56.517164+0000","last_peered":"2024-03-06T07:27:56.517164+0000","last_clean":"2024-03-06T07:27:56.517164+0000","last_became_active":"2024-03-06T07:27:55.036328+0000","last_became_peered":"2024-03-06T07:27:55.036328+0000","last_unstale":"2024-03-06T07:27:56.517164+0000","last_undegraded":"2024-03-06T07:27:56.517164+0000","last_fullsized":"2024-03-06T07:27:56.517164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516091+0000","last_change":"2024-03-06T07:27:54.022591+0000","last_active":"2024-03-06T07:27:56.516091+0000","last_peered":"2024-03-06T07:27:56.516091+0000","last_clean":"2024-03-06T07:27:56.516091+0000","last_became_active":"2024-03-06T07:27:54.022296+0000","last_became_peered":"2024-03-06T07:27:54.022296+0000","last_unstale":"2024-03-06T07:27:56.516091+0000","last_undegraded":"2024-03-06T07:27:56.516091+0000","last_fullsized":"2024-03-06T07:27:56.516091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:21.651372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.010897+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.005758+0000","last_became_peered":"2024-03-06T07:27:55.005758+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020051+0000","last_change":"2024-03-06T07:27:54.018547+0000","last_active":"2024-03-06T07:27:56.020051+0000","last_peered":"2024-03-06T07:27:56.020051+0000","last_clean":"2024-03-06T07:27:56.020051+0000","last_became_active":"2024-03-06T07:27:54.018215+0000","last_became_peered":"2024-03-06T07:27:54.018215+0000","last_unstale":"2024-03-06T07:27:56.020051+0000","last_undegraded":"2024-03-06T07:27:56.020051+0000","last_fullsized":"2024-03-06T07:27:56.020051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021795+0000","last_change":"2024-03-06T07:27:55.013861+0000","last_active":"2024-03-06T07:27:56.021795+0000","last_peered":"2024-03-06T07:27:56.021795+0000","last_clean":"2024-03-06T07:27:56.021795+0000","last_became_active":"2024-03-06T07:27:55.013498+0000","last_became_peered":"2024-03-06T07:27:55.013498+0000","last_unstale":"2024-03-06T07:27:56.021795+0000","last_undegraded":"2024-03-06T07:27:56.021795+0000","last_fullsized":"2024-03-06T07:27:56.021795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029229+0000","last_change":"2024-03-06T07:27:54.015491+0000","last_active":"2024-03-06T07:27:56.029229+0000","last_peered":"2024-03-06T07:27:56.029229+0000","last_clean":"2024-03-06T07:27:56.029229+0000","last_became_active":"2024-03-06T07:27:54.015207+0000","last_became_peered":"2024-03-06T07:27:54.015207+0000","last_unstale":"2024-03-06T07:27:56.029229+0000","last_undegraded":"2024-03-06T07:27:56.029229+0000","last_fullsized":"2024-03-06T07:27:56.029229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514806+0000","last_change":"2024-03-06T07:27:55.021833+0000","last_active":"2024-03-06T07:27:56.514806+0000","last_peered":"2024-03-06T07:27:56.514806+0000","last_clean":"2024-03-06T07:27:56.514806+0000","last_became_active":"2024-03-06T07:27:55.021478+0000","last_became_peered":"2024-03-06T07:27:55.021478+0000","last_unstale":"2024-03-06T07:27:56.514806+0000","last_undegraded":"2024-03-06T07:27:56.514806+0000","last_fullsized":"2024-03-06T07:27:56.514806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514469+0000","last_change":"2024-03-06T07:27:54.024577+0000","last_active":"2024-03-06T07:27:56.514469+0000","last_peered":"2024-03-06T07:27:56.514469+0000","last_clean":"2024-03-06T07:27:56.514469+0000","last_became_active":"2024-03-06T07:27:54.023972+0000","last_became_peered":"2024-03-06T07:27:54.023972+0000","last_unstale":"2024-03-06T07:27:56.514469+0000","last_undegraded":"2024-03-06T07:27:56.514469+0000","last_fullsized":"2024-03-06T07:27:56.514469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.025293+0000","last_change":"2024-03-06T07:27:55.020868+0000","last_active":"2024-03-06T07:27:56.025293+0000","last_peered":"2024-03-06T07:27:56.025293+0000","last_clean":"2024-03-06T07:27:56.025293+0000","last_became_active":"2024-03-06T07:27:55.019789+0000","last_became_peered":"2024-03-06T07:27:55.019789+0000","last_unstale":"2024-03-06T07:27:56.025293+0000","last_undegraded":"2024-03-06T07:27:56.025293+0000","last_fullsized":"2024-03-06T07:27:56.025293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:54.669987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019875+0000","last_change":"2024-03-06T07:27:54.014058+0000","last_active":"2024-03-06T07:27:56.019875+0000","last_peered":"2024-03-06T07:27:56.019875+0000","last_clean":"2024-03-06T07:27:56.019875+0000","last_became_active":"2024-03-06T07:27:54.013803+0000","last_became_peered":"2024-03-06T07:27:54.013803+0000","last_unstale":"2024-03-06T07:27:56.019875+0000","last_undegraded":"2024-03-06T07:27:56.019875+0000","last_fullsized":"2024-03-06T07:27:56.019875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021185+0000","last_change":"2024-03-06T07:27:55.020235+0000","last_active":"2024-03-06T07:27:56.021185+0000","last_peered":"2024-03-06T07:27:56.021185+0000","last_clean":"2024-03-06T07:27:56.021185+0000","last_became_active":"2024-03-06T07:27:55.015366+0000","last_became_peered":"2024-03-06T07:27:55.015366+0000","last_unstale":"2024-03-06T07:27:56.021185+0000","last_undegraded":"2024-03-06T07:27:56.021185+0000","last_fullsized":"2024-03-06T07:27:56.021185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019800+0000","last_change":"2024-03-06T07:27:54.005998+0000","last_active":"2024-03-06T07:27:56.019800+0000","last_peered":"2024-03-06T07:27:56.019800+0000","last_clean":"2024-03-06T07:27:56.019800+0000","last_became_active":"2024-03-06T07:27:54.005646+0000","last_became_peered":"2024-03-06T07:27:54.005646+0000","last_unstale":"2024-03-06T07:27:56.019800+0000","last_undegraded":"2024-03-06T07:27:56.019800+0000","last_fullsized":"2024-03-06T07:27:56.019800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021888+0000","last_change":"2024-03-06T07:27:55.005310+0000","last_active":"2024-03-06T07:27:56.021888+0000","last_peered":"2024-03-06T07:27:56.021888+0000","last_clean":"2024-03-06T07:27:56.021888+0000","last_became_active":"2024-03-06T07:27:55.004647+0000","last_became_peered":"2024-03-06T07:27:55.004647+0000","last_unstale":"2024-03-06T07:27:56.021888+0000","last_undegraded":"2024-03-06T07:27:56.021888+0000","last_fullsized":"2024-03-06T07:27:56.021888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515995+0000","last_change":"2024-03-06T07:27:54.027960+0000","last_active":"2024-03-06T07:27:56.515995+0000","last_peered":"2024-03-06T07:27:56.515995+0000","last_clean":"2024-03-06T07:27:56.515995+0000","last_became_active":"2024-03-06T07:27:54.027317+0000","last_became_peered":"2024-03-06T07:27:54.027317+0000","last_unstale":"2024-03-06T07:27:56.515995+0000","last_undegraded":"2024-03-06T07:27:56.515995+0000","last_fullsized":"2024-03-06T07:27:56.515995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:41.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026502+0000","last_change":"2024-03-06T07:27:55.025331+0000","last_active":"2024-03-06T07:27:56.026502+0000","last_peered":"2024-03-06T07:27:56.026502+0000","last_clean":"2024-03-06T07:27:56.026502+0000","last_became_active":"2024-03-06T07:27:55.024981+0000","last_became_peered":"2024-03-06T07:27:55.024981+0000","last_unstale":"2024-03-06T07:27:56.026502+0000","last_undegraded":"2024-03-06T07:27:56.026502+0000","last_fullsized":"2024-03-06T07:27:56.026502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516878+0000","last_change":"2024-03-06T07:27:55.023997+0000","last_active":"2024-03-06T07:27:56.516878+0000","last_peered":"2024-03-06T07:27:56.516878+0000","last_clean":"2024-03-06T07:27:56.516878+0000","last_became_active":"2024-03-06T07:27:55.023617+0000","last_became_peered":"2024-03-06T07:27:55.023617+0000","last_unstale":"2024-03-06T07:27:56.516878+0000","last_undegraded":"2024-03-06T07:27:56.516878+0000","last_fullsized":"2024-03-06T07:27:56.516878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515609+0000","last_change":"2024-03-06T07:27:54.028923+0000","last_active":"2024-03-06T07:27:56.515609+0000","last_peered":"2024-03-06T07:27:56.515609+0000","last_clean":"2024-03-06T07:27:56.515609+0000","last_became_active":"2024-03-06T07:27:54.028290+0000","last_became_peered":"2024-03-06T07:27:54.028290+0000","last_unstale":"2024-03-06T07:27:56.515609+0000","last_undegraded":"2024-03-06T07:27:56.515609+0000","last_fullsized":"2024-03-06T07:27:56.515609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:10.734558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518611+0000","last_change":"2024-03-06T07:27:55.020150+0000","last_active":"2024-03-06T07:27:56.518611+0000","last_peered":"2024-03-06T07:27:56.518611+0000","last_clean":"2024-03-06T07:27:56.518611+0000","last_became_active":"2024-03-06T07:27:55.019546+0000","last_became_peered":"2024-03-06T07:27:55.019546+0000","last_unstale":"2024-03-06T07:27:56.518611+0000","last_undegraded":"2024-03-06T07:27:56.518611+0000","last_fullsized":"2024-03-06T07:27:56.518611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515741+0000","last_change":"2024-03-06T07:27:54.027548+0000","last_active":"2024-03-06T07:27:56.515741+0000","last_peered":"2024-03-06T07:27:56.515741+0000","last_clean":"2024-03-06T07:27:56.515741+0000","last_became_active":"2024-03-06T07:27:54.027161+0000","last_became_peered":"2024-03-06T07:27:54.027161+0000","last_unstale":"2024-03-06T07:27:56.515741+0000","last_undegraded":"2024-03-06T07:27:56.515741+0000","last_fullsized":"2024-03-06T07:27:56.515741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514542+0000","last_change":"2024-03-06T07:27:55.019647+0000","last_active":"2024-03-06T07:27:56.514542+0000","last_peered":"2024-03-06T07:27:56.514542+0000","last_clean":"2024-03-06T07:27:56.514542+0000","last_became_active":"2024-03-06T07:27:55.018449+0000","last_became_peered":"2024-03-06T07:27:55.018449+0000","last_unstale":"2024-03-06T07:27:56.514542+0000","last_undegraded":"2024-03-06T07:27:56.514542+0000","last_fullsized":"2024-03-06T07:27:56.514542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021792+0000","last_change":"2024-03-06T07:27:54.008978+0000","last_active":"2024-03-06T07:27:56.021792+0000","last_peered":"2024-03-06T07:27:56.021792+0000","last_clean":"2024-03-06T07:27:56.021792+0000","last_became_active":"2024-03-06T07:27:54.004840+0000","last_became_peered":"2024-03-06T07:27:54.004840+0000","last_unstale":"2024-03-06T07:27:56.021792+0000","last_undegraded":"2024-03-06T07:27:56.021792+0000","last_fullsized":"2024-03-06T07:27:56.021792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021485+0000","last_change":"2024-03-06T07:27:55.013247+0000","last_active":"2024-03-06T07:27:56.021485+0000","last_peered":"2024-03-06T07:27:56.021485+0000","last_clean":"2024-03-06T07:27:56.021485+0000","last_became_active":"2024-03-06T07:27:55.012951+0000","last_became_peered":"2024-03-06T07:27:55.012951+0000","last_unstale":"2024-03-06T07:27:56.021485+0000","last_undegraded":"2024-03-06T07:27:56.021485+0000","last_fullsized":"2024-03-06T07:27:56.021485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514798+0000","last_change":"2024-03-06T07:27:54.019750+0000","last_active":"2024-03-06T07:27:56.514798+0000","last_peered":"2024-03-06T07:27:56.514798+0000","last_clean":"2024-03-06T07:27:56.514798+0000","last_became_active":"2024-03-06T07:27:54.019443+0000","last_became_peered":"2024-03-06T07:27:54.019443+0000","last_unstale":"2024-03-06T07:27:56.514798+0000","last_undegraded":"2024-03-06T07:27:56.514798+0000","last_fullsized":"2024-03-06T07:27:56.514798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:43.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:59.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:59.621 INFO:tasks.ceph.mds.c.smithi114.stderr:2024-03-06T07:27:59.618+0000 7f9f81600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:59.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.665+0000 7f7f0d400640 1 Processor -- start 2024-03-06T07:27:59.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.666+0000 7f7f0d400640 1 -- start start 2024-03-06T07:27:59.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.667+0000 7f7f0d400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7f0815fe60 0x7f7f08071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.667+0000 7f7f0d400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0806cef0 con 0x7f7f08144c00 2024-03-06T07:27:59.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.667+0000 7f7f0d400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0806d1d0 con 0x7f7f0815f9d0 2024-03-06T07:27:59.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.667+0000 7f7f0d400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0806d4b0 con 0x7f7f0815fe60 2024-03-06T07:27:59.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f06800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7f08144c00 0x7f7f0806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35954/0 (socket says 172.21.15.114:35954) 2024-03-06T07:27:59.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f06800640 1 -- 172.21.15.114:0/2062801898 learned_addr learned my addr 172.21.15.114:0/2062801898 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:27:59.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f07200640 1 --2- 172.21.15.114:0/2062801898 >> v2:172.21.15.177:3300/0 conn(0x7f7f0815fe60 0x7f7f08071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959396941 0 0) 0x7f7f0806cef0 con 0x7f7f08144c00 2024-03-06T07:27:59.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec003520 con 0x7f7f08144c00 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2699477362 0 0) 0x7f7eec003520 con 0x7f7f08144c00 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 >> v2:172.21.15.177:3300/0 conn(0x7f7f0815fe60 msgr2=0x7f7f08071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 --2- 172.21.15.114:0/2062801898 >> v2:172.21.15.177:3300/0 conn(0x7f7f0815fe60 0x7f7f08071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 >> v1:172.21.15.177:6789/0 conn(0x7f7f0815f9d0 legacy=0x7f7f08070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0806d790 con 0x7f7f08144c00 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.668+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7ef0001670 con 0x7f7f08144c00 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 >> v1:172.21.15.114:6789/0 conn(0x7f7f08144c00 legacy=0x7f7f0806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f05e00640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7eec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f05e00640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eec004800 0x7f7eec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f05e00640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7eec0051c0 0x7f7eec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0806d4b0 con 0x7f7eec0039b0 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec00c060 con 0x7f7eec004800 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec00c260 con 0x7f7eec0051c0 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f06800640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7eec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7efde00640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eec004800 0x7f7eec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f06800640 1 -- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7eec0051c0 msgr2=0x7f7eec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f06800640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7eec0051c0 0x7f7eec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f06800640 1 -- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eec004800 msgr2=0x7f7eec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f06800640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eec004800 0x7f7eec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f06800640 1 -- 172.21.15.114:0/2062801898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec003520 con 0x7f7eec0039b0 2024-03-06T07:27:59.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7f07200640 1 -- 172.21.15.114:0/2062801898 reap_dead start 2024-03-06T07:27:59.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.669+0000 7f7efde00640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7eec004800 0x7f7eec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.671+0000 7f7f06800640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7eec003dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000c210 tx=0x7f7eec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9858a2ad64e03677 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.671+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0007d30 con 0x7f7eec0039b0 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.671+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef0030070 con 0x7f7eec0039b0 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.671+0000 7f7f05e00640 1 -- 172.21.15.114:0/2062801898 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0005d70 con 0x7f7eec0039b0 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.672+0000 7f7f0d400640 1 -- 172.21.15.114:0/2062801898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 msgr2=0x7f7eec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.672+0000 7f7f0d400640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7eec003dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000c210 tx=0x7f7eec005830 comp rx=0 tx=0).stop 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.672+0000 7f7f0d400640 1 -- 172.21.15.114:0/2062801898 shutdown_connections 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.672+0000 7f7f0d400640 1 --2- 172.21.15.114:0/2062801898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7eec003dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.672+0000 7f7f0d400640 1 -- 172.21.15.114:0/2062801898 >> 172.21.15.114:0/2062801898 conn(0x7f7f080b1730 msgr2=0x7f7f080b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.672+0000 7f7f0d400640 1 -- 172.21.15.114:0/2062801898 shutdown_connections 2024-03-06T07:27:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.672+0000 7f7f0d400640 1 -- 172.21.15.114:0/2062801898 wait complete. 2024-03-06T07:27:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.674+0000 7f7f0d400640 1 Processor -- start 2024-03-06T07:27:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.674+0000 7f7f0d400640 1 -- start start 2024-03-06T07:27:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f0d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7f080b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f0d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0813f420 0x7f7f0815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f0d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f080ae9a0 0x7f7f081f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f0d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0806cb50 con 0x7f7eec0039b0 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f0d400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f08160730 con 0x7f7f0813f420 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f0d400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0806ccd0 con 0x7f7f080ae9a0 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f06800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7f080b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f06800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7f080b3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55692/0 (socket says 172.21.15.114:55692) 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.675+0000 7f7f06800640 1 -- 172.21.15.114:0/492169189 learned_addr learned my addr 172.21.15.114:0/492169189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7efde00640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0813f420 0x7f7f0815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f06800640 1 -- 172.21.15.114:0/492169189 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f080ae9a0 msgr2=0x7f7f081f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f06800640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f080ae9a0 0x7f7f081f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f06800640 1 -- 172.21.15.114:0/492169189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0813f420 msgr2=0x7f7f0815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f06800640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0813f420 0x7f7f0815cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f06800640 1 -- 172.21.15.114:0/492169189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080ad900 con 0x7f7eec0039b0 2024-03-06T07:27:59.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f06800640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7f080b3c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000c330 tx=0x7f7ef002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7eff400640 1 -- 172.21.15.114:0/492169189 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0004bf0 con 0x7f7eec0039b0 2024-03-06T07:27:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7eff400640 1 -- 172.21.15.114:0/492169189 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef0000ed0 con 0x7f7eec0039b0 2024-03-06T07:27:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f080adbe0 con 0x7f7eec0039b0 2024-03-06T07:27:59.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7eff400640 1 -- 172.21.15.114:0/492169189 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0007d10 con 0x7f7eec0039b0 2024-03-06T07:27:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.676+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f080ae1a0 con 0x7f7eec0039b0 2024-03-06T07:27:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.678+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0804f330 con 0x7f7eec0039b0 2024-03-06T07:27:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.678+0000 7f7eff400640 1 -- 172.21.15.114:0/492169189 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ef0048020 con 0x7f7eec0039b0 2024-03-06T07:27:59.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.679+0000 7f7eff400640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee406b500 0x7f7ee406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.679+0000 7f7eff400640 1 -- 172.21.15.114:0/492169189 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ef00ab560 con 0x7f7eec0039b0 2024-03-06T07:27:59.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.680+0000 7f7efde00640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee406b500 0x7f7ee406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.680+0000 7f7efde00640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee406b500 0x7f7ee406d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7f081f8110 tx=0x7f7ef8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.684+0000 7f7eff400640 1 -- 172.21.15.114:0/492169189 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ef007a340 con 0x7f7eec0039b0 2024-03-06T07:27:59.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.877+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f08072890 con 0x7f7ee406b500 2024-03-06T07:27:59.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.894+0000 7f7eff400640 1 -- 172.21.15.114:0/492169189 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251950 (secure 0 0 0) 0x7f7f08072890 con 0x7f7ee406b500 2024-03-06T07:27:59.899 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:27:59.901 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:27:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.898+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee406b500 msgr2=0x7f7ee406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.898+0000 7f7f0d400640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee406b500 0x7f7ee406d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7f081f8110 tx=0x7f7ef8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.898+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 msgr2=0x7f7f080b3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.898+0000 7f7f0d400640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7f080b3c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000c330 tx=0x7f7ef002e070 comp rx=0 tx=0).stop 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 shutdown_connections 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f080ae9a0 0x7f7f081f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0813f420 0x7f7f0815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee406b500 0x7f7ee406d9c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 --2- 172.21.15.114:0/492169189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eec0039b0 0x7f7f080b3c60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 >> 172.21.15.114:0/492169189 conn(0x7f7f080b1730 msgr2=0x7f7f0806e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 shutdown_connections 2024-03-06T07:27:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:27:59.899+0000 7f7f0d400640 1 -- 172.21.15.114:0/492169189 wait complete. 2024-03-06T07:27:59.926 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:27:59.100320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001263"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517399+0000","last_change":"2024-03-06T07:27:55.014476+0000","last_active":"2024-03-06T07:27:56.517399+0000","last_peered":"2024-03-06T07:27:56.517399+0000","last_clean":"2024-03-06T07:27:56.517399+0000","last_became_active":"2024-03-06T07:27:55.014169+0000","last_became_peered":"2024-03-06T07:27:55.014169+0000","last_unstale":"2024-03-06T07:27:56.517399+0000","last_undegraded":"2024-03-06T07:27:56.517399+0000","last_fullsized":"2024-03-06T07:27:56.517399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:20.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020046+0000","last_change":"2024-03-06T07:27:55.025640+0000","last_active":"2024-03-06T07:27:56.020046+0000","last_peered":"2024-03-06T07:27:56.020046+0000","last_clean":"2024-03-06T07:27:56.020046+0000","last_became_active":"2024-03-06T07:27:55.025215+0000","last_became_peered":"2024-03-06T07:27:55.025215+0000","last_unstale":"2024-03-06T07:27:56.020046+0000","last_undegraded":"2024-03-06T07:27:56.020046+0000","last_fullsized":"2024-03-06T07:27:56.020046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514320+0000","last_change":"2024-03-06T07:27:55.015505+0000","last_active":"2024-03-06T07:27:56.514320+0000","last_peered":"2024-03-06T07:27:56.514320+0000","last_clean":"2024-03-06T07:27:56.514320+0000","last_became_active":"2024-03-06T07:27:55.015199+0000","last_became_peered":"2024-03-06T07:27:55.015199+0000","last_unstale":"2024-03-06T07:27:56.514320+0000","last_undegraded":"2024-03-06T07:27:56.514320+0000","last_fullsized":"2024-03-06T07:27:56.514320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518415+0000","last_change":"2024-03-06T07:27:55.033897+0000","last_active":"2024-03-06T07:27:56.518415+0000","last_peered":"2024-03-06T07:27:56.518415+0000","last_clean":"2024-03-06T07:27:56.518415+0000","last_became_active":"2024-03-06T07:27:55.032987+0000","last_became_peered":"2024-03-06T07:27:55.032987+0000","last_unstale":"2024-03-06T07:27:56.518415+0000","last_undegraded":"2024-03-06T07:27:56.518415+0000","last_fullsized":"2024-03-06T07:27:56.518415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515678+0000","last_change":"2024-03-06T07:27:55.014597+0000","last_active":"2024-03-06T07:27:56.515678+0000","last_peered":"2024-03-06T07:27:56.515678+0000","last_clean":"2024-03-06T07:27:56.515678+0000","last_became_active":"2024-03-06T07:27:55.014206+0000","last_became_peered":"2024-03-06T07:27:55.014206+0000","last_unstale":"2024-03-06T07:27:56.515678+0000","last_undegraded":"2024-03-06T07:27:56.515678+0000","last_fullsized":"2024-03-06T07:27:56.515678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516836+0000","last_change":"2024-03-06T07:27:55.028575+0000","last_active":"2024-03-06T07:27:56.516836+0000","last_peered":"2024-03-06T07:27:56.516836+0000","last_clean":"2024-03-06T07:27:56.516836+0000","last_became_active":"2024-03-06T07:27:55.027801+0000","last_became_peered":"2024-03-06T07:27:55.027801+0000","last_unstale":"2024-03-06T07:27:56.516836+0000","last_undegraded":"2024-03-06T07:27:56.516836+0000","last_fullsized":"2024-03-06T07:27:56.516836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514995+0000","last_change":"2024-03-06T07:27:55.028417+0000","last_active":"2024-03-06T07:27:56.514995+0000","last_peered":"2024-03-06T07:27:56.514995+0000","last_clean":"2024-03-06T07:27:56.514995+0000","last_became_active":"2024-03-06T07:27:55.028108+0000","last_became_peered":"2024-03-06T07:27:55.028108+0000","last_unstale":"2024-03-06T07:27:56.514995+0000","last_undegraded":"2024-03-06T07:27:56.514995+0000","last_fullsized":"2024-03-06T07:27:56.514995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021025+0000","last_change":"2024-03-06T07:27:55.024045+0000","last_active":"2024-03-06T07:27:56.021025+0000","last_peered":"2024-03-06T07:27:56.021025+0000","last_clean":"2024-03-06T07:27:56.021025+0000","last_became_active":"2024-03-06T07:27:55.023528+0000","last_became_peered":"2024-03-06T07:27:55.023528+0000","last_unstale":"2024-03-06T07:27:56.021025+0000","last_undegraded":"2024-03-06T07:27:56.021025+0000","last_fullsized":"2024-03-06T07:27:56.021025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517632+0000","last_change":"2024-03-06T07:27:55.021303+0000","last_active":"2024-03-06T07:27:56.517632+0000","last_peered":"2024-03-06T07:27:56.517632+0000","last_clean":"2024-03-06T07:27:56.517632+0000","last_became_active":"2024-03-06T07:27:55.020953+0000","last_became_peered":"2024-03-06T07:27:55.020953+0000","last_unstale":"2024-03-06T07:27:56.517632+0000","last_undegraded":"2024-03-06T07:27:56.517632+0000","last_fullsized":"2024-03-06T07:27:56.517632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:30.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027586+0000","last_change":"2024-03-06T07:27:55.019249+0000","last_active":"2024-03-06T07:27:56.027586+0000","last_peered":"2024-03-06T07:27:56.027586+0000","last_clean":"2024-03-06T07:27:56.027586+0000","last_became_active":"2024-03-06T07:27:55.018947+0000","last_became_peered":"2024-03-06T07:27:55.018947+0000","last_unstale":"2024-03-06T07:27:56.027586+0000","last_undegraded":"2024-03-06T07:27:56.027586+0000","last_fullsized":"2024-03-06T07:27:56.027586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027423+0000","last_change":"2024-03-06T07:27:55.023288+0000","last_active":"2024-03-06T07:27:56.027423+0000","last_peered":"2024-03-06T07:27:56.027423+0000","last_clean":"2024-03-06T07:27:56.027423+0000","last_became_active":"2024-03-06T07:27:55.023020+0000","last_became_peered":"2024-03-06T07:27:55.023020+0000","last_unstale":"2024-03-06T07:27:56.027423+0000","last_undegraded":"2024-03-06T07:27:56.027423+0000","last_fullsized":"2024-03-06T07:27:56.027423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029290+0000","last_change":"2024-03-06T07:27:55.014888+0000","last_active":"2024-03-06T07:27:56.029290+0000","last_peered":"2024-03-06T07:27:56.029290+0000","last_clean":"2024-03-06T07:27:56.029290+0000","last_became_active":"2024-03-06T07:27:55.014394+0000","last_became_peered":"2024-03-06T07:27:55.014394+0000","last_unstale":"2024-03-06T07:27:56.029290+0000","last_undegraded":"2024-03-06T07:27:56.029290+0000","last_fullsized":"2024-03-06T07:27:56.029290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:58.566589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021843+0000","last_change":"2024-03-06T07:27:55.014936+0000","last_active":"2024-03-06T07:27:56.021843+0000","last_peered":"2024-03-06T07:27:56.021843+0000","last_clean":"2024-03-06T07:27:56.021843+0000","last_became_active":"2024-03-06T07:27:55.012277+0000","last_became_peered":"2024-03-06T07:27:55.012277+0000","last_unstale":"2024-03-06T07:27:56.021843+0000","last_undegraded":"2024-03-06T07:27:56.021843+0000","last_fullsized":"2024-03-06T07:27:56.021843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.012205+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.011928+0000","last_became_peered":"2024-03-06T07:27:55.011928+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019838+0000","last_change":"2024-03-06T07:27:55.030004+0000","last_active":"2024-03-06T07:27:56.019838+0000","last_peered":"2024-03-06T07:27:56.019838+0000","last_clean":"2024-03-06T07:27:56.019838+0000","last_became_active":"2024-03-06T07:27:55.029632+0000","last_became_peered":"2024-03-06T07:27:55.029632+0000","last_unstale":"2024-03-06T07:27:56.019838+0000","last_undegraded":"2024-03-06T07:27:56.019838+0000","last_fullsized":"2024-03-06T07:27:56.019838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515777+0000","last_change":"2024-03-06T07:27:55.028212+0000","last_active":"2024-03-06T07:27:56.515777+0000","last_peered":"2024-03-06T07:27:56.515777+0000","last_clean":"2024-03-06T07:27:56.515777+0000","last_became_active":"2024-03-06T07:27:55.027899+0000","last_became_peered":"2024-03-06T07:27:55.027899+0000","last_unstale":"2024-03-06T07:27:56.515777+0000","last_undegraded":"2024-03-06T07:27:56.515777+0000","last_fullsized":"2024-03-06T07:27:56.515777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519519+0000","last_change":"2024-03-06T07:27:55.024791+0000","last_active":"2024-03-06T07:27:56.519519+0000","last_peered":"2024-03-06T07:27:56.519519+0000","last_clean":"2024-03-06T07:27:56.519519+0000","last_became_active":"2024-03-06T07:27:55.023356+0000","last_became_peered":"2024-03-06T07:27:55.023356+0000","last_unstale":"2024-03-06T07:27:56.519519+0000","last_undegraded":"2024-03-06T07:27:56.519519+0000","last_fullsized":"2024-03-06T07:27:56.519519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514884+0000","last_change":"2024-03-06T07:27:54.994313+0000","last_active":"2024-03-06T07:27:56.514884+0000","last_peered":"2024-03-06T07:27:56.514884+0000","last_clean":"2024-03-06T07:27:56.514884+0000","last_became_active":"2024-03-06T07:27:54.993537+0000","last_became_peered":"2024-03-06T07:27:54.993537+0000","last_unstale":"2024-03-06T07:27:56.514884+0000","last_undegraded":"2024-03-06T07:27:56.514884+0000","last_fullsized":"2024-03-06T07:27:56.514884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:34.222639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516724+0000","last_change":"2024-03-06T07:27:55.016027+0000","last_active":"2024-03-06T07:27:56.516724+0000","last_peered":"2024-03-06T07:27:56.516724+0000","last_clean":"2024-03-06T07:27:56.516724+0000","last_became_active":"2024-03-06T07:27:55.015462+0000","last_became_peered":"2024-03-06T07:27:55.015462+0000","last_unstale":"2024-03-06T07:27:56.516724+0000","last_undegraded":"2024-03-06T07:27:56.516724+0000","last_fullsized":"2024-03-06T07:27:56.516724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027150+0000","last_change":"2024-03-06T07:27:55.015671+0000","last_active":"2024-03-06T07:27:56.027150+0000","last_peered":"2024-03-06T07:27:56.027150+0000","last_clean":"2024-03-06T07:27:56.027150+0000","last_became_active":"2024-03-06T07:27:55.015352+0000","last_became_peered":"2024-03-06T07:27:55.015352+0000","last_unstale":"2024-03-06T07:27:56.027150+0000","last_undegraded":"2024-03-06T07:27:56.027150+0000","last_fullsized":"2024-03-06T07:27:56.027150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513815+0000","last_change":"2024-03-06T07:27:55.026752+0000","last_active":"2024-03-06T07:27:56.513815+0000","last_peered":"2024-03-06T07:27:56.513815+0000","last_clean":"2024-03-06T07:27:56.513815+0000","last_became_active":"2024-03-06T07:27:55.026382+0000","last_became_peered":"2024-03-06T07:27:55.026382+0000","last_unstale":"2024-03-06T07:27:56.513815+0000","last_undegraded":"2024-03-06T07:27:56.513815+0000","last_fullsized":"2024-03-06T07:27:56.513815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517269+0000","last_change":"2024-03-06T07:27:55.024122+0000","last_active":"2024-03-06T07:27:56.517269+0000","last_peered":"2024-03-06T07:27:56.517269+0000","last_clean":"2024-03-06T07:27:56.517269+0000","last_became_active":"2024-03-06T07:27:55.023796+0000","last_became_peered":"2024-03-06T07:27:55.023796+0000","last_unstale":"2024-03-06T07:27:56.517269+0000","last_undegraded":"2024-03-06T07:27:56.517269+0000","last_fullsized":"2024-03-06T07:27:56.517269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021097+0000","last_change":"2024-03-06T07:27:55.020822+0000","last_active":"2024-03-06T07:27:56.021097+0000","last_peered":"2024-03-06T07:27:56.021097+0000","last_clean":"2024-03-06T07:27:56.021097+0000","last_became_active":"2024-03-06T07:27:55.020406+0000","last_became_peered":"2024-03-06T07:27:55.020406+0000","last_unstale":"2024-03-06T07:27:56.021097+0000","last_undegraded":"2024-03-06T07:27:56.021097+0000","last_fullsized":"2024-03-06T07:27:56.021097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516215+0000","last_change":"2024-03-06T07:27:55.016257+0000","last_active":"2024-03-06T07:27:56.516215+0000","last_peered":"2024-03-06T07:27:56.516215+0000","last_clean":"2024-03-06T07:27:56.516215+0000","last_became_active":"2024-03-06T07:27:55.013454+0000","last_became_peered":"2024-03-06T07:27:55.013454+0000","last_unstale":"2024-03-06T07:27:56.516215+0000","last_undegraded":"2024-03-06T07:27:56.516215+0000","last_fullsized":"2024-03-06T07:27:56.516215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021692+0000","last_change":"2024-03-06T07:27:55.014609+0000","last_active":"2024-03-06T07:27:56.021692+0000","last_peered":"2024-03-06T07:27:56.021692+0000","last_clean":"2024-03-06T07:27:56.021692+0000","last_became_active":"2024-03-06T07:27:55.014270+0000","last_became_peered":"2024-03-06T07:27:55.014270+0000","last_unstale":"2024-03-06T07:27:56.021692+0000","last_undegraded":"2024-03-06T07:27:56.021692+0000","last_fullsized":"2024-03-06T07:27:56.021692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027608+0000","last_change":"2024-03-06T07:27:55.012622+0000","last_active":"2024-03-06T07:27:56.027608+0000","last_peered":"2024-03-06T07:27:56.027608+0000","last_clean":"2024-03-06T07:27:56.027608+0000","last_became_active":"2024-03-06T07:27:55.012268+0000","last_became_peered":"2024-03-06T07:27:55.012268+0000","last_unstale":"2024-03-06T07:27:56.027608+0000","last_undegraded":"2024-03-06T07:27:56.027608+0000","last_fullsized":"2024-03-06T07:27:56.027608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019641+0000","last_change":"2024-03-06T07:27:55.025182+0000","last_active":"2024-03-06T07:27:56.019641+0000","last_peered":"2024-03-06T07:27:56.019641+0000","last_clean":"2024-03-06T07:27:56.019641+0000","last_became_active":"2024-03-06T07:27:55.024874+0000","last_became_peered":"2024-03-06T07:27:55.024874+0000","last_unstale":"2024-03-06T07:27:56.019641+0000","last_undegraded":"2024-03-06T07:27:56.019641+0000","last_fullsized":"2024-03-06T07:27:56.019641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.022062+0000","last_change":"2024-03-06T07:27:55.031727+0000","last_active":"2024-03-06T07:27:56.022062+0000","last_peered":"2024-03-06T07:27:56.022062+0000","last_clean":"2024-03-06T07:27:56.022062+0000","last_became_active":"2024-03-06T07:27:55.029328+0000","last_became_peered":"2024-03-06T07:27:55.029328+0000","last_unstale":"2024-03-06T07:27:56.022062+0000","last_undegraded":"2024-03-06T07:27:56.022062+0000","last_fullsized":"2024-03-06T07:27:56.022062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513894+0000","last_change":"2024-03-06T07:27:55.011251+0000","last_active":"2024-03-06T07:27:56.513894+0000","last_peered":"2024-03-06T07:27:56.513894+0000","last_clean":"2024-03-06T07:27:56.513894+0000","last_became_active":"2024-03-06T07:27:55.010892+0000","last_became_peered":"2024-03-06T07:27:55.010892+0000","last_unstale":"2024-03-06T07:27:56.513894+0000","last_undegraded":"2024-03-06T07:27:56.513894+0000","last_fullsized":"2024-03-06T07:27:56.513894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517664+0000","last_change":"2024-03-06T07:27:55.036627+0000","last_active":"2024-03-06T07:27:56.517664+0000","last_peered":"2024-03-06T07:27:56.517664+0000","last_clean":"2024-03-06T07:27:56.517664+0000","last_became_active":"2024-03-06T07:27:55.036245+0000","last_became_peered":"2024-03-06T07:27:55.036245+0000","last_unstale":"2024-03-06T07:27:56.517664+0000","last_undegraded":"2024-03-06T07:27:56.517664+0000","last_fullsized":"2024-03-06T07:27:56.517664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514454+0000","last_change":"2024-03-06T07:27:55.018254+0000","last_active":"2024-03-06T07:27:56.514454+0000","last_peered":"2024-03-06T07:27:56.514454+0000","last_clean":"2024-03-06T07:27:56.514454+0000","last_became_active":"2024-03-06T07:27:55.017832+0000","last_became_peered":"2024-03-06T07:27:55.017832+0000","last_unstale":"2024-03-06T07:27:56.514454+0000","last_undegraded":"2024-03-06T07:27:56.514454+0000","last_fullsized":"2024-03-06T07:27:56.514454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:11.749502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026113+0000","last_change":"2024-03-06T07:27:55.024465+0000","last_active":"2024-03-06T07:27:56.026113+0000","last_peered":"2024-03-06T07:27:56.026113+0000","last_clean":"2024-03-06T07:27:56.026113+0000","last_became_active":"2024-03-06T07:27:55.024173+0000","last_became_peered":"2024-03-06T07:27:55.024173+0000","last_unstale":"2024-03-06T07:27:56.026113+0000","last_undegraded":"2024-03-06T07:27:56.026113+0000","last_fullsized":"2024-03-06T07:27:56.026113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028167+0000","last_change":"2024-03-06T07:27:55.013597+0000","last_active":"2024-03-06T07:27:56.028167+0000","last_peered":"2024-03-06T07:27:56.028167+0000","last_clean":"2024-03-06T07:27:56.028167+0000","last_became_active":"2024-03-06T07:27:55.013199+0000","last_became_peered":"2024-03-06T07:27:55.013199+0000","last_unstale":"2024-03-06T07:27:56.028167+0000","last_undegraded":"2024-03-06T07:27:56.028167+0000","last_fullsized":"2024-03-06T07:27:56.028167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:13.574066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021899+0000","last_change":"2024-03-06T07:27:54.014875+0000","last_active":"2024-03-06T07:27:56.021899+0000","last_peered":"2024-03-06T07:27:56.021899+0000","last_clean":"2024-03-06T07:27:56.021899+0000","last_became_active":"2024-03-06T07:27:54.014526+0000","last_became_peered":"2024-03-06T07:27:54.014526+0000","last_unstale":"2024-03-06T07:27:56.021899+0000","last_undegraded":"2024-03-06T07:27:56.021899+0000","last_fullsized":"2024-03-06T07:27:56.021899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514018+0000","last_change":"2024-03-06T07:27:55.017432+0000","last_active":"2024-03-06T07:27:56.514018+0000","last_peered":"2024-03-06T07:27:56.514018+0000","last_clean":"2024-03-06T07:27:56.514018+0000","last_became_active":"2024-03-06T07:27:55.015135+0000","last_became_peered":"2024-03-06T07:27:55.015135+0000","last_unstale":"2024-03-06T07:27:56.514018+0000","last_undegraded":"2024-03-06T07:27:56.514018+0000","last_fullsized":"2024-03-06T07:27:56.514018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026039+0000","last_change":"2024-03-06T07:27:54.023420+0000","last_active":"2024-03-06T07:27:56.026039+0000","last_peered":"2024-03-06T07:27:56.026039+0000","last_clean":"2024-03-06T07:27:56.026039+0000","last_became_active":"2024-03-06T07:27:54.022665+0000","last_became_peered":"2024-03-06T07:27:54.022665+0000","last_unstale":"2024-03-06T07:27:56.026039+0000","last_undegraded":"2024-03-06T07:27:56.026039+0000","last_fullsized":"2024-03-06T07:27:56.026039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021422+0000","last_change":"2024-03-06T07:27:54.015366+0000","last_active":"2024-03-06T07:27:56.021422+0000","last_peered":"2024-03-06T07:27:56.021422+0000","last_clean":"2024-03-06T07:27:56.021422+0000","last_became_active":"2024-03-06T07:27:54.015155+0000","last_became_peered":"2024-03-06T07:27:54.015155+0000","last_unstale":"2024-03-06T07:27:56.021422+0000","last_undegraded":"2024-03-06T07:27:56.021422+0000","last_fullsized":"2024-03-06T07:27:56.021422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:04.758569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021171+0000","last_change":"2024-03-06T07:27:55.029914+0000","last_active":"2024-03-06T07:27:56.021171+0000","last_peered":"2024-03-06T07:27:56.021171+0000","last_clean":"2024-03-06T07:27:56.021171+0000","last_became_active":"2024-03-06T07:27:55.029599+0000","last_became_peered":"2024-03-06T07:27:55.029599+0000","last_unstale":"2024-03-06T07:27:56.021171+0000","last_undegraded":"2024-03-06T07:27:56.021171+0000","last_fullsized":"2024-03-06T07:27:56.021171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516973+0000","last_change":"2024-03-06T07:27:54.023284+0000","last_active":"2024-03-06T07:27:56.516973+0000","last_peered":"2024-03-06T07:27:56.516973+0000","last_clean":"2024-03-06T07:27:56.516973+0000","last_became_active":"2024-03-06T07:27:54.022909+0000","last_became_peered":"2024-03-06T07:27:54.022909+0000","last_unstale":"2024-03-06T07:27:56.516973+0000","last_undegraded":"2024-03-06T07:27:56.516973+0000","last_fullsized":"2024-03-06T07:27:56.516973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.016848+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.016590+0000","last_became_peered":"2024-03-06T07:27:55.016590+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028999+0000","last_change":"2024-03-06T07:27:54.025189+0000","last_active":"2024-03-06T07:27:56.028999+0000","last_peered":"2024-03-06T07:27:56.028999+0000","last_clean":"2024-03-06T07:27:56.028999+0000","last_became_active":"2024-03-06T07:27:54.024875+0000","last_became_peered":"2024-03-06T07:27:54.024875+0000","last_unstale":"2024-03-06T07:27:56.028999+0000","last_undegraded":"2024-03-06T07:27:56.028999+0000","last_fullsized":"2024-03-06T07:27:56.028999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519254+0000","last_change":"2024-03-06T07:27:55.018921+0000","last_active":"2024-03-06T07:27:56.519254+0000","last_peered":"2024-03-06T07:27:56.519254+0000","last_clean":"2024-03-06T07:27:56.519254+0000","last_became_active":"2024-03-06T07:27:55.018597+0000","last_became_peered":"2024-03-06T07:27:55.018597+0000","last_unstale":"2024-03-06T07:27:56.519254+0000","last_undegraded":"2024-03-06T07:27:56.519254+0000","last_fullsized":"2024-03-06T07:27:56.519254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519090+0000","last_change":"2024-03-06T07:27:54.025386+0000","last_active":"2024-03-06T07:27:56.519090+0000","last_peered":"2024-03-06T07:27:56.519090+0000","last_clean":"2024-03-06T07:27:56.519090+0000","last_became_active":"2024-03-06T07:27:54.020646+0000","last_became_peered":"2024-03-06T07:27:54.020646+0000","last_unstale":"2024-03-06T07:27:56.519090+0000","last_undegraded":"2024-03-06T07:27:56.519090+0000","last_fullsized":"2024-03-06T07:27:56.519090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514919+0000","last_change":"2024-03-06T07:27:55.022924+0000","last_active":"2024-03-06T07:27:56.514919+0000","last_peered":"2024-03-06T07:27:56.514919+0000","last_clean":"2024-03-06T07:27:56.514919+0000","last_became_active":"2024-03-06T07:27:55.022605+0000","last_became_peered":"2024-03-06T07:27:55.022605+0000","last_unstale":"2024-03-06T07:27:56.514919+0000","last_undegraded":"2024-03-06T07:27:56.514919+0000","last_fullsized":"2024-03-06T07:27:56.514919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019719+0000","last_change":"2024-03-06T07:27:54.012324+0000","last_active":"2024-03-06T07:27:56.019719+0000","last_peered":"2024-03-06T07:27:56.019719+0000","last_clean":"2024-03-06T07:27:56.019719+0000","last_became_active":"2024-03-06T07:27:54.012025+0000","last_became_peered":"2024-03-06T07:27:54.012025+0000","last_unstale":"2024-03-06T07:27:56.019719+0000","last_undegraded":"2024-03-06T07:27:56.019719+0000","last_fullsized":"2024-03-06T07:27:56.019719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029716+0000","last_change":"2024-03-06T07:27:55.028572+0000","last_active":"2024-03-06T07:27:56.029716+0000","last_peered":"2024-03-06T07:27:56.029716+0000","last_clean":"2024-03-06T07:27:56.029716+0000","last_became_active":"2024-03-06T07:27:55.026783+0000","last_became_peered":"2024-03-06T07:27:55.026783+0000","last_unstale":"2024-03-06T07:27:56.029716+0000","last_undegraded":"2024-03-06T07:27:56.029716+0000","last_fullsized":"2024-03-06T07:27:56.029716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:43.960764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027082+0000","last_change":"2024-03-06T07:27:54.011288+0000","last_active":"2024-03-06T07:27:56.027082+0000","last_peered":"2024-03-06T07:27:56.027082+0000","last_clean":"2024-03-06T07:27:56.027082+0000","last_became_active":"2024-03-06T07:27:54.010900+0000","last_became_peered":"2024-03-06T07:27:54.010900+0000","last_unstale":"2024-03-06T07:27:56.027082+0000","last_undegraded":"2024-03-06T07:27:56.027082+0000","last_fullsized":"2024-03-06T07:27:56.027082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515920+0000","last_change":"2024-03-06T07:27:55.035258+0000","last_active":"2024-03-06T07:27:56.515920+0000","last_peered":"2024-03-06T07:27:56.515920+0000","last_clean":"2024-03-06T07:27:56.515920+0000","last_became_active":"2024-03-06T07:27:55.034985+0000","last_became_peered":"2024-03-06T07:27:55.034985+0000","last_unstale":"2024-03-06T07:27:56.515920+0000","last_undegraded":"2024-03-06T07:27:56.515920+0000","last_fullsized":"2024-03-06T07:27:56.515920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513956+0000","last_change":"2024-03-06T07:27:54.019343+0000","last_active":"2024-03-06T07:27:56.513956+0000","last_peered":"2024-03-06T07:27:56.513956+0000","last_clean":"2024-03-06T07:27:56.513956+0000","last_became_active":"2024-03-06T07:27:54.019083+0000","last_became_peered":"2024-03-06T07:27:54.019083+0000","last_unstale":"2024-03-06T07:27:56.513956+0000","last_undegraded":"2024-03-06T07:27:56.513956+0000","last_fullsized":"2024-03-06T07:27:56.513956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:12.617582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.028252+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.027919+0000","last_became_peered":"2024-03-06T07:27:55.027919+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:57.800588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517776+0000","last_change":"2024-03-06T07:27:54.015808+0000","last_active":"2024-03-06T07:27:56.517776+0000","last_peered":"2024-03-06T07:27:56.517776+0000","last_clean":"2024-03-06T07:27:56.517776+0000","last_became_active":"2024-03-06T07:27:54.015279+0000","last_became_peered":"2024-03-06T07:27:54.015279+0000","last_unstale":"2024-03-06T07:27:56.517776+0000","last_undegraded":"2024-03-06T07:27:56.517776+0000","last_fullsized":"2024-03-06T07:27:56.517776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516046+0000","last_change":"2024-03-06T07:27:55.015432+0000","last_active":"2024-03-06T07:27:56.516046+0000","last_peered":"2024-03-06T07:27:56.516046+0000","last_clean":"2024-03-06T07:27:56.516046+0000","last_became_active":"2024-03-06T07:27:55.015132+0000","last_became_peered":"2024-03-06T07:27:55.015132+0000","last_unstale":"2024-03-06T07:27:56.516046+0000","last_undegraded":"2024-03-06T07:27:56.516046+0000","last_fullsized":"2024-03-06T07:27:56.516046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026103+0000","last_change":"2024-03-06T07:27:54.020993+0000","last_active":"2024-03-06T07:27:56.026103+0000","last_peered":"2024-03-06T07:27:56.026103+0000","last_clean":"2024-03-06T07:27:56.026103+0000","last_became_active":"2024-03-06T07:27:54.020618+0000","last_became_peered":"2024-03-06T07:27:54.020618+0000","last_unstale":"2024-03-06T07:27:56.026103+0000","last_undegraded":"2024-03-06T07:27:56.026103+0000","last_fullsized":"2024-03-06T07:27:56.026103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026185+0000","last_change":"2024-03-06T07:27:55.022089+0000","last_active":"2024-03-06T07:27:56.026185+0000","last_peered":"2024-03-06T07:27:56.026185+0000","last_clean":"2024-03-06T07:27:56.026185+0000","last_became_active":"2024-03-06T07:27:55.021113+0000","last_became_peered":"2024-03-06T07:27:55.021113+0000","last_unstale":"2024-03-06T07:27:56.026185+0000","last_undegraded":"2024-03-06T07:27:56.026185+0000","last_fullsized":"2024-03-06T07:27:56.026185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514788+0000","last_change":"2024-03-06T07:27:54.002389+0000","last_active":"2024-03-06T07:27:56.514788+0000","last_peered":"2024-03-06T07:27:56.514788+0000","last_clean":"2024-03-06T07:27:56.514788+0000","last_became_active":"2024-03-06T07:27:54.001928+0000","last_became_peered":"2024-03-06T07:27:54.001928+0000","last_unstale":"2024-03-06T07:27:56.514788+0000","last_undegraded":"2024-03-06T07:27:56.514788+0000","last_fullsized":"2024-03-06T07:27:56.514788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515180+0000","last_change":"2024-03-06T07:27:55.018703+0000","last_active":"2024-03-06T07:27:56.515180+0000","last_peered":"2024-03-06T07:27:56.515180+0000","last_clean":"2024-03-06T07:27:56.515180+0000","last_became_active":"2024-03-06T07:27:55.018315+0000","last_became_peered":"2024-03-06T07:27:55.018315+0000","last_unstale":"2024-03-06T07:27:56.515180+0000","last_undegraded":"2024-03-06T07:27:56.515180+0000","last_fullsized":"2024-03-06T07:27:56.515180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029303+0000","last_change":"2024-03-06T07:27:54.015632+0000","last_active":"2024-03-06T07:27:56.029303+0000","last_peered":"2024-03-06T07:27:56.029303+0000","last_clean":"2024-03-06T07:27:56.029303+0000","last_became_active":"2024-03-06T07:27:54.014810+0000","last_became_peered":"2024-03-06T07:27:54.014810+0000","last_unstale":"2024-03-06T07:27:56.029303+0000","last_undegraded":"2024-03-06T07:27:56.029303+0000","last_fullsized":"2024-03-06T07:27:56.029303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:09.001445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516849+0000","last_change":"2024-03-06T07:27:55.022324+0000","last_active":"2024-03-06T07:27:56.516849+0000","last_peered":"2024-03-06T07:27:56.516849+0000","last_clean":"2024-03-06T07:27:56.516849+0000","last_became_active":"2024-03-06T07:27:55.022013+0000","last_became_peered":"2024-03-06T07:27:55.022013+0000","last_unstale":"2024-03-06T07:27:56.516849+0000","last_undegraded":"2024-03-06T07:27:56.516849+0000","last_fullsized":"2024-03-06T07:27:56.516849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515058+0000","last_change":"2024-03-06T07:27:54.005847+0000","last_active":"2024-03-06T07:27:56.515058+0000","last_peered":"2024-03-06T07:27:56.515058+0000","last_clean":"2024-03-06T07:27:56.515058+0000","last_became_active":"2024-03-06T07:27:54.005460+0000","last_became_peered":"2024-03-06T07:27:54.005460+0000","last_unstale":"2024-03-06T07:27:56.515058+0000","last_undegraded":"2024-03-06T07:27:56.515058+0000","last_fullsized":"2024-03-06T07:27:56.515058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517605+0000","last_change":"2024-03-06T07:27:55.021338+0000","last_active":"2024-03-06T07:27:56.517605+0000","last_peered":"2024-03-06T07:27:56.517605+0000","last_clean":"2024-03-06T07:27:56.517605+0000","last_became_active":"2024-03-06T07:27:55.021017+0000","last_became_peered":"2024-03-06T07:27:55.021017+0000","last_unstale":"2024-03-06T07:27:56.517605+0000","last_undegraded":"2024-03-06T07:27:56.517605+0000","last_fullsized":"2024-03-06T07:27:56.517605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:47.579731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019961+0000","last_change":"2024-03-06T07:27:54.022289+0000","last_active":"2024-03-06T07:27:56.019961+0000","last_peered":"2024-03-06T07:27:56.019961+0000","last_clean":"2024-03-06T07:27:56.019961+0000","last_became_active":"2024-03-06T07:27:54.022098+0000","last_became_peered":"2024-03-06T07:27:54.022098+0000","last_unstale":"2024-03-06T07:27:56.019961+0000","last_undegraded":"2024-03-06T07:27:56.019961+0000","last_fullsized":"2024-03-06T07:27:56.019961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518836+0000","last_change":"2024-03-06T07:27:55.018543+0000","last_active":"2024-03-06T07:27:56.518836+0000","last_peered":"2024-03-06T07:27:56.518836+0000","last_clean":"2024-03-06T07:27:56.518836+0000","last_became_active":"2024-03-06T07:27:55.017787+0000","last_became_peered":"2024-03-06T07:27:55.017787+0000","last_unstale":"2024-03-06T07:27:56.518836+0000","last_undegraded":"2024-03-06T07:27:56.518836+0000","last_fullsized":"2024-03-06T07:27:56.518836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515131+0000","last_change":"2024-03-06T07:27:54.024731+0000","last_active":"2024-03-06T07:27:56.515131+0000","last_peered":"2024-03-06T07:27:56.515131+0000","last_clean":"2024-03-06T07:27:56.515131+0000","last_became_active":"2024-03-06T07:27:54.024389+0000","last_became_peered":"2024-03-06T07:27:54.024389+0000","last_unstale":"2024-03-06T07:27:56.515131+0000","last_undegraded":"2024-03-06T07:27:56.515131+0000","last_fullsized":"2024-03-06T07:27:56.515131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515569+0000","last_change":"2024-03-06T07:27:55.019440+0000","last_active":"2024-03-06T07:27:56.515569+0000","last_peered":"2024-03-06T07:27:56.515569+0000","last_clean":"2024-03-06T07:27:56.515569+0000","last_became_active":"2024-03-06T07:27:55.019018+0000","last_became_peered":"2024-03-06T07:27:55.019018+0000","last_unstale":"2024-03-06T07:27:56.515569+0000","last_undegraded":"2024-03-06T07:27:56.515569+0000","last_fullsized":"2024-03-06T07:27:56.515569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514828+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:27:56.514828+0000","last_peered":"2024-03-06T07:27:56.514828+0000","last_clean":"2024-03-06T07:27:56.514828+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:27:56.514828+0000","last_undegraded":"2024-03-06T07:27:56.514828+0000","last_fullsized":"2024-03-06T07:27:56.514828+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515546+0000","last_change":"2024-03-06T07:27:54.013591+0000","last_active":"2024-03-06T07:27:56.515546+0000","last_peered":"2024-03-06T07:27:56.515546+0000","last_clean":"2024-03-06T07:27:56.515546+0000","last_became_active":"2024-03-06T07:27:54.012154+0000","last_became_peered":"2024-03-06T07:27:54.012154+0000","last_unstale":"2024-03-06T07:27:56.515546+0000","last_undegraded":"2024-03-06T07:27:56.515546+0000","last_fullsized":"2024-03-06T07:27:56.515546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021505+0000","last_change":"2024-03-06T07:27:55.029404+0000","last_active":"2024-03-06T07:27:56.021505+0000","last_peered":"2024-03-06T07:27:56.021505+0000","last_clean":"2024-03-06T07:27:56.021505+0000","last_became_active":"2024-03-06T07:27:55.028890+0000","last_became_peered":"2024-03-06T07:27:55.028890+0000","last_unstale":"2024-03-06T07:27:56.021505+0000","last_undegraded":"2024-03-06T07:27:56.021505+0000","last_fullsized":"2024-03-06T07:27:56.021505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514470+0000","last_change":"2024-03-06T07:27:54.025195+0000","last_active":"2024-03-06T07:27:56.514470+0000","last_peered":"2024-03-06T07:27:56.514470+0000","last_clean":"2024-03-06T07:27:56.514470+0000","last_became_active":"2024-03-06T07:27:54.024942+0000","last_became_peered":"2024-03-06T07:27:54.024942+0000","last_unstale":"2024-03-06T07:27:56.514470+0000","last_undegraded":"2024-03-06T07:27:56.514470+0000","last_fullsized":"2024-03-06T07:27:56.514470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513738+0000","last_change":"2024-03-06T07:27:55.022642+0000","last_active":"2024-03-06T07:27:56.513738+0000","last_peered":"2024-03-06T07:27:56.513738+0000","last_clean":"2024-03-06T07:27:56.513738+0000","last_became_active":"2024-03-06T07:27:55.021096+0000","last_became_peered":"2024-03-06T07:27:55.021096+0000","last_unstale":"2024-03-06T07:27:56.513738+0000","last_undegraded":"2024-03-06T07:27:56.513738+0000","last_fullsized":"2024-03-06T07:27:56.513738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514360+0000","last_change":"2024-03-06T07:27:54.011028+0000","last_active":"2024-03-06T07:27:56.514360+0000","last_peered":"2024-03-06T07:27:56.514360+0000","last_clean":"2024-03-06T07:27:56.514360+0000","last_became_active":"2024-03-06T07:27:54.010767+0000","last_became_peered":"2024-03-06T07:27:54.010767+0000","last_unstale":"2024-03-06T07:27:56.514360+0000","last_undegraded":"2024-03-06T07:27:56.514360+0000","last_fullsized":"2024-03-06T07:27:56.514360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019932+0000","last_change":"2024-03-06T07:27:55.014190+0000","last_active":"2024-03-06T07:27:56.019932+0000","last_peered":"2024-03-06T07:27:56.019932+0000","last_clean":"2024-03-06T07:27:56.019932+0000","last_became_active":"2024-03-06T07:27:55.013895+0000","last_became_peered":"2024-03-06T07:27:55.013895+0000","last_unstale":"2024-03-06T07:27:56.019932+0000","last_undegraded":"2024-03-06T07:27:56.019932+0000","last_fullsized":"2024-03-06T07:27:56.019932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517915+0000","last_change":"2024-03-06T07:27:54.015537+0000","last_active":"2024-03-06T07:27:56.517915+0000","last_peered":"2024-03-06T07:27:56.517915+0000","last_clean":"2024-03-06T07:27:56.517915+0000","last_became_active":"2024-03-06T07:27:54.014754+0000","last_became_peered":"2024-03-06T07:27:54.014754+0000","last_unstale":"2024-03-06T07:27:56.517915+0000","last_undegraded":"2024-03-06T07:27:56.517915+0000","last_fullsized":"2024-03-06T07:27:56.517915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028699+0000","last_change":"2024-03-06T07:27:55.015839+0000","last_active":"2024-03-06T07:27:56.028699+0000","last_peered":"2024-03-06T07:27:56.028699+0000","last_clean":"2024-03-06T07:27:56.028699+0000","last_became_active":"2024-03-06T07:27:55.015468+0000","last_became_peered":"2024-03-06T07:27:55.015468+0000","last_unstale":"2024-03-06T07:27:56.028699+0000","last_undegraded":"2024-03-06T07:27:56.028699+0000","last_fullsized":"2024-03-06T07:27:56.028699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:41.571023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019749+0000","last_change":"2024-03-06T07:27:54.003778+0000","last_active":"2024-03-06T07:27:56.019749+0000","last_peered":"2024-03-06T07:27:56.019749+0000","last_clean":"2024-03-06T07:27:56.019749+0000","last_became_active":"2024-03-06T07:27:54.003482+0000","last_became_peered":"2024-03-06T07:27:54.003482+0000","last_unstale":"2024-03-06T07:27:56.019749+0000","last_undegraded":"2024-03-06T07:27:56.019749+0000","last_fullsized":"2024-03-06T07:27:56.019749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517164+0000","last_change":"2024-03-06T07:27:55.036670+0000","last_active":"2024-03-06T07:27:56.517164+0000","last_peered":"2024-03-06T07:27:56.517164+0000","last_clean":"2024-03-06T07:27:56.517164+0000","last_became_active":"2024-03-06T07:27:55.036328+0000","last_became_peered":"2024-03-06T07:27:55.036328+0000","last_unstale":"2024-03-06T07:27:56.517164+0000","last_undegraded":"2024-03-06T07:27:56.517164+0000","last_fullsized":"2024-03-06T07:27:56.517164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516091+0000","last_change":"2024-03-06T07:27:54.022591+0000","last_active":"2024-03-06T07:27:56.516091+0000","last_peered":"2024-03-06T07:27:56.516091+0000","last_clean":"2024-03-06T07:27:56.516091+0000","last_became_active":"2024-03-06T07:27:54.022296+0000","last_became_peered":"2024-03-06T07:27:54.022296+0000","last_unstale":"2024-03-06T07:27:56.516091+0000","last_undegraded":"2024-03-06T07:27:56.516091+0000","last_fullsized":"2024-03-06T07:27:56.516091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:21.651372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.010897+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.005758+0000","last_became_peered":"2024-03-06T07:27:55.005758+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020051+0000","last_change":"2024-03-06T07:27:54.018547+0000","last_active":"2024-03-06T07:27:56.020051+0000","last_peered":"2024-03-06T07:27:56.020051+0000","last_clean":"2024-03-06T07:27:56.020051+0000","last_became_active":"2024-03-06T07:27:54.018215+0000","last_became_peered":"2024-03-06T07:27:54.018215+0000","last_unstale":"2024-03-06T07:27:56.020051+0000","last_undegraded":"2024-03-06T07:27:56.020051+0000","last_fullsized":"2024-03-06T07:27:56.020051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021795+0000","last_change":"2024-03-06T07:27:55.013861+0000","last_active":"2024-03-06T07:27:56.021795+0000","last_peered":"2024-03-06T07:27:56.021795+0000","last_clean":"2024-03-06T07:27:56.021795+0000","last_became_active":"2024-03-06T07:27:55.013498+0000","last_became_peered":"2024-03-06T07:27:55.013498+0000","last_unstale":"2024-03-06T07:27:56.021795+0000","last_undegraded":"2024-03-06T07:27:56.021795+0000","last_fullsized":"2024-03-06T07:27:56.021795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029229+0000","last_change":"2024-03-06T07:27:54.015491+0000","last_active":"2024-03-06T07:27:56.029229+0000","last_peered":"2024-03-06T07:27:56.029229+0000","last_clean":"2024-03-06T07:27:56.029229+0000","last_became_active":"2024-03-06T07:27:54.015207+0000","last_became_peered":"2024-03-06T07:27:54.015207+0000","last_unstale":"2024-03-06T07:27:56.029229+0000","last_undegraded":"2024-03-06T07:27:56.029229+0000","last_fullsized":"2024-03-06T07:27:56.029229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514806+0000","last_change":"2024-03-06T07:27:55.021833+0000","last_active":"2024-03-06T07:27:56.514806+0000","last_peered":"2024-03-06T07:27:56.514806+0000","last_clean":"2024-03-06T07:27:56.514806+0000","last_became_active":"2024-03-06T07:27:55.021478+0000","last_became_peered":"2024-03-06T07:27:55.021478+0000","last_unstale":"2024-03-06T07:27:56.514806+0000","last_undegraded":"2024-03-06T07:27:56.514806+0000","last_fullsized":"2024-03-06T07:27:56.514806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514469+0000","last_change":"2024-03-06T07:27:54.024577+0000","last_active":"2024-03-06T07:27:56.514469+0000","last_peered":"2024-03-06T07:27:56.514469+0000","last_clean":"2024-03-06T07:27:56.514469+0000","last_became_active":"2024-03-06T07:27:54.023972+0000","last_became_peered":"2024-03-06T07:27:54.023972+0000","last_unstale":"2024-03-06T07:27:56.514469+0000","last_undegraded":"2024-03-06T07:27:56.514469+0000","last_fullsized":"2024-03-06T07:27:56.514469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.025293+0000","last_change":"2024-03-06T07:27:55.020868+0000","last_active":"2024-03-06T07:27:56.025293+0000","last_peered":"2024-03-06T07:27:56.025293+0000","last_clean":"2024-03-06T07:27:56.025293+0000","last_became_active":"2024-03-06T07:27:55.019789+0000","last_became_peered":"2024-03-06T07:27:55.019789+0000","last_unstale":"2024-03-06T07:27:56.025293+0000","last_undegraded":"2024-03-06T07:27:56.025293+0000","last_fullsized":"2024-03-06T07:27:56.025293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:54.669987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019875+0000","last_change":"2024-03-06T07:27:54.014058+0000","last_active":"2024-03-06T07:27:56.019875+0000","last_peered":"2024-03-06T07:27:56.019875+0000","last_clean":"2024-03-06T07:27:56.019875+0000","last_became_active":"2024-03-06T07:27:54.013803+0000","last_became_peered":"2024-03-06T07:27:54.013803+0000","last_unstale":"2024-03-06T07:27:56.019875+0000","last_undegraded":"2024-03-06T07:27:56.019875+0000","last_fullsized":"2024-03-06T07:27:56.019875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021185+0000","last_change":"2024-03-06T07:27:55.020235+0000","last_active":"2024-03-06T07:27:56.021185+0000","last_peered":"2024-03-06T07:27:56.021185+0000","last_clean":"2024-03-06T07:27:56.021185+0000","last_became_active":"2024-03-06T07:27:55.015366+0000","last_became_peered":"2024-03-06T07:27:55.015366+0000","last_unstale":"2024-03-06T07:27:56.021185+0000","last_undegraded":"2024-03-06T07:27:56.021185+0000","last_fullsized":"2024-03-06T07:27:56.021185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019800+0000","last_change":"2024-03-06T07:27:54.005998+0000","last_active":"2024-03-06T07:27:56.019800+0000","last_peered":"2024-03-06T07:27:56.019800+0000","last_clean":"2024-03-06T07:27:56.019800+0000","last_became_active":"2024-03-06T07:27:54.005646+0000","last_became_peered":"2024-03-06T07:27:54.005646+0000","last_unstale":"2024-03-06T07:27:56.019800+0000","last_undegraded":"2024-03-06T07:27:56.019800+0000","last_fullsized":"2024-03-06T07:27:56.019800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021888+0000","last_change":"2024-03-06T07:27:55.005310+0000","last_active":"2024-03-06T07:27:56.021888+0000","last_peered":"2024-03-06T07:27:56.021888+0000","last_clean":"2024-03-06T07:27:56.021888+0000","last_became_active":"2024-03-06T07:27:55.004647+0000","last_became_peered":"2024-03-06T07:27:55.004647+0000","last_unstale":"2024-03-06T07:27:56.021888+0000","last_undegraded":"2024-03-06T07:27:56.021888+0000","last_fullsized":"2024-03-06T07:27:56.021888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515995+0000","last_change":"2024-03-06T07:27:54.027960+0000","last_active":"2024-03-06T07:27:56.515995+0000","last_peered":"2024-03-06T07:27:56.515995+0000","last_clean":"2024-03-06T07:27:56.515995+0000","last_became_active":"2024-03-06T07:27:54.027317+0000","last_became_peered":"2024-03-06T07:27:54.027317+0000","last_unstale":"2024-03-06T07:27:56.515995+0000","last_undegraded":"2024-03-06T07:27:56.515995+0000","last_fullsized":"2024-03-06T07:27:56.515995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:41.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026502+0000","last_change":"2024-03-06T07:27:55.025331+0000","last_active":"2024-03-06T07:27:56.026502+0000","last_peered":"2024-03-06T07:27:56.026502+0000","last_clean":"2024-03-06T07:27:56.026502+0000","last_became_active":"2024-03-06T07:27:55.024981+0000","last_became_peered":"2024-03-06T07:27:55.024981+0000","last_unstale":"2024-03-06T07:27:56.026502+0000","last_undegraded":"2024-03-06T07:27:56.026502+0000","last_fullsized":"2024-03-06T07:27:56.026502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516878+0000","last_change":"2024-03-06T07:27:55.023997+0000","last_active":"2024-03-06T07:27:56.516878+0000","last_peered":"2024-03-06T07:27:56.516878+0000","last_clean":"2024-03-06T07:27:56.516878+0000","last_became_active":"2024-03-06T07:27:55.023617+0000","last_became_peered":"2024-03-06T07:27:55.023617+0000","last_unstale":"2024-03-06T07:27:56.516878+0000","last_undegraded":"2024-03-06T07:27:56.516878+0000","last_fullsized":"2024-03-06T07:27:56.516878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515609+0000","last_change":"2024-03-06T07:27:54.028923+0000","last_active":"2024-03-06T07:27:56.515609+0000","last_peered":"2024-03-06T07:27:56.515609+0000","last_clean":"2024-03-06T07:27:56.515609+0000","last_became_active":"2024-03-06T07:27:54.028290+0000","last_became_peered":"2024-03-06T07:27:54.028290+0000","last_unstale":"2024-03-06T07:27:56.515609+0000","last_undegraded":"2024-03-06T07:27:56.515609+0000","last_fullsized":"2024-03-06T07:27:56.515609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:10.734558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518611+0000","last_change":"2024-03-06T07:27:55.020150+0000","last_active":"2024-03-06T07:27:56.518611+0000","last_peered":"2024-03-06T07:27:56.518611+0000","last_clean":"2024-03-06T07:27:56.518611+0000","last_became_active":"2024-03-06T07:27:55.019546+0000","last_became_peered":"2024-03-06T07:27:55.019546+0000","last_unstale":"2024-03-06T07:27:56.518611+0000","last_undegraded":"2024-03-06T07:27:56.518611+0000","last_fullsized":"2024-03-06T07:27:56.518611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515741+0000","last_change":"2024-03-06T07:27:54.027548+0000","last_active":"2024-03-06T07:27:56.515741+0000","last_peered":"2024-03-06T07:27:56.515741+0000","last_clean":"2024-03-06T07:27:56.515741+0000","last_became_active":"2024-03-06T07:27:54.027161+0000","last_became_peered":"2024-03-06T07:27:54.027161+0000","last_unstale":"2024-03-06T07:27:56.515741+0000","last_undegraded":"2024-03-06T07:27:56.515741+0000","last_fullsized":"2024-03-06T07:27:56.515741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514542+0000","last_change":"2024-03-06T07:27:55.019647+0000","last_active":"2024-03-06T07:27:56.514542+0000","last_peered":"2024-03-06T07:27:56.514542+0000","last_clean":"2024-03-06T07:27:56.514542+0000","last_became_active":"2024-03-06T07:27:55.018449+0000","last_became_peered":"2024-03-06T07:27:55.018449+0000","last_unstale":"2024-03-06T07:27:56.514542+0000","last_undegraded":"2024-03-06T07:27:56.514542+0000","last_fullsized":"2024-03-06T07:27:56.514542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021792+0000","last_change":"2024-03-06T07:27:54.008978+0000","last_active":"2024-03-06T07:27:56.021792+0000","last_peered":"2024-03-06T07:27:56.021792+0000","last_clean":"2024-03-06T07:27:56.021792+0000","last_became_active":"2024-03-06T07:27:54.004840+0000","last_became_peered":"2024-03-06T07:27:54.004840+0000","last_unstale":"2024-03-06T07:27:56.021792+0000","last_undegraded":"2024-03-06T07:27:56.021792+0000","last_fullsized":"2024-03-06T07:27:56.021792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021485+0000","last_change":"2024-03-06T07:27:55.013247+0000","last_active":"2024-03-06T07:27:56.021485+0000","last_peered":"2024-03-06T07:27:56.021485+0000","last_clean":"2024-03-06T07:27:56.021485+0000","last_became_active":"2024-03-06T07:27:55.012951+0000","last_became_peered":"2024-03-06T07:27:55.012951+0000","last_unstale":"2024-03-06T07:27:56.021485+0000","last_undegraded":"2024-03-06T07:27:56.021485+0000","last_fullsized":"2024-03-06T07:27:56.021485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514798+0000","last_change":"2024-03-06T07:27:54.019750+0000","last_active":"2024-03-06T07:27:56.514798+0000","last_peered":"2024-03-06T07:27:56.514798+0000","last_clean":"2024-03-06T07:27:56.514798+0000","last_became_active":"2024-03-06T07:27:54.019443+0000","last_became_peered":"2024-03-06T07:27:54.019443+0000","last_unstale":"2024-03-06T07:27:56.514798+0000","last_undegraded":"2024-03-06T07:27:56.514798+0000","last_fullsized":"2024-03-06T07:27:56.514798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:43.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:59.929 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:59.930 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:27:59.930 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:27:59.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:27:59.930 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:28:00.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.167+0000 7f37a4600640 1 Processor -- start 2024-03-06T07:28:00.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.167+0000 7f37a4600640 1 -- start start 2024-03-06T07:28:00.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.168+0000 7f37a4600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c06d1f0 0x7f379c1627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.168+0000 7f37a4600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f379c06c2e0 0x7f379c15d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.168+0000 7f37a4600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0d0010 con 0x7f379c0cdd80 2024-03-06T07:28:00.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.168+0000 7f37a4600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379c0d0ec0 con 0x7f379c06c2e0 2024-03-06T07:28:00.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.168+0000 7f37a4600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0d02b0 con 0x7f379c06d1f0 2024-03-06T07:28:00.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.169+0000 7f37a1200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c06d1f0 0x7f379c1627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.169+0000 7f37a2600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f379c06c2e0 0x7f379c15d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.169+0000 7f37a1200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c06d1f0 0x7f379c1627a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58924/0 (socket says 172.21.15.114:58924) 2024-03-06T07:28:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.169+0000 7f37a1200640 1 -- 172.21.15.114:0/1731364615 learned_addr learned my addr 172.21.15.114:0/1731364615 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.170+0000 7f37a1200640 1 -- 172.21.15.114:0/1731364615 >> v2:172.21.15.177:3300/0 conn(0x7f379c06c2e0 msgr2=0x7f379c15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.170+0000 7f37a1200640 1 --2- 172.21.15.114:0/1731364615 >> v2:172.21.15.177:3300/0 conn(0x7f379c06c2e0 0x7f379c15d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.170+0000 7f37a1200640 1 -- 172.21.15.114:0/1731364615 >> v1:172.21.15.177:6789/0 conn(0x7f379c0cdd80 legacy=0x7f379c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.170+0000 7f37a1200640 1 -- 172.21.15.114:0/1731364615 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c0d0590 con 0x7f379c06d1f0 2024-03-06T07:28:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.170+0000 7f37a2600640 1 --2- 172.21.15.114:0/1731364615 >> v2:172.21.15.177:3300/0 conn(0x7f379c06c2e0 0x7f379c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.171+0000 7f37a1200640 1 --2- 172.21.15.114:0/1731364615 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c06d1f0 0x7f379c1627a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0666b5dc68e933f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.171+0000 7f379be00640 1 -- 172.21.15.114:0/1731364615 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3788034070 con 0x7f379c06d1f0 2024-03-06T07:28:00.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.171+0000 7f379be00640 1 -- 172.21.15.114:0/1731364615 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3788030840 con 0x7f379c06d1f0 2024-03-06T07:28:00.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.171+0000 7f379be00640 1 -- 172.21.15.114:0/1731364615 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3788030c10 con 0x7f379c06d1f0 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 -- 172.21.15.114:0/1731364615 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c06d1f0 msgr2=0x7f379c1627a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 --2- 172.21.15.114:0/1731364615 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c06d1f0 0x7f379c1627a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 -- 172.21.15.114:0/1731364615 shutdown_connections 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 --2- 172.21.15.114:0/1731364615 >> v2:172.21.15.177:3300/0 conn(0x7f379c06c2e0 0x7f379c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 --2- 172.21.15.114:0/1731364615 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c06d1f0 0x7f379c1627a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 -- 172.21.15.114:0/1731364615 >> 172.21.15.114:0/1731364615 conn(0x7f379c0b3870 msgr2=0x7f379c0b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 -- 172.21.15.114:0/1731364615 shutdown_connections 2024-03-06T07:28:00.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.172+0000 7f37a4600640 1 -- 172.21.15.114:0/1731364615 wait complete. 2024-03-06T07:28:00.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.174+0000 7f37a4600640 1 Processor -- start 2024-03-06T07:28:00.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.174+0000 7f37a4600640 1 -- start start 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a4600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f379c06c2e0 0x7f379c0b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a4600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f379c06d1f0 0x7f379c150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a4600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c0cdd80 0x7f379c1512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a4600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0ce160 con 0x7f379c06c2e0 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a4600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0d0b50 con 0x7f379c06d1f0 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a4600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0d1350 con 0x7f379c0cdd80 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f379c06c2e0 0x7f379c0b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f379c06c2e0 0x7f379c0b6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55696/0 (socket says 172.21.15.114:55696) 2024-03-06T07:28:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a1c00640 1 -- 172.21.15.114:0/425419608 learned_addr learned my addr 172.21.15.114:0/425419608 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:00.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a1200640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f379c06d1f0 0x7f379c150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a2600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c0cdd80 0x7f379c1512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a1c00640 1 -- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c0cdd80 msgr2=0x7f379c1512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.176+0000 7f37a1c00640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c0cdd80 0x7f379c1512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f37a1c00640 1 -- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f379c06d1f0 msgr2=0x7f379c150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f37a1c00640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f379c06d1f0 0x7f379c150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f37a1c00640 1 -- 172.21.15.114:0/425419608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c0c9d20 con 0x7f379c06c2e0 2024-03-06T07:28:00.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f37a1c00640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f379c06c2e0 0x7f379c0b6980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f379000c3f0 tx=0x7f379000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f3799600640 1 -- 172.21.15.114:0/425419608 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37900052d0 con 0x7f379c06c2e0 2024-03-06T07:28:00.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f3799600640 1 -- 172.21.15.114:0/425419608 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790005490 con 0x7f379c06c2e0 2024-03-06T07:28:00.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c0ca000 con 0x7f379c06c2e0 2024-03-06T07:28:00.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f3799600640 1 -- 172.21.15.114:0/425419608 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790007260 con 0x7f379c06c2e0 2024-03-06T07:28:00.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.177+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c0b6ec0 con 0x7f379c06c2e0 2024-03-06T07:28:00.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.178+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379c06bbc0 con 0x7f379c06c2e0 2024-03-06T07:28:00.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.179+0000 7f3799600640 1 -- 172.21.15.114:0/425419608 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3790007400 con 0x7f379c06c2e0 2024-03-06T07:28:00.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.180+0000 7f3799600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f377c06b4d0 0x7f377c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.180+0000 7f3799600640 1 -- 172.21.15.114:0/425419608 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3790089690 con 0x7f379c06c2e0 2024-03-06T07:28:00.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.181+0000 7f37a1200640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f377c06b4d0 0x7f377c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.181+0000 7f37a1200640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f377c06b4d0 0x7f377c06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3788036000 tx=0x7f3788008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.189+0000 7f3799600640 1 -- 172.21.15.114:0/425419608 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3790057e20 con 0x7f379c06c2e0 2024-03-06T07:28:00.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.435+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f379c1519f0 con 0x7f379c06c2e0 2024-03-06T07:28:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.444+0000 7f3799600640 1 -- 172.21.15.114:0/425419608 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171789 (secure 0 0 0) 0x7f379c1519f0 con 0x7f379c06c2e0 2024-03-06T07:28:00.450 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.448+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f377c06b4d0 msgr2=0x7f377c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.448+0000 7f37a4600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f377c06b4d0 0x7f377c06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3788036000 tx=0x7f3788008030 comp rx=0 tx=0).stop 2024-03-06T07:28:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.448+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f379c06c2e0 msgr2=0x7f379c0b6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.448+0000 7f37a4600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f379c06c2e0 0x7f379c0b6980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f379000c3f0 tx=0x7f379000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 shutdown_connections 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f379c0cdd80 0x7f379c1512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f379c06d1f0 0x7f379c150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f377c06b4d0 0x7f377c06d990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 --2- 172.21.15.114:0/425419608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f379c06c2e0 0x7f379c0b6980 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 >> 172.21.15.114:0/425419608 conn(0x7f379c0b3870 msgr2=0x7f379c070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 shutdown_connections 2024-03-06T07:28:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.449+0000 7f37a4600640 1 -- 172.21.15.114:0/425419608 wait complete. 2024-03-06T07:28:00.468 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3967440666},{"type":"v1","addr":"172.21.15.114:6833","nonce":3967440666}]},"active_addr":"172.21.15.114:6833/3967440666","active_change":"2024-03-06T07:27:35.075886+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4237165419}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":336081465}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":109358391}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":150756841}]}]} 2024-03-06T07:28:00.472 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:28:00.472 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:28:00.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.693+0000 7f384fe00640 1 Processor -- start 2024-03-06T07:28:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.693+0000 7f384fe00640 1 -- start start 2024-03-06T07:28:00.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 0x7f38500d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38500d1e90 0x7f38500d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3850161960 con 0x7f38500d28a0 2024-03-06T07:28:00.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38500cf6f0 con 0x7f38500d1e90 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38500cf9d0 con 0x7f3850161580 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f3846000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38500d1e90 0x7f38500d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384f400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f38500d28a0 0x7f38500d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35960/0 (socket says 172.21.15.114:35960) 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384f400640 1 -- 172.21.15.114:0/2211822122 learned_addr learned my addr 172.21.15.114:0/2211822122 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.696+0000 7f384ea00640 1 --2- 172.21.15.114:0/2211822122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 0x7f38500d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.697+0000 7f3846000640 1 -- 172.21.15.114:0/2211822122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 msgr2=0x7f38500d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.697+0000 7f3846000640 1 --2- 172.21.15.114:0/2211822122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 0x7f38500d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.697+0000 7f3846000640 1 -- 172.21.15.114:0/2211822122 >> v1:172.21.15.114:6789/0 conn(0x7f38500d28a0 legacy=0x7f38500d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.697+0000 7f3846000640 1 -- 172.21.15.114:0/2211822122 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38500cfcb0 con 0x7f38500d1e90 2024-03-06T07:28:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.697+0000 7f384e000640 1 -- 172.21.15.114:0/2211822122 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379746839 0 0) 0x7f3850161960 con 0x7f38500d28a0 2024-03-06T07:28:00.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.698+0000 7f3846000640 1 --2- 172.21.15.114:0/2211822122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d1e90 0x7f38500d2360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f38380097a0 tx=0x7f383802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c09742d9130532 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.698+0000 7f384e000640 1 -- 172.21.15.114:0/2211822122 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838035020 con 0x7f38500d1e90 2024-03-06T07:28:00.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.698+0000 7f384e000640 1 -- 172.21.15.114:0/2211822122 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3838031830 con 0x7f38500d1e90 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.698+0000 7f384e000640 1 -- 172.21.15.114:0/2211822122 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838031c00 con 0x7f38500d1e90 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 -- 172.21.15.114:0/2211822122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d1e90 msgr2=0x7f38500d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 --2- 172.21.15.114:0/2211822122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d1e90 0x7f38500d2360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f38380097a0 tx=0x7f383802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 -- 172.21.15.114:0/2211822122 shutdown_connections 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 --2- 172.21.15.114:0/2211822122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d1e90 0x7f38500d2360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 --2- 172.21.15.114:0/2211822122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 0x7f38500d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 -- 172.21.15.114:0/2211822122 >> 172.21.15.114:0/2211822122 conn(0x7f38500d3d80 msgr2=0x7f38500d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 -- 172.21.15.114:0/2211822122 shutdown_connections 2024-03-06T07:28:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.699+0000 7f384fe00640 1 -- 172.21.15.114:0/2211822122 wait complete. 2024-03-06T07:28:00.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.700+0000 7f384fe00640 1 Processor -- start 2024-03-06T07:28:00.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.701+0000 7f384fe00640 1 -- start start 2024-03-06T07:28:00.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.701+0000 7f384fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38500d1e90 0x7f385015dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.701+0000 7f384fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d28a0 0x7f385014d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.701+0000 7f384fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 0x7f385014ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.701+0000 7f384fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38500cf3a0 con 0x7f38500d28a0 2024-03-06T07:28:00.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.701+0000 7f384fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38500b39a0 con 0x7f38500d1e90 2024-03-06T07:28:00.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.701+0000 7f384fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38500b4ed0 con 0x7f3850161580 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d28a0 0x7f385014d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d28a0 0x7f385014d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55714/0 (socket says 172.21.15.114:55714) 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 -- 172.21.15.114:0/1713331740 learned_addr learned my addr 172.21.15.114:0/1713331740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f384ea00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38500d1e90 0x7f385015dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 -- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 msgr2=0x7f385014ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 0x7f385014ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 -- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38500d1e90 msgr2=0x7f385015dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38500d1e90 0x7f385015dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 -- 172.21.15.114:0/1713331740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385014dbf0 con 0x7f38500d28a0 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f384ea00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38500d1e90 0x7f385015dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.702+0000 7f3846000640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d28a0 0x7f385014d4c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3838000c00 tx=0x7f38380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.703+0000 7f3847400640 1 -- 172.21.15.114:0/1713331740 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838035070 con 0x7f38500d28a0 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.703+0000 7f3847400640 1 -- 172.21.15.114:0/1713331740 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3838004230 con 0x7f38500d28a0 2024-03-06T07:28:00.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.703+0000 7f3847400640 1 -- 172.21.15.114:0/1713331740 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383803a6f0 con 0x7f38500d28a0 2024-03-06T07:28:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.703+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385014ded0 con 0x7f38500d28a0 2024-03-06T07:28:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.703+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385014b050 con 0x7f38500d28a0 2024-03-06T07:28:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.704+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385014e3f0 con 0x7f38500d28a0 2024-03-06T07:28:00.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.705+0000 7f3847400640 1 -- 172.21.15.114:0/1713331740 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3838038020 con 0x7f38500d28a0 2024-03-06T07:28:00.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.706+0000 7f3847400640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b4d0 0x7f382c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.706+0000 7f3847400640 1 -- 172.21.15.114:0/1713331740 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38380abee0 con 0x7f38500d28a0 2024-03-06T07:28:00.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.706+0000 7f384ea00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b4d0 0x7f382c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.707+0000 7f384ea00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b4d0 0x7f382c06d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3840009f50 tx=0x7f384000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.710+0000 7f3847400640 1 -- 172.21.15.114:0/1713331740 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f385014e3f0 con 0x7f38500d28a0 2024-03-06T07:28:00.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.909+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f385004f330 con 0x7f38500d28a0 2024-03-06T07:28:00.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.911+0000 7f3847400640 1 -- 172.21.15.114:0/1713331740 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f3838040070 con 0x7f38500d28a0 2024-03-06T07:28:00.914 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:00.914 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:00.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b4d0 msgr2=0x7f382c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b4d0 0x7f382c06d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3840009f50 tx=0x7f384000b040 comp rx=0 tx=0).stop 2024-03-06T07:28:00.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d28a0 msgr2=0x7f385014d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d28a0 0x7f385014d4c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3838000c00 tx=0x7f38380046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 shutdown_connections 2024-03-06T07:28:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161580 0x7f385014ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b4d0 0x7f382c06d990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500d28a0 0x7f385014d4c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 --2- 172.21.15.114:0/1713331740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38500d1e90 0x7f385015dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 >> 172.21.15.114:0/1713331740 conn(0x7f38500d3d80 msgr2=0x7f38500b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 shutdown_connections 2024-03-06T07:28:00.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:00.914+0000 7f384fe00640 1 -- 172.21.15.114:0/1713331740 wait complete. 2024-03-06T07:28:00.935 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:28:00.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:01.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.140+0000 7f04bb000640 1 Processor -- start 2024-03-06T07:28:01.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.141+0000 7f04bb000640 1 -- start start 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.143+0000 7f04bb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40b4390 0x7f04b4160460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.143+0000 7f04bb000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f04b40cf9a0 0x7f04b40cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.143+0000 7f04bb000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40b3040 con 0x7f04b415f770 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.143+0000 7f04bb000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b415fb50 con 0x7f04b40cf9a0 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.143+0000 7f04bb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40cd3e0 con 0x7f04b40b4390 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04b9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f04b40cf9a0 0x7f04b40cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04b9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f04b40cf9a0 0x7f04b40cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55724/0 (socket says 172.21.15.114:55724) 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04b9000640 1 -- 172.21.15.114:0/2919903917 learned_addr learned my addr 172.21.15.114:0/2919903917 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04ab400640 1 --2- 172.21.15.114:0/2919903917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40b4390 0x7f04b4160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04b9000640 1 -- 172.21.15.114:0/2919903917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40b4390 msgr2=0x7f04b4160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04b9000640 1 --2- 172.21.15.114:0/2919903917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40b4390 0x7f04b4160460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04b9000640 1 -- 172.21.15.114:0/2919903917 >> v1:172.21.15.177:6789/0 conn(0x7f04b415f770 legacy=0x7f04b415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.144+0000 7f04b9000640 1 -- 172.21.15.114:0/2919903917 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40cd6c0 con 0x7f04b40cf9a0 2024-03-06T07:28:01.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.146+0000 7f04b9000640 1 --2- 172.21.15.114:0/2919903917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b40cf9a0 0x7f04b40cfe30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009870 tx=0x7f04ac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288e60d194b3ddf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.146+0000 7f04b3400640 1 -- 172.21.15.114:0/2919903917 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0317c0 con 0x7f04b40cf9a0 2024-03-06T07:28:01.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.146+0000 7f04b3400640 1 -- 172.21.15.114:0/2919903917 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac037070 con 0x7f04b40cf9a0 2024-03-06T07:28:01.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.146+0000 7f04b3400640 1 -- 172.21.15.114:0/2919903917 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac031b80 con 0x7f04b40cf9a0 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 -- 172.21.15.114:0/2919903917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b40cf9a0 msgr2=0x7f04b40cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 --2- 172.21.15.114:0/2919903917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b40cf9a0 0x7f04b40cfe30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009870 tx=0x7f04ac02b860 comp rx=0 tx=0).stop 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 -- 172.21.15.114:0/2919903917 shutdown_connections 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 --2- 172.21.15.114:0/2919903917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b40cf9a0 0x7f04b40cfe30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 --2- 172.21.15.114:0/2919903917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40b4390 0x7f04b4160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 -- 172.21.15.114:0/2919903917 >> 172.21.15.114:0/2919903917 conn(0x7f04b40b3870 msgr2=0x7f04b40b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 -- 172.21.15.114:0/2919903917 shutdown_connections 2024-03-06T07:28:01.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.147+0000 7f04bb000640 1 -- 172.21.15.114:0/2919903917 wait complete. 2024-03-06T07:28:01.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.149+0000 7f04bb000640 1 Processor -- start 2024-03-06T07:28:01.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.149+0000 7f04bb000640 1 -- start start 2024-03-06T07:28:01.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04bb000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b40b4390 0x7f04b40cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04bb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40cf9a0 0x7f04b415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04bb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b415f770 0x7f04b415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04bb000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40b34a0 con 0x7f04b415f770 2024-03-06T07:28:01.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04bb000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40b3320 con 0x7f04b40b4390 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04bb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b41609a0 con 0x7f04b40cf9a0 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04b9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b415f770 0x7f04b415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04b9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b415f770 0x7f04b415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55726/0 (socket says 172.21.15.114:55726) 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.150+0000 7f04b9000640 1 -- 172.21.15.114:0/3113191208 learned_addr learned my addr 172.21.15.114:0/3113191208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b3e00640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b40b4390 0x7f04b40cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04ab400640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40cf9a0 0x7f04b415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b9000640 1 -- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40cf9a0 msgr2=0x7f04b415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b9000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40cf9a0 0x7f04b415ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b9000640 1 -- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b40b4390 msgr2=0x7f04b40cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b9000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b40b4390 0x7f04b40cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b9000640 1 -- 172.21.15.114:0/3113191208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40cbcd0 con 0x7f04b415f770 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b3e00640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b40b4390 0x7f04b40cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:01.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b9000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b415f770 0x7f04b415e2f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04ac02b8a0 tx=0x7f04ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b0c00640 1 -- 172.21.15.114:0/3113191208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac032650 con 0x7f04b415f770 2024-03-06T07:28:01.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b0c00640 1 -- 172.21.15.114:0/3113191208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac032810 con 0x7f04b415f770 2024-03-06T07:28:01.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b40c8220 con 0x7f04b415f770 2024-03-06T07:28:01.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.151+0000 7f04b0c00640 1 -- 172.21.15.114:0/3113191208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac032c00 con 0x7f04b415f770 2024-03-06T07:28:01.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.152+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b40c7460 con 0x7f04b415f770 2024-03-06T07:28:01.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.153+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b404f330 con 0x7f04b415f770 2024-03-06T07:28:01.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.154+0000 7f04b0c00640 1 -- 172.21.15.114:0/3113191208 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04ac04c060 con 0x7f04b415f770 2024-03-06T07:28:01.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.155+0000 7f04b0c00640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f049006b4d0 0x7f049006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.155+0000 7f04b0c00640 1 -- 172.21.15.114:0/3113191208 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04ac0aba60 con 0x7f04b415f770 2024-03-06T07:28:01.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.155+0000 7f04b3e00640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f049006b4d0 0x7f049006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.156+0000 7f04b3e00640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f049006b4d0 0x7f049006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f04a4007e90 tx=0x7f04a40073a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.159+0000 7f04b0c00640 1 -- 172.21.15.114:0/3113191208 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04b40cbcd0 con 0x7f04b415f770 2024-03-06T07:28:01.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.355+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04b40b3650 con 0x7f04b415f770 2024-03-06T07:28:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.358+0000 7f04b0c00640 1 -- 172.21.15.114:0/3113191208 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f04ac040070 con 0x7f04b415f770 2024-03-06T07:28:01.361 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:01.361 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:01.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f049006b4d0 msgr2=0x7f049006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f049006b4d0 0x7f049006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f04a4007e90 tx=0x7f04a40073a0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b415f770 msgr2=0x7f04b415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b415f770 0x7f04b415e2f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04ac02b8a0 tx=0x7f04ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 shutdown_connections 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f049006b4d0 0x7f049006d990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04b415f770 0x7f04b415e2f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04b40cf9a0 0x7f04b415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 --2- 172.21.15.114:0/3113191208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04b40b4390 0x7f04b40cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.360+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 >> 172.21.15.114:0/3113191208 conn(0x7f04b40b3870 msgr2=0x7f04b40cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.361+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 shutdown_connections 2024-03-06T07:28:01.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.361+0000 7f04bb000640 1 -- 172.21.15.114:0/3113191208 wait complete. 2024-03-06T07:28:01.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:28:01.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:28:01.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:28:01.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:28:01.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:28:01.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:28:01.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:28:01.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:28:01.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.704+0000 7f6eee200640 1 Processor -- start 2024-03-06T07:28:01.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.704+0000 7f6eee200640 1 -- start start 2024-03-06T07:28:01.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.707+0000 7f8c6e800640 1 Processor -- start 2024-03-06T07:28:01.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.707+0000 7f6eee200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6ee00e3160 0x7f6ee00e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.707+0000 7f6eee200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 0x7f6ee004f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.709+0000 7f6eece00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6ee00e3160 0x7f6ee00e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.707+0000 7f8c6e800640 1 -- start start 2024-03-06T07:28:01.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.709+0000 7f6eece00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6ee00e3160 0x7f6ee00e7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55740/0 (socket says 172.21.15.114:55740) 2024-03-06T07:28:01.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.710+0000 7f6eee200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee0054580 con 0x7f6ee004fa10 2024-03-06T07:28:01.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.710+0000 7f6eee200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee0050150 con 0x7f6ee00e3160 2024-03-06T07:28:01.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.710+0000 7f6eee200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00062e0 con 0x7f6ee003aa10 2024-03-06T07:28:01.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.710+0000 7f8c6e800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8c600e7710 0x7f8c6004ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.711+0000 7f8c6e800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8c60053c40 0x7f8c60007020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.710+0000 7f6eece00640 1 -- 172.21.15.114:0/1193408753 learned_addr learned my addr 172.21.15.114:0/1193408753 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.711+0000 7f6eece00640 1 -- 172.21.15.114:0/1193408753 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 msgr2=0x7f6ee004f4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:28:01.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.711+0000 7f8c6e800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c60057d90 con 0x7f8c600589c0 2024-03-06T07:28:01.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.711+0000 7f6eece00640 1 --2- 172.21.15.114:0/1193408753 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 0x7f6ee004f4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.711+0000 7f6eece00640 1 -- 172.21.15.114:0/1193408753 >> v1:172.21.15.177:6789/0 conn(0x7f6ee004fa10 legacy=0x7f6ee0052fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:28:01.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.711+0000 7f6eece00640 1 -- 172.21.15.114:0/1193408753 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0054860 con 0x7f6ee00e3160 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.712+0000 7f8c6d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8c600e7710 0x7f8c6004ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.712+0000 7f8c6e800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c60006260 con 0x7f8c600e7710 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.712+0000 7f8c6e800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c60006540 con 0x7f8c60053c40 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.712+0000 7f8c6ca00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8c600589c0 0x7f8c60053530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35962/0 (socket says 172.21.15.114:35962) 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.712+0000 7f8c6ca00640 1 -- 172.21.15.114:0/2785811294 learned_addr learned my addr 172.21.15.114:0/2785811294 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.712+0000 7f8c6de00640 1 --2- 172.21.15.114:0/2785811294 >> v2:172.21.15.177:3300/0 conn(0x7f8c60053c40 0x7f8c60007020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.713+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772800482 0 0) 0x7f8c60057d90 con 0x7f8c600589c0 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.713+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c50003520 con 0x7f8c600589c0 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.713+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4152930475 0 0) 0x7f8c50003520 con 0x7f8c600589c0 2024-03-06T07:28:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.713+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 >> v2:172.21.15.177:3300/0 conn(0x7f8c60053c40 msgr2=0x7f8c60007020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.713+0000 7f8c67e00640 1 --2- 172.21.15.114:0/2785811294 >> v2:172.21.15.177:3300/0 conn(0x7f8c60053c40 0x7f8c60007020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.714+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 >> v2:172.21.15.114:3300/0 conn(0x7f8c600e7710 msgr2=0x7f8c6004ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.714+0000 7f8c67e00640 1 --2- 172.21.15.114:0/2785811294 >> v2:172.21.15.114:3300/0 conn(0x7f8c600e7710 0x7f8c6004ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.714+0000 7f6eece00640 1 --2- 172.21.15.114:0/1193408753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee00e7740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6ee806ab30 tx=0x7f6ee80a6b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8dc139a0dd0448 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.715+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c60006820 con 0x7f8c600589c0 2024-03-06T07:28:01.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.715+0000 7f6ee7400640 1 -- 172.21.15.114:0/1193408753 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee80b4070 con 0x7f6ee00e3160 2024-03-06T07:28:01.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.715+0000 7f6ee7400640 1 -- 172.21.15.114:0/1193408753 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee80a7a10 con 0x7f6ee00e3160 2024-03-06T07:28:01.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.715+0000 7f6ee7400640 1 -- 172.21.15.114:0/1193408753 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee80af440 con 0x7f6ee00e3160 2024-03-06T07:28:01.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.716+0000 7f21ff600640 1 Processor -- start 2024-03-06T07:28:01.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.716+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8c5c001670 con 0x7f8c600589c0 2024-03-06T07:28:01.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.717+0000 7f8c6de00640 1 --2- 172.21.15.114:0/2785811294 >> v2:172.21.15.177:3300/0 conn(0x7f8c60053c40 0x7f8c60007020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:01.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.717+0000 7f8c6d400640 1 --2- 172.21.15.114:0/2785811294 >> v2:172.21.15.114:3300/0 conn(0x7f8c600e7710 0x7f8c6004ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:01.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.717+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 >> v1:172.21.15.114:6789/0 conn(0x7f8c600589c0 legacy=0x7f8c60053530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.718+0000 7f8c67e00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c50003a80 0x7f8c50003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.718+0000 7f8c67e00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c50004800 0x7f8c50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.717+0000 7f6eee200640 1 -- 172.21.15.114:0/1193408753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 msgr2=0x7f6ee00e7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.718+0000 7f8c67e00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c500052e0 0x7f8c5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.717+0000 7f21ff600640 1 -- start start 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f21ff600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f21f8155060 0x7f21f8140e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f21ff600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f815b290 0x7f21f806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f21ff600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f21f806f070 0x7f21f8071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f21ff600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f8141360 con 0x7f21f8155060 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f21ff600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f815e880 con 0x7f21f806f070 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f21ff600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f80cc620 con 0x7f21f815b290 2024-03-06T07:28:01.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.717+0000 7f6eee200640 1 --2- 172.21.15.114:0/1193408753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee00e7740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6ee806ab30 tx=0x7f6ee80a6b80 comp rx=0 tx=0).stop 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60006260 con 0x7f8c500052e0 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60006540 con 0x7f8c50003a80 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5000c230 con 0x7f8c50004800 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f8c6ca00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c50004800 0x7f8c50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f8c6d400640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c50003a80 0x7f8c50003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f6eee200640 1 -- 172.21.15.114:0/1193408753 shutdown_connections 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f6eee200640 1 --2- 172.21.15.114:0/1193408753 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 0x7f6ee004f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f6eee200640 1 --2- 172.21.15.114:0/1193408753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee00e7740 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.719+0000 7f6eee200640 1 -- 172.21.15.114:0/1193408753 >> 172.21.15.114:0/1193408753 conn(0x7f6ee000a400 msgr2=0x7f6ee000a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.720+0000 7f21f7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f815b290 0x7f21f806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.720+0000 7f21fd600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f21f806f070 0x7f21f8071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.720+0000 7f21fd600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f21f806f070 0x7f21f8071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35588/0 (socket says 172.21.15.114:35588) 2024-03-06T07:28:01.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.720+0000 7f21fd600640 1 -- 172.21.15.114:0/1397048769 learned_addr learned my addr 172.21.15.114:0/1397048769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.720+0000 7f8c6de00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c500052e0 0x7f8c5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.721+0000 7f21fd600640 1 -- 172.21.15.114:0/1397048769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f815b290 msgr2=0x7f21f806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.721+0000 7f21fd600640 1 --2- 172.21.15.114:0/1397048769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f815b290 0x7f21f806eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.721+0000 7f21fd600640 1 -- 172.21.15.114:0/1397048769 >> v2:172.21.15.114:3300/0 conn(0x7f21f8155060 msgr2=0x7f21f8140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f21fd600640 1 --2- 172.21.15.114:0/1397048769 >> v2:172.21.15.114:3300/0 conn(0x7f21f8155060 0x7f21f8140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f21fd600640 1 -- 172.21.15.114:0/1397048769 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f815b7c0 con 0x7f21f806f070 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f9093c00640 1 Processor -- start 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f8c6ca00640 1 -- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c50003a80 msgr2=0x7f8c50003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f8c6ca00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c50003a80 0x7f8c50003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f8c6ca00640 1 -- 172.21.15.114:0/2785811294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c500052e0 msgr2=0x7f8c5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f8c6ca00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c500052e0 0x7f8c5000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f8c6ca00640 1 -- 172.21.15.114:0/2785811294 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c50003520 con 0x7f8c50004800 2024-03-06T07:28:01.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.723+0000 7f8c6de00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c500052e0 0x7f8c5000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.723+0000 7f8c6de00640 1 -- 172.21.15.114:0/2785811294 reap_dead start 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.721+0000 7f6eee200640 1 -- 172.21.15.114:0/1193408753 shutdown_connections 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.721+0000 7f6eee200640 1 -- 172.21.15.114:0/1193408753 wait complete. 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.722+0000 7f9093c00640 1 -- start start 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.724+0000 7f9093c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f908c0cec10 0x7f908c161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.724+0000 7f9093c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0cf2b0 con 0x7f908c162270 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.724+0000 7f9093c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0cf590 con 0x7f908c15e360 2024-03-06T07:28:01.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.724+0000 7f9093c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c1626c0 con 0x7f908c0cec10 2024-03-06T07:28:01.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.724+0000 7f21fd600640 1 --2- 172.21.15.114:0/1397048769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f806f070 0x7f21f8071c20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f21f815c400 tx=0x7f21f002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3623a60d6447a33c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f6eee200640 1 Processor -- start 2024-03-06T07:28:01.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f6eee200640 1 -- start start 2024-03-06T07:28:01.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f908be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f908c0cec10 0x7f908c161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f908be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f908c0cec10 0x7f908c161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35596/0 (socket says 172.21.15.114:35596) 2024-03-06T07:28:01.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f908be00640 1 -- 172.21.15.114:0/3773711764 learned_addr learned my addr 172.21.15.114:0/3773711764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f047a200640 1 Processor -- start 2024-03-06T07:28:01.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f8c6ca00640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c50004800 0x7f8c50004c80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c008fd0 tx=0x7f8c50005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4efc7d82893766d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.726+0000 7f908be00640 1 -- 172.21.15.114:0/3773711764 >> v1:172.21.15.177:6789/0 conn(0x7f908c15e360 legacy=0x7f908c0d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.726+0000 7f908be00640 1 -- 172.21.15.114:0/3773711764 >> v1:172.21.15.114:6789/0 conn(0x7f908c162270 legacy=0x7f908c0d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.726+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c004bc0 con 0x7f8c50004800 2024-03-06T07:28:01.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.726+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c005ad0 con 0x7f8c50004800 2024-03-06T07:28:01.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.726+0000 7f8c67e00640 1 -- 172.21.15.114:0/2785811294 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c0080d0 con 0x7f8c50004800 2024-03-06T07:28:01.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.726+0000 7f1465600640 1 Processor -- start 2024-03-06T07:28:01.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.726+0000 7f908be00640 1 -- 172.21.15.114:0/3773711764 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0cf870 con 0x7f908c0cec10 2024-03-06T07:28:01.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f8c6e800640 1 -- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c50004800 msgr2=0x7f8c50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f8c6e800640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c50004800 0x7f8c50004c80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c008fd0 tx=0x7f8c50005830 comp rx=0 tx=0).stop 2024-03-06T07:28:01.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.725+0000 7f047a200640 1 -- start start 2024-03-06T07:28:01.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f047a200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f04740cec10 0x7f0474161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f047a200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740cf2b0 con 0x7f047415e360 2024-03-06T07:28:01.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f047a200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740cf590 con 0x7f0474162270 2024-03-06T07:28:01.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f047a200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04741626c0 con 0x7f04740cec10 2024-03-06T07:28:01.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f6eee200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 0x7f6ee004e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f1465600640 1 -- start start 2024-03-06T07:28:01.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.727+0000 7f6eee200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ee004fa10 0x7f6ee00e4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eee200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee017a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f21f7400640 1 -- 172.21.15.114:0/1397048769 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f002ca80 con 0x7f21f806f070 2024-03-06T07:28:01.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f0473400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f047415e360 0x7f04740d12e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35978/0 (socket says 172.21.15.114:35978) 2024-03-06T07:28:01.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f21f7400640 1 -- 172.21.15.114:0/1397048769 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f002cc20 con 0x7f21f806f070 2024-03-06T07:28:01.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f21f7400640 1 -- 172.21.15.114:0/1397048769 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f00343f0 con 0x7f21f806f070 2024-03-06T07:28:01.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f0473400640 1 -- 172.21.15.114:0/136959613 learned_addr learned my addr 172.21.15.114:0/136959613 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eee200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00e3540 con 0x7f6ee00e3160 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eee200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0006530 con 0x7f6ee004fa10 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eee200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0007a60 con 0x7f6ee003aa10 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eed800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee017a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eece00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 0x7f6ee004e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eece00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 0x7f6ee004e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58982/0 (socket says 172.21.15.114:58982) 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eed800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee017a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55774/0 (socket says 172.21.15.114:55774) 2024-03-06T07:28:01.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6eece00640 1 -- 172.21.15.114:0/1069594257 learned_addr learned my addr 172.21.15.114:0/1069594257 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f908be00640 1 --2- 172.21.15.114:0/3773711764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 0x7f908c161d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9078009870 tx=0x7f907802bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73a7ebdd5c3e4568 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f8c6e800640 1 -- 172.21.15.114:0/2785811294 shutdown_connections 2024-03-06T07:28:01.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f8c6e800640 1 --2- 172.21.15.114:0/2785811294 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c50004800 0x7f8c50004c80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f8c6e800640 1 -- 172.21.15.114:0/2785811294 >> 172.21.15.114:0/2785811294 conn(0x7f8c60058ee0 msgr2=0x7f8c600592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.728+0000 7f6ee7e00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ee004fa10 0x7f6ee00e4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f8c6e800640 1 -- 172.21.15.114:0/2785811294 shutdown_connections 2024-03-06T07:28:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f6eed800640 1 -- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 msgr2=0x7f6ee004e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f6eed800640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ee003aa10 0x7f6ee004e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f6eed800640 1 -- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ee004fa10 msgr2=0x7f6ee00e4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f046aa00640 1 --2- 172.21.15.114:0/136959613 >> v2:172.21.15.177:3300/0 conn(0x7f04740cec10 0x7f0474161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f6eed800640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ee004fa10 0x7f6ee00e4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f6eed800640 1 -- 172.21.15.114:0/1069594257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee004d0d0 con 0x7f6ee00e3160 2024-03-06T07:28:01.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f908b400640 1 -- 172.21.15.114:0/3773711764 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907802d870 con 0x7f908c0cec10 2024-03-06T07:28:01.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182422135 0 0) 0x7f04740cf2b0 con 0x7f047415e360 2024-03-06T07:28:01.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f6eed800640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee017a3f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00b410 tx=0x7f6edc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f908b400640 1 -- 172.21.15.114:0/3773711764 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907802da10 con 0x7f908c0cec10 2024-03-06T07:28:01.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f908b400640 1 -- 172.21.15.114:0/3773711764 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907802dde0 con 0x7f908c0cec10 2024-03-06T07:28:01.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0458003520 con 0x7f047415e360 2024-03-06T07:28:01.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f6ee7e00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ee004fa10 0x7f6ee00e4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:01.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431395008 0 0) 0x7f04740cf590 con 0x7f0474162270 2024-03-06T07:28:01.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04740cf2b0 con 0x7f0474162270 2024-03-06T07:28:01.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc015020 con 0x7f6ee00e3160 2024-03-06T07:28:01.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc011830 con 0x7f6ee00e3160 2024-03-06T07:28:01.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432001165 0 0) 0x7f0458003520 con 0x7f047415e360 2024-03-06T07:28:01.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc011c20 con 0x7f6ee00e3160 2024-03-06T07:28:01.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 >> v2:172.21.15.177:3300/0 conn(0x7f04740cec10 msgr2=0x7f0474161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee004d3b0 con 0x7f6ee00e3160 2024-03-06T07:28:01.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee004d970 con 0x7f6ee00e3160 2024-03-06T07:28:01.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 --2- 172.21.15.114:0/136959613 >> v2:172.21.15.177:3300/0 conn(0x7f04740cec10 0x7f0474161d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 >> v1:172.21.15.177:6789/0 conn(0x7f0474162270 legacy=0x7f04740d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f9093c00640 1 -- 172.21.15.114:0/3773711764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 msgr2=0x7f908c161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f21ff600640 1 -- 172.21.15.114:0/1397048769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f806f070 msgr2=0x7f21f8071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f21ff600640 1 --2- 172.21.15.114:0/1397048769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f806f070 0x7f21f8071c20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f21f815c400 tx=0x7f21f002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f9093c00640 1 --2- 172.21.15.114:0/3773711764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 0x7f908c161d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9078009870 tx=0x7f907802bab0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04740cf870 con 0x7f047415e360 2024-03-06T07:28:01.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f045c001670 con 0x7f047415e360 2024-03-06T07:28:01.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 >> v1:172.21.15.114:6789/0 conn(0x7f047415e360 legacy=0x7f04740d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f1465600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f146015dfc0 0x7f14600b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f0472a00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458003b40 0x7f0458003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f1465600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14600cee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f1465600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600b4390 con 0x7f14600d1a80 2024-03-06T07:28:01.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f1465600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14600b4670 con 0x7f146015dfc0 2024-03-06T07:28:01.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.730+0000 7f1465600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14600b4950 con 0x7f14600d2b10 2024-03-06T07:28:01.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f9093c00640 1 -- 172.21.15.114:0/3773711764 shutdown_connections 2024-03-06T07:28:01.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f9093c00640 1 --2- 172.21.15.114:0/3773711764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 0x7f908c161d30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f0472a00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0458004800 0x7f0458004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f9093c00640 1 -- 172.21.15.114:0/3773711764 >> 172.21.15.114:0/3773711764 conn(0x7f908c0d3f50 msgr2=0x7f908c0d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f0472a00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04580051c0 0x7f045800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04741626c0 con 0x7f0458003b40 2024-03-06T07:28:01.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045800c010 con 0x7f0458004800 2024-03-06T07:28:01.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045800c210 con 0x7f04580051c0 2024-03-06T07:28:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f9093c00640 1 -- 172.21.15.114:0/3773711764 shutdown_connections 2024-03-06T07:28:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f21ff600640 1 -- 172.21.15.114:0/1397048769 shutdown_connections 2024-03-06T07:28:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f21ff600640 1 --2- 172.21.15.114:0/1397048769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f806f070 0x7f21f8071c20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f21ff600640 1 --2- 172.21.15.114:0/1397048769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f815b290 0x7f21f806eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f21ff600640 1 --2- 172.21.15.114:0/1397048769 >> v2:172.21.15.114:3300/0 conn(0x7f21f8155060 0x7f21f8140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f21ff600640 1 -- 172.21.15.114:0/1397048769 >> 172.21.15.114:0/1397048769 conn(0x7f21f80d0b90 msgr2=0x7f21f80d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f9093c00640 1 -- 172.21.15.114:0/3773711764 wait complete. 2024-03-06T07:28:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.729+0000 7f8c6e800640 1 -- 172.21.15.114:0/2785811294 wait complete. 2024-03-06T07:28:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f046aa00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0458004800 0x7f0458004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f0473e00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04580051c0 0x7f045800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f0473400640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458003b40 0x7f0458003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f145ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f146015dfc0 0x7f14600b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f145f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14600cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f145f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14600cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58986/0 (socket says 172.21.15.114:58986) 2024-03-06T07:28:01.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f145f400640 1 -- 172.21.15.114:0/2817632125 learned_addr learned my addr 172.21.15.114:0/2817632125 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.731+0000 7f145ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f146015dfc0 0x7f14600b3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55776/0 (socket says 172.21.15.114:55776) 2024-03-06T07:28:01.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ec4001030 con 0x7f6ee00e3160 2024-03-06T07:28:01.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.733+0000 7f21ff600640 1 -- 172.21.15.114:0/1397048769 shutdown_connections 2024-03-06T07:28:01.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f0473e00640 1 -- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0458004800 msgr2=0x7f0458004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f0473e00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0458004800 0x7f0458004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f0473e00640 1 -- 172.21.15.114:0/136959613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458003b40 msgr2=0x7f0458003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f0473e00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458003b40 0x7f0458003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f0473e00640 1 -- 172.21.15.114:0/136959613 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04740cf870 con 0x7f04580051c0 2024-03-06T07:28:01.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.734+0000 7f21ff600640 1 -- 172.21.15.114:0/1397048769 wait complete. 2024-03-06T07:28:01.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f145ea00640 1 -- 172.21.15.114:0/2817632125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 msgr2=0x7f14600cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.732+0000 7f145ea00640 1 --2- 172.21.15.114:0/2817632125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14600cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.733+0000 7f145ea00640 1 -- 172.21.15.114:0/2817632125 >> v1:172.21.15.177:6789/0 conn(0x7f14600d1a80 legacy=0x7f14600d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.733+0000 7f145ea00640 1 -- 172.21.15.114:0/2817632125 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600b4c30 con 0x7f146015dfc0 2024-03-06T07:28:01.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.734+0000 7f8c6e800640 1 Processor -- start 2024-03-06T07:28:01.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.734+0000 7fafd9600640 1 Processor -- start 2024-03-06T07:28:01.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.734+0000 7f145f400640 1 --2- 172.21.15.114:0/2817632125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14600cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:01.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.734+0000 7fafd9600640 1 -- start start 2024-03-06T07:28:01.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.734+0000 7f145ea00640 1 --2- 172.21.15.114:0/2817632125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f14600b3ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f144c0097a0 tx=0x7f144c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae60adb160d6c9de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.734+0000 7f0473e00640 1 -- 172.21.15.114:0/136959613 reap_dead start 2024-03-06T07:28:01.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.735+0000 7f8c6e800640 1 -- start start 2024-03-06T07:28:01.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.735+0000 7f145d600640 1 -- 172.21.15.114:0/2817632125 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c038070 con 0x7f146015dfc0 2024-03-06T07:28:01.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.735+0000 7f145d600640 1 -- 172.21.15.114:0/2817632125 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c031710 con 0x7f146015dfc0 2024-03-06T07:28:01.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.735+0000 7f145d600640 1 -- 172.21.15.114:0/2817632125 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c038070 con 0x7f146015dfc0 2024-03-06T07:28:01.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.735+0000 7f9093c00640 1 Processor -- start 2024-03-06T07:28:01.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7f9093c00640 1 -- start start 2024-03-06T07:28:01.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7fafd9600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafcc0dcd70 0x7fafcc0e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7fafd9600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fafcc0e31e0 0x7fafcc0336f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7fafd9600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafcc0e3ab0 con 0x7fafcc00c280 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7fafd9600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafcc03ab50 con 0x7fafcc0dcd70 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7fafd9600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafcc0e3630 con 0x7fafcc0e31e0 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7fafd2a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fafcc00c280 0x7fafcc053d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35990/0 (socket says 172.21.15.114:35990) 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7fafd2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafcc0dcd70 0x7fafcc0e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7fafd2a00640 1 -- 172.21.15.114:0/1401700023 learned_addr learned my addr 172.21.15.114:0/1401700023 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7f1465600640 1 -- 172.21.15.114:0/2817632125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 msgr2=0x7f14600b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7f1465600640 1 --2- 172.21.15.114:0/2817632125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f14600b3ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f144c0097a0 tx=0x7f144c02b7b0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7f0473e00640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04580051c0 0x7f045800b9c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f046c0018d0 tx=0x7f0458005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1328a19ad4715d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7fafd3e00640 1 --2- 172.21.15.114:0/1401700023 >> v2:172.21.15.177:3300/0 conn(0x7fafcc0e31e0 0x7fafcc0336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c0014f0 con 0x7f04580051c0 2024-03-06T07:28:01.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7fafd2000640 1 -- 172.21.15.114:0/1401700023 >> v2:172.21.15.177:3300/0 conn(0x7fafcc0e31e0 msgr2=0x7fafcc0336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7fafd2000640 1 --2- 172.21.15.114:0/1401700023 >> v2:172.21.15.177:3300/0 conn(0x7fafcc0e31e0 0x7fafcc0336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7fafd2000640 1 -- 172.21.15.114:0/1401700023 >> v1:172.21.15.114:6789/0 conn(0x7fafcc00c280 legacy=0x7fafcc053d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c0052d0 con 0x7f04580051c0 2024-03-06T07:28:01.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f1465600640 1 -- 172.21.15.114:0/2817632125 shutdown_connections 2024-03-06T07:28:01.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f1465600640 1 --2- 172.21.15.114:0/2817632125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14600cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f21ff600640 1 Processor -- start 2024-03-06T07:28:01.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7f8c6e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c50004800 0x7f8c6003c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7f8c6e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60052c60 0x7f8c600d4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7f8c6e800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c6003d140 0x7f8c6003d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f1465600640 1 --2- 172.21.15.114:0/2817632125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f14600b3ab0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f047a200640 1 -- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04580051c0 msgr2=0x7f045800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f1465600640 1 -- 172.21.15.114:0/2817632125 >> 172.21.15.114:0/2817632125 conn(0x7f146008d7f0 msgr2=0x7f146009d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f9093c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 0x7f908c14dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f9093c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908c15e360 0x7f908c0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f9093c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f908c162270 0x7f908c0b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f9093c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d23a0 con 0x7f908c15e360 2024-03-06T07:28:01.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f8c6e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60057b20 con 0x7f8c50004800 2024-03-06T07:28:01.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f8c6e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c600579a0 con 0x7f8c60052c60 2024-03-06T07:28:01.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f8c6e800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60056ab0 con 0x7f8c6003d140 2024-03-06T07:28:01.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f0472a00640 1 -- 172.21.15.114:0/136959613 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c0014f0 con 0x7f04580051c0 2024-03-06T07:28:01.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f047a200640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04580051c0 0x7f045800b9c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f046c0018d0 tx=0x7f0458005830 comp rx=0 tx=0).stop 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f9093c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0cfe30 con 0x7f908c0cec10 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f9093c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d2520 con 0x7f908c162270 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.736+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6edc018020 con 0x7f6ee00e3160 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7f6ee4c00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ec806b5a0 0x7f6ec806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.737+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6edc08a640 con 0x7f6ee00e3160 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f6ee4c00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f6ec80737c0 0x7f6ec8075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ec8076290 con 0x7f6ec80737c0 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6edc08aa30 con 0x7f6ee00e3160 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.738+0000 7f6eece00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ec806b5a0 0x7f6ec806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f6eece00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ec806b5a0 0x7f6ec806da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8069b90 tx=0x7f6ee806c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7fafd2000640 1 -- 172.21.15.114:0/1401700023 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc04f850 con 0x7fafcc0dcd70 2024-03-06T07:28:01.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f1465600640 1 -- 172.21.15.114:0/2817632125 shutdown_connections 2024-03-06T07:28:01.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f1465600640 1 -- 172.21.15.114:0/2817632125 wait complete. 2024-03-06T07:28:01.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f8c6ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60052c60 0x7f8c600d4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f8c6d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c50004800 0x7f8c6003c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f8c6d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c50004800 0x7f8c6003c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55804/0 (socket says 172.21.15.114:55804) 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f8c6d400640 1 -- 172.21.15.114:0/189397712 learned_addr learned my addr 172.21.15.114:0/189397712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f8c6de00640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c6003d140 0x7f8c6003d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f047a200640 1 -- 172.21.15.114:0/136959613 shutdown_connections 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f047a200640 1 --2- 172.21.15.114:0/136959613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04580051c0 0x7f045800b9c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f047a200640 1 -- 172.21.15.114:0/136959613 >> 172.21.15.114:0/136959613 conn(0x7f04740d3f50 msgr2=0x7f04740d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.739+0000 7f047a200640 1 -- 172.21.15.114:0/136959613 shutdown_connections 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f908be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908c15e360 0x7f908c0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f908be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908c15e360 0x7f908c0ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55810/0 (socket says 172.21.15.114:55810) 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f908be00640 1 -- 172.21.15.114:0/4287985076 learned_addr learned my addr 172.21.15.114:0/4287985076 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f9091200640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 0x7f908c14dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f9091c00640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f908c162270 0x7f908c0b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f8c6d400640 1 -- 172.21.15.114:0/189397712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c6003d140 msgr2=0x7f8c6003d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f8c6d400640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8c6003d140 0x7f8c6003d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f8c6d400640 1 -- 172.21.15.114:0/189397712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60052c60 msgr2=0x7f8c600d4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7fafd2000640 1 --2- 172.21.15.114:0/1401700023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0dcd70 0x7fafcc0e2ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00c660 tx=0x7fafc002e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b798cc5942bfa7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f8c6d400640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c60052c60 0x7f8c600d4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f8c6d400640 1 -- 172.21.15.114:0/189397712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6004ffb0 con 0x7f8c50004800 2024-03-06T07:28:01.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.740+0000 7f047a200640 1 -- 172.21.15.114:0/136959613 wait complete. 2024-03-06T07:28:01.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f9091200640 1 -- 172.21.15.114:0/4287985076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f908c162270 msgr2=0x7f908c0b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f9091200640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f908c162270 0x7f908c0b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f8c6d400640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c50004800 0x7f8c6003c3d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c540118d0 tx=0x7f8c54011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f9091200640 1 -- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908c15e360 msgr2=0x7f908c0ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f9091200640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908c15e360 0x7f908c0ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f9091200640 1 -- 172.21.15.114:0/4287985076 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c14b0f0 con 0x7f908c0cec10 2024-03-06T07:28:01.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f9091200640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 0x7f908c14dae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f908000c340 tx=0x7f908000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c54019070 con 0x7f8c50004800 2024-03-06T07:28:01.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f047a200640 1 Processor -- start 2024-03-06T07:28:01.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7fafd1600640 1 -- 172.21.15.114:0/1401700023 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc003b070 con 0x7fafcc0dcd70 2024-03-06T07:28:01.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080017070 con 0x7f908c0cec10 2024-03-06T07:28:01.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90800052f0 con 0x7f908c0cec10 2024-03-06T07:28:01.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080003830 con 0x7f908c0cec10 2024-03-06T07:28:01.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f9093c00640 1 -- 172.21.15.114:0/4287985076 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c14b3d0 con 0x7f908c0cec10 2024-03-06T07:28:01.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c54004220 con 0x7f8c50004800 2024-03-06T07:28:01.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5400f040 con 0x7f8c50004800 2024-03-06T07:28:01.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.741+0000 7f21ff600640 1 -- start start 2024-03-06T07:28:01.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f9093c00640 1 -- 172.21.15.114:0/4287985076 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c14b990 con 0x7f908c0cec10 2024-03-06T07:28:01.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f1465600640 1 Processor -- start 2024-03-06T07:28:01.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f1465600640 1 -- start start 2024-03-06T07:28:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f9093c00640 1 -- 172.21.15.114:0/4287985076 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f908c04f330 con 0x7f908c0cec10 2024-03-06T07:28:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f047a200640 1 -- start start 2024-03-06T07:28:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f047a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580051c0 0x7f04740b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f047a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04741f8320 0x7f04741e7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f047a200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04741e7a90 0x7f04740b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f047a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0474161d30 con 0x7f04580051c0 2024-03-06T07:28:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f047a200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740cfe30 con 0x7f04741f8320 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f047a200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0474161eb0 con 0x7f04741e7a90 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f0473400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580051c0 0x7f04740b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f0473400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580051c0 0x7f04740b6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55826/0 (socket says 172.21.15.114:55826) 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f0473400640 1 -- 172.21.15.114:0/4241151636 learned_addr learned my addr 172.21.15.114:0/4241151636 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f1465600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14600d1a80 0x7f146014bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f1465600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14601609a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f1465600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f1460160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f1465600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14600b4170 con 0x7f146015dfc0 2024-03-06T07:28:01.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f1465600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14600b3ff0 con 0x7f14600d1a80 2024-03-06T07:28:01.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f1465600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14600cf350 con 0x7f14600d2b10 2024-03-06T07:28:01.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7fafd1600640 1 -- 172.21.15.114:0/1401700023 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc002ee50 con 0x7fafcc0dcd70 2024-03-06T07:28:01.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.744+0000 7f046aa00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04741f8320 0x7f04741e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7fafd1600640 1 -- 172.21.15.114:0/1401700023 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0036660 con 0x7fafcc0dcd70 2024-03-06T07:28:01.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f145f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f1460160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f0473e00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04741e7a90 0x7f04740b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c600530e0 con 0x7f8c50004800 2024-03-06T07:28:01.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.742+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c60050250 con 0x7f8c50004800 2024-03-06T07:28:01.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c60005390 con 0x7f8c50004800 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7fafd9600640 1 -- 172.21.15.114:0/1401700023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0dcd70 msgr2=0x7fafcc0e2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.743+0000 7fafd9600640 1 --2- 172.21.15.114:0/1401700023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0dcd70 0x7fafcc0e2ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00c660 tx=0x7fafc002e0a0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7fafd9600640 1 -- 172.21.15.114:0/1401700023 shutdown_connections 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7fafd9600640 1 --2- 172.21.15.114:0/1401700023 >> v2:172.21.15.177:3300/0 conn(0x7fafcc0e31e0 0x7fafcc0336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7fafd9600640 1 --2- 172.21.15.114:0/1401700023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0dcd70 0x7fafcc0e2ca0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7fafd9600640 1 -- 172.21.15.114:0/1401700023 >> 172.21.15.114:0/1401700023 conn(0x7fafcc054ca0 msgr2=0x7fafcc0550b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7fafd9600640 1 -- 172.21.15.114:0/1401700023 shutdown_connections 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7fafd9600640 1 -- 172.21.15.114:0/1401700023 wait complete. 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f145ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14600d1a80 0x7f146014bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f145ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14600d1a80 0x7f146014bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35686/0 (socket says 172.21.15.114:35686) 2024-03-06T07:28:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f0473400640 1 -- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04741e7a90 msgr2=0x7f04740b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f0473400640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04741e7a90 0x7f04740b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f0473400640 1 -- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04741f8320 msgr2=0x7f04741e7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f0473400640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04741f8320 0x7f04741e7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f145f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f1460160ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55838/0 (socket says 172.21.15.114:55838) 2024-03-06T07:28:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f0473400640 1 -- 172.21.15.114:0/4241151636 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04741f8780 con 0x7f04580051c0 2024-03-06T07:28:01.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f0473e00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f04741e7a90 0x7f04740b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:01.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f6ee7e00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f6ec80737c0 0x7f6ec8075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.745+0000 7f145ea00640 1 -- 172.21.15.114:0/533784462 learned_addr learned my addr 172.21.15.114:0/533784462 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.746+0000 7f0473400640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580051c0 0x7f04740b6b70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f045c007360 tx=0x7f045c007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.746+0000 7f046be00640 1 -- 172.21.15.114:0/4241151636 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c01a070 con 0x7f04580051c0 2024-03-06T07:28:01.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.746+0000 7f046be00640 1 -- 172.21.15.114:0/4241151636 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c00fa20 con 0x7f04580051c0 2024-03-06T07:28:01.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.746+0000 7f046be00640 1 -- 172.21.15.114:0/4241151636 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c00fdf0 con 0x7f04580051c0 2024-03-06T07:28:01.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f145f400640 1 -- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 msgr2=0x7f14601609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f145f400640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14600d2b10 0x7f14601609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f145f400640 1 -- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14600d1a80 msgr2=0x7f146014bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f145f400640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14600d1a80 0x7f146014bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f145f400640 1 -- 172.21.15.114:0/533784462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146014c4e0 con 0x7f146015dfc0 2024-03-06T07:28:01.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f6ee7e00640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f6ec80737c0 0x7f6ec8075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7fafd9600640 1 Processor -- start 2024-03-06T07:28:01.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f908001a070 con 0x7f908c0cec10 2024-03-06T07:28:01.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.747+0000 7f145f400640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f1460160ee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f145800efc0 tx=0x7f145800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800cb20 con 0x7f146015dfc0 2024-03-06T07:28:01.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145800cce0 con 0x7f146015dfc0 2024-03-06T07:28:01.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458003c00 con 0x7f146015dfc0 2024-03-06T07:28:01.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04741e7f10 con 0x7f04580051c0 2024-03-06T07:28:01.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21ff600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21f806f070 0x7f21f8150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21ff600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f8155060 0x7f21f8140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21fcc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21f806f070 0x7f21f8150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21ff600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f815b290 0x7f21f8153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21ff600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f815c060 con 0x7f21f806f070 2024-03-06T07:28:01.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21ff600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f815bd80 con 0x7f21f815b290 2024-03-06T07:28:01.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21ff600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f8155440 con 0x7f21f8155060 2024-03-06T07:28:01.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146015f470 con 0x7f146015dfc0 2024-03-06T07:28:01.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.749+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146015fa30 con 0x7f146015dfc0 2024-03-06T07:28:01.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21fcc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21f806f070 0x7f21f8150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55852/0 (socket says 172.21.15.114:55852) 2024-03-06T07:28:01.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.748+0000 7f21fcc00640 1 -- 172.21.15.114:0/1479160153 learned_addr learned my addr 172.21.15.114:0/1479160153 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f21fcc00640 1 -- 172.21.15.114:0/1479160153 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f8155060 msgr2=0x7f21f8140260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:28:01.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f21fcc00640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f21f8155060 0x7f21f8140260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f21fcc00640 1 -- 172.21.15.114:0/1479160153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f815b290 msgr2=0x7f21f8153d20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f21fcc00640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21f815b290 0x7f21f8153d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f21fcc00640 1 -- 172.21.15.114:0/1479160153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f813f4a0 con 0x7f21f806f070 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.749+0000 7f9088c00640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b520 0x7f906c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.749+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9080089ba0 con 0x7f908c0cec10 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.749+0000 7f9088c00640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f906c0737a0 0x7f906c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.749+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f906c076270 con 0x7f906c0737a0 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.749+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9080089f50 con 0x7f908c0cec10 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f908be00640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b520 0x7f906c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f9091c00640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f906c0737a0 0x7f906c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.749+0000 7fafd9600640 1 -- start start 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7fafd9600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafcc00c280 0x7fafcc0dd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7fafd9600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafcc0dcd70 0x7fafcc047a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7fafd9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0e31e0 0x7fafcc04bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7fafd9600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0500f0 con 0x7fafcc0e31e0 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7fafd9600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc04fe10 con 0x7fafcc0dcd70 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7fafd9600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc0503d0 con 0x7fafcc00c280 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04740cbed0 con 0x7f04580051c0 2024-03-06T07:28:01.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.751+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== osd.5 v2:172.21.15.177:6816/2170499622 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6ec8076290 con 0x7f6ec80737c0 2024-03-06T07:28:01.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1460160080 con 0x7f146015dfc0 2024-03-06T07:28:01.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.750+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c5400f7c0 con 0x7f8c50004800 2024-03-06T07:28:01.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.751+0000 7fafd2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafcc0dcd70 0x7fafcc047a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.751+0000 7fafd2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafcc0dcd70 0x7fafcc047a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35700/0 (socket says 172.21.15.114:35700) 2024-03-06T07:28:01.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.751+0000 7fafd2000640 1 -- 172.21.15.114:0/4222100071 learned_addr learned my addr 172.21.15.114:0/4222100071 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.751+0000 7f908be00640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b520 0x7f906c06d9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f907802baf0 tx=0x7f9078008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.751+0000 7fafd3e00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0e31e0 0x7fafcc04bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.751+0000 7fafd2a00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafcc00c280 0x7fafcc0dd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7f9091c00640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f906c0737a0 0x7f906c075bc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7fafd2000640 1 -- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafcc00c280 msgr2=0x7fafcc0dd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7fafd2000640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafcc00c280 0x7fafcc0dd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7fafd2000640 1 -- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0e31e0 msgr2=0x7fafcc04bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7fafd2000640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0e31e0 0x7fafcc04bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7fafd2000640 1 -- 172.21.15.114:0/4222100071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc047f60 con 0x7fafcc0dcd70 2024-03-06T07:28:01.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7fafd3e00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafcc0e31e0 0x7fafcc04bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7f21fcc00640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21f806f070 0x7f21f8150280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00b4a0 tx=0x7f21ec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec014070 con 0x7f21f806f070 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec010610 con 0x7f21f806f070 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.752+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec0109e0 con 0x7f21f806f070 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.753+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f813f780 con 0x7f21f806f070 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.753+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f81407a0 con 0x7f21f806f070 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== osd.1 v2:172.21.15.114:6816/1955604845 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f906c076270 con 0x7f906c0737a0 2024-03-06T07:28:01.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.753+0000 7f8c65600640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c4806b5a0 0x7f8c4806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c5408aba0 con 0x7f8c50004800 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7f8c65600640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f8c48073820 0x7f8c48075c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 --> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c54001d30 con 0x7f8c48073820 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c5408af90 con 0x7f8c50004800 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.753+0000 7f046be00640 1 -- 172.21.15.114:0/4241151636 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f045c01d070 con 0x7f04580051c0 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7f046be00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f045006b520 0x7f045006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7f046aa00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f045006b520 0x7f045006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.755+0000 7f046be00640 1 -- 172.21.15.114:0/4241151636 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f045c08bca0 con 0x7f04580051c0 2024-03-06T07:28:01.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.754+0000 7fafd2000640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafcc0dcd70 0x7fafcc047a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fafc0000c00 tx=0x7fafc00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.755+0000 7f046aa00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f045006b520 0x7f045006d9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f047415ce40 tx=0x7f0464003bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.755+0000 7f8c6ca00640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c4806b5a0 0x7f8c4806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.755+0000 7f047a200640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f04741e8430 0x7f04740d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.755+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04740b6870 con 0x7f04741e8430 2024-03-06T07:28:01.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7f0473e00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f04741e8430 0x7f04740d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21ec017070 con 0x7f21f806f070 2024-03-06T07:28:01.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7f8c6de00640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f8c48073820 0x7f8c48075c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.757+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21f8140a60 con 0x7f21f806f070 2024-03-06T07:28:01.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.758+0000 7f0473e00640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f04741e8430 0x7f04740d12e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.758+0000 7f21f4c00640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f21d806b520 0x7f21d806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.757+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f145801b030 con 0x7f146015dfc0 2024-03-06T07:28:01.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.758+0000 7f1456a00640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144006b500 0x7f144006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.758+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f145808a3d0 con 0x7f146015dfc0 2024-03-06T07:28:01.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.758+0000 7f8c6de00640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f8c48073820 0x7f8c48075c40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f145ea00640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144006b500 0x7f144006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f1456a00640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x7f14400736a0 0x7f1440075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21ec0876e0 con 0x7f21f806f070 2024-03-06T07:28:01.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f21f4c00640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f21d8073720 0x7f21d8075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21ec013ec0 con 0x7f21d8073720 2024-03-06T07:28:01.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f145801ae80 con 0x7f14400736a0 2024-03-06T07:28:01.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1458091050 con 0x7f146015dfc0 2024-03-06T07:28:01.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21ec08e060 con 0x7f21f806f070 2024-03-06T07:28:01.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f145ea00640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144006b500 0x7f144006d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f144c0091d0 tx=0x7f144c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f156a600640 1 Processor -- start 2024-03-06T07:28:01.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.759+0000 7f156a600640 1 -- start start 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc003b040 con 0x7fafcc0dcd70 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0044070 con 0x7fafcc0dcd70 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc003f680 con 0x7fafcc0dcd70 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafcc037610 con 0x7fafcc0dcd70 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.756+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafcc035640 con 0x7fafcc0dcd70 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.757+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafac001030 con 0x7fafcc0dcd70 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.760+0000 7f046be00640 1 -- 172.21.15.114:0/4241151636 <== osd.6 v2:172.21.15.177:6808/4041662010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f04740b6870 con 0x7f04741e8430 2024-03-06T07:28:01.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.760+0000 7f8c6ca00640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c4806b5a0 0x7f8c4806da60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8c500057e0 tx=0x7f8c5c00a000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.760+0000 7f21f7e00640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f21d806b520 0x7f21d806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f145e000640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x7f14400736a0 0x7f1440075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== osd.2 v2:172.21.15.114:6808/1832993461 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8c54001d30 con 0x7f8c48073820 2024-03-06T07:28:01.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f21fd600640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f21d8073720 0x7f21d8075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f21f7e00640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f21d806b520 0x7f21d806d9e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21f81507c0 tx=0x7f21e4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.760+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafc003f820 con 0x7fafcc0dcd70 2024-03-06T07:28:01.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7fafcaa00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fafb406b4d0 0x7fafb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafc00acc50 con 0x7fafcc0dcd70 2024-03-06T07:28:01.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7f145e000640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x7f14400736a0 0x7f1440075ac0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f156a600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1564154490 0x7f15640cba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f156a600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15640ca1f0 con 0x7f15640cde80 2024-03-06T07:28:01.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f156a600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15640ca4d0 con 0x7f1564154870 2024-03-06T07:28:01.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.761+0000 7f156a600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640ca7b0 con 0x7f1564154490 2024-03-06T07:28:01.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7f155aa00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f1564154870 0x7f15640cd770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:47936/0 (socket says 172.21.15.114:47936) 2024-03-06T07:28:01.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7f155aa00640 1 -- 172.21.15.114:0/4242520797 learned_addr learned my addr 172.21.15.114:0/4242520797 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:01.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7fafcaa00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7fafb40736d0 0x7fafb4075af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafc003fe50 con 0x7fafb40736d0 2024-03-06T07:28:01.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafc00ad0e0 con 0x7fafcc0dcd70 2024-03-06T07:28:01.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7fafd2a00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fafb406b4d0 0x7fafb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.762+0000 7f1563400640 1 --2- 172.21.15.114:0/4242520797 >> v2:172.21.15.114:3300/0 conn(0x7f1564154490 0x7f15640cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.763+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636486485 0 0) 0x7f15640ca1f0 con 0x7f15640cde80 2024-03-06T07:28:01.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.763+0000 7fafd2a00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fafb406b4d0 0x7fafb406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fafd4062b90 tx=0x7fafd406d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.763+0000 7fafd3e00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7fafb40736d0 0x7fafb4075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.763+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1548003520 con 0x7f15640cde80 2024-03-06T07:28:01.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.763+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667936019 0 0) 0x7f15640ca4d0 con 0x7f1564154870 2024-03-06T07:28:01.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7fafd3e00640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7fafb40736d0 0x7fafb4075af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.763+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15640ca1f0 con 0x7f1564154870 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.763+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 12489325 0 0) 0x7f1548003520 con 0x7f15640cde80 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 >> v2:172.21.15.114:3300/0 conn(0x7f1564154490 msgr2=0x7f15640cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 --2- 172.21.15.114:0/4242520797 >> v2:172.21.15.114:3300/0 conn(0x7f1564154490 0x7f15640cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 >> v1:172.21.15.177:6789/0 conn(0x7f1564154870 legacy=0x7f15640cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640caa90 con 0x7f15640cde80 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f155c001670 con 0x7f15640cde80 2024-03-06T07:28:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1563400640 1 --2- 172.21.15.114:0/4242520797 >> v2:172.21.15.114:3300/0 conn(0x7f1564154490 0x7f15640cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 >> v1:172.21.15.114:6789/0 conn(0x7f15640cde80 legacy=0x7f156406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1548004c40 0x7f1548005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15480037a0 0x7f1548003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1548004360 0x7f154800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.765+0000 7f21fd600640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f21d8073720 0x7f21d8075b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640ca7b0 con 0x7f15480037a0 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.765+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== osd.7 v2:172.21.15.177:6824/1593615297 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f145801ae80 con 0x7f14400736a0 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154800c2a0 con 0x7f1548004c40 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.764+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154800c4a0 con 0x7f1548004360 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.765+0000 7f1563400640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1548004c40 0x7f1548005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.765+0000 7f1563e00640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1548004360 0x7f154800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.765+0000 7f1563e00640 1 -- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1548004c40 msgr2=0x7f1548005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.766+0000 7f1563e00640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1548004c40 0x7f1548005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.766+0000 7f1563e00640 1 -- 172.21.15.114:0/4242520797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15480037a0 msgr2=0x7f1548003c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.766+0000 7f1563e00640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15480037a0 0x7f1548003c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.766+0000 7f1563e00640 1 -- 172.21.15.114:0/4242520797 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1548003520 con 0x7f1548004360 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.767+0000 7f1563e00640 1 -- 172.21.15.114:0/4242520797 reap_dead start 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.767+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== osd.4 v2:172.21.15.177:6800/4111454897 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fafc003fe50 con 0x7fafb40736d0 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.767+0000 7f1563e00640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1548004360 0x7f154800b9c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f155c007ec0 tx=0x7f1548005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d385cce29ca99f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.768+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== osd.0 v2:172.21.15.114:6800/3895229755 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21ec013ec0 con 0x7f21d8073720 2024-03-06T07:28:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.774+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c039070 con 0x7f1548004360 2024-03-06T07:28:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.774+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c02c8b0 con 0x7f1548004360 2024-03-06T07:28:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.774+0000 7f1562a00640 1 -- 172.21.15.114:0/4242520797 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c007b10 con 0x7f1548004360 2024-03-06T07:28:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.775+0000 7f156a600640 1 -- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1548004360 msgr2=0x7f154800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.775+0000 7f156a600640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1548004360 0x7f154800b9c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f155c007ec0 tx=0x7f1548005830 comp rx=0 tx=0).stop 2024-03-06T07:28:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.778+0000 7f9093c00640 1 -- 172.21.15.114:0/4287985076 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f908c0ce2e0 con 0x7f906c0737a0 2024-03-06T07:28:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.778+0000 7f156a600640 1 -- 172.21.15.114:0/4242520797 shutdown_connections 2024-03-06T07:28:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.778+0000 7f156a600640 1 --2- 172.21.15.114:0/4242520797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1548004360 0x7f154800b9c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.778+0000 7f156a600640 1 -- 172.21.15.114:0/4242520797 >> 172.21.15.114:0/4242520797 conn(0x7f15640cc4f0 msgr2=0x7f15640cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.779+0000 7f9088c00640 1 -- 172.21.15.114:0/4287985076 <== osd.1 v2:172.21.15.114:6816/1955604845 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f908c0ce2e0 con 0x7f906c0737a0 2024-03-06T07:28:01.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.784+0000 7f156a600640 1 -- 172.21.15.114:0/4242520797 shutdown_connections 2024-03-06T07:28:01.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.784+0000 7f156a600640 1 -- 172.21.15.114:0/4242520797 wait complete. 2024-03-06T07:28:01.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.786+0000 7f156a600640 1 Processor -- start 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 -- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f906c0737a0 msgr2=0x7f906c075bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f906c0737a0 0x7f906c075bc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 -- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b520 msgr2=0x7f906c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b520 0x7f906c06d9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f907802baf0 tx=0x7f9078008030 comp rx=0 tx=0).stop 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 -- 172.21.15.114:0/4287985076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 msgr2=0x7f908c14dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 --2- 172.21.15.114:0/4287985076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908c0cec10 0x7f908c14dae0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f908000c340 tx=0x7f908000c800 comp rx=0 tx=0).stop 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f9091c00640 1 -- 172.21.15.114:0/4287985076 reap_dead start 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 -- 172.21.15.114:0/4287985076 shutdown_connections 2024-03-06T07:28:01.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f907e000640 1 -- 172.21.15.114:0/4287985076 >> 172.21.15.114:0/4287985076 conn(0x7f908c0d3f50 msgr2=0x7f908c15bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.787+0000 7f156a600640 1 -- start start 2024-03-06T07:28:01.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.789+0000 7f156a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1548004360 0x7f156414b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f156a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156414b9b0 0x7f15641f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f156a600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15641f3bc0 0x7f15641f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f907e000640 1 -- 172.21.15.114:0/4287985076 shutdown_connections 2024-03-06T07:28:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f907e000640 1 -- 172.21.15.114:0/4287985076 wait complete. 2024-03-06T07:28:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f1563400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1548004360 0x7f156414b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f1563400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1548004360 0x7f156414b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55884/0 (socket says 172.21.15.114:55884) 2024-03-06T07:28:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f1563400640 1 -- 172.21.15.114:0/618813594 learned_addr learned my addr 172.21.15.114:0/618813594 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156406ec70 con 0x7f1548004360 2024-03-06T07:28:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.790+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ec4002dd0 con 0x7f6ec80737c0 2024-03-06T07:28:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.792+0000 7f6ee4c00640 1 -- 172.21.15.114:0/1069594257 <== osd.5 v2:172.21.15.177:6816/2170499622 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6ec4002dd0 con 0x7f6ec80737c0 2024-03-06T07:28:01.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.793+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f6ec80737c0 msgr2=0x7f6ec8075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.793+0000 7f6eee200640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f6ec80737c0 0x7f6ec8075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.791+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640cb290 con 0x7f156414b9b0 2024-03-06T07:28:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.791+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156406edf0 con 0x7f15641f3bc0 2024-03-06T07:28:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.791+0000 7f1563400640 1 -- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15641f3bc0 msgr2=0x7f15641f4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.791+0000 7f155aa00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156414b9b0 0x7f15641f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.791+0000 7f1563e00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15641f3bc0 0x7f15641f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.792+0000 7f1563400640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15641f3bc0 0x7f15641f4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.792+0000 7f1563400640 1 -- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156414b9b0 msgr2=0x7f15641f3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.792+0000 7f1563400640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156414b9b0 0x7f15641f3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.792+0000 7f1563400640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156415b0b0 con 0x7f1548004360 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.792+0000 7f155aa00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156414b9b0 0x7f15641f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.793+0000 7f1563e00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15641f3bc0 0x7f15641f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.793+0000 7f1563400640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1548004360 0x7f156414b390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f155400c440 tx=0x7f155400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155400b760 con 0x7f1548004360 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155400b900 con 0x7f1548004360 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155400bcd0 con 0x7f1548004360 2024-03-06T07:28:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ec806b5a0 msgr2=0x7f6ec806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f6eee200640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ec806b5a0 0x7f6ec806da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8069b90 tx=0x7f6ee806c040 comp rx=0 tx=0).stop 2024-03-06T07:28:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 msgr2=0x7f6ee017a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f6eee200640 1 --2- 172.21.15.114:0/1069594257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ee00e3160 0x7f6ee017a3f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00b410 tx=0x7f6edc00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f6eed800640 1 -- 172.21.15.114:0/1069594257 reap_dead start 2024-03-06T07:28:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156415aaf0 con 0x7f1548004360 2024-03-06T07:28:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.794+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1564154de0 con 0x7f1548004360 2024-03-06T07:28:01.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.795+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f156404f330 con 0x7f1548004360 2024-03-06T07:28:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.796+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f04740d34e0 con 0x7f04741e8430 2024-03-06T07:28:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.797+0000 7f046be00640 1 -- 172.21.15.114:0/4241151636 <== osd.6 v2:172.21.15.177:6808/4041662010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f04740d34e0 con 0x7f04741e8430 2024-03-06T07:28:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.797+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 shutdown_connections 2024-03-06T07:28:01.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.797+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 >> 172.21.15.114:0/1069594257 conn(0x7f6ee000a400 msgr2=0x7f6ee0052fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.797+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f04741e8430 msgr2=0x7f04740d12e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.797+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 --> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c60039630 con 0x7f8c48073820 2024-03-06T07:28:01.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.798+0000 7f047a200640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f04741e8430 0x7f04740d12e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.798+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 shutdown_connections 2024-03-06T07:28:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.798+0000 7f6eee200640 1 -- 172.21.15.114:0/1069594257 wait complete. 2024-03-06T07:28:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.799+0000 7f8c65600640 1 -- 172.21.15.114:0/189397712 <== osd.2 v2:172.21.15.114:6808/1832993461 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8c60039630 con 0x7f8c48073820 2024-03-06T07:28:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.799+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1554005d10 con 0x7f1548004360 2024-03-06T07:28:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f045006b520 msgr2=0x7f045006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7f047a200640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f045006b520 0x7f045006d9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f047415ce40 tx=0x7f0464003bf0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580051c0 msgr2=0x7f04740b6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7f047a200640 1 --2- 172.21.15.114:0/4241151636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04580051c0 0x7f04740b6b70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f045c007360 tx=0x7f045c007820 comp rx=0 tx=0).stop 2024-03-06T07:28:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7f0473e00640 1 -- 172.21.15.114:0/4241151636 reap_dead start 2024-03-06T07:28:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 shutdown_connections 2024-03-06T07:28:01.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 >> 172.21.15.114:0/4241151636 conn(0x7f04740d3f50 msgr2=0x7f047406d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.800+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fafac002dd0 con 0x7fafb40736d0 2024-03-06T07:28:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.801+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 shutdown_connections 2024-03-06T07:28:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.801+0000 7f155be00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f154007c5e0 0x7f154007eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.801+0000 7f047a200640 1 -- 172.21.15.114:0/4241151636 wait complete. 2024-03-06T07:28:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.801+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f8c48073820 msgr2=0x7f8c48075c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.801+0000 7f8c6e800640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f8c48073820 0x7f8c48075c40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.801+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f155408bd70 con 0x7f1548004360 2024-03-06T07:28:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.801+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f146004f330 con 0x7f14400736a0 2024-03-06T07:28:01.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.802+0000 7f155be00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f1540084800 0x7f1540086c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.802+0000 7f155aa00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f154007c5e0 0x7f154007eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.802+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1554021e50 con 0x7f1540084800 2024-03-06T07:28:01.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f155408c160 con 0x7f1548004360 2024-03-06T07:28:01.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f155aa00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f154007c5e0 0x7f154007eaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1564062b90 tx=0x7f154c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f1563e00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f1540084800 0x7f1540086c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7fafcaa00640 1 -- 172.21.15.114:0/4222100071 <== osd.4 v2:172.21.15.177:6800/4111454897 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fafac002dd0 con 0x7fafb40736d0 2024-03-06T07:28:01.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21f8154e80 con 0x7f21d8073720 2024-03-06T07:28:01.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c4806b5a0 msgr2=0x7f8c4806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f8c6e800640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c4806b5a0 0x7f8c4806da60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8c500057e0 tx=0x7f8c5c00a000 comp rx=0 tx=0).stop 2024-03-06T07:28:01.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c50004800 msgr2=0x7f8c6003c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f8c6e800640 1 --2- 172.21.15.114:0/189397712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c50004800 0x7f8c6003c3d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c540118d0 tx=0x7f8c54011d90 comp rx=0 tx=0).stop 2024-03-06T07:28:01.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7f8c6de00640 1 -- 172.21.15.114:0/189397712 reap_dead start 2024-03-06T07:28:01.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.803+0000 7f1456a00640 1 -- 172.21.15.114:0/533784462 <== osd.7 v2:172.21.15.177:6824/1593615297 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f146004f330 con 0x7f14400736a0 2024-03-06T07:28:01.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7fafb40736d0 msgr2=0x7fafb4075af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7fafd9600640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7fafb40736d0 0x7fafb4075af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fafb406b4d0 msgr2=0x7fafb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7fafd9600640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fafb406b4d0 0x7fafb406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fafd4062b90 tx=0x7fafd406d040 comp rx=0 tx=0).stop 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafcc0dcd70 msgr2=0x7fafcc047a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7fafd9600640 1 --2- 172.21.15.114:0/4222100071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafcc0dcd70 0x7fafcc047a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fafc0000c00 tx=0x7fafc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 shutdown_connections 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 >> 172.21.15.114:0/189397712 conn(0x7f8c60058ee0 msgr2=0x7f8c60038380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7fafd3e00640 1 -- 172.21.15.114:0/4222100071 reap_dead start 2024-03-06T07:28:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.804+0000 7f21f4c00640 1 -- 172.21.15.114:0/1479160153 <== osd.0 v2:172.21.15.114:6800/3895229755 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f21f8154e80 con 0x7f21d8073720 2024-03-06T07:28:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 shutdown_connections 2024-03-06T07:28:01.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 >> 172.21.15.114:0/4222100071 conn(0x7fafcc054ca0 msgr2=0x7fafcc0dbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 shutdown_connections 2024-03-06T07:28:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7f8c6e800640 1 -- 172.21.15.114:0/189397712 wait complete. 2024-03-06T07:28:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7f1563e00640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f1540084800 0x7f1540086c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 shutdown_connections 2024-03-06T07:28:01.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7fafd9600640 1 -- 172.21.15.114:0/4222100071 wait complete. 2024-03-06T07:28:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.806+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== osd.3 v2:172.21.15.114:6818/735520392 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1554021e50 con 0x7f1540084800 2024-03-06T07:28:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f21d8073720 msgr2=0x7f21d8075b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7f21ff600640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f21d8073720 0x7f21d8075b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f21d806b520 msgr2=0x7f21d806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f21ff600640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f21d806b520 0x7f21d806d9e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21f81507c0 tx=0x7f21e4009190 comp rx=0 tx=0).stop 2024-03-06T07:28:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21f806f070 msgr2=0x7f21f8150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f21ff600640 1 --2- 172.21.15.114:0/1479160153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21f806f070 0x7f21f8150280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00b4a0 tx=0x7f21ec00b960 comp rx=0 tx=0).stop 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f21fd600640 1 -- 172.21.15.114:0/1479160153 reap_dead start 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x7f14400736a0 msgr2=0x7f1440075ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.805+0000 7f1465600640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x7f14400736a0 0x7f1440075ac0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144006b500 msgr2=0x7f144006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f1465600640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144006b500 0x7f144006d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f144c0091d0 tx=0x7f144c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 msgr2=0x7f1460160ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f1465600640 1 --2- 172.21.15.114:0/533784462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146015dfc0 0x7f1460160ee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f145800efc0 tx=0x7f145800c490 comp rx=0 tx=0).stop 2024-03-06T07:28:01.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f145f400640 1 -- 172.21.15.114:0/533784462 reap_dead start 2024-03-06T07:28:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 shutdown_connections 2024-03-06T07:28:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 >> 172.21.15.114:0/533784462 conn(0x7f146008d7f0 msgr2=0x7f146006c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.807+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 shutdown_connections 2024-03-06T07:28:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.808+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 shutdown_connections 2024-03-06T07:28:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.808+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 >> 172.21.15.114:0/1479160153 conn(0x7f21f80d0b90 msgr2=0x7f21f80cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.808+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 shutdown_connections 2024-03-06T07:28:01.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.808+0000 7f21ff600640 1 -- 172.21.15.114:0/1479160153 wait complete. 2024-03-06T07:28:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.808+0000 7f1465600640 1 -- 172.21.15.114:0/533784462 wait complete. 2024-03-06T07:28:01.830 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:01.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:28:01.831 INFO:teuthology.orchestra.run.smithi114.stdout:55834574853 2024-03-06T07:28:01.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:28:01.833 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:01.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:28:01.833 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:01.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:28:01.847 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.841+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f15640cd0a0 con 0x7f1540084800 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.842+0000 7f155be00640 1 -- 172.21.15.114:0/618813594 <== osd.3 v2:172.21.15.114:6818/735520392 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f15640cd0a0 con 0x7f1540084800 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.842+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f1540084800 msgr2=0x7f1540086c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.842+0000 7f156a600640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f1540084800 0x7f1540086c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f154007c5e0 msgr2=0x7f154007eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f154007c5e0 0x7f154007eaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1564062b90 tx=0x7f154c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1548004360 msgr2=0x7f156414b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 --2- 172.21.15.114:0/618813594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1548004360 0x7f156414b390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f155400c440 tx=0x7f155400c900 comp rx=0 tx=0).stop 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f1563e00640 1 -- 172.21.15.114:0/618813594 reap_dead start 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 shutdown_connections 2024-03-06T07:28:01.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 >> 172.21.15.114:0/618813594 conn(0x7f15640cc4f0 msgr2=0x7f15640c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 shutdown_connections 2024-03-06T07:28:01.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:01.843+0000 7f156a600640 1 -- 172.21.15.114:0/618813594 wait complete. 2024-03-06T07:28:01.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:28:01.849 INFO:teuthology.orchestra.run.smithi114.stdout:55834574853 2024-03-06T07:28:01.850 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:01.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:28:01.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:28:01.866 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:01.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:28:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.083+0000 7f5176600640 1 Processor -- start 2024-03-06T07:28:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.083+0000 7f5176600640 1 -- start start 2024-03-06T07:28:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.084+0000 7f5176600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f517015e360 0x7f51700cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.084+0000 7f5176600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f5170161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.085+0000 7f5176600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51700cf2b0 con 0x7f5170162270 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.085+0000 7f5176600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51700cf590 con 0x7f517015e360 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.085+0000 7f5176600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51700cf870 con 0x7f51700d12e0 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.085+0000 7f516f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f517015e360 0x7f51700cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.085+0000 7f516f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f517015e360 0x7f51700cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55894/0 (socket says 172.21.15.114:55894) 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.085+0000 7f516f400640 1 -- 172.21.15.114:0/3919227319 learned_addr learned my addr 172.21.15.114:0/3919227319 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.086+0000 7f516f400640 1 -- 172.21.15.114:0/3919227319 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 msgr2=0x7f5170161d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.086+0000 7f516f400640 1 --2- 172.21.15.114:0/3919227319 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f5170161d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.086+0000 7f516f400640 1 -- 172.21.15.114:0/3919227319 >> v1:172.21.15.177:6789/0 conn(0x7f5170162270 legacy=0x7f517015e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.086+0000 7f516f400640 1 -- 172.21.15.114:0/3919227319 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51700cfb50 con 0x7f517015e360 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.087+0000 7f516f400640 1 --2- 172.21.15.114:0/3919227319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f517015e360 0x7f51700cec10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5160009870 tx=0x7f516002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f265857f04ac5e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.088+0000 7f913b200640 1 Processor -- start 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.088+0000 7f516e000640 1 -- 172.21.15.114:0/3919227319 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160034070 con 0x7f517015e360 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.089+0000 7f516e000640 1 -- 172.21.15.114:0/3919227319 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5160030860 con 0x7f517015e360 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.089+0000 7f516e000640 1 -- 172.21.15.114:0/3919227319 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160030c30 con 0x7f517015e360 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.090+0000 7f5176600640 1 -- 172.21.15.114:0/3919227319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f517015e360 msgr2=0x7f51700cec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.090+0000 7f5176600640 1 --2- 172.21.15.114:0/3919227319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f517015e360 0x7f51700cec10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5160009870 tx=0x7f516002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.088+0000 7f913b200640 1 -- start start 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.090+0000 7f913b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c0b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.090+0000 7f913b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0af300 con 0x7f913c0b3a60 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.090+0000 7f913b200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0aed40 con 0x7f913c1623d0 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.090+0000 7f913b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0aeaf0 con 0x7f913c0ae6b0 2024-03-06T07:28:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.091+0000 7f9139400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c0b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.091+0000 7f9139400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c0b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59082/0 (socket says 172.21.15.114:59082) 2024-03-06T07:28:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.091+0000 7f9139e00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f913c1623d0 0x7f913c0d0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:47954/0 (socket says 172.21.15.114:47954) 2024-03-06T07:28:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.091+0000 7f9139e00640 1 -- 172.21.15.114:0/872757951 learned_addr learned my addr 172.21.15.114:0/872757951 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.091+0000 7f5176600640 1 -- 172.21.15.114:0/3919227319 shutdown_connections 2024-03-06T07:28:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.092+0000 7f5176600640 1 --2- 172.21.15.114:0/3919227319 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f5170161d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.092+0000 7f5176600640 1 --2- 172.21.15.114:0/3919227319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f517015e360 0x7f51700cec10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.092+0000 7f9139400640 1 -- 172.21.15.114:0/872757951 >> v1:172.21.15.177:6789/0 conn(0x7f913c1623d0 legacy=0x7f913c0d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.092+0000 7f9139400640 1 -- 172.21.15.114:0/872757951 >> v1:172.21.15.114:6789/0 conn(0x7f913c0b3a60 legacy=0x7f913c0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.092+0000 7f5176600640 1 -- 172.21.15.114:0/3919227319 >> 172.21.15.114:0/3919227319 conn(0x7f51700d3f50 msgr2=0x7f51700d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.093+0000 7f9139400640 1 -- 172.21.15.114:0/872757951 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0af020 con 0x7f913c0ae6b0 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.093+0000 7f5176600640 1 -- 172.21.15.114:0/3919227319 shutdown_connections 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.093+0000 7f5176600640 1 -- 172.21.15.114:0/3919227319 wait complete. 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.095+0000 7f9139400640 1 --2- 172.21.15.114:0/872757951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c0b3520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f91280097d0 tx=0x7f912802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1f1ac74bb1ece9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.096+0000 7f9138a00640 1 -- 172.21.15.114:0/872757951 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128035020 con 0x7f913c0ae6b0 2024-03-06T07:28:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.096+0000 7f9138a00640 1 -- 172.21.15.114:0/872757951 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9128031810 con 0x7f913c0ae6b0 2024-03-06T07:28:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.096+0000 7f9138a00640 1 -- 172.21.15.114:0/872757951 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128031be0 con 0x7f913c0ae6b0 2024-03-06T07:28:02.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.097+0000 7f5176600640 1 Processor -- start 2024-03-06T07:28:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.097+0000 7f5176600640 1 -- start start 2024-03-06T07:28:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.097+0000 7f913b200640 1 -- 172.21.15.114:0/872757951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 msgr2=0x7f913c0b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.097+0000 7f913b200640 1 --2- 172.21.15.114:0/872757951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c0b3520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f91280097d0 tx=0x7f912802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.098+0000 7f913b200640 1 -- 172.21.15.114:0/872757951 shutdown_connections 2024-03-06T07:28:02.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.098+0000 7f913b200640 1 --2- 172.21.15.114:0/872757951 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c0b3520 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.098+0000 7f913b200640 1 -- 172.21.15.114:0/872757951 >> 172.21.15.114:0/872757951 conn(0x7f913c06afc0 msgr2=0x7f913c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.098+0000 7ff624c00640 1 Processor -- start 2024-03-06T07:28:02.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.098+0000 7ff624c00640 1 -- start start 2024-03-06T07:28:02.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.099+0000 7f913b200640 1 -- 172.21.15.114:0/872757951 shutdown_connections 2024-03-06T07:28:02.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.099+0000 7f913b200640 1 -- 172.21.15.114:0/872757951 wait complete. 2024-03-06T07:28:02.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.099+0000 7f5fada00640 1 Processor -- start 2024-03-06T07:28:02.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.099+0000 7f5fada00640 1 -- start start 2024-03-06T07:28:02.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7ff624c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff6180e3160 0x7ff6180e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7f5176600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f51700cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7f5176600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517015e360 0x7f51700ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7ff624c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff61803c0a0 0x7ff618058050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7ff624c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff618053cb0 0x7ff618054140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7ff624c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff618058590 con 0x7ff6180e3160 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7ff624c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff618054680 con 0x7ff61803c0a0 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7ff624c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff618058810 con 0x7ff618053cb0 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.100+0000 7ff61ea00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff61803c0a0 0x7ff618058050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7ff61f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff6180e3160 0x7ff6180e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7ff61ea00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff61803c0a0 0x7ff618058050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35720/0 (socket says 172.21.15.114:35720) 2024-03-06T07:28:02.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7ff61ea00640 1 -- 172.21.15.114:0/3729645781 learned_addr learned my addr 172.21.15.114:0/3729645781 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f516f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f51700cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f516f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f51700cdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59090/0 (socket says 172.21.15.114:59090) 2024-03-06T07:28:02.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7ff61fe00640 1 --2- 172.21.15.114:0/3729645781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff618053cb0 0x7ff618054140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5176600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5170162270 0x7f51700b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5176600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700d18c0 con 0x7f5170162270 2024-03-06T07:28:02.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5176600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700d1740 con 0x7f517015e360 2024-03-06T07:28:02.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5176600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51700d2ca0 con 0x7f51700d12e0 2024-03-06T07:28:02.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f516ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517015e360 0x7f51700ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7f516ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517015e360 0x7f51700ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35724/0 (socket says 172.21.15.114:35724) 2024-03-06T07:28:02.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7f516f400640 1 -- 172.21.15.114:0/1850605437 learned_addr learned my addr 172.21.15.114:0/1850605437 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7ff61fe00640 1 -- 172.21.15.114:0/3729645781 >> v2:172.21.15.177:3300/0 conn(0x7ff61803c0a0 msgr2=0x7ff618058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7ff61fe00640 1 --2- 172.21.15.114:0/3729645781 >> v2:172.21.15.177:3300/0 conn(0x7ff61803c0a0 0x7ff618058050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7ff61fe00640 1 -- 172.21.15.114:0/3729645781 >> v2:172.21.15.114:3300/0 conn(0x7ff6180e3160 msgr2=0x7ff6180e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7ff61fe00640 1 --2- 172.21.15.114:0/3729645781 >> v2:172.21.15.114:3300/0 conn(0x7ff6180e3160 0x7ff6180e7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7ff61fe00640 1 -- 172.21.15.114:0/3729645781 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff618058af0 con 0x7ff618053cb0 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5fada00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5fa80d0590 0x7f5fa81402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5fada00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fa80b1d60 0x7f5fa80cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5fada00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa806cd60 con 0x7f5fa81407e0 2024-03-06T07:28:02.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5fada00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa806d690 con 0x7f5fa80d0590 2024-03-06T07:28:02.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.101+0000 7f5fada00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ce390 con 0x7f5fa80b1d60 2024-03-06T07:28:02.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7f5fa6000640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f5fa81407e0 0x7f5fa80b1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:47956/0 (socket says 172.21.15.114:47956) 2024-03-06T07:28:02.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7f5fa6000640 1 -- 172.21.15.114:0/166054816 learned_addr learned my addr 172.21.15.114:0/166054816 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:02.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7f913b200640 1 Processor -- start 2024-03-06T07:28:02.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.102+0000 7f516fe00640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5170162270 0x7f51700b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.103+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839900065 0 0) 0x7f5fa806cd60 con 0x7f5fa81407e0 2024-03-06T07:28:02.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.103+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f8c003520 con 0x7f5fa81407e0 2024-03-06T07:28:02.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.103+0000 7f913b200640 1 -- start start 2024-03-06T07:28:02.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.103+0000 7ff61fe00640 1 --2- 172.21.15.114:0/3729645781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff618053cb0 0x7ff618054140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff620064ac0 tx=0x7ff6200a6a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c758709761312d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.103+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2651358878 0 0) 0x7f5f8c003520 con 0x7f5fa81407e0 2024-03-06T07:28:02.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.103+0000 7f5fa7400640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fa80b1d60 0x7f5fa80cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fa80b1d60 msgr2=0x7f5fa80cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.103+0000 7f516fe00640 1 -- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 msgr2=0x7f51700cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fa80b1d60 0x7f5fa80cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 >> v2:172.21.15.177:3300/0 conn(0x7f5fa80d0590 msgr2=0x7f5fa81402a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:28:02.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 --2- 172.21.15.114:0/166054816 >> v2:172.21.15.177:3300/0 conn(0x7f5fa80d0590 0x7f5fa81402a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa806d040 con 0x7f5fa81407e0 2024-03-06T07:28:02.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5f94001670 con 0x7f5fa81407e0 2024-03-06T07:28:02.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 >> v1:172.21.15.177:6789/0 conn(0x7f5fa81407e0 legacy=0x7f5fa80b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7ff61e000640 1 -- 172.21.15.114:0/3729645781 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200aa910 con 0x7ff618053cb0 2024-03-06T07:28:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f516fe00640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f51700cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f516fe00640 1 -- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517015e360 msgr2=0x7f51700ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f516fe00640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517015e360 0x7f51700ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f516fe00640 1 -- 172.21.15.114:0/1850605437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51700c89a0 con 0x7f5170162270 2024-03-06T07:28:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7ff61e000640 1 -- 172.21.15.114:0/3729645781 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6200aaab0 con 0x7ff618053cb0 2024-03-06T07:28:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7ff61e000640 1 -- 172.21.15.114:0/3729645781 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200b2400 con 0x7ff618053cb0 2024-03-06T07:28:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f516f400640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f51700cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f516ea00640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517015e360 0x7f51700ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f913b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c163710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f913b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f913c0b3a60 0x7f913c163c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5f8c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f8c004800 0x7f5f8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8c0052e0 0x7f5f8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa806d690 con 0x7f5f8c0052e0 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ce390 con 0x7f5f8c004800 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.104+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c00c200 con 0x7f5f8c0039b0 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f5fa7400640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fa80b1d60 0x7f5fa80cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5f8c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f913b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913c1623d0 0x7f913c158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f913b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0ce940 con 0x7f913c1623d0 2024-03-06T07:28:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f913b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c06d5c0 con 0x7f913c0b3a60 2024-03-06T07:28:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f913b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0ceac0 con 0x7f913c0ae6b0 2024-03-06T07:28:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7f913a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913c1623d0 0x7f913c158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.105+0000 7ff624c00640 1 -- 172.21.15.114:0/3729645781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff618053cb0 msgr2=0x7ff618054140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f5fa7400640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8c0052e0 0x7f5f8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f516fe00640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5170162270 0x7f51700b1b60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f516800be00 tx=0x7f516800bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f913a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913c1623d0 0x7f913c158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55910/0 (socket says 172.21.15.114:55910) 2024-03-06T07:28:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f913a800640 1 -- 172.21.15.114:0/2593255740 learned_addr learned my addr 172.21.15.114:0/2593255740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 --2- 172.21.15.114:0/3729645781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff618053cb0 0x7ff618054140 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff620064ac0 tx=0x7ff6200a6a30 comp rx=0 tx=0).stop 2024-03-06T07:28:02.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f9139400640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f913c0b3a60 0x7f913c163c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 -- 172.21.15.114:0/3729645781 shutdown_connections 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 --2- 172.21.15.114:0/3729645781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff618053cb0 0x7ff618054140 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 --2- 172.21.15.114:0/3729645781 >> v2:172.21.15.177:3300/0 conn(0x7ff61803c0a0 0x7ff618058050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 --2- 172.21.15.114:0/3729645781 >> v2:172.21.15.114:3300/0 conn(0x7ff6180e3160 0x7ff6180e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 -- 172.21.15.114:0/3729645781 >> 172.21.15.114:0/3729645781 conn(0x7ff618058ee0 msgr2=0x7ff6180592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 -- 172.21.15.114:0/3729645781 shutdown_connections 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7ff624c00640 1 -- 172.21.15.114:0/3729645781 wait complete. 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f913a800640 1 -- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 msgr2=0x7f913c163710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f913a800640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c163710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f913a800640 1 -- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f913c0b3a60 msgr2=0x7f913c163c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f913a800640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f913c0b3a60 0x7f913c163c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f913a800640 1 -- 172.21.15.114:0/2593255740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c164290 con 0x7f913c1623d0 2024-03-06T07:28:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f9139400640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f913c0b3a60 0x7f913c163c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.106+0000 7f5167400640 1 -- 172.21.15.114:0/1850605437 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168003970 con 0x7f5170162270 2024-03-06T07:28:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5167400640 1 -- 172.21.15.114:0/1850605437 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5168003b10 con 0x7f5170162270 2024-03-06T07:28:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5167400640 1 -- 172.21.15.114:0/1850605437 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168003970 con 0x7f5170162270 2024-03-06T07:28:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f913a800640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913c1623d0 0x7f913c158c40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f913400c7a0 tx=0x7f913400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f9132000640 1 -- 172.21.15.114:0/2593255740 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134003b70 con 0x7f913c1623d0 2024-03-06T07:28:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f9132000640 1 -- 172.21.15.114:0/2593255740 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9134003d30 con 0x7f913c1623d0 2024-03-06T07:28:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5fa6a00640 1 -- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f8c004800 msgr2=0x7f5f8c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:28:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f8c004800 0x7f5f8c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5fa6a00640 1 -- 172.21.15.114:0/166054816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8c0052e0 msgr2=0x7f5f8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f8c0052e0 0x7f5f8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5fa6a00640 1 -- 172.21.15.114:0/166054816 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c003520 con 0x7f5f8c0039b0 2024-03-06T07:28:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f9132000640 1 -- 172.21.15.114:0/2593255740 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913400aa90 con 0x7f913c1623d0 2024-03-06T07:28:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51700b3060 con 0x7f5170162270 2024-03-06T07:28:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.107+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51700cd4b0 con 0x7f5170162270 2024-03-06T07:28:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.108+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c164570 con 0x7f913c1623d0 2024-03-06T07:28:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.108+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c164b30 con 0x7f913c1623d0 2024-03-06T07:28:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.110+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517004f330 con 0x7f5170162270 2024-03-06T07:28:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.111+0000 7ff624c00640 1 Processor -- start 2024-03-06T07:28:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.111+0000 7ff624c00640 1 -- start start 2024-03-06T07:28:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.112+0000 7f5167400640 1 -- 172.21.15.114:0/1850605437 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5168014440 con 0x7f5170162270 2024-03-06T07:28:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.112+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c04f330 con 0x7f913c1623d0 2024-03-06T07:28:02.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.112+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5f8c003df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009860 tx=0x7f5f8c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a3395b5f886cbbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7ff624c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff61803c0a0 0x7ff6180d1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98037070 con 0x7f5f8c0039b0 2024-03-06T07:28:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7ff624c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff618053cb0 0x7ff6180e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7ff624c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6180e3160 0x7ff61803d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7f5167400640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f515006b700 0x7f515006dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7f5167400640 1 -- 172.21.15.114:0/1850605437 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5168089d90 con 0x7f5170162270 2024-03-06T07:28:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f98032620 con 0x7f5f8c0039b0 2024-03-06T07:28:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.113+0000 7f5fa5600640 1 -- 172.21.15.114:0/166054816 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f980093b0 con 0x7f5f8c0039b0 2024-03-06T07:28:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7ff61f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff61803c0a0 0x7ff6180d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7f9132000640 1 -- 172.21.15.114:0/2593255740 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f913400ac30 con 0x7f913c1623d0 2024-03-06T07:28:02.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7f516f400640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f515006b700 0x7f515006dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7ff61f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff61803c0a0 0x7ff6180d1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59148/0 (socket says 172.21.15.114:59148) 2024-03-06T07:28:02.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7ff61f400640 1 -- 172.21.15.114:0/3204516774 learned_addr learned my addr 172.21.15.114:0/3204516774 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6180e3540 con 0x7ff6180e3160 2024-03-06T07:28:02.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618006640 con 0x7ff618053cb0 2024-03-06T07:28:02.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff618056e10 con 0x7ff61803c0a0 2024-03-06T07:28:02.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7f5fada00640 1 -- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 msgr2=0x7f5f8c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.114+0000 7f5fada00640 1 --2- 172.21.15.114:0/166054816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5f8c003df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009860 tx=0x7f5f8c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:02.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7ff61ea00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff618053cb0 0x7ff6180e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7f5fa7400640 1 -- 172.21.15.114:0/166054816 reap_dead start 2024-03-06T07:28:02.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7f5fada00640 1 -- 172.21.15.114:0/166054816 shutdown_connections 2024-03-06T07:28:02.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7f5fada00640 1 -- 172.21.15.114:0/166054816 >> 172.21.15.114:0/166054816 conn(0x7f5fa80cfa70 msgr2=0x7f5fa80cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7f5fada00640 1 -- 172.21.15.114:0/166054816 shutdown_connections 2024-03-06T07:28:02.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7f5fada00640 1 -- 172.21.15.114:0/166054816 wait complete. 2024-03-06T07:28:02.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7ff61fe00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6180e3160 0x7ff61803d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7f9132000640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f911c06b520 0x7f911c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.115+0000 7ff61f400640 1 -- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff618053cb0 msgr2=0x7ff6180e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7ff61f400640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff618053cb0 0x7ff6180e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7ff61f400640 1 -- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6180e3160 msgr2=0x7ff61803d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7ff61f400640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6180e3160 0x7ff61803d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7f9132000640 1 -- 172.21.15.114:0/2593255740 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9134056960 con 0x7f913c1623d0 2024-03-06T07:28:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7ff61f400640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61804cc20 con 0x7ff61803c0a0 2024-03-06T07:28:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7ff61ea00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff618053cb0 0x7ff6180e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7f9139e00640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f911c06b520 0x7f911c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7f516f400640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f515006b700 0x7f515006dbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51600090b0 tx=0x7f516003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.116+0000 7ff61f400640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff61803c0a0 0x7ff6180d1370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff61400d620 tx=0x7ff61400dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.117+0000 7ff613400640 1 -- 172.21.15.114:0/3204516774 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61400b620 con 0x7ff61803c0a0 2024-03-06T07:28:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.117+0000 7ff613400640 1 -- 172.21.15.114:0/3204516774 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff614019070 con 0x7ff61803c0a0 2024-03-06T07:28:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.117+0000 7ff613400640 1 -- 172.21.15.114:0/3204516774 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61400ba00 con 0x7ff61803c0a0 2024-03-06T07:28:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.117+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61804cf00 con 0x7ff61803c0a0 2024-03-06T07:28:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.117+0000 7f5fada00640 1 Processor -- start 2024-03-06T07:28:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.117+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61804d4c0 con 0x7ff61803c0a0 2024-03-06T07:28:02.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.118+0000 7f5fada00640 1 -- start start 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.118+0000 7f9139e00640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f911c06b520 0x7f911c06d9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f912c001550 tx=0x7f912c007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fada00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5fa8157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fada00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fa80c6ab0 0x7f5fa80c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fada00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fa80b08f0 0x7f5fa80c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fada00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80cd7a0 con 0x7f5fa80b08f0 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fada00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80cd620 con 0x7f5fa80c6ab0 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fada00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80cd920 con 0x7f5f8c0039b0 2024-03-06T07:28:02.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fa7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fa80b08f0 0x7f5fa80c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fa7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fa80b08f0 0x7f5fa80c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55928/0 (socket says 172.21.15.114:55928) 2024-03-06T07:28:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fa7400640 1 -- 172.21.15.114:0/2651082136 learned_addr learned my addr 172.21.15.114:0/2651082136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5fa8157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff618000ed0 con 0x7ff61803c0a0 2024-03-06T07:28:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.119+0000 7f5fa6000640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fa80c6ab0 0x7f5fa80c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.120+0000 7f5fa6a00640 1 -- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fa80c6ab0 msgr2=0x7f5fa80c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.120+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fa80c6ab0 0x7f5fa80c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.120+0000 7f5fa6a00640 1 -- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fa80b08f0 msgr2=0x7f5fa80c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.120+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fa80b08f0 0x7f5fa80c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.120+0000 7f5fa6a00640 1 -- 172.21.15.114:0/2651082136 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80b3810 con 0x7f5f8c0039b0 2024-03-06T07:28:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.121+0000 7f5fa6a00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5fa8157680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00c3d0 tx=0x7f5f98004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.121+0000 7ff613400640 1 -- 172.21.15.114:0/3204516774 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff61400bba0 con 0x7ff61803c0a0 2024-03-06T07:28:02.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.121+0000 7f5f9ea00640 1 -- 172.21.15.114:0/2651082136 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98042070 con 0x7f5f8c0039b0 2024-03-06T07:28:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.122+0000 7f5f9ea00640 1 -- 172.21.15.114:0/2651082136 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9803d660 con 0x7f5f8c0039b0 2024-03-06T07:28:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.122+0000 7f5f9ea00640 1 -- 172.21.15.114:0/2651082136 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98037040 con 0x7f5f8c0039b0 2024-03-06T07:28:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.122+0000 7f5fada00640 1 -- 172.21.15.114:0/2651082136 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa80c7d40 con 0x7f5f8c0039b0 2024-03-06T07:28:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.122+0000 7f5fada00640 1 -- 172.21.15.114:0/2651082136 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa80b3d60 con 0x7f5f8c0039b0 2024-03-06T07:28:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.123+0000 7ff613400640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff60006b520 0x7ff60006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.124+0000 7ff613400640 1 -- 172.21.15.114:0/3204516774 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff61408a640 con 0x7ff61803c0a0 2024-03-06T07:28:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.124+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa804f330 con 0x7f5f8c0039b0 2024-03-06T07:28:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.124+0000 7f5167400640 1 -- 172.21.15.114:0/1850605437 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51680585b0 con 0x7f5170162270 2024-03-06T07:28:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.124+0000 7ff61ea00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff60006b520 0x7ff60006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.125+0000 7ff61ea00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff60006b520 0x7ff60006d9e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff6180e2110 tx=0x7ff60c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.126+0000 7f5f9ea00640 1 -- 172.21.15.114:0/2651082136 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f98045070 con 0x7f5f8c0039b0 2024-03-06T07:28:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.126+0000 7f5f9ea00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f8806b4d0 0x7f5f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.127+0000 7f5f9ea00640 1 -- 172.21.15.114:0/2651082136 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f980ac810 con 0x7f5f8c0039b0 2024-03-06T07:28:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.127+0000 7f5fa6000640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f8806b4d0 0x7f5f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.128+0000 7f5fa6000640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f8806b4d0 0x7f5f8806d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5f94001eb0 tx=0x7f5f94001860 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.131+0000 7f9132000640 1 -- 172.21.15.114:0/2593255740 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9134058fc0 con 0x7f913c1623d0 2024-03-06T07:28:02.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.140+0000 7ff613400640 1 -- 172.21.15.114:0/3204516774 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff618000ed0 con 0x7ff61803c0a0 2024-03-06T07:28:02.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.153+0000 7f5f9ea00640 1 -- 172.21.15.114:0/2651082136 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f9807b140 con 0x7f5f8c0039b0 2024-03-06T07:28:02.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.207+0000 7f4d69c00640 1 Processor -- start 2024-03-06T07:28:02.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.208+0000 7f4d69c00640 1 -- start start 2024-03-06T07:28:02.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.209+0000 7f4d69c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4d6415e470 0x7f4d640d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.210+0000 7f4d69c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640d1f80 con 0x7f4d640d2860 2024-03-06T07:28:02.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.210+0000 7f4d69c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640cef80 con 0x7f4d640d2480 2024-03-06T07:28:02.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.210+0000 7f4d69c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d640d4da0 con 0x7f4d6415e470 2024-03-06T07:28:02.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.210+0000 7f4d63400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4d640d2860 0x7f4d640b3350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36020/0 (socket says 172.21.15.114:36020) 2024-03-06T07:28:02.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.210+0000 7f4d63400640 1 -- 172.21.15.114:0/1047533787 learned_addr learned my addr 172.21.15.114:0/1047533787 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:02.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.210+0000 7f4d62000640 1 --2- 172.21.15.114:0/1047533787 >> v2:172.21.15.177:3300/0 conn(0x7f4d6415e470 0x7f4d640d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.212+0000 7f4d61600640 1 -- 172.21.15.114:0/1047533787 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397715262 0 0) 0x7f4d640d1f80 con 0x7f4d640d2860 2024-03-06T07:28:02.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.212+0000 7f4d61600640 1 -- 172.21.15.114:0/1047533787 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d48003520 con 0x7f4d640d2860 2024-03-06T07:28:02.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.212+0000 7f4d62000640 1 -- 172.21.15.114:0/1047533787 >> v1:172.21.15.177:6789/0 conn(0x7f4d640d2480 legacy=0x7f4d6415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.212+0000 7f4d62000640 1 -- 172.21.15.114:0/1047533787 >> v1:172.21.15.114:6789/0 conn(0x7f4d640d2860 legacy=0x7f4d640b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.212+0000 7f4d62000640 1 -- 172.21.15.114:0/1047533787 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d640cf260 con 0x7f4d6415e470 2024-03-06T07:28:02.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.215+0000 7f4d62000640 1 --2- 172.21.15.114:0/1047533787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d6415e470 0x7f4d640d4860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c009d90 tx=0x7f4d4c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b45f1cf24f513bce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.215+0000 7f4d61600640 1 -- 172.21.15.114:0/1047533787 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c039070 con 0x7f4d6415e470 2024-03-06T07:28:02.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.215+0000 7f4d61600640 1 -- 172.21.15.114:0/1047533787 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c02cc60 con 0x7f4d6415e470 2024-03-06T07:28:02.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.215+0000 7f4d61600640 1 -- 172.21.15.114:0/1047533787 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c034660 con 0x7f4d6415e470 2024-03-06T07:28:02.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.216+0000 7f4d69c00640 1 -- 172.21.15.114:0/1047533787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d6415e470 msgr2=0x7f4d640d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.216+0000 7f4d69c00640 1 --2- 172.21.15.114:0/1047533787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d6415e470 0x7f4d640d4860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c009d90 tx=0x7f4d4c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.218+0000 7f4d69c00640 1 -- 172.21.15.114:0/1047533787 shutdown_connections 2024-03-06T07:28:02.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.218+0000 7f4d69c00640 1 --2- 172.21.15.114:0/1047533787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d6415e470 0x7f4d640d4860 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.218+0000 7f4d69c00640 1 -- 172.21.15.114:0/1047533787 >> 172.21.15.114:0/1047533787 conn(0x7f4d6409d240 msgr2=0x7f4d640d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.219+0000 7f4d69c00640 1 -- 172.21.15.114:0/1047533787 shutdown_connections 2024-03-06T07:28:02.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.219+0000 7f4d69c00640 1 -- 172.21.15.114:0/1047533787 wait complete. 2024-03-06T07:28:02.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.222+0000 7f4d69c00640 1 Processor -- start 2024-03-06T07:28:02.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.222+0000 7f4d69c00640 1 -- start start 2024-03-06T07:28:02.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.224+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640d2480 0x7f4d640b8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.224+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d640d2860 0x7f4d6415d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.224+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d6415e470 0x7f4d6409dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.225+0000 7f4d69c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d640cfb00 con 0x7f4d640d2860 2024-03-06T07:28:02.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.225+0000 7f4d69c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d640cf820 con 0x7f4d640d2480 2024-03-06T07:28:02.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.225+0000 7f4d69c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d640ce9f0 con 0x7f4d6415e470 2024-03-06T07:28:02.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.226+0000 7f4d62a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640d2480 0x7f4d640b8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.226+0000 7f4d62000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d640d2860 0x7f4d6415d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.226+0000 7f4d62000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d640d2860 0x7f4d6415d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55944/0 (socket says 172.21.15.114:55944) 2024-03-06T07:28:02.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.227+0000 7f4d63400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d6415e470 0x7f4d6409dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.227+0000 7f4d63400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d6415e470 0x7f4d6409dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59172/0 (socket says 172.21.15.114:59172) 2024-03-06T07:28:02.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.227+0000 7f4d63400640 1 -- 172.21.15.114:0/2008880966 learned_addr learned my addr 172.21.15.114:0/2008880966 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.227+0000 7f4d62a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640d2480 0x7f4d640b8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35782/0 (socket says 172.21.15.114:35782) 2024-03-06T07:28:02.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.228+0000 7f4d62000640 1 -- 172.21.15.114:0/2008880966 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d6415e470 msgr2=0x7f4d6409dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.228+0000 7f4d62000640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d6415e470 0x7f4d6409dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.228+0000 7f4d62000640 1 -- 172.21.15.114:0/2008880966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640d2480 msgr2=0x7f4d640b8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.228+0000 7f4d62000640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640d2480 0x7f4d640b8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.228+0000 7f4d62000640 1 -- 172.21.15.114:0/2008880966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d640b8a70 con 0x7f4d640d2860 2024-03-06T07:28:02.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.228+0000 7f4d62a00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640d2480 0x7f4d640b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.229+0000 7f4d62000640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d640d2860 0x7f4d6415d660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c033000 tx=0x7f4d4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.230+0000 7f4d5aa00640 1 -- 172.21.15.114:0/2008880966 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c044020 con 0x7f4d640d2860 2024-03-06T07:28:02.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.230+0000 7f4d5aa00640 1 -- 172.21.15.114:0/2008880966 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c0326f0 con 0x7f4d640d2860 2024-03-06T07:28:02.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.230+0000 7f4d5aa00640 1 -- 172.21.15.114:0/2008880966 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c039040 con 0x7f4d640d2860 2024-03-06T07:28:02.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.231+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d640b8d40 con 0x7f4d640d2860 2024-03-06T07:28:02.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.232+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d640b9300 con 0x7f4d640d2860 2024-03-06T07:28:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.235+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6404f330 con 0x7f4d640d2860 2024-03-06T07:28:02.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.236+0000 7f4d5aa00640 1 -- 172.21.15.114:0/2008880966 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d4c047020 con 0x7f4d640d2860 2024-03-06T07:28:02.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.237+0000 7f4d5aa00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.237+0000 7f4d5aa00640 1 -- 172.21.15.114:0/2008880966 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d4c0aceb0 con 0x7f4d640d2860 2024-03-06T07:28:02.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.239+0000 7f4d62a00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.244+0000 7fea8fe00640 1 Processor -- start 2024-03-06T07:28:02.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.244+0000 7fea8fe00640 1 -- start start 2024-03-06T07:28:02.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.245+0000 7f4d62a00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4d4006b4d0 0x7f4d4006d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c001700 tx=0x7f4d5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.250+0000 7f4d5aa00640 1 -- 172.21.15.114:0/2008880966 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d4c04e030 con 0x7f4d640d2860 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.246+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900d0c00 0x7fea900d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.246+0000 7fea8fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea900b2e50 0x7fea900b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.246+0000 7fea8fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea900b5240 con 0x7fea9015e200 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.246+0000 7fea8fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea900d2ec0 con 0x7fea900b2e50 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.246+0000 7fea8fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea900cf4d0 con 0x7fea900d0c00 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.247+0000 7fea8e000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900d0c00 0x7fea900d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.247+0000 7fea8e000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900d0c00 0x7fea900d4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59184/0 (socket says 172.21.15.114:59184) 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.247+0000 7fea8e000640 1 -- 172.21.15.114:0/785798990 learned_addr learned my addr 172.21.15.114:0/785798990 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.247+0000 7fea8e000640 1 -- 172.21.15.114:0/785798990 >> v2:172.21.15.114:3300/0 conn(0x7fea900b2e50 msgr2=0x7fea900b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.247+0000 7fea8e000640 1 --2- 172.21.15.114:0/785798990 >> v2:172.21.15.114:3300/0 conn(0x7fea900b2e50 0x7fea900b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.247+0000 7fea8e000640 1 -- 172.21.15.114:0/785798990 >> v1:172.21.15.114:6789/0 conn(0x7fea9015e200 legacy=0x7fea900d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.247+0000 7fea8e000640 1 -- 172.21.15.114:0/785798990 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900b5500 con 0x7fea900d0c00 2024-03-06T07:28:02.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.252+0000 7fea8e000640 1 --2- 172.21.15.114:0/785798990 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900d0c00 0x7fea900d4850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009870 tx=0x7fea7c03c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f3d11c8a7d50f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.253+0000 7fea8d600640 1 -- 172.21.15.114:0/785798990 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c045070 con 0x7fea900d0c00 2024-03-06T07:28:02.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.253+0000 7fea8d600640 1 -- 172.21.15.114:0/785798990 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c041850 con 0x7fea900d0c00 2024-03-06T07:28:02.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.253+0000 7fea8d600640 1 -- 172.21.15.114:0/785798990 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c041c20 con 0x7fea900d0c00 2024-03-06T07:28:02.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.258+0000 7fea8fe00640 1 -- 172.21.15.114:0/785798990 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900d0c00 msgr2=0x7fea900d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.258+0000 7fea8fe00640 1 --2- 172.21.15.114:0/785798990 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900d0c00 0x7fea900d4850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009870 tx=0x7fea7c03c990 comp rx=0 tx=0).stop 2024-03-06T07:28:02.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.258+0000 7fea8fe00640 1 -- 172.21.15.114:0/785798990 shutdown_connections 2024-03-06T07:28:02.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.258+0000 7fea8fe00640 1 --2- 172.21.15.114:0/785798990 >> v2:172.21.15.114:3300/0 conn(0x7fea900b2e50 0x7fea900b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.259+0000 7fea8fe00640 1 --2- 172.21.15.114:0/785798990 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900d0c00 0x7fea900d4850 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.259+0000 7fea8fe00640 1 -- 172.21.15.114:0/785798990 >> 172.21.15.114:0/785798990 conn(0x7fea900d3d30 msgr2=0x7fea900d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.259+0000 7fea8fe00640 1 -- 172.21.15.114:0/785798990 shutdown_connections 2024-03-06T07:28:02.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.259+0000 7fea8fe00640 1 -- 172.21.15.114:0/785798990 wait complete. 2024-03-06T07:28:02.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.263+0000 7fea8fe00640 1 Processor -- start 2024-03-06T07:28:02.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.263+0000 7fea8fe00640 1 -- start start 2024-03-06T07:28:02.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.265+0000 7f2060800640 1 Processor -- start 2024-03-06T07:28:02.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.265+0000 7f2060800640 1 -- start start 2024-03-06T07:28:02.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.265+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900b2e50 0x7fea900c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.265+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea900d0c00 0x7fea90156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.266+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9015e200 0x7fea900c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.266+0000 7fea8fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea900cfd60 con 0x7fea900d0c00 2024-03-06T07:28:02.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.266+0000 7fea8fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea900cfa80 con 0x7fea9015e200 2024-03-06T07:28:02.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.266+0000 7fea8fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9015e5e0 con 0x7fea900b2e50 2024-03-06T07:28:02.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.267+0000 7fea8f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9015e200 0x7fea900c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.267+0000 7fea8ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900b2e50 0x7fea900c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.267+0000 7fea8ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900b2e50 0x7fea900c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59198/0 (socket says 172.21.15.114:59198) 2024-03-06T07:28:02.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.267+0000 7fea8ea00640 1 -- 172.21.15.114:0/2714153844 learned_addr learned my addr 172.21.15.114:0/2714153844 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.267+0000 7fea8e000640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea900d0c00 0x7fea90156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.267+0000 7fea8e000640 1 -- 172.21.15.114:0/2714153844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900b2e50 msgr2=0x7fea900c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.268+0000 7fea8e000640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900b2e50 0x7fea900c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.268+0000 7fea8e000640 1 -- 172.21.15.114:0/2714153844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9015e200 msgr2=0x7fea900c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.268+0000 7fea8e000640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9015e200 0x7fea900c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.268+0000 7fea8e000640 1 -- 172.21.15.114:0/2714153844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900ca150 con 0x7fea900d0c00 2024-03-06T07:28:02.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.268+0000 7fea8e000640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea900d0c00 0x7fea90156d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea7c044000 tx=0x7fea7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7f2060800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f205815dfc0 0x7f205806d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7fea86a00640 1 -- 172.21.15.114:0/2714153844 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c059070 con 0x7fea900d0c00 2024-03-06T07:28:02.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea900c8b70 con 0x7fea900d0c00 2024-03-06T07:28:02.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea900c9130 con 0x7fea900d0c00 2024-03-06T07:28:02.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7f2060800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20580ad860 con 0x7f2058161cd0 2024-03-06T07:28:02.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7f2060800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20581621b0 con 0x7f20580b1d60 2024-03-06T07:28:02.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7f2060800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2058162490 con 0x7f205815dfc0 2024-03-06T07:28:02.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.269+0000 7f205e800640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f20580b1d60 0x7f20580add40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:47972/0 (socket says 172.21.15.114:47972) 2024-03-06T07:28:02.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.270+0000 7f205e800640 1 -- 172.21.15.114:0/2791604262 learned_addr learned my addr 172.21.15.114:0/2791604262 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:02.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.270+0000 7f205de00640 1 --2- 172.21.15.114:0/2791604262 >> v2:172.21.15.177:3300/0 conn(0x7f205815dfc0 0x7f205806d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.271+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141949258 0 0) 0x7f20581621b0 con 0x7f20580b1d60 2024-03-06T07:28:02.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.271+0000 7fea86a00640 1 -- 172.21.15.114:0/2714153844 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c04ad90 con 0x7fea900d0c00 2024-03-06T07:28:02.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.271+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9004f330 con 0x7fea900d0c00 2024-03-06T07:28:02.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.271+0000 7fea86a00640 1 -- 172.21.15.114:0/2714153844 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c045040 con 0x7fea900d0c00 2024-03-06T07:28:02.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.271+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2040003520 con 0x7f20580b1d60 2024-03-06T07:28:02.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7fea86a00640 1 -- 172.21.15.114:0/2714153844 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea7c05c070 con 0x7fea900d0c00 2024-03-06T07:28:02.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006425316 0 0) 0x7f20580ad860 con 0x7f2058161cd0 2024-03-06T07:28:02.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20581621b0 con 0x7f2058161cd0 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171671844 0 0) 0x7f2040003520 con 0x7f20580b1d60 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 >> v2:172.21.15.177:3300/0 conn(0x7f205815dfc0 msgr2=0x7f205806d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 --2- 172.21.15.114:0/2791604262 >> v2:172.21.15.177:3300/0 conn(0x7f205815dfc0 0x7f205806d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 >> v1:172.21.15.114:6789/0 conn(0x7f2058161cd0 legacy=0x7f20580b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2058162770 con 0x7f20580b1d60 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2048001670 con 0x7f20580b1d60 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 >> v1:172.21.15.177:6789/0 conn(0x7f20580b1d60 legacy=0x7f20580add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003b40 0x7f2040003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2040004800 0x7f2040004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20400051c0 0x7f204000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2058162490 con 0x7f2040003b40 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204000c010 con 0x7f20400051c0 2024-03-06T07:28:02.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.273+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204000c210 con 0x7f2040004800 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.274+0000 7f205e800640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20400051c0 0x7f204000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.274+0000 7f205de00640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003b40 0x7f2040003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.274+0000 7f205e800640 1 -- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2040004800 msgr2=0x7f2040004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.274+0000 7f205e800640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2040004800 0x7f2040004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.274+0000 7f205e800640 1 -- 172.21.15.114:0/2791604262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003b40 msgr2=0x7f2040003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.274+0000 7f205e800640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003b40 0x7f2040003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.274+0000 7f205e800640 1 -- 172.21.15.114:0/2791604262 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2040003520 con 0x7f20400051c0 2024-03-06T07:28:02.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.276+0000 7f205e800640 1 -- 172.21.15.114:0/2791604262 reap_dead start 2024-03-06T07:28:02.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.277+0000 7fea86a00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b500 0x7fea7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.277+0000 7fea86a00640 1 -- 172.21.15.114:0/2714153844 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea7c0bd420 con 0x7fea900d0c00 2024-03-06T07:28:02.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.278+0000 7fea8ea00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b500 0x7fea7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.279+0000 7fea8ea00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b500 0x7fea7006d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fea800099a0 tx=0x7fea800090e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.279+0000 7f205de00640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2040003b40 0x7f2040003f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:02.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.280+0000 7f205e800640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20400051c0 0x7f204000b9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2048007fc0 tx=0x7f2040005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d76d5ef93d469b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.280+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2048007be0 con 0x7f20400051c0 2024-03-06T07:28:02.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.281+0000 7f205ca00640 1 -- 172.21.15.114:0/2791604262 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2048005af0 con 0x7f20400051c0 2024-03-06T07:28:02.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.281+0000 7f2060800640 1 -- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20400051c0 msgr2=0x7f204000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.281+0000 7f2060800640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20400051c0 0x7f204000b9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2048007fc0 tx=0x7f2040005830 comp rx=0 tx=0).stop 2024-03-06T07:28:02.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.282+0000 7f2060800640 1 -- 172.21.15.114:0/2791604262 shutdown_connections 2024-03-06T07:28:02.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.282+0000 7f2060800640 1 --2- 172.21.15.114:0/2791604262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20400051c0 0x7f204000b9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.282+0000 7f2060800640 1 -- 172.21.15.114:0/2791604262 >> 172.21.15.114:0/2791604262 conn(0x7f20580cfa70 msgr2=0x7f20580cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.286+0000 7f2060800640 1 -- 172.21.15.114:0/2791604262 shutdown_connections 2024-03-06T07:28:02.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.289+0000 7fea86a00640 1 -- 172.21.15.114:0/2714153844 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea7c08bd70 con 0x7fea900d0c00 2024-03-06T07:28:02.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.286+0000 7f2060800640 1 -- 172.21.15.114:0/2791604262 wait complete. 2024-03-06T07:28:02.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.288+0000 7f2060800640 1 Processor -- start 2024-03-06T07:28:02.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.289+0000 7f2060800640 1 -- start start 2024-03-06T07:28:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.291+0000 7f2060800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20400051c0 0x7f205814c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.291+0000 7f2060800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20580af7d0 0x7f20580afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.291+0000 7f2060800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20580c72f0 0x7f20580c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.292+0000 7f205de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20400051c0 0x7f205814c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.292+0000 7f205de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20400051c0 0x7f205814c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59200/0 (socket says 172.21.15.114:59200) 2024-03-06T07:28:02.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.292+0000 7f205d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20580af7d0 0x7f20580afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.292+0000 7f205d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20580af7d0 0x7f20580afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35810/0 (socket says 172.21.15.114:35810) 2024-03-06T07:28:02.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.293+0000 7f205de00640 1 -- 172.21.15.114:0/614405970 learned_addr learned my addr 172.21.15.114:0/614405970 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.293+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205806ce40 con 0x7f20580c72f0 2024-03-06T07:28:02.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.293+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205806ccc0 con 0x7f20580af7d0 2024-03-06T07:28:02.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.293+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20580cf090 con 0x7f20400051c0 2024-03-06T07:28:02.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.294+0000 7f205d400640 1 -- 172.21.15.114:0/614405970 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20400051c0 msgr2=0x7f205814c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.294+0000 7f205d400640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20400051c0 0x7f205814c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.294+0000 7f205d400640 1 -- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20580c72f0 msgr2=0x7f20580c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.294+0000 7f205d400640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20580c72f0 0x7f20580c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.294+0000 7f205d400640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205814e910 con 0x7f20580af7d0 2024-03-06T07:28:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.294+0000 7f205d400640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20580af7d0 0x7f20580afc30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2058062b90 tx=0x7f204c00cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.296+0000 7f2052000640 1 -- 172.21.15.114:0/614405970 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c005270 con 0x7f20580af7d0 2024-03-06T07:28:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.297+0000 7f2052000640 1 -- 172.21.15.114:0/614405970 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204c005430 con 0x7f20580af7d0 2024-03-06T07:28:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.297+0000 7f2052000640 1 -- 172.21.15.114:0/614405970 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c00a850 con 0x7f20580af7d0 2024-03-06T07:28:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.297+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205814ebb0 con 0x7f20580af7d0 2024-03-06T07:28:02.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.297+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205814f170 con 0x7f20580af7d0 2024-03-06T07:28:02.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.299+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2034005290 con 0x7f20580af7d0 2024-03-06T07:28:02.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.302+0000 7f2052000640 1 -- 172.21.15.114:0/614405970 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f204c00a9f0 con 0x7f20580af7d0 2024-03-06T07:28:02.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.304+0000 7f2052000640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f203c06b5d0 0x7f203c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.304+0000 7f2052000640 1 -- 172.21.15.114:0/614405970 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f204c08a7f0 con 0x7f20580af7d0 2024-03-06T07:28:02.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.304+0000 7f205de00640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f203c06b5d0 0x7f203c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.306+0000 7f205de00640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f203c06b5d0 0x7f203c06da90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2054009fd0 tx=0x7f205400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.306+0000 7f17ea400640 1 Processor -- start 2024-03-06T07:28:02.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.306+0000 7f17ea400640 1 -- start start 2024-03-06T07:28:02.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.308+0000 7f17ea400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f17e40d2480 0x7f17e40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.308+0000 7f17ea400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f17e415e470 0x7f17e40b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.308+0000 7f17ea400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e40d1f80 con 0x7f17e40d2860 2024-03-06T07:28:02.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.308+0000 7f17ea400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e40d4da0 con 0x7f17e415e470 2024-03-06T07:28:02.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.308+0000 7f17ea400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e40cef80 con 0x7f17e40d2480 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.309+0000 7f17e3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f17e415e470 0x7f17e40b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.309+0000 7f17e3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f17e415e470 0x7f17e40b3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56012/0 (socket says 172.21.15.114:56012) 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.309+0000 7f17e3e00640 1 -- 172.21.15.114:0/651580467 learned_addr learned my addr 172.21.15.114:0/651580467 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.309+0000 7f17e3e00640 1 -- 172.21.15.114:0/651580467 >> v2:172.21.15.177:3300/0 conn(0x7f17e40d2480 msgr2=0x7f17e40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.309+0000 7f17e3e00640 1 --2- 172.21.15.114:0/651580467 >> v2:172.21.15.177:3300/0 conn(0x7f17e40d2480 0x7f17e40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.310+0000 7f17e3e00640 1 -- 172.21.15.114:0/651580467 >> v1:172.21.15.177:6789/0 conn(0x7f17e40d2860 legacy=0x7f17e415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.310+0000 7f17e3e00640 1 -- 172.21.15.114:0/651580467 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40cf260 con 0x7f17e415e470 2024-03-06T07:28:02.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.312+0000 7f17e3e00640 1 --2- 172.21.15.114:0/651580467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e415e470 0x7f17e40b3350 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17e409d970 tx=0x7f17d00344b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46eb2d7ed8752c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.312+0000 7f17e2000640 1 -- 172.21.15.114:0/651580467 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0040070 con 0x7f17e415e470 2024-03-06T07:28:02.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.312+0000 7f17e2000640 1 -- 172.21.15.114:0/651580467 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d003b600 con 0x7f17e415e470 2024-03-06T07:28:02.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.312+0000 7f17e2000640 1 -- 172.21.15.114:0/651580467 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0040070 con 0x7f17e415e470 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.313+0000 7f17ea400640 1 -- 172.21.15.114:0/651580467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e415e470 msgr2=0x7f17e40b3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.313+0000 7f17ea400640 1 --2- 172.21.15.114:0/651580467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e415e470 0x7f17e40b3350 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17e409d970 tx=0x7f17d00344b0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.314+0000 7f17ea400640 1 -- 172.21.15.114:0/651580467 shutdown_connections 2024-03-06T07:28:02.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.314+0000 7f17ea400640 1 --2- 172.21.15.114:0/651580467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e415e470 0x7f17e40b3350 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.314+0000 7f17ea400640 1 --2- 172.21.15.114:0/651580467 >> v2:172.21.15.177:3300/0 conn(0x7f17e40d2480 0x7f17e40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.314+0000 7f17ea400640 1 -- 172.21.15.114:0/651580467 >> 172.21.15.114:0/651580467 conn(0x7f17e409d240 msgr2=0x7f17e40d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.317+0000 7f17ea400640 1 -- 172.21.15.114:0/651580467 shutdown_connections 2024-03-06T07:28:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.317+0000 7f17ea400640 1 -- 172.21.15.114:0/651580467 wait complete. 2024-03-06T07:28:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.320+0000 7f17ea400640 1 Processor -- start 2024-03-06T07:28:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.320+0000 7f17ea400640 1 -- start start 2024-03-06T07:28:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.321+0000 7f2052000640 1 -- 172.21.15.114:0/614405970 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f204c039030 con 0x7f20580af7d0 2024-03-06T07:28:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.322+0000 7f17ea400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e40d2480 0x7f17e40cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.322+0000 7f17ea400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17e40d2860 0x7f17e40c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.322+0000 7f17ea400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17e415e470 0x7f17e4158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.323+0000 7f17ea400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40cfb00 con 0x7f17e40d2480 2024-03-06T07:28:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.323+0000 7f17ea400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40cf820 con 0x7f17e415e470 2024-03-06T07:28:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.323+0000 7f17ea400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40ce9f0 con 0x7f17e40d2860 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.324+0000 7f17e3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17e415e470 0x7f17e4158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.324+0000 7f17e3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17e415e470 0x7f17e4158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35828/0 (socket says 172.21.15.114:35828) 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.324+0000 7f17e3e00640 1 -- 172.21.15.114:0/1891194422 learned_addr learned my addr 172.21.15.114:0/1891194422 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.324+0000 7f17e3400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e40d2480 0x7f17e40cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.324+0000 7f17e2a00640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17e40d2860 0x7f17e40c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.325+0000 7f17e3e00640 1 -- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17e40d2860 msgr2=0x7f17e40c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.325+0000 7f17e3e00640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17e40d2860 0x7f17e40c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.325+0000 7f17e3e00640 1 -- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e40d2480 msgr2=0x7f17e40cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.325+0000 7f17e3e00640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e40d2480 0x7f17e40cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.325+0000 7f17e3e00640 1 -- 172.21.15.114:0/1891194422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e4159100 con 0x7f17e415e470 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.325+0000 7f17e3400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e40d2480 0x7f17e40cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.325+0000 7f17e3e00640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17e415e470 0x7f17e4158060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17d003e040 tx=0x7f17d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.326+0000 7f17db400640 1 -- 172.21.15.114:0/1891194422 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0040070 con 0x7f17e415e470 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.326+0000 7f17db400640 1 -- 172.21.15.114:0/1891194422 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d0014680 con 0x7f17e415e470 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.326+0000 7f17db400640 1 -- 172.21.15.114:0/1891194422 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0003810 con 0x7f17e415e470 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.326+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e406d220 con 0x7f17e415e470 2024-03-06T07:28:02.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.327+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e406d7e0 con 0x7f17e415e470 2024-03-06T07:28:02.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.329+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e404f330 con 0x7f17e415e470 2024-03-06T07:28:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.330+0000 7f17db400640 1 -- 172.21.15.114:0/1891194422 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17d00039b0 con 0x7f17e415e470 2024-03-06T07:28:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.331+0000 7f17db400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17c406b520 0x7f17c406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.332+0000 7f17db400640 1 -- 172.21.15.114:0/1891194422 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17d0079760 con 0x7f17e415e470 2024-03-06T07:28:02.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.332+0000 7f17e3400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17c406b520 0x7f17c406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.333+0000 7f17e3400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17c406b520 0x7f17c406d9e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17dc007eb0 tx=0x7f17dc0073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.355+0000 7f17db400640 1 -- 172.21.15.114:0/1891194422 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17d007bdc0 con 0x7f17e415e470 2024-03-06T07:28:02.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.539+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f913c0ccc10 con 0x7f913c1623d0 2024-03-06T07:28:02.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.540+0000 7f9132000640 1 -- 172.21.15.114:0/2593255740 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91340588c0 con 0x7f913c1623d0 2024-03-06T07:28:02.543 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:02.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.543+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f911c06b520 msgr2=0x7f911c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.544+0000 7f913b200640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f911c06b520 0x7f911c06d9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f912c001550 tx=0x7f912c007a40 comp rx=0 tx=0).stop 2024-03-06T07:28:02.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.544+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913c1623d0 msgr2=0x7f913c158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.544+0000 7f913b200640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913c1623d0 0x7f913c158c40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f913400c7a0 tx=0x7f913400cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:02.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.545+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 shutdown_connections 2024-03-06T07:28:02.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.545+0000 7f913b200640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f911c06b520 0x7f911c06d9e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.545+0000 7f913b200640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f913c1623d0 0x7f913c158c40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.545+0000 7f913b200640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f913c0b3a60 0x7f913c163c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.545+0000 7f913b200640 1 --2- 172.21.15.114:0/2593255740 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f913c0ae6b0 0x7f913c163710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.545+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 >> 172.21.15.114:0/2593255740 conn(0x7f913c06afc0 msgr2=0x7f913c0ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.546+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 shutdown_connections 2024-03-06T07:28:02.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.546+0000 7f913b200640 1 -- 172.21.15.114:0/2593255740 wait complete. 2024-03-06T07:28:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.573+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff618005390 con 0x7ff61803c0a0 2024-03-06T07:28:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.573+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f517008d9b0 con 0x7f5170162270 2024-03-06T07:28:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.574+0000 7ff613400640 1 -- 172.21.15.114:0/3204516774 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff614059510 con 0x7ff61803c0a0 2024-03-06T07:28:02.576 INFO:teuthology.orchestra.run.smithi114.stdout:55834574853 2024-03-06T07:28:02.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-03-06T07:28:02.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.576+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff60006b520 msgr2=0x7ff60006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.576+0000 7ff624c00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff60006b520 0x7ff60006d9e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff6180e2110 tx=0x7ff60c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.576+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff61803c0a0 msgr2=0x7ff6180d1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.576+0000 7ff624c00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff61803c0a0 0x7ff6180d1370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff61400d620 tx=0x7ff61400dae0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.577+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 shutdown_connections 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.577+0000 7ff624c00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff60006b520 0x7ff60006d9e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.577+0000 7ff624c00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6180e3160 0x7ff61803d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.577+0000 7ff624c00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff618053cb0 0x7ff6180e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.577+0000 7ff624c00640 1 --2- 172.21.15.114:0/3204516774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff61803c0a0 0x7ff6180d1370 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff61400d620 tx=0x7ff61400dae0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.577+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 >> 172.21.15.114:0/3204516774 conn(0x7ff618058ee0 msgr2=0x7ff618038940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.576+0000 7f5167400640 1 -- 172.21.15.114:0/1850605437 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f516805c650 con 0x7f5170162270 2024-03-06T07:28:02.581 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:02.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.578+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 shutdown_connections 2024-03-06T07:28:02.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.578+0000 7ff624c00640 1 -- 172.21.15.114:0/3204516774 wait complete. 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f515006b700 msgr2=0x7f515006dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f515006b700 0x7f515006dbc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51600090b0 tx=0x7f516003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5170162270 msgr2=0x7f51700b1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5170162270 0x7f51700b1b60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f516800be00 tx=0x7f516800bef0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 shutdown_connections 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f515006b700 0x7f515006dbc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5170162270 0x7f51700b1b60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f517015e360 0x7f51700ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 --2- 172.21.15.114:0/1850605437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51700d12e0 0x7f51700cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.580+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 >> 172.21.15.114:0/1850605437 conn(0x7f51700d3f50 msgr2=0x7f517015bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.581+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 shutdown_connections 2024-03-06T07:28:02.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.582+0000 7f5176600640 1 -- 172.21.15.114:0/1850605437 wait complete. 2024-03-06T07:28:02.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-03-06T07:28:02.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-06T07:28:02.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.607+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5fa80b3f30 con 0x7f5f8c0039b0 2024-03-06T07:28:02.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.607+0000 7f5f9ea00640 1 -- 172.21.15.114:0/2651082136 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f9807f1e0 con 0x7f5f8c0039b0 2024-03-06T07:28:02.610 INFO:teuthology.orchestra.run.smithi114.stdout:55834574852 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.610+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f8806b4d0 msgr2=0x7f5f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.610+0000 7f5f93e00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f8806b4d0 0x7f5f8806d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5f94001eb0 tx=0x7f5f94001860 comp rx=0 tx=0).stop 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.610+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 msgr2=0x7f5fa8157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.610+0000 7f5f93e00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5fa8157680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00c3d0 tx=0x7f5f98004700 comp rx=0 tx=0).stop 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 shutdown_connections 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f8806b4d0 0x7f5f8806d990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fa80b08f0 0x7f5fa80c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fa80c6ab0 0x7f5fa80c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 --2- 172.21.15.114:0/2651082136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f8c0039b0 0x7f5fa8157680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 >> 172.21.15.114:0/2651082136 conn(0x7f5fa80cfa70 msgr2=0x7f5fa81582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 shutdown_connections 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.611+0000 7f5f93e00640 1 -- 172.21.15.114:0/2651082136 wait complete. 2024-03-06T07:28:02.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2024-03-06T07:28:02.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.676+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2034005850 con 0x7f20580af7d0 2024-03-06T07:28:02.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.677+0000 7f2052000640 1 -- 172.21.15.114:0/614405970 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f204c019300 con 0x7f20580af7d0 2024-03-06T07:28:02.680 INFO:teuthology.orchestra.run.smithi114.stdout:55834574853 2024-03-06T07:28:02.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.679+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f203c06b5d0 msgr2=0x7f203c06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.679+0000 7f2060800640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f203c06b5d0 0x7f203c06da90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2054009fd0 tx=0x7f205400b040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.679+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20580af7d0 msgr2=0x7f20580afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.679+0000 7f2060800640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20580af7d0 0x7f20580afc30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2058062b90 tx=0x7f204c00cb90 comp rx=0 tx=0).stop 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 shutdown_connections 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f203c06b5d0 0x7f203c06da90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20580c72f0 0x7f20580c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20580af7d0 0x7f20580afc30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 --2- 172.21.15.114:0/614405970 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f20400051c0 0x7f205814c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 >> 172.21.15.114:0/614405970 conn(0x7f20580cfa70 msgr2=0x7f2058160820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 shutdown_connections 2024-03-06T07:28:02.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.680+0000 7f2060800640 1 -- 172.21.15.114:0/614405970 wait complete. 2024-03-06T07:28:02.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.688+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4d640d1490 con 0x7f4d640d2860 2024-03-06T07:28:02.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.688+0000 7f4d5aa00640 1 -- 172.21.15.114:0/2008880966 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d4c07bd20 con 0x7f4d640d2860 2024-03-06T07:28:02.691 INFO:teuthology.orchestra.run.smithi114.stdout:55834574852 2024-03-06T07:28:02.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.690+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4d4006b4d0 msgr2=0x7f4d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.690+0000 7f4d69c00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4d4006b4d0 0x7f4d4006d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c001700 tx=0x7f4d5c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.690+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d640d2860 msgr2=0x7f4d6415d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.690+0000 7f4d69c00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d640d2860 0x7f4d6415d660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c033000 tx=0x7f4d4c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 shutdown_connections 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4d6415e470 0x7f4d6409dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d640d2860 0x7f4d6415d660 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 --2- 172.21.15.114:0/2008880966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d640d2480 0x7f4d640b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 >> 172.21.15.114:0/2008880966 conn(0x7f4d6409d240 msgr2=0x7f4d640b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 shutdown_connections 2024-03-06T07:28:02.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.691+0000 7f4d69c00640 1 -- 172.21.15.114:0/2008880966 wait complete. 2024-03-06T07:28:02.706 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-03-06T07:28:02.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.705+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fea90068340 con 0x7fea900d0c00 2024-03-06T07:28:02.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.705+0000 7fea86a00640 1 -- 172.21.15.114:0/2714153844 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea7c08fe10 con 0x7fea900d0c00 2024-03-06T07:28:02.708 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:02.710 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.707+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b500 msgr2=0x7fea7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.707+0000 7fea8fe00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b500 0x7fea7006d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fea800099a0 tx=0x7fea800090e0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.707+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea900d0c00 msgr2=0x7fea90156d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.707+0000 7fea8fe00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea900d0c00 0x7fea90156d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea7c044000 tx=0x7fea7c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 shutdown_connections 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9015e200 0x7fea900c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b500 0x7fea7006d9c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea900d0c00 0x7fea90156d90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 --2- 172.21.15.114:0/2714153844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea900b2e50 0x7fea900c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 >> 172.21.15.114:0/2714153844 conn(0x7fea900d3d30 msgr2=0x7fea900b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 shutdown_connections 2024-03-06T07:28:02.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.708+0000 7fea8fe00640 1 -- 172.21.15.114:0/2714153844 wait complete. 2024-03-06T07:28:02.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-03-06T07:28:02.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.783+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f17e40d1490 con 0x7f17e415e470 2024-03-06T07:28:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.784+0000 7f17db400640 1 -- 172.21.15.114:0/1891194422 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f17d007b6c0 con 0x7f17e415e470 2024-03-06T07:28:02.787 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.787+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17c406b520 msgr2=0x7f17c406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.787+0000 7f17ea400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17c406b520 0x7f17c406d9e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17dc007eb0 tx=0x7f17dc0073c0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.787+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17e415e470 msgr2=0x7f17e4158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.787+0000 7f17ea400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17e415e470 0x7f17e4158060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17d003e040 tx=0x7f17d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 shutdown_connections 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17e415e470 0x7f17e4158060 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17e40d2860 0x7f17e40c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17c406b520 0x7f17c406d9e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 --2- 172.21.15.114:0/1891194422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17e40d2480 0x7f17e40cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 >> 172.21.15.114:0/1891194422 conn(0x7f17e409d240 msgr2=0x7f17e40d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 shutdown_connections 2024-03-06T07:28:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:02.788+0000 7f17ea400640 1 -- 172.21.15.114:0/1891194422 wait complete. 2024-03-06T07:28:02.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2024-03-06T07:28:03.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:28:03.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:28:03.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:28:03.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:28:03.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:28:03.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:28:03.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:28:03.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.847+0000 7f08f1400640 1 Processor -- start 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.847+0000 7f08f1400640 1 -- start start 2024-03-06T07:28:03.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.849+0000 7f08f1400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f08ec154400 0x7f08ec0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.849+0000 7f08f1400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08ec06ec70 0x7f08ec0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.849+0000 7f08f1400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0ca1f0 con 0x7f08ec1547e0 2024-03-06T07:28:03.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.849+0000 7f08f1400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec0ca4d0 con 0x7f08ec06ec70 2024-03-06T07:28:03.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.849+0000 7f08f1400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec0ca7b0 con 0x7f08ec154400 2024-03-06T07:28:03.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.852+0000 7f08e9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08ec06ec70 0x7f08ec0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.852+0000 7f08eb200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f08ec1547e0 0x7f08ec0cd770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36058/0 (socket says 172.21.15.114:36058) 2024-03-06T07:28:03.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.852+0000 7f08eb200640 1 -- 172.21.15.114:0/3287764793 learned_addr learned my addr 172.21.15.114:0/3287764793 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:03.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.853+0000 7f08e9e00640 1 -- 172.21.15.114:0/3287764793 >> v2:172.21.15.177:3300/0 conn(0x7f08ec154400 msgr2=0x7f08ec0cba30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:28:03.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.853+0000 7f08e9e00640 1 --2- 172.21.15.114:0/3287764793 >> v2:172.21.15.177:3300/0 conn(0x7f08ec154400 0x7f08ec0cba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.853+0000 7f08e9e00640 1 -- 172.21.15.114:0/3287764793 >> v1:172.21.15.114:6789/0 conn(0x7f08ec1547e0 legacy=0x7f08ec0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.853+0000 7f08e9e00640 1 -- 172.21.15.114:0/3287764793 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec0caa90 con 0x7f08ec06ec70 2024-03-06T07:28:03.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.855+0000 7f08e9e00640 1 --2- 172.21.15.114:0/3287764793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec06ec70 0x7f08ec0cbf70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0cd740 tx=0x7f08d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ae7113d04d6bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.856+0000 7f08e9400640 1 -- 172.21.15.114:0/3287764793 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d8034070 con 0x7f08ec06ec70 2024-03-06T07:28:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.856+0000 7f08e9400640 1 -- 172.21.15.114:0/3287764793 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d8030860 con 0x7f08ec06ec70 2024-03-06T07:28:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.856+0000 7f08e9400640 1 -- 172.21.15.114:0/3287764793 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d8034070 con 0x7f08ec06ec70 2024-03-06T07:28:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.857+0000 7f08f1400640 1 -- 172.21.15.114:0/3287764793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec06ec70 msgr2=0x7f08ec0cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.857+0000 7f08f1400640 1 --2- 172.21.15.114:0/3287764793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec06ec70 0x7f08ec0cbf70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0cd740 tx=0x7f08d802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.858+0000 7f08f1400640 1 -- 172.21.15.114:0/3287764793 shutdown_connections 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.858+0000 7f08f1400640 1 --2- 172.21.15.114:0/3287764793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec06ec70 0x7f08ec0cbf70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.858+0000 7f08f1400640 1 --2- 172.21.15.114:0/3287764793 >> v2:172.21.15.177:3300/0 conn(0x7f08ec154400 0x7f08ec0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.858+0000 7f08f1400640 1 -- 172.21.15.114:0/3287764793 >> 172.21.15.114:0/3287764793 conn(0x7f08ec0cc4f0 msgr2=0x7f08ec0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.858+0000 7f08f1400640 1 -- 172.21.15.114:0/3287764793 shutdown_connections 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.858+0000 7f08f1400640 1 -- 172.21.15.114:0/3287764793 wait complete. 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.861+0000 7f08f1400640 1 Processor -- start 2024-03-06T07:28:03.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.862+0000 7f08f1400640 1 -- start start 2024-03-06T07:28:03.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.864+0000 7f08f1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec06ec70 0x7f08ec15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.865+0000 7f08f1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec154400 0x7f08ec148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.865+0000 7f08f1400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08ec1547e0 0x7f08ec148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.865+0000 7f08ea800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec06ec70 0x7f08ec15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.865+0000 7f08ea800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec06ec70 0x7f08ec15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35850/0 (socket says 172.21.15.114:35850) 2024-03-06T07:28:03.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.866+0000 7f08f1400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0cde80 con 0x7f08ec154400 2024-03-06T07:28:03.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.866+0000 7f08f1400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0cb290 con 0x7f08ec06ec70 2024-03-06T07:28:03.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.866+0000 7f08f1400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0ce000 con 0x7f08ec1547e0 2024-03-06T07:28:03.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.866+0000 7f08e9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec154400 0x7f08ec148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.866+0000 7f08eb200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08ec1547e0 0x7f08ec148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.870+0000 7f2540a00640 1 Processor -- start 2024-03-06T07:28:03.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.870+0000 7f2540a00640 1 -- start start 2024-03-06T07:28:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.866+0000 7f08e9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec154400 0x7f08ec148540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56042/0 (socket says 172.21.15.114:56042) 2024-03-06T07:28:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.866+0000 7f08e9e00640 1 -- 172.21.15.114:0/917006594 learned_addr learned my addr 172.21.15.114:0/917006594 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.867+0000 7f08eb200640 1 -- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec06ec70 msgr2=0x7f08ec15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.867+0000 7f08eb200640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec06ec70 0x7f08ec15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.867+0000 7f08eb200640 1 -- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec154400 msgr2=0x7f08ec148540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.867+0000 7f08eb200640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec154400 0x7f08ec148540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.867+0000 7f08eb200640 1 -- 172.21.15.114:0/917006594 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec14d210 con 0x7f08ec1547e0 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.868+0000 7f08e9e00640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec154400 0x7f08ec148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.868+0000 7f08eb200640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08ec1547e0 0x7f08ec148a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c1b0 tx=0x7f08dc00c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.869+0000 7f08ea800640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec06ec70 0x7f08ec15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.869+0000 7f08e2a00640 1 -- 172.21.15.114:0/917006594 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc017070 con 0x7f08ec1547e0 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.869+0000 7f08e2a00640 1 -- 172.21.15.114:0/917006594 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc005ce0 con 0x7f08ec1547e0 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.869+0000 7f08e2a00640 1 -- 172.21.15.114:0/917006594 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0072f0 con 0x7f08ec1547e0 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.869+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec14d4f0 con 0x7f08ec1547e0 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.869+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec14dc50 con 0x7f08ec1547e0 2024-03-06T07:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.872+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c4005290 con 0x7f08ec1547e0 2024-03-06T07:28:03.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.873+0000 7f7a87800640 1 Processor -- start 2024-03-06T07:28:03.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.875+0000 7f08e2a00640 1 -- 172.21.15.114:0/917006594 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08dc01a070 con 0x7f08ec1547e0 2024-03-06T07:28:03.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.874+0000 7f7a87800640 1 -- start start 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.875+0000 7f7a87800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7a800d2480 0x7f7a800d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.875+0000 7f7a87800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7a800d2860 0x7f7a8015dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.875+0000 7f7a87800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800ce9f0 0x7f7a8015e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.875+0000 7f7a87800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a800d4da0 con 0x7f7a800d2480 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.875+0000 7f7a87800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a800d1f80 con 0x7f7a800d2860 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.875+0000 7f7a87800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800cef80 con 0x7f7a800ce9f0 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.876+0000 7f7a84e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7a800d2480 0x7f7a800d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.876+0000 7f7a84e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7a800d2480 0x7f7a800d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56050/0 (socket says 172.21.15.114:56050) 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.876+0000 7f7a84e00640 1 -- 172.21.15.114:0/1773920052 learned_addr learned my addr 172.21.15.114:0/1773920052 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.876+0000 7f7a85800640 1 --2- 172.21.15.114:0/1773920052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800ce9f0 0x7f7a8015e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.876+0000 7f7a77e00640 1 --2- 172.21.15.114:0/1773920052 >> v2:172.21.15.177:3300/0 conn(0x7f7a800d2860 0x7f7a8015dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.877+0000 7f2540a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f253c154870 0x7f253c0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.877+0000 7f2540a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f253c0cdcb0 0x7f253c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.877+0000 7f2540a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253c0ca1f0 con 0x7f253c154490 2024-03-06T07:28:03.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.877+0000 7f2540a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c0ce120 con 0x7f253c154870 2024-03-06T07:28:03.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.877+0000 7f2540a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c0ca4d0 con 0x7f253c0cdcb0 2024-03-06T07:28:03.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f08e2a00640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08cc06b5a0 0x7f08cc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f08e2a00640 1 -- 172.21.15.114:0/917006594 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08dc0892d0 con 0x7f08ec1547e0 2024-03-06T07:28:03.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f253a800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f253c0cdcb0 0x7f253c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f253a800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f253c0cdcb0 0x7f253c06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35876/0 (socket says 172.21.15.114:35876) 2024-03-06T07:28:03.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f253a800640 1 -- 172.21.15.114:0/861450109 learned_addr learned my addr 172.21.15.114:0/861450109 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:03.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.879+0000 7f2539400640 1 --2- 172.21.15.114:0/861450109 >> v2:172.21.15.114:3300/0 conn(0x7f253c154870 0x7f253c0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f7a84e00640 1 -- 172.21.15.114:0/1773920052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800ce9f0 msgr2=0x7f7a8015e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f7a84e00640 1 --2- 172.21.15.114:0/1773920052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800ce9f0 0x7f7a8015e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f7a84e00640 1 -- 172.21.15.114:0/1773920052 >> v2:172.21.15.177:3300/0 conn(0x7f7a800d2860 msgr2=0x7f7a8015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f7a84e00640 1 --2- 172.21.15.114:0/1773920052 >> v2:172.21.15.177:3300/0 conn(0x7f7a800d2860 0x7f7a8015dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.878+0000 7f7a84e00640 1 -- 172.21.15.114:0/1773920052 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a800cf260 con 0x7f7a800d2480 2024-03-06T07:28:03.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.880+0000 7f08ea800640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08cc06b5a0 0x7f08cc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.881+0000 7f7a84e00640 1 --2- 172.21.15.114:0/1773920052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800d4860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a8008c260 tx=0x7f7a7002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f577fbd35a8f505 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.882+0000 7f7a7fe00640 1 -- 172.21.15.114:0/1773920052 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7002cb20 con 0x7f7a800d2480 2024-03-06T07:28:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.880+0000 7f2539400640 1 -- 172.21.15.114:0/861450109 >> v2:172.21.15.177:3300/0 conn(0x7f253c0cdcb0 msgr2=0x7f253c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.880+0000 7f2539400640 1 --2- 172.21.15.114:0/861450109 >> v2:172.21.15.177:3300/0 conn(0x7f253c0cdcb0 0x7f253c06cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.880+0000 7f2539400640 1 -- 172.21.15.114:0/861450109 >> v1:172.21.15.114:6789/0 conn(0x7f253c154490 legacy=0x7f253c0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.880+0000 7f2539400640 1 -- 172.21.15.114:0/861450109 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c0ca7b0 con 0x7f253c154870 2024-03-06T07:28:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.881+0000 7f2539400640 1 --2- 172.21.15.114:0/861450109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154870 0x7f253c0cd770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f252c00ba70 tx=0x7f252c0301e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28566da463178bba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.882+0000 7f2538a00640 1 -- 172.21.15.114:0/861450109 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c03d070 con 0x7f253c154870 2024-03-06T07:28:03.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.882+0000 7f2538a00640 1 -- 172.21.15.114:0/861450109 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252c038440 con 0x7f253c154870 2024-03-06T07:28:03.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.883+0000 7f2538a00640 1 -- 172.21.15.114:0/861450109 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c038810 con 0x7f253c154870 2024-03-06T07:28:03.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.883+0000 7f7a7fe00640 1 -- 172.21.15.114:0/1773920052 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7002ccc0 con 0x7f7a800d2480 2024-03-06T07:28:03.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.883+0000 7f7a7fe00640 1 -- 172.21.15.114:0/1773920052 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70034610 con 0x7f7a800d2480 2024-03-06T07:28:03.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.883+0000 7f08ea800640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08cc06b5a0 0x7f08cc06da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f08e4007eb0 tx=0x7f08e40073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.883+0000 7f2540a00640 1 -- 172.21.15.114:0/861450109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154870 msgr2=0x7f253c0cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.883+0000 7f2540a00640 1 --2- 172.21.15.114:0/861450109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154870 0x7f253c0cd770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f252c00ba70 tx=0x7f252c0301e0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.884+0000 7f2540a00640 1 -- 172.21.15.114:0/861450109 shutdown_connections 2024-03-06T07:28:03.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.884+0000 7f2540a00640 1 --2- 172.21.15.114:0/861450109 >> v2:172.21.15.177:3300/0 conn(0x7f253c0cdcb0 0x7f253c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.884+0000 7f2540a00640 1 --2- 172.21.15.114:0/861450109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154870 0x7f253c0cd770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.884+0000 7f2540a00640 1 -- 172.21.15.114:0/861450109 >> 172.21.15.114:0/861450109 conn(0x7f253c0cc4f0 msgr2=0x7f253c0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.885+0000 7f7a87800640 1 -- 172.21.15.114:0/1773920052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 msgr2=0x7f7a800d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.885+0000 7f7a87800640 1 --2- 172.21.15.114:0/1773920052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800d4860 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a8008c260 tx=0x7f7a7002bf80 comp rx=0 tx=0).stop 2024-03-06T07:28:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.885+0000 7f7a87800640 1 -- 172.21.15.114:0/1773920052 shutdown_connections 2024-03-06T07:28:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.885+0000 7f7a87800640 1 --2- 172.21.15.114:0/1773920052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800ce9f0 0x7f7a8015e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.885+0000 7f7a87800640 1 --2- 172.21.15.114:0/1773920052 >> v2:172.21.15.177:3300/0 conn(0x7f7a800d2860 0x7f7a8015dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.885+0000 7f7a87800640 1 --2- 172.21.15.114:0/1773920052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800d4860 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.885+0000 7f7a87800640 1 -- 172.21.15.114:0/1773920052 >> 172.21.15.114:0/1773920052 conn(0x7f7a8009d240 msgr2=0x7f7a800d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.886+0000 7f2540a00640 1 -- 172.21.15.114:0/861450109 shutdown_connections 2024-03-06T07:28:03.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.886+0000 7f7a87800640 1 -- 172.21.15.114:0/1773920052 shutdown_connections 2024-03-06T07:28:03.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.886+0000 7f7a87800640 1 -- 172.21.15.114:0/1773920052 wait complete. 2024-03-06T07:28:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.886+0000 7f2540a00640 1 -- 172.21.15.114:0/861450109 wait complete. 2024-03-06T07:28:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.888+0000 7f2540a00640 1 Processor -- start 2024-03-06T07:28:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.889+0000 7f7a87800640 1 Processor -- start 2024-03-06T07:28:03.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.889+0000 7f7a87800640 1 -- start start 2024-03-06T07:28:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.889+0000 7f2540a00640 1 -- start start 2024-03-06T07:28:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.890+0000 7f2540a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f253c0cdcb0 0x7f253c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.890+0000 7f2540a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154490 0x7f253c0c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.890+0000 7f2540a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f253c154870 0x7f253c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.891+0000 7f7a87800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a800ce9f0 0x7f7a800b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.892+0000 7f7a87800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.892+0000 7f08e2a00640 1 -- 172.21.15.114:0/917006594 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08dc057ae0 con 0x7f08ec1547e0 2024-03-06T07:28:03.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.892+0000 7f7a84e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a800ce9f0 0x7f7a800b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.892+0000 7f7a77e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.892+0000 7f7a77e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800b24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56066/0 (socket says 172.21.15.114:56066) 2024-03-06T07:28:03.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.892+0000 7f7a84e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a800ce9f0 0x7f7a800b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35878/0 (socket says 172.21.15.114:35878) 2024-03-06T07:28:03.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.892+0000 7f7a87800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800d2860 0x7f7a8015d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.891+0000 7f2539e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f253c0cdcb0 0x7f253c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.893+0000 7f2539e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f253c0cdcb0 0x7f253c155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59224/0 (socket says 172.21.15.114:59224) 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.893+0000 7f2539400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154490 0x7f253c0c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.893+0000 7f2540a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0cb290 con 0x7f253c154490 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.893+0000 7f2540a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0cad70 con 0x7f253c154870 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.893+0000 7f2540a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0cc140 con 0x7f253c0cdcb0 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.894+0000 7f2539e00640 1 -- 172.21.15.114:0/1357767067 learned_addr learned my addr 172.21.15.114:0/1357767067 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.895+0000 7f2539e00640 1 -- 172.21.15.114:0/1357767067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f253c154870 msgr2=0x7f253c1598e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.895+0000 7f2539e00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f253c154870 0x7f253c1598e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.895+0000 7f2539e00640 1 -- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154490 msgr2=0x7f253c0c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.895+0000 7f2539e00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154490 0x7f253c0c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.893+0000 7f7a77e00640 1 -- 172.21.15.114:0/1945745096 learned_addr learned my addr 172.21.15.114:0/1945745096 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.896+0000 7f2539e00640 1 -- 172.21.15.114:0/1357767067 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c1f3410 con 0x7f253c0cdcb0 2024-03-06T07:28:03.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.896+0000 7f2539400640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154490 0x7f253c0c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.894+0000 7f7a87800640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800cfb00 con 0x7f7a800d2480 2024-03-06T07:28:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.894+0000 7f7a87800640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800cf820 con 0x7f7a800ce9f0 2024-03-06T07:28:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.894+0000 7f7a87800640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a80161e30 con 0x7f7a800d2860 2024-03-06T07:28:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.893+0000 7f7a85800640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800d2860 0x7f7a8015d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.896+0000 7f2539e00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f253c0cdcb0 0x7f253c155e40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f252800c450 tx=0x7f252800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.897+0000 7f2532000640 1 -- 172.21.15.114:0/1357767067 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25280052d0 con 0x7f253c0cdcb0 2024-03-06T07:28:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.897+0000 7f2532000640 1 -- 172.21.15.114:0/1357767067 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528005470 con 0x7f253c0cdcb0 2024-03-06T07:28:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.897+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c1f3750 con 0x7f253c0cdcb0 2024-03-06T07:28:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.898+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c1ef8b0 con 0x7f253c0cdcb0 2024-03-06T07:28:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.899+0000 7f2532000640 1 -- 172.21.15.114:0/1357767067 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528012870 con 0x7f253c0cdcb0 2024-03-06T07:28:03.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.900+0000 7f7a85800640 1 -- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a800ce9f0 msgr2=0x7f7a800b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.900+0000 7f7a85800640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a800ce9f0 0x7f7a800b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.900+0000 7f7a85800640 1 -- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 msgr2=0x7f7a800b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.901+0000 7f7a85800640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.901+0000 7f7a85800640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a800b3330 con 0x7f7a800d2860 2024-03-06T07:28:03.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.901+0000 7f7a77e00640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:03.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.905+0000 7f7a85800640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800d2860 0x7f7a8015d550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e7f0 tx=0x7f7a7800ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.903+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2514005290 con 0x7f253c0cdcb0 2024-03-06T07:28:03.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.903+0000 7f2532000640 1 -- 172.21.15.114:0/1357767067 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2528012a10 con 0x7f253c0cdcb0 2024-03-06T07:28:03.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.904+0000 7f2532000640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f251c06b4d0 0x7f251c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.905+0000 7f2532000640 1 -- 172.21.15.114:0/1357767067 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2528089d80 con 0x7f253c0cdcb0 2024-03-06T07:28:03.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.906+0000 7f2539400640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f251c06b4d0 0x7f251c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.907+0000 7f2539400640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f251c06b4d0 0x7f251c06d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f252c03a000 tx=0x7f252c034000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.917+0000 7f7a7d600640 1 -- 172.21.15.114:0/1945745096 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78017070 con 0x7f7a800d2860 2024-03-06T07:28:03.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.917+0000 7f7a87800640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a800b38b0 con 0x7f7a800d2860 2024-03-06T07:28:03.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.917+0000 7f7a7d600640 1 -- 172.21.15.114:0/1945745096 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7800ee40 con 0x7f7a800d2860 2024-03-06T07:28:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.917+0000 7f7a87800640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a800b3e70 con 0x7f7a800d2860 2024-03-06T07:28:03.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.917+0000 7f7a7d600640 1 -- 172.21.15.114:0/1945745096 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78017070 con 0x7f7a800d2860 2024-03-06T07:28:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.919+0000 7f2532000640 1 -- 172.21.15.114:0/1357767067 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f253c1f3750 con 0x7f253c0cdcb0 2024-03-06T07:28:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.920+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a50005290 con 0x7f7a800d2860 2024-03-06T07:28:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.922+0000 7f7a7d600640 1 -- 172.21.15.114:0/1945745096 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a7801a070 con 0x7f7a800d2860 2024-03-06T07:28:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.923+0000 7f7a7d600640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7a5c06b6d0 0x7f7a5c06db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.924+0000 7f7a7d600640 1 -- 172.21.15.114:0/1945745096 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a78089890 con 0x7f7a800d2860 2024-03-06T07:28:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.925+0000 7f7a84e00640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7a5c06b6d0 0x7f7a5c06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.926+0000 7f7a84e00640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7a5c06b6d0 0x7f7a5c06db90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a70035000 tx=0x7f7a70030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.938+0000 7f7a7d600640 1 -- 172.21.15.114:0/1945745096 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a780580b0 con 0x7f7a800d2860 2024-03-06T07:28:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.988+0000 7f51df400640 1 Processor -- start 2024-03-06T07:28:03.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.989+0000 7f51df400640 1 -- start start 2024-03-06T07:28:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.990+0000 7f51df400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f51d815e360 0x7f51d80cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.990+0000 7f51df400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80d12e0 0x7f51d8161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.991+0000 7f51df400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d80cf2b0 con 0x7f51d8162270 2024-03-06T07:28:03.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.991+0000 7f51df400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80cf590 con 0x7f51d815e360 2024-03-06T07:28:03.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.991+0000 7f51df400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80cf870 con 0x7f51d80d12e0 2024-03-06T07:28:03.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.993+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80d12e0 0x7f51d8161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.993+0000 7f51dd400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f51d8162270 0x7f51d815e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36082/0 (socket says 172.21.15.114:36082) 2024-03-06T07:28:03.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.993+0000 7f51dd400640 1 -- 172.21.15.114:0/1350336088 learned_addr learned my addr 172.21.15.114:0/1350336088 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:03.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.993+0000 7f51dca00640 1 --2- 172.21.15.114:0/1350336088 >> v2:172.21.15.114:3300/0 conn(0x7f51d815e360 0x7f51d80cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.994+0000 7f51dca00640 1 -- 172.21.15.114:0/1350336088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80d12e0 msgr2=0x7f51d8161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.994+0000 7f51dca00640 1 --2- 172.21.15.114:0/1350336088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80d12e0 0x7f51d8161d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.994+0000 7f51dca00640 1 -- 172.21.15.114:0/1350336088 >> v1:172.21.15.114:6789/0 conn(0x7f51d8162270 legacy=0x7f51d815e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.994+0000 7f51dca00640 1 -- 172.21.15.114:0/1350336088 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80cfb50 con 0x7f51d815e360 2024-03-06T07:28:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.996+0000 7f51dca00640 1 --2- 172.21.15.114:0/1350336088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d815e360 0x7f51d80cec10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0097a0 tx=0x7f51cc02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9204d682af2d2a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.997+0000 7f51d7400640 1 -- 172.21.15.114:0/1350336088 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc034070 con 0x7f51d815e360 2024-03-06T07:28:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.997+0000 7f51d7400640 1 -- 172.21.15.114:0/1350336088 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc030800 con 0x7f51d815e360 2024-03-06T07:28:03.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.997+0000 7f51d7400640 1 -- 172.21.15.114:0/1350336088 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc034070 con 0x7f51d815e360 2024-03-06T07:28:04.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.998+0000 7f51df400640 1 -- 172.21.15.114:0/1350336088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d815e360 msgr2=0x7f51d80cec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:03.998+0000 7f51df400640 1 --2- 172.21.15.114:0/1350336088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d815e360 0x7f51d80cec10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0097a0 tx=0x7f51cc02b940 comp rx=0 tx=0).stop 2024-03-06T07:28:04.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.002+0000 7f51df400640 1 -- 172.21.15.114:0/1350336088 shutdown_connections 2024-03-06T07:28:04.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.002+0000 7f51df400640 1 --2- 172.21.15.114:0/1350336088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80d12e0 0x7f51d8161d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.002+0000 7f51df400640 1 --2- 172.21.15.114:0/1350336088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d815e360 0x7f51d80cec10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.002+0000 7f51df400640 1 -- 172.21.15.114:0/1350336088 >> 172.21.15.114:0/1350336088 conn(0x7f51d80d3f50 msgr2=0x7f51d80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.002+0000 7f51df400640 1 -- 172.21.15.114:0/1350336088 shutdown_connections 2024-03-06T07:28:04.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.002+0000 7f51df400640 1 -- 172.21.15.114:0/1350336088 wait complete. 2024-03-06T07:28:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.006+0000 7f51df400640 1 Processor -- start 2024-03-06T07:28:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.006+0000 7f51df400640 1 -- start start 2024-03-06T07:28:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.011+0000 7f51df400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d80d12e0 0x7f51d8159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.012+0000 7f51df400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d8162270 0x7f51d80c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.012+0000 7f51df400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80b8ca0 0x7f51d80cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.013+0000 7f51df400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80d18c0 con 0x7f51d80d12e0 2024-03-06T07:28:04.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.013+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d8162270 0x7f51d80c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.013+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d8162270 0x7f51d80c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35894/0 (socket says 172.21.15.114:35894) 2024-03-06T07:28:04.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.013+0000 7f51d7e00640 1 -- 172.21.15.114:0/3388739459 learned_addr learned my addr 172.21.15.114:0/3388739459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.013+0000 7f51dd400640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80b8ca0 0x7f51d80cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.013+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80d1740 con 0x7f51d8162270 2024-03-06T07:28:04.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.013+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80d2ca0 con 0x7f51d80b8ca0 2024-03-06T07:28:04.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.014+0000 7f51d7e00640 1 -- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80b8ca0 msgr2=0x7f51d80cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.014+0000 7f51d7e00640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80b8ca0 0x7f51d80cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.014+0000 7f51d7e00640 1 -- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d80d12e0 msgr2=0x7f51d8159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.014+0000 7f51d7e00640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d80d12e0 0x7f51d8159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.016+0000 7ff50da00640 1 Processor -- start 2024-03-06T07:28:04.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.014+0000 7f51d7e00640 1 -- 172.21.15.114:0/3388739459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80b9240 con 0x7f51d8162270 2024-03-06T07:28:04.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.017+0000 7ff50da00640 1 -- start start 2024-03-06T07:28:04.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.016+0000 7f0850600640 1 Processor -- start 2024-03-06T07:28:04.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.017+0000 7f0850600640 1 -- start start 2024-03-06T07:28:04.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7f0850600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f084806ec70 0x7f0848154220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7f0850600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08480cde80 0x7f08480cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7f0850600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480ca1f0 con 0x7f08480cba30 2024-03-06T07:28:04.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7f0850600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08480ca4d0 con 0x7f084806ec70 2024-03-06T07:28:04.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7f0850600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08480ca7b0 con 0x7f08480cde80 2024-03-06T07:28:04.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7ff50da00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff50815e360 0x7ff5080cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f51d7e00640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d8162270 0x7f51d80c8300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51c400e9b0 tx=0x7f51c400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7ff50da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff508162440 0x7ff5080d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f084dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f084806ec70 0x7f0848154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7ff50da00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080cf2b0 con 0x7ff5080d12e0 2024-03-06T07:28:04.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7ff50da00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5080cf590 con 0x7ff50815e360 2024-03-06T07:28:04.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.019+0000 7ff50da00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080cf870 con 0x7ff508162440 2024-03-06T07:28:04.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f084dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f084806ec70 0x7f0848154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35902/0 (socket says 172.21.15.114:35902) 2024-03-06T07:28:04.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f084dc00640 1 -- 172.21.15.114:0/870021554 learned_addr learned my addr 172.21.15.114:0/870021554 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f084e600640 1 --2- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08480cde80 0x7f08480cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f51d4c00640 1 -- 172.21.15.114:0/3388739459 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c4017070 con 0x7f51d8162270 2024-03-06T07:28:04.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.021+0000 7ff507400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff508162440 0x7ff5080d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.021+0000 7f084dc00640 1 -- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08480cde80 msgr2=0x7f08480cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.021+0000 7f084dc00640 1 --2- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08480cde80 0x7f08480cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.021+0000 7f084dc00640 1 -- 172.21.15.114:0/870021554 >> v1:172.21.15.177:6789/0 conn(0x7f08480cba30 legacy=0x7f08480cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.021+0000 7f084dc00640 1 -- 172.21.15.114:0/870021554 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08480caa90 con 0x7f084806ec70 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f51d4c00640 1 -- 172.21.15.114:0/3388739459 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c400c9a0 con 0x7f51d8162270 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.020+0000 7f51d4c00640 1 -- 172.21.15.114:0/3388739459 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c400cd90 con 0x7f51d8162270 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.022+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d80b6d80 con 0x7f51d8162270 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.022+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d80cd5c0 con 0x7f51d8162270 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.022+0000 7ff506a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff50815e360 0x7ff5080cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.022+0000 7ff506a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff50815e360 0x7ff5080cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56096/0 (socket says 172.21.15.114:56096) 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.022+0000 7ff506a00640 1 -- 172.21.15.114:0/32389056 learned_addr learned my addr 172.21.15.114:0/32389056 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.022+0000 7ff506a00640 1 -- 172.21.15.114:0/32389056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff508162440 msgr2=0x7ff5080d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.022+0000 7ff506000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff5080d12e0 0x7ff508161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36096/0 (socket says 172.21.15.114:36096) 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.023+0000 7ff506a00640 1 --2- 172.21.15.114:0/32389056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff508162440 0x7ff5080d1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.023+0000 7ff506a00640 1 -- 172.21.15.114:0/32389056 >> v1:172.21.15.114:6789/0 conn(0x7ff5080d12e0 legacy=0x7ff508161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.023+0000 7f084dc00640 1 --2- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084806ec70 0x7f0848154220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f083c009870 tx=0x7f083c02ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f10b6e0286a3053 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.024+0000 7ff506a00640 1 -- 172.21.15.114:0/32389056 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080cfb50 con 0x7ff50815e360 2024-03-06T07:28:04.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.024+0000 7ff507400640 1 --2- 172.21.15.114:0/32389056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff508162440 0x7ff5080d1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:04.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.024+0000 7f0847e00640 1 -- 172.21.15.114:0/870021554 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c02f870 con 0x7f084806ec70 2024-03-06T07:28:04.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.026+0000 7ff506a00640 1 --2- 172.21.15.114:0/32389056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50815e360 0x7ff5080cec10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff500009e50 tx=0x7ff50002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0c2d1b4a9d804b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.026+0000 7ff505600640 1 -- 172.21.15.114:0/32389056 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50002cb90 con 0x7ff50815e360 2024-03-06T07:28:04.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.027+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b4005290 con 0x7f51d8162270 2024-03-06T07:28:04.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.027+0000 7ff505600640 1 -- 172.21.15.114:0/32389056 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50002cd30 con 0x7ff50815e360 2024-03-06T07:28:04.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.027+0000 7ff505600640 1 -- 172.21.15.114:0/32389056 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500034610 con 0x7ff50815e360 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.026+0000 7f0847e00640 1 -- 172.21.15.114:0/870021554 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c02fa10 con 0x7f084806ec70 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.027+0000 7f0847e00640 1 -- 172.21.15.114:0/870021554 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c02fe20 con 0x7f084806ec70 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.027+0000 7f51d4c00640 1 -- 172.21.15.114:0/3388739459 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51c401a070 con 0x7f51d8162270 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7ff50da00640 1 -- 172.21.15.114:0/32389056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50815e360 msgr2=0x7ff5080cec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7ff50da00640 1 --2- 172.21.15.114:0/32389056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50815e360 0x7ff5080cec10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff500009e50 tx=0x7ff50002bff0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7f0850600640 1 -- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084806ec70 msgr2=0x7f0848154220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7f0850600640 1 --2- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084806ec70 0x7f0848154220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f083c009870 tx=0x7f083c02ba50 comp rx=0 tx=0).stop 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7f0850600640 1 -- 172.21.15.114:0/870021554 shutdown_connections 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7f0850600640 1 --2- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08480cde80 0x7f08480cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7f0850600640 1 --2- 172.21.15.114:0/870021554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084806ec70 0x7f0848154220 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7f0850600640 1 -- 172.21.15.114:0/870021554 >> 172.21.15.114:0/870021554 conn(0x7f08480cc4f0 msgr2=0x7f08480cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7ff50da00640 1 -- 172.21.15.114:0/32389056 shutdown_connections 2024-03-06T07:28:04.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7ff50da00640 1 --2- 172.21.15.114:0/32389056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff508162440 0x7ff5080d1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7ff50da00640 1 --2- 172.21.15.114:0/32389056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50815e360 0x7ff5080cec10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.028+0000 7ff50da00640 1 -- 172.21.15.114:0/32389056 >> 172.21.15.114:0/32389056 conn(0x7ff5080d3f50 msgr2=0x7ff5080d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.029+0000 7ff50da00640 1 -- 172.21.15.114:0/32389056 shutdown_connections 2024-03-06T07:28:04.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.029+0000 7ff50da00640 1 -- 172.21.15.114:0/32389056 wait complete. 2024-03-06T07:28:04.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.029+0000 7f0850600640 1 -- 172.21.15.114:0/870021554 shutdown_connections 2024-03-06T07:28:04.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.029+0000 7f0850600640 1 -- 172.21.15.114:0/870021554 wait complete. 2024-03-06T07:28:04.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.030+0000 7f51d4c00640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51b806b500 0x7f51b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.031+0000 7f51d4c00640 1 -- 172.21.15.114:0/3388739459 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51c408a4d0 con 0x7f51d8162270 2024-03-06T07:28:04.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.031+0000 7f51dca00640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51b806b500 0x7f51b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.032+0000 7f0850600640 1 Processor -- start 2024-03-06T07:28:04.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.032+0000 7ff50da00640 1 Processor -- start 2024-03-06T07:28:04.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.032+0000 7ff50da00640 1 -- start start 2024-03-06T07:28:04.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.032+0000 7f51dca00640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51b806b500 0x7f51b806d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f51cc036000 tx=0x7f51cc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.032+0000 7f0850600640 1 -- start start 2024-03-06T07:28:04.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.034+0000 7f0850600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f084806ec70 0x7f08481376c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.034+0000 7ff50da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5080d12e0 0x7ff508161370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.034+0000 7f0850600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08480cba30 0x7f08480c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7ff50da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff50815e360 0x7ff5080b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7ff50da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff508162440 0x7ff5080b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7f0850600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08480cde80 0x7f0848159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7f084dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f084806ec70 0x7f08481376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7f084dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f084806ec70 0x7f08481376c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59282/0 (socket says 172.21.15.114:59282) 2024-03-06T07:28:04.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7f084dc00640 1 -- 172.21.15.114:0/200062946 learned_addr learned my addr 172.21.15.114:0/200062946 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7f084d200640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08480cba30 0x7f08480c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7f084e600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08480cde80 0x7f0848159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff507400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff508162440 0x7ff5080b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff507400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff508162440 0x7ff5080b8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56120/0 (socket says 172.21.15.114:56120) 2024-03-06T07:28:04.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff506000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff50815e360 0x7ff5080b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.035+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848154760 con 0x7f08480cde80 2024-03-06T07:28:04.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08480cb290 con 0x7f08480cba30 2024-03-06T07:28:04.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08481548e0 con 0x7f084806ec70 2024-03-06T07:28:04.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff506000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff50815e360 0x7ff5080b8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59294/0 (socket says 172.21.15.114:59294) 2024-03-06T07:28:04.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff506a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5080d12e0 0x7ff508161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff507400640 1 -- 172.21.15.114:0/2064878464 learned_addr learned my addr 172.21.15.114:0/2064878464 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff50da00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080d18f0 con 0x7ff508162440 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff50da00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080d1770 con 0x7ff5080d12e0 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.036+0000 7ff50da00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080d21d0 con 0x7ff50815e360 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.038+0000 7ff506a00640 1 -- 172.21.15.114:0/2064878464 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff50815e360 msgr2=0x7ff5080b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.038+0000 7ff506a00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff50815e360 0x7ff5080b8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.038+0000 7ff506a00640 1 -- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff508162440 msgr2=0x7ff5080b8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.037+0000 7f084e600640 1 -- 172.21.15.114:0/200062946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f084806ec70 msgr2=0x7f08481376c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.037+0000 7f084e600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f084806ec70 0x7f08481376c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.037+0000 7f084e600640 1 -- 172.21.15.114:0/200062946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08480cba30 msgr2=0x7f08480c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.037+0000 7f084e600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08480cba30 0x7f08480c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.037+0000 7f084e600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08480c9080 con 0x7f08480cde80 2024-03-06T07:28:04.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.038+0000 7ff506a00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff508162440 0x7ff5080b8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.038+0000 7ff506a00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080b6c10 con 0x7ff5080d12e0 2024-03-06T07:28:04.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.038+0000 7f084e600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08480cde80 0x7f0848159d10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f084000ebb0 tx=0x7f084000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.038+0000 7ff507400640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff508162440 0x7ff5080b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:04.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.039+0000 7f0845600640 1 -- 172.21.15.114:0/200062946 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840018020 con 0x7f08480cde80 2024-03-06T07:28:04.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.039+0000 7f0845600640 1 -- 172.21.15.114:0/200062946 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084000cd40 con 0x7f08480cde80 2024-03-06T07:28:04.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.040+0000 7ff506a00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5080d12e0 0x7ff508161370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff50808ce70 tx=0x7ff500004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.040+0000 7f0845600640 1 -- 172.21.15.114:0/200062946 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08400146c0 con 0x7f08480cde80 2024-03-06T07:28:04.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.040+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08480c9360 con 0x7f08480cde80 2024-03-06T07:28:04.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.040+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0848159510 con 0x7f08480cde80 2024-03-06T07:28:04.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.044+0000 7f51d4c00640 1 -- 172.21.15.114:0/3388739459 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51c4058cf0 con 0x7f51d8162270 2024-03-06T07:28:04.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.042+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084804f330 con 0x7f08480cde80 2024-03-06T07:28:04.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.045+0000 7f0845600640 1 -- 172.21.15.114:0/200062946 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0840005d10 con 0x7f08480cde80 2024-03-06T07:28:04.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.047+0000 7f0845600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f082806b5a0 0x7f082806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.047+0000 7f0845600640 1 -- 172.21.15.114:0/200062946 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f084008b580 con 0x7f08480cde80 2024-03-06T07:28:04.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.047+0000 7ff4fea00640 1 -- 172.21.15.114:0/2064878464 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500043020 con 0x7ff5080d12e0 2024-03-06T07:28:04.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.048+0000 7f084dc00640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f082806b5a0 0x7f082806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.049+0000 7f084dc00640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f082806b5a0 0x7f082806da60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f083c02ba90 tx=0x7f083c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.049+0000 7ff4fea00640 1 -- 172.21.15.114:0/2064878464 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50003e450 con 0x7ff5080d12e0 2024-03-06T07:28:04.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.049+0000 7ff4fea00640 1 -- 172.21.15.114:0/2064878464 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50003e840 con 0x7ff5080d12e0 2024-03-06T07:28:04.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.049+0000 7f9d41600640 1 Processor -- start 2024-03-06T07:28:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.050+0000 7ff50da00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50815b440 con 0x7ff5080d12e0 2024-03-06T07:28:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.050+0000 7f9d41600640 1 -- start start 2024-03-06T07:28:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.052+0000 7f9d41600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9d3c155060 0x7f9d3c140e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.052+0000 7f9d41600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c15b290 0x7f9d3c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.051+0000 7ff50da00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5080b7470 con 0x7ff5080d12e0 2024-03-06T07:28:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.052+0000 7f9d41600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3c15b7c0 con 0x7f9d3c06f070 2024-03-06T07:28:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.052+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc005290 con 0x7ff5080d12e0 2024-03-06T07:28:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.052+0000 7f9d41600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3c141360 con 0x7f9d3c155060 2024-03-06T07:28:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.052+0000 7f9d41600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c15e880 con 0x7f9d3c15b290 2024-03-06T07:28:04.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.053+0000 7f9d3aa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9d3c155060 0x7f9d3c140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.053+0000 7f9d3aa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9d3c155060 0x7f9d3c140e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:35928/0 (socket says 172.21.15.114:35928) 2024-03-06T07:28:04.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.053+0000 7f9d3aa00640 1 -- 172.21.15.114:0/4064316833 learned_addr learned my addr 172.21.15.114:0/4064316833 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.053+0000 7f9d3a000640 1 --2- 172.21.15.114:0/4064316833 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c15b290 0x7f9d3c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.055+0000 7f9d3a000640 1 -- 172.21.15.114:0/4064316833 >> v2:172.21.15.177:3300/0 conn(0x7f9d3c155060 msgr2=0x7f9d3c140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.055+0000 7f9d3a000640 1 --2- 172.21.15.114:0/4064316833 >> v2:172.21.15.177:3300/0 conn(0x7f9d3c155060 0x7f9d3c140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.055+0000 7f9d3a000640 1 -- 172.21.15.114:0/4064316833 >> v1:172.21.15.177:6789/0 conn(0x7f9d3c06f070 legacy=0x7f9d3c071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.055+0000 7f9d3a000640 1 -- 172.21.15.114:0/4064316833 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c15baa0 con 0x7f9d3c15b290 2024-03-06T07:28:04.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.056+0000 7f0845600640 1 -- 172.21.15.114:0/200062946 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0840022020 con 0x7f08480cde80 2024-03-06T07:28:04.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.057+0000 7ff4fea00640 1 -- 172.21.15.114:0/2064878464 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff500046020 con 0x7ff5080d12e0 2024-03-06T07:28:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.058+0000 7ff4fea00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4e806b5a0 0x7ff4e806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.059+0000 7ff4fea00640 1 -- 172.21.15.114:0/2064878464 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5000ad1c0 con 0x7ff5080d12e0 2024-03-06T07:28:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.059+0000 7f9d3a000640 1 --2- 172.21.15.114:0/4064316833 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c15b290 0x7f9d3c06eb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c007b70 tx=0x7f9d2c02fcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e10aa5097f8b4a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.059+0000 7f9d39600640 1 -- 172.21.15.114:0/4064316833 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c030670 con 0x7f9d3c15b290 2024-03-06T07:28:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.059+0000 7f9d39600640 1 -- 172.21.15.114:0/4064316833 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c030810 con 0x7f9d3c15b290 2024-03-06T07:28:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.059+0000 7f9d39600640 1 -- 172.21.15.114:0/4064316833 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c030670 con 0x7f9d3c15b290 2024-03-06T07:28:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.060+0000 7f9d41600640 1 -- 172.21.15.114:0/4064316833 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c15b290 msgr2=0x7f9d3c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.060+0000 7f9d41600640 1 --2- 172.21.15.114:0/4064316833 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c15b290 0x7f9d3c06eb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c007b70 tx=0x7f9d2c02fcd0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.061+0000 7ff506000640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4e806b5a0 0x7ff4e806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.061+0000 7f9d41600640 1 -- 172.21.15.114:0/4064316833 shutdown_connections 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.062+0000 7f9d41600640 1 --2- 172.21.15.114:0/4064316833 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c15b290 0x7f9d3c06eb30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.062+0000 7f9d41600640 1 --2- 172.21.15.114:0/4064316833 >> v2:172.21.15.177:3300/0 conn(0x7f9d3c155060 0x7f9d3c140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.062+0000 7f9d41600640 1 -- 172.21.15.114:0/4064316833 >> 172.21.15.114:0/4064316833 conn(0x7f9d3c0d0b90 msgr2=0x7f9d3c0d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.062+0000 7f9d41600640 1 -- 172.21.15.114:0/4064316833 shutdown_connections 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.062+0000 7ff506000640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4e806b5a0 0x7ff4e806da60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff5080d0430 tx=0x7ff4f4007e00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.062+0000 7f9d41600640 1 -- 172.21.15.114:0/4064316833 wait complete. 2024-03-06T07:28:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.066+0000 7f9d41600640 1 Processor -- start 2024-03-06T07:28:04.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.066+0000 7f9d41600640 1 -- start start 2024-03-06T07:28:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.068+0000 7ff4fea00640 1 -- 172.21.15.114:0/2064878464 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff50004d030 con 0x7ff5080d12e0 2024-03-06T07:28:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.068+0000 7f9d41600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d3c06f070 0x7f9d3c13d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.068+0000 7f9d41600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c155060 0x7f9d3c0b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.068+0000 7f9d41600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d3c15b290 0x7f9d3c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.068+0000 7f9d41600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c15c060 con 0x7f9d3c15b290 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.068+0000 7f9d41600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0cc620 con 0x7f9d3c06f070 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.068+0000 7f9d41600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c072330 con 0x7f9d3c155060 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.069+0000 7f9d3a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c155060 0x7f9d3c0b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.069+0000 7f9d3a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c155060 0x7f9d3c0b64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59318/0 (socket says 172.21.15.114:59318) 2024-03-06T07:28:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.069+0000 7f9d3a000640 1 -- 172.21.15.114:0/1792818323 learned_addr learned my addr 172.21.15.114:0/1792818323 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.069+0000 7f9d3b400640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d3c15b290 0x7f9d3c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.070+0000 7f9d3a000640 1 -- 172.21.15.114:0/1792818323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d3c06f070 msgr2=0x7f9d3c13d4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:28:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.070+0000 7f9d3a000640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d3c06f070 0x7f9d3c13d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.070+0000 7f9d3a000640 1 -- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d3c15b290 msgr2=0x7f9d3c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.070+0000 7f9d3a000640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d3c15b290 0x7f9d3c0b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.070+0000 7f9d3a000640 1 -- 172.21.15.114:0/1792818323 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c0b3f20 con 0x7f9d3c155060 2024-03-06T07:28:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.070+0000 7f9d3b400640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d3c15b290 0x7f9d3c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:04.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.073+0000 7f9d3a000640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c155060 0x7f9d3c0b64b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c000c00 tx=0x7f9d2c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.074+0000 7f9d32a00640 1 -- 172.21.15.114:0/1792818323 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c048030 con 0x7f9d3c155060 2024-03-06T07:28:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.074+0000 7f9d32a00640 1 -- 172.21.15.114:0/1792818323 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c031e10 con 0x7f9d3c155060 2024-03-06T07:28:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.075+0000 7f9d32a00640 1 -- 172.21.15.114:0/1792818323 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c00d660 con 0x7f9d3c155060 2024-03-06T07:28:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.080+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c143820 con 0x7f9d3c155060 2024-03-06T07:28:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.080+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c143de0 con 0x7f9d3c155060 2024-03-06T07:28:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.084+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c04f330 con 0x7f9d3c155060 2024-03-06T07:28:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.084+0000 7f9d32a00640 1 -- 172.21.15.114:0/1792818323 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d2c0072e0 con 0x7f9d3c155060 2024-03-06T07:28:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.086+0000 7f9d32a00640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.086+0000 7f9d32a00640 1 -- 172.21.15.114:0/1792818323 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d2c0b0710 con 0x7f9d3c155060 2024-03-06T07:28:04.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.087+0000 7f9d3aa00640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.088+0000 7f9d3aa00640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400e190 tx=0x7f9d3400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.102+0000 7f9d32a00640 1 -- 172.21.15.114:0/1792818323 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d2c052020 con 0x7f9d3c155060 2024-03-06T07:28:04.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.119+0000 7fbfa9000640 1 Processor -- start 2024-03-06T07:28:04.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.119+0000 7fbfa9000640 1 -- start start 2024-03-06T07:28:04.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.121+0000 7fbfa9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfa40cfd30 0x7fbfa4168d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.121+0000 7fbfa9000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa40cd320 con 0x7fbfa40cacb0 2024-03-06T07:28:04.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.121+0000 7fbfa9000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa40cca80 con 0x7fbfa406afc0 2024-03-06T07:28:04.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.121+0000 7fbfa9000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa40ccd60 con 0x7fbfa40cfd30 2024-03-06T07:28:04.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.124+0000 7fbfa1e00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fbfa406afc0 0x7fbfa406b3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:48000/0 (socket says 172.21.15.114:48000) 2024-03-06T07:28:04.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.124+0000 7fbfa1e00640 1 -- 172.21.15.114:0/1308569130 learned_addr learned my addr 172.21.15.114:0/1308569130 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:04.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.124+0000 7fbfa2800640 1 --2- 172.21.15.114:0/1308569130 >> v2:172.21.15.114:3300/0 conn(0x7fbfa40cfd30 0x7fbfa4168d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.124+0000 7fbfa0a00640 1 -- 172.21.15.114:0/1308569130 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470938435 0 0) 0x7fbfa40cca80 con 0x7fbfa406afc0 2024-03-06T07:28:04.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.125+0000 7fbfa0a00640 1 -- 172.21.15.114:0/1308569130 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf88003520 con 0x7fbfa406afc0 2024-03-06T07:28:04.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.125+0000 7fbfa0a00640 1 -- 172.21.15.114:0/1308569130 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510811807 0 0) 0x7fbfa40cd320 con 0x7fbfa40cacb0 2024-03-06T07:28:04.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.126+0000 7fbfa0a00640 1 -- 172.21.15.114:0/1308569130 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa40cca80 con 0x7fbfa40cacb0 2024-03-06T07:28:04.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.126+0000 7fbfa2800640 1 -- 172.21.15.114:0/1308569130 >> v1:172.21.15.177:6789/0 conn(0x7fbfa406afc0 legacy=0x7fbfa406b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.127+0000 7fbfa2800640 1 -- 172.21.15.114:0/1308569130 >> v1:172.21.15.114:6789/0 conn(0x7fbfa40cacb0 legacy=0x7fbfa40cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.127+0000 7fbfa2800640 1 -- 172.21.15.114:0/1308569130 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa40cd040 con 0x7fbfa40cfd30 2024-03-06T07:28:04.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.130+0000 7fbfa2800640 1 --2- 172.21.15.114:0/1308569130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa40cfd30 0x7fbfa4168d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009d90 tx=0x7fbf9c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370de3358548c9bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.130+0000 7fbfa0a00640 1 -- 172.21.15.114:0/1308569130 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c039070 con 0x7fbfa40cfd30 2024-03-06T07:28:04.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.130+0000 7fbfa0a00640 1 -- 172.21.15.114:0/1308569130 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c02cc90 con 0x7fbfa40cfd30 2024-03-06T07:28:04.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.130+0000 7fbfa0a00640 1 -- 172.21.15.114:0/1308569130 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c034640 con 0x7fbfa40cfd30 2024-03-06T07:28:04.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.133+0000 7fbfa9000640 1 -- 172.21.15.114:0/1308569130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa40cfd30 msgr2=0x7fbfa4168d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.133+0000 7fbfa9000640 1 --2- 172.21.15.114:0/1308569130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa40cfd30 0x7fbfa4168d90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009d90 tx=0x7fbf9c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.133+0000 7fbfa9000640 1 -- 172.21.15.114:0/1308569130 shutdown_connections 2024-03-06T07:28:04.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.133+0000 7fbfa9000640 1 --2- 172.21.15.114:0/1308569130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa40cfd30 0x7fbfa4168d90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.133+0000 7fbfa9000640 1 -- 172.21.15.114:0/1308569130 >> 172.21.15.114:0/1308569130 conn(0x7fbfa40d0b90 msgr2=0x7fbfa40d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.134+0000 7fbfa9000640 1 -- 172.21.15.114:0/1308569130 shutdown_connections 2024-03-06T07:28:04.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.134+0000 7fbfa9000640 1 -- 172.21.15.114:0/1308569130 wait complete. 2024-03-06T07:28:04.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.144+0000 7fbfa9000640 1 Processor -- start 2024-03-06T07:28:04.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.144+0000 7fbfa9000640 1 -- start start 2024-03-06T07:28:04.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.146+0000 7fbfa9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa406afc0 0x7fbfa415a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.146+0000 7fbfa9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa40cacb0 0x7fbfa413ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.146+0000 7fbfa9000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa40cfd30 0x7fbfa4157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.146+0000 7fbfa9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4144b00 con 0x7fbfa406afc0 2024-03-06T07:28:04.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.146+0000 7fbfa9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4144980 con 0x7fbfa40cacb0 2024-03-06T07:28:04.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.146+0000 7fbfa9000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa41692d0 con 0x7fbfa40cfd30 2024-03-06T07:28:04.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.150+0000 7fbfa1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa406afc0 0x7fbfa415a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.150+0000 7fbfa1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa406afc0 0x7fbfa415a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56160/0 (socket says 172.21.15.114:56160) 2024-03-06T07:28:04.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.150+0000 7fbfa1e00640 1 -- 172.21.15.114:0/3814931215 learned_addr learned my addr 172.21.15.114:0/3814931215 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.151+0000 7fbfa1400640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa40cacb0 0x7fbfa413ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.151+0000 7fbfa2800640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa40cfd30 0x7fbfa4157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.151+0000 7fbfa2800640 1 -- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa40cacb0 msgr2=0x7fbfa413ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.151+0000 7fbfa2800640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa40cacb0 0x7fbfa413ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.151+0000 7fbfa2800640 1 -- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa406afc0 msgr2=0x7fbfa415a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.153+0000 7fbfa2800640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa406afc0 0x7fbfa415a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.153+0000 7fbfa2800640 1 -- 172.21.15.114:0/3814931215 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa415a590 con 0x7fbfa40cfd30 2024-03-06T07:28:04.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.153+0000 7fbfa1e00640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa406afc0 0x7fbfa415a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:04.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.154+0000 7fbfa2800640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa40cfd30 0x7fbfa4157060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c033000 tx=0x7fbf9c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.155+0000 7fbf9a000640 1 -- 172.21.15.114:0/3814931215 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c039050 con 0x7fbfa40cfd30 2024-03-06T07:28:04.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.158+0000 7fbf9a000640 1 -- 172.21.15.114:0/3814931215 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c03a040 con 0x7fbfa40cfd30 2024-03-06T07:28:04.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.158+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa415c0f0 con 0x7fbfa40cfd30 2024-03-06T07:28:04.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.158+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa4155620 con 0x7fbfa40cfd30 2024-03-06T07:28:04.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.158+0000 7fbf9a000640 1 -- 172.21.15.114:0/3814931215 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c03e610 con 0x7fbfa40cfd30 2024-03-06T07:28:04.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.162+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7c005290 con 0x7fbfa40cfd30 2024-03-06T07:28:04.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.166+0000 7fbf9a000640 1 -- 172.21.15.114:0/3814931215 <== mon.2 v2:172.21.15.177:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf9c047020 con 0x7fbfa40cfd30 2024-03-06T07:28:04.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.167+0000 7fbf9a000640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf8406b5d0 0x7fbf8406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.167+0000 7fbf9a000640 1 -- 172.21.15.114:0/3814931215 <== mon.2 v2:172.21.15.177:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf9c0ac0d0 con 0x7fbfa40cfd30 2024-03-06T07:28:04.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.168+0000 7fbfa1e00640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf8406b5d0 0x7fbf8406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.171+0000 7fbfa1e00640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf8406b5d0 0x7fbf8406da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbf94000f60 tx=0x7fbf94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.182+0000 7fbf9a000640 1 -- 172.21.15.114:0/3814931215 <== mon.2 v2:172.21.15.177:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf9c04e020 con 0x7fbfa40cfd30 2024-03-06T07:28:04.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.304+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f08c4005580 con 0x7f08ec1547e0 2024-03-06T07:28:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.308+0000 7f08e2a00640 1 -- 172.21.15.114:0/917006594 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f08dc05bb80 con 0x7f08ec1547e0 2024-03-06T07:28:04.313 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08cc06b5a0 msgr2=0x7f08cc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08cc06b5a0 0x7f08cc06da60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f08e4007eb0 tx=0x7f08e40073c0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08ec1547e0 msgr2=0x7f08ec148a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08ec1547e0 0x7f08ec148a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c1b0 tx=0x7f08dc00c670 comp rx=0 tx=0).stop 2024-03-06T07:28:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 shutdown_connections 2024-03-06T07:28:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08ec1547e0 0x7f08ec148a80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08cc06b5a0 0x7f08cc06da60 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f08e4007eb0 tx=0x7f08e40073c0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08ec154400 0x7f08ec148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 --2- 172.21.15.114:0/917006594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec06ec70 0x7f08ec15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.312+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 >> 172.21.15.114:0/917006594 conn(0x7f08ec0cc4f0 msgr2=0x7f08ec15cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.316+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 shutdown_connections 2024-03-06T07:28:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.316+0000 7f08f1400640 1 -- 172.21.15.114:0/917006594 wait complete. 2024-03-06T07:28:04.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-03-06T07:28:04.365 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.368+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7a50005580 con 0x7f7a800d2860 2024-03-06T07:28:04.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.369+0000 7f7a7d600640 1 -- 172.21.15.114:0/1945745096 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7a7805c150 con 0x7f7a800d2860 2024-03-06T07:28:04.372 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:04.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.373+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7a5c06b6d0 msgr2=0x7f7a5c06db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.373+0000 7f7a76000640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7a5c06b6d0 0x7f7a5c06db90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a70035000 tx=0x7f7a70030000 comp rx=0 tx=0).stop 2024-03-06T07:28:04.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.373+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800d2860 msgr2=0x7f7a8015d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.373+0000 7f7a76000640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800d2860 0x7f7a8015d550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e7f0 tx=0x7f7a7800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 shutdown_connections 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a800d2860 0x7f7a8015d550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7a5c06b6d0 0x7f7a5c06db90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a800d2480 0x7f7a800b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 --2- 172.21.15.114:0/1945745096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a800ce9f0 0x7f7a800b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 >> 172.21.15.114:0/1945745096 conn(0x7f7a8009d240 msgr2=0x7f7a800b6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 shutdown_connections 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f7a76000640 1 -- 172.21.15.114:0/1945745096 wait complete. 2024-03-06T07:28:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.374+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2514005580 con 0x7f253c0cdcb0 2024-03-06T07:28:04.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.377+0000 7f2532000640 1 -- 172.21.15.114:0/1357767067 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2528058bf0 con 0x7f253c0cdcb0 2024-03-06T07:28:04.380 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.381+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f251c06b4d0 msgr2=0x7f251c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.381+0000 7f2540a00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f251c06b4d0 0x7f251c06d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f252c03a000 tx=0x7f252c034000 comp rx=0 tx=0).stop 2024-03-06T07:28:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.381+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f253c0cdcb0 msgr2=0x7f253c155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.381+0000 7f2540a00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f253c0cdcb0 0x7f253c155e40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f252800c450 tx=0x7f252800c910 comp rx=0 tx=0).stop 2024-03-06T07:28:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.382+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 shutdown_connections 2024-03-06T07:28:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.382+0000 7f2540a00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f253c154870 0x7f253c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.382+0000 7f2540a00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f251c06b4d0 0x7f251c06d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.382+0000 7f2540a00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f253c154490 0x7f253c0c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.382+0000 7f2540a00640 1 --2- 172.21.15.114:0/1357767067 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f253c0cdcb0 0x7f253c155e40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.382+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 >> 172.21.15.114:0/1357767067 conn(0x7f253c0cc4f0 msgr2=0x7f253c15cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.383+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 shutdown_connections 2024-03-06T07:28:04.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.383+0000 7f2540a00640 1 -- 172.21.15.114:0/1357767067 wait complete. 2024-03-06T07:28:04.400 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-06T07:28:04.401 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.406 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-06T07:28:04.406 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.434+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f51b4005580 con 0x7f51d8162270 2024-03-06T07:28:04.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.435+0000 7f51d4c00640 1 -- 172.21.15.114:0/3388739459 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51c405cd90 con 0x7f51d8162270 2024-03-06T07:28:04.438 INFO:teuthology.orchestra.run.smithi114.stdout:55834574853 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51b806b500 msgr2=0x7f51b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51b806b500 0x7f51b806d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f51cc036000 tx=0x7f51cc030000 comp rx=0 tx=0).stop 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d8162270 msgr2=0x7f51d80c8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d8162270 0x7f51d80c8300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51c400e9b0 tx=0x7f51c400ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 shutdown_connections 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d80b8ca0 0x7f51d80cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d8162270 0x7f51d80c8300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51b806b500 0x7f51b806d9c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 --2- 172.21.15.114:0/3388739459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51d80d12e0 0x7f51d8159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 >> 172.21.15.114:0/3388739459 conn(0x7f51d80d3f50 msgr2=0x7f51d806bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.438+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 shutdown_connections 2024-03-06T07:28:04.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.439+0000 7f51df400640 1 -- 172.21.15.114:0/3388739459 wait complete. 2024-03-06T07:28:04.458 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2024-03-06T07:28:04.458 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.500+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff4dc005850 con 0x7ff5080d12e0 2024-03-06T07:28:04.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.501+0000 7ff4fea00640 1 -- 172.21.15.114:0/2064878464 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff50007b9d0 con 0x7ff5080d12e0 2024-03-06T07:28:04.504 INFO:teuthology.orchestra.run.smithi114.stdout:55834574854 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4e806b5a0 msgr2=0x7ff4e806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4e806b5a0 0x7ff4e806da60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff5080d0430 tx=0x7ff4f4007e00 comp rx=0 tx=0).stop 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5080d12e0 msgr2=0x7ff508161370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5080d12e0 0x7ff508161370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff50808ce70 tx=0x7ff500004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 shutdown_connections 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4e806b5a0 0x7ff4e806da60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff508162440 0x7ff5080b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff50815e360 0x7ff5080b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 --2- 172.21.15.114:0/2064878464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5080d12e0 0x7ff508161370 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 >> 172.21.15.114:0/2064878464 conn(0x7ff5080d3f50 msgr2=0x7ff50815d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 shutdown_connections 2024-03-06T07:28:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.504+0000 7ff4f3e00640 1 -- 172.21.15.114:0/2064878464 wait complete. 2024-03-06T07:28:04.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.510+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f08480cbec0 con 0x7f08480cde80 2024-03-06T07:28:04.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.511+0000 7f0845600640 1 -- 172.21.15.114:0/200062946 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0840010040 con 0x7f08480cde80 2024-03-06T07:28:04.514 INFO:teuthology.orchestra.run.smithi114.stdout:55834574853 2024-03-06T07:28:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.514+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f082806b5a0 msgr2=0x7f082806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.514+0000 7f0850600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f082806b5a0 0x7f082806da60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f083c02ba90 tx=0x7f083c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.514+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08480cde80 msgr2=0x7f0848159d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.514+0000 7f0850600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08480cde80 0x7f0848159d10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f084000ebb0 tx=0x7f084000c490 comp rx=0 tx=0).stop 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 shutdown_connections 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f082806b5a0 0x7f082806da60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08480cde80 0x7f0848159d10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08480cba30 0x7f08480c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 --2- 172.21.15.114:0/200062946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f084806ec70 0x7f08481376c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 >> 172.21.15.114:0/200062946 conn(0x7f08480cc4f0 msgr2=0x7f084815cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 shutdown_connections 2024-03-06T07:28:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.515+0000 7f0850600640 1 -- 172.21.15.114:0/200062946 wait complete. 2024-03-06T07:28:04.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-03-06T07:28:04.525 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.535+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9d3c070120 con 0x7f9d3c155060 2024-03-06T07:28:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.536+0000 7f9d32a00640 1 -- 172.21.15.114:0/1792818323 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9d2c07f500 con 0x7f9d3c155060 2024-03-06T07:28:04.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-03-06T07:28:04.540 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.540 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d1c06b4d0 msgr2=0x7f9d1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400e190 tx=0x7f9d3400b040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c155060 msgr2=0x7f9d3c0b64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c155060 0x7f9d3c0b64b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c000c00 tx=0x7f9d2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 shutdown_connections 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d3c15b290 0x7f9d3c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d3c155060 0x7f9d3c0b64b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.539+0000 7f9d41600640 1 --2- 172.21.15.114:0/1792818323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d3c06f070 0x7f9d3c13d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.540+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 >> 172.21.15.114:0/1792818323 conn(0x7f9d3c0d0b90 msgr2=0x7f9d3c0ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.540+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 shutdown_connections 2024-03-06T07:28:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.540+0000 7f9d41600640 1 -- 172.21.15.114:0/1792818323 wait complete. 2024-03-06T07:28:04.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-03-06T07:28:04.559 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.595+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbf7c005580 con 0x7fbfa40cfd30 2024-03-06T07:28:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.596+0000 7fbf9a000640 1 -- 172.21.15.114:0/3814931215 <== mon.2 v2:172.21.15.177:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbf9c07a970 con 0x7fbfa40cfd30 2024-03-06T07:28:04.599 INFO:teuthology.orchestra.run.smithi114.stdout:55834574855 2024-03-06T07:28:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf8406b5d0 msgr2=0x7fbf8406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf8406b5d0 0x7fbf8406da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbf94000f60 tx=0x7fbf94009040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa40cfd30 msgr2=0x7fbfa4157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa40cfd30 0x7fbfa4157060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c033000 tx=0x7fbf9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 shutdown_connections 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfa40cfd30 0x7fbfa4157060 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfa40cacb0 0x7fbfa413ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf8406b5d0 0x7fbf8406da90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.598+0000 7fbfa9000640 1 --2- 172.21.15.114:0/3814931215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfa406afc0 0x7fbfa415a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.599+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 >> 172.21.15.114:0/3814931215 conn(0x7fbfa40d0b90 msgr2=0x7fbfa40cb510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.599+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 shutdown_connections 2024-03-06T07:28:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.599+0000 7fbfa9000640 1 -- 172.21.15.114:0/3814931215 wait complete. 2024-03-06T07:28:04.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-03-06T07:28:04.618 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:04.618 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:28:04.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.865+0000 7f236fe00640 1 Processor -- start 2024-03-06T07:28:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.865+0000 7f236fe00640 1 -- start start 2024-03-06T07:28:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700cc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f237015f0f0 0x7f23700ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23700ce7e0 0x7f2370161940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23700cec50 con 0x7f23700ce7e0 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236fe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237015f510 con 0x7f237015f0f0 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23700cd060 con 0x7f237015dfa0 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23700ce7e0 0x7f2370161940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23700ce7e0 0x7f2370161940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56166/0 (socket says 172.21.15.114:56166) 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236f400640 1 -- 172.21.15.114:0/791017240 learned_addr learned my addr 172.21.15.114:0/791017240 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236e000640 1 --2- 172.21.15.114:0/791017240 >> v2:172.21.15.177:3300/0 conn(0x7f237015f0f0 0x7f23700ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.867+0000 7f236ea00640 1 --2- 172.21.15.114:0/791017240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.868+0000 7f236f400640 1 -- 172.21.15.114:0/791017240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 msgr2=0x7f23700cc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.868+0000 7f236f400640 1 --2- 172.21.15.114:0/791017240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700cc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.868+0000 7f236f400640 1 -- 172.21.15.114:0/791017240 >> v2:172.21.15.177:3300/0 conn(0x7f237015f0f0 msgr2=0x7f23700ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.868+0000 7f236f400640 1 --2- 172.21.15.114:0/791017240 >> v2:172.21.15.177:3300/0 conn(0x7f237015f0f0 0x7f23700ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.868+0000 7f236f400640 1 -- 172.21.15.114:0/791017240 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23700cd340 con 0x7f23700ce7e0 2024-03-06T07:28:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.869+0000 7f236f400640 1 --2- 172.21.15.114:0/791017240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370161940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2370071ee0 tx=0x7f236802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0a73806823887b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.869+0000 7f236d600640 1 -- 172.21.15.114:0/791017240 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236802cb00 con 0x7f23700ce7e0 2024-03-06T07:28:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.869+0000 7f236d600640 1 -- 172.21.15.114:0/791017240 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236802cca0 con 0x7f23700ce7e0 2024-03-06T07:28:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.869+0000 7f236d600640 1 -- 172.21.15.114:0/791017240 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368034610 con 0x7f23700ce7e0 2024-03-06T07:28:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 -- 172.21.15.114:0/791017240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 msgr2=0x7f2370161940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 --2- 172.21.15.114:0/791017240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370161940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2370071ee0 tx=0x7f236802bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 -- 172.21.15.114:0/791017240 shutdown_connections 2024-03-06T07:28:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 --2- 172.21.15.114:0/791017240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370161940 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 --2- 172.21.15.114:0/791017240 >> v2:172.21.15.177:3300/0 conn(0x7f237015f0f0 0x7f23700ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 --2- 172.21.15.114:0/791017240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 -- 172.21.15.114:0/791017240 >> 172.21.15.114:0/791017240 conn(0x7f23700b3620 msgr2=0x7f23700b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 -- 172.21.15.114:0/791017240 shutdown_connections 2024-03-06T07:28:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.870+0000 7f236fe00640 1 -- 172.21.15.114:0/791017240 wait complete. 2024-03-06T07:28:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.872+0000 7f236fe00640 1 Processor -- start 2024-03-06T07:28:04.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.872+0000 7f236fe00640 1 -- start start 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f237015f0f0 0x7f23700caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700cdbe0 con 0x7f23700ce7e0 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700cd900 con 0x7f237015f0f0 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2370161e80 con 0x7f237015dfa0 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56180/0 (socket says 172.21.15.114:56180) 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 -- 172.21.15.114:0/2534819650 learned_addr learned my addr 172.21.15.114:0/2534819650 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236e000640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236f400640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f237015f0f0 0x7f23700caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 -- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 msgr2=0x7f23700b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 -- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f237015f0f0 msgr2=0x7f23700caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f237015f0f0 0x7f23700caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236ea00640 1 -- 172.21.15.114:0/2534819650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23700ae900 con 0x7f23700ce7e0 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.873+0000 7f236f400640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f237015f0f0 0x7f23700caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.874+0000 7f236ea00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370157110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f236000c820 tx=0x7f236000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.874+0000 7f2366a00640 1 -- 172.21.15.114:0/2534819650 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360015070 con 0x7f23700ce7e0 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.874+0000 7f2366a00640 1 -- 172.21.15.114:0/2534819650 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2360004220 con 0x7f23700ce7e0 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.874+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23700b0cf0 con 0x7f23700ce7e0 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.874+0000 7f2366a00640 1 -- 172.21.15.114:0/2534819650 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236000ce10 con 0x7f23700ce7e0 2024-03-06T07:28:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.874+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23701f4210 con 0x7f23700ce7e0 2024-03-06T07:28:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.875+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237004f330 con 0x7f23700ce7e0 2024-03-06T07:28:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.876+0000 7f2366a00640 1 -- 172.21.15.114:0/2534819650 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2360007830 con 0x7f23700ce7e0 2024-03-06T07:28:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.877+0000 7f2366a00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f235006b500 0x7f235006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.877+0000 7f2366a00640 1 -- 172.21.15.114:0/2534819650 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f236008a0b0 con 0x7f23700ce7e0 2024-03-06T07:28:04.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.877+0000 7f236e000640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f235006b500 0x7f235006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.878+0000 7f236e000640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f235006b500 0x7f235006d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f235c006fd0 tx=0x7f235c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:04.886+0000 7f2366a00640 1 -- 172.21.15.114:0/2534819650 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23700ae900 con 0x7f23700ce7e0 2024-03-06T07:28:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.128+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f237006fc80 con 0x7f235006b500 2024-03-06T07:28:05.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.135+0000 7f2366a00640 1 -- 172.21.15.114:0/2534819650 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f237006fc80 con 0x7f235006b500 2024-03-06T07:28:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:05.143 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:28:05.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f235006b500 msgr2=0x7f235006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f235006b500 0x7f235006d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f235c006fd0 tx=0x7f235c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:05.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 msgr2=0x7f2370157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370157110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f236000c820 tx=0x7f236000cce0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 shutdown_connections 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f237015f0f0 0x7f23700caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f237015dfa0 0x7f23700b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f235006b500 0x7f235006d9c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 --2- 172.21.15.114:0/2534819650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23700ce7e0 0x7f2370157110 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.140+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 >> 172.21.15.114:0/2534819650 conn(0x7f23700b3620 msgr2=0x7f23700b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.141+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 shutdown_connections 2024-03-06T07:28:05.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.141+0000 7f236fe00640 1 -- 172.21.15.114:0/2534819650 wait complete. 2024-03-06T07:28:05.159 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:28:05.102540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002964"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517399+0000","last_change":"2024-03-06T07:27:55.014476+0000","last_active":"2024-03-06T07:27:56.517399+0000","last_peered":"2024-03-06T07:27:56.517399+0000","last_clean":"2024-03-06T07:27:56.517399+0000","last_became_active":"2024-03-06T07:27:55.014169+0000","last_became_peered":"2024-03-06T07:27:55.014169+0000","last_unstale":"2024-03-06T07:27:56.517399+0000","last_undegraded":"2024-03-06T07:27:56.517399+0000","last_fullsized":"2024-03-06T07:27:56.517399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:20.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020046+0000","last_change":"2024-03-06T07:27:55.025640+0000","last_active":"2024-03-06T07:27:56.020046+0000","last_peered":"2024-03-06T07:27:56.020046+0000","last_clean":"2024-03-06T07:27:56.020046+0000","last_became_active":"2024-03-06T07:27:55.025215+0000","last_became_peered":"2024-03-06T07:27:55.025215+0000","last_unstale":"2024-03-06T07:27:56.020046+0000","last_undegraded":"2024-03-06T07:27:56.020046+0000","last_fullsized":"2024-03-06T07:27:56.020046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514320+0000","last_change":"2024-03-06T07:27:55.015505+0000","last_active":"2024-03-06T07:27:56.514320+0000","last_peered":"2024-03-06T07:27:56.514320+0000","last_clean":"2024-03-06T07:27:56.514320+0000","last_became_active":"2024-03-06T07:27:55.015199+0000","last_became_peered":"2024-03-06T07:27:55.015199+0000","last_unstale":"2024-03-06T07:27:56.514320+0000","last_undegraded":"2024-03-06T07:27:56.514320+0000","last_fullsized":"2024-03-06T07:27:56.514320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518415+0000","last_change":"2024-03-06T07:27:55.033897+0000","last_active":"2024-03-06T07:27:56.518415+0000","last_peered":"2024-03-06T07:27:56.518415+0000","last_clean":"2024-03-06T07:27:56.518415+0000","last_became_active":"2024-03-06T07:27:55.032987+0000","last_became_peered":"2024-03-06T07:27:55.032987+0000","last_unstale":"2024-03-06T07:27:56.518415+0000","last_undegraded":"2024-03-06T07:27:56.518415+0000","last_fullsized":"2024-03-06T07:27:56.518415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515678+0000","last_change":"2024-03-06T07:27:55.014597+0000","last_active":"2024-03-06T07:27:56.515678+0000","last_peered":"2024-03-06T07:27:56.515678+0000","last_clean":"2024-03-06T07:27:56.515678+0000","last_became_active":"2024-03-06T07:27:55.014206+0000","last_became_peered":"2024-03-06T07:27:55.014206+0000","last_unstale":"2024-03-06T07:27:56.515678+0000","last_undegraded":"2024-03-06T07:27:56.515678+0000","last_fullsized":"2024-03-06T07:27:56.515678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516836+0000","last_change":"2024-03-06T07:27:55.028575+0000","last_active":"2024-03-06T07:27:56.516836+0000","last_peered":"2024-03-06T07:27:56.516836+0000","last_clean":"2024-03-06T07:27:56.516836+0000","last_became_active":"2024-03-06T07:27:55.027801+0000","last_became_peered":"2024-03-06T07:27:55.027801+0000","last_unstale":"2024-03-06T07:27:56.516836+0000","last_undegraded":"2024-03-06T07:27:56.516836+0000","last_fullsized":"2024-03-06T07:27:56.516836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514995+0000","last_change":"2024-03-06T07:27:55.028417+0000","last_active":"2024-03-06T07:27:56.514995+0000","last_peered":"2024-03-06T07:27:56.514995+0000","last_clean":"2024-03-06T07:27:56.514995+0000","last_became_active":"2024-03-06T07:27:55.028108+0000","last_became_peered":"2024-03-06T07:27:55.028108+0000","last_unstale":"2024-03-06T07:27:56.514995+0000","last_undegraded":"2024-03-06T07:27:56.514995+0000","last_fullsized":"2024-03-06T07:27:56.514995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021025+0000","last_change":"2024-03-06T07:27:55.024045+0000","last_active":"2024-03-06T07:27:56.021025+0000","last_peered":"2024-03-06T07:27:56.021025+0000","last_clean":"2024-03-06T07:27:56.021025+0000","last_became_active":"2024-03-06T07:27:55.023528+0000","last_became_peered":"2024-03-06T07:27:55.023528+0000","last_unstale":"2024-03-06T07:27:56.021025+0000","last_undegraded":"2024-03-06T07:27:56.021025+0000","last_fullsized":"2024-03-06T07:27:56.021025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517632+0000","last_change":"2024-03-06T07:27:55.021303+0000","last_active":"2024-03-06T07:27:56.517632+0000","last_peered":"2024-03-06T07:27:56.517632+0000","last_clean":"2024-03-06T07:27:56.517632+0000","last_became_active":"2024-03-06T07:27:55.020953+0000","last_became_peered":"2024-03-06T07:27:55.020953+0000","last_unstale":"2024-03-06T07:27:56.517632+0000","last_undegraded":"2024-03-06T07:27:56.517632+0000","last_fullsized":"2024-03-06T07:27:56.517632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:30.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027586+0000","last_change":"2024-03-06T07:27:55.019249+0000","last_active":"2024-03-06T07:27:56.027586+0000","last_peered":"2024-03-06T07:27:56.027586+0000","last_clean":"2024-03-06T07:27:56.027586+0000","last_became_active":"2024-03-06T07:27:55.018947+0000","last_became_peered":"2024-03-06T07:27:55.018947+0000","last_unstale":"2024-03-06T07:27:56.027586+0000","last_undegraded":"2024-03-06T07:27:56.027586+0000","last_fullsized":"2024-03-06T07:27:56.027586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027423+0000","last_change":"2024-03-06T07:27:55.023288+0000","last_active":"2024-03-06T07:27:56.027423+0000","last_peered":"2024-03-06T07:27:56.027423+0000","last_clean":"2024-03-06T07:27:56.027423+0000","last_became_active":"2024-03-06T07:27:55.023020+0000","last_became_peered":"2024-03-06T07:27:55.023020+0000","last_unstale":"2024-03-06T07:27:56.027423+0000","last_undegraded":"2024-03-06T07:27:56.027423+0000","last_fullsized":"2024-03-06T07:27:56.027423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029290+0000","last_change":"2024-03-06T07:27:55.014888+0000","last_active":"2024-03-06T07:27:56.029290+0000","last_peered":"2024-03-06T07:27:56.029290+0000","last_clean":"2024-03-06T07:27:56.029290+0000","last_became_active":"2024-03-06T07:27:55.014394+0000","last_became_peered":"2024-03-06T07:27:55.014394+0000","last_unstale":"2024-03-06T07:27:56.029290+0000","last_undegraded":"2024-03-06T07:27:56.029290+0000","last_fullsized":"2024-03-06T07:27:56.029290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:58.566589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021843+0000","last_change":"2024-03-06T07:27:55.014936+0000","last_active":"2024-03-06T07:27:56.021843+0000","last_peered":"2024-03-06T07:27:56.021843+0000","last_clean":"2024-03-06T07:27:56.021843+0000","last_became_active":"2024-03-06T07:27:55.012277+0000","last_became_peered":"2024-03-06T07:27:55.012277+0000","last_unstale":"2024-03-06T07:27:56.021843+0000","last_undegraded":"2024-03-06T07:27:56.021843+0000","last_fullsized":"2024-03-06T07:27:56.021843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.012205+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.011928+0000","last_became_peered":"2024-03-06T07:27:55.011928+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019838+0000","last_change":"2024-03-06T07:27:55.030004+0000","last_active":"2024-03-06T07:27:56.019838+0000","last_peered":"2024-03-06T07:27:56.019838+0000","last_clean":"2024-03-06T07:27:56.019838+0000","last_became_active":"2024-03-06T07:27:55.029632+0000","last_became_peered":"2024-03-06T07:27:55.029632+0000","last_unstale":"2024-03-06T07:27:56.019838+0000","last_undegraded":"2024-03-06T07:27:56.019838+0000","last_fullsized":"2024-03-06T07:27:56.019838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515777+0000","last_change":"2024-03-06T07:27:55.028212+0000","last_active":"2024-03-06T07:27:56.515777+0000","last_peered":"2024-03-06T07:27:56.515777+0000","last_clean":"2024-03-06T07:27:56.515777+0000","last_became_active":"2024-03-06T07:27:55.027899+0000","last_became_peered":"2024-03-06T07:27:55.027899+0000","last_unstale":"2024-03-06T07:27:56.515777+0000","last_undegraded":"2024-03-06T07:27:56.515777+0000","last_fullsized":"2024-03-06T07:27:56.515777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519519+0000","last_change":"2024-03-06T07:27:55.024791+0000","last_active":"2024-03-06T07:27:56.519519+0000","last_peered":"2024-03-06T07:27:56.519519+0000","last_clean":"2024-03-06T07:27:56.519519+0000","last_became_active":"2024-03-06T07:27:55.023356+0000","last_became_peered":"2024-03-06T07:27:55.023356+0000","last_unstale":"2024-03-06T07:27:56.519519+0000","last_undegraded":"2024-03-06T07:27:56.519519+0000","last_fullsized":"2024-03-06T07:27:56.519519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514884+0000","last_change":"2024-03-06T07:27:54.994313+0000","last_active":"2024-03-06T07:27:56.514884+0000","last_peered":"2024-03-06T07:27:56.514884+0000","last_clean":"2024-03-06T07:27:56.514884+0000","last_became_active":"2024-03-06T07:27:54.993537+0000","last_became_peered":"2024-03-06T07:27:54.993537+0000","last_unstale":"2024-03-06T07:27:56.514884+0000","last_undegraded":"2024-03-06T07:27:56.514884+0000","last_fullsized":"2024-03-06T07:27:56.514884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:34.222639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516724+0000","last_change":"2024-03-06T07:27:55.016027+0000","last_active":"2024-03-06T07:27:56.516724+0000","last_peered":"2024-03-06T07:27:56.516724+0000","last_clean":"2024-03-06T07:27:56.516724+0000","last_became_active":"2024-03-06T07:27:55.015462+0000","last_became_peered":"2024-03-06T07:27:55.015462+0000","last_unstale":"2024-03-06T07:27:56.516724+0000","last_undegraded":"2024-03-06T07:27:56.516724+0000","last_fullsized":"2024-03-06T07:27:56.516724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027150+0000","last_change":"2024-03-06T07:27:55.015671+0000","last_active":"2024-03-06T07:27:56.027150+0000","last_peered":"2024-03-06T07:27:56.027150+0000","last_clean":"2024-03-06T07:27:56.027150+0000","last_became_active":"2024-03-06T07:27:55.015352+0000","last_became_peered":"2024-03-06T07:27:55.015352+0000","last_unstale":"2024-03-06T07:27:56.027150+0000","last_undegraded":"2024-03-06T07:27:56.027150+0000","last_fullsized":"2024-03-06T07:27:56.027150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513815+0000","last_change":"2024-03-06T07:27:55.026752+0000","last_active":"2024-03-06T07:27:56.513815+0000","last_peered":"2024-03-06T07:27:56.513815+0000","last_clean":"2024-03-06T07:27:56.513815+0000","last_became_active":"2024-03-06T07:27:55.026382+0000","last_became_peered":"2024-03-06T07:27:55.026382+0000","last_unstale":"2024-03-06T07:27:56.513815+0000","last_undegraded":"2024-03-06T07:27:56.513815+0000","last_fullsized":"2024-03-06T07:27:56.513815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517269+0000","last_change":"2024-03-06T07:27:55.024122+0000","last_active":"2024-03-06T07:27:56.517269+0000","last_peered":"2024-03-06T07:27:56.517269+0000","last_clean":"2024-03-06T07:27:56.517269+0000","last_became_active":"2024-03-06T07:27:55.023796+0000","last_became_peered":"2024-03-06T07:27:55.023796+0000","last_unstale":"2024-03-06T07:27:56.517269+0000","last_undegraded":"2024-03-06T07:27:56.517269+0000","last_fullsized":"2024-03-06T07:27:56.517269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021097+0000","last_change":"2024-03-06T07:27:55.020822+0000","last_active":"2024-03-06T07:27:56.021097+0000","last_peered":"2024-03-06T07:27:56.021097+0000","last_clean":"2024-03-06T07:27:56.021097+0000","last_became_active":"2024-03-06T07:27:55.020406+0000","last_became_peered":"2024-03-06T07:27:55.020406+0000","last_unstale":"2024-03-06T07:27:56.021097+0000","last_undegraded":"2024-03-06T07:27:56.021097+0000","last_fullsized":"2024-03-06T07:27:56.021097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516215+0000","last_change":"2024-03-06T07:27:55.016257+0000","last_active":"2024-03-06T07:27:56.516215+0000","last_peered":"2024-03-06T07:27:56.516215+0000","last_clean":"2024-03-06T07:27:56.516215+0000","last_became_active":"2024-03-06T07:27:55.013454+0000","last_became_peered":"2024-03-06T07:27:55.013454+0000","last_unstale":"2024-03-06T07:27:56.516215+0000","last_undegraded":"2024-03-06T07:27:56.516215+0000","last_fullsized":"2024-03-06T07:27:56.516215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021692+0000","last_change":"2024-03-06T07:27:55.014609+0000","last_active":"2024-03-06T07:27:56.021692+0000","last_peered":"2024-03-06T07:27:56.021692+0000","last_clean":"2024-03-06T07:27:56.021692+0000","last_became_active":"2024-03-06T07:27:55.014270+0000","last_became_peered":"2024-03-06T07:27:55.014270+0000","last_unstale":"2024-03-06T07:27:56.021692+0000","last_undegraded":"2024-03-06T07:27:56.021692+0000","last_fullsized":"2024-03-06T07:27:56.021692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027608+0000","last_change":"2024-03-06T07:27:55.012622+0000","last_active":"2024-03-06T07:27:56.027608+0000","last_peered":"2024-03-06T07:27:56.027608+0000","last_clean":"2024-03-06T07:27:56.027608+0000","last_became_active":"2024-03-06T07:27:55.012268+0000","last_became_peered":"2024-03-06T07:27:55.012268+0000","last_unstale":"2024-03-06T07:27:56.027608+0000","last_undegraded":"2024-03-06T07:27:56.027608+0000","last_fullsized":"2024-03-06T07:27:56.027608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019641+0000","last_change":"2024-03-06T07:27:55.025182+0000","last_active":"2024-03-06T07:27:56.019641+0000","last_peered":"2024-03-06T07:27:56.019641+0000","last_clean":"2024-03-06T07:27:56.019641+0000","last_became_active":"2024-03-06T07:27:55.024874+0000","last_became_peered":"2024-03-06T07:27:55.024874+0000","last_unstale":"2024-03-06T07:27:56.019641+0000","last_undegraded":"2024-03-06T07:27:56.019641+0000","last_fullsized":"2024-03-06T07:27:56.019641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.022062+0000","last_change":"2024-03-06T07:27:55.031727+0000","last_active":"2024-03-06T07:27:56.022062+0000","last_peered":"2024-03-06T07:27:56.022062+0000","last_clean":"2024-03-06T07:27:56.022062+0000","last_became_active":"2024-03-06T07:27:55.029328+0000","last_became_peered":"2024-03-06T07:27:55.029328+0000","last_unstale":"2024-03-06T07:27:56.022062+0000","last_undegraded":"2024-03-06T07:27:56.022062+0000","last_fullsized":"2024-03-06T07:27:56.022062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513894+0000","last_change":"2024-03-06T07:27:55.011251+0000","last_active":"2024-03-06T07:27:56.513894+0000","last_peered":"2024-03-06T07:27:56.513894+0000","last_clean":"2024-03-06T07:27:56.513894+0000","last_became_active":"2024-03-06T07:27:55.010892+0000","last_became_peered":"2024-03-06T07:27:55.010892+0000","last_unstale":"2024-03-06T07:27:56.513894+0000","last_undegraded":"2024-03-06T07:27:56.513894+0000","last_fullsized":"2024-03-06T07:27:56.513894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517664+0000","last_change":"2024-03-06T07:27:55.036627+0000","last_active":"2024-03-06T07:27:56.517664+0000","last_peered":"2024-03-06T07:27:56.517664+0000","last_clean":"2024-03-06T07:27:56.517664+0000","last_became_active":"2024-03-06T07:27:55.036245+0000","last_became_peered":"2024-03-06T07:27:55.036245+0000","last_unstale":"2024-03-06T07:27:56.517664+0000","last_undegraded":"2024-03-06T07:27:56.517664+0000","last_fullsized":"2024-03-06T07:27:56.517664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514454+0000","last_change":"2024-03-06T07:27:55.018254+0000","last_active":"2024-03-06T07:27:56.514454+0000","last_peered":"2024-03-06T07:27:56.514454+0000","last_clean":"2024-03-06T07:27:56.514454+0000","last_became_active":"2024-03-06T07:27:55.017832+0000","last_became_peered":"2024-03-06T07:27:55.017832+0000","last_unstale":"2024-03-06T07:27:56.514454+0000","last_undegraded":"2024-03-06T07:27:56.514454+0000","last_fullsized":"2024-03-06T07:27:56.514454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:11.749502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026113+0000","last_change":"2024-03-06T07:27:55.024465+0000","last_active":"2024-03-06T07:27:56.026113+0000","last_peered":"2024-03-06T07:27:56.026113+0000","last_clean":"2024-03-06T07:27:56.026113+0000","last_became_active":"2024-03-06T07:27:55.024173+0000","last_became_peered":"2024-03-06T07:27:55.024173+0000","last_unstale":"2024-03-06T07:27:56.026113+0000","last_undegraded":"2024-03-06T07:27:56.026113+0000","last_fullsized":"2024-03-06T07:27:56.026113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028167+0000","last_change":"2024-03-06T07:27:55.013597+0000","last_active":"2024-03-06T07:27:56.028167+0000","last_peered":"2024-03-06T07:27:56.028167+0000","last_clean":"2024-03-06T07:27:56.028167+0000","last_became_active":"2024-03-06T07:27:55.013199+0000","last_became_peered":"2024-03-06T07:27:55.013199+0000","last_unstale":"2024-03-06T07:27:56.028167+0000","last_undegraded":"2024-03-06T07:27:56.028167+0000","last_fullsized":"2024-03-06T07:27:56.028167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:13.574066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021899+0000","last_change":"2024-03-06T07:27:54.014875+0000","last_active":"2024-03-06T07:27:56.021899+0000","last_peered":"2024-03-06T07:27:56.021899+0000","last_clean":"2024-03-06T07:27:56.021899+0000","last_became_active":"2024-03-06T07:27:54.014526+0000","last_became_peered":"2024-03-06T07:27:54.014526+0000","last_unstale":"2024-03-06T07:27:56.021899+0000","last_undegraded":"2024-03-06T07:27:56.021899+0000","last_fullsized":"2024-03-06T07:27:56.021899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514018+0000","last_change":"2024-03-06T07:27:55.017432+0000","last_active":"2024-03-06T07:27:56.514018+0000","last_peered":"2024-03-06T07:27:56.514018+0000","last_clean":"2024-03-06T07:27:56.514018+0000","last_became_active":"2024-03-06T07:27:55.015135+0000","last_became_peered":"2024-03-06T07:27:55.015135+0000","last_unstale":"2024-03-06T07:27:56.514018+0000","last_undegraded":"2024-03-06T07:27:56.514018+0000","last_fullsized":"2024-03-06T07:27:56.514018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026039+0000","last_change":"2024-03-06T07:27:54.023420+0000","last_active":"2024-03-06T07:27:56.026039+0000","last_peered":"2024-03-06T07:27:56.026039+0000","last_clean":"2024-03-06T07:27:56.026039+0000","last_became_active":"2024-03-06T07:27:54.022665+0000","last_became_peered":"2024-03-06T07:27:54.022665+0000","last_unstale":"2024-03-06T07:27:56.026039+0000","last_undegraded":"2024-03-06T07:27:56.026039+0000","last_fullsized":"2024-03-06T07:27:56.026039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021422+0000","last_change":"2024-03-06T07:27:54.015366+0000","last_active":"2024-03-06T07:27:56.021422+0000","last_peered":"2024-03-06T07:27:56.021422+0000","last_clean":"2024-03-06T07:27:56.021422+0000","last_became_active":"2024-03-06T07:27:54.015155+0000","last_became_peered":"2024-03-06T07:27:54.015155+0000","last_unstale":"2024-03-06T07:27:56.021422+0000","last_undegraded":"2024-03-06T07:27:56.021422+0000","last_fullsized":"2024-03-06T07:27:56.021422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:04.758569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021171+0000","last_change":"2024-03-06T07:27:55.029914+0000","last_active":"2024-03-06T07:27:56.021171+0000","last_peered":"2024-03-06T07:27:56.021171+0000","last_clean":"2024-03-06T07:27:56.021171+0000","last_became_active":"2024-03-06T07:27:55.029599+0000","last_became_peered":"2024-03-06T07:27:55.029599+0000","last_unstale":"2024-03-06T07:27:56.021171+0000","last_undegraded":"2024-03-06T07:27:56.021171+0000","last_fullsized":"2024-03-06T07:27:56.021171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516973+0000","last_change":"2024-03-06T07:27:54.023284+0000","last_active":"2024-03-06T07:27:56.516973+0000","last_peered":"2024-03-06T07:27:56.516973+0000","last_clean":"2024-03-06T07:27:56.516973+0000","last_became_active":"2024-03-06T07:27:54.022909+0000","last_became_peered":"2024-03-06T07:27:54.022909+0000","last_unstale":"2024-03-06T07:27:56.516973+0000","last_undegraded":"2024-03-06T07:27:56.516973+0000","last_fullsized":"2024-03-06T07:27:56.516973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.016848+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.016590+0000","last_became_peered":"2024-03-06T07:27:55.016590+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028999+0000","last_change":"2024-03-06T07:27:54.025189+0000","last_active":"2024-03-06T07:27:56.028999+0000","last_peered":"2024-03-06T07:27:56.028999+0000","last_clean":"2024-03-06T07:27:56.028999+0000","last_became_active":"2024-03-06T07:27:54.024875+0000","last_became_peered":"2024-03-06T07:27:54.024875+0000","last_unstale":"2024-03-06T07:27:56.028999+0000","last_undegraded":"2024-03-06T07:27:56.028999+0000","last_fullsized":"2024-03-06T07:27:56.028999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519254+0000","last_change":"2024-03-06T07:27:55.018921+0000","last_active":"2024-03-06T07:27:56.519254+0000","last_peered":"2024-03-06T07:27:56.519254+0000","last_clean":"2024-03-06T07:27:56.519254+0000","last_became_active":"2024-03-06T07:27:55.018597+0000","last_became_peered":"2024-03-06T07:27:55.018597+0000","last_unstale":"2024-03-06T07:27:56.519254+0000","last_undegraded":"2024-03-06T07:27:56.519254+0000","last_fullsized":"2024-03-06T07:27:56.519254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519090+0000","last_change":"2024-03-06T07:27:54.025386+0000","last_active":"2024-03-06T07:27:56.519090+0000","last_peered":"2024-03-06T07:27:56.519090+0000","last_clean":"2024-03-06T07:27:56.519090+0000","last_became_active":"2024-03-06T07:27:54.020646+0000","last_became_peered":"2024-03-06T07:27:54.020646+0000","last_unstale":"2024-03-06T07:27:56.519090+0000","last_undegraded":"2024-03-06T07:27:56.519090+0000","last_fullsized":"2024-03-06T07:27:56.519090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514919+0000","last_change":"2024-03-06T07:27:55.022924+0000","last_active":"2024-03-06T07:27:56.514919+0000","last_peered":"2024-03-06T07:27:56.514919+0000","last_clean":"2024-03-06T07:27:56.514919+0000","last_became_active":"2024-03-06T07:27:55.022605+0000","last_became_peered":"2024-03-06T07:27:55.022605+0000","last_unstale":"2024-03-06T07:27:56.514919+0000","last_undegraded":"2024-03-06T07:27:56.514919+0000","last_fullsized":"2024-03-06T07:27:56.514919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019719+0000","last_change":"2024-03-06T07:27:54.012324+0000","last_active":"2024-03-06T07:27:56.019719+0000","last_peered":"2024-03-06T07:27:56.019719+0000","last_clean":"2024-03-06T07:27:56.019719+0000","last_became_active":"2024-03-06T07:27:54.012025+0000","last_became_peered":"2024-03-06T07:27:54.012025+0000","last_unstale":"2024-03-06T07:27:56.019719+0000","last_undegraded":"2024-03-06T07:27:56.019719+0000","last_fullsized":"2024-03-06T07:27:56.019719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029716+0000","last_change":"2024-03-06T07:27:55.028572+0000","last_active":"2024-03-06T07:27:56.029716+0000","last_peered":"2024-03-06T07:27:56.029716+0000","last_clean":"2024-03-06T07:27:56.029716+0000","last_became_active":"2024-03-06T07:27:55.026783+0000","last_became_peered":"2024-03-06T07:27:55.026783+0000","last_unstale":"2024-03-06T07:27:56.029716+0000","last_undegraded":"2024-03-06T07:27:56.029716+0000","last_fullsized":"2024-03-06T07:27:56.029716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:43.960764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027082+0000","last_change":"2024-03-06T07:27:54.011288+0000","last_active":"2024-03-06T07:27:56.027082+0000","last_peered":"2024-03-06T07:27:56.027082+0000","last_clean":"2024-03-06T07:27:56.027082+0000","last_became_active":"2024-03-06T07:27:54.010900+0000","last_became_peered":"2024-03-06T07:27:54.010900+0000","last_unstale":"2024-03-06T07:27:56.027082+0000","last_undegraded":"2024-03-06T07:27:56.027082+0000","last_fullsized":"2024-03-06T07:27:56.027082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515920+0000","last_change":"2024-03-06T07:27:55.035258+0000","last_active":"2024-03-06T07:27:56.515920+0000","last_peered":"2024-03-06T07:27:56.515920+0000","last_clean":"2024-03-06T07:27:56.515920+0000","last_became_active":"2024-03-06T07:27:55.034985+0000","last_became_peered":"2024-03-06T07:27:55.034985+0000","last_unstale":"2024-03-06T07:27:56.515920+0000","last_undegraded":"2024-03-06T07:27:56.515920+0000","last_fullsized":"2024-03-06T07:27:56.515920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513956+0000","last_change":"2024-03-06T07:27:54.019343+0000","last_active":"2024-03-06T07:27:56.513956+0000","last_peered":"2024-03-06T07:27:56.513956+0000","last_clean":"2024-03-06T07:27:56.513956+0000","last_became_active":"2024-03-06T07:27:54.019083+0000","last_became_peered":"2024-03-06T07:27:54.019083+0000","last_unstale":"2024-03-06T07:27:56.513956+0000","last_undegraded":"2024-03-06T07:27:56.513956+0000","last_fullsized":"2024-03-06T07:27:56.513956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:12.617582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.028252+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.027919+0000","last_became_peered":"2024-03-06T07:27:55.027919+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:57.800588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517776+0000","last_change":"2024-03-06T07:27:54.015808+0000","last_active":"2024-03-06T07:27:56.517776+0000","last_peered":"2024-03-06T07:27:56.517776+0000","last_clean":"2024-03-06T07:27:56.517776+0000","last_became_active":"2024-03-06T07:27:54.015279+0000","last_became_peered":"2024-03-06T07:27:54.015279+0000","last_unstale":"2024-03-06T07:27:56.517776+0000","last_undegraded":"2024-03-06T07:27:56.517776+0000","last_fullsized":"2024-03-06T07:27:56.517776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516046+0000","last_change":"2024-03-06T07:27:55.015432+0000","last_active":"2024-03-06T07:27:56.516046+0000","last_peered":"2024-03-06T07:27:56.516046+0000","last_clean":"2024-03-06T07:27:56.516046+0000","last_became_active":"2024-03-06T07:27:55.015132+0000","last_became_peered":"2024-03-06T07:27:55.015132+0000","last_unstale":"2024-03-06T07:27:56.516046+0000","last_undegraded":"2024-03-06T07:27:56.516046+0000","last_fullsized":"2024-03-06T07:27:56.516046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026103+0000","last_change":"2024-03-06T07:27:54.020993+0000","last_active":"2024-03-06T07:27:56.026103+0000","last_peered":"2024-03-06T07:27:56.026103+0000","last_clean":"2024-03-06T07:27:56.026103+0000","last_became_active":"2024-03-06T07:27:54.020618+0000","last_became_peered":"2024-03-06T07:27:54.020618+0000","last_unstale":"2024-03-06T07:27:56.026103+0000","last_undegraded":"2024-03-06T07:27:56.026103+0000","last_fullsized":"2024-03-06T07:27:56.026103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026185+0000","last_change":"2024-03-06T07:27:55.022089+0000","last_active":"2024-03-06T07:27:56.026185+0000","last_peered":"2024-03-06T07:27:56.026185+0000","last_clean":"2024-03-06T07:27:56.026185+0000","last_became_active":"2024-03-06T07:27:55.021113+0000","last_became_peered":"2024-03-06T07:27:55.021113+0000","last_unstale":"2024-03-06T07:27:56.026185+0000","last_undegraded":"2024-03-06T07:27:56.026185+0000","last_fullsized":"2024-03-06T07:27:56.026185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514788+0000","last_change":"2024-03-06T07:27:54.002389+0000","last_active":"2024-03-06T07:27:56.514788+0000","last_peered":"2024-03-06T07:27:56.514788+0000","last_clean":"2024-03-06T07:27:56.514788+0000","last_became_active":"2024-03-06T07:27:54.001928+0000","last_became_peered":"2024-03-06T07:27:54.001928+0000","last_unstale":"2024-03-06T07:27:56.514788+0000","last_undegraded":"2024-03-06T07:27:56.514788+0000","last_fullsized":"2024-03-06T07:27:56.514788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515180+0000","last_change":"2024-03-06T07:27:55.018703+0000","last_active":"2024-03-06T07:27:56.515180+0000","last_peered":"2024-03-06T07:27:56.515180+0000","last_clean":"2024-03-06T07:27:56.515180+0000","last_became_active":"2024-03-06T07:27:55.018315+0000","last_became_peered":"2024-03-06T07:27:55.018315+0000","last_unstale":"2024-03-06T07:27:56.515180+0000","last_undegraded":"2024-03-06T07:27:56.515180+0000","last_fullsized":"2024-03-06T07:27:56.515180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029303+0000","last_change":"2024-03-06T07:27:54.015632+0000","last_active":"2024-03-06T07:27:56.029303+0000","last_peered":"2024-03-06T07:27:56.029303+0000","last_clean":"2024-03-06T07:27:56.029303+0000","last_became_active":"2024-03-06T07:27:54.014810+0000","last_became_peered":"2024-03-06T07:27:54.014810+0000","last_unstale":"2024-03-06T07:27:56.029303+0000","last_undegraded":"2024-03-06T07:27:56.029303+0000","last_fullsized":"2024-03-06T07:27:56.029303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:09.001445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516849+0000","last_change":"2024-03-06T07:27:55.022324+0000","last_active":"2024-03-06T07:27:56.516849+0000","last_peered":"2024-03-06T07:27:56.516849+0000","last_clean":"2024-03-06T07:27:56.516849+0000","last_became_active":"2024-03-06T07:27:55.022013+0000","last_became_peered":"2024-03-06T07:27:55.022013+0000","last_unstale":"2024-03-06T07:27:56.516849+0000","last_undegraded":"2024-03-06T07:27:56.516849+0000","last_fullsized":"2024-03-06T07:27:56.516849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515058+0000","last_change":"2024-03-06T07:27:54.005847+0000","last_active":"2024-03-06T07:27:56.515058+0000","last_peered":"2024-03-06T07:27:56.515058+0000","last_clean":"2024-03-06T07:27:56.515058+0000","last_became_active":"2024-03-06T07:27:54.005460+0000","last_became_peered":"2024-03-06T07:27:54.005460+0000","last_unstale":"2024-03-06T07:27:56.515058+0000","last_undegraded":"2024-03-06T07:27:56.515058+0000","last_fullsized":"2024-03-06T07:27:56.515058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517605+0000","last_change":"2024-03-06T07:27:55.021338+0000","last_active":"2024-03-06T07:27:56.517605+0000","last_peered":"2024-03-06T07:27:56.517605+0000","last_clean":"2024-03-06T07:27:56.517605+0000","last_became_active":"2024-03-06T07:27:55.021017+0000","last_became_peered":"2024-03-06T07:27:55.021017+0000","last_unstale":"2024-03-06T07:27:56.517605+0000","last_undegraded":"2024-03-06T07:27:56.517605+0000","last_fullsized":"2024-03-06T07:27:56.517605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:47.579731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019961+0000","last_change":"2024-03-06T07:27:54.022289+0000","last_active":"2024-03-06T07:27:56.019961+0000","last_peered":"2024-03-06T07:27:56.019961+0000","last_clean":"2024-03-06T07:27:56.019961+0000","last_became_active":"2024-03-06T07:27:54.022098+0000","last_became_peered":"2024-03-06T07:27:54.022098+0000","last_unstale":"2024-03-06T07:27:56.019961+0000","last_undegraded":"2024-03-06T07:27:56.019961+0000","last_fullsized":"2024-03-06T07:27:56.019961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518836+0000","last_change":"2024-03-06T07:27:55.018543+0000","last_active":"2024-03-06T07:27:56.518836+0000","last_peered":"2024-03-06T07:27:56.518836+0000","last_clean":"2024-03-06T07:27:56.518836+0000","last_became_active":"2024-03-06T07:27:55.017787+0000","last_became_peered":"2024-03-06T07:27:55.017787+0000","last_unstale":"2024-03-06T07:27:56.518836+0000","last_undegraded":"2024-03-06T07:27:56.518836+0000","last_fullsized":"2024-03-06T07:27:56.518836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515131+0000","last_change":"2024-03-06T07:27:54.024731+0000","last_active":"2024-03-06T07:27:56.515131+0000","last_peered":"2024-03-06T07:27:56.515131+0000","last_clean":"2024-03-06T07:27:56.515131+0000","last_became_active":"2024-03-06T07:27:54.024389+0000","last_became_peered":"2024-03-06T07:27:54.024389+0000","last_unstale":"2024-03-06T07:27:56.515131+0000","last_undegraded":"2024-03-06T07:27:56.515131+0000","last_fullsized":"2024-03-06T07:27:56.515131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515569+0000","last_change":"2024-03-06T07:27:55.019440+0000","last_active":"2024-03-06T07:27:56.515569+0000","last_peered":"2024-03-06T07:27:56.515569+0000","last_clean":"2024-03-06T07:27:56.515569+0000","last_became_active":"2024-03-06T07:27:55.019018+0000","last_became_peered":"2024-03-06T07:27:55.019018+0000","last_unstale":"2024-03-06T07:27:56.515569+0000","last_undegraded":"2024-03-06T07:27:56.515569+0000","last_fullsized":"2024-03-06T07:27:56.515569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514828+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:27:56.514828+0000","last_peered":"2024-03-06T07:27:56.514828+0000","last_clean":"2024-03-06T07:27:56.514828+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:27:56.514828+0000","last_undegraded":"2024-03-06T07:27:56.514828+0000","last_fullsized":"2024-03-06T07:27:56.514828+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515546+0000","last_change":"2024-03-06T07:27:54.013591+0000","last_active":"2024-03-06T07:27:56.515546+0000","last_peered":"2024-03-06T07:27:56.515546+0000","last_clean":"2024-03-06T07:27:56.515546+0000","last_became_active":"2024-03-06T07:27:54.012154+0000","last_became_peered":"2024-03-06T07:27:54.012154+0000","last_unstale":"2024-03-06T07:27:56.515546+0000","last_undegraded":"2024-03-06T07:27:56.515546+0000","last_fullsized":"2024-03-06T07:27:56.515546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021505+0000","last_change":"2024-03-06T07:27:55.029404+0000","last_active":"2024-03-06T07:27:56.021505+0000","last_peered":"2024-03-06T07:27:56.021505+0000","last_clean":"2024-03-06T07:27:56.021505+0000","last_became_active":"2024-03-06T07:27:55.028890+0000","last_became_peered":"2024-03-06T07:27:55.028890+0000","last_unstale":"2024-03-06T07:27:56.021505+0000","last_undegraded":"2024-03-06T07:27:56.021505+0000","last_fullsized":"2024-03-06T07:27:56.021505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514470+0000","last_change":"2024-03-06T07:27:54.025195+0000","last_active":"2024-03-06T07:27:56.514470+0000","last_peered":"2024-03-06T07:27:56.514470+0000","last_clean":"2024-03-06T07:27:56.514470+0000","last_became_active":"2024-03-06T07:27:54.024942+0000","last_became_peered":"2024-03-06T07:27:54.024942+0000","last_unstale":"2024-03-06T07:27:56.514470+0000","last_undegraded":"2024-03-06T07:27:56.514470+0000","last_fullsized":"2024-03-06T07:27:56.514470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513738+0000","last_change":"2024-03-06T07:27:55.022642+0000","last_active":"2024-03-06T07:27:56.513738+0000","last_peered":"2024-03-06T07:27:56.513738+0000","last_clean":"2024-03-06T07:27:56.513738+0000","last_became_active":"2024-03-06T07:27:55.021096+0000","last_became_peered":"2024-03-06T07:27:55.021096+0000","last_unstale":"2024-03-06T07:27:56.513738+0000","last_undegraded":"2024-03-06T07:27:56.513738+0000","last_fullsized":"2024-03-06T07:27:56.513738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514360+0000","last_change":"2024-03-06T07:27:54.011028+0000","last_active":"2024-03-06T07:27:56.514360+0000","last_peered":"2024-03-06T07:27:56.514360+0000","last_clean":"2024-03-06T07:27:56.514360+0000","last_became_active":"2024-03-06T07:27:54.010767+0000","last_became_peered":"2024-03-06T07:27:54.010767+0000","last_unstale":"2024-03-06T07:27:56.514360+0000","last_undegraded":"2024-03-06T07:27:56.514360+0000","last_fullsized":"2024-03-06T07:27:56.514360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019932+0000","last_change":"2024-03-06T07:27:55.014190+0000","last_active":"2024-03-06T07:27:56.019932+0000","last_peered":"2024-03-06T07:27:56.019932+0000","last_clean":"2024-03-06T07:27:56.019932+0000","last_became_active":"2024-03-06T07:27:55.013895+0000","last_became_peered":"2024-03-06T07:27:55.013895+0000","last_unstale":"2024-03-06T07:27:56.019932+0000","last_undegraded":"2024-03-06T07:27:56.019932+0000","last_fullsized":"2024-03-06T07:27:56.019932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517915+0000","last_change":"2024-03-06T07:27:54.015537+0000","last_active":"2024-03-06T07:27:56.517915+0000","last_peered":"2024-03-06T07:27:56.517915+0000","last_clean":"2024-03-06T07:27:56.517915+0000","last_became_active":"2024-03-06T07:27:54.014754+0000","last_became_peered":"2024-03-06T07:27:54.014754+0000","last_unstale":"2024-03-06T07:27:56.517915+0000","last_undegraded":"2024-03-06T07:27:56.517915+0000","last_fullsized":"2024-03-06T07:27:56.517915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028699+0000","last_change":"2024-03-06T07:27:55.015839+0000","last_active":"2024-03-06T07:27:56.028699+0000","last_peered":"2024-03-06T07:27:56.028699+0000","last_clean":"2024-03-06T07:27:56.028699+0000","last_became_active":"2024-03-06T07:27:55.015468+0000","last_became_peered":"2024-03-06T07:27:55.015468+0000","last_unstale":"2024-03-06T07:27:56.028699+0000","last_undegraded":"2024-03-06T07:27:56.028699+0000","last_fullsized":"2024-03-06T07:27:56.028699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:41.571023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019749+0000","last_change":"2024-03-06T07:27:54.003778+0000","last_active":"2024-03-06T07:27:56.019749+0000","last_peered":"2024-03-06T07:27:56.019749+0000","last_clean":"2024-03-06T07:27:56.019749+0000","last_became_active":"2024-03-06T07:27:54.003482+0000","last_became_peered":"2024-03-06T07:27:54.003482+0000","last_unstale":"2024-03-06T07:27:56.019749+0000","last_undegraded":"2024-03-06T07:27:56.019749+0000","last_fullsized":"2024-03-06T07:27:56.019749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517164+0000","last_change":"2024-03-06T07:27:55.036670+0000","last_active":"2024-03-06T07:27:56.517164+0000","last_peered":"2024-03-06T07:27:56.517164+0000","last_clean":"2024-03-06T07:27:56.517164+0000","last_became_active":"2024-03-06T07:27:55.036328+0000","last_became_peered":"2024-03-06T07:27:55.036328+0000","last_unstale":"2024-03-06T07:27:56.517164+0000","last_undegraded":"2024-03-06T07:27:56.517164+0000","last_fullsized":"2024-03-06T07:27:56.517164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516091+0000","last_change":"2024-03-06T07:27:54.022591+0000","last_active":"2024-03-06T07:27:56.516091+0000","last_peered":"2024-03-06T07:27:56.516091+0000","last_clean":"2024-03-06T07:27:56.516091+0000","last_became_active":"2024-03-06T07:27:54.022296+0000","last_became_peered":"2024-03-06T07:27:54.022296+0000","last_unstale":"2024-03-06T07:27:56.516091+0000","last_undegraded":"2024-03-06T07:27:56.516091+0000","last_fullsized":"2024-03-06T07:27:56.516091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:21.651372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.010897+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.005758+0000","last_became_peered":"2024-03-06T07:27:55.005758+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020051+0000","last_change":"2024-03-06T07:27:54.018547+0000","last_active":"2024-03-06T07:27:56.020051+0000","last_peered":"2024-03-06T07:27:56.020051+0000","last_clean":"2024-03-06T07:27:56.020051+0000","last_became_active":"2024-03-06T07:27:54.018215+0000","last_became_peered":"2024-03-06T07:27:54.018215+0000","last_unstale":"2024-03-06T07:27:56.020051+0000","last_undegraded":"2024-03-06T07:27:56.020051+0000","last_fullsized":"2024-03-06T07:27:56.020051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021795+0000","last_change":"2024-03-06T07:27:55.013861+0000","last_active":"2024-03-06T07:27:56.021795+0000","last_peered":"2024-03-06T07:27:56.021795+0000","last_clean":"2024-03-06T07:27:56.021795+0000","last_became_active":"2024-03-06T07:27:55.013498+0000","last_became_peered":"2024-03-06T07:27:55.013498+0000","last_unstale":"2024-03-06T07:27:56.021795+0000","last_undegraded":"2024-03-06T07:27:56.021795+0000","last_fullsized":"2024-03-06T07:27:56.021795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029229+0000","last_change":"2024-03-06T07:27:54.015491+0000","last_active":"2024-03-06T07:27:56.029229+0000","last_peered":"2024-03-06T07:27:56.029229+0000","last_clean":"2024-03-06T07:27:56.029229+0000","last_became_active":"2024-03-06T07:27:54.015207+0000","last_became_peered":"2024-03-06T07:27:54.015207+0000","last_unstale":"2024-03-06T07:27:56.029229+0000","last_undegraded":"2024-03-06T07:27:56.029229+0000","last_fullsized":"2024-03-06T07:27:56.029229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514806+0000","last_change":"2024-03-06T07:27:55.021833+0000","last_active":"2024-03-06T07:27:56.514806+0000","last_peered":"2024-03-06T07:27:56.514806+0000","last_clean":"2024-03-06T07:27:56.514806+0000","last_became_active":"2024-03-06T07:27:55.021478+0000","last_became_peered":"2024-03-06T07:27:55.021478+0000","last_unstale":"2024-03-06T07:27:56.514806+0000","last_undegraded":"2024-03-06T07:27:56.514806+0000","last_fullsized":"2024-03-06T07:27:56.514806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514469+0000","last_change":"2024-03-06T07:27:54.024577+0000","last_active":"2024-03-06T07:27:56.514469+0000","last_peered":"2024-03-06T07:27:56.514469+0000","last_clean":"2024-03-06T07:27:56.514469+0000","last_became_active":"2024-03-06T07:27:54.023972+0000","last_became_peered":"2024-03-06T07:27:54.023972+0000","last_unstale":"2024-03-06T07:27:56.514469+0000","last_undegraded":"2024-03-06T07:27:56.514469+0000","last_fullsized":"2024-03-06T07:27:56.514469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.025293+0000","last_change":"2024-03-06T07:27:55.020868+0000","last_active":"2024-03-06T07:27:56.025293+0000","last_peered":"2024-03-06T07:27:56.025293+0000","last_clean":"2024-03-06T07:27:56.025293+0000","last_became_active":"2024-03-06T07:27:55.019789+0000","last_became_peered":"2024-03-06T07:27:55.019789+0000","last_unstale":"2024-03-06T07:27:56.025293+0000","last_undegraded":"2024-03-06T07:27:56.025293+0000","last_fullsized":"2024-03-06T07:27:56.025293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:54.669987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019875+0000","last_change":"2024-03-06T07:27:54.014058+0000","last_active":"2024-03-06T07:27:56.019875+0000","last_peered":"2024-03-06T07:27:56.019875+0000","last_clean":"2024-03-06T07:27:56.019875+0000","last_became_active":"2024-03-06T07:27:54.013803+0000","last_became_peered":"2024-03-06T07:27:54.013803+0000","last_unstale":"2024-03-06T07:27:56.019875+0000","last_undegraded":"2024-03-06T07:27:56.019875+0000","last_fullsized":"2024-03-06T07:27:56.019875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021185+0000","last_change":"2024-03-06T07:27:55.020235+0000","last_active":"2024-03-06T07:27:56.021185+0000","last_peered":"2024-03-06T07:27:56.021185+0000","last_clean":"2024-03-06T07:27:56.021185+0000","last_became_active":"2024-03-06T07:27:55.015366+0000","last_became_peered":"2024-03-06T07:27:55.015366+0000","last_unstale":"2024-03-06T07:27:56.021185+0000","last_undegraded":"2024-03-06T07:27:56.021185+0000","last_fullsized":"2024-03-06T07:27:56.021185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019800+0000","last_change":"2024-03-06T07:27:54.005998+0000","last_active":"2024-03-06T07:27:56.019800+0000","last_peered":"2024-03-06T07:27:56.019800+0000","last_clean":"2024-03-06T07:27:56.019800+0000","last_became_active":"2024-03-06T07:27:54.005646+0000","last_became_peered":"2024-03-06T07:27:54.005646+0000","last_unstale":"2024-03-06T07:27:56.019800+0000","last_undegraded":"2024-03-06T07:27:56.019800+0000","last_fullsized":"2024-03-06T07:27:56.019800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021888+0000","last_change":"2024-03-06T07:27:55.005310+0000","last_active":"2024-03-06T07:27:56.021888+0000","last_peered":"2024-03-06T07:27:56.021888+0000","last_clean":"2024-03-06T07:27:56.021888+0000","last_became_active":"2024-03-06T07:27:55.004647+0000","last_became_peered":"2024-03-06T07:27:55.004647+0000","last_unstale":"2024-03-06T07:27:56.021888+0000","last_undegraded":"2024-03-06T07:27:56.021888+0000","last_fullsized":"2024-03-06T07:27:56.021888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515995+0000","last_change":"2024-03-06T07:27:54.027960+0000","last_active":"2024-03-06T07:27:56.515995+0000","last_peered":"2024-03-06T07:27:56.515995+0000","last_clean":"2024-03-06T07:27:56.515995+0000","last_became_active":"2024-03-06T07:27:54.027317+0000","last_became_peered":"2024-03-06T07:27:54.027317+0000","last_unstale":"2024-03-06T07:27:56.515995+0000","last_undegraded":"2024-03-06T07:27:56.515995+0000","last_fullsized":"2024-03-06T07:27:56.515995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:41.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026502+0000","last_change":"2024-03-06T07:27:55.025331+0000","last_active":"2024-03-06T07:27:56.026502+0000","last_peered":"2024-03-06T07:27:56.026502+0000","last_clean":"2024-03-06T07:27:56.026502+0000","last_became_active":"2024-03-06T07:27:55.024981+0000","last_became_peered":"2024-03-06T07:27:55.024981+0000","last_unstale":"2024-03-06T07:27:56.026502+0000","last_undegraded":"2024-03-06T07:27:56.026502+0000","last_fullsized":"2024-03-06T07:27:56.026502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516878+0000","last_change":"2024-03-06T07:27:55.023997+0000","last_active":"2024-03-06T07:27:56.516878+0000","last_peered":"2024-03-06T07:27:56.516878+0000","last_clean":"2024-03-06T07:27:56.516878+0000","last_became_active":"2024-03-06T07:27:55.023617+0000","last_became_peered":"2024-03-06T07:27:55.023617+0000","last_unstale":"2024-03-06T07:27:56.516878+0000","last_undegraded":"2024-03-06T07:27:56.516878+0000","last_fullsized":"2024-03-06T07:27:56.516878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515609+0000","last_change":"2024-03-06T07:27:54.028923+0000","last_active":"2024-03-06T07:27:56.515609+0000","last_peered":"2024-03-06T07:27:56.515609+0000","last_clean":"2024-03-06T07:27:56.515609+0000","last_became_active":"2024-03-06T07:27:54.028290+0000","last_became_peered":"2024-03-06T07:27:54.028290+0000","last_unstale":"2024-03-06T07:27:56.515609+0000","last_undegraded":"2024-03-06T07:27:56.515609+0000","last_fullsized":"2024-03-06T07:27:56.515609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:10.734558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518611+0000","last_change":"2024-03-06T07:27:55.020150+0000","last_active":"2024-03-06T07:27:56.518611+0000","last_peered":"2024-03-06T07:27:56.518611+0000","last_clean":"2024-03-06T07:27:56.518611+0000","last_became_active":"2024-03-06T07:27:55.019546+0000","last_became_peered":"2024-03-06T07:27:55.019546+0000","last_unstale":"2024-03-06T07:27:56.518611+0000","last_undegraded":"2024-03-06T07:27:56.518611+0000","last_fullsized":"2024-03-06T07:27:56.518611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515741+0000","last_change":"2024-03-06T07:27:54.027548+0000","last_active":"2024-03-06T07:27:56.515741+0000","last_peered":"2024-03-06T07:27:56.515741+0000","last_clean":"2024-03-06T07:27:56.515741+0000","last_became_active":"2024-03-06T07:27:54.027161+0000","last_became_peered":"2024-03-06T07:27:54.027161+0000","last_unstale":"2024-03-06T07:27:56.515741+0000","last_undegraded":"2024-03-06T07:27:56.515741+0000","last_fullsized":"2024-03-06T07:27:56.515741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514542+0000","last_change":"2024-03-06T07:27:55.019647+0000","last_active":"2024-03-06T07:27:56.514542+0000","last_peered":"2024-03-06T07:27:56.514542+0000","last_clean":"2024-03-06T07:27:56.514542+0000","last_became_active":"2024-03-06T07:27:55.018449+0000","last_became_peered":"2024-03-06T07:27:55.018449+0000","last_unstale":"2024-03-06T07:27:56.514542+0000","last_undegraded":"2024-03-06T07:27:56.514542+0000","last_fullsized":"2024-03-06T07:27:56.514542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021792+0000","last_change":"2024-03-06T07:27:54.008978+0000","last_active":"2024-03-06T07:27:56.021792+0000","last_peered":"2024-03-06T07:27:56.021792+0000","last_clean":"2024-03-06T07:27:56.021792+0000","last_became_active":"2024-03-06T07:27:54.004840+0000","last_became_peered":"2024-03-06T07:27:54.004840+0000","last_unstale":"2024-03-06T07:27:56.021792+0000","last_undegraded":"2024-03-06T07:27:56.021792+0000","last_fullsized":"2024-03-06T07:27:56.021792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021485+0000","last_change":"2024-03-06T07:27:55.013247+0000","last_active":"2024-03-06T07:27:56.021485+0000","last_peered":"2024-03-06T07:27:56.021485+0000","last_clean":"2024-03-06T07:27:56.021485+0000","last_became_active":"2024-03-06T07:27:55.012951+0000","last_became_peered":"2024-03-06T07:27:55.012951+0000","last_unstale":"2024-03-06T07:27:56.021485+0000","last_undegraded":"2024-03-06T07:27:56.021485+0000","last_fullsized":"2024-03-06T07:27:56.021485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514798+0000","last_change":"2024-03-06T07:27:54.019750+0000","last_active":"2024-03-06T07:27:56.514798+0000","last_peered":"2024-03-06T07:27:56.514798+0000","last_clean":"2024-03-06T07:27:56.514798+0000","last_became_active":"2024-03-06T07:27:54.019443+0000","last_became_peered":"2024-03-06T07:27:54.019443+0000","last_unstale":"2024-03-06T07:27:56.514798+0000","last_undegraded":"2024-03-06T07:27:56.514798+0000","last_fullsized":"2024-03-06T07:27:56.514798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:43.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:05.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:05.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.399+0000 7f7fbfc00640 1 Processor -- start 2024-03-06T07:28:05.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.399+0000 7f7fbfc00640 1 -- start start 2024-03-06T07:28:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fbfc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fbfc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7fb815f470 0x7f7fb80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fbfc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb80cea60 con 0x7f7fb80d0df0 2024-03-06T07:28:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fbfc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb80d1260 con 0x7f7fb815f470 2024-03-06T07:28:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fbfc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80d2f60 con 0x7f7fb815e0d0 2024-03-06T07:28:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fafe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7fb815f470 0x7f7fb80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fafe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7fb815f470 0x7f7fb80d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56186/0 (socket says 172.21.15.114:56186) 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fafe00640 1 -- 172.21.15.114:0/455857105 learned_addr learned my addr 172.21.15.114:0/455857105 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.401+0000 7f7fbd200640 1 --2- 172.21.15.114:0/455857105 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.402+0000 7f7fafe00640 1 -- 172.21.15.114:0/455857105 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 msgr2=0x7f7fb80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.402+0000 7f7fafe00640 1 --2- 172.21.15.114:0/455857105 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.402+0000 7f7fafe00640 1 -- 172.21.15.114:0/455857105 >> v1:172.21.15.177:6789/0 conn(0x7f7fb80d0df0 legacy=0x7f7fb8160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.402+0000 7f7fafe00640 1 -- 172.21.15.114:0/455857105 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb80d2350 con 0x7f7fb815f470 2024-03-06T07:28:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.402+0000 7f7fbd200640 1 --2- 172.21.15.114:0/455857105 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:05.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.403+0000 7f7fafe00640 1 --2- 172.21.15.114:0/455857105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb80d08b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009870 tx=0x7f7fa002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895b8e980e715860 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.403+0000 7f7fb7e00640 1 -- 172.21.15.114:0/455857105 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa002f840 con 0x7f7fb815f470 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.403+0000 7f7fb7e00640 1 -- 172.21.15.114:0/455857105 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa002f9e0 con 0x7f7fb815f470 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.403+0000 7f7fb7e00640 1 -- 172.21.15.114:0/455857105 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa002fdb0 con 0x7f7fb815f470 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.403+0000 7f7fbfc00640 1 -- 172.21.15.114:0/455857105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 msgr2=0x7f7fb80d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.403+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/455857105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb80d08b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009870 tx=0x7f7fa002ba20 comp rx=0 tx=0).stop 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.404+0000 7f7fbfc00640 1 -- 172.21.15.114:0/455857105 shutdown_connections 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.404+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/455857105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb80d08b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.404+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/455857105 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.404+0000 7f7fbfc00640 1 -- 172.21.15.114:0/455857105 >> 172.21.15.114:0/455857105 conn(0x7f7fb80d3ec0 msgr2=0x7f7fb80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.404+0000 7f7fbfc00640 1 -- 172.21.15.114:0/455857105 shutdown_connections 2024-03-06T07:28:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.404+0000 7f7fbfc00640 1 -- 172.21.15.114:0/455857105 wait complete. 2024-03-06T07:28:05.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.405+0000 7f7fbfc00640 1 Processor -- start 2024-03-06T07:28:05.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.406+0000 7f7fbfc00640 1 -- start start 2024-03-06T07:28:05.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.406+0000 7f7fbfc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb80d0df0 0x7f7fb80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.406+0000 7f7fbfc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.406+0000 7f7fbfc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb8151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.406+0000 7f7fbfc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80d2bf0 con 0x7f7fb815f470 2024-03-06T07:28:05.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.406+0000 7f7fbfc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80d2910 con 0x7f7fb80d0df0 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.406+0000 7f7fbfc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb815fb30 con 0x7f7fb815e0d0 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb8151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb8151f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56190/0 (socket says 172.21.15.114:56190) 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 -- 172.21.15.114:0/257581190 learned_addr learned my addr 172.21.15.114:0/257581190 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbd200640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb80d0df0 0x7f7fb80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fafe00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 -- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 msgr2=0x7f7fb80b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80b6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 -- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb80d0df0 msgr2=0x7f7fb80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb80d0df0 0x7f7fb80b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 -- 172.21.15.114:0/257581190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb80c7910 con 0x7f7fb815f470 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.407+0000 7f7fbdc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb8151f30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000c760 tx=0x7f7fb000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.408+0000 7f7fb5600640 1 -- 172.21.15.114:0/257581190 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0016020 con 0x7f7fb815f470 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.408+0000 7f7fb5600640 1 -- 172.21.15.114:0/257581190 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb0006d50 con 0x7f7fb815f470 2024-03-06T07:28:05.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.408+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb80c7bf0 con 0x7f7fb815f470 2024-03-06T07:28:05.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.408+0000 7f7fb5600640 1 -- 172.21.15.114:0/257581190 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0003bb0 con 0x7f7fb815f470 2024-03-06T07:28:05.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.408+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb80c81b0 con 0x7f7fb815f470 2024-03-06T07:28:05.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.409+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb804f330 con 0x7f7fb815f470 2024-03-06T07:28:05.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.410+0000 7f7fb5600640 1 -- 172.21.15.114:0/257581190 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fb00050d0 con 0x7f7fb815f470 2024-03-06T07:28:05.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.411+0000 7f7fb5600640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f9406b4d0 0x7f7f9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.411+0000 7f7fb5600640 1 -- 172.21.15.114:0/257581190 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fb0089e00 con 0x7f7fb815f470 2024-03-06T07:28:05.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.411+0000 7f7fbd200640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f9406b4d0 0x7f7f9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.412+0000 7f7fbd200640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f9406b4d0 0x7f7f9406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fb80b17b0 tx=0x7f7fa8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.415+0000 7f7fb5600640 1 -- 172.21.15.114:0/257581190 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fb0020020 con 0x7f7fb815f470 2024-03-06T07:28:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.592+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fb8068500 con 0x7f7f9406b4d0 2024-03-06T07:28:05.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.610+0000 7f7fb5600640 1 -- 172.21.15.114:0/257581190 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f7fa800e850 con 0x7f7f9406b4d0 2024-03-06T07:28:05.616 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:05.617 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f9406b4d0 msgr2=0x7f7f9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f9406b4d0 0x7f7f9406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fb80b17b0 tx=0x7f7fa8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 msgr2=0x7f7fb8151f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb8151f30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000c760 tx=0x7f7fb000cb60 comp rx=0 tx=0).stop 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 shutdown_connections 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f9406b4d0 0x7f7f9406d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fb815f470 0x7f7fb8151f30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7fb815e0d0 0x7f7fb80b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 --2- 172.21.15.114:0/257581190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb80d0df0 0x7f7fb80b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 >> 172.21.15.114:0/257581190 conn(0x7f7fb80d3ec0 msgr2=0x7f7fb80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.615+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 shutdown_connections 2024-03-06T07:28:05.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.616+0000 7f7fbfc00640 1 -- 172.21.15.114:0/257581190 wait complete. 2024-03-06T07:28:05.634 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:28:05.102540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002964"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517399+0000","last_change":"2024-03-06T07:27:55.014476+0000","last_active":"2024-03-06T07:27:56.517399+0000","last_peered":"2024-03-06T07:27:56.517399+0000","last_clean":"2024-03-06T07:27:56.517399+0000","last_became_active":"2024-03-06T07:27:55.014169+0000","last_became_peered":"2024-03-06T07:27:55.014169+0000","last_unstale":"2024-03-06T07:27:56.517399+0000","last_undegraded":"2024-03-06T07:27:56.517399+0000","last_fullsized":"2024-03-06T07:27:56.517399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:20.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020046+0000","last_change":"2024-03-06T07:27:55.025640+0000","last_active":"2024-03-06T07:27:56.020046+0000","last_peered":"2024-03-06T07:27:56.020046+0000","last_clean":"2024-03-06T07:27:56.020046+0000","last_became_active":"2024-03-06T07:27:55.025215+0000","last_became_peered":"2024-03-06T07:27:55.025215+0000","last_unstale":"2024-03-06T07:27:56.020046+0000","last_undegraded":"2024-03-06T07:27:56.020046+0000","last_fullsized":"2024-03-06T07:27:56.020046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514320+0000","last_change":"2024-03-06T07:27:55.015505+0000","last_active":"2024-03-06T07:27:56.514320+0000","last_peered":"2024-03-06T07:27:56.514320+0000","last_clean":"2024-03-06T07:27:56.514320+0000","last_became_active":"2024-03-06T07:27:55.015199+0000","last_became_peered":"2024-03-06T07:27:55.015199+0000","last_unstale":"2024-03-06T07:27:56.514320+0000","last_undegraded":"2024-03-06T07:27:56.514320+0000","last_fullsized":"2024-03-06T07:27:56.514320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518415+0000","last_change":"2024-03-06T07:27:55.033897+0000","last_active":"2024-03-06T07:27:56.518415+0000","last_peered":"2024-03-06T07:27:56.518415+0000","last_clean":"2024-03-06T07:27:56.518415+0000","last_became_active":"2024-03-06T07:27:55.032987+0000","last_became_peered":"2024-03-06T07:27:55.032987+0000","last_unstale":"2024-03-06T07:27:56.518415+0000","last_undegraded":"2024-03-06T07:27:56.518415+0000","last_fullsized":"2024-03-06T07:27:56.518415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515678+0000","last_change":"2024-03-06T07:27:55.014597+0000","last_active":"2024-03-06T07:27:56.515678+0000","last_peered":"2024-03-06T07:27:56.515678+0000","last_clean":"2024-03-06T07:27:56.515678+0000","last_became_active":"2024-03-06T07:27:55.014206+0000","last_became_peered":"2024-03-06T07:27:55.014206+0000","last_unstale":"2024-03-06T07:27:56.515678+0000","last_undegraded":"2024-03-06T07:27:56.515678+0000","last_fullsized":"2024-03-06T07:27:56.515678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:14.645947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516836+0000","last_change":"2024-03-06T07:27:55.028575+0000","last_active":"2024-03-06T07:27:56.516836+0000","last_peered":"2024-03-06T07:27:56.516836+0000","last_clean":"2024-03-06T07:27:56.516836+0000","last_became_active":"2024-03-06T07:27:55.027801+0000","last_became_peered":"2024-03-06T07:27:55.027801+0000","last_unstale":"2024-03-06T07:27:56.516836+0000","last_undegraded":"2024-03-06T07:27:56.516836+0000","last_fullsized":"2024-03-06T07:27:56.516836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514995+0000","last_change":"2024-03-06T07:27:55.028417+0000","last_active":"2024-03-06T07:27:56.514995+0000","last_peered":"2024-03-06T07:27:56.514995+0000","last_clean":"2024-03-06T07:27:56.514995+0000","last_became_active":"2024-03-06T07:27:55.028108+0000","last_became_peered":"2024-03-06T07:27:55.028108+0000","last_unstale":"2024-03-06T07:27:56.514995+0000","last_undegraded":"2024-03-06T07:27:56.514995+0000","last_fullsized":"2024-03-06T07:27:56.514995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021025+0000","last_change":"2024-03-06T07:27:55.024045+0000","last_active":"2024-03-06T07:27:56.021025+0000","last_peered":"2024-03-06T07:27:56.021025+0000","last_clean":"2024-03-06T07:27:56.021025+0000","last_became_active":"2024-03-06T07:27:55.023528+0000","last_became_peered":"2024-03-06T07:27:55.023528+0000","last_unstale":"2024-03-06T07:27:56.021025+0000","last_undegraded":"2024-03-06T07:27:56.021025+0000","last_fullsized":"2024-03-06T07:27:56.021025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517632+0000","last_change":"2024-03-06T07:27:55.021303+0000","last_active":"2024-03-06T07:27:56.517632+0000","last_peered":"2024-03-06T07:27:56.517632+0000","last_clean":"2024-03-06T07:27:56.517632+0000","last_became_active":"2024-03-06T07:27:55.020953+0000","last_became_peered":"2024-03-06T07:27:55.020953+0000","last_unstale":"2024-03-06T07:27:56.517632+0000","last_undegraded":"2024-03-06T07:27:56.517632+0000","last_fullsized":"2024-03-06T07:27:56.517632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:30.042722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027586+0000","last_change":"2024-03-06T07:27:55.019249+0000","last_active":"2024-03-06T07:27:56.027586+0000","last_peered":"2024-03-06T07:27:56.027586+0000","last_clean":"2024-03-06T07:27:56.027586+0000","last_became_active":"2024-03-06T07:27:55.018947+0000","last_became_peered":"2024-03-06T07:27:55.018947+0000","last_unstale":"2024-03-06T07:27:56.027586+0000","last_undegraded":"2024-03-06T07:27:56.027586+0000","last_fullsized":"2024-03-06T07:27:56.027586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027423+0000","last_change":"2024-03-06T07:27:55.023288+0000","last_active":"2024-03-06T07:27:56.027423+0000","last_peered":"2024-03-06T07:27:56.027423+0000","last_clean":"2024-03-06T07:27:56.027423+0000","last_became_active":"2024-03-06T07:27:55.023020+0000","last_became_peered":"2024-03-06T07:27:55.023020+0000","last_unstale":"2024-03-06T07:27:56.027423+0000","last_undegraded":"2024-03-06T07:27:56.027423+0000","last_fullsized":"2024-03-06T07:27:56.027423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029290+0000","last_change":"2024-03-06T07:27:55.014888+0000","last_active":"2024-03-06T07:27:56.029290+0000","last_peered":"2024-03-06T07:27:56.029290+0000","last_clean":"2024-03-06T07:27:56.029290+0000","last_became_active":"2024-03-06T07:27:55.014394+0000","last_became_peered":"2024-03-06T07:27:55.014394+0000","last_unstale":"2024-03-06T07:27:56.029290+0000","last_undegraded":"2024-03-06T07:27:56.029290+0000","last_fullsized":"2024-03-06T07:27:56.029290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:58.566589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021843+0000","last_change":"2024-03-06T07:27:55.014936+0000","last_active":"2024-03-06T07:27:56.021843+0000","last_peered":"2024-03-06T07:27:56.021843+0000","last_clean":"2024-03-06T07:27:56.021843+0000","last_became_active":"2024-03-06T07:27:55.012277+0000","last_became_peered":"2024-03-06T07:27:55.012277+0000","last_unstale":"2024-03-06T07:27:56.021843+0000","last_undegraded":"2024-03-06T07:27:56.021843+0000","last_fullsized":"2024-03-06T07:27:56.021843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.012205+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.011928+0000","last_became_peered":"2024-03-06T07:27:55.011928+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019838+0000","last_change":"2024-03-06T07:27:55.030004+0000","last_active":"2024-03-06T07:27:56.019838+0000","last_peered":"2024-03-06T07:27:56.019838+0000","last_clean":"2024-03-06T07:27:56.019838+0000","last_became_active":"2024-03-06T07:27:55.029632+0000","last_became_peered":"2024-03-06T07:27:55.029632+0000","last_unstale":"2024-03-06T07:27:56.019838+0000","last_undegraded":"2024-03-06T07:27:56.019838+0000","last_fullsized":"2024-03-06T07:27:56.019838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515777+0000","last_change":"2024-03-06T07:27:55.028212+0000","last_active":"2024-03-06T07:27:56.515777+0000","last_peered":"2024-03-06T07:27:56.515777+0000","last_clean":"2024-03-06T07:27:56.515777+0000","last_became_active":"2024-03-06T07:27:55.027899+0000","last_became_peered":"2024-03-06T07:27:55.027899+0000","last_unstale":"2024-03-06T07:27:56.515777+0000","last_undegraded":"2024-03-06T07:27:56.515777+0000","last_fullsized":"2024-03-06T07:27:56.515777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519519+0000","last_change":"2024-03-06T07:27:55.024791+0000","last_active":"2024-03-06T07:27:56.519519+0000","last_peered":"2024-03-06T07:27:56.519519+0000","last_clean":"2024-03-06T07:27:56.519519+0000","last_became_active":"2024-03-06T07:27:55.023356+0000","last_became_peered":"2024-03-06T07:27:55.023356+0000","last_unstale":"2024-03-06T07:27:56.519519+0000","last_undegraded":"2024-03-06T07:27:56.519519+0000","last_fullsized":"2024-03-06T07:27:56.519519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514884+0000","last_change":"2024-03-06T07:27:54.994313+0000","last_active":"2024-03-06T07:27:56.514884+0000","last_peered":"2024-03-06T07:27:56.514884+0000","last_clean":"2024-03-06T07:27:56.514884+0000","last_became_active":"2024-03-06T07:27:54.993537+0000","last_became_peered":"2024-03-06T07:27:54.993537+0000","last_unstale":"2024-03-06T07:27:56.514884+0000","last_undegraded":"2024-03-06T07:27:56.514884+0000","last_fullsized":"2024-03-06T07:27:56.514884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:34.222639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516724+0000","last_change":"2024-03-06T07:27:55.016027+0000","last_active":"2024-03-06T07:27:56.516724+0000","last_peered":"2024-03-06T07:27:56.516724+0000","last_clean":"2024-03-06T07:27:56.516724+0000","last_became_active":"2024-03-06T07:27:55.015462+0000","last_became_peered":"2024-03-06T07:27:55.015462+0000","last_unstale":"2024-03-06T07:27:56.516724+0000","last_undegraded":"2024-03-06T07:27:56.516724+0000","last_fullsized":"2024-03-06T07:27:56.516724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027150+0000","last_change":"2024-03-06T07:27:55.015671+0000","last_active":"2024-03-06T07:27:56.027150+0000","last_peered":"2024-03-06T07:27:56.027150+0000","last_clean":"2024-03-06T07:27:56.027150+0000","last_became_active":"2024-03-06T07:27:55.015352+0000","last_became_peered":"2024-03-06T07:27:55.015352+0000","last_unstale":"2024-03-06T07:27:56.027150+0000","last_undegraded":"2024-03-06T07:27:56.027150+0000","last_fullsized":"2024-03-06T07:27:56.027150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513815+0000","last_change":"2024-03-06T07:27:55.026752+0000","last_active":"2024-03-06T07:27:56.513815+0000","last_peered":"2024-03-06T07:27:56.513815+0000","last_clean":"2024-03-06T07:27:56.513815+0000","last_became_active":"2024-03-06T07:27:55.026382+0000","last_became_peered":"2024-03-06T07:27:55.026382+0000","last_unstale":"2024-03-06T07:27:56.513815+0000","last_undegraded":"2024-03-06T07:27:56.513815+0000","last_fullsized":"2024-03-06T07:27:56.513815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517269+0000","last_change":"2024-03-06T07:27:55.024122+0000","last_active":"2024-03-06T07:27:56.517269+0000","last_peered":"2024-03-06T07:27:56.517269+0000","last_clean":"2024-03-06T07:27:56.517269+0000","last_became_active":"2024-03-06T07:27:55.023796+0000","last_became_peered":"2024-03-06T07:27:55.023796+0000","last_unstale":"2024-03-06T07:27:56.517269+0000","last_undegraded":"2024-03-06T07:27:56.517269+0000","last_fullsized":"2024-03-06T07:27:56.517269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021097+0000","last_change":"2024-03-06T07:27:55.020822+0000","last_active":"2024-03-06T07:27:56.021097+0000","last_peered":"2024-03-06T07:27:56.021097+0000","last_clean":"2024-03-06T07:27:56.021097+0000","last_became_active":"2024-03-06T07:27:55.020406+0000","last_became_peered":"2024-03-06T07:27:55.020406+0000","last_unstale":"2024-03-06T07:27:56.021097+0000","last_undegraded":"2024-03-06T07:27:56.021097+0000","last_fullsized":"2024-03-06T07:27:56.021097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516215+0000","last_change":"2024-03-06T07:27:55.016257+0000","last_active":"2024-03-06T07:27:56.516215+0000","last_peered":"2024-03-06T07:27:56.516215+0000","last_clean":"2024-03-06T07:27:56.516215+0000","last_became_active":"2024-03-06T07:27:55.013454+0000","last_became_peered":"2024-03-06T07:27:55.013454+0000","last_unstale":"2024-03-06T07:27:56.516215+0000","last_undegraded":"2024-03-06T07:27:56.516215+0000","last_fullsized":"2024-03-06T07:27:56.516215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021692+0000","last_change":"2024-03-06T07:27:55.014609+0000","last_active":"2024-03-06T07:27:56.021692+0000","last_peered":"2024-03-06T07:27:56.021692+0000","last_clean":"2024-03-06T07:27:56.021692+0000","last_became_active":"2024-03-06T07:27:55.014270+0000","last_became_peered":"2024-03-06T07:27:55.014270+0000","last_unstale":"2024-03-06T07:27:56.021692+0000","last_undegraded":"2024-03-06T07:27:56.021692+0000","last_fullsized":"2024-03-06T07:27:56.021692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027608+0000","last_change":"2024-03-06T07:27:55.012622+0000","last_active":"2024-03-06T07:27:56.027608+0000","last_peered":"2024-03-06T07:27:56.027608+0000","last_clean":"2024-03-06T07:27:56.027608+0000","last_became_active":"2024-03-06T07:27:55.012268+0000","last_became_peered":"2024-03-06T07:27:55.012268+0000","last_unstale":"2024-03-06T07:27:56.027608+0000","last_undegraded":"2024-03-06T07:27:56.027608+0000","last_fullsized":"2024-03-06T07:27:56.027608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019641+0000","last_change":"2024-03-06T07:27:55.025182+0000","last_active":"2024-03-06T07:27:56.019641+0000","last_peered":"2024-03-06T07:27:56.019641+0000","last_clean":"2024-03-06T07:27:56.019641+0000","last_became_active":"2024-03-06T07:27:55.024874+0000","last_became_peered":"2024-03-06T07:27:55.024874+0000","last_unstale":"2024-03-06T07:27:56.019641+0000","last_undegraded":"2024-03-06T07:27:56.019641+0000","last_fullsized":"2024-03-06T07:27:56.019641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.022062+0000","last_change":"2024-03-06T07:27:55.031727+0000","last_active":"2024-03-06T07:27:56.022062+0000","last_peered":"2024-03-06T07:27:56.022062+0000","last_clean":"2024-03-06T07:27:56.022062+0000","last_became_active":"2024-03-06T07:27:55.029328+0000","last_became_peered":"2024-03-06T07:27:55.029328+0000","last_unstale":"2024-03-06T07:27:56.022062+0000","last_undegraded":"2024-03-06T07:27:56.022062+0000","last_fullsized":"2024-03-06T07:27:56.022062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513894+0000","last_change":"2024-03-06T07:27:55.011251+0000","last_active":"2024-03-06T07:27:56.513894+0000","last_peered":"2024-03-06T07:27:56.513894+0000","last_clean":"2024-03-06T07:27:56.513894+0000","last_became_active":"2024-03-06T07:27:55.010892+0000","last_became_peered":"2024-03-06T07:27:55.010892+0000","last_unstale":"2024-03-06T07:27:56.513894+0000","last_undegraded":"2024-03-06T07:27:56.513894+0000","last_fullsized":"2024-03-06T07:27:56.513894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517664+0000","last_change":"2024-03-06T07:27:55.036627+0000","last_active":"2024-03-06T07:27:56.517664+0000","last_peered":"2024-03-06T07:27:56.517664+0000","last_clean":"2024-03-06T07:27:56.517664+0000","last_became_active":"2024-03-06T07:27:55.036245+0000","last_became_peered":"2024-03-06T07:27:55.036245+0000","last_unstale":"2024-03-06T07:27:56.517664+0000","last_undegraded":"2024-03-06T07:27:56.517664+0000","last_fullsized":"2024-03-06T07:27:56.517664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514454+0000","last_change":"2024-03-06T07:27:55.018254+0000","last_active":"2024-03-06T07:27:56.514454+0000","last_peered":"2024-03-06T07:27:56.514454+0000","last_clean":"2024-03-06T07:27:56.514454+0000","last_became_active":"2024-03-06T07:27:55.017832+0000","last_became_peered":"2024-03-06T07:27:55.017832+0000","last_unstale":"2024-03-06T07:27:56.514454+0000","last_undegraded":"2024-03-06T07:27:56.514454+0000","last_fullsized":"2024-03-06T07:27:56.514454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:11.749502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026113+0000","last_change":"2024-03-06T07:27:55.024465+0000","last_active":"2024-03-06T07:27:56.026113+0000","last_peered":"2024-03-06T07:27:56.026113+0000","last_clean":"2024-03-06T07:27:56.026113+0000","last_became_active":"2024-03-06T07:27:55.024173+0000","last_became_peered":"2024-03-06T07:27:55.024173+0000","last_unstale":"2024-03-06T07:27:56.026113+0000","last_undegraded":"2024-03-06T07:27:56.026113+0000","last_fullsized":"2024-03-06T07:27:56.026113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028167+0000","last_change":"2024-03-06T07:27:55.013597+0000","last_active":"2024-03-06T07:27:56.028167+0000","last_peered":"2024-03-06T07:27:56.028167+0000","last_clean":"2024-03-06T07:27:56.028167+0000","last_became_active":"2024-03-06T07:27:55.013199+0000","last_became_peered":"2024-03-06T07:27:55.013199+0000","last_unstale":"2024-03-06T07:27:56.028167+0000","last_undegraded":"2024-03-06T07:27:56.028167+0000","last_fullsized":"2024-03-06T07:27:56.028167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:13.574066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021899+0000","last_change":"2024-03-06T07:27:54.014875+0000","last_active":"2024-03-06T07:27:56.021899+0000","last_peered":"2024-03-06T07:27:56.021899+0000","last_clean":"2024-03-06T07:27:56.021899+0000","last_became_active":"2024-03-06T07:27:54.014526+0000","last_became_peered":"2024-03-06T07:27:54.014526+0000","last_unstale":"2024-03-06T07:27:56.021899+0000","last_undegraded":"2024-03-06T07:27:56.021899+0000","last_fullsized":"2024-03-06T07:27:56.021899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514018+0000","last_change":"2024-03-06T07:27:55.017432+0000","last_active":"2024-03-06T07:27:56.514018+0000","last_peered":"2024-03-06T07:27:56.514018+0000","last_clean":"2024-03-06T07:27:56.514018+0000","last_became_active":"2024-03-06T07:27:55.015135+0000","last_became_peered":"2024-03-06T07:27:55.015135+0000","last_unstale":"2024-03-06T07:27:56.514018+0000","last_undegraded":"2024-03-06T07:27:56.514018+0000","last_fullsized":"2024-03-06T07:27:56.514018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026039+0000","last_change":"2024-03-06T07:27:54.023420+0000","last_active":"2024-03-06T07:27:56.026039+0000","last_peered":"2024-03-06T07:27:56.026039+0000","last_clean":"2024-03-06T07:27:56.026039+0000","last_became_active":"2024-03-06T07:27:54.022665+0000","last_became_peered":"2024-03-06T07:27:54.022665+0000","last_unstale":"2024-03-06T07:27:56.026039+0000","last_undegraded":"2024-03-06T07:27:56.026039+0000","last_fullsized":"2024-03-06T07:27:56.026039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021422+0000","last_change":"2024-03-06T07:27:54.015366+0000","last_active":"2024-03-06T07:27:56.021422+0000","last_peered":"2024-03-06T07:27:56.021422+0000","last_clean":"2024-03-06T07:27:56.021422+0000","last_became_active":"2024-03-06T07:27:54.015155+0000","last_became_peered":"2024-03-06T07:27:54.015155+0000","last_unstale":"2024-03-06T07:27:56.021422+0000","last_undegraded":"2024-03-06T07:27:56.021422+0000","last_fullsized":"2024-03-06T07:27:56.021422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:04.758569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021171+0000","last_change":"2024-03-06T07:27:55.029914+0000","last_active":"2024-03-06T07:27:56.021171+0000","last_peered":"2024-03-06T07:27:56.021171+0000","last_clean":"2024-03-06T07:27:56.021171+0000","last_became_active":"2024-03-06T07:27:55.029599+0000","last_became_peered":"2024-03-06T07:27:55.029599+0000","last_unstale":"2024-03-06T07:27:56.021171+0000","last_undegraded":"2024-03-06T07:27:56.021171+0000","last_fullsized":"2024-03-06T07:27:56.021171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516973+0000","last_change":"2024-03-06T07:27:54.023284+0000","last_active":"2024-03-06T07:27:56.516973+0000","last_peered":"2024-03-06T07:27:56.516973+0000","last_clean":"2024-03-06T07:27:56.516973+0000","last_became_active":"2024-03-06T07:27:54.022909+0000","last_became_peered":"2024-03-06T07:27:54.022909+0000","last_unstale":"2024-03-06T07:27:56.516973+0000","last_undegraded":"2024-03-06T07:27:56.516973+0000","last_fullsized":"2024-03-06T07:27:56.516973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.016848+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.016590+0000","last_became_peered":"2024-03-06T07:27:55.016590+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028999+0000","last_change":"2024-03-06T07:27:54.025189+0000","last_active":"2024-03-06T07:27:56.028999+0000","last_peered":"2024-03-06T07:27:56.028999+0000","last_clean":"2024-03-06T07:27:56.028999+0000","last_became_active":"2024-03-06T07:27:54.024875+0000","last_became_peered":"2024-03-06T07:27:54.024875+0000","last_unstale":"2024-03-06T07:27:56.028999+0000","last_undegraded":"2024-03-06T07:27:56.028999+0000","last_fullsized":"2024-03-06T07:27:56.028999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519254+0000","last_change":"2024-03-06T07:27:55.018921+0000","last_active":"2024-03-06T07:27:56.519254+0000","last_peered":"2024-03-06T07:27:56.519254+0000","last_clean":"2024-03-06T07:27:56.519254+0000","last_became_active":"2024-03-06T07:27:55.018597+0000","last_became_peered":"2024-03-06T07:27:55.018597+0000","last_unstale":"2024-03-06T07:27:56.519254+0000","last_undegraded":"2024-03-06T07:27:56.519254+0000","last_fullsized":"2024-03-06T07:27:56.519254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.873362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.519090+0000","last_change":"2024-03-06T07:27:54.025386+0000","last_active":"2024-03-06T07:27:56.519090+0000","last_peered":"2024-03-06T07:27:56.519090+0000","last_clean":"2024-03-06T07:27:56.519090+0000","last_became_active":"2024-03-06T07:27:54.020646+0000","last_became_peered":"2024-03-06T07:27:54.020646+0000","last_unstale":"2024-03-06T07:27:56.519090+0000","last_undegraded":"2024-03-06T07:27:56.519090+0000","last_fullsized":"2024-03-06T07:27:56.519090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514919+0000","last_change":"2024-03-06T07:27:55.022924+0000","last_active":"2024-03-06T07:27:56.514919+0000","last_peered":"2024-03-06T07:27:56.514919+0000","last_clean":"2024-03-06T07:27:56.514919+0000","last_became_active":"2024-03-06T07:27:55.022605+0000","last_became_peered":"2024-03-06T07:27:55.022605+0000","last_unstale":"2024-03-06T07:27:56.514919+0000","last_undegraded":"2024-03-06T07:27:56.514919+0000","last_fullsized":"2024-03-06T07:27:56.514919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019719+0000","last_change":"2024-03-06T07:27:54.012324+0000","last_active":"2024-03-06T07:27:56.019719+0000","last_peered":"2024-03-06T07:27:56.019719+0000","last_clean":"2024-03-06T07:27:56.019719+0000","last_became_active":"2024-03-06T07:27:54.012025+0000","last_became_peered":"2024-03-06T07:27:54.012025+0000","last_unstale":"2024-03-06T07:27:56.019719+0000","last_undegraded":"2024-03-06T07:27:56.019719+0000","last_fullsized":"2024-03-06T07:27:56.019719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029716+0000","last_change":"2024-03-06T07:27:55.028572+0000","last_active":"2024-03-06T07:27:56.029716+0000","last_peered":"2024-03-06T07:27:56.029716+0000","last_clean":"2024-03-06T07:27:56.029716+0000","last_became_active":"2024-03-06T07:27:55.026783+0000","last_became_peered":"2024-03-06T07:27:55.026783+0000","last_unstale":"2024-03-06T07:27:56.029716+0000","last_undegraded":"2024-03-06T07:27:56.029716+0000","last_fullsized":"2024-03-06T07:27:56.029716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:43.960764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.027082+0000","last_change":"2024-03-06T07:27:54.011288+0000","last_active":"2024-03-06T07:27:56.027082+0000","last_peered":"2024-03-06T07:27:56.027082+0000","last_clean":"2024-03-06T07:27:56.027082+0000","last_became_active":"2024-03-06T07:27:54.010900+0000","last_became_peered":"2024-03-06T07:27:54.010900+0000","last_unstale":"2024-03-06T07:27:56.027082+0000","last_undegraded":"2024-03-06T07:27:56.027082+0000","last_fullsized":"2024-03-06T07:27:56.027082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515920+0000","last_change":"2024-03-06T07:27:55.035258+0000","last_active":"2024-03-06T07:27:56.515920+0000","last_peered":"2024-03-06T07:27:56.515920+0000","last_clean":"2024-03-06T07:27:56.515920+0000","last_became_active":"2024-03-06T07:27:55.034985+0000","last_became_peered":"2024-03-06T07:27:55.034985+0000","last_unstale":"2024-03-06T07:27:56.515920+0000","last_undegraded":"2024-03-06T07:27:56.515920+0000","last_fullsized":"2024-03-06T07:27:56.515920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513956+0000","last_change":"2024-03-06T07:27:54.019343+0000","last_active":"2024-03-06T07:27:56.513956+0000","last_peered":"2024-03-06T07:27:56.513956+0000","last_clean":"2024-03-06T07:27:56.513956+0000","last_became_active":"2024-03-06T07:27:54.019083+0000","last_became_peered":"2024-03-06T07:27:54.019083+0000","last_unstale":"2024-03-06T07:27:56.513956+0000","last_undegraded":"2024-03-06T07:27:56.513956+0000","last_fullsized":"2024-03-06T07:27:56.513956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:12.617582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513603+0000","last_change":"2024-03-06T07:27:55.028252+0000","last_active":"2024-03-06T07:27:56.513603+0000","last_peered":"2024-03-06T07:27:56.513603+0000","last_clean":"2024-03-06T07:27:56.513603+0000","last_became_active":"2024-03-06T07:27:55.027919+0000","last_became_peered":"2024-03-06T07:27:55.027919+0000","last_unstale":"2024-03-06T07:27:56.513603+0000","last_undegraded":"2024-03-06T07:27:56.513603+0000","last_fullsized":"2024-03-06T07:27:56.513603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:57.800588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517776+0000","last_change":"2024-03-06T07:27:54.015808+0000","last_active":"2024-03-06T07:27:56.517776+0000","last_peered":"2024-03-06T07:27:56.517776+0000","last_clean":"2024-03-06T07:27:56.517776+0000","last_became_active":"2024-03-06T07:27:54.015279+0000","last_became_peered":"2024-03-06T07:27:54.015279+0000","last_unstale":"2024-03-06T07:27:56.517776+0000","last_undegraded":"2024-03-06T07:27:56.517776+0000","last_fullsized":"2024-03-06T07:27:56.517776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516046+0000","last_change":"2024-03-06T07:27:55.015432+0000","last_active":"2024-03-06T07:27:56.516046+0000","last_peered":"2024-03-06T07:27:56.516046+0000","last_clean":"2024-03-06T07:27:56.516046+0000","last_became_active":"2024-03-06T07:27:55.015132+0000","last_became_peered":"2024-03-06T07:27:55.015132+0000","last_unstale":"2024-03-06T07:27:56.516046+0000","last_undegraded":"2024-03-06T07:27:56.516046+0000","last_fullsized":"2024-03-06T07:27:56.516046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026103+0000","last_change":"2024-03-06T07:27:54.020993+0000","last_active":"2024-03-06T07:27:56.026103+0000","last_peered":"2024-03-06T07:27:56.026103+0000","last_clean":"2024-03-06T07:27:56.026103+0000","last_became_active":"2024-03-06T07:27:54.020618+0000","last_became_peered":"2024-03-06T07:27:54.020618+0000","last_unstale":"2024-03-06T07:27:56.026103+0000","last_undegraded":"2024-03-06T07:27:56.026103+0000","last_fullsized":"2024-03-06T07:27:56.026103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026185+0000","last_change":"2024-03-06T07:27:55.022089+0000","last_active":"2024-03-06T07:27:56.026185+0000","last_peered":"2024-03-06T07:27:56.026185+0000","last_clean":"2024-03-06T07:27:56.026185+0000","last_became_active":"2024-03-06T07:27:55.021113+0000","last_became_peered":"2024-03-06T07:27:55.021113+0000","last_unstale":"2024-03-06T07:27:56.026185+0000","last_undegraded":"2024-03-06T07:27:56.026185+0000","last_fullsized":"2024-03-06T07:27:56.026185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:41.343748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514788+0000","last_change":"2024-03-06T07:27:54.002389+0000","last_active":"2024-03-06T07:27:56.514788+0000","last_peered":"2024-03-06T07:27:56.514788+0000","last_clean":"2024-03-06T07:27:56.514788+0000","last_became_active":"2024-03-06T07:27:54.001928+0000","last_became_peered":"2024-03-06T07:27:54.001928+0000","last_unstale":"2024-03-06T07:27:56.514788+0000","last_undegraded":"2024-03-06T07:27:56.514788+0000","last_fullsized":"2024-03-06T07:27:56.514788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:35.659909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515180+0000","last_change":"2024-03-06T07:27:55.018703+0000","last_active":"2024-03-06T07:27:56.515180+0000","last_peered":"2024-03-06T07:27:56.515180+0000","last_clean":"2024-03-06T07:27:56.515180+0000","last_became_active":"2024-03-06T07:27:55.018315+0000","last_became_peered":"2024-03-06T07:27:55.018315+0000","last_unstale":"2024-03-06T07:27:56.515180+0000","last_undegraded":"2024-03-06T07:27:56.515180+0000","last_fullsized":"2024-03-06T07:27:56.515180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:39.919518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029303+0000","last_change":"2024-03-06T07:27:54.015632+0000","last_active":"2024-03-06T07:27:56.029303+0000","last_peered":"2024-03-06T07:27:56.029303+0000","last_clean":"2024-03-06T07:27:56.029303+0000","last_became_active":"2024-03-06T07:27:54.014810+0000","last_became_peered":"2024-03-06T07:27:54.014810+0000","last_unstale":"2024-03-06T07:27:56.029303+0000","last_undegraded":"2024-03-06T07:27:56.029303+0000","last_fullsized":"2024-03-06T07:27:56.029303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:09.001445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516849+0000","last_change":"2024-03-06T07:27:55.022324+0000","last_active":"2024-03-06T07:27:56.516849+0000","last_peered":"2024-03-06T07:27:56.516849+0000","last_clean":"2024-03-06T07:27:56.516849+0000","last_became_active":"2024-03-06T07:27:55.022013+0000","last_became_peered":"2024-03-06T07:27:55.022013+0000","last_unstale":"2024-03-06T07:27:56.516849+0000","last_undegraded":"2024-03-06T07:27:56.516849+0000","last_fullsized":"2024-03-06T07:27:56.516849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:14.420738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515058+0000","last_change":"2024-03-06T07:27:54.005847+0000","last_active":"2024-03-06T07:27:56.515058+0000","last_peered":"2024-03-06T07:27:56.515058+0000","last_clean":"2024-03-06T07:27:56.515058+0000","last_became_active":"2024-03-06T07:27:54.005460+0000","last_became_peered":"2024-03-06T07:27:54.005460+0000","last_unstale":"2024-03-06T07:27:56.515058+0000","last_undegraded":"2024-03-06T07:27:56.515058+0000","last_fullsized":"2024-03-06T07:27:56.515058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517605+0000","last_change":"2024-03-06T07:27:55.021338+0000","last_active":"2024-03-06T07:27:56.517605+0000","last_peered":"2024-03-06T07:27:56.517605+0000","last_clean":"2024-03-06T07:27:56.517605+0000","last_became_active":"2024-03-06T07:27:55.021017+0000","last_became_peered":"2024-03-06T07:27:55.021017+0000","last_unstale":"2024-03-06T07:27:56.517605+0000","last_undegraded":"2024-03-06T07:27:56.517605+0000","last_fullsized":"2024-03-06T07:27:56.517605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:47.579731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019961+0000","last_change":"2024-03-06T07:27:54.022289+0000","last_active":"2024-03-06T07:27:56.019961+0000","last_peered":"2024-03-06T07:27:56.019961+0000","last_clean":"2024-03-06T07:27:56.019961+0000","last_became_active":"2024-03-06T07:27:54.022098+0000","last_became_peered":"2024-03-06T07:27:54.022098+0000","last_unstale":"2024-03-06T07:27:56.019961+0000","last_undegraded":"2024-03-06T07:27:56.019961+0000","last_fullsized":"2024-03-06T07:27:56.019961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518836+0000","last_change":"2024-03-06T07:27:55.018543+0000","last_active":"2024-03-06T07:27:56.518836+0000","last_peered":"2024-03-06T07:27:56.518836+0000","last_clean":"2024-03-06T07:27:56.518836+0000","last_became_active":"2024-03-06T07:27:55.017787+0000","last_became_peered":"2024-03-06T07:27:55.017787+0000","last_unstale":"2024-03-06T07:27:56.518836+0000","last_undegraded":"2024-03-06T07:27:56.518836+0000","last_fullsized":"2024-03-06T07:27:56.518836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:08.402370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515131+0000","last_change":"2024-03-06T07:27:54.024731+0000","last_active":"2024-03-06T07:27:56.515131+0000","last_peered":"2024-03-06T07:27:56.515131+0000","last_clean":"2024-03-06T07:27:56.515131+0000","last_became_active":"2024-03-06T07:27:54.024389+0000","last_became_peered":"2024-03-06T07:27:54.024389+0000","last_unstale":"2024-03-06T07:27:56.515131+0000","last_undegraded":"2024-03-06T07:27:56.515131+0000","last_fullsized":"2024-03-06T07:27:56.515131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515569+0000","last_change":"2024-03-06T07:27:55.019440+0000","last_active":"2024-03-06T07:27:56.515569+0000","last_peered":"2024-03-06T07:27:56.515569+0000","last_clean":"2024-03-06T07:27:56.515569+0000","last_became_active":"2024-03-06T07:27:55.019018+0000","last_became_peered":"2024-03-06T07:27:55.019018+0000","last_unstale":"2024-03-06T07:27:56.515569+0000","last_undegraded":"2024-03-06T07:27:56.515569+0000","last_fullsized":"2024-03-06T07:27:56.515569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:52.354424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514828+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:27:56.514828+0000","last_peered":"2024-03-06T07:27:56.514828+0000","last_clean":"2024-03-06T07:27:56.514828+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:27:56.514828+0000","last_undegraded":"2024-03-06T07:27:56.514828+0000","last_fullsized":"2024-03-06T07:27:56.514828+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515546+0000","last_change":"2024-03-06T07:27:54.013591+0000","last_active":"2024-03-06T07:27:56.515546+0000","last_peered":"2024-03-06T07:27:56.515546+0000","last_clean":"2024-03-06T07:27:56.515546+0000","last_became_active":"2024-03-06T07:27:54.012154+0000","last_became_peered":"2024-03-06T07:27:54.012154+0000","last_unstale":"2024-03-06T07:27:56.515546+0000","last_undegraded":"2024-03-06T07:27:56.515546+0000","last_fullsized":"2024-03-06T07:27:56.515546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021505+0000","last_change":"2024-03-06T07:27:55.029404+0000","last_active":"2024-03-06T07:27:56.021505+0000","last_peered":"2024-03-06T07:27:56.021505+0000","last_clean":"2024-03-06T07:27:56.021505+0000","last_became_active":"2024-03-06T07:27:55.028890+0000","last_became_peered":"2024-03-06T07:27:55.028890+0000","last_unstale":"2024-03-06T07:27:56.021505+0000","last_undegraded":"2024-03-06T07:27:56.021505+0000","last_fullsized":"2024-03-06T07:27:56.021505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514470+0000","last_change":"2024-03-06T07:27:54.025195+0000","last_active":"2024-03-06T07:27:56.514470+0000","last_peered":"2024-03-06T07:27:56.514470+0000","last_clean":"2024-03-06T07:27:56.514470+0000","last_became_active":"2024-03-06T07:27:54.024942+0000","last_became_peered":"2024-03-06T07:27:54.024942+0000","last_unstale":"2024-03-06T07:27:56.514470+0000","last_undegraded":"2024-03-06T07:27:56.514470+0000","last_fullsized":"2024-03-06T07:27:56.514470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.513738+0000","last_change":"2024-03-06T07:27:55.022642+0000","last_active":"2024-03-06T07:27:56.513738+0000","last_peered":"2024-03-06T07:27:56.513738+0000","last_clean":"2024-03-06T07:27:56.513738+0000","last_became_active":"2024-03-06T07:27:55.021096+0000","last_became_peered":"2024-03-06T07:27:55.021096+0000","last_unstale":"2024-03-06T07:27:56.513738+0000","last_undegraded":"2024-03-06T07:27:56.513738+0000","last_fullsized":"2024-03-06T07:27:56.513738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:03.195450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514360+0000","last_change":"2024-03-06T07:27:54.011028+0000","last_active":"2024-03-06T07:27:56.514360+0000","last_peered":"2024-03-06T07:27:56.514360+0000","last_clean":"2024-03-06T07:27:56.514360+0000","last_became_active":"2024-03-06T07:27:54.010767+0000","last_became_peered":"2024-03-06T07:27:54.010767+0000","last_unstale":"2024-03-06T07:27:56.514360+0000","last_undegraded":"2024-03-06T07:27:56.514360+0000","last_fullsized":"2024-03-06T07:27:56.514360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:12.559122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019932+0000","last_change":"2024-03-06T07:27:55.014190+0000","last_active":"2024-03-06T07:27:56.019932+0000","last_peered":"2024-03-06T07:27:56.019932+0000","last_clean":"2024-03-06T07:27:56.019932+0000","last_became_active":"2024-03-06T07:27:55.013895+0000","last_became_peered":"2024-03-06T07:27:55.013895+0000","last_unstale":"2024-03-06T07:27:56.019932+0000","last_undegraded":"2024-03-06T07:27:56.019932+0000","last_fullsized":"2024-03-06T07:27:56.019932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517915+0000","last_change":"2024-03-06T07:27:54.015537+0000","last_active":"2024-03-06T07:27:56.517915+0000","last_peered":"2024-03-06T07:27:56.517915+0000","last_clean":"2024-03-06T07:27:56.517915+0000","last_became_active":"2024-03-06T07:27:54.014754+0000","last_became_peered":"2024-03-06T07:27:54.014754+0000","last_unstale":"2024-03-06T07:27:56.517915+0000","last_undegraded":"2024-03-06T07:27:56.517915+0000","last_fullsized":"2024-03-06T07:27:56.517915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.028699+0000","last_change":"2024-03-06T07:27:55.015839+0000","last_active":"2024-03-06T07:27:56.028699+0000","last_peered":"2024-03-06T07:27:56.028699+0000","last_clean":"2024-03-06T07:27:56.028699+0000","last_became_active":"2024-03-06T07:27:55.015468+0000","last_became_peered":"2024-03-06T07:27:55.015468+0000","last_unstale":"2024-03-06T07:27:56.028699+0000","last_undegraded":"2024-03-06T07:27:56.028699+0000","last_fullsized":"2024-03-06T07:27:56.028699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:41.571023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019749+0000","last_change":"2024-03-06T07:27:54.003778+0000","last_active":"2024-03-06T07:27:56.019749+0000","last_peered":"2024-03-06T07:27:56.019749+0000","last_clean":"2024-03-06T07:27:56.019749+0000","last_became_active":"2024-03-06T07:27:54.003482+0000","last_became_peered":"2024-03-06T07:27:54.003482+0000","last_unstale":"2024-03-06T07:27:56.019749+0000","last_undegraded":"2024-03-06T07:27:56.019749+0000","last_fullsized":"2024-03-06T07:27:56.019749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:42.945821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.517164+0000","last_change":"2024-03-06T07:27:55.036670+0000","last_active":"2024-03-06T07:27:56.517164+0000","last_peered":"2024-03-06T07:27:56.517164+0000","last_clean":"2024-03-06T07:27:56.517164+0000","last_became_active":"2024-03-06T07:27:55.036328+0000","last_became_peered":"2024-03-06T07:27:55.036328+0000","last_unstale":"2024-03-06T07:27:56.517164+0000","last_undegraded":"2024-03-06T07:27:56.517164+0000","last_fullsized":"2024-03-06T07:27:56.517164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516091+0000","last_change":"2024-03-06T07:27:54.022591+0000","last_active":"2024-03-06T07:27:56.516091+0000","last_peered":"2024-03-06T07:27:56.516091+0000","last_clean":"2024-03-06T07:27:56.516091+0000","last_became_active":"2024-03-06T07:27:54.022296+0000","last_became_peered":"2024-03-06T07:27:54.022296+0000","last_unstale":"2024-03-06T07:27:56.516091+0000","last_undegraded":"2024-03-06T07:27:56.516091+0000","last_fullsized":"2024-03-06T07:27:56.516091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:21.651372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019687+0000","last_change":"2024-03-06T07:27:55.010897+0000","last_active":"2024-03-06T07:27:56.019687+0000","last_peered":"2024-03-06T07:27:56.019687+0000","last_clean":"2024-03-06T07:27:56.019687+0000","last_became_active":"2024-03-06T07:27:55.005758+0000","last_became_peered":"2024-03-06T07:27:55.005758+0000","last_unstale":"2024-03-06T07:27:56.019687+0000","last_undegraded":"2024-03-06T07:27:56.019687+0000","last_fullsized":"2024-03-06T07:27:56.019687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.020051+0000","last_change":"2024-03-06T07:27:54.018547+0000","last_active":"2024-03-06T07:27:56.020051+0000","last_peered":"2024-03-06T07:27:56.020051+0000","last_clean":"2024-03-06T07:27:56.020051+0000","last_became_active":"2024-03-06T07:27:54.018215+0000","last_became_peered":"2024-03-06T07:27:54.018215+0000","last_unstale":"2024-03-06T07:27:56.020051+0000","last_undegraded":"2024-03-06T07:27:56.020051+0000","last_fullsized":"2024-03-06T07:27:56.020051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021795+0000","last_change":"2024-03-06T07:27:55.013861+0000","last_active":"2024-03-06T07:27:56.021795+0000","last_peered":"2024-03-06T07:27:56.021795+0000","last_clean":"2024-03-06T07:27:56.021795+0000","last_became_active":"2024-03-06T07:27:55.013498+0000","last_became_peered":"2024-03-06T07:27:55.013498+0000","last_unstale":"2024-03-06T07:27:56.021795+0000","last_undegraded":"2024-03-06T07:27:56.021795+0000","last_fullsized":"2024-03-06T07:27:56.021795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:21.901760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.029229+0000","last_change":"2024-03-06T07:27:54.015491+0000","last_active":"2024-03-06T07:27:56.029229+0000","last_peered":"2024-03-06T07:27:56.029229+0000","last_clean":"2024-03-06T07:27:56.029229+0000","last_became_active":"2024-03-06T07:27:54.015207+0000","last_became_peered":"2024-03-06T07:27:54.015207+0000","last_unstale":"2024-03-06T07:27:56.029229+0000","last_undegraded":"2024-03-06T07:27:56.029229+0000","last_fullsized":"2024-03-06T07:27:56.029229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:40.556080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514806+0000","last_change":"2024-03-06T07:27:55.021833+0000","last_active":"2024-03-06T07:27:56.514806+0000","last_peered":"2024-03-06T07:27:56.514806+0000","last_clean":"2024-03-06T07:27:56.514806+0000","last_became_active":"2024-03-06T07:27:55.021478+0000","last_became_peered":"2024-03-06T07:27:55.021478+0000","last_unstale":"2024-03-06T07:27:56.514806+0000","last_undegraded":"2024-03-06T07:27:56.514806+0000","last_fullsized":"2024-03-06T07:27:56.514806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:18.259622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514469+0000","last_change":"2024-03-06T07:27:54.024577+0000","last_active":"2024-03-06T07:27:56.514469+0000","last_peered":"2024-03-06T07:27:56.514469+0000","last_clean":"2024-03-06T07:27:56.514469+0000","last_became_active":"2024-03-06T07:27:54.023972+0000","last_became_peered":"2024-03-06T07:27:54.023972+0000","last_unstale":"2024-03-06T07:27:56.514469+0000","last_undegraded":"2024-03-06T07:27:56.514469+0000","last_fullsized":"2024-03-06T07:27:56.514469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.025293+0000","last_change":"2024-03-06T07:27:55.020868+0000","last_active":"2024-03-06T07:27:56.025293+0000","last_peered":"2024-03-06T07:27:56.025293+0000","last_clean":"2024-03-06T07:27:56.025293+0000","last_became_active":"2024-03-06T07:27:55.019789+0000","last_became_peered":"2024-03-06T07:27:55.019789+0000","last_unstale":"2024-03-06T07:27:56.025293+0000","last_undegraded":"2024-03-06T07:27:56.025293+0000","last_fullsized":"2024-03-06T07:27:56.025293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:54.669987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019875+0000","last_change":"2024-03-06T07:27:54.014058+0000","last_active":"2024-03-06T07:27:56.019875+0000","last_peered":"2024-03-06T07:27:56.019875+0000","last_clean":"2024-03-06T07:27:56.019875+0000","last_became_active":"2024-03-06T07:27:54.013803+0000","last_became_peered":"2024-03-06T07:27:54.013803+0000","last_unstale":"2024-03-06T07:27:56.019875+0000","last_undegraded":"2024-03-06T07:27:56.019875+0000","last_fullsized":"2024-03-06T07:27:56.019875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.655043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021185+0000","last_change":"2024-03-06T07:27:55.020235+0000","last_active":"2024-03-06T07:27:56.021185+0000","last_peered":"2024-03-06T07:27:56.021185+0000","last_clean":"2024-03-06T07:27:56.021185+0000","last_became_active":"2024-03-06T07:27:55.015366+0000","last_became_peered":"2024-03-06T07:27:55.015366+0000","last_unstale":"2024-03-06T07:27:56.021185+0000","last_undegraded":"2024-03-06T07:27:56.021185+0000","last_fullsized":"2024-03-06T07:27:56.021185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:55.796987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.019800+0000","last_change":"2024-03-06T07:27:54.005998+0000","last_active":"2024-03-06T07:27:56.019800+0000","last_peered":"2024-03-06T07:27:56.019800+0000","last_clean":"2024-03-06T07:27:56.019800+0000","last_became_active":"2024-03-06T07:27:54.005646+0000","last_became_peered":"2024-03-06T07:27:54.005646+0000","last_unstale":"2024-03-06T07:27:56.019800+0000","last_undegraded":"2024-03-06T07:27:56.019800+0000","last_fullsized":"2024-03-06T07:27:56.019800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:57.551645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021888+0000","last_change":"2024-03-06T07:27:55.005310+0000","last_active":"2024-03-06T07:27:56.021888+0000","last_peered":"2024-03-06T07:27:56.021888+0000","last_clean":"2024-03-06T07:27:56.021888+0000","last_became_active":"2024-03-06T07:27:55.004647+0000","last_became_peered":"2024-03-06T07:27:55.004647+0000","last_unstale":"2024-03-06T07:27:56.021888+0000","last_undegraded":"2024-03-06T07:27:56.021888+0000","last_fullsized":"2024-03-06T07:27:56.021888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:10.823553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515995+0000","last_change":"2024-03-06T07:27:54.027960+0000","last_active":"2024-03-06T07:27:56.515995+0000","last_peered":"2024-03-06T07:27:56.515995+0000","last_clean":"2024-03-06T07:27:56.515995+0000","last_became_active":"2024-03-06T07:27:54.027317+0000","last_became_peered":"2024-03-06T07:27:54.027317+0000","last_unstale":"2024-03-06T07:27:56.515995+0000","last_undegraded":"2024-03-06T07:27:56.515995+0000","last_fullsized":"2024-03-06T07:27:56.515995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:41.117856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.026502+0000","last_change":"2024-03-06T07:27:55.025331+0000","last_active":"2024-03-06T07:27:56.026502+0000","last_peered":"2024-03-06T07:27:56.026502+0000","last_clean":"2024-03-06T07:27:56.026502+0000","last_became_active":"2024-03-06T07:27:55.024981+0000","last_became_peered":"2024-03-06T07:27:55.024981+0000","last_unstale":"2024-03-06T07:27:56.026502+0000","last_undegraded":"2024-03-06T07:27:56.026502+0000","last_fullsized":"2024-03-06T07:27:56.026502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:35.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.516878+0000","last_change":"2024-03-06T07:27:55.023997+0000","last_active":"2024-03-06T07:27:56.516878+0000","last_peered":"2024-03-06T07:27:56.516878+0000","last_clean":"2024-03-06T07:27:56.516878+0000","last_became_active":"2024-03-06T07:27:55.023617+0000","last_became_peered":"2024-03-06T07:27:55.023617+0000","last_unstale":"2024-03-06T07:27:56.516878+0000","last_undegraded":"2024-03-06T07:27:56.516878+0000","last_fullsized":"2024-03-06T07:27:56.516878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:25.469187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515609+0000","last_change":"2024-03-06T07:27:54.028923+0000","last_active":"2024-03-06T07:27:56.515609+0000","last_peered":"2024-03-06T07:27:56.515609+0000","last_clean":"2024-03-06T07:27:56.515609+0000","last_became_active":"2024-03-06T07:27:54.028290+0000","last_became_peered":"2024-03-06T07:27:54.028290+0000","last_unstale":"2024-03-06T07:27:56.515609+0000","last_undegraded":"2024-03-06T07:27:56.515609+0000","last_fullsized":"2024-03-06T07:27:56.515609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:10.734558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.518611+0000","last_change":"2024-03-06T07:27:55.020150+0000","last_active":"2024-03-06T07:27:56.518611+0000","last_peered":"2024-03-06T07:27:56.518611+0000","last_clean":"2024-03-06T07:27:56.518611+0000","last_became_active":"2024-03-06T07:27:55.019546+0000","last_became_peered":"2024-03-06T07:27:55.019546+0000","last_unstale":"2024-03-06T07:27:56.518611+0000","last_undegraded":"2024-03-06T07:27:56.518611+0000","last_fullsized":"2024-03-06T07:27:56.518611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:54.756676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.515741+0000","last_change":"2024-03-06T07:27:54.027548+0000","last_active":"2024-03-06T07:27:56.515741+0000","last_peered":"2024-03-06T07:27:56.515741+0000","last_clean":"2024-03-06T07:27:56.515741+0000","last_became_active":"2024-03-06T07:27:54.027161+0000","last_became_peered":"2024-03-06T07:27:54.027161+0000","last_unstale":"2024-03-06T07:27:56.515741+0000","last_undegraded":"2024-03-06T07:27:56.515741+0000","last_fullsized":"2024-03-06T07:27:56.515741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:33.207695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514542+0000","last_change":"2024-03-06T07:27:55.019647+0000","last_active":"2024-03-06T07:27:56.514542+0000","last_peered":"2024-03-06T07:27:56.514542+0000","last_clean":"2024-03-06T07:27:56.514542+0000","last_became_active":"2024-03-06T07:27:55.018449+0000","last_became_peered":"2024-03-06T07:27:55.018449+0000","last_unstale":"2024-03-06T07:27:56.514542+0000","last_undegraded":"2024-03-06T07:27:56.514542+0000","last_fullsized":"2024-03-06T07:27:56.514542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:51.164838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021792+0000","last_change":"2024-03-06T07:27:54.008978+0000","last_active":"2024-03-06T07:27:56.021792+0000","last_peered":"2024-03-06T07:27:56.021792+0000","last_clean":"2024-03-06T07:27:56.021792+0000","last_became_active":"2024-03-06T07:27:54.004840+0000","last_became_peered":"2024-03-06T07:27:54.004840+0000","last_unstale":"2024-03-06T07:27:56.021792+0000","last_undegraded":"2024-03-06T07:27:56.021792+0000","last_fullsized":"2024-03-06T07:27:56.021792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:08.575504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.021485+0000","last_change":"2024-03-06T07:27:55.013247+0000","last_active":"2024-03-06T07:27:56.021485+0000","last_peered":"2024-03-06T07:27:56.021485+0000","last_clean":"2024-03-06T07:27:56.021485+0000","last_became_active":"2024-03-06T07:27:55.012951+0000","last_became_peered":"2024-03-06T07:27:55.012951+0000","last_unstale":"2024-03-06T07:27:56.021485+0000","last_undegraded":"2024-03-06T07:27:56.021485+0000","last_fullsized":"2024-03-06T07:27:56.021485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:53.968464+0000","last_clean_scrub_stamp":"2024-03-06T07:27:53.968464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:02.636589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:27:56.514798+0000","last_change":"2024-03-06T07:27:54.019750+0000","last_active":"2024-03-06T07:27:56.514798+0000","last_peered":"2024-03-06T07:27:56.514798+0000","last_clean":"2024-03-06T07:27:56.514798+0000","last_became_active":"2024-03-06T07:27:54.019443+0000","last_became_peered":"2024-03-06T07:27:54.019443+0000","last_unstale":"2024-03-06T07:27:56.514798+0000","last_undegraded":"2024-03-06T07:27:56.514798+0000","last_fullsized":"2024-03-06T07:27:56.514798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:52.953520+0000","last_clean_scrub_stamp":"2024-03-06T07:27:52.953520+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:43.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:05.637 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:28:05.637 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:28:05.638 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:28:05.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:28:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.866+0000 7fcf0ec00640 1 Processor -- start 2024-03-06T07:28:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.867+0000 7fcf0ec00640 1 -- start start 2024-03-06T07:28:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0ec00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcf08144630 0x7fcf0815b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0ec00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf080b4590 0x7fcf080b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0ec00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0815c150 con 0x7fcf080729d0 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0ec00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf08144a10 con 0x7fcf080b4590 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0ec00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf080b4f40 con 0x7fcf08144630 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf080b4590 0x7fcf080b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf080b4590 0x7fcf080b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56196/0 (socket says 172.21.15.114:56196) 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.868+0000 7fcf0cc00640 1 -- 172.21.15.114:0/3919259561 learned_addr learned my addr 172.21.15.114:0/3919259561 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.869+0000 7fcf0cc00640 1 -- 172.21.15.114:0/3919259561 >> v2:172.21.15.177:3300/0 conn(0x7fcf08144630 msgr2=0x7fcf0815b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.869+0000 7fcf0cc00640 1 --2- 172.21.15.114:0/3919259561 >> v2:172.21.15.177:3300/0 conn(0x7fcf08144630 0x7fcf0815b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.869+0000 7fcf0cc00640 1 -- 172.21.15.114:0/3919259561 >> v1:172.21.15.177:6789/0 conn(0x7fcf080729d0 legacy=0x7fcf0815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.869+0000 7fcf0cc00640 1 -- 172.21.15.114:0/3919259561 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0815bb90 con 0x7fcf080b4590 2024-03-06T07:28:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.870+0000 7fcf0cc00640 1 --2- 172.21.15.114:0/3919259561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf080b4a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf0804fbf0 tx=0x7fcf0002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c075a345d4b5c44 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.870+0000 7fcf06a00640 1 -- 172.21.15.114:0/3919259561 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0002ca80 con 0x7fcf080b4590 2024-03-06T07:28:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.870+0000 7fcf06a00640 1 -- 172.21.15.114:0/3919259561 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0002cc20 con 0x7fcf080b4590 2024-03-06T07:28:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.870+0000 7fcf06a00640 1 -- 172.21.15.114:0/3919259561 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf000343f0 con 0x7fcf080b4590 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 -- 172.21.15.114:0/3919259561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 msgr2=0x7fcf080b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/3919259561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf080b4a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf0804fbf0 tx=0x7fcf0002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 -- 172.21.15.114:0/3919259561 shutdown_connections 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/3919259561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf080b4a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/3919259561 >> v2:172.21.15.177:3300/0 conn(0x7fcf08144630 0x7fcf0815b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 -- 172.21.15.114:0/3919259561 >> 172.21.15.114:0/3919259561 conn(0x7fcf080cfa30 msgr2=0x7fcf080cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 -- 172.21.15.114:0/3919259561 shutdown_connections 2024-03-06T07:28:05.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.871+0000 7fcf0ec00640 1 -- 172.21.15.114:0/3919259561 wait complete. 2024-03-06T07:28:05.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.873+0000 7fcf0ec00640 1 Processor -- start 2024-03-06T07:28:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.873+0000 7fcf0ec00640 1 -- start start 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf0ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf080729d0 0x7fcf08142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf0ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf081e3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf0ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf08144630 0x7fcf081f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf0ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080b2b60 con 0x7fcf080b4590 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf0ec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0815cab0 con 0x7fcf080729d0 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf0ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080b2e40 con 0x7fcf08144630 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf07400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf081e3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf07400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf081e3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56210/0 (socket says 172.21.15.114:56210) 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.874+0000 7fcf07400640 1 -- 172.21.15.114:0/570552749 learned_addr learned my addr 172.21.15.114:0/570552749 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf07400640 1 -- 172.21.15.114:0/570552749 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf08144630 msgr2=0x7fcf081f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf07400640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf08144630 0x7fcf081f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf07400640 1 -- 172.21.15.114:0/570552749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf080729d0 msgr2=0x7fcf08142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf07e00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf080729d0 0x7fcf08142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf07400640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf080729d0 0x7fcf08142590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf07400640 1 -- 172.21.15.114:0/570552749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf080ab790 con 0x7fcf080b4590 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf07400640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf081e3ba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c400 tx=0x7fcef400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fceffe00640 1 -- 172.21.15.114:0/570552749 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4017070 con 0x7fcf080b4590 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fceffe00640 1 -- 172.21.15.114:0/570552749 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef40035f0 con 0x7fcf080b4590 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf080aba70 con 0x7fcf080b4590 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fceffe00640 1 -- 172.21.15.114:0/570552749 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4003a20 con 0x7fcf080b4590 2024-03-06T07:28:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.875+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf080ac030 con 0x7fcf080b4590 2024-03-06T07:28:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.876+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0804f330 con 0x7fcf080b4590 2024-03-06T07:28:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.877+0000 7fceffe00640 1 -- 172.21.15.114:0/570552749 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcef401a070 con 0x7fcf080b4590 2024-03-06T07:28:05.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.879+0000 7fceffe00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcee806b4d0 0x7fcee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.879+0000 7fceffe00640 1 -- 172.21.15.114:0/570552749 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcef4089520 con 0x7fcf080b4590 2024-03-06T07:28:05.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.879+0000 7fcf07e00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcee806b4d0 0x7fcee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.880+0000 7fcf07e00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcee806b4d0 0x7fcee806d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcef800a770 tx=0x7fcef8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:05.883+0000 7fceffe00640 1 -- 172.21.15.114:0/570552749 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcef4057d40 con 0x7fcf080b4590 2024-03-06T07:28:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.176+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf0815c420 con 0x7fcf080b4590 2024-03-06T07:28:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.177+0000 7fceffe00640 1 -- 172.21.15.114:0/570552749 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fcef405bde0 con 0x7fcf080b4590 2024-03-06T07:28:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:06.180 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:28:06.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.180+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcee806b4d0 msgr2=0x7fcee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.180+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcee806b4d0 0x7fcee806d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcef800a770 tx=0x7fcef8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.180+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 msgr2=0x7fcf081e3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.180+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf081e3ba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcef400c400 tx=0x7fcef400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.180+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 shutdown_connections 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.181+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf08144630 0x7fcf081f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.181+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcee806b4d0 0x7fcee806d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.181+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf080b4590 0x7fcf081e3ba0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.181+0000 7fcf0ec00640 1 --2- 172.21.15.114:0/570552749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf080729d0 0x7fcf08142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.181+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 >> 172.21.15.114:0/570552749 conn(0x7fcf080cfa30 msgr2=0x7fcf0806e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.181+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 shutdown_connections 2024-03-06T07:28:06.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.181+0000 7fcf0ec00640 1 -- 172.21.15.114:0/570552749 wait complete. 2024-03-06T07:28:06.200 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:28:06.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:06.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.390+0000 7fee05400640 1 Processor -- start 2024-03-06T07:28:06.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.390+0000 7fee05400640 1 -- start start 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.391+0000 7fee05400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fee0015f9d0 0x7fee0015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.391+0000 7fee05400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0006cef0 con 0x7fee00144c00 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.392+0000 7fee05400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0006d1d0 con 0x7fee00070b10 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.392+0000 7fee05400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0006d4b0 con 0x7fee0015f9d0 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.392+0000 7fedfe800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fee00144c00 0x7fee0006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36110/0 (socket says 172.21.15.114:36110) 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.392+0000 7fedfe800640 1 -- 172.21.15.114:0/1717968016 learned_addr learned my addr 172.21.15.114:0/1717968016 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.392+0000 7fedf5e00640 1 --2- 172.21.15.114:0/1717968016 >> v2:172.21.15.177:3300/0 conn(0x7fee0015f9d0 0x7fee0015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.392+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182878424 0 0) 0x7fee0006cef0 con 0x7fee00144c00 2024-03-06T07:28:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.392+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede4003520 con 0x7fee00144c00 2024-03-06T07:28:06.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2422504598 0 0) 0x7fede4003520 con 0x7fee00144c00 2024-03-06T07:28:06.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 >> v2:172.21.15.177:3300/0 conn(0x7fee0015f9d0 msgr2=0x7fee0015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 --2- 172.21.15.114:0/1717968016 >> v2:172.21.15.177:3300/0 conn(0x7fee0015f9d0 0x7fee0015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 >> v1:172.21.15.177:6789/0 conn(0x7fee00070b10 legacy=0x7fee00070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0006d790 con 0x7fee00144c00 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fedf0001670 con 0x7fee00144c00 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 >> v1:172.21.15.114:6789/0 conn(0x7fee00144c00 legacy=0x7fee0006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fede4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede4004800 0x7fede4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede40051c0 0x7fede400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0006d4b0 con 0x7fede40039b0 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede400c060 con 0x7fede40051c0 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede400c260 con 0x7fede4004800 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfe800640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fede4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfe800640 1 -- 172.21.15.114:0/1717968016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede4004800 msgr2=0x7fede4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfe800640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fede4004800 0x7fede4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfe800640 1 -- 172.21.15.114:0/1717968016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede40051c0 msgr2=0x7fede400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfe800640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede40051c0 0x7fede400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedfe800640 1 -- 172.21.15.114:0/1717968016 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede4003520 con 0x7fede40039b0 2024-03-06T07:28:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.393+0000 7fedff200640 1 -- 172.21.15.114:0/1717968016 reap_dead start 2024-03-06T07:28:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.394+0000 7fedfe800640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fede4003dd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedf000c290 tx=0x7fede4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2607165a9e7d6ff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.395+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0007d30 con 0x7fede40039b0 2024-03-06T07:28:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.395+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf0030070 con 0x7fede40039b0 2024-03-06T07:28:06.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.395+0000 7fedfde00640 1 -- 172.21.15.114:0/1717968016 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0005d30 con 0x7fede40039b0 2024-03-06T07:28:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.395+0000 7fee05400640 1 -- 172.21.15.114:0/1717968016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 msgr2=0x7fede4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.395+0000 7fee05400640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fede4003dd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedf000c290 tx=0x7fede4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.396+0000 7fee05400640 1 -- 172.21.15.114:0/1717968016 shutdown_connections 2024-03-06T07:28:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.396+0000 7fee05400640 1 --2- 172.21.15.114:0/1717968016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fede4003dd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.396+0000 7fee05400640 1 -- 172.21.15.114:0/1717968016 >> 172.21.15.114:0/1717968016 conn(0x7fee000b1730 msgr2=0x7fee000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.396+0000 7fee05400640 1 -- 172.21.15.114:0/1717968016 shutdown_connections 2024-03-06T07:28:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.396+0000 7fee05400640 1 -- 172.21.15.114:0/1717968016 wait complete. 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.397+0000 7fee05400640 1 Processor -- start 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.397+0000 7fee05400640 1 -- start start 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.398+0000 7fee05400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fee001418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.398+0000 7fee05400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee000b2b30 0x7fee0015b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.398+0000 7fee05400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee0015ebd0 0x7fee00142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.398+0000 7fee05400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0006ccd0 con 0x7fede40039b0 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.398+0000 7fee05400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0006cb50 con 0x7fee000b2b30 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.398+0000 7fee05400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0006df90 con 0x7fee0015ebd0 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fee001418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fee001418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56222/0 (socket says 172.21.15.114:56222) 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 -- 172.21.15.114:0/2016022131 learned_addr learned my addr 172.21.15.114:0/2016022131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 -- 172.21.15.114:0/2016022131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee0015ebd0 msgr2=0x7fee00142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee0015ebd0 0x7fee00142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 -- 172.21.15.114:0/2016022131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee000b2b30 msgr2=0x7fee0015b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee000b2b30 0x7fee0015b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 -- 172.21.15.114:0/2016022131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0015db30 con 0x7fede40039b0 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedfe800640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fee001418c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fedf0001970 tx=0x7fedf0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.399+0000 7fedf7400640 1 -- 172.21.15.114:0/2016022131 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0045070 con 0x7fede40039b0 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.400+0000 7fedf7400640 1 -- 172.21.15.114:0/2016022131 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf0004bf0 con 0x7fede40039b0 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.400+0000 7fedf7400640 1 -- 172.21.15.114:0/2016022131 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0001c80 con 0x7fede40039b0 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.400+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0015de10 con 0x7fede40039b0 2024-03-06T07:28:06.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.400+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0015e3d0 con 0x7fede40039b0 2024-03-06T07:28:06.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.401+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0004f330 con 0x7fede40039b0 2024-03-06T07:28:06.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.402+0000 7fedf7400640 1 -- 172.21.15.114:0/2016022131 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedf0007070 con 0x7fede40039b0 2024-03-06T07:28:06.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.403+0000 7fedf7400640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feddc06b4d0 0x7feddc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.403+0000 7fedf7400640 1 -- 172.21.15.114:0/2016022131 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedf00ab240 con 0x7fede40039b0 2024-03-06T07:28:06.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.403+0000 7fedf5e00640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feddc06b4d0 0x7feddc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.404+0000 7fedf5e00640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feddc06b4d0 0x7feddc06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fede8009390 tx=0x7fede8009300 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.408+0000 7fedf7400640 1 -- 172.21.15.114:0/2016022131 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedf0079a70 con 0x7fede40039b0 2024-03-06T07:28:06.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.602+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee00072890 con 0x7fede40039b0 2024-03-06T07:28:06.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.604+0000 7fedf7400640 1 -- 172.21.15.114:0/2016022131 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fedf007db10 con 0x7fede40039b0 2024-03-06T07:28:06.607 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:06.607 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:06.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feddc06b4d0 msgr2=0x7feddc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feddc06b4d0 0x7feddc06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fede8009390 tx=0x7fede8009300 comp rx=0 tx=0).stop 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 msgr2=0x7fee001418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fee001418c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fedf0001970 tx=0x7fedf0001a40 comp rx=0 tx=0).stop 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 shutdown_connections 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee0015ebd0 0x7fee00142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee000b2b30 0x7fee0015b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feddc06b4d0 0x7feddc06d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 --2- 172.21.15.114:0/2016022131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fede40039b0 0x7fee001418c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.606+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 >> 172.21.15.114:0/2016022131 conn(0x7fee000b1730 msgr2=0x7fee000cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.607+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 shutdown_connections 2024-03-06T07:28:06.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.607+0000 7fee05400640 1 -- 172.21.15.114:0/2016022131 wait complete. 2024-03-06T07:28:06.630 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:06.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:06.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.829+0000 7fa4ec600640 1 Processor -- start 2024-03-06T07:28:06.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.829+0000 7fa4ec600640 1 -- start start 2024-03-06T07:28:06.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.830+0000 7fa4ec600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa4e4071b10 0x7fa4e415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.830+0000 7fa4ec600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e40b4340 0x7fa4e40b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.831+0000 7fa4ec600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e40d0030 con 0x7fa4e4071ef0 2024-03-06T07:28:06.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.831+0000 7fa4ec600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e40d0310 con 0x7fa4e4071b10 2024-03-06T07:28:06.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.831+0000 7fa4ec600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e40d05f0 con 0x7fa4e40b4340 2024-03-06T07:28:06.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.831+0000 7fa4ea600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e40b4340 0x7fa4e40b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.831+0000 7fa4ea600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e40b4340 0x7fa4e40b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59406/0 (socket says 172.21.15.114:59406) 2024-03-06T07:28:06.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.831+0000 7fa4ea600640 1 -- 172.21.15.114:0/2180080285 learned_addr learned my addr 172.21.15.114:0/2180080285 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:06.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.831+0000 7fa4e9c00640 1 --2- 172.21.15.114:0/2180080285 >> v2:172.21.15.177:3300/0 conn(0x7fa4e4071b10 0x7fa4e415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.832+0000 7fa4ea600640 1 -- 172.21.15.114:0/2180080285 >> v2:172.21.15.177:3300/0 conn(0x7fa4e4071b10 msgr2=0x7fa4e415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.832+0000 7fa4ea600640 1 --2- 172.21.15.114:0/2180080285 >> v2:172.21.15.177:3300/0 conn(0x7fa4e4071b10 0x7fa4e415c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.832+0000 7fa4ea600640 1 -- 172.21.15.114:0/2180080285 >> v1:172.21.15.177:6789/0 conn(0x7fa4e4071ef0 legacy=0x7fa4e4145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.832+0000 7fa4ea600640 1 -- 172.21.15.114:0/2180080285 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e40b3b40 con 0x7fa4e40b4340 2024-03-06T07:28:06.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.833+0000 7fa4ea600640 1 --2- 172.21.15.114:0/2180080285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e40b4340 0x7fa4e40b47d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009e50 tx=0x7fa4dc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e541fd2689a19ded server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.834+0000 7fa4e3e00640 1 -- 172.21.15.114:0/2180080285 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc02cb00 con 0x7fa4e40b4340 2024-03-06T07:28:06.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.834+0000 7fa4e3e00640 1 -- 172.21.15.114:0/2180080285 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc02cca0 con 0x7fa4e40b4340 2024-03-06T07:28:06.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.834+0000 7fa4e3e00640 1 -- 172.21.15.114:0/2180080285 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc034610 con 0x7fa4e40b4340 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.834+0000 7fa4ec600640 1 -- 172.21.15.114:0/2180080285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e40b4340 msgr2=0x7fa4e40b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.834+0000 7fa4ec600640 1 --2- 172.21.15.114:0/2180080285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e40b4340 0x7fa4e40b47d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009e50 tx=0x7fa4dc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.835+0000 7fa4ec600640 1 -- 172.21.15.114:0/2180080285 shutdown_connections 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.835+0000 7fa4ec600640 1 --2- 172.21.15.114:0/2180080285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e40b4340 0x7fa4e40b47d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.835+0000 7fa4ec600640 1 --2- 172.21.15.114:0/2180080285 >> v2:172.21.15.177:3300/0 conn(0x7fa4e4071b10 0x7fa4e415c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.835+0000 7fa4ec600640 1 -- 172.21.15.114:0/2180080285 >> 172.21.15.114:0/2180080285 conn(0x7fa4e40b2840 msgr2=0x7fa4e40b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.835+0000 7fa4ec600640 1 -- 172.21.15.114:0/2180080285 shutdown_connections 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.835+0000 7fa4ec600640 1 -- 172.21.15.114:0/2180080285 wait complete. 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.836+0000 7fa4ec600640 1 Processor -- start 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.837+0000 7fa4ec600640 1 -- start start 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.837+0000 7fa4ec600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e4071b10 0x7fa4e41403c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4ec600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e4071ef0 0x7fa4e41e4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4ec600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4e40b4340 0x7fa4e40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4ec600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e4072380 con 0x7fa4e4071ef0 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4ec600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e4145980 con 0x7fa4e40b4340 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4ec600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e40cfcc0 con 0x7fa4e4071b10 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4e9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e4071ef0 0x7fa4e41e4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4e9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e4071ef0 0x7fa4e41e4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56234/0 (socket says 172.21.15.114:56234) 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4e9200640 1 -- 172.21.15.114:0/3917837023 learned_addr learned my addr 172.21.15.114:0/3917837023 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4e9c00640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e4071b10 0x7fa4e41403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4ea600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4e40b4340 0x7fa4e40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4e9200640 1 -- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e4071b10 msgr2=0x7fa4e41403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.838+0000 7fa4e9200640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e4071b10 0x7fa4e41403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4e9200640 1 -- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4e40b4340 msgr2=0x7fa4e40ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4e9200640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4e40b4340 0x7fa4e40ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4e9200640 1 -- 172.21.15.114:0/3917837023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e41409e0 con 0x7fa4e4071ef0 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4ea600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4e40b4340 0x7fa4e40ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4e9200640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e4071ef0 0x7fa4e41e4570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c400 tx=0x7fa4d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4e1600640 1 -- 172.21.15.114:0/3917837023 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d00052d0 con 0x7fa4e4071ef0 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4e1600640 1 -- 172.21.15.114:0/3917837023 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d0005490 con 0x7fa4e4071ef0 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4e1600640 1 -- 172.21.15.114:0/3917837023 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0007260 con 0x7fa4e4071ef0 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e4140cc0 con 0x7fa4e4071ef0 2024-03-06T07:28:06.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.839+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e4141280 con 0x7fa4e4071ef0 2024-03-06T07:28:06.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.840+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e404f330 con 0x7fa4e4071ef0 2024-03-06T07:28:06.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.841+0000 7fa4e1600640 1 -- 172.21.15.114:0/3917837023 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4d0007400 con 0x7fa4e4071ef0 2024-03-06T07:28:06.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.843+0000 7fa4e1600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4c406b4d0 0x7fa4c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.843+0000 7fa4e1600640 1 -- 172.21.15.114:0/3917837023 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4d00897f0 con 0x7fa4e4071ef0 2024-03-06T07:28:06.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.843+0000 7fa4e9c00640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4c406b4d0 0x7fa4c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.843+0000 7fa4e9c00640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4c406b4d0 0x7fa4c406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009ad0 tx=0x7fa4d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:06.847+0000 7fa4e1600640 1 -- 172.21.15.114:0/3917837023 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4d00585d0 con 0x7fa4e4071ef0 2024-03-06T07:28:07.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.046+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa4e40ce240 con 0x7fa4e4071ef0 2024-03-06T07:28:07.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.047+0000 7fa4e1600640 1 -- 172.21.15.114:0/3917837023 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4d0057ed0 con 0x7fa4e4071ef0 2024-03-06T07:28:07.050 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4c406b4d0 msgr2=0x7fa4c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4c406b4d0 0x7fa4c406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009ad0 tx=0x7fa4d8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e4071ef0 msgr2=0x7fa4e41e4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e4071ef0 0x7fa4e41e4570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c400 tx=0x7fa4d000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 shutdown_connections 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4e40b4340 0x7fa4e40ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4c406b4d0 0x7fa4c406d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4e4071ef0 0x7fa4e41e4570 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 --2- 172.21.15.114:0/3917837023 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4e4071b10 0x7fa4e41403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 >> 172.21.15.114:0/3917837023 conn(0x7fa4e40b2840 msgr2=0x7fa4e40afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 shutdown_connections 2024-03-06T07:28:07.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.050+0000 7fa4ec600640 1 -- 172.21.15.114:0/3917837023 wait complete. 2024-03-06T07:28:07.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.278+0000 7fae6b800640 1 Processor -- start 2024-03-06T07:28:07.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.278+0000 7fae6b800640 1 -- start start 2024-03-06T07:28:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae6b800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae6406fef0 0x7fae6406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae6b800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae640d1260 con 0x7fae6415e3e0 2024-03-06T07:28:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae6b800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6406c320 con 0x7fae6415e000 2024-03-06T07:28:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae6b800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae6406c0d0 con 0x7fae6406fef0 2024-03-06T07:28:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae63e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae6406fef0 0x7fae6406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae69800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fae6415e3e0 0x7fae640ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36116/0 (socket says 172.21.15.114:36116) 2024-03-06T07:28:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae69800640 1 -- 172.21.15.114:0/3937609473 learned_addr learned my addr 172.21.15.114:0/3937609473 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae63e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae6406fef0 0x7fae6406f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56248/0 (socket says 172.21.15.114:56248) 2024-03-06T07:28:07.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae63e00640 1 -- 172.21.15.114:0/3937609473 >> v1:172.21.15.177:6789/0 conn(0x7fae6415e000 legacy=0x7fae6406f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.280+0000 7fae63e00640 1 -- 172.21.15.114:0/3937609473 >> v1:172.21.15.114:6789/0 conn(0x7fae6415e3e0 legacy=0x7fae640ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.281+0000 7fae63e00640 1 -- 172.21.15.114:0/3937609473 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6406c600 con 0x7fae6406fef0 2024-03-06T07:28:07.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.281+0000 7fae63400640 1 -- 172.21.15.114:0/3937609473 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855908344 0 0) 0x7fae640d1260 con 0x7fae6415e3e0 2024-03-06T07:28:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.282+0000 7fae63e00640 1 --2- 172.21.15.114:0/3937609473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae6406f130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae500097a0 tx=0x7fae5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad3a394cde44e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.282+0000 7fae63400640 1 -- 172.21.15.114:0/3937609473 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50035020 con 0x7fae6406fef0 2024-03-06T07:28:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.282+0000 7fae63400640 1 -- 172.21.15.114:0/3937609473 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50031830 con 0x7fae6406fef0 2024-03-06T07:28:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.282+0000 7fae63400640 1 -- 172.21.15.114:0/3937609473 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50031c00 con 0x7fae6406fef0 2024-03-06T07:28:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.282+0000 7fae6b800640 1 -- 172.21.15.114:0/3937609473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 msgr2=0x7fae6406f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.282+0000 7fae6b800640 1 --2- 172.21.15.114:0/3937609473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae6406f130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae500097a0 tx=0x7fae5002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.283+0000 7fae6b800640 1 -- 172.21.15.114:0/3937609473 shutdown_connections 2024-03-06T07:28:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.283+0000 7fae6b800640 1 --2- 172.21.15.114:0/3937609473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae6406f130 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.283+0000 7fae6b800640 1 -- 172.21.15.114:0/3937609473 >> 172.21.15.114:0/3937609473 conn(0x7fae640b3800 msgr2=0x7fae640b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.283+0000 7fae6b800640 1 -- 172.21.15.114:0/3937609473 shutdown_connections 2024-03-06T07:28:07.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.283+0000 7fae6b800640 1 -- 172.21.15.114:0/3937609473 wait complete. 2024-03-06T07:28:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.284+0000 7fae6b800640 1 Processor -- start 2024-03-06T07:28:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.284+0000 7fae6b800640 1 -- start start 2024-03-06T07:28:07.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.285+0000 7fae6b800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae641511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.285+0000 7fae6b800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6415e000 0x7fae64160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.285+0000 7fae6b800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae6415e3e0 0x7fae640c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.285+0000 7fae6b800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640d0ae0 con 0x7fae6406fef0 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae6b800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6406e060 con 0x7fae6415e3e0 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae6b800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640d0c60 con 0x7fae6415e000 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae641511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae641511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56256/0 (socket says 172.21.15.114:56256) 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 -- 172.21.15.114:0/408666475 learned_addr learned my addr 172.21.15.114:0/408666475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae69800640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae6415e3e0 0x7fae640c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 -- 172.21.15.114:0/408666475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6415e000 msgr2=0x7fae64160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6415e000 0x7fae64160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 -- 172.21.15.114:0/408666475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae6415e3e0 msgr2=0x7fae640c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae6415e3e0 0x7fae640c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 -- 172.21.15.114:0/408666475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae641611f0 con 0x7fae6406fef0 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.286+0000 7fae68e00640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae641511f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c430 tx=0x7fae5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.287+0000 7fae60c00640 1 -- 172.21.15.114:0/408666475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58011790 con 0x7fae6406fef0 2024-03-06T07:28:07.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.287+0000 7fae60c00640 1 -- 172.21.15.114:0/408666475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae58017070 con 0x7fae6406fef0 2024-03-06T07:28:07.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.287+0000 7fae60c00640 1 -- 172.21.15.114:0/408666475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58011bb0 con 0x7fae6406fef0 2024-03-06T07:28:07.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.287+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae641614d0 con 0x7fae6406fef0 2024-03-06T07:28:07.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.287+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae640c6430 con 0x7fae6406fef0 2024-03-06T07:28:07.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.288+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6404f330 con 0x7fae6406fef0 2024-03-06T07:28:07.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.289+0000 7fae60c00640 1 -- 172.21.15.114:0/408666475 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae5801c460 con 0x7fae6406fef0 2024-03-06T07:28:07.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.290+0000 7fae60c00640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae4406b520 0x7fae4406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.290+0000 7fae60c00640 1 -- 172.21.15.114:0/408666475 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae5808a0a0 con 0x7fae6406fef0 2024-03-06T07:28:07.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.290+0000 7fae63e00640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae4406b520 0x7fae4406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.291+0000 7fae63e00640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae4406b520 0x7fae4406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae50036000 tx=0x7fae50030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.294+0000 7fae60c00640 1 -- 172.21.15.114:0/408666475 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae58058f00 con 0x7fae6406fef0 2024-03-06T07:28:07.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.544+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae64070e30 con 0x7fae6406fef0 2024-03-06T07:28:07.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.545+0000 7fae60c00640 1 -- 172.21.15.114:0/408666475 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae58058800 con 0x7fae6406fef0 2024-03-06T07:28:07.548 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:07.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae4406b520 msgr2=0x7fae4406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae4406b520 0x7fae4406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae50036000 tx=0x7fae50030000 comp rx=0 tx=0).stop 2024-03-06T07:28:07.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 msgr2=0x7fae641511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae641511f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c430 tx=0x7fae5800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 shutdown_connections 2024-03-06T07:28:07.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae6415e3e0 0x7fae640c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6415e000 0x7fae64160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae4406b520 0x7fae4406d9e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 --2- 172.21.15.114:0/408666475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6406fef0 0x7fae641511f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 >> 172.21.15.114:0/408666475 conn(0x7fae640b3800 msgr2=0x7fae6406f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 shutdown_connections 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.547+0000 7fae6b800640 1 -- 172.21.15.114:0/408666475 wait complete. 2024-03-06T07:28:07.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.766+0000 7fadd3200640 1 Processor -- start 2024-03-06T07:28:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.766+0000 7fadd3200640 1 -- start start 2024-03-06T07:28:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.767+0000 7fadd3200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fadd406afc0 0x7fadd4160400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.767+0000 7fadd3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd40725d0 con 0x7fadd40cc9f0 2024-03-06T07:28:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.767+0000 7fadd3200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd406b5a0 con 0x7fadd415c050 2024-03-06T07:28:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.767+0000 7fadd3200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd4160940 con 0x7fadd406afc0 2024-03-06T07:28:07.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.768+0000 7fadd2800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fadd40cc9f0 0x7fadd40ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36130/0 (socket says 172.21.15.114:36130) 2024-03-06T07:28:07.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.768+0000 7fadd2800640 1 -- 172.21.15.114:0/1407262580 learned_addr learned my addr 172.21.15.114:0/1407262580 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:07.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.768+0000 7fadc9400640 1 --2- 172.21.15.114:0/1407262580 >> v2:172.21.15.177:3300/0 conn(0x7fadd406afc0 0x7fadd4160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.768+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779293011 0 0) 0x7fadd40725d0 con 0x7fadd40cc9f0 2024-03-06T07:28:07.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.768+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb8003520 con 0x7fadd40cc9f0 2024-03-06T07:28:07.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063057362 0 0) 0x7fadb8003520 con 0x7fadd40cc9f0 2024-03-06T07:28:07.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 >> v2:172.21.15.177:3300/0 conn(0x7fadd406afc0 msgr2=0x7fadd4160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 --2- 172.21.15.114:0/1407262580 >> v2:172.21.15.177:3300/0 conn(0x7fadd406afc0 0x7fadd4160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 >> v1:172.21.15.177:6789/0 conn(0x7fadd415c050 legacy=0x7fadd4072d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd406b880 con 0x7fadd40cc9f0 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fadcc0019f0 con 0x7fadd40cc9f0 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 >> v1:172.21.15.114:6789/0 conn(0x7fadd40cc9f0 legacy=0x7fadd40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadb80039b0 0x7fadb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadb8004800 0x7fadb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd4160940 con 0x7fadb80051c0 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb800c060 con 0x7fadb8004800 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb800c260 con 0x7fadb80039b0 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadd2800640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.769+0000 7fadc9400640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadb8004800 0x7fadb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.770+0000 7fadd1e00640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadb80039b0 0x7fadb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.770+0000 7fadd2800640 1 -- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadb80039b0 msgr2=0x7fadb8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.770+0000 7fadd2800640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadb80039b0 0x7fadb8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.770+0000 7fadd2800640 1 -- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadb8004800 msgr2=0x7fadb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.770+0000 7fadd2800640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadb8004800 0x7fadb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.770+0000 7fadd2800640 1 -- 172.21.15.114:0/1407262580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb8003520 con 0x7fadb80051c0 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.771+0000 7fadd2800640 1 -- 172.21.15.114:0/1407262580 reap_dead start 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.771+0000 7fadd2800640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadb800b9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00a850 tx=0x7fadb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ca7acb6161196b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.771+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00a4a0 con 0x7fadb80051c0 2024-03-06T07:28:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.771+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc034650 con 0x7fadb80051c0 2024-03-06T07:28:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.771+0000 7fadd1400640 1 -- 172.21.15.114:0/1407262580 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc034a40 con 0x7fadb80051c0 2024-03-06T07:28:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.771+0000 7fadd3200640 1 -- 172.21.15.114:0/1407262580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 msgr2=0x7fadb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.771+0000 7fadd3200640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadb800b9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00a850 tx=0x7fadb8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.772+0000 7fadd3200640 1 -- 172.21.15.114:0/1407262580 shutdown_connections 2024-03-06T07:28:07.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.772+0000 7fadd3200640 1 --2- 172.21.15.114:0/1407262580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadb800b9c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.772+0000 7fadd3200640 1 -- 172.21.15.114:0/1407262580 >> 172.21.15.114:0/1407262580 conn(0x7fadd40b1d00 msgr2=0x7fadd40b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.772+0000 7fadd3200640 1 -- 172.21.15.114:0/1407262580 shutdown_connections 2024-03-06T07:28:07.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.772+0000 7fadd3200640 1 -- 172.21.15.114:0/1407262580 wait complete. 2024-03-06T07:28:07.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.776+0000 7fadd3200640 1 Processor -- start 2024-03-06T07:28:07.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.776+0000 7fadd3200640 1 -- start start 2024-03-06T07:28:07.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadd4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadd40c56c0 0x7fadd40c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadd40c7ef0 0x7fadd40c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd406bce0 con 0x7fadb80051c0 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd406bb60 con 0x7fadd40c56c0 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd3200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd4160400 con 0x7fadd40c7ef0 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadd4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadd4099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56280/0 (socket says 172.21.15.114:56280) 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.778+0000 7fadd1e00640 1 -- 172.21.15.114:0/3951013203 learned_addr learned my addr 172.21.15.114:0/3951013203 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadd1e00640 1 -- 172.21.15.114:0/3951013203 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadd40c7ef0 msgr2=0x7fadd40c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadd1e00640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadd40c7ef0 0x7fadd40c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadc9400640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadd40c56c0 0x7fadd40c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadd1e00640 1 -- 172.21.15.114:0/3951013203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadd40c56c0 msgr2=0x7fadd40c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadd1e00640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadd40c56c0 0x7fadd40c7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadd1e00640 1 -- 172.21.15.114:0/3951013203 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd415f350 con 0x7fadb80051c0 2024-03-06T07:28:07.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadd1e00640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadd4099a40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fadc400e440 tx=0x7fadc400e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.779+0000 7fadcaa00640 1 -- 172.21.15.114:0/3951013203 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4019070 con 0x7fadb80051c0 2024-03-06T07:28:07.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.780+0000 7fadcaa00640 1 -- 172.21.15.114:0/3951013203 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc40052e0 con 0x7fadb80051c0 2024-03-06T07:28:07.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.780+0000 7fadcaa00640 1 -- 172.21.15.114:0/3951013203 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc40037f0 con 0x7fadb80051c0 2024-03-06T07:28:07.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.780+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40c5bc0 con 0x7fadb80051c0 2024-03-06T07:28:07.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.780+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd40c6180 con 0x7fadb80051c0 2024-03-06T07:28:07.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.781+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd40b1320 con 0x7fadb80051c0 2024-03-06T07:28:07.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.783+0000 7fadcaa00640 1 -- 172.21.15.114:0/3951013203 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadc401c070 con 0x7fadb80051c0 2024-03-06T07:28:07.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.784+0000 7fadcaa00640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fadb006b4d0 0x7fadb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.784+0000 7fadcaa00640 1 -- 172.21.15.114:0/3951013203 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadc40896a0 con 0x7fadb80051c0 2024-03-06T07:28:07.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.784+0000 7fadc9400640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fadb006b4d0 0x7fadb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.785+0000 7fadc9400640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fadb006b4d0 0x7fadb006d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fadd415fe80 tx=0x7fadbc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.792+0000 7fadcaa00640 1 -- 172.21.15.114:0/3951013203 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadd40b1320 con 0x7fadb80051c0 2024-03-06T07:28:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.988+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fadd40b5dd0 con 0x7fadb80051c0 2024-03-06T07:28:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.989+0000 7fadcaa00640 1 -- 172.21.15.114:0/3951013203 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fadd40b5dd0 con 0x7fadb80051c0 2024-03-06T07:28:07.992 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fadb006b4d0 msgr2=0x7fadb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fadb006b4d0 0x7fadb006d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fadd415fe80 tx=0x7fadbc008040 comp rx=0 tx=0).stop 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 msgr2=0x7fadd4099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadd4099a40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fadc400e440 tx=0x7fadc400e900 comp rx=0 tx=0).stop 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 shutdown_connections 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fadd40c7ef0 0x7fadd40c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fadd40c56c0 0x7fadd40c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fadb006b4d0 0x7fadb006d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 --2- 172.21.15.114:0/3951013203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadb80051c0 0x7fadd4099a40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 >> 172.21.15.114:0/3951013203 conn(0x7fadd40b1d00 msgr2=0x7fadd40cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 shutdown_connections 2024-03-06T07:28:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:07.992+0000 7fadd3200640 1 -- 172.21.15.114:0/3951013203 wait complete. 2024-03-06T07:28:08.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:08.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.213+0000 7f6857e00640 1 Processor -- start 2024-03-06T07:28:08.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.213+0000 7f6857e00640 1 -- start start 2024-03-06T07:28:08.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6857e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f685015e0d0 0x7f68500d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6857e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f685015f470 0x7f68500d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6857e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68500d1200 con 0x7f68500d0df0 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6857e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68500d2f60 con 0x7f685015e0d0 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6857e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68500cea60 con 0x7f685015f470 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6855400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f685015e0d0 0x7f68500d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6855400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f685015e0d0 0x7f68500d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49562/0 (socket says 172.21.15.114:49562) 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.214+0000 7f6855400640 1 -- 172.21.15.114:0/3341974329 learned_addr learned my addr 172.21.15.114:0/3341974329 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.215+0000 7f6855400640 1 -- 172.21.15.114:0/3341974329 >> v2:172.21.15.177:3300/0 conn(0x7f685015f470 msgr2=0x7f68500d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.215+0000 7f6854a00640 1 --2- 172.21.15.114:0/3341974329 >> v2:172.21.15.177:3300/0 conn(0x7f685015f470 0x7f68500d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.215+0000 7f6855400640 1 --2- 172.21.15.114:0/3341974329 >> v2:172.21.15.177:3300/0 conn(0x7f685015f470 0x7f68500d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.215+0000 7f6855400640 1 -- 172.21.15.114:0/3341974329 >> v1:172.21.15.177:6789/0 conn(0x7f68500d0df0 legacy=0x7f6850160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.215+0000 7f6855400640 1 -- 172.21.15.114:0/3341974329 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500d2350 con 0x7f685015e0d0 2024-03-06T07:28:08.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.215+0000 7f6854a00640 1 --2- 172.21.15.114:0/3341974329 >> v2:172.21.15.177:3300/0 conn(0x7f685015f470 0x7f68500d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.216+0000 7f6855400640 1 --2- 172.21.15.114:0/3341974329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500d1a90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68500c77f0 tx=0x7f684402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ccb9febc5f0ced server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.216+0000 7f684fe00640 1 -- 172.21.15.114:0/3341974329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844034070 con 0x7f685015e0d0 2024-03-06T07:28:08.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.216+0000 7f684fe00640 1 -- 172.21.15.114:0/3341974329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6844030860 con 0x7f685015e0d0 2024-03-06T07:28:08.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.216+0000 7f684fe00640 1 -- 172.21.15.114:0/3341974329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844030c30 con 0x7f685015e0d0 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 -- 172.21.15.114:0/3341974329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 msgr2=0x7f68500d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 --2- 172.21.15.114:0/3341974329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500d1a90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68500c77f0 tx=0x7f684402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 -- 172.21.15.114:0/3341974329 shutdown_connections 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 --2- 172.21.15.114:0/3341974329 >> v2:172.21.15.177:3300/0 conn(0x7f685015f470 0x7f68500d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 --2- 172.21.15.114:0/3341974329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500d1a90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 -- 172.21.15.114:0/3341974329 >> 172.21.15.114:0/3341974329 conn(0x7f68500d3ec0 msgr2=0x7f68500d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 -- 172.21.15.114:0/3341974329 shutdown_connections 2024-03-06T07:28:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.217+0000 7f6857e00640 1 -- 172.21.15.114:0/3341974329 wait complete. 2024-03-06T07:28:08.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.219+0000 7f6857e00640 1 Processor -- start 2024-03-06T07:28:08.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.219+0000 7f6857e00640 1 -- start start 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6857e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68500d0df0 0x7f68500b9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6857e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6857e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685015f470 0x7f68500cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6857e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68500d2bf0 con 0x7f685015e0d0 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6857e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68500d2910 con 0x7f685015f470 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6857e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685015fb30 con 0x7f68500d0df0 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6854a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6854a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49568/0 (socket says 172.21.15.114:49568) 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6854a00640 1 -- 172.21.15.114:0/687015310 learned_addr learned my addr 172.21.15.114:0/687015310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6855400640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68500d0df0 0x7f68500b9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6855e00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685015f470 0x7f68500cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6854a00640 1 -- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68500d0df0 msgr2=0x7f68500b9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6854a00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68500d0df0 0x7f68500b9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6854a00640 1 -- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685015f470 msgr2=0x7f68500cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.220+0000 7f6854a00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685015f470 0x7f68500cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.221+0000 7f6854a00640 1 -- 172.21.15.114:0/687015310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500b7cb0 con 0x7f685015e0d0 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.221+0000 7f6854a00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500cd160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f683c00c7f0 tx=0x7f683c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.221+0000 7f684d600640 1 -- 172.21.15.114:0/687015310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c015070 con 0x7f685015e0d0 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.221+0000 7f684d600640 1 -- 172.21.15.114:0/687015310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c007a30 con 0x7f685015e0d0 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.221+0000 7f684d600640 1 -- 172.21.15.114:0/687015310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c007e00 con 0x7f685015e0d0 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.221+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68500b6c10 con 0x7f685015e0d0 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.221+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68500b71d0 con 0x7f685015e0d0 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.223+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68500d3990 con 0x7f685015e0d0 2024-03-06T07:28:08.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.223+0000 7f684d600640 1 -- 172.21.15.114:0/687015310 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f683c018070 con 0x7f685015e0d0 2024-03-06T07:28:08.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.224+0000 7f684d600640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f683006b500 0x7f683006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.224+0000 7f684d600640 1 -- 172.21.15.114:0/687015310 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f683c08a0c0 con 0x7f685015e0d0 2024-03-06T07:28:08.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.225+0000 7f6855400640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f683006b500 0x7f683006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.225+0000 7f6855400640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f683006b500 0x7f683006d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6844036000 tx=0x7f684403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.233+0000 7f684d600640 1 -- 172.21.15.114:0/687015310 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68500d3990 con 0x7f685015e0d0 2024-03-06T07:28:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.465+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f685004f330 con 0x7f685015e0d0 2024-03-06T07:28:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.467+0000 7f684d600640 1 -- 172.21.15.114:0/687015310 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f683c058fd0 con 0x7f685015e0d0 2024-03-06T07:28:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:08.470 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.469+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f683006b500 msgr2=0x7f683006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.469+0000 7f6857e00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f683006b500 0x7f683006d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6844036000 tx=0x7f684403b040 comp rx=0 tx=0).stop 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.469+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 msgr2=0x7f68500cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.469+0000 7f6857e00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500cd160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f683c00c7f0 tx=0x7f683c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 shutdown_connections 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685015f470 0x7f68500cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f683006b500 0x7f683006d9c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f685015e0d0 0x7f68500cd160 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 --2- 172.21.15.114:0/687015310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68500d0df0 0x7f68500b9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 >> 172.21.15.114:0/687015310 conn(0x7f68500d3ec0 msgr2=0x7f68500b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 shutdown_connections 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.470+0000 7f6857e00640 1 -- 172.21.15.114:0/687015310 wait complete. 2024-03-06T07:28:08.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:08.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.682+0000 7f4627e00640 1 Processor -- start 2024-03-06T07:28:08.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.682+0000 7f4627e00640 1 -- start start 2024-03-06T07:28:08.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4627e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f462815e200 0x7f46280d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4627e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f46280d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4627e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280b5240 con 0x7f462815e5e0 2024-03-06T07:28:08.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4627e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46280d2ec0 con 0x7f462815e200 2024-03-06T07:28:08.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4627e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46280cf4d0 con 0x7f46280b2e50 2024-03-06T07:28:08.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4627400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f462815e5e0 0x7f46280d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49650/0 (socket says 172.21.15.114:49650) 2024-03-06T07:28:08.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4627400640 1 -- 172.21.15.114:0/4109228775 learned_addr learned my addr 172.21.15.114:0/4109228775 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:08.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f461fe00640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f46280d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.685+0000 7f4626a00640 1 --2- 172.21.15.114:0/4109228775 >> v2:172.21.15.177:3300/0 conn(0x7f462815e200 0x7f46280d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967604824 0 0) 0x7f46280b5240 con 0x7f462815e5e0 2024-03-06T07:28:08.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c003520 con 0x7f462815e5e0 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2326580311 0 0) 0x7f460c003520 con 0x7f462815e5e0 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 msgr2=0x7f46280d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f46280d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 >> v2:172.21.15.177:3300/0 conn(0x7f462815e200 msgr2=0x7f46280d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 --2- 172.21.15.114:0/4109228775 >> v2:172.21.15.177:3300/0 conn(0x7f462815e200 0x7f46280d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280b5500 con 0x7f462815e5e0 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f46200019f0 con 0x7f462815e5e0 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626a00640 1 --2- 172.21.15.114:0/4109228775 >> v2:172.21.15.177:3300/0 conn(0x7f462815e200 0x7f46280d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f461fe00640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f46280d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 >> v1:172.21.15.114:6789/0 conn(0x7f462815e5e0 legacy=0x7f46280d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f460c0039b0 0x7f460c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.686+0000 7f4626000640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f460c004800 0x7f460c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626000640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f460c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280d2ec0 con 0x7f460c0039b0 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280cf4d0 con 0x7f460c004800 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c00b9c0 con 0x7f46280b2e50 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626a00640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f460c0039b0 0x7f460c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4627400640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f460c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626a00640 1 -- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 msgr2=0x7f460c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626a00640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f460c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626a00640 1 -- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f460c004800 msgr2=0x7f460c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626a00640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f460c004800 0x7f460c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4626a00640 1 -- 172.21.15.114:0/4109228775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c003520 con 0x7f460c0039b0 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.687+0000 7f4627400640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46280b2e50 0x7f460c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.688+0000 7f4626a00640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f460c0039b0 0x7f460c003dd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4618009c90 tx=0x7f460c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbf6cd41a07a80c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.688+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618039070 con 0x7f460c0039b0 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.688+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461802c980 con 0x7f460c0039b0 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.688+0000 7f4626000640 1 -- 172.21.15.114:0/4109228775 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618009860 con 0x7f460c0039b0 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.689+0000 7f4627e00640 1 -- 172.21.15.114:0/4109228775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f460c0039b0 msgr2=0x7f460c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.689+0000 7f4627e00640 1 --2- 172.21.15.114:0/4109228775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f460c0039b0 0x7f460c003dd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4618009c90 tx=0x7f460c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.689+0000 7f4627400640 1 -- 172.21.15.114:0/4109228775 reap_dead start 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.689+0000 7f4627e00640 1 -- 172.21.15.114:0/4109228775 shutdown_connections 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.689+0000 7f4627e00640 1 -- 172.21.15.114:0/4109228775 >> 172.21.15.114:0/4109228775 conn(0x7f46280d3d30 msgr2=0x7f46280d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.689+0000 7f4627e00640 1 -- 172.21.15.114:0/4109228775 shutdown_connections 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.689+0000 7f4627e00640 1 -- 172.21.15.114:0/4109228775 wait complete. 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.691+0000 7f4627e00640 1 Processor -- start 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.691+0000 7f4627e00640 1 -- start start 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f460c0039b0 0x7f46280ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46281523f0 0x7f462814f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4628152b30 0x7f46280d1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280cfd60 con 0x7f4628152b30 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280cfa80 con 0x7f460c0039b0 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46280b4650 con 0x7f46281523f0 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4628152b30 0x7f46280d1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4628152b30 0x7f46280d1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49584/0 (socket says 172.21.15.114:49584) 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.692+0000 7f4627400640 1 -- 172.21.15.114:0/2518776842 learned_addr learned my addr 172.21.15.114:0/2518776842 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f461fe00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46281523f0 0x7f462814f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627400640 1 -- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46281523f0 msgr2=0x7f462814f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627400640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46281523f0 0x7f462814f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627400640 1 -- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f460c0039b0 msgr2=0x7f46280ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627400640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f460c0039b0 0x7f46280ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627400640 1 -- 172.21.15.114:0/2518776842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280c9220 con 0x7f4628152b30 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f461fe00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46281523f0 0x7f462814f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627400640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4628152b30 0x7f46280d1f10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f462815e9c0 tx=0x7f462000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f461ea00640 1 -- 172.21.15.114:0/2518776842 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462000bb80 con 0x7f4628152b30 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f461ea00640 1 -- 172.21.15.114:0/2518776842 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462000bd20 con 0x7f4628152b30 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46280c9500 con 0x7f4628152b30 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f461ea00640 1 -- 172.21.15.114:0/2518776842 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620007350 con 0x7f4628152b30 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.693+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46280c9ac0 con 0x7f4628152b30 2024-03-06T07:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.695+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462804f330 con 0x7f4628152b30 2024-03-06T07:28:08.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.696+0000 7f461ea00640 1 -- 172.21.15.114:0/2518776842 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f462001b020 con 0x7f4628152b30 2024-03-06T07:28:08.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.696+0000 7f461ea00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f460406b500 0x7f460406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.697+0000 7f461ea00640 1 -- 172.21.15.114:0/2518776842 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f462008a430 con 0x7f4628152b30 2024-03-06T07:28:08.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.697+0000 7f4626a00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f460406b500 0x7f460406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.697+0000 7f4626a00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f460406b500 0x7f460406d9c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4618036000 tx=0x7f461802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.701+0000 7f461ea00640 1 -- 172.21.15.114:0/2518776842 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4620059340 con 0x7f4628152b30 2024-03-06T07:28:08.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.896+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46280d4790 con 0x7f4628152b30 2024-03-06T07:28:08.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.898+0000 7f461ea00640 1 -- 172.21.15.114:0/2518776842 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f4620058c40 con 0x7f4628152b30 2024-03-06T07:28:08.901 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:08.901 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.900+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f460406b500 msgr2=0x7f460406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.900+0000 7f4627e00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f460406b500 0x7f460406d9c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4618036000 tx=0x7f461802e000 comp rx=0 tx=0).stop 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4628152b30 msgr2=0x7f46280d1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4628152b30 0x7f46280d1f10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f462815e9c0 tx=0x7f462000b4a0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 shutdown_connections 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f460406b500 0x7f460406d9c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4628152b30 0x7f46280d1f10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46281523f0 0x7f462814f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 --2- 172.21.15.114:0/2518776842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f460c0039b0 0x7f46280ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 >> 172.21.15.114:0/2518776842 conn(0x7f46280d3d30 msgr2=0x7f46280d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 shutdown_connections 2024-03-06T07:28:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:08.901+0000 7f4627e00640 1 -- 172.21.15.114:0/2518776842 wait complete. 2024-03-06T07:28:08.922 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:08.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:09.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.123+0000 7f94dee00640 1 Processor -- start 2024-03-06T07:28:09.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.124+0000 7f94dee00640 1 -- start start 2024-03-06T07:28:09.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dee00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d815fc10 0x7f94d8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dee00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f94d80b3f70 0x7f94d80ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dee00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d80b46c0 con 0x7f94d815fff0 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dee00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d80b49a0 con 0x7f94d80b3f70 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dee00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d8072c90 con 0x7f94d815fc10 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dce00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f94d80b3f70 0x7f94d80ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dce00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f94d80b3f70 0x7f94d80ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49590/0 (socket says 172.21.15.114:49590) 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.125+0000 7f94dce00640 1 -- 172.21.15.114:0/4139734098 learned_addr learned my addr 172.21.15.114:0/4139734098 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.126+0000 7f94d7e00640 1 --2- 172.21.15.114:0/4139734098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d815fc10 0x7f94d8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.126+0000 7f94dce00640 1 -- 172.21.15.114:0/4139734098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d815fc10 msgr2=0x7f94d8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.126+0000 7f94dce00640 1 --2- 172.21.15.114:0/4139734098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d815fc10 0x7f94d8070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.126+0000 7f94dce00640 1 -- 172.21.15.114:0/4139734098 >> v1:172.21.15.177:6789/0 conn(0x7f94d815fff0 legacy=0x7f94d8160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.126+0000 7f94dce00640 1 -- 172.21.15.114:0/4139734098 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d809dd80 con 0x7f94d80b3f70 2024-03-06T07:28:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.126+0000 7f94d7e00640 1 --2- 172.21.15.114:0/4139734098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d815fc10 0x7f94d8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.127+0000 7f94dce00640 1 --2- 172.21.15.114:0/4139734098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80b3f70 0x7f94d80ce4a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009e50 tx=0x7f94d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6673b049d0a84314 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.127+0000 7f94d6a00640 1 -- 172.21.15.114:0/4139734098 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d002ca70 con 0x7f94d80b3f70 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.127+0000 7f94d6a00640 1 -- 172.21.15.114:0/4139734098 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d002cc10 con 0x7f94d80b3f70 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.127+0000 7f94d6a00640 1 -- 172.21.15.114:0/4139734098 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d00343f0 con 0x7f94d80b3f70 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 -- 172.21.15.114:0/4139734098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80b3f70 msgr2=0x7f94d80ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 --2- 172.21.15.114:0/4139734098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80b3f70 0x7f94d80ce4a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009e50 tx=0x7f94d002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 -- 172.21.15.114:0/4139734098 shutdown_connections 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 --2- 172.21.15.114:0/4139734098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d80b3f70 0x7f94d80ce4a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 --2- 172.21.15.114:0/4139734098 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d815fc10 0x7f94d8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 -- 172.21.15.114:0/4139734098 >> 172.21.15.114:0/4139734098 conn(0x7f94d80b1940 msgr2=0x7f94d80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 -- 172.21.15.114:0/4139734098 shutdown_connections 2024-03-06T07:28:09.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.128+0000 7f94dee00640 1 -- 172.21.15.114:0/4139734098 wait complete. 2024-03-06T07:28:09.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.130+0000 7f94dee00640 1 Processor -- start 2024-03-06T07:28:09.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.130+0000 7f94dee00640 1 -- start start 2024-03-06T07:28:09.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dee00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d80b3f70 0x7f94d806c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d815fc10 0x7f94d806e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d815fff0 0x7f94d814a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d815c460 con 0x7f94d815fff0 2024-03-06T07:28:09.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d815c2e0 con 0x7f94d815fc10 2024-03-06T07:28:09.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dee00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d80ce9e0 con 0x7f94d80b3f70 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d815fff0 0x7f94d814a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d815fff0 0x7f94d814a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49606/0 (socket says 172.21.15.114:49606) 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.132+0000 7f94dce00640 1 -- 172.21.15.114:0/1308669490 learned_addr learned my addr 172.21.15.114:0/1308669490 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94d7e00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d80b3f70 0x7f94d806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94dce00640 1 -- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d80b3f70 msgr2=0x7f94d806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94dce00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d80b3f70 0x7f94d806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94d7400640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d815fc10 0x7f94d806e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94dce00640 1 -- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d815fc10 msgr2=0x7f94d806e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94dce00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d815fc10 0x7f94d806e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94dce00640 1 -- 172.21.15.114:0/1308669490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d806d000 con 0x7f94d815fff0 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94d7e00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d80b3f70 0x7f94d806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94dce00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d815fff0 0x7f94d814a370 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009e50 tx=0x7f94d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94cfe00640 1 -- 172.21.15.114:0/1308669490 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0043020 con 0x7f94d815fff0 2024-03-06T07:28:09.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94cfe00640 1 -- 172.21.15.114:0/1308669490 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d003e450 con 0x7f94d815fff0 2024-03-06T07:28:09.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.133+0000 7f94cfe00640 1 -- 172.21.15.114:0/1308669490 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0009510 con 0x7f94d815fff0 2024-03-06T07:28:09.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.134+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d806d2e0 con 0x7f94d815fff0 2024-03-06T07:28:09.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.134+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d806d8a0 con 0x7f94d815fff0 2024-03-06T07:28:09.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.135+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d809c6c0 con 0x7f94d815fff0 2024-03-06T07:28:09.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.136+0000 7f94cfe00640 1 -- 172.21.15.114:0/1308669490 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94d0046020 con 0x7f94d815fff0 2024-03-06T07:28:09.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.137+0000 7f94cfe00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f94b406b4d0 0x7f94b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.137+0000 7f94cfe00640 1 -- 172.21.15.114:0/1308669490 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94d00aca10 con 0x7f94d815fff0 2024-03-06T07:28:09.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.137+0000 7f94d7e00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f94b406b4d0 0x7f94b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.137+0000 7f94d7e00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f94b406b4d0 0x7f94b406d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006fd0 tx=0x7f94c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.145+0000 7f94cfe00640 1 -- 172.21.15.114:0/1308669490 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94d004d030 con 0x7f94d815fff0 2024-03-06T07:28:09.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.341+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94d804f330 con 0x7f94d815fff0 2024-03-06T07:28:09.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.341+0000 7f94cfe00640 1 -- 172.21.15.114:0/1308669490 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94d007b920 con 0x7f94d815fff0 2024-03-06T07:28:09.344 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f94b406b4d0 msgr2=0x7f94b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f94b406b4d0 0x7f94b406d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006fd0 tx=0x7f94c0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d815fff0 msgr2=0x7f94d814a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d815fff0 0x7f94d814a370 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009e50 tx=0x7f94d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 shutdown_connections 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f94b406b4d0 0x7f94b406d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94d815fff0 0x7f94d814a370 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94d815fc10 0x7f94d806e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 --2- 172.21.15.114:0/1308669490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f94d80b3f70 0x7f94d806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 >> 172.21.15.114:0/1308669490 conn(0x7f94d80b1940 msgr2=0x7f94d80ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 shutdown_connections 2024-03-06T07:28:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.344+0000 7f94dee00640 1 -- 172.21.15.114:0/1308669490 wait complete. 2024-03-06T07:28:09.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:09.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.562+0000 7fe440200640 1 Processor -- start 2024-03-06T07:28:09.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.563+0000 7fe440200640 1 -- start start 2024-03-06T07:28:09.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe440200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe43815fc10 0x7fe438070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe440200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4380b46c0 con 0x7fe43815fff0 2024-03-06T07:28:09.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe440200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4380b49a0 con 0x7fe4380b3f70 2024-03-06T07:28:09.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe440200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe438072c90 con 0x7fe43815fc10 2024-03-06T07:28:09.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe43d800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe43815fc10 0x7fe438070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe43d800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe43815fc10 0x7fe438070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49608/0 (socket says 172.21.15.114:49608) 2024-03-06T07:28:09.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe43d800640 1 -- 172.21.15.114:0/1086587537 learned_addr learned my addr 172.21.15.114:0/1086587537 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:09.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.565+0000 7fe43ce00640 1 --1- 172.21.15.114:0/1086587537 >> v1:172.21.15.114:6789/0 conn(0x7fe43815fff0 0x7fe438160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49658/0 (socket says 172.21.15.114:49658) 2024-03-06T07:28:09.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.566+0000 7fe43d800640 1 -- 172.21.15.114:0/1086587537 >> v1:172.21.15.177:6789/0 conn(0x7fe4380b3f70 legacy=0x7fe4380ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.566+0000 7fe43d800640 1 -- 172.21.15.114:0/1086587537 >> v1:172.21.15.114:6789/0 conn(0x7fe43815fff0 legacy=0x7fe438160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.566+0000 7fe43d800640 1 -- 172.21.15.114:0/1086587537 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43809dd80 con 0x7fe43815fc10 2024-03-06T07:28:09.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.566+0000 7fe437e00640 1 -- 172.21.15.114:0/1086587537 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506276222 0 0) 0x7fe4380b46c0 con 0x7fe43815fff0 2024-03-06T07:28:09.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.567+0000 7fe43d800640 1 --2- 172.21.15.114:0/1086587537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe43815fc10 0x7fe438070ff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009870 tx=0x7fe42c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d11345b94dad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.567+0000 7fe437e00640 1 -- 172.21.15.114:0/1086587537 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c034070 con 0x7fe43815fc10 2024-03-06T07:28:09.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.567+0000 7fe437e00640 1 -- 172.21.15.114:0/1086587537 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c030860 con 0x7fe43815fc10 2024-03-06T07:28:09.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.567+0000 7fe437e00640 1 -- 172.21.15.114:0/1086587537 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c030c30 con 0x7fe43815fc10 2024-03-06T07:28:09.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.568+0000 7fe440200640 1 -- 172.21.15.114:0/1086587537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe43815fc10 msgr2=0x7fe438070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.568+0000 7fe440200640 1 --2- 172.21.15.114:0/1086587537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe43815fc10 0x7fe438070ff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009870 tx=0x7fe42c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.568+0000 7fe440200640 1 -- 172.21.15.114:0/1086587537 shutdown_connections 2024-03-06T07:28:09.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.568+0000 7fe440200640 1 --2- 172.21.15.114:0/1086587537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe43815fc10 0x7fe438070ff0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.568+0000 7fe440200640 1 -- 172.21.15.114:0/1086587537 >> 172.21.15.114:0/1086587537 conn(0x7fe4380b1940 msgr2=0x7fe4380b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.568+0000 7fe440200640 1 -- 172.21.15.114:0/1086587537 shutdown_connections 2024-03-06T07:28:09.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.568+0000 7fe440200640 1 -- 172.21.15.114:0/1086587537 wait complete. 2024-03-06T07:28:09.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.569+0000 7fe440200640 1 Processor -- start 2024-03-06T07:28:09.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.570+0000 7fe440200640 1 -- start start 2024-03-06T07:28:09.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.570+0000 7fe440200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4380b3f70 0x7fe4380acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.570+0000 7fe440200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe43815fc10 0x7fe43806d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.570+0000 7fe440200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe43815fff0 0x7fe4381f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.570+0000 7fe440200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43815c460 con 0x7fe4380b3f70 2024-03-06T07:28:09.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.570+0000 7fe440200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43815c2e0 con 0x7fe43815fc10 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.570+0000 7fe440200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4380cebb0 con 0x7fe43815fff0 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4380b3f70 0x7fe4380acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4380b3f70 0x7fe4380acff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49614/0 (socket says 172.21.15.114:49614) 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 -- 172.21.15.114:0/2638073579 learned_addr learned my addr 172.21.15.114:0/2638073579 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 -- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe43815fff0 msgr2=0x7fe4381f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe43815fff0 0x7fe4381f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43ce00640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe43815fc10 0x7fe43806d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 -- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe43815fc10 msgr2=0x7fe43806d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe43815fc10 0x7fe43806d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 -- 172.21.15.114:0/2638073579 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43806cac0 con 0x7fe4380b3f70 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43ce00640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe43815fc10 0x7fe43806d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.571+0000 7fe43d800640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4380b3f70 0x7fe4380acff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009870 tx=0x7fe42c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.572+0000 7fe435600640 1 -- 172.21.15.114:0/2638073579 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c048070 con 0x7fe4380b3f70 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.572+0000 7fe435600640 1 -- 172.21.15.114:0/2638073579 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c030660 con 0x7fe4380b3f70 2024-03-06T07:28:09.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.572+0000 7fe435600640 1 -- 172.21.15.114:0/2638073579 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c034040 con 0x7fe4380b3f70 2024-03-06T07:28:09.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.572+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43806cda0 con 0x7fe4380b3f70 2024-03-06T07:28:09.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.572+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43806d360 con 0x7fe4380b3f70 2024-03-06T07:28:09.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.573+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43804f330 con 0x7fe4380b3f70 2024-03-06T07:28:09.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.574+0000 7fe435600640 1 -- 172.21.15.114:0/2638073579 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe42c037070 con 0x7fe4380b3f70 2024-03-06T07:28:09.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.575+0000 7fe435600640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe41806b4d0 0x7fe41806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.575+0000 7fe435600640 1 -- 172.21.15.114:0/2638073579 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe42c0ac120 con 0x7fe4380b3f70 2024-03-06T07:28:09.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.575+0000 7fe43ce00640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe41806b4d0 0x7fe41806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.576+0000 7fe43ce00640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe41806b4d0 0x7fe41806d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ad870 tx=0x7fe424009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.579+0000 7fe435600640 1 -- 172.21.15.114:0/2638073579 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe42c07af00 con 0x7fe4380b3f70 2024-03-06T07:28:09.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.778+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe43806ef30 con 0x7fe4380b3f70 2024-03-06T07:28:09.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.779+0000 7fe435600640 1 -- 172.21.15.114:0/2638073579 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe42c07a800 con 0x7fe4380b3f70 2024-03-06T07:28:09.782 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe41806b4d0 msgr2=0x7fe41806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe41806b4d0 0x7fe41806d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe4380ad870 tx=0x7fe424009040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4380b3f70 msgr2=0x7fe4380acff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4380b3f70 0x7fe4380acff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009870 tx=0x7fe42c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 shutdown_connections 2024-03-06T07:28:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe43815fff0 0x7fe4381f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe43815fc10 0x7fe43806d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe41806b4d0 0x7fe41806d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 --2- 172.21.15.114:0/2638073579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4380b3f70 0x7fe4380acff0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 >> 172.21.15.114:0/2638073579 conn(0x7fe4380b1940 msgr2=0x7fe43809ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 shutdown_connections 2024-03-06T07:28:09.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:09.782+0000 7fe440200640 1 -- 172.21.15.114:0/2638073579 wait complete. 2024-03-06T07:28:09.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:10.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.016+0000 7f8d4fe00640 1 Processor -- start 2024-03-06T07:28:10.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.016+0000 7f8d4fe00640 1 -- start start 2024-03-06T07:28:10.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.017+0000 7f8d4fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d5006da40 0x7f8d500ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8d500aef20 0x7f8d500ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5015c1f0 con 0x7f8d5015be10 2024-03-06T07:28:10.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d500cd200 con 0x7f8d5006da40 2024-03-06T07:28:10.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4fe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5006cac0 con 0x7f8d500aef20 2024-03-06T07:28:10.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d5006da40 0x7f8d500ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d5006da40 0x7f8d500ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49626/0 (socket says 172.21.15.114:49626) 2024-03-06T07:28:10.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4e000640 1 -- 172.21.15.114:0/2659626758 learned_addr learned my addr 172.21.15.114:0/2659626758 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:10.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4f400640 1 --2- 172.21.15.114:0/2659626758 >> v2:172.21.15.177:3300/0 conn(0x7f8d500aef20 0x7f8d500ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4e000640 1 -- 172.21.15.114:0/2659626758 >> v2:172.21.15.177:3300/0 conn(0x7f8d500aef20 msgr2=0x7f8d500ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4e000640 1 --2- 172.21.15.114:0/2659626758 >> v2:172.21.15.177:3300/0 conn(0x7f8d500aef20 0x7f8d500ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4e000640 1 -- 172.21.15.114:0/2659626758 >> v1:172.21.15.114:6789/0 conn(0x7f8d5015be10 legacy=0x7f8d5006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4e000640 1 -- 172.21.15.114:0/2659626758 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5006cda0 con 0x7f8d5006da40 2024-03-06T07:28:10.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.018+0000 7f8d4d600640 1 -- 172.21.15.114:0/2659626758 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147046820 0 0) 0x7f8d5015c1f0 con 0x7f8d5015be10 2024-03-06T07:28:10.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.019+0000 7f8d4e000640 1 --2- 172.21.15.114:0/2659626758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5006da40 0x7f8d500ccc20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d500af490 tx=0x7f8d3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fdbe4c07bcafe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.020+0000 7f8d4d600640 1 -- 172.21.15.114:0/2659626758 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c035020 con 0x7f8d5006da40 2024-03-06T07:28:10.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.020+0000 7f8d4d600640 1 -- 172.21.15.114:0/2659626758 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c031870 con 0x7f8d5006da40 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.020+0000 7f8d4d600640 1 -- 172.21.15.114:0/2659626758 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c031c40 con 0x7f8d5006da40 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.020+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2659626758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5006da40 msgr2=0x7f8d500ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.020+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2659626758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5006da40 0x7f8d500ccc20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d500af490 tx=0x7f8d3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.021+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2659626758 shutdown_connections 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.021+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2659626758 >> v2:172.21.15.177:3300/0 conn(0x7f8d500aef20 0x7f8d500ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.021+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2659626758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d5006da40 0x7f8d500ccc20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.021+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2659626758 >> 172.21.15.114:0/2659626758 conn(0x7f8d5006afc0 msgr2=0x7f8d5006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.021+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2659626758 shutdown_connections 2024-03-06T07:28:10.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.021+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2659626758 wait complete. 2024-03-06T07:28:10.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.023+0000 7f8d4fe00640 1 Processor -- start 2024-03-06T07:28:10.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.023+0000 7f8d4fe00640 1 -- start start 2024-03-06T07:28:10.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d5006da40 0x7f8d5014b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d500aef20 0x7f8d50159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5015be10 0x7f8d5015ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d500cda90 con 0x7f8d500aef20 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5006d080 con 0x7f8d5015be10 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50071ab0 con 0x7f8d5006da40 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d500aef20 0x7f8d50159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d500aef20 0x7f8d50159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49628/0 (socket says 172.21.15.114:49628) 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4e000640 1 -- 172.21.15.114:0/2427313008 learned_addr learned my addr 172.21.15.114:0/2427313008 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.024+0000 7f8d4ea00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d5006da40 0x7f8d5014b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4f400640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5015be10 0x7f8d5015ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4e000640 1 -- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d5006da40 msgr2=0x7f8d5014b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4e000640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d5006da40 0x7f8d5014b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4e000640 1 -- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5015be10 msgr2=0x7f8d5015ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4e000640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5015be10 0x7f8d5015ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4e000640 1 -- 172.21.15.114:0/2427313008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5014f5d0 con 0x7f8d500aef20 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4f400640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5015be10 0x7f8d5015ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:10.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4e000640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d500aef20 0x7f8d50159400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c000c00 tx=0x7f8d3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d46a00640 1 -- 172.21.15.114:0/2427313008 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c041070 con 0x7f8d500aef20 2024-03-06T07:28:10.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d46a00640 1 -- 172.21.15.114:0/2427313008 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c004030 con 0x7f8d500aef20 2024-03-06T07:28:10.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5014f8b0 con 0x7f8d500aef20 2024-03-06T07:28:10.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.025+0000 7f8d46a00640 1 -- 172.21.15.114:0/2427313008 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c035040 con 0x7f8d500aef20 2024-03-06T07:28:10.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.026+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5014be80 con 0x7f8d500aef20 2024-03-06T07:28:10.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.027+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5014fa80 con 0x7f8d500aef20 2024-03-06T07:28:10.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.028+0000 7f8d46a00640 1 -- 172.21.15.114:0/2427313008 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d3c038020 con 0x7f8d500aef20 2024-03-06T07:28:10.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.029+0000 7f8d46a00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8d3006b4d0 0x7f8d3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.029+0000 7f8d46a00640 1 -- 172.21.15.114:0/2427313008 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d3c0ac390 con 0x7f8d500aef20 2024-03-06T07:28:10.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.029+0000 7f8d4ea00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8d3006b4d0 0x7f8d3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.030+0000 7f8d4ea00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8d3006b4d0 0x7f8d3006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000f60 tx=0x7f8d40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.034+0000 7f8d46a00640 1 -- 172.21.15.114:0/2427313008 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d3c043350 con 0x7f8d500aef20 2024-03-06T07:28:10.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.304+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d5004f330 con 0x7f8d500aef20 2024-03-06T07:28:10.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.304+0000 7f8d46a00640 1 -- 172.21.15.114:0/2427313008 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d3c07b170 con 0x7f8d500aef20 2024-03-06T07:28:10.307 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8d3006b4d0 msgr2=0x7f8d3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8d3006b4d0 0x7f8d3006d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000f60 tx=0x7f8d40009040 comp rx=0 tx=0).stop 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d500aef20 msgr2=0x7f8d50159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d500aef20 0x7f8d50159400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c000c00 tx=0x7f8d3c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 shutdown_connections 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5015be10 0x7f8d5015ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8d3006b4d0 0x7f8d3006d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d500aef20 0x7f8d50159400 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 --2- 172.21.15.114:0/2427313008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d5006da40 0x7f8d5014b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 >> 172.21.15.114:0/2427313008 conn(0x7f8d5006afc0 msgr2=0x7f8d5015f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 shutdown_connections 2024-03-06T07:28:10.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.307+0000 7f8d4fe00640 1 -- 172.21.15.114:0/2427313008 wait complete. 2024-03-06T07:28:10.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.529+0000 7fdb2c000640 1 Processor -- start 2024-03-06T07:28:10.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.529+0000 7fdb2c000640 1 -- start start 2024-03-06T07:28:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.531+0000 7fdb2c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 0x7fdb240cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.531+0000 7fdb2c000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240cdc50 con 0x7fdb240cd240 2024-03-06T07:28:10.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.531+0000 7fdb2c000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240cd6f0 con 0x7fdb2415bfa0 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.531+0000 7fdb2c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240b5240 con 0x7fdb240cec90 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.531+0000 7fdb2a000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdb240cd240 0x7fdb240b56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49682/0 (socket says 172.21.15.114:49682) 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.531+0000 7fdb2a000640 1 -- 172.21.15.114:0/3959883874 learned_addr learned my addr 172.21.15.114:0/3959883874 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.531+0000 7fdb28c00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 0x7fdb240cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070084433 0 0) 0x7fdb240cdc50 con 0x7fdb240cd240 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb0c003520 con 0x7fdb240cd240 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589614532 0 0) 0x7fdb0c003520 con 0x7fdb240cd240 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 msgr2=0x7fdb240cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 0x7fdb240cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 >> v1:172.21.15.177:6789/0 conn(0x7fdb2415bfa0 legacy=0x7fdb240ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240cd970 con 0x7fdb240cd240 2024-03-06T07:28:10.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fdb1c0019f0 con 0x7fdb240cd240 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 >> v1:172.21.15.114:6789/0 conn(0x7fdb240cd240 legacy=0x7fdb240b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb0c0039e0 0x7fdb0c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb0c004800 0x7fdb0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 0x7fdb0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240b5240 con 0x7fdb0c004800 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c00baa0 con 0x7fdb0c0039e0 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c00bca0 con 0x7fdb240cec90 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.532+0000 7fdb28c00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb0c004800 0x7fdb0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb2a000640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 0x7fdb0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb28c00640 1 -- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 msgr2=0x7fdb0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb28c00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 0x7fdb0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb29600640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb0c0039e0 0x7fdb0c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb28c00640 1 -- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb0c0039e0 msgr2=0x7fdb0c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb28c00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb0c0039e0 0x7fdb0c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb28c00640 1 -- 172.21.15.114:0/3959883874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c003520 con 0x7fdb0c004800 2024-03-06T07:28:10.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.533+0000 7fdb2a000640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb240cec90 0x7fdb0c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.534+0000 7fdb28c00640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb0c004800 0x7fdb0c004c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009a90 tx=0x7fdb0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef3f38ce8264138 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.534+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10039070 con 0x7fdb0c004800 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.534+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1002c680 con 0x7fdb0c004800 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.534+0000 7fdb23e00640 1 -- 172.21.15.114:0/3959883874 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10009690 con 0x7fdb0c004800 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.535+0000 7fdb2c000640 1 -- 172.21.15.114:0/3959883874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb0c004800 msgr2=0x7fdb0c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.535+0000 7fdb2c000640 1 --2- 172.21.15.114:0/3959883874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb0c004800 0x7fdb0c004c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009a90 tx=0x7fdb0c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.535+0000 7fdb2a000640 1 -- 172.21.15.114:0/3959883874 reap_dead start 2024-03-06T07:28:10.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.535+0000 7fdb2c000640 1 -- 172.21.15.114:0/3959883874 shutdown_connections 2024-03-06T07:28:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.535+0000 7fdb2c000640 1 -- 172.21.15.114:0/3959883874 >> 172.21.15.114:0/3959883874 conn(0x7fdb240d1c90 msgr2=0x7fdb240d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.535+0000 7fdb2c000640 1 -- 172.21.15.114:0/3959883874 shutdown_connections 2024-03-06T07:28:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.535+0000 7fdb2c000640 1 -- 172.21.15.114:0/3959883874 wait complete. 2024-03-06T07:28:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.537+0000 7fdb2c000640 1 Processor -- start 2024-03-06T07:28:10.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.537+0000 7fdb2c000640 1 -- start start 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb2c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb0c004800 0x7fdb24150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb2c000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb240cb5e0 0x7fdb2406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb2c000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb240cba60 0x7fdb2406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb2c000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2415c380 con 0x7fdb240cb5e0 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb2c000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240b4ed0 con 0x7fdb240cba60 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb2c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240cf190 con 0x7fdb0c004800 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb240cb5e0 0x7fdb2406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb240cb5e0 0x7fdb2406ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49638/0 (socket says 172.21.15.114:49638) 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 -- 172.21.15.114:0/2580997025 learned_addr learned my addr 172.21.15.114:0/2580997025 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:10.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb29600640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb0c004800 0x7fdb24150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb2a000640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb240cba60 0x7fdb2406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 -- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb0c004800 msgr2=0x7fdb24150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb0c004800 0x7fdb24150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 -- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb240cba60 msgr2=0x7fdb2406f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb240cba60 0x7fdb2406f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.538+0000 7fdb28c00640 1 -- 172.21.15.114:0/2580997025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2415f180 con 0x7fdb240cb5e0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.539+0000 7fdb28c00640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb240cb5e0 0x7fdb2406ed20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009b80 tx=0x7fdb10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.539+0000 7fdb21600640 1 -- 172.21.15.114:0/2580997025 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10039050 con 0x7fdb240cb5e0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.539+0000 7fdb21600640 1 -- 172.21.15.114:0/2580997025 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb10045070 con 0x7fdb240cb5e0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.539+0000 7fdb21600640 1 -- 172.21.15.114:0/2580997025 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb10003a50 con 0x7fdb240cb5e0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.539+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2415f460 con 0x7fdb240cb5e0 2024-03-06T07:28:10.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.539+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb24142360 con 0x7fdb240cb5e0 2024-03-06T07:28:10.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.541+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2404f330 con 0x7fdb240cb5e0 2024-03-06T07:28:10.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.541+0000 7fdb21600640 1 -- 172.21.15.114:0/2580997025 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb10003bf0 con 0x7fdb240cb5e0 2024-03-06T07:28:10.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.542+0000 7fdb21600640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0406b500 0x7fdb0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.542+0000 7fdb21600640 1 -- 172.21.15.114:0/2580997025 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb100abd90 con 0x7fdb240cb5e0 2024-03-06T07:28:10.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.542+0000 7fdb29600640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0406b500 0x7fdb0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.543+0000 7fdb29600640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0406b500 0x7fdb0406d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb18000c00 tx=0x7fdb18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.546+0000 7fdb21600640 1 -- 172.21.15.114:0/2580997025 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb1004e020 con 0x7fdb240cb5e0 2024-03-06T07:28:10.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.779+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb240684a0 con 0x7fdb240cb5e0 2024-03-06T07:28:10.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.780+0000 7fdb21600640 1 -- 172.21.15.114:0/2580997025 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7fdb1007acc0 con 0x7fdb240cb5e0 2024-03-06T07:28:10.783 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:10.783 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:10.784 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0406b500 msgr2=0x7fdb0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0406b500 0x7fdb0406d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdb18000c00 tx=0x7fdb18009040 comp rx=0 tx=0).stop 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb240cb5e0 msgr2=0x7fdb2406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb240cb5e0 0x7fdb2406ed20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb10009b80 tx=0x7fdb10004730 comp rx=0 tx=0).stop 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 shutdown_connections 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb240cba60 0x7fdb2406f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0406b500 0x7fdb0406d9c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb240cb5e0 0x7fdb2406ed20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 --2- 172.21.15.114:0/2580997025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb0c004800 0x7fdb24150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 >> 172.21.15.114:0/2580997025 conn(0x7fdb240d1c90 msgr2=0x7fdb240b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 shutdown_connections 2024-03-06T07:28:10.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.783+0000 7fdb2c000640 1 -- 172.21.15.114:0/2580997025 wait complete. 2024-03-06T07:28:10.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:10.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.996+0000 7f5669400640 1 Processor -- start 2024-03-06T07:28:10.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.997+0000 7f5669400640 1 -- start start 2024-03-06T07:28:11.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.998+0000 7f5669400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f566415f9d0 0x7f566415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.998+0000 7f5669400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5664070b10 0x7f5664070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.998+0000 7f5669400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566406cef0 con 0x7f5664144c00 2024-03-06T07:28:11.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.998+0000 7f5669400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566406d1d0 con 0x7f566415f9d0 2024-03-06T07:28:11.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.998+0000 7f5669400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566406d4b0 con 0x7f5664070b10 2024-03-06T07:28:11.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5663200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5664070b10 0x7f5664070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5663200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5664070b10 0x7f5664070f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:39860/0 (socket says 172.21.15.114:39860) 2024-03-06T07:28:11.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5663200640 1 -- 172.21.15.114:0/2403931754 learned_addr learned my addr 172.21.15.114:0/2403931754 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:11.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5661e00640 1 --2- 172.21.15.114:0/2403931754 >> v2:172.21.15.177:3300/0 conn(0x7f566415f9d0 0x7f566415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5663200640 1 -- 172.21.15.114:0/2403931754 >> v2:172.21.15.177:3300/0 conn(0x7f566415f9d0 msgr2=0x7f566415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5663200640 1 --2- 172.21.15.114:0/2403931754 >> v2:172.21.15.177:3300/0 conn(0x7f566415f9d0 0x7f566415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5663200640 1 -- 172.21.15.114:0/2403931754 >> v1:172.21.15.177:6789/0 conn(0x7f5664144c00 legacy=0x7f566406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:10.999+0000 7f5663200640 1 -- 172.21.15.114:0/2403931754 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566406d790 con 0x7f5664070b10 2024-03-06T07:28:11.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.001+0000 7f5663200640 1 --2- 172.21.15.114:0/2403931754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5664070b10 0x7f5664070f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f565c009e50 tx=0x7f565c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=515288163d803405 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.001+0000 7f5661400640 1 -- 172.21.15.114:0/2403931754 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c02cb00 con 0x7f5664070b10 2024-03-06T07:28:11.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.001+0000 7f5661400640 1 -- 172.21.15.114:0/2403931754 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c02cca0 con 0x7f5664070b10 2024-03-06T07:28:11.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.001+0000 7f5661400640 1 -- 172.21.15.114:0/2403931754 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c034610 con 0x7f5664070b10 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 -- 172.21.15.114:0/2403931754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5664070b10 msgr2=0x7f5664070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 --2- 172.21.15.114:0/2403931754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5664070b10 0x7f5664070f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f565c009e50 tx=0x7f565c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 -- 172.21.15.114:0/2403931754 shutdown_connections 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 --2- 172.21.15.114:0/2403931754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5664070b10 0x7f5664070f40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 --2- 172.21.15.114:0/2403931754 >> v2:172.21.15.177:3300/0 conn(0x7f566415f9d0 0x7f566415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 -- 172.21.15.114:0/2403931754 >> 172.21.15.114:0/2403931754 conn(0x7f56640b1730 msgr2=0x7f56640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 -- 172.21.15.114:0/2403931754 shutdown_connections 2024-03-06T07:28:11.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.002+0000 7f5669400640 1 -- 172.21.15.114:0/2403931754 wait complete. 2024-03-06T07:28:11.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.004+0000 7f5669400640 1 Processor -- start 2024-03-06T07:28:11.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.004+0000 7f5669400640 1 -- start start 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5669400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5664070b10 0x7f56640ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5669400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5664144c00 0x7f56641f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5669400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f566415f9d0 0x7f5664142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5669400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566406ccd0 con 0x7f5664070b10 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5669400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566406cb50 con 0x7f5664144c00 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5669400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566406df90 con 0x7f566415f9d0 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5664070b10 0x7f56640ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5664070b10 0x7f56640ae870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49650/0 (socket says 172.21.15.114:49650) 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 -- 172.21.15.114:0/1667735199 learned_addr learned my addr 172.21.15.114:0/1667735199 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5661e00640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5664144c00 0x7f56641f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5663200640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f566415f9d0 0x7f5664142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 -- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f566415f9d0 msgr2=0x7f5664142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f566415f9d0 0x7f5664142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 -- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5664144c00 msgr2=0x7f56641f70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5664144c00 0x7f56641f70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.005+0000 7f5662800640 1 -- 172.21.15.114:0/1667735199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56641f7a90 con 0x7f5664070b10 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.006+0000 7f5662800640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5664070b10 0x7f56640ae870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f565400c450 tx=0x7f565400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.006+0000 7f565aa00640 1 -- 172.21.15.114:0/1667735199 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56540052d0 con 0x7f5664070b10 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.006+0000 7f565aa00640 1 -- 172.21.15.114:0/1667735199 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5654005470 con 0x7f5664070b10 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.006+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56641f7d70 con 0x7f5664070b10 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.006+0000 7f565aa00640 1 -- 172.21.15.114:0/1667735199 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654012870 con 0x7f5664070b10 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.006+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56640aedb0 con 0x7f5664070b10 2024-03-06T07:28:11.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.008+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566404f330 con 0x7f5664070b10 2024-03-06T07:28:11.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.008+0000 7f565aa00640 1 -- 172.21.15.114:0/1667735199 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5654012a10 con 0x7f5664070b10 2024-03-06T07:28:11.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.009+0000 7f565aa00640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f564406b4d0 0x7f564406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.009+0000 7f565aa00640 1 -- 172.21.15.114:0/1667735199 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5654089f50 con 0x7f5664070b10 2024-03-06T07:28:11.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.009+0000 7f5661e00640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f564406b4d0 0x7f564406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.010+0000 7f5661e00640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f564406b4d0 0x7f564406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5664150fd0 tx=0x7f5650008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.015+0000 7f565aa00640 1 -- 172.21.15.114:0/1667735199 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5654058d30 con 0x7f5664070b10 2024-03-06T07:28:11.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.220+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56641f7f40 con 0x7f5664070b10 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.222+0000 7f565aa00640 1 -- 172.21.15.114:0/1667735199 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f5654058630 con 0x7f5664070b10 2024-03-06T07:28:11.225 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:11.226 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f564406b4d0 msgr2=0x7f564406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f564406b4d0 0x7f564406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5664150fd0 tx=0x7f5650008040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5664070b10 msgr2=0x7f56640ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5664070b10 0x7f56640ae870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f565400c450 tx=0x7f565400c910 comp rx=0 tx=0).stop 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 shutdown_connections 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f566415f9d0 0x7f5664142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5664144c00 0x7f56641f70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f564406b4d0 0x7f564406d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 --2- 172.21.15.114:0/1667735199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5664070b10 0x7f56640ae870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 >> 172.21.15.114:0/1667735199 conn(0x7f56640b1730 msgr2=0x7f566415eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 shutdown_connections 2024-03-06T07:28:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.225+0000 7f5669400640 1 -- 172.21.15.114:0/1667735199 wait complete. 2024-03-06T07:28:11.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:11.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.435+0000 7f682ae00640 1 Processor -- start 2024-03-06T07:28:11.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.435+0000 7f682ae00640 1 -- start start 2024-03-06T07:28:11.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.436+0000 7f682ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f682415f9d0 0x7f682415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.436+0000 7f682ae00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682406cef0 con 0x7f6824070b10 2024-03-06T07:28:11.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.437+0000 7f682ae00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682406d1d0 con 0x7f6824144c00 2024-03-06T07:28:11.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.437+0000 7f682ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6824160370 con 0x7f682415f9d0 2024-03-06T07:28:11.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.437+0000 7f6828e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6824070b10 0x7f6824070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49690/0 (socket says 172.21.15.114:49690) 2024-03-06T07:28:11.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.437+0000 7f6828e00640 1 -- 172.21.15.114:0/2722821991 learned_addr learned my addr 172.21.15.114:0/2722821991 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:11.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.437+0000 7f681b400640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f682415f9d0 0x7f682415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.437+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757281490 0 0) 0x7f682406cef0 con 0x7f6824070b10 2024-03-06T07:28:11.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6808003520 con 0x7f6824070b10 2024-03-06T07:28:11.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938132388 0 0) 0x7f682406d1d0 con 0x7f6824144c00 2024-03-06T07:28:11.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f682406cef0 con 0x7f6824144c00 2024-03-06T07:28:11.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578892214 0 0) 0x7f6808003520 con 0x7f6824070b10 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f682415f9d0 msgr2=0x7f682415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f682415f9d0 0x7f682415fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 >> v1:172.21.15.177:6789/0 conn(0x7f6824144c00 legacy=0x7f682406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682406d4b0 con 0x7f6824070b10 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f681c001670 con 0x7f6824070b10 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 >> v1:172.21.15.114:6789/0 conn(0x7f6824070b10 legacy=0x7f6824070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6808003b70 0x7f6808003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6808004800 0x7f6808004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68080051c0 0x7f680800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680800bfb0 con 0x7f68080051c0 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680800c130 con 0x7f6808003b70 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6808010740 con 0x7f6808004800 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f681b400640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f682415f9d0 0x7f682415fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:11.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.438+0000 7f6828e00640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68080051c0 0x7f680800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f6823e00640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6808003b70 0x7f6808003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f681b400640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6808004800 0x7f6808004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f6828e00640 1 -- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6808004800 msgr2=0x7f6808004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f6828e00640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6808004800 0x7f6808004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f6828e00640 1 -- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6808003b70 msgr2=0x7f6808003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f6828e00640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6808003b70 0x7f6808003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f6828e00640 1 -- 172.21.15.114:0/2722821991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6808003520 con 0x7f68080051c0 2024-03-06T07:28:11.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.439+0000 7f6823e00640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6808003b70 0x7f6808003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:11.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.441+0000 7f6828e00640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68080051c0 0x7f680800b9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c190 tx=0x7f6808005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9c712aff258aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.441+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c007ce0 con 0x7f68080051c0 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.441+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c030070 con 0x7f68080051c0 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.441+0000 7f6823400640 1 -- 172.21.15.114:0/2722821991 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c005d50 con 0x7f68080051c0 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.442+0000 7f682ae00640 1 -- 172.21.15.114:0/2722821991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68080051c0 msgr2=0x7f680800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.442+0000 7f682ae00640 1 --2- 172.21.15.114:0/2722821991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68080051c0 0x7f680800b9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c190 tx=0x7f6808005830 comp rx=0 tx=0).stop 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.442+0000 7f6828e00640 1 -- 172.21.15.114:0/2722821991 reap_dead start 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.442+0000 7f682ae00640 1 -- 172.21.15.114:0/2722821991 shutdown_connections 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.442+0000 7f682ae00640 1 -- 172.21.15.114:0/2722821991 >> 172.21.15.114:0/2722821991 conn(0x7f68240b1730 msgr2=0x7f68240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.442+0000 7f682ae00640 1 -- 172.21.15.114:0/2722821991 shutdown_connections 2024-03-06T07:28:11.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.442+0000 7f682ae00640 1 -- 172.21.15.114:0/2722821991 wait complete. 2024-03-06T07:28:11.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.444+0000 7f682ae00640 1 Processor -- start 2024-03-06T07:28:11.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.444+0000 7f682ae00640 1 -- start start 2024-03-06T07:28:11.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f682ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080051c0 0x7f68241414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f682ae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824142f90 0x7f68240ab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f682ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68240a9450 0x7f68240ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f682ae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682406cb50 con 0x7f6824142f90 2024-03-06T07:28:11.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f682ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824160730 con 0x7f68240a9450 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f682ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682406ccd0 con 0x7f68080051c0 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f681b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824142f90 0x7f68240ab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f681b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824142f90 0x7f68240ab7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49670/0 (socket says 172.21.15.114:49670) 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.445+0000 7f681b400640 1 -- 172.21.15.114:0/2600178259 learned_addr learned my addr 172.21.15.114:0/2600178259 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f6823e00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080051c0 0x7f68241414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f6828e00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68240a9450 0x7f68240ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f681b400640 1 -- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080051c0 msgr2=0x7f68241414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f681b400640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080051c0 0x7f68241414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f681b400640 1 -- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68240a9450 msgr2=0x7f68240ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f681b400640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68240a9450 0x7f68240ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f681b400640 1 -- 172.21.15.114:0/2600178259 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240abd30 con 0x7f6824142f90 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f6828e00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68240a9450 0x7f68240ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f681b400640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824142f90 0x7f68240ab7f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f680c011830 tx=0x7f680c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f6820c00640 1 -- 172.21.15.114:0/2600178259 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c004220 con 0x7f6824142f90 2024-03-06T07:28:11.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f6820c00640 1 -- 172.21.15.114:0/2600178259 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c0043c0 con 0x7f6824142f90 2024-03-06T07:28:11.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68240acc30 con 0x7f6824142f90 2024-03-06T07:28:11.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.446+0000 7f6820c00640 1 -- 172.21.15.114:0/2600178259 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c00f980 con 0x7f6824142f90 2024-03-06T07:28:11.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.447+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6824141ad0 con 0x7f6824142f90 2024-03-06T07:28:11.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.448+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682404f330 con 0x7f6824142f90 2024-03-06T07:28:11.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.449+0000 7f6820c00640 1 -- 172.21.15.114:0/2600178259 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f680c00fb20 con 0x7f6824142f90 2024-03-06T07:28:11.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.449+0000 7f6820c00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f680006b500 0x7f680006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.450+0000 7f6820c00640 1 -- 172.21.15.114:0/2600178259 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f680c08a540 con 0x7f6824142f90 2024-03-06T07:28:11.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.450+0000 7f6823e00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f680006b500 0x7f680006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.450+0000 7f6823e00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f680006b500 0x7f680006d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f682415ec30 tx=0x7f6814009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.454+0000 7f6820c00640 1 -- 172.21.15.114:0/2600178259 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68240abd30 con 0x7f6824142f90 2024-03-06T07:28:11.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.682+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6824072890 con 0x7f6824142f90 2024-03-06T07:28:11.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.683+0000 7f6820c00640 1 -- 172.21.15.114:0/2600178259 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f680c019030 con 0x7f6824142f90 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:11.686 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f680006b500 msgr2=0x7f680006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f680006b500 0x7f680006d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f682415ec30 tx=0x7f6814009040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824142f90 msgr2=0x7f68240ab7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824142f90 0x7f68240ab7f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f680c011830 tx=0x7f680c011cf0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 shutdown_connections 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68240a9450 0x7f68240ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f680006b500 0x7f680006d9c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6824142f90 0x7f68240ab7f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 --2- 172.21.15.114:0/2600178259 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f68080051c0 0x7f68241414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 >> 172.21.15.114:0/2600178259 conn(0x7f68240b1730 msgr2=0x7f68240afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 shutdown_connections 2024-03-06T07:28:11.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:11.686+0000 7f682ae00640 1 -- 172.21.15.114:0/2600178259 wait complete. 2024-03-06T07:28:12.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:54.463887+0000', 'modified': '2024-03-06T07:27:57.520027+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4260}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4260': {'gid': 4260, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/3774250012', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 3774250012}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 3774250012}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4260, 'qdb_cluster': [4260]} 2024-03-06T07:28:12.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:12.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:28:12.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.318+0000 7f089a000640 1 Processor -- start 2024-03-06T07:28:12.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.318+0000 7f089a000640 1 -- start start 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.319+0000 7f089a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08940c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.319+0000 7f089a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08940c4d10 0x7f08940c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.319+0000 7f089a000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0894145b30 con 0x7f08940c56b0 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.319+0000 7f089a000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0894142ce0 con 0x7f08940c4d10 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.319+0000 7f089a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08940cbca0 con 0x7f08940c4930 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0892a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08940c4d10 0x7f08940c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0893e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f08940c56b0 0x7f089406d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49696/0 (socket says 172.21.15.114:49696) 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0893e00640 1 -- 172.21.15.114:0/1210715179 learned_addr learned my addr 172.21.15.114:0/1210715179 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:12.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0893400640 1 --2- 172.21.15.114:0/1210715179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08940c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0892000640 1 -- 172.21.15.114:0/1210715179 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522689818 0 0) 0x7f088c001490 con 0x7f08940c56b0 2024-03-06T07:28:12.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0892000640 1 -- 172.21.15.114:0/1210715179 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0878003520 con 0x7f08940c56b0 2024-03-06T07:28:12.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0892a00640 1 -- 172.21.15.114:0/1210715179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 msgr2=0x7f08940c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0892a00640 1 --2- 172.21.15.114:0/1210715179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08940c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0892a00640 1 -- 172.21.15.114:0/1210715179 >> v1:172.21.15.114:6789/0 conn(0x7f08940c56b0 legacy=0x7f089406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.320+0000 7f0892a00640 1 -- 172.21.15.114:0/1210715179 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08940cbf80 con 0x7f08940c4d10 2024-03-06T07:28:12.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.321+0000 7f0892a00640 1 --2- 172.21.15.114:0/1210715179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c4d10 0x7f08940c5170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08800097a0 tx=0x7f088002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3b1b63fe1c7144 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f0892000640 1 -- 172.21.15.114:0/1210715179 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800315a0 con 0x7f08940c4d10 2024-03-06T07:28:12.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f0892000640 1 -- 172.21.15.114:0/1210715179 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0880037070 con 0x7f08940c4d10 2024-03-06T07:28:12.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f0892000640 1 -- 172.21.15.114:0/1210715179 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880031960 con 0x7f08940c4d10 2024-03-06T07:28:12.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 -- 172.21.15.114:0/1210715179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c4d10 msgr2=0x7f08940c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 --2- 172.21.15.114:0/1210715179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c4d10 0x7f08940c5170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08800097a0 tx=0x7f088002b840 comp rx=0 tx=0).stop 2024-03-06T07:28:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 -- 172.21.15.114:0/1210715179 shutdown_connections 2024-03-06T07:28:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 --2- 172.21.15.114:0/1210715179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c4d10 0x7f08940c5170 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 --2- 172.21.15.114:0/1210715179 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08940c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 -- 172.21.15.114:0/1210715179 >> 172.21.15.114:0/1210715179 conn(0x7f08940cc4f0 msgr2=0x7f08940cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 -- 172.21.15.114:0/1210715179 shutdown_connections 2024-03-06T07:28:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.322+0000 7f089a000640 1 -- 172.21.15.114:0/1210715179 wait complete. 2024-03-06T07:28:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.324+0000 7f089a000640 1 Processor -- start 2024-03-06T07:28:12.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.324+0000 7f089a000640 1 -- start start 2024-03-06T07:28:12.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f089a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08941f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f089a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08940c4d10 0x7f08940cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f089a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c56b0 0x7f089413dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f089a000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940cc260 con 0x7f08940c56b0 2024-03-06T07:28:12.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f089a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08940ca3b0 con 0x7f08940c4d10 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f089a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08940c5b40 con 0x7f08940c4930 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c56b0 0x7f089413dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c56b0 0x7f089413dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49696/0 (socket says 172.21.15.114:49696) 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 -- 172.21.15.114:0/466651202 learned_addr learned my addr 172.21.15.114:0/466651202 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893400640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08941f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0892a00640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08940c4d10 0x7f08940cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 -- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 msgr2=0x7f08941f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08941f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 -- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08940c4d10 msgr2=0x7f08940cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08940c4d10 0x7f08940cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 -- 172.21.15.114:0/466651202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08941402c0 con 0x7f08940c56b0 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893400640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08941f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f0893e00640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c56b0 0x7f089413dbb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f088c00cad0 tx=0x7f088c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f088b400640 1 -- 172.21.15.114:0/466651202 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c017070 con 0x7f08940c56b0 2024-03-06T07:28:12.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f088b400640 1 -- 172.21.15.114:0/466651202 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c00abd0 con 0x7f08940c56b0 2024-03-06T07:28:12.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.326+0000 7f088b400640 1 -- 172.21.15.114:0/466651202 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c012400 con 0x7f08940c56b0 2024-03-06T07:28:12.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.327+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08940ce750 con 0x7f08940c56b0 2024-03-06T07:28:12.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.327+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089413eb60 con 0x7f08940c56b0 2024-03-06T07:28:12.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.329+0000 7f088b400640 1 -- 172.21.15.114:0/466651202 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f088c01a070 con 0x7f08940c56b0 2024-03-06T07:28:12.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.329+0000 7f088b400640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f087406b4d0 0x7f087406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.330+0000 7f088b400640 1 -- 172.21.15.114:0/466651202 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f088c089f00 con 0x7f08940c56b0 2024-03-06T07:28:12.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.330+0000 7f0893400640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f087406b4d0 0x7f087406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.330+0000 7f0893400640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f087406b4d0 0x7f087406d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0884006fd0 tx=0x7f0884008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.331+0000 7f089cc56740 1 Processor -- start 2024-03-06T07:28:12.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.331+0000 7f089cc56740 1 -- start start 2024-03-06T07:28:12.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.331+0000 7f089cc56740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed1666750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.331+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 0x55aed1696540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.331+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed1666c90 0x55aed1696a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f089cc56740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed15d0600 con 0x55aed16937b0 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed1579930 con 0x55aed1666c90 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aed15cc6a0 con 0x55aed1693b90 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed1666750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed1666750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49702/0 (socket says 172.21.15.114:49702) 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 -- 172.21.15.114:0/1193770580 learned_addr learned my addr 172.21.15.114:0/1193770580 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0892a00640 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 0x55aed1696540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 -- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 msgr2=0x55aed1696540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 0x55aed1696540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 -- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed1666c90 msgr2=0x55aed1696a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893e00640 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed1666c90 0x55aed1696a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed1666c90 0x55aed1696a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.332+0000 7f0893400640 1 -- 172.21.15.114:0/1193770580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aed12e52d0 con 0x55aed16937b0 2024-03-06T07:28:12.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.333+0000 7f0893400640 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed1666750 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0884004380 tx=0x7f0884011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce97a5d551da406 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.333+0000 7f087f400640 1 -- 172.21.15.114:0/1193770580 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088400f040 con 0x55aed16937b0 2024-03-06T07:28:12.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.333+0000 7f087f400640 1 -- 172.21.15.114:0/1193770580 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08840043b0 con 0x55aed16937b0 2024-03-06T07:28:12.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.333+0000 7f087f400640 1 -- 172.21.15.114:0/1193770580 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088400ee80 con 0x55aed16937b0 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 -- 172.21.15.114:0/1193770580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 msgr2=0x55aed1666750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed1666750 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0884004380 tx=0x7f0884011000 comp rx=0 tx=0).stop 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 -- 172.21.15.114:0/1193770580 shutdown_connections 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed1666c90 0x55aed1696a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 0x55aed1696540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 --2- 172.21.15.114:0/1193770580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed1666750 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 -- 172.21.15.114:0/1193770580 >> 172.21.15.114:0/1193770580 conn(0x55aed16689c0 msgr2=0x55aed16b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 -- 172.21.15.114:0/1193770580 shutdown_connections 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 -- 172.21.15.114:0/1193770580 wait complete. 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.334+0000 7f089cc56740 1 -- start start 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.335+0000 7f089cc56740 1 Processor -- start 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f089cc56740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed16c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 0x55aed168eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed168bc50 0x55aed168f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f089cc56740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed166af60 con 0x55aed16937b0 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed158cd90 con 0x55aed168bc50 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aed166a350 con 0x55aed1693b90 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f089cc56740 10 client.0 _ll_register_callbacks cb 0x55aed15c8350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed16c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed16c21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49718/0 (socket says 172.21.15.114:49718) 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 -- 172.21.15.114:0/368934407 learned_addr learned my addr 172.21.15.114:0/368934407 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 -- 172.21.15.114:0/368934407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 msgr2=0x55aed168eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 0x55aed168eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 -- 172.21.15.114:0/368934407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed168bc50 msgr2=0x55aed168f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed168bc50 0x55aed168f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 -- 172.21.15.114:0/368934407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aed13af6b0 con 0x55aed16937b0 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0893400640 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed16c21a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0884008010 tx=0x7f0884004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453fafaf9b50e4dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.336+0000 7f0873e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f0873e00640 1 -- 172.21.15.114:0/368934407 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884005ba0 con 0x55aed16937b0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f0873e00640 1 -- 172.21.15.114:0/368934407 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0884005d40 con 0x55aed16937b0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f08840043b0 con 0x55aed16937b0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f0873e00640 1 -- 172.21.15.114:0/368934407 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0884013860 con 0x55aed16937b0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 10 client.4650 fetch_fsmap learned FSMap version 8 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55aed168ceb0 con 0x55aed16937b0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f0873e00640 1 -- 172.21.15.114:0/368934407 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08840043b0 con 0x55aed16937b0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f0873e00640 1 -- 172.21.15.114:0/368934407 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0884012070 con 0x55aed16937b0 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 10 client.4650 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 10 client.4650 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 20 client.4650 populate_metadata read hostname 'smithi114' 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed1695a70 0x55aed16d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 4 client.4650 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:12.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.337+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 --> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aed15ddb40 con 0x55aed1695a70 2024-03-06T07:28:12.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.338+0000 7f0892a00640 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed1695a70 0x55aed16d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.338+0000 7f0892a00640 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed1695a70 0x55aed16d01d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb463c831379140 server_cookie=ea0b41dc69f8a172 in_seq=0 out_seq=0 2024-03-06T07:28:12.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.338+0000 7f0873e00640 10 client.4650 ms_handle_connect on v2:172.21.15.114:6835/3774250012 2024-03-06T07:28:12.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.339+0000 7f0873e00640 1 -- 172.21.15.114:0/368934407 <== mds.0 v2:172.21.15.114:6835/3774250012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aed15ddb40 con 0x55aed1695a70 2024-03-06T07:28:12.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.339+0000 7f0873e00640 10 client.4650 handle_command_reply: tid=0 2024-03-06T07:28:12.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.339+0000 7f089cc56740 1 client.4650 shutdown 2024-03-06T07:28:12.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 msgr2=0x55aed16c21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed16c21a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0884008010 tx=0x7f0884004730 comp rx=0 tx=0).stop 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 shutdown_connections 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed1695a70 0x55aed16d01d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed168bc50 0x55aed168f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed1693b90 0x55aed168eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f0873e00640 0 client.4650 ms_handle_reset on v2:172.21.15.114:6835/3774250012 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 --2- 172.21.15.114:0/368934407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed16937b0 0x55aed16c21a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 >> 172.21.15.114:0/368934407 conn(0x55aed1666c90 msgr2=0x55aed16b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 shutdown_connections 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 1 -- 172.21.15.114:0/368934407 wait complete. 2024-03-06T07:28:12.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.340+0000 7f089cc56740 20 client.4650 trim_cache size 0 max 16384 2024-03-06T07:28:12.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.358+0000 7f089cc56740 1 Processor -- start 2024-03-06T07:28:12.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.358+0000 7f089cc56740 1 -- start start 2024-03-06T07:28:12.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed168ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 0x55aed16b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f089cc56740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed16b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f089cc56740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894200310 con 0x55aed1693b90 2024-03-06T07:28:12.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894141a30 con 0x55aed16937b0 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08941e78c0 con 0x55aed168bc50 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed16b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed16b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49720/0 (socket says 172.21.15.114:49720) 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 -- 172.21.15.114:0/1437999347 learned_addr learned my addr 172.21.15.114:0/1437999347 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 -- 172.21.15.114:0/1437999347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 msgr2=0x55aed168ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 --2- 172.21.15.114:0/1437999347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed168ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 -- 172.21.15.114:0/1437999347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 msgr2=0x55aed16b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 --2- 172.21.15.114:0/1437999347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 0x55aed16b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.359+0000 7f0893e00640 1 -- 172.21.15.114:0/1437999347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aed16652c0 con 0x55aed1693b90 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f0893e00640 1 --2- 172.21.15.114:0/1437999347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed16b4f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f088c0883c0 tx=0x7f088c0587b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b00d80d1d0b76a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f087ea00640 1 -- 172.21.15.114:0/1437999347 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c056710 con 0x55aed1693b90 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f087ea00640 1 -- 172.21.15.114:0/1437999347 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c056970 con 0x55aed1693b90 2024-03-06T07:28:12.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f087ea00640 1 -- 172.21.15.114:0/1437999347 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c0608f0 con 0x55aed1693b90 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 -- 172.21.15.114:0/1437999347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 msgr2=0x55aed16b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 --2- 172.21.15.114:0/1437999347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed16b4f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f088c0883c0 tx=0x7f088c0587b0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 -- 172.21.15.114:0/1437999347 shutdown_connections 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 --2- 172.21.15.114:0/1437999347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed16b4f90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 --2- 172.21.15.114:0/1437999347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 0x55aed16b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 --2- 172.21.15.114:0/1437999347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed168ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 -- 172.21.15.114:0/1437999347 >> 172.21.15.114:0/1437999347 conn(0x55aed1666c90 msgr2=0x55aed16b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.360+0000 7f089cc56740 1 -- 172.21.15.114:0/1437999347 shutdown_connections 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.361+0000 7f089cc56740 1 -- 172.21.15.114:0/1437999347 wait complete. 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.361+0000 7f089cc56740 1 -- start start 2024-03-06T07:28:12.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.362+0000 7f089cc56740 1 Processor -- start 2024-03-06T07:28:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.363+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed166bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.363+0000 7f089cc56740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 0x55aed1720cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.363+0000 7f089cc56740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed1689850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.363+0000 7f089cc56740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aed168d190 con 0x55aed1693b90 2024-03-06T07:28:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.363+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894141430 con 0x55aed16937b0 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.363+0000 7f089cc56740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aed1694ea0 con 0x55aed168bc50 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.363+0000 7f089cc56740 10 client.0 _ll_register_callbacks cb 0x7f0884004250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed1689850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed1689850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49722/0 (socket says 172.21.15.114:49722) 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 -- 172.21.15.114:0/3524565479 learned_addr learned my addr 172.21.15.114:0/3524565479 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893400640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed166bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 -- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 msgr2=0x55aed166bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0892a00640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 0x55aed1720cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed166bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 -- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 msgr2=0x55aed1720cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 0x55aed1720cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 -- 172.21.15.114:0/3524565479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aed1721210 con 0x55aed1693b90 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893400640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed166bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:12.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0893e00640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed1689850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f088c06e040 tx=0x7f088c0015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c9b77ddd37ec1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0873e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0873e00640 1 -- 172.21.15.114:0/3524565479 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c086040 con 0x55aed1693b90 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f0873e00640 1 -- 172.21.15.114:0/3524565479 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f088c060af0 con 0x55aed1693b90 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.364+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f088c056970 con 0x55aed1693b90 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f0873e00640 1 -- 172.21.15.114:0/3524565479 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f088c077260 con 0x55aed1693b90 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 10 client.4656 fetch_fsmap learned FSMap version 8 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55aed1661ab0 con 0x55aed1693b90 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f0873e00640 1 -- 172.21.15.114:0/3524565479 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c056970 con 0x55aed1693b90 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f0873e00640 1 -- 172.21.15.114:0/3524565479 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f088c057ac0 con 0x55aed1693b90 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 10 client.4656 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 10 client.4656 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 20 client.4656 populate_metadata read hostname 'smithi114' 2024-03-06T07:28:12.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed16bfd60 0x55aed1665620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 4 client.4656 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 --> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aed168e430 con 0x55aed16bfd60 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.365+0000 7f0893400640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed16bfd60 0x55aed1665620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f0893400640 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed16bfd60 0x55aed1665620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=993cd1c66aef9f9b server_cookie=86c2e599afd640a5 in_seq=0 out_seq=0 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f0873e00640 10 client.4656 ms_handle_connect on v2:172.21.15.114:6835/3774250012 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f0873e00640 1 -- 172.21.15.114:0/3524565479 <== mds.0 v2:172.21.15.114:6835/3774250012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55aed168e430 con 0x55aed16bfd60 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f0873e00640 10 client.4656 handle_command_reply: tid=0 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 client.4656 shutdown 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 msgr2=0x55aed1689850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed1689850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f088c06e040 tx=0x7f088c0015f0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 shutdown_connections 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:6835/3774250012,v1:172.21.15.114:6837/3774250012] conn(0x55aed16bfd60 0x55aed1665620 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aed1693b90 0x55aed1689850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aed16937b0 0x55aed1720cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 --2- 172.21.15.114:0/3524565479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55aed168bc50 0x55aed166bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 >> 172.21.15.114:0/3524565479 conn(0x55aed1666c90 msgr2=0x55aed16b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f0873e00640 0 client.4656 ms_handle_reset on v2:172.21.15.114:6835/3774250012 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 shutdown_connections 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.366+0000 7f089cc56740 1 -- 172.21.15.114:0/3524565479 wait complete. 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089cc56740 20 client.4656 trim_cache size 0 max 16384 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 4260, 2024-03-06T07:28:12.373 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 17.359891601000001, 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 8, 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 20, 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 20, 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 22.759900113 2024-03-06T07:28:12.374 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f087406b4d0 msgr2=0x7f087406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f087406b4d0 0x7f087406d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0884006fd0 tx=0x7f0884008040 comp rx=0 tx=0).stop 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c56b0 msgr2=0x7f089413dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c56b0 0x7f089413dbb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f088c00cad0 tx=0x7f088c00cf90 comp rx=0 tx=0).stop 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 shutdown_connections 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f087406b4d0 0x7f087406d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08940c56b0 0x7f089413dbb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08940c4d10 0x7f08940cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 --2- 172.21.15.114:0/466651202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08940c4930 0x7f08941f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 >> 172.21.15.114:0/466651202 conn(0x7f08940cc4f0 msgr2=0x7f08941e6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 shutdown_connections 2024-03-06T07:28:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:12.367+0000 7f089a000640 1 -- 172.21.15.114:0/466651202 wait complete. 2024-03-06T07:28:12.386 INFO:teuthology.run_tasks:Running task kclient... 2024-03-06T07:28:12.410 INFO:tasks.kclient:Mounting kernel clients... 2024-03-06T07:28:12.410 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:28:12.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.410 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:28:12.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-03-06T07:28:12.563 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-03-06T07:28:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:12.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.569 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:28:12.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-03-06T07:28:12.704 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-03-06T07:28:12.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:12.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.708 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-03-06T07:28:12.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link delete ceph-brx 2024-03-06T07:28:12.871 INFO:teuthology.orchestra.run.smithi177.stderr:Cannot find device "ceph-brx" 2024-03-06T07:28:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:12.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.876 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-03-06T07:28:12.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:12.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link delete ceph-brx 2024-03-06T07:28:13.024 INFO:teuthology.orchestra.run.smithi177.stderr:Cannot find device "ceph-brx" 2024-03-06T07:28:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:13.028 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-06T07:28:13.029 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:28:13.029 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:13.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:13.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.242+0000 7fb575000640 1 Processor -- start 2024-03-06T07:28:13.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.242+0000 7fb575000640 1 -- start start 2024-03-06T07:28:13.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.243+0000 7fb575000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb570144c00 0x7fb57006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb575000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57015f9d0 0x7fb57015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb575000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb5700ce0a0 0x7fb5700ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb575000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb570160370 con 0x7fb570144c00 2024-03-06T07:28:13.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb575000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57006ee70 con 0x7fb5700ce0a0 2024-03-06T07:28:13.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb575000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57006cef0 con 0x7fb57015f9d0 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb56de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb570144c00 0x7fb57006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb56de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb570144c00 0x7fb57006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49734/0 (socket says 172.21.15.114:49734) 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb56de00640 1 -- 172.21.15.114:0/276025967 learned_addr learned my addr 172.21.15.114:0/276025967 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb56e800640 1 --2- 172.21.15.114:0/276025967 >> v2:172.21.15.177:3300/0 conn(0x7fb5700ce0a0 0x7fb5700ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb56d400640 1 --2- 172.21.15.114:0/276025967 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57015f9d0 0x7fb57015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.244+0000 7fb56de00640 1 -- 172.21.15.114:0/276025967 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57015f9d0 msgr2=0x7fb57015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.245+0000 7fb56de00640 1 --2- 172.21.15.114:0/276025967 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57015f9d0 0x7fb57015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.245+0000 7fb56de00640 1 -- 172.21.15.114:0/276025967 >> v2:172.21.15.177:3300/0 conn(0x7fb5700ce0a0 msgr2=0x7fb5700ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.245+0000 7fb56de00640 1 --2- 172.21.15.114:0/276025967 >> v2:172.21.15.177:3300/0 conn(0x7fb5700ce0a0 0x7fb5700ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.245+0000 7fb56de00640 1 -- 172.21.15.114:0/276025967 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57006d1d0 con 0x7fb570144c00 2024-03-06T07:28:13.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.245+0000 7fb56e800640 1 --2- 172.21.15.114:0/276025967 >> v2:172.21.15.177:3300/0 conn(0x7fb5700ce0a0 0x7fb5700ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.246+0000 7fb56de00640 1 --2- 172.21.15.114:0/276025967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb570144c00 0x7fb57006e930 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb5700737b0 tx=0x7fb56002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97885c1a94c700d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.246+0000 7fb56ca00640 1 -- 172.21.15.114:0/276025967 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560035020 con 0x7fb570144c00 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.246+0000 7fb56ca00640 1 -- 172.21.15.114:0/276025967 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb560031870 con 0x7fb570144c00 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.246+0000 7fb56ca00640 1 -- 172.21.15.114:0/276025967 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560031c40 con 0x7fb570144c00 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 -- 172.21.15.114:0/276025967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb570144c00 msgr2=0x7fb57006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 --2- 172.21.15.114:0/276025967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb570144c00 0x7fb57006e930 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb5700737b0 tx=0x7fb56002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 -- 172.21.15.114:0/276025967 shutdown_connections 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 --2- 172.21.15.114:0/276025967 >> v2:172.21.15.177:3300/0 conn(0x7fb5700ce0a0 0x7fb5700ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 --2- 172.21.15.114:0/276025967 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57015f9d0 0x7fb57015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 --2- 172.21.15.114:0/276025967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb570144c00 0x7fb57006e930 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 -- 172.21.15.114:0/276025967 >> 172.21.15.114:0/276025967 conn(0x7fb5700b1730 msgr2=0x7fb5700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 -- 172.21.15.114:0/276025967 shutdown_connections 2024-03-06T07:28:13.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.247+0000 7fb575000640 1 -- 172.21.15.114:0/276025967 wait complete. 2024-03-06T07:28:13.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.249+0000 7fb575000640 1 Processor -- start 2024-03-06T07:28:13.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.249+0000 7fb575000640 1 -- start start 2024-03-06T07:28:13.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb575000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5700ce0a0 0x7fb57015b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb575000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb570144c00 0x7fb5700cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb575000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57015f9d0 0x7fb5700cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb575000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb570160730 con 0x7fb57015f9d0 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb575000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57006d790 con 0x7fb5700ce0a0 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb575000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57006cb50 con 0x7fb570144c00 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb56e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57015f9d0 0x7fb5700cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb56e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57015f9d0 0x7fb5700cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49746/0 (socket says 172.21.15.114:49746) 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.250+0000 7fb56e800640 1 -- 172.21.15.114:0/1994545686 learned_addr learned my addr 172.21.15.114:0/1994545686 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56d400640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb570144c00 0x7fb5700cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56e800640 1 -- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb570144c00 msgr2=0x7fb5700cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56e800640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb570144c00 0x7fb5700cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56de00640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5700ce0a0 0x7fb57015b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56e800640 1 -- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5700ce0a0 msgr2=0x7fb57015b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56e800640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5700ce0a0 0x7fb57015b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56e800640 1 -- 172.21.15.114:0/1994545686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb570137ee0 con 0x7fb57015f9d0 2024-03-06T07:28:13.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb56e800640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57015f9d0 0x7fb5700cefc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb56800be90 tx=0x7fb56800e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb566000640 1 -- 172.21.15.114:0/1994545686 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56801a020 con 0x7fb57015f9d0 2024-03-06T07:28:13.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb566000640 1 -- 172.21.15.114:0/1994545686 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb568016420 con 0x7fb57015f9d0 2024-03-06T07:28:13.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5701381c0 con 0x7fb57015f9d0 2024-03-06T07:28:13.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.251+0000 7fb566000640 1 -- 172.21.15.114:0/1994545686 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568016810 con 0x7fb57015f9d0 2024-03-06T07:28:13.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.252+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb570138780 con 0x7fb57015f9d0 2024-03-06T07:28:13.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.253+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57013bad0 con 0x7fb57015f9d0 2024-03-06T07:28:13.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.254+0000 7fb566000640 1 -- 172.21.15.114:0/1994545686 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb56801d020 con 0x7fb57015f9d0 2024-03-06T07:28:13.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.254+0000 7fb566000640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.254+0000 7fb566000640 1 -- 172.21.15.114:0/1994545686 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb56808ebe0 con 0x7fb57015f9d0 2024-03-06T07:28:13.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.255+0000 7fb56de00640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.255+0000 7fb56de00640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb560036000 tx=0x7fb560008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.264+0000 7fb566000640 1 -- 172.21.15.114:0/1994545686 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb568024020 con 0x7fb57015f9d0 2024-03-06T07:28:13.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.497+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb57013a210 con 0x7fb57015f9d0 2024-03-06T07:28:13.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.497+0000 7fb566000640 1 -- 172.21.15.114:0/1994545686 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb56805da50 con 0x7fb57015f9d0 2024-03-06T07:28:13.500 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:13.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 msgr2=0x7fb55006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb560036000 tx=0x7fb560008030 comp rx=0 tx=0).stop 2024-03-06T07:28:13.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57015f9d0 msgr2=0x7fb5700cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57015f9d0 0x7fb5700cefc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb56800be90 tx=0x7fb56800e430 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 shutdown_connections 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb57015f9d0 0x7fb5700cefc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb570144c00 0x7fb5700cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 --2- 172.21.15.114:0/1994545686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5700ce0a0 0x7fb57015b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 >> 172.21.15.114:0/1994545686 conn(0x7fb5700b1730 msgr2=0x7fb5700b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 shutdown_connections 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:13.500+0000 7fb575000640 1 -- 172.21.15.114:0/1994545686 wait complete. 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:13.522 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:13.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:13.523 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:28:13.570 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:13.570 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:13.570 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42562sec preferred_lft 42562sec 2024-03-06T07:28:13.571 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:28:13.572 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:13.572 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:28:13.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:13.572 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:13.572 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:13.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:28:13.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-03-06T07:28:13.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-03-06T07:28:13.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:28:13.573 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:14.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:14.285 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:14.361 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:28:14.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:14.365 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:28:14.418 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:28:14.418 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:14.418 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:14.418 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:14.418 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:14.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:14.418 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:14.418 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:14.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:14.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:14.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:14.419 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:14.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:14.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:15.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:15.138 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:28:15.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:15.181 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-03-06T07:28:15.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:15.266 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:15.266 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:15.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:15.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:28:15.266 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:15.431 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:28:15.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:28:15.431 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:15.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:15.693 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:15.693 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:15.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:28:15.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:28:15.694 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:15.843 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:15.844 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:15.884 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:15.884 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:15.884 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:15.970 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:28:15.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:15.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:16.127 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:16.127 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:16.735 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:16.735 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:16.735 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:28:16.735 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:28:16.735 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:28:16.738 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:28:16.738 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:16.738 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:16.738 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:16.738 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:16.741 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:16.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:16.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:16.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:16.816 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:16.872 INFO:teuthology.orchestra.run.smithi114.stdout:client4531 2024-03-06T07:28:16.873 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:16.873 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:16.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:16.916 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:17.016 INFO:teuthology.orchestra.run.smithi114.stdout:client4531 2024-03-06T07:28:17.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.017 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:17.074 INFO:teuthology.orchestra.run.smithi114.stdout:client4531 2024-03-06T07:28:17.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.074 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/149c34a4-b335-4a4b-8a41-0fe21b0db808.client4531/status') 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.4531 (3)172.21.15.114:0/179128347 2024-03-06T07:28:17.163 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-03-06T07:28:17.164 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-03-06T07:28:17.164 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-03-06T07:28:17.164 INFO:teuthology.orchestra.run.smithi114.stderr:70 bytes copied, 8.0473e-05 s, 870 kB/s 2024-03-06T07:28:17.167 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-03-06T07:28:17.167 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:28:17.167 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:17.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:17.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.361+0000 7f974e400640 1 Processor -- start 2024-03-06T07:28:17.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.362+0000 7f974e400640 1 -- start start 2024-03-06T07:28:17.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.363+0000 7f974e400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97480d1cc0 0x7f97480b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.363+0000 7f974e400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480d4c50 con 0x7f97480b2de0 2024-03-06T07:28:17.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.363+0000 7f974e400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480cfa30 con 0x7f97480d24c0 2024-03-06T07:28:17.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.363+0000 7f974e400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97480d4a00 con 0x7f97480d1cc0 2024-03-06T07:28:17.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.363+0000 7f9747e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97480d1cc0 0x7f97480b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.363+0000 7f973fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f97480b2de0 0x7f97480b0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49704/0 (socket says 172.21.15.114:49704) 2024-03-06T07:28:17.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.363+0000 7f973fe00640 1 -- 172.21.15.114:0/2156732226 learned_addr learned my addr 172.21.15.114:0/2156732226 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:17.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.364+0000 7f9746a00640 1 -- 172.21.15.114:0/2156732226 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016852389 0 0) 0x7f97480d4c50 con 0x7f97480b2de0 2024-03-06T07:28:17.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.364+0000 7f9747e00640 1 -- 172.21.15.114:0/2156732226 >> v1:172.21.15.177:6789/0 conn(0x7f97480d24c0 legacy=0x7f97480d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.364+0000 7f9747e00640 1 -- 172.21.15.114:0/2156732226 >> v1:172.21.15.114:6789/0 conn(0x7f97480b2de0 legacy=0x7f97480b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.364+0000 7f9747e00640 1 -- 172.21.15.114:0/2156732226 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97480cfd10 con 0x7f97480d1cc0 2024-03-06T07:28:17.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.365+0000 7f9747e00640 1 --2- 172.21.15.114:0/2156732226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d1cc0 0x7f97480b11b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9740009d90 tx=0x7f974002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6e8e1d80b19e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.365+0000 7f9746a00640 1 -- 172.21.15.114:0/2156732226 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740039070 con 0x7f97480d1cc0 2024-03-06T07:28:17.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.365+0000 7f9746a00640 1 -- 172.21.15.114:0/2156732226 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974002cbd0 con 0x7f97480d1cc0 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.365+0000 7f9746a00640 1 -- 172.21.15.114:0/2156732226 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740034440 con 0x7f97480d1cc0 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.366+0000 7f974e400640 1 -- 172.21.15.114:0/2156732226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d1cc0 msgr2=0x7f97480b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.366+0000 7f974e400640 1 --2- 172.21.15.114:0/2156732226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d1cc0 0x7f97480b11b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9740009d90 tx=0x7f974002be20 comp rx=0 tx=0).stop 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.366+0000 7f974e400640 1 -- 172.21.15.114:0/2156732226 shutdown_connections 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.366+0000 7f974e400640 1 --2- 172.21.15.114:0/2156732226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d1cc0 0x7f97480b11b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.366+0000 7f974e400640 1 -- 172.21.15.114:0/2156732226 >> 172.21.15.114:0/2156732226 conn(0x7f97480d3ee0 msgr2=0x7f97480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.366+0000 7f974e400640 1 -- 172.21.15.114:0/2156732226 shutdown_connections 2024-03-06T07:28:17.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.366+0000 7f974e400640 1 -- 172.21.15.114:0/2156732226 wait complete. 2024-03-06T07:28:17.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.368+0000 7f974e400640 1 Processor -- start 2024-03-06T07:28:17.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.368+0000 7f974e400640 1 -- start start 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f974e400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97480b2de0 0x7f97480ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f974e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97480d1cc0 0x7f9748160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f974e400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d24c0 0x7f97480c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f974e400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480d0080 con 0x7f97480d24c0 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f974e400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480cf7e0 con 0x7f97480d1cc0 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f974e400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97480b16f0 con 0x7f97480b2de0 2024-03-06T07:28:17.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f9747e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d24c0 0x7f97480c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f9747e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d24c0 0x7f97480c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49768/0 (socket says 172.21.15.114:49768) 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.369+0000 7f9747e00640 1 -- 172.21.15.114:0/436975717 learned_addr learned my addr 172.21.15.114:0/436975717 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f973fe00640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97480d1cc0 0x7f9748160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97480b2de0 0x7f97480ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747e00640 1 -- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97480b2de0 msgr2=0x7f97480ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747e00640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97480b2de0 0x7f97480ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747e00640 1 -- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97480d1cc0 msgr2=0x7f9748160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747e00640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97480d1cc0 0x7f9748160940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747e00640 1 -- 172.21.15.114:0/436975717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97480cb0e0 con 0x7f97480d24c0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f973fe00640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97480d1cc0 0x7f9748160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97480b2de0 0x7f97480ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f9747e00640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d24c0 0x7f97480c8600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9740009d90 tx=0x7f9740004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f973f400640 1 -- 172.21.15.114:0/436975717 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740044070 con 0x7f97480d24c0 2024-03-06T07:28:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f973f400640 1 -- 172.21.15.114:0/436975717 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9740003990 con 0x7f97480d24c0 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97480cb3c0 con 0x7f97480d24c0 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.370+0000 7f973f400640 1 -- 172.21.15.114:0/436975717 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740039040 con 0x7f97480d24c0 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.371+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97480cac20 con 0x7f97480d24c0 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.372+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974804f330 con 0x7f97480d24c0 2024-03-06T07:28:17.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.372+0000 7f973f400640 1 -- 172.21.15.114:0/436975717 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9740004030 con 0x7f97480d24c0 2024-03-06T07:28:17.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.374+0000 7f973f400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f972406b4d0 0x7f972406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.374+0000 7f973f400640 1 -- 172.21.15.114:0/436975717 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97400acc60 con 0x7f97480d24c0 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.374+0000 7f9747400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f972406b4d0 0x7f972406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.375+0000 7f9747400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f972406b4d0 0x7f972406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97480c7b30 tx=0x7f9738008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.379+0000 7f973f400640 1 -- 172.21.15.114:0/436975717 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f974004e020 con 0x7f97480d24c0 2024-03-06T07:28:17.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.613+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f97480cf560 con 0x7f97480d24c0 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.613+0000 7f973f400640 1 -- 172.21.15.114:0/436975717 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f974007bad0 con 0x7f97480d24c0 2024-03-06T07:28:17.615 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.615+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f972406b4d0 msgr2=0x7f972406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.615+0000 7f974e400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f972406b4d0 0x7f972406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97480c7b30 tx=0x7f9738008040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.615+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d24c0 msgr2=0x7f97480c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.615+0000 7f974e400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d24c0 0x7f97480c8600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9740009d90 tx=0x7f9740004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 shutdown_connections 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f972406b4d0 0x7f972406d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97480d24c0 0x7f97480c8600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97480d1cc0 0x7f9748160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 --2- 172.21.15.114:0/436975717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97480b2de0 0x7f97480ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 >> 172.21.15.114:0/436975717 conn(0x7f97480d3ee0 msgr2=0x7f974806c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 shutdown_connections 2024-03-06T07:28:17.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:17.616+0000 7f974e400640 1 -- 172.21.15.114:0/436975717 wait complete. 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:17.635 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:28:17.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.635 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:28:17.680 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:17.680 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:17.680 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:17.680 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:17.680 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:17.680 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:17.681 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42558sec preferred_lft 42558sec 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:28:17.682 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:17.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.683 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:17.813 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:28:17.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.820 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:28:17.873 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:28:17.873 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:17.873 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:17.873 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:17.873 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:17.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.873 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:17.873 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:17.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:17.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:17.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:17.874 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:18.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:18.057 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:18.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:18.106 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-03-06T07:28:18.155 INFO:teuthology.orchestra.run.smithi114.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:28:18.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:18.156 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:18.156 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:18.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:18.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:28:18.156 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:18.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:18.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:28:18.355 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 42:cc:5d:e0:0d:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::40cc:5dff:fee0:d57/64 scope link proto kernel_ll 2024-03-06T07:28:18.356 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:18.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 42:cc:5d:e0:0d:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::40cc:5dff:fee0:d57/64 scope link proto kernel_ll 2024-03-06T07:28:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:18.449 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:28:18.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:28:18.450 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:18.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:18.713 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:28:18.713 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:28:18.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.1 up 2024-03-06T07:28:18.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:28:18.713 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:28:18.882 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:18.882 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:18.927 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:28:18.928 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:18.928 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:19.020 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-03-06T07:28:19.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:19.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:19.140 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:19.140 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:19.175 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:19.175 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:19.175 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:28:19.175 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 2@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:28:19.175 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:28:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-06T07:28:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:19.182 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:19.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:19.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:28:19.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:19.265 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:19.320 INFO:teuthology.orchestra.run.smithi114.stdout:client4534 2024-03-06T07:28:19.320 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:19.320 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:19.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:19.369 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run.smithi114.stdout:client4534 2024-03-06T07:28:19.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:19.466 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:19.520 INFO:teuthology.orchestra.run.smithi114.stdout:client4534 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:19.521 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/149c34a4-b335-4a4b-8a41-0fe21b0db808.client4534/status') 2024-03-06T07:28:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.4534 (3)172.21.15.114:0/3576273212 2024-03-06T07:28:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-03-06T07:28:19.601 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-03-06T07:28:19.601 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-03-06T07:28:19.601 INFO:teuthology.orchestra.run.smithi114.stderr:71 bytes copied, 6.461e-05 s, 1.1 MB/s 2024-03-06T07:28:19.604 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-06T07:28:19.605 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:28:19.605 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:19.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:19.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.813+0000 7f9f77200640 1 Processor -- start 2024-03-06T07:28:19.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.813+0000 7f9f77200640 1 -- start start 2024-03-06T07:28:19.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.815+0000 7f9f77200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 0x7f9f700cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.815+0000 7f9f77200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9f7015f4e0 0x7f9f700cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.815+0000 7f9f77200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700b1f00 con 0x7f9f7015f100 2024-03-06T07:28:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.815+0000 7f9f77200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f700b21a0 con 0x7f9f7015f4e0 2024-03-06T07:28:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.815+0000 7f9f77200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b2480 con 0x7f9f700b4140 2024-03-06T07:28:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.815+0000 7f9f75e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9f7015f100 0x7f9f700ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40456/0 (socket says 172.21.15.114:40456) 2024-03-06T07:28:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.815+0000 7f9f75e00640 1 -- 172.21.15.114:0/3785955034 learned_addr learned my addr 172.21.15.114:0/3785955034 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f75400640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 0x7f9f700cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f76800640 1 --2- 172.21.15.114:0/3785955034 >> v2:172.21.15.177:3300/0 conn(0x7f9f7015f4e0 0x7f9f700cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310706288 0 0) 0x7f9f700b1f00 con 0x7f9f7015f100 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f58003520 con 0x7f9f7015f100 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719875958 0 0) 0x7f9f58003520 con 0x7f9f7015f100 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 msgr2=0x7f9f700cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 0x7f9f700cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 >> v2:172.21.15.177:3300/0 conn(0x7f9f7015f4e0 msgr2=0x7f9f700cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 --2- 172.21.15.114:0/3785955034 >> v2:172.21.15.177:3300/0 conn(0x7f9f7015f4e0 0x7f9f700cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700b2760 con 0x7f9f7015f100 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f9f64001690 con 0x7f9f7015f100 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f75400640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 0x7f9f700cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 >> v1:172.21.15.114:6789/0 conn(0x7f9f7015f100 legacy=0x7f9f700ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580039b0 0x7f9f58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 0x7f9f58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f58004d40 0x7f9f580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b21a0 con 0x7f9f58004d40 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b2480 con 0x7f9f580039b0 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.816+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5800baf0 con 0x7f9f700b4140 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f76800640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f58004d40 0x7f9f580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f75400640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 0x7f9f58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f75e00640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580039b0 0x7f9f58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f76800640 1 -- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 msgr2=0x7f9f58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f76800640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f700b4140 0x7f9f58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f76800640 1 -- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580039b0 msgr2=0x7f9f58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f76800640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580039b0 0x7f9f58003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.817+0000 7f9f76800640 1 -- 172.21.15.114:0/3785955034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f58003520 con 0x7f9f58004d40 2024-03-06T07:28:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.819+0000 7f9f76800640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f58004d40 0x7f9f580051c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c007360 tx=0x7f9f58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2c10ff27716ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.820+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c039070 con 0x7f9f58004d40 2024-03-06T07:28:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.820+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c02ccf0 con 0x7f9f58004d40 2024-03-06T07:28:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.820+0000 7f9f74a00640 1 -- 172.21.15.114:0/3785955034 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c009be0 con 0x7f9f58004d40 2024-03-06T07:28:19.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.820+0000 7f9f77200640 1 -- 172.21.15.114:0/3785955034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f58004d40 msgr2=0x7f9f580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.820+0000 7f9f77200640 1 --2- 172.21.15.114:0/3785955034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f58004d40 0x7f9f580051c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c007360 tx=0x7f9f58005830 comp rx=0 tx=0).stop 2024-03-06T07:28:19.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.821+0000 7f9f76800640 1 -- 172.21.15.114:0/3785955034 reap_dead start 2024-03-06T07:28:19.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.821+0000 7f9f77200640 1 -- 172.21.15.114:0/3785955034 shutdown_connections 2024-03-06T07:28:19.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.821+0000 7f9f77200640 1 -- 172.21.15.114:0/3785955034 >> 172.21.15.114:0/3785955034 conn(0x7f9f700b3620 msgr2=0x7f9f700b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.821+0000 7f9f77200640 1 -- 172.21.15.114:0/3785955034 shutdown_connections 2024-03-06T07:28:19.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.821+0000 7f9f77200640 1 -- 172.21.15.114:0/3785955034 wait complete. 2024-03-06T07:28:19.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.822+0000 7f9f77200640 1 Processor -- start 2024-03-06T07:28:19.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.823+0000 7f9f77200640 1 -- start start 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.823+0000 7f9f77200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f70156e70 0x7f9f700cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.823+0000 7f9f77200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700c8b80 0x7f9f700cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f77200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f700b4af0 0x7f9f700c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f77200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b33b0 con 0x7f9f700c8b80 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f77200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b3230 con 0x7f9f700b4af0 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f77200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700cebd0 con 0x7f9f70156e70 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700c8b80 0x7f9f700cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700c8b80 0x7f9f700cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53526/0 (socket says 172.21.15.114:53526) 2024-03-06T07:28:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 -- 172.21.15.114:0/1654136500 learned_addr learned my addr 172.21.15.114:0/1654136500 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 -- 172.21.15.114:0/1654136500 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f70156e70 msgr2=0x7f9f700cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f70156e70 0x7f9f700cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f76800640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f700b4af0 0x7f9f700c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 -- 172.21.15.114:0/1654136500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f700b4af0 msgr2=0x7f9f700c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f700b4af0 0x7f9f700c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.824+0000 7f9f75400640 1 -- 172.21.15.114:0/1654136500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700b7710 con 0x7f9f700c8b80 2024-03-06T07:28:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.825+0000 7f9f75400640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700c8b80 0x7f9f700cceb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9f600115c0 tx=0x7f9f60011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.825+0000 7f9f6a000640 1 -- 172.21.15.114:0/1654136500 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60019070 con 0x7f9f700c8b80 2024-03-06T07:28:19.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.825+0000 7f9f6a000640 1 -- 172.21.15.114:0/1654136500 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f60004220 con 0x7f9f700c8b80 2024-03-06T07:28:19.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.825+0000 7f9f6a000640 1 -- 172.21.15.114:0/1654136500 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000f040 con 0x7f9f700c8b80 2024-03-06T07:28:19.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.825+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f700b5f30 con 0x7f9f700c8b80 2024-03-06T07:28:19.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.825+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f700b64f0 con 0x7f9f700c8b80 2024-03-06T07:28:19.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.826+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7004f330 con 0x7f9f700c8b80 2024-03-06T07:28:19.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.827+0000 7f9f6a000640 1 -- 172.21.15.114:0/1654136500 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f6000f6f0 con 0x7f9f700c8b80 2024-03-06T07:28:19.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.828+0000 7f9f6a000640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f5406b4d0 0x7f9f5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.828+0000 7f9f6a000640 1 -- 172.21.15.114:0/1654136500 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f6008a4b0 con 0x7f9f700c8b80 2024-03-06T07:28:19.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.828+0000 7f9f75e00640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f5406b4d0 0x7f9f5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.829+0000 7f9f75e00640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f5406b4d0 0x7f9f5406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f64001200 tx=0x7f9f64001110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:19.836+0000 7f9f6a000640 1 -- 172.21.15.114:0/1654136500 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f60059320 con 0x7f9f700c8b80 2024-03-06T07:28:20.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.077+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9f700c5050 con 0x7f9f700c8b80 2024-03-06T07:28:20.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.078+0000 7f9f6a000640 1 -- 172.21.15.114:0/1654136500 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9f60058c20 con 0x7f9f700c8b80 2024-03-06T07:28:20.080 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f5406b4d0 msgr2=0x7f9f5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f5406b4d0 0x7f9f5406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f64001200 tx=0x7f9f64001110 comp rx=0 tx=0).stop 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700c8b80 msgr2=0x7f9f700cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700c8b80 0x7f9f700cceb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9f600115c0 tx=0x7f9f60011a80 comp rx=0 tx=0).stop 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 shutdown_connections 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f700b4af0 0x7f9f700c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f5406b4d0 0x7f9f5406d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f700c8b80 0x7f9f700cceb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 --2- 172.21.15.114:0/1654136500 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f70156e70 0x7f9f700cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 >> 172.21.15.114:0/1654136500 conn(0x7f9f700b3620 msgr2=0x7f9f700aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 shutdown_connections 2024-03-06T07:28:20.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:20.081+0000 7f9f77200640 1 -- 172.21.15.114:0/1654136500 wait complete. 2024-03-06T07:28:20.100 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:20.100 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:20.100 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-03-06T07:28:20.100 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:28:20.100 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:20.100 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:20.101 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:20.101 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:20.101 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:28:20.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:20.101 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:20.158 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42583sec preferred_lft 42583sec 2024-03-06T07:28:20.159 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::ec4:7aff:feea:b04b/64 scope link proto kernel_ll 2024-03-06T07:28:20.160 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:20.160 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:28:20.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:20.161 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:20.161 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:20.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:28:20.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip addr flush dev ceph-brx 2024-03-06T07:28:20.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set ceph-brx up 2024-03-06T07:28:20.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:28:20.161 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:20.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:20.608 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:20.691 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-06T07:28:20.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:20.694 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-03-06T07:28:20.746 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-03-06T07:28:20.746 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:20.746 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:20.746 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:20.747 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:20.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:20.747 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:20.748 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:20.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:20.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:20.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:20.748 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:21.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:21.457 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-03-06T07:28:21.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:21.513 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list-id 2024-03-06T07:28:21.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:21.565 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:21.565 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:21.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:28:21.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:28:21.565 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:21.701 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:28:21.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:21.701 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:21.701 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:21.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:28:21.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:21.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:28:21.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:28:21.702 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:28:21.702 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:21.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:21.969 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:21.969 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:21.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.0 up 2024-03-06T07:28:21.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:28:21.969 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:22.137 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:22.137 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:22.183 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:28:22.183 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:22.183 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:22.275 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-06T07:28:22.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:22.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:22.417 INFO:teuthology.orchestra.run.smithi177.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:22.417 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:23.016 INFO:teuthology.orchestra.run.smithi177.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:23.016 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:23.016 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:28:23.016 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: trying mount with new device syntax: 1@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:28:23.017 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:28:23.021 INFO:teuthology.orchestra.run.smithi177.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-06T07:28:23.021 INFO:teuthology.orchestra.run.smithi177.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:23.021 INFO:teuthology.orchestra.run.smithi177.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:23.021 INFO:teuthology.orchestra.run.smithi177.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:23.021 INFO:teuthology.orchestra.run.smithi177.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:23.026 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:23.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.027 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:28:23.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.113 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:23.168 INFO:teuthology.orchestra.run.smithi177.stdout:client4680 2024-03-06T07:28:23.169 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:23.169 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:23.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.221 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:23.272 INFO:teuthology.orchestra.run.smithi177.stdout:client4680 2024-03-06T07:28:23.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.273 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:23.328 INFO:teuthology.orchestra.run.smithi177.stdout:client4680 2024-03-06T07:28:23.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.328 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/149c34a4-b335-4a4b-8a41-0fe21b0db808.client4680/status') 2024-03-06T07:28:23.415 INFO:teuthology.orchestra.run.smithi177.stdout:instance: client.4680 (3)172.21.15.177:0/4293383414 2024-03-06T07:28:23.415 INFO:teuthology.orchestra.run.smithi177.stdout:blocklisted: false 2024-03-06T07:28:23.416 INFO:teuthology.orchestra.run.smithi177.stderr:0+1 records in 2024-03-06T07:28:23.416 INFO:teuthology.orchestra.run.smithi177.stderr:0+1 records out 2024-03-06T07:28:23.416 INFO:teuthology.orchestra.run.smithi177.stderr:71 bytes copied, 0.000171775 s, 413 kB/s 2024-03-06T07:28:23.419 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-03-06T07:28:23.420 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:28:23.420 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:23.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:23.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.623+0000 7f3672400640 1 Processor -- start 2024-03-06T07:28:23.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.623+0000 7f3672400640 1 -- start start 2024-03-06T07:28:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f3672400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f366c15acb0 0x7f366c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f3672400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 0x7f366c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f3672400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c15b090 con 0x7f366c0b4680 2024-03-06T07:28:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f3672400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c0b2a30 con 0x7f366c15acb0 2024-03-06T07:28:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f3672400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b4360 con 0x7f366c06ec00 2024-03-06T07:28:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f3663e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f366c0b4680 0x7f366c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40472/0 (socket says 172.21.15.114:40472) 2024-03-06T07:28:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f3663e00640 1 -- 172.21.15.114:0/3675489364 learned_addr learned my addr 172.21.15.114:0/3675489364 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f366b400640 1 --2- 172.21.15.114:0/3675489364 >> v2:172.21.15.177:3300/0 conn(0x7f366c15acb0 0x7f366c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710633197 0 0) 0x7f366c15b090 con 0x7f366c0b4680 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.625+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3650003520 con 0x7f366c0b4680 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366be00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 0x7f366c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439086762 0 0) 0x7f3650003520 con 0x7f366c0b4680 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 msgr2=0x7f366c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 0x7f366c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 >> v2:172.21.15.177:3300/0 conn(0x7f366c15acb0 msgr2=0x7f366c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 --2- 172.21.15.114:0/3675489364 >> v2:172.21.15.177:3300/0 conn(0x7f366c15acb0 0x7f366c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c0b2cf0 con 0x7f366c0b4680 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f3654001670 con 0x7f366c0b4680 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 >> v1:172.21.15.114:6789/0 conn(0x7f366c0b4680 legacy=0x7f366c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36500039b0 0x7f3650003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 0x7f3650004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f36500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b2a30 con 0x7f3650004d40 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b4360 con 0x7f36500039b0 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.626+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365000b9f0 con 0x7f366c06ec00 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366be00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f36500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366b400640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36500039b0 0x7f3650003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f3663e00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 0x7f3650004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366be00640 1 -- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 msgr2=0x7f3650004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366be00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 0x7f3650004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366be00640 1 -- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36500039b0 msgr2=0x7f3650003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366be00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36500039b0 0x7f3650003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366be00640 1 -- 172.21.15.114:0/3675489364 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3650003520 con 0x7f3650004d40 2024-03-06T07:28:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f3663e00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c06ec00 0x7f3650004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.627+0000 7f366b400640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36500039b0 0x7f3650003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.628+0000 7f366be00640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f36500051c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3664007330 tx=0x7f3650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5079547b5a142b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.628+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664039070 con 0x7f3650004d40 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.628+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366402cca0 con 0x7f3650004d40 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.628+0000 7f366aa00640 1 -- 172.21.15.114:0/3675489364 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664009be0 con 0x7f3650004d40 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.629+0000 7f3672400640 1 -- 172.21.15.114:0/3675489364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 msgr2=0x7f36500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.629+0000 7f3672400640 1 --2- 172.21.15.114:0/3675489364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f36500051c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3664007330 tx=0x7f3650005830 comp rx=0 tx=0).stop 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.629+0000 7f366be00640 1 -- 172.21.15.114:0/3675489364 reap_dead start 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.629+0000 7f3672400640 1 -- 172.21.15.114:0/3675489364 shutdown_connections 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.629+0000 7f3672400640 1 -- 172.21.15.114:0/3675489364 >> 172.21.15.114:0/3675489364 conn(0x7f366c0b1730 msgr2=0x7f366c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.629+0000 7f3672400640 1 -- 172.21.15.114:0/3675489364 shutdown_connections 2024-03-06T07:28:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.629+0000 7f3672400640 1 -- 172.21.15.114:0/3675489364 wait complete. 2024-03-06T07:28:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.632+0000 7f3672400640 1 Processor -- start 2024-03-06T07:28:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.632+0000 7f3672400640 1 -- start start 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f3672400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f366c0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f3672400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f366c0ccfb0 0x7f366c14a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f3672400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c0ae070 0x7f366c0ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f3672400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b3d10 con 0x7f3650004d40 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f3672400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b32b0 con 0x7f366c0ccfb0 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f3672400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c0b4010 con 0x7f366c0ae070 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f366c0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f366c0ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53548/0 (socket says 172.21.15.114:53548) 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 -- 172.21.15.114:0/3432596102 learned_addr learned my addr 172.21.15.114:0/3432596102 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 -- 172.21.15.114:0/3432596102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c0ae070 msgr2=0x7f366c0ae4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c0ae070 0x7f366c0ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 -- 172.21.15.114:0/3432596102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f366c0ccfb0 msgr2=0x7f366c14a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f366c0ccfb0 0x7f366c14a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.634+0000 7f366b400640 1 -- 172.21.15.114:0/3432596102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c0cd520 con 0x7f3650004d40 2024-03-06T07:28:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.635+0000 7f366b400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f366c0ced80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f365c010a00 tx=0x7f365c010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.635+0000 7f3663400640 1 -- 172.21.15.114:0/3432596102 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c00f040 con 0x7f3650004d40 2024-03-06T07:28:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.635+0000 7f3663400640 1 -- 172.21.15.114:0/3432596102 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c00eb40 con 0x7f3650004d40 2024-03-06T07:28:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.635+0000 7f3663400640 1 -- 172.21.15.114:0/3432596102 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c01c400 con 0x7f3650004d40 2024-03-06T07:28:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.635+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c0cd800 con 0x7f3650004d40 2024-03-06T07:28:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.635+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c0cddc0 con 0x7f3650004d40 2024-03-06T07:28:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.636+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c04f330 con 0x7f3650004d40 2024-03-06T07:28:23.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.637+0000 7f3663400640 1 -- 172.21.15.114:0/3432596102 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f365c01c5a0 con 0x7f3650004d40 2024-03-06T07:28:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.638+0000 7f3663400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f364806b4d0 0x7f364806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.639+0000 7f3663400640 1 -- 172.21.15.114:0/3432596102 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f365c08a8f0 con 0x7f3650004d40 2024-03-06T07:28:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.639+0000 7f3663e00640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f364806b4d0 0x7f364806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.639+0000 7f3663e00640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f364806b4d0 0x7f364806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36540017e0 tx=0x7f3654009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.643+0000 7f3663400640 1 -- 172.21.15.114:0/3432596102 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f365c018030 con 0x7f3650004d40 2024-03-06T07:28:23.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.923+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f366c06ce60 con 0x7f3650004d40 2024-03-06T07:28:23.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.924+0000 7f3663400640 1 -- 172.21.15.114:0/3432596102 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f365c0596d0 con 0x7f3650004d40 2024-03-06T07:28:23.926 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f364806b4d0 msgr2=0x7f364806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f364806b4d0 0x7f364806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36540017e0 tx=0x7f3654009040 comp rx=0 tx=0).stop 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 msgr2=0x7f366c0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f366c0ced80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f365c010a00 tx=0x7f365c010ec0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 shutdown_connections 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f366c0ae070 0x7f366c0ae4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f366c0ccfb0 0x7f366c14a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f364806b4d0 0x7f364806d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 --2- 172.21.15.114:0/3432596102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3650004d40 0x7f366c0ced80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 >> 172.21.15.114:0/3432596102 conn(0x7f366c0b1730 msgr2=0x7f366c06d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 shutdown_connections 2024-03-06T07:28:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:23.927+0000 7f3672400640 1 -- 172.21.15.114:0/3432596102 wait complete. 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:23.949 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:28:23.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.950 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:24.008 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42579sec preferred_lft 42579sec 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::ec4:7aff:feea:b04b/64 scope link proto kernel_ll 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 3a:fd:c5:be:6c:3d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:24.009 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.010 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::4079:d2ff:fe9e:73c7/64 scope link proto kernel_ll 2024-03-06T07:28:24.010 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.010 INFO:teuthology.orchestra.run.smithi177.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:24.010 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 3a:fd:c5:be:6c:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:28:24.010 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::38fd:c5ff:febe:6c3d/64 scope link proto kernel_ll 2024-03-06T07:28:24.010 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.011 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:24.101 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-06T07:28:24.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.105 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:24.156 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:24.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.157 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:24.157 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:24.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:24.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:24.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:24.157 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:24.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.332 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-03-06T07:28:24.381 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:28:24.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.382 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list-id 2024-03-06T07:28:24.432 INFO:teuthology.orchestra.run.smithi177.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:28:24.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.433 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:24.433 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:24.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:28:24.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:28:24.433 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:24.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:28:24.666 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:24.697 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 02:e7:65:8e:1e:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::e7:65ff:fe8e:1e9d/64 scope link proto kernel_ll 2024-03-06T07:28:24.698 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.699 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 02:e7:65:8e:1e:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::e7:65ff:fe8e:1e9d/64 scope link proto kernel_ll 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:24.800 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:28:24.800 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:25.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:25.067 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-06T07:28:25.068 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-06T07:28:25.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.1 up 2024-03-06T07:28:25.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:28:25.068 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-06T07:28:25.233 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:25.233 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:25.272 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:28:25.273 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:25.273 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:25.363 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-03-06T07:28:25.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:25.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:25.498 INFO:teuthology.orchestra.run.smithi177.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:25.498 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:28:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: trying mount with new device syntax: 3@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:28:25.535 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:28:25.539 INFO:teuthology.orchestra.run.smithi177.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-06T07:28:25.539 INFO:teuthology.orchestra.run.smithi177.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:25.539 INFO:teuthology.orchestra.run.smithi177.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:25.539 INFO:teuthology.orchestra.run.smithi177.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:25.539 INFO:teuthology.orchestra.run.smithi177.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:25.543 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:25.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:25.543 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:25.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:25.619 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:25.667 INFO:teuthology.orchestra.run.smithi177.stdout:client4689 2024-03-06T07:28:25.667 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:25.668 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:25.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:25.762 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:25.815 INFO:teuthology.orchestra.run.smithi177.stdout:client4689 2024-03-06T07:28:25.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:25.815 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:25.866 INFO:teuthology.orchestra.run.smithi177.stdout:client4689 2024-03-06T07:28:25.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:25.866 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/149c34a4-b335-4a4b-8a41-0fe21b0db808.client4689/status') 2024-03-06T07:28:25.957 INFO:teuthology.orchestra.run.smithi177.stdout:instance: client.4689 (3)172.21.15.177:0/3342148983 2024-03-06T07:28:25.957 INFO:teuthology.orchestra.run.smithi177.stdout:blocklisted: false 2024-03-06T07:28:25.958 INFO:teuthology.orchestra.run.smithi177.stderr:0+1 records in 2024-03-06T07:28:25.958 INFO:teuthology.orchestra.run.smithi177.stderr:0+1 records out 2024-03-06T07:28:25.958 INFO:teuthology.orchestra.run.smithi177.stderr:71 bytes copied, 7.7341e-05 s, 918 kB/s 2024-03-06T07:28:25.962 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:28:25.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:26.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.190+0000 7f06ff800640 1 Processor -- start 2024-03-06T07:28:26.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.190+0000 7f06ff800640 1 -- start start 2024-03-06T07:28:26.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.191+0000 7f06ff800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f06f80af820 0x7f06f80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06ff800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06f80b38f0 0x7f06f806b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06ff800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b3d70 con 0x7f06f815e160 2024-03-06T07:28:26.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06ff800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f806b0f0 con 0x7f06f80af820 2024-03-06T07:28:26.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06ff800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b5620 con 0x7f06f80b38f0 2024-03-06T07:28:26.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06fce00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f06f815e160 0x7f06f8162620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40476/0 (socket says 172.21.15.114:40476) 2024-03-06T07:28:26.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06fce00640 1 -- 172.21.15.114:0/4259120471 learned_addr learned my addr 172.21.15.114:0/4259120471 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06fd800640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06f80b38f0 0x7f06f806b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554082894 0 0) 0x7f06f80b3d70 con 0x7f06f815e160 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.192+0000 7f06f7e00640 1 --2- 172.21.15.114:0/4259120471 >> v2:172.21.15.177:3300/0 conn(0x7f06f80af820 0x7f06f80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e0003520 con 0x7f06f815e160 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2690216024 0 0) 0x7f06e0003520 con 0x7f06f815e160 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06f80b38f0 msgr2=0x7f06f806b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06f80b38f0 0x7f06f806b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 >> v2:172.21.15.177:3300/0 conn(0x7f06f80af820 msgr2=0x7f06f80b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 --2- 172.21.15.114:0/4259120471 >> v2:172.21.15.177:3300/0 conn(0x7f06f80af820 0x7f06f80b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f806f1f0 con 0x7f06f815e160 2024-03-06T07:28:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f06ec001670 con 0x7f06f815e160 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 >> v1:172.21.15.114:6789/0 conn(0x7f06f815e160 legacy=0x7f06f8162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06f80b38f0 0x7f06e00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e0003f30 0x7f06e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f806b0f0 con 0x7f06e0004d40 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b5620 con 0x7f06e0003f30 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e000bac0 con 0x7f06f80b38f0 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.193+0000 7f06fd800640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.194+0000 7f06fd800640 1 -- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06f80b38f0 msgr2=0x7f06e00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.194+0000 7f06fd800640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06f80b38f0 0x7f06e00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.194+0000 7f06fd800640 1 -- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e0003f30 msgr2=0x7f06e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.194+0000 7f06fd800640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06e0003f30 0x7f06e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.194+0000 7f06fd800640 1 -- 172.21.15.114:0/4259120471 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e0003520 con 0x7f06e0004d40 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.195+0000 7f06fd800640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f06f000e170 tx=0x7f06e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f0630f3830ffff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.195+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0039070 con 0x7f06e0004d40 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.195+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f002cc20 con 0x7f06e0004d40 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.195+0000 7f06f7400640 1 -- 172.21.15.114:0/4259120471 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0009c10 con 0x7f06e0004d40 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.196+0000 7f06ff800640 1 -- 172.21.15.114:0/4259120471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06e0004d40 msgr2=0x7f06e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.196+0000 7f06ff800640 1 --2- 172.21.15.114:0/4259120471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06e0004d40 0x7f06e00051c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f06f000e170 tx=0x7f06e0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.196+0000 7f06fd800640 1 -- 172.21.15.114:0/4259120471 reap_dead start 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.196+0000 7f06ff800640 1 -- 172.21.15.114:0/4259120471 shutdown_connections 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.196+0000 7f06ff800640 1 -- 172.21.15.114:0/4259120471 >> 172.21.15.114:0/4259120471 conn(0x7f06f80d0340 msgr2=0x7f06f80d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.196+0000 7f06ff800640 1 -- 172.21.15.114:0/4259120471 shutdown_connections 2024-03-06T07:28:26.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.196+0000 7f06ff800640 1 -- 172.21.15.114:0/4259120471 wait complete. 2024-03-06T07:28:26.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.198+0000 7f06ff800640 1 Processor -- start 2024-03-06T07:28:26.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.198+0000 7f06ff800640 1 -- start start 2024-03-06T07:28:26.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06ff800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06e0004d40 0x7f06f8145d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06ff800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06f8154e00 0x7f06f8146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06ff800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f8146d50 0x7f06f8157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06ff800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f806ef10 con 0x7f06f8146d50 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06ff800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f806ec30 con 0x7f06f8154e00 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06ff800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f806b580 con 0x7f06e0004d40 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06fd800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f8146d50 0x7f06f8157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06fd800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f8146d50 0x7f06f8157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53576/0 (socket says 172.21.15.114:53576) 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.199+0000 7f06fd800640 1 -- 172.21.15.114:0/1578390581 learned_addr learned my addr 172.21.15.114:0/1578390581 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.200+0000 7f06fd800640 1 -- 172.21.15.114:0/1578390581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06e0004d40 msgr2=0x7f06f8145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.200+0000 7f06fd800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06e0004d40 0x7f06f8145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.200+0000 7f06fd800640 1 -- 172.21.15.114:0/1578390581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06f8154e00 msgr2=0x7f06f8146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.200+0000 7f06f7e00640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06f8154e00 0x7f06f8146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.200+0000 7f06fd800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06f8154e00 0x7f06f8146810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.200+0000 7f06fd800640 1 -- 172.21.15.114:0/1578390581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f81552c0 con 0x7f06f8146d50 2024-03-06T07:28:26.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.200+0000 7f06fd800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f8146d50 0x7f06f8157fe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f06f000e170 tx=0x7f06f0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.204+0000 7f06f4c00640 1 -- 172.21.15.114:0/1578390581 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0039070 con 0x7f06f8146d50 2024-03-06T07:28:26.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.204+0000 7f06f4c00640 1 -- 172.21.15.114:0/1578390581 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0044070 con 0x7f06f8146d50 2024-03-06T07:28:26.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.204+0000 7f06f4c00640 1 -- 172.21.15.114:0/1578390581 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0034e60 con 0x7f06f8146d50 2024-03-06T07:28:26.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.204+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f8155600 con 0x7f06f8146d50 2024-03-06T07:28:26.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.204+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f8155e40 con 0x7f06f8146d50 2024-03-06T07:28:26.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.205+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f804f330 con 0x7f06f8146d50 2024-03-06T07:28:26.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.207+0000 7f06f4c00640 1 -- 172.21.15.114:0/1578390581 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06f0005d10 con 0x7f06f8146d50 2024-03-06T07:28:26.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.208+0000 7f06f4c00640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06d806b4d0 0x7f06d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.208+0000 7f06f4c00640 1 -- 172.21.15.114:0/1578390581 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06f00ac050 con 0x7f06f8146d50 2024-03-06T07:28:26.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.208+0000 7f06fce00640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06d806b4d0 0x7f06d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.208+0000 7f06fce00640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06d806b4d0 0x7f06d806d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0079b0 tx=0x7f06ec0014c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.212+0000 7f06f4c00640 1 -- 172.21.15.114:0/1578390581 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06f004d020 con 0x7f06f8146d50 2024-03-06T07:28:26.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.437+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f80d11e0 con 0x7f06f8146d50 2024-03-06T07:28:26.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.439+0000 7f06f4c00640 1 -- 172.21.15.114:0/1578390581 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f06f8155600 con 0x7f06f8146d50 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:26.441 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:26.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.441+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06d806b4d0 msgr2=0x7f06d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.441+0000 7f06ff800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06d806b4d0 0x7f06d806d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0079b0 tx=0x7f06ec0014c0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.441+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f8146d50 msgr2=0x7f06f8157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.441+0000 7f06ff800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f8146d50 0x7f06f8157fe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f06f000e170 tx=0x7f06f0009750 comp rx=0 tx=0).stop 2024-03-06T07:28:26.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 shutdown_connections 2024-03-06T07:28:26.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06d806b4d0 0x7f06d806d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f8146d50 0x7f06f8157fe0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06f8154e00 0x7f06f8146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 --2- 172.21.15.114:0/1578390581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f06e0004d40 0x7f06f8145d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 >> 172.21.15.114:0/1578390581 conn(0x7f06f80d0340 msgr2=0x7f06f80ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 shutdown_connections 2024-03-06T07:28:26.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.442+0000 7f06ff800640 1 -- 172.21.15.114:0/1578390581 wait complete. 2024-03-06T07:28:26.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:26.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:26.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.667+0000 7fcfa9200640 1 Processor -- start 2024-03-06T07:28:26.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.668+0000 7fcfa9200640 1 -- start start 2024-03-06T07:28:26.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa9200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcfa40d07f0 0x7fcfa40d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa415e450 0x7fcfa40b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa9200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcfa40d1bf0 0x7fcfa40b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa9200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa40d4930 con 0x7fcfa40d07f0 2024-03-06T07:28:26.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa9200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa40b3730 con 0x7fcfa40d1bf0 2024-03-06T07:28:26.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4067b20 con 0x7fcfa415e450 2024-03-06T07:28:26.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa1e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcfa40d07f0 0x7fcfa40d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa1e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcfa40d07f0 0x7fcfa40d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53584/0 (socket says 172.21.15.114:53584) 2024-03-06T07:28:26.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.669+0000 7fcfa1e00640 1 -- 172.21.15.114:0/2785341382 learned_addr learned my addr 172.21.15.114:0/2785341382 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcf99400640 1 --2- 172.21.15.114:0/2785341382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa415e450 0x7fcfa40b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcfa2800640 1 --2- 172.21.15.114:0/2785341382 >> v2:172.21.15.177:3300/0 conn(0x7fcfa40d1bf0 0x7fcfa40b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcfa1e00640 1 -- 172.21.15.114:0/2785341382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa415e450 msgr2=0x7fcfa40b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcfa1e00640 1 --2- 172.21.15.114:0/2785341382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa415e450 0x7fcfa40b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcfa1e00640 1 -- 172.21.15.114:0/2785341382 >> v2:172.21.15.177:3300/0 conn(0x7fcfa40d1bf0 msgr2=0x7fcfa40b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcfa1e00640 1 --2- 172.21.15.114:0/2785341382 >> v2:172.21.15.177:3300/0 conn(0x7fcfa40d1bf0 0x7fcfa40b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcfa1e00640 1 -- 172.21.15.114:0/2785341382 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa40d1110 con 0x7fcfa40d07f0 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcfa2800640 1 --2- 172.21.15.114:0/2785341382 >> v2:172.21.15.177:3300/0 conn(0x7fcfa40d1bf0 0x7fcfa40b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.670+0000 7fcf99400640 1 --2- 172.21.15.114:0/2785341382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa415e450 0x7fcfa40b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.672+0000 7fcfa1e00640 1 --2- 172.21.15.114:0/2785341382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa40d07f0 0x7fcfa40d0bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40ce4f0 tx=0x7fcf9402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bf7b1fd95c57db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.672+0000 7fcfa1400640 1 -- 172.21.15.114:0/2785341382 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9402e070 con 0x7fcfa40d07f0 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.672+0000 7fcfa1400640 1 -- 172.21.15.114:0/2785341382 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94037070 con 0x7fcfa40d07f0 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.672+0000 7fcfa1400640 1 -- 172.21.15.114:0/2785341382 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94031a00 con 0x7fcfa40d07f0 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 -- 172.21.15.114:0/2785341382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa40d07f0 msgr2=0x7fcfa40d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 --2- 172.21.15.114:0/2785341382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa40d07f0 0x7fcfa40d0bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40ce4f0 tx=0x7fcf9402b880 comp rx=0 tx=0).stop 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 -- 172.21.15.114:0/2785341382 shutdown_connections 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 --2- 172.21.15.114:0/2785341382 >> v2:172.21.15.177:3300/0 conn(0x7fcfa40d1bf0 0x7fcfa40b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 --2- 172.21.15.114:0/2785341382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa415e450 0x7fcfa40b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 --2- 172.21.15.114:0/2785341382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa40d07f0 0x7fcfa40d0bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 -- 172.21.15.114:0/2785341382 >> 172.21.15.114:0/2785341382 conn(0x7fcfa40d3e10 msgr2=0x7fcfa40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 -- 172.21.15.114:0/2785341382 shutdown_connections 2024-03-06T07:28:26.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.673+0000 7fcfa9200640 1 -- 172.21.15.114:0/2785341382 wait complete. 2024-03-06T07:28:26.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.675+0000 7fcfa9200640 1 Processor -- start 2024-03-06T07:28:26.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.676+0000 7fcfa9200640 1 -- start start 2024-03-06T07:28:26.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa40d07f0 0x7fcfa414f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfa40d1bf0 0x7fcfa40caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa415e450 0x7fcfa4160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa9200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40b3c80 con 0x7fcfa415e450 2024-03-06T07:28:26.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa9200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40b39a0 con 0x7fcfa40d1bf0 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40b3f60 con 0x7fcfa40d07f0 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa415e450 0x7fcfa4160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa415e450 0x7fcfa4160f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53586/0 (socket says 172.21.15.114:53586) 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.677+0000 7fcfa2800640 1 -- 172.21.15.114:0/1794517202 learned_addr learned my addr 172.21.15.114:0/1794517202 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa1e00640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa40d07f0 0x7fcfa414f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcf99400640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfa40d1bf0 0x7fcfa40caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa2800640 1 -- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa40d07f0 msgr2=0x7fcfa414f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa2800640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa40d07f0 0x7fcfa414f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa2800640 1 -- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfa40d1bf0 msgr2=0x7fcfa40caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa2800640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfa40d1bf0 0x7fcfa40caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa2800640 1 -- 172.21.15.114:0/1794517202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa415fef0 con 0x7fcfa415e450 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa2800640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa415e450 0x7fcfa4160f90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00cf20 tx=0x7fcf9c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcf9aa00640 1 -- 172.21.15.114:0/1794517202 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c01a070 con 0x7fcfa415e450 2024-03-06T07:28:26.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcf9aa00640 1 -- 172.21.15.114:0/1794517202 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c00d840 con 0x7fcfa415e450 2024-03-06T07:28:26.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa41601d0 con 0x7fcfa415e450 2024-03-06T07:28:26.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.678+0000 7fcf9aa00640 1 -- 172.21.15.114:0/1794517202 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00dc30 con 0x7fcfa415e450 2024-03-06T07:28:26.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.679+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa4160790 con 0x7fcfa415e450 2024-03-06T07:28:26.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.680+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa404f330 con 0x7fcfa415e450 2024-03-06T07:28:26.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.682+0000 7fcf9aa00640 1 -- 172.21.15.114:0/1794517202 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf9c01d070 con 0x7fcfa415e450 2024-03-06T07:28:26.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.682+0000 7fcf9aa00640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf8006b500 0x7fcf8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.683+0000 7fcf9aa00640 1 -- 172.21.15.114:0/1794517202 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf9c08ab60 con 0x7fcfa415e450 2024-03-06T07:28:26.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.683+0000 7fcfa1e00640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf8006b500 0x7fcf8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.683+0000 7fcfa1e00640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf8006b500 0x7fcf8006d9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf94036000 tx=0x7fcf9403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.687+0000 7fcf9aa00640 1 -- 172.21.15.114:0/1794517202 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf9c0598b0 con 0x7fcfa415e450 2024-03-06T07:28:26.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.885+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfa40d21d0 con 0x7fcfa415e450 2024-03-06T07:28:26.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.886+0000 7fcf9aa00640 1 -- 172.21.15.114:0/1794517202 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf9c0591b0 con 0x7fcfa415e450 2024-03-06T07:28:26.888 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.888+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf8006b500 msgr2=0x7fcf8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.888+0000 7fcfa9200640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf8006b500 0x7fcf8006d9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf94036000 tx=0x7fcf9403d040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.888+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa415e450 msgr2=0x7fcfa4160f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.888+0000 7fcfa9200640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa415e450 0x7fcfa4160f90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c00cf20 tx=0x7fcf9c00a710 comp rx=0 tx=0).stop 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 shutdown_connections 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf8006b500 0x7fcf8006d9c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfa415e450 0x7fcfa4160f90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcfa40d1bf0 0x7fcfa40caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 --2- 172.21.15.114:0/1794517202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcfa40d07f0 0x7fcfa414f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 >> 172.21.15.114:0/1794517202 conn(0x7fcfa40d3e10 msgr2=0x7fcfa40b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 shutdown_connections 2024-03-06T07:28:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:26.889+0000 7fcfa9200640 1 -- 172.21.15.114:0/1794517202 wait complete. 2024-03-06T07:28:26.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.116+0000 7f5adf800640 1 Processor -- start 2024-03-06T07:28:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.117+0000 7f5adf800640 1 -- start start 2024-03-06T07:28:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5adf800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad815e450 0x7f5ad80cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5adf800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ad80b5490 0x7f5ad80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5adf800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad80b3730 con 0x7f5ad80d07f0 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5adf800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad80d4930 con 0x7f5ad80b5490 2024-03-06T07:28:27.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5adf800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad8067b20 con 0x7f5ad815e450 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5adce00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5ad80d07f0 0x7f5ad80d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40486/0 (socket says 172.21.15.114:40486) 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5adce00640 1 -- 172.21.15.114:0/991091801 learned_addr learned my addr 172.21.15.114:0/991091801 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.118+0000 7f5add800640 1 --2- 172.21.15.114:0/991091801 >> v2:172.21.15.114:3300/0 conn(0x7f5ad80b5490 0x7f5ad80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.119+0000 7f5ad7e00640 1 --2- 172.21.15.114:0/991091801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad815e450 0x7f5ad80cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.119+0000 7f5add800640 1 -- 172.21.15.114:0/991091801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad815e450 msgr2=0x7f5ad80cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.119+0000 7f5add800640 1 --2- 172.21.15.114:0/991091801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad815e450 0x7f5ad80cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.119+0000 7f5add800640 1 -- 172.21.15.114:0/991091801 >> v1:172.21.15.114:6789/0 conn(0x7f5ad80d07f0 legacy=0x7f5ad80d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.119+0000 7f5add800640 1 -- 172.21.15.114:0/991091801 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad80d4b80 con 0x7f5ad80b5490 2024-03-06T07:28:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.119+0000 7f5ad7400640 1 -- 172.21.15.114:0/991091801 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315045104 0 0) 0x7f5ad80b3730 con 0x7f5ad80d07f0 2024-03-06T07:28:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.120+0000 7f5add800640 1 --2- 172.21.15.114:0/991091801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad80b5490 0x7f5ad80cef60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009d90 tx=0x7f5ad002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f9893119560e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5ad7400640 1 -- 172.21.15.114:0/991091801 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0039070 con 0x7f5ad80b5490 2024-03-06T07:28:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5ad7400640 1 -- 172.21.15.114:0/991091801 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad002cbc0 con 0x7f5ad80b5490 2024-03-06T07:28:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5ad7400640 1 -- 172.21.15.114:0/991091801 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0034440 con 0x7f5ad80b5490 2024-03-06T07:28:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5adf800640 1 -- 172.21.15.114:0/991091801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad80b5490 msgr2=0x7f5ad80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5adf800640 1 --2- 172.21.15.114:0/991091801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad80b5490 0x7f5ad80cef60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009d90 tx=0x7f5ad002be10 comp rx=0 tx=0).stop 2024-03-06T07:28:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5adf800640 1 -- 172.21.15.114:0/991091801 shutdown_connections 2024-03-06T07:28:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5adf800640 1 --2- 172.21.15.114:0/991091801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad80b5490 0x7f5ad80cef60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5adf800640 1 --2- 172.21.15.114:0/991091801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad815e450 0x7f5ad80cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.121+0000 7f5adf800640 1 -- 172.21.15.114:0/991091801 >> 172.21.15.114:0/991091801 conn(0x7f5ad80d3e10 msgr2=0x7f5ad80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.122+0000 7f5adf800640 1 -- 172.21.15.114:0/991091801 shutdown_connections 2024-03-06T07:28:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.122+0000 7f5adf800640 1 -- 172.21.15.114:0/991091801 wait complete. 2024-03-06T07:28:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.124+0000 7f5adf800640 1 Processor -- start 2024-03-06T07:28:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.124+0000 7f5adf800640 1 -- start start 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5adf800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad80b5490 0x7f5ad80c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5adf800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d07f0 0x7f5ad80b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5adf800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad815e450 0x7f5ad80b9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5adf800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad80b3f60 con 0x7f5ad815e450 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5adf800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad80b3c80 con 0x7f5ad80d07f0 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5adf800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad80d1bf0 con 0x7f5ad80b5490 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5add800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad815e450 0x7f5ad80b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5add800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad815e450 0x7f5ad80b9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53606/0 (socket says 172.21.15.114:53606) 2024-03-06T07:28:27.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.125+0000 7f5add800640 1 -- 172.21.15.114:0/184316004 learned_addr learned my addr 172.21.15.114:0/184316004 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5adce00640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad80b5490 0x7f5ad80c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5ad7e00640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d07f0 0x7f5ad80b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5add800640 1 -- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad80b5490 msgr2=0x7f5ad80c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5add800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad80b5490 0x7f5ad80c7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5add800640 1 -- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d07f0 msgr2=0x7f5ad80b8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5add800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d07f0 0x7f5ad80b8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5add800640 1 -- 172.21.15.114:0/184316004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad80ca7c0 con 0x7f5ad815e450 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5ad7e00640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d07f0 0x7f5ad80b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.126+0000 7f5add800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad815e450 0x7f5ad80b9370 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0037040 tx=0x7f5ad0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.127+0000 7f5ad4c00640 1 -- 172.21.15.114:0/184316004 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0039050 con 0x7f5ad815e450 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.127+0000 7f5ad4c00640 1 -- 172.21.15.114:0/184316004 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0045070 con 0x7f5ad815e450 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.127+0000 7f5ad4c00640 1 -- 172.21.15.114:0/184316004 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad003e830 con 0x7f5ad815e450 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.127+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad80cb180 con 0x7f5ad815e450 2024-03-06T07:28:27.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.127+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad80cb740 con 0x7f5ad815e450 2024-03-06T07:28:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.128+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad8068230 con 0x7f5ad815e450 2024-03-06T07:28:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.129+0000 7f5ad4c00640 1 -- 172.21.15.114:0/184316004 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ad003e9d0 con 0x7f5ad815e450 2024-03-06T07:28:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.131+0000 7f5ad4c00640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5ab806b4d0 0x7f5ab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.131+0000 7f5ad4c00640 1 -- 172.21.15.114:0/184316004 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ad00ad3b0 con 0x7f5ad815e450 2024-03-06T07:28:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.131+0000 7f5adce00640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5ab806b4d0 0x7f5ab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.132+0000 7f5adce00640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5ab806b4d0 0x7f5ab806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5acc000f60 tx=0x7f5acc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.135+0000 7f5ad4c00640 1 -- 172.21.15.114:0/184316004 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ad8068230 con 0x7f5ad815e450 2024-03-06T07:28:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.334+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ad804f330 con 0x7f5ad815e450 2024-03-06T07:28:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.334+0000 7f5ad4c00640 1 -- 172.21.15.114:0/184316004 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ad007c2b0 con 0x7f5ad815e450 2024-03-06T07:28:27.336 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5ab806b4d0 msgr2=0x7f5ab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5ab806b4d0 0x7f5ab806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5acc000f60 tx=0x7f5acc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad815e450 msgr2=0x7f5ad80b9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad815e450 0x7f5ad80b9370 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0037040 tx=0x7f5ad0004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 shutdown_connections 2024-03-06T07:28:27.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5ab806b4d0 0x7f5ab806d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ad815e450 0x7f5ad80b9370 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ad80d07f0 0x7f5ad80b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 --2- 172.21.15.114:0/184316004 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ad80b5490 0x7f5ad80c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 >> 172.21.15.114:0/184316004 conn(0x7f5ad80d3e10 msgr2=0x7f5ad806c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 shutdown_connections 2024-03-06T07:28:27.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.337+0000 7f5adf800640 1 -- 172.21.15.114:0/184316004 wait complete. 2024-03-06T07:28:27.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.555+0000 7f9479800640 1 Processor -- start 2024-03-06T07:28:27.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.556+0000 7f9479800640 1 -- start start 2024-03-06T07:28:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.557+0000 7f9479800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f947415e130 0x7f94740cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.557+0000 7f9479800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94740d13e0 con 0x7f94740d25b0 2024-03-06T07:28:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.557+0000 7f9479800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94740cead0 con 0x7f94740cfaf0 2024-03-06T07:28:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.557+0000 7f9479800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94740cf120 con 0x7f947415e130 2024-03-06T07:28:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.557+0000 7f9473e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f947415e130 0x7f94740cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.558+0000 7f9473e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f947415e130 0x7f94740cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53620/0 (socket says 172.21.15.114:53620) 2024-03-06T07:28:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.558+0000 7f9473e00640 1 -- 172.21.15.114:0/1065614873 learned_addr learned my addr 172.21.15.114:0/1065614873 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:27.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.558+0000 7f9473e00640 1 -- 172.21.15.114:0/1065614873 >> v1:172.21.15.177:6789/0 conn(0x7f94740cfaf0 legacy=0x7f94740d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.558+0000 7f9473e00640 1 -- 172.21.15.114:0/1065614873 >> v1:172.21.15.114:6789/0 conn(0x7f94740d25b0 legacy=0x7f94740c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.558+0000 7f9473e00640 1 -- 172.21.15.114:0/1065614873 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740cedb0 con 0x7f947415e130 2024-03-06T07:28:27.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.558+0000 7f9473400640 1 -- 172.21.15.114:0/1065614873 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022691720 0 0) 0x7f94740d13e0 con 0x7f94740d25b0 2024-03-06T07:28:27.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.559+0000 7f9473e00640 1 --2- 172.21.15.114:0/1065614873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947415e130 0x7f94740cf5b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9464009870 tx=0x7f946402ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7ba4c557127546 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9473400640 1 -- 172.21.15.114:0/1065614873 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946402f870 con 0x7f947415e130 2024-03-06T07:28:27.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9473400640 1 -- 172.21.15.114:0/1065614873 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946402fa10 con 0x7f947415e130 2024-03-06T07:28:27.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9473400640 1 -- 172.21.15.114:0/1065614873 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946402fde0 con 0x7f947415e130 2024-03-06T07:28:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9479800640 1 -- 172.21.15.114:0/1065614873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947415e130 msgr2=0x7f94740cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9479800640 1 --2- 172.21.15.114:0/1065614873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947415e130 0x7f94740cf5b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9464009870 tx=0x7f946402ba50 comp rx=0 tx=0).stop 2024-03-06T07:28:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9479800640 1 -- 172.21.15.114:0/1065614873 shutdown_connections 2024-03-06T07:28:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9479800640 1 --2- 172.21.15.114:0/1065614873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f947415e130 0x7f94740cf5b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9479800640 1 -- 172.21.15.114:0/1065614873 >> 172.21.15.114:0/1065614873 conn(0x7f947409d070 msgr2=0x7f94740d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9479800640 1 -- 172.21.15.114:0/1065614873 shutdown_connections 2024-03-06T07:28:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.560+0000 7f9479800640 1 -- 172.21.15.114:0/1065614873 wait complete. 2024-03-06T07:28:27.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.562+0000 7f9479800640 1 Processor -- start 2024-03-06T07:28:27.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.562+0000 7f9479800640 1 -- start start 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9479800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94740cfaf0 0x7f94741f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9479800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94740d25b0 0x7f94740caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9479800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f947415e130 0x7f94740cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9479800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c8200 con 0x7f94740cfaf0 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9479800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740c8080 con 0x7f94740d25b0 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9479800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740d06b0 con 0x7f947415e130 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9473e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94740cfaf0 0x7f94741f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9473e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94740cfaf0 0x7f94741f0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53630/0 (socket says 172.21.15.114:53630) 2024-03-06T07:28:27.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.563+0000 7f9473e00640 1 -- 172.21.15.114:0/1515816758 learned_addr learned my addr 172.21.15.114:0/1515816758 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9473e00640 1 -- 172.21.15.114:0/1515816758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f947415e130 msgr2=0x7f94740cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9473e00640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f947415e130 0x7f94740cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9473e00640 1 -- 172.21.15.114:0/1515816758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94740d25b0 msgr2=0x7f94740caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9473e00640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94740d25b0 0x7f94740caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9473e00640 1 -- 172.21.15.114:0/1515816758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740cb9b0 con 0x7f94740cfaf0 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9473e00640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94740cfaf0 0x7f94741f0f60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f94640090b0 tx=0x7f9464004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9470c00640 1 -- 172.21.15.114:0/1515816758 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464047070 con 0x7f94740cfaf0 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9470c00640 1 -- 172.21.15.114:0/1515816758 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946402f830 con 0x7f94740cfaf0 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9470c00640 1 -- 172.21.15.114:0/1515816758 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464004270 con 0x7f94740cfaf0 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.564+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94740cbc90 con 0x7f94740cfaf0 2024-03-06T07:28:27.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.565+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94740cc250 con 0x7f94740cfaf0 2024-03-06T07:28:27.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.566+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947404f330 con 0x7f94740cfaf0 2024-03-06T07:28:27.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.567+0000 7f9470c00640 1 -- 172.21.15.114:0/1515816758 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9464003610 con 0x7f94740cfaf0 2024-03-06T07:28:27.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.567+0000 7f9470c00640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.568+0000 7f9470c00640 1 -- 172.21.15.114:0/1515816758 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94640ac4a0 con 0x7f94740cfaf0 2024-03-06T07:28:27.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.568+0000 7f946be00640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.568+0000 7f946be00640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f945c000f60 tx=0x7f945c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.572+0000 7f9470c00640 1 -- 172.21.15.114:0/1515816758 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9464033030 con 0x7f94740cfaf0 2024-03-06T07:28:27.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.766+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f947409d690 con 0x7f94740cfaf0 2024-03-06T07:28:27.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.767+0000 7f9470c00640 1 -- 172.21.15.114:0/1515816758 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f946402f9e0 con 0x7f94740cfaf0 2024-03-06T07:28:27.769 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f945006b4d0 msgr2=0x7f945006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f945c000f60 tx=0x7f945c0079b0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94740cfaf0 msgr2=0x7f94741f0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94740cfaf0 0x7f94741f0f60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f94640090b0 tx=0x7f9464004730 comp rx=0 tx=0).stop 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 shutdown_connections 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f947415e130 0x7f94740cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94740d25b0 0x7f94740caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 --2- 172.21.15.114:0/1515816758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94740cfaf0 0x7f94741f0f60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 >> 172.21.15.114:0/1515816758 conn(0x7f947409d070 msgr2=0x7f947406c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 shutdown_connections 2024-03-06T07:28:27.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.770+0000 7f9479800640 1 -- 172.21.15.114:0/1515816758 wait complete. 2024-03-06T07:28:27.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:27.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.987+0000 7f399da00640 1 Processor -- start 2024-03-06T07:28:27.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.987+0000 7f399da00640 1 -- start start 2024-03-06T07:28:27.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.989+0000 7f399da00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3998161580 0x7f39980d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.989+0000 7f399da00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f39980d1e90 0x7f39980d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.989+0000 7f399da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39980d28a0 0x7f39980d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.989+0000 7f399da00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3998161960 con 0x7f3998161580 2024-03-06T07:28:27.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.989+0000 7f399da00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39980cf6f0 con 0x7f39980d1e90 2024-03-06T07:28:27.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.989+0000 7f399da00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39980cf9d0 con 0x7f39980d28a0 2024-03-06T07:28:27.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3998161580 0x7f39980d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3998161580 0x7f39980d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55040/0 (socket says 172.21.15.114:55040) 2024-03-06T07:28:27.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 -- 172.21.15.114:0/3578786104 learned_addr learned my addr 172.21.15.114:0/3578786104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:27.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f399d000640 1 --2- 172.21.15.114:0/3578786104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39980d28a0 0x7f39980d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f398fe00640 1 --2- 172.21.15.114:0/3578786104 >> v2:172.21.15.177:3300/0 conn(0x7f39980d1e90 0x7f39980d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 -- 172.21.15.114:0/3578786104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39980d28a0 msgr2=0x7f39980d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 --2- 172.21.15.114:0/3578786104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39980d28a0 0x7f39980d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 -- 172.21.15.114:0/3578786104 >> v2:172.21.15.177:3300/0 conn(0x7f39980d1e90 msgr2=0x7f39980d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 --2- 172.21.15.114:0/3578786104 >> v2:172.21.15.177:3300/0 conn(0x7f39980d1e90 0x7f39980d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.990+0000 7f3997e00640 1 -- 172.21.15.114:0/3578786104 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39980cfcb0 con 0x7f3998161580 2024-03-06T07:28:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.992+0000 7f3997e00640 1 --2- 172.21.15.114:0/3578786104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998161580 0x7f39980d1950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f399808c9d0 tx=0x7f398802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925b6a1295b68284 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.992+0000 7f3997400640 1 -- 172.21.15.114:0/3578786104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988035020 con 0x7f3998161580 2024-03-06T07:28:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.992+0000 7f3997400640 1 -- 172.21.15.114:0/3578786104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3988031870 con 0x7f3998161580 2024-03-06T07:28:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.992+0000 7f3997400640 1 -- 172.21.15.114:0/3578786104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988031c40 con 0x7f3998161580 2024-03-06T07:28:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 -- 172.21.15.114:0/3578786104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998161580 msgr2=0x7f39980d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 --2- 172.21.15.114:0/3578786104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998161580 0x7f39980d1950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f399808c9d0 tx=0x7f398802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 -- 172.21.15.114:0/3578786104 shutdown_connections 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 --2- 172.21.15.114:0/3578786104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39980d28a0 0x7f39980d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 --2- 172.21.15.114:0/3578786104 >> v2:172.21.15.177:3300/0 conn(0x7f39980d1e90 0x7f39980d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 --2- 172.21.15.114:0/3578786104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3998161580 0x7f39980d1950 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 -- 172.21.15.114:0/3578786104 >> 172.21.15.114:0/3578786104 conn(0x7f39980d3d80 msgr2=0x7f39980d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 -- 172.21.15.114:0/3578786104 shutdown_connections 2024-03-06T07:28:27.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.993+0000 7f399da00640 1 -- 172.21.15.114:0/3578786104 wait complete. 2024-03-06T07:28:27.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.994+0000 7f399da00640 1 Processor -- start 2024-03-06T07:28:27.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.995+0000 7f399da00640 1 -- start start 2024-03-06T07:28:27.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.995+0000 7f399da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39980d1e90 0x7f3998152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.995+0000 7f399da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39980d28a0 0x7f39980b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.995+0000 7f399da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998161580 0x7f39980c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f399da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980cf3a0 con 0x7f39980d28a0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f399da00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980b39a0 con 0x7f39980d1e90 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f399da00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39980d0c20 con 0x7f3998161580 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39980d28a0 0x7f39980b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39980d28a0 0x7f39980b8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55052/0 (socket says 172.21.15.114:55052) 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 -- 172.21.15.114:0/3964142855 learned_addr learned my addr 172.21.15.114:0/3964142855 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f3997e00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39980d1e90 0x7f3998152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f399d000640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998161580 0x7f39980c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 -- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998161580 msgr2=0x7f39980c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998161580 0x7f39980c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 -- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39980d1e90 msgr2=0x7f3998152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39980d1e90 0x7f3998152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f398fe00640 1 -- 172.21.15.114:0/3964142855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998152f60 con 0x7f39980d28a0 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.996+0000 7f3997e00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39980d1e90 0x7f3998152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.997+0000 7f398fe00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39980d28a0 0x7f39980b8780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f398000d830 tx=0x7f398000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.997+0000 7f3994c00640 1 -- 172.21.15.114:0/3964142855 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000b6c0 con 0x7f39980d28a0 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.997+0000 7f3994c00640 1 -- 172.21.15.114:0/3964142855 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980019070 con 0x7f39980d28a0 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.997+0000 7f3994c00640 1 -- 172.21.15.114:0/3964142855 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000ba80 con 0x7f39980d28a0 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.997+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39980cc380 con 0x7f39980d28a0 2024-03-06T07:28:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.997+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39980cc940 con 0x7f39980d28a0 2024-03-06T07:28:28.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.998+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399804f330 con 0x7f39980d28a0 2024-03-06T07:28:28.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:27.999+0000 7f3994c00640 1 -- 172.21.15.114:0/3964142855 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f398001e460 con 0x7f39980d28a0 2024-03-06T07:28:28.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.000+0000 7f3994c00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f397406b4d0 0x7f397406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.000+0000 7f3994c00640 1 -- 172.21.15.114:0/3964142855 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f398008a950 con 0x7f39980d28a0 2024-03-06T07:28:28.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.000+0000 7f3997e00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f397406b4d0 0x7f397406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.001+0000 7f3997e00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f397406b4d0 0x7f397406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3988036000 tx=0x7f398803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.004+0000 7f3994c00640 1 -- 172.21.15.114:0/3964142855 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3980059730 con 0x7f39980d28a0 2024-03-06T07:28:28.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.202+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39980ce1b0 con 0x7f39980d28a0 2024-03-06T07:28:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.205+0000 7f3994c00640 1 -- 172.21.15.114:0/3964142855 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f3980059030 con 0x7f39980d28a0 2024-03-06T07:28:28.207 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:28.208 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f397406b4d0 msgr2=0x7f397406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f397406b4d0 0x7f397406d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3988036000 tx=0x7f398803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39980d28a0 msgr2=0x7f39980b8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39980d28a0 0x7f39980b8780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f398000d830 tx=0x7f398000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 shutdown_connections 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3998161580 0x7f39980c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f397406b4d0 0x7f397406d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39980d28a0 0x7f39980b8780 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 --2- 172.21.15.114:0/3964142855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39980d1e90 0x7f3998152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.208+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 >> 172.21.15.114:0/3964142855 conn(0x7f39980d3d80 msgr2=0x7f39980d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.209+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 shutdown_connections 2024-03-06T07:28:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.209+0000 7f399da00640 1 -- 172.21.15.114:0/3964142855 wait complete. 2024-03-06T07:28:28.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:28.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:28.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.430+0000 7f848be00640 1 Processor -- start 2024-03-06T07:28:28.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.430+0000 7f848be00640 1 -- start start 2024-03-06T07:28:28.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f848be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f848415bb90 0x7f848406dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f848be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415efa0 0x7f84840cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f848be00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8484071ea0 con 0x7f848406afc0 2024-03-06T07:28:28.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f848be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848415bf70 con 0x7f848415bb90 2024-03-06T07:28:28.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f848be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848406e380 con 0x7f848415efa0 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f8489400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f848415bb90 0x7f848406dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f8489400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f848415bb90 0x7f848406dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55064/0 (socket says 172.21.15.114:55064) 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f8489400640 1 -- 172.21.15.114:0/1303675939 learned_addr learned my addr 172.21.15.114:0/1303675939 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.432+0000 7f8489e00640 1 --1- 172.21.15.114:0/1303675939 >> v1:172.21.15.114:6789/0 conn(0x7f848406afc0 0x7f84840723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49130/0 (socket says 172.21.15.114:49130) 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.433+0000 7f8488a00640 1 --2- 172.21.15.114:0/1303675939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415efa0 0x7f84840cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.433+0000 7f8489400640 1 -- 172.21.15.114:0/1303675939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415efa0 msgr2=0x7f84840cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.433+0000 7f8489400640 1 --2- 172.21.15.114:0/1303675939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415efa0 0x7f84840cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.433+0000 7f8489400640 1 -- 172.21.15.114:0/1303675939 >> v1:172.21.15.114:6789/0 conn(0x7f848406afc0 legacy=0x7f84840723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.433+0000 7f8489400640 1 -- 172.21.15.114:0/1303675939 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848406e660 con 0x7f848415bb90 2024-03-06T07:28:28.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.433+0000 7f8483e00640 1 -- 172.21.15.114:0/1303675939 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961962386 0 0) 0x7f8484071ea0 con 0x7f848406afc0 2024-03-06T07:28:28.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.434+0000 7f8489400640 1 --2- 172.21.15.114:0/1303675939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415bb90 0x7f848406dcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8478009870 tx=0x7f847802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed8b86fd1080843 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f8483e00640 1 -- 172.21.15.114:0/1303675939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478034070 con 0x7f848415bb90 2024-03-06T07:28:28.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f8483e00640 1 -- 172.21.15.114:0/1303675939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478030840 con 0x7f848415bb90 2024-03-06T07:28:28.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f8483e00640 1 -- 172.21.15.114:0/1303675939 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478030c10 con 0x7f848415bb90 2024-03-06T07:28:28.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f848be00640 1 -- 172.21.15.114:0/1303675939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415bb90 msgr2=0x7f848406dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f848be00640 1 --2- 172.21.15.114:0/1303675939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415bb90 0x7f848406dcf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8478009870 tx=0x7f847802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:28.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f848be00640 1 -- 172.21.15.114:0/1303675939 shutdown_connections 2024-03-06T07:28:28.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f848be00640 1 --2- 172.21.15.114:0/1303675939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415efa0 0x7f84840cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f848be00640 1 --2- 172.21.15.114:0/1303675939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415bb90 0x7f848406dcf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.435+0000 7f848be00640 1 -- 172.21.15.114:0/1303675939 >> 172.21.15.114:0/1303675939 conn(0x7f84840b27d0 msgr2=0x7f84840b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.436+0000 7f848be00640 1 -- 172.21.15.114:0/1303675939 shutdown_connections 2024-03-06T07:28:28.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.436+0000 7f848be00640 1 -- 172.21.15.114:0/1303675939 wait complete. 2024-03-06T07:28:28.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.437+0000 7f848be00640 1 Processor -- start 2024-03-06T07:28:28.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.437+0000 7f848be00640 1 -- start start 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.438+0000 7f848be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f848406afc0 0x7f8484154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.438+0000 7f848be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415bb90 0x7f84840b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.438+0000 7f848be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415efa0 0x7f84841f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.438+0000 7f848be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848415f400 con 0x7f848415efa0 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.438+0000 7f848be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848406ec20 con 0x7f848406afc0 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.438+0000 7f848be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84840cf160 con 0x7f848415bb90 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415efa0 0x7f84841f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415efa0 0x7f84841f3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55070/0 (socket says 172.21.15.114:55070) 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 -- 172.21.15.114:0/3514937303 learned_addr learned my addr 172.21.15.114:0/3514937303 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8488a00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415bb90 0x7f84840b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489400640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f848406afc0 0x7f8484154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 -- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415bb90 msgr2=0x7f84840b6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415bb90 0x7f84840b6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 -- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f848406afc0 msgr2=0x7f8484154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f848406afc0 0x7f8484154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489e00640 1 -- 172.21.15.114:0/3514937303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84840b67a0 con 0x7f848415efa0 2024-03-06T07:28:28.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.439+0000 7f8489400640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f848406afc0 0x7f8484154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.440+0000 7f8489e00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415efa0 0x7f84841f3da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f847c00c7a0 tx=0x7f847c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.440+0000 7f8481600640 1 -- 172.21.15.114:0/3514937303 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c018070 con 0x7f848415efa0 2024-03-06T07:28:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.440+0000 7f8481600640 1 -- 172.21.15.114:0/3514937303 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c00a640 con 0x7f848415efa0 2024-03-06T07:28:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.440+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84840b53c0 con 0x7f848415efa0 2024-03-06T07:28:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.440+0000 7f8481600640 1 -- 172.21.15.114:0/3514937303 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c00aab0 con 0x7f848415efa0 2024-03-06T07:28:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.440+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84841550a0 con 0x7f848415efa0 2024-03-06T07:28:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.441+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848404f330 con 0x7f848415efa0 2024-03-06T07:28:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.442+0000 7f8481600640 1 -- 172.21.15.114:0/3514937303 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f847c01b070 con 0x7f848415efa0 2024-03-06T07:28:28.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.443+0000 7f8481600640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.443+0000 7f8481600640 1 -- 172.21.15.114:0/3514937303 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f847c0898c0 con 0x7f848415efa0 2024-03-06T07:28:28.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.443+0000 7f8489400640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.444+0000 7f8489400640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f846406b4d0 0x7f846406d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8478036000 tx=0x7f8478008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.448+0000 7f8481600640 1 -- 172.21.15.114:0/3514937303 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84840b67a0 con 0x7f848415efa0 2024-03-06T07:28:28.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.644+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f848415d7c0 con 0x7f848415efa0 2024-03-06T07:28:28.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.645+0000 7f8481600640 1 -- 172.21.15.114:0/3514937303 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f847c0586a0 con 0x7f848415efa0 2024-03-06T07:28:28.646 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:28.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.648+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f846406b4d0 msgr2=0x7f846406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.648+0000 7f848be00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f846406b4d0 0x7f846406d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8478036000 tx=0x7f8478008030 comp rx=0 tx=0).stop 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.648+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415efa0 msgr2=0x7f84841f3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.648+0000 7f848be00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415efa0 0x7f84841f3da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f847c00c7a0 tx=0x7f847c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 shutdown_connections 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f848415efa0 0x7f84841f3da0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f848415bb90 0x7f84840b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 --2- 172.21.15.114:0/3514937303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f848406afc0 0x7f8484154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 >> 172.21.15.114:0/3514937303 conn(0x7f84840b27d0 msgr2=0x7f84840cf3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 shutdown_connections 2024-03-06T07:28:28.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.649+0000 7f848be00640 1 -- 172.21.15.114:0/3514937303 wait complete. 2024-03-06T07:28:28.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.875+0000 7f52e6a00640 1 Processor -- start 2024-03-06T07:28:28.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.876+0000 7f52e6a00640 1 -- start start 2024-03-06T07:28:28.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.877+0000 7f52e6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52e015fc10 0x7f52e0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.877+0000 7f52e6a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00b46c0 con 0x7f52e00b3f70 2024-03-06T07:28:28.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.877+0000 7f52e6a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00b49a0 con 0x7f52e015fff0 2024-03-06T07:28:28.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.877+0000 7f52e6a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e0072c90 con 0x7f52e015fc10 2024-03-06T07:28:28.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.877+0000 7f52dfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52e015fc10 0x7f52e0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.878+0000 7f52e4a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f52e00b3f70 0x7f52e00ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49146/0 (socket says 172.21.15.114:49146) 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.878+0000 7f52e4a00640 1 -- 172.21.15.114:0/557291107 learned_addr learned my addr 172.21.15.114:0/557291107 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.878+0000 7f52dfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52e015fc10 0x7f52e0070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55078/0 (socket says 172.21.15.114:55078) 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.878+0000 7f52dfe00640 1 -- 172.21.15.114:0/557291107 >> v1:172.21.15.177:6789/0 conn(0x7f52e015fff0 legacy=0x7f52e0160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.878+0000 7f52dfe00640 1 -- 172.21.15.114:0/557291107 >> v1:172.21.15.114:6789/0 conn(0x7f52e00b3f70 legacy=0x7f52e00ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.878+0000 7f52df400640 1 -- 172.21.15.114:0/557291107 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932127389 0 0) 0x7f52e00b46c0 con 0x7f52e00b3f70 2024-03-06T07:28:28.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.878+0000 7f52dfe00640 1 -- 172.21.15.114:0/557291107 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e009dd80 con 0x7f52e015fc10 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.879+0000 7f52dfe00640 1 --2- 172.21.15.114:0/557291107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e0070ff0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52d00097a0 tx=0x7f52d002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e646bf36c6e585 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.880+0000 7f52df400640 1 -- 172.21.15.114:0/557291107 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0034070 con 0x7f52e015fc10 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.880+0000 7f52df400640 1 -- 172.21.15.114:0/557291107 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d0030610 con 0x7f52e015fc10 2024-03-06T07:28:28.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.880+0000 7f52df400640 1 -- 172.21.15.114:0/557291107 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d00309e0 con 0x7f52e015fc10 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.880+0000 7f52e6a00640 1 -- 172.21.15.114:0/557291107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 msgr2=0x7f52e0070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.880+0000 7f52e6a00640 1 --2- 172.21.15.114:0/557291107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e0070ff0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52d00097a0 tx=0x7f52d002b960 comp rx=0 tx=0).stop 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.881+0000 7f52e6a00640 1 -- 172.21.15.114:0/557291107 shutdown_connections 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.881+0000 7f52e6a00640 1 --2- 172.21.15.114:0/557291107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e0070ff0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.881+0000 7f52e6a00640 1 -- 172.21.15.114:0/557291107 >> 172.21.15.114:0/557291107 conn(0x7f52e00b1940 msgr2=0x7f52e00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.881+0000 7f52e6a00640 1 -- 172.21.15.114:0/557291107 shutdown_connections 2024-03-06T07:28:28.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.881+0000 7f52e6a00640 1 -- 172.21.15.114:0/557291107 wait complete. 2024-03-06T07:28:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.882+0000 7f52e6a00640 1 Processor -- start 2024-03-06T07:28:28.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.883+0000 7f52e6a00640 1 -- start start 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.883+0000 7f52e6a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52e00b3f70 0x7f52e009cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.883+0000 7f52e6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e009ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.883+0000 7f52e6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52e015fff0 0x7f52e01571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.883+0000 7f52e6a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015c460 con 0x7f52e015fc10 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.883+0000 7f52e6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015c2e0 con 0x7f52e015fff0 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.883+0000 7f52e6a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00cebb0 con 0x7f52e00b3f70 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e009ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e009ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55094/0 (socket says 172.21.15.114:55094) 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 -- 172.21.15.114:0/4187488088 learned_addr learned my addr 172.21.15.114:0/4187488088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52dfe00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52e00b3f70 0x7f52e009cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52e4a00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52e015fff0 0x7f52e01571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 -- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52e00b3f70 msgr2=0x7f52e009cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52e00b3f70 0x7f52e009cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 -- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52e015fff0 msgr2=0x7f52e01571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52e015fff0 0x7f52e01571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.884+0000 7f52d7400640 1 -- 172.21.15.114:0/4187488088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e009bf40 con 0x7f52e015fc10 2024-03-06T07:28:28.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.885+0000 7f52d7400640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e009ba00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c430 tx=0x7f52c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.885+0000 7f52dcc00640 1 -- 172.21.15.114:0/4187488088 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8017070 con 0x7f52e015fc10 2024-03-06T07:28:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.885+0000 7f52dcc00640 1 -- 172.21.15.114:0/4187488088 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c80052d0 con 0x7f52e015fc10 2024-03-06T07:28:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.885+0000 7f52dcc00640 1 -- 172.21.15.114:0/4187488088 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8003830 con 0x7f52e015fc10 2024-03-06T07:28:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.885+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e009ac40 con 0x7f52e015fc10 2024-03-06T07:28:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.885+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e009b200 con 0x7f52e015fc10 2024-03-06T07:28:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.886+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e004f330 con 0x7f52e015fc10 2024-03-06T07:28:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.887+0000 7f52dcc00640 1 -- 172.21.15.114:0/4187488088 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52c801a070 con 0x7f52e015fc10 2024-03-06T07:28:28.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.888+0000 7f52dcc00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52bc06b4d0 0x7f52bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.888+0000 7f52dcc00640 1 -- 172.21.15.114:0/4187488088 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52c8089620 con 0x7f52e015fc10 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.888+0000 7f52dfe00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52bc06b4d0 0x7f52bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.889+0000 7f52dfe00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52bc06b4d0 0x7f52bc06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52d0036000 tx=0x7f52d0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:28.892+0000 7f52dcc00640 1 -- 172.21.15.114:0/4187488088 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52c8058400 con 0x7f52e015fc10 2024-03-06T07:28:29.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.088+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f52e006ef30 con 0x7f52e015fc10 2024-03-06T07:28:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.088+0000 7f52dcc00640 1 -- 172.21.15.114:0/4187488088 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f52c8057d00 con 0x7f52e015fc10 2024-03-06T07:28:29.090 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.090+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52bc06b4d0 msgr2=0x7f52bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.090+0000 7f52e6a00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52bc06b4d0 0x7f52bc06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f52d0036000 tx=0x7f52d0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 msgr2=0x7f52e009ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e009ba00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c430 tx=0x7f52c800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 shutdown_connections 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52e015fff0 0x7f52e01571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52bc06b4d0 0x7f52bc06d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e015fc10 0x7f52e009ba00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 --2- 172.21.15.114:0/4187488088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52e00b3f70 0x7f52e009cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 >> 172.21.15.114:0/4187488088 conn(0x7f52e00b1940 msgr2=0x7f52e0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 shutdown_connections 2024-03-06T07:28:29.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.091+0000 7f52e6a00640 1 -- 172.21.15.114:0/4187488088 wait complete. 2024-03-06T07:28:29.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:29.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.314+0000 7f7082400640 1 Processor -- start 2024-03-06T07:28:29.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.314+0000 7f7082400640 1 -- start start 2024-03-06T07:28:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.315+0000 7f7082400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f707c06da40 0x7f707c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.315+0000 7f7082400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f707c0aef20 0x7f707c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.315+0000 7f7082400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707c15c1f0 con 0x7f707c15be10 2024-03-06T07:28:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.315+0000 7f7082400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707c0cd200 con 0x7f707c06da40 2024-03-06T07:28:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.315+0000 7f7082400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707c06cac0 con 0x7f707c0aef20 2024-03-06T07:28:29.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f707be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f707c06da40 0x7f707c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f707be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f707c06da40 0x7f707c0ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55108/0 (socket says 172.21.15.114:55108) 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f707be00640 1 -- 172.21.15.114:0/3192639544 learned_addr learned my addr 172.21.15.114:0/3192639544 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f7081a00640 1 --2- 172.21.15.114:0/3192639544 >> v2:172.21.15.177:3300/0 conn(0x7f707c0aef20 0x7f707c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f707be00640 1 -- 172.21.15.114:0/3192639544 >> v2:172.21.15.177:3300/0 conn(0x7f707c0aef20 msgr2=0x7f707c0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f707be00640 1 --2- 172.21.15.114:0/3192639544 >> v2:172.21.15.177:3300/0 conn(0x7f707c0aef20 0x7f707c0ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f707be00640 1 -- 172.21.15.114:0/3192639544 >> v1:172.21.15.177:6789/0 conn(0x7f707c15be10 legacy=0x7f707c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.316+0000 7f707be00640 1 -- 172.21.15.114:0/3192639544 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c06cda0 con 0x7f707c06da40 2024-03-06T07:28:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.318+0000 7f707be00640 1 --2- 172.21.15.114:0/3192639544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c06da40 0x7f707c0ccc20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f707c0af490 tx=0x7f706402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bd067c84f7e723 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.318+0000 7f707b400640 1 -- 172.21.15.114:0/3192639544 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706402e070 con 0x7f707c06da40 2024-03-06T07:28:29.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.318+0000 7f707b400640 1 -- 172.21.15.114:0/3192639544 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7064037070 con 0x7f707c06da40 2024-03-06T07:28:29.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.318+0000 7f707b400640 1 -- 172.21.15.114:0/3192639544 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064031a00 con 0x7f707c06da40 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 -- 172.21.15.114:0/3192639544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c06da40 msgr2=0x7f707c0ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 --2- 172.21.15.114:0/3192639544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c06da40 0x7f707c0ccc20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f707c0af490 tx=0x7f706402b880 comp rx=0 tx=0).stop 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 -- 172.21.15.114:0/3192639544 shutdown_connections 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 --2- 172.21.15.114:0/3192639544 >> v2:172.21.15.177:3300/0 conn(0x7f707c0aef20 0x7f707c0ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 --2- 172.21.15.114:0/3192639544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c06da40 0x7f707c0ccc20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 -- 172.21.15.114:0/3192639544 >> 172.21.15.114:0/3192639544 conn(0x7f707c06afc0 msgr2=0x7f707c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 -- 172.21.15.114:0/3192639544 shutdown_connections 2024-03-06T07:28:29.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.319+0000 7f7082400640 1 -- 172.21.15.114:0/3192639544 wait complete. 2024-03-06T07:28:29.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.321+0000 7f7082400640 1 Processor -- start 2024-03-06T07:28:29.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.321+0000 7f7082400640 1 -- start start 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7082400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c06da40 0x7f707c15f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7082400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c0aef20 0x7f707c14a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7082400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f707c15be10 0x7f707c15cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7082400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c0cda90 con 0x7f707c0aef20 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7082400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c06d080 con 0x7f707c06da40 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7082400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707c071ab0 con 0x7f707c15be10 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f707be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c0aef20 0x7f707c14a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f707be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c0aef20 0x7f707c14a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55114/0 (socket says 172.21.15.114:55114) 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f707be00640 1 -- 172.21.15.114:0/2688394807 learned_addr learned my addr 172.21.15.114:0/2688394807 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7081000640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c06da40 0x7f707c15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.322+0000 7f7081a00640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f707c15be10 0x7f707c15cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f707be00640 1 -- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f707c15be10 msgr2=0x7f707c15cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f707be00640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f707c15be10 0x7f707c15cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f707be00640 1 -- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c06da40 msgr2=0x7f707c15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f707be00640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c06da40 0x7f707c15f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f707be00640 1 -- 172.21.15.114:0/2688394807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c14aa50 con 0x7f707c0aef20 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f707be00640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c0aef20 0x7f707c14a510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7064000c00 tx=0x7f70640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f7078c00640 1 -- 172.21.15.114:0/2688394807 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064032670 con 0x7f707c0aef20 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f7078c00640 1 -- 172.21.15.114:0/2688394807 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7064032830 con 0x7f707c0aef20 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f7078c00640 1 -- 172.21.15.114:0/2688394807 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064032c20 con 0x7f707c0aef20 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c15cff0 con 0x7f707c0aef20 2024-03-06T07:28:29.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.323+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c159a70 con 0x7f707c0aef20 2024-03-06T07:28:29.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.324+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707c04f330 con 0x7f707c0aef20 2024-03-06T07:28:29.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.325+0000 7f7078c00640 1 -- 172.21.15.114:0/2688394807 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70640453f0 con 0x7f707c0aef20 2024-03-06T07:28:29.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.327+0000 7f7078c00640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f705806b4d0 0x7f705806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.327+0000 7f7078c00640 1 -- 172.21.15.114:0/2688394807 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70640abff0 con 0x7f707c0aef20 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.327+0000 7f7081000640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f705806b4d0 0x7f705806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.328+0000 7f7081000640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f705806b4d0 0x7f705806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f706c009f20 tx=0x7f706c0094d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.331+0000 7f7078c00640 1 -- 172.21.15.114:0/2688394807 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f707c15cff0 con 0x7f707c0aef20 2024-03-06T07:28:29.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.525+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f707c0cc1e0 con 0x7f707c0aef20 2024-03-06T07:28:29.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.526+0000 7f7078c00640 1 -- 172.21.15.114:0/2688394807 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7064040070 con 0x7f707c0aef20 2024-03-06T07:28:29.528 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f705806b4d0 msgr2=0x7f705806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f705806b4d0 0x7f705806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f706c009f20 tx=0x7f706c0094d0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c0aef20 msgr2=0x7f707c14a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c0aef20 0x7f707c14a510 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7064000c00 tx=0x7f70640046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 shutdown_connections 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f707c15be10 0x7f707c15cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f705806b4d0 0x7f705806d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f707c0aef20 0x7f707c14a510 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 --2- 172.21.15.114:0/2688394807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c06da40 0x7f707c15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 >> 172.21.15.114:0/2688394807 conn(0x7f707c06afc0 msgr2=0x7f707c0ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 shutdown_connections 2024-03-06T07:28:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.530+0000 7f7082400640 1 -- 172.21.15.114:0/2688394807 wait complete. 2024-03-06T07:28:29.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:29.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.746+0000 7f7cd9400640 1 Processor -- start 2024-03-06T07:28:29.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.746+0000 7f7cd9400640 1 -- start start 2024-03-06T07:28:29.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.748+0000 7f7cd9400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7cd40b4680 0x7f7cd406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.748+0000 7f7cd9400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd415b090 con 0x7f7cd4072340 2024-03-06T07:28:29.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.748+0000 7f7cd9400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd40b2a30 con 0x7f7cd415acb0 2024-03-06T07:28:29.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.748+0000 7f7cd9400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd40b4360 con 0x7f7cd40b4680 2024-03-06T07:28:29.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.748+0000 7f7cd3200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7cd4072340 0x7f7cd40ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49150/0 (socket says 172.21.15.114:49150) 2024-03-06T07:28:29.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.748+0000 7f7cd3200640 1 -- 172.21.15.114:0/4284633169 learned_addr learned my addr 172.21.15.114:0/4284633169 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:29.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279691305 0 0) 0x7f7cd415b090 con 0x7f7cd4072340 2024-03-06T07:28:29.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb8005730 con 0x7f7cd4072340 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cc9e00640 1 --2- 172.21.15.114:0/4284633169 >> v2:172.21.15.177:3300/0 conn(0x7f7cd40b4680 0x7f7cd406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1055324177 0 0) 0x7f7cb8005730 con 0x7f7cd4072340 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 >> v2:172.21.15.177:3300/0 conn(0x7f7cd40b4680 msgr2=0x7f7cd406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 --2- 172.21.15.114:0/4284633169 >> v2:172.21.15.177:3300/0 conn(0x7f7cd40b4680 0x7f7cd406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 >> v1:172.21.15.177:6789/0 conn(0x7f7cd415acb0 legacy=0x7f7cd40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd40b2cf0 con 0x7f7cd4072340 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7ccc001670 con 0x7f7cd4072340 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 >> v1:172.21.15.114:6789/0 conn(0x7f7cd4072340 legacy=0x7f7cd40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cb8005bc0 0x7f7cb8005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cb8006a10 0x7f7cb8006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cb800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40b4360 con 0x7f7cb80073d0 2024-03-06T07:28:29.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb800e320 con 0x7f7cb8005bc0 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.749+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb800e520 con 0x7f7cb8006a10 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd3200640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cb800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd2800640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cb8005bc0 0x7f7cb8005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cc9e00640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cb8006a10 0x7f7cb8006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd3200640 1 -- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cb8006a10 msgr2=0x7f7cb8006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd3200640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cb8006a10 0x7f7cb8006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd3200640 1 -- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cb8005bc0 msgr2=0x7f7cb8005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd3200640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cb8005bc0 0x7f7cb8005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd3200640 1 -- 172.21.15.114:0/4284633169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8005730 con 0x7f7cb80073d0 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.750+0000 7f7cd2800640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cb8005bc0 0x7f7cb8005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.751+0000 7f7cd3200640 1 -- 172.21.15.114:0/4284633169 reap_dead start 2024-03-06T07:28:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.751+0000 7f7cd3200640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cb800dbd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c240 tx=0x7f7cb8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58040c2403634f49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.751+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc007d10 con 0x7f7cb80073d0 2024-03-06T07:28:29.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.751+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc030070 con 0x7f7cb80073d0 2024-03-06T07:28:29.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.751+0000 7f7cd1e00640 1 -- 172.21.15.114:0/4284633169 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc02cd20 con 0x7f7cb80073d0 2024-03-06T07:28:29.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.752+0000 7f7cd9400640 1 -- 172.21.15.114:0/4284633169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 msgr2=0x7f7cb800dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.752+0000 7f7cd9400640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cb800dbd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c240 tx=0x7f7cb8007a40 comp rx=0 tx=0).stop 2024-03-06T07:28:29.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.752+0000 7f7cd9400640 1 -- 172.21.15.114:0/4284633169 shutdown_connections 2024-03-06T07:28:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.752+0000 7f7cd9400640 1 --2- 172.21.15.114:0/4284633169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cb800dbd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.752+0000 7f7cd9400640 1 -- 172.21.15.114:0/4284633169 >> 172.21.15.114:0/4284633169 conn(0x7f7cd40b1730 msgr2=0x7f7cd40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.752+0000 7f7cd9400640 1 -- 172.21.15.114:0/4284633169 shutdown_connections 2024-03-06T07:28:29.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.752+0000 7f7cd9400640 1 -- 172.21.15.114:0/4284633169 wait complete. 2024-03-06T07:28:29.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.754+0000 7f7cd9400640 1 Processor -- start 2024-03-06T07:28:29.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.754+0000 7f7cd9400640 1 -- start start 2024-03-06T07:28:29.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cd414a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cd4158260 0x7f7cd41586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd9400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cd406dbd0 0x7f7cd406e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40b4010 con 0x7f7cb80073d0 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd9400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40b32b0 con 0x7f7cd4158260 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd9400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd40b37d0 con 0x7f7cd406dbd0 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cd414a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cd414a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55142/0 (socket says 172.21.15.114:55142) 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.755+0000 7f7cd2800640 1 -- 172.21.15.114:0/1751683986 learned_addr learned my addr 172.21.15.114:0/1751683986 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cc9e00640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cd4158260 0x7f7cd41586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cd3200640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cd406dbd0 0x7f7cd406e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cd2800640 1 -- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cd406dbd0 msgr2=0x7f7cd406e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cd2800640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cd406dbd0 0x7f7cd406e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cd2800640 1 -- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cd4158260 msgr2=0x7f7cd41586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cd2800640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cd4158260 0x7f7cd41586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cd2800640 1 -- 172.21.15.114:0/1751683986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4158e00 con 0x7f7cb80073d0 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7cd2800640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cd414a480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400d910 tx=0x7f7cc400ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.756+0000 7f7ccb400640 1 -- 172.21.15.114:0/1751683986 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4003da0 con 0x7f7cb80073d0 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.757+0000 7f7ccb400640 1 -- 172.21.15.114:0/1751683986 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4017070 con 0x7f7cb80073d0 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.757+0000 7f7ccb400640 1 -- 172.21.15.114:0/1751683986 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400b990 con 0x7f7cb80073d0 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.757+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd41590e0 con 0x7f7cb80073d0 2024-03-06T07:28:29.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.757+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd41596a0 con 0x7f7cb80073d0 2024-03-06T07:28:29.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.758+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd404f330 con 0x7f7cb80073d0 2024-03-06T07:28:29.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.759+0000 7f7ccb400640 1 -- 172.21.15.114:0/1751683986 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cc400bb30 con 0x7f7cb80073d0 2024-03-06T07:28:29.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.760+0000 7f7ccb400640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7cb006b4d0 0x7f7cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.760+0000 7f7ccb400640 1 -- 172.21.15.114:0/1751683986 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cc408a3a0 con 0x7f7cb80073d0 2024-03-06T07:28:29.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.760+0000 7f7cc9e00640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7cb006b4d0 0x7f7cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.761+0000 7f7cc9e00640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7cb006b4d0 0x7f7cb006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40b1700 tx=0x7f7cbc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.764+0000 7f7ccb400640 1 -- 172.21.15.114:0/1751683986 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7cc400be20 con 0x7f7cb80073d0 2024-03-06T07:28:29.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.960+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd406ce60 con 0x7f7cb80073d0 2024-03-06T07:28:29.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.961+0000 7f7ccb400640 1 -- 172.21.15.114:0/1751683986 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f7cc4059220 con 0x7f7cb80073d0 2024-03-06T07:28:29.964 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:29.964 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:29.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7cb006b4d0 msgr2=0x7f7cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7cb006b4d0 0x7f7cb006d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40b1700 tx=0x7f7cbc008040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 msgr2=0x7f7cd414a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cd414a480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400d910 tx=0x7f7cc400ddd0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 shutdown_connections 2024-03-06T07:28:29.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7cd406dbd0 0x7f7cd406e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cd4158260 0x7f7cd41586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7cb006b4d0 0x7f7cb006d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 --2- 172.21.15.114:0/1751683986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cb80073d0 0x7f7cd414a480 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 >> 172.21.15.114:0/1751683986 conn(0x7f7cd40b1730 msgr2=0x7f7cd4155520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 shutdown_connections 2024-03-06T07:28:29.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:29.964+0000 7f7cd9400640 1 -- 172.21.15.114:0/1751683986 wait complete. 2024-03-06T07:28:29.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:29.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.187+0000 7f9746200640 1 Processor -- start 2024-03-06T07:28:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.187+0000 7f9746200640 1 -- start start 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.188+0000 7f9746200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f974015e360 0x7f97400cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.188+0000 7f9746200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97400d00d0 0x7f974006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.188+0000 7f9746200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97400d1fb0 con 0x7f974015df80 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.188+0000 7f9746200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97400d2a20 con 0x7f97400d00d0 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.188+0000 7f9746200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974009b100 con 0x7f974015e360 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97400d00d0 0x7f974006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97400d00d0 0x7f974006ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55144/0 (socket says 172.21.15.114:55144) 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973fe00640 1 -- 172.21.15.114:0/3106122145 learned_addr learned my addr 172.21.15.114:0/3106122145 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973ea00640 1 --2- 172.21.15.114:0/3106122145 >> v2:172.21.15.177:3300/0 conn(0x7f974015e360 0x7f97400cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973fe00640 1 -- 172.21.15.114:0/3106122145 >> v2:172.21.15.177:3300/0 conn(0x7f974015e360 msgr2=0x7f97400cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973fe00640 1 --2- 172.21.15.114:0/3106122145 >> v2:172.21.15.177:3300/0 conn(0x7f974015e360 0x7f97400cfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973fe00640 1 -- 172.21.15.114:0/3106122145 >> v1:172.21.15.177:6789/0 conn(0x7f974015df80 legacy=0x7f974006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973fe00640 1 -- 172.21.15.114:0/3106122145 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974009b3a0 con 0x7f97400d00d0 2024-03-06T07:28:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.189+0000 7f973ea00640 1 --2- 172.21.15.114:0/3106122145 >> v2:172.21.15.177:3300/0 conn(0x7f974015e360 0x7f97400cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.190+0000 7f973fe00640 1 --2- 172.21.15.114:0/3106122145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d00d0 0x7f974006ee40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f97400b77f0 tx=0x7f973802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7554fa4e6888d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.190+0000 7f973e000640 1 -- 172.21.15.114:0/3106122145 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973802ca80 con 0x7f97400d00d0 2024-03-06T07:28:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f973e000640 1 -- 172.21.15.114:0/3106122145 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973802cc20 con 0x7f97400d00d0 2024-03-06T07:28:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f973e000640 1 -- 172.21.15.114:0/3106122145 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380343f0 con 0x7f97400d00d0 2024-03-06T07:28:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 -- 172.21.15.114:0/3106122145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d00d0 msgr2=0x7f974006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 --2- 172.21.15.114:0/3106122145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d00d0 0x7f974006ee40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f97400b77f0 tx=0x7f973802bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 -- 172.21.15.114:0/3106122145 shutdown_connections 2024-03-06T07:28:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 --2- 172.21.15.114:0/3106122145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97400d00d0 0x7f974006ee40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 --2- 172.21.15.114:0/3106122145 >> v2:172.21.15.177:3300/0 conn(0x7f974015e360 0x7f97400cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 -- 172.21.15.114:0/3106122145 >> 172.21.15.114:0/3106122145 conn(0x7f97400b7320 msgr2=0x7f974009a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 -- 172.21.15.114:0/3106122145 shutdown_connections 2024-03-06T07:28:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.191+0000 7f9746200640 1 -- 172.21.15.114:0/3106122145 wait complete. 2024-03-06T07:28:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.193+0000 7f9746200640 1 Processor -- start 2024-03-06T07:28:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.193+0000 7f9746200640 1 -- start start 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f9746200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97400d00d0 0x7f974014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f9746200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f974015df80 0x7f97401492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f9746200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f974015e360 0x7f974006f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f9746200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974009b680 con 0x7f974015df80 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f9746200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97400d2500 con 0x7f97400d00d0 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f9746200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974006de40 con 0x7f974015e360 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f973ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f974015df80 0x7f97401492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f973ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f974015df80 0x7f97401492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55160/0 (socket says 172.21.15.114:55160) 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.194+0000 7f973ea00640 1 -- 172.21.15.114:0/3518957055 learned_addr learned my addr 172.21.15.114:0/3518957055 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973f400640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97400d00d0 0x7f974014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973fe00640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f974015e360 0x7f974006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973ea00640 1 -- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f974015e360 msgr2=0x7f974006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973ea00640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f974015e360 0x7f974006f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973ea00640 1 -- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97400d00d0 msgr2=0x7f974014c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973ea00640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97400d00d0 0x7f974014c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973ea00640 1 -- 172.21.15.114:0/3518957055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974015ea10 con 0x7f974015df80 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f973ea00640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f974015df80 0x7f97401492d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f972800c820 tx=0x7f972800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.195+0000 7f9737400640 1 -- 172.21.15.114:0/3518957055 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9728015070 con 0x7f974015df80 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.196+0000 7f9737400640 1 -- 172.21.15.114:0/3518957055 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9728007a30 con 0x7f974015df80 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.196+0000 7f9737400640 1 -- 172.21.15.114:0/3518957055 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9728007e00 con 0x7f974015df80 2024-03-06T07:28:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.196+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974014cb60 con 0x7f974015df80 2024-03-06T07:28:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.196+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9740149e20 con 0x7f974015df80 2024-03-06T07:28:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.197+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97400d2bb0 con 0x7f974015df80 2024-03-06T07:28:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.198+0000 7f9737400640 1 -- 172.21.15.114:0/3518957055 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9728018070 con 0x7f974015df80 2024-03-06T07:28:30.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.199+0000 7f9737400640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f971c06b500 0x7f971c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.199+0000 7f9737400640 1 -- 172.21.15.114:0/3518957055 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f972808a150 con 0x7f974015df80 2024-03-06T07:28:30.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.199+0000 7f973f400640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f971c06b500 0x7f971c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.200+0000 7f973f400640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f971c06b500 0x7f971c06d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9730004ea0 tx=0x7f9730009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.204+0000 7f9737400640 1 -- 172.21.15.114:0/3518957055 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97400d2bb0 con 0x7f974015df80 2024-03-06T07:28:30.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.412+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f974006db40 con 0x7f974015df80 2024-03-06T07:28:30.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.412+0000 7f9737400640 1 -- 172.21.15.114:0/3518957055 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f974006db40 con 0x7f974015df80 2024-03-06T07:28:30.414 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f971c06b500 msgr2=0x7f971c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f971c06b500 0x7f971c06d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9730004ea0 tx=0x7f9730009040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f974015df80 msgr2=0x7f97401492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f974015df80 0x7f97401492d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f972800c820 tx=0x7f972800cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 shutdown_connections 2024-03-06T07:28:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f974015e360 0x7f974006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f971c06b500 0x7f971c06d9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f974015df80 0x7f97401492d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 --2- 172.21.15.114:0/3518957055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97400d00d0 0x7f974014c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 >> 172.21.15.114:0/3518957055 conn(0x7f97400b7320 msgr2=0x7f97400b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 shutdown_connections 2024-03-06T07:28:30.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.415+0000 7f9746200640 1 -- 172.21.15.114:0/3518957055 wait complete. 2024-03-06T07:28:30.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:30.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.633+0000 7f611a000640 1 Processor -- start 2024-03-06T07:28:30.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.633+0000 7f611a000640 1 -- start start 2024-03-06T07:28:30.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f611a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f611406dcf0 0x7f611415efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f611a000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6114071ea0 con 0x7f611406afc0 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f611a000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611406e380 con 0x7f611415bb90 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f611a000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f611415bf70 con 0x7f611406dcf0 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f6112a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f611406dcf0 0x7f611415efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f6112a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f611406dcf0 0x7f611415efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55166/0 (socket says 172.21.15.114:55166) 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f6112a00640 1 -- 172.21.15.114:0/2708028517 learned_addr learned my addr 172.21.15.114:0/2708028517 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f6113e00640 1 --1- 172.21.15.114:0/2708028517 >> v1:172.21.15.114:6789/0 conn(0x7f611406afc0 0x7f61140723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49164/0 (socket says 172.21.15.114:49164) 2024-03-06T07:28:30.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f6112a00640 1 -- 172.21.15.114:0/2708028517 >> v1:172.21.15.177:6789/0 conn(0x7f611415bb90 legacy=0x7f61140cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f6112a00640 1 -- 172.21.15.114:0/2708028517 >> v1:172.21.15.114:6789/0 conn(0x7f611406afc0 legacy=0x7f61140723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.635+0000 7f6112a00640 1 -- 172.21.15.114:0/2708028517 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611406e660 con 0x7f611406dcf0 2024-03-06T07:28:30.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f6112a00640 1 --2- 172.21.15.114:0/2708028517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406dcf0 0x7f611415efa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61000098a0 tx=0x7f610002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7f34c48ac732c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f6112000640 1 -- 172.21.15.114:0/2708028517 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100034070 con 0x7f611406dcf0 2024-03-06T07:28:30.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f6112000640 1 -- 172.21.15.114:0/2708028517 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6100030840 con 0x7f611406dcf0 2024-03-06T07:28:30.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f6112000640 1 -- 172.21.15.114:0/2708028517 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100030c10 con 0x7f611406dcf0 2024-03-06T07:28:30.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f611a000640 1 -- 172.21.15.114:0/2708028517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406dcf0 msgr2=0x7f611415efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f611a000640 1 --2- 172.21.15.114:0/2708028517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406dcf0 0x7f611415efa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61000098a0 tx=0x7f610002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:30.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f611a000640 1 -- 172.21.15.114:0/2708028517 shutdown_connections 2024-03-06T07:28:30.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f611a000640 1 --2- 172.21.15.114:0/2708028517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406dcf0 0x7f611415efa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.637+0000 7f611a000640 1 -- 172.21.15.114:0/2708028517 >> 172.21.15.114:0/2708028517 conn(0x7f61140b27d0 msgr2=0x7f61140b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.638+0000 7f611a000640 1 -- 172.21.15.114:0/2708028517 shutdown_connections 2024-03-06T07:28:30.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.638+0000 7f611a000640 1 -- 172.21.15.114:0/2708028517 wait complete. 2024-03-06T07:28:30.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.639+0000 7f611a000640 1 Processor -- start 2024-03-06T07:28:30.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.639+0000 7f611a000640 1 -- start start 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.640+0000 7f611a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406afc0 0x7f6114145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.640+0000 7f611a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f611406dcf0 0x7f611415d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.640+0000 7f611a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f611415bb90 0x7f61141427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.640+0000 7f611a000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611406ec20 con 0x7f611406afc0 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.640+0000 7f611a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611415f4e0 con 0x7f611406dcf0 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.640+0000 7f611a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6114072ab0 con 0x7f611415bb90 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.640+0000 7f6113400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406afc0 0x7f6114145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406afc0 0x7f6114145c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55176/0 (socket says 172.21.15.114:55176) 2024-03-06T07:28:30.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 -- 172.21.15.114:0/227832718 learned_addr learned my addr 172.21.15.114:0/227832718 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:30.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6112a00640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f611406dcf0 0x7f611415d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113e00640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f611415bb90 0x7f61141427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 -- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f611415bb90 msgr2=0x7f61141427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f611415bb90 0x7f61141427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 -- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f611406dcf0 msgr2=0x7f611415d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f611406dcf0 0x7f611415d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 -- 172.21.15.114:0/227832718 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611415e010 con 0x7f611406afc0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6112a00640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f611406dcf0 0x7f611415d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.641+0000 7f6113400640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406afc0 0x7f6114145c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f610400b420 tx=0x7f610400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.642+0000 7f610b400640 1 -- 172.21.15.114:0/227832718 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104011830 con 0x7f611406afc0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.642+0000 7f610b400640 1 -- 172.21.15.114:0/227832718 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6104016070 con 0x7f611406afc0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.642+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611415dd30 con 0x7f611406afc0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.642+0000 7f610b400640 1 -- 172.21.15.114:0/227832718 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104011c10 con 0x7f611406afc0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.642+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61140b4c10 con 0x7f611406afc0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.643+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611404f330 con 0x7f611406afc0 2024-03-06T07:28:30.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.644+0000 7f610b400640 1 -- 172.21.15.114:0/227832718 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f610401b460 con 0x7f611406afc0 2024-03-06T07:28:30.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.645+0000 7f610b400640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f60f406b4d0 0x7f60f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.645+0000 7f610b400640 1 -- 172.21.15.114:0/227832718 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6104089d40 con 0x7f611406afc0 2024-03-06T07:28:30.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.645+0000 7f6112a00640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f60f406b4d0 0x7f60f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.646+0000 7f6112a00640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f60f406b4d0 0x7f60f406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6100036000 tx=0x7f610003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.650+0000 7f610b400640 1 -- 172.21.15.114:0/227832718 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6104020020 con 0x7f611406afc0 2024-03-06T07:28:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.843+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f611415f740 con 0x7f611406afc0 2024-03-06T07:28:30.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.844+0000 7f610b400640 1 -- 172.21.15.114:0/227832718 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6104058b20 con 0x7f611406afc0 2024-03-06T07:28:30.845 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:30.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.846+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f60f406b4d0 msgr2=0x7f60f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.846+0000 7f611a000640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f60f406b4d0 0x7f60f406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6100036000 tx=0x7f610003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.846+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406afc0 msgr2=0x7f6114145c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.846+0000 7f611a000640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406afc0 0x7f6114145c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f610400b420 tx=0x7f610400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 shutdown_connections 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f611415bb90 0x7f61141427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f611406dcf0 0x7f611415d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f60f406b4d0 0x7f60f406d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 --2- 172.21.15.114:0/227832718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f611406afc0 0x7f6114145c90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 >> 172.21.15.114:0/227832718 conn(0x7f61140b27d0 msgr2=0x7f61140d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 shutdown_connections 2024-03-06T07:28:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:30.847+0000 7f611a000640 1 -- 172.21.15.114:0/227832718 wait complete. 2024-03-06T07:28:30.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:31.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.137+0000 7fd6d3200640 1 Processor -- start 2024-03-06T07:28:31.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.138+0000 7fd6d3200640 1 -- start start 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.140+0000 7fd6d3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 0x7fd6d40b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.140+0000 7fd6d3200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd6d40b4d70 0x7fd6d406e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.140+0000 7fd6d3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d415e790 con 0x7fd6d4151bf0 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.140+0000 7fd6d3200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d415ea70 con 0x7fd6d40b4d70 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.140+0000 7fd6d3200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d415ed50 con 0x7fd6d406c7e0 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.140+0000 7fd6d1e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd6d4151bf0 0x7fd6d406c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49180/0 (socket says 172.21.15.114:49180) 2024-03-06T07:28:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.140+0000 7fd6d1e00640 1 -- 172.21.15.114:0/2649529915 learned_addr learned my addr 172.21.15.114:0/2649529915 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016567669 0 0) 0x7fd6d415e790 con 0x7fd6d4151bf0 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d2800640 1 --2- 172.21.15.114:0/2649529915 >> v2:172.21.15.177:3300/0 conn(0x7fd6d40b4d70 0x7fd6d406e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b8005730 con 0x7fd6d4151bf0 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6c9400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 0x7fd6d40b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332206901 0 0) 0x7fd6b8005730 con 0x7fd6d4151bf0 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 msgr2=0x7fd6d40b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 0x7fd6d40b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 >> v2:172.21.15.177:3300/0 conn(0x7fd6d40b4d70 msgr2=0x7fd6d406e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 --2- 172.21.15.114:0/2649529915 >> v2:172.21.15.177:3300/0 conn(0x7fd6d40b4d70 0x7fd6d406e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d415f030 con 0x7fd6d4151bf0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fd6c4001670 con 0x7fd6d4151bf0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 >> v1:172.21.15.114:6789/0 conn(0x7fd6d4151bf0 legacy=0x7fd6d406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8005bc0 0x7fd6b8005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8006a10 0x7fd6b8006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 0x7fd6b80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d415ea70 con 0x7fd6b8006a10 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d415ed50 con 0x7fd6b8005bc0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.141+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b800dc20 con 0x7fd6d406c7e0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6c9400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8006a10 0x7fd6b8006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6d1e00640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8005bc0 0x7fd6b8005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6d2800640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 0x7fd6b80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6c9400640 1 -- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 msgr2=0x7fd6b80073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6c9400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d406c7e0 0x7fd6b80073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6c9400640 1 -- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8005bc0 msgr2=0x7fd6b8005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6c9400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8005bc0 0x7fd6b8005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.142+0000 7fd6c9400640 1 -- 172.21.15.114:0/2649529915 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b8005730 con 0x7fd6b8006a10 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.143+0000 7fd6c9400640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8006a10 0x7fd6b8006e90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00ba30 tx=0x7fd6b8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92328c3c20619ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.143+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc036020 con 0x7fd6b8006a10 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.143+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc032640 con 0x7fd6b8006a10 2024-03-06T07:28:31.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.143+0000 7fd6d1400640 1 -- 172.21.15.114:0/2649529915 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00b600 con 0x7fd6b8006a10 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.144+0000 7fd6d3200640 1 -- 172.21.15.114:0/2649529915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8006a10 msgr2=0x7fd6b8006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.144+0000 7fd6d3200640 1 --2- 172.21.15.114:0/2649529915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8006a10 0x7fd6b8006e90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00ba30 tx=0x7fd6b8007a40 comp rx=0 tx=0).stop 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.144+0000 7fd6d2800640 1 -- 172.21.15.114:0/2649529915 reap_dead start 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.144+0000 7fd6d3200640 1 -- 172.21.15.114:0/2649529915 shutdown_connections 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.144+0000 7fd6d3200640 1 -- 172.21.15.114:0/2649529915 >> 172.21.15.114:0/2649529915 conn(0x7fd6d40b25f0 msgr2=0x7fd6d40b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.144+0000 7fd6d3200640 1 -- 172.21.15.114:0/2649529915 shutdown_connections 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.144+0000 7fd6d3200640 1 -- 172.21.15.114:0/2649529915 wait complete. 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.146+0000 7fd6d3200640 1 Processor -- start 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.146+0000 7fd6d3200640 1 -- start start 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6d3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8006a10 0x7fd6d413d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6d3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d40cf9c0 0x7fd6d40cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6d3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d413b530 0x7fd6d413b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6d3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d40b51d0 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6d3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d415c2c0 con 0x7fd6b8006a10 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6d3200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d40b5350 con 0x7fd6d413b530 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6c9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d40cf9c0 0x7fd6d40cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6c9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d40cf9c0 0x7fd6d40cfe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55204/0 (socket says 172.21.15.114:55204) 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.147+0000 7fd6c9400640 1 -- 172.21.15.114:0/3243730014 learned_addr learned my addr 172.21.15.114:0/3243730014 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6c9400640 1 -- 172.21.15.114:0/3243730014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d413b530 msgr2=0x7fd6d413b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6c9400640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d413b530 0x7fd6d413b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6c9400640 1 -- 172.21.15.114:0/3243730014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8006a10 msgr2=0x7fd6d413d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6c9400640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8006a10 0x7fd6d413d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6c9400640 1 -- 172.21.15.114:0/3243730014 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d413ad70 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6c9400640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d40cf9c0 0x7fd6d40cfe40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00bb20 tx=0x7fd6bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6caa00640 1 -- 172.21.15.114:0/3243730014 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc041070 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6caa00640 1 -- 172.21.15.114:0/3243730014 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc030a00 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6caa00640 1 -- 172.21.15.114:0/3243730014 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc036040 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d40d06b0 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.148+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d40d0c70 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.149+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d404f330 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.151+0000 7fd6caa00640 1 -- 172.21.15.114:0/3243730014 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6bc039020 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.152+0000 7fd6caa00640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd6b006b4d0 0x7fd6b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.152+0000 7fd6caa00640 1 -- 172.21.15.114:0/3243730014 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6bc0ac370 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.152+0000 7fd6d1e00640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd6b006b4d0 0x7fd6b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.153+0000 7fd6d1e00640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd6b006b4d0 0x7fd6b006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40058b0 tx=0x7fd6c4000f30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.157+0000 7fd6caa00640 1 -- 172.21.15.114:0/3243730014 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6bc07b160 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.359+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6d40b2e70 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.360+0000 7fd6caa00640 1 -- 172.21.15.114:0/3243730014 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6bc07aa60 con 0x7fd6d40cf9c0 2024-03-06T07:28:31.361 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd6b006b4d0 msgr2=0x7fd6b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd6b006b4d0 0x7fd6b006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40058b0 tx=0x7fd6c4000f30 comp rx=0 tx=0).stop 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d40cf9c0 msgr2=0x7fd6d40cfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d40cf9c0 0x7fd6d40cfe40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00bb20 tx=0x7fd6bc004730 comp rx=0 tx=0).stop 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 shutdown_connections 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6d413b530 0x7fd6d413b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd6b006b4d0 0x7fd6b006d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d40cf9c0 0x7fd6d40cfe40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 --2- 172.21.15.114:0/3243730014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6b8006a10 0x7fd6d413d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 >> 172.21.15.114:0/3243730014 conn(0x7fd6d40b25f0 msgr2=0x7fd6d415d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 shutdown_connections 2024-03-06T07:28:31.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.362+0000 7fd6d3200640 1 -- 172.21.15.114:0/3243730014 wait complete. 2024-03-06T07:28:31.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:31.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.577+0000 7fe5d7e00640 1 Processor -- start 2024-03-06T07:28:31.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.578+0000 7fe5d7e00640 1 -- start start 2024-03-06T07:28:31.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.579+0000 7fe5d7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5d80d27a0 0x7fe5d80b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.579+0000 7fe5d7e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d8161960 con 0x7fe5d80d2360 2024-03-06T07:28:31.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.579+0000 7fe5d7e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d80cf6f0 con 0x7fe5d8161580 2024-03-06T07:28:31.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.579+0000 7fe5d7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80cf9d0 con 0x7fe5d80d27a0 2024-03-06T07:28:31.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.579+0000 7fe5cfe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe5d80d2360 0x7fe5d80b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49196/0 (socket says 172.21.15.114:49196) 2024-03-06T07:28:31.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.579+0000 7fe5cfe00640 1 -- 172.21.15.114:0/4268326906 learned_addr learned my addr 172.21.15.114:0/4268326906 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d7400640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5d80d27a0 0x7fe5d80b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064470320 0 0) 0x7fe5d8161960 con 0x7fe5d80d2360 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5bc003520 con 0x7fe5d80d2360 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2957226731 0 0) 0x7fe5bc003520 con 0x7fe5d80d2360 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5d80d27a0 msgr2=0x7fe5d80b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5d80d27a0 0x7fe5d80b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 >> v1:172.21.15.177:6789/0 conn(0x7fe5d8161580 legacy=0x7fe5d80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d80cfcb0 con 0x7fe5d80d2360 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fe5c0001670 con 0x7fe5d80d2360 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 >> v1:172.21.15.114:6789/0 conn(0x7fe5d80d2360 legacy=0x7fe5d80b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc0039e0 0x7fe5bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5d80d27a0 0x7fe5bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5bc004d40 0x7fe5bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80cf9d0 con 0x7fe5bc0039e0 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc00bb50 con 0x7fe5bc004d40 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.580+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc00bd20 con 0x7fe5d80d27a0 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.581+0000 7fe5d6a00640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc0039e0 0x7fe5bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.581+0000 7fe5d6a00640 1 -- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5d80d27a0 msgr2=0x7fe5bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.581+0000 7fe5d6a00640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5d80d27a0 0x7fe5bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.581+0000 7fe5d6a00640 1 -- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5bc004d40 msgr2=0x7fe5bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.581+0000 7fe5d6a00640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5bc004d40 0x7fe5bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.581+0000 7fe5d6a00640 1 -- 172.21.15.114:0/4268326906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc003520 con 0x7fe5bc0039e0 2024-03-06T07:28:31.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.582+0000 7fe5d6a00640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc0039e0 0x7fe5bc003e00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8001780 tx=0x7fe5bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5300af9359b99d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.582+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8001370 con 0x7fe5bc0039e0 2024-03-06T07:28:31.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.582+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c80052d0 con 0x7fe5bc0039e0 2024-03-06T07:28:31.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.582+0000 7fe5d6000640 1 -- 172.21.15.114:0/4268326906 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8003830 con 0x7fe5bc0039e0 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.583+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4268326906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc0039e0 msgr2=0x7fe5bc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.583+0000 7fe5d7e00640 1 --2- 172.21.15.114:0/4268326906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc0039e0 0x7fe5bc003e00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8001780 tx=0x7fe5bc005830 comp rx=0 tx=0).stop 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.583+0000 7fe5d7400640 1 -- 172.21.15.114:0/4268326906 reap_dead start 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.583+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4268326906 shutdown_connections 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.583+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4268326906 >> 172.21.15.114:0/4268326906 conn(0x7fe5d80d3d80 msgr2=0x7fe5d80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.583+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4268326906 shutdown_connections 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.583+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4268326906 wait complete. 2024-03-06T07:28:31.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.585+0000 7fe5d7e00640 1 Processor -- start 2024-03-06T07:28:31.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.585+0000 7fe5d7e00640 1 -- start start 2024-03-06T07:28:31.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5d7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5bc0039e0 0x7fe5d80b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d814f140 0x7fe5d80b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5d7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5d80b6e20 0x7fe5d815d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5d7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80d2c30 con 0x7fe5d814f140 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5d7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80b39a0 con 0x7fe5d80b6e20 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5d7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80cf3a0 con 0x7fe5bc0039e0 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5cfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d814f140 0x7fe5d80b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5cfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d814f140 0x7fe5d80b68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55230/0 (socket says 172.21.15.114:55230) 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.586+0000 7fe5cfe00640 1 -- 172.21.15.114:0/4240411417 learned_addr learned my addr 172.21.15.114:0/4240411417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cfe00640 1 -- 172.21.15.114:0/4240411417 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5bc0039e0 msgr2=0x7fe5d80b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cfe00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5bc0039e0 0x7fe5d80b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cfe00640 1 -- 172.21.15.114:0/4240411417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5d80b6e20 msgr2=0x7fe5d815d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cfe00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5d80b6e20 0x7fe5d815d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cfe00640 1 -- 172.21.15.114:0/4240411417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d80b7380 con 0x7fe5d814f140 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cfe00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d814f140 0x7fe5d80b68e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000c7a0 tx=0x7fe5c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cf400640 1 -- 172.21.15.114:0/4240411417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0018070 con 0x7fe5d814f140 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cf400640 1 -- 172.21.15.114:0/4240411417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c00074e0 con 0x7fe5d814f140 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d80b7660 con 0x7fe5d814f140 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.587+0000 7fe5cf400640 1 -- 172.21.15.114:0/4240411417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0004a20 con 0x7fe5d814f140 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.588+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d80b7c20 con 0x7fe5d814f140 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.589+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d80ce110 con 0x7fe5d814f140 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.590+0000 7fe5cf400640 1 -- 172.21.15.114:0/4240411417 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5c001b070 con 0x7fe5d814f140 2024-03-06T07:28:31.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.591+0000 7fe5cf400640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5b406b500 0x7fe5b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.591+0000 7fe5cf400640 1 -- 172.21.15.114:0/4240411417 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5c0089a80 con 0x7fe5d814f140 2024-03-06T07:28:31.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.591+0000 7fe5d6a00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5b406b500 0x7fe5b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.592+0000 7fe5d6a00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5b406b500 0x7fe5b406d9c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8033000 tx=0x7fe5c802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.595+0000 7fe5cf400640 1 -- 172.21.15.114:0/4240411417 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5d80ce110 con 0x7fe5d814f140 2024-03-06T07:28:31.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.785+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5d804f330 con 0x7fe5d814f140 2024-03-06T07:28:31.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.787+0000 7fe5cf400640 1 -- 172.21.15.114:0/4240411417 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fe5c00589e0 con 0x7fe5d814f140 2024-03-06T07:28:31.789 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:31.789 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:31.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5b406b500 msgr2=0x7fe5b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5b406b500 0x7fe5b406d9c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8033000 tx=0x7fe5c802f000 comp rx=0 tx=0).stop 2024-03-06T07:28:31.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d814f140 msgr2=0x7fe5d80b68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d814f140 0x7fe5d80b68e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000c7a0 tx=0x7fe5c000cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:31.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 shutdown_connections 2024-03-06T07:28:31.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5d80b6e20 0x7fe5d815d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5b406b500 0x7fe5b406d9c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5d814f140 0x7fe5d80b68e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 --2- 172.21.15.114:0/4240411417 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe5bc0039e0 0x7fe5d80b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.789+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 >> 172.21.15.114:0/4240411417 conn(0x7fe5d80d3d80 msgr2=0x7fe5d80cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.790+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 shutdown_connections 2024-03-06T07:28:31.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:31.790+0000 7fe5d7e00640 1 -- 172.21.15.114:0/4240411417 wait complete. 2024-03-06T07:28:31.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:31.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:32.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.015+0000 7fb9e3400640 1 Processor -- start 2024-03-06T07:28:32.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.015+0000 7fb9e3400640 1 -- start start 2024-03-06T07:28:32.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.016+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 0x7fb9dc0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.016+0000 7fb9e3400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0b3730 con 0x7fb9dc15e450 2024-03-06T07:28:32.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.016+0000 7fb9e3400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc067b20 con 0x7fb9dc0d07f0 2024-03-06T07:28:32.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.017+0000 7fb9e3400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d4930 con 0x7fb9dc0b5490 2024-03-06T07:28:32.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.017+0000 7fb9d3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb9dc15e450 0x7fb9dc0cea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49202/0 (socket says 172.21.15.114:49202) 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.017+0000 7fb9d3e00640 1 -- 172.21.15.114:0/1972391811 learned_addr learned my addr 172.21.15.114:0/1972391811 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.017+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336058789 0 0) 0x7fb9dc0b3730 con 0x7fb9dc15e450 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.017+0000 7fb9e1400640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 0x7fb9dc0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.017+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c0003520 con 0x7fb9dc15e450 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20140281 0 0) 0x7fb9c0003520 con 0x7fb9dc15e450 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 msgr2=0x7fb9dc0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 0x7fb9dc0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 >> v1:172.21.15.177:6789/0 conn(0x7fb9dc0d07f0 legacy=0x7fb9dc0d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0d4b80 con 0x7fb9dc15e450 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb9c4001670 con 0x7fb9dc15e450 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 >> v1:172.21.15.114:6789/0 conn(0x7fb9dc15e450 legacy=0x7fb9dc0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9c00039e0 0x7fb9c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 0x7fb9c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c0004d40 0x7fb9c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d4930 con 0x7fb9c00039e0 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c000bba0 con 0x7fb9c0004d40 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c000bda0 con 0x7fb9dc0b5490 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9c00039e0 0x7fb9c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9e1400640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c0004d40 0x7fb9c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9e0a00640 1 -- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 msgr2=0x7fb9c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9d3e00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 0x7fb9c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.018+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9dc0b5490 0x7fb9c0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.019+0000 7fb9e0a00640 1 -- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c0004d40 msgr2=0x7fb9c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.019+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c0004d40 0x7fb9c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.019+0000 7fb9e0a00640 1 -- 172.21.15.114:0/1972391811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c0003520 con 0x7fb9c00039e0 2024-03-06T07:28:32.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.019+0000 7fb9e1400640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9c0004d40 0x7fb9c00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.020+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9c00039e0 0x7fb9c0003e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc001750 tx=0x7fb9c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764b4955384bb4cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.020+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc001370 con 0x7fb9c00039e0 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.020+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc0052d0 con 0x7fb9c00039e0 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.020+0000 7fb9dbe00640 1 -- 172.21.15.114:0/1972391811 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc003830 con 0x7fb9c00039e0 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.020+0000 7fb9e3400640 1 -- 172.21.15.114:0/1972391811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9c00039e0 msgr2=0x7fb9c0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.020+0000 7fb9e3400640 1 --2- 172.21.15.114:0/1972391811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9c00039e0 0x7fb9c0003e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc001750 tx=0x7fb9c0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.020+0000 7fb9e1400640 1 -- 172.21.15.114:0/1972391811 reap_dead start 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.021+0000 7fb9e3400640 1 -- 172.21.15.114:0/1972391811 shutdown_connections 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.021+0000 7fb9e3400640 1 -- 172.21.15.114:0/1972391811 >> 172.21.15.114:0/1972391811 conn(0x7fb9dc0d3e10 msgr2=0x7fb9dc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.021+0000 7fb9e3400640 1 -- 172.21.15.114:0/1972391811 shutdown_connections 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.021+0000 7fb9e3400640 1 -- 172.21.15.114:0/1972391811 wait complete. 2024-03-06T07:28:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.022+0000 7fb9e3400640 1 Processor -- start 2024-03-06T07:28:32.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.022+0000 7fb9e3400640 1 -- start start 2024-03-06T07:28:32.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.023+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c00039e0 0x7fb9dc151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.023+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9dc0ca550 0x7fb9dc14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.023+0000 7fb9e3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9dc14b060 0x7fb9dc0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.023+0000 7fb9e3400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0b3f60 con 0x7fb9dc14b060 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.023+0000 7fb9e3400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0b3c80 con 0x7fb9dc0ca550 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.023+0000 7fb9e3400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0d1bf0 con 0x7fb9c00039e0 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9dc14b060 0x7fb9dc0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9dc14b060 0x7fb9dc0cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55238/0 (socket says 172.21.15.114:55238) 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 -- 172.21.15.114:0/717881102 learned_addr learned my addr 172.21.15.114:0/717881102 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c00039e0 0x7fb9dc151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 -- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c00039e0 msgr2=0x7fb9dc151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c00039e0 0x7fb9dc151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 -- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9dc0ca550 msgr2=0x7fb9dc14db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9dc0ca550 0x7fb9dc14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 -- 172.21.15.114:0/717881102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc14e040 con 0x7fb9dc14b060 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c00039e0 0x7fb9dc151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9e1400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9dc14b060 0x7fb9dc0cc020 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009840 tx=0x7fb9d4009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.024+0000 7fb9d9600640 1 -- 172.21.15.114:0/717881102 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4019070 con 0x7fb9dc14b060 2024-03-06T07:28:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.025+0000 7fb9d9600640 1 -- 172.21.15.114:0/717881102 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d4007ce0 con 0x7fb9dc14b060 2024-03-06T07:28:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.025+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc0caf80 con 0x7fb9dc14b060 2024-03-06T07:28:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.025+0000 7fb9d9600640 1 -- 172.21.15.114:0/717881102 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4014640 con 0x7fb9dc14b060 2024-03-06T07:28:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.025+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc0cb540 con 0x7fb9dc14b060 2024-03-06T07:28:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.026+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc04f330 con 0x7fb9dc14b060 2024-03-06T07:28:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.027+0000 7fb9d9600640 1 -- 172.21.15.114:0/717881102 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9d4005d10 con 0x7fb9dc14b060 2024-03-06T07:28:32.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.028+0000 7fb9d9600640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb9b806b500 0x7fb9b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.028+0000 7fb9d9600640 1 -- 172.21.15.114:0/717881102 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9d4089bd0 con 0x7fb9dc14b060 2024-03-06T07:28:32.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.028+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb9b806b500 0x7fb9b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.029+0000 7fb9e0a00640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb9b806b500 0x7fb9b806d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc033000 tx=0x7fb9cc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.032+0000 7fb9d9600640 1 -- 172.21.15.114:0/717881102 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9dc14e040 con 0x7fb9dc14b060 2024-03-06T07:28:32.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.224+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9dc074a50 con 0x7fb9dc14b060 2024-03-06T07:28:32.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.225+0000 7fb9d9600640 1 -- 172.21.15.114:0/717881102 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9d4058a50 con 0x7fb9dc14b060 2024-03-06T07:28:32.226 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:32.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb9b806b500 msgr2=0x7fb9b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb9b806b500 0x7fb9b806d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc033000 tx=0x7fb9cc009030 comp rx=0 tx=0).stop 2024-03-06T07:28:32.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9dc14b060 msgr2=0x7fb9dc0cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9dc14b060 0x7fb9dc0cc020 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009840 tx=0x7fb9d4009d00 comp rx=0 tx=0).stop 2024-03-06T07:28:32.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 shutdown_connections 2024-03-06T07:28:32.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb9b806b500 0x7fb9b806d9c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9dc14b060 0x7fb9dc0cc020 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9dc0ca550 0x7fb9dc14db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 --2- 172.21.15.114:0/717881102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb9c00039e0 0x7fb9dc151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 >> 172.21.15.114:0/717881102 conn(0x7fb9dc0d3e10 msgr2=0x7fb9dc06cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 shutdown_connections 2024-03-06T07:28:32.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.227+0000 7fb9e3400640 1 -- 172.21.15.114:0/717881102 wait complete. 2024-03-06T07:28:32.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:32.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.447+0000 7fb0cd200640 1 Processor -- start 2024-03-06T07:28:32.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.448+0000 7fb0cd200640 1 -- start start 2024-03-06T07:28:32.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.450+0000 7fb0cd200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0c815e130 0x7fb0c80cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.450+0000 7fb0cd200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0c80cfaf0 0x7fb0c80d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.450+0000 7fb0cd200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c80d13e0 con 0x7fb0c80d23e0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.450+0000 7fb0cd200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c80cf120 con 0x7fb0c80cfaf0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.450+0000 7fb0cd200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c80cead0 con 0x7fb0c815e130 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.450+0000 7fb0bfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0c80cfaf0 0x7fb0c80d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0c7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb0c80d23e0 0x7fb0c80c7970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49212/0 (socket says 172.21.15.114:49212) 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0c7e00640 1 -- 172.21.15.114:0/3582039552 learned_addr learned my addr 172.21.15.114:0/3582039552 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0c7400640 1 --2- 172.21.15.114:0/3582039552 >> v2:172.21.15.177:3300/0 conn(0x7fb0c815e130 0x7fb0c80cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0bfe00640 1 -- 172.21.15.114:0/3582039552 >> v2:172.21.15.177:3300/0 conn(0x7fb0c815e130 msgr2=0x7fb0c80cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0bfe00640 1 --2- 172.21.15.114:0/3582039552 >> v2:172.21.15.177:3300/0 conn(0x7fb0c815e130 0x7fb0c80cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0bfe00640 1 -- 172.21.15.114:0/3582039552 >> v1:172.21.15.114:6789/0 conn(0x7fb0c80d23e0 legacy=0x7fb0c80c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0bfe00640 1 -- 172.21.15.114:0/3582039552 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c80cedb0 con 0x7fb0c80cfaf0 2024-03-06T07:28:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.451+0000 7fb0c6a00640 1 -- 172.21.15.114:0/3582039552 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221067108 0 0) 0x7fb0c80d13e0 con 0x7fb0c80d23e0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.452+0000 7fb0bfe00640 1 --2- 172.21.15.114:0/3582039552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c80cfaf0 0x7fb0c80d1ea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8067ce0 tx=0x7fb0b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798de8e3c133c2d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.452+0000 7fb0c6a00640 1 -- 172.21.15.114:0/3582039552 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0035020 con 0x7fb0c80cfaf0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.452+0000 7fb0c6a00640 1 -- 172.21.15.114:0/3582039552 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0031830 con 0x7fb0c80cfaf0 2024-03-06T07:28:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.452+0000 7fb0c6a00640 1 -- 172.21.15.114:0/3582039552 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0031c00 con 0x7fb0c80cfaf0 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 -- 172.21.15.114:0/3582039552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c80cfaf0 msgr2=0x7fb0c80d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 --2- 172.21.15.114:0/3582039552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c80cfaf0 0x7fb0c80d1ea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8067ce0 tx=0x7fb0b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 -- 172.21.15.114:0/3582039552 shutdown_connections 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 --2- 172.21.15.114:0/3582039552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c80cfaf0 0x7fb0c80d1ea0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 --2- 172.21.15.114:0/3582039552 >> v2:172.21.15.177:3300/0 conn(0x7fb0c815e130 0x7fb0c80cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 -- 172.21.15.114:0/3582039552 >> 172.21.15.114:0/3582039552 conn(0x7fb0c809d070 msgr2=0x7fb0c80d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 -- 172.21.15.114:0/3582039552 shutdown_connections 2024-03-06T07:28:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.453+0000 7fb0cd200640 1 -- 172.21.15.114:0/3582039552 wait complete. 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.456+0000 7fb0cd200640 1 Processor -- start 2024-03-06T07:28:32.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.456+0000 7fb0cd200640 1 -- start start 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0cd200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c80cfaf0 0x7fb0c80b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0cd200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0c80d23e0 0x7fb0c81603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0cd200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c815e130 0x7fb0c81553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0cd200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c8200 con 0x7fb0c815e130 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0cd200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c8080 con 0x7fb0c80cfaf0 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0cd200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80d06b0 con 0x7fb0c80d23e0 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c815e130 0x7fb0c81553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c815e130 0x7fb0c81553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55254/0 (socket says 172.21.15.114:55254) 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 -- 172.21.15.114:0/1679098556 learned_addr learned my addr 172.21.15.114:0/1679098556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0bfe00640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0c80d23e0 0x7fb0c81603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7400640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c80cfaf0 0x7fb0c80b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 -- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0c80d23e0 msgr2=0x7fb0c81603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0c80d23e0 0x7fb0c81603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 -- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c80cfaf0 msgr2=0x7fb0c80b6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c80cfaf0 0x7fb0c80b6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.458+0000 7fb0c7e00640 1 -- 172.21.15.114:0/1679098556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c815e720 con 0x7fb0c815e130 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.459+0000 7fb0c7e00640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c815e130 0x7fb0c81553d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000c7a0 tx=0x7fb0c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.459+0000 7fb0bf400640 1 -- 172.21.15.114:0/1679098556 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0018070 con 0x7fb0c815e130 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.459+0000 7fb0bf400640 1 -- 172.21.15.114:0/1679098556 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c000a640 con 0x7fb0c815e130 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.459+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c815ea00 con 0x7fb0c815e130 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.459+0000 7fb0bf400640 1 -- 172.21.15.114:0/1679098556 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c000aa70 con 0x7fb0c815e130 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.459+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c8160900 con 0x7fb0c815e130 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.460+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c804f330 con 0x7fb0c815e130 2024-03-06T07:28:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.461+0000 7fb0bf400640 1 -- 172.21.15.114:0/1679098556 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0c001b070 con 0x7fb0c815e130 2024-03-06T07:28:32.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.462+0000 7fb0bf400640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0a406b4d0 0x7fb0a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.462+0000 7fb0bf400640 1 -- 172.21.15.114:0/1679098556 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0c00898c0 con 0x7fb0c815e130 2024-03-06T07:28:32.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.462+0000 7fb0c7400640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0a406b4d0 0x7fb0a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.463+0000 7fb0c7400640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0a406b4d0 0x7fb0a406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8006fd0 tx=0x7fb0b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.470+0000 7fb0bf400640 1 -- 172.21.15.114:0/1679098556 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0c00586a0 con 0x7fb0c815e130 2024-03-06T07:28:32.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.652+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0c809d690 con 0x7fb0c815e130 2024-03-06T07:28:32.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.653+0000 7fb0bf400640 1 -- 172.21.15.114:0/1679098556 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0c0057fa0 con 0x7fb0c815e130 2024-03-06T07:28:32.654 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0a406b4d0 msgr2=0x7fb0a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0a406b4d0 0x7fb0a406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8006fd0 tx=0x7fb0b8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c815e130 msgr2=0x7fb0c81553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c815e130 0x7fb0c81553d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000c7a0 tx=0x7fb0c000cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 shutdown_connections 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0a406b4d0 0x7fb0a406d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0c815e130 0x7fb0c81553d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0c80d23e0 0x7fb0c81603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 --2- 172.21.15.114:0/1679098556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0c80cfaf0 0x7fb0c80b6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 >> 172.21.15.114:0/1679098556 conn(0x7fb0c809d070 msgr2=0x7fb0c80b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 shutdown_connections 2024-03-06T07:28:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.655+0000 7fb0cd200640 1 -- 172.21.15.114:0/1679098556 wait complete. 2024-03-06T07:28:32.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:32.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.870+0000 7f52a3000640 1 Processor -- start 2024-03-06T07:28:32.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.871+0000 7f52a3000640 1 -- start start 2024-03-06T07:28:32.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.872+0000 7f52a3000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f529c15fc10 0x7f529c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.872+0000 7f52a3000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f529c15fff0 0x7f529c160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.872+0000 7f52a3000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c0b46c0 con 0x7f529c0ce4a0 2024-03-06T07:28:32.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.872+0000 7f52a3000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c0b49a0 con 0x7f529c15fff0 2024-03-06T07:28:32.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.872+0000 7f52a3000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c072c90 con 0x7f529c15fc10 2024-03-06T07:28:32.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f5293e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f529c15fff0 0x7f529c160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f5293e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f529c15fff0 0x7f529c160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55266/0 (socket says 172.21.15.114:55266) 2024-03-06T07:28:32.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f5293e00640 1 -- 172.21.15.114:0/1271714334 learned_addr learned my addr 172.21.15.114:0/1271714334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:32.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f52a1000640 1 --1- 172.21.15.114:0/1271714334 >> v1:172.21.15.114:6789/0 conn(0x7f529c0ce4a0 0x7f529c0ce8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49228/0 (socket says 172.21.15.114:49228) 2024-03-06T07:28:32.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f529be00640 1 --2- 172.21.15.114:0/1271714334 >> v2:172.21.15.177:3300/0 conn(0x7f529c15fc10 0x7f529c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f5293e00640 1 -- 172.21.15.114:0/1271714334 >> v2:172.21.15.177:3300/0 conn(0x7f529c15fc10 msgr2=0x7f529c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f5293e00640 1 --2- 172.21.15.114:0/1271714334 >> v2:172.21.15.177:3300/0 conn(0x7f529c15fc10 0x7f529c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f5293e00640 1 -- 172.21.15.114:0/1271714334 >> v1:172.21.15.114:6789/0 conn(0x7f529c0ce4a0 legacy=0x7f529c0ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.873+0000 7f5293e00640 1 -- 172.21.15.114:0/1271714334 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c160990 con 0x7f529c15fff0 2024-03-06T07:28:32.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.874+0000 7f5293e00640 1 --2- 172.21.15.114:0/1271714334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c160450 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f529c0cf770 tx=0x7f528402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c5536bccbb71d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f529b400640 1 -- 172.21.15.114:0/1271714334 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284037070 con 0x7f529c15fff0 2024-03-06T07:28:32.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f529b400640 1 -- 172.21.15.114:0/1271714334 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284004230 con 0x7f529c15fff0 2024-03-06T07:28:32.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f529b400640 1 -- 172.21.15.114:0/1271714334 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284003610 con 0x7f529c15fff0 2024-03-06T07:28:32.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f52a3000640 1 -- 172.21.15.114:0/1271714334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 msgr2=0x7f529c160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f52a3000640 1 --2- 172.21.15.114:0/1271714334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c160450 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f529c0cf770 tx=0x7f528402b7e0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f52a3000640 1 -- 172.21.15.114:0/1271714334 shutdown_connections 2024-03-06T07:28:32.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f52a3000640 1 --2- 172.21.15.114:0/1271714334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c160450 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f52a3000640 1 --2- 172.21.15.114:0/1271714334 >> v2:172.21.15.177:3300/0 conn(0x7f529c15fc10 0x7f529c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f52a3000640 1 -- 172.21.15.114:0/1271714334 >> 172.21.15.114:0/1271714334 conn(0x7f529c0b1940 msgr2=0x7f529c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.875+0000 7f52a3000640 1 -- 172.21.15.114:0/1271714334 shutdown_connections 2024-03-06T07:28:32.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.876+0000 7f52a3000640 1 -- 172.21.15.114:0/1271714334 wait complete. 2024-03-06T07:28:32.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.877+0000 7f52a3000640 1 Processor -- start 2024-03-06T07:28:32.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.877+0000 7f52a3000640 1 -- start start 2024-03-06T07:28:32.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.878+0000 7f52a3000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f529c0ce4a0 0x7f529c14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.878+0000 7f52a3000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f529c15fc10 0x7f529c0adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.878+0000 7f52a3000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c0ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.878+0000 7f52a3000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c072a30 con 0x7f529c15fff0 2024-03-06T07:28:32.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.878+0000 7f52a3000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c0728b0 con 0x7f529c0ce4a0 2024-03-06T07:28:32.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.878+0000 7f52a3000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c15c2e0 con 0x7f529c15fc10 2024-03-06T07:28:32.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.878+0000 7f52a1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c0ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c0ae1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55274/0 (socket says 172.21.15.114:55274) 2024-03-06T07:28:32.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 -- 172.21.15.114:0/3229037138 learned_addr learned my addr 172.21.15.114:0/3229037138 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:32.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f529be00640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f529c0ce4a0 0x7f529c14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f5293e00640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f529c15fc10 0x7f529c0adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 -- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f529c15fc10 msgr2=0x7f529c0adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f529c15fc10 0x7f529c0adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 -- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f529c0ce4a0 msgr2=0x7f529c14e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f529c0ce4a0 0x7f529c14e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 -- 172.21.15.114:0/3229037138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c09cc60 con 0x7f529c15fff0 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.879+0000 7f52a1000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c0ae1c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f529400caf0 tx=0x7f52940020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.880+0000 7f5298c00640 1 -- 172.21.15.114:0/3229037138 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294018070 con 0x7f529c15fff0 2024-03-06T07:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.880+0000 7f5298c00640 1 -- 172.21.15.114:0/3229037138 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529400a760 con 0x7f529c15fff0 2024-03-06T07:28:32.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.880+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c09cf40 con 0x7f529c15fff0 2024-03-06T07:28:32.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.880+0000 7f5298c00640 1 -- 172.21.15.114:0/3229037138 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529400ab90 con 0x7f529c15fff0 2024-03-06T07:28:32.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.880+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c09d500 con 0x7f529c15fff0 2024-03-06T07:28:32.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.881+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529c14ea80 con 0x7f529c15fff0 2024-03-06T07:28:32.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.882+0000 7f5298c00640 1 -- 172.21.15.114:0/3229037138 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f529401b070 con 0x7f529c15fff0 2024-03-06T07:28:32.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.883+0000 7f5298c00640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f527806b4d0 0x7f527806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.883+0000 7f5298c00640 1 -- 172.21.15.114:0/3229037138 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52940899f0 con 0x7f529c15fff0 2024-03-06T07:28:32.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.883+0000 7f529be00640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f527806b4d0 0x7f527806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.884+0000 7f529be00640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f527806b4d0 0x7f527806d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f528c006fd0 tx=0x7f528c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:32.887+0000 7f5298c00640 1 -- 172.21.15.114:0/3229037138 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5294058900 con 0x7f529c15fff0 2024-03-06T07:28:33.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.082+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f529c04f330 con 0x7f529c15fff0 2024-03-06T07:28:33.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.083+0000 7f5298c00640 1 -- 172.21.15.114:0/3229037138 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5294058200 con 0x7f529c15fff0 2024-03-06T07:28:33.084 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f527806b4d0 msgr2=0x7f527806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f527806b4d0 0x7f527806d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f528c006fd0 tx=0x7f528c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 msgr2=0x7f529c0ae1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c0ae1c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f529400caf0 tx=0x7f52940020f0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 shutdown_connections 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f527806b4d0 0x7f527806d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f529c15fff0 0x7f529c0ae1c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f529c15fc10 0x7f529c0adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 --2- 172.21.15.114:0/3229037138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f529c0ce4a0 0x7f529c14e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 >> 172.21.15.114:0/3229037138 conn(0x7f529c0b1940 msgr2=0x7f529c0acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 shutdown_connections 2024-03-06T07:28:33.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.085+0000 7f52a3000640 1 -- 172.21.15.114:0/3229037138 wait complete. 2024-03-06T07:28:33.115 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:28:33.115 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:33.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:33.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.314+0000 7fbac8000640 1 Processor -- start 2024-03-06T07:28:33.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.314+0000 7fbac8000640 1 -- start start 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac8000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbac015f8d0 0x7fbac00cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac8000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac00ccc40 0x7fbac015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac8000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac0070de0 con 0x7fbac015f4f0 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac8000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac00cd0a0 con 0x7fbac00ccc40 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac8000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac0160a20 con 0x7fbac015f8d0 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac00ccc40 0x7fbac015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac00ccc40 0x7fbac015fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55276/0 (socket says 172.21.15.114:55276) 2024-03-06T07:28:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.317+0000 7fbac6000640 1 -- 172.21.15.114:0/4236976641 learned_addr learned my addr 172.21.15.114:0/4236976641 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.318+0000 7fbac6000640 1 -- 172.21.15.114:0/4236976641 >> v2:172.21.15.177:3300/0 conn(0x7fbac015f8d0 msgr2=0x7fbac00cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.318+0000 7fbac6000640 1 --2- 172.21.15.114:0/4236976641 >> v2:172.21.15.177:3300/0 conn(0x7fbac015f8d0 0x7fbac00cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.318+0000 7fbac6000640 1 -- 172.21.15.114:0/4236976641 >> v1:172.21.15.177:6789/0 conn(0x7fbac015f4f0 legacy=0x7fbac00706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.318+0000 7fbac6000640 1 -- 172.21.15.114:0/4236976641 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac006ef50 con 0x7fbac00ccc40 2024-03-06T07:28:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.319+0000 7fbac6000640 1 --2- 172.21.15.114:0/4236976641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac00ccc40 0x7fbac015fd50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbac00ce570 tx=0x7fbab802bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a69c776e3c1295 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.319+0000 7fbabfe00640 1 -- 172.21.15.114:0/4236976641 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab802caa0 con 0x7fbac00ccc40 2024-03-06T07:28:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.319+0000 7fbabfe00640 1 -- 172.21.15.114:0/4236976641 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab802cc40 con 0x7fbac00ccc40 2024-03-06T07:28:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.319+0000 7fbabfe00640 1 -- 172.21.15.114:0/4236976641 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab80343f0 con 0x7fbac00ccc40 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.320+0000 7fbac8000640 1 -- 172.21.15.114:0/4236976641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac00ccc40 msgr2=0x7fbac015fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.320+0000 7fbac8000640 1 --2- 172.21.15.114:0/4236976641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac00ccc40 0x7fbac015fd50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbac00ce570 tx=0x7fbab802bf00 comp rx=0 tx=0).stop 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.320+0000 7fbac8000640 1 -- 172.21.15.114:0/4236976641 shutdown_connections 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.320+0000 7fbac8000640 1 --2- 172.21.15.114:0/4236976641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac00ccc40 0x7fbac015fd50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.320+0000 7fbac8000640 1 --2- 172.21.15.114:0/4236976641 >> v2:172.21.15.177:3300/0 conn(0x7fbac015f8d0 0x7fbac00cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.320+0000 7fbac8000640 1 -- 172.21.15.114:0/4236976641 >> 172.21.15.114:0/4236976641 conn(0x7fbac00b3800 msgr2=0x7fbac00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.321+0000 7fbac8000640 1 -- 172.21.15.114:0/4236976641 shutdown_connections 2024-03-06T07:28:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.321+0000 7fbac8000640 1 -- 172.21.15.114:0/4236976641 wait complete. 2024-03-06T07:28:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.322+0000 7fbac8000640 1 Processor -- start 2024-03-06T07:28:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.322+0000 7fbac8000640 1 -- start start 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.323+0000 7fbac8000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac00ccc40 0x7fbac015b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.323+0000 7fbac8000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac015f4f0 0x7fbac00afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.323+0000 7fbac8000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac015f8d0 0x7fbac00b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.323+0000 7fbac8000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0160410 con 0x7fbac015f4f0 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.323+0000 7fbac8000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0160290 con 0x7fbac00ccc40 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.323+0000 7fbac8000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0160590 con 0x7fbac015f8d0 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac015f4f0 0x7fbac00afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac015f4f0 0x7fbac00afb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55290/0 (socket says 172.21.15.114:55290) 2024-03-06T07:28:33.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 -- 172.21.15.114:0/1382928115 learned_addr learned my addr 172.21.15.114:0/1382928115 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac5600640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac00ccc40 0x7fbac015b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac6000640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac015f8d0 0x7fbac00b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 -- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac015f8d0 msgr2=0x7fbac00b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac015f8d0 0x7fbac00b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 -- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac00ccc40 msgr2=0x7fbac015b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac00ccc40 0x7fbac015b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac4c00640 1 -- 172.21.15.114:0/1382928115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac01f44c0 con 0x7fbac015f4f0 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.324+0000 7fbac5600640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac00ccc40 0x7fbac015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.325+0000 7fbac4c00640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac015f4f0 0x7fbac00afb30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b410 tx=0x7fbaac009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.325+0000 7fbabd600640 1 -- 172.21.15.114:0/1382928115 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac015020 con 0x7fbac015f4f0 2024-03-06T07:28:33.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.325+0000 7fbabd600640 1 -- 172.21.15.114:0/1382928115 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac011850 con 0x7fbac015f4f0 2024-03-06T07:28:33.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.325+0000 7fbabd600640 1 -- 172.21.15.114:0/1382928115 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac011c40 con 0x7fbac015f4f0 2024-03-06T07:28:33.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.325+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac015b6d0 con 0x7fbac015f4f0 2024-03-06T07:28:33.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.325+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac01f4a60 con 0x7fbac015f4f0 2024-03-06T07:28:33.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.326+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac00d11c0 con 0x7fbac015f4f0 2024-03-06T07:28:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.327+0000 7fbabd600640 1 -- 172.21.15.114:0/1382928115 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaac018020 con 0x7fbac015f4f0 2024-03-06T07:28:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.328+0000 7fbabd600640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.328+0000 7fbabd600640 1 -- 172.21.15.114:0/1382928115 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbaac089e50 con 0x7fbac015f4f0 2024-03-06T07:28:33.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.328+0000 7fbac5600640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.329+0000 7fbac5600640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbac00b7b20 tx=0x7fbab4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.335+0000 7fbabd600640 1 -- 172.21.15.114:0/1382928115 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbac00d11c0 con 0x7fbac015f4f0 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.529+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbac004f330 con 0x7fbac015f4f0 2024-03-06T07:28:33.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.532+0000 7fbabd600640 1 -- 172.21.15.114:0/1382928115 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fbaac01f030 con 0x7fbac015f4f0 2024-03-06T07:28:33.534 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:33.534 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:27:56.006199+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:33.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.534+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 msgr2=0x7fbaa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.534+0000 7fbac8000640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbac00b7b20 tx=0x7fbab4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac015f4f0 msgr2=0x7fbac00afb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac015f4f0 0x7fbac00afb30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b410 tx=0x7fbaac009f90 comp rx=0 tx=0).stop 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 shutdown_connections 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac015f8d0 0x7fbac00b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac015f4f0 0x7fbac00afb30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 --2- 172.21.15.114:0/1382928115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac00ccc40 0x7fbac015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 >> 172.21.15.114:0/1382928115 conn(0x7fbac00b3800 msgr2=0x7fbac00b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 shutdown_connections 2024-03-06T07:28:33.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.535+0000 7fbac8000640 1 -- 172.21.15.114:0/1382928115 wait complete. 2024-03-06T07:28:33.554 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:33.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:33.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.766+0000 7f42b9c00640 1 Processor -- start 2024-03-06T07:28:33.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.766+0000 7f42b9c00640 1 -- start start 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b9c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42b415c050 0x7f42b406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b9c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b40cc9f0 0x7f42b40cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b9c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b40725d0 con 0x7f42b4160400 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b9c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b406b5a0 con 0x7f42b415c050 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b9c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b406b880 con 0x7f42b40cc9f0 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42b415c050 0x7f42b406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42b415c050 0x7f42b406afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55294/0 (socket says 172.21.15.114:55294) 2024-03-06T07:28:33.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.768+0000 7f42b3e00640 1 -- 172.21.15.114:0/960600208 learned_addr learned my addr 172.21.15.114:0/960600208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:33.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.769+0000 7f42b9200640 1 --2- 172.21.15.114:0/960600208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b40cc9f0 0x7f42b40cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.769+0000 7f42b3e00640 1 -- 172.21.15.114:0/960600208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b40cc9f0 msgr2=0x7f42b40cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.769+0000 7f42b3e00640 1 --2- 172.21.15.114:0/960600208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b40cc9f0 0x7f42b40cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.769+0000 7f42b3e00640 1 -- 172.21.15.114:0/960600208 >> v1:172.21.15.177:6789/0 conn(0x7f42b4160400 legacy=0x7f42b4072d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.769+0000 7f42b3e00640 1 -- 172.21.15.114:0/960600208 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b4072840 con 0x7f42b415c050 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.770+0000 7f42b3e00640 1 --2- 172.21.15.114:0/960600208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b406afc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009870 tx=0x7f42a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3bf2f4b4a943d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.770+0000 7f42b2a00640 1 -- 172.21.15.114:0/960600208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4035020 con 0x7f42b415c050 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.770+0000 7f42b2a00640 1 -- 172.21.15.114:0/960600208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a4031640 con 0x7f42b415c050 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.770+0000 7f42b2a00640 1 -- 172.21.15.114:0/960600208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a402d040 con 0x7f42b415c050 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 -- 172.21.15.114:0/960600208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 msgr2=0x7f42b406afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 --2- 172.21.15.114:0/960600208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b406afc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009870 tx=0x7f42a402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 -- 172.21.15.114:0/960600208 shutdown_connections 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 --2- 172.21.15.114:0/960600208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b40cc9f0 0x7f42b40cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 --2- 172.21.15.114:0/960600208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b406afc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 -- 172.21.15.114:0/960600208 >> 172.21.15.114:0/960600208 conn(0x7f42b40b1d00 msgr2=0x7f42b40b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 -- 172.21.15.114:0/960600208 shutdown_connections 2024-03-06T07:28:33.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.771+0000 7f42b9c00640 1 -- 172.21.15.114:0/960600208 wait complete. 2024-03-06T07:28:33.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.773+0000 7f42b9c00640 1 Processor -- start 2024-03-06T07:28:33.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.773+0000 7f42b9c00640 1 -- start start 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42b40cc9f0 0x7f42b415d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b9c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b40998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b9c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b4160400 0x7f42b40c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b9c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40735f0 con 0x7f42b415c050 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b9c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4073470 con 0x7f42b40cc9f0 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b9c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b406bb60 con 0x7f42b4160400 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b40998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b40998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55310/0 (socket says 172.21.15.114:55310) 2024-03-06T07:28:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b3400640 1 -- 172.21.15.114:0/1907887455 learned_addr learned my addr 172.21.15.114:0/1907887455 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b9200640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b4160400 0x7f42b40c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.774+0000 7f42b3e00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42b40cc9f0 0x7f42b415d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b3400640 1 -- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b4160400 msgr2=0x7f42b40c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b3400640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b4160400 0x7f42b40c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b3400640 1 -- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42b40cc9f0 msgr2=0x7f42b415d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b3400640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42b40cc9f0 0x7f42b415d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b3400640 1 -- 172.21.15.114:0/1907887455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b41f6300 con 0x7f42b415c050 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b3e00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42b40cc9f0 0x7f42b415d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b3400640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b40998b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f42a000b3f0 tx=0x7f42a000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42abe00640 1 -- 172.21.15.114:0/1907887455 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0015020 con 0x7f42b415c050 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42abe00640 1 -- 172.21.15.114:0/1907887455 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0011830 con 0x7f42b415c050 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42abe00640 1 -- 172.21.15.114:0/1907887455 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0011c20 con 0x7f42b415c050 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b415c640 con 0x7f42b415c050 2024-03-06T07:28:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.775+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b415cc00 con 0x7f42b415c050 2024-03-06T07:28:33.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.777+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b404f330 con 0x7f42b415c050 2024-03-06T07:28:33.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.778+0000 7f42abe00640 1 -- 172.21.15.114:0/1907887455 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42a0018020 con 0x7f42b415c050 2024-03-06T07:28:33.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.779+0000 7f42abe00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f429406b4d0 0x7f429406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.779+0000 7f42abe00640 1 -- 172.21.15.114:0/1907887455 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42a0089e40 con 0x7f42b415c050 2024-03-06T07:28:33.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.779+0000 7f42b3e00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f429406b4d0 0x7f429406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.780+0000 7f42b3e00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f429406b4d0 0x7f429406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f42a4036000 tx=0x7f42a4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.786+0000 7f42abe00640 1 -- 172.21.15.114:0/1907887455 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42a001f030 con 0x7f42b415c050 2024-03-06T07:28:33.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.984+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42b40cd720 con 0x7f42b415c050 2024-03-06T07:28:33.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.985+0000 7f42abe00640 1 -- 172.21.15.114:0/1907887455 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f42a0058cc0 con 0x7f42b415c050 2024-03-06T07:28:33.986 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f429406b4d0 msgr2=0x7f429406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f429406b4d0 0x7f429406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f42a4036000 tx=0x7f42a4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 msgr2=0x7f42b40998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b40998b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f42a000b3f0 tx=0x7f42a000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 shutdown_connections 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42b4160400 0x7f42b40c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f429406b4d0 0x7f429406d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42b415c050 0x7f42b40998b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 --2- 172.21.15.114:0/1907887455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42b40cc9f0 0x7f42b415d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 >> 172.21.15.114:0/1907887455 conn(0x7f42b40b1d00 msgr2=0x7f42b40cb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 shutdown_connections 2024-03-06T07:28:33.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:33.987+0000 7f42b9c00640 1 -- 172.21.15.114:0/1907887455 wait complete. 2024-03-06T07:28:34.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:34.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.205+0000 7f14ea800640 1 Processor -- start 2024-03-06T07:28:34.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.205+0000 7f14ea800640 1 -- start start 2024-03-06T07:28:34.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.206+0000 7f14ea800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f14e40d06d0 0x7f14e40d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.206+0000 7f14ea800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14e40d1db0 0x7f14e40d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.206+0000 7f14ea800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e40d0ff0 con 0x7f14e40b4b90 2024-03-06T07:28:34.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.206+0000 7f14ea800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e40cf2f0 con 0x7f14e40d1db0 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.206+0000 7f14ea800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e40cf5d0 con 0x7f14e40d06d0 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14e40d1db0 0x7f14e40d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f14e40d1db0 0x7f14e40d2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55322/0 (socket says 172.21.15.114:55322) 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9e00640 1 -- 172.21.15.114:0/3827640523 learned_addr learned my addr 172.21.15.114:0/3827640523 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9400640 1 --2- 172.21.15.114:0/3827640523 >> v2:172.21.15.177:3300/0 conn(0x7f14e40d06d0 0x7f14e40d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9e00640 1 -- 172.21.15.114:0/3827640523 >> v2:172.21.15.177:3300/0 conn(0x7f14e40d06d0 msgr2=0x7f14e40d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9e00640 1 --2- 172.21.15.114:0/3827640523 >> v2:172.21.15.177:3300/0 conn(0x7f14e40d06d0 0x7f14e40d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9e00640 1 -- 172.21.15.114:0/3827640523 >> v1:172.21.15.114:6789/0 conn(0x7f14e40b4b90 legacy=0x7f14e40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.207+0000 7f14e9e00640 1 -- 172.21.15.114:0/3827640523 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e40cf8b0 con 0x7f14e40d1db0 2024-03-06T07:28:34.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.208+0000 7f14e9e00640 1 --2- 172.21.15.114:0/3827640523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40d2240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f14e408e0f0 tx=0x7f14dc02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c22a459dc1ffbb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14e3e00640 1 -- 172.21.15.114:0/3827640523 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02cb10 con 0x7f14e40d1db0 2024-03-06T07:28:34.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14e3e00640 1 -- 172.21.15.114:0/3827640523 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc02ccb0 con 0x7f14e40d1db0 2024-03-06T07:28:34.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14e3e00640 1 -- 172.21.15.114:0/3827640523 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02ce60 con 0x7f14e40d1db0 2024-03-06T07:28:34.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14ea800640 1 -- 172.21.15.114:0/3827640523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 msgr2=0x7f14e40d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14ea800640 1 --2- 172.21.15.114:0/3827640523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40d2240 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f14e408e0f0 tx=0x7f14dc02bf70 comp rx=0 tx=0).stop 2024-03-06T07:28:34.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14ea800640 1 -- 172.21.15.114:0/3827640523 shutdown_connections 2024-03-06T07:28:34.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14ea800640 1 --2- 172.21.15.114:0/3827640523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40d2240 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.209+0000 7f14ea800640 1 --2- 172.21.15.114:0/3827640523 >> v2:172.21.15.177:3300/0 conn(0x7f14e40d06d0 0x7f14e40d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.210+0000 7f14ea800640 1 -- 172.21.15.114:0/3827640523 >> 172.21.15.114:0/3827640523 conn(0x7f14e409d070 msgr2=0x7f14e40d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.210+0000 7f14ea800640 1 -- 172.21.15.114:0/3827640523 shutdown_connections 2024-03-06T07:28:34.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.210+0000 7f14ea800640 1 -- 172.21.15.114:0/3827640523 wait complete. 2024-03-06T07:28:34.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.211+0000 7f14ea800640 1 Processor -- start 2024-03-06T07:28:34.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.211+0000 7f14ea800640 1 -- start start 2024-03-06T07:28:34.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.212+0000 7f14ea800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e40b4b90 0x7f14e4158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.212+0000 7f14ea800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e40d06d0 0x7f14e40cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.212+0000 7f14ea800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.212+0000 7f14ea800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e40d2780 con 0x7f14e40d1db0 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.212+0000 7f14ea800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e40b0960 con 0x7f14e40d06d0 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.212+0000 7f14ea800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e40d2900 con 0x7f14e40b4b90 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.212+0000 7f14e9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55334/0 (socket says 172.21.15.114:55334) 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 -- 172.21.15.114:0/1156409932 learned_addr learned my addr 172.21.15.114:0/1156409932 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9400640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e40b4b90 0x7f14e4158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e8a00640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e40d06d0 0x7f14e40cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 -- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e40b4b90 msgr2=0x7f14e4158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e40b4b90 0x7f14e4158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 -- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e40d06d0 msgr2=0x7f14e40cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e40d06d0 0x7f14e40cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 -- 172.21.15.114:0/1156409932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e41594e0 con 0x7f14e40d1db0 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e8a00640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e40d06d0 0x7f14e40cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9400640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e40b4b90 0x7f14e4158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.213+0000 7f14e9e00640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40caed0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f14dc02c580 tx=0x7f14dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.214+0000 7f14e1600640 1 -- 172.21.15.114:0/1156409932 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc043020 con 0x7f14e40d1db0 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.214+0000 7f14e1600640 1 -- 172.21.15.114:0/1156409932 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc03e450 con 0x7f14e40d1db0 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.214+0000 7f14e1600640 1 -- 172.21.15.114:0/1156409932 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02ce60 con 0x7f14e40d1db0 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.214+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e40cb900 con 0x7f14e40d1db0 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.214+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e40cbec0 con 0x7f14e40d1db0 2024-03-06T07:28:34.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.215+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e404f330 con 0x7f14e40d1db0 2024-03-06T07:28:34.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.216+0000 7f14e1600640 1 -- 172.21.15.114:0/1156409932 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14dc046020 con 0x7f14e40d1db0 2024-03-06T07:28:34.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.217+0000 7f14e1600640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14c806b500 0x7f14c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.217+0000 7f14e1600640 1 -- 172.21.15.114:0/1156409932 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14dc0ac6a0 con 0x7f14e40d1db0 2024-03-06T07:28:34.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.217+0000 7f14e9400640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14c806b500 0x7f14c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.217+0000 7f14e9400640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14c806b500 0x7f14c806d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14d8006fd0 tx=0x7f14d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.221+0000 7f14e1600640 1 -- 172.21.15.114:0/1156409932 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14dc04d070 con 0x7f14e40d1db0 2024-03-06T07:28:34.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.452+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f14e40d1480 con 0x7f14e40d1db0 2024-03-06T07:28:34.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.453+0000 7f14e1600640 1 -- 172.21.15.114:0/1156409932 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f14dc07b470 con 0x7f14e40d1db0 2024-03-06T07:28:34.455 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:34.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14c806b500 msgr2=0x7f14c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14c806b500 0x7f14c806d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f14d8006fd0 tx=0x7f14d8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 msgr2=0x7f14e40caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40caed0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f14dc02c580 tx=0x7f14dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:34.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 shutdown_connections 2024-03-06T07:28:34.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14c806b500 0x7f14c806d9c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e40d1db0 0x7f14e40caed0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14e40d06d0 0x7f14e40cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 --2- 172.21.15.114:0/1156409932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14e40b4b90 0x7f14e4158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 >> 172.21.15.114:0/1156409932 conn(0x7f14e409d070 msgr2=0x7f14e40d2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 shutdown_connections 2024-03-06T07:28:34.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.457+0000 7f14ea800640 1 -- 172.21.15.114:0/1156409932 wait complete. 2024-03-06T07:28:34.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:34.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.740+0000 7fe902e00640 1 Processor -- start 2024-03-06T07:28:34.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.740+0000 7fe902e00640 1 -- start start 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe902e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8fc15df80 0x7fe8fc06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe902e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8fc0d02a0 0x7fe8fc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe902e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc0d1fb0 con 0x7fe8fc15e360 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe902e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8fc0d2a20 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe902e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8fc09b100 con 0x7fe8fc15df80 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe900e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8fc0d02a0 0x7fe8fc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe900e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8fc0d02a0 0x7fe8fc06ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55344/0 (socket says 172.21.15.114:55344) 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe900e00640 1 -- 172.21.15.114:0/2643670897 learned_addr learned my addr 172.21.15.114:0/2643670897 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe8fb400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe8fc15e360 0x7fe8fc0cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49242/0 (socket says 172.21.15.114:49242) 2024-03-06T07:28:34.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.742+0000 7fe8fbe00640 1 --2- 172.21.15.114:0/2643670897 >> v2:172.21.15.177:3300/0 conn(0x7fe8fc15df80 0x7fe8fc06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.743+0000 7fe900e00640 1 -- 172.21.15.114:0/2643670897 >> v2:172.21.15.177:3300/0 conn(0x7fe8fc15df80 msgr2=0x7fe8fc06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.743+0000 7fe900e00640 1 --2- 172.21.15.114:0/2643670897 >> v2:172.21.15.177:3300/0 conn(0x7fe8fc15df80 0x7fe8fc06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.743+0000 7fe900e00640 1 -- 172.21.15.114:0/2643670897 >> v1:172.21.15.114:6789/0 conn(0x7fe8fc15e360 legacy=0x7fe8fc0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.743+0000 7fe900e00640 1 -- 172.21.15.114:0/2643670897 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc09b3a0 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.743+0000 7fe8faa00640 1 -- 172.21.15.114:0/2643670897 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377361609 0 0) 0x7fe8fc0d1fb0 con 0x7fe8fc15e360 2024-03-06T07:28:34.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.744+0000 7fe900e00640 1 --2- 172.21.15.114:0/2643670897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc06ee40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc071810 tx=0x7fe8f402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac8bfc6382ccfaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.744+0000 7fe8faa00640 1 -- 172.21.15.114:0/2643670897 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f402ca80 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.744+0000 7fe8faa00640 1 -- 172.21.15.114:0/2643670897 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f402cc20 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.744+0000 7fe8faa00640 1 -- 172.21.15.114:0/2643670897 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40343f0 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 -- 172.21.15.114:0/2643670897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 msgr2=0x7fe8fc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 --2- 172.21.15.114:0/2643670897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc06ee40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc071810 tx=0x7fe8f402bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 -- 172.21.15.114:0/2643670897 shutdown_connections 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 --2- 172.21.15.114:0/2643670897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc06ee40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 --2- 172.21.15.114:0/2643670897 >> v2:172.21.15.177:3300/0 conn(0x7fe8fc15df80 0x7fe8fc06c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 -- 172.21.15.114:0/2643670897 >> 172.21.15.114:0/2643670897 conn(0x7fe8fc0b7320 msgr2=0x7fe8fc09a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 -- 172.21.15.114:0/2643670897 shutdown_connections 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.745+0000 7fe902e00640 1 -- 172.21.15.114:0/2643670897 wait complete. 2024-03-06T07:28:34.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.747+0000 7fe902e00640 1 Processor -- start 2024-03-06T07:28:34.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.747+0000 7fe902e00640 1 -- start start 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe902e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe902e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8fc15df80 0x7fe8fc1f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe902e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8fc15e360 0x7fe8fc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe902e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc06c610 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe902e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc0d2500 con 0x7fe8fc15e360 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe902e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc09b680 con 0x7fe8fc15df80 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55360/0 (socket says 172.21.15.114:55360) 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 -- 172.21.15.114:0/3071186817 learned_addr learned my addr 172.21.15.114:0/3071186817 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe900e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8fc15e360 0x7fe8fc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fb400640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8fc15df80 0x7fe8fc1f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 -- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8fc15df80 msgr2=0x7fe8fc1f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8fc15df80 0x7fe8fc1f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 -- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8fc15e360 msgr2=0x7fe8fc0b6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8fc15e360 0x7fe8fc0b6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fbe00640 1 -- 172.21.15.114:0/3071186817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc1f40b0 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe8fb400640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8fc15df80 0x7fe8fc1f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.748+0000 7fe900e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8fc15e360 0x7fe8fc0b6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.749+0000 7fe8fbe00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc159a40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00d800 tx=0x7fe8ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.749+0000 7fe8f3e00640 1 -- 172.21.15.114:0/3071186817 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec00b660 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.749+0000 7fe8f3e00640 1 -- 172.21.15.114:0/3071186817 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec019070 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.749+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc1f4390 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.749+0000 7fe8f3e00640 1 -- 172.21.15.114:0/3071186817 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec00ba20 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.749+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc0b5c10 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.750+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc0b5a00 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.751+0000 7fe8f3e00640 1 -- 172.21.15.114:0/3071186817 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8ec00bbc0 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.752+0000 7fe8f3e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8dc06b4d0 0x7fe8dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.752+0000 7fe8f3e00640 1 -- 172.21.15.114:0/3071186817 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8ec08a6a0 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.752+0000 7fe8fb400640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8dc06b4d0 0x7fe8dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.753+0000 7fe8fb400640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8dc06b4d0 0x7fe8dc06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc15a340 tx=0x7fe8e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.757+0000 7fe8f3e00640 1 -- 172.21.15.114:0/3071186817 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8ec01c360 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.992+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8fc04f330 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.992+0000 7fe8f3e00640 1 -- 172.21.15.114:0/3071186817 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8ec059510 con 0x7fe8fc0d02a0 2024-03-06T07:28:34.994 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8dc06b4d0 msgr2=0x7fe8dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8dc06b4d0 0x7fe8dc06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc15a340 tx=0x7fe8e8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 msgr2=0x7fe8fc159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc159a40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00d800 tx=0x7fe8ec00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 shutdown_connections 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8fc15e360 0x7fe8fc0b6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8fc15df80 0x7fe8fc1f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8dc06b4d0 0x7fe8dc06d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 --2- 172.21.15.114:0/3071186817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8fc0d02a0 0x7fe8fc159a40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 >> 172.21.15.114:0/3071186817 conn(0x7fe8fc0b7320 msgr2=0x7fe8fc0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 shutdown_connections 2024-03-06T07:28:34.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:34.995+0000 7fe902e00640 1 -- 172.21.15.114:0/3071186817 wait complete. 2024-03-06T07:28:35.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:28:35.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.217+0000 7f7db7c00640 1 Processor -- start 2024-03-06T07:28:35.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.217+0000 7f7db7c00640 1 -- start start 2024-03-06T07:28:35.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7db7c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b2e60 0x7f7db00cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7db7c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00b3340 con 0x7f7db00c5970 2024-03-06T07:28:35.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7db7c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00cd570 con 0x7f7db015f730 2024-03-06T07:28:35.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7db7c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cd320 con 0x7f7db00b2e60 2024-03-06T07:28:35.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7db5c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7db00c5970 0x7f7db00c5e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49254/0 (socket says 172.21.15.114:49254) 2024-03-06T07:28:35.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7db5c00640 1 -- 172.21.15.114:0/307485287 learned_addr learned my addr 172.21.15.114:0/307485287 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:35.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7dafe00640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b2e60 0x7f7db00cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.219+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258821100 0 0) 0x7f7db00b3340 con 0x7f7db00c5970 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d98003520 con 0x7f7db00c5970 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1250829493 0 0) 0x7f7d98003520 con 0x7f7db00c5970 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b2e60 msgr2=0x7f7db00cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b2e60 0x7f7db00cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 >> v1:172.21.15.177:6789/0 conn(0x7f7db015f730 legacy=0x7f7db00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db00cd850 con 0x7f7db00c5970 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7da80019f0 con 0x7f7db00c5970 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 >> v1:172.21.15.114:6789/0 conn(0x7f7db00c5970 legacy=0x7f7db00c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b2e60 0x7f7d980039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7d98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d98004d40 0x7f7d980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cd320 con 0x7f7d98003f60 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9800baa0 con 0x7f7d98004d40 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9800bca0 con 0x7f7db00b2e60 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.220+0000 7f7dafe00640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7d98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.221+0000 7f7dafe00640 1 -- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b2e60 msgr2=0x7f7d980039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.221+0000 7f7dafe00640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b2e60 0x7f7d980039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.221+0000 7f7dafe00640 1 -- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d98004d40 msgr2=0x7f7d980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.221+0000 7f7dafe00640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d98004d40 0x7f7d980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.221+0000 7f7dafe00640 1 -- 172.21.15.114:0/307485287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98003520 con 0x7f7d98003f60 2024-03-06T07:28:35.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.222+0000 7f7dafe00640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7d98004800 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009ac0 tx=0x7f7d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cbdea6fde0398c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.222+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c039070 con 0x7f7d98003f60 2024-03-06T07:28:35.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.222+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c02c710 con 0x7f7d98003f60 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.222+0000 7f7daf400640 1 -- 172.21.15.114:0/307485287 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c009690 con 0x7f7d98003f60 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.223+0000 7f7db7c00640 1 -- 172.21.15.114:0/307485287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 msgr2=0x7f7d98004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.223+0000 7f7db7c00640 1 --2- 172.21.15.114:0/307485287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7d98004800 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009ac0 tx=0x7f7d98005830 comp rx=0 tx=0).stop 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.223+0000 7f7db5c00640 1 -- 172.21.15.114:0/307485287 reap_dead start 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.223+0000 7f7db7c00640 1 -- 172.21.15.114:0/307485287 shutdown_connections 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.223+0000 7f7db7c00640 1 -- 172.21.15.114:0/307485287 >> 172.21.15.114:0/307485287 conn(0x7f7db00b3800 msgr2=0x7f7db00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.223+0000 7f7db7c00640 1 -- 172.21.15.114:0/307485287 shutdown_connections 2024-03-06T07:28:35.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.223+0000 7f7db7c00640 1 -- 172.21.15.114:0/307485287 wait complete. 2024-03-06T07:28:35.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.225+0000 7f7db7c00640 1 Processor -- start 2024-03-06T07:28:35.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.226+0000 7f7db7c00640 1 -- start start 2024-03-06T07:28:35.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.226+0000 7f7db7c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7db0150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db7c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db00c7120 0x7f7db00c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db7c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b54c0 0x7f7db015dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db7c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db015fb10 con 0x7f7d98003f60 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db7c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00cdbc0 con 0x7f7db00c7120 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db7c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db015fc90 con 0x7f7db00b54c0 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7db0150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7db0150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55378/0 (socket says 172.21.15.114:55378) 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 -- 172.21.15.114:0/3657754283 learned_addr learned my addr 172.21.15.114:0/3657754283 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 -- 172.21.15.114:0/3657754283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b54c0 msgr2=0x7f7db015dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b54c0 0x7f7db015dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 -- 172.21.15.114:0/3657754283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db00c7120 msgr2=0x7f7db00c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db00c7120 0x7f7db00c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 -- 172.21.15.114:0/3657754283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db0150a80 con 0x7f7d98003f60 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.227+0000 7f7db5200640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7db0150540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c530 tx=0x7f7da400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.228+0000 7f7dacc00640 1 -- 172.21.15.114:0/3657754283 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da40052d0 con 0x7f7d98003f60 2024-03-06T07:28:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.228+0000 7f7dacc00640 1 -- 172.21.15.114:0/3657754283 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da4005470 con 0x7f7d98003f60 2024-03-06T07:28:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.228+0000 7f7dacc00640 1 -- 172.21.15.114:0/3657754283 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4012870 con 0x7f7d98003f60 2024-03-06T07:28:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.228+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db00c6340 con 0x7f7d98003f60 2024-03-06T07:28:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.228+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db00c68a0 con 0x7f7d98003f60 2024-03-06T07:28:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.229+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db004f330 con 0x7f7d98003f60 2024-03-06T07:28:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.230+0000 7f7dacc00640 1 -- 172.21.15.114:0/3657754283 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7da4012a10 con 0x7f7d98003f60 2024-03-06T07:28:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.231+0000 7f7dacc00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9006b500 0x7f7d9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.231+0000 7f7dacc00640 1 -- 172.21.15.114:0/3657754283 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7da4089f10 con 0x7f7d98003f60 2024-03-06T07:28:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.231+0000 7f7dafe00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9006b500 0x7f7d9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.232+0000 7f7dafe00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9006b500 0x7f7d9006d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c036000 tx=0x7f7d9c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.235+0000 7f7dacc00640 1 -- 172.21.15.114:0/3657754283 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7db00c6340 con 0x7f7d98003f60 2024-03-06T07:28:35.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:35.520+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f7db006fb30 con 0x7f7d98003f60 2024-03-06T07:28:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.279+0000 7f7dacc00640 1 -- 172.21.15.114:0/3657754283 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v57) v1 ==== 135+0+0 (secure 0 0 0) 0x7f7da4058cf0 con 0x7f7d98003f60 2024-03-06T07:28:36.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.283+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9006b500 msgr2=0x7f7d9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.283+0000 7f7db7c00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9006b500 0x7f7d9006d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c036000 tx=0x7f7d9c02e000 comp rx=0 tx=0).stop 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.283+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 msgr2=0x7f7db0150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.283+0000 7f7db7c00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7db0150540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c530 tx=0x7f7da400c9f0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 shutdown_connections 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db00b54c0 0x7f7db015dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db00c7120 0x7f7db00c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9006b500 0x7f7d9006d9c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 --2- 172.21.15.114:0/3657754283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d98003f60 0x7f7db0150540 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 >> 172.21.15.114:0/3657754283 conn(0x7f7db00b3800 msgr2=0x7f7db0152450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 shutdown_connections 2024-03-06T07:28:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.284+0000 7f7db7c00640 1 -- 172.21.15.114:0/3657754283 wait complete. 2024-03-06T07:28:36.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:36.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.505+0000 7fca1b800640 1 Processor -- start 2024-03-06T07:28:36.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.506+0000 7fca1b800640 1 -- start start 2024-03-06T07:28:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.507+0000 7fca1b800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca141618e0 0x7fca140ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.507+0000 7fca1b800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca140b2d80 0x7fca140cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.507+0000 7fca1b800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca140cf9b0 con 0x7fca140cf520 2024-03-06T07:28:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.507+0000 7fca1b800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca140d0d20 con 0x7fca140b2d80 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.507+0000 7fca1b800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca140b32f0 con 0x7fca141618e0 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca13e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca140b2d80 0x7fca140cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca13e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca140b2d80 0x7fca140cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55388/0 (socket says 172.21.15.114:55388) 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca13e00640 1 -- 172.21.15.114:0/2867258510 learned_addr learned my addr 172.21.15.114:0/2867258510 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca18e00640 1 --2- 172.21.15.114:0/2867258510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca141618e0 0x7fca140ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca13e00640 1 -- 172.21.15.114:0/2867258510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca141618e0 msgr2=0x7fca140ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca13e00640 1 --2- 172.21.15.114:0/2867258510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca141618e0 0x7fca140ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca13e00640 1 -- 172.21.15.114:0/2867258510 >> v1:172.21.15.114:6789/0 conn(0x7fca140cf520 legacy=0x7fca140d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.508+0000 7fca13e00640 1 -- 172.21.15.114:0/2867258510 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140d0f70 con 0x7fca140b2d80 2024-03-06T07:28:36.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.509+0000 7fca13e00640 1 --2- 172.21.15.114:0/2867258510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140cefe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fca000098a0 tx=0x7fca0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9a0cd9cd1cfeb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca13400640 1 -- 172.21.15.114:0/2867258510 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00035020 con 0x7fca140b2d80 2024-03-06T07:28:36.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca13400640 1 -- 172.21.15.114:0/2867258510 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca00031640 con 0x7fca140b2d80 2024-03-06T07:28:36.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca13400640 1 -- 172.21.15.114:0/2867258510 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0002d040 con 0x7fca140b2d80 2024-03-06T07:28:36.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca1b800640 1 -- 172.21.15.114:0/2867258510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 msgr2=0x7fca140cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca1b800640 1 --2- 172.21.15.114:0/2867258510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140cefe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fca000098a0 tx=0x7fca0002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca1b800640 1 -- 172.21.15.114:0/2867258510 shutdown_connections 2024-03-06T07:28:36.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca1b800640 1 --2- 172.21.15.114:0/2867258510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140cefe0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca1b800640 1 --2- 172.21.15.114:0/2867258510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca141618e0 0x7fca140ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.510+0000 7fca1b800640 1 -- 172.21.15.114:0/2867258510 >> 172.21.15.114:0/2867258510 conn(0x7fca140d3e90 msgr2=0x7fca140d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.511+0000 7fca1b800640 1 -- 172.21.15.114:0/2867258510 shutdown_connections 2024-03-06T07:28:36.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.511+0000 7fca1b800640 1 -- 172.21.15.114:0/2867258510 wait complete. 2024-03-06T07:28:36.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.513+0000 7fca1b800640 1 Processor -- start 2024-03-06T07:28:36.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.513+0000 7fca1b800640 1 -- start start 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca1b800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca1b800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca140cf520 0x7fca140cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca1b800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca141618e0 0x7fca140b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca1b800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca14161cc0 con 0x7fca140b2d80 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca1b800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca140d1530 con 0x7fca141618e0 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca1b800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca140b1c50 con 0x7fca140cf520 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca18e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca18e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55404/0 (socket says 172.21.15.114:55404) 2024-03-06T07:28:36.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.514+0000 7fca18e00640 1 -- 172.21.15.114:0/263344097 learned_addr learned my addr 172.21.15.114:0/263344097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:36.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca18e00640 1 -- 172.21.15.114:0/263344097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca140cf520 msgr2=0x7fca140cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca18e00640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca140cf520 0x7fca140cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca18e00640 1 -- 172.21.15.114:0/263344097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca141618e0 msgr2=0x7fca140b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca18e00640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca141618e0 0x7fca140b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca18e00640 1 -- 172.21.15.114:0/263344097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140cc4a0 con 0x7fca140b2d80 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca18e00640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140c79c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fca0800d5f0 tx=0x7fca0800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca10c00640 1 -- 172.21.15.114:0/263344097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca080125f0 con 0x7fca140b2d80 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca10c00640 1 -- 172.21.15.114:0/263344097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca08019070 con 0x7fca140b2d80 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.515+0000 7fca10c00640 1 -- 172.21.15.114:0/263344097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca080129d0 con 0x7fca140b2d80 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.516+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca140cc780 con 0x7fca140b2d80 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.516+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca140ccd40 con 0x7fca140b2d80 2024-03-06T07:28:36.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.517+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1404f330 con 0x7fca140b2d80 2024-03-06T07:28:36.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.519+0000 7fca10c00640 1 -- 172.21.15.114:0/263344097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca08012b70 con 0x7fca140b2d80 2024-03-06T07:28:36.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.520+0000 7fca10c00640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.521+0000 7fca10c00640 1 -- 172.21.15.114:0/263344097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fca0808a690 con 0x7fca140b2d80 2024-03-06T07:28:36.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.521+0000 7fca13e00640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.521+0000 7fca13e00640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9f406b4d0 0x7fc9f406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fca00036000 tx=0x7fca00030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.524+0000 7fca10c00640 1 -- 172.21.15.114:0/263344097 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca08059480 con 0x7fca140b2d80 2024-03-06T07:28:36.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.720+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fca140b2950 con 0x7fca140b2d80 2024-03-06T07:28:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.726+0000 7fca10c00640 1 -- 172.21.15.114:0/263344097 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca08058d80 con 0x7fca140b2d80 2024-03-06T07:28:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.729+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9f406b4d0 msgr2=0x7fc9f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.729+0000 7fca1b800640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9f406b4d0 0x7fc9f406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fca00036000 tx=0x7fca00030000 comp rx=0 tx=0).stop 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.729+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 msgr2=0x7fca140c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.729+0000 7fca1b800640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140c79c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fca0800d5f0 tx=0x7fca0800dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 shutdown_connections 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca141618e0 0x7fca140b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fca140cf520 0x7fca140cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9f406b4d0 0x7fc9f406d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 --2- 172.21.15.114:0/263344097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca140b2d80 0x7fca140c79c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 >> 172.21.15.114:0/263344097 conn(0x7fca140d3e90 msgr2=0x7fca14160760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 shutdown_connections 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:36.730+0000 7fca1b800640 1 -- 172.21.15.114:0/263344097 wait complete. 2024-03-06T07:28:36.750 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:36.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:36.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:36.972 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:36.972 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:36.972 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:37.029 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:37.029 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:37.076 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:37.076 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:37.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:37.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.415+0000 7f0f95600640 1 Processor -- start 2024-03-06T07:28:37.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.415+0000 7f0f95600640 1 -- start start 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f95600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f90144c00 0x7f0f9006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f95600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0f9015fe30 0x7f0f900ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f95600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9006ee70 con 0x7f0f9015f9d0 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f95600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9006cef0 con 0x7f0f9015fe30 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f95600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9006d1d0 con 0x7f0f90144c00 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f8f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0f9015fe30 0x7f0f900ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f8f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0f9015fe30 0x7f0f900ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55412/0 (socket says 172.21.15.114:55412) 2024-03-06T07:28:37.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.417+0000 7f0f8f400640 1 -- 172.21.15.114:0/2797134075 learned_addr learned my addr 172.21.15.114:0/2797134075 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:37.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.418+0000 7f0f8ea00640 1 --2- 172.21.15.114:0/2797134075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f90144c00 0x7f0f9006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.418+0000 7f0f8f400640 1 -- 172.21.15.114:0/2797134075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f90144c00 msgr2=0x7f0f9006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.418+0000 7f0f8f400640 1 --2- 172.21.15.114:0/2797134075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f90144c00 0x7f0f9006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.418+0000 7f0f8f400640 1 -- 172.21.15.114:0/2797134075 >> v1:172.21.15.177:6789/0 conn(0x7f0f9015f9d0 legacy=0x7f0f900ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.418+0000 7f0f8f400640 1 -- 172.21.15.114:0/2797134075 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9006d4b0 con 0x7f0f9015fe30 2024-03-06T07:28:37.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.418+0000 7f0f8ea00640 1 --2- 172.21.15.114:0/2797134075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f90144c00 0x7f0f9006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:37.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.419+0000 7f0f8f400640 1 --2- 172.21.15.114:0/2797134075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900ce7b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009e50 tx=0x7f0f8802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8551c868e333bc00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.420+0000 7f0f8e000640 1 -- 172.21.15.114:0/2797134075 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8802cb00 con 0x7f0f9015fe30 2024-03-06T07:28:37.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.420+0000 7f0f8e000640 1 -- 172.21.15.114:0/2797134075 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8802cca0 con 0x7f0f9015fe30 2024-03-06T07:28:37.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.420+0000 7f0f8e000640 1 -- 172.21.15.114:0/2797134075 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88034610 con 0x7f0f9015fe30 2024-03-06T07:28:37.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.420+0000 7f0f95600640 1 -- 172.21.15.114:0/2797134075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 msgr2=0x7f0f900ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.420+0000 7f0f95600640 1 --2- 172.21.15.114:0/2797134075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900ce7b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009e50 tx=0x7f0f8802bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:37.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.421+0000 7f0f95600640 1 -- 172.21.15.114:0/2797134075 shutdown_connections 2024-03-06T07:28:37.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.421+0000 7f0f95600640 1 --2- 172.21.15.114:0/2797134075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900ce7b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.421+0000 7f0f95600640 1 --2- 172.21.15.114:0/2797134075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f90144c00 0x7f0f9006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.421+0000 7f0f95600640 1 -- 172.21.15.114:0/2797134075 >> 172.21.15.114:0/2797134075 conn(0x7f0f900b1730 msgr2=0x7f0f900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.421+0000 7f0f95600640 1 -- 172.21.15.114:0/2797134075 shutdown_connections 2024-03-06T07:28:37.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.421+0000 7f0f95600640 1 -- 172.21.15.114:0/2797134075 wait complete. 2024-03-06T07:28:37.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.422+0000 7f0f95600640 1 Processor -- start 2024-03-06T07:28:37.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.423+0000 7f0f95600640 1 -- start start 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.423+0000 7f0f95600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f90144c00 0x7f0f900b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.423+0000 7f0f95600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f9015f9d0 0x7f0f9013c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.423+0000 7f0f95600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.423+0000 7f0f95600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f90160730 con 0x7f0f9015fe30 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.423+0000 7f0f95600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f90160280 con 0x7f0f90144c00 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.423+0000 7f0f95600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9006cb50 con 0x7f0f9015f9d0 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55422/0 (socket says 172.21.15.114:55422) 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 -- 172.21.15.114:0/1875603815 learned_addr learned my addr 172.21.15.114:0/1875603815 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f86000640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f9015f9d0 0x7f0f9013c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8ea00640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f90144c00 0x7f0f900b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 -- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f9015f9d0 msgr2=0x7f0f9013c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f9015f9d0 0x7f0f9013c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 -- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f90144c00 msgr2=0x7f0f900b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f90144c00 0x7f0f900b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8f400640 1 -- 172.21.15.114:0/1875603815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9013d120 con 0x7f0f9015fe30 2024-03-06T07:28:37.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.424+0000 7f0f8ea00640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f90144c00 0x7f0f900b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:37.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.425+0000 7f0f8f400640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900b3580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009e50 tx=0x7f0f88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.425+0000 7f0f87400640 1 -- 172.21.15.114:0/1875603815 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88043020 con 0x7f0f9015fe30 2024-03-06T07:28:37.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.425+0000 7f0f87400640 1 -- 172.21.15.114:0/1875603815 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8802ce50 con 0x7f0f9015fe30 2024-03-06T07:28:37.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.425+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9013d400 con 0x7f0f9015fe30 2024-03-06T07:28:37.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.425+0000 7f0f87400640 1 -- 172.21.15.114:0/1875603815 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88009510 con 0x7f0f9015fe30 2024-03-06T07:28:37.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.425+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9015b800 con 0x7f0f9015fe30 2024-03-06T07:28:37.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.426+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9004f330 con 0x7f0f9015fe30 2024-03-06T07:28:37.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.428+0000 7f0f87400640 1 -- 172.21.15.114:0/1875603815 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f88046020 con 0x7f0f9015fe30 2024-03-06T07:28:37.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.429+0000 7f0f87400640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.429+0000 7f0f87400640 1 -- 172.21.15.114:0/1875603815 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0f880ac8c0 con 0x7f0f9015fe30 2024-03-06T07:28:37.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.429+0000 7f0f8ea00640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.430+0000 7f0f8ea00640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0f80006fd0 tx=0x7f0f80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.434+0000 7f0f87400640 1 -- 172.21.15.114:0/1875603815 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f8804d030 con 0x7f0f9015fe30 2024-03-06T07:28:37.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.628+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f9006dd20 con 0x7f0f9015fe30 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.631+0000 7f0f87400640 1 -- 172.21.15.114:0/1875603815 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f0f8807b6b0 con 0x7f0f9015fe30 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":22,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:36.265733+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f6c06b500 msgr2=0x7f0f6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0f80006fd0 tx=0x7f0f80008040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 msgr2=0x7f0f900b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900b3580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009e50 tx=0x7f0f88004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 shutdown_connections 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0f6c06b500 0x7f0f6c06d9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f9015fe30 0x7f0f900b3580 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0f9015f9d0 0x7f0f9013c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 --2- 172.21.15.114:0/1875603815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f90144c00 0x7f0f900b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.633+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 >> 172.21.15.114:0/1875603815 conn(0x7f0f900b1730 msgr2=0x7f0f9006e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.634+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 shutdown_connections 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.634+0000 7f0f95600640 1 -- 172.21.15.114:0/1875603815 wait complete. 2024-03-06T07:28:37.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:37.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.861+0000 7f45b7e00640 1 Processor -- start 2024-03-06T07:28:37.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.861+0000 7f45b7e00640 1 -- start start 2024-03-06T07:28:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.862+0000 7f45b7e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f45b80d1ea0 0x7f45b80d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.862+0000 7f45b7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80c7970 0x7f45b80c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.862+0000 7f45b7e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b80d13e0 con 0x7f45b815e130 2024-03-06T07:28:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.862+0000 7f45b7e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b80d2840 con 0x7f45b80d1ea0 2024-03-06T07:28:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.862+0000 7f45b7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80cf120 con 0x7f45b80c7970 2024-03-06T07:28:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.862+0000 7f45b6a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f45b815e130 0x7f45b80cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52158/0 (socket says 172.21.15.114:52158) 2024-03-06T07:28:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.862+0000 7f45b6a00640 1 -- 172.21.15.114:0/661582260 learned_addr learned my addr 172.21.15.114:0/661582260 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:37.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b6000640 1 --2- 172.21.15.114:0/661582260 >> v2:172.21.15.177:3300/0 conn(0x7f45b80d1ea0 0x7f45b80d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b7400640 1 --2- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80c7970 0x7f45b80c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b5600640 1 -- 172.21.15.114:0/661582260 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871158675 0 0) 0x7f45b80d13e0 con 0x7f45b815e130 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b5600640 1 -- 172.21.15.114:0/661582260 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c003520 con 0x7f45b815e130 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b6000640 1 -- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80c7970 msgr2=0x7f45b80c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b6000640 1 --2- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80c7970 0x7f45b80c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b6000640 1 -- 172.21.15.114:0/661582260 >> v1:172.21.15.114:6789/0 conn(0x7f45b815e130 legacy=0x7f45b80cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b6000640 1 -- 172.21.15.114:0/661582260 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b80cead0 con 0x7f45b80d1ea0 2024-03-06T07:28:37.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.863+0000 7f45b7400640 1 --2- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80c7970 0x7f45b80c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.865+0000 7f45b6000640 1 --2- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b80d1ea0 0x7f45b80d2300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f45a00097a0 tx=0x7f45a002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6c493db8c7f6c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.865+0000 7f45b5600640 1 -- 172.21.15.114:0/661582260 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0035020 con 0x7f45b80d1ea0 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.865+0000 7f45b5600640 1 -- 172.21.15.114:0/661582260 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a0031810 con 0x7f45b80d1ea0 2024-03-06T07:28:37.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.865+0000 7f45b5600640 1 -- 172.21.15.114:0/661582260 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0031bc0 con 0x7f45b80d1ea0 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 -- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b80d1ea0 msgr2=0x7f45b80d2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 --2- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b80d1ea0 0x7f45b80d2300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f45a00097a0 tx=0x7f45a002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 -- 172.21.15.114:0/661582260 shutdown_connections 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 --2- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80c7970 0x7f45b80c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 --2- 172.21.15.114:0/661582260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b80d1ea0 0x7f45b80d2300 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 -- 172.21.15.114:0/661582260 >> 172.21.15.114:0/661582260 conn(0x7f45b809d070 msgr2=0x7f45b80d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 -- 172.21.15.114:0/661582260 shutdown_connections 2024-03-06T07:28:37.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.866+0000 7f45b7e00640 1 -- 172.21.15.114:0/661582260 wait complete. 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.868+0000 7f45b7e00640 1 Processor -- start 2024-03-06T07:28:37.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.868+0000 7f45b7e00640 1 -- start start 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b80c7970 0x7f45b815b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80d1ea0 0x7f45b81605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b815e130 0x7f45b80d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80cfcc0 con 0x7f45b80c7970 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80c8340 con 0x7f45b815e130 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80d16c0 con 0x7f45b80d1ea0 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b80c7970 0x7f45b815b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b80c7970 0x7f45b815b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52210/0 (socket says 172.21.15.114:52210) 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 -- 172.21.15.114:0/1987217694 learned_addr learned my addr 172.21.15.114:0/1987217694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6000640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80d1ea0 0x7f45b81605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b7400640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b815e130 0x7f45b80d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 -- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80d1ea0 msgr2=0x7f45b81605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80d1ea0 0x7f45b81605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 -- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b815e130 msgr2=0x7f45b80d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b815e130 0x7f45b80d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.869+0000 7f45b6a00640 1 -- 172.21.15.114:0/1987217694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b80cbe70 con 0x7f45b80c7970 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.870+0000 7f45b7400640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b815e130 0x7f45b80d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.870+0000 7f45b6a00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b80c7970 0x7f45b815b330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45a800cb40 tx=0x7f45a800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.870+0000 7f45aea00640 1 -- 172.21.15.114:0/1987217694 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8017070 con 0x7f45b80c7970 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.870+0000 7f45aea00640 1 -- 172.21.15.114:0/1987217694 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a800aca0 con 0x7f45b80c7970 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.870+0000 7f45aea00640 1 -- 172.21.15.114:0/1987217694 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8012620 con 0x7f45b80c7970 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.870+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b80cc150 con 0x7f45b80c7970 2024-03-06T07:28:37.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.870+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b80cc710 con 0x7f45b80c7970 2024-03-06T07:28:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.872+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b80ccaf0 con 0x7f45b80c7970 2024-03-06T07:28:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.872+0000 7f45aea00640 1 -- 172.21.15.114:0/1987217694 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45a801a070 con 0x7f45b80c7970 2024-03-06T07:28:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.873+0000 7f45aea00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459406b500 0x7f459406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.873+0000 7f45aea00640 1 -- 172.21.15.114:0/1987217694 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f45a808a0d0 con 0x7f45b80c7970 2024-03-06T07:28:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.874+0000 7f45b6000640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459406b500 0x7f459406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.874+0000 7f45b6000640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459406b500 0x7f459406d9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f45a0036000 tx=0x7f45a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:37.878+0000 7f45aea00640 1 -- 172.21.15.114:0/1987217694 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45b80ccaf0 con 0x7f45b80c7970 2024-03-06T07:28:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.138+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f45b804f330 con 0x7f45b80c7970 2024-03-06T07:28:38.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.139+0000 7f45aea00640 1 -- 172.21.15.114:0/1987217694 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45a8058eb0 con 0x7f45b80c7970 2024-03-06T07:28:38.141 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459406b500 msgr2=0x7f459406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459406b500 0x7f459406d9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f45a0036000 tx=0x7f45a0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b80c7970 msgr2=0x7f45b815b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b80c7970 0x7f45b815b330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45a800cb40 tx=0x7f45a800a060 comp rx=0 tx=0).stop 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 shutdown_connections 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45b815e130 0x7f45b80d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b80d1ea0 0x7f45b81605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459406b500 0x7f459406d9c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 --2- 172.21.15.114:0/1987217694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b80c7970 0x7f45b815b330 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.142+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 >> 172.21.15.114:0/1987217694 conn(0x7f45b809d070 msgr2=0x7f45b80c84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.143+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 shutdown_connections 2024-03-06T07:28:38.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.143+0000 7f45b7e00640 1 -- 172.21.15.114:0/1987217694 wait complete. 2024-03-06T07:28:38.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:38.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.363+0000 7fb559c00640 1 Processor -- start 2024-03-06T07:28:38.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.363+0000 7fb559c00640 1 -- start start 2024-03-06T07:28:38.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb559c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb554161b50 0x7fb5540d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb559c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5540cecd0 con 0x7fb5540d4a00 2024-03-06T07:28:38.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb559c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5540cfc80 con 0x7fb5540b5490 2024-03-06T07:28:38.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb559c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5540cef40 con 0x7fb554161b50 2024-03-06T07:28:38.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb552a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb554161b50 0x7fb5540d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb552a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb554161b50 0x7fb5540d05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52216/0 (socket says 172.21.15.114:52216) 2024-03-06T07:28:38.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb552000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb5540d4a00 0x7fb5540d0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52172/0 (socket says 172.21.15.114:52172) 2024-03-06T07:28:38.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.366+0000 7fb552a00640 1 -- 172.21.15.114:0/1591635178 learned_addr learned my addr 172.21.15.114:0/1591635178 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:38.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.367+0000 7fb552a00640 1 -- 172.21.15.114:0/1591635178 >> v1:172.21.15.177:6789/0 conn(0x7fb5540b5490 legacy=0x7fb5540c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.367+0000 7fb552a00640 1 -- 172.21.15.114:0/1591635178 >> v1:172.21.15.114:6789/0 conn(0x7fb5540d4a00 legacy=0x7fb5540d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.367+0000 7fb552a00640 1 -- 172.21.15.114:0/1591635178 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540c7b60 con 0x7fb554161b50 2024-03-06T07:28:38.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.367+0000 7fb551600640 1 -- 172.21.15.114:0/1591635178 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346905437 0 0) 0x7fb5540cecd0 con 0x7fb5540d4a00 2024-03-06T07:28:38.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.368+0000 7fb552a00640 1 --2- 172.21.15.114:0/1591635178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb5540d05a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb544009870 tx=0x7fb54402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e65cf15627e16e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.368+0000 7fb551600640 1 -- 172.21.15.114:0/1591635178 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544034070 con 0x7fb554161b50 2024-03-06T07:28:38.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.368+0000 7fb551600640 1 -- 172.21.15.114:0/1591635178 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb544030860 con 0x7fb554161b50 2024-03-06T07:28:38.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.368+0000 7fb551600640 1 -- 172.21.15.114:0/1591635178 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544030c30 con 0x7fb554161b50 2024-03-06T07:28:38.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.368+0000 7fb559c00640 1 -- 172.21.15.114:0/1591635178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 msgr2=0x7fb5540d05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.368+0000 7fb559c00640 1 --2- 172.21.15.114:0/1591635178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb5540d05a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb544009870 tx=0x7fb54402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.369+0000 7fb559c00640 1 -- 172.21.15.114:0/1591635178 shutdown_connections 2024-03-06T07:28:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.369+0000 7fb559c00640 1 --2- 172.21.15.114:0/1591635178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb5540d05a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.369+0000 7fb559c00640 1 -- 172.21.15.114:0/1591635178 >> 172.21.15.114:0/1591635178 conn(0x7fb5540d3e70 msgr2=0x7fb5540d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.369+0000 7fb559c00640 1 -- 172.21.15.114:0/1591635178 shutdown_connections 2024-03-06T07:28:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.369+0000 7fb559c00640 1 -- 172.21.15.114:0/1591635178 wait complete. 2024-03-06T07:28:38.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.370+0000 7fb559c00640 1 Processor -- start 2024-03-06T07:28:38.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.370+0000 7fb559c00640 1 -- start start 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.371+0000 7fb559c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5540b5490 0x7fb5540b7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.371+0000 7fb559c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5540d4a00 0x7fb5540b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.371+0000 7fb559c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb55414bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.371+0000 7fb559c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540c8910 con 0x7fb554161b50 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.371+0000 7fb559c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540c8790 con 0x7fb5540d4a00 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.371+0000 7fb559c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540c8a90 con 0x7fb5540b5490 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb55414bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb55414bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52228/0 (socket says 172.21.15.114:52228) 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 -- 172.21.15.114:0/4060132683 learned_addr learned my addr 172.21.15.114:0/4060132683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb552000640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5540d4a00 0x7fb5540b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb552a00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5540b5490 0x7fb5540b7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 -- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5540b5490 msgr2=0x7fb5540b7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5540b5490 0x7fb5540b7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 -- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5540d4a00 msgr2=0x7fb5540b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5540d4a00 0x7fb5540b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 -- 172.21.15.114:0/4060132683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540b88f0 con 0x7fb554161b50 2024-03-06T07:28:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.372+0000 7fb553400640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb55414bd90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c7b0 tx=0x7fb54c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.373+0000 7fb54aa00640 1 -- 172.21.15.114:0/4060132683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c004b60 con 0x7fb554161b50 2024-03-06T07:28:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.373+0000 7fb54aa00640 1 -- 172.21.15.114:0/4060132683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c004d00 con 0x7fb554161b50 2024-03-06T07:28:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.373+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5540b8610 con 0x7fb554161b50 2024-03-06T07:28:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.373+0000 7fb54aa00640 1 -- 172.21.15.114:0/4060132683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c00aa90 con 0x7fb554161b50 2024-03-06T07:28:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.373+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55414a150 con 0x7fb554161b50 2024-03-06T07:28:38.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.374+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55404f330 con 0x7fb554161b50 2024-03-06T07:28:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.376+0000 7fb54aa00640 1 -- 172.21.15.114:0/4060132683 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb54c00ac30 con 0x7fb554161b50 2024-03-06T07:28:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.377+0000 7fb54aa00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb53406b500 0x7fb53406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.377+0000 7fb54aa00640 1 -- 172.21.15.114:0/4060132683 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb54c08a4a0 con 0x7fb554161b50 2024-03-06T07:28:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.377+0000 7fb552a00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb53406b500 0x7fb53406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.378+0000 7fb552a00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb53406b500 0x7fb53406d9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb544036000 tx=0x7fb54403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.383+0000 7fb54aa00640 1 -- 172.21.15.114:0/4060132683 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb54c020070 con 0x7fb554161b50 2024-03-06T07:28:38.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.578+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5540d16d0 con 0x7fb554161b50 2024-03-06T07:28:38.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.579+0000 7fb54aa00640 1 -- 172.21.15.114:0/4060132683 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb5540b88f0 con 0x7fb554161b50 2024-03-06T07:28:38.581 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb53406b500 msgr2=0x7fb53406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb53406b500 0x7fb53406d9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb544036000 tx=0x7fb54403b040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 msgr2=0x7fb55414bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb55414bd90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00c7b0 tx=0x7fb54c00cc70 comp rx=0 tx=0).stop 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 shutdown_connections 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb53406b500 0x7fb53406d9c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb554161b50 0x7fb55414bd90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5540d4a00 0x7fb5540b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 --2- 172.21.15.114:0/4060132683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5540b5490 0x7fb5540b7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 >> 172.21.15.114:0/4060132683 conn(0x7fb5540d3e70 msgr2=0x7fb5540c8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 shutdown_connections 2024-03-06T07:28:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.582+0000 7fb559c00640 1 -- 172.21.15.114:0/4060132683 wait complete. 2024-03-06T07:28:38.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:38.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.811+0000 7f0e9dc00640 1 Processor -- start 2024-03-06T07:28:38.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.811+0000 7f0e9dc00640 1 -- start start 2024-03-06T07:28:38.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e9dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0e980d07f0 0x7f0e980d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e9dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0e9815e450 0x7f0e980b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e9dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e980d1bf0 0x7f0e980b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e9dc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980d4930 con 0x7f0e9815e450 2024-03-06T07:28:38.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e9dc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980b3730 con 0x7f0e980d07f0 2024-03-06T07:28:38.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e9dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98067b20 con 0x7f0e980d1bf0 2024-03-06T07:28:38.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e8e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0e9815e450 0x7f0e980b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e8e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0e9815e450 0x7f0e980b4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52234/0 (socket says 172.21.15.114:52234) 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e8e000640 1 -- 172.21.15.114:0/395109959 learned_addr learned my addr 172.21.15.114:0/395109959 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e96a00640 1 --2- 172.21.15.114:0/395109959 >> v2:172.21.15.177:3300/0 conn(0x7f0e980d07f0 0x7f0e980d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.813+0000 7f0e97400640 1 --2- 172.21.15.114:0/395109959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e980d1bf0 0x7f0e980b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.814+0000 7f0e8e000640 1 -- 172.21.15.114:0/395109959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e980d1bf0 msgr2=0x7f0e980b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.814+0000 7f0e8e000640 1 --2- 172.21.15.114:0/395109959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e980d1bf0 0x7f0e980b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.814+0000 7f0e8e000640 1 -- 172.21.15.114:0/395109959 >> v2:172.21.15.177:3300/0 conn(0x7f0e980d07f0 msgr2=0x7f0e980d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.814+0000 7f0e8e000640 1 --2- 172.21.15.114:0/395109959 >> v2:172.21.15.177:3300/0 conn(0x7f0e980d07f0 0x7f0e980d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.814+0000 7f0e8e000640 1 -- 172.21.15.114:0/395109959 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980d1110 con 0x7f0e9815e450 2024-03-06T07:28:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.814+0000 7f0e96a00640 1 --2- 172.21.15.114:0/395109959 >> v2:172.21.15.177:3300/0 conn(0x7f0e980d07f0 0x7f0e980d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:38.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.815+0000 7f0e8e000640 1 --2- 172.21.15.114:0/395109959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e9815e450 0x7f0e980b4480 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0e9808d7b0 tx=0x7f0e8002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedc611331647f3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.815+0000 7f0e96000640 1 -- 172.21.15.114:0/395109959 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8002e070 con 0x7f0e9815e450 2024-03-06T07:28:38.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.815+0000 7f0e96000640 1 -- 172.21.15.114:0/395109959 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e80037070 con 0x7f0e9815e450 2024-03-06T07:28:38.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.815+0000 7f0e96000640 1 -- 172.21.15.114:0/395109959 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80031a00 con 0x7f0e9815e450 2024-03-06T07:28:38.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 -- 172.21.15.114:0/395109959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e9815e450 msgr2=0x7f0e980b4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/395109959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e9815e450 0x7f0e980b4480 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0e9808d7b0 tx=0x7f0e8002b880 comp rx=0 tx=0).stop 2024-03-06T07:28:38.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 -- 172.21.15.114:0/395109959 shutdown_connections 2024-03-06T07:28:38.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/395109959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e980d1bf0 0x7f0e980b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/395109959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e9815e450 0x7f0e980b4480 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/395109959 >> v2:172.21.15.177:3300/0 conn(0x7f0e980d07f0 0x7f0e980d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 -- 172.21.15.114:0/395109959 >> 172.21.15.114:0/395109959 conn(0x7f0e980d3e10 msgr2=0x7f0e980d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 -- 172.21.15.114:0/395109959 shutdown_connections 2024-03-06T07:28:38.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.816+0000 7f0e9dc00640 1 -- 172.21.15.114:0/395109959 wait complete. 2024-03-06T07:28:38.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.818+0000 7f0e9dc00640 1 Processor -- start 2024-03-06T07:28:38.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.818+0000 7f0e9dc00640 1 -- start start 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e9dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e980d07f0 0x7f0e98151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e9dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e980d1bf0 0x7f0e981e67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e9dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e9815e450 0x7f0e981f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e9dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b3c80 con 0x7f0e980d07f0 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e9dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b39a0 con 0x7f0e980d1bf0 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e9dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b3f60 con 0x7f0e9815e450 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e96a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e980d07f0 0x7f0e98151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e96a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e980d07f0 0x7f0e98151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52248/0 (socket says 172.21.15.114:52248) 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.819+0000 7f0e96a00640 1 -- 172.21.15.114:0/2779245123 learned_addr learned my addr 172.21.15.114:0/2779245123 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e8e000640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e980d1bf0 0x7f0e981e67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e97400640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e9815e450 0x7f0e981f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e96a00640 1 -- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e9815e450 msgr2=0x7f0e981f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e96a00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e9815e450 0x7f0e981f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e96a00640 1 -- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e980d1bf0 msgr2=0x7f0e981e67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e96a00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e980d1bf0 0x7f0e981e67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e96a00640 1 -- 172.21.15.114:0/2779245123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980cc000 con 0x7f0e980d07f0 2024-03-06T07:28:38.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e97400640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e9815e450 0x7f0e981f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e96a00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e980d07f0 0x7f0e98151280 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800d800 tx=0x7f0e8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.820+0000 7f0e8f400640 1 -- 172.21.15.114:0/2779245123 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88018070 con 0x7f0e980d07f0 2024-03-06T07:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.821+0000 7f0e8f400640 1 -- 172.21.15.114:0/2779245123 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e88004030 con 0x7f0e980d07f0 2024-03-06T07:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.821+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e980cc2e0 con 0x7f0e980d07f0 2024-03-06T07:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.821+0000 7f0e8f400640 1 -- 172.21.15.114:0/2779245123 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88004400 con 0x7f0e980d07f0 2024-03-06T07:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.821+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e980cc8a0 con 0x7f0e980d07f0 2024-03-06T07:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.822+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9804f330 con 0x7f0e980d07f0 2024-03-06T07:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.823+0000 7f0e8f400640 1 -- 172.21.15.114:0/2779245123 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e880096a0 con 0x7f0e980d07f0 2024-03-06T07:28:38.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.824+0000 7f0e8f400640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e7406b4d0 0x7f0e7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.824+0000 7f0e8f400640 1 -- 172.21.15.114:0/2779245123 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0e8808a720 con 0x7f0e980d07f0 2024-03-06T07:28:38.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.824+0000 7f0e8e000640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e7406b4d0 0x7f0e7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.825+0000 7f0e8e000640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e7406b4d0 0x7f0e7406d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0e80036000 tx=0x7f0e80008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:38.829+0000 7f0e8f400640 1 -- 172.21.15.114:0/2779245123 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e880593e0 con 0x7f0e980d07f0 2024-03-06T07:28:39.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.024+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0e980d21d0 con 0x7f0e980d07f0 2024-03-06T07:28:39.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.025+0000 7f0e8f400640 1 -- 172.21.15.114:0/2779245123 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0e88058ce0 con 0x7f0e980d07f0 2024-03-06T07:28:39.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:39.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.027+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e7406b4d0 msgr2=0x7f0e7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.027+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e7406b4d0 0x7f0e7406d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0e80036000 tx=0x7f0e80008030 comp rx=0 tx=0).stop 2024-03-06T07:28:39.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.027+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e980d07f0 msgr2=0x7f0e98151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.027+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e980d07f0 0x7f0e98151280 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800d800 tx=0x7f0e8800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.027+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 shutdown_connections 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.027+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0e9815e450 0x7f0e981f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.027+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e980d1bf0 0x7f0e981e67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.028+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e7406b4d0 0x7f0e7406d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.028+0000 7f0e9dc00640 1 --2- 172.21.15.114:0/2779245123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e980d07f0 0x7f0e98151280 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.028+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 >> 172.21.15.114:0/2779245123 conn(0x7f0e980d3e10 msgr2=0x7f0e980b13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.028+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 shutdown_connections 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.028+0000 7f0e9dc00640 1 -- 172.21.15.114:0/2779245123 wait complete. 2024-03-06T07:28:39.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:39.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.267+0000 7f7564c00640 1 Processor -- start 2024-03-06T07:28:39.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.267+0000 7f7564c00640 1 -- start start 2024-03-06T07:28:39.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.268+0000 7f7564c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f75600cec10 0x7f7560161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f7564c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600cf2b0 con 0x7f7560162270 2024-03-06T07:28:39.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f7564c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600cf590 con 0x7f756015e360 2024-03-06T07:28:39.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f7564c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75601626c0 con 0x7f75600cec10 2024-03-06T07:28:39.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f755e800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7560162270 0x7f75600d1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52182/0 (socket says 172.21.15.114:52182) 2024-03-06T07:28:39.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f755e800640 1 -- 172.21.15.114:0/2618173175 learned_addr learned my addr 172.21.15.114:0/2618173175 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f755d400640 1 --2- 172.21.15.114:0/2618173175 >> v2:172.21.15.177:3300/0 conn(0x7f75600cec10 0x7f7560161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054560620 0 0) 0x7f75600cf2b0 con 0x7f7560162270 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.269+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7544003520 con 0x7f7560162270 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2564478373 0 0) 0x7f7544003520 con 0x7f7560162270 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 >> v2:172.21.15.177:3300/0 conn(0x7f75600cec10 msgr2=0x7f7560161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 --2- 172.21.15.114:0/2618173175 >> v2:172.21.15.177:3300/0 conn(0x7f75600cec10 0x7f7560161d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 >> v1:172.21.15.177:6789/0 conn(0x7f756015e360 legacy=0x7f75600d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75600cf870 con 0x7f7560162270 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f75580019f0 con 0x7f7560162270 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 >> v1:172.21.15.114:6789/0 conn(0x7f7560162270 legacy=0x7f75600d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75440039b0 0x7f7544003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f7544004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75440051c0 0x7f754400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75601626c0 con 0x7f7544004800 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754400c070 con 0x7f75440039b0 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754400c270 con 0x7f75440051c0 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755d400640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f7544004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.270+0000 7f755d400640 1 -- 172.21.15.114:0/2618173175 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75440051c0 msgr2=0x7f754400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.271+0000 7f755d400640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75440051c0 0x7f754400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.271+0000 7f755d400640 1 -- 172.21.15.114:0/2618173175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75440039b0 msgr2=0x7f7544003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.271+0000 7f755d400640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75440039b0 0x7f7544003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.271+0000 7f755d400640 1 -- 172.21.15.114:0/2618173175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7544003520 con 0x7f7544004800 2024-03-06T07:28:39.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.271+0000 7f755e800640 1 -- 172.21.15.114:0/2618173175 reap_dead start 2024-03-06T07:28:39.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.272+0000 7f755d400640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f7544004c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f754c009ac0 tx=0x7f7544005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74812aedc64ff28c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.272+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c039070 con 0x7f7544004800 2024-03-06T07:28:39.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.272+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c02c6b0 con 0x7f7544004800 2024-03-06T07:28:39.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.272+0000 7f755ca00640 1 -- 172.21.15.114:0/2618173175 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c009690 con 0x7f7544004800 2024-03-06T07:28:39.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.272+0000 7f7564c00640 1 -- 172.21.15.114:0/2618173175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 msgr2=0x7f7544004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.272+0000 7f7564c00640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f7544004c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f754c009ac0 tx=0x7f7544005830 comp rx=0 tx=0).stop 2024-03-06T07:28:39.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.273+0000 7f7564c00640 1 -- 172.21.15.114:0/2618173175 shutdown_connections 2024-03-06T07:28:39.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.273+0000 7f7564c00640 1 --2- 172.21.15.114:0/2618173175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f7544004c80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.273+0000 7f7564c00640 1 -- 172.21.15.114:0/2618173175 >> 172.21.15.114:0/2618173175 conn(0x7f75600d3f50 msgr2=0x7f75600d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.273+0000 7f7564c00640 1 -- 172.21.15.114:0/2618173175 shutdown_connections 2024-03-06T07:28:39.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.273+0000 7f7564c00640 1 -- 172.21.15.114:0/2618173175 wait complete. 2024-03-06T07:28:39.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.275+0000 7f7564c00640 1 Processor -- start 2024-03-06T07:28:39.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.276+0000 7f7564c00640 1 -- start start 2024-03-06T07:28:39.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f7564c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f75601594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f7564c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75601f78d0 0x7f75600cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f7564c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75600cb560 0x7f75600d2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f7564c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7560161d30 con 0x7f7544004800 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f7564c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600cfe30 con 0x7f75600cb560 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f7564c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7560161eb0 con 0x7f75601f78d0 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f755de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f75601594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f755de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f75601594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52274/0 (socket says 172.21.15.114:52274) 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.277+0000 7f755de00640 1 -- 172.21.15.114:0/1397922230 learned_addr learned my addr 172.21.15.114:0/1397922230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f755de00640 1 -- 172.21.15.114:0/1397922230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75601f78d0 msgr2=0x7f75600cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f755de00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75601f78d0 0x7f75600cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f755de00640 1 -- 172.21.15.114:0/1397922230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75600cb560 msgr2=0x7f75600d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f755de00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75600cb560 0x7f75600d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f755de00640 1 -- 172.21.15.114:0/1397922230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75601f8320 con 0x7f7544004800 2024-03-06T07:28:39.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f755de00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f75601594d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f755000c460 tx=0x7f755000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f7556000640 1 -- 172.21.15.114:0/1397922230 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550017070 con 0x7f7544004800 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f7556000640 1 -- 172.21.15.114:0/1397922230 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75500052f0 con 0x7f7544004800 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.278+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75601f8600 con 0x7f7544004800 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.279+0000 7f7556000640 1 -- 172.21.15.114:0/1397922230 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550003830 con 0x7f7544004800 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.279+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75600b2bc0 con 0x7f7544004800 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.280+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756004f330 con 0x7f7544004800 2024-03-06T07:28:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.281+0000 7f7556000640 1 -- 172.21.15.114:0/1397922230 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f755001a070 con 0x7f7544004800 2024-03-06T07:28:39.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.282+0000 7f7556000640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f754006b4d0 0x7f754006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.282+0000 7f7556000640 1 -- 172.21.15.114:0/1397922230 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7550089650 con 0x7f7544004800 2024-03-06T07:28:39.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.282+0000 7f755d400640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f754006b4d0 0x7f754006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.283+0000 7f755d400640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f754006b4d0 0x7f754006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f754c036000 tx=0x7f754c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.286+0000 7f7556000640 1 -- 172.21.15.114:0/1397922230 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7550058430 con 0x7f7544004800 2024-03-06T07:28:39.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.495+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f756008d9b0 con 0x7f7544004800 2024-03-06T07:28:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.496+0000 7f7556000640 1 -- 172.21.15.114:0/1397922230 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7550057d30 con 0x7f7544004800 2024-03-06T07:28:39.498 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f754006b4d0 msgr2=0x7f754006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f754006b4d0 0x7f754006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f754c036000 tx=0x7f754c02e000 comp rx=0 tx=0).stop 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 msgr2=0x7f75601594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f75601594d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f755000c460 tx=0x7f755000c920 comp rx=0 tx=0).stop 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 shutdown_connections 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75600cb560 0x7f75600d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75601f78d0 0x7f75600cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f754006b4d0 0x7f754006d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 --2- 172.21.15.114:0/1397922230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7544004800 0x7f75601594d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 >> 172.21.15.114:0/1397922230 conn(0x7f75600d3f50 msgr2=0x7f756015b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 shutdown_connections 2024-03-06T07:28:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:39.499+0000 7f7564c00640 1 -- 172.21.15.114:0/1397922230 wait complete. 2024-03-06T07:28:39.520 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:28:39.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:39.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-06T07:28:39.597 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:28:39.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:39.598 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:39.648 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:39.648 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:39.703 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:39.703 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:39.756 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:39.756 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:39.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:40.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.000+0000 7f1d68200640 1 Processor -- start 2024-03-06T07:28:40.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.000+0000 7f1d68200640 1 -- start start 2024-03-06T07:28:40.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.001+0000 7f1d68200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1d6015eff0 0x7f1d60070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d68200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1d6006e4d0 0x7f1d6006e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d68200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d60071d30 con 0x7f1d600af1c0 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d68200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6006ee80 con 0x7f1d6006e4d0 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d68200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d60071260 con 0x7f1d6015eff0 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d66200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1d6006e4d0 0x7f1d6006e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d65800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1d600af1c0 0x7f1d6006cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52198/0 (socket says 172.21.15.114:52198) 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d66200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1d6006e4d0 0x7f1d6006e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52278/0 (socket says 172.21.15.114:52278) 2024-03-06T07:28:40.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d65800640 1 -- 172.21.15.114:0/2063262520 learned_addr learned my addr 172.21.15.114:0/2063262520 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:40.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d64e00640 1 --2- 172.21.15.114:0/2063262520 >> v2:172.21.15.177:3300/0 conn(0x7f1d6015eff0 0x7f1d60070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.002+0000 7f1d66200640 1 -- 172.21.15.114:0/2063262520 >> v2:172.21.15.177:3300/0 conn(0x7f1d6015eff0 msgr2=0x7f1d60070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.003+0000 7f1d66200640 1 --2- 172.21.15.114:0/2063262520 >> v2:172.21.15.177:3300/0 conn(0x7f1d6015eff0 0x7f1d60070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.003+0000 7f1d66200640 1 -- 172.21.15.114:0/2063262520 >> v1:172.21.15.114:6789/0 conn(0x7f1d600af1c0 legacy=0x7f1d6006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.003+0000 7f1d5fe00640 1 -- 172.21.15.114:0/2063262520 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068945284 0 0) 0x7f1d54001110 con 0x7f1d600af1c0 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.003+0000 7f1d66200640 1 -- 172.21.15.114:0/2063262520 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6015f4a0 con 0x7f1d6006e4d0 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.003+0000 7f1d64e00640 1 --2- 172.21.15.114:0/2063262520 >> v2:172.21.15.177:3300/0 conn(0x7f1d6015eff0 0x7f1d60070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.004+0000 7f1d66200640 1 --2- 172.21.15.114:0/2063262520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6006e4d0 0x7f1d6006e940 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009d90 tx=0x7f1d5802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dca123c43c00443 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.004+0000 7f1d5fe00640 1 -- 172.21.15.114:0/2063262520 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58039070 con 0x7f1d6006e4d0 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.004+0000 7f1d5fe00640 1 -- 172.21.15.114:0/2063262520 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5802cc60 con 0x7f1d6006e4d0 2024-03-06T07:28:40.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.004+0000 7f1d5fe00640 1 -- 172.21.15.114:0/2063262520 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58034660 con 0x7f1d6006e4d0 2024-03-06T07:28:40.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 -- 172.21.15.114:0/2063262520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6006e4d0 msgr2=0x7f1d6006e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 --2- 172.21.15.114:0/2063262520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6006e4d0 0x7f1d6006e940 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009d90 tx=0x7f1d5802beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 -- 172.21.15.114:0/2063262520 shutdown_connections 2024-03-06T07:28:40.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 --2- 172.21.15.114:0/2063262520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6006e4d0 0x7f1d6006e940 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 --2- 172.21.15.114:0/2063262520 >> v2:172.21.15.177:3300/0 conn(0x7f1d6015eff0 0x7f1d60070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 -- 172.21.15.114:0/2063262520 >> 172.21.15.114:0/2063262520 conn(0x7f1d600b1730 msgr2=0x7f1d600b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 -- 172.21.15.114:0/2063262520 shutdown_connections 2024-03-06T07:28:40.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.005+0000 7f1d68200640 1 -- 172.21.15.114:0/2063262520 wait complete. 2024-03-06T07:28:40.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.006+0000 7f1d68200640 1 Processor -- start 2024-03-06T07:28:40.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.007+0000 7f1d68200640 1 -- start start 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.007+0000 7f1d68200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1d6006e4d0 0x7f1d601e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.007+0000 7f1d68200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d600af1c0 0x7f1d6014b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.007+0000 7f1d68200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6015eff0 0x7f1d600cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.007+0000 7f1d68200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d60071a50 con 0x7f1d6015eff0 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.007+0000 7f1d68200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d60071770 con 0x7f1d600af1c0 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.007+0000 7f1d68200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d600b30b0 con 0x7f1d6006e4d0 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6015eff0 0x7f1d600cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6015eff0 0x7f1d600cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52294/0 (socket says 172.21.15.114:52294) 2024-03-06T07:28:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 -- 172.21.15.114:0/1425723540 learned_addr learned my addr 172.21.15.114:0/1425723540 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d64e00640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d600af1c0 0x7f1d6014b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d65800640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1d6006e4d0 0x7f1d601e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 -- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1d6006e4d0 msgr2=0x7f1d601e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1d6006e4d0 0x7f1d601e75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 -- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d600af1c0 msgr2=0x7f1d6014b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d600af1c0 0x7f1d6014b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.008+0000 7f1d66200640 1 -- 172.21.15.114:0/1425723540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d600ce300 con 0x7f1d6015eff0 2024-03-06T07:28:40.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.009+0000 7f1d66200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6015eff0 0x7f1d600cda00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009d90 tx=0x7f1d58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.009+0000 7f1d5d600640 1 -- 172.21.15.114:0/1425723540 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58044020 con 0x7f1d6015eff0 2024-03-06T07:28:40.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.009+0000 7f1d5d600640 1 -- 172.21.15.114:0/1425723540 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5803a040 con 0x7f1d6015eff0 2024-03-06T07:28:40.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.009+0000 7f1d5d600640 1 -- 172.21.15.114:0/1425723540 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58039050 con 0x7f1d6015eff0 2024-03-06T07:28:40.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.009+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d600ce5e0 con 0x7f1d6015eff0 2024-03-06T07:28:40.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.009+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d600ceba0 con 0x7f1d6015eff0 2024-03-06T07:28:40.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.010+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6004f330 con 0x7f1d6015eff0 2024-03-06T07:28:40.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.011+0000 7f1d5d600640 1 -- 172.21.15.114:0/1425723540 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d58003bd0 con 0x7f1d6015eff0 2024-03-06T07:28:40.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.012+0000 7f1d5d600640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1d4006b4d0 0x7f1d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.012+0000 7f1d5d600640 1 -- 172.21.15.114:0/1425723540 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f1d580ac980 con 0x7f1d6015eff0 2024-03-06T07:28:40.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.012+0000 7f1d65800640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1d4006b4d0 0x7f1d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.013+0000 7f1d65800640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1d4006b4d0 0x7f1d4006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d54000f60 tx=0x7f1d54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.017+0000 7f1d5d600640 1 -- 172.21.15.114:0/1425723540 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d5804e060 con 0x7f1d6015eff0 2024-03-06T07:28:40.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.215+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d60160610 con 0x7f1d6015eff0 2024-03-06T07:28:40.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.217+0000 7f1d5d600640 1 -- 172.21.15.114:0/1425723540 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f1d6015f4a0 con 0x7f1d6015eff0 2024-03-06T07:28:40.219 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:40.219 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":22,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:36.265733+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1d4006b4d0 msgr2=0x7f1d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1d4006b4d0 0x7f1d4006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d54000f60 tx=0x7f1d54009040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6015eff0 msgr2=0x7f1d600cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6015eff0 0x7f1d600cda00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009d90 tx=0x7f1d58004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 shutdown_connections 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1d4006b4d0 0x7f1d4006d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d6015eff0 0x7f1d600cda00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d600af1c0 0x7f1d6014b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 --2- 172.21.15.114:0/1425723540 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1d6006e4d0 0x7f1d601e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 >> 172.21.15.114:0/1425723540 conn(0x7f1d600b1730 msgr2=0x7f1d60161050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 shutdown_connections 2024-03-06T07:28:40.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.220+0000 7f1d68200640 1 -- 172.21.15.114:0/1425723540 wait complete. 2024-03-06T07:28:40.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:40.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:40.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.448+0000 7f102b200640 1 Processor -- start 2024-03-06T07:28:40.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.449+0000 7f102b200640 1 -- start start 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f102b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1024066e00 0x7f10240af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f102b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f102406afc0 0x7f102406b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f102b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240b2d80 con 0x7f102415c2a0 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f102b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10240b0690 con 0x7f102406afc0 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f102b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240ad870 con 0x7f1024066e00 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f1029200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f102406afc0 0x7f102406b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f1023e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f102415c2a0 0x7f10240ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52204/0 (socket says 172.21.15.114:52204) 2024-03-06T07:28:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.450+0000 7f1023e00640 1 -- 172.21.15.114:0/1030465469 learned_addr learned my addr 172.21.15.114:0/1030465469 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.451+0000 7f101b400640 1 --2- 172.21.15.114:0/1030465469 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1024066e00 0x7f10240af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.451+0000 7f1029200640 1 -- 172.21.15.114:0/1030465469 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1024066e00 msgr2=0x7f10240af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.451+0000 7f1029200640 1 --2- 172.21.15.114:0/1030465469 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1024066e00 0x7f10240af9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.451+0000 7f1029200640 1 -- 172.21.15.114:0/1030465469 >> v1:172.21.15.114:6789/0 conn(0x7f102415c2a0 legacy=0x7f10240ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.451+0000 7f1023400640 1 -- 172.21.15.114:0/1030465469 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612153518 0 0) 0x7f10240b2d80 con 0x7f102415c2a0 2024-03-06T07:28:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.451+0000 7f1029200640 1 -- 172.21.15.114:0/1030465469 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240731b0 con 0x7f102406afc0 2024-03-06T07:28:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.451+0000 7f101b400640 1 --2- 172.21.15.114:0/1030465469 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1024066e00 0x7f10240af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.452+0000 7f1029200640 1 --2- 172.21.15.114:0/1030465469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f102406afc0 0x7f102406b450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f101c009dc0 tx=0x7f101c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497d4cc9dc84e25f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.452+0000 7f1023400640 1 -- 172.21.15.114:0/1030465469 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c039070 con 0x7f102406afc0 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.452+0000 7f1023400640 1 -- 172.21.15.114:0/1030465469 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c02cc40 con 0x7f102406afc0 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.452+0000 7f1023400640 1 -- 172.21.15.114:0/1030465469 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c034440 con 0x7f102406afc0 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 -- 172.21.15.114:0/1030465469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f102406afc0 msgr2=0x7f102406b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 --2- 172.21.15.114:0/1030465469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f102406afc0 0x7f102406b450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f101c009dc0 tx=0x7f101c02be90 comp rx=0 tx=0).stop 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 -- 172.21.15.114:0/1030465469 shutdown_connections 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 --2- 172.21.15.114:0/1030465469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f102406afc0 0x7f102406b450 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 --2- 172.21.15.114:0/1030465469 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1024066e00 0x7f10240af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 -- 172.21.15.114:0/1030465469 >> 172.21.15.114:0/1030465469 conn(0x7f10240b1e80 msgr2=0x7f10240b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 -- 172.21.15.114:0/1030465469 shutdown_connections 2024-03-06T07:28:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.453+0000 7f102b200640 1 -- 172.21.15.114:0/1030465469 wait complete. 2024-03-06T07:28:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.455+0000 7f102b200640 1 Processor -- start 2024-03-06T07:28:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.455+0000 7f102b200640 1 -- start start 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f102b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1024066e00 0x7f10241581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f102b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f102406afc0 0x7f1024158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f102b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f102415c2a0 0x7f10240c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f102b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102415c680 con 0x7f1024066e00 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f102b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240b0320 con 0x7f102415c2a0 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f102b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240b29a0 con 0x7f102406afc0 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1024066e00 0x7f10241581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1024066e00 0x7f10241581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52304/0 (socket says 172.21.15.114:52304) 2024-03-06T07:28:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 -- 172.21.15.114:0/4110612535 learned_addr learned my addr 172.21.15.114:0/4110612535 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 -- 172.21.15.114:0/4110612535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f102406afc0 msgr2=0x7f1024158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f102406afc0 0x7f1024158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 -- 172.21.15.114:0/4110612535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f102415c2a0 msgr2=0x7f10240c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f102415c2a0 0x7f10240c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.456+0000 7f1023e00640 1 -- 172.21.15.114:0/4110612535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102415dce0 con 0x7f1024066e00 2024-03-06T07:28:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.457+0000 7f1023e00640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1024066e00 0x7f10241581d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f101400c460 tx=0x7f101400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.457+0000 7f1020c00640 1 -- 172.21.15.114:0/4110612535 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1014018070 con 0x7f1024066e00 2024-03-06T07:28:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.457+0000 7f1020c00640 1 -- 172.21.15.114:0/4110612535 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1014007500 con 0x7f1024066e00 2024-03-06T07:28:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.457+0000 7f1020c00640 1 -- 172.21.15.114:0/4110612535 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1014004a20 con 0x7f1024066e00 2024-03-06T07:28:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.457+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102415dfa0 con 0x7f1024066e00 2024-03-06T07:28:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.457+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102415e560 con 0x7f1024066e00 2024-03-06T07:28:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.459+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102404f330 con 0x7f1024066e00 2024-03-06T07:28:40.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.459+0000 7f1020c00640 1 -- 172.21.15.114:0/4110612535 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1014004bc0 con 0x7f1024066e00 2024-03-06T07:28:40.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.460+0000 7f1020c00640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f100006b4d0 0x7f100006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.460+0000 7f1020c00640 1 -- 172.21.15.114:0/4110612535 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f10140894e0 con 0x7f1024066e00 2024-03-06T07:28:40.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.461+0000 7f101b400640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f100006b4d0 0x7f100006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.461+0000 7f101b400640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f100006b4d0 0x7f100006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f100c006fd0 tx=0x7f100c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.465+0000 7f1020c00640 1 -- 172.21.15.114:0/4110612535 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1014058360 con 0x7f1024066e00 2024-03-06T07:28:40.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.660+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10240ad370 con 0x7f1024066e00 2024-03-06T07:28:40.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.661+0000 7f1020c00640 1 -- 172.21.15.114:0/4110612535 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1014057c60 con 0x7f1024066e00 2024-03-06T07:28:40.662 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f100006b4d0 msgr2=0x7f100006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f100006b4d0 0x7f100006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f100c006fd0 tx=0x7f100c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1024066e00 msgr2=0x7f10241581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1024066e00 0x7f10241581d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f101400c460 tx=0x7f101400c920 comp rx=0 tx=0).stop 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 shutdown_connections 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f102415c2a0 0x7f10240c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f102406afc0 0x7f1024158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f100006b4d0 0x7f100006d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 --2- 172.21.15.114:0/4110612535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1024066e00 0x7f10241581d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.663+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 >> 172.21.15.114:0/4110612535 conn(0x7f10240b1e80 msgr2=0x7f10240b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.664+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 shutdown_connections 2024-03-06T07:28:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.664+0000 7f102b200640 1 -- 172.21.15.114:0/4110612535 wait complete. 2024-03-06T07:28:40.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:40.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.888+0000 7f8b49a00640 1 Processor -- start 2024-03-06T07:28:40.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.888+0000 7f8b49a00640 1 -- start start 2024-03-06T07:28:40.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b49a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8b4415fc10 0x7f8b44070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b49a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b440b3f70 0x7f8b440ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b49a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440b46c0 con 0x7f8b4415fff0 2024-03-06T07:28:40.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b49a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b440b49a0 con 0x7f8b440b3f70 2024-03-06T07:28:40.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b49a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b44072c90 con 0x7f8b4415fc10 2024-03-06T07:28:40.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b43400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b440b3f70 0x7f8b440ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b43400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b440b3f70 0x7f8b440ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52306/0 (socket says 172.21.15.114:52306) 2024-03-06T07:28:40.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.890+0000 7f8b43400640 1 -- 172.21.15.114:0/2444224872 learned_addr learned my addr 172.21.15.114:0/2444224872 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:40.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.891+0000 7f8b42a00640 1 --2- 172.21.15.114:0/2444224872 >> v2:172.21.15.177:3300/0 conn(0x7f8b4415fc10 0x7f8b44070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.891+0000 7f8b43400640 1 -- 172.21.15.114:0/2444224872 >> v2:172.21.15.177:3300/0 conn(0x7f8b4415fc10 msgr2=0x7f8b44070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.891+0000 7f8b43400640 1 --2- 172.21.15.114:0/2444224872 >> v2:172.21.15.177:3300/0 conn(0x7f8b4415fc10 0x7f8b44070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.891+0000 7f8b43400640 1 -- 172.21.15.114:0/2444224872 >> v1:172.21.15.177:6789/0 conn(0x7f8b4415fff0 legacy=0x7f8b44160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.891+0000 7f8b43400640 1 -- 172.21.15.114:0/2444224872 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4409dd80 con 0x7f8b440b3f70 2024-03-06T07:28:40.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.892+0000 7f8b43400640 1 --2- 172.21.15.114:0/2444224872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b440b3f70 0x7f8b440ce4a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8b440cf770 tx=0x7f8b3c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a224e7b1c4daff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b41600640 1 -- 172.21.15.114:0/2444224872 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c02ca80 con 0x7f8b440b3f70 2024-03-06T07:28:40.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b41600640 1 -- 172.21.15.114:0/2444224872 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c02cc20 con 0x7f8b440b3f70 2024-03-06T07:28:40.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b41600640 1 -- 172.21.15.114:0/2444224872 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c0343f0 con 0x7f8b440b3f70 2024-03-06T07:28:40.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 -- 172.21.15.114:0/2444224872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b440b3f70 msgr2=0x7f8b440ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 --2- 172.21.15.114:0/2444224872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b440b3f70 0x7f8b440ce4a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8b440cf770 tx=0x7f8b3c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 -- 172.21.15.114:0/2444224872 shutdown_connections 2024-03-06T07:28:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 --2- 172.21.15.114:0/2444224872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b440b3f70 0x7f8b440ce4a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 --2- 172.21.15.114:0/2444224872 >> v2:172.21.15.177:3300/0 conn(0x7f8b4415fc10 0x7f8b44070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 -- 172.21.15.114:0/2444224872 >> 172.21.15.114:0/2444224872 conn(0x7f8b440b1940 msgr2=0x7f8b440b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 -- 172.21.15.114:0/2444224872 shutdown_connections 2024-03-06T07:28:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.893+0000 7f8b49a00640 1 -- 172.21.15.114:0/2444224872 wait complete. 2024-03-06T07:28:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.895+0000 7f8b49a00640 1 Processor -- start 2024-03-06T07:28:40.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.895+0000 7f8b49a00640 1 -- start start 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b49a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b440b3f70 0x7f8b441598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b49a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b4415fc10 0x7f8b4414fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b49a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b4415fff0 0x7f8b4414ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b49a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4415c460 con 0x7f8b4415fc10 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b49a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4415c2e0 con 0x7f8b4415fff0 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b49a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b440ce9e0 con 0x7f8b440b3f70 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b42000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b4415fc10 0x7f8b4414fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b42000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b4415fc10 0x7f8b4414fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52318/0 (socket says 172.21.15.114:52318) 2024-03-06T07:28:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.896+0000 7f8b42000640 1 -- 172.21.15.114:0/1268291006 learned_addr learned my addr 172.21.15.114:0/1268291006 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:40.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b42000640 1 -- 172.21.15.114:0/1268291006 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b440b3f70 msgr2=0x7f8b441598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b42000640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b440b3f70 0x7f8b441598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b42000640 1 -- 172.21.15.114:0/1268291006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b4415fff0 msgr2=0x7f8b4414ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b43400640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b4415fff0 0x7f8b4414ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b42000640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b4415fff0 0x7f8b4414ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b42000640 1 -- 172.21.15.114:0/1268291006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4415caa0 con 0x7f8b4415fc10 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b42000640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b4415fc10 0x7f8b4414fa00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000b410 tx=0x7f8b3000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.897+0000 7f8b3aa00640 1 -- 172.21.15.114:0/1268291006 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30015020 con 0x7f8b4415fc10 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.898+0000 7f8b3aa00640 1 -- 172.21.15.114:0/1268291006 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30011830 con 0x7f8b4415fc10 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.898+0000 7f8b3aa00640 1 -- 172.21.15.114:0/1268291006 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30011c20 con 0x7f8b4415fc10 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.898+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4415cd80 con 0x7f8b4415fc10 2024-03-06T07:28:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.898+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4414e790 con 0x7f8b4415fc10 2024-03-06T07:28:40.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.899+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4404f330 con 0x7f8b4415fc10 2024-03-06T07:28:40.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.900+0000 7f8b3aa00640 1 -- 172.21.15.114:0/1268291006 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b30018020 con 0x7f8b4415fc10 2024-03-06T07:28:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.901+0000 7f8b3aa00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b2406b4d0 0x7f8b2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.901+0000 7f8b3aa00640 1 -- 172.21.15.114:0/1268291006 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8b3008a2a0 con 0x7f8b4415fc10 2024-03-06T07:28:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.901+0000 7f8b42a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b2406b4d0 0x7f8b2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.902+0000 7f8b42a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b2406b4d0 0x7f8b2406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8b34006fd0 tx=0x7f8b34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:40.913+0000 7f8b3aa00640 1 -- 172.21.15.114:0/1268291006 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b3001f020 con 0x7f8b4415fc10 2024-03-06T07:28:41.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.108+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8b4414e960 con 0x7f8b4415fc10 2024-03-06T07:28:41.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.109+0000 7f8b3aa00640 1 -- 172.21.15.114:0/1268291006 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8b300172c0 con 0x7f8b4415fc10 2024-03-06T07:28:41.111 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:41.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.111+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b2406b4d0 msgr2=0x7f8b2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.111+0000 7f8b49a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b2406b4d0 0x7f8b2406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8b34006fd0 tx=0x7f8b34008040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.111+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b4415fc10 msgr2=0x7f8b4414fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.111+0000 7f8b49a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b4415fc10 0x7f8b4414fa00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000b410 tx=0x7f8b3000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 shutdown_connections 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b4415fff0 0x7f8b4414ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b2406b4d0 0x7f8b2406d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b4415fc10 0x7f8b4414fa00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 --2- 172.21.15.114:0/1268291006 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b440b3f70 0x7f8b441598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 >> 172.21.15.114:0/1268291006 conn(0x7f8b440b1940 msgr2=0x7f8b440ace20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 shutdown_connections 2024-03-06T07:28:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.112+0000 7f8b49a00640 1 -- 172.21.15.114:0/1268291006 wait complete. 2024-03-06T07:28:41.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.340+0000 7fdd0dc00640 1 Processor -- start 2024-03-06T07:28:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.340+0000 7fdd0dc00640 1 -- start start 2024-03-06T07:28:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.342+0000 7fdd0dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdd081618e0 0x7fdd080ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.342+0000 7fdd0dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.342+0000 7fdd0dc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd080b32f0 con 0x7fdd080b2d80 2024-03-06T07:28:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.342+0000 7fdd0dc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd080d0d20 con 0x7fdd081618e0 2024-03-06T07:28:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.342+0000 7fdd0dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080d0f70 con 0x7fdd080cf6f0 2024-03-06T07:28:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd07400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd07400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:55562/0 (socket says 172.21.15.114:55562) 2024-03-06T07:28:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd07400640 1 -- 172.21.15.114:0/380808930 learned_addr learned my addr 172.21.15.114:0/380808930 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd06a00640 1 --2- 172.21.15.114:0/380808930 >> v2:172.21.15.177:3300/0 conn(0x7fdd081618e0 0x7fdd080ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd06a00640 1 -- 172.21.15.114:0/380808930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 msgr2=0x7fdd080b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd06a00640 1 --2- 172.21.15.114:0/380808930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd06a00640 1 -- 172.21.15.114:0/380808930 >> v1:172.21.15.177:6789/0 conn(0x7fdd080b2d80 legacy=0x7fdd080cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.343+0000 7fdd06a00640 1 -- 172.21.15.114:0/380808930 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd080d1250 con 0x7fdd081618e0 2024-03-06T07:28:41.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.345+0000 7fdd06a00640 1 --2- 172.21.15.114:0/380808930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 0x7fdd080ceaa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80097a0 tx=0x7fdcf802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54b0dab5fb0bb814 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.345+0000 7fdd05600640 1 -- 172.21.15.114:0/380808930 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8035020 con 0x7fdd081618e0 2024-03-06T07:28:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.345+0000 7fdd05600640 1 -- 172.21.15.114:0/380808930 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8031810 con 0x7fdd081618e0 2024-03-06T07:28:41.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.345+0000 7fdd05600640 1 -- 172.21.15.114:0/380808930 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8031be0 con 0x7fdd081618e0 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 -- 172.21.15.114:0/380808930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 msgr2=0x7fdd080ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/380808930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 0x7fdd080ceaa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80097a0 tx=0x7fdcf802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 -- 172.21.15.114:0/380808930 shutdown_connections 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/380808930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/380808930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 0x7fdd080ceaa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 -- 172.21.15.114:0/380808930 >> 172.21.15.114:0/380808930 conn(0x7fdd080d3e90 msgr2=0x7fdd080d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 -- 172.21.15.114:0/380808930 shutdown_connections 2024-03-06T07:28:41.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.346+0000 7fdd0dc00640 1 -- 172.21.15.114:0/380808930 wait complete. 2024-03-06T07:28:41.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.348+0000 7fdd0dc00640 1 Processor -- start 2024-03-06T07:28:41.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.348+0000 7fdd0dc00640 1 -- start start 2024-03-06T07:28:41.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd0dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd080b2d80 0x7fdd08157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd0dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd0dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 0x7fdd0806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd0dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080d2da0 con 0x7fdd080b2d80 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd0dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd08161cc0 con 0x7fdd081618e0 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd0dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080b1180 con 0x7fdd080cf6f0 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd080b2d80 0x7fdd08157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd080b2d80 0x7fdd08157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52334/0 (socket says 172.21.15.114:52334) 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.349+0000 7fdd06a00640 1 -- 172.21.15.114:0/959813426 learned_addr learned my addr 172.21.15.114:0/959813426 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd07400640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 0x7fdd0806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd06000640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd06a00640 1 -- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 msgr2=0x7fdd080b8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd06a00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd06a00640 1 -- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 msgr2=0x7fdd0806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd06a00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 0x7fdd0806cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd06a00640 1 -- 172.21.15.114:0/959813426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08158900 con 0x7fdd080b2d80 2024-03-06T07:28:41.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd06a00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd080b2d80 0x7fdd08157a80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdcf802b8f0 tx=0x7fdcf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdcfea00640 1 -- 172.21.15.114:0/959813426 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8040070 con 0x7fdd080b2d80 2024-03-06T07:28:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdcfea00640 1 -- 172.21.15.114:0/959813426 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8039cf0 con 0x7fdd080b2d80 2024-03-06T07:28:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdcfea00640 1 -- 172.21.15.114:0/959813426 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8035040 con 0x7fdd080b2d80 2024-03-06T07:28:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.350+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd080b4bf0 con 0x7fdd080b2d80 2024-03-06T07:28:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.351+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd081f7ab0 con 0x7fdd080b2d80 2024-03-06T07:28:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.352+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0804f330 con 0x7fdd080b2d80 2024-03-06T07:28:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.352+0000 7fdcfea00640 1 -- 172.21.15.114:0/959813426 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcf804b060 con 0x7fdd080b2d80 2024-03-06T07:28:41.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.353+0000 7fdcfea00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdce806b4d0 0x7fdce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.353+0000 7fdcfea00640 1 -- 172.21.15.114:0/959813426 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fdcf80abe60 con 0x7fdd080b2d80 2024-03-06T07:28:41.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.354+0000 7fdd06000640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdce806b4d0 0x7fdce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.354+0000 7fdd06000640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdce806b4d0 0x7fdce806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd080c7400 tx=0x7fdcf4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.358+0000 7fdcfea00640 1 -- 172.21.15.114:0/959813426 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcf807abb0 con 0x7fdd080b2d80 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.552+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd08158b70 con 0x7fdd080b2d80 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.553+0000 7fdcfea00640 1 -- 172.21.15.114:0/959813426 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdcf807a4b0 con 0x7fdd080b2d80 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdce806b4d0 msgr2=0x7fdce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdce806b4d0 0x7fdce806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd080c7400 tx=0x7fdcf4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd080b2d80 msgr2=0x7fdd08157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd080b2d80 0x7fdd08157a80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdcf802b8f0 tx=0x7fdcf80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 shutdown_connections 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd081618e0 0x7fdd0806cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdd080cf6f0 0x7fdd080b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdce806b4d0 0x7fdce806d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 --2- 172.21.15.114:0/959813426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd080b2d80 0x7fdd08157a80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 >> 172.21.15.114:0/959813426 conn(0x7fdd080d3e90 msgr2=0x7fdd0815ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 shutdown_connections 2024-03-06T07:28:41.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.555+0000 7fdd0dc00640 1 -- 172.21.15.114:0/959813426 wait complete. 2024-03-06T07:28:41.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:41.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.769+0000 7fc22d000640 1 Processor -- start 2024-03-06T07:28:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.769+0000 7fc22d000640 1 -- start start 2024-03-06T07:28:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.770+0000 7fc22d000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc22815bfc0 0x7fc2280b1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.770+0000 7fc22d000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc2280b1bb0 0x7fc2280d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.770+0000 7fc22d000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22815c3a0 con 0x7fc2280b4390 2024-03-06T07:28:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.770+0000 7fc22d000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2280cdc20 con 0x7fc2280b1bb0 2024-03-06T07:28:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.770+0000 7fc22d000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280cd660 con 0x7fc22815bfc0 2024-03-06T07:28:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.771+0000 7fc226800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc2280b4390 0x7fc2280d0780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52208/0 (socket says 172.21.15.114:52208) 2024-03-06T07:28:41.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.771+0000 7fc226800640 1 -- 172.21.15.114:0/1125981702 learned_addr learned my addr 172.21.15.114:0/1125981702 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.771+0000 7fc225400640 1 --2- 172.21.15.114:0/1125981702 >> v2:172.21.15.177:3300/0 conn(0x7fc2280b1bb0 0x7fc2280d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.771+0000 7fc225e00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc22815bfc0 0x7fc2280b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.771+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593944439 0 0) 0x7fc22815c3a0 con 0x7fc2280b4390 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.771+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc20c003520 con 0x7fc2280b4390 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3012494100 0 0) 0x7fc20c003520 con 0x7fc2280b4390 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc22815bfc0 msgr2=0x7fc2280b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc22815bfc0 0x7fc2280b1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 >> v2:172.21.15.177:3300/0 conn(0x7fc2280b1bb0 msgr2=0x7fc2280d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 --2- 172.21.15.114:0/1125981702 >> v2:172.21.15.177:3300/0 conn(0x7fc2280b1bb0 0x7fc2280d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2280cd940 con 0x7fc2280b4390 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fc2200019f0 con 0x7fc2280b4390 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 >> v1:172.21.15.114:6789/0 conn(0x7fc2280b4390 legacy=0x7fc2280d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc20c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc22815bfc0 0x7fc20c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc20c004d40 0x7fc20c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280cd660 con 0x7fc20c0039b0 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c00b9c0 con 0x7fc20c004d40 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c00bb40 con 0x7fc22815bfc0 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.772+0000 7fc225e00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc20c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.773+0000 7fc225e00640 1 -- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc22815bfc0 msgr2=0x7fc20c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.773+0000 7fc225e00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc22815bfc0 0x7fc20c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.773+0000 7fc225e00640 1 -- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc20c004d40 msgr2=0x7fc20c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.773+0000 7fc225e00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc20c004d40 0x7fc20c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.773+0000 7fc225e00640 1 -- 172.21.15.114:0/1125981702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c003520 con 0x7fc20c0039b0 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc225e00640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc20c003dd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc218009ac0 tx=0x7fc20c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce72406b2400c42e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc218039070 con 0x7fc20c0039b0 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21802c710 con 0x7fc20c0039b0 2024-03-06T07:28:41.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc224a00640 1 -- 172.21.15.114:0/1125981702 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc218009690 con 0x7fc20c0039b0 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc22d000640 1 -- 172.21.15.114:0/1125981702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 msgr2=0x7fc20c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc22d000640 1 --2- 172.21.15.114:0/1125981702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc20c003dd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc218009ac0 tx=0x7fc20c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc226800640 1 -- 172.21.15.114:0/1125981702 reap_dead start 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc22d000640 1 -- 172.21.15.114:0/1125981702 shutdown_connections 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.774+0000 7fc22d000640 1 -- 172.21.15.114:0/1125981702 >> 172.21.15.114:0/1125981702 conn(0x7fc2280b3800 msgr2=0x7fc2280b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.775+0000 7fc22d000640 1 -- 172.21.15.114:0/1125981702 shutdown_connections 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.775+0000 7fc22d000640 1 -- 172.21.15.114:0/1125981702 wait complete. 2024-03-06T07:28:41.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.776+0000 7fc22d000640 1 Processor -- start 2024-03-06T07:28:41.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.776+0000 7fc22d000640 1 -- start start 2024-03-06T07:28:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.777+0000 7fc22d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc22815d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.777+0000 7fc22d000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc228159b70 0x7fc2280caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.777+0000 7fc22d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280cb030 0x7fc2281f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.777+0000 7fc22d000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280cd2f0 con 0x7fc20c0039b0 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.777+0000 7fc22d000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280cfd20 con 0x7fc2280cb030 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.777+0000 7fc22d000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2280b1670 con 0x7fc228159b70 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc22815d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc22815d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52352/0 (socket says 172.21.15.114:52352) 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 -- 172.21.15.114:0/2511200100 learned_addr learned my addr 172.21.15.114:0/2511200100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 -- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc228159b70 msgr2=0x7fc2280caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc226800640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280cb030 0x7fc2281f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc228159b70 0x7fc2280caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 -- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280cb030 msgr2=0x7fc2281f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280cb030 0x7fc2281f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 -- 172.21.15.114:0/2511200100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22815c6f0 con 0x7fc20c0039b0 2024-03-06T07:28:41.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc226800640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280cb030 0x7fc2281f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.778+0000 7fc225e00640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc22815d4b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc218000c00 tx=0x7fc218004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.779+0000 7fc21e000640 1 -- 172.21.15.114:0/2511200100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc218042020 con 0x7fc20c0039b0 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.779+0000 7fc21e000640 1 -- 172.21.15.114:0/2511200100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21803a040 con 0x7fc20c0039b0 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.779+0000 7fc21e000640 1 -- 172.21.15.114:0/2511200100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc218039070 con 0x7fc20c0039b0 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.779+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22815c9d0 con 0x7fc20c0039b0 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.779+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc228159fd0 con 0x7fc20c0039b0 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.780+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22804f330 con 0x7fc20c0039b0 2024-03-06T07:28:41.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.781+0000 7fc21e000640 1 -- 172.21.15.114:0/2511200100 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc218045020 con 0x7fc20c0039b0 2024-03-06T07:28:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.781+0000 7fc21e000640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc20806b4d0 0x7fc20806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.782+0000 7fc21e000640 1 -- 172.21.15.114:0/2511200100 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc2180ab740 con 0x7fc20c0039b0 2024-03-06T07:28:41.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.782+0000 7fc225400640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc20806b4d0 0x7fc20806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.782+0000 7fc225400640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc20806b4d0 0x7fc20806d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc2140093d0 tx=0x7fc214009300 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.786+0000 7fc21e000640 1 -- 172.21.15.114:0/2511200100 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc21804c070 con 0x7fc20c0039b0 2024-03-06T07:28:41.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.992+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc2280cb410 con 0x7fc20c0039b0 2024-03-06T07:28:41.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.993+0000 7fc21e000640 1 -- 172.21.15.114:0/2511200100 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc21807a520 con 0x7fc20c0039b0 2024-03-06T07:28:41.995 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc20806b4d0 msgr2=0x7fc20806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc20806b4d0 0x7fc20806d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc2140093d0 tx=0x7fc214009300 comp rx=0 tx=0).stop 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 msgr2=0x7fc22815d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc22815d4b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc218000c00 tx=0x7fc218004730 comp rx=0 tx=0).stop 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 shutdown_connections 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280cb030 0x7fc2281f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc228159b70 0x7fc2280caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc20806b4d0 0x7fc20806d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 --2- 172.21.15.114:0/2511200100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc20c0039b0 0x7fc22815d4b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 >> 172.21.15.114:0/2511200100 conn(0x7fc2280b3800 msgr2=0x7fc22806edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 shutdown_connections 2024-03-06T07:28:41.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:41.996+0000 7fc22d000640 1 -- 172.21.15.114:0/2511200100 wait complete. 2024-03-06T07:28:42.019 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-06T07:28:42.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.019 DEBUG:teuthology.orchestra.run.smithi177:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-06T07:28:42.126 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:28:42.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:42.127 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:42.193 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:42.193 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:42.247 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:42.247 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:42.295 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:28:42.295 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:42.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:42.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.618+0000 7f95d8a00640 1 Processor -- start 2024-03-06T07:28:42.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.619+0000 7f95d8a00640 1 -- start start 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.620+0000 7f95d8a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d4161580 0x7f95d40d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.620+0000 7f95d8a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f95d40d1e90 0x7f95d40d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.620+0000 7f95d8a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d4161960 con 0x7f95d40d28a0 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.620+0000 7f95d8a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d40cf6f0 con 0x7f95d40d1e90 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.620+0000 7f95d8a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d40cf9d0 con 0x7f95d4161580 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.620+0000 7f95d2a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f95d40d1e90 0x7f95d40d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.620+0000 7f95d2a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f95d40d1e90 0x7f95d40d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52354/0 (socket says 172.21.15.114:52354) 2024-03-06T07:28:42.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.621+0000 7f95d2a00640 1 -- 172.21.15.114:0/1834749538 learned_addr learned my addr 172.21.15.114:0/1834749538 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:42.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.621+0000 7f95d3400640 1 --2- 172.21.15.114:0/1834749538 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d4161580 0x7f95d40d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.621+0000 7f95d2a00640 1 -- 172.21.15.114:0/1834749538 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d4161580 msgr2=0x7f95d40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.621+0000 7f95d2a00640 1 --2- 172.21.15.114:0/1834749538 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d4161580 0x7f95d40d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.621+0000 7f95d2a00640 1 -- 172.21.15.114:0/1834749538 >> v1:172.21.15.177:6789/0 conn(0x7f95d40d28a0 legacy=0x7f95d40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.621+0000 7f95d2a00640 1 -- 172.21.15.114:0/1834749538 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d40cfcb0 con 0x7f95d40d1e90 2024-03-06T07:28:42.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.622+0000 7f95d2a00640 1 --2- 172.21.15.114:0/1834749538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d40d1e90 0x7f95d40d2360 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c24080c52897a1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.622+0000 7f95d2000640 1 -- 172.21.15.114:0/1834749538 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc035020 con 0x7f95d40d1e90 2024-03-06T07:28:42.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.623+0000 7f95d2000640 1 -- 172.21.15.114:0/1834749538 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc031640 con 0x7f95d40d1e90 2024-03-06T07:28:42.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.623+0000 7f95d2000640 1 -- 172.21.15.114:0/1834749538 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc02d040 con 0x7f95d40d1e90 2024-03-06T07:28:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.623+0000 7f95d8a00640 1 -- 172.21.15.114:0/1834749538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d40d1e90 msgr2=0x7f95d40d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.623+0000 7f95d8a00640 1 --2- 172.21.15.114:0/1834749538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d40d1e90 0x7f95d40d2360 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.623+0000 7f95d8a00640 1 -- 172.21.15.114:0/1834749538 shutdown_connections 2024-03-06T07:28:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.623+0000 7f95d8a00640 1 --2- 172.21.15.114:0/1834749538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d40d1e90 0x7f95d40d2360 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.623+0000 7f95d8a00640 1 --2- 172.21.15.114:0/1834749538 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d4161580 0x7f95d40d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.624+0000 7f95d8a00640 1 -- 172.21.15.114:0/1834749538 >> 172.21.15.114:0/1834749538 conn(0x7f95d40d3d80 msgr2=0x7f95d40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.624+0000 7f95d8a00640 1 -- 172.21.15.114:0/1834749538 shutdown_connections 2024-03-06T07:28:42.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.624+0000 7f95d8a00640 1 -- 172.21.15.114:0/1834749538 wait complete. 2024-03-06T07:28:42.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.625+0000 7f95d8a00640 1 Processor -- start 2024-03-06T07:28:42.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.626+0000 7f95d8a00640 1 -- start start 2024-03-06T07:28:42.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.626+0000 7f95d8a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d40d1e90 0x7f95d40cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.626+0000 7f95d8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95d40d28a0 0x7f95d41f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.626+0000 7f95d8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d4161580 0x7f95d40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.626+0000 7f95d8a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d40cf3a0 con 0x7f95d4161580 2024-03-06T07:28:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.626+0000 7f95d8a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d40b39a0 con 0x7f95d40d28a0 2024-03-06T07:28:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.626+0000 7f95d8a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d40b4ed0 con 0x7f95d40d1e90 2024-03-06T07:28:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d4161580 0x7f95d40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d4161580 0x7f95d40c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52358/0 (socket says 172.21.15.114:52358) 2024-03-06T07:28:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 -- 172.21.15.114:0/2784143223 learned_addr learned my addr 172.21.15.114:0/2784143223 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d2a00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95d40d28a0 0x7f95d41f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3400640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d40d1e90 0x7f95d40cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 -- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d40d1e90 msgr2=0x7f95d40cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d40d1e90 0x7f95d40cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 -- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95d40d28a0 msgr2=0x7f95d41f6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95d40d28a0 0x7f95d41f6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.627+0000 7f95d3e00640 1 -- 172.21.15.114:0/2784143223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d40cc7c0 con 0x7f95d4161580 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.628+0000 7f95d3e00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d4161580 0x7f95d40c93d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00c410 tx=0x7f95cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.628+0000 7f95cb400640 1 -- 172.21.15.114:0/2784143223 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc011790 con 0x7f95d4161580 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.628+0000 7f95cb400640 1 -- 172.21.15.114:0/2784143223 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc017070 con 0x7f95d4161580 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.628+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d40cbc40 con 0x7f95d4161580 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.628+0000 7f95cb400640 1 -- 172.21.15.114:0/2784143223 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc011bb0 con 0x7f95d4161580 2024-03-06T07:28:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.628+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d40cc200 con 0x7f95d4161580 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.629+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d40d18b0 con 0x7f95d4161580 2024-03-06T07:28:42.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.630+0000 7f95cb400640 1 -- 172.21.15.114:0/2784143223 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95cc01c460 con 0x7f95d4161580 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.631+0000 7f95cb400640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f95b006b500 0x7f95b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.631+0000 7f95cb400640 1 -- 172.21.15.114:0/2784143223 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f95cc089e40 con 0x7f95d4161580 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.631+0000 7f95d3400640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f95b006b500 0x7f95b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.632+0000 7f95d3400640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f95b006b500 0x7f95b006d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009e90 tx=0x7f95c400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.636+0000 7f95cb400640 1 -- 172.21.15.114:0/2784143223 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95cc021020 con 0x7f95d4161580 2024-03-06T07:28:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.833+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95d404f330 con 0x7f95d4161580 2024-03-06T07:28:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.834+0000 7f95cb400640 1 -- 172.21.15.114:0/2784143223 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f95cc058c30 con 0x7f95d4161580 2024-03-06T07:28:42.836 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:42.837 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":22,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:36.265733+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f95b006b500 msgr2=0x7f95b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f95b006b500 0x7f95b006d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009e90 tx=0x7f95c400b040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d4161580 msgr2=0x7f95d40c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d4161580 0x7f95d40c93d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00c410 tx=0x7f95cc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 shutdown_connections 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f95b006b500 0x7f95b006d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95d4161580 0x7f95d40c93d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95d40d28a0 0x7f95d41f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 --2- 172.21.15.114:0/2784143223 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f95d40d1e90 0x7f95d40cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 >> 172.21.15.114:0/2784143223 conn(0x7f95d40d3d80 msgr2=0x7f95d40cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 shutdown_connections 2024-03-06T07:28:42.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:42.837+0000 7f95d8a00640 1 -- 172.21.15.114:0/2784143223 wait complete. 2024-03-06T07:28:42.858 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:42.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.071+0000 7f3b05000640 1 Processor -- start 2024-03-06T07:28:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.071+0000 7f3b05000640 1 -- start start 2024-03-06T07:28:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3b05000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3b0015be10 0x7f3b000ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3b05000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0015c1f0 con 0x7f3b0006da40 2024-03-06T07:28:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3b05000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0006cac0 con 0x7f3b000aef20 2024-03-06T07:28:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3b05000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b000cd200 con 0x7f3b0015be10 2024-03-06T07:28:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3affe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3b0006da40 0x7f3b00071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52212/0 (socket says 172.21.15.114:52212) 2024-03-06T07:28:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3affe00640 1 -- 172.21.15.114:0/2191257151 learned_addr learned my addr 172.21.15.114:0/2191257151 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3aff400640 1 --2- 172.21.15.114:0/2191257151 >> v2:172.21.15.177:3300/0 conn(0x7f3b0015be10 0x7f3b000ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.073+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476823880 0 0) 0x7f3b0015c1f0 con 0x7f3b0006da40 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae4003520 con 0x7f3b0006da40 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741516294 0 0) 0x7f3ae4003520 con 0x7f3b0006da40 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 >> v2:172.21.15.177:3300/0 conn(0x7f3b0015be10 msgr2=0x7f3b000ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 --2- 172.21.15.114:0/2191257151 >> v2:172.21.15.177:3300/0 conn(0x7f3b0015be10 0x7f3b000ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 >> v1:172.21.15.177:6789/0 conn(0x7f3b000aef20 legacy=0x7f3b0006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0006cda0 con 0x7f3b0006da40 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f3af80019f0 con 0x7f3b0006da40 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 >> v1:172.21.15.114:6789/0 conn(0x7f3b0006da40 legacy=0x7f3b00071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4003a00 0x7f3ae4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ae4004800 0x7f3ae4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae40051c0 0x7f3ae400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b000cd200 con 0x7f3ae4004800 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae400c0c0 con 0x7f3ae40051c0 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae400c2c0 con 0x7f3ae4003a00 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.074+0000 7f3afea00640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ae4004800 0x7f3ae4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.075+0000 7f3afea00640 1 -- 172.21.15.114:0/2191257151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4003a00 msgr2=0x7f3ae4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.075+0000 7f3afea00640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4003a00 0x7f3ae4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.075+0000 7f3afea00640 1 -- 172.21.15.114:0/2191257151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae40051c0 msgr2=0x7f3ae400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.075+0000 7f3afea00640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ae40051c0 0x7f3ae400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.075+0000 7f3afea00640 1 -- 172.21.15.114:0/2191257151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4003520 con 0x7f3ae4004800 2024-03-06T07:28:43.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.075+0000 7f3affe00640 1 -- 172.21.15.114:0/2191257151 reap_dead start 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.076+0000 7f3afea00640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ae4004800 0x7f3ae4004c80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3aec001750 tx=0x7f3ae4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e8dc396802c6ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.076+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec001370 con 0x7f3ae4004800 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.076+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec0052d0 con 0x7f3ae4004800 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.076+0000 7f3afe000640 1 -- 172.21.15.114:0/2191257151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec003830 con 0x7f3ae4004800 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.077+0000 7f3b05000640 1 -- 172.21.15.114:0/2191257151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ae4004800 msgr2=0x7f3ae4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.077+0000 7f3b05000640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ae4004800 0x7f3ae4004c80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3aec001750 tx=0x7f3ae4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.077+0000 7f3b05000640 1 -- 172.21.15.114:0/2191257151 shutdown_connections 2024-03-06T07:28:43.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.077+0000 7f3b05000640 1 --2- 172.21.15.114:0/2191257151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ae4004800 0x7f3ae4004c80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.077+0000 7f3b05000640 1 -- 172.21.15.114:0/2191257151 >> 172.21.15.114:0/2191257151 conn(0x7f3b0006afc0 msgr2=0x7f3b0006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.077+0000 7f3b05000640 1 -- 172.21.15.114:0/2191257151 shutdown_connections 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.077+0000 7f3b05000640 1 -- 172.21.15.114:0/2191257151 wait complete. 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.080+0000 7f3b05000640 1 Processor -- start 2024-03-06T07:28:43.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.080+0000 7f3b05000640 1 -- start start 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.081+0000 7f3b05000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4004800 0x7f3b0014c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3b05000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b0015aa00 0x7f3b0014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3b05000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b001585e0 0x7f3b00158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3b05000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b000cda90 con 0x7f3b001585e0 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3b05000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0006d080 con 0x7f3b0015aa00 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3b05000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b000ccc20 con 0x7f3ae4004800 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b001585e0 0x7f3b00158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b001585e0 0x7f3b00158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52368/0 (socket says 172.21.15.114:52368) 2024-03-06T07:28:43.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 -- 172.21.15.114:0/653454198 learned_addr learned my addr 172.21.15.114:0/653454198 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3afea00640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b0015aa00 0x7f3b0014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3aff400640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4004800 0x7f3b0014c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 -- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4004800 msgr2=0x7f3b0014c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4004800 0x7f3b0014c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 -- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b0015aa00 msgr2=0x7f3b0014f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b0015aa00 0x7f3b0014f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3affe00640 1 -- 172.21.15.114:0/653454198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0014c940 con 0x7f3b001585e0 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3aff400640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4004800 0x7f3b0014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.082+0000 7f3afea00640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b0015aa00 0x7f3b0014f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.083+0000 7f3affe00640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b001585e0 0x7f3b00158a60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3b000ce1d0 tx=0x7f3af800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.083+0000 7f3af7400640 1 -- 172.21.15.114:0/653454198 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800cea0 con 0x7f3b001585e0 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.083+0000 7f3af7400640 1 -- 172.21.15.114:0/653454198 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af800a610 con 0x7f3b001585e0 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.083+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0014fb20 con 0x7f3b001585e0 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.083+0000 7f3af7400640 1 -- 172.21.15.114:0/653454198 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800a9e0 con 0x7f3b001585e0 2024-03-06T07:28:43.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.083+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b001500e0 con 0x7f3b001585e0 2024-03-06T07:28:43.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.084+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0004f330 con 0x7f3b001585e0 2024-03-06T07:28:43.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.085+0000 7f3af7400640 1 -- 172.21.15.114:0/653454198 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3af800ab80 con 0x7f3b001585e0 2024-03-06T07:28:43.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.086+0000 7f3af7400640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ae006b500 0x7f3ae006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.086+0000 7f3af7400640 1 -- 172.21.15.114:0/653454198 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f3af8089a30 con 0x7f3b001585e0 2024-03-06T07:28:43.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.086+0000 7f3aff400640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ae006b500 0x7f3ae006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.087+0000 7f3aff400640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ae006b500 0x7f3ae006d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b000686c0 tx=0x7f3af0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.091+0000 7f3af7400640 1 -- 172.21.15.114:0/653454198 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3af8058780 con 0x7f3b001585e0 2024-03-06T07:28:43.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.287+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b000b11d0 con 0x7f3b001585e0 2024-03-06T07:28:43.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.288+0000 7f3af7400640 1 -- 172.21.15.114:0/653454198 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3af8058080 con 0x7f3b001585e0 2024-03-06T07:28:43.290 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:43.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.290+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ae006b500 msgr2=0x7f3ae006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.290+0000 7f3b05000640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ae006b500 0x7f3ae006d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b000686c0 tx=0x7f3af0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:43.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.290+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b001585e0 msgr2=0x7f3b00158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.290+0000 7f3b05000640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b001585e0 0x7f3b00158a60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3b000ce1d0 tx=0x7f3af800cb90 comp rx=0 tx=0).stop 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 shutdown_connections 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ae006b500 0x7f3ae006d9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b001585e0 0x7f3b00158a60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b0015aa00 0x7f3b0014f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 --2- 172.21.15.114:0/653454198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ae4004800 0x7f3b0014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 >> 172.21.15.114:0/653454198 conn(0x7f3b0006afc0 msgr2=0x7f3b001565e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 shutdown_connections 2024-03-06T07:28:43.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.291+0000 7f3b05000640 1 -- 172.21.15.114:0/653454198 wait complete. 2024-03-06T07:28:43.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:43.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.517+0000 7f7c6ec00640 1 Processor -- start 2024-03-06T07:28:43.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.517+0000 7f7c6ec00640 1 -- start start 2024-03-06T07:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c6ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c680d07f0 0x7f7c680d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c6ec00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c6815e450 0x7f7c680b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c6ec00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7c680d1bf0 0x7f7c680b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c6ec00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c680d4930 con 0x7f7c6815e450 2024-03-06T07:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c6ec00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c680b3730 con 0x7f7c680d1bf0 2024-03-06T07:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c6ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68067b20 con 0x7f7c680d07f0 2024-03-06T07:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c67400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c6815e450 0x7f7c680b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c67400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c6815e450 0x7f7c680b4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52380/0 (socket says 172.21.15.114:52380) 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c67400640 1 -- 172.21.15.114:0/1021695943 learned_addr learned my addr 172.21.15.114:0/1021695943 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.519+0000 7f7c6cc00640 1 --2- 172.21.15.114:0/1021695943 >> v2:172.21.15.177:3300/0 conn(0x7f7c680d1bf0 0x7f7c680b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.520+0000 7f7c67e00640 1 --2- 172.21.15.114:0/1021695943 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c680d07f0 0x7f7c680d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.520+0000 7f7c67400640 1 -- 172.21.15.114:0/1021695943 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c680d07f0 msgr2=0x7f7c680d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.520+0000 7f7c67400640 1 --2- 172.21.15.114:0/1021695943 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c680d07f0 0x7f7c680d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.520+0000 7f7c67400640 1 -- 172.21.15.114:0/1021695943 >> v2:172.21.15.177:3300/0 conn(0x7f7c680d1bf0 msgr2=0x7f7c680b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.520+0000 7f7c67400640 1 --2- 172.21.15.114:0/1021695943 >> v2:172.21.15.177:3300/0 conn(0x7f7c680d1bf0 0x7f7c680b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.520+0000 7f7c67400640 1 -- 172.21.15.114:0/1021695943 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c680d1110 con 0x7f7c6815e450 2024-03-06T07:28:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.520+0000 7f7c6cc00640 1 --2- 172.21.15.114:0/1021695943 >> v2:172.21.15.177:3300/0 conn(0x7f7c680d1bf0 0x7f7c680b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.521+0000 7f7c67400640 1 --2- 172.21.15.114:0/1021695943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c6815e450 0x7f7c680b4480 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7c680ce4f0 tx=0x7f7c5402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfe26c02f4f8aca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.521+0000 7f7c66a00640 1 -- 172.21.15.114:0/1021695943 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5402e070 con 0x7f7c6815e450 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.521+0000 7f7c66a00640 1 -- 172.21.15.114:0/1021695943 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c54037070 con 0x7f7c6815e450 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.521+0000 7f7c66a00640 1 -- 172.21.15.114:0/1021695943 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54031a00 con 0x7f7c6815e450 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 -- 172.21.15.114:0/1021695943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c6815e450 msgr2=0x7f7c680b4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/1021695943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c6815e450 0x7f7c680b4480 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7c680ce4f0 tx=0x7f7c5402b880 comp rx=0 tx=0).stop 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 -- 172.21.15.114:0/1021695943 shutdown_connections 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/1021695943 >> v2:172.21.15.177:3300/0 conn(0x7f7c680d1bf0 0x7f7c680b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/1021695943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c6815e450 0x7f7c680b4480 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/1021695943 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c680d07f0 0x7f7c680d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 -- 172.21.15.114:0/1021695943 >> 172.21.15.114:0/1021695943 conn(0x7f7c680d3e10 msgr2=0x7f7c680d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 -- 172.21.15.114:0/1021695943 shutdown_connections 2024-03-06T07:28:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.522+0000 7f7c6ec00640 1 -- 172.21.15.114:0/1021695943 wait complete. 2024-03-06T07:28:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.523+0000 7f7c6ec00640 1 Processor -- start 2024-03-06T07:28:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.524+0000 7f7c6ec00640 1 -- start start 2024-03-06T07:28:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.524+0000 7f7c6ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c680d07f0 0x7f7c6814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.524+0000 7f7c6ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680d1bf0 0x7f7c68160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c6ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c6815e450 0x7f7c68160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c6ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680b3c80 con 0x7f7c680d1bf0 2024-03-06T07:28:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c6ec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680b39a0 con 0x7f7c680d07f0 2024-03-06T07:28:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c6ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680b3f60 con 0x7f7c6815e450 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680d1bf0 0x7f7c68160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680d1bf0 0x7f7c68160840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52384/0 (socket says 172.21.15.114:52384) 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 -- 172.21.15.114:0/3209551493 learned_addr learned my addr 172.21.15.114:0/3209551493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67e00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c680d07f0 0x7f7c6814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 -- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c6815e450 msgr2=0x7f7c68160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c6cc00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c6815e450 0x7f7c68160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c6815e450 0x7f7c68160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 -- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c680d07f0 msgr2=0x7f7c6814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c680d07f0 0x7f7c6814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67400640 1 -- 172.21.15.114:0/3209551493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68158c30 con 0x7f7c680d1bf0 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.525+0000 7f7c67e00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c680d07f0 0x7f7c6814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.526+0000 7f7c67400640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680d1bf0 0x7f7c68160840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7c5402b8c0 tx=0x7f7c540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.526+0000 7f7c5fe00640 1 -- 172.21.15.114:0/3209551493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54032880 con 0x7f7c680d1bf0 2024-03-06T07:28:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.526+0000 7f7c5fe00640 1 -- 172.21.15.114:0/3209551493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c54032a40 con 0x7f7c680d1bf0 2024-03-06T07:28:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.526+0000 7f7c5fe00640 1 -- 172.21.15.114:0/3209551493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54032e30 con 0x7f7c680d1bf0 2024-03-06T07:28:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.526+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c680cd310 con 0x7f7c680d1bf0 2024-03-06T07:28:43.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.526+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c680cd8d0 con 0x7f7c680d1bf0 2024-03-06T07:28:43.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.528+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6804f330 con 0x7f7c680d1bf0 2024-03-06T07:28:43.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.528+0000 7f7c5fe00640 1 -- 172.21.15.114:0/3209551493 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c5404c060 con 0x7f7c680d1bf0 2024-03-06T07:28:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.530+0000 7f7c5fe00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7c4806b4d0 0x7f7c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.530+0000 7f7c5fe00640 1 -- 172.21.15.114:0/3209551493 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7c540abf10 con 0x7f7c680d1bf0 2024-03-06T07:28:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.530+0000 7f7c67e00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7c4806b4d0 0x7f7c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.531+0000 7f7c67e00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7c4806b4d0 0x7f7c4806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c68158b70 tx=0x7f7c58008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.534+0000 7f7c5fe00640 1 -- 172.21.15.114:0/3209551493 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c54040070 con 0x7f7c680d1bf0 2024-03-06T07:28:43.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.770+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c680d21d0 con 0x7f7c680d1bf0 2024-03-06T07:28:43.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.770+0000 7f7c5fe00640 1 -- 172.21.15.114:0/3209551493 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c5407acf0 con 0x7f7c680d1bf0 2024-03-06T07:28:43.772 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7c4806b4d0 msgr2=0x7f7c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7c4806b4d0 0x7f7c4806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7c68158b70 tx=0x7f7c58008040 comp rx=0 tx=0).stop 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680d1bf0 msgr2=0x7f7c68160840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680d1bf0 0x7f7c68160840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7c5402b8c0 tx=0x7f7c540046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 shutdown_connections 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c6815e450 0x7f7c68160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7c4806b4d0 0x7f7c4806d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680d1bf0 0x7f7c68160840 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 --2- 172.21.15.114:0/3209551493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c680d07f0 0x7f7c6814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 >> 172.21.15.114:0/3209551493 conn(0x7f7c680d3e10 msgr2=0x7f7c680b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 shutdown_connections 2024-03-06T07:28:43.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.773+0000 7f7c6ec00640 1 -- 172.21.15.114:0/3209551493 wait complete. 2024-03-06T07:28:43.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:43.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.995+0000 7f644aa00640 1 Processor -- start 2024-03-06T07:28:43.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.995+0000 7f644aa00640 1 -- start start 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f644aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64440b2c40 0x7f644415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f644aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f644406e2f0 0x7f64440cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f644aa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644406cba0 con 0x7f644406d960 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f644aa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644415f700 con 0x7f644406e2f0 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f644aa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644406d4d0 con 0x7f64440b2c40 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f6448a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f644406e2f0 0x7f64440cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f6448a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f644406e2f0 0x7f64440cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52392/0 (socket says 172.21.15.114:52392) 2024-03-06T07:28:43.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.997+0000 7f6448a00640 1 -- 172.21.15.114:0/4089123520 learned_addr learned my addr 172.21.15.114:0/4089123520 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:44.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.998+0000 7f6443e00640 1 --2- 172.21.15.114:0/4089123520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64440b2c40 0x7f644415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.998+0000 7f6448a00640 1 -- 172.21.15.114:0/4089123520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64440b2c40 msgr2=0x7f644415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.998+0000 7f6448a00640 1 --2- 172.21.15.114:0/4089123520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64440b2c40 0x7f644415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.998+0000 7f6448a00640 1 -- 172.21.15.114:0/4089123520 >> v1:172.21.15.114:6789/0 conn(0x7f644406d960 legacy=0x7f64440b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.998+0000 7f6448a00640 1 -- 172.21.15.114:0/4089123520 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644406ce80 con 0x7f644406e2f0 2024-03-06T07:28:44.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:43.999+0000 7f6448a00640 1 --2- 172.21.15.114:0/4089123520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f644406e2f0 0x7f64440cda90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f643c009e50 tx=0x7f643c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c8df77ae6d8298 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.000+0000 7f6442a00640 1 -- 172.21.15.114:0/4089123520 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c02cb00 con 0x7f644406e2f0 2024-03-06T07:28:44.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.000+0000 7f6442a00640 1 -- 172.21.15.114:0/4089123520 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c02cca0 con 0x7f644406e2f0 2024-03-06T07:28:44.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.000+0000 7f6442a00640 1 -- 172.21.15.114:0/4089123520 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c034610 con 0x7f644406e2f0 2024-03-06T07:28:44.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.000+0000 7f644aa00640 1 -- 172.21.15.114:0/4089123520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f644406e2f0 msgr2=0x7f64440cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.000+0000 7f644aa00640 1 --2- 172.21.15.114:0/4089123520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f644406e2f0 0x7f64440cda90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f643c009e50 tx=0x7f643c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:44.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.001+0000 7f644aa00640 1 -- 172.21.15.114:0/4089123520 shutdown_connections 2024-03-06T07:28:44.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.001+0000 7f644aa00640 1 --2- 172.21.15.114:0/4089123520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f644406e2f0 0x7f64440cda90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.001+0000 7f644aa00640 1 --2- 172.21.15.114:0/4089123520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64440b2c40 0x7f644415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.001+0000 7f644aa00640 1 -- 172.21.15.114:0/4089123520 >> 172.21.15.114:0/4089123520 conn(0x7f64440b1940 msgr2=0x7f64440b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.001+0000 7f644aa00640 1 -- 172.21.15.114:0/4089123520 shutdown_connections 2024-03-06T07:28:44.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.001+0000 7f644aa00640 1 -- 172.21.15.114:0/4089123520 wait complete. 2024-03-06T07:28:44.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.002+0000 7f644aa00640 1 Processor -- start 2024-03-06T07:28:44.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.002+0000 7f644aa00640 1 -- start start 2024-03-06T07:28:44.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.003+0000 7f644aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f644406d960 0x7f6444159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.003+0000 7f644aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f644406e2f0 0x7f6444149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.003+0000 7f644aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64440b2c40 0x7f644414a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.003+0000 7f644aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644406e8d0 con 0x7f64440b2c40 2024-03-06T07:28:44.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.003+0000 7f644aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644406e750 con 0x7f644406d960 2024-03-06T07:28:44.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.003+0000 7f644aa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64440cdfd0 con 0x7f644406e2f0 2024-03-06T07:28:44.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64440b2c40 0x7f644414a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64440b2c40 0x7f644414a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52404/0 (socket says 172.21.15.114:52404) 2024-03-06T07:28:44.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 -- 172.21.15.114:0/3297785729 learned_addr learned my addr 172.21.15.114:0/3297785729 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:44.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6443e00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f644406d960 0x7f6444159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 -- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f644406e2f0 msgr2=0x7f6444149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6443400640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f644406e2f0 0x7f6444149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f644406e2f0 0x7f6444149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 -- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f644406d960 msgr2=0x7f6444159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f644406d960 0x7f6444159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 -- 172.21.15.114:0/3297785729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440c9060 con 0x7f64440b2c40 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.004+0000 7f6448a00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64440b2c40 0x7f644414a390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f643c036040 tx=0x7f643c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.005+0000 7f643be00640 1 -- 172.21.15.114:0/3297785729 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c043020 con 0x7f64440b2c40 2024-03-06T07:28:44.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.005+0000 7f643be00640 1 -- 172.21.15.114:0/3297785729 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c02ce50 con 0x7f64440b2c40 2024-03-06T07:28:44.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.005+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64440c9340 con 0x7f64440b2c40 2024-03-06T07:28:44.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.005+0000 7f643be00640 1 -- 172.21.15.114:0/3297785729 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c009510 con 0x7f64440b2c40 2024-03-06T07:28:44.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.005+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644414a8d0 con 0x7f64440b2c40 2024-03-06T07:28:44.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.006+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64440c9510 con 0x7f64440b2c40 2024-03-06T07:28:44.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.007+0000 7f643be00640 1 -- 172.21.15.114:0/3297785729 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f643c046020 con 0x7f64440b2c40 2024-03-06T07:28:44.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.008+0000 7f643be00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f642406b4d0 0x7f642406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.008+0000 7f643be00640 1 -- 172.21.15.114:0/3297785729 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f643c0aca40 con 0x7f64440b2c40 2024-03-06T07:28:44.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.008+0000 7f6443e00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f642406b4d0 0x7f642406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.009+0000 7f6443e00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f642406b4d0 0x7f642406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6434006fd0 tx=0x7f6434008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.013+0000 7f643be00640 1 -- 172.21.15.114:0/3297785729 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f643c04d030 con 0x7f64440b2c40 2024-03-06T07:28:44.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.209+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f644404f330 con 0x7f64440b2c40 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.210+0000 7f643be00640 1 -- 172.21.15.114:0/3297785729 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f643c07b700 con 0x7f64440b2c40 2024-03-06T07:28:44.212 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f642406b4d0 msgr2=0x7f642406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f642406b4d0 0x7f642406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6434006fd0 tx=0x7f6434008040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64440b2c40 msgr2=0x7f644414a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64440b2c40 0x7f644414a390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f643c036040 tx=0x7f643c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 shutdown_connections 2024-03-06T07:28:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f642406b4d0 0x7f642406d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64440b2c40 0x7f644414a390 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f644406e2f0 0x7f6444149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 --2- 172.21.15.114:0/3297785729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f644406d960 0x7f6444159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 >> 172.21.15.114:0/3297785729 conn(0x7f64440b1940 msgr2=0x7f64440b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 shutdown_connections 2024-03-06T07:28:44.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.213+0000 7f644aa00640 1 -- 172.21.15.114:0/3297785729 wait complete. 2024-03-06T07:28:44.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:44.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.428+0000 7f425e200640 1 Processor -- start 2024-03-06T07:28:44.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.429+0000 7f425e200640 1 -- start start 2024-03-06T07:28:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f425e200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42580c4930 0x7f42580c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f425e200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f42580c4d10 0x7f42580c5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f425e200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c56b0 0x7f425806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f425e200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4258145b30 con 0x7f42580c4930 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f425e200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4258142ce0 con 0x7f42580c4d10 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f425e200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42580cbca0 con 0x7f42580c56b0 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42580c4930 0x7f42580c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42580c4930 0x7f42580c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52418/0 (socket says 172.21.15.114:52418) 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 -- 172.21.15.114:0/2743306842 learned_addr learned my addr 172.21.15.114:0/2743306842 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:44.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 -- 172.21.15.114:0/2743306842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c56b0 msgr2=0x7f425806d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 --2- 172.21.15.114:0/2743306842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c56b0 0x7f425806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 -- 172.21.15.114:0/2743306842 >> v2:172.21.15.177:3300/0 conn(0x7f42580c4d10 msgr2=0x7f42580c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 --2- 172.21.15.114:0/2743306842 >> v2:172.21.15.177:3300/0 conn(0x7f42580c4d10 0x7f42580c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.431+0000 7f4257400640 1 -- 172.21.15.114:0/2743306842 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42580cbf80 con 0x7f42580c4930 2024-03-06T07:28:44.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.433+0000 7f4257400640 1 --2- 172.21.15.114:0/2743306842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c4930 0x7f42580c6460 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4258068c90 tx=0x7f424802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9962078880336fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.433+0000 7f4256000640 1 -- 172.21.15.114:0/2743306842 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424802e070 con 0x7f42580c4930 2024-03-06T07:28:44.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.433+0000 7f4256000640 1 -- 172.21.15.114:0/2743306842 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4248037070 con 0x7f42580c4930 2024-03-06T07:28:44.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.433+0000 7f4256000640 1 -- 172.21.15.114:0/2743306842 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4248031a00 con 0x7f42580c4930 2024-03-06T07:28:44.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.433+0000 7f425e200640 1 -- 172.21.15.114:0/2743306842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c4930 msgr2=0x7f42580c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.433+0000 7f425e200640 1 --2- 172.21.15.114:0/2743306842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c4930 0x7f42580c6460 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4258068c90 tx=0x7f424802b880 comp rx=0 tx=0).stop 2024-03-06T07:28:44.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.434+0000 7f425e200640 1 -- 172.21.15.114:0/2743306842 shutdown_connections 2024-03-06T07:28:44.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.434+0000 7f425e200640 1 --2- 172.21.15.114:0/2743306842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c56b0 0x7f425806d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.434+0000 7f425e200640 1 --2- 172.21.15.114:0/2743306842 >> v2:172.21.15.177:3300/0 conn(0x7f42580c4d10 0x7f42580c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.434+0000 7f425e200640 1 --2- 172.21.15.114:0/2743306842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c4930 0x7f42580c6460 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.434+0000 7f425e200640 1 -- 172.21.15.114:0/2743306842 >> 172.21.15.114:0/2743306842 conn(0x7f42580cc4f0 msgr2=0x7f42580cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.434+0000 7f425e200640 1 -- 172.21.15.114:0/2743306842 shutdown_connections 2024-03-06T07:28:44.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.434+0000 7f425e200640 1 -- 172.21.15.114:0/2743306842 wait complete. 2024-03-06T07:28:44.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.436+0000 7f425e200640 1 Processor -- start 2024-03-06T07:28:44.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.436+0000 7f425e200640 1 -- start start 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f425e200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c4930 0x7f42580b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f425e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42580c4d10 0x7f42580c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f425e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c56b0 0x7f42581e4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f425e200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42580cc260 con 0x7f42580c56b0 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f425e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42580ca3b0 con 0x7f42580c4d10 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f425e200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42580c5b40 con 0x7f42580c4930 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f4257e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c56b0 0x7f42581e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f4257e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c56b0 0x7f42581e4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52432/0 (socket says 172.21.15.114:52432) 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.437+0000 7f4257e00640 1 -- 172.21.15.114:0/1922361392 learned_addr learned my addr 172.21.15.114:0/1922361392 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4257400640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c4930 0x7f42580b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4256a00640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42580c4d10 0x7f42580c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4257e00640 1 -- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c4930 msgr2=0x7f42580b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4257e00640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c4930 0x7f42580b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4257e00640 1 -- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42580c4d10 msgr2=0x7f42580c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4257e00640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42580c4d10 0x7f42580c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4257e00640 1 -- 172.21.15.114:0/1922361392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42580c8de0 con 0x7f42580c56b0 2024-03-06T07:28:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.438+0000 7f4257e00640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c56b0 0x7f42581e4710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f425000b9e0 tx=0x7f425000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.439+0000 7f424f400640 1 -- 172.21.15.114:0/1922361392 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250019070 con 0x7f42580c56b0 2024-03-06T07:28:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.439+0000 7f424f400640 1 -- 172.21.15.114:0/1922361392 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425000cac0 con 0x7f42580c56b0 2024-03-06T07:28:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.439+0000 7f424f400640 1 -- 172.21.15.114:0/1922361392 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250014440 con 0x7f42580c56b0 2024-03-06T07:28:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.439+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42581e4c50 con 0x7f42580c56b0 2024-03-06T07:28:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.439+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42580c7dc0 con 0x7f42580c56b0 2024-03-06T07:28:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.440+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425804f330 con 0x7f42580c56b0 2024-03-06T07:28:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.442+0000 7f424f400640 1 -- 172.21.15.114:0/1922361392 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f425001c070 con 0x7f42580c56b0 2024-03-06T07:28:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.443+0000 7f424f400640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f423806b4d0 0x7f423806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.444+0000 7f424f400640 1 -- 172.21.15.114:0/1922361392 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4250089ec0 con 0x7f42580c56b0 2024-03-06T07:28:44.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.444+0000 7f4257400640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f423806b4d0 0x7f423806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.445+0000 7f4257400640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f423806b4d0 0x7f423806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4248036000 tx=0x7f424803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.448+0000 7f424f400640 1 -- 172.21.15.114:0/1922361392 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4250058d40 con 0x7f42580c56b0 2024-03-06T07:28:44.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.671+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f42580c7b70 con 0x7f42580c56b0 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.671+0000 7f424f400640 1 -- 172.21.15.114:0/1922361392 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4250058640 con 0x7f42580c56b0 2024-03-06T07:28:44.673 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f423806b4d0 msgr2=0x7f423806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f423806b4d0 0x7f423806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4248036000 tx=0x7f424803d040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c56b0 msgr2=0x7f42581e4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c56b0 0x7f42581e4710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f425000b9e0 tx=0x7f425000bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 shutdown_connections 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f423806b4d0 0x7f423806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42580c56b0 0x7f42581e4710 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42580c4d10 0x7f42580c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 --2- 172.21.15.114:0/1922361392 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42580c4930 0x7f42580b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 >> 172.21.15.114:0/1922361392 conn(0x7f42580cc4f0 msgr2=0x7f42580aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 shutdown_connections 2024-03-06T07:28:44.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:44.674+0000 7f425e200640 1 -- 172.21.15.114:0/1922361392 wait complete. 2024-03-06T07:28:44.694 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-06T07:28:44.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:44.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-06T07:28:44.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:44.743 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:44.784 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:28:44.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:44.785 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:44.832 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:44.832 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:44.883 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:44.883 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:44.981 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:28:44.981 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:45.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:45.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.246+0000 7f7e5a200640 1 Processor -- start 2024-03-06T07:28:45.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.246+0000 7f7e5a200640 1 -- start start 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e5a200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7e54144c00 0x7f7e5406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e5a200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e5415f9d0 0x7f7e5415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e5a200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e540ce0a0 0x7f7e540ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e5a200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5406ee70 con 0x7f7e5415f9d0 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e5a200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5406cef0 con 0x7f7e54144c00 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e5a200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5406d1d0 con 0x7f7e540ce0a0 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e52a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e5415f9d0 0x7f7e5415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e52a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e5415f9d0 0x7f7e5415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52448/0 (socket says 172.21.15.114:52448) 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e52a00640 1 -- 172.21.15.114:0/3139756496 learned_addr learned my addr 172.21.15.114:0/3139756496 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e53e00640 1 --2- 172.21.15.114:0/3139756496 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e540ce0a0 0x7f7e540ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.248+0000 7f7e53400640 1 --2- 172.21.15.114:0/3139756496 >> v2:172.21.15.177:3300/0 conn(0x7f7e54144c00 0x7f7e5406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.249+0000 7f7e52a00640 1 -- 172.21.15.114:0/3139756496 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e540ce0a0 msgr2=0x7f7e540ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.249+0000 7f7e52a00640 1 --2- 172.21.15.114:0/3139756496 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e540ce0a0 0x7f7e540ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.249+0000 7f7e52a00640 1 -- 172.21.15.114:0/3139756496 >> v2:172.21.15.177:3300/0 conn(0x7f7e54144c00 msgr2=0x7f7e5406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.249+0000 7f7e52a00640 1 --2- 172.21.15.114:0/3139756496 >> v2:172.21.15.177:3300/0 conn(0x7f7e54144c00 0x7f7e5406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.249+0000 7f7e52a00640 1 -- 172.21.15.114:0/3139756496 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5406d4b0 con 0x7f7e5415f9d0 2024-03-06T07:28:45.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.250+0000 7f7e52a00640 1 --2- 172.21.15.114:0/3139756496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e5415fe60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7e540b25f0 tx=0x7f7e3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfc3aabdcf0b2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.250+0000 7f7e52000640 1 -- 172.21.15.114:0/3139756496 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c035020 con 0x7f7e5415f9d0 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.250+0000 7f7e52000640 1 -- 172.21.15.114:0/3139756496 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c031870 con 0x7f7e5415f9d0 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.250+0000 7f7e52000640 1 -- 172.21.15.114:0/3139756496 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c031c40 con 0x7f7e5415f9d0 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 -- 172.21.15.114:0/3139756496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 msgr2=0x7f7e5415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3139756496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e5415fe60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7e540b25f0 tx=0x7f7e3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 -- 172.21.15.114:0/3139756496 shutdown_connections 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3139756496 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e540ce0a0 0x7f7e540ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3139756496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e5415fe60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3139756496 >> v2:172.21.15.177:3300/0 conn(0x7f7e54144c00 0x7f7e5406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 -- 172.21.15.114:0/3139756496 >> 172.21.15.114:0/3139756496 conn(0x7f7e540b1730 msgr2=0x7f7e540b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 -- 172.21.15.114:0/3139756496 shutdown_connections 2024-03-06T07:28:45.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.251+0000 7f7e5a200640 1 -- 172.21.15.114:0/3139756496 wait complete. 2024-03-06T07:28:45.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.254+0000 7f7e5a200640 1 Processor -- start 2024-03-06T07:28:45.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.254+0000 7f7e5a200640 1 -- start start 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.255+0000 7f7e5a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e540ce0a0 0x7f7e54138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e5a200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e54144c00 0x7f7e54140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e5a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e54141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e5a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5406cb50 con 0x7f7e5415f9d0 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e5a200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e54160730 con 0x7f7e540ce0a0 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e5a200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5406ccd0 con 0x7f7e54144c00 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e54141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e54141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52462/0 (socket says 172.21.15.114:52462) 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 -- 172.21.15.114:0/3188565040 learned_addr learned my addr 172.21.15.114:0/3188565040 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e52a00640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e54144c00 0x7f7e54140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53400640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e540ce0a0 0x7f7e54138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 -- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e54144c00 msgr2=0x7f7e54140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e54144c00 0x7f7e54140d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 -- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e540ce0a0 msgr2=0x7f7e54138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e540ce0a0 0x7f7e54138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.256+0000 7f7e53e00640 1 -- 172.21.15.114:0/3188565040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e54139fc0 con 0x7f7e5415f9d0 2024-03-06T07:28:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.257+0000 7f7e53e00640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e54141290 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400eed0 tx=0x7f7e4400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.257+0000 7f7e4b400640 1 -- 172.21.15.114:0/3188565040 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e44018020 con 0x7f7e5415f9d0 2024-03-06T07:28:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.257+0000 7f7e4b400640 1 -- 172.21.15.114:0/3188565040 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4400bd20 con 0x7f7e5415f9d0 2024-03-06T07:28:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.257+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5413a2a0 con 0x7f7e5415f9d0 2024-03-06T07:28:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.257+0000 7f7e4b400640 1 -- 172.21.15.114:0/3188565040 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e44014640 con 0x7f7e5415f9d0 2024-03-06T07:28:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.257+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e54139080 con 0x7f7e5415f9d0 2024-03-06T07:28:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.258+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5404f330 con 0x7f7e5415f9d0 2024-03-06T07:28:45.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.260+0000 7f7e4b400640 1 -- 172.21.15.114:0/3188565040 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e44005d10 con 0x7f7e5415f9d0 2024-03-06T07:28:45.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.261+0000 7f7e4b400640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e3006b4d0 0x7f7e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.261+0000 7f7e4b400640 1 -- 172.21.15.114:0/3188565040 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7e4408ab40 con 0x7f7e5415f9d0 2024-03-06T07:28:45.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.261+0000 7f7e53400640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e3006b4d0 0x7f7e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.262+0000 7f7e53400640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e3006b4d0 0x7f7e3006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c006fd0 tx=0x7f7e4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.271+0000 7f7e4b400640 1 -- 172.21.15.114:0/3188565040 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e44022030 con 0x7f7e5415f9d0 2024-03-06T07:28:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.468+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e54072890 con 0x7f7e5415f9d0 2024-03-06T07:28:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.470+0000 7f7e4b400640 1 -- 172.21.15.114:0/3188565040 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f7e44059930 con 0x7f7e5415f9d0 2024-03-06T07:28:45.472 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:45.472 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":22,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:36.265733+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.472+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e3006b4d0 msgr2=0x7f7e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.472+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e3006b4d0 0x7f7e3006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c006fd0 tx=0x7f7e4c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.472+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 msgr2=0x7f7e54141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.472+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e54141290 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400eed0 tx=0x7f7e4400b480 comp rx=0 tx=0).stop 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 shutdown_connections 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e3006b4d0 0x7f7e3006d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e5415f9d0 0x7f7e54141290 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e54144c00 0x7f7e54140d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 --2- 172.21.15.114:0/3188565040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e540ce0a0 0x7f7e54138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 >> 172.21.15.114:0/3188565040 conn(0x7f7e540b1730 msgr2=0x7f7e54070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 shutdown_connections 2024-03-06T07:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.473+0000 7f7e5a200640 1 -- 172.21.15.114:0/3188565040 wait complete. 2024-03-06T07:28:45.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:45.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:45.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.695+0000 7f3246a00640 1 Processor -- start 2024-03-06T07:28:45.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.695+0000 7f3246a00640 1 -- start start 2024-03-06T07:28:45.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.696+0000 7f3246a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f32400b4b90 0x7f32400cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.696+0000 7f3246a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f32400d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.696+0000 7f3246a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32400cf2f0 con 0x7f32400d06d0 2024-03-06T07:28:45.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.696+0000 7f3246a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32400cf5d0 con 0x7f32400b4b90 2024-03-06T07:28:45.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.696+0000 7f3246a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32400cf8b0 con 0x7f32400d1db0 2024-03-06T07:28:45.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3244c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f32400b4b90 0x7f32400cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3244c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f32400b4b90 0x7f32400cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52466/0 (socket says 172.21.15.114:52466) 2024-03-06T07:28:45.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3245600640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f32400d06d0 0x7f32400d0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52230/0 (socket says 172.21.15.114:52230) 2024-03-06T07:28:45.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3244c00640 1 -- 172.21.15.114:0/2415917323 learned_addr learned my addr 172.21.15.114:0/2415917323 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:45.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3246000640 1 --2- 172.21.15.114:0/2415917323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f32400d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3244c00640 1 -- 172.21.15.114:0/2415917323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 msgr2=0x7f32400d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3244c00640 1 --2- 172.21.15.114:0/2415917323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f32400d2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3244c00640 1 -- 172.21.15.114:0/2415917323 >> v1:172.21.15.114:6789/0 conn(0x7f32400d06d0 legacy=0x7f32400d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.697+0000 7f3244c00640 1 -- 172.21.15.114:0/2415917323 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32400cfb90 con 0x7f32400b4b90 2024-03-06T07:28:45.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.698+0000 7f3244c00640 1 --2- 172.21.15.114:0/2415917323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400b4b90 0x7f32400cea10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f322c009870 tx=0x7f322c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769705028f397c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f323fe00640 1 -- 172.21.15.114:0/2415917323 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c034070 con 0x7f32400b4b90 2024-03-06T07:28:45.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f323fe00640 1 -- 172.21.15.114:0/2415917323 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c030840 con 0x7f32400b4b90 2024-03-06T07:28:45.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f323fe00640 1 -- 172.21.15.114:0/2415917323 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c030c10 con 0x7f32400b4b90 2024-03-06T07:28:45.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 -- 172.21.15.114:0/2415917323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400b4b90 msgr2=0x7f32400cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 --2- 172.21.15.114:0/2415917323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400b4b90 0x7f32400cea10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f322c009870 tx=0x7f322c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:45.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 -- 172.21.15.114:0/2415917323 shutdown_connections 2024-03-06T07:28:45.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 --2- 172.21.15.114:0/2415917323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f32400d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 --2- 172.21.15.114:0/2415917323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400b4b90 0x7f32400cea10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 -- 172.21.15.114:0/2415917323 >> 172.21.15.114:0/2415917323 conn(0x7f324009d070 msgr2=0x7f32400d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 -- 172.21.15.114:0/2415917323 shutdown_connections 2024-03-06T07:28:45.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.699+0000 7f3246a00640 1 -- 172.21.15.114:0/2415917323 wait complete. 2024-03-06T07:28:45.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.701+0000 7f3246a00640 1 Processor -- start 2024-03-06T07:28:45.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.701+0000 7f3246a00640 1 -- start start 2024-03-06T07:28:45.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3246a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32400b4b90 0x7f32400ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3246a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400d06d0 0x7f3240160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3246a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f324014d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3246a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32400cf0d0 con 0x7f32400d06d0 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3246a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32400cef50 con 0x7f32400b4b90 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3246a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32400d2780 con 0x7f32400d1db0 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3244c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400d06d0 0x7f3240160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3244c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400d06d0 0x7f3240160460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52482/0 (socket says 172.21.15.114:52482) 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.702+0000 7f3244c00640 1 -- 172.21.15.114:0/1949694673 learned_addr learned my addr 172.21.15.114:0/1949694673 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3246000640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f324014d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3245600640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32400b4b90 0x7f32400ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3244c00640 1 -- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 msgr2=0x7f324014d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3244c00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f324014d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3244c00640 1 -- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32400b4b90 msgr2=0x7f32400ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3244c00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32400b4b90 0x7f32400ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3244c00640 1 -- 172.21.15.114:0/1949694673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3240160dd0 con 0x7f32400d06d0 2024-03-06T07:28:45.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f3244c00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400d06d0 0x7f3240160460 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f322c000c00 tx=0x7f322c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f323d600640 1 -- 172.21.15.114:0/1949694673 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c048070 con 0x7f32400d06d0 2024-03-06T07:28:45.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.703+0000 7f323d600640 1 -- 172.21.15.114:0/1949694673 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c030640 con 0x7f32400d06d0 2024-03-06T07:28:45.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.704+0000 7f323d600640 1 -- 172.21.15.114:0/1949694673 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c034040 con 0x7f32400d06d0 2024-03-06T07:28:45.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.704+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32401610b0 con 0x7f32400d06d0 2024-03-06T07:28:45.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.704+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3240161670 con 0x7f32400d06d0 2024-03-06T07:28:45.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.705+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324004f330 con 0x7f32400d06d0 2024-03-06T07:28:45.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.706+0000 7f323d600640 1 -- 172.21.15.114:0/1949694673 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f322c04b070 con 0x7f32400d06d0 2024-03-06T07:28:45.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.706+0000 7f323d600640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f322006b4d0 0x7f322006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.707+0000 7f323d600640 1 -- 172.21.15.114:0/1949694673 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f322c0ac150 con 0x7f32400d06d0 2024-03-06T07:28:45.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.707+0000 7f3245600640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f322006b4d0 0x7f322006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.707+0000 7f3245600640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f322006b4d0 0x7f322006d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3234000f60 tx=0x7f3234009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.711+0000 7f323d600640 1 -- 172.21.15.114:0/1949694673 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f322c07b060 con 0x7f32400d06d0 2024-03-06T07:28:45.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.903+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32400b55a0 con 0x7f32400d06d0 2024-03-06T07:28:45.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.904+0000 7f323d600640 1 -- 172.21.15.114:0/1949694673 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f322c07a960 con 0x7f32400d06d0 2024-03-06T07:28:45.907 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f322006b4d0 msgr2=0x7f322006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f322006b4d0 0x7f322006d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3234000f60 tx=0x7f3234009040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400d06d0 msgr2=0x7f3240160460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400d06d0 0x7f3240160460 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f322c000c00 tx=0x7f322c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 shutdown_connections 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f32400d1db0 0x7f324014d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f322006b4d0 0x7f322006d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32400d06d0 0x7f3240160460 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 --2- 172.21.15.114:0/1949694673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32400b4b90 0x7f32400ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 >> 172.21.15.114:0/1949694673 conn(0x7f324009d070 msgr2=0x7f32400b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.906+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 shutdown_connections 2024-03-06T07:28:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:45.907+0000 7f3246a00640 1 -- 172.21.15.114:0/1949694673 wait complete. 2024-03-06T07:28:45.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:46.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.101+0000 7f1c4c800640 1 Processor -- start 2024-03-06T07:28:46.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.101+0000 7f1c4c800640 1 -- start start 2024-03-06T07:28:46.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.102+0000 7f1c4c800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c440b2c40 0x7f1c4415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.102+0000 7f1c4c800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1c4406e2f0 0x7f1c440cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.102+0000 7f1c4c800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4406cba0 con 0x7f1c4406d960 2024-03-06T07:28:46.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.102+0000 7f1c4c800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c4415f700 con 0x7f1c4406e2f0 2024-03-06T07:28:46.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.102+0000 7f1c4c800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4406d4d0 con 0x7f1c440b2c40 2024-03-06T07:28:46.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.103+0000 7f1c49e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c440b2c40 0x7f1c4415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.103+0000 7f1c49e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c440b2c40 0x7f1c4415f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:55722/0 (socket says 172.21.15.114:55722) 2024-03-06T07:28:46.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.103+0000 7f1c49e00640 1 -- 172.21.15.114:0/433568704 learned_addr learned my addr 172.21.15.114:0/433568704 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:46.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.103+0000 7f1c49400640 1 --1- 172.21.15.114:0/433568704 >> v1:172.21.15.177:6789/0 conn(0x7f1c4406d960 0x7f1c440b3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:51312/0 (socket says 172.21.15.114:51312) 2024-03-06T07:28:46.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.103+0000 7f1c4a800640 1 --2- 172.21.15.114:0/433568704 >> v2:172.21.15.177:3300/0 conn(0x7f1c4406e2f0 0x7f1c440cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.104+0000 7f1c49e00640 1 -- 172.21.15.114:0/433568704 >> v2:172.21.15.177:3300/0 conn(0x7f1c4406e2f0 msgr2=0x7f1c440cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.104+0000 7f1c49e00640 1 --2- 172.21.15.114:0/433568704 >> v2:172.21.15.177:3300/0 conn(0x7f1c4406e2f0 0x7f1c440cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.104+0000 7f1c49e00640 1 -- 172.21.15.114:0/433568704 >> v1:172.21.15.177:6789/0 conn(0x7f1c4406d960 legacy=0x7f1c440b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.104+0000 7f1c49e00640 1 -- 172.21.15.114:0/433568704 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4406ce80 con 0x7f1c440b2c40 2024-03-06T07:28:46.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.104+0000 7f1c48a00640 1 -- 172.21.15.114:0/433568704 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030779556 0 0) 0x7f1c4406cba0 con 0x7f1c4406d960 2024-03-06T07:28:46.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.105+0000 7f1c49e00640 1 --2- 172.21.15.114:0/433568704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c440b2c40 0x7f1c4415f1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009870 tx=0x7f1c3802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593a43ab1b79b433 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.105+0000 7f1c48a00640 1 -- 172.21.15.114:0/433568704 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38034070 con 0x7f1c440b2c40 2024-03-06T07:28:46.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.105+0000 7f1c48a00640 1 -- 172.21.15.114:0/433568704 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c38030840 con 0x7f1c440b2c40 2024-03-06T07:28:46.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.105+0000 7f1c48a00640 1 -- 172.21.15.114:0/433568704 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38030c10 con 0x7f1c440b2c40 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 -- 172.21.15.114:0/433568704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c440b2c40 msgr2=0x7f1c4415f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 --2- 172.21.15.114:0/433568704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c440b2c40 0x7f1c4415f1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009870 tx=0x7f1c3802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 -- 172.21.15.114:0/433568704 shutdown_connections 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 --2- 172.21.15.114:0/433568704 >> v2:172.21.15.177:3300/0 conn(0x7f1c4406e2f0 0x7f1c440cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 --2- 172.21.15.114:0/433568704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c440b2c40 0x7f1c4415f1c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 -- 172.21.15.114:0/433568704 >> 172.21.15.114:0/433568704 conn(0x7f1c440b1940 msgr2=0x7f1c440b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 -- 172.21.15.114:0/433568704 shutdown_connections 2024-03-06T07:28:46.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.106+0000 7f1c4c800640 1 -- 172.21.15.114:0/433568704 wait complete. 2024-03-06T07:28:46.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.108+0000 7f1c4c800640 1 Processor -- start 2024-03-06T07:28:46.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.108+0000 7f1c4c800640 1 -- start start 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4c800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c4406d960 0x7f1c4409a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4c800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c4406e2f0 0x7f1c4409c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4c800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c440b2c40 0x7f1c4415f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4c800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4406e8d0 con 0x7f1c440b2c40 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4c800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4406e750 con 0x7f1c4406e2f0 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4c800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440cdfd0 con 0x7f1c4406d960 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c440b2c40 0x7f1c4415f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c440b2c40 0x7f1c4415f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52496/0 (socket says 172.21.15.114:52496) 2024-03-06T07:28:46.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c4a800640 1 -- 172.21.15.114:0/3797593649 learned_addr learned my addr 172.21.15.114:0/3797593649 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c49e00640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c4406d960 0x7f1c4409a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.109+0000 7f1c49400640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c4406e2f0 0x7f1c4409c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4a800640 1 -- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c4406d960 msgr2=0x7f1c4409a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4a800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c4406d960 0x7f1c4409a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4a800640 1 -- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c4406e2f0 msgr2=0x7f1c4409c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4a800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c4406e2f0 0x7f1c4409c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4a800640 1 -- 172.21.15.114:0/3797593649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4409b340 con 0x7f1c440b2c40 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c49e00640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c4406d960 0x7f1c4409a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4a800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c440b2c40 0x7f1c4415f990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009dd0 tx=0x7f1c4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c3e000640 1 -- 172.21.15.114:0/3797593649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40018020 con 0x7f1c440b2c40 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c3e000640 1 -- 172.21.15.114:0/3797593649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40014480 con 0x7f1c440b2c40 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c3e000640 1 -- 172.21.15.114:0/3797593649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40014870 con 0x7f1c440b2c40 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4409b620 con 0x7f1c440b2c40 2024-03-06T07:28:46.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.110+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4409bbe0 con 0x7f1c440b2c40 2024-03-06T07:28:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.111+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4404f330 con 0x7f1c440b2c40 2024-03-06T07:28:46.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.112+0000 7f1c3e000640 1 -- 172.21.15.114:0/3797593649 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c40005d10 con 0x7f1c440b2c40 2024-03-06T07:28:46.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.113+0000 7f1c3e000640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c2806b4d0 0x7f1c2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.113+0000 7f1c3e000640 1 -- 172.21.15.114:0/3797593649 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f1c4008af70 con 0x7f1c440b2c40 2024-03-06T07:28:46.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.114+0000 7f1c49e00640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c2806b4d0 0x7f1c2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.114+0000 7f1c49e00640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c2806b4d0 0x7f1c2806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c38036000 tx=0x7f1c38008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.118+0000 7f1c3e000640 1 -- 172.21.15.114:0/3797593649 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c40022070 con 0x7f1c440b2c40 2024-03-06T07:28:46.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.309+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1c440cda90 con 0x7f1c440b2c40 2024-03-06T07:28:46.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.310+0000 7f1c3e000640 1 -- 172.21.15.114:0/3797593649 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1c40010040 con 0x7f1c440b2c40 2024-03-06T07:28:46.312 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c2806b4d0 msgr2=0x7f1c2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c2806b4d0 0x7f1c2806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c38036000 tx=0x7f1c38008030 comp rx=0 tx=0).stop 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c440b2c40 msgr2=0x7f1c4415f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c440b2c40 0x7f1c4415f990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009dd0 tx=0x7f1c4000c490 comp rx=0 tx=0).stop 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 shutdown_connections 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c2806b4d0 0x7f1c2806d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c440b2c40 0x7f1c4415f990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c4406e2f0 0x7f1c4409c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 --2- 172.21.15.114:0/3797593649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c4406d960 0x7f1c4409a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 >> 172.21.15.114:0/3797593649 conn(0x7f1c440b1940 msgr2=0x7f1c4415d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 shutdown_connections 2024-03-06T07:28:46.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.312+0000 7f1c4c800640 1 -- 172.21.15.114:0/3797593649 wait complete. 2024-03-06T07:28:46.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:46.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.548+0000 7fb3a7200640 1 Processor -- start 2024-03-06T07:28:46.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.548+0000 7fb3a7200640 1 -- start start 2024-03-06T07:28:46.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.549+0000 7fb3a7200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb3a815be10 0x7fb3a80ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.549+0000 7fb3a7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3a80aef20 0x7fb3a80ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.549+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a806d330 0x7fb3a806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.549+0000 7fb3a7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a80cd200 con 0x7fb3a80aef20 2024-03-06T07:28:46.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.549+0000 7fb3a7200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a815c1f0 con 0x7fb3a815be10 2024-03-06T07:28:46.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.549+0000 7fb3a7200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a806cac0 con 0x7fb3a806d330 2024-03-06T07:28:46.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3a80aef20 0x7fb3a80ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb3a80aef20 0x7fb3a80ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52502/0 (socket says 172.21.15.114:52502) 2024-03-06T07:28:46.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 -- 172.21.15.114:0/187192958 learned_addr learned my addr 172.21.15.114:0/187192958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:46.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb3a6800640 1 --2- 172.21.15.114:0/187192958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a806d330 0x7fb3a806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb3a5e00640 1 --2- 172.21.15.114:0/187192958 >> v2:172.21.15.177:3300/0 conn(0x7fb3a815be10 0x7fb3a80ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 -- 172.21.15.114:0/187192958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a806d330 msgr2=0x7fb3a806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 --2- 172.21.15.114:0/187192958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a806d330 0x7fb3a806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 -- 172.21.15.114:0/187192958 >> v2:172.21.15.177:3300/0 conn(0x7fb3a815be10 msgr2=0x7fb3a80ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 --2- 172.21.15.114:0/187192958 >> v2:172.21.15.177:3300/0 conn(0x7fb3a815be10 0x7fb3a80ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb39fe00640 1 -- 172.21.15.114:0/187192958 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a806cda0 con 0x7fb3a80aef20 2024-03-06T07:28:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.550+0000 7fb3a6800640 1 --2- 172.21.15.114:0/187192958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a806d330 0x7fb3a806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.551+0000 7fb39fe00640 1 --2- 172.21.15.114:0/187192958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a80aef20 0x7fb3a80ce2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80d09c0 tx=0x7fb39002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784a52754dafba19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a5400640 1 -- 172.21.15.114:0/187192958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390035020 con 0x7fb3a80aef20 2024-03-06T07:28:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a5400640 1 -- 172.21.15.114:0/187192958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39002d040 con 0x7fb3a80aef20 2024-03-06T07:28:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a5400640 1 -- 172.21.15.114:0/187192958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390031a80 con 0x7fb3a80aef20 2024-03-06T07:28:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a7200640 1 -- 172.21.15.114:0/187192958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a80aef20 msgr2=0x7fb3a80ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a7200640 1 --2- 172.21.15.114:0/187192958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a80aef20 0x7fb3a80ce2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80d09c0 tx=0x7fb39002b900 comp rx=0 tx=0).stop 2024-03-06T07:28:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a7200640 1 -- 172.21.15.114:0/187192958 shutdown_connections 2024-03-06T07:28:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a7200640 1 --2- 172.21.15.114:0/187192958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a806d330 0x7fb3a806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a7200640 1 --2- 172.21.15.114:0/187192958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a80aef20 0x7fb3a80ce2b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a7200640 1 --2- 172.21.15.114:0/187192958 >> v2:172.21.15.177:3300/0 conn(0x7fb3a815be10 0x7fb3a80ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.552+0000 7fb3a7200640 1 -- 172.21.15.114:0/187192958 >> 172.21.15.114:0/187192958 conn(0x7fb3a806afc0 msgr2=0x7fb3a806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.553+0000 7fb3a7200640 1 -- 172.21.15.114:0/187192958 shutdown_connections 2024-03-06T07:28:46.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.553+0000 7fb3a7200640 1 -- 172.21.15.114:0/187192958 wait complete. 2024-03-06T07:28:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.555+0000 7fb3a7200640 1 Processor -- start 2024-03-06T07:28:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.555+0000 7fb3a7200640 1 -- start start 2024-03-06T07:28:46.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a806d330 0x7fb3a8158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a80aef20 0x7fb3a80b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a815be10 0x7fb3a8147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80cda90 con 0x7fb3a815be10 2024-03-06T07:28:46.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a806d080 con 0x7fb3a806d330 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a7200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8071ab0 con 0x7fb3a80aef20 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a815be10 0x7fb3a8147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a815be10 0x7fb3a8147210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52510/0 (socket says 172.21.15.114:52510) 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.557+0000 7fb3a6800640 1 -- 172.21.15.114:0/3112672830 learned_addr learned my addr 172.21.15.114:0/3112672830 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb39fe00640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a80aef20 0x7fb3a80b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a5e00640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a806d330 0x7fb3a8158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a6800640 1 -- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a80aef20 msgr2=0x7fb3a80b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a6800640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a80aef20 0x7fb3a80b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a6800640 1 -- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a806d330 msgr2=0x7fb3a8158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a6800640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a806d330 0x7fb3a8158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a6800640 1 -- 172.21.15.114:0/3112672830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8155df0 con 0x7fb3a815be10 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a5e00640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a806d330 0x7fb3a8158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb3a6800640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a815be10 0x7fb3a8147210 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0007e10 tx=0x7fb3a000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.558+0000 7fb39e000640 1 -- 172.21.15.114:0/3112672830 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a001a070 con 0x7fb3a815be10 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.559+0000 7fb39e000640 1 -- 172.21.15.114:0/3112672830 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a0003b90 con 0x7fb3a815be10 2024-03-06T07:28:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.559+0000 7fb39e000640 1 -- 172.21.15.114:0/3112672830 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0004030 con 0x7fb3a815be10 2024-03-06T07:28:46.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.559+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a81f42f0 con 0x7fb3a815be10 2024-03-06T07:28:46.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.559+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a81f48b0 con 0x7fb3a815be10 2024-03-06T07:28:46.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.560+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a804f330 con 0x7fb3a815be10 2024-03-06T07:28:46.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.561+0000 7fb39e000640 1 -- 172.21.15.114:0/3112672830 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3a001d070 con 0x7fb3a815be10 2024-03-06T07:28:46.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.562+0000 7fb39e000640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb38406b4d0 0x7fb38406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.562+0000 7fb39e000640 1 -- 172.21.15.114:0/3112672830 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb3a008a410 con 0x7fb3a815be10 2024-03-06T07:28:46.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.562+0000 7fb3a5e00640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb38406b4d0 0x7fb38406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.563+0000 7fb3a5e00640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb38406b4d0 0x7fb38406d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb398006fd0 tx=0x7fb398008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.567+0000 7fb39e000640 1 -- 172.21.15.114:0/3112672830 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3a0010070 con 0x7fb3a815be10 2024-03-06T07:28:46.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.765+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3a8157df0 con 0x7fb3a815be10 2024-03-06T07:28:46.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.766+0000 7fb39e000640 1 -- 172.21.15.114:0/3112672830 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3a0058c40 con 0x7fb3a815be10 2024-03-06T07:28:46.768 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb38406b4d0 msgr2=0x7fb38406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb38406b4d0 0x7fb38406d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb398006fd0 tx=0x7fb398008040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a815be10 msgr2=0x7fb3a8147210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a815be10 0x7fb3a8147210 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0007e10 tx=0x7fb3a000d4a0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 shutdown_connections 2024-03-06T07:28:46.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb38406b4d0 0x7fb38406d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3a815be10 0x7fb3a8147210 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb3a80aef20 0x7fb3a80b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 --2- 172.21.15.114:0/3112672830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3a806d330 0x7fb3a8158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 >> 172.21.15.114:0/3112672830 conn(0x7fb3a806afc0 msgr2=0x7fb3a80af890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 shutdown_connections 2024-03-06T07:28:46.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.768+0000 7fb3a7200640 1 -- 172.21.15.114:0/3112672830 wait complete. 2024-03-06T07:28:46.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:46.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.994+0000 7f1455800640 1 Processor -- start 2024-03-06T07:28:46.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.994+0000 7f1455800640 1 -- start start 2024-03-06T07:28:46.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f1455800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 0x7f14500c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f1455800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f14500c5880 0x7f1450165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f1455800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1450145b30 con 0x7f14500c4d10 2024-03-06T07:28:46.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f1455800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1450142ce0 con 0x7f14500c5880 2024-03-06T07:28:46.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f1455800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14500cbca0 con 0x7f14500c4930 2024-03-06T07:28:46.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f144e000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f14500c4d10 0x7f14500c5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52238/0 (socket says 172.21.15.114:52238) 2024-03-06T07:28:46.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f144e000640 1 -- 172.21.15.114:0/539299193 learned_addr learned my addr 172.21.15.114:0/539299193 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f144ea00640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 0x7f14500c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f144f400640 1 --2- 172.21.15.114:0/539299193 >> v2:172.21.15.177:3300/0 conn(0x7f14500c5880 0x7f1450165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.996+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219180607 0 0) 0x7f1438001490 con 0x7f14500c4d10 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1430003520 con 0x7f14500c4d10 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846609729 0 0) 0x7f1430003520 con 0x7f14500c4d10 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 msgr2=0x7f14500c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 0x7f14500c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 >> v2:172.21.15.177:3300/0 conn(0x7f14500c5880 msgr2=0x7f1450165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 --2- 172.21.15.114:0/539299193 >> v2:172.21.15.177:3300/0 conn(0x7f14500c5880 0x7f1450165b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500cbf80 con 0x7f14500c4d10 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f1438001bc0 con 0x7f14500c4d10 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 >> v1:172.21.15.114:6789/0 conn(0x7f14500c4d10 legacy=0x7f14500c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14300039b0 0x7f1430003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1430004800 0x7f1430004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 0x7f14300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1450142ce0 con 0x7f1430004800 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14500cbca0 con 0x7f14300039b0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143000b9c0 con 0x7f14500c4930 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.997+0000 7f144e000640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1430004800 0x7f1430004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.998+0000 7f144f400640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 0x7f14300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.998+0000 7f144ea00640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14300039b0 0x7f1430003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.998+0000 7f144e000640 1 -- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 msgr2=0x7f14300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.998+0000 7f144e000640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500c4930 0x7f14300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.998+0000 7f144e000640 1 -- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14300039b0 msgr2=0x7f1430003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.998+0000 7f144e000640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14300039b0 0x7f1430003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:46.998+0000 7f144e000640 1 -- 172.21.15.114:0/539299193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1430003520 con 0x7f1430004800 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.000+0000 7f144e000640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1430004800 0x7f1430004c80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f143800aaa0 tx=0x7f1430005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0d25890a02a723 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.000+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438038070 con 0x7f1430004800 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.000+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14380072f0 con 0x7f1430004800 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.000+0000 7f144d600640 1 -- 172.21.15.114:0/539299193 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143800a690 con 0x7f1430004800 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.000+0000 7f1455800640 1 -- 172.21.15.114:0/539299193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1430004800 msgr2=0x7f1430004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.000+0000 7f1455800640 1 --2- 172.21.15.114:0/539299193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1430004800 0x7f1430004c80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f143800aaa0 tx=0x7f1430005830 comp rx=0 tx=0).stop 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.000+0000 7f144f400640 1 -- 172.21.15.114:0/539299193 reap_dead start 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.001+0000 7f1455800640 1 -- 172.21.15.114:0/539299193 shutdown_connections 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.001+0000 7f1455800640 1 -- 172.21.15.114:0/539299193 >> 172.21.15.114:0/539299193 conn(0x7f14500cc4f0 msgr2=0x7f14500cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.001+0000 7f1455800640 1 -- 172.21.15.114:0/539299193 shutdown_connections 2024-03-06T07:28:47.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.001+0000 7f1455800640 1 -- 172.21.15.114:0/539299193 wait complete. 2024-03-06T07:28:47.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.002+0000 7f1455800640 1 Processor -- start 2024-03-06T07:28:47.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.003+0000 7f1455800640 1 -- start start 2024-03-06T07:28:47.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f1455800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1430004800 0x7f14501400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f1455800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14501e5840 0x7f14500aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f1455800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500b1470 0x7f14500b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f1455800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145006e600 con 0x7f14501e5840 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f1455800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14500ca3b0 con 0x7f1430004800 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f1455800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14500cc260 con 0x7f14500b1470 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14501e5840 0x7f14500aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14501e5840 0x7f14500aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52522/0 (socket says 172.21.15.114:52522) 2024-03-06T07:28:47.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 -- 172.21.15.114:0/3143597395 learned_addr learned my addr 172.21.15.114:0/3143597395 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144ea00640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1430004800 0x7f14501400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144f400640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500b1470 0x7f14500b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 -- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500b1470 msgr2=0x7f14500b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500b1470 0x7f14500b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 -- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1430004800 msgr2=0x7f14501400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1430004800 0x7f14501400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144e000640 1 -- 172.21.15.114:0/3143597395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14501405e0 con 0x7f14501e5840 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144ea00640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1430004800 0x7f14501400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.004+0000 7f144f400640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500b1470 0x7f14500b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.005+0000 7f144e000640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14501e5840 0x7f14500aff40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f143800aaa0 tx=0x7f145006f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.005+0000 7f1446a00640 1 -- 172.21.15.114:0/3143597395 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438038030 con 0x7f14501e5840 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.005+0000 7f1446a00640 1 -- 172.21.15.114:0/3143597395 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14380074a0 con 0x7f14501e5840 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.005+0000 7f1446a00640 1 -- 172.21.15.114:0/3143597395 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438007850 con 0x7f14501e5840 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.005+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500b08e0 con 0x7f14501e5840 2024-03-06T07:28:47.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.005+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14500c6f30 con 0x7f14501e5840 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.006+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145004f330 con 0x7f14501e5840 2024-03-06T07:28:47.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.007+0000 7f1446a00640 1 -- 172.21.15.114:0/3143597395 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1438033430 con 0x7f14501e5840 2024-03-06T07:28:47.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.009+0000 7f1446a00640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f142c06b500 0x7f142c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.009+0000 7f1446a00640 1 -- 172.21.15.114:0/3143597395 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f14380abe10 con 0x7f14501e5840 2024-03-06T07:28:47.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.009+0000 7f144ea00640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f142c06b500 0x7f142c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.010+0000 7f144ea00640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f142c06b500 0x7f142c06d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14400098b0 tx=0x7f1440008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.013+0000 7f1446a00640 1 -- 172.21.15.114:0/3143597395 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1450145b30 con 0x7f14501e5840 2024-03-06T07:28:47.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.222+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f14500b5a60 con 0x7f14501e5840 2024-03-06T07:28:47.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.223+0000 7f1446a00640 1 -- 172.21.15.114:0/3143597395 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f143807a6d0 con 0x7f14501e5840 2024-03-06T07:28:47.225 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:28:47.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f142c06b500 msgr2=0x7f142c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f142c06b500 0x7f142c06d9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14400098b0 tx=0x7f1440008040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14501e5840 msgr2=0x7f14500aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14501e5840 0x7f14500aff40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f143800aaa0 tx=0x7f145006f050 comp rx=0 tx=0).stop 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 shutdown_connections 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14500b1470 0x7f14500b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f142c06b500 0x7f142c06d9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14501e5840 0x7f14500aff40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.225+0000 7f1455800640 1 --2- 172.21.15.114:0/3143597395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1430004800 0x7f14501400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.226+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 >> 172.21.15.114:0/3143597395 conn(0x7f14500cc4f0 msgr2=0x7f14501e77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.226+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 shutdown_connections 2024-03-06T07:28:47.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.226+0000 7f1455800640 1 -- 172.21.15.114:0/3143597395 wait complete. 2024-03-06T07:28:47.247 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-06T07:28:47.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.247 DEBUG:teuthology.orchestra.run.smithi177:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-06T07:28:47.357 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi177.front.sepia.ceph.com 2024-03-06T07:28:47.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.357 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:47.402 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:47.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:47.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.613+0000 7f99f6600640 1 Processor -- start 2024-03-06T07:28:47.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.613+0000 7f99f6600640 1 -- start start 2024-03-06T07:28:47.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.614+0000 7f99f6600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f99f00aedc0 0x7f99f00cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99f6600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f015c240 con 0x7f99f00cc800 2024-03-06T07:28:47.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99f6600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f015c500 con 0x7f99f01561f0 2024-03-06T07:28:47.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99f6600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f015ce30 con 0x7f99f00aedc0 2024-03-06T07:28:47.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99f5c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f99f00cc800 0x7f99f015f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52246/0 (socket says 172.21.15.114:52246) 2024-03-06T07:28:47.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99f5c00640 1 -- 172.21.15.114:0/980311142 learned_addr learned my addr 172.21.15.114:0/980311142 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99efe00640 1 --2- 172.21.15.114:0/980311142 >> v2:172.21.15.177:3300/0 conn(0x7f99f00aedc0 0x7f99f00cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379529652 0 0) 0x7f99f015c240 con 0x7f99f00cc800 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.615+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d8003520 con 0x7f99f00cc800 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2920228989 0 0) 0x7f99d8003520 con 0x7f99f00cc800 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 >> v2:172.21.15.177:3300/0 conn(0x7f99f00aedc0 msgr2=0x7f99f00cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 --2- 172.21.15.114:0/980311142 >> v2:172.21.15.177:3300/0 conn(0x7f99f00aedc0 0x7f99f00cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 >> v1:172.21.15.177:6789/0 conn(0x7f99f01561f0 legacy=0x7f99f00ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f015c7e0 con 0x7f99f00cc800 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f99e80019f0 con 0x7f99f00cc800 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 >> v1:172.21.15.114:6789/0 conn(0x7f99f00cc800 legacy=0x7f99f015f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d80039b0 0x7f99d8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d8004800 0x7f99d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99d8005210 0x7f99d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f015ce30 con 0x7f99d80039b0 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d800c140 con 0x7f99d8005210 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d800c340 con 0x7f99d8004800 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.616+0000 7f99f5200640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d80039b0 0x7f99d8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99efe00640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d8004800 0x7f99d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5200640 1 -- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d8004800 msgr2=0x7f99d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5200640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d8004800 0x7f99d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5c00640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99d8005210 0x7f99d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5200640 1 -- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99d8005210 msgr2=0x7f99d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5200640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99d8005210 0x7f99d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5200640 1 -- 172.21.15.114:0/980311142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d8003520 con 0x7f99d80039b0 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99efe00640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d8004800 0x7f99d8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5c00640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99d8005210 0x7f99d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.617+0000 7f99f5c00640 1 -- 172.21.15.114:0/980311142 reap_dead start 2024-03-06T07:28:47.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.618+0000 7f99f5200640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d80039b0 0x7f99d8003df0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f99e4001750 tx=0x7f99d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7f0e9c0b34aea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.618+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e4001370 con 0x7f99d80039b0 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.618+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e40052d0 con 0x7f99d80039b0 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.618+0000 7f99ef400640 1 -- 172.21.15.114:0/980311142 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e4003830 con 0x7f99d80039b0 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.619+0000 7f99f6600640 1 -- 172.21.15.114:0/980311142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d80039b0 msgr2=0x7f99d8003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.619+0000 7f99f6600640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d80039b0 0x7f99d8003df0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f99e4001750 tx=0x7f99d8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.619+0000 7f99f6600640 1 -- 172.21.15.114:0/980311142 shutdown_connections 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.619+0000 7f99f6600640 1 --2- 172.21.15.114:0/980311142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99d80039b0 0x7f99d8003df0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.619+0000 7f99f6600640 1 -- 172.21.15.114:0/980311142 >> 172.21.15.114:0/980311142 conn(0x7f99f006afc0 msgr2=0x7f99f006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.619+0000 7f99f6600640 1 -- 172.21.15.114:0/980311142 shutdown_connections 2024-03-06T07:28:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.619+0000 7f99f6600640 1 -- 172.21.15.114:0/980311142 wait complete. 2024-03-06T07:28:47.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.621+0000 7f99f6600640 1 Processor -- start 2024-03-06T07:28:47.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.621+0000 7f99f6600640 1 -- start start 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99f6600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d80039b0 0x7f99f00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99f6600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f00cdb30 0x7f99f0154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99f6600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f0157be0 0x7f99f0071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99f6600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f01565d0 con 0x7f99f00cdb30 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99f6600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f00ccc30 con 0x7f99f0157be0 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99f6600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0156750 con 0x7f99d80039b0 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99efe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f00cdb30 0x7f99f0154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99efe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f00cdb30 0x7f99f0154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52542/0 (socket says 172.21.15.114:52542) 2024-03-06T07:28:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.622+0000 7f99efe00640 1 -- 172.21.15.114:0/2079217598 learned_addr learned my addr 172.21.15.114:0/2079217598 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99efe00640 1 -- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d80039b0 msgr2=0x7f99f00cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99f5200640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d80039b0 0x7f99f00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99f5c00640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f0157be0 0x7f99f0071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99efe00640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d80039b0 0x7f99f00cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99efe00640 1 -- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f0157be0 msgr2=0x7f99f0071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99efe00640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f0157be0 0x7f99f0071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99efe00640 1 -- 172.21.15.114:0/2079217598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0154780 con 0x7f99f00cdb30 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99f5200640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d80039b0 0x7f99f00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99f5c00640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f0157be0 0x7f99f0071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99efe00640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f00cdb30 0x7f99f0154240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e570 tx=0x7f99dc00ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99ecc00640 1 -- 172.21.15.114:0/2079217598 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc017070 con 0x7f99f00cdb30 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99ecc00640 1 -- 172.21.15.114:0/2079217598 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc013610 con 0x7f99f00cdb30 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99ecc00640 1 -- 172.21.15.114:0/2079217598 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc00f040 con 0x7f99f00cdb30 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.623+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f00ce010 con 0x7f99f00cdb30 2024-03-06T07:28:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.624+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f00ce5d0 con 0x7f99f00cdb30 2024-03-06T07:28:47.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.625+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f004f330 con 0x7f99f00cdb30 2024-03-06T07:28:47.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.626+0000 7f99ecc00640 1 -- 172.21.15.114:0/2079217598 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99dc01a070 con 0x7f99f00cdb30 2024-03-06T07:28:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.627+0000 7f99ecc00640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99d006b500 0x7f99d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.627+0000 7f99ecc00640 1 -- 172.21.15.114:0/2079217598 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f99dc089c20 con 0x7f99f00cdb30 2024-03-06T07:28:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.627+0000 7f99f5200640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99d006b500 0x7f99d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.628+0000 7f99f5200640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99d006b500 0x7f99d006d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f99e4033000 tx=0x7f99e4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.631+0000 7f99ecc00640 1 -- 172.21.15.114:0/2079217598 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99dc058970 con 0x7f99f00cdb30 2024-03-06T07:28:47.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.866+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99f00b2b10 con 0x7f99f00cdb30 2024-03-06T07:28:47.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.867+0000 7f99ecc00640 1 -- 172.21.15.114:0/2079217598 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f99dc058270 con 0x7f99f00cdb30 2024-03-06T07:28:47.870 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:47.870 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:47.870 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99d006b500 msgr2=0x7f99d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99d006b500 0x7f99d006d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f99e4033000 tx=0x7f99e4009030 comp rx=0 tx=0).stop 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f00cdb30 msgr2=0x7f99f0154240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f00cdb30 0x7f99f0154240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e570 tx=0x7f99dc00ea30 comp rx=0 tx=0).stop 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 shutdown_connections 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99f0157be0 0x7f99f0071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99d006b500 0x7f99d006d9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99f00cdb30 0x7f99f0154240 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 --2- 172.21.15.114:0/2079217598 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99d80039b0 0x7f99f00cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 >> 172.21.15.114:0/2079217598 conn(0x7f99f006afc0 msgr2=0x7f99f0071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 shutdown_connections 2024-03-06T07:28:47.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:47.870+0000 7f99f6600640 1 -- 172.21.15.114:0/2079217598 wait complete. 2024-03-06T07:28:47.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:48.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.082+0000 7fac1d600640 1 Processor -- start 2024-03-06T07:28:48.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.082+0000 7fac1d600640 1 -- start start 2024-03-06T07:28:48.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac1d600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac1815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac1d600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac18070b10 0x7fac18070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac1d600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1806cef0 con 0x7fac18144c00 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac1d600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac18160370 con 0x7fac18070b10 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac1d600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1806d1d0 con 0x7fac1815f9d0 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac17400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac18070b10 0x7fac18070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac17400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac18070b10 0x7fac18070f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51138/0 (socket says 172.21.15.114:51138) 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac17400640 1 -- 172.21.15.114:0/449315647 learned_addr learned my addr 172.21.15.114:0/449315647 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.084+0000 7fac0fe00640 1 --2- 172.21.15.114:0/449315647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac1815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.085+0000 7fac17400640 1 -- 172.21.15.114:0/449315647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 msgr2=0x7fac1815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.085+0000 7fac17400640 1 --2- 172.21.15.114:0/449315647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac1815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.085+0000 7fac17400640 1 -- 172.21.15.114:0/449315647 >> v1:172.21.15.177:6789/0 conn(0x7fac18144c00 legacy=0x7fac1806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.085+0000 7fac17400640 1 -- 172.21.15.114:0/449315647 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1806d4b0 con 0x7fac18070b10 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.086+0000 7fac17400640 1 --2- 172.21.15.114:0/449315647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac18070f60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fac10009e50 tx=0x7fac1002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1be642bee4e8e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.086+0000 7fac16000640 1 -- 172.21.15.114:0/449315647 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1002cb00 con 0x7fac18070b10 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.086+0000 7fac16000640 1 -- 172.21.15.114:0/449315647 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1002cca0 con 0x7fac18070b10 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.086+0000 7fac16000640 1 -- 172.21.15.114:0/449315647 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10034610 con 0x7fac18070b10 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.086+0000 7fac1d600640 1 -- 172.21.15.114:0/449315647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 msgr2=0x7fac18070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.086+0000 7fac1d600640 1 --2- 172.21.15.114:0/449315647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac18070f60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fac10009e50 tx=0x7fac1002bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.087+0000 7fac1d600640 1 -- 172.21.15.114:0/449315647 shutdown_connections 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.087+0000 7fac1d600640 1 --2- 172.21.15.114:0/449315647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac18070f60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.087+0000 7fac1d600640 1 --2- 172.21.15.114:0/449315647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac1815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.087+0000 7fac1d600640 1 -- 172.21.15.114:0/449315647 >> 172.21.15.114:0/449315647 conn(0x7fac180b1730 msgr2=0x7fac180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.087+0000 7fac1d600640 1 -- 172.21.15.114:0/449315647 shutdown_connections 2024-03-06T07:28:48.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.087+0000 7fac1d600640 1 -- 172.21.15.114:0/449315647 wait complete. 2024-03-06T07:28:48.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.088+0000 7fac1d600640 1 Processor -- start 2024-03-06T07:28:48.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.088+0000 7fac1d600640 1 -- start start 2024-03-06T07:28:48.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.089+0000 7fac1d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac180b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.089+0000 7fac1d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac18144c00 0x7fac180b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.089+0000 7fac1d600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac181f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.089+0000 7fac1d600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1806cb50 con 0x7fac18070b10 2024-03-06T07:28:48.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.089+0000 7fac1d600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac18160730 con 0x7fac18144c00 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.089+0000 7fac1d600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1806ccd0 con 0x7fac1815f9d0 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.089+0000 7fac16a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac180b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac180b43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51146/0 (socket says 172.21.15.114:51146) 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 -- 172.21.15.114:0/3570517121 learned_addr learned my addr 172.21.15.114:0/3570517121 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac0fe00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac18144c00 0x7fac180b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 -- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 msgr2=0x7fac181f4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac17400640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac181f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac181f4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 -- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac18144c00 msgr2=0x7fac180b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac18144c00 0x7fac180b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 -- 172.21.15.114:0/3570517121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac181f79e0 con 0x7fac18070b10 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac0fe00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac18144c00 0x7fac180b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac17400640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac181f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.090+0000 7fac16a00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac180b43a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c450 tx=0x7fac0800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.091+0000 7fac0ea00640 1 -- 172.21.15.114:0/3570517121 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac080052d0 con 0x7fac18070b10 2024-03-06T07:28:48.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.091+0000 7fac0ea00640 1 -- 172.21.15.114:0/3570517121 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac08005470 con 0x7fac18070b10 2024-03-06T07:28:48.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.091+0000 7fac0ea00640 1 -- 172.21.15.114:0/3570517121 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac08012870 con 0x7fac18070b10 2024-03-06T07:28:48.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.091+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac181f7cc0 con 0x7fac18070b10 2024-03-06T07:28:48.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.091+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac181f8280 con 0x7fac18070b10 2024-03-06T07:28:48.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.092+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1804f330 con 0x7fac18070b10 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.094+0000 7fac0ea00640 1 -- 172.21.15.114:0/3570517121 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac08012a10 con 0x7fac18070b10 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.094+0000 7fac0ea00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabf406b4d0 0x7fabf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.094+0000 7fac0ea00640 1 -- 172.21.15.114:0/3570517121 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fac08089e40 con 0x7fac18070b10 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.095+0000 7fac0fe00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabf406b4d0 0x7fabf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.095+0000 7fac0fe00640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabf406b4d0 0x7fabf406d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac180af0a0 tx=0x7fac00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.099+0000 7fac0ea00640 1 -- 172.21.15.114:0/3570517121 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac08058cc0 con 0x7fac18070b10 2024-03-06T07:28:48.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.289+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac18072890 con 0x7fac18070b10 2024-03-06T07:28:48.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.291+0000 7fac0ea00640 1 -- 172.21.15.114:0/3570517121 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16950 (secure 0 0 0) 0x7fac080585c0 con 0x7fac18070b10 2024-03-06T07:28:48.294 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":22,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:36.265733+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabf406b4d0 msgr2=0x7fabf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabf406b4d0 0x7fabf406d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac180af0a0 tx=0x7fac00008040 comp rx=0 tx=0).stop 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 msgr2=0x7fac180b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac180b43a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c450 tx=0x7fac0800c910 comp rx=0 tx=0).stop 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 shutdown_connections 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac1815f9d0 0x7fac181f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac18144c00 0x7fac180b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabf406b4d0 0x7fabf406d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 --2- 172.21.15.114:0/3570517121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac18070b10 0x7fac180b43a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 >> 172.21.15.114:0/3570517121 conn(0x7fac180b1730 msgr2=0x7fac180af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 shutdown_connections 2024-03-06T07:28:48.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.293+0000 7fac1d600640 1 -- 172.21.15.114:0/3570517121 wait complete. 2024-03-06T07:28:48.313 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:52.531713+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:53.453989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:48.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:48.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.512+0000 7f65ef200640 1 Processor -- start 2024-03-06T07:28:48.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.512+0000 7f65ef200640 1 -- start start 2024-03-06T07:28:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.514+0000 7f65ef200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65e8160700 0x7f65e8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.514+0000 7f65ef200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80b46c0 con 0x7f65e80b3f70 2024-03-06T07:28:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.514+0000 7f65ef200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80b49a0 con 0x7f65e815fc10 2024-03-06T07:28:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.514+0000 7f65ef200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8072c90 con 0x7f65e8160700 2024-03-06T07:28:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.514+0000 7f65e7400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f65e80b3f70 0x7f65e80ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56876/0 (socket says 172.21.15.114:56876) 2024-03-06T07:28:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.514+0000 7f65e7400640 1 -- 172.21.15.114:0/1206045920 learned_addr learned my addr 172.21.15.114:0/1206045920 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65ed200640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65e8160700 0x7f65e8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960578242 0 0) 0x7f65e80b46c0 con 0x7f65e80b3f70 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65cc003520 con 0x7f65e80b3f70 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1221094086 0 0) 0x7f65cc003520 con 0x7f65e80b3f70 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65e8160700 msgr2=0x7f65e8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65e8160700 0x7f65e8070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 >> v1:172.21.15.177:6789/0 conn(0x7f65e815fc10 legacy=0x7f65e815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e809dd80 con 0x7f65e80b3f70 2024-03-06T07:28:48.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f65d0001670 con 0x7f65e80b3f70 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 >> v1:172.21.15.114:6789/0 conn(0x7f65e80b3f70 legacy=0x7f65e80ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65cc0039e0 0x7f65cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65e8160700 0x7f65cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8072c90 con 0x7f65cc004800 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc00ba70 con 0x7f65cc0039e0 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.515+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc00bc20 con 0x7f65e8160700 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.516+0000 7f65e7400640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.516+0000 7f65e7e00640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65cc0039e0 0x7f65cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.516+0000 7f65e7400640 1 -- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65e8160700 msgr2=0x7f65cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.516+0000 7f65e7400640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65e8160700 0x7f65cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.516+0000 7f65e7400640 1 -- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65cc0039e0 msgr2=0x7f65cc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.516+0000 7f65e7400640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65cc0039e0 0x7f65cc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.516+0000 7f65e7400640 1 -- 172.21.15.114:0/1206045920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc003520 con 0x7f65cc004800 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.517+0000 7f65e7400640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f65d0001eb0 tx=0x7f65cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f35fd4ba6397da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.517+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0007c80 con 0x7f65cc004800 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.517+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0030070 con 0x7f65cc004800 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.517+0000 7f65e6a00640 1 -- 172.21.15.114:0/1206045920 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0005d30 con 0x7f65cc004800 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.518+0000 7f65ef200640 1 -- 172.21.15.114:0/1206045920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65cc004800 msgr2=0x7f65cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.518+0000 7f65ef200640 1 --2- 172.21.15.114:0/1206045920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f65d0001eb0 tx=0x7f65cc005830 comp rx=0 tx=0).stop 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.518+0000 7f65ed200640 1 -- 172.21.15.114:0/1206045920 reap_dead start 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.518+0000 7f65ef200640 1 -- 172.21.15.114:0/1206045920 shutdown_connections 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.518+0000 7f65ef200640 1 -- 172.21.15.114:0/1206045920 >> 172.21.15.114:0/1206045920 conn(0x7f65e80b1940 msgr2=0x7f65e80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.518+0000 7f65ef200640 1 -- 172.21.15.114:0/1206045920 shutdown_connections 2024-03-06T07:28:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.518+0000 7f65ef200640 1 -- 172.21.15.114:0/1206045920 wait complete. 2024-03-06T07:28:48.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.520+0000 7f65ef200640 1 Processor -- start 2024-03-06T07:28:48.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.520+0000 7f65ef200640 1 -- start start 2024-03-06T07:28:48.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65ef200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65cc004800 0x7f65e809d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65ef200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e81585b0 0x7f65e81462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65ef200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e8158a30 0x7f65e8146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65ef200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e815c460 con 0x7f65e81585b0 2024-03-06T07:28:48.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65ef200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e815c2e0 con 0x7f65e8158a30 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65ef200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8070ff0 con 0x7f65cc004800 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e81585b0 0x7f65e81462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e81585b0 0x7f65e81462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51164/0 (socket says 172.21.15.114:51164) 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 -- 172.21.15.114:0/3022695017 learned_addr learned my addr 172.21.15.114:0/3022695017 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7e00640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65cc004800 0x7f65e809d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65ed200640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e8158a30 0x7f65e8146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 -- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65cc004800 msgr2=0x7f65e809d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65cc004800 0x7f65e809d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 -- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e8158a30 msgr2=0x7f65e8146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e8158a30 0x7f65e8146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.521+0000 7f65e7400640 1 -- 172.21.15.114:0/3022695017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8157510 con 0x7f65e81585b0 2024-03-06T07:28:48.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.522+0000 7f65e7400640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e81585b0 0x7f65e81462c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f65d000c220 tx=0x7f65d0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.522+0000 7f65dfe00640 1 -- 172.21.15.114:0/3022695017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0045070 con 0x7f65e81585b0 2024-03-06T07:28:48.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.522+0000 7f65dfe00640 1 -- 172.21.15.114:0/3022695017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d002cbc0 con 0x7f65e81585b0 2024-03-06T07:28:48.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.522+0000 7f65dfe00640 1 -- 172.21.15.114:0/3022695017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0007c60 con 0x7f65e81585b0 2024-03-06T07:28:48.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.522+0000 7f65ef200640 1 -- 172.21.15.114:0/3022695017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e81577f0 con 0x7f65e81585b0 2024-03-06T07:28:48.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.522+0000 7f65ef200640 1 -- 172.21.15.114:0/3022695017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e8157db0 con 0x7f65e81585b0 2024-03-06T07:28:48.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.523+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e804f330 con 0x7f65e81585b0 2024-03-06T07:28:48.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.524+0000 7f65dfe00640 1 -- 172.21.15.114:0/3022695017 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65d0007070 con 0x7f65e81585b0 2024-03-06T07:28:48.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.525+0000 7f65dfe00640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f65c406b500 0x7f65c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.525+0000 7f65dfe00640 1 -- 172.21.15.114:0/3022695017 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f65d00ab450 con 0x7f65e81585b0 2024-03-06T07:28:48.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.525+0000 7f65e7e00640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f65c406b500 0x7f65c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.526+0000 7f65e7e00640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f65c406b500 0x7f65c406d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f65d8007fd0 tx=0x7f65d8003970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.530+0000 7f65dfe00640 1 -- 172.21.15.114:0/3022695017 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65d0079b50 con 0x7f65e81585b0 2024-03-06T07:28:48.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.835+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65e806ef30 con 0x7f65e81585b0 2024-03-06T07:28:48.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.836+0000 7f65dfe00640 1 -- 172.21.15.114:0/3022695017 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65d007dbf0 con 0x7f65e81585b0 2024-03-06T07:28:48.839 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:28:48.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f65c406b500 msgr2=0x7f65c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f65c406b500 0x7f65c406d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f65d8007fd0 tx=0x7f65d8003970 comp rx=0 tx=0).stop 2024-03-06T07:28:48.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e81585b0 msgr2=0x7f65e81462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e81585b0 0x7f65e81462c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f65d000c220 tx=0x7f65d0001970 comp rx=0 tx=0).stop 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 shutdown_connections 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65e8158a30 0x7f65e8146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f65c406b500 0x7f65c406d9c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e81585b0 0x7f65e81462c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 --2- 172.21.15.114:0/3022695017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65cc004800 0x7f65e809d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.839+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 >> 172.21.15.114:0/3022695017 conn(0x7f65e80b1940 msgr2=0x7f65e80af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.840+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 shutdown_connections 2024-03-06T07:28:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:48.840+0000 7f65dd600640 1 -- 172.21.15.114:0/3022695017 wait complete. 2024-03-06T07:28:48.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:49.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.055+0000 7ff943000640 1 Processor -- start 2024-03-06T07:28:49.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.055+0000 7ff943000640 1 -- start start 2024-03-06T07:28:49.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.057+0000 7ff943000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff93c0b4b90 0x7ff93c0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.057+0000 7ff943000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d1db0 0x7ff93c0d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.057+0000 7ff943000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93c0cf2f0 con 0x7ff93c0d06d0 2024-03-06T07:28:49.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.057+0000 7ff943000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93c0cf5d0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.057+0000 7ff943000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0cf8b0 con 0x7ff93c0d1db0 2024-03-06T07:28:49.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.058+0000 7ff942600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d1db0 0x7ff93c0d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.058+0000 7ff942600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d1db0 0x7ff93c0d2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:54236/0 (socket says 172.21.15.114:54236) 2024-03-06T07:28:49.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.058+0000 7ff942600640 1 -- 172.21.15.114:0/1881139571 learned_addr learned my addr 172.21.15.114:0/1881139571 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:49.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.058+0000 7ff941200640 1 --2- 172.21.15.114:0/1881139571 >> v2:172.21.15.177:3300/0 conn(0x7ff93c0b4b90 0x7ff93c0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.059+0000 7ff942600640 1 -- 172.21.15.114:0/1881139571 >> v2:172.21.15.177:3300/0 conn(0x7ff93c0b4b90 msgr2=0x7ff93c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.059+0000 7ff942600640 1 --2- 172.21.15.114:0/1881139571 >> v2:172.21.15.177:3300/0 conn(0x7ff93c0b4b90 0x7ff93c0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.059+0000 7ff942600640 1 -- 172.21.15.114:0/1881139571 >> v1:172.21.15.177:6789/0 conn(0x7ff93c0d06d0 legacy=0x7ff93c0d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.059+0000 7ff942600640 1 -- 172.21.15.114:0/1881139571 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c0cfb90 con 0x7ff93c0d1db0 2024-03-06T07:28:49.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.060+0000 7ff942600640 1 --2- 172.21.15.114:0/1881139571 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d1db0 0x7ff93c0d2240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff934009e50 tx=0x7ff93402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90281c36320a2905 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.061+0000 7ff93be00640 1 -- 172.21.15.114:0/1881139571 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93402cb00 con 0x7ff93c0d1db0 2024-03-06T07:28:49.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.061+0000 7ff93be00640 1 -- 172.21.15.114:0/1881139571 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93402cca0 con 0x7ff93c0d1db0 2024-03-06T07:28:49.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.061+0000 7ff93be00640 1 -- 172.21.15.114:0/1881139571 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934034610 con 0x7ff93c0d1db0 2024-03-06T07:28:49.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.061+0000 7ff943000640 1 -- 172.21.15.114:0/1881139571 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d1db0 msgr2=0x7ff93c0d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.061+0000 7ff943000640 1 --2- 172.21.15.114:0/1881139571 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d1db0 0x7ff93c0d2240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff934009e50 tx=0x7ff93402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.062+0000 7ff943000640 1 -- 172.21.15.114:0/1881139571 shutdown_connections 2024-03-06T07:28:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.062+0000 7ff943000640 1 --2- 172.21.15.114:0/1881139571 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d1db0 0x7ff93c0d2240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.062+0000 7ff943000640 1 --2- 172.21.15.114:0/1881139571 >> v2:172.21.15.177:3300/0 conn(0x7ff93c0b4b90 0x7ff93c0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.062+0000 7ff943000640 1 -- 172.21.15.114:0/1881139571 >> 172.21.15.114:0/1881139571 conn(0x7ff93c09d070 msgr2=0x7ff93c0d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.062+0000 7ff943000640 1 -- 172.21.15.114:0/1881139571 shutdown_connections 2024-03-06T07:28:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.062+0000 7ff943000640 1 -- 172.21.15.114:0/1881139571 wait complete. 2024-03-06T07:28:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.064+0000 7ff943000640 1 Processor -- start 2024-03-06T07:28:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.065+0000 7ff943000640 1 -- start start 2024-03-06T07:28:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.066+0000 7ff943000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c0b4b90 0x7ff93c1622d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.066+0000 7ff943000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d06d0 0x7ff93c158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.066+0000 7ff943000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff93c0d1db0 0x7ff93c0b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.066+0000 7ff943000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0cf0d0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.066+0000 7ff943000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0cef50 con 0x7ff93c0d1db0 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.066+0000 7ff943000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0d2780 con 0x7ff93c0d06d0 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.066+0000 7ff941c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c0b4b90 0x7ff93c1622d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c0b4b90 0x7ff93c1622d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51170/0 (socket says 172.21.15.114:51170) 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 -- 172.21.15.114:0/2326215194 learned_addr learned my addr 172.21.15.114:0/2326215194 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941200640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d06d0 0x7ff93c158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff942600640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff93c0d1db0 0x7ff93c0b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 -- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d06d0 msgr2=0x7ff93c158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d06d0 0x7ff93c158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 -- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff93c0d1db0 msgr2=0x7ff93c0b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff93c0d1db0 0x7ff93c0b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 -- 172.21.15.114:0/2326215194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c161230 con 0x7ff93c0b4b90 2024-03-06T07:28:49.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.067+0000 7ff941c00640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c0b4b90 0x7ff93c1622d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c450 tx=0x7ff93000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.068+0000 7ff939600640 1 -- 172.21.15.114:0/2326215194 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9300052d0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.068+0000 7ff939600640 1 -- 172.21.15.114:0/2326215194 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff930005470 con 0x7ff93c0b4b90 2024-03-06T07:28:49.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.068+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93c161510 con 0x7ff93c0b4b90 2024-03-06T07:28:49.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.068+0000 7ff939600640 1 -- 172.21.15.114:0/2326215194 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930012870 con 0x7ff93c0b4b90 2024-03-06T07:28:49.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.068+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93c161db0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.069+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93c1616e0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.070+0000 7ff939600640 1 -- 172.21.15.114:0/2326215194 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff930012a10 con 0x7ff93c0b4b90 2024-03-06T07:28:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.071+0000 7ff939600640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.071+0000 7ff939600640 1 -- 172.21.15.114:0/2326215194 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ff930089ee0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.071+0000 7ff941200640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.072+0000 7ff941200640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff91c06b4d0 0x7ff91c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff93c160a60 tx=0x7ff928008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.075+0000 7ff939600640 1 -- 172.21.15.114:0/2326215194 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff930058cc0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.270+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff93c04f330 con 0x7ff93c0b4b90 2024-03-06T07:28:49.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.271+0000 7ff939600640 1 -- 172.21.15.114:0/2326215194 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9300585c0 con 0x7ff93c0b4b90 2024-03-06T07:28:49.274 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:28:49.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff91c06b4d0 msgr2=0x7ff91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff91c06b4d0 0x7ff91c06d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff93c160a60 tx=0x7ff928008040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c0b4b90 msgr2=0x7ff93c1622d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c0b4b90 0x7ff93c1622d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c450 tx=0x7ff93000c910 comp rx=0 tx=0).stop 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 shutdown_connections 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff93c0d1db0 0x7ff93c0b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff93c0d06d0 0x7ff93c158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 --2- 172.21.15.114:0/2326215194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93c0b4b90 0x7ff93c1622d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.273+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 >> 172.21.15.114:0/2326215194 conn(0x7ff93c09d070 msgr2=0x7ff93c15dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.274+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 shutdown_connections 2024-03-06T07:28:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.274+0000 7ff943000640 1 -- 172.21.15.114:0/2326215194 wait complete. 2024-03-06T07:28:49.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.495+0000 7fd84d400640 1 Processor -- start 2024-03-06T07:28:49.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.496+0000 7fd84d400640 1 -- start start 2024-03-06T07:28:49.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd84d400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd84815e000 0x7fd84806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd84d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd84806fef0 0x7fd8480d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd84d400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8480d1260 con 0x7fd84815e3e0 2024-03-06T07:28:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd84d400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8480d1020 con 0x7fd84806fef0 2024-03-06T07:28:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd84d400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84806c0d0 con 0x7fd84815e000 2024-03-06T07:28:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd847200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd84806fef0 0x7fd8480d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd847200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd84806fef0 0x7fd8480d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51186/0 (socket says 172.21.15.114:51186) 2024-03-06T07:28:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd847200640 1 -- 172.21.15.114:0/280485972 learned_addr learned my addr 172.21.15.114:0/280485972 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd847200640 1 -- 172.21.15.114:0/280485972 >> v2:172.21.15.177:3300/0 conn(0x7fd84815e000 msgr2=0x7fd84806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.497+0000 7fd847200640 1 --2- 172.21.15.114:0/280485972 >> v2:172.21.15.177:3300/0 conn(0x7fd84815e000 0x7fd84806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.498+0000 7fd847200640 1 -- 172.21.15.114:0/280485972 >> v1:172.21.15.177:6789/0 conn(0x7fd84815e3e0 legacy=0x7fd84806f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.498+0000 7fd847200640 1 -- 172.21.15.114:0/280485972 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84806c320 con 0x7fd84806fef0 2024-03-06T07:28:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.499+0000 7fd847200640 1 --2- 172.21.15.114:0/280485972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84806fef0 0x7fd8480d0ae0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd848071b20 tx=0x7fd84002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27249ac90c05c30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.499+0000 7fd845e00640 1 -- 172.21.15.114:0/280485972 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84002ca80 con 0x7fd84806fef0 2024-03-06T07:28:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.499+0000 7fd845e00640 1 -- 172.21.15.114:0/280485972 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84002cc20 con 0x7fd84806fef0 2024-03-06T07:28:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.499+0000 7fd845e00640 1 -- 172.21.15.114:0/280485972 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8400343f0 con 0x7fd84806fef0 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.499+0000 7fd84d400640 1 -- 172.21.15.114:0/280485972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84806fef0 msgr2=0x7fd8480d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.499+0000 7fd84d400640 1 --2- 172.21.15.114:0/280485972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84806fef0 0x7fd8480d0ae0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd848071b20 tx=0x7fd84002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.500+0000 7fd84d400640 1 -- 172.21.15.114:0/280485972 shutdown_connections 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.500+0000 7fd84d400640 1 --2- 172.21.15.114:0/280485972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84806fef0 0x7fd8480d0ae0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.500+0000 7fd84d400640 1 --2- 172.21.15.114:0/280485972 >> v2:172.21.15.177:3300/0 conn(0x7fd84815e000 0x7fd84806f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.500+0000 7fd84d400640 1 -- 172.21.15.114:0/280485972 >> 172.21.15.114:0/280485972 conn(0x7fd8480b3800 msgr2=0x7fd8480b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.500+0000 7fd84d400640 1 -- 172.21.15.114:0/280485972 shutdown_connections 2024-03-06T07:28:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.500+0000 7fd84d400640 1 -- 172.21.15.114:0/280485972 wait complete. 2024-03-06T07:28:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.501+0000 7fd84d400640 1 Processor -- start 2024-03-06T07:28:49.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.502+0000 7fd84d400640 1 -- start start 2024-03-06T07:28:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.502+0000 7fd84d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd84806fef0 0x7fd84814a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.502+0000 7fd84d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84815e000 0x7fd8480b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.502+0000 7fd84d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd84815e3e0 0x7fd84806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd84d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480ce4c0 con 0x7fd84815e000 2024-03-06T07:28:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd84d400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480ce340 con 0x7fd84806fef0 2024-03-06T07:28:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd84d400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480ce640 con 0x7fd84815e3e0 2024-03-06T07:28:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84815e000 0x7fd8480b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84815e000 0x7fd8480b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51196/0 (socket says 172.21.15.114:51196) 2024-03-06T07:28:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 -- 172.21.15.114:0/2097158930 learned_addr learned my addr 172.21.15.114:0/2097158930 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd846800640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd84806fef0 0x7fd84814a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 -- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd84815e3e0 msgr2=0x7fd84806ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd847200640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd84815e3e0 0x7fd84806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd84815e3e0 0x7fd84806ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 -- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd84806fef0 msgr2=0x7fd84814a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd84806fef0 0x7fd84814a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd83fe00640 1 -- 172.21.15.114:0/2097158930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8480b7920 con 0x7fd84815e000 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd846800640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd84806fef0 0x7fd84814a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.503+0000 7fd847200640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd84815e3e0 0x7fd84806ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.504+0000 7fd83fe00640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84815e000 0x7fd8480b6cf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd83000b3f0 tx=0x7fd83000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.504+0000 7fd83ea00640 1 -- 172.21.15.114:0/2097158930 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830015020 con 0x7fd84815e000 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.504+0000 7fd83ea00640 1 -- 172.21.15.114:0/2097158930 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd830011830 con 0x7fd84815e000 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.504+0000 7fd83ea00640 1 -- 172.21.15.114:0/2097158930 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830011c20 con 0x7fd84815e000 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.504+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8480b7640 con 0x7fd84815e000 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.504+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84815feb0 con 0x7fd84815e000 2024-03-06T07:28:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.505+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8480cfbe0 con 0x7fd84815e000 2024-03-06T07:28:49.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.507+0000 7fd83ea00640 1 -- 172.21.15.114:0/2097158930 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd830018020 con 0x7fd84815e000 2024-03-06T07:28:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.508+0000 7fd83ea00640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd82406b4d0 0x7fd82406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.508+0000 7fd83ea00640 1 -- 172.21.15.114:0/2097158930 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd830089f50 con 0x7fd84815e000 2024-03-06T07:28:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.508+0000 7fd846800640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd82406b4d0 0x7fd82406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.509+0000 7fd846800640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd82406b4d0 0x7fd82406d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd838006fd0 tx=0x7fd838008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.512+0000 7fd83ea00640 1 -- 172.21.15.114:0/2097158930 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8480cfbe0 con 0x7fd84815e000 2024-03-06T07:28:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.903+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd84815dbb0 con 0x7fd84815e000 2024-03-06T07:28:49.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.904+0000 7fd83ea00640 1 -- 172.21.15.114:0/2097158930 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd84815dbb0 con 0x7fd84815e000 2024-03-06T07:28:49.906 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:28:49.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd82406b4d0 msgr2=0x7fd82406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd82406b4d0 0x7fd82406d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd838006fd0 tx=0x7fd838008040 comp rx=0 tx=0).stop 2024-03-06T07:28:49.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84815e000 msgr2=0x7fd8480b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84815e000 0x7fd8480b6cf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd83000b3f0 tx=0x7fd83000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 shutdown_connections 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd84815e3e0 0x7fd84806ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd82406b4d0 0x7fd82406d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84815e000 0x7fd8480b6cf0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 --2- 172.21.15.114:0/2097158930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd84806fef0 0x7fd84814a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 >> 172.21.15.114:0/2097158930 conn(0x7fd8480b3800 msgr2=0x7fd84815a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 shutdown_connections 2024-03-06T07:28:49.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:49.907+0000 7fd84d400640 1 -- 172.21.15.114:0/2097158930 wait complete. 2024-03-06T07:28:49.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:50.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.128+0000 7f6bc9e00640 1 Processor -- start 2024-03-06T07:28:50.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.128+0000 7f6bc9e00640 1 -- start start 2024-03-06T07:28:50.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bc9e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 0x7f6bc406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bc9e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6bc4072340 0x7f6bc406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bc9e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc415b090 con 0x7f6bc415acb0 2024-03-06T07:28:50.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bc9e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc40b2a30 con 0x7f6bc4072340 2024-03-06T07:28:50.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bc9e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b4360 con 0x7f6bc40b4680 2024-03-06T07:28:50.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bc2a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6bc415acb0 0x7f6bc40b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56886/0 (socket says 172.21.15.114:56886) 2024-03-06T07:28:50.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bc2a00640 1 -- 172.21.15.114:0/1641808315 learned_addr learned my addr 172.21.15.114:0/1641808315 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:50.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.131+0000 7f6bbbe00640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 0x7f6bc406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1641808315 >> v2:172.21.15.177:3300/0 conn(0x7f6bc4072340 0x7f6bc406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310607448 0 0) 0x7f6bc415b090 con 0x7f6bc415acb0 2024-03-06T07:28:50.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba8003520 con 0x7f6bc415acb0 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3233286541 0 0) 0x7f6ba8003520 con 0x7f6bc415acb0 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 msgr2=0x7f6bc406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 0x7f6bc406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 >> v2:172.21.15.177:3300/0 conn(0x7f6bc4072340 msgr2=0x7f6bc406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 --2- 172.21.15.114:0/1641808315 >> v2:172.21.15.177:3300/0 conn(0x7f6bc4072340 0x7f6bc406ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc40b2cf0 con 0x7f6bc415acb0 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6bb4001670 con 0x7f6bc415acb0 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 >> v1:172.21.15.114:6789/0 conn(0x7f6bc415acb0 legacy=0x7f6bc40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 0x7f6ba80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8003f80 0x7f6ba8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ba8004d40 0x7f6ba80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b2a30 con 0x7f6ba8004d40 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b4360 con 0x7f6ba8003f80 2024-03-06T07:28:50.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.132+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba800b9c0 con 0x7f6bc40b4680 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ba8004d40 0x7f6ba80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bc2a00640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 0x7f6ba80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bbbe00640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8003f80 0x7f6ba8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bc3400640 1 -- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 msgr2=0x7f6ba80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc40b4680 0x7f6ba80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bc3400640 1 -- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8003f80 msgr2=0x7f6ba8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8003f80 0x7f6ba8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bc3400640 1 -- 172.21.15.114:0/1641808315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba8003520 con 0x7f6ba8004d40 2024-03-06T07:28:50.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.133+0000 7f6bbbe00640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8003f80 0x7f6ba8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.134+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ba8004d40 0x7f6ba80051c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00e170 tx=0x7f6ba8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78124d2592c640bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.134+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc039070 con 0x7f6ba8004d40 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.134+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc02cc30 con 0x7f6ba8004d40 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc2000640 1 -- 172.21.15.114:0/1641808315 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc009c10 con 0x7f6ba8004d40 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1641808315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ba8004d40 msgr2=0x7f6ba80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc9e00640 1 --2- 172.21.15.114:0/1641808315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ba8004d40 0x7f6ba80051c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00e170 tx=0x7f6ba8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc3400640 1 -- 172.21.15.114:0/1641808315 reap_dead start 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1641808315 shutdown_connections 2024-03-06T07:28:50.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1641808315 >> 172.21.15.114:0/1641808315 conn(0x7f6bc40b1730 msgr2=0x7f6bc40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1641808315 shutdown_connections 2024-03-06T07:28:50.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.135+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1641808315 wait complete. 2024-03-06T07:28:50.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.137+0000 7f6bc9e00640 1 Processor -- start 2024-03-06T07:28:50.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.137+0000 7f6bc9e00640 1 -- start start 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8004d40 0x7f6bc4142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc9e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc415dbc0 0x7f6bc41447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bc4144d10 0x7f6bc4157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc9e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b4010 con 0x7f6bc4144d10 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc9e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b32b0 con 0x7f6ba8004d40 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc9e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40ccfb0 con 0x7f6bc415dbc0 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bc4144d10 0x7f6bc4157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bc4144d10 0x7f6bc4157ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51212/0 (socket says 172.21.15.114:51212) 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.138+0000 7f6bc3400640 1 -- 172.21.15.114:0/1598966508 learned_addr learned my addr 172.21.15.114:0/1598966508 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:50.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bbbe00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc415dbc0 0x7f6bc41447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc2a00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8004d40 0x7f6bc4142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc3400640 1 -- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc415dbc0 msgr2=0x7f6bc41447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc415dbc0 0x7f6bc41447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc3400640 1 -- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8004d40 msgr2=0x7f6bc4142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8004d40 0x7f6bc4142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc3400640 1 -- 172.21.15.114:0/1598966508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc40a9810 con 0x7f6bc4144d10 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bbbe00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc415dbc0 0x7f6bc41447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc3400640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bc4144d10 0x7f6bc4157ad0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009f90 tx=0x7f6bbc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bbb400640 1 -- 172.21.15.114:0/1598966508 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc043020 con 0x7f6bc4144d10 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bbb400640 1 -- 172.21.15.114:0/1598966508 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc03a040 con 0x7f6bc4144d10 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc4142740 con 0x7f6bc4144d10 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.139+0000 7f6bbb400640 1 -- 172.21.15.114:0/1598966508 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc039070 con 0x7f6bc4144d10 2024-03-06T07:28:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.140+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc4142d00 con 0x7f6bc4144d10 2024-03-06T07:28:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.141+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc404f330 con 0x7f6bc4144d10 2024-03-06T07:28:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.142+0000 7f6bbb400640 1 -- 172.21.15.114:0/1598966508 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bbc046070 con 0x7f6bc4144d10 2024-03-06T07:28:50.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.143+0000 7f6bbb400640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ba006b500 0x7f6ba006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.143+0000 7f6bbb400640 1 -- 172.21.15.114:0/1598966508 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6bbc0abdb0 con 0x7f6bc4144d10 2024-03-06T07:28:50.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.143+0000 7f6bc2a00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ba006b500 0x7f6ba006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.144+0000 7f6bc2a00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ba006b500 0x7f6ba006d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4001eb0 tx=0x7f6bb4001bd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.148+0000 7f6bbb400640 1 -- 172.21.15.114:0/1598966508 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bbc0b0070 con 0x7f6bc4144d10 2024-03-06T07:28:50.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.382+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bc406ce60 con 0x7f6bc4144d10 2024-03-06T07:28:50.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.383+0000 7f6bbb400640 1 -- 172.21.15.114:0/1598966508 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f6bbc07a420 con 0x7f6bc4144d10 2024-03-06T07:28:50.386 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:50.386 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:50.386 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:28:50.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.385+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ba006b500 msgr2=0x7f6ba006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.385+0000 7f6bc9e00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ba006b500 0x7f6ba006d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4001eb0 tx=0x7f6bb4001bd0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.385+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bc4144d10 msgr2=0x7f6bc4157ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.385+0000 7f6bc9e00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bc4144d10 0x7f6bc4157ad0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009f90 tx=0x7f6bbc009750 comp rx=0 tx=0).stop 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 shutdown_connections 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ba006b500 0x7f6ba006d9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bc4144d10 0x7f6bc4157ad0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bc415dbc0 0x7f6bc41447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 --2- 172.21.15.114:0/1598966508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ba8004d40 0x7f6bc4142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 >> 172.21.15.114:0/1598966508 conn(0x7f6bc40b1730 msgr2=0x7f6bc40afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 shutdown_connections 2024-03-06T07:28:50.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.386+0000 7f6bc9e00640 1 -- 172.21.15.114:0/1598966508 wait complete. 2024-03-06T07:28:50.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:50.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.596+0000 7fe4f0c00640 1 Processor -- start 2024-03-06T07:28:50.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.596+0000 7fe4f0c00640 1 -- start start 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.597+0000 7fe4f0c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4ec06cb50 0x7fe4ec06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.597+0000 7fe4f0c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.597+0000 7fe4f0c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec0717c0 con 0x7fe4ec0cf430 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.597+0000 7fe4f0c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ec071aa0 con 0x7fe4ec06cb50 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.597+0000 7fe4f0c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec071d80 con 0x7fe4ec0b2d10 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4e1400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4ec06cb50 0x7fe4ec06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4e1400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4ec06cb50 0x7fe4ec06cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51220/0 (socket says 172.21.15.114:51220) 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4e1400640 1 -- 172.21.15.114:0/2699177581 learned_addr learned my addr 172.21.15.114:0/2699177581 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4ea800640 1 --2- 172.21.15.114:0/2699177581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4e1400640 1 -- 172.21.15.114:0/2699177581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 msgr2=0x7fe4ec0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4e1400640 1 --2- 172.21.15.114:0/2699177581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec0b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4e1400640 1 -- 172.21.15.114:0/2699177581 >> v1:172.21.15.177:6789/0 conn(0x7fe4ec0cf430 legacy=0x7fe4ec070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.598+0000 7fe4e1400640 1 -- 172.21.15.114:0/2699177581 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec072060 con 0x7fe4ec06cb50 2024-03-06T07:28:50.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.600+0000 7fe4e1400640 1 --2- 172.21.15.114:0/2699177581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec06cb50 0x7fe4ec06cfe0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009870 tx=0x7fe4d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed52c869f6a72e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.600+0000 7fe4e9400640 1 -- 172.21.15.114:0/2699177581 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4035020 con 0x7fe4ec06cb50 2024-03-06T07:28:50.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.600+0000 7fe4e9400640 1 -- 172.21.15.114:0/2699177581 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d4031640 con 0x7fe4ec06cb50 2024-03-06T07:28:50.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.600+0000 7fe4e9400640 1 -- 172.21.15.114:0/2699177581 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d402d040 con 0x7fe4ec06cb50 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 -- 172.21.15.114:0/2699177581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec06cb50 msgr2=0x7fe4ec06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/2699177581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec06cb50 0x7fe4ec06cfe0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009870 tx=0x7fe4d402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 -- 172.21.15.114:0/2699177581 shutdown_connections 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/2699177581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec0b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/2699177581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec06cb50 0x7fe4ec06cfe0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 -- 172.21.15.114:0/2699177581 >> 172.21.15.114:0/2699177581 conn(0x7fe4ec0b1730 msgr2=0x7fe4ec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 -- 172.21.15.114:0/2699177581 shutdown_connections 2024-03-06T07:28:50.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.601+0000 7fe4f0c00640 1 -- 172.21.15.114:0/2699177581 wait complete. 2024-03-06T07:28:50.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.603+0000 7fe4f0c00640 1 Processor -- start 2024-03-06T07:28:50.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.603+0000 7fe4f0c00640 1 -- start start 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4f0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4ec06cb50 0x7fe4ec1fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4f0c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4f0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec0cf430 0x7fe4ec151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4f0c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec162b70 con 0x7fe4ec0cf430 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4f0c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec071430 con 0x7fe4ec06cb50 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4f0c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec162cf0 con 0x7fe4ec0b2d10 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4ea800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec0cf430 0x7fe4ec151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4ea800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec0cf430 0x7fe4ec151260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51228/0 (socket says 172.21.15.114:51228) 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4ea800640 1 -- 172.21.15.114:0/136532139 learned_addr learned my addr 172.21.15.114:0/136532139 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4e9e00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4ec06cb50 0x7fe4ec1fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4e1400640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4ea800640 1 -- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 msgr2=0x7fe4ec150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4ea800640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4ea800640 1 -- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4ec06cb50 msgr2=0x7fe4ec1fc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.604+0000 7fe4ea800640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4ec06cb50 0x7fe4ec1fc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.605+0000 7fe4ea800640 1 -- 172.21.15.114:0/136532139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec152d70 con 0x7fe4ec0cf430 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.605+0000 7fe4ea800640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec0cf430 0x7fe4ec151260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe4e400d800 tx=0x7fe4e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.605+0000 7fe4e2a00640 1 -- 172.21.15.114:0/136532139 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e400b6c0 con 0x7fe4ec0cf430 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.605+0000 7fe4e2a00640 1 -- 172.21.15.114:0/136532139 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e4019070 con 0x7fe4ec0cf430 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.605+0000 7fe4e2a00640 1 -- 172.21.15.114:0/136532139 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e400bac0 con 0x7fe4ec0cf430 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.605+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec153030 con 0x7fe4ec0cf430 2024-03-06T07:28:50.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.605+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec15e3e0 con 0x7fe4ec0cf430 2024-03-06T07:28:50.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.606+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec04f330 con 0x7fe4ec0cf430 2024-03-06T07:28:50.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.607+0000 7fe4e2a00640 1 -- 172.21.15.114:0/136532139 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4e400bc60 con 0x7fe4ec0cf430 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.608+0000 7fe4e2a00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe4c806b4d0 0x7fe4c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.608+0000 7fe4e2a00640 1 -- 172.21.15.114:0/136532139 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe4e408a730 con 0x7fe4ec0cf430 2024-03-06T07:28:50.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.609+0000 7fe4e9e00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe4c806b4d0 0x7fe4c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.609+0000 7fe4e9e00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe4c806b4d0 0x7fe4c806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009f20 tx=0x7fe4dc0094d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.613+0000 7fe4e2a00640 1 -- 172.21.15.114:0/136532139 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4e4059510 con 0x7fe4ec0cf430 2024-03-06T07:28:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.810+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4ec0b4a40 con 0x7fe4ec0cf430 2024-03-06T07:28:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.811+0000 7fe4e2a00640 1 -- 172.21.15.114:0/136532139 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16950 (secure 0 0 0) 0x7fe4e4058e10 con 0x7fe4ec0cf430 2024-03-06T07:28:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:50.814 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":22,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:36.265733+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe4c806b4d0 msgr2=0x7fe4c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe4c806b4d0 0x7fe4c806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009f20 tx=0x7fe4dc0094d0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec0cf430 msgr2=0x7fe4ec151260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec0cf430 0x7fe4ec151260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe4e400d800 tx=0x7fe4e400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 shutdown_connections 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe4c806b4d0 0x7fe4c806d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4ec0cf430 0x7fe4ec151260 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4ec0b2d10 0x7fe4ec150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 --2- 172.21.15.114:0/136532139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4ec06cb50 0x7fe4ec1fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 >> 172.21.15.114:0/136532139 conn(0x7fe4ec0b1730 msgr2=0x7fe4ec06cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 shutdown_connections 2024-03-06T07:28:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:50.815+0000 7fe4f0c00640 1 -- 172.21.15.114:0/136532139 wait complete. 2024-03-06T07:28:50.840 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:50.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:51.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.044+0000 7f0cc7000640 1 Processor -- start 2024-03-06T07:28:51.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.044+0000 7f0cc7000640 1 -- start start 2024-03-06T07:28:51.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.046+0000 7f0cc7000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 0x7f0cc015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.046+0000 7f0cc7000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc006cef0 con 0x7f0cc0144c00 2024-03-06T07:28:51.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.046+0000 7f0cc7000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc006d1d0 con 0x7f0cc0070b10 2024-03-06T07:28:51.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.046+0000 7f0cc7000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0160370 con 0x7f0cc015f9d0 2024-03-06T07:28:51.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.046+0000 7f0cbfe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0cc0144c00 0x7f0cc006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56900/0 (socket says 172.21.15.114:56900) 2024-03-06T07:28:51.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.046+0000 7f0cbfe00640 1 -- 172.21.15.114:0/496097844 learned_addr learned my addr 172.21.15.114:0/496097844 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:51.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbf400640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 0x7f0cc015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277737506 0 0) 0x7f0cc006cef0 con 0x7f0cc0144c00 2024-03-06T07:28:51.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca4003520 con 0x7f0cc0144c00 2024-03-06T07:28:51.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1087709323 0 0) 0x7f0ca4003520 con 0x7f0cc0144c00 2024-03-06T07:28:51.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 msgr2=0x7f0cc015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 0x7f0cc015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 >> v1:172.21.15.177:6789/0 conn(0x7f0cc0070b10 legacy=0x7f0cc0070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc006d4b0 con 0x7f0cc0144c00 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f0cb0001670 con 0x7f0cc0144c00 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 >> v1:172.21.15.114:6789/0 conn(0x7f0cc0144c00 legacy=0x7f0cc006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 0x7f0ca40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca4003f60 0x7f0ca4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.047+0000 7f0cbea00640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ca4004d40 0x7f0ca40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca400ba70 con 0x7f0ca4004d40 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca400bc20 con 0x7f0ca4003f60 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca400be20 con 0x7f0cc015f9d0 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cc5000640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ca4004d40 0x7f0ca40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cbfe00640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 0x7f0ca40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cbf400640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca4003f60 0x7f0ca4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cc5000640 1 -- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 msgr2=0x7f0ca40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cc5000640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 0x7f0ca40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cc5000640 1 -- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca4003f60 msgr2=0x7f0ca4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cc5000640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ca4003f60 0x7f0ca4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cc5000640 1 -- 172.21.15.114:0/496097844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4003520 con 0x7f0ca4004d40 2024-03-06T07:28:51.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.048+0000 7f0cbfe00640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0cc015f9d0 0x7f0ca40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:51.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.050+0000 7f0cc5000640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ca4004d40 0x7f0ca40051c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8001e50 tx=0x7f0ca4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab86b3e5831c1b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.050+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8001a70 con 0x7f0ca4004d40 2024-03-06T07:28:51.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.050+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb8003b70 con 0x7f0ca4004d40 2024-03-06T07:28:51.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.050+0000 7f0cbea00640 1 -- 172.21.15.114:0/496097844 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb80050d0 con 0x7f0ca4004d40 2024-03-06T07:28:51.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.051+0000 7f0cc7000640 1 -- 172.21.15.114:0/496097844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ca4004d40 msgr2=0x7f0ca40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.051+0000 7f0cc7000640 1 --2- 172.21.15.114:0/496097844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ca4004d40 0x7f0ca40051c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8001e50 tx=0x7f0ca4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:51.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.051+0000 7f0cc5000640 1 -- 172.21.15.114:0/496097844 reap_dead start 2024-03-06T07:28:51.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.051+0000 7f0cc7000640 1 -- 172.21.15.114:0/496097844 shutdown_connections 2024-03-06T07:28:51.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.051+0000 7f0cc7000640 1 -- 172.21.15.114:0/496097844 >> 172.21.15.114:0/496097844 conn(0x7f0cc00b1730 msgr2=0x7f0cc00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.051+0000 7f0cc7000640 1 -- 172.21.15.114:0/496097844 shutdown_connections 2024-03-06T07:28:51.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.051+0000 7f0cc7000640 1 -- 172.21.15.114:0/496097844 wait complete. 2024-03-06T07:28:51.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.053+0000 7f0cc7000640 1 Processor -- start 2024-03-06T07:28:51.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.053+0000 7f0cc7000640 1 -- start start 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc7000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ca4004d40 0x7f0cc013a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc7000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0cc015c3b0 0x7f0cc015e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc7000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc015c830 0x7f0cc015e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc7000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc006cb50 con 0x7f0cc015c830 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc7000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0160730 con 0x7f0cc015c3b0 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc7000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc006ccd0 con 0x7f0ca4004d40 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc015c830 0x7f0cc015e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc015c830 0x7f0cc015e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51252/0 (socket says 172.21.15.114:51252) 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cc5000640 1 -- 172.21.15.114:0/3602203720 learned_addr learned my addr 172.21.15.114:0/3602203720 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:51.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cbf400640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0cc015c3b0 0x7f0cc015e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.054+0000 7f0cbfe00640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ca4004d40 0x7f0cc013a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cc5000640 1 -- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ca4004d40 msgr2=0x7f0cc013a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cc5000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ca4004d40 0x7f0cc013a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cc5000640 1 -- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0cc015c3b0 msgr2=0x7f0cc015e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cc5000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0cc015c3b0 0x7f0cc015e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cc5000640 1 -- 172.21.15.114:0/3602203720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc013a570 con 0x7f0cc015c830 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cc5000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc015c830 0x7f0cc015e580 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8001f70 tx=0x7f0cb80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cb7e00640 1 -- 172.21.15.114:0/3602203720 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8007db0 con 0x7f0cc015c830 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cb7e00640 1 -- 172.21.15.114:0/3602203720 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb8003970 con 0x7f0cc015c830 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cb7e00640 1 -- 172.21.15.114:0/3602203720 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb802f6a0 con 0x7f0cc015c830 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.055+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc015f050 con 0x7f0cc015c830 2024-03-06T07:28:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.056+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc015f610 con 0x7f0cc015c830 2024-03-06T07:28:51.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.057+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc004f330 con 0x7f0cc015c830 2024-03-06T07:28:51.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.057+0000 7f0cb7e00640 1 -- 172.21.15.114:0/3602203720 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cb802f840 con 0x7f0cc015c830 2024-03-06T07:28:51.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.058+0000 7f0cb7e00640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ca006b500 0x7f0ca006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.058+0000 7f0cb7e00640 1 -- 172.21.15.114:0/3602203720 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0cb80abae0 con 0x7f0cc015c830 2024-03-06T07:28:51.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.059+0000 7f0cbfe00640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ca006b500 0x7f0ca006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.059+0000 7f0cbfe00640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ca006b500 0x7f0ca006d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0001030 tx=0x7f0cb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.063+0000 7f0cb7e00640 1 -- 172.21.15.114:0/3602203720 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cb8033020 con 0x7f0cc015c830 2024-03-06T07:28:51.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.344+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cc00b1480 con 0x7f0cc015c830 2024-03-06T07:28:51.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.345+0000 7f0cb7e00640 1 -- 172.21.15.114:0/3602203720 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f0cb807a8c0 con 0x7f0cc015c830 2024-03-06T07:28:51.348 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:51.348 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:51.348 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ca006b500 msgr2=0x7f0ca006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ca006b500 0x7f0ca006d9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0001030 tx=0x7f0cb0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc015c830 msgr2=0x7f0cc015e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc015c830 0x7f0cc015e580 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8001f70 tx=0x7f0cb80014a0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 shutdown_connections 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ca006b500 0x7f0ca006d9c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc015c830 0x7f0cc015e580 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0cc015c3b0 0x7f0cc015e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 --2- 172.21.15.114:0/3602203720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ca4004d40 0x7f0cc013a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 >> 172.21.15.114:0/3602203720 conn(0x7f0cc00b1730 msgr2=0x7f0cc0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 shutdown_connections 2024-03-06T07:28:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.348+0000 7f0cc7000640 1 -- 172.21.15.114:0/3602203720 wait complete. 2024-03-06T07:28:51.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:51.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.568+0000 7f3854c00640 1 Processor -- start 2024-03-06T07:28:51.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.568+0000 7f3854c00640 1 -- start start 2024-03-06T07:28:51.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.569+0000 7f3854c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3850161380 0x7f38500ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.569+0000 7f3854c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f38500b26a0 0x7f38500ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.569+0000 7f3854c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38500cf000 con 0x7f3850161760 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.569+0000 7f3854c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38500cd570 con 0x7f3850161380 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.569+0000 7f3854c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38500cd850 con 0x7f38500b26a0 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3850161380 0x7f38500ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3850161380 0x7f38500ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51258/0 (socket says 172.21.15.114:51258) 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f3846a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3850161760 0x7f3850161c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56902/0 (socket says 172.21.15.114:56902) 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384f400640 1 -- 172.21.15.114:0/1988780728 learned_addr learned my addr 172.21.15.114:0/1988780728 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384fe00640 1 --2- 172.21.15.114:0/1988780728 >> v2:172.21.15.177:3300/0 conn(0x7f38500b26a0 0x7f38500ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384f400640 1 -- 172.21.15.114:0/1988780728 >> v2:172.21.15.177:3300/0 conn(0x7f38500b26a0 msgr2=0x7f38500ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384f400640 1 --2- 172.21.15.114:0/1988780728 >> v2:172.21.15.177:3300/0 conn(0x7f38500b26a0 0x7f38500ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384f400640 1 -- 172.21.15.114:0/1988780728 >> v1:172.21.15.114:6789/0 conn(0x7f3850161760 legacy=0x7f3850161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384f400640 1 -- 172.21.15.114:0/1988780728 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38500cdb30 con 0x7f3850161380 2024-03-06T07:28:51.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.570+0000 7f384ea00640 1 -- 172.21.15.114:0/1988780728 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004373688 0 0) 0x7f38500cf000 con 0x7f3850161760 2024-03-06T07:28:51.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f384f400640 1 --2- 172.21.15.114:0/1988780728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3850161380 0x7f38500ce430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3850071f50 tx=0x7f384002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba1ca8fe7700176 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f384ea00640 1 -- 172.21.15.114:0/1988780728 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840035020 con 0x7f3850161380 2024-03-06T07:28:51.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f384ea00640 1 -- 172.21.15.114:0/1988780728 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3840031870 con 0x7f3850161380 2024-03-06T07:28:51.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f384ea00640 1 -- 172.21.15.114:0/1988780728 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840031c40 con 0x7f3850161380 2024-03-06T07:28:51.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f3854c00640 1 -- 172.21.15.114:0/1988780728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3850161380 msgr2=0x7f38500ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f3854c00640 1 --2- 172.21.15.114:0/1988780728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3850161380 0x7f38500ce430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3850071f50 tx=0x7f384002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:51.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f3854c00640 1 -- 172.21.15.114:0/1988780728 shutdown_connections 2024-03-06T07:28:51.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.572+0000 7f3854c00640 1 --2- 172.21.15.114:0/1988780728 >> v2:172.21.15.177:3300/0 conn(0x7f38500b26a0 0x7f38500ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.573+0000 7f3854c00640 1 --2- 172.21.15.114:0/1988780728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3850161380 0x7f38500ce430 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.573+0000 7f3854c00640 1 -- 172.21.15.114:0/1988780728 >> 172.21.15.114:0/1988780728 conn(0x7f38500b3690 msgr2=0x7f38500b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.573+0000 7f3854c00640 1 -- 172.21.15.114:0/1988780728 shutdown_connections 2024-03-06T07:28:51.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.573+0000 7f3854c00640 1 -- 172.21.15.114:0/1988780728 wait complete. 2024-03-06T07:28:51.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.574+0000 7f3854c00640 1 Processor -- start 2024-03-06T07:28:51.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.574+0000 7f3854c00640 1 -- start start 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.575+0000 7f3854c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500b26a0 0x7f38501f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.575+0000 7f3854c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161380 0x7f3850160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.575+0000 7f3854c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3850161760 0x7f38500c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.575+0000 7f3854c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38500d0a50 con 0x7f38500b26a0 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.575+0000 7f3854c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38500d08d0 con 0x7f3850161760 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.575+0000 7f3854c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38500d0bd0 con 0x7f3850161380 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500b26a0 0x7f38501f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500b26a0 0x7f38501f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51266/0 (socket says 172.21.15.114:51266) 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 -- 172.21.15.114:0/890525236 learned_addr learned my addr 172.21.15.114:0/890525236 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:51.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384fe00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3850161760 0x7f38500c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f3846a00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161380 0x7f3850160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 -- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161380 msgr2=0x7f3850160570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161380 0x7f3850160570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 -- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3850161760 msgr2=0x7f38500c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3850161760 0x7f38500c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f384f400640 1 -- 172.21.15.114:0/890525236 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38501f7f80 con 0x7f38500b26a0 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.576+0000 7f3846a00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161380 0x7f3850160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.577+0000 7f384f400640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500b26a0 0x7f38501f7a40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3840000c00 tx=0x7f38400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.577+0000 7f3847e00640 1 -- 172.21.15.114:0/890525236 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840040070 con 0x7f38500b26a0 2024-03-06T07:28:51.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.577+0000 7f3847e00640 1 -- 172.21.15.114:0/890525236 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3840030d00 con 0x7f38500b26a0 2024-03-06T07:28:51.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.577+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3850160ab0 con 0x7f38500b26a0 2024-03-06T07:28:51.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.577+0000 7f3847e00640 1 -- 172.21.15.114:0/890525236 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840035040 con 0x7f38500b26a0 2024-03-06T07:28:51.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.577+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38500c7e80 con 0x7f38500b26a0 2024-03-06T07:28:51.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.578+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385004f330 con 0x7f38500b26a0 2024-03-06T07:28:51.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.579+0000 7f3847e00640 1 -- 172.21.15.114:0/890525236 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3840038020 con 0x7f38500b26a0 2024-03-06T07:28:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.580+0000 7f3847e00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b500 0x7f382c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.581+0000 7f3847e00640 1 -- 172.21.15.114:0/890525236 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f38400ac070 con 0x7f38500b26a0 2024-03-06T07:28:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.581+0000 7f3846a00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b500 0x7f382c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.581+0000 7f3846a00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b500 0x7f382c06d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f385006db40 tx=0x7f3838009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.585+0000 7f3847e00640 1 -- 172.21.15.114:0/890525236 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38501f7f80 con 0x7f38500b26a0 2024-03-06T07:28:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.818+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38500b7ac0 con 0x7f38500b26a0 2024-03-06T07:28:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.819+0000 7f3847e00640 1 -- 172.21.15.114:0/890525236 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4983 (secure 0 0 0) 0x7f384007ae60 con 0x7f38500b26a0 2024-03-06T07:28:51.822 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:51.822 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3864220883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3864220883},{"type":"v1","addr":"172.21.15.177:6833","nonce":3864220883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3773586248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3773586248},{"type":"v1","addr":"172.21.15.177:6835","nonce":3773586248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1663866098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1663866098},{"type":"v1","addr":"172.21.15.114:6836","nonce":1663866098}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:54.463887+0000","modified":"2024-03-06T07:27:57.520027+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3774250012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3774250012},{"type":"v1","addr":"172.21.15.114:6837","nonce":3774250012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":1}]} 2024-03-06T07:28:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.821+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b500 msgr2=0x7f382c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.821+0000 7f3854c00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b500 0x7f382c06d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f385006db40 tx=0x7f3838009040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.821+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500b26a0 msgr2=0x7f38501f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.821+0000 7f3854c00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500b26a0 0x7f38501f7a40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3840000c00 tx=0x7f38400046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 shutdown_connections 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3850161760 0x7f38500c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3850161380 0x7f3850160570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f382c06b500 0x7f382c06d9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 --2- 172.21.15.114:0/890525236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38500b26a0 0x7f38501f7a40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 >> 172.21.15.114:0/890525236 conn(0x7f38500b3690 msgr2=0x7f38500cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 shutdown_connections 2024-03-06T07:28:51.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:51.822+0000 7f3854c00640 1 -- 172.21.15.114:0/890525236 wait complete. 2024-03-06T07:28:51.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:52.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.036+0000 7f3ebb000640 1 Processor -- start 2024-03-06T07:28:52.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.036+0000 7f3ebb000640 1 -- start start 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.037+0000 7f3ebb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb415e0d0 0x7f3eb40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.037+0000 7f3ebb000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3eb415f470 0x7f3eb40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3ebb000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb40cea60 con 0x7f3eb40d0df0 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3ebb000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb40d1260 con 0x7f3eb415f470 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3ebb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40d2f60 con 0x7f3eb415e0d0 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3eb415f470 0x7f3eb40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3eb415f470 0x7f3eb40d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51280/0 (socket says 172.21.15.114:51280) 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3400640 1 -- 172.21.15.114:0/340506836 learned_addr learned my addr 172.21.15.114:0/340506836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3e00640 1 --2- 172.21.15.114:0/340506836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb415e0d0 0x7f3eb40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3400640 1 -- 172.21.15.114:0/340506836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb415e0d0 msgr2=0x7f3eb40d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3400640 1 --2- 172.21.15.114:0/340506836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb415e0d0 0x7f3eb40d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3400640 1 -- 172.21.15.114:0/340506836 >> v1:172.21.15.177:6789/0 conn(0x7f3eb40d0df0 legacy=0x7f3eb4160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3400640 1 -- 172.21.15.114:0/340506836 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40d2350 con 0x7f3eb415f470 2024-03-06T07:28:52.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.038+0000 7f3eb3e00640 1 --2- 172.21.15.114:0/340506836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb415e0d0 0x7f3eb40d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.039+0000 7f3eb3400640 1 --2- 172.21.15.114:0/340506836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40d08b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009870 tx=0x7f3ea002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb765b45c0ef084f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3eb2a00640 1 -- 172.21.15.114:0/340506836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0034070 con 0x7f3eb415f470 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3eb2a00640 1 -- 172.21.15.114:0/340506836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0030840 con 0x7f3eb415f470 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3eb2a00640 1 -- 172.21.15.114:0/340506836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0030c10 con 0x7f3eb415f470 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 -- 172.21.15.114:0/340506836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 msgr2=0x7f3eb40d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 --2- 172.21.15.114:0/340506836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40d08b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009870 tx=0x7f3ea002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 -- 172.21.15.114:0/340506836 shutdown_connections 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 --2- 172.21.15.114:0/340506836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40d08b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 --2- 172.21.15.114:0/340506836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb415e0d0 0x7f3eb40d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 -- 172.21.15.114:0/340506836 >> 172.21.15.114:0/340506836 conn(0x7f3eb40d3ec0 msgr2=0x7f3eb40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 -- 172.21.15.114:0/340506836 shutdown_connections 2024-03-06T07:28:52.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.040+0000 7f3ebb000640 1 -- 172.21.15.114:0/340506836 wait complete. 2024-03-06T07:28:52.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.042+0000 7f3ebb000640 1 Processor -- start 2024-03-06T07:28:52.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.042+0000 7f3ebb000640 1 -- start start 2024-03-06T07:28:52.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3ebb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb40d0df0 0x7f3eb40cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3ebb000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb415e0d0 0x7f3eb4152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3ebb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3ebb000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40d2bf0 con 0x7f3eb415f470 2024-03-06T07:28:52.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3ebb000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40d2910 con 0x7f3eb415e0d0 2024-03-06T07:28:52.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3ebb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb415fb30 con 0x7f3eb40d0df0 2024-03-06T07:28:52.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3eb9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.043+0000 7f3eb9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51296/0 (socket says 172.21.15.114:51296) 2024-03-06T07:28:52.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb9000640 1 -- 172.21.15.114:0/1508654840 learned_addr learned my addr 172.21.15.114:0/1508654840 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:52.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb3e00640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb40d0df0 0x7f3eb40cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb3400640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb415e0d0 0x7f3eb4152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb9000640 1 -- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb40d0df0 msgr2=0x7f3eb40cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb9000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb40d0df0 0x7f3eb40cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb9000640 1 -- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb415e0d0 msgr2=0x7f3eb4152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb9000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb415e0d0 0x7f3eb4152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb9000640 1 -- 172.21.15.114:0/1508654840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb4153340 con 0x7f3eb415f470 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.044+0000 7f3eb9000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40cb9a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00b970 tx=0x7f3eac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.045+0000 7f3eabe00640 1 -- 172.21.15.114:0/1508654840 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac019070 con 0x7f3eb415f470 2024-03-06T07:28:52.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.045+0000 7f3eabe00640 1 -- 172.21.15.114:0/1508654840 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac00cb40 con 0x7f3eb415f470 2024-03-06T07:28:52.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.045+0000 7f3eabe00640 1 -- 172.21.15.114:0/1508654840 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac014440 con 0x7f3eb415f470 2024-03-06T07:28:52.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.045+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb40cbee0 con 0x7f3eb415f470 2024-03-06T07:28:52.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.045+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb40cc440 con 0x7f3eb415f470 2024-03-06T07:28:52.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.046+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb404f330 con 0x7f3eb415f470 2024-03-06T07:28:52.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.047+0000 7f3eabe00640 1 -- 172.21.15.114:0/1508654840 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3eac01c070 con 0x7f3eb415f470 2024-03-06T07:28:52.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.048+0000 7f3eabe00640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e9406b500 0x7f3e9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.048+0000 7f3eabe00640 1 -- 172.21.15.114:0/1508654840 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f3eac089f00 con 0x7f3eb415f470 2024-03-06T07:28:52.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.048+0000 7f3eb3e00640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e9406b500 0x7f3e9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.049+0000 7f3eb3e00640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e9406b500 0x7f3e9406d9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400a950 tx=0x7f3ea4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.057+0000 7f3eabe00640 1 -- 172.21.15.114:0/1508654840 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3eb40cbee0 con 0x7f3eb415f470 2024-03-06T07:28:52.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.250+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eb408d920 con 0x7f3eb415f470 2024-03-06T07:28:52.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.252+0000 7f3eabe00640 1 -- 172.21.15.114:0/1508654840 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16950 (secure 0 0 0) 0x7f3eac058c50 con 0x7f3eb415f470 2024-03-06T07:28:52.256 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:52.256 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":22,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:36.265733+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:52.531713+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:53.453989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:52.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e9406b500 msgr2=0x7f3e9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e9406b500 0x7f3e9406d9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400a950 tx=0x7f3ea4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:52.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 msgr2=0x7f3eb40cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40cb9a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00b970 tx=0x7f3eac00be30 comp rx=0 tx=0).stop 2024-03-06T07:28:52.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 shutdown_connections 2024-03-06T07:28:52.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e9406b500 0x7f3e9406d9c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3eb415f470 0x7f3eb40cb9a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb415e0d0 0x7f3eb4152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 --2- 172.21.15.114:0/1508654840 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3eb40d0df0 0x7f3eb40cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.255+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 >> 172.21.15.114:0/1508654840 conn(0x7f3eb40d3ec0 msgr2=0x7f3eb40b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.256+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 shutdown_connections 2024-03-06T07:28:52.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.256+0000 7f3ebb000640 1 -- 172.21.15.114:0/1508654840 wait complete. 2024-03-06T07:28:52.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:52.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.470+0000 7f29c7e00640 1 Processor -- start 2024-03-06T07:28:52.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.470+0000 7f29c7e00640 1 -- start start 2024-03-06T07:28:52.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.471+0000 7f29c7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f29c0164b30 0x7f29c0166f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.471+0000 7f29c7e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c0145b30 con 0x7f29c00c6460 2024-03-06T07:28:52.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.471+0000 7f29c7e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c0142ce0 con 0x7f29c00c4930 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.471+0000 7f29c7e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c00c68c0 con 0x7f29c0164b30 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29c4a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f29c00c6460 0x7f29c00c5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56904/0 (socket says 172.21.15.114:56904) 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29c4a00640 1 -- 172.21.15.114:0/181098796 learned_addr learned my addr 172.21.15.114:0/181098796 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29c5e00640 1 --2- 172.21.15.114:0/181098796 >> v2:172.21.15.114:3300/0 conn(0x7f29c0164b30 0x7f29c0166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29bfe00640 1 -- 172.21.15.114:0/181098796 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365634365 0 0) 0x7f29ac001110 con 0x7f29c00c6460 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29bfe00640 1 -- 172.21.15.114:0/181098796 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a8003520 con 0x7f29c00c6460 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29c5e00640 1 -- 172.21.15.114:0/181098796 >> v1:172.21.15.177:6789/0 conn(0x7f29c00c4930 legacy=0x7f29c00c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29c5e00640 1 -- 172.21.15.114:0/181098796 >> v1:172.21.15.114:6789/0 conn(0x7f29c00c6460 legacy=0x7f29c00c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.472+0000 7f29c5e00640 1 -- 172.21.15.114:0/181098796 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c00cbca0 con 0x7f29c0164b30 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.474+0000 7f29c5e00640 1 --2- 172.21.15.114:0/181098796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c0166f20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7e37dba9153ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.474+0000 7f29bfe00640 1 -- 172.21.15.114:0/181098796 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8039070 con 0x7f29c0164b30 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.474+0000 7f29bfe00640 1 -- 172.21.15.114:0/181098796 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b802cc60 con 0x7f29c0164b30 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.474+0000 7f29bfe00640 1 -- 172.21.15.114:0/181098796 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8034660 con 0x7f29c0164b30 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.475+0000 7f29c7e00640 1 -- 172.21.15.114:0/181098796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 msgr2=0x7f29c0166f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.475+0000 7f29c7e00640 1 --2- 172.21.15.114:0/181098796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c0166f20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.475+0000 7f29c7e00640 1 -- 172.21.15.114:0/181098796 shutdown_connections 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.475+0000 7f29c7e00640 1 --2- 172.21.15.114:0/181098796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c0166f20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.475+0000 7f29c7e00640 1 -- 172.21.15.114:0/181098796 >> 172.21.15.114:0/181098796 conn(0x7f29c00cc4f0 msgr2=0x7f29c00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.475+0000 7f29c7e00640 1 -- 172.21.15.114:0/181098796 shutdown_connections 2024-03-06T07:28:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.475+0000 7f29c7e00640 1 -- 172.21.15.114:0/181098796 wait complete. 2024-03-06T07:28:52.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.477+0000 7f29c7e00640 1 Processor -- start 2024-03-06T07:28:52.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.477+0000 7f29c7e00640 1 -- start start 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29c00c4930 0x7f29c015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29c00c6460 0x7f29c00c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c00c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c00ca3b0 con 0x7f29c0164b30 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c00ca0d0 con 0x7f29c00c4930 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c006e600 con 0x7f29c00c6460 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c00c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c00c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51312/0 (socket says 172.21.15.114:51312) 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 -- 172.21.15.114:0/3268981399 learned_addr learned my addr 172.21.15.114:0/3268981399 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 -- 172.21.15.114:0/3268981399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29c00c6460 msgr2=0x7f29c00c7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29c00c6460 0x7f29c00c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 -- 172.21.15.114:0/3268981399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29c00c4930 msgr2=0x7f29c015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29c00c4930 0x7f29c015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.478+0000 7f29c5e00640 1 -- 172.21.15.114:0/3268981399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c00c7560 con 0x7f29c0164b30 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.479+0000 7f29c5e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c00c8510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.479+0000 7f29bd600640 1 -- 172.21.15.114:0/3268981399 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8039050 con 0x7f29c0164b30 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.479+0000 7f29bd600640 1 -- 172.21.15.114:0/3268981399 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b8045070 con 0x7f29c0164b30 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.479+0000 7f29bd600640 1 -- 172.21.15.114:0/3268981399 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8009450 con 0x7f29c0164b30 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.479+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c00c9550 con 0x7f29c0164b30 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.479+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c00c9830 con 0x7f29c0164b30 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.480+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c004f330 con 0x7f29c0164b30 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.481+0000 7f29bd600640 1 -- 172.21.15.114:0/3268981399 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29b803e850 con 0x7f29c0164b30 2024-03-06T07:28:52.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.482+0000 7f29bd600640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29a006b4d0 0x7f29a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.482+0000 7f29bd600640 1 -- 172.21.15.114:0/3268981399 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f29b80acf70 con 0x7f29c0164b30 2024-03-06T07:28:52.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.482+0000 7f29c5400640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29a006b4d0 0x7f29a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.483+0000 7f29c5400640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29a006b4d0 0x7f29a006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f60 tx=0x7f29b40079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.486+0000 7f29bd600640 1 -- 172.21.15.114:0/3268981399 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29b8032e20 con 0x7f29c0164b30 2024-03-06T07:28:52.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.722+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f29c01415c0 con 0x7f29c0164b30 2024-03-06T07:28:52.726 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:28:52.723+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:52.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.727+0000 7f29bd600640 1 -- 172.21.15.114:0/3268981399 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f29b807bcc0 con 0x7f29c0164b30 2024-03-06T07:28:52.730 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:52.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.730+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29a006b4d0 msgr2=0x7f29a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.730+0000 7f29c7e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29a006b4d0 0x7f29a006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f60 tx=0x7f29b40079d0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.730+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 msgr2=0x7f29c00c8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.730+0000 7f29c7e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c00c8510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 shutdown_connections 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29a006b4d0 0x7f29a006d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29c0164b30 0x7f29c00c8510 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29c00c6460 0x7f29c00c7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 --2- 172.21.15.114:0/3268981399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29c00c4930 0x7f29c015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 >> 172.21.15.114:0/3268981399 conn(0x7f29c00cc4f0 msgr2=0x7f29c00c4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 shutdown_connections 2024-03-06T07:28:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.731+0000 7f29c7e00640 1 -- 172.21.15.114:0/3268981399 wait complete. 2024-03-06T07:28:52.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:52.756 INFO:tasks.ceph.mds.c.smithi114.stderr: -7752> 2024-03-06T07:27:59.618+0000 7f9f81600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:52.821 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.925+0000 7f5a02e00640 1 Processor -- start 2024-03-06T07:28:52.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.925+0000 7f5a02e00640 1 -- start start 2024-03-06T07:28:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.926+0000 7f5a02e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f59fc0d1ea0 0x7f59fc0d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.926+0000 7f5a02e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 0x7f59fc0c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.926+0000 7f5a02e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc0d13e0 con 0x7f59fc15e130 2024-03-06T07:28:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.926+0000 7f5a02e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0d2840 con 0x7f59fc0d1ea0 2024-03-06T07:28:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.926+0000 7f5a02e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0cf120 con 0x7f59fc0c7970 2024-03-06T07:28:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f5a01a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f59fc15e130 0x7f59fc0cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56928/0 (socket says 172.21.15.114:56928) 2024-03-06T07:28:52.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f5a01a00640 1 -- 172.21.15.114:0/3445944891 learned_addr learned my addr 172.21.15.114:0/3445944891 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f5a01000640 1 --2- 172.21.15.114:0/3445944891 >> v2:172.21.15.177:3300/0 conn(0x7f59fc0d1ea0 0x7f59fc0d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f5a02400640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 0x7f59fc0c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207566552 0 0) 0x7f59fc0d13e0 con 0x7f59fc15e130 2024-03-06T07:28:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e4003520 con 0x7f59fc15e130 2024-03-06T07:28:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3965938099 0 0) 0x7f59e4003520 con 0x7f59fc15e130 2024-03-06T07:28:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 msgr2=0x7f59fc0c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f59fbe00640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 0x7f59fc0c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.927+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 >> v2:172.21.15.177:3300/0 conn(0x7f59fc0d1ea0 msgr2=0x7f59fc0d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 --2- 172.21.15.114:0/3445944891 >> v2:172.21.15.177:3300/0 conn(0x7f59fc0d1ea0 0x7f59fc0d2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0cead0 con 0x7f59fc15e130 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f59f0001670 con 0x7f59fc15e130 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 >> v1:172.21.15.114:6789/0 conn(0x7f59fc15e130 legacy=0x7f59fc0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 0x7f59e40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e4003f80 0x7f59e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0cf120 con 0x7f59e4004d40 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e400b9c0 con 0x7f59e4003f80 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e400bb40 con 0x7f59fc0c7970 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a02400640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a01a00640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 0x7f59e40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a01000640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e4003f80 0x7f59e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a02400640 1 -- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 msgr2=0x7f59e40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a02400640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 0x7f59e40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a02400640 1 -- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e4003f80 msgr2=0x7f59e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a02400640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e4003f80 0x7f59e4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a02400640 1 -- 172.21.15.114:0/3445944891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e4003520 con 0x7f59e4004d40 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.928+0000 7f5a01a00640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc0c7970 0x7f59e40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.929+0000 7f5a02400640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59e40051c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f59f400e170 tx=0x7f59e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36212ad49eb97f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4039070 con 0x7f59e4004d40 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f402cbf0 con 0x7f59e4004d40 2024-03-06T07:28:52.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f59fbe00640 1 -- 172.21.15.114:0/3445944891 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4009c10 con 0x7f59e4004d40 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f5a02e00640 1 -- 172.21.15.114:0/3445944891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 msgr2=0x7f59e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f5a02e00640 1 --2- 172.21.15.114:0/3445944891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59e40051c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f59f400e170 tx=0x7f59e4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f5a02400640 1 -- 172.21.15.114:0/3445944891 reap_dead start 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f5a02e00640 1 -- 172.21.15.114:0/3445944891 shutdown_connections 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f5a02e00640 1 -- 172.21.15.114:0/3445944891 >> 172.21.15.114:0/3445944891 conn(0x7f59fc09d070 msgr2=0x7f59fc0d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f5a02e00640 1 -- 172.21.15.114:0/3445944891 shutdown_connections 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.930+0000 7f5a02e00640 1 -- 172.21.15.114:0/3445944891 wait complete. 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.932+0000 7f5a02e00640 1 Processor -- start 2024-03-06T07:28:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.932+0000 7f5a02e00640 1 -- start start 2024-03-06T07:28:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a02e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59fc1f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a02e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc1f3f10 0x7f59fc15f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a02e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc15faa0 0x7f59fc06c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a02e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0d16c0 con 0x7f59e4004d40 2024-03-06T07:28:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a02e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0cfcc0 con 0x7f59fc1f3f10 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a02e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0c7e00 con 0x7f59fc15faa0 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a01a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59fc1f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a01a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59fc1f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51362/0 (socket says 172.21.15.114:51362) 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.933+0000 7f5a01a00640 1 -- 172.21.15.114:0/636307626 learned_addr learned my addr 172.21.15.114:0/636307626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a01a00640 1 -- 172.21.15.114:0/636307626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc15faa0 msgr2=0x7f59fc06c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a01a00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc15faa0 0x7f59fc06c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a01a00640 1 -- 172.21.15.114:0/636307626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc1f3f10 msgr2=0x7f59fc15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a01a00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc1f3f10 0x7f59fc15f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a01a00640 1 -- 172.21.15.114:0/636307626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc15ffb0 con 0x7f59e4004d40 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a01a00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59fc1f6ea0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f59f000d940 tx=0x7f59f000de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f59f9600640 1 -- 172.21.15.114:0/636307626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0019070 con 0x7f59e4004d40 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f59f9600640 1 -- 172.21.15.114:0/636307626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f00074e0 con 0x7f59e4004d40 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f59f9600640 1 -- 172.21.15.114:0/636307626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f00049e0 con 0x7f59e4004d40 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc160290 con 0x7f59e4004d40 2024-03-06T07:28:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.934+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc1610b0 con 0x7f59e4004d40 2024-03-06T07:28:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.935+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc0c87e0 con 0x7f59e4004d40 2024-03-06T07:28:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.936+0000 7f59f9600640 1 -- 172.21.15.114:0/636307626 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59f001c070 con 0x7f59e4004d40 2024-03-06T07:28:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.937+0000 7f59f9600640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59dc06b4d0 0x7f59dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.937+0000 7f59f9600640 1 -- 172.21.15.114:0/636307626 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f59f0089ed0 con 0x7f59e4004d40 2024-03-06T07:28:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.937+0000 7f5a01000640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59dc06b4d0 0x7f59dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.938+0000 7f5a01000640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59dc06b4d0 0x7f59dc06d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f59fc160cc0 tx=0x7f59e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:52.941+0000 7f59f9600640 1 -- 172.21.15.114:0/636307626 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59fc0c87e0 con 0x7f59e4004d40 2024-03-06T07:28:53.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.164+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f59fc04f330 con 0x7f59e4004d40 2024-03-06T07:28:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.731+0000 7f59f9600640 1 -- 172.21.15.114:0/636307626 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f59f0058c40 con 0x7f59e4004d40 2024-03-06T07:28:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.734+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59dc06b4d0 msgr2=0x7f59dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.734+0000 7f5a02e00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59dc06b4d0 0x7f59dc06d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f59fc160cc0 tx=0x7f59e8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.734+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 msgr2=0x7f59fc1f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.734+0000 7f5a02e00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59fc1f6ea0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f59f000d940 tx=0x7f59f000de00 comp rx=0 tx=0).stop 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 shutdown_connections 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc15faa0 0x7f59fc06c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc1f3f10 0x7f59fc15f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59dc06b4d0 0x7f59dc06d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 --2- 172.21.15.114:0/636307626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e4004d40 0x7f59fc1f6ea0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 >> 172.21.15.114:0/636307626 conn(0x7f59fc09d070 msgr2=0x7f59fc15bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 shutdown_connections 2024-03-06T07:28:53.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.735+0000 7f5a02e00640 1 -- 172.21.15.114:0/636307626 wait complete. 2024-03-06T07:28:53.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:53.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.971+0000 7f9ab2e00640 1 Processor -- start 2024-03-06T07:28:53.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.972+0000 7f9ab2e00640 1 -- start start 2024-03-06T07:28:53.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.973+0000 7f9ab2e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9aac15c160 0x7f9aac0cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.973+0000 7f9ab2e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0cd240 0x7f9aac0d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.973+0000 7f9ab2e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0cd790 con 0x7f9aac0d0740 2024-03-06T07:28:53.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.973+0000 7f9ab2e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aac0cda70 con 0x7f9aac15c160 2024-03-06T07:28:53.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.973+0000 7f9ab2e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0b12a0 con 0x7f9aac0cd240 2024-03-06T07:28:53.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.974+0000 7f9aabe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9aac15c160 0x7f9aac0cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.974+0000 7f9aabe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9aac15c160 0x7f9aac0cf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:46424/0 (socket says 172.21.15.114:46424) 2024-03-06T07:28:53.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.974+0000 7f9aabe00640 1 -- 172.21.15.114:0/2816654115 learned_addr learned my addr 172.21.15.114:0/2816654115 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:53.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.974+0000 7f9aab400640 1 --2- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0cd240 0x7f9aac0d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.975+0000 7f9aabe00640 1 -- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0cd240 msgr2=0x7f9aac0d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.975+0000 7f9aabe00640 1 --2- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0cd240 0x7f9aac0d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.975+0000 7f9aabe00640 1 -- 172.21.15.114:0/2816654115 >> v1:172.21.15.177:6789/0 conn(0x7f9aac0d0740 legacy=0x7f9aac0ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.975+0000 7f9aabe00640 1 -- 172.21.15.114:0/2816654115 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0b1580 con 0x7f9aac15c160 2024-03-06T07:28:53.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.976+0000 7f9aabe00640 1 --2- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac15c160 0x7f9aac0cf9b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009870 tx=0x7f9a9c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac5a6cbb97ca1eae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.976+0000 7f9aaaa00640 1 -- 172.21.15.114:0/2816654115 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c035020 con 0x7f9aac15c160 2024-03-06T07:28:53.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.976+0000 7f9aaaa00640 1 -- 172.21.15.114:0/2816654115 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c031640 con 0x7f9aac15c160 2024-03-06T07:28:53.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.976+0000 7f9aaaa00640 1 -- 172.21.15.114:0/2816654115 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c02d040 con 0x7f9aac15c160 2024-03-06T07:28:53.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 -- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac15c160 msgr2=0x7f9aac0cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac15c160 0x7f9aac0cf9b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009870 tx=0x7f9a9c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 -- 172.21.15.114:0/2816654115 shutdown_connections 2024-03-06T07:28:53.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0cd240 0x7f9aac0d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/2816654115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac15c160 0x7f9aac0cf9b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 -- 172.21.15.114:0/2816654115 >> 172.21.15.114:0/2816654115 conn(0x7f9aac0d1c20 msgr2=0x7f9aac0d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 -- 172.21.15.114:0/2816654115 shutdown_connections 2024-03-06T07:28:53.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.977+0000 7f9ab2e00640 1 -- 172.21.15.114:0/2816654115 wait complete. 2024-03-06T07:28:53.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.980+0000 7f9ab2e00640 1 Processor -- start 2024-03-06T07:28:53.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.980+0000 7f9ab2e00640 1 -- start start 2024-03-06T07:28:53.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab2e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac0cd240 0x7f9aac154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab2e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0d0740 0x7f9aac157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab2e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac15c160 0x7f9aac06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab2e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0b2c30 con 0x7f9aac15c160 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab2e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0b2ab0 con 0x7f9aac0cd240 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab2e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0b2db0 con 0x7f9aac0d0740 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac15c160 0x7f9aac06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac15c160 0x7f9aac06d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51370/0 (socket says 172.21.15.114:51370) 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.981+0000 7f9ab0e00640 1 -- 172.21.15.114:0/1642572946 learned_addr learned my addr 172.21.15.114:0/1642572946 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9aab400640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0d0740 0x7f9aac157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9aabe00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac0cd240 0x7f9aac154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9ab0e00640 1 -- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0d0740 msgr2=0x7f9aac157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9ab0e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0d0740 0x7f9aac157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9ab0e00640 1 -- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac0cd240 msgr2=0x7f9aac154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9ab0e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac0cd240 0x7f9aac154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9ab0e00640 1 -- 172.21.15.114:0/1642572946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac15a460 con 0x7f9aac15c160 2024-03-06T07:28:53.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9aabe00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac0cd240 0x7f9aac154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:53.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9ab0e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac15c160 0x7f9aac06d7f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400c3c0 tx=0x7f9aa400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.982+0000 7f9aa3e00640 1 -- 172.21.15.114:0/1642572946 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4017070 con 0x7f9aac15c160 2024-03-06T07:28:53.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.983+0000 7f9aa3e00640 1 -- 172.21.15.114:0/1642572946 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa40052d0 con 0x7f9aac15c160 2024-03-06T07:28:53.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.983+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac15a740 con 0x7f9aac15c160 2024-03-06T07:28:53.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.983+0000 7f9aa3e00640 1 -- 172.21.15.114:0/1642572946 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4003830 con 0x7f9aac15c160 2024-03-06T07:28:53.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.983+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac156910 con 0x7f9aac15c160 2024-03-06T07:28:53.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.984+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aac068ee0 con 0x7f9aac15c160 2024-03-06T07:28:53.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.985+0000 7f9aa3e00640 1 -- 172.21.15.114:0/1642572946 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9aa401a070 con 0x7f9aac15c160 2024-03-06T07:28:53.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.985+0000 7f9aa3e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a8806b4d0 0x7f9a8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.986+0000 7f9aa3e00640 1 -- 172.21.15.114:0/1642572946 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f9aa4089690 con 0x7f9aac15c160 2024-03-06T07:28:53.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.986+0000 7f9aabe00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a8806b4d0 0x7f9a8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.986+0000 7f9aabe00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a8806b4d0 0x7f9a8806d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c036000 tx=0x7f9a9c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:53.990+0000 7f9aa3e00640 1 -- 172.21.15.114:0/1642572946 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9aac068ee0 con 0x7f9aac15c160 2024-03-06T07:28:54.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.185+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9aac04f3a0 con 0x7f9aac15c160 2024-03-06T07:28:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.741+0000 7f9aa3e00640 1 -- 172.21.15.114:0/1642572946 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9aa4058420 con 0x7f9aac15c160 2024-03-06T07:28:54.744 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:54.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a8806b4d0 msgr2=0x7f9a8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a8806b4d0 0x7f9a8806d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c036000 tx=0x7f9a9c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac15c160 msgr2=0x7f9aac06d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac15c160 0x7f9aac06d7f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400c3c0 tx=0x7f9aa400c7c0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 shutdown_connections 2024-03-06T07:28:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a8806b4d0 0x7f9a8806d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9aac15c160 0x7f9aac06d7f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9aac0d0740 0x7f9aac157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 --2- 172.21.15.114:0/1642572946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9aac0cd240 0x7f9aac154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.745+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 >> 172.21.15.114:0/1642572946 conn(0x7f9aac0d1c20 msgr2=0x7f9aac06ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.746+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 shutdown_connections 2024-03-06T07:28:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.746+0000 7f9ab2e00640 1 -- 172.21.15.114:0/1642572946 wait complete. 2024-03-06T07:28:54.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:54.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.978+0000 7f2d0d000640 1 Processor -- start 2024-03-06T07:28:54.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.978+0000 7f2d0d000640 1 -- start start 2024-03-06T07:28:54.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.979+0000 7f2d0d000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2d08161820 0x7f2d080d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.979+0000 7f2d0d000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d080cfd20 0x7f2d080d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.980+0000 7f2d0d000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d080b2e00 con 0x7f2d080d2810 2024-03-06T07:28:54.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.980+0000 7f2d0d000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d080cf5b0 con 0x7f2d08161820 2024-03-06T07:28:54.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.980+0000 7f2d0d000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080d4b50 con 0x7f2d080cfd20 2024-03-06T07:28:54.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.980+0000 7f2d05e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2d08161820 0x7f2d080d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.980+0000 7f2d06800640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f2d080d2810 0x7f2d080b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:38882/0 (socket says 172.21.15.114:38882) 2024-03-06T07:28:54.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.981+0000 7f2d05400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d080cfd20 0x7f2d080d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.981+0000 7f2d06800640 1 -- 172.21.15.114:0/3306466168 learned_addr learned my addr 172.21.15.114:0/3306466168 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.981+0000 7f2d05e00640 1 -- 172.21.15.114:0/3306466168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d080cfd20 msgr2=0x7f2d080d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.981+0000 7f2d05e00640 1 --2- 172.21.15.114:0/3306466168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d080cfd20 0x7f2d080d22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.981+0000 7f2d05e00640 1 -- 172.21.15.114:0/3306466168 >> v1:172.21.15.177:6789/0 conn(0x7f2d080d2810 legacy=0x7f2d080b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.981+0000 7f2d05e00640 1 -- 172.21.15.114:0/3306466168 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080cf800 con 0x7f2d08161820 2024-03-06T07:28:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.983+0000 7f2d05e00640 1 --2- 172.21.15.114:0/3306466168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d08161820 0x7f2d080d1d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009d90 tx=0x7f2d0002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23830b2852975949 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.983+0000 7f2d04a00640 1 -- 172.21.15.114:0/3306466168 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00039070 con 0x7f2d08161820 2024-03-06T07:28:54.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.983+0000 7f2d04a00640 1 -- 172.21.15.114:0/3306466168 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0002cbd0 con 0x7f2d08161820 2024-03-06T07:28:54.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.983+0000 7f2d04a00640 1 -- 172.21.15.114:0/3306466168 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00034440 con 0x7f2d08161820 2024-03-06T07:28:54.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 -- 172.21.15.114:0/3306466168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d08161820 msgr2=0x7f2d080d1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 --2- 172.21.15.114:0/3306466168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d08161820 0x7f2d080d1d90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009d90 tx=0x7f2d0002be20 comp rx=0 tx=0).stop 2024-03-06T07:28:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 -- 172.21.15.114:0/3306466168 shutdown_connections 2024-03-06T07:28:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 --2- 172.21.15.114:0/3306466168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d080cfd20 0x7f2d080d22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 --2- 172.21.15.114:0/3306466168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d08161820 0x7f2d080d1d90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 -- 172.21.15.114:0/3306466168 >> 172.21.15.114:0/3306466168 conn(0x7f2d080d3e90 msgr2=0x7f2d080d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 -- 172.21.15.114:0/3306466168 shutdown_connections 2024-03-06T07:28:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.984+0000 7f2d0d000640 1 -- 172.21.15.114:0/3306466168 wait complete. 2024-03-06T07:28:54.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.985+0000 7f2d0d000640 1 Processor -- start 2024-03-06T07:28:54.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.986+0000 7f2d0d000640 1 -- start start 2024-03-06T07:28:54.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.986+0000 7f2d0d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d080cfd20 0x7f2d08150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.986+0000 7f2d0d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d080d2810 0x7f2d080caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.986+0000 7f2d0d000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d08161820 0x7f2d080b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.986+0000 7f2d0d000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080d13b0 con 0x7f2d080cfd20 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.986+0000 7f2d0d000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08161c00 con 0x7f2d080d2810 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.986+0000 7f2d0d000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080d1530 con 0x7f2d08161820 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d080cfd20 0x7f2d08150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d080cfd20 0x7f2d08150360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51380/0 (socket says 172.21.15.114:51380) 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 -- 172.21.15.114:0/4100361488 learned_addr learned my addr 172.21.15.114:0/4100361488 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05400640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d080d2810 0x7f2d080caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d06800640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d08161820 0x7f2d080b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 -- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d08161820 msgr2=0x7f2d080b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d08161820 0x7f2d080b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 -- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d080d2810 msgr2=0x7f2d080caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d080d2810 0x7f2d080caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d05e00640 1 -- 172.21.15.114:0/4100361488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080cb8d0 con 0x7f2d080cfd20 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.987+0000 7f2d06800640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d08161820 0x7f2d080b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.988+0000 7f2d05e00640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d080cfd20 0x7f2d08150360 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009eb0 tx=0x7f2d00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.988+0000 7f2cfe000640 1 -- 172.21.15.114:0/4100361488 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00039050 con 0x7f2d080cfd20 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.988+0000 7f2cfe000640 1 -- 172.21.15.114:0/4100361488 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d00045070 con 0x7f2d080cfd20 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.988+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d080cbbb0 con 0x7f2d080cfd20 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.988+0000 7f2cfe000640 1 -- 172.21.15.114:0/4100361488 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00009450 con 0x7f2d080cfd20 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.988+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d080cc170 con 0x7f2d080cfd20 2024-03-06T07:28:54.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.989+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d080cc5b0 con 0x7f2d080cfd20 2024-03-06T07:28:54.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.990+0000 7f2cfe000640 1 -- 172.21.15.114:0/4100361488 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d0003e850 con 0x7f2d080cfd20 2024-03-06T07:28:54.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.991+0000 7f2cfe000640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ce406b4d0 0x7f2ce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.991+0000 7f2cfe000640 1 -- 172.21.15.114:0/4100361488 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f2d000acf00 con 0x7f2d080cfd20 2024-03-06T07:28:54.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.991+0000 7f2d05400640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ce406b4d0 0x7f2ce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.992+0000 7f2d05400640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ce406b4d0 0x7f2ce406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2d080b2460 tx=0x7f2cf0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:54.996+0000 7f2cfe000640 1 -- 172.21.15.114:0/4100361488 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d00032e20 con 0x7f2d080cfd20 2024-03-06T07:28:55.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.195+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2d080d0350 con 0x7f2d080cfd20 2024-03-06T07:28:55.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.751+0000 7f2cfe000640 1 -- 172.21.15.114:0/4100361488 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2d0007bd20 con 0x7f2d080cfd20 2024-03-06T07:28:55.755 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.755+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ce406b4d0 msgr2=0x7f2ce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.755+0000 7f2d0d000640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ce406b4d0 0x7f2ce406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2d080b2460 tx=0x7f2cf0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.755+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d080cfd20 msgr2=0x7f2d08150360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.755+0000 7f2d0d000640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d080cfd20 0x7f2d08150360 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009eb0 tx=0x7f2d00004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 shutdown_connections 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d08161820 0x7f2d080b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d080d2810 0x7f2d080caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ce406b4d0 0x7f2ce406d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 --2- 172.21.15.114:0/4100361488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d080cfd20 0x7f2d08150360 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 >> 172.21.15.114:0/4100361488 conn(0x7f2d080d3e90 msgr2=0x7f2d080d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 shutdown_connections 2024-03-06T07:28:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:55.756+0000 7f2d0d000640 1 -- 172.21.15.114:0/4100361488 wait complete. 2024-03-06T07:28:55.786 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:55.786 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:55.786 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:55.786 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:55.786 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:55.786 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:55.787 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:55.787 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:55.787 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:55.787 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:55.787 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:55.787 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:55.787 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:55.787 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:55.788 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:55.788 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:55.808 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:55.809 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:55.809 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:55.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:55.812 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:55.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:55.814 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:55.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:55.856 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:55.856 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:55.860 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:55.860 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:55.861 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:55.861 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:55.866 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:55.867 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:55.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:56.016 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:28:56.017 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:28:56.021 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:28:56.022 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:28:56.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.051+0000 7f3b46a00640 1 Processor -- start 2024-03-06T07:28:56.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.052+0000 7f3b46a00640 1 -- start start 2024-03-06T07:28:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.053+0000 7f3b46a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3b401444e0 0x7f3b400cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.053+0000 7f3b46a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b400c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.053+0000 7f3b46a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b400ae380 con 0x7f3b400c4e70 2024-03-06T07:28:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.053+0000 7f3b46a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b400c52d0 con 0x7f3b401444e0 2024-03-06T07:28:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.053+0000 7f3b46a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400ae660 con 0x7f3b401474c0 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.053+0000 7f3b3fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3b401444e0 0x7f3b400cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b3fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3b401444e0 0x7f3b400cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:46500/0 (socket says 172.21.15.114:46500) 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b44a00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f3b400c4e70 0x7f3b40161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:38906/0 (socket says 172.21.15.114:38906) 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b3fe00640 1 -- 172.21.15.114:0/2011749675 learned_addr learned my addr 172.21.15.114:0/2011749675 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b3f400640 1 --2- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b400c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b3fe00640 1 -- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 msgr2=0x7f3b400c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b3fe00640 1 --2- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b400c4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b3fe00640 1 -- 172.21.15.114:0/2011749675 >> v1:172.21.15.177:6789/0 conn(0x7f3b400c4e70 legacy=0x7f3b40161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.054+0000 7f3b3fe00640 1 -- 172.21.15.114:0/2011749675 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400ae940 con 0x7f3b401444e0 2024-03-06T07:28:56.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.055+0000 7f3b3fe00640 1 --2- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b401444e0 0x7f3b400cd010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009e50 tx=0x7f3b3002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6b16f1dfb2930d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.055+0000 7f3b3ea00640 1 -- 172.21.15.114:0/2011749675 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3002cb00 con 0x7f3b401444e0 2024-03-06T07:28:56.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.055+0000 7f3b3ea00640 1 -- 172.21.15.114:0/2011749675 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3002cca0 con 0x7f3b401444e0 2024-03-06T07:28:56.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.055+0000 7f3b3ea00640 1 -- 172.21.15.114:0/2011749675 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30034610 con 0x7f3b401444e0 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 -- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b401444e0 msgr2=0x7f3b400cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 --2- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b401444e0 0x7f3b400cd010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009e50 tx=0x7f3b3002bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 -- 172.21.15.114:0/2011749675 shutdown_connections 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 --2- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b400c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 --2- 172.21.15.114:0/2011749675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b401444e0 0x7f3b400cd010 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 -- 172.21.15.114:0/2011749675 >> 172.21.15.114:0/2011749675 conn(0x7f3b400cc4f0 msgr2=0x7f3b400cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 -- 172.21.15.114:0/2011749675 shutdown_connections 2024-03-06T07:28:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.056+0000 7f3b46a00640 1 -- 172.21.15.114:0/2011749675 wait complete. 2024-03-06T07:28:56.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.058+0000 7f3b46a00640 1 Processor -- start 2024-03-06T07:28:56.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.058+0000 7f3b46a00640 1 -- start start 2024-03-06T07:28:56.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.059+0000 7f3b46a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b400c4e70 0x7f3b40156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.059+0000 7f3b46a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b401444e0 0x7f3b400abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.059+0000 7f3b46a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b401f7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.059+0000 7f3b46a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b40147cb0 con 0x7f3b401444e0 2024-03-06T07:28:56.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.059+0000 7f3b46a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400aef00 con 0x7f3b400c4e70 2024-03-06T07:28:56.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.059+0000 7f3b46a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b401448c0 con 0x7f3b401474c0 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b401444e0 0x7f3b400abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b400c4e70 0x7f3b40156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b401444e0 0x7f3b400abf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51406/0 (socket says 172.21.15.114:51406) 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3f400640 1 -- 172.21.15.114:0/43053932 learned_addr learned my addr 172.21.15.114:0/43053932 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b44a00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b401f7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3fe00640 1 -- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 msgr2=0x7f3b401f7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3fe00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b401f7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3fe00640 1 -- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b401444e0 msgr2=0x7f3b400abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3fe00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b401444e0 0x7f3b400abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3fe00640 1 -- 172.21.15.114:0/43053932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4015e0d0 con 0x7f3b400c4e70 2024-03-06T07:28:56.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.060+0000 7f3b3f400640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b401444e0 0x7f3b400abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:56.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.061+0000 7f3b3fe00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b400c4e70 0x7f3b40156770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3b40073740 tx=0x7f3b30004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.061+0000 7f3b37e00640 1 -- 172.21.15.114:0/43053932 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30043020 con 0x7f3b400c4e70 2024-03-06T07:28:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.061+0000 7f3b37e00640 1 -- 172.21.15.114:0/43053932 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3002ce50 con 0x7f3b400c4e70 2024-03-06T07:28:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.061+0000 7f3b37e00640 1 -- 172.21.15.114:0/43053932 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30009510 con 0x7f3b400c4e70 2024-03-06T07:28:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.061+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4013de70 con 0x7f3b400c4e70 2024-03-06T07:28:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.061+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4013e430 con 0x7f3b400c4e70 2024-03-06T07:28:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.062+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4004f330 con 0x7f3b400c4e70 2024-03-06T07:28:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.063+0000 7f3b37e00640 1 -- 172.21.15.114:0/43053932 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b30046020 con 0x7f3b400c4e70 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.064+0000 7f3b37e00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1c06b4d0 0x7f3b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.064+0000 7f3b37e00640 1 -- 172.21.15.114:0/43053932 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3b300ac3b0 con 0x7f3b400c4e70 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.064+0000 7f3b3f400640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1c06b4d0 0x7f3b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.065+0000 7f3b3f400640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1c06b4d0 0x7f3b1c06d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3b400ad810 tx=0x7f3b28009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.070+0000 7f3b37e00640 1 -- 172.21.15.114:0/43053932 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b3004d030 con 0x7f3b400c4e70 2024-03-06T07:28:56.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.276+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3b400c5720 con 0x7f3b400c4e70 2024-03-06T07:28:56.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.780+0000 7f3b37e00640 1 -- 172.21.15.114:0/43053932 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3b3007ae40 con 0x7f3b400c4e70 2024-03-06T07:28:56.784 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1c06b4d0 msgr2=0x7f3b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1c06b4d0 0x7f3b1c06d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3b400ad810 tx=0x7f3b28009190 comp rx=0 tx=0).stop 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b400c4e70 msgr2=0x7f3b40156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b400c4e70 0x7f3b40156770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3b40073740 tx=0x7f3b30004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 shutdown_connections 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b401474c0 0x7f3b401f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1c06b4d0 0x7f3b1c06d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b401444e0 0x7f3b400abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 --2- 172.21.15.114:0/43053932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b400c4e70 0x7f3b40156770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.786+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 >> 172.21.15.114:0/43053932 conn(0x7f3b400cc4f0 msgr2=0x7f3b40070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.787+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 shutdown_connections 2024-03-06T07:28:56.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:56.787+0000 7f3b46a00640 1 -- 172.21.15.114:0/43053932 wait complete. 2024-03-06T07:28:56.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:57.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.006+0000 7fae62c00640 1 Processor -- start 2024-03-06T07:28:57.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.006+0000 7fae62c00640 1 -- start start 2024-03-06T07:28:57.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.007+0000 7fae62c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c1561f0 0x7fae5c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.007+0000 7fae62c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae5c0cc9d0 0x7fae5c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.007+0000 7fae62c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c15c240 con 0x7fae5c0aebf0 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.007+0000 7fae62c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c15ce30 con 0x7fae5c0cc9d0 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.007+0000 7fae62c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c15c500 con 0x7fae5c1561f0 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae62200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae5c0cc9d0 0x7fae5c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae60e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fae5c0aebf0 0x7fae5c0cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56938/0 (socket says 172.21.15.114:56938) 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae62200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae5c0cc9d0 0x7fae5c15f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51414/0 (socket says 172.21.15.114:51414) 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae60e00640 1 -- 172.21.15.114:0/1320734819 learned_addr learned my addr 172.21.15.114:0/1320734819 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:57.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae61800640 1 --2- 172.21.15.114:0/1320734819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c1561f0 0x7fae5c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae62200640 1 -- 172.21.15.114:0/1320734819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c1561f0 msgr2=0x7fae5c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae62200640 1 --2- 172.21.15.114:0/1320734819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c1561f0 0x7fae5c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae62200640 1 -- 172.21.15.114:0/1320734819 >> v1:172.21.15.114:6789/0 conn(0x7fae5c0aebf0 legacy=0x7fae5c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae62200640 1 -- 172.21.15.114:0/1320734819 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c15c7e0 con 0x7fae5c0cc9d0 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.008+0000 7fae5be00640 1 -- 172.21.15.114:0/1320734819 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229672500 0 0) 0x7fae5c15c240 con 0x7fae5c0aebf0 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.009+0000 7fae62200640 1 --2- 172.21.15.114:0/1320734819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0cc9d0 0x7fae5c15f830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fae54009d90 tx=0x7fae5402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943ff2e213c83ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae5be00640 1 -- 172.21.15.114:0/1320734819 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54039070 con 0x7fae5c0cc9d0 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae5be00640 1 -- 172.21.15.114:0/1320734819 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5402cc50 con 0x7fae5c0cc9d0 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae5be00640 1 -- 172.21.15.114:0/1320734819 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54034440 con 0x7fae5c0cc9d0 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae62c00640 1 -- 172.21.15.114:0/1320734819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0cc9d0 msgr2=0x7fae5c15f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae62c00640 1 --2- 172.21.15.114:0/1320734819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0cc9d0 0x7fae5c15f830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fae54009d90 tx=0x7fae5402bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae62c00640 1 -- 172.21.15.114:0/1320734819 shutdown_connections 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae62c00640 1 --2- 172.21.15.114:0/1320734819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0cc9d0 0x7fae5c15f830 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae62c00640 1 --2- 172.21.15.114:0/1320734819 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c1561f0 0x7fae5c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae62c00640 1 -- 172.21.15.114:0/1320734819 >> 172.21.15.114:0/1320734819 conn(0x7fae5c06afc0 msgr2=0x7fae5c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.010+0000 7fae62c00640 1 -- 172.21.15.114:0/1320734819 shutdown_connections 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.011+0000 7fae62c00640 1 -- 172.21.15.114:0/1320734819 wait complete. 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.012+0000 7fae62c00640 1 Processor -- start 2024-03-06T07:28:57.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.012+0000 7fae62c00640 1 -- start start 2024-03-06T07:28:57.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae62c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0aebf0 0x7fae5c0cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae62c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c0cc9d0 0x7fae5c158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae62c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae5c1561f0 0x7fae5c153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae62c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c156750 con 0x7fae5c0aebf0 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae62c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c1565d0 con 0x7fae5c1561f0 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae62c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c15fd70 con 0x7fae5c0cc9d0 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae61800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0aebf0 0x7fae5c0cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.013+0000 7fae61800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0aebf0 0x7fae5c0cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51428/0 (socket says 172.21.15.114:51428) 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae61800640 1 -- 172.21.15.114:0/2004640012 learned_addr learned my addr 172.21.15.114:0/2004640012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae61800640 1 -- 172.21.15.114:0/2004640012 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c0cc9d0 msgr2=0x7fae5c158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae61800640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c0cc9d0 0x7fae5c158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae61800640 1 -- 172.21.15.114:0/2004640012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae5c1561f0 msgr2=0x7fae5c153fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae61800640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae5c1561f0 0x7fae5c153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae61800640 1 -- 172.21.15.114:0/2004640012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c1585c0 con 0x7fae5c0aebf0 2024-03-06T07:28:57.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae61800640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0aebf0 0x7fae5c0cd290 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fae5000d620 tx=0x7fae5000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae59600640 1 -- 172.21.15.114:0/2004640012 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5000b620 con 0x7fae5c0aebf0 2024-03-06T07:28:57.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae59600640 1 -- 172.21.15.114:0/2004640012 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50019070 con 0x7fae5c0aebf0 2024-03-06T07:28:57.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.014+0000 7fae59600640 1 -- 172.21.15.114:0/2004640012 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5000ba00 con 0x7fae5c0aebf0 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.015+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c0cd870 con 0x7fae5c0aebf0 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.015+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c0cde30 con 0x7fae5c0aebf0 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.016+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c04f330 con 0x7fae5c0aebf0 2024-03-06T07:28:57.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.016+0000 7fae59600640 1 -- 172.21.15.114:0/2004640012 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae5000bba0 con 0x7fae5c0aebf0 2024-03-06T07:28:57.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.017+0000 7fae59600640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b4d0 0x7fae3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.017+0000 7fae59600640 1 -- 172.21.15.114:0/2004640012 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fae5008a370 con 0x7fae5c0aebf0 2024-03-06T07:28:57.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.018+0000 7fae60e00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b4d0 0x7fae3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.018+0000 7fae60e00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b4d0 0x7fae3c06d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fae48007fd0 tx=0x7fae48007900 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.022+0000 7fae59600640 1 -- 172.21.15.114:0/2004640012 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae500594a0 con 0x7fae5c0aebf0 2024-03-06T07:28:57.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.290+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fae5c0b2b10 con 0x7fae5c0aebf0 2024-03-06T07:28:57.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.291+0000 7fae59600640 1 -- 172.21.15.114:0/2004640012 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fae50058da0 con 0x7fae5c0aebf0 2024-03-06T07:28:57.293 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:28:57.293 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:28:57.294 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:28:57.296 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.297 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:28:57.298 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.300 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:28:57.301 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:28:57.302 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDiGuhlmWi+ARAASWXcRYEci7IMYqPIW8DVAA==", 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.305 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDiGuhlF4C+ARAAcJ8a89kmbY0rSkz0hFGgIA==", 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDiGuhly5S+ARAAmAdvA3J/drjkL8QRe73IWg==", 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:28:57.306 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDiGuhlW6m+ARAA59jdLWTOgdJvuLjs/AdQ7Q==", 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDiGuhl+L6+ARAA+BBh22MPReW5Ha44EhWIgQ==", 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:28:57.307 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDiGuhl8M6+ARAALuDJTGk0xBxjNthLF1RZTA==", 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:28:57.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:28:57.309 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.293+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b4d0 msgr2=0x7fae3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.293+0000 7fae62c00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b4d0 0x7fae3c06d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fae48007fd0 tx=0x7fae48007900 comp rx=0 tx=0).stop 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.293+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0aebf0 msgr2=0x7fae5c0cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.293+0000 7fae62c00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0aebf0 0x7fae5c0cd290 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fae5000d620 tx=0x7fae5000dae0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.293+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 shutdown_connections 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.293+0000 7fae62c00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae5c1561f0 0x7fae5c153fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.293+0000 7fae62c00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae5c0cc9d0 0x7fae5c158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.294+0000 7fae62c00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b4d0 0x7fae3c06d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.294+0000 7fae62c00640 1 --2- 172.21.15.114:0/2004640012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae5c0aebf0 0x7fae5c0cd290 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.294+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 >> 172.21.15.114:0/2004640012 conn(0x7fae5c06afc0 msgr2=0x7fae5c0b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.294+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 shutdown_connections 2024-03-06T07:28:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.294+0000 7fae62c00640 1 -- 172.21.15.114:0/2004640012 wait complete. 2024-03-06T07:28:57.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.519+0000 7ffbcf000640 1 Processor -- start 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.519+0000 7ffbcf000640 1 -- start start 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbcf000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffbc81444e0 0x7ffbc80cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbcf000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffbc81474c0 0x7ffbc80c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbcf000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc80ae380 con 0x7ffbc80c4e70 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbcf000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc80ae660 con 0x7ffbc81444e0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbcf000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc80ae940 con 0x7ffbc81474c0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbc7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffbc81444e0 0x7ffbc80cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbc7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffbc81444e0 0x7ffbc80cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51436/0 (socket says 172.21.15.114:51436) 2024-03-06T07:28:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.521+0000 7ffbc7e00640 1 -- 172.21.15.114:0/1666469492 learned_addr learned my addr 172.21.15.114:0/1666469492 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.522+0000 7ffbc7e00640 1 -- 172.21.15.114:0/1666469492 >> v2:172.21.15.177:3300/0 conn(0x7ffbc81474c0 msgr2=0x7ffbc80c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.522+0000 7ffbc7400640 1 --2- 172.21.15.114:0/1666469492 >> v2:172.21.15.177:3300/0 conn(0x7ffbc81474c0 0x7ffbc80c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.522+0000 7ffbc7e00640 1 --2- 172.21.15.114:0/1666469492 >> v2:172.21.15.177:3300/0 conn(0x7ffbc81474c0 0x7ffbc80c4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.522+0000 7ffbc7e00640 1 -- 172.21.15.114:0/1666469492 >> v1:172.21.15.177:6789/0 conn(0x7ffbc80c4e70 legacy=0x7ffbc8161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.522+0000 7ffbc7e00640 1 -- 172.21.15.114:0/1666469492 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc80aec20 con 0x7ffbc81444e0 2024-03-06T07:28:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.522+0000 7ffbc7400640 1 --2- 172.21.15.114:0/1666469492 >> v2:172.21.15.177:3300/0 conn(0x7ffbc81474c0 0x7ffbc80c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.523+0000 7ffbc7e00640 1 --2- 172.21.15.114:0/1666469492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc81444e0 0x7ffbc80cd010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8068c90 tx=0x7ffbb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537ed51709b5b9e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.523+0000 7ffbc6a00640 1 -- 172.21.15.114:0/1666469492 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8035020 con 0x7ffbc81444e0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.523+0000 7ffbc6a00640 1 -- 172.21.15.114:0/1666469492 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb8031870 con 0x7ffbc81444e0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.523+0000 7ffbc6a00640 1 -- 172.21.15.114:0/1666469492 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8031c40 con 0x7ffbc81444e0 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 -- 172.21.15.114:0/1666469492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc81444e0 msgr2=0x7ffbc80cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 --2- 172.21.15.114:0/1666469492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc81444e0 0x7ffbc80cd010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8068c90 tx=0x7ffbb802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 -- 172.21.15.114:0/1666469492 shutdown_connections 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 --2- 172.21.15.114:0/1666469492 >> v2:172.21.15.177:3300/0 conn(0x7ffbc81474c0 0x7ffbc80c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 --2- 172.21.15.114:0/1666469492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc81444e0 0x7ffbc80cd010 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 -- 172.21.15.114:0/1666469492 >> 172.21.15.114:0/1666469492 conn(0x7ffbc80cc4f0 msgr2=0x7ffbc80cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 -- 172.21.15.114:0/1666469492 shutdown_connections 2024-03-06T07:28:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.524+0000 7ffbcf000640 1 -- 172.21.15.114:0/1666469492 wait complete. 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.526+0000 7ffbcf000640 1 Processor -- start 2024-03-06T07:28:57.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.526+0000 7ffbcf000640 1 -- start start 2024-03-06T07:28:57.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbcf000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80c4e70 0x7ffbc815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbcf000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbc81444e0 0x7ffbc80b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbcf000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc81474c0 0x7ffbc80b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbcf000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc81448c0 con 0x7ffbc80c4e70 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbcf000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc8147cb0 con 0x7ffbc81474c0 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbcf000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc8144a40 con 0x7ffbc81444e0 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbc7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80c4e70 0x7ffbc815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbc7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80c4e70 0x7ffbc815ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51450/0 (socket says 172.21.15.114:51450) 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.527+0000 7ffbc7e00640 1 -- 172.21.15.114:0/3206176930 learned_addr learned my addr 172.21.15.114:0/3206176930 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbc7e00640 1 -- 172.21.15.114:0/3206176930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbc81444e0 msgr2=0x7ffbc80b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbc7e00640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbc81444e0 0x7ffbc80b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbc7e00640 1 -- 172.21.15.114:0/3206176930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc81474c0 msgr2=0x7ffbc80b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbc7e00640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc81474c0 0x7ffbc80b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbc7e00640 1 -- 172.21.15.114:0/3206176930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc80b5010 con 0x7ffbc80c4e70 2024-03-06T07:28:57.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbc7e00640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80c4e70 0x7ffbc815ddc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8000c00 tx=0x7ffbb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbbfe00640 1 -- 172.21.15.114:0/3206176930 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8040070 con 0x7ffbc80c4e70 2024-03-06T07:28:57.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbbfe00640 1 -- 172.21.15.114:0/3206176930 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb803a420 con 0x7ffbc80c4e70 2024-03-06T07:28:57.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbbfe00640 1 -- 172.21.15.114:0/3206176930 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8035040 con 0x7ffbc80c4e70 2024-03-06T07:28:57.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc80b52f0 con 0x7ffbc80c4e70 2024-03-06T07:28:57.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.528+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc806d8a0 con 0x7ffbc80c4e70 2024-03-06T07:28:57.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.530+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc804f330 con 0x7ffbc80c4e70 2024-03-06T07:28:57.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.531+0000 7ffbbfe00640 1 -- 172.21.15.114:0/3206176930 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbb804b060 con 0x7ffbc80c4e70 2024-03-06T07:28:57.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.531+0000 7ffbbfe00640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffba806b500 0x7ffba806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.531+0000 7ffbbfe00640 1 -- 172.21.15.114:0/3206176930 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffbb80abde0 con 0x7ffbc80c4e70 2024-03-06T07:28:57.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.532+0000 7ffbc7400640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffba806b500 0x7ffba806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.532+0000 7ffbc7400640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffba806b500 0x7ffba806d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80b1960 tx=0x7ffbb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.536+0000 7ffbbfe00640 1 -- 172.21.15.114:0/3206176930 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbb807b040 con 0x7ffbc80c4e70 2024-03-06T07:28:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.808+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7ffbc80af530 con 0x7ffbc80c4e70 2024-03-06T07:28:57.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.813+0000 7ffbbfe00640 1 -- 172.21.15.114:0/3206176930 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7ffbb807a940 con 0x7ffbc80c4e70 2024-03-06T07:28:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.815+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffba806b500 msgr2=0x7ffba806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.815+0000 7ffbcf000640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffba806b500 0x7ffba806d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80b1960 tx=0x7ffbb4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.815+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80c4e70 msgr2=0x7ffbc815ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.815+0000 7ffbcf000640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80c4e70 0x7ffbc815ddc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8000c00 tx=0x7ffbb80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 shutdown_connections 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc81474c0 0x7ffbc80b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbc81444e0 0x7ffbc80b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffba806b500 0x7ffba806d9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 --2- 172.21.15.114:0/3206176930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80c4e70 0x7ffbc815ddc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 >> 172.21.15.114:0/3206176930 conn(0x7ffbc80cc4f0 msgr2=0x7ffbc8071b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 shutdown_connections 2024-03-06T07:28:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:57.816+0000 7ffbcf000640 1 -- 172.21.15.114:0/3206176930 wait complete. 2024-03-06T07:28:57.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:28:58.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.031+0000 7f9e0a800640 1 Processor -- start 2024-03-06T07:28:58.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.032+0000 7f9e0a800640 1 -- start start 2024-03-06T07:28:58.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.033+0000 7f9e0a800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9e0406d0a0 0x7f9e040ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.033+0000 7f9e0a800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e040b4570 con 0x7f9e0415e2c0 2024-03-06T07:28:58.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.033+0000 7f9e0a800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e040d0260 con 0x7f9e0406c0d0 2024-03-06T07:28:58.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.033+0000 7f9e0a800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e040b4320 con 0x7f9e0406d0a0 2024-03-06T07:28:58.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e03400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9e0415e2c0 0x7f9e040cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51532/0 (socket says 172.21.15.114:51532) 2024-03-06T07:28:58.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e03400640 1 -- 172.21.15.114:0/2335298673 learned_addr learned my addr 172.21.15.114:0/2335298673 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:58.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e03e00640 1 --2- 172.21.15.114:0/2335298673 >> v2:172.21.15.177:3300/0 conn(0x7f9e0406d0a0 0x7f9e040ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665710759 0 0) 0x7f9e040b4570 con 0x7f9e0415e2c0 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de8003520 con 0x7f9e0415e2c0 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4294533959 0 0) 0x7f9de8003520 con 0x7f9e0415e2c0 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 >> v2:172.21.15.177:3300/0 conn(0x7f9e0406d0a0 msgr2=0x7f9e040ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 --2- 172.21.15.114:0/2335298673 >> v2:172.21.15.177:3300/0 conn(0x7f9e0406d0a0 0x7f9e040ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 >> v1:172.21.15.177:6789/0 conn(0x7f9e0406c0d0 legacy=0x7f9e040d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e040cff10 con 0x7f9e0415e2c0 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f9df4001670 con 0x7f9e0415e2c0 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.034+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 >> v1:172.21.15.114:6789/0 conn(0x7f9e0415e2c0 legacy=0x7f9e040cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02000640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de8003a00 0x7f9de8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02000640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9de8004800 0x7f9de8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02000640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de80051c0 0x7f9de800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040b4320 con 0x7f9de8003a00 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de800c0c0 con 0x7f9de80051c0 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de800c2c0 con 0x7f9de8004800 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03400640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de8003a00 0x7f9de8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03e00640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de80051c0 0x7f9de800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02a00640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9de8004800 0x7f9de8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03400640 1 -- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9de8004800 msgr2=0x7f9de8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03400640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9de8004800 0x7f9de8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03400640 1 -- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de80051c0 msgr2=0x7f9de800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03400640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de80051c0 0x7f9de800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03400640 1 -- 172.21.15.114:0/2335298673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8003520 con 0x7f9de8003a00 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e02a00640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9de8004800 0x7f9de8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03e00640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de80051c0 0x7f9de800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.035+0000 7f9e03e00640 1 -- 172.21.15.114:0/2335298673 reap_dead start 2024-03-06T07:28:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.036+0000 7f9e03400640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de8003a00 0x7f9de8003e20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c190 tx=0x7f9de8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286067ac00eba86f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4007c60 con 0x7f9de8003a00 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df4030070 con 0x7f9de8003a00 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e02000640 1 -- 172.21.15.114:0/2335298673 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4005d50 con 0x7f9de8003a00 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e0a800640 1 -- 172.21.15.114:0/2335298673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de8003a00 msgr2=0x7f9de8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e0a800640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de8003a00 0x7f9de8003e20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c190 tx=0x7f9de8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e0a800640 1 -- 172.21.15.114:0/2335298673 shutdown_connections 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e0a800640 1 --2- 172.21.15.114:0/2335298673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de8003a00 0x7f9de8003e20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e0a800640 1 -- 172.21.15.114:0/2335298673 >> 172.21.15.114:0/2335298673 conn(0x7f9e040b3800 msgr2=0x7f9e040b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e0a800640 1 -- 172.21.15.114:0/2335298673 shutdown_connections 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.037+0000 7f9e0a800640 1 -- 172.21.15.114:0/2335298673 wait complete. 2024-03-06T07:28:58.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.039+0000 7f9e0a800640 1 Processor -- start 2024-03-06T07:28:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.039+0000 7f9e0a800640 1 -- start start 2024-03-06T07:28:58.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e0a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de8003a00 0x7f9e0409f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e0a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0415afd0 0x7f9e040aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e0a800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e0415b450 0x7f9e040b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e0a800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040cc320 con 0x7f9e0415afd0 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e0a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040d05d0 con 0x7f9de8003a00 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e0a800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040cc4a0 con 0x7f9e0415b450 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e02a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0415afd0 0x7f9e040aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.040+0000 7f9e02a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0415afd0 0x7f9e040aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42086/0 (socket says 172.21.15.114:42086) 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e02a00640 1 -- 172.21.15.114:0/3602930655 learned_addr learned my addr 172.21.15.114:0/3602930655 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e02a00640 1 -- 172.21.15.114:0/3602930655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e0415b450 msgr2=0x7f9e040b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e02a00640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e0415b450 0x7f9e040b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e02a00640 1 -- 172.21.15.114:0/3602930655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de8003a00 msgr2=0x7f9e0409f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e02a00640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de8003a00 0x7f9e0409f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e02a00640 1 -- 172.21.15.114:0/3602930655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0409fb90 con 0x7f9e0415afd0 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e02a00640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0415afd0 0x7f9e040aff40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9df000d400 tx=0x7f9df000d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9dfb400640 1 -- 172.21.15.114:0/3602930655 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0018070 con 0x7f9e0415afd0 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9dfb400640 1 -- 172.21.15.114:0/3602930655 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df0003610 con 0x7f9e0415afd0 2024-03-06T07:28:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9dfb400640 1 -- 172.21.15.114:0/3602930655 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0003a40 con 0x7f9e0415afd0 2024-03-06T07:28:58.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.041+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e041f77a0 con 0x7f9e0415afd0 2024-03-06T07:28:58.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.042+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e041f7d60 con 0x7f9e0415afd0 2024-03-06T07:28:58.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.043+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0404f330 con 0x7f9e0415afd0 2024-03-06T07:28:58.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.043+0000 7f9dfb400640 1 -- 172.21.15.114:0/3602930655 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9df001b070 con 0x7f9e0415afd0 2024-03-06T07:28:58.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.044+0000 7f9dfb400640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9de406b4d0 0x7f9de406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.044+0000 7f9dfb400640 1 -- 172.21.15.114:0/3602930655 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9df0089580 con 0x7f9e0415afd0 2024-03-06T07:28:58.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.045+0000 7f9e03400640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9de406b4d0 0x7f9de406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.045+0000 7f9e03400640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9de406b4d0 0x7f9de406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c160 tx=0x7f9df4007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.049+0000 7f9dfb400640 1 -- 172.21.15.114:0/3602930655 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9df00580f0 con 0x7f9e0415afd0 2024-03-06T07:28:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.315+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f9e040b0c00 con 0x7f9e0415afd0 2024-03-06T07:28:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.321+0000 7f9dfb400640 1 -- 172.21.15.114:0/3602930655 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9df005c190 con 0x7f9e0415afd0 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9de406b4d0 msgr2=0x7f9de406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9de406b4d0 0x7f9de406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c160 tx=0x7f9df4007000 comp rx=0 tx=0).stop 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0415afd0 msgr2=0x7f9e040aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0415afd0 0x7f9e040aff40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9df000d400 tx=0x7f9df000d8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 shutdown_connections 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e0415b450 0x7f9e040b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9de406b4d0 0x7f9de406d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0415afd0 0x7f9e040aff40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 --2- 172.21.15.114:0/3602930655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9de8003a00 0x7f9e0409f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 >> 172.21.15.114:0/3602930655 conn(0x7f9e040b3800 msgr2=0x7f9e040b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 shutdown_connections 2024-03-06T07:28:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.324+0000 7f9e0a800640 1 -- 172.21.15.114:0/3602930655 wait complete. 2024-03-06T07:28:58.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:28:58.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.553+0000 7fbac3c00640 1 Processor -- start 2024-03-06T07:28:58.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.553+0000 7fbac3c00640 1 -- start start 2024-03-06T07:28:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.554+0000 7fbac3c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbabc0c5a20 0x7fbabc06e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.554+0000 7fbac3c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbabc15f830 con 0x7fbabc15b070 2024-03-06T07:28:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.555+0000 7fbac3c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbabc0cc000 con 0x7fbabc15ac90 2024-03-06T07:28:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.555+0000 7fbac3c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbabc0ce360 con 0x7fbabc0c5a20 2024-03-06T07:28:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.555+0000 7fbab3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbabc15b070 0x7fbabc15f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51540/0 (socket says 172.21.15.114:51540) 2024-03-06T07:28:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.555+0000 7fbab3e00640 1 -- 172.21.15.114:0/2870776640 learned_addr learned my addr 172.21.15.114:0/2870776640 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:28:58.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.555+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2870776640 >> v2:172.21.15.177:3300/0 conn(0x7fbabc0c5a20 0x7fbabc06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.555+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332954230 0 0) 0x7fbabc15f830 con 0x7fbabc15b070 2024-03-06T07:28:58.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.555+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa0003520 con 0x7fbabc15b070 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188307811 0 0) 0x7fbaa0003520 con 0x7fbabc15b070 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 >> v2:172.21.15.177:3300/0 conn(0x7fbabc0c5a20 msgr2=0x7fbabc06e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 --2- 172.21.15.114:0/2870776640 >> v2:172.21.15.177:3300/0 conn(0x7fbabc0c5a20 0x7fbabc06e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 >> v1:172.21.15.177:6789/0 conn(0x7fbabc15ac90 legacy=0x7fbabc15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc0ce640 con 0x7fbabc15b070 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fbaa4001670 con 0x7fbabc15b070 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 >> v1:172.21.15.114:6789/0 conn(0x7fbabc15b070 legacy=0x7fbabc15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00039b0 0x7fbaa0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa0004800 0x7fbaa0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa00051c0 0x7fbaa000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc0ce360 con 0x7fbaa00051c0 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa000e370 con 0x7fbaa0004800 2024-03-06T07:28:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa000e570 con 0x7fbaa00039b0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa00051c0 0x7fbaa000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbab3e00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa0004800 0x7fbaa0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbac1c00640 1 -- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00039b0 msgr2=0x7fbaa0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.556+0000 7fbac1200640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00039b0 0x7fbaa0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00039b0 0x7fbaa0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbac1c00640 1 -- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa0004800 msgr2=0x7fbaa0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa0004800 0x7fbaa0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbac1c00640 1 -- 172.21.15.114:0/2870776640 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa0003520 con 0x7fbaa00051c0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbac1200640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00039b0 0x7fbaa0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbab3e00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa0004800 0x7fbaa0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.557+0000 7fbac1c00640 1 -- 172.21.15.114:0/2870776640 reap_dead start 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.558+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa00051c0 0x7fbaa000dbd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009f90 tx=0x7fbaa0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6485335c37ca9bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.558+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4039070 con 0x7fbaa00051c0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.558+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab402cc20 con 0x7fbaa00051c0 2024-03-06T07:28:58.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.558+0000 7fbabbe00640 1 -- 172.21.15.114:0/2870776640 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4009c10 con 0x7fbaa00051c0 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.558+0000 7fbac3c00640 1 -- 172.21.15.114:0/2870776640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa00051c0 msgr2=0x7fbaa000dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.558+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa00051c0 0x7fbaa000dbd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009f90 tx=0x7fbaa0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.559+0000 7fbac3c00640 1 -- 172.21.15.114:0/2870776640 shutdown_connections 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.559+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2870776640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa00051c0 0x7fbaa000dbd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.559+0000 7fbac3c00640 1 -- 172.21.15.114:0/2870776640 >> 172.21.15.114:0/2870776640 conn(0x7fbabc0cc4f0 msgr2=0x7fbabc0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.559+0000 7fbac3c00640 1 -- 172.21.15.114:0/2870776640 shutdown_connections 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.559+0000 7fbac3c00640 1 -- 172.21.15.114:0/2870776640 wait complete. 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.560+0000 7fbac3c00640 1 Processor -- start 2024-03-06T07:28:58.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.560+0000 7fbac3c00640 1 -- start start 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.561+0000 7fbac3c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00051c0 0x7fbabc145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.561+0000 7fbac3c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbabc0c6ce0 0x7fbabc149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.561+0000 7fbac3c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbabc0ad860 0x7fbabc0add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.561+0000 7fbac3c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc0ceee0 con 0x7fbabc0ad860 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.561+0000 7fbac3c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc0cec00 con 0x7fbabc0c6ce0 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.561+0000 7fbac3c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbabc15ea10 con 0x7fbaa00051c0 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbabc0ad860 0x7fbabc0add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbabc0ad860 0x7fbabc0add10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42098/0 (socket says 172.21.15.114:42098) 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 -- 172.21.15.114:0/2039918440 learned_addr learned my addr 172.21.15.114:0/2039918440 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbab3e00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbabc0c6ce0 0x7fbabc149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 -- 172.21.15.114:0/2039918440 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00051c0 msgr2=0x7fbabc145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00051c0 0x7fbabc145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 -- 172.21.15.114:0/2039918440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbabc0c6ce0 msgr2=0x7fbabc149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbabc0c6ce0 0x7fbabc149050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 -- 172.21.15.114:0/2039918440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc149590 con 0x7fbabc0ad860 2024-03-06T07:28:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.562+0000 7fbac1c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbabc0ad860 0x7fbabc0add10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbab402c690 tx=0x7fbab4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.563+0000 7fbab9600640 1 -- 172.21.15.114:0/2039918440 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4046070 con 0x7fbabc0ad860 2024-03-06T07:28:58.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.563+0000 7fbab9600640 1 -- 172.21.15.114:0/2039918440 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab403e480 con 0x7fbabc0ad860 2024-03-06T07:28:58.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.563+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbabc1558b0 con 0x7fbabc0ad860 2024-03-06T07:28:58.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.563+0000 7fbab9600640 1 -- 172.21.15.114:0/2039918440 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4039040 con 0x7fbabc0ad860 2024-03-06T07:28:58.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.563+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbabc0c98d0 con 0x7fbabc0ad860 2024-03-06T07:28:58.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.564+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc04f330 con 0x7fbabc0ad860 2024-03-06T07:28:58.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.565+0000 7fbab9600640 1 -- 172.21.15.114:0/2039918440 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbab4049070 con 0x7fbabc0ad860 2024-03-06T07:28:58.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.566+0000 7fbab9600640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9806b500 0x7fba9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.566+0000 7fbab9600640 1 -- 172.21.15.114:0/2039918440 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbab40ab740 con 0x7fbabc0ad860 2024-03-06T07:28:58.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.566+0000 7fbac1200640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9806b500 0x7fba9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.567+0000 7fbac1200640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9806b500 0x7fba9806d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbabc15e270 tx=0x7fbaac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.571+0000 7fbab9600640 1 -- 172.21.15.114:0/2039918440 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbabc149590 con 0x7fbabc0ad860 2024-03-06T07:28:58.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.838+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fbabc0c7130 con 0x7fbabc0ad860 2024-03-06T07:28:58.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.843+0000 7fbab9600640 1 -- 172.21.15.114:0/2039918440 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fbab407a2a0 con 0x7fbabc0ad860 2024-03-06T07:28:58.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9806b500 msgr2=0x7fba9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9806b500 0x7fba9806d9c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbabc15e270 tx=0x7fbaac009040 comp rx=0 tx=0).stop 2024-03-06T07:28:58.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbabc0ad860 msgr2=0x7fbabc0add10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbabc0ad860 0x7fbabc0add10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbab402c690 tx=0x7fbab4009750 comp rx=0 tx=0).stop 2024-03-06T07:28:58.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 shutdown_connections 2024-03-06T07:28:58.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9806b500 0x7fba9806d9c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbabc0ad860 0x7fbabc0add10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbabc0c6ce0 0x7fbabc149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 --2- 172.21.15.114:0/2039918440 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa00051c0 0x7fbabc145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 >> 172.21.15.114:0/2039918440 conn(0x7fbabc0cc4f0 msgr2=0x7fbabc071f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.846+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 shutdown_connections 2024-03-06T07:28:58.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:58.847+0000 7fbac3c00640 1 -- 172.21.15.114:0/2039918440 wait complete. 2024-03-06T07:28:58.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:28:59.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.066+0000 7f8be2600640 1 Processor -- start 2024-03-06T07:28:59.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.066+0000 7f8be2600640 1 -- start start 2024-03-06T07:28:59.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.067+0000 7f8be2600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0cd1b0 0x7f8bdc072200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.067+0000 7f8be2600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8bdc0b03a0 0x7f8bdc06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.068+0000 7f8be2600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc0cb820 con 0x7f8bdc071e20 2024-03-06T07:28:59.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.068+0000 7f8be2600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cc3d0 con 0x7f8bdc0b03a0 2024-03-06T07:28:59.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.068+0000 7f8be2600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc072740 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.068+0000 7f8bdbe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8bdc0b03a0 0x7f8bdc06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.068+0000 7f8bd3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0cd1b0 0x7f8bdc072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.068+0000 7f8bdbe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8bdc0b03a0 0x7f8bdc06dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:52318/0 (socket says 172.21.15.114:52318) 2024-03-06T07:28:59.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.068+0000 7f8bdbe00640 1 -- 172.21.15.114:0/348923148 learned_addr learned my addr 172.21.15.114:0/348923148 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:59.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.069+0000 7f8bdbe00640 1 -- 172.21.15.114:0/348923148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0cd1b0 msgr2=0x7f8bdc072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.069+0000 7f8bdbe00640 1 --2- 172.21.15.114:0/348923148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0cd1b0 0x7f8bdc072200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.069+0000 7f8bdbe00640 1 -- 172.21.15.114:0/348923148 >> v1:172.21.15.177:6789/0 conn(0x7f8bdc071e20 legacy=0x7f8bdc0cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.069+0000 7f8bdbe00640 1 -- 172.21.15.114:0/348923148 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc0cbaa0 con 0x7f8bdc0b03a0 2024-03-06T07:28:59.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.070+0000 7f8bdbe00640 1 --2- 172.21.15.114:0/348923148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc0b03a0 0x7f8bdc06dba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009e50 tx=0x7f8bd402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f569ecfda4e807a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.071+0000 7f8bdaa00640 1 -- 172.21.15.114:0/348923148 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd402cb00 con 0x7f8bdc0b03a0 2024-03-06T07:28:59.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.071+0000 7f8bdaa00640 1 -- 172.21.15.114:0/348923148 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd402cca0 con 0x7f8bdc0b03a0 2024-03-06T07:28:59.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.071+0000 7f8bdaa00640 1 -- 172.21.15.114:0/348923148 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4034610 con 0x7f8bdc0b03a0 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 -- 172.21.15.114:0/348923148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc0b03a0 msgr2=0x7f8bdc06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 --2- 172.21.15.114:0/348923148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc0b03a0 0x7f8bdc06dba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009e50 tx=0x7f8bd402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 -- 172.21.15.114:0/348923148 shutdown_connections 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 --2- 172.21.15.114:0/348923148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc0b03a0 0x7f8bdc06dba0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 --2- 172.21.15.114:0/348923148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0cd1b0 0x7f8bdc072200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 -- 172.21.15.114:0/348923148 >> 172.21.15.114:0/348923148 conn(0x7f8bdc0d0df0 msgr2=0x7f8bdc0d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 -- 172.21.15.114:0/348923148 shutdown_connections 2024-03-06T07:28:59.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.072+0000 7f8be2600640 1 -- 172.21.15.114:0/348923148 wait complete. 2024-03-06T07:28:59.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.074+0000 7f8be2600640 1 Processor -- start 2024-03-06T07:28:59.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.074+0000 7f8be2600640 1 -- start start 2024-03-06T07:28:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8be2600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc071e20 0x7f8bdc15c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8be2600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0b03a0 0x7f8bdc0af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8be2600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bdc0cd1b0 0x7f8bdc0af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8be2600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc15b080 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8be2600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cc060 con 0x7f8bdc071e20 2024-03-06T07:28:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8be2600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc072b10 con 0x7f8bdc0b03a0 2024-03-06T07:28:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bdc0cd1b0 0x7f8bdc0af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bdc0cd1b0 0x7f8bdc0af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42100/0 (socket says 172.21.15.114:42100) 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 -- 172.21.15.114:0/1858426698 learned_addr learned my addr 172.21.15.114:0/1858426698 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdb400640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc071e20 0x7f8bdc15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bd3e00640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0b03a0 0x7f8bdc0af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 -- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0b03a0 msgr2=0x7f8bdc0af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0b03a0 0x7f8bdc0af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 -- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc071e20 msgr2=0x7f8bdc15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc071e20 0x7f8bdc15c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.075+0000 7f8bdbe00640 1 -- 172.21.15.114:0/1858426698 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc15ce70 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.076+0000 7f8bdbe00640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bdc0cd1b0 0x7f8bdc0af760 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4036040 tx=0x7f8bd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.076+0000 7f8bd3400640 1 -- 172.21.15.114:0/1858426698 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4043020 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.076+0000 7f8bd3400640 1 -- 172.21.15.114:0/1858426698 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd402ce50 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.076+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc15d150 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.076+0000 7f8bd3400640 1 -- 172.21.15.114:0/1858426698 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4009510 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.076+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bdc15d710 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.077+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bdc04f330 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.078+0000 7f8bd3400640 1 -- 172.21.15.114:0/1858426698 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bd4046020 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.079+0000 7f8bd3400640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8bb806b500 0x7f8bb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.080+0000 7f8bd3400640 1 -- 172.21.15.114:0/1858426698 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8bd40ac3c0 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.080+0000 7f8bdb400640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8bb806b500 0x7f8bb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.080+0000 7f8bdb400640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8bb806b500 0x7f8bb806d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc007fd0 tx=0x7f8bcc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.084+0000 7f8bd3400640 1 -- 172.21.15.114:0/1858426698 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8bd404d030 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.332+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f8bdc1e3ac0 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.337+0000 7f8bd3400640 1 -- 172.21.15.114:0/1858426698 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8bd407b6a0 con 0x7f8bdc0cd1b0 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8bb806b500 msgr2=0x7f8bb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8bb806b500 0x7f8bb806d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc007fd0 tx=0x7f8bcc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bdc0cd1b0 msgr2=0x7f8bdc0af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bdc0cd1b0 0x7f8bdc0af760 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4036040 tx=0x7f8bd4004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 shutdown_connections 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8bb806b500 0x7f8bb806d9c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bdc0cd1b0 0x7f8bdc0af760 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8bdc0b03a0 0x7f8bdc0af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 --2- 172.21.15.114:0/1858426698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bdc071e20 0x7f8bdc15c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 >> 172.21.15.114:0/1858426698 conn(0x7f8bdc0d0df0 msgr2=0x7f8bdc0cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 shutdown_connections 2024-03-06T07:28:59.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.339+0000 7f8be2600640 1 -- 172.21.15.114:0/1858426698 wait complete. 2024-03-06T07:28:59.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:28:59.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.552+0000 7f67b3000640 1 Processor -- start 2024-03-06T07:28:59.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.552+0000 7f67b3000640 1 -- start start 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b3000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67ac0b40c0 0x7f67ac0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b3000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f67ac0ae670 0x7f67ac15e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b3000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac1620c0 0x7f67ac0b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b3000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ac161900 con 0x7f67ac0b40c0 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b3000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ac0b49e0 con 0x7f67ac0ae670 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b3000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac15ebd0 con 0x7f67ac1620c0 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b1c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67ac0b40c0 0x7f67ac0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b1c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67ac0b40c0 0x7f67ac0b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42116/0 (socket says 172.21.15.114:42116) 2024-03-06T07:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.554+0000 7f67b1c00640 1 -- 172.21.15.114:0/4065237428 learned_addr learned my addr 172.21.15.114:0/4065237428 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:59.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.555+0000 7f67b1200640 1 --2- 172.21.15.114:0/4065237428 >> v2:172.21.15.177:3300/0 conn(0x7f67ac0ae670 0x7f67ac15e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.555+0000 7f67b2600640 1 --2- 172.21.15.114:0/4065237428 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac1620c0 0x7f67ac0b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.555+0000 7f67b1c00640 1 -- 172.21.15.114:0/4065237428 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac1620c0 msgr2=0x7f67ac0b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.555+0000 7f67b1c00640 1 --2- 172.21.15.114:0/4065237428 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac1620c0 0x7f67ac0b28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.555+0000 7f67b1c00640 1 -- 172.21.15.114:0/4065237428 >> v2:172.21.15.177:3300/0 conn(0x7f67ac0ae670 msgr2=0x7f67ac15e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.555+0000 7f67b1c00640 1 --2- 172.21.15.114:0/4065237428 >> v2:172.21.15.177:3300/0 conn(0x7f67ac0ae670 0x7f67ac15e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.555+0000 7f67b1c00640 1 -- 172.21.15.114:0/4065237428 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac15eeb0 con 0x7f67ac0b40c0 2024-03-06T07:28:59.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.556+0000 7f67b1c00640 1 --2- 172.21.15.114:0/4065237428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac0b40c0 0x7f67ac0b44a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0adc70 tx=0x7f67a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4902ce198a4a404 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.557+0000 7f67abe00640 1 -- 172.21.15.114:0/4065237428 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0035020 con 0x7f67ac0b40c0 2024-03-06T07:28:59.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.557+0000 7f67abe00640 1 -- 172.21.15.114:0/4065237428 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a0031870 con 0x7f67ac0b40c0 2024-03-06T07:28:59.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.557+0000 7f67abe00640 1 -- 172.21.15.114:0/4065237428 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0031c40 con 0x7f67ac0b40c0 2024-03-06T07:28:59.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 -- 172.21.15.114:0/4065237428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac0b40c0 msgr2=0x7f67ac0b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 --2- 172.21.15.114:0/4065237428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac0b40c0 0x7f67ac0b44a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0adc70 tx=0x7f67a002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 -- 172.21.15.114:0/4065237428 shutdown_connections 2024-03-06T07:28:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 --2- 172.21.15.114:0/4065237428 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac1620c0 0x7f67ac0b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 --2- 172.21.15.114:0/4065237428 >> v2:172.21.15.177:3300/0 conn(0x7f67ac0ae670 0x7f67ac15e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 --2- 172.21.15.114:0/4065237428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac0b40c0 0x7f67ac0b44a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 -- 172.21.15.114:0/4065237428 >> 172.21.15.114:0/4065237428 conn(0x7f67ac06afc0 msgr2=0x7f67ac06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 -- 172.21.15.114:0/4065237428 shutdown_connections 2024-03-06T07:28:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.558+0000 7f67b3000640 1 -- 172.21.15.114:0/4065237428 wait complete. 2024-03-06T07:28:59.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.560+0000 7f67b3000640 1 Processor -- start 2024-03-06T07:28:59.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.560+0000 7f67b3000640 1 -- start start 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b3000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac0ae670 0x7f67ac0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b3000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67ac0b40c0 0x7f67ac15ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b3000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac1620c0 0x7f67ac14a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b3000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac15e800 con 0x7f67ac1620c0 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b3000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac161ba0 con 0x7f67ac0b40c0 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b3000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac15e980 con 0x7f67ac0ae670 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac1620c0 0x7f67ac14a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac1620c0 0x7f67ac14a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42118/0 (socket says 172.21.15.114:42118) 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 -- 172.21.15.114:0/535084976 learned_addr learned my addr 172.21.15.114:0/535084976 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:28:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b1200640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67ac0b40c0 0x7f67ac15ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b1c00640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac0ae670 0x7f67ac0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 -- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac0ae670 msgr2=0x7f67ac0b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac0ae670 0x7f67ac0b15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 -- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67ac0b40c0 msgr2=0x7f67ac15ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67ac0b40c0 0x7f67ac15ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.561+0000 7f67b2600640 1 -- 172.21.15.114:0/535084976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0b1cf0 con 0x7f67ac1620c0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.562+0000 7f67b1c00640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac0ae670 0x7f67ac0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.562+0000 7f67b2600640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac1620c0 0x7f67ac14a3f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f67a4007e40 tx=0x7f67a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.562+0000 7f67a9600640 1 -- 172.21.15.114:0/535084976 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4014620 con 0x7f67ac1620c0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.562+0000 7f67a9600640 1 -- 172.21.15.114:0/535084976 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a4019070 con 0x7f67ac1620c0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.562+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac0b1fd0 con 0x7f67ac1620c0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.562+0000 7f67a9600640 1 -- 172.21.15.114:0/535084976 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4014a20 con 0x7f67ac1620c0 2024-03-06T07:28:59.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.562+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac0b2590 con 0x7f67ac1620c0 2024-03-06T07:28:59.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.563+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac04f330 con 0x7f67ac1620c0 2024-03-06T07:28:59.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.564+0000 7f67a9600640 1 -- 172.21.15.114:0/535084976 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67a4014bc0 con 0x7f67ac1620c0 2024-03-06T07:28:59.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.566+0000 7f67a9600640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f678c06b4d0 0x7f678c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.566+0000 7f67a9600640 1 -- 172.21.15.114:0/535084976 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f67a408a800 con 0x7f67ac1620c0 2024-03-06T07:28:59.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.566+0000 7f67b1c00640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f678c06b4d0 0x7f678c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.566+0000 7f67b1c00640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f678c06b4d0 0x7f678c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f67a0036000 tx=0x7f67a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.570+0000 7f67a9600640 1 -- 172.21.15.114:0/535084976 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67a40599c0 con 0x7f67ac1620c0 2024-03-06T07:28:59.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.840+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f67ac0adf40 con 0x7f67ac1620c0 2024-03-06T07:28:59.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.844+0000 7f67a9600640 1 -- 172.21.15.114:0/535084976 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f67a40592c0 con 0x7f67ac1620c0 2024-03-06T07:28:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.846+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f678c06b4d0 msgr2=0x7f678c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.846+0000 7f67b3000640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f678c06b4d0 0x7f678c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f67a0036000 tx=0x7f67a0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.846+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac1620c0 msgr2=0x7f67ac14a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.846+0000 7f67b3000640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac1620c0 0x7f67ac14a3f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f67a4007e40 tx=0x7f67a400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 shutdown_connections 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f678c06b4d0 0x7f678c06d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ac1620c0 0x7f67ac14a3f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67ac0b40c0 0x7f67ac15ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 --2- 172.21.15.114:0/535084976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67ac0ae670 0x7f67ac0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 >> 172.21.15.114:0/535084976 conn(0x7f67ac06afc0 msgr2=0x7f67ac0d0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 shutdown_connections 2024-03-06T07:28:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:28:59.847+0000 7f67b3000640 1 -- 172.21.15.114:0/535084976 wait complete. 2024-03-06T07:28:59.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:29:00.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.065+0000 7fd507a00640 1 Processor -- start 2024-03-06T07:29:00.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.066+0000 7fd507a00640 1 -- start start 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.067+0000 7fd507a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 0x7fd5000cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.067+0000 7fd507a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5001474c0 0x7fd5000cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.067+0000 7fd507a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5000ae380 con 0x7fd5001444e0 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.067+0000 7fd507a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5000ae660 con 0x7fd5001474c0 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.067+0000 7fd507a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5000ae940 con 0x7fd5000c5040 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd505a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5001474c0 0x7fd5000cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd505a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5001474c0 0x7fd5000cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42128/0 (socket says 172.21.15.114:42128) 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd505a00640 1 -- 172.21.15.114:0/1056517790 learned_addr learned my addr 172.21.15.114:0/1056517790 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd4f7e00640 1 --2- 172.21.15.114:0/1056517790 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 0x7fd5000cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd505a00640 1 -- 172.21.15.114:0/1056517790 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 msgr2=0x7fd5000cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd505a00640 1 --2- 172.21.15.114:0/1056517790 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 0x7fd5000cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd505a00640 1 -- 172.21.15.114:0/1056517790 >> v1:172.21.15.177:6789/0 conn(0x7fd5001444e0 legacy=0x7fd5000c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.068+0000 7fd505a00640 1 -- 172.21.15.114:0/1056517790 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5000aec20 con 0x7fd5001474c0 2024-03-06T07:29:00.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.069+0000 7fd505a00640 1 --2- 172.21.15.114:0/1056517790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001474c0 0x7fd5000cd920 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009e50 tx=0x7fd4f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4cab771de92ec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd4ffe00640 1 -- 172.21.15.114:0/1056517790 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f802ca70 con 0x7fd5001474c0 2024-03-06T07:29:00.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd4ffe00640 1 -- 172.21.15.114:0/1056517790 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f802cc10 con 0x7fd5001474c0 2024-03-06T07:29:00.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd4ffe00640 1 -- 172.21.15.114:0/1056517790 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80343f0 con 0x7fd5001474c0 2024-03-06T07:29:00.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd507a00640 1 -- 172.21.15.114:0/1056517790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001474c0 msgr2=0x7fd5000cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd507a00640 1 --2- 172.21.15.114:0/1056517790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001474c0 0x7fd5000cd920 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009e50 tx=0x7fd4f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd507a00640 1 -- 172.21.15.114:0/1056517790 shutdown_connections 2024-03-06T07:29:00.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd507a00640 1 --2- 172.21.15.114:0/1056517790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001474c0 0x7fd5000cd920 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd507a00640 1 --2- 172.21.15.114:0/1056517790 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 0x7fd5000cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.070+0000 7fd507a00640 1 -- 172.21.15.114:0/1056517790 >> 172.21.15.114:0/1056517790 conn(0x7fd5000cc4f0 msgr2=0x7fd5000cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.071+0000 7fd507a00640 1 -- 172.21.15.114:0/1056517790 shutdown_connections 2024-03-06T07:29:00.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.071+0000 7fd507a00640 1 -- 172.21.15.114:0/1056517790 wait complete. 2024-03-06T07:29:00.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.073+0000 7fd507a00640 1 Processor -- start 2024-03-06T07:29:00.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.073+0000 7fd507a00640 1 -- start start 2024-03-06T07:29:00.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd507a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 0x7fd5000c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd507a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001444e0 0x7fd50013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd507a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5001474c0 0x7fd5000cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd507a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5001448c0 con 0x7fd5001444e0 2024-03-06T07:29:00.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd507a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd500147cb0 con 0x7fd5001474c0 2024-03-06T07:29:00.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd507a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd500144a40 con 0x7fd5000c5040 2024-03-06T07:29:00.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd4f7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001444e0 0x7fd50013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd4f7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001444e0 0x7fd50013f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42144/0 (socket says 172.21.15.114:42144) 2024-03-06T07:29:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.074+0000 7fd4f7e00640 1 -- 172.21.15.114:0/3353684920 learned_addr learned my addr 172.21.15.114:0/3353684920 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4f7e00640 1 -- 172.21.15.114:0/3353684920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 msgr2=0x7fd5000c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4f7e00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 0x7fd5000c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4f7e00640 1 -- 172.21.15.114:0/3353684920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5001474c0 msgr2=0x7fd5000cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4f7e00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5001474c0 0x7fd5000cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4f7e00640 1 -- 172.21.15.114:0/3353684920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50013fef0 con 0x7fd5001444e0 2024-03-06T07:29:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4f7e00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001444e0 0x7fd50013f6d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000d800 tx=0x7fd4f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4fd600640 1 -- 172.21.15.114:0/3353684920 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f000b6c0 con 0x7fd5001444e0 2024-03-06T07:29:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4fd600640 1 -- 172.21.15.114:0/3353684920 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f0019070 con 0x7fd5001444e0 2024-03-06T07:29:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd4fd600640 1 -- 172.21.15.114:0/3353684920 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f000ba80 con 0x7fd5001444e0 2024-03-06T07:29:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.075+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5001401d0 con 0x7fd5001444e0 2024-03-06T07:29:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.076+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5000c7280 con 0x7fd5001444e0 2024-03-06T07:29:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.077+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50004f330 con 0x7fd5001444e0 2024-03-06T07:29:00.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.077+0000 7fd4fd600640 1 -- 172.21.15.114:0/3353684920 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4f000bc20 con 0x7fd5001444e0 2024-03-06T07:29:00.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.078+0000 7fd4fd600640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4dc06b4d0 0x7fd4dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.078+0000 7fd4fd600640 1 -- 172.21.15.114:0/3353684920 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd4f008a430 con 0x7fd5001444e0 2024-03-06T07:29:00.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.079+0000 7fd505000640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4dc06b4d0 0x7fd4dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.079+0000 7fd505000640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4dc06b4d0 0x7fd4dc06d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd5000c6df0 tx=0x7fd4e80079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.083+0000 7fd4fd600640 1 -- 172.21.15.114:0/3353684920 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4f0059560 con 0x7fd5001444e0 2024-03-06T07:29:00.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.332+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fd50013fc10 con 0x7fd5001444e0 2024-03-06T07:29:00.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.336+0000 7fd4fd600640 1 -- 172.21.15.114:0/3353684920 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd4f0058e60 con 0x7fd5001444e0 2024-03-06T07:29:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4dc06b4d0 msgr2=0x7fd4dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4dc06b4d0 0x7fd4dc06d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd5000c6df0 tx=0x7fd4e80079b0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001444e0 msgr2=0x7fd50013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001444e0 0x7fd50013f6d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000d800 tx=0x7fd4f000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 shutdown_connections 2024-03-06T07:29:00.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5001474c0 0x7fd5000cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4dc06b4d0 0x7fd4dc06d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5001444e0 0x7fd50013f6d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 --2- 172.21.15.114:0/3353684920 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5000c5040 0x7fd5000c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.339+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 >> 172.21.15.114:0/3353684920 conn(0x7fd5000cc4f0 msgr2=0x7fd50006dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.340+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 shutdown_connections 2024-03-06T07:29:00.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.340+0000 7fd507a00640 1 -- 172.21.15.114:0/3353684920 wait complete. 2024-03-06T07:29:00.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:00.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.548+0000 7fd327000640 1 Processor -- start 2024-03-06T07:29:00.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.548+0000 7fd327000640 1 -- start start 2024-03-06T07:29:00.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.549+0000 7fd327000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd32015acb0 0x7fd3200b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.549+0000 7fd327000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3200b4680 0x7fd32006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.549+0000 7fd327000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32015b090 con 0x7fd320072340 2024-03-06T07:29:00.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.549+0000 7fd327000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200b2a30 con 0x7fd32015acb0 2024-03-06T07:29:00.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.549+0000 7fd327000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b4360 con 0x7fd3200b4680 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd31fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd32015acb0 0x7fd3200b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd31fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd32015acb0 0x7fd3200b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42146/0 (socket says 172.21.15.114:42146) 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd31fe00640 1 -- 172.21.15.114:0/1914984783 learned_addr learned my addr 172.21.15.114:0/1914984783 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd317400640 1 --2- 172.21.15.114:0/1914984783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3200b4680 0x7fd32006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd31fe00640 1 -- 172.21.15.114:0/1914984783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3200b4680 msgr2=0x7fd32006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd31fe00640 1 --2- 172.21.15.114:0/1914984783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3200b4680 0x7fd32006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd31fe00640 1 -- 172.21.15.114:0/1914984783 >> v1:172.21.15.177:6789/0 conn(0x7fd320072340 legacy=0x7fd3200ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.550+0000 7fd31fe00640 1 -- 172.21.15.114:0/1914984783 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200b2cf0 con 0x7fd32015acb0 2024-03-06T07:29:00.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.551+0000 7fd31fe00640 1 --2- 172.21.15.114:0/1914984783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015acb0 0x7fd3200b37d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd308009870 tx=0x7fd30802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89550077b1d1a174 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.551+0000 7fd31f400640 1 -- 172.21.15.114:0/1914984783 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd308035020 con 0x7fd32015acb0 2024-03-06T07:29:00.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.551+0000 7fd31f400640 1 -- 172.21.15.114:0/1914984783 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd308031640 con 0x7fd32015acb0 2024-03-06T07:29:00.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.551+0000 7fd31f400640 1 -- 172.21.15.114:0/1914984783 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30802d040 con 0x7fd32015acb0 2024-03-06T07:29:00.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 -- 172.21.15.114:0/1914984783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015acb0 msgr2=0x7fd3200b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 --2- 172.21.15.114:0/1914984783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015acb0 0x7fd3200b37d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd308009870 tx=0x7fd30802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 -- 172.21.15.114:0/1914984783 shutdown_connections 2024-03-06T07:29:00.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 --2- 172.21.15.114:0/1914984783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3200b4680 0x7fd32006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 --2- 172.21.15.114:0/1914984783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015acb0 0x7fd3200b37d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 -- 172.21.15.114:0/1914984783 >> 172.21.15.114:0/1914984783 conn(0x7fd3200b1730 msgr2=0x7fd3200b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 -- 172.21.15.114:0/1914984783 shutdown_connections 2024-03-06T07:29:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.552+0000 7fd327000640 1 -- 172.21.15.114:0/1914984783 wait complete. 2024-03-06T07:29:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.554+0000 7fd327000640 1 Processor -- start 2024-03-06T07:29:00.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.554+0000 7fd327000640 1 -- start start 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd327000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd320072340 0x7fd32015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd327000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200b4680 0x7fd32015b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd327000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd32015acb0 0x7fd32015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd327000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b3d10 con 0x7fd3200b4680 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd327000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b32b0 con 0x7fd32015acb0 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd327000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b4010 con 0x7fd320072340 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd317400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200b4680 0x7fd32015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd317400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200b4680 0x7fd32015b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42162/0 (socket says 172.21.15.114:42162) 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd317400640 1 -- 172.21.15.114:0/3582139171 learned_addr learned my addr 172.21.15.114:0/3582139171 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd31fe00640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd320072340 0x7fd32015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd325000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd32015acb0 0x7fd32015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.555+0000 7fd317400640 1 -- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd320072340 msgr2=0x7fd32015ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd317400640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd320072340 0x7fd32015ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd317400640 1 -- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd32015acb0 msgr2=0x7fd32015b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd317400640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd32015acb0 0x7fd32015b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd317400640 1 -- 172.21.15.114:0/3582139171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32015bce0 con 0x7fd3200b4680 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd325000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd32015acb0 0x7fd32015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd317400640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200b4680 0x7fd32015b260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd310007f20 tx=0x7fd31000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd31cc00640 1 -- 172.21.15.114:0/3582139171 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310014620 con 0x7fd3200b4680 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd31cc00640 1 -- 172.21.15.114:0/3582139171 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd310019070 con 0x7fd3200b4680 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd31cc00640 1 -- 172.21.15.114:0/3582139171 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310014a00 con 0x7fd3200b4680 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32015bf50 con 0x7fd3200b4680 2024-03-06T07:29:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.556+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32015c510 con 0x7fd3200b4680 2024-03-06T07:29:00.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.557+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32004f330 con 0x7fd3200b4680 2024-03-06T07:29:00.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.558+0000 7fd31cc00640 1 -- 172.21.15.114:0/3582139171 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd310014ba0 con 0x7fd3200b4680 2024-03-06T07:29:00.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.559+0000 7fd31cc00640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd2fc06b4d0 0x7fd2fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.559+0000 7fd31cc00640 1 -- 172.21.15.114:0/3582139171 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd31008a8a0 con 0x7fd3200b4680 2024-03-06T07:29:00.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.559+0000 7fd31fe00640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd2fc06b4d0 0x7fd2fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.560+0000 7fd31fe00640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd2fc06b4d0 0x7fd2fc06d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd308036000 tx=0x7fd30803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.564+0000 7fd31cc00640 1 -- 172.21.15.114:0/3582139171 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3100599d0 con 0x7fd3200b4680 2024-03-06T07:29:00.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.763+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd32006ce60 con 0x7fd3200b4680 2024-03-06T07:29:00.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.765+0000 7fd31cc00640 1 -- 172.21.15.114:0/3582139171 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12939 (secure 0 0 0) 0x7fd3100592d0 con 0x7fd3200b4680 2024-03-06T07:29:00.768 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:00.768 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:28:56.791426+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd2fc06b4d0 msgr2=0x7fd2fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd2fc06b4d0 0x7fd2fc06d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd308036000 tx=0x7fd30803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200b4680 msgr2=0x7fd32015b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200b4680 0x7fd32015b260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd310007f20 tx=0x7fd31000e570 comp rx=0 tx=0).stop 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 shutdown_connections 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd32015acb0 0x7fd32015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd2fc06b4d0 0x7fd2fc06d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200b4680 0x7fd32015b260 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 --2- 172.21.15.114:0/3582139171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd320072340 0x7fd32015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 >> 172.21.15.114:0/3582139171 conn(0x7fd3200b1730 msgr2=0x7fd3200cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 shutdown_connections 2024-03-06T07:29:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.768+0000 7fd327000640 1 -- 172.21.15.114:0/3582139171 wait complete. 2024-03-06T07:29:00.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:00.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:00.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.982+0000 7fc577e00640 1 Processor -- start 2024-03-06T07:29:00.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.982+0000 7fc577e00640 1 -- start start 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.983+0000 7fc577e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc57015fc10 0x7fc570070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.983+0000 7fc577e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc57015fff0 0x7fc570160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc577e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5700b46c0 con 0x7fc5700ce4a0 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc577e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5700b49a0 con 0x7fc57015fff0 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc577e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc570072c90 con 0x7fc57015fc10 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc574a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc57015fff0 0x7fc570160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc574a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc57015fff0 0x7fc570160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42166/0 (socket says 172.21.15.114:42166) 2024-03-06T07:29:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc574a00640 1 -- 172.21.15.114:0/3668478989 learned_addr learned my addr 172.21.15.114:0/3668478989 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc575e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc5700ce4a0 0x7fc5700ce8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51544/0 (socket says 172.21.15.114:51544) 2024-03-06T07:29:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc575400640 1 --2- 172.21.15.114:0/3668478989 >> v2:172.21.15.177:3300/0 conn(0x7fc57015fc10 0x7fc570070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc574a00640 1 -- 172.21.15.114:0/3668478989 >> v2:172.21.15.177:3300/0 conn(0x7fc57015fc10 msgr2=0x7fc570070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc574a00640 1 --2- 172.21.15.114:0/3668478989 >> v2:172.21.15.177:3300/0 conn(0x7fc57015fc10 0x7fc570070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc574a00640 1 -- 172.21.15.114:0/3668478989 >> v1:172.21.15.114:6789/0 conn(0x7fc5700ce4a0 legacy=0x7fc5700ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc574a00640 1 -- 172.21.15.114:0/3668478989 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc570160990 con 0x7fc57015fff0 2024-03-06T07:29:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.984+0000 7fc56fe00640 1 -- 172.21.15.114:0/3668478989 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977318645 0 0) 0x7fc5700b46c0 con 0x7fc5700ce4a0 2024-03-06T07:29:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc574a00640 1 --2- 172.21.15.114:0/3668478989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fff0 0x7fc570160450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc5700cf770 tx=0x7fc55c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239def539d1f9078 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc56fe00640 1 -- 172.21.15.114:0/3668478989 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c034070 con 0x7fc57015fff0 2024-03-06T07:29:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc56fe00640 1 -- 172.21.15.114:0/3668478989 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c030860 con 0x7fc57015fff0 2024-03-06T07:29:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc56fe00640 1 -- 172.21.15.114:0/3668478989 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c030c30 con 0x7fc57015fff0 2024-03-06T07:29:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc577e00640 1 -- 172.21.15.114:0/3668478989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fff0 msgr2=0x7fc570160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc577e00640 1 --2- 172.21.15.114:0/3668478989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fff0 0x7fc570160450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc5700cf770 tx=0x7fc55c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc577e00640 1 -- 172.21.15.114:0/3668478989 shutdown_connections 2024-03-06T07:29:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc577e00640 1 --2- 172.21.15.114:0/3668478989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fff0 0x7fc570160450 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc577e00640 1 --2- 172.21.15.114:0/3668478989 >> v2:172.21.15.177:3300/0 conn(0x7fc57015fc10 0x7fc570070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.986+0000 7fc577e00640 1 -- 172.21.15.114:0/3668478989 >> 172.21.15.114:0/3668478989 conn(0x7fc5700b1940 msgr2=0x7fc5700b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.987+0000 7fc577e00640 1 -- 172.21.15.114:0/3668478989 shutdown_connections 2024-03-06T07:29:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.987+0000 7fc577e00640 1 -- 172.21.15.114:0/3668478989 wait complete. 2024-03-06T07:29:00.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.988+0000 7fc577e00640 1 Processor -- start 2024-03-06T07:29:00.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.988+0000 7fc577e00640 1 -- start start 2024-03-06T07:29:00.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.989+0000 7fc577e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5700ce4a0 0x7fc57014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.989+0000 7fc577e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fc10 0x7fc570156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.989+0000 7fc577e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc57015fff0 0x7fc5701572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.989+0000 7fc577e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc570072a30 con 0x7fc57015fc10 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.989+0000 7fc577e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700728b0 con 0x7fc57015fff0 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.989+0000 7fc577e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57015c2e0 con 0x7fc5700ce4a0 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fc10 0x7fc570156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fc10 0x7fc570156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42180/0 (socket says 172.21.15.114:42180) 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 -- 172.21.15.114:0/2490689122 learned_addr learned my addr 172.21.15.114:0/2490689122 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc575e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc57015fff0 0x7fc5701572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 -- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5700ce4a0 msgr2=0x7fc57014f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc575400640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5700ce4a0 0x7fc57014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5700ce4a0 0x7fc57014f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 -- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc57015fff0 msgr2=0x7fc5701572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc57015fff0 0x7fc5701572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 -- 172.21.15.114:0/2490689122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57006d6a0 con 0x7fc57015fc10 2024-03-06T07:29:00.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc575e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc57015fff0 0x7fc5701572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.990+0000 7fc574a00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fc10 0x7fc570156d70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc55c02b9e0 tx=0x7fc55c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.991+0000 7fc56d600640 1 -- 172.21.15.114:0/2490689122 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c034040 con 0x7fc57015fc10 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.991+0000 7fc56d600640 1 -- 172.21.15.114:0/2490689122 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c039d90 con 0x7fc57015fc10 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.991+0000 7fc56d600640 1 -- 172.21.15.114:0/2490689122 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c043610 con 0x7fc57015fc10 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.991+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57006d980 con 0x7fc57015fc10 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.991+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57006df40 con 0x7fc57015fc10 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.992+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57004f330 con 0x7fc57015fc10 2024-03-06T07:29:00.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.993+0000 7fc56d600640 1 -- 172.21.15.114:0/2490689122 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc55c04a060 con 0x7fc57015fc10 2024-03-06T07:29:00.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.994+0000 7fc56d600640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc55006b4d0 0x7fc55006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.994+0000 7fc56d600640 1 -- 172.21.15.114:0/2490689122 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc55c0abff0 con 0x7fc57015fc10 2024-03-06T07:29:00.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.994+0000 7fc575400640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc55006b4d0 0x7fc55006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.995+0000 7fc575400640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc55006b4d0 0x7fc55006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc564006fd0 tx=0x7fc564008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:00.998+0000 7fc56d600640 1 -- 172.21.15.114:0/2490689122 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc55c03e070 con 0x7fc57015fc10 2024-03-06T07:29:01.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.195+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5700b2d40 con 0x7fc57015fc10 2024-03-06T07:29:01.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.196+0000 7fc56d600640 1 -- 172.21.15.114:0/2490689122 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc55c037360 con 0x7fc57015fc10 2024-03-06T07:29:01.198 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.198+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc55006b4d0 msgr2=0x7fc55006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.198+0000 7fc577e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc55006b4d0 0x7fc55006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc564006fd0 tx=0x7fc564008040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.198+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fc10 msgr2=0x7fc570156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.198+0000 7fc577e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fc10 0x7fc570156d70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc55c02b9e0 tx=0x7fc55c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 shutdown_connections 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc57015fff0 0x7fc5701572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc55006b4d0 0x7fc55006d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc57015fc10 0x7fc570156d70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 --2- 172.21.15.114:0/2490689122 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5700ce4a0 0x7fc57014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 >> 172.21.15.114:0/2490689122 conn(0x7fc5700b1940 msgr2=0x7fc5700ace40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 shutdown_connections 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.199+0000 7fc577e00640 1 -- 172.21.15.114:0/2490689122 wait complete. 2024-03-06T07:29:01.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:29:01.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.426+0000 7ffa1b400640 1 Processor -- start 2024-03-06T07:29:01.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.426+0000 7ffa1b400640 1 -- start start 2024-03-06T07:29:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.426+0000 7ffa1b400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 0x7ffa14071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.426+0000 7ffa1b400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1406cef0 con 0x7ffa1415f9d0 2024-03-06T07:29:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.426+0000 7ffa1b400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1406d1d0 con 0x7ffa14144c00 2024-03-06T07:29:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.427+0000 7ffa1b400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1406d4b0 con 0x7ffa1415fe60 2024-03-06T07:29:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.427+0000 7ffa0be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffa1415f9d0 0x7ffa14070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51556/0 (socket says 172.21.15.114:51556) 2024-03-06T07:29:01.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.427+0000 7ffa0be00640 1 -- 172.21.15.114:0/724227931 learned_addr learned my addr 172.21.15.114:0/724227931 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:01.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.427+0000 7ffa19400640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 0x7ffa14071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.427+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757951652 0 0) 0x7ffa1406cef0 con 0x7ffa1415f9d0 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.427+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f4003520 con 0x7ffa1415f9d0 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 821711054 0 0) 0x7ff9f4003520 con 0x7ffa1415f9d0 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 msgr2=0x7ffa14071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 0x7ffa14071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 >> v1:172.21.15.177:6789/0 conn(0x7ffa14144c00 legacy=0x7ffa1406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1406d790 con 0x7ffa1415f9d0 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7ffa04001670 con 0x7ffa1415f9d0 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 >> v1:172.21.15.114:6789/0 conn(0x7ffa1415f9d0 legacy=0x7ffa14070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 0x7ff9f40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f4003fb0 0x7ff9f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9f4004d40 0x7ff9f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1406d4b0 con 0x7ff9f4003fb0 2024-03-06T07:29:01.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f400bb00 con 0x7ff9f4004d40 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f400bd00 con 0x7ffa1415fe60 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa0be00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f4003fb0 0x7ff9f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa18a00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 0x7ff9f40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa19400640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9f4004d40 0x7ff9f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa0be00640 1 -- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 msgr2=0x7ff9f40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa0be00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 0x7ff9f40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa0be00640 1 -- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9f4004d40 msgr2=0x7ff9f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa0be00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9f4004d40 0x7ff9f40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa0be00640 1 -- 172.21.15.114:0/724227931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f4003520 con 0x7ff9f4003fb0 2024-03-06T07:29:01.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.428+0000 7ffa18a00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa1415fe60 0x7ff9f40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa0be00640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f4003fb0 0x7ff9f4004800 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c190 tx=0x7ff9f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac6b2be2ae9f358 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04007c60 con 0x7ff9f4003fb0 2024-03-06T07:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa04030070 con 0x7ff9f4003fb0 2024-03-06T07:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa13e00640 1 -- 172.21.15.114:0/724227931 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04005d90 con 0x7ff9f4003fb0 2024-03-06T07:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa1b400640 1 -- 172.21.15.114:0/724227931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f4003fb0 msgr2=0x7ff9f4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa1b400640 1 --2- 172.21.15.114:0/724227931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9f4003fb0 0x7ff9f4004800 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c190 tx=0x7ff9f4005830 comp rx=0 tx=0).stop 2024-03-06T07:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa19400640 1 -- 172.21.15.114:0/724227931 reap_dead start 2024-03-06T07:29:01.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa1b400640 1 -- 172.21.15.114:0/724227931 shutdown_connections 2024-03-06T07:29:01.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa1b400640 1 -- 172.21.15.114:0/724227931 >> 172.21.15.114:0/724227931 conn(0x7ffa140b1730 msgr2=0x7ffa140b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa1b400640 1 -- 172.21.15.114:0/724227931 shutdown_connections 2024-03-06T07:29:01.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.430+0000 7ffa1b400640 1 -- 172.21.15.114:0/724227931 wait complete. 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.432+0000 7ffa1b400640 1 Processor -- start 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.432+0000 7ffa1b400640 1 -- start start 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa1b400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9f4003fb0 0x7ffa14139350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa1b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa1415ab60 0x7ffa1415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa1b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa141e55c0 0x7ffa141e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa1b400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1406cb50 con 0x7ffa1415ab60 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa1b400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14160730 con 0x7ffa141e55c0 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa1b400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1406ccd0 con 0x7ff9f4003fb0 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa1415ab60 0x7ffa1415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa1415ab60 0x7ffa1415afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42194/0 (socket says 172.21.15.114:42194) 2024-03-06T07:29:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.433+0000 7ffa0be00640 1 -- 172.21.15.114:0/1218531152 learned_addr learned my addr 172.21.15.114:0/1218531152 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:01.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa0be00640 1 -- 172.21.15.114:0/1218531152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9f4003fb0 msgr2=0x7ffa14139350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa0be00640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9f4003fb0 0x7ffa14139350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa0be00640 1 -- 172.21.15.114:0/1218531152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa141e55c0 msgr2=0x7ffa141e5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa0be00640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa141e55c0 0x7ffa141e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa0be00640 1 -- 172.21.15.114:0/1218531152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14139890 con 0x7ffa1415ab60 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa0be00640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa1415ab60 0x7ffa1415afe0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c1c0 tx=0x7ffa04001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa11600640 1 -- 172.21.15.114:0/1218531152 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04007c60 con 0x7ffa1415ab60 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa11600640 1 -- 172.21.15.114:0/1218531152 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0402cba0 con 0x7ffa1415ab60 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa141e6280 con 0x7ffa1415ab60 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.434+0000 7ffa11600640 1 -- 172.21.15.114:0/1218531152 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04007070 con 0x7ffa1415ab60 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.435+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa141e6840 con 0x7ffa1415ab60 2024-03-06T07:29:01.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.436+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1404f330 con 0x7ffa1415ab60 2024-03-06T07:29:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.437+0000 7ffa11600640 1 -- 172.21.15.114:0/1218531152 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa0403f070 con 0x7ffa1415ab60 2024-03-06T07:29:01.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.437+0000 7ffa11600640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9f006b4d0 0x7ff9f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.438+0000 7ffa11600640 1 -- 172.21.15.114:0/1218531152 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffa040ab2f0 con 0x7ffa1415ab60 2024-03-06T07:29:01.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.438+0000 7ffa18a00640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9f006b4d0 0x7ff9f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.438+0000 7ffa18a00640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9f006b4d0 0x7ff9f006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc001670 tx=0x7ff9fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.442+0000 7ffa11600640 1 -- 172.21.15.114:0/1218531152 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa0403fe40 con 0x7ffa1415ab60 2024-03-06T07:29:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.643+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ffa14072890 con 0x7ffa1415ab60 2024-03-06T07:29:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.644+0000 7ffa11600640 1 -- 172.21.15.114:0/1218531152 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7ffa0407a420 con 0x7ffa1415ab60 2024-03-06T07:29:01.646 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:01.647 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:29:01.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:01.647 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:29:01.647 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:29:01.647 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:01.647 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9f006b4d0 msgr2=0x7ff9f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9f006b4d0 0x7ff9f006d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc001670 tx=0x7ff9fc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa1415ab60 msgr2=0x7ffa1415afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa1415ab60 0x7ffa1415afe0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c1c0 tx=0x7ffa04001970 comp rx=0 tx=0).stop 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 shutdown_connections 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa141e55c0 0x7ffa141e5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9f006b4d0 0x7ff9f006d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa1415ab60 0x7ffa1415afe0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 --2- 172.21.15.114:0/1218531152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff9f4003fb0 0x7ffa14139350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 >> 172.21.15.114:0/1218531152 conn(0x7ffa140b1730 msgr2=0x7ffa140ceec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 shutdown_connections 2024-03-06T07:29:01.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.647+0000 7ffa1b400640 1 -- 172.21.15.114:0/1218531152 wait complete. 2024-03-06T07:29:01.667 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:29:01.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:29:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.875+0000 7f0d62c00640 1 Processor -- start 2024-03-06T07:29:01.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.875+0000 7f0d62c00640 1 -- start start 2024-03-06T07:29:01.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.876+0000 7f0d62c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0d5c160110 0x7f0d5c06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.876+0000 7f0d62c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0d5c0ced20 0x7f0d5c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.876+0000 7f0d62c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5c0cf1e0 con 0x7f0d5c145980 2024-03-06T07:29:01.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.876+0000 7f0d62c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c070e40 con 0x7f0d5c160110 2024-03-06T07:29:01.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.876+0000 7f0d62c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c0cf4c0 con 0x7f0d5c0ced20 2024-03-06T07:29:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0d5c145980 0x7f0d5c15fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51564/0 (socket says 172.21.15.114:51564) 2024-03-06T07:29:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5be00640 1 -- 172.21.15.114:0/4118214473 learned_addr learned my addr 172.21.15.114:0/4118214473 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5b400640 1 --2- 172.21.15.114:0/4118214473 >> v2:172.21.15.114:3300/0 conn(0x7f0d5c160110 0x7f0d5c06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d60c00640 1 --2- 172.21.15.114:0/4118214473 >> v2:172.21.15.177:3300/0 conn(0x7f0d5c0ced20 0x7f0d5c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5aa00640 1 -- 172.21.15.114:0/4118214473 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220587958 0 0) 0x7f0d5c0cf1e0 con 0x7f0d5c145980 2024-03-06T07:29:01.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5aa00640 1 -- 172.21.15.114:0/4118214473 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d40003520 con 0x7f0d5c145980 2024-03-06T07:29:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5b400640 1 -- 172.21.15.114:0/4118214473 >> v2:172.21.15.177:3300/0 conn(0x7f0d5c0ced20 msgr2=0x7f0d5c06d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5b400640 1 --2- 172.21.15.114:0/4118214473 >> v2:172.21.15.177:3300/0 conn(0x7f0d5c0ced20 0x7f0d5c06d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5b400640 1 -- 172.21.15.114:0/4118214473 >> v1:172.21.15.114:6789/0 conn(0x7f0d5c145980 legacy=0x7f0d5c15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.877+0000 7f0d5b400640 1 -- 172.21.15.114:0/4118214473 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c0cf7a0 con 0x7f0d5c160110 2024-03-06T07:29:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.879+0000 7f0d5b400640 1 --2- 172.21.15.114:0/4118214473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c06ceb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c15f940 tx=0x7f0d4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef26ec888c4e0067 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.879+0000 7f0d5aa00640 1 -- 172.21.15.114:0/4118214473 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48035020 con 0x7f0d5c160110 2024-03-06T07:29:01.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.879+0000 7f0d5aa00640 1 -- 172.21.15.114:0/4118214473 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d48031830 con 0x7f0d5c160110 2024-03-06T07:29:01.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.879+0000 7f0d5aa00640 1 -- 172.21.15.114:0/4118214473 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48031be0 con 0x7f0d5c160110 2024-03-06T07:29:01.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 -- 172.21.15.114:0/4118214473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 msgr2=0x7f0d5c06ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 --2- 172.21.15.114:0/4118214473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c06ceb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c15f940 tx=0x7f0d4802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 -- 172.21.15.114:0/4118214473 shutdown_connections 2024-03-06T07:29:01.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 --2- 172.21.15.114:0/4118214473 >> v2:172.21.15.177:3300/0 conn(0x7f0d5c0ced20 0x7f0d5c06d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 --2- 172.21.15.114:0/4118214473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c06ceb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 -- 172.21.15.114:0/4118214473 >> 172.21.15.114:0/4118214473 conn(0x7f0d5c0cfa80 msgr2=0x7f0d5c0cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 -- 172.21.15.114:0/4118214473 shutdown_connections 2024-03-06T07:29:01.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.880+0000 7f0d62c00640 1 -- 172.21.15.114:0/4118214473 wait complete. 2024-03-06T07:29:01.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.882+0000 7f0d62c00640 1 Processor -- start 2024-03-06T07:29:01.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.882+0000 7f0d62c00640 1 -- start start 2024-03-06T07:29:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d62c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d5c0ced20 0x7f0d5c142070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d62c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d5c145980 0x7f0d5c144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d62c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c15d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d62c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c1608c0 con 0x7f0d5c160110 2024-03-06T07:29:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d62c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c160740 con 0x7f0d5c145980 2024-03-06T07:29:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d62c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c0722e0 con 0x7f0d5c0ced20 2024-03-06T07:29:01.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c15d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42214/0 (socket says 172.21.15.114:42214) 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 -- 172.21.15.114:0/1014896488 learned_addr learned my addr 172.21.15.114:0/1014896488 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d5b400640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d5c145980 0x7f0d5c144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d5be00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d5c0ced20 0x7f0d5c142070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 -- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d5c0ced20 msgr2=0x7f0d5c142070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d5c0ced20 0x7f0d5c142070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 -- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d5c145980 msgr2=0x7f0d5c144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d5c145980 0x7f0d5c144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d60c00640 1 -- 172.21.15.114:0/1014896488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c15bd10 con 0x7f0d5c160110 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.884+0000 7f0d5b400640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d5c145980 0x7f0d5c144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:01.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.885+0000 7f0d60c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c15d490 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0d54007e90 tx=0x7f0d5400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.885+0000 7f0d53e00640 1 -- 172.21.15.114:0/1014896488 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54014420 con 0x7f0d5c160110 2024-03-06T07:29:01.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.885+0000 7f0d53e00640 1 -- 172.21.15.114:0/1014896488 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d54019070 con 0x7f0d5c160110 2024-03-06T07:29:01.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.885+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c142770 con 0x7f0d5c160110 2024-03-06T07:29:01.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.885+0000 7f0d53e00640 1 -- 172.21.15.114:0/1014896488 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54014800 con 0x7f0d5c160110 2024-03-06T07:29:01.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.885+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c142d30 con 0x7f0d5c160110 2024-03-06T07:29:01.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.886+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c04f330 con 0x7f0d5c160110 2024-03-06T07:29:01.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.887+0000 7f0d53e00640 1 -- 172.21.15.114:0/1014896488 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d540149a0 con 0x7f0d5c160110 2024-03-06T07:29:01.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.888+0000 7f0d53e00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.888+0000 7f0d53e00640 1 -- 172.21.15.114:0/1014896488 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0d5408a7b0 con 0x7f0d5c160110 2024-03-06T07:29:01.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.888+0000 7f0d5be00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.889+0000 7f0d5be00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000f60 tx=0x7f0d4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:01.891+0000 7f0d53e00640 1 -- 172.21.15.114:0/1014896488 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d540598e0 con 0x7f0d5c160110 2024-03-06T07:29:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.086+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0d5c072920 con 0x7f0d5c160110 2024-03-06T07:29:02.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.814+0000 7f0d53e00640 1 -- 172.21.15.114:0/1014896488 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0d540591e0 con 0x7f0d5c160110 2024-03-06T07:29:02.816 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:29:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0d3c06b4d0 msgr2=0x7f0d3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000f60 tx=0x7f0d4c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 msgr2=0x7f0d5c15d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c15d490 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0d54007e90 tx=0x7f0d5400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 shutdown_connections 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0d3c06b4d0 0x7f0d3c06d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d5c160110 0x7f0d5c15d490 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d5c145980 0x7f0d5c144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 --2- 172.21.15.114:0/1014896488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d5c0ced20 0x7f0d5c142070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.818+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 >> 172.21.15.114:0/1014896488 conn(0x7f0d5c0cfa80 msgr2=0x7f0d5c06d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.819+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 shutdown_connections 2024-03-06T07:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:02.819+0000 7f0d62c00640 1 -- 172.21.15.114:0/1014896488 wait complete. 2024-03-06T07:29:02.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:29:03.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.050+0000 7fd08ec00640 1 Processor -- start 2024-03-06T07:29:03.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.051+0000 7fd08ec00640 1 -- start start 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.052+0000 7fd08ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3650 0x7fd0880727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.052+0000 7fd08ec00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0880cda90 0x7fd08806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.052+0000 7fd08ec00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880719e0 con 0x7fd0880b3df0 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.052+0000 7fd08ec00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd088072310 con 0x7fd0880cda90 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.052+0000 7fd08ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd088071cc0 con 0x7fd0880b3650 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd08cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0880cda90 0x7fd08806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd08cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0880cda90 0x7fd08806c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42218/0 (socket says 172.21.15.114:42218) 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd08cc00640 1 -- 172.21.15.114:0/196565806 learned_addr learned my addr 172.21.15.114:0/196565806 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd087e00640 1 --1- 172.21.15.114:0/196565806 >> v1:172.21.15.114:6789/0 conn(0x7fd0880b3df0 0x7fd0880b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51578/0 (socket says 172.21.15.114:51578) 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd07f400640 1 --2- 172.21.15.114:0/196565806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3650 0x7fd0880727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd08cc00640 1 -- 172.21.15.114:0/196565806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3650 msgr2=0x7fd0880727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd08cc00640 1 --2- 172.21.15.114:0/196565806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3650 0x7fd0880727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd08cc00640 1 -- 172.21.15.114:0/196565806 >> v1:172.21.15.114:6789/0 conn(0x7fd0880b3df0 legacy=0x7fd0880b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd08cc00640 1 -- 172.21.15.114:0/196565806 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd088071fa0 con 0x7fd0880cda90 2024-03-06T07:29:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.053+0000 7fd087400640 1 -- 172.21.15.114:0/196565806 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201940464 0 0) 0x7fd0880719e0 con 0x7fd0880b3df0 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.054+0000 7fd08cc00640 1 --2- 172.21.15.114:0/196565806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08806c990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd080009e50 tx=0x7fd08002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78588a25184aa162 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.054+0000 7fd087400640 1 -- 172.21.15.114:0/196565806 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08002cb00 con 0x7fd0880cda90 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.054+0000 7fd087400640 1 -- 172.21.15.114:0/196565806 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08002cca0 con 0x7fd0880cda90 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.054+0000 7fd087400640 1 -- 172.21.15.114:0/196565806 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080034610 con 0x7fd0880cda90 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 -- 172.21.15.114:0/196565806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 msgr2=0x7fd08806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 --2- 172.21.15.114:0/196565806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08806c990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd080009e50 tx=0x7fd08002bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 -- 172.21.15.114:0/196565806 shutdown_connections 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 --2- 172.21.15.114:0/196565806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08806c990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 --2- 172.21.15.114:0/196565806 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3650 0x7fd0880727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 -- 172.21.15.114:0/196565806 >> 172.21.15.114:0/196565806 conn(0x7fd0880b1940 msgr2=0x7fd0880b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 -- 172.21.15.114:0/196565806 shutdown_connections 2024-03-06T07:29:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.055+0000 7fd08ec00640 1 -- 172.21.15.114:0/196565806 wait complete. 2024-03-06T07:29:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.057+0000 7fd08ec00640 1 Processor -- start 2024-03-06T07:29:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.057+0000 7fd08ec00640 1 -- start start 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0880b3650 0x7fd0880ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3df0 0x7fd08809cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08809d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08815b3c0 con 0x7fd0880cda90 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08ec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08815b240 con 0x7fd0880b3650 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880cf510 con 0x7fd0880b3df0 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08809d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08809d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42220/0 (socket says 172.21.15.114:42220) 2024-03-06T07:29:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.058+0000 7fd08cc00640 1 -- 172.21.15.114:0/1281364600 learned_addr learned my addr 172.21.15.114:0/1281364600 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd08cc00640 1 -- 172.21.15.114:0/1281364600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3df0 msgr2=0x7fd08809cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd08cc00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3df0 0x7fd08809cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd08cc00640 1 -- 172.21.15.114:0/1281364600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0880b3650 msgr2=0x7fd0880ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd08cc00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0880b3650 0x7fd0880ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd08cc00640 1 -- 172.21.15.114:0/1281364600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0881f8640 con 0x7fd0880cda90 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd08cc00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08809d330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd080036040 tx=0x7fd080004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd084c00640 1 -- 172.21.15.114:0/1281364600 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080044020 con 0x7fd0880cda90 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd084c00640 1 -- 172.21.15.114:0/1281364600 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08002ce50 con 0x7fd0880cda90 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd084c00640 1 -- 172.21.15.114:0/1281364600 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080009510 con 0x7fd0880cda90 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.059+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0881f4310 con 0x7fd0880cda90 2024-03-06T07:29:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.060+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0881f48d0 con 0x7fd0880cda90 2024-03-06T07:29:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.061+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0880cdf60 con 0x7fd0880cda90 2024-03-06T07:29:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.062+0000 7fd084c00640 1 -- 172.21.15.114:0/1281364600 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd080047020 con 0x7fd0880cda90 2024-03-06T07:29:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.063+0000 7fd084c00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd06406b4d0 0x7fd06406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.063+0000 7fd084c00640 1 -- 172.21.15.114:0/1281364600 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd0800aca60 con 0x7fd0880cda90 2024-03-06T07:29:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.063+0000 7fd087e00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd06406b4d0 0x7fd06406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.064+0000 7fd087e00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd06406b4d0 0x7fd06406d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd078007fd0 tx=0x7fd078007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.068+0000 7fd084c00640 1 -- 172.21.15.114:0/1281364600 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0880cdf60 con 0x7fd0880cda90 2024-03-06T07:29:03.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.268+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd0880cc0e0 con 0x7fd0880cda90 2024-03-06T07:29:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.829+0000 7fd084c00640 1 -- 172.21.15.114:0/1281364600 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd08004e030 con 0x7fd0880cda90 2024-03-06T07:29:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd06406b4d0 msgr2=0x7fd06406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd06406b4d0 0x7fd06406d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd078007fd0 tx=0x7fd078007960 comp rx=0 tx=0).stop 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 msgr2=0x7fd08809d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08809d330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd080036040 tx=0x7fd080004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 shutdown_connections 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd06406b4d0 0x7fd06406d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0880cda90 0x7fd08809d330 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0880b3df0 0x7fd08809cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 --2- 172.21.15.114:0/1281364600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0880b3650 0x7fd0880ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.834+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 >> 172.21.15.114:0/1281364600 conn(0x7fd0880b1940 msgr2=0x7fd08806dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.835+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 shutdown_connections 2024-03-06T07:29:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:03.835+0000 7fd08ec00640 1 -- 172.21.15.114:0/1281364600 wait complete. 2024-03-06T07:29:03.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:29:04.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.057+0000 7f8800c00640 1 Processor -- start 2024-03-06T07:29:04.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.058+0000 7f8800c00640 1 -- start start 2024-03-06T07:29:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f8800c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87fc0b3df0 0x7f87fc0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f8800c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f87fc0b3650 0x7f87fc0b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f8800c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f8800c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc072310 con 0x7f87fc0b3df0 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f8800c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc0b4710 con 0x7f87fc0b3650 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f8800c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0719e0 con 0x7f87fc0cda90 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f87f9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87fc0b3df0 0x7f87fc0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f87f9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87fc0b3df0 0x7f87fc0727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42234/0 (socket says 172.21.15.114:42234) 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.059+0000 7f87f9e00640 1 -- 172.21.15.114:0/2212118025 learned_addr learned my addr 172.21.15.114:0/2212118025 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.060+0000 7f87fa800640 1 --2- 172.21.15.114:0/2212118025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.060+0000 7f87f9e00640 1 -- 172.21.15.114:0/2212118025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 msgr2=0x7f87fc06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.060+0000 7f87f9e00640 1 --2- 172.21.15.114:0/2212118025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc06c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.060+0000 7f87f9e00640 1 -- 172.21.15.114:0/2212118025 >> v2:172.21.15.177:3300/0 conn(0x7f87fc0b3650 msgr2=0x7f87fc0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.060+0000 7f87f9e00640 1 --2- 172.21.15.114:0/2212118025 >> v2:172.21.15.177:3300/0 conn(0x7f87fc0b3650 0x7f87fc0b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.060+0000 7f87f9e00640 1 -- 172.21.15.114:0/2212118025 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc071cc0 con 0x7f87fc0b3df0 2024-03-06T07:29:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.060+0000 7f87fa800640 1 --2- 172.21.15.114:0/2212118025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.061+0000 7f87f9e00640 1 --2- 172.21.15.114:0/2212118025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3df0 0x7f87fc0727a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0b4c90 tx=0x7f87ec02db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ac9162d44aa56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.061+0000 7f87f9400640 1 -- 172.21.15.114:0/2212118025 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec02f8e0 con 0x7f87fc0b3df0 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.061+0000 7f87f9400640 1 -- 172.21.15.114:0/2212118025 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec02fa80 con 0x7f87fc0b3df0 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.061+0000 7f87f9400640 1 -- 172.21.15.114:0/2212118025 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec02fe50 con 0x7f87fc0b3df0 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 -- 172.21.15.114:0/2212118025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3df0 msgr2=0x7f87fc0727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 --2- 172.21.15.114:0/2212118025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3df0 0x7f87fc0727a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0b4c90 tx=0x7f87ec02db20 comp rx=0 tx=0).stop 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 -- 172.21.15.114:0/2212118025 shutdown_connections 2024-03-06T07:29:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 --2- 172.21.15.114:0/2212118025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 --2- 172.21.15.114:0/2212118025 >> v2:172.21.15.177:3300/0 conn(0x7f87fc0b3650 0x7f87fc0b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 --2- 172.21.15.114:0/2212118025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3df0 0x7f87fc0727a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 -- 172.21.15.114:0/2212118025 >> 172.21.15.114:0/2212118025 conn(0x7f87fc0b1940 msgr2=0x7f87fc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 -- 172.21.15.114:0/2212118025 shutdown_connections 2024-03-06T07:29:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.062+0000 7f8800c00640 1 -- 172.21.15.114:0/2212118025 wait complete. 2024-03-06T07:29:04.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.064+0000 7f8800c00640 1 Processor -- start 2024-03-06T07:29:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.064+0000 7f8800c00640 1 -- start start 2024-03-06T07:29:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.065+0000 7f8800c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3650 0x7f87fc1459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.065+0000 7f8800c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0b3df0 0x7f87fc0acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.065+0000 7f8800c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.065+0000 7f8800c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc15b240 con 0x7f87fc0b3650 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.065+0000 7f8800c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0b3b30 con 0x7f87fc0b3df0 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.065+0000 7f8800c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc15b3c0 con 0x7f87fc0cda90 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.065+0000 7f87f9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3650 0x7f87fc1459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3650 0x7f87fc1459a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42238/0 (socket says 172.21.15.114:42238) 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 -- 172.21.15.114:0/4173375515 learned_addr learned my addr 172.21.15.114:0/4173375515 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87fa800640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f3e00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0b3df0 0x7f87fc0acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 -- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 msgr2=0x7f87fc15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc15c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 -- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0b3df0 msgr2=0x7f87fc0acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0b3df0 0x7f87fc0acd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 -- 172.21.15.114:0/4173375515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc0ad7a0 con 0x7f87fc0b3650 2024-03-06T07:29:04.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.066+0000 7f87f9e00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3650 0x7f87fc1459a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f87ec02db60 tx=0x7f87ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.067+0000 7f87f2000640 1 -- 172.21.15.114:0/4173375515 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec046020 con 0x7f87fc0b3650 2024-03-06T07:29:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.067+0000 7f87f2000640 1 -- 172.21.15.114:0/4173375515 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec037df0 con 0x7f87fc0b3650 2024-03-06T07:29:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.067+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc0ada80 con 0x7f87fc0b3650 2024-03-06T07:29:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.067+0000 7f87f2000640 1 -- 172.21.15.114:0/4173375515 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0426b0 con 0x7f87fc0b3650 2024-03-06T07:29:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.067+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc146420 con 0x7f87fc0b3650 2024-03-06T07:29:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.068+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc04f330 con 0x7f87fc0b3650 2024-03-06T07:29:04.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.069+0000 7f87f2000640 1 -- 172.21.15.114:0/4173375515 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87ec049020 con 0x7f87fc0b3650 2024-03-06T07:29:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.070+0000 7f87f2000640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.070+0000 7f87f2000640 1 -- 172.21.15.114:0/4173375515 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f87ec0ac520 con 0x7f87fc0b3650 2024-03-06T07:29:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.070+0000 7f87f3e00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.071+0000 7f87f3e00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f87fc15e610 tx=0x7f87e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.075+0000 7f87f2000640 1 -- 172.21.15.114:0/4173375515 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87ec07b3f0 con 0x7f87fc0b3650 2024-03-06T07:29:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.311+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f87fc09da20 con 0x7f87fc0b3650 2024-03-06T07:29:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.853+0000 7f87f2000640 1 -- 172.21.15.114:0/4173375515 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f87ec07acf0 con 0x7f87fc0b3650 2024-03-06T07:29:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.862+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 msgr2=0x7f87d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.862+0000 7f8800c00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f87fc15e610 tx=0x7f87e4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.863+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3650 msgr2=0x7f87fc1459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.863+0000 7f8800c00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3650 0x7f87fc1459a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f87ec02db60 tx=0x7f87ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.864+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 shutdown_connections 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.864+0000 7f8800c00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc0cda90 0x7f87fc15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.864+0000 7f8800c00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0b3df0 0x7f87fc0acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.864+0000 7f8800c00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.864+0000 7f8800c00640 1 --2- 172.21.15.114:0/4173375515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0b3650 0x7f87fc1459a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.864+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 >> 172.21.15.114:0/4173375515 conn(0x7f87fc0b1940 msgr2=0x7f87fc14fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.865+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 shutdown_connections 2024-03-06T07:29:04.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:04.865+0000 7f8800c00640 1 -- 172.21.15.114:0/4173375515 wait complete. 2024-03-06T07:29:04.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:05.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.064+0000 7fd066000640 1 Processor -- start 2024-03-06T07:29:05.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.064+0000 7fd066000640 1 -- start start 2024-03-06T07:29:05.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.065+0000 7fd066000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015fe60 0x7fd060071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.065+0000 7fd066000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06006cef0 con 0x7fd06015f9d0 2024-03-06T07:29:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.065+0000 7fd066000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06006d1d0 con 0x7fd060144c00 2024-03-06T07:29:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.065+0000 7fd066000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06006d4b0 con 0x7fd06015fe60 2024-03-06T07:29:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05ea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd06015f9d0 0x7fd060070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51584/0 (socket says 172.21.15.114:51584) 2024-03-06T07:29:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05ea00640 1 -- 172.21.15.114:0/3587423110 learned_addr learned my addr 172.21.15.114:0/3587423110 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05fe00640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015fe60 0x7fd060071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781320664 0 0) 0x7fd06006cef0 con 0x7fd06015f9d0 2024-03-06T07:29:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd044003520 con 0x7fd06015f9d0 2024-03-06T07:29:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187344363 0 0) 0x7fd044003520 con 0x7fd06015f9d0 2024-03-06T07:29:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015fe60 msgr2=0x7fd060071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015fe60 0x7fd060071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 >> v1:172.21.15.177:6789/0 conn(0x7fd060144c00 legacy=0x7fd06006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06006d790 con 0x7fd06015f9d0 2024-03-06T07:29:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fd04c001670 con 0x7fd06015f9d0 2024-03-06T07:29:05.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 >> v1:172.21.15.114:6789/0 conn(0x7fd06015f9d0 legacy=0x7fd060070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.066+0000 7fd05e000640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd044003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05e000640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015fe60 0x7fd044004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05e000640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd044004d40 0x7fd0440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06006d4b0 con 0x7fd0440039e0 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04400ba70 con 0x7fd044004d40 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04400bc70 con 0x7fd06015fe60 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05f400640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd044003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05f400640 1 -- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015fe60 msgr2=0x7fd044004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05f400640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015fe60 0x7fd044004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05f400640 1 -- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd044004d40 msgr2=0x7fd0440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05f400640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd044004d40 0x7fd0440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.067+0000 7fd05f400640 1 -- 172.21.15.114:0/3587423110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044003520 con 0x7fd0440039e0 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.068+0000 7fd05f400640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd044003e00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd050001720 tx=0x7fd044005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c46f9a87e7a968 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.068+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050001370 con 0x7fd0440039e0 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.068+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0500052d0 con 0x7fd0440039e0 2024-03-06T07:29:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.068+0000 7fd05e000640 1 -- 172.21.15.114:0/3587423110 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050003830 con 0x7fd0440039e0 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.069+0000 7fd066000640 1 -- 172.21.15.114:0/3587423110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 msgr2=0x7fd044003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.069+0000 7fd066000640 1 --2- 172.21.15.114:0/3587423110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd044003e00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd050001720 tx=0x7fd044005830 comp rx=0 tx=0).stop 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.069+0000 7fd05fe00640 1 -- 172.21.15.114:0/3587423110 reap_dead start 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.069+0000 7fd066000640 1 -- 172.21.15.114:0/3587423110 shutdown_connections 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.069+0000 7fd066000640 1 -- 172.21.15.114:0/3587423110 >> 172.21.15.114:0/3587423110 conn(0x7fd0600b1730 msgr2=0x7fd0600b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.069+0000 7fd066000640 1 -- 172.21.15.114:0/3587423110 shutdown_connections 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.069+0000 7fd066000640 1 -- 172.21.15.114:0/3587423110 wait complete. 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.071+0000 7fd066000640 1 Processor -- start 2024-03-06T07:29:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.071+0000 7fd066000640 1 -- start start 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd066000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd060138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd066000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015ee70 0x7fd06015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd066000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06015b1d0 0x7fd06015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd066000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06006cb50 con 0x7fd0440039e0 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd066000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd060160730 con 0x7fd06015b1d0 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd066000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06006ccd0 con 0x7fd06015ee70 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd060138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd060138d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42252/0 (socket says 172.21.15.114:42252) 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 -- 172.21.15.114:0/795100414 learned_addr learned my addr 172.21.15.114:0/795100414 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 -- 172.21.15.114:0/795100414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015ee70 msgr2=0x7fd06015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015ee70 0x7fd06015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 -- 172.21.15.114:0/795100414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06015b1d0 msgr2=0x7fd06015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06015b1d0 0x7fd06015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.072+0000 7fd05f400640 1 -- 172.21.15.114:0/795100414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06015cfc0 con 0x7fd0440039e0 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.073+0000 7fd05f400640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd060138d20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd050008fd0 tx=0x7fd050001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.073+0000 7fd057400640 1 -- 172.21.15.114:0/795100414 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050041070 con 0x7fd0440039e0 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.073+0000 7fd057400640 1 -- 172.21.15.114:0/795100414 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd050000ed0 con 0x7fd0440039e0 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.073+0000 7fd057400640 1 -- 172.21.15.114:0/795100414 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050007a30 con 0x7fd0440039e0 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.073+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06015e390 con 0x7fd0440039e0 2024-03-06T07:29:05.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.073+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06015e950 con 0x7fd0440039e0 2024-03-06T07:29:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.074+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06015e180 con 0x7fd0440039e0 2024-03-06T07:29:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.075+0000 7fd057400640 1 -- 172.21.15.114:0/795100414 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd050044070 con 0x7fd0440039e0 2024-03-06T07:29:05.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.076+0000 7fd057400640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd04006b4d0 0x7fd04006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.076+0000 7fd057400640 1 -- 172.21.15.114:0/795100414 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0500abd00 con 0x7fd0440039e0 2024-03-06T07:29:05.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.076+0000 7fd05ea00640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd04006b4d0 0x7fd04006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.077+0000 7fd05ea00640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd04006b4d0 0x7fd04006d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd04c001050 tx=0x7fd04c001540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.081+0000 7fd057400640 1 -- 172.21.15.114:0/795100414 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd05007ab60 con 0x7fd0440039e0 2024-03-06T07:29:05.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.275+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd06004f330 con 0x7fd0440039e0 2024-03-06T07:29:05.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.278+0000 7fd057400640 1 -- 172.21.15.114:0/795100414 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16730 (secure 0 0 0) 0x7fd05007a460 con 0x7fd0440039e0 2024-03-06T07:29:05.280 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:05.280 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":32,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:04.312903+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:02.138886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:03.320986+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:05.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd04006b4d0 msgr2=0x7fd04006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd04006b4d0 0x7fd04006d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd04c001050 tx=0x7fd04c001540 comp rx=0 tx=0).stop 2024-03-06T07:29:05.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 msgr2=0x7fd060138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd060138d20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd050008fd0 tx=0x7fd050001120 comp rx=0 tx=0).stop 2024-03-06T07:29:05.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 shutdown_connections 2024-03-06T07:29:05.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06015b1d0 0x7fd06015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd06015ee70 0x7fd06015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd04006b4d0 0x7fd04006d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 --2- 172.21.15.114:0/795100414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0440039e0 0x7fd060138d20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 >> 172.21.15.114:0/795100414 conn(0x7fd0600b1730 msgr2=0x7fd0600b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.280+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 shutdown_connections 2024-03-06T07:29:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.281+0000 7fd066000640 1 -- 172.21.15.114:0/795100414 wait complete. 2024-03-06T07:29:05.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:05.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.485+0000 7eff00a00640 1 Processor -- start 2024-03-06T07:29:05.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.485+0000 7eff00a00640 1 -- start start 2024-03-06T07:29:05.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7eff00a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efefc1620c0 0x7efefc0b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7eff00a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc161900 con 0x7efefc0b40c0 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7eff00a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc15ebd0 con 0x7efefc0ae670 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7eff00a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc15eeb0 con 0x7efefc1620c0 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7efefa800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7efefc0b40c0 0x7efefc0b44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51590/0 (socket says 172.21.15.114:51590) 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7efefa800640 1 -- 172.21.15.114:0/764823724 learned_addr learned my addr 172.21.15.114:0/764823724 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7efefb200640 1 --2- 172.21.15.114:0/764823724 >> v2:172.21.15.114:3300/0 conn(0x7efefc1620c0 0x7efefc0b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7efef9400640 1 -- 172.21.15.114:0/764823724 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158820167 0 0) 0x7efefc161900 con 0x7efefc0b40c0 2024-03-06T07:29:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7efefb200640 1 -- 172.21.15.114:0/764823724 >> v1:172.21.15.177:6789/0 conn(0x7efefc0ae670 legacy=0x7efefc15e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.487+0000 7efefb200640 1 -- 172.21.15.114:0/764823724 >> v1:172.21.15.114:6789/0 conn(0x7efefc0b40c0 legacy=0x7efefc0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.488+0000 7efefb200640 1 -- 172.21.15.114:0/764823724 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc15f190 con 0x7efefc1620c0 2024-03-06T07:29:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7efefb200640 1 --2- 172.21.15.114:0/764823724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc1620c0 0x7efefc0b28e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efef4009d90 tx=0x7efef402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b413377fe4551a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7efef9400640 1 -- 172.21.15.114:0/764823724 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4039070 con 0x7efefc1620c0 2024-03-06T07:29:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7efef9400640 1 -- 172.21.15.114:0/764823724 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef402cc60 con 0x7efefc1620c0 2024-03-06T07:29:05.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7efef9400640 1 -- 172.21.15.114:0/764823724 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4034660 con 0x7efefc1620c0 2024-03-06T07:29:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7eff00a00640 1 -- 172.21.15.114:0/764823724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc1620c0 msgr2=0x7efefc0b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7eff00a00640 1 --2- 172.21.15.114:0/764823724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc1620c0 0x7efefc0b28e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efef4009d90 tx=0x7efef402beb0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7eff00a00640 1 -- 172.21.15.114:0/764823724 shutdown_connections 2024-03-06T07:29:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7eff00a00640 1 --2- 172.21.15.114:0/764823724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc1620c0 0x7efefc0b28e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7eff00a00640 1 -- 172.21.15.114:0/764823724 >> 172.21.15.114:0/764823724 conn(0x7efefc06afc0 msgr2=0x7efefc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7eff00a00640 1 -- 172.21.15.114:0/764823724 shutdown_connections 2024-03-06T07:29:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.489+0000 7eff00a00640 1 -- 172.21.15.114:0/764823724 wait complete. 2024-03-06T07:29:05.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.491+0000 7eff00a00640 1 Processor -- start 2024-03-06T07:29:05.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.491+0000 7eff00a00640 1 -- start start 2024-03-06T07:29:05.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7eff00a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc0ae670 0x7efefc152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7eff00a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efefc0b40c0 0x7efefc1577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7eff00a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc1620c0 0x7efefc160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7eff00a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc1587e0 con 0x7efefc0ae670 2024-03-06T07:29:05.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7eff00a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc158660 con 0x7efefc1620c0 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7eff00a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc162570 con 0x7efefc0b40c0 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7efefa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc0ae670 0x7efefc152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7efefa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc0ae670 0x7efefc152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42278/0 (socket says 172.21.15.114:42278) 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.492+0000 7efefa800640 1 -- 172.21.15.114:0/1754388185 learned_addr learned my addr 172.21.15.114:0/1754388185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efefa800640 1 -- 172.21.15.114:0/1754388185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efefc0b40c0 msgr2=0x7efefc1577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efefa800640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efefc0b40c0 0x7efefc1577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efefa800640 1 -- 172.21.15.114:0/1754388185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc1620c0 msgr2=0x7efefc160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efefa800640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc1620c0 0x7efefc160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efefa800640 1 -- 172.21.15.114:0/1754388185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc1f3e70 con 0x7efefc0ae670 2024-03-06T07:29:05.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efefa800640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc0ae670 0x7efefc152600 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efeec00c4d0 tx=0x7efeec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efef2a00640 1 -- 172.21.15.114:0/1754388185 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec018070 con 0x7efefc0ae670 2024-03-06T07:29:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efef2a00640 1 -- 172.21.15.114:0/1754388185 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec007500 con 0x7efefc0ae670 2024-03-06T07:29:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc1f4150 con 0x7efefc0ae670 2024-03-06T07:29:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7efef2a00640 1 -- 172.21.15.114:0/1754388185 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec004a20 con 0x7efefc0ae670 2024-03-06T07:29:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.493+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc1f4710 con 0x7efefc0ae670 2024-03-06T07:29:05.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.494+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc04f330 con 0x7efefc0ae670 2024-03-06T07:29:05.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.495+0000 7efef2a00640 1 -- 172.21.15.114:0/1754388185 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efeec01b070 con 0x7efefc0ae670 2024-03-06T07:29:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.496+0000 7efef2a00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efed806b4d0 0x7efed806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.496+0000 7efef2a00640 1 -- 172.21.15.114:0/1754388185 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efeec0895e0 con 0x7efefc0ae670 2024-03-06T07:29:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.496+0000 7efef9e00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efed806b4d0 0x7efed806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.497+0000 7efef9e00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efed806b4d0 0x7efed806d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efefc153130 tx=0x7efee40079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.501+0000 7efef2a00640 1 -- 172.21.15.114:0/1754388185 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efeec0583c0 con 0x7efefc0ae670 2024-03-06T07:29:05.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.680+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efefc0b11a0 con 0x7efefc0ae670 2024-03-06T07:29:05.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.681+0000 7efef2a00640 1 -- 172.21.15.114:0/1754388185 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16730 (secure 0 0 0) 0x7efeec057cc0 con 0x7efefc0ae670 2024-03-06T07:29:05.683 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:05.683 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":32,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:04.312903+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:02.138886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:03.320986+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efed806b4d0 msgr2=0x7efed806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efed806b4d0 0x7efed806d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efefc153130 tx=0x7efee40079b0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc0ae670 msgr2=0x7efefc152600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc0ae670 0x7efefc152600 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efeec00c4d0 tx=0x7efeec00c990 comp rx=0 tx=0).stop 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 shutdown_connections 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efefc1620c0 0x7efefc160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efefc0b40c0 0x7efefc1577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efed806b4d0 0x7efed806d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 --2- 172.21.15.114:0/1754388185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efefc0ae670 0x7efefc152600 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 >> 172.21.15.114:0/1754388185 conn(0x7efefc06afc0 msgr2=0x7efefc15d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 shutdown_connections 2024-03-06T07:29:05.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.684+0000 7eff00a00640 1 -- 172.21.15.114:0/1754388185 wait complete. 2024-03-06T07:29:05.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:29:05.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.914+0000 7fea37800640 1 Processor -- start 2024-03-06T07:29:05.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.914+0000 7fea37800640 1 -- start start 2024-03-06T07:29:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.915+0000 7fea37800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea300cda90 0x7fea300727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.915+0000 7fea37800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea300719e0 con 0x7fea300b3df0 2024-03-06T07:29:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.915+0000 7fea37800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea30071cc0 con 0x7fea300b3650 2024-03-06T07:29:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.915+0000 7fea37800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea30072310 con 0x7fea300cda90 2024-03-06T07:29:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.916+0000 7fea35800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea300cda90 0x7fea300727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.916+0000 7fea35800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea300cda90 0x7fea300727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42282/0 (socket says 172.21.15.114:42282) 2024-03-06T07:29:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.916+0000 7fea35800640 1 -- 172.21.15.114:0/1067835175 learned_addr learned my addr 172.21.15.114:0/1067835175 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.916+0000 7fea35800640 1 -- 172.21.15.114:0/1067835175 >> v1:172.21.15.177:6789/0 conn(0x7fea300b3650 legacy=0x7fea3006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.916+0000 7fea35800640 1 -- 172.21.15.114:0/1067835175 >> v1:172.21.15.114:6789/0 conn(0x7fea300b3df0 legacy=0x7fea300b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.916+0000 7fea35800640 1 -- 172.21.15.114:0/1067835175 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30071fa0 con 0x7fea300cda90 2024-03-06T07:29:05.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.917+0000 7fea35800640 1 --2- 172.21.15.114:0/1067835175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300cda90 0x7fea300727a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fea28009e50 tx=0x7fea2802e190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd05cabe9aaa227 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.917+0000 7fea2f400640 1 -- 172.21.15.114:0/1067835175 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2802ed30 con 0x7fea300cda90 2024-03-06T07:29:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea2f400640 1 -- 172.21.15.114:0/1067835175 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea280363f0 con 0x7fea300cda90 2024-03-06T07:29:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea2f400640 1 -- 172.21.15.114:0/1067835175 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea280367c0 con 0x7fea300cda90 2024-03-06T07:29:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea37800640 1 -- 172.21.15.114:0/1067835175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300cda90 msgr2=0x7fea300727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea37800640 1 --2- 172.21.15.114:0/1067835175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300cda90 0x7fea300727a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fea28009e50 tx=0x7fea2802e190 comp rx=0 tx=0).stop 2024-03-06T07:29:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea37800640 1 -- 172.21.15.114:0/1067835175 shutdown_connections 2024-03-06T07:29:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea37800640 1 --2- 172.21.15.114:0/1067835175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300cda90 0x7fea300727a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea37800640 1 -- 172.21.15.114:0/1067835175 >> 172.21.15.114:0/1067835175 conn(0x7fea300b1940 msgr2=0x7fea300b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea37800640 1 -- 172.21.15.114:0/1067835175 shutdown_connections 2024-03-06T07:29:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.918+0000 7fea37800640 1 -- 172.21.15.114:0/1067835175 wait complete. 2024-03-06T07:29:05.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.920+0000 7fea37800640 1 Processor -- start 2024-03-06T07:29:05.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.920+0000 7fea37800640 1 -- start start 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea37800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea300b3650 0x7fea30154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea37800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300b3df0 0x7fea3009cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea37800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea300cda90 0x7fea3009a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea37800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3015b3c0 con 0x7fea300b3df0 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea37800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3015b240 con 0x7fea300cda90 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea37800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea300cf510 con 0x7fea300b3650 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300b3df0 0x7fea3009cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300b3df0 0x7fea3009cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42288/0 (socket says 172.21.15.114:42288) 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 -- 172.21.15.114:0/4178689745 learned_addr learned my addr 172.21.15.114:0/4178689745 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 -- 172.21.15.114:0/4178689745 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea300b3650 msgr2=0x7fea30154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea300b3650 0x7fea30154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 -- 172.21.15.114:0/4178689745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea300cda90 msgr2=0x7fea3009a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea300cda90 0x7fea3009a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.921+0000 7fea2fe00640 1 -- 172.21.15.114:0/4178689745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3009c300 con 0x7fea300b3df0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.922+0000 7fea2fe00640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300b3df0 0x7fea3009cac0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00c430 tx=0x7fea1c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.922+0000 7fea2cc00640 1 -- 172.21.15.114:0/4178689745 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c011790 con 0x7fea300b3df0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.922+0000 7fea2cc00640 1 -- 172.21.15.114:0/4178689745 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1c017070 con 0x7fea300b3df0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.922+0000 7fea2cc00640 1 -- 172.21.15.114:0/4178689745 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c011bb0 con 0x7fea300b3df0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.922+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3009b390 con 0x7fea300b3df0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.922+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3009c5c0 con 0x7fea300b3df0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.923+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3004f330 con 0x7fea300b3df0 2024-03-06T07:29:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.924+0000 7fea2cc00640 1 -- 172.21.15.114:0/4178689745 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea1c01c460 con 0x7fea300b3df0 2024-03-06T07:29:05.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.925+0000 7fea2cc00640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea1006b4d0 0x7fea1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.925+0000 7fea2cc00640 1 -- 172.21.15.114:0/4178689745 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea1c08a0f0 con 0x7fea300b3df0 2024-03-06T07:29:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.925+0000 7fea34e00640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea1006b4d0 0x7fea1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.926+0000 7fea34e00640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea1006b4d0 0x7fea1006d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fea24000f60 tx=0x7fea240079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:05.934+0000 7fea2cc00640 1 -- 172.21.15.114:0/4178689745 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea1c058f60 con 0x7fea300b3df0 2024-03-06T07:29:06.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.156+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fea300cc0e0 con 0x7fea300b3df0 2024-03-06T07:29:06.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.876+0000 7fea2cc00640 1 -- 172.21.15.114:0/4178689745 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fea1c058860 con 0x7fea300b3df0 2024-03-06T07:29:06.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.879+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea1006b4d0 msgr2=0x7fea1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.879+0000 7fea37800640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea1006b4d0 0x7fea1006d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fea24000f60 tx=0x7fea240079d0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.879+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300b3df0 msgr2=0x7fea3009cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.879+0000 7fea37800640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300b3df0 0x7fea3009cac0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00c430 tx=0x7fea1c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 shutdown_connections 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea300cda90 0x7fea3009a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea1006b4d0 0x7fea1006d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea300b3df0 0x7fea3009cac0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 --2- 172.21.15.114:0/4178689745 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea300b3650 0x7fea30154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 >> 172.21.15.114:0/4178689745 conn(0x7fea300b1940 msgr2=0x7fea300cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 shutdown_connections 2024-03-06T07:29:06.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:06.880+0000 7fea37800640 1 -- 172.21.15.114:0/4178689745 wait complete. 2024-03-06T07:29:06.910 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:29:06.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:29:07.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.121+0000 7fa013200640 1 Processor -- start 2024-03-06T07:29:07.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.122+0000 7fa013200640 1 -- start start 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.123+0000 7fa013200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa014145600 0x7fa0140d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.123+0000 7fa013200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa0140719f0 0x7fa014071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.123+0000 7fa013200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0141459e0 con 0x7fa01406fdd0 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.123+0000 7fa013200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01406f940 con 0x7fa0140719f0 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.124+0000 7fa013200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01406ed30 con 0x7fa014145600 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.124+0000 7fa00be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa01406fdd0 0x7fa01415cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51608/0 (socket says 172.21.15.114:51608) 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.124+0000 7fa00be00640 1 -- 172.21.15.114:0/1911835783 learned_addr learned my addr 172.21.15.114:0/1911835783 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:07.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.124+0000 7fa012800640 1 --2- 172.21.15.114:0/1911835783 >> v2:172.21.15.114:3300/0 conn(0x7fa0140719f0 0x7fa014071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.124+0000 7fa011e00640 1 --2- 172.21.15.114:0/1911835783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa014145600 0x7fa0140d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.124+0000 7fa011400640 1 -- 172.21.15.114:0/1911835783 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260916936 0 0) 0x7fa0141459e0 con 0x7fa01406fdd0 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.124+0000 7fa011400640 1 -- 172.21.15.114:0/1911835783 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff8003520 con 0x7fa01406fdd0 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.125+0000 7fa012800640 1 -- 172.21.15.114:0/1911835783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa014145600 msgr2=0x7fa0140d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.125+0000 7fa012800640 1 --2- 172.21.15.114:0/1911835783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa014145600 0x7fa0140d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.125+0000 7fa012800640 1 -- 172.21.15.114:0/1911835783 >> v1:172.21.15.114:6789/0 conn(0x7fa01406fdd0 legacy=0x7fa01415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.125+0000 7fa012800640 1 -- 172.21.15.114:0/1911835783 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01406f010 con 0x7fa0140719f0 2024-03-06T07:29:07.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.126+0000 7fa012800640 1 --2- 172.21.15.114:0/1911835783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0140719f0 0x7fa014071ea0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d90 tx=0x7fa00c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933b5d10efcafc7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.126+0000 7fa011400640 1 -- 172.21.15.114:0/1911835783 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c039070 con 0x7fa0140719f0 2024-03-06T07:29:07.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.126+0000 7fa011400640 1 -- 172.21.15.114:0/1911835783 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c02cc50 con 0x7fa0140719f0 2024-03-06T07:29:07.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.126+0000 7fa011400640 1 -- 172.21.15.114:0/1911835783 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c034440 con 0x7fa0140719f0 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.126+0000 7fa013200640 1 -- 172.21.15.114:0/1911835783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0140719f0 msgr2=0x7fa014071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.126+0000 7fa013200640 1 --2- 172.21.15.114:0/1911835783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0140719f0 0x7fa014071ea0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d90 tx=0x7fa00c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.127+0000 7fa013200640 1 -- 172.21.15.114:0/1911835783 shutdown_connections 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.127+0000 7fa013200640 1 --2- 172.21.15.114:0/1911835783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0140719f0 0x7fa014071ea0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.127+0000 7fa013200640 1 --2- 172.21.15.114:0/1911835783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa014145600 0x7fa0140d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.127+0000 7fa013200640 1 -- 172.21.15.114:0/1911835783 >> 172.21.15.114:0/1911835783 conn(0x7fa0140d0a20 msgr2=0x7fa0140d0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.127+0000 7fa013200640 1 -- 172.21.15.114:0/1911835783 shutdown_connections 2024-03-06T07:29:07.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.127+0000 7fa013200640 1 -- 172.21.15.114:0/1911835783 wait complete. 2024-03-06T07:29:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.128+0000 7fa013200640 1 Processor -- start 2024-03-06T07:29:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.129+0000 7fa013200640 1 -- start start 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.129+0000 7fa013200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa01406fdd0 0x7fa01415edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.129+0000 7fa013200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0140719f0 0x7fa01415dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.129+0000 7fa013200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa014145600 0x7fa01415e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.129+0000 7fa013200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140ce920 con 0x7fa014145600 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.129+0000 7fa013200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01406f5d0 con 0x7fa0140719f0 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa013200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0140ceaa0 con 0x7fa01406fdd0 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa014145600 0x7fa01415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa014145600 0x7fa01415e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42302/0 (socket says 172.21.15.114:42302) 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 -- 172.21.15.114:0/3191268266 learned_addr learned my addr 172.21.15.114:0/3191268266 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa00be00640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0140719f0 0x7fa01415dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 -- 172.21.15.114:0/3191268266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa01406fdd0 msgr2=0x7fa01415edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa01406fdd0 0x7fa01415edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 -- 172.21.15.114:0/3191268266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0140719f0 msgr2=0x7fa01415dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0140719f0 0x7fa01415dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.130+0000 7fa012800640 1 -- 172.21.15.114:0/3191268266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0141f4d30 con 0x7fa014145600 2024-03-06T07:29:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.131+0000 7fa012800640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa014145600 0x7fa01415e270 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d90 tx=0x7fa00c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.131+0000 7fa00a000640 1 -- 172.21.15.114:0/3191268266 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c039050 con 0x7fa014145600 2024-03-06T07:29:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.131+0000 7fa00a000640 1 -- 172.21.15.114:0/3191268266 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c045070 con 0x7fa014145600 2024-03-06T07:29:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.131+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01415c5a0 con 0x7fa014145600 2024-03-06T07:29:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.131+0000 7fa00a000640 1 -- 172.21.15.114:0/3191268266 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c03e610 con 0x7fa014145600 2024-03-06T07:29:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.131+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01415f330 con 0x7fa014145600 2024-03-06T07:29:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.133+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01404f330 con 0x7fa014145600 2024-03-06T07:29:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.134+0000 7fa00a000640 1 -- 172.21.15.114:0/3191268266 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa00c03e7b0 con 0x7fa014145600 2024-03-06T07:29:07.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.134+0000 7fa00a000640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.135+0000 7fa00a000640 1 -- 172.21.15.114:0/3191268266 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa00c0acd60 con 0x7fa014145600 2024-03-06T07:29:07.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.135+0000 7fa011e00640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.136+0000 7fa011e00640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ff006b4d0 0x7f9ff006d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa004006fd0 tx=0x7fa004008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.139+0000 7fa00a000640 1 -- 172.21.15.114:0/3191268266 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa00c07bcf0 con 0x7fa014145600 2024-03-06T07:29:07.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.378+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa0140b2050 con 0x7fa014145600 2024-03-06T07:29:07.797 INFO:tasks.ceph.mds.c.smithi114.stderr:2024-03-06T07:29:07.795+0000 7f92b3a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:07.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.887+0000 7fa00a000640 1 -- 172.21.15.114:0/3191268266 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa00c07b5f0 con 0x7fa014145600 2024-03-06T07:29:07.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.891+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ff006b4d0 msgr2=0x7f9ff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.891+0000 7fa013200640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ff006b4d0 0x7f9ff006d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa004006fd0 tx=0x7fa004008040 comp rx=0 tx=0).stop 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.891+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa014145600 msgr2=0x7fa01415e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.891+0000 7fa013200640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa014145600 0x7fa01415e270 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009d90 tx=0x7fa00c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.891+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 shutdown_connections 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.891+0000 7fa013200640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ff006b4d0 0x7f9ff006d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.891+0000 7fa013200640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa014145600 0x7fa01415e270 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.892+0000 7fa013200640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0140719f0 0x7fa01415dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.892+0000 7fa013200640 1 --2- 172.21.15.114:0/3191268266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa01406fdd0 0x7fa01415edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.892+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 >> 172.21.15.114:0/3191268266 conn(0x7fa0140d0a20 msgr2=0x7fa0140d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.892+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 shutdown_connections 2024-03-06T07:29:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:07.892+0000 7fa013200640 1 -- 172.21.15.114:0/3191268266 wait complete. 2024-03-06T07:29:07.914 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:29:07.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:08.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.122+0000 7f4306a00640 1 Processor -- start 2024-03-06T07:29:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.122+0000 7f4306a00640 1 -- start start 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f4306a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f430015f730 0x7f43000cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f4306a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f43000c5970 0x7f43000c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f4306a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43000b3340 con 0x7f43000b2e60 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f4306a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43000cd320 con 0x7f43000c5970 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f4306a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43000cdbc0 con 0x7f430015f730 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f4304a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f43000c5970 0x7f43000c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f42ff400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f43000b2e60 0x7f43000cfd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58792/0 (socket says 172.21.15.114:58792) 2024-03-06T07:29:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.124+0000 7f42ff400640 1 -- 172.21.15.114:0/1365574531 learned_addr learned my addr 172.21.15.114:0/1365574531 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.125+0000 7f42ffe00640 1 --2- 172.21.15.114:0/1365574531 >> v2:172.21.15.177:3300/0 conn(0x7f430015f730 0x7f43000cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.125+0000 7f42fea00640 1 -- 172.21.15.114:0/1365574531 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995160233 0 0) 0x7f43000b3340 con 0x7f43000b2e60 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.125+0000 7f4304a00640 1 -- 172.21.15.114:0/1365574531 >> v2:172.21.15.177:3300/0 conn(0x7f430015f730 msgr2=0x7f43000cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.125+0000 7f4304a00640 1 --2- 172.21.15.114:0/1365574531 >> v2:172.21.15.177:3300/0 conn(0x7f430015f730 0x7f43000cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.125+0000 7f4304a00640 1 -- 172.21.15.114:0/1365574531 >> v1:172.21.15.114:6789/0 conn(0x7f43000b2e60 legacy=0x7f43000cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.125+0000 7f4304a00640 1 -- 172.21.15.114:0/1365574531 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43000cd570 con 0x7f43000c5970 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.126+0000 7f4304a00640 1 --2- 172.21.15.114:0/1365574531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000c5970 0x7f43000c5de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4300071dc0 tx=0x7f42f802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acecfc8341d1d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.126+0000 7f42fea00640 1 -- 172.21.15.114:0/1365574531 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8039070 con 0x7f43000c5970 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.126+0000 7f42fea00640 1 -- 172.21.15.114:0/1365574531 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f802cc70 con 0x7f43000c5970 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.126+0000 7f42fea00640 1 -- 172.21.15.114:0/1365574531 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8034660 con 0x7f43000c5970 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 -- 172.21.15.114:0/1365574531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000c5970 msgr2=0x7f43000c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 --2- 172.21.15.114:0/1365574531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000c5970 0x7f43000c5de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4300071dc0 tx=0x7f42f802bec0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 -- 172.21.15.114:0/1365574531 shutdown_connections 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 --2- 172.21.15.114:0/1365574531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000c5970 0x7f43000c5de0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 --2- 172.21.15.114:0/1365574531 >> v2:172.21.15.177:3300/0 conn(0x7f430015f730 0x7f43000cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 -- 172.21.15.114:0/1365574531 >> 172.21.15.114:0/1365574531 conn(0x7f43000b3800 msgr2=0x7f43000b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 -- 172.21.15.114:0/1365574531 shutdown_connections 2024-03-06T07:29:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.127+0000 7f4306a00640 1 -- 172.21.15.114:0/1365574531 wait complete. 2024-03-06T07:29:08.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.129+0000 7f4306a00640 1 Processor -- start 2024-03-06T07:29:08.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.129+0000 7f4306a00640 1 -- start start 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f4306a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000b2e60 0x7f4300152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f4306a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43000c5970 0x7f43000b7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f4306a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f430015f730 0x7f4300151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f4306a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430015fb10 con 0x7f43000b2e60 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f4306a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43000d0ae0 con 0x7f43000c5970 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f4306a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430015fc90 con 0x7f430015f730 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000b2e60 0x7f4300152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000b2e60 0x7f4300152000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40080/0 (socket says 172.21.15.114:40080) 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 -- 172.21.15.114:0/4174610422 learned_addr learned my addr 172.21.15.114:0/4174610422 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 -- 172.21.15.114:0/4174610422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f430015f730 msgr2=0x7f4300151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f430015f730 0x7f4300151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 -- 172.21.15.114:0/4174610422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43000c5970 msgr2=0x7f43000b7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43000c5970 0x7f43000b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.131+0000 7f42ffe00640 1 -- 172.21.15.114:0/4174610422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43001f8530 con 0x7f43000b2e60 2024-03-06T07:29:08.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.132+0000 7f42ffe00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000b2e60 0x7f4300152000 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f42f000d800 tx=0x7f42f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.132+0000 7f42f7e00640 1 -- 172.21.15.114:0/4174610422 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000b6c0 con 0x7f43000b2e60 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.132+0000 7f42f7e00640 1 -- 172.21.15.114:0/4174610422 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f0019070 con 0x7f43000b2e60 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.132+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43000c4990 con 0x7f43000b2e60 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.132+0000 7f42f7e00640 1 -- 172.21.15.114:0/4174610422 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000ba60 con 0x7f43000b2e60 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.132+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43000c4f50 con 0x7f43000b2e60 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.133+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430006bbc0 con 0x7f43000b2e60 2024-03-06T07:29:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.134+0000 7f42f7e00640 1 -- 172.21.15.114:0/4174610422 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42f000bc00 con 0x7f43000b2e60 2024-03-06T07:29:08.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.135+0000 7f42f7e00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42e006b4d0 0x7f42e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.135+0000 7f42f7e00640 1 -- 172.21.15.114:0/4174610422 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42f008a6f0 con 0x7f43000b2e60 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.135+0000 7f42ff400640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42e006b4d0 0x7f42e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.136+0000 7f42ff400640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42e006b4d0 0x7f42e006d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f42ec007fd0 tx=0x7f42ec007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.144+0000 7f42f7e00640 1 -- 172.21.15.114:0/4174610422 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42f0011340 con 0x7f43000b2e60 2024-03-06T07:29:08.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.377+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f430004f330 con 0x7f43000b2e60 2024-03-06T07:29:08.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.378+0000 7f42f7e00640 1 -- 172.21.15.114:0/4174610422 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f42f0059560 con 0x7f43000b2e60 2024-03-06T07:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:08.380 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42e006b4d0 msgr2=0x7f42e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42e006b4d0 0x7f42e006d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f42ec007fd0 tx=0x7f42ec007960 comp rx=0 tx=0).stop 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000b2e60 msgr2=0x7f4300152000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000b2e60 0x7f4300152000 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f42f000d800 tx=0x7f42f000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 shutdown_connections 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f430015f730 0x7f4300151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43000c5970 0x7f43000b7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42e006b4d0 0x7f42e006d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 --2- 172.21.15.114:0/4174610422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43000b2e60 0x7f4300152000 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 >> 172.21.15.114:0/4174610422 conn(0x7f43000b3800 msgr2=0x7f43000ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 shutdown_connections 2024-03-06T07:29:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.381+0000 7f4306a00640 1 -- 172.21.15.114:0/4174610422 wait complete. 2024-03-06T07:29:08.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:08.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.593+0000 7f79c3200640 1 Processor -- start 2024-03-06T07:29:08.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.593+0000 7f79c3200640 1 -- start start 2024-03-06T07:29:08.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79c3200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79bc151660 0x7f79bc071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79c3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc15dff0 0x7f79bc15e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79c3200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79bc071a60 con 0x7f79bc0cf430 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79c3200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79bc0b3ee0 con 0x7f79bc151660 2024-03-06T07:29:08.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79c3200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc0b35b0 con 0x7f79bc15dff0 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79bbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79bc151660 0x7f79bc071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79bbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79bc151660 0x7f79bc071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40096/0 (socket says 172.21.15.114:40096) 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.596+0000 7f79bbe00640 1 -- 172.21.15.114:0/3635311799 learned_addr learned my addr 172.21.15.114:0/3635311799 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.597+0000 7f79bbe00640 1 -- 172.21.15.114:0/3635311799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc15dff0 msgr2=0x7f79bc15e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.597+0000 7f79c1200640 1 --2- 172.21.15.114:0/3635311799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc15dff0 0x7f79bc15e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.597+0000 7f79bbe00640 1 --2- 172.21.15.114:0/3635311799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc15dff0 0x7f79bc15e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.597+0000 7f79bbe00640 1 -- 172.21.15.114:0/3635311799 >> v1:172.21.15.177:6789/0 conn(0x7f79bc0cf430 legacy=0x7f79bc06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.597+0000 7f79bbe00640 1 -- 172.21.15.114:0/3635311799 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc0b3890 con 0x7f79bc151660 2024-03-06T07:29:08.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.597+0000 7f79c1200640 1 --2- 172.21.15.114:0/3635311799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc15dff0 0x7f79bc15e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.598+0000 7f79bbe00640 1 --2- 172.21.15.114:0/3635311799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc071520 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00af60 tx=0x7f79ac02ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7e1fcdc4bf6358 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.598+0000 7f79baa00640 1 -- 172.21.15.114:0/3635311799 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac02d850 con 0x7f79bc151660 2024-03-06T07:29:08.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.598+0000 7f79baa00640 1 -- 172.21.15.114:0/3635311799 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ac02d9f0 con 0x7f79bc151660 2024-03-06T07:29:08.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.598+0000 7f79baa00640 1 -- 172.21.15.114:0/3635311799 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ac02ddc0 con 0x7f79bc151660 2024-03-06T07:29:08.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 -- 172.21.15.114:0/3635311799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 msgr2=0x7f79bc071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 --2- 172.21.15.114:0/3635311799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc071520 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00af60 tx=0x7f79ac02ceb0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 -- 172.21.15.114:0/3635311799 shutdown_connections 2024-03-06T07:29:08.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 --2- 172.21.15.114:0/3635311799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc15dff0 0x7f79bc15e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 --2- 172.21.15.114:0/3635311799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc071520 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 -- 172.21.15.114:0/3635311799 >> 172.21.15.114:0/3635311799 conn(0x7f79bc0b1730 msgr2=0x7f79bc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 -- 172.21.15.114:0/3635311799 shutdown_connections 2024-03-06T07:29:08.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.599+0000 7f79c3200640 1 -- 172.21.15.114:0/3635311799 wait complete. 2024-03-06T07:29:08.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.601+0000 7f79c3200640 1 Processor -- start 2024-03-06T07:29:08.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.601+0000 7f79c3200640 1 -- start start 2024-03-06T07:29:08.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.602+0000 7f79c3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc0cf430 0x7f79bc0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79c3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc0aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79c3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79bc15dff0 0x7f79bc14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79c3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc06daf0 con 0x7f79bc151660 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79c3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc071d30 con 0x7f79bc15dff0 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79c3200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc06cb50 con 0x7f79bc0cf430 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc0aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc0aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40112/0 (socket says 172.21.15.114:40112) 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 -- 172.21.15.114:0/2352647811 learned_addr learned my addr 172.21.15.114:0/2352647811 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bbe00640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc0cf430 0x7f79bc0a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79c1200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79bc15dff0 0x7f79bc14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 -- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc0cf430 msgr2=0x7f79bc0a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc0cf430 0x7f79bc0a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 -- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79bc15dff0 msgr2=0x7f79bc14fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79bc15dff0 0x7f79bc14fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79bb400640 1 -- 172.21.15.114:0/2352647811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc0af300 con 0x7f79bc151660 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.603+0000 7f79c1200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79bc15dff0 0x7f79bc14fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.604+0000 7f79bb400640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc0aeb20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79a400c460 tx=0x7f79a400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.604+0000 7f79b3e00640 1 -- 172.21.15.114:0/2352647811 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4016020 con 0x7f79bc151660 2024-03-06T07:29:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.604+0000 7f79b3e00640 1 -- 172.21.15.114:0/2352647811 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a4012870 con 0x7f79bc151660 2024-03-06T07:29:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.604+0000 7f79b3e00640 1 -- 172.21.15.114:0/2352647811 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4012c60 con 0x7f79bc151660 2024-03-06T07:29:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.604+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79bc0af060 con 0x7f79bc151660 2024-03-06T07:29:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.604+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79bc1f7bd0 con 0x7f79bc151660 2024-03-06T07:29:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.605+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79bc04f330 con 0x7f79bc151660 2024-03-06T07:29:08.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.606+0000 7f79b3e00640 1 -- 172.21.15.114:0/2352647811 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79a4019020 con 0x7f79bc151660 2024-03-06T07:29:08.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.607+0000 7f79b3e00640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799806b500 0x7f799806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.607+0000 7f79b3e00640 1 -- 172.21.15.114:0/2352647811 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79a4089ec0 con 0x7f79bc151660 2024-03-06T07:29:08.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.607+0000 7f79bbe00640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799806b500 0x7f799806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.608+0000 7f79bbe00640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799806b500 0x7f799806d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79ac036000 tx=0x7f79ac031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.612+0000 7f79b3e00640 1 -- 172.21.15.114:0/2352647811 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79a4020030 con 0x7f79bc151660 2024-03-06T07:29:08.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.807+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79bc06f3c0 con 0x7f79bc151660 2024-03-06T07:29:08.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.808+0000 7f79b3e00640 1 -- 172.21.15.114:0/2352647811 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f79a4058d30 con 0x7f79bc151660 2024-03-06T07:29:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:08.810 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":33,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:05.853392+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:02.138886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:03.320986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799806b500 msgr2=0x7f799806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799806b500 0x7f799806d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79ac036000 tx=0x7f79ac031000 comp rx=0 tx=0).stop 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 msgr2=0x7f79bc0aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc0aeb20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79a400c460 tx=0x7f79a400c920 comp rx=0 tx=0).stop 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 shutdown_connections 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79bc15dff0 0x7f79bc14fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799806b500 0x7f799806d9c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79bc151660 0x7f79bc0aeb20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 --2- 172.21.15.114:0/2352647811 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79bc0cf430 0x7f79bc0a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 >> 172.21.15.114:0/2352647811 conn(0x7f79bc0b1730 msgr2=0x7f79bc06dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 shutdown_connections 2024-03-06T07:29:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:08.811+0000 7f79c3200640 1 -- 172.21.15.114:0/2352647811 wait complete. 2024-03-06T07:29:08.831 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:29:08.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.031+0000 7f8400e00640 1 Processor -- start 2024-03-06T07:29:09.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.031+0000 7f8400e00640 1 -- start start 2024-03-06T07:29:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.032+0000 7f8400e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0d24c0 0x7f83fc0d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.032+0000 7f8400e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f83fc0b2de0 0x7f83fc0d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.032+0000 7f8400e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f83fc0d07f0 0x7f83fc0d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.032+0000 7f8400e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83fc0d4a00 con 0x7f83fc0b2de0 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.032+0000 7f8400e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83fc0cf7e0 con 0x7f83fc0d07f0 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.032+0000 7f8400e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc0d0080 con 0x7f83fc0d24c0 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f83fc0b2de0 0x7f83fc0d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f83fc0b2de0 0x7f83fc0d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40128/0 (socket says 172.21.15.114:40128) 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 -- 172.21.15.114:0/4216376895 learned_addr learned my addr 172.21.15.114:0/4216376895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83fa800640 1 --2- 172.21.15.114:0/4216376895 >> v2:172.21.15.177:3300/0 conn(0x7f83fc0d07f0 0x7f83fc0d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9e00640 1 --2- 172.21.15.114:0/4216376895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0d24c0 0x7f83fc0d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 -- 172.21.15.114:0/4216376895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0d24c0 msgr2=0x7f83fc0d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 --2- 172.21.15.114:0/4216376895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0d24c0 0x7f83fc0d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 -- 172.21.15.114:0/4216376895 >> v2:172.21.15.177:3300/0 conn(0x7f83fc0d07f0 msgr2=0x7f83fc0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 --2- 172.21.15.114:0/4216376895 >> v2:172.21.15.177:3300/0 conn(0x7f83fc0d07f0 0x7f83fc0d0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9400640 1 -- 172.21.15.114:0/4216376895 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc0d2de0 con 0x7f83fc0b2de0 2024-03-06T07:29:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.033+0000 7f83f9e00640 1 --2- 172.21.15.114:0/4216376895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0d24c0 0x7f83fc0d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.034+0000 7f83f9400640 1 --2- 172.21.15.114:0/4216376895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0b2de0 0x7f83fc0d28a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0ce760 tx=0x7f83e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c088cf81a9be95ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.035+0000 7f83f8a00640 1 -- 172.21.15.114:0/4216376895 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e802e070 con 0x7f83fc0b2de0 2024-03-06T07:29:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.035+0000 7f83f8a00640 1 -- 172.21.15.114:0/4216376895 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e8037070 con 0x7f83fc0b2de0 2024-03-06T07:29:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.035+0000 7f83f8a00640 1 -- 172.21.15.114:0/4216376895 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8031a00 con 0x7f83fc0b2de0 2024-03-06T07:29:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.035+0000 7f8400e00640 1 -- 172.21.15.114:0/4216376895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0b2de0 msgr2=0x7f83fc0d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.035+0000 7f8400e00640 1 --2- 172.21.15.114:0/4216376895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0b2de0 0x7f83fc0d28a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0ce760 tx=0x7f83e802b880 comp rx=0 tx=0).stop 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.036+0000 7f8400e00640 1 -- 172.21.15.114:0/4216376895 shutdown_connections 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.036+0000 7f8400e00640 1 --2- 172.21.15.114:0/4216376895 >> v2:172.21.15.177:3300/0 conn(0x7f83fc0d07f0 0x7f83fc0d0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.036+0000 7f8400e00640 1 --2- 172.21.15.114:0/4216376895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0b2de0 0x7f83fc0d28a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.036+0000 7f8400e00640 1 --2- 172.21.15.114:0/4216376895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0d24c0 0x7f83fc0d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.036+0000 7f8400e00640 1 -- 172.21.15.114:0/4216376895 >> 172.21.15.114:0/4216376895 conn(0x7f83fc0d3ee0 msgr2=0x7f83fc0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.036+0000 7f8400e00640 1 -- 172.21.15.114:0/4216376895 shutdown_connections 2024-03-06T07:29:09.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.036+0000 7f8400e00640 1 -- 172.21.15.114:0/4216376895 wait complete. 2024-03-06T07:29:09.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.037+0000 7f8400e00640 1 Processor -- start 2024-03-06T07:29:09.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.037+0000 7f8400e00640 1 -- start start 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.038+0000 7f8400e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0b2de0 0x7f83fc0cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.038+0000 7f8400e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83fc0d07f0 0x7f83fc0cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.038+0000 7f8400e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0d24c0 0x7f83fc0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.038+0000 7f8400e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc0cfd10 con 0x7f83fc0d24c0 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.038+0000 7f8400e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc0cfa30 con 0x7f83fc0d07f0 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.038+0000 7f8400e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83fc15e5b0 con 0x7f83fc0b2de0 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0d24c0 0x7f83fc0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0d24c0 0x7f83fc0cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40140/0 (socket says 172.21.15.114:40140) 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 -- 172.21.15.114:0/1813427635 learned_addr learned my addr 172.21.15.114:0/1813427635 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83f9400640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83fc0d07f0 0x7f83fc0cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83f9e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0b2de0 0x7f83fc0cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 -- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0b2de0 msgr2=0x7f83fc0cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0b2de0 0x7f83fc0cd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 -- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83fc0d07f0 msgr2=0x7f83fc0cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83fc0d07f0 0x7f83fc0cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 -- 172.21.15.114:0/1813427635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc0c7d50 con 0x7f83fc0d24c0 2024-03-06T07:29:09.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.039+0000 7f83fa800640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0d24c0 0x7f83fc0cc540 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f83f4007d20 tx=0x7f83f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.040+0000 7f83f2000640 1 -- 172.21.15.114:0/1813427635 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f4014420 con 0x7f83fc0d24c0 2024-03-06T07:29:09.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.040+0000 7f83f2000640 1 -- 172.21.15.114:0/1813427635 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f4019070 con 0x7f83fc0d24c0 2024-03-06T07:29:09.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.040+0000 7f83f2000640 1 -- 172.21.15.114:0/1813427635 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f4014820 con 0x7f83fc0d24c0 2024-03-06T07:29:09.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.040+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83fc147c80 con 0x7f83fc0d24c0 2024-03-06T07:29:09.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.040+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83fc148240 con 0x7f83fc0d24c0 2024-03-06T07:29:09.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.041+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83fc04f330 con 0x7f83fc0d24c0 2024-03-06T07:29:09.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.042+0000 7f83f2000640 1 -- 172.21.15.114:0/1813427635 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83f40149c0 con 0x7f83fc0d24c0 2024-03-06T07:29:09.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.043+0000 7f83f2000640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f83dc06b4d0 0x7f83dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.043+0000 7f83f2000640 1 -- 172.21.15.114:0/1813427635 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83f408a9d0 con 0x7f83fc0d24c0 2024-03-06T07:29:09.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.044+0000 7f83f9e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f83dc06b4d0 0x7f83dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.044+0000 7f83f9e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f83dc06b4d0 0x7f83dc06d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00a990 tx=0x7f83ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.048+0000 7f83f2000640 1 -- 172.21.15.114:0/1813427635 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83f4059840 con 0x7f83fc0d24c0 2024-03-06T07:29:09.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.242+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83fc06d230 con 0x7f83dc06b4d0 2024-03-06T07:29:09.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.263+0000 7f83f2000640 1 -- 172.21.15.114:0/1813427635 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251951 (secure 0 0 0) 0x7f83fc06d230 con 0x7f83dc06b4d0 2024-03-06T07:29:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:09.270 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f83dc06b4d0 msgr2=0x7f83dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f83dc06b4d0 0x7f83dc06d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00a990 tx=0x7f83ec008040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0d24c0 msgr2=0x7f83fc0cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0d24c0 0x7f83fc0cc540 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f83f4007d20 tx=0x7f83f400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 shutdown_connections 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f83dc06b4d0 0x7f83dc06d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83fc0d24c0 0x7f83fc0cc540 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83fc0d07f0 0x7f83fc0cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 --2- 172.21.15.114:0/1813427635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83fc0b2de0 0x7f83fc0cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 >> 172.21.15.114:0/1813427635 conn(0x7f83fc0d3ee0 msgr2=0x7f83fc0b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 shutdown_connections 2024-03-06T07:29:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.269+0000 7f8400e00640 1 -- 172.21.15.114:0/1813427635 wait complete. 2024-03-06T07:29:09.290 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-06T07:29:09.120412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":182,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220328,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754392,"statfs":{"total":773094113280,"available":772868497408,"internally_reserved":0,"allocated":4235264,"data_stored":2359780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001197"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201506+0000","last_change":"2024-03-06T07:29:04.868489+0000","last_active":"2024-03-06T07:29:06.201506+0000","last_peered":"2024-03-06T07:29:06.201506+0000","last_clean":"2024-03-06T07:29:06.201506+0000","last_became_active":"2024-03-06T07:29:04.868206+0000","last_became_peered":"2024-03-06T07:29:04.868206+0000","last_unstale":"2024-03-06T07:29:06.201506+0000","last_undegraded":"2024-03-06T07:29:06.201506+0000","last_fullsized":"2024-03-06T07:29:06.201506+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:46.317405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867783+0000","last_change":"2024-03-06T07:29:04.882576+0000","last_active":"2024-03-06T07:29:05.867783+0000","last_peered":"2024-03-06T07:29:05.867783+0000","last_clean":"2024-03-06T07:29:05.867783+0000","last_became_active":"2024-03-06T07:29:04.882298+0000","last_became_peered":"2024-03-06T07:29:04.882298+0000","last_unstale":"2024-03-06T07:29:05.867783+0000","last_undegraded":"2024-03-06T07:29:05.867783+0000","last_fullsized":"2024-03-06T07:29:05.867783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:30.478664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199255+0000","last_change":"2024-03-06T07:29:04.886479+0000","last_active":"2024-03-06T07:29:06.199255+0000","last_peered":"2024-03-06T07:29:06.199255+0000","last_clean":"2024-03-06T07:29:06.199255+0000","last_became_active":"2024-03-06T07:29:04.886139+0000","last_became_peered":"2024-03-06T07:29:04.886139+0000","last_unstale":"2024-03-06T07:29:06.199255+0000","last_undegraded":"2024-03-06T07:29:06.199255+0000","last_fullsized":"2024-03-06T07:29:06.199255+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:10.951882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874887+0000","last_change":"2024-03-06T07:29:04.886643+0000","last_active":"2024-03-06T07:29:05.874887+0000","last_peered":"2024-03-06T07:29:05.874887+0000","last_clean":"2024-03-06T07:29:05.874887+0000","last_became_active":"2024-03-06T07:29:04.886371+0000","last_became_peered":"2024-03-06T07:29:04.886371+0000","last_unstale":"2024-03-06T07:29:05.874887+0000","last_undegraded":"2024-03-06T07:29:05.874887+0000","last_fullsized":"2024-03-06T07:29:05.874887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:36.195095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.196865+0000","last_change":"2024-03-06T07:29:04.873221+0000","last_active":"2024-03-06T07:29:06.196865+0000","last_peered":"2024-03-06T07:29:06.196865+0000","last_clean":"2024-03-06T07:29:06.196865+0000","last_became_active":"2024-03-06T07:29:04.872941+0000","last_became_peered":"2024-03-06T07:29:04.872941+0000","last_unstale":"2024-03-06T07:29:06.196865+0000","last_undegraded":"2024-03-06T07:29:06.196865+0000","last_fullsized":"2024-03-06T07:29:06.196865+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:33.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867244+0000","last_change":"2024-03-06T07:29:04.874030+0000","last_active":"2024-03-06T07:29:05.867244+0000","last_peered":"2024-03-06T07:29:05.867244+0000","last_clean":"2024-03-06T07:29:05.867244+0000","last_became_active":"2024-03-06T07:29:04.873748+0000","last_became_peered":"2024-03-06T07:29:04.873748+0000","last_unstale":"2024-03-06T07:29:05.867244+0000","last_undegraded":"2024-03-06T07:29:05.867244+0000","last_fullsized":"2024-03-06T07:29:05.867244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:14.905700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200103+0000","last_change":"2024-03-06T07:29:04.876229+0000","last_active":"2024-03-06T07:29:06.200103+0000","last_peered":"2024-03-06T07:29:06.200103+0000","last_clean":"2024-03-06T07:29:06.200103+0000","last_became_active":"2024-03-06T07:29:04.875681+0000","last_became_peered":"2024-03-06T07:29:04.875681+0000","last_unstale":"2024-03-06T07:29:06.200103+0000","last_undegraded":"2024-03-06T07:29:06.200103+0000","last_fullsized":"2024-03-06T07:29:06.200103+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:04.236079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197968+0000","last_change":"2024-03-06T07:29:04.879283+0000","last_active":"2024-03-06T07:29:06.197968+0000","last_peered":"2024-03-06T07:29:06.197968+0000","last_clean":"2024-03-06T07:29:06.197968+0000","last_became_active":"2024-03-06T07:29:04.878997+0000","last_became_peered":"2024-03-06T07:29:04.878997+0000","last_unstale":"2024-03-06T07:29:06.197968+0000","last_undegraded":"2024-03-06T07:29:06.197968+0000","last_fullsized":"2024-03-06T07:29:06.197968+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:41.329709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.871488+0000","last_change":"2024-03-06T07:29:04.882788+0000","last_active":"2024-03-06T07:29:05.871488+0000","last_peered":"2024-03-06T07:29:05.871488+0000","last_clean":"2024-03-06T07:29:05.871488+0000","last_became_active":"2024-03-06T07:29:04.881964+0000","last_became_peered":"2024-03-06T07:29:04.881964+0000","last_unstale":"2024-03-06T07:29:05.871488+0000","last_undegraded":"2024-03-06T07:29:05.871488+0000","last_fullsized":"2024-03-06T07:29:05.871488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:50.833591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874788+0000","last_change":"2024-03-06T07:29:04.890469+0000","last_active":"2024-03-06T07:29:05.874788+0000","last_peered":"2024-03-06T07:29:05.874788+0000","last_clean":"2024-03-06T07:29:05.874788+0000","last_became_active":"2024-03-06T07:29:04.890176+0000","last_became_peered":"2024-03-06T07:29:04.890176+0000","last_unstale":"2024-03-06T07:29:05.874788+0000","last_undegraded":"2024-03-06T07:29:05.874788+0000","last_fullsized":"2024-03-06T07:29:05.874788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:28.827720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201913+0000","last_change":"2024-03-06T07:29:04.877245+0000","last_active":"2024-03-06T07:29:06.201913+0000","last_peered":"2024-03-06T07:29:06.201913+0000","last_clean":"2024-03-06T07:29:06.201913+0000","last_became_active":"2024-03-06T07:29:04.875833+0000","last_became_peered":"2024-03-06T07:29:04.875833+0000","last_unstale":"2024-03-06T07:29:06.201913+0000","last_undegraded":"2024-03-06T07:29:06.201913+0000","last_fullsized":"2024-03-06T07:29:06.201913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:32.161449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198923+0000","last_change":"2024-03-06T07:29:04.870836+0000","last_active":"2024-03-06T07:29:06.198923+0000","last_peered":"2024-03-06T07:29:06.198923+0000","last_clean":"2024-03-06T07:29:06.198923+0000","last_became_active":"2024-03-06T07:29:04.870531+0000","last_became_peered":"2024-03-06T07:29:04.870531+0000","last_unstale":"2024-03-06T07:29:06.198923+0000","last_undegraded":"2024-03-06T07:29:06.198923+0000","last_fullsized":"2024-03-06T07:29:06.198923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:39.099541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.196988+0000","last_change":"2024-03-06T07:29:04.873330+0000","last_active":"2024-03-06T07:29:06.196988+0000","last_peered":"2024-03-06T07:29:06.196988+0000","last_clean":"2024-03-06T07:29:06.196988+0000","last_became_active":"2024-03-06T07:29:04.873030+0000","last_became_peered":"2024-03-06T07:29:04.873030+0000","last_unstale":"2024-03-06T07:29:06.196988+0000","last_undegraded":"2024-03-06T07:29:06.196988+0000","last_fullsized":"2024-03-06T07:29:06.196988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:39.555158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867091+0000","last_change":"2024-03-06T07:29:04.870436+0000","last_active":"2024-03-06T07:29:05.867091+0000","last_peered":"2024-03-06T07:29:05.867091+0000","last_clean":"2024-03-06T07:29:05.867091+0000","last_became_active":"2024-03-06T07:29:04.870126+0000","last_became_peered":"2024-03-06T07:29:04.870126+0000","last_unstale":"2024-03-06T07:29:05.867091+0000","last_undegraded":"2024-03-06T07:29:05.867091+0000","last_fullsized":"2024-03-06T07:29:05.867091+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:10.951882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197996+0000","last_change":"2024-03-06T07:29:04.858586+0000","last_active":"2024-03-06T07:29:06.197996+0000","last_peered":"2024-03-06T07:29:06.197996+0000","last_clean":"2024-03-06T07:29:06.197996+0000","last_became_active":"2024-03-06T07:29:04.857982+0000","last_became_peered":"2024-03-06T07:29:04.857982+0000","last_unstale":"2024-03-06T07:29:06.197996+0000","last_undegraded":"2024-03-06T07:29:06.197996+0000","last_fullsized":"2024-03-06T07:29:06.197996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:07.259591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198700+0000","last_change":"2024-03-06T07:29:04.886988+0000","last_active":"2024-03-06T07:29:06.198700+0000","last_peered":"2024-03-06T07:29:06.198700+0000","last_clean":"2024-03-06T07:29:06.198700+0000","last_became_active":"2024-03-06T07:29:04.886486+0000","last_became_peered":"2024-03-06T07:29:04.886486+0000","last_unstale":"2024-03-06T07:29:06.198700+0000","last_undegraded":"2024-03-06T07:29:06.198700+0000","last_fullsized":"2024-03-06T07:29:06.198700+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:39.166593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.202705+0000","last_change":"2024-03-06T07:29:04.875170+0000","last_active":"2024-03-06T07:29:06.202705+0000","last_peered":"2024-03-06T07:29:06.202705+0000","last_clean":"2024-03-06T07:29:06.202705+0000","last_became_active":"2024-03-06T07:29:04.874879+0000","last_became_peered":"2024-03-06T07:29:04.874879+0000","last_unstale":"2024-03-06T07:29:06.202705+0000","last_undegraded":"2024-03-06T07:29:06.202705+0000","last_fullsized":"2024-03-06T07:29:06.202705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:30.478664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867811+0000","last_change":"2024-03-06T07:29:04.880904+0000","last_active":"2024-03-06T07:29:05.867811+0000","last_peered":"2024-03-06T07:29:05.867811+0000","last_clean":"2024-03-06T07:29:05.867811+0000","last_became_active":"2024-03-06T07:29:04.880613+0000","last_became_peered":"2024-03-06T07:29:04.880613+0000","last_unstale":"2024-03-06T07:29:05.867811+0000","last_undegraded":"2024-03-06T07:29:05.867811+0000","last_fullsized":"2024-03-06T07:29:05.867811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:37.927828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867286+0000","last_change":"2024-03-06T07:29:04.884106+0000","last_active":"2024-03-06T07:29:05.867286+0000","last_peered":"2024-03-06T07:29:05.867286+0000","last_clean":"2024-03-06T07:29:05.867286+0000","last_became_active":"2024-03-06T07:29:04.883849+0000","last_became_peered":"2024-03-06T07:29:04.883849+0000","last_unstale":"2024-03-06T07:29:05.867286+0000","last_undegraded":"2024-03-06T07:29:05.867286+0000","last_fullsized":"2024-03-06T07:29:05.867286+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:49.801021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197227+0000","last_change":"2024-03-06T07:29:04.862260+0000","last_active":"2024-03-06T07:29:06.197227+0000","last_peered":"2024-03-06T07:29:06.197227+0000","last_clean":"2024-03-06T07:29:06.197227+0000","last_became_active":"2024-03-06T07:29:04.861983+0000","last_became_peered":"2024-03-06T07:29:04.861983+0000","last_unstale":"2024-03-06T07:29:06.197227+0000","last_undegraded":"2024-03-06T07:29:06.197227+0000","last_fullsized":"2024-03-06T07:29:06.197227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:21.071800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870395+0000","last_change":"2024-03-06T07:29:04.875003+0000","last_active":"2024-03-06T07:29:05.870395+0000","last_peered":"2024-03-06T07:29:05.870395+0000","last_clean":"2024-03-06T07:29:05.870395+0000","last_became_active":"2024-03-06T07:29:04.874164+0000","last_became_peered":"2024-03-06T07:29:04.874164+0000","last_unstale":"2024-03-06T07:29:05.870395+0000","last_undegraded":"2024-03-06T07:29:05.870395+0000","last_fullsized":"2024-03-06T07:29:05.870395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:02.588765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870122+0000","last_change":"2024-03-06T07:29:04.889597+0000","last_active":"2024-03-06T07:29:05.870122+0000","last_peered":"2024-03-06T07:29:05.870122+0000","last_clean":"2024-03-06T07:29:05.870122+0000","last_became_active":"2024-03-06T07:29:04.889239+0000","last_became_peered":"2024-03-06T07:29:04.889239+0000","last_unstale":"2024-03-06T07:29:05.870122+0000","last_undegraded":"2024-03-06T07:29:05.870122+0000","last_fullsized":"2024-03-06T07:29:05.870122+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:39.668562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866012+0000","last_change":"2024-03-06T07:29:04.872684+0000","last_active":"2024-03-06T07:29:05.866012+0000","last_peered":"2024-03-06T07:29:05.866012+0000","last_clean":"2024-03-06T07:29:05.866012+0000","last_became_active":"2024-03-06T07:29:04.872298+0000","last_became_peered":"2024-03-06T07:29:04.872298+0000","last_unstale":"2024-03-06T07:29:05.866012+0000","last_undegraded":"2024-03-06T07:29:05.866012+0000","last_fullsized":"2024-03-06T07:29:05.866012+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:46.317405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200274+0000","last_change":"2024-03-06T07:29:04.857280+0000","last_active":"2024-03-06T07:29:06.200274+0000","last_peered":"2024-03-06T07:29:06.200274+0000","last_clean":"2024-03-06T07:29:06.200274+0000","last_became_active":"2024-03-06T07:29:04.856703+0000","last_became_peered":"2024-03-06T07:29:04.856703+0000","last_unstale":"2024-03-06T07:29:06.200274+0000","last_undegraded":"2024-03-06T07:29:06.200274+0000","last_fullsized":"2024-03-06T07:29:06.200274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:49.801021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875738+0000","last_change":"2024-03-06T07:29:04.876472+0000","last_active":"2024-03-06T07:29:05.875738+0000","last_peered":"2024-03-06T07:29:05.875738+0000","last_clean":"2024-03-06T07:29:05.875738+0000","last_became_active":"2024-03-06T07:29:04.876151+0000","last_became_peered":"2024-03-06T07:29:04.876151+0000","last_unstale":"2024-03-06T07:29:05.875738+0000","last_undegraded":"2024-03-06T07:29:05.875738+0000","last_fullsized":"2024-03-06T07:29:05.875738+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:27.306762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867878+0000","last_change":"2024-03-06T07:29:04.872591+0000","last_active":"2024-03-06T07:29:05.867878+0000","last_peered":"2024-03-06T07:29:05.867878+0000","last_clean":"2024-03-06T07:29:05.867878+0000","last_became_active":"2024-03-06T07:29:04.872305+0000","last_became_peered":"2024-03-06T07:29:04.872305+0000","last_unstale":"2024-03-06T07:29:05.867878+0000","last_undegraded":"2024-03-06T07:29:05.867878+0000","last_fullsized":"2024-03-06T07:29:05.867878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:46.613573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875132+0000","last_change":"2024-03-06T07:29:04.887259+0000","last_active":"2024-03-06T07:29:05.875132+0000","last_peered":"2024-03-06T07:29:05.875132+0000","last_clean":"2024-03-06T07:29:05.875132+0000","last_became_active":"2024-03-06T07:29:04.886964+0000","last_became_peered":"2024-03-06T07:29:04.886964+0000","last_unstale":"2024-03-06T07:29:05.875132+0000","last_undegraded":"2024-03-06T07:29:05.875132+0000","last_fullsized":"2024-03-06T07:29:05.875132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.050440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199011+0000","last_change":"2024-03-06T07:29:04.865474+0000","last_active":"2024-03-06T07:29:06.199011+0000","last_peered":"2024-03-06T07:29:06.199011+0000","last_clean":"2024-03-06T07:29:06.199011+0000","last_became_active":"2024-03-06T07:29:04.865124+0000","last_became_peered":"2024-03-06T07:29:04.865124+0000","last_unstale":"2024-03-06T07:29:06.199011+0000","last_undegraded":"2024-03-06T07:29:06.199011+0000","last_fullsized":"2024-03-06T07:29:06.199011+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:39.166593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870764+0000","last_change":"2024-03-06T07:29:04.883964+0000","last_active":"2024-03-06T07:29:05.870764+0000","last_peered":"2024-03-06T07:29:05.870764+0000","last_clean":"2024-03-06T07:29:05.870764+0000","last_became_active":"2024-03-06T07:29:04.883636+0000","last_became_peered":"2024-03-06T07:29:04.883636+0000","last_unstale":"2024-03-06T07:29:05.870764+0000","last_undegraded":"2024-03-06T07:29:05.870764+0000","last_fullsized":"2024-03-06T07:29:05.870764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:00.120836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199767+0000","last_change":"2024-03-06T07:29:04.875428+0000","last_active":"2024-03-06T07:29:06.199767+0000","last_peered":"2024-03-06T07:29:06.199767+0000","last_clean":"2024-03-06T07:29:06.199767+0000","last_became_active":"2024-03-06T07:29:04.875036+0000","last_became_peered":"2024-03-06T07:29:04.875036+0000","last_unstale":"2024-03-06T07:29:06.199767+0000","last_undegraded":"2024-03-06T07:29:06.199767+0000","last_fullsized":"2024-03-06T07:29:06.199767+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:10.951882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199720+0000","last_change":"2024-03-06T07:29:04.859104+0000","last_active":"2024-03-06T07:29:06.199720+0000","last_peered":"2024-03-06T07:29:06.199720+0000","last_clean":"2024-03-06T07:29:06.199720+0000","last_became_active":"2024-03-06T07:29:04.858444+0000","last_became_peered":"2024-03-06T07:29:04.858444+0000","last_unstale":"2024-03-06T07:29:06.199720+0000","last_undegraded":"2024-03-06T07:29:06.199720+0000","last_fullsized":"2024-03-06T07:29:06.199720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.639921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197795+0000","last_change":"2024-03-06T07:29:04.878191+0000","last_active":"2024-03-06T07:29:06.197795+0000","last_peered":"2024-03-06T07:29:06.197795+0000","last_clean":"2024-03-06T07:29:06.197795+0000","last_became_active":"2024-03-06T07:29:04.877879+0000","last_became_peered":"2024-03-06T07:29:04.877879+0000","last_unstale":"2024-03-06T07:29:06.197795+0000","last_undegraded":"2024-03-06T07:29:06.197795+0000","last_fullsized":"2024-03-06T07:29:06.197795+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:07.259591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866369+0000","last_change":"2024-03-06T07:29:04.882320+0000","last_active":"2024-03-06T07:29:05.866369+0000","last_peered":"2024-03-06T07:29:05.866369+0000","last_clean":"2024-03-06T07:29:05.866369+0000","last_became_active":"2024-03-06T07:29:04.881904+0000","last_became_peered":"2024-03-06T07:29:04.881904+0000","last_unstale":"2024-03-06T07:29:05.866369+0000","last_undegraded":"2024-03-06T07:29:05.866369+0000","last_fullsized":"2024-03-06T07:29:05.866369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:02.588765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201673+0000","last_change":"2024-03-06T07:29:03.873666+0000","last_active":"2024-03-06T07:29:06.201673+0000","last_peered":"2024-03-06T07:29:06.201673+0000","last_clean":"2024-03-06T07:29:06.201673+0000","last_became_active":"2024-03-06T07:29:03.873263+0000","last_became_peered":"2024-03-06T07:29:03.873263+0000","last_unstale":"2024-03-06T07:29:06.201673+0000","last_undegraded":"2024-03-06T07:29:06.201673+0000","last_fullsized":"2024-03-06T07:29:06.201673+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201722+0000","last_change":"2024-03-06T07:29:04.886962+0000","last_active":"2024-03-06T07:29:06.201722+0000","last_peered":"2024-03-06T07:29:06.201722+0000","last_clean":"2024-03-06T07:29:06.201722+0000","last_became_active":"2024-03-06T07:29:04.886690+0000","last_became_peered":"2024-03-06T07:29:04.886690+0000","last_unstale":"2024-03-06T07:29:06.201722+0000","last_undegraded":"2024-03-06T07:29:06.201722+0000","last_fullsized":"2024-03-06T07:29:06.201722+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:39.668562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866400+0000","last_change":"2024-03-06T07:29:03.869646+0000","last_active":"2024-03-06T07:29:05.866400+0000","last_peered":"2024-03-06T07:29:05.866400+0000","last_clean":"2024-03-06T07:29:05.866400+0000","last_became_active":"2024-03-06T07:29:03.869102+0000","last_became_peered":"2024-03-06T07:29:03.869102+0000","last_unstale":"2024-03-06T07:29:05.866400+0000","last_undegraded":"2024-03-06T07:29:05.866400+0000","last_fullsized":"2024-03-06T07:29:05.866400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:07.769778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.871280+0000","last_change":"2024-03-06T07:29:04.887273+0000","last_active":"2024-03-06T07:29:05.871280+0000","last_peered":"2024-03-06T07:29:05.871280+0000","last_clean":"2024-03-06T07:29:05.871280+0000","last_became_active":"2024-03-06T07:29:04.886013+0000","last_became_peered":"2024-03-06T07:29:04.886013+0000","last_unstale":"2024-03-06T07:29:05.871280+0000","last_undegraded":"2024-03-06T07:29:05.871280+0000","last_fullsized":"2024-03-06T07:29:05.871280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:28.827720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867339+0000","last_change":"2024-03-06T07:29:03.859102+0000","last_active":"2024-03-06T07:29:05.867339+0000","last_peered":"2024-03-06T07:29:05.867339+0000","last_clean":"2024-03-06T07:29:05.867339+0000","last_became_active":"2024-03-06T07:29:03.858905+0000","last_became_peered":"2024-03-06T07:29:03.858905+0000","last_unstale":"2024-03-06T07:29:05.867339+0000","last_undegraded":"2024-03-06T07:29:05.867339+0000","last_fullsized":"2024-03-06T07:29:05.867339+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:38.653487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875253+0000","last_change":"2024-03-06T07:29:04.883060+0000","last_active":"2024-03-06T07:29:05.875253+0000","last_peered":"2024-03-06T07:29:05.875253+0000","last_clean":"2024-03-06T07:29:05.875253+0000","last_became_active":"2024-03-06T07:29:04.882761+0000","last_became_peered":"2024-03-06T07:29:04.882761+0000","last_unstale":"2024-03-06T07:29:05.875253+0000","last_undegraded":"2024-03-06T07:29:05.875253+0000","last_fullsized":"2024-03-06T07:29:05.875253+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:59.871873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197179+0000","last_change":"2024-03-06T07:29:03.854977+0000","last_active":"2024-03-06T07:29:06.197179+0000","last_peered":"2024-03-06T07:29:06.197179+0000","last_clean":"2024-03-06T07:29:06.197179+0000","last_became_active":"2024-03-06T07:29:03.854659+0000","last_became_peered":"2024-03-06T07:29:03.854659+0000","last_unstale":"2024-03-06T07:29:06.197179+0000","last_undegraded":"2024-03-06T07:29:06.197179+0000","last_fullsized":"2024-03-06T07:29:06.197179+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:38.875235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201287+0000","last_change":"2024-03-06T07:29:04.878839+0000","last_active":"2024-03-06T07:29:06.201287+0000","last_peered":"2024-03-06T07:29:06.201287+0000","last_clean":"2024-03-06T07:29:06.201287+0000","last_became_active":"2024-03-06T07:29:04.877469+0000","last_became_peered":"2024-03-06T07:29:04.877469+0000","last_unstale":"2024-03-06T07:29:06.201287+0000","last_undegraded":"2024-03-06T07:29:06.201287+0000","last_fullsized":"2024-03-06T07:29:06.201287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:49.801021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870554+0000","last_change":"2024-03-06T07:29:03.865856+0000","last_active":"2024-03-06T07:29:05.870554+0000","last_peered":"2024-03-06T07:29:05.870554+0000","last_clean":"2024-03-06T07:29:05.870554+0000","last_became_active":"2024-03-06T07:29:03.864750+0000","last_became_peered":"2024-03-06T07:29:03.864750+0000","last_unstale":"2024-03-06T07:29:05.870554+0000","last_undegraded":"2024-03-06T07:29:05.870554+0000","last_fullsized":"2024-03-06T07:29:05.870554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:27.937514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199437+0000","last_change":"2024-03-06T07:29:04.873261+0000","last_active":"2024-03-06T07:29:06.199437+0000","last_peered":"2024-03-06T07:29:06.199437+0000","last_clean":"2024-03-06T07:29:06.199437+0000","last_became_active":"2024-03-06T07:29:04.872397+0000","last_became_peered":"2024-03-06T07:29:04.872397+0000","last_unstale":"2024-03-06T07:29:06.199437+0000","last_undegraded":"2024-03-06T07:29:06.199437+0000","last_fullsized":"2024-03-06T07:29:06.199437+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:04.236079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869903+0000","last_change":"2024-03-06T07:29:03.864850+0000","last_active":"2024-03-06T07:29:05.869903+0000","last_peered":"2024-03-06T07:29:05.869903+0000","last_clean":"2024-03-06T07:29:05.869903+0000","last_became_active":"2024-03-06T07:29:03.861142+0000","last_became_peered":"2024-03-06T07:29:03.861142+0000","last_unstale":"2024-03-06T07:29:05.869903+0000","last_undegraded":"2024-03-06T07:29:05.869903+0000","last_fullsized":"2024-03-06T07:29:05.869903+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:32.423149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870091+0000","last_change":"2024-03-06T07:29:04.881202+0000","last_active":"2024-03-06T07:29:05.870091+0000","last_peered":"2024-03-06T07:29:05.870091+0000","last_clean":"2024-03-06T07:29:05.870091+0000","last_became_active":"2024-03-06T07:29:04.880435+0000","last_became_peered":"2024-03-06T07:29:04.880435+0000","last_unstale":"2024-03-06T07:29:05.870091+0000","last_undegraded":"2024-03-06T07:29:05.870091+0000","last_fullsized":"2024-03-06T07:29:05.870091+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:46.317405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866734+0000","last_change":"2024-03-06T07:29:03.858472+0000","last_active":"2024-03-06T07:29:05.866734+0000","last_peered":"2024-03-06T07:29:05.866734+0000","last_clean":"2024-03-06T07:29:05.866734+0000","last_became_active":"2024-03-06T07:29:03.858161+0000","last_became_peered":"2024-03-06T07:29:03.858161+0000","last_unstale":"2024-03-06T07:29:05.866734+0000","last_undegraded":"2024-03-06T07:29:05.866734+0000","last_fullsized":"2024-03-06T07:29:05.866734+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:45.302331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874505+0000","last_change":"2024-03-06T07:29:04.879185+0000","last_active":"2024-03-06T07:29:05.874505+0000","last_peered":"2024-03-06T07:29:05.874505+0000","last_clean":"2024-03-06T07:29:05.874505+0000","last_became_active":"2024-03-06T07:29:04.878856+0000","last_became_peered":"2024-03-06T07:29:04.878856+0000","last_unstale":"2024-03-06T07:29:05.874505+0000","last_undegraded":"2024-03-06T07:29:05.874505+0000","last_fullsized":"2024-03-06T07:29:05.874505+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:00.120836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869691+0000","last_change":"2024-03-06T07:29:03.867364+0000","last_active":"2024-03-06T07:29:05.869691+0000","last_peered":"2024-03-06T07:29:05.869691+0000","last_clean":"2024-03-06T07:29:05.869691+0000","last_became_active":"2024-03-06T07:29:03.866841+0000","last_became_peered":"2024-03-06T07:29:03.866841+0000","last_unstale":"2024-03-06T07:29:05.869691+0000","last_undegraded":"2024-03-06T07:29:05.869691+0000","last_fullsized":"2024-03-06T07:29:05.869691+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:44.258813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200120+0000","last_change":"2024-03-06T07:29:04.887789+0000","last_active":"2024-03-06T07:29:06.200120+0000","last_peered":"2024-03-06T07:29:06.200120+0000","last_clean":"2024-03-06T07:29:06.200120+0000","last_became_active":"2024-03-06T07:29:04.887437+0000","last_became_peered":"2024-03-06T07:29:04.887437+0000","last_unstale":"2024-03-06T07:29:06.200120+0000","last_undegraded":"2024-03-06T07:29:06.200120+0000","last_fullsized":"2024-03-06T07:29:06.200120+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:14.905700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867651+0000","last_change":"2024-03-06T07:29:03.869315+0000","last_active":"2024-03-06T07:29:05.867651+0000","last_peered":"2024-03-06T07:29:05.867651+0000","last_clean":"2024-03-06T07:29:05.867651+0000","last_became_active":"2024-03-06T07:29:03.869070+0000","last_became_peered":"2024-03-06T07:29:03.869070+0000","last_unstale":"2024-03-06T07:29:05.867651+0000","last_undegraded":"2024-03-06T07:29:05.867651+0000","last_fullsized":"2024-03-06T07:29:05.867651+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:56.207229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867603+0000","last_change":"2024-03-06T07:29:04.881850+0000","last_active":"2024-03-06T07:29:05.867603+0000","last_peered":"2024-03-06T07:29:05.867603+0000","last_clean":"2024-03-06T07:29:05.867603+0000","last_became_active":"2024-03-06T07:29:04.881556+0000","last_became_peered":"2024-03-06T07:29:04.881556+0000","last_unstale":"2024-03-06T07:29:05.867603+0000","last_undegraded":"2024-03-06T07:29:05.867603+0000","last_fullsized":"2024-03-06T07:29:05.867603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:32.161449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866500+0000","last_change":"2024-03-06T07:29:03.857566+0000","last_active":"2024-03-06T07:29:05.866500+0000","last_peered":"2024-03-06T07:29:05.866500+0000","last_clean":"2024-03-06T07:29:05.866500+0000","last_became_active":"2024-03-06T07:29:03.856632+0000","last_became_peered":"2024-03-06T07:29:03.856632+0000","last_unstale":"2024-03-06T07:29:05.866500+0000","last_undegraded":"2024-03-06T07:29:05.866500+0000","last_fullsized":"2024-03-06T07:29:05.866500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.478639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867353+0000","last_change":"2024-03-06T07:29:04.869814+0000","last_active":"2024-03-06T07:29:05.867353+0000","last_peered":"2024-03-06T07:29:05.867353+0000","last_clean":"2024-03-06T07:29:05.867353+0000","last_became_active":"2024-03-06T07:29:04.868733+0000","last_became_peered":"2024-03-06T07:29:04.868733+0000","last_unstale":"2024-03-06T07:29:05.867353+0000","last_undegraded":"2024-03-06T07:29:05.867353+0000","last_fullsized":"2024-03-06T07:29:05.867353+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:16.747901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199404+0000","last_change":"2024-03-06T07:29:03.871624+0000","last_active":"2024-03-06T07:29:06.199404+0000","last_peered":"2024-03-06T07:29:06.199404+0000","last_clean":"2024-03-06T07:29:06.199404+0000","last_became_active":"2024-03-06T07:29:03.871249+0000","last_became_peered":"2024-03-06T07:29:03.871249+0000","last_unstale":"2024-03-06T07:29:06.199404+0000","last_undegraded":"2024-03-06T07:29:06.199404+0000","last_fullsized":"2024-03-06T07:29:06.199404+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:38.653487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198847+0000","last_change":"2024-03-06T07:29:04.881197+0000","last_active":"2024-03-06T07:29:06.198847+0000","last_peered":"2024-03-06T07:29:06.198847+0000","last_clean":"2024-03-06T07:29:06.198847+0000","last_became_active":"2024-03-06T07:29:04.880923+0000","last_became_peered":"2024-03-06T07:29:04.880923+0000","last_unstale":"2024-03-06T07:29:06.198847+0000","last_undegraded":"2024-03-06T07:29:06.198847+0000","last_fullsized":"2024-03-06T07:29:06.198847+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:41.329709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867400+0000","last_change":"2024-03-06T07:29:03.860307+0000","last_active":"2024-03-06T07:29:05.867400+0000","last_peered":"2024-03-06T07:29:05.867400+0000","last_clean":"2024-03-06T07:29:05.867400+0000","last_became_active":"2024-03-06T07:29:03.860043+0000","last_became_peered":"2024-03-06T07:29:03.860043+0000","last_unstale":"2024-03-06T07:29:05.867400+0000","last_undegraded":"2024-03-06T07:29:05.867400+0000","last_fullsized":"2024-03-06T07:29:05.867400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:01.573690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874727+0000","last_change":"2024-03-06T07:29:04.875307+0000","last_active":"2024-03-06T07:29:05.874727+0000","last_peered":"2024-03-06T07:29:05.874727+0000","last_clean":"2024-03-06T07:29:05.874727+0000","last_became_active":"2024-03-06T07:29:04.874985+0000","last_became_peered":"2024-03-06T07:29:04.874985+0000","last_unstale":"2024-03-06T07:29:05.874727+0000","last_undegraded":"2024-03-06T07:29:05.874727+0000","last_fullsized":"2024-03-06T07:29:05.874727+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:50.833591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870892+0000","last_change":"2024-03-06T07:29:03.873904+0000","last_active":"2024-03-06T07:29:05.870892+0000","last_peered":"2024-03-06T07:29:05.870892+0000","last_clean":"2024-03-06T07:29:05.870892+0000","last_became_active":"2024-03-06T07:29:03.873433+0000","last_became_peered":"2024-03-06T07:29:03.873433+0000","last_unstale":"2024-03-06T07:29:05.870892+0000","last_undegraded":"2024-03-06T07:29:05.870892+0000","last_fullsized":"2024-03-06T07:29:05.870892+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:38.540083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867151+0000","last_change":"2024-03-06T07:29:04.883358+0000","last_active":"2024-03-06T07:29:05.867151+0000","last_peered":"2024-03-06T07:29:05.867151+0000","last_clean":"2024-03-06T07:29:05.867151+0000","last_became_active":"2024-03-06T07:29:04.882897+0000","last_became_peered":"2024-03-06T07:29:04.882897+0000","last_unstale":"2024-03-06T07:29:05.867151+0000","last_undegraded":"2024-03-06T07:29:05.867151+0000","last_fullsized":"2024-03-06T07:29:05.867151+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:04.236079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874791+0000","last_change":"2024-03-06T07:29:03.863024+0000","last_active":"2024-03-06T07:29:05.874791+0000","last_peered":"2024-03-06T07:29:05.874791+0000","last_clean":"2024-03-06T07:29:05.874791+0000","last_became_active":"2024-03-06T07:29:03.861857+0000","last_became_peered":"2024-03-06T07:29:03.861857+0000","last_unstale":"2024-03-06T07:29:05.874791+0000","last_undegraded":"2024-03-06T07:29:05.874791+0000","last_fullsized":"2024-03-06T07:29:05.874791+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.478639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198292+0000","last_change":"2024-03-06T07:29:04.883037+0000","last_active":"2024-03-06T07:29:06.198292+0000","last_peered":"2024-03-06T07:29:06.198292+0000","last_clean":"2024-03-06T07:29:06.198292+0000","last_became_active":"2024-03-06T07:29:04.882734+0000","last_became_peered":"2024-03-06T07:29:04.882734+0000","last_unstale":"2024-03-06T07:29:06.198292+0000","last_undegraded":"2024-03-06T07:29:06.198292+0000","last_fullsized":"2024-03-06T07:29:06.198292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:39.099541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866693+0000","last_change":"2024-03-06T07:29:03.871751+0000","last_active":"2024-03-06T07:29:05.866693+0000","last_peered":"2024-03-06T07:29:05.866693+0000","last_clean":"2024-03-06T07:29:05.866693+0000","last_became_active":"2024-03-06T07:29:03.871452+0000","last_became_peered":"2024-03-06T07:29:03.871452+0000","last_unstale":"2024-03-06T07:29:05.866693+0000","last_undegraded":"2024-03-06T07:29:05.866693+0000","last_fullsized":"2024-03-06T07:29:05.866693+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:47.139784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200110+0000","last_change":"2024-03-06T07:29:04.876892+0000","last_active":"2024-03-06T07:29:06.200110+0000","last_peered":"2024-03-06T07:29:06.200110+0000","last_clean":"2024-03-06T07:29:06.200110+0000","last_became_active":"2024-03-06T07:29:04.876489+0000","last_became_peered":"2024-03-06T07:29:04.876489+0000","last_unstale":"2024-03-06T07:29:06.200110+0000","last_undegraded":"2024-03-06T07:29:06.200110+0000","last_fullsized":"2024-03-06T07:29:06.200110+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:38.880329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866796+0000","last_change":"2024-03-06T07:29:03.860168+0000","last_active":"2024-03-06T07:29:05.866796+0000","last_peered":"2024-03-06T07:29:05.866796+0000","last_clean":"2024-03-06T07:29:05.866796+0000","last_became_active":"2024-03-06T07:29:03.859890+0000","last_became_peered":"2024-03-06T07:29:03.859890+0000","last_unstale":"2024-03-06T07:29:05.866796+0000","last_undegraded":"2024-03-06T07:29:05.866796+0000","last_fullsized":"2024-03-06T07:29:05.866796+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:37.865255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870480+0000","last_change":"2024-03-06T07:29:04.880998+0000","last_active":"2024-03-06T07:29:05.870480+0000","last_peered":"2024-03-06T07:29:05.870480+0000","last_clean":"2024-03-06T07:29:05.870480+0000","last_became_active":"2024-03-06T07:29:04.880779+0000","last_became_peered":"2024-03-06T07:29:04.880779+0000","last_unstale":"2024-03-06T07:29:05.870480+0000","last_undegraded":"2024-03-06T07:29:05.870480+0000","last_fullsized":"2024-03-06T07:29:05.870480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:57.222304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875634+0000","last_change":"2024-03-06T07:29:03.861596+0000","last_active":"2024-03-06T07:29:05.875634+0000","last_peered":"2024-03-06T07:29:05.875634+0000","last_clean":"2024-03-06T07:29:05.875634+0000","last_became_active":"2024-03-06T07:29:03.861269+0000","last_became_peered":"2024-03-06T07:29:03.861269+0000","last_unstale":"2024-03-06T07:29:05.875634+0000","last_undegraded":"2024-03-06T07:29:05.875634+0000","last_fullsized":"2024-03-06T07:29:05.875634+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:17.253100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200296+0000","last_change":"2024-03-06T07:29:04.878303+0000","last_active":"2024-03-06T07:29:06.200296+0000","last_peered":"2024-03-06T07:29:06.200296+0000","last_clean":"2024-03-06T07:29:06.200296+0000","last_became_active":"2024-03-06T07:29:04.877889+0000","last_became_peered":"2024-03-06T07:29:04.877889+0000","last_unstale":"2024-03-06T07:29:06.200296+0000","last_undegraded":"2024-03-06T07:29:06.200296+0000","last_fullsized":"2024-03-06T07:29:06.200296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:16.747901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870778+0000","last_change":"2024-03-06T07:29:03.870746+0000","last_active":"2024-03-06T07:29:05.870778+0000","last_peered":"2024-03-06T07:29:05.870778+0000","last_clean":"2024-03-06T07:29:05.870778+0000","last_became_active":"2024-03-06T07:29:03.870176+0000","last_became_peered":"2024-03-06T07:29:03.870176+0000","last_unstale":"2024-03-06T07:29:05.870778+0000","last_undegraded":"2024-03-06T07:29:05.870778+0000","last_fullsized":"2024-03-06T07:29:05.870778+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:41.650116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874948+0000","last_change":"2024-03-06T07:29:04.880917+0000","last_active":"2024-03-06T07:29:05.874948+0000","last_peered":"2024-03-06T07:29:05.874948+0000","last_clean":"2024-03-06T07:29:05.874948+0000","last_became_active":"2024-03-06T07:29:04.880401+0000","last_became_peered":"2024-03-06T07:29:04.880401+0000","last_unstale":"2024-03-06T07:29:05.874948+0000","last_undegraded":"2024-03-06T07:29:05.874948+0000","last_fullsized":"2024-03-06T07:29:05.874948+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:55.768199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870416+0000","last_change":"2024-03-06T07:29:03.869114+0000","last_active":"2024-03-06T07:29:05.870416+0000","last_peered":"2024-03-06T07:29:05.870416+0000","last_clean":"2024-03-06T07:29:05.870416+0000","last_became_active":"2024-03-06T07:29:03.868791+0000","last_became_peered":"2024-03-06T07:29:03.868791+0000","last_unstale":"2024-03-06T07:29:05.870416+0000","last_undegraded":"2024-03-06T07:29:05.870416+0000","last_fullsized":"2024-03-06T07:29:05.870416+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197070+0000","last_change":"2024-03-06T07:29:04.868441+0000","last_active":"2024-03-06T07:29:06.197070+0000","last_peered":"2024-03-06T07:29:06.197070+0000","last_clean":"2024-03-06T07:29:06.197070+0000","last_became_active":"2024-03-06T07:29:04.867986+0000","last_became_peered":"2024-03-06T07:29:04.867986+0000","last_unstale":"2024-03-06T07:29:06.197070+0000","last_undegraded":"2024-03-06T07:29:06.197070+0000","last_fullsized":"2024-03-06T07:29:06.197070+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:42.665190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.871226+0000","last_change":"2024-03-06T07:29:03.867613+0000","last_active":"2024-03-06T07:29:05.871226+0000","last_peered":"2024-03-06T07:29:05.871226+0000","last_clean":"2024-03-06T07:29:05.871226+0000","last_became_active":"2024-03-06T07:29:03.867296+0000","last_became_peered":"2024-03-06T07:29:03.867296+0000","last_unstale":"2024-03-06T07:29:05.871226+0000","last_undegraded":"2024-03-06T07:29:05.871226+0000","last_fullsized":"2024-03-06T07:29:05.871226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:15.312779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.196975+0000","last_change":"2024-03-06T07:29:04.874430+0000","last_active":"2024-03-06T07:29:06.196975+0000","last_peered":"2024-03-06T07:29:06.196975+0000","last_clean":"2024-03-06T07:29:06.196975+0000","last_became_active":"2024-03-06T07:29:04.874102+0000","last_became_peered":"2024-03-06T07:29:04.874102+0000","last_unstale":"2024-03-06T07:29:06.196975+0000","last_undegraded":"2024-03-06T07:29:06.196975+0000","last_fullsized":"2024-03-06T07:29:06.196975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:28.952589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.865991+0000","last_change":"2024-03-06T07:29:03.854465+0000","last_active":"2024-03-06T07:29:05.865991+0000","last_peered":"2024-03-06T07:29:05.865991+0000","last_clean":"2024-03-06T07:29:05.865991+0000","last_became_active":"2024-03-06T07:29:03.853828+0000","last_became_peered":"2024-03-06T07:29:03.853828+0000","last_unstale":"2024-03-06T07:29:05.865991+0000","last_undegraded":"2024-03-06T07:29:05.865991+0000","last_fullsized":"2024-03-06T07:29:05.865991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:23.980924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197369+0000","last_change":"2024-03-06T07:29:04.877720+0000","last_active":"2024-03-06T07:29:06.197369+0000","last_peered":"2024-03-06T07:29:06.197369+0000","last_clean":"2024-03-06T07:29:06.197369+0000","last_became_active":"2024-03-06T07:29:04.877371+0000","last_became_peered":"2024-03-06T07:29:04.877371+0000","last_unstale":"2024-03-06T07:29:06.197369+0000","last_undegraded":"2024-03-06T07:29:06.197369+0000","last_fullsized":"2024-03-06T07:29:06.197369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:24.995998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201190+0000","last_change":"2024-03-06T07:29:03.868968+0000","last_active":"2024-03-06T07:29:06.201190+0000","last_peered":"2024-03-06T07:29:06.201190+0000","last_clean":"2024-03-06T07:29:06.201190+0000","last_became_active":"2024-03-06T07:29:03.867933+0000","last_became_peered":"2024-03-06T07:29:03.867933+0000","last_unstale":"2024-03-06T07:29:06.201190+0000","last_undegraded":"2024-03-06T07:29:06.201190+0000","last_fullsized":"2024-03-06T07:29:06.201190+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:41.650116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197224+0000","last_change":"2024-03-06T07:29:04.863142+0000","last_active":"2024-03-06T07:29:06.197224+0000","last_peered":"2024-03-06T07:29:06.197224+0000","last_clean":"2024-03-06T07:29:06.197224+0000","last_became_active":"2024-03-06T07:29:04.862866+0000","last_became_peered":"2024-03-06T07:29:04.862866+0000","last_unstale":"2024-03-06T07:29:06.197224+0000","last_undegraded":"2024-03-06T07:29:06.197224+0000","last_fullsized":"2024-03-06T07:29:06.197224+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:16.327853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869657+0000","last_change":"2024-03-06T07:29:03.864280+0000","last_active":"2024-03-06T07:29:05.869657+0000","last_peered":"2024-03-06T07:29:05.869657+0000","last_clean":"2024-03-06T07:29:05.869657+0000","last_became_active":"2024-03-06T07:29:03.863397+0000","last_became_peered":"2024-03-06T07:29:03.863397+0000","last_unstale":"2024-03-06T07:29:05.869657+0000","last_undegraded":"2024-03-06T07:29:05.869657+0000","last_fullsized":"2024-03-06T07:29:05.869657+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:14.593755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200626+0000","last_change":"2024-03-06T07:29:04.874502+0000","last_active":"2024-03-06T07:29:06.200626+0000","last_peered":"2024-03-06T07:29:06.200626+0000","last_clean":"2024-03-06T07:29:06.200626+0000","last_became_active":"2024-03-06T07:29:04.874240+0000","last_became_peered":"2024-03-06T07:29:04.874240+0000","last_unstale":"2024-03-06T07:29:06.200626+0000","last_undegraded":"2024-03-06T07:29:06.200626+0000","last_fullsized":"2024-03-06T07:29:06.200626+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:14.905700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197415+0000","last_change":"2024-03-06T07:29:03.872538+0000","last_active":"2024-03-06T07:29:06.197415+0000","last_peered":"2024-03-06T07:29:06.197415+0000","last_clean":"2024-03-06T07:29:06.197415+0000","last_became_active":"2024-03-06T07:29:03.871251+0000","last_became_peered":"2024-03-06T07:29:03.871251+0000","last_unstale":"2024-03-06T07:29:06.197415+0000","last_undegraded":"2024-03-06T07:29:06.197415+0000","last_fullsized":"2024-03-06T07:29:06.197415+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:45.302331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197256+0000","last_change":"2024-03-06T07:29:04.863561+0000","last_active":"2024-03-06T07:29:06.197256+0000","last_peered":"2024-03-06T07:29:06.197256+0000","last_clean":"2024-03-06T07:29:06.197256+0000","last_became_active":"2024-03-06T07:29:04.863282+0000","last_became_peered":"2024-03-06T07:29:04.863282+0000","last_unstale":"2024-03-06T07:29:06.197256+0000","last_undegraded":"2024-03-06T07:29:06.197256+0000","last_fullsized":"2024-03-06T07:29:06.197256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:44.493714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875760+0000","last_change":"2024-03-06T07:29:03.858375+0000","last_active":"2024-03-06T07:29:05.875760+0000","last_peered":"2024-03-06T07:29:05.875760+0000","last_clean":"2024-03-06T07:29:05.875760+0000","last_became_active":"2024-03-06T07:29:03.858097+0000","last_became_peered":"2024-03-06T07:29:03.858097+0000","last_unstale":"2024-03-06T07:29:05.875760+0000","last_undegraded":"2024-03-06T07:29:05.875760+0000","last_fullsized":"2024-03-06T07:29:05.875760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:07.769778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200569+0000","last_change":"2024-03-06T07:29:04.873219+0000","last_active":"2024-03-06T07:29:06.200569+0000","last_peered":"2024-03-06T07:29:06.200569+0000","last_clean":"2024-03-06T07:29:06.200569+0000","last_became_active":"2024-03-06T07:29:04.872693+0000","last_became_peered":"2024-03-06T07:29:04.872693+0000","last_unstale":"2024-03-06T07:29:06.200569+0000","last_undegraded":"2024-03-06T07:29:06.200569+0000","last_fullsized":"2024-03-06T07:29:06.200569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:32.161449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866475+0000","last_change":"2024-03-06T07:29:03.853739+0000","last_active":"2024-03-06T07:29:05.866475+0000","last_peered":"2024-03-06T07:29:05.866475+0000","last_clean":"2024-03-06T07:29:05.866475+0000","last_became_active":"2024-03-06T07:29:03.853042+0000","last_became_peered":"2024-03-06T07:29:03.853042+0000","last_unstale":"2024-03-06T07:29:05.866475+0000","last_undegraded":"2024-03-06T07:29:05.866475+0000","last_fullsized":"2024-03-06T07:29:05.866475+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866205+0000","last_change":"2024-03-06T07:29:04.876775+0000","last_active":"2024-03-06T07:29:05.866205+0000","last_peered":"2024-03-06T07:29:05.866205+0000","last_clean":"2024-03-06T07:29:05.866205+0000","last_became_active":"2024-03-06T07:29:04.875987+0000","last_became_peered":"2024-03-06T07:29:04.875987+0000","last_unstale":"2024-03-06T07:29:05.866205+0000","last_undegraded":"2024-03-06T07:29:05.866205+0000","last_fullsized":"2024-03-06T07:29:05.866205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:39.668562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875019+0000","last_change":"2024-03-06T07:29:03.855200+0000","last_active":"2024-03-06T07:29:05.875019+0000","last_peered":"2024-03-06T07:29:05.875019+0000","last_clean":"2024-03-06T07:29:05.875019+0000","last_became_active":"2024-03-06T07:29:03.854863+0000","last_became_peered":"2024-03-06T07:29:03.854863+0000","last_unstale":"2024-03-06T07:29:05.875019+0000","last_undegraded":"2024-03-06T07:29:05.875019+0000","last_fullsized":"2024-03-06T07:29:05.875019+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201222+0000","last_change":"2024-03-06T07:29:04.867972+0000","last_active":"2024-03-06T07:29:06.201222+0000","last_peered":"2024-03-06T07:29:06.201222+0000","last_clean":"2024-03-06T07:29:06.201222+0000","last_became_active":"2024-03-06T07:29:04.866461+0000","last_became_peered":"2024-03-06T07:29:04.866461+0000","last_unstale":"2024-03-06T07:29:06.201222+0000","last_undegraded":"2024-03-06T07:29:06.201222+0000","last_fullsized":"2024-03-06T07:29:06.201222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:48.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866167+0000","last_change":"2024-03-06T07:29:03.860203+0000","last_active":"2024-03-06T07:29:05.866167+0000","last_peered":"2024-03-06T07:29:05.866167+0000","last_clean":"2024-03-06T07:29:05.866167+0000","last_became_active":"2024-03-06T07:29:03.860001+0000","last_became_peered":"2024-03-06T07:29:03.860001+0000","last_unstale":"2024-03-06T07:29:05.866167+0000","last_undegraded":"2024-03-06T07:29:05.866167+0000","last_fullsized":"2024-03-06T07:29:05.866167+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:17.253100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.865919+0000","last_change":"2024-03-06T07:29:04.874410+0000","last_active":"2024-03-06T07:29:05.865919+0000","last_peered":"2024-03-06T07:29:05.865919+0000","last_clean":"2024-03-06T07:29:05.865919+0000","last_became_active":"2024-03-06T07:29:04.873804+0000","last_became_peered":"2024-03-06T07:29:04.873804+0000","last_unstale":"2024-03-06T07:29:05.865919+0000","last_undegraded":"2024-03-06T07:29:05.865919+0000","last_fullsized":"2024-03-06T07:29:05.865919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:48.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874901+0000","last_change":"2024-03-06T07:29:03.864706+0000","last_active":"2024-03-06T07:29:05.874901+0000","last_peered":"2024-03-06T07:29:05.874901+0000","last_clean":"2024-03-06T07:29:05.874901+0000","last_became_active":"2024-03-06T07:29:03.864011+0000","last_became_peered":"2024-03-06T07:29:03.864011+0000","last_unstale":"2024-03-06T07:29:05.874901+0000","last_undegraded":"2024-03-06T07:29:05.874901+0000","last_fullsized":"2024-03-06T07:29:05.874901+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:23.980924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866900+0000","last_change":"2024-03-06T07:29:04.878373+0000","last_active":"2024-03-06T07:29:05.866900+0000","last_peered":"2024-03-06T07:29:05.866900+0000","last_clean":"2024-03-06T07:29:05.866900+0000","last_became_active":"2024-03-06T07:29:04.878111+0000","last_became_peered":"2024-03-06T07:29:04.878111+0000","last_unstale":"2024-03-06T07:29:05.866900+0000","last_undegraded":"2024-03-06T07:29:05.866900+0000","last_fullsized":"2024-03-06T07:29:05.866900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:39.166593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869897+0000","last_change":"2024-03-06T07:29:03.867397+0000","last_active":"2024-03-06T07:29:05.869897+0000","last_peered":"2024-03-06T07:29:05.869897+0000","last_clean":"2024-03-06T07:29:05.869897+0000","last_became_active":"2024-03-06T07:29:03.867116+0000","last_became_peered":"2024-03-06T07:29:03.867116+0000","last_unstale":"2024-03-06T07:29:05.869897+0000","last_undegraded":"2024-03-06T07:29:05.869897+0000","last_fullsized":"2024-03-06T07:29:05.869897+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:34.707979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869740+0000","last_change":"2024-03-06T07:29:04.886296+0000","last_active":"2024-03-06T07:29:05.869740+0000","last_peered":"2024-03-06T07:29:05.869740+0000","last_clean":"2024-03-06T07:29:05.869740+0000","last_became_active":"2024-03-06T07:29:04.886005+0000","last_became_peered":"2024-03-06T07:29:04.886005+0000","last_unstale":"2024-03-06T07:29:05.869740+0000","last_undegraded":"2024-03-06T07:29:05.869740+0000","last_fullsized":"2024-03-06T07:29:05.869740+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:48.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866642+0000","last_change":"2024-03-06T07:29:03.871145+0000","last_active":"2024-03-06T07:29:05.866642+0000","last_peered":"2024-03-06T07:29:05.866642+0000","last_clean":"2024-03-06T07:29:05.866642+0000","last_became_active":"2024-03-06T07:29:03.870797+0000","last_became_peered":"2024-03-06T07:29:03.870797+0000","last_unstale":"2024-03-06T07:29:05.866642+0000","last_undegraded":"2024-03-06T07:29:05.866642+0000","last_fullsized":"2024-03-06T07:29:05.866642+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.624847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201836+0000","last_change":"2024-03-06T07:29:04.883862+0000","last_active":"2024-03-06T07:29:06.201836+0000","last_peered":"2024-03-06T07:29:06.201836+0000","last_clean":"2024-03-06T07:29:06.201836+0000","last_became_active":"2024-03-06T07:29:04.883554+0000","last_became_peered":"2024-03-06T07:29:04.883554+0000","last_unstale":"2024-03-06T07:29:06.201836+0000","last_undegraded":"2024-03-06T07:29:06.201836+0000","last_fullsized":"2024-03-06T07:29:06.201836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:16.747901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200922+0000","last_change":"2024-03-06T07:29:03.872234+0000","last_active":"2024-03-06T07:29:06.200922+0000","last_peered":"2024-03-06T07:29:06.200922+0000","last_clean":"2024-03-06T07:29:06.200922+0000","last_became_active":"2024-03-06T07:29:03.870531+0000","last_became_peered":"2024-03-06T07:29:03.870531+0000","last_unstale":"2024-03-06T07:29:06.200922+0000","last_undegraded":"2024-03-06T07:29:06.200922+0000","last_fullsized":"2024-03-06T07:29:06.200922+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:15.312779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":193,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198848+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:29:06.198848+0000","last_peered":"2024-03-06T07:29:06.198848+0000","last_clean":"2024-03-06T07:29:06.198848+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:29:06.198848+0000","last_undegraded":"2024-03-06T07:29:06.198848+0000","last_fullsized":"2024-03-06T07:29:06.198848+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":146848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":737146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:09.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:09.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.535+0000 7f16f3e00640 1 Processor -- start 2024-03-06T07:29:09.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.536+0000 7f16f3e00640 1 -- start start 2024-03-06T07:29:09.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.537+0000 7f16f3e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16ec144c00 0x7f16ec06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.537+0000 7f16f3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16ec15f9d0 0x7f16ec15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.537+0000 7f16f3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec0ce0a0 0x7f16ec0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.537+0000 7f16f3e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec06ee70 con 0x7f16ec15f9d0 2024-03-06T07:29:09.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.537+0000 7f16f3e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec06cef0 con 0x7f16ec144c00 2024-03-06T07:29:09.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.537+0000 7f16f3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec06d1d0 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16ec15f9d0 0x7f16ec15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16ec15f9d0 0x7f16ec15fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40154/0 (socket says 172.21.15.114:40154) 2024-03-06T07:29:09.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 -- 172.21.15.114:0/3577278517 learned_addr learned my addr 172.21.15.114:0/3577278517 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f1e00640 1 --2- 172.21.15.114:0/3577278517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec0ce0a0 0x7f16ec0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f1400640 1 --2- 172.21.15.114:0/3577278517 >> v2:172.21.15.177:3300/0 conn(0x7f16ec144c00 0x7f16ec06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 -- 172.21.15.114:0/3577278517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec0ce0a0 msgr2=0x7f16ec0ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 --2- 172.21.15.114:0/3577278517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec0ce0a0 0x7f16ec0ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 -- 172.21.15.114:0/3577278517 >> v2:172.21.15.177:3300/0 conn(0x7f16ec144c00 msgr2=0x7f16ec06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 --2- 172.21.15.114:0/3577278517 >> v2:172.21.15.177:3300/0 conn(0x7f16ec144c00 0x7f16ec06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.538+0000 7f16f0a00640 1 -- 172.21.15.114:0/3577278517 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec06d4b0 con 0x7f16ec15f9d0 2024-03-06T07:29:09.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.539+0000 7f16f0a00640 1 --2- 172.21.15.114:0/3577278517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec15f9d0 0x7f16ec15fe60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0b25f0 tx=0x7f16d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca39cf6209d87840 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16ebe00640 1 -- 172.21.15.114:0/3577278517 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d8035020 con 0x7f16ec15f9d0 2024-03-06T07:29:09.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16ebe00640 1 -- 172.21.15.114:0/3577278517 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d8031870 con 0x7f16ec15f9d0 2024-03-06T07:29:09.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16ebe00640 1 -- 172.21.15.114:0/3577278517 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d8031c40 con 0x7f16ec15f9d0 2024-03-06T07:29:09.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 -- 172.21.15.114:0/3577278517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec15f9d0 msgr2=0x7f16ec15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 --2- 172.21.15.114:0/3577278517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec15f9d0 0x7f16ec15fe60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0b25f0 tx=0x7f16d802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 -- 172.21.15.114:0/3577278517 shutdown_connections 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 --2- 172.21.15.114:0/3577278517 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec0ce0a0 0x7f16ec0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 --2- 172.21.15.114:0/3577278517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec15f9d0 0x7f16ec15fe60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 --2- 172.21.15.114:0/3577278517 >> v2:172.21.15.177:3300/0 conn(0x7f16ec144c00 0x7f16ec06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 -- 172.21.15.114:0/3577278517 >> 172.21.15.114:0/3577278517 conn(0x7f16ec0b1730 msgr2=0x7f16ec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 -- 172.21.15.114:0/3577278517 shutdown_connections 2024-03-06T07:29:09.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.540+0000 7f16f3e00640 1 -- 172.21.15.114:0/3577278517 wait complete. 2024-03-06T07:29:09.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.542+0000 7f16f3e00640 1 Processor -- start 2024-03-06T07:29:09.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.542+0000 7f16f3e00640 1 -- start start 2024-03-06T07:29:09.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.543+0000 7f16f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec0ce0a0 0x7f16ec1e8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.543+0000 7f16f3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16ec144c00 0x7f16ec15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.543+0000 7f16f3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec15f9d0 0x7f16ec15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.543+0000 7f16f3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec06cb50 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.543+0000 7f16f3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec160730 con 0x7f16ec144c00 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.543+0000 7f16f3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec06ccd0 con 0x7f16ec15f9d0 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.543+0000 7f16f1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec0ce0a0 0x7f16ec1e8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec0ce0a0 0x7f16ec1e8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40162/0 (socket says 172.21.15.114:40162) 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 -- 172.21.15.114:0/4000981995 learned_addr learned my addr 172.21.15.114:0/4000981995 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1e00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec15f9d0 0x7f16ec15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f0a00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16ec144c00 0x7f16ec15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 -- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec15f9d0 msgr2=0x7f16ec15b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec15f9d0 0x7f16ec15b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 -- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16ec144c00 msgr2=0x7f16ec15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16ec144c00 0x7f16ec15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 -- 172.21.15.114:0/4000981995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec1e7c30 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f0a00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16ec144c00 0x7f16ec15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.544+0000 7f16f1400640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec0ce0a0 0x7f16ec1e8e70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f16e000d800 tx=0x7f16e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.545+0000 7f16e9600640 1 -- 172.21.15.114:0/4000981995 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000b6c0 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.545+0000 7f16e9600640 1 -- 172.21.15.114:0/4000981995 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e0019070 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.545+0000 7f16e9600640 1 -- 172.21.15.114:0/4000981995 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000ba80 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.545+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec1e7f10 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.545+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec1e84d0 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.546+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec04f330 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.547+0000 7f16e9600640 1 -- 172.21.15.114:0/4000981995 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16e000bc20 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.548+0000 7f16e9600640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f16cc06b500 0x7f16cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.548+0000 7f16e9600640 1 -- 172.21.15.114:0/4000981995 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16e008a760 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.548+0000 7f16f0a00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f16cc06b500 0x7f16cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.549+0000 7f16f0a00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f16cc06b500 0x7f16cc06d9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f16d8036000 tx=0x7f16d8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.552+0000 7f16e9600640 1 -- 172.21.15.114:0/4000981995 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16e001c360 con 0x7f16ec0ce0a0 2024-03-06T07:29:09.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.823+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16ec06c780 con 0x7f16cc06b500 2024-03-06T07:29:09.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.839+0000 7f16e9600640 1 -- 172.21.15.114:0/4000981995 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251951 (secure 0 0 0) 0x7f16ec06c780 con 0x7f16cc06b500 2024-03-06T07:29:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:09.845 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f16cc06b500 msgr2=0x7f16cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f16cc06b500 0x7f16cc06d9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f16d8036000 tx=0x7f16d8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec0ce0a0 msgr2=0x7f16ec1e8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec0ce0a0 0x7f16ec1e8e70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f16e000d800 tx=0x7f16e000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 shutdown_connections 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16ec15f9d0 0x7f16ec15b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16ec144c00 0x7f16ec15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f16cc06b500 0x7f16cc06d9c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 --2- 172.21.15.114:0/4000981995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16ec0ce0a0 0x7f16ec1e8e70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 >> 172.21.15.114:0/4000981995 conn(0x7f16ec0b1730 msgr2=0x7f16ec0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 shutdown_connections 2024-03-06T07:29:09.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:09.845+0000 7f16f3e00640 1 -- 172.21.15.114:0/4000981995 wait complete. 2024-03-06T07:29:09.873 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-03-06T07:29:09.120412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":182,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220328,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754392,"statfs":{"total":773094113280,"available":772868497408,"internally_reserved":0,"allocated":4235264,"data_stored":2359780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001197"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201506+0000","last_change":"2024-03-06T07:29:04.868489+0000","last_active":"2024-03-06T07:29:06.201506+0000","last_peered":"2024-03-06T07:29:06.201506+0000","last_clean":"2024-03-06T07:29:06.201506+0000","last_became_active":"2024-03-06T07:29:04.868206+0000","last_became_peered":"2024-03-06T07:29:04.868206+0000","last_unstale":"2024-03-06T07:29:06.201506+0000","last_undegraded":"2024-03-06T07:29:06.201506+0000","last_fullsized":"2024-03-06T07:29:06.201506+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:46.317405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867783+0000","last_change":"2024-03-06T07:29:04.882576+0000","last_active":"2024-03-06T07:29:05.867783+0000","last_peered":"2024-03-06T07:29:05.867783+0000","last_clean":"2024-03-06T07:29:05.867783+0000","last_became_active":"2024-03-06T07:29:04.882298+0000","last_became_peered":"2024-03-06T07:29:04.882298+0000","last_unstale":"2024-03-06T07:29:05.867783+0000","last_undegraded":"2024-03-06T07:29:05.867783+0000","last_fullsized":"2024-03-06T07:29:05.867783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:30.478664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199255+0000","last_change":"2024-03-06T07:29:04.886479+0000","last_active":"2024-03-06T07:29:06.199255+0000","last_peered":"2024-03-06T07:29:06.199255+0000","last_clean":"2024-03-06T07:29:06.199255+0000","last_became_active":"2024-03-06T07:29:04.886139+0000","last_became_peered":"2024-03-06T07:29:04.886139+0000","last_unstale":"2024-03-06T07:29:06.199255+0000","last_undegraded":"2024-03-06T07:29:06.199255+0000","last_fullsized":"2024-03-06T07:29:06.199255+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:10.951882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874887+0000","last_change":"2024-03-06T07:29:04.886643+0000","last_active":"2024-03-06T07:29:05.874887+0000","last_peered":"2024-03-06T07:29:05.874887+0000","last_clean":"2024-03-06T07:29:05.874887+0000","last_became_active":"2024-03-06T07:29:04.886371+0000","last_became_peered":"2024-03-06T07:29:04.886371+0000","last_unstale":"2024-03-06T07:29:05.874887+0000","last_undegraded":"2024-03-06T07:29:05.874887+0000","last_fullsized":"2024-03-06T07:29:05.874887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:36.195095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.196865+0000","last_change":"2024-03-06T07:29:04.873221+0000","last_active":"2024-03-06T07:29:06.196865+0000","last_peered":"2024-03-06T07:29:06.196865+0000","last_clean":"2024-03-06T07:29:06.196865+0000","last_became_active":"2024-03-06T07:29:04.872941+0000","last_became_peered":"2024-03-06T07:29:04.872941+0000","last_unstale":"2024-03-06T07:29:06.196865+0000","last_undegraded":"2024-03-06T07:29:06.196865+0000","last_fullsized":"2024-03-06T07:29:06.196865+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:33.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867244+0000","last_change":"2024-03-06T07:29:04.874030+0000","last_active":"2024-03-06T07:29:05.867244+0000","last_peered":"2024-03-06T07:29:05.867244+0000","last_clean":"2024-03-06T07:29:05.867244+0000","last_became_active":"2024-03-06T07:29:04.873748+0000","last_became_peered":"2024-03-06T07:29:04.873748+0000","last_unstale":"2024-03-06T07:29:05.867244+0000","last_undegraded":"2024-03-06T07:29:05.867244+0000","last_fullsized":"2024-03-06T07:29:05.867244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:14.905700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200103+0000","last_change":"2024-03-06T07:29:04.876229+0000","last_active":"2024-03-06T07:29:06.200103+0000","last_peered":"2024-03-06T07:29:06.200103+0000","last_clean":"2024-03-06T07:29:06.200103+0000","last_became_active":"2024-03-06T07:29:04.875681+0000","last_became_peered":"2024-03-06T07:29:04.875681+0000","last_unstale":"2024-03-06T07:29:06.200103+0000","last_undegraded":"2024-03-06T07:29:06.200103+0000","last_fullsized":"2024-03-06T07:29:06.200103+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:04.236079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197968+0000","last_change":"2024-03-06T07:29:04.879283+0000","last_active":"2024-03-06T07:29:06.197968+0000","last_peered":"2024-03-06T07:29:06.197968+0000","last_clean":"2024-03-06T07:29:06.197968+0000","last_became_active":"2024-03-06T07:29:04.878997+0000","last_became_peered":"2024-03-06T07:29:04.878997+0000","last_unstale":"2024-03-06T07:29:06.197968+0000","last_undegraded":"2024-03-06T07:29:06.197968+0000","last_fullsized":"2024-03-06T07:29:06.197968+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:41.329709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.871488+0000","last_change":"2024-03-06T07:29:04.882788+0000","last_active":"2024-03-06T07:29:05.871488+0000","last_peered":"2024-03-06T07:29:05.871488+0000","last_clean":"2024-03-06T07:29:05.871488+0000","last_became_active":"2024-03-06T07:29:04.881964+0000","last_became_peered":"2024-03-06T07:29:04.881964+0000","last_unstale":"2024-03-06T07:29:05.871488+0000","last_undegraded":"2024-03-06T07:29:05.871488+0000","last_fullsized":"2024-03-06T07:29:05.871488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:50.833591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874788+0000","last_change":"2024-03-06T07:29:04.890469+0000","last_active":"2024-03-06T07:29:05.874788+0000","last_peered":"2024-03-06T07:29:05.874788+0000","last_clean":"2024-03-06T07:29:05.874788+0000","last_became_active":"2024-03-06T07:29:04.890176+0000","last_became_peered":"2024-03-06T07:29:04.890176+0000","last_unstale":"2024-03-06T07:29:05.874788+0000","last_undegraded":"2024-03-06T07:29:05.874788+0000","last_fullsized":"2024-03-06T07:29:05.874788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:28.827720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201913+0000","last_change":"2024-03-06T07:29:04.877245+0000","last_active":"2024-03-06T07:29:06.201913+0000","last_peered":"2024-03-06T07:29:06.201913+0000","last_clean":"2024-03-06T07:29:06.201913+0000","last_became_active":"2024-03-06T07:29:04.875833+0000","last_became_peered":"2024-03-06T07:29:04.875833+0000","last_unstale":"2024-03-06T07:29:06.201913+0000","last_undegraded":"2024-03-06T07:29:06.201913+0000","last_fullsized":"2024-03-06T07:29:06.201913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:32.161449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198923+0000","last_change":"2024-03-06T07:29:04.870836+0000","last_active":"2024-03-06T07:29:06.198923+0000","last_peered":"2024-03-06T07:29:06.198923+0000","last_clean":"2024-03-06T07:29:06.198923+0000","last_became_active":"2024-03-06T07:29:04.870531+0000","last_became_peered":"2024-03-06T07:29:04.870531+0000","last_unstale":"2024-03-06T07:29:06.198923+0000","last_undegraded":"2024-03-06T07:29:06.198923+0000","last_fullsized":"2024-03-06T07:29:06.198923+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:39.099541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.196988+0000","last_change":"2024-03-06T07:29:04.873330+0000","last_active":"2024-03-06T07:29:06.196988+0000","last_peered":"2024-03-06T07:29:06.196988+0000","last_clean":"2024-03-06T07:29:06.196988+0000","last_became_active":"2024-03-06T07:29:04.873030+0000","last_became_peered":"2024-03-06T07:29:04.873030+0000","last_unstale":"2024-03-06T07:29:06.196988+0000","last_undegraded":"2024-03-06T07:29:06.196988+0000","last_fullsized":"2024-03-06T07:29:06.196988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:39.555158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867091+0000","last_change":"2024-03-06T07:29:04.870436+0000","last_active":"2024-03-06T07:29:05.867091+0000","last_peered":"2024-03-06T07:29:05.867091+0000","last_clean":"2024-03-06T07:29:05.867091+0000","last_became_active":"2024-03-06T07:29:04.870126+0000","last_became_peered":"2024-03-06T07:29:04.870126+0000","last_unstale":"2024-03-06T07:29:05.867091+0000","last_undegraded":"2024-03-06T07:29:05.867091+0000","last_fullsized":"2024-03-06T07:29:05.867091+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:10.951882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197996+0000","last_change":"2024-03-06T07:29:04.858586+0000","last_active":"2024-03-06T07:29:06.197996+0000","last_peered":"2024-03-06T07:29:06.197996+0000","last_clean":"2024-03-06T07:29:06.197996+0000","last_became_active":"2024-03-06T07:29:04.857982+0000","last_became_peered":"2024-03-06T07:29:04.857982+0000","last_unstale":"2024-03-06T07:29:06.197996+0000","last_undegraded":"2024-03-06T07:29:06.197996+0000","last_fullsized":"2024-03-06T07:29:06.197996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:07.259591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198700+0000","last_change":"2024-03-06T07:29:04.886988+0000","last_active":"2024-03-06T07:29:06.198700+0000","last_peered":"2024-03-06T07:29:06.198700+0000","last_clean":"2024-03-06T07:29:06.198700+0000","last_became_active":"2024-03-06T07:29:04.886486+0000","last_became_peered":"2024-03-06T07:29:04.886486+0000","last_unstale":"2024-03-06T07:29:06.198700+0000","last_undegraded":"2024-03-06T07:29:06.198700+0000","last_fullsized":"2024-03-06T07:29:06.198700+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:39.166593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.202705+0000","last_change":"2024-03-06T07:29:04.875170+0000","last_active":"2024-03-06T07:29:06.202705+0000","last_peered":"2024-03-06T07:29:06.202705+0000","last_clean":"2024-03-06T07:29:06.202705+0000","last_became_active":"2024-03-06T07:29:04.874879+0000","last_became_peered":"2024-03-06T07:29:04.874879+0000","last_unstale":"2024-03-06T07:29:06.202705+0000","last_undegraded":"2024-03-06T07:29:06.202705+0000","last_fullsized":"2024-03-06T07:29:06.202705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:30.478664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867811+0000","last_change":"2024-03-06T07:29:04.880904+0000","last_active":"2024-03-06T07:29:05.867811+0000","last_peered":"2024-03-06T07:29:05.867811+0000","last_clean":"2024-03-06T07:29:05.867811+0000","last_became_active":"2024-03-06T07:29:04.880613+0000","last_became_peered":"2024-03-06T07:29:04.880613+0000","last_unstale":"2024-03-06T07:29:05.867811+0000","last_undegraded":"2024-03-06T07:29:05.867811+0000","last_fullsized":"2024-03-06T07:29:05.867811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:37.927828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867286+0000","last_change":"2024-03-06T07:29:04.884106+0000","last_active":"2024-03-06T07:29:05.867286+0000","last_peered":"2024-03-06T07:29:05.867286+0000","last_clean":"2024-03-06T07:29:05.867286+0000","last_became_active":"2024-03-06T07:29:04.883849+0000","last_became_peered":"2024-03-06T07:29:04.883849+0000","last_unstale":"2024-03-06T07:29:05.867286+0000","last_undegraded":"2024-03-06T07:29:05.867286+0000","last_fullsized":"2024-03-06T07:29:05.867286+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:49.801021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197227+0000","last_change":"2024-03-06T07:29:04.862260+0000","last_active":"2024-03-06T07:29:06.197227+0000","last_peered":"2024-03-06T07:29:06.197227+0000","last_clean":"2024-03-06T07:29:06.197227+0000","last_became_active":"2024-03-06T07:29:04.861983+0000","last_became_peered":"2024-03-06T07:29:04.861983+0000","last_unstale":"2024-03-06T07:29:06.197227+0000","last_undegraded":"2024-03-06T07:29:06.197227+0000","last_fullsized":"2024-03-06T07:29:06.197227+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:21.071800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870395+0000","last_change":"2024-03-06T07:29:04.875003+0000","last_active":"2024-03-06T07:29:05.870395+0000","last_peered":"2024-03-06T07:29:05.870395+0000","last_clean":"2024-03-06T07:29:05.870395+0000","last_became_active":"2024-03-06T07:29:04.874164+0000","last_became_peered":"2024-03-06T07:29:04.874164+0000","last_unstale":"2024-03-06T07:29:05.870395+0000","last_undegraded":"2024-03-06T07:29:05.870395+0000","last_fullsized":"2024-03-06T07:29:05.870395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:02.588765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870122+0000","last_change":"2024-03-06T07:29:04.889597+0000","last_active":"2024-03-06T07:29:05.870122+0000","last_peered":"2024-03-06T07:29:05.870122+0000","last_clean":"2024-03-06T07:29:05.870122+0000","last_became_active":"2024-03-06T07:29:04.889239+0000","last_became_peered":"2024-03-06T07:29:04.889239+0000","last_unstale":"2024-03-06T07:29:05.870122+0000","last_undegraded":"2024-03-06T07:29:05.870122+0000","last_fullsized":"2024-03-06T07:29:05.870122+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:39.668562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866012+0000","last_change":"2024-03-06T07:29:04.872684+0000","last_active":"2024-03-06T07:29:05.866012+0000","last_peered":"2024-03-06T07:29:05.866012+0000","last_clean":"2024-03-06T07:29:05.866012+0000","last_became_active":"2024-03-06T07:29:04.872298+0000","last_became_peered":"2024-03-06T07:29:04.872298+0000","last_unstale":"2024-03-06T07:29:05.866012+0000","last_undegraded":"2024-03-06T07:29:05.866012+0000","last_fullsized":"2024-03-06T07:29:05.866012+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:46.317405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200274+0000","last_change":"2024-03-06T07:29:04.857280+0000","last_active":"2024-03-06T07:29:06.200274+0000","last_peered":"2024-03-06T07:29:06.200274+0000","last_clean":"2024-03-06T07:29:06.200274+0000","last_became_active":"2024-03-06T07:29:04.856703+0000","last_became_peered":"2024-03-06T07:29:04.856703+0000","last_unstale":"2024-03-06T07:29:06.200274+0000","last_undegraded":"2024-03-06T07:29:06.200274+0000","last_fullsized":"2024-03-06T07:29:06.200274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:49.801021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875738+0000","last_change":"2024-03-06T07:29:04.876472+0000","last_active":"2024-03-06T07:29:05.875738+0000","last_peered":"2024-03-06T07:29:05.875738+0000","last_clean":"2024-03-06T07:29:05.875738+0000","last_became_active":"2024-03-06T07:29:04.876151+0000","last_became_peered":"2024-03-06T07:29:04.876151+0000","last_unstale":"2024-03-06T07:29:05.875738+0000","last_undegraded":"2024-03-06T07:29:05.875738+0000","last_fullsized":"2024-03-06T07:29:05.875738+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:27.306762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867878+0000","last_change":"2024-03-06T07:29:04.872591+0000","last_active":"2024-03-06T07:29:05.867878+0000","last_peered":"2024-03-06T07:29:05.867878+0000","last_clean":"2024-03-06T07:29:05.867878+0000","last_became_active":"2024-03-06T07:29:04.872305+0000","last_became_peered":"2024-03-06T07:29:04.872305+0000","last_unstale":"2024-03-06T07:29:05.867878+0000","last_undegraded":"2024-03-06T07:29:05.867878+0000","last_fullsized":"2024-03-06T07:29:05.867878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:46.613573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875132+0000","last_change":"2024-03-06T07:29:04.887259+0000","last_active":"2024-03-06T07:29:05.875132+0000","last_peered":"2024-03-06T07:29:05.875132+0000","last_clean":"2024-03-06T07:29:05.875132+0000","last_became_active":"2024-03-06T07:29:04.886964+0000","last_became_peered":"2024-03-06T07:29:04.886964+0000","last_unstale":"2024-03-06T07:29:05.875132+0000","last_undegraded":"2024-03-06T07:29:05.875132+0000","last_fullsized":"2024-03-06T07:29:05.875132+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.050440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199011+0000","last_change":"2024-03-06T07:29:04.865474+0000","last_active":"2024-03-06T07:29:06.199011+0000","last_peered":"2024-03-06T07:29:06.199011+0000","last_clean":"2024-03-06T07:29:06.199011+0000","last_became_active":"2024-03-06T07:29:04.865124+0000","last_became_peered":"2024-03-06T07:29:04.865124+0000","last_unstale":"2024-03-06T07:29:06.199011+0000","last_undegraded":"2024-03-06T07:29:06.199011+0000","last_fullsized":"2024-03-06T07:29:06.199011+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:39.166593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870764+0000","last_change":"2024-03-06T07:29:04.883964+0000","last_active":"2024-03-06T07:29:05.870764+0000","last_peered":"2024-03-06T07:29:05.870764+0000","last_clean":"2024-03-06T07:29:05.870764+0000","last_became_active":"2024-03-06T07:29:04.883636+0000","last_became_peered":"2024-03-06T07:29:04.883636+0000","last_unstale":"2024-03-06T07:29:05.870764+0000","last_undegraded":"2024-03-06T07:29:05.870764+0000","last_fullsized":"2024-03-06T07:29:05.870764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:00.120836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199767+0000","last_change":"2024-03-06T07:29:04.875428+0000","last_active":"2024-03-06T07:29:06.199767+0000","last_peered":"2024-03-06T07:29:06.199767+0000","last_clean":"2024-03-06T07:29:06.199767+0000","last_became_active":"2024-03-06T07:29:04.875036+0000","last_became_peered":"2024-03-06T07:29:04.875036+0000","last_unstale":"2024-03-06T07:29:06.199767+0000","last_undegraded":"2024-03-06T07:29:06.199767+0000","last_fullsized":"2024-03-06T07:29:06.199767+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:10:10.951882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199720+0000","last_change":"2024-03-06T07:29:04.859104+0000","last_active":"2024-03-06T07:29:06.199720+0000","last_peered":"2024-03-06T07:29:06.199720+0000","last_clean":"2024-03-06T07:29:06.199720+0000","last_became_active":"2024-03-06T07:29:04.858444+0000","last_became_peered":"2024-03-06T07:29:04.858444+0000","last_unstale":"2024-03-06T07:29:06.199720+0000","last_undegraded":"2024-03-06T07:29:06.199720+0000","last_fullsized":"2024-03-06T07:29:06.199720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:25.639921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197795+0000","last_change":"2024-03-06T07:29:04.878191+0000","last_active":"2024-03-06T07:29:06.197795+0000","last_peered":"2024-03-06T07:29:06.197795+0000","last_clean":"2024-03-06T07:29:06.197795+0000","last_became_active":"2024-03-06T07:29:04.877879+0000","last_became_peered":"2024-03-06T07:29:04.877879+0000","last_unstale":"2024-03-06T07:29:06.197795+0000","last_undegraded":"2024-03-06T07:29:06.197795+0000","last_fullsized":"2024-03-06T07:29:06.197795+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:07.259591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866369+0000","last_change":"2024-03-06T07:29:04.882320+0000","last_active":"2024-03-06T07:29:05.866369+0000","last_peered":"2024-03-06T07:29:05.866369+0000","last_clean":"2024-03-06T07:29:05.866369+0000","last_became_active":"2024-03-06T07:29:04.881904+0000","last_became_peered":"2024-03-06T07:29:04.881904+0000","last_unstale":"2024-03-06T07:29:05.866369+0000","last_undegraded":"2024-03-06T07:29:05.866369+0000","last_fullsized":"2024-03-06T07:29:05.866369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:02.588765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201673+0000","last_change":"2024-03-06T07:29:03.873666+0000","last_active":"2024-03-06T07:29:06.201673+0000","last_peered":"2024-03-06T07:29:06.201673+0000","last_clean":"2024-03-06T07:29:06.201673+0000","last_became_active":"2024-03-06T07:29:03.873263+0000","last_became_peered":"2024-03-06T07:29:03.873263+0000","last_unstale":"2024-03-06T07:29:06.201673+0000","last_undegraded":"2024-03-06T07:29:06.201673+0000","last_fullsized":"2024-03-06T07:29:06.201673+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201722+0000","last_change":"2024-03-06T07:29:04.886962+0000","last_active":"2024-03-06T07:29:06.201722+0000","last_peered":"2024-03-06T07:29:06.201722+0000","last_clean":"2024-03-06T07:29:06.201722+0000","last_became_active":"2024-03-06T07:29:04.886690+0000","last_became_peered":"2024-03-06T07:29:04.886690+0000","last_unstale":"2024-03-06T07:29:06.201722+0000","last_undegraded":"2024-03-06T07:29:06.201722+0000","last_fullsized":"2024-03-06T07:29:06.201722+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:39.668562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866400+0000","last_change":"2024-03-06T07:29:03.869646+0000","last_active":"2024-03-06T07:29:05.866400+0000","last_peered":"2024-03-06T07:29:05.866400+0000","last_clean":"2024-03-06T07:29:05.866400+0000","last_became_active":"2024-03-06T07:29:03.869102+0000","last_became_peered":"2024-03-06T07:29:03.869102+0000","last_unstale":"2024-03-06T07:29:05.866400+0000","last_undegraded":"2024-03-06T07:29:05.866400+0000","last_fullsized":"2024-03-06T07:29:05.866400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:07.769778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.871280+0000","last_change":"2024-03-06T07:29:04.887273+0000","last_active":"2024-03-06T07:29:05.871280+0000","last_peered":"2024-03-06T07:29:05.871280+0000","last_clean":"2024-03-06T07:29:05.871280+0000","last_became_active":"2024-03-06T07:29:04.886013+0000","last_became_peered":"2024-03-06T07:29:04.886013+0000","last_unstale":"2024-03-06T07:29:05.871280+0000","last_undegraded":"2024-03-06T07:29:05.871280+0000","last_fullsized":"2024-03-06T07:29:05.871280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:28.827720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867339+0000","last_change":"2024-03-06T07:29:03.859102+0000","last_active":"2024-03-06T07:29:05.867339+0000","last_peered":"2024-03-06T07:29:05.867339+0000","last_clean":"2024-03-06T07:29:05.867339+0000","last_became_active":"2024-03-06T07:29:03.858905+0000","last_became_peered":"2024-03-06T07:29:03.858905+0000","last_unstale":"2024-03-06T07:29:05.867339+0000","last_undegraded":"2024-03-06T07:29:05.867339+0000","last_fullsized":"2024-03-06T07:29:05.867339+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:38.653487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875253+0000","last_change":"2024-03-06T07:29:04.883060+0000","last_active":"2024-03-06T07:29:05.875253+0000","last_peered":"2024-03-06T07:29:05.875253+0000","last_clean":"2024-03-06T07:29:05.875253+0000","last_became_active":"2024-03-06T07:29:04.882761+0000","last_became_peered":"2024-03-06T07:29:04.882761+0000","last_unstale":"2024-03-06T07:29:05.875253+0000","last_undegraded":"2024-03-06T07:29:05.875253+0000","last_fullsized":"2024-03-06T07:29:05.875253+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:59.871873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197179+0000","last_change":"2024-03-06T07:29:03.854977+0000","last_active":"2024-03-06T07:29:06.197179+0000","last_peered":"2024-03-06T07:29:06.197179+0000","last_clean":"2024-03-06T07:29:06.197179+0000","last_became_active":"2024-03-06T07:29:03.854659+0000","last_became_peered":"2024-03-06T07:29:03.854659+0000","last_unstale":"2024-03-06T07:29:06.197179+0000","last_undegraded":"2024-03-06T07:29:06.197179+0000","last_fullsized":"2024-03-06T07:29:06.197179+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:38.875235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201287+0000","last_change":"2024-03-06T07:29:04.878839+0000","last_active":"2024-03-06T07:29:06.201287+0000","last_peered":"2024-03-06T07:29:06.201287+0000","last_clean":"2024-03-06T07:29:06.201287+0000","last_became_active":"2024-03-06T07:29:04.877469+0000","last_became_peered":"2024-03-06T07:29:04.877469+0000","last_unstale":"2024-03-06T07:29:06.201287+0000","last_undegraded":"2024-03-06T07:29:06.201287+0000","last_fullsized":"2024-03-06T07:29:06.201287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:49.801021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870554+0000","last_change":"2024-03-06T07:29:03.865856+0000","last_active":"2024-03-06T07:29:05.870554+0000","last_peered":"2024-03-06T07:29:05.870554+0000","last_clean":"2024-03-06T07:29:05.870554+0000","last_became_active":"2024-03-06T07:29:03.864750+0000","last_became_peered":"2024-03-06T07:29:03.864750+0000","last_unstale":"2024-03-06T07:29:05.870554+0000","last_undegraded":"2024-03-06T07:29:05.870554+0000","last_fullsized":"2024-03-06T07:29:05.870554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:27.937514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199437+0000","last_change":"2024-03-06T07:29:04.873261+0000","last_active":"2024-03-06T07:29:06.199437+0000","last_peered":"2024-03-06T07:29:06.199437+0000","last_clean":"2024-03-06T07:29:06.199437+0000","last_became_active":"2024-03-06T07:29:04.872397+0000","last_became_peered":"2024-03-06T07:29:04.872397+0000","last_unstale":"2024-03-06T07:29:06.199437+0000","last_undegraded":"2024-03-06T07:29:06.199437+0000","last_fullsized":"2024-03-06T07:29:06.199437+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:04.236079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869903+0000","last_change":"2024-03-06T07:29:03.864850+0000","last_active":"2024-03-06T07:29:05.869903+0000","last_peered":"2024-03-06T07:29:05.869903+0000","last_clean":"2024-03-06T07:29:05.869903+0000","last_became_active":"2024-03-06T07:29:03.861142+0000","last_became_peered":"2024-03-06T07:29:03.861142+0000","last_unstale":"2024-03-06T07:29:05.869903+0000","last_undegraded":"2024-03-06T07:29:05.869903+0000","last_fullsized":"2024-03-06T07:29:05.869903+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:32.423149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870091+0000","last_change":"2024-03-06T07:29:04.881202+0000","last_active":"2024-03-06T07:29:05.870091+0000","last_peered":"2024-03-06T07:29:05.870091+0000","last_clean":"2024-03-06T07:29:05.870091+0000","last_became_active":"2024-03-06T07:29:04.880435+0000","last_became_peered":"2024-03-06T07:29:04.880435+0000","last_unstale":"2024-03-06T07:29:05.870091+0000","last_undegraded":"2024-03-06T07:29:05.870091+0000","last_fullsized":"2024-03-06T07:29:05.870091+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:46.317405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866734+0000","last_change":"2024-03-06T07:29:03.858472+0000","last_active":"2024-03-06T07:29:05.866734+0000","last_peered":"2024-03-06T07:29:05.866734+0000","last_clean":"2024-03-06T07:29:05.866734+0000","last_became_active":"2024-03-06T07:29:03.858161+0000","last_became_peered":"2024-03-06T07:29:03.858161+0000","last_unstale":"2024-03-06T07:29:05.866734+0000","last_undegraded":"2024-03-06T07:29:05.866734+0000","last_fullsized":"2024-03-06T07:29:05.866734+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:45.302331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874505+0000","last_change":"2024-03-06T07:29:04.879185+0000","last_active":"2024-03-06T07:29:05.874505+0000","last_peered":"2024-03-06T07:29:05.874505+0000","last_clean":"2024-03-06T07:29:05.874505+0000","last_became_active":"2024-03-06T07:29:04.878856+0000","last_became_peered":"2024-03-06T07:29:04.878856+0000","last_unstale":"2024-03-06T07:29:05.874505+0000","last_undegraded":"2024-03-06T07:29:05.874505+0000","last_fullsized":"2024-03-06T07:29:05.874505+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:00.120836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869691+0000","last_change":"2024-03-06T07:29:03.867364+0000","last_active":"2024-03-06T07:29:05.869691+0000","last_peered":"2024-03-06T07:29:05.869691+0000","last_clean":"2024-03-06T07:29:05.869691+0000","last_became_active":"2024-03-06T07:29:03.866841+0000","last_became_peered":"2024-03-06T07:29:03.866841+0000","last_unstale":"2024-03-06T07:29:05.869691+0000","last_undegraded":"2024-03-06T07:29:05.869691+0000","last_fullsized":"2024-03-06T07:29:05.869691+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:44.258813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200120+0000","last_change":"2024-03-06T07:29:04.887789+0000","last_active":"2024-03-06T07:29:06.200120+0000","last_peered":"2024-03-06T07:29:06.200120+0000","last_clean":"2024-03-06T07:29:06.200120+0000","last_became_active":"2024-03-06T07:29:04.887437+0000","last_became_peered":"2024-03-06T07:29:04.887437+0000","last_unstale":"2024-03-06T07:29:06.200120+0000","last_undegraded":"2024-03-06T07:29:06.200120+0000","last_fullsized":"2024-03-06T07:29:06.200120+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:14.905700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867651+0000","last_change":"2024-03-06T07:29:03.869315+0000","last_active":"2024-03-06T07:29:05.867651+0000","last_peered":"2024-03-06T07:29:05.867651+0000","last_clean":"2024-03-06T07:29:05.867651+0000","last_became_active":"2024-03-06T07:29:03.869070+0000","last_became_peered":"2024-03-06T07:29:03.869070+0000","last_unstale":"2024-03-06T07:29:05.867651+0000","last_undegraded":"2024-03-06T07:29:05.867651+0000","last_fullsized":"2024-03-06T07:29:05.867651+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:56.207229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867603+0000","last_change":"2024-03-06T07:29:04.881850+0000","last_active":"2024-03-06T07:29:05.867603+0000","last_peered":"2024-03-06T07:29:05.867603+0000","last_clean":"2024-03-06T07:29:05.867603+0000","last_became_active":"2024-03-06T07:29:04.881556+0000","last_became_peered":"2024-03-06T07:29:04.881556+0000","last_unstale":"2024-03-06T07:29:05.867603+0000","last_undegraded":"2024-03-06T07:29:05.867603+0000","last_fullsized":"2024-03-06T07:29:05.867603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:32.161449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866500+0000","last_change":"2024-03-06T07:29:03.857566+0000","last_active":"2024-03-06T07:29:05.866500+0000","last_peered":"2024-03-06T07:29:05.866500+0000","last_clean":"2024-03-06T07:29:05.866500+0000","last_became_active":"2024-03-06T07:29:03.856632+0000","last_became_peered":"2024-03-06T07:29:03.856632+0000","last_unstale":"2024-03-06T07:29:05.866500+0000","last_undegraded":"2024-03-06T07:29:05.866500+0000","last_fullsized":"2024-03-06T07:29:05.866500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.478639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867353+0000","last_change":"2024-03-06T07:29:04.869814+0000","last_active":"2024-03-06T07:29:05.867353+0000","last_peered":"2024-03-06T07:29:05.867353+0000","last_clean":"2024-03-06T07:29:05.867353+0000","last_became_active":"2024-03-06T07:29:04.868733+0000","last_became_peered":"2024-03-06T07:29:04.868733+0000","last_unstale":"2024-03-06T07:29:05.867353+0000","last_undegraded":"2024-03-06T07:29:05.867353+0000","last_fullsized":"2024-03-06T07:29:05.867353+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:16.747901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.199404+0000","last_change":"2024-03-06T07:29:03.871624+0000","last_active":"2024-03-06T07:29:06.199404+0000","last_peered":"2024-03-06T07:29:06.199404+0000","last_clean":"2024-03-06T07:29:06.199404+0000","last_became_active":"2024-03-06T07:29:03.871249+0000","last_became_peered":"2024-03-06T07:29:03.871249+0000","last_unstale":"2024-03-06T07:29:06.199404+0000","last_undegraded":"2024-03-06T07:29:06.199404+0000","last_fullsized":"2024-03-06T07:29:06.199404+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:38.653487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198847+0000","last_change":"2024-03-06T07:29:04.881197+0000","last_active":"2024-03-06T07:29:06.198847+0000","last_peered":"2024-03-06T07:29:06.198847+0000","last_clean":"2024-03-06T07:29:06.198847+0000","last_became_active":"2024-03-06T07:29:04.880923+0000","last_became_peered":"2024-03-06T07:29:04.880923+0000","last_unstale":"2024-03-06T07:29:06.198847+0000","last_undegraded":"2024-03-06T07:29:06.198847+0000","last_fullsized":"2024-03-06T07:29:06.198847+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:41.329709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867400+0000","last_change":"2024-03-06T07:29:03.860307+0000","last_active":"2024-03-06T07:29:05.867400+0000","last_peered":"2024-03-06T07:29:05.867400+0000","last_clean":"2024-03-06T07:29:05.867400+0000","last_became_active":"2024-03-06T07:29:03.860043+0000","last_became_peered":"2024-03-06T07:29:03.860043+0000","last_unstale":"2024-03-06T07:29:05.867400+0000","last_undegraded":"2024-03-06T07:29:05.867400+0000","last_fullsized":"2024-03-06T07:29:05.867400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:01.573690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874727+0000","last_change":"2024-03-06T07:29:04.875307+0000","last_active":"2024-03-06T07:29:05.874727+0000","last_peered":"2024-03-06T07:29:05.874727+0000","last_clean":"2024-03-06T07:29:05.874727+0000","last_became_active":"2024-03-06T07:29:04.874985+0000","last_became_peered":"2024-03-06T07:29:04.874985+0000","last_unstale":"2024-03-06T07:29:05.874727+0000","last_undegraded":"2024-03-06T07:29:05.874727+0000","last_fullsized":"2024-03-06T07:29:05.874727+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:50.833591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870892+0000","last_change":"2024-03-06T07:29:03.873904+0000","last_active":"2024-03-06T07:29:05.870892+0000","last_peered":"2024-03-06T07:29:05.870892+0000","last_clean":"2024-03-06T07:29:05.870892+0000","last_became_active":"2024-03-06T07:29:03.873433+0000","last_became_peered":"2024-03-06T07:29:03.873433+0000","last_unstale":"2024-03-06T07:29:05.870892+0000","last_undegraded":"2024-03-06T07:29:05.870892+0000","last_fullsized":"2024-03-06T07:29:05.870892+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:38.540083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.867151+0000","last_change":"2024-03-06T07:29:04.883358+0000","last_active":"2024-03-06T07:29:05.867151+0000","last_peered":"2024-03-06T07:29:05.867151+0000","last_clean":"2024-03-06T07:29:05.867151+0000","last_became_active":"2024-03-06T07:29:04.882897+0000","last_became_peered":"2024-03-06T07:29:04.882897+0000","last_unstale":"2024-03-06T07:29:05.867151+0000","last_undegraded":"2024-03-06T07:29:05.867151+0000","last_fullsized":"2024-03-06T07:29:05.867151+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:04.236079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874791+0000","last_change":"2024-03-06T07:29:03.863024+0000","last_active":"2024-03-06T07:29:05.874791+0000","last_peered":"2024-03-06T07:29:05.874791+0000","last_clean":"2024-03-06T07:29:05.874791+0000","last_became_active":"2024-03-06T07:29:03.861857+0000","last_became_peered":"2024-03-06T07:29:03.861857+0000","last_unstale":"2024-03-06T07:29:05.874791+0000","last_undegraded":"2024-03-06T07:29:05.874791+0000","last_fullsized":"2024-03-06T07:29:05.874791+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.478639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198292+0000","last_change":"2024-03-06T07:29:04.883037+0000","last_active":"2024-03-06T07:29:06.198292+0000","last_peered":"2024-03-06T07:29:06.198292+0000","last_clean":"2024-03-06T07:29:06.198292+0000","last_became_active":"2024-03-06T07:29:04.882734+0000","last_became_peered":"2024-03-06T07:29:04.882734+0000","last_unstale":"2024-03-06T07:29:06.198292+0000","last_undegraded":"2024-03-06T07:29:06.198292+0000","last_fullsized":"2024-03-06T07:29:06.198292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:39.099541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866693+0000","last_change":"2024-03-06T07:29:03.871751+0000","last_active":"2024-03-06T07:29:05.866693+0000","last_peered":"2024-03-06T07:29:05.866693+0000","last_clean":"2024-03-06T07:29:05.866693+0000","last_became_active":"2024-03-06T07:29:03.871452+0000","last_became_peered":"2024-03-06T07:29:03.871452+0000","last_unstale":"2024-03-06T07:29:05.866693+0000","last_undegraded":"2024-03-06T07:29:05.866693+0000","last_fullsized":"2024-03-06T07:29:05.866693+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:47.139784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200110+0000","last_change":"2024-03-06T07:29:04.876892+0000","last_active":"2024-03-06T07:29:06.200110+0000","last_peered":"2024-03-06T07:29:06.200110+0000","last_clean":"2024-03-06T07:29:06.200110+0000","last_became_active":"2024-03-06T07:29:04.876489+0000","last_became_peered":"2024-03-06T07:29:04.876489+0000","last_unstale":"2024-03-06T07:29:06.200110+0000","last_undegraded":"2024-03-06T07:29:06.200110+0000","last_fullsized":"2024-03-06T07:29:06.200110+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:38.880329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866796+0000","last_change":"2024-03-06T07:29:03.860168+0000","last_active":"2024-03-06T07:29:05.866796+0000","last_peered":"2024-03-06T07:29:05.866796+0000","last_clean":"2024-03-06T07:29:05.866796+0000","last_became_active":"2024-03-06T07:29:03.859890+0000","last_became_peered":"2024-03-06T07:29:03.859890+0000","last_unstale":"2024-03-06T07:29:05.866796+0000","last_undegraded":"2024-03-06T07:29:05.866796+0000","last_fullsized":"2024-03-06T07:29:05.866796+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:37.865255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870480+0000","last_change":"2024-03-06T07:29:04.880998+0000","last_active":"2024-03-06T07:29:05.870480+0000","last_peered":"2024-03-06T07:29:05.870480+0000","last_clean":"2024-03-06T07:29:05.870480+0000","last_became_active":"2024-03-06T07:29:04.880779+0000","last_became_peered":"2024-03-06T07:29:04.880779+0000","last_unstale":"2024-03-06T07:29:05.870480+0000","last_undegraded":"2024-03-06T07:29:05.870480+0000","last_fullsized":"2024-03-06T07:29:05.870480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:57.222304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875634+0000","last_change":"2024-03-06T07:29:03.861596+0000","last_active":"2024-03-06T07:29:05.875634+0000","last_peered":"2024-03-06T07:29:05.875634+0000","last_clean":"2024-03-06T07:29:05.875634+0000","last_became_active":"2024-03-06T07:29:03.861269+0000","last_became_peered":"2024-03-06T07:29:03.861269+0000","last_unstale":"2024-03-06T07:29:05.875634+0000","last_undegraded":"2024-03-06T07:29:05.875634+0000","last_fullsized":"2024-03-06T07:29:05.875634+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:17.253100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200296+0000","last_change":"2024-03-06T07:29:04.878303+0000","last_active":"2024-03-06T07:29:06.200296+0000","last_peered":"2024-03-06T07:29:06.200296+0000","last_clean":"2024-03-06T07:29:06.200296+0000","last_became_active":"2024-03-06T07:29:04.877889+0000","last_became_peered":"2024-03-06T07:29:04.877889+0000","last_unstale":"2024-03-06T07:29:06.200296+0000","last_undegraded":"2024-03-06T07:29:06.200296+0000","last_fullsized":"2024-03-06T07:29:06.200296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:16.747901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870778+0000","last_change":"2024-03-06T07:29:03.870746+0000","last_active":"2024-03-06T07:29:05.870778+0000","last_peered":"2024-03-06T07:29:05.870778+0000","last_clean":"2024-03-06T07:29:05.870778+0000","last_became_active":"2024-03-06T07:29:03.870176+0000","last_became_peered":"2024-03-06T07:29:03.870176+0000","last_unstale":"2024-03-06T07:29:05.870778+0000","last_undegraded":"2024-03-06T07:29:05.870778+0000","last_fullsized":"2024-03-06T07:29:05.870778+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:41.650116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874948+0000","last_change":"2024-03-06T07:29:04.880917+0000","last_active":"2024-03-06T07:29:05.874948+0000","last_peered":"2024-03-06T07:29:05.874948+0000","last_clean":"2024-03-06T07:29:05.874948+0000","last_became_active":"2024-03-06T07:29:04.880401+0000","last_became_peered":"2024-03-06T07:29:04.880401+0000","last_unstale":"2024-03-06T07:29:05.874948+0000","last_undegraded":"2024-03-06T07:29:05.874948+0000","last_fullsized":"2024-03-06T07:29:05.874948+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:55.768199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.870416+0000","last_change":"2024-03-06T07:29:03.869114+0000","last_active":"2024-03-06T07:29:05.870416+0000","last_peered":"2024-03-06T07:29:05.870416+0000","last_clean":"2024-03-06T07:29:05.870416+0000","last_became_active":"2024-03-06T07:29:03.868791+0000","last_became_peered":"2024-03-06T07:29:03.868791+0000","last_unstale":"2024-03-06T07:29:05.870416+0000","last_undegraded":"2024-03-06T07:29:05.870416+0000","last_fullsized":"2024-03-06T07:29:05.870416+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197070+0000","last_change":"2024-03-06T07:29:04.868441+0000","last_active":"2024-03-06T07:29:06.197070+0000","last_peered":"2024-03-06T07:29:06.197070+0000","last_clean":"2024-03-06T07:29:06.197070+0000","last_became_active":"2024-03-06T07:29:04.867986+0000","last_became_peered":"2024-03-06T07:29:04.867986+0000","last_unstale":"2024-03-06T07:29:06.197070+0000","last_undegraded":"2024-03-06T07:29:06.197070+0000","last_fullsized":"2024-03-06T07:29:06.197070+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:42.665190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.871226+0000","last_change":"2024-03-06T07:29:03.867613+0000","last_active":"2024-03-06T07:29:05.871226+0000","last_peered":"2024-03-06T07:29:05.871226+0000","last_clean":"2024-03-06T07:29:05.871226+0000","last_became_active":"2024-03-06T07:29:03.867296+0000","last_became_peered":"2024-03-06T07:29:03.867296+0000","last_unstale":"2024-03-06T07:29:05.871226+0000","last_undegraded":"2024-03-06T07:29:05.871226+0000","last_fullsized":"2024-03-06T07:29:05.871226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:15.312779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.196975+0000","last_change":"2024-03-06T07:29:04.874430+0000","last_active":"2024-03-06T07:29:06.196975+0000","last_peered":"2024-03-06T07:29:06.196975+0000","last_clean":"2024-03-06T07:29:06.196975+0000","last_became_active":"2024-03-06T07:29:04.874102+0000","last_became_peered":"2024-03-06T07:29:04.874102+0000","last_unstale":"2024-03-06T07:29:06.196975+0000","last_undegraded":"2024-03-06T07:29:06.196975+0000","last_fullsized":"2024-03-06T07:29:06.196975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:28.952589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.865991+0000","last_change":"2024-03-06T07:29:03.854465+0000","last_active":"2024-03-06T07:29:05.865991+0000","last_peered":"2024-03-06T07:29:05.865991+0000","last_clean":"2024-03-06T07:29:05.865991+0000","last_became_active":"2024-03-06T07:29:03.853828+0000","last_became_peered":"2024-03-06T07:29:03.853828+0000","last_unstale":"2024-03-06T07:29:05.865991+0000","last_undegraded":"2024-03-06T07:29:05.865991+0000","last_fullsized":"2024-03-06T07:29:05.865991+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:23.980924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197369+0000","last_change":"2024-03-06T07:29:04.877720+0000","last_active":"2024-03-06T07:29:06.197369+0000","last_peered":"2024-03-06T07:29:06.197369+0000","last_clean":"2024-03-06T07:29:06.197369+0000","last_became_active":"2024-03-06T07:29:04.877371+0000","last_became_peered":"2024-03-06T07:29:04.877371+0000","last_unstale":"2024-03-06T07:29:06.197369+0000","last_undegraded":"2024-03-06T07:29:06.197369+0000","last_fullsized":"2024-03-06T07:29:06.197369+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:24.995998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201190+0000","last_change":"2024-03-06T07:29:03.868968+0000","last_active":"2024-03-06T07:29:06.201190+0000","last_peered":"2024-03-06T07:29:06.201190+0000","last_clean":"2024-03-06T07:29:06.201190+0000","last_became_active":"2024-03-06T07:29:03.867933+0000","last_became_peered":"2024-03-06T07:29:03.867933+0000","last_unstale":"2024-03-06T07:29:06.201190+0000","last_undegraded":"2024-03-06T07:29:06.201190+0000","last_fullsized":"2024-03-06T07:29:06.201190+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:41.650116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197224+0000","last_change":"2024-03-06T07:29:04.863142+0000","last_active":"2024-03-06T07:29:06.197224+0000","last_peered":"2024-03-06T07:29:06.197224+0000","last_clean":"2024-03-06T07:29:06.197224+0000","last_became_active":"2024-03-06T07:29:04.862866+0000","last_became_peered":"2024-03-06T07:29:04.862866+0000","last_unstale":"2024-03-06T07:29:06.197224+0000","last_undegraded":"2024-03-06T07:29:06.197224+0000","last_fullsized":"2024-03-06T07:29:06.197224+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:16.327853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869657+0000","last_change":"2024-03-06T07:29:03.864280+0000","last_active":"2024-03-06T07:29:05.869657+0000","last_peered":"2024-03-06T07:29:05.869657+0000","last_clean":"2024-03-06T07:29:05.869657+0000","last_became_active":"2024-03-06T07:29:03.863397+0000","last_became_peered":"2024-03-06T07:29:03.863397+0000","last_unstale":"2024-03-06T07:29:05.869657+0000","last_undegraded":"2024-03-06T07:29:05.869657+0000","last_fullsized":"2024-03-06T07:29:05.869657+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:14.593755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200626+0000","last_change":"2024-03-06T07:29:04.874502+0000","last_active":"2024-03-06T07:29:06.200626+0000","last_peered":"2024-03-06T07:29:06.200626+0000","last_clean":"2024-03-06T07:29:06.200626+0000","last_became_active":"2024-03-06T07:29:04.874240+0000","last_became_peered":"2024-03-06T07:29:04.874240+0000","last_unstale":"2024-03-06T07:29:06.200626+0000","last_undegraded":"2024-03-06T07:29:06.200626+0000","last_fullsized":"2024-03-06T07:29:06.200626+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:14.905700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197415+0000","last_change":"2024-03-06T07:29:03.872538+0000","last_active":"2024-03-06T07:29:06.197415+0000","last_peered":"2024-03-06T07:29:06.197415+0000","last_clean":"2024-03-06T07:29:06.197415+0000","last_became_active":"2024-03-06T07:29:03.871251+0000","last_became_peered":"2024-03-06T07:29:03.871251+0000","last_unstale":"2024-03-06T07:29:06.197415+0000","last_undegraded":"2024-03-06T07:29:06.197415+0000","last_fullsized":"2024-03-06T07:29:06.197415+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:45.302331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.197256+0000","last_change":"2024-03-06T07:29:04.863561+0000","last_active":"2024-03-06T07:29:06.197256+0000","last_peered":"2024-03-06T07:29:06.197256+0000","last_clean":"2024-03-06T07:29:06.197256+0000","last_became_active":"2024-03-06T07:29:04.863282+0000","last_became_peered":"2024-03-06T07:29:04.863282+0000","last_unstale":"2024-03-06T07:29:06.197256+0000","last_undegraded":"2024-03-06T07:29:06.197256+0000","last_fullsized":"2024-03-06T07:29:06.197256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:44.493714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875760+0000","last_change":"2024-03-06T07:29:03.858375+0000","last_active":"2024-03-06T07:29:05.875760+0000","last_peered":"2024-03-06T07:29:05.875760+0000","last_clean":"2024-03-06T07:29:05.875760+0000","last_became_active":"2024-03-06T07:29:03.858097+0000","last_became_peered":"2024-03-06T07:29:03.858097+0000","last_unstale":"2024-03-06T07:29:05.875760+0000","last_undegraded":"2024-03-06T07:29:05.875760+0000","last_fullsized":"2024-03-06T07:29:05.875760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:07.769778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200569+0000","last_change":"2024-03-06T07:29:04.873219+0000","last_active":"2024-03-06T07:29:06.200569+0000","last_peered":"2024-03-06T07:29:06.200569+0000","last_clean":"2024-03-06T07:29:06.200569+0000","last_became_active":"2024-03-06T07:29:04.872693+0000","last_became_peered":"2024-03-06T07:29:04.872693+0000","last_unstale":"2024-03-06T07:29:06.200569+0000","last_undegraded":"2024-03-06T07:29:06.200569+0000","last_fullsized":"2024-03-06T07:29:06.200569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:32.161449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866475+0000","last_change":"2024-03-06T07:29:03.853739+0000","last_active":"2024-03-06T07:29:05.866475+0000","last_peered":"2024-03-06T07:29:05.866475+0000","last_clean":"2024-03-06T07:29:05.866475+0000","last_became_active":"2024-03-06T07:29:03.853042+0000","last_became_peered":"2024-03-06T07:29:03.853042+0000","last_unstale":"2024-03-06T07:29:05.866475+0000","last_undegraded":"2024-03-06T07:29:05.866475+0000","last_fullsized":"2024-03-06T07:29:05.866475+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866205+0000","last_change":"2024-03-06T07:29:04.876775+0000","last_active":"2024-03-06T07:29:05.866205+0000","last_peered":"2024-03-06T07:29:05.866205+0000","last_clean":"2024-03-06T07:29:05.866205+0000","last_became_active":"2024-03-06T07:29:04.875987+0000","last_became_peered":"2024-03-06T07:29:04.875987+0000","last_unstale":"2024-03-06T07:29:05.866205+0000","last_undegraded":"2024-03-06T07:29:05.866205+0000","last_fullsized":"2024-03-06T07:29:05.866205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:39.668562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.875019+0000","last_change":"2024-03-06T07:29:03.855200+0000","last_active":"2024-03-06T07:29:05.875019+0000","last_peered":"2024-03-06T07:29:05.875019+0000","last_clean":"2024-03-06T07:29:05.875019+0000","last_became_active":"2024-03-06T07:29:03.854863+0000","last_became_peered":"2024-03-06T07:29:03.854863+0000","last_unstale":"2024-03-06T07:29:05.875019+0000","last_undegraded":"2024-03-06T07:29:05.875019+0000","last_fullsized":"2024-03-06T07:29:05.875019+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:20.056726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201222+0000","last_change":"2024-03-06T07:29:04.867972+0000","last_active":"2024-03-06T07:29:06.201222+0000","last_peered":"2024-03-06T07:29:06.201222+0000","last_clean":"2024-03-06T07:29:06.201222+0000","last_became_active":"2024-03-06T07:29:04.866461+0000","last_became_peered":"2024-03-06T07:29:04.866461+0000","last_unstale":"2024-03-06T07:29:06.201222+0000","last_undegraded":"2024-03-06T07:29:06.201222+0000","last_fullsized":"2024-03-06T07:29:06.201222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:48.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866167+0000","last_change":"2024-03-06T07:29:03.860203+0000","last_active":"2024-03-06T07:29:05.866167+0000","last_peered":"2024-03-06T07:29:05.866167+0000","last_clean":"2024-03-06T07:29:05.866167+0000","last_became_active":"2024-03-06T07:29:03.860001+0000","last_became_peered":"2024-03-06T07:29:03.860001+0000","last_unstale":"2024-03-06T07:29:05.866167+0000","last_undegraded":"2024-03-06T07:29:05.866167+0000","last_fullsized":"2024-03-06T07:29:05.866167+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:17.253100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.865919+0000","last_change":"2024-03-06T07:29:04.874410+0000","last_active":"2024-03-06T07:29:05.865919+0000","last_peered":"2024-03-06T07:29:05.865919+0000","last_clean":"2024-03-06T07:29:05.865919+0000","last_became_active":"2024-03-06T07:29:04.873804+0000","last_became_peered":"2024-03-06T07:29:04.873804+0000","last_unstale":"2024-03-06T07:29:05.865919+0000","last_undegraded":"2024-03-06T07:29:05.865919+0000","last_fullsized":"2024-03-06T07:29:05.865919+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:48.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.874901+0000","last_change":"2024-03-06T07:29:03.864706+0000","last_active":"2024-03-06T07:29:05.874901+0000","last_peered":"2024-03-06T07:29:05.874901+0000","last_clean":"2024-03-06T07:29:05.874901+0000","last_became_active":"2024-03-06T07:29:03.864011+0000","last_became_peered":"2024-03-06T07:29:03.864011+0000","last_unstale":"2024-03-06T07:29:05.874901+0000","last_undegraded":"2024-03-06T07:29:05.874901+0000","last_fullsized":"2024-03-06T07:29:05.874901+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:23.980924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866900+0000","last_change":"2024-03-06T07:29:04.878373+0000","last_active":"2024-03-06T07:29:05.866900+0000","last_peered":"2024-03-06T07:29:05.866900+0000","last_clean":"2024-03-06T07:29:05.866900+0000","last_became_active":"2024-03-06T07:29:04.878111+0000","last_became_peered":"2024-03-06T07:29:04.878111+0000","last_unstale":"2024-03-06T07:29:05.866900+0000","last_undegraded":"2024-03-06T07:29:05.866900+0000","last_fullsized":"2024-03-06T07:29:05.866900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:39.166593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869897+0000","last_change":"2024-03-06T07:29:03.867397+0000","last_active":"2024-03-06T07:29:05.869897+0000","last_peered":"2024-03-06T07:29:05.869897+0000","last_clean":"2024-03-06T07:29:05.869897+0000","last_became_active":"2024-03-06T07:29:03.867116+0000","last_became_peered":"2024-03-06T07:29:03.867116+0000","last_unstale":"2024-03-06T07:29:05.869897+0000","last_undegraded":"2024-03-06T07:29:05.869897+0000","last_fullsized":"2024-03-06T07:29:05.869897+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:34.707979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.869740+0000","last_change":"2024-03-06T07:29:04.886296+0000","last_active":"2024-03-06T07:29:05.869740+0000","last_peered":"2024-03-06T07:29:05.869740+0000","last_clean":"2024-03-06T07:29:05.869740+0000","last_became_active":"2024-03-06T07:29:04.886005+0000","last_became_peered":"2024-03-06T07:29:04.886005+0000","last_unstale":"2024-03-06T07:29:05.869740+0000","last_undegraded":"2024-03-06T07:29:05.869740+0000","last_fullsized":"2024-03-06T07:29:05.869740+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:48.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:05.866642+0000","last_change":"2024-03-06T07:29:03.871145+0000","last_active":"2024-03-06T07:29:05.866642+0000","last_peered":"2024-03-06T07:29:05.866642+0000","last_clean":"2024-03-06T07:29:05.866642+0000","last_became_active":"2024-03-06T07:29:03.870797+0000","last_became_peered":"2024-03-06T07:29:03.870797+0000","last_unstale":"2024-03-06T07:29:05.866642+0000","last_undegraded":"2024-03-06T07:29:05.866642+0000","last_fullsized":"2024-03-06T07:29:05.866642+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:24.624847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.201836+0000","last_change":"2024-03-06T07:29:04.883862+0000","last_active":"2024-03-06T07:29:06.201836+0000","last_peered":"2024-03-06T07:29:06.201836+0000","last_clean":"2024-03-06T07:29:06.201836+0000","last_became_active":"2024-03-06T07:29:04.883554+0000","last_became_peered":"2024-03-06T07:29:04.883554+0000","last_unstale":"2024-03-06T07:29:06.201836+0000","last_undegraded":"2024-03-06T07:29:06.201836+0000","last_fullsized":"2024-03-06T07:29:06.201836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:03.816051+0000","last_clean_scrub_stamp":"2024-03-06T07:29:03.816051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:16.747901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.200922+0000","last_change":"2024-03-06T07:29:03.872234+0000","last_active":"2024-03-06T07:29:06.200922+0000","last_peered":"2024-03-06T07:29:06.200922+0000","last_clean":"2024-03-06T07:29:06.200922+0000","last_became_active":"2024-03-06T07:29:03.870531+0000","last_became_peered":"2024-03-06T07:29:03.870531+0000","last_unstale":"2024-03-06T07:29:06.200922+0000","last_undegraded":"2024-03-06T07:29:06.200922+0000","last_fullsized":"2024-03-06T07:29:06.200922+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:02.800977+0000","last_clean_scrub_stamp":"2024-03-06T07:29:02.800977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:15.312779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":193,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:29:06.198848+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:29:06.198848+0000","last_peered":"2024-03-06T07:29:06.198848+0000","last_clean":"2024-03-06T07:29:06.198848+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:29:06.198848+0000","last_undegraded":"2024-03-06T07:29:06.198848+0000","last_fullsized":"2024-03-06T07:29:06.198848+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":146848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":737146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:09.877 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:29:09.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:29:10.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.131+0000 7fa9a2a00640 1 Processor -- start 2024-03-06T07:29:10.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.131+0000 7fa9a2a00640 1 -- start start 2024-03-06T07:29:10.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.133+0000 7fa9a2a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 0x7fa99c06d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.133+0000 7fa9a2a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c162300 con 0x7fa99c15dde0 2024-03-06T07:29:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.133+0000 7fa9a2a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c1625e0 con 0x7fa99c06afc0 2024-03-06T07:29:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.133+0000 7fa9a2a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1628c0 con 0x7fa99c06cbf0 2024-03-06T07:29:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.133+0000 7fa9a1600640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa99c15dde0 0x7fa99c161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58806/0 (socket says 172.21.15.114:58806) 2024-03-06T07:29:10.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.133+0000 7fa9a1600640 1 -- 172.21.15.114:0/2899464866 learned_addr learned my addr 172.21.15.114:0/2899464866 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:10.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.133+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060777831 0 0) 0x7fa99c162300 con 0x7fa99c15dde0 2024-03-06T07:29:10.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa9a0c00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 0x7fa99c06d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa984003520 con 0x7fa99c15dde0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3821402243 0 0) 0x7fa984003520 con 0x7fa99c15dde0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 msgr2=0x7fa99c06d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 0x7fa99c06d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 >> v1:172.21.15.177:6789/0 conn(0x7fa99c06afc0 legacy=0x7fa99c0ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0ad860 con 0x7fa99c15dde0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fa990001670 con 0x7fa99c15dde0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 >> v1:172.21.15.114:6789/0 conn(0x7fa99c15dde0 legacy=0x7fa99c161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9840039e0 0x7fa984003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa984004800 0x7fa984004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 0x7fa984005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1628c0 con 0x7fa984004800 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98400bb50 con 0x7fa9840039e0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.134+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98400bd50 con 0x7fa99c06cbf0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a0c00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa984004800 0x7fa984004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a1600640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9840039e0 0x7fa984003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a2000640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 0x7fa984005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a0c00640 1 -- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 msgr2=0x7fa984005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a0c00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c06cbf0 0x7fa984005270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a0c00640 1 -- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9840039e0 msgr2=0x7fa984003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a0c00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9840039e0 0x7fa984003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.135+0000 7fa9a0c00640 1 -- 172.21.15.114:0/2899464866 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa984003520 con 0x7fa984004800 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.136+0000 7fa9a0c00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa984004800 0x7fa984004c80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa988009ac0 tx=0x7fa984005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a443a4721c239b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.136+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988039070 con 0x7fa984004800 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.136+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98802c6b0 con 0x7fa984004800 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.136+0000 7fa99be00640 1 -- 172.21.15.114:0/2899464866 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988009690 con 0x7fa984004800 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.137+0000 7fa9a2a00640 1 -- 172.21.15.114:0/2899464866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa984004800 msgr2=0x7fa984004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.137+0000 7fa9a2a00640 1 --2- 172.21.15.114:0/2899464866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa984004800 0x7fa984004c80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa988009ac0 tx=0x7fa984005830 comp rx=0 tx=0).stop 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.137+0000 7fa9a2000640 1 -- 172.21.15.114:0/2899464866 reap_dead start 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.137+0000 7fa9a2a00640 1 -- 172.21.15.114:0/2899464866 shutdown_connections 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.137+0000 7fa9a2a00640 1 -- 172.21.15.114:0/2899464866 >> 172.21.15.114:0/2899464866 conn(0x7fa99c06c0d0 msgr2=0x7fa99c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.137+0000 7fa9a2a00640 1 -- 172.21.15.114:0/2899464866 shutdown_connections 2024-03-06T07:29:10.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.137+0000 7fa9a2a00640 1 -- 172.21.15.114:0/2899464866 wait complete. 2024-03-06T07:29:10.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.139+0000 7fa9a2a00640 1 Processor -- start 2024-03-06T07:29:10.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.139+0000 7fa9a2a00640 1 -- start start 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa984004800 0x7fa99c14b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c0ccc90 0x7fa99c0b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa99c158a70 0x7fa99c0b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c15f350 con 0x7fa99c158a70 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c15f1d0 con 0x7fa984004800 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0ae770 con 0x7fa99c0ccc90 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa99c158a70 0x7fa99c0b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa99c158a70 0x7fa99c0b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40178/0 (socket says 172.21.15.114:40178) 2024-03-06T07:29:10.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.140+0000 7fa9a2000640 1 -- 172.21.15.114:0/1657912792 learned_addr learned my addr 172.21.15.114:0/1657912792 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:10.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a0c00640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c0ccc90 0x7fa99c0b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a1600640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa984004800 0x7fa99c14b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a2000640 1 -- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c0ccc90 msgr2=0x7fa99c0b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a2000640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c0ccc90 0x7fa99c0b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a2000640 1 -- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa984004800 msgr2=0x7fa99c14b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a2000640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa984004800 0x7fa99c14b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a2000640 1 -- 172.21.15.114:0/1657912792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0cd200 con 0x7fa99c158a70 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa9a2000640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa99c158a70 0x7fa99c0b1ca0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa99c06f800 tx=0x7fa99400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.141+0000 7fa999600640 1 -- 172.21.15.114:0/1657912792 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994019070 con 0x7fa99c158a70 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.142+0000 7fa999600640 1 -- 172.21.15.114:0/1657912792 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994003b70 con 0x7fa99c158a70 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.142+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c0cd4e0 con 0x7fa99c158a70 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.142+0000 7fa999600640 1 -- 172.21.15.114:0/1657912792 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9940050d0 con 0x7fa99c158a70 2024-03-06T07:29:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.142+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c0cdaa0 con 0x7fa99c158a70 2024-03-06T07:29:10.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.143+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c075450 con 0x7fa99c158a70 2024-03-06T07:29:10.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.144+0000 7fa999600640 1 -- 172.21.15.114:0/1657912792 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa99401c070 con 0x7fa99c158a70 2024-03-06T07:29:10.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.145+0000 7fa999600640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa97c06b500 0x7fa97c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.145+0000 7fa999600640 1 -- 172.21.15.114:0/1657912792 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9940895d0 con 0x7fa99c158a70 2024-03-06T07:29:10.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.146+0000 7fa9a1600640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa97c06b500 0x7fa97c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.146+0000 7fa9a1600640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa97c06b500 0x7fa97c06d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa990001050 tx=0x7fa990009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.149+0000 7fa999600640 1 -- 172.21.15.114:0/1657912792 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa994058450 con 0x7fa99c158a70 2024-03-06T07:29:10.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.414+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa99c06bca0 con 0x7fa99c158a70 2024-03-06T07:29:10.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.418+0000 7fa999600640 1 -- 172.21.15.114:0/1657912792 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa994057d50 con 0x7fa99c158a70 2024-03-06T07:29:10.420 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:29:10.420 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:29:10.420 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:29:10.420 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:29:10.420 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:10.421 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:29:10.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa97c06b500 msgr2=0x7fa97c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa97c06b500 0x7fa97c06d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa990001050 tx=0x7fa990009040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa99c158a70 msgr2=0x7fa99c0b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa99c158a70 0x7fa99c0b1ca0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa99c06f800 tx=0x7fa99400dc00 comp rx=0 tx=0).stop 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 shutdown_connections 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa97c06b500 0x7fa97c06d9c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa99c158a70 0x7fa99c0b1ca0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa99c0ccc90 0x7fa99c0b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 --2- 172.21.15.114:0/1657912792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa984004800 0x7fa99c14b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 >> 172.21.15.114:0/1657912792 conn(0x7fa99c06c0d0 msgr2=0x7fa99c0b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 shutdown_connections 2024-03-06T07:29:10.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.422+0000 7fa9a2a00640 1 -- 172.21.15.114:0/1657912792 wait complete. 2024-03-06T07:29:10.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:10.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.625+0000 7fa9e9000640 1 Processor -- start 2024-03-06T07:29:10.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.625+0000 7fa9e9000640 1 -- start start 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e9000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa9e415f950 0x7fa9e415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9e406afc0 0x7fa9e406b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e9000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e415d2f0 con 0x7fa9e4145540 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e9000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e406b970 con 0x7fa9e406afc0 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e9000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e40ce960 con 0x7fa9e415f950 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa9e4145540 0x7fa9e406e0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58822/0 (socket says 172.21.15.114:58822) 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e3400640 1 -- 172.21.15.114:0/1442247034 learned_addr learned my addr 172.21.15.114:0/1442247034 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e3e00640 1 --2- 172.21.15.114:0/1442247034 >> v2:172.21.15.114:3300/0 conn(0x7fa9e406afc0 0x7fa9e406b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.627+0000 7fa9e2a00640 1 --2- 172.21.15.114:0/1442247034 >> v2:172.21.15.177:3300/0 conn(0x7fa9e415f950 0x7fa9e415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.628+0000 7fa9e2000640 1 -- 172.21.15.114:0/1442247034 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286207469 0 0) 0x7fa9e415d2f0 con 0x7fa9e4145540 2024-03-06T07:29:10.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.628+0000 7fa9e2000640 1 -- 172.21.15.114:0/1442247034 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c8003520 con 0x7fa9e4145540 2024-03-06T07:29:10.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.628+0000 7fa9e3e00640 1 -- 172.21.15.114:0/1442247034 >> v2:172.21.15.177:3300/0 conn(0x7fa9e415f950 msgr2=0x7fa9e415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.628+0000 7fa9e3e00640 1 --2- 172.21.15.114:0/1442247034 >> v2:172.21.15.177:3300/0 conn(0x7fa9e415f950 0x7fa9e415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.628+0000 7fa9e3e00640 1 -- 172.21.15.114:0/1442247034 >> v1:172.21.15.114:6789/0 conn(0x7fa9e4145540 legacy=0x7fa9e406e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.628+0000 7fa9e3e00640 1 -- 172.21.15.114:0/1442247034 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e415fed0 con 0x7fa9e406afc0 2024-03-06T07:29:10.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.629+0000 7fa9e3e00640 1 --2- 172.21.15.114:0/1442247034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e406b430 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40b5fd0 tx=0x7fa9dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894545ae20b3dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.629+0000 7fa9e2000640 1 -- 172.21.15.114:0/1442247034 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc039070 con 0x7fa9e406afc0 2024-03-06T07:29:10.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.629+0000 7fa9e2000640 1 -- 172.21.15.114:0/1442247034 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc02cc80 con 0x7fa9e406afc0 2024-03-06T07:29:10.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.629+0000 7fa9e2000640 1 -- 172.21.15.114:0/1442247034 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc034640 con 0x7fa9e406afc0 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 -- 172.21.15.114:0/1442247034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 msgr2=0x7fa9e406b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 --2- 172.21.15.114:0/1442247034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e406b430 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40b5fd0 tx=0x7fa9dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 -- 172.21.15.114:0/1442247034 shutdown_connections 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 --2- 172.21.15.114:0/1442247034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e406b430 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 --2- 172.21.15.114:0/1442247034 >> v2:172.21.15.177:3300/0 conn(0x7fa9e415f950 0x7fa9e415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 -- 172.21.15.114:0/1442247034 >> 172.21.15.114:0/1442247034 conn(0x7fa9e406c0d0 msgr2=0x7fa9e406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 -- 172.21.15.114:0/1442247034 shutdown_connections 2024-03-06T07:29:10.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.630+0000 7fa9e9000640 1 -- 172.21.15.114:0/1442247034 wait complete. 2024-03-06T07:29:10.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.632+0000 7fa9e9000640 1 Processor -- start 2024-03-06T07:29:10.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.632+0000 7fa9e9000640 1 -- start start 2024-03-06T07:29:10.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e4156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9e4145540 0x7fa9e415da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e9000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9e415f950 0x7fa9e413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4160770 con 0x7fa9e406afc0 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4160490 con 0x7fa9e4145540 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e9000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4145920 con 0x7fa9e415f950 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e4156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e4156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40182/0 (socket says 172.21.15.114:40182) 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e3400640 1 -- 172.21.15.114:0/642506490 learned_addr learned my addr 172.21.15.114:0/642506490 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e2a00640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9e4145540 0x7fa9e415da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.633+0000 7fa9e3e00640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9e415f950 0x7fa9e413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9e3400640 1 -- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9e415f950 msgr2=0x7fa9e413ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9e3400640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9e415f950 0x7fa9e413ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9e3400640 1 -- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9e4145540 msgr2=0x7fa9e415da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9e3400640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9e4145540 0x7fa9e415da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9e3400640 1 -- 172.21.15.114:0/642506490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e415ed70 con 0x7fa9e406afc0 2024-03-06T07:29:10.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9e3400640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e4156080 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400c430 tx=0x7fa9d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9db400640 1 -- 172.21.15.114:0/642506490 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4017070 con 0x7fa9e406afc0 2024-03-06T07:29:10.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9db400640 1 -- 172.21.15.114:0/642506490 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4005ce0 con 0x7fa9e406afc0 2024-03-06T07:29:10.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9db400640 1 -- 172.21.15.114:0/642506490 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40072f0 con 0x7fa9e406afc0 2024-03-06T07:29:10.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.634+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e415a020 con 0x7fa9e406afc0 2024-03-06T07:29:10.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.635+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e415f2f0 con 0x7fa9e406afc0 2024-03-06T07:29:10.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.636+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e404f330 con 0x7fa9e406afc0 2024-03-06T07:29:10.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.637+0000 7fa9db400640 1 -- 172.21.15.114:0/642506490 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9d401a070 con 0x7fa9e406afc0 2024-03-06T07:29:10.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.637+0000 7fa9db400640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9c406b500 0x7fa9c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.638+0000 7fa9db400640 1 -- 172.21.15.114:0/642506490 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9d4089680 con 0x7fa9e406afc0 2024-03-06T07:29:10.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.638+0000 7fa9e2a00640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9c406b500 0x7fa9c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.638+0000 7fa9e2a00640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9c406b500 0x7fa9c406d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4159d10 tx=0x7fa9d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.646+0000 7fa9db400640 1 -- 172.21.15.114:0/642506490 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9d40584f0 con 0x7fa9e406afc0 2024-03-06T07:29:10.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.875+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9e406bc60 con 0x7fa9e406afc0 2024-03-06T07:29:10.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.876+0000 7fa9db400640 1 -- 172.21.15.114:0/642506490 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fa9d4057df0 con 0x7fa9e406afc0 2024-03-06T07:29:10.878 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:10.878 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:10.878 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:10.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.878+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9c406b500 msgr2=0x7fa9c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.878+0000 7fa9e9000640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9c406b500 0x7fa9c406d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4159d10 tx=0x7fa9d0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.878+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 msgr2=0x7fa9e4156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.878+0000 7fa9e9000640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e4156080 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400c430 tx=0x7fa9d400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 shutdown_connections 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9e415f950 0x7fa9e413ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9e4145540 0x7fa9e415da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9c406b500 0x7fa9c406d9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 --2- 172.21.15.114:0/642506490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e406afc0 0x7fa9e4156080 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 >> 172.21.15.114:0/642506490 conn(0x7fa9e406c0d0 msgr2=0x7fa9e406e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 shutdown_connections 2024-03-06T07:29:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:10.879+0000 7fa9e9000640 1 -- 172.21.15.114:0/642506490 wait complete. 2024-03-06T07:29:11.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:04.313533+0000', 'modified': '2024-03-06T07:29:07.379375+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5088}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5088': {'gid': 5088, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.114:6837/3915705340', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 3915705340}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 3915705340}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5088, 'qdb_cluster': [5088]} 2024-03-06T07:29:11.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:11.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:29:11.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.549+0000 7f39c9400640 1 Processor -- start 2024-03-06T07:29:11.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.550+0000 7f39c9400640 1 -- start start 2024-03-06T07:29:11.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.551+0000 7f39c9400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c40cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.551+0000 7f39c9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39c415f4e0 0x7f39c40cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.551+0000 7f39c9400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40b1f00 con 0x7f39c415f100 2024-03-06T07:29:11.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.551+0000 7f39c9400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c40b21a0 con 0x7f39c415f4e0 2024-03-06T07:29:11.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.551+0000 7f39c9400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40b2480 con 0x7f39c40b4140 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.551+0000 7f39c8a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39c415f4e0 0x7f39c40cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.552+0000 7f39c8a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39c415f4e0 0x7f39c40cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40184/0 (socket says 172.21.15.114:40184) 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.552+0000 7f39c8a00640 1 -- 172.21.15.114:0/3940154876 learned_addr learned my addr 172.21.15.114:0/3940154876 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.552+0000 7f39c3400640 1 --2- 172.21.15.114:0/3940154876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c40cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.552+0000 7f39c8a00640 1 -- 172.21.15.114:0/3940154876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 msgr2=0x7f39c40cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.552+0000 7f39c8a00640 1 --2- 172.21.15.114:0/3940154876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c40cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.552+0000 7f39c8a00640 1 -- 172.21.15.114:0/3940154876 >> v1:172.21.15.177:6789/0 conn(0x7f39c415f100 legacy=0x7f39c40ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.552+0000 7f39c8a00640 1 -- 172.21.15.114:0/3940154876 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c40b2760 con 0x7f39c415f4e0 2024-03-06T07:29:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.553+0000 7f39c8a00640 1 --2- 172.21.15.114:0/3940154876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f4e0 0x7f39c40cca50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009870 tx=0x7f39bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3cf3dcb06902e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.554+0000 7f39c2a00640 1 -- 172.21.15.114:0/3940154876 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc035020 con 0x7f39c415f4e0 2024-03-06T07:29:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.554+0000 7f39c2a00640 1 -- 172.21.15.114:0/3940154876 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc031640 con 0x7f39c415f4e0 2024-03-06T07:29:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.554+0000 7f39c2a00640 1 -- 172.21.15.114:0/3940154876 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc02d040 con 0x7f39c415f4e0 2024-03-06T07:29:11.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.554+0000 7f39c9400640 1 -- 172.21.15.114:0/3940154876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f4e0 msgr2=0x7f39c40cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.554+0000 7f39c9400640 1 --2- 172.21.15.114:0/3940154876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f4e0 0x7f39c40cca50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009870 tx=0x7f39bc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.555+0000 7f39c9400640 1 -- 172.21.15.114:0/3940154876 shutdown_connections 2024-03-06T07:29:11.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.555+0000 7f39c9400640 1 --2- 172.21.15.114:0/3940154876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f4e0 0x7f39c40cca50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.555+0000 7f39c9400640 1 --2- 172.21.15.114:0/3940154876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c40cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.555+0000 7f39c9400640 1 -- 172.21.15.114:0/3940154876 >> 172.21.15.114:0/3940154876 conn(0x7f39c40b3620 msgr2=0x7f39c40b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.555+0000 7f39c9400640 1 -- 172.21.15.114:0/3940154876 shutdown_connections 2024-03-06T07:29:11.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.555+0000 7f39c9400640 1 -- 172.21.15.114:0/3940154876 wait complete. 2024-03-06T07:29:11.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.556+0000 7f39c9400640 1 Processor -- start 2024-03-06T07:29:11.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.556+0000 7f39c9400640 1 -- start start 2024-03-06T07:29:11.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.557+0000 7f39c9400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.557+0000 7f39c9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f100 0x7f39c40cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.557+0000 7f39c9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39c415f4e0 0x7f39c40cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.557+0000 7f39c9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40b33b0 con 0x7f39c415f100 2024-03-06T07:29:11.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.557+0000 7f39c9400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40b3230 con 0x7f39c415f4e0 2024-03-06T07:29:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.557+0000 7f39c9400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40cf110 con 0x7f39c40b4140 2024-03-06T07:29:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f100 0x7f39c40cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f100 0x7f39c40cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40192/0 (socket says 172.21.15.114:40192) 2024-03-06T07:29:11.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 -- 172.21.15.114:0/2768265217 learned_addr learned my addr 172.21.15.114:0/2768265217 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c8a00640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39c415f4e0 0x7f39c40cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3e00640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 -- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 msgr2=0x7f39c414f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c414f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 -- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39c415f4e0 msgr2=0x7f39c40cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39c415f4e0 0x7f39c40cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 -- 172.21.15.114:0/2768265217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c414f780 con 0x7f39c415f100 2024-03-06T07:29:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3e00640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c8a00640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39c415f4e0 0x7f39c40cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.558+0000 7f39c3400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f100 0x7f39c40cadd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f39b400efc0 tx=0x7f39b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.559+0000 7f39bbe00640 1 -- 172.21.15.114:0/2768265217 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4018020 con 0x7f39c415f100 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.559+0000 7f39bbe00640 1 -- 172.21.15.114:0/2768265217 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4009e40 con 0x7f39c415f100 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.559+0000 7f39bbe00640 1 -- 172.21.15.114:0/2768265217 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4014680 con 0x7f39c415f100 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.559+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c40c9be0 con 0x7f39c415f100 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.559+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c40ca1a0 con 0x7f39c415f100 2024-03-06T07:29:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.561+0000 7f39bbe00640 1 -- 172.21.15.114:0/2768265217 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39b4005d10 con 0x7f39c415f100 2024-03-06T07:29:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.562+0000 7f39bbe00640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f39a006b4d0 0x7f39a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.562+0000 7f39bbe00640 1 -- 172.21.15.114:0/2768265217 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39b408af20 con 0x7f39c415f100 2024-03-06T07:29:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.562+0000 7f39c3e00640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f39a006b4d0 0x7f39a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.563+0000 7f39c3e00640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f39a006b4d0 0x7f39a006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f39ac000f60 tx=0x7f39ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.564+0000 7f39cd805740 1 Processor -- start 2024-03-06T07:29:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.564+0000 7f39cd805740 1 -- start start 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 0x557e47bf5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39cd805740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c25b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47bf6280 0x557e47c26070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39cd805740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e47b5b600 con 0x557e47c23180 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e47b04930 con 0x557e47bf6280 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x557e47b576a0 con 0x557e47c22da0 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c25b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c25b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40196/0 (socket says 172.21.15.114:40196) 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 -- 172.21.15.114:0/4149985989 learned_addr learned my addr 172.21.15.114:0/4149985989 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 -- 172.21.15.114:0/4149985989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 msgr2=0x557e47bf5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 --2- 172.21.15.114:0/4149985989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 0x557e47bf5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 -- 172.21.15.114:0/4149985989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47bf6280 msgr2=0x557e47c26070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 --2- 172.21.15.114:0/4149985989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47bf6280 0x557e47c26070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.565+0000 7f39c3400640 1 -- 172.21.15.114:0/4149985989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e478702d0 con 0x557e47c23180 2024-03-06T07:29:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.566+0000 7f39c3400640 1 --2- 172.21.15.114:0/4149985989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c25b30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f39b4089380 tx=0x7f39b4059750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cfe9e27ac93129 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.566+0000 7f39b3e00640 1 -- 172.21.15.114:0/4149985989 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4087070 con 0x557e47c23180 2024-03-06T07:29:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.566+0000 7f39b3e00640 1 -- 172.21.15.114:0/4149985989 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4071210 con 0x557e47c23180 2024-03-06T07:29:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.566+0000 7f39b3e00640 1 -- 172.21.15.114:0/4149985989 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4071070 con 0x557e47c23180 2024-03-06T07:29:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 -- 172.21.15.114:0/4149985989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 msgr2=0x557e47c25b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 --2- 172.21.15.114:0/4149985989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c25b30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f39b4089380 tx=0x7f39b4059750 comp rx=0 tx=0).stop 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 -- 172.21.15.114:0/4149985989 shutdown_connections 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 --2- 172.21.15.114:0/4149985989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47bf6280 0x557e47c26070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 --2- 172.21.15.114:0/4149985989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c25b30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 --2- 172.21.15.114:0/4149985989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 0x557e47bf5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 -- 172.21.15.114:0/4149985989 >> 172.21.15.114:0/4149985989 conn(0x557e47bf7fb0 msgr2=0x557e47c40b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 -- 172.21.15.114:0/4149985989 shutdown_connections 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 -- 172.21.15.114:0/4149985989 wait complete. 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.567+0000 7f39cd805740 1 -- start start 2024-03-06T07:29:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.568+0000 7f39cd805740 1 Processor -- start 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39cd805740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c3b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c23180 0x557e47c54750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22340 0x557e47c54c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39cd805740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e47bfa550 con 0x557e47c22da0 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e47b17d90 con 0x557e47c23180 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x557e47bf9940 con 0x557e47c22340 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39cd805740 10 client.0 _ll_register_callbacks cb 0x557e47b53350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c3b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c3b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40204/0 (socket says 172.21.15.114:40204) 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 -- 172.21.15.114:0/72974422 learned_addr learned my addr 172.21.15.114:0/72974422 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 -- 172.21.15.114:0/72974422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22340 msgr2=0x557e47c54c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22340 0x557e47c54c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 -- 172.21.15.114:0/72974422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c23180 msgr2=0x557e47c54750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3400640 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c23180 0x557e47c54750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c23180 0x557e47c54750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.569+0000 7f39c3e00640 1 -- 172.21.15.114:0/72974422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e4793a6b0 con 0x557e47c22da0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39c3e00640 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c3b820 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00dc30 tx=0x7f39ac014420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e0f075c4124b97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39b0c00640 1 -- 172.21.15.114:0/72974422 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac005110 con 0x557e47c22da0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39b0c00640 1 -- 172.21.15.114:0/72974422 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39ac0052b0 con 0x557e47c22da0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39b4071210 con 0x557e47c22da0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39b0c00640 1 -- 172.21.15.114:0/72974422 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39ac0054b0 con 0x557e47c22da0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39cd805740 10 client.5265 fetch_fsmap learned FSMap version 17 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.570+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x557e47c39100 con 0x557e47c22da0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39b0c00640 1 -- 172.21.15.114:0/72974422 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4071210 con 0x557e47c22da0 2024-03-06T07:29:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39b0c00640 1 -- 172.21.15.114:0/72974422 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f39ac029050 con 0x557e47c22da0 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39cd805740 10 client.5265 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39cd805740 10 client.5265 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39cd805740 20 client.5265 populate_metadata read hostname 'smithi114' 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39cd805740 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47c518e0 0x557e47c5fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39cd805740 4 client.5265 mds_command: new command op to 5088 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 --> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e47b68b40 con 0x557e47c518e0 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.571+0000 7f39c3400640 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47c518e0 0x557e47c5fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.572+0000 7f39c3400640 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47c518e0 0x557e47c5fba0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=649897ce29cbfafa server_cookie=ed55914f641b7510 in_seq=0 out_seq=0 2024-03-06T07:29:11.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.572+0000 7f39b0c00640 10 client.5265 ms_handle_connect on v2:172.21.15.114:6836/3915705340 2024-03-06T07:29:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.572+0000 7f39b0c00640 1 -- 172.21.15.114:0/72974422 <== mds.0 v2:172.21.15.114:6836/3915705340 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557e47b68b40 con 0x557e47c518e0 2024-03-06T07:29:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.572+0000 7f39b0c00640 10 client.5265 handle_command_reply: tid=0 2024-03-06T07:29:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.572+0000 7f39cd805740 1 client.5265 shutdown 2024-03-06T07:29:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 msgr2=0x557e47c3b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c3b820 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00dc30 tx=0x7f39ac014420 comp rx=0 tx=0).stop 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 shutdown_connections 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47c518e0 0x557e47c5fba0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22340 0x557e47c54c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c23180 0x557e47c54750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 --2- 172.21.15.114:0/72974422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c3b820 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39b0c00640 0 client.5265 ms_handle_reset on v2:172.21.15.114:6836/3915705340 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 >> 172.21.15.114:0/72974422 conn(0x557e47bf6280 msgr2=0x557e47c44040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 shutdown_connections 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 1 -- 172.21.15.114:0/72974422 wait complete. 2024-03-06T07:29:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.573+0000 7f39cd805740 20 client.5265 trim_cache size 0 max 16384 2024-03-06T07:29:11.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.591+0000 7f39cd805740 1 Processor -- start 2024-03-06T07:29:11.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.591+0000 7f39cd805740 1 -- start start 2024-03-06T07:29:11.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c3b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 0x557e47c37650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39cd805740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c37b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39cd805740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c406dfe0 con 0x557e47c23180 2024-03-06T07:29:11.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40c63e0 con 0x557e47c22340 2024-03-06T07:29:11.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c4150a00 con 0x557e47c22da0 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39c8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c37b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39c8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c37b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40212/0 (socket says 172.21.15.114:40212) 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.592+0000 7f39c8a00640 1 -- 172.21.15.114:0/4062396967 learned_addr learned my addr 172.21.15.114:0/4062396967 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39c8a00640 1 -- 172.21.15.114:0/4062396967 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 msgr2=0x557e47c37650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39c8a00640 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 0x557e47c37650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39c8a00640 1 -- 172.21.15.114:0/4062396967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 msgr2=0x557e47c3b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39c3e00640 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c3b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39c8a00640 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c3b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39c8a00640 1 -- 172.21.15.114:0/4062396967 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e47bf07c0 con 0x557e47c23180 2024-03-06T07:29:11.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39c8a00640 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c37b90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f39bc02bf00 tx=0x7f39bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368a9188ac87e7d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39b3400640 1 -- 172.21.15.114:0/4062396967 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc035040 con 0x557e47c23180 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39b3400640 1 -- 172.21.15.114:0/4062396967 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc031640 con 0x557e47c23180 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.593+0000 7f39b3400640 1 -- 172.21.15.114:0/4062396967 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc031a10 con 0x557e47c23180 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 -- 172.21.15.114:0/4062396967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 msgr2=0x557e47c37b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c37b90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f39bc02bf00 tx=0x7f39bc004730 comp rx=0 tx=0).stop 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 -- 172.21.15.114:0/4062396967 shutdown_connections 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c23180 0x557e47c37b90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c22da0 0x557e47c37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 --2- 172.21.15.114:0/4062396967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c3b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 -- 172.21.15.114:0/4062396967 >> 172.21.15.114:0/4062396967 conn(0x557e47bf6280 msgr2=0x557e47c26150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 -- 172.21.15.114:0/4062396967 shutdown_connections 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.594+0000 7f39cd805740 1 -- 172.21.15.114:0/4062396967 wait complete. 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.595+0000 7f39cd805740 1 -- start start 2024-03-06T07:29:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.595+0000 7f39cd805740 1 Processor -- start 2024-03-06T07:29:11.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c37650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39cd805740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c37b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39cd805740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c23180 0x557e47c364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39cd805740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e47c3bd60 con 0x557e47c22da0 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e47c396c0 con 0x557e47c22340 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39cd805740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x557e47c39f10 con 0x557e47c23180 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39cd805740 10 client.0 _ll_register_callbacks cb 0x7f39ac001300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c37b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c37b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40224/0 (socket says 172.21.15.114:40224) 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.596+0000 7f39c3400640 1 -- 172.21.15.114:0/3961943073 learned_addr learned my addr 172.21.15.114:0/3961943073 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c3e00640 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c37650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c8a00640 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c23180 0x557e47c364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c3400640 1 -- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c23180 msgr2=0x557e47c364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c3400640 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c23180 0x557e47c364c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c3400640 1 -- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 msgr2=0x557e47c37650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c3400640 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c37650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c3400640 1 -- 172.21.15.114:0/3961943073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e47c53e70 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39c3400640 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c37b90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f39b4059790 tx=0x7f39b4089310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b67ec2e59e7d7c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39b0c00640 1 -- 172.21.15.114:0/3961943073 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4087040 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39b0c00640 1 -- 172.21.15.114:0/3961943073 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39b406ed00 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39bc031640 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39b0c00640 1 -- 172.21.15.114:0/3961943073 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39b407d050 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39cd805740 10 client.5271 fetch_fsmap learned FSMap version 17 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.597+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x557e47c38280 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39b0c00640 1 -- 172.21.15.114:0/3961943073 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc031640 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39b0c00640 1 -- 172.21.15.114:0/3961943073 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f39b4080020 con 0x557e47c22da0 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39cd805740 10 client.5271 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39cd805740 10 client.5271 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39cd805740 20 client.5271 populate_metadata read hostname 'smithi114' 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39cd805740 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47cb0260 0x557e47cb0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39cd805740 4 client.5271 mds_command: new command op to 5088 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 --> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557e47b62e90 con 0x557e47cb0260 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.598+0000 7f39c3e00640 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47cb0260 0x557e47cb0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39c3e00640 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47cb0260 0x557e47cb0640 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94ce922b3884e742 server_cookie=940e345e90e9e23a in_seq=0 out_seq=0 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39b0c00640 10 client.5271 ms_handle_connect on v2:172.21.15.114:6836/3915705340 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39b0c00640 1 -- 172.21.15.114:0/3961943073 <== mds.0 v2:172.21.15.114:6836/3915705340 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557e47b62e90 con 0x557e47cb0260 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39b0c00640 10 client.5271 handle_command_reply: tid=0 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 client.5271 shutdown 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 msgr2=0x557e47c37b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c37b90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f39b4059790 tx=0x7f39b4089310 comp rx=0 tx=0).stop 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 shutdown_connections 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:6836/3915705340,v1:172.21.15.114:6837/3915705340] conn(0x557e47cb0260 0x557e47cb0640 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x557e47c23180 0x557e47c364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557e47c22da0 0x557e47c37b90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 --2- 172.21.15.114:0/3961943073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e47c22340 0x557e47c37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 >> 172.21.15.114:0/3961943073 conn(0x557e47bf6280 msgr2=0x557e47bfcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39b0c00640 0 client.5271 ms_handle_reset on v2:172.21.15.114:6836/3915705340 2024-03-06T07:29:11.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 shutdown_connections 2024-03-06T07:29:11.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.599+0000 7f39cd805740 1 -- 172.21.15.114:0/3961943073 wait complete. 2024-03-06T07:29:11.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39cd805740 20 client.5271 trim_cache size 0 max 16384 2024-03-06T07:29:11.604 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5088, 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 6.7445392249999996, 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 17, 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 33, 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 33, 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 15.574145857 2024-03-06T07:29:11.605 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f39a006b4d0 msgr2=0x7f39a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f39a006b4d0 0x7f39a006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f39ac000f60 tx=0x7f39ac009040 comp rx=0 tx=0).stop 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f100 msgr2=0x7f39c40cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f100 0x7f39c40cadd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f39b400efc0 tx=0x7f39b400c490 comp rx=0 tx=0).stop 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 shutdown_connections 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39c415f4e0 0x7f39c40cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f39a006b4d0 0x7f39a006d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39c415f100 0x7f39c40cadd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 --2- 172.21.15.114:0/2768265217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39c40b4140 0x7f39c414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 >> 172.21.15.114:0/2768265217 conn(0x7f39c40b3620 msgr2=0x7f39c40b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 shutdown_connections 2024-03-06T07:29:11.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.600+0000 7f39c9400640 1 -- 172.21.15.114:0/2768265217 wait complete. 2024-03-06T07:29:11.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.816+0000 7f39bce00640 1 Processor -- start 2024-03-06T07:29:11.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.816+0000 7f39bce00640 1 -- start start 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.818+0000 7f39bce00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f39b815f470 0x7f39b80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39bce00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39b80d0df0 0x7f39b8160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39bce00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b80d1200 con 0x7f39b815e0d0 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39bce00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80d2f60 con 0x7f39b80d0df0 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39bce00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80cea60 con 0x7f39b815f470 2024-03-06T07:29:11.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39b6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39b80d0df0 0x7f39b8160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39b6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39b80d0df0 0x7f39b8160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40230/0 (socket says 172.21.15.114:40230) 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39b6800640 1 -- 172.21.15.114:0/2489410402 learned_addr learned my addr 172.21.15.114:0/2489410402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.819+0000 7f39b5400640 1 --2- 172.21.15.114:0/2489410402 >> v2:172.21.15.177:3300/0 conn(0x7f39b815f470 0x7f39b80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.820+0000 7f39b6800640 1 -- 172.21.15.114:0/2489410402 >> v2:172.21.15.177:3300/0 conn(0x7f39b815f470 msgr2=0x7f39b80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.820+0000 7f39b6800640 1 --2- 172.21.15.114:0/2489410402 >> v2:172.21.15.177:3300/0 conn(0x7f39b815f470 0x7f39b80d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.820+0000 7f39b6800640 1 -- 172.21.15.114:0/2489410402 >> v1:172.21.15.177:6789/0 conn(0x7f39b815e0d0 legacy=0x7f39b80d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.820+0000 7f39b6800640 1 -- 172.21.15.114:0/2489410402 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80d2350 con 0x7f39b80d0df0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.821+0000 7f39b6800640 1 --2- 172.21.15.114:0/2489410402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80d0df0 0x7f39b8160dc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f39b80c77f0 tx=0x7f39b002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab34f43b9667bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39b4a00640 1 -- 172.21.15.114:0/2489410402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b002cbb0 con 0x7f39b80d0df0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39b4a00640 1 -- 172.21.15.114:0/2489410402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b002cd50 con 0x7f39b80d0df0 2024-03-06T07:29:11.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39b4a00640 1 -- 172.21.15.114:0/2489410402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0034610 con 0x7f39b80d0df0 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39bce00640 1 -- 172.21.15.114:0/2489410402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80d0df0 msgr2=0x7f39b8160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39bce00640 1 --2- 172.21.15.114:0/2489410402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80d0df0 0x7f39b8160dc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f39b80c77f0 tx=0x7f39b002c010 comp rx=0 tx=0).stop 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39bce00640 1 -- 172.21.15.114:0/2489410402 shutdown_connections 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39bce00640 1 --2- 172.21.15.114:0/2489410402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b80d0df0 0x7f39b8160dc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39bce00640 1 --2- 172.21.15.114:0/2489410402 >> v2:172.21.15.177:3300/0 conn(0x7f39b815f470 0x7f39b80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.822+0000 7f39bce00640 1 -- 172.21.15.114:0/2489410402 >> 172.21.15.114:0/2489410402 conn(0x7f39b80d3ec0 msgr2=0x7f39b80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.823+0000 7f39bce00640 1 -- 172.21.15.114:0/2489410402 shutdown_connections 2024-03-06T07:29:11.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.823+0000 7f39bce00640 1 -- 172.21.15.114:0/2489410402 wait complete. 2024-03-06T07:29:11.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.824+0000 7f39bce00640 1 Processor -- start 2024-03-06T07:29:11.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.824+0000 7f39bce00640 1 -- start start 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.825+0000 7f39bce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b80d0df0 0x7f39b8155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.825+0000 7f39bce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b815e0d0 0x7f39b81f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.825+0000 7f39bce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b815f470 0x7f39b80b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.825+0000 7f39bce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80d2bf0 con 0x7f39b815f470 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.825+0000 7f39bce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80d2910 con 0x7f39b815e0d0 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.825+0000 7f39bce00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b815fb30 con 0x7f39b80d0df0 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b815f470 0x7f39b80b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b815f470 0x7f39b80b4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40238/0 (socket says 172.21.15.114:40238) 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 -- 172.21.15.114:0/1266242129 learned_addr learned my addr 172.21.15.114:0/1266242129 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b5400640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b815e0d0 0x7f39b81f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 -- 172.21.15.114:0/1266242129 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b80d0df0 msgr2=0x7f39b8155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b80d0df0 0x7f39b8155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 -- 172.21.15.114:0/1266242129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b815e0d0 msgr2=0x7f39b81f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b815e0d0 0x7f39b81f49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 -- 172.21.15.114:0/1266242129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b81f4f00 con 0x7f39b815f470 2024-03-06T07:29:11.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39b6800640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b815f470 0x7f39b80b4960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f40 tx=0x7f39b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.826+0000 7f39ae000640 1 -- 172.21.15.114:0/1266242129 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0043020 con 0x7f39b815f470 2024-03-06T07:29:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.827+0000 7f39ae000640 1 -- 172.21.15.114:0/1266242129 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b003e450 con 0x7f39b815f470 2024-03-06T07:29:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.827+0000 7f39ae000640 1 -- 172.21.15.114:0/1266242129 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0009510 con 0x7f39b815f470 2024-03-06T07:29:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.827+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b80cb530 con 0x7f39b815f470 2024-03-06T07:29:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.827+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b80cbaf0 con 0x7f39b815f470 2024-03-06T07:29:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.828+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b804f330 con 0x7f39b815f470 2024-03-06T07:29:11.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.829+0000 7f39ae000640 1 -- 172.21.15.114:0/1266242129 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39b0046020 con 0x7f39b815f470 2024-03-06T07:29:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.830+0000 7f39ae000640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.830+0000 7f39ae000640 1 -- 172.21.15.114:0/1266242129 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39b00ac9a0 con 0x7f39b815f470 2024-03-06T07:29:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.830+0000 7f39b5e00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.831+0000 7f39b5e00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399806b4d0 0x7f399806d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f39a8000f60 tx=0x7f39a80079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:11.835+0000 7f39ae000640 1 -- 172.21.15.114:0/1266242129 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39b004d030 con 0x7f39b815f470 2024-03-06T07:29:12.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.072+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f39b80d3490 con 0x7f39b815f470 2024-03-06T07:29:12.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.072+0000 7f39ae000640 1 -- 172.21.15.114:0/1266242129 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f39b007b810 con 0x7f39b815f470 2024-03-06T07:29:12.074 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:12.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399806b4d0 msgr2=0x7f399806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399806b4d0 0x7f399806d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f39a8000f60 tx=0x7f39a80079d0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b815f470 msgr2=0x7f39b80b4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b815f470 0x7f39b80b4960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f40 tx=0x7f39b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 shutdown_connections 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b815f470 0x7f39b80b4960 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b815e0d0 0x7f39b81f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 --2- 172.21.15.114:0/1266242129 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b80d0df0 0x7f39b8155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 >> 172.21.15.114:0/1266242129 conn(0x7f39b80d3ec0 msgr2=0x7f39b806bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.075+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 shutdown_connections 2024-03-06T07:29:12.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:12.076+0000 7f39bce00640 1 -- 172.21.15.114:0/1266242129 wait complete. 2024-03-06T07:29:12.094 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:12.095 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:12.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.095 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:29:12.145 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42503sec preferred_lft 42503sec 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:12.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.148 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:29:12.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.227 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:29:12.269 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:29:12.269 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:12.269 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:29:12.269 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:29:12.269 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:12.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.270 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:29:12.270 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:29:12.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:29:12.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:29:12.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:29:12.270 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:29:12.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.481 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:29:12.529 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:29:12.529 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:29:12.530 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:12.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:29:12.622 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:12.622 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:12.671 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:12.671 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:12.718 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:29:12.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:12.850 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:12.850 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:12.884 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:12.884 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:12.884 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:29:12.884 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:29:12.884 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:29:12.886 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:29:12.886 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:12.887 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:12.887 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:12.887 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:12.890 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:12.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:12.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:12.958 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:13.017 INFO:teuthology.orchestra.run.smithi114.stdout:client5280 2024-03-06T07:29:13.018 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:29:13.018 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> subdir = "delete_me" 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> size = 1024 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> file_count = 1000 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> for i in range(0, file_count): 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:29:13.067 DEBUG:teuthology.orchestra.run.smithi114:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:29:13.068 DEBUG:teuthology.orchestra.run.smithi114:> f.write(size * '"'"'x'"'"') 2024-03-06T07:29:13.068 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:29:14.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:14.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:14.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.380+0000 7f74e1600640 1 Processor -- start 2024-03-06T07:29:14.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.380+0000 7f74e1600640 1 -- start start 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.382+0000 7f74e1600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74dc161580 0x7f74dc0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.382+0000 7f74e1600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.382+0000 7f74e1600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc161960 con 0x7f74dc0d28a0 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.382+0000 7f74e1600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc0cf6f0 con 0x7f74dc161580 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.382+0000 7f74e1600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0cf9d0 con 0x7f74dc0d1e90 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.382+0000 7f74dbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74dc161580 0x7f74dc0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74dbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74dc161580 0x7f74dc0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40240/0 (socket says 172.21.15.114:40240) 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74dbe00640 1 -- 172.21.15.114:0/2857664458 learned_addr learned my addr 172.21.15.114:0/2857664458 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74d3400640 1 --2- 172.21.15.114:0/2857664458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74dbe00640 1 -- 172.21.15.114:0/2857664458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 msgr2=0x7f74dc0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74dbe00640 1 --2- 172.21.15.114:0/2857664458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74dbe00640 1 -- 172.21.15.114:0/2857664458 >> v1:172.21.15.114:6789/0 conn(0x7f74dc0d28a0 legacy=0x7f74dc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74dbe00640 1 -- 172.21.15.114:0/2857664458 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0cfcb0 con 0x7f74dc161580 2024-03-06T07:29:14.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74d3400640 1 --2- 172.21.15.114:0/2857664458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.383+0000 7f74db400640 1 -- 172.21.15.114:0/2857664458 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843317602 0 0) 0x7f74dc161960 con 0x7f74dc0d28a0 2024-03-06T07:29:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.384+0000 7f74dbe00640 1 --2- 172.21.15.114:0/2857664458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc161580 0x7f74dc0d1950 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009870 tx=0x7f74cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b7dcdbbf2e36b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.384+0000 7f74db400640 1 -- 172.21.15.114:0/2857664458 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc034070 con 0x7f74dc161580 2024-03-06T07:29:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.384+0000 7f74db400640 1 -- 172.21.15.114:0/2857664458 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc030840 con 0x7f74dc161580 2024-03-06T07:29:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.384+0000 7f74db400640 1 -- 172.21.15.114:0/2857664458 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc030c10 con 0x7f74dc161580 2024-03-06T07:29:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 -- 172.21.15.114:0/2857664458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc161580 msgr2=0x7f74dc0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 --2- 172.21.15.114:0/2857664458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc161580 0x7f74dc0d1950 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009870 tx=0x7f74cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:29:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 -- 172.21.15.114:0/2857664458 shutdown_connections 2024-03-06T07:29:14.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 --2- 172.21.15.114:0/2857664458 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 --2- 172.21.15.114:0/2857664458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc161580 0x7f74dc0d1950 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 -- 172.21.15.114:0/2857664458 >> 172.21.15.114:0/2857664458 conn(0x7f74dc0d3d80 msgr2=0x7f74dc0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 -- 172.21.15.114:0/2857664458 shutdown_connections 2024-03-06T07:29:14.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.385+0000 7f74e1600640 1 -- 172.21.15.114:0/2857664458 wait complete. 2024-03-06T07:29:14.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.388+0000 7f74e1600640 1 Processor -- start 2024-03-06T07:29:14.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.388+0000 7f74e1600640 1 -- start start 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74e1600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74e1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc0d28a0 0x7f74dc150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74e1600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74dc161580 0x7f74dc1505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74e1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0cf3a0 con 0x7f74dc0d28a0 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74e1600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0b39a0 con 0x7f74dc161580 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74e1600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0b4ed0 con 0x7f74dc0d1e90 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74d3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc0d28a0 0x7f74dc150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74d3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc0d28a0 0x7f74dc150090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40256/0 (socket says 172.21.15.114:40256) 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74d3400640 1 -- 172.21.15.114:0/479206136 learned_addr learned my addr 172.21.15.114:0/479206136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74dbe00640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.390+0000 7f74e0c00640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74dc161580 0x7f74dc1505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d3400640 1 -- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 msgr2=0x7f74dc0cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d3400640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d3400640 1 -- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74dc161580 msgr2=0x7f74dc1505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d3400640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74dc161580 0x7f74dc1505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d3400640 1 -- 172.21.15.114:0/479206136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0c9f90 con 0x7f74dc0d28a0 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74e0c00640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74dc161580 0x7f74dc1505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d3400640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc0d28a0 0x7f74dc150090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c7f0 tx=0x7f74c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d8c00640 1 -- 172.21.15.114:0/479206136 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4015070 con 0x7f74dc0d28a0 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d8c00640 1 -- 172.21.15.114:0/479206136 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4007a30 con 0x7f74dc0d28a0 2024-03-06T07:29:14.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc0ca270 con 0x7f74dc0d28a0 2024-03-06T07:29:14.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74d8c00640 1 -- 172.21.15.114:0/479206136 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4007e00 con 0x7f74dc0d28a0 2024-03-06T07:29:14.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.391+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc0cab10 con 0x7f74dc0d28a0 2024-03-06T07:29:14.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.393+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc0ca440 con 0x7f74dc0d28a0 2024-03-06T07:29:14.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.395+0000 7f74d8c00640 1 -- 172.21.15.114:0/479206136 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74c4018070 con 0x7f74dc0d28a0 2024-03-06T07:29:14.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.396+0000 7f74d8c00640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74b806b4d0 0x7f74b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.396+0000 7f74d8c00640 1 -- 172.21.15.114:0/479206136 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74c408a1c0 con 0x7f74dc0d28a0 2024-03-06T07:29:14.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.396+0000 7f74dbe00640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74b806b4d0 0x7f74b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.397+0000 7f74dbe00640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74b806b4d0 0x7f74b806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f74cc036000 tx=0x7f74cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.401+0000 7f74d8c00640 1 -- 172.21.15.114:0/479206136 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74c4059030 con 0x7f74dc0d28a0 2024-03-06T07:29:14.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.632+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74dc04f330 con 0x7f74dc0d28a0 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.633+0000 7f74d8c00640 1 -- 172.21.15.114:0/479206136 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f74c4058930 con 0x7f74dc0d28a0 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:14.635 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74b806b4d0 msgr2=0x7f74b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74b806b4d0 0x7f74b806d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f74cc036000 tx=0x7f74cc008030 comp rx=0 tx=0).stop 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc0d28a0 msgr2=0x7f74dc150090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc0d28a0 0x7f74dc150090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c7f0 tx=0x7f74c400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 shutdown_connections 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74dc161580 0x7f74dc1505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74b806b4d0 0x7f74b806d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74dc0d28a0 0x7f74dc150090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 --2- 172.21.15.114:0/479206136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74dc0d1e90 0x7f74dc0cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 >> 172.21.15.114:0/479206136 conn(0x7f74dc0d3d80 msgr2=0x7f74dc0cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 shutdown_connections 2024-03-06T07:29:14.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:14.636+0000 7f74e1600640 1 -- 172.21.15.114:0/479206136 wait complete. 2024-03-06T07:29:14.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:29:14.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:14.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:15.246 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:29:15.246 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:15.246 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:29:15.247 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:29:15.247 DEBUG:teuthology.orchestra.run.smithi114:> import stat 2024-03-06T07:29:15.247 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:29:15.247 DEBUG:teuthology.orchestra.run.smithi114:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:29:15.247 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:29:15.353 INFO:teuthology.orchestra.run.smithi114.stdout:1099511627776 2024-03-06T07:29:15.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:15.369 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:29:15.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.511+0000 7fd267c4af40 1 Processor -- start 2024-03-06T07:29:15.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.511+0000 7fd267c4af40 1 -- start start 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd267c4af40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31737d0 0x560eb31a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd267c4af40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31beb40 0x560eb32411f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd267c4af40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 0x560eb322efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd267c4af40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb324ad90 con 0x560eb31beb40 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd267c4af40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb311bfd0 con 0x560eb31737d0 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd267c4af40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x560eb324af10 con 0x560eb324a260 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd263c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31beb40 0x560eb32411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd263c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31beb40 0x560eb32411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40262/0 (socket says 172.21.15.114:40262) 2024-03-06T07:29:15.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.512+0000 7fd263c00640 1 -- 172.21.15.114:0/2167573899 learned_addr learned my addr 172.21.15.114:0/2167573899 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:15.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd265000640 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 0x560eb322efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd264600640 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31737d0 0x560eb31a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd263c00640 1 -- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 msgr2=0x560eb322efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd263c00640 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 0x560eb322efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd263c00640 1 -- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31737d0 msgr2=0x560eb31a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd263c00640 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31737d0 0x560eb31a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd263c00640 1 -- 172.21.15.114:0/2167573899 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eb31a5050 con 0x560eb31beb40 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd263c00640 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31beb40 0x560eb32411f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd250001d80 tx=0x7fd250001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac94c6578482da02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd262800640 1 -- 172.21.15.114:0/2167573899 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25000bd70 con 0x560eb31beb40 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd262800640 1 -- 172.21.15.114:0/2167573899 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25000f400 con 0x560eb31beb40 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.513+0000 7fd262800640 1 -- 172.21.15.114:0/2167573899 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25000f7f0 con 0x560eb31beb40 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 -- 172.21.15.114:0/2167573899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31beb40 msgr2=0x560eb32411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31beb40 0x560eb32411f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd250001d80 tx=0x7fd250001e70 comp rx=0 tx=0).stop 2024-03-06T07:29:15.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 -- 172.21.15.114:0/2167573899 shutdown_connections 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 0x560eb322efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31beb40 0x560eb32411f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 --2- 172.21.15.114:0/2167573899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31737d0 0x560eb31a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 -- 172.21.15.114:0/2167573899 >> 172.21.15.114:0/2167573899 conn(0x560eb31a2bf0 msgr2=0x560eb3248960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 -- 172.21.15.114:0/2167573899 shutdown_connections 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.514+0000 7fd267c4af40 1 -- 172.21.15.114:0/2167573899 wait complete. 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.515+0000 7fd267c4af40 1 Processor -- start 2024-03-06T07:29:15.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.515+0000 7fd267c4af40 1 -- start start 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd267c4af40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31737d0 0x560eb322e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd267c4af40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31beb40 0x560eb32319a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd267c4af40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 0x560eb3231ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd267c4af40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb3173c90 con 0x560eb31737d0 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd267c4af40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eb32486f0 con 0x560eb31beb40 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd267c4af40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x560eb31a4d70 con 0x560eb324a260 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd264600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31737d0 0x560eb322e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd264600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31737d0 0x560eb322e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40266/0 (socket says 172.21.15.114:40266) 2024-03-06T07:29:15.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.516+0000 7fd264600640 1 -- 172.21.15.114:0/1625569751 learned_addr learned my addr 172.21.15.114:0/1625569751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd264600640 1 -- 172.21.15.114:0/1625569751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 msgr2=0x560eb3231ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd264600640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560eb324a260 0x560eb3231ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd264600640 1 -- 172.21.15.114:0/1625569751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31beb40 msgr2=0x560eb32319a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd264600640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb31beb40 0x560eb32319a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd264600640 1 -- 172.21.15.114:0/1625569751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eb3232700 con 0x560eb31737d0 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd264600640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31737d0 0x560eb322e130 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd25401ced0 tx=0x7fd254002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd261400640 1 -- 172.21.15.114:0/1625569751 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254061020 con 0x560eb31737d0 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd261400640 1 -- 172.21.15.114:0/1625569751 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254052970 con 0x560eb31737d0 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560eb32329e0 con 0x560eb31737d0 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd261400640 1 -- 172.21.15.114:0/1625569751 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254057040 con 0x560eb31737d0 2024-03-06T07:29:15.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.517+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560eb322e670 con 0x560eb31737d0 2024-03-06T07:29:15.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.520+0000 7fd261400640 1 -- 172.21.15.114:0/1625569751 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd254064020 con 0x560eb31737d0 2024-03-06T07:29:15.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.521+0000 7fd261400640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd244069eb0 0x7fd24406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.521+0000 7fd261400640 1 -- 172.21.15.114:0/1625569751 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2540ca900 con 0x560eb31737d0 2024-03-06T07:29:15.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.521+0000 7fd263c00640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd244069eb0 0x7fd24406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.521+0000 7fd267c4af40 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x560eb319edc0 0x560eb322f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.521+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x560eb322f7c0 con 0x560eb319edc0 2024-03-06T07:29:15.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.522+0000 7fd265000640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x560eb319edc0 0x560eb322f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.522+0000 7fd263c00640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd244069eb0 0x7fd24406c370 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x560eb31bdd80 tx=0x7fd25000f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.522+0000 7fd265000640 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x560eb319edc0 0x560eb322f280 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd265000640 1 -- 172.21.15.114:0/1625569751 <== osd.3 v2:172.21.15.114:6818/735520392 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7fd25c00d510 con 0x560eb319edc0 2024-03-06T07:29:15.525 INFO:teuthology.orchestra.run.smithi114.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:29:15.000000+0000, size 0 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x560eb319edc0 msgr2=0x560eb322f280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x560eb319edc0 0x560eb322f280 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd244069eb0 msgr2=0x7fd24406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd244069eb0 0x7fd24406c370 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x560eb31bdd80 tx=0x7fd25000f000 comp rx=0 tx=0).stop 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31737d0 msgr2=0x560eb322e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 --2- 172.21.15.114:0/1625569751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560eb31737d0 0x560eb322e130 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd25401ced0 tx=0x7fd254002fb0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd265000640 1 -- 172.21.15.114:0/1625569751 reap_dead start 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 shutdown_connections 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 >> 172.21.15.114:0/1625569751 conn(0x560eb31a2bf0 msgr2=0x560eb3248960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 shutdown_connections 2024-03-06T07:29:15.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:15.523+0000 7fd267c4af40 1 -- 172.21.15.114:0/1625569751 wait complete. 2024-03-06T07:29:15.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:15.531 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:29:18.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:18.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.387+0000 7f0bcda00640 1 Processor -- start 2024-03-06T07:29:18.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.387+0000 7f0bcda00640 1 -- start start 2024-03-06T07:29:18.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 0x7f0bc806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bcda00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0bc815f9d0 0x7f0bc815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bcda00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0bc80ce0a0 0x7f0bc80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bcda00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc8160370 con 0x7f0bc80ce0a0 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bcda00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc806ee70 con 0x7f0bc815f9d0 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bcda00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc806cef0 con 0x7f0bc8144c00 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bc7400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0bc80ce0a0 0x7f0bc80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bc7400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0bc80ce0a0 0x7f0bc80ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47828/0 (socket says 172.21.15.114:47828) 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.389+0000 7f0bc7400640 1 -- 172.21.15.114:0/1610995047 learned_addr learned my addr 172.21.15.114:0/1610995047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.390+0000 7f0bbe000640 1 --2- 172.21.15.114:0/1610995047 >> v2:172.21.15.177:3300/0 conn(0x7f0bc815f9d0 0x7f0bc815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.390+0000 7f0bc7400640 1 -- 172.21.15.114:0/1610995047 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 msgr2=0x7f0bc806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.390+0000 7f0bc7400640 1 --2- 172.21.15.114:0/1610995047 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 0x7f0bc806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.390+0000 7f0bc7400640 1 -- 172.21.15.114:0/1610995047 >> v2:172.21.15.177:3300/0 conn(0x7f0bc815f9d0 msgr2=0x7f0bc815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.390+0000 7f0bc7400640 1 --2- 172.21.15.114:0/1610995047 >> v2:172.21.15.177:3300/0 conn(0x7f0bc815f9d0 0x7f0bc815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.390+0000 7f0bc7400640 1 -- 172.21.15.114:0/1610995047 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc806d1d0 con 0x7f0bc80ce0a0 2024-03-06T07:29:18.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.390+0000 7f0bbe000640 1 --2- 172.21.15.114:0/1610995047 >> v2:172.21.15.177:3300/0 conn(0x7f0bc815f9d0 0x7f0bc815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:18.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.391+0000 7f0bc7400640 1 --2- 172.21.15.114:0/1610995047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc80ce4f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80b27f0 tx=0x7f0bc002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c20b1fbbc89807a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.391+0000 7f0bc6000640 1 -- 172.21.15.114:0/1610995047 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc002ca70 con 0x7f0bc80ce0a0 2024-03-06T07:29:18.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.391+0000 7f0bc6000640 1 -- 172.21.15.114:0/1610995047 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc002cc10 con 0x7f0bc80ce0a0 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.391+0000 7f0bc6000640 1 -- 172.21.15.114:0/1610995047 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc00343f0 con 0x7f0bc80ce0a0 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 -- 172.21.15.114:0/1610995047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc80ce0a0 msgr2=0x7f0bc80ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 --2- 172.21.15.114:0/1610995047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc80ce4f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80b27f0 tx=0x7f0bc002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 -- 172.21.15.114:0/1610995047 shutdown_connections 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 --2- 172.21.15.114:0/1610995047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc80ce4f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 --2- 172.21.15.114:0/1610995047 >> v2:172.21.15.177:3300/0 conn(0x7f0bc815f9d0 0x7f0bc815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 --2- 172.21.15.114:0/1610995047 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 0x7f0bc806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 -- 172.21.15.114:0/1610995047 >> 172.21.15.114:0/1610995047 conn(0x7f0bc80b1730 msgr2=0x7f0bc80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 -- 172.21.15.114:0/1610995047 shutdown_connections 2024-03-06T07:29:18.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.392+0000 7f0bcda00640 1 -- 172.21.15.114:0/1610995047 wait complete. 2024-03-06T07:29:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.394+0000 7f0bcda00640 1 Processor -- start 2024-03-06T07:29:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.394+0000 7f0bcda00640 1 -- start start 2024-03-06T07:29:18.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc81434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 0x7f0bc8142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc815f9d0 0x7f0bc815dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bcda00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc8160730 con 0x7f0bc815f9d0 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bcda00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc806d790 con 0x7f0bc80ce0a0 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bcda00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc806cb50 con 0x7f0bc8144c00 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc815f9d0 0x7f0bc815dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc815f9d0 0x7f0bc815dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47830/0 (socket says 172.21.15.114:47830) 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 -- 172.21.15.114:0/68348088 learned_addr learned my addr 172.21.15.114:0/68348088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 -- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 msgr2=0x7f0bc8142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 0x7f0bc8142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 -- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc80ce0a0 msgr2=0x7f0bc81434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc6a00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc81434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc81434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc7400640 1 -- 172.21.15.114:0/68348088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc815e290 con 0x7f0bc815f9d0 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.395+0000 7f0bc6a00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc81434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.396+0000 7f0bc7400640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc815f9d0 0x7f0bc815dd50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0036040 tx=0x7f0bc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.396+0000 7f0bbf400640 1 -- 172.21.15.114:0/68348088 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0043020 con 0x7f0bc815f9d0 2024-03-06T07:29:18.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.396+0000 7f0bbf400640 1 -- 172.21.15.114:0/68348088 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc003e450 con 0x7f0bc815f9d0 2024-03-06T07:29:18.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.396+0000 7f0bbf400640 1 -- 172.21.15.114:0/68348088 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0009510 con 0x7f0bc815f9d0 2024-03-06T07:29:18.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.396+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc815e560 con 0x7f0bc815f9d0 2024-03-06T07:29:18.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.396+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc815eb20 con 0x7f0bc815f9d0 2024-03-06T07:29:18.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.398+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc815ecf0 con 0x7f0bc815f9d0 2024-03-06T07:29:18.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.398+0000 7f0bbf400640 1 -- 172.21.15.114:0/68348088 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bc0046020 con 0x7f0bc815f9d0 2024-03-06T07:29:18.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.399+0000 7f0bbf400640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ba406b4d0 0x7f0ba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.399+0000 7f0bbf400640 1 -- 172.21.15.114:0/68348088 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bc00aced0 con 0x7f0bc815f9d0 2024-03-06T07:29:18.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.399+0000 7f0bc6a00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ba406b4d0 0x7f0ba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.400+0000 7f0bc6a00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ba406b4d0 0x7f0ba406d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8006fd0 tx=0x7f0bb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.408+0000 7f0bbf400640 1 -- 172.21.15.114:0/68348088 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bc815ecf0 con 0x7f0bc815f9d0 2024-03-06T07:29:18.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.770+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bc804f330 con 0x7f0bc815f9d0 2024-03-06T07:29:18.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.774+0000 7f0bbf400640 1 -- 172.21.15.114:0/68348088 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f0bc004d020 con 0x7f0bc815f9d0 2024-03-06T07:29:18.776 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:18.776 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:18.776 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:18.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.778+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ba406b4d0 msgr2=0x7f0ba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.778+0000 7f0bcda00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ba406b4d0 0x7f0ba406d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8006fd0 tx=0x7f0bb8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:18.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.778+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc815f9d0 msgr2=0x7f0bc815dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.778+0000 7f0bcda00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc815f9d0 0x7f0bc815dd50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0036040 tx=0x7f0bc0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:18.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.779+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 shutdown_connections 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.779+0000 7f0bcda00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0ba406b4d0 0x7f0ba406d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.779+0000 7f0bcda00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc815f9d0 0x7f0bc815dd50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.779+0000 7f0bcda00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0bc8144c00 0x7f0bc8142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.779+0000 7f0bcda00640 1 --2- 172.21.15.114:0/68348088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bc80ce0a0 0x7f0bc81434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.779+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 >> 172.21.15.114:0/68348088 conn(0x7f0bc80b1730 msgr2=0x7f0bc8071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.779+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 shutdown_connections 2024-03-06T07:29:18.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:18.780+0000 7f0bcda00640 1 -- 172.21.15.114:0/68348088 wait complete. 2024-03-06T07:29:18.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:29:21.251 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:29:21.251 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:21.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:21.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.419+0000 7feb25a00640 1 Processor -- start 2024-03-06T07:29:21.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.419+0000 7feb25a00640 1 -- start start 2024-03-06T07:29:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.420+0000 7feb25a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb2015f9d0 0x7feb2015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.420+0000 7feb25a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2006cef0 con 0x7feb20144c00 2024-03-06T07:29:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.420+0000 7feb25a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2006d1d0 con 0x7feb20070b10 2024-03-06T07:29:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.420+0000 7feb25a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2006d4b0 con 0x7feb2015f9d0 2024-03-06T07:29:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.421+0000 7feb17e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb2015f9d0 0x7feb2015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.421+0000 7feb1ea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7feb20144c00 0x7feb2006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43366/0 (socket says 172.21.15.114:43366) 2024-03-06T07:29:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.421+0000 7feb1ea00640 1 -- 172.21.15.114:0/3903874423 learned_addr learned my addr 172.21.15.114:0/3903874423 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.421+0000 7feb17e00640 1 -- 172.21.15.114:0/3903874423 >> v1:172.21.15.177:6789/0 conn(0x7feb20070b10 legacy=0x7feb20070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.421+0000 7feb17e00640 1 -- 172.21.15.114:0/3903874423 >> v1:172.21.15.114:6789/0 conn(0x7feb20144c00 legacy=0x7feb2006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.421+0000 7feb1e000640 1 -- 172.21.15.114:0/3903874423 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024504746 0 0) 0x7feb2006cef0 con 0x7feb20144c00 2024-03-06T07:29:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.421+0000 7feb17e00640 1 -- 172.21.15.114:0/3903874423 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2006d790 con 0x7feb2015f9d0 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.422+0000 7feb17e00640 1 --2- 172.21.15.114:0/3903874423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2015f9d0 0x7feb2015fe60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7feb080097a0 tx=0x7feb0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5580070f3daa1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.422+0000 7feb1e000640 1 -- 172.21.15.114:0/3903874423 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08035020 con 0x7feb2015f9d0 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.422+0000 7feb1e000640 1 -- 172.21.15.114:0/3903874423 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08031830 con 0x7feb2015f9d0 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.422+0000 7feb1e000640 1 -- 172.21.15.114:0/3903874423 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08031c00 con 0x7feb2015f9d0 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.423+0000 7feb25a00640 1 -- 172.21.15.114:0/3903874423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2015f9d0 msgr2=0x7feb2015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.423+0000 7feb25a00640 1 --2- 172.21.15.114:0/3903874423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2015f9d0 0x7feb2015fe60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7feb080097a0 tx=0x7feb0802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.423+0000 7feb25a00640 1 -- 172.21.15.114:0/3903874423 shutdown_connections 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.423+0000 7feb25a00640 1 --2- 172.21.15.114:0/3903874423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb2015f9d0 0x7feb2015fe60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.423+0000 7feb25a00640 1 -- 172.21.15.114:0/3903874423 >> 172.21.15.114:0/3903874423 conn(0x7feb200b1730 msgr2=0x7feb200b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.423+0000 7feb25a00640 1 -- 172.21.15.114:0/3903874423 shutdown_connections 2024-03-06T07:29:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.423+0000 7feb25a00640 1 -- 172.21.15.114:0/3903874423 wait complete. 2024-03-06T07:29:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.425+0000 7feb25a00640 1 Processor -- start 2024-03-06T07:29:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.425+0000 7feb25a00640 1 -- start start 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb25a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb20070b10 0x7feb2013c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb25a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb20144c00 0x7feb2015e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb25a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb2015f9d0 0x7feb2015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb25a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2006ccd0 con 0x7feb20144c00 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb25a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2006cb50 con 0x7feb20070b10 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb25a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2006df90 con 0x7feb2015f9d0 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb17e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb20144c00 0x7feb2015e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb17e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb20144c00 0x7feb2015e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47844/0 (socket says 172.21.15.114:47844) 2024-03-06T07:29:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb17e00640 1 -- 172.21.15.114:0/2528539908 learned_addr learned my addr 172.21.15.114:0/2528539908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb1ea00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb20070b10 0x7feb2013c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb17e00640 1 -- 172.21.15.114:0/2528539908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb2015f9d0 msgr2=0x7feb2015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb17e00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb2015f9d0 0x7feb2015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb17e00640 1 -- 172.21.15.114:0/2528539908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb20070b10 msgr2=0x7feb2013c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.426+0000 7feb17e00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb20070b10 0x7feb2013c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.427+0000 7feb17e00640 1 -- 172.21.15.114:0/2528539908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2013c820 con 0x7feb20144c00 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.427+0000 7feb17e00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb20144c00 0x7feb2015e1b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7feb08034040 tx=0x7feb080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.427+0000 7feb16a00640 1 -- 172.21.15.114:0/2528539908 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08004230 con 0x7feb20144c00 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.427+0000 7feb16a00640 1 -- 172.21.15.114:0/2528539908 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb080043f0 con 0x7feb20144c00 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.427+0000 7feb16a00640 1 -- 172.21.15.114:0/2528539908 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08003850 con 0x7feb20144c00 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.427+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2015ebb0 con 0x7feb20144c00 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.427+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2015f170 con 0x7feb20144c00 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.428+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2004f330 con 0x7feb20144c00 2024-03-06T07:29:21.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.429+0000 7feb16a00640 1 -- 172.21.15.114:0/2528539908 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb08038020 con 0x7feb20144c00 2024-03-06T07:29:21.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.430+0000 7feb16a00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feafc06b4d0 0x7feafc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.430+0000 7feb16a00640 1 -- 172.21.15.114:0/2528539908 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb080abd70 con 0x7feb20144c00 2024-03-06T07:29:21.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.430+0000 7feb1ea00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feafc06b4d0 0x7feafc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.431+0000 7feb1ea00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feafc06b4d0 0x7feafc06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feb10007fd0 tx=0x7feb10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.435+0000 7feb16a00640 1 -- 172.21.15.114:0/2528539908 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb08040070 con 0x7feb20144c00 2024-03-06T07:29:21.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.657+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb200721b0 con 0x7feb20144c00 2024-03-06T07:29:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.657+0000 7feb16a00640 1 -- 172.21.15.114:0/2528539908 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7feb0807abf0 con 0x7feb20144c00 2024-03-06T07:29:21.659 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:21.659 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feafc06b4d0 msgr2=0x7feafc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feafc06b4d0 0x7feafc06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feb10007fd0 tx=0x7feb10009040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb20144c00 msgr2=0x7feb2015e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb20144c00 0x7feb2015e1b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7feb08034040 tx=0x7feb080046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 shutdown_connections 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb2015f9d0 0x7feb2015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feafc06b4d0 0x7feafc06d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb20144c00 0x7feb2015e1b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 --2- 172.21.15.114:0/2528539908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb20070b10 0x7feb2013c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 >> 172.21.15.114:0/2528539908 conn(0x7feb200b1730 msgr2=0x7feb200b0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 shutdown_connections 2024-03-06T07:29:21.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:21.660+0000 7feb25a00640 1 -- 172.21.15.114:0/2528539908 wait complete. 2024-03-06T07:29:21.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:21.900 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:29:21.900 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:29:21.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.131+0000 7f75f7e00640 1 Processor -- start 2024-03-06T07:29:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.131+0000 7f75f7e00640 1 -- start start 2024-03-06T07:29:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.132+0000 7f75f7e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f75f8161580 0x7f75f80d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.132+0000 7f75f7e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f8161960 con 0x7f75f80d1e90 2024-03-06T07:29:22.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.132+0000 7f75f7e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f80cf6f0 con 0x7f75f80d2a70 2024-03-06T07:29:22.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.132+0000 7f75f7e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f80cf9d0 con 0x7f75f8161580 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.132+0000 7f75f6000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f75f80d1e90 0x7f75f80d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43378/0 (socket says 172.21.15.114:43378) 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.132+0000 7f75f6000640 1 -- 172.21.15.114:0/3337049388 learned_addr learned my addr 172.21.15.114:0/3337049388 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f6a00640 1 --2- 172.21.15.114:0/3337049388 >> v2:172.21.15.177:3300/0 conn(0x7f75f8161580 0x7f75f80d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927453661 0 0) 0x7f75f8161960 con 0x7f75f80d1e90 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc003520 con 0x7f75f80d1e90 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865213246 0 0) 0x7f75dc003520 con 0x7f75f80d1e90 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 >> v2:172.21.15.177:3300/0 conn(0x7f75f8161580 msgr2=0x7f75f80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 --2- 172.21.15.114:0/3337049388 >> v2:172.21.15.177:3300/0 conn(0x7f75f8161580 0x7f75f80d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 >> v1:172.21.15.177:6789/0 conn(0x7f75f80d2a70 legacy=0x7f75f80b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f80cfcb0 con 0x7f75f80d1e90 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f75e4001670 con 0x7f75f80d1e90 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 >> v1:172.21.15.114:6789/0 conn(0x7f75f80d1e90 legacy=0x7f75f80d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0039b0 0x7f75dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75dc004800 0x7f75dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.133+0000 7f75f5600640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75dc0051c0 0x7f75dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80cf9d0 con 0x7f75dc0051c0 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc00c060 con 0x7f75dc004800 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc00c260 con 0x7f75dc0039b0 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f7400640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75dc0051c0 0x7f75dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f6000640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75dc004800 0x7f75dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f6a00640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0039b0 0x7f75dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f7400640 1 -- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0039b0 msgr2=0x7f75dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f7400640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0039b0 0x7f75dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f7400640 1 -- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75dc004800 msgr2=0x7f75dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f7400640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75dc004800 0x7f75dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f7400640 1 -- 172.21.15.114:0/3337049388 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc003520 con 0x7f75dc0051c0 2024-03-06T07:29:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.134+0000 7f75f6a00640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0039b0 0x7f75dc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.135+0000 7f75f7400640 1 -- 172.21.15.114:0/3337049388 reap_dead start 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.135+0000 7f75f7400640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75dc0051c0 0x7f75dc00b9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f75f0001e20 tx=0x7f75dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd47ae20e50b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.135+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0001a70 con 0x7f75dc0051c0 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.135+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f0003b70 con 0x7f75dc0051c0 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.135+0000 7f75f5600640 1 -- 172.21.15.114:0/3337049388 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f00050d0 con 0x7f75dc0051c0 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.136+0000 7f75f7e00640 1 -- 172.21.15.114:0/3337049388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75dc0051c0 msgr2=0x7f75dc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.136+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75dc0051c0 0x7f75dc00b9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f75f0001e20 tx=0x7f75dc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.136+0000 7f75f7e00640 1 -- 172.21.15.114:0/3337049388 shutdown_connections 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.136+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3337049388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75dc0051c0 0x7f75dc00b9c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.136+0000 7f75f7e00640 1 -- 172.21.15.114:0/3337049388 >> 172.21.15.114:0/3337049388 conn(0x7f75f80d3d80 msgr2=0x7f75f80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.136+0000 7f75f7e00640 1 -- 172.21.15.114:0/3337049388 shutdown_connections 2024-03-06T07:29:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.136+0000 7f75f7e00640 1 -- 172.21.15.114:0/3337049388 wait complete. 2024-03-06T07:29:22.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.138+0000 7f75f7e00640 1 Processor -- start 2024-03-06T07:29:22.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.139+0000 7f75f7e00640 1 -- start start 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.140+0000 7f75f7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0051c0 0x7f75f814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.140+0000 7f75f7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75f80c7730 0x7f75f80b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.140+0000 7f75f7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75f80b84a0 0x7f75f81f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.140+0000 7f75f7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80cf3a0 con 0x7f75f80c7730 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.140+0000 7f75f7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80b39a0 con 0x7f75f80b84a0 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.140+0000 7f75f7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80d1950 con 0x7f75dc0051c0 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.140+0000 7f75f6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75f80c7730 0x7f75f80b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75f80c7730 0x7f75f80b7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47874/0 (socket says 172.21.15.114:47874) 2024-03-06T07:29:22.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 -- 172.21.15.114:0/3723011829 learned_addr learned my addr 172.21.15.114:0/3723011829 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6a00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0051c0 0x7f75f814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f7400640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75f80b84a0 0x7f75f81f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 -- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0051c0 msgr2=0x7f75f814d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0051c0 0x7f75f814d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 -- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75f80b84a0 msgr2=0x7f75f81f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75f80b84a0 0x7f75f81f7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 -- 172.21.15.114:0/3723011829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f814def0 con 0x7f75f80c7730 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f7400640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75f80b84a0 0x7f75f81f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.141+0000 7f75f6000640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75f80c7730 0x7f75f80b7f60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f75e400e860 tx=0x7f75e400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.142+0000 7f75eea00640 1 -- 172.21.15.114:0/3723011829 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e400ee50 con 0x7f75f80c7730 2024-03-06T07:29:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.142+0000 7f75eea00640 1 -- 172.21.15.114:0/3723011829 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4018070 con 0x7f75f80c7730 2024-03-06T07:29:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.142+0000 7f75eea00640 1 -- 172.21.15.114:0/3723011829 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e400cb10 con 0x7f75f80c7730 2024-03-06T07:29:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.142+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f81504a0 con 0x7f75f80c7730 2024-03-06T07:29:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.142+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f8150a60 con 0x7f75f80c7730 2024-03-06T07:29:22.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.143+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f80b88b0 con 0x7f75f80c7730 2024-03-06T07:29:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.145+0000 7f75eea00640 1 -- 172.21.15.114:0/3723011829 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75e400ccb0 con 0x7f75f80c7730 2024-03-06T07:29:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.146+0000 7f75eea00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75d806b500 0x7f75d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.146+0000 7f75eea00640 1 -- 172.21.15.114:0/3723011829 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75e4089d60 con 0x7f75f80c7730 2024-03-06T07:29:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.147+0000 7f75f6a00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75d806b500 0x7f75d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.147+0000 7f75f6a00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75d806b500 0x7f75d806d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009aa0 tx=0x7f75e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.154+0000 7f75eea00640 1 -- 172.21.15.114:0/3723011829 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75f80b88b0 con 0x7f75f80c7730 2024-03-06T07:29:22.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.391+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75f804f330 con 0x7f75f80c7730 2024-03-06T07:29:22.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.392+0000 7f75eea00640 1 -- 172.21.15.114:0/3723011829 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f75e4058b40 con 0x7f75f80c7730 2024-03-06T07:29:22.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:22.394 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:22.394 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:22.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.394+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75d806b500 msgr2=0x7f75d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.394+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75d806b500 0x7f75d806d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009aa0 tx=0x7f75e8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.394+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75f80c7730 msgr2=0x7f75f80b7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.394+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75f80c7730 0x7f75f80b7f60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f75e400e860 tx=0x7f75e400ed20 comp rx=0 tx=0).stop 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 shutdown_connections 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75f80b84a0 0x7f75f81f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75d806b500 0x7f75d806d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75f80c7730 0x7f75f80b7f60 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 --2- 172.21.15.114:0/3723011829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75dc0051c0 0x7f75f814d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 >> 172.21.15.114:0/3723011829 conn(0x7f75f80d3d80 msgr2=0x7f75f815e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 shutdown_connections 2024-03-06T07:29:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.395+0000 7f75f7e00640 1 -- 172.21.15.114:0/3723011829 wait complete. 2024-03-06T07:29:22.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:22.622 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:29:22.623 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:29:22.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.819+0000 7fb1cac00640 1 Processor -- start 2024-03-06T07:29:22.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.819+0000 7fb1cac00640 1 -- start start 2024-03-06T07:29:22.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1cac00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c415e070 0x7fb1c4066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1cac00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb1c40cd860 0x7fb1c406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1cac00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c406eb10 con 0x7fb1c406b500 2024-03-06T07:29:22.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1cac00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c406b970 con 0x7fb1c40cd860 2024-03-06T07:29:22.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1cac00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40b5010 con 0x7fb1c415e070 2024-03-06T07:29:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1c3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c415e070 0x7fb1c4066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1c3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c415e070 0x7fb1c4066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:60360/0 (socket says 172.21.15.114:60360) 2024-03-06T07:29:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1c3e00640 1 -- 172.21.15.114:0/885040593 learned_addr learned my addr 172.21.15.114:0/885040593 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.821+0000 7fb1bbe00640 1 --2- 172.21.15.114:0/885040593 >> v2:172.21.15.177:3300/0 conn(0x7fb1c40cd860 0x7fb1c406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.822+0000 7fb1c3e00640 1 -- 172.21.15.114:0/885040593 >> v2:172.21.15.177:3300/0 conn(0x7fb1c40cd860 msgr2=0x7fb1c406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.822+0000 7fb1c3e00640 1 --2- 172.21.15.114:0/885040593 >> v2:172.21.15.177:3300/0 conn(0x7fb1c40cd860 0x7fb1c406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.822+0000 7fb1c3e00640 1 -- 172.21.15.114:0/885040593 >> v1:172.21.15.177:6789/0 conn(0x7fb1c406b500 legacy=0x7fb1c4161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.822+0000 7fb1c3e00640 1 -- 172.21.15.114:0/885040593 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c40b52f0 con 0x7fb1c415e070 2024-03-06T07:29:22.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.823+0000 7fb1c3e00640 1 --2- 172.21.15.114:0/885040593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c415e070 0x7fb1c4066d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009870 tx=0x7fb1b402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e98f73d1ebc5151 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.824+0000 7fb1c3400640 1 -- 172.21.15.114:0/885040593 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4034070 con 0x7fb1c415e070 2024-03-06T07:29:22.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.824+0000 7fb1c3400640 1 -- 172.21.15.114:0/885040593 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1b4030840 con 0x7fb1c415e070 2024-03-06T07:29:22.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.824+0000 7fb1c3400640 1 -- 172.21.15.114:0/885040593 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4030c10 con 0x7fb1c415e070 2024-03-06T07:29:22.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.824+0000 7fb1cac00640 1 -- 172.21.15.114:0/885040593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c415e070 msgr2=0x7fb1c4066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.824+0000 7fb1cac00640 1 --2- 172.21.15.114:0/885040593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c415e070 0x7fb1c4066d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009870 tx=0x7fb1b402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:22.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.825+0000 7fb1cac00640 1 -- 172.21.15.114:0/885040593 shutdown_connections 2024-03-06T07:29:22.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.825+0000 7fb1cac00640 1 --2- 172.21.15.114:0/885040593 >> v2:172.21.15.177:3300/0 conn(0x7fb1c40cd860 0x7fb1c406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.825+0000 7fb1cac00640 1 --2- 172.21.15.114:0/885040593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c415e070 0x7fb1c4066d00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.825+0000 7fb1cac00640 1 -- 172.21.15.114:0/885040593 >> 172.21.15.114:0/885040593 conn(0x7fb1c40b1e10 msgr2=0x7fb1c40b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.825+0000 7fb1cac00640 1 -- 172.21.15.114:0/885040593 shutdown_connections 2024-03-06T07:29:22.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.825+0000 7fb1cac00640 1 -- 172.21.15.114:0/885040593 wait complete. 2024-03-06T07:29:22.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.827+0000 7fb1cac00640 1 Processor -- start 2024-03-06T07:29:22.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.827+0000 7fb1cac00640 1 -- start start 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.828+0000 7fb1cac00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c406b500 0x7fb1c4099010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.828+0000 7fb1cac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c40cd860 0x7fb1c40cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.828+0000 7fb1cac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c415e070 0x7fb1c40d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.828+0000 7fb1cac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40b5b90 con 0x7fb1c415e070 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.828+0000 7fb1cac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40b58b0 con 0x7fb1c40cd860 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.828+0000 7fb1cac00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c415e450 con 0x7fb1c406b500 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c415e070 0x7fb1c40d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c415e070 0x7fb1c40d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47888/0 (socket says 172.21.15.114:47888) 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 -- 172.21.15.114:0/4071234594 learned_addr learned my addr 172.21.15.114:0/4071234594 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c3e00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c406b500 0x7fb1c4099010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1bbe00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c40cd860 0x7fb1c40cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 -- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c406b500 msgr2=0x7fb1c4099010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c406b500 0x7fb1c4099010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 -- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c40cd860 msgr2=0x7fb1c40cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c40cd860 0x7fb1c40cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1c8c00640 1 -- 172.21.15.114:0/4071234594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c40ce990 con 0x7fb1c415e070 2024-03-06T07:29:22.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.829+0000 7fb1bbe00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c40cd860 0x7fb1c40cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.830+0000 7fb1c8c00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c415e070 0x7fb1c40d0370 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c6b0 tx=0x7fb1bc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.830+0000 7fb1c0c00640 1 -- 172.21.15.114:0/4071234594 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc003b70 con 0x7fb1c415e070 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.830+0000 7fb1c0c00640 1 -- 172.21.15.114:0/4071234594 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc003d30 con 0x7fb1c415e070 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.830+0000 7fb1c0c00640 1 -- 172.21.15.114:0/4071234594 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc005310 con 0x7fb1c415e070 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.830+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c40cec70 con 0x7fb1c415e070 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.830+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c40cf230 con 0x7fb1c415e070 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.831+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c404f330 con 0x7fb1c415e070 2024-03-06T07:29:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.832+0000 7fb1c0c00640 1 -- 172.21.15.114:0/4071234594 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1bc0054b0 con 0x7fb1c415e070 2024-03-06T07:29:22.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.833+0000 7fb1c0c00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a006b4d0 0x7fb1a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.833+0000 7fb1c0c00640 1 -- 172.21.15.114:0/4071234594 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1bc08a000 con 0x7fb1c415e070 2024-03-06T07:29:22.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.833+0000 7fb1c3e00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a006b4d0 0x7fb1a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.834+0000 7fb1c3e00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a006b4d0 0x7fb1a006d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4036000 tx=0x7fb1b403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:22.838+0000 7fb1c0c00640 1 -- 172.21.15.114:0/4071234594 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1bc058f10 con 0x7fb1c415e070 2024-03-06T07:29:23.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.074+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1c406f0b0 con 0x7fb1c415e070 2024-03-06T07:29:23.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.074+0000 7fb1c0c00640 1 -- 172.21.15.114:0/4071234594 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fb1bc058810 con 0x7fb1c415e070 2024-03-06T07:29:23.076 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:23.076 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:23.076 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a006b4d0 msgr2=0x7fb1a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a006b4d0 0x7fb1a006d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4036000 tx=0x7fb1b403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c415e070 msgr2=0x7fb1c40d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c415e070 0x7fb1c40d0370 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c6b0 tx=0x7fb1bc00cb70 comp rx=0 tx=0).stop 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 shutdown_connections 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a006b4d0 0x7fb1a006d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c415e070 0x7fb1c40d0370 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c40cd860 0x7fb1c40cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 --2- 172.21.15.114:0/4071234594 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c406b500 0x7fb1c4099010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 >> 172.21.15.114:0/4071234594 conn(0x7fb1c40b1e10 msgr2=0x7fb1c40ad150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 shutdown_connections 2024-03-06T07:29:23.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.077+0000 7fb1cac00640 1 -- 172.21.15.114:0/4071234594 wait complete. 2024-03-06T07:29:23.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:29:23.315 INFO:teuthology.orchestra.run.smithi114.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:29:23.316 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:29:23.316 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:29:23.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.413+0000 7f8feeb30f40 1 Processor -- start 2024-03-06T07:29:23.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.414+0000 7f8feeb30f40 1 -- start start 2024-03-06T07:29:23.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.414+0000 7f8feeb30f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65990 0x55bc67c3bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8feeb30f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67b65d70 0x55bc67c22840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8feeb30f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67c237e0 0x55bc67b96d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8feeb30f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc67bafba0 con 0x55bc67c237e0 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8feeb30f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc67b0bfd0 con 0x55bc67b65990 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8feeb30f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc67b64d10 con 0x55bc67b65d70 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67c237e0 0x55bc67b96d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67c237e0 0x55bc67b96d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47908/0 (socket says 172.21.15.114:47908) 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 -- 172.21.15.114:0/2332862898 learned_addr learned my addr 172.21.15.114:0/2332862898 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8febe00640 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65990 0x55bc67c3bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67b65d70 0x55bc67c22840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 -- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67b65d70 msgr2=0x55bc67c22840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67b65d70 0x55bc67c22840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 -- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65990 msgr2=0x55bc67c3bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65990 0x55bc67c3bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.415+0000 7f8fec800640 1 -- 172.21.15.114:0/2332862898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc67b97a00 con 0x55bc67c237e0 2024-03-06T07:29:23.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.416+0000 7f8fec800640 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67c237e0 0x55bc67b96d50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c4e0 tx=0x7f8fe400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e6d34df0dbb265 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.416+0000 7f8feaa00640 1 -- 172.21.15.114:0/2332862898 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40116b0 con 0x55bc67c237e0 2024-03-06T07:29:23.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.416+0000 7f8feaa00640 1 -- 172.21.15.114:0/2332862898 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4017070 con 0x55bc67c237e0 2024-03-06T07:29:23.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.416+0000 7f8feaa00640 1 -- 172.21.15.114:0/2332862898 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4011a90 con 0x55bc67c237e0 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 -- 172.21.15.114:0/2332862898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67c237e0 msgr2=0x55bc67b96d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67c237e0 0x55bc67b96d50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c4e0 tx=0x7f8fe400c9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 -- 172.21.15.114:0/2332862898 shutdown_connections 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67c237e0 0x55bc67b96d50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67b65d70 0x55bc67c22840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/2332862898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65990 0x55bc67c3bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 -- 172.21.15.114:0/2332862898 >> 172.21.15.114:0/2332862898 conn(0x55bc67b94c50 msgr2=0x55bc67c22130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 -- 172.21.15.114:0/2332862898 shutdown_connections 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 -- 172.21.15.114:0/2332862898 wait complete. 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.417+0000 7f8feeb30f40 1 Processor -- start 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.418+0000 7f8feeb30f40 1 -- start start 2024-03-06T07:29:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.418+0000 7f8feeb30f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67b65990 0x55bc67c22020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8feeb30f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65d70 0x55bc67b972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8feeb30f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67c237e0 0x55bc67ba7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8feeb30f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc67b97810 con 0x55bc67b65990 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8feeb30f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc67c1d830 con 0x55bc67b65d70 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8feeb30f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc67c226b0 con 0x55bc67c237e0 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67b65990 0x55bc67c22020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67b65990 0x55bc67c22020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47914/0 (socket says 172.21.15.114:47914) 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 -- 172.21.15.114:0/218301221 learned_addr learned my addr 172.21.15.114:0/218301221 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65d70 0x55bc67b972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 -- 172.21.15.114:0/218301221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67c237e0 msgr2=0x55bc67ba7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8fec800640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67c237e0 0x55bc67ba7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55bc67c237e0 0x55bc67ba7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 -- 172.21.15.114:0/218301221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65d70 msgr2=0x55bc67b972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc67b65d70 0x55bc67b972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.419+0000 7f8febe00640 1 -- 172.21.15.114:0/218301221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc67c1dad0 con 0x55bc67b65990 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.420+0000 7f8febe00640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67b65990 0x55bc67c22020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00ded0 tx=0x7f8fdc00b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.420+0000 7f8fe9600640 1 -- 172.21.15.114:0/218301221 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc014020 con 0x55bc67b65990 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.420+0000 7f8fe9600640 1 -- 172.21.15.114:0/218301221 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc00be10 con 0x55bc67b65990 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.420+0000 7f8fe9600640 1 -- 172.21.15.114:0/218301221 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc008030 con 0x55bc67b65990 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.420+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bc67c1ddb0 con 0x55bc67b65990 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.420+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc67ba81b0 con 0x55bc67b65990 2024-03-06T07:29:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.423+0000 7f8fe9600640 1 -- 172.21.15.114:0/218301221 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fdc017020 con 0x55bc67b65990 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.424+0000 7f8fe9600640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc8069ee0 0x7f8fc806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.424+0000 7f8fe9600640 1 -- 172.21.15.114:0/218301221 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fdc088e50 con 0x55bc67b65990 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.424+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc8069ee0 0x7f8fc806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.424+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x55bc67c1ee00 0x55bc67c1f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.424+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x55bc67c37030 con 0x55bc67c1ee00 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.425+0000 7f8fec800640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x55bc67c1ee00 0x55bc67c1f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.425+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc8069ee0 0x7f8fc806c3a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4054000 tx=0x7f8fd404e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.425+0000 7f8fec800640 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x55bc67c1ee00 0x55bc67c1f240 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.425+0000 7f8fec800640 1 -- 172.21.15.114:0/218301221 <== osd.3 v2:172.21.15.114:6818/735520392 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f8fe40116b0 con 0x55bc67c1ee00 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x55bc67c1ee00 msgr2=0x55bc67c1f240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x55bc67c1ee00 0x55bc67c1f240 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc8069ee0 msgr2=0x7f8fc806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc8069ee0 0x7f8fc806c3a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4054000 tx=0x7f8fd404e000 comp rx=0 tx=0).stop 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67b65990 msgr2=0x55bc67c22020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 --2- 172.21.15.114:0/218301221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bc67b65990 0x55bc67c22020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00ded0 tx=0x7f8fdc00b5e0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8fec800640 1 -- 172.21.15.114:0/218301221 reap_dead start 2024-03-06T07:29:23.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 shutdown_connections 2024-03-06T07:29:23.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 >> 172.21.15.114:0/218301221 conn(0x55bc67b94c50 msgr2=0x55bc67c39040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 shutdown_connections 2024-03-06T07:29:23.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.426+0000 7f8feeb30f40 1 -- 172.21.15.114:0/218301221 wait complete. 2024-03-06T07:29:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:23.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:29:23.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.622+0000 7fe8e1200640 1 Processor -- start 2024-03-06T07:29:23.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.622+0000 7fe8e1200640 1 -- start start 2024-03-06T07:29:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.624+0000 7fe8e1200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc0b54e0 0x7fe8dc0ce070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.624+0000 7fe8e1200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc071280 con 0x7fe8dc1566a0 2024-03-06T07:29:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.624+0000 7fe8e1200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc072df0 con 0x7fe8dc071d20 2024-03-06T07:29:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.624+0000 7fe8e1200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0ccbd0 con 0x7fe8dc0b54e0 2024-03-06T07:29:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.624+0000 7fe8d9e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe8dc1566a0 0x7fe8dc15ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43404/0 (socket says 172.21.15.114:43404) 2024-03-06T07:29:23.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.624+0000 7fe8d9e00640 1 -- 172.21.15.114:0/878100679 learned_addr learned my addr 172.21.15.114:0/878100679 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.624+0000 7fe8da800640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc0b54e0 0x7fe8dc0ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640011532 0 0) 0x7fe8dc071280 con 0x7fe8dc1566a0 2024-03-06T07:29:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c0003520 con 0x7fe8dc1566a0 2024-03-06T07:29:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734641975 0 0) 0x7fe8c0003520 con 0x7fe8dc1566a0 2024-03-06T07:29:23.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc0b54e0 msgr2=0x7fe8dc0ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc0b54e0 0x7fe8dc0ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 >> v1:172.21.15.177:6789/0 conn(0x7fe8dc071d20 legacy=0x7fe8dc0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc0cc8f0 con 0x7fe8dc1566a0 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fe8cc001670 con 0x7fe8dc1566a0 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 >> v1:172.21.15.114:6789/0 conn(0x7fe8dc1566a0 legacy=0x7fe8dc15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c00039e0 0x7fe8c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc0b54e0 0x7fe8c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c0004d40 0x7fe8c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0ccbd0 con 0x7fe8c0004d40 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c000bad0 con 0x7fe8c00039e0 2024-03-06T07:29:23.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.625+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c000bcd0 con 0x7fe8dc0b54e0 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.626+0000 7fe8da800640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c0004d40 0x7fe8c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.626+0000 7fe8d9e00640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c00039e0 0x7fe8c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.626+0000 7fe8da800640 1 -- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc0b54e0 msgr2=0x7fe8c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.626+0000 7fe8da800640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc0b54e0 0x7fe8c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.626+0000 7fe8da800640 1 -- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c00039e0 msgr2=0x7fe8c0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.626+0000 7fe8da800640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c00039e0 0x7fe8c0003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.626+0000 7fe8da800640 1 -- 172.21.15.114:0/878100679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c0003520 con 0x7fe8c0004d40 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.627+0000 7fe8da800640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c0004d40 0x7fe8c00051c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc070a80 tx=0x7fe8c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8d7cc87dc15b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.627+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4039070 con 0x7fe8c0004d40 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.627+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d402cb90 con 0x7fe8c0004d40 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.627+0000 7fe8d8a00640 1 -- 172.21.15.114:0/878100679 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4009c10 con 0x7fe8c0004d40 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.628+0000 7fe8e1200640 1 -- 172.21.15.114:0/878100679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c0004d40 msgr2=0x7fe8c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.628+0000 7fe8e1200640 1 --2- 172.21.15.114:0/878100679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c0004d40 0x7fe8c00051c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc070a80 tx=0x7fe8c0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.628+0000 7fe8da800640 1 -- 172.21.15.114:0/878100679 reap_dead start 2024-03-06T07:29:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.628+0000 7fe8e1200640 1 -- 172.21.15.114:0/878100679 shutdown_connections 2024-03-06T07:29:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.628+0000 7fe8e1200640 1 -- 172.21.15.114:0/878100679 >> 172.21.15.114:0/878100679 conn(0x7fe8dc0cfa30 msgr2=0x7fe8dc0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.628+0000 7fe8e1200640 1 -- 172.21.15.114:0/878100679 shutdown_connections 2024-03-06T07:29:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.628+0000 7fe8e1200640 1 -- 172.21.15.114:0/878100679 wait complete. 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.629+0000 7fe8e1200640 1 Processor -- start 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.630+0000 7fe8e1200640 1 -- start start 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.630+0000 7fe8e1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c0004d40 0x7fe8dc14c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.630+0000 7fe8e1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8dc141400 0x7fe8dc14e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.630+0000 7fe8e1200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc15aeb0 0x7fe8dc15b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.630+0000 7fe8e1200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0cc4a0 con 0x7fe8dc141400 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.630+0000 7fe8e1200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc071800 con 0x7fe8c0004d40 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.630+0000 7fe8e1200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0cc620 con 0x7fe8dc15aeb0 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8dc141400 0x7fe8dc14e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8dc141400 0x7fe8dc14e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47930/0 (socket says 172.21.15.114:47930) 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 -- 172.21.15.114:0/269884710 learned_addr learned my addr 172.21.15.114:0/269884710 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9e00640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c0004d40 0x7fe8dc14c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 -- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc15aeb0 msgr2=0x7fe8dc15b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8da800640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc15aeb0 0x7fe8dc15b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc15aeb0 0x7fe8dc15b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 -- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c0004d40 msgr2=0x7fe8dc14c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c0004d40 0x7fe8dc14c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9400640 1 -- 172.21.15.114:0/269884710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc14cb90 con 0x7fe8dc141400 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.631+0000 7fe8d9e00640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c0004d40 0x7fe8dc14c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.632+0000 7fe8d9400640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8dc141400 0x7fe8dc14e590 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800c450 tx=0x7fe8c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.632+0000 7fe8d2000640 1 -- 172.21.15.114:0/269884710 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c80052d0 con 0x7fe8dc141400 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.632+0000 7fe8d2000640 1 -- 172.21.15.114:0/269884710 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8005470 con 0x7fe8dc141400 2024-03-06T07:29:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.632+0000 7fe8d2000640 1 -- 172.21.15.114:0/269884710 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8012870 con 0x7fe8dc141400 2024-03-06T07:29:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.632+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc14ce70 con 0x7fe8dc141400 2024-03-06T07:29:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.632+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc14d430 con 0x7fe8dc141400 2024-03-06T07:29:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.633+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc04f330 con 0x7fe8dc141400 2024-03-06T07:29:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.635+0000 7fe8d2000640 1 -- 172.21.15.114:0/269884710 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8c8012a10 con 0x7fe8dc141400 2024-03-06T07:29:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.636+0000 7fe8d2000640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8bc06b500 0x7fe8bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.636+0000 7fe8d2000640 1 -- 172.21.15.114:0/269884710 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8c8089df0 con 0x7fe8dc141400 2024-03-06T07:29:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.636+0000 7fe8d9e00640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8bc06b500 0x7fe8bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.637+0000 7fe8d9e00640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8bc06b500 0x7fe8bc06d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0017e0 tx=0x7fe8cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.641+0000 7fe8d2000640 1 -- 172.21.15.114:0/269884710 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8c8058c60 con 0x7fe8dc141400 2024-03-06T07:29:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.903+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fe8dc155270 con 0x7fe8dc141400 2024-03-06T07:29:23.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.904+0000 7fe8d2000640 1 -- 172.21.15.114:0/269884710 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1663 (secure 0 0 0) 0x7fe8c8058560 con 0x7fe8dc141400 2024-03-06T07:29:23.905 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772855111680, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 239001600, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 239001600, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00030914941453374922, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "num_osds": 8, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_osds": 8, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "stats_by_class": { 2024-03-06T07:29:23.906 INFO:teuthology.orchestra.run.smithi114.stdout: "ssd": { 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772855111680, 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 239001600, 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 239001600, 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00030914941453374922 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "name": ".mgr", 2024-03-06T07:29:23.907 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 1, 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 590368, 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 2, 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 1160, 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 1187840, 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 1.6179528756765649e-06, 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367080570880 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_metadata", 2024-03-06T07:29:23.908 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 4, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 7564641, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 24, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 14832, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 15187968, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 2.0687084543169476e-05, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367080570880 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_data", 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5, 2024-03-06T07:29:23.909 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 0, 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 1000, 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 0, 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 0, 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 0, 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367080570880 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:29:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8bc06b500 msgr2=0x7fe8bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8bc06b500 0x7fe8bc06d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0017e0 tx=0x7fe8cc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8dc141400 msgr2=0x7fe8dc14e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8dc141400 0x7fe8dc14e590 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800c450 tx=0x7fe8c800c910 comp rx=0 tx=0).stop 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 shutdown_connections 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8dc15aeb0 0x7fe8dc15b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8bc06b500 0x7fe8bc06d9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8dc141400 0x7fe8dc14e590 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 --2- 172.21.15.114:0/269884710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c0004d40 0x7fe8dc14c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.906+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 >> 172.21.15.114:0/269884710 conn(0x7fe8dc0cfa30 msgr2=0x7fe8dc0cef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.907+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 shutdown_connections 2024-03-06T07:29:23.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:23.907+0000 7fe8e1200640 1 -- 172.21.15.114:0/269884710 wait complete. 2024-03-06T07:29:23.932 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-06T07:29:28.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:29:29.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.143+0000 7f060e200640 1 Processor -- start 2024-03-06T07:29:29.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.143+0000 7f060e200640 1 -- start start 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.145+0000 7f060e200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 0x7f06080ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.145+0000 7f060e200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f06080d1870 0x7f06080cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.145+0000 7f060e200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06080d0e20 con 0x7f060815e390 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.145+0000 7f060e200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06080cc7e0 con 0x7f06080d1870 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.145+0000 7f060e200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06080d1100 con 0x7f060806ff60 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.146+0000 7f0607400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f060815e390 0x7f060806f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49068/0 (socket says 172.21.15.114:49068) 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.146+0000 7f0607400640 1 -- 172.21.15.114:0/617047652 learned_addr learned my addr 172.21.15.114:0/617047652 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.146+0000 7f0607e00640 1 --2- 172.21.15.114:0/617047652 >> v2:172.21.15.177:3300/0 conn(0x7f06080d1870 0x7f06080cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.146+0000 7f0606a00640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 0x7f06080ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.146+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558716963 0 0) 0x7f06080d0e20 con 0x7f060815e390 2024-03-06T07:29:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.146+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ec003520 con 0x7f060815e390 2024-03-06T07:29:29.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3870840113 0 0) 0x7f05ec003520 con 0x7f060815e390 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 msgr2=0x7f06080ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 0x7f06080ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 >> v2:172.21.15.177:3300/0 conn(0x7f06080d1870 msgr2=0x7f06080cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 --2- 172.21.15.114:0/617047652 >> v2:172.21.15.177:3300/0 conn(0x7f06080d1870 0x7f06080cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06080cc470 con 0x7f060815e390 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f05f8001670 con 0x7f060815e390 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 >> v1:172.21.15.114:6789/0 conn(0x7f060815e390 legacy=0x7f060806f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 0x7f05ec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05ec003f30 0x7f05ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05ec004d40 0x7f05ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06080cc7e0 con 0x7f05ec003f30 2024-03-06T07:29:29.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06080d1100 con 0x7f05ec004d40 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ec00b9c0 con 0x7f060806ff60 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606a00640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05ec003f30 0x7f05ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0607400640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 0x7f05ec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606a00640 1 -- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 msgr2=0x7f05ec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606a00640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 0x7f05ec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606a00640 1 -- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05ec004d40 msgr2=0x7f05ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606a00640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05ec004d40 0x7f05ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0606a00640 1 -- 172.21.15.114:0/617047652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec003520 con 0x7f05ec003f30 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.147+0000 7f0607400640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f060806ff60 0x7f05ec0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.149+0000 7f0606a00640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05ec003f30 0x7f05ec004800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009ac0 tx=0x7f05ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151be503e0419676 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.149+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4039070 con 0x7f05ec003f30 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.149+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f402c6b0 con 0x7f05ec003f30 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.149+0000 7f0606000640 1 -- 172.21.15.114:0/617047652 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4009690 con 0x7f05ec003f30 2024-03-06T07:29:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.149+0000 7f060e200640 1 -- 172.21.15.114:0/617047652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05ec003f30 msgr2=0x7f05ec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.149+0000 7f060e200640 1 --2- 172.21.15.114:0/617047652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05ec003f30 0x7f05ec004800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009ac0 tx=0x7f05ec005830 comp rx=0 tx=0).stop 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.149+0000 7f0607e00640 1 -- 172.21.15.114:0/617047652 reap_dead start 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.150+0000 7f060e200640 1 -- 172.21.15.114:0/617047652 shutdown_connections 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.150+0000 7f060e200640 1 -- 172.21.15.114:0/617047652 >> 172.21.15.114:0/617047652 conn(0x7f06080b3870 msgr2=0x7f06080b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.150+0000 7f060e200640 1 -- 172.21.15.114:0/617047652 shutdown_connections 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.150+0000 7f060e200640 1 -- 172.21.15.114:0/617047652 wait complete. 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.151+0000 7f060e200640 1 Processor -- start 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.151+0000 7f060e200640 1 -- start start 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.152+0000 7f060e200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f05ec003f30 0x7f06081551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.152+0000 7f060e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06080b5370 0x7f06081601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.152+0000 7f060e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06080b6d40 0x7f06080b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.152+0000 7f060e200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060806f1d0 con 0x7f06080b6d40 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.152+0000 7f060e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060806f050 con 0x7f06080b5370 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.152+0000 7f060e200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060806f350 con 0x7f05ec003f30 2024-03-06T07:29:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06080b6d40 0x7f06080b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06080b6d40 0x7f06080b71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36032/0 (socket says 172.21.15.114:36032) 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 -- 172.21.15.114:0/242237096 learned_addr learned my addr 172.21.15.114:0/242237096 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607400640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f05ec003f30 0x7f06081551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0606a00640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06080b5370 0x7f06081601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 -- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f05ec003f30 msgr2=0x7f06081551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f05ec003f30 0x7f06081551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 -- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06080b5370 msgr2=0x7f06081601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06080b5370 0x7f06081601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 -- 172.21.15.114:0/242237096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608155720 con 0x7f06080b6d40 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607400640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f05ec003f30 0x7f06081551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.153+0000 7f0607e00640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06080b6d40 0x7f06080b71c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0608071eb0 tx=0x7f060000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.154+0000 7f05ff400640 1 -- 172.21.15.114:0/242237096 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000ba80 con 0x7f06080b6d40 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.154+0000 7f05ff400640 1 -- 172.21.15.114:0/242237096 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060001a070 con 0x7f06080b6d40 2024-03-06T07:29:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.154+0000 7f05ff400640 1 -- 172.21.15.114:0/242237096 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000be40 con 0x7f06080b6d40 2024-03-06T07:29:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.154+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0608160a00 con 0x7f06080b6d40 2024-03-06T07:29:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.154+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0608160fc0 con 0x7f06080b6d40 2024-03-06T07:29:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.155+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060804f330 con 0x7f06080b6d40 2024-03-06T07:29:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.156+0000 7f05ff400640 1 -- 172.21.15.114:0/242237096 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0600005d10 con 0x7f06080b6d40 2024-03-06T07:29:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.157+0000 7f05ff400640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f05e806b500 0x7f05e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.157+0000 7f05ff400640 1 -- 172.21.15.114:0/242237096 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f060008a1c0 con 0x7f06080b6d40 2024-03-06T07:29:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.157+0000 7f0607400640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f05e806b500 0x7f05e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.158+0000 7f0607400640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f05e806b500 0x7f05e806d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05f80018e0 tx=0x7f05f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.162+0000 7f05ff400640 1 -- 172.21.15.114:0/242237096 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0608155720 con 0x7f06080b6d40 2024-03-06T07:29:29.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.431+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f06080cdc60 con 0x7f06080b6d40 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.432+0000 7f05ff400640 1 -- 172.21.15.114:0/242237096 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1658 (secure 0 0 0) 0x7f0600059040 con 0x7f06080b6d40 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772840587264, 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 253526016, 2024-03-06T07:29:29.434 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 253526016, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00032793681020848453, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "num_osds": 8, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_osds": 8, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "stats_by_class": { 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "ssd": { 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772840587264, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 253526016, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 253526016, 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00032793681020848453 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:29.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "name": ".mgr", 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 1, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 590368, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 2, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 1160, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 1187840, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 1.6179402564375778e-06, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367083421696 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:29.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_metadata", 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 4, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 1274376, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 23, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 2552, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 2613248, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 3.5594616747403052e-06, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367083421696 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_data", 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5, 2024-03-06T07:29:29.437 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 0, 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 0, 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 0, 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 0, 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 0, 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367083421696 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:29:29.438 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.434+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f05e806b500 msgr2=0x7f05e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.434+0000 7f060e200640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f05e806b500 0x7f05e806d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05f80018e0 tx=0x7f05f8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06080b6d40 msgr2=0x7f06080b71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06080b6d40 0x7f06080b71c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0608071eb0 tx=0x7f060000b480 comp rx=0 tx=0).stop 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 shutdown_connections 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f05e806b500 0x7f05e806d9c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06080b6d40 0x7f06080b71c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06080b5370 0x7f06081601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 --2- 172.21.15.114:0/242237096 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f05ec003f30 0x7f06081551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 >> 172.21.15.114:0/242237096 conn(0x7f06080b3870 msgr2=0x7f060806f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 shutdown_connections 2024-03-06T07:29:29.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:29.435+0000 7f060e200640 1 -- 172.21.15.114:0/242237096 wait complete. 2024-03-06T07:29:29.453 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:29:29.454 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:29.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:29.549 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:29.665 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:29.665 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:29:29.665 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:29.717 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:29:29.717 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:29.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.012+0000 7fd40de00640 1 Processor -- start 2024-03-06T07:29:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.012+0000 7fd40de00640 1 -- start start 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.014+0000 7fd40de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd40806ec00 0x7fd4080ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.014+0000 7fd40de00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40815b090 con 0x7fd4080b4680 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.014+0000 7fd40de00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080b2a30 con 0x7fd40815acb0 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.014+0000 7fd40de00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4080b4360 con 0x7fd40806ec00 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.014+0000 7fd406000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd4080b4680 0x7fd40806e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49074/0 (socket says 172.21.15.114:49074) 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.014+0000 7fd406000640 1 -- 172.21.15.114:0/4266167089 learned_addr learned my addr 172.21.15.114:0/4266167089 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:30.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.014+0000 7fd407400640 1 --2- 172.21.15.114:0/4266167089 >> v2:172.21.15.114:3300/0 conn(0x7fd40806ec00 0x7fd4080ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.015+0000 7fd405600640 1 -- 172.21.15.114:0/4266167089 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035642002 0 0) 0x7fd40815b090 con 0x7fd4080b4680 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.015+0000 7fd407400640 1 -- 172.21.15.114:0/4266167089 >> v1:172.21.15.177:6789/0 conn(0x7fd40815acb0 legacy=0x7fd4080b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.015+0000 7fd407400640 1 -- 172.21.15.114:0/4266167089 >> v1:172.21.15.114:6789/0 conn(0x7fd4080b4680 legacy=0x7fd40806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.015+0000 7fd407400640 1 -- 172.21.15.114:0/4266167089 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4080b2cf0 con 0x7fd40806ec00 2024-03-06T07:29:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.016+0000 7fd407400640 1 --2- 172.21.15.114:0/4266167089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd4080ccfb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd400009d90 tx=0x7fd40002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a012fc5ffca4837d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.016+0000 7fd405600640 1 -- 172.21.15.114:0/4266167089 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400039070 con 0x7fd40806ec00 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.016+0000 7fd405600640 1 -- 172.21.15.114:0/4266167089 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40002ccf0 con 0x7fd40806ec00 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.016+0000 7fd405600640 1 -- 172.21.15.114:0/4266167089 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400034660 con 0x7fd40806ec00 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.017+0000 7fd40de00640 1 -- 172.21.15.114:0/4266167089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 msgr2=0x7fd4080ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.017+0000 7fd40de00640 1 --2- 172.21.15.114:0/4266167089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd4080ccfb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd400009d90 tx=0x7fd40002bf40 comp rx=0 tx=0).stop 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.017+0000 7fd40de00640 1 -- 172.21.15.114:0/4266167089 shutdown_connections 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.017+0000 7fd40de00640 1 --2- 172.21.15.114:0/4266167089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd4080ccfb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.017+0000 7fd40de00640 1 -- 172.21.15.114:0/4266167089 >> 172.21.15.114:0/4266167089 conn(0x7fd4080b1730 msgr2=0x7fd4080b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.017+0000 7fd40de00640 1 -- 172.21.15.114:0/4266167089 shutdown_connections 2024-03-06T07:29:30.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.017+0000 7fd40de00640 1 -- 172.21.15.114:0/4266167089 wait complete. 2024-03-06T07:29:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.020+0000 7fd40de00640 1 Processor -- start 2024-03-06T07:29:30.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.020+0000 7fd40de00640 1 -- start start 2024-03-06T07:29:30.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd40de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd40814acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd40de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4080b4680 0x7fd4080ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd40de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd40815acb0 0x7fd4080ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd40de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b4010 con 0x7fd40806ec00 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd40de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b32b0 con 0x7fd4080b4680 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd40de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd408072340 con 0x7fd40815acb0 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd406a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd40814acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd406a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd40814acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36042/0 (socket says 172.21.15.114:36042) 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.021+0000 7fd406a00640 1 -- 172.21.15.114:0/3194622794 learned_addr learned my addr 172.21.15.114:0/3194622794 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd406000640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4080b4680 0x7fd4080ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd407400640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd40815acb0 0x7fd4080ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd406a00640 1 -- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd40815acb0 msgr2=0x7fd4080ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd406a00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd40815acb0 0x7fd4080ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd406a00640 1 -- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4080b4680 msgr2=0x7fd4080ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd406a00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4080b4680 0x7fd4080ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd406a00640 1 -- 172.21.15.114:0/3194622794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40815d700 con 0x7fd40806ec00 2024-03-06T07:29:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd406a00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd40814acf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c3c0 tx=0x7fd3f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.022+0000 7fd3ff400640 1 -- 172.21.15.114:0/3194622794 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f80052d0 con 0x7fd40806ec00 2024-03-06T07:29:30.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.023+0000 7fd3ff400640 1 -- 172.21.15.114:0/3194622794 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f8005490 con 0x7fd40806ec00 2024-03-06T07:29:30.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.023+0000 7fd3ff400640 1 -- 172.21.15.114:0/3194622794 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8007260 con 0x7fd40806ec00 2024-03-06T07:29:30.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.023+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40815c660 con 0x7fd40806ec00 2024-03-06T07:29:30.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.023+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40815cc20 con 0x7fd40806ec00 2024-03-06T07:29:30.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.024+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40804f330 con 0x7fd40806ec00 2024-03-06T07:29:30.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.025+0000 7fd3ff400640 1 -- 172.21.15.114:0/3194622794 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3f8007400 con 0x7fd40806ec00 2024-03-06T07:29:30.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.026+0000 7fd3ff400640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3e806b4d0 0x7fd3e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.026+0000 7fd3ff400640 1 -- 172.21.15.114:0/3194622794 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3f80896f0 con 0x7fd40806ec00 2024-03-06T07:29:30.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.026+0000 7fd406000640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3e806b4d0 0x7fd3e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.027+0000 7fd406000640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3e806b4d0 0x7fd3e806d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd4080cf560 tx=0x7fd3f4007b90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.037+0000 7fd3ff400640 1 -- 172.21.15.114:0/3194622794 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3f8058560 con 0x7fd40806ec00 2024-03-06T07:29:30.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.234+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd40806ce60 con 0x7fd40806ec00 2024-03-06T07:29:30.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.236+0000 7fd3ff400640 1 -- 172.21.15.114:0/3194622794 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16717 (secure 0 0 0) 0x7fd3f8057e60 con 0x7fd40806ec00 2024-03-06T07:29:30.238 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":33,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:05.853392+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:02.138886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:03.320986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3e806b4d0 msgr2=0x7fd3e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3e806b4d0 0x7fd3e806d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd4080cf560 tx=0x7fd3f4007b90 comp rx=0 tx=0).stop 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 msgr2=0x7fd40814acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd40814acf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c3c0 tx=0x7fd3f800c880 comp rx=0 tx=0).stop 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 shutdown_connections 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd40815acb0 0x7fd4080ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4080b4680 0x7fd4080ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3e806b4d0 0x7fd3e806d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 --2- 172.21.15.114:0/3194622794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd40806ec00 0x7fd40814acf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 >> 172.21.15.114:0/3194622794 conn(0x7fd4080b1730 msgr2=0x7fd40815f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 shutdown_connections 2024-03-06T07:29:30.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.239+0000 7fd40de00640 1 -- 172.21.15.114:0/3194622794 wait complete. 2024-03-06T07:29:30.259 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:02.138886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:03.320986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:30.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.453+0000 7fd012600640 1 Processor -- start 2024-03-06T07:29:30.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.454+0000 7fd012600640 1 -- start start 2024-03-06T07:29:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.456+0000 7fd012600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 0x7fd00c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.456+0000 7fd012600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd00c0b3650 0x7fd00c0b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.456+0000 7fd012600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00c0b4710 con 0x7fd00c0cda90 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd012600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00c072310 con 0x7fd00c0b3650 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd012600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c0719e0 con 0x7fd00c0b3df0 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd003e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd00c0b3650 0x7fd00c0b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd003e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd00c0b3650 0x7fd00c0b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36054/0 (socket says 172.21.15.114:36054) 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd003e00640 1 -- 172.21.15.114:0/3320560676 learned_addr learned my addr 172.21.15.114:0/3320560676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd00be00640 1 --1- 172.21.15.114:0/3320560676 >> v1:172.21.15.114:6789/0 conn(0x7fd00c0cda90 0x7fd00c06c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49076/0 (socket says 172.21.15.114:49076) 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd00b400640 1 --2- 172.21.15.114:0/3320560676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 0x7fd00c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd003e00640 1 -- 172.21.15.114:0/3320560676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 msgr2=0x7fd00c0727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd003e00640 1 --2- 172.21.15.114:0/3320560676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 0x7fd00c0727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd003e00640 1 -- 172.21.15.114:0/3320560676 >> v1:172.21.15.114:6789/0 conn(0x7fd00c0cda90 legacy=0x7fd00c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.457+0000 7fd003e00640 1 -- 172.21.15.114:0/3320560676 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c071cc0 con 0x7fd00c0b3650 2024-03-06T07:29:30.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.459+0000 7fd003e00640 1 --2- 172.21.15.114:0/3320560676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c0b41d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009870 tx=0x7fcff402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05a45130b7468c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.459+0000 7fd00aa00640 1 -- 172.21.15.114:0/3320560676 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4034070 con 0x7fd00c0b3650 2024-03-06T07:29:30.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.459+0000 7fd00aa00640 1 -- 172.21.15.114:0/3320560676 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff4030840 con 0x7fd00c0b3650 2024-03-06T07:29:30.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.459+0000 7fd00aa00640 1 -- 172.21.15.114:0/3320560676 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4030c10 con 0x7fd00c0b3650 2024-03-06T07:29:30.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.459+0000 7fd012600640 1 -- 172.21.15.114:0/3320560676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 msgr2=0x7fd00c0b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.459+0000 7fd012600640 1 --2- 172.21.15.114:0/3320560676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c0b41d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009870 tx=0x7fcff402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:30.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.460+0000 7fd012600640 1 -- 172.21.15.114:0/3320560676 shutdown_connections 2024-03-06T07:29:30.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.460+0000 7fd012600640 1 --2- 172.21.15.114:0/3320560676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c0b41d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.460+0000 7fd012600640 1 --2- 172.21.15.114:0/3320560676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 0x7fd00c0727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.460+0000 7fd012600640 1 -- 172.21.15.114:0/3320560676 >> 172.21.15.114:0/3320560676 conn(0x7fd00c0b1940 msgr2=0x7fd00c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.460+0000 7fd012600640 1 -- 172.21.15.114:0/3320560676 shutdown_connections 2024-03-06T07:29:30.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.460+0000 7fd012600640 1 -- 172.21.15.114:0/3320560676 wait complete. 2024-03-06T07:29:30.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.461+0000 7fd012600640 1 Processor -- start 2024-03-06T07:29:30.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd012600640 1 -- start start 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd012600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c14f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd012600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 0x7fd00c0ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd012600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd00c0cda90 0x7fd00c0aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd012600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c15b240 con 0x7fd00c0b3650 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd012600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c0b3b30 con 0x7fd00c0cda90 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd012600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c15b3c0 con 0x7fd00c0b3df0 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd00b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c14f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd00b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c14f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36070/0 (socket says 172.21.15.114:36070) 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.462+0000 7fd00b400640 1 -- 172.21.15.114:0/933545613 learned_addr learned my addr 172.21.15.114:0/933545613 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.463+0000 7fd00be00640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd00c0cda90 0x7fd00c0aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.463+0000 7fd00b400640 1 -- 172.21.15.114:0/933545613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 msgr2=0x7fd00c0ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.463+0000 7fd00b400640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 0x7fd00c0ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.463+0000 7fd00b400640 1 -- 172.21.15.114:0/933545613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd00c0cda90 msgr2=0x7fd00c0aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.463+0000 7fd00b400640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd00c0cda90 0x7fd00c0aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.463+0000 7fd00b400640 1 -- 172.21.15.114:0/933545613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c09ab70 con 0x7fd00c0b3650 2024-03-06T07:29:30.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.464+0000 7fd00b400640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c14f8f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00d830 tx=0x7fcffc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.464+0000 7fd003400640 1 -- 172.21.15.114:0/933545613 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc00b6c0 con 0x7fd00c0b3650 2024-03-06T07:29:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.464+0000 7fd003400640 1 -- 172.21.15.114:0/933545613 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc019070 con 0x7fd00c0b3650 2024-03-06T07:29:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.464+0000 7fd003400640 1 -- 172.21.15.114:0/933545613 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc00ba80 con 0x7fd00c0b3650 2024-03-06T07:29:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.464+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00c09ae50 con 0x7fd00c0b3650 2024-03-06T07:29:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.464+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c09b410 con 0x7fd00c0b3650 2024-03-06T07:29:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.465+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00c04f330 con 0x7fd00c0b3650 2024-03-06T07:29:30.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.466+0000 7fd003400640 1 -- 172.21.15.114:0/933545613 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcffc01e460 con 0x7fd00c0b3650 2024-03-06T07:29:30.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.467+0000 7fd003400640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcfe806b4d0 0x7fcfe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.467+0000 7fd003400640 1 -- 172.21.15.114:0/933545613 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcffc08a940 con 0x7fd00c0b3650 2024-03-06T07:29:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.467+0000 7fd003e00640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcfe806b4d0 0x7fcfe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.468+0000 7fd003e00640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcfe806b4d0 0x7fcfe806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcff4036000 tx=0x7fcff403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.471+0000 7fd003400640 1 -- 172.21.15.114:0/933545613 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcffc0597b0 con 0x7fd00c0b3650 2024-03-06T07:29:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.694+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd00c0cc0e0 con 0x7fd00c0b3650 2024-03-06T07:29:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.695+0000 7fd003400640 1 -- 172.21.15.114:0/933545613 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcffc0590b0 con 0x7fd00c0b3650 2024-03-06T07:29:30.697 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:29:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcfe806b4d0 msgr2=0x7fcfe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcfe806b4d0 0x7fcfe806d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcff4036000 tx=0x7fcff403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 msgr2=0x7fd00c14f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c14f8f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00d830 tx=0x7fcffc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 shutdown_connections 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd00c0cda90 0x7fd00c0aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd00c0b3df0 0x7fd00c0ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcfe806b4d0 0x7fcfe806d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.698+0000 7fd012600640 1 --2- 172.21.15.114:0/933545613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd00c0b3650 0x7fd00c14f8f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.699+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 >> 172.21.15.114:0/933545613 conn(0x7fd00c0b1940 msgr2=0x7fd00c153a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.699+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 shutdown_connections 2024-03-06T07:29:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.699+0000 7fd012600640 1 -- 172.21.15.114:0/933545613 wait complete. 2024-03-06T07:29:30.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:30.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.925+0000 7f0abf800640 1 Processor -- start 2024-03-06T07:29:30.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.926+0000 7f0abf800640 1 -- start start 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.928+0000 7f0abf800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab80d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.928+0000 7f0abf800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0ab80d1bf0 0x7f0ab80cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.928+0000 7f0abf800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab80b3730 con 0x7f0ab815e450 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.928+0000 7f0abf800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab80d4930 con 0x7f0ab80d1bf0 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.928+0000 7f0abf800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8067b20 con 0x7f0ab80d07f0 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abd800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0ab80d1bf0 0x7f0ab80cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abd800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0ab80d1bf0 0x7f0ab80cfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36074/0 (socket says 172.21.15.114:36074) 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abd800640 1 -- 172.21.15.114:0/726107349 learned_addr learned my addr 172.21.15.114:0/726107349 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0ab7e00640 1 --1- 172.21.15.114:0/726107349 >> v1:172.21.15.114:6789/0 conn(0x7f0ab815e450 0x7f0ab80b4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49090/0 (socket says 172.21.15.114:49090) 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abce00640 1 --2- 172.21.15.114:0/726107349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab80d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abd800640 1 -- 172.21.15.114:0/726107349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 msgr2=0x7f0ab80d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abd800640 1 --2- 172.21.15.114:0/726107349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab80d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abd800640 1 -- 172.21.15.114:0/726107349 >> v1:172.21.15.114:6789/0 conn(0x7f0ab815e450 legacy=0x7f0ab80b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.929+0000 7f0abd800640 1 -- 172.21.15.114:0/726107349 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80d1110 con 0x7f0ab80d1bf0 2024-03-06T07:29:30.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.930+0000 7f0abd800640 1 --2- 172.21.15.114:0/726107349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80cfa30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009e50 tx=0x7f0ab002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677d16f06ce7656b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.930+0000 7f0ab7400640 1 -- 172.21.15.114:0/726107349 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab002ca70 con 0x7f0ab80d1bf0 2024-03-06T07:29:30.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0ab7400640 1 -- 172.21.15.114:0/726107349 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab002cc10 con 0x7f0ab80d1bf0 2024-03-06T07:29:30.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0ab7400640 1 -- 172.21.15.114:0/726107349 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00343f0 con 0x7f0ab80d1bf0 2024-03-06T07:29:30.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 -- 172.21.15.114:0/726107349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab80d1bf0 msgr2=0x7f0ab80cfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 --2- 172.21.15.114:0/726107349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80cfa30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009e50 tx=0x7f0ab002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 -- 172.21.15.114:0/726107349 shutdown_connections 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 --2- 172.21.15.114:0/726107349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80cfa30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 --2- 172.21.15.114:0/726107349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab80d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 -- 172.21.15.114:0/726107349 >> 172.21.15.114:0/726107349 conn(0x7f0ab80d3e10 msgr2=0x7f0ab80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 -- 172.21.15.114:0/726107349 shutdown_connections 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.931+0000 7f0abf800640 1 -- 172.21.15.114:0/726107349 wait complete. 2024-03-06T07:29:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.933+0000 7f0abf800640 1 Processor -- start 2024-03-06T07:29:30.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.933+0000 7f0abf800640 1 -- start start 2024-03-06T07:29:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abf800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab8157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abf800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abf800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab815e450 0x7f0ab80cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abf800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80b3c80 con 0x7f0ab815e450 2024-03-06T07:29:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abf800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80b39a0 con 0x7f0ab80d1bf0 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abf800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80b3f60 con 0x7f0ab80d07f0 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abd800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab815e450 0x7f0ab80cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abd800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab815e450 0x7f0ab80cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36090/0 (socket says 172.21.15.114:36090) 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.934+0000 7f0abd800640 1 -- 172.21.15.114:0/166790532 learned_addr learned my addr 172.21.15.114:0/166790532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abce00640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab8157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0ab7e00640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abd800640 1 -- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 msgr2=0x7f0ab8157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abd800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab8157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abd800640 1 -- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ab80d1bf0 msgr2=0x7f0ab80ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abd800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abd800640 1 -- 172.21.15.114:0/166790532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80b82f0 con 0x7f0ab815e450 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0ab7e00640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abd800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab815e450 0x7f0ab80cd1a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036040 tx=0x7f0ab0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0ab4c00640 1 -- 172.21.15.114:0/166790532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0043020 con 0x7f0ab815e450 2024-03-06T07:29:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0ab4c00640 1 -- 172.21.15.114:0/166790532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab003e450 con 0x7f0ab815e450 2024-03-06T07:29:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0ab4c00640 1 -- 172.21.15.114:0/166790532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0009510 con 0x7f0ab815e450 2024-03-06T07:29:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.935+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab80b85d0 con 0x7f0ab815e450 2024-03-06T07:29:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.936+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab8158230 con 0x7f0ab815e450 2024-03-06T07:29:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.937+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab804f330 con 0x7f0ab815e450 2024-03-06T07:29:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.938+0000 7f0ab4c00640 1 -- 172.21.15.114:0/166790532 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ab0046020 con 0x7f0ab815e450 2024-03-06T07:29:30.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.939+0000 7f0ab4c00640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9806b4d0 0x7f0a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.939+0000 7f0ab4c00640 1 -- 172.21.15.114:0/166790532 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ab00ac9b0 con 0x7f0ab815e450 2024-03-06T07:29:30.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.940+0000 7f0abce00640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9806b4d0 0x7f0a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.940+0000 7f0abce00640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9806b4d0 0x7f0a9806d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0aac006fd0 tx=0x7f0aac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:30.944+0000 7f0ab4c00640 1 -- 172.21.15.114:0/166790532 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ab004d030 con 0x7f0ab815e450 2024-03-06T07:29:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.143+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ab80d21d0 con 0x7f0ab815e450 2024-03-06T07:29:31.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.144+0000 7f0ab4c00640 1 -- 172.21.15.114:0/166790532 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0ab007b820 con 0x7f0ab815e450 2024-03-06T07:29:31.146 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9806b4d0 msgr2=0x7f0a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9806b4d0 0x7f0a9806d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0aac006fd0 tx=0x7f0aac008040 comp rx=0 tx=0).stop 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab815e450 msgr2=0x7f0ab80cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab815e450 0x7f0ab80cd1a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036040 tx=0x7f0ab0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 shutdown_connections 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9806b4d0 0x7f0a9806d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ab815e450 0x7f0ab80cd1a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ab80d1bf0 0x7f0ab80ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 --2- 172.21.15.114:0/166790532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ab80d07f0 0x7f0ab8157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 >> 172.21.15.114:0/166790532 conn(0x7f0ab80d3e10 msgr2=0x7f0ab80b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 shutdown_connections 2024-03-06T07:29:31.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.147+0000 7f0abf800640 1 -- 172.21.15.114:0/166790532 wait complete. 2024-03-06T07:29:31.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:31.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.354+0000 7ff373000640 1 Processor -- start 2024-03-06T07:29:31.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.355+0000 7ff373000640 1 -- start start 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff373000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff36c15fc10 0x7ff36c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff373000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff36c15fff0 0x7ff36c160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff373000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 0x7ff36c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff373000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff36c0b46c0 con 0x7ff36c15fff0 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff373000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff36c0b49a0 con 0x7ff36c15fc10 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff373000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c072c90 con 0x7ff36c0ce4a0 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff363400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff36c15fff0 0x7ff36c160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff363400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff36c15fff0 0x7ff36c160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36102/0 (socket says 172.21.15.114:36102) 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff363400640 1 -- 172.21.15.114:0/2837928988 learned_addr learned my addr 172.21.15.114:0/2837928988 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.358+0000 7ff36be00640 1 --2- 172.21.15.114:0/2837928988 >> v2:172.21.15.177:3300/0 conn(0x7ff36c15fc10 0x7ff36c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.359+0000 7ff371000640 1 --2- 172.21.15.114:0/2837928988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 0x7ff36c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.359+0000 7ff363400640 1 -- 172.21.15.114:0/2837928988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 msgr2=0x7ff36c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.359+0000 7ff363400640 1 --2- 172.21.15.114:0/2837928988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 0x7ff36c0ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.359+0000 7ff363400640 1 -- 172.21.15.114:0/2837928988 >> v2:172.21.15.177:3300/0 conn(0x7ff36c15fc10 msgr2=0x7ff36c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.359+0000 7ff363400640 1 --2- 172.21.15.114:0/2837928988 >> v2:172.21.15.177:3300/0 conn(0x7ff36c15fc10 0x7ff36c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.359+0000 7ff363400640 1 -- 172.21.15.114:0/2837928988 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c160990 con 0x7ff36c15fff0 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.360+0000 7ff363400640 1 --2- 172.21.15.114:0/2837928988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fff0 0x7ff36c160450 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0cf770 tx=0x7ff35402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a7d14e6aa16e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.360+0000 7ff36b400640 1 -- 172.21.15.114:0/2837928988 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354035020 con 0x7ff36c15fff0 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.360+0000 7ff36b400640 1 -- 172.21.15.114:0/2837928988 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354031870 con 0x7ff36c15fff0 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.360+0000 7ff36b400640 1 -- 172.21.15.114:0/2837928988 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354031c40 con 0x7ff36c15fff0 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 -- 172.21.15.114:0/2837928988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fff0 msgr2=0x7ff36c160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 --2- 172.21.15.114:0/2837928988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fff0 0x7ff36c160450 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0cf770 tx=0x7ff35402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 -- 172.21.15.114:0/2837928988 shutdown_connections 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 --2- 172.21.15.114:0/2837928988 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 0x7ff36c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 --2- 172.21.15.114:0/2837928988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fff0 0x7ff36c160450 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 --2- 172.21.15.114:0/2837928988 >> v2:172.21.15.177:3300/0 conn(0x7ff36c15fc10 0x7ff36c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 -- 172.21.15.114:0/2837928988 >> 172.21.15.114:0/2837928988 conn(0x7ff36c0b1940 msgr2=0x7ff36c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 -- 172.21.15.114:0/2837928988 shutdown_connections 2024-03-06T07:29:31.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.361+0000 7ff373000640 1 -- 172.21.15.114:0/2837928988 wait complete. 2024-03-06T07:29:31.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.364+0000 7ff373000640 1 Processor -- start 2024-03-06T07:29:31.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.364+0000 7ff373000640 1 -- start start 2024-03-06T07:29:31.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.365+0000 7ff373000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 0x7ff36c1580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.365+0000 7ff373000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fc10 0x7ff36c1f5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.365+0000 7ff373000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c15fff0 0x7ff36c14e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.365+0000 7ff373000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c072a30 con 0x7ff36c15fc10 2024-03-06T07:29:31.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.365+0000 7ff373000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c0728b0 con 0x7ff36c15fff0 2024-03-06T07:29:31.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.365+0000 7ff373000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c15c2e0 con 0x7ff36c0ce4a0 2024-03-06T07:29:31.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fc10 0x7ff36c1f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fc10 0x7ff36c1f5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36110/0 (socket says 172.21.15.114:36110) 2024-03-06T07:29:31.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 -- 172.21.15.114:0/166534395 learned_addr learned my addr 172.21.15.114:0/166534395 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 -- 172.21.15.114:0/166534395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 msgr2=0x7ff36c1580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 0x7ff36c1580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 -- 172.21.15.114:0/166534395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c15fff0 msgr2=0x7ff36c14e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c15fff0 0x7ff36c14e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 -- 172.21.15.114:0/166534395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c1f5940 con 0x7ff36c15fc10 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.366+0000 7ff363400640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fc10 0x7ff36c1f5210 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff35402b950 tx=0x7ff3540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.367+0000 7ff368c00640 1 -- 172.21.15.114:0/166534395 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354040070 con 0x7ff36c15fc10 2024-03-06T07:29:31.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.367+0000 7ff368c00640 1 -- 172.21.15.114:0/166534395 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354030d00 con 0x7ff36c15fc10 2024-03-06T07:29:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.367+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c1f5c20 con 0x7ff36c15fc10 2024-03-06T07:29:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.367+0000 7ff368c00640 1 -- 172.21.15.114:0/166534395 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354035040 con 0x7ff36c15fc10 2024-03-06T07:29:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.367+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36c14e8f0 con 0x7ff36c15fc10 2024-03-06T07:29:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.368+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36c04f330 con 0x7ff36c15fc10 2024-03-06T07:29:31.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.369+0000 7ff368c00640 1 -- 172.21.15.114:0/166534395 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff35404b060 con 0x7ff36c15fc10 2024-03-06T07:29:31.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.370+0000 7ff368c00640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff34806b4d0 0x7ff34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.370+0000 7ff368c00640 1 -- 172.21.15.114:0/166534395 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3540abfc0 con 0x7ff36c15fc10 2024-03-06T07:29:31.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.370+0000 7ff36be00640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff34806b4d0 0x7ff34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.371+0000 7ff36be00640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff34806b4d0 0x7ff34806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0070b0 tx=0x7ff35c007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.374+0000 7ff368c00640 1 -- 172.21.15.114:0/166534395 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff35407ada0 con 0x7ff36c15fc10 2024-03-06T07:29:31.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.571+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff36c06ef30 con 0x7ff36c15fc10 2024-03-06T07:29:31.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.572+0000 7ff368c00640 1 -- 172.21.15.114:0/166534395 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff35407a6a0 con 0x7ff36c15fc10 2024-03-06T07:29:31.573 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff34806b4d0 msgr2=0x7ff34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff34806b4d0 0x7ff34806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0070b0 tx=0x7ff35c007040 comp rx=0 tx=0).stop 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fc10 msgr2=0x7ff36c1f5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fc10 0x7ff36c1f5210 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff35402b950 tx=0x7ff3540046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 shutdown_connections 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c15fff0 0x7ff36c14e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff34806b4d0 0x7ff34806d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff36c15fc10 0x7ff36c1f5210 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 --2- 172.21.15.114:0/166534395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff36c0ce4a0 0x7ff36c1580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.574+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 >> 172.21.15.114:0/166534395 conn(0x7ff36c0b1940 msgr2=0x7ff36c0cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.575+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 shutdown_connections 2024-03-06T07:29:31.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.575+0000 7ff373000640 1 -- 172.21.15.114:0/166534395 wait complete. 2024-03-06T07:29:31.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:31.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.795+0000 7f5695200640 1 Processor -- start 2024-03-06T07:29:31.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.795+0000 7f5695200640 1 -- start start 2024-03-06T07:29:31.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.797+0000 7f5695200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56900d1660 0x7f56900d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.797+0000 7f5695200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56900cf3a0 con 0x7f56900cd470 2024-03-06T07:29:31.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.797+0000 7f5695200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569006cd30 con 0x7f56900ce590 2024-03-06T07:29:31.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.797+0000 7f5695200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569006cae0 con 0x7f56900d1660 2024-03-06T07:29:31.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.797+0000 7f568e800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56900d1660 0x7f56900d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.797+0000 7f568e800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56900d1660 0x7f56900d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36120/0 (socket says 172.21.15.114:36120) 2024-03-06T07:29:31.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.797+0000 7f568e800640 1 -- 172.21.15.114:0/1957100306 learned_addr learned my addr 172.21.15.114:0/1957100306 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:31.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.798+0000 7f568e800640 1 -- 172.21.15.114:0/1957100306 >> v1:172.21.15.177:6789/0 conn(0x7f56900ce590 legacy=0x7f56900ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.798+0000 7f568e800640 1 -- 172.21.15.114:0/1957100306 >> v1:172.21.15.114:6789/0 conn(0x7f56900cd470 legacy=0x7f56900cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.798+0000 7f568e800640 1 -- 172.21.15.114:0/1957100306 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56900b0e30 con 0x7f56900d1660 2024-03-06T07:29:31.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.799+0000 7f568e800640 1 --2- 172.21.15.114:0/1957100306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f56900d0470 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5688009e50 tx=0x7f568802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1588ff6bd44e70f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.799+0000 7f568d400640 1 -- 172.21.15.114:0/1957100306 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568802cb10 con 0x7f56900d1660 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.799+0000 7f568d400640 1 -- 172.21.15.114:0/1957100306 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568802ccb0 con 0x7f56900d1660 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.799+0000 7f568d400640 1 -- 172.21.15.114:0/1957100306 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568802ce60 con 0x7f56900d1660 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.800+0000 7f5695200640 1 -- 172.21.15.114:0/1957100306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 msgr2=0x7f56900d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.800+0000 7f5695200640 1 --2- 172.21.15.114:0/1957100306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f56900d0470 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5688009e50 tx=0x7f568802bf70 comp rx=0 tx=0).stop 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.800+0000 7f5695200640 1 -- 172.21.15.114:0/1957100306 shutdown_connections 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.800+0000 7f5695200640 1 --2- 172.21.15.114:0/1957100306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f56900d0470 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.800+0000 7f5695200640 1 -- 172.21.15.114:0/1957100306 >> 172.21.15.114:0/1957100306 conn(0x7f56900d1d10 msgr2=0x7f56900d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.800+0000 7f5695200640 1 -- 172.21.15.114:0/1957100306 shutdown_connections 2024-03-06T07:29:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.800+0000 7f5695200640 1 -- 172.21.15.114:0/1957100306 wait complete. 2024-03-06T07:29:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.802+0000 7f5695200640 1 Processor -- start 2024-03-06T07:29:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.802+0000 7f5695200640 1 -- start start 2024-03-06T07:29:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f5695200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56900cd470 0x7f56900703d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f5695200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56900ce590 0x7f569015cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f5695200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f5690150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f5695200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5690161980 con 0x7f56900d1660 2024-03-06T07:29:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f5695200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569006d0a0 con 0x7f56900ce590 2024-03-06T07:29:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f5695200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5690161b00 con 0x7f56900cd470 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f568e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f5690150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f568e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f5690150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36128/0 (socket says 172.21.15.114:36128) 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f568e800640 1 -- 172.21.15.114:0/3708865110 learned_addr learned my addr 172.21.15.114:0/3708865110 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f5685400640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56900ce590 0x7f569015cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f568de00640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56900cd470 0x7f56900703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.803+0000 7f568e800640 1 -- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56900cd470 msgr2=0x7f56900703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f568e800640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56900cd470 0x7f56900703d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f568e800640 1 -- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56900ce590 msgr2=0x7f569015cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f568e800640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56900ce590 0x7f569015cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f568e800640 1 -- 172.21.15.114:0/3708865110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690159e30 con 0x7f56900d1660 2024-03-06T07:29:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f568de00640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56900cd470 0x7f56900703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:31.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f5685400640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56900ce590 0x7f569015cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:31.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f568e800640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f5690150060 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5688036040 tx=0x7f5688004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f5686a00640 1 -- 172.21.15.114:0/3708865110 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5688043020 con 0x7f56900d1660 2024-03-06T07:29:31.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f5686a00640 1 -- 172.21.15.114:0/3708865110 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568803e450 con 0x7f56900d1660 2024-03-06T07:29:31.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f5686a00640 1 -- 172.21.15.114:0/3708865110 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568802ce60 con 0x7f56900d1660 2024-03-06T07:29:31.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569015a110 con 0x7f56900d1660 2024-03-06T07:29:31.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.804+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56900cac80 con 0x7f56900d1660 2024-03-06T07:29:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.805+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56900b0270 con 0x7f56900d1660 2024-03-06T07:29:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.807+0000 7f5686a00640 1 -- 172.21.15.114:0/3708865110 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5688046020 con 0x7f56900d1660 2024-03-06T07:29:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.808+0000 7f5686a00640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f566c06b4d0 0x7f566c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.808+0000 7f5686a00640 1 -- 172.21.15.114:0/3708865110 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56880acb10 con 0x7f56900d1660 2024-03-06T07:29:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.808+0000 7f568de00640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f566c06b4d0 0x7f566c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.809+0000 7f568de00640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f566c06b4d0 0x7f566c06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f568000a830 tx=0x7f5680008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:31.812+0000 7f5686a00640 1 -- 172.21.15.114:0/3708865110 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56900b0270 con 0x7f56900d1660 2024-03-06T07:29:32.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.022+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f569004f330 con 0x7f56900d1660 2024-03-06T07:29:32.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.023+0000 7f5686a00640 1 -- 172.21.15.114:0/3708865110 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7f568804d030 con 0x7f56900d1660 2024-03-06T07:29:32.025 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f566c06b4d0 msgr2=0x7f566c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f566c06b4d0 0x7f566c06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f568000a830 tx=0x7f5680008040 comp rx=0 tx=0).stop 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 msgr2=0x7f5690150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f5690150060 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5688036040 tx=0x7f5688004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 shutdown_connections 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f566c06b4d0 0x7f566c06d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56900d1660 0x7f5690150060 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56900ce590 0x7f569015cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 --2- 172.21.15.114:0/3708865110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56900cd470 0x7f56900703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 >> 172.21.15.114:0/3708865110 conn(0x7f56900d1d10 msgr2=0x7f56900afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 shutdown_connections 2024-03-06T07:29:32.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.026+0000 7f5695200640 1 -- 172.21.15.114:0/3708865110 wait complete. 2024-03-06T07:29:32.046 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:29:32.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:32.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:32.159 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:29:32.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:32.159 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:32.203 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:29:32.203 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:32.244 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:32.244 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:29:32.245 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:32.281 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:32.281 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:29:32.281 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:32.334 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:32.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:32.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.532+0000 7fa2bd000640 1 Processor -- start 2024-03-06T07:29:32.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.532+0000 7fa2bd000640 1 -- start start 2024-03-06T07:29:32.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2bd000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2b80d1cc0 0x7fa2b80b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2bd000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b80d4c50 con 0x7fa2b80d24c0 2024-03-06T07:29:32.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2bd000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b80cfa30 con 0x7fa2b80b2de0 2024-03-06T07:29:32.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2bd000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b80d4a00 con 0x7fa2b80d1cc0 2024-03-06T07:29:32.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2b6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2b80d1cc0 0x7fa2b80b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2b6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa2b80d1cc0 0x7fa2b80b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36138/0 (socket says 172.21.15.114:36138) 2024-03-06T07:29:32.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2b6800640 1 -- 172.21.15.114:0/1292792735 learned_addr learned my addr 172.21.15.114:0/1292792735 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:32.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.533+0000 7fa2b5e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa2b80d24c0 0x7fa2b80d28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49114/0 (socket says 172.21.15.114:49114) 2024-03-06T07:29:32.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.534+0000 7fa2b6800640 1 -- 172.21.15.114:0/1292792735 >> v1:172.21.15.177:6789/0 conn(0x7fa2b80b2de0 legacy=0x7fa2b80b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.534+0000 7fa2b6800640 1 -- 172.21.15.114:0/1292792735 >> v1:172.21.15.114:6789/0 conn(0x7fa2b80d24c0 legacy=0x7fa2b80d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.534+0000 7fa2b6800640 1 -- 172.21.15.114:0/1292792735 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80cfd10 con 0x7fa2b80d1cc0 2024-03-06T07:29:32.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.535+0000 7fa2b6800640 1 --2- 172.21.15.114:0/1292792735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80d1cc0 0x7fa2b80b11b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009e50 tx=0x7fa2b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b6805bd671ed88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.535+0000 7fa2b5400640 1 -- 172.21.15.114:0/1292792735 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b002ca80 con 0x7fa2b80d1cc0 2024-03-06T07:29:32.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.535+0000 7fa2b5400640 1 -- 172.21.15.114:0/1292792735 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b002cc20 con 0x7fa2b80d1cc0 2024-03-06T07:29:32.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.535+0000 7fa2b5400640 1 -- 172.21.15.114:0/1292792735 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b00343f0 con 0x7fa2b80d1cc0 2024-03-06T07:29:32.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.536+0000 7fa2bd000640 1 -- 172.21.15.114:0/1292792735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80d1cc0 msgr2=0x7fa2b80b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.536+0000 7fa2bd000640 1 --2- 172.21.15.114:0/1292792735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80d1cc0 0x7fa2b80b11b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009e50 tx=0x7fa2b002bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.536+0000 7fa2bd000640 1 -- 172.21.15.114:0/1292792735 shutdown_connections 2024-03-06T07:29:32.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.536+0000 7fa2bd000640 1 --2- 172.21.15.114:0/1292792735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80d1cc0 0x7fa2b80b11b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.536+0000 7fa2bd000640 1 -- 172.21.15.114:0/1292792735 >> 172.21.15.114:0/1292792735 conn(0x7fa2b80d3ee0 msgr2=0x7fa2b80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.536+0000 7fa2bd000640 1 -- 172.21.15.114:0/1292792735 shutdown_connections 2024-03-06T07:29:32.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.536+0000 7fa2bd000640 1 -- 172.21.15.114:0/1292792735 wait complete. 2024-03-06T07:29:32.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.538+0000 7fa2bd000640 1 Processor -- start 2024-03-06T07:29:32.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.538+0000 7fa2bd000640 1 -- start start 2024-03-06T07:29:32.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2bd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80b2de0 0x7fa2b80c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2bd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2b80d1cc0 0x7fa2b8152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2bd000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2b80d24c0 0x7fa2b80cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2bd000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80d0080 con 0x7fa2b80b2de0 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2bd000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80cf7e0 con 0x7fa2b80d1cc0 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2bd000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80b16f0 con 0x7fa2b80d24c0 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2b5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80b2de0 0x7fa2b80c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2b5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80b2de0 0x7fa2b80c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36150/0 (socket says 172.21.15.114:36150) 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2b5e00640 1 -- 172.21.15.114:0/3183856475 learned_addr learned my addr 172.21.15.114:0/3183856475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2b5e00640 1 -- 172.21.15.114:0/3183856475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2b80d24c0 msgr2=0x7fa2b80cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.539+0000 7fa2b5e00640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2b80d24c0 0x7fa2b80cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2b5e00640 1 -- 172.21.15.114:0/3183856475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2b80d1cc0 msgr2=0x7fa2b8152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2b5e00640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2b80d1cc0 0x7fa2b8152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2b5e00640 1 -- 172.21.15.114:0/3183856475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80cd970 con 0x7fa2b80b2de0 2024-03-06T07:29:32.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2b5e00640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80b2de0 0x7fa2b80c8420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c3f0 tx=0x7fa2a800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2ae000640 1 -- 172.21.15.114:0/3183856475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80074e0 con 0x7fa2b80b2de0 2024-03-06T07:29:32.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2ae000640 1 -- 172.21.15.114:0/3183856475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8007680 con 0x7fa2b80b2de0 2024-03-06T07:29:32.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2ae000640 1 -- 172.21.15.114:0/3183856475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8004a00 con 0x7fa2b80b2de0 2024-03-06T07:29:32.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b80c8ab0 con 0x7fa2b80b2de0 2024-03-06T07:29:32.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.540+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b80c9070 con 0x7fa2b80b2de0 2024-03-06T07:29:32.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.542+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b804f330 con 0x7fa2b80b2de0 2024-03-06T07:29:32.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.542+0000 7fa2ae000640 1 -- 172.21.15.114:0/3183856475 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2a801a020 con 0x7fa2b80b2de0 2024-03-06T07:29:32.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.543+0000 7fa2ae000640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa29406b4d0 0x7fa29406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.543+0000 7fa2ae000640 1 -- 172.21.15.114:0/3183856475 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2a8089a90 con 0x7fa2b80b2de0 2024-03-06T07:29:32.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.544+0000 7fa2afe00640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa29406b4d0 0x7fa29406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.544+0000 7fa2afe00640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa29406b4d0 0x7fa29406d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8161450 tx=0x7fa2a00079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.548+0000 7fa2ae000640 1 -- 172.21.15.114:0/3183856475 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2a8010300 con 0x7fa2b80b2de0 2024-03-06T07:29:32.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.831+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2b80d22a0 con 0x7fa2b80b2de0 2024-03-06T07:29:32.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.832+0000 7fa2ae000640 1 -- 172.21.15.114:0/3183856475 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fa2a80587e0 con 0x7fa2b80b2de0 2024-03-06T07:29:32.834 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:32.834 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:32.834 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:32.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.834+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa29406b4d0 msgr2=0x7fa29406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.834+0000 7fa2bd000640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa29406b4d0 0x7fa29406d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8161450 tx=0x7fa2a00079b0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.834+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80b2de0 msgr2=0x7fa2b80c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.834+0000 7fa2bd000640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80b2de0 0x7fa2b80c8420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c3f0 tx=0x7fa2a800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 shutdown_connections 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2b80d24c0 0x7fa2b80cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2b80d1cc0 0x7fa2b8152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa29406b4d0 0x7fa29406d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 --2- 172.21.15.114:0/3183856475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2b80b2de0 0x7fa2b80c8420 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 >> 172.21.15.114:0/3183856475 conn(0x7fa2b80d3ee0 msgr2=0x7fa2b80d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 shutdown_connections 2024-03-06T07:29:32.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:32.835+0000 7fa2bd000640 1 -- 172.21.15.114:0/3183856475 wait complete. 2024-03-06T07:29:32.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:33.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.050+0000 7fef78e00640 1 Processor -- start 2024-03-06T07:29:33.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.050+0000 7fef78e00640 1 -- start start 2024-03-06T07:29:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef78e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fef741561f0 0x7fef740ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef78e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef740aebf0 0x7fef740cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef78e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef740cc800 0x7fef740cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef78e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7415ce30 con 0x7fef740cc800 2024-03-06T07:29:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef78e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7415c240 con 0x7fef741561f0 2024-03-06T07:29:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef78e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7415c500 con 0x7fef740aebf0 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef73e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef740cc800 0x7fef740cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef73e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef740cc800 0x7fef740cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36152/0 (socket says 172.21.15.114:36152) 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef73e00640 1 -- 172.21.15.114:0/1753125120 learned_addr learned my addr 172.21.15.114:0/1753125120 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.052+0000 7fef6be00640 1 --2- 172.21.15.114:0/1753125120 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef740aebf0 0x7fef740cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.053+0000 7fef73400640 1 --2- 172.21.15.114:0/1753125120 >> v2:172.21.15.177:3300/0 conn(0x7fef741561f0 0x7fef740ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.053+0000 7fef73e00640 1 -- 172.21.15.114:0/1753125120 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef740aebf0 msgr2=0x7fef740cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.053+0000 7fef73e00640 1 --2- 172.21.15.114:0/1753125120 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef740aebf0 0x7fef740cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.053+0000 7fef73e00640 1 -- 172.21.15.114:0/1753125120 >> v2:172.21.15.177:3300/0 conn(0x7fef741561f0 msgr2=0x7fef740ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.053+0000 7fef73e00640 1 --2- 172.21.15.114:0/1753125120 >> v2:172.21.15.177:3300/0 conn(0x7fef741561f0 0x7fef740ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.053+0000 7fef73e00640 1 -- 172.21.15.114:0/1753125120 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7415c7e0 con 0x7fef740cc800 2024-03-06T07:29:33.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.053+0000 7fef73400640 1 --2- 172.21.15.114:0/1753125120 >> v2:172.21.15.177:3300/0 conn(0x7fef741561f0 0x7fef740ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.054+0000 7fef73e00640 1 --2- 172.21.15.114:0/1753125120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740cc800 0x7fef740cf800 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fef740b4db0 tx=0x7fef6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe595ce8c096b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.054+0000 7fef72a00640 1 -- 172.21.15.114:0/1753125120 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c02ca70 con 0x7fef740cc800 2024-03-06T07:29:33.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.054+0000 7fef72a00640 1 -- 172.21.15.114:0/1753125120 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c02cc10 con 0x7fef740cc800 2024-03-06T07:29:33.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.054+0000 7fef72a00640 1 -- 172.21.15.114:0/1753125120 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0343f0 con 0x7fef740cc800 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 -- 172.21.15.114:0/1753125120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740cc800 msgr2=0x7fef740cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 --2- 172.21.15.114:0/1753125120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740cc800 0x7fef740cf800 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fef740b4db0 tx=0x7fef6c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 -- 172.21.15.114:0/1753125120 shutdown_connections 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 --2- 172.21.15.114:0/1753125120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740cc800 0x7fef740cf800 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 --2- 172.21.15.114:0/1753125120 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef740aebf0 0x7fef740cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 --2- 172.21.15.114:0/1753125120 >> v2:172.21.15.177:3300/0 conn(0x7fef741561f0 0x7fef740ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 -- 172.21.15.114:0/1753125120 >> 172.21.15.114:0/1753125120 conn(0x7fef7406afc0 msgr2=0x7fef7406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 -- 172.21.15.114:0/1753125120 shutdown_connections 2024-03-06T07:29:33.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.055+0000 7fef78e00640 1 -- 172.21.15.114:0/1753125120 wait complete. 2024-03-06T07:29:33.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.057+0000 7fef78e00640 1 Processor -- start 2024-03-06T07:29:33.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.057+0000 7fef78e00640 1 -- start start 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef78e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740aebf0 0x7fef740a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef78e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef740cc800 0x7fef740b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef78e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef741561f0 0x7fef74149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef78e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef741565d0 con 0x7fef740aebf0 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef78e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef740ccc90 con 0x7fef740cc800 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef78e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef74156750 con 0x7fef741561f0 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef73400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740aebf0 0x7fef740a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef73400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740aebf0 0x7fef740a9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36156/0 (socket says 172.21.15.114:36156) 2024-03-06T07:29:33.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.058+0000 7fef73400640 1 -- 172.21.15.114:0/3413968732 learned_addr learned my addr 172.21.15.114:0/3413968732 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:33.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef73e00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef741561f0 0x7fef74149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef73400640 1 -- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef741561f0 msgr2=0x7fef74149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef6be00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef740cc800 0x7fef740b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef73400640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef741561f0 0x7fef74149ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef73400640 1 -- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef740cc800 msgr2=0x7fef740b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef73400640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef740cc800 0x7fef740b0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef73400640 1 -- 172.21.15.114:0/3413968732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef740a99d0 con 0x7fef740aebf0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef73400640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740aebf0 0x7fef740a9490 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fef6400c820 tx=0x7fef6400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef6b400640 1 -- 172.21.15.114:0/3413968732 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64015070 con 0x7fef740aebf0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef6b400640 1 -- 172.21.15.114:0/3413968732 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef64007a30 con 0x7fef740aebf0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef6b400640 1 -- 172.21.15.114:0/3413968732 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64007e00 con 0x7fef740aebf0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef74158360 con 0x7fef740aebf0 2024-03-06T07:29:33.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.059+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef74158c40 con 0x7fef740aebf0 2024-03-06T07:29:33.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.061+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7404f330 con 0x7fef740aebf0 2024-03-06T07:29:33.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.063+0000 7fef6b400640 1 -- 172.21.15.114:0/3413968732 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef64018070 con 0x7fef740aebf0 2024-03-06T07:29:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.063+0000 7fef6b400640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5006b4d0 0x7fef5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.064+0000 7fef6b400640 1 -- 172.21.15.114:0/3413968732 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef6408a0b0 con 0x7fef740aebf0 2024-03-06T07:29:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.064+0000 7fef6be00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5006b4d0 0x7fef5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.065+0000 7fef6be00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5006b4d0 0x7fef5006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fef5c006fd0 tx=0x7fef5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.068+0000 7fef6b400640 1 -- 172.21.15.114:0/3413968732 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef64058fc0 con 0x7fef740aebf0 2024-03-06T07:29:33.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.294+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef740b2b10 con 0x7fef740aebf0 2024-03-06T07:29:33.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.296+0000 7fef6b400640 1 -- 172.21.15.114:0/3413968732 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16717 (secure 0 0 0) 0x7fef640588c0 con 0x7fef740aebf0 2024-03-06T07:29:33.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:33.298 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":33,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:05.853392+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:02.138886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:03.320986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:33.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5006b4d0 msgr2=0x7fef5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5006b4d0 0x7fef5006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fef5c006fd0 tx=0x7fef5c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740aebf0 msgr2=0x7fef740a9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740aebf0 0x7fef740a9490 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fef6400c820 tx=0x7fef6400cc20 comp rx=0 tx=0).stop 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 shutdown_connections 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef741561f0 0x7fef74149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef740cc800 0x7fef740b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5006b4d0 0x7fef5006d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 --2- 172.21.15.114:0/3413968732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef740aebf0 0x7fef740a9490 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 >> 172.21.15.114:0/3413968732 conn(0x7fef7406afc0 msgr2=0x7fef740b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.298+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 shutdown_connections 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.299+0000 7fef78e00640 1 -- 172.21.15.114:0/3413968732 wait complete. 2024-03-06T07:29:33.318 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:02.138886+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:03.320986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:33.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:33.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.513+0000 7f9696e00640 1 Processor -- start 2024-03-06T07:29:33.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.513+0000 7f9696e00640 1 -- start start 2024-03-06T07:29:33.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.514+0000 7f9696e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f96900b4de0 0x7f96900d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.514+0000 7f9696e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900ced50 0x7f96900cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.514+0000 7f9696e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96900d4c40 con 0x7f969015f490 2024-03-06T07:29:33.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.514+0000 7f9696e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96900d49f0 con 0x7f96900b4de0 2024-03-06T07:29:33.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.514+0000 7f9696e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96900d24b0 con 0x7f96900ced50 2024-03-06T07:29:33.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.515+0000 7f9694e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900ced50 0x7f96900cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.515+0000 7f968f400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f96900b4de0 0x7f96900d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.515+0000 7f9694e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900ced50 0x7f96900cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:55420/0 (socket says 172.21.15.114:55420) 2024-03-06T07:29:33.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.515+0000 7f9694e00640 1 -- 172.21.15.114:0/867292644 learned_addr learned my addr 172.21.15.114:0/867292644 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:33.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.516+0000 7f9694e00640 1 -- 172.21.15.114:0/867292644 >> v2:172.21.15.177:3300/0 conn(0x7f96900b4de0 msgr2=0x7f96900d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.516+0000 7f9694e00640 1 --2- 172.21.15.114:0/867292644 >> v2:172.21.15.177:3300/0 conn(0x7f96900b4de0 0x7f96900d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.516+0000 7f9694e00640 1 -- 172.21.15.114:0/867292644 >> v1:172.21.15.177:6789/0 conn(0x7f969015f490 legacy=0x7f96900d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.516+0000 7f9694e00640 1 -- 172.21.15.114:0/867292644 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96900d2700 con 0x7f96900ced50 2024-03-06T07:29:33.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.517+0000 7f9694e00640 1 --2- 172.21.15.114:0/867292644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900ced50 0x7f96900cf1c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9688009e50 tx=0x7f968802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16858da0da14ac4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.517+0000 7f968ea00640 1 -- 172.21.15.114:0/867292644 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968802ca80 con 0x7f96900ced50 2024-03-06T07:29:33.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f968ea00640 1 -- 172.21.15.114:0/867292644 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968802cc20 con 0x7f96900ced50 2024-03-06T07:29:33.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f968ea00640 1 -- 172.21.15.114:0/867292644 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96880343f0 con 0x7f96900ced50 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 -- 172.21.15.114:0/867292644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900ced50 msgr2=0x7f96900cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 --2- 172.21.15.114:0/867292644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900ced50 0x7f96900cf1c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9688009e50 tx=0x7f968802bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 -- 172.21.15.114:0/867292644 shutdown_connections 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 --2- 172.21.15.114:0/867292644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900ced50 0x7f96900cf1c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 --2- 172.21.15.114:0/867292644 >> v2:172.21.15.177:3300/0 conn(0x7f96900b4de0 0x7f96900d0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 -- 172.21.15.114:0/867292644 >> 172.21.15.114:0/867292644 conn(0x7f96900d3ed0 msgr2=0x7f96900d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 -- 172.21.15.114:0/867292644 shutdown_connections 2024-03-06T07:29:33.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.518+0000 7f9696e00640 1 -- 172.21.15.114:0/867292644 wait complete. 2024-03-06T07:29:33.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.520+0000 7f9696e00640 1 Processor -- start 2024-03-06T07:29:33.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.520+0000 7f9696e00640 1 -- start start 2024-03-06T07:29:33.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f9696e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900b4de0 0x7f969015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f9696e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96900ced50 0x7f96900cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f9696e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969015f490 0x7f96900d0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f9696e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969015fd50 con 0x7f96900ced50 2024-03-06T07:29:33.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f9696e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96900d2cc0 con 0x7f969015f490 2024-03-06T07:29:33.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f9696e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969015f870 con 0x7f96900b4de0 2024-03-06T07:29:33.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f968f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96900ced50 0x7f96900cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f968f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96900ced50 0x7f96900cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36160/0 (socket says 172.21.15.114:36160) 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f968f400640 1 -- 172.21.15.114:0/1946138783 learned_addr learned my addr 172.21.15.114:0/1946138783 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.521+0000 7f968fe00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900b4de0 0x7f969015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f9694e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969015f490 0x7f96900d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f968f400640 1 -- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900b4de0 msgr2=0x7f969015ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f968f400640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900b4de0 0x7f969015ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f968f400640 1 -- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969015f490 msgr2=0x7f96900d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f968f400640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969015f490 0x7f96900d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f968f400640 1 -- 172.21.15.114:0/1946138783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96901f42f0 con 0x7f96900ced50 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f968f400640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96900ced50 0x7f96900cb9b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f967c00e870 tx=0x7f967c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f9687e00640 1 -- 172.21.15.114:0/1946138783 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c017070 con 0x7f96900ced50 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f9687e00640 1 -- 172.21.15.114:0/1946138783 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c00c940 con 0x7f96900ced50 2024-03-06T07:29:33.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f9687e00640 1 -- 172.21.15.114:0/1946138783 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c00cd30 con 0x7f96900ced50 2024-03-06T07:29:33.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.522+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96900cc400 con 0x7f96900ced50 2024-03-06T07:29:33.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.523+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96900cc9c0 con 0x7f96900ced50 2024-03-06T07:29:33.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.524+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690161690 con 0x7f96900ced50 2024-03-06T07:29:33.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.525+0000 7f9687e00640 1 -- 172.21.15.114:0/1946138783 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f967c01a070 con 0x7f96900ced50 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.525+0000 7f9687e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f967006b500 0x7f967006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.525+0000 7f9687e00640 1 -- 172.21.15.114:0/1946138783 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f967c08a080 con 0x7f96900ced50 2024-03-06T07:29:33.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.526+0000 7f968fe00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f967006b500 0x7f967006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.526+0000 7f968fe00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f967006b500 0x7f967006d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9680007fd0 tx=0x7f9680009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.530+0000 7f9687e00640 1 -- 172.21.15.114:0/1946138783 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9690161690 con 0x7f96900ced50 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.724+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f969004f330 con 0x7f96900ced50 2024-03-06T07:29:33.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.725+0000 7f9687e00640 1 -- 172.21.15.114:0/1946138783 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f967c058ef0 con 0x7f96900ced50 2024-03-06T07:29:33.726 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f967006b500 msgr2=0x7f967006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f967006b500 0x7f967006d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9680007fd0 tx=0x7f9680009040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96900ced50 msgr2=0x7f96900cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96900ced50 0x7f96900cb9b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f967c00e870 tx=0x7f967c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 shutdown_connections 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f969015f490 0x7f96900d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f967006b500 0x7f967006d9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96900ced50 0x7f96900cb9b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 --2- 172.21.15.114:0/1946138783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96900b4de0 0x7f969015ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 >> 172.21.15.114:0/1946138783 conn(0x7f96900d3ed0 msgr2=0x7f969006c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 shutdown_connections 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.727+0000 7f9696e00640 1 -- 172.21.15.114:0/1946138783 wait complete. 2024-03-06T07:29:33.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:33.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.953+0000 7ff169600640 1 Processor -- start 2024-03-06T07:29:33.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.953+0000 7ff169600640 1 -- start start 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.955+0000 7ff169600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff16415df80 0x7ff16406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff169600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1640d1fb0 con 0x7ff1640d02a0 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff169600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16409b100 con 0x7ff16415e360 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff169600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1640d2a20 con 0x7ff16415df80 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff162a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff16415df80 0x7ff16406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff162a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff16415df80 0x7ff16406c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36174/0 (socket says 172.21.15.114:36174) 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff162a00640 1 -- 172.21.15.114:0/1892024996 learned_addr learned my addr 172.21.15.114:0/1892024996 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff163400640 1 --1- 172.21.15.114:0/1892024996 >> v1:172.21.15.114:6789/0 conn(0x7ff1640d02a0 0x7ff16406ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49124/0 (socket says 172.21.15.114:49124) 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff162a00640 1 -- 172.21.15.114:0/1892024996 >> v1:172.21.15.177:6789/0 conn(0x7ff16415e360 legacy=0x7ff1640cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff162a00640 1 -- 172.21.15.114:0/1892024996 >> v1:172.21.15.114:6789/0 conn(0x7ff1640d02a0 legacy=0x7ff16406ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff162a00640 1 -- 172.21.15.114:0/1892024996 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16409b3a0 con 0x7ff16415df80 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.956+0000 7ff162000640 1 -- 172.21.15.114:0/1892024996 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961445709 0 0) 0x7ff1640d1fb0 con 0x7ff1640d02a0 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.957+0000 7ff162a00640 1 --2- 172.21.15.114:0/1892024996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff16406c0d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff16406cbb0 tx=0x7ff15402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da5f199fc9b0b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.958+0000 7ff162000640 1 -- 172.21.15.114:0/1892024996 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154035020 con 0x7ff16415df80 2024-03-06T07:29:33.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.958+0000 7ff162000640 1 -- 172.21.15.114:0/1892024996 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff154031640 con 0x7ff16415df80 2024-03-06T07:29:33.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.958+0000 7ff162000640 1 -- 172.21.15.114:0/1892024996 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15402d040 con 0x7ff16415df80 2024-03-06T07:29:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.958+0000 7ff169600640 1 -- 172.21.15.114:0/1892024996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 msgr2=0x7ff16406c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.958+0000 7ff169600640 1 --2- 172.21.15.114:0/1892024996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff16406c0d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff16406cbb0 tx=0x7ff15402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.959+0000 7ff169600640 1 -- 172.21.15.114:0/1892024996 shutdown_connections 2024-03-06T07:29:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.959+0000 7ff169600640 1 --2- 172.21.15.114:0/1892024996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff16406c0d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.959+0000 7ff169600640 1 -- 172.21.15.114:0/1892024996 >> 172.21.15.114:0/1892024996 conn(0x7ff1640b7320 msgr2=0x7ff16409a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.959+0000 7ff169600640 1 -- 172.21.15.114:0/1892024996 shutdown_connections 2024-03-06T07:29:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.959+0000 7ff169600640 1 -- 172.21.15.114:0/1892024996 wait complete. 2024-03-06T07:29:33.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.960+0000 7ff169600640 1 Processor -- start 2024-03-06T07:29:33.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.960+0000 7ff169600640 1 -- start start 2024-03-06T07:29:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.961+0000 7ff169600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1640d02a0 0x7ff1640b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.961+0000 7ff169600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff1640affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.961+0000 7ff169600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff16415e360 0x7ff1641531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.961+0000 7ff169600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16406c610 con 0x7ff16415df80 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.961+0000 7ff169600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1640d2500 con 0x7ff16415e360 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.961+0000 7ff169600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16409b680 con 0x7ff1640d02a0 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.961+0000 7ff15a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff1640affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff1640affb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36186/0 (socket says 172.21.15.114:36186) 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 -- 172.21.15.114:0/4183365012 learned_addr learned my addr 172.21.15.114:0/4183365012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 -- 172.21.15.114:0/4183365012 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1640d02a0 msgr2=0x7ff1640b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1640d02a0 0x7ff1640b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff163400640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff16415e360 0x7ff1641531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 -- 172.21.15.114:0/4183365012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff16415e360 msgr2=0x7ff1641531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff16415e360 0x7ff1641531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 -- 172.21.15.114:0/4183365012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1640b6f90 con 0x7ff16415df80 2024-03-06T07:29:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15a000640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff1640affb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00c430 tx=0x7ff14c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.962+0000 7ff15b400640 1 -- 172.21.15.114:0/4183365012 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c011790 con 0x7ff16415df80 2024-03-06T07:29:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.963+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1640b49b0 con 0x7ff16415df80 2024-03-06T07:29:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.963+0000 7ff15b400640 1 -- 172.21.15.114:0/4183365012 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c017070 con 0x7ff16415df80 2024-03-06T07:29:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.963+0000 7ff15b400640 1 -- 172.21.15.114:0/4183365012 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c011bb0 con 0x7ff16415df80 2024-03-06T07:29:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.963+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1640b4f70 con 0x7ff16415df80 2024-03-06T07:29:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.964+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1640b5940 con 0x7ff16415df80 2024-03-06T07:29:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.965+0000 7ff15b400640 1 -- 172.21.15.114:0/4183365012 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff14c01c460 con 0x7ff16415df80 2024-03-06T07:29:33.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.965+0000 7ff15b400640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff14006b4d0 0x7ff14006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.966+0000 7ff15b400640 1 -- 172.21.15.114:0/4183365012 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff14c08a170 con 0x7ff16415df80 2024-03-06T07:29:33.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.966+0000 7ff162a00640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff14006b4d0 0x7ff14006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.966+0000 7ff162a00640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff14006b4d0 0x7ff14006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff154036000 tx=0x7ff154030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:33.970+0000 7ff15b400640 1 -- 172.21.15.114:0/4183365012 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff14c058f60 con 0x7ff16415df80 2024-03-06T07:29:34.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.177+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff16404f330 con 0x7ff16415df80 2024-03-06T07:29:34.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.177+0000 7ff15b400640 1 -- 172.21.15.114:0/4183365012 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff14c058860 con 0x7ff16415df80 2024-03-06T07:29:34.180 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff14006b4d0 msgr2=0x7ff14006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff14006b4d0 0x7ff14006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff154036000 tx=0x7ff154030000 comp rx=0 tx=0).stop 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 msgr2=0x7ff1640affb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff1640affb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00c430 tx=0x7ff14c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 shutdown_connections 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff16415e360 0x7ff1641531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff14006b4d0 0x7ff14006d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff16415df80 0x7ff1640affb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 --2- 172.21.15.114:0/4183365012 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff1640d02a0 0x7ff1640b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 >> 172.21.15.114:0/4183365012 conn(0x7ff1640b7320 msgr2=0x7ff16415c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 shutdown_connections 2024-03-06T07:29:34.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.180+0000 7ff169600640 1 -- 172.21.15.114:0/4183365012 wait complete. 2024-03-06T07:29:34.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:34.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.402+0000 7f24ee800640 1 Processor -- start 2024-03-06T07:29:34.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.403+0000 7f24ee800640 1 -- start start 2024-03-06T07:29:34.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.404+0000 7f24ee800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f24e80b5490 0x7f24e80cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.404+0000 7f24ee800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e80b3730 con 0x7f24e815e450 2024-03-06T07:29:34.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.404+0000 7f24ee800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e8067b20 con 0x7f24e80d07f0 2024-03-06T07:29:34.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.404+0000 7f24ee800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e80d4930 con 0x7f24e80b5490 2024-03-06T07:29:34.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.405+0000 7f24e7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f24e80b5490 0x7f24e80cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.405+0000 7f24dfe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f24e815e450 0x7f24e80cea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49134/0 (socket says 172.21.15.114:49134) 2024-03-06T07:29:34.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.405+0000 7f24dfe00640 1 -- 172.21.15.114:0/3098406559 learned_addr learned my addr 172.21.15.114:0/3098406559 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:34.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.405+0000 7f24e6a00640 1 -- 172.21.15.114:0/3098406559 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173004214 0 0) 0x7f24e80b3730 con 0x7f24e815e450 2024-03-06T07:29:34.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.405+0000 7f24e7e00640 1 -- 172.21.15.114:0/3098406559 >> v1:172.21.15.177:6789/0 conn(0x7f24e80d07f0 legacy=0x7f24e80d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.405+0000 7f24e7e00640 1 -- 172.21.15.114:0/3098406559 >> v1:172.21.15.114:6789/0 conn(0x7f24e815e450 legacy=0x7f24e80cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.405+0000 7f24e7e00640 1 -- 172.21.15.114:0/3098406559 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80d4b80 con 0x7f24e80b5490 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.406+0000 7f24e7e00640 1 --2- 172.21.15.114:0/3098406559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e80b5490 0x7f24e80cf130 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009d90 tx=0x7f24e002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf01b0f9b45ada server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24e6a00640 1 -- 172.21.15.114:0/3098406559 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0039070 con 0x7f24e80b5490 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24e6a00640 1 -- 172.21.15.114:0/3098406559 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e002cbd0 con 0x7f24e80b5490 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24e6a00640 1 -- 172.21.15.114:0/3098406559 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0034440 con 0x7f24e80b5490 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24ee800640 1 -- 172.21.15.114:0/3098406559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e80b5490 msgr2=0x7f24e80cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24ee800640 1 --2- 172.21.15.114:0/3098406559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e80b5490 0x7f24e80cf130 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009d90 tx=0x7f24e002be20 comp rx=0 tx=0).stop 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24ee800640 1 -- 172.21.15.114:0/3098406559 shutdown_connections 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24ee800640 1 --2- 172.21.15.114:0/3098406559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e80b5490 0x7f24e80cf130 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24ee800640 1 -- 172.21.15.114:0/3098406559 >> 172.21.15.114:0/3098406559 conn(0x7f24e80d3e10 msgr2=0x7f24e80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24ee800640 1 -- 172.21.15.114:0/3098406559 shutdown_connections 2024-03-06T07:29:34.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.407+0000 7f24ee800640 1 -- 172.21.15.114:0/3098406559 wait complete. 2024-03-06T07:29:34.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.409+0000 7f24ee800640 1 Processor -- start 2024-03-06T07:29:34.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.409+0000 7f24ee800640 1 -- start start 2024-03-06T07:29:34.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24ee800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24e80b5490 0x7f24e81586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24ee800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e80d07f0 0x7f24e80b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24ee800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e815e450 0x7f24e80b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24ee800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80b3f60 con 0x7f24e815e450 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24ee800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80b3c80 con 0x7f24e80d07f0 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24ee800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80d1bf0 con 0x7f24e80b5490 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24e7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e815e450 0x7f24e80b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24e7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e815e450 0x7f24e80b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36206/0 (socket says 172.21.15.114:36206) 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.410+0000 7f24e7e00640 1 -- 172.21.15.114:0/2704877316 learned_addr learned my addr 172.21.15.114:0/2704877316 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24e7e00640 1 -- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24e80b5490 msgr2=0x7f24e81586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24e7400640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24e80b5490 0x7f24e81586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24dfe00640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e80d07f0 0x7f24e80b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24e7e00640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24e80b5490 0x7f24e81586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24e7e00640 1 -- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e80d07f0 msgr2=0x7f24e80b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24e7e00640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e80d07f0 0x7f24e80b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24e7e00640 1 -- 172.21.15.114:0/2704877316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80cdb50 con 0x7f24e815e450 2024-03-06T07:29:34.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24dfe00640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e80d07f0 0x7f24e80b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:34.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24e7e00640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e815e450 0x7f24e80b1410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009eb0 tx=0x7f24e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24df400640 1 -- 172.21.15.114:0/2704877316 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0044030 con 0x7f24e815e450 2024-03-06T07:29:34.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.411+0000 7f24df400640 1 -- 172.21.15.114:0/2704877316 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e002c9d0 con 0x7f24e815e450 2024-03-06T07:29:34.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.412+0000 7f24df400640 1 -- 172.21.15.114:0/2704877316 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0039040 con 0x7f24e815e450 2024-03-06T07:29:34.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.412+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e8158be0 con 0x7f24e815e450 2024-03-06T07:29:34.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.412+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e80cbda0 con 0x7f24e815e450 2024-03-06T07:29:34.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.413+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e804f330 con 0x7f24e815e450 2024-03-06T07:29:34.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.414+0000 7f24df400640 1 -- 172.21.15.114:0/2704877316 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24e0047070 con 0x7f24e815e450 2024-03-06T07:29:34.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.415+0000 7f24df400640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f24c406b4d0 0x7f24c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.415+0000 7f24df400640 1 -- 172.21.15.114:0/2704877316 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24e00acbc0 con 0x7f24e815e450 2024-03-06T07:29:34.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.415+0000 7f24e7400640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f24c406b4d0 0x7f24c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.415+0000 7f24e7400640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f24c406b4d0 0x7f24c406d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f24d8003d90 tx=0x7f24d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.419+0000 7f24df400640 1 -- 172.21.15.114:0/2704877316 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24e004e070 con 0x7f24e815e450 2024-03-06T07:29:34.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.614+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24e80ce2d0 con 0x7f24e815e450 2024-03-06T07:29:34.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.615+0000 7f24df400640 1 -- 172.21.15.114:0/2704877316 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24e80cdb50 con 0x7f24e815e450 2024-03-06T07:29:34.618 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f24c406b4d0 msgr2=0x7f24c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f24c406b4d0 0x7f24c406d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f24d8003d90 tx=0x7f24d8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e815e450 msgr2=0x7f24e80b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e815e450 0x7f24e80b1410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009eb0 tx=0x7f24e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 shutdown_connections 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f24c406b4d0 0x7f24c406d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24e815e450 0x7f24e80b1410 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e80d07f0 0x7f24e80b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 --2- 172.21.15.114:0/2704877316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24e80b5490 0x7f24e81586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 >> 172.21.15.114:0/2704877316 conn(0x7f24e80d3e10 msgr2=0x7f24e80d41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 shutdown_connections 2024-03-06T07:29:34.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.618+0000 7f24ee800640 1 -- 172.21.15.114:0/2704877316 wait complete. 2024-03-06T07:29:34.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:34.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.846+0000 7f1209200640 1 Processor -- start 2024-03-06T07:29:34.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.846+0000 7f1209200640 1 -- start start 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.847+0000 7f1209200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12040af820 0x7f12040b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.847+0000 7f1209200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f12040b38f0 0x7f120406b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.847+0000 7f1209200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040b3da0 con 0x7f120415e160 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.847+0000 7f1209200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120406b0f0 con 0x7f12040af820 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.847+0000 7f1209200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040b5620 con 0x7f12040b38f0 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1201400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12040af820 0x7f12040b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1201400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12040af820 0x7f12040b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36222/0 (socket says 172.21.15.114:36222) 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1201400640 1 -- 172.21.15.114:0/3099337645 learned_addr learned my addr 172.21.15.114:0/3099337645 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1202800640 1 --2- 172.21.15.114:0/3099337645 >> v2:172.21.15.177:3300/0 conn(0x7f12040b38f0 0x7f120406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1201400640 1 -- 172.21.15.114:0/3099337645 >> v2:172.21.15.177:3300/0 conn(0x7f12040b38f0 msgr2=0x7f120406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1201400640 1 --2- 172.21.15.114:0/3099337645 >> v2:172.21.15.177:3300/0 conn(0x7f12040b38f0 0x7f120406b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1201400640 1 -- 172.21.15.114:0/3099337645 >> v1:172.21.15.177:6789/0 conn(0x7f120415e160 legacy=0x7f1204162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.848+0000 7f1201400640 1 -- 172.21.15.114:0/3099337645 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120406f1f0 con 0x7f12040af820 2024-03-06T07:29:34.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.850+0000 7f1201400640 1 --2- 172.21.15.114:0/3099337645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12040af820 0x7f12040b33b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f120406e460 tx=0x7f11f002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd204ec6d3e36eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.850+0000 7f1200a00640 1 -- 172.21.15.114:0/3099337645 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f002e070 con 0x7f12040af820 2024-03-06T07:29:34.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.850+0000 7f1200a00640 1 -- 172.21.15.114:0/3099337645 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f0037070 con 0x7f12040af820 2024-03-06T07:29:34.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.850+0000 7f1200a00640 1 -- 172.21.15.114:0/3099337645 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0031a00 con 0x7f12040af820 2024-03-06T07:29:34.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 -- 172.21.15.114:0/3099337645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12040af820 msgr2=0x7f12040b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 --2- 172.21.15.114:0/3099337645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12040af820 0x7f12040b33b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f120406e460 tx=0x7f11f002b880 comp rx=0 tx=0).stop 2024-03-06T07:29:34.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 -- 172.21.15.114:0/3099337645 shutdown_connections 2024-03-06T07:29:34.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 --2- 172.21.15.114:0/3099337645 >> v2:172.21.15.177:3300/0 conn(0x7f12040b38f0 0x7f120406b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 --2- 172.21.15.114:0/3099337645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12040af820 0x7f12040b33b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 -- 172.21.15.114:0/3099337645 >> 172.21.15.114:0/3099337645 conn(0x7f12040d0340 msgr2=0x7f12040d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 -- 172.21.15.114:0/3099337645 shutdown_connections 2024-03-06T07:29:34.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.851+0000 7f1209200640 1 -- 172.21.15.114:0/3099337645 wait complete. 2024-03-06T07:29:34.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.852+0000 7f1209200640 1 Processor -- start 2024-03-06T07:29:34.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.853+0000 7f1209200640 1 -- start start 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.853+0000 7f1209200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12040af820 0x7f1204152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.853+0000 7f1209200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12040b38f0 0x7f12041f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.853+0000 7f1209200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f120415e160 0x7f120415e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.853+0000 7f1209200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120406ef10 con 0x7f120415e160 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.853+0000 7f1209200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120406ec30 con 0x7f12040b38f0 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1209200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120406bac0 con 0x7f12040af820 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f120415e160 0x7f120415e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f120415e160 0x7f120415e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36238/0 (socket says 172.21.15.114:36238) 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 -- 172.21.15.114:0/3381437529 learned_addr learned my addr 172.21.15.114:0/3381437529 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:34.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1201400640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12040b38f0 0x7f12041f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1201e00640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12040af820 0x7f1204152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 -- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12040af820 msgr2=0x7f1204152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12040af820 0x7f1204152e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 -- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12040b38f0 msgr2=0x7f12041f7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12040b38f0 0x7f12041f7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.854+0000 7f1202800640 1 -- 172.21.15.114:0/3381437529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204154300 con 0x7f120415e160 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.855+0000 7f1202800640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f120415e160 0x7f120415e540 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f11fc007dc0 tx=0x7f11fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.855+0000 7f11fa000640 1 -- 172.21.15.114:0/3381437529 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc014420 con 0x7f120415e160 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.855+0000 7f11fa000640 1 -- 172.21.15.114:0/3381437529 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc019070 con 0x7f120415e160 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.855+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12041f7fc0 con 0x7f120415e160 2024-03-06T07:29:34.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.855+0000 7f11fa000640 1 -- 172.21.15.114:0/3381437529 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc014820 con 0x7f120415e160 2024-03-06T07:29:34.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.855+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12041f8580 con 0x7f120415e160 2024-03-06T07:29:34.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.856+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1204074ae0 con 0x7f120415e160 2024-03-06T07:29:34.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.857+0000 7f11fa000640 1 -- 172.21.15.114:0/3381437529 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11fc0149c0 con 0x7f120415e160 2024-03-06T07:29:34.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.858+0000 7f11fa000640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11e406b4d0 0x7f11e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.858+0000 7f11fa000640 1 -- 172.21.15.114:0/3381437529 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11fc08ab00 con 0x7f120415e160 2024-03-06T07:29:34.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.858+0000 7f1201e00640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11e406b4d0 0x7f11e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.859+0000 7f1201e00640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11e406b4d0 0x7f11e406d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f11f4006fd0 tx=0x7f11f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:34.863+0000 7f11fa000640 1 -- 172.21.15.114:0/3381437529 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11fc0598e0 con 0x7f120415e160 2024-03-06T07:29:35.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.097+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1204153490 con 0x7f120415e160 2024-03-06T07:29:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.098+0000 7f11fa000640 1 -- 172.21.15.114:0/3381437529 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f11fc0591e0 con 0x7f120415e160 2024-03-06T07:29:35.101 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:35.101 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:35.101 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11e406b4d0 msgr2=0x7f11e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11e406b4d0 0x7f11e406d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f11f4006fd0 tx=0x7f11f4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f120415e160 msgr2=0x7f120415e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f120415e160 0x7f120415e540 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f11fc007dc0 tx=0x7f11fc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 shutdown_connections 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11e406b4d0 0x7f11e406d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f120415e160 0x7f120415e540 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12040b38f0 0x7f12041f7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 --2- 172.21.15.114:0/3381437529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12040af820 0x7f1204152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 >> 172.21.15.114:0/3381437529 conn(0x7f12040d0340 msgr2=0x7f120406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 shutdown_connections 2024-03-06T07:29:35.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.101+0000 7f1209200640 1 -- 172.21.15.114:0/3381437529 wait complete. 2024-03-06T07:29:35.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:35.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.316+0000 7fad2d200640 1 Processor -- start 2024-03-06T07:29:35.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.316+0000 7fad2d200640 1 -- start start 2024-03-06T07:29:35.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.317+0000 7fad2d200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fad280b4680 0x7fad2806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad2d200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad28072340 0x7fad2806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad2d200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2815b090 con 0x7fad2815acb0 2024-03-06T07:29:35.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad2d200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad280b2a30 con 0x7fad28072340 2024-03-06T07:29:35.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad2d200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad280b4360 con 0x7fad280b4680 2024-03-06T07:29:35.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad26800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad28072340 0x7fad2806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad26800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad28072340 0x7fad2806ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36254/0 (socket says 172.21.15.114:36254) 2024-03-06T07:29:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad26800640 1 -- 172.21.15.114:0/330787160 learned_addr learned my addr 172.21.15.114:0/330787160 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad1d400640 1 --2- 172.21.15.114:0/330787160 >> v2:172.21.15.177:3300/0 conn(0x7fad280b4680 0x7fad2806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad26800640 1 -- 172.21.15.114:0/330787160 >> v2:172.21.15.177:3300/0 conn(0x7fad280b4680 msgr2=0x7fad2806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad26800640 1 --2- 172.21.15.114:0/330787160 >> v2:172.21.15.177:3300/0 conn(0x7fad280b4680 0x7fad2806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.318+0000 7fad26800640 1 -- 172.21.15.114:0/330787160 >> v1:172.21.15.177:6789/0 conn(0x7fad2815acb0 legacy=0x7fad280b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.319+0000 7fad26800640 1 -- 172.21.15.114:0/330787160 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad280b2cf0 con 0x7fad28072340 2024-03-06T07:29:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.320+0000 7fad26800640 1 --2- 172.21.15.114:0/330787160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad28072340 0x7fad2806ea30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fad28068560 tx=0x7fad2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0738b82c1cd711e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad25400640 1 -- 172.21.15.114:0/330787160 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad20035020 con 0x7fad28072340 2024-03-06T07:29:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad25400640 1 -- 172.21.15.114:0/330787160 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad20031870 con 0x7fad28072340 2024-03-06T07:29:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad25400640 1 -- 172.21.15.114:0/330787160 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad20031c40 con 0x7fad28072340 2024-03-06T07:29:35.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 -- 172.21.15.114:0/330787160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad28072340 msgr2=0x7fad2806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 --2- 172.21.15.114:0/330787160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad28072340 0x7fad2806ea30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fad28068560 tx=0x7fad2002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:35.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 -- 172.21.15.114:0/330787160 shutdown_connections 2024-03-06T07:29:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 --2- 172.21.15.114:0/330787160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad28072340 0x7fad2806ea30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 --2- 172.21.15.114:0/330787160 >> v2:172.21.15.177:3300/0 conn(0x7fad280b4680 0x7fad2806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 -- 172.21.15.114:0/330787160 >> 172.21.15.114:0/330787160 conn(0x7fad280b1730 msgr2=0x7fad280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 -- 172.21.15.114:0/330787160 shutdown_connections 2024-03-06T07:29:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.321+0000 7fad2d200640 1 -- 172.21.15.114:0/330787160 wait complete. 2024-03-06T07:29:35.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.323+0000 7fad2d200640 1 Processor -- start 2024-03-06T07:29:35.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.323+0000 7fad2d200640 1 -- start start 2024-03-06T07:29:35.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad2d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad28072340 0x7fad2814d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad2d200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad280b4680 0x7fad2813bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad2d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2815acb0 0x7fad2813c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad2d200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad280b4010 con 0x7fad2815acb0 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad2d200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad280b32b0 con 0x7fad28072340 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad2d200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad280ccfb0 con 0x7fad280b4680 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad26800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2815acb0 0x7fad2813c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad26800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2815acb0 0x7fad2813c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36264/0 (socket says 172.21.15.114:36264) 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.324+0000 7fad26800640 1 -- 172.21.15.114:0/1786176820 learned_addr learned my addr 172.21.15.114:0/1786176820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad25e00640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad28072340 0x7fad2814d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad1d400640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad280b4680 0x7fad2813bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad26800640 1 -- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad280b4680 msgr2=0x7fad2813bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad26800640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad280b4680 0x7fad2813bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad26800640 1 -- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad28072340 msgr2=0x7fad2814d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad26800640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad28072340 0x7fad2814d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad26800640 1 -- 172.21.15.114:0/1786176820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad28154ee0 con 0x7fad2815acb0 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad25e00640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad28072340 0x7fad2814d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.325+0000 7fad26800640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2815acb0 0x7fad2813c060 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fad20009960 tx=0x7fad200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.326+0000 7fad1ea00640 1 -- 172.21.15.114:0/1786176820 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad20040070 con 0x7fad2815acb0 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.326+0000 7fad1ea00640 1 -- 172.21.15.114:0/1786176820 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad20030d00 con 0x7fad2815acb0 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.326+0000 7fad1ea00640 1 -- 172.21.15.114:0/1786176820 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad20035040 con 0x7fad2815acb0 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.326+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad281551c0 con 0x7fad2815acb0 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.326+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad28155780 con 0x7fad2815acb0 2024-03-06T07:29:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.327+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2806df30 con 0x7fad2815acb0 2024-03-06T07:29:35.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.329+0000 7fad1ea00640 1 -- 172.21.15.114:0/1786176820 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad20038020 con 0x7fad2815acb0 2024-03-06T07:29:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.330+0000 7fad1ea00640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fad0406b4d0 0x7fad0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.330+0000 7fad1ea00640 1 -- 172.21.15.114:0/1786176820 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad200ac050 con 0x7fad2815acb0 2024-03-06T07:29:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.330+0000 7fad25e00640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fad0406b4d0 0x7fad0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.331+0000 7fad25e00640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fad0406b4d0 0x7fad0406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fad18007e90 tx=0x7fad180073a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.335+0000 7fad1ea00640 1 -- 172.21.15.114:0/1786176820 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad2806df30 con 0x7fad2815acb0 2024-03-06T07:29:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.530+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad2804f330 con 0x7fad2815acb0 2024-03-06T07:29:35.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.532+0000 7fad1ea00640 1 -- 172.21.15.114:0/1786176820 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16717 (secure 0 0 0) 0x7fad2007aed0 con 0x7fad2815acb0 2024-03-06T07:29:35.535 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:35.535 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":33,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:05.853392+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:02.138886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:03.320986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:35.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fad0406b4d0 msgr2=0x7fad0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fad0406b4d0 0x7fad0406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fad18007e90 tx=0x7fad180073a0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2815acb0 msgr2=0x7fad2813c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2815acb0 0x7fad2813c060 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fad20009960 tx=0x7fad200046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 shutdown_connections 2024-03-06T07:29:35.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fad0406b4d0 0x7fad0406d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad2815acb0 0x7fad2813c060 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fad280b4680 0x7fad2813bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 --2- 172.21.15.114:0/1786176820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad28072340 0x7fad2814d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 >> 172.21.15.114:0/1786176820 conn(0x7fad280b1730 msgr2=0x7fad2815edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 shutdown_connections 2024-03-06T07:29:35.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.535+0000 7fad2d200640 1 -- 172.21.15.114:0/1786176820 wait complete. 2024-03-06T07:29:35.556 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:35.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:35.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.759+0000 7f79be400640 1 Processor -- start 2024-03-06T07:29:35.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.759+0000 7f79be400640 1 -- start start 2024-03-06T07:29:35.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79be400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79b80af820 0x7f79b80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79be400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f79b80b38f0 0x7f79b806b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79be400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b80b3da0 con 0x7f79b815e160 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79be400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b806b0f0 con 0x7f79b80af820 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79be400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b80b5620 con 0x7f79b80b38f0 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79aea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79b80af820 0x7f79b80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79aea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79b80af820 0x7f79b80b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36274/0 (socket says 172.21.15.114:36274) 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79b7400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f79b815e160 0x7f79b8162620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49142/0 (socket says 172.21.15.114:49142) 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.761+0000 7f79aea00640 1 -- 172.21.15.114:0/2951929031 learned_addr learned my addr 172.21.15.114:0/2951929031 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.762+0000 7f79b7e00640 1 --2- 172.21.15.114:0/2951929031 >> v2:172.21.15.177:3300/0 conn(0x7f79b80b38f0 0x7f79b806b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.762+0000 7f79aea00640 1 -- 172.21.15.114:0/2951929031 >> v2:172.21.15.177:3300/0 conn(0x7f79b80b38f0 msgr2=0x7f79b806b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.762+0000 7f79aea00640 1 --2- 172.21.15.114:0/2951929031 >> v2:172.21.15.177:3300/0 conn(0x7f79b80b38f0 0x7f79b806b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.762+0000 7f79aea00640 1 -- 172.21.15.114:0/2951929031 >> v1:172.21.15.114:6789/0 conn(0x7f79b815e160 legacy=0x7f79b8162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.762+0000 7f79aea00640 1 -- 172.21.15.114:0/2951929031 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b806f1f0 con 0x7f79b80af820 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.762+0000 7f79b6a00640 1 -- 172.21.15.114:0/2951929031 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115728780 0 0) 0x7f79a8001490 con 0x7f79b815e160 2024-03-06T07:29:35.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.762+0000 7f79b7e00640 1 --2- 172.21.15.114:0/2951929031 >> v2:172.21.15.177:3300/0 conn(0x7f79b80b38f0 0x7f79b806b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.763+0000 7f79aea00640 1 --2- 172.21.15.114:0/2951929031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80af820 0x7f79b80b33b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f79b806e460 tx=0x7f79a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eee08745f22c29d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.763+0000 7f79b6a00640 1 -- 172.21.15.114:0/2951929031 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0035020 con 0x7f79b80af820 2024-03-06T07:29:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.763+0000 7f79b6a00640 1 -- 172.21.15.114:0/2951929031 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0031870 con 0x7f79b80af820 2024-03-06T07:29:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.763+0000 7f79b6a00640 1 -- 172.21.15.114:0/2951929031 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0031c40 con 0x7f79b80af820 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 -- 172.21.15.114:0/2951929031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80af820 msgr2=0x7f79b80b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 --2- 172.21.15.114:0/2951929031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80af820 0x7f79b80b33b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f79b806e460 tx=0x7f79a002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 -- 172.21.15.114:0/2951929031 shutdown_connections 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 --2- 172.21.15.114:0/2951929031 >> v2:172.21.15.177:3300/0 conn(0x7f79b80b38f0 0x7f79b806b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 --2- 172.21.15.114:0/2951929031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80af820 0x7f79b80b33b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 -- 172.21.15.114:0/2951929031 >> 172.21.15.114:0/2951929031 conn(0x7f79b80d0340 msgr2=0x7f79b80d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 -- 172.21.15.114:0/2951929031 shutdown_connections 2024-03-06T07:29:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.764+0000 7f79be400640 1 -- 172.21.15.114:0/2951929031 wait complete. 2024-03-06T07:29:35.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.766+0000 7f79be400640 1 Processor -- start 2024-03-06T07:29:35.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.766+0000 7f79be400640 1 -- start start 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79be400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80af820 0x7f79b8144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79be400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80b38f0 0x7f79b8146f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79be400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b815e160 0x7f79b8147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79be400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b806ef10 con 0x7f79b80b38f0 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79be400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b806ec30 con 0x7f79b815e160 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79be400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b806bac0 con 0x7f79b80af820 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80b38f0 0x7f79b8146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80b38f0 0x7f79b8146f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36288/0 (socket says 172.21.15.114:36288) 2024-03-06T07:29:35.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79aea00640 1 -- 172.21.15.114:0/1118376529 learned_addr learned my addr 172.21.15.114:0/1118376529 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79b7400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80af820 0x7f79b8144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79b7e00640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b815e160 0x7f79b8147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79aea00640 1 -- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80af820 msgr2=0x7f79b8144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.767+0000 7f79aea00640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80af820 0x7f79b8144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79aea00640 1 -- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b815e160 msgr2=0x7f79b8147480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79aea00640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b815e160 0x7f79b8147480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79aea00640 1 -- 172.21.15.114:0/1118376529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b8146c80 con 0x7f79b80b38f0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79b7e00640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b815e160 0x7f79b8147480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79aea00640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80b38f0 0x7f79b8146f40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f79a0000c00 tx=0x7f79a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79afe00640 1 -- 172.21.15.114:0/1118376529 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0035040 con 0x7f79b80b38f0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79afe00640 1 -- 172.21.15.114:0/1118376529 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0045480 con 0x7f79b80b38f0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79afe00640 1 -- 172.21.15.114:0/1118376529 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0045870 con 0x7f79b80b38f0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b8145be0 con 0x7f79b80b38f0 2024-03-06T07:29:35.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.768+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b81461a0 con 0x7f79b80b38f0 2024-03-06T07:29:35.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.769+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8074ae0 con 0x7f79b80b38f0 2024-03-06T07:29:35.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.770+0000 7f79afe00640 1 -- 172.21.15.114:0/1118376529 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79a0038020 con 0x7f79b80b38f0 2024-03-06T07:29:35.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.771+0000 7f79afe00640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799406b4d0 0x7f799406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.771+0000 7f79afe00640 1 -- 172.21.15.114:0/1118376529 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79a00ac710 con 0x7f79b80b38f0 2024-03-06T07:29:35.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.772+0000 7f79b7400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799406b4d0 0x7f799406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.772+0000 7f79b7400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799406b4d0 0x7f799406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f79a8007ed0 tx=0x7f79a8007e60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:35.776+0000 7f79afe00640 1 -- 172.21.15.114:0/1118376529 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79a007b580 con 0x7f79b80b38f0 2024-03-06T07:29:36.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.022+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b804f330 con 0x7f79b80b38f0 2024-03-06T07:29:36.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.023+0000 7f79afe00640 1 -- 172.21.15.114:0/1118376529 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f79a007ae80 con 0x7f79b80b38f0 2024-03-06T07:29:36.026 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:36.026 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:36.027 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799406b4d0 msgr2=0x7f799406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799406b4d0 0x7f799406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f79a8007ed0 tx=0x7f79a8007e60 comp rx=0 tx=0).stop 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80b38f0 msgr2=0x7f79b8146f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80b38f0 0x7f79b8146f40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f79a0000c00 tx=0x7f79a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 shutdown_connections 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79b815e160 0x7f79b8147480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f799406b4d0 0x7f799406d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79b80b38f0 0x7f79b8146f40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 --2- 172.21.15.114:0/1118376529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79b80af820 0x7f79b8144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 >> 172.21.15.114:0/1118376529 conn(0x7f79b80d0340 msgr2=0x7f79b80b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 shutdown_connections 2024-03-06T07:29:36.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.026+0000 7f79be400640 1 -- 172.21.15.114:0/1118376529 wait complete. 2024-03-06T07:29:36.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.248+0000 7f7983400640 1 Processor -- start 2024-03-06T07:29:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.248+0000 7f7983400640 1 -- start start 2024-03-06T07:29:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.250+0000 7f7983400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c15acb0 0x7f797c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.250+0000 7f7983400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f797c06ec00 0x7f797c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.250+0000 7f7983400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c15b090 con 0x7f797c0b4680 2024-03-06T07:29:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.250+0000 7f7983400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797c0b2a30 con 0x7f797c06ec00 2024-03-06T07:29:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.250+0000 7f7983400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f797c0b4360 con 0x7f797c15acb0 2024-03-06T07:29:36.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.251+0000 7f7981400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f797c06ec00 0x7f797c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.251+0000 7f7980a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c15acb0 0x7f797c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.251+0000 7f7980a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c15acb0 0x7f797c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:55476/0 (socket says 172.21.15.114:55476) 2024-03-06T07:29:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.251+0000 7f7980a00640 1 -- 172.21.15.114:0/1835141154 learned_addr learned my addr 172.21.15.114:0/1835141154 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.251+0000 7f7981400640 1 -- 172.21.15.114:0/1835141154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c15acb0 msgr2=0x7f797c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.251+0000 7f7981400640 1 --2- 172.21.15.114:0/1835141154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c15acb0 0x7f797c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.252+0000 7f7981400640 1 -- 172.21.15.114:0/1835141154 >> v1:172.21.15.177:6789/0 conn(0x7f797c0b4680 legacy=0x7f797c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.252+0000 7f7981400640 1 -- 172.21.15.114:0/1835141154 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c0b2cf0 con 0x7f797c06ec00 2024-03-06T07:29:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.252+0000 7f797be00640 1 -- 172.21.15.114:0/1835141154 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740989172 0 0) 0x7f797c15b090 con 0x7f797c0b4680 2024-03-06T07:29:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.253+0000 7f7981400640 1 --2- 172.21.15.114:0/1835141154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c06ec00 0x7f797c0ccfb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7974009d90 tx=0x7f797402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f39aa4428e4d26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.253+0000 7f797be00640 1 -- 172.21.15.114:0/1835141154 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974039070 con 0x7f797c06ec00 2024-03-06T07:29:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.253+0000 7f797be00640 1 -- 172.21.15.114:0/1835141154 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797402cc50 con 0x7f797c06ec00 2024-03-06T07:29:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.253+0000 7f797be00640 1 -- 172.21.15.114:0/1835141154 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974034440 con 0x7f797c06ec00 2024-03-06T07:29:36.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 -- 172.21.15.114:0/1835141154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c06ec00 msgr2=0x7f797c0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 --2- 172.21.15.114:0/1835141154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c06ec00 0x7f797c0ccfb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7974009d90 tx=0x7f797402bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 -- 172.21.15.114:0/1835141154 shutdown_connections 2024-03-06T07:29:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 --2- 172.21.15.114:0/1835141154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c06ec00 0x7f797c0ccfb0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 --2- 172.21.15.114:0/1835141154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c15acb0 0x7f797c0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 -- 172.21.15.114:0/1835141154 >> 172.21.15.114:0/1835141154 conn(0x7f797c0b1730 msgr2=0x7f797c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 -- 172.21.15.114:0/1835141154 shutdown_connections 2024-03-06T07:29:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.254+0000 7f7983400640 1 -- 172.21.15.114:0/1835141154 wait complete. 2024-03-06T07:29:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.255+0000 7f7983400640 1 Processor -- start 2024-03-06T07:29:36.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.256+0000 7f7983400640 1 -- start start 2024-03-06T07:29:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7983400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c06ec00 0x7f797c145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7983400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f797c0b4680 0x7f797c0ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7983400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c15acb0 0x7f797c0cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7983400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c0b3d10 con 0x7f797c15acb0 2024-03-06T07:29:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7983400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c0b32b0 con 0x7f797c0b4680 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7983400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f797c0b4010 con 0x7f797c06ec00 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c15acb0 0x7f797c0cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c15acb0 0x7f797c0cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36318/0 (socket says 172.21.15.114:36318) 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 -- 172.21.15.114:0/1352649584 learned_addr learned my addr 172.21.15.114:0/1352649584 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7973e00640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f797c0b4680 0x7f797c0ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7980a00640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c06ec00 0x7f797c145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 -- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c06ec00 msgr2=0x7f797c145410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c06ec00 0x7f797c145410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 -- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f797c0b4680 msgr2=0x7f797c0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f797c0b4680 0x7f797c0ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.257+0000 7f7981400640 1 -- 172.21.15.114:0/1352649584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c0cd1c0 con 0x7f797c15acb0 2024-03-06T07:29:36.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.258+0000 7f7981400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c15acb0 0x7f797c0cef10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7974009d90 tx=0x7f7974004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.258+0000 7f7979600640 1 -- 172.21.15.114:0/1352649584 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974044020 con 0x7f797c15acb0 2024-03-06T07:29:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.258+0000 7f7979600640 1 -- 172.21.15.114:0/1352649584 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797403a040 con 0x7f797c15acb0 2024-03-06T07:29:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.258+0000 7f7979600640 1 -- 172.21.15.114:0/1352649584 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974039050 con 0x7f797c15acb0 2024-03-06T07:29:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.258+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c0cd4a0 con 0x7f797c15acb0 2024-03-06T07:29:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.258+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c0cda60 con 0x7f797c15acb0 2024-03-06T07:29:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.259+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c15a810 con 0x7f797c15acb0 2024-03-06T07:29:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.260+0000 7f7979600640 1 -- 172.21.15.114:0/1352649584 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7974003990 con 0x7f797c15acb0 2024-03-06T07:29:36.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.262+0000 7f7979600640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f795806b4d0 0x7f795806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.262+0000 7f7979600640 1 -- 172.21.15.114:0/1352649584 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79740ac6a0 con 0x7f797c15acb0 2024-03-06T07:29:36.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.262+0000 7f7980a00640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f795806b4d0 0x7f795806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.263+0000 7f7980a00640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f795806b4d0 0x7f795806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f796c009a00 tx=0x7f796c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.266+0000 7f7979600640 1 -- 172.21.15.114:0/1352649584 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f797c15a810 con 0x7f797c15acb0 2024-03-06T07:29:36.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.502+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f797c04f330 con 0x7f797c15acb0 2024-03-06T07:29:36.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.503+0000 7f7979600640 1 -- 172.21.15.114:0/1352649584 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f797404e060 con 0x7f797c15acb0 2024-03-06T07:29:36.508 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:36.508 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:04.313533+0000","modified":"2024-03-06T07:29:07.379375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5088},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5088":{"gid":5088,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6837/3915705340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3915705340},{"type":"v1","addr":"172.21.15.114:6837","nonce":3915705340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5088,"qdb_cluster":[5088]},"id":2}]} 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f795806b4d0 msgr2=0x7f795806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f795806b4d0 0x7f795806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f796c009a00 tx=0x7f796c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c15acb0 msgr2=0x7f797c0cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c15acb0 0x7f797c0cef10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7974009d90 tx=0x7f7974004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 shutdown_connections 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f795806b4d0 0x7f795806d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797c15acb0 0x7f797c0cef10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f797c0b4680 0x7f797c0ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 --2- 172.21.15.114:0/1352649584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f797c06ec00 0x7f797c145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 >> 172.21.15.114:0/1352649584 conn(0x7f797c0b1730 msgr2=0x7f797c072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 shutdown_connections 2024-03-06T07:29:36.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.506+0000 7f7983400640 1 -- 172.21.15.114:0/1352649584 wait complete. 2024-03-06T07:29:36.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.720+0000 7f6a45e00640 1 Processor -- start 2024-03-06T07:29:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.720+0000 7f6a45e00640 1 -- start start 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a45e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a4015e0d0 0x7f6a400d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a45e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6a400d0fc0 0x7f6a40160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a45e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400cea60 con 0x7f6a4015f470 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a45e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400d2f60 con 0x7f6a4015e0d0 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a45e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400d2350 con 0x7f6a400d0fc0 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a3ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a4015e0d0 0x7f6a400d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a37e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6a4015f470 0x7f6a400d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49154/0 (socket says 172.21.15.114:49154) 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a37e00640 1 -- 172.21.15.114:0/2580845172 learned_addr learned my addr 172.21.15.114:0/2580845172 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:36.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a3ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a4015e0d0 0x7f6a400d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36320/0 (socket says 172.21.15.114:36320) 2024-03-06T07:29:36.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.722+0000 7f6a3f400640 1 --2- 172.21.15.114:0/2580845172 >> v2:172.21.15.177:3300/0 conn(0x7f6a400d0fc0 0x7f6a40160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.723+0000 7f6a3ea00640 1 -- 172.21.15.114:0/2580845172 >> v2:172.21.15.177:3300/0 conn(0x7f6a400d0fc0 msgr2=0x7f6a40160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.723+0000 7f6a3ea00640 1 --2- 172.21.15.114:0/2580845172 >> v2:172.21.15.177:3300/0 conn(0x7f6a400d0fc0 0x7f6a40160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.723+0000 7f6a3ea00640 1 -- 172.21.15.114:0/2580845172 >> v1:172.21.15.114:6789/0 conn(0x7f6a4015f470 legacy=0x7f6a400d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.723+0000 7f6a3ea00640 1 -- 172.21.15.114:0/2580845172 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400d2630 con 0x7f6a4015e0d0 2024-03-06T07:29:36.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.724+0000 7f6a3ea00640 1 --2- 172.21.15.114:0/2580845172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015e0d0 0x7f6a400d1a90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6a400caf20 tx=0x7f6a3002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b029dec1d91a6bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.724+0000 7f6a3e000640 1 -- 172.21.15.114:0/2580845172 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30034070 con 0x7f6a4015e0d0 2024-03-06T07:29:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.724+0000 7f6a3e000640 1 -- 172.21.15.114:0/2580845172 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a30030610 con 0x7f6a4015e0d0 2024-03-06T07:29:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.724+0000 7f6a3e000640 1 -- 172.21.15.114:0/2580845172 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a300309e0 con 0x7f6a4015e0d0 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 -- 172.21.15.114:0/2580845172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015e0d0 msgr2=0x7f6a400d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 --2- 172.21.15.114:0/2580845172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015e0d0 0x7f6a400d1a90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6a400caf20 tx=0x7f6a3002b960 comp rx=0 tx=0).stop 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 -- 172.21.15.114:0/2580845172 shutdown_connections 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 --2- 172.21.15.114:0/2580845172 >> v2:172.21.15.177:3300/0 conn(0x7f6a400d0fc0 0x7f6a40160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 --2- 172.21.15.114:0/2580845172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015e0d0 0x7f6a400d1a90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 -- 172.21.15.114:0/2580845172 >> 172.21.15.114:0/2580845172 conn(0x7f6a400d3ec0 msgr2=0x7f6a400d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 -- 172.21.15.114:0/2580845172 shutdown_connections 2024-03-06T07:29:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.725+0000 7f6a45e00640 1 -- 172.21.15.114:0/2580845172 wait complete. 2024-03-06T07:29:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.727+0000 7f6a45e00640 1 Processor -- start 2024-03-06T07:29:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.728+0000 7f6a45e00640 1 -- start start 2024-03-06T07:29:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.729+0000 7f6a45e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6a400d0fc0 0x7f6a4014a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.729+0000 7f6a45e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a4015e0d0 0x7f6a4014ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.729+0000 7f6a45e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015f470 0x7f6a400c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.729+0000 7f6a45e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4015fb30 con 0x7f6a4015f470 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.729+0000 7f6a45e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400d2bf0 con 0x7f6a4015e0d0 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.729+0000 7f6a45e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400d1fd0 con 0x7f6a400d0fc0 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015f470 0x7f6a400c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015f470 0x7f6a400c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36328/0 (socket says 172.21.15.114:36328) 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 -- 172.21.15.114:0/838541074 learned_addr learned my addr 172.21.15.114:0/838541074 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3ea00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6a400d0fc0 0x7f6a4014a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 -- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6a400d0fc0 msgr2=0x7f6a4014a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a37e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a4015e0d0 0x7f6a4014ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6a400d0fc0 0x7f6a4014a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 -- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a4015e0d0 msgr2=0x7f6a4014ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a4015e0d0 0x7f6a4014ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 -- 172.21.15.114:0/838541074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a401f3d40 con 0x7f6a4015f470 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a37e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a4015e0d0 0x7f6a4014ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.730+0000 7f6a3f400640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015f470 0x7f6a400c88e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6a38007e40 tx=0x7f6a3800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.731+0000 7f6a37400640 1 -- 172.21.15.114:0/838541074 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38014620 con 0x7f6a4015f470 2024-03-06T07:29:36.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.731+0000 7f6a37400640 1 -- 172.21.15.114:0/838541074 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a38019070 con 0x7f6a4015f470 2024-03-06T07:29:36.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.731+0000 7f6a37400640 1 -- 172.21.15.114:0/838541074 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38014a20 con 0x7f6a4015f470 2024-03-06T07:29:36.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.731+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a400c9330 con 0x7f6a4015f470 2024-03-06T07:29:36.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.731+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a400c9bd0 con 0x7f6a4015f470 2024-03-06T07:29:36.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.732+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4004f330 con 0x7f6a4015f470 2024-03-06T07:29:36.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.733+0000 7f6a37400640 1 -- 172.21.15.114:0/838541074 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a38014bc0 con 0x7f6a4015f470 2024-03-06T07:29:36.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.735+0000 7f6a37400640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.735+0000 7f6a37400640 1 -- 172.21.15.114:0/838541074 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a3808ab40 con 0x7f6a4015f470 2024-03-06T07:29:36.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.735+0000 7f6a3ea00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.736+0000 7f6a3ea00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6a30036000 tx=0x7f6a3003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.741+0000 7f6a37400640 1 -- 172.21.15.114:0/838541074 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a380599c0 con 0x7f6a4015f470 2024-03-06T07:29:36.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.946+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a4008d920 con 0x7f6a4015f470 2024-03-06T07:29:36.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.949+0000 7f6a37400640 1 -- 172.21.15.114:0/838541074 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f6a380592c0 con 0x7f6a4015f470 2024-03-06T07:29:36.952 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:36.952 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":33,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:05.853392+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:02.138886+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:03.320986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:36.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.951+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6a1c06b4d0 msgr2=0x7f6a1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.951+0000 7f6a45e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6a30036000 tx=0x7f6a3003b040 comp rx=0 tx=0).stop 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.951+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015f470 msgr2=0x7f6a400c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015f470 0x7f6a400c88e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6a38007e40 tx=0x7f6a3800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 shutdown_connections 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a4015f470 0x7f6a400c88e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a4015e0d0 0x7f6a4014ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 --2- 172.21.15.114:0/838541074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6a400d0fc0 0x7f6a4014a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 >> 172.21.15.114:0/838541074 conn(0x7f6a400d3ec0 msgr2=0x7f6a4015e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 shutdown_connections 2024-03-06T07:29:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:36.952+0000 7f6a45e00640 1 -- 172.21.15.114:0/838541074 wait complete. 2024-03-06T07:29:36.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:37.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.169+0000 7f27ffa00640 1 Processor -- start 2024-03-06T07:29:37.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.169+0000 7f27ffa00640 1 -- start start 2024-03-06T07:29:37.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27ffa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f27f815e360 0x7f27f80cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27ffa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27f80d12e0 0x7f27f8161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27ffa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f815e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27ffa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f80cf2b0 con 0x7f27f80d12e0 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27ffa00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f80cf590 con 0x7f27f815e360 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27ffa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f80cf870 con 0x7f27f8162270 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27efe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27f80d12e0 0x7f27f8161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27efe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27f80d12e0 0x7f27f8161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36342/0 (socket says 172.21.15.114:36342) 2024-03-06T07:29:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27efe00640 1 -- 172.21.15.114:0/3850996876 learned_addr learned my addr 172.21.15.114:0/3850996876 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.171+0000 7f27fd000640 1 --2- 172.21.15.114:0/3850996876 >> v2:172.21.15.177:3300/0 conn(0x7f27f815e360 0x7f27f80cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.172+0000 7f27fda00640 1 --2- 172.21.15.114:0/3850996876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f815e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.172+0000 7f27efe00640 1 -- 172.21.15.114:0/3850996876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 msgr2=0x7f27f815e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.172+0000 7f27efe00640 1 --2- 172.21.15.114:0/3850996876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f815e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.172+0000 7f27efe00640 1 -- 172.21.15.114:0/3850996876 >> v2:172.21.15.177:3300/0 conn(0x7f27f815e360 msgr2=0x7f27f80cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.172+0000 7f27efe00640 1 --2- 172.21.15.114:0/3850996876 >> v2:172.21.15.177:3300/0 conn(0x7f27f815e360 0x7f27f80cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.172+0000 7f27efe00640 1 -- 172.21.15.114:0/3850996876 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f80cfb50 con 0x7f27f80d12e0 2024-03-06T07:29:37.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.172+0000 7f27fda00640 1 --2- 172.21.15.114:0/3850996876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f815e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.173+0000 7f27efe00640 1 --2- 172.21.15.114:0/3850996876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f80d12e0 0x7f27f8161d30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f27f80d4fc0 tx=0x7f27e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8282fb8cc0d4729 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.173+0000 7f27f7e00640 1 -- 172.21.15.114:0/3850996876 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0034070 con 0x7f27f80d12e0 2024-03-06T07:29:37.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.173+0000 7f27f7e00640 1 -- 172.21.15.114:0/3850996876 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0030860 con 0x7f27f80d12e0 2024-03-06T07:29:37.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.173+0000 7f27f7e00640 1 -- 172.21.15.114:0/3850996876 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0030c30 con 0x7f27f80d12e0 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 -- 172.21.15.114:0/3850996876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f80d12e0 msgr2=0x7f27f8161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 --2- 172.21.15.114:0/3850996876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f80d12e0 0x7f27f8161d30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f27f80d4fc0 tx=0x7f27e002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 -- 172.21.15.114:0/3850996876 shutdown_connections 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 --2- 172.21.15.114:0/3850996876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f815e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 --2- 172.21.15.114:0/3850996876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f80d12e0 0x7f27f8161d30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 --2- 172.21.15.114:0/3850996876 >> v2:172.21.15.177:3300/0 conn(0x7f27f815e360 0x7f27f80cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 -- 172.21.15.114:0/3850996876 >> 172.21.15.114:0/3850996876 conn(0x7f27f80d3f50 msgr2=0x7f27f80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 -- 172.21.15.114:0/3850996876 shutdown_connections 2024-03-06T07:29:37.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.174+0000 7f27ffa00640 1 -- 172.21.15.114:0/3850996876 wait complete. 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.176+0000 7f27ffa00640 1 Processor -- start 2024-03-06T07:29:37.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.176+0000 7f27ffa00640 1 -- start start 2024-03-06T07:29:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27ffa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f80d12e0 0x7f27f8158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27ffa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f815e360 0x7f27f8161a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27ffa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f80cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27ffa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f80d18f0 con 0x7f27f815e360 2024-03-06T07:29:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27ffa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f80d1770 con 0x7f27f80d12e0 2024-03-06T07:29:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27ffa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f815ec80 con 0x7f27f8162270 2024-03-06T07:29:37.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27efe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f815e360 0x7f27f8161a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27efe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f815e360 0x7f27f8161a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36350/0 (socket says 172.21.15.114:36350) 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.177+0000 7f27efe00640 1 -- 172.21.15.114:0/2662074713 learned_addr learned my addr 172.21.15.114:0/2662074713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27fd000640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f80d12e0 0x7f27f8158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27efe00640 1 -- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 msgr2=0x7f27f80cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27fda00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f80cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27efe00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f80cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27efe00640 1 -- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f80d12e0 msgr2=0x7f27f8158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27efe00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f80d12e0 0x7f27f8158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27efe00640 1 -- 172.21.15.114:0/2662074713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f81609d0 con 0x7f27f815e360 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27efe00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f815e360 0x7f27f8161a70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000c00 tx=0x7f27e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27f5600640 1 -- 172.21.15.114:0/2662074713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0048070 con 0x7f27f815e360 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27f5600640 1 -- 172.21.15.114:0/2662074713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0030660 con 0x7f27f815e360 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.178+0000 7f27f5600640 1 -- 172.21.15.114:0/2662074713 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0034040 con 0x7f27f815e360 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.179+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f8160cb0 con 0x7f27f815e360 2024-03-06T07:29:37.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.179+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f8161270 con 0x7f27f815e360 2024-03-06T07:29:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.180+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f804f330 con 0x7f27f815e360 2024-03-06T07:29:37.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.181+0000 7f27f5600640 1 -- 172.21.15.114:0/2662074713 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27e004b070 con 0x7f27f815e360 2024-03-06T07:29:37.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.182+0000 7f27f5600640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f27d406b4d0 0x7f27d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.182+0000 7f27f5600640 1 -- 172.21.15.114:0/2662074713 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27e00ac560 con 0x7f27f815e360 2024-03-06T07:29:37.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.182+0000 7f27fd000640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f27d406b4d0 0x7f27d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.183+0000 7f27fd000640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f27d406b4d0 0x7f27d406d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f27e8004410 tx=0x7f27e80039f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.186+0000 7f27f5600640 1 -- 172.21.15.114:0/2662074713 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27e007b4f0 con 0x7f27f815e360 2024-03-06T07:29:37.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.422+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f27f808d6a0 con 0x7f27f815e360 2024-03-06T07:29:37.426 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:29:37.424+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:37.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.428+0000 7f27f5600640 1 -- 172.21.15.114:0/2662074713 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f27e007adf0 con 0x7f27f815e360 2024-03-06T07:29:37.430 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:37.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f27d406b4d0 msgr2=0x7f27d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f27d406b4d0 0x7f27d406d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f27e8004410 tx=0x7f27e80039f0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f815e360 msgr2=0x7f27f8161a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f815e360 0x7f27f8161a70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000c00 tx=0x7f27e0004730 comp rx=0 tx=0).stop 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 shutdown_connections 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27f8162270 0x7f27f80cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f27d406b4d0 0x7f27d406d990 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f27e8004410 tx=0x7f27e80039f0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27f815e360 0x7f27f8161a70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 --2- 172.21.15.114:0/2662074713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f80d12e0 0x7f27f8158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.431+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 >> 172.21.15.114:0/2662074713 conn(0x7f27f80d3f50 msgr2=0x7f27f815bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.432+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 shutdown_connections 2024-03-06T07:29:37.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.432+0000 7f27ffa00640 1 -- 172.21.15.114:0/2662074713 wait complete. 2024-03-06T07:29:37.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:37.507 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:29:37.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.634+0000 7f79de800640 1 Processor -- start 2024-03-06T07:29:37.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.634+0000 7f79de800640 1 -- start start 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.635+0000 7f79de800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79d80b3650 0x7f79d80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.635+0000 7f79de800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80cda90 0x7f79d806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.635+0000 7f79de800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d80719e0 con 0x7f79d80b3df0 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79de800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d8072310 con 0x7f79d80b3650 2024-03-06T07:29:37.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79de800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d8071cc0 con 0x7f79d80cda90 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79d80b3650 0x7f79d80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d7400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f79d80b3df0 0x7f79d80b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49176/0 (socket says 172.21.15.114:49176) 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f79d80b3650 0x7f79d80727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36380/0 (socket says 172.21.15.114:36380) 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d7400640 1 -- 172.21.15.114:0/377071015 learned_addr learned my addr 172.21.15.114:0/377071015 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:37.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d7e00640 1 --2- 172.21.15.114:0/377071015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80cda90 0x7f79d806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d6a00640 1 -- 172.21.15.114:0/377071015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80cda90 msgr2=0x7f79d806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d6a00640 1 --2- 172.21.15.114:0/377071015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80cda90 0x7f79d806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d6a00640 1 -- 172.21.15.114:0/377071015 >> v1:172.21.15.114:6789/0 conn(0x7f79d80b3df0 legacy=0x7f79d80b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d6a00640 1 -- 172.21.15.114:0/377071015 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8071fa0 con 0x7f79d80b3650 2024-03-06T07:29:37.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d7e00640 1 --2- 172.21.15.114:0/377071015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80cda90 0x7f79d806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:37.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.636+0000 7f79d6000640 1 -- 172.21.15.114:0/377071015 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129641377 0 0) 0x7f79d80719e0 con 0x7f79d80b3df0 2024-03-06T07:29:37.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.637+0000 7f79d6a00640 1 --2- 172.21.15.114:0/377071015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d80727a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f79c40097a0 tx=0x7f79c402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0b3a6ae62558f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79d6000640 1 -- 172.21.15.114:0/377071015 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4035020 con 0x7f79d80b3650 2024-03-06T07:29:37.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79d6000640 1 -- 172.21.15.114:0/377071015 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c4031810 con 0x7f79d80b3650 2024-03-06T07:29:37.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79d6000640 1 -- 172.21.15.114:0/377071015 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c4031be0 con 0x7f79d80b3650 2024-03-06T07:29:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 -- 172.21.15.114:0/377071015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 msgr2=0x7f79d80727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 --2- 172.21.15.114:0/377071015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d80727a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f79c40097a0 tx=0x7f79c402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 -- 172.21.15.114:0/377071015 shutdown_connections 2024-03-06T07:29:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 --2- 172.21.15.114:0/377071015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80cda90 0x7f79d806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 --2- 172.21.15.114:0/377071015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d80727a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 -- 172.21.15.114:0/377071015 >> 172.21.15.114:0/377071015 conn(0x7f79d80b1940 msgr2=0x7f79d80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 -- 172.21.15.114:0/377071015 shutdown_connections 2024-03-06T07:29:37.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.638+0000 7f79de800640 1 -- 172.21.15.114:0/377071015 wait complete. 2024-03-06T07:29:37.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.640+0000 7f79de800640 1 Processor -- start 2024-03-06T07:29:37.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.640+0000 7f79de800640 1 -- start start 2024-03-06T07:29:37.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79de800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d81511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79de800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80b3df0 0x7f79d81501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79de800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d80cda90 0x7f79d814ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79de800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d815b3c0 con 0x7f79d80b3650 2024-03-06T07:29:37.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79de800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d815b240 con 0x7f79d80cda90 2024-03-06T07:29:37.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79de800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d80cf510 con 0x7f79d80b3df0 2024-03-06T07:29:37.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79d7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d81511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79d7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d81511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36386/0 (socket says 172.21.15.114:36386) 2024-03-06T07:29:37.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.641+0000 7f79d7400640 1 -- 172.21.15.114:0/2579146550 learned_addr learned my addr 172.21.15.114:0/2579146550 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:37.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79d7400640 1 -- 172.21.15.114:0/2579146550 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80b3df0 msgr2=0x7f79d81501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79d7400640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80b3df0 0x7f79d81501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79d7400640 1 -- 172.21.15.114:0/2579146550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d80cda90 msgr2=0x7f79d814ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79d7400640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d80cda90 0x7f79d814ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79d7400640 1 -- 172.21.15.114:0/2579146550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d8151aa0 con 0x7f79d80b3650 2024-03-06T07:29:37.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79d7400640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d81511a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f79c800c4d0 tx=0x7f79c800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79cf400640 1 -- 172.21.15.114:0/2579146550 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8018070 con 0x7f79d80b3650 2024-03-06T07:29:37.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79cf400640 1 -- 172.21.15.114:0/2579146550 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c8007500 con 0x7f79d80b3650 2024-03-06T07:29:37.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79cf400640 1 -- 172.21.15.114:0/2579146550 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8004a20 con 0x7f79d80b3650 2024-03-06T07:29:37.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.642+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d8151d80 con 0x7f79d80b3650 2024-03-06T07:29:37.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.643+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d8152340 con 0x7f79d80b3650 2024-03-06T07:29:37.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.644+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d8072ec0 con 0x7f79d80b3650 2024-03-06T07:29:37.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.645+0000 7f79cf400640 1 -- 172.21.15.114:0/2579146550 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79c801b070 con 0x7f79d80b3650 2024-03-06T07:29:37.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.646+0000 7f79cf400640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79b806b4d0 0x7f79b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.646+0000 7f79cf400640 1 -- 172.21.15.114:0/2579146550 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f79c8089570 con 0x7f79d80b3650 2024-03-06T07:29:37.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.646+0000 7f79d6a00640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79b806b4d0 0x7f79b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.646+0000 7f79d6a00640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79b806b4d0 0x7f79b806d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f79c4036000 tx=0x7f79c4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.650+0000 7f79cf400640 1 -- 172.21.15.114:0/2579146550 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79d8072ec0 con 0x7f79d80b3650 2024-03-06T07:29:37.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:37.880+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f79d80cc0e0 con 0x7f79d80b3650 2024-03-06T07:29:38.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.432+0000 7f79cf400640 1 -- 172.21.15.114:0/2579146550 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f79c8058390 con 0x7f79d80b3650 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79b806b4d0 msgr2=0x7f79b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79b806b4d0 0x7f79b806d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f79c4036000 tx=0x7f79c4030000 comp rx=0 tx=0).stop 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 msgr2=0x7f79d81511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d81511a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f79c800c4d0 tx=0x7f79c800c990 comp rx=0 tx=0).stop 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 shutdown_connections 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79d80cda90 0x7f79d814ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79d80b3df0 0x7f79d81501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79b806b4d0 0x7f79b806d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 --2- 172.21.15.114:0/2579146550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79d80b3650 0x7f79d81511a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 >> 172.21.15.114:0/2579146550 conn(0x7f79d80b1940 msgr2=0x7f79d806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 shutdown_connections 2024-03-06T07:29:38.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.435+0000 7f79de800640 1 -- 172.21.15.114:0/2579146550 wait complete. 2024-03-06T07:29:38.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:29:38.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.663+0000 7f7f67c00640 1 Processor -- start 2024-03-06T07:29:38.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.663+0000 7f7f67c00640 1 -- start start 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f67c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7f60145750 0x7f7f60072260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f67c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7f6006dd30 0x7f7f60165ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f67c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f600cdbd0 con 0x7f7f600727a0 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f67c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f600cd980 con 0x7f7f6006dd30 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f67c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f600cdeb0 con 0x7f7f60145750 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f65c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7f6006dd30 0x7f7f60165ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f65c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7f6006dd30 0x7f7f60165ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45720/0 (socket says 172.21.15.114:45720) 2024-03-06T07:29:38.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.665+0000 7f7f65c00640 1 -- 172.21.15.114:0/1891556121 learned_addr learned my addr 172.21.15.114:0/1891556121 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.666+0000 7f7f65200640 1 --2- 172.21.15.114:0/1891556121 >> v2:172.21.15.177:3300/0 conn(0x7f7f60145750 0x7f7f60072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.666+0000 7f7f65c00640 1 -- 172.21.15.114:0/1891556121 >> v2:172.21.15.177:3300/0 conn(0x7f7f60145750 msgr2=0x7f7f60072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.666+0000 7f7f65c00640 1 --2- 172.21.15.114:0/1891556121 >> v2:172.21.15.177:3300/0 conn(0x7f7f60145750 0x7f7f60072260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.666+0000 7f7f65c00640 1 -- 172.21.15.114:0/1891556121 >> v1:172.21.15.114:6789/0 conn(0x7f7f600727a0 legacy=0x7f7f6006d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.666+0000 7f7f65c00640 1 -- 172.21.15.114:0/1891556121 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f600ce190 con 0x7f7f6006dd30 2024-03-06T07:29:38.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.666+0000 7f7f5fe00640 1 -- 172.21.15.114:0/1891556121 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554310185 0 0) 0x7f7f600cdbd0 con 0x7f7f600727a0 2024-03-06T07:29:38.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.667+0000 7f7f65c00640 1 --2- 172.21.15.114:0/1891556121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6006dd30 0x7f7f60165ab0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7f6006eb20 tx=0x7f7f5802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a759f957ea9cd44a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.667+0000 7f7f5fe00640 1 -- 172.21.15.114:0/1891556121 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5802ca80 con 0x7f7f6006dd30 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.667+0000 7f7f5fe00640 1 -- 172.21.15.114:0/1891556121 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5802cc20 con 0x7f7f6006dd30 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.667+0000 7f7f5fe00640 1 -- 172.21.15.114:0/1891556121 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f580343f0 con 0x7f7f6006dd30 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 -- 172.21.15.114:0/1891556121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6006dd30 msgr2=0x7f7f60165ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 --2- 172.21.15.114:0/1891556121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6006dd30 0x7f7f60165ab0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7f6006eb20 tx=0x7f7f5802bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 -- 172.21.15.114:0/1891556121 shutdown_connections 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 --2- 172.21.15.114:0/1891556121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6006dd30 0x7f7f60165ab0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 --2- 172.21.15.114:0/1891556121 >> v2:172.21.15.177:3300/0 conn(0x7f7f60145750 0x7f7f60072260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 -- 172.21.15.114:0/1891556121 >> 172.21.15.114:0/1891556121 conn(0x7f7f600cfa80 msgr2=0x7f7f600cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 -- 172.21.15.114:0/1891556121 shutdown_connections 2024-03-06T07:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.668+0000 7f7f67c00640 1 -- 172.21.15.114:0/1891556121 wait complete. 2024-03-06T07:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.670+0000 7f7f67c00640 1 Processor -- start 2024-03-06T07:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.670+0000 7f7f67c00640 1 -- start start 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.671+0000 7f7f67c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f6006dd30 0x7f7f601e2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.671+0000 7f7f67c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f600727a0 0x7f7f601e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.671+0000 7f7f67c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60145750 0x7f7f601f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.671+0000 7f7f67c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600cd5d0 con 0x7f7f60145750 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.671+0000 7f7f67c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f600cd450 con 0x7f7f600727a0 2024-03-06T07:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.671+0000 7f7f67c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6015c440 con 0x7f7f6006dd30 2024-03-06T07:29:38.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60145750 0x7f7f601f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60145750 0x7f7f601f7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45728/0 (socket says 172.21.15.114:45728) 2024-03-06T07:29:38.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 -- 172.21.15.114:0/3140655446 learned_addr learned my addr 172.21.15.114:0/3140655446 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:38.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f57e00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f600727a0 0x7f7f601e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65200640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f6006dd30 0x7f7f601e2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 -- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f6006dd30 msgr2=0x7f7f601e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f6006dd30 0x7f7f601e2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 -- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f600727a0 msgr2=0x7f7f601e14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f600727a0 0x7f7f601e14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f65c00640 1 -- 172.21.15.114:0/3140655446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f601e1d30 con 0x7f7f60145750 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.672+0000 7f7f57e00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f600727a0 0x7f7f601e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.673+0000 7f7f65c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60145750 0x7f7f601f7990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7f5802c4f0 tx=0x7f7f58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.673+0000 7f7f5d600640 1 -- 172.21.15.114:0/3140655446 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58043030 con 0x7f7f60145750 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.673+0000 7f7f5d600640 1 -- 172.21.15.114:0/3140655446 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f58003990 con 0x7f7f60145750 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.673+0000 7f7f5d600640 1 -- 172.21.15.114:0/3140655446 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58009510 con 0x7f7f60145750 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.673+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f601e2010 con 0x7f7f60145750 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.673+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f601e25d0 con 0x7f7f60145750 2024-03-06T07:29:38.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.674+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f60153910 con 0x7f7f60145750 2024-03-06T07:29:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.675+0000 7f7f5d600640 1 -- 172.21.15.114:0/3140655446 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f5802ca90 con 0x7f7f60145750 2024-03-06T07:29:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.676+0000 7f7f5d600640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.676+0000 7f7f5d600640 1 -- 172.21.15.114:0/3140655446 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7f580ac660 con 0x7f7f60145750 2024-03-06T07:29:38.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.676+0000 7f7f65200640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.677+0000 7f7f65200640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7f50006fd0 tx=0x7f7f50008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.680+0000 7f7f5d600640 1 -- 172.21.15.114:0/3140655446 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f60153910 con 0x7f7f60145750 2024-03-06T07:29:38.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:38.861+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7f6004f3a0 con 0x7f7f60145750 2024-03-06T07:29:39.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.435+0000 7f7f5d600640 1 -- 172.21.15.114:0/3140655446 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7f5804d020 con 0x7f7f60145750 2024-03-06T07:29:39.438 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:29:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.438+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f3c06b4d0 msgr2=0x7f7f3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.438+0000 7f7f67c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7f50006fd0 tx=0x7f7f50008040 comp rx=0 tx=0).stop 2024-03-06T07:29:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.438+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60145750 msgr2=0x7f7f601f7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.438+0000 7f7f67c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60145750 0x7f7f601f7990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7f5802c4f0 tx=0x7f7f58004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 shutdown_connections 2024-03-06T07:29:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f3c06b4d0 0x7f7f3c06d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60145750 0x7f7f601f7990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f600727a0 0x7f7f601e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 --2- 172.21.15.114:0/3140655446 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f6006dd30 0x7f7f601e2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 >> 172.21.15.114:0/3140655446 conn(0x7f7f600cfa80 msgr2=0x7f7f6015b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 shutdown_connections 2024-03-06T07:29:39.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.439+0000 7f7f67c00640 1 -- 172.21.15.114:0/3140655446 wait complete. 2024-03-06T07:29:39.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:29:39.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.668+0000 7faab6400640 1 Processor -- start 2024-03-06T07:29:39.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.668+0000 7faab6400640 1 -- start start 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faab6400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7faab00b7810 0x7faab00d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faab6400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faab00d1020 0x7faab006c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faab6400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab00cc5a0 con 0x7faab00b4b70 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faab6400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab00cc350 con 0x7faab00d1020 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faab6400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab00cced0 con 0x7faab00b7810 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faaaea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faab00d1020 0x7faab006c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faaafe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7faab00b4b70 0x7faab00cd360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58776/0 (socket says 172.21.15.114:58776) 2024-03-06T07:29:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.670+0000 7faaafe00640 1 -- 172.21.15.114:0/3531624410 learned_addr learned my addr 172.21.15.114:0/3531624410 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.671+0000 7faaaf400640 1 --2- 172.21.15.114:0/3531624410 >> v2:172.21.15.177:3300/0 conn(0x7faab00b7810 0x7faab00d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.671+0000 7faaaea00640 1 -- 172.21.15.114:0/3531624410 >> v2:172.21.15.177:3300/0 conn(0x7faab00b7810 msgr2=0x7faab00d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.671+0000 7faaaea00640 1 --2- 172.21.15.114:0/3531624410 >> v2:172.21.15.177:3300/0 conn(0x7faab00b7810 0x7faab00d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.671+0000 7faaaea00640 1 -- 172.21.15.114:0/3531624410 >> v1:172.21.15.114:6789/0 conn(0x7faab00b4b70 legacy=0x7faab00cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.671+0000 7faaaea00640 1 -- 172.21.15.114:0/3531624410 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab00cc880 con 0x7faab00d1020 2024-03-06T07:29:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.671+0000 7faaae000640 1 -- 172.21.15.114:0/3531624410 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666217002 0 0) 0x7faab00cc5a0 con 0x7faab00b4b70 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.672+0000 7faaaea00640 1 --2- 172.21.15.114:0/3531624410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00d1020 0x7faab006c2e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7faab0071850 tx=0x7faa9c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a66beae9572d1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.672+0000 7faaae000640 1 -- 172.21.15.114:0/3531624410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c034070 con 0x7faab00d1020 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.672+0000 7faaae000640 1 -- 172.21.15.114:0/3531624410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9c030610 con 0x7faab00d1020 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.672+0000 7faaae000640 1 -- 172.21.15.114:0/3531624410 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c0309e0 con 0x7faab00d1020 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 -- 172.21.15.114:0/3531624410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00d1020 msgr2=0x7faab006c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 --2- 172.21.15.114:0/3531624410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00d1020 0x7faab006c2e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7faab0071850 tx=0x7faa9c02b960 comp rx=0 tx=0).stop 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 -- 172.21.15.114:0/3531624410 shutdown_connections 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 --2- 172.21.15.114:0/3531624410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00d1020 0x7faab006c2e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 --2- 172.21.15.114:0/3531624410 >> v2:172.21.15.177:3300/0 conn(0x7faab00b7810 0x7faab00d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 -- 172.21.15.114:0/3531624410 >> 172.21.15.114:0/3531624410 conn(0x7faab00b3870 msgr2=0x7faab00b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 -- 172.21.15.114:0/3531624410 shutdown_connections 2024-03-06T07:29:39.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.673+0000 7faab6400640 1 -- 172.21.15.114:0/3531624410 wait complete. 2024-03-06T07:29:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.675+0000 7faab6400640 1 Processor -- start 2024-03-06T07:29:39.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.675+0000 7faab6400640 1 -- start start 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faab6400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faab00b4b70 0x7faab01f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faab6400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00b7810 0x7faab00ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faab6400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faab00d1020 0x7faab00c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faab6400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab006cf60 con 0x7faab00b7810 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faab6400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab00b7bf0 con 0x7faab00d1020 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faab6400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab006d0e0 con 0x7faab00b4b70 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00b7810 0x7faab00ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00b7810 0x7faab00ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45742/0 (socket says 172.21.15.114:45742) 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 -- 172.21.15.114:0/4081629564 learned_addr learned my addr 172.21.15.114:0/4081629564 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:39.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaf400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faab00b4b70 0x7faab01f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaafe00640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faab00d1020 0x7faab00c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 -- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faab00b4b70 msgr2=0x7faab01f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faab00b4b70 0x7faab01f4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 -- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faab00d1020 msgr2=0x7faab00c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faab00d1020 0x7faab00c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaaea00640 1 -- 172.21.15.114:0/4081629564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab01f49d0 con 0x7faab00b7810 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.676+0000 7faaafe00640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faab00d1020 0x7faab00c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.677+0000 7faaaea00640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00b7810 0x7faab00ca1a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7faa9c000c00 tx=0x7faa9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.677+0000 7faaa7400640 1 -- 172.21.15.114:0/4081629564 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c034040 con 0x7faab00b7810 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.677+0000 7faaa7400640 1 -- 172.21.15.114:0/4081629564 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9c004030 con 0x7faab00b7810 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.677+0000 7faaa7400640 1 -- 172.21.15.114:0/4081629564 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9c004400 con 0x7faab00b7810 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.677+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab00c5ce0 con 0x7faab00b7810 2024-03-06T07:29:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.677+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab00c6260 con 0x7faab00b7810 2024-03-06T07:29:39.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.678+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab004f330 con 0x7faab00b7810 2024-03-06T07:29:39.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.679+0000 7faaa7400640 1 -- 172.21.15.114:0/4081629564 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa9c037070 con 0x7faab00b7810 2024-03-06T07:29:39.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.680+0000 7faaa7400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa9006b4d0 0x7faa9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.680+0000 7faaa7400640 1 -- 172.21.15.114:0/4081629564 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5357+0+0 (secure 0 0 0) 0x7faa9c0abd70 con 0x7faab00b7810 2024-03-06T07:29:39.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.680+0000 7faaaf400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa9006b4d0 0x7faa9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.681+0000 7faaaf400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa9006b4d0 0x7faa9006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faab00b6b90 tx=0x7faaa0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.685+0000 7faaa7400640 1 -- 172.21.15.114:0/4081629564 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faab01f49d0 con 0x7faab00b7810 2024-03-06T07:29:39.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:39.879+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faab00b3650 con 0x7faab00b7810 2024-03-06T07:29:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.446+0000 7faaa7400640 1 -- 172.21.15.114:0/4081629564 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 169+0+0 (secure 0 0 0) 0x7faa9c07ad40 con 0x7faab00b7810 2024-03-06T07:29:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:29:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa9006b4d0 msgr2=0x7faa9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa9006b4d0 0x7faa9006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faab00b6b90 tx=0x7faaa0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00b7810 msgr2=0x7faab00ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00b7810 0x7faab00ca1a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7faa9c000c00 tx=0x7faa9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 shutdown_connections 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faab00d1020 0x7faab00c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa9006b4d0 0x7faa9006d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faab00b7810 0x7faab00ca1a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 --2- 172.21.15.114:0/4081629564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faab00b4b70 0x7faab01f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.449+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 >> 172.21.15.114:0/4081629564 conn(0x7faab00b3870 msgr2=0x7faab00d1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.450+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 shutdown_connections 2024-03-06T07:29:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.450+0000 7faab6400640 1 -- 172.21.15.114:0/4081629564 wait complete. 2024-03-06T07:29:40.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:29:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.663+0000 7f45b6800640 1 Processor -- start 2024-03-06T07:29:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.664+0000 7f45b6800640 1 -- start start 2024-03-06T07:29:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.665+0000 7f45b6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b00b3f70 0x7f45b0160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.665+0000 7f45b6800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b00b46c0 con 0x7f45b015fc10 2024-03-06T07:29:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.665+0000 7f45b6800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b00b49a0 con 0x7f45b0070ff0 2024-03-06T07:29:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.665+0000 7f45b6800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0071480 con 0x7f45b00b3f70 2024-03-06T07:29:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.665+0000 7f45af400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f45b015fc10 0x7f45b015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58794/0 (socket says 172.21.15.114:58794) 2024-03-06T07:29:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.665+0000 7f45af400640 1 -- 172.21.15.114:0/1843456549 learned_addr learned my addr 172.21.15.114:0/1843456549 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996155662 0 0) 0x7f45b00b46c0 con 0x7f45b015fc10 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45a6a00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b00b3f70 0x7f45b0160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4594003520 con 0x7f45b015fc10 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3695220165 0 0) 0x7f4594003520 con 0x7f45b015fc10 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b00b3f70 msgr2=0x7f45b0160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b00b3f70 0x7f45b0160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 >> v1:172.21.15.177:6789/0 conn(0x7f45b0070ff0 legacy=0x7f45b00ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0072c90 con 0x7f45b015fc10 2024-03-06T07:29:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f45a0001670 con 0x7f45b015fc10 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 >> v1:172.21.15.114:6789/0 conn(0x7f45b015fc10 legacy=0x7f45b015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45940039e0 0x7f4594003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4594004800 0x7f4594004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.666+0000 7f45aea00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b00b3f70 0x7f4594005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0071480 con 0x7f4594004800 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459400bb50 con 0x7f45940039e0 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459400bd50 con 0x7f45b00b3f70 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45a6a00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4594004800 0x7f4594004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45af400640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45940039e0 0x7f4594003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45a6a00640 1 -- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b00b3f70 msgr2=0x7f4594005250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45a6a00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b00b3f70 0x7f4594005250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45a6a00640 1 -- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45940039e0 msgr2=0x7f4594003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45a6a00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45940039e0 0x7f4594003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.667+0000 7f45a6a00640 1 -- 172.21.15.114:0/1843456549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594003520 con 0x7f4594004800 2024-03-06T07:29:40.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.668+0000 7f45a6a00640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4594004800 0x7f4594004c80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f459800ba00 tx=0x7f4594005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fd71ebea901419 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598036020 con 0x7f4594004800 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4598032630 con 0x7f4594004800 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45aea00640 1 -- 172.21.15.114:0/1843456549 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459800b600 con 0x7f4594004800 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45b6800640 1 -- 172.21.15.114:0/1843456549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4594004800 msgr2=0x7f4594004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45b6800640 1 --2- 172.21.15.114:0/1843456549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4594004800 0x7f4594004c80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f459800ba00 tx=0x7f4594005830 comp rx=0 tx=0).stop 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45afe00640 1 -- 172.21.15.114:0/1843456549 reap_dead start 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45b6800640 1 -- 172.21.15.114:0/1843456549 shutdown_connections 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45b6800640 1 -- 172.21.15.114:0/1843456549 >> 172.21.15.114:0/1843456549 conn(0x7f45b00b1940 msgr2=0x7f45b00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45b6800640 1 -- 172.21.15.114:0/1843456549 shutdown_connections 2024-03-06T07:29:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.669+0000 7f45b6800640 1 -- 172.21.15.114:0/1843456549 wait complete. 2024-03-06T07:29:40.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.671+0000 7f45b6800640 1 Processor -- start 2024-03-06T07:29:40.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.671+0000 7f45b6800640 1 -- start start 2024-03-06T07:29:40.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.672+0000 7f45b6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4594004800 0x7f45b014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.672+0000 7f45b6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b006e540 0x7f45b00ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.672+0000 7f45b6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b014bc20 0x7f45b014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.672+0000 7f45b6800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0072a30 con 0x7f45b006e540 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.672+0000 7f45b6800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b00728b0 con 0x7f4594004800 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.672+0000 7f45b6800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015c2e0 con 0x7f45b014bc20 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b006e540 0x7f45b00ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b006e540 0x7f45b00ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45760/0 (socket says 172.21.15.114:45760) 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 -- 172.21.15.114:0/2466338618 learned_addr learned my addr 172.21.15.114:0/2466338618 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45af400640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4594004800 0x7f45b014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 -- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b014bc20 msgr2=0x7f45b014c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45afe00640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b014bc20 0x7f45b014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b014bc20 0x7f45b014c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 -- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4594004800 msgr2=0x7f45b014ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4594004800 0x7f45b014ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 -- 172.21.15.114:0/2466338618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b009ab70 con 0x7f45b006e540 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45af400640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4594004800 0x7f45b014ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.673+0000 7f45a6a00640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b006e540 0x7f45b00ae210 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4598035040 tx=0x7f4598004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.674+0000 7f45a7e00640 1 -- 172.21.15.114:0/2466338618 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598041070 con 0x7f45b006e540 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.674+0000 7f45a7e00640 1 -- 172.21.15.114:0/2466338618 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4598030a40 con 0x7f45b006e540 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.674+0000 7f45a7e00640 1 -- 172.21.15.114:0/2466338618 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598036040 con 0x7f45b006e540 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.674+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b009ae50 con 0x7f45b006e540 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.674+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b014e300 con 0x7f45b006e540 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.675+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b004f330 con 0x7f45b006e540 2024-03-06T07:29:40.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.676+0000 7f45a7e00640 1 -- 172.21.15.114:0/2466338618 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4598039020 con 0x7f45b006e540 2024-03-06T07:29:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.677+0000 7f45a7e00640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458c06b4d0 0x7f458c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.677+0000 7f45a7e00640 1 -- 172.21.15.114:0/2466338618 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f45980ac140 con 0x7f45b006e540 2024-03-06T07:29:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.677+0000 7f45af400640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458c06b4d0 0x7f458c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.678+0000 7f45af400640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458c06b4d0 0x7f458c06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f45a0001c00 tx=0x7f45a0000f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.682+0000 7f45a7e00640 1 -- 172.21.15.114:0/2466338618 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f459807b230 con 0x7f45b006e540 2024-03-06T07:29:40.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.889+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f45b00b2210 con 0x7f45b006e540 2024-03-06T07:29:40.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.896+0000 7f45a7e00640 1 -- 172.21.15.114:0/2466338618 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f459807ab30 con 0x7f45b006e540 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458c06b4d0 msgr2=0x7f458c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458c06b4d0 0x7f458c06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f45a0001c00 tx=0x7f45a0000f90 comp rx=0 tx=0).stop 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b006e540 msgr2=0x7f45b00ae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b006e540 0x7f45b00ae210 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4598035040 tx=0x7f4598004730 comp rx=0 tx=0).stop 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 shutdown_connections 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45b014bc20 0x7f45b014c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458c06b4d0 0x7f458c06d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45b006e540 0x7f45b00ae210 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 --2- 172.21.15.114:0/2466338618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4594004800 0x7f45b014ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 >> 172.21.15.114:0/2466338618 conn(0x7f45b00b1940 msgr2=0x7f45b00b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 shutdown_connections 2024-03-06T07:29:40.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:40.899+0000 7f45b6800640 1 -- 172.21.15.114:0/2466338618 wait complete. 2024-03-06T07:29:40.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:29:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.111+0000 7fdc99800640 1 Processor -- start 2024-03-06T07:29:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.112+0000 7fdc99800640 1 -- start start 2024-03-06T07:29:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.113+0000 7fdc99800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc9406cb50 0x7fdc9406cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.113+0000 7fdc99800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdc940b2d10 0x7fdc940b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.113+0000 7fdc99800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc940717c0 con 0x7fdc940cf430 2024-03-06T07:29:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.113+0000 7fdc99800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9406d4f0 con 0x7fdc940b2d10 2024-03-06T07:29:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.113+0000 7fdc99800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94071aa0 con 0x7fdc9406cb50 2024-03-06T07:29:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.113+0000 7fdc92a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdc940cf430 0x7fdc94070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58796/0 (socket says 172.21.15.114:58796) 2024-03-06T07:29:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.113+0000 7fdc92a00640 1 -- 172.21.15.114:0/817621493 learned_addr learned my addr 172.21.15.114:0/817621493 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066746278 0 0) 0x7fdc940717c0 con 0x7fdc940cf430 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc93400640 1 --2- 172.21.15.114:0/817621493 >> v2:172.21.15.177:3300/0 conn(0x7fdc940b2d10 0x7fdc940b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc78003520 con 0x7fdc940cf430 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc92000640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc9406cb50 0x7fdc9406cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515519162 0 0) 0x7fdc78003520 con 0x7fdc940cf430 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc9406cb50 msgr2=0x7fdc9406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc9406cb50 0x7fdc9406cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 >> v2:172.21.15.177:3300/0 conn(0x7fdc940b2d10 msgr2=0x7fdc940b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 --2- 172.21.15.114:0/817621493 >> v2:172.21.15.177:3300/0 conn(0x7fdc940b2d10 0x7fdc940b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc94071d80 con 0x7fdc940cf430 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fdc840019f0 con 0x7fdc940cf430 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc93400640 1 --2- 172.21.15.114:0/817621493 >> v2:172.21.15.177:3300/0 conn(0x7fdc940b2d10 0x7fdc940b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 >> v1:172.21.15.114:6789/0 conn(0x7fdc940cf430 legacy=0x7fdc94070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc780039b0 0x7fdc78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.114+0000 7fdc91600640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc78004800 0x7fdc78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc91600640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc9406cb50 0x7fdc780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94071aa0 con 0x7fdc780039b0 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7800ba20 con 0x7fdc78004800 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7800bc20 con 0x7fdc9406cb50 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc92a00640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc780039b0 0x7fdc78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc92000640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc78004800 0x7fdc78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc92a00640 1 -- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc9406cb50 msgr2=0x7fdc780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc92a00640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc9406cb50 0x7fdc780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc92a00640 1 -- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc78004800 msgr2=0x7fdc78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc92a00640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc78004800 0x7fdc78004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.115+0000 7fdc92a00640 1 -- 172.21.15.114:0/817621493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc78003520 con 0x7fdc780039b0 2024-03-06T07:29:41.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.116+0000 7fdc92a00640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc780039b0 0x7fdc78003dd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdc84002230 tx=0x7fdc78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ebc1e0085bfdc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.116+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8400a4e0 con 0x7fdc780039b0 2024-03-06T07:29:41.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.116+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc84034650 con 0x7fdc780039b0 2024-03-06T07:29:41.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.116+0000 7fdc91600640 1 -- 172.21.15.114:0/817621493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84034a40 con 0x7fdc780039b0 2024-03-06T07:29:41.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.117+0000 7fdc99800640 1 -- 172.21.15.114:0/817621493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc780039b0 msgr2=0x7fdc78003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.117+0000 7fdc99800640 1 --2- 172.21.15.114:0/817621493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc780039b0 0x7fdc78003dd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdc84002230 tx=0x7fdc78005830 comp rx=0 tx=0).stop 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.117+0000 7fdc93400640 1 -- 172.21.15.114:0/817621493 reap_dead start 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.117+0000 7fdc99800640 1 -- 172.21.15.114:0/817621493 shutdown_connections 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.117+0000 7fdc99800640 1 -- 172.21.15.114:0/817621493 >> 172.21.15.114:0/817621493 conn(0x7fdc940b1730 msgr2=0x7fdc940b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.117+0000 7fdc99800640 1 -- 172.21.15.114:0/817621493 shutdown_connections 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.117+0000 7fdc99800640 1 -- 172.21.15.114:0/817621493 wait complete. 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.119+0000 7fdc99800640 1 Processor -- start 2024-03-06T07:29:41.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.119+0000 7fdc99800640 1 -- start start 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc99800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc780039b0 0x7fdc940a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc99800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc941fc930 0x7fdc941fc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc99800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc941f7bd0 0x7fdc941f8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc99800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94071430 con 0x7fdc941f7bd0 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc99800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94072340 con 0x7fdc941fc930 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc99800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94162b70 con 0x7fdc780039b0 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc93400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc941f7bd0 0x7fdc941f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc93400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc941f7bd0 0x7fdc941f8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45782/0 (socket says 172.21.15.114:45782) 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc93400640 1 -- 172.21.15.114:0/1828370174 learned_addr learned my addr 172.21.15.114:0/1828370174 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:41.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc93400640 1 -- 172.21.15.114:0/1828370174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc780039b0 msgr2=0x7fdc940a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc93400640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc780039b0 0x7fdc940a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc93400640 1 -- 172.21.15.114:0/1828370174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc941fc930 msgr2=0x7fdc941fc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.120+0000 7fdc93400640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc941fc930 0x7fdc941fc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.121+0000 7fdc93400640 1 -- 172.21.15.114:0/1828370174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc941ebfb0 con 0x7fdc941f7bd0 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.121+0000 7fdc93400640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc941f7bd0 0x7fdc941f8050 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdc94073390 tx=0x7fdc8c00e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.121+0000 7fdc8aa00640 1 -- 172.21.15.114:0/1828370174 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c01a070 con 0x7fdc941f7bd0 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.121+0000 7fdc8aa00640 1 -- 172.21.15.114:0/1828370174 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c004230 con 0x7fdc941f7bd0 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.121+0000 7fdc8aa00640 1 -- 172.21.15.114:0/1828370174 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c00f040 con 0x7fdc941f7bd0 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.121+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc941ec250 con 0x7fdc941f7bd0 2024-03-06T07:29:41.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.121+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc941ec810 con 0x7fdc941f7bd0 2024-03-06T07:29:41.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.122+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc940b0810 con 0x7fdc941f7bd0 2024-03-06T07:29:41.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.124+0000 7fdc8aa00640 1 -- 172.21.15.114:0/1828370174 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc8c01d070 con 0x7fdc941f7bd0 2024-03-06T07:29:41.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.125+0000 7fdc8aa00640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdc7006b4d0 0x7fdc7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.125+0000 7fdc8aa00640 1 -- 172.21.15.114:0/1828370174 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdc8c088cd0 con 0x7fdc941f7bd0 2024-03-06T07:29:41.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.125+0000 7fdc92a00640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdc7006b4d0 0x7fdc7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.126+0000 7fdc92a00640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdc7006b4d0 0x7fdc7006d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc84034000 tx=0x7fdc84007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.129+0000 7fdc8aa00640 1 -- 172.21.15.114:0/1828370174 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc8c050e10 con 0x7fdc941f7bd0 2024-03-06T07:29:41.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.400+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fdc9404f330 con 0x7fdc941f7bd0 2024-03-06T07:29:41.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.449+0000 7fdc8aa00640 1 -- 172.21.15.114:0/1828370174 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v113) v1 ==== 132+0+0 (secure 0 0 0) 0x7fdc940b0810 con 0x7fdc941f7bd0 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdc7006b4d0 msgr2=0x7fdc7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdc7006b4d0 0x7fdc7006d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc84034000 tx=0x7fdc84007000 comp rx=0 tx=0).stop 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc941f7bd0 msgr2=0x7fdc941f8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc941f7bd0 0x7fdc941f8050 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdc94073390 tx=0x7fdc8c00e8a0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 shutdown_connections 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdc7006b4d0 0x7fdc7006d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc941f7bd0 0x7fdc941f8050 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc941fc930 0x7fdc941fc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 --2- 172.21.15.114:0/1828370174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdc780039b0 0x7fdc940a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 >> 172.21.15.114:0/1828370174 conn(0x7fdc940b1730 msgr2=0x7fdc94070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 shutdown_connections 2024-03-06T07:29:41.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.452+0000 7fdc99800640 1 -- 172.21.15.114:0/1828370174 wait complete. 2024-03-06T07:29:41.474 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:29:41.474 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:29:41.474 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:29:41.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:41.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.678+0000 7fe077200640 1 Processor -- start 2024-03-06T07:29:41.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.678+0000 7fe077200640 1 -- start start 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.679+0000 7fe077200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe070144c00 0x7fe07006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.679+0000 7fe077200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe07015f9d0 0x7fe07015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe077200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 0x7fe0700ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe077200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07006ee70 con 0x7fe070144c00 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe077200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07006cef0 con 0x7fe07015f9d0 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe077200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07006d1d0 con 0x7fe0700ce0a0 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe070144c00 0x7fe07006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe070144c00 0x7fe07006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45794/0 (socket says 172.21.15.114:45794) 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 -- 172.21.15.114:0/607007236 learned_addr learned my addr 172.21.15.114:0/607007236 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 -- 172.21.15.114:0/607007236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 msgr2=0x7fe0700ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 --2- 172.21.15.114:0/607007236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 0x7fe0700ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 -- 172.21.15.114:0/607007236 >> v2:172.21.15.177:3300/0 conn(0x7fe07015f9d0 msgr2=0x7fe07015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 --2- 172.21.15.114:0/607007236 >> v2:172.21.15.177:3300/0 conn(0x7fe07015f9d0 0x7fe07015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.680+0000 7fe06fe00640 1 -- 172.21.15.114:0/607007236 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07006d4b0 con 0x7fe070144c00 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.681+0000 7fe06fe00640 1 --2- 172.21.15.114:0/607007236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe070144c00 0x7fe07006e930 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe0700b26f0 tx=0x7fe06002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389cc3ad95731605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe06f400640 1 -- 172.21.15.114:0/607007236 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06002ca80 con 0x7fe070144c00 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe06f400640 1 -- 172.21.15.114:0/607007236 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06002cc20 con 0x7fe070144c00 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe06f400640 1 -- 172.21.15.114:0/607007236 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0600343f0 con 0x7fe070144c00 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 -- 172.21.15.114:0/607007236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe070144c00 msgr2=0x7fe07006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 --2- 172.21.15.114:0/607007236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe070144c00 0x7fe07006e930 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe0700b26f0 tx=0x7fe06002bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 -- 172.21.15.114:0/607007236 shutdown_connections 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 --2- 172.21.15.114:0/607007236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 0x7fe0700ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 --2- 172.21.15.114:0/607007236 >> v2:172.21.15.177:3300/0 conn(0x7fe07015f9d0 0x7fe07015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 --2- 172.21.15.114:0/607007236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe070144c00 0x7fe07006e930 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 -- 172.21.15.114:0/607007236 >> 172.21.15.114:0/607007236 conn(0x7fe0700b1730 msgr2=0x7fe0700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 -- 172.21.15.114:0/607007236 shutdown_connections 2024-03-06T07:29:41.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.682+0000 7fe077200640 1 -- 172.21.15.114:0/607007236 wait complete. 2024-03-06T07:29:41.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.684+0000 7fe077200640 1 Processor -- start 2024-03-06T07:29:41.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.684+0000 7fe077200640 1 -- start start 2024-03-06T07:29:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe077200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 0x7fe0701390b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe077200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe070144c00 0x7fe07015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe077200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe07015f9d0 0x7fe07015b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe077200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07006cb50 con 0x7fe07015f9d0 2024-03-06T07:29:41.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe077200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070160730 con 0x7fe070144c00 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe077200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07006ccd0 con 0x7fe0700ce0a0 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe075200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe07015f9d0 0x7fe07015b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe075200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe07015f9d0 0x7fe07015b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45806/0 (socket says 172.21.15.114:45806) 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.685+0000 7fe075200640 1 -- 172.21.15.114:0/1189753327 learned_addr learned my addr 172.21.15.114:0/1189753327 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe075200640 1 -- 172.21.15.114:0/1189753327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 msgr2=0x7fe0701390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe075200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 0x7fe0701390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe075200640 1 -- 172.21.15.114:0/1189753327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe070144c00 msgr2=0x7fe07015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe075200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe070144c00 0x7fe07015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe075200640 1 -- 172.21.15.114:0/1189753327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0701f5460 con 0x7fe07015f9d0 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe075200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe07015f9d0 0x7fe07015b120 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe06800b440 tx=0x7fe06800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe06cc00640 1 -- 172.21.15.114:0/1189753327 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068015020 con 0x7fe07015f9d0 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe06cc00640 1 -- 172.21.15.114:0/1189753327 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe068011830 con 0x7fe07015f9d0 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe06cc00640 1 -- 172.21.15.114:0/1189753327 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068011c00 con 0x7fe07015f9d0 2024-03-06T07:29:41.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0701f5740 con 0x7fe07015f9d0 2024-03-06T07:29:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.686+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0701f5d00 con 0x7fe07015f9d0 2024-03-06T07:29:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.687+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07004f330 con 0x7fe07015f9d0 2024-03-06T07:29:41.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.688+0000 7fe06cc00640 1 -- 172.21.15.114:0/1189753327 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe068018020 con 0x7fe07015f9d0 2024-03-06T07:29:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.689+0000 7fe06cc00640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe04c06b4d0 0x7fe04c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.689+0000 7fe06cc00640 1 -- 172.21.15.114:0/1189753327 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe068089e30 con 0x7fe07015f9d0 2024-03-06T07:29:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.689+0000 7fe06fe00640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe04c06b4d0 0x7fe04c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.690+0000 7fe06fe00640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe04c06b4d0 0x7fe04c06d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe060035000 tx=0x7fe06002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.693+0000 7fe06cc00640 1 -- 172.21.15.114:0/1189753327 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe06801f020 con 0x7fe07015f9d0 2024-03-06T07:29:41.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.876+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe070072890 con 0x7fe07015f9d0 2024-03-06T07:29:41.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.878+0000 7fe06cc00640 1 -- 172.21.15.114:0/1189753327 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13070 (secure 0 0 0) 0x7fe0680172c0 con 0x7fe07015f9d0 2024-03-06T07:29:41.880 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:41.881 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":36,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:40.437250+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/3915705340":"2024-03-07T07:29:37.425210+0000","172.21.15.114:6836/3915705340":"2024-03-07T07:29:37.425210+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:41.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe04c06b4d0 msgr2=0x7fe04c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe04c06b4d0 0x7fe04c06d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe060035000 tx=0x7fe06002e000 comp rx=0 tx=0).stop 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe07015f9d0 msgr2=0x7fe07015b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe07015f9d0 0x7fe07015b120 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe06800b440 tx=0x7fe06800b900 comp rx=0 tx=0).stop 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 shutdown_connections 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe04c06b4d0 0x7fe04c06d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe07015f9d0 0x7fe07015b120 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe070144c00 0x7fe07015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 --2- 172.21.15.114:0/1189753327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0700ce0a0 0x7fe0701390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 >> 172.21.15.114:0/1189753327 conn(0x7fe0700b1730 msgr2=0x7fe070070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.880+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 shutdown_connections 2024-03-06T07:29:41.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:41.881+0000 7fe077200640 1 -- 172.21.15.114:0/1189753327 wait complete. 2024-03-06T07:29:41.900 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:41.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:42.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.097+0000 7f33fa200640 1 Processor -- start 2024-03-06T07:29:42.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.097+0000 7f33fa200640 1 -- start start 2024-03-06T07:29:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.098+0000 7f33fa200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f33f415be10 0x7f33f40ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33fa200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f33f406da40 0x7f33f40ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33fa200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f415c1f0 con 0x7f33f40aef20 2024-03-06T07:29:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33fa200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f40cd200 con 0x7f33f406da40 2024-03-06T07:29:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33fa200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f406cac0 con 0x7f33f415be10 2024-03-06T07:29:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33f9800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f33f406da40 0x7f33f40ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33f9800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f33f406da40 0x7f33f40ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45816/0 (socket says 172.21.15.114:45816) 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33f9800640 1 -- 172.21.15.114:0/1412027609 learned_addr learned my addr 172.21.15.114:0/1412027609 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33f8e00640 1 --2- 172.21.15.114:0/1412027609 >> v2:172.21.15.177:3300/0 conn(0x7f33f415be10 0x7f33f40ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33f9800640 1 -- 172.21.15.114:0/1412027609 >> v2:172.21.15.177:3300/0 conn(0x7f33f415be10 msgr2=0x7f33f40ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.099+0000 7f33f9800640 1 --2- 172.21.15.114:0/1412027609 >> v2:172.21.15.177:3300/0 conn(0x7f33f415be10 0x7f33f40ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.100+0000 7f33f9800640 1 -- 172.21.15.114:0/1412027609 >> v1:172.21.15.177:6789/0 conn(0x7f33f40aef20 legacy=0x7f33f406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.100+0000 7f33f9800640 1 -- 172.21.15.114:0/1412027609 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f406cda0 con 0x7f33f406da40 2024-03-06T07:29:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.101+0000 7f33f9800640 1 --2- 172.21.15.114:0/1412027609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f406da40 0x7f33f40ce2b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f33f40d09c0 tx=0x7f33ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a1cea46b8443f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.101+0000 7f33f3400640 1 -- 172.21.15.114:0/1412027609 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec02ca80 con 0x7f33f406da40 2024-03-06T07:29:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.101+0000 7f33f3400640 1 -- 172.21.15.114:0/1412027609 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec02cc20 con 0x7f33f406da40 2024-03-06T07:29:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.101+0000 7f33f3400640 1 -- 172.21.15.114:0/1412027609 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0343f0 con 0x7f33f406da40 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 -- 172.21.15.114:0/1412027609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f406da40 msgr2=0x7f33f40ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 --2- 172.21.15.114:0/1412027609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f406da40 0x7f33f40ce2b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f33f40d09c0 tx=0x7f33ec02bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 -- 172.21.15.114:0/1412027609 shutdown_connections 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 --2- 172.21.15.114:0/1412027609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f406da40 0x7f33f40ce2b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 --2- 172.21.15.114:0/1412027609 >> v2:172.21.15.177:3300/0 conn(0x7f33f415be10 0x7f33f40ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 -- 172.21.15.114:0/1412027609 >> 172.21.15.114:0/1412027609 conn(0x7f33f406afc0 msgr2=0x7f33f406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 -- 172.21.15.114:0/1412027609 shutdown_connections 2024-03-06T07:29:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.102+0000 7f33fa200640 1 -- 172.21.15.114:0/1412027609 wait complete. 2024-03-06T07:29:42.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.105+0000 7f33fa200640 1 Processor -- start 2024-03-06T07:29:42.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.105+0000 7f33fa200640 1 -- start start 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.106+0000 7f33fa200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33f406da40 0x7f33f40b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.106+0000 7f33fa200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f33f40aef20 0x7f33f414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.106+0000 7f33fa200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f415be10 0x7f33f40b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.106+0000 7f33fa200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f40cda90 con 0x7f33f415be10 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.106+0000 7f33fa200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f406d080 con 0x7f33f406da40 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.106+0000 7f33fa200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4071ab0 con 0x7f33f40aef20 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f415be10 0x7f33f40b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f415be10 0x7f33f40b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45824/0 (socket says 172.21.15.114:45824) 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 -- 172.21.15.114:0/1436024011 learned_addr learned my addr 172.21.15.114:0/1436024011 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f8e00640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33f406da40 0x7f33f40b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 -- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f33f40aef20 msgr2=0x7f33f414dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f3e00640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f33f40aef20 0x7f33f414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f33f40aef20 0x7f33f414dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 -- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33f406da40 msgr2=0x7f33f40b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33f406da40 0x7f33f40b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f9800640 1 -- 172.21.15.114:0/1436024011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f414e710 con 0x7f33f415be10 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f3e00640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f33f40aef20 0x7f33f414dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.107+0000 7f33f8e00640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33f406da40 0x7f33f40b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.108+0000 7f33f9800640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f415be10 0x7f33f40b3650 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009f40 tx=0x7f33ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.108+0000 7f33f0c00640 1 -- 172.21.15.114:0/1436024011 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec043020 con 0x7f33f415be10 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.108+0000 7f33f0c00640 1 -- 172.21.15.114:0/1436024011 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec03e450 con 0x7f33f415be10 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.108+0000 7f33f0c00640 1 -- 172.21.15.114:0/1436024011 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec009510 con 0x7f33f415be10 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.108+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f41f3ad0 con 0x7f33f415be10 2024-03-06T07:29:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.108+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f41f4090 con 0x7f33f415be10 2024-03-06T07:29:42.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.109+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f404f330 con 0x7f33f415be10 2024-03-06T07:29:42.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.110+0000 7f33f0c00640 1 -- 172.21.15.114:0/1436024011 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33ec046020 con 0x7f33f415be10 2024-03-06T07:29:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.111+0000 7f33f0c00640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33d406b4d0 0x7f33d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.111+0000 7f33f0c00640 1 -- 172.21.15.114:0/1436024011 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f33ec0ac7f0 con 0x7f33f415be10 2024-03-06T07:29:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.111+0000 7f33f8e00640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33d406b4d0 0x7f33d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.112+0000 7f33f8e00640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33d406b4d0 0x7f33d406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f33e8006fd0 tx=0x7f33e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.116+0000 7f33f0c00640 1 -- 172.21.15.114:0/1436024011 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33ec04d030 con 0x7f33f415be10 2024-03-06T07:29:42.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.312+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33f40ccbe0 con 0x7f33f415be10 2024-03-06T07:29:42.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.313+0000 7f33f0c00640 1 -- 172.21.15.114:0/1436024011 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33ec07b9f0 con 0x7f33f415be10 2024-03-06T07:29:42.316 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33d406b4d0 msgr2=0x7f33d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33d406b4d0 0x7f33d406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f33e8006fd0 tx=0x7f33e8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f415be10 msgr2=0x7f33f40b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f415be10 0x7f33f40b3650 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009f40 tx=0x7f33ec004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 shutdown_connections 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33d406b4d0 0x7f33d406d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33f415be10 0x7f33f40b3650 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f33f40aef20 0x7f33f414dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 --2- 172.21.15.114:0/1436024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33f406da40 0x7f33f40b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 >> 172.21.15.114:0/1436024011 conn(0x7f33f406afc0 msgr2=0x7f33f4159b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 shutdown_connections 2024-03-06T07:29:42.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.316+0000 7f33fa200640 1 -- 172.21.15.114:0/1436024011 wait complete. 2024-03-06T07:29:42.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:29:42.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.539+0000 7f0996a00640 1 Processor -- start 2024-03-06T07:29:42.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.539+0000 7f0996a00640 1 -- start start 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.540+0000 7f0996a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f099015f9d0 0x7f099015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.540+0000 7f0996a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0990070b10 0x7f0990070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.540+0000 7f0996a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099006cef0 con 0x7f0990144c00 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.540+0000 7f0996a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099006d1d0 con 0x7f0990070b10 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.540+0000 7f0996a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099006d4b0 con 0x7f099015f9d0 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0994a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0990070b10 0x7f0990070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0994a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0990070b10 0x7f0990070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45830/0 (socket says 172.21.15.114:45830) 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0994a00640 1 -- 172.21.15.114:0/3816631317 learned_addr learned my addr 172.21.15.114:0/3816631317 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0987e00640 1 --2- 172.21.15.114:0/3816631317 >> v2:172.21.15.177:3300/0 conn(0x7f099015f9d0 0x7f099015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0994a00640 1 -- 172.21.15.114:0/3816631317 >> v2:172.21.15.177:3300/0 conn(0x7f099015f9d0 msgr2=0x7f099015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0994a00640 1 --2- 172.21.15.114:0/3816631317 >> v2:172.21.15.177:3300/0 conn(0x7f099015f9d0 0x7f099015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0994a00640 1 -- 172.21.15.114:0/3816631317 >> v1:172.21.15.177:6789/0 conn(0x7f0990144c00 legacy=0x7f099006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.541+0000 7f0994a00640 1 -- 172.21.15.114:0/3816631317 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099006d790 con 0x7f0990070b10 2024-03-06T07:29:42.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.542+0000 7f0994a00640 1 --2- 172.21.15.114:0/3816631317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0990070b10 0x7f0990070f20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f09900b26f0 tx=0x7f098802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce964f1aa2266b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f098f400640 1 -- 172.21.15.114:0/3816631317 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098802ca80 con 0x7f0990070b10 2024-03-06T07:29:42.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f098f400640 1 -- 172.21.15.114:0/3816631317 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098802cc20 con 0x7f0990070b10 2024-03-06T07:29:42.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f098f400640 1 -- 172.21.15.114:0/3816631317 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09880343f0 con 0x7f0990070b10 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 -- 172.21.15.114:0/3816631317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0990070b10 msgr2=0x7f0990070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 --2- 172.21.15.114:0/3816631317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0990070b10 0x7f0990070f20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f09900b26f0 tx=0x7f098802bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 -- 172.21.15.114:0/3816631317 shutdown_connections 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 --2- 172.21.15.114:0/3816631317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0990070b10 0x7f0990070f20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 --2- 172.21.15.114:0/3816631317 >> v2:172.21.15.177:3300/0 conn(0x7f099015f9d0 0x7f099015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 -- 172.21.15.114:0/3816631317 >> 172.21.15.114:0/3816631317 conn(0x7f09900b1730 msgr2=0x7f09900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 -- 172.21.15.114:0/3816631317 shutdown_connections 2024-03-06T07:29:42.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.543+0000 7f0996a00640 1 -- 172.21.15.114:0/3816631317 wait complete. 2024-03-06T07:29:42.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.545+0000 7f0996a00640 1 Processor -- start 2024-03-06T07:29:42.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.545+0000 7f0996a00640 1 -- start start 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0996a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0990070b10 0x7f099015b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0996a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0990144c00 0x7f09900b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0996a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f099015f9d0 0x7f0990140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0996a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099006ccd0 con 0x7f099015f9d0 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0996a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099006cb50 con 0x7f0990070b10 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0996a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099006df90 con 0x7f0990144c00 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0994a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f099015f9d0 0x7f0990140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0994a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f099015f9d0 0x7f0990140130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45842/0 (socket says 172.21.15.114:45842) 2024-03-06T07:29:42.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.546+0000 7f0994a00640 1 -- 172.21.15.114:0/1438342578 learned_addr learned my addr 172.21.15.114:0/1438342578 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f098fe00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0990070b10 0x7f099015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f0994a00640 1 -- 172.21.15.114:0/1438342578 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0990144c00 msgr2=0x7f09900b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f0994a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0990144c00 0x7f09900b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f0994a00640 1 -- 172.21.15.114:0/1438342578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0990070b10 msgr2=0x7f099015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f0994a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0990070b10 0x7f099015b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f0994a00640 1 -- 172.21.15.114:0/1438342578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09900b3bd0 con 0x7f099015f9d0 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f0994a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f099015f9d0 0x7f0990140130 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0988036040 tx=0x7f0988004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f098cc00640 1 -- 172.21.15.114:0/1438342578 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988043020 con 0x7f099015f9d0 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f098cc00640 1 -- 172.21.15.114:0/1438342578 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098803e450 con 0x7f099015f9d0 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.547+0000 7f098cc00640 1 -- 172.21.15.114:0/1438342578 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988009510 con 0x7f099015f9d0 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.548+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09900b3eb0 con 0x7f099015f9d0 2024-03-06T07:29:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.548+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09901f9a20 con 0x7f099015f9d0 2024-03-06T07:29:42.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.549+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099004f330 con 0x7f099015f9d0 2024-03-06T07:29:42.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.550+0000 7f098cc00640 1 -- 172.21.15.114:0/1438342578 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0988046020 con 0x7f099015f9d0 2024-03-06T07:29:42.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.550+0000 7f098cc00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f096c06b4d0 0x7f096c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.551+0000 7f098cc00640 1 -- 172.21.15.114:0/1438342578 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09880ac760 con 0x7f099015f9d0 2024-03-06T07:29:42.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.551+0000 7f098fe00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f096c06b4d0 0x7f096c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.552+0000 7f098fe00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f096c06b4d0 0x7f096c06d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f09800015d0 tx=0x7f0980009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.559+0000 7f098cc00640 1 -- 172.21.15.114:0/1438342578 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f098804d030 con 0x7f099015f9d0 2024-03-06T07:29:42.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:42.826+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f09901f9bf0 con 0x7f099015f9d0 2024-03-06T07:29:43.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.462+0000 7f098cc00640 1 -- 172.21.15.114:0/1438342578 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v115) v1 ==== 136+0+0 (secure 0 0 0) 0x7f098807b9f0 con 0x7f099015f9d0 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f096c06b4d0 msgr2=0x7f096c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f096c06b4d0 0x7f096c06d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f09800015d0 tx=0x7f0980009040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f099015f9d0 msgr2=0x7f0990140130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f099015f9d0 0x7f0990140130 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0988036040 tx=0x7f0988004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 shutdown_connections 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f096c06b4d0 0x7f096c06d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f099015f9d0 0x7f0990140130 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0990144c00 0x7f09900b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 --2- 172.21.15.114:0/1438342578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0990070b10 0x7f099015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 >> 172.21.15.114:0/1438342578 conn(0x7f09900b1730 msgr2=0x7f0990142bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 shutdown_connections 2024-03-06T07:29:43.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.466+0000 7f0996a00640 1 -- 172.21.15.114:0/1438342578 wait complete. 2024-03-06T07:29:43.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:29:43.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.698+0000 7fba67200640 1 Processor -- start 2024-03-06T07:29:43.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.698+0000 7fba67200640 1 -- start start 2024-03-06T07:29:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.699+0000 7fba67200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fba68154dc0 0x7fba68161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.699+0000 7fba67200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba680cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.699+0000 7fba67200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680cfb30 con 0x7fba681551a0 2024-03-06T07:29:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.700+0000 7fba67200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6815e540 con 0x7fba68154dc0 2024-03-06T07:29:43.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.700+0000 7fba67200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba680cfe10 con 0x7fba680aeff0 2024-03-06T07:29:43.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.700+0000 7fba65e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fba68154dc0 0x7fba68161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.700+0000 7fba66800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba680cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.700+0000 7fba65e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fba68154dc0 0x7fba68161b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:57856/0 (socket says 172.21.15.114:57856) 2024-03-06T07:29:43.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.700+0000 7fba65e00640 1 -- 172.21.15.114:0/3010668298 learned_addr learned my addr 172.21.15.114:0/3010668298 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.701+0000 7fba65e00640 1 -- 172.21.15.114:0/3010668298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 msgr2=0x7fba680cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.701+0000 7fba65e00640 1 --2- 172.21.15.114:0/3010668298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba680cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.701+0000 7fba65e00640 1 -- 172.21.15.114:0/3010668298 >> v1:172.21.15.177:6789/0 conn(0x7fba681551a0 legacy=0x7fba680ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.701+0000 7fba65e00640 1 -- 172.21.15.114:0/3010668298 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680d00f0 con 0x7fba68154dc0 2024-03-06T07:29:43.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.702+0000 7fba65e00640 1 --2- 172.21.15.114:0/3010668298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 0x7fba68161b30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fba58009870 tx=0x7fba5802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=756e3e31dbb35127 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba65400640 1 -- 172.21.15.114:0/3010668298 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58034070 con 0x7fba68154dc0 2024-03-06T07:29:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba65400640 1 -- 172.21.15.114:0/3010668298 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58030840 con 0x7fba68154dc0 2024-03-06T07:29:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba65400640 1 -- 172.21.15.114:0/3010668298 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58030c10 con 0x7fba68154dc0 2024-03-06T07:29:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 -- 172.21.15.114:0/3010668298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 msgr2=0x7fba68161b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 --2- 172.21.15.114:0/3010668298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 0x7fba68161b30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fba58009870 tx=0x7fba5802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 -- 172.21.15.114:0/3010668298 shutdown_connections 2024-03-06T07:29:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 --2- 172.21.15.114:0/3010668298 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba680cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 --2- 172.21.15.114:0/3010668298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 0x7fba68161b30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 -- 172.21.15.114:0/3010668298 >> 172.21.15.114:0/3010668298 conn(0x7fba6806afc0 msgr2=0x7fba6806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 -- 172.21.15.114:0/3010668298 shutdown_connections 2024-03-06T07:29:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.703+0000 7fba67200640 1 -- 172.21.15.114:0/3010668298 wait complete. 2024-03-06T07:29:43.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.705+0000 7fba67200640 1 Processor -- start 2024-03-06T07:29:43.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.705+0000 7fba67200640 1 -- start start 2024-03-06T07:29:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.706+0000 7fba67200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba6814c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.706+0000 7fba67200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 0x7fba680b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.706+0000 7fba67200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba681551a0 0x7fba680b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.706+0000 7fba67200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68162070 con 0x7fba681551a0 2024-03-06T07:29:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.706+0000 7fba67200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680cf820 con 0x7fba68154dc0 2024-03-06T07:29:43.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.706+0000 7fba67200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba680b2080 con 0x7fba680aeff0 2024-03-06T07:29:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.706+0000 7fba66800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba681551a0 0x7fba680b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba681551a0 0x7fba680b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45846/0 (socket says 172.21.15.114:45846) 2024-03-06T07:29:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 -- 172.21.15.114:0/1715904670 learned_addr learned my addr 172.21.15.114:0/1715904670 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba5d400640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 0x7fba680b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba65e00640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba6814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 -- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 msgr2=0x7fba6814c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba6814c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 -- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 msgr2=0x7fba680b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 0x7fba680b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 -- 172.21.15.114:0/1715904670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba68155db0 con 0x7fba681551a0 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba66800640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba681551a0 0x7fba680b1b40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fba6000efc0 tx=0x7fba6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.707+0000 7fba5ea00640 1 -- 172.21.15.114:0/1715904670 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60018020 con 0x7fba681551a0 2024-03-06T07:29:43.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.708+0000 7fba5ea00640 1 -- 172.21.15.114:0/1715904670 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60014480 con 0x7fba681551a0 2024-03-06T07:29:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.708+0000 7fba5ea00640 1 -- 172.21.15.114:0/1715904670 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60014870 con 0x7fba681551a0 2024-03-06T07:29:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.708+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba68156050 con 0x7fba681551a0 2024-03-06T07:29:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.708+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba68156610 con 0x7fba681551a0 2024-03-06T07:29:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.709+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6804f330 con 0x7fba681551a0 2024-03-06T07:29:43.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.710+0000 7fba5ea00640 1 -- 172.21.15.114:0/1715904670 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba60005d10 con 0x7fba681551a0 2024-03-06T07:29:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.711+0000 7fba5ea00640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.711+0000 7fba5ea00640 1 -- 172.21.15.114:0/1715904670 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fba6008ab40 con 0x7fba681551a0 2024-03-06T07:29:43.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.711+0000 7fba65e00640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.712+0000 7fba65e00640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba4406b4d0 0x7fba4406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fba58036000 tx=0x7fba5803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.715+0000 7fba5ea00640 1 -- 172.21.15.114:0/1715904670 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba60022070 con 0x7fba681551a0 2024-03-06T07:29:43.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.912+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fba680add40 con 0x7fba681551a0 2024-03-06T07:29:43.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.918+0000 7fba5ea00640 1 -- 172.21.15.114:0/1715904670 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fba60010040 con 0x7fba681551a0 2024-03-06T07:29:43.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.920+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba4406b4d0 msgr2=0x7fba4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.920+0000 7fba67200640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba4406b4d0 0x7fba4406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fba58036000 tx=0x7fba5803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.920+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba681551a0 msgr2=0x7fba680b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.920+0000 7fba67200640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba681551a0 0x7fba680b1b40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fba6000efc0 tx=0x7fba6000c490 comp rx=0 tx=0).stop 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 shutdown_connections 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba681551a0 0x7fba680b1b40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba68154dc0 0x7fba680b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 --2- 172.21.15.114:0/1715904670 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba680aeff0 0x7fba6814c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 >> 172.21.15.114:0/1715904670 conn(0x7fba6806afc0 msgr2=0x7fba680ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 shutdown_connections 2024-03-06T07:29:43.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:43.921+0000 7fba67200640 1 -- 172.21.15.114:0/1715904670 wait complete. 2024-03-06T07:29:43.943 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:43.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:44.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:44.171 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:44.171 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:29:44.171 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:44.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:44.228 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:29:44.228 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:44.285 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:44.285 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:29:44.285 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:44.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:44.342 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:29:44.342 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:44.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:44.394 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:29:44.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.610+0000 7f7c20200640 1 Processor -- start 2024-03-06T07:29:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.610+0000 7f7c20200640 1 -- start start 2024-03-06T07:29:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.612+0000 7f7c20200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 0x7f7c1815eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.612+0000 7f7c20200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7c1806e4d0 0x7f7c1806e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.612+0000 7f7c20200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c18071d30 con 0x7f7c18070d20 2024-03-06T07:29:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.612+0000 7f7c20200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c18071490 con 0x7f7c1806e4d0 2024-03-06T07:29:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.612+0000 7f7c20200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c18071770 con 0x7f7c180af1c0 2024-03-06T07:29:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.612+0000 7f7c1ce00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7c18070d20 0x7f7c1806cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58808/0 (socket says 172.21.15.114:58808) 2024-03-06T07:29:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.612+0000 7f7c1ce00640 1 -- 172.21.15.114:0/22463851 learned_addr learned my addr 172.21.15.114:0/22463851 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c1d800640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 0x7f7c1815eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c1e200640 1 --2- 172.21.15.114:0/22463851 >> v2:172.21.15.177:3300/0 conn(0x7f7c1806e4d0 0x7f7c1806e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884377687 0 0) 0x7f7c04001110 con 0x7f7c18070d20 2024-03-06T07:29:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c00003520 con 0x7f7c18070d20 2024-03-06T07:29:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004654544 0 0) 0x7f7c00003520 con 0x7f7c18070d20 2024-03-06T07:29:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 msgr2=0x7f7c1815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 0x7f7c1815eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 >> v2:172.21.15.177:3300/0 conn(0x7f7c1806e4d0 msgr2=0x7f7c1806e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 --2- 172.21.15.114:0/22463851 >> v2:172.21.15.177:3300/0 conn(0x7f7c1806e4d0 0x7f7c1806e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18071a50 con 0x7f7c18070d20 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7c04001840 con 0x7f7c18070d20 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 >> v1:172.21.15.114:6789/0 conn(0x7f7c18070d20 legacy=0x7f7c1806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 0x7f7c000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c00003f30 0x7f7c00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c00004d40 0x7f7c000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c18071490 con 0x7f7c00004d40 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c18071770 con 0x7f7c00003f30 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.613+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0000b9c0 con 0x7f7c180af1c0 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1e200640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c00004d40 0x7f7c000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1d800640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 0x7f7c000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1ce00640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c00003f30 0x7f7c00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1e200640 1 -- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 msgr2=0x7f7c000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1e200640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c180af1c0 0x7f7c000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1e200640 1 -- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c00003f30 msgr2=0x7f7c00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1e200640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c00003f30 0x7f7c00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.614+0000 7f7c1e200640 1 -- 172.21.15.114:0/22463851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00003520 con 0x7f7c00004d40 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.616+0000 7f7c1e200640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c00004d40 0x7f7c000051c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009fc0 tx=0x7f7c00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24697a61001dabb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.616+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10039070 con 0x7f7c00004d40 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.616+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1002cc40 con 0x7f7c00004d40 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.616+0000 7f7c17e00640 1 -- 172.21.15.114:0/22463851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c10009c10 con 0x7f7c00004d40 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.616+0000 7f7c20200640 1 -- 172.21.15.114:0/22463851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c00004d40 msgr2=0x7f7c000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.616+0000 7f7c20200640 1 --2- 172.21.15.114:0/22463851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c00004d40 0x7f7c000051c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009fc0 tx=0x7f7c00005830 comp rx=0 tx=0).stop 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.616+0000 7f7c1e200640 1 -- 172.21.15.114:0/22463851 reap_dead start 2024-03-06T07:29:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.617+0000 7f7c20200640 1 -- 172.21.15.114:0/22463851 shutdown_connections 2024-03-06T07:29:44.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.617+0000 7f7c20200640 1 -- 172.21.15.114:0/22463851 >> 172.21.15.114:0/22463851 conn(0x7f7c180b1730 msgr2=0x7f7c180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.617+0000 7f7c20200640 1 -- 172.21.15.114:0/22463851 shutdown_connections 2024-03-06T07:29:44.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.617+0000 7f7c20200640 1 -- 172.21.15.114:0/22463851 wait complete. 2024-03-06T07:29:44.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.618+0000 7f7c20200640 1 Processor -- start 2024-03-06T07:29:44.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.619+0000 7f7c20200640 1 -- start start 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.619+0000 7f7c20200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18141df0 0x7f7c18158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.619+0000 7f7c20200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c181421d0 0x7f7c18155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.619+0000 7f7c20200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c18152260 0x7f7c18152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.619+0000 7f7c20200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1806d370 con 0x7f7c18141df0 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.619+0000 7f7c20200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180711a0 con 0x7f7c181421d0 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.619+0000 7f7c20200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1815eff0 con 0x7f7c18152260 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18141df0 0x7f7c18158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18141df0 0x7f7c18158d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45862/0 (socket says 172.21.15.114:45862) 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 -- 172.21.15.114:0/1849811438 learned_addr learned my addr 172.21.15.114:0/1849811438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1ce00640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c181421d0 0x7f7c18155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1e200640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c18152260 0x7f7c18152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 -- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c18152260 msgr2=0x7f7c18152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c18152260 0x7f7c18152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 -- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c181421d0 msgr2=0x7f7c18155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c181421d0 0x7f7c18155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1d800640 1 -- 172.21.15.114:0/1849811438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c181555d0 con 0x7f7c18141df0 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.620+0000 7f7c1ce00640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c181421d0 0x7f7c18155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.621+0000 7f7c1d800640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18141df0 0x7f7c18158d60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c011600 tx=0x7f7c0c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.621+0000 7f7c15600640 1 -- 172.21.15.114:0/1849811438 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00f920 con 0x7f7c18141df0 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.621+0000 7f7c15600640 1 -- 172.21.15.114:0/1849811438 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c00fac0 con 0x7f7c18141df0 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.621+0000 7f7c15600640 1 -- 172.21.15.114:0/1849811438 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00fe70 con 0x7f7c18141df0 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.621+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c18159420 con 0x7f7c18141df0 2024-03-06T07:29:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.621+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c181599e0 con 0x7f7c18141df0 2024-03-06T07:29:44.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.622+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1804f330 con 0x7f7c18141df0 2024-03-06T07:29:44.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.623+0000 7f7c15600640 1 -- 172.21.15.114:0/1849811438 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c0c015460 con 0x7f7c18141df0 2024-03-06T07:29:44.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.624+0000 7f7c15600640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bf806b550 0x7f7bf806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.624+0000 7f7c15600640 1 -- 172.21.15.114:0/1849811438 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7c0c08a7d0 con 0x7f7c18141df0 2024-03-06T07:29:44.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.624+0000 7f7c1ce00640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bf806b550 0x7f7bf806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.625+0000 7f7c1ce00640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bf806b550 0x7f7bf806da10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7c04004840 tx=0x7f7c04000fe0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.628+0000 7f7c15600640 1 -- 172.21.15.114:0/1849811438 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c0c019030 con 0x7f7c18141df0 2024-03-06T07:29:44.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.867+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c180b4360 con 0x7f7c18141df0 2024-03-06T07:29:44.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.868+0000 7f7c15600640 1 -- 172.21.15.114:0/1849811438 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f7c0c0599b0 con 0x7f7c18141df0 2024-03-06T07:29:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:44.871 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1079445588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1079445588},{"type":"v1","addr":"172.21.15.114:6835","nonce":1079445588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4592,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/783279268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":783279268},{"type":"v1","addr":"172.21.15.177:6835","nonce":783279268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3549417735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3549417735},{"type":"v1","addr":"172.21.15.177:6833","nonce":3549417735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3936379832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3936379832},{"type":"v1","addr":"172.21.15.114:6837","nonce":3936379832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-03-06T07:29:44.871 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 19 2024-03-06T07:29:44.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bf806b550 msgr2=0x7f7bf806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bf806b550 0x7f7bf806da10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7c04004840 tx=0x7f7c04000fe0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18141df0 msgr2=0x7f7c18158d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18141df0 0x7f7c18158d60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c011600 tx=0x7f7c0c011ac0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 shutdown_connections 2024-03-06T07:29:44.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c18152260 0x7f7c18152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c181421d0 0x7f7c18155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bf806b550 0x7f7bf806da10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 --2- 172.21.15.114:0/1849811438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18141df0 0x7f7c18158d60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.871+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 >> 172.21.15.114:0/1849811438 conn(0x7f7c180b1730 msgr2=0x7f7c181600b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.872+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 shutdown_connections 2024-03-06T07:29:44.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:44.872+0000 7f7c20200640 1 -- 172.21.15.114:0/1849811438 wait complete. 2024-03-06T07:29:44.893 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:29:44.893 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:29:44.893 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:29:44.893 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:44.893 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:29:44.893 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:29:44.894 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:29:44.894 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:44.894 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:29:44.894 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:29:44.894 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:29:44.894 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:44.894 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:29:44.895 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:29:44.895 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:29:44.895 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:44.910 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:29:44.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:29:44.916 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:29:44.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:29:44.917 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:29:44.917 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:29:44.922 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:29:44.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:29:44.964 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:29:44.964 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:44.968 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:29:44.968 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:44.970 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:29:44.970 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:44.973 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:29:44.974 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:44.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:29:45.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:45.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:45.118 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:29:45.123 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:29:45.124 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:29:45.130 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:29:45.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.159+0000 7f5f81200640 1 Processor -- start 2024-03-06T07:29:45.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.159+0000 7f5f81200640 1 -- start start 2024-03-06T07:29:45.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.160+0000 7f5f81200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f7c0d0600 0x7f5f7c0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.160+0000 7f5f81200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0b2a60 0x7f5f7c0d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f81200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5f7c0d18e0 0x7f5f7c0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f81200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0cf090 con 0x7f5f7c0d0600 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f81200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d0f20 con 0x7f5f7c0d18e0 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f81200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c15f030 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f7c0d0600 0x7f5f7c0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f7c0d0600 0x7f5f7c0d4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45898/0 (socket says 172.21.15.114:45898) 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7b400640 1 -- 172.21.15.114:0/4146460419 learned_addr learned my addr 172.21.15.114:0/4146460419 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7aa00640 1 --2- 172.21.15.114:0/4146460419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0b2a60 0x7f5f7c0d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7be00640 1 --2- 172.21.15.114:0/4146460419 >> v2:172.21.15.177:3300/0 conn(0x7f5f7c0d18e0 0x7f5f7c0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7aa00640 1 -- 172.21.15.114:0/4146460419 >> v2:172.21.15.177:3300/0 conn(0x7f5f7c0d18e0 msgr2=0x7f5f7c0d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7aa00640 1 --2- 172.21.15.114:0/4146460419 >> v2:172.21.15.177:3300/0 conn(0x7f5f7c0d18e0 0x7f5f7c0d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7aa00640 1 -- 172.21.15.114:0/4146460419 >> v2:172.21.15.114:3300/0 conn(0x7f5f7c0d0600 msgr2=0x7f5f7c0d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7aa00640 1 --2- 172.21.15.114:0/4146460419 >> v2:172.21.15.114:3300/0 conn(0x7f5f7c0d0600 0x7f5f7c0d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7aa00640 1 -- 172.21.15.114:0/4146460419 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c15f310 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.161+0000 7f5f7b400640 1 --2- 172.21.15.114:0/4146460419 >> v2:172.21.15.114:3300/0 conn(0x7f5f7c0d0600 0x7f5f7c0d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:45.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.163+0000 7f5f7aa00640 1 --2- 172.21.15.114:0/4146460419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0b2a60 0x7f5f7c0d09e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c0d3790 tx=0x7f5f6802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f475e0e8fc2e0b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.163+0000 7f5f7a000640 1 -- 172.21.15.114:0/4146460419 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f680315a0 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.163+0000 7f5f7a000640 1 -- 172.21.15.114:0/4146460419 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f68037070 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.163+0000 7f5f7a000640 1 -- 172.21.15.114:0/4146460419 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f68031960 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.163+0000 7f5f81200640 1 -- 172.21.15.114:0/4146460419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0b2a60 msgr2=0x7f5f7c0d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.163+0000 7f5f81200640 1 --2- 172.21.15.114:0/4146460419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0b2a60 0x7f5f7c0d09e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c0d3790 tx=0x7f5f6802b840 comp rx=0 tx=0).stop 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.164+0000 7f5f81200640 1 -- 172.21.15.114:0/4146460419 shutdown_connections 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.164+0000 7f5f81200640 1 --2- 172.21.15.114:0/4146460419 >> v2:172.21.15.177:3300/0 conn(0x7f5f7c0d18e0 0x7f5f7c0d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.164+0000 7f5f81200640 1 --2- 172.21.15.114:0/4146460419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0b2a60 0x7f5f7c0d09e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.164+0000 7f5f81200640 1 --2- 172.21.15.114:0/4146460419 >> v2:172.21.15.114:3300/0 conn(0x7f5f7c0d0600 0x7f5f7c0d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.164+0000 7f5f81200640 1 -- 172.21.15.114:0/4146460419 >> 172.21.15.114:0/4146460419 conn(0x7f5f7c0d3d10 msgr2=0x7f5f7c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.164+0000 7f5f81200640 1 -- 172.21.15.114:0/4146460419 shutdown_connections 2024-03-06T07:29:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.164+0000 7f5f81200640 1 -- 172.21.15.114:0/4146460419 wait complete. 2024-03-06T07:29:45.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.165+0000 7f5f81200640 1 Processor -- start 2024-03-06T07:29:45.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.165+0000 7f5f81200640 1 -- start start 2024-03-06T07:29:45.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.166+0000 7f5f81200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f7c0b2a60 0x7f5f7c159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.166+0000 7f5f81200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0d0600 0x7f5f7c152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.166+0000 7f5f81200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f7c0d18e0 0x7f5f7c153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.166+0000 7f5f81200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0b3250 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.166+0000 7f5f81200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0ced20 con 0x7f5f7c0d18e0 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.166+0000 7f5f81200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c15ecf0 con 0x7f5f7c0d0600 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f7c0b2a60 0x7f5f7c159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f7c0b2a60 0x7f5f7c159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45904/0 (socket says 172.21.15.114:45904) 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 -- 172.21.15.114:0/163914883 learned_addr learned my addr 172.21.15.114:0/163914883 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7be00640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f7c0d18e0 0x7f5f7c153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7aa00640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0d0600 0x7f5f7c152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 -- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0d0600 msgr2=0x7f5f7c152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0d0600 0x7f5f7c152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 -- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f7c0d18e0 msgr2=0x7f5f7c153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f7c0d18e0 0x7f5f7c153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 -- 172.21.15.114:0/163914883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0b6dc0 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.167+0000 7f5f7b400640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f7c0b2a60 0x7f5f7c159bb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00e9b0 tx=0x7f5f6c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.168+0000 7f5f73400640 1 -- 172.21.15.114:0/163914883 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c00ca00 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.168+0000 7f5f73400640 1 -- 172.21.15.114:0/163914883 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c00cba0 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.168+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c0b70a0 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.168+0000 7f5f73400640 1 -- 172.21.15.114:0/163914883 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c004030 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.168+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c0b7660 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.169+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c04f330 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.170+0000 7f5f73400640 1 -- 172.21.15.114:0/163914883 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f6c0041d0 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.171+0000 7f5f73400640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.171+0000 7f5f73400640 1 -- 172.21.15.114:0/163914883 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5f6c08a340 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.171+0000 7f5f7aa00640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.172+0000 7f5f7aa00640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5f68036000 tx=0x7f5f6803d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.176+0000 7f5f73400640 1 -- 172.21.15.114:0/163914883 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f6c017020 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.382+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5f7c08d7e0 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.485+0000 7f5f73400640 1 -- 172.21.15.114:0/163914883 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5f6c059410 con 0x7f5f7c0b2a60 2024-03-06T07:29:45.488 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.488+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f5c06b4d0 msgr2=0x7f5f5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.488+0000 7f5f81200640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5f68036000 tx=0x7f5f6803d040 comp rx=0 tx=0).stop 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f7c0b2a60 msgr2=0x7f5f7c159bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f7c0b2a60 0x7f5f7c159bb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00e9b0 tx=0x7f5f6c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 shutdown_connections 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f7c0d18e0 0x7f5f7c153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5f7c0d0600 0x7f5f7c152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5f5c06b4d0 0x7f5f5c06d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 --2- 172.21.15.114:0/163914883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f7c0b2a60 0x7f5f7c159bb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 >> 172.21.15.114:0/163914883 conn(0x7f5f7c0d3d10 msgr2=0x7f5f7c0b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 shutdown_connections 2024-03-06T07:29:45.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.489+0000 7f5f81200640 1 -- 172.21.15.114:0/163914883 wait complete. 2024-03-06T07:29:45.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:29:45.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.749+0000 7f4839c00640 1 Processor -- start 2024-03-06T07:29:45.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.749+0000 7f4839c00640 1 -- start start 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f4839c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4834144c00 0x7f483406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f4839c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f483415fe60 0x7f48340ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f4839c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483406ee70 con 0x7f483415f9d0 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f4839c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483406cef0 con 0x7f4834144c00 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f4839c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483406d1d0 con 0x7f483415fe60 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f4832a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4834144c00 0x7f483406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f482a000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f483415f9d0 0x7f48340ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58814/0 (socket says 172.21.15.114:58814) 2024-03-06T07:29:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.751+0000 7f482a000640 1 -- 172.21.15.114:0/2444717698 learned_addr learned my addr 172.21.15.114:0/2444717698 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.752+0000 7f4833400640 1 --2- 172.21.15.114:0/2444717698 >> v2:172.21.15.177:3300/0 conn(0x7f483415fe60 0x7f48340ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.752+0000 7f4832a00640 1 -- 172.21.15.114:0/2444717698 >> v2:172.21.15.177:3300/0 conn(0x7f483415fe60 msgr2=0x7f48340ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.752+0000 7f4832a00640 1 --2- 172.21.15.114:0/2444717698 >> v2:172.21.15.177:3300/0 conn(0x7f483415fe60 0x7f48340ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.752+0000 7f4832a00640 1 -- 172.21.15.114:0/2444717698 >> v1:172.21.15.114:6789/0 conn(0x7f483415f9d0 legacy=0x7f48340ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.752+0000 7f4832000640 1 -- 172.21.15.114:0/2444717698 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160750353 0 0) 0x7f481c001110 con 0x7f483415f9d0 2024-03-06T07:29:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.752+0000 7f4832a00640 1 -- 172.21.15.114:0/2444717698 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483406d4b0 con 0x7f4834144c00 2024-03-06T07:29:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.752+0000 7f4833400640 1 --2- 172.21.15.114:0/2444717698 >> v2:172.21.15.177:3300/0 conn(0x7f483415fe60 0x7f48340ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.753+0000 7f4832a00640 1 --2- 172.21.15.114:0/2444717698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483406e930 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f483406e500 tx=0x7f482402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b8c349275adffc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.753+0000 7f4832000640 1 -- 172.21.15.114:0/2444717698 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824035020 con 0x7f4834144c00 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.753+0000 7f4832000640 1 -- 172.21.15.114:0/2444717698 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824031830 con 0x7f4834144c00 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.753+0000 7f4832000640 1 -- 172.21.15.114:0/2444717698 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824031c00 con 0x7f4834144c00 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 -- 172.21.15.114:0/2444717698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 msgr2=0x7f483406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 --2- 172.21.15.114:0/2444717698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483406e930 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f483406e500 tx=0x7f482402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 -- 172.21.15.114:0/2444717698 shutdown_connections 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 --2- 172.21.15.114:0/2444717698 >> v2:172.21.15.177:3300/0 conn(0x7f483415fe60 0x7f48340ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 --2- 172.21.15.114:0/2444717698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483406e930 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 -- 172.21.15.114:0/2444717698 >> 172.21.15.114:0/2444717698 conn(0x7f48340b1730 msgr2=0x7f48340b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 -- 172.21.15.114:0/2444717698 shutdown_connections 2024-03-06T07:29:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.754+0000 7f4839c00640 1 -- 172.21.15.114:0/2444717698 wait complete. 2024-03-06T07:29:45.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.756+0000 7f4839c00640 1 Processor -- start 2024-03-06T07:29:45.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.756+0000 7f4839c00640 1 -- start start 2024-03-06T07:29:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4839c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483413a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4839c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f483415f9d0 0x7f48341392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4839c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f483415fe60 0x7f483415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4839c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4834160730 con 0x7f4834144c00 2024-03-06T07:29:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4839c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4834160270 con 0x7f483415f9d0 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4839c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483406cb50 con 0x7f483415fe60 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483413a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483413a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45922/0 (socket says 172.21.15.114:45922) 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 -- 172.21.15.114:0/1205879488 learned_addr learned my addr 172.21.15.114:0/1205879488 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 -- 172.21.15.114:0/1205879488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f483415fe60 msgr2=0x7f483415ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f483415fe60 0x7f483415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 -- 172.21.15.114:0/1205879488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f483415f9d0 msgr2=0x7f48341392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f483415f9d0 0x7f48341392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.757+0000 7f4832a00640 1 -- 172.21.15.114:0/1205879488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483413b630 con 0x7f4834144c00 2024-03-06T07:29:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.758+0000 7f4832a00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483413a460 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4824000c00 tx=0x7f48240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.758+0000 7f482b400640 1 -- 172.21.15.114:0/1205879488 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824041070 con 0x7f4834144c00 2024-03-06T07:29:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.758+0000 7f482b400640 1 -- 172.21.15.114:0/1205879488 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824004230 con 0x7f4834144c00 2024-03-06T07:29:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.758+0000 7f482b400640 1 -- 172.21.15.114:0/1205879488 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824035040 con 0x7f4834144c00 2024-03-06T07:29:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.758+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483413ad90 con 0x7f4834144c00 2024-03-06T07:29:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.758+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483413b350 con 0x7f4834144c00 2024-03-06T07:29:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.759+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483404f330 con 0x7f4834144c00 2024-03-06T07:29:45.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.760+0000 7f482b400640 1 -- 172.21.15.114:0/1205879488 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4824038020 con 0x7f4834144c00 2024-03-06T07:29:45.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.761+0000 7f482b400640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f481006b500 0x7f481006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.761+0000 7f482b400640 1 -- 172.21.15.114:0/1205879488 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f48240aba80 con 0x7f4834144c00 2024-03-06T07:29:45.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.761+0000 7f482a000640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f481006b500 0x7f481006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.762+0000 7f482a000640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f481006b500 0x7f481006d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f48341f7910 tx=0x7f481c001310 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:45.765+0000 7f482b400640 1 -- 172.21.15.114:0/1205879488 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48240ade20 con 0x7f4834144c00 2024-03-06T07:29:46.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.029+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f48340b0c20 con 0x7f4834144c00 2024-03-06T07:29:46.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.030+0000 7f482b400640 1 -- 172.21.15.114:0/1205879488 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f482407abb0 con 0x7f4834144c00 2024-03-06T07:29:46.032 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:29:46.033 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.035 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.036 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.038 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.039 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow r", 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.040 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:29:46.042 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:29:46.043 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:29:46.044 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:29:46.044 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:46.044 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:29:46.044 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.044 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:46.044 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:29:46.044 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f481006b500 msgr2=0x7f481006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f481006b500 0x7f481006d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f48341f7910 tx=0x7f481c001310 comp rx=0 tx=0).stop 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 msgr2=0x7f483413a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483413a460 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4824000c00 tx=0x7f48240046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 shutdown_connections 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f483415fe60 0x7f483415ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f483415f9d0 0x7f48341392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f481006b500 0x7f481006d9c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 --2- 172.21.15.114:0/1205879488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4834144c00 0x7f483413a460 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 >> 172.21.15.114:0/1205879488 conn(0x7f48340b1730 msgr2=0x7f4834142d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 shutdown_connections 2024-03-06T07:29:46.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.032+0000 7f4839c00640 1 -- 172.21.15.114:0/1205879488 wait complete. 2024-03-06T07:29:46.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.244+0000 7fb234e00640 1 Processor -- start 2024-03-06T07:29:46.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.244+0000 7fb234e00640 1 -- start start 2024-03-06T07:29:46.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.245+0000 7fb234e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23006ec00 0x7fb2300ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.245+0000 7fb234e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23015b090 con 0x7fb23015acb0 2024-03-06T07:29:46.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.245+0000 7fb234e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2300b2a30 con 0x7fb2300b4680 2024-03-06T07:29:46.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.245+0000 7fb234e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300b4360 con 0x7fb23006ec00 2024-03-06T07:29:46.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.245+0000 7fb22de00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb23015acb0 0x7fb2300b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58828/0 (socket says 172.21.15.114:58828) 2024-03-06T07:29:46.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.245+0000 7fb22de00640 1 -- 172.21.15.114:0/1325448923 learned_addr learned my addr 172.21.15.114:0/1325448923 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539490368 0 0) 0x7fb23015b090 con 0x7fb23015acb0 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22e800640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23006ec00 0x7fb2300ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb214005730 con 0x7fb23015acb0 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066840029 0 0) 0x7fb214005730 con 0x7fb23015acb0 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23006ec00 msgr2=0x7fb2300ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23006ec00 0x7fb2300ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 >> v1:172.21.15.177:6789/0 conn(0x7fb2300b4680 legacy=0x7fb23006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2300b2cf0 con 0x7fb23015acb0 2024-03-06T07:29:46.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb220001670 con 0x7fb23015acb0 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 >> v1:172.21.15.114:6789/0 conn(0x7fb23015acb0 legacy=0x7fb2300b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb214006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23006ec00 0x7fb214006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214006f50 0x7fb2140073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300b4360 con 0x7fb214005bf0 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21400dcb0 con 0x7fb214006f50 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.246+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21400deb0 con 0x7fb23006ec00 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.247+0000 7fb22de00640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb214006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.247+0000 7fb22de00640 1 -- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23006ec00 msgr2=0x7fb214006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.247+0000 7fb22de00640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23006ec00 0x7fb214006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.247+0000 7fb22de00640 1 -- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214006f50 msgr2=0x7fb2140073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.247+0000 7fb22de00640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb214006f50 0x7fb2140073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.247+0000 7fb22de00640 1 -- 172.21.15.114:0/1325448923 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb214005730 con 0x7fb214005bf0 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.248+0000 7fb22de00640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb214006010 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c210 tx=0x7fb214007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495f6899081e69a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.248+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220007d10 con 0x7fb214005bf0 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.248+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb220030070 con 0x7fb214005bf0 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.248+0000 7fb22d400640 1 -- 172.21.15.114:0/1325448923 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220005d30 con 0x7fb214005bf0 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.249+0000 7fb234e00640 1 -- 172.21.15.114:0/1325448923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 msgr2=0x7fb214006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.249+0000 7fb234e00640 1 --2- 172.21.15.114:0/1325448923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb214006010 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c210 tx=0x7fb214007a40 comp rx=0 tx=0).stop 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.249+0000 7fb22e800640 1 -- 172.21.15.114:0/1325448923 reap_dead start 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.249+0000 7fb234e00640 1 -- 172.21.15.114:0/1325448923 shutdown_connections 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.249+0000 7fb234e00640 1 -- 172.21.15.114:0/1325448923 >> 172.21.15.114:0/1325448923 conn(0x7fb2300b1730 msgr2=0x7fb2300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.249+0000 7fb234e00640 1 -- 172.21.15.114:0/1325448923 shutdown_connections 2024-03-06T07:29:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.249+0000 7fb234e00640 1 -- 172.21.15.114:0/1325448923 wait complete. 2024-03-06T07:29:46.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.251+0000 7fb234e00640 1 Processor -- start 2024-03-06T07:29:46.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.251+0000 7fb234e00640 1 -- start start 2024-03-06T07:29:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb234e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb2300ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb234e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23013f660 0x7fb2301ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb234e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb230140ce0 0x7fb230141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb234e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300b4010 con 0x7fb214005bf0 2024-03-06T07:29:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb234e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300b32b0 con 0x7fb230140ce0 2024-03-06T07:29:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb234e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb230072340 con 0x7fb23013f660 2024-03-06T07:29:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb2300ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb2300ce490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45942/0 (socket says 172.21.15.114:45942) 2024-03-06T07:29:46.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 -- 172.21.15.114:0/1537656046 learned_addr learned my addr 172.21.15.114:0/1537656046 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 -- 172.21.15.114:0/1537656046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23013f660 msgr2=0x7fb2301ef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23013f660 0x7fb2301ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 -- 172.21.15.114:0/1537656046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb230140ce0 msgr2=0x7fb230141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb230140ce0 0x7fb230141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.252+0000 7fb22de00640 1 -- 172.21.15.114:0/1537656046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23013fdd0 con 0x7fb214005bf0 2024-03-06T07:29:46.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.253+0000 7fb22de00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb2300ce490 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c210 tx=0x7fb2300b4af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.253+0000 7fb226a00640 1 -- 172.21.15.114:0/1537656046 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220007d10 con 0x7fb214005bf0 2024-03-06T07:29:46.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.253+0000 7fb226a00640 1 -- 172.21.15.114:0/1537656046 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb220004bf0 con 0x7fb214005bf0 2024-03-06T07:29:46.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.253+0000 7fb226a00640 1 -- 172.21.15.114:0/1537656046 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220007070 con 0x7fb214005bf0 2024-03-06T07:29:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.253+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2301400b0 con 0x7fb214005bf0 2024-03-06T07:29:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.253+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb230140670 con 0x7fb214005bf0 2024-03-06T07:29:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.255+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2301416a0 con 0x7fb214005bf0 2024-03-06T07:29:46.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.255+0000 7fb226a00640 1 -- 172.21.15.114:0/1537656046 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb220007230 con 0x7fb214005bf0 2024-03-06T07:29:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.256+0000 7fb226a00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb20c06b500 0x7fb20c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.256+0000 7fb226a00640 1 -- 172.21.15.114:0/1537656046 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb2200ab630 con 0x7fb214005bf0 2024-03-06T07:29:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.257+0000 7fb225400640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb20c06b500 0x7fb20c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.257+0000 7fb225400640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb20c06b500 0x7fb20c06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb230145360 tx=0x7fb218007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.260+0000 7fb226a00640 1 -- 172.21.15.114:0/1537656046 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2301416a0 con 0x7fb214005bf0 2024-03-06T07:29:46.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.457+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb23004f330 con 0x7fb214005bf0 2024-03-06T07:29:46.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.459+0000 7fb226a00640 1 -- 172.21.15.114:0/1537656046 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+12939 (secure 0 0 0) 0x7fb22007a6d0 con 0x7fb214005bf0 2024-03-06T07:29:46.462 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:46.462 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":40,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:45.501764+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:46.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb20c06b500 msgr2=0x7fb20c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb20c06b500 0x7fb20c06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb230145360 tx=0x7fb218007990 comp rx=0 tx=0).stop 2024-03-06T07:29:46.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 msgr2=0x7fb2300ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb2300ce490 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c210 tx=0x7fb2300b4af0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 shutdown_connections 2024-03-06T07:29:46.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb230140ce0 0x7fb230141160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb23013f660 0x7fb2301ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb20c06b500 0x7fb20c06d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 --2- 172.21.15.114:0/1537656046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb214005bf0 0x7fb2300ce490 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.461+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 >> 172.21.15.114:0/1537656046 conn(0x7fb2300b1730 msgr2=0x7fb2300b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.462+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 shutdown_connections 2024-03-06T07:29:46.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.462+0000 7fb234e00640 1 -- 172.21.15.114:0/1537656046 wait complete. 2024-03-06T07:29:46.483 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:46.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:46.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.678+0000 7f8344e00640 1 Processor -- start 2024-03-06T07:29:46.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.678+0000 7f8344e00640 1 -- start start 2024-03-06T07:29:46.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f8344e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f834006da40 0x7f83400ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f8344e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834015c1f0 con 0x7f834015be10 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f8344e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834006cac0 con 0x7f83400aef20 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f8344e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83400cd200 con 0x7f834006da40 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833f400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f834015be10 0x7f834006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58844/0 (socket says 172.21.15.114:58844) 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833f400640 1 -- 172.21.15.114:0/1383965530 learned_addr learned my addr 172.21.15.114:0/1383965530 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833ea00640 1 --2- 172.21.15.114:0/1383965530 >> v2:172.21.15.114:3300/0 conn(0x7f834006da40 0x7f83400ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833e000640 1 -- 172.21.15.114:0/1383965530 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59972112 0 0) 0x7f834015c1f0 con 0x7f834015be10 2024-03-06T07:29:46.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833e000640 1 -- 172.21.15.114:0/1383965530 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8324003520 con 0x7f834015be10 2024-03-06T07:29:46.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833ea00640 1 -- 172.21.15.114:0/1383965530 >> v1:172.21.15.177:6789/0 conn(0x7f83400aef20 legacy=0x7f8340071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833ea00640 1 -- 172.21.15.114:0/1383965530 >> v1:172.21.15.114:6789/0 conn(0x7f834015be10 legacy=0x7f834006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.680+0000 7f833ea00640 1 -- 172.21.15.114:0/1383965530 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834006cda0 con 0x7f834006da40 2024-03-06T07:29:46.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.682+0000 7f833ea00640 1 --2- 172.21.15.114:0/1383965530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834006da40 0x7f83400ccc20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f832c0097a0 tx=0x7f832c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2757610e15ece3b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.682+0000 7f833e000640 1 -- 172.21.15.114:0/1383965530 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c0315a0 con 0x7f834006da40 2024-03-06T07:29:46.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.682+0000 7f833e000640 1 -- 172.21.15.114:0/1383965530 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c037070 con 0x7f834006da40 2024-03-06T07:29:46.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.682+0000 7f833e000640 1 -- 172.21.15.114:0/1383965530 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c031960 con 0x7f834006da40 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.683+0000 7f8344e00640 1 -- 172.21.15.114:0/1383965530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834006da40 msgr2=0x7f83400ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.683+0000 7f8344e00640 1 --2- 172.21.15.114:0/1383965530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834006da40 0x7f83400ccc20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f832c0097a0 tx=0x7f832c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.683+0000 7f8344e00640 1 -- 172.21.15.114:0/1383965530 shutdown_connections 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.683+0000 7f8344e00640 1 --2- 172.21.15.114:0/1383965530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f834006da40 0x7f83400ccc20 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.683+0000 7f8344e00640 1 -- 172.21.15.114:0/1383965530 >> 172.21.15.114:0/1383965530 conn(0x7f834006afc0 msgr2=0x7f834006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.683+0000 7f8344e00640 1 -- 172.21.15.114:0/1383965530 shutdown_connections 2024-03-06T07:29:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.683+0000 7f8344e00640 1 -- 172.21.15.114:0/1383965530 wait complete. 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.684+0000 7f8344e00640 1 Processor -- start 2024-03-06T07:29:46.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.685+0000 7f8344e00640 1 -- start start 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.685+0000 7f8344e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f834006da40 0x7f8340142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.685+0000 7f8344e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83400aef20 0x7f83400b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.685+0000 7f8344e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f834015be10 0x7f834015a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f8344e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83400cda90 con 0x7f83400aef20 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f8344e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834006d080 con 0x7f834006da40 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f8344e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83400ce2b0 con 0x7f834015be10 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83400aef20 0x7f83400b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83400aef20 0x7f83400b4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45960/0 (socket says 172.21.15.114:45960) 2024-03-06T07:29:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 -- 172.21.15.114:0/1961152235 learned_addr learned my addr 172.21.15.114:0/1961152235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833f400640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f834006da40 0x7f8340142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 -- 172.21.15.114:0/1961152235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f834015be10 msgr2=0x7f834015a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f834015be10 0x7f834015a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 -- 172.21.15.114:0/1961152235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f834006da40 msgr2=0x7f8340142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f834006da40 0x7f8340142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 -- 172.21.15.114:0/1961152235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83401561f0 con 0x7f83400aef20 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.686+0000 7f833ea00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83400aef20 0x7f83400b4100 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f832c0098c0 tx=0x7f832c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.687+0000 7f8337400640 1 -- 172.21.15.114:0/1961152235 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c032630 con 0x7f83400aef20 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.687+0000 7f8337400640 1 -- 172.21.15.114:0/1961152235 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c0327d0 con 0x7f83400aef20 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.687+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83401564d0 con 0x7f83400aef20 2024-03-06T07:29:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.687+0000 7f8337400640 1 -- 172.21.15.114:0/1961152235 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c032bc0 con 0x7f83400aef20 2024-03-06T07:29:46.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.687+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8340156a90 con 0x7f83400aef20 2024-03-06T07:29:46.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.688+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834004f330 con 0x7f83400aef20 2024-03-06T07:29:46.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.689+0000 7f8337400640 1 -- 172.21.15.114:0/1961152235 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f832c04d3f0 con 0x7f83400aef20 2024-03-06T07:29:46.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.689+0000 7f8337400640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.690+0000 7f8337400640 1 -- 172.21.15.114:0/1961152235 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f832c02f030 con 0x7f83400aef20 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.690+0000 7f833f400640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.690+0000 7f833f400640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f832006b500 0x7f832006d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8330000f60 tx=0x7f8330009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.694+0000 7f8337400640 1 -- 172.21.15.114:0/1961152235 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f832c040070 con 0x7f83400aef20 2024-03-06T07:29:46.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.887+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8340155750 con 0x7f83400aef20 2024-03-06T07:29:46.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.888+0000 7f8337400640 1 -- 172.21.15.114:0/1961152235 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f832c07a3e0 con 0x7f83400aef20 2024-03-06T07:29:46.891 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:29:46.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.890+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f832006b500 msgr2=0x7f832006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.890+0000 7f8344e00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f832006b500 0x7f832006d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8330000f60 tx=0x7f8330009040 comp rx=0 tx=0).stop 2024-03-06T07:29:46.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.890+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83400aef20 msgr2=0x7f83400b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.890+0000 7f8344e00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83400aef20 0x7f83400b4100 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f832c0098c0 tx=0x7f832c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 shutdown_connections 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f834015be10 0x7f834015a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f832006b500 0x7f832006d9c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83400aef20 0x7f83400b4100 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 --2- 172.21.15.114:0/1961152235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f834006da40 0x7f8340142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 >> 172.21.15.114:0/1961152235 conn(0x7f834006afc0 msgr2=0x7f83400cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 shutdown_connections 2024-03-06T07:29:46.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:46.891+0000 7f8344e00640 1 -- 172.21.15.114:0/1961152235 wait complete. 2024-03-06T07:29:46.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:29:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.120+0000 7f5c52c00640 1 Processor -- start 2024-03-06T07:29:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.121+0000 7f5c52c00640 1 -- start start 2024-03-06T07:29:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.122+0000 7f5c52c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5c4c15acb0 0x7f5c4c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.122+0000 7f5c52c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c06ec00 0x7f5c4c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.122+0000 7f5c52c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4c15b090 con 0x7f5c4c0b4680 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.122+0000 7f5c52c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4c0b2a30 con 0x7f5c4c15acb0 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.122+0000 7f5c52c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c0b4360 con 0x7f5c4c06ec00 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c4be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5c4c15acb0 0x7f5c4c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c4be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5c4c15acb0 0x7f5c4c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45964/0 (socket says 172.21.15.114:45964) 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c4be00640 1 -- 172.21.15.114:0/1352887311 learned_addr learned my addr 172.21.15.114:0/1352887311 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c50c00640 1 --2- 172.21.15.114:0/1352887311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c06ec00 0x7f5c4c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c4be00640 1 -- 172.21.15.114:0/1352887311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c06ec00 msgr2=0x7f5c4c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c4be00640 1 --2- 172.21.15.114:0/1352887311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c06ec00 0x7f5c4c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c4be00640 1 -- 172.21.15.114:0/1352887311 >> v1:172.21.15.114:6789/0 conn(0x7f5c4c0b4680 legacy=0x7f5c4c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.123+0000 7f5c4be00640 1 -- 172.21.15.114:0/1352887311 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c0b2cf0 con 0x7f5c4c15acb0 2024-03-06T07:29:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.124+0000 7f5c4be00640 1 --2- 172.21.15.114:0/1352887311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c15acb0 0x7f5c4c0b37d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009870 tx=0x7f5c3c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd730f2f2444765 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.124+0000 7f5c4b400640 1 -- 172.21.15.114:0/1352887311 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c034070 con 0x7f5c4c15acb0 2024-03-06T07:29:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.124+0000 7f5c4b400640 1 -- 172.21.15.114:0/1352887311 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c030840 con 0x7f5c4c15acb0 2024-03-06T07:29:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.124+0000 7f5c4b400640 1 -- 172.21.15.114:0/1352887311 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c030c10 con 0x7f5c4c15acb0 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 -- 172.21.15.114:0/1352887311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c15acb0 msgr2=0x7f5c4c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 --2- 172.21.15.114:0/1352887311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c15acb0 0x7f5c4c0b37d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009870 tx=0x7f5c3c02b980 comp rx=0 tx=0).stop 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 -- 172.21.15.114:0/1352887311 shutdown_connections 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 --2- 172.21.15.114:0/1352887311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c06ec00 0x7f5c4c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 --2- 172.21.15.114:0/1352887311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c15acb0 0x7f5c4c0b37d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 -- 172.21.15.114:0/1352887311 >> 172.21.15.114:0/1352887311 conn(0x7f5c4c0b1730 msgr2=0x7f5c4c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 -- 172.21.15.114:0/1352887311 shutdown_connections 2024-03-06T07:29:47.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.125+0000 7f5c52c00640 1 -- 172.21.15.114:0/1352887311 wait complete. 2024-03-06T07:29:47.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.127+0000 7f5c52c00640 1 Processor -- start 2024-03-06T07:29:47.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.127+0000 7f5c52c00640 1 -- start start 2024-03-06T07:29:47.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c52c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c06ec00 0x7f5c4c0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c52c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c0b4680 0x7f5c4c149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c52c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4c15acb0 0x7f5c4c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c52c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c0b3d10 con 0x7f5c4c06ec00 2024-03-06T07:29:47.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c52c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c0b32b0 con 0x7f5c4c15acb0 2024-03-06T07:29:47.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c52c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c0b4010 con 0x7f5c4c0b4680 2024-03-06T07:29:47.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c4be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c06ec00 0x7f5c4c0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c4be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c06ec00 0x7f5c4c0c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45966/0 (socket says 172.21.15.114:45966) 2024-03-06T07:29:47.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.128+0000 7f5c4be00640 1 -- 172.21.15.114:0/451118843 learned_addr learned my addr 172.21.15.114:0/451118843 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:47.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c43400640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c0b4680 0x7f5c4c149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c50c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4c15acb0 0x7f5c4c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c4be00640 1 -- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c0b4680 msgr2=0x7f5c4c149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c4be00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c0b4680 0x7f5c4c149670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c4be00640 1 -- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4c15acb0 msgr2=0x7f5c4c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c4be00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4c15acb0 0x7f5c4c152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c4be00640 1 -- 172.21.15.114:0/451118843 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c0c9a90 con 0x7f5c4c06ec00 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c4be00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c06ec00 0x7f5c4c0c9260 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c02b9c0 tx=0x7f5c3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c48c00640 1 -- 172.21.15.114:0/451118843 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c034040 con 0x7f5c4c06ec00 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c48c00640 1 -- 172.21.15.114:0/451118843 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c039d90 con 0x7f5c4c06ec00 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c0c9d70 con 0x7f5c4c06ec00 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.129+0000 7f5c48c00640 1 -- 172.21.15.114:0/451118843 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c043610 con 0x7f5c4c06ec00 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.130+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c0ca330 con 0x7f5c4c06ec00 2024-03-06T07:29:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.131+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4c04f330 con 0x7f5c4c06ec00 2024-03-06T07:29:47.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.132+0000 7f5c48c00640 1 -- 172.21.15.114:0/451118843 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c3c04a060 con 0x7f5c4c06ec00 2024-03-06T07:29:47.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.132+0000 7f5c48c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5c2806b4d0 0x7f5c2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.133+0000 7f5c48c00640 1 -- 172.21.15.114:0/451118843 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5c3c0abfc0 con 0x7f5c4c06ec00 2024-03-06T07:29:47.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.133+0000 7f5c43400640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5c2806b4d0 0x7f5c2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.133+0000 7f5c43400640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5c2806b4d0 0x7f5c2806d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c15d290 tx=0x7f5c34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.137+0000 7f5c48c00640 1 -- 172.21.15.114:0/451118843 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c3c03e070 con 0x7f5c4c06ec00 2024-03-06T07:29:47.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.334+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5c4c06ce60 con 0x7f5c4c06ec00 2024-03-06T07:29:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.334+0000 7f5c48c00640 1 -- 172.21.15.114:0/451118843 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v40) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5c3c037e40 con 0x7f5c4c06ec00 2024-03-06T07:29:47.337 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:47.337 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:29:47.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:29:47.337 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:29:47.337 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:29:47.337 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:29:47.338 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5c2806b4d0 msgr2=0x7f5c2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5c2806b4d0 0x7f5c2806d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c15d290 tx=0x7f5c34009040 comp rx=0 tx=0).stop 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c06ec00 msgr2=0x7f5c4c0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c06ec00 0x7f5c4c0c9260 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c02b9c0 tx=0x7f5c3c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 shutdown_connections 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4c15acb0 0x7f5c4c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5c4c0b4680 0x7f5c4c149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5c2806b4d0 0x7f5c2806d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 --2- 172.21.15.114:0/451118843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c4c06ec00 0x7f5c4c0c9260 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 >> 172.21.15.114:0/451118843 conn(0x7f5c4c0b1730 msgr2=0x7f5c4c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 shutdown_connections 2024-03-06T07:29:47.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.337+0000 7f5c52c00640 1 -- 172.21.15.114:0/451118843 wait complete. 2024-03-06T07:29:47.358 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:29:47.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:29:47.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.569+0000 7f0ced400640 1 Processor -- start 2024-03-06T07:29:47.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.569+0000 7f0ced400640 1 -- start start 2024-03-06T07:29:47.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.571+0000 7f0ced400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0ce806e570 0x7f0ce80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.571+0000 7f0ced400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce80b4990 0x7f0ce8161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.571+0000 7f0ced400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce80adf70 con 0x7f0ce815e170 2024-03-06T07:29:47.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.571+0000 7f0ced400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce806ea70 con 0x7f0ce806e570 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.571+0000 7f0ced400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b3690 con 0x7f0ce80b4990 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.571+0000 7f0ce5e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0ce806e570 0x7f0ce80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.571+0000 7f0ce7200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce80b4990 0x7f0ce8161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce6800640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f0ce815e170 0x7f0ce80ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:55208/0 (socket says 172.21.15.114:55208) 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce5e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0ce806e570 0x7f0ce80b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:58010/0 (socket says 172.21.15.114:58010) 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce6800640 1 -- 172.21.15.114:0/1451734841 learned_addr learned my addr 172.21.15.114:0/1451734841 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:47.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce7200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce80b4990 0x7f0ce8161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:37298/0 (socket says 172.21.15.114:37298) 2024-03-06T07:29:47.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce5e00640 1 -- 172.21.15.114:0/1451734841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce80b4990 msgr2=0x7f0ce8161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce5e00640 1 --2- 172.21.15.114:0/1451734841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce80b4990 0x7f0ce8161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce5e00640 1 -- 172.21.15.114:0/1451734841 >> v1:172.21.15.177:6789/0 conn(0x7f0ce815e170 legacy=0x7f0ce80ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.572+0000 7f0ce5400640 1 -- 172.21.15.114:0/1451734841 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209744058 0 0) 0x7f0ce80adf70 con 0x7f0ce815e170 2024-03-06T07:29:47.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.573+0000 7f0ce5e00640 1 -- 172.21.15.114:0/1451734841 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80b3970 con 0x7f0ce806e570 2024-03-06T07:29:47.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.574+0000 7f0ce5e00640 1 --2- 172.21.15.114:0/1451734841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 0x7f0ce80b4450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00097a0 tx=0x7f0cd002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5758e113bae0b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.575+0000 7f0ce5400640 1 -- 172.21.15.114:0/1451734841 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00315a0 con 0x7f0ce806e570 2024-03-06T07:29:47.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.575+0000 7f0ce5400640 1 -- 172.21.15.114:0/1451734841 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd0037070 con 0x7f0ce806e570 2024-03-06T07:29:47.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.575+0000 7f0ce5400640 1 -- 172.21.15.114:0/1451734841 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0031960 con 0x7f0ce806e570 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 -- 172.21.15.114:0/1451734841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 msgr2=0x7f0ce80b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 --2- 172.21.15.114:0/1451734841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 0x7f0ce80b4450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00097a0 tx=0x7f0cd002b840 comp rx=0 tx=0).stop 2024-03-06T07:29:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 -- 172.21.15.114:0/1451734841 shutdown_connections 2024-03-06T07:29:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 --2- 172.21.15.114:0/1451734841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce80b4990 0x7f0ce8161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 --2- 172.21.15.114:0/1451734841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 0x7f0ce80b4450 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 -- 172.21.15.114:0/1451734841 >> 172.21.15.114:0/1451734841 conn(0x7f0ce80cfa80 msgr2=0x7f0ce80cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 -- 172.21.15.114:0/1451734841 shutdown_connections 2024-03-06T07:29:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.576+0000 7f0ced400640 1 -- 172.21.15.114:0/1451734841 wait complete. 2024-03-06T07:29:47.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.578+0000 7f0ced400640 1 Processor -- start 2024-03-06T07:29:47.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.578+0000 7f0ced400640 1 -- start start 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.579+0000 7f0ced400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 0x7f0ce80ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.579+0000 7f0ced400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce80b4990 0x7f0ce80b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.579+0000 7f0ced400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce815e170 0x7f0ce80b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.579+0000 7f0ced400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b4dd0 con 0x7f0ce80b4990 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.579+0000 7f0ced400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b3f30 con 0x7f0ce806e570 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.579+0000 7f0ced400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce806df80 con 0x7f0ce815e170 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce80b4990 0x7f0ce80b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce80b4990 0x7f0ce80b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45974/0 (socket says 172.21.15.114:45974) 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 -- 172.21.15.114:0/1116594889 learned_addr learned my addr 172.21.15.114:0/1116594889 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce6800640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 0x7f0ce80ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce7200640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce815e170 0x7f0ce80b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 -- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce815e170 msgr2=0x7f0ce80b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce815e170 0x7f0ce80b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 -- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 msgr2=0x7f0ce80ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 0x7f0ce80ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce5e00640 1 -- 172.21.15.114:0/1116594889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80ae6a0 con 0x7f0ce80b4990 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.580+0000 7f0ce7200640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce815e170 0x7f0ce80b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.581+0000 7f0ce5e00640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce80b4990 0x7f0ce80b0af0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0034040 tx=0x7f0cd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.581+0000 7f0cdea00640 1 -- 172.21.15.114:0/1116594889 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0032630 con 0x7f0ce80b4990 2024-03-06T07:29:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.581+0000 7f0cdea00640 1 -- 172.21.15.114:0/1116594889 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd00327d0 con 0x7f0ce80b4990 2024-03-06T07:29:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.581+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce80af830 con 0x7f0ce80b4990 2024-03-06T07:29:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.581+0000 7f0cdea00640 1 -- 172.21.15.114:0/1116594889 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0032bc0 con 0x7f0ce80b4990 2024-03-06T07:29:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.581+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce81e7e90 con 0x7f0ce80b4990 2024-03-06T07:29:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.582+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce804f330 con 0x7f0ce80b4990 2024-03-06T07:29:47.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.583+0000 7f0cdea00640 1 -- 172.21.15.114:0/1116594889 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cd004e430 con 0x7f0ce80b4990 2024-03-06T07:29:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.584+0000 7f0cdea00640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cc406b4d0 0x7f0cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.584+0000 7f0cdea00640 1 -- 172.21.15.114:0/1116594889 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0cd002f030 con 0x7f0ce80b4990 2024-03-06T07:29:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.584+0000 7f0ce6800640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cc406b4d0 0x7f0cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.585+0000 7f0ce6800640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cc406b4d0 0x7f0cc406d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00012e0 tx=0x7f0ce0005eb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.589+0000 7f0cdea00640 1 -- 172.21.15.114:0/1116594889 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cd0040070 con 0x7f0ce80b4990 2024-03-06T07:29:47.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:47.783+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0ce80b2e30 con 0x7f0ce80b4990 2024-03-06T07:29:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.514+0000 7f0cdea00640 1 -- 172.21.15.114:0/1116594889 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0cd007ae90 con 0x7f0ce80b4990 2024-03-06T07:29:48.517 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:29:48.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.518+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cc406b4d0 msgr2=0x7f0cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.518+0000 7f0ced400640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cc406b4d0 0x7f0cc406d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00012e0 tx=0x7f0ce0005eb0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.518+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce80b4990 msgr2=0x7f0ce80b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.518+0000 7f0ced400640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce80b4990 0x7f0ce80b0af0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0034040 tx=0x7f0cd00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.518+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 shutdown_connections 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.519+0000 7f0ced400640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0ce815e170 0x7f0ce80b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.519+0000 7f0ced400640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cc406b4d0 0x7f0cc406d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.519+0000 7f0ced400640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce80b4990 0x7f0ce80b0af0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.519+0000 7f0ced400640 1 --2- 172.21.15.114:0/1116594889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ce806e570 0x7f0ce80ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.519+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 >> 172.21.15.114:0/1116594889 conn(0x7f0ce80cfa80 msgr2=0x7f0ce80cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.519+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 shutdown_connections 2024-03-06T07:29:48.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.519+0000 7f0ced400640 1 -- 172.21.15.114:0/1116594889 wait complete. 2024-03-06T07:29:48.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:29:48.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.741+0000 7fb732200640 1 Processor -- start 2024-03-06T07:29:48.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.741+0000 7fb732200640 1 -- start start 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb732200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb72c0b3650 0x7fb72c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb732200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0cda90 0x7fb72c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb732200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0719e0 con 0x7fb72c0b3df0 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb732200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c072310 con 0x7fb72c0b3650 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb732200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c071cc0 con 0x7fb72c0cda90 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb72aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb72c0b3650 0x7fb72c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb72aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb72c0b3650 0x7fb72c0727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48850/0 (socket says 172.21.15.114:48850) 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb72aa00640 1 -- 172.21.15.114:0/975864168 learned_addr learned my addr 172.21.15.114:0/975864168 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.743+0000 7fb72b400640 1 --1- 172.21.15.114:0/975864168 >> v1:172.21.15.114:6789/0 conn(0x7fb72c0b3df0 0x7fb72c0b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34454/0 (socket says 172.21.15.114:34454) 2024-03-06T07:29:48.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.744+0000 7fb72be00640 1 --2- 172.21.15.114:0/975864168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0cda90 0x7fb72c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.744+0000 7fb72aa00640 1 -- 172.21.15.114:0/975864168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0cda90 msgr2=0x7fb72c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.744+0000 7fb72aa00640 1 --2- 172.21.15.114:0/975864168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0cda90 0x7fb72c06c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.744+0000 7fb72aa00640 1 -- 172.21.15.114:0/975864168 >> v1:172.21.15.114:6789/0 conn(0x7fb72c0b3df0 legacy=0x7fb72c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.744+0000 7fb72aa00640 1 -- 172.21.15.114:0/975864168 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c071fa0 con 0x7fb72c0b3650 2024-03-06T07:29:48.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.745+0000 7fb72aa00640 1 --2- 172.21.15.114:0/975864168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3650 0x7fb72c0727a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb718009870 tx=0x7fb71802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bcfa019a90cd06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.745+0000 7fb72a000640 1 -- 172.21.15.114:0/975864168 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718034070 con 0x7fb72c0b3650 2024-03-06T07:29:48.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.745+0000 7fb72a000640 1 -- 172.21.15.114:0/975864168 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb718030840 con 0x7fb72c0b3650 2024-03-06T07:29:48.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.745+0000 7fb72a000640 1 -- 172.21.15.114:0/975864168 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718030c10 con 0x7fb72c0b3650 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 -- 172.21.15.114:0/975864168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3650 msgr2=0x7fb72c0727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 --2- 172.21.15.114:0/975864168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3650 0x7fb72c0727a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb718009870 tx=0x7fb71802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 -- 172.21.15.114:0/975864168 shutdown_connections 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 --2- 172.21.15.114:0/975864168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0cda90 0x7fb72c06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 --2- 172.21.15.114:0/975864168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3650 0x7fb72c0727a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 -- 172.21.15.114:0/975864168 >> 172.21.15.114:0/975864168 conn(0x7fb72c0b1940 msgr2=0x7fb72c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 -- 172.21.15.114:0/975864168 shutdown_connections 2024-03-06T07:29:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.746+0000 7fb732200640 1 -- 172.21.15.114:0/975864168 wait complete. 2024-03-06T07:29:48.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.748+0000 7fb732200640 1 Processor -- start 2024-03-06T07:29:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.748+0000 7fb732200640 1 -- start start 2024-03-06T07:29:48.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb732200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0b3650 0x7fb72c0cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb732200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3df0 0x7fb72c1f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb732200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c0cda90 0x7fb72c06cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb732200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c15b3c0 con 0x7fb72c0b3df0 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb732200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c15b240 con 0x7fb72c0cda90 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb732200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c0cf510 con 0x7fb72c0b3650 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb72aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3df0 0x7fb72c1f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb72aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3df0 0x7fb72c1f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48864/0 (socket says 172.21.15.114:48864) 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.749+0000 7fb72aa00640 1 -- 172.21.15.114:0/648972616 learned_addr learned my addr 172.21.15.114:0/648972616 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb72aa00640 1 -- 172.21.15.114:0/648972616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0b3650 msgr2=0x7fb72c0cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb72aa00640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0b3650 0x7fb72c0cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb72be00640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c0cda90 0x7fb72c06cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb72aa00640 1 -- 172.21.15.114:0/648972616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c0cda90 msgr2=0x7fb72c06cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb72aa00640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c0cda90 0x7fb72c06cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb72aa00640 1 -- 172.21.15.114:0/648972616 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c155a90 con 0x7fb72c0b3df0 2024-03-06T07:29:48.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb72aa00640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3df0 0x7fb72c1f4760 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb718000c00 tx=0x7fb7180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb723400640 1 -- 172.21.15.114:0/648972616 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718034040 con 0x7fb72c0b3df0 2024-03-06T07:29:48.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb723400640 1 -- 172.21.15.114:0/648972616 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb718039d90 con 0x7fb72c0b3df0 2024-03-06T07:29:48.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.750+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c155d70 con 0x7fb72c0b3df0 2024-03-06T07:29:48.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.751+0000 7fb723400640 1 -- 172.21.15.114:0/648972616 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718043630 con 0x7fb72c0b3df0 2024-03-06T07:29:48.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.751+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c156330 con 0x7fb72c0b3df0 2024-03-06T07:29:48.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.752+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c04f330 con 0x7fb72c0b3df0 2024-03-06T07:29:48.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.753+0000 7fb723400640 1 -- 172.21.15.114:0/648972616 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb71804a060 con 0x7fb72c0b3df0 2024-03-06T07:29:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.754+0000 7fb723400640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb70c06b4d0 0x7fb70c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.754+0000 7fb723400640 1 -- 172.21.15.114:0/648972616 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb7180ac5f0 con 0x7fb72c0b3df0 2024-03-06T07:29:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.754+0000 7fb72b400640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb70c06b4d0 0x7fb70c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.755+0000 7fb72b400640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb70c06b4d0 0x7fb70c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb71c000f60 tx=0x7fb71c007a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.758+0000 7fb723400640 1 -- 172.21.15.114:0/648972616 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb71803e070 con 0x7fb72c0b3df0 2024-03-06T07:29:48.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:48.960+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb72c0b4550 con 0x7fb72c0b3df0 2024-03-06T07:29:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.529+0000 7fb723400640 1 -- 172.21.15.114:0/648972616 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v42) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb71804a350 con 0x7fb72c0b3df0 2024-03-06T07:29:49.532 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb70c06b4d0 msgr2=0x7fb70c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb70c06b4d0 0x7fb70c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb71c000f60 tx=0x7fb71c007a00 comp rx=0 tx=0).stop 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3df0 msgr2=0x7fb72c1f4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3df0 0x7fb72c1f4760 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb718000c00 tx=0x7fb7180046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 shutdown_connections 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb72c0cda90 0x7fb72c06cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb70c06b4d0 0x7fb70c06d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb72c0b3df0 0x7fb72c1f4760 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 --2- 172.21.15.114:0/648972616 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb72c0b3650 0x7fb72c0cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.532+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 >> 172.21.15.114:0/648972616 conn(0x7fb72c0b1940 msgr2=0x7fb72c0b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.533+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 shutdown_connections 2024-03-06T07:29:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.533+0000 7fb732200640 1 -- 172.21.15.114:0/648972616 wait complete. 2024-03-06T07:29:49.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:29:49.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.750+0000 7fcc83a00640 1 Processor -- start 2024-03-06T07:29:49.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.750+0000 7fcc83a00640 1 -- start start 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc83a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc7c15fc10 0x7fcc7c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc83a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc83a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0b46c0 con 0x7fcc7c15fff0 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc83a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c0b49a0 con 0x7fcc7c15fc10 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc83a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c072c90 con 0x7fcc7c0b3f70 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc81000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc7c15fc10 0x7fcc7c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc7be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fcc7c15fff0 0x7fcc7c160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34466/0 (socket says 172.21.15.114:34466) 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc81000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc7c15fc10 0x7fcc7c070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48880/0 (socket says 172.21.15.114:48880) 2024-03-06T07:29:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc7be00640 1 -- 172.21.15.114:0/1209231439 learned_addr learned my addr 172.21.15.114:0/1209231439 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:49.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc81a00640 1 --2- 172.21.15.114:0/1209231439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc81000640 1 -- 172.21.15.114:0/1209231439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 msgr2=0x7fcc7c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:49.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc81000640 1 --2- 172.21.15.114:0/1209231439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc81000640 1 -- 172.21.15.114:0/1209231439 >> v1:172.21.15.114:6789/0 conn(0x7fcc7c15fff0 legacy=0x7fcc7c160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.752+0000 7fcc81000640 1 -- 172.21.15.114:0/1209231439 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c09dd80 con 0x7fcc7c15fc10 2024-03-06T07:29:49.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.753+0000 7fcc81000640 1 --2- 172.21.15.114:0/1209231439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c070ff0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcc700097a0 tx=0x7fcc7002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0f31b956d7a2ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.753+0000 7fcc7b400640 1 -- 172.21.15.114:0/1209231439 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70035020 con 0x7fcc7c15fc10 2024-03-06T07:29:49.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc7b400640 1 -- 172.21.15.114:0/1209231439 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc70031810 con 0x7fcc7c15fc10 2024-03-06T07:29:49.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc7b400640 1 -- 172.21.15.114:0/1209231439 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70031be0 con 0x7fcc7c15fc10 2024-03-06T07:29:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 -- 172.21.15.114:0/1209231439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fc10 msgr2=0x7fcc7c070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1209231439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c070ff0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcc700097a0 tx=0x7fcc7002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 -- 172.21.15.114:0/1209231439 shutdown_connections 2024-03-06T07:29:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1209231439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1209231439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c070ff0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 -- 172.21.15.114:0/1209231439 >> 172.21.15.114:0/1209231439 conn(0x7fcc7c0b1940 msgr2=0x7fcc7c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 -- 172.21.15.114:0/1209231439 shutdown_connections 2024-03-06T07:29:49.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.754+0000 7fcc83a00640 1 -- 172.21.15.114:0/1209231439 wait complete. 2024-03-06T07:29:49.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.756+0000 7fcc83a00640 1 Processor -- start 2024-03-06T07:29:49.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.756+0000 7fcc83a00640 1 -- start start 2024-03-06T07:29:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc83a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc83a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c14d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc83a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fff0 0x7fcc7c09bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc83a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c15c460 con 0x7fcc7c15fff0 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc83a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c15c2e0 con 0x7fcc7c15fc10 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc83a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0ce9e0 con 0x7fcc7c0b3f70 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc81a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fff0 0x7fcc7c09bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc81a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fff0 0x7fcc7c09bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48896/0 (socket says 172.21.15.114:48896) 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.757+0000 7fcc81a00640 1 -- 172.21.15.114:0/1733325757 learned_addr learned my addr 172.21.15.114:0/1733325757 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81000640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc7be00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c14d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81a00640 1 -- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 msgr2=0x7fcc7c14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c14c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81a00640 1 -- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc7c15fc10 msgr2=0x7fcc7c14d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c14d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81a00640 1 -- 172.21.15.114:0/1733325757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c14e340 con 0x7fcc7c15fff0 2024-03-06T07:29:49.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81000640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc7be00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc81a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fff0 0x7fcc7c09bf10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcc74007e60 tx=0x7fcc7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.758+0000 7fcc78c00640 1 -- 172.21.15.114:0/1733325757 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74014620 con 0x7fcc7c15fff0 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.759+0000 7fcc78c00640 1 -- 172.21.15.114:0/1733325757 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc74019070 con 0x7fcc7c15fff0 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.759+0000 7fcc78c00640 1 -- 172.21.15.114:0/1733325757 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74014a00 con 0x7fcc7c15fff0 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.759+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c15dd00 con 0x7fcc7c15fff0 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.759+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c15e2c0 con 0x7fcc7c15fff0 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.760+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c15daf0 con 0x7fcc7c15fff0 2024-03-06T07:29:49.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.761+0000 7fcc78c00640 1 -- 172.21.15.114:0/1733325757 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc74014ba0 con 0x7fcc7c15fff0 2024-03-06T07:29:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.762+0000 7fcc78c00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc6006b4d0 0x7fcc6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.762+0000 7fcc78c00640 1 -- 172.21.15.114:0/1733325757 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fcc7408ab60 con 0x7fcc7c15fff0 2024-03-06T07:29:49.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.762+0000 7fcc81000640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc6006b4d0 0x7fcc6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.763+0000 7fcc81000640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc6006b4d0 0x7fcc6006d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcc70036000 tx=0x7fcc70008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.767+0000 7fcc78c00640 1 -- 172.21.15.114:0/1733325757 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc740599a0 con 0x7fcc7c15fff0 2024-03-06T07:29:49.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:49.992+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcc7c04f330 con 0x7fcc7c15fff0 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.548+0000 7fcc78c00640 1 -- 172.21.15.114:0/1733325757 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7fcc740592a0 con 0x7fcc7c15fff0 2024-03-06T07:29:50.550 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.555+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc6006b4d0 msgr2=0x7fcc6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.555+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc6006b4d0 0x7fcc6006d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcc70036000 tx=0x7fcc70008030 comp rx=0 tx=0).stop 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.555+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fff0 msgr2=0x7fcc7c09bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.555+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fff0 0x7fcc7c09bf10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcc74007e60 tx=0x7fcc7400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.556+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 shutdown_connections 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.556+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc6006b4d0 0x7fcc6006d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.556+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc7c15fff0 0x7fcc7c09bf10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.556+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc7c15fc10 0x7fcc7c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.556+0000 7fcc83a00640 1 --2- 172.21.15.114:0/1733325757 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc7c0b3f70 0x7fcc7c14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.556+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 >> 172.21.15.114:0/1733325757 conn(0x7fcc7c0b1940 msgr2=0x7fcc7c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.556+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 shutdown_connections 2024-03-06T07:29:50.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.557+0000 7fcc83a00640 1 -- 172.21.15.114:0/1733325757 wait complete. 2024-03-06T07:29:50.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:50.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.765+0000 7ff4e5400640 1 Processor -- start 2024-03-06T07:29:50.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.765+0000 7ff4e5400640 1 -- start start 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.766+0000 7ff4e5400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e015a940 0x7ff4e00b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.766+0000 7ff4e5400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff4e00cf370 0x7ff4e006d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.766+0000 7ff4e5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4e00cc2a0 0x7ff4e00cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.766+0000 7ff4e5400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e00b28e0 con 0x7ff4e00cc2a0 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.766+0000 7ff4e5400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e00b2b30 con 0x7ff4e00cf370 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.766+0000 7ff4e5400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e00b2e10 con 0x7ff4e015a940 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4e00cc2a0 0x7ff4e00cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4e00cc2a0 0x7ff4e00cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48908/0 (socket says 172.21.15.114:48908) 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 -- 172.21.15.114:0/1579503144 learned_addr learned my addr 172.21.15.114:0/1579503144 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4dfe00640 1 --2- 172.21.15.114:0/1579503144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e015a940 0x7ff4e00b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4df400640 1 --2- 172.21.15.114:0/1579503144 >> v2:172.21.15.177:3300/0 conn(0x7ff4e00cf370 0x7ff4e006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 -- 172.21.15.114:0/1579503144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e015a940 msgr2=0x7ff4e00b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 --2- 172.21.15.114:0/1579503144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e015a940 0x7ff4e00b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 -- 172.21.15.114:0/1579503144 >> v2:172.21.15.177:3300/0 conn(0x7ff4e00cf370 msgr2=0x7ff4e006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 --2- 172.21.15.114:0/1579503144 >> v2:172.21.15.177:3300/0 conn(0x7ff4e00cf370 0x7ff4e006d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4e4a00640 1 -- 172.21.15.114:0/1579503144 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e006ca50 con 0x7ff4e00cc2a0 2024-03-06T07:29:50.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.767+0000 7ff4df400640 1 --2- 172.21.15.114:0/1579503144 >> v2:172.21.15.177:3300/0 conn(0x7ff4e00cf370 0x7ff4e006d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.768+0000 7ff4e4a00640 1 --2- 172.21.15.114:0/1579503144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e00cc2a0 0x7ff4e00cc750 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff4e006eb60 tx=0x7ff4d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3afffef00fd712 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4dea00640 1 -- 172.21.15.114:0/1579503144 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d802ca70 con 0x7ff4e00cc2a0 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4dea00640 1 -- 172.21.15.114:0/1579503144 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d802cc10 con 0x7ff4e00cc2a0 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4dea00640 1 -- 172.21.15.114:0/1579503144 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d80343f0 con 0x7ff4e00cc2a0 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 -- 172.21.15.114:0/1579503144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e00cc2a0 msgr2=0x7ff4e00cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 --2- 172.21.15.114:0/1579503144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e00cc2a0 0x7ff4e00cc750 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff4e006eb60 tx=0x7ff4d802bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 -- 172.21.15.114:0/1579503144 shutdown_connections 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 --2- 172.21.15.114:0/1579503144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e00cc2a0 0x7ff4e00cc750 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 --2- 172.21.15.114:0/1579503144 >> v2:172.21.15.177:3300/0 conn(0x7ff4e00cf370 0x7ff4e006d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 --2- 172.21.15.114:0/1579503144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e015a940 0x7ff4e00b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 -- 172.21.15.114:0/1579503144 >> 172.21.15.114:0/1579503144 conn(0x7ff4e006afc0 msgr2=0x7ff4e006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 -- 172.21.15.114:0/1579503144 shutdown_connections 2024-03-06T07:29:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.769+0000 7ff4e5400640 1 -- 172.21.15.114:0/1579503144 wait complete. 2024-03-06T07:29:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.771+0000 7ff4e5400640 1 Processor -- start 2024-03-06T07:29:50.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.771+0000 7ff4e5400640 1 -- start start 2024-03-06T07:29:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e5400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e00cc2a0 0x7ff4e0144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4e00cf370 0x7ff4e0159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e015a940 0x7ff4e0146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e5400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e00b4b20 con 0x7ff4e015a940 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e5400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e00b30f0 con 0x7ff4e00cf370 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e5400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e006d790 con 0x7ff4e00cc2a0 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e015a940 0x7ff4e0146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e015a940 0x7ff4e0146460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48916/0 (socket says 172.21.15.114:48916) 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.772+0000 7ff4e4a00640 1 -- 172.21.15.114:0/2826329617 learned_addr learned my addr 172.21.15.114:0/2826329617 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4dfe00640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e00cc2a0 0x7ff4e0144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4df400640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4e00cf370 0x7ff4e0159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4e4a00640 1 -- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e00cc2a0 msgr2=0x7ff4e0144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4e4a00640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e00cc2a0 0x7ff4e0144c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4e4a00640 1 -- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4e00cf370 msgr2=0x7ff4e0159ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4e4a00640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4e00cf370 0x7ff4e0159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4e4a00640 1 -- 172.21.15.114:0/2826329617 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e01456a0 con 0x7ff4e015a940 2024-03-06T07:29:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4e4a00640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e015a940 0x7ff4e0146460 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8009f40 tx=0x7ff4d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4d7e00640 1 -- 172.21.15.114:0/2826329617 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8043020 con 0x7ff4e015a940 2024-03-06T07:29:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4d7e00640 1 -- 172.21.15.114:0/2826329617 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d803e450 con 0x7ff4e015a940 2024-03-06T07:29:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4d7e00640 1 -- 172.21.15.114:0/2826329617 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8009510 con 0x7ff4e015a940 2024-03-06T07:29:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.773+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e0145980 con 0x7ff4e015a940 2024-03-06T07:29:50.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.774+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e0145f40 con 0x7ff4e015a940 2024-03-06T07:29:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.775+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e004f330 con 0x7ff4e015a940 2024-03-06T07:29:50.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.775+0000 7ff4d7e00640 1 -- 172.21.15.114:0/2826329617 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4d8046020 con 0x7ff4e015a940 2024-03-06T07:29:50.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.777+0000 7ff4d7e00640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4c006b4d0 0x7ff4c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.777+0000 7ff4d7e00640 1 -- 172.21.15.114:0/2826329617 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4d80ac770 con 0x7ff4e015a940 2024-03-06T07:29:50.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.777+0000 7ff4dfe00640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4c006b4d0 0x7ff4c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.778+0000 7ff4dfe00640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4c006b4d0 0x7ff4c006d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0006fd0 tx=0x7ff4d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.782+0000 7ff4d7e00640 1 -- 172.21.15.114:0/2826329617 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4d804d070 con 0x7ff4e015a940 2024-03-06T07:29:50.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.978+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4e00b4790 con 0x7ff4e015a940 2024-03-06T07:29:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.980+0000 7ff4d7e00640 1 -- 172.21.15.114:0/2826329617 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16674 (secure 0 0 0) 0x7ff4d807b5e0 con 0x7ff4e015a940 2024-03-06T07:29:50.983 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:50.983 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":43,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:49.994347+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:47.835295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:49.012214+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.983+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4c006b4d0 msgr2=0x7ff4c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.983+0000 7ff4e5400640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4c006b4d0 0x7ff4c006d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0006fd0 tx=0x7ff4d0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.983+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e015a940 msgr2=0x7ff4e0146460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.983+0000 7ff4e5400640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e015a940 0x7ff4e0146460 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8009f40 tx=0x7ff4d8004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 shutdown_connections 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff4c006b4d0 0x7ff4c006d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4e015a940 0x7ff4e0146460 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4e00cf370 0x7ff4e0159ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 --2- 172.21.15.114:0/2826329617 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4e00cc2a0 0x7ff4e0144c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 >> 172.21.15.114:0/2826329617 conn(0x7ff4e006afc0 msgr2=0x7ff4e00cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 shutdown_connections 2024-03-06T07:29:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:50.984+0000 7ff4e5400640 1 -- 172.21.15.114:0/2826329617 wait complete. 2024-03-06T07:29:51.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:51.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.200+0000 7f3cabe00640 1 Processor -- start 2024-03-06T07:29:51.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.200+0000 7f3cabe00640 1 -- start start 2024-03-06T07:29:51.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.201+0000 7f3cabe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3ca415acb0 0x7f3ca40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.201+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca40b4680 0x7f3ca406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.201+0000 7f3cabe00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca415b090 con 0x7f3ca4072340 2024-03-06T07:29:51.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.201+0000 7f3cabe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca40b2a30 con 0x7f3ca415acb0 2024-03-06T07:29:51.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.201+0000 7f3cabe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca40b4360 con 0x7f3ca40b4680 2024-03-06T07:29:51.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.202+0000 7f3ca9400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3ca415acb0 0x7f3ca40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.202+0000 7f3ca8a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca40b4680 0x7f3ca406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.202+0000 7f3ca9400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3ca415acb0 0x7f3ca40b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:57600/0 (socket says 172.21.15.114:57600) 2024-03-06T07:29:51.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.202+0000 7f3ca9400640 1 -- 172.21.15.114:0/3076237133 learned_addr learned my addr 172.21.15.114:0/3076237133 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:51.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.202+0000 7f3ca8a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca40b4680 0x7f3ca406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:56820/0 (socket says 172.21.15.114:56820) 2024-03-06T07:29:51.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.202+0000 7f3ca9e00640 1 --1- 172.21.15.114:0/3076237133 >> v1:172.21.15.177:6789/0 conn(0x7f3ca4072340 0x7f3ca40ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:35194/0 (socket says 172.21.15.114:35194) 2024-03-06T07:29:51.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.203+0000 7f3ca9400640 1 -- 172.21.15.114:0/3076237133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca40b4680 msgr2=0x7f3ca406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.203+0000 7f3ca9400640 1 --2- 172.21.15.114:0/3076237133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca40b4680 0x7f3ca406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.203+0000 7f3ca9400640 1 -- 172.21.15.114:0/3076237133 >> v1:172.21.15.177:6789/0 conn(0x7f3ca4072340 legacy=0x7f3ca40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.203+0000 7f3ca9400640 1 -- 172.21.15.114:0/3076237133 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca40b2cf0 con 0x7f3ca415acb0 2024-03-06T07:29:51.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.204+0000 7f3ca9400640 1 --2- 172.21.15.114:0/3076237133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 0x7f3ca40b37d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009870 tx=0x7f3c9802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3e2e49cfab0921 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.205+0000 7f3ca3e00640 1 -- 172.21.15.114:0/3076237133 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98034070 con 0x7f3ca415acb0 2024-03-06T07:29:51.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.205+0000 7f3ca3e00640 1 -- 172.21.15.114:0/3076237133 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98030840 con 0x7f3ca415acb0 2024-03-06T07:29:51.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.205+0000 7f3ca3e00640 1 -- 172.21.15.114:0/3076237133 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98030c10 con 0x7f3ca415acb0 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 -- 172.21.15.114:0/3076237133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 msgr2=0x7f3ca40b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 --2- 172.21.15.114:0/3076237133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 0x7f3ca40b37d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009870 tx=0x7f3c9802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 -- 172.21.15.114:0/3076237133 shutdown_connections 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 --2- 172.21.15.114:0/3076237133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca40b4680 0x7f3ca406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 --2- 172.21.15.114:0/3076237133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 0x7f3ca40b37d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 -- 172.21.15.114:0/3076237133 >> 172.21.15.114:0/3076237133 conn(0x7f3ca40b1730 msgr2=0x7f3ca40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 -- 172.21.15.114:0/3076237133 shutdown_connections 2024-03-06T07:29:51.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.206+0000 7f3cabe00640 1 -- 172.21.15.114:0/3076237133 wait complete. 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.207+0000 7f3cabe00640 1 Processor -- start 2024-03-06T07:29:51.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.208+0000 7f3cabe00640 1 -- start start 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.208+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca4072340 0x7f3ca41f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.208+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca40b4680 0x7f3ca41f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.208+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 0x7f3ca4149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.208+0000 7f3cabe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca40b3d10 con 0x7f3ca40b4680 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.208+0000 7f3cabe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca40b32b0 con 0x7f3ca415acb0 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.208+0000 7f3cabe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca40b4010 con 0x7f3ca4072340 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca40b4680 0x7f3ca41f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca40b4680 0x7f3ca41f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48930/0 (socket says 172.21.15.114:48930) 2024-03-06T07:29:51.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 -- 172.21.15.114:0/1588178040 learned_addr learned my addr 172.21.15.114:0/1588178040 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca9e00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 0x7f3ca4149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 -- 172.21.15.114:0/1588178040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca4072340 msgr2=0x7f3ca41f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca4072340 0x7f3ca41f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 -- 172.21.15.114:0/1588178040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 msgr2=0x7f3ca4149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 0x7f3ca4149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 -- 172.21.15.114:0/1588178040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca4150ca0 con 0x7f3ca40b4680 2024-03-06T07:29:51.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.209+0000 7f3ca8a00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca40b4680 0x7f3ca41f6d80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400e9e0 tx=0x7f3c9400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.210+0000 7f3ca1600640 1 -- 172.21.15.114:0/1588178040 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94017070 con 0x7f3ca40b4680 2024-03-06T07:29:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.210+0000 7f3ca1600640 1 -- 172.21.15.114:0/1588178040 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9400c9d0 con 0x7f3ca40b4680 2024-03-06T07:29:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.210+0000 7f3ca1600640 1 -- 172.21.15.114:0/1588178040 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400cdc0 con 0x7f3ca40b4680 2024-03-06T07:29:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.210+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca41f72c0 con 0x7f3ca40b4680 2024-03-06T07:29:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.210+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca4150f40 con 0x7f3ca40b4680 2024-03-06T07:29:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.211+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca404f330 con 0x7f3ca40b4680 2024-03-06T07:29:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.212+0000 7f3ca1600640 1 -- 172.21.15.114:0/1588178040 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c9401a070 con 0x7f3ca40b4680 2024-03-06T07:29:51.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.213+0000 7f3ca1600640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3c8806b4d0 0x7f3c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.213+0000 7f3ca1600640 1 -- 172.21.15.114:0/1588178040 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c9408a2e0 con 0x7f3ca40b4680 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.213+0000 7f3ca9400640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3c8806b4d0 0x7f3c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.214+0000 7f3ca9400640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3c8806b4d0 0x7f3c8806d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c98036000 tx=0x7f3c9803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.217+0000 7f3ca1600640 1 -- 172.21.15.114:0/1588178040 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c940590c0 con 0x7f3ca40b4680 2024-03-06T07:29:51.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.403+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ca406ce60 con 0x7f3ca40b4680 2024-03-06T07:29:51.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.405+0000 7f3ca1600640 1 -- 172.21.15.114:0/1588178040 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16674 (secure 0 0 0) 0x7f3c940589c0 con 0x7f3ca40b4680 2024-03-06T07:29:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:51.407 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":43,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:49.994347+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:47.835295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:49.012214+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3c8806b4d0 msgr2=0x7f3c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3c8806b4d0 0x7f3c8806d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c98036000 tx=0x7f3c9803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca40b4680 msgr2=0x7f3ca41f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca40b4680 0x7f3ca41f6d80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400e9e0 tx=0x7f3c9400eea0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 shutdown_connections 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ca415acb0 0x7f3ca4149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3c8806b4d0 0x7f3c8806d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ca40b4680 0x7f3ca41f6d80 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 --2- 172.21.15.114:0/1588178040 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3ca4072340 0x7f3ca41f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 >> 172.21.15.114:0/1588178040 conn(0x7f3ca40b1730 msgr2=0x7f3ca415e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 shutdown_connections 2024-03-06T07:29:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.408+0000 7f3cabe00640 1 -- 172.21.15.114:0/1588178040 wait complete. 2024-03-06T07:29:51.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:29:51.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.631+0000 7f7091600640 1 Processor -- start 2024-03-06T07:29:51.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.631+0000 7f7091600640 1 -- start start 2024-03-06T07:29:51.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f7091600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f708c0b3df0 0x7f708c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f7091600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0cda90 0x7f708c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f7091600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708c0719e0 con 0x7f708c0b3650 2024-03-06T07:29:51.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f7091600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708c072310 con 0x7f708c0b3df0 2024-03-06T07:29:51.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f7091600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c071cc0 con 0x7f708c0cda90 2024-03-06T07:29:51.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f708aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f708c0b3df0 0x7f708c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f708aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f708c0b3df0 0x7f708c0727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48934/0 (socket says 172.21.15.114:48934) 2024-03-06T07:29:51.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.632+0000 7f708aa00640 1 -- 172.21.15.114:0/3742911851 learned_addr learned my addr 172.21.15.114:0/3742911851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:51.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.633+0000 7f708aa00640 1 -- 172.21.15.114:0/3742911851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0cda90 msgr2=0x7f708c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.633+0000 7f708aa00640 1 --2- 172.21.15.114:0/3742911851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0cda90 0x7f708c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.633+0000 7f708aa00640 1 -- 172.21.15.114:0/3742911851 >> v1:172.21.15.177:6789/0 conn(0x7f708c0b3650 legacy=0x7f708c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.633+0000 7f708aa00640 1 -- 172.21.15.114:0/3742911851 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c071fa0 con 0x7f708c0b3df0 2024-03-06T07:29:51.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.634+0000 7f708aa00640 1 --2- 172.21.15.114:0/3742911851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3df0 0x7f708c0727a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f707c00d150 tx=0x7f707c02e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f7d97d19f33f0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.634+0000 7f708a000640 1 -- 172.21.15.114:0/3742911851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c02ea70 con 0x7f708c0b3df0 2024-03-06T07:29:51.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.634+0000 7f708a000640 1 -- 172.21.15.114:0/3742911851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c02ec10 con 0x7f708c0b3df0 2024-03-06T07:29:51.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.634+0000 7f708a000640 1 -- 172.21.15.114:0/3742911851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c009600 con 0x7f708c0b3df0 2024-03-06T07:29:51.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 -- 172.21.15.114:0/3742911851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3df0 msgr2=0x7f708c0727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 --2- 172.21.15.114:0/3742911851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3df0 0x7f708c0727a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f707c00d150 tx=0x7f707c02e0d0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 -- 172.21.15.114:0/3742911851 shutdown_connections 2024-03-06T07:29:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 --2- 172.21.15.114:0/3742911851 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0cda90 0x7f708c06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 --2- 172.21.15.114:0/3742911851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3df0 0x7f708c0727a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 -- 172.21.15.114:0/3742911851 >> 172.21.15.114:0/3742911851 conn(0x7f708c0b1940 msgr2=0x7f708c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 -- 172.21.15.114:0/3742911851 shutdown_connections 2024-03-06T07:29:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.635+0000 7f7091600640 1 -- 172.21.15.114:0/3742911851 wait complete. 2024-03-06T07:29:51.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.637+0000 7f7091600640 1 Processor -- start 2024-03-06T07:29:51.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.637+0000 7f7091600640 1 -- start start 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f7091600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3650 0x7f708c155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f7091600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0b3df0 0x7f708c09d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f7091600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f708c0cda90 0x7f708c14e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f7091600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c15b3c0 con 0x7f708c0b3650 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f7091600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c15b240 con 0x7f708c0cda90 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f7091600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c0cf510 con 0x7f708c0b3df0 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3650 0x7f708c155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3650 0x7f708c155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48938/0 (socket says 172.21.15.114:48938) 2024-03-06T07:29:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 -- 172.21.15.114:0/1837488453 learned_addr learned my addr 172.21.15.114:0/1837488453 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 -- 172.21.15.114:0/1837488453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0b3df0 msgr2=0x7f708c09d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0b3df0 0x7f708c09d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 -- 172.21.15.114:0/1837488453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f708c0cda90 msgr2=0x7f708c14e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f708c0cda90 0x7f708c14e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.638+0000 7f708aa00640 1 -- 172.21.15.114:0/1837488453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c09ad20 con 0x7f708c0b3650 2024-03-06T07:29:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.639+0000 7f708aa00640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3650 0x7f708c155b20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f707c000f80 tx=0x7f707c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.639+0000 7f7083400640 1 -- 172.21.15.114:0/1837488453 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c00b720 con 0x7f708c0b3650 2024-03-06T07:29:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.639+0000 7f7083400640 1 -- 172.21.15.114:0/1837488453 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c045070 con 0x7f708c0b3650 2024-03-06T07:29:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.639+0000 7f7083400640 1 -- 172.21.15.114:0/1837488453 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c036a60 con 0x7f708c0b3650 2024-03-06T07:29:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.639+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c09b000 con 0x7f708c0b3650 2024-03-06T07:29:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.639+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c09b5c0 con 0x7f708c0b3650 2024-03-06T07:29:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.641+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c09b8f0 con 0x7f708c0b3650 2024-03-06T07:29:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.641+0000 7f7083400640 1 -- 172.21.15.114:0/1837488453 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f707c036c00 con 0x7f708c0b3650 2024-03-06T07:29:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.643+0000 7f7083400640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f706806b4d0 0x7f706806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.643+0000 7f7083400640 1 -- 172.21.15.114:0/1837488453 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f707c0ad240 con 0x7f708c0b3650 2024-03-06T07:29:51.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.643+0000 7f7082000640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f706806b4d0 0x7f706806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.644+0000 7f7082000640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f706806b4d0 0x7f706806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f708c150ab0 tx=0x7f7074006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.647+0000 7f7083400640 1 -- 172.21.15.114:0/1837488453 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f707c07c150 con 0x7f708c0b3650 2024-03-06T07:29:51.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:51.903+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f708c04f330 con 0x7f708c0b3650 2024-03-06T07:29:52.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.561+0000 7f7083400640 1 -- 172.21.15.114:0/1837488453 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f707c07ba50 con 0x7f708c0b3650 2024-03-06T07:29:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f706806b4d0 msgr2=0x7f706806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f706806b4d0 0x7f706806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f708c150ab0 tx=0x7f7074006ba0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3650 msgr2=0x7f708c155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3650 0x7f708c155b20 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f707c000f80 tx=0x7f707c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 shutdown_connections 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f708c0cda90 0x7f708c14e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f708c0b3df0 0x7f708c09d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f706806b4d0 0x7f706806d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 --2- 172.21.15.114:0/1837488453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c0b3650 0x7f708c155b20 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.565+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 >> 172.21.15.114:0/1837488453 conn(0x7f708c0b1940 msgr2=0x7f708c0ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.566+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 shutdown_connections 2024-03-06T07:29:52.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.566+0000 7f7091600640 1 -- 172.21.15.114:0/1837488453 wait complete. 2024-03-06T07:29:52.588 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:29:52.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:29:52.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.805+0000 7f37d0c00640 1 Processor -- start 2024-03-06T07:29:52.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.805+0000 7f37d0c00640 1 -- start start 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.807+0000 7f37d0c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f37cc15fc10 0x7f37cc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.807+0000 7f37d0c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.807+0000 7f37d0c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc0b46c0 con 0x7f37cc15fff0 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.807+0000 7f37d0c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc0b49a0 con 0x7f37cc15fc10 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.807+0000 7f37d0c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc072c90 con 0x7f37cc0b3f70 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.808+0000 7f37c9e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f37cc15fc10 0x7f37cc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.808+0000 7f37ca800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.808+0000 7f37ca800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc0ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:56866/0 (socket says 172.21.15.114:56866) 2024-03-06T07:29:52.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.808+0000 7f37c9e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f37cc15fc10 0x7f37cc070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:57630/0 (socket says 172.21.15.114:57630) 2024-03-06T07:29:52.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.808+0000 7f37ca800640 1 -- 172.21.15.114:0/3844737908 learned_addr learned my addr 172.21.15.114:0/3844737908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:52.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.808+0000 7f37ca800640 1 -- 172.21.15.114:0/3844737908 >> v2:172.21.15.177:3300/0 conn(0x7f37cc15fc10 msgr2=0x7f37cc070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.808+0000 7f37ca800640 1 --2- 172.21.15.114:0/3844737908 >> v2:172.21.15.177:3300/0 conn(0x7f37cc15fc10 0x7f37cc070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.809+0000 7f37ca800640 1 -- 172.21.15.114:0/3844737908 >> v1:172.21.15.177:6789/0 conn(0x7f37cc15fff0 legacy=0x7f37cc160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.809+0000 7f37ca800640 1 -- 172.21.15.114:0/3844737908 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc09dd80 con 0x7f37cc0b3f70 2024-03-06T07:29:52.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.810+0000 7f37ca800640 1 --2- 172.21.15.114:0/3844737908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc0ce4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009e50 tx=0x7f37c402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6276476efcf4b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.810+0000 7f37c9400640 1 -- 172.21.15.114:0/3844737908 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c402cb00 con 0x7f37cc0b3f70 2024-03-06T07:29:52.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.810+0000 7f37c9400640 1 -- 172.21.15.114:0/3844737908 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c402cca0 con 0x7f37cc0b3f70 2024-03-06T07:29:52.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.810+0000 7f37c9400640 1 -- 172.21.15.114:0/3844737908 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4034610 con 0x7f37cc0b3f70 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 -- 172.21.15.114:0/3844737908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 msgr2=0x7f37cc0ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3844737908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc0ce4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009e50 tx=0x7f37c402bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 -- 172.21.15.114:0/3844737908 shutdown_connections 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3844737908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc0ce4a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3844737908 >> v2:172.21.15.177:3300/0 conn(0x7f37cc15fc10 0x7f37cc070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 -- 172.21.15.114:0/3844737908 >> 172.21.15.114:0/3844737908 conn(0x7f37cc0b1940 msgr2=0x7f37cc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 -- 172.21.15.114:0/3844737908 shutdown_connections 2024-03-06T07:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.811+0000 7f37d0c00640 1 -- 172.21.15.114:0/3844737908 wait complete. 2024-03-06T07:29:52.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.813+0000 7f37d0c00640 1 Processor -- start 2024-03-06T07:29:52.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.813+0000 7f37d0c00640 1 -- start start 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37d0c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc09d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37d0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37cc15fc10 0x7f37cc15b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37d0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37cc15fff0 0x7f37cc1f7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37d0c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc15c460 con 0x7f37cc15fff0 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37d0c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc15c2e0 con 0x7f37cc15fc10 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37d0c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0ce9e0 con 0x7f37cc0b3f70 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37ca800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37cc15fff0 0x7f37cc1f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37ca800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37cc15fff0 0x7f37cc1f7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48944/0 (socket says 172.21.15.114:48944) 2024-03-06T07:29:52.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.814+0000 7f37ca800640 1 -- 172.21.15.114:0/3093580688 learned_addr learned my addr 172.21.15.114:0/3093580688 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:52.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37c9e00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc09d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37c1400640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37cc15fc10 0x7f37cc15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37ca800640 1 -- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 msgr2=0x7f37cc09d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37ca800640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc09d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37ca800640 1 -- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37cc15fc10 msgr2=0x7f37cc15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37ca800640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37cc15fc10 0x7f37cc15b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37ca800640 1 -- 172.21.15.114:0/3093580688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc15b9c0 con 0x7f37cc15fff0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37c1400640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37cc15fc10 0x7f37cc15b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37ca800640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37cc15fff0 0x7f37cc1f7630 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f37c4036040 tx=0x7f37c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37c2a00640 1 -- 172.21.15.114:0/3093580688 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4043020 con 0x7f37cc15fff0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37c2a00640 1 -- 172.21.15.114:0/3093580688 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c402ce50 con 0x7f37cc15fff0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37c2a00640 1 -- 172.21.15.114:0/3093580688 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c403e690 con 0x7f37cc15fff0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.815+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc09c590 con 0x7f37cc15fff0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.816+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc09cb50 con 0x7f37cc15fff0 2024-03-06T07:29:52.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.817+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc04f330 con 0x7f37cc15fff0 2024-03-06T07:29:52.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.818+0000 7f37c2a00640 1 -- 172.21.15.114:0/3093580688 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37c4046020 con 0x7f37cc15fff0 2024-03-06T07:29:52.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.819+0000 7f37c2a00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37a806b500 0x7f37a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.819+0000 7f37c2a00640 1 -- 172.21.15.114:0/3093580688 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37c40ac6b0 con 0x7f37cc15fff0 2024-03-06T07:29:52.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.819+0000 7f37c9e00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37a806b500 0x7f37a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.820+0000 7f37c9e00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37a806b500 0x7f37a806d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009a10 tx=0x7f37bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:52.823+0000 7f37c2a00640 1 -- 172.21.15.114:0/3093580688 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37c404d030 con 0x7f37cc15fff0 2024-03-06T07:29:53.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.061+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f37cc06ef30 con 0x7f37cc15fff0 2024-03-06T07:29:53.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.574+0000 7f37c2a00640 1 -- 172.21.15.114:0/3093580688 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f37cc15b9c0 con 0x7f37cc15fff0 2024-03-06T07:29:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37a806b500 msgr2=0x7f37a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37a806b500 0x7f37a806d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009a10 tx=0x7f37bc008040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37cc15fff0 msgr2=0x7f37cc1f7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37cc15fff0 0x7f37cc1f7630 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f37c4036040 tx=0x7f37c4004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 shutdown_connections 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37a806b500 0x7f37a806d9c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37cc15fff0 0x7f37cc1f7630 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37cc15fc10 0x7f37cc15b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 --2- 172.21.15.114:0/3093580688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37cc0b3f70 0x7f37cc09d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.578+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 >> 172.21.15.114:0/3093580688 conn(0x7f37cc0b1940 msgr2=0x7f37cc0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.579+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 shutdown_connections 2024-03-06T07:29:53.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.579+0000 7f37d0c00640 1 -- 172.21.15.114:0/3093580688 wait complete. 2024-03-06T07:29:53.602 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:29:53.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:53.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.809+0000 7fc9ed800640 1 Processor -- start 2024-03-06T07:29:53.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.809+0000 7fc9ed800640 1 -- start start 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9ed800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9e80d2360 0x7fc9e80d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9ed800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e8161960 con 0x7fc9e80b3ec0 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9ed800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e80cf6f0 con 0x7fc9e8161580 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9ed800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e80d2ce0 con 0x7fc9e80d2360 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9df400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9e80d2360 0x7fc9e80d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9ece00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc9e80b3ec0 0x7fc9e80b4310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34470/0 (socket says 172.21.15.114:34470) 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9df400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9e80d2360 0x7fc9e80d27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48948/0 (socket says 172.21.15.114:48948) 2024-03-06T07:29:53.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.810+0000 7fc9ece00640 1 -- 172.21.15.114:0/3972000208 learned_addr learned my addr 172.21.15.114:0/3972000208 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.811+0000 7fc9e7400640 1 -- 172.21.15.114:0/3972000208 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749378336 0 0) 0x7fc9e8161960 con 0x7fc9e80b3ec0 2024-03-06T07:29:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.811+0000 7fc9e7400640 1 -- 172.21.15.114:0/3972000208 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9cc003520 con 0x7fc9e80b3ec0 2024-03-06T07:29:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.811+0000 7fc9df400640 1 -- 172.21.15.114:0/3972000208 >> v1:172.21.15.177:6789/0 conn(0x7fc9e8161580 legacy=0x7fc9e80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.811+0000 7fc9df400640 1 -- 172.21.15.114:0/3972000208 >> v1:172.21.15.114:6789/0 conn(0x7fc9e80b3ec0 legacy=0x7fc9e80b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.811+0000 7fc9df400640 1 -- 172.21.15.114:0/3972000208 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80cf9d0 con 0x7fc9e80d2360 2024-03-06T07:29:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.813+0000 7fc9df400640 1 --2- 172.21.15.114:0/3972000208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e80d27a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00097a0 tx=0x7fc9d002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30906f077291a3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.813+0000 7fc9e7400640 1 -- 172.21.15.114:0/3972000208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0034070 con 0x7fc9e80d2360 2024-03-06T07:29:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.813+0000 7fc9e7400640 1 -- 172.21.15.114:0/3972000208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0030610 con 0x7fc9e80d2360 2024-03-06T07:29:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.813+0000 7fc9e7400640 1 -- 172.21.15.114:0/3972000208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00309c0 con 0x7fc9e80d2360 2024-03-06T07:29:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.813+0000 7fc9ed800640 1 -- 172.21.15.114:0/3972000208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 msgr2=0x7fc9e80d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.813+0000 7fc9ed800640 1 --2- 172.21.15.114:0/3972000208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e80d27a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00097a0 tx=0x7fc9d002b960 comp rx=0 tx=0).stop 2024-03-06T07:29:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.814+0000 7fc9ed800640 1 -- 172.21.15.114:0/3972000208 shutdown_connections 2024-03-06T07:29:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.814+0000 7fc9ed800640 1 --2- 172.21.15.114:0/3972000208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e80d27a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.814+0000 7fc9ed800640 1 -- 172.21.15.114:0/3972000208 >> 172.21.15.114:0/3972000208 conn(0x7fc9e80d3d80 msgr2=0x7fc9e80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.814+0000 7fc9ed800640 1 -- 172.21.15.114:0/3972000208 shutdown_connections 2024-03-06T07:29:53.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.814+0000 7fc9ed800640 1 -- 172.21.15.114:0/3972000208 wait complete. 2024-03-06T07:29:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.815+0000 7fc9ed800640 1 Processor -- start 2024-03-06T07:29:53.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.815+0000 7fc9ed800640 1 -- start start 2024-03-06T07:29:53.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.816+0000 7fc9ed800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9e80b3ec0 0x7fc9e80c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.816+0000 7fc9ed800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e814f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.816+0000 7fc9ed800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9e8161580 0x7fc9e80cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.816+0000 7fc9ed800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80b39a0 con 0x7fc9e80d2360 2024-03-06T07:29:53.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.816+0000 7fc9ed800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80b36c0 con 0x7fc9e8161580 2024-03-06T07:29:53.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.816+0000 7fc9ed800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80cf3a0 con 0x7fc9e80b3ec0 2024-03-06T07:29:53.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e814f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e814f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48950/0 (socket says 172.21.15.114:48950) 2024-03-06T07:29:53.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 -- 172.21.15.114:0/628022419 learned_addr learned my addr 172.21.15.114:0/628022419 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9ece00640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9e8161580 0x7fc9e80cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 -- 172.21.15.114:0/628022419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9e80b3ec0 msgr2=0x7fc9e80c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9e80b3ec0 0x7fc9e80c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 -- 172.21.15.114:0/628022419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9e8161580 msgr2=0x7fc9e80cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9e8161580 0x7fc9e80cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 -- 172.21.15.114:0/628022419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80cc5a0 con 0x7fc9e80d2360 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.817+0000 7fc9df400640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e814f260 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0000c00 tx=0x7fc9d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.818+0000 7fc9e4c00640 1 -- 172.21.15.114:0/628022419 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0034040 con 0x7fc9e80d2360 2024-03-06T07:29:53.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.818+0000 7fc9e4c00640 1 -- 172.21.15.114:0/628022419 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0030410 con 0x7fc9e80d2360 2024-03-06T07:29:53.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.818+0000 7fc9e4c00640 1 -- 172.21.15.114:0/628022419 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0004230 con 0x7fc9e80d2360 2024-03-06T07:29:53.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.818+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e80cc880 con 0x7fc9e80d2360 2024-03-06T07:29:53.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.818+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e80cce40 con 0x7fc9e80d2360 2024-03-06T07:29:53.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.819+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e80d2f50 con 0x7fc9e80d2360 2024-03-06T07:29:53.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.820+0000 7fc9e4c00640 1 -- 172.21.15.114:0/628022419 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9d004c060 con 0x7fc9e80d2360 2024-03-06T07:29:53.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.821+0000 7fc9e4c00640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9c406b4d0 0x7fc9c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.821+0000 7fc9e4c00640 1 -- 172.21.15.114:0/628022419 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9d00abd50 con 0x7fc9e80d2360 2024-03-06T07:29:53.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.821+0000 7fc9e7e00640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9c406b4d0 0x7fc9c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.821+0000 7fc9e7e00640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9c406b4d0 0x7fc9c406d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8000f60 tx=0x7fc9d80079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:53.825+0000 7fc9e4c00640 1 -- 172.21.15.114:0/628022419 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9e80d2f50 con 0x7fc9e80d2360 2024-03-06T07:29:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.098+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9e808d820 con 0x7fc9e80d2360 2024-03-06T07:29:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.099+0000 7fc9e4c00640 1 -- 172.21.15.114:0/628022419 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fc9e808d820 con 0x7fc9e80d2360 2024-03-06T07:29:54.101 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:54.101 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4721,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/308840931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":308840931},{"type":"v1","addr":"172.21.15.177:6833","nonce":308840931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/4238819696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":4238819696},{"type":"v1","addr":"172.21.15.177:6835","nonce":4238819696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4960,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/379671784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":379671784},{"type":"v1","addr":"172.21.15.114:6835","nonce":379671784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:49.994729+0000","modified":"2024-03-06T07:29:53.062242+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5568":{"gid":5568,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2123564678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":2123564678},{"type":"v1","addr":"172.21.15.114:6837","nonce":2123564678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5568,"qdb_cluster":[5568]},"id":3}]} 2024-03-06T07:29:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 26 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9c406b4d0 msgr2=0x7fc9c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9c406b4d0 0x7fc9c406d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8000f60 tx=0x7fc9d80079d0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 msgr2=0x7fc9e814f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e814f260 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0000c00 tx=0x7fc9d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 shutdown_connections 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9e8161580 0x7fc9e80cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9c406b4d0 0x7fc9c406d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9e80d2360 0x7fc9e814f260 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 --2- 172.21.15.114:0/628022419 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9e80b3ec0 0x7fc9e80c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 >> 172.21.15.114:0/628022419 conn(0x7fc9e80d3d80 msgr2=0x7fc9e806c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 shutdown_connections 2024-03-06T07:29:54.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.102+0000 7fc9ed800640 1 -- 172.21.15.114:0/628022419 wait complete. 2024-03-06T07:29:54.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:54.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.326+0000 7f7f95000640 1 Processor -- start 2024-03-06T07:29:54.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.326+0000 7f7f95000640 1 -- start start 2024-03-06T07:29:54.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.327+0000 7f7f95000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 0x7f7f9006b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f95000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9015fed0 con 0x7f7f9015f8f0 2024-03-06T07:29:54.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f95000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f901601b0 con 0x7f7f90156320 2024-03-06T07:29:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f95000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f90071dc0 con 0x7f7f9006afc0 2024-03-06T07:29:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f8ea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7f9015f8f0 0x7f7f900ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34474/0 (socket says 172.21.15.114:34474) 2024-03-06T07:29:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f8ea00640 1 -- 172.21.15.114:0/1230630621 learned_addr learned my addr 172.21.15.114:0/1230630621 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f8fe00640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 0x7f7f9006b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176288375 0 0) 0x7f7f9015fed0 con 0x7f7f9015f8f0 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.328+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f74003520 con 0x7f7f9015f8f0 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1370610053 0 0) 0x7f7f74003520 con 0x7f7f9015f8f0 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 msgr2=0x7f7f9006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 0x7f7f9006b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 >> v1:172.21.15.177:6789/0 conn(0x7f7f90156320 legacy=0x7f7f9015cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f90160490 con 0x7f7f9015f8f0 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7f7c001690 con 0x7f7f9015f8f0 2024-03-06T07:29:54.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 >> v1:172.21.15.114:6789/0 conn(0x7f7f9015f8f0 legacy=0x7f7f900ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f740039e0 0x7f7f74003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 0x7f7f7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f90071dc0 con 0x7f7f74004800 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f74005550 con 0x7f7f740039e0 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7400bf00 con 0x7f7f9006afc0 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.329+0000 7f7f8ea00640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.330+0000 7f7f8f400640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f740039e0 0x7f7f74003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.330+0000 7f7f8fe00640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 0x7f7f7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.330+0000 7f7f8ea00640 1 -- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 msgr2=0x7f7f7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.330+0000 7f7f8ea00640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f9006afc0 0x7f7f7400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.330+0000 7f7f8ea00640 1 -- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f740039e0 msgr2=0x7f7f74003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.330+0000 7f7f8ea00640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f740039e0 0x7f7f74003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.330+0000 7f7f8ea00640 1 -- 172.21.15.114:0/1230630621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f74003520 con 0x7f7f74004800 2024-03-06T07:29:54.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.331+0000 7f7f8ea00640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f74004c80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00c0e0 tx=0x7f7f74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bb3310c13fdfbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.331+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c007c40 con 0x7f7f74004800 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.331+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c030070 con 0x7f7f74004800 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.331+0000 7f7f8e000640 1 -- 172.21.15.114:0/1230630621 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c005d10 con 0x7f7f74004800 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.332+0000 7f7f95000640 1 -- 172.21.15.114:0/1230630621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 msgr2=0x7f7f74004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.332+0000 7f7f95000640 1 --2- 172.21.15.114:0/1230630621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f74004c80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00c0e0 tx=0x7f7f74005830 comp rx=0 tx=0).stop 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.332+0000 7f7f8fe00640 1 -- 172.21.15.114:0/1230630621 reap_dead start 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.332+0000 7f7f95000640 1 -- 172.21.15.114:0/1230630621 shutdown_connections 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.332+0000 7f7f95000640 1 -- 172.21.15.114:0/1230630621 >> 172.21.15.114:0/1230630621 conn(0x7f7f9006c0d0 msgr2=0x7f7f9006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.332+0000 7f7f95000640 1 -- 172.21.15.114:0/1230630621 shutdown_connections 2024-03-06T07:29:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.332+0000 7f7f95000640 1 -- 172.21.15.114:0/1230630621 wait complete. 2024-03-06T07:29:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.334+0000 7f7f95000640 1 Processor -- start 2024-03-06T07:29:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.334+0000 7f7f95000640 1 -- start start 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f95000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f90148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f95000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f90149160 0x7f7f9014b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f95000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f9014b980 0x7f7f90150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f95000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9006b400 con 0x7f7f74004800 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f95000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f90156700 con 0x7f7f9014b980 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f95000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9006b580 con 0x7f7f90149160 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f8f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f90148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f8f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f90148c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48968/0 (socket says 172.21.15.114:48968) 2024-03-06T07:29:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.335+0000 7f7f8f400640 1 -- 172.21.15.114:0/178951797 learned_addr learned my addr 172.21.15.114:0/178951797 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8fe00640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f9014b980 0x7f7f90150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8ea00640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f90149160 0x7f7f9014b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8f400640 1 -- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f90149160 msgr2=0x7f7f9014b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8f400640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f90149160 0x7f7f9014b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8f400640 1 -- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f9014b980 msgr2=0x7f7f90150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8f400640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f9014b980 0x7f7f90150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8f400640 1 -- 172.21.15.114:0/178951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9014fba0 con 0x7f7f74004800 2024-03-06T07:29:54.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f8f400640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f90148c20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000e440 tx=0x7f7f8000e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f87400640 1 -- 172.21.15.114:0/178951797 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f800052e0 con 0x7f7f74004800 2024-03-06T07:29:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f87400640 1 -- 172.21.15.114:0/178951797 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80018070 con 0x7f7f74004800 2024-03-06T07:29:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9014fe80 con 0x7f7f74004800 2024-03-06T07:29:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f87400640 1 -- 172.21.15.114:0/178951797 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80003810 con 0x7f7f74004800 2024-03-06T07:29:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.336+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f90150440 con 0x7f7f74004800 2024-03-06T07:29:54.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.338+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9014bd90 con 0x7f7f74004800 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.338+0000 7f7f87400640 1 -- 172.21.15.114:0/178951797 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f800039b0 con 0x7f7f74004800 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.339+0000 7f7f87400640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f7006b4d0 0x7f7f7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.339+0000 7f7f87400640 1 -- 172.21.15.114:0/178951797 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f80089e00 con 0x7f7f74004800 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.340+0000 7f7f8ea00640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f7006b4d0 0x7f7f7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.340+0000 7f7f8ea00640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f7006b4d0 0x7f7f7006d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c034000 tx=0x7f7f7c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.344+0000 7f7f87400640 1 -- 172.21.15.114:0/178951797 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f9014bd90 con 0x7f7f74004800 2024-03-06T07:29:54.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.542+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f901e6990 con 0x7f7f74004800 2024-03-06T07:29:54.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.544+0000 7f7f87400640 1 -- 172.21.15.114:0/178951797 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f7f80058c70 con 0x7f7f74004800 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:54.546 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:51.547619+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:47.835295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:49.012214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:54.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f7006b4d0 msgr2=0x7f7f7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f7006b4d0 0x7f7f7006d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c034000 tx=0x7f7f7c007000 comp rx=0 tx=0).stop 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 msgr2=0x7f7f90148c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f90148c20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000e440 tx=0x7f7f8000e900 comp rx=0 tx=0).stop 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 shutdown_connections 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f9014b980 0x7f7f90150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f90149160 0x7f7f9014b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f7006b4d0 0x7f7f7006d990 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c034000 tx=0x7f7f7c007000 comp rx=0 tx=0).stop 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 --2- 172.21.15.114:0/178951797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f74004800 0x7f7f90148c20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 >> 172.21.15.114:0/178951797 conn(0x7f7f9006c0d0 msgr2=0x7f7f901544c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.546+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 shutdown_connections 2024-03-06T07:29:54.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.547+0000 7f7f95000640 1 -- 172.21.15.114:0/178951797 wait complete. 2024-03-06T07:29:54.567 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:29:54.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.756+0000 7f200ee00640 1 Processor -- start 2024-03-06T07:29:54.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.757+0000 7f200ee00640 1 -- start start 2024-03-06T07:29:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.758+0000 7f200ee00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f200815e0d0 0x7f200806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.758+0000 7f200ee00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20080b1440 con 0x7f200815e4b0 2024-03-06T07:29:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.758+0000 7f200ee00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20080b16e0 con 0x7f20080af950 2024-03-06T07:29:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.758+0000 7f200ee00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200806b500 con 0x7f200815e0d0 2024-03-06T07:29:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.758+0000 7f200d000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f200815e4b0 0x7f20080af240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34490/0 (socket says 172.21.15.114:34490) 2024-03-06T07:29:54.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.758+0000 7f200d000640 1 -- 172.21.15.114:0/2125533882 learned_addr learned my addr 172.21.15.114:0/2125533882 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:54.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f200da00640 1 --2- 172.21.15.114:0/2125533882 >> v2:172.21.15.177:3300/0 conn(0x7f200815e0d0 0x7f200806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983769406 0 0) 0x7f20080b1440 con 0x7f200815e4b0 2024-03-06T07:29:54.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff0003520 con 0x7f200815e4b0 2024-03-06T07:29:54.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776695474 0 0) 0x7f1ff0003520 con 0x7f200815e4b0 2024-03-06T07:29:54.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 >> v2:172.21.15.177:3300/0 conn(0x7f200815e0d0 msgr2=0x7f200806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 --2- 172.21.15.114:0/2125533882 >> v2:172.21.15.177:3300/0 conn(0x7f200815e0d0 0x7f200806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 >> v1:172.21.15.177:6789/0 conn(0x7f20080af950 legacy=0x7f20080ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080b19c0 con 0x7f200815e4b0 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f1ff4001670 con 0x7f200815e4b0 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 >> v1:172.21.15.114:6789/0 conn(0x7f200815e4b0 legacy=0x7f20080af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ff0003a00 0x7f1ff0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff0004800 0x7f1ff0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.759+0000 7f2007e00640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff00051c0 0x7f1ff000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200806b500 con 0x7f1ff0004800 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000c0a0 con 0x7f1ff0003a00 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000c2a0 con 0x7f1ff00051c0 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200d000640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff0004800 0x7f1ff0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200da00640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ff0003a00 0x7f1ff0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200e400640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff00051c0 0x7f1ff000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200d000640 1 -- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff00051c0 msgr2=0x7f1ff000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200d000640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff00051c0 0x7f1ff000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200d000640 1 -- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ff0003a00 msgr2=0x7f1ff0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200d000640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ff0003a00 0x7f1ff0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200d000640 1 -- 172.21.15.114:0/2125533882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0003520 con 0x7f1ff0004800 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200e400640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff00051c0 0x7f1ff000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.760+0000 7f200e400640 1 -- 172.21.15.114:0/2125533882 reap_dead start 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.761+0000 7f200d000640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff0004800 0x7f1ff0004c80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400c0e0 tx=0x7f1ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d82cbd4764486c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.761+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4007c20 con 0x7f1ff0004800 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4030070 con 0x7f1ff0004800 2024-03-06T07:29:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f2007e00640 1 -- 172.21.15.114:0/2125533882 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4005d70 con 0x7f1ff0004800 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f200ee00640 1 -- 172.21.15.114:0/2125533882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff0004800 msgr2=0x7f1ff0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f200ee00640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff0004800 0x7f1ff0004c80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400c0e0 tx=0x7f1ff0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f200ee00640 1 -- 172.21.15.114:0/2125533882 shutdown_connections 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f200ee00640 1 --2- 172.21.15.114:0/2125533882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ff0004800 0x7f1ff0004c80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f200ee00640 1 -- 172.21.15.114:0/2125533882 >> 172.21.15.114:0/2125533882 conn(0x7f20080b4fe0 msgr2=0x7f20080b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f200ee00640 1 -- 172.21.15.114:0/2125533882 shutdown_connections 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.762+0000 7f200ee00640 1 -- 172.21.15.114:0/2125533882 wait complete. 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.764+0000 7f200ee00640 1 Processor -- start 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.764+0000 7f200ee00640 1 -- start start 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200ee00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff0004800 0x7f20081f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200ee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2008161220 0x7f20081536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200ee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20081f4ce0 0x7f2008158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200ee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200806c690 con 0x7f2008161220 2024-03-06T07:29:54.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200ee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200806c0d0 con 0x7f20081f4ce0 2024-03-06T07:29:54.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200ee00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20080b5db0 con 0x7f1ff0004800 2024-03-06T07:29:54.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2008161220 0x7f20081536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2008161220 0x7f20081536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48974/0 (socket says 172.21.15.114:48974) 2024-03-06T07:29:54.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.765+0000 7f200d000640 1 -- 172.21.15.114:0/930743462 learned_addr learned my addr 172.21.15.114:0/930743462 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:54.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200e400640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20081f4ce0 0x7f2008158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200da00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff0004800 0x7f20081f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200d000640 1 -- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff0004800 msgr2=0x7f20081f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200d000640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff0004800 0x7f20081f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200d000640 1 -- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20081f4ce0 msgr2=0x7f2008158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200d000640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20081f4ce0 0x7f2008158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200d000640 1 -- 172.21.15.114:0/930743462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2008153c30 con 0x7f2008161220 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200e400640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20081f4ce0 0x7f2008158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200da00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff0004800 0x7f20081f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200d000640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2008161220 0x7f20081536f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ff402c480 tx=0x7f20080b18b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f2005600640 1 -- 172.21.15.114:0/930743462 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4004bf0 con 0x7f2008161220 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f2005600640 1 -- 172.21.15.114:0/930743462 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4000ed0 con 0x7f2008161220 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f2005600640 1 -- 172.21.15.114:0/930743462 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4007070 con 0x7f2008161220 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.766+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20081616a0 con 0x7f2008161220 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.767+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20081f43c0 con 0x7f2008161220 2024-03-06T07:29:54.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.768+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200804f330 con 0x7f2008161220 2024-03-06T07:29:54.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.769+0000 7f2005600640 1 -- 172.21.15.114:0/930743462 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ff4048020 con 0x7f2008161220 2024-03-06T07:29:54.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.770+0000 7f2005600640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1fe806b4d0 0x7f1fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.770+0000 7f2005600640 1 -- 172.21.15.114:0/930743462 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ff40ab3e0 con 0x7f2008161220 2024-03-06T07:29:54.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.770+0000 7f200da00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1fe806b4d0 0x7f1fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.771+0000 7f200da00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1fe806b4d0 0x7f1fe806d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f20081511e0 tx=0x7f1ffc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.774+0000 7f2005600640 1 -- 172.21.15.114:0/930743462 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2008153c30 con 0x7f2008161220 2024-03-06T07:29:54.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.964+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20080d0780 con 0x7f1fe806b4d0 2024-03-06T07:29:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.976+0000 7f2005600640 1 -- 172.21.15.114:0/930743462 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237202 (secure 0 0 0) 0x7f20080d0780 con 0x7f1fe806b4d0 2024-03-06T07:29:54.981 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:54.982 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:29:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1fe806b4d0 msgr2=0x7f1fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1fe806b4d0 0x7f1fe806d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f20081511e0 tx=0x7f1ffc008040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2008161220 msgr2=0x7f20081536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2008161220 0x7f20081536f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ff402c480 tx=0x7f20080b18b0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 shutdown_connections 2024-03-06T07:29:54.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20081f4ce0 0x7f2008158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1fe806b4d0 0x7f1fe806d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2008161220 0x7f20081536f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 --2- 172.21.15.114:0/930743462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ff0004800 0x7f20081f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 >> 172.21.15.114:0/930743462 conn(0x7f20080b4fe0 msgr2=0x7f20080ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 shutdown_connections 2024-03-06T07:29:54.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:54.982+0000 7f200ee00640 1 -- 172.21.15.114:0/930743462 wait complete. 2024-03-06T07:29:55.002 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-06T07:29:53.133644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":250132,"kb_used_data":4808,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724588,"statfs":{"total":773094113280,"available":772837978112,"internally_reserved":0,"allocated":4923392,"data_stored":2659224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":251055667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.611157"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:49.516837+0000","last_change":"2024-03-06T07:29:49.516837+0000","last_active":"2024-03-06T07:29:49.516837+0000","last_peered":"2024-03-06T07:29:49.516837+0000","last_clean":"2024-03-06T07:29:49.516837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:49.516837+0000","last_undegraded":"2024-03-06T07:29:49.516837+0000","last_fullsized":"2024-03-06T07:29:49.516837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:29:48.501959+0000","last_change":"2024-03-06T07:29:48.501959+0000","last_active":"2024-03-06T07:29:48.501959+0000","last_peered":"2024-03-06T07:29:48.501959+0000","last_clean":"2024-03-06T07:29:48.501959+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:48.501959+0000","last_undegraded":"2024-03-06T07:29:48.501959+0000","last_fullsized":"2024-03-06T07:29:48.501959+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-06T07:29:46.511377+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:29:46.511377+0000","last_peered":"2024-03-06T07:29:46.511377+0000","last_clean":"2024-03-06T07:29:46.511377+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:29:46.511377+0000","last_undegraded":"2024-03-06T07:29:46.511377+0000","last_fullsized":"2024-03-06T07:29:46.511377+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":775179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":466944,"data_stored":184811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31636,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":466944,"data_stored":184811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31640,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":466944,"data_stored":184811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":466944,"data_stored":184811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27488,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":466944,"data_stored":184811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31640,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":466944,"data_stored":184811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":775179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:55.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:55.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.242+0000 7f8aa0600640 1 Processor -- start 2024-03-06T07:29:55.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.242+0000 7f8aa0600640 1 -- start start 2024-03-06T07:29:55.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.245+0000 7f8aa0600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8a9815acb0 0x7f8a980b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.245+0000 7f8aa0600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a9806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.245+0000 7f8aa0600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9815b090 con 0x7f8a98072340 2024-03-06T07:29:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.245+0000 7f8aa0600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a980b2a30 con 0x7f8a9815acb0 2024-03-06T07:29:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.245+0000 7f8aa0600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980b4360 con 0x7f8a980b4680 2024-03-06T07:29:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8a9815acb0 0x7f8a980b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9d200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a9806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8a9815acb0 0x7f8a980b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:57682/0 (socket says 172.21.15.114:57682) 2024-03-06T07:29:55.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9dc00640 1 -- 172.21.15.114:0/1879712463 learned_addr learned my addr 172.21.15.114:0/1879712463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9d200640 1 -- 172.21.15.114:0/1879712463 >> v2:172.21.15.177:3300/0 conn(0x7f8a9815acb0 msgr2=0x7f8a980b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9d200640 1 --2- 172.21.15.114:0/1879712463 >> v2:172.21.15.177:3300/0 conn(0x7f8a9815acb0 0x7f8a980b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9d200640 1 -- 172.21.15.114:0/1879712463 >> v1:172.21.15.177:6789/0 conn(0x7f8a98072340 legacy=0x7f8a980ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.246+0000 7f8a9d200640 1 -- 172.21.15.114:0/1879712463 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a980b2cf0 con 0x7f8a980b4680 2024-03-06T07:29:55.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.248+0000 7f8a9d200640 1 --2- 172.21.15.114:0/1879712463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a9806e4f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a840097a0 tx=0x7f8a8402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d66ba2c90f94502 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.248+0000 7f8a97e00640 1 -- 172.21.15.114:0/1879712463 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84035020 con 0x7f8a980b4680 2024-03-06T07:29:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.248+0000 7f8a97e00640 1 -- 172.21.15.114:0/1879712463 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a84031810 con 0x7f8a980b4680 2024-03-06T07:29:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.248+0000 7f8a97e00640 1 -- 172.21.15.114:0/1879712463 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84031be0 con 0x7f8a980b4680 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 -- 172.21.15.114:0/1879712463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 msgr2=0x7f8a9806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 --2- 172.21.15.114:0/1879712463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a9806e4f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a840097a0 tx=0x7f8a8402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 -- 172.21.15.114:0/1879712463 shutdown_connections 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 --2- 172.21.15.114:0/1879712463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a9806e4f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 --2- 172.21.15.114:0/1879712463 >> v2:172.21.15.177:3300/0 conn(0x7f8a9815acb0 0x7f8a980b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 -- 172.21.15.114:0/1879712463 >> 172.21.15.114:0/1879712463 conn(0x7f8a980b1730 msgr2=0x7f8a980b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 -- 172.21.15.114:0/1879712463 shutdown_connections 2024-03-06T07:29:55.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.249+0000 7f8aa0600640 1 -- 172.21.15.114:0/1879712463 wait complete. 2024-03-06T07:29:55.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.253+0000 7f8aa0600640 1 Processor -- start 2024-03-06T07:29:55.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.253+0000 7f8aa0600640 1 -- start start 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8aa0600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98072340 0x7f8a98154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8aa0600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a980b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8aa0600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a9815acb0 0x7f8a9814c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8aa0600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980b3d10 con 0x7f8a98072340 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8aa0600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980b32b0 con 0x7f8a9815acb0 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8aa0600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980b4010 con 0x7f8a980b4680 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8a9dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98072340 0x7f8a98154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8a9dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98072340 0x7f8a98154fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48976/0 (socket says 172.21.15.114:48976) 2024-03-06T07:29:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8a9dc00640 1 -- 172.21.15.114:0/3614389468 learned_addr learned my addr 172.21.15.114:0/3614389468 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.255+0000 7f8a9d200640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a980b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9e600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a9815acb0 0x7f8a9814c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9dc00640 1 -- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 msgr2=0x7f8a980b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9dc00640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a980b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9dc00640 1 -- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a9815acb0 msgr2=0x7f8a9814c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9dc00640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a9815acb0 0x7f8a9814c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9dc00640 1 -- 172.21.15.114:0/3614389468 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9815da10 con 0x7f8a98072340 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9e600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a9815acb0 0x7f8a9814c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a9dc00640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98072340 0x7f8a98154fe0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009fd0 tx=0x7f8a8c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a95600640 1 -- 172.21.15.114:0/3614389468 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c017070 con 0x7f8a98072340 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a95600640 1 -- 172.21.15.114:0/3614389468 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c00c940 con 0x7f8a98072340 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9815b300 con 0x7f8a98072340 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.256+0000 7f8a95600640 1 -- 172.21.15.114:0/3614389468 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c00cd10 con 0x7f8a98072340 2024-03-06T07:29:55.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.257+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9815dcb0 con 0x7f8a98072340 2024-03-06T07:29:55.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.258+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a981555e0 con 0x7f8a98072340 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.259+0000 7f8a95600640 1 -- 172.21.15.114:0/3614389468 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a8c01a070 con 0x7f8a98072340 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.260+0000 7f8a95600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a7806b4d0 0x7f8a7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.260+0000 7f8a95600640 1 -- 172.21.15.114:0/3614389468 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a8c08a100 con 0x7f8a98072340 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.260+0000 7f8a9d200640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a7806b4d0 0x7f8a7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.261+0000 7f8a9d200640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a7806b4d0 0x7f8a7806d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8a84036000 tx=0x7f8a84008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.264+0000 7f8a95600640 1 -- 172.21.15.114:0/3614389468 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a8c059010 con 0x7f8a98072340 2024-03-06T07:29:55.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.455+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a9806ce60 con 0x7f8a7806b4d0 2024-03-06T07:29:55.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.467+0000 7f8a95600640 1 -- 172.21.15.114:0/3614389468 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251956 (secure 0 0 0) 0x7f8a84086850 con 0x7f8a7806b4d0 2024-03-06T07:29:55.472 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:55.473 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:29:55.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a7806b4d0 msgr2=0x7f8a7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a7806b4d0 0x7f8a7806d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8a84036000 tx=0x7f8a84008030 comp rx=0 tx=0).stop 2024-03-06T07:29:55.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98072340 msgr2=0x7f8a98154fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98072340 0x7f8a98154fe0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009fd0 tx=0x7f8a8c00ed40 comp rx=0 tx=0).stop 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 shutdown_connections 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a9815acb0 0x7f8a9814c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a980b4680 0x7f8a980b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a7806b4d0 0x7f8a7806d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 --2- 172.21.15.114:0/3614389468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a98072340 0x7f8a98154fe0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 >> 172.21.15.114:0/3614389468 conn(0x7f8a980b1730 msgr2=0x7f8a980af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 shutdown_connections 2024-03-06T07:29:55.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.473+0000 7f8aa0600640 1 -- 172.21.15.114:0/3614389468 wait complete. 2024-03-06T07:29:55.494 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-06T07:29:55.134258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250600,"kb_used_data":5256,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724120,"statfs":{"total":773094113280,"available":772837498880,"internally_reserved":0,"allocated":5382144,"data_stored":2865316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":251055667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001790"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208742+0000","last_change":"2024-03-06T07:29:50.562058+0000","last_active":"2024-03-06T07:29:52.208742+0000","last_peered":"2024-03-06T07:29:52.208742+0000","last_clean":"2024-03-06T07:29:52.208742+0000","last_became_active":"2024-03-06T07:29:50.561789+0000","last_became_peered":"2024-03-06T07:29:50.561789+0000","last_unstale":"2024-03-06T07:29:52.208742+0000","last_undegraded":"2024-03-06T07:29:52.208742+0000","last_fullsized":"2024-03-06T07:29:52.208742+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:26.094622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207715+0000","last_change":"2024-03-06T07:29:50.567753+0000","last_active":"2024-03-06T07:29:52.207715+0000","last_peered":"2024-03-06T07:29:52.207715+0000","last_clean":"2024-03-06T07:29:52.207715+0000","last_became_active":"2024-03-06T07:29:50.567424+0000","last_became_peered":"2024-03-06T07:29:50.567424+0000","last_unstale":"2024-03-06T07:29:52.207715+0000","last_undegraded":"2024-03-06T07:29:52.207715+0000","last_fullsized":"2024-03-06T07:29:52.207715+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:24.360707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572574+0000","last_change":"2024-03-06T07:29:50.559739+0000","last_active":"2024-03-06T07:29:51.572574+0000","last_peered":"2024-03-06T07:29:51.572574+0000","last_clean":"2024-03-06T07:29:51.572574+0000","last_became_active":"2024-03-06T07:29:50.559475+0000","last_became_peered":"2024-03-06T07:29:50.559475+0000","last_unstale":"2024-03-06T07:29:51.572574+0000","last_undegraded":"2024-03-06T07:29:51.572574+0000","last_fullsized":"2024-03-06T07:29:51.572574+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:17.116705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206854+0000","last_change":"2024-03-06T07:29:50.573393+0000","last_active":"2024-03-06T07:29:52.206854+0000","last_peered":"2024-03-06T07:29:52.206854+0000","last_clean":"2024-03-06T07:29:52.206854+0000","last_became_active":"2024-03-06T07:29:50.572926+0000","last_became_peered":"2024-03-06T07:29:50.572926+0000","last_unstale":"2024-03-06T07:29:52.206854+0000","last_undegraded":"2024-03-06T07:29:52.206854+0000","last_fullsized":"2024-03-06T07:29:52.206854+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:24.360707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208144+0000","last_change":"2024-03-06T07:29:50.573968+0000","last_active":"2024-03-06T07:29:52.208144+0000","last_peered":"2024-03-06T07:29:52.208144+0000","last_clean":"2024-03-06T07:29:52.208144+0000","last_became_active":"2024-03-06T07:29:50.573635+0000","last_became_peered":"2024-03-06T07:29:50.573635+0000","last_unstale":"2024-03-06T07:29:52.208144+0000","last_undegraded":"2024-03-06T07:29:52.208144+0000","last_fullsized":"2024-03-06T07:29:52.208144+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:45.695291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567594+0000","last_change":"2024-03-06T07:29:50.555805+0000","last_active":"2024-03-06T07:29:51.567594+0000","last_peered":"2024-03-06T07:29:51.567594+0000","last_clean":"2024-03-06T07:29:51.567594+0000","last_became_active":"2024-03-06T07:29:50.555188+0000","last_became_peered":"2024-03-06T07:29:50.555188+0000","last_unstale":"2024-03-06T07:29:51.567594+0000","last_undegraded":"2024-03-06T07:29:51.567594+0000","last_fullsized":"2024-03-06T07:29:51.567594+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:06.558708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208061+0000","last_change":"2024-03-06T07:29:50.571488+0000","last_active":"2024-03-06T07:29:52.208061+0000","last_peered":"2024-03-06T07:29:52.208061+0000","last_clean":"2024-03-06T07:29:52.208061+0000","last_became_active":"2024-03-06T07:29:50.571215+0000","last_became_peered":"2024-03-06T07:29:50.571215+0000","last_unstale":"2024-03-06T07:29:52.208061+0000","last_undegraded":"2024-03-06T07:29:52.208061+0000","last_fullsized":"2024-03-06T07:29:52.208061+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:17.326685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206204+0000","last_change":"2024-03-06T07:29:50.572305+0000","last_active":"2024-03-06T07:29:52.206204+0000","last_peered":"2024-03-06T07:29:52.206204+0000","last_clean":"2024-03-06T07:29:52.206204+0000","last_became_active":"2024-03-06T07:29:50.571808+0000","last_became_peered":"2024-03-06T07:29:50.571808+0000","last_unstale":"2024-03-06T07:29:52.206204+0000","last_undegraded":"2024-03-06T07:29:52.206204+0000","last_fullsized":"2024-03-06T07:29:52.206204+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:56.015309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207459+0000","last_change":"2024-03-06T07:29:50.581574+0000","last_active":"2024-03-06T07:29:52.207459+0000","last_peered":"2024-03-06T07:29:52.207459+0000","last_clean":"2024-03-06T07:29:52.207459+0000","last_became_active":"2024-03-06T07:29:50.581273+0000","last_became_peered":"2024-03-06T07:29:50.581273+0000","last_unstale":"2024-03-06T07:29:52.207459+0000","last_undegraded":"2024-03-06T07:29:52.207459+0000","last_fullsized":"2024-03-06T07:29:52.207459+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:17.326685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207398+0000","last_change":"2024-03-06T07:29:50.581512+0000","last_active":"2024-03-06T07:29:52.207398+0000","last_peered":"2024-03-06T07:29:52.207398+0000","last_clean":"2024-03-06T07:29:52.207398+0000","last_became_active":"2024-03-06T07:29:50.580456+0000","last_became_peered":"2024-03-06T07:29:50.580456+0000","last_unstale":"2024-03-06T07:29:52.207398+0000","last_undegraded":"2024-03-06T07:29:52.207398+0000","last_fullsized":"2024-03-06T07:29:52.207398+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:07.158209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207099+0000","last_change":"2024-03-06T07:29:50.561896+0000","last_active":"2024-03-06T07:29:52.207099+0000","last_peered":"2024-03-06T07:29:52.207099+0000","last_clean":"2024-03-06T07:29:52.207099+0000","last_became_active":"2024-03-06T07:29:50.561610+0000","last_became_peered":"2024-03-06T07:29:50.561610+0000","last_unstale":"2024-03-06T07:29:52.207099+0000","last_undegraded":"2024-03-06T07:29:52.207099+0000","last_fullsized":"2024-03-06T07:29:52.207099+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:45.695291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208716+0000","last_change":"2024-03-06T07:29:50.563113+0000","last_active":"2024-03-06T07:29:52.208716+0000","last_peered":"2024-03-06T07:29:52.208716+0000","last_clean":"2024-03-06T07:29:52.208716+0000","last_became_active":"2024-03-06T07:29:50.560856+0000","last_became_peered":"2024-03-06T07:29:50.560856+0000","last_unstale":"2024-03-06T07:29:52.208716+0000","last_undegraded":"2024-03-06T07:29:52.208716+0000","last_fullsized":"2024-03-06T07:29:52.208716+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:07.158209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565786+0000","last_change":"2024-03-06T07:29:50.559522+0000","last_active":"2024-03-06T07:29:51.565786+0000","last_peered":"2024-03-06T07:29:51.565786+0000","last_clean":"2024-03-06T07:29:51.565786+0000","last_became_active":"2024-03-06T07:29:50.559229+0000","last_became_peered":"2024-03-06T07:29:50.559229+0000","last_unstale":"2024-03-06T07:29:51.565786+0000","last_undegraded":"2024-03-06T07:29:51.565786+0000","last_fullsized":"2024-03-06T07:29:51.565786+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:02.448688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206162+0000","last_change":"2024-03-06T07:29:50.582976+0000","last_active":"2024-03-06T07:29:52.206162+0000","last_peered":"2024-03-06T07:29:52.206162+0000","last_clean":"2024-03-06T07:29:52.206162+0000","last_became_active":"2024-03-06T07:29:50.582735+0000","last_became_peered":"2024-03-06T07:29:50.582735+0000","last_unstale":"2024-03-06T07:29:52.206162+0000","last_undegraded":"2024-03-06T07:29:52.206162+0000","last_fullsized":"2024-03-06T07:29:52.206162+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:44.925680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565022+0000","last_change":"2024-03-06T07:29:50.560797+0000","last_active":"2024-03-06T07:29:51.565022+0000","last_peered":"2024-03-06T07:29:51.565022+0000","last_clean":"2024-03-06T07:29:51.565022+0000","last_became_active":"2024-03-06T07:29:50.559499+0000","last_became_peered":"2024-03-06T07:29:50.559499+0000","last_unstale":"2024-03-06T07:29:51.565022+0000","last_undegraded":"2024-03-06T07:29:51.565022+0000","last_fullsized":"2024-03-06T07:29:51.565022+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:00.236412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565200+0000","last_change":"2024-03-06T07:29:50.556870+0000","last_active":"2024-03-06T07:29:51.565200+0000","last_peered":"2024-03-06T07:29:51.565200+0000","last_clean":"2024-03-06T07:29:51.565200+0000","last_became_active":"2024-03-06T07:29:50.556251+0000","last_became_peered":"2024-03-06T07:29:50.556251+0000","last_unstale":"2024-03-06T07:29:51.565200+0000","last_undegraded":"2024-03-06T07:29:51.565200+0000","last_fullsized":"2024-03-06T07:29:51.565200+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:19.963413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572804+0000","last_change":"2024-03-06T07:29:50.557806+0000","last_active":"2024-03-06T07:29:51.572804+0000","last_peered":"2024-03-06T07:29:51.572804+0000","last_clean":"2024-03-06T07:29:51.572804+0000","last_became_active":"2024-03-06T07:29:50.557292+0000","last_became_peered":"2024-03-06T07:29:50.557292+0000","last_unstale":"2024-03-06T07:29:51.572804+0000","last_undegraded":"2024-03-06T07:29:51.572804+0000","last_fullsized":"2024-03-06T07:29:51.572804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:44.925680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.573001+0000","last_change":"2024-03-06T07:29:50.581521+0000","last_active":"2024-03-06T07:29:51.573001+0000","last_peered":"2024-03-06T07:29:51.573001+0000","last_clean":"2024-03-06T07:29:51.573001+0000","last_became_active":"2024-03-06T07:29:50.581174+0000","last_became_peered":"2024-03-06T07:29:50.581174+0000","last_unstale":"2024-03-06T07:29:51.573001+0000","last_undegraded":"2024-03-06T07:29:51.573001+0000","last_fullsized":"2024-03-06T07:29:51.573001+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:17.536713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566190+0000","last_change":"2024-03-06T07:29:50.553996+0000","last_active":"2024-03-06T07:29:51.566190+0000","last_peered":"2024-03-06T07:29:51.566190+0000","last_clean":"2024-03-06T07:29:51.566190+0000","last_became_active":"2024-03-06T07:29:50.553276+0000","last_became_peered":"2024-03-06T07:29:50.553276+0000","last_unstale":"2024-03-06T07:29:51.566190+0000","last_undegraded":"2024-03-06T07:29:51.566190+0000","last_fullsized":"2024-03-06T07:29:51.566190+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:06.131618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566475+0000","last_change":"2024-03-06T07:29:50.557488+0000","last_active":"2024-03-06T07:29:51.566475+0000","last_peered":"2024-03-06T07:29:51.566475+0000","last_clean":"2024-03-06T07:29:51.566475+0000","last_became_active":"2024-03-06T07:29:50.557176+0000","last_became_peered":"2024-03-06T07:29:50.557176+0000","last_unstale":"2024-03-06T07:29:51.566475+0000","last_undegraded":"2024-03-06T07:29:51.566475+0000","last_fullsized":"2024-03-06T07:29:51.566475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:16.179450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565409+0000","last_change":"2024-03-06T07:29:50.555171+0000","last_active":"2024-03-06T07:29:51.565409+0000","last_peered":"2024-03-06T07:29:51.565409+0000","last_clean":"2024-03-06T07:29:51.565409+0000","last_became_active":"2024-03-06T07:29:50.549572+0000","last_became_peered":"2024-03-06T07:29:50.549572+0000","last_unstale":"2024-03-06T07:29:51.565409+0000","last_undegraded":"2024-03-06T07:29:51.565409+0000","last_fullsized":"2024-03-06T07:29:51.565409+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:37.733061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206168+0000","last_change":"2024-03-06T07:29:50.575109+0000","last_active":"2024-03-06T07:29:52.206168+0000","last_peered":"2024-03-06T07:29:52.206168+0000","last_clean":"2024-03-06T07:29:52.206168+0000","last_became_active":"2024-03-06T07:29:50.570810+0000","last_became_peered":"2024-03-06T07:29:50.570810+0000","last_unstale":"2024-03-06T07:29:52.206168+0000","last_undegraded":"2024-03-06T07:29:52.206168+0000","last_fullsized":"2024-03-06T07:29:52.206168+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:17.536713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565059+0000","last_change":"2024-03-06T07:29:50.552386+0000","last_active":"2024-03-06T07:29:51.565059+0000","last_peered":"2024-03-06T07:29:51.565059+0000","last_clean":"2024-03-06T07:29:51.565059+0000","last_became_active":"2024-03-06T07:29:50.551939+0000","last_became_peered":"2024-03-06T07:29:50.551939+0000","last_unstale":"2024-03-06T07:29:51.565059+0000","last_undegraded":"2024-03-06T07:29:51.565059+0000","last_fullsized":"2024-03-06T07:29:51.565059+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:18.879688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.564923+0000","last_change":"2024-03-06T07:29:50.554217+0000","last_active":"2024-03-06T07:29:51.564923+0000","last_peered":"2024-03-06T07:29:51.564923+0000","last_clean":"2024-03-06T07:29:51.564923+0000","last_became_active":"2024-03-06T07:29:50.553830+0000","last_became_peered":"2024-03-06T07:29:50.553830+0000","last_unstale":"2024-03-06T07:29:51.564923+0000","last_undegraded":"2024-03-06T07:29:51.564923+0000","last_fullsized":"2024-03-06T07:29:51.564923+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:01.378687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566999+0000","last_change":"2024-03-06T07:29:50.564220+0000","last_active":"2024-03-06T07:29:51.566999+0000","last_peered":"2024-03-06T07:29:51.566999+0000","last_clean":"2024-03-06T07:29:51.566999+0000","last_became_active":"2024-03-06T07:29:50.563905+0000","last_became_peered":"2024-03-06T07:29:50.563905+0000","last_unstale":"2024-03-06T07:29:51.566999+0000","last_undegraded":"2024-03-06T07:29:51.566999+0000","last_fullsized":"2024-03-06T07:29:51.566999+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:41.833195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567326+0000","last_change":"2024-03-06T07:29:50.564390+0000","last_active":"2024-03-06T07:29:51.567326+0000","last_peered":"2024-03-06T07:29:51.567326+0000","last_clean":"2024-03-06T07:29:51.567326+0000","last_became_active":"2024-03-06T07:29:50.561610+0000","last_became_peered":"2024-03-06T07:29:50.561610+0000","last_unstale":"2024-03-06T07:29:51.567326+0000","last_undegraded":"2024-03-06T07:29:51.567326+0000","last_fullsized":"2024-03-06T07:29:51.567326+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:28:32.314359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572886+0000","last_change":"2024-03-06T07:29:50.562661+0000","last_active":"2024-03-06T07:29:51.572886+0000","last_peered":"2024-03-06T07:29:51.572886+0000","last_clean":"2024-03-06T07:29:51.572886+0000","last_became_active":"2024-03-06T07:29:50.562365+0000","last_became_peered":"2024-03-06T07:29:50.562365+0000","last_unstale":"2024-03-06T07:29:51.572886+0000","last_undegraded":"2024-03-06T07:29:51.572886+0000","last_fullsized":"2024-03-06T07:29:51.572886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:33.047728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208694+0000","last_change":"2024-03-06T07:29:50.561972+0000","last_active":"2024-03-06T07:29:52.208694+0000","last_peered":"2024-03-06T07:29:52.208694+0000","last_clean":"2024-03-06T07:29:52.208694+0000","last_became_active":"2024-03-06T07:29:50.561324+0000","last_became_peered":"2024-03-06T07:29:50.561324+0000","last_unstale":"2024-03-06T07:29:52.208694+0000","last_undegraded":"2024-03-06T07:29:52.208694+0000","last_fullsized":"2024-03-06T07:29:52.208694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:04.580321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565529+0000","last_change":"2024-03-06T07:29:50.563338+0000","last_active":"2024-03-06T07:29:51.565529+0000","last_peered":"2024-03-06T07:29:51.565529+0000","last_clean":"2024-03-06T07:29:51.565529+0000","last_became_active":"2024-03-06T07:29:50.563051+0000","last_became_peered":"2024-03-06T07:29:50.563051+0000","last_unstale":"2024-03-06T07:29:51.565529+0000","last_undegraded":"2024-03-06T07:29:51.565529+0000","last_fullsized":"2024-03-06T07:29:51.565529+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:26.390356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572323+0000","last_change":"2024-03-06T07:29:50.561601+0000","last_active":"2024-03-06T07:29:51.572323+0000","last_peered":"2024-03-06T07:29:51.572323+0000","last_clean":"2024-03-06T07:29:51.572323+0000","last_became_active":"2024-03-06T07:29:50.561306+0000","last_became_peered":"2024-03-06T07:29:50.561306+0000","last_unstale":"2024-03-06T07:29:51.572323+0000","last_undegraded":"2024-03-06T07:29:51.572323+0000","last_fullsized":"2024-03-06T07:29:51.572323+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:09.575158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565250+0000","last_change":"2024-03-06T07:29:50.569474+0000","last_active":"2024-03-06T07:29:51.565250+0000","last_peered":"2024-03-06T07:29:51.565250+0000","last_clean":"2024-03-06T07:29:51.565250+0000","last_became_active":"2024-03-06T07:29:50.569174+0000","last_became_peered":"2024-03-06T07:29:50.569174+0000","last_unstale":"2024-03-06T07:29:51.565250+0000","last_undegraded":"2024-03-06T07:29:51.565250+0000","last_fullsized":"2024-03-06T07:29:51.565250+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:30.682988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565451+0000","last_change":"2024-03-06T07:29:50.558014+0000","last_active":"2024-03-06T07:29:51.565451+0000","last_peered":"2024-03-06T07:29:51.565451+0000","last_clean":"2024-03-06T07:29:51.565451+0000","last_became_active":"2024-03-06T07:29:50.554107+0000","last_became_peered":"2024-03-06T07:29:50.554107+0000","last_unstale":"2024-03-06T07:29:51.565451+0000","last_undegraded":"2024-03-06T07:29:51.565451+0000","last_fullsized":"2024-03-06T07:29:51.565451+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:35.501808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.032960+0000","last_change":"2024-03-06T07:29:50.569940+0000","last_active":"2024-03-06T07:29:52.032960+0000","last_peered":"2024-03-06T07:29:52.032960+0000","last_clean":"2024-03-06T07:29:52.032960+0000","last_became_active":"2024-03-06T07:29:50.569624+0000","last_became_peered":"2024-03-06T07:29:50.569624+0000","last_unstale":"2024-03-06T07:29:52.032960+0000","last_undegraded":"2024-03-06T07:29:52.032960+0000","last_fullsized":"2024-03-06T07:29:52.032960+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:48.398258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567363+0000","last_change":"2024-03-06T07:29:49.566013+0000","last_active":"2024-03-06T07:29:51.567363+0000","last_peered":"2024-03-06T07:29:51.567363+0000","last_clean":"2024-03-06T07:29:51.567363+0000","last_became_active":"2024-03-06T07:29:49.565826+0000","last_became_peered":"2024-03-06T07:29:49.565826+0000","last_unstale":"2024-03-06T07:29:51.567363+0000","last_undegraded":"2024-03-06T07:29:51.567363+0000","last_fullsized":"2024-03-06T07:29:51.567363+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:32.032849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567180+0000","last_change":"2024-03-06T07:29:50.561717+0000","last_active":"2024-03-06T07:29:51.567180+0000","last_peered":"2024-03-06T07:29:51.567180+0000","last_clean":"2024-03-06T07:29:51.567180+0000","last_became_active":"2024-03-06T07:29:50.561414+0000","last_became_peered":"2024-03-06T07:29:50.561414+0000","last_unstale":"2024-03-06T07:29:51.567180+0000","last_undegraded":"2024-03-06T07:29:51.567180+0000","last_fullsized":"2024-03-06T07:29:51.567180+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:09.575158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.573151+0000","last_change":"2024-03-06T07:29:49.565960+0000","last_active":"2024-03-06T07:29:51.573151+0000","last_peered":"2024-03-06T07:29:51.573151+0000","last_clean":"2024-03-06T07:29:51.573151+0000","last_became_active":"2024-03-06T07:29:49.565622+0000","last_became_peered":"2024-03-06T07:29:49.565622+0000","last_unstale":"2024-03-06T07:29:51.573151+0000","last_undegraded":"2024-03-06T07:29:51.573151+0000","last_fullsized":"2024-03-06T07:29:51.573151+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:22.643749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.209124+0000","last_change":"2024-03-06T07:29:50.562199+0000","last_active":"2024-03-06T07:29:52.209124+0000","last_peered":"2024-03-06T07:29:52.209124+0000","last_clean":"2024-03-06T07:29:52.209124+0000","last_became_active":"2024-03-06T07:29:50.561715+0000","last_became_peered":"2024-03-06T07:29:50.561715+0000","last_unstale":"2024-03-06T07:29:52.209124+0000","last_undegraded":"2024-03-06T07:29:52.209124+0000","last_fullsized":"2024-03-06T07:29:52.209124+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:17.326685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.032886+0000","last_change":"2024-03-06T07:29:49.557706+0000","last_active":"2024-03-06T07:29:52.032886+0000","last_peered":"2024-03-06T07:29:52.032886+0000","last_clean":"2024-03-06T07:29:52.032886+0000","last_became_active":"2024-03-06T07:29:49.557382+0000","last_became_peered":"2024-03-06T07:29:49.557382+0000","last_unstale":"2024-03-06T07:29:52.032886+0000","last_undegraded":"2024-03-06T07:29:52.032886+0000","last_fullsized":"2024-03-06T07:29:52.032886+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:00.363808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.032853+0000","last_change":"2024-03-06T07:29:50.561244+0000","last_active":"2024-03-06T07:29:52.032853+0000","last_peered":"2024-03-06T07:29:52.032853+0000","last_clean":"2024-03-06T07:29:52.032853+0000","last_became_active":"2024-03-06T07:29:50.558759+0000","last_became_peered":"2024-03-06T07:29:50.558759+0000","last_unstale":"2024-03-06T07:29:52.032853+0000","last_undegraded":"2024-03-06T07:29:52.032853+0000","last_fullsized":"2024-03-06T07:29:52.032853+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:06.131618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208756+0000","last_change":"2024-03-06T07:29:49.570530+0000","last_active":"2024-03-06T07:29:52.208756+0000","last_peered":"2024-03-06T07:29:52.208756+0000","last_clean":"2024-03-06T07:29:52.208756+0000","last_became_active":"2024-03-06T07:29:49.570159+0000","last_became_peered":"2024-03-06T07:29:49.570159+0000","last_unstale":"2024-03-06T07:29:52.208756+0000","last_undegraded":"2024-03-06T07:29:52.208756+0000","last_fullsized":"2024-03-06T07:29:52.208756+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:32.032849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572319+0000","last_change":"2024-03-06T07:29:50.561804+0000","last_active":"2024-03-06T07:29:51.572319+0000","last_peered":"2024-03-06T07:29:51.572319+0000","last_clean":"2024-03-06T07:29:51.572319+0000","last_became_active":"2024-03-06T07:29:50.561543+0000","last_became_peered":"2024-03-06T07:29:50.561543+0000","last_unstale":"2024-03-06T07:29:51.572319+0000","last_undegraded":"2024-03-06T07:29:51.572319+0000","last_fullsized":"2024-03-06T07:29:51.572319+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:41.833195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565660+0000","last_change":"2024-03-06T07:29:49.561264+0000","last_active":"2024-03-06T07:29:51.565660+0000","last_peered":"2024-03-06T07:29:51.565660+0000","last_clean":"2024-03-06T07:29:51.565660+0000","last_became_active":"2024-03-06T07:29:49.560173+0000","last_became_peered":"2024-03-06T07:29:49.560173+0000","last_unstale":"2024-03-06T07:29:51.565660+0000","last_undegraded":"2024-03-06T07:29:51.565660+0000","last_fullsized":"2024-03-06T07:29:51.565660+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:18.017800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206636+0000","last_change":"2024-03-06T07:29:50.579114+0000","last_active":"2024-03-06T07:29:52.206636+0000","last_peered":"2024-03-06T07:29:52.206636+0000","last_clean":"2024-03-06T07:29:52.206636+0000","last_became_active":"2024-03-06T07:29:50.578944+0000","last_became_peered":"2024-03-06T07:29:50.578944+0000","last_unstale":"2024-03-06T07:29:52.206636+0000","last_undegraded":"2024-03-06T07:29:52.206636+0000","last_fullsized":"2024-03-06T07:29:52.206636+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:19.032678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572749+0000","last_change":"2024-03-06T07:29:49.570246+0000","last_active":"2024-03-06T07:29:51.572749+0000","last_peered":"2024-03-06T07:29:51.572749+0000","last_clean":"2024-03-06T07:29:51.572749+0000","last_became_active":"2024-03-06T07:29:49.569956+0000","last_became_peered":"2024-03-06T07:29:49.569956+0000","last_unstale":"2024-03-06T07:29:51.572749+0000","last_undegraded":"2024-03-06T07:29:51.572749+0000","last_fullsized":"2024-03-06T07:29:51.572749+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:46.791258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206184+0000","last_change":"2024-03-06T07:29:50.572063+0000","last_active":"2024-03-06T07:29:52.206184+0000","last_peered":"2024-03-06T07:29:52.206184+0000","last_clean":"2024-03-06T07:29:52.206184+0000","last_became_active":"2024-03-06T07:29:50.571455+0000","last_became_peered":"2024-03-06T07:29:50.571455+0000","last_unstale":"2024-03-06T07:29:52.206184+0000","last_undegraded":"2024-03-06T07:29:52.206184+0000","last_fullsized":"2024-03-06T07:29:52.206184+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:33.047728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207224+0000","last_change":"2024-03-06T07:29:49.554790+0000","last_active":"2024-03-06T07:29:52.207224+0000","last_peered":"2024-03-06T07:29:52.207224+0000","last_clean":"2024-03-06T07:29:52.207224+0000","last_became_active":"2024-03-06T07:29:49.554354+0000","last_became_peered":"2024-03-06T07:29:49.554354+0000","last_unstale":"2024-03-06T07:29:52.207224+0000","last_undegraded":"2024-03-06T07:29:52.207224+0000","last_fullsized":"2024-03-06T07:29:52.207224+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:24.897038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565626+0000","last_change":"2024-03-06T07:29:50.552277+0000","last_active":"2024-03-06T07:29:51.565626+0000","last_peered":"2024-03-06T07:29:51.565626+0000","last_clean":"2024-03-06T07:29:51.565626+0000","last_became_active":"2024-03-06T07:29:50.551948+0000","last_became_peered":"2024-03-06T07:29:50.551948+0000","last_unstale":"2024-03-06T07:29:51.565626+0000","last_undegraded":"2024-03-06T07:29:51.565626+0000","last_fullsized":"2024-03-06T07:29:51.565626+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:05.897228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206494+0000","last_change":"2024-03-06T07:29:49.553294+0000","last_active":"2024-03-06T07:29:52.206494+0000","last_peered":"2024-03-06T07:29:52.206494+0000","last_clean":"2024-03-06T07:29:52.206494+0000","last_became_active":"2024-03-06T07:29:49.552937+0000","last_became_peered":"2024-03-06T07:29:49.552937+0000","last_unstale":"2024-03-06T07:29:52.206494+0000","last_undegraded":"2024-03-06T07:29:52.206494+0000","last_fullsized":"2024-03-06T07:29:52.206494+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:46.791258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.573119+0000","last_change":"2024-03-06T07:29:50.581518+0000","last_active":"2024-03-06T07:29:51.573119+0000","last_peered":"2024-03-06T07:29:51.573119+0000","last_clean":"2024-03-06T07:29:51.573119+0000","last_became_active":"2024-03-06T07:29:50.581163+0000","last_became_peered":"2024-03-06T07:29:50.581163+0000","last_unstale":"2024-03-06T07:29:51.573119+0000","last_undegraded":"2024-03-06T07:29:51.573119+0000","last_fullsized":"2024-03-06T07:29:51.573119+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:56.015309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572958+0000","last_change":"2024-03-06T07:29:49.562050+0000","last_active":"2024-03-06T07:29:51.572958+0000","last_peered":"2024-03-06T07:29:51.572958+0000","last_clean":"2024-03-06T07:29:51.572958+0000","last_became_active":"2024-03-06T07:29:49.561569+0000","last_became_peered":"2024-03-06T07:29:49.561569+0000","last_unstale":"2024-03-06T07:29:51.572958+0000","last_undegraded":"2024-03-06T07:29:51.572958+0000","last_fullsized":"2024-03-06T07:29:51.572958+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:24.458577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208979+0000","last_change":"2024-03-06T07:29:50.558748+0000","last_active":"2024-03-06T07:29:52.208979+0000","last_peered":"2024-03-06T07:29:52.208979+0000","last_clean":"2024-03-06T07:29:52.208979+0000","last_became_active":"2024-03-06T07:29:50.555799+0000","last_became_peered":"2024-03-06T07:29:50.555799+0000","last_unstale":"2024-03-06T07:29:52.208979+0000","last_undegraded":"2024-03-06T07:29:52.208979+0000","last_fullsized":"2024-03-06T07:29:52.208979+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:48.997693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207585+0000","last_change":"2024-03-06T07:29:49.560300+0000","last_active":"2024-03-06T07:29:52.207585+0000","last_peered":"2024-03-06T07:29:52.207585+0000","last_clean":"2024-03-06T07:29:52.207585+0000","last_became_active":"2024-03-06T07:29:49.560000+0000","last_became_peered":"2024-03-06T07:29:49.560000+0000","last_unstale":"2024-03-06T07:29:52.207585+0000","last_undegraded":"2024-03-06T07:29:52.207585+0000","last_fullsized":"2024-03-06T07:29:52.207585+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:01.591115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566019+0000","last_change":"2024-03-06T07:29:50.566187+0000","last_active":"2024-03-06T07:29:51.566019+0000","last_peered":"2024-03-06T07:29:51.566019+0000","last_clean":"2024-03-06T07:29:51.566019+0000","last_became_active":"2024-03-06T07:29:50.565871+0000","last_became_peered":"2024-03-06T07:29:50.565871+0000","last_unstale":"2024-03-06T07:29:51.566019+0000","last_undegraded":"2024-03-06T07:29:51.566019+0000","last_fullsized":"2024-03-06T07:29:51.566019+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:33.508587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208636+0000","last_change":"2024-03-06T07:29:49.569063+0000","last_active":"2024-03-06T07:29:52.208636+0000","last_peered":"2024-03-06T07:29:52.208636+0000","last_clean":"2024-03-06T07:29:52.208636+0000","last_became_active":"2024-03-06T07:29:49.568697+0000","last_became_peered":"2024-03-06T07:29:49.568697+0000","last_unstale":"2024-03-06T07:29:52.208636+0000","last_undegraded":"2024-03-06T07:29:52.208636+0000","last_fullsized":"2024-03-06T07:29:52.208636+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:16.521834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208298+0000","last_change":"2024-03-06T07:29:50.570736+0000","last_active":"2024-03-06T07:29:52.208298+0000","last_peered":"2024-03-06T07:29:52.208298+0000","last_clean":"2024-03-06T07:29:52.208298+0000","last_became_active":"2024-03-06T07:29:50.570470+0000","last_became_peered":"2024-03-06T07:29:50.570470+0000","last_unstale":"2024-03-06T07:29:52.208298+0000","last_undegraded":"2024-03-06T07:29:52.208298+0000","last_fullsized":"2024-03-06T07:29:52.208298+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:35.908558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566625+0000","last_change":"2024-03-06T07:29:49.559466+0000","last_active":"2024-03-06T07:29:51.566625+0000","last_peered":"2024-03-06T07:29:51.566625+0000","last_clean":"2024-03-06T07:29:51.566625+0000","last_became_active":"2024-03-06T07:29:49.559183+0000","last_became_peered":"2024-03-06T07:29:49.559183+0000","last_unstale":"2024-03-06T07:29:51.566625+0000","last_undegraded":"2024-03-06T07:29:51.566625+0000","last_fullsized":"2024-03-06T07:29:51.566625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:11.992670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565667+0000","last_change":"2024-03-06T07:29:50.576050+0000","last_active":"2024-03-06T07:29:51.565667+0000","last_peered":"2024-03-06T07:29:51.565667+0000","last_clean":"2024-03-06T07:29:51.565667+0000","last_became_active":"2024-03-06T07:29:50.575548+0000","last_became_peered":"2024-03-06T07:29:50.575548+0000","last_unstale":"2024-03-06T07:29:51.565667+0000","last_undegraded":"2024-03-06T07:29:51.565667+0000","last_fullsized":"2024-03-06T07:29:51.565667+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:52.342437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567769+0000","last_change":"2024-03-06T07:29:49.563163+0000","last_active":"2024-03-06T07:29:51.567769+0000","last_peered":"2024-03-06T07:29:51.567769+0000","last_clean":"2024-03-06T07:29:51.567769+0000","last_became_active":"2024-03-06T07:29:49.562849+0000","last_became_peered":"2024-03-06T07:29:49.562849+0000","last_unstale":"2024-03-06T07:29:51.567769+0000","last_undegraded":"2024-03-06T07:29:51.567769+0000","last_fullsized":"2024-03-06T07:29:51.567769+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:29.453665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206837+0000","last_change":"2024-03-06T07:29:50.569109+0000","last_active":"2024-03-06T07:29:52.206837+0000","last_peered":"2024-03-06T07:29:52.206837+0000","last_clean":"2024-03-06T07:29:52.206837+0000","last_became_active":"2024-03-06T07:29:50.568249+0000","last_became_peered":"2024-03-06T07:29:50.568249+0000","last_unstale":"2024-03-06T07:29:52.206837+0000","last_undegraded":"2024-03-06T07:29:52.206837+0000","last_fullsized":"2024-03-06T07:29:52.206837+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:30.468543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207603+0000","last_change":"2024-03-06T07:29:49.564969+0000","last_active":"2024-03-06T07:29:52.207603+0000","last_peered":"2024-03-06T07:29:52.207603+0000","last_clean":"2024-03-06T07:29:52.207603+0000","last_became_active":"2024-03-06T07:29:49.564628+0000","last_became_peered":"2024-03-06T07:29:49.564628+0000","last_unstale":"2024-03-06T07:29:52.207603+0000","last_undegraded":"2024-03-06T07:29:52.207603+0000","last_fullsized":"2024-03-06T07:29:52.207603+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:16.101827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.564655+0000","last_change":"2024-03-06T07:29:50.564039+0000","last_active":"2024-03-06T07:29:51.564655+0000","last_peered":"2024-03-06T07:29:51.564655+0000","last_clean":"2024-03-06T07:29:51.564655+0000","last_became_active":"2024-03-06T07:29:50.563026+0000","last_became_peered":"2024-03-06T07:29:50.563026+0000","last_unstale":"2024-03-06T07:29:51.564655+0000","last_undegraded":"2024-03-06T07:29:51.564655+0000","last_fullsized":"2024-03-06T07:29:51.564655+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:46.738548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565404+0000","last_change":"2024-03-06T07:29:49.563135+0000","last_active":"2024-03-06T07:29:51.565404+0000","last_peered":"2024-03-06T07:29:51.565404+0000","last_clean":"2024-03-06T07:29:51.565404+0000","last_became_active":"2024-03-06T07:29:49.562808+0000","last_became_peered":"2024-03-06T07:29:49.562808+0000","last_unstale":"2024-03-06T07:29:51.565404+0000","last_undegraded":"2024-03-06T07:29:51.565404+0000","last_fullsized":"2024-03-06T07:29:51.565404+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:16.521834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208304+0000","last_change":"2024-03-06T07:29:50.557902+0000","last_active":"2024-03-06T07:29:52.208304+0000","last_peered":"2024-03-06T07:29:52.208304+0000","last_clean":"2024-03-06T07:29:52.208304+0000","last_became_active":"2024-03-06T07:29:50.557578+0000","last_became_peered":"2024-03-06T07:29:50.557578+0000","last_unstale":"2024-03-06T07:29:52.208304+0000","last_undegraded":"2024-03-06T07:29:52.208304+0000","last_fullsized":"2024-03-06T07:29:52.208304+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:27.805322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":22,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207044+0000","last_change":"2024-03-06T07:29:49.563265+0000","last_active":"2024-03-06T07:29:52.207044+0000","last_peered":"2024-03-06T07:29:52.207044+0000","last_clean":"2024-03-06T07:29:52.207044+0000","last_became_active":"2024-03-06T07:29:49.562867+0000","last_became_peered":"2024-03-06T07:29:49.562867+0000","last_unstale":"2024-03-06T07:29:52.207044+0000","last_undegraded":"2024-03-06T07:29:52.207044+0000","last_fullsized":"2024-03-06T07:29:52.207044+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:17.252307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567103+0000","last_change":"2024-03-06T07:29:50.567319+0000","last_active":"2024-03-06T07:29:51.567103+0000","last_peered":"2024-03-06T07:29:51.567103+0000","last_clean":"2024-03-06T07:29:51.567103+0000","last_became_active":"2024-03-06T07:29:50.566765+0000","last_became_peered":"2024-03-06T07:29:50.566765+0000","last_unstale":"2024-03-06T07:29:51.567103+0000","last_undegraded":"2024-03-06T07:29:51.567103+0000","last_fullsized":"2024-03-06T07:29:51.567103+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:23.628615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567007+0000","last_change":"2024-03-06T07:29:49.563172+0000","last_active":"2024-03-06T07:29:51.567007+0000","last_peered":"2024-03-06T07:29:51.567007+0000","last_clean":"2024-03-06T07:29:51.567007+0000","last_became_active":"2024-03-06T07:29:49.562881+0000","last_became_peered":"2024-03-06T07:29:49.562881+0000","last_unstale":"2024-03-06T07:29:51.567007+0000","last_undegraded":"2024-03-06T07:29:51.567007+0000","last_fullsized":"2024-03-06T07:29:51.567007+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:51.945499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207612+0000","last_change":"2024-03-06T07:29:50.572340+0000","last_active":"2024-03-06T07:29:52.207612+0000","last_peered":"2024-03-06T07:29:52.207612+0000","last_clean":"2024-03-06T07:29:52.207612+0000","last_became_active":"2024-03-06T07:29:50.572024+0000","last_became_peered":"2024-03-06T07:29:50.572024+0000","last_unstale":"2024-03-06T07:29:52.207612+0000","last_undegraded":"2024-03-06T07:29:52.207612+0000","last_fullsized":"2024-03-06T07:29:52.207612+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:35.908558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.573264+0000","last_change":"2024-03-06T07:29:49.552935+0000","last_active":"2024-03-06T07:29:51.573264+0000","last_peered":"2024-03-06T07:29:51.573264+0000","last_clean":"2024-03-06T07:29:51.573264+0000","last_became_active":"2024-03-06T07:29:49.552616+0000","last_became_peered":"2024-03-06T07:29:49.552616+0000","last_unstale":"2024-03-06T07:29:51.573264+0000","last_undegraded":"2024-03-06T07:29:51.573264+0000","last_fullsized":"2024-03-06T07:29:51.573264+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:00.363808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566423+0000","last_change":"2024-03-06T07:29:50.548527+0000","last_active":"2024-03-06T07:29:51.566423+0000","last_peered":"2024-03-06T07:29:51.566423+0000","last_clean":"2024-03-06T07:29:51.566423+0000","last_became_active":"2024-03-06T07:29:50.546667+0000","last_became_peered":"2024-03-06T07:29:50.546667+0000","last_unstale":"2024-03-06T07:29:51.566423+0000","last_undegraded":"2024-03-06T07:29:51.566423+0000","last_fullsized":"2024-03-06T07:29:51.566423+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:48.398258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207719+0000","last_change":"2024-03-06T07:29:49.555548+0000","last_active":"2024-03-06T07:29:52.207719+0000","last_peered":"2024-03-06T07:29:52.207719+0000","last_clean":"2024-03-06T07:29:52.207719+0000","last_became_active":"2024-03-06T07:29:49.555189+0000","last_became_peered":"2024-03-06T07:29:49.555189+0000","last_unstale":"2024-03-06T07:29:52.207719+0000","last_undegraded":"2024-03-06T07:29:52.207719+0000","last_fullsized":"2024-03-06T07:29:52.207719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:40.818317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565636+0000","last_change":"2024-03-06T07:29:50.561979+0000","last_active":"2024-03-06T07:29:51.565636+0000","last_peered":"2024-03-06T07:29:51.565636+0000","last_clean":"2024-03-06T07:29:51.565636+0000","last_became_active":"2024-03-06T07:29:50.561653+0000","last_became_peered":"2024-03-06T07:29:50.561653+0000","last_unstale":"2024-03-06T07:29:51.565636+0000","last_undegraded":"2024-03-06T07:29:51.565636+0000","last_fullsized":"2024-03-06T07:29:51.565636+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:18.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566113+0000","last_change":"2024-03-06T07:29:49.565741+0000","last_active":"2024-03-06T07:29:51.566113+0000","last_peered":"2024-03-06T07:29:51.566113+0000","last_clean":"2024-03-06T07:29:51.566113+0000","last_became_active":"2024-03-06T07:29:49.565445+0000","last_became_peered":"2024-03-06T07:29:49.565445+0000","last_unstale":"2024-03-06T07:29:51.566113+0000","last_undegraded":"2024-03-06T07:29:51.566113+0000","last_fullsized":"2024-03-06T07:29:51.566113+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:55.000431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565087+0000","last_change":"2024-03-06T07:29:50.556044+0000","last_active":"2024-03-06T07:29:51.565087+0000","last_peered":"2024-03-06T07:29:51.565087+0000","last_clean":"2024-03-06T07:29:51.565087+0000","last_became_active":"2024-03-06T07:29:50.555540+0000","last_became_peered":"2024-03-06T07:29:50.555540+0000","last_unstale":"2024-03-06T07:29:51.565087+0000","last_undegraded":"2024-03-06T07:29:51.565087+0000","last_fullsized":"2024-03-06T07:29:51.565087+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:00.859543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.566004+0000","last_change":"2024-03-06T07:29:49.561263+0000","last_active":"2024-03-06T07:29:51.566004+0000","last_peered":"2024-03-06T07:29:51.566004+0000","last_clean":"2024-03-06T07:29:51.566004+0000","last_became_active":"2024-03-06T07:29:49.559687+0000","last_became_peered":"2024-03-06T07:29:49.559687+0000","last_unstale":"2024-03-06T07:29:51.566004+0000","last_undegraded":"2024-03-06T07:29:51.566004+0000","last_fullsized":"2024-03-06T07:29:51.566004+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:23.345829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.032347+0000","last_change":"2024-03-06T07:29:50.572590+0000","last_active":"2024-03-06T07:29:52.032347+0000","last_peered":"2024-03-06T07:29:52.032347+0000","last_clean":"2024-03-06T07:29:52.032347+0000","last_became_active":"2024-03-06T07:29:50.570186+0000","last_became_peered":"2024-03-06T07:29:50.570186+0000","last_unstale":"2024-03-06T07:29:52.032347+0000","last_undegraded":"2024-03-06T07:29:52.032347+0000","last_fullsized":"2024-03-06T07:29:52.032347+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:18.342484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207380+0000","last_change":"2024-03-06T07:29:49.568087+0000","last_active":"2024-03-06T07:29:52.207380+0000","last_peered":"2024-03-06T07:29:52.207380+0000","last_clean":"2024-03-06T07:29:52.207380+0000","last_became_active":"2024-03-06T07:29:49.567388+0000","last_became_peered":"2024-03-06T07:29:49.567388+0000","last_unstale":"2024-03-06T07:29:52.207380+0000","last_undegraded":"2024-03-06T07:29:52.207380+0000","last_fullsized":"2024-03-06T07:29:52.207380+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:08.560279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565834+0000","last_change":"2024-03-06T07:29:50.554466+0000","last_active":"2024-03-06T07:29:51.565834+0000","last_peered":"2024-03-06T07:29:51.565834+0000","last_clean":"2024-03-06T07:29:51.565834+0000","last_became_active":"2024-03-06T07:29:50.553783+0000","last_became_peered":"2024-03-06T07:29:50.553783+0000","last_unstale":"2024-03-06T07:29:51.565834+0000","last_undegraded":"2024-03-06T07:29:51.565834+0000","last_fullsized":"2024-03-06T07:29:51.565834+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:17.862236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.205951+0000","last_change":"2024-03-06T07:29:49.559887+0000","last_active":"2024-03-06T07:29:52.205951+0000","last_peered":"2024-03-06T07:29:52.205951+0000","last_clean":"2024-03-06T07:29:52.205951+0000","last_became_active":"2024-03-06T07:29:49.559549+0000","last_became_peered":"2024-03-06T07:29:49.559549+0000","last_unstale":"2024-03-06T07:29:52.205951+0000","last_undegraded":"2024-03-06T07:29:52.205951+0000","last_fullsized":"2024-03-06T07:29:52.205951+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:11.726980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206084+0000","last_change":"2024-03-06T07:29:50.575818+0000","last_active":"2024-03-06T07:29:52.206084+0000","last_peered":"2024-03-06T07:29:52.206084+0000","last_clean":"2024-03-06T07:29:52.206084+0000","last_became_active":"2024-03-06T07:29:50.575501+0000","last_became_peered":"2024-03-06T07:29:50.575501+0000","last_unstale":"2024-03-06T07:29:52.206084+0000","last_undegraded":"2024-03-06T07:29:52.206084+0000","last_fullsized":"2024-03-06T07:29:52.206084+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:17.116705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206509+0000","last_change":"2024-03-06T07:29:49.563648+0000","last_active":"2024-03-06T07:29:52.206509+0000","last_peered":"2024-03-06T07:29:52.206509+0000","last_clean":"2024-03-06T07:29:52.206509+0000","last_became_active":"2024-03-06T07:29:49.563328+0000","last_became_peered":"2024-03-06T07:29:49.563328+0000","last_unstale":"2024-03-06T07:29:52.206509+0000","last_undegraded":"2024-03-06T07:29:52.206509+0000","last_fullsized":"2024-03-06T07:29:52.206509+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:24.897038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.567361+0000","last_change":"2024-03-06T07:29:50.575086+0000","last_active":"2024-03-06T07:29:51.567361+0000","last_peered":"2024-03-06T07:29:51.567361+0000","last_clean":"2024-03-06T07:29:51.567361+0000","last_became_active":"2024-03-06T07:29:50.574807+0000","last_became_peered":"2024-03-06T07:29:50.574807+0000","last_unstale":"2024-03-06T07:29:51.567361+0000","last_undegraded":"2024-03-06T07:29:51.567361+0000","last_fullsized":"2024-03-06T07:29:51.567361+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:25.911917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"44'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.570560+0000","last_change":"2024-03-06T07:29:49.563242+0000","last_active":"2024-03-06T07:29:51.570560+0000","last_peered":"2024-03-06T07:29:51.570560+0000","last_clean":"2024-03-06T07:29:51.570560+0000","last_became_active":"2024-03-06T07:29:49.562890+0000","last_became_peered":"2024-03-06T07:29:49.562890+0000","last_unstale":"2024-03-06T07:29:51.570560+0000","last_undegraded":"2024-03-06T07:29:51.570560+0000","last_fullsized":"2024-03-06T07:29:51.570560+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:43.910801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207338+0000","last_change":"2024-03-06T07:29:50.563822+0000","last_active":"2024-03-06T07:29:52.207338+0000","last_peered":"2024-03-06T07:29:52.207338+0000","last_clean":"2024-03-06T07:29:52.207338+0000","last_became_active":"2024-03-06T07:29:50.563542+0000","last_became_peered":"2024-03-06T07:29:50.563542+0000","last_unstale":"2024-03-06T07:29:52.207338+0000","last_undegraded":"2024-03-06T07:29:52.207338+0000","last_fullsized":"2024-03-06T07:29:52.207338+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:04.580321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208653+0000","last_change":"2024-03-06T07:29:49.569394+0000","last_active":"2024-03-06T07:29:52.208653+0000","last_peered":"2024-03-06T07:29:52.208653+0000","last_clean":"2024-03-06T07:29:52.208653+0000","last_became_active":"2024-03-06T07:29:49.569085+0000","last_became_peered":"2024-03-06T07:29:49.569085+0000","last_unstale":"2024-03-06T07:29:52.208653+0000","last_undegraded":"2024-03-06T07:29:52.208653+0000","last_fullsized":"2024-03-06T07:29:52.208653+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:55.000431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.208507+0000","last_change":"2024-03-06T07:29:50.558071+0000","last_active":"2024-03-06T07:29:52.208507+0000","last_peered":"2024-03-06T07:29:52.208507+0000","last_clean":"2024-03-06T07:29:52.208507+0000","last_became_active":"2024-03-06T07:29:50.557710+0000","last_became_peered":"2024-03-06T07:29:50.557710+0000","last_unstale":"2024-03-06T07:29:52.208507+0000","last_undegraded":"2024-03-06T07:29:52.208507+0000","last_fullsized":"2024-03-06T07:29:52.208507+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:10.781950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.032317+0000","last_change":"2024-03-06T07:29:49.557882+0000","last_active":"2024-03-06T07:29:52.032317+0000","last_peered":"2024-03-06T07:29:52.032317+0000","last_clean":"2024-03-06T07:29:52.032317+0000","last_became_active":"2024-03-06T07:29:49.557589+0000","last_became_peered":"2024-03-06T07:29:49.557589+0000","last_unstale":"2024-03-06T07:29:52.032317+0000","last_undegraded":"2024-03-06T07:29:52.032317+0000","last_fullsized":"2024-03-06T07:29:52.032317+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:17.864810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565242+0000","last_change":"2024-03-06T07:29:50.570487+0000","last_active":"2024-03-06T07:29:51.565242+0000","last_peered":"2024-03-06T07:29:51.565242+0000","last_clean":"2024-03-06T07:29:51.565242+0000","last_became_active":"2024-03-06T07:29:50.569326+0000","last_became_peered":"2024-03-06T07:29:50.569326+0000","last_unstale":"2024-03-06T07:29:51.565242+0000","last_undegraded":"2024-03-06T07:29:51.565242+0000","last_fullsized":"2024-03-06T07:29:51.565242+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:16.101419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207921+0000","last_change":"2024-03-06T07:29:49.567854+0000","last_active":"2024-03-06T07:29:52.207921+0000","last_peered":"2024-03-06T07:29:52.207921+0000","last_clean":"2024-03-06T07:29:52.207921+0000","last_became_active":"2024-03-06T07:29:49.567457+0000","last_became_peered":"2024-03-06T07:29:49.567457+0000","last_unstale":"2024-03-06T07:29:52.207921+0000","last_undegraded":"2024-03-06T07:29:52.207921+0000","last_fullsized":"2024-03-06T07:29:52.207921+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:43.910801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.565556+0000","last_change":"2024-03-06T07:29:50.567599+0000","last_active":"2024-03-06T07:29:51.565556+0000","last_peered":"2024-03-06T07:29:51.565556+0000","last_clean":"2024-03-06T07:29:51.565556+0000","last_became_active":"2024-03-06T07:29:50.566983+0000","last_became_peered":"2024-03-06T07:29:50.566983+0000","last_unstale":"2024-03-06T07:29:51.565556+0000","last_undegraded":"2024-03-06T07:29:51.565556+0000","last_fullsized":"2024-03-06T07:29:51.565556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:32.481790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572715+0000","last_change":"2024-03-06T07:29:49.563568+0000","last_active":"2024-03-06T07:29:51.572715+0000","last_peered":"2024-03-06T07:29:51.572715+0000","last_clean":"2024-03-06T07:29:51.572715+0000","last_became_active":"2024-03-06T07:29:49.563250+0000","last_became_peered":"2024-03-06T07:29:49.563250+0000","last_unstale":"2024-03-06T07:29:51.572715+0000","last_undegraded":"2024-03-06T07:29:51.572715+0000","last_fullsized":"2024-03-06T07:29:51.572715+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:01.591115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207134+0000","last_change":"2024-03-06T07:29:50.559010+0000","last_active":"2024-03-06T07:29:52.207134+0000","last_peered":"2024-03-06T07:29:52.207134+0000","last_clean":"2024-03-06T07:29:52.207134+0000","last_became_active":"2024-03-06T07:29:50.555394+0000","last_became_peered":"2024-03-06T07:29:50.555394+0000","last_unstale":"2024-03-06T07:29:52.207134+0000","last_undegraded":"2024-03-06T07:29:52.207134+0000","last_fullsized":"2024-03-06T07:29:52.207134+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:30.468543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.032638+0000","last_change":"2024-03-06T07:29:49.565696+0000","last_active":"2024-03-06T07:29:52.032638+0000","last_peered":"2024-03-06T07:29:52.032638+0000","last_clean":"2024-03-06T07:29:52.032638+0000","last_became_active":"2024-03-06T07:29:49.565414+0000","last_became_peered":"2024-03-06T07:29:49.565414+0000","last_unstale":"2024-03-06T07:29:52.032638+0000","last_undegraded":"2024-03-06T07:29:52.032638+0000","last_fullsized":"2024-03-06T07:29:52.032638+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:22.643749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:51.572316+0000","last_change":"2024-03-06T07:29:50.555956+0000","last_active":"2024-03-06T07:29:51.572316+0000","last_peered":"2024-03-06T07:29:51.572316+0000","last_clean":"2024-03-06T07:29:51.572316+0000","last_became_active":"2024-03-06T07:29:50.554170+0000","last_became_peered":"2024-03-06T07:29:50.554170+0000","last_unstale":"2024-03-06T07:29:51.572316+0000","last_undegraded":"2024-03-06T07:29:51.572316+0000","last_fullsized":"2024-03-06T07:29:51.572316+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:25.911917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206792+0000","last_change":"2024-03-06T07:29:49.564805+0000","last_active":"2024-03-06T07:29:52.206792+0000","last_peered":"2024-03-06T07:29:52.206792+0000","last_clean":"2024-03-06T07:29:52.206792+0000","last_became_active":"2024-03-06T07:29:49.564533+0000","last_became_peered":"2024-03-06T07:29:49.564533+0000","last_unstale":"2024-03-06T07:29:52.206792+0000","last_undegraded":"2024-03-06T07:29:52.206792+0000","last_fullsized":"2024-03-06T07:29:52.206792+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:41.957223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207554+0000","last_change":"2024-03-06T07:29:50.560511+0000","last_active":"2024-03-06T07:29:52.207554+0000","last_peered":"2024-03-06T07:29:52.207554+0000","last_clean":"2024-03-06T07:29:52.207554+0000","last_became_active":"2024-03-06T07:29:50.560177+0000","last_became_peered":"2024-03-06T07:29:50.560177+0000","last_unstale":"2024-03-06T07:29:52.207554+0000","last_undegraded":"2024-03-06T07:29:52.207554+0000","last_fullsized":"2024-03-06T07:29:52.207554+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:49.516837+0000","last_clean_scrub_stamp":"2024-03-06T07:29:49.516837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:27.805322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.207980+0000","last_change":"2024-03-06T07:29:49.565525+0000","last_active":"2024-03-06T07:29:52.207980+0000","last_peered":"2024-03-06T07:29:52.207980+0000","last_clean":"2024-03-06T07:29:52.207980+0000","last_became_active":"2024-03-06T07:29:49.560896+0000","last_became_peered":"2024-03-06T07:29:49.560896+0000","last_unstale":"2024-03-06T07:29:52.207980+0000","last_undegraded":"2024-03-06T07:29:52.207980+0000","last_fullsized":"2024-03-06T07:29:52.207980+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:48.501959+0000","last_clean_scrub_stamp":"2024-03-06T07:29:48.501959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:47.383380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:52.206424+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:29:52.206424+0000","last_peered":"2024-03-06T07:29:52.206424+0000","last_clean":"2024-03-06T07:29:52.206424+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:29:52.206424+0000","last_undegraded":"2024-03-06T07:29:52.206424+0000","last_fullsized":"2024-03-06T07:29:52.206424+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1114112,"data_stored":800428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":210642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":210040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":211582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":210608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":211604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":210040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1114112,"data_stored":800372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:55.498 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:29:55.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:29:55.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.749+0000 7ffbed600640 1 Processor -- start 2024-03-06T07:29:55.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.749+0000 7ffbed600640 1 -- start start 2024-03-06T07:29:55.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.751+0000 7ffbed600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.751+0000 7ffbed600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffbe815e3e0 0x7ffbe80d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.751+0000 7ffbed600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe80d1260 con 0x7ffbe815e000 2024-03-06T07:29:55.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.751+0000 7ffbed600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe80d1020 con 0x7ffbe815e3e0 2024-03-06T07:29:55.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.751+0000 7ffbed600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe806c0d0 con 0x7ffbe806fef0 2024-03-06T07:29:55.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.752+0000 7ffbdfe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.752+0000 7ffbe7400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffbe815e3e0 0x7ffbe80d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.752+0000 7ffbdfe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe806f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:56956/0 (socket says 172.21.15.114:56956) 2024-03-06T07:29:55.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.752+0000 7ffbdfe00640 1 -- 172.21.15.114:0/940572092 learned_addr learned my addr 172.21.15.114:0/940572092 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:55.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.753+0000 7ffbdfe00640 1 -- 172.21.15.114:0/940572092 >> v2:172.21.15.177:3300/0 conn(0x7ffbe815e3e0 msgr2=0x7ffbe80d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.753+0000 7ffbdfe00640 1 --2- 172.21.15.114:0/940572092 >> v2:172.21.15.177:3300/0 conn(0x7ffbe815e3e0 0x7ffbe80d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.753+0000 7ffbdfe00640 1 -- 172.21.15.114:0/940572092 >> v1:172.21.15.177:6789/0 conn(0x7ffbe815e000 legacy=0x7ffbe806f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.753+0000 7ffbdfe00640 1 -- 172.21.15.114:0/940572092 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe806c320 con 0x7ffbe806fef0 2024-03-06T07:29:55.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.755+0000 7ffbdfe00640 1 --2- 172.21.15.114:0/940572092 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe806f130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00098a0 tx=0x7ffbd002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba0cd0f856bfc09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.755+0000 7ffbe6000640 1 -- 172.21.15.114:0/940572092 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0034070 con 0x7ffbe806fef0 2024-03-06T07:29:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.755+0000 7ffbe6000640 1 -- 172.21.15.114:0/940572092 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd0030840 con 0x7ffbe806fef0 2024-03-06T07:29:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.755+0000 7ffbe6000640 1 -- 172.21.15.114:0/940572092 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0030c10 con 0x7ffbe806fef0 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 -- 172.21.15.114:0/940572092 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 msgr2=0x7ffbe806f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 --2- 172.21.15.114:0/940572092 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe806f130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00098a0 tx=0x7ffbd002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 -- 172.21.15.114:0/940572092 shutdown_connections 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 --2- 172.21.15.114:0/940572092 >> v2:172.21.15.177:3300/0 conn(0x7ffbe815e3e0 0x7ffbe80d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 --2- 172.21.15.114:0/940572092 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe806f130 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 -- 172.21.15.114:0/940572092 >> 172.21.15.114:0/940572092 conn(0x7ffbe80b3800 msgr2=0x7ffbe80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 -- 172.21.15.114:0/940572092 shutdown_connections 2024-03-06T07:29:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.756+0000 7ffbed600640 1 -- 172.21.15.114:0/940572092 wait complete. 2024-03-06T07:29:55.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.758+0000 7ffbed600640 1 Processor -- start 2024-03-06T07:29:55.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.758+0000 7ffbed600640 1 -- start start 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.759+0000 7ffbed600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe8152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.759+0000 7ffbed600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe815e000 0x7ffbe80c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.759+0000 7ffbed600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbe815e3e0 0x7ffbe80b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.759+0000 7ffbed600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80ce4c0 con 0x7ffbe815e3e0 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.759+0000 7ffbed600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80ce340 con 0x7ffbe815e000 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.759+0000 7ffbed600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80ce640 con 0x7ffbe806fef0 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbe815e3e0 0x7ffbe80b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbe815e3e0 0x7ffbe80b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48984/0 (socket says 172.21.15.114:48984) 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 -- 172.21.15.114:0/2340079653 learned_addr learned my addr 172.21.15.114:0/2340079653 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe6a00640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe8152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbdfe00640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe815e000 0x7ffbe80c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 -- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 msgr2=0x7ffbe8152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe8152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 -- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe815e000 msgr2=0x7ffbe80c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe815e000 0x7ffbe80c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.760+0000 7ffbe7400640 1 -- 172.21.15.114:0/2340079653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe80b4d00 con 0x7ffbe815e3e0 2024-03-06T07:29:55.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.761+0000 7ffbe7400640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbe815e3e0 0x7ffbe80b01b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ffbe000efc0 tx=0x7ffbe000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.761+0000 7ffbdea00640 1 -- 172.21.15.114:0/2340079653 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0018020 con 0x7ffbe815e3e0 2024-03-06T07:29:55.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.761+0000 7ffbdea00640 1 -- 172.21.15.114:0/2340079653 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe0014480 con 0x7ffbe815e3e0 2024-03-06T07:29:55.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.761+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe80c7840 con 0x7ffbe815e3e0 2024-03-06T07:29:55.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.761+0000 7ffbdea00640 1 -- 172.21.15.114:0/2340079653 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0014870 con 0x7ffbe815e3e0 2024-03-06T07:29:55.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.761+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe80c7560 con 0x7ffbe815e3e0 2024-03-06T07:29:55.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.762+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe804f330 con 0x7ffbe815e3e0 2024-03-06T07:29:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.763+0000 7ffbdea00640 1 -- 172.21.15.114:0/2340079653 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbe0005d10 con 0x7ffbe815e3e0 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.764+0000 7ffbdea00640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffbc406b4d0 0x7ffbc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.764+0000 7ffbdea00640 1 -- 172.21.15.114:0/2340079653 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbe008af50 con 0x7ffbe815e3e0 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.764+0000 7ffbe6a00640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffbc406b4d0 0x7ffbc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.765+0000 7ffbe6a00640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffbc406b4d0 0x7ffbc406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8000f60 tx=0x7ffbd8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:55.769+0000 7ffbdea00640 1 -- 172.21.15.114:0/2340079653 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbe0022030 con 0x7ffbe815e3e0 2024-03-06T07:29:56.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.041+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ffbe815d820 con 0x7ffbe815e3e0 2024-03-06T07:29:56.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.045+0000 7ffbdea00640 1 -- 172.21.15.114:0/2340079653 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7ffbe80b4d00 con 0x7ffbe815e3e0 2024-03-06T07:29:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:29:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:29:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:29:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:29:56.047 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:29:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffbc406b4d0 msgr2=0x7ffbc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffbc406b4d0 0x7ffbc406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8000f60 tx=0x7ffbd8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbe815e3e0 msgr2=0x7ffbe80b01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbe815e3e0 0x7ffbe80b01b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ffbe000efc0 tx=0x7ffbe000c490 comp rx=0 tx=0).stop 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 shutdown_connections 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffbc406b4d0 0x7ffbc406d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbe815e3e0 0x7ffbe80b01b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbe815e000 0x7ffbe80c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 --2- 172.21.15.114:0/2340079653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffbe806fef0 0x7ffbe8152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 >> 172.21.15.114:0/2340079653 conn(0x7ffbe80b3800 msgr2=0x7ffbe80d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.048+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 shutdown_connections 2024-03-06T07:29:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.049+0000 7ffbed600640 1 -- 172.21.15.114:0/2340079653 wait complete. 2024-03-06T07:29:56.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:56.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.276+0000 7f7ad1a00640 1 Processor -- start 2024-03-06T07:29:56.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.276+0000 7f7ad1a00640 1 -- start start 2024-03-06T07:29:56.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.277+0000 7f7ad1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc15acb0 0x7f7acc0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.277+0000 7f7ad1a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7acc0b4680 0x7f7acc06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.277+0000 7f7ad1a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc15b090 con 0x7f7acc072340 2024-03-06T07:29:56.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.277+0000 7f7ad1a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc0b2a30 con 0x7f7acc0b4680 2024-03-06T07:29:56.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.277+0000 7f7ad1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0b4360 con 0x7f7acc15acb0 2024-03-06T07:29:56.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.278+0000 7f7acaa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc15acb0 0x7f7acc0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.278+0000 7f7aca000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7acc0b4680 0x7f7acc06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.278+0000 7f7acaa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc15acb0 0x7f7acc0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:56986/0 (socket says 172.21.15.114:56986) 2024-03-06T07:29:56.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.278+0000 7f7acaa00640 1 -- 172.21.15.114:0/1837228132 learned_addr learned my addr 172.21.15.114:0/1837228132 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:56.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.279+0000 7f7acaa00640 1 -- 172.21.15.114:0/1837228132 >> v2:172.21.15.177:3300/0 conn(0x7f7acc0b4680 msgr2=0x7f7acc06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.279+0000 7f7acaa00640 1 --2- 172.21.15.114:0/1837228132 >> v2:172.21.15.177:3300/0 conn(0x7f7acc0b4680 0x7f7acc06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.279+0000 7f7acaa00640 1 -- 172.21.15.114:0/1837228132 >> v1:172.21.15.177:6789/0 conn(0x7f7acc072340 legacy=0x7f7acc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.279+0000 7f7acaa00640 1 -- 172.21.15.114:0/1837228132 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0b2cf0 con 0x7f7acc15acb0 2024-03-06T07:29:56.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.280+0000 7f7acaa00640 1 --2- 172.21.15.114:0/1837228132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc15acb0 0x7f7acc0b37d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009e50 tx=0x7f7abc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45660ce7481b3d8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.280+0000 7f7ac9600640 1 -- 172.21.15.114:0/1837228132 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc02cb00 con 0x7f7acc15acb0 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ac9600640 1 -- 172.21.15.114:0/1837228132 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc02cca0 con 0x7f7acc15acb0 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ac9600640 1 -- 172.21.15.114:0/1837228132 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc034610 con 0x7f7acc15acb0 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 -- 172.21.15.114:0/1837228132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc15acb0 msgr2=0x7f7acc0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/1837228132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc15acb0 0x7f7acc0b37d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009e50 tx=0x7f7abc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 -- 172.21.15.114:0/1837228132 shutdown_connections 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/1837228132 >> v2:172.21.15.177:3300/0 conn(0x7f7acc0b4680 0x7f7acc06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/1837228132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc15acb0 0x7f7acc0b37d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 -- 172.21.15.114:0/1837228132 >> 172.21.15.114:0/1837228132 conn(0x7f7acc0b1730 msgr2=0x7f7acc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 -- 172.21.15.114:0/1837228132 shutdown_connections 2024-03-06T07:29:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.281+0000 7f7ad1a00640 1 -- 172.21.15.114:0/1837228132 wait complete. 2024-03-06T07:29:56.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.283+0000 7f7ad1a00640 1 Processor -- start 2024-03-06T07:29:56.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.283+0000 7f7ad1a00640 1 -- start start 2024-03-06T07:29:56.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.284+0000 7f7ad1a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7acc072340 0x7f7acc145260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.284+0000 7f7ad1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc0b4680 0x7f7acc0c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.284+0000 7f7ad1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7acc15acb0 0x7f7acc06d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.284+0000 7f7ad1a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0b3d10 con 0x7f7acc072340 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.284+0000 7f7ad1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0b32b0 con 0x7f7acc15acb0 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.284+0000 7f7ad1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0b4010 con 0x7f7acc0b4680 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.284+0000 7f7acaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7acc072340 0x7f7acc145260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7acc072340 0x7f7acc145260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48998/0 (socket says 172.21.15.114:48998) 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 -- 172.21.15.114:0/2393216761 learned_addr learned my addr 172.21.15.114:0/2393216761 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7aca000640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc0b4680 0x7f7acc0c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acb400640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7acc15acb0 0x7f7acc06d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 -- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc0b4680 msgr2=0x7f7acc0c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc0b4680 0x7f7acc0c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 -- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7acc15acb0 msgr2=0x7f7acc06d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7acc15acb0 0x7f7acc06d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 -- 172.21.15.114:0/2393216761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0ca0c0 con 0x7f7acc072340 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.285+0000 7f7acaa00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7acc072340 0x7f7acc145260 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7abc000f80 tx=0x7f7abc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.286+0000 7f7ac2a00640 1 -- 172.21.15.114:0/2393216761 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc044020 con 0x7f7acc072340 2024-03-06T07:29:56.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.286+0000 7f7ac2a00640 1 -- 172.21.15.114:0/2393216761 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc02ce50 con 0x7f7acc072340 2024-03-06T07:29:56.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.286+0000 7f7ac2a00640 1 -- 172.21.15.114:0/2393216761 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc009510 con 0x7f7acc072340 2024-03-06T07:29:56.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.286+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc0ca3a0 con 0x7f7acc072340 2024-03-06T07:29:56.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.286+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc0ca960 con 0x7f7acc072340 2024-03-06T07:29:56.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.287+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc04f330 con 0x7f7acc072340 2024-03-06T07:29:56.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.288+0000 7f7ac2a00640 1 -- 172.21.15.114:0/2393216761 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7abc047020 con 0x7f7acc072340 2024-03-06T07:29:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.289+0000 7f7ac2a00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7aac06b4d0 0x7f7aac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.289+0000 7f7ac2a00640 1 -- 172.21.15.114:0/2393216761 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7abc0acb90 con 0x7f7acc072340 2024-03-06T07:29:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.289+0000 7f7aca000640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7aac06b4d0 0x7f7aac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.290+0000 7f7aca000640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7aac06b4d0 0x7f7aac06d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0ce890 tx=0x7f7ab8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.293+0000 7f7ac2a00640 1 -- 172.21.15.114:0/2393216761 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7abc04e030 con 0x7f7acc072340 2024-03-06T07:29:56.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T07:29:56.498+0000 7fa65f800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:56.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.526+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7acc1457a0 con 0x7f7acc072340 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.527+0000 7f7ac2a00640 1 -- 172.21.15.114:0/2393216761 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f7abc07b980 con 0x7f7acc072340 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4721,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/308840931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":308840931},{"type":"v1","addr":"172.21.15.177:6833","nonce":308840931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/4238819696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":4238819696},{"type":"v1","addr":"172.21.15.177:6835","nonce":4238819696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4960,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/379671784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":379671784},{"type":"v1","addr":"172.21.15.114:6835","nonce":379671784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:49.994729+0000","modified":"2024-03-06T07:29:53.062242+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5568":{"gid":5568,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2123564678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":2123564678},{"type":"v1","addr":"172.21.15.114:6837","nonce":2123564678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5568,"qdb_cluster":[5568]},"id":3}]} 2024-03-06T07:29:56.529 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 26 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7aac06b4d0 msgr2=0x7f7aac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7aac06b4d0 0x7f7aac06d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0ce890 tx=0x7f7ab8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7acc072340 msgr2=0x7f7acc145260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7acc072340 0x7f7acc145260 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7abc000f80 tx=0x7f7abc004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 shutdown_connections 2024-03-06T07:29:56.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7acc15acb0 0x7f7acc06d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7acc0b4680 0x7f7acc0c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7aac06b4d0 0x7f7aac06d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 --2- 172.21.15.114:0/2393216761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7acc072340 0x7f7acc145260 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 >> 172.21.15.114:0/2393216761 conn(0x7f7acc0b1730 msgr2=0x7f7acc0cdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 shutdown_connections 2024-03-06T07:29:56.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:56.530+0000 7f7ad1a00640 1 -- 172.21.15.114:0/2393216761 wait complete. 2024-03-06T07:29:56.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:49.994729+0000', 'modified': '2024-03-06T07:29:53.062242+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5568}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5568': {'gid': 5568, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/2123564678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 2123564678}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 2123564678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5568, 'qdb_cluster': [5568]} 2024-03-06T07:29:56.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:56.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:29:57.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.164+0000 7fa745e00640 1 Processor -- start 2024-03-06T07:29:57.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.164+0000 7fa745e00640 1 -- start start 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa745e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa7401444e0 0x7fa7400cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa745e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa7401474c0 0x7fa7400c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa745e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400ae380 con 0x7fa7400c4e70 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa745e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7400ae660 con 0x7fa7401474c0 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa745e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7400ae940 con 0x7fa7401444e0 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa73e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa7401474c0 0x7fa7400c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa73e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa7401474c0 0x7fa7400c4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49006/0 (socket says 172.21.15.114:49006) 2024-03-06T07:29:57.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.166+0000 7fa73e000640 1 -- 172.21.15.114:0/593035283 learned_addr learned my addr 172.21.15.114:0/593035283 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.167+0000 7fa73ea00640 1 --2- 172.21.15.114:0/593035283 >> v2:172.21.15.177:3300/0 conn(0x7fa7401444e0 0x7fa7400cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.167+0000 7fa73e000640 1 -- 172.21.15.114:0/593035283 >> v2:172.21.15.177:3300/0 conn(0x7fa7401444e0 msgr2=0x7fa7400cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.167+0000 7fa73e000640 1 --2- 172.21.15.114:0/593035283 >> v2:172.21.15.177:3300/0 conn(0x7fa7401444e0 0x7fa7400cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.167+0000 7fa73e000640 1 -- 172.21.15.114:0/593035283 >> v1:172.21.15.177:6789/0 conn(0x7fa7400c4e70 legacy=0x7fa740161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.167+0000 7fa73e000640 1 -- 172.21.15.114:0/593035283 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400aec20 con 0x7fa7401474c0 2024-03-06T07:29:57.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.168+0000 7fa73e000640 1 --2- 172.21.15.114:0/593035283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401474c0 0x7fa7400c4930 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa7400af3c0 tx=0x7fa72c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef1fe04c6f26b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa73d600640 1 -- 172.21.15.114:0/593035283 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c035020 con 0x7fa7401474c0 2024-03-06T07:29:57.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa73d600640 1 -- 172.21.15.114:0/593035283 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c031870 con 0x7fa7401474c0 2024-03-06T07:29:57.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa73d600640 1 -- 172.21.15.114:0/593035283 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c031c40 con 0x7fa7401474c0 2024-03-06T07:29:57.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa745e00640 1 -- 172.21.15.114:0/593035283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401474c0 msgr2=0x7fa7400c4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa745e00640 1 --2- 172.21.15.114:0/593035283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401474c0 0x7fa7400c4930 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa7400af3c0 tx=0x7fa72c02b910 comp rx=0 tx=0).stop 2024-03-06T07:29:57.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa745e00640 1 -- 172.21.15.114:0/593035283 shutdown_connections 2024-03-06T07:29:57.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa745e00640 1 --2- 172.21.15.114:0/593035283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401474c0 0x7fa7400c4930 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa745e00640 1 --2- 172.21.15.114:0/593035283 >> v2:172.21.15.177:3300/0 conn(0x7fa7401444e0 0x7fa7400cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.169+0000 7fa745e00640 1 -- 172.21.15.114:0/593035283 >> 172.21.15.114:0/593035283 conn(0x7fa7400cc4f0 msgr2=0x7fa7400cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.170+0000 7fa745e00640 1 -- 172.21.15.114:0/593035283 shutdown_connections 2024-03-06T07:29:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.170+0000 7fa745e00640 1 -- 172.21.15.114:0/593035283 wait complete. 2024-03-06T07:29:57.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.171+0000 7fa745e00640 1 Processor -- start 2024-03-06T07:29:57.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.171+0000 7fa745e00640 1 -- start start 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.172+0000 7fa745e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7400c4e70 0x7fa7401f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.172+0000 7fa745e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401444e0 0x7fa7400afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.172+0000 7fa745e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7401474c0 0x7fa74015dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.172+0000 7fa745e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401448c0 con 0x7fa7401444e0 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.172+0000 7fa745e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740147cb0 con 0x7fa7401474c0 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.172+0000 7fa745e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa740144a40 con 0x7fa7400c4e70 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.172+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401444e0 0x7fa7400afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401444e0 0x7fa7400afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49016/0 (socket says 172.21.15.114:49016) 2024-03-06T07:29:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 -- 172.21.15.114:0/3329900563 learned_addr learned my addr 172.21.15.114:0/3329900563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 -- 172.21.15.114:0/3329900563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7400c4e70 msgr2=0x7fa7401f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7400c4e70 0x7fa7401f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73f400640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7401474c0 0x7fa74015dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 -- 172.21.15.114:0/3329900563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7401474c0 msgr2=0x7fa74015dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7401474c0 0x7fa74015dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 -- 172.21.15.114:0/3329900563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74015e110 con 0x7fa7401444e0 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.173+0000 7fa73e000640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401444e0 0x7fa7400afed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa72c02b950 tx=0x7fa72c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.174+0000 7fa737400640 1 -- 172.21.15.114:0/3329900563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c040070 con 0x7fa7401444e0 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.174+0000 7fa737400640 1 -- 172.21.15.114:0/3329900563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c030d00 con 0x7fa7401444e0 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.174+0000 7fa737400640 1 -- 172.21.15.114:0/3329900563 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c035040 con 0x7fa7401444e0 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.174+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7400c9d10 con 0x7fa7401444e0 2024-03-06T07:29:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.174+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7400ca2d0 con 0x7fa7401444e0 2024-03-06T07:29:57.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.177+0000 7fa737400640 1 -- 172.21.15.114:0/3329900563 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa72c038020 con 0x7fa7401444e0 2024-03-06T07:29:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.178+0000 7fa748c19740 1 Processor -- start 2024-03-06T07:29:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.178+0000 7fa748c19740 1 -- start start 2024-03-06T07:29:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.178+0000 7fa737400640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa72006b500 0x7fa72006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa737400640 1 -- 172.21.15.114:0/3329900563 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa72c0abfd0 con 0x7fa7401444e0 2024-03-06T07:29:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa73f400640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa72006b500 0x7fa72006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808c0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa748c19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808f0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560808c0c90 0x5560808f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa748c19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55608082a600 con 0x5560808edb90 2024-03-06T07:29:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560807d3930 con 0x5560808ed7b0 2024-03-06T07:29:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.179+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560808266a0 con 0x5560808c0c90 2024-03-06T07:29:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73f400640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa72006b500 0x7fa72006d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa738007fd0 tx=0x7fa738009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808f0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49026/0 (socket says 172.21.15.114:49026) 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 -- 172.21.15.114:0/850058765 learned_addr learned my addr 172.21.15.114:0/850058765 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73ea00640 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 -- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560808c0c90 msgr2=0x5560808f0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73f400640 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560808c0c90 0x5560808f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560808c0c90 0x5560808f0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 -- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 msgr2=0x5560808c0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808c0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 -- 172.21.15.114:0/850058765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55608053f2d0 con 0x5560808edb90 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73ea00640 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.180+0000 7fa73e000640 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808f0540 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa72c031fd0 tx=0x7fa72c031970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b3935b185278dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa72b400640 1 -- 172.21.15.114:0/850058765 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c07e890 con 0x5560808edb90 2024-03-06T07:29:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa72b400640 1 -- 172.21.15.114:0/850058765 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c0a3e50 con 0x5560808edb90 2024-03-06T07:29:57.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa72b400640 1 -- 172.21.15.114:0/850058765 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c082e50 con 0x5560808edb90 2024-03-06T07:29:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 -- 172.21.15.114:0/850058765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 msgr2=0x5560808f0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808f0540 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa72c031fd0 tx=0x7fa72c031970 comp rx=0 tx=0).stop 2024-03-06T07:29:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 -- 172.21.15.114:0/850058765 shutdown_connections 2024-03-06T07:29:57.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x5560808c0c90 0x5560808f0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808f0540 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 --2- 172.21.15.114:0/850058765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 -- 172.21.15.114:0/850058765 >> 172.21.15.114:0/850058765 conn(0x5560808c29c0 msgr2=0x55608090b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.181+0000 7fa748c19740 1 -- 172.21.15.114:0/850058765 shutdown_connections 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.182+0000 7fa748c19740 1 -- 172.21.15.114:0/850058765 wait complete. 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.182+0000 7fa748c19740 1 -- start start 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.182+0000 7fa748c19740 1 Processor -- start 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.183+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x556080902bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.183+0000 7fa748c19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808df310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.183+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x556080905b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.183+0000 7fa748c19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55608082a600 con 0x5560808edb90 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.183+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560807e6d90 con 0x5560808ed7b0 2024-03-06T07:29:57.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.183+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560808c4f60 con 0x55608091e790 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa748c19740 10 client.0 _ll_register_callbacks cb 0x556080822350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808df310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808df310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49038/0 (socket says 172.21.15.114:49038) 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 -- 172.21.15.114:0/1028440879 learned_addr learned my addr 172.21.15.114:0/1028440879 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 -- 172.21.15.114:0/1028440879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 msgr2=0x556080905b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x556080905b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 -- 172.21.15.114:0/1028440879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 msgr2=0x556080902bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x556080902bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 -- 172.21.15.114:0/1028440879 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560806096b0 con 0x5560808edb90 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa73e000640 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808df310 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa72c031fd0 tx=0x7fa72c04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2ec773a9976abd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.184+0000 7fa71fe00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa71fe00640 1 -- 172.21.15.114:0/1028440879 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c082e50 con 0x5560808edb90 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa71fe00640 1 -- 172.21.15.114:0/1028440879 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa72c079b90 con 0x5560808edb90 2024-03-06T07:29:57.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa72c0a3e50 con 0x5560808edb90 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa71fe00640 1 -- 172.21.15.114:0/1028440879 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa72c07e890 con 0x5560808edb90 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 10 client.5688 fetch_fsmap learned FSMap version 26 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x556080903a30 con 0x5560808edb90 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa71fe00640 1 -- 172.21.15.114:0/1028440879 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c0a3e50 con 0x5560808edb90 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa71fe00640 1 -- 172.21.15.114:0/1028440879 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa72c099080 con 0x5560808edb90 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 10 client.5688 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 10 client.5688 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 20 client.5688 populate_metadata read hostname 'smithi114' 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x556080927100 0x5560809294f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 4 client.5688 mds_command: new command op to 5568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.185+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 --> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556080837b40 con 0x556080927100 2024-03-06T07:29:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.186+0000 7fa73ea00640 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x556080927100 0x5560809294f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.186+0000 7fa71fe00640 10 client.5688 ms_handle_connect on v2:172.21.15.114:6836/2123564678 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.186+0000 7fa73ea00640 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x556080927100 0x5560809294f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20201ee73919c806 server_cookie=a1a65fa0dd0978e5 in_seq=0 out_seq=0 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.187+0000 7fa71fe00640 1 -- 172.21.15.114:0/1028440879 <== mds.0 v2:172.21.15.114:6836/2123564678 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556080837b40 con 0x556080927100 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.187+0000 7fa71fe00640 10 client.5688 handle_command_reply: tid=0 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 client.5688 shutdown 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 msgr2=0x5560808df310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808df310 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa72c031fd0 tx=0x7fa72c04b000 comp rx=0 tx=0).stop 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 shutdown_connections 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x556080905b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x556080927100 0x5560809294f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x5560808df310 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 --2- 172.21.15.114:0/1028440879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x556080902bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 >> 172.21.15.114:0/1028440879 conn(0x5560808c0c90 msgr2=0x5560808c5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa71fe00640 0 client.5688 ms_handle_reset on v2:172.21.15.114:6836/2123564678 2024-03-06T07:29:57.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 shutdown_connections 2024-03-06T07:29:57.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 1 -- 172.21.15.114:0/1028440879 wait complete. 2024-03-06T07:29:57.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.188+0000 7fa748c19740 20 client.5688 trim_cache size 0 max 16384 2024-03-06T07:29:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.206+0000 7fa748c19740 1 Processor -- start 2024-03-06T07:29:57.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.206+0000 7fa748c19740 1 -- start start 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808e1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa748c19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x55608090e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x5560808f4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa748c19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400cbcf0 con 0x5560808edb90 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400c8800 con 0x5560808ed7b0 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400cb6f0 con 0x55608091e790 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x55608090e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa73e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x55608090e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49048/0 (socket says 172.21.15.114:49048) 2024-03-06T07:29:57.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.207+0000 7fa73e000640 1 -- 172.21.15.114:0/2580644528 learned_addr learned my addr 172.21.15.114:0/2580644528 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa73e000640 1 -- 172.21.15.114:0/2580644528 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 msgr2=0x5560808f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa73e000640 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x5560808f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa73e000640 1 -- 172.21.15.114:0/2580644528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 msgr2=0x5560808e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa73ea00640 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808e1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa73e000640 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808e1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa73e000640 1 -- 172.21.15.114:0/2580644528 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560808c0730 con 0x5560808edb90 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa73e000640 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x55608090e0f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa72c07a440 tx=0x7fa72c090070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67eb021fcc066e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa72aa00640 1 -- 172.21.15.114:0/2580644528 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c07a280 con 0x5560808edb90 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa72aa00640 1 -- 172.21.15.114:0/2580644528 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c0a3e50 con 0x5560808edb90 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.208+0000 7fa72aa00640 1 -- 172.21.15.114:0/2580644528 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c079b90 con 0x5560808edb90 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 -- 172.21.15.114:0/2580644528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 msgr2=0x55608090e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x55608090e0f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa72c07a440 tx=0x7fa72c090070 comp rx=0 tx=0).stop 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 -- 172.21.15.114:0/2580644528 shutdown_connections 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x5560808f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808edb90 0x55608090e0f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 --2- 172.21.15.114:0/2580644528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808ed7b0 0x5560808e1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 -- 172.21.15.114:0/2580644528 >> 172.21.15.114:0/2580644528 conn(0x5560808c0c90 msgr2=0x556080905b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 -- 172.21.15.114:0/2580644528 shutdown_connections 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.209+0000 7fa748c19740 1 -- 172.21.15.114:0/2580644528 wait complete. 2024-03-06T07:29:57.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.210+0000 7fa748c19740 1 -- start start 2024-03-06T07:29:57.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.210+0000 7fa748c19740 1 Processor -- start 2024-03-06T07:29:57.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa748c19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808ed7b0 0x556080970ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808edb90 0x5560808c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa748c19740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x5560808c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa748c19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55608082a600 con 0x5560808ed7b0 2024-03-06T07:29:57.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560807e6d90 con 0x5560808edb90 2024-03-06T07:29:57.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa748c19740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560808f2b20 con 0x55608091e790 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa748c19740 10 client.0 _ll_register_callbacks cb 0x7fa72c091070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa73ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808ed7b0 0x556080970ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa73ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808ed7b0 0x556080970ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49062/0 (socket says 172.21.15.114:49062) 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.212+0000 7fa73ea00640 1 -- 172.21.15.114:0/1586197795 learned_addr learned my addr 172.21.15.114:0/1586197795 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa73ea00640 1 -- 172.21.15.114:0/1586197795 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 msgr2=0x5560808c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa73ea00640 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x5560808c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa73ea00640 1 -- 172.21.15.114:0/1586197795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808edb90 msgr2=0x5560808c4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa73ea00640 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808edb90 0x5560808c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa73ea00640 1 -- 172.21.15.114:0/1586197795 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55608091ecc0 con 0x5560808ed7b0 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa73ea00640 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808ed7b0 0x556080970ef0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa730000c00 tx=0x7fa730009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e27468a10982f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa71fe00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa71fe00640 1 -- 172.21.15.114:0/1586197795 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000f040 con 0x5560808ed7b0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa71fe00640 1 -- 172.21.15.114:0/1586197795 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa730018240 con 0x5560808ed7b0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.213+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa72c0a3e50 con 0x5560808ed7b0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa71fe00640 1 -- 172.21.15.114:0/1586197795 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa730029050 con 0x5560808ed7b0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 10 client.5694 fetch_fsmap learned FSMap version 26 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x5560809714e0 con 0x5560808ed7b0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa71fe00640 1 -- 172.21.15.114:0/1586197795 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c0a3e50 con 0x5560808ed7b0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa71fe00640 1 -- 172.21.15.114:0/1586197795 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa73002c020 con 0x5560808ed7b0 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 10 client.5694 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 10 client.5694 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 20 client.5694 populate_metadata read hostname 'smithi114' 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x55608097aaf0 0x55608092d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 4 client.5694 mds_command: new command op to 5568 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:57.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 --> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556080831e90 con 0x55608097aaf0 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.214+0000 7fa73e000640 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x55608097aaf0 0x55608092d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa71fe00640 10 client.5694 ms_handle_connect on v2:172.21.15.114:6836/2123564678 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa73e000640 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x55608097aaf0 0x55608092d130 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107706bcaadaba87 server_cookie=d705d64c245c84aa in_seq=0 out_seq=0 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa71fe00640 1 -- 172.21.15.114:0/1586197795 <== mds.0 v2:172.21.15.114:6836/2123564678 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x556080831e90 con 0x55608097aaf0 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa71fe00640 10 client.5694 handle_command_reply: tid=0 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 client.5694 shutdown 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808ed7b0 msgr2=0x556080970ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808ed7b0 0x556080970ef0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa730000c00 tx=0x7fa730009d70 comp rx=0 tx=0).stop 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 shutdown_connections 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55608091e790 0x5560808c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5560808edb90 0x5560808c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:6836/2123564678,v1:172.21.15.114:6837/2123564678] conn(0x55608097aaf0 0x55608092d130 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 --2- 172.21.15.114:0/1586197795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5560808ed7b0 0x556080970ef0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa71fe00640 0 client.5694 ms_handle_reset on v2:172.21.15.114:6836/2123564678 2024-03-06T07:29:57.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 >> 172.21.15.114:0/1586197795 conn(0x5560808c0c90 msgr2=0x556080905b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.215+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 shutdown_connections 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa748c19740 1 -- 172.21.15.114:0/1586197795 wait complete. 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa748c19740 20 client.5694 trim_cache size 0 max 16384 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5568, 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:29:57.220 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 6.6665117040000004, 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 26, 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 44, 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 44, 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 12.081040725999999 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa72006b500 msgr2=0x7fa72006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa72006b500 0x7fa72006d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa738007fd0 tx=0x7fa738009040 comp rx=0 tx=0).stop 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401444e0 msgr2=0x7fa7400afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401444e0 0x7fa7400afed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa72c02b950 tx=0x7fa72c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 shutdown_connections 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7401474c0 0x7fa74015dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa72006b500 0x7fa72006d9c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7401444e0 0x7fa7400afed0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 --2- 172.21.15.114:0/3329900563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa7400c4e70 0x7fa7401f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.216+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 >> 172.21.15.114:0/3329900563 conn(0x7fa7400cc4f0 msgr2=0x7fa740140de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.217+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 shutdown_connections 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.217+0000 7fa745e00640 1 -- 172.21.15.114:0/3329900563 wait complete. 2024-03-06T07:29:57.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.439+0000 7fed10600640 1 Processor -- start 2024-03-06T07:29:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.439+0000 7fed10600640 1 -- start start 2024-03-06T07:29:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.441+0000 7fed10600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fed0815e160 0x7fed080b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.441+0000 7fed10600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e540 0x7fed080ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.441+0000 7fed10600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0806cc80 con 0x7fed080cf170 2024-03-06T07:29:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.441+0000 7fed10600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0806cf60 con 0x7fed0815e160 2024-03-06T07:29:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.441+0000 7fed10600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0806d240 con 0x7fed0815e540 2024-03-06T07:29:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.441+0000 7fed0dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fed0815e160 0x7fed080b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.441+0000 7fed0d200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e540 0x7fed080ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.442+0000 7fed0dc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fed0815e160 0x7fed080b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:57818/0 (socket says 172.21.15.114:57818) 2024-03-06T07:29:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.442+0000 7fed0dc00640 1 -- 172.21.15.114:0/4061105804 learned_addr learned my addr 172.21.15.114:0/4061105804 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.442+0000 7fed0dc00640 1 -- 172.21.15.114:0/4061105804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e540 msgr2=0x7fed080ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.442+0000 7fed0dc00640 1 --2- 172.21.15.114:0/4061105804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e540 0x7fed080ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.442+0000 7fed0dc00640 1 -- 172.21.15.114:0/4061105804 >> v1:172.21.15.177:6789/0 conn(0x7fed080cf170 legacy=0x7fed080cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.442+0000 7fed0dc00640 1 -- 172.21.15.114:0/4061105804 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080b33a0 con 0x7fed0815e160 2024-03-06T07:29:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.444+0000 7fed0dc00640 1 --2- 172.21.15.114:0/4061105804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed0815e160 0x7fed080b2e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009870 tx=0x7fecfc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b95215aa92d4131 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.445+0000 7fed07e00640 1 -- 172.21.15.114:0/4061105804 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc034070 con 0x7fed0815e160 2024-03-06T07:29:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.445+0000 7fed07e00640 1 -- 172.21.15.114:0/4061105804 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc030840 con 0x7fed0815e160 2024-03-06T07:29:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.445+0000 7fed07e00640 1 -- 172.21.15.114:0/4061105804 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc030c10 con 0x7fed0815e160 2024-03-06T07:29:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 -- 172.21.15.114:0/4061105804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed0815e160 msgr2=0x7fed080b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 --2- 172.21.15.114:0/4061105804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed0815e160 0x7fed080b2e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009870 tx=0x7fecfc02b980 comp rx=0 tx=0).stop 2024-03-06T07:29:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 -- 172.21.15.114:0/4061105804 shutdown_connections 2024-03-06T07:29:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 --2- 172.21.15.114:0/4061105804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e540 0x7fed080ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 --2- 172.21.15.114:0/4061105804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed0815e160 0x7fed080b2e60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 -- 172.21.15.114:0/4061105804 >> 172.21.15.114:0/4061105804 conn(0x7fed080b3800 msgr2=0x7fed080b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 -- 172.21.15.114:0/4061105804 shutdown_connections 2024-03-06T07:29:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.446+0000 7fed10600640 1 -- 172.21.15.114:0/4061105804 wait complete. 2024-03-06T07:29:57.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.448+0000 7fed10600640 1 Processor -- start 2024-03-06T07:29:57.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.448+0000 7fed10600640 1 -- start start 2024-03-06T07:29:57.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.449+0000 7fed10600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed080cf170 0x7fed0814c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.449+0000 7fed10600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e160 0x7fed08146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.449+0000 7fed10600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0815e540 0x7fed08147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.449+0000 7fed10600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080cea30 con 0x7fed0815e540 2024-03-06T07:29:57.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.449+0000 7fed10600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080cd890 con 0x7fed080cf170 2024-03-06T07:29:57.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.449+0000 7fed10600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed080cebb0 con 0x7fed0815e160 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0815e540 0x7fed08147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0815e540 0x7fed08147090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49074/0 (socket says 172.21.15.114:49074) 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 -- 172.21.15.114:0/403471201 learned_addr learned my addr 172.21.15.114:0/403471201 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0d200640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e160 0x7fed08146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0dc00640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed080cf170 0x7fed0814c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 -- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e160 msgr2=0x7fed08146b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e160 0x7fed08146b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 -- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed080cf170 msgr2=0x7fed0814c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed080cf170 0x7fed0814c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.450+0000 7fed0e600640 1 -- 172.21.15.114:0/403471201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080c5840 con 0x7fed0815e540 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.451+0000 7fed0e600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0815e540 0x7fed08147090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fed0000c730 tx=0x7fed0000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.451+0000 7fed05600640 1 -- 172.21.15.114:0/403471201 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00003b70 con 0x7fed0815e540 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.451+0000 7fed05600640 1 -- 172.21.15.114:0/403471201 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00003d10 con 0x7fed0815e540 2024-03-06T07:29:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.451+0000 7fed05600640 1 -- 172.21.15.114:0/403471201 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000aa90 con 0x7fed0815e540 2024-03-06T07:29:57.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.451+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed080c5ab0 con 0x7fed0815e540 2024-03-06T07:29:57.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.451+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed080c6070 con 0x7fed0815e540 2024-03-06T07:29:57.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.452+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0804f330 con 0x7fed0815e540 2024-03-06T07:29:57.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.453+0000 7fed05600640 1 -- 172.21.15.114:0/403471201 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed0000ac30 con 0x7fed0815e540 2024-03-06T07:29:57.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.454+0000 7fed05600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece806b4d0 0x7fece806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.454+0000 7fed05600640 1 -- 172.21.15.114:0/403471201 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed0008a090 con 0x7fed0815e540 2024-03-06T07:29:57.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.455+0000 7fed0dc00640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece806b4d0 0x7fece806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.455+0000 7fed0dc00640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece806b4d0 0x7fece806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fecfc036000 tx=0x7fecfc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.459+0000 7fed05600640 1 -- 172.21.15.114:0/403471201 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed00058fa0 con 0x7fed0815e540 2024-03-06T07:29:57.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.699+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fed080cdcc0 con 0x7fed0815e540 2024-03-06T07:29:57.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.700+0000 7fed05600640 1 -- 172.21.15.114:0/403471201 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7fed000588a0 con 0x7fed0815e540 2024-03-06T07:29:57.701 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:57.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.702+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece806b4d0 msgr2=0x7fece806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.702+0000 7fed10600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece806b4d0 0x7fece806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fecfc036000 tx=0x7fecfc03b040 comp rx=0 tx=0).stop 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.702+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0815e540 msgr2=0x7fed08147090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.702+0000 7fed10600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0815e540 0x7fed08147090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fed0000c730 tx=0x7fed0000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 shutdown_connections 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece806b4d0 0x7fece806d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0815e540 0x7fed08147090 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0815e160 0x7fed08146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 --2- 172.21.15.114:0/403471201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed080cf170 0x7fed0814c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 >> 172.21.15.114:0/403471201 conn(0x7fed080b3800 msgr2=0x7fed080d2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 shutdown_connections 2024-03-06T07:29:57.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:57.703+0000 7fed10600640 1 -- 172.21.15.114:0/403471201 wait complete. 2024-03-06T07:29:57.721 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:57.722 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:57.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.722 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:57.765 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42458sec preferred_lft 42458sec 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:57.766 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:57.767 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:57.767 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:29:57.767 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:57.767 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:57.767 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:57.767 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:29:57.767 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:57.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.768 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:57.892 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:29:57.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.897 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:29:57.950 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:29:57.950 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:29:57.951 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:57.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.952 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:29:57.952 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:29:57.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:29:57.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:29:57.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:29:57.952 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:29:58.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:58.119 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:29:58.167 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:29:58.167 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:29:58.168 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:58.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:58.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:29:58.248 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:58.248 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:58.293 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:58.293 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:58.293 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:58.383 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:29:58.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:58.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:58.505 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:58.505 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:58.539 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:58.539 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:58.539 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:29:58.539 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:29:58.539 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:29:58.541 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:29:58.541 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:58.541 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:58.541 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:58.541 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:58.545 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:58.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:58.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:58.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:58.619 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:58.674 INFO:teuthology.orchestra.run.smithi114.stdout:client5044 2024-03-06T07:29:58.674 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:29:58.674 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:58.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:29:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.962+0000 7f0a2c000640 1 Processor -- start 2024-03-06T07:29:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.962+0000 7f0a2c000640 1 -- start start 2024-03-06T07:29:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.965+0000 7f0a2c000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0a240b3310 0x7f0a2415fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.965+0000 7f0a2c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 0x7f0a240cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.965+0000 7f0a2c000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a240ccc40 con 0x7f0a240ce3a0 2024-03-06T07:29:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.965+0000 7f0a2c000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a240d05c0 con 0x7f0a240b3310 2024-03-06T07:29:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.965+0000 7f0a2c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240d08a0 con 0x7f0a241629e0 2024-03-06T07:29:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.966+0000 7f0a2a000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0a240ce3a0 0x7f0a240cd120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43390/0 (socket says 172.21.15.114:43390) 2024-03-06T07:29:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.966+0000 7f0a2a000640 1 -- 172.21.15.114:0/560662008 learned_addr learned my addr 172.21.15.114:0/560662008 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:29:58.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.966+0000 7f0a29600640 1 --2- 172.21.15.114:0/560662008 >> v2:172.21.15.177:3300/0 conn(0x7f0a240b3310 0x7f0a2415fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.966+0000 7f0a28c00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 0x7f0a240cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.966+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809407852 0 0) 0x7f0a240ccc40 con 0x7f0a240ce3a0 2024-03-06T07:29:58.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.966+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c003520 con 0x7f0a240ce3a0 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225543927 0 0) 0x7f0a0c003520 con 0x7f0a240ce3a0 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 msgr2=0x7f0a240cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 0x7f0a240cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 >> v2:172.21.15.177:3300/0 conn(0x7f0a240b3310 msgr2=0x7f0a2415fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 --2- 172.21.15.114:0/560662008 >> v2:172.21.15.177:3300/0 conn(0x7f0a240b3310 0x7f0a2415fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a240d0b80 con 0x7f0a240ce3a0 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f0a1c0019f0 con 0x7f0a240ce3a0 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 >> v1:172.21.15.114:6789/0 conn(0x7f0a240ce3a0 legacy=0x7f0a240cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 0x7f0a0c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a0c003f30 0x7f0a0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a0c004d40 0x7f0a0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240d05c0 con 0x7f0a0c003f30 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240d08a0 con 0x7f0a0c004d40 2024-03-06T07:29:58.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c00ba10 con 0x7f0a241629e0 2024-03-06T07:29:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a28c00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a0c003f30 0x7f0a0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a29600640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 0x7f0a0c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a28c00640 1 -- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 msgr2=0x7f0a0c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a28c00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a241629e0 0x7f0a0c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a28c00640 1 -- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a0c004d40 msgr2=0x7f0a0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.967+0000 7f0a28c00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a0c004d40 0x7f0a0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.968+0000 7f0a28c00640 1 -- 172.21.15.114:0/560662008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c003520 con 0x7f0a0c003f30 2024-03-06T07:29:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.969+0000 7f0a28c00640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a0c003f30 0x7f0a0c004800 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009af0 tx=0x7f0a0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189e52f422377315 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.969+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10039070 con 0x7f0a0c003f30 2024-03-06T07:29:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.969+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1002c6b0 con 0x7f0a0c003f30 2024-03-06T07:29:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.969+0000 7f0a23e00640 1 -- 172.21.15.114:0/560662008 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10009690 con 0x7f0a0c003f30 2024-03-06T07:29:58.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.969+0000 7f0a2c000640 1 -- 172.21.15.114:0/560662008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a0c003f30 msgr2=0x7f0a0c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.969+0000 7f0a2c000640 1 --2- 172.21.15.114:0/560662008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a0c003f30 0x7f0a0c004800 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009af0 tx=0x7f0a0c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.969+0000 7f0a2a000640 1 -- 172.21.15.114:0/560662008 reap_dead start 2024-03-06T07:29:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.970+0000 7f0a2c000640 1 -- 172.21.15.114:0/560662008 shutdown_connections 2024-03-06T07:29:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.970+0000 7f0a2c000640 1 -- 172.21.15.114:0/560662008 >> 172.21.15.114:0/560662008 conn(0x7f0a240b3790 msgr2=0x7f0a240b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.970+0000 7f0a2c000640 1 -- 172.21.15.114:0/560662008 shutdown_connections 2024-03-06T07:29:58.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.970+0000 7f0a2c000640 1 -- 172.21.15.114:0/560662008 wait complete. 2024-03-06T07:29:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.971+0000 7f0a2c000640 1 Processor -- start 2024-03-06T07:29:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.971+0000 7f0a2c000640 1 -- start start 2024-03-06T07:29:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.972+0000 7f0a2c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a0c003f30 0x7f0a24153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.972+0000 7f0a2c000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a240c8ea0 0x7f0a240b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.972+0000 7f0a2c000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a240b69a0 0x7f0a2415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.972+0000 7f0a2c000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240ce9b0 con 0x7f0a240c8ea0 2024-03-06T07:29:58.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.972+0000 7f0a2c000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240ce830 con 0x7f0a240b69a0 2024-03-06T07:29:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.972+0000 7f0a2c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240ceb30 con 0x7f0a0c003f30 2024-03-06T07:29:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a240c8ea0 0x7f0a240b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a240c8ea0 0x7f0a240b6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47612/0 (socket says 172.21.15.114:47612) 2024-03-06T07:29:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 -- 172.21.15.114:0/900499353 learned_addr learned my addr 172.21.15.114:0/900499353 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a29600640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a0c003f30 0x7f0a24153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 -- 172.21.15.114:0/900499353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a0c003f30 msgr2=0x7f0a24153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a0c003f30 0x7f0a24153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 -- 172.21.15.114:0/900499353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a240b69a0 msgr2=0x7f0a2415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a240b69a0 0x7f0a2415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 -- 172.21.15.114:0/900499353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a241e7ce0 con 0x7f0a240c8ea0 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.973+0000 7f0a28c00640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a240c8ea0 0x7f0a240b6460 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0a10037040 tx=0x7f0a10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.974+0000 7f0a21600640 1 -- 172.21.15.114:0/900499353 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10042020 con 0x7f0a240c8ea0 2024-03-06T07:29:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.974+0000 7f0a21600640 1 -- 172.21.15.114:0/900499353 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1003a040 con 0x7f0a240c8ea0 2024-03-06T07:29:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.974+0000 7f0a21600640 1 -- 172.21.15.114:0/900499353 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10039070 con 0x7f0a240c8ea0 2024-03-06T07:29:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.974+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a241e7fc0 con 0x7f0a240c8ea0 2024-03-06T07:29:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.974+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a241e8580 con 0x7f0a240c8ea0 2024-03-06T07:29:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.975+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2404f330 con 0x7f0a240c8ea0 2024-03-06T07:29:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.976+0000 7f0a21600640 1 -- 172.21.15.114:0/900499353 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a10045020 con 0x7f0a240c8ea0 2024-03-06T07:29:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.977+0000 7f0a21600640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a0406b500 0x7f0a0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.977+0000 7f0a21600640 1 -- 172.21.15.114:0/900499353 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a100ab350 con 0x7f0a240c8ea0 2024-03-06T07:29:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.977+0000 7f0a29600640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a0406b500 0x7f0a0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.978+0000 7f0a29600640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a0406b500 0x7f0a0406d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0a2415d810 tx=0x7f0a18008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:58.982+0000 7f0a21600640 1 -- 172.21.15.114:0/900499353 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a1004c020 con 0x7f0a240c8ea0 2024-03-06T07:29:59.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.219+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f0a24072e80 con 0x7f0a240c8ea0 2024-03-06T07:29:59.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.224+0000 7f0a21600640 1 -- 172.21.15.114:0/900499353 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0a1007a130 con 0x7f0a240c8ea0 2024-03-06T07:29:59.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a0406b500 msgr2=0x7f0a0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a0406b500 0x7f0a0406d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0a2415d810 tx=0x7f0a18008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a240c8ea0 msgr2=0x7f0a240b6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a240c8ea0 0x7f0a240b6460 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0a10037040 tx=0x7f0a10004730 comp rx=0 tx=0).stop 2024-03-06T07:29:59.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 shutdown_connections 2024-03-06T07:29:59.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a240b69a0 0x7f0a2415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a0406b500 0x7f0a0406d9c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a240c8ea0 0x7f0a240b6460 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 --2- 172.21.15.114:0/900499353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0a0c003f30 0x7f0a24153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.228+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 >> 172.21.15.114:0/900499353 conn(0x7f0a240b3790 msgr2=0x7f0a2406d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.229+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 shutdown_connections 2024-03-06T07:29:59.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:29:59.229+0000 7f0a2c000640 1 -- 172.21.15.114:0/900499353 wait complete. 2024-03-06T07:30:09.251 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:09.251 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> import uuid 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> n = 51 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = False 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> if 0 > 0: 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = True 2024-03-06T07:30:09.306 DEBUG:teuthology.orchestra.run.smithi114:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> dpath = os.path.dirname(path) 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> fnameprefix = os.path.basename(path) 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> try: 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> for i in range(n): 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> f.write(f"{i}") 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> f.flush() 2024-03-06T07:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(f.fileno()) 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> os.unlink(fpath) 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> if create_hard_links: 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> for j in range(0): 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> if True: 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> finally: 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> os.close(dirfd) 2024-03-06T07:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:30:09.523 INFO:teuthology.orchestra.run.smithi114.stderr:Traceback (most recent call last): 2024-03-06T07:30:09.524 INFO:teuthology.orchestra.run.smithi114.stderr: File "", line 33, in 2024-03-06T07:30:09.524 INFO:teuthology.orchestra.run.smithi114.stderr:OSError: [Errno 28] No space left on device 2024-03-06T07:30:09.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:09 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:30:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:30:09.544 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:09.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:09.684 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:09.806 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:09.806 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:09.806 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:09.851 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:09.851 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:09.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.176+0000 7f1abb200640 1 Processor -- start 2024-03-06T07:30:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.176+0000 7f1abb200640 1 -- start start 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1abb200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ab4145540 0x7f1ab406e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1abb200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1ab415f950 0x7f1ab415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1abb200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab406e5e0 con 0x7f1ab406afc0 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1abb200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab40ce960 con 0x7f1ab4145540 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1abb200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab415d2f0 con 0x7f1ab415f950 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ab4145540 0x7f1ab406e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ab4145540 0x7f1ab406e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47682/0 (socket says 172.21.15.114:47682) 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9e00640 1 -- 172.21.15.114:0/2780412268 learned_addr learned my addr 172.21.15.114:0/2780412268 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1aba800640 1 --1- 172.21.15.114:0/2780412268 >> v1:172.21.15.114:6789/0 conn(0x7f1ab406afc0 0x7f1ab40cedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56234/0 (socket says 172.21.15.114:56234) 2024-03-06T07:30:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9400640 1 --2- 172.21.15.114:0/2780412268 >> v2:172.21.15.177:3300/0 conn(0x7f1ab415f950 0x7f1ab415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9e00640 1 -- 172.21.15.114:0/2780412268 >> v2:172.21.15.177:3300/0 conn(0x7f1ab415f950 msgr2=0x7f1ab415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9e00640 1 --2- 172.21.15.114:0/2780412268 >> v2:172.21.15.177:3300/0 conn(0x7f1ab415f950 0x7f1ab415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9e00640 1 -- 172.21.15.114:0/2780412268 >> v1:172.21.15.114:6789/0 conn(0x7f1ab406afc0 legacy=0x7f1ab40cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.178+0000 7f1ab9e00640 1 -- 172.21.15.114:0/2780412268 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab415fed0 con 0x7f1ab4145540 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1ab9e00640 1 --2- 172.21.15.114:0/2780412268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab4145540 0x7f1ab406e0a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40b5fd0 tx=0x7f1aa802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f42da53bf0e8730 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1ab8a00640 1 -- 172.21.15.114:0/2780412268 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8034070 con 0x7f1ab4145540 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1ab8a00640 1 -- 172.21.15.114:0/2780412268 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa8030860 con 0x7f1ab4145540 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1ab8a00640 1 -- 172.21.15.114:0/2780412268 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8030c30 con 0x7f1ab4145540 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1abb200640 1 -- 172.21.15.114:0/2780412268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab4145540 msgr2=0x7f1ab406e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1abb200640 1 --2- 172.21.15.114:0/2780412268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab4145540 0x7f1ab406e0a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40b5fd0 tx=0x7f1aa802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1abb200640 1 -- 172.21.15.114:0/2780412268 shutdown_connections 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1abb200640 1 --2- 172.21.15.114:0/2780412268 >> v2:172.21.15.177:3300/0 conn(0x7f1ab415f950 0x7f1ab415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1abb200640 1 --2- 172.21.15.114:0/2780412268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab4145540 0x7f1ab406e0a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.180+0000 7f1abb200640 1 -- 172.21.15.114:0/2780412268 >> 172.21.15.114:0/2780412268 conn(0x7f1ab406c0d0 msgr2=0x7f1ab406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.181+0000 7f1abb200640 1 -- 172.21.15.114:0/2780412268 shutdown_connections 2024-03-06T07:30:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.181+0000 7f1abb200640 1 -- 172.21.15.114:0/2780412268 wait complete. 2024-03-06T07:30:10.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.182+0000 7f1abb200640 1 Processor -- start 2024-03-06T07:30:10.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.182+0000 7f1abb200640 1 -- start start 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.183+0000 7f1abb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ab406afc0 0x7f1ab415eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.183+0000 7f1abb200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ab4145540 0x7f1ab40aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.183+0000 7f1abb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab415f950 0x7f1ab415afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.183+0000 7f1abb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4160770 con 0x7f1ab415f950 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.183+0000 7f1abb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4160490 con 0x7f1ab406afc0 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.183+0000 7f1abb200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4145920 con 0x7f1ab4145540 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab415f950 0x7f1ab415afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab415f950 0x7f1ab415afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47696/0 (socket says 172.21.15.114:47696) 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 -- 172.21.15.114:0/2712450611 learned_addr learned my addr 172.21.15.114:0/2712450611 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1ab9e00640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ab406afc0 0x7f1ab415eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1ab9400640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ab4145540 0x7f1ab40aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 -- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ab4145540 msgr2=0x7f1ab40aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ab4145540 0x7f1ab40aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 -- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ab406afc0 msgr2=0x7f1ab415eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ab406afc0 0x7f1ab415eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.184+0000 7f1aba800640 1 -- 172.21.15.114:0/2712450611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab40b4a50 con 0x7f1ab415f950 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.185+0000 7f1aba800640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab415f950 0x7f1ab415afd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c780 tx=0x7f1ab000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.185+0000 7f1aae000640 1 -- 172.21.15.114:0/2712450611 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0017070 con 0x7f1ab415f950 2024-03-06T07:30:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.185+0000 7f1aae000640 1 -- 172.21.15.114:0/2712450611 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab000a640 con 0x7f1ab415f950 2024-03-06T07:30:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.185+0000 7f1aae000640 1 -- 172.21.15.114:0/2712450611 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab000aa30 con 0x7f1ab415f950 2024-03-06T07:30:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.185+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab40b4d30 con 0x7f1ab415f950 2024-03-06T07:30:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.185+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab40b52f0 con 0x7f1ab415f950 2024-03-06T07:30:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.186+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab404f330 con 0x7f1ab415f950 2024-03-06T07:30:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.187+0000 7f1aae000640 1 -- 172.21.15.114:0/2712450611 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ab001a070 con 0x7f1ab415f950 2024-03-06T07:30:10.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.188+0000 7f1aae000640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a9806b4d0 0x7f1a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.188+0000 7f1aae000640 1 -- 172.21.15.114:0/2712450611 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ab0089710 con 0x7f1ab415f950 2024-03-06T07:30:10.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.188+0000 7f1ab9e00640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a9806b4d0 0x7f1a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.189+0000 7f1ab9e00640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a9806b4d0 0x7f1a9806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8036000 tx=0x7f1aa803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.193+0000 7f1aae000640 1 -- 172.21.15.114:0/2712450611 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ab406e5e0 con 0x7f1ab415f950 2024-03-06T07:30:10.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.393+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ab406bc60 con 0x7f1ab415f950 2024-03-06T07:30:10.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.394+0000 7f1aae000640 1 -- 172.21.15.114:0/2712450611 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f1ab0057f40 con 0x7f1ab415f950 2024-03-06T07:30:10.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:10.396 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:51.547619+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:47.835295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:49.012214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a9806b4d0 msgr2=0x7f1a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a9806b4d0 0x7f1a9806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8036000 tx=0x7f1aa803b040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab415f950 msgr2=0x7f1ab415afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab415f950 0x7f1ab415afd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c780 tx=0x7f1ab000cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 shutdown_connections 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a9806b4d0 0x7f1a9806d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ab415f950 0x7f1ab415afd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ab4145540 0x7f1ab40aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 --2- 172.21.15.114:0/2712450611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ab406afc0 0x7f1ab415eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 >> 172.21.15.114:0/2712450611 conn(0x7f1ab406c0d0 msgr2=0x7f1ab406e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 shutdown_connections 2024-03-06T07:30:10.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.397+0000 7f1abb200640 1 -- 172.21.15.114:0/2712450611 wait complete. 2024-03-06T07:30:10.425 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:47.835295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:49.012214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:10.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:10.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.636+0000 7f1c2ca00640 1 Processor -- start 2024-03-06T07:30:10.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.636+0000 7f1c2ca00640 1 -- start start 2024-03-06T07:30:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c2ca00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1c2815e130 0x7f1c280cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c2ca00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1c280cfaf0 0x7f1c280d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c2ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c280d23e0 0x7f1c280c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c2ca00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280cf120 con 0x7f1c280cfaf0 2024-03-06T07:30:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c2ca00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280d13e0 con 0x7f1c2815e130 2024-03-06T07:30:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c2ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280cead0 con 0x7f1c280d23e0 2024-03-06T07:30:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c1fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1c280cfaf0 0x7f1c280d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c1fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1c280cfaf0 0x7f1c280d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47712/0 (socket says 172.21.15.114:47712) 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.639+0000 7f1c1fe00640 1 -- 172.21.15.114:0/545056172 learned_addr learned my addr 172.21.15.114:0/545056172 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.640+0000 7f1c26800640 1 --2- 172.21.15.114:0/545056172 >> v2:172.21.15.177:3300/0 conn(0x7f1c2815e130 0x7f1c280cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.640+0000 7f1c27200640 1 --2- 172.21.15.114:0/545056172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c280d23e0 0x7f1c280c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.640+0000 7f1c1fe00640 1 -- 172.21.15.114:0/545056172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c280d23e0 msgr2=0x7f1c280c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.640+0000 7f1c1fe00640 1 --2- 172.21.15.114:0/545056172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c280d23e0 0x7f1c280c7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.640+0000 7f1c1fe00640 1 -- 172.21.15.114:0/545056172 >> v2:172.21.15.177:3300/0 conn(0x7f1c2815e130 msgr2=0x7f1c280cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.640+0000 7f1c1fe00640 1 --2- 172.21.15.114:0/545056172 >> v2:172.21.15.177:3300/0 conn(0x7f1c2815e130 0x7f1c280cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.640+0000 7f1c1fe00640 1 -- 172.21.15.114:0/545056172 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280cedb0 con 0x7f1c280cfaf0 2024-03-06T07:30:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.641+0000 7f1c1fe00640 1 --2- 172.21.15.114:0/545056172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c280d1ea0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1c28067ce0 tx=0x7f1c1002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d7c0e34c14938f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.641+0000 7f1c25e00640 1 -- 172.21.15.114:0/545056172 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10035020 con 0x7f1c280cfaf0 2024-03-06T07:30:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.641+0000 7f1c25e00640 1 -- 172.21.15.114:0/545056172 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1002d040 con 0x7f1c280cfaf0 2024-03-06T07:30:10.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.641+0000 7f1c25e00640 1 -- 172.21.15.114:0/545056172 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10031a80 con 0x7f1c280cfaf0 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 -- 172.21.15.114:0/545056172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 msgr2=0x7f1c280d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/545056172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c280d1ea0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1c28067ce0 tx=0x7f1c1002b900 comp rx=0 tx=0).stop 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 -- 172.21.15.114:0/545056172 shutdown_connections 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/545056172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c280d23e0 0x7f1c280c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/545056172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c280d1ea0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/545056172 >> v2:172.21.15.177:3300/0 conn(0x7f1c2815e130 0x7f1c280cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 -- 172.21.15.114:0/545056172 >> 172.21.15.114:0/545056172 conn(0x7f1c2809d070 msgr2=0x7f1c280d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 -- 172.21.15.114:0/545056172 shutdown_connections 2024-03-06T07:30:10.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.642+0000 7f1c2ca00640 1 -- 172.21.15.114:0/545056172 wait complete. 2024-03-06T07:30:10.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.645+0000 7f1c2ca00640 1 Processor -- start 2024-03-06T07:30:10.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.645+0000 7f1c2ca00640 1 -- start start 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c2ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c2815f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c2ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c280d23e0 0x7f1c280cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c2ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c2815e130 0x7f1c28150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c2ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280c8030 con 0x7f1c280cfaf0 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c2ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280c7eb0 con 0x7f1c280d23e0 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c2ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280c81b0 con 0x7f1c2815e130 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c26800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c2815f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c26800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c2815f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47714/0 (socket says 172.21.15.114:47714) 2024-03-06T07:30:10.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.647+0000 7f1c26800640 1 -- 172.21.15.114:0/1461201217 learned_addr learned my addr 172.21.15.114:0/1461201217 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.648+0000 7f1c26800640 1 -- 172.21.15.114:0/1461201217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c2815e130 msgr2=0x7f1c28150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.648+0000 7f1c26800640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c2815e130 0x7f1c28150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.648+0000 7f1c26800640 1 -- 172.21.15.114:0/1461201217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c280d23e0 msgr2=0x7f1c280cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.648+0000 7f1c26800640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c280d23e0 0x7f1c280cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.648+0000 7f1c26800640 1 -- 172.21.15.114:0/1461201217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28150b50 con 0x7f1c280cfaf0 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.648+0000 7f1c26800640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c2815f560 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c7f0 tx=0x7f1c1800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.648+0000 7f1c1ea00640 1 -- 172.21.15.114:0/1461201217 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18015070 con 0x7f1c280cfaf0 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.649+0000 7f1c1ea00640 1 -- 172.21.15.114:0/1461201217 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c18004220 con 0x7f1c280cfaf0 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.649+0000 7f1c1ea00640 1 -- 172.21.15.114:0/1461201217 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18007620 con 0x7f1c280cfaf0 2024-03-06T07:30:10.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.649+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c281605e0 con 0x7f1c280cfaf0 2024-03-06T07:30:10.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.649+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2815fd40 con 0x7f1c280cfaf0 2024-03-06T07:30:10.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.650+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2804f330 con 0x7f1c280cfaf0 2024-03-06T07:30:10.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.651+0000 7f1c1ea00640 1 -- 172.21.15.114:0/1461201217 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c180079d0 con 0x7f1c280cfaf0 2024-03-06T07:30:10.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.652+0000 7f1c1ea00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.652+0000 7f1c1ea00640 1 -- 172.21.15.114:0/1461201217 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c1808a540 con 0x7f1c280cfaf0 2024-03-06T07:30:10.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.653+0000 7f1c1fe00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.653+0000 7f1c1fe00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c0406b4d0 0x7f1c0406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1c10036000 tx=0x7f1c10030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.657+0000 7f1c1ea00640 1 -- 172.21.15.114:0/1461201217 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c180593b0 con 0x7f1c280cfaf0 2024-03-06T07:30:10.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.850+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c2809d690 con 0x7f1c280cfaf0 2024-03-06T07:30:10.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.851+0000 7f1c1ea00640 1 -- 172.21.15.114:0/1461201217 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c18058cb0 con 0x7f1c280cfaf0 2024-03-06T07:30:10.853 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:30:10.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c0406b4d0 msgr2=0x7f1c0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c0406b4d0 0x7f1c0406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1c10036000 tx=0x7f1c10030000 comp rx=0 tx=0).stop 2024-03-06T07:30:10.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 msgr2=0x7f1c2815f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c2815f560 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800c7f0 tx=0x7f1c1800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 shutdown_connections 2024-03-06T07:30:10.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1c2815e130 0x7f1c28150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c280d23e0 0x7f1c280cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 --2- 172.21.15.114:0/1461201217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c280cfaf0 0x7f1c2815f560 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 >> 172.21.15.114:0/1461201217 conn(0x7f1c2809d070 msgr2=0x7f1c280c84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.853+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 shutdown_connections 2024-03-06T07:30:10.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:10.854+0000 7f1c2ca00640 1 -- 172.21.15.114:0/1461201217 wait complete. 2024-03-06T07:30:10.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:11.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.082+0000 7f7d7ca00640 1 Processor -- start 2024-03-06T07:30:11.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.082+0000 7f7d7ca00640 1 -- start start 2024-03-06T07:30:11.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.083+0000 7f7d7ca00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7d78158950 0x7f7d780cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.083+0000 7f7d7ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d780cfc10 0x7f7d780cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.083+0000 7f7d7ca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d7815b130 con 0x7f7d78158570 2024-03-06T07:30:11.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.083+0000 7f7d7ca00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780af170 con 0x7f7d78158950 2024-03-06T07:30:11.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.083+0000 7f7d7ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7815b3f0 con 0x7f7d780cfc10 2024-03-06T07:30:11.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.084+0000 7f7d76800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7d78158570 0x7f7d780ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56238/0 (socket says 172.21.15.114:56238) 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.084+0000 7f7d76800640 1 -- 172.21.15.114:0/759944115 learned_addr learned my addr 172.21.15.114:0/759944115 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.084+0000 7f7d77200640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d780cfc10 0x7f7d780cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.084+0000 7f7d6de00640 1 --2- 172.21.15.114:0/759944115 >> v2:172.21.15.177:3300/0 conn(0x7f7d78158950 0x7f7d780cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.084+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960059953 0 0) 0x7f7d7815b130 con 0x7f7d78158570 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d5c003520 con 0x7f7d78158570 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 159562874 0 0) 0x7f7d5c003520 con 0x7f7d78158570 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d780cfc10 msgr2=0x7f7d780cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d780cfc10 0x7f7d780cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 >> v2:172.21.15.177:3300/0 conn(0x7f7d78158950 msgr2=0x7f7d780cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 --2- 172.21.15.114:0/759944115 >> v2:172.21.15.177:3300/0 conn(0x7f7d78158950 0x7f7d780cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7815b6d0 con 0x7f7d78158570 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7d68001670 con 0x7f7d78158570 2024-03-06T07:30:11.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 >> v1:172.21.15.114:6789/0 conn(0x7f7d78158570 legacy=0x7f7d780ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c0039b0 0x7f7d5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d5c004800 0x7f7d5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d780cfc10 0x7f7d5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780af170 con 0x7f7d5c0039b0 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7815b3f0 con 0x7f7d5c004800 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c00dc20 con 0x7f7d780cfc10 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.085+0000 7f7d76800640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c0039b0 0x7f7d5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.086+0000 7f7d76800640 1 -- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d780cfc10 msgr2=0x7f7d5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.086+0000 7f7d76800640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d780cfc10 0x7f7d5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.086+0000 7f7d76800640 1 -- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d5c004800 msgr2=0x7f7d5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.086+0000 7f7d76800640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d5c004800 0x7f7d5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.086+0000 7f7d76800640 1 -- 172.21.15.114:0/759944115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c003520 con 0x7f7d5c0039b0 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.087+0000 7f7d76800640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c0039b0 0x7f7d5c003dd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800c190 tx=0x7f7d5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a252d1d9cd55a910 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.087+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68007c80 con 0x7f7d5c0039b0 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.087+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d68030070 con 0x7f7d5c0039b0 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.087+0000 7f7d75e00640 1 -- 172.21.15.114:0/759944115 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68005d30 con 0x7f7d5c0039b0 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.088+0000 7f7d7ca00640 1 -- 172.21.15.114:0/759944115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c0039b0 msgr2=0x7f7d5c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.088+0000 7f7d7ca00640 1 --2- 172.21.15.114:0/759944115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d5c0039b0 0x7f7d5c003dd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800c190 tx=0x7f7d5c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.088+0000 7f7d77200640 1 -- 172.21.15.114:0/759944115 reap_dead start 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.088+0000 7f7d7ca00640 1 -- 172.21.15.114:0/759944115 shutdown_connections 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.088+0000 7f7d7ca00640 1 -- 172.21.15.114:0/759944115 >> 172.21.15.114:0/759944115 conn(0x7f7d7806afc0 msgr2=0x7f7d7806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.088+0000 7f7d7ca00640 1 -- 172.21.15.114:0/759944115 shutdown_connections 2024-03-06T07:30:11.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.088+0000 7f7d7ca00640 1 -- 172.21.15.114:0/759944115 wait complete. 2024-03-06T07:30:11.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.089+0000 7f7d7ca00640 1 Processor -- start 2024-03-06T07:30:11.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.090+0000 7f7d7ca00640 1 -- start start 2024-03-06T07:30:11.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.090+0000 7f7d7ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d5c0039b0 0x7f7d7814f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.090+0000 7f7d7ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d78156cc0 0x7f7d7815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d7ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7815cfc0 0x7f7d7815d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d7ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780aedc0 con 0x7f7d7815cfc0 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d7ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780b4950 con 0x7f7d5c0039b0 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d7ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78071d50 con 0x7f7d78156cc0 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7815cfc0 0x7f7d7815d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7815cfc0 0x7f7d7815d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47738/0 (socket says 172.21.15.114:47738) 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 -- 172.21.15.114:0/2381965382 learned_addr learned my addr 172.21.15.114:0/2381965382 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 -- 172.21.15.114:0/2381965382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d78156cc0 msgr2=0x7f7d7815a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d78156cc0 0x7f7d7815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 -- 172.21.15.114:0/2381965382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d5c0039b0 msgr2=0x7f7d7814f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d5c0039b0 0x7f7d7814f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 -- 172.21.15.114:0/2381965382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7815a930 con 0x7f7d7815cfc0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.091+0000 7f7d77200640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7815cfc0 0x7f7d7815d440 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7d7815c2a0 tx=0x7f7d7000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.092+0000 7f7d6f400640 1 -- 172.21.15.114:0/2381965382 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000bb10 con 0x7f7d7815cfc0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.092+0000 7f7d6f400640 1 -- 172.21.15.114:0/2381965382 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7001a070 con 0x7f7d7815cfc0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.092+0000 7f7d6f400640 1 -- 172.21.15.114:0/2381965382 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70014400 con 0x7f7d7815cfc0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.092+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d78071420 con 0x7f7d7815cfc0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.092+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d780719e0 con 0x7f7d7815cfc0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.093+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7815b0f0 con 0x7f7d7815cfc0 2024-03-06T07:30:11.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.094+0000 7f7d6f400640 1 -- 172.21.15.114:0/2381965382 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d70005d10 con 0x7f7d7815cfc0 2024-03-06T07:30:11.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.095+0000 7f7d6f400640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d5406b4d0 0x7f7d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.095+0000 7f7d6f400640 1 -- 172.21.15.114:0/2381965382 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d7008a4d0 con 0x7f7d7815cfc0 2024-03-06T07:30:11.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.095+0000 7f7d76800640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d5406b4d0 0x7f7d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.096+0000 7f7d76800640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d5406b4d0 0x7f7d5406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7d68034000 tx=0x7f7d68039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.099+0000 7f7d6f400640 1 -- 172.21.15.114:0/2381965382 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d7815b0f0 con 0x7f7d7815cfc0 2024-03-06T07:30:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.324+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d7804f330 con 0x7f7d7815cfc0 2024-03-06T07:30:11.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.325+0000 7f7d6f400640 1 -- 172.21.15.114:0/2381965382 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d70024020 con 0x7f7d7815cfc0 2024-03-06T07:30:11.327 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:30:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d5406b4d0 msgr2=0x7f7d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d5406b4d0 0x7f7d5406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7d68034000 tx=0x7f7d68039040 comp rx=0 tx=0).stop 2024-03-06T07:30:11.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7815cfc0 msgr2=0x7f7d7815d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7815cfc0 0x7f7d7815d440 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7d7815c2a0 tx=0x7f7d7000b480 comp rx=0 tx=0).stop 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 shutdown_connections 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d5406b4d0 0x7f7d5406d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7815cfc0 0x7f7d7815d440 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7d78156cc0 0x7f7d7815a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 --2- 172.21.15.114:0/2381965382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d5c0039b0 0x7f7d7814f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.327+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 >> 172.21.15.114:0/2381965382 conn(0x7f7d7806afc0 msgr2=0x7f7d780ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.328+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 shutdown_connections 2024-03-06T07:30:11.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.328+0000 7f7d7ca00640 1 -- 172.21.15.114:0/2381965382 wait complete. 2024-03-06T07:30:11.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.562+0000 7f6125000640 1 Processor -- start 2024-03-06T07:30:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.562+0000 7f6125000640 1 -- start start 2024-03-06T07:30:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.563+0000 7f6125000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f612015d9f0 0x7f61200ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f6125000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61200cef20 con 0x7f612006e510 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f6125000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61200b3e70 con 0x7f6120072840 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f6125000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61200b4bf0 con 0x7f612015d9f0 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f612015d9f0 0x7f61200ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f612015d9f0 0x7f61200ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47746/0 (socket says 172.21.15.114:47746) 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611f400640 1 -- 172.21.15.114:0/3124037751 learned_addr learned my addr 172.21.15.114:0/3124037751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f612006e510 0x7f612006c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56240/0 (socket says 172.21.15.114:56240) 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611f400640 1 -- 172.21.15.114:0/3124037751 >> v1:172.21.15.177:6789/0 conn(0x7f6120072840 legacy=0x7f612006de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611f400640 1 -- 172.21.15.114:0/3124037751 >> v1:172.21.15.114:6789/0 conn(0x7f612006e510 legacy=0x7f612006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611f400640 1 -- 172.21.15.114:0/3124037751 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61200cf1a0 con 0x7f612015d9f0 2024-03-06T07:30:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.564+0000 7f611e000640 1 -- 172.21.15.114:0/3124037751 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10058281 0 0) 0x7f6118001490 con 0x7f612006e510 2024-03-06T07:30:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.566+0000 7f611f400640 1 --2- 172.21.15.114:0/3124037751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612015d9f0 0x7f61200ce9e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f612006c000 tx=0x7f611002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237aaab086ce5036 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.566+0000 7f611e000640 1 -- 172.21.15.114:0/3124037751 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110034070 con 0x7f612015d9f0 2024-03-06T07:30:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.566+0000 7f611e000640 1 -- 172.21.15.114:0/3124037751 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6110030610 con 0x7f612015d9f0 2024-03-06T07:30:11.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.566+0000 7f611e000640 1 -- 172.21.15.114:0/3124037751 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61100309e0 con 0x7f612015d9f0 2024-03-06T07:30:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.566+0000 7f6125000640 1 -- 172.21.15.114:0/3124037751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612015d9f0 msgr2=0x7f61200ce9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.567+0000 7f6125000640 1 --2- 172.21.15.114:0/3124037751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612015d9f0 0x7f61200ce9e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f612006c000 tx=0x7f611002b960 comp rx=0 tx=0).stop 2024-03-06T07:30:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.567+0000 7f6125000640 1 -- 172.21.15.114:0/3124037751 shutdown_connections 2024-03-06T07:30:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.567+0000 7f6125000640 1 --2- 172.21.15.114:0/3124037751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612015d9f0 0x7f61200ce9e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.567+0000 7f6125000640 1 -- 172.21.15.114:0/3124037751 >> 172.21.15.114:0/3124037751 conn(0x7f612006afc0 msgr2=0x7f612006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.567+0000 7f6125000640 1 -- 172.21.15.114:0/3124037751 shutdown_connections 2024-03-06T07:30:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.567+0000 7f6125000640 1 -- 172.21.15.114:0/3124037751 wait complete. 2024-03-06T07:30:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.568+0000 7f6125000640 1 Processor -- start 2024-03-06T07:30:11.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.568+0000 7f6125000640 1 -- start start 2024-03-06T07:30:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.569+0000 7f6125000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612006e510 0x7f612014c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.569+0000 7f6125000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6120072840 0x7f6120157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.569+0000 7f6125000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612015d9f0 0x7f61201583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.569+0000 7f6125000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61200b43f0 con 0x7f612006e510 2024-03-06T07:30:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.569+0000 7f6125000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612015ddd0 con 0x7f612015d9f0 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.569+0000 7f6125000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6120070970 con 0x7f6120072840 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612006e510 0x7f612014c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612006e510 0x7f612014c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47762/0 (socket says 172.21.15.114:47762) 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 -- 172.21.15.114:0/1895350479 learned_addr learned my addr 172.21.15.114:0/1895350479 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 -- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6120072840 msgr2=0x7f6120157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611ea00640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6120072840 0x7f6120157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6120072840 0x7f6120157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 -- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612015d9f0 msgr2=0x7f61201583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611fe00640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612015d9f0 0x7f61201583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612015d9f0 0x7f61201583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611f400640 1 -- 172.21.15.114:0/1895350479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6120156510 con 0x7f612006e510 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.570+0000 7f611fe00640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612015d9f0 0x7f61201583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.571+0000 7f611f400640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612006e510 0x7f612014c280 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f61100090b0 tx=0x7f61100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.571+0000 7f6117400640 1 -- 172.21.15.114:0/1895350479 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110034040 con 0x7f612006e510 2024-03-06T07:30:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.571+0000 7f6117400640 1 -- 172.21.15.114:0/1895350479 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6110004230 con 0x7f612006e510 2024-03-06T07:30:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.571+0000 7f6117400640 1 -- 172.21.15.114:0/1895350479 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611003a890 con 0x7f612006e510 2024-03-06T07:30:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.571+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61201567f0 con 0x7f612006e510 2024-03-06T07:30:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.571+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612014c7c0 con 0x7f612006e510 2024-03-06T07:30:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.572+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612004f330 con 0x7f612006e510 2024-03-06T07:30:11.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.573+0000 7f6117400640 1 -- 172.21.15.114:0/1895350479 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6110037070 con 0x7f612006e510 2024-03-06T07:30:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.574+0000 7f6117400640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610006b4d0 0x7f610006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.574+0000 7f6117400640 1 -- 172.21.15.114:0/1895350479 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61100abf70 con 0x7f612006e510 2024-03-06T07:30:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.575+0000 7f611ea00640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610006b4d0 0x7f610006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.575+0000 7f611ea00640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610006b4d0 0x7f610006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f61200aec60 tx=0x7f610c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.579+0000 7f6117400640 1 -- 172.21.15.114:0/1895350479 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f611003f070 con 0x7f612006e510 2024-03-06T07:30:11.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.861+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f61200afea0 con 0x7f612006e510 2024-03-06T07:30:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.861+0000 7f6117400640 1 -- 172.21.15.114:0/1895350479 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f611007ade0 con 0x7f612006e510 2024-03-06T07:30:11.863 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610006b4d0 msgr2=0x7f610006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610006b4d0 0x7f610006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f61200aec60 tx=0x7f610c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612006e510 msgr2=0x7f612014c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612006e510 0x7f612014c280 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f61100090b0 tx=0x7f61100046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 shutdown_connections 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612015d9f0 0x7f61201583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6120072840 0x7f6120157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610006b4d0 0x7f610006d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 --2- 172.21.15.114:0/1895350479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612006e510 0x7f612014c280 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 >> 172.21.15.114:0/1895350479 conn(0x7f612006afc0 msgr2=0x7f61200cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 shutdown_connections 2024-03-06T07:30:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:11.864+0000 7f6125000640 1 -- 172.21.15.114:0/1895350479 wait complete. 2024-03-06T07:30:11.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.077+0000 7fbf79000640 1 Processor -- start 2024-03-06T07:30:12.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.077+0000 7fbf79000640 1 -- start start 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.079+0000 7fbf79000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbf740c6460 0x7fbf740c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.079+0000 7fbf79000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c5960 0x7fbf74165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.079+0000 7fbf79000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74145b30 con 0x7fbf740c4930 2024-03-06T07:30:12.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.079+0000 7fbf79000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf740c68c0 con 0x7fbf740c6460 2024-03-06T07:30:12.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.079+0000 7fbf79000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74142ce0 con 0x7fbf740c5960 2024-03-06T07:30:12.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.079+0000 7fbf72800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c5960 0x7fbf74165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf71400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbf740c6460 0x7fbf740c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf71e00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fbf740c4930 0x7fbf740c4d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:59642/0 (socket says 172.21.15.114:59642) 2024-03-06T07:30:12.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf71e00640 1 -- 172.21.15.114:0/802916767 learned_addr learned my addr 172.21.15.114:0/802916767 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:12.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf71400640 1 -- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c5960 msgr2=0x7fbf74165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf71400640 1 --2- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c5960 0x7fbf74165b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf71400640 1 -- 172.21.15.114:0/802916767 >> v1:172.21.15.177:6789/0 conn(0x7fbf740c4930 legacy=0x7fbf740c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf71400640 1 -- 172.21.15.114:0/802916767 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740cbca0 con 0x7fbf740c6460 2024-03-06T07:30:12.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.080+0000 7fbf72800640 1 --2- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c5960 0x7fbf74165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:12.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.082+0000 7fbf71400640 1 --2- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c6460 0x7fbf740c5420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf600097a0 tx=0x7fbf6002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9230c3ff36668635 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.082+0000 7fbf70a00640 1 -- 172.21.15.114:0/802916767 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60031780 con 0x7fbf740c6460 2024-03-06T07:30:12.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.082+0000 7fbf70a00640 1 -- 172.21.15.114:0/802916767 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf60037070 con 0x7fbf740c6460 2024-03-06T07:30:12.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.082+0000 7fbf70a00640 1 -- 172.21.15.114:0/802916767 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60031b40 con 0x7fbf740c6460 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.082+0000 7fbf79000640 1 -- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c6460 msgr2=0x7fbf740c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.082+0000 7fbf79000640 1 --2- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c6460 0x7fbf740c5420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf600097a0 tx=0x7fbf6002b820 comp rx=0 tx=0).stop 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.083+0000 7fbf79000640 1 -- 172.21.15.114:0/802916767 shutdown_connections 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.083+0000 7fbf79000640 1 --2- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c5960 0x7fbf74165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.083+0000 7fbf79000640 1 --2- 172.21.15.114:0/802916767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c6460 0x7fbf740c5420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.083+0000 7fbf79000640 1 -- 172.21.15.114:0/802916767 >> 172.21.15.114:0/802916767 conn(0x7fbf740cc4f0 msgr2=0x7fbf740cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.083+0000 7fbf79000640 1 -- 172.21.15.114:0/802916767 shutdown_connections 2024-03-06T07:30:12.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.083+0000 7fbf79000640 1 -- 172.21.15.114:0/802916767 wait complete. 2024-03-06T07:30:12.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.084+0000 7fbf79000640 1 Processor -- start 2024-03-06T07:30:12.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.085+0000 7fbf79000640 1 -- start start 2024-03-06T07:30:12.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf79000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c4930 0x7fbf740c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf79000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c5960 0x7fbf740c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf79000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf740c6460 0x7fbf740c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf79000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740ca3b0 con 0x7fbf740c6460 2024-03-06T07:30:12.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf79000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf740ca0d0 con 0x7fbf740c5960 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf79000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7406e600 con 0x7fbf740c4930 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf72800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf740c6460 0x7fbf740c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf72800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf740c6460 0x7fbf740c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47776/0 (socket says 172.21.15.114:47776) 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf72800640 1 -- 172.21.15.114:0/1865682693 learned_addr learned my addr 172.21.15.114:0/1865682693 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf71e00640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c4930 0x7fbf740c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf71400640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c5960 0x7fbf740c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.086+0000 7fbf72800640 1 -- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c4930 msgr2=0x7fbf740c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf72800640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c4930 0x7fbf740c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf72800640 1 -- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c5960 msgr2=0x7fbf740c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf72800640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c5960 0x7fbf740c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf72800640 1 -- 172.21.15.114:0/1865682693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740c9590 con 0x7fbf740c6460 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf71400640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c5960 0x7fbf740c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:12.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf72800640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf740c6460 0x7fbf740c8f70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00efc0 tx=0x7fbf6c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf6a000640 1 -- 172.21.15.114:0/1865682693 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c018030 con 0x7fbf740c6460 2024-03-06T07:30:12.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf6a000640 1 -- 172.21.15.114:0/1865682693 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c00cd30 con 0x7fbf740c6460 2024-03-06T07:30:12.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf6a000640 1 -- 172.21.15.114:0/1865682693 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003bb0 con 0x7fbf740c6460 2024-03-06T07:30:12.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf740cd650 con 0x7fbf740c6460 2024-03-06T07:30:12.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.087+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf740cdc10 con 0x7fbf740c6460 2024-03-06T07:30:12.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.088+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf740cb530 con 0x7fbf740c6460 2024-03-06T07:30:12.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.089+0000 7fbf6a000640 1 -- 172.21.15.114:0/1865682693 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf6c005d10 con 0x7fbf740c6460 2024-03-06T07:30:12.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.090+0000 7fbf6a000640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5406b4d0 0x7fbf5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.090+0000 7fbf6a000640 1 -- 172.21.15.114:0/1865682693 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf6c08ac10 con 0x7fbf740c6460 2024-03-06T07:30:12.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.091+0000 7fbf71e00640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5406b4d0 0x7fbf5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.091+0000 7fbf71e00640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5406b4d0 0x7fbf5406d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbf64005880 tx=0x7fbf64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.095+0000 7fbf6a000640 1 -- 172.21.15.114:0/1865682693 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf740cb530 con 0x7fbf740c6460 2024-03-06T07:30:12.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.304+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbf7404f330 con 0x7fbf740c6460 2024-03-06T07:30:12.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.305+0000 7fbf6a000640 1 -- 172.21.15.114:0/1865682693 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbf6c022070 con 0x7fbf740c6460 2024-03-06T07:30:12.307 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:30:12.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.307+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5406b4d0 msgr2=0x7fbf5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.307+0000 7fbf79000640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5406b4d0 0x7fbf5406d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbf64005880 tx=0x7fbf64009040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.307+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf740c6460 msgr2=0x7fbf740c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.307+0000 7fbf79000640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf740c6460 0x7fbf740c8f70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00efc0 tx=0x7fbf6c00c490 comp rx=0 tx=0).stop 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.307+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 shutdown_connections 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.308+0000 7fbf79000640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5406b4d0 0x7fbf5406d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.308+0000 7fbf79000640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf740c6460 0x7fbf740c8f70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.308+0000 7fbf79000640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf740c5960 0x7fbf740c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.308+0000 7fbf79000640 1 --2- 172.21.15.114:0/1865682693 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf740c4930 0x7fbf740c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.308+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 >> 172.21.15.114:0/1865682693 conn(0x7fbf740cc4f0 msgr2=0x7fbf741e3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.308+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 shutdown_connections 2024-03-06T07:30:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.308+0000 7fbf79000640 1 -- 172.21.15.114:0/1865682693 wait complete. 2024-03-06T07:30:12.329 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:30:12.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:12.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:12.497 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:30:12.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:12.497 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:12.539 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:30:12.539 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:12.580 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:12.581 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:12.581 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:12.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:12.619 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:30:12.619 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:12.670 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:12.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:12.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.867+0000 7fc088200640 1 Processor -- start 2024-03-06T07:30:12.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.867+0000 7fc088200640 1 -- start start 2024-03-06T07:30:12.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.870+0000 7fc088200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc08015e160 0x7fc080162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.870+0000 7fc088200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc0800af820 0x7fc0800b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.870+0000 7fc088200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc080162b60 con 0x7fc0800b38f0 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.870+0000 7fc088200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08006b0f0 con 0x7fc0800af820 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.870+0000 7fc088200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0800b5620 con 0x7fc08015e160 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.870+0000 7fc084e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc0800af820 0x7fc0800b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.870+0000 7fc086200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc0800b38f0 0x7fc08006d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56246/0 (socket says 172.21.15.114:56246) 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.871+0000 7fc086200640 1 -- 172.21.15.114:0/3322295876 learned_addr learned my addr 172.21.15.114:0/3322295876 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.871+0000 7fc085800640 1 --2- 172.21.15.114:0/3322295876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc08015e160 0x7fc080162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.871+0000 7fc084e00640 1 -- 172.21.15.114:0/3322295876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc08015e160 msgr2=0x7fc080162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.871+0000 7fc084e00640 1 --2- 172.21.15.114:0/3322295876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc08015e160 0x7fc080162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.871+0000 7fc084e00640 1 -- 172.21.15.114:0/3322295876 >> v1:172.21.15.114:6789/0 conn(0x7fc0800b38f0 legacy=0x7fc08006d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.871+0000 7fc084e00640 1 -- 172.21.15.114:0/3322295876 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0800b3d70 con 0x7fc0800af820 2024-03-06T07:30:12.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.871+0000 7fc07fe00640 1 -- 172.21.15.114:0/3322295876 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344741886 0 0) 0x7fc078001490 con 0x7fc0800b38f0 2024-03-06T07:30:12.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.872+0000 7fc084e00640 1 --2- 172.21.15.114:0/3322295876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc0800b33b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0097a0 tx=0x7fc06c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45adad7d03fc6b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.872+0000 7fc07fe00640 1 -- 172.21.15.114:0/3322295876 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c0315a0 con 0x7fc0800af820 2024-03-06T07:30:12.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.872+0000 7fc07fe00640 1 -- 172.21.15.114:0/3322295876 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c037070 con 0x7fc0800af820 2024-03-06T07:30:12.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.872+0000 7fc07fe00640 1 -- 172.21.15.114:0/3322295876 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c031960 con 0x7fc0800af820 2024-03-06T07:30:12.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 -- 172.21.15.114:0/3322295876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 msgr2=0x7fc0800b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 --2- 172.21.15.114:0/3322295876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc0800b33b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0097a0 tx=0x7fc06c02b840 comp rx=0 tx=0).stop 2024-03-06T07:30:12.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 -- 172.21.15.114:0/3322295876 shutdown_connections 2024-03-06T07:30:12.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 --2- 172.21.15.114:0/3322295876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc0800b33b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 --2- 172.21.15.114:0/3322295876 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc08015e160 0x7fc080162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 -- 172.21.15.114:0/3322295876 >> 172.21.15.114:0/3322295876 conn(0x7fc0800d0340 msgr2=0x7fc0800d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 -- 172.21.15.114:0/3322295876 shutdown_connections 2024-03-06T07:30:12.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.873+0000 7fc088200640 1 -- 172.21.15.114:0/3322295876 wait complete. 2024-03-06T07:30:12.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.876+0000 7fc088200640 1 Processor -- start 2024-03-06T07:30:12.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.876+0000 7fc088200640 1 -- start start 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.877+0000 7fc088200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc080158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.877+0000 7fc088200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0800b38f0 0x7fc0800cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.877+0000 7fc088200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc08015e160 0x7fc08015fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.877+0000 7fc088200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08006ec30 con 0x7fc0800af820 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.877+0000 7fc088200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0800afce0 con 0x7fc08015e160 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.877+0000 7fc088200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08006ef10 con 0x7fc0800b38f0 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc080158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc080158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47790/0 (socket says 172.21.15.114:47790) 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 -- 172.21.15.114:0/1074926411 learned_addr learned my addr 172.21.15.114:0/1074926411 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 -- 172.21.15.114:0/1074926411 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0800b38f0 msgr2=0x7fc0800cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0800b38f0 0x7fc0800cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 -- 172.21.15.114:0/1074926411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc08015e160 msgr2=0x7fc08015fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc08015e160 0x7fc08015fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 -- 172.21.15.114:0/1074926411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08015e9d0 con 0x7fc0800af820 2024-03-06T07:30:12.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.878+0000 7fc085800640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc080158300 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc07400d800 tx=0x7fc07400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.879+0000 7fc07d600640 1 -- 172.21.15.114:0/1074926411 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400b6c0 con 0x7fc0800af820 2024-03-06T07:30:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.879+0000 7fc07d600640 1 -- 172.21.15.114:0/1074926411 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc074019070 con 0x7fc0800af820 2024-03-06T07:30:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.879+0000 7fc07d600640 1 -- 172.21.15.114:0/1074926411 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400ba80 con 0x7fc0800af820 2024-03-06T07:30:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.879+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08015ecb0 con 0x7fc0800af820 2024-03-06T07:30:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.879+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08015f270 con 0x7fc0800af820 2024-03-06T07:30:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.880+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc080074ae0 con 0x7fc0800af820 2024-03-06T07:30:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.881+0000 7fc07d600640 1 -- 172.21.15.114:0/1074926411 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc07400bc20 con 0x7fc0800af820 2024-03-06T07:30:12.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.882+0000 7fc07d600640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc06006b4d0 0x7fc06006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.882+0000 7fc07d600640 1 -- 172.21.15.114:0/1074926411 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc07408a6c0 con 0x7fc0800af820 2024-03-06T07:30:12.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.883+0000 7fc084e00640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc06006b4d0 0x7fc06006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.883+0000 7fc084e00640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc06006b4d0 0x7fc06006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc06c036000 tx=0x7fc06c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:12.887+0000 7fc07d600640 1 -- 172.21.15.114:0/1074926411 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc074059530 con 0x7fc0800af820 2024-03-06T07:30:13.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.121+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc08004f330 con 0x7fc0800af820 2024-03-06T07:30:13.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.122+0000 7fc07d600640 1 -- 172.21.15.114:0/1074926411 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fc074058e30 con 0x7fc0800af820 2024-03-06T07:30:13.124 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:13.124 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4721,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/308840931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":308840931},{"type":"v1","addr":"172.21.15.177:6833","nonce":308840931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/4238819696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":4238819696},{"type":"v1","addr":"172.21.15.177:6835","nonce":4238819696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4960,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/379671784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":379671784},{"type":"v1","addr":"172.21.15.114:6835","nonce":379671784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:49.994729+0000","modified":"2024-03-06T07:29:53.062242+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5568":{"gid":5568,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2123564678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":2123564678},{"type":"v1","addr":"172.21.15.114:6837","nonce":2123564678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5568,"qdb_cluster":[5568]},"id":3}]} 2024-03-06T07:30:13.124 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 26 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc06006b4d0 msgr2=0x7fc06006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc06006b4d0 0x7fc06006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc06c036000 tx=0x7fc06c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 msgr2=0x7fc080158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc080158300 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc07400d800 tx=0x7fc07400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 shutdown_connections 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc08015e160 0x7fc08015fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0800b38f0 0x7fc0800cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc06006b4d0 0x7fc06006d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 --2- 172.21.15.114:0/1074926411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0800af820 0x7fc080158300 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 >> 172.21.15.114:0/1074926411 conn(0x7fc0800d0340 msgr2=0x7fc08006bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.125+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 shutdown_connections 2024-03-06T07:30:13.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.126+0000 7fc088200640 1 -- 172.21.15.114:0/1074926411 wait complete. 2024-03-06T07:30:13.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:13.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.365+0000 7feb74800640 1 Processor -- start 2024-03-06T07:30:13.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.365+0000 7feb74800640 1 -- start start 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.367+0000 7feb74800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.367+0000 7feb74800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb6c072310 0x7feb6c06f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.367+0000 7feb74800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6c0cfe30 con 0x7feb6c0cf170 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.367+0000 7feb74800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6c0cfb50 con 0x7feb6c072310 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.367+0000 7feb74800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c06ed90 con 0x7feb6c144660 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.367+0000 7feb72800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb6c072310 0x7feb6c06f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.368+0000 7feb72800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb6c072310 0x7feb6c06f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47804/0 (socket says 172.21.15.114:47804) 2024-03-06T07:30:13.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.368+0000 7feb72800640 1 -- 172.21.15.114:0/1307691773 learned_addr learned my addr 172.21.15.114:0/1307691773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:13.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.368+0000 7feb71e00640 1 --2- 172.21.15.114:0/1307691773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.368+0000 7feb72800640 1 -- 172.21.15.114:0/1307691773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 msgr2=0x7feb6c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.368+0000 7feb72800640 1 --2- 172.21.15.114:0/1307691773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.368+0000 7feb72800640 1 -- 172.21.15.114:0/1307691773 >> v1:172.21.15.177:6789/0 conn(0x7feb6c0cf170 legacy=0x7feb6c071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.368+0000 7feb72800640 1 -- 172.21.15.114:0/1307691773 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c06f070 con 0x7feb6c072310 2024-03-06T07:30:13.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.369+0000 7feb72800640 1 --2- 172.21.15.114:0/1307691773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c072310 0x7feb6c06f870 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feb68009e50 tx=0x7feb6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad72aab5d0b890a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.369+0000 7feb70a00640 1 -- 172.21.15.114:0/1307691773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6802cb00 con 0x7feb6c072310 2024-03-06T07:30:13.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.369+0000 7feb70a00640 1 -- 172.21.15.114:0/1307691773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6802cca0 con 0x7feb6c072310 2024-03-06T07:30:13.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.369+0000 7feb70a00640 1 -- 172.21.15.114:0/1307691773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb68034610 con 0x7feb6c072310 2024-03-06T07:30:13.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 -- 172.21.15.114:0/1307691773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c072310 msgr2=0x7feb6c06f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 --2- 172.21.15.114:0/1307691773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c072310 0x7feb6c06f870 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feb68009e50 tx=0x7feb6802bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:13.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 -- 172.21.15.114:0/1307691773 shutdown_connections 2024-03-06T07:30:13.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 --2- 172.21.15.114:0/1307691773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c072310 0x7feb6c06f870 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 --2- 172.21.15.114:0/1307691773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 -- 172.21.15.114:0/1307691773 >> 172.21.15.114:0/1307691773 conn(0x7feb6c0d0b90 msgr2=0x7feb6c0d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 -- 172.21.15.114:0/1307691773 shutdown_connections 2024-03-06T07:30:13.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.370+0000 7feb74800640 1 -- 172.21.15.114:0/1307691773 wait complete. 2024-03-06T07:30:13.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.372+0000 7feb74800640 1 Processor -- start 2024-03-06T07:30:13.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.372+0000 7feb74800640 1 -- start start 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb74800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb6c072310 0x7feb6c143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb74800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c0cf170 0x7feb6c168190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb74800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb74800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c0cc6d0 con 0x7feb6c0cf170 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb74800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c0cc550 con 0x7feb6c072310 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb74800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c0cf5b0 con 0x7feb6c144660 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb71400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c0cf170 0x7feb6c168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb71400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c0cf170 0x7feb6c168190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47816/0 (socket says 172.21.15.114:47816) 2024-03-06T07:30:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.373+0000 7feb71400640 1 -- 172.21.15.114:0/3584174595 learned_addr learned my addr 172.21.15.114:0/3584174595 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb71e00640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb6c072310 0x7feb6c143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb72800640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb71400640 1 -- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 msgr2=0x7feb6c0b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb71400640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb71400640 1 -- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb6c072310 msgr2=0x7feb6c143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb71400640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb6c072310 0x7feb6c143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb71400640 1 -- 172.21.15.114:0/3584174595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c15c4d0 con 0x7feb6c0cf170 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb71400640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c0cf170 0x7feb6c168190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00c400 tx=0x7feb5c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.374+0000 7feb66000640 1 -- 172.21.15.114:0/3584174595 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c0052d0 con 0x7feb6c0cf170 2024-03-06T07:30:13.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.375+0000 7feb66000640 1 -- 172.21.15.114:0/3584174595 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5c005490 con 0x7feb6c0cf170 2024-03-06T07:30:13.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.375+0000 7feb66000640 1 -- 172.21.15.114:0/3584174595 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5c007260 con 0x7feb6c0cf170 2024-03-06T07:30:13.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.375+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6c15c7b0 con 0x7feb6c0cf170 2024-03-06T07:30:13.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.375+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6c142d10 con 0x7feb6c0cf170 2024-03-06T07:30:13.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.376+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6c15ccd0 con 0x7feb6c0cf170 2024-03-06T07:30:13.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.377+0000 7feb66000640 1 -- 172.21.15.114:0/3584174595 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb5c007400 con 0x7feb6c0cf170 2024-03-06T07:30:13.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.378+0000 7feb66000640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb5006b4d0 0x7feb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.378+0000 7feb66000640 1 -- 172.21.15.114:0/3584174595 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb5c0897f0 con 0x7feb6c0cf170 2024-03-06T07:30:13.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.378+0000 7feb71e00640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb5006b4d0 0x7feb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.379+0000 7feb71e00640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb5006b4d0 0x7feb5006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7feb60009a30 tx=0x7feb60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.382+0000 7feb66000640 1 -- 172.21.15.114:0/3584174595 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb5c0585d0 con 0x7feb6c0cf170 2024-03-06T07:30:13.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.580+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb6c04f330 con 0x7feb6c0cf170 2024-03-06T07:30:13.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.582+0000 7feb66000640 1 -- 172.21.15.114:0/3584174595 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16661 (secure 0 0 0) 0x7feb5c057ed0 con 0x7feb6c0cf170 2024-03-06T07:30:13.584 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:13.584 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:51.547619+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:47.835295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:49.012214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb5006b4d0 msgr2=0x7feb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb5006b4d0 0x7feb5006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7feb60009a30 tx=0x7feb60008040 comp rx=0 tx=0).stop 2024-03-06T07:30:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c0cf170 msgr2=0x7feb6c168190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c0cf170 0x7feb6c168190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7feb5c00c400 tx=0x7feb5c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 shutdown_connections 2024-03-06T07:30:13.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb6c144660 0x7feb6c0b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb5006b4d0 0x7feb5006d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb6c0cf170 0x7feb6c168190 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 --2- 172.21.15.114:0/3584174595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb6c072310 0x7feb6c143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 >> 172.21.15.114:0/3584174595 conn(0x7feb6c0d0b90 msgr2=0x7feb6c0ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.585+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 shutdown_connections 2024-03-06T07:30:13.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.586+0000 7feb74800640 1 -- 172.21.15.114:0/3584174595 wait complete. 2024-03-06T07:30:13.607 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:47.835295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:49.012214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:13.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:13.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.815+0000 7f35fec00640 1 Processor -- start 2024-03-06T07:30:13.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.815+0000 7f35fec00640 1 -- start start 2024-03-06T07:30:13.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.818+0000 7f35fec00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f35f80d4a00 0x7f35f80d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.818+0000 7f35fec00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35f80b5490 0x7f35f80c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.818+0000 7f35fec00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f80cecd0 con 0x7f35f8161b50 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.818+0000 7f35fec00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80cef40 con 0x7f35f80b5490 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.818+0000 7f35fec00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80cfc80 con 0x7f35f80d4a00 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35fcc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35f80b5490 0x7f35f80c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35f7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f35f8161b50 0x7f35f80d05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56258/0 (socket says 172.21.15.114:56258) 2024-03-06T07:30:13.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35f7e00640 1 -- 172.21.15.114:0/811524328 learned_addr learned my addr 172.21.15.114:0/811524328 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35f7400640 1 --2- 172.21.15.114:0/811524328 >> v2:172.21.15.177:3300/0 conn(0x7f35f80d4a00 0x7f35f80d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35fcc00640 1 -- 172.21.15.114:0/811524328 >> v2:172.21.15.177:3300/0 conn(0x7f35f80d4a00 msgr2=0x7f35f80d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35fcc00640 1 --2- 172.21.15.114:0/811524328 >> v2:172.21.15.177:3300/0 conn(0x7f35f80d4a00 0x7f35f80d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35fcc00640 1 -- 172.21.15.114:0/811524328 >> v1:172.21.15.114:6789/0 conn(0x7f35f8161b50 legacy=0x7f35f80d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.819+0000 7f35fcc00640 1 -- 172.21.15.114:0/811524328 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f80c7b60 con 0x7f35f80b5490 2024-03-06T07:30:13.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.821+0000 7f35fcc00640 1 --2- 172.21.15.114:0/811524328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80b5490 0x7f35f80c8080 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f35f80b6a60 tx=0x7f35f002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1dff27923d35955 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.821+0000 7f35f6a00640 1 -- 172.21.15.114:0/811524328 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0039070 con 0x7f35f80b5490 2024-03-06T07:30:13.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.821+0000 7f35f6a00640 1 -- 172.21.15.114:0/811524328 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f002cc70 con 0x7f35f80b5490 2024-03-06T07:30:13.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.821+0000 7f35f6a00640 1 -- 172.21.15.114:0/811524328 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0034660 con 0x7f35f80b5490 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 -- 172.21.15.114:0/811524328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80b5490 msgr2=0x7f35f80c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 --2- 172.21.15.114:0/811524328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80b5490 0x7f35f80c8080 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f35f80b6a60 tx=0x7f35f002bec0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 -- 172.21.15.114:0/811524328 shutdown_connections 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 --2- 172.21.15.114:0/811524328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80b5490 0x7f35f80c8080 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 --2- 172.21.15.114:0/811524328 >> v2:172.21.15.177:3300/0 conn(0x7f35f80d4a00 0x7f35f80d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 -- 172.21.15.114:0/811524328 >> 172.21.15.114:0/811524328 conn(0x7f35f80d3e70 msgr2=0x7f35f80d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 -- 172.21.15.114:0/811524328 shutdown_connections 2024-03-06T07:30:13.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.822+0000 7f35fec00640 1 -- 172.21.15.114:0/811524328 wait complete. 2024-03-06T07:30:13.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.824+0000 7f35fec00640 1 Processor -- start 2024-03-06T07:30:13.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.824+0000 7f35fec00640 1 -- start start 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35fec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35f80b5490 0x7f35f8149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35fec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80d4a00 0x7f35f8159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35fec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35f8161b50 0x7f35f815a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35fec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c8740 con 0x7f35f80d4a00 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35fec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c85c0 con 0x7f35f80b5490 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35fec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c88c0 con 0x7f35f8161b50 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35f7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80d4a00 0x7f35f8159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.825+0000 7f35f7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80d4a00 0x7f35f8159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47832/0 (socket says 172.21.15.114:47832) 2024-03-06T07:30:13.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7400640 1 -- 172.21.15.114:0/1656772049 learned_addr learned my addr 172.21.15.114:0/1656772049 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7e00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35f80b5490 0x7f35f8149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35fcc00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35f8161b50 0x7f35f815a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7400640 1 -- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35f8161b50 msgr2=0x7f35f815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7400640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35f8161b50 0x7f35f815a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7400640 1 -- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35f80b5490 msgr2=0x7f35f8149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7400640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35f80b5490 0x7f35f8149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7400640 1 -- 172.21.15.114:0/1656772049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f81f4740 con 0x7f35f80d4a00 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35f7400640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80d4a00 0x7f35f8159d30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f35e400d800 tx=0x7f35e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35efe00640 1 -- 172.21.15.114:0/1656772049 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e400b6c0 con 0x7f35f80d4a00 2024-03-06T07:30:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.826+0000 7f35efe00640 1 -- 172.21.15.114:0/1656772049 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e4019070 con 0x7f35f80d4a00 2024-03-06T07:30:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.827+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f814a360 con 0x7f35f80d4a00 2024-03-06T07:30:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.827+0000 7f35efe00640 1 -- 172.21.15.114:0/1656772049 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e400ba80 con 0x7f35f80d4a00 2024-03-06T07:30:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.827+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f815ab80 con 0x7f35f80d4a00 2024-03-06T07:30:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.828+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f804f330 con 0x7f35f80d4a00 2024-03-06T07:30:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.829+0000 7f35efe00640 1 -- 172.21.15.114:0/1656772049 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35e400bc20 con 0x7f35f80d4a00 2024-03-06T07:30:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.829+0000 7f35efe00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f35d806b4d0 0x7f35d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.830+0000 7f35efe00640 1 -- 172.21.15.114:0/1656772049 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35e408a750 con 0x7f35f80d4a00 2024-03-06T07:30:13.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.830+0000 7f35f7e00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f35d806b4d0 0x7f35d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.831+0000 7f35f7e00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f35d806b4d0 0x7f35d806d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f35e8000f60 tx=0x7f35e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:13.834+0000 7f35efe00640 1 -- 172.21.15.114:0/1656772049 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35e4059530 con 0x7f35f80d4a00 2024-03-06T07:30:14.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.029+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35f80d16d0 con 0x7f35f80d4a00 2024-03-06T07:30:14.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.030+0000 7f35efe00640 1 -- 172.21.15.114:0/1656772049 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35e4058e30 con 0x7f35f80d4a00 2024-03-06T07:30:14.031 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:30:14.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f35d806b4d0 msgr2=0x7f35d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f35d806b4d0 0x7f35d806d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f35e8000f60 tx=0x7f35e8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:14.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80d4a00 msgr2=0x7f35f8159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80d4a00 0x7f35f8159d30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f35e400d800 tx=0x7f35e400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 shutdown_connections 2024-03-06T07:30:14.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35f8161b50 0x7f35f815a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f35d806b4d0 0x7f35d806d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35f80d4a00 0x7f35f8159d30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 --2- 172.21.15.114:0/1656772049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35f80b5490 0x7f35f8149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 >> 172.21.15.114:0/1656772049 conn(0x7f35f80d3e70 msgr2=0x7f35f80c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 shutdown_connections 2024-03-06T07:30:14.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.032+0000 7f35fec00640 1 -- 172.21.15.114:0/1656772049 wait complete. 2024-03-06T07:30:14.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:14.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.257+0000 7fec4b200640 1 Processor -- start 2024-03-06T07:30:14.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.258+0000 7fec4b200640 1 -- start start 2024-03-06T07:30:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.260+0000 7fec4b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fec440b3b40 0x7fec440b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec4b200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fec4406dda0 0x7fec4406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec4b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440b04b0 con 0x7fec44158770 2024-03-06T07:30:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec4b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4415ef20 con 0x7fec440b3b40 2024-03-06T07:30:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec4b200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4415e8d0 con 0x7fec4406dda0 2024-03-06T07:30:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec43400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fec440b3b40 0x7fec440b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec43e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fec44158770 0x7fec440afa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56260/0 (socket says 172.21.15.114:56260) 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec43e00640 1 -- 172.21.15.114:0/3638560485 learned_addr learned my addr 172.21.15.114:0/3638560485 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec49200640 1 --2- 172.21.15.114:0/3638560485 >> v2:172.21.15.177:3300/0 conn(0x7fec4406dda0 0x7fec4406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec43400640 1 -- 172.21.15.114:0/3638560485 >> v2:172.21.15.177:3300/0 conn(0x7fec4406dda0 msgr2=0x7fec4406e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec43400640 1 --2- 172.21.15.114:0/3638560485 >> v2:172.21.15.177:3300/0 conn(0x7fec4406dda0 0x7fec4406e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec43400640 1 -- 172.21.15.114:0/3638560485 >> v1:172.21.15.114:6789/0 conn(0x7fec44158770 legacy=0x7fec440afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec43400640 1 -- 172.21.15.114:0/3638560485 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4415ebb0 con 0x7fec440b3b40 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec42a00640 1 -- 172.21.15.114:0/3638560485 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177924437 0 0) 0x7fec440b04b0 con 0x7fec44158770 2024-03-06T07:30:14.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.261+0000 7fec49200640 1 --2- 172.21.15.114:0/3638560485 >> v2:172.21.15.177:3300/0 conn(0x7fec4406dda0 0x7fec4406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:14.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.262+0000 7fec43400640 1 --2- 172.21.15.114:0/3638560485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec440b3fd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fec440b57a0 tx=0x7fec2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64e450afff8027e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec42a00640 1 -- 172.21.15.114:0/3638560485 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c035020 con 0x7fec440b3b40 2024-03-06T07:30:14.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec42a00640 1 -- 172.21.15.114:0/3638560485 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c031830 con 0x7fec440b3b40 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec42a00640 1 -- 172.21.15.114:0/3638560485 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c031c00 con 0x7fec440b3b40 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 -- 172.21.15.114:0/3638560485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 msgr2=0x7fec440b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 --2- 172.21.15.114:0/3638560485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec440b3fd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fec440b57a0 tx=0x7fec2c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 -- 172.21.15.114:0/3638560485 shutdown_connections 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 --2- 172.21.15.114:0/3638560485 >> v2:172.21.15.177:3300/0 conn(0x7fec4406dda0 0x7fec4406e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 --2- 172.21.15.114:0/3638560485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec440b3fd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 -- 172.21.15.114:0/3638560485 >> 172.21.15.114:0/3638560485 conn(0x7fec440b2840 msgr2=0x7fec440b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 -- 172.21.15.114:0/3638560485 shutdown_connections 2024-03-06T07:30:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.263+0000 7fec4b200640 1 -- 172.21.15.114:0/3638560485 wait complete. 2024-03-06T07:30:14.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.266+0000 7fec4b200640 1 Processor -- start 2024-03-06T07:30:14.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.266+0000 7fec4b200640 1 -- start start 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec4b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec4406dda0 0x7fec441f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec4b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec441f3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec4b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec44158770 0x7fec4415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec4b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440b0120 con 0x7fec440b3b40 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec4b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440b5d70 con 0x7fec44158770 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec4b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4415f3b0 con 0x7fec4406dda0 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec441f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec441f3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47846/0 (socket says 172.21.15.114:47846) 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 -- 172.21.15.114:0/2754969890 learned_addr learned my addr 172.21.15.114:0/2754969890 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:14.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43e00640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec4406dda0 0x7fec441f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 -- 172.21.15.114:0/2754969890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec4406dda0 msgr2=0x7fec441f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec4406dda0 0x7fec441f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 -- 172.21.15.114:0/2754969890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec44158770 msgr2=0x7fec4415ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec44158770 0x7fec4415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.268+0000 7fec43400640 1 -- 172.21.15.114:0/2754969890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec441f3df0 con 0x7fec440b3b40 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.269+0000 7fec43400640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec441f3530 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009890 tx=0x7fec2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.269+0000 7fec3be00640 1 -- 172.21.15.114:0/2754969890 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c041070 con 0x7fec440b3b40 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.269+0000 7fec3be00640 1 -- 172.21.15.114:0/2754969890 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c03aa30 con 0x7fec440b3b40 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.269+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec441f40d0 con 0x7fec440b3b40 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.269+0000 7fec3be00640 1 -- 172.21.15.114:0/2754969890 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c035040 con 0x7fec440b3b40 2024-03-06T07:30:14.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.269+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec441f6ba0 con 0x7fec440b3b40 2024-03-06T07:30:14.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.270+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4404f330 con 0x7fec440b3b40 2024-03-06T07:30:14.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.271+0000 7fec3be00640 1 -- 172.21.15.114:0/2754969890 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec2c038020 con 0x7fec440b3b40 2024-03-06T07:30:14.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.272+0000 7fec3be00640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec2006b4d0 0x7fec2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.273+0000 7fec3be00640 1 -- 172.21.15.114:0/2754969890 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec2c0abc70 con 0x7fec440b3b40 2024-03-06T07:30:14.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.273+0000 7fec43e00640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec2006b4d0 0x7fec2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.273+0000 7fec43e00640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec2006b4d0 0x7fec2006d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fec340012e0 tx=0x7fec34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.277+0000 7fec3be00640 1 -- 172.21.15.114:0/2754969890 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec2c07ab80 con 0x7fec440b3b40 2024-03-06T07:30:14.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.565+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec441f6fc0 con 0x7fec440b3b40 2024-03-06T07:30:14.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.566+0000 7fec3be00640 1 -- 172.21.15.114:0/2754969890 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec2c07a480 con 0x7fec440b3b40 2024-03-06T07:30:14.567 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec2006b4d0 msgr2=0x7fec2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec2006b4d0 0x7fec2006d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fec340012e0 tx=0x7fec34009040 comp rx=0 tx=0).stop 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 msgr2=0x7fec441f3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec441f3530 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009890 tx=0x7fec2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 shutdown_connections 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec44158770 0x7fec4415ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec2006b4d0 0x7fec2006d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec440b3b40 0x7fec441f3530 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 --2- 172.21.15.114:0/2754969890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec4406dda0 0x7fec441f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 >> 172.21.15.114:0/2754969890 conn(0x7fec440b2840 msgr2=0x7fec440b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 shutdown_connections 2024-03-06T07:30:14.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.568+0000 7fec4b200640 1 -- 172.21.15.114:0/2754969890 wait complete. 2024-03-06T07:30:14.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:14.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.780+0000 7f3889000640 1 Processor -- start 2024-03-06T07:30:14.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.780+0000 7f3889000640 1 -- start start 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3889000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f38840b5490 0x7f38840cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3889000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840b3730 con 0x7f38840d07f0 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3889000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3884067b20 con 0x7f388415e450 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3889000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840d4930 con 0x7f38840b5490 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3881e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f38840d07f0 0x7f38840d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56268/0 (socket says 172.21.15.114:56268) 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3881e00640 1 -- 172.21.15.114:0/2589471091 learned_addr learned my addr 172.21.15.114:0/2589471091 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3882800640 1 --2- 172.21.15.114:0/2589471091 >> v2:172.21.15.177:3300/0 conn(0x7f38840b5490 0x7f38840cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389216476 0 0) 0x7f38840b3730 con 0x7f38840d07f0 2024-03-06T07:30:14.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.782+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3868003520 con 0x7f38840d07f0 2024-03-06T07:30:14.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3488749997 0 0) 0x7f3868003520 con 0x7f38840d07f0 2024-03-06T07:30:14.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 >> v2:172.21.15.177:3300/0 conn(0x7f38840b5490 msgr2=0x7f38840cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 --2- 172.21.15.114:0/2589471091 >> v2:172.21.15.177:3300/0 conn(0x7f38840b5490 0x7f38840cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 >> v1:172.21.15.177:6789/0 conn(0x7f388415e450 legacy=0x7f38840cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38840d4b80 con 0x7f38840d07f0 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f3874001670 con 0x7f38840d07f0 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 >> v1:172.21.15.114:6789/0 conn(0x7f38840d07f0 legacy=0x7f38840d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3868003a00 0x7f3868003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3868004800 0x7f3868004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3868005210 0x7f386800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840d4930 con 0x7f3868004800 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386800c160 con 0x7f3868003a00 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386800c360 con 0x7f3868005210 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.783+0000 7f3881400640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3868004800 0x7f3868004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.784+0000 7f3881400640 1 -- 172.21.15.114:0/2589471091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3868005210 msgr2=0x7f386800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.784+0000 7f3881400640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3868005210 0x7f386800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.784+0000 7f3881400640 1 -- 172.21.15.114:0/2589471091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3868003a00 msgr2=0x7f3868003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.784+0000 7f3881400640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3868003a00 0x7f3868003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.784+0000 7f3881400640 1 -- 172.21.15.114:0/2589471091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3868003520 con 0x7f3868004800 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.784+0000 7f3882800640 1 -- 172.21.15.114:0/2589471091 reap_dead start 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3881400640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3868004800 0x7f3868004c80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f388415e830 tx=0x7f3868005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e380fbc0c75398 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870001370 con 0x7f3868004800 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38700052d0 con 0x7f3868004800 2024-03-06T07:30:14.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3880a00640 1 -- 172.21.15.114:0/2589471091 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870003830 con 0x7f3868004800 2024-03-06T07:30:14.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3889000640 1 -- 172.21.15.114:0/2589471091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3868004800 msgr2=0x7f3868004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3889000640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3868004800 0x7f3868004c80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f388415e830 tx=0x7f3868005830 comp rx=0 tx=0).stop 2024-03-06T07:30:14.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3889000640 1 -- 172.21.15.114:0/2589471091 shutdown_connections 2024-03-06T07:30:14.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3889000640 1 --2- 172.21.15.114:0/2589471091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3868004800 0x7f3868004c80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3889000640 1 -- 172.21.15.114:0/2589471091 >> 172.21.15.114:0/2589471091 conn(0x7f38840d3e10 msgr2=0x7f38840d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.785+0000 7f3889000640 1 -- 172.21.15.114:0/2589471091 shutdown_connections 2024-03-06T07:30:14.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.786+0000 7f3889000640 1 -- 172.21.15.114:0/2589471091 wait complete. 2024-03-06T07:30:14.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.787+0000 7f3889000640 1 Processor -- start 2024-03-06T07:30:14.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.787+0000 7f3889000640 1 -- start start 2024-03-06T07:30:14.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3889000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3868004800 0x7f388409de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3889000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38840a0fd0 0x7f38840a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3889000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38840b75f0 0x7f38840b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3889000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840b3f60 con 0x7f38840b75f0 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3889000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840b3c80 con 0x7f3868004800 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3889000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38840d1bf0 con 0x7f38840a0fd0 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3882800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38840b75f0 0x7f38840b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3882800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38840b75f0 0x7f38840b7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47864/0 (socket says 172.21.15.114:47864) 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.788+0000 7f3882800640 1 -- 172.21.15.114:0/3262537234 learned_addr learned my addr 172.21.15.114:0/3262537234 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3882800640 1 -- 172.21.15.114:0/3262537234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38840a0fd0 msgr2=0x7f38840a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3882800640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38840a0fd0 0x7f38840a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3882800640 1 -- 172.21.15.114:0/3262537234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3868004800 msgr2=0x7f388409de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3882800640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3868004800 0x7f388409de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3882800640 1 -- 172.21.15.114:0/3262537234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388409ff30 con 0x7f38840b75f0 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3882800640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38840b75f0 0x7f38840b7a80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f388408e120 tx=0x7f387c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f387a000640 1 -- 172.21.15.114:0/3262537234 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c00bc80 con 0x7f38840b75f0 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f387a000640 1 -- 172.21.15.114:0/3262537234 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c01a070 con 0x7f38840b75f0 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f387a000640 1 -- 172.21.15.114:0/3262537234 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c014640 con 0x7f38840b75f0 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38840a0210 con 0x7f38840b75f0 2024-03-06T07:30:14.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.789+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38840a07d0 con 0x7f38840b75f0 2024-03-06T07:30:14.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.790+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3884072be0 con 0x7f38840b75f0 2024-03-06T07:30:14.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.792+0000 7f387a000640 1 -- 172.21.15.114:0/3262537234 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f387c013070 con 0x7f38840b75f0 2024-03-06T07:30:14.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.792+0000 7f387a000640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.792+0000 7f387a000640 1 -- 172.21.15.114:0/3262537234 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f387c08a600 con 0x7f38840b75f0 2024-03-06T07:30:14.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.793+0000 7f3881e00640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.793+0000 7f3881e00640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f386406b4d0 0x7f386406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f38840686c0 tx=0x7f3874001500 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.797+0000 7f387a000640 1 -- 172.21.15.114:0/3262537234 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3884072be0 con 0x7f38840b75f0 2024-03-06T07:30:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.981+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f38840d21d0 con 0x7f38840b75f0 2024-03-06T07:30:14.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.982+0000 7f387a000640 1 -- 172.21.15.114:0/3262537234 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) v1 ==== 100+0+11 (secure 0 0 0) 0x7f387c00be20 con 0x7f38840b75f0 2024-03-06T07:30:14.984 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:30:14.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.984+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f386406b4d0 msgr2=0x7f386406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.984+0000 7f3889000640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f386406b4d0 0x7f386406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f38840686c0 tx=0x7f3874001500 comp rx=0 tx=0).stop 2024-03-06T07:30:14.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.984+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38840b75f0 msgr2=0x7f38840b7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.984+0000 7f3889000640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38840b75f0 0x7f38840b7a80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f388408e120 tx=0x7f387c00b480 comp rx=0 tx=0).stop 2024-03-06T07:30:14.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 shutdown_connections 2024-03-06T07:30:14.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38840b75f0 0x7f38840b7a80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38840a0fd0 0x7f38840a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 --2- 172.21.15.114:0/3262537234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3868004800 0x7f388409de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 >> 172.21.15.114:0/3262537234 conn(0x7f38840d3e10 msgr2=0x7f38840b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 shutdown_connections 2024-03-06T07:30:14.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:14.985+0000 7f3889000640 1 -- 172.21.15.114:0/3262537234 wait complete. 2024-03-06T07:30:15.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:15.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:15.126 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:15.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.203+0000 7fae49600640 1 Processor -- start 2024-03-06T07:30:15.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.203+0000 7fae49600640 1 -- start start 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.204+0000 7fae49600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae4415e160 0x7fae44162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.204+0000 7fae49600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fae440af820 0x7fae440b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.204+0000 7fae49600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae44162b60 con 0x7fae440b38f0 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.204+0000 7fae49600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4406b0f0 con 0x7fae4415e160 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.204+0000 7fae49600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae440b5620 con 0x7fae440af820 2024-03-06T07:30:15.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae42a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae4415e160 0x7fae44162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae42000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fae440af820 0x7fae440b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae43400640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fae440b38f0 0x7fae4406d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:59662/0 (socket says 172.21.15.114:59662) 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae43400640 1 -- 172.21.15.114:0/2548939417 learned_addr learned my addr 172.21.15.114:0/2548939417 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae42000640 1 -- 172.21.15.114:0/2548939417 >> v2:172.21.15.114:3300/0 conn(0x7fae4415e160 msgr2=0x7fae44162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae42000640 1 --2- 172.21.15.114:0/2548939417 >> v2:172.21.15.114:3300/0 conn(0x7fae4415e160 0x7fae44162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae42000640 1 -- 172.21.15.114:0/2548939417 >> v1:172.21.15.177:6789/0 conn(0x7fae440b38f0 legacy=0x7fae4406d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae42000640 1 -- 172.21.15.114:0/2548939417 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae440b3da0 con 0x7fae440af820 2024-03-06T07:30:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.205+0000 7fae42a00640 1 --2- 172.21.15.114:0/2548939417 >> v2:172.21.15.114:3300/0 conn(0x7fae4415e160 0x7fae44162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:15.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.207+0000 7fae42000640 1 --2- 172.21.15.114:0/2548939417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440af820 0x7fae440b33b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae440d11b0 tx=0x7fae3002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fcdd7a426db7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.207+0000 7fae41600640 1 -- 172.21.15.114:0/2548939417 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae300315a0 con 0x7fae440af820 2024-03-06T07:30:15.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.207+0000 7fae41600640 1 -- 172.21.15.114:0/2548939417 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae30037070 con 0x7fae440af820 2024-03-06T07:30:15.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.207+0000 7fae41600640 1 -- 172.21.15.114:0/2548939417 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae30031960 con 0x7fae440af820 2024-03-06T07:30:15.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.207+0000 7fae49600640 1 -- 172.21.15.114:0/2548939417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440af820 msgr2=0x7fae440b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.207+0000 7fae49600640 1 --2- 172.21.15.114:0/2548939417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440af820 0x7fae440b33b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae440d11b0 tx=0x7fae3002b840 comp rx=0 tx=0).stop 2024-03-06T07:30:15.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.208+0000 7fae49600640 1 -- 172.21.15.114:0/2548939417 shutdown_connections 2024-03-06T07:30:15.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.208+0000 7fae49600640 1 --2- 172.21.15.114:0/2548939417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440af820 0x7fae440b33b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.208+0000 7fae49600640 1 --2- 172.21.15.114:0/2548939417 >> v2:172.21.15.114:3300/0 conn(0x7fae4415e160 0x7fae44162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.208+0000 7fae49600640 1 -- 172.21.15.114:0/2548939417 >> 172.21.15.114:0/2548939417 conn(0x7fae440d0340 msgr2=0x7fae440d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.208+0000 7fae49600640 1 -- 172.21.15.114:0/2548939417 shutdown_connections 2024-03-06T07:30:15.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.208+0000 7fae49600640 1 -- 172.21.15.114:0/2548939417 wait complete. 2024-03-06T07:30:15.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.209+0000 7fae49600640 1 Processor -- start 2024-03-06T07:30:15.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.209+0000 7fae49600640 1 -- start start 2024-03-06T07:30:15.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.210+0000 7fae49600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae440af820 0x7fae440ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.210+0000 7fae49600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440b38f0 0x7fae44151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.210+0000 7fae49600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae4415e160 0x7fae44151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.210+0000 7fae49600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4406ec30 con 0x7fae4415e160 2024-03-06T07:30:15.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.210+0000 7fae49600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae440afce0 con 0x7fae440b38f0 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.210+0000 7fae49600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4406ef10 con 0x7fae440af820 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae4415e160 0x7fae44151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae4415e160 0x7fae44151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47884/0 (socket says 172.21.15.114:47884) 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 -- 172.21.15.114:0/3295456769 learned_addr learned my addr 172.21.15.114:0/3295456769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae42a00640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae440af820 0x7fae440ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae42000640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440b38f0 0x7fae44151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 -- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae440af820 msgr2=0x7fae440ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae440af820 0x7fae440ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 -- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440b38f0 msgr2=0x7fae44151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440b38f0 0x7fae44151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 -- 172.21.15.114:0/3295456769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae440cd110 con 0x7fae4415e160 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae42a00640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae440af820 0x7fae440ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae42000640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440b38f0 0x7fae44151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.211+0000 7fae43400640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae4415e160 0x7fae44151430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00ca70 tx=0x7fae3c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.212+0000 7fae3aa00640 1 -- 172.21.15.114:0/3295456769 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c017070 con 0x7fae4415e160 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.212+0000 7fae3aa00640 1 -- 172.21.15.114:0/3295456769 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c00abd0 con 0x7fae4415e160 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.212+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae440cd3f0 con 0x7fae4415e160 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.212+0000 7fae3aa00640 1 -- 172.21.15.114:0/3295456769 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c012400 con 0x7fae4415e160 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.212+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae440cd9b0 con 0x7fae4415e160 2024-03-06T07:30:15.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.213+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4404f330 con 0x7fae4415e160 2024-03-06T07:30:15.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.214+0000 7fae3aa00640 1 -- 172.21.15.114:0/3295456769 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae3c01a070 con 0x7fae4415e160 2024-03-06T07:30:15.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.215+0000 7fae3aa00640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae2406b4d0 0x7fae2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.215+0000 7fae3aa00640 1 -- 172.21.15.114:0/3295456769 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae3c089d80 con 0x7fae4415e160 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.215+0000 7fae42a00640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae2406b4d0 0x7fae2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.216+0000 7fae42a00640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae2406b4d0 0x7fae2406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fae4415fa00 tx=0x7fae34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.222+0000 7fae3aa00640 1 -- 172.21.15.114:0/3295456769 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae44162b60 con 0x7fae4415e160 2024-03-06T07:30:15.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.456+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae4415d8c0 con 0x7fae4415e160 2024-03-06T07:30:15.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.456+0000 7fae3aa00640 1 -- 172.21.15.114:0/3295456769 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fae3c0585b0 con 0x7fae4415e160 2024-03-06T07:30:15.458 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:15.458 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4721,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/308840931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":308840931},{"type":"v1","addr":"172.21.15.177:6833","nonce":308840931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/4238819696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":4238819696},{"type":"v1","addr":"172.21.15.177:6835","nonce":4238819696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4960,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/379671784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":379671784},{"type":"v1","addr":"172.21.15.114:6835","nonce":379671784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:49.994729+0000","modified":"2024-03-06T07:29:53.062242+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5568":{"gid":5568,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2123564678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":2123564678},{"type":"v1","addr":"172.21.15.114:6837","nonce":2123564678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5568,"qdb_cluster":[5568]},"id":3}]} 2024-03-06T07:30:15.458 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 26 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae2406b4d0 msgr2=0x7fae2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae2406b4d0 0x7fae2406d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fae4415fa00 tx=0x7fae34008040 comp rx=0 tx=0).stop 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae4415e160 msgr2=0x7fae44151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae4415e160 0x7fae44151430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00ca70 tx=0x7fae3c00cf30 comp rx=0 tx=0).stop 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 shutdown_connections 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae2406b4d0 0x7fae2406d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae4415e160 0x7fae44151430 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae440b38f0 0x7fae44151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 --2- 172.21.15.114:0/3295456769 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae440af820 0x7fae440ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 >> 172.21.15.114:0/3295456769 conn(0x7fae440d0340 msgr2=0x7fae440b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 shutdown_connections 2024-03-06T07:30:15.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.459+0000 7fae49600640 1 -- 172.21.15.114:0/3295456769 wait complete. 2024-03-06T07:30:15.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:15.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.675+0000 7f01d6e00640 1 Processor -- start 2024-03-06T07:30:15.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.676+0000 7f01d6e00640 1 -- start start 2024-03-06T07:30:15.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.677+0000 7f01d6e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f01d0144c00 0x7f01d006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.677+0000 7f01d6e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01d015f9d0 0x7f01d015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.677+0000 7f01d6e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d006ee70 con 0x7f01d00ce0a0 2024-03-06T07:30:15.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.677+0000 7f01d6e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d006cef0 con 0x7f01d015f9d0 2024-03-06T07:30:15.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.677+0000 7f01d6e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d006d1d0 con 0x7f01d0144c00 2024-03-06T07:30:15.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01c7400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01d015f9d0 0x7f01d015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01c7400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01d015f9d0 0x7f01d015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47890/0 (socket says 172.21.15.114:47890) 2024-03-06T07:30:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01c7400640 1 -- 172.21.15.114:0/4003186789 learned_addr learned my addr 172.21.15.114:0/4003186789 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01cfe00640 1 --2- 172.21.15.114:0/4003186789 >> v2:172.21.15.177:3300/0 conn(0x7f01d0144c00 0x7f01d006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01c7400640 1 -- 172.21.15.114:0/4003186789 >> v2:172.21.15.177:3300/0 conn(0x7f01d0144c00 msgr2=0x7f01d006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01c7400640 1 --2- 172.21.15.114:0/4003186789 >> v2:172.21.15.177:3300/0 conn(0x7f01d0144c00 0x7f01d006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01c7400640 1 -- 172.21.15.114:0/4003186789 >> v1:172.21.15.177:6789/0 conn(0x7f01d00ce0a0 legacy=0x7f01d00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.678+0000 7f01c7400640 1 -- 172.21.15.114:0/4003186789 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d006d4b0 con 0x7f01d015f9d0 2024-03-06T07:30:15.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.680+0000 7f01c7400640 1 --2- 172.21.15.114:0/4003186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d015f9d0 0x7f01d015fe60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f01d00b25f0 tx=0x7f01c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1373d40464f9ec68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.680+0000 7f01cf400640 1 -- 172.21.15.114:0/4003186789 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0035020 con 0x7f01d015f9d0 2024-03-06T07:30:15.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.680+0000 7f01cf400640 1 -- 172.21.15.114:0/4003186789 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c0031870 con 0x7f01d015f9d0 2024-03-06T07:30:15.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.680+0000 7f01cf400640 1 -- 172.21.15.114:0/4003186789 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0031c40 con 0x7f01d015f9d0 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 -- 172.21.15.114:0/4003186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d015f9d0 msgr2=0x7f01d015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 --2- 172.21.15.114:0/4003186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d015f9d0 0x7f01d015fe60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f01d00b25f0 tx=0x7f01c002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 -- 172.21.15.114:0/4003186789 shutdown_connections 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 --2- 172.21.15.114:0/4003186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d015f9d0 0x7f01d015fe60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 --2- 172.21.15.114:0/4003186789 >> v2:172.21.15.177:3300/0 conn(0x7f01d0144c00 0x7f01d006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 -- 172.21.15.114:0/4003186789 >> 172.21.15.114:0/4003186789 conn(0x7f01d00b1730 msgr2=0x7f01d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 -- 172.21.15.114:0/4003186789 shutdown_connections 2024-03-06T07:30:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.681+0000 7f01d6e00640 1 -- 172.21.15.114:0/4003186789 wait complete. 2024-03-06T07:30:15.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.684+0000 7f01d6e00640 1 Processor -- start 2024-03-06T07:30:15.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.684+0000 7f01d6e00640 1 -- start start 2024-03-06T07:30:15.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01d6e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01d00ce0a0 0x7f01d01393e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01d6e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0144c00 0x7f01d013e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01d6e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01d015f9d0 0x7f01d0142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01d6e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d006cb50 con 0x7f01d0144c00 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01d6e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0160730 con 0x7f01d015f9d0 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01d6e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d006ccd0 con 0x7f01d00ce0a0 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01c7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0144c00 0x7f01d013e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01c7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0144c00 0x7f01d013e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47898/0 (socket says 172.21.15.114:47898) 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.686+0000 7f01c7400640 1 -- 172.21.15.114:0/3192930687 learned_addr learned my addr 172.21.15.114:0/3192930687 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01cfe00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01d00ce0a0 0x7f01d01393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01c7400640 1 -- 172.21.15.114:0/3192930687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01d00ce0a0 msgr2=0x7f01d01393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01c7400640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01d00ce0a0 0x7f01d01393e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01c7400640 1 -- 172.21.15.114:0/3192930687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01d015f9d0 msgr2=0x7f01d0142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01c7400640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01d015f9d0 0x7f01d0142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01c7400640 1 -- 172.21.15.114:0/3192930687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d0139920 con 0x7f01d0144c00 2024-03-06T07:30:15.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01c7400640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0144c00 0x7f01d013e200 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f01c002b950 tx=0x7f01c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01ccc00640 1 -- 172.21.15.114:0/3192930687 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0040070 con 0x7f01d0144c00 2024-03-06T07:30:15.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01ccc00640 1 -- 172.21.15.114:0/3192930687 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c0030d00 con 0x7f01d0144c00 2024-03-06T07:30:15.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01ccc00640 1 -- 172.21.15.114:0/3192930687 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0035040 con 0x7f01d0144c00 2024-03-06T07:30:15.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.687+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d013d440 con 0x7f01d0144c00 2024-03-06T07:30:15.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.688+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d013da00 con 0x7f01d0144c00 2024-03-06T07:30:15.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.689+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d004f330 con 0x7f01d0144c00 2024-03-06T07:30:15.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.690+0000 7f01ccc00640 1 -- 172.21.15.114:0/3192930687 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01c004b060 con 0x7f01d0144c00 2024-03-06T07:30:15.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.690+0000 7f01ccc00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f01ac06b4d0 0x7f01ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.691+0000 7f01ccc00640 1 -- 172.21.15.114:0/3192930687 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01c00abfd0 con 0x7f01d0144c00 2024-03-06T07:30:15.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.691+0000 7f01cfe00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f01ac06b4d0 0x7f01ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.692+0000 7f01cfe00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f01ac06b4d0 0x7f01ac06d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f01b8006fd0 tx=0x7f01b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.695+0000 7f01ccc00640 1 -- 172.21.15.114:0/3192930687 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01d0139920 con 0x7f01d0144c00 2024-03-06T07:30:15.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.918+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01d01600a0 con 0x7f01d0144c00 2024-03-06T07:30:15.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.921+0000 7f01ccc00640 1 -- 172.21.15.114:0/3192930687 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f01d006ee70 con 0x7f01d0144c00 2024-03-06T07:30:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:15.923 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:51.547619+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:47.835295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:49.012214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:15.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.923+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f01ac06b4d0 msgr2=0x7f01ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f01ac06b4d0 0x7f01ac06d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f01b8006fd0 tx=0x7f01b8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0144c00 msgr2=0x7f01d013e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0144c00 0x7f01d013e200 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f01c002b950 tx=0x7f01c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 shutdown_connections 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01d015f9d0 0x7f01d0142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f01ac06b4d0 0x7f01ac06d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0144c00 0x7f01d013e200 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 --2- 172.21.15.114:0/3192930687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01d00ce0a0 0x7f01d01393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 >> 172.21.15.114:0/3192930687 conn(0x7f01d00b1730 msgr2=0x7f01d0145370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 shutdown_connections 2024-03-06T07:30:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:15.924+0000 7f01d6e00640 1 -- 172.21.15.114:0/3192930687 wait complete. 2024-03-06T07:30:15.946 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:30:15.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:16.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.142+0000 7ff146000640 1 Processor -- start 2024-03-06T07:30:16.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.142+0000 7ff146000640 1 -- start start 2024-03-06T07:30:16.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.143+0000 7ff146000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff14015f9d0 0x7ff14015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.143+0000 7ff146000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff140070b10 0x7ff140070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.143+0000 7ff146000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14006cef0 con 0x7ff140144c00 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.143+0000 7ff146000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14006d1d0 con 0x7ff14015f9d0 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.143+0000 7ff146000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14006d4b0 con 0x7ff140070b10 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff137e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff14015f9d0 0x7ff14015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff137e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff14015f9d0 0x7ff14015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47900/0 (socket says 172.21.15.114:47900) 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff137e00640 1 -- 172.21.15.114:0/3384371564 learned_addr learned my addr 172.21.15.114:0/3384371564 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff13fe00640 1 --2- 172.21.15.114:0/3384371564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff140070b10 0x7ff140070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff137e00640 1 -- 172.21.15.114:0/3384371564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff140070b10 msgr2=0x7ff140070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff137e00640 1 --2- 172.21.15.114:0/3384371564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff140070b10 0x7ff140070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff137e00640 1 -- 172.21.15.114:0/3384371564 >> v1:172.21.15.177:6789/0 conn(0x7ff140144c00 legacy=0x7ff14006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.144+0000 7ff137e00640 1 -- 172.21.15.114:0/3384371564 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14006d790 con 0x7ff14015f9d0 2024-03-06T07:30:16.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.145+0000 7ff137e00640 1 --2- 172.21.15.114:0/3384371564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff14015f9d0 0x7ff14015fe60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff128009870 tx=0x7ff12802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2965948a72607d4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.145+0000 7ff13ea00640 1 -- 172.21.15.114:0/3384371564 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128035020 con 0x7ff14015f9d0 2024-03-06T07:30:16.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff13ea00640 1 -- 172.21.15.114:0/3384371564 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff128031640 con 0x7ff14015f9d0 2024-03-06T07:30:16.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff13ea00640 1 -- 172.21.15.114:0/3384371564 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12802d040 con 0x7ff14015f9d0 2024-03-06T07:30:16.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 -- 172.21.15.114:0/3384371564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff14015f9d0 msgr2=0x7ff14015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 --2- 172.21.15.114:0/3384371564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff14015f9d0 0x7ff14015fe60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff128009870 tx=0x7ff12802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 -- 172.21.15.114:0/3384371564 shutdown_connections 2024-03-06T07:30:16.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 --2- 172.21.15.114:0/3384371564 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff140070b10 0x7ff140070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 --2- 172.21.15.114:0/3384371564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff14015f9d0 0x7ff14015fe60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 -- 172.21.15.114:0/3384371564 >> 172.21.15.114:0/3384371564 conn(0x7ff1400b1730 msgr2=0x7ff1400b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 -- 172.21.15.114:0/3384371564 shutdown_connections 2024-03-06T07:30:16.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.146+0000 7ff146000640 1 -- 172.21.15.114:0/3384371564 wait complete. 2024-03-06T07:30:16.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.148+0000 7ff146000640 1 Processor -- start 2024-03-06T07:30:16.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.148+0000 7ff146000640 1 -- start start 2024-03-06T07:30:16.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff146000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff140070b10 0x7ff14013b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff146000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff140144c00 0x7ff14013bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff146000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff14015f9d0 0x7ff140139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff146000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14006ccd0 con 0x7ff140144c00 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff146000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14006cb50 con 0x7ff140070b10 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff146000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14006df90 con 0x7ff14015f9d0 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff137e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff140144c00 0x7ff14013bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff137e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff140144c00 0x7ff14013bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47906/0 (socket says 172.21.15.114:47906) 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.149+0000 7ff137e00640 1 -- 172.21.15.114:0/3147038514 learned_addr learned my addr 172.21.15.114:0/3147038514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137e00640 1 -- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff14015f9d0 msgr2=0x7ff140139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff13fe00640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff14015f9d0 0x7ff140139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137e00640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff14015f9d0 0x7ff140139ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff13f400640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff140070b10 0x7ff14013b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137e00640 1 -- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff140070b10 msgr2=0x7ff14013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137e00640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff140070b10 0x7ff14013b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137e00640 1 -- 172.21.15.114:0/3147038514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14013aef0 con 0x7ff140144c00 2024-03-06T07:30:16.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137e00640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff140144c00 0x7ff14013bda0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff128000c00 tx=0x7ff1280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137400640 1 -- 172.21.15.114:0/3147038514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128040070 con 0x7ff140144c00 2024-03-06T07:30:16.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137400640 1 -- 172.21.15.114:0/3147038514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1280399f0 con 0x7ff140144c00 2024-03-06T07:30:16.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff137400640 1 -- 172.21.15.114:0/3147038514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128035040 con 0x7ff140144c00 2024-03-06T07:30:16.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.150+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14013ac10 con 0x7ff140144c00 2024-03-06T07:30:16.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.151+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1401e59d0 con 0x7ff140144c00 2024-03-06T07:30:16.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.152+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff140067830 con 0x7ff140144c00 2024-03-06T07:30:16.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.153+0000 7ff137400640 1 -- 172.21.15.114:0/3147038514 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff128038020 con 0x7ff140144c00 2024-03-06T07:30:16.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.154+0000 7ff137400640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff11c06b4d0 0x7ff11c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.154+0000 7ff137400640 1 -- 172.21.15.114:0/3147038514 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1280abb90 con 0x7ff140144c00 2024-03-06T07:30:16.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.155+0000 7ff13f400640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff11c06b4d0 0x7ff11c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.155+0000 7ff13f400640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff11c06b4d0 0x7ff11c06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff130003d90 tx=0x7ff130009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.159+0000 7ff137400640 1 -- 172.21.15.114:0/3147038514 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff140067830 con 0x7ff140144c00 2024-03-06T07:30:16.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.383+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff140139a50 con 0x7ff140144c00 2024-03-06T07:30:16.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.384+0000 7ff137400640 1 -- 172.21.15.114:0/3147038514 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4983 (secure 0 0 0) 0x7ff12807aaa0 con 0x7ff140144c00 2024-03-06T07:30:16.385 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:16.386 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4721,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/308840931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":308840931},{"type":"v1","addr":"172.21.15.177:6833","nonce":308840931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/4238819696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":4238819696},{"type":"v1","addr":"172.21.15.177:6835","nonce":4238819696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4960,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/379671784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":379671784},{"type":"v1","addr":"172.21.15.114:6835","nonce":379671784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:49.994729+0000","modified":"2024-03-06T07:29:53.062242+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5568":{"gid":5568,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2123564678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":2123564678},{"type":"v1","addr":"172.21.15.114:6837","nonce":2123564678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5568,"qdb_cluster":[5568]},"id":3}]} 2024-03-06T07:30:16.386 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 26 2024-03-06T07:30:16.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.386+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff11c06b4d0 msgr2=0x7ff11c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.386+0000 7ff146000640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff11c06b4d0 0x7ff11c06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff130003d90 tx=0x7ff130009040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.386+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff140144c00 msgr2=0x7ff14013bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.386+0000 7ff146000640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff140144c00 0x7ff14013bda0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff128000c00 tx=0x7ff1280046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 shutdown_connections 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff14015f9d0 0x7ff140139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff11c06b4d0 0x7ff11c06d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff140144c00 0x7ff14013bda0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 --2- 172.21.15.114:0/3147038514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff140070b10 0x7ff14013b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 >> 172.21.15.114:0/3147038514 conn(0x7ff1400b1730 msgr2=0x7ff1400ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 shutdown_connections 2024-03-06T07:30:16.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.387+0000 7ff146000640 1 -- 172.21.15.114:0/3147038514 wait complete. 2024-03-06T07:30:16.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.593+0000 7fb332800640 1 Processor -- start 2024-03-06T07:30:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.593+0000 7fb332800640 1 -- start start 2024-03-06T07:30:16.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb332800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb32c161580 0x7fb32c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb332800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb32c0d1e90 0x7fb32c0d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb332800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb332800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c161960 con 0x7fb32c161580 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb332800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0cf6f0 con 0x7fb32c0d1e90 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb332800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0cf9d0 con 0x7fb32c0d28a0 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb331400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb32c161580 0x7fb32c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb331400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb32c161580 0x7fb32c0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47918/0 (socket says 172.21.15.114:47918) 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb331400640 1 -- 172.21.15.114:0/2525912676 learned_addr learned my addr 172.21.15.114:0/2525912676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb330a00640 1 --2- 172.21.15.114:0/2525912676 >> v2:172.21.15.177:3300/0 conn(0x7fb32c0d1e90 0x7fb32c0d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb331e00640 1 --2- 172.21.15.114:0/2525912676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb331400640 1 -- 172.21.15.114:0/2525912676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 msgr2=0x7fb32c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.595+0000 7fb331400640 1 --2- 172.21.15.114:0/2525912676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c0d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.596+0000 7fb331400640 1 -- 172.21.15.114:0/2525912676 >> v2:172.21.15.177:3300/0 conn(0x7fb32c0d1e90 msgr2=0x7fb32c0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.596+0000 7fb331400640 1 --2- 172.21.15.114:0/2525912676 >> v2:172.21.15.177:3300/0 conn(0x7fb32c0d1e90 0x7fb32c0d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.596+0000 7fb331400640 1 -- 172.21.15.114:0/2525912676 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0cfcb0 con 0x7fb32c161580 2024-03-06T07:30:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.597+0000 7fb331400640 1 --2- 172.21.15.114:0/2525912676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c161580 0x7fb32c0d1950 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb32c08c9d0 tx=0x7fb32002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe19064c4192734 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.597+0000 7fb32be00640 1 -- 172.21.15.114:0/2525912676 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32002ca70 con 0x7fb32c161580 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.597+0000 7fb32be00640 1 -- 172.21.15.114:0/2525912676 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32002cc10 con 0x7fb32c161580 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.597+0000 7fb32be00640 1 -- 172.21.15.114:0/2525912676 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3200343f0 con 0x7fb32c161580 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.597+0000 7fb332800640 1 -- 172.21.15.114:0/2525912676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c161580 msgr2=0x7fb32c0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.597+0000 7fb332800640 1 --2- 172.21.15.114:0/2525912676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c161580 0x7fb32c0d1950 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb32c08c9d0 tx=0x7fb32002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.598+0000 7fb332800640 1 -- 172.21.15.114:0/2525912676 shutdown_connections 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.598+0000 7fb332800640 1 --2- 172.21.15.114:0/2525912676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.598+0000 7fb332800640 1 --2- 172.21.15.114:0/2525912676 >> v2:172.21.15.177:3300/0 conn(0x7fb32c0d1e90 0x7fb32c0d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.598+0000 7fb332800640 1 --2- 172.21.15.114:0/2525912676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c161580 0x7fb32c0d1950 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.598+0000 7fb332800640 1 -- 172.21.15.114:0/2525912676 >> 172.21.15.114:0/2525912676 conn(0x7fb32c0d3d80 msgr2=0x7fb32c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.598+0000 7fb332800640 1 -- 172.21.15.114:0/2525912676 shutdown_connections 2024-03-06T07:30:16.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.598+0000 7fb332800640 1 -- 172.21.15.114:0/2525912676 wait complete. 2024-03-06T07:30:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.599+0000 7fb332800640 1 Processor -- start 2024-03-06T07:30:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.599+0000 7fb332800640 1 -- start start 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.600+0000 7fb332800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c0d1e90 0x7fb32c0c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.600+0000 7fb332800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c1f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.600+0000 7fb332800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c161580 0x7fb32c0b8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.600+0000 7fb332800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0cf3a0 con 0x7fb32c0d1e90 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.600+0000 7fb332800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0b39a0 con 0x7fb32c161580 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.600+0000 7fb332800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0d0c20 con 0x7fb32c0d28a0 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c0d1e90 0x7fb32c0c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c0d1e90 0x7fb32c0c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47934/0 (socket says 172.21.15.114:47934) 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 -- 172.21.15.114:0/3452142273 learned_addr learned my addr 172.21.15.114:0/3452142273 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331e00640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c161580 0x7fb32c0b8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 -- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 msgr2=0x7fb32c1f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb330a00640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c1f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c1f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 -- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c161580 msgr2=0x7fb32c0b8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c161580 0x7fb32c0b8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 -- 172.21.15.114:0/3452142273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0b7da0 con 0x7fb32c0d1e90 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.601+0000 7fb331400640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c0d1e90 0x7fb32c0c7e20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb320000f80 tx=0x7fb320004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.602+0000 7fb329600640 1 -- 172.21.15.114:0/3452142273 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320043020 con 0x7fb32c0d1e90 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.602+0000 7fb329600640 1 -- 172.21.15.114:0/3452142273 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32003e450 con 0x7fb32c0d1e90 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.602+0000 7fb329600640 1 -- 172.21.15.114:0/3452142273 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32003e840 con 0x7fb32c0d1e90 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.602+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c0b8080 con 0x7fb32c0d1e90 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.602+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c0b8640 con 0x7fb32c0d1e90 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.603+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c04f330 con 0x7fb32c0d1e90 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.604+0000 7fb329600640 1 -- 172.21.15.114:0/3452142273 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb320046020 con 0x7fb32c0d1e90 2024-03-06T07:30:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.605+0000 7fb329600640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb30c06b4d0 0x7fb30c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.605+0000 7fb329600640 1 -- 172.21.15.114:0/3452142273 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3200acaf0 con 0x7fb32c0d1e90 2024-03-06T07:30:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.605+0000 7fb330a00640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb30c06b4d0 0x7fb30c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.606+0000 7fb330a00640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb30c06b4d0 0x7fb30c06d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb318006fd0 tx=0x7fb318008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.610+0000 7fb329600640 1 -- 172.21.15.114:0/3452142273 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb32004d030 con 0x7fb32c0d1e90 2024-03-06T07:30:16.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.846+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb32c0d3b60 con 0x7fb32c0d1e90 2024-03-06T07:30:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.847+0000 7fb329600640 1 -- 172.21.15.114:0/3452142273 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fb32007b9e0 con 0x7fb32c0d1e90 2024-03-06T07:30:16.848 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:16.848 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4721,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/308840931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":308840931},{"type":"v1","addr":"172.21.15.177:6833","nonce":308840931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/4238819696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":4238819696},{"type":"v1","addr":"172.21.15.177:6835","nonce":4238819696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4960,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/379671784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":379671784},{"type":"v1","addr":"172.21.15.114:6835","nonce":379671784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:49.994729+0000","modified":"2024-03-06T07:29:53.062242+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5568":{"gid":5568,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2123564678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":2123564678},{"type":"v1","addr":"172.21.15.114:6837","nonce":2123564678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5568,"qdb_cluster":[5568]},"id":3}]} 2024-03-06T07:30:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 26 2024-03-06T07:30:16.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb30c06b4d0 msgr2=0x7fb30c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb30c06b4d0 0x7fb30c06d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb318006fd0 tx=0x7fb318008040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c0d1e90 msgr2=0x7fb32c0c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c0d1e90 0x7fb32c0c7e20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb320000f80 tx=0x7fb320004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 shutdown_connections 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb32c161580 0x7fb32c0b8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb32c0d28a0 0x7fb32c1f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb30c06b4d0 0x7fb30c06d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.849+0000 7fb332800640 1 --2- 172.21.15.114:0/3452142273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb32c0d1e90 0x7fb32c0c7e20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.850+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 >> 172.21.15.114:0/3452142273 conn(0x7fb32c0d3d80 msgr2=0x7fb32c0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.850+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 shutdown_connections 2024-03-06T07:30:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:16.850+0000 7fb332800640 1 -- 172.21.15.114:0/3452142273 wait complete. 2024-03-06T07:30:16.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.078+0000 7fc0a9800640 1 Processor -- start 2024-03-06T07:30:17.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.079+0000 7fc0a9800640 1 -- start start 2024-03-06T07:30:17.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.080+0000 7fc0a9800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc0a415a940 0x7fc0a40b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.080+0000 7fc0a9800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a40b2b30 con 0x7fc0a40cf370 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.080+0000 7fc0a9800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a40b2e10 con 0x7fc0a40cc2a0 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.080+0000 7fc0a9800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a40b28e0 con 0x7fc0a415a940 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.080+0000 7fc0a3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc0a415a940 0x7fc0a40b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.081+0000 7fc0a3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc0a415a940 0x7fc0a40b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47948/0 (socket says 172.21.15.114:47948) 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.081+0000 7fc09b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc0a40cf370 0x7fc0a406d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56270/0 (socket says 172.21.15.114:56270) 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.081+0000 7fc0a3e00640 1 -- 172.21.15.114:0/1103878216 learned_addr learned my addr 172.21.15.114:0/1103878216 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.081+0000 7fc0a3e00640 1 -- 172.21.15.114:0/1103878216 >> v1:172.21.15.177:6789/0 conn(0x7fc0a40cc2a0 legacy=0x7fc0a40cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.081+0000 7fc0a3e00640 1 -- 172.21.15.114:0/1103878216 >> v1:172.21.15.114:6789/0 conn(0x7fc0a40cf370 legacy=0x7fc0a406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.081+0000 7fc0a3e00640 1 -- 172.21.15.114:0/1103878216 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a406ca50 con 0x7fc0a415a940 2024-03-06T07:30:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.082+0000 7fc0a3e00640 1 --2- 172.21.15.114:0/1103878216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a415a940 0x7fc0a40b45e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc094009870 tx=0x7fc09402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89d07497822266d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.082+0000 7fc0a3400640 1 -- 172.21.15.114:0/1103878216 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094034070 con 0x7fc0a415a940 2024-03-06T07:30:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.082+0000 7fc0a3400640 1 -- 172.21.15.114:0/1103878216 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc094030860 con 0x7fc0a415a940 2024-03-06T07:30:17.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.082+0000 7fc0a3400640 1 -- 172.21.15.114:0/1103878216 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094030c30 con 0x7fc0a415a940 2024-03-06T07:30:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.083+0000 7fc0a9800640 1 -- 172.21.15.114:0/1103878216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a415a940 msgr2=0x7fc0a40b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.083+0000 7fc0a9800640 1 --2- 172.21.15.114:0/1103878216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a415a940 0x7fc0a40b45e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc094009870 tx=0x7fc09402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.083+0000 7fc0a9800640 1 -- 172.21.15.114:0/1103878216 shutdown_connections 2024-03-06T07:30:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.083+0000 7fc0a9800640 1 --2- 172.21.15.114:0/1103878216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a415a940 0x7fc0a40b45e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.083+0000 7fc0a9800640 1 -- 172.21.15.114:0/1103878216 >> 172.21.15.114:0/1103878216 conn(0x7fc0a406afc0 msgr2=0x7fc0a406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.083+0000 7fc0a9800640 1 -- 172.21.15.114:0/1103878216 shutdown_connections 2024-03-06T07:30:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.083+0000 7fc0a9800640 1 -- 172.21.15.114:0/1103878216 wait complete. 2024-03-06T07:30:17.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.085+0000 7fc0a9800640 1 Processor -- start 2024-03-06T07:30:17.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.085+0000 7fc0a9800640 1 -- start start 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a9800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0a40cc2a0 0x7fc0a41f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a40cf370 0x7fc0a4148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a415a940 0x7fc0a4145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a9800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40b4b20 con 0x7fc0a40cf370 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a40b30f0 con 0x7fc0a415a940 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a9800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a415ad20 con 0x7fc0a40cc2a0 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc09b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a40cf370 0x7fc0a4148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc09b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a40cf370 0x7fc0a4148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47954/0 (socket says 172.21.15.114:47954) 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc09b400640 1 -- 172.21.15.114:0/567602656 learned_addr learned my addr 172.21.15.114:0/567602656 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a8e00640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a415a940 0x7fc0a4145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc09b400640 1 -- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0a40cc2a0 msgr2=0x7fc0a41f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc0a3e00640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0a40cc2a0 0x7fc0a41f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc09b400640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0a40cc2a0 0x7fc0a41f3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.086+0000 7fc09b400640 1 -- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a415a940 msgr2=0x7fc0a4145360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc09b400640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a415a940 0x7fc0a4145360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc09b400640 1 -- 172.21.15.114:0/567602656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a4148c40 con 0x7fc0a40cf370 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc09b400640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a40cf370 0x7fc0a4148700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00c400 tx=0x7fc08c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc0a0c00640 1 -- 172.21.15.114:0/567602656 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0074e0 con 0x7fc0a40cf370 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc0a0c00640 1 -- 172.21.15.114:0/567602656 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c007680 con 0x7fc0a40cf370 2024-03-06T07:30:17.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a41f40a0 con 0x7fc0a40cf370 2024-03-06T07:30:17.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc0a0c00640 1 -- 172.21.15.114:0/567602656 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c004a00 con 0x7fc0a40cf370 2024-03-06T07:30:17.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.087+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a41f4940 con 0x7fc0a40cf370 2024-03-06T07:30:17.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.088+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a41f4270 con 0x7fc0a40cf370 2024-03-06T07:30:17.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.090+0000 7fc0a0c00640 1 -- 172.21.15.114:0/567602656 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc08c01a020 con 0x7fc0a40cf370 2024-03-06T07:30:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.091+0000 7fc0a0c00640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc08006b4d0 0x7fc08006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.091+0000 7fc0a0c00640 1 -- 172.21.15.114:0/567602656 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc08c089a20 con 0x7fc0a40cf370 2024-03-06T07:30:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.091+0000 7fc0a3e00640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc08006b4d0 0x7fc08006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.092+0000 7fc0a3e00640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc08006b4d0 0x7fc08006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc094036000 tx=0x7fc09403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.096+0000 7fc0a0c00640 1 -- 172.21.15.114:0/567602656 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0a4148c40 con 0x7fc0a40cf370 2024-03-06T07:30:17.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.294+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0a404f330 con 0x7fc0a40cf370 2024-03-06T07:30:17.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.299+0000 7fc0a0c00640 1 -- 172.21.15.114:0/567602656 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16661 (secure 0 0 0) 0x7fc08c010300 con 0x7fc0a40cf370 2024-03-06T07:30:17.301 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:17.301 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:29:51.547619+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:47.835295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:49.012214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc08006b4d0 msgr2=0x7fc08006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc08006b4d0 0x7fc08006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc094036000 tx=0x7fc09403b040 comp rx=0 tx=0).stop 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a40cf370 msgr2=0x7fc0a4148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a40cf370 0x7fc0a4148700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00c400 tx=0x7fc08c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 shutdown_connections 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc0a415a940 0x7fc0a4145360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc08006b4d0 0x7fc08006d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a40cf370 0x7fc0a4148700 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 --2- 172.21.15.114:0/567602656 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc0a40cc2a0 0x7fc0a41f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 >> 172.21.15.114:0/567602656 conn(0x7fc0a406afc0 msgr2=0x7fc0a4070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 shutdown_connections 2024-03-06T07:30:17.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.302+0000 7fc0a9800640 1 -- 172.21.15.114:0/567602656 wait complete. 2024-03-06T07:30:17.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:30:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.525+0000 7f567f000640 1 Processor -- start 2024-03-06T07:30:17.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.525+0000 7f567f000640 1 -- start start 2024-03-06T07:30:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.526+0000 7f567f000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56780c4930 0x7f56780c6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f567f000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f56780c5880 0x7f5678165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f567f000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5678145b30 con 0x7f56780c4d10 2024-03-06T07:30:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f567f000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5678142ce0 con 0x7f56780c4930 2024-03-06T07:30:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f567f000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56780cbca0 con 0x7f56780c5880 2024-03-06T07:30:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f5677e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56780c4930 0x7f56780c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f566fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f56780c4d10 0x7f56780c5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56282/0 (socket says 172.21.15.114:56282) 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f566fe00640 1 -- 172.21.15.114:0/3000877221 learned_addr learned my addr 172.21.15.114:0/3000877221 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f567d000640 1 --2- 172.21.15.114:0/3000877221 >> v2:172.21.15.177:3300/0 conn(0x7f56780c5880 0x7f5678165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f5677e00640 1 -- 172.21.15.114:0/3000877221 >> v2:172.21.15.177:3300/0 conn(0x7f56780c5880 msgr2=0x7f5678165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f5677e00640 1 --2- 172.21.15.114:0/3000877221 >> v2:172.21.15.177:3300/0 conn(0x7f56780c5880 0x7f5678165b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.527+0000 7f5677e00640 1 -- 172.21.15.114:0/3000877221 >> v1:172.21.15.114:6789/0 conn(0x7f56780c4d10 legacy=0x7f56780c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.528+0000 7f5677400640 1 -- 172.21.15.114:0/3000877221 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135935712 0 0) 0x7f5660001110 con 0x7f56780c4d10 2024-03-06T07:30:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.528+0000 7f5677e00640 1 -- 172.21.15.114:0/3000877221 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56780cbf80 con 0x7f56780c4930 2024-03-06T07:30:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.529+0000 7f5677e00640 1 --2- 172.21.15.114:0/3000877221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4930 0x7f56780c6460 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5678073310 tx=0x7f566802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90319cc891944ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.529+0000 7f5677400640 1 -- 172.21.15.114:0/3000877221 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56680315a0 con 0x7f56780c4930 2024-03-06T07:30:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.529+0000 7f5677400640 1 -- 172.21.15.114:0/3000877221 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668037070 con 0x7f56780c4930 2024-03-06T07:30:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.529+0000 7f5677400640 1 -- 172.21.15.114:0/3000877221 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668031960 con 0x7f56780c4930 2024-03-06T07:30:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 -- 172.21.15.114:0/3000877221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4930 msgr2=0x7f56780c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 --2- 172.21.15.114:0/3000877221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4930 0x7f56780c6460 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5678073310 tx=0x7f566802b840 comp rx=0 tx=0).stop 2024-03-06T07:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 -- 172.21.15.114:0/3000877221 shutdown_connections 2024-03-06T07:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 --2- 172.21.15.114:0/3000877221 >> v2:172.21.15.177:3300/0 conn(0x7f56780c5880 0x7f5678165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 --2- 172.21.15.114:0/3000877221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4930 0x7f56780c6460 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 -- 172.21.15.114:0/3000877221 >> 172.21.15.114:0/3000877221 conn(0x7f56780cc4f0 msgr2=0x7f56780cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 -- 172.21.15.114:0/3000877221 shutdown_connections 2024-03-06T07:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.530+0000 7f567f000640 1 -- 172.21.15.114:0/3000877221 wait complete. 2024-03-06T07:30:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.532+0000 7f567f000640 1 Processor -- start 2024-03-06T07:30:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.532+0000 7f567f000640 1 -- start start 2024-03-06T07:30:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f567f000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780c4930 0x7f56780b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f567f000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4d10 0x7f56780b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f567f000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56780c5880 0x7f56780b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f567f000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567806e600 con 0x7f56780c4d10 2024-03-06T07:30:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f567f000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780ca3b0 con 0x7f56780c4930 2024-03-06T07:30:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f567f000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56780cc260 con 0x7f56780c5880 2024-03-06T07:30:17.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f566fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4d10 0x7f56780b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f566fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4d10 0x7f56780b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47978/0 (socket says 172.21.15.114:47978) 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f566fe00640 1 -- 172.21.15.114:0/2426189211 learned_addr learned my addr 172.21.15.114:0/2426189211 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f5677e00640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780c4930 0x7f56780b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.534+0000 7f567d000640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56780c5880 0x7f56780b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f566fe00640 1 -- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56780c5880 msgr2=0x7f56780b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f566fe00640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56780c5880 0x7f56780b1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f566fe00640 1 -- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780c4930 msgr2=0x7f56780b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f566fe00640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780c4930 0x7f56780b0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f566fe00640 1 -- 172.21.15.114:0/2426189211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56780ce230 con 0x7f56780c4d10 2024-03-06T07:30:17.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f566fe00640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4d10 0x7f56780b0ac0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f566000c780 tx=0x7f566000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f5674c00640 1 -- 172.21.15.114:0/2426189211 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5660018070 con 0x7f56780c4d10 2024-03-06T07:30:17.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f5674c00640 1 -- 172.21.15.114:0/2426189211 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56600072d0 con 0x7f56780c4d10 2024-03-06T07:30:17.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f5674c00640 1 -- 172.21.15.114:0/2426189211 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56600047e0 con 0x7f56780c4d10 2024-03-06T07:30:17.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56780ce510 con 0x7f56780c4d10 2024-03-06T07:30:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.535+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56780cead0 con 0x7f56780c4d10 2024-03-06T07:30:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.537+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567804f330 con 0x7f56780c4d10 2024-03-06T07:30:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.537+0000 7f5674c00640 1 -- 172.21.15.114:0/2426189211 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5660004980 con 0x7f56780c4d10 2024-03-06T07:30:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.538+0000 7f5674c00640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565406b4d0 0x7f565406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.538+0000 7f5674c00640 1 -- 172.21.15.114:0/2426189211 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56600895a0 con 0x7f56780c4d10 2024-03-06T07:30:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.539+0000 7f5677e00640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565406b4d0 0x7f565406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.539+0000 7f5677e00640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565406b4d0 0x7f565406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5668036000 tx=0x7f5668008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.543+0000 7f5674c00640 1 -- 172.21.15.114:0/2426189211 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5678145b30 con 0x7f56780c4d10 2024-03-06T07:30:17.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.775+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5678143dc0 con 0x7f56780c4d10 2024-03-06T07:30:17.778 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:30:17.777+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:30:17.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.780+0000 7f5674c00640 1 -- 172.21.15.114:0/2426189211 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5660004c70 con 0x7f56780c4d10 2024-03-06T07:30:17.781 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:30:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.783+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565406b4d0 msgr2=0x7f565406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.783+0000 7f567f000640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565406b4d0 0x7f565406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5668036000 tx=0x7f5668008030 comp rx=0 tx=0).stop 2024-03-06T07:30:17.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.783+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4d10 msgr2=0x7f56780b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.783+0000 7f567f000640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4d10 0x7f56780b0ac0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f566000c780 tx=0x7f566000cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 shutdown_connections 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56780c5880 0x7f56780b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565406b4d0 0x7f565406d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56780c4d10 0x7f56780b0ac0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 --2- 172.21.15.114:0/2426189211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780c4930 0x7f56780b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 >> 172.21.15.114:0/2426189211 conn(0x7f56780cc4f0 msgr2=0x7f56781448a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 shutdown_connections 2024-03-06T07:30:17.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:17.784+0000 7f567f000640 1 -- 172.21.15.114:0/2426189211 wait complete. 2024-03-06T07:30:17.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:30:17.864 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:30:18.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.003+0000 7f36d5a00640 1 Processor -- start 2024-03-06T07:30:18.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.003+0000 7f36d5a00640 1 -- start start 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.004+0000 7f36d5a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f36d00ccb80 0x7f36d006b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.004+0000 7f36d5a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d0161b00 0x7f36d0161f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.004+0000 7f36d5a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d015e790 con 0x7f36d0066d00 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.004+0000 7f36d5a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d0158950 con 0x7f36d00ccb80 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.004+0000 7f36d5a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d015e530 con 0x7f36d0161b00 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36cfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f36d00ccb80 0x7f36d006b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36c7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f36d0066d00 0x7f36d00ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39848/0 (socket says 172.21.15.114:39848) 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36c7e00640 1 -- 172.21.15.114:0/2506201596 learned_addr learned my addr 172.21.15.114:0/2506201596 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36cfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f36d00ccb80 0x7f36d006b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40934/0 (socket says 172.21.15.114:40934) 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36d5000640 1 --2- 172.21.15.114:0/2506201596 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d0161b00 0x7f36d0161f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36cfe00640 1 -- 172.21.15.114:0/2506201596 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d0161b00 msgr2=0x7f36d0161f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36cfe00640 1 --2- 172.21.15.114:0/2506201596 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d0161b00 0x7f36d0161f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36cfe00640 1 -- 172.21.15.114:0/2506201596 >> v1:172.21.15.114:6789/0 conn(0x7f36d0066d00 legacy=0x7f36d00ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36cfe00640 1 -- 172.21.15.114:0/2506201596 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d015ea70 con 0x7f36d00ccb80 2024-03-06T07:30:18.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36cf400640 1 -- 172.21.15.114:0/2506201596 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304174745 0 0) 0x7f36d015e790 con 0x7f36d0066d00 2024-03-06T07:30:18.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.005+0000 7f36d5000640 1 --2- 172.21.15.114:0/2506201596 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d0161b00 0x7f36d0161f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36cfe00640 1 --2- 172.21.15.114:0/2506201596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d00ccb80 0x7f36d006b9d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f36c00097d0 tx=0x7f36c002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbebe2d147443722 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36cf400640 1 -- 172.21.15.114:0/2506201596 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0035020 con 0x7f36d00ccb80 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36cf400640 1 -- 172.21.15.114:0/2506201596 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0031800 con 0x7f36d00ccb80 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36cf400640 1 -- 172.21.15.114:0/2506201596 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0031bd0 con 0x7f36d00ccb80 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36d5a00640 1 -- 172.21.15.114:0/2506201596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d00ccb80 msgr2=0x7f36d006b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2506201596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d00ccb80 0x7f36d006b9d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f36c00097d0 tx=0x7f36c002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36d5a00640 1 -- 172.21.15.114:0/2506201596 shutdown_connections 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2506201596 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d0161b00 0x7f36d0161f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.007+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2506201596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d00ccb80 0x7f36d006b9d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.008+0000 7f36d5a00640 1 -- 172.21.15.114:0/2506201596 >> 172.21.15.114:0/2506201596 conn(0x7f36d00b1d00 msgr2=0x7f36d00b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.008+0000 7f36d5a00640 1 -- 172.21.15.114:0/2506201596 shutdown_connections 2024-03-06T07:30:18.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.008+0000 7f36d5a00640 1 -- 172.21.15.114:0/2506201596 wait complete. 2024-03-06T07:30:18.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.009+0000 7f36d5a00640 1 Processor -- start 2024-03-06T07:30:18.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.009+0000 7f36d5a00640 1 -- start start 2024-03-06T07:30:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.010+0000 7f36d5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d0066d00 0x7f36d00c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.010+0000 7f36d5a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d00ccb80 0x7f36d0094760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.010+0000 7f36d5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0161b00 0x7f36d00ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.010+0000 7f36d5a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0067160 con 0x7f36d0161b00 2024-03-06T07:30:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.010+0000 7f36d5a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d015f030 con 0x7f36d0066d00 2024-03-06T07:30:18.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.010+0000 7f36d5a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d00af6d0 con 0x7f36d00ccb80 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0161b00 0x7f36d00ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0161b00 0x7f36d00ae670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40948/0 (socket says 172.21.15.114:40948) 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 -- 172.21.15.114:0/2500681432 learned_addr learned my addr 172.21.15.114:0/2500681432 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36cfe00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d0066d00 0x7f36d00c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36c7e00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d00ccb80 0x7f36d0094760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 -- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d00ccb80 msgr2=0x7f36d0094760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d00ccb80 0x7f36d0094760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 -- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d0066d00 msgr2=0x7f36d00c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d0066d00 0x7f36d00c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.011+0000 7f36d5000640 1 -- 172.21.15.114:0/2500681432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d00c5710 con 0x7f36d0161b00 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.012+0000 7f36d5000640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0161b00 0x7f36d00ae670 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f36c800cd80 tx=0x7f36c800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.012+0000 7f36ccc00640 1 -- 172.21.15.114:0/2500681432 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c800f680 con 0x7f36d0161b00 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.012+0000 7f36ccc00640 1 -- 172.21.15.114:0/2500681432 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c800f840 con 0x7f36d0161b00 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.012+0000 7f36ccc00640 1 -- 172.21.15.114:0/2500681432 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c800fc30 con 0x7f36d0161b00 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.012+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d00c4b90 con 0x7f36d0161b00 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.012+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d00c5150 con 0x7f36d0161b00 2024-03-06T07:30:18.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.013+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d004f330 con 0x7f36d0161b00 2024-03-06T07:30:18.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.014+0000 7f36ccc00640 1 -- 172.21.15.114:0/2500681432 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36c80213f0 con 0x7f36d0161b00 2024-03-06T07:30:18.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.015+0000 7f36ccc00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36ac06b4d0 0x7f36ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.015+0000 7f36ccc00640 1 -- 172.21.15.114:0/2500681432 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f36c808b120 con 0x7f36d0161b00 2024-03-06T07:30:18.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.015+0000 7f36cfe00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36ac06b4d0 0x7f36ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.016+0000 7f36cfe00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36ac06b4d0 0x7f36ac06d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f36c0036000 tx=0x7f36c0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.020+0000 7f36ccc00640 1 -- 172.21.15.114:0/2500681432 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36c8014030 con 0x7f36d0161b00 2024-03-06T07:30:18.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.257+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f36d0071420 con 0x7f36d0161b00 2024-03-06T07:30:18.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.783+0000 7f36ccc00640 1 -- 172.21.15.114:0/2500681432 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f36c8059ea0 con 0x7f36d0161b00 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.787+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36ac06b4d0 msgr2=0x7f36ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.787+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36ac06b4d0 0x7f36ac06d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f36c0036000 tx=0x7f36c0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.787+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0161b00 msgr2=0x7f36d00ae670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.787+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0161b00 0x7f36d00ae670 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f36c800cd80 tx=0x7f36c800a650 comp rx=0 tx=0).stop 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 shutdown_connections 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36ac06b4d0 0x7f36ac06d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0161b00 0x7f36d00ae670 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36d00ccb80 0x7f36d0094760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 --2- 172.21.15.114:0/2500681432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36d0066d00 0x7f36d00c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 >> 172.21.15.114:0/2500681432 conn(0x7f36d00b1d00 msgr2=0x7f36d0074130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 shutdown_connections 2024-03-06T07:30:18.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:18.788+0000 7f36d5a00640 1 -- 172.21.15.114:0/2500681432 wait complete. 2024-03-06T07:30:18.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.017+0000 7f105d000640 1 Processor -- start 2024-03-06T07:30:19.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.018+0000 7f105d000640 1 -- start start 2024-03-06T07:30:19.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.019+0000 7f105d000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 0x7f1058166ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.019+0000 7f105d000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1058072970 con 0x7f10580cd450 2024-03-06T07:30:19.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.019+0000 7f105d000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580cdbd0 con 0x7f1058145750 2024-03-06T07:30:19.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.019+0000 7f105d000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10580cd980 con 0x7f105806dd30 2024-03-06T07:30:19.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f104d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f10580cd450 0x7f105806d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39858/0 (socket says 172.21.15.114:39858) 2024-03-06T07:30:19.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f104d400640 1 -- 172.21.15.114:0/1399383842 learned_addr learned my addr 172.21.15.114:0/1399383842 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:19.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f1056800640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 0x7f1058166ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397658830 0 0) 0x7f1058072970 con 0x7f10580cd450 2024-03-06T07:30:19.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c003520 con 0x7f10580cd450 2024-03-06T07:30:19.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941621846 0 0) 0x7f103c003520 con 0x7f10580cd450 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 msgr2=0x7f1058166ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f1055400640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 0x7f1058166ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.020+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 >> v1:172.21.15.177:6789/0 conn(0x7f1058145750 legacy=0x7f1058072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580cdeb0 con 0x7f10580cd450 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f1040001670 con 0x7f10580cd450 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 >> v1:172.21.15.114:6789/0 conn(0x7f10580cd450 legacy=0x7f105806d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 0x7f103c003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103c003fd0 0x7f103c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c004d40 0x7f103c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580cd980 con 0x7f103c004d40 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c00bbb0 con 0x7f103c003fd0 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c00bdb0 con 0x7f105806dd30 2024-03-06T07:30:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1056800640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c004d40 0x7f103c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1055e00640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 0x7f103c003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f104d400640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103c003fd0 0x7f103c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1056800640 1 -- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 msgr2=0x7f103c003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1056800640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f105806dd30 0x7f103c003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1056800640 1 -- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103c003fd0 msgr2=0x7f103c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1056800640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103c003fd0 0x7f103c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f1056800640 1 -- 172.21.15.114:0/1399383842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c003520 con 0x7f103c004d40 2024-03-06T07:30:19.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.021+0000 7f104d400640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f103c003fd0 0x7f103c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:19.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.023+0000 7f1056800640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c004d40 0x7f103c0051c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1050009fc0 tx=0x7f103c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfae5b7de13b625a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.023+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050039070 con 0x7f103c004d40 2024-03-06T07:30:19.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.023+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105002cc20 con 0x7f103c004d40 2024-03-06T07:30:19.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.023+0000 7f1055400640 1 -- 172.21.15.114:0/1399383842 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050009c10 con 0x7f103c004d40 2024-03-06T07:30:19.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.023+0000 7f105d000640 1 -- 172.21.15.114:0/1399383842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c004d40 msgr2=0x7f103c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.023+0000 7f105d000640 1 --2- 172.21.15.114:0/1399383842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f103c004d40 0x7f103c0051c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1050009fc0 tx=0x7f103c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.023+0000 7f1056800640 1 -- 172.21.15.114:0/1399383842 reap_dead start 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.024+0000 7f105d000640 1 -- 172.21.15.114:0/1399383842 shutdown_connections 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.024+0000 7f105d000640 1 -- 172.21.15.114:0/1399383842 >> 172.21.15.114:0/1399383842 conn(0x7f10580cfa80 msgr2=0x7f10580cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.024+0000 7f105d000640 1 -- 172.21.15.114:0/1399383842 shutdown_connections 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.024+0000 7f105d000640 1 -- 172.21.15.114:0/1399383842 wait complete. 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.025+0000 7f105d000640 1 Processor -- start 2024-03-06T07:30:19.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.026+0000 7f105d000640 1 -- start start 2024-03-06T07:30:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.026+0000 7f105d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10581566b0 0x7f1058142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.026+0000 7f105d000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1058156a90 0x7f1058128dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.026+0000 7f105d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1058144410 0x7f1058129310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.026+0000 7f105d000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105815c440 con 0x7f10581566b0 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.026+0000 7f105d000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580ce470 con 0x7f1058144410 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.026+0000 7f105d000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105815c5c0 con 0x7f1058156a90 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10581566b0 0x7f1058142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10581566b0 0x7f1058142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40978/0 (socket says 172.21.15.114:40978) 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 -- 172.21.15.114:0/1116039131 learned_addr learned my addr 172.21.15.114:0/1116039131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1056800640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1058144410 0x7f1058129310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 -- 172.21.15.114:0/1116039131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1058156a90 msgr2=0x7f1058128dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1058156a90 0x7f1058128dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 -- 172.21.15.114:0/1116039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1058144410 msgr2=0x7f1058129310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1058144410 0x7f1058129310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 -- 172.21.15.114:0/1116039131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10581430c0 con 0x7f10581566b0 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.027+0000 7f1055e00640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10581566b0 0x7f1058142b80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f104800e580 tx=0x7f104800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.028+0000 7f104ea00640 1 -- 172.21.15.114:0/1116039131 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10480052e0 con 0x7f10581566b0 2024-03-06T07:30:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.028+0000 7f104ea00640 1 -- 172.21.15.114:0/1116039131 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1048018070 con 0x7f10581566b0 2024-03-06T07:30:19.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.028+0000 7f104ea00640 1 -- 172.21.15.114:0/1116039131 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048003810 con 0x7f10581566b0 2024-03-06T07:30:19.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.028+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1058143370 con 0x7f10581566b0 2024-03-06T07:30:19.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.028+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1058143930 con 0x7f10581566b0 2024-03-06T07:30:19.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.029+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105804f3a0 con 0x7f10581566b0 2024-03-06T07:30:19.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.030+0000 7f104ea00640 1 -- 172.21.15.114:0/1116039131 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10480039b0 con 0x7f10581566b0 2024-03-06T07:30:19.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.031+0000 7f104ea00640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f103406b500 0x7f103406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.032+0000 7f104ea00640 1 -- 172.21.15.114:0/1116039131 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1048089f70 con 0x7f10581566b0 2024-03-06T07:30:19.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.032+0000 7f104d400640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f103406b500 0x7f103406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.032+0000 7f104d400640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f103406b500 0x7f103406d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1040001eb0 tx=0x7f1040009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.036+0000 7f104ea00640 1 -- 172.21.15.114:0/1116039131 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1048058d00 con 0x7f10581566b0 2024-03-06T07:30:19.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.232+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10580d0750 con 0x7f10581566b0 2024-03-06T07:30:19.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.792+0000 7f104ea00640 1 -- 172.21.15.114:0/1116039131 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1048058600 con 0x7f10581566b0 2024-03-06T07:30:19.794 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:19.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f103406b500 msgr2=0x7f103406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f103406b500 0x7f103406d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1040001eb0 tx=0x7f1040009040 comp rx=0 tx=0).stop 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10581566b0 msgr2=0x7f1058142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10581566b0 0x7f1058142b80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f104800e580 tx=0x7f104800ea40 comp rx=0 tx=0).stop 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 shutdown_connections 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1058144410 0x7f1058129310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1058156a90 0x7f1058128dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f103406b500 0x7f103406d9c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 --2- 172.21.15.114:0/1116039131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10581566b0 0x7f1058142b80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 >> 172.21.15.114:0/1116039131 conn(0x7f10580cfa80 msgr2=0x7f1058140650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.797+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 shutdown_connections 2024-03-06T07:30:19.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:19.798+0000 7f105d000640 1 -- 172.21.15.114:0/1116039131 wait complete. 2024-03-06T07:30:19.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:20.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.029+0000 7efce1800640 1 Processor -- start 2024-03-06T07:30:20.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.029+0000 7efce1800640 1 -- start start 2024-03-06T07:30:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efce1800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efcdc0b5320 0x7efcdc0cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efce1800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efcdc0d18f0 0x7efcdc0d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efce1800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc0d1d80 con 0x7efcdc0cfb90 2024-03-06T07:30:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efce1800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc0d2990 con 0x7efcdc0b5320 2024-03-06T07:30:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efce1800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc0d2060 con 0x7efcdc0d18f0 2024-03-06T07:30:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efcdbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efcdc0b5320 0x7efcdc0cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efcdbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efcdc0b5320 0x7efcdc0cf650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40982/0 (socket says 172.21.15.114:40982) 2024-03-06T07:30:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efcdbe00640 1 -- 172.21.15.114:0/3125717899 learned_addr learned my addr 172.21.15.114:0/3125717899 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efcd3400640 1 --2- 172.21.15.114:0/3125717899 >> v2:172.21.15.177:3300/0 conn(0x7efcdc0d18f0 0x7efcdc0d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.031+0000 7efcdbe00640 1 -- 172.21.15.114:0/3125717899 >> v2:172.21.15.177:3300/0 conn(0x7efcdc0d18f0 msgr2=0x7efcdc0d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.032+0000 7efcdbe00640 1 --2- 172.21.15.114:0/3125717899 >> v2:172.21.15.177:3300/0 conn(0x7efcdc0d18f0 0x7efcdc0d0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.032+0000 7efcdbe00640 1 -- 172.21.15.114:0/3125717899 >> v1:172.21.15.177:6789/0 conn(0x7efcdc0cfb90 legacy=0x7efcdc161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.032+0000 7efcdbe00640 1 -- 172.21.15.114:0/3125717899 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0d2340 con 0x7efcdc0b5320 2024-03-06T07:30:20.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.033+0000 7efcdbe00640 1 --2- 172.21.15.114:0/3125717899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc0cf650 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7efcdc08c5c0 tx=0x7efccc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f3dd43c0ba8c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.033+0000 7efcdb400640 1 -- 172.21.15.114:0/3125717899 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc02ca80 con 0x7efcdc0b5320 2024-03-06T07:30:20.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.033+0000 7efcdb400640 1 -- 172.21.15.114:0/3125717899 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc02cc20 con 0x7efcdc0b5320 2024-03-06T07:30:20.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.033+0000 7efcdb400640 1 -- 172.21.15.114:0/3125717899 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc0343f0 con 0x7efcdc0b5320 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 -- 172.21.15.114:0/3125717899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 msgr2=0x7efcdc0cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 --2- 172.21.15.114:0/3125717899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc0cf650 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7efcdc08c5c0 tx=0x7efccc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 -- 172.21.15.114:0/3125717899 shutdown_connections 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 --2- 172.21.15.114:0/3125717899 >> v2:172.21.15.177:3300/0 conn(0x7efcdc0d18f0 0x7efcdc0d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 --2- 172.21.15.114:0/3125717899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc0cf650 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 -- 172.21.15.114:0/3125717899 >> 172.21.15.114:0/3125717899 conn(0x7efcdc0d3d20 msgr2=0x7efcdc0d4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 -- 172.21.15.114:0/3125717899 shutdown_connections 2024-03-06T07:30:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.034+0000 7efce1800640 1 -- 172.21.15.114:0/3125717899 wait complete. 2024-03-06T07:30:20.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.036+0000 7efce1800640 1 Processor -- start 2024-03-06T07:30:20.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.036+0000 7efce1800640 1 -- start start 2024-03-06T07:30:20.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efce1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc14f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efce1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcdc0cfb90 0x7efcdc0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efce1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcdc0cd210 0x7efcdc15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efce1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0b7d00 con 0x7efcdc0b5320 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efce1800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0b7b80 con 0x7efcdc0cfb90 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efce1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0d0c70 con 0x7efcdc0cd210 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efcdbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc14f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efcdbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc14f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40990/0 (socket says 172.21.15.114:40990) 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.037+0000 7efcdbe00640 1 -- 172.21.15.114:0/3454493971 learned_addr learned my addr 172.21.15.114:0/3454493971 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcd3400640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcdc0cfb90 0x7efcdc0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcdbe00640 1 -- 172.21.15.114:0/3454493971 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcdc0cd210 msgr2=0x7efcdc15aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcdbe00640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcdc0cd210 0x7efcdc15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcdbe00640 1 -- 172.21.15.114:0/3454493971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcdc0cfb90 msgr2=0x7efcdc0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcdbe00640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcdc0cfb90 0x7efcdc0cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcdbe00640 1 -- 172.21.15.114:0/3454493971 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0ca020 con 0x7efcdc0b5320 2024-03-06T07:30:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcdbe00640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc14f040 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7efccc036040 tx=0x7efccc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.038+0000 7efcd8c00640 1 -- 172.21.15.114:0/3454493971 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc043020 con 0x7efcdc0b5320 2024-03-06T07:30:20.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.039+0000 7efcd8c00640 1 -- 172.21.15.114:0/3454493971 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc03e450 con 0x7efcdc0b5320 2024-03-06T07:30:20.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.039+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc14f580 con 0x7efcdc0b5320 2024-03-06T07:30:20.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.039+0000 7efcd8c00640 1 -- 172.21.15.114:0/3454493971 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc03e840 con 0x7efcdc0b5320 2024-03-06T07:30:20.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.039+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc0c8510 con 0x7efcdc0b5320 2024-03-06T07:30:20.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.040+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc0cd630 con 0x7efcdc0b5320 2024-03-06T07:30:20.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.041+0000 7efcd8c00640 1 -- 172.21.15.114:0/3454493971 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efccc046020 con 0x7efcdc0b5320 2024-03-06T07:30:20.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.042+0000 7efcd8c00640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.042+0000 7efcd8c00640 1 -- 172.21.15.114:0/3454493971 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5357+0+0 (secure 0 0 0) 0x7efccc0acb50 con 0x7efcdc0b5320 2024-03-06T07:30:20.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.042+0000 7efcd3400640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.043+0000 7efcd3400640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0cc930 tx=0x7efcc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.047+0000 7efcd8c00640 1 -- 172.21.15.114:0/3454493971 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efccc04d030 con 0x7efcdc0b5320 2024-03-06T07:30:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.243+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efcdc04f330 con 0x7efcdc0b5320 2024-03-06T07:30:20.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.804+0000 7efcd8c00640 1 -- 172.21.15.114:0/3454493971 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) v1 ==== 169+0+0 (secure 0 0 0) 0x7efccc07ba00 con 0x7efcdc0b5320 2024-03-06T07:30:20.807 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.808+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 msgr2=0x7efcb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0cc930 tx=0x7efcc4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 msgr2=0x7efcdc14f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc14f040 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7efccc036040 tx=0x7efccc004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:20.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 shutdown_connections 2024-03-06T07:30:20.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcdc0cd210 0x7efcdc15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcdc0cfb90 0x7efcdc0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 --2- 172.21.15.114:0/3454493971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcdc0b5320 0x7efcdc14f040 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.809+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 >> 172.21.15.114:0/3454493971 conn(0x7efcdc0d3d20 msgr2=0x7efcdc0b9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.810+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 shutdown_connections 2024-03-06T07:30:20.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:20.810+0000 7efce1800640 1 -- 172.21.15.114:0/3454493971 wait complete. 2024-03-06T07:30:20.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:30:21.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.039+0000 7f1b1b000640 1 Processor -- start 2024-03-06T07:30:21.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.039+0000 7f1b1b000640 1 -- start start 2024-03-06T07:30:21.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b1b000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b1415e130 0x7f1b140cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b1b000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140cfaf0 0x7f1b140d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b1b000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b140d13e0 con 0x7f1b140d23e0 2024-03-06T07:30:21.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b1b000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b140cf120 con 0x7f1b1415e130 2024-03-06T07:30:21.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b1b000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140cead0 con 0x7f1b140cfaf0 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b19c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b1415e130 0x7f1b140cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b19c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b1415e130 0x7f1b140cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40998/0 (socket says 172.21.15.114:40998) 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b19c00640 1 -- 172.21.15.114:0/1057941236 learned_addr learned my addr 172.21.15.114:0/1057941236 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b19200640 1 --2- 172.21.15.114:0/1057941236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140cfaf0 0x7f1b140d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b19c00640 1 -- 172.21.15.114:0/1057941236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140cfaf0 msgr2=0x7f1b140d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.041+0000 7f1b19c00640 1 --2- 172.21.15.114:0/1057941236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140cfaf0 0x7f1b140d1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.042+0000 7f1b19c00640 1 -- 172.21.15.114:0/1057941236 >> v1:172.21.15.114:6789/0 conn(0x7f1b140d23e0 legacy=0x7f1b140c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.042+0000 7f1b19c00640 1 -- 172.21.15.114:0/1057941236 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b140cedb0 con 0x7f1b1415e130 2024-03-06T07:30:21.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.042+0000 7f1b19200640 1 --2- 172.21.15.114:0/1057941236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140cfaf0 0x7f1b140d1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:21.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.043+0000 7f1b19c00640 1 --2- 172.21.15.114:0/1057941236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b1415e130 0x7f1b140cf5b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009870 tx=0x7f1b0802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3855d840b67a82e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.043+0000 7f1b13e00640 1 -- 172.21.15.114:0/1057941236 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08037070 con 0x7f1b1415e130 2024-03-06T07:30:21.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.043+0000 7f1b13e00640 1 -- 172.21.15.114:0/1057941236 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b08004230 con 0x7f1b1415e130 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.043+0000 7f1b13e00640 1 -- 172.21.15.114:0/1057941236 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08003610 con 0x7f1b1415e130 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.043+0000 7f1b1b000640 1 -- 172.21.15.114:0/1057941236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b1415e130 msgr2=0x7f1b140cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.043+0000 7f1b1b000640 1 --2- 172.21.15.114:0/1057941236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b1415e130 0x7f1b140cf5b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009870 tx=0x7f1b0802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.044+0000 7f1b1b000640 1 -- 172.21.15.114:0/1057941236 shutdown_connections 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.044+0000 7f1b1b000640 1 --2- 172.21.15.114:0/1057941236 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140cfaf0 0x7f1b140d1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.044+0000 7f1b1b000640 1 --2- 172.21.15.114:0/1057941236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b1415e130 0x7f1b140cf5b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.044+0000 7f1b1b000640 1 -- 172.21.15.114:0/1057941236 >> 172.21.15.114:0/1057941236 conn(0x7f1b1409d070 msgr2=0x7f1b140d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.044+0000 7f1b1b000640 1 -- 172.21.15.114:0/1057941236 shutdown_connections 2024-03-06T07:30:21.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.044+0000 7f1b1b000640 1 -- 172.21.15.114:0/1057941236 wait complete. 2024-03-06T07:30:21.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.045+0000 7f1b1b000640 1 Processor -- start 2024-03-06T07:30:21.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.045+0000 7f1b1b000640 1 -- start start 2024-03-06T07:30:21.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.046+0000 7f1b1b000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b140cfaf0 0x7f1b1415e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.046+0000 7f1b1b000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140d23e0 0x7f1b140b9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.046+0000 7f1b1b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b1415e130 0x7f1b14149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.046+0000 7f1b1b000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140c8200 con 0x7f1b140cfaf0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.046+0000 7f1b1b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140c8080 con 0x7f1b1415e130 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.046+0000 7f1b1b000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140d06b0 con 0x7f1b140d23e0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b140cfaf0 0x7f1b1415e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b140cfaf0 0x7f1b1415e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41006/0 (socket says 172.21.15.114:41006) 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 -- 172.21.15.114:0/2008194565 learned_addr learned my addr 172.21.15.114:0/2008194565 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 -- 172.21.15.114:0/2008194565 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140d23e0 msgr2=0x7f1b140b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140d23e0 0x7f1b140b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 -- 172.21.15.114:0/2008194565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b1415e130 msgr2=0x7f1b14149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b1415e130 0x7f1b14149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 -- 172.21.15.114:0/2008194565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1415d8b0 con 0x7f1b140cfaf0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.047+0000 7f1b19c00640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b140cfaf0 0x7f1b1415e510 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009870 tx=0x7f1b080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.048+0000 7f1b11600640 1 -- 172.21.15.114:0/2008194565 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08041070 con 0x7f1b140cfaf0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.048+0000 7f1b11600640 1 -- 172.21.15.114:0/2008194565 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b08030730 con 0x7f1b140cfaf0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.048+0000 7f1b11600640 1 -- 172.21.15.114:0/2008194565 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08037040 con 0x7f1b140cfaf0 2024-03-06T07:30:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.048+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1415eaf0 con 0x7f1b140cfaf0 2024-03-06T07:30:21.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.048+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b140cc3f0 con 0x7f1b140cfaf0 2024-03-06T07:30:21.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.049+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1404f330 con 0x7f1b140cfaf0 2024-03-06T07:30:21.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.050+0000 7f1b11600640 1 -- 172.21.15.114:0/2008194565 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b08044070 con 0x7f1b140cfaf0 2024-03-06T07:30:21.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.051+0000 7f1b11600640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1af406b4d0 0x7f1af406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.051+0000 7f1b11600640 1 -- 172.21.15.114:0/2008194565 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1b080abcc0 con 0x7f1b140cfaf0 2024-03-06T07:30:21.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.051+0000 7f1b19200640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1af406b4d0 0x7f1af406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.052+0000 7f1b19200640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1af406b4d0 0x7f1af406d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b00006fd0 tx=0x7f1b00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.055+0000 7f1b11600640 1 -- 172.21.15.114:0/2008194565 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b1415d8b0 con 0x7f1b140cfaf0 2024-03-06T07:30:21.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.250+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1b1409d690 con 0x7f1b140cfaf0 2024-03-06T07:30:21.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.256+0000 7f1b11600640 1 -- 172.21.15.114:0/2008194565 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1b0807adb0 con 0x7f1b140cfaf0 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1af406b4d0 msgr2=0x7f1af406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1af406b4d0 0x7f1af406d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b00006fd0 tx=0x7f1b00008040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b140cfaf0 msgr2=0x7f1b1415e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b140cfaf0 0x7f1b1415e510 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009870 tx=0x7f1b080046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 shutdown_connections 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b1415e130 0x7f1b14149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1b140d23e0 0x7f1b140b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1af406b4d0 0x7f1af406d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 --2- 172.21.15.114:0/2008194565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b140cfaf0 0x7f1b1415e510 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 >> 172.21.15.114:0/2008194565 conn(0x7f1b1409d070 msgr2=0x7f1b140d1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 shutdown_connections 2024-03-06T07:30:21.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.259+0000 7f1b1b000640 1 -- 172.21.15.114:0/2008194565 wait complete. 2024-03-06T07:30:21.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:30:21.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.487+0000 7f3538000640 1 Processor -- start 2024-03-06T07:30:21.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.487+0000 7f3538000640 1 -- start start 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.488+0000 7f3538000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3530160700 0x7f3530070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.488+0000 7f3538000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35300b46c0 con 0x7f353015fc10 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.488+0000 7f3538000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35300b49a0 con 0x7f35300b3f70 2024-03-06T07:30:21.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.489+0000 7f3538000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3530072c90 con 0x7f3530160700 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.489+0000 7f3535600640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f353015fc10 0x7f353015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39864/0 (socket says 172.21.15.114:39864) 2024-03-06T07:30:21.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.489+0000 7f3535600640 1 -- 172.21.15.114:0/2904392882 learned_addr learned my addr 172.21.15.114:0/2904392882 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.489+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193938146 0 0) 0x7f35300b46c0 con 0x7f353015fc10 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.489+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3518003520 con 0x7f353015fc10 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.489+0000 7f3536000640 1 --2- 172.21.15.114:0/2904392882 >> v2:172.21.15.177:3300/0 conn(0x7f3530160700 0x7f3530070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.489+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 803465684 0 0) 0x7f3518003520 con 0x7f353015fc10 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 >> v2:172.21.15.177:3300/0 conn(0x7f3530160700 msgr2=0x7f3530070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 --2- 172.21.15.114:0/2904392882 >> v2:172.21.15.177:3300/0 conn(0x7f3530160700 0x7f3530070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 >> v1:172.21.15.177:6789/0 conn(0x7f35300b3f70 legacy=0x7f35300ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353009dd80 con 0x7f353015fc10 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f3524001670 con 0x7f353015fc10 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 >> v1:172.21.15.114:6789/0 conn(0x7f353015fc10 legacy=0x7f353015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180039b0 0x7f3518003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3518004800 0x7f3518004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35180051c0 0x7f351800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3530072c90 con 0x7f35180051c0 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351800c040 con 0x7f35180039b0 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351800c240 con 0x7f3518004800 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3536000640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35180051c0 0x7f351800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3535600640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180039b0 0x7f3518003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3536000640 1 -- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3518004800 msgr2=0x7f3518004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3536000640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3518004800 0x7f3518004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3536000640 1 -- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180039b0 msgr2=0x7f3518003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3536000640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180039b0 0x7f3518003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3536000640 1 -- 172.21.15.114:0/2904392882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3518003520 con 0x7f35180051c0 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.490+0000 7f3535600640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180039b0 0x7f3518003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.491+0000 7f3536000640 1 -- 172.21.15.114:0/2904392882 reap_dead start 2024-03-06T07:30:21.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.491+0000 7f3536000640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35180051c0 0x7f351800b9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3528009fc0 tx=0x7f3518005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec65d29e7039c1f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528009c10 con 0x7f35180051c0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352802cc60 con 0x7f35180051c0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f352fe00640 1 -- 172.21.15.114:0/2904392882 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528003bd0 con 0x7f35180051c0 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f3538000640 1 -- 172.21.15.114:0/2904392882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35180051c0 msgr2=0x7f351800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f3538000640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35180051c0 0x7f351800b9c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3528009fc0 tx=0x7f3518005830 comp rx=0 tx=0).stop 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f3538000640 1 -- 172.21.15.114:0/2904392882 shutdown_connections 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f3538000640 1 --2- 172.21.15.114:0/2904392882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35180051c0 0x7f351800b9c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f3538000640 1 -- 172.21.15.114:0/2904392882 >> 172.21.15.114:0/2904392882 conn(0x7f35300b1940 msgr2=0x7f35300b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f3538000640 1 -- 172.21.15.114:0/2904392882 shutdown_connections 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.492+0000 7f3538000640 1 -- 172.21.15.114:0/2904392882 wait complete. 2024-03-06T07:30:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.494+0000 7f3538000640 1 Processor -- start 2024-03-06T07:30:21.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.494+0000 7f3538000640 1 -- start start 2024-03-06T07:30:21.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3538000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180051c0 0x7f35301475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3538000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f353014fac0 0x7f353006e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3538000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3530151120 0x7f35301515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3538000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353015c460 con 0x7f353014fac0 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3538000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353015c2e0 con 0x7f35180051c0 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3538000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3530070ff0 con 0x7f3530151120 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3534c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f353014fac0 0x7f353006e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.495+0000 7f3534c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f353014fac0 0x7f353006e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41026/0 (socket says 172.21.15.114:41026) 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3534c00640 1 -- 172.21.15.114:0/777710849 learned_addr learned my addr 172.21.15.114:0/777710849 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3535600640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180051c0 0x7f35301475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3534c00640 1 -- 172.21.15.114:0/777710849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3530151120 msgr2=0x7f35301515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3534c00640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3530151120 0x7f35301515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3534c00640 1 -- 172.21.15.114:0/777710849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180051c0 msgr2=0x7f35301475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3534c00640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180051c0 0x7f35301475e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3534c00640 1 -- 172.21.15.114:0/777710849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353006e760 con 0x7f353014fac0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f3534c00640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f353014fac0 0x7f353006e220 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f351c00c3e0 tx=0x7f351c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.496+0000 7f352d600640 1 -- 172.21.15.114:0/777710849 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c011790 con 0x7f353014fac0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.497+0000 7f352d600640 1 -- 172.21.15.114:0/777710849 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351c017070 con 0x7f353014fac0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.497+0000 7f352d600640 1 -- 172.21.15.114:0/777710849 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c011bb0 con 0x7f353014fac0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.497+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353006cb30 con 0x7f353014fac0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.497+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353006d0f0 con 0x7f353014fac0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.498+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f353004f330 con 0x7f353014fac0 2024-03-06T07:30:21.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.499+0000 7f352d600640 1 -- 172.21.15.114:0/777710849 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f351c01c460 con 0x7f353014fac0 2024-03-06T07:30:21.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.500+0000 7f352d600640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f351006b4d0 0x7f351006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.500+0000 7f352d600640 1 -- 172.21.15.114:0/777710849 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f351c089df0 con 0x7f353014fac0 2024-03-06T07:30:21.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.500+0000 7f3535600640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f351006b4d0 0x7f351006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.501+0000 7f3535600640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f351006b4d0 0x7f351006d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f35240079d0 tx=0x7f3524009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.504+0000 7f352d600640 1 -- 172.21.15.114:0/777710849 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f351c058ed0 con 0x7f353014fac0 2024-03-06T07:30:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.700+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f353006ef30 con 0x7f353014fac0 2024-03-06T07:30:21.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.706+0000 7f352d600640 1 -- 172.21.15.114:0/777710849 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v15) v1 ==== 108+0+0 (secure 0 0 0) 0x7f351c0587d0 con 0x7f353014fac0 2024-03-06T07:30:21.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f351006b4d0 msgr2=0x7f351006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f351006b4d0 0x7f351006d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f35240079d0 tx=0x7f3524009040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f353014fac0 msgr2=0x7f353006e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f353014fac0 0x7f353006e220 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f351c00c3e0 tx=0x7f351c00c8a0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 shutdown_connections 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3530151120 0x7f35301515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f351006b4d0 0x7f351006d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f353014fac0 0x7f353006e220 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 --2- 172.21.15.114:0/777710849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180051c0 0x7f35301475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.708+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 >> 172.21.15.114:0/777710849 conn(0x7f35300b1940 msgr2=0x7f35300b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.709+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 shutdown_connections 2024-03-06T07:30:21.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.709+0000 7f3538000640 1 -- 172.21.15.114:0/777710849 wait complete. 2024-03-06T07:30:21.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:30:21.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.924+0000 7f9daa400640 1 Processor -- start 2024-03-06T07:30:21.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.924+0000 7f9daa400640 1 -- start start 2024-03-06T07:30:21.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.925+0000 7f9daa400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 0x7f9da40ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.925+0000 7f9daa400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9da40aebf0 0x7f9da40cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9daa400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da415c240 con 0x7f9da40cc800 2024-03-06T07:30:21.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9daa400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da415ce30 con 0x7f9da40aebf0 2024-03-06T07:30:21.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9daa400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da415c500 con 0x7f9da41561f0 2024-03-06T07:30:21.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9da9a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9da40cc800 0x7f9da40cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39878/0 (socket says 172.21.15.114:39878) 2024-03-06T07:30:21.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9da9a00640 1 -- 172.21.15.114:0/2208933166 learned_addr learned my addr 172.21.15.114:0/2208933166 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:21.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9da3e00640 1 --2- 172.21.15.114:0/2208933166 >> v2:172.21.15.177:3300/0 conn(0x7f9da40aebf0 0x7f9da40cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759813100 0 0) 0x7f9da415c240 con 0x7f9da40cc800 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9da9000640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 0x7f9da40ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.926+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d88003520 con 0x7f9da40cc800 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539388656 0 0) 0x7f9d88003520 con 0x7f9da40cc800 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 msgr2=0x7f9da40ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 0x7f9da40ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 >> v2:172.21.15.177:3300/0 conn(0x7f9da40aebf0 msgr2=0x7f9da40cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 --2- 172.21.15.114:0/2208933166 >> v2:172.21.15.177:3300/0 conn(0x7f9da40aebf0 0x7f9da40cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da415c7e0 con 0x7f9da40cc800 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f9d9c0019f0 con 0x7f9da40cc800 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 >> v1:172.21.15.114:6789/0 conn(0x7f9da40cc800 legacy=0x7f9da40cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d880039b0 0x7f9d88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 0x7f9d88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9d880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da415ce30 con 0x7f9d88004d40 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da415c500 con 0x7f9d880039b0 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8800fdd0 con 0x7f9da41561f0 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da9a00640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9d880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da9000640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d880039b0 0x7f9d88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.927+0000 7f9da3e00640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 0x7f9d88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.928+0000 7f9da9a00640 1 -- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 msgr2=0x7f9d88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.928+0000 7f9da9a00640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41561f0 0x7f9d88004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.928+0000 7f9da9a00640 1 -- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d880039b0 msgr2=0x7f9d88003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.928+0000 7f9da9a00640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d880039b0 0x7f9d88003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.928+0000 7f9da9a00640 1 -- 172.21.15.114:0/2208933166 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d88003520 con 0x7f9d88004d40 2024-03-06T07:30:21.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.928+0000 7f9da9000640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d880039b0 0x7f9d88003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.929+0000 7f9da9a00640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9d880051c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00a950 tx=0x7f9d88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352484218c2c0ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.929+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00a4c0 con 0x7f9d88004d40 2024-03-06T07:30:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.929+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c034650 con 0x7f9d88004d40 2024-03-06T07:30:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.929+0000 7f9da3400640 1 -- 172.21.15.114:0/2208933166 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c034a40 con 0x7f9d88004d40 2024-03-06T07:30:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.929+0000 7f9daa400640 1 -- 172.21.15.114:0/2208933166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 msgr2=0x7f9d880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.929+0000 7f9daa400640 1 --2- 172.21.15.114:0/2208933166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9d880051c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00a950 tx=0x7f9d88005830 comp rx=0 tx=0).stop 2024-03-06T07:30:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.930+0000 7f9da9a00640 1 -- 172.21.15.114:0/2208933166 reap_dead start 2024-03-06T07:30:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.930+0000 7f9daa400640 1 -- 172.21.15.114:0/2208933166 shutdown_connections 2024-03-06T07:30:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.930+0000 7f9daa400640 1 -- 172.21.15.114:0/2208933166 >> 172.21.15.114:0/2208933166 conn(0x7f9da406afc0 msgr2=0x7f9da406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.930+0000 7f9daa400640 1 -- 172.21.15.114:0/2208933166 shutdown_connections 2024-03-06T07:30:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.930+0000 7f9daa400640 1 -- 172.21.15.114:0/2208933166 wait complete. 2024-03-06T07:30:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.931+0000 7f9daa400640 1 Processor -- start 2024-03-06T07:30:21.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.932+0000 7f9daa400640 1 -- start start 2024-03-06T07:30:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.932+0000 7f9daa400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9da41ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.932+0000 7f9daa400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9da4159e60 0x7f9da415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.932+0000 7f9daa400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41577d0 0x7f9da4157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.932+0000 7f9daa400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da41565d0 con 0x7f9d88004d40 2024-03-06T07:30:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.932+0000 7f9daa400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da40ccc90 con 0x7f9da4159e60 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.932+0000 7f9daa400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4156750 con 0x7f9da41577d0 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9da41ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9da41ef4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41050/0 (socket says 172.21.15.114:41050) 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 -- 172.21.15.114:0/52077135 learned_addr learned my addr 172.21.15.114:0/52077135 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da3e00640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9da4159e60 0x7f9da415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 -- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41577d0 msgr2=0x7f9da4157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41577d0 0x7f9da4157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 -- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9da4159e60 msgr2=0x7f9da415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9da4159e60 0x7f9da415a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da9000640 1 -- 172.21.15.114:0/52077135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da4145e70 con 0x7f9d88004d40 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.933+0000 7f9da3e00640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9da4159e60 0x7f9da415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.934+0000 7f9da9000640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9da41ef4d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400e840 tx=0x7f9d9400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.934+0000 7f9da0c00640 1 -- 172.21.15.114:0/52077135 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94017070 con 0x7f9d88004d40 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.934+0000 7f9da0c00640 1 -- 172.21.15.114:0/52077135 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9400c650 con 0x7f9d88004d40 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.934+0000 7f9da0c00640 1 -- 172.21.15.114:0/52077135 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9400f040 con 0x7f9d88004d40 2024-03-06T07:30:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.934+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da4146150 con 0x7f9d88004d40 2024-03-06T07:30:21.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.934+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da4146710 con 0x7f9d88004d40 2024-03-06T07:30:21.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.935+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da404f330 con 0x7f9d88004d40 2024-03-06T07:30:21.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.936+0000 7f9da0c00640 1 -- 172.21.15.114:0/52077135 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d9401a070 con 0x7f9d88004d40 2024-03-06T07:30:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.937+0000 7f9da0c00640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d8006b4d0 0x7f9d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.937+0000 7f9da0c00640 1 -- 172.21.15.114:0/52077135 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9d94089a50 con 0x7f9d88004d40 2024-03-06T07:30:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.937+0000 7f9da3e00640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d8006b4d0 0x7f9d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.938+0000 7f9da3e00640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d8006b4d0 0x7f9d8006d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9da40b0aa0 tx=0x7f9d8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:21.942+0000 7f9da0c00640 1 -- 172.21.15.114:0/52077135 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d94058bd0 con 0x7f9d88004d40 2024-03-06T07:30:22.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.212+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f9da40b2b10 con 0x7f9d88004d40 2024-03-06T07:30:22.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.816+0000 7f9da0c00640 1 -- 172.21.15.114:0/52077135 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v147) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9d940584d0 con 0x7f9d88004d40 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d8006b4d0 msgr2=0x7f9d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d8006b4d0 0x7f9d8006d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9da40b0aa0 tx=0x7f9d8c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 msgr2=0x7f9da41ef4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9da41ef4d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9d9400e840 tx=0x7f9d9400ed00 comp rx=0 tx=0).stop 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 shutdown_connections 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9da41577d0 0x7f9da4157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9da4159e60 0x7f9da415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d8006b4d0 0x7f9d8006d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 --2- 172.21.15.114:0/52077135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d88004d40 0x7f9da41ef4d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 >> 172.21.15.114:0/52077135 conn(0x7f9da406afc0 msgr2=0x7f9da40af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 shutdown_connections 2024-03-06T07:30:22.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:22.820+0000 7f9daa400640 1 -- 172.21.15.114:0/52077135 wait complete. 2024-03-06T07:30:22.843 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:30:22.843 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:30:22.843 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:30:22.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:23.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.038+0000 7fd2b9000640 1 Processor -- start 2024-03-06T07:30:23.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.038+0000 7fd2b9000640 1 -- start start 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.039+0000 7fd2b9000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd2b415acb0 0x7fd2b40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.039+0000 7fd2b9000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 0x7fd2b40ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.039+0000 7fd2b9000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b415b090 con 0x7fd2b40b4680 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.039+0000 7fd2b9000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b40b2a30 con 0x7fd2b415acb0 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.039+0000 7fd2b9000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b4360 con 0x7fd2b406ec00 2024-03-06T07:30:23.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.040+0000 7fd2a9400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd2b40b4680 0x7fd2b406e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39884/0 (socket says 172.21.15.114:39884) 2024-03-06T07:30:23.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.040+0000 7fd2a9400640 1 -- 172.21.15.114:0/3145540258 learned_addr learned my addr 172.21.15.114:0/3145540258 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:23.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.040+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/3145540258 >> v2:172.21.15.177:3300/0 conn(0x7fd2b415acb0 0x7fd2b40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.040+0000 7fd2b2800640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 0x7fd2b40ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.040+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636119910 0 0) 0x7fd2b415b090 con 0x7fd2b40b4680 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.040+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd298005730 con 0x7fd2b40b4680 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4057555297 0 0) 0x7fd298005730 con 0x7fd2b40b4680 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 msgr2=0x7fd2b40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 0x7fd2b40ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 >> v2:172.21.15.177:3300/0 conn(0x7fd2b415acb0 msgr2=0x7fd2b40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 --2- 172.21.15.114:0/3145540258 >> v2:172.21.15.177:3300/0 conn(0x7fd2b415acb0 0x7fd2b40b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40b2cf0 con 0x7fd2b40b4680 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fd29c001670 con 0x7fd2b40b4680 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 >> v1:172.21.15.114:6789/0 conn(0x7fd2b40b4680 legacy=0x7fd2b406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd298005bc0 0x7fd298005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298006a10 0x7fd298006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 0x7fd2980073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b2a30 con 0x7fd298005bc0 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b4360 con 0x7fd298006a10 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29800dbd0 con 0x7fd2b406ec00 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.041+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd298005bc0 0x7fd298005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2a9400640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298006a10 0x7fd298006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2b1e00640 1 -- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 msgr2=0x7fd2980073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2b2800640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 0x7fd2980073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b406ec00 0x7fd2980073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2b1e00640 1 -- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298006a10 msgr2=0x7fd298006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298006a10 0x7fd298006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2b1e00640 1 -- 172.21.15.114:0/3145540258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298005730 con 0x7fd298005bc0 2024-03-06T07:30:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.042+0000 7fd2a9400640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298006a10 0x7fd298006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.043+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd298005bc0 0x7fd298005fe0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009ac0 tx=0x7fd298007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ded549bd8e70bea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.043+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4039070 con 0x7fd298005bc0 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.043+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a402c710 con 0x7fd298005bc0 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.043+0000 7fd2b1400640 1 -- 172.21.15.114:0/3145540258 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4009690 con 0x7fd298005bc0 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.044+0000 7fd2b9000640 1 -- 172.21.15.114:0/3145540258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd298005bc0 msgr2=0x7fd298005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.044+0000 7fd2b9000640 1 --2- 172.21.15.114:0/3145540258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd298005bc0 0x7fd298005fe0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009ac0 tx=0x7fd298007a40 comp rx=0 tx=0).stop 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.044+0000 7fd2b2800640 1 -- 172.21.15.114:0/3145540258 reap_dead start 2024-03-06T07:30:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.044+0000 7fd2b9000640 1 -- 172.21.15.114:0/3145540258 shutdown_connections 2024-03-06T07:30:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.044+0000 7fd2b9000640 1 -- 172.21.15.114:0/3145540258 >> 172.21.15.114:0/3145540258 conn(0x7fd2b40b1730 msgr2=0x7fd2b40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.044+0000 7fd2b9000640 1 -- 172.21.15.114:0/3145540258 shutdown_connections 2024-03-06T07:30:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.044+0000 7fd2b9000640 1 -- 172.21.15.114:0/3145540258 wait complete. 2024-03-06T07:30:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.046+0000 7fd2b9000640 1 Processor -- start 2024-03-06T07:30:23.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.046+0000 7fd2b9000640 1 -- start start 2024-03-06T07:30:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298005bc0 0x7fd2b41485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b9000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b4159b60 0x7fd2b4159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2b41551d0 0x7fd2b4155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b3d10 con 0x7fd2b41551d0 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b32b0 con 0x7fd298005bc0 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b9000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b4010 con 0x7fd2b4159b60 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2b41551d0 0x7fd2b4155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2b41551d0 0x7fd2b4155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41068/0 (socket says 172.21.15.114:41068) 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 -- 172.21.15.114:0/2634572883 learned_addr learned my addr 172.21.15.114:0/2634572883 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298005bc0 0x7fd2b41485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 -- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b4159b60 msgr2=0x7fd2b4159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2a9400640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b4159b60 0x7fd2b4159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b4159b60 0x7fd2b4159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 -- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298005bc0 msgr2=0x7fd2b41485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298005bc0 0x7fd2b41485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b2800640 1 -- 172.21.15.114:0/2634572883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b4148b30 con 0x7fd2b41551d0 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.047+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298005bc0 0x7fd2b41485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.048+0000 7fd2b2800640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2b41551d0 0x7fd2b4155650 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40cf970 tx=0x7fd2ac00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.048+0000 7fd2aaa00640 1 -- 172.21.15.114:0/2634572883 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac01c070 con 0x7fd2b41551d0 2024-03-06T07:30:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.048+0000 7fd2aaa00640 1 -- 172.21.15.114:0/2634572883 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac017400 con 0x7fd2b41551d0 2024-03-06T07:30:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.048+0000 7fd2aaa00640 1 -- 172.21.15.114:0/2634572883 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0177f0 con 0x7fd2b41551d0 2024-03-06T07:30:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.048+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b4157ee0 con 0x7fd2b41551d0 2024-03-06T07:30:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.048+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b41584a0 con 0x7fd2b41551d0 2024-03-06T07:30:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.049+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b404f330 con 0x7fd2b41551d0 2024-03-06T07:30:23.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.050+0000 7fd2aaa00640 1 -- 172.21.15.114:0/2634572883 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2ac01f070 con 0x7fd2b41551d0 2024-03-06T07:30:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.051+0000 7fd2aaa00640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd29006b4d0 0x7fd29006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.051+0000 7fd2aaa00640 1 -- 172.21.15.114:0/2634572883 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd2ac089fe0 con 0x7fd2b41551d0 2024-03-06T07:30:23.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.051+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd29006b4d0 0x7fd29006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.052+0000 7fd2b1e00640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd29006b4d0 0x7fd29006d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4036000 tx=0x7fd2a4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.055+0000 7fd2aaa00640 1 -- 172.21.15.114:0/2634572883 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2b4148b30 con 0x7fd2b41551d0 2024-03-06T07:30:23.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.238+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2b406ce60 con 0x7fd2b41551d0 2024-03-06T07:30:23.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.240+0000 7fd2aaa00640 1 -- 172.21.15.114:0/2634572883 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+13070 (secure 0 0 0) 0x7fd2ac012070 con 0x7fd2b41551d0 2024-03-06T07:30:23.243 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:23.243 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":47,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:20.793958+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/2123564678":"2024-03-07T07:30:17.777299+0000","172.21.15.114:6836/2123564678":"2024-03-07T07:30:17.777299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:23.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd29006b4d0 msgr2=0x7fd29006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd29006b4d0 0x7fd29006d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4036000 tx=0x7fd2a4030000 comp rx=0 tx=0).stop 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2b41551d0 msgr2=0x7fd2b4155650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2b41551d0 0x7fd2b4155650 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40cf970 tx=0x7fd2ac00f4a0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 shutdown_connections 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd29006b4d0 0x7fd29006d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2b41551d0 0x7fd2b4155650 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2b4159b60 0x7fd2b4159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 --2- 172.21.15.114:0/2634572883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd298005bc0 0x7fd2b41485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.242+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 >> 172.21.15.114:0/2634572883 conn(0x7fd2b40b1730 msgr2=0x7fd2b4071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.243+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 shutdown_connections 2024-03-06T07:30:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.243+0000 7fd2b9000640 1 -- 172.21.15.114:0/2634572883 wait complete. 2024-03-06T07:30:23.264 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:23.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:23.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.470+0000 7fe8d0800640 1 Processor -- start 2024-03-06T07:30:23.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.470+0000 7fe8d0800640 1 -- start start 2024-03-06T07:30:23.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.472+0000 7fe8d0800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c80b2c40 0x7fe8c815f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.472+0000 7fe8d0800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8c806d960 0x7fe8c80b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.472+0000 7fe8d0800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c80b3560 con 0x7fe8c806e2f0 2024-03-06T07:30:23.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.472+0000 7fe8d0800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c815f700 con 0x7fe8c806d960 2024-03-06T07:30:23.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.472+0000 7fe8d0800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c806d4d0 con 0x7fe8c80b2c40 2024-03-06T07:30:23.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.472+0000 7fe8cd400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8c806d960 0x7fe8c80b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8cde00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c80b2c40 0x7fe8c815f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8cd400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8c806d960 0x7fe8c80b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:41214/0 (socket says 172.21.15.114:41214) 2024-03-06T07:30:23.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8ce800640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fe8c806e2f0 0x7fe8c8161d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:35642/0 (socket says 172.21.15.114:35642) 2024-03-06T07:30:23.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8cd400640 1 -- 172.21.15.114:0/3428700116 learned_addr learned my addr 172.21.15.114:0/3428700116 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:23.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8cd400640 1 -- 172.21.15.114:0/3428700116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c80b2c40 msgr2=0x7fe8c815f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8cd400640 1 --2- 172.21.15.114:0/3428700116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c80b2c40 0x7fe8c815f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8cd400640 1 -- 172.21.15.114:0/3428700116 >> v1:172.21.15.177:6789/0 conn(0x7fe8c806e2f0 legacy=0x7fe8c8161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.473+0000 7fe8cd400640 1 -- 172.21.15.114:0/3428700116 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c806cba0 con 0x7fe8c806d960 2024-03-06T07:30:23.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.475+0000 7fe8cd400640 1 --2- 172.21.15.114:0/3428700116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c806d960 0x7fe8c80b3020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009870 tx=0x7fe8b402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6faa5a9873d5e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.475+0000 7fe8cca00640 1 -- 172.21.15.114:0/3428700116 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4034070 con 0x7fe8c806d960 2024-03-06T07:30:23.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.475+0000 7fe8cca00640 1 -- 172.21.15.114:0/3428700116 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b4030840 con 0x7fe8c806d960 2024-03-06T07:30:23.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.475+0000 7fe8cca00640 1 -- 172.21.15.114:0/3428700116 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4030c10 con 0x7fe8c806d960 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 -- 172.21.15.114:0/3428700116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c806d960 msgr2=0x7fe8c80b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3428700116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c806d960 0x7fe8c80b3020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009870 tx=0x7fe8b402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 -- 172.21.15.114:0/3428700116 shutdown_connections 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3428700116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c806d960 0x7fe8c80b3020 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3428700116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c80b2c40 0x7fe8c815f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 -- 172.21.15.114:0/3428700116 >> 172.21.15.114:0/3428700116 conn(0x7fe8c80b1940 msgr2=0x7fe8c80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 -- 172.21.15.114:0/3428700116 shutdown_connections 2024-03-06T07:30:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.476+0000 7fe8d0800640 1 -- 172.21.15.114:0/3428700116 wait complete. 2024-03-06T07:30:23.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.478+0000 7fe8d0800640 1 Processor -- start 2024-03-06T07:30:23.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.479+0000 7fe8d0800640 1 -- start start 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8d0800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c806d960 0x7fe8c81f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8d0800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c806e2f0 0x7fe8c815fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8d0800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c80b2c40 0x7fe8c8146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8d0800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c806e750 con 0x7fe8c806e2f0 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8d0800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c806d160 con 0x7fe8c80b2c40 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8d0800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c806e8d0 con 0x7fe8c806d960 2024-03-06T07:30:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c806e2f0 0x7fe8c815fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c806e2f0 0x7fe8c815fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41074/0 (socket says 172.21.15.114:41074) 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 -- 172.21.15.114:0/3802426142 learned_addr learned my addr 172.21.15.114:0/3802426142 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cde00640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c806d960 0x7fe8c81f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8ce800640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c80b2c40 0x7fe8c8146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 -- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c806d960 msgr2=0x7fe8c81f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c806d960 0x7fe8c81f7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 -- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c80b2c40 msgr2=0x7fe8c8146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c80b2c40 0x7fe8c8146bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.480+0000 7fe8cd400640 1 -- 172.21.15.114:0/3802426142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c809bae0 con 0x7fe8c806e2f0 2024-03-06T07:30:23.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.481+0000 7fe8cd400640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c806e2f0 0x7fe8c815fb00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009990 tx=0x7fe8b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.481+0000 7fe8c2000640 1 -- 172.21.15.114:0/3802426142 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4034040 con 0x7fe8c806e2f0 2024-03-06T07:30:23.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.481+0000 7fe8c2000640 1 -- 172.21.15.114:0/3802426142 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b4039d90 con 0x7fe8c806e2f0 2024-03-06T07:30:23.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.481+0000 7fe8c2000640 1 -- 172.21.15.114:0/3802426142 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4043610 con 0x7fe8c806e2f0 2024-03-06T07:30:23.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.481+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c8147b70 con 0x7fe8c806e2f0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.481+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c81f8220 con 0x7fe8c806e2f0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.483+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c804f330 con 0x7fe8c806e2f0 2024-03-06T07:30:23.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.483+0000 7fe8c2000640 1 -- 172.21.15.114:0/3802426142 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8b404a060 con 0x7fe8c806e2f0 2024-03-06T07:30:23.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.484+0000 7fe8c2000640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8a806b4d0 0x7fe8a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.484+0000 7fe8c2000640 1 -- 172.21.15.114:0/3802426142 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe8b40abeb0 con 0x7fe8c806e2f0 2024-03-06T07:30:23.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.485+0000 7fe8cde00640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8a806b4d0 0x7fe8a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.485+0000 7fe8cde00640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8a806b4d0 0x7fe8a806d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009ad0 tx=0x7fe8bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.490+0000 7fe8c2000640 1 -- 172.21.15.114:0/3802426142 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8b403e070 con 0x7fe8c806e2f0 2024-03-06T07:30:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.684+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8c80cd460 con 0x7fe8c806e2f0 2024-03-06T07:30:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.685+0000 7fe8c2000640 1 -- 172.21.15.114:0/3802426142 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe8b407b020 con 0x7fe8c806e2f0 2024-03-06T07:30:23.688 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:30:23.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8a806b4d0 msgr2=0x7fe8a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8a806b4d0 0x7fe8a806d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009ad0 tx=0x7fe8bc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:23.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c806e2f0 msgr2=0x7fe8c815fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c806e2f0 0x7fe8c815fb00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009990 tx=0x7fe8b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 shutdown_connections 2024-03-06T07:30:23.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c80b2c40 0x7fe8c8146bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe8a806b4d0 0x7fe8a806d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8c806e2f0 0x7fe8c815fb00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 --2- 172.21.15.114:0/3802426142 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8c806d960 0x7fe8c81f7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.687+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 >> 172.21.15.114:0/3802426142 conn(0x7fe8c80b1940 msgr2=0x7fe8c8071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.688+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 shutdown_connections 2024-03-06T07:30:23.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.688+0000 7fe8d0800640 1 -- 172.21.15.114:0/3802426142 wait complete. 2024-03-06T07:30:23.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:30:23.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.921+0000 7f0431e00640 1 Processor -- start 2024-03-06T07:30:23.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.922+0000 7f0431e00640 1 -- start start 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.923+0000 7f0431e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c0b4680 0x7f042c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.923+0000 7f0431e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f042c072340 0x7f042c06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.923+0000 7f0431e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042c15b090 con 0x7f042c15acb0 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.923+0000 7f0431e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042c0b2a30 con 0x7f042c072340 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.923+0000 7f0431e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0b4360 con 0x7f042c0b4680 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.923+0000 7f042aa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f042c15acb0 0x7f042c0b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39888/0 (socket says 172.21.15.114:39888) 2024-03-06T07:30:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.923+0000 7f042aa00640 1 -- 172.21.15.114:0/1173500016 learned_addr learned my addr 172.21.15.114:0/1173500016 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f0422000640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c0b4680 0x7f042c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55659202 0 0) 0x7f042c15b090 con 0x7f042c15acb0 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0410003520 con 0x7f042c15acb0 2024-03-06T07:30:23.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042b400640 1 --2- 172.21.15.114:0/1173500016 >> v2:172.21.15.177:3300/0 conn(0x7f042c072340 0x7f042c06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3828858034 0 0) 0x7f0410003520 con 0x7f042c15acb0 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c0b4680 msgr2=0x7f042c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c0b4680 0x7f042c06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 >> v2:172.21.15.177:3300/0 conn(0x7f042c072340 msgr2=0x7f042c06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 --2- 172.21.15.114:0/1173500016 >> v2:172.21.15.177:3300/0 conn(0x7f042c072340 0x7f042c06ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c0b2cf0 con 0x7f042c15acb0 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f041c0019f0 con 0x7f042c15acb0 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 >> v1:172.21.15.114:6789/0 conn(0x7f042c15acb0 legacy=0x7f042c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.924+0000 7f042a000640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f0410003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042a000640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c0b4680 0x7f0410004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042a000640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0410004d40 0x7f04100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0b2a30 con 0x7f04100039b0 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0b4360 con 0x7f0410004d40 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041000dc00 con 0x7f042c0b4680 2024-03-06T07:30:23.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042aa00640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f0410003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042aa00640 1 -- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c0b4680 msgr2=0x7f0410004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042aa00640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c0b4680 0x7f0410004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042aa00640 1 -- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0410004d40 msgr2=0x7f04100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042aa00640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0410004d40 0x7f04100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.925+0000 7f042aa00640 1 -- 172.21.15.114:0/1173500016 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0410003520 con 0x7f04100039b0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.926+0000 7f042aa00640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f0410003dd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f041c00a8a0 tx=0x7f0410005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d590cbbdd4208a7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.926+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00a4c0 con 0x7f04100039b0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.926+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c034650 con 0x7f04100039b0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.926+0000 7f042a000640 1 -- 172.21.15.114:0/1173500016 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c034a40 con 0x7f04100039b0 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.927+0000 7f0431e00640 1 -- 172.21.15.114:0/1173500016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 msgr2=0x7f0410003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.927+0000 7f0431e00640 1 --2- 172.21.15.114:0/1173500016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f0410003dd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f041c00a8a0 tx=0x7f0410005830 comp rx=0 tx=0).stop 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.927+0000 7f042b400640 1 -- 172.21.15.114:0/1173500016 reap_dead start 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.927+0000 7f0431e00640 1 -- 172.21.15.114:0/1173500016 shutdown_connections 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.927+0000 7f0431e00640 1 -- 172.21.15.114:0/1173500016 >> 172.21.15.114:0/1173500016 conn(0x7f042c0b1730 msgr2=0x7f042c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.927+0000 7f0431e00640 1 -- 172.21.15.114:0/1173500016 shutdown_connections 2024-03-06T07:30:23.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.927+0000 7f0431e00640 1 -- 172.21.15.114:0/1173500016 wait complete. 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.929+0000 7f0431e00640 1 Processor -- start 2024-03-06T07:30:23.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.929+0000 7f0431e00640 1 -- start start 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f0431e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f042c152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f0431e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f042c14e2a0 0x7f042c142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f0431e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c14e7f0 0x7f042c143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f0431e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0b4010 con 0x7f04100039b0 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f0431e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0b32b0 con 0x7f042c14e2a0 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f0431e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c0ccfb0 con 0x7f042c14e7f0 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f042aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f042c152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f042aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f042c152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41098/0 (socket says 172.21.15.114:41098) 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.930+0000 7f042aa00640 1 -- 172.21.15.114:0/374405192 learned_addr learned my addr 172.21.15.114:0/374405192 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f0422000640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f042c14e2a0 0x7f042c142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f042aa00640 1 -- 172.21.15.114:0/374405192 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c14e7f0 msgr2=0x7f042c143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f042aa00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c14e7f0 0x7f042c143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f042aa00640 1 -- 172.21.15.114:0/374405192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f042c14e2a0 msgr2=0x7f042c142ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f042aa00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f042c14e2a0 0x7f042c142ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f042aa00640 1 -- 172.21.15.114:0/374405192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c1f30d0 con 0x7f04100039b0 2024-03-06T07:30:23.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f042aa00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f042c152c20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f041c00a9c0 tx=0x7f041c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f0423e00640 1 -- 172.21.15.114:0/374405192 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c0019c0 con 0x7f04100039b0 2024-03-06T07:30:23.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f0423e00640 1 -- 172.21.15.114:0/374405192 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c03a040 con 0x7f04100039b0 2024-03-06T07:30:23.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f0423e00640 1 -- 172.21.15.114:0/374405192 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c034a40 con 0x7f04100039b0 2024-03-06T07:30:23.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042c1f33b0 con 0x7f04100039b0 2024-03-06T07:30:23.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.931+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042c1f3970 con 0x7f04100039b0 2024-03-06T07:30:23.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.933+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042c04f330 con 0x7f04100039b0 2024-03-06T07:30:23.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.933+0000 7f0423e00640 1 -- 172.21.15.114:0/374405192 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f041c046020 con 0x7f04100039b0 2024-03-06T07:30:23.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.934+0000 7f0423e00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f040806b4d0 0x7f040806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.934+0000 7f0423e00640 1 -- 172.21.15.114:0/374405192 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f041c0abe20 con 0x7f04100039b0 2024-03-06T07:30:23.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.935+0000 7f0422000640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f040806b4d0 0x7f040806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.935+0000 7f0422000640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f040806b4d0 0x7f040806d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f042c0cbe50 tx=0x7f0414008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:23.939+0000 7f0423e00640 1 -- 172.21.15.114:0/374405192 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f041c04d020 con 0x7f04100039b0 2024-03-06T07:30:24.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.206+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f042c15a2c0 con 0x7f04100039b0 2024-03-06T07:30:24.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.831+0000 7f0423e00640 1 -- 172.21.15.114:0/374405192 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v149) v1 ==== 147+0+0 (secure 0 0 0) 0x7f041c07af00 con 0x7f04100039b0 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f040806b4d0 msgr2=0x7f040806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f040806b4d0 0x7f040806d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f042c0cbe50 tx=0x7f0414008040 comp rx=0 tx=0).stop 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 msgr2=0x7f042c152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f042c152c20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f041c00a9c0 tx=0x7f041c001dc0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 shutdown_connections 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f042c14e7f0 0x7f042c143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f042c14e2a0 0x7f042c142ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f040806b4d0 0x7f040806d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 --2- 172.21.15.114:0/374405192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04100039b0 0x7f042c152c20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 >> 172.21.15.114:0/374405192 conn(0x7f042c0b1730 msgr2=0x7f042c1557e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 shutdown_connections 2024-03-06T07:30:24.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:24.836+0000 7f0431e00640 1 -- 172.21.15.114:0/374405192 wait complete. 2024-03-06T07:30:24.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:30:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.070+0000 7f38c3600640 1 Processor -- start 2024-03-06T07:30:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.070+0000 7f38c3600640 1 -- start start 2024-03-06T07:30:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.072+0000 7f38c3600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f38bc0b2c40 0x7f38bc15f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.072+0000 7f38c3600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06d960 0x7f38bc0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.072+0000 7f38c3600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38bc06e2f0 0x7f38bc161d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.072+0000 7f38c3600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc15f700 con 0x7f38bc06e2f0 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.072+0000 7f38c3600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc06d4d0 con 0x7f38bc0b2c40 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.072+0000 7f38c3600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc0b3560 con 0x7f38bc06d960 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.072+0000 7f38c1600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38bc06e2f0 0x7f38bc161d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c1600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38bc06e2f0 0x7f38bc161d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41108/0 (socket says 172.21.15.114:41108) 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c1600640 1 -- 172.21.15.114:0/3040756157 learned_addr learned my addr 172.21.15.114:0/3040756157 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38bbe00640 1 --2- 172.21.15.114:0/3040756157 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06d960 0x7f38bc0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c0c00640 1 --2- 172.21.15.114:0/3040756157 >> v2:172.21.15.177:3300/0 conn(0x7f38bc0b2c40 0x7f38bc15f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c1600640 1 -- 172.21.15.114:0/3040756157 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06d960 msgr2=0x7f38bc0b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c1600640 1 --2- 172.21.15.114:0/3040756157 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06d960 0x7f38bc0b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c1600640 1 -- 172.21.15.114:0/3040756157 >> v2:172.21.15.177:3300/0 conn(0x7f38bc0b2c40 msgr2=0x7f38bc15f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c1600640 1 --2- 172.21.15.114:0/3040756157 >> v2:172.21.15.177:3300/0 conn(0x7f38bc0b2c40 0x7f38bc15f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c1600640 1 -- 172.21.15.114:0/3040756157 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc06cba0 con 0x7f38bc06e2f0 2024-03-06T07:30:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.073+0000 7f38c0c00640 1 --2- 172.21.15.114:0/3040756157 >> v2:172.21.15.177:3300/0 conn(0x7f38bc0b2c40 0x7f38bc15f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.074+0000 7f38c1600640 1 --2- 172.21.15.114:0/3040756157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc06e2f0 0x7f38bc161d50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f38bc06c6e0 tx=0x7f38b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f3d0313a07544f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.074+0000 7f38bb400640 1 -- 172.21.15.114:0/3040756157 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b402ca70 con 0x7f38bc06e2f0 2024-03-06T07:30:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38bb400640 1 -- 172.21.15.114:0/3040756157 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b402cc10 con 0x7f38bc06e2f0 2024-03-06T07:30:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38bb400640 1 -- 172.21.15.114:0/3040756157 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40343f0 con 0x7f38bc06e2f0 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 -- 172.21.15.114:0/3040756157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc06e2f0 msgr2=0x7f38bc161d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 --2- 172.21.15.114:0/3040756157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc06e2f0 0x7f38bc161d50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f38bc06c6e0 tx=0x7f38b402bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 -- 172.21.15.114:0/3040756157 shutdown_connections 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 --2- 172.21.15.114:0/3040756157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc06e2f0 0x7f38bc161d50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 --2- 172.21.15.114:0/3040756157 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06d960 0x7f38bc0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 --2- 172.21.15.114:0/3040756157 >> v2:172.21.15.177:3300/0 conn(0x7f38bc0b2c40 0x7f38bc15f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 -- 172.21.15.114:0/3040756157 >> 172.21.15.114:0/3040756157 conn(0x7f38bc0b1940 msgr2=0x7f38bc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 -- 172.21.15.114:0/3040756157 shutdown_connections 2024-03-06T07:30:25.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.075+0000 7f38c3600640 1 -- 172.21.15.114:0/3040756157 wait complete. 2024-03-06T07:30:25.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.077+0000 7f38c3600640 1 Processor -- start 2024-03-06T07:30:25.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.077+0000 7f38c3600640 1 -- start start 2024-03-06T07:30:25.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c3600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38bc06d960 0x7f38bc0aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c3600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06e2f0 0x7f38bc09d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c3600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc0b2c40 0x7f38bc14df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c3600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc06e750 con 0x7f38bc0b2c40 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c3600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc06d160 con 0x7f38bc06d960 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c3600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc06e8d0 con 0x7f38bc06e2f0 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc0b2c40 0x7f38bc14df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc0b2c40 0x7f38bc14df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41112/0 (socket says 172.21.15.114:41112) 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.078+0000 7f38c1600640 1 -- 172.21.15.114:0/987198392 learned_addr learned my addr 172.21.15.114:0/987198392 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38c1600640 1 -- 172.21.15.114:0/987198392 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06e2f0 msgr2=0x7f38bc09d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38c1600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06e2f0 0x7f38bc09d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38c1600640 1 -- 172.21.15.114:0/987198392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38bc06d960 msgr2=0x7f38bc0aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38c1600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38bc06d960 0x7f38bc0aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38c1600640 1 -- 172.21.15.114:0/987198392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc14e490 con 0x7f38bc0b2c40 2024-03-06T07:30:25.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38c1600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc0b2c40 0x7f38bc14df50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38b4036040 tx=0x7f38b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38b8c00640 1 -- 172.21.15.114:0/987198392 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4043020 con 0x7f38bc0b2c40 2024-03-06T07:30:25.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38b8c00640 1 -- 172.21.15.114:0/987198392 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b403e450 con 0x7f38bc0b2c40 2024-03-06T07:30:25.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38b8c00640 1 -- 172.21.15.114:0/987198392 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4009510 con 0x7f38bc0b2c40 2024-03-06T07:30:25.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.079+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc160140 con 0x7f38bc0b2c40 2024-03-06T07:30:25.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.080+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc09ab90 con 0x7f38bc0b2c40 2024-03-06T07:30:25.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.081+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc160660 con 0x7f38bc0b2c40 2024-03-06T07:30:25.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.082+0000 7f38b8c00640 1 -- 172.21.15.114:0/987198392 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38b4046020 con 0x7f38bc0b2c40 2024-03-06T07:30:25.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.083+0000 7f38b8c00640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f389c06b4d0 0x7f389c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.083+0000 7f38b8c00640 1 -- 172.21.15.114:0/987198392 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f38b40ac9f0 con 0x7f38bc0b2c40 2024-03-06T07:30:25.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.083+0000 7f38c0c00640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f389c06b4d0 0x7f389c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.084+0000 7f38c0c00640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f389c06b4d0 0x7f389c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f38b000a8b0 tx=0x7f38b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.088+0000 7f38b8c00640 1 -- 172.21.15.114:0/987198392 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38bc160660 con 0x7f38bc0b2c40 2024-03-06T07:30:25.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.286+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f38bc04f330 con 0x7f38bc0b2c40 2024-03-06T07:30:25.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.291+0000 7f38b8c00640 1 -- 172.21.15.114:0/987198392 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f38b404d020 con 0x7f38bc0b2c40 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f389c06b4d0 msgr2=0x7f389c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f389c06b4d0 0x7f389c06d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f38b000a8b0 tx=0x7f38b0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc0b2c40 msgr2=0x7f38bc14df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc0b2c40 0x7f38bc14df50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38b4036040 tx=0x7f38b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 shutdown_connections 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f389c06b4d0 0x7f389c06d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38bc0b2c40 0x7f38bc14df50 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f38bc06e2f0 0x7f38bc09d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 --2- 172.21.15.114:0/987198392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f38bc06d960 0x7f38bc0aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.294+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 >> 172.21.15.114:0/987198392 conn(0x7f38bc0b1940 msgr2=0x7f38bc15db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.295+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 shutdown_connections 2024-03-06T07:30:25.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.295+0000 7f38c3600640 1 -- 172.21.15.114:0/987198392 wait complete. 2024-03-06T07:30:25.317 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:25.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:25.426 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:25.547 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:25.547 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:25.547 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:25.603 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:25.603 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:30:25.603 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:25.657 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:25.658 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:25.658 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:25.713 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:25.713 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:30:25.713 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:25.768 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:25.768 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:25.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.984+0000 7f5998e00640 1 Processor -- start 2024-03-06T07:30:25.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.984+0000 7f5998e00640 1 -- start start 2024-03-06T07:30:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.986+0000 7f5998e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f59940cc9d0 0x7f599406d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.986+0000 7f5998e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599415c240 con 0x7f59940aedc0 2024-03-06T07:30:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.986+0000 7f5998e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599415c500 con 0x7f59941561f0 2024-03-06T07:30:25.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.986+0000 7f5998e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599415ce30 con 0x7f59940cc9d0 2024-03-06T07:30:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.986+0000 7f5992a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f59940aedc0 0x7f59940cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39894/0 (socket says 172.21.15.114:39894) 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.986+0000 7f5992a00640 1 -- 172.21.15.114:0/1364306136 learned_addr learned my addr 172.21.15.114:0/1364306136 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.986+0000 7f5993e00640 1 --2- 172.21.15.114:0/1364306136 >> v2:172.21.15.177:3300/0 conn(0x7f59940cc9d0 0x7f599406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244305943 0 0) 0x7f599415c240 con 0x7f59940aedc0 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5978003520 con 0x7f59940aedc0 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1697695261 0 0) 0x7f5978003520 con 0x7f59940aedc0 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 >> v2:172.21.15.177:3300/0 conn(0x7f59940cc9d0 msgr2=0x7f599406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 --2- 172.21.15.114:0/1364306136 >> v2:172.21.15.177:3300/0 conn(0x7f59940cc9d0 0x7f599406d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 >> v1:172.21.15.177:6789/0 conn(0x7f59941561f0 legacy=0x7f59940ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599415c7e0 con 0x7f59940aedc0 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5980001670 con 0x7f59940aedc0 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 >> v1:172.21.15.114:6789/0 conn(0x7f59940aedc0 legacy=0x7f59940cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780039b0 0x7f5978003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5978004800 0x7f5978004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59780051c0 0x7f597800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599415ce30 con 0x7f59780051c0 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597800c060 con 0x7f5978004800 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597800c260 con 0x7f59780039b0 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.987+0000 7f5993e00640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59780051c0 0x7f597800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.988+0000 7f5992a00640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5978004800 0x7f5978004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.988+0000 7f5993400640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780039b0 0x7f5978003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.988+0000 7f5993e00640 1 -- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780039b0 msgr2=0x7f5978003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.988+0000 7f5993e00640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780039b0 0x7f5978003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.988+0000 7f5993e00640 1 -- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5978004800 msgr2=0x7f5978004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.988+0000 7f5993e00640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5978004800 0x7f5978004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.988+0000 7f5993e00640 1 -- 172.21.15.114:0/1364306136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978003520 con 0x7f59780051c0 2024-03-06T07:30:25.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.989+0000 7f5993e00640 1 -- 172.21.15.114:0/1364306136 reap_dead start 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.989+0000 7f5993e00640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59780051c0 0x7f597800b9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f598c009fc0 tx=0x7f5978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5577ad0e7227cff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.989+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c009c10 con 0x7f59780051c0 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.989+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598c02cca0 con 0x7f59780051c0 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.989+0000 7f5992000640 1 -- 172.21.15.114:0/1364306136 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c003bd0 con 0x7f59780051c0 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.990+0000 7f5998e00640 1 -- 172.21.15.114:0/1364306136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59780051c0 msgr2=0x7f597800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.990+0000 7f5998e00640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59780051c0 0x7f597800b9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f598c009fc0 tx=0x7f5978005830 comp rx=0 tx=0).stop 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.990+0000 7f5998e00640 1 -- 172.21.15.114:0/1364306136 shutdown_connections 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.990+0000 7f5998e00640 1 --2- 172.21.15.114:0/1364306136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59780051c0 0x7f597800b9c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.990+0000 7f5998e00640 1 -- 172.21.15.114:0/1364306136 >> 172.21.15.114:0/1364306136 conn(0x7f599406afc0 msgr2=0x7f599406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.990+0000 7f5998e00640 1 -- 172.21.15.114:0/1364306136 shutdown_connections 2024-03-06T07:30:25.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.990+0000 7f5998e00640 1 -- 172.21.15.114:0/1364306136 wait complete. 2024-03-06T07:30:25.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.992+0000 7f5998e00640 1 Processor -- start 2024-03-06T07:30:25.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.993+0000 7f5998e00640 1 -- start start 2024-03-06T07:30:25.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5998e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780051c0 0x7f599413d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5998e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5994158f70 0x7f599415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5998e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59941593f0 0x7f599415ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5998e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994156750 con 0x7f5994158f70 2024-03-06T07:30:25.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5998e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941565d0 con 0x7f59941593f0 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5998e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59940cc2c0 con 0x7f59780051c0 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5992a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5994158f70 0x7f599415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5992a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5994158f70 0x7f599415b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41132/0 (socket says 172.21.15.114:41132) 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.994+0000 7f5992a00640 1 -- 172.21.15.114:0/4285640248 learned_addr learned my addr 172.21.15.114:0/4285640248 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5992a00640 1 -- 172.21.15.114:0/4285640248 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780051c0 msgr2=0x7f599413d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5992a00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780051c0 0x7f599413d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5992a00640 1 -- 172.21.15.114:0/4285640248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59941593f0 msgr2=0x7f599415ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5992a00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59941593f0 0x7f599415ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5992a00640 1 -- 172.21.15.114:0/4285640248 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5994157ed0 con 0x7f5994158f70 2024-03-06T07:30:25.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5992a00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5994158f70 0x7f599415b4c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f598000e7b0 tx=0x7f598000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f598b400640 1 -- 172.21.15.114:0/4285640248 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000c870 con 0x7f5994158f70 2024-03-06T07:30:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f598b400640 1 -- 172.21.15.114:0/4285640248 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5980018070 con 0x7f5994158f70 2024-03-06T07:30:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f598b400640 1 -- 172.21.15.114:0/4285640248 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000cc50 con 0x7f5994158f70 2024-03-06T07:30:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59941581b0 con 0x7f5994158f70 2024-03-06T07:30:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.995+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5994158770 con 0x7f5994158f70 2024-03-06T07:30:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.997+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599404f330 con 0x7f5994158f70 2024-03-06T07:30:26.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.998+0000 7f598b400640 1 -- 172.21.15.114:0/4285640248 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5980013420 con 0x7f5994158f70 2024-03-06T07:30:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.998+0000 7f598b400640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f597406b500 0x7f597406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.999+0000 7f598b400640 1 -- 172.21.15.114:0/4285640248 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5980089ad0 con 0x7f5994158f70 2024-03-06T07:30:26.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:25.999+0000 7f5993400640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f597406b500 0x7f597406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.000+0000 7f5993400640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f597406b500 0x7f597406d9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f599415a9d0 tx=0x7f5984009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.005+0000 7f598b400640 1 -- 172.21.15.114:0/4285640248 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5980058bb0 con 0x7f5994158f70 2024-03-06T07:30:26.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.239+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59940b2b10 con 0x7f5994158f70 2024-03-06T07:30:26.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.240+0000 7f598b400640 1 -- 172.21.15.114:0/4285640248 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f59800584b0 con 0x7f5994158f70 2024-03-06T07:30:26.243 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:26.243 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4721,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/308840931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":308840931},{"type":"v1","addr":"172.21.15.177:6833","nonce":308840931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/4238819696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":4238819696},{"type":"v1","addr":"172.21.15.177:6835","nonce":4238819696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4960,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/379671784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":379671784},{"type":"v1","addr":"172.21.15.114:6835","nonce":379671784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/2130353724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":2130353724},{"type":"v1","addr":"172.21.15.114:6837","nonce":2130353724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[]} 2024-03-06T07:30:26.243 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 28 2024-03-06T07:30:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f597406b500 msgr2=0x7f597406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f597406b500 0x7f597406d9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f599415a9d0 tx=0x7f5984009040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5994158f70 msgr2=0x7f599415b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5994158f70 0x7f599415b4c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f598000e7b0 tx=0x7f598000ec70 comp rx=0 tx=0).stop 2024-03-06T07:30:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 shutdown_connections 2024-03-06T07:30:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59941593f0 0x7f599415ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f597406b500 0x7f597406d9c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5994158f70 0x7f599415b4c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 --2- 172.21.15.114:0/4285640248 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59780051c0 0x7f599413d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 >> 172.21.15.114:0/4285640248 conn(0x7f599406afc0 msgr2=0x7f59940b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 shutdown_connections 2024-03-06T07:30:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.243+0000 7f5998e00640 1 -- 172.21.15.114:0/4285640248 wait complete. 2024-03-06T07:30:26.267 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:26.267 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:26.267 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:26.267 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:26.268 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:26.268 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:26.268 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:26.268 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:26.268 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:26.268 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:30:26.269 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:26.269 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:26.269 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:26.269 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:30:26.269 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:26.269 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:26.285 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:26.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:26.287 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:26.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:26.290 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:26.291 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:26.295 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:26.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:26.334 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:26.335 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:26.340 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:26.341 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:26.341 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:26.342 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:26.348 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:26.348 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:26.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:26.491 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:30:26.491 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:30:26.503 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:30:26.504 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:30:26.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.529+0000 7f09d6c00640 1 Processor -- start 2024-03-06T07:30:26.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.529+0000 7f09d6c00640 1 -- start start 2024-03-06T07:30:26.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09d6c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01444e0 0x7f09d00cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09d6c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f09d01474c0 0x7f09d00c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09d6c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f09d00c4e70 0x7f09d0161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09d6c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00c52d0 con 0x7f09d00c4e70 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09d6c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00ae380 con 0x7f09d01474c0 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09d6c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00ae660 con 0x7f09d01444e0 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09d4c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f09d00c4e70 0x7f09d0161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09cfe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01444e0 0x7f09d00cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09cfe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01444e0 0x7f09d00cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:47684/0 (socket says 172.21.15.114:47684) 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09cfe00640 1 -- 172.21.15.114:0/2245332518 learned_addr learned my addr 172.21.15.114:0/2245332518 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.531+0000 7f09cf400640 1 --2- 172.21.15.114:0/2245332518 >> v2:172.21.15.177:3300/0 conn(0x7f09d01474c0 0x7f09d00c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.532+0000 7f09cfe00640 1 -- 172.21.15.114:0/2245332518 >> v2:172.21.15.177:3300/0 conn(0x7f09d01474c0 msgr2=0x7f09d00c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.532+0000 7f09cfe00640 1 --2- 172.21.15.114:0/2245332518 >> v2:172.21.15.177:3300/0 conn(0x7f09d01474c0 0x7f09d00c4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.532+0000 7f09cfe00640 1 -- 172.21.15.114:0/2245332518 >> v2:172.21.15.114:3300/0 conn(0x7f09d00c4e70 msgr2=0x7f09d0161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.532+0000 7f09cfe00640 1 --2- 172.21.15.114:0/2245332518 >> v2:172.21.15.114:3300/0 conn(0x7f09d00c4e70 0x7f09d0161d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.532+0000 7f09cfe00640 1 -- 172.21.15.114:0/2245332518 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00ae940 con 0x7f09d01444e0 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.533+0000 7f09cfe00640 1 --2- 172.21.15.114:0/2245332518 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01444e0 0x7f09d00cd010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09d00737d0 tx=0x7f09c002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d26e34cd3908800 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.533+0000 7f09cea00640 1 -- 172.21.15.114:0/2245332518 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0035020 con 0x7f09d01444e0 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.533+0000 7f09cea00640 1 -- 172.21.15.114:0/2245332518 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c0031870 con 0x7f09d01444e0 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.533+0000 7f09cea00640 1 -- 172.21.15.114:0/2245332518 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0031c40 con 0x7f09d01444e0 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 -- 172.21.15.114:0/2245332518 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01444e0 msgr2=0x7f09d00cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2245332518 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01444e0 0x7f09d00cd010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09d00737d0 tx=0x7f09c002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 -- 172.21.15.114:0/2245332518 shutdown_connections 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2245332518 >> v2:172.21.15.114:3300/0 conn(0x7f09d00c4e70 0x7f09d0161d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2245332518 >> v2:172.21.15.177:3300/0 conn(0x7f09d01474c0 0x7f09d00c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2245332518 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01444e0 0x7f09d00cd010 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 -- 172.21.15.114:0/2245332518 >> 172.21.15.114:0/2245332518 conn(0x7f09d00cc4f0 msgr2=0x7f09d00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 -- 172.21.15.114:0/2245332518 shutdown_connections 2024-03-06T07:30:26.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.534+0000 7f09d6c00640 1 -- 172.21.15.114:0/2245332518 wait complete. 2024-03-06T07:30:26.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.535+0000 7f09d6c00640 1 Processor -- start 2024-03-06T07:30:26.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.536+0000 7f09d6c00640 1 -- start start 2024-03-06T07:30:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.536+0000 7f09d6c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d00c4e70 0x7f09d0140190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.536+0000 7f09d6c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d01444e0 0x7f09d00cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.536+0000 7f09d6c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01474c0 0x7f09d00b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.536+0000 7f09d6c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0147cb0 con 0x7f09d00c4e70 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.536+0000 7f09d6c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00aef00 con 0x7f09d01444e0 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.536+0000 7f09d6c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01448c0 con 0x7f09d01474c0 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d00c4e70 0x7f09d0140190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d00c4e70 0x7f09d0140190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41194/0 (socket says 172.21.15.114:41194) 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 -- 172.21.15.114:0/2404968516 learned_addr learned my addr 172.21.15.114:0/2404968516 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09d4c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01474c0 0x7f09d00b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cf400640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d01444e0 0x7f09d00cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 -- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01474c0 msgr2=0x7f09d00b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01474c0 0x7f09d00b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 -- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d01444e0 msgr2=0x7f09d00cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d01444e0 0x7f09d00cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 -- 172.21.15.114:0/2404968516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d01408c0 con 0x7f09d00c4e70 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09d4c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01474c0 0x7f09d00b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.537+0000 7f09cfe00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d00c4e70 0x7f09d0140190 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f09c002b950 tx=0x7f09c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.538+0000 7f09c7e00640 1 -- 172.21.15.114:0/2404968516 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0040070 con 0x7f09d00c4e70 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.538+0000 7f09c7e00640 1 -- 172.21.15.114:0/2404968516 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c0031e10 con 0x7f09d00c4e70 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.538+0000 7f09c7e00640 1 -- 172.21.15.114:0/2404968516 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0035040 con 0x7f09d00c4e70 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.538+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d01f7620 con 0x7f09d00c4e70 2024-03-06T07:30:26.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.538+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d01f7be0 con 0x7f09d00c4e70 2024-03-06T07:30:26.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.539+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d004f330 con 0x7f09d00c4e70 2024-03-06T07:30:26.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.540+0000 7f09c7e00640 1 -- 172.21.15.114:0/2404968516 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09c004b060 con 0x7f09d00c4e70 2024-03-06T07:30:26.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.541+0000 7f09c7e00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09b006b500 0x7f09b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.541+0000 7f09c7e00640 1 -- 172.21.15.114:0/2404968516 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09c00abb00 con 0x7f09d00c4e70 2024-03-06T07:30:26.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.541+0000 7f09cf400640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09b006b500 0x7f09b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.542+0000 7f09cf400640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09b006b500 0x7f09b006d9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f09d00c8700 tx=0x7f09bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.546+0000 7f09c7e00640 1 -- 172.21.15.114:0/2404968516 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09c002f320 con 0x7f09d00c4e70 2024-03-06T07:30:26.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.744+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f09d006e0e0 con 0x7f09d00c4e70 2024-03-06T07:30:26.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.862+0000 7f09c7e00640 1 -- 172.21.15.114:0/2404968516 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v49) v1 ==== 97+0+0 (secure 0 0 0) 0x7f09c007acf0 con 0x7f09d00c4e70 2024-03-06T07:30:26.865 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.865+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09b006b500 msgr2=0x7f09b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.865+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09b006b500 0x7f09b006d9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f09d00c8700 tx=0x7f09bc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.865+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d00c4e70 msgr2=0x7f09d0140190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.865+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d00c4e70 0x7f09d0140190 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f09c002b950 tx=0x7f09c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 shutdown_connections 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f09d01474c0 0x7f09d00b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d01444e0 0x7f09d00cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09b006b500 0x7f09b006d9c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 --2- 172.21.15.114:0/2404968516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09d00c4e70 0x7f09d0140190 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 >> 172.21.15.114:0/2404968516 conn(0x7f09d00cc4f0 msgr2=0x7f09d00ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 shutdown_connections 2024-03-06T07:30:26.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:26.866+0000 7f09d6c00640 1 -- 172.21.15.114:0/2404968516 wait complete. 2024-03-06T07:30:26.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.084+0000 7f7702200640 1 Processor -- start 2024-03-06T07:30:27.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.084+0000 7f7702200640 1 -- start start 2024-03-06T07:30:27.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.085+0000 7f7702200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc15f9d0 0x7f76fc15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f7702200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc06cef0 con 0x7f76fc070b10 2024-03-06T07:30:27.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f7702200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc06d1d0 con 0x7f76fc144c00 2024-03-06T07:30:27.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f7702200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc160370 con 0x7f76fc15f9d0 2024-03-06T07:30:27.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f76fbe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f76fc070b10 0x7f76fc070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39922/0 (socket says 172.21.15.114:39922) 2024-03-06T07:30:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f76fbe00640 1 -- 172.21.15.114:0/3559296568 learned_addr learned my addr 172.21.15.114:0/3559296568 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:27.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f76f2a00640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc15f9d0 0x7f76fc15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504044157 0 0) 0x7f76fc06cef0 con 0x7f76fc070b10 2024-03-06T07:30:27.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.086+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e0003520 con 0x7f76fc070b10 2024-03-06T07:30:27.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1812823428 0 0) 0x7f76e0003520 con 0x7f76fc070b10 2024-03-06T07:30:27.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc15f9d0 msgr2=0x7f76fc15fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc15f9d0 0x7f76fc15fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 >> v1:172.21.15.177:6789/0 conn(0x7f76fc144c00 legacy=0x7f76fc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc06d4b0 con 0x7f76fc070b10 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f76f4001670 con 0x7f76fc070b10 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 >> v1:172.21.15.114:6789/0 conn(0x7f76fc070b10 legacy=0x7f76fc070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76e00039e0 0x7f76e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc15f9d0 0x7f76e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e0004d40 0x7f76e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e000ba70 con 0x7f76e00039e0 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e000bc20 con 0x7f76e0004d40 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e000be20 con 0x7f76fc15f9d0 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.087+0000 7f76fb400640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76e00039e0 0x7f76e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.088+0000 7f76fb400640 1 -- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc15f9d0 msgr2=0x7f76e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.088+0000 7f76fb400640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc15f9d0 0x7f76e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.088+0000 7f76fb400640 1 -- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e0004d40 msgr2=0x7f76e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.088+0000 7f76fb400640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e0004d40 0x7f76e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.088+0000 7f76fb400640 1 -- 172.21.15.114:0/3559296568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e0003520 con 0x7f76e00039e0 2024-03-06T07:30:27.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f76fb400640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76e00039e0 0x7f76e0003e00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f76ec001e20 tx=0x7f76e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2441d3902da386 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec001a70 con 0x7f76e00039e0 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ec003b70 con 0x7f76e00039e0 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f76faa00640 1 -- 172.21.15.114:0/3559296568 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec0050d0 con 0x7f76e00039e0 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f7702200640 1 -- 172.21.15.114:0/3559296568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76e00039e0 msgr2=0x7f76e0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f7702200640 1 --2- 172.21.15.114:0/3559296568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76e00039e0 0x7f76e0003e00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f76ec001e20 tx=0x7f76e0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f76fbe00640 1 -- 172.21.15.114:0/3559296568 reap_dead start 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f7702200640 1 -- 172.21.15.114:0/3559296568 shutdown_connections 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.089+0000 7f7702200640 1 -- 172.21.15.114:0/3559296568 >> 172.21.15.114:0/3559296568 conn(0x7f76fc0b1730 msgr2=0x7f76fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.090+0000 7f7702200640 1 -- 172.21.15.114:0/3559296568 shutdown_connections 2024-03-06T07:30:27.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.090+0000 7f7702200640 1 -- 172.21.15.114:0/3559296568 wait complete. 2024-03-06T07:30:27.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.091+0000 7f7702200640 1 Processor -- start 2024-03-06T07:30:27.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.091+0000 7f7702200640 1 -- start start 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.092+0000 7f7702200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e00039e0 0x7f76fc13afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.092+0000 7f7702200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76fc139d70 0x7f76fc0b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.092+0000 7f7702200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0b4010 0x7f76fc13d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.092+0000 7f7702200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc06cb50 con 0x7f76fc139d70 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.092+0000 7f7702200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc160730 con 0x7f76e00039e0 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.092+0000 7f7702200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc06ccd0 con 0x7f76fc0b4010 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.092+0000 7f76f2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76fc139d70 0x7f76fc0b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76fc139d70 0x7f76fc0b3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41220/0 (socket says 172.21.15.114:41220) 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 -- 172.21.15.114:0/1444110923 learned_addr learned my addr 172.21.15.114:0/1444110923 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 -- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0b4010 msgr2=0x7f76fc13d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76fbe00640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0b4010 0x7f76fc13d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76fb400640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e00039e0 0x7f76fc13afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0b4010 0x7f76fc13d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 -- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e00039e0 msgr2=0x7f76fc13afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e00039e0 0x7f76fc13afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 -- 172.21.15.114:0/1444110923 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc158be0 con 0x7f76fc139d70 2024-03-06T07:30:27.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f2a00640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76fc139d70 0x7f76fc0b3ad0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f76e400e7c0 tx=0x7f76e400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.093+0000 7f76f3e00640 1 -- 172.21.15.114:0/1444110923 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4017070 con 0x7f76fc139d70 2024-03-06T07:30:27.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.094+0000 7f76f3e00640 1 -- 172.21.15.114:0/1444110923 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e400c940 con 0x7f76fc139d70 2024-03-06T07:30:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.094+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76fc158ec0 con 0x7f76fc139d70 2024-03-06T07:30:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.094+0000 7f76f3e00640 1 -- 172.21.15.114:0/1444110923 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e400cd10 con 0x7f76fc139d70 2024-03-06T07:30:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.094+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc157370 con 0x7f76fc139d70 2024-03-06T07:30:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.095+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76fc157180 con 0x7f76fc139d70 2024-03-06T07:30:27.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.096+0000 7f76f3e00640 1 -- 172.21.15.114:0/1444110923 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76e401a070 con 0x7f76fc139d70 2024-03-06T07:30:27.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.097+0000 7f76f3e00640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76d806b4d0 0x7f76d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.097+0000 7f76f3e00640 1 -- 172.21.15.114:0/1444110923 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f76e4089c30 con 0x7f76fc139d70 2024-03-06T07:30:27.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.097+0000 7f76fb400640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76d806b4d0 0x7f76d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.097+0000 7f76fb400640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76d806b4d0 0x7f76d806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f76ec034000 tx=0x7f76ec039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.101+0000 7f76f3e00640 1 -- 172.21.15.114:0/1444110923 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76e4058e90 con 0x7f76fc139d70 2024-03-06T07:30:27.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.347+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f76fc04f330 con 0x7f76fc139d70 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.348+0000 7f76f3e00640 1 -- 172.21.15.114:0/1444110923 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f76e4058790 con 0x7f76fc139d70 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:30:27.351 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.352 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.353 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:30:27.354 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:30:27.358 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow r", 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.359 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:30:27.360 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.361 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:30:27.362 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:30:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76d806b4d0 msgr2=0x7f76d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76d806b4d0 0x7f76d806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f76ec034000 tx=0x7f76ec039040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76fc139d70 msgr2=0x7f76fc0b3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76fc139d70 0x7f76fc0b3ad0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f76e400e7c0 tx=0x7f76e400ec80 comp rx=0 tx=0).stop 2024-03-06T07:30:27.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 shutdown_connections 2024-03-06T07:30:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76fc0b4010 0x7f76fc13d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76d806b4d0 0x7f76d806d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76fc139d70 0x7f76fc0b3ad0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 --2- 172.21.15.114:0/1444110923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76e00039e0 0x7f76fc13afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 >> 172.21.15.114:0/1444110923 conn(0x7f76fc0b1730 msgr2=0x7f76fc0affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 shutdown_connections 2024-03-06T07:30:27.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.351+0000 7f7702200640 1 -- 172.21.15.114:0/1444110923 wait complete. 2024-03-06T07:30:27.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:27.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.573+0000 7fc777e00640 1 Processor -- start 2024-03-06T07:30:27.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.573+0000 7fc777e00640 1 -- start start 2024-03-06T07:30:27.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.574+0000 7fc777e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc77815dcd0 0x7fc7780b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.574+0000 7fc777e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815e0b0 0x7fc77806f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.574+0000 7fc777e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77806e780 con 0x7fc7780b39d0 2024-03-06T07:30:27.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.574+0000 7fc777e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc77806d770 con 0x7fc77815dcd0 2024-03-06T07:30:27.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.574+0000 7fc777e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77806ee50 con 0x7fc77815e0b0 2024-03-06T07:30:27.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.575+0000 7fc776a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc77815dcd0 0x7fc7780b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.575+0000 7fc777400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc7780b39d0 0x7fc7780b3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39934/0 (socket says 172.21.15.114:39934) 2024-03-06T07:30:27.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.575+0000 7fc777400640 1 -- 172.21.15.114:0/1675114221 learned_addr learned my addr 172.21.15.114:0/1675114221 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:27.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.575+0000 7fc76e000640 1 --2- 172.21.15.114:0/1675114221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815e0b0 0x7fc77806f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.576+0000 7fc776a00640 1 -- 172.21.15.114:0/1675114221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815e0b0 msgr2=0x7fc77806f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.576+0000 7fc776a00640 1 --2- 172.21.15.114:0/1675114221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815e0b0 0x7fc77806f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.576+0000 7fc776a00640 1 -- 172.21.15.114:0/1675114221 >> v1:172.21.15.114:6789/0 conn(0x7fc7780b39d0 legacy=0x7fc7780b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.576+0000 7fc776a00640 1 -- 172.21.15.114:0/1675114221 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77806d9c0 con 0x7fc77815dcd0 2024-03-06T07:30:27.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.576+0000 7fc776000640 1 -- 172.21.15.114:0/1675114221 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213838502 0 0) 0x7fc77806e780 con 0x7fc7780b39d0 2024-03-06T07:30:27.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.577+0000 7fc776a00640 1 --2- 172.21.15.114:0/1675114221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815dcd0 0x7fc7780b5860 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc768009d90 tx=0x7fc76802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d27b27f9ded1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.577+0000 7fc776000640 1 -- 172.21.15.114:0/1675114221 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768039070 con 0x7fc77815dcd0 2024-03-06T07:30:27.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.577+0000 7fc776000640 1 -- 172.21.15.114:0/1675114221 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76802cbc0 con 0x7fc77815dcd0 2024-03-06T07:30:27.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.577+0000 7fc776000640 1 -- 172.21.15.114:0/1675114221 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768034440 con 0x7fc77815dcd0 2024-03-06T07:30:27.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.577+0000 7fc777e00640 1 -- 172.21.15.114:0/1675114221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815dcd0 msgr2=0x7fc7780b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.577+0000 7fc777e00640 1 --2- 172.21.15.114:0/1675114221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815dcd0 0x7fc7780b5860 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc768009d90 tx=0x7fc76802be10 comp rx=0 tx=0).stop 2024-03-06T07:30:27.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.578+0000 7fc777e00640 1 -- 172.21.15.114:0/1675114221 shutdown_connections 2024-03-06T07:30:27.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.578+0000 7fc777e00640 1 --2- 172.21.15.114:0/1675114221 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815e0b0 0x7fc77806f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.578+0000 7fc777e00640 1 --2- 172.21.15.114:0/1675114221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815dcd0 0x7fc7780b5860 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.578+0000 7fc777e00640 1 -- 172.21.15.114:0/1675114221 >> 172.21.15.114:0/1675114221 conn(0x7fc77806c0d0 msgr2=0x7fc77806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.578+0000 7fc777e00640 1 -- 172.21.15.114:0/1675114221 shutdown_connections 2024-03-06T07:30:27.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.578+0000 7fc777e00640 1 -- 172.21.15.114:0/1675114221 wait complete. 2024-03-06T07:30:27.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.579+0000 7fc777e00640 1 Processor -- start 2024-03-06T07:30:27.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.580+0000 7fc777e00640 1 -- start start 2024-03-06T07:30:27.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.580+0000 7fc777e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7780b39d0 0x7fc778154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.580+0000 7fc777e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815dcd0 0x7fc778140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.580+0000 7fc777e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815e0b0 0x7fc7780b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77806e260 con 0x7fc77815e0b0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77806df80 con 0x7fc7780b39d0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780d06e0 con 0x7fc77815dcd0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815e0b0 0x7fc7780b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815e0b0 0x7fc7780b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41226/0 (socket says 172.21.15.114:41226) 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 -- 172.21.15.114:0/2969862175 learned_addr learned my addr 172.21.15.114:0/2969862175 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc776a00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7780b39d0 0x7fc778154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc76e000640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815dcd0 0x7fc778140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 -- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815dcd0 msgr2=0x7fc778140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815dcd0 0x7fc778140490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 -- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7780b39d0 msgr2=0x7fc778154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7780b39d0 0x7fc778154a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.581+0000 7fc777400640 1 -- 172.21.15.114:0/2969862175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77815fd30 con 0x7fc77815e0b0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.582+0000 7fc777400640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815e0b0 0x7fc7780b26b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc77000c450 tx=0x7fc77000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.582+0000 7fc76fe00640 1 -- 172.21.15.114:0/2969862175 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770018070 con 0x7fc77815e0b0 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.582+0000 7fc76fe00640 1 -- 172.21.15.114:0/2969862175 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc770007500 con 0x7fc77815e0b0 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.582+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc778151d40 con 0x7fc77815e0b0 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.582+0000 7fc76fe00640 1 -- 172.21.15.114:0/2969862175 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc770004a20 con 0x7fc77815e0b0 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.582+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc778152300 con 0x7fc77815e0b0 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.583+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77804f330 con 0x7fc77815e0b0 2024-03-06T07:30:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.584+0000 7fc76fe00640 1 -- 172.21.15.114:0/2969862175 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc77001b070 con 0x7fc77815e0b0 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.585+0000 7fc76fe00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.585+0000 7fc76fe00640 1 -- 172.21.15.114:0/2969862175 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc770089190 con 0x7fc77815e0b0 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.585+0000 7fc776a00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.586+0000 7fc776a00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc75406b4d0 0x7fc75406d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc768036000 tx=0x7fc768030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.589+0000 7fc76fe00640 1 -- 172.21.15.114:0/2969862175 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7700582c0 con 0x7fc77815e0b0 2024-03-06T07:30:27.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.786+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc77806c700 con 0x7fc77815e0b0 2024-03-06T07:30:27.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.788+0000 7fc76fe00640 1 -- 172.21.15.114:0/2969862175 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+12939 (secure 0 0 0) 0x7fc770057bc0 con 0x7fc77815e0b0 2024-03-06T07:30:27.791 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:27.791 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":51,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:26.874889+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc75406b4d0 msgr2=0x7fc75406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc75406b4d0 0x7fc75406d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc768036000 tx=0x7fc768030000 comp rx=0 tx=0).stop 2024-03-06T07:30:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815e0b0 msgr2=0x7fc7780b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815e0b0 0x7fc7780b26b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc77000c450 tx=0x7fc77000c910 comp rx=0 tx=0).stop 2024-03-06T07:30:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 shutdown_connections 2024-03-06T07:30:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc77815e0b0 0x7fc7780b26b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc77815dcd0 0x7fc778140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 --2- 172.21.15.114:0/2969862175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7780b39d0 0x7fc778154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 >> 172.21.15.114:0/2969862175 conn(0x7fc77806c0d0 msgr2=0x7fc7780b5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.790+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 shutdown_connections 2024-03-06T07:30:27.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:27.791+0000 7fc777e00640 1 -- 172.21.15.114:0/2969862175 wait complete. 2024-03-06T07:30:27.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:27.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:28.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.022+0000 7f02cd200640 1 Processor -- start 2024-03-06T07:30:28.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.022+0000 7f02cd200640 1 -- start start 2024-03-06T07:30:28.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02cd200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02c80b3650 0x7f02c80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02cd200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80cda90 0x7f02c806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02cd200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c80719e0 con 0x7f02c80b3df0 2024-03-06T07:30:28.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02cd200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c8072310 con 0x7f02c80b3650 2024-03-06T07:30:28.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02cd200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c8071cc0 con 0x7f02c80cda90 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02c80b3650 0x7f02c80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c5e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f02c80b3df0 0x7f02c80b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42020/0 (socket says 172.21.15.114:42020) 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c5e00640 1 -- 172.21.15.114:0/2798536471 learned_addr learned my addr 172.21.15.114:0/2798536471 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c5400640 1 -- 172.21.15.114:0/2798536471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80cda90 msgr2=0x7f02c806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c4a00640 1 -- 172.21.15.114:0/2798536471 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285395371 0 0) 0x7f02c80719e0 con 0x7f02c80b3df0 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c5400640 1 --2- 172.21.15.114:0/2798536471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80cda90 0x7f02c806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c5400640 1 -- 172.21.15.114:0/2798536471 >> v1:172.21.15.114:6789/0 conn(0x7f02c80b3df0 legacy=0x7f02c80b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.024+0000 7f02c5400640 1 -- 172.21.15.114:0/2798536471 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c8071fa0 con 0x7f02c80b3650 2024-03-06T07:30:28.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.026+0000 7f02c5400640 1 --2- 172.21.15.114:0/2798536471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3650 0x7f02c80727a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009d90 tx=0x7f02b002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e93e985e7484fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.026+0000 7f02c4a00640 1 -- 172.21.15.114:0/2798536471 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0039070 con 0x7f02c80b3650 2024-03-06T07:30:28.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.026+0000 7f02c4a00640 1 -- 172.21.15.114:0/2798536471 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b002cc50 con 0x7f02c80b3650 2024-03-06T07:30:28.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.026+0000 7f02c4a00640 1 -- 172.21.15.114:0/2798536471 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0034440 con 0x7f02c80b3650 2024-03-06T07:30:28.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 -- 172.21.15.114:0/2798536471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3650 msgr2=0x7f02c80727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 --2- 172.21.15.114:0/2798536471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3650 0x7f02c80727a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009d90 tx=0x7f02b002bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 -- 172.21.15.114:0/2798536471 shutdown_connections 2024-03-06T07:30:28.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 --2- 172.21.15.114:0/2798536471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80cda90 0x7f02c806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 --2- 172.21.15.114:0/2798536471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3650 0x7f02c80727a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 -- 172.21.15.114:0/2798536471 >> 172.21.15.114:0/2798536471 conn(0x7f02c80b1940 msgr2=0x7f02c80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 -- 172.21.15.114:0/2798536471 shutdown_connections 2024-03-06T07:30:28.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.027+0000 7f02cd200640 1 -- 172.21.15.114:0/2798536471 wait complete. 2024-03-06T07:30:28.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.028+0000 7f02cd200640 1 Processor -- start 2024-03-06T07:30:28.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.029+0000 7f02cd200640 1 -- start start 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.029+0000 7f02cd200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80b3650 0x7f02c80c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.029+0000 7f02cd200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3df0 0x7f02c809d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.029+0000 7f02cd200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02c80cda90 0x7f02c80acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.029+0000 7f02cd200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c815b3c0 con 0x7f02c80b3df0 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.029+0000 7f02cd200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c815b240 con 0x7f02c80cda90 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.029+0000 7f02cd200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80cf510 con 0x7f02c80b3650 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3df0 0x7f02c809d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3df0 0x7f02c809d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39740/0 (socket says 172.21.15.114:39740) 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 -- 172.21.15.114:0/1669115344 learned_addr learned my addr 172.21.15.114:0/1669115344 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5e00640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80b3650 0x7f02c80c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c6800640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02c80cda90 0x7f02c80acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 -- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80b3650 msgr2=0x7f02c80c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80b3650 0x7f02c80c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 -- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02c80cda90 msgr2=0x7f02c80acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02c80cda90 0x7f02c80acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.030+0000 7f02c5400640 1 -- 172.21.15.114:0/1669115344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80c9500 con 0x7f02c80b3df0 2024-03-06T07:30:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.031+0000 7f02c5400640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3df0 0x7f02c809d210 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f02b0037040 tx=0x7f02b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.031+0000 7f02be000640 1 -- 172.21.15.114:0/1669115344 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0044020 con 0x7f02c80b3df0 2024-03-06T07:30:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.031+0000 7f02be000640 1 -- 172.21.15.114:0/1669115344 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b003f470 con 0x7f02c80b3df0 2024-03-06T07:30:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.031+0000 7f02be000640 1 -- 172.21.15.114:0/1669115344 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0039040 con 0x7f02c80b3df0 2024-03-06T07:30:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.031+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c81f4830 con 0x7f02c80b3df0 2024-03-06T07:30:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.031+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c81f50d0 con 0x7f02c80b3df0 2024-03-06T07:30:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.032+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c804fb80 con 0x7f02c80b3df0 2024-03-06T07:30:28.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.033+0000 7f02be000640 1 -- 172.21.15.114:0/1669115344 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02b0047020 con 0x7f02c80b3df0 2024-03-06T07:30:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.035+0000 7f02be000640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02a406b4d0 0x7f02a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.035+0000 7f02be000640 1 -- 172.21.15.114:0/1669115344 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f02b00ac8d0 con 0x7f02c80b3df0 2024-03-06T07:30:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.035+0000 7f02c5e00640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02a406b4d0 0x7f02a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.035+0000 7f02c5e00640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02a406b4d0 0x7f02a406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000f60 tx=0x7f02b8007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.039+0000 7f02be000640 1 -- 172.21.15.114:0/1669115344 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02c804fb80 con 0x7f02c80b3df0 2024-03-06T07:30:28.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.238+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02c804f330 con 0x7f02c80b3df0 2024-03-06T07:30:28.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.238+0000 7f02be000640 1 -- 172.21.15.114:0/1669115344 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02b004e030 con 0x7f02c80b3df0 2024-03-06T07:30:28.241 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02a406b4d0 msgr2=0x7f02a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02a406b4d0 0x7f02a406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000f60 tx=0x7f02b8007b50 comp rx=0 tx=0).stop 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3df0 msgr2=0x7f02c809d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3df0 0x7f02c809d210 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f02b0037040 tx=0x7f02b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 shutdown_connections 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02c80cda90 0x7f02c80acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02a406b4d0 0x7f02a406d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c80b3df0 0x7f02c809d210 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 --2- 172.21.15.114:0/1669115344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02c80b3650 0x7f02c80c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 >> 172.21.15.114:0/1669115344 conn(0x7f02c80b1940 msgr2=0x7f02c80b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 shutdown_connections 2024-03-06T07:30:28.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.241+0000 7f02cd200640 1 -- 172.21.15.114:0/1669115344 wait complete. 2024-03-06T07:30:28.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.468+0000 7f9849000640 1 Processor -- start 2024-03-06T07:30:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.468+0000 7f9849000640 1 -- start start 2024-03-06T07:30:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9849000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f984415fe60 0x7f9844071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9849000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984406cef0 con 0x7f9844144c00 2024-03-06T07:30:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9849000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984406d1d0 con 0x7f984415f9d0 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9849000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984406d4b0 con 0x7f984415fe60 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9841e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9844144c00 0x7f984406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42032/0 (socket says 172.21.15.114:42032) 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9841e00640 1 -- 172.21.15.114:0/751750703 learned_addr learned my addr 172.21.15.114:0/751750703 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9842800640 1 --2- 172.21.15.114:0/751750703 >> v2:172.21.15.114:3300/0 conn(0x7f984415fe60 0x7f9844071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9841400640 1 -- 172.21.15.114:0/751750703 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623607988 0 0) 0x7f984406cef0 con 0x7f9844144c00 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9841400640 1 -- 172.21.15.114:0/751750703 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9828003520 con 0x7f9844144c00 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9842800640 1 -- 172.21.15.114:0/751750703 >> v1:172.21.15.177:6789/0 conn(0x7f984415f9d0 legacy=0x7f9844070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.470+0000 7f9842800640 1 -- 172.21.15.114:0/751750703 >> v1:172.21.15.114:6789/0 conn(0x7f9844144c00 legacy=0x7f984406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.471+0000 7f9842800640 1 -- 172.21.15.114:0/751750703 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984406d790 con 0x7f984415fe60 2024-03-06T07:30:28.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9842800640 1 --2- 172.21.15.114:0/751750703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f9844071220 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f983c009d90 tx=0x7f983c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0731fad4a85051 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9841400640 1 -- 172.21.15.114:0/751750703 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c039070 con 0x7f984415fe60 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9841400640 1 -- 172.21.15.114:0/751750703 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c02cc60 con 0x7f984415fe60 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9841400640 1 -- 172.21.15.114:0/751750703 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c034660 con 0x7f984415fe60 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9849000640 1 -- 172.21.15.114:0/751750703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 msgr2=0x7f9844071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9849000640 1 --2- 172.21.15.114:0/751750703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f9844071220 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f983c009d90 tx=0x7f983c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9849000640 1 -- 172.21.15.114:0/751750703 shutdown_connections 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.472+0000 7f9849000640 1 --2- 172.21.15.114:0/751750703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f9844071220 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.473+0000 7f9849000640 1 -- 172.21.15.114:0/751750703 >> 172.21.15.114:0/751750703 conn(0x7f98440b1730 msgr2=0x7f98440b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.473+0000 7f9849000640 1 -- 172.21.15.114:0/751750703 shutdown_connections 2024-03-06T07:30:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.473+0000 7f9849000640 1 -- 172.21.15.114:0/751750703 wait complete. 2024-03-06T07:30:28.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.474+0000 7f9849000640 1 Processor -- start 2024-03-06T07:30:28.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.474+0000 7f9849000640 1 -- start start 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.475+0000 7f9849000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9844144c00 0x7f9844139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.475+0000 7f9849000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f984415f9d0 0x7f98440cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.475+0000 7f9849000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f984413c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.475+0000 7f9849000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984406cb50 con 0x7f984415fe60 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.475+0000 7f9849000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9844160730 con 0x7f9844144c00 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.475+0000 7f9849000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984406ccd0 con 0x7f984415f9d0 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f984413c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f984413c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39752/0 (socket says 172.21.15.114:39752) 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 -- 172.21.15.114:0/1228754123 learned_addr learned my addr 172.21.15.114:0/1228754123 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9841e00640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9844144c00 0x7f9844139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f983be00640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f984415f9d0 0x7f98440cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 -- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f984415f9d0 msgr2=0x7f98440cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f984415f9d0 0x7f98440cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 -- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9844144c00 msgr2=0x7f9844139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9844144c00 0x7f9844139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 -- 172.21.15.114:0/1228754123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984413bbc0 con 0x7f984415fe60 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f983be00640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f984415f9d0 0x7f98440cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.476+0000 7f9842800640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f984413c980 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f983c009d90 tx=0x7f983c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.477+0000 7f983a000640 1 -- 172.21.15.114:0/1228754123 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c039050 con 0x7f984415fe60 2024-03-06T07:30:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.477+0000 7f983a000640 1 -- 172.21.15.114:0/1228754123 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c045070 con 0x7f984415fe60 2024-03-06T07:30:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.477+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984413bea0 con 0x7f984415fe60 2024-03-06T07:30:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.477+0000 7f983a000640 1 -- 172.21.15.114:0/1228754123 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c009450 con 0x7f984415fe60 2024-03-06T07:30:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.477+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984413a4c0 con 0x7f984415fe60 2024-03-06T07:30:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.478+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984404f330 con 0x7f984415fe60 2024-03-06T07:30:28.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.479+0000 7f983a000640 1 -- 172.21.15.114:0/1228754123 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f983c03e850 con 0x7f984415fe60 2024-03-06T07:30:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.480+0000 7f983a000640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f982006b500 0x7f982006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.480+0000 7f983a000640 1 -- 172.21.15.114:0/1228754123 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f983c0accc0 con 0x7f984415fe60 2024-03-06T07:30:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.480+0000 7f9841e00640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f982006b500 0x7f982006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.481+0000 7f9841e00640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f982006b500 0x7f982006d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9834005880 tx=0x7f9834009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.485+0000 7f983a000640 1 -- 172.21.15.114:0/1228754123 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f983c032e20 con 0x7f984415fe60 2024-03-06T07:30:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.687+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f98440b1d60 con 0x7f984415fe60 2024-03-06T07:30:28.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.688+0000 7f983a000640 1 -- 172.21.15.114:0/1228754123 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v51) v1 ==== 84+0+65 (secure 0 0 0) 0x7f983c07bfe0 con 0x7f984415fe60 2024-03-06T07:30:28.690 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:28.690 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:30:28.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:30:28.691 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:30:28.691 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:30:28.691 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:30:28.691 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.690+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f982006b500 msgr2=0x7f982006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.690+0000 7f9849000640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f982006b500 0x7f982006d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9834005880 tx=0x7f9834009040 comp rx=0 tx=0).stop 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.690+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 msgr2=0x7f984413c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.690+0000 7f9849000640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f984413c980 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f983c009d90 tx=0x7f983c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 shutdown_connections 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f982006b500 0x7f982006d9c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f984415fe60 0x7f984413c980 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f984415f9d0 0x7f98440cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 --2- 172.21.15.114:0/1228754123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9844144c00 0x7f9844139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 >> 172.21.15.114:0/1228754123 conn(0x7f98440b1730 msgr2=0x7f98440af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 shutdown_connections 2024-03-06T07:30:28.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.691+0000 7f9849000640 1 -- 172.21.15.114:0/1228754123 wait complete. 2024-03-06T07:30:28.711 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:28.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.923+0000 7fd7db200640 1 Processor -- start 2024-03-06T07:30:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.923+0000 7fd7db200640 1 -- start start 2024-03-06T07:30:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.925+0000 7fd7db200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd7d4158070 0x7fd7d415b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.925+0000 7fd7db200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7d40cc3a0 0x7fd7d40cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.925+0000 7fd7db200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d40cccf0 con 0x7fd7d40b4b40 2024-03-06T07:30:28.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.925+0000 7fd7db200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d40728e0 con 0x7fd7d40cc3a0 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.925+0000 7fd7db200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d4072bc0 con 0x7fd7d4158070 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7da800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7d40cc3a0 0x7fd7d40cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7d9400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd7d40b4b40 0x7fd7d4072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42038/0 (socket says 172.21.15.114:42038) 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7d9400640 1 -- 172.21.15.114:0/2647668794 learned_addr learned my addr 172.21.15.114:0/2647668794 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7d9e00640 1 --2- 172.21.15.114:0/2647668794 >> v2:172.21.15.177:3300/0 conn(0x7fd7d4158070 0x7fd7d415b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7da800640 1 -- 172.21.15.114:0/2647668794 >> v2:172.21.15.177:3300/0 conn(0x7fd7d4158070 msgr2=0x7fd7d415b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7da800640 1 --2- 172.21.15.114:0/2647668794 >> v2:172.21.15.177:3300/0 conn(0x7fd7d4158070 0x7fd7d415b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7da800640 1 -- 172.21.15.114:0/2647668794 >> v1:172.21.15.114:6789/0 conn(0x7fd7d40b4b40 legacy=0x7fd7d4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7da800640 1 -- 172.21.15.114:0/2647668794 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d415aa50 con 0x7fd7d40cc3a0 2024-03-06T07:30:28.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7d9e00640 1 --2- 172.21.15.114:0/2647668794 >> v2:172.21.15.177:3300/0 conn(0x7fd7d4158070 0x7fd7d415b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:28.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.926+0000 7fd7d8a00640 1 -- 172.21.15.114:0/2647668794 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578720264 0 0) 0x7fd7c0001110 con 0x7fd7d40b4b40 2024-03-06T07:30:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.927+0000 7fd7da800640 1 --2- 172.21.15.114:0/2647668794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d40cc3a0 0x7fd7d40cc7b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd7d406ee80 tx=0x7fd7d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db9f68280e95c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.927+0000 7fd7d8a00640 1 -- 172.21.15.114:0/2647668794 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0039070 con 0x7fd7d40cc3a0 2024-03-06T07:30:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.927+0000 7fd7d8a00640 1 -- 172.21.15.114:0/2647668794 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d002cc80 con 0x7fd7d40cc3a0 2024-03-06T07:30:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7d8a00640 1 -- 172.21.15.114:0/2647668794 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0034660 con 0x7fd7d40cc3a0 2024-03-06T07:30:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 -- 172.21.15.114:0/2647668794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d40cc3a0 msgr2=0x7fd7d40cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 --2- 172.21.15.114:0/2647668794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d40cc3a0 0x7fd7d40cc7b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd7d406ee80 tx=0x7fd7d002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 -- 172.21.15.114:0/2647668794 shutdown_connections 2024-03-06T07:30:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 --2- 172.21.15.114:0/2647668794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d40cc3a0 0x7fd7d40cc7b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 --2- 172.21.15.114:0/2647668794 >> v2:172.21.15.177:3300/0 conn(0x7fd7d4158070 0x7fd7d415b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 -- 172.21.15.114:0/2647668794 >> 172.21.15.114:0/2647668794 conn(0x7fd7d40b18a0 msgr2=0x7fd7d40b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 -- 172.21.15.114:0/2647668794 shutdown_connections 2024-03-06T07:30:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.928+0000 7fd7db200640 1 -- 172.21.15.114:0/2647668794 wait complete. 2024-03-06T07:30:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.930+0000 7fd7db200640 1 Processor -- start 2024-03-06T07:30:28.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.930+0000 7fd7db200640 1 -- start start 2024-03-06T07:30:28.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7db200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7d40b4b40 0x7fd7d414e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7db200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7d40cc3a0 0x7fd7d41f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7db200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d4158070 0x7fd7d41564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7db200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d406afc0 con 0x7fd7d4158070 2024-03-06T07:30:28.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7db200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d415b010 con 0x7fd7d40cc3a0 2024-03-06T07:30:28.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7db200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d406b140 con 0x7fd7d40b4b40 2024-03-06T07:30:28.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7da800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d4158070 0x7fd7d41564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7da800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d4158070 0x7fd7d41564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39772/0 (socket says 172.21.15.114:39772) 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.931+0000 7fd7da800640 1 -- 172.21.15.114:0/3387935810 learned_addr learned my addr 172.21.15.114:0/3387935810 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7da800640 1 -- 172.21.15.114:0/3387935810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7d40b4b40 msgr2=0x7fd7d414e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7da800640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7d40b4b40 0x7fd7d414e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7da800640 1 -- 172.21.15.114:0/3387935810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7d40cc3a0 msgr2=0x7fd7d41f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7da800640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7d40cc3a0 0x7fd7d41f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7da800640 1 -- 172.21.15.114:0/3387935810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d4167520 con 0x7fd7d4158070 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7da800640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d4158070 0x7fd7d41564b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009e80 tx=0x7fd7d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7ce000640 1 -- 172.21.15.114:0/3387935810 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0044020 con 0x7fd7d4158070 2024-03-06T07:30:28.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7ce000640 1 -- 172.21.15.114:0/3387935810 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d003a040 con 0x7fd7d4158070 2024-03-06T07:30:28.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7ce000640 1 -- 172.21.15.114:0/3387935810 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0039050 con 0x7fd7d4158070 2024-03-06T07:30:28.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d4167800 con 0x7fd7d4158070 2024-03-06T07:30:28.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.932+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d4167dc0 con 0x7fd7d4158070 2024-03-06T07:30:28.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.934+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d404f330 con 0x7fd7d4158070 2024-03-06T07:30:28.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.934+0000 7fd7ce000640 1 -- 172.21.15.114:0/3387935810 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7d0003b90 con 0x7fd7d4158070 2024-03-06T07:30:28.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.935+0000 7fd7ce000640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7b406b4d0 0x7fd7b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.935+0000 7fd7ce000640 1 -- 172.21.15.114:0/3387935810 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd7d00ac670 con 0x7fd7d4158070 2024-03-06T07:30:28.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.935+0000 7fd7d9e00640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7b406b4d0 0x7fd7b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.936+0000 7fd7d9e00640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7b406b4d0 0x7fd7b406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8006fd0 tx=0x7fd7c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:28.940+0000 7fd7ce000640 1 -- 172.21.15.114:0/3387935810 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7d004e020 con 0x7fd7d4158070 2024-03-06T07:30:29.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.139+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd7d40cf0f0 con 0x7fd7d4158070 2024-03-06T07:30:29.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.883+0000 7fd7ce000640 1 -- 172.21.15.114:0/3387935810 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v52) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd7d007b830 con 0x7fd7d4158070 2024-03-06T07:30:29.886 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7b406b4d0 msgr2=0x7fd7b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7b406b4d0 0x7fd7b406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8006fd0 tx=0x7fd7c8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d4158070 msgr2=0x7fd7d41564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d4158070 0x7fd7d41564b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009e80 tx=0x7fd7d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 shutdown_connections 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7b406b4d0 0x7fd7b406d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7d4158070 0x7fd7d41564b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7d40cc3a0 0x7fd7d41f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 --2- 172.21.15.114:0/3387935810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7d40b4b40 0x7fd7d414e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 >> 172.21.15.114:0/3387935810 conn(0x7fd7d40b18a0 msgr2=0x7fd7d4150f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 shutdown_connections 2024-03-06T07:30:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:29.887+0000 7fd7db200640 1 -- 172.21.15.114:0/3387935810 wait complete. 2024-03-06T07:30:29.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:30.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.113+0000 7fef4de00640 1 Processor -- start 2024-03-06T07:30:30.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.113+0000 7fef4de00640 1 -- start start 2024-03-06T07:30:30.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.115+0000 7fef4de00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fef4815fc10 0x7fef48070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.115+0000 7fef4de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef480ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.115+0000 7fef4de00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef480b46c0 con 0x7fef4815fff0 2024-03-06T07:30:30.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.115+0000 7fef4de00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef480b49a0 con 0x7fef4815fc10 2024-03-06T07:30:30.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.115+0000 7fef4de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef48072c90 con 0x7fef480b3f70 2024-03-06T07:30:30.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.116+0000 7fef47400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef480ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.116+0000 7fef47400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef480ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:39486/0 (socket says 172.21.15.114:39486) 2024-03-06T07:30:30.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.116+0000 7fef47400640 1 -- 172.21.15.114:0/3883870823 learned_addr learned my addr 172.21.15.114:0/3883870823 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:30.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.116+0000 7fef46a00640 1 --2- 172.21.15.114:0/3883870823 >> v2:172.21.15.177:3300/0 conn(0x7fef4815fc10 0x7fef48070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.117+0000 7fef47400640 1 -- 172.21.15.114:0/3883870823 >> v2:172.21.15.177:3300/0 conn(0x7fef4815fc10 msgr2=0x7fef48070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:30.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.117+0000 7fef47400640 1 --2- 172.21.15.114:0/3883870823 >> v2:172.21.15.177:3300/0 conn(0x7fef4815fc10 0x7fef48070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.117+0000 7fef47400640 1 -- 172.21.15.114:0/3883870823 >> v1:172.21.15.177:6789/0 conn(0x7fef4815fff0 legacy=0x7fef48160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.117+0000 7fef47400640 1 -- 172.21.15.114:0/3883870823 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4809dd80 con 0x7fef480b3f70 2024-03-06T07:30:30.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.118+0000 7fef47400640 1 --2- 172.21.15.114:0/3883870823 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef480ce4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fef40009e50 tx=0x7fef4002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dcc6cd3b8ca5eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.119+0000 7fef45600640 1 -- 172.21.15.114:0/3883870823 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4002cb00 con 0x7fef480b3f70 2024-03-06T07:30:30.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.119+0000 7fef45600640 1 -- 172.21.15.114:0/3883870823 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4002cca0 con 0x7fef480b3f70 2024-03-06T07:30:30.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.119+0000 7fef45600640 1 -- 172.21.15.114:0/3883870823 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40034610 con 0x7fef480b3f70 2024-03-06T07:30:30.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 -- 172.21.15.114:0/3883870823 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 msgr2=0x7fef480ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 --2- 172.21.15.114:0/3883870823 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef480ce4a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fef40009e50 tx=0x7fef4002bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:30.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 -- 172.21.15.114:0/3883870823 shutdown_connections 2024-03-06T07:30:30.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 --2- 172.21.15.114:0/3883870823 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef480ce4a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 --2- 172.21.15.114:0/3883870823 >> v2:172.21.15.177:3300/0 conn(0x7fef4815fc10 0x7fef48070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 -- 172.21.15.114:0/3883870823 >> 172.21.15.114:0/3883870823 conn(0x7fef480b1940 msgr2=0x7fef480b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 -- 172.21.15.114:0/3883870823 shutdown_connections 2024-03-06T07:30:30.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.120+0000 7fef4de00640 1 -- 172.21.15.114:0/3883870823 wait complete. 2024-03-06T07:30:30.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.122+0000 7fef4de00640 1 Processor -- start 2024-03-06T07:30:30.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.122+0000 7fef4de00640 1 -- start start 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef4de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef4806e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef4de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef4815fc10 0x7fef481f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef4de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4815fff0 0x7fef48154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef4de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4815c460 con 0x7fef4815fff0 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef4de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4815c2e0 con 0x7fef4815fc10 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef4de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef480ce9e0 con 0x7fef480b3f70 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef47400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4815fff0 0x7fef48154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef47400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4815fff0 0x7fef48154770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39776/0 (socket says 172.21.15.114:39776) 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef47400640 1 -- 172.21.15.114:0/3241712472 learned_addr learned my addr 172.21.15.114:0/3241712472 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef46000640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef4815fc10 0x7fef481f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.123+0000 7fef46a00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef4806e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef47400640 1 -- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 msgr2=0x7fef4806e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef47400640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef4806e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef47400640 1 -- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef4815fc10 msgr2=0x7fef481f40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef47400640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef4815fc10 0x7fef481f40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef47400640 1 -- 172.21.15.114:0/3241712472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef481f4820 con 0x7fef4815fff0 2024-03-06T07:30:30.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef47400640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4815fff0 0x7fef48154770 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fef40036040 tx=0x7fef40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef3f400640 1 -- 172.21.15.114:0/3241712472 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40043020 con 0x7fef4815fff0 2024-03-06T07:30:30.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef3f400640 1 -- 172.21.15.114:0/3241712472 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4002ce50 con 0x7fef4815fff0 2024-03-06T07:30:30.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef481f4b00 con 0x7fef4815fff0 2024-03-06T07:30:30.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef3f400640 1 -- 172.21.15.114:0/3241712472 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40009510 con 0x7fef4815fff0 2024-03-06T07:30:30.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.124+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef48154cb0 con 0x7fef4815fff0 2024-03-06T07:30:30.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.126+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4804f330 con 0x7fef4815fff0 2024-03-06T07:30:30.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.126+0000 7fef3f400640 1 -- 172.21.15.114:0/3241712472 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef40046020 con 0x7fef4815fff0 2024-03-06T07:30:30.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.128+0000 7fef3f400640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2806b4d0 0x7fef2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.128+0000 7fef3f400640 1 -- 172.21.15.114:0/3241712472 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fef400ac510 con 0x7fef4815fff0 2024-03-06T07:30:30.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.128+0000 7fef46a00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2806b4d0 0x7fef2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.129+0000 7fef46a00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2806b4d0 0x7fef2806d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ad0 tx=0x7fef38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.133+0000 7fef3f400640 1 -- 172.21.15.114:0/3241712472 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef4004d030 con 0x7fef4815fff0 2024-03-06T07:30:30.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.331+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fef4806ef30 con 0x7fef4815fff0 2024-03-06T07:30:30.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.890+0000 7fef3f400640 1 -- 172.21.15.114:0/3241712472 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v53) v1 ==== 170+0+0 (secure 0 0 0) 0x7fef4007b560 con 0x7fef4815fff0 2024-03-06T07:30:30.893 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2806b4d0 msgr2=0x7fef2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2806b4d0 0x7fef2806d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fef38009ad0 tx=0x7fef38008040 comp rx=0 tx=0).stop 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4815fff0 msgr2=0x7fef48154770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4815fff0 0x7fef48154770 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fef40036040 tx=0x7fef40004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 shutdown_connections 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2806b4d0 0x7fef2806d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4815fff0 0x7fef48154770 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef4815fc10 0x7fef481f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 --2- 172.21.15.114:0/3241712472 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b3f70 0x7fef4806e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 >> 172.21.15.114:0/3241712472 conn(0x7fef480b1940 msgr2=0x7fef48070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 shutdown_connections 2024-03-06T07:30:30.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:30.894+0000 7fef4de00640 1 -- 172.21.15.114:0/3241712472 wait complete. 2024-03-06T07:30:30.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.115+0000 7f50d0a00640 1 Processor -- start 2024-03-06T07:30:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.115+0000 7f50d0a00640 1 -- start start 2024-03-06T07:30:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50d0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f50cc15df80 0x7f50cc06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50d0a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15e360 0x7f50cc0cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50d0a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f50cc0d00d0 0x7f50cc161d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50d0a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc0d2a20 con 0x7f50cc15df80 2024-03-06T07:30:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50d0a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc0d1fb0 con 0x7f50cc0d00d0 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50d0a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc09b100 con 0x7f50cc15e360 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50c9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f50cc15df80 0x7f50cc06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50c9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f50cc15df80 0x7f50cc06c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39786/0 (socket says 172.21.15.114:39786) 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.117+0000 7f50c9e00640 1 -- 172.21.15.114:0/3546900068 learned_addr learned my addr 172.21.15.114:0/3546900068 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.118+0000 7f50c9e00640 1 -- 172.21.15.114:0/3546900068 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15e360 msgr2=0x7f50cc0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.118+0000 7f50c9400640 1 --2- 172.21.15.114:0/3546900068 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15e360 0x7f50cc0cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.118+0000 7f50c9e00640 1 --2- 172.21.15.114:0/3546900068 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15e360 0x7f50cc0cfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.118+0000 7f50c9e00640 1 -- 172.21.15.114:0/3546900068 >> v2:172.21.15.177:3300/0 conn(0x7f50cc0d00d0 msgr2=0x7f50cc161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.118+0000 7f50c9e00640 1 --2- 172.21.15.114:0/3546900068 >> v2:172.21.15.177:3300/0 conn(0x7f50cc0d00d0 0x7f50cc161d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.118+0000 7f50c9e00640 1 -- 172.21.15.114:0/3546900068 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc09b3a0 con 0x7f50cc15df80 2024-03-06T07:30:31.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.119+0000 7f50c9e00640 1 --2- 172.21.15.114:0/3546900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15df80 0x7f50cc06c0d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0cfa60 tx=0x7f50bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e9002de47253f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.119+0000 7f50c8a00640 1 -- 172.21.15.114:0/3546900068 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc035020 con 0x7f50cc15df80 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.119+0000 7f50c8a00640 1 -- 172.21.15.114:0/3546900068 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc031870 con 0x7f50cc15df80 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.119+0000 7f50c8a00640 1 -- 172.21.15.114:0/3546900068 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc031c40 con 0x7f50cc15df80 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 -- 172.21.15.114:0/3546900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15df80 msgr2=0x7f50cc06c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 --2- 172.21.15.114:0/3546900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15df80 0x7f50cc06c0d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0cfa60 tx=0x7f50bc02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 -- 172.21.15.114:0/3546900068 shutdown_connections 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 --2- 172.21.15.114:0/3546900068 >> v2:172.21.15.177:3300/0 conn(0x7f50cc0d00d0 0x7f50cc161d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 --2- 172.21.15.114:0/3546900068 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15e360 0x7f50cc0cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 --2- 172.21.15.114:0/3546900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15df80 0x7f50cc06c0d0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 -- 172.21.15.114:0/3546900068 >> 172.21.15.114:0/3546900068 conn(0x7f50cc0b7320 msgr2=0x7f50cc09a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 -- 172.21.15.114:0/3546900068 shutdown_connections 2024-03-06T07:30:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.120+0000 7f50d0a00640 1 -- 172.21.15.114:0/3546900068 wait complete. 2024-03-06T07:30:31.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.122+0000 7f50d0a00640 1 Processor -- start 2024-03-06T07:30:31.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.122+0000 7f50d0a00640 1 -- start start 2024-03-06T07:30:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50d0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50cc0d00d0 0x7f50cc14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50d0a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15df80 0x7f50cc09f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50d0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15e360 0x7f50cc09d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50d0a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc06c610 con 0x7f50cc15e360 2024-03-06T07:30:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50d0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0d2500 con 0x7f50cc0d00d0 2024-03-06T07:30:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50d0a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc09b680 con 0x7f50cc15df80 2024-03-06T07:30:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50ca800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15e360 0x7f50cc09d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50ca800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15e360 0x7f50cc09d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39802/0 (socket says 172.21.15.114:39802) 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50ca800640 1 -- 172.21.15.114:0/2603154563 learned_addr learned my addr 172.21.15.114:0/2603154563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.123+0000 7f50c9400640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15df80 0x7f50cc09f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50c9e00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50cc0d00d0 0x7f50cc14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50ca800640 1 -- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15df80 msgr2=0x7f50cc09f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50ca800640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15df80 0x7f50cc09f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50ca800640 1 -- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50cc0d00d0 msgr2=0x7f50cc14e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50ca800640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50cc0d00d0 0x7f50cc14e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50ca800640 1 -- 172.21.15.114:0/2603154563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc15ec40 con 0x7f50cc15e360 2024-03-06T07:30:31.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50ca800640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15e360 0x7f50cc09d420 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f50c400b440 tx=0x7f50c400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50c2000640 1 -- 172.21.15.114:0/2603154563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4015020 con 0x7f50cc15e360 2024-03-06T07:30:31.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50c2000640 1 -- 172.21.15.114:0/2603154563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c4011830 con 0x7f50cc15e360 2024-03-06T07:30:31.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50c2000640 1 -- 172.21.15.114:0/2603154563 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4011c20 con 0x7f50cc15e360 2024-03-06T07:30:31.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc15ef20 con 0x7f50cc15e360 2024-03-06T07:30:31.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.124+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc15f4e0 con 0x7f50cc15e360 2024-03-06T07:30:31.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.126+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc04f330 con 0x7f50cc15e360 2024-03-06T07:30:31.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.127+0000 7f50c2000640 1 -- 172.21.15.114:0/2603154563 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50c4018020 con 0x7f50cc15e360 2024-03-06T07:30:31.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.128+0000 7f50c2000640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f50a806b4d0 0x7f50a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.128+0000 7f50c2000640 1 -- 172.21.15.114:0/2603154563 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f50c408a700 con 0x7f50cc15e360 2024-03-06T07:30:31.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.128+0000 7f50c9e00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f50a806b4d0 0x7f50a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.129+0000 7f50c9e00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f50a806b4d0 0x7f50a806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f50bc036000 tx=0x7f50bc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.132+0000 7f50c2000640 1 -- 172.21.15.114:0/2603154563 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50c401f020 con 0x7f50cc15e360 2024-03-06T07:30:31.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.360+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f50cc074b70 con 0x7f50cc15e360 2024-03-06T07:30:31.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.938+0000 7f50c2000640 1 -- 172.21.15.114:0/2603154563 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v31) v1 ==== 172+0+0 (secure 0 0 0) 0x7f50c400f2e0 con 0x7f50cc15e360 2024-03-06T07:30:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:30:31.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f50a806b4d0 msgr2=0x7f50a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f50a806b4d0 0x7f50a806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f50bc036000 tx=0x7f50bc03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:31.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15e360 msgr2=0x7f50cc09d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15e360 0x7f50cc09d420 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f50c400b440 tx=0x7f50c400b900 comp rx=0 tx=0).stop 2024-03-06T07:30:31.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 shutdown_connections 2024-03-06T07:30:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f50a806b4d0 0x7f50a806d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50cc15e360 0x7f50cc09d420 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50cc15df80 0x7f50cc09f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 --2- 172.21.15.114:0/2603154563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50cc0d00d0 0x7f50cc14e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 >> 172.21.15.114:0/2603154563 conn(0x7f50cc0b7320 msgr2=0x7f50cc0b6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 shutdown_connections 2024-03-06T07:30:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:31.952+0000 7f50d0a00640 1 -- 172.21.15.114:0/2603154563 wait complete. 2024-03-06T07:30:31.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.160+0000 7f9a07400640 1 Processor -- start 2024-03-06T07:30:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.160+0000 7f9a07400640 1 -- start start 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.161+0000 7f9a07400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a0015eff0 0x7f9a00070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a07400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0006e4d0 0x7f9a0006e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a07400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a00071d30 con 0x7f9a000af1c0 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a07400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a00071260 con 0x7f9a0015eff0 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a07400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0015f4a0 con 0x7f9a0006e4d0 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f99ffe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a0015eff0 0x7f9a00070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a04a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9a000af1c0 0x7f9a0006cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42040/0 (socket says 172.21.15.114:42040) 2024-03-06T07:30:32.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a04a00640 1 -- 172.21.15.114:0/136539903 learned_addr learned my addr 172.21.15.114:0/136539903 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a05400640 1 --2- 172.21.15.114:0/136539903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0006e4d0 0x7f9a0006e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f99ffe00640 1 -- 172.21.15.114:0/136539903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0006e4d0 msgr2=0x7f9a0006e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f99ffe00640 1 --2- 172.21.15.114:0/136539903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0006e4d0 0x7f9a0006e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f99ffe00640 1 -- 172.21.15.114:0/136539903 >> v1:172.21.15.114:6789/0 conn(0x7f9a000af1c0 legacy=0x7f9a0006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f99ffe00640 1 -- 172.21.15.114:0/136539903 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a00071490 con 0x7f9a0015eff0 2024-03-06T07:30:32.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.162+0000 7f9a05400640 1 --2- 172.21.15.114:0/136539903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0006e4d0 0x7f9a0006e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.163+0000 7f99ffe00640 1 --2- 172.21.15.114:0/136539903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0015eff0 0x7f9a00070d20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097d0 tx=0x7f99ec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857651411fca143b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f99ff400640 1 -- 172.21.15.114:0/136539903 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec035020 con 0x7f9a0015eff0 2024-03-06T07:30:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f99ff400640 1 -- 172.21.15.114:0/136539903 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec031810 con 0x7f9a0015eff0 2024-03-06T07:30:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f99ff400640 1 -- 172.21.15.114:0/136539903 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec031be0 con 0x7f9a0015eff0 2024-03-06T07:30:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 -- 172.21.15.114:0/136539903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0015eff0 msgr2=0x7f9a00070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 --2- 172.21.15.114:0/136539903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0015eff0 0x7f9a00070d20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097d0 tx=0x7f99ec02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 -- 172.21.15.114:0/136539903 shutdown_connections 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 --2- 172.21.15.114:0/136539903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0006e4d0 0x7f9a0006e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 --2- 172.21.15.114:0/136539903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0015eff0 0x7f9a00070d20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 -- 172.21.15.114:0/136539903 >> 172.21.15.114:0/136539903 conn(0x7f9a000b1730 msgr2=0x7f9a000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 -- 172.21.15.114:0/136539903 shutdown_connections 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.164+0000 7f9a07400640 1 -- 172.21.15.114:0/136539903 wait complete. 2024-03-06T07:30:32.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.166+0000 7f9a07400640 1 Processor -- start 2024-03-06T07:30:32.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.166+0000 7f9a07400640 1 -- start start 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a07400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0006e4d0 0x7f9a000cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a07400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a000af1c0 0x7f9a0014b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a07400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0015eff0 0x7f9a0015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a07400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a000b30b0 con 0x7f9a0006e4d0 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a07400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00071a50 con 0x7f9a000af1c0 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a07400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0006d370 con 0x7f9a0015eff0 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a04a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0006e4d0 0x7f9a000cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a04a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0006e4d0 0x7f9a000cf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39816/0 (socket says 172.21.15.114:39816) 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.167+0000 7f9a04a00640 1 -- 172.21.15.114:0/4005404151 learned_addr learned my addr 172.21.15.114:0/4005404151 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f9a04a00640 1 -- 172.21.15.114:0/4005404151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0015eff0 msgr2=0x7f9a0015cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f9a04a00640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0015eff0 0x7f9a0015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f9a04a00640 1 -- 172.21.15.114:0/4005404151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a000af1c0 msgr2=0x7f9a0014b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f9a04a00640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a000af1c0 0x7f9a0014b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f9a04a00640 1 -- 172.21.15.114:0/4005404151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a000ce0a0 con 0x7f9a0006e4d0 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f9a04a00640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0006e4d0 0x7f9a000cf140 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f99f400c400 tx=0x7f99f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f99fcc00640 1 -- 172.21.15.114:0/4005404151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f4017070 con 0x7f9a0006e4d0 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f99fcc00640 1 -- 172.21.15.114:0/4005404151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f4005ce0 con 0x7f9a0006e4d0 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f99fcc00640 1 -- 172.21.15.114:0/4005404151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f40072f0 con 0x7f9a0006e4d0 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.168+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a000ce380 con 0x7f9a0006e4d0 2024-03-06T07:30:32.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.169+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a000ce940 con 0x7f9a0006e4d0 2024-03-06T07:30:32.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.169+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a000ced70 con 0x7f9a0006e4d0 2024-03-06T07:30:32.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.171+0000 7f99fcc00640 1 -- 172.21.15.114:0/4005404151 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99f401a070 con 0x7f9a0006e4d0 2024-03-06T07:30:32.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.171+0000 7f99fcc00640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99e006b4d0 0x7f99e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.171+0000 7f99fcc00640 1 -- 172.21.15.114:0/4005404151 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99f40896e0 con 0x7f9a0006e4d0 2024-03-06T07:30:32.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.172+0000 7f99ffe00640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99e006b4d0 0x7f99e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.172+0000 7f99ffe00640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99e006b4d0 0x7f99e006d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f99ec036000 tx=0x7f99ec030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.176+0000 7f99fcc00640 1 -- 172.21.15.114:0/4005404151 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a00071d30 con 0x7f9a0006e4d0 2024-03-06T07:30:32.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.370+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a000b1f00 con 0x7f9a0006e4d0 2024-03-06T07:30:32.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.373+0000 7f99fcc00640 1 -- 172.21.15.114:0/4005404151 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16726 (secure 0 0 0) 0x7f99f40584c0 con 0x7f9a0006e4d0 2024-03-06T07:30:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":54,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:31.362457+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99e006b4d0 msgr2=0x7f99e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99e006b4d0 0x7f99e006d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f99ec036000 tx=0x7f99ec030000 comp rx=0 tx=0).stop 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0006e4d0 msgr2=0x7f9a000cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0006e4d0 0x7f9a000cf140 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f99f400c400 tx=0x7f99f400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 shutdown_connections 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a0015eff0 0x7f9a0015cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a000af1c0 0x7f9a0014b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99e006b4d0 0x7f99e006d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 --2- 172.21.15.114:0/4005404151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a0006e4d0 0x7f9a000cf140 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 >> 172.21.15.114:0/4005404151 conn(0x7f9a000b1730 msgr2=0x7f9a0015e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 shutdown_connections 2024-03-06T07:30:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.376+0000 7f9a07400640 1 -- 172.21.15.114:0/4005404151 wait complete. 2024-03-06T07:30:32.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:32.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.598+0000 7f146ca00640 1 Processor -- start 2024-03-06T07:30:32.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.598+0000 7f146ca00640 1 -- start start 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.599+0000 7f146ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14680cf370 0x7f146806d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.599+0000 7f146ca00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f14680cc2a0 0x7f14680cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.599+0000 7f146ca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14680b2b30 con 0x7f146815a940 2024-03-06T07:30:32.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.599+0000 7f146ca00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14680b28e0 con 0x7f14680cc2a0 2024-03-06T07:30:32.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.599+0000 7f146ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14680b2e10 con 0x7f14680cf370 2024-03-06T07:30:32.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.600+0000 7f1467400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f146815a940 0x7f14680b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42046/0 (socket says 172.21.15.114:42046) 2024-03-06T07:30:32.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.600+0000 7f1467400640 1 -- 172.21.15.114:0/347468537 learned_addr learned my addr 172.21.15.114:0/347468537 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.600+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383719564 0 0) 0x7f14680b2b30 con 0x7f146815a940 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.600+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1448003520 con 0x7f146815a940 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.600+0000 7f145fe00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14680cf370 0x7f146806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.600+0000 7f1467e00640 1 --2- 172.21.15.114:0/347468537 >> v2:172.21.15.177:3300/0 conn(0x7f14680cc2a0 0x7f14680cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212602557 0 0) 0x7f1448003520 con 0x7f146815a940 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14680cf370 msgr2=0x7f146806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14680cf370 0x7f146806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 >> v2:172.21.15.177:3300/0 conn(0x7f14680cc2a0 msgr2=0x7f14680cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 --2- 172.21.15.114:0/347468537 >> v2:172.21.15.177:3300/0 conn(0x7f14680cc2a0 0x7f14680cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146806ca50 con 0x7f146815a940 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f1458001670 con 0x7f146815a940 2024-03-06T07:30:32.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 >> v1:172.21.15.114:6789/0 conn(0x7f146815a940 legacy=0x7f14680b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14680cf370 0x7f14480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1448004030 0x7f1448004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1448004d40 0x7f14480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680b28e0 con 0x7f1448004030 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680b2e10 con 0x7f1448004d40 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144800bb20 con 0x7f14680cf370 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f145fe00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1448004030 0x7f1448004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f145fe00640 1 -- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14680cf370 msgr2=0x7f14480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f145fe00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14680cf370 0x7f14480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f145fe00640 1 -- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1448004d40 msgr2=0x7f14480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f145fe00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1448004d40 0x7f14480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.601+0000 7f145fe00640 1 -- 172.21.15.114:0/347468537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1448003520 con 0x7f1448004030 2024-03-06T07:30:32.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.602+0000 7f145fe00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1448004030 0x7f1448004800 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f145000ba30 tx=0x7f1448005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12dd9c8cdb5e419 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450036020 con 0x7f1448004030 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1450032680 con 0x7f1448004030 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f1466a00640 1 -- 172.21.15.114:0/347468537 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145000b600 con 0x7f1448004030 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f146ca00640 1 -- 172.21.15.114:0/347468537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1448004030 msgr2=0x7f1448004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f146ca00640 1 --2- 172.21.15.114:0/347468537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1448004030 0x7f1448004800 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f145000ba30 tx=0x7f1448005830 comp rx=0 tx=0).stop 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f1467e00640 1 -- 172.21.15.114:0/347468537 reap_dead start 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f146ca00640 1 -- 172.21.15.114:0/347468537 shutdown_connections 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f146ca00640 1 -- 172.21.15.114:0/347468537 >> 172.21.15.114:0/347468537 conn(0x7f146806afc0 msgr2=0x7f146806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f146ca00640 1 -- 172.21.15.114:0/347468537 shutdown_connections 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.603+0000 7f146ca00640 1 -- 172.21.15.114:0/347468537 wait complete. 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.605+0000 7f146ca00640 1 Processor -- start 2024-03-06T07:30:32.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.605+0000 7f146ca00640 1 -- start start 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f146ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1448004030 0x7f146814a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f146ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14680b0430 0x7f146815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f146ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1468154790 0x7f1468154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f146ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146815ad20 con 0x7f14680b0430 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f146ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14680b30f0 con 0x7f1448004030 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f146ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146815aea0 con 0x7f1468154790 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f145fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14680b0430 0x7f146815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.606+0000 7f145fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14680b0430 0x7f146815b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39848/0 (socket says 172.21.15.114:39848) 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145fe00640 1 -- 172.21.15.114:0/2832294476 learned_addr learned my addr 172.21.15.114:0/2832294476 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145fe00640 1 -- 172.21.15.114:0/2832294476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1468154790 msgr2=0x7f1468154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145fe00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1468154790 0x7f1468154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145fe00640 1 -- 172.21.15.114:0/2832294476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1448004030 msgr2=0x7f146814a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145fe00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1448004030 0x7f146814a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145fe00640 1 -- 172.21.15.114:0/2832294476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146815b560 con 0x7f14680b0430 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145fe00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14680b0430 0x7f146815b020 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f145000bb50 tx=0x7f1450004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145f400640 1 -- 172.21.15.114:0/2832294476 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450041070 con 0x7f14680b0430 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145f400640 1 -- 172.21.15.114:0/2832294476 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1450030a90 con 0x7f14680b0430 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f145f400640 1 -- 172.21.15.114:0/2832294476 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450036040 con 0x7f14680b0430 2024-03-06T07:30:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.607+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146815b800 con 0x7f14680b0430 2024-03-06T07:30:32.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.608+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146815bdc0 con 0x7f14680b0430 2024-03-06T07:30:32.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.609+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1468155150 con 0x7f14680b0430 2024-03-06T07:30:32.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.610+0000 7f145f400640 1 -- 172.21.15.114:0/2832294476 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1450039020 con 0x7f14680b0430 2024-03-06T07:30:32.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.611+0000 7f145f400640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144406b4d0 0x7f144406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.611+0000 7f145f400640 1 -- 172.21.15.114:0/2832294476 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14500ac650 con 0x7f14680b0430 2024-03-06T07:30:32.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.611+0000 7f1467400640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144406b4d0 0x7f144406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.612+0000 7f1467400640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144406b4d0 0x7f144406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f14580079b0 tx=0x7f1458001560 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.615+0000 7f145f400640 1 -- 172.21.15.114:0/2832294476 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f145007b3a0 con 0x7f14680b0430 2024-03-06T07:30:32.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.806+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f146804f330 con 0x7f14680b0430 2024-03-06T07:30:32.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.809+0000 7f145f400640 1 -- 172.21.15.114:0/2832294476 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16726 (secure 0 0 0) 0x7f145007aca0 con 0x7f14680b0430 2024-03-06T07:30:32.812 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:32.812 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":54,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:31.362457+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:32.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.811+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144406b4d0 msgr2=0x7f144406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.811+0000 7f146ca00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144406b4d0 0x7f144406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f14580079b0 tx=0x7f1458001560 comp rx=0 tx=0).stop 2024-03-06T07:30:32.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.811+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14680b0430 msgr2=0x7f146815b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.811+0000 7f146ca00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14680b0430 0x7f146815b020 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f145000bb50 tx=0x7f1450004730 comp rx=0 tx=0).stop 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.811+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 shutdown_connections 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.812+0000 7f146ca00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1468154790 0x7f1468154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.812+0000 7f146ca00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f144406b4d0 0x7f144406d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.812+0000 7f146ca00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14680b0430 0x7f146815b020 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.812+0000 7f146ca00640 1 --2- 172.21.15.114:0/2832294476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1448004030 0x7f146814a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.812+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 >> 172.21.15.114:0/2832294476 conn(0x7f146806afc0 msgr2=0x7f14680cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.812+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 shutdown_connections 2024-03-06T07:30:32.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:32.812+0000 7f146ca00640 1 -- 172.21.15.114:0/2832294476 wait complete. 2024-03-06T07:30:32.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:33.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.065+0000 7f3d13c00640 1 Processor -- start 2024-03-06T07:30:33.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.065+0000 7f3d13c00640 1 -- start start 2024-03-06T07:30:33.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.067+0000 7f3d13c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 0x7f3d0c06dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.067+0000 7f3d13c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c071ea0 con 0x7f3d0c15efa0 2024-03-06T07:30:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.067+0000 7f3d13c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c06e380 con 0x7f3d0c06afc0 2024-03-06T07:30:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.067+0000 7f3d13c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c15bf70 con 0x7f3d0c15bb90 2024-03-06T07:30:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.067+0000 7f3d0be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3d0c15efa0 0x7f3d0c0cec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42048/0 (socket says 172.21.15.114:42048) 2024-03-06T07:30:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.067+0000 7f3d0be00640 1 -- 172.21.15.114:0/2722809149 learned_addr learned my addr 172.21.15.114:0/2722809149 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:33.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.067+0000 7f3d11200640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 0x7f3d0c06dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120321941 0 0) 0x7f3d0c071ea0 con 0x7f3d0c15efa0 2024-03-06T07:30:33.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf4003520 con 0x7f3d0c15efa0 2024-03-06T07:30:33.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 501905341 0 0) 0x7f3cf4003520 con 0x7f3d0c15efa0 2024-03-06T07:30:33.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 msgr2=0x7f3d0c06dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 0x7f3d0c06dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 >> v1:172.21.15.177:6789/0 conn(0x7f3d0c06afc0 legacy=0x7f3d0c0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c06e660 con 0x7f3d0c15efa0 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f3cf8001670 con 0x7f3d0c15efa0 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 >> v1:172.21.15.114:6789/0 conn(0x7f3d0c15efa0 legacy=0x7f3d0c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf40039e0 0x7f3cf4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 0x7f3cf4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf4004d40 0x7f3cf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c15bf70 con 0x7f3cf4004d40 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf400ba70 con 0x7f3cf40039e0 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf400bc70 con 0x7f3d0c15bb90 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.068+0000 7f3d11c00640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf4004d40 0x7f3cf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d11200640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf40039e0 0x7f3cf4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d0be00640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 0x7f3cf4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d11c00640 1 -- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 msgr2=0x7f3cf4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d11c00640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 0x7f3cf4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d11c00640 1 -- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf40039e0 msgr2=0x7f3cf4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d11c00640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf40039e0 0x7f3cf4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d11c00640 1 -- 172.21.15.114:0/2722809149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf4003520 con 0x7f3cf4004d40 2024-03-06T07:30:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d0be00640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0c15bb90 0x7f3cf4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.069+0000 7f3d11200640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf40039e0 0x7f3cf4003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.070+0000 7f3d11c00640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf4004d40 0x7f3cf40051c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3d04001e20 tx=0x7f3cf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2933e62d7802a84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.070+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04001a70 con 0x7f3cf4004d40 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.070+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d04003b70 con 0x7f3cf4004d40 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.070+0000 7f3d0b400640 1 -- 172.21.15.114:0/2722809149 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d040050d0 con 0x7f3cf4004d40 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.071+0000 7f3d13c00640 1 -- 172.21.15.114:0/2722809149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf4004d40 msgr2=0x7f3cf40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.071+0000 7f3d13c00640 1 --2- 172.21.15.114:0/2722809149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf4004d40 0x7f3cf40051c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3d04001e20 tx=0x7f3cf4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.071+0000 7f3d11c00640 1 -- 172.21.15.114:0/2722809149 reap_dead start 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.071+0000 7f3d13c00640 1 -- 172.21.15.114:0/2722809149 shutdown_connections 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.071+0000 7f3d13c00640 1 -- 172.21.15.114:0/2722809149 >> 172.21.15.114:0/2722809149 conn(0x7f3d0c0b27d0 msgr2=0x7f3d0c0b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.071+0000 7f3d13c00640 1 -- 172.21.15.114:0/2722809149 shutdown_connections 2024-03-06T07:30:33.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.071+0000 7f3d13c00640 1 -- 172.21.15.114:0/2722809149 wait complete. 2024-03-06T07:30:33.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.073+0000 7f3d13c00640 1 Processor -- start 2024-03-06T07:30:33.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.073+0000 7f3d13c00640 1 -- start start 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d13c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf4004d40 0x7f3d0c1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d13c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c155d90 0x7f3d0c156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d13c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c0cc1f0 0x7f3d0c0cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d13c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c15f400 con 0x7f3d0c155d90 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d13c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c06ec20 con 0x7f3d0c0cc1f0 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d13c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c06dcf0 con 0x7f3cf4004d40 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c155d90 0x7f3d0c156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c155d90 0x7f3d0c156210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39872/0 (socket says 172.21.15.114:39872) 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 -- 172.21.15.114:0/3709251286 learned_addr learned my addr 172.21.15.114:0/3709251286 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d11200640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf4004d40 0x7f3d0c1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 -- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf4004d40 msgr2=0x7f3d0c1543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf4004d40 0x7f3d0c1543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 -- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c0cc1f0 msgr2=0x7f3d0c0cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d11c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c0cc1f0 0x7f3d0c0cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c0cc1f0 0x7f3d0c0cc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d0be00640 1 -- 172.21.15.114:0/3709251286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0ca950 con 0x7f3d0c155d90 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.074+0000 7f3d11200640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf4004d40 0x7f3d0c1543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.075+0000 7f3d0be00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c155d90 0x7f3d0c156210 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8001200 tx=0x7f3cf800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.075+0000 7f3d08c00640 1 -- 172.21.15.114:0/3709251286 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8019070 con 0x7f3d0c155d90 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.075+0000 7f3d08c00640 1 -- 172.21.15.114:0/3709251286 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf80074e0 con 0x7f3d0c155d90 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.075+0000 7f3d08c00640 1 -- 172.21.15.114:0/3709251286 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80049e0 con 0x7f3d0c155d90 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.075+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0c0cac30 con 0x7f3d0c155d90 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.075+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0c0cb1f0 con 0x7f3d0c155d90 2024-03-06T07:30:33.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.076+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0c0690e0 con 0x7f3d0c155d90 2024-03-06T07:30:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.077+0000 7f3d08c00640 1 -- 172.21.15.114:0/3709251286 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cf801c070 con 0x7f3d0c155d90 2024-03-06T07:30:33.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.078+0000 7f3d08c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3cec06b500 0x7f3cec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.079+0000 7f3d08c00640 1 -- 172.21.15.114:0/3709251286 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cf8089d20 con 0x7f3d0c155d90 2024-03-06T07:30:33.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.079+0000 7f3d11200640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3cec06b500 0x7f3cec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.079+0000 7f3d11200640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3cec06b500 0x7f3cec06d9c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009a80 tx=0x7f3d00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.083+0000 7f3d08c00640 1 -- 172.21.15.114:0/3709251286 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d0c0690e0 con 0x7f3d0c155d90 2024-03-06T07:30:33.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.321+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3d0c04f330 con 0x7f3d0c155d90 2024-03-06T07:30:33.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.960+0000 7f3d08c00640 1 -- 172.21.15.114:0/3709251286 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3cf8058c50 con 0x7f3d0c155d90 2024-03-06T07:30:33.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.964+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3cec06b500 msgr2=0x7f3cec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.964+0000 7f3d13c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3cec06b500 0x7f3cec06d9c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009a80 tx=0x7f3d00008040 comp rx=0 tx=0).stop 2024-03-06T07:30:33.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.964+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c155d90 msgr2=0x7f3d0c156210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.964+0000 7f3d13c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c155d90 0x7f3d0c156210 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8001200 tx=0x7f3cf800dd40 comp rx=0 tx=0).stop 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 shutdown_connections 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c0cc1f0 0x7f3d0c0cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3cec06b500 0x7f3cec06d9c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c155d90 0x7f3d0c156210 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 --2- 172.21.15.114:0/3709251286 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3cf4004d40 0x7f3d0c1543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 >> 172.21.15.114:0/3709251286 conn(0x7f3d0c0b27d0 msgr2=0x7f3d0c0cef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 shutdown_connections 2024-03-06T07:30:33.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:33.965+0000 7f3d13c00640 1 -- 172.21.15.114:0/3709251286 wait complete. 2024-03-06T07:30:33.994 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:33.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:34.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.188+0000 7f6c02c00640 1 Processor -- start 2024-03-06T07:30:34.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.188+0000 7f6c02c00640 1 -- start start 2024-03-06T07:30:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.190+0000 7f6c02c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6bfc160700 0x7f6bfc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.190+0000 7f6c02c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bfc0b46c0 con 0x7f6bfc0b3f70 2024-03-06T07:30:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.190+0000 7f6c02c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bfc0b49a0 con 0x7f6bfc15fc10 2024-03-06T07:30:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.190+0000 7f6c02c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bfc072c90 con 0x7f6bfc160700 2024-03-06T07:30:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.190+0000 7f6bf3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6bfc0b3f70 0x7f6bfc0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42064/0 (socket says 172.21.15.114:42064) 2024-03-06T07:30:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.190+0000 7f6bf3400640 1 -- 172.21.15.114:0/3744877100 learned_addr learned my addr 172.21.15.114:0/3744877100 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6c00c00640 1 --2- 172.21.15.114:0/3744877100 >> v2:172.21.15.177:3300/0 conn(0x7f6bfc160700 0x7f6bfc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321664422 0 0) 0x7f6bfc0b46c0 con 0x7f6bfc0b3f70 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be0003520 con 0x7f6bfc0b3f70 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1664118271 0 0) 0x7f6be0003520 con 0x7f6bfc0b3f70 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 >> v2:172.21.15.177:3300/0 conn(0x7f6bfc160700 msgr2=0x7f6bfc070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 --2- 172.21.15.114:0/3744877100 >> v2:172.21.15.177:3300/0 conn(0x7f6bfc160700 0x7f6bfc070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 >> v1:172.21.15.177:6789/0 conn(0x7f6bfc15fc10 legacy=0x7f6bfc15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc09dd80 con 0x7f6bfc0b3f70 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6be4001670 con 0x7f6bfc0b3f70 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 >> v1:172.21.15.114:6789/0 conn(0x7f6bfc0b3f70 legacy=0x7f6bfc0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be00039b0 0x7f6be0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6be0004800 0x7f6be0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00051c0 0x7f6be000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc072c90 con 0x7f6be00039b0 2024-03-06T07:30:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be000c060 con 0x7f6be00051c0 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.191+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be000c260 con 0x7f6be0004800 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be00039b0 0x7f6be0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6bf3400640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6be0004800 0x7f6be0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6c00c00640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00051c0 0x7f6be000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6bfbe00640 1 -- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6be0004800 msgr2=0x7f6be0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6be0004800 0x7f6be0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6bfbe00640 1 -- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00051c0 msgr2=0x7f6be000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00051c0 0x7f6be000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6bfbe00640 1 -- 172.21.15.114:0/3744877100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be0003520 con 0x7f6be00039b0 2024-03-06T07:30:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.192+0000 7f6c00c00640 1 -- 172.21.15.114:0/3744877100 reap_dead start 2024-03-06T07:30:34.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.194+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be00039b0 0x7f6be0003dd0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6bec001750 tx=0x7f6be0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb50693e4b4709 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.194+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec001370 con 0x7f6be00039b0 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.194+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bec0052d0 con 0x7f6be00039b0 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.194+0000 7f6bfb400640 1 -- 172.21.15.114:0/3744877100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec003830 con 0x7f6be00039b0 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.195+0000 7f6c02c00640 1 -- 172.21.15.114:0/3744877100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be00039b0 msgr2=0x7f6be0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.195+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be00039b0 0x7f6be0003dd0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6bec001750 tx=0x7f6be0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.195+0000 7f6c02c00640 1 -- 172.21.15.114:0/3744877100 shutdown_connections 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.195+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3744877100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be00039b0 0x7f6be0003dd0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.195+0000 7f6c02c00640 1 -- 172.21.15.114:0/3744877100 >> 172.21.15.114:0/3744877100 conn(0x7f6bfc0b1940 msgr2=0x7f6bfc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.195+0000 7f6c02c00640 1 -- 172.21.15.114:0/3744877100 shutdown_connections 2024-03-06T07:30:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.195+0000 7f6c02c00640 1 -- 172.21.15.114:0/3744877100 wait complete. 2024-03-06T07:30:34.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.197+0000 7f6c02c00640 1 Processor -- start 2024-03-06T07:30:34.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.197+0000 7f6c02c00640 1 -- start start 2024-03-06T07:30:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c02c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00039b0 0x7f6bfc147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c02c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bfc143e30 0x7f6bfc15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c02c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bfc158350 0x7f6bfc1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c02c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc15c460 con 0x7f6bfc158350 2024-03-06T07:30:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c02c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc15c2e0 con 0x7f6be00039b0 2024-03-06T07:30:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c02c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc070ff0 con 0x7f6bfc143e30 2024-03-06T07:30:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bfc158350 0x7f6bfc1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bfc158350 0x7f6bfc1587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39894/0 (socket says 172.21.15.114:39894) 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 -- 172.21.15.114:0/3659055258 learned_addr learned my addr 172.21.15.114:0/3659055258 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00039b0 0x7f6bfc147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 -- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bfc143e30 msgr2=0x7f6bfc15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bfc143e30 0x7f6bfc15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 -- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00039b0 msgr2=0x7f6bfc147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00039b0 0x7f6bfc147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6c00c00640 1 -- 172.21.15.114:0/3659055258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc15b3f0 con 0x7f6bfc158350 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.198+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00039b0 0x7f6bfc147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.199+0000 7f6c00c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bfc158350 0x7f6bfc1587d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc070bc0 tx=0x7f6bf40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.199+0000 7f6bf8c00640 1 -- 172.21.15.114:0/3659055258 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4011e40 con 0x7f6bfc158350 2024-03-06T07:30:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.199+0000 7f6bf8c00640 1 -- 172.21.15.114:0/3659055258 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf401b070 con 0x7f6bfc158350 2024-03-06T07:30:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.199+0000 7f6bf8c00640 1 -- 172.21.15.114:0/3659055258 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4003990 con 0x7f6bfc158350 2024-03-06T07:30:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.199+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bfc15b670 con 0x7f6bfc158350 2024-03-06T07:30:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.199+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bfc1e65d0 con 0x7f6bfc158350 2024-03-06T07:30:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.201+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bfc1e63c0 con 0x7f6bfc158350 2024-03-06T07:30:34.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.201+0000 7f6bf8c00640 1 -- 172.21.15.114:0/3659055258 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bf4005d10 con 0x7f6bfc158350 2024-03-06T07:30:34.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.202+0000 7f6bf8c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6bd806b500 0x7f6bd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.202+0000 7f6bf8c00640 1 -- 172.21.15.114:0/3659055258 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bf408a570 con 0x7f6bfc158350 2024-03-06T07:30:34.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.202+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6bd806b500 0x7f6bd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.203+0000 7f6bfbe00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6bd806b500 0x7f6bd806d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6bec033000 tx=0x7f6bec009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.207+0000 7f6bf8c00640 1 -- 172.21.15.114:0/3659055258 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bfc15b3f0 con 0x7f6bfc158350 2024-03-06T07:30:34.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.442+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6bfc04f330 con 0x7f6bfc158350 2024-03-06T07:30:34.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.964+0000 7f6bf8c00640 1 -- 172.21.15.114:0/3659055258 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6bf4088e20 con 0x7f6bfc158350 2024-03-06T07:30:34.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6bd806b500 msgr2=0x7f6bd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6bd806b500 0x7f6bd806d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6bec033000 tx=0x7f6bec009030 comp rx=0 tx=0).stop 2024-03-06T07:30:34.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bfc158350 msgr2=0x7f6bfc1587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bfc158350 0x7f6bfc1587d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc070bc0 tx=0x7f6bf40115a0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 shutdown_connections 2024-03-06T07:30:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6bd806b500 0x7f6bd806d9c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bfc158350 0x7f6bfc1587d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6bfc143e30 0x7f6bfc15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 --2- 172.21.15.114:0/3659055258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6be00039b0 0x7f6bfc147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 >> 172.21.15.114:0/3659055258 conn(0x7f6bfc0b1940 msgr2=0x7f6bfc0cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.967+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 shutdown_connections 2024-03-06T07:30:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:34.968+0000 7f6c02c00640 1 -- 172.21.15.114:0/3659055258 wait complete. 2024-03-06T07:30:34.989 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:34.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.242+0000 7f0c0b200640 1 Processor -- start 2024-03-06T07:30:35.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.242+0000 7f0c0b200640 1 -- start start 2024-03-06T07:30:35.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.243+0000 7f0c0b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c0b40c0 0x7f0c0c0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.243+0000 7f0c0b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c0c1620c0 0x7f0c0c0b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.243+0000 7f0c0b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c161900 con 0x7f0c0c0ae670 2024-03-06T07:30:35.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.243+0000 7f0c0b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c0b49e0 con 0x7f0c0c1620c0 2024-03-06T07:30:35.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.243+0000 7f0c0b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c15ebd0 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c0a800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c0c1620c0 0x7f0c0c0b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c01400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0c0c0ae670 0x7f0c0c15e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42074/0 (socket says 172.21.15.114:42074) 2024-03-06T07:30:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c01400640 1 -- 172.21.15.114:0/1890818651 learned_addr learned my addr 172.21.15.114:0/1890818651 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c09e00640 1 --2- 172.21.15.114:0/1890818651 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c0b40c0 0x7f0c0c0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c09400640 1 -- 172.21.15.114:0/1890818651 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113049180 0 0) 0x7f0c0c161900 con 0x7f0c0c0ae670 2024-03-06T07:30:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c0a800640 1 -- 172.21.15.114:0/1890818651 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c0b40c0 msgr2=0x7f0c0c0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c0a800640 1 --2- 172.21.15.114:0/1890818651 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c0b40c0 0x7f0c0c0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c0a800640 1 -- 172.21.15.114:0/1890818651 >> v1:172.21.15.114:6789/0 conn(0x7f0c0c0ae670 legacy=0x7f0c0c15e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.244+0000 7f0c0a800640 1 -- 172.21.15.114:0/1890818651 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c15eeb0 con 0x7f0c0c1620c0 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.246+0000 7f0c0a800640 1 --2- 172.21.15.114:0/1890818651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c1620c0 0x7f0c0c0b28e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009d90 tx=0x7f0c0402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebfb1a11b507562 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.246+0000 7f0c09400640 1 -- 172.21.15.114:0/1890818651 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04039070 con 0x7f0c0c1620c0 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.246+0000 7f0c09400640 1 -- 172.21.15.114:0/1890818651 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0402cc50 con 0x7f0c0c1620c0 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.246+0000 7f0c09400640 1 -- 172.21.15.114:0/1890818651 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04034440 con 0x7f0c0c1620c0 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 -- 172.21.15.114:0/1890818651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c1620c0 msgr2=0x7f0c0c0b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 --2- 172.21.15.114:0/1890818651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c1620c0 0x7f0c0c0b28e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009d90 tx=0x7f0c0402bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 -- 172.21.15.114:0/1890818651 shutdown_connections 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 --2- 172.21.15.114:0/1890818651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c1620c0 0x7f0c0c0b28e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 --2- 172.21.15.114:0/1890818651 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c0b40c0 0x7f0c0c0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 -- 172.21.15.114:0/1890818651 >> 172.21.15.114:0/1890818651 conn(0x7f0c0c06afc0 msgr2=0x7f0c0c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 -- 172.21.15.114:0/1890818651 shutdown_connections 2024-03-06T07:30:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.247+0000 7f0c0b200640 1 -- 172.21.15.114:0/1890818651 wait complete. 2024-03-06T07:30:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.248+0000 7f0c0b200640 1 Processor -- start 2024-03-06T07:30:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.249+0000 7f0c0b200640 1 -- start start 2024-03-06T07:30:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c0b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c0ae670 0x7f0c0c0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c0b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c0b40c0 0x7f0c0c1535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c0b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c1620c0 0x7f0c0c15f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c0b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c158660 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c0b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c161ba0 con 0x7f0c0c0ae670 2024-03-06T07:30:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c0b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c1587e0 con 0x7f0c0c1620c0 2024-03-06T07:30:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c0b40c0 0x7f0c0c1535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c0b40c0 0x7f0c0c1535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39916/0 (socket says 172.21.15.114:39916) 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 -- 172.21.15.114:0/3676483057 learned_addr learned my addr 172.21.15.114:0/3676483057 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c09e00640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c0ae670 0x7f0c0c0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c0a800640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c1620c0 0x7f0c0c15f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 -- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c1620c0 msgr2=0x7f0c0c15f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c1620c0 0x7f0c0c15f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 -- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c0ae670 msgr2=0x7f0c0c0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c0ae670 0x7f0c0c0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.250+0000 7f0c01400640 1 -- 172.21.15.114:0/3676483057 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c155c50 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.251+0000 7f0c01400640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c0b40c0 0x7f0c0c1535a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c460 tx=0x7f0bf400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.251+0000 7f0c02a00640 1 -- 172.21.15.114:0/3676483057 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4018070 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.251+0000 7f0c02a00640 1 -- 172.21.15.114:0/3676483057 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf4007500 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.251+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c155f30 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.251+0000 7f0c02a00640 1 -- 172.21.15.114:0/3676483057 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4004a20 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.251+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c1564f0 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.252+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c04f330 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.253+0000 7f0c02a00640 1 -- 172.21.15.114:0/3676483057 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bf401b070 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.254+0000 7f0c02a00640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0be806b4d0 0x7f0be806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.254+0000 7f0c02a00640 1 -- 172.21.15.114:0/3676483057 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bf4089600 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.254+0000 7f0c09e00640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0be806b4d0 0x7f0be806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.255+0000 7f0c09e00640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0be806b4d0 0x7f0be806d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc006fd0 tx=0x7f0bfc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.259+0000 7f0c02a00640 1 -- 172.21.15.114:0/3676483057 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bf4058350 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.501+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c0c15ddb0 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.502+0000 7f0c02a00640 1 -- 172.21.15.114:0/3676483057 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f0bf4057c50 con 0x7f0c0c0b40c0 2024-03-06T07:30:35.505 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:35.505 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:30:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0be806b4d0 msgr2=0x7f0be806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0be806b4d0 0x7f0be806d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc006fd0 tx=0x7f0bfc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c0b40c0 msgr2=0x7f0c0c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c0b40c0 0x7f0c0c1535a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c460 tx=0x7f0bf400c920 comp rx=0 tx=0).stop 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 shutdown_connections 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0c0c1620c0 0x7f0c0c15f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0be806b4d0 0x7f0be806d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c0c0b40c0 0x7f0c0c1535a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 --2- 172.21.15.114:0/3676483057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c0ae670 0x7f0c0c0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 >> 172.21.15.114:0/3676483057 conn(0x7f0c0c06afc0 msgr2=0x7f0c0c15e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 shutdown_connections 2024-03-06T07:30:35.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.505+0000 7f0c0b200640 1 -- 172.21.15.114:0/3676483057 wait complete. 2024-03-06T07:30:35.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:35.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.712+0000 7f4863000640 1 Processor -- start 2024-03-06T07:30:35.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.712+0000 7f4863000640 1 -- start start 2024-03-06T07:30:35.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f4863000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0cff20 0x7f485c0b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f4863000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f485c0b73e0 0x7f485c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f4863000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c0b5e00 con 0x7f485c0b7860 2024-03-06T07:30:35.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f4863000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c0b6a10 con 0x7f485c0b73e0 2024-03-06T07:30:35.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f4863000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0b60e0 con 0x7f485c0cff20 2024-03-06T07:30:35.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f485be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0cff20 0x7f485c0b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f485be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0cff20 0x7f485c0b6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:39614/0 (socket says 172.21.15.114:39614) 2024-03-06T07:30:35.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.714+0000 7f485be00640 1 -- 172.21.15.114:0/3337276768 learned_addr learned my addr 172.21.15.114:0/3337276768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:35.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.715+0000 7f4853400640 1 --2- 172.21.15.114:0/3337276768 >> v2:172.21.15.177:3300/0 conn(0x7f485c0b73e0 0x7f485c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.715+0000 7f485be00640 1 -- 172.21.15.114:0/3337276768 >> v2:172.21.15.177:3300/0 conn(0x7f485c0b73e0 msgr2=0x7f485c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.715+0000 7f485be00640 1 --2- 172.21.15.114:0/3337276768 >> v2:172.21.15.177:3300/0 conn(0x7f485c0b73e0 0x7f485c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.715+0000 7f485be00640 1 -- 172.21.15.114:0/3337276768 >> v1:172.21.15.177:6789/0 conn(0x7f485c0b7860 legacy=0x7f485c0d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.715+0000 7f485be00640 1 -- 172.21.15.114:0/3337276768 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c0b63c0 con 0x7f485c0cff20 2024-03-06T07:30:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.717+0000 7f485be00640 1 --2- 172.21.15.114:0/3337276768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0cff20 0x7f485c0b6ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f484c0098a0 tx=0x7f484c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f56c3360cf954aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.717+0000 7f485b400640 1 -- 172.21.15.114:0/3337276768 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c034070 con 0x7f485c0cff20 2024-03-06T07:30:35.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.717+0000 7f485b400640 1 -- 172.21.15.114:0/3337276768 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c030840 con 0x7f485c0cff20 2024-03-06T07:30:35.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.717+0000 7f485b400640 1 -- 172.21.15.114:0/3337276768 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c030c10 con 0x7f485c0cff20 2024-03-06T07:30:35.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 -- 172.21.15.114:0/3337276768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0cff20 msgr2=0x7f485c0b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 --2- 172.21.15.114:0/3337276768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0cff20 0x7f485c0b6ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f484c0098a0 tx=0x7f484c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 -- 172.21.15.114:0/3337276768 shutdown_connections 2024-03-06T07:30:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 --2- 172.21.15.114:0/3337276768 >> v2:172.21.15.177:3300/0 conn(0x7f485c0b73e0 0x7f485c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 --2- 172.21.15.114:0/3337276768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0cff20 0x7f485c0b6ea0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 -- 172.21.15.114:0/3337276768 >> 172.21.15.114:0/3337276768 conn(0x7f485c0d1c90 msgr2=0x7f485c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 -- 172.21.15.114:0/3337276768 shutdown_connections 2024-03-06T07:30:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.718+0000 7f4863000640 1 -- 172.21.15.114:0/3337276768 wait complete. 2024-03-06T07:30:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.719+0000 7f4863000640 1 Processor -- start 2024-03-06T07:30:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.719+0000 7f4863000640 1 -- start start 2024-03-06T07:30:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.720+0000 7f4863000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0b73e0 0x7f485c1f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.720+0000 7f4863000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0b7860 0x7f485c15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.720+0000 7f4863000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f485c0cff20 0x7f485c09dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.720+0000 7f4863000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c161a50 con 0x7f485c0b7860 2024-03-06T07:30:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.720+0000 7f4863000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0b5af0 con 0x7f485c0cff20 2024-03-06T07:30:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.720+0000 7f4863000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c0d0f50 con 0x7f485c0b73e0 2024-03-06T07:30:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.720+0000 7f4853400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0b7860 0x7f485c15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0b7860 0x7f485c15c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39924/0 (socket says 172.21.15.114:39924) 2024-03-06T07:30:35.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 -- 172.21.15.114:0/583735864 learned_addr learned my addr 172.21.15.114:0/583735864 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 -- 172.21.15.114:0/583735864 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0b73e0 msgr2=0x7f485c1f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0b73e0 0x7f485c1f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 -- 172.21.15.114:0/583735864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f485c0cff20 msgr2=0x7f485c09dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f485c0cff20 0x7f485c09dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 -- 172.21.15.114:0/583735864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c1f3a10 con 0x7f485c0b7860 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4853400640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0b7860 0x7f485c15c590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f484400d800 tx=0x7f484400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.721+0000 7f4858c00640 1 -- 172.21.15.114:0/583735864 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484400b6c0 con 0x7f485c0b7860 2024-03-06T07:30:35.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.722+0000 7f4858c00640 1 -- 172.21.15.114:0/583735864 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4844019070 con 0x7f485c0b7860 2024-03-06T07:30:35.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.722+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c09bb90 con 0x7f485c0b7860 2024-03-06T07:30:35.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.722+0000 7f4858c00640 1 -- 172.21.15.114:0/583735864 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484400ba80 con 0x7f485c0b7860 2024-03-06T07:30:35.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.722+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c15d0f0 con 0x7f485c0b7860 2024-03-06T07:30:35.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.723+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c04f330 con 0x7f485c0b7860 2024-03-06T07:30:35.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.724+0000 7f4858c00640 1 -- 172.21.15.114:0/583735864 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f484400bc20 con 0x7f485c0b7860 2024-03-06T07:30:35.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.724+0000 7f4858c00640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f483806b500 0x7f483806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.725+0000 7f4858c00640 1 -- 172.21.15.114:0/583735864 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f484408a670 con 0x7f485c0b7860 2024-03-06T07:30:35.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.725+0000 7f485be00640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f483806b500 0x7f483806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.725+0000 7f485be00640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f483806b500 0x7f483806d9c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f484c036000 tx=0x7f484c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.729+0000 7f4858c00640 1 -- 172.21.15.114:0/583735864 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f484401c360 con 0x7f485c0b7860 2024-03-06T07:30:35.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.917+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f485c0684a0 con 0x7f485c0b7860 2024-03-06T07:30:35.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.919+0000 7f4858c00640 1 -- 172.21.15.114:0/583735864 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16713 (secure 0 0 0) 0x7f48440594e0 con 0x7f485c0b7860 2024-03-06T07:30:35.922 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:35.922 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:32.939149+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f483806b500 msgr2=0x7f483806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f483806b500 0x7f483806d9c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f484c036000 tx=0x7f484c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0b7860 msgr2=0x7f485c15c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0b7860 0x7f485c15c590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f484400d800 tx=0x7f484400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 shutdown_connections 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f485c0cff20 0x7f485c09dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f483806b500 0x7f483806d9c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f485c0b7860 0x7f485c15c590 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 --2- 172.21.15.114:0/583735864 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f485c0b73e0 0x7f485c1f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 >> 172.21.15.114:0/583735864 conn(0x7f485c0d1c90 msgr2=0x7f485c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 shutdown_connections 2024-03-06T07:30:35.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:35.923+0000 7f4863000640 1 -- 172.21.15.114:0/583735864 wait complete. 2024-03-06T07:30:35.950 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:35.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.136+0000 7f2ab3000640 1 Processor -- start 2024-03-06T07:30:36.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.136+0000 7f2ab3000640 1 -- start start 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.138+0000 7f2ab3000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2aac0b75b0 0x7f2aac0b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.138+0000 7f2ab3000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2aac0b5020 0x7f2aac0b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.138+0000 7f2ab3000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0b5e00 con 0x7f2aac0cff20 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.138+0000 7f2ab3000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac0b6a10 con 0x7f2aac0b75b0 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.138+0000 7f2ab3000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac0b60e0 con 0x7f2aac0b5020 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2aab400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2aac0b75b0 0x7f2aac0b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2aab400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2aac0b75b0 0x7f2aac0b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39932/0 (socket says 172.21.15.114:39932) 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2aab400640 1 -- 172.21.15.114:0/262184038 learned_addr learned my addr 172.21.15.114:0/262184038 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2ab1000640 1 --2- 172.21.15.114:0/262184038 >> v2:172.21.15.177:3300/0 conn(0x7f2aac0b5020 0x7f2aac0b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2aab400640 1 -- 172.21.15.114:0/262184038 >> v2:172.21.15.177:3300/0 conn(0x7f2aac0b5020 msgr2=0x7f2aac0b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2aab400640 1 --2- 172.21.15.114:0/262184038 >> v2:172.21.15.177:3300/0 conn(0x7f2aac0b5020 0x7f2aac0b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2aab400640 1 -- 172.21.15.114:0/262184038 >> v1:172.21.15.177:6789/0 conn(0x7f2aac0cff20 legacy=0x7f2aac0b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.139+0000 7f2aab400640 1 -- 172.21.15.114:0/262184038 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0b63c0 con 0x7f2aac0b75b0 2024-03-06T07:30:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.140+0000 7f2aab400640 1 --2- 172.21.15.114:0/262184038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b75b0 0x7f2aac0b4ae0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0c8cf0 tx=0x7f2a9802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f8425f43bf354 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.141+0000 7f2aaaa00640 1 -- 172.21.15.114:0/262184038 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9802e070 con 0x7f2aac0b75b0 2024-03-06T07:30:36.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.141+0000 7f2aaaa00640 1 -- 172.21.15.114:0/262184038 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98037070 con 0x7f2aac0b75b0 2024-03-06T07:30:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.141+0000 7f2aaaa00640 1 -- 172.21.15.114:0/262184038 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98031a00 con 0x7f2aac0b75b0 2024-03-06T07:30:36.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 -- 172.21.15.114:0/262184038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b75b0 msgr2=0x7f2aac0b4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 --2- 172.21.15.114:0/262184038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b75b0 0x7f2aac0b4ae0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0c8cf0 tx=0x7f2a9802b880 comp rx=0 tx=0).stop 2024-03-06T07:30:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 -- 172.21.15.114:0/262184038 shutdown_connections 2024-03-06T07:30:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 --2- 172.21.15.114:0/262184038 >> v2:172.21.15.177:3300/0 conn(0x7f2aac0b5020 0x7f2aac0b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 --2- 172.21.15.114:0/262184038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b75b0 0x7f2aac0b4ae0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 -- 172.21.15.114:0/262184038 >> 172.21.15.114:0/262184038 conn(0x7f2aac0d1c90 msgr2=0x7f2aac0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 -- 172.21.15.114:0/262184038 shutdown_connections 2024-03-06T07:30:36.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.142+0000 7f2ab3000640 1 -- 172.21.15.114:0/262184038 wait complete. 2024-03-06T07:30:36.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.145+0000 7f2ab3000640 1 Processor -- start 2024-03-06T07:30:36.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.145+0000 7f2ab3000640 1 -- start start 2024-03-06T07:30:36.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2ab3000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b5020 0x7f2aac146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2ab3000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aac0b75b0 0x7f2aac0c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2ab3000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aac0cff20 0x7f2aac0c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2ab3000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0b7a30 con 0x7f2aac0b5020 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2ab3000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac161a50 con 0x7f2aac0cff20 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2ab3000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0b7bb0 con 0x7f2aac0b75b0 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2aabe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b5020 0x7f2aac146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2aabe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b5020 0x7f2aac146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39938/0 (socket says 172.21.15.114:39938) 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.147+0000 7f2aabe00640 1 -- 172.21.15.114:0/3617469816 learned_addr learned my addr 172.21.15.114:0/3617469816 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aabe00640 1 -- 172.21.15.114:0/3617469816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aac0b75b0 msgr2=0x7f2aac0c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aabe00640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aac0b75b0 0x7f2aac0c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aabe00640 1 -- 172.21.15.114:0/3617469816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aac0cff20 msgr2=0x7f2aac0c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aabe00640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aac0cff20 0x7f2aac0c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aabe00640 1 -- 172.21.15.114:0/3617469816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0c8700 con 0x7f2aac0b5020 2024-03-06T07:30:36.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aabe00640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b5020 0x7f2aac146a00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00b420 tx=0x7f2a9c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aa3e00640 1 -- 172.21.15.114:0/3617469816 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c011830 con 0x7f2aac0b5020 2024-03-06T07:30:36.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aa3e00640 1 -- 172.21.15.114:0/3617469816 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c016070 con 0x7f2aac0b5020 2024-03-06T07:30:36.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.148+0000 7f2aa3e00640 1 -- 172.21.15.114:0/3617469816 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c011c10 con 0x7f2aac0b5020 2024-03-06T07:30:36.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.149+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac0c7660 con 0x7f2aac0b5020 2024-03-06T07:30:36.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.149+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac0c7c20 con 0x7f2aac0b5020 2024-03-06T07:30:36.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.150+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac04f330 con 0x7f2aac0b5020 2024-03-06T07:30:36.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.151+0000 7f2aa3e00640 1 -- 172.21.15.114:0/3617469816 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a9c01b460 con 0x7f2aac0b5020 2024-03-06T07:30:36.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.152+0000 7f2aa3e00640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.152+0000 7f2aa3e00640 1 -- 172.21.15.114:0/3617469816 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a9c089c70 con 0x7f2aac0b5020 2024-03-06T07:30:36.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.152+0000 7f2aab400640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.153+0000 7f2aab400640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2a98036000 tx=0x7f2a98030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.161+0000 7f2aa3e00640 1 -- 172.21.15.114:0/3617469816 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a9c020020 con 0x7f2aac0b5020 2024-03-06T07:30:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.355+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2aac06c8c0 con 0x7f2a8c06b4d0 2024-03-06T07:30:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.365+0000 7f2aa3e00640 1 -- 172.21.15.114:0/3617469816 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251912 (secure 0 0 0) 0x7f2aac06c8c0 con 0x7f2a8c06b4d0 2024-03-06T07:30:36.371 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:30:36.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.371+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8c06b4d0 msgr2=0x7f2a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.371+0000 7f2ab3000640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2a98036000 tx=0x7f2a98030000 comp rx=0 tx=0).stop 2024-03-06T07:30:36.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.371+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b5020 msgr2=0x7f2aac146a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.371+0000 7f2ab3000640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b5020 0x7f2aac146a00 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00b420 tx=0x7f2a9c00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 shutdown_connections 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aac0cff20 0x7f2aac0c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aac0b75b0 0x7f2aac0c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8c06b4d0 0x7f2a8c06d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 --2- 172.21.15.114:0/3617469816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aac0b5020 0x7f2aac146a00 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 >> 172.21.15.114:0/3617469816 conn(0x7f2aac0d1c90 msgr2=0x7f2aac06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 shutdown_connections 2024-03-06T07:30:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.372+0000 7f2ab3000640 1 -- 172.21.15.114:0/3617469816 wait complete. 2024-03-06T07:30:36.394 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-03-06T07:30:35.146795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255720,"kb_used_data":6280,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719000,"statfs":{"total":773094113280,"available":772832256000,"internally_reserved":0,"allocated":6430720,"data_stored":3320452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.259613"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.967521+0000","last_change":"2024-03-06T07:30:31.962485+0000","last_active":"2024-03-06T07:30:31.967521+0000","last_peered":"2024-03-06T07:30:31.967521+0000","last_clean":"2024-03-06T07:30:31.967521+0000","last_became_active":"2024-03-06T07:30:31.962150+0000","last_became_peered":"2024-03-06T07:30:31.962150+0000","last_unstale":"2024-03-06T07:30:31.967521+0000","last_undegraded":"2024-03-06T07:30:31.967521+0000","last_fullsized":"2024-03-06T07:30:31.967521+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:23.201204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.968945+0000","last_change":"2024-03-06T07:30:31.956884+0000","last_active":"2024-03-06T07:30:31.968945+0000","last_peered":"2024-03-06T07:30:31.968945+0000","last_clean":"2024-03-06T07:30:31.968945+0000","last_became_active":"2024-03-06T07:30:31.956525+0000","last_became_peered":"2024-03-06T07:30:31.956525+0000","last_unstale":"2024-03-06T07:30:31.968945+0000","last_undegraded":"2024-03-06T07:30:31.968945+0000","last_fullsized":"2024-03-06T07:30:31.968945+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:47.388772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958588+0000","last_change":"2024-03-06T07:30:31.952795+0000","last_active":"2024-03-06T07:30:32.958588+0000","last_peered":"2024-03-06T07:30:32.958588+0000","last_clean":"2024-03-06T07:30:32.958588+0000","last_became_active":"2024-03-06T07:30:31.952464+0000","last_became_peered":"2024-03-06T07:30:31.952464+0000","last_unstale":"2024-03-06T07:30:32.958588+0000","last_undegraded":"2024-03-06T07:30:32.958588+0000","last_fullsized":"2024-03-06T07:30:32.958588+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:30.346901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961893+0000","last_change":"2024-03-06T07:30:31.948779+0000","last_active":"2024-03-06T07:30:32.961893+0000","last_peered":"2024-03-06T07:30:32.961893+0000","last_clean":"2024-03-06T07:30:32.961893+0000","last_became_active":"2024-03-06T07:30:31.947906+0000","last_became_peered":"2024-03-06T07:30:31.947906+0000","last_unstale":"2024-03-06T07:30:32.961893+0000","last_undegraded":"2024-03-06T07:30:32.961893+0000","last_fullsized":"2024-03-06T07:30:32.961893+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:46.744237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.955355+0000","last_change":"2024-03-06T07:30:31.947260+0000","last_active":"2024-03-06T07:30:31.955355+0000","last_peered":"2024-03-06T07:30:31.955355+0000","last_clean":"2024-03-06T07:30:31.955355+0000","last_became_active":"2024-03-06T07:30:31.946938+0000","last_became_peered":"2024-03-06T07:30:31.946938+0000","last_unstale":"2024-03-06T07:30:31.955355+0000","last_undegraded":"2024-03-06T07:30:31.955355+0000","last_fullsized":"2024-03-06T07:30:31.955355+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:47.499627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.952002+0000","last_change":"2024-03-06T07:30:31.948067+0000","last_active":"2024-03-06T07:30:31.952002+0000","last_peered":"2024-03-06T07:30:31.952002+0000","last_clean":"2024-03-06T07:30:31.952002+0000","last_became_active":"2024-03-06T07:30:31.947771+0000","last_became_peered":"2024-03-06T07:30:31.947771+0000","last_unstale":"2024-03-06T07:30:31.952002+0000","last_undegraded":"2024-03-06T07:30:31.952002+0000","last_fullsized":"2024-03-06T07:30:31.952002+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:59.378303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961603+0000","last_change":"2024-03-06T07:30:31.951565+0000","last_active":"2024-03-06T07:30:32.961603+0000","last_peered":"2024-03-06T07:30:32.961603+0000","last_clean":"2024-03-06T07:30:32.961603+0000","last_became_active":"2024-03-06T07:30:31.950242+0000","last_became_peered":"2024-03-06T07:30:31.950242+0000","last_unstale":"2024-03-06T07:30:32.961603+0000","last_undegraded":"2024-03-06T07:30:32.961603+0000","last_fullsized":"2024-03-06T07:30:32.961603+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:33.683988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972949+0000","last_change":"2024-03-06T07:30:31.954173+0000","last_active":"2024-03-06T07:30:31.972949+0000","last_peered":"2024-03-06T07:30:31.972949+0000","last_clean":"2024-03-06T07:30:31.972949+0000","last_became_active":"2024-03-06T07:30:31.953613+0000","last_became_peered":"2024-03-06T07:30:31.953613+0000","last_unstale":"2024-03-06T07:30:31.972949+0000","last_undegraded":"2024-03-06T07:30:31.972949+0000","last_fullsized":"2024-03-06T07:30:31.972949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:14.442253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962333+0000","last_change":"2024-03-06T07:30:31.951629+0000","last_active":"2024-03-06T07:30:32.962333+0000","last_peered":"2024-03-06T07:30:32.962333+0000","last_clean":"2024-03-06T07:30:32.962333+0000","last_became_active":"2024-03-06T07:30:31.950143+0000","last_became_peered":"2024-03-06T07:30:31.950143+0000","last_unstale":"2024-03-06T07:30:32.962333+0000","last_undegraded":"2024-03-06T07:30:32.962333+0000","last_fullsized":"2024-03-06T07:30:32.962333+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:52.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.958734+0000","last_change":"2024-03-06T07:30:31.941159+0000","last_active":"2024-03-06T07:30:31.958734+0000","last_peered":"2024-03-06T07:30:31.958734+0000","last_clean":"2024-03-06T07:30:31.958734+0000","last_became_active":"2024-03-06T07:30:31.940348+0000","last_became_peered":"2024-03-06T07:30:31.940348+0000","last_unstale":"2024-03-06T07:30:31.958734+0000","last_undegraded":"2024-03-06T07:30:31.958734+0000","last_fullsized":"2024-03-06T07:30:31.958734+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:07.462631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.967860+0000","last_change":"2024-03-06T07:30:31.958085+0000","last_active":"2024-03-06T07:30:31.967860+0000","last_peered":"2024-03-06T07:30:31.967860+0000","last_clean":"2024-03-06T07:30:31.967860+0000","last_became_active":"2024-03-06T07:30:31.957416+0000","last_became_peered":"2024-03-06T07:30:31.957416+0000","last_unstale":"2024-03-06T07:30:31.967860+0000","last_undegraded":"2024-03-06T07:30:31.967860+0000","last_fullsized":"2024-03-06T07:30:31.967860+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:12.050997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962271+0000","last_change":"2024-03-06T07:30:31.944864+0000","last_active":"2024-03-06T07:30:32.962271+0000","last_peered":"2024-03-06T07:30:32.962271+0000","last_clean":"2024-03-06T07:30:32.962271+0000","last_became_active":"2024-03-06T07:30:31.944562+0000","last_became_peered":"2024-03-06T07:30:31.944562+0000","last_unstale":"2024-03-06T07:30:32.962271+0000","last_undegraded":"2024-03-06T07:30:32.962271+0000","last_fullsized":"2024-03-06T07:30:32.962271+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:30.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.955091+0000","last_change":"2024-03-06T07:30:31.945006+0000","last_active":"2024-03-06T07:30:31.955091+0000","last_peered":"2024-03-06T07:30:31.955091+0000","last_clean":"2024-03-06T07:30:31.955091+0000","last_became_active":"2024-03-06T07:30:31.944719+0000","last_became_peered":"2024-03-06T07:30:31.944719+0000","last_unstale":"2024-03-06T07:30:31.955091+0000","last_undegraded":"2024-03-06T07:30:31.955091+0000","last_fullsized":"2024-03-06T07:30:31.955091+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:41.604421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960815+0000","last_change":"2024-03-06T07:30:31.956694+0000","last_active":"2024-03-06T07:30:32.960815+0000","last_peered":"2024-03-06T07:30:32.960815+0000","last_clean":"2024-03-06T07:30:32.960815+0000","last_became_active":"2024-03-06T07:30:31.955990+0000","last_became_peered":"2024-03-06T07:30:31.955990+0000","last_unstale":"2024-03-06T07:30:32.960815+0000","last_undegraded":"2024-03-06T07:30:32.960815+0000","last_fullsized":"2024-03-06T07:30:32.960815+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:07.842923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962144+0000","last_change":"2024-03-06T07:30:31.964070+0000","last_active":"2024-03-06T07:30:32.962144+0000","last_peered":"2024-03-06T07:30:32.962144+0000","last_clean":"2024-03-06T07:30:32.962144+0000","last_became_active":"2024-03-06T07:30:31.963786+0000","last_became_peered":"2024-03-06T07:30:31.963786+0000","last_unstale":"2024-03-06T07:30:32.962144+0000","last_undegraded":"2024-03-06T07:30:32.962144+0000","last_fullsized":"2024-03-06T07:30:32.962144+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:35.862798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959715+0000","last_change":"2024-03-06T07:30:31.952755+0000","last_active":"2024-03-06T07:30:31.959715+0000","last_peered":"2024-03-06T07:30:31.959715+0000","last_clean":"2024-03-06T07:30:31.959715+0000","last_became_active":"2024-03-06T07:30:31.952413+0000","last_became_peered":"2024-03-06T07:30:31.952413+0000","last_unstale":"2024-03-06T07:30:31.959715+0000","last_undegraded":"2024-03-06T07:30:31.959715+0000","last_fullsized":"2024-03-06T07:30:31.959715+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:27.188131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959983+0000","last_change":"2024-03-06T07:30:31.951631+0000","last_active":"2024-03-06T07:30:31.959983+0000","last_peered":"2024-03-06T07:30:31.959983+0000","last_clean":"2024-03-06T07:30:31.959983+0000","last_became_active":"2024-03-06T07:30:31.951358+0000","last_became_peered":"2024-03-06T07:30:31.951358+0000","last_unstale":"2024-03-06T07:30:31.959983+0000","last_undegraded":"2024-03-06T07:30:31.959983+0000","last_fullsized":"2024-03-06T07:30:31.959983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:55.224508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959290+0000","last_change":"2024-03-06T07:30:31.951322+0000","last_active":"2024-03-06T07:30:32.959290+0000","last_peered":"2024-03-06T07:30:32.959290+0000","last_clean":"2024-03-06T07:30:32.959290+0000","last_became_active":"2024-03-06T07:30:31.951031+0000","last_became_peered":"2024-03-06T07:30:31.951031+0000","last_unstale":"2024-03-06T07:30:32.959290+0000","last_undegraded":"2024-03-06T07:30:32.959290+0000","last_fullsized":"2024-03-06T07:30:32.959290+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:54.804220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.950906+0000","last_change":"2024-03-06T07:30:31.945498+0000","last_active":"2024-03-06T07:30:31.950906+0000","last_peered":"2024-03-06T07:30:31.950906+0000","last_clean":"2024-03-06T07:30:31.950906+0000","last_became_active":"2024-03-06T07:30:31.945175+0000","last_became_peered":"2024-03-06T07:30:31.945175+0000","last_unstale":"2024-03-06T07:30:31.950906+0000","last_undegraded":"2024-03-06T07:30:31.950906+0000","last_fullsized":"2024-03-06T07:30:31.950906+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:33.710446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970476+0000","last_change":"2024-03-06T07:30:31.959202+0000","last_active":"2024-03-06T07:30:31.970476+0000","last_peered":"2024-03-06T07:30:31.970476+0000","last_clean":"2024-03-06T07:30:31.970476+0000","last_became_active":"2024-03-06T07:30:31.950560+0000","last_became_peered":"2024-03-06T07:30:31.950560+0000","last_unstale":"2024-03-06T07:30:31.970476+0000","last_undegraded":"2024-03-06T07:30:31.970476+0000","last_fullsized":"2024-03-06T07:30:31.970476+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:29.489129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958504+0000","last_change":"2024-03-06T07:30:31.952895+0000","last_active":"2024-03-06T07:30:32.958504+0000","last_peered":"2024-03-06T07:30:32.958504+0000","last_clean":"2024-03-06T07:30:32.958504+0000","last_became_active":"2024-03-06T07:30:31.950563+0000","last_became_peered":"2024-03-06T07:30:31.950563+0000","last_unstale":"2024-03-06T07:30:32.958504+0000","last_undegraded":"2024-03-06T07:30:32.958504+0000","last_fullsized":"2024-03-06T07:30:32.958504+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:38.629962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970342+0000","last_change":"2024-03-06T07:30:31.957698+0000","last_active":"2024-03-06T07:30:31.970342+0000","last_peered":"2024-03-06T07:30:31.970342+0000","last_clean":"2024-03-06T07:30:31.970342+0000","last_became_active":"2024-03-06T07:30:31.956919+0000","last_became_peered":"2024-03-06T07:30:31.956919+0000","last_unstale":"2024-03-06T07:30:31.970342+0000","last_undegraded":"2024-03-06T07:30:31.970342+0000","last_fullsized":"2024-03-06T07:30:31.970342+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:05.494883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957408+0000","last_change":"2024-03-06T07:30:31.962125+0000","last_active":"2024-03-06T07:30:32.957408+0000","last_peered":"2024-03-06T07:30:32.957408+0000","last_clean":"2024-03-06T07:30:32.957408+0000","last_became_active":"2024-03-06T07:30:31.961857+0000","last_became_peered":"2024-03-06T07:30:31.961857+0000","last_unstale":"2024-03-06T07:30:32.957408+0000","last_undegraded":"2024-03-06T07:30:32.957408+0000","last_fullsized":"2024-03-06T07:30:32.957408+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:57.463832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.956512+0000","last_change":"2024-03-06T07:30:31.951432+0000","last_active":"2024-03-06T07:30:31.956512+0000","last_peered":"2024-03-06T07:30:31.956512+0000","last_clean":"2024-03-06T07:30:31.956512+0000","last_became_active":"2024-03-06T07:30:31.951124+0000","last_became_peered":"2024-03-06T07:30:31.951124+0000","last_unstale":"2024-03-06T07:30:31.956512+0000","last_undegraded":"2024-03-06T07:30:31.956512+0000","last_fullsized":"2024-03-06T07:30:31.956512+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:42.227552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964089+0000","last_change":"2024-03-06T07:30:31.957551+0000","last_active":"2024-03-06T07:30:31.964089+0000","last_peered":"2024-03-06T07:30:31.964089+0000","last_clean":"2024-03-06T07:30:31.964089+0000","last_became_active":"2024-03-06T07:30:31.957237+0000","last_became_peered":"2024-03-06T07:30:31.957237+0000","last_unstale":"2024-03-06T07:30:31.964089+0000","last_undegraded":"2024-03-06T07:30:31.964089+0000","last_fullsized":"2024-03-06T07:30:31.964089+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:29.766268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961752+0000","last_change":"2024-03-06T07:30:31.953339+0000","last_active":"2024-03-06T07:30:32.961752+0000","last_peered":"2024-03-06T07:30:32.961752+0000","last_clean":"2024-03-06T07:30:32.961752+0000","last_became_active":"2024-03-06T07:30:31.953019+0000","last_became_peered":"2024-03-06T07:30:31.953019+0000","last_unstale":"2024-03-06T07:30:32.961752+0000","last_undegraded":"2024-03-06T07:30:32.961752+0000","last_fullsized":"2024-03-06T07:30:32.961752+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:48.453243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.962962+0000","last_change":"2024-03-06T07:30:31.954458+0000","last_active":"2024-03-06T07:30:31.962962+0000","last_peered":"2024-03-06T07:30:31.962962+0000","last_clean":"2024-03-06T07:30:31.962962+0000","last_became_active":"2024-03-06T07:30:31.953639+0000","last_became_peered":"2024-03-06T07:30:31.953639+0000","last_unstale":"2024-03-06T07:30:31.962962+0000","last_undegraded":"2024-03-06T07:30:31.962962+0000","last_fullsized":"2024-03-06T07:30:31.962962+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:31.307116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.955395+0000","last_change":"2024-03-06T07:30:31.946986+0000","last_active":"2024-03-06T07:30:31.955395+0000","last_peered":"2024-03-06T07:30:31.955395+0000","last_clean":"2024-03-06T07:30:31.955395+0000","last_became_active":"2024-03-06T07:30:31.946682+0000","last_became_peered":"2024-03-06T07:30:31.946682+0000","last_unstale":"2024-03-06T07:30:31.955395+0000","last_undegraded":"2024-03-06T07:30:31.955395+0000","last_fullsized":"2024-03-06T07:30:31.955395+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:59.710493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.973701+0000","last_change":"2024-03-06T07:30:31.960089+0000","last_active":"2024-03-06T07:30:31.973701+0000","last_peered":"2024-03-06T07:30:31.973701+0000","last_clean":"2024-03-06T07:30:31.973701+0000","last_became_active":"2024-03-06T07:30:31.959808+0000","last_became_peered":"2024-03-06T07:30:31.959808+0000","last_unstale":"2024-03-06T07:30:31.973701+0000","last_undegraded":"2024-03-06T07:30:31.973701+0000","last_fullsized":"2024-03-06T07:30:31.973701+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:27.562274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957988+0000","last_change":"2024-03-06T07:30:31.953670+0000","last_active":"2024-03-06T07:30:32.957988+0000","last_peered":"2024-03-06T07:30:32.957988+0000","last_clean":"2024-03-06T07:30:32.957988+0000","last_became_active":"2024-03-06T07:30:31.952056+0000","last_became_peered":"2024-03-06T07:30:31.952056+0000","last_unstale":"2024-03-06T07:30:32.957988+0000","last_undegraded":"2024-03-06T07:30:32.957988+0000","last_fullsized":"2024-03-06T07:30:32.957988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:12.259535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.967783+0000","last_change":"2024-03-06T07:30:31.963413+0000","last_active":"2024-03-06T07:30:31.967783+0000","last_peered":"2024-03-06T07:30:31.967783+0000","last_clean":"2024-03-06T07:30:31.967783+0000","last_became_active":"2024-03-06T07:30:31.962593+0000","last_became_peered":"2024-03-06T07:30:31.962593+0000","last_unstale":"2024-03-06T07:30:31.967783+0000","last_undegraded":"2024-03-06T07:30:31.967783+0000","last_fullsized":"2024-03-06T07:30:31.967783+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:50.943167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.966913+0000","last_change":"2024-03-06T07:30:31.957159+0000","last_active":"2024-03-06T07:30:31.966913+0000","last_peered":"2024-03-06T07:30:31.966913+0000","last_clean":"2024-03-06T07:30:31.966913+0000","last_became_active":"2024-03-06T07:30:31.956852+0000","last_became_peered":"2024-03-06T07:30:31.956852+0000","last_unstale":"2024-03-06T07:30:31.966913+0000","last_undegraded":"2024-03-06T07:30:31.966913+0000","last_fullsized":"2024-03-06T07:30:31.966913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:46.744237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959381+0000","last_change":"2024-03-06T07:30:31.953077+0000","last_active":"2024-03-06T07:30:31.959381+0000","last_peered":"2024-03-06T07:30:31.959381+0000","last_clean":"2024-03-06T07:30:31.959381+0000","last_became_active":"2024-03-06T07:30:31.952615+0000","last_became_peered":"2024-03-06T07:30:31.952615+0000","last_unstale":"2024-03-06T07:30:31.959381+0000","last_undegraded":"2024-03-06T07:30:31.959381+0000","last_fullsized":"2024-03-06T07:30:31.959381+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:10.166452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"54'4","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.976168+0000","last_change":"2024-03-06T07:30:30.932288+0000","last_active":"2024-03-06T07:30:31.976168+0000","last_peered":"2024-03-06T07:30:31.976168+0000","last_clean":"2024-03-06T07:30:31.976168+0000","last_became_active":"2024-03-06T07:30:30.931989+0000","last_became_peered":"2024-03-06T07:30:30.931989+0000","last_unstale":"2024-03-06T07:30:31.976168+0000","last_undegraded":"2024-03-06T07:30:31.976168+0000","last_fullsized":"2024-03-06T07:30:31.976168+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:44.920712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964213+0000","last_change":"2024-03-06T07:30:31.951364+0000","last_active":"2024-03-06T07:30:31.964213+0000","last_peered":"2024-03-06T07:30:31.964213+0000","last_clean":"2024-03-06T07:30:31.964213+0000","last_became_active":"2024-03-06T07:30:31.951085+0000","last_became_peered":"2024-03-06T07:30:31.951085+0000","last_unstale":"2024-03-06T07:30:31.964213+0000","last_undegraded":"2024-03-06T07:30:31.964213+0000","last_fullsized":"2024-03-06T07:30:31.964213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:41.604421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977248+0000","last_change":"2024-03-06T07:30:30.931196+0000","last_active":"2024-03-06T07:30:31.977248+0000","last_peered":"2024-03-06T07:30:31.977248+0000","last_clean":"2024-03-06T07:30:31.977248+0000","last_became_active":"2024-03-06T07:30:30.930810+0000","last_became_peered":"2024-03-06T07:30:30.930810+0000","last_unstale":"2024-03-06T07:30:31.977248+0000","last_undegraded":"2024-03-06T07:30:31.977248+0000","last_fullsized":"2024-03-06T07:30:31.977248+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:41.119688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970038+0000","last_change":"2024-03-06T07:30:31.965160+0000","last_active":"2024-03-06T07:30:31.970038+0000","last_peered":"2024-03-06T07:30:31.970038+0000","last_clean":"2024-03-06T07:30:31.970038+0000","last_became_active":"2024-03-06T07:30:31.964872+0000","last_became_peered":"2024-03-06T07:30:31.964872+0000","last_unstale":"2024-03-06T07:30:31.970038+0000","last_undegraded":"2024-03-06T07:30:31.970038+0000","last_fullsized":"2024-03-06T07:30:31.970038+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:30.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.975415+0000","last_change":"2024-03-06T07:30:30.925000+0000","last_active":"2024-03-06T07:30:31.975415+0000","last_peered":"2024-03-06T07:30:31.975415+0000","last_clean":"2024-03-06T07:30:31.975415+0000","last_became_active":"2024-03-06T07:30:30.924386+0000","last_became_peered":"2024-03-06T07:30:30.924386+0000","last_unstale":"2024-03-06T07:30:31.975415+0000","last_undegraded":"2024-03-06T07:30:31.975415+0000","last_fullsized":"2024-03-06T07:30:31.975415+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:15.803474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.963031+0000","last_change":"2024-03-06T07:30:31.954716+0000","last_active":"2024-03-06T07:30:31.963031+0000","last_peered":"2024-03-06T07:30:31.963031+0000","last_clean":"2024-03-06T07:30:31.963031+0000","last_became_active":"2024-03-06T07:30:31.954153+0000","last_became_peered":"2024-03-06T07:30:31.954153+0000","last_unstale":"2024-03-06T07:30:31.963031+0000","last_undegraded":"2024-03-06T07:30:31.963031+0000","last_fullsized":"2024-03-06T07:30:31.963031+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:48.453243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962541+0000","last_change":"2024-03-06T07:30:30.930436+0000","last_active":"2024-03-06T07:30:32.962541+0000","last_peered":"2024-03-06T07:30:32.962541+0000","last_clean":"2024-03-06T07:30:32.962541+0000","last_became_active":"2024-03-06T07:30:30.927208+0000","last_became_peered":"2024-03-06T07:30:30.927208+0000","last_unstale":"2024-03-06T07:30:32.962541+0000","last_undegraded":"2024-03-06T07:30:32.962541+0000","last_fullsized":"2024-03-06T07:30:32.962541+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:32.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.969896+0000","last_change":"2024-03-06T07:30:31.957530+0000","last_active":"2024-03-06T07:30:31.969896+0000","last_peered":"2024-03-06T07:30:31.969896+0000","last_clean":"2024-03-06T07:30:31.969896+0000","last_became_active":"2024-03-06T07:30:31.956496+0000","last_became_peered":"2024-03-06T07:30:31.956496+0000","last_unstale":"2024-03-06T07:30:31.969896+0000","last_undegraded":"2024-03-06T07:30:31.969896+0000","last_fullsized":"2024-03-06T07:30:31.969896+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:54.804220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960694+0000","last_change":"2024-03-06T07:30:30.923626+0000","last_active":"2024-03-06T07:30:32.960694+0000","last_peered":"2024-03-06T07:30:32.960694+0000","last_clean":"2024-03-06T07:30:32.960694+0000","last_became_active":"2024-03-06T07:30:30.923283+0000","last_became_peered":"2024-03-06T07:30:30.923283+0000","last_unstale":"2024-03-06T07:30:32.960694+0000","last_undegraded":"2024-03-06T07:30:32.960694+0000","last_fullsized":"2024-03-06T07:30:32.960694+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:18.090061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.960410+0000","last_change":"2024-03-06T07:30:31.948958+0000","last_active":"2024-03-06T07:30:31.960410+0000","last_peered":"2024-03-06T07:30:31.960410+0000","last_clean":"2024-03-06T07:30:31.960410+0000","last_became_active":"2024-03-06T07:30:31.948443+0000","last_became_peered":"2024-03-06T07:30:31.948443+0000","last_unstale":"2024-03-06T07:30:31.960410+0000","last_undegraded":"2024-03-06T07:30:31.960410+0000","last_fullsized":"2024-03-06T07:30:31.960410+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:30.346901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972133+0000","last_change":"2024-03-06T07:30:30.923998+0000","last_active":"2024-03-06T07:30:31.972133+0000","last_peered":"2024-03-06T07:30:31.972133+0000","last_clean":"2024-03-06T07:30:31.972133+0000","last_became_active":"2024-03-06T07:30:30.923701+0000","last_became_peered":"2024-03-06T07:30:30.923701+0000","last_unstale":"2024-03-06T07:30:31.972133+0000","last_undegraded":"2024-03-06T07:30:31.972133+0000","last_fullsized":"2024-03-06T07:30:31.972133+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:32.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.951029+0000","last_change":"2024-03-06T07:30:31.944334+0000","last_active":"2024-03-06T07:30:31.951029+0000","last_peered":"2024-03-06T07:30:31.951029+0000","last_clean":"2024-03-06T07:30:31.951029+0000","last_became_active":"2024-03-06T07:30:31.944050+0000","last_became_peered":"2024-03-06T07:30:31.944050+0000","last_unstale":"2024-03-06T07:30:31.951029+0000","last_undegraded":"2024-03-06T07:30:31.951029+0000","last_fullsized":"2024-03-06T07:30:31.951029+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:28.106557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962838+0000","last_change":"2024-03-06T07:30:30.927100+0000","last_active":"2024-03-06T07:30:32.962838+0000","last_peered":"2024-03-06T07:30:32.962838+0000","last_clean":"2024-03-06T07:30:32.962838+0000","last_became_active":"2024-03-06T07:30:30.925900+0000","last_became_peered":"2024-03-06T07:30:30.925900+0000","last_unstale":"2024-03-06T07:30:32.962838+0000","last_undegraded":"2024-03-06T07:30:32.962838+0000","last_fullsized":"2024-03-06T07:30:32.962838+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:49.142087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972912+0000","last_change":"2024-03-06T07:30:31.958940+0000","last_active":"2024-03-06T07:30:31.972912+0000","last_peered":"2024-03-06T07:30:31.972912+0000","last_clean":"2024-03-06T07:30:31.972912+0000","last_became_active":"2024-03-06T07:30:31.958306+0000","last_became_peered":"2024-03-06T07:30:31.958306+0000","last_unstale":"2024-03-06T07:30:31.972912+0000","last_undegraded":"2024-03-06T07:30:31.972912+0000","last_fullsized":"2024-03-06T07:30:31.972912+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:24.763133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961103+0000","last_change":"2024-03-06T07:30:30.919561+0000","last_active":"2024-03-06T07:30:32.961103+0000","last_peered":"2024-03-06T07:30:32.961103+0000","last_clean":"2024-03-06T07:30:32.961103+0000","last_became_active":"2024-03-06T07:30:30.918505+0000","last_became_peered":"2024-03-06T07:30:30.918505+0000","last_unstale":"2024-03-06T07:30:32.961103+0000","last_undegraded":"2024-03-06T07:30:32.961103+0000","last_fullsized":"2024-03-06T07:30:32.961103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:56.458418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964227+0000","last_change":"2024-03-06T07:30:31.964911+0000","last_active":"2024-03-06T07:30:32.964227+0000","last_peered":"2024-03-06T07:30:32.964227+0000","last_clean":"2024-03-06T07:30:32.964227+0000","last_became_active":"2024-03-06T07:30:31.964621+0000","last_became_peered":"2024-03-06T07:30:31.964621+0000","last_unstale":"2024-03-06T07:30:32.964227+0000","last_undegraded":"2024-03-06T07:30:32.964227+0000","last_fullsized":"2024-03-06T07:30:32.964227+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:03.199829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960036+0000","last_change":"2024-03-06T07:30:30.935100+0000","last_active":"2024-03-06T07:30:32.960036+0000","last_peered":"2024-03-06T07:30:32.960036+0000","last_clean":"2024-03-06T07:30:32.960036+0000","last_became_active":"2024-03-06T07:30:30.934888+0000","last_became_peered":"2024-03-06T07:30:30.934888+0000","last_unstale":"2024-03-06T07:30:32.960036+0000","last_undegraded":"2024-03-06T07:30:32.960036+0000","last_fullsized":"2024-03-06T07:30:32.960036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:51.419184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959791+0000","last_change":"2024-03-06T07:30:31.959935+0000","last_active":"2024-03-06T07:30:32.959791+0000","last_peered":"2024-03-06T07:30:32.959791+0000","last_clean":"2024-03-06T07:30:32.959791+0000","last_became_active":"2024-03-06T07:30:31.959601+0000","last_became_peered":"2024-03-06T07:30:31.959601+0000","last_unstale":"2024-03-06T07:30:32.959791+0000","last_undegraded":"2024-03-06T07:30:32.959791+0000","last_fullsized":"2024-03-06T07:30:32.959791+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:14.442253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.933587+0000","last_change":"2024-03-06T07:30:30.926935+0000","last_active":"2024-03-06T07:30:31.933587+0000","last_peered":"2024-03-06T07:30:31.933587+0000","last_clean":"2024-03-06T07:30:31.933587+0000","last_became_active":"2024-03-06T07:30:30.926637+0000","last_became_peered":"2024-03-06T07:30:30.926637+0000","last_unstale":"2024-03-06T07:30:31.933587+0000","last_undegraded":"2024-03-06T07:30:31.933587+0000","last_fullsized":"2024-03-06T07:30:31.933587+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:11.248525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964356+0000","last_change":"2024-03-06T07:30:31.955590+0000","last_active":"2024-03-06T07:30:31.964356+0000","last_peered":"2024-03-06T07:30:31.964356+0000","last_clean":"2024-03-06T07:30:31.964356+0000","last_became_active":"2024-03-06T07:30:31.955144+0000","last_became_peered":"2024-03-06T07:30:31.955144+0000","last_unstale":"2024-03-06T07:30:31.964356+0000","last_undegraded":"2024-03-06T07:30:31.964356+0000","last_fullsized":"2024-03-06T07:30:31.964356+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:57.469428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959020+0000","last_change":"2024-03-06T07:30:30.932132+0000","last_active":"2024-03-06T07:30:32.959020+0000","last_peered":"2024-03-06T07:30:32.959020+0000","last_clean":"2024-03-06T07:30:32.959020+0000","last_became_active":"2024-03-06T07:30:30.931384+0000","last_became_peered":"2024-03-06T07:30:30.931384+0000","last_unstale":"2024-03-06T07:30:32.959020+0000","last_undegraded":"2024-03-06T07:30:32.959020+0000","last_fullsized":"2024-03-06T07:30:32.959020+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:02.188819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958729+0000","last_change":"2024-03-06T07:30:31.963602+0000","last_active":"2024-03-06T07:30:32.958729+0000","last_peered":"2024-03-06T07:30:32.958729+0000","last_clean":"2024-03-06T07:30:32.958729+0000","last_became_active":"2024-03-06T07:30:31.962944+0000","last_became_peered":"2024-03-06T07:30:31.962944+0000","last_unstale":"2024-03-06T07:30:32.958729+0000","last_undegraded":"2024-03-06T07:30:32.958729+0000","last_fullsized":"2024-03-06T07:30:32.958729+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:55.224508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"54'6","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963632+0000","last_change":"2024-03-06T07:30:30.924534+0000","last_active":"2024-03-06T07:30:32.963632+0000","last_peered":"2024-03-06T07:30:32.963632+0000","last_clean":"2024-03-06T07:30:32.963632+0000","last_became_active":"2024-03-06T07:30:30.924231+0000","last_became_peered":"2024-03-06T07:30:30.924231+0000","last_unstale":"2024-03-06T07:30:32.963632+0000","last_undegraded":"2024-03-06T07:30:32.963632+0000","last_fullsized":"2024-03-06T07:30:32.963632+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.838789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963222+0000","last_change":"2024-03-06T07:30:31.957577+0000","last_active":"2024-03-06T07:30:32.963222+0000","last_peered":"2024-03-06T07:30:32.963222+0000","last_clean":"2024-03-06T07:30:32.963222+0000","last_became_active":"2024-03-06T07:30:31.957255+0000","last_became_peered":"2024-03-06T07:30:31.957255+0000","last_unstale":"2024-03-06T07:30:32.963222+0000","last_undegraded":"2024-03-06T07:30:32.963222+0000","last_fullsized":"2024-03-06T07:30:32.963222+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:45.674954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977778+0000","last_change":"2024-03-06T07:30:30.927655+0000","last_active":"2024-03-06T07:30:31.977778+0000","last_peered":"2024-03-06T07:30:31.977778+0000","last_clean":"2024-03-06T07:30:31.977778+0000","last_became_active":"2024-03-06T07:30:30.927399+0000","last_became_peered":"2024-03-06T07:30:30.927399+0000","last_unstale":"2024-03-06T07:30:31.977778+0000","last_undegraded":"2024-03-06T07:30:31.977778+0000","last_fullsized":"2024-03-06T07:30:31.977778+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:44.663944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958918+0000","last_change":"2024-03-06T07:30:31.943368+0000","last_active":"2024-03-06T07:30:32.958918+0000","last_peered":"2024-03-06T07:30:32.958918+0000","last_clean":"2024-03-06T07:30:32.958918+0000","last_became_active":"2024-03-06T07:30:31.943076+0000","last_became_peered":"2024-03-06T07:30:31.943076+0000","last_unstale":"2024-03-06T07:30:32.958918+0000","last_undegraded":"2024-03-06T07:30:32.958918+0000","last_fullsized":"2024-03-06T07:30:32.958918+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:22.621712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.922041+0000","last_change":"2024-03-06T07:30:30.928717+0000","last_active":"2024-03-06T07:30:31.922041+0000","last_peered":"2024-03-06T07:30:31.922041+0000","last_clean":"2024-03-06T07:30:31.922041+0000","last_became_active":"2024-03-06T07:30:30.928398+0000","last_became_peered":"2024-03-06T07:30:30.928398+0000","last_unstale":"2024-03-06T07:30:31.922041+0000","last_undegraded":"2024-03-06T07:30:31.922041+0000","last_fullsized":"2024-03-06T07:30:31.922041+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:29.354693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960869+0000","last_change":"2024-03-06T07:30:31.959404+0000","last_active":"2024-03-06T07:30:32.960869+0000","last_peered":"2024-03-06T07:30:32.960869+0000","last_clean":"2024-03-06T07:30:32.960869+0000","last_became_active":"2024-03-06T07:30:31.958438+0000","last_became_peered":"2024-03-06T07:30:31.958438+0000","last_unstale":"2024-03-06T07:30:32.960869+0000","last_undegraded":"2024-03-06T07:30:32.960869+0000","last_fullsized":"2024-03-06T07:30:32.960869+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:45.931722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962950+0000","last_change":"2024-03-06T07:30:30.924253+0000","last_active":"2024-03-06T07:30:32.962950+0000","last_peered":"2024-03-06T07:30:32.962950+0000","last_clean":"2024-03-06T07:30:32.962950+0000","last_became_active":"2024-03-06T07:30:30.923993+0000","last_became_peered":"2024-03-06T07:30:30.923993+0000","last_unstale":"2024-03-06T07:30:32.962950+0000","last_undegraded":"2024-03-06T07:30:32.962950+0000","last_fullsized":"2024-03-06T07:30:32.962950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:06.831913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957627+0000","last_change":"2024-03-06T07:30:31.963332+0000","last_active":"2024-03-06T07:30:32.957627+0000","last_peered":"2024-03-06T07:30:32.957627+0000","last_clean":"2024-03-06T07:30:32.957627+0000","last_became_active":"2024-03-06T07:30:31.962957+0000","last_became_peered":"2024-03-06T07:30:31.962957+0000","last_unstale":"2024-03-06T07:30:32.957627+0000","last_undegraded":"2024-03-06T07:30:32.957627+0000","last_fullsized":"2024-03-06T07:30:32.957627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:42.130697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957138+0000","last_change":"2024-03-06T07:30:30.932567+0000","last_active":"2024-03-06T07:30:32.957138+0000","last_peered":"2024-03-06T07:30:32.957138+0000","last_clean":"2024-03-06T07:30:32.957138+0000","last_became_active":"2024-03-06T07:30:30.932240+0000","last_became_peered":"2024-03-06T07:30:30.932240+0000","last_unstale":"2024-03-06T07:30:32.957138+0000","last_undegraded":"2024-03-06T07:30:32.957138+0000","last_fullsized":"2024-03-06T07:30:32.957138+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:34.851788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959683+0000","last_change":"2024-03-06T07:30:31.957151+0000","last_active":"2024-03-06T07:30:32.959683+0000","last_peered":"2024-03-06T07:30:32.959683+0000","last_clean":"2024-03-06T07:30:32.959683+0000","last_became_active":"2024-03-06T07:30:31.956339+0000","last_became_peered":"2024-03-06T07:30:31.956339+0000","last_unstale":"2024-03-06T07:30:32.959683+0000","last_undegraded":"2024-03-06T07:30:32.959683+0000","last_fullsized":"2024-03-06T07:30:32.959683+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:56.746980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977305+0000","last_change":"2024-03-06T07:30:30.928024+0000","last_active":"2024-03-06T07:30:31.977305+0000","last_peered":"2024-03-06T07:30:31.977305+0000","last_clean":"2024-03-06T07:30:31.977305+0000","last_became_active":"2024-03-06T07:30:30.925279+0000","last_became_peered":"2024-03-06T07:30:30.925279+0000","last_unstale":"2024-03-06T07:30:31.977305+0000","last_undegraded":"2024-03-06T07:30:31.977305+0000","last_fullsized":"2024-03-06T07:30:31.977305+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:37.618952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964465+0000","last_change":"2024-03-06T07:30:31.969822+0000","last_active":"2024-03-06T07:30:32.964465+0000","last_peered":"2024-03-06T07:30:32.964465+0000","last_clean":"2024-03-06T07:30:32.964465+0000","last_became_active":"2024-03-06T07:30:31.969243+0000","last_became_peered":"2024-03-06T07:30:31.969243+0000","last_unstale":"2024-03-06T07:30:32.964465+0000","last_undegraded":"2024-03-06T07:30:32.964465+0000","last_fullsized":"2024-03-06T07:30:32.964465+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:12.259535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.971467+0000","last_change":"2024-03-06T07:30:30.928639+0000","last_active":"2024-03-06T07:30:31.971467+0000","last_peered":"2024-03-06T07:30:31.971467+0000","last_clean":"2024-03-06T07:30:31.971467+0000","last_became_active":"2024-03-06T07:30:30.928398+0000","last_became_peered":"2024-03-06T07:30:30.928398+0000","last_unstale":"2024-03-06T07:30:31.971467+0000","last_undegraded":"2024-03-06T07:30:31.971467+0000","last_fullsized":"2024-03-06T07:30:31.971467+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:34.851788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.973603+0000","last_change":"2024-03-06T07:30:31.960234+0000","last_active":"2024-03-06T07:30:31.973603+0000","last_peered":"2024-03-06T07:30:31.973603+0000","last_clean":"2024-03-06T07:30:31.973603+0000","last_became_active":"2024-03-06T07:30:31.959337+0000","last_became_peered":"2024-03-06T07:30:31.959337+0000","last_unstale":"2024-03-06T07:30:31.973603+0000","last_undegraded":"2024-03-06T07:30:31.973603+0000","last_fullsized":"2024-03-06T07:30:31.973603+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:38.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963932+0000","last_change":"2024-03-06T07:30:30.928134+0000","last_active":"2024-03-06T07:30:32.963932+0000","last_peered":"2024-03-06T07:30:32.963932+0000","last_clean":"2024-03-06T07:30:32.963932+0000","last_became_active":"2024-03-06T07:30:30.927804+0000","last_became_peered":"2024-03-06T07:30:30.927804+0000","last_unstale":"2024-03-06T07:30:32.963932+0000","last_undegraded":"2024-03-06T07:30:32.963932+0000","last_fullsized":"2024-03-06T07:30:32.963932+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:34.851788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959006+0000","last_change":"2024-03-06T07:30:31.957489+0000","last_active":"2024-03-06T07:30:32.959006+0000","last_peered":"2024-03-06T07:30:32.959006+0000","last_clean":"2024-03-06T07:30:32.959006+0000","last_became_active":"2024-03-06T07:30:31.957216+0000","last_became_peered":"2024-03-06T07:30:31.957216+0000","last_unstale":"2024-03-06T07:30:32.959006+0000","last_undegraded":"2024-03-06T07:30:32.959006+0000","last_fullsized":"2024-03-06T07:30:32.959006+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:47.388772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.940720+0000","last_change":"2024-03-06T07:30:30.924150+0000","last_active":"2024-03-06T07:30:31.940720+0000","last_peered":"2024-03-06T07:30:31.940720+0000","last_clean":"2024-03-06T07:30:31.940720+0000","last_became_active":"2024-03-06T07:30:30.923745+0000","last_became_peered":"2024-03-06T07:30:30.923745+0000","last_unstale":"2024-03-06T07:30:31.940720+0000","last_undegraded":"2024-03-06T07:30:31.940720+0000","last_fullsized":"2024-03-06T07:30:31.940720+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:02.188819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963433+0000","last_change":"2024-03-06T07:30:31.956449+0000","last_active":"2024-03-06T07:30:32.963433+0000","last_peered":"2024-03-06T07:30:32.963433+0000","last_clean":"2024-03-06T07:30:32.963433+0000","last_became_active":"2024-03-06T07:30:31.956185+0000","last_became_peered":"2024-03-06T07:30:31.956185+0000","last_unstale":"2024-03-06T07:30:32.963433+0000","last_undegraded":"2024-03-06T07:30:32.963433+0000","last_fullsized":"2024-03-06T07:30:32.963433+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:57.463832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977698+0000","last_change":"2024-03-06T07:30:30.918948+0000","last_active":"2024-03-06T07:30:31.977698+0000","last_peered":"2024-03-06T07:30:31.977698+0000","last_clean":"2024-03-06T07:30:31.977698+0000","last_became_active":"2024-03-06T07:30:30.918671+0000","last_became_peered":"2024-03-06T07:30:30.918671+0000","last_unstale":"2024-03-06T07:30:31.977698+0000","last_undegraded":"2024-03-06T07:30:31.977698+0000","last_fullsized":"2024-03-06T07:30:31.977698+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:56.452822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.951641+0000","last_change":"2024-03-06T07:30:31.944826+0000","last_active":"2024-03-06T07:30:31.951641+0000","last_peered":"2024-03-06T07:30:31.951641+0000","last_clean":"2024-03-06T07:30:31.951641+0000","last_became_active":"2024-03-06T07:30:31.944523+0000","last_became_peered":"2024-03-06T07:30:31.944523+0000","last_unstale":"2024-03-06T07:30:31.951641+0000","last_undegraded":"2024-03-06T07:30:31.951641+0000","last_fullsized":"2024-03-06T07:30:31.951641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:27.274064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.938608+0000","last_change":"2024-03-06T07:30:30.923028+0000","last_active":"2024-03-06T07:30:31.938608+0000","last_peered":"2024-03-06T07:30:31.938608+0000","last_clean":"2024-03-06T07:30:31.938608+0000","last_became_active":"2024-03-06T07:30:30.922730+0000","last_became_peered":"2024-03-06T07:30:30.922730+0000","last_unstale":"2024-03-06T07:30:31.938608+0000","last_undegraded":"2024-03-06T07:30:31.938608+0000","last_fullsized":"2024-03-06T07:30:31.938608+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:16.265557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959460+0000","last_change":"2024-03-06T07:30:31.952694+0000","last_active":"2024-03-06T07:30:31.959460+0000","last_peered":"2024-03-06T07:30:31.959460+0000","last_clean":"2024-03-06T07:30:31.959460+0000","last_became_active":"2024-03-06T07:30:31.952438+0000","last_became_peered":"2024-03-06T07:30:31.952438+0000","last_unstale":"2024-03-06T07:30:31.959460+0000","last_undegraded":"2024-03-06T07:30:31.959460+0000","last_fullsized":"2024-03-06T07:30:31.959460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:44.915416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961419+0000","last_change":"2024-03-06T07:30:30.919320+0000","last_active":"2024-03-06T07:30:32.961419+0000","last_peered":"2024-03-06T07:30:32.961419+0000","last_clean":"2024-03-06T07:30:32.961419+0000","last_became_active":"2024-03-06T07:30:30.919011+0000","last_became_peered":"2024-03-06T07:30:30.919011+0000","last_unstale":"2024-03-06T07:30:32.961419+0000","last_undegraded":"2024-03-06T07:30:32.961419+0000","last_fullsized":"2024-03-06T07:30:32.961419+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:44.920712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964300+0000","last_change":"2024-03-06T07:30:31.947473+0000","last_active":"2024-03-06T07:30:32.964300+0000","last_peered":"2024-03-06T07:30:32.964300+0000","last_clean":"2024-03-06T07:30:32.964300+0000","last_became_active":"2024-03-06T07:30:31.947192+0000","last_became_peered":"2024-03-06T07:30:31.947192+0000","last_unstale":"2024-03-06T07:30:32.964300+0000","last_undegraded":"2024-03-06T07:30:32.964300+0000","last_fullsized":"2024-03-06T07:30:32.964300+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:42.130697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.974356+0000","last_change":"2024-03-06T07:30:30.917523+0000","last_active":"2024-03-06T07:30:31.974356+0000","last_peered":"2024-03-06T07:30:31.974356+0000","last_clean":"2024-03-06T07:30:31.974356+0000","last_became_active":"2024-03-06T07:30:30.917235+0000","last_became_peered":"2024-03-06T07:30:30.917235+0000","last_unstale":"2024-03-06T07:30:31.974356+0000","last_undegraded":"2024-03-06T07:30:31.974356+0000","last_fullsized":"2024-03-06T07:30:31.974356+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:27.095547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964515+0000","last_change":"2024-03-06T07:30:31.945698+0000","last_active":"2024-03-06T07:30:32.964515+0000","last_peered":"2024-03-06T07:30:32.964515+0000","last_clean":"2024-03-06T07:30:32.964515+0000","last_became_active":"2024-03-06T07:30:31.943698+0000","last_became_peered":"2024-03-06T07:30:31.943698+0000","last_unstale":"2024-03-06T07:30:32.964515+0000","last_undegraded":"2024-03-06T07:30:32.964515+0000","last_fullsized":"2024-03-06T07:30:32.964515+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:38.629962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.955903+0000","last_change":"2024-03-06T07:30:30.930515+0000","last_active":"2024-03-06T07:30:32.955903+0000","last_peered":"2024-03-06T07:30:32.955903+0000","last_clean":"2024-03-06T07:30:32.955903+0000","last_became_active":"2024-03-06T07:30:30.930223+0000","last_became_peered":"2024-03-06T07:30:30.930223+0000","last_unstale":"2024-03-06T07:30:32.955903+0000","last_undegraded":"2024-03-06T07:30:32.955903+0000","last_fullsized":"2024-03-06T07:30:32.955903+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:51.419184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964329+0000","last_change":"2024-03-06T07:30:31.955384+0000","last_active":"2024-03-06T07:30:31.964329+0000","last_peered":"2024-03-06T07:30:31.964329+0000","last_clean":"2024-03-06T07:30:31.964329+0000","last_became_active":"2024-03-06T07:30:31.954792+0000","last_became_peered":"2024-03-06T07:30:31.954792+0000","last_unstale":"2024-03-06T07:30:31.964329+0000","last_undegraded":"2024-03-06T07:30:31.964329+0000","last_fullsized":"2024-03-06T07:30:31.964329+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:42.227552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.921804+0000","last_change":"2024-03-06T07:30:30.915129+0000","last_active":"2024-03-06T07:30:31.921804+0000","last_peered":"2024-03-06T07:30:31.921804+0000","last_clean":"2024-03-06T07:30:31.921804+0000","last_became_active":"2024-03-06T07:30:30.914662+0000","last_became_peered":"2024-03-06T07:30:30.914662+0000","last_unstale":"2024-03-06T07:30:31.921804+0000","last_undegraded":"2024-03-06T07:30:31.921804+0000","last_fullsized":"2024-03-06T07:30:31.921804+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:06.451621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.953852+0000","last_change":"2024-03-06T07:30:31.949441+0000","last_active":"2024-03-06T07:30:31.953852+0000","last_peered":"2024-03-06T07:30:31.953852+0000","last_clean":"2024-03-06T07:30:31.953852+0000","last_became_active":"2024-03-06T07:30:31.949160+0000","last_became_peered":"2024-03-06T07:30:31.949160+0000","last_unstale":"2024-03-06T07:30:31.953852+0000","last_undegraded":"2024-03-06T07:30:31.953852+0000","last_fullsized":"2024-03-06T07:30:31.953852+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:31.877967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962831+0000","last_change":"2024-03-06T07:30:30.926512+0000","last_active":"2024-03-06T07:30:32.962831+0000","last_peered":"2024-03-06T07:30:32.962831+0000","last_clean":"2024-03-06T07:30:32.962831+0000","last_became_active":"2024-03-06T07:30:30.926186+0000","last_became_peered":"2024-03-06T07:30:30.926186+0000","last_unstale":"2024-03-06T07:30:32.962831+0000","last_undegraded":"2024-03-06T07:30:32.962831+0000","last_fullsized":"2024-03-06T07:30:32.962831+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:51.419184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970362+0000","last_change":"2024-03-06T07:30:31.957935+0000","last_active":"2024-03-06T07:30:31.970362+0000","last_peered":"2024-03-06T07:30:31.970362+0000","last_clean":"2024-03-06T07:30:31.970362+0000","last_became_active":"2024-03-06T07:30:31.957258+0000","last_became_peered":"2024-03-06T07:30:31.957258+0000","last_unstale":"2024-03-06T07:30:31.970362+0000","last_undegraded":"2024-03-06T07:30:31.970362+0000","last_fullsized":"2024-03-06T07:30:31.970362+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:56.746980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.938489+0000","last_change":"2024-03-06T07:30:30.927710+0000","last_active":"2024-03-06T07:30:31.938489+0000","last_peered":"2024-03-06T07:30:31.938489+0000","last_clean":"2024-03-06T07:30:31.938489+0000","last_became_active":"2024-03-06T07:30:30.927410+0000","last_became_peered":"2024-03-06T07:30:30.927410+0000","last_unstale":"2024-03-06T07:30:31.938489+0000","last_undegraded":"2024-03-06T07:30:31.938489+0000","last_fullsized":"2024-03-06T07:30:31.938489+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.162321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959310+0000","last_change":"2024-03-06T07:30:31.953994+0000","last_active":"2024-03-06T07:30:32.959310+0000","last_peered":"2024-03-06T07:30:32.959310+0000","last_clean":"2024-03-06T07:30:32.959310+0000","last_became_active":"2024-03-06T07:30:31.953659+0000","last_became_peered":"2024-03-06T07:30:31.953659+0000","last_unstale":"2024-03-06T07:30:32.959310+0000","last_undegraded":"2024-03-06T07:30:32.959310+0000","last_fullsized":"2024-03-06T07:30:32.959310+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:05.494883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961076+0000","last_change":"2024-03-06T07:30:30.923679+0000","last_active":"2024-03-06T07:30:32.961076+0000","last_peered":"2024-03-06T07:30:32.961076+0000","last_clean":"2024-03-06T07:30:32.961076+0000","last_became_active":"2024-03-06T07:30:30.923350+0000","last_became_peered":"2024-03-06T07:30:30.923350+0000","last_unstale":"2024-03-06T07:30:32.961076+0000","last_undegraded":"2024-03-06T07:30:32.961076+0000","last_fullsized":"2024-03-06T07:30:32.961076+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:11.039987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959264+0000","last_change":"2024-03-06T07:30:31.942901+0000","last_active":"2024-03-06T07:30:31.959264+0000","last_peered":"2024-03-06T07:30:31.959264+0000","last_clean":"2024-03-06T07:30:31.959264+0000","last_became_active":"2024-03-06T07:30:31.942606+0000","last_became_peered":"2024-03-06T07:30:31.942606+0000","last_unstale":"2024-03-06T07:30:31.959264+0000","last_undegraded":"2024-03-06T07:30:31.959264+0000","last_fullsized":"2024-03-06T07:30:31.959264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:30.365703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.938752+0000","last_change":"2024-03-06T07:30:30.921904+0000","last_active":"2024-03-06T07:30:31.938752+0000","last_peered":"2024-03-06T07:30:31.938752+0000","last_clean":"2024-03-06T07:30:31.938752+0000","last_became_active":"2024-03-06T07:30:30.921679+0000","last_became_peered":"2024-03-06T07:30:30.921679+0000","last_unstale":"2024-03-06T07:30:31.938752+0000","last_undegraded":"2024-03-06T07:30:31.938752+0000","last_fullsized":"2024-03-06T07:30:31.938752+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:51.138949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972927+0000","last_change":"2024-03-06T07:30:31.959158+0000","last_active":"2024-03-06T07:30:31.972927+0000","last_peered":"2024-03-06T07:30:31.972927+0000","last_clean":"2024-03-06T07:30:31.972927+0000","last_became_active":"2024-03-06T07:30:31.958637+0000","last_became_peered":"2024-03-06T07:30:31.958637+0000","last_unstale":"2024-03-06T07:30:31.972927+0000","last_undegraded":"2024-03-06T07:30:31.972927+0000","last_fullsized":"2024-03-06T07:30:31.972927+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:33.705056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.929793+0000","last_change":"2024-03-06T07:30:30.919539+0000","last_active":"2024-03-06T07:30:31.929793+0000","last_peered":"2024-03-06T07:30:31.929793+0000","last_clean":"2024-03-06T07:30:31.929793+0000","last_became_active":"2024-03-06T07:30:30.919273+0000","last_became_peered":"2024-03-06T07:30:30.919273+0000","last_unstale":"2024-03-06T07:30:31.929793+0000","last_undegraded":"2024-03-06T07:30:31.929793+0000","last_fullsized":"2024-03-06T07:30:31.929793+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:29.776982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.961219+0000","last_change":"2024-03-06T07:30:31.953890+0000","last_active":"2024-03-06T07:30:31.961219+0000","last_peered":"2024-03-06T07:30:31.961219+0000","last_clean":"2024-03-06T07:30:31.961219+0000","last_became_active":"2024-03-06T07:30:31.953251+0000","last_became_peered":"2024-03-06T07:30:31.953251+0000","last_unstale":"2024-03-06T07:30:31.961219+0000","last_undegraded":"2024-03-06T07:30:31.961219+0000","last_fullsized":"2024-03-06T07:30:31.961219+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:22.621712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.921597+0000","last_change":"2024-03-06T07:30:30.930617+0000","last_active":"2024-03-06T07:30:31.921597+0000","last_peered":"2024-03-06T07:30:31.921597+0000","last_clean":"2024-03-06T07:30:31.921597+0000","last_became_active":"2024-03-06T07:30:30.930166+0000","last_became_peered":"2024-03-06T07:30:30.930166+0000","last_unstale":"2024-03-06T07:30:31.921597+0000","last_undegraded":"2024-03-06T07:30:31.921597+0000","last_fullsized":"2024-03-06T07:30:31.921597+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:51.138949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":233,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957295+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:30:32.957295+0000","last_peered":"2024-03-06T07:30:32.957295+0000","last_clean":"2024-03-06T07:30:32.957295+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:30:32.957295+0000","last_undegraded":"2024-03-06T07:30:32.957295+0000","last_fullsized":"2024-03-06T07:30:32.957295+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":1249280,"data_stored":857390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":266862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":267514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":659456,"data_stored":268474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":266918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":266918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":651264,"data_stored":266952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1257472,"data_stored":859424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:36.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:36.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.636+0000 7f73e3e00640 1 Processor -- start 2024-03-06T07:30:36.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.637+0000 7f73e3e00640 1 -- start start 2024-03-06T07:30:36.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.639+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 0x7f73dc071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.639+0000 7f73e3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73dc0cf3e0 0x7f73dc0724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e3e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f73dc0c9ec0 0x7f73dc0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e3e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73dc0cabf0 con 0x7f73dc0cf3e0 2024-03-06T07:30:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e3e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73dc0ce900 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0cebe0 con 0x7f73dc144660 2024-03-06T07:30:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73dc0cf3e0 0x7f73dc0724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73dc0cf3e0 0x7f73dc0724b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39954/0 (socket says 172.21.15.114:39954) 2024-03-06T07:30:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e0a00640 1 -- 172.21.15.114:0/981878173 learned_addr learned my addr 172.21.15.114:0/981878173 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e1400640 1 --2- 172.21.15.114:0/981878173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 0x7f73dc071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e1e00640 1 --2- 172.21.15.114:0/981878173 >> v2:172.21.15.177:3300/0 conn(0x7f73dc0c9ec0 0x7f73dc0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e0a00640 1 -- 172.21.15.114:0/981878173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 msgr2=0x7f73dc071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.640+0000 7f73e0a00640 1 --2- 172.21.15.114:0/981878173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 0x7f73dc071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.641+0000 7f73e0a00640 1 -- 172.21.15.114:0/981878173 >> v2:172.21.15.177:3300/0 conn(0x7f73dc0c9ec0 msgr2=0x7f73dc0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.641+0000 7f73e0a00640 1 --2- 172.21.15.114:0/981878173 >> v2:172.21.15.177:3300/0 conn(0x7f73dc0c9ec0 0x7f73dc0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.641+0000 7f73e0a00640 1 -- 172.21.15.114:0/981878173 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc0ceec0 con 0x7f73dc0cf3e0 2024-03-06T07:30:36.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.642+0000 7f73e0a00640 1 --2- 172.21.15.114:0/981878173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0cf3e0 0x7f73dc0724b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0d1bf0 tx=0x7f73c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51f2adf8df62e34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.642+0000 7f73dbe00640 1 -- 172.21.15.114:0/981878173 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8035020 con 0x7f73dc0cf3e0 2024-03-06T07:30:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.642+0000 7f73dbe00640 1 -- 172.21.15.114:0/981878173 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c8031870 con 0x7f73dc0cf3e0 2024-03-06T07:30:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.642+0000 7f73dbe00640 1 -- 172.21.15.114:0/981878173 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8031c40 con 0x7f73dc0cf3e0 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 -- 172.21.15.114:0/981878173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0cf3e0 msgr2=0x7f73dc0724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 --2- 172.21.15.114:0/981878173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0cf3e0 0x7f73dc0724b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0d1bf0 tx=0x7f73c802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 -- 172.21.15.114:0/981878173 shutdown_connections 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 --2- 172.21.15.114:0/981878173 >> v2:172.21.15.177:3300/0 conn(0x7f73dc0c9ec0 0x7f73dc0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 --2- 172.21.15.114:0/981878173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0cf3e0 0x7f73dc0724b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 --2- 172.21.15.114:0/981878173 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 0x7f73dc071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 -- 172.21.15.114:0/981878173 >> 172.21.15.114:0/981878173 conn(0x7f73dc0d0b90 msgr2=0x7f73dc0d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 -- 172.21.15.114:0/981878173 shutdown_connections 2024-03-06T07:30:36.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.643+0000 7f73e3e00640 1 -- 172.21.15.114:0/981878173 wait complete. 2024-03-06T07:30:36.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.646+0000 7f73e3e00640 1 Processor -- start 2024-03-06T07:30:36.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.646+0000 7f73e3e00640 1 -- start start 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0c9ec0 0x7f73dc06efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73dc0cf3e0 0x7f73dc1601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 0x7f73dc1f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0ca9d0 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0ca850 con 0x7f73dc0cf3e0 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc161bf0 con 0x7f73dc144660 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0c9ec0 0x7f73dc06efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0c9ec0 0x7f73dc06efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39970/0 (socket says 172.21.15.114:39970) 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.648+0000 7f73e1400640 1 -- 172.21.15.114:0/1476311395 learned_addr learned my addr 172.21.15.114:0/1476311395 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e1400640 1 -- 172.21.15.114:0/1476311395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 msgr2=0x7f73dc1f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e1400640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 0x7f73dc1f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e1400640 1 -- 172.21.15.114:0/1476311395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73dc0cf3e0 msgr2=0x7f73dc1601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e1400640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73dc0cf3e0 0x7f73dc1601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e1400640 1 -- 172.21.15.114:0/1476311395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc0b46b0 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e1400640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0c9ec0 0x7f73dc06efa0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f73d000d5f0 tx=0x7f73d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73d9600640 1 -- 172.21.15.114:0/1476311395 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d00125f0 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73d9600640 1 -- 172.21.15.114:0/1476311395 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d0019070 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73d9600640 1 -- 172.21.15.114:0/1476311395 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d00129d0 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73dc0b4990 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.649+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73dc0b4f50 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.651+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73dc04f330 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.651+0000 7f73d9600640 1 -- 172.21.15.114:0/1476311395 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73d0012b70 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.652+0000 7f73d9600640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73bc06b4d0 0x7f73bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.652+0000 7f73d9600640 1 -- 172.21.15.114:0/1476311395 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73d008a6b0 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.653+0000 7f73e0a00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73bc06b4d0 0x7f73bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.653+0000 7f73e0a00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73bc06b4d0 0x7f73bc06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f73c8036000 tx=0x7f73c8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.657+0000 7f73d9600640 1 -- 172.21.15.114:0/1476311395 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73d0059490 con 0x7f73dc0c9ec0 2024-03-06T07:30:36.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.846+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73dc143c60 con 0x7f73bc06b4d0 2024-03-06T07:30:36.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.855+0000 7f73d9600640 1 -- 172.21.15.114:0/1476311395 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251912 (secure 0 0 0) 0x7f73dc143c60 con 0x7f73bc06b4d0 2024-03-06T07:30:36.862 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:36.863 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:30:36.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T07:30:36.860+0000 7f1e47c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73bc06b4d0 msgr2=0x7f73bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73bc06b4d0 0x7f73bc06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f73c8036000 tx=0x7f73c8030000 comp rx=0 tx=0).stop 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0c9ec0 msgr2=0x7f73dc06efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0c9ec0 0x7f73dc06efa0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f73d000d5f0 tx=0x7f73d000dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 shutdown_connections 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73dc144660 0x7f73dc1f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73dc0cf3e0 0x7f73dc1601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73bc06b4d0 0x7f73bc06d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 --2- 172.21.15.114:0/1476311395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73dc0c9ec0 0x7f73dc06efa0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 >> 172.21.15.114:0/1476311395 conn(0x7f73dc0d0b90 msgr2=0x7f73dc1428d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 shutdown_connections 2024-03-06T07:30:36.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:36.861+0000 7f73e3e00640 1 -- 172.21.15.114:0/1476311395 wait complete. 2024-03-06T07:30:36.883 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-03-06T07:30:35.146795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255720,"kb_used_data":6280,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719000,"statfs":{"total":773094113280,"available":772832256000,"internally_reserved":0,"allocated":6430720,"data_stored":3320452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.259613"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.967521+0000","last_change":"2024-03-06T07:30:31.962485+0000","last_active":"2024-03-06T07:30:31.967521+0000","last_peered":"2024-03-06T07:30:31.967521+0000","last_clean":"2024-03-06T07:30:31.967521+0000","last_became_active":"2024-03-06T07:30:31.962150+0000","last_became_peered":"2024-03-06T07:30:31.962150+0000","last_unstale":"2024-03-06T07:30:31.967521+0000","last_undegraded":"2024-03-06T07:30:31.967521+0000","last_fullsized":"2024-03-06T07:30:31.967521+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:23.201204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.968945+0000","last_change":"2024-03-06T07:30:31.956884+0000","last_active":"2024-03-06T07:30:31.968945+0000","last_peered":"2024-03-06T07:30:31.968945+0000","last_clean":"2024-03-06T07:30:31.968945+0000","last_became_active":"2024-03-06T07:30:31.956525+0000","last_became_peered":"2024-03-06T07:30:31.956525+0000","last_unstale":"2024-03-06T07:30:31.968945+0000","last_undegraded":"2024-03-06T07:30:31.968945+0000","last_fullsized":"2024-03-06T07:30:31.968945+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:47.388772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958588+0000","last_change":"2024-03-06T07:30:31.952795+0000","last_active":"2024-03-06T07:30:32.958588+0000","last_peered":"2024-03-06T07:30:32.958588+0000","last_clean":"2024-03-06T07:30:32.958588+0000","last_became_active":"2024-03-06T07:30:31.952464+0000","last_became_peered":"2024-03-06T07:30:31.952464+0000","last_unstale":"2024-03-06T07:30:32.958588+0000","last_undegraded":"2024-03-06T07:30:32.958588+0000","last_fullsized":"2024-03-06T07:30:32.958588+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:30.346901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961893+0000","last_change":"2024-03-06T07:30:31.948779+0000","last_active":"2024-03-06T07:30:32.961893+0000","last_peered":"2024-03-06T07:30:32.961893+0000","last_clean":"2024-03-06T07:30:32.961893+0000","last_became_active":"2024-03-06T07:30:31.947906+0000","last_became_peered":"2024-03-06T07:30:31.947906+0000","last_unstale":"2024-03-06T07:30:32.961893+0000","last_undegraded":"2024-03-06T07:30:32.961893+0000","last_fullsized":"2024-03-06T07:30:32.961893+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:46.744237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.955355+0000","last_change":"2024-03-06T07:30:31.947260+0000","last_active":"2024-03-06T07:30:31.955355+0000","last_peered":"2024-03-06T07:30:31.955355+0000","last_clean":"2024-03-06T07:30:31.955355+0000","last_became_active":"2024-03-06T07:30:31.946938+0000","last_became_peered":"2024-03-06T07:30:31.946938+0000","last_unstale":"2024-03-06T07:30:31.955355+0000","last_undegraded":"2024-03-06T07:30:31.955355+0000","last_fullsized":"2024-03-06T07:30:31.955355+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:47.499627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.952002+0000","last_change":"2024-03-06T07:30:31.948067+0000","last_active":"2024-03-06T07:30:31.952002+0000","last_peered":"2024-03-06T07:30:31.952002+0000","last_clean":"2024-03-06T07:30:31.952002+0000","last_became_active":"2024-03-06T07:30:31.947771+0000","last_became_peered":"2024-03-06T07:30:31.947771+0000","last_unstale":"2024-03-06T07:30:31.952002+0000","last_undegraded":"2024-03-06T07:30:31.952002+0000","last_fullsized":"2024-03-06T07:30:31.952002+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:59.378303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961603+0000","last_change":"2024-03-06T07:30:31.951565+0000","last_active":"2024-03-06T07:30:32.961603+0000","last_peered":"2024-03-06T07:30:32.961603+0000","last_clean":"2024-03-06T07:30:32.961603+0000","last_became_active":"2024-03-06T07:30:31.950242+0000","last_became_peered":"2024-03-06T07:30:31.950242+0000","last_unstale":"2024-03-06T07:30:32.961603+0000","last_undegraded":"2024-03-06T07:30:32.961603+0000","last_fullsized":"2024-03-06T07:30:32.961603+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:33.683988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972949+0000","last_change":"2024-03-06T07:30:31.954173+0000","last_active":"2024-03-06T07:30:31.972949+0000","last_peered":"2024-03-06T07:30:31.972949+0000","last_clean":"2024-03-06T07:30:31.972949+0000","last_became_active":"2024-03-06T07:30:31.953613+0000","last_became_peered":"2024-03-06T07:30:31.953613+0000","last_unstale":"2024-03-06T07:30:31.972949+0000","last_undegraded":"2024-03-06T07:30:31.972949+0000","last_fullsized":"2024-03-06T07:30:31.972949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:14.442253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962333+0000","last_change":"2024-03-06T07:30:31.951629+0000","last_active":"2024-03-06T07:30:32.962333+0000","last_peered":"2024-03-06T07:30:32.962333+0000","last_clean":"2024-03-06T07:30:32.962333+0000","last_became_active":"2024-03-06T07:30:31.950143+0000","last_became_peered":"2024-03-06T07:30:31.950143+0000","last_unstale":"2024-03-06T07:30:32.962333+0000","last_undegraded":"2024-03-06T07:30:32.962333+0000","last_fullsized":"2024-03-06T07:30:32.962333+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:52.430194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.958734+0000","last_change":"2024-03-06T07:30:31.941159+0000","last_active":"2024-03-06T07:30:31.958734+0000","last_peered":"2024-03-06T07:30:31.958734+0000","last_clean":"2024-03-06T07:30:31.958734+0000","last_became_active":"2024-03-06T07:30:31.940348+0000","last_became_peered":"2024-03-06T07:30:31.940348+0000","last_unstale":"2024-03-06T07:30:31.958734+0000","last_undegraded":"2024-03-06T07:30:31.958734+0000","last_fullsized":"2024-03-06T07:30:31.958734+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:07.462631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.967860+0000","last_change":"2024-03-06T07:30:31.958085+0000","last_active":"2024-03-06T07:30:31.967860+0000","last_peered":"2024-03-06T07:30:31.967860+0000","last_clean":"2024-03-06T07:30:31.967860+0000","last_became_active":"2024-03-06T07:30:31.957416+0000","last_became_peered":"2024-03-06T07:30:31.957416+0000","last_unstale":"2024-03-06T07:30:31.967860+0000","last_undegraded":"2024-03-06T07:30:31.967860+0000","last_fullsized":"2024-03-06T07:30:31.967860+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:12.050997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962271+0000","last_change":"2024-03-06T07:30:31.944864+0000","last_active":"2024-03-06T07:30:32.962271+0000","last_peered":"2024-03-06T07:30:32.962271+0000","last_clean":"2024-03-06T07:30:32.962271+0000","last_became_active":"2024-03-06T07:30:31.944562+0000","last_became_peered":"2024-03-06T07:30:31.944562+0000","last_unstale":"2024-03-06T07:30:32.962271+0000","last_undegraded":"2024-03-06T07:30:32.962271+0000","last_fullsized":"2024-03-06T07:30:32.962271+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:30.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.955091+0000","last_change":"2024-03-06T07:30:31.945006+0000","last_active":"2024-03-06T07:30:31.955091+0000","last_peered":"2024-03-06T07:30:31.955091+0000","last_clean":"2024-03-06T07:30:31.955091+0000","last_became_active":"2024-03-06T07:30:31.944719+0000","last_became_peered":"2024-03-06T07:30:31.944719+0000","last_unstale":"2024-03-06T07:30:31.955091+0000","last_undegraded":"2024-03-06T07:30:31.955091+0000","last_fullsized":"2024-03-06T07:30:31.955091+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:41.604421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960815+0000","last_change":"2024-03-06T07:30:31.956694+0000","last_active":"2024-03-06T07:30:32.960815+0000","last_peered":"2024-03-06T07:30:32.960815+0000","last_clean":"2024-03-06T07:30:32.960815+0000","last_became_active":"2024-03-06T07:30:31.955990+0000","last_became_peered":"2024-03-06T07:30:31.955990+0000","last_unstale":"2024-03-06T07:30:32.960815+0000","last_undegraded":"2024-03-06T07:30:32.960815+0000","last_fullsized":"2024-03-06T07:30:32.960815+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:07.842923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962144+0000","last_change":"2024-03-06T07:30:31.964070+0000","last_active":"2024-03-06T07:30:32.962144+0000","last_peered":"2024-03-06T07:30:32.962144+0000","last_clean":"2024-03-06T07:30:32.962144+0000","last_became_active":"2024-03-06T07:30:31.963786+0000","last_became_peered":"2024-03-06T07:30:31.963786+0000","last_unstale":"2024-03-06T07:30:32.962144+0000","last_undegraded":"2024-03-06T07:30:32.962144+0000","last_fullsized":"2024-03-06T07:30:32.962144+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:35.862798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959715+0000","last_change":"2024-03-06T07:30:31.952755+0000","last_active":"2024-03-06T07:30:31.959715+0000","last_peered":"2024-03-06T07:30:31.959715+0000","last_clean":"2024-03-06T07:30:31.959715+0000","last_became_active":"2024-03-06T07:30:31.952413+0000","last_became_peered":"2024-03-06T07:30:31.952413+0000","last_unstale":"2024-03-06T07:30:31.959715+0000","last_undegraded":"2024-03-06T07:30:31.959715+0000","last_fullsized":"2024-03-06T07:30:31.959715+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:27.188131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959983+0000","last_change":"2024-03-06T07:30:31.951631+0000","last_active":"2024-03-06T07:30:31.959983+0000","last_peered":"2024-03-06T07:30:31.959983+0000","last_clean":"2024-03-06T07:30:31.959983+0000","last_became_active":"2024-03-06T07:30:31.951358+0000","last_became_peered":"2024-03-06T07:30:31.951358+0000","last_unstale":"2024-03-06T07:30:31.959983+0000","last_undegraded":"2024-03-06T07:30:31.959983+0000","last_fullsized":"2024-03-06T07:30:31.959983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:55.224508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959290+0000","last_change":"2024-03-06T07:30:31.951322+0000","last_active":"2024-03-06T07:30:32.959290+0000","last_peered":"2024-03-06T07:30:32.959290+0000","last_clean":"2024-03-06T07:30:32.959290+0000","last_became_active":"2024-03-06T07:30:31.951031+0000","last_became_peered":"2024-03-06T07:30:31.951031+0000","last_unstale":"2024-03-06T07:30:32.959290+0000","last_undegraded":"2024-03-06T07:30:32.959290+0000","last_fullsized":"2024-03-06T07:30:32.959290+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:54.804220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.950906+0000","last_change":"2024-03-06T07:30:31.945498+0000","last_active":"2024-03-06T07:30:31.950906+0000","last_peered":"2024-03-06T07:30:31.950906+0000","last_clean":"2024-03-06T07:30:31.950906+0000","last_became_active":"2024-03-06T07:30:31.945175+0000","last_became_peered":"2024-03-06T07:30:31.945175+0000","last_unstale":"2024-03-06T07:30:31.950906+0000","last_undegraded":"2024-03-06T07:30:31.950906+0000","last_fullsized":"2024-03-06T07:30:31.950906+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:33.710446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970476+0000","last_change":"2024-03-06T07:30:31.959202+0000","last_active":"2024-03-06T07:30:31.970476+0000","last_peered":"2024-03-06T07:30:31.970476+0000","last_clean":"2024-03-06T07:30:31.970476+0000","last_became_active":"2024-03-06T07:30:31.950560+0000","last_became_peered":"2024-03-06T07:30:31.950560+0000","last_unstale":"2024-03-06T07:30:31.970476+0000","last_undegraded":"2024-03-06T07:30:31.970476+0000","last_fullsized":"2024-03-06T07:30:31.970476+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:29.489129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958504+0000","last_change":"2024-03-06T07:30:31.952895+0000","last_active":"2024-03-06T07:30:32.958504+0000","last_peered":"2024-03-06T07:30:32.958504+0000","last_clean":"2024-03-06T07:30:32.958504+0000","last_became_active":"2024-03-06T07:30:31.950563+0000","last_became_peered":"2024-03-06T07:30:31.950563+0000","last_unstale":"2024-03-06T07:30:32.958504+0000","last_undegraded":"2024-03-06T07:30:32.958504+0000","last_fullsized":"2024-03-06T07:30:32.958504+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:38.629962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970342+0000","last_change":"2024-03-06T07:30:31.957698+0000","last_active":"2024-03-06T07:30:31.970342+0000","last_peered":"2024-03-06T07:30:31.970342+0000","last_clean":"2024-03-06T07:30:31.970342+0000","last_became_active":"2024-03-06T07:30:31.956919+0000","last_became_peered":"2024-03-06T07:30:31.956919+0000","last_unstale":"2024-03-06T07:30:31.970342+0000","last_undegraded":"2024-03-06T07:30:31.970342+0000","last_fullsized":"2024-03-06T07:30:31.970342+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:05.494883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957408+0000","last_change":"2024-03-06T07:30:31.962125+0000","last_active":"2024-03-06T07:30:32.957408+0000","last_peered":"2024-03-06T07:30:32.957408+0000","last_clean":"2024-03-06T07:30:32.957408+0000","last_became_active":"2024-03-06T07:30:31.961857+0000","last_became_peered":"2024-03-06T07:30:31.961857+0000","last_unstale":"2024-03-06T07:30:32.957408+0000","last_undegraded":"2024-03-06T07:30:32.957408+0000","last_fullsized":"2024-03-06T07:30:32.957408+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:57.463832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.956512+0000","last_change":"2024-03-06T07:30:31.951432+0000","last_active":"2024-03-06T07:30:31.956512+0000","last_peered":"2024-03-06T07:30:31.956512+0000","last_clean":"2024-03-06T07:30:31.956512+0000","last_became_active":"2024-03-06T07:30:31.951124+0000","last_became_peered":"2024-03-06T07:30:31.951124+0000","last_unstale":"2024-03-06T07:30:31.956512+0000","last_undegraded":"2024-03-06T07:30:31.956512+0000","last_fullsized":"2024-03-06T07:30:31.956512+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:42.227552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964089+0000","last_change":"2024-03-06T07:30:31.957551+0000","last_active":"2024-03-06T07:30:31.964089+0000","last_peered":"2024-03-06T07:30:31.964089+0000","last_clean":"2024-03-06T07:30:31.964089+0000","last_became_active":"2024-03-06T07:30:31.957237+0000","last_became_peered":"2024-03-06T07:30:31.957237+0000","last_unstale":"2024-03-06T07:30:31.964089+0000","last_undegraded":"2024-03-06T07:30:31.964089+0000","last_fullsized":"2024-03-06T07:30:31.964089+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:29.766268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961752+0000","last_change":"2024-03-06T07:30:31.953339+0000","last_active":"2024-03-06T07:30:32.961752+0000","last_peered":"2024-03-06T07:30:32.961752+0000","last_clean":"2024-03-06T07:30:32.961752+0000","last_became_active":"2024-03-06T07:30:31.953019+0000","last_became_peered":"2024-03-06T07:30:31.953019+0000","last_unstale":"2024-03-06T07:30:32.961752+0000","last_undegraded":"2024-03-06T07:30:32.961752+0000","last_fullsized":"2024-03-06T07:30:32.961752+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:48.453243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.962962+0000","last_change":"2024-03-06T07:30:31.954458+0000","last_active":"2024-03-06T07:30:31.962962+0000","last_peered":"2024-03-06T07:30:31.962962+0000","last_clean":"2024-03-06T07:30:31.962962+0000","last_became_active":"2024-03-06T07:30:31.953639+0000","last_became_peered":"2024-03-06T07:30:31.953639+0000","last_unstale":"2024-03-06T07:30:31.962962+0000","last_undegraded":"2024-03-06T07:30:31.962962+0000","last_fullsized":"2024-03-06T07:30:31.962962+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:31.307116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.955395+0000","last_change":"2024-03-06T07:30:31.946986+0000","last_active":"2024-03-06T07:30:31.955395+0000","last_peered":"2024-03-06T07:30:31.955395+0000","last_clean":"2024-03-06T07:30:31.955395+0000","last_became_active":"2024-03-06T07:30:31.946682+0000","last_became_peered":"2024-03-06T07:30:31.946682+0000","last_unstale":"2024-03-06T07:30:31.955395+0000","last_undegraded":"2024-03-06T07:30:31.955395+0000","last_fullsized":"2024-03-06T07:30:31.955395+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:59.710493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.973701+0000","last_change":"2024-03-06T07:30:31.960089+0000","last_active":"2024-03-06T07:30:31.973701+0000","last_peered":"2024-03-06T07:30:31.973701+0000","last_clean":"2024-03-06T07:30:31.973701+0000","last_became_active":"2024-03-06T07:30:31.959808+0000","last_became_peered":"2024-03-06T07:30:31.959808+0000","last_unstale":"2024-03-06T07:30:31.973701+0000","last_undegraded":"2024-03-06T07:30:31.973701+0000","last_fullsized":"2024-03-06T07:30:31.973701+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:27.562274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957988+0000","last_change":"2024-03-06T07:30:31.953670+0000","last_active":"2024-03-06T07:30:32.957988+0000","last_peered":"2024-03-06T07:30:32.957988+0000","last_clean":"2024-03-06T07:30:32.957988+0000","last_became_active":"2024-03-06T07:30:31.952056+0000","last_became_peered":"2024-03-06T07:30:31.952056+0000","last_unstale":"2024-03-06T07:30:32.957988+0000","last_undegraded":"2024-03-06T07:30:32.957988+0000","last_fullsized":"2024-03-06T07:30:32.957988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:12.259535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.967783+0000","last_change":"2024-03-06T07:30:31.963413+0000","last_active":"2024-03-06T07:30:31.967783+0000","last_peered":"2024-03-06T07:30:31.967783+0000","last_clean":"2024-03-06T07:30:31.967783+0000","last_became_active":"2024-03-06T07:30:31.962593+0000","last_became_peered":"2024-03-06T07:30:31.962593+0000","last_unstale":"2024-03-06T07:30:31.967783+0000","last_undegraded":"2024-03-06T07:30:31.967783+0000","last_fullsized":"2024-03-06T07:30:31.967783+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:50.943167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.966913+0000","last_change":"2024-03-06T07:30:31.957159+0000","last_active":"2024-03-06T07:30:31.966913+0000","last_peered":"2024-03-06T07:30:31.966913+0000","last_clean":"2024-03-06T07:30:31.966913+0000","last_became_active":"2024-03-06T07:30:31.956852+0000","last_became_peered":"2024-03-06T07:30:31.956852+0000","last_unstale":"2024-03-06T07:30:31.966913+0000","last_undegraded":"2024-03-06T07:30:31.966913+0000","last_fullsized":"2024-03-06T07:30:31.966913+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:46.744237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959381+0000","last_change":"2024-03-06T07:30:31.953077+0000","last_active":"2024-03-06T07:30:31.959381+0000","last_peered":"2024-03-06T07:30:31.959381+0000","last_clean":"2024-03-06T07:30:31.959381+0000","last_became_active":"2024-03-06T07:30:31.952615+0000","last_became_peered":"2024-03-06T07:30:31.952615+0000","last_unstale":"2024-03-06T07:30:31.959381+0000","last_undegraded":"2024-03-06T07:30:31.959381+0000","last_fullsized":"2024-03-06T07:30:31.959381+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:10.166452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"54'4","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.976168+0000","last_change":"2024-03-06T07:30:30.932288+0000","last_active":"2024-03-06T07:30:31.976168+0000","last_peered":"2024-03-06T07:30:31.976168+0000","last_clean":"2024-03-06T07:30:31.976168+0000","last_became_active":"2024-03-06T07:30:30.931989+0000","last_became_peered":"2024-03-06T07:30:30.931989+0000","last_unstale":"2024-03-06T07:30:31.976168+0000","last_undegraded":"2024-03-06T07:30:31.976168+0000","last_fullsized":"2024-03-06T07:30:31.976168+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:44.920712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964213+0000","last_change":"2024-03-06T07:30:31.951364+0000","last_active":"2024-03-06T07:30:31.964213+0000","last_peered":"2024-03-06T07:30:31.964213+0000","last_clean":"2024-03-06T07:30:31.964213+0000","last_became_active":"2024-03-06T07:30:31.951085+0000","last_became_peered":"2024-03-06T07:30:31.951085+0000","last_unstale":"2024-03-06T07:30:31.964213+0000","last_undegraded":"2024-03-06T07:30:31.964213+0000","last_fullsized":"2024-03-06T07:30:31.964213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:41.604421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977248+0000","last_change":"2024-03-06T07:30:30.931196+0000","last_active":"2024-03-06T07:30:31.977248+0000","last_peered":"2024-03-06T07:30:31.977248+0000","last_clean":"2024-03-06T07:30:31.977248+0000","last_became_active":"2024-03-06T07:30:30.930810+0000","last_became_peered":"2024-03-06T07:30:30.930810+0000","last_unstale":"2024-03-06T07:30:31.977248+0000","last_undegraded":"2024-03-06T07:30:31.977248+0000","last_fullsized":"2024-03-06T07:30:31.977248+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:41.119688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970038+0000","last_change":"2024-03-06T07:30:31.965160+0000","last_active":"2024-03-06T07:30:31.970038+0000","last_peered":"2024-03-06T07:30:31.970038+0000","last_clean":"2024-03-06T07:30:31.970038+0000","last_became_active":"2024-03-06T07:30:31.964872+0000","last_became_peered":"2024-03-06T07:30:31.964872+0000","last_unstale":"2024-03-06T07:30:31.970038+0000","last_undegraded":"2024-03-06T07:30:31.970038+0000","last_fullsized":"2024-03-06T07:30:31.970038+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:30.787992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.975415+0000","last_change":"2024-03-06T07:30:30.925000+0000","last_active":"2024-03-06T07:30:31.975415+0000","last_peered":"2024-03-06T07:30:31.975415+0000","last_clean":"2024-03-06T07:30:31.975415+0000","last_became_active":"2024-03-06T07:30:30.924386+0000","last_became_peered":"2024-03-06T07:30:30.924386+0000","last_unstale":"2024-03-06T07:30:31.975415+0000","last_undegraded":"2024-03-06T07:30:31.975415+0000","last_fullsized":"2024-03-06T07:30:31.975415+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:15.803474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.963031+0000","last_change":"2024-03-06T07:30:31.954716+0000","last_active":"2024-03-06T07:30:31.963031+0000","last_peered":"2024-03-06T07:30:31.963031+0000","last_clean":"2024-03-06T07:30:31.963031+0000","last_became_active":"2024-03-06T07:30:31.954153+0000","last_became_peered":"2024-03-06T07:30:31.954153+0000","last_unstale":"2024-03-06T07:30:31.963031+0000","last_undegraded":"2024-03-06T07:30:31.963031+0000","last_fullsized":"2024-03-06T07:30:31.963031+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:48.453243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962541+0000","last_change":"2024-03-06T07:30:30.930436+0000","last_active":"2024-03-06T07:30:32.962541+0000","last_peered":"2024-03-06T07:30:32.962541+0000","last_clean":"2024-03-06T07:30:32.962541+0000","last_became_active":"2024-03-06T07:30:30.927208+0000","last_became_peered":"2024-03-06T07:30:30.927208+0000","last_unstale":"2024-03-06T07:30:32.962541+0000","last_undegraded":"2024-03-06T07:30:32.962541+0000","last_fullsized":"2024-03-06T07:30:32.962541+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:32.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.969896+0000","last_change":"2024-03-06T07:30:31.957530+0000","last_active":"2024-03-06T07:30:31.969896+0000","last_peered":"2024-03-06T07:30:31.969896+0000","last_clean":"2024-03-06T07:30:31.969896+0000","last_became_active":"2024-03-06T07:30:31.956496+0000","last_became_peered":"2024-03-06T07:30:31.956496+0000","last_unstale":"2024-03-06T07:30:31.969896+0000","last_undegraded":"2024-03-06T07:30:31.969896+0000","last_fullsized":"2024-03-06T07:30:31.969896+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:54.804220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960694+0000","last_change":"2024-03-06T07:30:30.923626+0000","last_active":"2024-03-06T07:30:32.960694+0000","last_peered":"2024-03-06T07:30:32.960694+0000","last_clean":"2024-03-06T07:30:32.960694+0000","last_became_active":"2024-03-06T07:30:30.923283+0000","last_became_peered":"2024-03-06T07:30:30.923283+0000","last_unstale":"2024-03-06T07:30:32.960694+0000","last_undegraded":"2024-03-06T07:30:32.960694+0000","last_fullsized":"2024-03-06T07:30:32.960694+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:18.090061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.960410+0000","last_change":"2024-03-06T07:30:31.948958+0000","last_active":"2024-03-06T07:30:31.960410+0000","last_peered":"2024-03-06T07:30:31.960410+0000","last_clean":"2024-03-06T07:30:31.960410+0000","last_became_active":"2024-03-06T07:30:31.948443+0000","last_became_peered":"2024-03-06T07:30:31.948443+0000","last_unstale":"2024-03-06T07:30:31.960410+0000","last_undegraded":"2024-03-06T07:30:31.960410+0000","last_fullsized":"2024-03-06T07:30:31.960410+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:30.346901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972133+0000","last_change":"2024-03-06T07:30:30.923998+0000","last_active":"2024-03-06T07:30:31.972133+0000","last_peered":"2024-03-06T07:30:31.972133+0000","last_clean":"2024-03-06T07:30:31.972133+0000","last_became_active":"2024-03-06T07:30:30.923701+0000","last_became_peered":"2024-03-06T07:30:30.923701+0000","last_unstale":"2024-03-06T07:30:31.972133+0000","last_undegraded":"2024-03-06T07:30:31.972133+0000","last_fullsized":"2024-03-06T07:30:31.972133+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:32.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.951029+0000","last_change":"2024-03-06T07:30:31.944334+0000","last_active":"2024-03-06T07:30:31.951029+0000","last_peered":"2024-03-06T07:30:31.951029+0000","last_clean":"2024-03-06T07:30:31.951029+0000","last_became_active":"2024-03-06T07:30:31.944050+0000","last_became_peered":"2024-03-06T07:30:31.944050+0000","last_unstale":"2024-03-06T07:30:31.951029+0000","last_undegraded":"2024-03-06T07:30:31.951029+0000","last_fullsized":"2024-03-06T07:30:31.951029+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:28.106557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962838+0000","last_change":"2024-03-06T07:30:30.927100+0000","last_active":"2024-03-06T07:30:32.962838+0000","last_peered":"2024-03-06T07:30:32.962838+0000","last_clean":"2024-03-06T07:30:32.962838+0000","last_became_active":"2024-03-06T07:30:30.925900+0000","last_became_peered":"2024-03-06T07:30:30.925900+0000","last_unstale":"2024-03-06T07:30:32.962838+0000","last_undegraded":"2024-03-06T07:30:32.962838+0000","last_fullsized":"2024-03-06T07:30:32.962838+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:49.142087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972912+0000","last_change":"2024-03-06T07:30:31.958940+0000","last_active":"2024-03-06T07:30:31.972912+0000","last_peered":"2024-03-06T07:30:31.972912+0000","last_clean":"2024-03-06T07:30:31.972912+0000","last_became_active":"2024-03-06T07:30:31.958306+0000","last_became_peered":"2024-03-06T07:30:31.958306+0000","last_unstale":"2024-03-06T07:30:31.972912+0000","last_undegraded":"2024-03-06T07:30:31.972912+0000","last_fullsized":"2024-03-06T07:30:31.972912+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:24.763133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961103+0000","last_change":"2024-03-06T07:30:30.919561+0000","last_active":"2024-03-06T07:30:32.961103+0000","last_peered":"2024-03-06T07:30:32.961103+0000","last_clean":"2024-03-06T07:30:32.961103+0000","last_became_active":"2024-03-06T07:30:30.918505+0000","last_became_peered":"2024-03-06T07:30:30.918505+0000","last_unstale":"2024-03-06T07:30:32.961103+0000","last_undegraded":"2024-03-06T07:30:32.961103+0000","last_fullsized":"2024-03-06T07:30:32.961103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:56.458418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964227+0000","last_change":"2024-03-06T07:30:31.964911+0000","last_active":"2024-03-06T07:30:32.964227+0000","last_peered":"2024-03-06T07:30:32.964227+0000","last_clean":"2024-03-06T07:30:32.964227+0000","last_became_active":"2024-03-06T07:30:31.964621+0000","last_became_peered":"2024-03-06T07:30:31.964621+0000","last_unstale":"2024-03-06T07:30:32.964227+0000","last_undegraded":"2024-03-06T07:30:32.964227+0000","last_fullsized":"2024-03-06T07:30:32.964227+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:03.199829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960036+0000","last_change":"2024-03-06T07:30:30.935100+0000","last_active":"2024-03-06T07:30:32.960036+0000","last_peered":"2024-03-06T07:30:32.960036+0000","last_clean":"2024-03-06T07:30:32.960036+0000","last_became_active":"2024-03-06T07:30:30.934888+0000","last_became_peered":"2024-03-06T07:30:30.934888+0000","last_unstale":"2024-03-06T07:30:32.960036+0000","last_undegraded":"2024-03-06T07:30:32.960036+0000","last_fullsized":"2024-03-06T07:30:32.960036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:51.419184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959791+0000","last_change":"2024-03-06T07:30:31.959935+0000","last_active":"2024-03-06T07:30:32.959791+0000","last_peered":"2024-03-06T07:30:32.959791+0000","last_clean":"2024-03-06T07:30:32.959791+0000","last_became_active":"2024-03-06T07:30:31.959601+0000","last_became_peered":"2024-03-06T07:30:31.959601+0000","last_unstale":"2024-03-06T07:30:32.959791+0000","last_undegraded":"2024-03-06T07:30:32.959791+0000","last_fullsized":"2024-03-06T07:30:32.959791+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:14.442253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.933587+0000","last_change":"2024-03-06T07:30:30.926935+0000","last_active":"2024-03-06T07:30:31.933587+0000","last_peered":"2024-03-06T07:30:31.933587+0000","last_clean":"2024-03-06T07:30:31.933587+0000","last_became_active":"2024-03-06T07:30:30.926637+0000","last_became_peered":"2024-03-06T07:30:30.926637+0000","last_unstale":"2024-03-06T07:30:31.933587+0000","last_undegraded":"2024-03-06T07:30:31.933587+0000","last_fullsized":"2024-03-06T07:30:31.933587+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:11.248525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964356+0000","last_change":"2024-03-06T07:30:31.955590+0000","last_active":"2024-03-06T07:30:31.964356+0000","last_peered":"2024-03-06T07:30:31.964356+0000","last_clean":"2024-03-06T07:30:31.964356+0000","last_became_active":"2024-03-06T07:30:31.955144+0000","last_became_peered":"2024-03-06T07:30:31.955144+0000","last_unstale":"2024-03-06T07:30:31.964356+0000","last_undegraded":"2024-03-06T07:30:31.964356+0000","last_fullsized":"2024-03-06T07:30:31.964356+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:57.469428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959020+0000","last_change":"2024-03-06T07:30:30.932132+0000","last_active":"2024-03-06T07:30:32.959020+0000","last_peered":"2024-03-06T07:30:32.959020+0000","last_clean":"2024-03-06T07:30:32.959020+0000","last_became_active":"2024-03-06T07:30:30.931384+0000","last_became_peered":"2024-03-06T07:30:30.931384+0000","last_unstale":"2024-03-06T07:30:32.959020+0000","last_undegraded":"2024-03-06T07:30:32.959020+0000","last_fullsized":"2024-03-06T07:30:32.959020+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:02.188819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958729+0000","last_change":"2024-03-06T07:30:31.963602+0000","last_active":"2024-03-06T07:30:32.958729+0000","last_peered":"2024-03-06T07:30:32.958729+0000","last_clean":"2024-03-06T07:30:32.958729+0000","last_became_active":"2024-03-06T07:30:31.962944+0000","last_became_peered":"2024-03-06T07:30:31.962944+0000","last_unstale":"2024-03-06T07:30:32.958729+0000","last_undegraded":"2024-03-06T07:30:32.958729+0000","last_fullsized":"2024-03-06T07:30:32.958729+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:55.224508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"54'6","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963632+0000","last_change":"2024-03-06T07:30:30.924534+0000","last_active":"2024-03-06T07:30:32.963632+0000","last_peered":"2024-03-06T07:30:32.963632+0000","last_clean":"2024-03-06T07:30:32.963632+0000","last_became_active":"2024-03-06T07:30:30.924231+0000","last_became_peered":"2024-03-06T07:30:30.924231+0000","last_unstale":"2024-03-06T07:30:32.963632+0000","last_undegraded":"2024-03-06T07:30:32.963632+0000","last_fullsized":"2024-03-06T07:30:32.963632+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.838789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963222+0000","last_change":"2024-03-06T07:30:31.957577+0000","last_active":"2024-03-06T07:30:32.963222+0000","last_peered":"2024-03-06T07:30:32.963222+0000","last_clean":"2024-03-06T07:30:32.963222+0000","last_became_active":"2024-03-06T07:30:31.957255+0000","last_became_peered":"2024-03-06T07:30:31.957255+0000","last_unstale":"2024-03-06T07:30:32.963222+0000","last_undegraded":"2024-03-06T07:30:32.963222+0000","last_fullsized":"2024-03-06T07:30:32.963222+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:45.674954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977778+0000","last_change":"2024-03-06T07:30:30.927655+0000","last_active":"2024-03-06T07:30:31.977778+0000","last_peered":"2024-03-06T07:30:31.977778+0000","last_clean":"2024-03-06T07:30:31.977778+0000","last_became_active":"2024-03-06T07:30:30.927399+0000","last_became_peered":"2024-03-06T07:30:30.927399+0000","last_unstale":"2024-03-06T07:30:31.977778+0000","last_undegraded":"2024-03-06T07:30:31.977778+0000","last_fullsized":"2024-03-06T07:30:31.977778+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:44.663944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.958918+0000","last_change":"2024-03-06T07:30:31.943368+0000","last_active":"2024-03-06T07:30:32.958918+0000","last_peered":"2024-03-06T07:30:32.958918+0000","last_clean":"2024-03-06T07:30:32.958918+0000","last_became_active":"2024-03-06T07:30:31.943076+0000","last_became_peered":"2024-03-06T07:30:31.943076+0000","last_unstale":"2024-03-06T07:30:32.958918+0000","last_undegraded":"2024-03-06T07:30:32.958918+0000","last_fullsized":"2024-03-06T07:30:32.958918+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:22.621712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.922041+0000","last_change":"2024-03-06T07:30:30.928717+0000","last_active":"2024-03-06T07:30:31.922041+0000","last_peered":"2024-03-06T07:30:31.922041+0000","last_clean":"2024-03-06T07:30:31.922041+0000","last_became_active":"2024-03-06T07:30:30.928398+0000","last_became_peered":"2024-03-06T07:30:30.928398+0000","last_unstale":"2024-03-06T07:30:31.922041+0000","last_undegraded":"2024-03-06T07:30:31.922041+0000","last_fullsized":"2024-03-06T07:30:31.922041+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:29.354693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.960869+0000","last_change":"2024-03-06T07:30:31.959404+0000","last_active":"2024-03-06T07:30:32.960869+0000","last_peered":"2024-03-06T07:30:32.960869+0000","last_clean":"2024-03-06T07:30:32.960869+0000","last_became_active":"2024-03-06T07:30:31.958438+0000","last_became_peered":"2024-03-06T07:30:31.958438+0000","last_unstale":"2024-03-06T07:30:32.960869+0000","last_undegraded":"2024-03-06T07:30:32.960869+0000","last_fullsized":"2024-03-06T07:30:32.960869+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:45.931722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962950+0000","last_change":"2024-03-06T07:30:30.924253+0000","last_active":"2024-03-06T07:30:32.962950+0000","last_peered":"2024-03-06T07:30:32.962950+0000","last_clean":"2024-03-06T07:30:32.962950+0000","last_became_active":"2024-03-06T07:30:30.923993+0000","last_became_peered":"2024-03-06T07:30:30.923993+0000","last_unstale":"2024-03-06T07:30:32.962950+0000","last_undegraded":"2024-03-06T07:30:32.962950+0000","last_fullsized":"2024-03-06T07:30:32.962950+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:06.831913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957627+0000","last_change":"2024-03-06T07:30:31.963332+0000","last_active":"2024-03-06T07:30:32.957627+0000","last_peered":"2024-03-06T07:30:32.957627+0000","last_clean":"2024-03-06T07:30:32.957627+0000","last_became_active":"2024-03-06T07:30:31.962957+0000","last_became_peered":"2024-03-06T07:30:31.962957+0000","last_unstale":"2024-03-06T07:30:32.957627+0000","last_undegraded":"2024-03-06T07:30:32.957627+0000","last_fullsized":"2024-03-06T07:30:32.957627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:42.130697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957138+0000","last_change":"2024-03-06T07:30:30.932567+0000","last_active":"2024-03-06T07:30:32.957138+0000","last_peered":"2024-03-06T07:30:32.957138+0000","last_clean":"2024-03-06T07:30:32.957138+0000","last_became_active":"2024-03-06T07:30:30.932240+0000","last_became_peered":"2024-03-06T07:30:30.932240+0000","last_unstale":"2024-03-06T07:30:32.957138+0000","last_undegraded":"2024-03-06T07:30:32.957138+0000","last_fullsized":"2024-03-06T07:30:32.957138+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:34.851788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959683+0000","last_change":"2024-03-06T07:30:31.957151+0000","last_active":"2024-03-06T07:30:32.959683+0000","last_peered":"2024-03-06T07:30:32.959683+0000","last_clean":"2024-03-06T07:30:32.959683+0000","last_became_active":"2024-03-06T07:30:31.956339+0000","last_became_peered":"2024-03-06T07:30:31.956339+0000","last_unstale":"2024-03-06T07:30:32.959683+0000","last_undegraded":"2024-03-06T07:30:32.959683+0000","last_fullsized":"2024-03-06T07:30:32.959683+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:56.746980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977305+0000","last_change":"2024-03-06T07:30:30.928024+0000","last_active":"2024-03-06T07:30:31.977305+0000","last_peered":"2024-03-06T07:30:31.977305+0000","last_clean":"2024-03-06T07:30:31.977305+0000","last_became_active":"2024-03-06T07:30:30.925279+0000","last_became_peered":"2024-03-06T07:30:30.925279+0000","last_unstale":"2024-03-06T07:30:31.977305+0000","last_undegraded":"2024-03-06T07:30:31.977305+0000","last_fullsized":"2024-03-06T07:30:31.977305+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:37.618952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964465+0000","last_change":"2024-03-06T07:30:31.969822+0000","last_active":"2024-03-06T07:30:32.964465+0000","last_peered":"2024-03-06T07:30:32.964465+0000","last_clean":"2024-03-06T07:30:32.964465+0000","last_became_active":"2024-03-06T07:30:31.969243+0000","last_became_peered":"2024-03-06T07:30:31.969243+0000","last_unstale":"2024-03-06T07:30:32.964465+0000","last_undegraded":"2024-03-06T07:30:32.964465+0000","last_fullsized":"2024-03-06T07:30:32.964465+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:12.259535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.971467+0000","last_change":"2024-03-06T07:30:30.928639+0000","last_active":"2024-03-06T07:30:31.971467+0000","last_peered":"2024-03-06T07:30:31.971467+0000","last_clean":"2024-03-06T07:30:31.971467+0000","last_became_active":"2024-03-06T07:30:30.928398+0000","last_became_peered":"2024-03-06T07:30:30.928398+0000","last_unstale":"2024-03-06T07:30:31.971467+0000","last_undegraded":"2024-03-06T07:30:31.971467+0000","last_fullsized":"2024-03-06T07:30:31.971467+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:34.851788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.973603+0000","last_change":"2024-03-06T07:30:31.960234+0000","last_active":"2024-03-06T07:30:31.973603+0000","last_peered":"2024-03-06T07:30:31.973603+0000","last_clean":"2024-03-06T07:30:31.973603+0000","last_became_active":"2024-03-06T07:30:31.959337+0000","last_became_peered":"2024-03-06T07:30:31.959337+0000","last_unstale":"2024-03-06T07:30:31.973603+0000","last_undegraded":"2024-03-06T07:30:31.973603+0000","last_fullsized":"2024-03-06T07:30:31.973603+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:38.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963932+0000","last_change":"2024-03-06T07:30:30.928134+0000","last_active":"2024-03-06T07:30:32.963932+0000","last_peered":"2024-03-06T07:30:32.963932+0000","last_clean":"2024-03-06T07:30:32.963932+0000","last_became_active":"2024-03-06T07:30:30.927804+0000","last_became_peered":"2024-03-06T07:30:30.927804+0000","last_unstale":"2024-03-06T07:30:32.963932+0000","last_undegraded":"2024-03-06T07:30:32.963932+0000","last_fullsized":"2024-03-06T07:30:32.963932+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:34.851788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959006+0000","last_change":"2024-03-06T07:30:31.957489+0000","last_active":"2024-03-06T07:30:32.959006+0000","last_peered":"2024-03-06T07:30:32.959006+0000","last_clean":"2024-03-06T07:30:32.959006+0000","last_became_active":"2024-03-06T07:30:31.957216+0000","last_became_peered":"2024-03-06T07:30:31.957216+0000","last_unstale":"2024-03-06T07:30:32.959006+0000","last_undegraded":"2024-03-06T07:30:32.959006+0000","last_fullsized":"2024-03-06T07:30:32.959006+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:47.388772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.940720+0000","last_change":"2024-03-06T07:30:30.924150+0000","last_active":"2024-03-06T07:30:31.940720+0000","last_peered":"2024-03-06T07:30:31.940720+0000","last_clean":"2024-03-06T07:30:31.940720+0000","last_became_active":"2024-03-06T07:30:30.923745+0000","last_became_peered":"2024-03-06T07:30:30.923745+0000","last_unstale":"2024-03-06T07:30:31.940720+0000","last_undegraded":"2024-03-06T07:30:31.940720+0000","last_fullsized":"2024-03-06T07:30:31.940720+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:02.188819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.963433+0000","last_change":"2024-03-06T07:30:31.956449+0000","last_active":"2024-03-06T07:30:32.963433+0000","last_peered":"2024-03-06T07:30:32.963433+0000","last_clean":"2024-03-06T07:30:32.963433+0000","last_became_active":"2024-03-06T07:30:31.956185+0000","last_became_peered":"2024-03-06T07:30:31.956185+0000","last_unstale":"2024-03-06T07:30:32.963433+0000","last_undegraded":"2024-03-06T07:30:32.963433+0000","last_fullsized":"2024-03-06T07:30:32.963433+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:57.463832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.977698+0000","last_change":"2024-03-06T07:30:30.918948+0000","last_active":"2024-03-06T07:30:31.977698+0000","last_peered":"2024-03-06T07:30:31.977698+0000","last_clean":"2024-03-06T07:30:31.977698+0000","last_became_active":"2024-03-06T07:30:30.918671+0000","last_became_peered":"2024-03-06T07:30:30.918671+0000","last_unstale":"2024-03-06T07:30:31.977698+0000","last_undegraded":"2024-03-06T07:30:31.977698+0000","last_fullsized":"2024-03-06T07:30:31.977698+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:56.452822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.951641+0000","last_change":"2024-03-06T07:30:31.944826+0000","last_active":"2024-03-06T07:30:31.951641+0000","last_peered":"2024-03-06T07:30:31.951641+0000","last_clean":"2024-03-06T07:30:31.951641+0000","last_became_active":"2024-03-06T07:30:31.944523+0000","last_became_peered":"2024-03-06T07:30:31.944523+0000","last_unstale":"2024-03-06T07:30:31.951641+0000","last_undegraded":"2024-03-06T07:30:31.951641+0000","last_fullsized":"2024-03-06T07:30:31.951641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:02:27.274064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.938608+0000","last_change":"2024-03-06T07:30:30.923028+0000","last_active":"2024-03-06T07:30:31.938608+0000","last_peered":"2024-03-06T07:30:31.938608+0000","last_clean":"2024-03-06T07:30:31.938608+0000","last_became_active":"2024-03-06T07:30:30.922730+0000","last_became_peered":"2024-03-06T07:30:30.922730+0000","last_unstale":"2024-03-06T07:30:31.938608+0000","last_undegraded":"2024-03-06T07:30:31.938608+0000","last_fullsized":"2024-03-06T07:30:31.938608+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:16.265557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959460+0000","last_change":"2024-03-06T07:30:31.952694+0000","last_active":"2024-03-06T07:30:31.959460+0000","last_peered":"2024-03-06T07:30:31.959460+0000","last_clean":"2024-03-06T07:30:31.959460+0000","last_became_active":"2024-03-06T07:30:31.952438+0000","last_became_peered":"2024-03-06T07:30:31.952438+0000","last_unstale":"2024-03-06T07:30:31.959460+0000","last_undegraded":"2024-03-06T07:30:31.959460+0000","last_fullsized":"2024-03-06T07:30:31.959460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:44.915416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961419+0000","last_change":"2024-03-06T07:30:30.919320+0000","last_active":"2024-03-06T07:30:32.961419+0000","last_peered":"2024-03-06T07:30:32.961419+0000","last_clean":"2024-03-06T07:30:32.961419+0000","last_became_active":"2024-03-06T07:30:30.919011+0000","last_became_peered":"2024-03-06T07:30:30.919011+0000","last_unstale":"2024-03-06T07:30:32.961419+0000","last_undegraded":"2024-03-06T07:30:32.961419+0000","last_fullsized":"2024-03-06T07:30:32.961419+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:44.920712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964300+0000","last_change":"2024-03-06T07:30:31.947473+0000","last_active":"2024-03-06T07:30:32.964300+0000","last_peered":"2024-03-06T07:30:32.964300+0000","last_clean":"2024-03-06T07:30:32.964300+0000","last_became_active":"2024-03-06T07:30:31.947192+0000","last_became_peered":"2024-03-06T07:30:31.947192+0000","last_unstale":"2024-03-06T07:30:32.964300+0000","last_undegraded":"2024-03-06T07:30:32.964300+0000","last_fullsized":"2024-03-06T07:30:32.964300+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:42.130697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"54'2","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.974356+0000","last_change":"2024-03-06T07:30:30.917523+0000","last_active":"2024-03-06T07:30:31.974356+0000","last_peered":"2024-03-06T07:30:31.974356+0000","last_clean":"2024-03-06T07:30:31.974356+0000","last_became_active":"2024-03-06T07:30:30.917235+0000","last_became_peered":"2024-03-06T07:30:30.917235+0000","last_unstale":"2024-03-06T07:30:31.974356+0000","last_undegraded":"2024-03-06T07:30:31.974356+0000","last_fullsized":"2024-03-06T07:30:31.974356+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:27.095547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.964515+0000","last_change":"2024-03-06T07:30:31.945698+0000","last_active":"2024-03-06T07:30:32.964515+0000","last_peered":"2024-03-06T07:30:32.964515+0000","last_clean":"2024-03-06T07:30:32.964515+0000","last_became_active":"2024-03-06T07:30:31.943698+0000","last_became_peered":"2024-03-06T07:30:31.943698+0000","last_unstale":"2024-03-06T07:30:32.964515+0000","last_undegraded":"2024-03-06T07:30:32.964515+0000","last_fullsized":"2024-03-06T07:30:32.964515+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:38.629962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"54'1","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.955903+0000","last_change":"2024-03-06T07:30:30.930515+0000","last_active":"2024-03-06T07:30:32.955903+0000","last_peered":"2024-03-06T07:30:32.955903+0000","last_clean":"2024-03-06T07:30:32.955903+0000","last_became_active":"2024-03-06T07:30:30.930223+0000","last_became_peered":"2024-03-06T07:30:30.930223+0000","last_unstale":"2024-03-06T07:30:32.955903+0000","last_undegraded":"2024-03-06T07:30:32.955903+0000","last_fullsized":"2024-03-06T07:30:32.955903+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:51.419184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.964329+0000","last_change":"2024-03-06T07:30:31.955384+0000","last_active":"2024-03-06T07:30:31.964329+0000","last_peered":"2024-03-06T07:30:31.964329+0000","last_clean":"2024-03-06T07:30:31.964329+0000","last_became_active":"2024-03-06T07:30:31.954792+0000","last_became_peered":"2024-03-06T07:30:31.954792+0000","last_unstale":"2024-03-06T07:30:31.964329+0000","last_undegraded":"2024-03-06T07:30:31.964329+0000","last_fullsized":"2024-03-06T07:30:31.964329+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:42.227552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.921804+0000","last_change":"2024-03-06T07:30:30.915129+0000","last_active":"2024-03-06T07:30:31.921804+0000","last_peered":"2024-03-06T07:30:31.921804+0000","last_clean":"2024-03-06T07:30:31.921804+0000","last_became_active":"2024-03-06T07:30:30.914662+0000","last_became_peered":"2024-03-06T07:30:30.914662+0000","last_unstale":"2024-03-06T07:30:31.921804+0000","last_undegraded":"2024-03-06T07:30:31.921804+0000","last_fullsized":"2024-03-06T07:30:31.921804+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:06.451621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.953852+0000","last_change":"2024-03-06T07:30:31.949441+0000","last_active":"2024-03-06T07:30:31.953852+0000","last_peered":"2024-03-06T07:30:31.953852+0000","last_clean":"2024-03-06T07:30:31.953852+0000","last_became_active":"2024-03-06T07:30:31.949160+0000","last_became_peered":"2024-03-06T07:30:31.949160+0000","last_unstale":"2024-03-06T07:30:31.953852+0000","last_undegraded":"2024-03-06T07:30:31.953852+0000","last_fullsized":"2024-03-06T07:30:31.953852+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:31.877967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.962831+0000","last_change":"2024-03-06T07:30:30.926512+0000","last_active":"2024-03-06T07:30:32.962831+0000","last_peered":"2024-03-06T07:30:32.962831+0000","last_clean":"2024-03-06T07:30:32.962831+0000","last_became_active":"2024-03-06T07:30:30.926186+0000","last_became_peered":"2024-03-06T07:30:30.926186+0000","last_unstale":"2024-03-06T07:30:32.962831+0000","last_undegraded":"2024-03-06T07:30:32.962831+0000","last_fullsized":"2024-03-06T07:30:32.962831+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:51.419184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.970362+0000","last_change":"2024-03-06T07:30:31.957935+0000","last_active":"2024-03-06T07:30:31.970362+0000","last_peered":"2024-03-06T07:30:31.970362+0000","last_clean":"2024-03-06T07:30:31.970362+0000","last_became_active":"2024-03-06T07:30:31.957258+0000","last_became_peered":"2024-03-06T07:30:31.957258+0000","last_unstale":"2024-03-06T07:30:31.970362+0000","last_undegraded":"2024-03-06T07:30:31.970362+0000","last_fullsized":"2024-03-06T07:30:31.970362+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:56.746980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.938489+0000","last_change":"2024-03-06T07:30:30.927710+0000","last_active":"2024-03-06T07:30:31.938489+0000","last_peered":"2024-03-06T07:30:31.938489+0000","last_clean":"2024-03-06T07:30:31.938489+0000","last_became_active":"2024-03-06T07:30:30.927410+0000","last_became_peered":"2024-03-06T07:30:30.927410+0000","last_unstale":"2024-03-06T07:30:31.938489+0000","last_undegraded":"2024-03-06T07:30:31.938489+0000","last_fullsized":"2024-03-06T07:30:31.938489+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:08.162321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.959310+0000","last_change":"2024-03-06T07:30:31.953994+0000","last_active":"2024-03-06T07:30:32.959310+0000","last_peered":"2024-03-06T07:30:32.959310+0000","last_clean":"2024-03-06T07:30:32.959310+0000","last_became_active":"2024-03-06T07:30:31.953659+0000","last_became_peered":"2024-03-06T07:30:31.953659+0000","last_unstale":"2024-03-06T07:30:32.959310+0000","last_undegraded":"2024-03-06T07:30:32.959310+0000","last_fullsized":"2024-03-06T07:30:32.959310+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:05.494883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.961076+0000","last_change":"2024-03-06T07:30:30.923679+0000","last_active":"2024-03-06T07:30:32.961076+0000","last_peered":"2024-03-06T07:30:32.961076+0000","last_clean":"2024-03-06T07:30:32.961076+0000","last_became_active":"2024-03-06T07:30:30.923350+0000","last_became_peered":"2024-03-06T07:30:30.923350+0000","last_unstale":"2024-03-06T07:30:32.961076+0000","last_undegraded":"2024-03-06T07:30:32.961076+0000","last_fullsized":"2024-03-06T07:30:32.961076+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:11.039987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.959264+0000","last_change":"2024-03-06T07:30:31.942901+0000","last_active":"2024-03-06T07:30:31.959264+0000","last_peered":"2024-03-06T07:30:31.959264+0000","last_clean":"2024-03-06T07:30:31.959264+0000","last_became_active":"2024-03-06T07:30:31.942606+0000","last_became_peered":"2024-03-06T07:30:31.942606+0000","last_unstale":"2024-03-06T07:30:31.959264+0000","last_undegraded":"2024-03-06T07:30:31.959264+0000","last_fullsized":"2024-03-06T07:30:31.959264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:30.365703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.938752+0000","last_change":"2024-03-06T07:30:30.921904+0000","last_active":"2024-03-06T07:30:31.938752+0000","last_peered":"2024-03-06T07:30:31.938752+0000","last_clean":"2024-03-06T07:30:31.938752+0000","last_became_active":"2024-03-06T07:30:30.921679+0000","last_became_peered":"2024-03-06T07:30:30.921679+0000","last_unstale":"2024-03-06T07:30:31.938752+0000","last_undegraded":"2024-03-06T07:30:31.938752+0000","last_fullsized":"2024-03-06T07:30:31.938752+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:51.138949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.972927+0000","last_change":"2024-03-06T07:30:31.959158+0000","last_active":"2024-03-06T07:30:31.972927+0000","last_peered":"2024-03-06T07:30:31.972927+0000","last_clean":"2024-03-06T07:30:31.972927+0000","last_became_active":"2024-03-06T07:30:31.958637+0000","last_became_peered":"2024-03-06T07:30:31.958637+0000","last_unstale":"2024-03-06T07:30:31.972927+0000","last_undegraded":"2024-03-06T07:30:31.972927+0000","last_fullsized":"2024-03-06T07:30:31.972927+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:33.705056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.929793+0000","last_change":"2024-03-06T07:30:30.919539+0000","last_active":"2024-03-06T07:30:31.929793+0000","last_peered":"2024-03-06T07:30:31.929793+0000","last_clean":"2024-03-06T07:30:31.929793+0000","last_became_active":"2024-03-06T07:30:30.919273+0000","last_became_peered":"2024-03-06T07:30:30.919273+0000","last_unstale":"2024-03-06T07:30:31.929793+0000","last_undegraded":"2024-03-06T07:30:31.929793+0000","last_fullsized":"2024-03-06T07:30:31.929793+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:29.776982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.961219+0000","last_change":"2024-03-06T07:30:31.953890+0000","last_active":"2024-03-06T07:30:31.961219+0000","last_peered":"2024-03-06T07:30:31.961219+0000","last_clean":"2024-03-06T07:30:31.961219+0000","last_became_active":"2024-03-06T07:30:31.953251+0000","last_became_peered":"2024-03-06T07:30:31.953251+0000","last_unstale":"2024-03-06T07:30:31.961219+0000","last_undegraded":"2024-03-06T07:30:31.961219+0000","last_fullsized":"2024-03-06T07:30:31.961219+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:30.884847+0000","last_clean_scrub_stamp":"2024-03-06T07:30:30.884847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:36:22.621712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:30:31.921597+0000","last_change":"2024-03-06T07:30:30.930617+0000","last_active":"2024-03-06T07:30:31.921597+0000","last_peered":"2024-03-06T07:30:31.921597+0000","last_clean":"2024-03-06T07:30:31.921597+0000","last_became_active":"2024-03-06T07:30:30.930166+0000","last_became_peered":"2024-03-06T07:30:30.930166+0000","last_unstale":"2024-03-06T07:30:31.921597+0000","last_undegraded":"2024-03-06T07:30:31.921597+0000","last_fullsized":"2024-03-06T07:30:31.921597+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:29.873837+0000","last_clean_scrub_stamp":"2024-03-06T07:30:29.873837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:51.138949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":233,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:30:32.957295+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:30:32.957295+0000","last_peered":"2024-03-06T07:30:32.957295+0000","last_clean":"2024-03-06T07:30:32.957295+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:30:32.957295+0000","last_undegraded":"2024-03-06T07:30:32.957295+0000","last_fullsized":"2024-03-06T07:30:32.957295+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":1249280,"data_stored":857390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":266862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":267514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":659456,"data_stored":268474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":266918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":655360,"data_stored":266918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":651264,"data_stored":266952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1257472,"data_stored":859424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:36.887 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:36.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:37.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.141+0000 7f71ea600640 1 Processor -- start 2024-03-06T07:30:37.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.141+0000 7f71ea600640 1 -- start start 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.144+0000 7f71ea600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f71e415dde0 0x7f71e4161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.144+0000 7f71ea600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f71e406cbf0 0x7f71e406d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.144+0000 7f71ea600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4162300 con 0x7f71e406afc0 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71ea600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e41625e0 con 0x7f71e406cbf0 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71ea600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e41628c0 con 0x7f71e415dde0 2024-03-06T07:30:37.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e9c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f71e406afc0 0x7f71e40ae060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42086/0 (socket says 172.21.15.114:42086) 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e9c00640 1 -- 172.21.15.114:0/3070256385 learned_addr learned my addr 172.21.15.114:0/3070256385 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e3e00640 1 --2- 172.21.15.114:0/3070256385 >> v2:172.21.15.114:3300/0 conn(0x7f71e406cbf0 0x7f71e406d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e9200640 1 --2- 172.21.15.114:0/3070256385 >> v2:172.21.15.177:3300/0 conn(0x7f71e415dde0 0x7f71e4161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e3400640 1 -- 172.21.15.114:0/3070256385 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171633532 0 0) 0x7f71e4162300 con 0x7f71e406afc0 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e3e00640 1 -- 172.21.15.114:0/3070256385 >> v2:172.21.15.177:3300/0 conn(0x7f71e415dde0 msgr2=0x7f71e4161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e3e00640 1 --2- 172.21.15.114:0/3070256385 >> v2:172.21.15.177:3300/0 conn(0x7f71e415dde0 0x7f71e4161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e3e00640 1 -- 172.21.15.114:0/3070256385 >> v1:172.21.15.114:6789/0 conn(0x7f71e406afc0 legacy=0x7f71e40ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.145+0000 7f71e3e00640 1 -- 172.21.15.114:0/3070256385 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e40ad860 con 0x7f71e406cbf0 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.147+0000 7f71e3e00640 1 --2- 172.21.15.114:0/3070256385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e406cbf0 0x7f71e406d6e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f71e406d400 tx=0x7f71d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd8d9af241429dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.147+0000 7f71e3400640 1 -- 172.21.15.114:0/3070256385 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d00315a0 con 0x7f71e406cbf0 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.147+0000 7f71e3400640 1 -- 172.21.15.114:0/3070256385 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d0037070 con 0x7f71e406cbf0 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.147+0000 7f71e3400640 1 -- 172.21.15.114:0/3070256385 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0031960 con 0x7f71e406cbf0 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.147+0000 7f71ea600640 1 -- 172.21.15.114:0/3070256385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e406cbf0 msgr2=0x7f71e406d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.147+0000 7f71ea600640 1 --2- 172.21.15.114:0/3070256385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e406cbf0 0x7f71e406d6e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f71e406d400 tx=0x7f71d002b840 comp rx=0 tx=0).stop 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.148+0000 7f71ea600640 1 -- 172.21.15.114:0/3070256385 shutdown_connections 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.148+0000 7f71ea600640 1 --2- 172.21.15.114:0/3070256385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e406cbf0 0x7f71e406d6e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.148+0000 7f71ea600640 1 --2- 172.21.15.114:0/3070256385 >> v2:172.21.15.177:3300/0 conn(0x7f71e415dde0 0x7f71e4161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.148+0000 7f71ea600640 1 -- 172.21.15.114:0/3070256385 >> 172.21.15.114:0/3070256385 conn(0x7f71e406c0d0 msgr2=0x7f71e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.148+0000 7f71ea600640 1 -- 172.21.15.114:0/3070256385 shutdown_connections 2024-03-06T07:30:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.148+0000 7f71ea600640 1 -- 172.21.15.114:0/3070256385 wait complete. 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.149+0000 7f71ea600640 1 Processor -- start 2024-03-06T07:30:37.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.149+0000 7f71ea600640 1 -- start start 2024-03-06T07:30:37.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.150+0000 7f71ea600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f71e406afc0 0x7f71e4146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.150+0000 7f71ea600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71e406cbf0 0x7f71e4157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.150+0000 7f71ea600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e415dde0 0x7f71e40b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.150+0000 7f71ea600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e406dc20 con 0x7f71e415dde0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.150+0000 7f71ea600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4162010 con 0x7f71e406cbf0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.150+0000 7f71ea600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e406dda0 con 0x7f71e406afc0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e415dde0 0x7f71e40b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e415dde0 0x7f71e40b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39982/0 (socket says 172.21.15.114:39982) 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 -- 172.21.15.114:0/1594435782 learned_addr learned my addr 172.21.15.114:0/1594435782 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9200640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f71e406afc0 0x7f71e4146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e3e00640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71e406cbf0 0x7f71e4157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 -- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f71e406afc0 msgr2=0x7f71e4146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f71e406afc0 0x7f71e4146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 -- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71e406cbf0 msgr2=0x7f71e4157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71e406cbf0 0x7f71e4157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 -- 172.21.15.114:0/1594435782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e41f6c80 con 0x7f71e415dde0 2024-03-06T07:30:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.151+0000 7f71e9c00640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e415dde0 0x7f71e40b1630 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c7d0 tx=0x7f71dc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.152+0000 7f71e0c00640 1 -- 172.21.15.114:0/1594435782 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc018070 con 0x7f71e415dde0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.152+0000 7f71e0c00640 1 -- 172.21.15.114:0/1594435782 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc00a640 con 0x7f71e415dde0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.152+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e41f6f60 con 0x7f71e415dde0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.152+0000 7f71e0c00640 1 -- 172.21.15.114:0/1594435782 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc00aa70 con 0x7f71e415dde0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.152+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41f7520 con 0x7f71e415dde0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.153+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e4075450 con 0x7f71e415dde0 2024-03-06T07:30:37.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.154+0000 7f71e0c00640 1 -- 172.21.15.114:0/1594435782 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71dc01b070 con 0x7f71e415dde0 2024-03-06T07:30:37.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.155+0000 7f71e0c00640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f71c406b4d0 0x7f71c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.155+0000 7f71e0c00640 1 -- 172.21.15.114:0/1594435782 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71dc089820 con 0x7f71e415dde0 2024-03-06T07:30:37.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.155+0000 7f71e9200640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f71c406b4d0 0x7f71c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.156+0000 7f71e9200640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f71c406b4d0 0x7f71c406d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006fd0 tx=0x7f71d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.159+0000 7f71e0c00640 1 -- 172.21.15.114:0/1594435782 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71dc0586a0 con 0x7f71e415dde0 2024-03-06T07:30:37.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.550+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f71e406bca0 con 0x7f71e415dde0 2024-03-06T07:30:37.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.553+0000 7f71e0c00640 1 -- 172.21.15.114:0/1594435782 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f71dc057fa0 con 0x7f71e415dde0 2024-03-06T07:30:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:30:37.556 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:30:37.556 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:30:37.556 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:30:37.556 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:37.556 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:30:37.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.555+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f71c406b4d0 msgr2=0x7f71c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.555+0000 7f71ea600640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f71c406b4d0 0x7f71c406d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f71d8006fd0 tx=0x7f71d8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e415dde0 msgr2=0x7f71e40b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e415dde0 0x7f71e40b1630 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c7d0 tx=0x7f71dc00cc90 comp rx=0 tx=0).stop 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 shutdown_connections 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f71c406b4d0 0x7f71c406d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e415dde0 0x7f71e40b1630 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71e406cbf0 0x7f71e4157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 --2- 172.21.15.114:0/1594435782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f71e406afc0 0x7f71e4146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 >> 172.21.15.114:0/1594435782 conn(0x7f71e406c0d0 msgr2=0x7f71e4071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 shutdown_connections 2024-03-06T07:30:37.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.556+0000 7f71ea600640 1 -- 172.21.15.114:0/1594435782 wait complete. 2024-03-06T07:30:37.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:37.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.798+0000 7f4915e00640 1 Processor -- start 2024-03-06T07:30:37.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.798+0000 7f4915e00640 1 -- start start 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.798+0000 7f4915e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4910161580 0x7f49100d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.798+0000 7f4915e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d1e90 0x7f49100d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.798+0000 7f4915e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4910161960 con 0x7f49100d28a0 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.798+0000 7f4915e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100cf6f0 con 0x7f4910161580 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.798+0000 7f4915e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100cf9d0 con 0x7f49100d1e90 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4914a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4910161580 0x7f49100d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4914a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4910161580 0x7f49100d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39996/0 (socket says 172.21.15.114:39996) 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4914a00640 1 -- 172.21.15.114:0/1349758515 learned_addr learned my addr 172.21.15.114:0/1349758515 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4915400640 1 --1- 172.21.15.114:0/1349758515 >> v1:172.21.15.114:6789/0 conn(0x7f49100d28a0 0x7f49100d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42090/0 (socket says 172.21.15.114:42090) 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4907e00640 1 --2- 172.21.15.114:0/1349758515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d1e90 0x7f49100d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4914a00640 1 -- 172.21.15.114:0/1349758515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d1e90 msgr2=0x7f49100d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4914a00640 1 --2- 172.21.15.114:0/1349758515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d1e90 0x7f49100d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4914a00640 1 -- 172.21.15.114:0/1349758515 >> v1:172.21.15.114:6789/0 conn(0x7f49100d28a0 legacy=0x7f49100d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.799+0000 7f4914a00640 1 -- 172.21.15.114:0/1349758515 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100cfcb0 con 0x7f4910161580 2024-03-06T07:30:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.800+0000 7f4914a00640 1 --2- 172.21.15.114:0/1349758515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f49100d1950 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4900009870 tx=0x7f490002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6424a487b67b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f490fe00640 1 -- 172.21.15.114:0/1349758515 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900034070 con 0x7f4910161580 2024-03-06T07:30:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f490fe00640 1 -- 172.21.15.114:0/1349758515 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4900030840 con 0x7f4910161580 2024-03-06T07:30:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f490fe00640 1 -- 172.21.15.114:0/1349758515 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900030c10 con 0x7f4910161580 2024-03-06T07:30:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 -- 172.21.15.114:0/1349758515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 msgr2=0x7f49100d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 --2- 172.21.15.114:0/1349758515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f49100d1950 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4900009870 tx=0x7f490002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 -- 172.21.15.114:0/1349758515 shutdown_connections 2024-03-06T07:30:37.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 --2- 172.21.15.114:0/1349758515 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d1e90 0x7f49100d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 --2- 172.21.15.114:0/1349758515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f49100d1950 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 -- 172.21.15.114:0/1349758515 >> 172.21.15.114:0/1349758515 conn(0x7f49100d3d80 msgr2=0x7f49100d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 -- 172.21.15.114:0/1349758515 shutdown_connections 2024-03-06T07:30:37.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.801+0000 7f4915e00640 1 -- 172.21.15.114:0/1349758515 wait complete. 2024-03-06T07:30:37.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.803+0000 7f4915e00640 1 Processor -- start 2024-03-06T07:30:37.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.803+0000 7f4915e00640 1 -- start start 2024-03-06T07:30:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.804+0000 7f4915e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49100d1e90 0x7f49100ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.804+0000 7f4915e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d28a0 0x7f49100c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.804+0000 7f4915e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f491015ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.804+0000 7f4915e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100cf3a0 con 0x7f4910161580 2024-03-06T07:30:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.804+0000 7f4915e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100b39a0 con 0x7f49100d1e90 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.804+0000 7f4915e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100b4ed0 con 0x7f49100d28a0 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f491015ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f491015ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40000/0 (socket says 172.21.15.114:40000) 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 -- 172.21.15.114:0/2267767152 learned_addr learned my addr 172.21.15.114:0/2267767152 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4914a00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49100d1e90 0x7f49100ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4907e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d28a0 0x7f49100c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 -- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d28a0 msgr2=0x7f49100c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d28a0 0x7f49100c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 -- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49100d1e90 msgr2=0x7f49100ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49100d1e90 0x7f49100ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 -- 172.21.15.114:0/2267767152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100c7b80 con 0x7f4910161580 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4907e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d28a0 0x7f49100c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.805+0000 7f4915400640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f491015ac10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f490800c7b0 tx=0x7f490800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.806+0000 7f490d600640 1 -- 172.21.15.114:0/2267767152 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908017070 con 0x7f4910161580 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.806+0000 7f490d600640 1 -- 172.21.15.114:0/2267767152 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490800a640 con 0x7f4910161580 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.806+0000 7f490d600640 1 -- 172.21.15.114:0/2267767152 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490800a9f0 con 0x7f4910161580 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.806+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49100c7e60 con 0x7f4910161580 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.806+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491015b150 con 0x7f4910161580 2024-03-06T07:30:37.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.807+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49100d0130 con 0x7f4910161580 2024-03-06T07:30:37.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.808+0000 7f490d600640 1 -- 172.21.15.114:0/2267767152 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f490801a070 con 0x7f4910161580 2024-03-06T07:30:37.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.809+0000 7f490d600640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f48ec06b4d0 0x7f48ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.809+0000 7f490d600640 1 -- 172.21.15.114:0/2267767152 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4908089730 con 0x7f4910161580 2024-03-06T07:30:37.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.809+0000 7f4914a00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f48ec06b4d0 0x7f48ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.810+0000 7f4914a00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f48ec06b4d0 0x7f48ec06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4900036000 tx=0x7f490003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:37.815+0000 7f490d600640 1 -- 172.21.15.114:0/2267767152 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49100d0130 con 0x7f4910161580 2024-03-06T07:30:38.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.050+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f491004f330 con 0x7f4910161580 2024-03-06T07:30:38.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.050+0000 7f490d600640 1 -- 172.21.15.114:0/2267767152 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f4908057f50 con 0x7f4910161580 2024-03-06T07:30:38.053 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:38.054 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:30:38.054 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f48ec06b4d0 msgr2=0x7f48ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f48ec06b4d0 0x7f48ec06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4900036000 tx=0x7f490003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 msgr2=0x7f491015ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f491015ac10 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f490800c7b0 tx=0x7f490800cc70 comp rx=0 tx=0).stop 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 shutdown_connections 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f48ec06b4d0 0x7f48ec06d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4910161580 0x7f491015ac10 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f49100d28a0 0x7f49100c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 --2- 172.21.15.114:0/2267767152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49100d1e90 0x7f49100ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 >> 172.21.15.114:0/2267767152 conn(0x7f49100d3d80 msgr2=0x7f49100d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 shutdown_connections 2024-03-06T07:30:38.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.053+0000 7f4915e00640 1 -- 172.21.15.114:0/2267767152 wait complete. 2024-03-06T07:30:38.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:31.363295+0000', 'modified': '2024-03-06T07:30:34.444971+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5904}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5904': {'gid': 5904, 'name': 'a', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/4290788305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 4290788305}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 4290788305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5904, 'qdb_cluster': [5904]} 2024-03-06T07:30:38.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:38.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:30:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.690+0000 7f4fcd600640 1 Processor -- start 2024-03-06T07:30:38.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fcd600640 1 -- start start 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fcd600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fc81617d0 0x7f4fc80ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fcd600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4fc80d0f20 0x7f4fc80cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fcd600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc80cf6d0 con 0x7f4fc80b6d90 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fcd600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc80cff70 con 0x7f4fc81617d0 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fcd600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc80cf9b0 con 0x7f4fc80d0f20 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fc6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fc81617d0 0x7f4fc80ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fc6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fc81617d0 0x7f4fc80ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52770/0 (socket says 172.21.15.114:52770) 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.691+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3737698761 learned_addr learned my addr 172.21.15.114:0/3737698761 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.693+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3737698761 >> v2:172.21.15.177:3300/0 conn(0x7f4fc80d0f20 msgr2=0x7f4fc80cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.693+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3737698761 >> v2:172.21.15.177:3300/0 conn(0x7f4fc80d0f20 0x7f4fc80cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.693+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3737698761 >> v1:172.21.15.177:6789/0 conn(0x7f4fc80b6d90 legacy=0x7f4fc80d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.693+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3737698761 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc80cfc90 con 0x7f4fc81617d0 2024-03-06T07:30:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.694+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3737698761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc81617d0 0x7f4fc80ceb50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4fc808e6b0 tx=0x7f4fb802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48887fd5d0e08536 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.694+0000 7f4fc6000640 1 -- 172.21.15.114:0/3737698761 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb802e070 con 0x7f4fc81617d0 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.694+0000 7f4fc6000640 1 -- 172.21.15.114:0/3737698761 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8037070 con 0x7f4fc81617d0 2024-03-06T07:30:38.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.694+0000 7f4fc6000640 1 -- 172.21.15.114:0/3737698761 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8031a00 con 0x7f4fc81617d0 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 -- 172.21.15.114:0/3737698761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc81617d0 msgr2=0x7f4fc80ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 --2- 172.21.15.114:0/3737698761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc81617d0 0x7f4fc80ceb50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4fc808e6b0 tx=0x7f4fb802b880 comp rx=0 tx=0).stop 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 -- 172.21.15.114:0/3737698761 shutdown_connections 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 --2- 172.21.15.114:0/3737698761 >> v2:172.21.15.177:3300/0 conn(0x7f4fc80d0f20 0x7f4fc80cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 --2- 172.21.15.114:0/3737698761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc81617d0 0x7f4fc80ceb50 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 -- 172.21.15.114:0/3737698761 >> 172.21.15.114:0/3737698761 conn(0x7f4fc809d210 msgr2=0x7f4fc80d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 -- 172.21.15.114:0/3737698761 shutdown_connections 2024-03-06T07:30:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.695+0000 7f4fcd600640 1 -- 172.21.15.114:0/3737698761 wait complete. 2024-03-06T07:30:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.697+0000 7f4fcd600640 1 Processor -- start 2024-03-06T07:30:38.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.698+0000 7f4fcd600640 1 -- start start 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.699+0000 7f4fcd600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fc80b6d90 0x7f4fc80cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.699+0000 7f4fcd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc80d0f20 0x7f4fc8152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.699+0000 7f4fcd600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fc81617d0 0x7f4fc80c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.699+0000 7f4fcd600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80b5350 con 0x7f4fc80d0f20 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.699+0000 7f4fcd600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc8161bb0 con 0x7f4fc80b6d90 2024-03-06T07:30:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.699+0000 7f4fcd600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80b54d0 con 0x7f4fc81617d0 2024-03-06T07:30:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc80d0f20 0x7f4fc8152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc80d0f20 0x7f4fc8152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52786/0 (socket says 172.21.15.114:52786) 2024-03-06T07:30:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 -- 172.21.15.114:0/2757678766 learned_addr learned my addr 172.21.15.114:0/2757678766 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fc80b6d90 0x7f4fc80cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fc7400640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fc81617d0 0x7f4fc80c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 -- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fc81617d0 msgr2=0x7f4fc80c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fc81617d0 0x7f4fc80c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 -- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fc80b6d90 msgr2=0x7f4fc80cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fc80b6d90 0x7f4fc80cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 -- 172.21.15.114:0/2757678766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc80cc740 con 0x7f4fc80d0f20 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fc80b6d90 0x7f4fc80cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fc7400640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fc81617d0 0x7f4fc80c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.700+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc80d0f20 0x7f4fc8152200 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000b410 tx=0x7f4fb000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.701+0000 7f4fbea00640 1 -- 172.21.15.114:0/2757678766 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0015020 con 0x7f4fc80d0f20 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.701+0000 7f4fbea00640 1 -- 172.21.15.114:0/2757678766 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb0011830 con 0x7f4fc80d0f20 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.701+0000 7f4fbea00640 1 -- 172.21.15.114:0/2757678766 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0011c20 con 0x7f4fc80d0f20 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.701+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc80cca20 con 0x7f4fc80d0f20 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.701+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc8152740 con 0x7f4fc80d0f20 2024-03-06T07:30:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.703+0000 7f4fbea00640 1 -- 172.21.15.114:0/2757678766 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fb0018020 con 0x7f4fc80d0f20 2024-03-06T07:30:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.705+0000 7f4fbea00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fa406b4d0 0x7f4fa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.705+0000 7f4fbea00640 1 -- 172.21.15.114:0/2757678766 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fb008a170 con 0x7f4fc80d0f20 2024-03-06T07:30:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.705+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fa406b4d0 0x7f4fa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.705+0000 7f4fd0413740 1 Processor -- start 2024-03-06T07:30:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fd0413740 1 -- start start 2024-03-06T07:30:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fa406b4d0 0x7f4fa406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8036000 tx=0x7f4fb8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf8477b0 0x55c8bf81a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 0x55c8bf84a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf81ac90 0x55c8bf84aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fd0413740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf784600 con 0x55c8bf81ac90 2024-03-06T07:30:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf72d930 con 0x55c8bf847b90 2024-03-06T07:30:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.706+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf7806a0 con 0x55c8bf8477b0 2024-03-06T07:30:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf81ac90 0x55c8bf84aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf81ac90 0x55c8bf84aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52792/0 (socket says 172.21.15.114:52792) 2024-03-06T07:30:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 -- 172.21.15.114:0/663214768 learned_addr learned my addr 172.21.15.114:0/663214768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 -- 172.21.15.114:0/663214768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf8477b0 msgr2=0x55c8bf81a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf8477b0 0x55c8bf81a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 -- 172.21.15.114:0/663214768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 msgr2=0x55c8bf84a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 0x55c8bf84a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 0x55c8bf84a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.707+0000 7f4fc7400640 1 -- 172.21.15.114:0/663214768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8bf4992d0 con 0x55c8bf81ac90 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.708+0000 7f4fc7400640 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf81ac90 0x55c8bf84aa80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0007f60 tx=0x7f4fc0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12746dd2ea04560 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.708+0000 7f4fb7000640 1 -- 172.21.15.114:0/663214768 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0014070 con 0x55c8bf81ac90 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.708+0000 7f4fb7000640 1 -- 172.21.15.114:0/663214768 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc000f850 con 0x55c8bf81ac90 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.708+0000 7f4fb7000640 1 -- 172.21.15.114:0/663214768 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc000fc20 con 0x55c8bf81ac90 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.708+0000 7f4fd0413740 1 -- 172.21.15.114:0/663214768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf81ac90 msgr2=0x55c8bf84aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.708+0000 7f4fd0413740 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf81ac90 0x55c8bf84aa80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0007f60 tx=0x7f4fc0007f90 comp rx=0 tx=0).stop 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.709+0000 7f4fd0413740 1 -- 172.21.15.114:0/663214768 shutdown_connections 2024-03-06T07:30:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.709+0000 7f4fd0413740 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf81ac90 0x55c8bf84aa80 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.709+0000 7f4fd0413740 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 0x55c8bf84a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.709+0000 7f4fd0413740 1 --2- 172.21.15.114:0/663214768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf8477b0 0x55c8bf81a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.709+0000 7f4fd0413740 1 -- 172.21.15.114:0/663214768 >> 172.21.15.114:0/663214768 conn(0x55c8bf81c9c0 msgr2=0x55c8bf865540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.709+0000 7f4fd0413740 1 -- 172.21.15.114:0/663214768 shutdown_connections 2024-03-06T07:30:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.709+0000 7f4fd0413740 1 -- 172.21.15.114:0/663214768 wait complete. 2024-03-06T07:30:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.710+0000 7f4fd0413740 1 -- start start 2024-03-06T07:30:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.710+0000 7f4fd0413740 1 Processor -- start 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 0x55c8bf865fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8431d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf840400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fd0413740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf81ef60 con 0x55c8bf847b90 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf740d90 con 0x55c8bf8477b0 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf81e350 con 0x55c8bf84d3f0 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fd0413740 10 client.0 _ll_register_callbacks cb 0x55c8bf77c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52796/0 (socket says 172.21.15.114:52796) 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 -- 172.21.15.114:0/1939373136 learned_addr learned my addr 172.21.15.114:0/1939373136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 -- 172.21.15.114:0/1939373136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 msgr2=0x55c8bf840400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf840400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 -- 172.21.15.114:0/1939373136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 msgr2=0x55c8bf865fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 0x55c8bf865fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.712+0000 7f4fbfe00640 1 -- 172.21.15.114:0/1939373136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8bf5636b0 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8431d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0060ac0 tx=0x7f4fb0058a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1413c2b66a5a0280 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fab400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fab400640 1 -- 172.21.15.114:0/1939373136 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0070020 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fab400640 1 -- 172.21.15.114:0/1939373136 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fb0054f50 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4fc000f850 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fab400640 1 -- 172.21.15.114:0/1939373136 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fb0075050 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fd0413740 10 client.6057 fetch_fsmap learned FSMap version 35 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55c8bf84c910 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.713+0000 7f4fab400640 1 -- 172.21.15.114:0/1939373136 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc000f850 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fab400640 1 -- 172.21.15.114:0/1939373136 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 35) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4fb00772e0 con 0x55c8bf847b90 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fd0413740 10 client.6057 fetch_fsmap finished waiting for FSMap version 35 2024-03-06T07:30:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fd0413740 10 client.6057 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fd0413740 20 client.6057 populate_metadata read hostname 'smithi114' 2024-03-06T07:30:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fd0413740 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf879640 0x55c8bf879a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fd0413740 4 client.6057 mds_command: new command op to 5904 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 --> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8bf791b40 con 0x55c8bf879640 2024-03-06T07:30:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.714+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf879640 0x55c8bf879a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.715+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf879640 0x55c8bf879a20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b741ca3da3997d0 server_cookie=b937043331105c90 in_seq=0 out_seq=0 2024-03-06T07:30:38.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.715+0000 7f4fab400640 10 client.6057 ms_handle_connect on v2:172.21.15.114:6835/4290788305 2024-03-06T07:30:38.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fab400640 1 -- 172.21.15.114:0/1939373136 <== mds.0 v2:172.21.15.114:6835/4290788305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c8bf791b40 con 0x55c8bf879640 2024-03-06T07:30:38.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fab400640 10 client.6057 handle_command_reply: tid=0 2024-03-06T07:30:38.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 client.6057 shutdown 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 msgr2=0x55c8bf8431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8431d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0060ac0 tx=0x7f4fb0058a20 comp rx=0 tx=0).stop 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 shutdown_connections 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf879640 0x55c8bf879a20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf840400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8431d0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 --2- 172.21.15.114:0/1939373136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 0x55c8bf865fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 >> 172.21.15.114:0/1939373136 conn(0x55c8bf81ac90 msgr2=0x55c8bf819eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fab400640 0 client.6057 ms_handle_reset on v2:172.21.15.114:6835/4290788305 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.716+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 shutdown_connections 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.717+0000 7f4fd0413740 1 -- 172.21.15.114:0/1939373136 wait complete. 2024-03-06T07:30:38.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.717+0000 7f4fd0413740 20 client.6057 trim_cache size 0 max 16384 2024-03-06T07:30:38.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 Processor -- start 2024-03-06T07:30:38.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 -- start start 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf8477b0 0x55c8bf84ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 0x55c8bf841b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf8d89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc8200690 con 0x55c8bf8477b0 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80cb5f0 con 0x55c8bf847b90 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.734+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80c7f00 con 0x55c8bf84d3f0 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf8477b0 0x55c8bf84ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf8477b0 0x55c8bf84ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52798/0 (socket says 172.21.15.114:52798) 2024-03-06T07:30:38.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3806331776 learned_addr learned my addr 172.21.15.114:0/3806331776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3806331776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 msgr2=0x55c8bf8d89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3806331776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf8d89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3806331776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 msgr2=0x55c8bf841b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3806331776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 0x55c8bf841b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 -- 172.21.15.114:0/3806331776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8bf81c120 con 0x55c8bf8477b0 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3806331776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf8477b0 0x55c8bf84ddb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f4fb802be90 tx=0x7f4fb8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c9ead19587ce81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fb6600640 1 -- 172.21.15.114:0/3806331776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8003810 con 0x55c8bf8477b0 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fb6600640 1 -- 172.21.15.114:0/3806331776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8054070 con 0x55c8bf8477b0 2024-03-06T07:30:38.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.736+0000 7f4fb6600640 1 -- 172.21.15.114:0/3806331776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8003bb0 con 0x55c8bf8477b0 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 -- 172.21.15.114:0/3806331776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf8477b0 msgr2=0x55c8bf84ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3806331776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf8477b0 0x55c8bf84ddb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f4fb802be90 tx=0x7f4fb8031000 comp rx=0 tx=0).stop 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 -- 172.21.15.114:0/3806331776 shutdown_connections 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3806331776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf8d89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3806331776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf847b90 0x55c8bf841b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3806331776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf8477b0 0x55c8bf84ddb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 -- 172.21.15.114:0/3806331776 >> 172.21.15.114:0/3806331776 conn(0x55c8bf81ac90 msgr2=0x55c8bf867c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 -- 172.21.15.114:0/3806331776 shutdown_connections 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.737+0000 7f4fd0413740 1 -- 172.21.15.114:0/3806331776 wait complete. 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.738+0000 7f4fd0413740 1 -- start start 2024-03-06T07:30:38.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.738+0000 7f4fd0413740 1 Processor -- start 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 0x55c8bf8498a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8dd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fd0413740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf866f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fd0413740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf8437b0 con 0x55c8bf847b90 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80caff0 con 0x55c8bf8477b0 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fd0413740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c8bf848fa0 con 0x55c8bf84d3f0 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fd0413740 10 client.0 _ll_register_callbacks cb 0x7f4fb0058730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8dd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8dd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52810/0 (socket says 172.21.15.114:52810) 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.739+0000 7f4fbfe00640 1 -- 172.21.15.114:0/3021935841 learned_addr learned my addr 172.21.15.114:0/3021935841 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fbfe00640 1 -- 172.21.15.114:0/3021935841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 msgr2=0x55c8bf866f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf866f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fbfe00640 1 -- 172.21.15.114:0/3021935841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 msgr2=0x55c8bf8498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 0x55c8bf8498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 0x55c8bf8498a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fbfe00640 1 -- 172.21.15.114:0/3021935841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8bf81a0d0 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fbfe00640 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8dd470 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0058870 tx=0x7f4fb0004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54437be0e41c22b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fab400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fab400640 1 -- 172.21.15.114:0/3021935841 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0070070 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fab400640 1 -- 172.21.15.114:0/3021935841 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fb0056be0 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.740+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4fb8054070 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fab400640 1 -- 172.21.15.114:0/3021935841 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fb0070210 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 10 client.6063 fetch_fsmap learned FSMap version 35 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55c8bf87fc90 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fab400640 1 -- 172.21.15.114:0/3021935841 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8054070 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fab400640 1 -- 172.21.15.114:0/3021935841 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 35) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4fb0057d30 con 0x55c8bf847b90 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 10 client.6063 fetch_fsmap finished waiting for FSMap version 35 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 10 client.6063 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 20 client.6063 populate_metadata read hostname 'smithi114' 2024-03-06T07:30:38.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf84e010 0x55c8bf81f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 4 client.6063 mds_command: new command op to 5904 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:38.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 --> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c8bf78be90 con 0x55c8bf84e010 2024-03-06T07:30:38.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.741+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf84e010 0x55c8bf81f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fc6a00640 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf84e010 0x55c8bf81f450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd4a2a7c6b3170f7 server_cookie=d61766f35c75c8fb in_seq=0 out_seq=0 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fab400640 10 client.6063 ms_handle_connect on v2:172.21.15.114:6835/4290788305 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fab400640 1 -- 172.21.15.114:0/3021935841 <== mds.0 v2:172.21.15.114:6835/4290788305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c8bf78be90 con 0x55c8bf84e010 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fab400640 10 client.6063 handle_command_reply: tid=0 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 client.6063 shutdown 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 msgr2=0x55c8bf8dd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8dd470 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0058870 tx=0x7f4fb0004640 comp rx=0 tx=0).stop 2024-03-06T07:30:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 shutdown_connections 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:6835/4290788305,v1:172.21.15.114:6837/4290788305] conn(0x55c8bf84e010 0x55c8bf81f450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55c8bf84d3f0 0x55c8bf866f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8bf847b90 0x55c8bf8dd470 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 --2- 172.21.15.114:0/3021935841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c8bf8477b0 0x55c8bf8498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fab400640 0 client.6063 ms_handle_reset on v2:172.21.15.114:6835/4290788305 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.742+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 >> 172.21.15.114:0/3021935841 conn(0x55c8bf81ac90 msgr2=0x55c8bf8159e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 shutdown_connections 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fd0413740 1 -- 172.21.15.114:0/3021935841 wait complete. 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fd0413740 20 client.6063 trim_cache size 0 max 16384 2024-03-06T07:30:38.748 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 5904, 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 6.8031410489999997, 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 35, 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 55, 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 55, 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 12.247564537000001 2024-03-06T07:30:38.749 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fa406b4d0 msgr2=0x7f4fa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fa406b4d0 0x7f4fa406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8036000 tx=0x7f4fb8008030 comp rx=0 tx=0).stop 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc80d0f20 msgr2=0x7f4fc8152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc80d0f20 0x7f4fc8152200 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000b410 tx=0x7f4fb000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 shutdown_connections 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4fc81617d0 0x7f4fc80c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4fa406b4d0 0x7f4fa406d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fc80d0f20 0x7f4fc8152200 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 --2- 172.21.15.114:0/2757678766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fc80b6d90 0x7f4fc80cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 >> 172.21.15.114:0/2757678766 conn(0x7f4fc809d210 msgr2=0x7f4fc80ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.743+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 shutdown_connections 2024-03-06T07:30:38.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.744+0000 7f4fcd600640 1 -- 172.21.15.114:0/2757678766 wait complete. 2024-03-06T07:30:38.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:38.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.956+0000 7f8feae00640 1 Processor -- start 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.956+0000 7f8feae00640 1 -- start start 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.957+0000 7f8feae00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8fe40d24c0 0x7f8fe40d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.957+0000 7f8feae00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fe40b0aa0 0x7f8fe40b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.957+0000 7f8feae00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe40d4c50 con 0x7f8fe40b2de0 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.957+0000 7f8feae00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe40d4a00 con 0x7f8fe40b0aa0 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.957+0000 7f8feae00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe40cf7e0 con 0x7f8fe40d24c0 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe8e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fe40b0aa0 0x7f8fe40b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8fe40b2de0 0x7f8fe40d28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36824/0 (socket says 172.21.15.114:36824) 2024-03-06T07:30:38.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe3400640 1 -- 172.21.15.114:0/2829411362 learned_addr learned my addr 172.21.15.114:0/2829411362 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:38.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/2829411362 >> v2:172.21.15.177:3300/0 conn(0x7f8fe40d24c0 0x7f8fe40d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe8e00640 1 -- 172.21.15.114:0/2829411362 >> v2:172.21.15.177:3300/0 conn(0x7f8fe40d24c0 msgr2=0x7f8fe40d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:38.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe8e00640 1 --2- 172.21.15.114:0/2829411362 >> v2:172.21.15.177:3300/0 conn(0x7f8fe40d24c0 0x7f8fe40d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe8e00640 1 -- 172.21.15.114:0/2829411362 >> v1:172.21.15.114:6789/0 conn(0x7f8fe40b2de0 legacy=0x7f8fe40d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe8e00640 1 -- 172.21.15.114:0/2829411362 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe40cfa30 con 0x7f8fe40b0aa0 2024-03-06T07:30:38.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.958+0000 7f8fe2a00640 1 -- 172.21.15.114:0/2829411362 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251746024 0 0) 0x7f8fe40d4c50 con 0x7f8fe40b2de0 2024-03-06T07:30:38.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.959+0000 7f8fe8e00640 1 --2- 172.21.15.114:0/2829411362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40b0aa0 0x7f8fe40b0f30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8fe408d300 tx=0x7f8fdc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7ddac92d592a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8fe2a00640 1 -- 172.21.15.114:0/2829411362 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc039070 con 0x7f8fe40b0aa0 2024-03-06T07:30:38.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8fe2a00640 1 -- 172.21.15.114:0/2829411362 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc02cbf0 con 0x7f8fe40b0aa0 2024-03-06T07:30:38.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8fe2a00640 1 -- 172.21.15.114:0/2829411362 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc034440 con 0x7f8fe40b0aa0 2024-03-06T07:30:38.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 -- 172.21.15.114:0/2829411362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40b0aa0 msgr2=0x7f8fe40b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 --2- 172.21.15.114:0/2829411362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40b0aa0 0x7f8fe40b0f30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8fe408d300 tx=0x7f8fdc02be40 comp rx=0 tx=0).stop 2024-03-06T07:30:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 -- 172.21.15.114:0/2829411362 shutdown_connections 2024-03-06T07:30:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 --2- 172.21.15.114:0/2829411362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40b0aa0 0x7f8fe40b0f30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 --2- 172.21.15.114:0/2829411362 >> v2:172.21.15.177:3300/0 conn(0x7f8fe40d24c0 0x7f8fe40d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 -- 172.21.15.114:0/2829411362 >> 172.21.15.114:0/2829411362 conn(0x7f8fe40d3ee0 msgr2=0x7f8fe40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 -- 172.21.15.114:0/2829411362 shutdown_connections 2024-03-06T07:30:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.960+0000 7f8feae00640 1 -- 172.21.15.114:0/2829411362 wait complete. 2024-03-06T07:30:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.963+0000 7f8feae00640 1 Processor -- start 2024-03-06T07:30:38.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.963+0000 7f8feae00640 1 -- start start 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8feae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fe40b0aa0 0x7f8fe4152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8feae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fe40b2de0 0x7f8fe415fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8feae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40d24c0 0x7f8fe40c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8feae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe415e5b0 con 0x7f8fe40d24c0 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8feae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe40d0080 con 0x7f8fe40b0aa0 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8feae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe415e730 con 0x7f8fe40b2de0 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8fe8e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40d24c0 0x7f8fe40c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8fe8e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40d24c0 0x7f8fe40c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52828/0 (socket says 172.21.15.114:52828) 2024-03-06T07:30:38.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.965+0000 7f8fe8e00640 1 -- 172.21.15.114:0/808746104 learned_addr learned my addr 172.21.15.114:0/808746104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:38.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fe40b0aa0 0x7f8fe4152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe3400640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fe40b2de0 0x7f8fe415fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe8e00640 1 -- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fe40b2de0 msgr2=0x7f8fe415fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe8e00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fe40b2de0 0x7f8fe415fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe8e00640 1 -- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fe40b0aa0 msgr2=0x7f8fe4152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe8e00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fe40b0aa0 0x7f8fe4152070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe8e00640 1 -- 172.21.15.114:0/808746104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe41525b0 con 0x7f8fe40d24c0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fe8e00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40d24c0 0x7f8fe40c8420 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc02c450 tx=0x7f8fdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fdbe00640 1 -- 172.21.15.114:0/808746104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc044020 con 0x7f8fe40d24c0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fdbe00640 1 -- 172.21.15.114:0/808746104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc03f470 con 0x7f8fe40d24c0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe415eb00 con 0x7f8fe40d24c0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.966+0000 7f8fdbe00640 1 -- 172.21.15.114:0/808746104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc039040 con 0x7f8fe40d24c0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.967+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe415f0c0 con 0x7f8fe40d24c0 2024-03-06T07:30:38.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.968+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe404f330 con 0x7f8fe40d24c0 2024-03-06T07:30:38.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.969+0000 7f8fdbe00640 1 -- 172.21.15.114:0/808746104 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fdc047020 con 0x7f8fe40d24c0 2024-03-06T07:30:38.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.969+0000 7f8fdbe00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc406b4d0 0x7f8fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:38.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.969+0000 7f8fdbe00640 1 -- 172.21.15.114:0/808746104 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fdc0acd80 con 0x7f8fe40d24c0 2024-03-06T07:30:38.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.970+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc406b4d0 0x7f8fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:38.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.970+0000 7f8fe3e00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc406b4d0 0x7f8fc406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4006fd0 tx=0x7f8fd4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:38.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:38.974+0000 7f8fdbe00640 1 -- 172.21.15.114:0/808746104 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fdc04e030 con 0x7f8fe40d24c0 2024-03-06T07:30:39.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.258+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8fe40d2b50 con 0x7f8fe40d24c0 2024-03-06T07:30:39.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.259+0000 7f8fdbe00640 1 -- 172.21.15.114:0/808746104 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8fdc07bc00 con 0x7f8fe40d24c0 2024-03-06T07:30:39.261 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc406b4d0 msgr2=0x7f8fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc406b4d0 0x7f8fc406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4006fd0 tx=0x7f8fd4008040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40d24c0 msgr2=0x7f8fe40c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40d24c0 0x7f8fe40c8420 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc02c450 tx=0x7f8fdc004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 shutdown_connections 2024-03-06T07:30:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fc406b4d0 0x7f8fc406d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fe40d24c0 0x7f8fe40c8420 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fe40b2de0 0x7f8fe415fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 --2- 172.21.15.114:0/808746104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fe40b0aa0 0x7f8fe4152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.261+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 >> 172.21.15.114:0/808746104 conn(0x7f8fe40d3ee0 msgr2=0x7f8fe40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.262+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 shutdown_connections 2024-03-06T07:30:39.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:39.262+0000 7f8feae00640 1 -- 172.21.15.114:0/808746104 wait complete. 2024-03-06T07:30:39.283 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:30:39.283 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:39.283 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:30:39.283 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:39.283 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:39.284 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:30:39.284 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:30:39.284 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:39.284 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:39.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.284 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:39.335 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42416sec preferred_lft 42416sec 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:39.336 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:30:39.337 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:39.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.339 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:39.425 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:30:39.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.429 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.482 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:30:39.482 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:30:39.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:39.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:39.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:39.483 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:30:39.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.658 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:30:39.704 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:39.705 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:39.706 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:39.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:30:39.780 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:39.780 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:39.825 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:39.825 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:39.825 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:39.922 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:30:39.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:40.046 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:40.046 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:40.081 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:40.081 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:40.081 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:30:40.081 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:30:40.081 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:30:40.084 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:30:40.084 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:30:40.084 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:30:40.084 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:30:40.084 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:30:40.087 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:30:40.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:40.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:40.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:40.164 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:30:40.215 INFO:teuthology.orchestra.run.smithi114.stdout:client4895 2024-03-06T07:30:40.215 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:40.215 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:40.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:30:40.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.505+0000 7fc9c6e00640 1 Processor -- start 2024-03-06T07:30:40.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.505+0000 7fc9c6e00640 1 -- start start 2024-03-06T07:30:40.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9c6e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9c00b3df0 0x7fc9c00727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9c6e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00b3650 0x7fc9c00b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9c6e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c00b4710 con 0x7fc9c00cda90 2024-03-06T07:30:40.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9c6e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c0072310 con 0x7fc9c00b3df0 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9c6e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c00719e0 con 0x7fc9c00b3650 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9bfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9c00b3df0 0x7fc9c00727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9bfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9c00b3df0 0x7fc9c00727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52834/0 (socket says 172.21.15.114:52834) 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.507+0000 7fc9bfe00640 1 -- 172.21.15.114:0/2511260732 learned_addr learned my addr 172.21.15.114:0/2511260732 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.508+0000 7fc9bf400640 1 --2- 172.21.15.114:0/2511260732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00b3650 0x7fc9c00b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.508+0000 7fc9bfe00640 1 -- 172.21.15.114:0/2511260732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00b3650 msgr2=0x7fc9c00b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.508+0000 7fc9bfe00640 1 --2- 172.21.15.114:0/2511260732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00b3650 0x7fc9c00b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.508+0000 7fc9bfe00640 1 -- 172.21.15.114:0/2511260732 >> v1:172.21.15.177:6789/0 conn(0x7fc9c00cda90 legacy=0x7fc9c006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.508+0000 7fc9bfe00640 1 -- 172.21.15.114:0/2511260732 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c0071cc0 con 0x7fc9c00b3df0 2024-03-06T07:30:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.509+0000 7fc9bfe00640 1 --2- 172.21.15.114:0/2511260732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c00727a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000bb10 tx=0x7fc9b002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b567dbab95710957 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.509+0000 7fc9bea00640 1 -- 172.21.15.114:0/2511260732 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b003b070 con 0x7fc9c00b3df0 2024-03-06T07:30:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.509+0000 7fc9bea00640 1 -- 172.21.15.114:0/2511260732 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b0036440 con 0x7fc9c00b3df0 2024-03-06T07:30:40.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.509+0000 7fc9bea00640 1 -- 172.21.15.114:0/2511260732 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0036810 con 0x7fc9c00b3df0 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 -- 172.21.15.114:0/2511260732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 msgr2=0x7fc9c00727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/2511260732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c00727a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000bb10 tx=0x7fc9b002e0c0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 -- 172.21.15.114:0/2511260732 shutdown_connections 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/2511260732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00b3650 0x7fc9c00b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/2511260732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c00727a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 -- 172.21.15.114:0/2511260732 >> 172.21.15.114:0/2511260732 conn(0x7fc9c00b1940 msgr2=0x7fc9c00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 -- 172.21.15.114:0/2511260732 shutdown_connections 2024-03-06T07:30:40.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.510+0000 7fc9c6e00640 1 -- 172.21.15.114:0/2511260732 wait complete. 2024-03-06T07:30:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.512+0000 7fc9c6e00640 1 Processor -- start 2024-03-06T07:30:40.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.512+0000 7fc9c6e00640 1 -- start start 2024-03-06T07:30:40.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9c6e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9c00b3650 0x7fc9c0150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9c6e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c009c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9c6e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00cda90 0x7fc9c0152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9c6e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c015b240 con 0x7fc9c00b3df0 2024-03-06T07:30:40.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9c6e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c00b3b30 con 0x7fc9c00b3650 2024-03-06T07:30:40.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9c6e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c015b3c0 con 0x7fc9c00cda90 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9bf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c009c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9bf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c009c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52840/0 (socket says 172.21.15.114:52840) 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.513+0000 7fc9bf400640 1 -- 172.21.15.114:0/3534286193 learned_addr learned my addr 172.21.15.114:0/3534286193 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bfe00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9c00b3650 0x7fc9c0150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bf400640 1 -- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00cda90 msgr2=0x7fc9c0152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9c4e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00cda90 0x7fc9c0152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bf400640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00cda90 0x7fc9c0152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bf400640 1 -- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9c00b3650 msgr2=0x7fc9c0150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bf400640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9c00b3650 0x7fc9c0150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bf400640 1 -- 172.21.15.114:0/3534286193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c009cc50 con 0x7fc9c00b3df0 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bfe00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9c00b3650 0x7fc9c0150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9bf400640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c009c2b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00c820 tx=0x7fc9ac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.514+0000 7fc9b7e00640 1 -- 172.21.15.114:0/3534286193 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac015070 con 0x7fc9c00b3df0 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.515+0000 7fc9b7e00640 1 -- 172.21.15.114:0/3534286193 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac004240 con 0x7fc9c00b3df0 2024-03-06T07:30:40.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.515+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c009cf30 con 0x7fc9c00b3df0 2024-03-06T07:30:40.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.515+0000 7fc9b7e00640 1 -- 172.21.15.114:0/3534286193 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac00ce10 con 0x7fc9c00b3df0 2024-03-06T07:30:40.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.515+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c0150c40 con 0x7fc9c00b3df0 2024-03-06T07:30:40.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.516+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c004f330 con 0x7fc9c00b3df0 2024-03-06T07:30:40.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.517+0000 7fc9b7e00640 1 -- 172.21.15.114:0/3534286193 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9ac007830 con 0x7fc9c00b3df0 2024-03-06T07:30:40.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.517+0000 7fc9b7e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9a006b4d0 0x7fc9a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.518+0000 7fc9b7e00640 1 -- 172.21.15.114:0/3534286193 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9ac08a0f0 con 0x7fc9c00b3df0 2024-03-06T07:30:40.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.518+0000 7fc9bfe00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9a006b4d0 0x7fc9a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.519+0000 7fc9bfe00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9a006b4d0 0x7fc9a006d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0038000 tx=0x7fc9b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.523+0000 7fc9b7e00640 1 -- 172.21.15.114:0/3534286193 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9ac058f60 con 0x7fc9c00b3df0 2024-03-06T07:30:40.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.856+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fc9c00cc0e0 con 0x7fc9c00b3df0 2024-03-06T07:30:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.863+0000 7fc9b7e00640 1 -- 172.21.15.114:0/3534286193 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc9ac058860 con 0x7fc9c00b3df0 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9a006b4d0 msgr2=0x7fc9a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9a006b4d0 0x7fc9a006d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0038000 tx=0x7fc9b0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 msgr2=0x7fc9c009c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c009c2b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00c820 tx=0x7fc9ac00cce0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 shutdown_connections 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9c00cda90 0x7fc9c0152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9a006b4d0 0x7fc9a006d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9c00b3df0 0x7fc9c009c2b0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 --2- 172.21.15.114:0/3534286193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9c00b3650 0x7fc9c0150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.866+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 >> 172.21.15.114:0/3534286193 conn(0x7fc9c00b1940 msgr2=0x7fc9c0155bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.867+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 shutdown_connections 2024-03-06T07:30:40.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:40.867+0000 7fc9c6e00640 1 -- 172.21.15.114:0/3534286193 wait complete. 2024-03-06T07:30:40.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:30:41.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.095+0000 7f063a000640 1 Processor -- start 2024-03-06T07:30:41.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.095+0000 7f063a000640 1 -- start start 2024-03-06T07:30:41.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.097+0000 7f063a000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0634160700 0x7f0634070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.097+0000 7f063a000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340b46c0 con 0x7f06340b3f70 2024-03-06T07:30:41.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.097+0000 7f063a000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340b49a0 con 0x7f063415fc10 2024-03-06T07:30:41.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.097+0000 7f063a000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0634072c90 con 0x7f0634160700 2024-03-06T07:30:41.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.097+0000 7f062aa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f06340b3f70 0x7f06340ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36834/0 (socket says 172.21.15.114:36834) 2024-03-06T07:30:41.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.097+0000 7f062aa00640 1 -- 172.21.15.114:0/1223613326 learned_addr learned my addr 172.21.15.114:0/1223613326 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0633e00640 1 --2- 172.21.15.114:0/1223613326 >> v2:172.21.15.177:3300/0 conn(0x7f0634160700 0x7f0634070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526180654 0 0) 0x7f06340b46c0 con 0x7f06340b3f70 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0618003520 con 0x7f06340b3f70 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860696775 0 0) 0x7f0618003520 con 0x7f06340b3f70 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 >> v2:172.21.15.177:3300/0 conn(0x7f0634160700 msgr2=0x7f0634070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 --2- 172.21.15.114:0/1223613326 >> v2:172.21.15.177:3300/0 conn(0x7f0634160700 0x7f0634070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 >> v1:172.21.15.177:6789/0 conn(0x7f063415fc10 legacy=0x7f063415fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063409dd80 con 0x7f06340b3f70 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f061c001670 con 0x7f06340b3f70 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 >> v1:172.21.15.114:6789/0 conn(0x7f06340b3f70 legacy=0x7f06340ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06180039b0 0x7f0618003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0618004800 0x7f0618004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180051c0 0x7f061800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634072c90 con 0x7f06180039b0 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061800c110 con 0x7f06180051c0 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061800c310 con 0x7f0618004800 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.098+0000 7f0633e00640 1 --2- 172.21.15.114:0/1223613326 >> v2:172.21.15.177:3300/0 conn(0x7f0634160700 0x7f0634070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633400640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06180039b0 0x7f0618003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f062aa00640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0618004800 0x7f0618004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633e00640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180051c0 0x7f061800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633400640 1 -- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0618004800 msgr2=0x7f0618004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633400640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0618004800 0x7f0618004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633400640 1 -- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180051c0 msgr2=0x7f061800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633400640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180051c0 0x7f061800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633400640 1 -- 172.21.15.114:0/1223613326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0618003520 con 0x7f06180039b0 2024-03-06T07:30:41.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.099+0000 7f0633e00640 1 -- 172.21.15.114:0/1223613326 reap_dead start 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.100+0000 7f0633400640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06180039b0 0x7f0618003dd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0624001e50 tx=0x7f0618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e4e4b8c3357dee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.100+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624001a70 con 0x7f06180039b0 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.100+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624003b70 con 0x7f06180039b0 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.100+0000 7f0632a00640 1 -- 172.21.15.114:0/1223613326 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06240050d0 con 0x7f06180039b0 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.101+0000 7f063a000640 1 -- 172.21.15.114:0/1223613326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06180039b0 msgr2=0x7f0618003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.101+0000 7f063a000640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06180039b0 0x7f0618003dd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0624001e50 tx=0x7f0618005830 comp rx=0 tx=0).stop 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.101+0000 7f063a000640 1 -- 172.21.15.114:0/1223613326 shutdown_connections 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.101+0000 7f063a000640 1 --2- 172.21.15.114:0/1223613326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06180039b0 0x7f0618003dd0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.101+0000 7f063a000640 1 -- 172.21.15.114:0/1223613326 >> 172.21.15.114:0/1223613326 conn(0x7f06340b1940 msgr2=0x7f06340b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.101+0000 7f063a000640 1 -- 172.21.15.114:0/1223613326 shutdown_connections 2024-03-06T07:30:41.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.101+0000 7f063a000640 1 -- 172.21.15.114:0/1223613326 wait complete. 2024-03-06T07:30:41.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.103+0000 7f063a000640 1 Processor -- start 2024-03-06T07:30:41.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.103+0000 7f063a000640 1 -- start start 2024-03-06T07:30:41.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f063a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180039b0 0x7f063409c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f063a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f063409d1e0 0x7f063415e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f063a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f063406de60 0x7f063406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f063a000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063415c460 con 0x7f063406de60 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f063a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063415c2e0 con 0x7f06180039b0 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f063a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0634070ff0 con 0x7f063409d1e0 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f0633e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f063406de60 0x7f063406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f0633e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f063406de60 0x7f063406e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52866/0 (socket says 172.21.15.114:52866) 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.104+0000 7f0633e00640 1 -- 172.21.15.114:0/2362847046 learned_addr learned my addr 172.21.15.114:0/2362847046 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f062aa00640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f063409d1e0 0x7f063415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f0633e00640 1 -- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f063409d1e0 msgr2=0x7f063415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f0633e00640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f063409d1e0 0x7f063415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f0633e00640 1 -- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180039b0 msgr2=0x7f063409c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f0633400640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180039b0 0x7f063409c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f0633e00640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180039b0 0x7f063409c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f0633e00640 1 -- 172.21.15.114:0/2362847046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063415c830 con 0x7f063406de60 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f062aa00640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f063409d1e0 0x7f063415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f0633e00640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f063406de60 0x7f063406e2e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0634070bc0 tx=0x7f062c0108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.105+0000 7f062be00640 1 -- 172.21.15.114:0/2362847046 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c00f040 con 0x7f063406de60 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.106+0000 7f062be00640 1 -- 172.21.15.114:0/2362847046 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c01b070 con 0x7f063406de60 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.106+0000 7f062be00640 1 -- 172.21.15.114:0/2362847046 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c015ae0 con 0x7f063406de60 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.106+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063415cb10 con 0x7f063406de60 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.106+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063409d660 con 0x7f063406de60 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.107+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063404f330 con 0x7f063406de60 2024-03-06T07:30:41.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.108+0000 7f062be00640 1 -- 172.21.15.114:0/2362847046 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f062c015c80 con 0x7f063406de60 2024-03-06T07:30:41.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.109+0000 7f062be00640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f061006b500 0x7f061006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.109+0000 7f062be00640 1 -- 172.21.15.114:0/2362847046 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f062c08a240 con 0x7f063406de60 2024-03-06T07:30:41.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.109+0000 7f0633400640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f061006b500 0x7f061006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.110+0000 7f0633400640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f061006b500 0x7f061006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0624034000 tx=0x7f0624009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.113+0000 7f062be00640 1 -- 172.21.15.114:0/2362847046 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f062c0590b0 con 0x7f063406de60 2024-03-06T07:30:41.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.312+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f06340b16b0 con 0x7f063406de60 2024-03-06T07:30:41.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.318+0000 7f062be00640 1 -- 172.21.15.114:0/2362847046 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f062c08a020 con 0x7f063406de60 2024-03-06T07:30:41.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f061006b500 msgr2=0x7f061006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f061006b500 0x7f061006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0624034000 tx=0x7f0624009030 comp rx=0 tx=0).stop 2024-03-06T07:30:41.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f063406de60 msgr2=0x7f063406e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f063406de60 0x7f063406e2e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0634070bc0 tx=0x7f062c0108b0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 shutdown_connections 2024-03-06T07:30:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f061006b500 0x7f061006d9c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f063406de60 0x7f063406e2e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f063409d1e0 0x7f063415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 --2- 172.21.15.114:0/2362847046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06180039b0 0x7f063409c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.321+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 >> 172.21.15.114:0/2362847046 conn(0x7f06340b1940 msgr2=0x7f063415fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.322+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 shutdown_connections 2024-03-06T07:30:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:41.322+0000 7f063a000640 1 -- 172.21.15.114:0/2362847046 wait complete. 2024-03-06T07:30:45.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:45.265 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:51.349 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:51.350 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> import uuid 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> n = 50 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = False 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> if 0 > 0: 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = True 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:51.408 DEBUG:teuthology.orchestra.run.smithi114:> dpath = os.path.dirname(path) 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> fnameprefix = os.path.basename(path) 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> try: 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> for i in range(n): 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> f.write(f"{i}") 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> f.flush() 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(f.fileno()) 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:51.409 DEBUG:teuthology.orchestra.run.smithi114:> os.unlink(fpath) 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> if create_hard_links: 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> for j in range(0): 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> if True: 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> finally: 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> os.close(dirfd) 2024-03-06T07:30:51.410 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:30:51.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:51.704 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:51.705 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:51.794 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:51.794 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:51.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:52.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.125+0000 7f58da200640 1 Processor -- start 2024-03-06T07:30:52.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.125+0000 7f58da200640 1 -- start start 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.126+0000 7f58da200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58d415acb0 0x7f58d40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58da200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b4680 0x7f58d406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58da200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f58d4072340 0x7f58d406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58da200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d415b090 con 0x7f58d415acb0 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58da200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d40b2a30 con 0x7f58d4072340 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58da200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40b4360 con 0x7f58d40b4680 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58d415acb0 0x7f58d40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58d415acb0 0x7f58d40b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48888/0 (socket says 172.21.15.114:48888) 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 -- 172.21.15.114:0/1849697874 learned_addr learned my addr 172.21.15.114:0/1849697874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 -- 172.21.15.114:0/1849697874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b4680 msgr2=0x7f58d406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 --2- 172.21.15.114:0/1849697874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b4680 0x7f58d406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 -- 172.21.15.114:0/1849697874 >> v2:172.21.15.177:3300/0 conn(0x7f58d4072340 msgr2=0x7f58d406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 --2- 172.21.15.114:0/1849697874 >> v2:172.21.15.177:3300/0 conn(0x7f58d4072340 0x7f58d406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.127+0000 7f58d3400640 1 -- 172.21.15.114:0/1849697874 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40b2cf0 con 0x7f58d415acb0 2024-03-06T07:30:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.128+0000 7f58d3400640 1 --2- 172.21.15.114:0/1849697874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d40b37d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f58d4070aa0 tx=0x7f58c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60cd270cf67c1a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58d2a00640 1 -- 172.21.15.114:0/1849697874 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4035020 con 0x7f58d415acb0 2024-03-06T07:30:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58d2a00640 1 -- 172.21.15.114:0/1849697874 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c4031870 con 0x7f58d415acb0 2024-03-06T07:30:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58d2a00640 1 -- 172.21.15.114:0/1849697874 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4031c40 con 0x7f58d415acb0 2024-03-06T07:30:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 -- 172.21.15.114:0/1849697874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 msgr2=0x7f58d40b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 --2- 172.21.15.114:0/1849697874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d40b37d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f58d4070aa0 tx=0x7f58c402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 -- 172.21.15.114:0/1849697874 shutdown_connections 2024-03-06T07:30:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 --2- 172.21.15.114:0/1849697874 >> v2:172.21.15.177:3300/0 conn(0x7f58d4072340 0x7f58d406ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 --2- 172.21.15.114:0/1849697874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b4680 0x7f58d406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 --2- 172.21.15.114:0/1849697874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d40b37d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 -- 172.21.15.114:0/1849697874 >> 172.21.15.114:0/1849697874 conn(0x7f58d40b1730 msgr2=0x7f58d40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 -- 172.21.15.114:0/1849697874 shutdown_connections 2024-03-06T07:30:52.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.129+0000 7f58da200640 1 -- 172.21.15.114:0/1849697874 wait complete. 2024-03-06T07:30:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.132+0000 7f58da200640 1 Processor -- start 2024-03-06T07:30:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.132+0000 7f58da200640 1 -- start start 2024-03-06T07:30:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58da200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d4072340 0x7f58d414e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58da200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b4680 0x7f58d41f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58da200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d4140450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58da200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40b3d10 con 0x7f58d415acb0 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58da200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40b32b0 con 0x7f58d40b4680 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58da200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40b4010 con 0x7f58d4072340 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58d3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d4140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58d3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d4140450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48894/0 (socket says 172.21.15.114:48894) 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.134+0000 7f58d3e00640 1 -- 172.21.15.114:0/1006661885 learned_addr learned my addr 172.21.15.114:0/1006661885 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58d3400640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d4072340 0x7f58d414e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58cbe00640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b4680 0x7f58d41f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58d3e00640 1 -- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d4072340 msgr2=0x7f58d414e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58d3e00640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d4072340 0x7f58d414e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58d3e00640 1 -- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b4680 msgr2=0x7f58d41f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58d3e00640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b4680 0x7f58d41f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58d3e00640 1 -- 172.21.15.114:0/1006661885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4143910 con 0x7f58d415acb0 2024-03-06T07:30:52.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58d3e00640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d4140450 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00b970 tx=0x7f58cc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58cb400640 1 -- 172.21.15.114:0/1006661885 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc019070 con 0x7f58d415acb0 2024-03-06T07:30:52.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58cb400640 1 -- 172.21.15.114:0/1006661885 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc00cb60 con 0x7f58d415acb0 2024-03-06T07:30:52.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d414ecb0 con 0x7f58d415acb0 2024-03-06T07:30:52.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.135+0000 7f58cb400640 1 -- 172.21.15.114:0/1006661885 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc014440 con 0x7f58d415acb0 2024-03-06T07:30:52.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.136+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d41f2430 con 0x7f58d415acb0 2024-03-06T07:30:52.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.137+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d4153f80 con 0x7f58d415acb0 2024-03-06T07:30:52.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.139+0000 7f58cb400640 1 -- 172.21.15.114:0/1006661885 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58cc01c070 con 0x7f58d415acb0 2024-03-06T07:30:52.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.139+0000 7f58cb400640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.140+0000 7f58cb400640 1 -- 172.21.15.114:0/1006661885 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58cc089f90 con 0x7f58d415acb0 2024-03-06T07:30:52.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.140+0000 7f58d3400640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.140+0000 7f58d3400640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f58c4036000 tx=0x7f58c403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.144+0000 7f58cb400640 1 -- 172.21.15.114:0/1006661885 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58d4153f80 con 0x7f58d415acb0 2024-03-06T07:30:52.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.340+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58d415f3a0 con 0x7f58d415acb0 2024-03-06T07:30:52.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.343+0000 7f58cb400640 1 -- 172.21.15.114:0/1006661885 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16713 (secure 0 0 0) 0x7f58cc081e20 con 0x7f58d415acb0 2024-03-06T07:30:52.345 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:52.345 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:32.939149+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 msgr2=0x7f58b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f58c4036000 tx=0x7f58c403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 msgr2=0x7f58d4140450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d4140450 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00b970 tx=0x7f58cc00be30 comp rx=0 tx=0).stop 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 shutdown_connections 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d415acb0 0x7f58d4140450 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b4680 0x7f58d41f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 --2- 172.21.15.114:0/1006661885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d4072340 0x7f58d414e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 >> 172.21.15.114:0/1006661885 conn(0x7f58d40b1730 msgr2=0x7f58d406d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 shutdown_connections 2024-03-06T07:30:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.346+0000 7f58da200640 1 -- 172.21.15.114:0/1006661885 wait complete. 2024-03-06T07:30:52.369 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:29.190809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:30.382190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:52.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:52.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.570+0000 7fce22e00640 1 Processor -- start 2024-03-06T07:30:52.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.570+0000 7fce22e00640 1 -- start start 2024-03-06T07:30:52.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce22e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c06d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce22e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fce1c163800 0x7fce1c0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce22e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1c15cf70 con 0x7fce1c06e060 2024-03-06T07:30:52.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce22e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1c0b3890 con 0x7fce1c163800 2024-03-06T07:30:52.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce22e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c06c990 con 0x7fce1c15cb90 2024-03-06T07:30:52.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce1be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c06d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce1b400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fce1c163800 0x7fce1c0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce1be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c06d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:50252/0 (socket says 172.21.15.114:50252) 2024-03-06T07:30:52.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce1be00640 1 -- 172.21.15.114:0/2073692273 learned_addr learned my addr 172.21.15.114:0/2073692273 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:52.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.572+0000 7fce1b400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fce1c163800 0x7fce1c0b2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:54482/0 (socket says 172.21.15.114:54482) 2024-03-06T07:30:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.573+0000 7fce1b400640 1 -- 172.21.15.114:0/2073692273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 msgr2=0x7fce1c06d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.573+0000 7fce1b400640 1 --2- 172.21.15.114:0/2073692273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c06d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.573+0000 7fce1b400640 1 -- 172.21.15.114:0/2073692273 >> v1:172.21.15.177:6789/0 conn(0x7fce1c06e060 legacy=0x7fce1c0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.573+0000 7fce1b400640 1 -- 172.21.15.114:0/2073692273 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c0b35b0 con 0x7fce1c163800 2024-03-06T07:30:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.575+0000 7fce1b400640 1 --2- 172.21.15.114:0/2073692273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0b2c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fce080097a0 tx=0x7fce0802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2de3124275fc0117 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.575+0000 7fce1aa00640 1 -- 172.21.15.114:0/2073692273 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08035020 con 0x7fce1c163800 2024-03-06T07:30:52.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.575+0000 7fce1aa00640 1 -- 172.21.15.114:0/2073692273 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce08031830 con 0x7fce1c163800 2024-03-06T07:30:52.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.575+0000 7fce1aa00640 1 -- 172.21.15.114:0/2073692273 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08031c00 con 0x7fce1c163800 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 -- 172.21.15.114:0/2073692273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 msgr2=0x7fce1c0b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 --2- 172.21.15.114:0/2073692273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0b2c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fce080097a0 tx=0x7fce0802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 -- 172.21.15.114:0/2073692273 shutdown_connections 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 --2- 172.21.15.114:0/2073692273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0b2c40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 --2- 172.21.15.114:0/2073692273 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c06d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 -- 172.21.15.114:0/2073692273 >> 172.21.15.114:0/2073692273 conn(0x7fce1c0b1940 msgr2=0x7fce1c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 -- 172.21.15.114:0/2073692273 shutdown_connections 2024-03-06T07:30:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.576+0000 7fce22e00640 1 -- 172.21.15.114:0/2073692273 wait complete. 2024-03-06T07:30:52.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.579+0000 7fce22e00640 1 Processor -- start 2024-03-06T07:30:52.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.579+0000 7fce22e00640 1 -- start start 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce22e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce1c06e060 0x7fce1c156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce22e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c09b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce22e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce22e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c06cec0 con 0x7fce1c06e060 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce22e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c06d230 con 0x7fce1c163800 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce22e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c06dc00 con 0x7fce1c15cb90 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce1be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce1c06e060 0x7fce1c156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce1be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce1c06e060 0x7fce1c156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48908/0 (socket says 172.21.15.114:48908) 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce1be00640 1 -- 172.21.15.114:0/3783559666 learned_addr learned my addr 172.21.15.114:0/3783559666 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce20e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.581+0000 7fce1b400640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c09b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce1be00640 1 -- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 msgr2=0x7fce1c09b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce1be00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c09b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce1be00640 1 -- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 msgr2=0x7fce1c0ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce1be00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0ae2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce1be00640 1 -- 172.21.15.114:0/3783559666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1c09cee0 con 0x7fce1c06e060 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce20e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0ae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce1b400640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c09b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce1be00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce1c06e060 0x7fce1c156100 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009fd0 tx=0x7fce0c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce13e00640 1 -- 172.21.15.114:0/3783559666 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c017070 con 0x7fce1c06e060 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce13e00640 1 -- 172.21.15.114:0/3783559666 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c00c940 con 0x7fce1c06e060 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.582+0000 7fce13e00640 1 -- 172.21.15.114:0/3783559666 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c00cd10 con 0x7fce1c06e060 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.583+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c09d1c0 con 0x7fce1c06e060 2024-03-06T07:30:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.583+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c09d780 con 0x7fce1c06e060 2024-03-06T07:30:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.584+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1c04f330 con 0x7fce1c06e060 2024-03-06T07:30:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.585+0000 7fce13e00640 1 -- 172.21.15.114:0/3783559666 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce0c01a070 con 0x7fce1c06e060 2024-03-06T07:30:52.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.586+0000 7fce13e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcdfc06b4d0 0x7fcdfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.586+0000 7fce13e00640 1 -- 172.21.15.114:0/3783559666 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce0c08a200 con 0x7fce1c06e060 2024-03-06T07:30:52.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.586+0000 7fce1b400640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcdfc06b4d0 0x7fcdfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.587+0000 7fce1b400640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcdfc06b4d0 0x7fcdfc06d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fce08036000 tx=0x7fce0803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.590+0000 7fce13e00640 1 -- 172.21.15.114:0/3783559666 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce0c059070 con 0x7fce1c06e060 2024-03-06T07:30:52.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.819+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce1c0b2530 con 0x7fce1c06e060 2024-03-06T07:30:52.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.820+0000 7fce13e00640 1 -- 172.21.15.114:0/3783559666 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce0c058970 con 0x7fce1c06e060 2024-03-06T07:30:52.822 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:30:52.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcdfc06b4d0 msgr2=0x7fcdfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcdfc06b4d0 0x7fcdfc06d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fce08036000 tx=0x7fce0803c040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce1c06e060 msgr2=0x7fce1c156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce1c06e060 0x7fce1c156100 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009fd0 tx=0x7fce0c00ed40 comp rx=0 tx=0).stop 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 shutdown_connections 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce1c163800 0x7fce1c0ae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fce1c15cb90 0x7fce1c09b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcdfc06b4d0 0x7fcdfc06d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 --2- 172.21.15.114:0/3783559666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce1c06e060 0x7fce1c156100 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 >> 172.21.15.114:0/3783559666 conn(0x7fce1c0b1940 msgr2=0x7fce1c0b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 shutdown_connections 2024-03-06T07:30:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:52.823+0000 7fce22e00640 1 -- 172.21.15.114:0/3783559666 wait complete. 2024-03-06T07:30:52.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:53.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.069+0000 7fb279c00640 1 Processor -- start 2024-03-06T07:30:53.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.069+0000 7fb279c00640 1 -- start start 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.070+0000 7fb279c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb27415bb90 0x7fb27406dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.070+0000 7fb279c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb27415efa0 0x7fb2740cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.070+0000 7fb279c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb274071ea0 con 0x7fb27406afc0 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.070+0000 7fb279c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27415bf70 con 0x7fb27415bb90 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.070+0000 7fb279c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27406e380 con 0x7fb27415efa0 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb27415bb90 0x7fb27406dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb27415bb90 0x7fb27406dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48912/0 (socket says 172.21.15.114:48912) 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272a00640 1 -- 172.21.15.114:0/1976820883 learned_addr learned my addr 172.21.15.114:0/1976820883 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272000640 1 --2- 172.21.15.114:0/1976820883 >> v2:172.21.15.177:3300/0 conn(0x7fb27415efa0 0x7fb2740cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272a00640 1 -- 172.21.15.114:0/1976820883 >> v2:172.21.15.177:3300/0 conn(0x7fb27415efa0 msgr2=0x7fb2740cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272a00640 1 --2- 172.21.15.114:0/1976820883 >> v2:172.21.15.177:3300/0 conn(0x7fb27415efa0 0x7fb2740cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272a00640 1 -- 172.21.15.114:0/1976820883 >> v1:172.21.15.177:6789/0 conn(0x7fb27406afc0 legacy=0x7fb2740723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272a00640 1 -- 172.21.15.114:0/1976820883 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27406e660 con 0x7fb27415bb90 2024-03-06T07:30:53.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.071+0000 7fb272000640 1 --2- 172.21.15.114:0/1976820883 >> v2:172.21.15.177:3300/0 conn(0x7fb27415efa0 0x7fb2740cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.073+0000 7fb272a00640 1 --2- 172.21.15.114:0/1976820883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27415bb90 0x7fb27406dcf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb264009870 tx=0x7fb26402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c4257735e440c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.073+0000 7fb271600640 1 -- 172.21.15.114:0/1976820883 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264035020 con 0x7fb27415bb90 2024-03-06T07:30:53.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.073+0000 7fb271600640 1 -- 172.21.15.114:0/1976820883 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb264031870 con 0x7fb27415bb90 2024-03-06T07:30:53.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.073+0000 7fb271600640 1 -- 172.21.15.114:0/1976820883 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264031c40 con 0x7fb27415bb90 2024-03-06T07:30:53.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.073+0000 7fb279c00640 1 -- 172.21.15.114:0/1976820883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27415bb90 msgr2=0x7fb27406dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.073+0000 7fb279c00640 1 --2- 172.21.15.114:0/1976820883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27415bb90 0x7fb27406dcf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb264009870 tx=0x7fb26402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:53.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.074+0000 7fb279c00640 1 -- 172.21.15.114:0/1976820883 shutdown_connections 2024-03-06T07:30:53.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.074+0000 7fb279c00640 1 --2- 172.21.15.114:0/1976820883 >> v2:172.21.15.177:3300/0 conn(0x7fb27415efa0 0x7fb2740cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.074+0000 7fb279c00640 1 --2- 172.21.15.114:0/1976820883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27415bb90 0x7fb27406dcf0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.074+0000 7fb279c00640 1 -- 172.21.15.114:0/1976820883 >> 172.21.15.114:0/1976820883 conn(0x7fb2740b27d0 msgr2=0x7fb2740b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.074+0000 7fb279c00640 1 -- 172.21.15.114:0/1976820883 shutdown_connections 2024-03-06T07:30:53.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.074+0000 7fb279c00640 1 -- 172.21.15.114:0/1976820883 wait complete. 2024-03-06T07:30:53.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.075+0000 7fb279c00640 1 Processor -- start 2024-03-06T07:30:53.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.075+0000 7fb279c00640 1 -- start start 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.076+0000 7fb279c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27406afc0 0x7fb27413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.076+0000 7fb279c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27415bb90 0x7fb274142850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.076+0000 7fb279c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27415efa0 0x7fb274140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.076+0000 7fb279c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27415f430 con 0x7fb27406afc0 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.076+0000 7fb279c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27406ec20 con 0x7fb27415bb90 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.076+0000 7fb279c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740cf160 con 0x7fb27415efa0 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.076+0000 7fb272a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27406afc0 0x7fb27413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27406afc0 0x7fb27413fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48924/0 (socket says 172.21.15.114:48924) 2024-03-06T07:30:53.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272a00640 1 -- 172.21.15.114:0/4196760976 learned_addr learned my addr 172.21.15.114:0/4196760976 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272000640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27415bb90 0x7fb274142850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb273400640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27415efa0 0x7fb274140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272a00640 1 -- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27415efa0 msgr2=0x7fb274140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272a00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27415efa0 0x7fb274140810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272a00640 1 -- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27415bb90 msgr2=0x7fb274142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272a00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27415bb90 0x7fb274142850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb272a00640 1 -- 172.21.15.114:0/4196760976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274143210 con 0x7fb27406afc0 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.077+0000 7fb273400640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27415efa0 0x7fb274140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.078+0000 7fb272a00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27406afc0 0x7fb27413fe60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb264009870 tx=0x7fb2640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.078+0000 7fb26aa00640 1 -- 172.21.15.114:0/4196760976 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264040070 con 0x7fb27406afc0 2024-03-06T07:30:53.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.078+0000 7fb26aa00640 1 -- 172.21.15.114:0/4196760976 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26403a420 con 0x7fb27406afc0 2024-03-06T07:30:53.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.078+0000 7fb26aa00640 1 -- 172.21.15.114:0/4196760976 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264035040 con 0x7fb27406afc0 2024-03-06T07:30:53.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.078+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2741434f0 con 0x7fb27406afc0 2024-03-06T07:30:53.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.078+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2741403a0 con 0x7fb27406afc0 2024-03-06T07:30:53.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.079+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27404f330 con 0x7fb27406afc0 2024-03-06T07:30:53.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.080+0000 7fb26aa00640 1 -- 172.21.15.114:0/4196760976 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb26404b060 con 0x7fb27406afc0 2024-03-06T07:30:53.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.081+0000 7fb26aa00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb25406b520 0x7fb25406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.081+0000 7fb26aa00640 1 -- 172.21.15.114:0/4196760976 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2640ac190 con 0x7fb27406afc0 2024-03-06T07:30:53.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.082+0000 7fb272000640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb25406b520 0x7fb25406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.082+0000 7fb272000640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb25406b520 0x7fb25406d9e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb2740b1ef0 tx=0x7fb260008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.090+0000 7fb26aa00640 1 -- 172.21.15.114:0/4196760976 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb26407aff0 con 0x7fb27406afc0 2024-03-06T07:30:53.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.289+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2740cdf60 con 0x7fb27406afc0 2024-03-06T07:30:53.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.290+0000 7fb26aa00640 1 -- 172.21.15.114:0/4196760976 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb26407a8f0 con 0x7fb27406afc0 2024-03-06T07:30:53.292 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:30:53.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.292+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb25406b520 msgr2=0x7fb25406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.292+0000 7fb279c00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb25406b520 0x7fb25406d9e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb2740b1ef0 tx=0x7fb260008040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.292+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27406afc0 msgr2=0x7fb27413fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.292+0000 7fb279c00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27406afc0 0x7fb27413fe60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb264009870 tx=0x7fb2640046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 shutdown_connections 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb27415efa0 0x7fb274140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb27415bb90 0x7fb274142850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb25406b520 0x7fb25406d9e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 --2- 172.21.15.114:0/4196760976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb27406afc0 0x7fb27413fe60 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 >> 172.21.15.114:0/4196760976 conn(0x7fb2740b27d0 msgr2=0x7fb27415e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 shutdown_connections 2024-03-06T07:30:53.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.293+0000 7fb279c00640 1 -- 172.21.15.114:0/4196760976 wait complete. 2024-03-06T07:30:53.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:53.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.519+0000 7ff4b7200640 1 Processor -- start 2024-03-06T07:30:53.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.519+0000 7ff4b7200640 1 -- start start 2024-03-06T07:30:53.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.520+0000 7ff4b7200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b815e130 0x7ff4b80cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.520+0000 7ff4b7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4b80d25b0 0x7ff4b80c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.520+0000 7ff4b7200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b80d13e0 con 0x7ff4b80cfaf0 2024-03-06T07:30:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b80cf120 con 0x7ff4b80d25b0 2024-03-06T07:30:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b7200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80cead0 con 0x7ff4b815e130 2024-03-06T07:30:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4b80d25b0 0x7ff4b80c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff4b80d25b0 0x7ff4b80c7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48940/0 (socket says 172.21.15.114:48940) 2024-03-06T07:30:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b6800640 1 -- 172.21.15.114:0/1626764648 learned_addr learned my addr 172.21.15.114:0/1626764648 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b5e00640 1 --2- 172.21.15.114:0/1626764648 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b815e130 0x7ff4b80cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b6800640 1 -- 172.21.15.114:0/1626764648 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b815e130 msgr2=0x7ff4b80cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b6800640 1 --2- 172.21.15.114:0/1626764648 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b815e130 0x7ff4b80cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b6800640 1 -- 172.21.15.114:0/1626764648 >> v1:172.21.15.177:6789/0 conn(0x7ff4b80cfaf0 legacy=0x7ff4b80d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.521+0000 7ff4b6800640 1 -- 172.21.15.114:0/1626764648 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b80cedb0 con 0x7ff4b80d25b0 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.522+0000 7ff4b6800640 1 --2- 172.21.15.114:0/1626764648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80c7970 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009e50 tx=0x7ff4b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726b109ddc7523b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.523+0000 7ff4b4a00640 1 -- 172.21.15.114:0/1626764648 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b002ca70 con 0x7ff4b80d25b0 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.523+0000 7ff4b4a00640 1 -- 172.21.15.114:0/1626764648 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b002cc10 con 0x7ff4b80d25b0 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.523+0000 7ff4b4a00640 1 -- 172.21.15.114:0/1626764648 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b00343f0 con 0x7ff4b80d25b0 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 -- 172.21.15.114:0/1626764648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 msgr2=0x7ff4b80c7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 --2- 172.21.15.114:0/1626764648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80c7970 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009e50 tx=0x7ff4b002bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 -- 172.21.15.114:0/1626764648 shutdown_connections 2024-03-06T07:30:53.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 --2- 172.21.15.114:0/1626764648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80c7970 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 --2- 172.21.15.114:0/1626764648 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b815e130 0x7ff4b80cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 -- 172.21.15.114:0/1626764648 >> 172.21.15.114:0/1626764648 conn(0x7ff4b809d070 msgr2=0x7ff4b80d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 -- 172.21.15.114:0/1626764648 shutdown_connections 2024-03-06T07:30:53.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.524+0000 7ff4b7200640 1 -- 172.21.15.114:0/1626764648 wait complete. 2024-03-06T07:30:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.526+0000 7ff4b7200640 1 Processor -- start 2024-03-06T07:30:53.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.527+0000 7ff4b7200640 1 -- start start 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b7200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b80cfaf0 0x7ff4b80b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4b815e130 0x7ff4b80b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80c8030 con 0x7ff4b80d25b0 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80c7eb0 con 0x7ff4b815e130 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b7200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80c81b0 con 0x7ff4b80cfaf0 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80b83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48946/0 (socket says 172.21.15.114:48946) 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.528+0000 7ff4b5400640 1 -- 172.21.15.114:0/2377409396 learned_addr learned my addr 172.21.15.114:0/2377409396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:53.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b5e00640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b80cfaf0 0x7ff4b80b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b6800640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4b815e130 0x7ff4b80b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b5400640 1 -- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b80cfaf0 msgr2=0x7ff4b80b7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b5400640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b80cfaf0 0x7ff4b80b7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b5400640 1 -- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4b815e130 msgr2=0x7ff4b80b8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b5400640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4b815e130 0x7ff4b80b8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b5400640 1 -- 172.21.15.114:0/2377409396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b815c940 con 0x7ff4b80d25b0 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b6800640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4b815e130 0x7ff4b80b8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4b5400640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80b83c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400b3c0 tx=0x7ff4a400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.529+0000 7ff4ae000640 1 -- 172.21.15.114:0/2377409396 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4011830 con 0x7ff4b80d25b0 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.530+0000 7ff4ae000640 1 -- 172.21.15.114:0/2377409396 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a4016070 con 0x7ff4b80d25b0 2024-03-06T07:30:53.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.530+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b809f160 con 0x7ff4b80d25b0 2024-03-06T07:30:53.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.530+0000 7ff4ae000640 1 -- 172.21.15.114:0/2377409396 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4011c10 con 0x7ff4b80d25b0 2024-03-06T07:30:53.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.530+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b809f720 con 0x7ff4b80d25b0 2024-03-06T07:30:53.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.531+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b804f330 con 0x7ff4b80d25b0 2024-03-06T07:30:53.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.532+0000 7ff4ae000640 1 -- 172.21.15.114:0/2377409396 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4a401b460 con 0x7ff4b80d25b0 2024-03-06T07:30:53.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.533+0000 7ff4ae000640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff49806b4d0 0x7ff49806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.533+0000 7ff4ae000640 1 -- 172.21.15.114:0/2377409396 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4a4089cb0 con 0x7ff4b80d25b0 2024-03-06T07:30:53.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.533+0000 7ff4b5e00640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff49806b4d0 0x7ff49806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.533+0000 7ff4b5e00640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff49806b4d0 0x7ff49806d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8006fd0 tx=0x7ff4a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.537+0000 7ff4ae000640 1 -- 172.21.15.114:0/2377409396 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4a4020020 con 0x7ff4b80d25b0 2024-03-06T07:30:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.731+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff4b809d690 con 0x7ff4b80d25b0 2024-03-06T07:30:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.732+0000 7ff4ae000640 1 -- 172.21.15.114:0/2377409396 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4b815c940 con 0x7ff4b80d25b0 2024-03-06T07:30:53.734 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:30:53.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.734+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff49806b4d0 msgr2=0x7ff49806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.734+0000 7ff4b7200640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff49806b4d0 0x7ff49806d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8006fd0 tx=0x7ff4a8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.734+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 msgr2=0x7ff4b80b83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.734+0000 7ff4b7200640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80b83c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400b3c0 tx=0x7ff4a400b880 comp rx=0 tx=0).stop 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 shutdown_connections 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4b815e130 0x7ff4b80b8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff49806b4d0 0x7ff49806d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4b80d25b0 0x7ff4b80b83c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 --2- 172.21.15.114:0/2377409396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff4b80cfaf0 0x7ff4b80b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 >> 172.21.15.114:0/2377409396 conn(0x7ff4b809d070 msgr2=0x7ff4b80c8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 shutdown_connections 2024-03-06T07:30:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.735+0000 7ff4b7200640 1 -- 172.21.15.114:0/2377409396 wait complete. 2024-03-06T07:30:53.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:53.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.953+0000 7f5a00a00640 1 Processor -- start 2024-03-06T07:30:53.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.953+0000 7f5a00a00640 1 -- start start 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f5a00a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f59fc164b30 0x7f59fc166f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f5a00a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc145b30 con 0x7f59fc0c6460 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f5a00a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc142ce0 con 0x7f59fc0c4930 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f5a00a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0c68c0 con 0x7f59fc164b30 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f59f1400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f59fc0c6460 0x7f59fc0c5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49160/0 (socket says 172.21.15.114:49160) 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f59f1400640 1 -- 172.21.15.114:0/1171466097 learned_addr learned my addr 172.21.15.114:0/1171466097 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980360658 0 0) 0x7f59ec001110 con 0x7f59fc0c6460 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.955+0000 7f59fa800640 1 --2- 172.21.15.114:0/1171466097 >> v2:172.21.15.177:3300/0 conn(0x7f59fc164b30 0x7f59fc166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e0003520 con 0x7f59fc0c6460 2024-03-06T07:30:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552613085 0 0) 0x7f59e0003520 con 0x7f59fc0c6460 2024-03-06T07:30:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 >> v2:172.21.15.177:3300/0 conn(0x7f59fc164b30 msgr2=0x7f59fc166f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 --2- 172.21.15.114:0/1171466097 >> v2:172.21.15.177:3300/0 conn(0x7f59fc164b30 0x7f59fc166f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 >> v1:172.21.15.177:6789/0 conn(0x7f59fc0c4930 legacy=0x7f59fc0c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0cbca0 con 0x7f59fc0c6460 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f59ec001840 con 0x7f59fc0c6460 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 >> v1:172.21.15.114:6789/0 conn(0x7f59fc0c6460 legacy=0x7f59fc0c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e0003a00 0x7f59e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e0004800 0x7f59e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0c68c0 con 0x7f59e0005290 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e000c1e0 con 0x7f59e0003a00 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e000c3e0 con 0x7f59e0004800 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59fa800640 1 --2- 172.21.15.114:0/1171466097 >> v2:172.21.15.177:3300/0 conn(0x7f59fc164b30 0x7f59fc166f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.956+0000 7f59fa800640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.957+0000 7f59f1400640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e0004800 0x7f59e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.957+0000 7f59f9e00640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e0003a00 0x7f59e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.957+0000 7f59fa800640 1 -- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e0004800 msgr2=0x7f59e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.957+0000 7f59fa800640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e0004800 0x7f59e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.957+0000 7f59fa800640 1 -- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e0003a00 msgr2=0x7f59e0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.957+0000 7f59fa800640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e0003a00 0x7f59e0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.957+0000 7f59fa800640 1 -- 172.21.15.114:0/1171466097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e0003520 con 0x7f59e0005290 2024-03-06T07:30:53.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.958+0000 7f59fa800640 1 -- 172.21.15.114:0/1171466097 reap_dead start 2024-03-06T07:30:53.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.958+0000 7f59fa800640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59e000b9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f59f40073b0 tx=0x7f59e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab786638ad1f32e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.958+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4039070 con 0x7f59e0005290 2024-03-06T07:30:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.958+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f402cd90 con 0x7f59e0005290 2024-03-06T07:30:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.958+0000 7f59f9400640 1 -- 172.21.15.114:0/1171466097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4009c70 con 0x7f59e0005290 2024-03-06T07:30:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.959+0000 7f5a00a00640 1 -- 172.21.15.114:0/1171466097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 msgr2=0x7f59e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.959+0000 7f5a00a00640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59e000b9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f59f40073b0 tx=0x7f59e0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.959+0000 7f5a00a00640 1 -- 172.21.15.114:0/1171466097 shutdown_connections 2024-03-06T07:30:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.959+0000 7f5a00a00640 1 --2- 172.21.15.114:0/1171466097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59e000b9c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.959+0000 7f5a00a00640 1 -- 172.21.15.114:0/1171466097 >> 172.21.15.114:0/1171466097 conn(0x7f59fc0cc4f0 msgr2=0x7f59fc0cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.959+0000 7f5a00a00640 1 -- 172.21.15.114:0/1171466097 shutdown_connections 2024-03-06T07:30:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.959+0000 7f5a00a00640 1 -- 172.21.15.114:0/1171466097 wait complete. 2024-03-06T07:30:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.961+0000 7f5a00a00640 1 Processor -- start 2024-03-06T07:30:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.961+0000 7f5a00a00640 1 -- start start 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f5a00a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59fc140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f5a00a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc141d60 0x7f59fc0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f5a00a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc071d40 0x7f59fc159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f5a00a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0ca3b0 con 0x7f59e0005290 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f5a00a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0ca0d0 con 0x7f59fc071d40 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f5a00a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0cc260 con 0x7f59fc141d60 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59fc140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59fc140220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48958/0 (socket says 172.21.15.114:48958) 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 -- 172.21.15.114:0/709020118 learned_addr learned my addr 172.21.15.114:0/709020118 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f1400640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc141d60 0x7f59fc0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59fa800640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc071d40 0x7f59fc159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 -- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc141d60 msgr2=0x7f59fc0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc141d60 0x7f59fc0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 -- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc071d40 msgr2=0x7f59fc159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc071d40 0x7f59fc159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.962+0000 7f59f9e00640 1 -- 172.21.15.114:0/709020118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0c8310 con 0x7f59e0005290 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.963+0000 7f59f9e00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59fc140220 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f59e400e460 tx=0x7f59e400e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.963+0000 7f59f2a00640 1 -- 172.21.15.114:0/709020118 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40135f0 con 0x7f59e0005290 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.963+0000 7f59f2a00640 1 -- 172.21.15.114:0/709020118 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e401a070 con 0x7f59e0005290 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.963+0000 7f59f2a00640 1 -- 172.21.15.114:0/709020118 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40139d0 con 0x7f59e0005290 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.963+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc140800 con 0x7f59e0005290 2024-03-06T07:30:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.963+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc140dc0 con 0x7f59e0005290 2024-03-06T07:30:53.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.965+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc04f330 con 0x7f59e0005290 2024-03-06T07:30:53.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.965+0000 7f59f2a00640 1 -- 172.21.15.114:0/709020118 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59e4013b70 con 0x7f59e0005290 2024-03-06T07:30:53.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.967+0000 7f59f2a00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59d806b4d0 0x7f59d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.967+0000 7f59f2a00640 1 -- 172.21.15.114:0/709020118 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59e408a2c0 con 0x7f59e0005290 2024-03-06T07:30:53.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.967+0000 7f59f1400640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59d806b4d0 0x7f59d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.968+0000 7f59f1400640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59d806b4d0 0x7f59d806d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0012f0 tx=0x7f59ec000f70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:53.971+0000 7f59f2a00640 1 -- 172.21.15.114:0/709020118 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59e4024020 con 0x7f59e0005290 2024-03-06T07:30:54.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.178+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f59fc0cd590 con 0x7f59e0005290 2024-03-06T07:30:54.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.179+0000 7f59f2a00640 1 -- 172.21.15.114:0/709020118 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) v1 ==== 80+0+0 (secure 0 0 0) 0x7f59e4059130 con 0x7f59e0005290 2024-03-06T07:30:54.181 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59d806b4d0 msgr2=0x7f59d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59d806b4d0 0x7f59d806d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0012f0 tx=0x7f59ec000f70 comp rx=0 tx=0).stop 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 msgr2=0x7f59fc140220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59fc140220 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f59e400e460 tx=0x7f59e400e920 comp rx=0 tx=0).stop 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 shutdown_connections 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59fc071d40 0x7f59fc159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59fc141d60 0x7f59fc0c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59d806b4d0 0x7f59d806d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 --2- 172.21.15.114:0/709020118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59e0005290 0x7f59fc140220 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 >> 172.21.15.114:0/709020118 conn(0x7f59fc0cc4f0 msgr2=0x7f59fc06d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 shutdown_connections 2024-03-06T07:30:54.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.182+0000 7f5a00a00640 1 -- 172.21.15.114:0/709020118 wait complete. 2024-03-06T07:30:54.204 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:30:54.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:54.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:54.373 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:30:54.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:54.373 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:54.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.659+0000 7f7f7b200640 1 Processor -- start 2024-03-06T07:30:54.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.660+0000 7f7f7b200640 1 -- start start 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.661+0000 7f7f7b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7f7c0aedc0 0x7f7f7c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.661+0000 7f7f7b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.661+0000 7f7f7b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c15c240 con 0x7f7f7c1561f0 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.661+0000 7f7f7b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7c15ce30 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.661+0000 7f7f7b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c15c500 con 0x7f7f7c0cc800 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.661+0000 7f7f71400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7f7c0aedc0 0x7f7f7c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.661+0000 7f7f79e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7f7c1561f0 0x7f7f7c0ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49174/0 (socket says 172.21.15.114:49174) 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.662+0000 7f7f79e00640 1 -- 172.21.15.114:0/2166197144 learned_addr learned my addr 172.21.15.114:0/2166197144 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:54.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.662+0000 7f7f7a800640 1 --2- 172.21.15.114:0/2166197144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.662+0000 7f7f79400640 1 -- 172.21.15.114:0/2166197144 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698067962 0 0) 0x7f7f7c15c240 con 0x7f7f7c1561f0 2024-03-06T07:30:54.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.662+0000 7f7f71400640 1 -- 172.21.15.114:0/2166197144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 msgr2=0x7f7f7c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.662+0000 7f7f71400640 1 --2- 172.21.15.114:0/2166197144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.662+0000 7f7f71400640 1 -- 172.21.15.114:0/2166197144 >> v1:172.21.15.114:6789/0 conn(0x7f7f7c1561f0 legacy=0x7f7f7c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.662+0000 7f7f71400640 1 -- 172.21.15.114:0/2166197144 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c15c7e0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.664+0000 7f7f71400640 1 --2- 172.21.15.114:0/2166197144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c0cc2c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7f640097a0 tx=0x7f7f6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99912d27ae3f3ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.664+0000 7f7f79400640 1 -- 172.21.15.114:0/2166197144 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64035020 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.664+0000 7f7f79400640 1 -- 172.21.15.114:0/2166197144 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f64031830 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.664+0000 7f7f79400640 1 -- 172.21.15.114:0/2166197144 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f64031c00 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 -- 172.21.15.114:0/2166197144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 msgr2=0x7f7f7c0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 --2- 172.21.15.114:0/2166197144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c0cc2c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7f640097a0 tx=0x7f7f6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 -- 172.21.15.114:0/2166197144 shutdown_connections 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 --2- 172.21.15.114:0/2166197144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 --2- 172.21.15.114:0/2166197144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c0cc2c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 -- 172.21.15.114:0/2166197144 >> 172.21.15.114:0/2166197144 conn(0x7f7f7c06afc0 msgr2=0x7f7f7c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 -- 172.21.15.114:0/2166197144 shutdown_connections 2024-03-06T07:30:54.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.665+0000 7f7f7b200640 1 -- 172.21.15.114:0/2166197144 wait complete. 2024-03-06T07:30:54.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.667+0000 7f7f7b200640 1 Processor -- start 2024-03-06T07:30:54.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.668+0000 7f7f7b200640 1 -- start start 2024-03-06T07:30:54.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.669+0000 7f7f7b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c1e43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.669+0000 7f7f7b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c1f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.669+0000 7f7f7b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f7c1561f0 0x7f7f7c152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.669+0000 7f7f7b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c1565d0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.669+0000 7f7f7b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0ccc70 con 0x7f7f7c1561f0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.669+0000 7f7f7b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c156750 con 0x7f7f7c0cc800 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c1e43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c1e43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48988/0 (socket says 172.21.15.114:48988) 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 -- 172.21.15.114:0/1279902310 learned_addr learned my addr 172.21.15.114:0/1279902310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 -- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 msgr2=0x7f7f7c1f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f71400640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c1f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f7a800640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f7c1561f0 0x7f7f7c152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c1f2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 -- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f7c1561f0 msgr2=0x7f7f7c152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f7c1561f0 0x7f7f7c152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 -- 172.21.15.114:0/1279902310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c1f1b90 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.670+0000 7f7f79e00640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c1e43d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00c430 tx=0x7f7f6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.671+0000 7f7f72a00640 1 -- 172.21.15.114:0/1279902310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c0074e0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.671+0000 7f7f72a00640 1 -- 172.21.15.114:0/1279902310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6c0047e0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.671+0000 7f7f72a00640 1 -- 172.21.15.114:0/1279902310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c004bd0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.671+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c1e4ad0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.671+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7c1f2860 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.672+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7c155770 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.673+0000 7f7f72a00640 1 -- 172.21.15.114:0/1279902310 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f6c011070 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.674+0000 7f7f72a00640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f5806b4d0 0x7f7f5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.674+0000 7f7f72a00640 1 -- 172.21.15.114:0/1279902310 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f6c089e70 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.675+0000 7f7f71400640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f5806b4d0 0x7f7f5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.675+0000 7f7f71400640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f5806b4d0 0x7f7f5806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7f64036000 tx=0x7f7f64008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.679+0000 7f7f72a00640 1 -- 172.21.15.114:0/1279902310 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f6c058cf0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.949+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f7c04f330 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.950+0000 7f7f72a00640 1 -- 172.21.15.114:0/1279902310 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f7f6c0585f0 con 0x7f7f7c0aedc0 2024-03-06T07:30:54.951 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:54.951 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:30:54.952 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f5806b4d0 msgr2=0x7f7f5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f5806b4d0 0x7f7f5806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7f64036000 tx=0x7f7f64008030 comp rx=0 tx=0).stop 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 msgr2=0x7f7f7c1e43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c1e43d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00c430 tx=0x7f7f6c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 shutdown_connections 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f7c1561f0 0x7f7f7c152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f7c0cc800 0x7f7f7c1f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f5806b4d0 0x7f7f5806d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 --2- 172.21.15.114:0/1279902310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f7c0aedc0 0x7f7f7c1e43d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 >> 172.21.15.114:0/1279902310 conn(0x7f7f7c06afc0 msgr2=0x7f7f7c071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 shutdown_connections 2024-03-06T07:30:54.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:54.952+0000 7f7f7b200640 1 -- 172.21.15.114:0/1279902310 wait complete. 2024-03-06T07:30:54.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:30:55.178 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:30:55.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:55.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.365+0000 7f1611000640 1 Processor -- start 2024-03-06T07:30:55.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.365+0000 7f1611000640 1 -- start start 2024-03-06T07:30:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.367+0000 7f1611000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f160c0cfc30 0x7f160c0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f1611000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f160c0d0bf0 0x7f160c0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f1611000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f1611000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160c0af8e0 con 0x7f160c0cfc30 2024-03-06T07:30:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f1611000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160c0afbc0 con 0x7f160c0d0bf0 2024-03-06T07:30:55.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f1611000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160c0afea0 con 0x7f160c0d0010 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f160c0cfc30 0x7f160c0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f160c0cfc30 0x7f160c0d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49002/0 (socket says 172.21.15.114:49002) 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 -- 172.21.15.114:0/3346732399 learned_addr learned my addr 172.21.15.114:0/3346732399 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160be00640 1 --2- 172.21.15.114:0/3346732399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160aa00640 1 --2- 172.21.15.114:0/3346732399 >> v2:172.21.15.177:3300/0 conn(0x7f160c0d0bf0 0x7f160c0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 -- 172.21.15.114:0/3346732399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 msgr2=0x7f160c0727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 --2- 172.21.15.114:0/3346732399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 -- 172.21.15.114:0/3346732399 >> v2:172.21.15.177:3300/0 conn(0x7f160c0d0bf0 msgr2=0x7f160c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 --2- 172.21.15.114:0/3346732399 >> v2:172.21.15.177:3300/0 conn(0x7f160c0d0bf0 0x7f160c0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.368+0000 7f160b400640 1 -- 172.21.15.114:0/3346732399 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c0b0180 con 0x7f160c0cfc30 2024-03-06T07:30:55.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.369+0000 7f160aa00640 1 --2- 172.21.15.114:0/3346732399 >> v2:172.21.15.177:3300/0 conn(0x7f160c0d0bf0 0x7f160c0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.369+0000 7f160b400640 1 --2- 172.21.15.114:0/3346732399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0cfc30 0x7f160c0d06b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f160c06cf30 tx=0x7f15fc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f868591f365fda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.370+0000 7f160a000640 1 -- 172.21.15.114:0/3346732399 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc035020 con 0x7f160c0cfc30 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.370+0000 7f160a000640 1 -- 172.21.15.114:0/3346732399 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15fc02d040 con 0x7f160c0cfc30 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.370+0000 7f160a000640 1 -- 172.21.15.114:0/3346732399 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15fc031a80 con 0x7f160c0cfc30 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 -- 172.21.15.114:0/3346732399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0cfc30 msgr2=0x7f160c0d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 --2- 172.21.15.114:0/3346732399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0cfc30 0x7f160c0d06b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f160c06cf30 tx=0x7f15fc02b900 comp rx=0 tx=0).stop 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 -- 172.21.15.114:0/3346732399 shutdown_connections 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 --2- 172.21.15.114:0/3346732399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 --2- 172.21.15.114:0/3346732399 >> v2:172.21.15.177:3300/0 conn(0x7f160c0d0bf0 0x7f160c0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 --2- 172.21.15.114:0/3346732399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0cfc30 0x7f160c0d06b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 -- 172.21.15.114:0/3346732399 >> 172.21.15.114:0/3346732399 conn(0x7f160c06c0d0 msgr2=0x7f160c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 -- 172.21.15.114:0/3346732399 shutdown_connections 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.371+0000 7f1611000640 1 -- 172.21.15.114:0/3346732399 wait complete. 2024-03-06T07:30:55.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.372+0000 7f1611000640 1 Processor -- start 2024-03-06T07:30:55.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.373+0000 7f1611000640 1 -- start start 2024-03-06T07:30:55.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.373+0000 7f1611000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f160c0cfc30 0x7f160c14c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.373+0000 7f1611000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.373+0000 7f1611000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0d0bf0 0x7f160c0b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f1611000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c06b390 con 0x7f160c0d0bf0 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f1611000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c06b210 con 0x7f160c0cfc30 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f1611000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160c06b510 con 0x7f160c0d0010 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0d0bf0 0x7f160c0b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0d0bf0 0x7f160c0b1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49018/0 (socket says 172.21.15.114:49018) 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 -- 172.21.15.114:0/2533654475 learned_addr learned my addr 172.21.15.114:0/2533654475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160aa00640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160b400640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f160c0cfc30 0x7f160c14c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 -- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 msgr2=0x7f160c0b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 -- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f160c0cfc30 msgr2=0x7f160c14c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f160c0cfc30 0x7f160c14c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 -- 172.21.15.114:0/2533654475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c1611c0 con 0x7f160c0d0bf0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.374+0000 7f160be00640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0d0bf0 0x7f160c0b1390 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1604007e90 tx=0x7f160400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.375+0000 7f1603400640 1 -- 172.21.15.114:0/2533654475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604014420 con 0x7f160c0d0bf0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.375+0000 7f1603400640 1 -- 172.21.15.114:0/2533654475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1604019070 con 0x7f160c0d0bf0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.375+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c156390 con 0x7f160c0d0bf0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.375+0000 7f1603400640 1 -- 172.21.15.114:0/2533654475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604014820 con 0x7f160c0d0bf0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.375+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c156950 con 0x7f160c0d0bf0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.376+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c04f330 con 0x7f160c0d0bf0 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.377+0000 7f1603400640 1 -- 172.21.15.114:0/2533654475 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16040149c0 con 0x7f160c0d0bf0 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.378+0000 7f1603400640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15ec06b4d0 0x7f15ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.378+0000 7f1603400640 1 -- 172.21.15.114:0/2533654475 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f160408aaa0 con 0x7f160c0d0bf0 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.378+0000 7f160b400640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15ec06b4d0 0x7f15ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.379+0000 7f160b400640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15ec06b4d0 0x7f15ec06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f15fc036000 tx=0x7f15fc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.382+0000 7f1603400640 1 -- 172.21.15.114:0/2533654475 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1604059910 con 0x7f160c0d0bf0 2024-03-06T07:30:55.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.606+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f160c14cb80 con 0x7f160c0d0bf0 2024-03-06T07:30:55.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.607+0000 7f1603400640 1 -- 172.21.15.114:0/2533654475 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f1604059210 con 0x7f160c0d0bf0 2024-03-06T07:30:55.609 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:55.609 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:30:55.610 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15ec06b4d0 msgr2=0x7f15ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15ec06b4d0 0x7f15ec06d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f15fc036000 tx=0x7f15fc03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0d0bf0 msgr2=0x7f160c0b1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0d0bf0 0x7f160c0b1390 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1604007e90 tx=0x7f160400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 shutdown_connections 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15ec06b4d0 0x7f15ec06d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160c0d0bf0 0x7f160c0b1390 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f160c0d0010 0x7f160c0b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 --2- 172.21.15.114:0/2533654475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f160c0cfc30 0x7f160c14c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 >> 172.21.15.114:0/2533654475 conn(0x7f160c06c0d0 msgr2=0x7f160c0b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 shutdown_connections 2024-03-06T07:30:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:55.610+0000 7f1611000640 1 -- 172.21.15.114:0/2533654475 wait complete. 2024-03-06T07:30:55.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:30:55.858 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:30:55.858 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:55.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:56.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.044+0000 7f9ae1e00640 1 Processor -- start 2024-03-06T07:30:56.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.044+0000 7f9ae1e00640 1 -- start start 2024-03-06T07:30:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.045+0000 7f9ae1e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9adc15eff0 0x7f9adc070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.045+0000 7f9ae1e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9adc06e4d0 0x7f9adc06e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.045+0000 7f9ae1e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc071d30 con 0x7f9adc0af1c0 2024-03-06T07:30:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.045+0000 7f9ae1e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9adc06ee80 con 0x7f9adc06e4d0 2024-03-06T07:30:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.045+0000 7f9ae1e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9adc071260 con 0x7f9adc15eff0 2024-03-06T07:30:56.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9adb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9adc06e4d0 0x7f9adc06e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9adaa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9adc0af1c0 0x7f9adc06cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49186/0 (socket says 172.21.15.114:49186) 2024-03-06T07:30:56.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9adaa00640 1 -- 172.21.15.114:0/1937520543 learned_addr learned my addr 172.21.15.114:0/1937520543 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:56.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9ada000640 1 --2- 172.21.15.114:0/1937520543 >> v2:172.21.15.177:3300/0 conn(0x7f9adc15eff0 0x7f9adc070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9adb400640 1 -- 172.21.15.114:0/1937520543 >> v2:172.21.15.177:3300/0 conn(0x7f9adc15eff0 msgr2=0x7f9adc070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9adb400640 1 --2- 172.21.15.114:0/1937520543 >> v2:172.21.15.177:3300/0 conn(0x7f9adc15eff0 0x7f9adc070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9adb400640 1 -- 172.21.15.114:0/1937520543 >> v1:172.21.15.114:6789/0 conn(0x7f9adc0af1c0 legacy=0x7f9adc06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.046+0000 7f9adb400640 1 -- 172.21.15.114:0/1937520543 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc15f4a0 con 0x7f9adc06e4d0 2024-03-06T07:30:56.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.047+0000 7f9adb400640 1 --2- 172.21.15.114:0/1937520543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc06e4d0 0x7f9adc06e940 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009d90 tx=0x7f9ad402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b7313f7329c8c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ad9600640 1 -- 172.21.15.114:0/1937520543 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4039070 con 0x7f9adc06e4d0 2024-03-06T07:30:56.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ad9600640 1 -- 172.21.15.114:0/1937520543 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad402cc60 con 0x7f9adc06e4d0 2024-03-06T07:30:56.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ad9600640 1 -- 172.21.15.114:0/1937520543 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4034660 con 0x7f9adc06e4d0 2024-03-06T07:30:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1937520543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc06e4d0 msgr2=0x7f9adc06e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1937520543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc06e4d0 0x7f9adc06e940 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009d90 tx=0x7f9ad402beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1937520543 shutdown_connections 2024-03-06T07:30:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1937520543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc06e4d0 0x7f9adc06e940 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1937520543 >> v2:172.21.15.177:3300/0 conn(0x7f9adc15eff0 0x7f9adc070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1937520543 >> 172.21.15.114:0/1937520543 conn(0x7f9adc0b1730 msgr2=0x7f9adc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1937520543 shutdown_connections 2024-03-06T07:30:56.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.048+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1937520543 wait complete. 2024-03-06T07:30:56.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.051+0000 7f9ae1e00640 1 Processor -- start 2024-03-06T07:30:56.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.051+0000 7f9ae1e00640 1 -- start start 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.052+0000 7f9ae1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9adc06e4d0 0x7f9adc148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.052+0000 7f9ae1e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9adc0af1c0 0x7f9adc14b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.052+0000 7f9ae1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc15eff0 0x7f9adc1f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.052+0000 7f9ae1e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc071a50 con 0x7f9adc15eff0 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.052+0000 7f9ae1e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc071770 con 0x7f9adc06e4d0 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.052+0000 7f9ae1e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0b30b0 con 0x7f9adc0af1c0 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc15eff0 0x7f9adc1f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc15eff0 0x7f9adc1f7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49036/0 (socket says 172.21.15.114:49036) 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 -- 172.21.15.114:0/1854939414 learned_addr learned my addr 172.21.15.114:0/1854939414 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adaa00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9adc06e4d0 0x7f9adc148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 -- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9adc0af1c0 msgr2=0x7f9adc14b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9ada000640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9adc0af1c0 0x7f9adc14b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9adc0af1c0 0x7f9adc14b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 -- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9adc06e4d0 msgr2=0x7f9adc148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9adc06e4d0 0x7f9adc148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adb400640 1 -- 172.21.15.114:0/1854939414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc1f7f30 con 0x7f9adc15eff0 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9adaa00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9adc06e4d0 0x7f9adc148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.053+0000 7f9ada000640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9adc0af1c0 0x7f9adc14b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:56.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.054+0000 7f9adb400640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc15eff0 0x7f9adc1f7570 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009eb0 tx=0x7f9ad4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.054+0000 7f9ad3400640 1 -- 172.21.15.114:0/1854939414 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4039050 con 0x7f9adc15eff0 2024-03-06T07:30:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.054+0000 7f9ad3400640 1 -- 172.21.15.114:0/1854939414 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad4045070 con 0x7f9adc15eff0 2024-03-06T07:30:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.054+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc1f8210 con 0x7f9adc15eff0 2024-03-06T07:30:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.054+0000 7f9ad3400640 1 -- 172.21.15.114:0/1854939414 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad403e830 con 0x7f9adc15eff0 2024-03-06T07:30:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.054+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc14acb0 con 0x7f9adc15eff0 2024-03-06T07:30:56.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.056+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc04f330 con 0x7f9adc15eff0 2024-03-06T07:30:56.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.056+0000 7f9ad3400640 1 -- 172.21.15.114:0/1854939414 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ad403e9d0 con 0x7f9adc15eff0 2024-03-06T07:30:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.057+0000 7f9ad3400640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ab806b4d0 0x7f9ab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.057+0000 7f9ad3400640 1 -- 172.21.15.114:0/1854939414 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ad40acfe0 con 0x7f9adc15eff0 2024-03-06T07:30:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.057+0000 7f9adaa00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ab806b4d0 0x7f9ab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.058+0000 7f9adaa00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ab806b4d0 0x7f9ab806d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9acc000f60 tx=0x7f9acc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.062+0000 7f9ad3400640 1 -- 172.21.15.114:0/1854939414 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9adc15f4a0 con 0x7f9adc15eff0 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.392+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9adc0b1510 con 0x7f9adc15eff0 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.393+0000 7f9ad3400640 1 -- 172.21.15.114:0/1854939414 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9ad4032e20 con 0x7f9adc15eff0 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ab806b4d0 msgr2=0x7f9ab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ab806b4d0 0x7f9ab806d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9acc000f60 tx=0x7f9acc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc15eff0 msgr2=0x7f9adc1f7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc15eff0 0x7f9adc1f7570 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009eb0 tx=0x7f9ad4004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 shutdown_connections 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9ab806b4d0 0x7f9ab806d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9adc15eff0 0x7f9adc1f7570 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9adc0af1c0 0x7f9adc14b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 --2- 172.21.15.114:0/1854939414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9adc06e4d0 0x7f9adc148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 >> 172.21.15.114:0/1854939414 conn(0x7f9adc0b1730 msgr2=0x7f9adc0b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 shutdown_connections 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:56.395+0000 7f9ae1e00640 1 -- 172.21.15.114:0/1854939414 wait complete. 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:56.415 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:56.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.416 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:56.458 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42399sec preferred_lft 42399sec 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:56.459 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:56.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.460 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:56.592 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:30:56.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.598 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:30:56.649 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:30:56.649 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:56.649 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:30:56.649 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:30:56.649 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:56.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.650 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:30:56.650 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:30:56.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:30:56.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:30:56.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:30:56.650 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:30:56.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.820 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:30:56.864 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:56.864 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:56.865 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:56.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:30:56.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:56.996 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:57.045 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:57.045 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:57.045 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:57.090 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:30:57.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:57.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:57.269 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:57.269 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:57.301 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:57.301 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:57.301 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:30:57.301 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:30:57.301 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:30:57.303 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:30:57.303 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:30:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:30:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:30:57.304 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:30:57.307 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:30:57.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:57.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:57.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:57.390 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:30:57.435 INFO:teuthology.orchestra.run.smithi114.stdout:client4919 2024-03-06T07:30:57.435 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:57.435 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:57.523 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:57.523 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> import uuid 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> n = 75 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = False 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> if 0 > 0: 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = True 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> dpath = os.path.dirname(path) 2024-03-06T07:30:57.613 DEBUG:teuthology.orchestra.run.smithi114:> fnameprefix = os.path.basename(path) 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> try: 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> for i in range(n): 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> f.write(f"{i}") 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> f.flush() 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(f.fileno()) 2024-03-06T07:30:57.614 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> os.unlink(fpath) 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> if create_hard_links: 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> for j in range(0): 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> if True: 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> finally: 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> os.close(dirfd) 2024-03-06T07:30:57.615 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:30:58.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:58.689 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:58.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:58.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:58.971 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:58.971 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:58.971 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:59.022 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:30:59.022 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:59.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:59.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.316+0000 7f7496400640 1 Processor -- start 2024-03-06T07:30:59.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.316+0000 7f7496400640 1 -- start start 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.318+0000 7f7496400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f749015fe60 0x7f7490071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.318+0000 7f7496400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749006cef0 con 0x7f749015f9d0 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.318+0000 7f7496400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749006d1d0 con 0x7f7490144c00 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.318+0000 7f7496400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749006d4b0 con 0x7f749015fe60 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.318+0000 7f748ea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f749015f9d0 0x7f7490070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33200/0 (socket says 172.21.15.114:33200) 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.318+0000 7f748ea00640 1 -- 172.21.15.114:0/3817745467 learned_addr learned my addr 172.21.15.114:0/3817745467 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.318+0000 7f748fe00640 1 --2- 172.21.15.114:0/3817745467 >> v2:172.21.15.177:3300/0 conn(0x7f749015fe60 0x7f7490071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948375687 0 0) 0x7f749006cef0 con 0x7f749015f9d0 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7474003520 con 0x7f749015f9d0 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 488002745 0 0) 0x7f7474003520 con 0x7f749015f9d0 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 >> v2:172.21.15.177:3300/0 conn(0x7f749015fe60 msgr2=0x7f7490071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 --2- 172.21.15.114:0/3817745467 >> v2:172.21.15.177:3300/0 conn(0x7f749015fe60 0x7f7490071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 >> v1:172.21.15.177:6789/0 conn(0x7f7490144c00 legacy=0x7f749006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749006d790 con 0x7f749015f9d0 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7478001670 con 0x7f749015f9d0 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 >> v1:172.21.15.114:6789/0 conn(0x7f749015f9d0 legacy=0x7f7490070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474003a00 0x7f7474003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7474004800 0x7f7474004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74740051c0 0x7f747400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749006d4b0 con 0x7f7474004800 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747400c0c0 con 0x7f74740051c0 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747400c2c0 con 0x7f7474003a00 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.319+0000 7f748ea00640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7474004800 0x7f7474004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.320+0000 7f748ea00640 1 -- 172.21.15.114:0/3817745467 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474003a00 msgr2=0x7f7474003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.320+0000 7f748ea00640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474003a00 0x7f7474003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.320+0000 7f748ea00640 1 -- 172.21.15.114:0/3817745467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74740051c0 msgr2=0x7f747400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.320+0000 7f748ea00640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74740051c0 0x7f747400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.320+0000 7f748ea00640 1 -- 172.21.15.114:0/3817745467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7474003520 con 0x7f7474004800 2024-03-06T07:30:59.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.320+0000 7f748fe00640 1 -- 172.21.15.114:0/3817745467 reap_dead start 2024-03-06T07:30:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.321+0000 7f748ea00640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7474004800 0x7f7474004c80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7478007c20 tx=0x7f7474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c835ce74e836eaad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.321+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478007c60 con 0x7f7474004800 2024-03-06T07:30:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.321+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478034070 con 0x7f7474004800 2024-03-06T07:30:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.321+0000 7f748e000640 1 -- 172.21.15.114:0/3817745467 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478005b10 con 0x7f7474004800 2024-03-06T07:30:59.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.322+0000 7f7496400640 1 -- 172.21.15.114:0/3817745467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7474004800 msgr2=0x7f7474004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.322+0000 7f7496400640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7474004800 0x7f7474004c80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7478007c20 tx=0x7f7474005830 comp rx=0 tx=0).stop 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.322+0000 7f7496400640 1 -- 172.21.15.114:0/3817745467 shutdown_connections 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.322+0000 7f7496400640 1 --2- 172.21.15.114:0/3817745467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7474004800 0x7f7474004c80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.322+0000 7f7496400640 1 -- 172.21.15.114:0/3817745467 >> 172.21.15.114:0/3817745467 conn(0x7f74900b1730 msgr2=0x7f74900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.322+0000 7f7496400640 1 -- 172.21.15.114:0/3817745467 shutdown_connections 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.322+0000 7f7496400640 1 -- 172.21.15.114:0/3817745467 wait complete. 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.324+0000 7f7496400640 1 Processor -- start 2024-03-06T07:30:59.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.324+0000 7f7496400640 1 -- start start 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f7496400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474004800 0x7f749015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f7496400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749013b0f0 0x7f7490142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f7496400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f749013a050 0x7f749013a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f7496400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749006cb50 con 0x7f749013b0f0 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f7496400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7490160730 con 0x7f749013a050 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f7496400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749006ccd0 con 0x7f7474004800 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f748ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749013b0f0 0x7f7490142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f748ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749013b0f0 0x7f7490142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51876/0 (socket says 172.21.15.114:51876) 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f748ea00640 1 -- 172.21.15.114:0/3353247266 learned_addr learned my addr 172.21.15.114:0/3353247266 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f748fe00640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f749013a050 0x7f749013a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.325+0000 7f748f400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474004800 0x7f749015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f748ea00640 1 -- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474004800 msgr2=0x7f749015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f748ea00640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474004800 0x7f749015b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f748ea00640 1 -- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f749013a050 msgr2=0x7f749013a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f748ea00640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f749013a050 0x7f749013a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f748ea00640 1 -- 172.21.15.114:0/3353247266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490159ca0 con 0x7f749013b0f0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f748fe00640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f749013a050 0x7f749013a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f748ea00640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749013b0f0 0x7f7490142e00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f74780308d0 tx=0x7f7478001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f7487400640 1 -- 172.21.15.114:0/3353247266 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478049070 con 0x7f749013b0f0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f7487400640 1 -- 172.21.15.114:0/3353247266 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478000ed0 con 0x7f749013b0f0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7490159f80 con 0x7f749013b0f0 2024-03-06T07:30:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f7487400640 1 -- 172.21.15.114:0/3353247266 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478007c60 con 0x7f749013b0f0 2024-03-06T07:30:59.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.326+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749015a540 con 0x7f749013b0f0 2024-03-06T07:30:59.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.327+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749004f330 con 0x7f749013b0f0 2024-03-06T07:30:59.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.329+0000 7f7487400640 1 -- 172.21.15.114:0/3353247266 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74780072b0 con 0x7f749013b0f0 2024-03-06T07:30:59.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.329+0000 7f7487400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f746c06b500 0x7f746c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.329+0000 7f7487400640 1 -- 172.21.15.114:0/3353247266 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74780af360 con 0x7f749013b0f0 2024-03-06T07:30:59.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.330+0000 7f748f400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f746c06b500 0x7f746c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.330+0000 7f748f400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f746c06b500 0x7f746c06d9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7480001d50 tx=0x7f7480009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.334+0000 7f7487400640 1 -- 172.21.15.114:0/3353247266 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f747807e150 con 0x7f749013b0f0 2024-03-06T07:30:59.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.530+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7490072890 con 0x7f749013b0f0 2024-03-06T07:30:59.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.531+0000 7f7487400640 1 -- 172.21.15.114:0/3353247266 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16713 (secure 0 0 0) 0x7f7478004e20 con 0x7f749013b0f0 2024-03-06T07:30:59.533 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:30:59.533 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:32.939149+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.533+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f746c06b500 msgr2=0x7f746c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f746c06b500 0x7f746c06d9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7480001d50 tx=0x7f7480009040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749013b0f0 msgr2=0x7f7490142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749013b0f0 0x7f7490142e00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f74780308d0 tx=0x7f7478001970 comp rx=0 tx=0).stop 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 shutdown_connections 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f749013a050 0x7f749013a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f746c06b500 0x7f746c06d9c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749013b0f0 0x7f7490142e00 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 --2- 172.21.15.114:0/3353247266 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7474004800 0x7f749015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 >> 172.21.15.114:0/3353247266 conn(0x7f74900b1730 msgr2=0x7f74900b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 shutdown_connections 2024-03-06T07:30:59.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.534+0000 7f7496400640 1 -- 172.21.15.114:0/3353247266 wait complete. 2024-03-06T07:30:59.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:29.190809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:30.382190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:59.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:59.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.757+0000 7f5e98800640 1 Processor -- start 2024-03-06T07:30:59.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.758+0000 7f5e98800640 1 -- start start 2024-03-06T07:30:59.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.759+0000 7f5e98800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5e900c9090 0x7f5e901690f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.759+0000 7f5e98800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900cecd0 con 0x7f5e90161b50 2024-03-06T07:30:59.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.759+0000 7f5e98800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900cfc80 con 0x7f5e900d4a00 2024-03-06T07:30:59.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.759+0000 7f5e98800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e900d13c0 con 0x7f5e900c9090 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.759+0000 7f5e95e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5e90161b50 0x7f5e900d05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33216/0 (socket says 172.21.15.114:33216) 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.759+0000 7f5e95e00640 1 -- 172.21.15.114:0/3573300168 learned_addr learned my addr 172.21.15.114:0/3573300168 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.759+0000 7f5e96800640 1 --2- 172.21.15.114:0/3573300168 >> v2:172.21.15.114:3300/0 conn(0x7f5e900c9090 0x7f5e901690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.760+0000 7f5e94a00640 1 -- 172.21.15.114:0/3573300168 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180215084 0 0) 0x7f5e900cecd0 con 0x7f5e90161b50 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.760+0000 7f5e94a00640 1 -- 172.21.15.114:0/3573300168 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e78003520 con 0x7f5e90161b50 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.760+0000 7f5e96800640 1 -- 172.21.15.114:0/3573300168 >> v1:172.21.15.177:6789/0 conn(0x7f5e900d4a00 legacy=0x7f5e900d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.760+0000 7f5e96800640 1 -- 172.21.15.114:0/3573300168 >> v1:172.21.15.114:6789/0 conn(0x7f5e90161b50 legacy=0x7f5e900d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.760+0000 7f5e96800640 1 -- 172.21.15.114:0/3573300168 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900c7b60 con 0x7f5e900c9090 2024-03-06T07:30:59.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.761+0000 7f5e96800640 1 --2- 172.21.15.114:0/3573300168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e901690f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009d90 tx=0x7f5e8c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ffe7e817a88339 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.761+0000 7f5e94a00640 1 -- 172.21.15.114:0/3573300168 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c039070 con 0x7f5e900c9090 2024-03-06T07:30:59.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e94a00640 1 -- 172.21.15.114:0/3573300168 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c02cd00 con 0x7f5e900c9090 2024-03-06T07:30:59.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e94a00640 1 -- 172.21.15.114:0/3573300168 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c034660 con 0x7f5e900c9090 2024-03-06T07:30:59.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e98800640 1 -- 172.21.15.114:0/3573300168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 msgr2=0x7f5e901690f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e98800640 1 --2- 172.21.15.114:0/3573300168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e901690f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009d90 tx=0x7f5e8c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:59.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e98800640 1 -- 172.21.15.114:0/3573300168 shutdown_connections 2024-03-06T07:30:59.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e98800640 1 --2- 172.21.15.114:0/3573300168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e901690f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e98800640 1 -- 172.21.15.114:0/3573300168 >> 172.21.15.114:0/3573300168 conn(0x7f5e900d3e70 msgr2=0x7f5e900d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e98800640 1 -- 172.21.15.114:0/3573300168 shutdown_connections 2024-03-06T07:30:59.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.762+0000 7f5e98800640 1 -- 172.21.15.114:0/3573300168 wait complete. 2024-03-06T07:30:59.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.764+0000 7f5e98800640 1 Processor -- start 2024-03-06T07:30:59.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.764+0000 7f5e98800640 1 -- start start 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e98800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e9014d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e98800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e900d4a00 0x7f5e900cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e98800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e90161b50 0x7f5e900cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e98800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900cef40 con 0x7f5e900c9090 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e98800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900b56a0 con 0x7f5e900d4a00 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e98800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e90169630 con 0x7f5e90161b50 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e95e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e9014d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e95e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e9014d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51898/0 (socket says 172.21.15.114:51898) 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.765+0000 7f5e95e00640 1 -- 172.21.15.114:0/493912046 learned_addr learned my addr 172.21.15.114:0/493912046 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e95e00640 1 -- 172.21.15.114:0/493912046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e90161b50 msgr2=0x7f5e900cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e95e00640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e90161b50 0x7f5e900cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e95e00640 1 -- 172.21.15.114:0/493912046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e900d4a00 msgr2=0x7f5e900cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e95400640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e900d4a00 0x7f5e900cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e95e00640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e900d4a00 0x7f5e900cb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e95e00640 1 -- 172.21.15.114:0/493912046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e90150610 con 0x7f5e900c9090 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e95e00640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e9014d550 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c530 tx=0x7f5e8400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e8a000640 1 -- 172.21.15.114:0/493912046 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84017070 con 0x7f5e900c9090 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e8a000640 1 -- 172.21.15.114:0/493912046 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e84005ce0 con 0x7f5e900c9090 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e8a000640 1 -- 172.21.15.114:0/493912046 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e840072b0 con 0x7f5e900c9090 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e900cbdc0 con 0x7f5e900c9090 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.766+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9014e060 con 0x7f5e900c9090 2024-03-06T07:30:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.767+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9004f330 con 0x7f5e900c9090 2024-03-06T07:30:59.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.769+0000 7f5e8a000640 1 -- 172.21.15.114:0/493912046 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e8401a070 con 0x7f5e900c9090 2024-03-06T07:30:59.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.770+0000 7f5e8a000640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5e7406b4d0 0x7f5e7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.770+0000 7f5e8a000640 1 -- 172.21.15.114:0/493912046 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e84089640 con 0x7f5e900c9090 2024-03-06T07:30:59.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.771+0000 7f5e95400640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5e7406b4d0 0x7f5e7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.771+0000 7f5e95400640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5e7406b4d0 0x7f5e7406d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5e901f7730 tx=0x7f5e80009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.775+0000 7f5e8a000640 1 -- 172.21.15.114:0/493912046 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e84058580 con 0x7f5e900c9090 2024-03-06T07:30:59.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.970+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e900d16d0 con 0x7f5e900c9090 2024-03-06T07:30:59.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.971+0000 7f5e8a000640 1 -- 172.21.15.114:0/493912046 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e84057e80 con 0x7f5e900c9090 2024-03-06T07:30:59.973 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5e7406b4d0 msgr2=0x7f5e7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5e7406b4d0 0x7f5e7406d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5e901f7730 tx=0x7f5e80009040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 msgr2=0x7f5e9014d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e9014d550 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c530 tx=0x7f5e8400c9f0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 shutdown_connections 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5e90161b50 0x7f5e900cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e900d4a00 0x7f5e900cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5e7406b4d0 0x7f5e7406d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 --2- 172.21.15.114:0/493912046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e900c9090 0x7f5e9014d550 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 >> 172.21.15.114:0/493912046 conn(0x7f5e900d3e70 msgr2=0x7f5e9006c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 shutdown_connections 2024-03-06T07:30:59.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:30:59.974+0000 7f5e98800640 1 -- 172.21.15.114:0/493912046 wait complete. 2024-03-06T07:30:59.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:00.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.192+0000 7f29f4400640 1 Processor -- start 2024-03-06T07:31:00.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.192+0000 7f29f4400640 1 -- start start 2024-03-06T07:31:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f4400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0b3f70 0x7f29ec160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f4400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f29ec070ff0 0x7f29ec0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f4400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec0b46c0 con 0x7f29ec15fc10 2024-03-06T07:31:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f4400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ec071480 con 0x7f29ec070ff0 2024-03-06T07:31:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f4400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec0b49a0 con 0x7f29ec0b3f70 2024-03-06T07:31:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f2400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f29ec070ff0 0x7f29ec0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f1a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f29ec15fc10 0x7f29ec15fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33228/0 (socket says 172.21.15.114:33228) 2024-03-06T07:31:00.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f1a00640 1 -- 172.21.15.114:0/962237924 learned_addr learned my addr 172.21.15.114:0/962237924 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:00.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.194+0000 7f29f1000640 1 --2- 172.21.15.114:0/962237924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0b3f70 0x7f29ec160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.195+0000 7f29f2400640 1 -- 172.21.15.114:0/962237924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0b3f70 msgr2=0x7f29ec160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.195+0000 7f29f2400640 1 --2- 172.21.15.114:0/962237924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0b3f70 0x7f29ec160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.195+0000 7f29f2400640 1 -- 172.21.15.114:0/962237924 >> v1:172.21.15.114:6789/0 conn(0x7f29ec15fc10 legacy=0x7f29ec15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.195+0000 7f29f2400640 1 -- 172.21.15.114:0/962237924 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec072c90 con 0x7f29ec070ff0 2024-03-06T07:31:00.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.196+0000 7f29f2400640 1 --2- 172.21.15.114:0/962237924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec070ff0 0x7f29ec0ce4a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009d90 tx=0x7f29e402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd979f93047fbcde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.196+0000 7f29ebe00640 1 -- 172.21.15.114:0/962237924 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4039070 con 0x7f29ec070ff0 2024-03-06T07:31:00.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.196+0000 7f29ebe00640 1 -- 172.21.15.114:0/962237924 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e402cc60 con 0x7f29ec070ff0 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.196+0000 7f29ebe00640 1 -- 172.21.15.114:0/962237924 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4034660 con 0x7f29ec070ff0 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.196+0000 7f29f4400640 1 -- 172.21.15.114:0/962237924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec070ff0 msgr2=0x7f29ec0ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.196+0000 7f29f4400640 1 --2- 172.21.15.114:0/962237924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec070ff0 0x7f29ec0ce4a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009d90 tx=0x7f29e402beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.197+0000 7f29f4400640 1 -- 172.21.15.114:0/962237924 shutdown_connections 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.197+0000 7f29f4400640 1 --2- 172.21.15.114:0/962237924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec070ff0 0x7f29ec0ce4a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.197+0000 7f29f4400640 1 --2- 172.21.15.114:0/962237924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec0b3f70 0x7f29ec160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.197+0000 7f29f4400640 1 -- 172.21.15.114:0/962237924 >> 172.21.15.114:0/962237924 conn(0x7f29ec0b1940 msgr2=0x7f29ec0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.197+0000 7f29f4400640 1 -- 172.21.15.114:0/962237924 shutdown_connections 2024-03-06T07:31:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.197+0000 7f29f4400640 1 -- 172.21.15.114:0/962237924 wait complete. 2024-03-06T07:31:00.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.198+0000 7f29f4400640 1 Processor -- start 2024-03-06T07:31:00.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.199+0000 7f29f4400640 1 -- start start 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.199+0000 7f29f4400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec070ff0 0x7f29ec148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.199+0000 7f29f4400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec0b3f70 0x7f29ec0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.199+0000 7f29f4400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29ec15fc10 0x7f29ec0c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.199+0000 7f29f4400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec072a30 con 0x7f29ec0b3f70 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.199+0000 7f29f4400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec0728b0 con 0x7f29ec15fc10 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f4400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec15c2e0 con 0x7f29ec070ff0 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec0b3f70 0x7f29ec0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec0b3f70 0x7f29ec0c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51918/0 (socket says 172.21.15.114:51918) 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 -- 172.21.15.114:0/938807633 learned_addr learned my addr 172.21.15.114:0/938807633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1a00640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec070ff0 0x7f29ec148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f2400640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29ec15fc10 0x7f29ec0c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 -- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec070ff0 msgr2=0x7f29ec148680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec070ff0 0x7f29ec148680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 -- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29ec15fc10 msgr2=0x7f29ec0c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29ec15fc10 0x7f29ec0c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.200+0000 7f29f1000640 1 -- 172.21.15.114:0/938807633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec09b0a0 con 0x7f29ec0b3f70 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.201+0000 7f29f1000640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec0b3f70 0x7f29ec0c8fc0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f29d800d7d0 tx=0x7f29d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.201+0000 7f29e9600640 1 -- 172.21.15.114:0/938807633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d800b6c0 con 0x7f29ec0b3f70 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.201+0000 7f29e9600640 1 -- 172.21.15.114:0/938807633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d8019070 con 0x7f29ec0b3f70 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.201+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec09b380 con 0x7f29ec0b3f70 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.201+0000 7f29e9600640 1 -- 172.21.15.114:0/938807633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d800bac0 con 0x7f29ec0b3f70 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.201+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec09b940 con 0x7f29ec0b3f70 2024-03-06T07:31:00.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.202+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ec04f330 con 0x7f29ec0b3f70 2024-03-06T07:31:00.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.204+0000 7f29e9600640 1 -- 172.21.15.114:0/938807633 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29d800bc60 con 0x7f29ec0b3f70 2024-03-06T07:31:00.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.204+0000 7f29e9600640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29cc06b4d0 0x7f29cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.205+0000 7f29e9600640 1 -- 172.21.15.114:0/938807633 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29d808b160 con 0x7f29ec0b3f70 2024-03-06T07:31:00.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.205+0000 7f29f1a00640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29cc06b4d0 0x7f29cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.206+0000 7f29f1a00640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29cc06b4d0 0x7f29cc06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000f60 tx=0x7f29e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.208+0000 7f29e9600640 1 -- 172.21.15.114:0/938807633 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29d805a0c0 con 0x7f29ec0b3f70 2024-03-06T07:31:00.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.408+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f29ec0b2d40 con 0x7f29ec0b3f70 2024-03-06T07:31:00.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.409+0000 7f29e9600640 1 -- 172.21.15.114:0/938807633 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29d80599c0 con 0x7f29ec0b3f70 2024-03-06T07:31:00.411 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29cc06b4d0 msgr2=0x7f29cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29cc06b4d0 0x7f29cc06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000f60 tx=0x7f29e0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec0b3f70 msgr2=0x7f29ec0c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec0b3f70 0x7f29ec0c8fc0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f29d800d7d0 tx=0x7f29d800dc90 comp rx=0 tx=0).stop 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 shutdown_connections 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29ec15fc10 0x7f29ec0c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f29cc06b4d0 0x7f29cc06d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29ec0b3f70 0x7f29ec0c8fc0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 --2- 172.21.15.114:0/938807633 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f29ec070ff0 0x7f29ec148680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 >> 172.21.15.114:0/938807633 conn(0x7f29ec0b1940 msgr2=0x7f29ec0adef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 shutdown_connections 2024-03-06T07:31:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.412+0000 7f29f4400640 1 -- 172.21.15.114:0/938807633 wait complete. 2024-03-06T07:31:00.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:00.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.635+0000 7fd6c0e00640 1 Processor -- start 2024-03-06T07:31:00.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.635+0000 7fd6c0e00640 1 -- start start 2024-03-06T07:31:00.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6c0e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 0x7fd6bc0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6c0e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd6bc06da40 0x7fd6bc0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6c0e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc15c1f0 con 0x7fd6bc0aef20 2024-03-06T07:31:00.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6c0e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6bc0cd200 con 0x7fd6bc06da40 2024-03-06T07:31:00.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6c0e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc06cac0 con 0x7fd6bc15be10 2024-03-06T07:31:00.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6bbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd6bc06da40 0x7fd6bc0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6bbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd6bc06da40 0x7fd6bc0ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51930/0 (socket says 172.21.15.114:51930) 2024-03-06T07:31:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6bbe00640 1 -- 172.21.15.114:0/2369878688 learned_addr learned my addr 172.21.15.114:0/2369878688 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.637+0000 7fd6bb400640 1 --2- 172.21.15.114:0/2369878688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 0x7fd6bc0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.638+0000 7fd6bbe00640 1 -- 172.21.15.114:0/2369878688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 msgr2=0x7fd6bc0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.638+0000 7fd6bbe00640 1 --2- 172.21.15.114:0/2369878688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 0x7fd6bc0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.638+0000 7fd6bbe00640 1 -- 172.21.15.114:0/2369878688 >> v1:172.21.15.177:6789/0 conn(0x7fd6bc0aef20 legacy=0x7fd6bc06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.638+0000 7fd6bbe00640 1 -- 172.21.15.114:0/2369878688 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc06cda0 con 0x7fd6bc06da40 2024-03-06T07:31:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.639+0000 7fd6bbe00640 1 --2- 172.21.15.114:0/2369878688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc0ce2b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009e50 tx=0x7fd6b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508ced076608b3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.639+0000 7fd6ba000640 1 -- 172.21.15.114:0/2369878688 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b402ca70 con 0x7fd6bc06da40 2024-03-06T07:31:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.639+0000 7fd6ba000640 1 -- 172.21.15.114:0/2369878688 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b402cc10 con 0x7fd6bc06da40 2024-03-06T07:31:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.639+0000 7fd6ba000640 1 -- 172.21.15.114:0/2369878688 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40343f0 con 0x7fd6bc06da40 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 -- 172.21.15.114:0/2369878688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 msgr2=0x7fd6bc0ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/2369878688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc0ce2b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009e50 tx=0x7fd6b402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 -- 172.21.15.114:0/2369878688 shutdown_connections 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/2369878688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc0ce2b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/2369878688 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 0x7fd6bc0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 -- 172.21.15.114:0/2369878688 >> 172.21.15.114:0/2369878688 conn(0x7fd6bc06afc0 msgr2=0x7fd6bc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 -- 172.21.15.114:0/2369878688 shutdown_connections 2024-03-06T07:31:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.640+0000 7fd6c0e00640 1 -- 172.21.15.114:0/2369878688 wait complete. 2024-03-06T07:31:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.642+0000 7fd6c0e00640 1 Processor -- start 2024-03-06T07:31:00.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.642+0000 7fd6c0e00640 1 -- start start 2024-03-06T07:31:00.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6c0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6c0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6bc0aef20 0x7fd6bc1403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6c0e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 0x7fd6bc15b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6c0e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc0cda90 con 0x7fd6bc06da40 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6c0e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc06d080 con 0x7fd6bc0aef20 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6c0e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc071ab0 con 0x7fd6bc15be10 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc15cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51944/0 (socket says 172.21.15.114:51944) 2024-03-06T07:31:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 -- 172.21.15.114:0/210822727 learned_addr learned my addr 172.21.15.114:0/210822727 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 -- 172.21.15.114:0/210822727 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 msgr2=0x7fd6bc15b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 0x7fd6bc15b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 -- 172.21.15.114:0/210822727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6bc0aef20 msgr2=0x7fd6bc1403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6bc0aef20 0x7fd6bc1403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.643+0000 7fd6bb400640 1 -- 172.21.15.114:0/210822727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc15d360 con 0x7fd6bc06da40 2024-03-06T07:31:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.644+0000 7fd6bb400640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc15cd40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00d5f0 tx=0x7fd6ac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.644+0000 7fd6b3400640 1 -- 172.21.15.114:0/210822727 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0125f0 con 0x7fd6bc06da40 2024-03-06T07:31:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.644+0000 7fd6b3400640 1 -- 172.21.15.114:0/210822727 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac019070 con 0x7fd6bc06da40 2024-03-06T07:31:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.644+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc15c560 con 0x7fd6bc06da40 2024-03-06T07:31:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.644+0000 7fd6b3400640 1 -- 172.21.15.114:0/210822727 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0129d0 con 0x7fd6bc06da40 2024-03-06T07:31:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.644+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc15f9e0 con 0x7fd6bc06da40 2024-03-06T07:31:00.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.646+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc04f330 con 0x7fd6bc06da40 2024-03-06T07:31:00.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.646+0000 7fd6b3400640 1 -- 172.21.15.114:0/210822727 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6ac012b70 con 0x7fd6bc06da40 2024-03-06T07:31:00.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.647+0000 7fd6b3400640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd69c06b4d0 0x7fd69c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.647+0000 7fd6b3400640 1 -- 172.21.15.114:0/210822727 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6ac08a5f0 con 0x7fd6bc06da40 2024-03-06T07:31:00.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.647+0000 7fd6baa00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd69c06b4d0 0x7fd69c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.648+0000 7fd6baa00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd69c06b4d0 0x7fd69c06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc1591f0 tx=0x7fd6a80079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.652+0000 7fd6b3400640 1 -- 172.21.15.114:0/210822727 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6bc15d360 con 0x7fd6bc06da40 2024-03-06T07:31:00.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.862+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6bc160270 con 0x7fd6bc06da40 2024-03-06T07:31:00.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.863+0000 7fd6b3400640 1 -- 172.21.15.114:0/210822727 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6ac059470 con 0x7fd6bc06da40 2024-03-06T07:31:00.865 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:31:00.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.865+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd69c06b4d0 msgr2=0x7fd69c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.865+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd69c06b4d0 0x7fd69c06d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc1591f0 tx=0x7fd6a80079b0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.865+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 msgr2=0x7fd6bc15cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.865+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc15cd40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00d5f0 tx=0x7fd6ac00dab0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.865+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 shutdown_connections 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.866+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd6bc15be10 0x7fd6bc15b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.866+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6bc0aef20 0x7fd6bc1403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.866+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd69c06b4d0 0x7fd69c06d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.866+0000 7fd6c0e00640 1 --2- 172.21.15.114:0/210822727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6bc06da40 0x7fd6bc15cd40 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.866+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 >> 172.21.15.114:0/210822727 conn(0x7fd6bc06afc0 msgr2=0x7fd6bc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.866+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 shutdown_connections 2024-03-06T07:31:00.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:00.866+0000 7fd6c0e00640 1 -- 172.21.15.114:0/210822727 wait complete. 2024-03-06T07:31:00.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:01.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.090+0000 7f470ea00640 1 Processor -- start 2024-03-06T07:31:01.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.090+0000 7f470ea00640 1 -- start start 2024-03-06T07:31:01.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f470ea00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f470815e2c0 0x7f47080cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f470ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f470806c0d0 0x7f47080d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f470ea00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080b4570 con 0x7f47080cc860 2024-03-06T07:31:01.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f470ea00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47080b4320 con 0x7f470815e2c0 2024-03-06T07:31:01.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f470ea00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47080d05d0 con 0x7f470806c0d0 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f46ffe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f470806c0d0 0x7f47080d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f46ffe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f470806c0d0 0x7f47080d0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:38472/0 (socket says 172.21.15.114:38472) 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f46ffe00640 1 -- 172.21.15.114:0/4170888542 learned_addr learned my addr 172.21.15.114:0/4170888542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.092+0000 7f4707e00640 1 --2- 172.21.15.114:0/4170888542 >> v2:172.21.15.177:3300/0 conn(0x7f470815e2c0 0x7f47080cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.093+0000 7f46ffe00640 1 -- 172.21.15.114:0/4170888542 >> v2:172.21.15.177:3300/0 conn(0x7f470815e2c0 msgr2=0x7f47080cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.093+0000 7f46ffe00640 1 --2- 172.21.15.114:0/4170888542 >> v2:172.21.15.177:3300/0 conn(0x7f470815e2c0 0x7f47080cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.093+0000 7f46ffe00640 1 -- 172.21.15.114:0/4170888542 >> v1:172.21.15.177:6789/0 conn(0x7f47080cc860 legacy=0x7f47080ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.093+0000 7f46ffe00640 1 -- 172.21.15.114:0/4170888542 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47080d0260 con 0x7f470806c0d0 2024-03-06T07:31:01.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.094+0000 7f46ffe00640 1 --2- 172.21.15.114:0/4170888542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f470806c0d0 0x7f47080d0a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009870 tx=0x7f46f002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d84ee1d8e489bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.095+0000 7f4707400640 1 -- 172.21.15.114:0/4170888542 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0034070 con 0x7f470806c0d0 2024-03-06T07:31:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.095+0000 7f4707400640 1 -- 172.21.15.114:0/4170888542 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f0030840 con 0x7f470806c0d0 2024-03-06T07:31:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.095+0000 7f4707400640 1 -- 172.21.15.114:0/4170888542 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0030c10 con 0x7f470806c0d0 2024-03-06T07:31:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.095+0000 7f470ea00640 1 -- 172.21.15.114:0/4170888542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f470806c0d0 msgr2=0x7f47080d0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.095+0000 7f470ea00640 1 --2- 172.21.15.114:0/4170888542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f470806c0d0 0x7f47080d0a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009870 tx=0x7f46f002b980 comp rx=0 tx=0).stop 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.096+0000 7f470ea00640 1 -- 172.21.15.114:0/4170888542 shutdown_connections 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.096+0000 7f470ea00640 1 --2- 172.21.15.114:0/4170888542 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f470806c0d0 0x7f47080d0a60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.096+0000 7f470ea00640 1 --2- 172.21.15.114:0/4170888542 >> v2:172.21.15.177:3300/0 conn(0x7f470815e2c0 0x7f47080cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.096+0000 7f470ea00640 1 -- 172.21.15.114:0/4170888542 >> 172.21.15.114:0/4170888542 conn(0x7f47080b3800 msgr2=0x7f47080b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.096+0000 7f470ea00640 1 -- 172.21.15.114:0/4170888542 shutdown_connections 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.096+0000 7f470ea00640 1 -- 172.21.15.114:0/4170888542 wait complete. 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.097+0000 7f470ea00640 1 Processor -- start 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.097+0000 7f470ea00640 1 -- start start 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.098+0000 7f470ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f470806c0d0 0x7f470815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.098+0000 7f470ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47080cc860 0x7f47080c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.098+0000 7f470ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f470815e2c0 0x7f47080caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.098+0000 7f470ea00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47080cec60 con 0x7f470815e2c0 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.098+0000 7f470ea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47080b0e20 con 0x7f470806c0d0 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.098+0000 7f470ea00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47080cede0 con 0x7f47080cc860 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f470815e2c0 0x7f47080caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f470815e2c0 0x7f47080caa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51952/0 (socket says 172.21.15.114:51952) 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 -- 172.21.15.114:0/753050390 learned_addr learned my addr 172.21.15.114:0/753050390 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f4707e00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f470806c0d0 0x7f470815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f46ffe00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47080cc860 0x7f47080c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 -- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47080cc860 msgr2=0x7f47080c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47080cc860 0x7f47080c6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 -- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f470806c0d0 msgr2=0x7f470815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f470806c0d0 0x7f470815ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f470ca00640 1 -- 172.21.15.114:0/753050390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470815b380 con 0x7f470815e2c0 2024-03-06T07:31:01.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.099+0000 7f4707e00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f470806c0d0 0x7f470815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:01.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.100+0000 7f470ca00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f470815e2c0 0x7f47080caa70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f470000c6b0 tx=0x7f470000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.100+0000 7f4704c00640 1 -- 172.21.15.114:0/753050390 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700003b70 con 0x7f470815e2c0 2024-03-06T07:31:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.100+0000 7f4704c00640 1 -- 172.21.15.114:0/753050390 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4700003d30 con 0x7f470815e2c0 2024-03-06T07:31:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.100+0000 7f4704c00640 1 -- 172.21.15.114:0/753050390 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700005310 con 0x7f470815e2c0 2024-03-06T07:31:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.100+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47080c6e80 con 0x7f470815e2c0 2024-03-06T07:31:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.100+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47080c7440 con 0x7f470815e2c0 2024-03-06T07:31:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.102+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470804f330 con 0x7f470815e2c0 2024-03-06T07:31:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.102+0000 7f4704c00640 1 -- 172.21.15.114:0/753050390 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47000054b0 con 0x7f470815e2c0 2024-03-06T07:31:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.103+0000 7f4704c00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f46e406b4d0 0x7f46e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.103+0000 7f4704c00640 1 -- 172.21.15.114:0/753050390 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f470008a070 con 0x7f470815e2c0 2024-03-06T07:31:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.103+0000 7f4707e00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f46e406b4d0 0x7f46e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.104+0000 7f4707e00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f46e406b4d0 0x7f46e406d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009af0 tx=0x7f46f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.111+0000 7f4704c00640 1 -- 172.21.15.114:0/753050390 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4700058ee0 con 0x7f470815e2c0 2024-03-06T07:31:01.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.355+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f47080b0c00 con 0x7f470815e2c0 2024-03-06T07:31:01.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.356+0000 7f4704c00640 1 -- 172.21.15.114:0/753050390 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) v1 ==== 80+0+0 (secure 0 0 0) 0x7f47000587e0 con 0x7f470815e2c0 2024-03-06T07:31:01.357 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:31:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f46e406b4d0 msgr2=0x7f46e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f46e406b4d0 0x7f46e406d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009af0 tx=0x7f46f8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f470815e2c0 msgr2=0x7f47080caa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f470815e2c0 0x7f47080caa70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f470000c6b0 tx=0x7f470000cb70 comp rx=0 tx=0).stop 2024-03-06T07:31:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 shutdown_connections 2024-03-06T07:31:01.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f46e406b4d0 0x7f46e406d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f470815e2c0 0x7f47080caa70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47080cc860 0x7f47080c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 --2- 172.21.15.114:0/753050390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f470806c0d0 0x7f470815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.358+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 >> 172.21.15.114:0/753050390 conn(0x7f47080b3800 msgr2=0x7f47080b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.359+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 shutdown_connections 2024-03-06T07:31:01.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.359+0000 7f470ea00640 1 -- 172.21.15.114:0/753050390 wait complete. 2024-03-06T07:31:01.378 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:31:01.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:01.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:01.606 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:31:01.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:01.606 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:01.646 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:31:01.646 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:01.690 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:01.690 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:01.690 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:01.729 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:01.730 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:31:01.730 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:01.784 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:01.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:01.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.979+0000 7f1483800640 1 Processor -- start 2024-03-06T07:31:01.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.980+0000 7f1483800640 1 -- start start 2024-03-06T07:31:01.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.981+0000 7f1483800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c15acb0 0x7f147c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.981+0000 7f1483800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f147c0b4680 0x7f147c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.981+0000 7f1483800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147c15b090 con 0x7f147c072340 2024-03-06T07:31:01.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.981+0000 7f1483800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147c0b2a30 con 0x7f147c0b4680 2024-03-06T07:31:01.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.981+0000 7f1483800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0b4360 con 0x7f147c15acb0 2024-03-06T07:31:01.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.982+0000 7f1480e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c15acb0 0x7f147c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.982+0000 7f1480e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c15acb0 0x7f147c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:38500/0 (socket says 172.21.15.114:38500) 2024-03-06T07:31:01.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.982+0000 7f1480e00640 1 -- 172.21.15.114:0/2985954658 learned_addr learned my addr 172.21.15.114:0/2985954658 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:01.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.982+0000 7f147be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f147c0b4680 0x7f147c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.983+0000 7f1480e00640 1 -- 172.21.15.114:0/2985954658 >> v2:172.21.15.177:3300/0 conn(0x7f147c0b4680 msgr2=0x7f147c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.983+0000 7f1480e00640 1 --2- 172.21.15.114:0/2985954658 >> v2:172.21.15.177:3300/0 conn(0x7f147c0b4680 0x7f147c06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.983+0000 7f1480e00640 1 -- 172.21.15.114:0/2985954658 >> v1:172.21.15.177:6789/0 conn(0x7f147c072340 legacy=0x7f147c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.983+0000 7f1480e00640 1 -- 172.21.15.114:0/2985954658 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0b2cf0 con 0x7f147c15acb0 2024-03-06T07:31:01.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.984+0000 7f1480e00640 1 --2- 172.21.15.114:0/2985954658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c15acb0 0x7f147c0b37d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1470009870 tx=0x7f147002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a47dd2a7ef4630c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.984+0000 7f147b400640 1 -- 172.21.15.114:0/2985954658 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470034070 con 0x7f147c15acb0 2024-03-06T07:31:01.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.984+0000 7f147b400640 1 -- 172.21.15.114:0/2985954658 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1470030840 con 0x7f147c15acb0 2024-03-06T07:31:01.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f147b400640 1 -- 172.21.15.114:0/2985954658 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470030c10 con 0x7f147c15acb0 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 -- 172.21.15.114:0/2985954658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c15acb0 msgr2=0x7f147c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 --2- 172.21.15.114:0/2985954658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c15acb0 0x7f147c0b37d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1470009870 tx=0x7f147002b980 comp rx=0 tx=0).stop 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 -- 172.21.15.114:0/2985954658 shutdown_connections 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 --2- 172.21.15.114:0/2985954658 >> v2:172.21.15.177:3300/0 conn(0x7f147c0b4680 0x7f147c06e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 --2- 172.21.15.114:0/2985954658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c15acb0 0x7f147c0b37d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 -- 172.21.15.114:0/2985954658 >> 172.21.15.114:0/2985954658 conn(0x7f147c0b1730 msgr2=0x7f147c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 -- 172.21.15.114:0/2985954658 shutdown_connections 2024-03-06T07:31:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.985+0000 7f1483800640 1 -- 172.21.15.114:0/2985954658 wait complete. 2024-03-06T07:31:01.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.987+0000 7f1483800640 1 Processor -- start 2024-03-06T07:31:01.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.987+0000 7f1483800640 1 -- start start 2024-03-06T07:31:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1483800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c072340 0x7f147c167370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1483800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f147c0b4680 0x7f147c1453b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1483800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f147c15acb0 0x7f147c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1483800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0b3d10 con 0x7f147c15acb0 2024-03-06T07:31:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1483800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0b32b0 con 0x7f147c0b4680 2024-03-06T07:31:01.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1483800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147c0b4010 con 0x7f147c072340 2024-03-06T07:31:01.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1481800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f147c15acb0 0x7f147c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f147be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f147c0b4680 0x7f147c1453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1480e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c072340 0x7f147c167370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1481800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f147c15acb0 0x7f147c0ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51966/0 (socket says 172.21.15.114:51966) 2024-03-06T07:31:01.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.988+0000 7f1481800640 1 -- 172.21.15.114:0/2471827301 learned_addr learned my addr 172.21.15.114:0/2471827301 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:01.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1481800640 1 -- 172.21.15.114:0/2471827301 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c072340 msgr2=0x7f147c167370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1481800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c072340 0x7f147c167370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1481800640 1 -- 172.21.15.114:0/2471827301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f147c0b4680 msgr2=0x7f147c1453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1481800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f147c0b4680 0x7f147c1453b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1481800640 1 -- 172.21.15.114:0/2471827301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0c9170 con 0x7f147c15acb0 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1481800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f147c15acb0 0x7f147c0ca1f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f147400c6e0 tx=0x7f147400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1478c00640 1 -- 172.21.15.114:0/2471827301 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474003b70 con 0x7f147c15acb0 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1478c00640 1 -- 172.21.15.114:0/2471827301 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474003d30 con 0x7f147c15acb0 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.989+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c0c9450 con 0x7f147c15acb0 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.990+0000 7f1478c00640 1 -- 172.21.15.114:0/2471827301 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474005310 con 0x7f147c15acb0 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.990+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c0c9cf0 con 0x7f147c15acb0 2024-03-06T07:31:01.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.991+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c04f330 con 0x7f147c15acb0 2024-03-06T07:31:01.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.992+0000 7f1478c00640 1 -- 172.21.15.114:0/2471827301 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14740054b0 con 0x7f147c15acb0 2024-03-06T07:31:01.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.992+0000 7f1478c00640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f145c06b4d0 0x7f145c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.993+0000 7f1478c00640 1 -- 172.21.15.114:0/2471827301 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1474089f90 con 0x7f147c15acb0 2024-03-06T07:31:01.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.993+0000 7f1480e00640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f145c06b4d0 0x7f145c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:01.993+0000 7f1480e00640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f145c06b4d0 0x7f145c06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1470036000 tx=0x7f147003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.000+0000 7f1478c00640 1 -- 172.21.15.114:0/2471827301 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1474058740 con 0x7f147c15acb0 2024-03-06T07:31:02.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.232+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f147c0c9620 con 0x7f147c15acb0 2024-03-06T07:31:02.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.233+0000 7f1478c00640 1 -- 172.21.15.114:0/2471827301 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f147405c7e0 con 0x7f147c15acb0 2024-03-06T07:31:02.235 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:02.235 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:31:02.236 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f145c06b4d0 msgr2=0x7f145c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f145c06b4d0 0x7f145c06d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1470036000 tx=0x7f147003b040 comp rx=0 tx=0).stop 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f147c15acb0 msgr2=0x7f147c0ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f147c15acb0 0x7f147c0ca1f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f147400c6e0 tx=0x7f147400cba0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 shutdown_connections 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f145c06b4d0 0x7f145c06d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f147c15acb0 0x7f147c0ca1f0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f147c0b4680 0x7f147c1453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 --2- 172.21.15.114:0/2471827301 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f147c072340 0x7f147c167370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 >> 172.21.15.114:0/2471827301 conn(0x7f147c0b1730 msgr2=0x7f147c0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 shutdown_connections 2024-03-06T07:31:02.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.236+0000 7f1483800640 1 -- 172.21.15.114:0/2471827301 wait complete. 2024-03-06T07:31:02.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:02.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.450+0000 7fd052800640 1 Processor -- start 2024-03-06T07:31:02.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.450+0000 7fd052800640 1 -- start start 2024-03-06T07:31:02.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.451+0000 7fd052800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 0x7fd04c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.451+0000 7fd052800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c15c240 con 0x7fd04c0cc9d0 2024-03-06T07:31:02.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.451+0000 7fd052800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c15c500 con 0x7fd04c0aebf0 2024-03-06T07:31:02.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.451+0000 7fd052800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c15ce30 con 0x7fd04c1561f0 2024-03-06T07:31:02.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd051e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd04c0cc9d0 0x7fd04c15f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33234/0 (socket says 172.21.15.114:33234) 2024-03-06T07:31:02.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd051e00640 1 -- 172.21.15.114:0/4077027076 learned_addr learned my addr 172.21.15.114:0/4077027076 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208754446 0 0) 0x7fd04c15c240 con 0x7fd04c0cc9d0 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd051400640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 0x7fd04c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd034003520 con 0x7fd04c0cc9d0 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3651449033 0 0) 0x7fd034003520 con 0x7fd04c0cc9d0 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 msgr2=0x7fd04c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 0x7fd04c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 >> v1:172.21.15.177:6789/0 conn(0x7fd04c0aebf0 legacy=0x7fd04c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c15c7e0 con 0x7fd04c0cc9d0 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.452+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fd0440019f0 con 0x7fd04c0cc9d0 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 >> v1:172.21.15.114:6789/0 conn(0x7fd04c0cc9d0 legacy=0x7fd04c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd04be00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0340039e0 0x7fd034003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd04be00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 0x7fd034004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd04be00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd034004d40 0x7fd0340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c15ce30 con 0x7fd034004d40 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03400ba70 con 0x7fd0340039e0 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03400bc70 con 0x7fd04c1561f0 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd051e00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd034004d40 0x7fd0340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd051400640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0340039e0 0x7fd034003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd050a00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 0x7fd034004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd051e00640 1 -- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 msgr2=0x7fd034004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd051e00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 0x7fd034004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd051e00640 1 -- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0340039e0 msgr2=0x7fd034003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd051e00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0340039e0 0x7fd034003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd051e00640 1 -- 172.21.15.114:0/4077027076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd034003520 con 0x7fd034004d40 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.453+0000 7fd050a00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c1561f0 0x7fd034004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.454+0000 7fd051e00640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd034004d40 0x7fd0340051c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd04400a830 tx=0x7fd034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5742d21e6b13fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04400a480 con 0x7fd034004d40 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044034630 con 0x7fd034004d40 2024-03-06T07:31:02.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd04be00640 1 -- 172.21.15.114:0/4077027076 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044034a40 con 0x7fd034004d40 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd052800640 1 -- 172.21.15.114:0/4077027076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd034004d40 msgr2=0x7fd0340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd052800640 1 --2- 172.21.15.114:0/4077027076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd034004d40 0x7fd0340051c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd04400a830 tx=0x7fd034005830 comp rx=0 tx=0).stop 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd051e00640 1 -- 172.21.15.114:0/4077027076 reap_dead start 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd052800640 1 -- 172.21.15.114:0/4077027076 shutdown_connections 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd052800640 1 -- 172.21.15.114:0/4077027076 >> 172.21.15.114:0/4077027076 conn(0x7fd04c06afc0 msgr2=0x7fd04c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd052800640 1 -- 172.21.15.114:0/4077027076 shutdown_connections 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.455+0000 7fd052800640 1 -- 172.21.15.114:0/4077027076 wait complete. 2024-03-06T07:31:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.458+0000 7fd052800640 1 Processor -- start 2024-03-06T07:31:02.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.458+0000 7fd052800640 1 -- start start 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd052800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd034004d40 0x7fd04c1e44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd052800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c153800 0x7fd04c1e5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd052800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd04c15e460 0x7fd04c145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd052800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1565d0 con 0x7fd04c15e460 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd052800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c06e360 con 0x7fd034004d40 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd052800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c156750 con 0x7fd04c153800 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd04c15e460 0x7fd04c145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd04c15e460 0x7fd04c145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51980/0 (socket says 172.21.15.114:51980) 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 -- 172.21.15.114:0/2200686252 learned_addr learned my addr 172.21.15.114:0/2200686252 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051400640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd034004d40 0x7fd04c1e44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 -- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c153800 msgr2=0x7fd04c1e5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd050a00640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c153800 0x7fd04c1e5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c153800 0x7fd04c1e5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 -- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd034004d40 msgr2=0x7fd04c1e44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd034004d40 0x7fd04c1e44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.460+0000 7fd051e00640 1 -- 172.21.15.114:0/2200686252 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c1e4a10 con 0x7fd04c15e460 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.461+0000 7fd051e00640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd04c15e460 0x7fd04c145e80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd044001cf0 tx=0x7fd04c15cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.461+0000 7fd049600640 1 -- 172.21.15.114:0/2200686252 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04400a480 con 0x7fd04c15e460 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.461+0000 7fd049600640 1 -- 172.21.15.114:0/2200686252 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044034430 con 0x7fd04c15e460 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.461+0000 7fd049600640 1 -- 172.21.15.114:0/2200686252 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044004d70 con 0x7fd04c15e460 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.461+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c1e4cb0 con 0x7fd04c15e460 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.461+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c1e5270 con 0x7fd04c15e460 2024-03-06T07:31:02.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.462+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c04f330 con 0x7fd04c15e460 2024-03-06T07:31:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.464+0000 7fd049600640 1 -- 172.21.15.114:0/2200686252 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd044047020 con 0x7fd04c15e460 2024-03-06T07:31:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.465+0000 7fd049600640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd02c06b4d0 0x7fd02c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.465+0000 7fd049600640 1 -- 172.21.15.114:0/2200686252 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0440abf10 con 0x7fd04c15e460 2024-03-06T07:31:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.465+0000 7fd051400640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd02c06b4d0 0x7fd02c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.466+0000 7fd051400640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd02c06b4d0 0x7fd02c06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd04c06ba80 tx=0x7fd040008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.470+0000 7fd049600640 1 -- 172.21.15.114:0/2200686252 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd04407ae90 con 0x7fd04c15e460 2024-03-06T07:31:02.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.839+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd04c0b2b10 con 0x7fd04c15e460 2024-03-06T07:31:02.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.841+0000 7fd049600640 1 -- 172.21.15.114:0/2200686252 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16713 (secure 0 0 0) 0x7fd04407a790 con 0x7fd04c15e460 2024-03-06T07:31:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:02.843 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:32.939149+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.843+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd02c06b4d0 msgr2=0x7fd02c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.843+0000 7fd052800640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd02c06b4d0 0x7fd02c06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd04c06ba80 tx=0x7fd040008040 comp rx=0 tx=0).stop 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.843+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd04c15e460 msgr2=0x7fd04c145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.843+0000 7fd052800640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd04c15e460 0x7fd04c145e80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd044001cf0 tx=0x7fd04c15cda0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 shutdown_connections 2024-03-06T07:31:02.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd02c06b4d0 0x7fd02c06d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd04c15e460 0x7fd04c145e80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd04c153800 0x7fd04c1e5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 --2- 172.21.15.114:0/2200686252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd034004d40 0x7fd04c1e44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 >> 172.21.15.114:0/2200686252 conn(0x7fd04c06afc0 msgr2=0x7fd04c0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 shutdown_connections 2024-03-06T07:31:02.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:02.844+0000 7fd052800640 1 -- 172.21.15.114:0/2200686252 wait complete. 2024-03-06T07:31:02.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:29.190809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:30.382190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:02.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:03.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.051+0000 7fb142e00640 1 Processor -- start 2024-03-06T07:31:03.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.051+0000 7fb142e00640 1 -- start start 2024-03-06T07:31:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.052+0000 7fb142e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb13c0b3f70 0x7fb13c160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.052+0000 7fb142e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c071440 con 0x7fb13c070ff0 2024-03-06T07:31:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.052+0000 7fb142e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c0b46c0 con 0x7fb13c15fc10 2024-03-06T07:31:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.052+0000 7fb142e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c0b49a0 con 0x7fb13c0b3f70 2024-03-06T07:31:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.052+0000 7fb140e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb13c070ff0 0x7fb13c0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33250/0 (socket says 172.21.15.114:33250) 2024-03-06T07:31:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.052+0000 7fb140e00640 1 -- 172.21.15.114:0/453184131 learned_addr learned my addr 172.21.15.114:0/453184131 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb133e00640 1 --2- 172.21.15.114:0/453184131 >> v2:172.21.15.177:3300/0 conn(0x7fb13c0b3f70 0x7fb13c160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544696545 0 0) 0x7fb13c071440 con 0x7fb13c070ff0 2024-03-06T07:31:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb120003520 con 0x7fb13c070ff0 2024-03-06T07:31:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4216885024 0 0) 0x7fb120003520 con 0x7fb13c070ff0 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 >> v2:172.21.15.177:3300/0 conn(0x7fb13c0b3f70 msgr2=0x7fb13c160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 --2- 172.21.15.114:0/453184131 >> v2:172.21.15.177:3300/0 conn(0x7fb13c0b3f70 0x7fb13c160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 >> v1:172.21.15.177:6789/0 conn(0x7fb13c15fc10 legacy=0x7fb13c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c072c90 con 0x7fb13c070ff0 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb1340019f0 con 0x7fb13c070ff0 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 >> v1:172.21.15.114:6789/0 conn(0x7fb13c070ff0 legacy=0x7fb13c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1200039b0 0x7fb120003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb120004800 0x7fb120004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200051c0 0x7fb12000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0b49a0 con 0x7fb1200039b0 2024-03-06T07:31:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12000c060 con 0x7fb120004800 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.053+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12000c260 con 0x7fb1200051c0 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.054+0000 7fb13be00640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1200039b0 0x7fb120003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.054+0000 7fb13be00640 1 -- 172.21.15.114:0/453184131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200051c0 msgr2=0x7fb12000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.054+0000 7fb13be00640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200051c0 0x7fb12000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.054+0000 7fb13be00640 1 -- 172.21.15.114:0/453184131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb120004800 msgr2=0x7fb120004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.054+0000 7fb13be00640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb120004800 0x7fb120004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.054+0000 7fb13be00640 1 -- 172.21.15.114:0/453184131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb120003520 con 0x7fb1200039b0 2024-03-06T07:31:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.054+0000 7fb140e00640 1 -- 172.21.15.114:0/453184131 reap_dead start 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.055+0000 7fb13be00640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1200039b0 0x7fb120003dd0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb12c001750 tx=0x7fb120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5d465ef9fe40f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.055+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c001370 con 0x7fb1200039b0 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.055+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c0052d0 con 0x7fb1200039b0 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.055+0000 7fb13b400640 1 -- 172.21.15.114:0/453184131 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c003830 con 0x7fb1200039b0 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.056+0000 7fb142e00640 1 -- 172.21.15.114:0/453184131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1200039b0 msgr2=0x7fb120003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.056+0000 7fb142e00640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1200039b0 0x7fb120003dd0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb12c001750 tx=0x7fb120005830 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.056+0000 7fb142e00640 1 -- 172.21.15.114:0/453184131 shutdown_connections 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.056+0000 7fb142e00640 1 --2- 172.21.15.114:0/453184131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1200039b0 0x7fb120003dd0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.056+0000 7fb142e00640 1 -- 172.21.15.114:0/453184131 >> 172.21.15.114:0/453184131 conn(0x7fb13c0b1940 msgr2=0x7fb13c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.056+0000 7fb142e00640 1 -- 172.21.15.114:0/453184131 shutdown_connections 2024-03-06T07:31:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.056+0000 7fb142e00640 1 -- 172.21.15.114:0/453184131 wait complete. 2024-03-06T07:31:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.058+0000 7fb142e00640 1 Processor -- start 2024-03-06T07:31:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.058+0000 7fb142e00640 1 -- start start 2024-03-06T07:31:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb142e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200039b0 0x7fb13c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb142e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb13c1582e0 0x7fb13c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb142e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13c158740 0x7fb13c0ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb142e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c072a30 con 0x7fb13c158740 2024-03-06T07:31:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb142e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c0728b0 con 0x7fb13c1582e0 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb142e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15c2e0 con 0x7fb1200039b0 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb140e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13c158740 0x7fb13c0ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb140e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13c158740 0x7fb13c0ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52000/0 (socket says 172.21.15.114:52000) 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb140e00640 1 -- 172.21.15.114:0/1063137462 learned_addr learned my addr 172.21.15.114:0/1063137462 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb133e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb13c1582e0 0x7fb13c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.059+0000 7fb13be00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200039b0 0x7fb13c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb140e00640 1 -- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200039b0 msgr2=0x7fb13c06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb140e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200039b0 0x7fb13c06cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb140e00640 1 -- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb13c1582e0 msgr2=0x7fb13c0c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb140e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb13c1582e0 0x7fb13c0c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb140e00640 1 -- 172.21.15.114:0/1063137462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c14a100 con 0x7fb13c158740 2024-03-06T07:31:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb140e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13c158740 0x7fb13c0ca140 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb13c070bc0 tx=0x7fb13400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb138c00640 1 -- 172.21.15.114:0/1063137462 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134004d60 con 0x7fb13c158740 2024-03-06T07:31:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb138c00640 1 -- 172.21.15.114:0/1063137462 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13400a610 con 0x7fb13c158740 2024-03-06T07:31:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c14a3e0 con 0x7fb13c158740 2024-03-06T07:31:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb138c00640 1 -- 172.21.15.114:0/1063137462 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13400aa00 con 0x7fb13c158740 2024-03-06T07:31:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.060+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c14a9a0 con 0x7fb13c158740 2024-03-06T07:31:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.062+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c04f330 con 0x7fb13c158740 2024-03-06T07:31:03.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.063+0000 7fb138c00640 1 -- 172.21.15.114:0/1063137462 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb134011070 con 0x7fb13c158740 2024-03-06T07:31:03.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.064+0000 7fb138c00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.064+0000 7fb138c00640 1 -- 172.21.15.114:0/1063137462 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb134089ca0 con 0x7fb13c158740 2024-03-06T07:31:03.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.064+0000 7fb13be00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.065+0000 7fb13be00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb12c033000 tx=0x7fb12c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.071+0000 7fb138c00640 1 -- 172.21.15.114:0/1063137462 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb13401a060 con 0x7fb13c158740 2024-03-06T07:31:03.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.264+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb13c06ef30 con 0x7fb13c158740 2024-03-06T07:31:03.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.265+0000 7fb138c00640 1 -- 172.21.15.114:0/1063137462 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1340589f0 con 0x7fb13c158740 2024-03-06T07:31:03.266 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 msgr2=0x7fb11806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb12c033000 tx=0x7fb12c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13c158740 msgr2=0x7fb13c0ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13c158740 0x7fb13c0ca140 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb13c070bc0 tx=0x7fb13400cbe0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 shutdown_connections 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb13c158740 0x7fb13c0ca140 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb13c1582e0 0x7fb13c0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 --2- 172.21.15.114:0/1063137462 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1200039b0 0x7fb13c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.267+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 >> 172.21.15.114:0/1063137462 conn(0x7fb13c0b1940 msgr2=0x7fb13c0ae330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.268+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 shutdown_connections 2024-03-06T07:31:03.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.268+0000 7fb142e00640 1 -- 172.21.15.114:0/1063137462 wait complete. 2024-03-06T07:31:03.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.486+0000 7f9d87e00640 1 Processor -- start 2024-03-06T07:31:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.486+0000 7f9d87e00640 1 -- start start 2024-03-06T07:31:03.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.488+0000 7f9d87e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9d800b3f70 0x7f9d80160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.488+0000 7f9d87e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d80071440 con 0x7f9d8015fc10 2024-03-06T07:31:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.488+0000 7f9d87e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d800b46c0 con 0x7f9d80070ff0 2024-03-06T07:31:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.488+0000 7f9d87e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d800b49a0 con 0x7f9d800b3f70 2024-03-06T07:31:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.488+0000 7f9d85400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9d8015fc10 0x7f9d8015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33260/0 (socket says 172.21.15.114:33260) 2024-03-06T07:31:03.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.488+0000 7f9d85400640 1 -- 172.21.15.114:0/3213203200 learned_addr learned my addr 172.21.15.114:0/3213203200 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d84a00640 1 --2- 172.21.15.114:0/3213203200 >> v2:172.21.15.177:3300/0 conn(0x7f9d800b3f70 0x7f9d80160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628018948 0 0) 0x7f9d80071440 con 0x7f9d8015fc10 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d68003520 con 0x7f9d8015fc10 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1794632780 0 0) 0x7f9d68003520 con 0x7f9d8015fc10 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 >> v2:172.21.15.177:3300/0 conn(0x7f9d800b3f70 msgr2=0x7f9d80160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 --2- 172.21.15.114:0/3213203200 >> v2:172.21.15.177:3300/0 conn(0x7f9d800b3f70 0x7f9d80160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 >> v1:172.21.15.177:6789/0 conn(0x7f9d80070ff0 legacy=0x7f9d800ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d80072c90 con 0x7f9d8015fc10 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f9d74001670 con 0x7f9d8015fc10 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 >> v1:172.21.15.114:6789/0 conn(0x7f9d8015fc10 legacy=0x7f9d8015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d68003a00 0x7f9d68003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d68004800 0x7f9d68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800b49a0 con 0x7f9d680051c0 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6800c0c0 con 0x7f9d68003a00 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6800c2c0 con 0x7f9d68004800 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.489+0000 7f9d85e00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.490+0000 7f9d85e00640 1 -- 172.21.15.114:0/3213203200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d68004800 msgr2=0x7f9d68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.490+0000 7f9d85e00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d68004800 0x7f9d68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.490+0000 7f9d85e00640 1 -- 172.21.15.114:0/3213203200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d68003a00 msgr2=0x7f9d68003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.490+0000 7f9d85e00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d68003a00 0x7f9d68003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.490+0000 7f9d85e00640 1 -- 172.21.15.114:0/3213203200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68003520 con 0x7f9d680051c0 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.491+0000 7f9d85e00640 1 -- 172.21.15.114:0/3213203200 reap_dead start 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.491+0000 7f9d85e00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d6800b9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9d78001e50 tx=0x7f9d68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760d808a08af873a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.491+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d78001a70 con 0x7f9d680051c0 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.491+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d78003bb0 con 0x7f9d680051c0 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.491+0000 7f9d7fe00640 1 -- 172.21.15.114:0/3213203200 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d780050d0 con 0x7f9d680051c0 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.491+0000 7f9d87e00640 1 -- 172.21.15.114:0/3213203200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 msgr2=0x7f9d6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.491+0000 7f9d87e00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d6800b9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9d78001e50 tx=0x7f9d68005830 comp rx=0 tx=0).stop 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.492+0000 7f9d87e00640 1 -- 172.21.15.114:0/3213203200 shutdown_connections 2024-03-06T07:31:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.492+0000 7f9d87e00640 1 --2- 172.21.15.114:0/3213203200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d6800b9c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.492+0000 7f9d87e00640 1 -- 172.21.15.114:0/3213203200 >> 172.21.15.114:0/3213203200 conn(0x7f9d800b1940 msgr2=0x7f9d800b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.492+0000 7f9d87e00640 1 -- 172.21.15.114:0/3213203200 shutdown_connections 2024-03-06T07:31:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.492+0000 7f9d87e00640 1 -- 172.21.15.114:0/3213203200 wait complete. 2024-03-06T07:31:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.493+0000 7f9d87e00640 1 Processor -- start 2024-03-06T07:31:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.493+0000 7f9d87e00640 1 -- start start 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.494+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d8014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.494+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d8006ddb0 0x7f9d8006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.494+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d8006d7a0 0x7f9d80150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.494+0000 7f9d87e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80072a30 con 0x7f9d680051c0 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.494+0000 7f9d87e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800728b0 con 0x7f9d8006ddb0 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.494+0000 7f9d87e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8015c2e0 con 0x7f9d8006d7a0 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d8014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d8014c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52016/0 (socket says 172.21.15.114:52016) 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 -- 172.21.15.114:0/4037337890 learned_addr learned my addr 172.21.15.114:0/4037337890 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 -- 172.21.15.114:0/4037337890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d8006d7a0 msgr2=0x7f9d80150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d8006d7a0 0x7f9d80150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 -- 172.21.15.114:0/4037337890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d8006ddb0 msgr2=0x7f9d8006d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d8006ddb0 0x7f9d8006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 -- 172.21.15.114:0/4037337890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8006e240 con 0x7f9d680051c0 2024-03-06T07:31:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.495+0000 7f9d85400640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d8014c370 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c730 tx=0x7f9d7400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.496+0000 7f9d7d600640 1 -- 172.21.15.114:0/4037337890 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74018070 con 0x7f9d680051c0 2024-03-06T07:31:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.496+0000 7f9d7d600640 1 -- 172.21.15.114:0/4037337890 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d740072d0 con 0x7f9d680051c0 2024-03-06T07:31:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.496+0000 7f9d7d600640 1 -- 172.21.15.114:0/4037337890 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740047e0 con 0x7f9d680051c0 2024-03-06T07:31:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.496+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8006e520 con 0x7f9d680051c0 2024-03-06T07:31:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.496+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d801506b0 con 0x7f9d680051c0 2024-03-06T07:31:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.497+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8004f330 con 0x7f9d680051c0 2024-03-06T07:31:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.498+0000 7f9d7d600640 1 -- 172.21.15.114:0/4037337890 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d7401b070 con 0x7f9d680051c0 2024-03-06T07:31:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.499+0000 7f9d7d600640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d6006b4d0 0x7f9d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.499+0000 7f9d7d600640 1 -- 172.21.15.114:0/4037337890 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d74089640 con 0x7f9d680051c0 2024-03-06T07:31:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.499+0000 7f9d84a00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d6006b4d0 0x7f9d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.500+0000 7f9d84a00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d6006b4d0 0x7f9d6006d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006a90 tx=0x7f9d6c006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.503+0000 7f9d7d600640 1 -- 172.21.15.114:0/4037337890 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d740584b0 con 0x7f9d680051c0 2024-03-06T07:31:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.691+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d800aeb20 con 0x7f9d680051c0 2024-03-06T07:31:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.691+0000 7f9d7d600640 1 -- 172.21.15.114:0/4037337890 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d7405ce20 con 0x7f9d680051c0 2024-03-06T07:31:03.693 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:31:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d6006b4d0 msgr2=0x7f9d6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d6006b4d0 0x7f9d6006d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006a90 tx=0x7f9d6c006a20 comp rx=0 tx=0).stop 2024-03-06T07:31:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 msgr2=0x7f9d8014c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d8014c370 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c730 tx=0x7f9d7400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 shutdown_connections 2024-03-06T07:31:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9d8006d7a0 0x7f9d80150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d8006ddb0 0x7f9d8006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9d6006b4d0 0x7f9d6006d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 --2- 172.21.15.114:0/4037337890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d680051c0 0x7f9d8014c370 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 >> 172.21.15.114:0/4037337890 conn(0x7f9d800b1940 msgr2=0x7f9d800cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 shutdown_connections 2024-03-06T07:31:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.694+0000 7f9d87e00640 1 -- 172.21.15.114:0/4037337890 wait complete. 2024-03-06T07:31:03.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.903+0000 7f6996c00640 1 Processor -- start 2024-03-06T07:31:03.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.903+0000 7f6996c00640 1 -- start start 2024-03-06T07:31:03.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.905+0000 7f6996c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f699015bb90 0x7f699006dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.905+0000 7f6996c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f699015efa0 0x7f69900cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.905+0000 7f6996c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6990071ea0 con 0x7f699006afc0 2024-03-06T07:31:03.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.905+0000 7f6996c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699015bf70 con 0x7f699015efa0 2024-03-06T07:31:03.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.905+0000 7f6996c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699006e380 con 0x7f699015bb90 2024-03-06T07:31:03.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f6994c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f699006afc0 0x7f69900723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33276/0 (socket says 172.21.15.114:33276) 2024-03-06T07:31:03.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f6994c00640 1 -- 172.21.15.114:0/2966156436 learned_addr learned my addr 172.21.15.114:0/2966156436 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f698f400640 1 --2- 172.21.15.114:0/2966156436 >> v2:172.21.15.177:3300/0 conn(0x7f699015efa0 0x7f69900cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f698fe00640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f699015bb90 0x7f699006dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856361830 0 0) 0x7f6990071ea0 con 0x7f699006afc0 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6974003520 con 0x7f699006afc0 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1796481218 0 0) 0x7f6974003520 con 0x7f699006afc0 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f699015bb90 msgr2=0x7f699006dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.906+0000 7f698ea00640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f699015bb90 0x7f699006dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 >> v2:172.21.15.177:3300/0 conn(0x7f699015efa0 msgr2=0x7f69900cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 --2- 172.21.15.114:0/2966156436 >> v2:172.21.15.177:3300/0 conn(0x7f699015efa0 0x7f69900cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699006e660 con 0x7f699006afc0 2024-03-06T07:31:03.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6988001670 con 0x7f699006afc0 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698f400640 1 --2- 172.21.15.114:0/2966156436 >> v2:172.21.15.177:3300/0 conn(0x7f699015efa0 0x7f69900cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 >> v1:172.21.15.114:6789/0 conn(0x7f699006afc0 legacy=0x7f69900723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f699015bb90 0x7f69740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6974003f30 0x7f6974004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6974004d40 0x7f69740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699015bf70 con 0x7f6974003f30 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699006e380 con 0x7f6974004d40 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697400ba10 con 0x7f699015bb90 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698f400640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6974003f30 0x7f6974004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698f400640 1 -- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f699015bb90 msgr2=0x7f69740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698f400640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f699015bb90 0x7f69740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698f400640 1 -- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6974004d40 msgr2=0x7f69740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698f400640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6974004d40 0x7f69740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.907+0000 7f698f400640 1 -- 172.21.15.114:0/2966156436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974003520 con 0x7f6974003f30 2024-03-06T07:31:03.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.908+0000 7f698f400640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6974003f30 0x7f6974004800 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f697c0098a0 tx=0x7f6974005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21783912322927e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00f040 con 0x7f6974003f30 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c004230 con 0x7f6974003f30 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f698ea00640 1 -- 172.21.15.114:0/2966156436 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c003610 con 0x7f6974003f30 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f6996c00640 1 -- 172.21.15.114:0/2966156436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6974003f30 msgr2=0x7f6974004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f6996c00640 1 --2- 172.21.15.114:0/2966156436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6974003f30 0x7f6974004800 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f697c0098a0 tx=0x7f6974005830 comp rx=0 tx=0).stop 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f6994c00640 1 -- 172.21.15.114:0/2966156436 reap_dead start 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f6996c00640 1 -- 172.21.15.114:0/2966156436 shutdown_connections 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f6996c00640 1 -- 172.21.15.114:0/2966156436 >> 172.21.15.114:0/2966156436 conn(0x7f69900b27d0 msgr2=0x7f69900b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f6996c00640 1 -- 172.21.15.114:0/2966156436 shutdown_connections 2024-03-06T07:31:03.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.909+0000 7f6996c00640 1 -- 172.21.15.114:0/2966156436 wait complete. 2024-03-06T07:31:03.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.911+0000 7f6996c00640 1 Processor -- start 2024-03-06T07:31:03.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.911+0000 7f6996c00640 1 -- start start 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f6996c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6974003f30 0x7f69901545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f6996c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69900b3880 0x7f69900b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f6996c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6990160a50 0x7f6990160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f6996c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699015f400 con 0x7f69900b3880 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f6996c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699006ec20 con 0x7f6990160a50 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f6996c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69900cf160 con 0x7f6974003f30 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f698f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69900b3880 0x7f69900b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f698f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69900b3880 0x7f69900b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52046/0 (socket says 172.21.15.114:52046) 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.912+0000 7f698f400640 1 -- 172.21.15.114:0/462598470 learned_addr learned my addr 172.21.15.114:0/462598470 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f698fe00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6974003f30 0x7f69901545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f6994c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6990160a50 0x7f6990160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f698f400640 1 -- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6974003f30 msgr2=0x7f69901545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f698f400640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6974003f30 0x7f69901545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f698f400640 1 -- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6990160a50 msgr2=0x7f6990160ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f698f400640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6990160a50 0x7f6990160ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f698f400640 1 -- 172.21.15.114:0/462598470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6990154af0 con 0x7f69900b3880 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f6994c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6990160a50 0x7f6990160ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.913+0000 7f698f400640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69900b3880 0x7f69900b5800 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f697c02bff0 tx=0x7f697c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.914+0000 7f6987e00640 1 -- 172.21.15.114:0/462598470 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c047070 con 0x7f69900b3880 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.914+0000 7f6987e00640 1 -- 172.21.15.114:0/462598470 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c042400 con 0x7f69900b3880 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.914+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69901f49c0 con 0x7f69900b3880 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.914+0000 7f6987e00640 1 -- 172.21.15.114:0/462598470 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c042810 con 0x7f69900b3880 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.914+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69901f4120 con 0x7f69900b3880 2024-03-06T07:31:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.915+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699004f330 con 0x7f69900b3880 2024-03-06T07:31:03.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.917+0000 7f6987e00640 1 -- 172.21.15.114:0/462598470 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f697c04a070 con 0x7f69900b3880 2024-03-06T07:31:03.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.917+0000 7f6987e00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f697006b4d0 0x7f697006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.918+0000 7f6987e00640 1 -- 172.21.15.114:0/462598470 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f697c0abc20 con 0x7f69900b3880 2024-03-06T07:31:03.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.918+0000 7f698fe00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f697006b4d0 0x7f697006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.918+0000 7f698fe00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f697006b4d0 0x7f697006d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6980007230 tx=0x7f698000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:03.923+0000 7f6987e00640 1 -- 172.21.15.114:0/462598470 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6990154af0 con 0x7f69900b3880 2024-03-06T07:31:04.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.174+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f699015d2d0 con 0x7f69900b3880 2024-03-06T07:31:04.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.175+0000 7f6987e00640 1 -- 172.21.15.114:0/462598470 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7f697c033070 con 0x7f69900b3880 2024-03-06T07:31:04.176 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f697006b4d0 msgr2=0x7f697006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f697006b4d0 0x7f697006d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6980007230 tx=0x7f698000b040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69900b3880 msgr2=0x7f69900b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69900b3880 0x7f69900b5800 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f697c02bff0 tx=0x7f697c004730 comp rx=0 tx=0).stop 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 shutdown_connections 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6990160a50 0x7f6990160ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f697006b4d0 0x7f697006d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69900b3880 0x7f69900b5800 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 --2- 172.21.15.114:0/462598470 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6974003f30 0x7f69901545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 >> 172.21.15.114:0/462598470 conn(0x7f69900b27d0 msgr2=0x7f69900b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 shutdown_connections 2024-03-06T07:31:04.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.178+0000 7f6996c00640 1 -- 172.21.15.114:0/462598470 wait complete. 2024-03-06T07:31:04.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.387+0000 7f296e000640 1 Processor -- start 2024-03-06T07:31:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.387+0000 7f296e000640 1 -- start start 2024-03-06T07:31:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.388+0000 7f296e000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f296815f9d0 0x7f296815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f296e000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296806cef0 con 0x7f2968144c00 2024-03-06T07:31:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f296e000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296806d1d0 con 0x7f2968070b10 2024-03-06T07:31:04.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f296e000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296806d4b0 con 0x7f296815f9d0 2024-03-06T07:31:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f2967400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2968144c00 0x7f296806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33278/0 (socket says 172.21.15.114:33278) 2024-03-06T07:31:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f2967400640 1 -- 172.21.15.114:0/4070495199 learned_addr learned my addr 172.21.15.114:0/4070495199 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998213576 0 0) 0x7f296806cef0 con 0x7f2968144c00 2024-03-06T07:31:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f294c003520 con 0x7f2968144c00 2024-03-06T07:31:04.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.389+0000 7f295ea00640 1 --2- 172.21.15.114:0/4070495199 >> v2:172.21.15.177:3300/0 conn(0x7f296815f9d0 0x7f296815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376481907 0 0) 0x7f294c003520 con 0x7f2968144c00 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 >> v2:172.21.15.177:3300/0 conn(0x7f296815f9d0 msgr2=0x7f296815fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 --2- 172.21.15.114:0/4070495199 >> v2:172.21.15.177:3300/0 conn(0x7f296815f9d0 0x7f296815fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 >> v1:172.21.15.177:6789/0 conn(0x7f2968070b10 legacy=0x7f2968070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296806d790 con 0x7f2968144c00 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f29580019f0 con 0x7f2968144c00 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 >> v1:172.21.15.114:6789/0 conn(0x7f2968144c00 legacy=0x7f296806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c003a00 0x7f294c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f294c004800 0x7f294c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f294c005210 0x7f294c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296806d4b0 con 0x7f294c005210 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c00c140 con 0x7f294c003a00 2024-03-06T07:31:04.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c00c340 con 0x7f294c004800 2024-03-06T07:31:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2967e00640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f294c005210 0x7f294c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2967e00640 1 -- 172.21.15.114:0/4070495199 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f294c004800 msgr2=0x7f294c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2967400640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c003a00 0x7f294c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2967e00640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f294c004800 0x7f294c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2967e00640 1 -- 172.21.15.114:0/4070495199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c003a00 msgr2=0x7f294c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2967e00640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c003a00 0x7f294c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.390+0000 7f2967e00640 1 -- 172.21.15.114:0/4070495199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c003520 con 0x7f294c005210 2024-03-06T07:31:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.391+0000 7f2967e00640 1 -- 172.21.15.114:0/4070495199 reap_dead start 2024-03-06T07:31:04.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.391+0000 7f2967e00640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f294c005210 0x7f294c00b9c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f2960003d80 tx=0x7f294c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89389e33a37beb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2960003980 con 0x7f294c005210 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296002be50 con 0x7f294c005210 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f2966a00640 1 -- 172.21.15.114:0/4070495199 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2960007480 con 0x7f294c005210 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f296e000640 1 -- 172.21.15.114:0/4070495199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f294c005210 msgr2=0x7f294c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f296e000640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f294c005210 0x7f294c00b9c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f2960003d80 tx=0x7f294c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f296e000640 1 -- 172.21.15.114:0/4070495199 shutdown_connections 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f296e000640 1 --2- 172.21.15.114:0/4070495199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f294c005210 0x7f294c00b9c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f296e000640 1 -- 172.21.15.114:0/4070495199 >> 172.21.15.114:0/4070495199 conn(0x7f29680b1730 msgr2=0x7f29680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f296e000640 1 -- 172.21.15.114:0/4070495199 shutdown_connections 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.392+0000 7f296e000640 1 -- 172.21.15.114:0/4070495199 wait complete. 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.394+0000 7f296e000640 1 Processor -- start 2024-03-06T07:31:04.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.394+0000 7f296e000640 1 -- start start 2024-03-06T07:31:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f296e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c005210 0x7f29680a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f296e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f296813fbf0 0x7f296815c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f296e000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f296815cab0 0x7f296815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f296e000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296806ccd0 con 0x7f296813fbf0 2024-03-06T07:31:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f296e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296806cb50 con 0x7f294c005210 2024-03-06T07:31:04.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f296e000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296806df90 con 0x7f296815cab0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f295ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f296813fbf0 0x7f296815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f295ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f296813fbf0 0x7f296815c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52060/0 (socket says 172.21.15.114:52060) 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.395+0000 7f295ea00640 1 -- 172.21.15.114:0/3320980810 learned_addr learned my addr 172.21.15.114:0/3320980810 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295ea00640 1 -- 172.21.15.114:0/3320980810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f296815cab0 msgr2=0x7f296815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295ea00640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f296815cab0 0x7f296815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295ea00640 1 -- 172.21.15.114:0/3320980810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c005210 msgr2=0x7f29680a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295ea00640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c005210 0x7f29680a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295ea00640 1 -- 172.21.15.114:0/3320980810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296813d2a0 con 0x7f296813fbf0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295ea00640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f296813fbf0 0x7f296815c570 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f295000d860 tx=0x7f295000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295fe00640 1 -- 172.21.15.114:0/3320980810 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2950018020 con 0x7f296813fbf0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295fe00640 1 -- 172.21.15.114:0/3320980810 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295000de50 con 0x7f296813fbf0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f295fe00640 1 -- 172.21.15.114:0/3320980810 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295000baf0 con 0x7f296813fbf0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296813d580 con 0x7f296813fbf0 2024-03-06T07:31:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.396+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296813db40 con 0x7f296813fbf0 2024-03-06T07:31:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.397+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296804f330 con 0x7f296813fbf0 2024-03-06T07:31:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.398+0000 7f295fe00640 1 -- 172.21.15.114:0/3320980810 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f295001b020 con 0x7f296813fbf0 2024-03-06T07:31:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.399+0000 7f295fe00640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f294406b500 0x7f294406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.400+0000 7f295fe00640 1 -- 172.21.15.114:0/3320980810 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f295008a4a0 con 0x7f296813fbf0 2024-03-06T07:31:04.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.400+0000 7f2967400640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f294406b500 0x7f294406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.400+0000 7f2967400640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f294406b500 0x7f294406d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2958001350 tx=0x7f2958009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.404+0000 7f295fe00640 1 -- 172.21.15.114:0/3320980810 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2950022030 con 0x7f296813fbf0 2024-03-06T07:31:04.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.635+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2968072890 con 0x7f296813fbf0 2024-03-06T07:31:04.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.637+0000 7f295fe00640 1 -- 172.21.15.114:0/3320980810 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f2950059280 con 0x7f296813fbf0 2024-03-06T07:31:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:04.639 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:31:04.639 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f294406b500 msgr2=0x7f294406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f294406b500 0x7f294406d9c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2958001350 tx=0x7f2958009040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f296813fbf0 msgr2=0x7f296815c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f296813fbf0 0x7f296815c570 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f295000d860 tx=0x7f295000dd20 comp rx=0 tx=0).stop 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 shutdown_connections 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f296815cab0 0x7f296815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f294406b500 0x7f294406d9c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f296813fbf0 0x7f296815c570 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 --2- 172.21.15.114:0/3320980810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f294c005210 0x7f29680a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 >> 172.21.15.114:0/3320980810 conn(0x7f29680b1730 msgr2=0x7f29681452e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 shutdown_connections 2024-03-06T07:31:04.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.639+0000 7f296e000640 1 -- 172.21.15.114:0/3320980810 wait complete. 2024-03-06T07:31:04.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.857+0000 7f659fe00640 1 Processor -- start 2024-03-06T07:31:04.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.857+0000 7f659fe00640 1 -- start start 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f65a01561f0 0x7f65a00ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a00aebf0 0x7f65a00cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f65a00cc800 0x7f65a00cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a015ce30 con 0x7f65a01561f0 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659fe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a015c240 con 0x7f65a00cc800 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a015c500 con 0x7f65a00aebf0 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f65a01561f0 0x7f65a00ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f65a01561f0 0x7f65a00ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52076/0 (socket says 172.21.15.114:52076) 2024-03-06T07:31:04.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.859+0000 7f659ea00640 1 -- 172.21.15.114:0/1100609411 learned_addr learned my addr 172.21.15.114:0/1100609411 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.860+0000 7f6597e00640 1 --2- 172.21.15.114:0/1100609411 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a00aebf0 0x7f65a00cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.860+0000 7f659f400640 1 --2- 172.21.15.114:0/1100609411 >> v2:172.21.15.177:3300/0 conn(0x7f65a00cc800 0x7f65a00cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.860+0000 7f659ea00640 1 -- 172.21.15.114:0/1100609411 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a00aebf0 msgr2=0x7f65a00cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.860+0000 7f659ea00640 1 --2- 172.21.15.114:0/1100609411 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a00aebf0 0x7f65a00cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.860+0000 7f659ea00640 1 -- 172.21.15.114:0/1100609411 >> v2:172.21.15.177:3300/0 conn(0x7f65a00cc800 msgr2=0x7f65a00cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.860+0000 7f659ea00640 1 --2- 172.21.15.114:0/1100609411 >> v2:172.21.15.177:3300/0 conn(0x7f65a00cc800 0x7f65a00cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.860+0000 7f659ea00640 1 -- 172.21.15.114:0/1100609411 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a015c7e0 con 0x7f65a01561f0 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.861+0000 7f659ea00640 1 --2- 172.21.15.114:0/1100609411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a01561f0 0x7f65a00ae6b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f65a00cc1f0 tx=0x7f659002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddfe3df945380ad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659e000640 1 -- 172.21.15.114:0/1100609411 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590035020 con 0x7f65a01561f0 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659e000640 1 -- 172.21.15.114:0/1100609411 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590031870 con 0x7f65a01561f0 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659e000640 1 -- 172.21.15.114:0/1100609411 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590031c40 con 0x7f65a01561f0 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 -- 172.21.15.114:0/1100609411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a01561f0 msgr2=0x7f65a00ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 --2- 172.21.15.114:0/1100609411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a01561f0 0x7f65a00ae6b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f65a00cc1f0 tx=0x7f659002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:04.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 -- 172.21.15.114:0/1100609411 shutdown_connections 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 --2- 172.21.15.114:0/1100609411 >> v2:172.21.15.177:3300/0 conn(0x7f65a00cc800 0x7f65a00cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 --2- 172.21.15.114:0/1100609411 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a00aebf0 0x7f65a00cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 --2- 172.21.15.114:0/1100609411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a01561f0 0x7f65a00ae6b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 -- 172.21.15.114:0/1100609411 >> 172.21.15.114:0/1100609411 conn(0x7f65a006afc0 msgr2=0x7f65a006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 -- 172.21.15.114:0/1100609411 shutdown_connections 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.862+0000 7f659fe00640 1 -- 172.21.15.114:0/1100609411 wait complete. 2024-03-06T07:31:04.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.864+0000 7f659fe00640 1 Processor -- start 2024-03-06T07:31:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.864+0000 7f659fe00640 1 -- start start 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f659fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65a00aebf0 0x7f65a00adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f659fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a00cc800 0x7f65a00cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f659fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a01561f0 0x7f65a0149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f659fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01565d0 con 0x7f65a00cc800 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f659fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00ccc90 con 0x7f65a00aebf0 2024-03-06T07:31:04.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f659fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a0156750 con 0x7f65a01561f0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f6597e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a00cc800 0x7f65a00cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f6597e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a00cc800 0x7f65a00cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52086/0 (socket says 172.21.15.114:52086) 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.865+0000 7f6597e00640 1 -- 172.21.15.114:0/949897827 learned_addr learned my addr 172.21.15.114:0/949897827 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f659ea00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65a00aebf0 0x7f65a00adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f659f400640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a01561f0 0x7f65a0149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6597e00640 1 -- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a01561f0 msgr2=0x7f65a0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6597e00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a01561f0 0x7f65a0149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6597e00640 1 -- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65a00aebf0 msgr2=0x7f65a00adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6597e00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65a00aebf0 0x7f65a00adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6597e00640 1 -- 172.21.15.114:0/949897827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a00ce8c0 con 0x7f65a00cc800 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6597e00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a00cc800 0x7f65a00cde70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f658800d800 tx=0x7f658800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6596a00640 1 -- 172.21.15.114:0/949897827 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588018030 con 0x7f65a00cc800 2024-03-06T07:31:04.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6596a00640 1 -- 172.21.15.114:0/949897827 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65880096a0 con 0x7f65a00cc800 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a00ceba0 con 0x7f65a00cc800 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.866+0000 7f6596a00640 1 -- 172.21.15.114:0/949897827 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588009870 con 0x7f65a00cc800 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.867+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a00cf160 con 0x7f65a00cc800 2024-03-06T07:31:04.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.868+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a004f330 con 0x7f65a00cc800 2024-03-06T07:31:04.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.869+0000 7f6596a00640 1 -- 172.21.15.114:0/949897827 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f658801b070 con 0x7f65a00cc800 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.870+0000 7f6596a00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f657c06b500 0x7f657c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.870+0000 7f6596a00640 1 -- 172.21.15.114:0/949897827 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f658808a3e0 con 0x7f65a00cc800 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.870+0000 7f659ea00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f657c06b500 0x7f657c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.871+0000 7f659ea00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f657c06b500 0x7f657c06d9c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6590036000 tx=0x7f659003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:04.875+0000 7f6596a00640 1 -- 172.21.15.114:0/949897827 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65880592f0 con 0x7f65a00cc800 2024-03-06T07:31:05.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.070+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65a00b2b10 con 0x7f65a00cc800 2024-03-06T07:31:05.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.072+0000 7f6596a00640 1 -- 172.21.15.114:0/949897827 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16713 (secure 0 0 0) 0x7f6588058bf0 con 0x7f65a00cc800 2024-03-06T07:31:05.074 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:05.075 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:32.939149+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f657c06b500 msgr2=0x7f657c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f657c06b500 0x7f657c06d9c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6590036000 tx=0x7f659003c040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a00cc800 msgr2=0x7f65a00cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a00cc800 0x7f65a00cde70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f658800d800 tx=0x7f658800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 shutdown_connections 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f65a01561f0 0x7f65a0149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f657c06b500 0x7f657c06d9c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65a00cc800 0x7f65a00cde70 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 --2- 172.21.15.114:0/949897827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f65a00aebf0 0x7f65a00adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 >> 172.21.15.114:0/949897827 conn(0x7f65a006afc0 msgr2=0x7f65a006d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 shutdown_connections 2024-03-06T07:31:05.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.075+0000 7f659fe00640 1 -- 172.21.15.114:0/949897827 wait complete. 2024-03-06T07:31:05.095 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:05.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.292+0000 7fc925a00640 1 Processor -- start 2024-03-06T07:31:05.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.292+0000 7fc925a00640 1 -- start start 2024-03-06T07:31:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.293+0000 7fc925a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc9200b75b0 0x7fc9200b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.293+0000 7fc925a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9200b5e00 con 0x7fc9200b5020 2024-03-06T07:31:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.293+0000 7fc925a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9200b60e0 con 0x7fc9200cff20 2024-03-06T07:31:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.293+0000 7fc925a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9200b5bc0 con 0x7fc9200b75b0 2024-03-06T07:31:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91f400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc9200b5020 0x7fc9200b54b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33294/0 (socket says 172.21.15.114:33294) 2024-03-06T07:31:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91f400640 1 -- 172.21.15.114:0/3833273532 learned_addr learned my addr 172.21.15.114:0/3833273532 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:05.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657452095 0 0) 0x7fc9200b5e00 con 0x7fc9200b5020 2024-03-06T07:31:05.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91e000640 1 --2- 172.21.15.114:0/3833273532 >> v2:172.21.15.177:3300/0 conn(0x7fc9200b75b0 0x7fc9200b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc904003520 con 0x7fc9200b5020 2024-03-06T07:31:05.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 31679270 0 0) 0x7fc904003520 con 0x7fc9200b5020 2024-03-06T07:31:05.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 >> v2:172.21.15.177:3300/0 conn(0x7fc9200b75b0 msgr2=0x7fc9200b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 --2- 172.21.15.114:0/3833273532 >> v2:172.21.15.177:3300/0 conn(0x7fc9200b75b0 0x7fc9200b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 >> v1:172.21.15.177:6789/0 conn(0x7fc9200cff20 legacy=0x7fc9200b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9200b63c0 con 0x7fc9200b5020 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.294+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fc9180019f0 con 0x7fc9200b5020 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 >> v1:172.21.15.114:6789/0 conn(0x7fc9200b5020 legacy=0x7fc9200b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91d600640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc904003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91d600640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc904004800 0x7fc904004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91d600640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc904005210 0x7fc90400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9200b5bc0 con 0x7fc904003a00 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90400c160 con 0x7fc904005210 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90400c360 con 0x7fc904004800 2024-03-06T07:31:05.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91ea00640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc904003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91ea00640 1 -- 172.21.15.114:0/3833273532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc904004800 msgr2=0x7fc904004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91ea00640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc904004800 0x7fc904004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91ea00640 1 -- 172.21.15.114:0/3833273532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc904005210 msgr2=0x7fc90400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91ea00640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc904005210 0x7fc90400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91f400640 1 -- 172.21.15.114:0/3833273532 reap_dead start 2024-03-06T07:31:05.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.295+0000 7fc91ea00640 1 -- 172.21.15.114:0/3833273532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc904003520 con 0x7fc904003a00 2024-03-06T07:31:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.297+0000 7fc91ea00640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc904003e20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc910001720 tx=0x7fc904005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e79542565284d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.297+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910001370 con 0x7fc904003a00 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.297+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9100052d0 con 0x7fc904003a00 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.297+0000 7fc91d600640 1 -- 172.21.15.114:0/3833273532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910003830 con 0x7fc904003a00 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.298+0000 7fc925a00640 1 -- 172.21.15.114:0/3833273532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 msgr2=0x7fc904003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.298+0000 7fc925a00640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc904003e20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc910001720 tx=0x7fc904005830 comp rx=0 tx=0).stop 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.298+0000 7fc925a00640 1 -- 172.21.15.114:0/3833273532 shutdown_connections 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.298+0000 7fc925a00640 1 --2- 172.21.15.114:0/3833273532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc904003e20 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.298+0000 7fc925a00640 1 -- 172.21.15.114:0/3833273532 >> 172.21.15.114:0/3833273532 conn(0x7fc9200d1c90 msgr2=0x7fc9200d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.298+0000 7fc925a00640 1 -- 172.21.15.114:0/3833273532 shutdown_connections 2024-03-06T07:31:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.298+0000 7fc925a00640 1 -- 172.21.15.114:0/3833273532 wait complete. 2024-03-06T07:31:05.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.300+0000 7fc925a00640 1 Processor -- start 2024-03-06T07:31:05.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.300+0000 7fc925a00640 1 -- start start 2024-03-06T07:31:05.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc925a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc9200cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc925a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92015af00 0x7fc9200cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc925a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92015d510 0x7fc92015d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc925a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc920161a50 con 0x7fc904003a00 2024-03-06T07:31:05.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc925a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9200b6a10 con 0x7fc92015d510 2024-03-06T07:31:05.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc925a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9200b7a30 con 0x7fc92015af00 2024-03-06T07:31:05.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc9200cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc9200cac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52116/0 (socket says 172.21.15.114:52116) 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 -- 172.21.15.114:0/2022364919 learned_addr learned my addr 172.21.15.114:0/2022364919 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91f400640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92015d510 0x7fc92015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 -- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92015af00 msgr2=0x7fc9200cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92015af00 0x7fc9200cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 -- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92015d510 msgr2=0x7fc92015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92015d510 0x7fc92015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91ea00640 1 -- 172.21.15.114:0/2022364919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92015b380 con 0x7fc904003a00 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.301+0000 7fc91f400640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92015d510 0x7fc92015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.302+0000 7fc91ea00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc9200cac50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc91002bc90 tx=0x7fc910001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.302+0000 7fc916a00640 1 -- 172.21.15.114:0/2022364919 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910041070 con 0x7fc904003a00 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.302+0000 7fc916a00640 1 -- 172.21.15.114:0/2022364919 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc910005480 con 0x7fc904003a00 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.302+0000 7fc916a00640 1 -- 172.21.15.114:0/2022364919 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91003d640 con 0x7fc904003a00 2024-03-06T07:31:05.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.302+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9200cb350 con 0x7fc904003a00 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.302+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92015ca30 con 0x7fc904003a00 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.303+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9200cf520 con 0x7fc904003a00 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.304+0000 7fc916a00640 1 -- 172.21.15.114:0/2022364919 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc910044070 con 0x7fc904003a00 2024-03-06T07:31:05.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.305+0000 7fc916a00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90006b500 0x7fc90006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.305+0000 7fc916a00640 1 -- 172.21.15.114:0/2022364919 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9100abdf0 con 0x7fc904003a00 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.305+0000 7fc91e000640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90006b500 0x7fc90006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.306+0000 7fc91e000640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90006b500 0x7fc90006d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc90c0093d0 tx=0x7fc90c009300 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.309+0000 7fc916a00640 1 -- 172.21.15.114:0/2022364919 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9200cf520 con 0x7fc904003a00 2024-03-06T07:31:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.535+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc92004f330 con 0x7fc904003a00 2024-03-06T07:31:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.536+0000 7fc916a00640 1 -- 172.21.15.114:0/2022364919 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fc91007ad00 con 0x7fc904003a00 2024-03-06T07:31:05.538 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:05.538 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:31:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:31:05.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90006b500 msgr2=0x7fc90006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90006b500 0x7fc90006d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc90c0093d0 tx=0x7fc90c009300 comp rx=0 tx=0).stop 2024-03-06T07:31:05.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 msgr2=0x7fc9200cac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc9200cac50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc91002bc90 tx=0x7fc910001120 comp rx=0 tx=0).stop 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 shutdown_connections 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92015d510 0x7fc92015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92015af00 0x7fc9200cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90006b500 0x7fc90006d9c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 --2- 172.21.15.114:0/2022364919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc904003a00 0x7fc9200cac50 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.538+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 >> 172.21.15.114:0/2022364919 conn(0x7fc9200d1c90 msgr2=0x7fc920074030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.539+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 shutdown_connections 2024-03-06T07:31:05.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.539+0000 7fc925a00640 1 -- 172.21.15.114:0/2022364919 wait complete. 2024-03-06T07:31:05.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.753+0000 7f8dc7a00640 1 Processor -- start 2024-03-06T07:31:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.753+0000 7f8dc7a00640 1 -- start start 2024-03-06T07:31:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dc7a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8dc015f470 0x7f8dc00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dc7a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc0160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dc7a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc00cea60 con 0x7f8dc015e0d0 2024-03-06T07:31:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dc7a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc00d2f60 con 0x7f8dc015f470 2024-03-06T07:31:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dc7a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00d1240 con 0x7f8dc00d0df0 2024-03-06T07:31:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dbfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8dc015f470 0x7f8dc00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dbfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8dc015f470 0x7f8dc00d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52130/0 (socket says 172.21.15.114:52130) 2024-03-06T07:31:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dbfe00640 1 -- 172.21.15.114:0/310530456 learned_addr learned my addr 172.21.15.114:0/310530456 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dc5a00640 1 --2- 172.21.15.114:0/310530456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc0160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dbfe00640 1 -- 172.21.15.114:0/310530456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 msgr2=0x7f8dc0160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dbfe00640 1 --2- 172.21.15.114:0/310530456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc0160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.755+0000 7f8dbfe00640 1 -- 172.21.15.114:0/310530456 >> v1:172.21.15.177:6789/0 conn(0x7f8dc015e0d0 legacy=0x7f8dc00d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.756+0000 7f8dbfe00640 1 -- 172.21.15.114:0/310530456 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00d2350 con 0x7f8dc015f470 2024-03-06T07:31:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.756+0000 7f8dc5a00640 1 --2- 172.21.15.114:0/310530456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc0160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.757+0000 7f8dbfe00640 1 --2- 172.21.15.114:0/310530456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015f470 0x7f8dc00d08b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009870 tx=0x7f8dac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0020c09ecc46137 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.757+0000 7f8dbf400640 1 -- 172.21.15.114:0/310530456 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac034070 con 0x7f8dc015f470 2024-03-06T07:31:05.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.757+0000 7f8dbf400640 1 -- 172.21.15.114:0/310530456 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac030840 con 0x7f8dc015f470 2024-03-06T07:31:05.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.757+0000 7f8dbf400640 1 -- 172.21.15.114:0/310530456 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac030c10 con 0x7f8dc015f470 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 -- 172.21.15.114:0/310530456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015f470 msgr2=0x7f8dc00d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/310530456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015f470 0x7f8dc00d08b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009870 tx=0x7f8dac02b980 comp rx=0 tx=0).stop 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 -- 172.21.15.114:0/310530456 shutdown_connections 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/310530456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc0160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/310530456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015f470 0x7f8dc00d08b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 -- 172.21.15.114:0/310530456 >> 172.21.15.114:0/310530456 conn(0x7f8dc00d3ec0 msgr2=0x7f8dc00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 -- 172.21.15.114:0/310530456 shutdown_connections 2024-03-06T07:31:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.758+0000 7f8dc7a00640 1 -- 172.21.15.114:0/310530456 wait complete. 2024-03-06T07:31:05.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.760+0000 7f8dc7a00640 1 Processor -- start 2024-03-06T07:31:05.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.761+0000 7f8dc7a00640 1 -- start start 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.762+0000 7f8dc7a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc01515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.762+0000 7f8dc7a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015e0d0 0x7f8dc00cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.762+0000 7f8dc7a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dc015f470 0x7f8dc00b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.762+0000 7f8dc7a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00d2bf0 con 0x7f8dc015e0d0 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.762+0000 7f8dc7a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00d2910 con 0x7f8dc015f470 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.762+0000 7f8dc7a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc015fb30 con 0x7f8dc00d0df0 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015e0d0 0x7f8dc00cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015e0d0 0x7f8dc00cc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52134/0 (socket says 172.21.15.114:52134) 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 -- 172.21.15.114:0/3653293537 learned_addr learned my addr 172.21.15.114:0/3653293537 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:05.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dc5a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dc015f470 0x7f8dc00b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 -- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 msgr2=0x7f8dc01515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dc5000640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc01515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc01515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 -- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dc015f470 msgr2=0x7f8dc00b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dc015f470 0x7f8dc00b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 -- 172.21.15.114:0/3653293537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00cd410 con 0x7f8dc015e0d0 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dc5a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dc015f470 0x7f8dc00b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dc5000640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc01515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:05.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.763+0000 7f8dbfe00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015e0d0 0x7f8dc00cc970 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8dac000c00 tx=0x7f8dac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.764+0000 7f8dbcc00640 1 -- 172.21.15.114:0/3653293537 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac048070 con 0x7f8dc015e0d0 2024-03-06T07:31:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.764+0000 7f8dbcc00640 1 -- 172.21.15.114:0/3653293537 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac030640 con 0x7f8dc015e0d0 2024-03-06T07:31:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.764+0000 7f8dbcc00640 1 -- 172.21.15.114:0/3653293537 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac034040 con 0x7f8dc015e0d0 2024-03-06T07:31:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.764+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc00cd6f0 con 0x7f8dc015e0d0 2024-03-06T07:31:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.764+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc0151b20 con 0x7f8dc015e0d0 2024-03-06T07:31:05.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.765+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc004f330 con 0x7f8dc015e0d0 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.767+0000 7f8dbcc00640 1 -- 172.21.15.114:0/3653293537 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dac037070 con 0x7f8dc015e0d0 2024-03-06T07:31:05.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.768+0000 7f8dbcc00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8da006b500 0x7f8da006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.768+0000 7f8dbcc00640 1 -- 172.21.15.114:0/3653293537 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dac0ac130 con 0x7f8dc015e0d0 2024-03-06T07:31:05.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.768+0000 7f8dc5000640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8da006b500 0x7f8da006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.769+0000 7f8dc5000640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8da006b500 0x7f8da006d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8db400a830 tx=0x7f8db4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:05.773+0000 7f8dbcc00640 1 -- 172.21.15.114:0/3653293537 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8dac07b0c0 con 0x7f8dc015e0d0 2024-03-06T07:31:06.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.002+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dc0151d80 con 0x7f8dc015e0d0 2024-03-06T07:31:06.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.003+0000 7f8dbcc00640 1 -- 172.21.15.114:0/3653293537 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f8dac07a9c0 con 0x7f8dc015e0d0 2024-03-06T07:31:06.005 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:06.005 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:31.363295+0000","modified":"2024-03-06T07:30:34.444971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5904":{"gid":5904,"name":"a","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/4290788305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4290788305},{"type":"v1","addr":"172.21.15.114:6837","nonce":4290788305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5904,"qdb_cluster":[5904]},"id":4}]} 2024-03-06T07:31:06.005 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 35 2024-03-06T07:31:06.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8da006b500 msgr2=0x7f8da006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8da006b500 0x7f8da006d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8db400a830 tx=0x7f8db4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015e0d0 msgr2=0x7f8dc00cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015e0d0 0x7f8dc00cc970 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8dac000c00 tx=0x7f8dac004730 comp rx=0 tx=0).stop 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 shutdown_connections 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8dc015f470 0x7f8dc00b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8da006b500 0x7f8da006d9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc015e0d0 0x7f8dc00cc970 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 --2- 172.21.15.114:0/3653293537 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8dc00d0df0 0x7f8dc01515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.005+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 >> 172.21.15.114:0/3653293537 conn(0x7f8dc00d3ec0 msgr2=0x7f8dc00d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.006+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 shutdown_connections 2024-03-06T07:31:06.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.006+0000 7f8dc7a00640 1 -- 172.21.15.114:0/3653293537 wait complete. 2024-03-06T07:31:06.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.228+0000 7f50a2400640 1 Processor -- start 2024-03-06T07:31:06.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.228+0000 7f50a2400640 1 -- start start 2024-03-06T07:31:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f50a2400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15e0d0 0x7f509c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f50a2400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f509c15f470 0x7f509c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f50a2400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f509c0d0df0 0x7f509c160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f50a2400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c0d1260 con 0x7f509c0d0df0 2024-03-06T07:31:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f50a2400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c0d2f60 con 0x7f509c15f470 2024-03-06T07:31:06.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f50a2400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0cea60 con 0x7f509c15e0d0 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f509be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f509c0d0df0 0x7f509c160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f509be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f509c0d0df0 0x7f509c160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52140/0 (socket says 172.21.15.114:52140) 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f509be00640 1 -- 172.21.15.114:0/1219783724 learned_addr learned my addr 172.21.15.114:0/1219783724 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f509b400640 1 --2- 172.21.15.114:0/1219783724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15e0d0 0x7f509c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f5093e00640 1 --2- 172.21.15.114:0/1219783724 >> v2:172.21.15.177:3300/0 conn(0x7f509c15f470 0x7f509c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f509be00640 1 -- 172.21.15.114:0/1219783724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15e0d0 msgr2=0x7f509c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.230+0000 7f509be00640 1 --2- 172.21.15.114:0/1219783724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15e0d0 0x7f509c0d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.231+0000 7f509be00640 1 -- 172.21.15.114:0/1219783724 >> v2:172.21.15.177:3300/0 conn(0x7f509c15f470 msgr2=0x7f509c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.231+0000 7f509be00640 1 --2- 172.21.15.114:0/1219783724 >> v2:172.21.15.177:3300/0 conn(0x7f509c15f470 0x7f509c0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.231+0000 7f509be00640 1 -- 172.21.15.114:0/1219783724 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c0d2350 con 0x7f509c0d0df0 2024-03-06T07:31:06.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.231+0000 7f5093e00640 1 --2- 172.21.15.114:0/1219783724 >> v2:172.21.15.177:3300/0 conn(0x7f509c15f470 0x7f509c0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.232+0000 7f509be00640 1 --2- 172.21.15.114:0/1219783724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c0d0df0 0x7f509c160dc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f509c08de20 tx=0x7f509402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafa62c08cebc9b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.232+0000 7f509aa00640 1 -- 172.21.15.114:0/1219783724 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509402cb00 con 0x7f509c0d0df0 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.232+0000 7f509aa00640 1 -- 172.21.15.114:0/1219783724 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509402cca0 con 0x7f509c0d0df0 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.232+0000 7f509aa00640 1 -- 172.21.15.114:0/1219783724 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094034610 con 0x7f509c0d0df0 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.232+0000 7f50a2400640 1 -- 172.21.15.114:0/1219783724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c0d0df0 msgr2=0x7f509c160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.232+0000 7f50a2400640 1 --2- 172.21.15.114:0/1219783724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c0d0df0 0x7f509c160dc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f509c08de20 tx=0x7f509402bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.233+0000 7f50a2400640 1 -- 172.21.15.114:0/1219783724 shutdown_connections 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.233+0000 7f50a2400640 1 --2- 172.21.15.114:0/1219783724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c0d0df0 0x7f509c160dc0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.233+0000 7f50a2400640 1 --2- 172.21.15.114:0/1219783724 >> v2:172.21.15.177:3300/0 conn(0x7f509c15f470 0x7f509c0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.233+0000 7f50a2400640 1 --2- 172.21.15.114:0/1219783724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15e0d0 0x7f509c0d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.233+0000 7f50a2400640 1 -- 172.21.15.114:0/1219783724 >> 172.21.15.114:0/1219783724 conn(0x7f509c0d3ec0 msgr2=0x7f509c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.233+0000 7f50a2400640 1 -- 172.21.15.114:0/1219783724 shutdown_connections 2024-03-06T07:31:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.233+0000 7f50a2400640 1 -- 172.21.15.114:0/1219783724 wait complete. 2024-03-06T07:31:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.234+0000 7f50a2400640 1 Processor -- start 2024-03-06T07:31:06.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.234+0000 7f50a2400640 1 -- start start 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.235+0000 7f50a2400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f509c0d0df0 0x7f509c0c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.235+0000 7f50a2400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c15e0d0 0x7f509c0c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.235+0000 7f50a2400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15f470 0x7f509c15acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.235+0000 7f50a2400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0d2bf0 con 0x7f509c15e0d0 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.235+0000 7f50a2400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0d2910 con 0x7f509c0d0df0 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.235+0000 7f50a2400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c15fb30 con 0x7f509c15f470 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c15e0d0 0x7f509c0c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c15e0d0 0x7f509c0c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52144/0 (socket says 172.21.15.114:52144) 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 -- 172.21.15.114:0/595939994 learned_addr learned my addr 172.21.15.114:0/595939994 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f509be00640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15f470 0x7f509c15acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f509b400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f509c0d0df0 0x7f509c0c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 -- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15f470 msgr2=0x7f509c15acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15f470 0x7f509c15acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 -- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f509c0d0df0 msgr2=0x7f509c0c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f509c0d0df0 0x7f509c0c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.236+0000 7f5093e00640 1 -- 172.21.15.114:0/595939994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c1f3c10 con 0x7f509c15e0d0 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.237+0000 7f5093e00640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c15e0d0 0x7f509c0c8650 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f508400c820 tx=0x7f508400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.237+0000 7f5093400640 1 -- 172.21.15.114:0/595939994 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084015070 con 0x7f509c15e0d0 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.237+0000 7f5093400640 1 -- 172.21.15.114:0/595939994 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5084007a30 con 0x7f509c15e0d0 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.237+0000 7f5093400640 1 -- 172.21.15.114:0/595939994 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084007e00 con 0x7f509c15e0d0 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.237+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c1f3ef0 con 0x7f509c15e0d0 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.237+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c1f44b0 con 0x7f509c15e0d0 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.238+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c04f330 con 0x7f509c15e0d0 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.239+0000 7f5093400640 1 -- 172.21.15.114:0/595939994 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5084018070 con 0x7f509c15e0d0 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.240+0000 7f5093400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f507806b4d0 0x7f507806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.240+0000 7f5093400640 1 -- 172.21.15.114:0/595939994 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f508408a190 con 0x7f509c15e0d0 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.240+0000 7f509b400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f507806b4d0 0x7f507806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.241+0000 7f509b400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f507806b4d0 0x7f507806d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f508c006fd0 tx=0x7f508c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.245+0000 7f5093400640 1 -- 172.21.15.114:0/595939994 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5084058f70 con 0x7f509c15e0d0 2024-03-06T07:31:06.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.441+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f509c08d920 con 0x7f509c15e0d0 2024-03-06T07:31:06.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.443+0000 7f5093400640 1 -- 172.21.15.114:0/595939994 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16713 (secure 0 0 0) 0x7f5084058870 con 0x7f509c15e0d0 2024-03-06T07:31:06.445 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:06.445 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:30:32.939149+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:29.190809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:30.382190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:06.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.445+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f507806b4d0 msgr2=0x7f507806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.445+0000 7f50a2400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f507806b4d0 0x7f507806d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f508c006fd0 tx=0x7f508c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.445+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c15e0d0 msgr2=0x7f509c0c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.445+0000 7f50a2400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c15e0d0 0x7f509c0c8650 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f508400c820 tx=0x7f508400cc20 comp rx=0 tx=0).stop 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 shutdown_connections 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f509c15f470 0x7f509c15acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f507806b4d0 0x7f507806d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c15e0d0 0x7f509c0c8650 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 --2- 172.21.15.114:0/595939994 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f509c0d0df0 0x7f509c0c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 >> 172.21.15.114:0/595939994 conn(0x7f509c0d3ec0 msgr2=0x7f509c06c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 shutdown_connections 2024-03-06T07:31:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.446+0000 7f50a2400640 1 -- 172.21.15.114:0/595939994 wait complete. 2024-03-06T07:31:06.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:31:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.663+0000 7fe199e00640 1 Processor -- start 2024-03-06T07:31:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.663+0000 7fe199e00640 1 -- start start 2024-03-06T07:31:06.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.664+0000 7fe199e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe1941474c0 0x7fe1940cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.664+0000 7fe199e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1940ae380 con 0x7fe1941444e0 2024-03-06T07:31:06.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.664+0000 7fe199e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1940ae660 con 0x7fe1940c5040 2024-03-06T07:31:06.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.664+0000 7fe199e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1940ae940 con 0x7fe1941474c0 2024-03-06T07:31:06.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.665+0000 7fe192a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe1941444e0 0x7fe1940c4930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33302/0 (socket says 172.21.15.114:33302) 2024-03-06T07:31:06.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.665+0000 7fe192a00640 1 -- 172.21.15.114:0/2678431922 learned_addr learned my addr 172.21.15.114:0/2678431922 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:06.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.665+0000 7fe193400640 1 --2- 172.21.15.114:0/2678431922 >> v2:172.21.15.114:3300/0 conn(0x7fe1941474c0 0x7fe1940cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.665+0000 7fe192000640 1 -- 172.21.15.114:0/2678431922 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124127130 0 0) 0x7fe1940ae380 con 0x7fe1941444e0 2024-03-06T07:31:06.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.665+0000 7fe192000640 1 -- 172.21.15.114:0/2678431922 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe178003520 con 0x7fe1941444e0 2024-03-06T07:31:06.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.665+0000 7fe193400640 1 -- 172.21.15.114:0/2678431922 >> v1:172.21.15.177:6789/0 conn(0x7fe1940c5040 legacy=0x7fe1940cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.665+0000 7fe193400640 1 -- 172.21.15.114:0/2678431922 >> v1:172.21.15.114:6789/0 conn(0x7fe1941444e0 legacy=0x7fe1940c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.666+0000 7fe193400640 1 -- 172.21.15.114:0/2678431922 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940aec20 con 0x7fe1941474c0 2024-03-06T07:31:06.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.667+0000 7fe193400640 1 --2- 172.21.15.114:0/2678431922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe1940cd010 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009d90 tx=0x7fe18c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935282ba4ec19c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.667+0000 7fe192000640 1 -- 172.21.15.114:0/2678431922 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c039070 con 0x7fe1941474c0 2024-03-06T07:31:06.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.667+0000 7fe192000640 1 -- 172.21.15.114:0/2678431922 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c02cd00 con 0x7fe1941474c0 2024-03-06T07:31:06.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.667+0000 7fe192000640 1 -- 172.21.15.114:0/2678431922 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c034640 con 0x7fe1941474c0 2024-03-06T07:31:06.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.667+0000 7fe199e00640 1 -- 172.21.15.114:0/2678431922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 msgr2=0x7fe1940cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.667+0000 7fe199e00640 1 --2- 172.21.15.114:0/2678431922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe1940cd010 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009d90 tx=0x7fe18c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:31:06.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.668+0000 7fe199e00640 1 -- 172.21.15.114:0/2678431922 shutdown_connections 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.668+0000 7fe199e00640 1 --2- 172.21.15.114:0/2678431922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe1940cd010 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.668+0000 7fe199e00640 1 -- 172.21.15.114:0/2678431922 >> 172.21.15.114:0/2678431922 conn(0x7fe1940cc4f0 msgr2=0x7fe1940cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.668+0000 7fe199e00640 1 -- 172.21.15.114:0/2678431922 shutdown_connections 2024-03-06T07:31:06.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.668+0000 7fe199e00640 1 -- 172.21.15.114:0/2678431922 wait complete. 2024-03-06T07:31:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.669+0000 7fe199e00640 1 Processor -- start 2024-03-06T07:31:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.669+0000 7fe199e00640 1 -- start start 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.670+0000 7fe199e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1940c5040 0x7fe1941e6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.670+0000 7fe199e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1941444e0 0x7fe1941f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.670+0000 7fe199e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe194145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.670+0000 7fe199e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941448c0 con 0x7fe1941474c0 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.670+0000 7fe199e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194147cb0 con 0x7fe1940c5040 2024-03-06T07:31:06.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.670+0000 7fe199e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe194144a40 con 0x7fe1941444e0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe194145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe194145a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52168/0 (socket says 172.21.15.114:52168) 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 -- 172.21.15.114:0/238445530 learned_addr learned my addr 172.21.15.114:0/238445530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe192a00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1940c5040 0x7fe1941e6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe18be00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1941444e0 0x7fe1941f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 -- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1941444e0 msgr2=0x7fe1941f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1941444e0 0x7fe1941f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 -- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1940c5040 msgr2=0x7fe1941e6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1940c5040 0x7fe1941e6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 -- 172.21.15.114:0/238445530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe194144cf0 con 0x7fe1941474c0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe192a00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1940c5040 0x7fe1941e6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.671+0000 7fe193400640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe194145a90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe18c037040 tx=0x7fe18c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.672+0000 7fe18b400640 1 -- 172.21.15.114:0/238445530 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c044020 con 0x7fe1941474c0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.672+0000 7fe18b400640 1 -- 172.21.15.114:0/238445530 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c03a040 con 0x7fe1941474c0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.672+0000 7fe18b400640 1 -- 172.21.15.114:0/238445530 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c039050 con 0x7fe1941474c0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.672+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe194144fd0 con 0x7fe1941474c0 2024-03-06T07:31:06.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.672+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe194145590 con 0x7fe1941474c0 2024-03-06T07:31:06.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.673+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19404f330 con 0x7fe1941474c0 2024-03-06T07:31:06.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.674+0000 7fe18b400640 1 -- 172.21.15.114:0/238445530 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe18c003bb0 con 0x7fe1941474c0 2024-03-06T07:31:06.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.675+0000 7fe18b400640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.675+0000 7fe18b400640 1 -- 172.21.15.114:0/238445530 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe18c0ac640 con 0x7fe1941474c0 2024-03-06T07:31:06.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.675+0000 7fe192a00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.676+0000 7fe192a00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe17006b4d0 0x7fe17006d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe1840011e0 tx=0x7fe184009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.679+0000 7fe18b400640 1 -- 172.21.15.114:0/238445530 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe18c04e060 con 0x7fe1941474c0 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.912+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe194074c50 con 0x7fe1941474c0 2024-03-06T07:31:06.915 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:31:06.913+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:31:06.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.916+0000 7fe18b400640 1 -- 172.21.15.114:0/238445530 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe18c07b610 con 0x7fe1941474c0 2024-03-06T07:31:06.918 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:31:06.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe17006b4d0 msgr2=0x7fe17006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe17006b4d0 0x7fe17006d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe1840011e0 tx=0x7fe184009040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 msgr2=0x7fe194145a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe194145a90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe18c037040 tx=0x7fe18c004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:06.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 shutdown_connections 2024-03-06T07:31:06.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1941474c0 0x7fe194145a90 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe1941444e0 0x7fe1941f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 --2- 172.21.15.114:0/238445530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1940c5040 0x7fe1941e6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.920+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 >> 172.21.15.114:0/238445530 conn(0x7fe1940cc4f0 msgr2=0x7fe194072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.923+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 shutdown_connections 2024-03-06T07:31:06.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:06.923+0000 7fe199e00640 1 -- 172.21.15.114:0/238445530 wait complete. 2024-03-06T07:31:06.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:31:07.024 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:31:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.128+0000 7f993e400640 1 Processor -- start 2024-03-06T07:31:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.128+0000 7f993e400640 1 -- start start 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.129+0000 7f993e400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 0x7f99380727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f993e400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99380b3650 0x7f99380b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f993e400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f99380cda90 0x7f993806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f993e400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9938072310 con 0x7f99380b3650 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f993e400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99380b4710 con 0x7f99380cda90 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f993e400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99380719e0 con 0x7f99380b3df0 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99380b3650 0x7f99380b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f99380b3650 0x7f99380b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52210/0 (socket says 172.21.15.114:52210) 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 -- 172.21.15.114:0/2894443471 learned_addr learned my addr 172.21.15.114:0/2894443471 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f9937400640 1 --2- 172.21.15.114:0/2894443471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 0x7f99380727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f9937e00640 1 --2- 172.21.15.114:0/2894443471 >> v2:172.21.15.177:3300/0 conn(0x7f99380cda90 0x7f993806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 -- 172.21.15.114:0/2894443471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 msgr2=0x7f99380727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 --2- 172.21.15.114:0/2894443471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 0x7f99380727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 -- 172.21.15.114:0/2894443471 >> v2:172.21.15.177:3300/0 conn(0x7f99380cda90 msgr2=0x7f993806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 --2- 172.21.15.114:0/2894443471 >> v2:172.21.15.177:3300/0 conn(0x7f99380cda90 0x7f993806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.130+0000 7f992ea00640 1 -- 172.21.15.114:0/2894443471 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9938071cc0 con 0x7f99380b3650 2024-03-06T07:31:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.131+0000 7f992ea00640 1 --2- 172.21.15.114:0/2894443471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f99380b41d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f99380af840 tx=0x7f992002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2104a5556bf2eebc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f9936a00640 1 -- 172.21.15.114:0/2894443471 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920035020 con 0x7f99380b3650 2024-03-06T07:31:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f9936a00640 1 -- 172.21.15.114:0/2894443471 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9920031870 con 0x7f99380b3650 2024-03-06T07:31:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f9936a00640 1 -- 172.21.15.114:0/2894443471 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920031c40 con 0x7f99380b3650 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f993e400640 1 -- 172.21.15.114:0/2894443471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 msgr2=0x7f99380b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f993e400640 1 --2- 172.21.15.114:0/2894443471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f99380b41d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f99380af840 tx=0x7f992002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f993e400640 1 -- 172.21.15.114:0/2894443471 shutdown_connections 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f993e400640 1 --2- 172.21.15.114:0/2894443471 >> v2:172.21.15.177:3300/0 conn(0x7f99380cda90 0x7f993806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f993e400640 1 --2- 172.21.15.114:0/2894443471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f99380b41d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f993e400640 1 --2- 172.21.15.114:0/2894443471 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 0x7f99380727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.132+0000 7f993e400640 1 -- 172.21.15.114:0/2894443471 >> 172.21.15.114:0/2894443471 conn(0x7f99380b1940 msgr2=0x7f99380b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.133+0000 7f993e400640 1 -- 172.21.15.114:0/2894443471 shutdown_connections 2024-03-06T07:31:07.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.133+0000 7f993e400640 1 -- 172.21.15.114:0/2894443471 wait complete. 2024-03-06T07:31:07.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.134+0000 7f993e400640 1 Processor -- start 2024-03-06T07:31:07.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.134+0000 7f993e400640 1 -- start start 2024-03-06T07:31:07.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.135+0000 7f993e400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f9938156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.135+0000 7f993e400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 0x7f993814f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.135+0000 7f993e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99380cda90 0x7f993809b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.135+0000 7f993e400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993815b240 con 0x7f99380b3650 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.135+0000 7f993e400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99380b3b30 con 0x7f99380cda90 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.135+0000 7f993e400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993815b3c0 con 0x7f99380b3df0 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f9938156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f9938156340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52214/0 (socket says 172.21.15.114:52214) 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 -- 172.21.15.114:0/313930934 learned_addr learned my addr 172.21.15.114:0/313930934 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937e00640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99380cda90 0x7f993809b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 -- 172.21.15.114:0/313930934 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 msgr2=0x7f993814f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 0x7f993814f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 -- 172.21.15.114:0/313930934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99380cda90 msgr2=0x7f993809b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99380cda90 0x7f993809b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 -- 172.21.15.114:0/313930934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99380cab70 con 0x7f99380b3650 2024-03-06T07:31:07.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.136+0000 7f9937400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f9938156340 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f992800d830 tx=0x7f992800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.137+0000 7f992fe00640 1 -- 172.21.15.114:0/313930934 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928018030 con 0x7f99380b3650 2024-03-06T07:31:07.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.137+0000 7f992fe00640 1 -- 172.21.15.114:0/313930934 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992800be10 con 0x7f99380b3650 2024-03-06T07:31:07.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.137+0000 7f992fe00640 1 -- 172.21.15.114:0/313930934 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99280098b0 con 0x7f99380b3650 2024-03-06T07:31:07.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.137+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99381f8000 con 0x7f99380b3650 2024-03-06T07:31:07.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.137+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99381f85c0 con 0x7f99380b3650 2024-03-06T07:31:07.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.138+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993804f330 con 0x7f99380b3650 2024-03-06T07:31:07.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.139+0000 7f992fe00640 1 -- 172.21.15.114:0/313930934 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f992801b070 con 0x7f99380b3650 2024-03-06T07:31:07.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.139+0000 7f992fe00640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f991406b4d0 0x7f991406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.140+0000 7f992fe00640 1 -- 172.21.15.114:0/313930934 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f992808a400 con 0x7f99380b3650 2024-03-06T07:31:07.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.140+0000 7f992ea00640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f991406b4d0 0x7f991406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.140+0000 7f992ea00640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f991406b4d0 0x7f991406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9920036000 tx=0x7f9920030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.146+0000 7f992fe00640 1 -- 172.21.15.114:0/313930934 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9928059210 con 0x7f99380b3650 2024-03-06T07:31:07.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.382+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f99380b4360 con 0x7f99380b3650 2024-03-06T07:31:07.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.927+0000 7f992fe00640 1 -- 172.21.15.114:0/313930934 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9928058b10 con 0x7f99380b3650 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f991406b4d0 msgr2=0x7f991406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f991406b4d0 0x7f991406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9920036000 tx=0x7f9920030000 comp rx=0 tx=0).stop 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 msgr2=0x7f9938156340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f9938156340 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f992800d830 tx=0x7f992800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 shutdown_connections 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99380cda90 0x7f993809b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99380b3df0 0x7f993814f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f991406b4d0 0x7f991406d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 --2- 172.21.15.114:0/313930934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99380b3650 0x7f9938156340 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.931+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 >> 172.21.15.114:0/313930934 conn(0x7f99380b1940 msgr2=0x7f9938153c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.932+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 shutdown_connections 2024-03-06T07:31:07.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:07.932+0000 7f993e400640 1 -- 172.21.15.114:0/313930934 wait complete. 2024-03-06T07:31:07.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:31:08.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.157+0000 7fccc1600640 1 Processor -- start 2024-03-06T07:31:08.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.157+0000 7fccc1600640 1 -- start start 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccc1600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc15df30 0x7fccbc0cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccc1600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccbc0d47b0 0x7fccbc0d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccc1600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc0cfa00 con 0x7fccbc0d06a0 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccc1600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc0d1860 con 0x7fccbc0d47b0 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccc1600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc0d1ab0 con 0x7fccbc15df30 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccbb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccbc0d47b0 0x7fccbc0d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccbb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccbc0d47b0 0x7fccbc0d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52584/0 (socket says 172.21.15.114:52584) 2024-03-06T07:31:08.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.159+0000 7fccbb400640 1 -- 172.21.15.114:0/1715328849 learned_addr learned my addr 172.21.15.114:0/1715328849 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.160+0000 7fccbbe00640 1 --2- 172.21.15.114:0/1715328849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc15df30 0x7fccbc0cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.160+0000 7fccbb400640 1 -- 172.21.15.114:0/1715328849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc15df30 msgr2=0x7fccbc0cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.160+0000 7fccbb400640 1 --2- 172.21.15.114:0/1715328849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc15df30 0x7fccbc0cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.160+0000 7fccbb400640 1 -- 172.21.15.114:0/1715328849 >> v1:172.21.15.114:6789/0 conn(0x7fccbc0d06a0 legacy=0x7fccbc0c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.160+0000 7fccbb400640 1 -- 172.21.15.114:0/1715328849 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc0d1d90 con 0x7fccbc0d47b0 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.160+0000 7fccbaa00640 1 -- 172.21.15.114:0/1715328849 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418122629 0 0) 0x7fccbc0cfa00 con 0x7fccbc0d06a0 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.161+0000 7fccbb400640 1 --2- 172.21.15.114:0/1715328849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d47b0 0x7fccbc0d2590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009e50 tx=0x7fcca402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff09d443c09a0d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.161+0000 7fccbaa00640 1 -- 172.21.15.114:0/1715328849 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca402ca70 con 0x7fccbc0d47b0 2024-03-06T07:31:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.161+0000 7fccbaa00640 1 -- 172.21.15.114:0/1715328849 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca402cc10 con 0x7fccbc0d47b0 2024-03-06T07:31:08.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.161+0000 7fccbaa00640 1 -- 172.21.15.114:0/1715328849 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca40343f0 con 0x7fccbc0d47b0 2024-03-06T07:31:08.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 -- 172.21.15.114:0/1715328849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d47b0 msgr2=0x7fccbc0d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 --2- 172.21.15.114:0/1715328849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d47b0 0x7fccbc0d2590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009e50 tx=0x7fcca402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 -- 172.21.15.114:0/1715328849 shutdown_connections 2024-03-06T07:31:08.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 --2- 172.21.15.114:0/1715328849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d47b0 0x7fccbc0d2590 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 --2- 172.21.15.114:0/1715328849 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc15df30 0x7fccbc0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 -- 172.21.15.114:0/1715328849 >> 172.21.15.114:0/1715328849 conn(0x7fccbc0d3c90 msgr2=0x7fccbc0d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 -- 172.21.15.114:0/1715328849 shutdown_connections 2024-03-06T07:31:08.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.162+0000 7fccc1600640 1 -- 172.21.15.114:0/1715328849 wait complete. 2024-03-06T07:31:08.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.164+0000 7fccc1600640 1 Processor -- start 2024-03-06T07:31:08.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.164+0000 7fccc1600640 1 -- start start 2024-03-06T07:31:08.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccc1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d06a0 0x7fccbc156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccc1600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc0d47b0 0x7fccbc0cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccc1600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc15df30 0x7fccbc0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccc1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc0d2c50 con 0x7fccbc0d06a0 2024-03-06T07:31:08.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccc1600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc0d2ad0 con 0x7fccbc15df30 2024-03-06T07:31:08.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccc1600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc15e310 con 0x7fccbc0d47b0 2024-03-06T07:31:08.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d06a0 0x7fccbc156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d06a0 0x7fccbc156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52598/0 (socket says 172.21.15.114:52598) 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 -- 172.21.15.114:0/87200709 learned_addr learned my addr 172.21.15.114:0/87200709 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbb400640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc0d47b0 0x7fccbc0cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 -- 172.21.15.114:0/87200709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc0d47b0 msgr2=0x7fccbc0cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc0d47b0 0x7fccbc0cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 -- 172.21.15.114:0/87200709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc15df30 msgr2=0x7fccbc0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc15df30 0x7fccbc0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.165+0000 7fccbbe00640 1 -- 172.21.15.114:0/87200709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc1531c0 con 0x7fccbc0d06a0 2024-03-06T07:31:08.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.166+0000 7fccbbe00640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d06a0 0x7fccbc156190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fccac00d5f0 tx=0x7fccac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.166+0000 7fccb3e00640 1 -- 172.21.15.114:0/87200709 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac0125f0 con 0x7fccbc0d06a0 2024-03-06T07:31:08.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.166+0000 7fccb3e00640 1 -- 172.21.15.114:0/87200709 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac019070 con 0x7fccbc0d06a0 2024-03-06T07:31:08.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.166+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccbc0cd330 con 0x7fccbc0d06a0 2024-03-06T07:31:08.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.166+0000 7fccb3e00640 1 -- 172.21.15.114:0/87200709 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac0129d0 con 0x7fccbc0d06a0 2024-03-06T07:31:08.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.166+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc156e70 con 0x7fccbc0d06a0 2024-03-06T07:31:08.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.167+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccbc04f330 con 0x7fccbc0d06a0 2024-03-06T07:31:08.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.168+0000 7fccb3e00640 1 -- 172.21.15.114:0/87200709 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccac012b70 con 0x7fccbc0d06a0 2024-03-06T07:31:08.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.169+0000 7fccb3e00640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc9806b4d0 0x7fcc9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.169+0000 7fccb3e00640 1 -- 172.21.15.114:0/87200709 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fccac08a7a0 con 0x7fccbc0d06a0 2024-03-06T07:31:08.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.169+0000 7fccbb400640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc9806b4d0 0x7fcc9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.170+0000 7fccbb400640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc9806b4d0 0x7fcc9806d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcca4035000 tx=0x7fcca4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.174+0000 7fccb3e00640 1 -- 172.21.15.114:0/87200709 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccac059490 con 0x7fccbc0d06a0 2024-03-06T07:31:08.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.368+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fccbc08d430 con 0x7fccbc0d06a0 2024-03-06T07:31:08.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.931+0000 7fccb3e00640 1 -- 172.21.15.114:0/87200709 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) v1 ==== 181+0+0 (secure 0 0 0) 0x7fccac058d90 con 0x7fccbc0d06a0 2024-03-06T07:31:08.933 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc9806b4d0 msgr2=0x7fcc9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc9806b4d0 0x7fcc9806d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcca4035000 tx=0x7fcca4030000 comp rx=0 tx=0).stop 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d06a0 msgr2=0x7fccbc156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d06a0 0x7fccbc156190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fccac00d5f0 tx=0x7fccac00dab0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 shutdown_connections 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccbc15df30 0x7fccbc0ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccbc0d47b0 0x7fccbc0cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc9806b4d0 0x7fcc9806d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 --2- 172.21.15.114:0/87200709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccbc0d06a0 0x7fccbc156190 secure :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fccac00d5f0 tx=0x7fccac00dab0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.937+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 >> 172.21.15.114:0/87200709 conn(0x7fccbc0d3c90 msgr2=0x7fccbc15e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.938+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 shutdown_connections 2024-03-06T07:31:08.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:08.938+0000 7fccc1600640 1 -- 172.21.15.114:0/87200709 wait complete. 2024-03-06T07:31:08.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:31:09.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.161+0000 7fb153e00640 1 Processor -- start 2024-03-06T07:31:09.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.162+0000 7fb153e00640 1 -- start start 2024-03-06T07:31:09.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.163+0000 7fb153e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1541452f0 0x7fb154163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.163+0000 7fb153e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb154163580 0x7fb15406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.163+0000 7fb153e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1540ce6b0 con 0x7fb15406b500 2024-03-06T07:31:09.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.163+0000 7fb153e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1541456d0 con 0x7fb154163580 2024-03-06T07:31:09.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.163+0000 7fb153e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540cf2c0 con 0x7fb1541452f0 2024-03-06T07:31:09.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.163+0000 7fb153400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb15406b500 0x7fb154165e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55064/0 (socket says 172.21.15.114:55064) 2024-03-06T07:31:09.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.163+0000 7fb153400640 1 -- 172.21.15.114:0/716011817 learned_addr learned my addr 172.21.15.114:0/716011817 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:09.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb14be00640 1 --2- 172.21.15.114:0/716011817 >> v2:172.21.15.177:3300/0 conn(0x7fb154163580 0x7fb15406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147544422 0 0) 0x7fb1540ce6b0 con 0x7fb15406b500 2024-03-06T07:31:09.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb138003520 con 0x7fb15406b500 2024-03-06T07:31:09.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152a00640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1541452f0 0x7fb154163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1608734694 0 0) 0x7fb138003520 con 0x7fb15406b500 2024-03-06T07:31:09.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1541452f0 msgr2=0x7fb154163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1541452f0 0x7fb154163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 >> v2:172.21.15.177:3300/0 conn(0x7fb154163580 msgr2=0x7fb15406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 --2- 172.21.15.114:0/716011817 >> v2:172.21.15.177:3300/0 conn(0x7fb154163580 0x7fb15406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540ce990 con 0x7fb15406b500 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb14c0019f0 con 0x7fb15406b500 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb14be00640 1 --2- 172.21.15.114:0/716011817 >> v2:172.21.15.177:3300/0 conn(0x7fb154163580 0x7fb15406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 >> v1:172.21.15.114:6789/0 conn(0x7fb15406b500 legacy=0x7fb154165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1541452f0 0x7fb1380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb138003fe0 0x7fb138004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138004d40 0x7fb1380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541456d0 con 0x7fb138003fe0 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540cf2c0 con 0x7fb138004d40 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.164+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13800bb80 con 0x7fb1541452f0 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb14be00640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb138003fe0 0x7fb138004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb14be00640 1 -- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1541452f0 msgr2=0x7fb1380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb14be00640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1541452f0 0x7fb1380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb153400640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138004d40 0x7fb1380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb14be00640 1 -- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138004d40 msgr2=0x7fb1380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb14be00640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138004d40 0x7fb1380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb14be00640 1 -- 172.21.15.114:0/716011817 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138003520 con 0x7fb138003fe0 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.165+0000 7fb153400640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138004d40 0x7fb1380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.166+0000 7fb14be00640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb138003fe0 0x7fb138004800 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0098a0 tx=0x7fb138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb96aaac5306fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.166+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c00f040 con 0x7fb138003fe0 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c004230 con 0x7fb138003fe0 2024-03-06T07:31:09.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb152000640 1 -- 172.21.15.114:0/716011817 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c003610 con 0x7fb138003fe0 2024-03-06T07:31:09.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb153e00640 1 -- 172.21.15.114:0/716011817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb138003fe0 msgr2=0x7fb138004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb153e00640 1 --2- 172.21.15.114:0/716011817 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb138003fe0 0x7fb138004800 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0098a0 tx=0x7fb138005830 comp rx=0 tx=0).stop 2024-03-06T07:31:09.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb153400640 1 -- 172.21.15.114:0/716011817 reap_dead start 2024-03-06T07:31:09.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb153e00640 1 -- 172.21.15.114:0/716011817 shutdown_connections 2024-03-06T07:31:09.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb153e00640 1 -- 172.21.15.114:0/716011817 >> 172.21.15.114:0/716011817 conn(0x7fb1540b18a0 msgr2=0x7fb1540b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb153e00640 1 -- 172.21.15.114:0/716011817 shutdown_connections 2024-03-06T07:31:09.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.167+0000 7fb153e00640 1 -- 172.21.15.114:0/716011817 wait complete. 2024-03-06T07:31:09.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.169+0000 7fb153e00640 1 Processor -- start 2024-03-06T07:31:09.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.169+0000 7fb153e00640 1 -- start start 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.170+0000 7fb153e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb138003fe0 0x7fb1541ec6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.170+0000 7fb153e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb154163960 0x7fb15415f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.170+0000 7fb153e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb154164fa0 0x7fb154165420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.170+0000 7fb153e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15406b930 con 0x7fb154163960 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.170+0000 7fb153e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540cf750 con 0x7fb154164fa0 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.170+0000 7fb153e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540cef50 con 0x7fb138003fe0 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb154163960 0x7fb15415f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb154163960 0x7fb15415f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52618/0 (socket says 172.21.15.114:52618) 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 -- 172.21.15.114:0/1327812979 learned_addr learned my addr 172.21.15.114:0/1327812979 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb153400640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb154164fa0 0x7fb154165420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb152a00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb138003fe0 0x7fb1541ec6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 -- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb138003fe0 msgr2=0x7fb1541ec6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb138003fe0 0x7fb1541ec6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 -- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb154164fa0 msgr2=0x7fb154165420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb154164fa0 0x7fb154165420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 -- 172.21.15.114:0/1327812979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1541ecc20 con 0x7fb154163960 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb153400640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb154164fa0 0x7fb154165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb152a00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb138003fe0 0x7fb1541ec6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.171+0000 7fb14be00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb154163960 0x7fb15415f1e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb13c02bff0 tx=0x7fb13c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.172+0000 7fb14aa00640 1 -- 172.21.15.114:0/1327812979 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c047070 con 0x7fb154163960 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.172+0000 7fb14aa00640 1 -- 172.21.15.114:0/1327812979 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c042400 con 0x7fb154163960 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.172+0000 7fb14aa00640 1 -- 172.21.15.114:0/1327812979 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0427d0 con 0x7fb154163960 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.172+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb154163f00 con 0x7fb154163960 2024-03-06T07:31:09.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.172+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1541644c0 con 0x7fb154163960 2024-03-06T07:31:09.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.173+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15404f330 con 0x7fb154163960 2024-03-06T07:31:09.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.174+0000 7fb14aa00640 1 -- 172.21.15.114:0/1327812979 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb13c04a070 con 0x7fb154163960 2024-03-06T07:31:09.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.175+0000 7fb14aa00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb13006b4d0 0x7fb13006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.175+0000 7fb14aa00640 1 -- 172.21.15.114:0/1327812979 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fb13c0abb40 con 0x7fb154163960 2024-03-06T07:31:09.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.175+0000 7fb152a00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb13006b4d0 0x7fb13006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.176+0000 7fb152a00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb13006b4d0 0x7fb13006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb144006a90 tx=0x7fb144008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.179+0000 7fb14aa00640 1 -- 172.21.15.114:0/1327812979 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb13c033070 con 0x7fb154163960 2024-03-06T07:31:09.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.374+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb1540ce360 con 0x7fb154163960 2024-03-06T07:31:09.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.944+0000 7fb14aa00640 1 -- 172.21.15.114:0/1327812979 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb13c07a9f0 con 0x7fb154163960 2024-03-06T07:31:09.946 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.947+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb13006b4d0 msgr2=0x7fb13006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.947+0000 7fb153e00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb13006b4d0 0x7fb13006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb144006a90 tx=0x7fb144008040 comp rx=0 tx=0).stop 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.947+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb154163960 msgr2=0x7fb15415f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.947+0000 7fb153e00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb154163960 0x7fb15415f1e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb13c02bff0 tx=0x7fb13c004730 comp rx=0 tx=0).stop 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 shutdown_connections 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb154164fa0 0x7fb154165420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb13006b4d0 0x7fb13006d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb154163960 0x7fb15415f1e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 --2- 172.21.15.114:0/1327812979 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb138003fe0 0x7fb1541ec6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 >> 172.21.15.114:0/1327812979 conn(0x7fb1540b18a0 msgr2=0x7fb154161a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 shutdown_connections 2024-03-06T07:31:09.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:09.948+0000 7fb153e00640 1 -- 172.21.15.114:0/1327812979 wait complete. 2024-03-06T07:31:09.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:31:10.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.171+0000 7f7c15800640 1 Processor -- start 2024-03-06T07:31:10.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.172+0000 7f7c15800640 1 -- start start 2024-03-06T07:31:10.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.173+0000 7f7c15800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c100b3f70 0x7f7c10160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.173+0000 7f7c15800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7c10070ff0 0x7f7c100ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.173+0000 7f7c15800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c10071440 con 0x7f7c1015fc10 2024-03-06T07:31:10.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.173+0000 7f7c15800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c100b46c0 con 0x7f7c100b3f70 2024-03-06T07:31:10.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.173+0000 7f7c15800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c100b49a0 con 0x7f7c10070ff0 2024-03-06T07:31:10.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c07e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c100b3f70 0x7f7c10160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c07e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c100b3f70 0x7f7c10160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52632/0 (socket says 172.21.15.114:52632) 2024-03-06T07:31:10.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c07e00640 1 -- 172.21.15.114:0/624903884 learned_addr learned my addr 172.21.15.114:0/624903884 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:10.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c0ea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7c1015fc10 0x7f7c1015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55070/0 (socket says 172.21.15.114:55070) 2024-03-06T07:31:10.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c0f400640 1 --2- 172.21.15.114:0/624903884 >> v2:172.21.15.177:3300/0 conn(0x7f7c10070ff0 0x7f7c100ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c07e00640 1 -- 172.21.15.114:0/624903884 >> v2:172.21.15.177:3300/0 conn(0x7f7c10070ff0 msgr2=0x7f7c100ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c07e00640 1 --2- 172.21.15.114:0/624903884 >> v2:172.21.15.177:3300/0 conn(0x7f7c10070ff0 0x7f7c100ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c07e00640 1 -- 172.21.15.114:0/624903884 >> v1:172.21.15.114:6789/0 conn(0x7f7c1015fc10 legacy=0x7f7c1015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c07e00640 1 -- 172.21.15.114:0/624903884 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c10072c90 con 0x7f7c100b3f70 2024-03-06T07:31:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c0e000640 1 -- 172.21.15.114:0/624903884 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536701221 0 0) 0x7f7c10071440 con 0x7f7c1015fc10 2024-03-06T07:31:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.174+0000 7f7c0f400640 1 --2- 172.21.15.114:0/624903884 >> v2:172.21.15.177:3300/0 conn(0x7f7c10070ff0 0x7f7c100ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.175+0000 7f7c07e00640 1 --2- 172.21.15.114:0/624903884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c100b3f70 0x7f7c10160700 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7c100cf6a0 tx=0x7f7bf802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc49c66cade4d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.175+0000 7f7c0e000640 1 -- 172.21.15.114:0/624903884 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8034070 con 0x7f7c100b3f70 2024-03-06T07:31:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c0e000640 1 -- 172.21.15.114:0/624903884 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf8030860 con 0x7f7c100b3f70 2024-03-06T07:31:10.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c0e000640 1 -- 172.21.15.114:0/624903884 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8030c30 con 0x7f7c100b3f70 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 -- 172.21.15.114:0/624903884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c100b3f70 msgr2=0x7f7c10160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 --2- 172.21.15.114:0/624903884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c100b3f70 0x7f7c10160700 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7c100cf6a0 tx=0x7f7bf802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 -- 172.21.15.114:0/624903884 shutdown_connections 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 --2- 172.21.15.114:0/624903884 >> v2:172.21.15.177:3300/0 conn(0x7f7c10070ff0 0x7f7c100ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 --2- 172.21.15.114:0/624903884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c100b3f70 0x7f7c10160700 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 -- 172.21.15.114:0/624903884 >> 172.21.15.114:0/624903884 conn(0x7f7c100b1940 msgr2=0x7f7c100b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 -- 172.21.15.114:0/624903884 shutdown_connections 2024-03-06T07:31:10.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.176+0000 7f7c15800640 1 -- 172.21.15.114:0/624903884 wait complete. 2024-03-06T07:31:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.178+0000 7f7c15800640 1 Processor -- start 2024-03-06T07:31:10.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.178+0000 7f7c15800640 1 -- start start 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c15800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c10070ff0 0x7f7c101551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c15800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c100b3f70 0x7f7c1014e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c15800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c1015fc10 0x7f7c101f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c15800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c10072a30 con 0x7f7c10070ff0 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c15800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100728b0 con 0x7f7c100b3f70 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c15800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1015c2e0 con 0x7f7c1015fc10 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c0ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c10070ff0 0x7f7c101551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c0ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c10070ff0 0x7f7c101551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52634/0 (socket says 172.21.15.114:52634) 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.179+0000 7f7c0ea00640 1 -- 172.21.15.114:0/3983887869 learned_addr learned my addr 172.21.15.114:0/3983887869 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c07e00640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c100b3f70 0x7f7c1014e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c0ea00640 1 -- 172.21.15.114:0/3983887869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c1015fc10 msgr2=0x7f7c101f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c0ea00640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c1015fc10 0x7f7c101f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c0ea00640 1 -- 172.21.15.114:0/3983887869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c100b3f70 msgr2=0x7f7c1014e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c0ea00640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c100b3f70 0x7f7c1014e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c0ea00640 1 -- 172.21.15.114:0/3983887869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1009b4a0 con 0x7f7c10070ff0 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c0ea00640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c10070ff0 0x7f7c101551e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c460 tx=0x7f7c0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c06a00640 1 -- 172.21.15.114:0/3983887869 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00018070 con 0x7f7c10070ff0 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c06a00640 1 -- 172.21.15.114:0/3983887869 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00007500 con 0x7f7c10070ff0 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1009b780 con 0x7f7c10070ff0 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c06a00640 1 -- 172.21.15.114:0/3983887869 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00004a20 con 0x7f7c10070ff0 2024-03-06T07:31:10.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.180+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1009bd40 con 0x7f7c10070ff0 2024-03-06T07:31:10.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.182+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1004f330 con 0x7f7c10070ff0 2024-03-06T07:31:10.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.182+0000 7f7c06a00640 1 -- 172.21.15.114:0/3983887869 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c0001b070 con 0x7f7c10070ff0 2024-03-06T07:31:10.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.183+0000 7f7c06a00640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bec06b4d0 0x7f7bec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.183+0000 7f7c06a00640 1 -- 172.21.15.114:0/3983887869 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7c000891d0 con 0x7f7c10070ff0 2024-03-06T07:31:10.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.183+0000 7f7c07e00640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bec06b4d0 0x7f7bec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.184+0000 7f7c07e00640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bec06b4d0 0x7f7bec06d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8036000 tx=0x7f7bf803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.188+0000 7f7c06a00640 1 -- 172.21.15.114:0/3983887869 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c00058350 con 0x7f7c10070ff0 2024-03-06T07:31:10.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.368+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7c1006ef30 con 0x7f7c10070ff0 2024-03-06T07:31:10.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.374+0000 7f7c06a00640 1 -- 172.21.15.114:0/3983887869 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7c00087e20 con 0x7f7c10070ff0 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bec06b4d0 msgr2=0x7f7bec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bec06b4d0 0x7f7bec06d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8036000 tx=0x7f7bf803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c10070ff0 msgr2=0x7f7c101551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c10070ff0 0x7f7c101551e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000c460 tx=0x7f7c0000c920 comp rx=0 tx=0).stop 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 shutdown_connections 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7c1015fc10 0x7f7c101f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c100b3f70 0x7f7c1014e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bec06b4d0 0x7f7bec06d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 --2- 172.21.15.114:0/3983887869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c10070ff0 0x7f7c101551e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 >> 172.21.15.114:0/3983887869 conn(0x7f7c100b1940 msgr2=0x7f7c100cee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 shutdown_connections 2024-03-06T07:31:10.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.377+0000 7f7c15800640 1 -- 172.21.15.114:0/3983887869 wait complete. 2024-03-06T07:31:10.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:31:10.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.598+0000 7f2311800640 1 Processor -- start 2024-03-06T07:31:10.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.598+0000 7f2311800640 1 -- start start 2024-03-06T07:31:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.599+0000 7f2311800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c06da40 0x7f230c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.599+0000 7f2311800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c15c1f0 con 0x7f230c15be10 2024-03-06T07:31:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.599+0000 7f2311800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c06cac0 con 0x7f230c0aef20 2024-03-06T07:31:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.599+0000 7f2311800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0cd200 con 0x7f230c06da40 2024-03-06T07:31:10.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f230c15be10 0x7f230c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55076/0 (socket says 172.21.15.114:55076) 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230be00640 1 -- 172.21.15.114:0/1836021385 learned_addr learned my addr 172.21.15.114:0/1836021385 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507858641 0 0) 0x7f230c15c1f0 con 0x7f230c15be10 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0003520 con 0x7f230c15be10 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230b400640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c06da40 0x7f230c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773229785 0 0) 0x7f22f0003520 con 0x7f230c15be10 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c06da40 msgr2=0x7f230c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c06da40 0x7f230c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 >> v1:172.21.15.177:6789/0 conn(0x7f230c0aef20 legacy=0x7f230c071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c06cda0 con 0x7f230c15be10 2024-03-06T07:31:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f22fc001670 con 0x7f230c15be10 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.600+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 >> v1:172.21.15.114:6789/0 conn(0x7f230c15be10 legacy=0x7f230c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230aa00640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c06da40 0x7f22f00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230aa00640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f22f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230aa00640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f0004d40 0x7f22f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0cd200 con 0x7f22f0004060 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f000bc00 con 0x7f22f0004d40 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f000be00 con 0x7f230c06da40 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230b400640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f22f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230b400640 1 -- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c06da40 msgr2=0x7f22f00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230b400640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c06da40 0x7f22f00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230b400640 1 -- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f0004d40 msgr2=0x7f22f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f2310e00640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f0004d40 0x7f22f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230b400640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f0004d40 0x7f22f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f230b400640 1 -- 172.21.15.114:0/1836021385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0003520 con 0x7f22f0004060 2024-03-06T07:31:10.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.601+0000 7f2310e00640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22f0004d40 0x7f22f00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.602+0000 7f230b400640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f22f0004800 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f22f800ba30 tx=0x7f22f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b6d5a58a33c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.602+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f8036020 con 0x7f22f0004060 2024-03-06T07:31:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.602+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8032680 con 0x7f22f0004060 2024-03-06T07:31:10.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.602+0000 7f230aa00640 1 -- 172.21.15.114:0/1836021385 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f800b600 con 0x7f22f0004060 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.603+0000 7f2311800640 1 -- 172.21.15.114:0/1836021385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 msgr2=0x7f22f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.603+0000 7f2311800640 1 --2- 172.21.15.114:0/1836021385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f22f0004800 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f22f800ba30 tx=0x7f22f0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.603+0000 7f2310e00640 1 -- 172.21.15.114:0/1836021385 reap_dead start 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.603+0000 7f2311800640 1 -- 172.21.15.114:0/1836021385 shutdown_connections 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.603+0000 7f2311800640 1 -- 172.21.15.114:0/1836021385 >> 172.21.15.114:0/1836021385 conn(0x7f230c06afc0 msgr2=0x7f230c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.603+0000 7f2311800640 1 -- 172.21.15.114:0/1836021385 shutdown_connections 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.603+0000 7f2311800640 1 -- 172.21.15.114:0/1836021385 wait complete. 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.605+0000 7f2311800640 1 Processor -- start 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.605+0000 7f2311800640 1 -- start start 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f2311800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f230c158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f2311800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c157910 0x7f230c13f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f2311800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f230c15eb20 0x7f230c0b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f2311800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0cda90 con 0x7f22f0004060 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f2311800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c06d080 con 0x7f230c15eb20 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f2311800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0ccc20 con 0x7f230c157910 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f230c158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f230c158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52648/0 (socket says 172.21.15.114:52648) 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 -- 172.21.15.114:0/3942385522 learned_addr learned my addr 172.21.15.114:0/3942385522 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 -- 172.21.15.114:0/3942385522 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c157910 msgr2=0x7f230c13f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c157910 0x7f230c13f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 -- 172.21.15.114:0/3942385522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f230c15eb20 msgr2=0x7f230c0b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f230c15eb20 0x7f230c0b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.606+0000 7f230be00640 1 -- 172.21.15.114:0/3942385522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c13f820 con 0x7f22f0004060 2024-03-06T07:31:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.607+0000 7f230be00640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f230c158f20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c750 tx=0x7f22fc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.607+0000 7f2303e00640 1 -- 172.21.15.114:0/3942385522 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc018070 con 0x7f22f0004060 2024-03-06T07:31:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.607+0000 7f2303e00640 1 -- 172.21.15.114:0/3942385522 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc0072e0 con 0x7f22f0004060 2024-03-06T07:31:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.607+0000 7f2303e00640 1 -- 172.21.15.114:0/3942385522 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0047e0 con 0x7f22f0004060 2024-03-06T07:31:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.607+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c157e80 con 0x7f22f0004060 2024-03-06T07:31:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.607+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c158440 con 0x7f22f0004060 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.608+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c04f330 con 0x7f22f0004060 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.609+0000 7f2303e00640 1 -- 172.21.15.114:0/3942385522 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22fc004980 con 0x7f22f0004060 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.610+0000 7f2303e00640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f22ec06b4d0 0x7f22ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.610+0000 7f2303e00640 1 -- 172.21.15.114:0/3942385522 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f22fc089450 con 0x7f22f0004060 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.610+0000 7f230b400640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f22ec06b4d0 0x7f22ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.611+0000 7f230b400640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f22ec06b4d0 0x7f22ec06d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f22f800ba00 tx=0x7f22f8032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.614+0000 7f2303e00640 1 -- 172.21.15.114:0/3942385522 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f230c13f820 con 0x7f22f0004060 2024-03-06T07:31:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.792+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f230c0afb60 con 0x7f22f0004060 2024-03-06T07:31:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.799+0000 7f2303e00640 1 -- 172.21.15.114:0/3942385522 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v20) v1 ==== 101+0+0 (secure 0 0 0) 0x7f22fc004c70 con 0x7f22f0004060 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f22ec06b4d0 msgr2=0x7f22ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f22ec06b4d0 0x7f22ec06d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f22f800ba00 tx=0x7f22f8032000 comp rx=0 tx=0).stop 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 msgr2=0x7f230c158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f230c158f20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c750 tx=0x7f22fc00cc10 comp rx=0 tx=0).stop 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 shutdown_connections 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f230c15eb20 0x7f230c0b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f230c157910 0x7f230c13f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f22ec06b4d0 0x7f22ec06d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 --2- 172.21.15.114:0/3942385522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f0004060 0x7f230c158f20 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.801+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 >> 172.21.15.114:0/3942385522 conn(0x7f230c06afc0 msgr2=0x7f230c071ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.802+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 shutdown_connections 2024-03-06T07:31:10.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:10.802+0000 7f2311800640 1 -- 172.21.15.114:0/3942385522 wait complete. 2024-03-06T07:31:10.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:31:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.028+0000 7f00f9200640 1 Processor -- start 2024-03-06T07:31:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.028+0000 7f00f9200640 1 -- start start 2024-03-06T07:31:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.029+0000 7f00f9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 0x7f00f415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.029+0000 7f00f9200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f00f406d960 0x7f00f40b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.029+0000 7f00f9200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f40b3560 con 0x7f00f406e2f0 2024-03-06T07:31:11.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.029+0000 7f00f9200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f415f700 con 0x7f00f406d960 2024-03-06T07:31:11.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.029+0000 7f00f9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f406d4d0 con 0x7f00f40b2c40 2024-03-06T07:31:11.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.030+0000 7f00f2800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f00f406e2f0 0x7f00f4161d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55092/0 (socket says 172.21.15.114:55092) 2024-03-06T07:31:11.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.030+0000 7f00f2800640 1 -- 172.21.15.114:0/790094019 learned_addr learned my addr 172.21.15.114:0/790094019 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.030+0000 7f00e9400640 1 --2- 172.21.15.114:0/790094019 >> v2:172.21.15.177:3300/0 conn(0x7f00f406d960 0x7f00f40b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.030+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273034601 0 0) 0x7f00f40b3560 con 0x7f00f406e2f0 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.030+0000 7f00f1e00640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 0x7f00f415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.030+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d8003520 con 0x7f00f406e2f0 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3371121544 0 0) 0x7f00d8003520 con 0x7f00f406e2f0 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 msgr2=0x7f00f415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 0x7f00f415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 >> v2:172.21.15.177:3300/0 conn(0x7f00f406d960 msgr2=0x7f00f40b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 --2- 172.21.15.114:0/790094019 >> v2:172.21.15.177:3300/0 conn(0x7f00f406d960 0x7f00f40b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f406cba0 con 0x7f00f406e2f0 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f00ec0019f0 con 0x7f00f406e2f0 2024-03-06T07:31:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 >> v1:172.21.15.114:6789/0 conn(0x7f00f406e2f0 legacy=0x7f00f4161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00d80039b0 0x7f00d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 0x7f00d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00d8004d40 0x7f00d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f406d4d0 con 0x7f00d80039b0 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d800ba70 con 0x7f00d8004d40 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d800bbf0 con 0x7f00f40b2c40 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1e00640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00d80039b0 0x7f00d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00e9400640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 0x7f00d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1e00640 1 -- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 msgr2=0x7f00d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1e00640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00f40b2c40 0x7f00d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1e00640 1 -- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00d8004d40 msgr2=0x7f00d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1e00640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00d8004d40 0x7f00d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.031+0000 7f00f1e00640 1 -- 172.21.15.114:0/790094019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d8003520 con 0x7f00d80039b0 2024-03-06T07:31:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.032+0000 7f00f1e00640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00d80039b0 0x7f00d8003dd0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f00e400ba60 tx=0x7f00d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394acf4dd7b5949d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.033+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4036020 con 0x7f00d80039b0 2024-03-06T07:31:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.033+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e4032630 con 0x7f00d80039b0 2024-03-06T07:31:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.033+0000 7f00f1400640 1 -- 172.21.15.114:0/790094019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e400b600 con 0x7f00d80039b0 2024-03-06T07:31:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.033+0000 7f00f9200640 1 -- 172.21.15.114:0/790094019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00d80039b0 msgr2=0x7f00d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.033+0000 7f00f9200640 1 --2- 172.21.15.114:0/790094019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00d80039b0 0x7f00d8003dd0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f00e400ba60 tx=0x7f00d8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.033+0000 7f00f2800640 1 -- 172.21.15.114:0/790094019 reap_dead start 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.034+0000 7f00f9200640 1 -- 172.21.15.114:0/790094019 shutdown_connections 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.034+0000 7f00f9200640 1 -- 172.21.15.114:0/790094019 >> 172.21.15.114:0/790094019 conn(0x7f00f40b1940 msgr2=0x7f00f40b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.034+0000 7f00f9200640 1 -- 172.21.15.114:0/790094019 shutdown_connections 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.034+0000 7f00f9200640 1 -- 172.21.15.114:0/790094019 wait complete. 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.035+0000 7f00f9200640 1 Processor -- start 2024-03-06T07:31:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.035+0000 7f00f9200640 1 -- start start 2024-03-06T07:31:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.036+0000 7f00f9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00d80039b0 0x7f00f40ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.036+0000 7f00f9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f4151630 0x7f00f4150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.036+0000 7f00f9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00f40ad8a0 0x7f00f4156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.036+0000 7f00f9200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f406e750 con 0x7f00f40ad8a0 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.036+0000 7f00f9200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f406d160 con 0x7f00f4151630 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.036+0000 7f00f9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f406e8d0 con 0x7f00d80039b0 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00f40ad8a0 0x7f00f4156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00f40ad8a0 0x7f00f4156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52678/0 (socket says 172.21.15.114:52678) 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 -- 172.21.15.114:0/4226236052 learned_addr learned my addr 172.21.15.114:0/4226236052 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00e9400640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f4151630 0x7f00f4150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f1e00640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00d80039b0 0x7f00f40ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 -- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00d80039b0 msgr2=0x7f00f40ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00d80039b0 0x7f00f40ae250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 -- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f4151630 msgr2=0x7f00f4150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f4151630 0x7f00f4150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 -- 172.21.15.114:0/4226236052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f40ae790 con 0x7f00f40ad8a0 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00e9400640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f4151630 0x7f00f4150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00f2800640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00f40ad8a0 0x7f00f4156be0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f00f406c6e0 tx=0x7f00ec00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.037+0000 7f00eaa00640 1 -- 172.21.15.114:0/4226236052 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec00a610 con 0x7f00f40ad8a0 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.038+0000 7f00eaa00640 1 -- 172.21.15.114:0/4226236052 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec00a7b0 con 0x7f00f40ad8a0 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.038+0000 7f00eaa00640 1 -- 172.21.15.114:0/4226236052 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec00aba0 con 0x7f00f40ad8a0 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.038+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f4151400 con 0x7f00f40ad8a0 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.038+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f40aed10 con 0x7f00f40ad8a0 2024-03-06T07:31:11.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.039+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f404f330 con 0x7f00f40ad8a0 2024-03-06T07:31:11.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.040+0000 7f00eaa00640 1 -- 172.21.15.114:0/4226236052 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00ec011070 con 0x7f00f40ad8a0 2024-03-06T07:31:11.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.040+0000 7f00eaa00640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f00d006b500 0x7f00d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.040+0000 7f00eaa00640 1 -- 172.21.15.114:0/4226236052 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f00ec089ae0 con 0x7f00f40ad8a0 2024-03-06T07:31:11.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.041+0000 7f00f1e00640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f00d006b500 0x7f00d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.041+0000 7f00f1e00640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f00d006b500 0x7f00d006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f00e400ba30 tx=0x7f00e4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.045+0000 7f00eaa00640 1 -- 172.21.15.114:0/4226236052 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00ec058c50 con 0x7f00f40ad8a0 2024-03-06T07:31:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.229+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f00f40b2530 con 0x7f00f40ad8a0 2024-03-06T07:31:11.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.233+0000 7f00eaa00640 1 -- 172.21.15.114:0/4226236052 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v21) v1 ==== 108+0+0 (secure 0 0 0) 0x7f00ec058550 con 0x7f00f40ad8a0 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f00d006b500 msgr2=0x7f00d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f00d006b500 0x7f00d006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f00e400ba30 tx=0x7f00e4030000 comp rx=0 tx=0).stop 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00f40ad8a0 msgr2=0x7f00f4156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00f40ad8a0 0x7f00f4156be0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f00f406c6e0 tx=0x7f00ec00ccd0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 shutdown_connections 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f00d006b500 0x7f00d006d9c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00f40ad8a0 0x7f00f4156be0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00f4151630 0x7f00f4150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 --2- 172.21.15.114:0/4226236052 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00d80039b0 0x7f00f40ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 >> 172.21.15.114:0/4226236052 conn(0x7f00f40b1940 msgr2=0x7f00f415d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 shutdown_connections 2024-03-06T07:31:11.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.237+0000 7f00f9200640 1 -- 172.21.15.114:0/4226236052 wait complete. 2024-03-06T07:31:11.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:31:11.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.455+0000 7f5901c00640 1 Processor -- start 2024-03-06T07:31:11.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.455+0000 7f5901c00640 1 -- start start 2024-03-06T07:31:11.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f5901c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58fc144c00 0x7f58fc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f5901c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc06ee70 con 0x7f58fc15fe60 2024-03-06T07:31:11.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f5901c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc06cef0 con 0x7f58fc15f9d0 2024-03-06T07:31:11.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f5901c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58fc06d1d0 con 0x7f58fc144c00 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f58faa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58fc144c00 0x7f58fc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f58faa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58fc144c00 0x7f58fc06e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52690/0 (socket says 172.21.15.114:52690) 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f58faa00640 1 -- 172.21.15.114:0/1003779892 learned_addr learned my addr 172.21.15.114:0/1003779892 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.457+0000 7f58fb400640 1 --1- 172.21.15.114:0/1003779892 >> v1:172.21.15.114:6789/0 conn(0x7f58fc15fe60 0x7f58fc0ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55108/0 (socket says 172.21.15.114:55108) 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.458+0000 7f58faa00640 1 -- 172.21.15.114:0/1003779892 >> v1:172.21.15.177:6789/0 conn(0x7f58fc15f9d0 legacy=0x7f58fc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.458+0000 7f58faa00640 1 -- 172.21.15.114:0/1003779892 >> v1:172.21.15.114:6789/0 conn(0x7f58fc15fe60 legacy=0x7f58fc0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.458+0000 7f58faa00640 1 -- 172.21.15.114:0/1003779892 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc06d4b0 con 0x7f58fc144c00 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.458+0000 7f58fa000640 1 -- 172.21.15.114:0/1003779892 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506518688 0 0) 0x7f58f4001490 con 0x7f58fc15fe60 2024-03-06T07:31:11.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.459+0000 7f58faa00640 1 --2- 172.21.15.114:0/1003779892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc144c00 0x7f58fc06e930 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009870 tx=0x7f58ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77cf478d3389da8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.459+0000 7f58fa000640 1 -- 172.21.15.114:0/1003779892 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec034070 con 0x7f58fc144c00 2024-03-06T07:31:11.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.459+0000 7f58fa000640 1 -- 172.21.15.114:0/1003779892 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ec030860 con 0x7f58fc144c00 2024-03-06T07:31:11.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.459+0000 7f58fa000640 1 -- 172.21.15.114:0/1003779892 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec030c30 con 0x7f58fc144c00 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.460+0000 7f5901c00640 1 -- 172.21.15.114:0/1003779892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc144c00 msgr2=0x7f58fc06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.460+0000 7f5901c00640 1 --2- 172.21.15.114:0/1003779892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc144c00 0x7f58fc06e930 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009870 tx=0x7f58ec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.460+0000 7f5901c00640 1 -- 172.21.15.114:0/1003779892 shutdown_connections 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.460+0000 7f5901c00640 1 --2- 172.21.15.114:0/1003779892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc144c00 0x7f58fc06e930 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.460+0000 7f5901c00640 1 -- 172.21.15.114:0/1003779892 >> 172.21.15.114:0/1003779892 conn(0x7f58fc0b1730 msgr2=0x7f58fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.460+0000 7f5901c00640 1 -- 172.21.15.114:0/1003779892 shutdown_connections 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.460+0000 7f5901c00640 1 -- 172.21.15.114:0/1003779892 wait complete. 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.462+0000 7f5901c00640 1 Processor -- start 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.462+0000 7f5901c00640 1 -- start start 2024-03-06T07:31:11.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.462+0000 7f5901c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc144c00 0x7f58fc13c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f5901c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc15f9d0 0x7f58fc15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f5901c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc15fe60 0x7f58fc13a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f5901c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc160730 con 0x7f58fc15f9d0 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f5901c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc160270 con 0x7f58fc15fe60 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f5901c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0ceec0 con 0x7f58fc144c00 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc15f9d0 0x7f58fc15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc15f9d0 0x7f58fc15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52692/0 (socket says 172.21.15.114:52692) 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 -- 172.21.15.114:0/850182588 learned_addr learned my addr 172.21.15.114:0/850182588 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58faa00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc144c00 0x7f58fc13c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58fb400640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc15fe60 0x7f58fc13a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 -- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc144c00 msgr2=0x7f58fc13c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc144c00 0x7f58fc13c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 -- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc15fe60 msgr2=0x7f58fc13a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc15fe60 0x7f58fc13a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58f2000640 1 -- 172.21.15.114:0/850182588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc13a920 con 0x7f58fc15f9d0 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.463+0000 7f58faa00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc144c00 0x7f58fc13c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.464+0000 7f58f2000640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc15f9d0 0x7f58fc15b290 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c430 tx=0x7f58e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.464+0000 7f58f3400640 1 -- 172.21.15.114:0/850182588 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4011790 con 0x7f58fc15f9d0 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.464+0000 7f58f3400640 1 -- 172.21.15.114:0/850182588 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e4017070 con 0x7f58fc15f9d0 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.464+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc13abd0 con 0x7f58fc15f9d0 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.464+0000 7f58f3400640 1 -- 172.21.15.114:0/850182588 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4011bb0 con 0x7f58fc15f9d0 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.464+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc13e880 con 0x7f58fc15f9d0 2024-03-06T07:31:11.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.465+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc06cb50 con 0x7f58fc15f9d0 2024-03-06T07:31:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.466+0000 7f58f3400640 1 -- 172.21.15.114:0/850182588 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58e401c460 con 0x7f58fc15f9d0 2024-03-06T07:31:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.467+0000 7f58f3400640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58d806b4d0 0x7f58d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.467+0000 7f58f3400640 1 -- 172.21.15.114:0/850182588 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f58e4089e80 con 0x7f58fc15f9d0 2024-03-06T07:31:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.467+0000 7f58faa00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58d806b4d0 0x7f58d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.468+0000 7f58faa00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58d806b4d0 0x7f58d806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f58ec036000 tx=0x7f58ec03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.471+0000 7f58f3400640 1 -- 172.21.15.114:0/850182588 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58fc06cb50 con 0x7f58fc15f9d0 2024-03-06T07:31:11.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.718+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f58fc04f330 con 0x7f58fc15f9d0 2024-03-06T07:31:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.949+0000 7f58f3400640 1 -- 172.21.15.114:0/850182588 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v189) v1 ==== 144+0+0 (secure 0 0 0) 0x7f58e4058f60 con 0x7f58fc15f9d0 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58d806b4d0 msgr2=0x7f58d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58d806b4d0 0x7f58d806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f58ec036000 tx=0x7f58ec03b040 comp rx=0 tx=0).stop 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc15f9d0 msgr2=0x7f58fc15b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc15f9d0 0x7f58fc15b290 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c430 tx=0x7f58e400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 shutdown_connections 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58fc15fe60 0x7f58fc13a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58d806b4d0 0x7f58d806d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58fc15f9d0 0x7f58fc15b290 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 --2- 172.21.15.114:0/850182588 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58fc144c00 0x7f58fc13c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.952+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 >> 172.21.15.114:0/850182588 conn(0x7f58fc0b1730 msgr2=0x7f58fc071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.953+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 shutdown_connections 2024-03-06T07:31:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:11.953+0000 7f5901c00640 1 -- 172.21.15.114:0/850182588 wait complete. 2024-03-06T07:31:11.975 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:31:11.975 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:31:11.975 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:31:11.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:12.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.169+0000 7f635dc00640 1 Processor -- start 2024-03-06T07:31:12.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.169+0000 7f635dc00640 1 -- start start 2024-03-06T07:31:12.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.170+0000 7f635dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63580cf430 0x7f6358070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f635dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635806d260 0x7f63580b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f635dc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63580717c0 con 0x7f6358071260 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f635dc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6358071aa0 con 0x7f63580cf430 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f635dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6358071d80 con 0x7f635806d260 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6356a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63580cf430 0x7f6358070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6356000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6358071260 0x7f635806cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55116/0 (socket says 172.21.15.114:55116) 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6356000640 1 -- 172.21.15.114:0/1853589399 learned_addr learned my addr 172.21.15.114:0/1853589399 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6357400640 1 --2- 172.21.15.114:0/1853589399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635806d260 0x7f63580b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6356a00640 1 -- 172.21.15.114:0/1853589399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635806d260 msgr2=0x7f63580b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6356a00640 1 --2- 172.21.15.114:0/1853589399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635806d260 0x7f63580b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6356a00640 1 -- 172.21.15.114:0/1853589399 >> v1:172.21.15.114:6789/0 conn(0x7f6358071260 legacy=0x7f635806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6356a00640 1 -- 172.21.15.114:0/1853589399 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358072060 con 0x7f63580cf430 2024-03-06T07:31:12.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6357400640 1 --2- 172.21.15.114:0/1853589399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635806d260 0x7f63580b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.171+0000 7f6355600640 1 -- 172.21.15.114:0/1853589399 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890070851 0 0) 0x7f63580717c0 con 0x7f6358071260 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f6356a00640 1 --2- 172.21.15.114:0/1853589399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63580cf430 0x7f6358070d20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f63500097a0 tx=0x7f635002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9143aa519ded1da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f6355600640 1 -- 172.21.15.114:0/1853589399 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350035020 con 0x7f63580cf430 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f6355600640 1 -- 172.21.15.114:0/1853589399 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6350031830 con 0x7f63580cf430 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f6355600640 1 -- 172.21.15.114:0/1853589399 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350031c00 con 0x7f63580cf430 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f635dc00640 1 -- 172.21.15.114:0/1853589399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63580cf430 msgr2=0x7f6358070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f635dc00640 1 --2- 172.21.15.114:0/1853589399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63580cf430 0x7f6358070d20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f63500097a0 tx=0x7f635002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f635dc00640 1 -- 172.21.15.114:0/1853589399 shutdown_connections 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f635dc00640 1 --2- 172.21.15.114:0/1853589399 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635806d260 0x7f63580b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f635dc00640 1 --2- 172.21.15.114:0/1853589399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63580cf430 0x7f6358070d20 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f635dc00640 1 -- 172.21.15.114:0/1853589399 >> 172.21.15.114:0/1853589399 conn(0x7f63580b1730 msgr2=0x7f63580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.173+0000 7f635dc00640 1 -- 172.21.15.114:0/1853589399 shutdown_connections 2024-03-06T07:31:12.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.174+0000 7f635dc00640 1 -- 172.21.15.114:0/1853589399 wait complete. 2024-03-06T07:31:12.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.175+0000 7f635dc00640 1 Processor -- start 2024-03-06T07:31:12.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.175+0000 7f635dc00640 1 -- start start 2024-03-06T07:31:12.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.176+0000 7f635dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635806d260 0x7f6358152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.176+0000 7f635dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6358071260 0x7f635814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.176+0000 7f635dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63580cf430 0x7f6358157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.176+0000 7f635dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6358162cf0 con 0x7f635806d260 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.176+0000 7f635dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6358162b70 con 0x7f63580cf430 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.176+0000 7f635dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63580b3250 con 0x7f6358071260 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635806d260 0x7f6358152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635806d260 0x7f6358152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52706/0 (socket says 172.21.15.114:52706) 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 -- 172.21.15.114:0/1679068154 learned_addr learned my addr 172.21.15.114:0/1679068154 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356000640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6358071260 0x7f635814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6357400640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63580cf430 0x7f6358157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 -- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6358071260 msgr2=0x7f635814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6358071260 0x7f635814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 -- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63580cf430 msgr2=0x7f6358157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63580cf430 0x7f6358157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 -- 172.21.15.114:0/1679068154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635815dc50 con 0x7f635806d260 2024-03-06T07:31:12.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6357400640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63580cf430 0x7f6358157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.177+0000 7f6356a00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635806d260 0x7f6358152590 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f63500098c0 tx=0x7f63500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.178+0000 7f634ea00640 1 -- 172.21.15.114:0/1679068154 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350040070 con 0x7f635806d260 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.178+0000 7f634ea00640 1 -- 172.21.15.114:0/1679068154 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6350039a00 con 0x7f635806d260 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.178+0000 7f634ea00640 1 -- 172.21.15.114:0/1679068154 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6350035040 con 0x7f635806d260 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.178+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635815df30 con 0x7f635806d260 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.178+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6358153540 con 0x7f635806d260 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.179+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63580cef70 con 0x7f635806d260 2024-03-06T07:31:12.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.180+0000 7f634ea00640 1 -- 172.21.15.114:0/1679068154 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6350038020 con 0x7f635806d260 2024-03-06T07:31:12.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.181+0000 7f634ea00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f633406b4d0 0x7f633406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.181+0000 7f634ea00640 1 -- 172.21.15.114:0/1679068154 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f63500ab970 con 0x7f635806d260 2024-03-06T07:31:12.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.181+0000 7f6356000640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f633406b4d0 0x7f633406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.182+0000 7f6356000640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f633406b4d0 0x7f633406d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f63581fb8e0 tx=0x7f6340007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.185+0000 7f634ea00640 1 -- 172.21.15.114:0/1679068154 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f635007aae0 con 0x7f635806d260 2024-03-06T07:31:12.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.383+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63580b4a40 con 0x7f635806d260 2024-03-06T07:31:12.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.385+0000 7f634ea00640 1 -- 172.21.15.114:0/1679068154 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+13070 (secure 0 0 0) 0x7f635007a3e0 con 0x7f635806d260 2024-03-06T07:31:12.388 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:12.388 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":58,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:09.932841+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/4290788305":"2024-03-07T07:31:06.913957+0000","172.21.15.114:6835/4290788305":"2024-03-07T07:31:06.913957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f633406b4d0 msgr2=0x7f633406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f633406b4d0 0x7f633406d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f63581fb8e0 tx=0x7f6340007b50 comp rx=0 tx=0).stop 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635806d260 msgr2=0x7f6358152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635806d260 0x7f6358152590 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f63500098c0 tx=0x7f63500046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 shutdown_connections 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63580cf430 0x7f6358157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6358071260 0x7f635814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f633406b4d0 0x7f633406d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 --2- 172.21.15.114:0/1679068154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635806d260 0x7f6358152590 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 >> 172.21.15.114:0/1679068154 conn(0x7f63580b1730 msgr2=0x7f63580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 shutdown_connections 2024-03-06T07:31:12.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.388+0000 7f635dc00640 1 -- 172.21.15.114:0/1679068154 wait complete. 2024-03-06T07:31:12.410 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:12.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:12.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.623+0000 7f411ba00640 1 Processor -- start 2024-03-06T07:31:12.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.624+0000 7f411ba00640 1 -- start start 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f411ba00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f411406fef0 0x7f411406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f411ba00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f411415e3e0 0x7f41140d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f411ba00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140d1260 con 0x7f411415e000 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f411ba00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41140d1020 con 0x7f411415e3e0 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f411ba00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411406c0d0 con 0x7f411406fef0 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f4119000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f411415e000 0x7f411406f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55126/0 (socket says 172.21.15.114:55126) 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f4119000640 1 -- 172.21.15.114:0/689256129 learned_addr learned my addr 172.21.15.114:0/689256129 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.625+0000 7f4119a00640 1 --2- 172.21.15.114:0/689256129 >> v2:172.21.15.114:3300/0 conn(0x7f411415e3e0 0x7f41140d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.626+0000 7f4113e00640 1 --2- 172.21.15.114:0/689256129 >> v2:172.21.15.177:3300/0 conn(0x7f411406fef0 0x7f411406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.626+0000 7f4119a00640 1 -- 172.21.15.114:0/689256129 >> v2:172.21.15.177:3300/0 conn(0x7f411406fef0 msgr2=0x7f411406f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.626+0000 7f4119a00640 1 --2- 172.21.15.114:0/689256129 >> v2:172.21.15.177:3300/0 conn(0x7f411406fef0 0x7f411406f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.626+0000 7f4119a00640 1 -- 172.21.15.114:0/689256129 >> v1:172.21.15.114:6789/0 conn(0x7f411415e000 legacy=0x7f411406f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.626+0000 7f4119a00640 1 -- 172.21.15.114:0/689256129 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411406c320 con 0x7f411415e3e0 2024-03-06T07:31:12.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.626+0000 7f4113e00640 1 --2- 172.21.15.114:0/689256129 >> v2:172.21.15.177:3300/0 conn(0x7f411406fef0 0x7f411406f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.626+0000 7f4113400640 1 -- 172.21.15.114:0/689256129 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402290173 0 0) 0x7f41140d1260 con 0x7f411415e000 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.627+0000 7f4119a00640 1 --2- 172.21.15.114:0/689256129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411415e3e0 0x7f41140d0ae0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4114067cb0 tx=0x7f410c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672fdc9951b9c150 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.627+0000 7f4113400640 1 -- 172.21.15.114:0/689256129 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c035020 con 0x7f411415e3e0 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.627+0000 7f4113400640 1 -- 172.21.15.114:0/689256129 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c031830 con 0x7f411415e3e0 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.627+0000 7f4113400640 1 -- 172.21.15.114:0/689256129 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c031c00 con 0x7f411415e3e0 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 -- 172.21.15.114:0/689256129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411415e3e0 msgr2=0x7f41140d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 --2- 172.21.15.114:0/689256129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411415e3e0 0x7f41140d0ae0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4114067cb0 tx=0x7f410c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 -- 172.21.15.114:0/689256129 shutdown_connections 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 --2- 172.21.15.114:0/689256129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411415e3e0 0x7f41140d0ae0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 --2- 172.21.15.114:0/689256129 >> v2:172.21.15.177:3300/0 conn(0x7f411406fef0 0x7f411406f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 -- 172.21.15.114:0/689256129 >> 172.21.15.114:0/689256129 conn(0x7f41140b3800 msgr2=0x7f41140b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 -- 172.21.15.114:0/689256129 shutdown_connections 2024-03-06T07:31:12.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.628+0000 7f411ba00640 1 -- 172.21.15.114:0/689256129 wait complete. 2024-03-06T07:31:12.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.630+0000 7f411ba00640 1 Processor -- start 2024-03-06T07:31:12.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.630+0000 7f411ba00640 1 -- start start 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f411ba00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411406fef0 0x7f411415a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f411ba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f411415e000 0x7f41140b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f411ba00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f411415e3e0 0x7f41140c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f411ba00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140ce4c0 con 0x7f411406fef0 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f411ba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140ce340 con 0x7f411415e000 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f411ba00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140ce640 con 0x7f411415e3e0 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f4119000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411406fef0 0x7f411415a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f4119000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411406fef0 0x7f411415a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52722/0 (socket says 172.21.15.114:52722) 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f4119000640 1 -- 172.21.15.114:0/1347542739 learned_addr learned my addr 172.21.15.114:0/1347542739 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f4119a00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f411415e3e0 0x7f41140c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.631+0000 7f4113e00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f411415e000 0x7f41140b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4119000640 1 -- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f411415e3e0 msgr2=0x7f41140c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4119000640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f411415e3e0 0x7f41140c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4119000640 1 -- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f411415e000 msgr2=0x7f41140b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4119000640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f411415e000 0x7f41140b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4119000640 1 -- 172.21.15.114:0/1347542739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41140b6e60 con 0x7f411406fef0 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4113e00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f411415e000 0x7f41140b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4119a00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f411415e3e0 0x7f41140c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4119000640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411406fef0 0x7f411415a480 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f410400c500 tx=0x7f410400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4110c00640 1 -- 172.21.15.114:0/1347542739 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104018070 con 0x7f411406fef0 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4110c00640 1 -- 172.21.15.114:0/1347542739 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104007580 con 0x7f411406fef0 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f4110c00640 1 -- 172.21.15.114:0/1347542739 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104004a20 con 0x7f411406fef0 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41140b7140 con 0x7f411406fef0 2024-03-06T07:31:12.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.632+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411415f480 con 0x7f411406fef0 2024-03-06T07:31:12.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.633+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411404f330 con 0x7f411406fef0 2024-03-06T07:31:12.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.634+0000 7f4110c00640 1 -- 172.21.15.114:0/1347542739 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f410401b070 con 0x7f411406fef0 2024-03-06T07:31:12.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.635+0000 7f4110c00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f40f006b4d0 0x7f40f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.635+0000 7f4110c00640 1 -- 172.21.15.114:0/1347542739 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4104089240 con 0x7f411406fef0 2024-03-06T07:31:12.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.635+0000 7f4113e00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f40f006b4d0 0x7f40f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.636+0000 7f4113e00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f40f006b4d0 0x7f40f006d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f411415ecc0 tx=0x7f40fc007530 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.640+0000 7f4110c00640 1 -- 172.21.15.114:0/1347542739 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41040583b0 con 0x7f411406fef0 2024-03-06T07:31:12.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.833+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41140d1a90 con 0x7f411406fef0 2024-03-06T07:31:12.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.834+0000 7f4110c00640 1 -- 172.21.15.114:0/1347542739 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4104057cb0 con 0x7f411406fef0 2024-03-06T07:31:12.837 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:31:12.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.836+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f40f006b4d0 msgr2=0x7f40f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.836+0000 7f411ba00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f40f006b4d0 0x7f40f006d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f411415ecc0 tx=0x7f40fc007530 comp rx=0 tx=0).stop 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.836+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411406fef0 msgr2=0x7f411415a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.836+0000 7f411ba00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411406fef0 0x7f411415a480 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f410400c500 tx=0x7f410400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 shutdown_connections 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f411415e3e0 0x7f41140c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f411415e000 0x7f41140b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f40f006b4d0 0x7f40f006d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 --2- 172.21.15.114:0/1347542739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f411406fef0 0x7f411415a480 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 >> 172.21.15.114:0/1347542739 conn(0x7f41140b3800 msgr2=0x7f41140b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 shutdown_connections 2024-03-06T07:31:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:12.837+0000 7f411ba00640 1 -- 172.21.15.114:0/1347542739 wait complete. 2024-03-06T07:31:12.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:31:13.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.060+0000 7fe9d6800640 1 Processor -- start 2024-03-06T07:31:13.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.060+0000 7fe9d6800640 1 -- start start 2024-03-06T07:31:13.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.061+0000 7fe9d6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d015f9d0 0x7fe9d015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9d6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe9d0070b10 0x7fe9d0070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9d6800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d006cef0 con 0x7fe9d0144c00 2024-03-06T07:31:13.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9d6800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d0160370 con 0x7fe9d0070b10 2024-03-06T07:31:13.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9d6800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d006d1d0 con 0x7fe9d015f9d0 2024-03-06T07:31:13.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9cfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe9d0070b10 0x7fe9d0070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9cfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe9d0070b10 0x7fe9d0070f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52732/0 (socket says 172.21.15.114:52732) 2024-03-06T07:31:13.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9cfe00640 1 -- 172.21.15.114:0/602831371 learned_addr learned my addr 172.21.15.114:0/602831371 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:13.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9cfe00640 1 -- 172.21.15.114:0/602831371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d015f9d0 msgr2=0x7fe9d015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9cfe00640 1 --2- 172.21.15.114:0/602831371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d015f9d0 0x7fe9d015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9cfe00640 1 -- 172.21.15.114:0/602831371 >> v1:172.21.15.114:6789/0 conn(0x7fe9d0144c00 legacy=0x7fe9d006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.062+0000 7fe9cfe00640 1 -- 172.21.15.114:0/602831371 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d006d4b0 con 0x7fe9d0070b10 2024-03-06T07:31:13.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.063+0000 7fe9cfe00640 1 --2- 172.21.15.114:0/602831371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0070b10 0x7fe9d0070f60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009e50 tx=0x7fe9c802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e471e598518efdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9ce000640 1 -- 172.21.15.114:0/602831371 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c802cb00 con 0x7fe9d0070b10 2024-03-06T07:31:13.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9ce000640 1 -- 172.21.15.114:0/602831371 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c802cca0 con 0x7fe9d0070b10 2024-03-06T07:31:13.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9ce000640 1 -- 172.21.15.114:0/602831371 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8034610 con 0x7fe9d0070b10 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 -- 172.21.15.114:0/602831371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0070b10 msgr2=0x7fe9d0070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 --2- 172.21.15.114:0/602831371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0070b10 0x7fe9d0070f60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009e50 tx=0x7fe9c802bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 -- 172.21.15.114:0/602831371 shutdown_connections 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 --2- 172.21.15.114:0/602831371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0070b10 0x7fe9d0070f60 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 --2- 172.21.15.114:0/602831371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d015f9d0 0x7fe9d015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 -- 172.21.15.114:0/602831371 >> 172.21.15.114:0/602831371 conn(0x7fe9d00b1730 msgr2=0x7fe9d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 -- 172.21.15.114:0/602831371 shutdown_connections 2024-03-06T07:31:13.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.064+0000 7fe9d6800640 1 -- 172.21.15.114:0/602831371 wait complete. 2024-03-06T07:31:13.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.066+0000 7fe9d6800640 1 Processor -- start 2024-03-06T07:31:13.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.066+0000 7fe9d6800640 1 -- start start 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9d6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d0070b10 0x7fe9d015b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9d6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0144c00 0x7fe9d01e9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9d6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9d015f9d0 0x7fe9d00b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9d6800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d006cb50 con 0x7fe9d0144c00 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9d6800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d0160730 con 0x7fe9d015f9d0 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9d6800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d006ccd0 con 0x7fe9d0070b10 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9cea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0144c00 0x7fe9d01e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9cea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0144c00 0x7fe9d01e9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52734/0 (socket says 172.21.15.114:52734) 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.067+0000 7fe9cea00640 1 -- 172.21.15.114:0/3201596233 learned_addr learned my addr 172.21.15.114:0/3201596233 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9cf400640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d0070b10 0x7fe9d015b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9cea00640 1 -- 172.21.15.114:0/3201596233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d0070b10 msgr2=0x7fe9d015b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9cea00640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d0070b10 0x7fe9d015b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9cea00640 1 -- 172.21.15.114:0/3201596233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9d015f9d0 msgr2=0x7fe9d00b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9cea00640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9d015f9d0 0x7fe9d00b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9cea00640 1 -- 172.21.15.114:0/3201596233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d015bcb0 con 0x7fe9d0144c00 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9cea00640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0144c00 0x7fe9d01e9340 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00b4d0 tx=0x7fe9bc00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9c7400640 1 -- 172.21.15.114:0/3201596233 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc014070 con 0x7fe9d0144c00 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9c7400640 1 -- 172.21.15.114:0/3201596233 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc010870 con 0x7fe9d0144c00 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9c7400640 1 -- 172.21.15.114:0/3201596233 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc010c60 con 0x7fe9d0144c00 2024-03-06T07:31:13.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d01e82a0 con 0x7fe9d0144c00 2024-03-06T07:31:13.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.068+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d01e8860 con 0x7fe9d0144c00 2024-03-06T07:31:13.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.069+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d00b1550 con 0x7fe9d0144c00 2024-03-06T07:31:13.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.070+0000 7fe9c7400640 1 -- 172.21.15.114:0/3201596233 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9bc017070 con 0x7fe9d0144c00 2024-03-06T07:31:13.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.071+0000 7fe9c7400640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe9b006b4d0 0x7fe9b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.071+0000 7fe9c7400640 1 -- 172.21.15.114:0/3201596233 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe9bc089db0 con 0x7fe9d0144c00 2024-03-06T07:31:13.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.071+0000 7fe9cf400640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe9b006b4d0 0x7fe9b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.072+0000 7fe9cf400640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe9b006b4d0 0x7fe9b006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0071a30 tx=0x7fe9c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.076+0000 7fe9c7400640 1 -- 172.21.15.114:0/3201596233 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9d00b1550 con 0x7fe9d0144c00 2024-03-06T07:31:13.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.334+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fe9d0072890 con 0x7fe9d0144c00 2024-03-06T07:31:13.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.961+0000 7fe9c7400640 1 -- 172.21.15.114:0/3201596233 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v191) v1 ==== 143+0+0 (secure 0 0 0) 0x7fe9bc08e070 con 0x7fe9d0144c00 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe9b006b4d0 msgr2=0x7fe9b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe9b006b4d0 0x7fe9b006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0071a30 tx=0x7fe9c0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0144c00 msgr2=0x7fe9d01e9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0144c00 0x7fe9d01e9340 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00b4d0 tx=0x7fe9bc00b990 comp rx=0 tx=0).stop 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 shutdown_connections 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe9d015f9d0 0x7fe9d00b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe9b006b4d0 0x7fe9b006d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9d0144c00 0x7fe9d01e9340 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 --2- 172.21.15.114:0/3201596233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe9d0070b10 0x7fe9d015b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 >> 172.21.15.114:0/3201596233 conn(0x7fe9d00b1730 msgr2=0x7fe9d00ae0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 shutdown_connections 2024-03-06T07:31:13.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:13.965+0000 7fe9d6800640 1 -- 172.21.15.114:0/3201596233 wait complete. 2024-03-06T07:31:13.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:31:14.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.198+0000 7f39b4c00640 1 Processor -- start 2024-03-06T07:31:14.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.198+0000 7f39b4c00640 1 -- start start 2024-03-06T07:31:14.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.199+0000 7f39b4c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39b00b3f70 0x7f39b0160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.199+0000 7f39b4c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39b4c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b46c0 con 0x7f39b015fc10 2024-03-06T07:31:14.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39b4c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b0071480 con 0x7f39b00b3f70 2024-03-06T07:31:14.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39b4c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00b49a0 con 0x7f39b0070ff0 2024-03-06T07:31:14.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ade00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f39b015fc10 0x7f39b015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55138/0 (socket says 172.21.15.114:55138) 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ade00640 1 -- 172.21.15.114:0/285233885 learned_addr learned my addr 172.21.15.114:0/285233885 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ad400640 1 --2- 172.21.15.114:0/285233885 >> v2:172.21.15.114:3300/0 conn(0x7f39b00b3f70 0x7f39b0160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39aca00640 1 -- 172.21.15.114:0/285233885 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168624429 0 0) 0x7f39b00b46c0 con 0x7f39b015fc10 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ae800640 1 --2- 172.21.15.114:0/285233885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39aca00640 1 -- 172.21.15.114:0/285233885 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994003520 con 0x7f39b015fc10 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ad400640 1 -- 172.21.15.114:0/285233885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 msgr2=0x7f39b00ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ad400640 1 --2- 172.21.15.114:0/285233885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b00ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ad400640 1 -- 172.21.15.114:0/285233885 >> v1:172.21.15.114:6789/0 conn(0x7f39b015fc10 legacy=0x7f39b015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ad400640 1 -- 172.21.15.114:0/285233885 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b0072c90 con 0x7f39b00b3f70 2024-03-06T07:31:14.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.200+0000 7f39ae800640 1 --2- 172.21.15.114:0/285233885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:14.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.201+0000 7f39ad400640 1 --2- 172.21.15.114:0/285233885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b00b3f70 0x7f39b0160700 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097a0 tx=0x7f399c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9b7dc0364efbdd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39aca00640 1 -- 172.21.15.114:0/285233885 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c035020 con 0x7f39b00b3f70 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39aca00640 1 -- 172.21.15.114:0/285233885 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c031810 con 0x7f39b00b3f70 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39aca00640 1 -- 172.21.15.114:0/285233885 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c031be0 con 0x7f39b00b3f70 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39b4c00640 1 -- 172.21.15.114:0/285233885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b00b3f70 msgr2=0x7f39b0160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39b4c00640 1 --2- 172.21.15.114:0/285233885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b00b3f70 0x7f39b0160700 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097a0 tx=0x7f399c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39b4c00640 1 -- 172.21.15.114:0/285233885 shutdown_connections 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39b4c00640 1 --2- 172.21.15.114:0/285233885 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39b4c00640 1 --2- 172.21.15.114:0/285233885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b00b3f70 0x7f39b0160700 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.202+0000 7f39b4c00640 1 -- 172.21.15.114:0/285233885 >> 172.21.15.114:0/285233885 conn(0x7f39b00b1940 msgr2=0x7f39b00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.203+0000 7f39b4c00640 1 -- 172.21.15.114:0/285233885 shutdown_connections 2024-03-06T07:31:14.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.203+0000 7f39b4c00640 1 -- 172.21.15.114:0/285233885 wait complete. 2024-03-06T07:31:14.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.204+0000 7f39b4c00640 1 Processor -- start 2024-03-06T07:31:14.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.204+0000 7f39b4c00640 1 -- start start 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.205+0000 7f39b4c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b0141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.205+0000 7f39b4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b00b3f70 0x7f39b015c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.205+0000 7f39b4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b015fc10 0x7f39b01413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.205+0000 7f39b4c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0072a30 con 0x7f39b015fc10 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.205+0000 7f39b4c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00728b0 con 0x7f39b00b3f70 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.205+0000 7f39b4c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b015c2e0 con 0x7f39b0070ff0 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b015fc10 0x7f39b01413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b015fc10 0x7f39b01413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52756/0 (socket says 172.21.15.114:52756) 2024-03-06T07:31:14.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 -- 172.21.15.114:0/2142955776 learned_addr learned my addr 172.21.15.114:0/2142955776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:14.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 -- 172.21.15.114:0/2142955776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 msgr2=0x7f39b0141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b0141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ad400640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b00b3f70 0x7f39b015c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 -- 172.21.15.114:0/2142955776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b00b3f70 msgr2=0x7f39b015c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b00b3f70 0x7f39b015c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 -- 172.21.15.114:0/2142955776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b014cc50 con 0x7f39b015fc10 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39ae800640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b015fc10 0x7f39b01413f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f39a800cef0 tx=0x7f39a800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.206+0000 7f39a6000640 1 -- 172.21.15.114:0/2142955776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a801a070 con 0x7f39b015fc10 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.207+0000 7f39a6000640 1 -- 172.21.15.114:0/2142955776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a800d860 con 0x7f39b015fc10 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.207+0000 7f39a6000640 1 -- 172.21.15.114:0/2142955776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a800dc50 con 0x7f39b015fc10 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.207+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b009bbc0 con 0x7f39b015fc10 2024-03-06T07:31:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.207+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b009c180 con 0x7f39b015fc10 2024-03-06T07:31:14.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.208+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b004f330 con 0x7f39b015fc10 2024-03-06T07:31:14.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.209+0000 7f39a6000640 1 -- 172.21.15.114:0/2142955776 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39a801d070 con 0x7f39b015fc10 2024-03-06T07:31:14.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.209+0000 7f39a6000640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399006b4d0 0x7f399006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.209+0000 7f39a6000640 1 -- 172.21.15.114:0/2142955776 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f39a808a870 con 0x7f39b015fc10 2024-03-06T07:31:14.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.210+0000 7f39ade00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399006b4d0 0x7f399006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.210+0000 7f39ade00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399006b4d0 0x7f399006d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f39a0001530 tx=0x7f39a00078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.214+0000 7f39a6000640 1 -- 172.21.15.114:0/2142955776 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39a8059a10 con 0x7f39b015fc10 2024-03-06T07:31:14.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.395+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f39b006ef30 con 0x7f39b015fc10 2024-03-06T07:31:14.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.402+0000 7f39a6000640 1 -- 172.21.15.114:0/2142955776 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f39a8059310 con 0x7f39b015fc10 2024-03-06T07:31:14.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399006b4d0 msgr2=0x7f399006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399006b4d0 0x7f399006d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f39a0001530 tx=0x7f39a00078f0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b015fc10 msgr2=0x7f39b01413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b015fc10 0x7f39b01413f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f39a800cef0 tx=0x7f39a800a710 comp rx=0 tx=0).stop 2024-03-06T07:31:14.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 shutdown_connections 2024-03-06T07:31:14.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f399006b4d0 0x7f399006d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39b015fc10 0x7f39b01413f0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39b00b3f70 0x7f39b015c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 --2- 172.21.15.114:0/2142955776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f39b0070ff0 0x7f39b0141db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.404+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 >> 172.21.15.114:0/2142955776 conn(0x7f39b00b1940 msgr2=0x7f39b00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.405+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 shutdown_connections 2024-03-06T07:31:14.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:14.405+0000 7f39b4c00640 1 -- 172.21.15.114:0/2142955776 wait complete. 2024-03-06T07:31:14.424 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:14.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:14.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:14.647 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:14.647 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:14.647 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:14.701 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:14.701 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:31:14.702 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:14.758 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:14.758 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:14.758 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:14.814 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:14.814 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:31:14.814 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:14.873 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:14.873 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:31:14.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.083+0000 7fdba3200640 1 Processor -- start 2024-03-06T07:31:15.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.083+0000 7fdba3200640 1 -- start start 2024-03-06T07:31:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.084+0000 7fdba3200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdba40cc9d0 0x7fdba406d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.084+0000 7fdba3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba415c240 con 0x7fdba41561f0 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.084+0000 7fdba3200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba415c500 con 0x7fdba40aedc0 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.084+0000 7fdba3200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba415ce30 con 0x7fdba40cc9d0 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdba41561f0 0x7fdba40ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55144/0 (socket says 172.21.15.114:55144) 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1e00640 1 -- 172.21.15.114:0/3737627890 learned_addr learned my addr 172.21.15.114:0/3737627890 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250992637 0 0) 0x7fdba415c240 con 0x7fdba41561f0 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba2800640 1 --2- 172.21.15.114:0/3737627890 >> v2:172.21.15.177:3300/0 conn(0x7fdba40cc9d0 0x7fdba406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb88003520 con 0x7fdba41561f0 2024-03-06T07:31:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360205506 0 0) 0x7fdb88003520 con 0x7fdba41561f0 2024-03-06T07:31:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 >> v2:172.21.15.177:3300/0 conn(0x7fdba40cc9d0 msgr2=0x7fdba406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 --2- 172.21.15.114:0/3737627890 >> v2:172.21.15.177:3300/0 conn(0x7fdba40cc9d0 0x7fdba406d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 >> v1:172.21.15.177:6789/0 conn(0x7fdba40aedc0 legacy=0x7fdba40cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba415c7e0 con 0x7fdba41561f0 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.085+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fdb94001670 con 0x7fdba41561f0 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 >> v1:172.21.15.114:6789/0 conn(0x7fdba41561f0 legacy=0x7fdba40ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1400640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb88003a00 0x7fdb88003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1400640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb88004800 0x7fdb88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1400640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb880051c0 0x7fdb8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba415ce30 con 0x7fdb880051c0 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8800c0f0 con 0x7fdb88003a00 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8800c2f0 con 0x7fdb88004800 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba2800640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb880051c0 0x7fdb8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdb9be00640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb88004800 0x7fdb88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba1e00640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb88003a00 0x7fdb88003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba2800640 1 -- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb88004800 msgr2=0x7fdb88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba2800640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb88004800 0x7fdb88004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba2800640 1 -- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb88003a00 msgr2=0x7fdb88003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba2800640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb88003a00 0x7fdb88003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.086+0000 7fdba2800640 1 -- 172.21.15.114:0/3737627890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88003520 con 0x7fdb880051c0 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.087+0000 7fdba1e00640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb88003a00 0x7fdb88003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.087+0000 7fdba2800640 1 -- 172.21.15.114:0/3737627890 reap_dead start 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.088+0000 7fdba2800640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb880051c0 0x7fdb8800b9c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009fc0 tx=0x7fdb88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4514e119a7741e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.088+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c009c10 con 0x7fdb880051c0 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.088+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c02cca0 con 0x7fdb880051c0 2024-03-06T07:31:15.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.088+0000 7fdba1400640 1 -- 172.21.15.114:0/3737627890 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c003bd0 con 0x7fdb880051c0 2024-03-06T07:31:15.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.088+0000 7fdba3200640 1 -- 172.21.15.114:0/3737627890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb880051c0 msgr2=0x7fdb8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.088+0000 7fdba3200640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb880051c0 0x7fdb8800b9c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009fc0 tx=0x7fdb88005830 comp rx=0 tx=0).stop 2024-03-06T07:31:15.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.089+0000 7fdba3200640 1 -- 172.21.15.114:0/3737627890 shutdown_connections 2024-03-06T07:31:15.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.089+0000 7fdba3200640 1 --2- 172.21.15.114:0/3737627890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb880051c0 0x7fdb8800b9c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.089+0000 7fdba3200640 1 -- 172.21.15.114:0/3737627890 >> 172.21.15.114:0/3737627890 conn(0x7fdba406afc0 msgr2=0x7fdba406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.089+0000 7fdba3200640 1 -- 172.21.15.114:0/3737627890 shutdown_connections 2024-03-06T07:31:15.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.089+0000 7fdba3200640 1 -- 172.21.15.114:0/3737627890 wait complete. 2024-03-06T07:31:15.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.090+0000 7fdba3200640 1 Processor -- start 2024-03-06T07:31:15.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.090+0000 7fdba3200640 1 -- start start 2024-03-06T07:31:15.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.091+0000 7fdba3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb880051c0 0x7fdba40a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.091+0000 7fdba3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba415a4b0 0x7fdba4159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.091+0000 7fdba3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdba415a930 0x7fdba40ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.091+0000 7fdba3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4156750 con 0x7fdba415a930 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.091+0000 7fdba3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba41565d0 con 0x7fdba415a4b0 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.091+0000 7fdba3200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40cc2c0 con 0x7fdb880051c0 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdba415a930 0x7fdba40ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdba415a930 0x7fdba40ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52768/0 (socket says 172.21.15.114:52768) 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 -- 172.21.15.114:0/1721801385 learned_addr learned my addr 172.21.15.114:0/1721801385 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 -- 172.21.15.114:0/1721801385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb880051c0 msgr2=0x7fdba40a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb880051c0 0x7fdba40a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 -- 172.21.15.114:0/1721801385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba415a4b0 msgr2=0x7fdba4159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba415a4b0 0x7fdba4159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 -- 172.21.15.114:0/1721801385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4157cb0 con 0x7fdba415a930 2024-03-06T07:31:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdba2800640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdba415a930 0x7fdba40ae1e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00e200 tx=0x7fdb9c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.092+0000 7fdb9a000640 1 -- 172.21.15.114:0/1721801385 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c009c10 con 0x7fdba415a930 2024-03-06T07:31:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.093+0000 7fdb9a000640 1 -- 172.21.15.114:0/1721801385 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c044070 con 0x7fdba415a930 2024-03-06T07:31:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.093+0000 7fdb9a000640 1 -- 172.21.15.114:0/1721801385 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c003b90 con 0x7fdba415a930 2024-03-06T07:31:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.093+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba4157f90 con 0x7fdba415a930 2024-03-06T07:31:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.093+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba4158830 con 0x7fdba415a930 2024-03-06T07:31:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.094+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba404f330 con 0x7fdba415a930 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.095+0000 7fdb9a000640 1 -- 172.21.15.114:0/1721801385 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb9c005d10 con 0x7fdba415a930 2024-03-06T07:31:15.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.096+0000 7fdb9a000640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb8006b4d0 0x7fdb8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.096+0000 7fdb9a000640 1 -- 172.21.15.114:0/1721801385 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdb9c0ac530 con 0x7fdba415a930 2024-03-06T07:31:15.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.096+0000 7fdba1e00640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb8006b4d0 0x7fdb8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.097+0000 7fdba1e00640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb8006b4d0 0x7fdb8006d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdb940078d0 tx=0x7fdb94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.101+0000 7fdb9a000640 1 -- 172.21.15.114:0/1721801385 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb9c0333a0 con 0x7fdba415a930 2024-03-06T07:31:15.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.334+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba40b2b10 con 0x7fdba415a930 2024-03-06T07:31:15.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.335+0000 7fdb9a000640 1 -- 172.21.15.114:0/1721801385 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fdb9c07b7d0 con 0x7fdba415a930 2024-03-06T07:31:15.338 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:15.338 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/262539266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":262539266},{"type":"v1","addr":"172.21.15.114:6836","nonce":262539266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":4847,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/2750482346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2750482346},{"type":"v1","addr":"172.21.15.177:6834","nonce":2750482346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5149,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/301657919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":301657919},{"type":"v1","addr":"172.21.15.177:6835","nonce":301657919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6243,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/4245376178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":4245376178},{"type":"v1","addr":"172.21.15.114:6837","nonce":4245376178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[]} 2024-03-06T07:31:15.338 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 37 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb8006b4d0 msgr2=0x7fdb8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb8006b4d0 0x7fdb8006d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdb940078d0 tx=0x7fdb94009040 comp rx=0 tx=0).stop 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdba415a930 msgr2=0x7fdba40ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdba415a930 0x7fdba40ae1e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00e200 tx=0x7fdb9c009750 comp rx=0 tx=0).stop 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 shutdown_connections 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb8006b4d0 0x7fdb8006d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdba415a930 0x7fdba40ae1e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba415a4b0 0x7fdba4159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 --2- 172.21.15.114:0/1721801385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb880051c0 0x7fdba40a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 >> 172.21.15.114:0/1721801385 conn(0x7fdba406afc0 msgr2=0x7fdba40b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 shutdown_connections 2024-03-06T07:31:15.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.338+0000 7fdba3200640 1 -- 172.21.15.114:0/1721801385 wait complete. 2024-03-06T07:31:15.359 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:31:15.359 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:31:15.360 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:31:15.360 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:15.360 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:31:15.360 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:31:15.360 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:31:15.360 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:15.360 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:31:15.361 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:31:15.361 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:31:15.361 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:15.361 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:31:15.361 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:31:15.361 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:31:15.361 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:15.377 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:31:15.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:31:15.378 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:31:15.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:31:15.385 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:31:15.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:31:15.388 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:31:15.388 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:31:15.425 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:31:15.425 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:15.430 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:31:15.430 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:15.435 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:31:15.435 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:15.441 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:31:15.441 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:15.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:31:15.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:15.491 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:15.572 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:31:15.577 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:31:15.592 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:31:15.600 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:31:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.616+0000 7fbacbc00640 1 Processor -- start 2024-03-06T07:31:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.617+0000 7fbacbc00640 1 -- start start 2024-03-06T07:31:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbacbc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac4160400 0x7fbac406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbacbc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac406b500 0x7fbac41414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbacbc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac415f940 con 0x7fbac413ef20 2024-03-06T07:31:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbacbc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac406e9c0 con 0x7fbac406b500 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbacbc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac415ce80 con 0x7fbac4160400 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbac9200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbac413ef20 0x7fbac415c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55186/0 (socket says 172.21.15.114:55186) 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbac9c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac406b500 0x7fbac41414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbac9200640 1 -- 172.21.15.114:0/422635529 learned_addr learned my addr 172.21.15.114:0/422635529 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.618+0000 7fbac3e00640 1 --2- 172.21.15.114:0/422635529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac4160400 0x7fbac406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.619+0000 7fbac9c00640 1 -- 172.21.15.114:0/422635529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac4160400 msgr2=0x7fbac406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.619+0000 7fbac9c00640 1 --2- 172.21.15.114:0/422635529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac4160400 0x7fbac406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.619+0000 7fbac9c00640 1 -- 172.21.15.114:0/422635529 >> v1:172.21.15.114:6789/0 conn(0x7fbac413ef20 legacy=0x7fbac415c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.619+0000 7fbac3400640 1 -- 172.21.15.114:0/422635529 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869982456 0 0) 0x7fbac415f940 con 0x7fbac413ef20 2024-03-06T07:31:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.619+0000 7fbac9c00640 1 -- 172.21.15.114:0/422635529 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac415d160 con 0x7fbac406b500 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.620+0000 7fbac9c00640 1 --2- 172.21.15.114:0/422635529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac406b500 0x7fbac41414b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009d90 tx=0x7fbabc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16002eb67421db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.620+0000 7fbac3400640 1 -- 172.21.15.114:0/422635529 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc039070 con 0x7fbac406b500 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.620+0000 7fbac3400640 1 -- 172.21.15.114:0/422635529 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc02cc50 con 0x7fbac406b500 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.620+0000 7fbac3400640 1 -- 172.21.15.114:0/422635529 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc034440 con 0x7fbac406b500 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 -- 172.21.15.114:0/422635529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac406b500 msgr2=0x7fbac41414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 --2- 172.21.15.114:0/422635529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac406b500 0x7fbac41414b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009d90 tx=0x7fbabc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 -- 172.21.15.114:0/422635529 shutdown_connections 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 --2- 172.21.15.114:0/422635529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac406b500 0x7fbac41414b0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 --2- 172.21.15.114:0/422635529 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac4160400 0x7fbac406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 -- 172.21.15.114:0/422635529 >> 172.21.15.114:0/422635529 conn(0x7fbac40d0b90 msgr2=0x7fbac40d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 -- 172.21.15.114:0/422635529 shutdown_connections 2024-03-06T07:31:15.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.621+0000 7fbacbc00640 1 -- 172.21.15.114:0/422635529 wait complete. 2024-03-06T07:31:15.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.622+0000 7fbacbc00640 1 Processor -- start 2024-03-06T07:31:15.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.623+0000 7fbacbc00640 1 -- start start 2024-03-06T07:31:15.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.623+0000 7fbacbc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac406b500 0x7fbac41e5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.623+0000 7fbacbc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac413ef20 0x7fbac40d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.623+0000 7fbacbc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4160400 0x7fbac41e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbacbc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac40700e0 con 0x7fbac413ef20 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbacbc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac415fee0 con 0x7fbac4160400 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbacbc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac415ca60 con 0x7fbac406b500 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac413ef20 0x7fbac40d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac413ef20 0x7fbac40d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52830/0 (socket says 172.21.15.114:52830) 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 -- 172.21.15.114:0/3232091933 learned_addr learned my addr 172.21.15.114:0/3232091933 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac9c00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4160400 0x7fbac41e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac9200640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac406b500 0x7fbac41e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 -- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac406b500 msgr2=0x7fbac41e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac406b500 0x7fbac41e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 -- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4160400 msgr2=0x7fbac41e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4160400 0x7fbac41e6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.624+0000 7fbac3e00640 1 -- 172.21.15.114:0/3232091933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac41e5e60 con 0x7fbac413ef20 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.625+0000 7fbac3e00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac413ef20 0x7fbac40d02f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbab000e8a0 tx=0x7fbab000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.625+0000 7fbac0c00640 1 -- 172.21.15.114:0/3232091933 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0017070 con 0x7fbac413ef20 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.625+0000 7fbac0c00640 1 -- 172.21.15.114:0/3232091933 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000c8e0 con 0x7fbac413ef20 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.625+0000 7fbac0c00640 1 -- 172.21.15.114:0/3232091933 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000ccd0 con 0x7fbac413ef20 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.625+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac41e6140 con 0x7fbac413ef20 2024-03-06T07:31:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.625+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac41e6700 con 0x7fbac413ef20 2024-03-06T07:31:15.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.626+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac404f330 con 0x7fbac413ef20 2024-03-06T07:31:15.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.627+0000 7fbac0c00640 1 -- 172.21.15.114:0/3232091933 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbab001a070 con 0x7fbac413ef20 2024-03-06T07:31:15.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.628+0000 7fbac0c00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa406b500 0x7fbaa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.628+0000 7fbac0c00640 1 -- 172.21.15.114:0/3232091933 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbab008a120 con 0x7fbac413ef20 2024-03-06T07:31:15.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.628+0000 7fbac9200640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa406b500 0x7fbaa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.629+0000 7fbac9200640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa406b500 0x7fbaa406d9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbab8007f70 tx=0x7fbab8007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.632+0000 7fbac0c00640 1 -- 172.21.15.114:0/3232091933 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbab0058b20 con 0x7fbac413ef20 2024-03-06T07:31:15.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.840+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbac40d1aa0 con 0x7fbac413ef20 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.986+0000 7fbac0c00640 1 -- 172.21.15.114:0/3232091933 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v60) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbab005cbc0 con 0x7fbac413ef20 2024-03-06T07:31:15.989 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:31:15.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.989+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa406b500 msgr2=0x7fbaa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.989+0000 7fbacbc00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa406b500 0x7fbaa406d9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fbab8007f70 tx=0x7fbab8007f00 comp rx=0 tx=0).stop 2024-03-06T07:31:15.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.989+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac413ef20 msgr2=0x7fbac40d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.989+0000 7fbacbc00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac413ef20 0x7fbac40d02f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbab000e8a0 tx=0x7fbab000ed60 comp rx=0 tx=0).stop 2024-03-06T07:31:15.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 shutdown_connections 2024-03-06T07:31:15.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac4160400 0x7fbac41e6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa406b500 0x7fbaa406d9c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac413ef20 0x7fbac40d02f0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 --2- 172.21.15.114:0/3232091933 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac406b500 0x7fbac41e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 >> 172.21.15.114:0/3232091933 conn(0x7fbac40d0b90 msgr2=0x7fbac40cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 shutdown_connections 2024-03-06T07:31:15.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:15.990+0000 7fbacbc00640 1 -- 172.21.15.114:0/3232091933 wait complete. 2024-03-06T07:31:16.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:31:16.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.264+0000 7fda8c000640 1 Processor -- start 2024-03-06T07:31:16.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.264+0000 7fda8c000640 1 -- start start 2024-03-06T07:31:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.265+0000 7fda8c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 0x7fda8415e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.265+0000 7fda8c000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840b35b0 con 0x7fda840cf430 2024-03-06T07:31:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.265+0000 7fda8c000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840b3890 con 0x7fda84151660 2024-03-06T07:31:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.265+0000 7fda8c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda840b3ee0 con 0x7fda8415dff0 2024-03-06T07:31:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda88c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fda840cf430 0x7fda8406d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55190/0 (socket says 172.21.15.114:55190) 2024-03-06T07:31:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda88c00640 1 -- 172.21.15.114:0/3600079445 learned_addr learned my addr 172.21.15.114:0/3600079445 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda8a000640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 0x7fda8415e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940131835 0 0) 0x7fda840b35b0 con 0x7fda840cf430 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6c003520 con 0x7fda840cf430 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169807930 0 0) 0x7fda6c003520 con 0x7fda840cf430 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 msgr2=0x7fda8415e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 0x7fda8415e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 >> v1:172.21.15.177:6789/0 conn(0x7fda84151660 legacy=0x7fda84071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840b3b70 con 0x7fda840cf430 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fda70001670 con 0x7fda840cf430 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.266+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 >> v1:172.21.15.114:6789/0 conn(0x7fda840cf430 legacy=0x7fda8406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda83e00640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c0039e0 0x7fda6c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda83e00640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 0x7fda6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda83e00640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda6c004d40 0x7fda6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda840b3ee0 con 0x7fda6c004d40 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00bbb0 con 0x7fda6c0039e0 2024-03-06T07:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00bdb0 con 0x7fda8415dff0 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda8a000640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda6c004d40 0x7fda6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda88c00640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 0x7fda6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda89600640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c0039e0 0x7fda6c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda8a000640 1 -- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 msgr2=0x7fda6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda8a000640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda8415dff0 0x7fda6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda8a000640 1 -- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c0039e0 msgr2=0x7fda6c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda8a000640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c0039e0 0x7fda6c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.267+0000 7fda8a000640 1 -- 172.21.15.114:0/3600079445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c003520 con 0x7fda6c004d40 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.268+0000 7fda8a000640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda6c004d40 0x7fda6c0051c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009fc0 tx=0x7fda6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65d5f161ea91a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c03b070 con 0x7fda6c004d40 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c02ee30 con 0x7fda6c004d40 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda83e00640 1 -- 172.21.15.114:0/3600079445 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c009c10 con 0x7fda6c004d40 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda8c000640 1 -- 172.21.15.114:0/3600079445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda6c004d40 msgr2=0x7fda6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda8c000640 1 --2- 172.21.15.114:0/3600079445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda6c004d40 0x7fda6c0051c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009fc0 tx=0x7fda6c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda8a000640 1 -- 172.21.15.114:0/3600079445 reap_dead start 2024-03-06T07:31:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda8c000640 1 -- 172.21.15.114:0/3600079445 shutdown_connections 2024-03-06T07:31:16.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda8c000640 1 -- 172.21.15.114:0/3600079445 >> 172.21.15.114:0/3600079445 conn(0x7fda840b1730 msgr2=0x7fda840b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda8c000640 1 -- 172.21.15.114:0/3600079445 shutdown_connections 2024-03-06T07:31:16.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.269+0000 7fda8c000640 1 -- 172.21.15.114:0/3600079445 wait complete. 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.271+0000 7fda8c000640 1 Processor -- start 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.271+0000 7fda8c000640 1 -- start start 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda8c000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c004d40 0x7fda84159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda8c000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda841f8370 0x7fda841f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda8c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda841f46d0 0x7fda84146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda8c000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8415e480 con 0x7fda841f8370 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda8c000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8406daf0 con 0x7fda6c004d40 2024-03-06T07:31:16.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda8c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8415e600 con 0x7fda841f46d0 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda88c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda841f8370 0x7fda841f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda88c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda841f8370 0x7fda841f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52848/0 (socket says 172.21.15.114:52848) 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.272+0000 7fda88c00640 1 -- 172.21.15.114:0/2112648366 learned_addr learned my addr 172.21.15.114:0/2112648366 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda88c00640 1 -- 172.21.15.114:0/2112648366 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda841f46d0 msgr2=0x7fda84146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda88c00640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda841f46d0 0x7fda84146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda88c00640 1 -- 172.21.15.114:0/2112648366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c004d40 msgr2=0x7fda84159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda88c00640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c004d40 0x7fda84159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda88c00640 1 -- 172.21.15.114:0/2112648366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda841f50c0 con 0x7fda841f8370 2024-03-06T07:31:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda88c00640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda841f8370 0x7fda841f4190 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e730 tx=0x7fda7000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda81600640 1 -- 172.21.15.114:0/2112648366 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000c640 con 0x7fda841f8370 2024-03-06T07:31:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda81600640 1 -- 172.21.15.114:0/2112648366 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda70018070 con 0x7fda841f8370 2024-03-06T07:31:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda841f4b50 con 0x7fda841f8370 2024-03-06T07:31:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda81600640 1 -- 172.21.15.114:0/2112648366 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000ca60 con 0x7fda841f8370 2024-03-06T07:31:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.273+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda841f7890 con 0x7fda841f8370 2024-03-06T07:31:16.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.275+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8404f330 con 0x7fda841f8370 2024-03-06T07:31:16.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.275+0000 7fda81600640 1 -- 172.21.15.114:0/2112648366 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda70013420 con 0x7fda841f8370 2024-03-06T07:31:16.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.276+0000 7fda81600640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda6406b500 0x7fda6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.276+0000 7fda81600640 1 -- 172.21.15.114:0/2112648366 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fda70089b10 con 0x7fda841f8370 2024-03-06T07:31:16.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.276+0000 7fda89600640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda6406b500 0x7fda6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.277+0000 7fda89600640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda6406b500 0x7fda6406d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fda78001640 tx=0x7fda78009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.281+0000 7fda81600640 1 -- 172.21.15.114:0/2112648366 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda841f50c0 con 0x7fda841f8370 2024-03-06T07:31:16.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.528+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fda840b1d60 con 0x7fda841f8370 2024-03-06T07:31:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.529+0000 7fda81600640 1 -- 172.21.15.114:0/2112648366 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fda70058d70 con 0x7fda841f8370 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:31:16.534 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:31:16.536 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow r", 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.541 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.543 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:31:16.544 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:31:16.545 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.531+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda6406b500 msgr2=0x7fda6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.531+0000 7fda8c000640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda6406b500 0x7fda6406d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fda78001640 tx=0x7fda78009040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.531+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda841f8370 msgr2=0x7fda841f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.531+0000 7fda8c000640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda841f8370 0x7fda841f4190 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e730 tx=0x7fda7000ebf0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.531+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 shutdown_connections 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.531+0000 7fda8c000640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda841f46d0 0x7fda84146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.532+0000 7fda8c000640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda6406b500 0x7fda6406d9c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.532+0000 7fda8c000640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda841f8370 0x7fda841f4190 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.532+0000 7fda8c000640 1 --2- 172.21.15.114:0/2112648366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda6c004d40 0x7fda84159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.532+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 >> 172.21.15.114:0/2112648366 conn(0x7fda840b1730 msgr2=0x7fda840b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.532+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 shutdown_connections 2024-03-06T07:31:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.532+0000 7fda8c000640 1 -- 172.21.15.114:0/2112648366 wait complete. 2024-03-06T07:31:16.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.758+0000 7f2453400640 1 Processor -- start 2024-03-06T07:31:16.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.759+0000 7f2453400640 1 -- start start 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.760+0000 7f2453400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f244c0b2de0 0x7f244c0d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.760+0000 7f2453400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b0aa0 0x7f244c0b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.760+0000 7f2453400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0d4c50 con 0x7f244c0d24c0 2024-03-06T07:31:16.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.760+0000 7f2453400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c0d4a00 con 0x7f244c0b2de0 2024-03-06T07:31:16.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.760+0000 7f2453400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0cf7e0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.760+0000 7f2451400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b0aa0 0x7f244c0b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f2450a00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f244c0d24c0 0x7f244c0d28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:51952/0 (socket says 172.21.15.114:51952) 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f2450a00640 1 -- 172.21.15.114:0/276109912 learned_addr learned my addr 172.21.15.114:0/276109912 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f2451400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b0aa0 0x7f244c0b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:36368/0 (socket says 172.21.15.114:36368) 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f244be00640 1 --2- 172.21.15.114:0/276109912 >> v2:172.21.15.177:3300/0 conn(0x7f244c0b2de0 0x7f244c0d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f2451400640 1 -- 172.21.15.114:0/276109912 >> v2:172.21.15.177:3300/0 conn(0x7f244c0b2de0 msgr2=0x7f244c0d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f2451400640 1 --2- 172.21.15.114:0/276109912 >> v2:172.21.15.177:3300/0 conn(0x7f244c0b2de0 0x7f244c0d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f2451400640 1 -- 172.21.15.114:0/276109912 >> v1:172.21.15.177:6789/0 conn(0x7f244c0d24c0 legacy=0x7f244c0d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.761+0000 7f2451400640 1 -- 172.21.15.114:0/276109912 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c0cfa30 con 0x7f244c0b0aa0 2024-03-06T07:31:16.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.763+0000 7f2451400640 1 --2- 172.21.15.114:0/276109912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b0aa0 0x7f244c0b0f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f24440097a0 tx=0x7f244402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b6c014c71e345dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.763+0000 7f244b400640 1 -- 172.21.15.114:0/276109912 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24440315a0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f244b400640 1 -- 172.21.15.114:0/276109912 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2444037070 con 0x7f244c0b0aa0 2024-03-06T07:31:16.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f244b400640 1 -- 172.21.15.114:0/276109912 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444031960 con 0x7f244c0b0aa0 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 -- 172.21.15.114:0/276109912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b0aa0 msgr2=0x7f244c0b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 --2- 172.21.15.114:0/276109912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b0aa0 0x7f244c0b0f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f24440097a0 tx=0x7f244402b840 comp rx=0 tx=0).stop 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 -- 172.21.15.114:0/276109912 shutdown_connections 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 --2- 172.21.15.114:0/276109912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b0aa0 0x7f244c0b0f30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 --2- 172.21.15.114:0/276109912 >> v2:172.21.15.177:3300/0 conn(0x7f244c0b2de0 0x7f244c0d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 -- 172.21.15.114:0/276109912 >> 172.21.15.114:0/276109912 conn(0x7f244c0d3ee0 msgr2=0x7f244c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 -- 172.21.15.114:0/276109912 shutdown_connections 2024-03-06T07:31:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.764+0000 7f2453400640 1 -- 172.21.15.114:0/276109912 wait complete. 2024-03-06T07:31:16.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.766+0000 7f2453400640 1 Processor -- start 2024-03-06T07:31:16.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.766+0000 7f2453400640 1 -- start start 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2453400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f244c0b0aa0 0x7f244c153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2453400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b2de0 0x7f244c0b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2453400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c0d24c0 0x7f244c0b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2453400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c15e5b0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2453400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0d0080 con 0x7f244c0d24c0 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2453400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c15e730 con 0x7f244c0b2de0 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2450a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f244c0b0aa0 0x7f244c153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2450a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f244c0b0aa0 0x7f244c153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52862/0 (socket says 172.21.15.114:52862) 2024-03-06T07:31:16.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.767+0000 7f2450a00640 1 -- 172.21.15.114:0/3139511678 learned_addr learned my addr 172.21.15.114:0/3139511678 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2451400640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c0d24c0 0x7f244c0b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f244be00640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b2de0 0x7f244c0b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2450a00640 1 -- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b2de0 msgr2=0x7f244c0b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2450a00640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b2de0 0x7f244c0b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2450a00640 1 -- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c0d24c0 msgr2=0x7f244c0b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2450a00640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c0d24c0 0x7f244c0b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2450a00640 1 -- 172.21.15.114:0/3139511678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c15eee0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2450a00640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f244c0b0aa0 0x7f244c153240 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f243c00ca70 tx=0x7f243c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2448c00640 1 -- 172.21.15.114:0/3139511678 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c017070 con 0x7f244c0b0aa0 2024-03-06T07:31:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2448c00640 1 -- 172.21.15.114:0/3139511678 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243c00abd0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2448c00640 1 -- 172.21.15.114:0/3139511678 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c012400 con 0x7f244c0b0aa0 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c0b80e0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.768+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c0b86a0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.770+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c04f330 con 0x7f244c0b0aa0 2024-03-06T07:31:16.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.770+0000 7f2448c00640 1 -- 172.21.15.114:0/3139511678 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f243c01a070 con 0x7f244c0b0aa0 2024-03-06T07:31:16.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.771+0000 7f2448c00640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f242806b4d0 0x7f242806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.771+0000 7f2448c00640 1 -- 172.21.15.114:0/3139511678 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f243c089c90 con 0x7f244c0b0aa0 2024-03-06T07:31:16.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.771+0000 7f244be00640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f242806b4d0 0x7f242806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.772+0000 7f244be00640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f242806b4d0 0x7f242806d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f244c1f7850 tx=0x7f2434008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.776+0000 7f2448c00640 1 -- 172.21.15.114:0/3139511678 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f243c058dc0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.971+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f244c0d22a0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.973+0000 7f2448c00640 1 -- 172.21.15.114:0/3139511678 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+12939 (secure 0 0 0) 0x7f243c0586c0 con 0x7f244c0b0aa0 2024-03-06T07:31:16.976 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:16.976 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":62,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:16.000887+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:16.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.975+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f242806b4d0 msgr2=0x7f242806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.975+0000 7f2453400640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f242806b4d0 0x7f242806d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f244c1f7850 tx=0x7f2434008040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.975+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f244c0b0aa0 msgr2=0x7f244c153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.975+0000 7f2453400640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f244c0b0aa0 0x7f244c153240 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f243c00ca70 tx=0x7f243c00cf30 comp rx=0 tx=0).stop 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 shutdown_connections 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c0d24c0 0x7f244c0b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f244c0b2de0 0x7f244c0b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f242806b4d0 0x7f242806d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 --2- 172.21.15.114:0/3139511678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f244c0b0aa0 0x7f244c153240 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 >> 172.21.15.114:0/3139511678 conn(0x7f244c0d3ee0 msgr2=0x7f244c0d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 shutdown_connections 2024-03-06T07:31:16.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:16.976+0000 7f2453400640 1 -- 172.21.15.114:0/3139511678 wait complete. 2024-03-06T07:31:16.999 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:16.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:17.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.205+0000 7fc61be00640 1 Processor -- start 2024-03-06T07:31:17.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.206+0000 7fc61be00640 1 -- start start 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.207+0000 7fc61be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc61c15be10 0x7fc61c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.207+0000 7fc61be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc61c06da40 0x7fc61c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.207+0000 7fc61be00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61c15c1f0 con 0x7fc61c0aef20 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.207+0000 7fc61be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61c0cd200 con 0x7fc61c15be10 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.207+0000 7fc61be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61c06cac0 con 0x7fc61c06da40 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc61aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc61c15be10 0x7fc61c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc61aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc61c15be10 0x7fc61c0ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52874/0 (socket says 172.21.15.114:52874) 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc61aa00640 1 -- 172.21.15.114:0/2219213262 learned_addr learned my addr 172.21.15.114:0/2219213262 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc612000640 1 --1- 172.21.15.114:0/2219213262 >> v1:172.21.15.114:6789/0 conn(0x7fc61c0aef20 0x7fc61c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55198/0 (socket says 172.21.15.114:55198) 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc61aa00640 1 -- 172.21.15.114:0/2219213262 >> v2:172.21.15.177:3300/0 conn(0x7fc61c06da40 msgr2=0x7fc61c0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc61aa00640 1 --2- 172.21.15.114:0/2219213262 >> v2:172.21.15.177:3300/0 conn(0x7fc61c06da40 0x7fc61c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc61aa00640 1 -- 172.21.15.114:0/2219213262 >> v1:172.21.15.114:6789/0 conn(0x7fc61c0aef20 legacy=0x7fc61c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.208+0000 7fc61aa00640 1 -- 172.21.15.114:0/2219213262 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c06cda0 con 0x7fc61c15be10 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.209+0000 7fc61aa00640 1 --2- 172.21.15.114:0/2219213262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c15be10 0x7fc61c0ccc20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0b1500 tx=0x7fc60c02b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a31a96b5c15ffc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.209+0000 7fc61a000640 1 -- 172.21.15.114:0/2219213262 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c034070 con 0x7fc61c15be10 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.209+0000 7fc61a000640 1 -- 172.21.15.114:0/2219213262 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c030890 con 0x7fc61c15be10 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.209+0000 7fc61a000640 1 -- 172.21.15.114:0/2219213262 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c030c60 con 0x7fc61c15be10 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 -- 172.21.15.114:0/2219213262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c15be10 msgr2=0x7fc61c0ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 --2- 172.21.15.114:0/2219213262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c15be10 0x7fc61c0ccc20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0b1500 tx=0x7fc60c02b9d0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 -- 172.21.15.114:0/2219213262 shutdown_connections 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 --2- 172.21.15.114:0/2219213262 >> v2:172.21.15.177:3300/0 conn(0x7fc61c06da40 0x7fc61c0ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 --2- 172.21.15.114:0/2219213262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c15be10 0x7fc61c0ccc20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 -- 172.21.15.114:0/2219213262 >> 172.21.15.114:0/2219213262 conn(0x7fc61c06afc0 msgr2=0x7fc61c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 -- 172.21.15.114:0/2219213262 shutdown_connections 2024-03-06T07:31:17.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.210+0000 7fc61be00640 1 -- 172.21.15.114:0/2219213262 wait complete. 2024-03-06T07:31:17.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.212+0000 7fc61be00640 1 Processor -- start 2024-03-06T07:31:17.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.213+0000 7fc61be00640 1 -- start start 2024-03-06T07:31:17.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc61be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc61c06da40 0x7fc61c1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc61be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c0aef20 0x7fc61c15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc61be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc61c15be10 0x7fc61c14ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc61be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c0cda90 con 0x7fc61c0aef20 2024-03-06T07:31:17.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc61be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c06d080 con 0x7fc61c15be10 2024-03-06T07:31:17.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc61be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c071ab0 con 0x7fc61c06da40 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c0aef20 0x7fc61c15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c0aef20 0x7fc61c15a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52888/0 (socket says 172.21.15.114:52888) 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 -- 172.21.15.114:0/3358156408 learned_addr learned my addr 172.21.15.114:0/3358156408 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc61aa00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc61c06da40 0x7fc61c1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 -- 172.21.15.114:0/3358156408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc61c06da40 msgr2=0x7fc61c1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc61c06da40 0x7fc61c1591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 -- 172.21.15.114:0/3358156408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc61c15be10 msgr2=0x7fc61c14ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc61c15be10 0x7fc61c14ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.214+0000 7fc612000640 1 -- 172.21.15.114:0/3358156408 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c159850 con 0x7fc61c0aef20 2024-03-06T07:31:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.215+0000 7fc612000640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c0aef20 0x7fc61c15a8f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c430 tx=0x7fc60400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.215+0000 7fc613400640 1 -- 172.21.15.114:0/3358156408 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6040074e0 con 0x7fc61c0aef20 2024-03-06T07:31:17.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.215+0000 7fc613400640 1 -- 172.21.15.114:0/3358156408 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc604007680 con 0x7fc61c0aef20 2024-03-06T07:31:17.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.215+0000 7fc613400640 1 -- 172.21.15.114:0/3358156408 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc604004a00 con 0x7fc61c0aef20 2024-03-06T07:31:17.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.215+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c159b30 con 0x7fc61c0aef20 2024-03-06T07:31:17.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.215+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c15a0f0 con 0x7fc61c0aef20 2024-03-06T07:31:17.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.216+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61c04f330 con 0x7fc61c0aef20 2024-03-06T07:31:17.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.217+0000 7fc613400640 1 -- 172.21.15.114:0/3358156408 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc60401a020 con 0x7fc61c0aef20 2024-03-06T07:31:17.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.218+0000 7fc613400640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.218+0000 7fc613400640 1 -- 172.21.15.114:0/3358156408 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc604089730 con 0x7fc61c0aef20 2024-03-06T07:31:17.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.219+0000 7fc61aa00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.219+0000 7fc61aa00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f806b4d0 0x7fc5f806d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036000 tx=0x7fc60c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.223+0000 7fc613400640 1 -- 172.21.15.114:0/3358156408 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc604058860 con 0x7fc61c0aef20 2024-03-06T07:31:17.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.416+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc61c0b4500 con 0x7fc61c0aef20 2024-03-06T07:31:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.418+0000 7fc613400640 1 -- 172.21.15.114:0/3358156408 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc604058160 con 0x7fc61c0aef20 2024-03-06T07:31:17.420 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:31:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f806b4d0 msgr2=0x7fc5f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f806b4d0 0x7fc5f806d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036000 tx=0x7fc60c03b040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c0aef20 msgr2=0x7fc61c15a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c0aef20 0x7fc61c15a8f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c430 tx=0x7fc60400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 shutdown_connections 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc61c15be10 0x7fc61c14ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c0aef20 0x7fc61c15a8f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.420+0000 7fc61be00640 1 --2- 172.21.15.114:0/3358156408 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc61c06da40 0x7fc61c1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.421+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 >> 172.21.15.114:0/3358156408 conn(0x7fc61c06afc0 msgr2=0x7fc61c0affa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.421+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 shutdown_connections 2024-03-06T07:31:17.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.421+0000 7fc61be00640 1 -- 172.21.15.114:0/3358156408 wait complete. 2024-03-06T07:31:17.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:31:17.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.642+0000 7fc9fd400640 1 Processor -- start 2024-03-06T07:31:17.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.642+0000 7fc9fd400640 1 -- start start 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.643+0000 7fc9fd400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9f8144c00 0x7fc9f806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9fd400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f815fe60 0x7fc9f80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9fd400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f806ee70 con 0x7fc9f815f9d0 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9fd400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f806cef0 con 0x7fc9f8144c00 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9fd400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f806d1d0 con 0x7fc9f815fe60 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9f6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9f8144c00 0x7fc9f806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9f6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9f8144c00 0x7fc9f806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52896/0 (socket says 172.21.15.114:52896) 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9f6800640 1 -- 172.21.15.114:0/2700386785 learned_addr learned my addr 172.21.15.114:0/2700386785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:17.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.644+0000 7fc9f7200640 1 --2- 172.21.15.114:0/2700386785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f815fe60 0x7fc9f80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.645+0000 7fc9f6800640 1 -- 172.21.15.114:0/2700386785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f815fe60 msgr2=0x7fc9f80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.645+0000 7fc9f6800640 1 --2- 172.21.15.114:0/2700386785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f815fe60 0x7fc9f80ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.645+0000 7fc9f6800640 1 -- 172.21.15.114:0/2700386785 >> v1:172.21.15.177:6789/0 conn(0x7fc9f815f9d0 legacy=0x7fc9f80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.645+0000 7fc9f6800640 1 -- 172.21.15.114:0/2700386785 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f806d4b0 con 0x7fc9f8144c00 2024-03-06T07:31:17.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.645+0000 7fc9f7200640 1 --2- 172.21.15.114:0/2700386785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f815fe60 0x7fc9f80ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:17.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.646+0000 7fc9f6800640 1 --2- 172.21.15.114:0/2700386785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f8144c00 0x7fc9f806e930 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009870 tx=0x7fc9e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e3aa58cbfe40ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9f5400640 1 -- 172.21.15.114:0/2700386785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8034070 con 0x7fc9f8144c00 2024-03-06T07:31:17.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9f5400640 1 -- 172.21.15.114:0/2700386785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e8030840 con 0x7fc9f8144c00 2024-03-06T07:31:17.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9f5400640 1 -- 172.21.15.114:0/2700386785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8030c10 con 0x7fc9f8144c00 2024-03-06T07:31:17.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 -- 172.21.15.114:0/2700386785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f8144c00 msgr2=0x7fc9f806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2700386785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f8144c00 0x7fc9f806e930 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009870 tx=0x7fc9e802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:17.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 -- 172.21.15.114:0/2700386785 shutdown_connections 2024-03-06T07:31:17.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2700386785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f815fe60 0x7fc9f80ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2700386785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f8144c00 0x7fc9f806e930 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 -- 172.21.15.114:0/2700386785 >> 172.21.15.114:0/2700386785 conn(0x7fc9f80b1730 msgr2=0x7fc9f80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 -- 172.21.15.114:0/2700386785 shutdown_connections 2024-03-06T07:31:17.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.647+0000 7fc9fd400640 1 -- 172.21.15.114:0/2700386785 wait complete. 2024-03-06T07:31:17.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.649+0000 7fc9fd400640 1 Processor -- start 2024-03-06T07:31:17.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.649+0000 7fc9fd400640 1 -- start start 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.650+0000 7fc9fd400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f8144c00 0x7fc9f81508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.650+0000 7fc9fd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f815f9d0 0x7fc9f8141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.650+0000 7fc9fd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f815fe60 0x7fc9f8141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.650+0000 7fc9fd400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8160730 con 0x7fc9f815f9d0 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.650+0000 7fc9fd400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8160290 con 0x7fc9f815fe60 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.650+0000 7fc9fd400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f806cb50 con 0x7fc9f8144c00 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.650+0000 7fc9f5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f815f9d0 0x7fc9f8141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f815f9d0 0x7fc9f8141a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52910/0 (socket says 172.21.15.114:52910) 2024-03-06T07:31:17.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 -- 172.21.15.114:0/2619767437 learned_addr learned my addr 172.21.15.114:0/2619767437 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f6800640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f8144c00 0x7fc9f81508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 -- 172.21.15.114:0/2619767437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f8144c00 msgr2=0x7fc9f81508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f8144c00 0x7fc9f81508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 -- 172.21.15.114:0/2619767437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f815fe60 msgr2=0x7fc9f8141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f815fe60 0x7fc9f8141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 -- 172.21.15.114:0/2619767437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f80ae4b0 con 0x7fc9f815f9d0 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9f5e00640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f815f9d0 0x7fc9f8141a10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400c790 tx=0x7fc9e400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9eea00640 1 -- 172.21.15.114:0/2619767437 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4016020 con 0x7fc9f815f9d0 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9eea00640 1 -- 172.21.15.114:0/2619767437 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e4006d50 con 0x7fc9f815f9d0 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f80b2bd0 con 0x7fc9f815f9d0 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.651+0000 7fc9eea00640 1 -- 172.21.15.114:0/2619767437 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4003bb0 con 0x7fc9f815f9d0 2024-03-06T07:31:17.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.652+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f80b3190 con 0x7fc9f815f9d0 2024-03-06T07:31:17.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.653+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f804f330 con 0x7fc9f815f9d0 2024-03-06T07:31:17.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.654+0000 7fc9eea00640 1 -- 172.21.15.114:0/2619767437 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9e400a850 con 0x7fc9f815f9d0 2024-03-06T07:31:17.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.654+0000 7fc9eea00640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d806b4d0 0x7fc9d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.655+0000 7fc9eea00640 1 -- 172.21.15.114:0/2619767437 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc9e4089bc0 con 0x7fc9f815f9d0 2024-03-06T07:31:17.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.655+0000 7fc9f6800640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d806b4d0 0x7fc9d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.655+0000 7fc9f6800640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d806b4d0 0x7fc9d806d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8036000 tx=0x7fc9e803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.659+0000 7fc9eea00640 1 -- 172.21.15.114:0/2619767437 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9e4020020 con 0x7fc9f815f9d0 2024-03-06T07:31:17.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.857+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc9f80b04a0 con 0x7fc9f815f9d0 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.858+0000 7fc9eea00640 1 -- 172.21.15.114:0/2619767437 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v62) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc9f806ee70 con 0x7fc9f815f9d0 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:31:17.861 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:31:17.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d806b4d0 msgr2=0x7fc9d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d806b4d0 0x7fc9d806d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8036000 tx=0x7fc9e803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f815f9d0 msgr2=0x7fc9f8141a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f815f9d0 0x7fc9f8141a10 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400c790 tx=0x7fc9e400cc50 comp rx=0 tx=0).stop 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 shutdown_connections 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f815fe60 0x7fc9f8141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d806b4d0 0x7fc9d806d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f815f9d0 0x7fc9f8141a10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 --2- 172.21.15.114:0/2619767437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f8144c00 0x7fc9f81508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.860+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 >> 172.21.15.114:0/2619767437 conn(0x7fc9f80b1730 msgr2=0x7fc9f8145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.861+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 shutdown_connections 2024-03-06T07:31:17.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:17.861+0000 7fc9fd400640 1 -- 172.21.15.114:0/2619767437 wait complete. 2024-03-06T07:31:17.881 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:31:17.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:31:18.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.086+0000 7f850d400640 1 Processor -- start 2024-03-06T07:31:18.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.087+0000 7f850d400640 1 -- start start 2024-03-06T07:31:18.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.088+0000 7f850d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f85080b5260 0x7f850806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.088+0000 7f850d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8508072b70 0x7f850806e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.088+0000 7f850d400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85080b41c0 con 0x7f8508072790 2024-03-06T07:31:18.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.088+0000 7f850d400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85080b44a0 con 0x7f8508072b70 2024-03-06T07:31:18.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.088+0000 7f850d400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85080b4780 con 0x7f85080b5260 2024-03-06T07:31:18.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8506800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8508072790 0x7f850815eed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36112/0 (socket says 172.21.15.114:36112) 2024-03-06T07:31:18.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8506800640 1 -- 172.21.15.114:0/331922171 learned_addr learned my addr 172.21.15.114:0/331922171 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:18.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8507200640 1 --2- 172.21.15.114:0/331922171 >> v2:172.21.15.114:3300/0 conn(0x7f8508072b70 0x7f850806e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8505e00640 1 --2- 172.21.15.114:0/331922171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f85080b5260 0x7f850806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8505400640 1 -- 172.21.15.114:0/331922171 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175525361 0 0) 0x7f85080b41c0 con 0x7f8508072790 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8505400640 1 -- 172.21.15.114:0/331922171 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ec003520 con 0x7f8508072790 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8507200640 1 -- 172.21.15.114:0/331922171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f85080b5260 msgr2=0x7f850806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8507200640 1 --2- 172.21.15.114:0/331922171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f85080b5260 0x7f850806e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8507200640 1 -- 172.21.15.114:0/331922171 >> v1:172.21.15.114:6789/0 conn(0x7f8508072790 legacy=0x7f850815eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.089+0000 7f8507200640 1 -- 172.21.15.114:0/331922171 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85080b4a60 con 0x7f8508072b70 2024-03-06T07:31:18.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f8507200640 1 --2- 172.21.15.114:0/331922171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8508072b70 0x7f850806e810 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f85000097a0 tx=0x7f850002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8062d777b10a8b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f8505400640 1 -- 172.21.15.114:0/331922171 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500035020 con 0x7f8508072b70 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f8505400640 1 -- 172.21.15.114:0/331922171 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500031830 con 0x7f8508072b70 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f8505400640 1 -- 172.21.15.114:0/331922171 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500031c00 con 0x7f8508072b70 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f850d400640 1 -- 172.21.15.114:0/331922171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8508072b70 msgr2=0x7f850806e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f850d400640 1 --2- 172.21.15.114:0/331922171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8508072b70 0x7f850806e810 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f85000097a0 tx=0x7f850002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f850d400640 1 -- 172.21.15.114:0/331922171 shutdown_connections 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f850d400640 1 --2- 172.21.15.114:0/331922171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8508072b70 0x7f850806e810 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f850d400640 1 --2- 172.21.15.114:0/331922171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f85080b5260 0x7f850806e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.091+0000 7f850d400640 1 -- 172.21.15.114:0/331922171 >> 172.21.15.114:0/331922171 conn(0x7f85080b1c50 msgr2=0x7f85080b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.092+0000 7f850d400640 1 -- 172.21.15.114:0/331922171 shutdown_connections 2024-03-06T07:31:18.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.092+0000 7f850d400640 1 -- 172.21.15.114:0/331922171 wait complete. 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.093+0000 7f850d400640 1 Processor -- start 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.093+0000 7f850d400640 1 -- start start 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.094+0000 7f850d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8508072790 0x7f8508154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.094+0000 7f850d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8508072b70 0x7f850813ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.094+0000 7f850d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f850813bf80 0x7f8508161a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.094+0000 7f850d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850815e6a0 con 0x7f850813bf80 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.094+0000 7f850d400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85080b3e40 con 0x7f8508072b70 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.094+0000 7f850d400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850815e820 con 0x7f8508072790 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f850813bf80 0x7f8508161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f850813bf80 0x7f8508161a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56930/0 (socket says 172.21.15.114:56930) 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 -- 172.21.15.114:0/1614579687 learned_addr learned my addr 172.21.15.114:0/1614579687 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 -- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8508072790 msgr2=0x7f8508154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8506800640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8508072790 0x7f8508154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8505e00640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8508072b70 0x7f850813ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8508072790 0x7f8508154900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 -- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8508072b70 msgr2=0x7f850813ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8508072b70 0x7f850813ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 -- 172.21.15.114:0/1614579687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85081395f0 con 0x7f850813bf80 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.095+0000 7f8507200640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f850813bf80 0x7f8508161a20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f85000097a0 tx=0x7f85000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.096+0000 7f84fea00640 1 -- 172.21.15.114:0/1614579687 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500035040 con 0x7f850813bf80 2024-03-06T07:31:18.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.096+0000 7f84fea00640 1 -- 172.21.15.114:0/1614579687 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850003b840 con 0x7f850813bf80 2024-03-06T07:31:18.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.096+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85081398d0 con 0x7f850813bf80 2024-03-06T07:31:18.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.096+0000 7f84fea00640 1 -- 172.21.15.114:0/1614579687 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850003bc30 con 0x7f850813bf80 2024-03-06T07:31:18.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.096+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8508139e90 con 0x7f850813bf80 2024-03-06T07:31:18.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.097+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850804f330 con 0x7f850813bf80 2024-03-06T07:31:18.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.098+0000 7f84fea00640 1 -- 172.21.15.114:0/1614579687 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8500038020 con 0x7f850813bf80 2024-03-06T07:31:18.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.099+0000 7f84fea00640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f84e406b4d0 0x7f84e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.099+0000 7f84fea00640 1 -- 172.21.15.114:0/1614579687 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f85000abf80 con 0x7f850813bf80 2024-03-06T07:31:18.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.099+0000 7f8506800640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f84e406b4d0 0x7f84e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.100+0000 7f8506800640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f84e406b4d0 0x7f84e406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f84f0000f60 tx=0x7f84f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.103+0000 7f84fea00640 1 -- 172.21.15.114:0/1614579687 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8500045e90 con 0x7f850813bf80 2024-03-06T07:31:18.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:18.298+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f850806f770 con 0x7f850813bf80 2024-03-06T07:31:19.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.011+0000 7f84fea00640 1 -- 172.21.15.114:0/1614579687 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v63) v1 ==== 138+0+0 (secure 0 0 0) 0x7f850007b140 con 0x7f850813bf80 2024-03-06T07:31:19.014 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:31:19.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.016+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f84e406b4d0 msgr2=0x7f84e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.016+0000 7f850d400640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f84e406b4d0 0x7f84e406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f84f0000f60 tx=0x7f84f0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.016+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f850813bf80 msgr2=0x7f8508161a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.016+0000 7f850d400640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f850813bf80 0x7f8508161a20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f85000097a0 tx=0x7f85000046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 shutdown_connections 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f84e406b4d0 0x7f84e406d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f850813bf80 0x7f8508161a20 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8508072b70 0x7f850813ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 --2- 172.21.15.114:0/1614579687 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8508072790 0x7f8508154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 >> 172.21.15.114:0/1614579687 conn(0x7f85080b1c50 msgr2=0x7f850813fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 shutdown_connections 2024-03-06T07:31:19.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.017+0000 7f850d400640 1 -- 172.21.15.114:0/1614579687 wait complete. 2024-03-06T07:31:19.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:31:19.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.249+0000 7f1745800640 1 Processor -- start 2024-03-06T07:31:19.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.250+0000 7f1745800640 1 -- start start 2024-03-06T07:31:19.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.251+0000 7f1745800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f17400d1f20 0x7f17400d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.251+0000 7f1745800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f174015e2d0 0x7f17400d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.251+0000 7f1745800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17400b3190 con 0x7f17400cf240 2024-03-06T07:31:19.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.251+0000 7f1745800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17400d2840 con 0x7f17400d1f20 2024-03-06T07:31:19.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.251+0000 7f1745800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17400cfae0 con 0x7f174015e2d0 2024-03-06T07:31:19.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.252+0000 7f173f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f174015e2d0 0x7f17400d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.252+0000 7f173f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f174015e2d0 0x7f17400d2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:56534/0 (socket says 172.21.15.114:56534) 2024-03-06T07:31:19.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.252+0000 7f173f400640 1 -- 172.21.15.114:0/879242780 learned_addr learned my addr 172.21.15.114:0/879242780 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:19.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.252+0000 7f173fe00640 1 --2- 172.21.15.114:0/879242780 >> v2:172.21.15.177:3300/0 conn(0x7f17400d1f20 0x7f17400d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.253+0000 7f173fe00640 1 -- 172.21.15.114:0/879242780 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f174015e2d0 msgr2=0x7f17400d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.253+0000 7f173fe00640 1 --2- 172.21.15.114:0/879242780 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f174015e2d0 0x7f17400d2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.253+0000 7f173fe00640 1 -- 172.21.15.114:0/879242780 >> v1:172.21.15.177:6789/0 conn(0x7f17400cf240 legacy=0x7f17400d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.253+0000 7f173fe00640 1 -- 172.21.15.114:0/879242780 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17400cf800 con 0x7f17400d1f20 2024-03-06T07:31:19.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.254+0000 7f173fe00640 1 --2- 172.21.15.114:0/879242780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400d1f20 0x7f17400d1860 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1730009d90 tx=0x7f173002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=801ba8f8af785048 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f173ea00640 1 -- 172.21.15.114:0/879242780 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730039070 con 0x7f17400d1f20 2024-03-06T07:31:19.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f173ea00640 1 -- 172.21.15.114:0/879242780 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173002cc60 con 0x7f17400d1f20 2024-03-06T07:31:19.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f173ea00640 1 -- 172.21.15.114:0/879242780 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730034660 con 0x7f17400d1f20 2024-03-06T07:31:19.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f1745800640 1 -- 172.21.15.114:0/879242780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400d1f20 msgr2=0x7f17400d1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f1745800640 1 --2- 172.21.15.114:0/879242780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400d1f20 0x7f17400d1860 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1730009d90 tx=0x7f173002beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f1745800640 1 -- 172.21.15.114:0/879242780 shutdown_connections 2024-03-06T07:31:19.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f1745800640 1 --2- 172.21.15.114:0/879242780 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f174015e2d0 0x7f17400d2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f1745800640 1 --2- 172.21.15.114:0/879242780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400d1f20 0x7f17400d1860 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.255+0000 7f1745800640 1 -- 172.21.15.114:0/879242780 >> 172.21.15.114:0/879242780 conn(0x7f17400d3c90 msgr2=0x7f17400d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.256+0000 7f1745800640 1 -- 172.21.15.114:0/879242780 shutdown_connections 2024-03-06T07:31:19.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.256+0000 7f1745800640 1 -- 172.21.15.114:0/879242780 wait complete. 2024-03-06T07:31:19.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.257+0000 7f1745800640 1 Processor -- start 2024-03-06T07:31:19.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.257+0000 7f1745800640 1 -- start start 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.258+0000 7f1745800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400cf240 0x7f17401f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.258+0000 7f1745800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17400d1f20 0x7f17400b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.258+0000 7f1745800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f174015e2d0 0x7f17400cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.258+0000 7f1745800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17400b4c40 con 0x7f174015e2d0 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.258+0000 7f1745800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17400b3710 con 0x7f17400cf240 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.258+0000 7f1745800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17400b4dc0 con 0x7f17400d1f20 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f174015e2d0 0x7f17400cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f174015e2d0 0x7f17400cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56936/0 (socket says 172.21.15.114:56936) 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 -- 172.21.15.114:0/1199039075 learned_addr learned my addr 172.21.15.114:0/1199039075 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f173f400640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17400d1f20 0x7f17400b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f173fe00640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400cf240 0x7f17401f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 -- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17400d1f20 msgr2=0x7f17400b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17400d1f20 0x7f17400b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 -- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400cf240 msgr2=0x7f17401f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400cf240 0x7f17401f4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 -- 172.21.15.114:0/1199039075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17400b7760 con 0x7f174015e2d0 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f173f400640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17400d1f20 0x7f17400b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.259+0000 7f1744e00640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f174015e2d0 0x7f17400cb1f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1738001eb0 tx=0x7f173800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.260+0000 7f1737e00640 1 -- 172.21.15.114:0/1199039075 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738017070 con 0x7f174015e2d0 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.260+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17400b9740 con 0x7f174015e2d0 2024-03-06T07:31:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.260+0000 7f1737e00640 1 -- 172.21.15.114:0/1199039075 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17380052d0 con 0x7f174015e2d0 2024-03-06T07:31:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.260+0000 7f1737e00640 1 -- 172.21.15.114:0/1199039075 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738003810 con 0x7f174015e2d0 2024-03-06T07:31:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.260+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17400b7a20 con 0x7f174015e2d0 2024-03-06T07:31:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.261+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1740075480 con 0x7f174015e2d0 2024-03-06T07:31:19.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.262+0000 7f1737e00640 1 -- 172.21.15.114:0/1199039075 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f173801a070 con 0x7f174015e2d0 2024-03-06T07:31:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.263+0000 7f1737e00640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f171c06b4d0 0x7f171c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.263+0000 7f1737e00640 1 -- 172.21.15.114:0/1199039075 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f17380894f0 con 0x7f174015e2d0 2024-03-06T07:31:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.264+0000 7f173fe00640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f171c06b4d0 0x7f171c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.264+0000 7f173fe00640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f171c06b4d0 0x7f171c06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1730036000 tx=0x7f1730030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.268+0000 7f1737e00640 1 -- 172.21.15.114:0/1199039075 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1738058430 con 0x7f174015e2d0 2024-03-06T07:31:19.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:19.466+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f174006cb60 con 0x7f174015e2d0 2024-03-06T07:31:20.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.026+0000 7f1737e00640 1 -- 172.21.15.114:0/1199039075 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v64) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1738057d30 con 0x7f174015e2d0 2024-03-06T07:31:20.029 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.029+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f171c06b4d0 msgr2=0x7f171c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.029+0000 7f1745800640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f171c06b4d0 0x7f171c06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1730036000 tx=0x7f1730030000 comp rx=0 tx=0).stop 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.029+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f174015e2d0 msgr2=0x7f17400cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.029+0000 7f1745800640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f174015e2d0 0x7f17400cb1f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1738001eb0 tx=0x7f173800c890 comp rx=0 tx=0).stop 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 shutdown_connections 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f171c06b4d0 0x7f171c06d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f174015e2d0 0x7f17400cb1f0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f17400d1f20 0x7f17400b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 --2- 172.21.15.114:0/1199039075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17400cf240 0x7f17401f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 >> 172.21.15.114:0/1199039075 conn(0x7f17400d3c90 msgr2=0x7f17400b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 shutdown_connections 2024-03-06T07:31:20.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.030+0000 7f1745800640 1 -- 172.21.15.114:0/1199039075 wait complete. 2024-03-06T07:31:20.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:31:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.248+0000 7fcf30c00640 1 Processor -- start 2024-03-06T07:31:20.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.248+0000 7fcf30c00640 1 -- start start 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.249+0000 7fcf30c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcf2c0b3650 0x7fcf2c0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.249+0000 7fcf30c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf2c0cda90 0x7fcf2c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf30c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c0719e0 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf30c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c072310 con 0x7fcf2c0cda90 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf30c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c071cc0 con 0x7fcf2c0b3650 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf2a800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf2c0cda90 0x7fcf2c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf2a800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf2c0cda90 0x7fcf2c06c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56940/0 (socket says 172.21.15.114:56940) 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf2a800640 1 -- 172.21.15.114:0/3677969884 learned_addr learned my addr 172.21.15.114:0/3677969884 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf29e00640 1 --1- 172.21.15.114:0/3677969884 >> v1:172.21.15.114:6789/0 conn(0x7fcf2c0b3df0 0x7fcf2c0b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36118/0 (socket says 172.21.15.114:36118) 2024-03-06T07:31:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf29400640 1 --2- 172.21.15.114:0/3677969884 >> v2:172.21.15.177:3300/0 conn(0x7fcf2c0b3650 0x7fcf2c0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf2a800640 1 -- 172.21.15.114:0/3677969884 >> v2:172.21.15.177:3300/0 conn(0x7fcf2c0b3650 msgr2=0x7fcf2c0727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf2a800640 1 --2- 172.21.15.114:0/3677969884 >> v2:172.21.15.177:3300/0 conn(0x7fcf2c0b3650 0x7fcf2c0727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf2a800640 1 -- 172.21.15.114:0/3677969884 >> v1:172.21.15.114:6789/0 conn(0x7fcf2c0b3df0 legacy=0x7fcf2c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf2a800640 1 -- 172.21.15.114:0/3677969884 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c071fa0 con 0x7fcf2c0cda90 2024-03-06T07:31:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.250+0000 7fcf28a00640 1 -- 172.21.15.114:0/3677969884 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366607083 0 0) 0x7fcf2c0719e0 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.251+0000 7fcf2a800640 1 --2- 172.21.15.114:0/3677969884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0cda90 0x7fcf2c06c990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0b4c90 tx=0x7fcf2402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be675380699bb524 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf28a00640 1 -- 172.21.15.114:0/3677969884 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2402cb20 con 0x7fcf2c0cda90 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf28a00640 1 -- 172.21.15.114:0/3677969884 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2402ccc0 con 0x7fcf2c0cda90 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf28a00640 1 -- 172.21.15.114:0/3677969884 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24034610 con 0x7fcf2c0cda90 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 -- 172.21.15.114:0/3677969884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0cda90 msgr2=0x7fcf2c06c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 --2- 172.21.15.114:0/3677969884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0cda90 0x7fcf2c06c990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0b4c90 tx=0x7fcf2402bf80 comp rx=0 tx=0).stop 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 -- 172.21.15.114:0/3677969884 shutdown_connections 2024-03-06T07:31:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 --2- 172.21.15.114:0/3677969884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0cda90 0x7fcf2c06c990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 --2- 172.21.15.114:0/3677969884 >> v2:172.21.15.177:3300/0 conn(0x7fcf2c0b3650 0x7fcf2c0727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 -- 172.21.15.114:0/3677969884 >> 172.21.15.114:0/3677969884 conn(0x7fcf2c0b1940 msgr2=0x7fcf2c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 -- 172.21.15.114:0/3677969884 shutdown_connections 2024-03-06T07:31:20.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.252+0000 7fcf30c00640 1 -- 172.21.15.114:0/3677969884 wait complete. 2024-03-06T07:31:20.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.254+0000 7fcf30c00640 1 Processor -- start 2024-03-06T07:31:20.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.254+0000 7fcf30c00640 1 -- start start 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.255+0000 7fcf30c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c0b3650 0x7fcf2c149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.255+0000 7fcf30c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0b3df0 0x7fcf2c1f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.255+0000 7fcf30c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf2c0cda90 0x7fcf2c145d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.255+0000 7fcf30c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c15b3c0 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.255+0000 7fcf30c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c15b240 con 0x7fcf2c0b3650 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.255+0000 7fcf30c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0cf510 con 0x7fcf2c0cda90 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.255+0000 7fcf29400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0b3df0 0x7fcf2c1f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0b3df0 0x7fcf2c1f1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56948/0 (socket says 172.21.15.114:56948) 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 -- 172.21.15.114:0/2434739285 learned_addr learned my addr 172.21.15.114:0/2434739285 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29e00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c0b3650 0x7fcf2c149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf2a800640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf2c0cda90 0x7fcf2c145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 -- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf2c0cda90 msgr2=0x7fcf2c145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf2c0cda90 0x7fcf2c145d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 -- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c0b3650 msgr2=0x7fcf2c149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c0b3650 0x7fcf2c149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 -- 172.21.15.114:0/2434739285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c15c4a0 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29e00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c0b3650 0x7fcf2c149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.256+0000 7fcf29400640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0b3df0 0x7fcf2c1f1db0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800d620 tx=0x7fcf1800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.257+0000 7fcf22000640 1 -- 172.21.15.114:0/2434739285 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800b620 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.257+0000 7fcf22000640 1 -- 172.21.15.114:0/2434739285 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18019070 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.257+0000 7fcf22000640 1 -- 172.21.15.114:0/2434739285 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800ba00 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.257+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c15c780 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.257+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c15cd40 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.258+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c0b1760 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.259+0000 7fcf22000640 1 -- 172.21.15.114:0/2434739285 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf1800bba0 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.260+0000 7fcf22000640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf0c06b4d0 0x7fcf0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.260+0000 7fcf22000640 1 -- 172.21.15.114:0/2434739285 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fcf1808b030 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.260+0000 7fcf29e00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf0c06b4d0 0x7fcf0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.261+0000 7fcf29e00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf0c06b4d0 0x7fcf0c06d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c000f60 tx=0x7fcf1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.265+0000 7fcf22000640 1 -- 172.21.15.114:0/2434739285 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf2c0b1760 con 0x7fcf2c0b3df0 2024-03-06T07:31:20.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:20.501+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcf2c04f330 con 0x7fcf2c0b3df0 2024-03-06T07:31:21.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.059+0000 7fcf22000640 1 -- 172.21.15.114:0/2434739285 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v40) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcf18059ff0 con 0x7fcf2c0b3df0 2024-03-06T07:31:21.064 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.065+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf0c06b4d0 msgr2=0x7fcf0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.065+0000 7fcf30c00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf0c06b4d0 0x7fcf0c06d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c000f60 tx=0x7fcf1c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.065+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0b3df0 msgr2=0x7fcf2c1f1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.065+0000 7fcf30c00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0b3df0 0x7fcf2c1f1db0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800d620 tx=0x7fcf1800dae0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.067+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 shutdown_connections 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.067+0000 7fcf30c00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf2c0cda90 0x7fcf2c145d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.067+0000 7fcf30c00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf0c06b4d0 0x7fcf0c06d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.067+0000 7fcf30c00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf2c0b3df0 0x7fcf2c1f1db0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.067+0000 7fcf30c00640 1 --2- 172.21.15.114:0/2434739285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c0b3650 0x7fcf2c149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.067+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 >> 172.21.15.114:0/2434739285 conn(0x7fcf2c0b1940 msgr2=0x7fcf2c0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.067+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 shutdown_connections 2024-03-06T07:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.068+0000 7fcf30c00640 1 -- 172.21.15.114:0/2434739285 wait complete. 2024-03-06T07:31:21.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:21.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.271+0000 7f41d6a00640 1 Processor -- start 2024-03-06T07:31:21.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.271+0000 7f41d6a00640 1 -- start start 2024-03-06T07:31:21.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.273+0000 7f41d6a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d015f9d0 0x7f41d015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.273+0000 7f41d6a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d006cef0 con 0x7f41d0144c00 2024-03-06T07:31:21.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.273+0000 7f41d6a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d006d1d0 con 0x7f41d0070b10 2024-03-06T07:31:21.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.273+0000 7f41d6a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d0160370 con 0x7f41d015f9d0 2024-03-06T07:31:21.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.273+0000 7f41cfe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f41d0144c00 0x7f41d006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36122/0 (socket says 172.21.15.114:36122) 2024-03-06T07:31:21.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.273+0000 7f41cfe00640 1 -- 172.21.15.114:0/3297637976 learned_addr learned my addr 172.21.15.114:0/3297637976 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:21.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.273+0000 7f41c7e00640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d015f9d0 0x7f41d015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524970465 0 0) 0x7f41d006cef0 con 0x7f41d0144c00 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b4003520 con 0x7f41d0144c00 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400088172 0 0) 0x7f41b4003520 con 0x7f41d0144c00 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d015f9d0 msgr2=0x7f41d015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d015f9d0 0x7f41d015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 >> v1:172.21.15.177:6789/0 conn(0x7f41d0070b10 legacy=0x7f41d0070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d006d4b0 con 0x7f41d0144c00 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f41c0001670 con 0x7f41d0144c00 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 >> v1:172.21.15.114:6789/0 conn(0x7f41d0144c00 legacy=0x7f41d006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41b40039e0 0x7f41b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b4004800 0x7f41b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d015f9d0 0x7f41b4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b400ff60 con 0x7f41b40039e0 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4010110 con 0x7f41b4004800 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4010310 con 0x7f41d015f9d0 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.274+0000 7f41cfe00640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41b40039e0 0x7f41b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.275+0000 7f41c7e00640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b4004800 0x7f41b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.275+0000 7f41cfe00640 1 -- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d015f9d0 msgr2=0x7f41b4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.275+0000 7f41cfe00640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d015f9d0 0x7f41b4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.275+0000 7f41cfe00640 1 -- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b4004800 msgr2=0x7f41b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.275+0000 7f41cfe00640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b4004800 0x7f41b4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.275+0000 7f41cfe00640 1 -- 172.21.15.114:0/3297637976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b4003520 con 0x7f41b40039e0 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41cfe00640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41b40039e0 0x7f41b4003e00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f41c000c190 tx=0x7f41b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a177596ebf48ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0007c60 con 0x7f41b40039e0 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c0030070 con 0x7f41b40039e0 2024-03-06T07:31:21.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41cf400640 1 -- 172.21.15.114:0/3297637976 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0005d10 con 0x7f41b40039e0 2024-03-06T07:31:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41d6a00640 1 -- 172.21.15.114:0/3297637976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41b40039e0 msgr2=0x7f41b4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41d6a00640 1 --2- 172.21.15.114:0/3297637976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41b40039e0 0x7f41b4003e00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f41c000c190 tx=0x7f41b4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41d4a00640 1 -- 172.21.15.114:0/3297637976 reap_dead start 2024-03-06T07:31:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41d6a00640 1 -- 172.21.15.114:0/3297637976 shutdown_connections 2024-03-06T07:31:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.276+0000 7f41d6a00640 1 -- 172.21.15.114:0/3297637976 >> 172.21.15.114:0/3297637976 conn(0x7f41d00b1730 msgr2=0x7f41d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.277+0000 7f41d6a00640 1 -- 172.21.15.114:0/3297637976 shutdown_connections 2024-03-06T07:31:21.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.277+0000 7f41d6a00640 1 -- 172.21.15.114:0/3297637976 wait complete. 2024-03-06T07:31:21.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.278+0000 7f41d6a00640 1 Processor -- start 2024-03-06T07:31:21.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.278+0000 7f41d6a00640 1 -- start start 2024-03-06T07:31:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.279+0000 7f41d6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b40039e0 0x7f41d0156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.279+0000 7f41d6a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d0142fe0 0x7f41d00b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.279+0000 7f41d6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41d00b2d60 0x7f41d00ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.279+0000 7f41d6a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d006cb50 con 0x7f41d00b2d60 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.279+0000 7f41d6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d0160730 con 0x7f41b40039e0 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.279+0000 7f41d6a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d006ccd0 con 0x7f41d0142fe0 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.279+0000 7f41d4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41d00b2d60 0x7f41d00ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41d00b2d60 0x7f41d00ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56962/0 (socket says 172.21.15.114:56962) 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 -- 172.21.15.114:0/1679340874 learned_addr learned my addr 172.21.15.114:0/1679340874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 -- 172.21.15.114:0/1679340874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d0142fe0 msgr2=0x7f41d00b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d0142fe0 0x7f41d00b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 -- 172.21.15.114:0/1679340874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b40039e0 msgr2=0x7f41d0156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41cfe00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b40039e0 0x7f41d0156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b40039e0 0x7f41d0156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 -- 172.21.15.114:0/1679340874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d00b3270 con 0x7f41d00b2d60 2024-03-06T07:31:21.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41d4a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41d00b2d60 0x7f41d00ce0d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f41d006e500 tx=0x7f41c800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.280+0000 7f41ccc00640 1 -- 172.21.15.114:0/1679340874 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8017070 con 0x7f41d00b2d60 2024-03-06T07:31:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.281+0000 7f41ccc00640 1 -- 172.21.15.114:0/1679340874 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c8005d60 con 0x7f41d00b2d60 2024-03-06T07:31:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.281+0000 7f41ccc00640 1 -- 172.21.15.114:0/1679340874 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c800a880 con 0x7f41d00b2d60 2024-03-06T07:31:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.281+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d0142500 con 0x7f41d00b2d60 2024-03-06T07:31:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.281+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d0156d80 con 0x7f41d00b2d60 2024-03-06T07:31:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.282+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d004f330 con 0x7f41d00b2d60 2024-03-06T07:31:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.283+0000 7f41ccc00640 1 -- 172.21.15.114:0/1679340874 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41c801a070 con 0x7f41d00b2d60 2024-03-06T07:31:21.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.284+0000 7f41ccc00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f41ac06b500 0x7f41ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.284+0000 7f41ccc00640 1 -- 172.21.15.114:0/1679340874 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41c8089260 con 0x7f41d00b2d60 2024-03-06T07:31:21.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.284+0000 7f41cfe00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f41ac06b500 0x7f41ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.284+0000 7f41cfe00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f41ac06b500 0x7f41ac06d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f41c000c160 tx=0x7f41c0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.288+0000 7f41ccc00640 1 -- 172.21.15.114:0/1679340874 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41c80580e0 con 0x7f41d00b2d60 2024-03-06T07:31:21.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.475+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41d0072890 con 0x7f41d00b2d60 2024-03-06T07:31:21.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.477+0000 7f41ccc00640 1 -- 172.21.15.114:0/1679340874 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f41c80579e0 con 0x7f41d00b2d60 2024-03-06T07:31:21.481 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:21.481 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":65,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:20.504104+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:18.351057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:19.518313+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:21.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f41ac06b500 msgr2=0x7f41ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f41ac06b500 0x7f41ac06d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f41c000c160 tx=0x7f41c0009030 comp rx=0 tx=0).stop 2024-03-06T07:31:21.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41d00b2d60 msgr2=0x7f41d00ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41d00b2d60 0x7f41d00ce0d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f41d006e500 tx=0x7f41c800cb90 comp rx=0 tx=0).stop 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 shutdown_connections 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f41ac06b500 0x7f41ac06d9c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41d00b2d60 0x7f41d00ce0d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f41d0142fe0 0x7f41d00b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 --2- 172.21.15.114:0/1679340874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41b40039e0 0x7f41d0156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.480+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 >> 172.21.15.114:0/1679340874 conn(0x7f41d00b1730 msgr2=0x7f41d0143e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.481+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 shutdown_connections 2024-03-06T07:31:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.481+0000 7f41d6a00640 1 -- 172.21.15.114:0/1679340874 wait complete. 2024-03-06T07:31:21.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.688+0000 7f6935600640 1 Processor -- start 2024-03-06T07:31:21.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.689+0000 7f6935600640 1 -- start start 2024-03-06T07:31:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.690+0000 7f6935600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f69300b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.690+0000 7f6935600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f693006ec00 0x7f69300ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.690+0000 7f6935600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693015b090 con 0x7f69300b4680 2024-03-06T07:31:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.690+0000 7f6935600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69300b2a30 con 0x7f693006ec00 2024-03-06T07:31:21.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.690+0000 7f6935600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b4360 con 0x7f693015acb0 2024-03-06T07:31:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.691+0000 7f692ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f69300b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.691+0000 7f692ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f69300b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:56576/0 (socket says 172.21.15.114:56576) 2024-03-06T07:31:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.691+0000 7f692ea00640 1 -- 172.21.15.114:0/3006474860 learned_addr learned my addr 172.21.15.114:0/3006474860 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:21.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.691+0000 7f692f400640 1 --2- 172.21.15.114:0/3006474860 >> v2:172.21.15.177:3300/0 conn(0x7f693006ec00 0x7f69300ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.692+0000 7f692ea00640 1 -- 172.21.15.114:0/3006474860 >> v2:172.21.15.177:3300/0 conn(0x7f693006ec00 msgr2=0x7f69300ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.692+0000 7f692ea00640 1 --2- 172.21.15.114:0/3006474860 >> v2:172.21.15.177:3300/0 conn(0x7f693006ec00 0x7f69300ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.692+0000 7f692ea00640 1 -- 172.21.15.114:0/3006474860 >> v1:172.21.15.177:6789/0 conn(0x7f69300b4680 legacy=0x7f693006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.692+0000 7f692ea00640 1 -- 172.21.15.114:0/3006474860 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300b2cf0 con 0x7f693015acb0 2024-03-06T07:31:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.692+0000 7f692e000640 1 -- 172.21.15.114:0/3006474860 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951976462 0 0) 0x7f693015b090 con 0x7f69300b4680 2024-03-06T07:31:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.693+0000 7f692ea00640 1 --2- 172.21.15.114:0/3006474860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f69300b37d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6920009870 tx=0x7f692002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4a9beab75a69cc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.693+0000 7f692e000640 1 -- 172.21.15.114:0/3006474860 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920034070 con 0x7f693015acb0 2024-03-06T07:31:21.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.693+0000 7f692e000640 1 -- 172.21.15.114:0/3006474860 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6920030840 con 0x7f693015acb0 2024-03-06T07:31:21.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.693+0000 7f692e000640 1 -- 172.21.15.114:0/3006474860 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920030c10 con 0x7f693015acb0 2024-03-06T07:31:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 -- 172.21.15.114:0/3006474860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 msgr2=0x7f69300b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 --2- 172.21.15.114:0/3006474860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f69300b37d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6920009870 tx=0x7f692002b980 comp rx=0 tx=0).stop 2024-03-06T07:31:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 -- 172.21.15.114:0/3006474860 shutdown_connections 2024-03-06T07:31:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 --2- 172.21.15.114:0/3006474860 >> v2:172.21.15.177:3300/0 conn(0x7f693006ec00 0x7f69300ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 --2- 172.21.15.114:0/3006474860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f69300b37d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 -- 172.21.15.114:0/3006474860 >> 172.21.15.114:0/3006474860 conn(0x7f69300b1730 msgr2=0x7f69300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 -- 172.21.15.114:0/3006474860 shutdown_connections 2024-03-06T07:31:21.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.694+0000 7f6935600640 1 -- 172.21.15.114:0/3006474860 wait complete. 2024-03-06T07:31:21.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.696+0000 7f6935600640 1 Processor -- start 2024-03-06T07:31:21.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.696+0000 7f6935600640 1 -- start start 2024-03-06T07:31:21.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f6935600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f693006ec00 0x7f6930147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f6935600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69300b4680 0x7f693014ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f6935600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f693014d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f6935600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b3d10 con 0x7f693006ec00 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f6935600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b32b0 con 0x7f69300b4680 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f6935600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b4010 con 0x7f693015acb0 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f692ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f693006ec00 0x7f6930147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f692ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f693006ec00 0x7f6930147070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56974/0 (socket says 172.21.15.114:56974) 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.697+0000 7f692ea00640 1 -- 172.21.15.114:0/4187397416 learned_addr learned my addr 172.21.15.114:0/4187397416 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692ea00640 1 -- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 msgr2=0x7f693014d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692f400640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f693014d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f6926000640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69300b4680 0x7f693014ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692ea00640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f693014d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692ea00640 1 -- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69300b4680 msgr2=0x7f693014ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692ea00640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69300b4680 0x7f693014ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692ea00640 1 -- 172.21.15.114:0/4187397416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693014efd0 con 0x7f693006ec00 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692f400640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f693014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f692ea00640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f693006ec00 0x7f6930147070 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f69200090b0 tx=0x7f69200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.698+0000 7f6927400640 1 -- 172.21.15.114:0/4187397416 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920034040 con 0x7f693006ec00 2024-03-06T07:31:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.699+0000 7f6927400640 1 -- 172.21.15.114:0/4187397416 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692003ab90 con 0x7f693006ec00 2024-03-06T07:31:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.699+0000 7f6927400640 1 -- 172.21.15.114:0/4187397416 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920004030 con 0x7f693006ec00 2024-03-06T07:31:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.699+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693014d9f0 con 0x7f693006ec00 2024-03-06T07:31:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.699+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693014dfb0 con 0x7f693006ec00 2024-03-06T07:31:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.700+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693004f330 con 0x7f693006ec00 2024-03-06T07:31:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.701+0000 7f6927400640 1 -- 172.21.15.114:0/4187397416 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6920037070 con 0x7f693006ec00 2024-03-06T07:31:21.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.701+0000 7f6927400640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f690c06b500 0x7f690c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.702+0000 7f6927400640 1 -- 172.21.15.114:0/4187397416 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69200abff0 con 0x7f693006ec00 2024-03-06T07:31:21.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.702+0000 7f6926000640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f690c06b500 0x7f690c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.702+0000 7f6926000640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f690c06b500 0x7f690c06d9c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f69301475b0 tx=0x7f6918009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.706+0000 7f6927400640 1 -- 172.21.15.114:0/4187397416 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f692003f070 con 0x7f693006ec00 2024-03-06T07:31:21.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.885+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6930153510 con 0x7f693006ec00 2024-03-06T07:31:21.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.887+0000 7f6927400640 1 -- 172.21.15.114:0/4187397416 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16729 (secure 0 0 0) 0x7f692007afc0 con 0x7f693006ec00 2024-03-06T07:31:21.890 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:21.890 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":65,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:20.504104+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:18.351057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:19.518313+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f690c06b500 msgr2=0x7f690c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f690c06b500 0x7f690c06d9c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f69301475b0 tx=0x7f6918009040 comp rx=0 tx=0).stop 2024-03-06T07:31:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f693006ec00 msgr2=0x7f6930147070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f693006ec00 0x7f6930147070 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f69200090b0 tx=0x7f69200046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 shutdown_connections 2024-03-06T07:31:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f693015acb0 0x7f693014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69300b4680 0x7f693014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f690c06b500 0x7f690c06d9c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 --2- 172.21.15.114:0/4187397416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f693006ec00 0x7f6930147070 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 >> 172.21.15.114:0/4187397416 conn(0x7f69300b1730 msgr2=0x7f693014f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 shutdown_connections 2024-03-06T07:31:21.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:21.890+0000 7f6935600640 1 -- 172.21.15.114:0/4187397416 wait complete. 2024-03-06T07:31:21.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:31:22.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.123+0000 7f1813000640 1 Processor -- start 2024-03-06T07:31:22.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.123+0000 7f1813000640 1 -- start start 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.125+0000 7f1813000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fc10 0x7f180c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.125+0000 7f1813000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f180c0b3f70 0x7f180c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.125+0000 7f1813000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180c0b46c0 con 0x7f180c15fff0 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.125+0000 7f1813000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180c0b49a0 con 0x7f180c0b3f70 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.125+0000 7f1813000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c072c90 con 0x7f180c15fc10 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.126+0000 7f1803e00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f180c15fff0 0x7f180c160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:60932/0 (socket says 172.21.15.114:60932) 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.126+0000 7f1803e00640 1 -- 172.21.15.114:0/731970174 learned_addr learned my addr 172.21.15.114:0/731970174 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.126+0000 7f180be00640 1 --2- 172.21.15.114:0/731970174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fc10 0x7f180c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.126+0000 7f1811000640 1 --2- 172.21.15.114:0/731970174 >> v2:172.21.15.114:3300/0 conn(0x7f180c0b3f70 0x7f180c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.127+0000 7f1811000640 1 -- 172.21.15.114:0/731970174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fc10 msgr2=0x7f180c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.127+0000 7f1811000640 1 --2- 172.21.15.114:0/731970174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fc10 0x7f180c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.127+0000 7f1811000640 1 -- 172.21.15.114:0/731970174 >> v1:172.21.15.177:6789/0 conn(0x7f180c15fff0 legacy=0x7f180c160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.127+0000 7f1811000640 1 -- 172.21.15.114:0/731970174 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c09dd80 con 0x7f180c0b3f70 2024-03-06T07:31:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.129+0000 7f1811000640 1 --2- 172.21.15.114:0/731970174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c0b3f70 0x7f180c0ce4a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f1804009d90 tx=0x7f180402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b07b274e08d2ada server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.129+0000 7f180b400640 1 -- 172.21.15.114:0/731970174 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804039070 con 0x7f180c0b3f70 2024-03-06T07:31:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.129+0000 7f180b400640 1 -- 172.21.15.114:0/731970174 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180402cc50 con 0x7f180c0b3f70 2024-03-06T07:31:22.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.129+0000 7f180b400640 1 -- 172.21.15.114:0/731970174 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804034440 con 0x7f180c0b3f70 2024-03-06T07:31:22.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 -- 172.21.15.114:0/731970174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c0b3f70 msgr2=0x7f180c0ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 --2- 172.21.15.114:0/731970174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c0b3f70 0x7f180c0ce4a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f1804009d90 tx=0x7f180402bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 -- 172.21.15.114:0/731970174 shutdown_connections 2024-03-06T07:31:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 --2- 172.21.15.114:0/731970174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c0b3f70 0x7f180c0ce4a0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 --2- 172.21.15.114:0/731970174 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fc10 0x7f180c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 -- 172.21.15.114:0/731970174 >> 172.21.15.114:0/731970174 conn(0x7f180c0b1940 msgr2=0x7f180c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 -- 172.21.15.114:0/731970174 shutdown_connections 2024-03-06T07:31:22.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.130+0000 7f1813000640 1 -- 172.21.15.114:0/731970174 wait complete. 2024-03-06T07:31:22.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.132+0000 7f1813000640 1 Processor -- start 2024-03-06T07:31:22.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.132+0000 7f1813000640 1 -- start start 2024-03-06T07:31:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1813000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f180c0b3f70 0x7f180c14a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1813000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c15fc10 0x7f180c06d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1813000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fff0 0x7f180c06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1813000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c15c460 con 0x7f180c15fc10 2024-03-06T07:31:22.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1813000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c15c2e0 con 0x7f180c0b3f70 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1813000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0ce9e0 con 0x7f180c15fff0 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1803e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c15fc10 0x7f180c06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1803e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c15fc10 0x7f180c06d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57004/0 (socket says 172.21.15.114:57004) 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.133+0000 7f1803e00640 1 -- 172.21.15.114:0/3497805476 learned_addr learned my addr 172.21.15.114:0/3497805476 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f180be00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f180c0b3f70 0x7f180c14a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1803e00640 1 -- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fff0 msgr2=0x7f180c06dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1811000640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fff0 0x7f180c06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1803e00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fff0 0x7f180c06dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1803e00640 1 -- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f180c0b3f70 msgr2=0x7f180c14a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1803e00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f180c0b3f70 0x7f180c14a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1803e00640 1 -- 172.21.15.114:0/3497805476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c0c9160 con 0x7f180c15fc10 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f180be00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f180c0b3f70 0x7f180c14a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1803e00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c15fc10 0x7f180c06d860 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c3c0 tx=0x7f17f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1808c00640 1 -- 172.21.15.114:0/3497805476 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4017070 con 0x7f180c15fc10 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1808c00640 1 -- 172.21.15.114:0/3497805476 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f4005ce0 con 0x7f180c15fc10 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180c157d10 con 0x7f180c15fc10 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.134+0000 7f1808c00640 1 -- 172.21.15.114:0/3497805476 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f40072f0 con 0x7f180c15fc10 2024-03-06T07:31:22.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.135+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180c0c9400 con 0x7f180c15fc10 2024-03-06T07:31:22.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.136+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180c04f330 con 0x7f180c15fc10 2024-03-06T07:31:22.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.137+0000 7f1808c00640 1 -- 172.21.15.114:0/3497805476 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17f401a070 con 0x7f180c15fc10 2024-03-06T07:31:22.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.137+0000 7f1808c00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.137+0000 7f1808c00640 1 -- 172.21.15.114:0/3497805476 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17f4089610 con 0x7f180c15fc10 2024-03-06T07:31:22.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.138+0000 7f180be00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.138+0000 7f180be00640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17e806b4d0 0x7f17e806d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009ac0 tx=0x7f17fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.144+0000 7f1808c00640 1 -- 172.21.15.114:0/3497805476 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17f4058480 con 0x7f180c15fc10 2024-03-06T07:31:22.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:22.421+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f180c06ef30 con 0x7f180c15fc10 2024-03-06T07:31:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.079+0000 7f1808c00640 1 -- 172.21.15.114:0/3497805476 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) v1 ==== 118+0+0 (secure 0 0 0) 0x7f17f4057d80 con 0x7f180c15fc10 2024-03-06T07:31:23.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.082+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17e806b4d0 msgr2=0x7f17e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.082+0000 7f1813000640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17e806b4d0 0x7f17e806d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009ac0 tx=0x7f17fc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.082+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c15fc10 msgr2=0x7f180c06d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.082+0000 7f1813000640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c15fc10 0x7f180c06d860 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c3c0 tx=0x7f17f400c880 comp rx=0 tx=0).stop 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 shutdown_connections 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f180c15fff0 0x7f180c06dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f180c15fc10 0x7f180c06d860 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 --2- 172.21.15.114:0/3497805476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f180c0b3f70 0x7f180c14a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 >> 172.21.15.114:0/3497805476 conn(0x7f180c0b1940 msgr2=0x7f180c0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 shutdown_connections 2024-03-06T07:31:23.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.083+0000 7f1813000640 1 -- 172.21.15.114:0/3497805476 wait complete. 2024-03-06T07:31:23.109 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:31:23.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:31:23.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.317+0000 7f1536400640 1 Processor -- start 2024-03-06T07:31:23.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.317+0000 7f1536400640 1 -- start start 2024-03-06T07:31:23.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f1536400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f153015e3f0 0x7f15300d2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f1536400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15300d1060 0x7f15300d2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f1536400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15300d1d00 0x7f15300ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f1536400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300b3ce0 con 0x7f15300d1d00 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f1536400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300d4e00 con 0x7f153015e3f0 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f1536400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300b33b0 con 0x7f15300d1060 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f152fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15300d1d00 0x7f15300ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f152fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15300d1d00 0x7f15300ceac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57016/0 (socket says 172.21.15.114:57016) 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.320+0000 7f152fe00640 1 -- 172.21.15.114:0/4251166332 learned_addr learned my addr 172.21.15.114:0/4251166332 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f1527e00640 1 --2- 172.21.15.114:0/4251166332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15300d1060 0x7f15300d2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f152f400640 1 --2- 172.21.15.114:0/4251166332 >> v2:172.21.15.177:3300/0 conn(0x7f153015e3f0 0x7f15300d2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f152fe00640 1 -- 172.21.15.114:0/4251166332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15300d1060 msgr2=0x7f15300d2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f152fe00640 1 --2- 172.21.15.114:0/4251166332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15300d1060 0x7f15300d2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f152fe00640 1 -- 172.21.15.114:0/4251166332 >> v2:172.21.15.177:3300/0 conn(0x7f153015e3f0 msgr2=0x7f15300d2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f152fe00640 1 --2- 172.21.15.114:0/4251166332 >> v2:172.21.15.177:3300/0 conn(0x7f153015e3f0 0x7f15300d2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f152fe00640 1 -- 172.21.15.114:0/4251166332 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300b3690 con 0x7f15300d1d00 2024-03-06T07:31:23.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.321+0000 7f152f400640 1 --2- 172.21.15.114:0/4251166332 >> v2:172.21.15.177:3300/0 conn(0x7f153015e3f0 0x7f15300d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.322+0000 7f152fe00640 1 --2- 172.21.15.114:0/4251166332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1d00 0x7f15300ceac0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f153008d300 tx=0x7f152802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31985bbf4d0212d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f152ea00640 1 -- 172.21.15.114:0/4251166332 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528035020 con 0x7f15300d1d00 2024-03-06T07:31:23.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f152ea00640 1 -- 172.21.15.114:0/4251166332 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1528031870 con 0x7f15300d1d00 2024-03-06T07:31:23.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f152ea00640 1 -- 172.21.15.114:0/4251166332 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1528031c40 con 0x7f15300d1d00 2024-03-06T07:31:23.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f1536400640 1 -- 172.21.15.114:0/4251166332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1d00 msgr2=0x7f15300ceac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f1536400640 1 --2- 172.21.15.114:0/4251166332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1d00 0x7f15300ceac0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f153008d300 tx=0x7f152802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:23.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f1536400640 1 -- 172.21.15.114:0/4251166332 shutdown_connections 2024-03-06T07:31:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f1536400640 1 --2- 172.21.15.114:0/4251166332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1d00 0x7f15300ceac0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f1536400640 1 --2- 172.21.15.114:0/4251166332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15300d1060 0x7f15300d2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f1536400640 1 --2- 172.21.15.114:0/4251166332 >> v2:172.21.15.177:3300/0 conn(0x7f153015e3f0 0x7f15300d2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.323+0000 7f1536400640 1 -- 172.21.15.114:0/4251166332 >> 172.21.15.114:0/4251166332 conn(0x7f15300d3f20 msgr2=0x7f15300d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.324+0000 7f1536400640 1 -- 172.21.15.114:0/4251166332 shutdown_connections 2024-03-06T07:31:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.324+0000 7f1536400640 1 -- 172.21.15.114:0/4251166332 wait complete. 2024-03-06T07:31:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.326+0000 7f1536400640 1 Processor -- start 2024-03-06T07:31:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.327+0000 7f1536400640 1 -- start start 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f1536400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1060 0x7f15301f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f1536400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15300d1d00 0x7f15301583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f1536400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153015e3f0 0x7f153006cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f1536400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300d4bc0 con 0x7f15300d1060 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f1536400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15300d4a40 con 0x7f15300d1d00 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f1536400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300cf000 con 0x7f153015e3f0 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f152f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1060 0x7f15301f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f152f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1060 0x7f15301f4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57024/0 (socket says 172.21.15.114:57024) 2024-03-06T07:31:23.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.328+0000 7f152f400640 1 -- 172.21.15.114:0/1468619080 learned_addr learned my addr 172.21.15.114:0/1468619080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f152f400640 1 -- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153015e3f0 msgr2=0x7f153006cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f152fe00640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153015e3f0 0x7f153006cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f1527e00640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15300d1d00 0x7f15301583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f152f400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153015e3f0 0x7f153006cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f152f400640 1 -- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15300d1d00 msgr2=0x7f15301583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f152f400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15300d1d00 0x7f15301583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f152f400640 1 -- 172.21.15.114:0/1468619080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1530158c50 con 0x7f15300d1060 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f152f400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1060 0x7f15301f4e60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f152000ce30 tx=0x7f152000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.329+0000 7f1527400640 1 -- 172.21.15.114:0/1468619080 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152000f890 con 0x7f15300d1060 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.330+0000 7f1527400640 1 -- 172.21.15.114:0/1468619080 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152000fa30 con 0x7f15300d1060 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.330+0000 7f1527400640 1 -- 172.21.15.114:0/1468619080 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152000fe00 con 0x7f15300d1060 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.330+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1530158f30 con 0x7f15300d1060 2024-03-06T07:31:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.330+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15301594f0 con 0x7f15300d1060 2024-03-06T07:31:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.331+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153004f330 con 0x7f15300d1060 2024-03-06T07:31:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.332+0000 7f1527400640 1 -- 172.21.15.114:0/1468619080 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1520018400 con 0x7f15300d1060 2024-03-06T07:31:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.332+0000 7f1527400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f150c06b4d0 0x7f150c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.333+0000 7f1527400640 1 -- 172.21.15.114:0/1468619080 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f152008b070 con 0x7f15300d1060 2024-03-06T07:31:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.333+0000 7f1527e00640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f150c06b4d0 0x7f150c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.334+0000 7f1527e00640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f150c06b4d0 0x7f150c06d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f15300d0b70 tx=0x7f1518008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.337+0000 7f1527400640 1 -- 172.21.15.114:0/1468619080 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1520014030 con 0x7f15300d1060 2024-03-06T07:31:23.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:23.696+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f15300d22e0 con 0x7f15300d1060 2024-03-06T07:31:24.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.091+0000 7f1527400640 1 -- 172.21.15.114:0/1468619080 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7f152000ae20 con 0x7f15300d1060 2024-03-06T07:31:24.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f150c06b4d0 msgr2=0x7f150c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f150c06b4d0 0x7f150c06d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f15300d0b70 tx=0x7f1518008040 comp rx=0 tx=0).stop 2024-03-06T07:31:24.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1060 msgr2=0x7f15301f4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1060 0x7f15301f4e60 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f152000ce30 tx=0x7f152000a650 comp rx=0 tx=0).stop 2024-03-06T07:31:24.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 shutdown_connections 2024-03-06T07:31:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f153015e3f0 0x7f153006cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15300d1d00 0x7f15301583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f150c06b4d0 0x7f150c06d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 --2- 172.21.15.114:0/1468619080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15300d1060 0x7f15301f4e60 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.094+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 >> 172.21.15.114:0/1468619080 conn(0x7f15300d3f20 msgr2=0x7f153006ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.095+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 shutdown_connections 2024-03-06T07:31:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.095+0000 7f1536400640 1 -- 172.21.15.114:0/1468619080 wait complete. 2024-03-06T07:31:24.121 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:31:24.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:24.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.326+0000 7f2c7c200640 1 Processor -- start 2024-03-06T07:31:24.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.326+0000 7f2c7c200640 1 -- start start 2024-03-06T07:31:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.327+0000 7f2c7c200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c7c200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2c7415fe30 0x7f2c740ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c7c200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7406ee70 con 0x7f2c7415f9d0 2024-03-06T07:31:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c7c200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7406cef0 con 0x7f2c7415fe30 2024-03-06T07:31:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c7c200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7406d1d0 con 0x7f2c74144c00 2024-03-06T07:31:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c78e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2c7415f9d0 0x7f2c740ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36130/0 (socket says 172.21.15.114:36130) 2024-03-06T07:31:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c78e00640 1 -- 172.21.15.114:0/936695986 learned_addr learned my addr 172.21.15.114:0/936695986 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:24.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c7a200640 1 --2- 172.21.15.114:0/936695986 >> v2:172.21.15.114:3300/0 conn(0x7f2c7415fe30 0x7f2c740ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.328+0000 7f2c79800640 1 --2- 172.21.15.114:0/936695986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.329+0000 7f2c73e00640 1 -- 172.21.15.114:0/936695986 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907582635 0 0) 0x7f2c60001110 con 0x7f2c7415f9d0 2024-03-06T07:31:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.329+0000 7f2c73e00640 1 -- 172.21.15.114:0/936695986 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c5c003520 con 0x7f2c7415f9d0 2024-03-06T07:31:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.329+0000 7f2c7a200640 1 -- 172.21.15.114:0/936695986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 msgr2=0x7f2c7406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.329+0000 7f2c7a200640 1 --2- 172.21.15.114:0/936695986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.329+0000 7f2c7a200640 1 -- 172.21.15.114:0/936695986 >> v1:172.21.15.114:6789/0 conn(0x7f2c7415f9d0 legacy=0x7f2c740ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.329+0000 7f2c7a200640 1 -- 172.21.15.114:0/936695986 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7406d4b0 con 0x7f2c7415fe30 2024-03-06T07:31:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.330+0000 7f2c7a200640 1 --2- 172.21.15.114:0/936695986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce7b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009d90 tx=0x7f2c6c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465452d17ac4c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.330+0000 7f2c73e00640 1 -- 172.21.15.114:0/936695986 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c039070 con 0x7f2c7415fe30 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.330+0000 7f2c73e00640 1 -- 172.21.15.114:0/936695986 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c02cc60 con 0x7f2c7415fe30 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.330+0000 7f2c73e00640 1 -- 172.21.15.114:0/936695986 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c034660 con 0x7f2c7415fe30 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 -- 172.21.15.114:0/936695986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415fe30 msgr2=0x7f2c740ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 --2- 172.21.15.114:0/936695986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce7b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009d90 tx=0x7f2c6c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 -- 172.21.15.114:0/936695986 shutdown_connections 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 --2- 172.21.15.114:0/936695986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce7b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 --2- 172.21.15.114:0/936695986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 -- 172.21.15.114:0/936695986 >> 172.21.15.114:0/936695986 conn(0x7f2c740b1730 msgr2=0x7f2c740b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 -- 172.21.15.114:0/936695986 shutdown_connections 2024-03-06T07:31:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.331+0000 7f2c7c200640 1 -- 172.21.15.114:0/936695986 wait complete. 2024-03-06T07:31:24.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.333+0000 7f2c7c200640 1 Processor -- start 2024-03-06T07:31:24.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.333+0000 7f2c7c200640 1 -- start start 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7c200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7c200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415f9d0 0x7f2c74158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7c200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7c200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74160730 con 0x7f2c7415f9d0 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7c200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74160280 con 0x7f2c7415fe30 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7c200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7406cb50 con 0x7f2c74144c00 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415f9d0 0x7f2c74158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415f9d0 0x7f2c74158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57038/0 (socket says 172.21.15.114:57038) 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 -- 172.21.15.114:0/3745793694 learned_addr learned my addr 172.21.15.114:0/3745793694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c79800640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7a200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 -- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 msgr2=0x7f2c7415c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7415c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 -- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c7415fe30 msgr2=0x7f2c740ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c78e00640 1 -- 172.21.15.114:0/3745793694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7413c930 con 0x7f2c7415f9d0 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.334+0000 7f2c7a200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.335+0000 7f2c78e00640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415f9d0 0x7f2c74158280 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000c760 tx=0x7f2c6000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.335+0000 7f2c71600640 1 -- 172.21.15.114:0/3745793694 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60017070 con 0x7f2c7415f9d0 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.335+0000 7f2c71600640 1 -- 172.21.15.114:0/3745793694 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c60005ad0 con 0x7f2c7415f9d0 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.335+0000 7f2c71600640 1 -- 172.21.15.114:0/3745793694 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60007e20 con 0x7f2c7415f9d0 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.335+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7415cd00 con 0x7f2c7415f9d0 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.335+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7413cbd0 con 0x7f2c7415f9d0 2024-03-06T07:31:24.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.337+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7404f330 con 0x7f2c7415f9d0 2024-03-06T07:31:24.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.338+0000 7f2c71600640 1 -- 172.21.15.114:0/3745793694 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c6001a070 con 0x7f2c7415f9d0 2024-03-06T07:31:24.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.339+0000 7f2c71600640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5406b4d0 0x7f2c5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.340+0000 7f2c71600640 1 -- 172.21.15.114:0/3745793694 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c60089560 con 0x7f2c7415f9d0 2024-03-06T07:31:24.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.340+0000 7f2c79800640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5406b4d0 0x7f2c5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.340+0000 7f2c79800640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5406b4d0 0x7f2c5406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2c68006fd0 tx=0x7f2c68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.344+0000 7f2c71600640 1 -- 172.21.15.114:0/3745793694 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c7406ee70 con 0x7f2c7415f9d0 2024-03-06T07:31:24.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.582+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c7415f7b0 con 0x7f2c7415f9d0 2024-03-06T07:31:24.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.584+0000 7f2c71600640 1 -- 172.21.15.114:0/3745793694 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f2c60058470 con 0x7f2c7415f9d0 2024-03-06T07:31:24.587 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:24.587 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:24.587 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5406b4d0 msgr2=0x7f2c5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5406b4d0 0x7f2c5406d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2c68006fd0 tx=0x7f2c68008040 comp rx=0 tx=0).stop 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415f9d0 msgr2=0x7f2c74158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415f9d0 0x7f2c74158280 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000c760 tx=0x7f2c6000cc20 comp rx=0 tx=0).stop 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 shutdown_connections 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c7415fe30 0x7f2c740ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5406b4d0 0x7f2c5406d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c7415f9d0 0x7f2c74158280 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 --2- 172.21.15.114:0/3745793694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c74144c00 0x7f2c7415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 >> 172.21.15.114:0/3745793694 conn(0x7f2c740b1730 msgr2=0x7f2c74143370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 shutdown_connections 2024-03-06T07:31:24.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.587+0000 7f2c7c200640 1 -- 172.21.15.114:0/3745793694 wait complete. 2024-03-06T07:31:24.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.792+0000 7fd41a400640 1 Processor -- start 2024-03-06T07:31:24.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.792+0000 7fd41a400640 1 -- start start 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.793+0000 7fd41a400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd414161580 0x7fd4140d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.793+0000 7fd41a400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd4140d2a70 0x7fd4140b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.793+0000 7fd41a400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd414161960 con 0x7fd4140d1e90 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.793+0000 7fd41a400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4140cf6f0 con 0x7fd4140d2a70 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.793+0000 7fd41a400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140cf9d0 con 0x7fd414161580 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.794+0000 7fd419000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd414161580 0x7fd4140d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.794+0000 7fd419000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd414161580 0x7fd4140d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:56624/0 (socket says 172.21.15.114:56624) 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.794+0000 7fd419000640 1 -- 172.21.15.114:0/2678292694 learned_addr learned my addr 172.21.15.114:0/2678292694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.794+0000 7fd419a00640 1 --2- 172.21.15.114:0/2678292694 >> v2:172.21.15.177:3300/0 conn(0x7fd4140d2a70 0x7fd4140b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.795+0000 7fd413400640 1 -- 172.21.15.114:0/2678292694 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596550544 0 0) 0x7fd414161960 con 0x7fd4140d1e90 2024-03-06T07:31:24.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.795+0000 7fd419000640 1 -- 172.21.15.114:0/2678292694 >> v2:172.21.15.177:3300/0 conn(0x7fd4140d2a70 msgr2=0x7fd4140b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.795+0000 7fd419000640 1 --2- 172.21.15.114:0/2678292694 >> v2:172.21.15.177:3300/0 conn(0x7fd4140d2a70 0x7fd4140b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.795+0000 7fd419000640 1 -- 172.21.15.114:0/2678292694 >> v1:172.21.15.177:6789/0 conn(0x7fd4140d1e90 legacy=0x7fd4140d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.795+0000 7fd419000640 1 -- 172.21.15.114:0/2678292694 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4140cfcb0 con 0x7fd414161580 2024-03-06T07:31:24.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.796+0000 7fd419000640 1 --2- 172.21.15.114:0/2678292694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd414161580 0x7fd4140d1950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd408009870 tx=0x7fd40802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde166103a65c8d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.797+0000 7fd413400640 1 -- 172.21.15.114:0/2678292694 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408034070 con 0x7fd414161580 2024-03-06T07:31:24.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.797+0000 7fd413400640 1 -- 172.21.15.114:0/2678292694 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd408030840 con 0x7fd414161580 2024-03-06T07:31:24.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.797+0000 7fd413400640 1 -- 172.21.15.114:0/2678292694 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408030c10 con 0x7fd414161580 2024-03-06T07:31:24.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.797+0000 7fd41a400640 1 -- 172.21.15.114:0/2678292694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd414161580 msgr2=0x7fd4140d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.797+0000 7fd41a400640 1 --2- 172.21.15.114:0/2678292694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd414161580 0x7fd4140d1950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd408009870 tx=0x7fd40802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:24.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.798+0000 7fd41a400640 1 -- 172.21.15.114:0/2678292694 shutdown_connections 2024-03-06T07:31:24.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.798+0000 7fd41a400640 1 --2- 172.21.15.114:0/2678292694 >> v2:172.21.15.177:3300/0 conn(0x7fd4140d2a70 0x7fd4140b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.798+0000 7fd41a400640 1 --2- 172.21.15.114:0/2678292694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd414161580 0x7fd4140d1950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.798+0000 7fd41a400640 1 -- 172.21.15.114:0/2678292694 >> 172.21.15.114:0/2678292694 conn(0x7fd4140d3d80 msgr2=0x7fd4140d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.798+0000 7fd41a400640 1 -- 172.21.15.114:0/2678292694 shutdown_connections 2024-03-06T07:31:24.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.798+0000 7fd41a400640 1 -- 172.21.15.114:0/2678292694 wait complete. 2024-03-06T07:31:24.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.799+0000 7fd41a400640 1 Processor -- start 2024-03-06T07:31:24.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.799+0000 7fd41a400640 1 -- start start 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.800+0000 7fd41a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4140d1e90 0x7fd4141f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.800+0000 7fd41a400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4140d2a70 0x7fd414160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.800+0000 7fd41a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd414161580 0x7fd4140b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.800+0000 7fd41a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140cf3a0 con 0x7fd414161580 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.800+0000 7fd41a400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140b39a0 con 0x7fd4140d1e90 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.800+0000 7fd41a400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140b4400 con 0x7fd4140d2a70 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd414161580 0x7fd4140b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd414161580 0x7fd4140b79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57048/0 (socket says 172.21.15.114:57048) 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 -- 172.21.15.114:0/3636197232 learned_addr learned my addr 172.21.15.114:0/3636197232 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd413e00640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4140d2a70 0x7fd414160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419000640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4140d1e90 0x7fd4141f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 -- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4140d2a70 msgr2=0x7fd414160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4140d2a70 0x7fd414160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 -- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4140d1e90 msgr2=0x7fd4141f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4140d1e90 0x7fd4141f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.801+0000 7fd419a00640 1 -- 172.21.15.114:0/3636197232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4140b86b0 con 0x7fd414161580 2024-03-06T07:31:24.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.802+0000 7fd419a00640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd414161580 0x7fd4140b79c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd40c00efc0 tx=0x7fd40c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.802+0000 7fd410c00640 1 -- 172.21.15.114:0/3636197232 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40c018020 con 0x7fd414161580 2024-03-06T07:31:24.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.802+0000 7fd410c00640 1 -- 172.21.15.114:0/3636197232 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40c014480 con 0x7fd414161580 2024-03-06T07:31:24.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.802+0000 7fd410c00640 1 -- 172.21.15.114:0/3636197232 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40c014870 con 0x7fd414161580 2024-03-06T07:31:24.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.802+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4140b8990 con 0x7fd414161580 2024-03-06T07:31:24.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.802+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4140cb380 con 0x7fd414161580 2024-03-06T07:31:24.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.803+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4140cb580 con 0x7fd414161580 2024-03-06T07:31:24.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.804+0000 7fd410c00640 1 -- 172.21.15.114:0/3636197232 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd40c005d10 con 0x7fd414161580 2024-03-06T07:31:24.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.805+0000 7fd410c00640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3f406b4d0 0x7fd3f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.805+0000 7fd410c00640 1 -- 172.21.15.114:0/3636197232 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd40c08ad70 con 0x7fd414161580 2024-03-06T07:31:24.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.805+0000 7fd419000640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3f406b4d0 0x7fd3f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.806+0000 7fd419000640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3f406b4d0 0x7fd3f406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd408036000 tx=0x7fd408008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:24.811+0000 7fd410c00640 1 -- 172.21.15.114:0/3636197232 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd40c022070 con 0x7fd414161580 2024-03-06T07:31:25.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.007+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd41404f330 con 0x7fd414161580 2024-03-06T07:31:25.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.009+0000 7fd410c00640 1 -- 172.21.15.114:0/3636197232 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16716 (secure 0 0 0) 0x7fd40c010040 con 0x7fd414161580 2024-03-06T07:31:25.012 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:25.012 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":66,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:22.058205+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:18.351057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:19.518313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:25.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.011+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3f406b4d0 msgr2=0x7fd3f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.011+0000 7fd41a400640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3f406b4d0 0x7fd3f406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd408036000 tx=0x7fd408008030 comp rx=0 tx=0).stop 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd414161580 msgr2=0x7fd4140b79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd414161580 0x7fd4140b79c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd40c00efc0 tx=0x7fd40c00c490 comp rx=0 tx=0).stop 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 shutdown_connections 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3f406b4d0 0x7fd3f406d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd414161580 0x7fd4140b79c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4140d2a70 0x7fd414160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 --2- 172.21.15.114:0/3636197232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4140d1e90 0x7fd4141f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 >> 172.21.15.114:0/3636197232 conn(0x7fd4140d3d80 msgr2=0x7fd4140b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 shutdown_connections 2024-03-06T07:31:25.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.012+0000 7fd41a400640 1 -- 172.21.15.114:0/3636197232 wait complete. 2024-03-06T07:31:25.033 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:31:25.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:25.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.267+0000 7f1607e00640 1 Processor -- start 2024-03-06T07:31:25.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.267+0000 7f1607e00640 1 -- start start 2024-03-06T07:31:25.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.268+0000 7f1607e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 0x7f16080aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.268+0000 7f1607e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16080cfe20 0x7f16080cd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1607e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160806e1b0 con 0x7f160806dbf0 2024-03-06T07:31:25.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1607e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16080d06a0 con 0x7f16080cfe20 2024-03-06T07:31:25.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1607e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16080cd4f0 con 0x7f1608159cc0 2024-03-06T07:31:25.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1607400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16080cfe20 0x7f16080cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f15ffe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f160806dbf0 0x7f16080cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36134/0 (socket says 172.21.15.114:36134) 2024-03-06T07:31:25.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f15ffe00640 1 -- 172.21.15.114:0/4169118036 learned_addr learned my addr 172.21.15.114:0/4169118036 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:25.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1606a00640 1 --2- 172.21.15.114:0/4169118036 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 0x7f16080aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1607400640 1 -- 172.21.15.114:0/4169118036 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 msgr2=0x7f16080aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1607400640 1 --2- 172.21.15.114:0/4169118036 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 0x7f16080aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.269+0000 7f1607400640 1 -- 172.21.15.114:0/4169118036 >> v1:172.21.15.114:6789/0 conn(0x7f160806dbf0 legacy=0x7f16080cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.270+0000 7f1607400640 1 -- 172.21.15.114:0/4169118036 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16080b4530 con 0x7f16080cfe20 2024-03-06T07:31:25.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.271+0000 7f1607400640 1 --2- 172.21.15.114:0/4169118036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16080cfe20 0x7f16080cd980 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f160002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2682f799a68fc9ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.271+0000 7f1606000640 1 -- 172.21.15.114:0/4169118036 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1600039070 con 0x7f16080cfe20 2024-03-06T07:31:25.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.271+0000 7f1606000640 1 -- 172.21.15.114:0/4169118036 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160002cc50 con 0x7f16080cfe20 2024-03-06T07:31:25.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.271+0000 7f1606000640 1 -- 172.21.15.114:0/4169118036 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1600034440 con 0x7f16080cfe20 2024-03-06T07:31:25.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.271+0000 7f1607e00640 1 -- 172.21.15.114:0/4169118036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16080cfe20 msgr2=0x7f16080cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.271+0000 7f1607e00640 1 --2- 172.21.15.114:0/4169118036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16080cfe20 0x7f16080cd980 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f160002bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.272+0000 7f1607e00640 1 -- 172.21.15.114:0/4169118036 shutdown_connections 2024-03-06T07:31:25.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.272+0000 7f1607e00640 1 --2- 172.21.15.114:0/4169118036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16080cfe20 0x7f16080cd980 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.272+0000 7f1607e00640 1 --2- 172.21.15.114:0/4169118036 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 0x7f16080aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.272+0000 7f1607e00640 1 -- 172.21.15.114:0/4169118036 >> 172.21.15.114:0/4169118036 conn(0x7f160806afc0 msgr2=0x7f160806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.272+0000 7f1607e00640 1 -- 172.21.15.114:0/4169118036 shutdown_connections 2024-03-06T07:31:25.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.272+0000 7f1607e00640 1 -- 172.21.15.114:0/4169118036 wait complete. 2024-03-06T07:31:25.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.273+0000 7f1607e00640 1 Processor -- start 2024-03-06T07:31:25.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.273+0000 7f1607e00640 1 -- start start 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.274+0000 7f1607e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160806dbf0 0x7f16081448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.274+0000 7f1607e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16080cfe20 0x7f16081468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.274+0000 7f1607e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 0x7f16080cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.274+0000 7f1607e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160815a0a0 con 0x7f160806dbf0 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.274+0000 7f1607e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16080b4af0 con 0x7f16080cfe20 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.274+0000 7f1607e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160815a220 con 0x7f1608159cc0 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160806dbf0 0x7f16081448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160806dbf0 0x7f16081448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57070/0 (socket says 172.21.15.114:57070) 2024-03-06T07:31:25.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 -- 172.21.15.114:0/2907477501 learned_addr learned my addr 172.21.15.114:0/2907477501 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:25.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 -- 172.21.15.114:0/2907477501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 msgr2=0x7f16080cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 0x7f16080cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 -- 172.21.15.114:0/2907477501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16080cfe20 msgr2=0x7f16081468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16080cfe20 0x7f16081468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 -- 172.21.15.114:0/2907477501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16081484b0 con 0x7f160806dbf0 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f1606a00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160806dbf0 0x7f16081448a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f15f800d5f0 tx=0x7f15f800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.275+0000 7f15ff400640 1 -- 172.21.15.114:0/2907477501 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80125f0 con 0x7f160806dbf0 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.276+0000 7f15ff400640 1 -- 172.21.15.114:0/2907477501 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8019070 con 0x7f160806dbf0 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.276+0000 7f15ff400640 1 -- 172.21.15.114:0/2907477501 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80129d0 con 0x7f160806dbf0 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.276+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1608148790 con 0x7f160806dbf0 2024-03-06T07:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.276+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1608148d50 con 0x7f160806dbf0 2024-03-06T07:31:25.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.277+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160804f330 con 0x7f160806dbf0 2024-03-06T07:31:25.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.278+0000 7f15ff400640 1 -- 172.21.15.114:0/2907477501 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15f8012b70 con 0x7f160806dbf0 2024-03-06T07:31:25.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.279+0000 7f15ff400640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e406b4d0 0x7f15e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.280+0000 7f15ff400640 1 -- 172.21.15.114:0/2907477501 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15f808a570 con 0x7f160806dbf0 2024-03-06T07:31:25.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.280+0000 7f15ffe00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e406b4d0 0x7f15e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.280+0000 7f15ffe00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e406b4d0 0x7f15e406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f15f0000f60 tx=0x7f15f0007a80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.287+0000 7f15ff400640 1 -- 172.21.15.114:0/2907477501 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15f8059480 con 0x7f160806dbf0 2024-03-06T07:31:25.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.629+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f160806d4c0 con 0x7f15e406b4d0 2024-03-06T07:31:25.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.648+0000 7f15ff400640 1 -- 172.21.15.114:0/2907477501 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252067 (secure 0 0 0) 0x7f160806d4c0 con 0x7f15e406b4d0 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:25.656 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e406b4d0 msgr2=0x7f15e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e406b4d0 0x7f15e406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f15f0000f60 tx=0x7f15f0007a80 comp rx=0 tx=0).stop 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160806dbf0 msgr2=0x7f16081448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160806dbf0 0x7f16081448a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f15f800d5f0 tx=0x7f15f800dab0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 shutdown_connections 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1608159cc0 0x7f16080cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16080cfe20 0x7f16081468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e406b4d0 0x7f15e406d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 --2- 172.21.15.114:0/2907477501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f160806dbf0 0x7f16081448a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.653+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 >> 172.21.15.114:0/2907477501 conn(0x7f160806afc0 msgr2=0x7f16080b05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.654+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 shutdown_connections 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.654+0000 7f1607e00640 1 -- 172.21.15.114:0/2907477501 wait complete. 2024-03-06T07:31:25.675 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-03-06T07:31:25.160937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256992,"kb_used_data":7496,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717728,"statfs":{"total":773094113280,"available":772830953472,"internally_reserved":0,"allocated":7675904,"data_stored":3876452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001287"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132334+0000","last_change":"2024-03-06T07:31:21.085906+0000","last_active":"2024-03-06T07:31:22.132334+0000","last_peered":"2024-03-06T07:31:22.132334+0000","last_clean":"2024-03-06T07:31:22.132334+0000","last_became_active":"2024-03-06T07:31:21.085459+0000","last_became_peered":"2024-03-06T07:31:21.085459+0000","last_unstale":"2024-03-06T07:31:22.132334+0000","last_undegraded":"2024-03-06T07:31:22.132334+0000","last_fullsized":"2024-03-06T07:31:22.132334+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:01.102594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075780+0000","last_change":"2024-03-06T07:31:21.072937+0000","last_active":"2024-03-06T07:31:22.075780+0000","last_peered":"2024-03-06T07:31:22.075780+0000","last_clean":"2024-03-06T07:31:22.075780+0000","last_became_active":"2024-03-06T07:31:21.072539+0000","last_became_peered":"2024-03-06T07:31:21.072539+0000","last_unstale":"2024-03-06T07:31:22.075780+0000","last_undegraded":"2024-03-06T07:31:22.075780+0000","last_fullsized":"2024-03-06T07:31:22.075780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:31.371528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072543+0000","last_change":"2024-03-06T07:31:21.057219+0000","last_active":"2024-03-06T07:31:22.072543+0000","last_peered":"2024-03-06T07:31:22.072543+0000","last_clean":"2024-03-06T07:31:22.072543+0000","last_became_active":"2024-03-06T07:31:21.056836+0000","last_became_peered":"2024-03-06T07:31:21.056836+0000","last_unstale":"2024-03-06T07:31:22.072543+0000","last_undegraded":"2024-03-06T07:31:22.072543+0000","last_fullsized":"2024-03-06T07:31:22.072543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:58.875123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128099+0000","last_change":"2024-03-06T07:31:21.070818+0000","last_active":"2024-03-06T07:31:22.128099+0000","last_peered":"2024-03-06T07:31:22.128099+0000","last_clean":"2024-03-06T07:31:22.128099+0000","last_became_active":"2024-03-06T07:31:21.070495+0000","last_became_peered":"2024-03-06T07:31:21.070495+0000","last_unstale":"2024-03-06T07:31:22.128099+0000","last_undegraded":"2024-03-06T07:31:22.128099+0000","last_fullsized":"2024-03-06T07:31:22.128099+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:13.892288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074570+0000","last_change":"2024-03-06T07:31:21.064066+0000","last_active":"2024-03-06T07:31:22.074570+0000","last_peered":"2024-03-06T07:31:22.074570+0000","last_clean":"2024-03-06T07:31:22.074570+0000","last_became_active":"2024-03-06T07:31:21.063775+0000","last_became_peered":"2024-03-06T07:31:21.063775+0000","last_unstale":"2024-03-06T07:31:22.074570+0000","last_undegraded":"2024-03-06T07:31:22.074570+0000","last_fullsized":"2024-03-06T07:31:22.074570+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:48.507457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074314+0000","last_change":"2024-03-06T07:31:21.071128+0000","last_active":"2024-03-06T07:31:22.074314+0000","last_peered":"2024-03-06T07:31:22.074314+0000","last_clean":"2024-03-06T07:31:22.074314+0000","last_became_active":"2024-03-06T07:31:21.070829+0000","last_became_peered":"2024-03-06T07:31:21.070829+0000","last_unstale":"2024-03-06T07:31:22.074314+0000","last_undegraded":"2024-03-06T07:31:22.074314+0000","last_fullsized":"2024-03-06T07:31:22.074314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:14.734900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130325+0000","last_change":"2024-03-06T07:31:21.066739+0000","last_active":"2024-03-06T07:31:22.130325+0000","last_peered":"2024-03-06T07:31:22.130325+0000","last_clean":"2024-03-06T07:31:22.130325+0000","last_became_active":"2024-03-06T07:31:21.066028+0000","last_became_peered":"2024-03-06T07:31:21.066028+0000","last_unstale":"2024-03-06T07:31:22.130325+0000","last_undegraded":"2024-03-06T07:31:22.130325+0000","last_fullsized":"2024-03-06T07:31:22.130325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:36.454862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074242+0000","last_change":"2024-03-06T07:31:21.064469+0000","last_active":"2024-03-06T07:31:22.074242+0000","last_peered":"2024-03-06T07:31:22.074242+0000","last_clean":"2024-03-06T07:31:22.074242+0000","last_became_active":"2024-03-06T07:31:21.064146+0000","last_became_peered":"2024-03-06T07:31:21.064146+0000","last_unstale":"2024-03-06T07:31:22.074242+0000","last_undegraded":"2024-03-06T07:31:22.074242+0000","last_fullsized":"2024-03-06T07:31:22.074242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:17.499867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074912+0000","last_change":"2024-03-06T07:31:21.070404+0000","last_active":"2024-03-06T07:31:22.074912+0000","last_peered":"2024-03-06T07:31:22.074912+0000","last_clean":"2024-03-06T07:31:22.074912+0000","last_became_active":"2024-03-06T07:31:21.070072+0000","last_became_peered":"2024-03-06T07:31:21.070072+0000","last_unstale":"2024-03-06T07:31:22.074912+0000","last_undegraded":"2024-03-06T07:31:22.074912+0000","last_fullsized":"2024-03-06T07:31:22.074912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:34.044570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134114+0000","last_change":"2024-03-06T07:31:21.082310+0000","last_active":"2024-03-06T07:31:22.134114+0000","last_peered":"2024-03-06T07:31:22.134114+0000","last_clean":"2024-03-06T07:31:22.134114+0000","last_became_active":"2024-03-06T07:31:21.081984+0000","last_became_peered":"2024-03-06T07:31:21.081984+0000","last_unstale":"2024-03-06T07:31:22.134114+0000","last_undegraded":"2024-03-06T07:31:22.134114+0000","last_fullsized":"2024-03-06T07:31:22.134114+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:31.371528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132250+0000","last_change":"2024-03-06T07:31:21.077321+0000","last_active":"2024-03-06T07:31:22.132250+0000","last_peered":"2024-03-06T07:31:22.132250+0000","last_clean":"2024-03-06T07:31:22.132250+0000","last_became_active":"2024-03-06T07:31:21.076973+0000","last_became_peered":"2024-03-06T07:31:21.076973+0000","last_unstale":"2024-03-06T07:31:22.132250+0000","last_undegraded":"2024-03-06T07:31:22.132250+0000","last_fullsized":"2024-03-06T07:31:22.132250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:57.957027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128169+0000","last_change":"2024-03-06T07:31:21.083693+0000","last_active":"2024-03-06T07:31:22.128169+0000","last_peered":"2024-03-06T07:31:22.128169+0000","last_clean":"2024-03-06T07:31:22.128169+0000","last_became_active":"2024-03-06T07:31:21.083375+0000","last_became_peered":"2024-03-06T07:31:21.083375+0000","last_unstale":"2024-03-06T07:31:22.128169+0000","last_undegraded":"2024-03-06T07:31:22.128169+0000","last_fullsized":"2024-03-06T07:31:22.128169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:22.834210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127561+0000","last_change":"2024-03-06T07:31:21.059505+0000","last_active":"2024-03-06T07:31:22.127561+0000","last_peered":"2024-03-06T07:31:22.127561+0000","last_clean":"2024-03-06T07:31:22.127561+0000","last_became_active":"2024-03-06T07:31:21.059128+0000","last_became_peered":"2024-03-06T07:31:21.059128+0000","last_unstale":"2024-03-06T07:31:22.127561+0000","last_undegraded":"2024-03-06T07:31:22.127561+0000","last_fullsized":"2024-03-06T07:31:22.127561+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:18.120622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074391+0000","last_change":"2024-03-06T07:31:21.074713+0000","last_active":"2024-03-06T07:31:22.074391+0000","last_peered":"2024-03-06T07:31:22.074391+0000","last_clean":"2024-03-06T07:31:22.074391+0000","last_became_active":"2024-03-06T07:31:21.074420+0000","last_became_peered":"2024-03-06T07:31:21.074420+0000","last_unstale":"2024-03-06T07:31:22.074391+0000","last_undegraded":"2024-03-06T07:31:22.074391+0000","last_fullsized":"2024-03-06T07:31:22.074391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:52.328983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127475+0000","last_change":"2024-03-06T07:31:21.057982+0000","last_active":"2024-03-06T07:31:22.127475+0000","last_peered":"2024-03-06T07:31:22.127475+0000","last_clean":"2024-03-06T07:31:22.127475+0000","last_became_active":"2024-03-06T07:31:21.057012+0000","last_became_peered":"2024-03-06T07:31:21.057012+0000","last_unstale":"2024-03-06T07:31:22.127475+0000","last_undegraded":"2024-03-06T07:31:22.127475+0000","last_fullsized":"2024-03-06T07:31:22.127475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:51.875945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131610+0000","last_change":"2024-03-06T07:31:21.074696+0000","last_active":"2024-03-06T07:31:22.131610+0000","last_peered":"2024-03-06T07:31:22.131610+0000","last_clean":"2024-03-06T07:31:22.131610+0000","last_became_active":"2024-03-06T07:31:21.074365+0000","last_became_peered":"2024-03-06T07:31:21.074365+0000","last_unstale":"2024-03-06T07:31:22.131610+0000","last_undegraded":"2024-03-06T07:31:22.131610+0000","last_fullsized":"2024-03-06T07:31:22.131610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:50.583417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075051+0000","last_change":"2024-03-06T07:31:21.059534+0000","last_active":"2024-03-06T07:31:22.075051+0000","last_peered":"2024-03-06T07:31:22.075051+0000","last_clean":"2024-03-06T07:31:22.075051+0000","last_became_active":"2024-03-06T07:31:21.059265+0000","last_became_peered":"2024-03-06T07:31:21.059265+0000","last_unstale":"2024-03-06T07:31:22.075051+0000","last_undegraded":"2024-03-06T07:31:22.075051+0000","last_fullsized":"2024-03-06T07:31:22.075051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:16.317285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128848+0000","last_change":"2024-03-06T07:31:21.069478+0000","last_active":"2024-03-06T07:31:22.128848+0000","last_peered":"2024-03-06T07:31:22.128848+0000","last_clean":"2024-03-06T07:31:22.128848+0000","last_became_active":"2024-03-06T07:31:21.069215+0000","last_became_peered":"2024-03-06T07:31:21.069215+0000","last_unstale":"2024-03-06T07:31:22.128848+0000","last_undegraded":"2024-03-06T07:31:22.128848+0000","last_fullsized":"2024-03-06T07:31:22.128848+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:16.691429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074774+0000","last_change":"2024-03-06T07:31:21.065658+0000","last_active":"2024-03-06T07:31:22.074774+0000","last_peered":"2024-03-06T07:31:22.074774+0000","last_clean":"2024-03-06T07:31:22.074774+0000","last_became_active":"2024-03-06T07:31:21.065308+0000","last_became_peered":"2024-03-06T07:31:21.065308+0000","last_unstale":"2024-03-06T07:31:22.074774+0000","last_undegraded":"2024-03-06T07:31:22.074774+0000","last_fullsized":"2024-03-06T07:31:22.074774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:13.803132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127963+0000","last_change":"2024-03-06T07:31:21.068191+0000","last_active":"2024-03-06T07:31:22.127963+0000","last_peered":"2024-03-06T07:31:22.127963+0000","last_clean":"2024-03-06T07:31:22.127963+0000","last_became_active":"2024-03-06T07:31:21.067928+0000","last_became_peered":"2024-03-06T07:31:21.067928+0000","last_unstale":"2024-03-06T07:31:22.127963+0000","last_undegraded":"2024-03-06T07:31:22.127963+0000","last_fullsized":"2024-03-06T07:31:22.127963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:43.982756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131397+0000","last_change":"2024-03-06T07:31:21.074913+0000","last_active":"2024-03-06T07:31:22.131397+0000","last_peered":"2024-03-06T07:31:22.131397+0000","last_clean":"2024-03-06T07:31:22.131397+0000","last_became_active":"2024-03-06T07:31:21.073358+0000","last_became_peered":"2024-03-06T07:31:21.073358+0000","last_unstale":"2024-03-06T07:31:22.131397+0000","last_undegraded":"2024-03-06T07:31:22.131397+0000","last_fullsized":"2024-03-06T07:31:22.131397+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:14.734900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132986+0000","last_change":"2024-03-06T07:31:21.082910+0000","last_active":"2024-03-06T07:31:22.132986+0000","last_peered":"2024-03-06T07:31:22.132986+0000","last_clean":"2024-03-06T07:31:22.132986+0000","last_became_active":"2024-03-06T07:31:21.082612+0000","last_became_peered":"2024-03-06T07:31:21.082612+0000","last_unstale":"2024-03-06T07:31:22.132986+0000","last_undegraded":"2024-03-06T07:31:22.132986+0000","last_fullsized":"2024-03-06T07:31:22.132986+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:54.016506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130987+0000","last_change":"2024-03-06T07:31:21.070797+0000","last_active":"2024-03-06T07:31:22.130987+0000","last_peered":"2024-03-06T07:31:22.130987+0000","last_clean":"2024-03-06T07:31:22.130987+0000","last_became_active":"2024-03-06T07:31:21.070519+0000","last_became_peered":"2024-03-06T07:31:21.070519+0000","last_unstale":"2024-03-06T07:31:22.130987+0000","last_undegraded":"2024-03-06T07:31:22.130987+0000","last_fullsized":"2024-03-06T07:31:22.130987+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:59.352745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130736+0000","last_change":"2024-03-06T07:31:21.075341+0000","last_active":"2024-03-06T07:31:22.130736+0000","last_peered":"2024-03-06T07:31:22.130736+0000","last_clean":"2024-03-06T07:31:22.130736+0000","last_became_active":"2024-03-06T07:31:21.075021+0000","last_became_peered":"2024-03-06T07:31:21.075021+0000","last_unstale":"2024-03-06T07:31:22.130736+0000","last_undegraded":"2024-03-06T07:31:22.130736+0000","last_fullsized":"2024-03-06T07:31:22.130736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:43.982756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134392+0000","last_change":"2024-03-06T07:31:21.077095+0000","last_active":"2024-03-06T07:31:22.134392+0000","last_peered":"2024-03-06T07:31:22.134392+0000","last_clean":"2024-03-06T07:31:22.134392+0000","last_became_active":"2024-03-06T07:31:21.076068+0000","last_became_peered":"2024-03-06T07:31:21.076068+0000","last_unstale":"2024-03-06T07:31:22.134392+0000","last_undegraded":"2024-03-06T07:31:22.134392+0000","last_fullsized":"2024-03-06T07:31:22.134392+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:09.875896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132325+0000","last_change":"2024-03-06T07:31:21.075318+0000","last_active":"2024-03-06T07:31:22.132325+0000","last_peered":"2024-03-06T07:31:22.132325+0000","last_clean":"2024-03-06T07:31:22.132325+0000","last_became_active":"2024-03-06T07:31:21.074971+0000","last_became_peered":"2024-03-06T07:31:21.074971+0000","last_unstale":"2024-03-06T07:31:22.132325+0000","last_undegraded":"2024-03-06T07:31:22.132325+0000","last_fullsized":"2024-03-06T07:31:22.132325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:44.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072716+0000","last_change":"2024-03-06T07:31:21.074573+0000","last_active":"2024-03-06T07:31:22.072716+0000","last_peered":"2024-03-06T07:31:22.072716+0000","last_clean":"2024-03-06T07:31:22.072716+0000","last_became_active":"2024-03-06T07:31:21.074251+0000","last_became_peered":"2024-03-06T07:31:21.074251+0000","last_unstale":"2024-03-06T07:31:22.072716+0000","last_undegraded":"2024-03-06T07:31:22.072716+0000","last_fullsized":"2024-03-06T07:31:22.072716+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:23.355027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133705+0000","last_change":"2024-03-06T07:31:21.084306+0000","last_active":"2024-03-06T07:31:22.133705+0000","last_peered":"2024-03-06T07:31:22.133705+0000","last_clean":"2024-03-06T07:31:22.133705+0000","last_became_active":"2024-03-06T07:31:21.083673+0000","last_became_peered":"2024-03-06T07:31:21.083673+0000","last_unstale":"2024-03-06T07:31:22.133705+0000","last_undegraded":"2024-03-06T07:31:22.133705+0000","last_fullsized":"2024-03-06T07:31:22.133705+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:51.005686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072226+0000","last_change":"2024-03-06T07:31:21.064761+0000","last_active":"2024-03-06T07:31:22.072226+0000","last_peered":"2024-03-06T07:31:22.072226+0000","last_clean":"2024-03-06T07:31:22.072226+0000","last_became_active":"2024-03-06T07:31:21.064390+0000","last_became_peered":"2024-03-06T07:31:21.064390+0000","last_unstale":"2024-03-06T07:31:22.072226+0000","last_undegraded":"2024-03-06T07:31:22.072226+0000","last_fullsized":"2024-03-06T07:31:22.072226+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.052290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133819+0000","last_change":"2024-03-06T07:31:21.062778+0000","last_active":"2024-03-06T07:31:22.133819+0000","last_peered":"2024-03-06T07:31:22.133819+0000","last_clean":"2024-03-06T07:31:22.133819+0000","last_became_active":"2024-03-06T07:31:21.062374+0000","last_became_peered":"2024-03-06T07:31:21.062374+0000","last_unstale":"2024-03-06T07:31:22.133819+0000","last_undegraded":"2024-03-06T07:31:22.133819+0000","last_fullsized":"2024-03-06T07:31:22.133819+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131322+0000","last_change":"2024-03-06T07:31:21.083905+0000","last_active":"2024-03-06T07:31:22.131322+0000","last_peered":"2024-03-06T07:31:22.131322+0000","last_clean":"2024-03-06T07:31:22.131322+0000","last_became_active":"2024-03-06T07:31:21.081704+0000","last_became_peered":"2024-03-06T07:31:21.081704+0000","last_unstale":"2024-03-06T07:31:22.131322+0000","last_undegraded":"2024-03-06T07:31:22.131322+0000","last_fullsized":"2024-03-06T07:31:22.131322+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:27.536304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074764+0000","last_change":"2024-03-06T07:31:21.057192+0000","last_active":"2024-03-06T07:31:22.074764+0000","last_peered":"2024-03-06T07:31:22.074764+0000","last_clean":"2024-03-06T07:31:22.074764+0000","last_became_active":"2024-03-06T07:31:21.055921+0000","last_became_peered":"2024-03-06T07:31:21.055921+0000","last_unstale":"2024-03-06T07:31:22.074764+0000","last_undegraded":"2024-03-06T07:31:22.074764+0000","last_fullsized":"2024-03-06T07:31:22.074764+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:50.583417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132601+0000","last_change":"2024-03-06T07:31:21.081485+0000","last_active":"2024-03-06T07:31:22.132601+0000","last_peered":"2024-03-06T07:31:22.132601+0000","last_clean":"2024-03-06T07:31:22.132601+0000","last_became_active":"2024-03-06T07:31:21.081134+0000","last_became_peered":"2024-03-06T07:31:21.081134+0000","last_unstale":"2024-03-06T07:31:22.132601+0000","last_undegraded":"2024-03-06T07:31:22.132601+0000","last_fullsized":"2024-03-06T07:31:22.132601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:18.106636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"65'4","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133523+0000","last_change":"2024-03-06T07:31:20.063659+0000","last_active":"2024-03-06T07:31:22.133523+0000","last_peered":"2024-03-06T07:31:22.133523+0000","last_clean":"2024-03-06T07:31:22.133523+0000","last_became_active":"2024-03-06T07:31:20.062726+0000","last_became_peered":"2024-03-06T07:31:20.062726+0000","last_unstale":"2024-03-06T07:31:22.133523+0000","last_undegraded":"2024-03-06T07:31:22.133523+0000","last_fullsized":"2024-03-06T07:31:22.133523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:49.893699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075278+0000","last_change":"2024-03-06T07:31:21.066149+0000","last_active":"2024-03-06T07:31:22.075278+0000","last_peered":"2024-03-06T07:31:22.075278+0000","last_clean":"2024-03-06T07:31:22.075278+0000","last_became_active":"2024-03-06T07:31:21.060334+0000","last_became_peered":"2024-03-06T07:31:21.060334+0000","last_unstale":"2024-03-06T07:31:22.075278+0000","last_undegraded":"2024-03-06T07:31:22.075278+0000","last_fullsized":"2024-03-06T07:31:22.075278+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:59.295606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128008+0000","last_change":"2024-03-06T07:31:20.047815+0000","last_active":"2024-03-06T07:31:22.128008+0000","last_peered":"2024-03-06T07:31:22.128008+0000","last_clean":"2024-03-06T07:31:22.128008+0000","last_became_active":"2024-03-06T07:31:20.047436+0000","last_became_peered":"2024-03-06T07:31:20.047436+0000","last_unstale":"2024-03-06T07:31:22.128008+0000","last_undegraded":"2024-03-06T07:31:22.128008+0000","last_fullsized":"2024-03-06T07:31:22.128008+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:08.860785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075608+0000","last_change":"2024-03-06T07:31:21.066378+0000","last_active":"2024-03-06T07:31:22.075608+0000","last_peered":"2024-03-06T07:31:22.075608+0000","last_clean":"2024-03-06T07:31:22.075608+0000","last_became_active":"2024-03-06T07:31:21.065862+0000","last_became_peered":"2024-03-06T07:31:21.065862+0000","last_unstale":"2024-03-06T07:31:22.075608+0000","last_undegraded":"2024-03-06T07:31:22.075608+0000","last_fullsized":"2024-03-06T07:31:22.075608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:51.005686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131787+0000","last_change":"2024-03-06T07:31:20.064805+0000","last_active":"2024-03-06T07:31:22.131787+0000","last_peered":"2024-03-06T07:31:22.131787+0000","last_clean":"2024-03-06T07:31:22.131787+0000","last_became_active":"2024-03-06T07:31:20.063617+0000","last_became_peered":"2024-03-06T07:31:20.063617+0000","last_unstale":"2024-03-06T07:31:22.131787+0000","last_undegraded":"2024-03-06T07:31:22.131787+0000","last_fullsized":"2024-03-06T07:31:22.131787+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:44.861023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132600+0000","last_change":"2024-03-06T07:31:21.075738+0000","last_active":"2024-03-06T07:31:22.132600+0000","last_peered":"2024-03-06T07:31:22.132600+0000","last_clean":"2024-03-06T07:31:22.132600+0000","last_became_active":"2024-03-06T07:31:21.075465+0000","last_became_peered":"2024-03-06T07:31:21.075465+0000","last_unstale":"2024-03-06T07:31:22.132600+0000","last_undegraded":"2024-03-06T07:31:22.132600+0000","last_fullsized":"2024-03-06T07:31:22.132600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:54.601086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.076005+0000","last_change":"2024-03-06T07:31:20.060231+0000","last_active":"2024-03-06T07:31:22.076005+0000","last_peered":"2024-03-06T07:31:22.076005+0000","last_clean":"2024-03-06T07:31:22.076005+0000","last_became_active":"2024-03-06T07:31:20.059932+0000","last_became_peered":"2024-03-06T07:31:20.059932+0000","last_unstale":"2024-03-06T07:31:22.076005+0000","last_undegraded":"2024-03-06T07:31:22.076005+0000","last_fullsized":"2024-03-06T07:31:22.076005+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:44.861023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131153+0000","last_change":"2024-03-06T07:31:21.067967+0000","last_active":"2024-03-06T07:31:22.131153+0000","last_peered":"2024-03-06T07:31:22.131153+0000","last_clean":"2024-03-06T07:31:22.131153+0000","last_became_active":"2024-03-06T07:31:21.067558+0000","last_became_peered":"2024-03-06T07:31:21.067558+0000","last_unstale":"2024-03-06T07:31:22.131153+0000","last_undegraded":"2024-03-06T07:31:22.131153+0000","last_fullsized":"2024-03-06T07:31:22.131153+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:02.154662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072511+0000","last_change":"2024-03-06T07:31:20.062132+0000","last_active":"2024-03-06T07:31:22.072511+0000","last_peered":"2024-03-06T07:31:22.072511+0000","last_clean":"2024-03-06T07:31:22.072511+0000","last_became_active":"2024-03-06T07:31:20.060895+0000","last_became_peered":"2024-03-06T07:31:20.060895+0000","last_unstale":"2024-03-06T07:31:22.072511+0000","last_undegraded":"2024-03-06T07:31:22.072511+0000","last_fullsized":"2024-03-06T07:31:22.072511+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:53.585975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131827+0000","last_change":"2024-03-06T07:31:21.080386+0000","last_active":"2024-03-06T07:31:22.131827+0000","last_peered":"2024-03-06T07:31:22.131827+0000","last_clean":"2024-03-06T07:31:22.131827+0000","last_became_active":"2024-03-06T07:31:21.073882+0000","last_became_peered":"2024-03-06T07:31:21.073882+0000","last_unstale":"2024-03-06T07:31:22.131827+0000","last_undegraded":"2024-03-06T07:31:22.131827+0000","last_fullsized":"2024-03-06T07:31:22.131827+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:13.803132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074882+0000","last_change":"2024-03-06T07:31:20.052215+0000","last_active":"2024-03-06T07:31:22.074882+0000","last_peered":"2024-03-06T07:31:22.074882+0000","last_clean":"2024-03-06T07:31:22.074882+0000","last_became_active":"2024-03-06T07:31:20.051935+0000","last_became_peered":"2024-03-06T07:31:20.051935+0000","last_unstale":"2024-03-06T07:31:22.074882+0000","last_undegraded":"2024-03-06T07:31:22.074882+0000","last_fullsized":"2024-03-06T07:31:22.074882+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:30.244740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.073975+0000","last_change":"2024-03-06T07:31:21.076556+0000","last_active":"2024-03-06T07:31:22.073975+0000","last_peered":"2024-03-06T07:31:22.073975+0000","last_clean":"2024-03-06T07:31:22.073975+0000","last_became_active":"2024-03-06T07:31:21.076275+0000","last_became_peered":"2024-03-06T07:31:21.076275+0000","last_unstale":"2024-03-06T07:31:22.073975+0000","last_undegraded":"2024-03-06T07:31:22.073975+0000","last_fullsized":"2024-03-06T07:31:22.073975+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:24.991953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127744+0000","last_change":"2024-03-06T07:31:20.059231+0000","last_active":"2024-03-06T07:31:22.127744+0000","last_peered":"2024-03-06T07:31:22.127744+0000","last_clean":"2024-03-06T07:31:22.127744+0000","last_became_active":"2024-03-06T07:31:20.058946+0000","last_became_peered":"2024-03-06T07:31:20.058946+0000","last_unstale":"2024-03-06T07:31:22.127744+0000","last_undegraded":"2024-03-06T07:31:22.127744+0000","last_fullsized":"2024-03-06T07:31:22.127744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:18.460944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127630+0000","last_change":"2024-03-06T07:31:21.066893+0000","last_active":"2024-03-06T07:31:22.127630+0000","last_peered":"2024-03-06T07:31:22.127630+0000","last_clean":"2024-03-06T07:31:22.127630+0000","last_became_active":"2024-03-06T07:31:21.065636+0000","last_became_peered":"2024-03-06T07:31:21.065636+0000","last_unstale":"2024-03-06T07:31:22.127630+0000","last_undegraded":"2024-03-06T07:31:22.127630+0000","last_fullsized":"2024-03-06T07:31:22.127630+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:15.795338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.073971+0000","last_change":"2024-03-06T07:31:20.048745+0000","last_active":"2024-03-06T07:31:22.073971+0000","last_peered":"2024-03-06T07:31:22.073971+0000","last_clean":"2024-03-06T07:31:22.073971+0000","last_became_active":"2024-03-06T07:31:20.048441+0000","last_became_peered":"2024-03-06T07:31:20.048441+0000","last_unstale":"2024-03-06T07:31:22.073971+0000","last_undegraded":"2024-03-06T07:31:22.073971+0000","last_fullsized":"2024-03-06T07:31:22.073971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:00.165040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132731+0000","last_change":"2024-03-06T07:31:21.078336+0000","last_active":"2024-03-06T07:31:22.132731+0000","last_peered":"2024-03-06T07:31:22.132731+0000","last_clean":"2024-03-06T07:31:22.132731+0000","last_became_active":"2024-03-06T07:31:21.078019+0000","last_became_peered":"2024-03-06T07:31:21.078019+0000","last_unstale":"2024-03-06T07:31:22.132731+0000","last_undegraded":"2024-03-06T07:31:22.132731+0000","last_fullsized":"2024-03-06T07:31:22.132731+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:01.616697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134120+0000","last_change":"2024-03-06T07:31:20.064730+0000","last_active":"2024-03-06T07:31:22.134120+0000","last_peered":"2024-03-06T07:31:22.134120+0000","last_clean":"2024-03-06T07:31:22.134120+0000","last_became_active":"2024-03-06T07:31:20.063871+0000","last_became_peered":"2024-03-06T07:31:20.063871+0000","last_unstale":"2024-03-06T07:31:22.134120+0000","last_undegraded":"2024-03-06T07:31:22.134120+0000","last_fullsized":"2024-03-06T07:31:22.134120+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:12.877176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132763+0000","last_change":"2024-03-06T07:31:21.083063+0000","last_active":"2024-03-06T07:31:22.132763+0000","last_peered":"2024-03-06T07:31:22.132763+0000","last_clean":"2024-03-06T07:31:22.132763+0000","last_became_active":"2024-03-06T07:31:21.082427+0000","last_became_peered":"2024-03-06T07:31:21.082427+0000","last_unstale":"2024-03-06T07:31:22.132763+0000","last_undegraded":"2024-03-06T07:31:22.132763+0000","last_fullsized":"2024-03-06T07:31:22.132763+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:30.095004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075750+0000","last_change":"2024-03-06T07:31:20.063163+0000","last_active":"2024-03-06T07:31:22.075750+0000","last_peered":"2024-03-06T07:31:22.075750+0000","last_clean":"2024-03-06T07:31:22.075750+0000","last_became_active":"2024-03-06T07:31:20.061717+0000","last_became_peered":"2024-03-06T07:31:20.061717+0000","last_unstale":"2024-03-06T07:31:22.075750+0000","last_undegraded":"2024-03-06T07:31:22.075750+0000","last_fullsized":"2024-03-06T07:31:22.075750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:35.502814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.076097+0000","last_change":"2024-03-06T07:31:21.071252+0000","last_active":"2024-03-06T07:31:22.076097+0000","last_peered":"2024-03-06T07:31:22.076097+0000","last_clean":"2024-03-06T07:31:22.076097+0000","last_became_active":"2024-03-06T07:31:21.070677+0000","last_became_peered":"2024-03-06T07:31:21.070677+0000","last_unstale":"2024-03-06T07:31:22.076097+0000","last_undegraded":"2024-03-06T07:31:22.076097+0000","last_fullsized":"2024-03-06T07:31:22.076097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:09.875896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"65'3","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072831+0000","last_change":"2024-03-06T07:31:20.056117+0000","last_active":"2024-03-06T07:31:22.072831+0000","last_peered":"2024-03-06T07:31:22.072831+0000","last_clean":"2024-03-06T07:31:22.072831+0000","last_became_active":"2024-03-06T07:31:20.055715+0000","last_became_peered":"2024-03-06T07:31:20.055715+0000","last_unstale":"2024-03-06T07:31:22.072831+0000","last_undegraded":"2024-03-06T07:31:22.072831+0000","last_fullsized":"2024-03-06T07:31:22.072831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:17.091524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074361+0000","last_change":"2024-03-06T07:31:21.065295+0000","last_active":"2024-03-06T07:31:22.074361+0000","last_peered":"2024-03-06T07:31:22.074361+0000","last_clean":"2024-03-06T07:31:22.074361+0000","last_became_active":"2024-03-06T07:31:21.064998+0000","last_became_peered":"2024-03-06T07:31:21.064998+0000","last_unstale":"2024-03-06T07:31:22.074361+0000","last_undegraded":"2024-03-06T07:31:22.074361+0000","last_fullsized":"2024-03-06T07:31:22.074361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:41.559348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"65'6","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128343+0000","last_change":"2024-03-06T07:31:20.049330+0000","last_active":"2024-03-06T07:31:22.128343+0000","last_peered":"2024-03-06T07:31:22.128343+0000","last_clean":"2024-03-06T07:31:22.128343+0000","last_became_active":"2024-03-06T07:31:20.049008+0000","last_became_peered":"2024-03-06T07:31:20.049008+0000","last_unstale":"2024-03-06T07:31:22.128343+0000","last_undegraded":"2024-03-06T07:31:22.128343+0000","last_fullsized":"2024-03-06T07:31:22.128343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:43.338550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131195+0000","last_change":"2024-03-06T07:31:21.086442+0000","last_active":"2024-03-06T07:31:22.131195+0000","last_peered":"2024-03-06T07:31:22.131195+0000","last_clean":"2024-03-06T07:31:22.131195+0000","last_became_active":"2024-03-06T07:31:21.085366+0000","last_became_peered":"2024-03-06T07:31:21.085366+0000","last_unstale":"2024-03-06T07:31:22.131195+0000","last_undegraded":"2024-03-06T07:31:22.131195+0000","last_fullsized":"2024-03-06T07:31:22.131195+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:04.425146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127979+0000","last_change":"2024-03-06T07:31:20.057733+0000","last_active":"2024-03-06T07:31:22.127979+0000","last_peered":"2024-03-06T07:31:22.127979+0000","last_clean":"2024-03-06T07:31:22.127979+0000","last_became_active":"2024-03-06T07:31:20.057444+0000","last_became_peered":"2024-03-06T07:31:20.057444+0000","last_unstale":"2024-03-06T07:31:22.127979+0000","last_undegraded":"2024-03-06T07:31:22.127979+0000","last_fullsized":"2024-03-06T07:31:22.127979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:00.373578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133445+0000","last_change":"2024-03-06T07:31:21.078875+0000","last_active":"2024-03-06T07:31:22.133445+0000","last_peered":"2024-03-06T07:31:22.133445+0000","last_clean":"2024-03-06T07:31:22.133445+0000","last_became_active":"2024-03-06T07:31:21.074059+0000","last_became_peered":"2024-03-06T07:31:21.074059+0000","last_unstale":"2024-03-06T07:31:22.133445+0000","last_undegraded":"2024-03-06T07:31:22.133445+0000","last_fullsized":"2024-03-06T07:31:22.133445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:51.875945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074950+0000","last_change":"2024-03-06T07:31:20.053542+0000","last_active":"2024-03-06T07:31:22.074950+0000","last_peered":"2024-03-06T07:31:22.074950+0000","last_clean":"2024-03-06T07:31:22.074950+0000","last_became_active":"2024-03-06T07:31:20.051991+0000","last_became_peered":"2024-03-06T07:31:20.051991+0000","last_unstale":"2024-03-06T07:31:22.074950+0000","last_undegraded":"2024-03-06T07:31:22.074950+0000","last_fullsized":"2024-03-06T07:31:22.074950+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:00.373578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127541+0000","last_change":"2024-03-06T07:31:21.059392+0000","last_active":"2024-03-06T07:31:22.127541+0000","last_peered":"2024-03-06T07:31:22.127541+0000","last_clean":"2024-03-06T07:31:22.127541+0000","last_became_active":"2024-03-06T07:31:21.059089+0000","last_became_peered":"2024-03-06T07:31:21.059089+0000","last_unstale":"2024-03-06T07:31:22.127541+0000","last_undegraded":"2024-03-06T07:31:22.127541+0000","last_fullsized":"2024-03-06T07:31:22.127541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130501+0000","last_change":"2024-03-06T07:31:20.054898+0000","last_active":"2024-03-06T07:31:22.130501+0000","last_peered":"2024-03-06T07:31:22.130501+0000","last_clean":"2024-03-06T07:31:22.130501+0000","last_became_active":"2024-03-06T07:31:20.054535+0000","last_became_peered":"2024-03-06T07:31:20.054535+0000","last_unstale":"2024-03-06T07:31:22.130501+0000","last_undegraded":"2024-03-06T07:31:22.130501+0000","last_fullsized":"2024-03-06T07:31:22.130501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:35.537736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074513+0000","last_change":"2024-03-06T07:31:21.054190+0000","last_active":"2024-03-06T07:31:22.074513+0000","last_peered":"2024-03-06T07:31:22.074513+0000","last_clean":"2024-03-06T07:31:22.074513+0000","last_became_active":"2024-03-06T07:31:21.053700+0000","last_became_peered":"2024-03-06T07:31:21.053700+0000","last_unstale":"2024-03-06T07:31:22.074513+0000","last_undegraded":"2024-03-06T07:31:22.074513+0000","last_fullsized":"2024-03-06T07:31:22.074513+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:16.403218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"65'8","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131891+0000","last_change":"2024-03-06T07:31:20.061201+0000","last_active":"2024-03-06T07:31:22.131891+0000","last_peered":"2024-03-06T07:31:22.131891+0000","last_clean":"2024-03-06T07:31:22.131891+0000","last_became_active":"2024-03-06T07:31:20.060919+0000","last_became_peered":"2024-03-06T07:31:20.060919+0000","last_unstale":"2024-03-06T07:31:22.131891+0000","last_undegraded":"2024-03-06T07:31:22.131891+0000","last_fullsized":"2024-03-06T07:31:22.131891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:53.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134521+0000","last_change":"2024-03-06T07:31:21.070760+0000","last_active":"2024-03-06T07:31:22.134521+0000","last_peered":"2024-03-06T07:31:22.134521+0000","last_clean":"2024-03-06T07:31:22.134521+0000","last_became_active":"2024-03-06T07:31:21.070421+0000","last_became_peered":"2024-03-06T07:31:21.070421+0000","last_unstale":"2024-03-06T07:31:22.134521+0000","last_undegraded":"2024-03-06T07:31:22.134521+0000","last_fullsized":"2024-03-06T07:31:22.134521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:50.908811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075978+0000","last_change":"2024-03-06T07:31:20.060259+0000","last_active":"2024-03-06T07:31:22.075978+0000","last_peered":"2024-03-06T07:31:22.075978+0000","last_clean":"2024-03-06T07:31:22.075978+0000","last_became_active":"2024-03-06T07:31:20.059947+0000","last_became_peered":"2024-03-06T07:31:20.059947+0000","last_unstale":"2024-03-06T07:31:22.075978+0000","last_undegraded":"2024-03-06T07:31:22.075978+0000","last_fullsized":"2024-03-06T07:31:22.075978+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:43.338550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133214+0000","last_change":"2024-03-06T07:31:21.075262+0000","last_active":"2024-03-06T07:31:22.133214+0000","last_peered":"2024-03-06T07:31:22.133214+0000","last_clean":"2024-03-06T07:31:22.133214+0000","last_became_active":"2024-03-06T07:31:21.074848+0000","last_became_peered":"2024-03-06T07:31:21.074848+0000","last_unstale":"2024-03-06T07:31:22.133214+0000","last_undegraded":"2024-03-06T07:31:22.133214+0000","last_fullsized":"2024-03-06T07:31:22.133214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:48.328865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134410+0000","last_change":"2024-03-06T07:31:20.064445+0000","last_active":"2024-03-06T07:31:22.134410+0000","last_peered":"2024-03-06T07:31:22.134410+0000","last_clean":"2024-03-06T07:31:22.134410+0000","last_became_active":"2024-03-06T07:31:20.064168+0000","last_became_peered":"2024-03-06T07:31:20.064168+0000","last_unstale":"2024-03-06T07:31:22.134410+0000","last_undegraded":"2024-03-06T07:31:22.134410+0000","last_fullsized":"2024-03-06T07:31:22.134410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:02.556295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128243+0000","last_change":"2024-03-06T07:31:21.079636+0000","last_active":"2024-03-06T07:31:22.128243+0000","last_peered":"2024-03-06T07:31:22.128243+0000","last_clean":"2024-03-06T07:31:22.128243+0000","last_became_active":"2024-03-06T07:31:21.079384+0000","last_became_peered":"2024-03-06T07:31:21.079384+0000","last_unstale":"2024-03-06T07:31:22.128243+0000","last_undegraded":"2024-03-06T07:31:22.128243+0000","last_fullsized":"2024-03-06T07:31:22.128243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:18.618283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075322+0000","last_change":"2024-03-06T07:31:20.059890+0000","last_active":"2024-03-06T07:31:22.075322+0000","last_peered":"2024-03-06T07:31:22.075322+0000","last_clean":"2024-03-06T07:31:22.075322+0000","last_became_active":"2024-03-06T07:31:20.059612+0000","last_became_peered":"2024-03-06T07:31:20.059612+0000","last_unstale":"2024-03-06T07:31:22.075322+0000","last_undegraded":"2024-03-06T07:31:22.075322+0000","last_fullsized":"2024-03-06T07:31:22.075322+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.918263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074569+0000","last_change":"2024-03-06T07:31:21.064475+0000","last_active":"2024-03-06T07:31:22.074569+0000","last_peered":"2024-03-06T07:31:22.074569+0000","last_clean":"2024-03-06T07:31:22.074569+0000","last_became_active":"2024-03-06T07:31:21.064195+0000","last_became_peered":"2024-03-06T07:31:21.064195+0000","last_unstale":"2024-03-06T07:31:22.074569+0000","last_undegraded":"2024-03-06T07:31:22.074569+0000","last_fullsized":"2024-03-06T07:31:22.074569+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:21.007122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133070+0000","last_change":"2024-03-06T07:31:20.065063+0000","last_active":"2024-03-06T07:31:22.133070+0000","last_peered":"2024-03-06T07:31:22.133070+0000","last_clean":"2024-03-06T07:31:22.133070+0000","last_became_active":"2024-03-06T07:31:20.064566+0000","last_became_peered":"2024-03-06T07:31:20.064566+0000","last_unstale":"2024-03-06T07:31:22.133070+0000","last_undegraded":"2024-03-06T07:31:22.133070+0000","last_fullsized":"2024-03-06T07:31:22.133070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:08.860785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072394+0000","last_change":"2024-03-06T07:31:21.063158+0000","last_active":"2024-03-06T07:31:22.072394+0000","last_peered":"2024-03-06T07:31:22.072394+0000","last_clean":"2024-03-06T07:31:22.072394+0000","last_became_active":"2024-03-06T07:31:21.062893+0000","last_became_peered":"2024-03-06T07:31:21.062893+0000","last_unstale":"2024-03-06T07:31:22.072394+0000","last_undegraded":"2024-03-06T07:31:22.072394+0000","last_fullsized":"2024-03-06T07:31:22.072394+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:07.101394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075146+0000","last_change":"2024-03-06T07:31:20.054569+0000","last_active":"2024-03-06T07:31:22.075146+0000","last_peered":"2024-03-06T07:31:22.075146+0000","last_clean":"2024-03-06T07:31:22.075146+0000","last_became_active":"2024-03-06T07:31:20.054115+0000","last_became_peered":"2024-03-06T07:31:20.054115+0000","last_unstale":"2024-03-06T07:31:22.075146+0000","last_undegraded":"2024-03-06T07:31:22.075146+0000","last_fullsized":"2024-03-06T07:31:22.075146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:53.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072307+0000","last_change":"2024-03-06T07:31:21.056323+0000","last_active":"2024-03-06T07:31:22.072307+0000","last_peered":"2024-03-06T07:31:22.072307+0000","last_clean":"2024-03-06T07:31:22.072307+0000","last_became_active":"2024-03-06T07:31:21.056042+0000","last_became_peered":"2024-03-06T07:31:21.056042+0000","last_unstale":"2024-03-06T07:31:22.072307+0000","last_undegraded":"2024-03-06T07:31:22.072307+0000","last_fullsized":"2024-03-06T07:31:22.072307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:06.914166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130797+0000","last_change":"2024-03-06T07:31:20.050862+0000","last_active":"2024-03-06T07:31:22.130797+0000","last_peered":"2024-03-06T07:31:22.130797+0000","last_clean":"2024-03-06T07:31:22.130797+0000","last_became_active":"2024-03-06T07:31:20.050589+0000","last_became_peered":"2024-03-06T07:31:20.050589+0000","last_unstale":"2024-03-06T07:31:22.130797+0000","last_undegraded":"2024-03-06T07:31:22.130797+0000","last_fullsized":"2024-03-06T07:31:22.130797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:16.484756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072230+0000","last_change":"2024-03-06T07:31:21.056338+0000","last_active":"2024-03-06T07:31:22.072230+0000","last_peered":"2024-03-06T07:31:22.072230+0000","last_clean":"2024-03-06T07:31:22.072230+0000","last_became_active":"2024-03-06T07:31:21.055778+0000","last_became_peered":"2024-03-06T07:31:21.055778+0000","last_unstale":"2024-03-06T07:31:22.072230+0000","last_undegraded":"2024-03-06T07:31:22.072230+0000","last_fullsized":"2024-03-06T07:31:22.072230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:36.206563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127332+0000","last_change":"2024-03-06T07:31:20.050121+0000","last_active":"2024-03-06T07:31:22.127332+0000","last_peered":"2024-03-06T07:31:22.127332+0000","last_clean":"2024-03-06T07:31:22.127332+0000","last_became_active":"2024-03-06T07:31:20.048492+0000","last_became_peered":"2024-03-06T07:31:20.048492+0000","last_unstale":"2024-03-06T07:31:22.127332+0000","last_undegraded":"2024-03-06T07:31:22.127332+0000","last_fullsized":"2024-03-06T07:31:22.127332+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:30.356417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127907+0000","last_change":"2024-03-06T07:31:21.057834+0000","last_active":"2024-03-06T07:31:22.127907+0000","last_peered":"2024-03-06T07:31:22.127907+0000","last_clean":"2024-03-06T07:31:22.127907+0000","last_became_active":"2024-03-06T07:31:21.056626+0000","last_became_peered":"2024-03-06T07:31:21.056626+0000","last_unstale":"2024-03-06T07:31:22.127907+0000","last_undegraded":"2024-03-06T07:31:22.127907+0000","last_fullsized":"2024-03-06T07:31:22.127907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:10.546879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075427+0000","last_change":"2024-03-06T07:31:20.061495+0000","last_active":"2024-03-06T07:31:22.075427+0000","last_peered":"2024-03-06T07:31:22.075427+0000","last_clean":"2024-03-06T07:31:22.075427+0000","last_became_active":"2024-03-06T07:31:20.060383+0000","last_became_peered":"2024-03-06T07:31:20.060383+0000","last_unstale":"2024-03-06T07:31:22.075427+0000","last_undegraded":"2024-03-06T07:31:22.075427+0000","last_fullsized":"2024-03-06T07:31:22.075427+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:10.735755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133831+0000","last_change":"2024-03-06T07:31:21.066729+0000","last_active":"2024-03-06T07:31:22.133831+0000","last_peered":"2024-03-06T07:31:22.133831+0000","last_clean":"2024-03-06T07:31:22.133831+0000","last_became_active":"2024-03-06T07:31:21.066358+0000","last_became_peered":"2024-03-06T07:31:21.066358+0000","last_unstale":"2024-03-06T07:31:22.133831+0000","last_undegraded":"2024-03-06T07:31:22.133831+0000","last_fullsized":"2024-03-06T07:31:22.133831+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:18.120622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"66'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:23.073584+0000","last_change":"2024-03-06T07:31:20.059415+0000","last_active":"2024-03-06T07:31:23.073584+0000","last_peered":"2024-03-06T07:31:23.073584+0000","last_clean":"2024-03-06T07:31:23.073584+0000","last_became_active":"2024-03-06T07:31:20.059067+0000","last_became_peered":"2024-03-06T07:31:20.059067+0000","last_unstale":"2024-03-06T07:31:23.073584+0000","last_undegraded":"2024-03-06T07:31:23.073584+0000","last_fullsized":"2024-03-06T07:31:23.073584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:13.719789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131892+0000","last_change":"2024-03-06T07:31:21.081861+0000","last_active":"2024-03-06T07:31:22.131892+0000","last_peered":"2024-03-06T07:31:22.131892+0000","last_clean":"2024-03-06T07:31:22.131892+0000","last_became_active":"2024-03-06T07:31:21.081178+0000","last_became_peered":"2024-03-06T07:31:21.081178+0000","last_unstale":"2024-03-06T07:31:22.131892+0000","last_undegraded":"2024-03-06T07:31:22.131892+0000","last_fullsized":"2024-03-06T07:31:22.131892+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:46.802945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074731+0000","last_change":"2024-03-06T07:31:20.052615+0000","last_active":"2024-03-06T07:31:22.074731+0000","last_peered":"2024-03-06T07:31:22.074731+0000","last_clean":"2024-03-06T07:31:22.074731+0000","last_became_active":"2024-03-06T07:31:20.051001+0000","last_became_peered":"2024-03-06T07:31:20.051001+0000","last_unstale":"2024-03-06T07:31:22.074731+0000","last_undegraded":"2024-03-06T07:31:22.074731+0000","last_fullsized":"2024-03-06T07:31:22.074731+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:07.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133213+0000","last_change":"2024-03-06T07:31:21.086159+0000","last_active":"2024-03-06T07:31:22.133213+0000","last_peered":"2024-03-06T07:31:22.133213+0000","last_clean":"2024-03-06T07:31:22.133213+0000","last_became_active":"2024-03-06T07:31:21.085619+0000","last_became_peered":"2024-03-06T07:31:21.085619+0000","last_unstale":"2024-03-06T07:31:22.133213+0000","last_undegraded":"2024-03-06T07:31:22.133213+0000","last_fullsized":"2024-03-06T07:31:22.133213+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:34.645483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127745+0000","last_change":"2024-03-06T07:31:20.059873+0000","last_active":"2024-03-06T07:31:22.127745+0000","last_peered":"2024-03-06T07:31:22.127745+0000","last_clean":"2024-03-06T07:31:22.127745+0000","last_became_active":"2024-03-06T07:31:20.059624+0000","last_became_peered":"2024-03-06T07:31:20.059624+0000","last_unstale":"2024-03-06T07:31:22.127745+0000","last_undegraded":"2024-03-06T07:31:22.127745+0000","last_fullsized":"2024-03-06T07:31:22.127745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:49.893699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074461+0000","last_change":"2024-03-06T07:31:21.058580+0000","last_active":"2024-03-06T07:31:22.074461+0000","last_peered":"2024-03-06T07:31:22.074461+0000","last_clean":"2024-03-06T07:31:22.074461+0000","last_became_active":"2024-03-06T07:31:21.058278+0000","last_became_peered":"2024-03-06T07:31:21.058278+0000","last_unstale":"2024-03-06T07:31:22.074461+0000","last_undegraded":"2024-03-06T07:31:22.074461+0000","last_fullsized":"2024-03-06T07:31:22.074461+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:05.943638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074595+0000","last_change":"2024-03-06T07:31:20.053971+0000","last_active":"2024-03-06T07:31:22.074595+0000","last_peered":"2024-03-06T07:31:22.074595+0000","last_clean":"2024-03-06T07:31:22.074595+0000","last_became_active":"2024-03-06T07:31:20.053541+0000","last_became_peered":"2024-03-06T07:31:20.053541+0000","last_unstale":"2024-03-06T07:31:22.074595+0000","last_undegraded":"2024-03-06T07:31:22.074595+0000","last_fullsized":"2024-03-06T07:31:22.074595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.963841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074701+0000","last_change":"2024-03-06T07:31:21.066218+0000","last_active":"2024-03-06T07:31:22.074701+0000","last_peered":"2024-03-06T07:31:22.074701+0000","last_clean":"2024-03-06T07:31:22.074701+0000","last_became_active":"2024-03-06T07:31:21.065378+0000","last_became_peered":"2024-03-06T07:31:21.065378+0000","last_unstale":"2024-03-06T07:31:22.074701+0000","last_undegraded":"2024-03-06T07:31:22.074701+0000","last_fullsized":"2024-03-06T07:31:22.074701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:54.016506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128499+0000","last_change":"2024-03-06T07:31:20.061631+0000","last_active":"2024-03-06T07:31:22.128499+0000","last_peered":"2024-03-06T07:31:22.128499+0000","last_clean":"2024-03-06T07:31:22.128499+0000","last_became_active":"2024-03-06T07:31:20.060893+0000","last_became_peered":"2024-03-06T07:31:20.060893+0000","last_unstale":"2024-03-06T07:31:22.128499+0000","last_undegraded":"2024-03-06T07:31:22.128499+0000","last_fullsized":"2024-03-06T07:31:22.128499+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:26.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127559+0000","last_change":"2024-03-06T07:31:21.060706+0000","last_active":"2024-03-06T07:31:22.127559+0000","last_peered":"2024-03-06T07:31:22.127559+0000","last_clean":"2024-03-06T07:31:22.127559+0000","last_became_active":"2024-03-06T07:31:21.060308+0000","last_became_peered":"2024-03-06T07:31:21.060308+0000","last_unstale":"2024-03-06T07:31:22.127559+0000","last_undegraded":"2024-03-06T07:31:22.127559+0000","last_fullsized":"2024-03-06T07:31:22.127559+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:03.571407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132320+0000","last_change":"2024-03-06T07:31:20.063597+0000","last_active":"2024-03-06T07:31:22.132320+0000","last_peered":"2024-03-06T07:31:22.132320+0000","last_clean":"2024-03-06T07:31:22.132320+0000","last_became_active":"2024-03-06T07:31:20.062770+0000","last_became_peered":"2024-03-06T07:31:20.062770+0000","last_unstale":"2024-03-06T07:31:22.132320+0000","last_undegraded":"2024-03-06T07:31:22.132320+0000","last_fullsized":"2024-03-06T07:31:22.132320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.918263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074008+0000","last_change":"2024-03-06T07:31:21.056252+0000","last_active":"2024-03-06T07:31:22.074008+0000","last_peered":"2024-03-06T07:31:22.074008+0000","last_clean":"2024-03-06T07:31:22.074008+0000","last_became_active":"2024-03-06T07:31:21.055928+0000","last_became_peered":"2024-03-06T07:31:21.055928+0000","last_unstale":"2024-03-06T07:31:22.074008+0000","last_undegraded":"2024-03-06T07:31:22.074008+0000","last_fullsized":"2024-03-06T07:31:22.074008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:36.552847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075024+0000","last_change":"2024-03-06T07:31:20.053091+0000","last_active":"2024-03-06T07:31:22.075024+0000","last_peered":"2024-03-06T07:31:22.075024+0000","last_clean":"2024-03-06T07:31:22.075024+0000","last_became_active":"2024-03-06T07:31:20.052758+0000","last_became_peered":"2024-03-06T07:31:20.052758+0000","last_unstale":"2024-03-06T07:31:22.075024+0000","last_undegraded":"2024-03-06T07:31:22.075024+0000","last_fullsized":"2024-03-06T07:31:22.075024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:26.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072257+0000","last_change":"2024-03-06T07:31:21.058169+0000","last_active":"2024-03-06T07:31:22.072257+0000","last_peered":"2024-03-06T07:31:22.072257+0000","last_clean":"2024-03-06T07:31:22.072257+0000","last_became_active":"2024-03-06T07:31:21.057552+0000","last_became_peered":"2024-03-06T07:31:21.057552+0000","last_unstale":"2024-03-06T07:31:22.072257+0000","last_undegraded":"2024-03-06T07:31:22.072257+0000","last_fullsized":"2024-03-06T07:31:22.072257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:32.727509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074916+0000","last_change":"2024-03-06T07:31:20.054533+0000","last_active":"2024-03-06T07:31:22.074916+0000","last_peered":"2024-03-06T07:31:22.074916+0000","last_clean":"2024-03-06T07:31:22.074916+0000","last_became_active":"2024-03-06T07:31:20.054264+0000","last_became_peered":"2024-03-06T07:31:20.054264+0000","last_unstale":"2024-03-06T07:31:22.074916+0000","last_undegraded":"2024-03-06T07:31:22.074916+0000","last_fullsized":"2024-03-06T07:31:22.074916+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:18.460944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":253,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130893+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:31:22.130893+0000","last_peered":"2024-03-06T07:31:22.130893+0000","last_clean":"2024-03-06T07:31:22.130893+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:31:22.130893+0000","last_undegraded":"2024-03-06T07:31:22.130893+0000","last_fullsized":"2024-03-06T07:31:22.130893+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":927346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":336454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":336452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":336432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":336474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574895,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":337974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":823296,"data_stored":338590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1396736,"data_stored":926730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:25.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:25.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.919+0000 7f31cc200640 1 Processor -- start 2024-03-06T07:31:25.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.919+0000 7f31cc200640 1 -- start start 2024-03-06T07:31:25.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31cc200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c4144c00 0x7f31c406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31cc200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f31c415f9d0 0x7f31c415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31cc200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f31c40ce0a0 0x7f31c40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31cc200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c4160370 con 0x7f31c415f9d0 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31cc200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c406ee70 con 0x7f31c40ce0a0 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31cc200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c406cef0 con 0x7f31c4144c00 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31c8e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f31c415f9d0 0x7f31c415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31c8e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f31c415f9d0 0x7f31c415fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57086/0 (socket says 172.21.15.114:57086) 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.921+0000 7f31c8e00640 1 -- 172.21.15.114:0/1986359528 learned_addr learned my addr 172.21.15.114:0/1986359528 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.922+0000 7f31c9800640 1 --2- 172.21.15.114:0/1986359528 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c4144c00 0x7f31c406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.922+0000 7f31ca200640 1 --2- 172.21.15.114:0/1986359528 >> v2:172.21.15.177:3300/0 conn(0x7f31c40ce0a0 0x7f31c40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.922+0000 7f31c8e00640 1 -- 172.21.15.114:0/1986359528 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c4144c00 msgr2=0x7f31c406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.922+0000 7f31c8e00640 1 --2- 172.21.15.114:0/1986359528 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c4144c00 0x7f31c406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.922+0000 7f31c8e00640 1 -- 172.21.15.114:0/1986359528 >> v2:172.21.15.177:3300/0 conn(0x7f31c40ce0a0 msgr2=0x7f31c40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.922+0000 7f31c8e00640 1 --2- 172.21.15.114:0/1986359528 >> v2:172.21.15.177:3300/0 conn(0x7f31c40ce0a0 0x7f31c40ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.922+0000 7f31c8e00640 1 -- 172.21.15.114:0/1986359528 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c406d1d0 con 0x7f31c415f9d0 2024-03-06T07:31:25.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.924+0000 7f31c8e00640 1 --2- 172.21.15.114:0/1986359528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c415f9d0 0x7f31c415fe30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f31c40737b0 tx=0x7f31b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec13f3de73bc25e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.924+0000 7f31c3e00640 1 -- 172.21.15.114:0/1986359528 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0035020 con 0x7f31c415f9d0 2024-03-06T07:31:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.924+0000 7f31c3e00640 1 -- 172.21.15.114:0/1986359528 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b0031870 con 0x7f31c415f9d0 2024-03-06T07:31:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.924+0000 7f31c3e00640 1 -- 172.21.15.114:0/1986359528 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0031c40 con 0x7f31c415f9d0 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 -- 172.21.15.114:0/1986359528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c415f9d0 msgr2=0x7f31c415fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 --2- 172.21.15.114:0/1986359528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c415f9d0 0x7f31c415fe30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f31c40737b0 tx=0x7f31b002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 -- 172.21.15.114:0/1986359528 shutdown_connections 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 --2- 172.21.15.114:0/1986359528 >> v2:172.21.15.177:3300/0 conn(0x7f31c40ce0a0 0x7f31c40ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 --2- 172.21.15.114:0/1986359528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c415f9d0 0x7f31c415fe30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 --2- 172.21.15.114:0/1986359528 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c4144c00 0x7f31c406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 -- 172.21.15.114:0/1986359528 >> 172.21.15.114:0/1986359528 conn(0x7f31c40b1730 msgr2=0x7f31c40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 -- 172.21.15.114:0/1986359528 shutdown_connections 2024-03-06T07:31:25.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.925+0000 7f31cc200640 1 -- 172.21.15.114:0/1986359528 wait complete. 2024-03-06T07:31:25.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.927+0000 7f31cc200640 1 Processor -- start 2024-03-06T07:31:25.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.927+0000 7f31cc200640 1 -- start start 2024-03-06T07:31:25.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31cc200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c40ce0a0 0x7f31c40b3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31cc200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c4144c00 0x7f31c40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31cc200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c415f9d0 0x7f31c415add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31cc200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4160730 con 0x7f31c4144c00 2024-03-06T07:31:25.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31cc200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c406d790 con 0x7f31c40ce0a0 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31cc200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c406cb50 con 0x7f31c415f9d0 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c4144c00 0x7f31c40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c4144c00 0x7f31c40b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57094/0 (socket says 172.21.15.114:57094) 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 -- 172.21.15.114:0/2240463193 learned_addr learned my addr 172.21.15.114:0/2240463193 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c9800640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c40ce0a0 0x7f31c40b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31ca200640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c415f9d0 0x7f31c415add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 -- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c415f9d0 msgr2=0x7f31c415add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c415f9d0 0x7f31c415add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 -- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c40ce0a0 msgr2=0x7f31c40b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c40ce0a0 0x7f31c40b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.928+0000 7f31c8e00640 1 -- 172.21.15.114:0/2240463193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c415f330 con 0x7f31c4144c00 2024-03-06T07:31:25.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.929+0000 7f31c8e00640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c4144c00 0x7f31c40b3db0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f31b002b950 tx=0x7f31b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.929+0000 7f31c1600640 1 -- 172.21.15.114:0/2240463193 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0041070 con 0x7f31c4144c00 2024-03-06T07:31:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.929+0000 7f31c1600640 1 -- 172.21.15.114:0/2240463193 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b003ac80 con 0x7f31c4144c00 2024-03-06T07:31:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.929+0000 7f31c1600640 1 -- 172.21.15.114:0/2240463193 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0035040 con 0x7f31c4144c00 2024-03-06T07:31:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.929+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c413c130 con 0x7f31c4144c00 2024-03-06T07:31:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.929+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c413cba0 con 0x7f31c4144c00 2024-03-06T07:31:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.930+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c404f330 con 0x7f31c4144c00 2024-03-06T07:31:25.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.932+0000 7f31c1600640 1 -- 172.21.15.114:0/2240463193 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31b004c060 con 0x7f31c4144c00 2024-03-06T07:31:25.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.932+0000 7f31c1600640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f31a406b4d0 0x7f31a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.933+0000 7f31c1600640 1 -- 172.21.15.114:0/2240463193 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31b00ac070 con 0x7f31c4144c00 2024-03-06T07:31:25.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.933+0000 7f31c9800640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f31a406b4d0 0x7f31a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.933+0000 7f31c9800640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f31a406b4d0 0x7f31a406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f31b8006fd0 tx=0x7f31b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:25.938+0000 7f31c1600640 1 -- 172.21.15.114:0/2240463193 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31b002f320 con 0x7f31c4144c00 2024-03-06T07:31:26.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.131+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31c413c410 con 0x7f31a406b4d0 2024-03-06T07:31:26.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.141+0000 7f31c1600640 1 -- 172.21.15.114:0/2240463193 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252067 (secure 0 0 0) 0x7f31c413c410 con 0x7f31a406b4d0 2024-03-06T07:31:26.148 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:26.149 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f31a406b4d0 msgr2=0x7f31a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f31a406b4d0 0x7f31a406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f31b8006fd0 tx=0x7f31b8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c4144c00 msgr2=0x7f31c40b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c4144c00 0x7f31c40b3db0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f31b002b950 tx=0x7f31b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 shutdown_connections 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f31c415f9d0 0x7f31c415add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f31a406b4d0 0x7f31a406d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31c4144c00 0x7f31c40b3db0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 --2- 172.21.15.114:0/2240463193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31c40ce0a0 0x7f31c40b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 >> 172.21.15.114:0/2240463193 conn(0x7f31c40b1730 msgr2=0x7f31c40b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 shutdown_connections 2024-03-06T07:31:26.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.147+0000 7f31cc200640 1 -- 172.21.15.114:0/2240463193 wait complete. 2024-03-06T07:31:26.167 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-03-06T07:31:25.160937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256992,"kb_used_data":7496,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717728,"statfs":{"total":773094113280,"available":772830953472,"internally_reserved":0,"allocated":7675904,"data_stored":3876452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001287"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132334+0000","last_change":"2024-03-06T07:31:21.085906+0000","last_active":"2024-03-06T07:31:22.132334+0000","last_peered":"2024-03-06T07:31:22.132334+0000","last_clean":"2024-03-06T07:31:22.132334+0000","last_became_active":"2024-03-06T07:31:21.085459+0000","last_became_peered":"2024-03-06T07:31:21.085459+0000","last_unstale":"2024-03-06T07:31:22.132334+0000","last_undegraded":"2024-03-06T07:31:22.132334+0000","last_fullsized":"2024-03-06T07:31:22.132334+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:01.102594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075780+0000","last_change":"2024-03-06T07:31:21.072937+0000","last_active":"2024-03-06T07:31:22.075780+0000","last_peered":"2024-03-06T07:31:22.075780+0000","last_clean":"2024-03-06T07:31:22.075780+0000","last_became_active":"2024-03-06T07:31:21.072539+0000","last_became_peered":"2024-03-06T07:31:21.072539+0000","last_unstale":"2024-03-06T07:31:22.075780+0000","last_undegraded":"2024-03-06T07:31:22.075780+0000","last_fullsized":"2024-03-06T07:31:22.075780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:31.371528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072543+0000","last_change":"2024-03-06T07:31:21.057219+0000","last_active":"2024-03-06T07:31:22.072543+0000","last_peered":"2024-03-06T07:31:22.072543+0000","last_clean":"2024-03-06T07:31:22.072543+0000","last_became_active":"2024-03-06T07:31:21.056836+0000","last_became_peered":"2024-03-06T07:31:21.056836+0000","last_unstale":"2024-03-06T07:31:22.072543+0000","last_undegraded":"2024-03-06T07:31:22.072543+0000","last_fullsized":"2024-03-06T07:31:22.072543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:58.875123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128099+0000","last_change":"2024-03-06T07:31:21.070818+0000","last_active":"2024-03-06T07:31:22.128099+0000","last_peered":"2024-03-06T07:31:22.128099+0000","last_clean":"2024-03-06T07:31:22.128099+0000","last_became_active":"2024-03-06T07:31:21.070495+0000","last_became_peered":"2024-03-06T07:31:21.070495+0000","last_unstale":"2024-03-06T07:31:22.128099+0000","last_undegraded":"2024-03-06T07:31:22.128099+0000","last_fullsized":"2024-03-06T07:31:22.128099+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:13.892288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074570+0000","last_change":"2024-03-06T07:31:21.064066+0000","last_active":"2024-03-06T07:31:22.074570+0000","last_peered":"2024-03-06T07:31:22.074570+0000","last_clean":"2024-03-06T07:31:22.074570+0000","last_became_active":"2024-03-06T07:31:21.063775+0000","last_became_peered":"2024-03-06T07:31:21.063775+0000","last_unstale":"2024-03-06T07:31:22.074570+0000","last_undegraded":"2024-03-06T07:31:22.074570+0000","last_fullsized":"2024-03-06T07:31:22.074570+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:48.507457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074314+0000","last_change":"2024-03-06T07:31:21.071128+0000","last_active":"2024-03-06T07:31:22.074314+0000","last_peered":"2024-03-06T07:31:22.074314+0000","last_clean":"2024-03-06T07:31:22.074314+0000","last_became_active":"2024-03-06T07:31:21.070829+0000","last_became_peered":"2024-03-06T07:31:21.070829+0000","last_unstale":"2024-03-06T07:31:22.074314+0000","last_undegraded":"2024-03-06T07:31:22.074314+0000","last_fullsized":"2024-03-06T07:31:22.074314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:14.734900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130325+0000","last_change":"2024-03-06T07:31:21.066739+0000","last_active":"2024-03-06T07:31:22.130325+0000","last_peered":"2024-03-06T07:31:22.130325+0000","last_clean":"2024-03-06T07:31:22.130325+0000","last_became_active":"2024-03-06T07:31:21.066028+0000","last_became_peered":"2024-03-06T07:31:21.066028+0000","last_unstale":"2024-03-06T07:31:22.130325+0000","last_undegraded":"2024-03-06T07:31:22.130325+0000","last_fullsized":"2024-03-06T07:31:22.130325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:36.454862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074242+0000","last_change":"2024-03-06T07:31:21.064469+0000","last_active":"2024-03-06T07:31:22.074242+0000","last_peered":"2024-03-06T07:31:22.074242+0000","last_clean":"2024-03-06T07:31:22.074242+0000","last_became_active":"2024-03-06T07:31:21.064146+0000","last_became_peered":"2024-03-06T07:31:21.064146+0000","last_unstale":"2024-03-06T07:31:22.074242+0000","last_undegraded":"2024-03-06T07:31:22.074242+0000","last_fullsized":"2024-03-06T07:31:22.074242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:17.499867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074912+0000","last_change":"2024-03-06T07:31:21.070404+0000","last_active":"2024-03-06T07:31:22.074912+0000","last_peered":"2024-03-06T07:31:22.074912+0000","last_clean":"2024-03-06T07:31:22.074912+0000","last_became_active":"2024-03-06T07:31:21.070072+0000","last_became_peered":"2024-03-06T07:31:21.070072+0000","last_unstale":"2024-03-06T07:31:22.074912+0000","last_undegraded":"2024-03-06T07:31:22.074912+0000","last_fullsized":"2024-03-06T07:31:22.074912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:34.044570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134114+0000","last_change":"2024-03-06T07:31:21.082310+0000","last_active":"2024-03-06T07:31:22.134114+0000","last_peered":"2024-03-06T07:31:22.134114+0000","last_clean":"2024-03-06T07:31:22.134114+0000","last_became_active":"2024-03-06T07:31:21.081984+0000","last_became_peered":"2024-03-06T07:31:21.081984+0000","last_unstale":"2024-03-06T07:31:22.134114+0000","last_undegraded":"2024-03-06T07:31:22.134114+0000","last_fullsized":"2024-03-06T07:31:22.134114+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:31.371528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132250+0000","last_change":"2024-03-06T07:31:21.077321+0000","last_active":"2024-03-06T07:31:22.132250+0000","last_peered":"2024-03-06T07:31:22.132250+0000","last_clean":"2024-03-06T07:31:22.132250+0000","last_became_active":"2024-03-06T07:31:21.076973+0000","last_became_peered":"2024-03-06T07:31:21.076973+0000","last_unstale":"2024-03-06T07:31:22.132250+0000","last_undegraded":"2024-03-06T07:31:22.132250+0000","last_fullsized":"2024-03-06T07:31:22.132250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:57.957027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128169+0000","last_change":"2024-03-06T07:31:21.083693+0000","last_active":"2024-03-06T07:31:22.128169+0000","last_peered":"2024-03-06T07:31:22.128169+0000","last_clean":"2024-03-06T07:31:22.128169+0000","last_became_active":"2024-03-06T07:31:21.083375+0000","last_became_peered":"2024-03-06T07:31:21.083375+0000","last_unstale":"2024-03-06T07:31:22.128169+0000","last_undegraded":"2024-03-06T07:31:22.128169+0000","last_fullsized":"2024-03-06T07:31:22.128169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:22.834210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127561+0000","last_change":"2024-03-06T07:31:21.059505+0000","last_active":"2024-03-06T07:31:22.127561+0000","last_peered":"2024-03-06T07:31:22.127561+0000","last_clean":"2024-03-06T07:31:22.127561+0000","last_became_active":"2024-03-06T07:31:21.059128+0000","last_became_peered":"2024-03-06T07:31:21.059128+0000","last_unstale":"2024-03-06T07:31:22.127561+0000","last_undegraded":"2024-03-06T07:31:22.127561+0000","last_fullsized":"2024-03-06T07:31:22.127561+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:18.120622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074391+0000","last_change":"2024-03-06T07:31:21.074713+0000","last_active":"2024-03-06T07:31:22.074391+0000","last_peered":"2024-03-06T07:31:22.074391+0000","last_clean":"2024-03-06T07:31:22.074391+0000","last_became_active":"2024-03-06T07:31:21.074420+0000","last_became_peered":"2024-03-06T07:31:21.074420+0000","last_unstale":"2024-03-06T07:31:22.074391+0000","last_undegraded":"2024-03-06T07:31:22.074391+0000","last_fullsized":"2024-03-06T07:31:22.074391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:52.328983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127475+0000","last_change":"2024-03-06T07:31:21.057982+0000","last_active":"2024-03-06T07:31:22.127475+0000","last_peered":"2024-03-06T07:31:22.127475+0000","last_clean":"2024-03-06T07:31:22.127475+0000","last_became_active":"2024-03-06T07:31:21.057012+0000","last_became_peered":"2024-03-06T07:31:21.057012+0000","last_unstale":"2024-03-06T07:31:22.127475+0000","last_undegraded":"2024-03-06T07:31:22.127475+0000","last_fullsized":"2024-03-06T07:31:22.127475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:51.875945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131610+0000","last_change":"2024-03-06T07:31:21.074696+0000","last_active":"2024-03-06T07:31:22.131610+0000","last_peered":"2024-03-06T07:31:22.131610+0000","last_clean":"2024-03-06T07:31:22.131610+0000","last_became_active":"2024-03-06T07:31:21.074365+0000","last_became_peered":"2024-03-06T07:31:21.074365+0000","last_unstale":"2024-03-06T07:31:22.131610+0000","last_undegraded":"2024-03-06T07:31:22.131610+0000","last_fullsized":"2024-03-06T07:31:22.131610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:50.583417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075051+0000","last_change":"2024-03-06T07:31:21.059534+0000","last_active":"2024-03-06T07:31:22.075051+0000","last_peered":"2024-03-06T07:31:22.075051+0000","last_clean":"2024-03-06T07:31:22.075051+0000","last_became_active":"2024-03-06T07:31:21.059265+0000","last_became_peered":"2024-03-06T07:31:21.059265+0000","last_unstale":"2024-03-06T07:31:22.075051+0000","last_undegraded":"2024-03-06T07:31:22.075051+0000","last_fullsized":"2024-03-06T07:31:22.075051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:16.317285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128848+0000","last_change":"2024-03-06T07:31:21.069478+0000","last_active":"2024-03-06T07:31:22.128848+0000","last_peered":"2024-03-06T07:31:22.128848+0000","last_clean":"2024-03-06T07:31:22.128848+0000","last_became_active":"2024-03-06T07:31:21.069215+0000","last_became_peered":"2024-03-06T07:31:21.069215+0000","last_unstale":"2024-03-06T07:31:22.128848+0000","last_undegraded":"2024-03-06T07:31:22.128848+0000","last_fullsized":"2024-03-06T07:31:22.128848+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:16.691429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074774+0000","last_change":"2024-03-06T07:31:21.065658+0000","last_active":"2024-03-06T07:31:22.074774+0000","last_peered":"2024-03-06T07:31:22.074774+0000","last_clean":"2024-03-06T07:31:22.074774+0000","last_became_active":"2024-03-06T07:31:21.065308+0000","last_became_peered":"2024-03-06T07:31:21.065308+0000","last_unstale":"2024-03-06T07:31:22.074774+0000","last_undegraded":"2024-03-06T07:31:22.074774+0000","last_fullsized":"2024-03-06T07:31:22.074774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:13.803132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127963+0000","last_change":"2024-03-06T07:31:21.068191+0000","last_active":"2024-03-06T07:31:22.127963+0000","last_peered":"2024-03-06T07:31:22.127963+0000","last_clean":"2024-03-06T07:31:22.127963+0000","last_became_active":"2024-03-06T07:31:21.067928+0000","last_became_peered":"2024-03-06T07:31:21.067928+0000","last_unstale":"2024-03-06T07:31:22.127963+0000","last_undegraded":"2024-03-06T07:31:22.127963+0000","last_fullsized":"2024-03-06T07:31:22.127963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:43.982756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131397+0000","last_change":"2024-03-06T07:31:21.074913+0000","last_active":"2024-03-06T07:31:22.131397+0000","last_peered":"2024-03-06T07:31:22.131397+0000","last_clean":"2024-03-06T07:31:22.131397+0000","last_became_active":"2024-03-06T07:31:21.073358+0000","last_became_peered":"2024-03-06T07:31:21.073358+0000","last_unstale":"2024-03-06T07:31:22.131397+0000","last_undegraded":"2024-03-06T07:31:22.131397+0000","last_fullsized":"2024-03-06T07:31:22.131397+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:14.734900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132986+0000","last_change":"2024-03-06T07:31:21.082910+0000","last_active":"2024-03-06T07:31:22.132986+0000","last_peered":"2024-03-06T07:31:22.132986+0000","last_clean":"2024-03-06T07:31:22.132986+0000","last_became_active":"2024-03-06T07:31:21.082612+0000","last_became_peered":"2024-03-06T07:31:21.082612+0000","last_unstale":"2024-03-06T07:31:22.132986+0000","last_undegraded":"2024-03-06T07:31:22.132986+0000","last_fullsized":"2024-03-06T07:31:22.132986+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:54.016506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130987+0000","last_change":"2024-03-06T07:31:21.070797+0000","last_active":"2024-03-06T07:31:22.130987+0000","last_peered":"2024-03-06T07:31:22.130987+0000","last_clean":"2024-03-06T07:31:22.130987+0000","last_became_active":"2024-03-06T07:31:21.070519+0000","last_became_peered":"2024-03-06T07:31:21.070519+0000","last_unstale":"2024-03-06T07:31:22.130987+0000","last_undegraded":"2024-03-06T07:31:22.130987+0000","last_fullsized":"2024-03-06T07:31:22.130987+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:59.352745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130736+0000","last_change":"2024-03-06T07:31:21.075341+0000","last_active":"2024-03-06T07:31:22.130736+0000","last_peered":"2024-03-06T07:31:22.130736+0000","last_clean":"2024-03-06T07:31:22.130736+0000","last_became_active":"2024-03-06T07:31:21.075021+0000","last_became_peered":"2024-03-06T07:31:21.075021+0000","last_unstale":"2024-03-06T07:31:22.130736+0000","last_undegraded":"2024-03-06T07:31:22.130736+0000","last_fullsized":"2024-03-06T07:31:22.130736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:43.982756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134392+0000","last_change":"2024-03-06T07:31:21.077095+0000","last_active":"2024-03-06T07:31:22.134392+0000","last_peered":"2024-03-06T07:31:22.134392+0000","last_clean":"2024-03-06T07:31:22.134392+0000","last_became_active":"2024-03-06T07:31:21.076068+0000","last_became_peered":"2024-03-06T07:31:21.076068+0000","last_unstale":"2024-03-06T07:31:22.134392+0000","last_undegraded":"2024-03-06T07:31:22.134392+0000","last_fullsized":"2024-03-06T07:31:22.134392+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:09.875896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132325+0000","last_change":"2024-03-06T07:31:21.075318+0000","last_active":"2024-03-06T07:31:22.132325+0000","last_peered":"2024-03-06T07:31:22.132325+0000","last_clean":"2024-03-06T07:31:22.132325+0000","last_became_active":"2024-03-06T07:31:21.074971+0000","last_became_peered":"2024-03-06T07:31:21.074971+0000","last_unstale":"2024-03-06T07:31:22.132325+0000","last_undegraded":"2024-03-06T07:31:22.132325+0000","last_fullsized":"2024-03-06T07:31:22.132325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:44.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072716+0000","last_change":"2024-03-06T07:31:21.074573+0000","last_active":"2024-03-06T07:31:22.072716+0000","last_peered":"2024-03-06T07:31:22.072716+0000","last_clean":"2024-03-06T07:31:22.072716+0000","last_became_active":"2024-03-06T07:31:21.074251+0000","last_became_peered":"2024-03-06T07:31:21.074251+0000","last_unstale":"2024-03-06T07:31:22.072716+0000","last_undegraded":"2024-03-06T07:31:22.072716+0000","last_fullsized":"2024-03-06T07:31:22.072716+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:23.355027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133705+0000","last_change":"2024-03-06T07:31:21.084306+0000","last_active":"2024-03-06T07:31:22.133705+0000","last_peered":"2024-03-06T07:31:22.133705+0000","last_clean":"2024-03-06T07:31:22.133705+0000","last_became_active":"2024-03-06T07:31:21.083673+0000","last_became_peered":"2024-03-06T07:31:21.083673+0000","last_unstale":"2024-03-06T07:31:22.133705+0000","last_undegraded":"2024-03-06T07:31:22.133705+0000","last_fullsized":"2024-03-06T07:31:22.133705+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:51.005686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072226+0000","last_change":"2024-03-06T07:31:21.064761+0000","last_active":"2024-03-06T07:31:22.072226+0000","last_peered":"2024-03-06T07:31:22.072226+0000","last_clean":"2024-03-06T07:31:22.072226+0000","last_became_active":"2024-03-06T07:31:21.064390+0000","last_became_peered":"2024-03-06T07:31:21.064390+0000","last_unstale":"2024-03-06T07:31:22.072226+0000","last_undegraded":"2024-03-06T07:31:22.072226+0000","last_fullsized":"2024-03-06T07:31:22.072226+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:17.052290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133819+0000","last_change":"2024-03-06T07:31:21.062778+0000","last_active":"2024-03-06T07:31:22.133819+0000","last_peered":"2024-03-06T07:31:22.133819+0000","last_clean":"2024-03-06T07:31:22.133819+0000","last_became_active":"2024-03-06T07:31:21.062374+0000","last_became_peered":"2024-03-06T07:31:21.062374+0000","last_unstale":"2024-03-06T07:31:22.133819+0000","last_undegraded":"2024-03-06T07:31:22.133819+0000","last_fullsized":"2024-03-06T07:31:22.133819+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131322+0000","last_change":"2024-03-06T07:31:21.083905+0000","last_active":"2024-03-06T07:31:22.131322+0000","last_peered":"2024-03-06T07:31:22.131322+0000","last_clean":"2024-03-06T07:31:22.131322+0000","last_became_active":"2024-03-06T07:31:21.081704+0000","last_became_peered":"2024-03-06T07:31:21.081704+0000","last_unstale":"2024-03-06T07:31:22.131322+0000","last_undegraded":"2024-03-06T07:31:22.131322+0000","last_fullsized":"2024-03-06T07:31:22.131322+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:27.536304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074764+0000","last_change":"2024-03-06T07:31:21.057192+0000","last_active":"2024-03-06T07:31:22.074764+0000","last_peered":"2024-03-06T07:31:22.074764+0000","last_clean":"2024-03-06T07:31:22.074764+0000","last_became_active":"2024-03-06T07:31:21.055921+0000","last_became_peered":"2024-03-06T07:31:21.055921+0000","last_unstale":"2024-03-06T07:31:22.074764+0000","last_undegraded":"2024-03-06T07:31:22.074764+0000","last_fullsized":"2024-03-06T07:31:22.074764+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:50.583417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132601+0000","last_change":"2024-03-06T07:31:21.081485+0000","last_active":"2024-03-06T07:31:22.132601+0000","last_peered":"2024-03-06T07:31:22.132601+0000","last_clean":"2024-03-06T07:31:22.132601+0000","last_became_active":"2024-03-06T07:31:21.081134+0000","last_became_peered":"2024-03-06T07:31:21.081134+0000","last_unstale":"2024-03-06T07:31:22.132601+0000","last_undegraded":"2024-03-06T07:31:22.132601+0000","last_fullsized":"2024-03-06T07:31:22.132601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:18.106636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"65'4","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133523+0000","last_change":"2024-03-06T07:31:20.063659+0000","last_active":"2024-03-06T07:31:22.133523+0000","last_peered":"2024-03-06T07:31:22.133523+0000","last_clean":"2024-03-06T07:31:22.133523+0000","last_became_active":"2024-03-06T07:31:20.062726+0000","last_became_peered":"2024-03-06T07:31:20.062726+0000","last_unstale":"2024-03-06T07:31:22.133523+0000","last_undegraded":"2024-03-06T07:31:22.133523+0000","last_fullsized":"2024-03-06T07:31:22.133523+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:49.893699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075278+0000","last_change":"2024-03-06T07:31:21.066149+0000","last_active":"2024-03-06T07:31:22.075278+0000","last_peered":"2024-03-06T07:31:22.075278+0000","last_clean":"2024-03-06T07:31:22.075278+0000","last_became_active":"2024-03-06T07:31:21.060334+0000","last_became_peered":"2024-03-06T07:31:21.060334+0000","last_unstale":"2024-03-06T07:31:22.075278+0000","last_undegraded":"2024-03-06T07:31:22.075278+0000","last_fullsized":"2024-03-06T07:31:22.075278+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:59.295606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128008+0000","last_change":"2024-03-06T07:31:20.047815+0000","last_active":"2024-03-06T07:31:22.128008+0000","last_peered":"2024-03-06T07:31:22.128008+0000","last_clean":"2024-03-06T07:31:22.128008+0000","last_became_active":"2024-03-06T07:31:20.047436+0000","last_became_peered":"2024-03-06T07:31:20.047436+0000","last_unstale":"2024-03-06T07:31:22.128008+0000","last_undegraded":"2024-03-06T07:31:22.128008+0000","last_fullsized":"2024-03-06T07:31:22.128008+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:08.860785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075608+0000","last_change":"2024-03-06T07:31:21.066378+0000","last_active":"2024-03-06T07:31:22.075608+0000","last_peered":"2024-03-06T07:31:22.075608+0000","last_clean":"2024-03-06T07:31:22.075608+0000","last_became_active":"2024-03-06T07:31:21.065862+0000","last_became_peered":"2024-03-06T07:31:21.065862+0000","last_unstale":"2024-03-06T07:31:22.075608+0000","last_undegraded":"2024-03-06T07:31:22.075608+0000","last_fullsized":"2024-03-06T07:31:22.075608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:51.005686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131787+0000","last_change":"2024-03-06T07:31:20.064805+0000","last_active":"2024-03-06T07:31:22.131787+0000","last_peered":"2024-03-06T07:31:22.131787+0000","last_clean":"2024-03-06T07:31:22.131787+0000","last_became_active":"2024-03-06T07:31:20.063617+0000","last_became_peered":"2024-03-06T07:31:20.063617+0000","last_unstale":"2024-03-06T07:31:22.131787+0000","last_undegraded":"2024-03-06T07:31:22.131787+0000","last_fullsized":"2024-03-06T07:31:22.131787+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:44.861023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132600+0000","last_change":"2024-03-06T07:31:21.075738+0000","last_active":"2024-03-06T07:31:22.132600+0000","last_peered":"2024-03-06T07:31:22.132600+0000","last_clean":"2024-03-06T07:31:22.132600+0000","last_became_active":"2024-03-06T07:31:21.075465+0000","last_became_peered":"2024-03-06T07:31:21.075465+0000","last_unstale":"2024-03-06T07:31:22.132600+0000","last_undegraded":"2024-03-06T07:31:22.132600+0000","last_fullsized":"2024-03-06T07:31:22.132600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:54.601086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.076005+0000","last_change":"2024-03-06T07:31:20.060231+0000","last_active":"2024-03-06T07:31:22.076005+0000","last_peered":"2024-03-06T07:31:22.076005+0000","last_clean":"2024-03-06T07:31:22.076005+0000","last_became_active":"2024-03-06T07:31:20.059932+0000","last_became_peered":"2024-03-06T07:31:20.059932+0000","last_unstale":"2024-03-06T07:31:22.076005+0000","last_undegraded":"2024-03-06T07:31:22.076005+0000","last_fullsized":"2024-03-06T07:31:22.076005+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:44.861023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131153+0000","last_change":"2024-03-06T07:31:21.067967+0000","last_active":"2024-03-06T07:31:22.131153+0000","last_peered":"2024-03-06T07:31:22.131153+0000","last_clean":"2024-03-06T07:31:22.131153+0000","last_became_active":"2024-03-06T07:31:21.067558+0000","last_became_peered":"2024-03-06T07:31:21.067558+0000","last_unstale":"2024-03-06T07:31:22.131153+0000","last_undegraded":"2024-03-06T07:31:22.131153+0000","last_fullsized":"2024-03-06T07:31:22.131153+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:02.154662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072511+0000","last_change":"2024-03-06T07:31:20.062132+0000","last_active":"2024-03-06T07:31:22.072511+0000","last_peered":"2024-03-06T07:31:22.072511+0000","last_clean":"2024-03-06T07:31:22.072511+0000","last_became_active":"2024-03-06T07:31:20.060895+0000","last_became_peered":"2024-03-06T07:31:20.060895+0000","last_unstale":"2024-03-06T07:31:22.072511+0000","last_undegraded":"2024-03-06T07:31:22.072511+0000","last_fullsized":"2024-03-06T07:31:22.072511+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:53.585975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131827+0000","last_change":"2024-03-06T07:31:21.080386+0000","last_active":"2024-03-06T07:31:22.131827+0000","last_peered":"2024-03-06T07:31:22.131827+0000","last_clean":"2024-03-06T07:31:22.131827+0000","last_became_active":"2024-03-06T07:31:21.073882+0000","last_became_peered":"2024-03-06T07:31:21.073882+0000","last_unstale":"2024-03-06T07:31:22.131827+0000","last_undegraded":"2024-03-06T07:31:22.131827+0000","last_fullsized":"2024-03-06T07:31:22.131827+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:13.803132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074882+0000","last_change":"2024-03-06T07:31:20.052215+0000","last_active":"2024-03-06T07:31:22.074882+0000","last_peered":"2024-03-06T07:31:22.074882+0000","last_clean":"2024-03-06T07:31:22.074882+0000","last_became_active":"2024-03-06T07:31:20.051935+0000","last_became_peered":"2024-03-06T07:31:20.051935+0000","last_unstale":"2024-03-06T07:31:22.074882+0000","last_undegraded":"2024-03-06T07:31:22.074882+0000","last_fullsized":"2024-03-06T07:31:22.074882+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:30.244740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.073975+0000","last_change":"2024-03-06T07:31:21.076556+0000","last_active":"2024-03-06T07:31:22.073975+0000","last_peered":"2024-03-06T07:31:22.073975+0000","last_clean":"2024-03-06T07:31:22.073975+0000","last_became_active":"2024-03-06T07:31:21.076275+0000","last_became_peered":"2024-03-06T07:31:21.076275+0000","last_unstale":"2024-03-06T07:31:22.073975+0000","last_undegraded":"2024-03-06T07:31:22.073975+0000","last_fullsized":"2024-03-06T07:31:22.073975+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:24.991953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127744+0000","last_change":"2024-03-06T07:31:20.059231+0000","last_active":"2024-03-06T07:31:22.127744+0000","last_peered":"2024-03-06T07:31:22.127744+0000","last_clean":"2024-03-06T07:31:22.127744+0000","last_became_active":"2024-03-06T07:31:20.058946+0000","last_became_peered":"2024-03-06T07:31:20.058946+0000","last_unstale":"2024-03-06T07:31:22.127744+0000","last_undegraded":"2024-03-06T07:31:22.127744+0000","last_fullsized":"2024-03-06T07:31:22.127744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:18.460944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127630+0000","last_change":"2024-03-06T07:31:21.066893+0000","last_active":"2024-03-06T07:31:22.127630+0000","last_peered":"2024-03-06T07:31:22.127630+0000","last_clean":"2024-03-06T07:31:22.127630+0000","last_became_active":"2024-03-06T07:31:21.065636+0000","last_became_peered":"2024-03-06T07:31:21.065636+0000","last_unstale":"2024-03-06T07:31:22.127630+0000","last_undegraded":"2024-03-06T07:31:22.127630+0000","last_fullsized":"2024-03-06T07:31:22.127630+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:15.795338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.073971+0000","last_change":"2024-03-06T07:31:20.048745+0000","last_active":"2024-03-06T07:31:22.073971+0000","last_peered":"2024-03-06T07:31:22.073971+0000","last_clean":"2024-03-06T07:31:22.073971+0000","last_became_active":"2024-03-06T07:31:20.048441+0000","last_became_peered":"2024-03-06T07:31:20.048441+0000","last_unstale":"2024-03-06T07:31:22.073971+0000","last_undegraded":"2024-03-06T07:31:22.073971+0000","last_fullsized":"2024-03-06T07:31:22.073971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:00.165040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132731+0000","last_change":"2024-03-06T07:31:21.078336+0000","last_active":"2024-03-06T07:31:22.132731+0000","last_peered":"2024-03-06T07:31:22.132731+0000","last_clean":"2024-03-06T07:31:22.132731+0000","last_became_active":"2024-03-06T07:31:21.078019+0000","last_became_peered":"2024-03-06T07:31:21.078019+0000","last_unstale":"2024-03-06T07:31:22.132731+0000","last_undegraded":"2024-03-06T07:31:22.132731+0000","last_fullsized":"2024-03-06T07:31:22.132731+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:01.616697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134120+0000","last_change":"2024-03-06T07:31:20.064730+0000","last_active":"2024-03-06T07:31:22.134120+0000","last_peered":"2024-03-06T07:31:22.134120+0000","last_clean":"2024-03-06T07:31:22.134120+0000","last_became_active":"2024-03-06T07:31:20.063871+0000","last_became_peered":"2024-03-06T07:31:20.063871+0000","last_unstale":"2024-03-06T07:31:22.134120+0000","last_undegraded":"2024-03-06T07:31:22.134120+0000","last_fullsized":"2024-03-06T07:31:22.134120+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:12.877176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132763+0000","last_change":"2024-03-06T07:31:21.083063+0000","last_active":"2024-03-06T07:31:22.132763+0000","last_peered":"2024-03-06T07:31:22.132763+0000","last_clean":"2024-03-06T07:31:22.132763+0000","last_became_active":"2024-03-06T07:31:21.082427+0000","last_became_peered":"2024-03-06T07:31:21.082427+0000","last_unstale":"2024-03-06T07:31:22.132763+0000","last_undegraded":"2024-03-06T07:31:22.132763+0000","last_fullsized":"2024-03-06T07:31:22.132763+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:30.095004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075750+0000","last_change":"2024-03-06T07:31:20.063163+0000","last_active":"2024-03-06T07:31:22.075750+0000","last_peered":"2024-03-06T07:31:22.075750+0000","last_clean":"2024-03-06T07:31:22.075750+0000","last_became_active":"2024-03-06T07:31:20.061717+0000","last_became_peered":"2024-03-06T07:31:20.061717+0000","last_unstale":"2024-03-06T07:31:22.075750+0000","last_undegraded":"2024-03-06T07:31:22.075750+0000","last_fullsized":"2024-03-06T07:31:22.075750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:35.502814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.076097+0000","last_change":"2024-03-06T07:31:21.071252+0000","last_active":"2024-03-06T07:31:22.076097+0000","last_peered":"2024-03-06T07:31:22.076097+0000","last_clean":"2024-03-06T07:31:22.076097+0000","last_became_active":"2024-03-06T07:31:21.070677+0000","last_became_peered":"2024-03-06T07:31:21.070677+0000","last_unstale":"2024-03-06T07:31:22.076097+0000","last_undegraded":"2024-03-06T07:31:22.076097+0000","last_fullsized":"2024-03-06T07:31:22.076097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:09.875896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"65'3","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072831+0000","last_change":"2024-03-06T07:31:20.056117+0000","last_active":"2024-03-06T07:31:22.072831+0000","last_peered":"2024-03-06T07:31:22.072831+0000","last_clean":"2024-03-06T07:31:22.072831+0000","last_became_active":"2024-03-06T07:31:20.055715+0000","last_became_peered":"2024-03-06T07:31:20.055715+0000","last_unstale":"2024-03-06T07:31:22.072831+0000","last_undegraded":"2024-03-06T07:31:22.072831+0000","last_fullsized":"2024-03-06T07:31:22.072831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:17.091524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074361+0000","last_change":"2024-03-06T07:31:21.065295+0000","last_active":"2024-03-06T07:31:22.074361+0000","last_peered":"2024-03-06T07:31:22.074361+0000","last_clean":"2024-03-06T07:31:22.074361+0000","last_became_active":"2024-03-06T07:31:21.064998+0000","last_became_peered":"2024-03-06T07:31:21.064998+0000","last_unstale":"2024-03-06T07:31:22.074361+0000","last_undegraded":"2024-03-06T07:31:22.074361+0000","last_fullsized":"2024-03-06T07:31:22.074361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:41.559348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"65'6","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128343+0000","last_change":"2024-03-06T07:31:20.049330+0000","last_active":"2024-03-06T07:31:22.128343+0000","last_peered":"2024-03-06T07:31:22.128343+0000","last_clean":"2024-03-06T07:31:22.128343+0000","last_became_active":"2024-03-06T07:31:20.049008+0000","last_became_peered":"2024-03-06T07:31:20.049008+0000","last_unstale":"2024-03-06T07:31:22.128343+0000","last_undegraded":"2024-03-06T07:31:22.128343+0000","last_fullsized":"2024-03-06T07:31:22.128343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:43.338550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131195+0000","last_change":"2024-03-06T07:31:21.086442+0000","last_active":"2024-03-06T07:31:22.131195+0000","last_peered":"2024-03-06T07:31:22.131195+0000","last_clean":"2024-03-06T07:31:22.131195+0000","last_became_active":"2024-03-06T07:31:21.085366+0000","last_became_peered":"2024-03-06T07:31:21.085366+0000","last_unstale":"2024-03-06T07:31:22.131195+0000","last_undegraded":"2024-03-06T07:31:22.131195+0000","last_fullsized":"2024-03-06T07:31:22.131195+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:04.425146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127979+0000","last_change":"2024-03-06T07:31:20.057733+0000","last_active":"2024-03-06T07:31:22.127979+0000","last_peered":"2024-03-06T07:31:22.127979+0000","last_clean":"2024-03-06T07:31:22.127979+0000","last_became_active":"2024-03-06T07:31:20.057444+0000","last_became_peered":"2024-03-06T07:31:20.057444+0000","last_unstale":"2024-03-06T07:31:22.127979+0000","last_undegraded":"2024-03-06T07:31:22.127979+0000","last_fullsized":"2024-03-06T07:31:22.127979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:00.373578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133445+0000","last_change":"2024-03-06T07:31:21.078875+0000","last_active":"2024-03-06T07:31:22.133445+0000","last_peered":"2024-03-06T07:31:22.133445+0000","last_clean":"2024-03-06T07:31:22.133445+0000","last_became_active":"2024-03-06T07:31:21.074059+0000","last_became_peered":"2024-03-06T07:31:21.074059+0000","last_unstale":"2024-03-06T07:31:22.133445+0000","last_undegraded":"2024-03-06T07:31:22.133445+0000","last_fullsized":"2024-03-06T07:31:22.133445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:51.875945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074950+0000","last_change":"2024-03-06T07:31:20.053542+0000","last_active":"2024-03-06T07:31:22.074950+0000","last_peered":"2024-03-06T07:31:22.074950+0000","last_clean":"2024-03-06T07:31:22.074950+0000","last_became_active":"2024-03-06T07:31:20.051991+0000","last_became_peered":"2024-03-06T07:31:20.051991+0000","last_unstale":"2024-03-06T07:31:22.074950+0000","last_undegraded":"2024-03-06T07:31:22.074950+0000","last_fullsized":"2024-03-06T07:31:22.074950+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:00.373578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127541+0000","last_change":"2024-03-06T07:31:21.059392+0000","last_active":"2024-03-06T07:31:22.127541+0000","last_peered":"2024-03-06T07:31:22.127541+0000","last_clean":"2024-03-06T07:31:22.127541+0000","last_became_active":"2024-03-06T07:31:21.059089+0000","last_became_peered":"2024-03-06T07:31:21.059089+0000","last_unstale":"2024-03-06T07:31:22.127541+0000","last_undegraded":"2024-03-06T07:31:22.127541+0000","last_fullsized":"2024-03-06T07:31:22.127541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:35.936843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130501+0000","last_change":"2024-03-06T07:31:20.054898+0000","last_active":"2024-03-06T07:31:22.130501+0000","last_peered":"2024-03-06T07:31:22.130501+0000","last_clean":"2024-03-06T07:31:22.130501+0000","last_became_active":"2024-03-06T07:31:20.054535+0000","last_became_peered":"2024-03-06T07:31:20.054535+0000","last_unstale":"2024-03-06T07:31:22.130501+0000","last_undegraded":"2024-03-06T07:31:22.130501+0000","last_fullsized":"2024-03-06T07:31:22.130501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:35.537736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074513+0000","last_change":"2024-03-06T07:31:21.054190+0000","last_active":"2024-03-06T07:31:22.074513+0000","last_peered":"2024-03-06T07:31:22.074513+0000","last_clean":"2024-03-06T07:31:22.074513+0000","last_became_active":"2024-03-06T07:31:21.053700+0000","last_became_peered":"2024-03-06T07:31:21.053700+0000","last_unstale":"2024-03-06T07:31:22.074513+0000","last_undegraded":"2024-03-06T07:31:22.074513+0000","last_fullsized":"2024-03-06T07:31:22.074513+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:16.403218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"65'8","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131891+0000","last_change":"2024-03-06T07:31:20.061201+0000","last_active":"2024-03-06T07:31:22.131891+0000","last_peered":"2024-03-06T07:31:22.131891+0000","last_clean":"2024-03-06T07:31:22.131891+0000","last_became_active":"2024-03-06T07:31:20.060919+0000","last_became_peered":"2024-03-06T07:31:20.060919+0000","last_unstale":"2024-03-06T07:31:22.131891+0000","last_undegraded":"2024-03-06T07:31:22.131891+0000","last_fullsized":"2024-03-06T07:31:22.131891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:53.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134521+0000","last_change":"2024-03-06T07:31:21.070760+0000","last_active":"2024-03-06T07:31:22.134521+0000","last_peered":"2024-03-06T07:31:22.134521+0000","last_clean":"2024-03-06T07:31:22.134521+0000","last_became_active":"2024-03-06T07:31:21.070421+0000","last_became_peered":"2024-03-06T07:31:21.070421+0000","last_unstale":"2024-03-06T07:31:22.134521+0000","last_undegraded":"2024-03-06T07:31:22.134521+0000","last_fullsized":"2024-03-06T07:31:22.134521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:50.908811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075978+0000","last_change":"2024-03-06T07:31:20.060259+0000","last_active":"2024-03-06T07:31:22.075978+0000","last_peered":"2024-03-06T07:31:22.075978+0000","last_clean":"2024-03-06T07:31:22.075978+0000","last_became_active":"2024-03-06T07:31:20.059947+0000","last_became_peered":"2024-03-06T07:31:20.059947+0000","last_unstale":"2024-03-06T07:31:22.075978+0000","last_undegraded":"2024-03-06T07:31:22.075978+0000","last_fullsized":"2024-03-06T07:31:22.075978+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:43.338550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133214+0000","last_change":"2024-03-06T07:31:21.075262+0000","last_active":"2024-03-06T07:31:22.133214+0000","last_peered":"2024-03-06T07:31:22.133214+0000","last_clean":"2024-03-06T07:31:22.133214+0000","last_became_active":"2024-03-06T07:31:21.074848+0000","last_became_peered":"2024-03-06T07:31:21.074848+0000","last_unstale":"2024-03-06T07:31:22.133214+0000","last_undegraded":"2024-03-06T07:31:22.133214+0000","last_fullsized":"2024-03-06T07:31:22.133214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:48.328865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.134410+0000","last_change":"2024-03-06T07:31:20.064445+0000","last_active":"2024-03-06T07:31:22.134410+0000","last_peered":"2024-03-06T07:31:22.134410+0000","last_clean":"2024-03-06T07:31:22.134410+0000","last_became_active":"2024-03-06T07:31:20.064168+0000","last_became_peered":"2024-03-06T07:31:20.064168+0000","last_unstale":"2024-03-06T07:31:22.134410+0000","last_undegraded":"2024-03-06T07:31:22.134410+0000","last_fullsized":"2024-03-06T07:31:22.134410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:02.556295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128243+0000","last_change":"2024-03-06T07:31:21.079636+0000","last_active":"2024-03-06T07:31:22.128243+0000","last_peered":"2024-03-06T07:31:22.128243+0000","last_clean":"2024-03-06T07:31:22.128243+0000","last_became_active":"2024-03-06T07:31:21.079384+0000","last_became_peered":"2024-03-06T07:31:21.079384+0000","last_unstale":"2024-03-06T07:31:22.128243+0000","last_undegraded":"2024-03-06T07:31:22.128243+0000","last_fullsized":"2024-03-06T07:31:22.128243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:18.618283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075322+0000","last_change":"2024-03-06T07:31:20.059890+0000","last_active":"2024-03-06T07:31:22.075322+0000","last_peered":"2024-03-06T07:31:22.075322+0000","last_clean":"2024-03-06T07:31:22.075322+0000","last_became_active":"2024-03-06T07:31:20.059612+0000","last_became_peered":"2024-03-06T07:31:20.059612+0000","last_unstale":"2024-03-06T07:31:22.075322+0000","last_undegraded":"2024-03-06T07:31:22.075322+0000","last_fullsized":"2024-03-06T07:31:22.075322+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.918263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074569+0000","last_change":"2024-03-06T07:31:21.064475+0000","last_active":"2024-03-06T07:31:22.074569+0000","last_peered":"2024-03-06T07:31:22.074569+0000","last_clean":"2024-03-06T07:31:22.074569+0000","last_became_active":"2024-03-06T07:31:21.064195+0000","last_became_peered":"2024-03-06T07:31:21.064195+0000","last_unstale":"2024-03-06T07:31:22.074569+0000","last_undegraded":"2024-03-06T07:31:22.074569+0000","last_fullsized":"2024-03-06T07:31:22.074569+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:21.007122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133070+0000","last_change":"2024-03-06T07:31:20.065063+0000","last_active":"2024-03-06T07:31:22.133070+0000","last_peered":"2024-03-06T07:31:22.133070+0000","last_clean":"2024-03-06T07:31:22.133070+0000","last_became_active":"2024-03-06T07:31:20.064566+0000","last_became_peered":"2024-03-06T07:31:20.064566+0000","last_unstale":"2024-03-06T07:31:22.133070+0000","last_undegraded":"2024-03-06T07:31:22.133070+0000","last_fullsized":"2024-03-06T07:31:22.133070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:08.860785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072394+0000","last_change":"2024-03-06T07:31:21.063158+0000","last_active":"2024-03-06T07:31:22.072394+0000","last_peered":"2024-03-06T07:31:22.072394+0000","last_clean":"2024-03-06T07:31:22.072394+0000","last_became_active":"2024-03-06T07:31:21.062893+0000","last_became_peered":"2024-03-06T07:31:21.062893+0000","last_unstale":"2024-03-06T07:31:22.072394+0000","last_undegraded":"2024-03-06T07:31:22.072394+0000","last_fullsized":"2024-03-06T07:31:22.072394+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:07.101394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075146+0000","last_change":"2024-03-06T07:31:20.054569+0000","last_active":"2024-03-06T07:31:22.075146+0000","last_peered":"2024-03-06T07:31:22.075146+0000","last_clean":"2024-03-06T07:31:22.075146+0000","last_became_active":"2024-03-06T07:31:20.054115+0000","last_became_peered":"2024-03-06T07:31:20.054115+0000","last_unstale":"2024-03-06T07:31:22.075146+0000","last_undegraded":"2024-03-06T07:31:22.075146+0000","last_fullsized":"2024-03-06T07:31:22.075146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:53.608925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072307+0000","last_change":"2024-03-06T07:31:21.056323+0000","last_active":"2024-03-06T07:31:22.072307+0000","last_peered":"2024-03-06T07:31:22.072307+0000","last_clean":"2024-03-06T07:31:22.072307+0000","last_became_active":"2024-03-06T07:31:21.056042+0000","last_became_peered":"2024-03-06T07:31:21.056042+0000","last_unstale":"2024-03-06T07:31:22.072307+0000","last_undegraded":"2024-03-06T07:31:22.072307+0000","last_fullsized":"2024-03-06T07:31:22.072307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:06.914166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130797+0000","last_change":"2024-03-06T07:31:20.050862+0000","last_active":"2024-03-06T07:31:22.130797+0000","last_peered":"2024-03-06T07:31:22.130797+0000","last_clean":"2024-03-06T07:31:22.130797+0000","last_became_active":"2024-03-06T07:31:20.050589+0000","last_became_peered":"2024-03-06T07:31:20.050589+0000","last_unstale":"2024-03-06T07:31:22.130797+0000","last_undegraded":"2024-03-06T07:31:22.130797+0000","last_fullsized":"2024-03-06T07:31:22.130797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:16.484756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072230+0000","last_change":"2024-03-06T07:31:21.056338+0000","last_active":"2024-03-06T07:31:22.072230+0000","last_peered":"2024-03-06T07:31:22.072230+0000","last_clean":"2024-03-06T07:31:22.072230+0000","last_became_active":"2024-03-06T07:31:21.055778+0000","last_became_peered":"2024-03-06T07:31:21.055778+0000","last_unstale":"2024-03-06T07:31:22.072230+0000","last_undegraded":"2024-03-06T07:31:22.072230+0000","last_fullsized":"2024-03-06T07:31:22.072230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:36.206563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127332+0000","last_change":"2024-03-06T07:31:20.050121+0000","last_active":"2024-03-06T07:31:22.127332+0000","last_peered":"2024-03-06T07:31:22.127332+0000","last_clean":"2024-03-06T07:31:22.127332+0000","last_became_active":"2024-03-06T07:31:20.048492+0000","last_became_peered":"2024-03-06T07:31:20.048492+0000","last_unstale":"2024-03-06T07:31:22.127332+0000","last_undegraded":"2024-03-06T07:31:22.127332+0000","last_fullsized":"2024-03-06T07:31:22.127332+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:30.356417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127907+0000","last_change":"2024-03-06T07:31:21.057834+0000","last_active":"2024-03-06T07:31:22.127907+0000","last_peered":"2024-03-06T07:31:22.127907+0000","last_clean":"2024-03-06T07:31:22.127907+0000","last_became_active":"2024-03-06T07:31:21.056626+0000","last_became_peered":"2024-03-06T07:31:21.056626+0000","last_unstale":"2024-03-06T07:31:22.127907+0000","last_undegraded":"2024-03-06T07:31:22.127907+0000","last_fullsized":"2024-03-06T07:31:22.127907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:10.546879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075427+0000","last_change":"2024-03-06T07:31:20.061495+0000","last_active":"2024-03-06T07:31:22.075427+0000","last_peered":"2024-03-06T07:31:22.075427+0000","last_clean":"2024-03-06T07:31:22.075427+0000","last_became_active":"2024-03-06T07:31:20.060383+0000","last_became_peered":"2024-03-06T07:31:20.060383+0000","last_unstale":"2024-03-06T07:31:22.075427+0000","last_undegraded":"2024-03-06T07:31:22.075427+0000","last_fullsized":"2024-03-06T07:31:22.075427+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:10.735755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133831+0000","last_change":"2024-03-06T07:31:21.066729+0000","last_active":"2024-03-06T07:31:22.133831+0000","last_peered":"2024-03-06T07:31:22.133831+0000","last_clean":"2024-03-06T07:31:22.133831+0000","last_became_active":"2024-03-06T07:31:21.066358+0000","last_became_peered":"2024-03-06T07:31:21.066358+0000","last_unstale":"2024-03-06T07:31:22.133831+0000","last_undegraded":"2024-03-06T07:31:22.133831+0000","last_fullsized":"2024-03-06T07:31:22.133831+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:18.120622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"66'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:23.073584+0000","last_change":"2024-03-06T07:31:20.059415+0000","last_active":"2024-03-06T07:31:23.073584+0000","last_peered":"2024-03-06T07:31:23.073584+0000","last_clean":"2024-03-06T07:31:23.073584+0000","last_became_active":"2024-03-06T07:31:20.059067+0000","last_became_peered":"2024-03-06T07:31:20.059067+0000","last_unstale":"2024-03-06T07:31:23.073584+0000","last_undegraded":"2024-03-06T07:31:23.073584+0000","last_fullsized":"2024-03-06T07:31:23.073584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:13.719789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.131892+0000","last_change":"2024-03-06T07:31:21.081861+0000","last_active":"2024-03-06T07:31:22.131892+0000","last_peered":"2024-03-06T07:31:22.131892+0000","last_clean":"2024-03-06T07:31:22.131892+0000","last_became_active":"2024-03-06T07:31:21.081178+0000","last_became_peered":"2024-03-06T07:31:21.081178+0000","last_unstale":"2024-03-06T07:31:22.131892+0000","last_undegraded":"2024-03-06T07:31:22.131892+0000","last_fullsized":"2024-03-06T07:31:22.131892+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:46.802945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074731+0000","last_change":"2024-03-06T07:31:20.052615+0000","last_active":"2024-03-06T07:31:22.074731+0000","last_peered":"2024-03-06T07:31:22.074731+0000","last_clean":"2024-03-06T07:31:22.074731+0000","last_became_active":"2024-03-06T07:31:20.051001+0000","last_became_peered":"2024-03-06T07:31:20.051001+0000","last_unstale":"2024-03-06T07:31:22.074731+0000","last_undegraded":"2024-03-06T07:31:22.074731+0000","last_fullsized":"2024-03-06T07:31:22.074731+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:15:07.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.133213+0000","last_change":"2024-03-06T07:31:21.086159+0000","last_active":"2024-03-06T07:31:22.133213+0000","last_peered":"2024-03-06T07:31:22.133213+0000","last_clean":"2024-03-06T07:31:22.133213+0000","last_became_active":"2024-03-06T07:31:21.085619+0000","last_became_peered":"2024-03-06T07:31:21.085619+0000","last_unstale":"2024-03-06T07:31:22.133213+0000","last_undegraded":"2024-03-06T07:31:22.133213+0000","last_fullsized":"2024-03-06T07:31:22.133213+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:34.645483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127745+0000","last_change":"2024-03-06T07:31:20.059873+0000","last_active":"2024-03-06T07:31:22.127745+0000","last_peered":"2024-03-06T07:31:22.127745+0000","last_clean":"2024-03-06T07:31:22.127745+0000","last_became_active":"2024-03-06T07:31:20.059624+0000","last_became_peered":"2024-03-06T07:31:20.059624+0000","last_unstale":"2024-03-06T07:31:22.127745+0000","last_undegraded":"2024-03-06T07:31:22.127745+0000","last_fullsized":"2024-03-06T07:31:22.127745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:49.893699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074461+0000","last_change":"2024-03-06T07:31:21.058580+0000","last_active":"2024-03-06T07:31:22.074461+0000","last_peered":"2024-03-06T07:31:22.074461+0000","last_clean":"2024-03-06T07:31:22.074461+0000","last_became_active":"2024-03-06T07:31:21.058278+0000","last_became_peered":"2024-03-06T07:31:21.058278+0000","last_unstale":"2024-03-06T07:31:22.074461+0000","last_undegraded":"2024-03-06T07:31:22.074461+0000","last_fullsized":"2024-03-06T07:31:22.074461+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:05.943638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074595+0000","last_change":"2024-03-06T07:31:20.053971+0000","last_active":"2024-03-06T07:31:22.074595+0000","last_peered":"2024-03-06T07:31:22.074595+0000","last_clean":"2024-03-06T07:31:22.074595+0000","last_became_active":"2024-03-06T07:31:20.053541+0000","last_became_peered":"2024-03-06T07:31:20.053541+0000","last_unstale":"2024-03-06T07:31:22.074595+0000","last_undegraded":"2024-03-06T07:31:22.074595+0000","last_fullsized":"2024-03-06T07:31:22.074595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.963841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074701+0000","last_change":"2024-03-06T07:31:21.066218+0000","last_active":"2024-03-06T07:31:22.074701+0000","last_peered":"2024-03-06T07:31:22.074701+0000","last_clean":"2024-03-06T07:31:22.074701+0000","last_became_active":"2024-03-06T07:31:21.065378+0000","last_became_peered":"2024-03-06T07:31:21.065378+0000","last_unstale":"2024-03-06T07:31:22.074701+0000","last_undegraded":"2024-03-06T07:31:22.074701+0000","last_fullsized":"2024-03-06T07:31:22.074701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:54.016506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.128499+0000","last_change":"2024-03-06T07:31:20.061631+0000","last_active":"2024-03-06T07:31:22.128499+0000","last_peered":"2024-03-06T07:31:22.128499+0000","last_clean":"2024-03-06T07:31:22.128499+0000","last_became_active":"2024-03-06T07:31:20.060893+0000","last_became_peered":"2024-03-06T07:31:20.060893+0000","last_unstale":"2024-03-06T07:31:22.128499+0000","last_undegraded":"2024-03-06T07:31:22.128499+0000","last_fullsized":"2024-03-06T07:31:22.128499+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:26.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.127559+0000","last_change":"2024-03-06T07:31:21.060706+0000","last_active":"2024-03-06T07:31:22.127559+0000","last_peered":"2024-03-06T07:31:22.127559+0000","last_clean":"2024-03-06T07:31:22.127559+0000","last_became_active":"2024-03-06T07:31:21.060308+0000","last_became_peered":"2024-03-06T07:31:21.060308+0000","last_unstale":"2024-03-06T07:31:22.127559+0000","last_undegraded":"2024-03-06T07:31:22.127559+0000","last_fullsized":"2024-03-06T07:31:22.127559+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:03.571407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.132320+0000","last_change":"2024-03-06T07:31:20.063597+0000","last_active":"2024-03-06T07:31:22.132320+0000","last_peered":"2024-03-06T07:31:22.132320+0000","last_clean":"2024-03-06T07:31:22.132320+0000","last_became_active":"2024-03-06T07:31:20.062770+0000","last_became_peered":"2024-03-06T07:31:20.062770+0000","last_unstale":"2024-03-06T07:31:22.132320+0000","last_undegraded":"2024-03-06T07:31:22.132320+0000","last_fullsized":"2024-03-06T07:31:22.132320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.918263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074008+0000","last_change":"2024-03-06T07:31:21.056252+0000","last_active":"2024-03-06T07:31:22.074008+0000","last_peered":"2024-03-06T07:31:22.074008+0000","last_clean":"2024-03-06T07:31:22.074008+0000","last_became_active":"2024-03-06T07:31:21.055928+0000","last_became_peered":"2024-03-06T07:31:21.055928+0000","last_unstale":"2024-03-06T07:31:22.074008+0000","last_undegraded":"2024-03-06T07:31:22.074008+0000","last_fullsized":"2024-03-06T07:31:22.074008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:36.552847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.075024+0000","last_change":"2024-03-06T07:31:20.053091+0000","last_active":"2024-03-06T07:31:22.075024+0000","last_peered":"2024-03-06T07:31:22.075024+0000","last_clean":"2024-03-06T07:31:22.075024+0000","last_became_active":"2024-03-06T07:31:20.052758+0000","last_became_peered":"2024-03-06T07:31:20.052758+0000","last_unstale":"2024-03-06T07:31:22.075024+0000","last_undegraded":"2024-03-06T07:31:22.075024+0000","last_fullsized":"2024-03-06T07:31:22.075024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:26.744005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.072257+0000","last_change":"2024-03-06T07:31:21.058169+0000","last_active":"2024-03-06T07:31:22.072257+0000","last_peered":"2024-03-06T07:31:22.072257+0000","last_clean":"2024-03-06T07:31:22.072257+0000","last_became_active":"2024-03-06T07:31:21.057552+0000","last_became_peered":"2024-03-06T07:31:21.057552+0000","last_unstale":"2024-03-06T07:31:22.072257+0000","last_undegraded":"2024-03-06T07:31:22.072257+0000","last_fullsized":"2024-03-06T07:31:22.072257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:20.014001+0000","last_clean_scrub_stamp":"2024-03-06T07:31:20.014001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:32.727509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.074916+0000","last_change":"2024-03-06T07:31:20.054533+0000","last_active":"2024-03-06T07:31:22.074916+0000","last_peered":"2024-03-06T07:31:22.074916+0000","last_clean":"2024-03-06T07:31:22.074916+0000","last_became_active":"2024-03-06T07:31:20.054264+0000","last_became_peered":"2024-03-06T07:31:20.054264+0000","last_unstale":"2024-03-06T07:31:22.074916+0000","last_undegraded":"2024-03-06T07:31:22.074916+0000","last_fullsized":"2024-03-06T07:31:22.074916+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:18.998889+0000","last_clean_scrub_stamp":"2024-03-06T07:31:18.998889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:18.460944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":253,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-06T07:31:22.130893+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:31:22.130893+0000","last_peered":"2024-03-06T07:31:22.130893+0000","last_clean":"2024-03-06T07:31:22.130893+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:31:22.130893+0000","last_undegraded":"2024-03-06T07:31:22.130893+0000","last_fullsized":"2024-03-06T07:31:22.130893+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":927346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":336454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":336452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":336432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":336474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574895,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":337974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":823296,"data_stored":338590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1396736,"data_stored":926730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:26.171 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:31:26.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:31:26.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.481+0000 7f84a1000640 1 Processor -- start 2024-03-06T07:31:26.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.482+0000 7f84a1000640 1 -- start start 2024-03-06T07:31:26.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.483+0000 7f84a1000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f849c0d2900 0x7f849c0ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.483+0000 7f84a1000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c0b3340 con 0x7f849c0b50f0 2024-03-06T07:31:26.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.483+0000 7f84a1000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c0d1ed0 con 0x7f849c15e270 2024-03-06T07:31:26.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.483+0000 7f84a1000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849c0b3600 con 0x7f849c0d2900 2024-03-06T07:31:26.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f849c0b50f0 0x7f849c0ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36148/0 (socket says 172.21.15.114:36148) 2024-03-06T07:31:26.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849be00640 1 -- 172.21.15.114:0/294217444 learned_addr learned my addr 172.21.15.114:0/294217444 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:26.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10769664 0 0) 0x7f849c0b3340 con 0x7f849c0b50f0 2024-03-06T07:31:26.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8480003520 con 0x7f849c0b50f0 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849aa00640 1 --2- 172.21.15.114:0/294217444 >> v2:172.21.15.177:3300/0 conn(0x7f849c0d2900 0x7f849c0ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1737929 0 0) 0x7f8480003520 con 0x7f849c0b50f0 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 >> v2:172.21.15.177:3300/0 conn(0x7f849c0d2900 msgr2=0x7f849c0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 --2- 172.21.15.114:0/294217444 >> v2:172.21.15.177:3300/0 conn(0x7f849c0d2900 0x7f849c0ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 >> v1:172.21.15.177:6789/0 conn(0x7f849c15e270 legacy=0x7f849c0d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c0b38e0 con 0x7f849c0b50f0 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8494001670 con 0x7f849c0b50f0 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.484+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 >> v1:172.21.15.114:6789/0 conn(0x7f849c0b50f0 legacy=0x7f849c0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849a000640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84800039b0 0x7f8480003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849a000640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8480004800 0x7f8480004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849a000640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800053c0 0x7f848000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0b3600 con 0x7f84800039b0 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848000c250 con 0x7f84800053c0 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848000c450 con 0x7f8480004800 2024-03-06T07:31:26.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849b400640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84800039b0 0x7f8480003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849be00640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800053c0 0x7f848000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849aa00640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8480004800 0x7f8480004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849b400640 1 -- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8480004800 msgr2=0x7f8480004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849b400640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8480004800 0x7f8480004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849b400640 1 -- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800053c0 msgr2=0x7f848000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849b400640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800053c0 0x7f848000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849b400640 1 -- 172.21.15.114:0/294217444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480003520 con 0x7f84800039b0 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849be00640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800053c0 0x7f848000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:26.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.485+0000 7f849be00640 1 -- 172.21.15.114:0/294217444 reap_dead start 2024-03-06T07:31:26.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.486+0000 7f849b400640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84800039b0 0x7f8480003dd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f848c001e50 tx=0x7f8480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fbbf576dafd077 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c001a70 con 0x7f84800039b0 2024-03-06T07:31:26.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c003b70 con 0x7f84800039b0 2024-03-06T07:31:26.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f849a000640 1 -- 172.21.15.114:0/294217444 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c0050d0 con 0x7f84800039b0 2024-03-06T07:31:26.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f84a1000640 1 -- 172.21.15.114:0/294217444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84800039b0 msgr2=0x7f8480003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f84a1000640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84800039b0 0x7f8480003dd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f848c001e50 tx=0x7f8480005830 comp rx=0 tx=0).stop 2024-03-06T07:31:26.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f84a1000640 1 -- 172.21.15.114:0/294217444 shutdown_connections 2024-03-06T07:31:26.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f84a1000640 1 --2- 172.21.15.114:0/294217444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84800039b0 0x7f8480003dd0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f84a1000640 1 -- 172.21.15.114:0/294217444 >> 172.21.15.114:0/294217444 conn(0x7f849c0d3dc0 msgr2=0x7f849c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f84a1000640 1 -- 172.21.15.114:0/294217444 shutdown_connections 2024-03-06T07:31:26.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.487+0000 7f84a1000640 1 -- 172.21.15.114:0/294217444 wait complete. 2024-03-06T07:31:26.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.489+0000 7f84a1000640 1 Processor -- start 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.489+0000 7f84a1000640 1 -- start start 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f84a1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800039b0 0x7f849c1570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f84a1000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c0b98a0 0x7f849c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f84a1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0b0f10 0x7f849c0b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f84a1000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c15e650 con 0x7f849c0b0f10 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f84a1000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0d1ba0 con 0x7f84800039b0 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f84a1000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0d48e0 con 0x7f849c0b98a0 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f849be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0b0f10 0x7f849c0b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f849be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0b0f10 0x7f849c0b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57102/0 (socket says 172.21.15.114:57102) 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.490+0000 7f849be00640 1 -- 172.21.15.114:0/75993461 learned_addr learned my addr 172.21.15.114:0/75993461 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849aa00640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c0b98a0 0x7f849c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849b400640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800039b0 0x7f849c1570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849be00640 1 -- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c0b98a0 msgr2=0x7f849c0b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849be00640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c0b98a0 0x7f849c0b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849be00640 1 -- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800039b0 msgr2=0x7f849c1570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849be00640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800039b0 0x7f849c1570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849be00640 1 -- 172.21.15.114:0/75993461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c161230 con 0x7f849c0b0f10 2024-03-06T07:31:26.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f849be00640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0b0f10 0x7f849c0b13a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f849c0a1d40 tx=0x7f849400d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.491+0000 7f8493400640 1 -- 172.21.15.114:0/75993461 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494007440 con 0x7f849c0b0f10 2024-03-06T07:31:26.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.492+0000 7f8493400640 1 -- 172.21.15.114:0/75993461 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8494007600 con 0x7f849c0b0f10 2024-03-06T07:31:26.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.492+0000 7f8493400640 1 -- 172.21.15.114:0/75993461 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494004a20 con 0x7f849c0b0f10 2024-03-06T07:31:26.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.492+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c161510 con 0x7f849c0b0f10 2024-03-06T07:31:26.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.492+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c0b6e80 con 0x7f849c0b0f10 2024-03-06T07:31:26.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.493+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c075450 con 0x7f849c0b0f10 2024-03-06T07:31:26.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.494+0000 7f8493400640 1 -- 172.21.15.114:0/75993461 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f849401b020 con 0x7f849c0b0f10 2024-03-06T07:31:26.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.494+0000 7f8493400640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847806b4d0 0x7f847806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.495+0000 7f8493400640 1 -- 172.21.15.114:0/75993461 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8494089c90 con 0x7f849c0b0f10 2024-03-06T07:31:26.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.495+0000 7f849b400640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847806b4d0 0x7f847806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.495+0000 7f849b400640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847806b4d0 0x7f847806d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f848c034000 tx=0x7f848c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.499+0000 7f8493400640 1 -- 172.21.15.114:0/75993461 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8494058a70 con 0x7f849c0b0f10 2024-03-06T07:31:26.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.765+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f849c0b3340 con 0x7f849c0b0f10 2024-03-06T07:31:26.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.768+0000 7f8493400640 1 -- 172.21.15.114:0/75993461 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8494058370 con 0x7f849c0b0f10 2024-03-06T07:31:26.771 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:31:26.771 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:31:26.771 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:31:26.771 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:31:26.771 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:26.772 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.771+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847806b4d0 msgr2=0x7f847806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.771+0000 7f84a1000640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847806b4d0 0x7f847806d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f848c034000 tx=0x7f848c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0b0f10 msgr2=0x7f849c0b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0b0f10 0x7f849c0b13a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f849c0a1d40 tx=0x7f849400d950 comp rx=0 tx=0).stop 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 shutdown_connections 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847806b4d0 0x7f847806d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c0b0f10 0x7f849c0b13a0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c0b98a0 0x7f849c0b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 --2- 172.21.15.114:0/75993461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84800039b0 0x7f849c1570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 >> 172.21.15.114:0/75993461 conn(0x7f849c0d3dc0 msgr2=0x7f849c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 shutdown_connections 2024-03-06T07:31:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.772+0000 7f84a1000640 1 -- 172.21.15.114:0/75993461 wait complete. 2024-03-06T07:31:26.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:26.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.982+0000 7f24c2200640 1 Processor -- start 2024-03-06T07:31:26.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.982+0000 7f24c2200640 1 -- start start 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.983+0000 7f24c2200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f24bc161580 0x7f24bc0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.983+0000 7f24c2200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f24bc0d2a70 0x7f24bc0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.983+0000 7f24c2200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24bc161960 con 0x7f24bc0d1e90 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.983+0000 7f24c2200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24bc0cf6f0 con 0x7f24bc161580 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.983+0000 7f24c2200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24bc0cf9d0 con 0x7f24bc0d2a70 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24c0e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f24bc161580 0x7f24bc0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24bbe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f24bc0d1e90 0x7f24bc0d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36160/0 (socket says 172.21.15.114:36160) 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24bbe00640 1 -- 172.21.15.114:0/2551299793 learned_addr learned my addr 172.21.15.114:0/2551299793 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24c1800640 1 --2- 172.21.15.114:0/2551299793 >> v2:172.21.15.177:3300/0 conn(0x7f24bc0d2a70 0x7f24bc0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24c0e00640 1 -- 172.21.15.114:0/2551299793 >> v2:172.21.15.177:3300/0 conn(0x7f24bc0d2a70 msgr2=0x7f24bc0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24c0e00640 1 --2- 172.21.15.114:0/2551299793 >> v2:172.21.15.177:3300/0 conn(0x7f24bc0d2a70 0x7f24bc0b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24c0e00640 1 -- 172.21.15.114:0/2551299793 >> v1:172.21.15.114:6789/0 conn(0x7f24bc0d1e90 legacy=0x7f24bc0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.984+0000 7f24c0e00640 1 -- 172.21.15.114:0/2551299793 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0cfcb0 con 0x7f24bc161580 2024-03-06T07:31:26.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.985+0000 7f24c0e00640 1 --2- 172.21.15.114:0/2551299793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc161580 0x7f24bc0d1950 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f24bc08d180 tx=0x7f24b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61aad55a26906671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24bb400640 1 -- 172.21.15.114:0/2551299793 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0035020 con 0x7f24bc161580 2024-03-06T07:31:26.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24bb400640 1 -- 172.21.15.114:0/2551299793 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b0031830 con 0x7f24bc161580 2024-03-06T07:31:26.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24bb400640 1 -- 172.21.15.114:0/2551299793 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0031c00 con 0x7f24bc161580 2024-03-06T07:31:26.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 -- 172.21.15.114:0/2551299793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc161580 msgr2=0x7f24bc0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 --2- 172.21.15.114:0/2551299793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc161580 0x7f24bc0d1950 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f24bc08d180 tx=0x7f24b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 -- 172.21.15.114:0/2551299793 shutdown_connections 2024-03-06T07:31:26.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 --2- 172.21.15.114:0/2551299793 >> v2:172.21.15.177:3300/0 conn(0x7f24bc0d2a70 0x7f24bc0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 --2- 172.21.15.114:0/2551299793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc161580 0x7f24bc0d1950 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 -- 172.21.15.114:0/2551299793 >> 172.21.15.114:0/2551299793 conn(0x7f24bc0d3d80 msgr2=0x7f24bc0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 -- 172.21.15.114:0/2551299793 shutdown_connections 2024-03-06T07:31:26.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.986+0000 7f24c2200640 1 -- 172.21.15.114:0/2551299793 wait complete. 2024-03-06T07:31:26.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.988+0000 7f24c2200640 1 Processor -- start 2024-03-06T07:31:26.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.988+0000 7f24c2200640 1 -- start start 2024-03-06T07:31:26.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.989+0000 7f24c2200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc0d1e90 0x7f24bc0c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.989+0000 7f24c2200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24bc0d2a70 0x7f24bc0b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.989+0000 7f24c2200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24bc161580 0x7f24bc1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.989+0000 7f24c2200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0cf3a0 con 0x7f24bc0d1e90 2024-03-06T07:31:26.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.989+0000 7f24c2200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0b39a0 con 0x7f24bc0d2a70 2024-03-06T07:31:26.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.989+0000 7f24c2200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0b4400 con 0x7f24bc161580 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc0d1e90 0x7f24bc0c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc0d1e90 0x7f24bc0c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57120/0 (socket says 172.21.15.114:57120) 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 -- 172.21.15.114:0/3853075370 learned_addr learned my addr 172.21.15.114:0/3853075370 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 -- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24bc161580 msgr2=0x7f24bc1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c1800640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24bc161580 0x7f24bc1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24bc161580 0x7f24bc1513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24bbe00640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24bc0d2a70 0x7f24bc0b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 -- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24bc0d2a70 msgr2=0x7f24bc0b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24bc0d2a70 0x7f24bc0b89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.990+0000 7f24c0e00640 1 -- 172.21.15.114:0/3853075370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0c8870 con 0x7f24bc0d1e90 2024-03-06T07:31:26.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.991+0000 7f24c0e00640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc0d1e90 0x7f24bc0c81b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f24b0000c00 tx=0x7f24b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.991+0000 7f24b8c00640 1 -- 172.21.15.114:0/3853075370 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0040070 con 0x7f24bc0d1e90 2024-03-06T07:31:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.991+0000 7f24b8c00640 1 -- 172.21.15.114:0/3853075370 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b0039a20 con 0x7f24bc0d1e90 2024-03-06T07:31:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.991+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc0c8b50 con 0x7f24bc0d1e90 2024-03-06T07:31:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.991+0000 7f24b8c00640 1 -- 172.21.15.114:0/3853075370 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0035040 con 0x7f24bc0d1e90 2024-03-06T07:31:26.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.991+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24bc150ea0 con 0x7f24bc0d1e90 2024-03-06T07:31:26.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.992+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24bc04f330 con 0x7f24bc0d1e90 2024-03-06T07:31:26.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.994+0000 7f24b8c00640 1 -- 172.21.15.114:0/3853075370 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24b004b060 con 0x7f24bc0d1e90 2024-03-06T07:31:26.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.995+0000 7f24b8c00640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f249c06b4d0 0x7f249c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.995+0000 7f24b8c00640 1 -- 172.21.15.114:0/3853075370 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24b00abab0 con 0x7f24bc0d1e90 2024-03-06T07:31:26.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.996+0000 7f24bbe00640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f249c06b4d0 0x7f249c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:26.996+0000 7f24bbe00640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f249c06b4d0 0x7f249c06d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f24bc154ff0 tx=0x7f24a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T07:31:26.997+0000 7fafa1e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:27.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.000+0000 7f24b8c00640 1 -- 172.21.15.114:0/3853075370 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24b007a930 con 0x7f24bc0d1e90 2024-03-06T07:31:27.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.245+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24bc0d3b60 con 0x7f24bc0d1e90 2024-03-06T07:31:27.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.246+0000 7f24b8c00640 1 -- 172.21.15.114:0/3853075370 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f24b007a230 con 0x7f24bc0d1e90 2024-03-06T07:31:27.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:27.248 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:27.248 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f249c06b4d0 msgr2=0x7f249c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f249c06b4d0 0x7f249c06d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f24bc154ff0 tx=0x7f24a8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc0d1e90 msgr2=0x7f24bc0c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc0d1e90 0x7f24bc0c81b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f24b0000c00 tx=0x7f24b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 shutdown_connections 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f24bc161580 0x7f24bc1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24bc0d2a70 0x7f24bc0b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f249c06b4d0 0x7f249c06d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 --2- 172.21.15.114:0/3853075370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24bc0d1e90 0x7f24bc0c81b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 >> 172.21.15.114:0/3853075370 conn(0x7f24bc0d3d80 msgr2=0x7f24bc0b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 shutdown_connections 2024-03-06T07:31:27.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.249+0000 7f24c2200640 1 -- 172.21.15.114:0/3853075370 wait complete. 2024-03-06T07:31:27.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:20.504747+0000', 'modified': '2024-03-06T07:31:23.698546+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6354}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6354': {'gid': 6354, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/3105808425', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 3105808425}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 3105808425}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6354, 'qdb_cluster': [6354]} 2024-03-06T07:31:27.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:31:27.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:31:27.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.882+0000 7fb881600640 1 Processor -- start 2024-03-06T07:31:27.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.882+0000 7fb881600640 1 -- start start 2024-03-06T07:31:27.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb881600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb87c0d1cf0 0x7fb87c0d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb881600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0cf940 con 0x7fb87c0d2c00 2024-03-06T07:31:27.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb881600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0cfc20 con 0x7fb87c161ad0 2024-03-06T07:31:27.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb881600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c0cff00 con 0x7fb87c0d1cf0 2024-03-06T07:31:27.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb87a000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb87c0d2c00 0x7fb87c0b0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43806/0 (socket says 172.21.15.114:43806) 2024-03-06T07:31:27.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb87a000640 1 -- 172.21.15.114:0/4097087732 learned_addr learned my addr 172.21.15.114:0/4097087732 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807098974 0 0) 0x7fb87c0cf940 con 0x7fb87c0d2c00 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb860003520 con 0x7fb87c0d2c00 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.884+0000 7fb87b400640 1 --2- 172.21.15.114:0/4097087732 >> v2:172.21.15.177:3300/0 conn(0x7fb87c0d1cf0 0x7fb87c0d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 679108797 0 0) 0x7fb860003520 con 0x7fb87c0d2c00 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 >> v2:172.21.15.177:3300/0 conn(0x7fb87c0d1cf0 msgr2=0x7fb87c0d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 --2- 172.21.15.114:0/4097087732 >> v2:172.21.15.177:3300/0 conn(0x7fb87c0d1cf0 0x7fb87c0d4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 >> v1:172.21.15.177:6789/0 conn(0x7fb87c161ad0 legacy=0x7fb87c0d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0d01e0 con 0x7fb87c0d2c00 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb868001670 con 0x7fb87c0d2c00 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 >> v1:172.21.15.114:6789/0 conn(0x7fb87c0d2c00 legacy=0x7fb87c0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8600039b0 0x7fb860003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb8600052e0 0x7fb86000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0cff00 con 0x7fb860004800 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86000c1b0 con 0x7fb8600039b0 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86000c3b0 con 0x7fb8600052e0 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb87a000640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.885+0000 7fb87aa00640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8600039b0 0x7fb860003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.886+0000 7fb87a000640 1 -- 172.21.15.114:0/4097087732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb8600052e0 msgr2=0x7fb86000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.886+0000 7fb87a000640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb8600052e0 0x7fb86000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.886+0000 7fb87a000640 1 -- 172.21.15.114:0/4097087732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8600039b0 msgr2=0x7fb860003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.886+0000 7fb87a000640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8600039b0 0x7fb860003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.886+0000 7fb87a000640 1 -- 172.21.15.114:0/4097087732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb860003520 con 0x7fb860004800 2024-03-06T07:31:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.886+0000 7fb87b400640 1 -- 172.21.15.114:0/4097087732 reap_dead start 2024-03-06T07:31:27.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.887+0000 7fb87a000640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb860004800 0x7fb860004c80 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb868001eb0 tx=0x7fb860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3a075fb88daa07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.887+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868007d30 con 0x7fb860004800 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.887+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868030070 con 0x7fb860004800 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.887+0000 7fb879600640 1 -- 172.21.15.114:0/4097087732 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868005d70 con 0x7fb860004800 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.888+0000 7fb881600640 1 -- 172.21.15.114:0/4097087732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb860004800 msgr2=0x7fb860004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.888+0000 7fb881600640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb860004800 0x7fb860004c80 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fb868001eb0 tx=0x7fb860005830 comp rx=0 tx=0).stop 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.888+0000 7fb881600640 1 -- 172.21.15.114:0/4097087732 shutdown_connections 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.888+0000 7fb881600640 1 --2- 172.21.15.114:0/4097087732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb860004800 0x7fb860004c80 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.888+0000 7fb881600640 1 -- 172.21.15.114:0/4097087732 >> 172.21.15.114:0/4097087732 conn(0x7fb87c0d3f10 msgr2=0x7fb87c0d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.888+0000 7fb881600640 1 -- 172.21.15.114:0/4097087732 shutdown_connections 2024-03-06T07:31:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.888+0000 7fb881600640 1 -- 172.21.15.114:0/4097087732 wait complete. 2024-03-06T07:31:27.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.890+0000 7fb881600640 1 Processor -- start 2024-03-06T07:31:27.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.890+0000 7fb881600640 1 -- start start 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb881600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb860004800 0x7fb87c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb881600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb87c0b8380 0x7fb87c1503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb881600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb87c0b6b00 0x7fb87c0b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb881600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0d4dc0 con 0x7fb87c0b6b00 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb881600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0d4c40 con 0x7fb87c0b8380 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb881600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0d24f0 con 0x7fb860004800 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb87b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb87c0b6b00 0x7fb87c0b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb87b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb87c0b6b00 0x7fb87c0b8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33102/0 (socket says 172.21.15.114:33102) 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb87b400640 1 -- 172.21.15.114:0/1186463958 learned_addr learned my addr 172.21.15.114:0/1186463958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb87aa00640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb860004800 0x7fb87c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.891+0000 7fb87a000640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb87c0b8380 0x7fb87c1503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb87b400640 1 -- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb860004800 msgr2=0x7fb87c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb87b400640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb860004800 0x7fb87c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb87b400640 1 -- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb87c0b8380 msgr2=0x7fb87c1503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb87b400640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb87c0b8380 0x7fb87c1503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb87b400640 1 -- 172.21.15.114:0/1186463958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c0b72e0 con 0x7fb87c0b6b00 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb87aa00640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb860004800 0x7fb87c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb87b400640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb87c0b6b00 0x7fb87c0b8a60 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb87c06b950 tx=0x7fb87400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb872a00640 1 -- 172.21.15.114:0/1186463958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87400ed90 con 0x7fb87c0b6b00 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb872a00640 1 -- 172.21.15.114:0/1186463958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874019070 con 0x7fb87c0b6b00 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb872a00640 1 -- 172.21.15.114:0/1186463958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874014660 con 0x7fb87c0b6b00 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c0b75c0 con 0x7fb87c0b6b00 2024-03-06T07:31:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.892+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c0b7b80 con 0x7fb87c0b6b00 2024-03-06T07:31:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.894+0000 7fb872a00640 1 -- 172.21.15.114:0/1186463958 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb874014800 con 0x7fb87c0b6b00 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.895+0000 7fb872a00640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.895+0000 7fb872a00640 1 -- 172.21.15.114:0/1186463958 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb87408a7d0 con 0x7fb87c0b6b00 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.895+0000 7fb87aa00640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.896+0000 7fb87aa00640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb85c06b4d0 0x7fb85c06d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0055f0 tx=0x7fb86c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.897+0000 7fb88438d740 1 Processor -- start 2024-03-06T07:31:27.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.897+0000 7fb88438d740 1 -- start start 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb88438d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec107c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 0x55cec10ac540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec107cc90 0x55cec10aca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb88438d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec0fe6600 con 0x55cec10a97b0 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec0f8f930 con 0x55cec10a9b90 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cec0fe26a0 con 0x55cec107cc90 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb87aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec107c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb87aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec107c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33118/0 (socket says 172.21.15.114:33118) 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.899+0000 7fb87aa00640 1 -- 172.21.15.114:0/3225233768 learned_addr learned my addr 172.21.15.114:0/3225233768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:27.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87b400640 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec107cc90 0x55cec10aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87aa00640 1 -- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec107cc90 msgr2=0x55cec10aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87aa00640 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec107cc90 0x55cec10aca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87a000640 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 0x55cec10ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87aa00640 1 -- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 msgr2=0x55cec10ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87aa00640 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 0x55cec10ac540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87aa00640 1 -- 172.21.15.114:0/3225233768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cec0cfb2d0 con 0x55cec10a97b0 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb87aa00640 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec107c750 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0016d0 tx=0x7fb86c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976acb33eadcbfe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb867000640 1 -- 172.21.15.114:0/3225233768 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c027070 con 0x55cec10a97b0 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb867000640 1 -- 172.21.15.114:0/3225233768 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c014870 con 0x55cec10a97b0 2024-03-06T07:31:27.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.900+0000 7fb867000640 1 -- 172.21.15.114:0/3225233768 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c014c20 con 0x55cec10a97b0 2024-03-06T07:31:27.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 -- 172.21.15.114:0/3225233768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 msgr2=0x55cec107c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec107c750 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0016d0 tx=0x7fb86c013000 comp rx=0 tx=0).stop 2024-03-06T07:31:27.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 -- 172.21.15.114:0/3225233768 shutdown_connections 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec107cc90 0x55cec10aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 0x55cec10ac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 --2- 172.21.15.114:0/3225233768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec107c750 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 -- 172.21.15.114:0/3225233768 >> 172.21.15.114:0/3225233768 conn(0x55cec107e9c0 msgr2=0x55cec10c7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 -- 172.21.15.114:0/3225233768 shutdown_connections 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.901+0000 7fb88438d740 1 -- 172.21.15.114:0/3225233768 wait complete. 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.902+0000 7fb88438d740 1 -- start start 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.902+0000 7fb88438d740 1 Processor -- start 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb88438d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec10af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a9b90 0x55cec10af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a8b70 0x55cec10c5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb88438d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec1080f60 con 0x55cec10a97b0 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec0fa2d90 con 0x55cec10a8b70 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cec1080350 con 0x55cec10a9b90 2024-03-06T07:31:27.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb88438d740 10 client.0 _ll_register_callbacks cb 0x55cec0fde350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb87aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec10af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb87aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec10af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33126/0 (socket says 172.21.15.114:33126) 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.903+0000 7fb87aa00640 1 -- 172.21.15.114:0/289274197 learned_addr learned my addr 172.21.15.114:0/289274197 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb87aa00640 1 -- 172.21.15.114:0/289274197 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a9b90 msgr2=0x55cec10af9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb87aa00640 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a9b90 0x55cec10af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb87aa00640 1 -- 172.21.15.114:0/289274197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a8b70 msgr2=0x55cec10c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb87aa00640 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a8b70 0x55cec10c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb87aa00640 1 -- 172.21.15.114:0/289274197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cec0dc56b0 con 0x55cec10a97b0 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb87aa00640 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec10af490 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb86c025040 tx=0x7fb86c001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f07e98ca50ae8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb85b400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:31:27.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb85b400640 1 -- 172.21.15.114:0/289274197 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c030020 con 0x55cec10a97b0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb85b400640 1 -- 172.21.15.114:0/289274197 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb86c023070 con 0x55cec10a97b0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.904+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb86c014870 con 0x55cec10a97b0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb85b400640 1 -- 172.21.15.114:0/289274197 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb86c027040 con 0x55cec10a97b0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 10 client.6498 fetch_fsmap learned FSMap version 44 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55cec10c3200 con 0x55cec10a97b0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb85b400640 1 -- 172.21.15.114:0/289274197 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c014870 con 0x55cec10a97b0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb85b400640 1 -- 172.21.15.114:0/289274197 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 44) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb86c03d020 con 0x55cec10a97b0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 10 client.6498 fetch_fsmap finished waiting for FSMap version 44 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 10 client.6498 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 20 client.6498 populate_metadata read hostname 'smithi114' 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c3ea0 0x55cec10e5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 4 client.6498 mds_command: new command op to 6354 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 --> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cec0ff3b40 con 0x55cec10c3ea0 2024-03-06T07:31:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.905+0000 7fb87a000640 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c3ea0 0x55cec10e5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.906+0000 7fb85b400640 10 client.6498 ms_handle_connect on v2:172.21.15.114:6836/3105808425 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.906+0000 7fb87a000640 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c3ea0 0x55cec10e5b90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84284525cf088071 server_cookie=c98195f88f36e284 in_seq=0 out_seq=0 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb85b400640 1 -- 172.21.15.114:0/289274197 <== mds.0 v2:172.21.15.114:6836/3105808425 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cec0ff3b40 con 0x55cec10c3ea0 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb85b400640 10 client.6498 handle_command_reply: tid=0 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 client.6498 shutdown 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 msgr2=0x55cec10af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec10af490 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb86c025040 tx=0x7fb86c001190 comp rx=0 tx=0).stop 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 shutdown_connections 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c3ea0 0x55cec10e5b90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a8b70 0x55cec10c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a9b90 0x55cec10af9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 --2- 172.21.15.114:0/289274197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec10af490 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 >> 172.21.15.114:0/289274197 conn(0x55cec107cc90 msgr2=0x55cec10ac000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb85b400640 0 client.6498 ms_handle_reset on v2:172.21.15.114:6836/3105808425 2024-03-06T07:31:27.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 shutdown_connections 2024-03-06T07:31:27.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 1 -- 172.21.15.114:0/289274197 wait complete. 2024-03-06T07:31:27.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.907+0000 7fb88438d740 20 client.6498 trim_cache size 0 max 16384 2024-03-06T07:31:27.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.925+0000 7fb88438d740 1 Processor -- start 2024-03-06T07:31:27.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.925+0000 7fb88438d740 1 -- start start 2024-03-06T07:31:27.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.926+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.926+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a97b0 0x55cec10e9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.926+0000 7fb88438d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a9b90 0x55cec10ea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.926+0000 7fb88438d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c9160 con 0x55cec10a9b90 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.926+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c8560 con 0x55cec10a97b0 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.926+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0c8b60 con 0x55cec10a8b70 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a9b90 0x55cec10ea0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a9b90 0x55cec10ea0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33132/0 (socket says 172.21.15.114:33132) 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 -- 172.21.15.114:0/56793677 learned_addr learned my addr 172.21.15.114:0/56793677 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87aa00640 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87a000640 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a97b0 0x55cec10e9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 -- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 msgr2=0x55cec10c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 -- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a97b0 msgr2=0x55cec10e9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a97b0 0x55cec10e9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 -- 172.21.15.114:0/56793677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cec10c8570 con 0x55cec10a9b90 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.927+0000 7fb87b400640 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a9b90 0x55cec10ea0f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb874002230 tx=0x7fb8740595c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618cceb7407d1234 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb866600640 1 -- 172.21.15.114:0/56793677 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87405d010 con 0x55cec10a9b90 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb866600640 1 -- 172.21.15.114:0/56793677 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874061660 con 0x55cec10a9b90 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb866600640 1 -- 172.21.15.114:0/56793677 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874012dc0 con 0x55cec10a9b90 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 -- 172.21.15.114:0/56793677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a9b90 msgr2=0x55cec10ea0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a9b90 0x55cec10ea0f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb874002230 tx=0x7fb8740595c0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 -- 172.21.15.114:0/56793677 shutdown_connections 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a9b90 0x55cec10ea0f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a97b0 0x55cec10e9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 --2- 172.21.15.114:0/56793677 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 -- 172.21.15.114:0/56793677 >> 172.21.15.114:0/56793677 conn(0x55cec107cc90 msgr2=0x55cec10c7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 -- 172.21.15.114:0/56793677 shutdown_connections 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.928+0000 7fb88438d740 1 -- 172.21.15.114:0/56793677 wait complete. 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.929+0000 7fb88438d740 1 -- start start 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.930+0000 7fb88438d740 1 Processor -- start 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.931+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10e0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.931+0000 7fb88438d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec11379e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.931+0000 7fb88438d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 0x55cec10c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.931+0000 7fb88438d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec10c3d20 con 0x55cec10a97b0 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec10d7d40 con 0x55cec10a9b90 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb88438d740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cec10c42a0 con 0x55cec10a8b70 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb88438d740 10 client.0 _ll_register_callbacks cb 0x7fb86c025070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec11379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec11379e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33136/0 (socket says 172.21.15.114:33136) 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 -- 172.21.15.114:0/1576108845 learned_addr learned my addr 172.21.15.114:0/1576108845 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87aa00640 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10e0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 -- 172.21.15.114:0/1576108845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 msgr2=0x55cec10e0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10e0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 -- 172.21.15.114:0/1576108845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 msgr2=0x55cec10c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 0x55cec10c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.932+0000 7fb87a000640 1 -- 172.21.15.114:0/1576108845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cec107b7c0 con 0x55cec10a97b0 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb87a000640 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec11379e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb86800c300 tx=0x7fb868001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fc39a9d46b67eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb85b400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:31:27.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb85b400640 1 -- 172.21.15.114:0/1576108845 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86804a060 con 0x55cec10a97b0 2024-03-06T07:31:27.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb85b400640 1 -- 172.21.15.114:0/1576108845 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb868001470 con 0x55cec10a97b0 2024-03-06T07:31:27.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb874061660 con 0x55cec10a97b0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb85b400640 1 -- 172.21.15.114:0/1576108845 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb868001690 con 0x55cec10a97b0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb88438d740 10 client.6504 fetch_fsmap learned FSMap version 44 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) v3 -- 0x55cec1082300 con 0x55cec10a97b0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.933+0000 7fb85b400640 1 -- 172.21.15.114:0/1576108845 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874061660 con 0x55cec10a97b0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb85b400640 1 -- 172.21.15.114:0/1576108845 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 44) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb86802c8b0 con 0x55cec10a97b0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb88438d740 10 client.6504 fetch_fsmap finished waiting for FSMap version 44 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb88438d740 10 client.6504 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb88438d740 20 client.6504 populate_metadata read hostname 'smithi114' 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb88438d740 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c4cc0 0x55cec113b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb88438d740 4 client.6504 mds_command: new command op to 6354 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 --> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cec0fede90 con 0x55cec10c4cc0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb87aa00640 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c4cc0 0x55cec113b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb87aa00640 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c4cc0 0x55cec113b6e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dcc4d33c8916441 server_cookie=80174dbcf3863ceb in_seq=0 out_seq=0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.934+0000 7fb85b400640 10 client.6504 ms_handle_connect on v2:172.21.15.114:6836/3105808425 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb85b400640 1 -- 172.21.15.114:0/1576108845 <== mds.0 v2:172.21.15.114:6836/3105808425 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55cec0fede90 con 0x55cec10c4cc0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb85b400640 10 client.6504 handle_command_reply: tid=0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 client.6504 shutdown 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 msgr2=0x55cec11379e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec11379e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb86800c300 tx=0x7fb868001a40 comp rx=0 tx=0).stop 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 shutdown_connections 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:6836/3105808425,v1:172.21.15.114:6837/3105808425] conn(0x55cec10c4cc0 0x55cec113b6e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cec10a9b90 0x55cec10c1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cec10a97b0 0x55cec11379e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 --2- 172.21.15.114:0/1576108845 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cec10a8b70 0x55cec10e0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb85b400640 0 client.6504 ms_handle_reset on v2:172.21.15.114:6836/3105808425 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 >> 172.21.15.114:0/1576108845 conn(0x55cec107cc90 msgr2=0x55cec10c1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 shutdown_connections 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 1 -- 172.21.15.114:0/1576108845 wait complete. 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.935+0000 7fb88438d740 20 client.6504 trim_cache size 0 max 16384 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 6354, 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 6.8764259489999997, 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 44, 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 66, 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 66, 2024-03-06T07:31:27.941 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 12.355168536000001 2024-03-06T07:31:27.941 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:31:27.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb85c06b4d0 msgr2=0x7fb85c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb85c06b4d0 0x7fb85c06d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0055f0 tx=0x7fb86c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb87c0b6b00 msgr2=0x7fb87c0b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb87c0b6b00 0x7fb87c0b8a60 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb87c06b950 tx=0x7fb87400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 shutdown_connections 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb87c0b6b00 0x7fb87c0b8a60 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb87c0b8380 0x7fb87c1503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 --2- 172.21.15.114:0/1186463958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb860004800 0x7fb87c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 >> 172.21.15.114:0/1186463958 conn(0x7fb87c0d3f10 msgr2=0x7fb87c160b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 shutdown_connections 2024-03-06T07:31:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:27.936+0000 7fb881600640 1 -- 172.21.15.114:0/1186463958 wait complete. 2024-03-06T07:31:27.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:31:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.193+0000 7fccd1800640 1 Processor -- start 2024-03-06T07:31:28.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.193+0000 7fccd1800640 1 -- start start 2024-03-06T07:31:28.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.195+0000 7fccd1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc15f950 0x7fcccc15cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.195+0000 7fccd1800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcccc06afc0 0x7fcccc06b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.195+0000 7fccd1800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc15d2f0 con 0x7fcccc145540 2024-03-06T07:31:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.195+0000 7fccd1800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0ce960 con 0x7fcccc06afc0 2024-03-06T07:31:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.195+0000 7fccd1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc15fed0 con 0x7fcccc15f950 2024-03-06T07:31:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fcccb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc15f950 0x7fcccc15cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fccd0e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcccc06afc0 0x7fcccc06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fcccb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc15f950 0x7fcccc15cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:41372/0 (socket says 172.21.15.114:41372) 2024-03-06T07:31:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fcccb400640 1 -- 172.21.15.114:0/3569154013 learned_addr learned my addr 172.21.15.114:0/3569154013 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fcccb400640 1 -- 172.21.15.114:0/3569154013 >> v2:172.21.15.177:3300/0 conn(0x7fcccc06afc0 msgr2=0x7fcccc06b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fcccb400640 1 --2- 172.21.15.114:0/3569154013 >> v2:172.21.15.177:3300/0 conn(0x7fcccc06afc0 0x7fcccc06b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fcccb400640 1 -- 172.21.15.114:0/3569154013 >> v1:172.21.15.177:6789/0 conn(0x7fcccc145540 legacy=0x7fcccc06e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.196+0000 7fcccb400640 1 -- 172.21.15.114:0/3569154013 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc1601b0 con 0x7fcccc15f950 2024-03-06T07:31:28.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.198+0000 7fcccb400640 1 --2- 172.21.15.114:0/3569154013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc15f950 0x7fcccc15cdb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009870 tx=0x7fccb802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7a04c21db4ed57f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.198+0000 7fcccaa00640 1 -- 172.21.15.114:0/3569154013 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8034070 con 0x7fcccc15f950 2024-03-06T07:31:28.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fcccaa00640 1 -- 172.21.15.114:0/3569154013 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb8030840 con 0x7fcccc15f950 2024-03-06T07:31:28.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fcccaa00640 1 -- 172.21.15.114:0/3569154013 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8030c10 con 0x7fcccc15f950 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 -- 172.21.15.114:0/3569154013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc15f950 msgr2=0x7fcccc15cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 --2- 172.21.15.114:0/3569154013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc15f950 0x7fcccc15cdb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009870 tx=0x7fccb802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 -- 172.21.15.114:0/3569154013 shutdown_connections 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 --2- 172.21.15.114:0/3569154013 >> v2:172.21.15.177:3300/0 conn(0x7fcccc06afc0 0x7fcccc06b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 --2- 172.21.15.114:0/3569154013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc15f950 0x7fcccc15cdb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 -- 172.21.15.114:0/3569154013 >> 172.21.15.114:0/3569154013 conn(0x7fcccc06c0d0 msgr2=0x7fcccc06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 -- 172.21.15.114:0/3569154013 shutdown_connections 2024-03-06T07:31:28.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.199+0000 7fccd1800640 1 -- 172.21.15.114:0/3569154013 wait complete. 2024-03-06T07:31:28.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.201+0000 7fccd1800640 1 Processor -- start 2024-03-06T07:31:28.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.202+0000 7fccd1800640 1 -- start start 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc06afc0 0x7fcccc0b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc145540 0x7fcccc0aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcccc15f950 0x7fcccc15ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc145920 con 0x7fcccc06afc0 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd1800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc160770 con 0x7fcccc15f950 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0cedf0 con 0x7fcccc145540 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc06afc0 0x7fcccc0b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc06afc0 0x7fcccc0b56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33150/0 (socket says 172.21.15.114:33150) 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 -- 172.21.15.114:0/2171880485 learned_addr learned my addr 172.21.15.114:0/2171880485 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccb400640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc145540 0x7fcccc0aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd0e00640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcccc15f950 0x7fcccc15ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 -- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc145540 msgr2=0x7fcccc0aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc145540 0x7fcccc0aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 -- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcccc15f950 msgr2=0x7fcccc15ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcccc15f950 0x7fcccc15ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fcccbe00640 1 -- 172.21.15.114:0/2171880485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0af330 con 0x7fcccc06afc0 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.203+0000 7fccd0e00640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcccc15f950 0x7fcccc15ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.204+0000 7fcccbe00640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc06afc0 0x7fcccc0b56e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00c400 tx=0x7fccbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.204+0000 7fccc3e00640 1 -- 172.21.15.114:0/2171880485 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0052d0 con 0x7fcccc06afc0 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.204+0000 7fccc3e00640 1 -- 172.21.15.114:0/2171880485 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc005490 con 0x7fcccc06afc0 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.204+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc0af610 con 0x7fcccc06afc0 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.204+0000 7fccc3e00640 1 -- 172.21.15.114:0/2171880485 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc007260 con 0x7fcccc06afc0 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.204+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc0afbd0 con 0x7fcccc06afc0 2024-03-06T07:31:28.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.206+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc04f330 con 0x7fcccc06afc0 2024-03-06T07:31:28.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.207+0000 7fccc3e00640 1 -- 172.21.15.114:0/2171880485 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccbc007400 con 0x7fcccc06afc0 2024-03-06T07:31:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.208+0000 7fccc3e00640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccac06b4d0 0x7fccac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.208+0000 7fccc3e00640 1 -- 172.21.15.114:0/2171880485 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccbc089780 con 0x7fcccc06afc0 2024-03-06T07:31:28.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.209+0000 7fcccb400640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccac06b4d0 0x7fccac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.209+0000 7fcccb400640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccac06b4d0 0x7fccac06d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fccb8036000 tx=0x7fccb803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.213+0000 7fccc3e00640 1 -- 172.21.15.114:0/2171880485 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccbc058600 con 0x7fcccc06afc0 2024-03-06T07:31:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.461+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcccc06bc60 con 0x7fcccc06afc0 2024-03-06T07:31:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.462+0000 7fccc3e00640 1 -- 172.21.15.114:0/2171880485 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) v1 ==== 53+0+73 (secure 0 0 0) 0x7fccbc057f00 con 0x7fcccc06afc0 2024-03-06T07:31:28.464 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:31:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccac06b4d0 msgr2=0x7fccac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccac06b4d0 0x7fccac06d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fccb8036000 tx=0x7fccb803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc06afc0 msgr2=0x7fcccc0b56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc06afc0 0x7fcccc0b56e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00c400 tx=0x7fccbc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 shutdown_connections 2024-03-06T07:31:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcccc15f950 0x7fcccc15ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcccc145540 0x7fcccc0aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccac06b4d0 0x7fccac06d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 --2- 172.21.15.114:0/2171880485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcccc06afc0 0x7fcccc0b56e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.464+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 >> 172.21.15.114:0/2171880485 conn(0x7fcccc06c0d0 msgr2=0x7fcccc072280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.465+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 shutdown_connections 2024-03-06T07:31:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:28.465+0000 7fccd1800640 1 -- 172.21.15.114:0/2171880485 wait complete. 2024-03-06T07:31:28.483 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:31:28.483 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:31:28.484 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:31:28.484 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:31:28.484 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:28.484 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:31:28.484 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:31:28.484 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:31:28.484 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:28.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.484 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:31:28.524 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:31:28.524 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:31:28.524 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:31:28.524 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:28.524 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42367sec preferred_lft 42367sec 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:31:28.525 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:31:28.526 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:28.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.528 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:31:28.647 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:31:28.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.653 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:31:28.701 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:31:28.701 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:31:28.701 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:31:28.701 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:31:28.701 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:31:28.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.701 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:31:28.701 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:31:28.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:31:28.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:31:28.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:31:28.702 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:31:28.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.878 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:31:28.924 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:28.924 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:31:28.925 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:28.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:28.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:31:29.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:29.004 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:29.048 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:29.048 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:29.048 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:29.135 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:31:29.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:29.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:29.311 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:29.312 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:29.345 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:29.345 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:29.345 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:31:29.345 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:31:29.345 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:31:29.347 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:31:29.347 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:31:29.347 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:31:29.347 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:31:29.348 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:31:29.351 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:31:29.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:29.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:29.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:29.434 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:31:29.497 INFO:teuthology.orchestra.run.smithi114.stdout:client5413 2024-03-06T07:31:29.497 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:29.497 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:29.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:31:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.788+0000 7f728d400640 1 Processor -- start 2024-03-06T07:31:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.788+0000 7f728d400640 1 -- start start 2024-03-06T07:31:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.790+0000 7f728d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 0x7f7288070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.790+0000 7f728d400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72880b46c0 con 0x7f728815fff0 2024-03-06T07:31:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.790+0000 7f728d400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72880b49a0 con 0x7f72880b3f70 2024-03-06T07:31:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.790+0000 7f728d400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7288072c90 con 0x7f728815fc10 2024-03-06T07:31:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.790+0000 7f727de00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f728815fff0 0x7f7288160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43810/0 (socket says 172.21.15.114:43810) 2024-03-06T07:31:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.790+0000 7f727de00640 1 -- 172.21.15.114:0/2277139112 learned_addr learned my addr 172.21.15.114:0/2277139112 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7286800640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 0x7f7288070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824726019 0 0) 0x7f72880b46c0 con 0x7f728815fff0 2024-03-06T07:31:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f726c003520 con 0x7f728815fff0 2024-03-06T07:31:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215082306 0 0) 0x7f726c003520 con 0x7f728815fff0 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 msgr2=0x7f7288070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 0x7f7288070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 >> v1:172.21.15.177:6789/0 conn(0x7f72880b3f70 legacy=0x7f72880ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728809dd80 con 0x7f728815fff0 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7270001670 con 0x7f728815fff0 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 >> v1:172.21.15.114:6789/0 conn(0x7f728815fff0 legacy=0x7f7288160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 0x7f726c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f726c003f60 0x7f726c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726c004d40 0x7f726c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7288072c90 con 0x7f726c003f60 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c00baa0 con 0x7f726c004d40 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.791+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c00bca0 con 0x7f728815fc10 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f727de00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f726c003f60 0x7f726c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f7286800640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 0x7f726c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f7287200640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726c004d40 0x7f726c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f727de00640 1 -- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 msgr2=0x7f726c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f727de00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 0x7f726c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f727de00640 1 -- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726c004d40 msgr2=0x7f726c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f727de00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726c004d40 0x7f726c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f727de00640 1 -- 172.21.15.114:0/2277139112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c003520 con 0x7f726c003f60 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f7286800640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f728815fc10 0x7f726c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.792+0000 7f7287200640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726c004d40 0x7f726c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.793+0000 7f727de00640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f726c003f60 0x7f726c004800 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f727000c210 tx=0x7f726c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bc770cb7688962 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.793+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270007d30 con 0x7f726c003f60 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.793+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7270030070 con 0x7f726c003f60 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f7285e00640 1 -- 172.21.15.114:0/2277139112 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727002cda0 con 0x7f726c003f60 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f728d400640 1 -- 172.21.15.114:0/2277139112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f726c003f60 msgr2=0x7f726c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f728d400640 1 --2- 172.21.15.114:0/2277139112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f726c003f60 0x7f726c004800 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f727000c210 tx=0x7f726c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f7287200640 1 -- 172.21.15.114:0/2277139112 reap_dead start 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f728d400640 1 -- 172.21.15.114:0/2277139112 shutdown_connections 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f728d400640 1 -- 172.21.15.114:0/2277139112 >> 172.21.15.114:0/2277139112 conn(0x7f72880b1940 msgr2=0x7f72880b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f728d400640 1 -- 172.21.15.114:0/2277139112 shutdown_connections 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.794+0000 7f728d400640 1 -- 172.21.15.114:0/2277139112 wait complete. 2024-03-06T07:31:29.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.796+0000 7f728d400640 1 Processor -- start 2024-03-06T07:31:29.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.796+0000 7f728d400640 1 -- start start 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f728d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f726c003f60 0x7f728809ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f728d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728814ec50 0x7f728814f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f728d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f728806d9e0 0x7f728806de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f728d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728815c460 con 0x7f728814ec50 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f728d400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728815c2e0 con 0x7f728806d9e0 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f728d400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7288070ff0 con 0x7f726c003f60 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f727de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728814ec50 0x7f728814f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f727de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728814ec50 0x7f728814f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33170/0 (socket says 172.21.15.114:33170) 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.797+0000 7f727de00640 1 -- 172.21.15.114:0/2301226355 learned_addr learned my addr 172.21.15.114:0/2301226355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727de00640 1 -- 172.21.15.114:0/2301226355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f726c003f60 msgr2=0x7f728809ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727de00640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f726c003f60 0x7f728809ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727de00640 1 -- 172.21.15.114:0/2301226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f728806d9e0 msgr2=0x7f728806de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f7287200640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f728806d9e0 0x7f728806de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727de00640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f728806d9e0 0x7f728806de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727de00640 1 -- 172.21.15.114:0/2301226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728814de90 con 0x7f728814ec50 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727de00640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728814ec50 0x7f728814f0d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f727002c570 tx=0x7f7270001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727f400640 1 -- 172.21.15.114:0/2301226355 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270045070 con 0x7f728814ec50 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727f400640 1 -- 172.21.15.114:0/2301226355 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7270000ed0 con 0x7f728814ec50 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728814e170 con 0x7f728814ec50 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.798+0000 7f727f400640 1 -- 172.21.15.114:0/2301226355 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270004e30 con 0x7f728814ec50 2024-03-06T07:31:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.799+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728806e910 con 0x7f728814ec50 2024-03-06T07:31:29.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.800+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728804f330 con 0x7f728814ec50 2024-03-06T07:31:29.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.801+0000 7f727f400640 1 -- 172.21.15.114:0/2301226355 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7270007070 con 0x7f728814ec50 2024-03-06T07:31:29.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.801+0000 7f727f400640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f726406b500 0x7f726406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.801+0000 7f727f400640 1 -- 172.21.15.114:0/2301226355 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72700ab420 con 0x7f728814ec50 2024-03-06T07:31:29.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.802+0000 7f7286800640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f726406b500 0x7f726406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.802+0000 7f7286800640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f726406b500 0x7f726406d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f72780099e0 tx=0x7f7278008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:29.806+0000 7f727f400640 1 -- 172.21.15.114:0/2301226355 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f727007a280 con 0x7f728814ec50 2024-03-06T07:31:30.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.002+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f72880b2d40 con 0x7f728814ec50 2024-03-06T07:31:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.008+0000 7f727f400640 1 -- 172.21.15.114:0/2301226355 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7270079b80 con 0x7f728814ec50 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f726406b500 msgr2=0x7f726406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f726406b500 0x7f726406d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f72780099e0 tx=0x7f7278008040 comp rx=0 tx=0).stop 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728814ec50 msgr2=0x7f728814f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728814ec50 0x7f728814f0d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f727002c570 tx=0x7f7270001a40 comp rx=0 tx=0).stop 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 shutdown_connections 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f728806d9e0 0x7f728806de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f726406b500 0x7f726406d9c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728814ec50 0x7f728814f0d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 --2- 172.21.15.114:0/2301226355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f726c003f60 0x7f728809ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 >> 172.21.15.114:0/2301226355 conn(0x7f72880b1940 msgr2=0x7f7288156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 shutdown_connections 2024-03-06T07:31:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:30.011+0000 7f728d400640 1 -- 172.21.15.114:0/2301226355 wait complete. 2024-03-06T07:31:40.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.248+0000 7f76de200640 1 Processor -- start 2024-03-06T07:31:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.248+0000 7f76de200640 1 -- start start 2024-03-06T07:31:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.249+0000 7f76de200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d80cf430 0x7f76d8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76de200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76d8071260 0x7f76d806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76de200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d80717c0 con 0x7f76d806d090 2024-03-06T07:31:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76de200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d806d4c0 con 0x7f76d8071260 2024-03-06T07:31:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76de200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8071aa0 con 0x7f76d80cf430 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76d8071260 0x7f76d806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76d8071260 0x7f76d806cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55528/0 (socket says 172.21.15.114:55528) 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d6a00640 1 -- 172.21.15.114:0/2347737631 learned_addr learned my addr 172.21.15.114:0/2347737631 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d7400640 1 --2- 172.21.15.114:0/2347737631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d80cf430 0x7f76d8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d6a00640 1 -- 172.21.15.114:0/2347737631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d80cf430 msgr2=0x7f76d8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d6a00640 1 --2- 172.21.15.114:0/2347737631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d80cf430 0x7f76d8070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d6a00640 1 -- 172.21.15.114:0/2347737631 >> v1:172.21.15.177:6789/0 conn(0x7f76d806d090 legacy=0x7f76d8165f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.250+0000 7f76d6a00640 1 -- 172.21.15.114:0/2347737631 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d8071d80 con 0x7f76d8071260 2024-03-06T07:31:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.252+0000 7f76d6a00640 1 --2- 172.21.15.114:0/2347737631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d8071260 0x7f76d806cb50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009870 tx=0x7f76c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2743df5214e62f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.252+0000 7f76d6000640 1 -- 172.21.15.114:0/2347737631 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4035020 con 0x7f76d8071260 2024-03-06T07:31:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.252+0000 7f76d6000640 1 -- 172.21.15.114:0/2347737631 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c4031640 con 0x7f76d8071260 2024-03-06T07:31:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.252+0000 7f76d6000640 1 -- 172.21.15.114:0/2347737631 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c402d040 con 0x7f76d8071260 2024-03-06T07:31:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 -- 172.21.15.114:0/2347737631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d8071260 msgr2=0x7f76d806cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 --2- 172.21.15.114:0/2347737631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d8071260 0x7f76d806cb50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009870 tx=0x7f76c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 -- 172.21.15.114:0/2347737631 shutdown_connections 2024-03-06T07:31:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 --2- 172.21.15.114:0/2347737631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d8071260 0x7f76d806cb50 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 --2- 172.21.15.114:0/2347737631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d80cf430 0x7f76d8070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 -- 172.21.15.114:0/2347737631 >> 172.21.15.114:0/2347737631 conn(0x7f76d80b1730 msgr2=0x7f76d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 -- 172.21.15.114:0/2347737631 shutdown_connections 2024-03-06T07:31:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.253+0000 7f76de200640 1 -- 172.21.15.114:0/2347737631 wait complete. 2024-03-06T07:31:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.255+0000 7f76de200640 1 Processor -- start 2024-03-06T07:31:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.255+0000 7f76de200640 1 -- start start 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.256+0000 7f76de200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d806d090 0x7f76d8166850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.256+0000 7f76de200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76d8071260 0x7f76d8143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.256+0000 7f76de200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d80cf430 0x7f76d81f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.256+0000 7f76de200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8162b70 con 0x7f76d80cf430 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.256+0000 7f76de200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8072340 con 0x7f76d8071260 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.256+0000 7f76de200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8162cf0 con 0x7f76d806d090 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d80cf430 0x7f76d81f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d80cf430 0x7f76d81f8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55530/0 (socket says 172.21.15.114:55530) 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 -- 172.21.15.114:0/1695439386 learned_addr learned my addr 172.21.15.114:0/1695439386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7400640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d806d090 0x7f76d8166850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d6a00640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76d8071260 0x7f76d8143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 -- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d806d090 msgr2=0x7f76d8166850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d806d090 0x7f76d8166850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 -- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76d8071260 msgr2=0x7f76d8143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76d8071260 0x7f76d8143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.257+0000 7f76d7e00640 1 -- 172.21.15.114:0/1695439386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d81fc770 con 0x7f76d80cf430 2024-03-06T07:31:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.258+0000 7f76d7e00640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d80cf430 0x7f76d81f8260 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f76d000c7b0 tx=0x7f76d000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.258+0000 7f76cf400640 1 -- 172.21.15.114:0/1695439386 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0017070 con 0x7f76d80cf430 2024-03-06T07:31:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.258+0000 7f76cf400640 1 -- 172.21.15.114:0/1695439386 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d0003da0 con 0x7f76d80cf430 2024-03-06T07:31:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.258+0000 7f76cf400640 1 -- 172.21.15.114:0/1695439386 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d00052b0 con 0x7f76d80cf430 2024-03-06T07:31:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.258+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d8165a90 con 0x7f76d80cf430 2024-03-06T07:31:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.258+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d8166050 con 0x7f76d80cf430 2024-03-06T07:31:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.259+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d804f330 con 0x7f76d80cf430 2024-03-06T07:31:40.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.261+0000 7f76cf400640 1 -- 172.21.15.114:0/1695439386 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76d0005450 con 0x7f76d80cf430 2024-03-06T07:31:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.262+0000 7f76cf400640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76b806b4d0 0x7f76b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.262+0000 7f76cf400640 1 -- 172.21.15.114:0/1695439386 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76d0089840 con 0x7f76d80cf430 2024-03-06T07:31:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.263+0000 7f76d7400640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76b806b4d0 0x7f76b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.263+0000 7f76d7400640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76b806b4d0 0x7f76b806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f76c8006fd0 tx=0x7f76c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.267+0000 7f76cf400640 1 -- 172.21.15.114:0/1695439386 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76d0058670 con 0x7f76d80cf430 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.501+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d80b4a40 con 0x7f76d80cf430 2024-03-06T07:31:40.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.501+0000 7f76cf400640 1 -- 172.21.15.114:0/1695439386 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f76d0057f70 con 0x7f76d80cf430 2024-03-06T07:31:40.503 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:40.503 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:40.504 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76b806b4d0 msgr2=0x7f76b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76b806b4d0 0x7f76b806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f76c8006fd0 tx=0x7f76c8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d80cf430 msgr2=0x7f76d81f8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d80cf430 0x7f76d81f8260 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f76d000c7b0 tx=0x7f76d000cc70 comp rx=0 tx=0).stop 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 shutdown_connections 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f76b806b4d0 0x7f76b806d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d80cf430 0x7f76d81f8260 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76d8071260 0x7f76d8143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 --2- 172.21.15.114:0/1695439386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f76d806d090 0x7f76d8166850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 >> 172.21.15.114:0/1695439386 conn(0x7f76d80b1730 msgr2=0x7f76d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.504+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 shutdown_connections 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:40.505+0000 7f76de200640 1 -- 172.21.15.114:0/1695439386 wait complete. 2024-03-06T07:31:40.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:40.733 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:31:40.733 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:31:40.734 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:40.734 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:40.785 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:31:40.785 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:31:40.785 DEBUG:teuthology.orchestra.run.smithi114:> import uuid 2024-03-06T07:31:40.785 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> n = 100 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = False 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> if 0 > 0: 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = True 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> dpath = os.path.dirname(path) 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> fnameprefix = os.path.basename(path) 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> try: 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> for i in range(n): 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> f.write(f"{i}") 2024-03-06T07:31:40.786 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> f.flush() 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(f.fileno()) 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> if True: 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> os.unlink(fpath) 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> if create_hard_links: 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> for j in range(0): 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> if True: 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> finally: 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> os.close(dirfd) 2024-03-06T07:31:40.787 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:31:41.050 INFO:teuthology.orchestra.run.smithi114.stderr:Traceback (most recent call last): 2024-03-06T07:31:41.050 INFO:teuthology.orchestra.run.smithi114.stderr: File "", line 26, in 2024-03-06T07:31:41.050 INFO:teuthology.orchestra.run.smithi114.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:31:41.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:31:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:41.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:41.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.304+0000 7fa6ff400640 1 Processor -- start 2024-03-06T07:31:41.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.304+0000 7fa6ff400640 1 -- start start 2024-03-06T07:31:41.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.305+0000 7fa6ff400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6f8144c00 0x7fa6f806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.305+0000 7fa6ff400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa6f815fe60 0x7fa6f80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.305+0000 7fa6ff400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f806ee70 con 0x7fa6f815f9d0 2024-03-06T07:31:41.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.305+0000 7fa6ff400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f806cef0 con 0x7fa6f8144c00 2024-03-06T07:31:41.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.305+0000 7fa6ff400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f806d1d0 con 0x7fa6f815fe60 2024-03-06T07:31:41.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fca00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6f8144c00 0x7fa6f806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fca00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6f8144c00 0x7fa6f806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55540/0 (socket says 172.21.15.114:55540) 2024-03-06T07:31:41.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fca00640 1 -- 172.21.15.114:0/998456606 learned_addr learned my addr 172.21.15.114:0/998456606 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:41.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fca00640 1 -- 172.21.15.114:0/998456606 >> v2:172.21.15.177:3300/0 conn(0x7fa6f815fe60 msgr2=0x7fa6f80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fd400640 1 --2- 172.21.15.114:0/998456606 >> v2:172.21.15.177:3300/0 conn(0x7fa6f815fe60 0x7fa6f80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fca00640 1 --2- 172.21.15.114:0/998456606 >> v2:172.21.15.177:3300/0 conn(0x7fa6f815fe60 0x7fa6f80ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fca00640 1 -- 172.21.15.114:0/998456606 >> v1:172.21.15.177:6789/0 conn(0x7fa6f815f9d0 legacy=0x7fa6f80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:41.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.306+0000 7fa6fca00640 1 -- 172.21.15.114:0/998456606 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f806d4b0 con 0x7fa6f8144c00 2024-03-06T07:31:41.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.307+0000 7fa6fca00640 1 --2- 172.21.15.114:0/998456606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f8144c00 0x7fa6f806e930 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806e500 tx=0x7fa6e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea73fc211fc7b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6f7e00640 1 -- 172.21.15.114:0/998456606 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8035020 con 0x7fa6f8144c00 2024-03-06T07:31:41.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6f7e00640 1 -- 172.21.15.114:0/998456606 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e8031870 con 0x7fa6f8144c00 2024-03-06T07:31:41.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6f7e00640 1 -- 172.21.15.114:0/998456606 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8031c40 con 0x7fa6f8144c00 2024-03-06T07:31:41.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 -- 172.21.15.114:0/998456606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f8144c00 msgr2=0x7fa6f806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 --2- 172.21.15.114:0/998456606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f8144c00 0x7fa6f806e930 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806e500 tx=0x7fa6e802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:41.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 -- 172.21.15.114:0/998456606 shutdown_connections 2024-03-06T07:31:41.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 --2- 172.21.15.114:0/998456606 >> v2:172.21.15.177:3300/0 conn(0x7fa6f815fe60 0x7fa6f80ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 --2- 172.21.15.114:0/998456606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f8144c00 0x7fa6f806e930 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 -- 172.21.15.114:0/998456606 >> 172.21.15.114:0/998456606 conn(0x7fa6f80b1730 msgr2=0x7fa6f80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 -- 172.21.15.114:0/998456606 shutdown_connections 2024-03-06T07:31:41.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.308+0000 7fa6ff400640 1 -- 172.21.15.114:0/998456606 wait complete. 2024-03-06T07:31:41.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.311+0000 7fa6ff400640 1 Processor -- start 2024-03-06T07:31:41.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.311+0000 7fa6ff400640 1 -- start start 2024-03-06T07:31:41.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6ff400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6f8144c00 0x7fa6f81f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6ff400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6f815f9d0 0x7fa6f81f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6ff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f815fe60 0x7fa6f8139600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6ff400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8160730 con 0x7fa6f815fe60 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6ff400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f8160270 con 0x7fa6f815f9d0 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6ff400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f806cb50 con 0x7fa6f8144c00 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6fd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f815fe60 0x7fa6f8139600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6fd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f815fe60 0x7fa6f8139600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55546/0 (socket says 172.21.15.114:55546) 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.313+0000 7fa6fd400640 1 -- 172.21.15.114:0/1923962461 learned_addr learned my addr 172.21.15.114:0/1923962461 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6efe00640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6f815f9d0 0x7fa6f81f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6fca00640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6f8144c00 0x7fa6f81f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6fd400640 1 -- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6f8144c00 msgr2=0x7fa6f81f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6fd400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6f8144c00 0x7fa6f81f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6fd400640 1 -- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6f815f9d0 msgr2=0x7fa6f81f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6fd400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6f815f9d0 0x7fa6f81f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6fd400640 1 -- 172.21.15.114:0/1923962461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f8139b40 con 0x7fa6f815fe60 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6fd400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f815fe60 0x7fa6f8139600 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cd70 tx=0x7fa6f000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.314+0000 7fa6f5600640 1 -- 172.21.15.114:0/1923962461 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0003b90 con 0x7fa6f815fe60 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.315+0000 7fa6f5600640 1 -- 172.21.15.114:0/1923962461 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f0003d30 con 0x7fa6f815fe60 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.315+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f81f56d0 con 0x7fa6f815fe60 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.315+0000 7fa6f5600640 1 -- 172.21.15.114:0/1923962461 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0004240 con 0x7fa6f815fe60 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.315+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f813c960 con 0x7fa6f815fe60 2024-03-06T07:31:41.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.316+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f804f330 con 0x7fa6f815fe60 2024-03-06T07:31:41.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.317+0000 7fa6f5600640 1 -- 172.21.15.114:0/1923962461 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6f00043e0 con 0x7fa6f815fe60 2024-03-06T07:31:41.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.318+0000 7fa6f5600640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6d406b4d0 0x7fa6d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:41.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.318+0000 7fa6f5600640 1 -- 172.21.15.114:0/1923962461 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa6f008ae00 con 0x7fa6f815fe60 2024-03-06T07:31:41.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.318+0000 7fa6fca00640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6d406b4d0 0x7fa6d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:41.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.319+0000 7fa6fca00640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6d406b4d0 0x7fa6d406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8036000 tx=0x7fa6e803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:41.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.325+0000 7fa6f5600640 1 -- 172.21.15.114:0/1923962461 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6f8139b40 con 0x7fa6f815fe60 2024-03-06T07:31:41.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.570+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6f8142c10 con 0x7fa6f815fe60 2024-03-06T07:31:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.570+0000 7fa6f5600640 1 -- 172.21.15.114:0/1923962461 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fa6f0059ba0 con 0x7fa6f815fe60 2024-03-06T07:31:41.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:41.572 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6d406b4d0 msgr2=0x7fa6d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6d406b4d0 0x7fa6d406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8036000 tx=0x7fa6e803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f815fe60 msgr2=0x7fa6f8139600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f815fe60 0x7fa6f8139600 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cd70 tx=0x7fa6f000a660 comp rx=0 tx=0).stop 2024-03-06T07:31:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 shutdown_connections 2024-03-06T07:31:41.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6d406b4d0 0x7fa6d406d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f815fe60 0x7fa6f8139600 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6f815f9d0 0x7fa6f81f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 --2- 172.21.15.114:0/1923962461 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6f8144c00 0x7fa6f81f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 >> 172.21.15.114:0/1923962461 conn(0x7fa6f80b1730 msgr2=0x7fa6f8070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 shutdown_connections 2024-03-06T07:31:41.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:41.573+0000 7fa6ff400640 1 -- 172.21.15.114:0/1923962461 wait complete. 2024-03-06T07:31:41.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:41.818 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:31:41.818 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:31:41.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.015+0000 7fa1bd000640 1 Processor -- start 2024-03-06T07:31:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.015+0000 7fa1bd000640 1 -- start start 2024-03-06T07:31:42.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.017+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 0x7fa1b815fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.017+0000 7fa1bd000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b80b3040 con 0x7fa1b80b4390 2024-03-06T07:31:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.017+0000 7fa1bd000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b80cd3e0 con 0x7fa1b80cf9a0 2024-03-06T07:31:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.017+0000 7fa1bd000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b815fb50 con 0x7fa1b815f770 2024-03-06T07:31:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.017+0000 7fa1ad400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa1b80b4390 0x7fa1b8160290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59814/0 (socket says 172.21.15.114:59814) 2024-03-06T07:31:42.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.017+0000 7fa1ad400640 1 -- 172.21.15.114:0/4213164513 learned_addr learned my addr 172.21.15.114:0/4213164513 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.017+0000 7fa1b5e00640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 0x7fa1b815fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563042982 0 0) 0x7fa1b80b3040 con 0x7fa1b80b4390 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c003520 con 0x7fa1b80b4390 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 254053400 0 0) 0x7fa19c003520 con 0x7fa1b80b4390 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 msgr2=0x7fa1b815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 0x7fa1b815fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 >> v1:172.21.15.177:6789/0 conn(0x7fa1b80cf9a0 legacy=0x7fa1b80cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b80cd6c0 con 0x7fa1b80b4390 2024-03-06T07:31:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fa1a0001670 con 0x7fa1b80b4390 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 >> v1:172.21.15.114:6789/0 conn(0x7fa1b80b4390 legacy=0x7fa1b8160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c0039e0 0x7fa19c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19c004800 0x7fa19c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 0x7fa19c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c00bac0 con 0x7fa19c004800 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c00bc70 con 0x7fa19c0039e0 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.018+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19c00be70 con 0x7fa1b815f770 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1ad400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19c004800 0x7fa19c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1b5e00640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c0039e0 0x7fa19c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1b6800640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 0x7fa19c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1ad400640 1 -- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 msgr2=0x7fa19c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1ad400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b815f770 0x7fa19c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1ad400640 1 -- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c0039e0 msgr2=0x7fa19c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1ad400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c0039e0 0x7fa19c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.019+0000 7fa1ad400640 1 -- 172.21.15.114:0/4213164513 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c003520 con 0x7fa19c004800 2024-03-06T07:31:42.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.020+0000 7fa1ad400640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19c004800 0x7fa19c004c80 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80b4790 tx=0x7fa19c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3376e9babb33d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.020+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0007d30 con 0x7fa19c004800 2024-03-06T07:31:42.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.020+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a0030070 con 0x7fa19c004800 2024-03-06T07:31:42.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.020+0000 7fa1b5400640 1 -- 172.21.15.114:0/4213164513 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0005d30 con 0x7fa19c004800 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.021+0000 7fa1bd000640 1 -- 172.21.15.114:0/4213164513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19c004800 msgr2=0x7fa19c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.021+0000 7fa1bd000640 1 --2- 172.21.15.114:0/4213164513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19c004800 0x7fa19c004c80 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80b4790 tx=0x7fa19c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.021+0000 7fa1b6800640 1 -- 172.21.15.114:0/4213164513 reap_dead start 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.021+0000 7fa1bd000640 1 -- 172.21.15.114:0/4213164513 shutdown_connections 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.021+0000 7fa1bd000640 1 -- 172.21.15.114:0/4213164513 >> 172.21.15.114:0/4213164513 conn(0x7fa1b80b3870 msgr2=0x7fa1b80b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.021+0000 7fa1bd000640 1 -- 172.21.15.114:0/4213164513 shutdown_connections 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.021+0000 7fa1bd000640 1 -- 172.21.15.114:0/4213164513 wait complete. 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.023+0000 7fa1bd000640 1 Processor -- start 2024-03-06T07:31:42.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.023+0000 7fa1bd000640 1 -- start start 2024-03-06T07:31:42.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c004800 0x7fa1b8156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b80cb730 0x7fa1b80c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1b80c9d60 0x7fa1b81f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1bd000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80b34a0 con 0x7fa1b80c9d60 2024-03-06T07:31:42.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1bd000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80b3320 con 0x7fa19c004800 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1bd000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81609a0 con 0x7fa1b80cb730 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1b6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1b80c9d60 0x7fa1b81f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1b6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1b80c9d60 0x7fa1b81f79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55552/0 (socket says 172.21.15.114:55552) 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1b6800640 1 -- 172.21.15.114:0/609077838 learned_addr learned my addr 172.21.15.114:0/609077838 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1ad400640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b80cb730 0x7fa1b80c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.024+0000 7fa1b5e00640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c004800 0x7fa1b8156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1b6800640 1 -- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b80cb730 msgr2=0x7fa1b80c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1b6800640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b80cb730 0x7fa1b80c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1b6800640 1 -- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c004800 msgr2=0x7fa1b8156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1b6800640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c004800 0x7fa1b8156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1b6800640 1 -- 172.21.15.114:0/609077838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b81574b0 con 0x7fa1b80c9d60 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1b6800640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1b80c9d60 0x7fa1b81f79c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0000c00 tx=0x7fa1b000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1aea00640 1 -- 172.21.15.114:0/609077838 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0019070 con 0x7fa1b80c9d60 2024-03-06T07:31:42.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1aea00640 1 -- 172.21.15.114:0/609077838 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b00052e0 con 0x7fa1b80c9d60 2024-03-06T07:31:42.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1aea00640 1 -- 172.21.15.114:0/609077838 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0003830 con 0x7fa1b80c9d60 2024-03-06T07:31:42.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b815ed80 con 0x7fa1b80c9d60 2024-03-06T07:31:42.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.025+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b815b9c0 con 0x7fa1b80c9d60 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.027+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b80b41b0 con 0x7fa1b80c9d60 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.027+0000 7fa1aea00640 1 -- 172.21.15.114:0/609077838 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1b001c070 con 0x7fa1b80c9d60 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.028+0000 7fa1aea00640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa19406b500 0x7fa19406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.028+0000 7fa1aea00640 1 -- 172.21.15.114:0/609077838 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa1b0089650 con 0x7fa1b80c9d60 2024-03-06T07:31:42.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.029+0000 7fa1b5e00640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa19406b500 0x7fa19406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.029+0000 7fa1b5e00640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa19406b500 0x7fa19406d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8007520 tx=0x7fa1a800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.033+0000 7fa1aea00640 1 -- 172.21.15.114:0/609077838 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1b80b41b0 con 0x7fa1b80c9d60 2024-03-06T07:31:42.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.273+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1b804f330 con 0x7fa1b80c9d60 2024-03-06T07:31:42.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.274+0000 7fa1aea00640 1 -- 172.21.15.114:0/609077838 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fa1b0058490 con 0x7fa1b80c9d60 2024-03-06T07:31:42.276 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:42.276 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:42.276 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:42.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.276+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa19406b500 msgr2=0x7fa19406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.276+0000 7fa1bd000640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa19406b500 0x7fa19406d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8007520 tx=0x7fa1a800b040 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.276+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1b80c9d60 msgr2=0x7fa1b81f79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.276+0000 7fa1bd000640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1b80c9d60 0x7fa1b81f79c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0000c00 tx=0x7fa1b000e8a0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 shutdown_connections 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa19406b500 0x7fa19406d9c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1b80c9d60 0x7fa1b81f79c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1b80cb730 0x7fa1b80c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 --2- 172.21.15.114:0/609077838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19c004800 0x7fa1b8156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 >> 172.21.15.114:0/609077838 conn(0x7fa1b80b3870 msgr2=0x7fa1b806d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 shutdown_connections 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:42.277+0000 7fa1bd000640 1 -- 172.21.15.114:0/609077838 wait complete. 2024-03-06T07:31:42.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:42.504 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:31:42.504 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:31:42.504 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:31:45.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:45.615 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:47.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:47.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.716+0000 7fbf85a00640 1 Processor -- start 2024-03-06T07:31:47.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.716+0000 7fbf85a00640 1 -- start start 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf85a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbf80144c00 0x7fbf8006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf85a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbf8015fe60 0x7fbf800ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf85a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8006ee70 con 0x7fbf8015f9d0 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf85a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8006cef0 con 0x7fbf8015fe60 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf85a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8006d1d0 con 0x7fbf80144c00 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf7f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbf8015fe60 0x7fbf800ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf7f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbf8015fe60 0x7fbf800ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55566/0 (socket says 172.21.15.114:55566) 2024-03-06T07:31:47.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.718+0000 7fbf7f400640 1 -- 172.21.15.114:0/1322702665 learned_addr learned my addr 172.21.15.114:0/1322702665 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.719+0000 7fbf7ea00640 1 --2- 172.21.15.114:0/1322702665 >> v2:172.21.15.177:3300/0 conn(0x7fbf80144c00 0x7fbf8006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.719+0000 7fbf7f400640 1 -- 172.21.15.114:0/1322702665 >> v2:172.21.15.177:3300/0 conn(0x7fbf80144c00 msgr2=0x7fbf8006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.719+0000 7fbf7f400640 1 --2- 172.21.15.114:0/1322702665 >> v2:172.21.15.177:3300/0 conn(0x7fbf80144c00 0x7fbf8006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.719+0000 7fbf7f400640 1 -- 172.21.15.114:0/1322702665 >> v1:172.21.15.114:6789/0 conn(0x7fbf8015f9d0 legacy=0x7fbf800ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.719+0000 7fbf7f400640 1 -- 172.21.15.114:0/1322702665 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8006d4b0 con 0x7fbf8015fe60 2024-03-06T07:31:47.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.719+0000 7fbf7e000640 1 -- 172.21.15.114:0/1322702665 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738655098 0 0) 0x7fbf68001110 con 0x7fbf8015f9d0 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.720+0000 7fbf7f400640 1 --2- 172.21.15.114:0/1322702665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800ce7b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbf800b25f0 tx=0x7fbf7802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fe16327ed10d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.720+0000 7fbf7e000640 1 -- 172.21.15.114:0/1322702665 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7802cb10 con 0x7fbf8015fe60 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.720+0000 7fbf7e000640 1 -- 172.21.15.114:0/1322702665 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7802ccb0 con 0x7fbf8015fe60 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.720+0000 7fbf7e000640 1 -- 172.21.15.114:0/1322702665 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7802ce60 con 0x7fbf8015fe60 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 -- 172.21.15.114:0/1322702665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 msgr2=0x7fbf800ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1322702665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800ce7b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fbf800b25f0 tx=0x7fbf7802bf70 comp rx=0 tx=0).stop 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 -- 172.21.15.114:0/1322702665 shutdown_connections 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1322702665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800ce7b0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1322702665 >> v2:172.21.15.177:3300/0 conn(0x7fbf80144c00 0x7fbf8006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 -- 172.21.15.114:0/1322702665 >> 172.21.15.114:0/1322702665 conn(0x7fbf800b1730 msgr2=0x7fbf800b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 -- 172.21.15.114:0/1322702665 shutdown_connections 2024-03-06T07:31:47.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.721+0000 7fbf85a00640 1 -- 172.21.15.114:0/1322702665 wait complete. 2024-03-06T07:31:47.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.724+0000 7fbf85a00640 1 Processor -- start 2024-03-06T07:31:47.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.724+0000 7fbf85a00640 1 -- start start 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf85a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf80144c00 0x7fbf801f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf85a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf8015f9d0 0x7fbf800b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf85a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf85a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf80160730 con 0x7fbf8015fe60 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf85a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf80160270 con 0x7fbf80144c00 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf85a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8006cb50 con 0x7fbf8015f9d0 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55574/0 (socket says 172.21.15.114:55574) 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 -- 172.21.15.114:0/1716915071 learned_addr learned my addr 172.21.15.114:0/1716915071 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7ea00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf80144c00 0x7fbf801f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf77e00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf8015f9d0 0x7fbf800b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 -- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf8015f9d0 msgr2=0x7fbf800b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf8015f9d0 0x7fbf800b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 -- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf80144c00 msgr2=0x7fbf801f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf80144c00 0x7fbf801f7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7f400640 1 -- 172.21.15.114:0/1716915071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf800b40b0 con 0x7fbf8015fe60 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf7ea00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf80144c00 0x7fbf801f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.726+0000 7fbf77e00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf8015f9d0 0x7fbf800b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.727+0000 7fbf7f400640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800b3a80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbf78036040 tx=0x7fbf78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.727+0000 7fbf76a00640 1 -- 172.21.15.114:0/1716915071 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78043030 con 0x7fbf8015fe60 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.727+0000 7fbf76a00640 1 -- 172.21.15.114:0/1716915071 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7803e450 con 0x7fbf8015fe60 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.727+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf800b4390 con 0x7fbf8015fe60 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.727+0000 7fbf76a00640 1 -- 172.21.15.114:0/1716915071 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7802ce60 con 0x7fbf8015fe60 2024-03-06T07:31:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.727+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf800b4950 con 0x7fbf8015fe60 2024-03-06T07:31:47.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.728+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf800a9e50 con 0x7fbf8015fe60 2024-03-06T07:31:47.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.730+0000 7fbf76a00640 1 -- 172.21.15.114:0/1716915071 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf78046070 con 0x7fbf8015fe60 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.730+0000 7fbf76a00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.731+0000 7fbf76a00640 1 -- 172.21.15.114:0/1716915071 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbf780ac6a0 con 0x7fbf8015fe60 2024-03-06T07:31:47.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.731+0000 7fbf7ea00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.731+0000 7fbf7ea00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbf70006fd0 tx=0x7fbf70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.735+0000 7fbf76a00640 1 -- 172.21.15.114:0/1716915071 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf7804d070 con 0x7fbf8015fe60 2024-03-06T07:31:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.976+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf80072890 con 0x7fbf8015fe60 2024-03-06T07:31:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.977+0000 7fbf76a00640 1 -- 172.21.15.114:0/1716915071 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fbf7807b560 con 0x7fbf8015fe60 2024-03-06T07:31:47.979 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:47.979 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5c06b4d0 msgr2=0x7fbf5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbf70006fd0 tx=0x7fbf70008040 comp rx=0 tx=0).stop 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 msgr2=0x7fbf800b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800b3a80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbf78036040 tx=0x7fbf78004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 shutdown_connections 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbf5c06b4d0 0x7fbf5c06d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8015fe60 0x7fbf800b3a80 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbf8015f9d0 0x7fbf800b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 --2- 172.21.15.114:0/1716915071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf80144c00 0x7fbf801f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 >> 172.21.15.114:0/1716915071 conn(0x7fbf800b1730 msgr2=0x7fbf8015e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.980+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 shutdown_connections 2024-03-06T07:31:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:47.981+0000 7fbf85a00640 1 -- 172.21.15.114:0/1716915071 wait complete. 2024-03-06T07:31:48.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:48.235 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":7}} 2024-03-06T07:31:48.235 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 7 } } 2024-03-06T07:31:48.235 DEBUG:tasks.ceph_test_case:wait_until_equal: 7 != 10, waiting (timeout=60)... 2024-03-06T07:31:53.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:53.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.447+0000 7fbcd1a00640 1 Processor -- start 2024-03-06T07:31:53.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.447+0000 7fbcd1a00640 1 -- start start 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcd1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc144c00 0x7fbccc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcd1a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbccc15f9d0 0x7fbccc15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcd1a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbccc0ce0a0 0x7fbccc0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcd1a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc160370 con 0x7fbccc15f9d0 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcd1a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc06ee70 con 0x7fbccc0ce0a0 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcd1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc06cef0 con 0x7fbccc144c00 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcc2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbccc15f9d0 0x7fbccc15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcc2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbccc15f9d0 0x7fbccc15fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33278/0 (socket says 172.21.15.114:33278) 2024-03-06T07:31:53.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbcc2000640 1 -- 172.21.15.114:0/1885345726 learned_addr learned my addr 172.21.15.114:0/1885345726 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:53.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbccaa00640 1 --2- 172.21.15.114:0/1885345726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc144c00 0x7fbccc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.449+0000 7fbccb400640 1 --2- 172.21.15.114:0/1885345726 >> v2:172.21.15.177:3300/0 conn(0x7fbccc0ce0a0 0x7fbccc0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.450+0000 7fbcc2000640 1 -- 172.21.15.114:0/1885345726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc144c00 msgr2=0x7fbccc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.450+0000 7fbcc2000640 1 --2- 172.21.15.114:0/1885345726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc144c00 0x7fbccc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.450+0000 7fbcc2000640 1 -- 172.21.15.114:0/1885345726 >> v2:172.21.15.177:3300/0 conn(0x7fbccc0ce0a0 msgr2=0x7fbccc0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.450+0000 7fbcc2000640 1 --2- 172.21.15.114:0/1885345726 >> v2:172.21.15.177:3300/0 conn(0x7fbccc0ce0a0 0x7fbccc0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.450+0000 7fbcc2000640 1 -- 172.21.15.114:0/1885345726 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc06d1d0 con 0x7fbccc15f9d0 2024-03-06T07:31:53.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.451+0000 7fbcc2000640 1 --2- 172.21.15.114:0/1885345726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc15fe30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0737b0 tx=0x7fbcb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f5ec71e71819e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.451+0000 7fbcca000640 1 -- 172.21.15.114:0/1885345726 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4035020 con 0x7fbccc15f9d0 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.451+0000 7fbcca000640 1 -- 172.21.15.114:0/1885345726 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb4031870 con 0x7fbccc15f9d0 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.451+0000 7fbcca000640 1 -- 172.21.15.114:0/1885345726 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4031c40 con 0x7fbccc15f9d0 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 -- 172.21.15.114:0/1885345726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 msgr2=0x7fbccc15fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/1885345726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc15fe30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0737b0 tx=0x7fbcb402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 -- 172.21.15.114:0/1885345726 shutdown_connections 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/1885345726 >> v2:172.21.15.177:3300/0 conn(0x7fbccc0ce0a0 0x7fbccc0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/1885345726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc15fe30 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/1885345726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc144c00 0x7fbccc06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 -- 172.21.15.114:0/1885345726 >> 172.21.15.114:0/1885345726 conn(0x7fbccc0b1730 msgr2=0x7fbccc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 -- 172.21.15.114:0/1885345726 shutdown_connections 2024-03-06T07:31:53.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.452+0000 7fbcd1a00640 1 -- 172.21.15.114:0/1885345726 wait complete. 2024-03-06T07:31:53.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.454+0000 7fbcd1a00640 1 Processor -- start 2024-03-06T07:31:53.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.454+0000 7fbcd1a00640 1 -- start start 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcd1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc0ce0a0 0x7fbccc0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcd1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc144c00 0x7fbccc13bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcd1a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc13c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcd1a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc160730 con 0x7fbccc15f9d0 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcd1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc06d790 con 0x7fbccc144c00 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcd1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc06cb50 con 0x7fbccc0ce0a0 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc13c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc13c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33282/0 (socket says 172.21.15.114:33282) 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 -- 172.21.15.114:0/2120969770 learned_addr learned my addr 172.21.15.114:0/2120969770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcc2000640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc144c00 0x7fbccc13bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccaa00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc0ce0a0 0x7fbccc0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 -- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc0ce0a0 msgr2=0x7fbccc0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc0ce0a0 0x7fbccc0ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 -- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc144c00 msgr2=0x7fbccc13bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc144c00 0x7fbccc13bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbccb400640 1 -- 172.21.15.114:0/2120969770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc072230 con 0x7fbccc15f9d0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.455+0000 7fbcc2000640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc144c00 0x7fbccc13bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.456+0000 7fbccb400640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc13c4b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4007d50 tx=0x7fbcc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.456+0000 7fbcc3400640 1 -- 172.21.15.114:0/2120969770 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4014420 con 0x7fbccc15f9d0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.456+0000 7fbcc3400640 1 -- 172.21.15.114:0/2120969770 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4019070 con 0x7fbccc15f9d0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.456+0000 7fbcc3400640 1 -- 172.21.15.114:0/2120969770 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4014820 con 0x7fbccc15f9d0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.456+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc071f50 con 0x7fbccc15f9d0 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.456+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc158840 con 0x7fbccc15f9d0 2024-03-06T07:31:53.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.457+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc04f330 con 0x7fbccc15f9d0 2024-03-06T07:31:53.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.458+0000 7fbcc3400640 1 -- 172.21.15.114:0/2120969770 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcc40149c0 con 0x7fbccc15f9d0 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.460+0000 7fbcc3400640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbca806b500 0x7fbca806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.460+0000 7fbcc3400640 1 -- 172.21.15.114:0/2120969770 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbcc408aa10 con 0x7fbccc15f9d0 2024-03-06T07:31:53.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.460+0000 7fbccaa00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbca806b500 0x7fbca806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.461+0000 7fbccaa00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbca806b500 0x7fbca806d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc009ad0 tx=0x7fbcbc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.465+0000 7fbcc3400640 1 -- 172.21.15.114:0/2120969770 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcc4059840 con 0x7fbccc15f9d0 2024-03-06T07:31:53.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.707+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbccc145160 con 0x7fbccc15f9d0 2024-03-06T07:31:53.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.708+0000 7fbcc3400640 1 -- 172.21.15.114:0/2120969770 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fbcc405f070 con 0x7fbccc15f9d0 2024-03-06T07:31:53.710 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:53.711 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:53.711 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbca806b500 msgr2=0x7fbca806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbca806b500 0x7fbca806d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc009ad0 tx=0x7fbcbc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 msgr2=0x7fbccc13c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc13c4b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4007d50 tx=0x7fbcc400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 shutdown_connections 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbca806b500 0x7fbca806d9c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbccc15f9d0 0x7fbccc13c4b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc144c00 0x7fbccc13bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 --2- 172.21.15.114:0/2120969770 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbccc0ce0a0 0x7fbccc0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 >> 172.21.15.114:0/2120969770 conn(0x7fbccc0b1730 msgr2=0x7fbccc06dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 shutdown_connections 2024-03-06T07:31:53.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:53.711+0000 7fbcd1a00640 1 -- 172.21.15.114:0/2120969770 wait complete. 2024-03-06T07:31:53.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:53.960 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:31:53.960 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:31:53.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:54.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.179+0000 7f7a0f200640 1 Processor -- start 2024-03-06T07:31:54.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.179+0000 7f7a0f200640 1 -- start start 2024-03-06T07:31:54.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.181+0000 7f7a0f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 0x7f7a100d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.181+0000 7f7a0f200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a10161960 con 0x7f7a100d1e90 2024-03-06T07:31:54.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.181+0000 7f7a0f200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a100cf6f0 con 0x7f7a100d2a70 2024-03-06T07:31:54.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.181+0000 7f7a0f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100cf9d0 con 0x7f7a10161580 2024-03-06T07:31:54.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.181+0000 7f7a0d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7a100d1e90 0x7f7a100d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36288/0 (socket says 172.21.15.114:36288) 2024-03-06T07:31:54.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.181+0000 7f7a0d400640 1 -- 172.21.15.114:0/1363145570 learned_addr learned my addr 172.21.15.114:0/1363145570 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:54.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874234324 0 0) 0x7f7a10161960 con 0x7f7a100d1e90 2024-03-06T07:31:54.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f4003520 con 0x7f7a100d1e90 2024-03-06T07:31:54.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0de00640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 0x7f7a100d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266463049 0 0) 0x7f79f4003520 con 0x7f7a100d1e90 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 msgr2=0x7f7a100d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 0x7f7a100d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 >> v1:172.21.15.177:6789/0 conn(0x7f7a100d2a70 legacy=0x7f7a100b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a100cfcb0 con 0x7f7a100d1e90 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f79fc001670 con 0x7f7a100d1e90 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 >> v1:172.21.15.114:6789/0 conn(0x7f7a100d1e90 legacy=0x7f7a100d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79f40039e0 0x7f79f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f4004800 0x7f79f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 0x7f79f4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100cf9d0 con 0x7f79f4004800 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f400bb50 con 0x7f79f40039e0 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.182+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f400bd50 con 0x7f7a10161580 2024-03-06T07:31:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0d400640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f4004800 0x7f79f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0e800640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 0x7f79f4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0d400640 1 -- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 msgr2=0x7f79f4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0d400640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 0x7f79f4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0d400640 1 -- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79f40039e0 msgr2=0x7f79f4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0d400640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79f40039e0 0x7f79f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0d400640 1 -- 172.21.15.114:0/1363145570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4003520 con 0x7f79f4004800 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.183+0000 7f7a0e800640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7a10161580 0x7f79f4005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:54.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.184+0000 7f7a0d400640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f4004800 0x7f79f4004c80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00c0e0 tx=0x7f79f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c110c65d66219a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.184+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc007c40 con 0x7f79f4004800 2024-03-06T07:31:54.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.184+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79fc030070 con 0x7f79f4004800 2024-03-06T07:31:54.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.184+0000 7f7a0ca00640 1 -- 172.21.15.114:0/1363145570 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc005d30 con 0x7f79f4004800 2024-03-06T07:31:54.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.185+0000 7f7a0f200640 1 -- 172.21.15.114:0/1363145570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f4004800 msgr2=0x7f79f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.185+0000 7f7a0f200640 1 --2- 172.21.15.114:0/1363145570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f4004800 0x7f79f4004c80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f79fc00c0e0 tx=0x7f79f4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.185+0000 7f7a0e800640 1 -- 172.21.15.114:0/1363145570 reap_dead start 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.185+0000 7f7a0f200640 1 -- 172.21.15.114:0/1363145570 shutdown_connections 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.185+0000 7f7a0f200640 1 -- 172.21.15.114:0/1363145570 >> 172.21.15.114:0/1363145570 conn(0x7f7a100d3d80 msgr2=0x7f7a100d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.185+0000 7f7a0f200640 1 -- 172.21.15.114:0/1363145570 shutdown_connections 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.185+0000 7f7a0f200640 1 -- 172.21.15.114:0/1363145570 wait complete. 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.187+0000 7f7a0f200640 1 Processor -- start 2024-03-06T07:31:54.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.187+0000 7f7a0f200640 1 -- start start 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79f4004800 0x7f7a1014b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100b6970 0x7f7a100b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a100cd4c0 0x7f7a100cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100cf3a0 con 0x7f7a100cd4c0 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100b39a0 con 0x7f7a100b6970 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100d1950 con 0x7f79f4004800 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a100cd4c0 0x7f7a100cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a100cd4c0 0x7f7a100cd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33300/0 (socket says 172.21.15.114:33300) 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 -- 172.21.15.114:0/693494019 learned_addr learned my addr 172.21.15.114:0/693494019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0d400640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100b6970 0x7f7a100b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 -- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79f4004800 msgr2=0x7f7a1014b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0de00640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79f4004800 0x7f7a1014b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79f4004800 0x7f7a1014b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 -- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100b6970 msgr2=0x7f7a100b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100b6970 0x7f7a100b6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.188+0000 7f7a0e800640 1 -- 172.21.15.114:0/693494019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1014bdf0 con 0x7f7a100cd4c0 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.189+0000 7f7a0e800640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a100cd4c0 0x7f7a100cd940 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7a10073740 tx=0x7f7a0800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.189+0000 7f7a06000640 1 -- 172.21.15.114:0/693494019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08006d50 con 0x7f7a100cd4c0 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.189+0000 7f7a06000640 1 -- 172.21.15.114:0/693494019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a08003970 con 0x7f7a100cd4c0 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.189+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a100cc700 con 0x7f7a100cd4c0 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.189+0000 7f7a06000640 1 -- 172.21.15.114:0/693494019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08003d60 con 0x7f7a100cd4c0 2024-03-06T07:31:54.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.189+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a100cccc0 con 0x7f7a100cd4c0 2024-03-06T07:31:54.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.191+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1004f330 con 0x7f7a100cd4c0 2024-03-06T07:31:54.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.191+0000 7f7a06000640 1 -- 172.21.15.114:0/693494019 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a0801b020 con 0x7f7a100cd4c0 2024-03-06T07:31:54.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.193+0000 7f7a06000640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79f006b500 0x7f79f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.193+0000 7f7a06000640 1 -- 172.21.15.114:0/693494019 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7a08089cf0 con 0x7f7a100cd4c0 2024-03-06T07:31:54.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.193+0000 7f7a0de00640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79f006b500 0x7f79f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.194+0000 7f7a0de00640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79f006b500 0x7f79f006d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006a90 tx=0x7f7a00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.197+0000 7f7a06000640 1 -- 172.21.15.114:0/693494019 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a08058b20 con 0x7f7a100cd4c0 2024-03-06T07:31:54.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.565+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a100d3b60 con 0x7f7a100cd4c0 2024-03-06T07:31:54.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.566+0000 7f7a06000640 1 -- 172.21.15.114:0/693494019 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f7a08058420 con 0x7f7a100cd4c0 2024-03-06T07:31:54.568 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:54.568 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:54.569 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79f006b500 msgr2=0x7f79f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79f006b500 0x7f79f006d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006a90 tx=0x7f7a00008040 comp rx=0 tx=0).stop 2024-03-06T07:31:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a100cd4c0 msgr2=0x7f7a100cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a100cd4c0 0x7f7a100cd940 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7a10073740 tx=0x7f7a0800dc00 comp rx=0 tx=0).stop 2024-03-06T07:31:54.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 shutdown_connections 2024-03-06T07:31:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f79f006b500 0x7f79f006d9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a100cd4c0 0x7f7a100cd940 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100b6970 0x7f7a100b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 --2- 172.21.15.114:0/693494019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79f4004800 0x7f7a1014b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.569+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 >> 172.21.15.114:0/693494019 conn(0x7f7a100d3d80 msgr2=0x7f7a1015e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.570+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 shutdown_connections 2024-03-06T07:31:54.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:54.570+0000 7f7a0f200640 1 -- 172.21.15.114:0/693494019 wait complete. 2024-03-06T07:31:54.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:31:54.807 INFO:teuthology.orchestra.run.smithi114.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:31:54.808 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:31:54.808 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:54.808 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> import uuid 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> n = 10 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = False 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> if 0 > 0: 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> create_hard_links = True 2024-03-06T07:31:54.855 DEBUG:teuthology.orchestra.run.smithi114:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> dpath = os.path.dirname(path) 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> fnameprefix = os.path.basename(path) 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> try: 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> for i in range(n): 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> f.write(f"{i}") 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> f.flush() 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(f.fileno()) 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> if True: 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> os.unlink(fpath) 2024-03-06T07:31:54.856 DEBUG:teuthology.orchestra.run.smithi114:> if True: 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> if create_hard_links: 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> for j in range(0): 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> if False: 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> os.fsync(dirfd) 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> finally: 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> os.close(dirfd) 2024-03-06T07:31:54.857 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:31:55.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:31:55.149 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:55.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:55.291 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:55.410 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:55.410 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:55.410 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:55.462 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:55.463 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:55.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:55.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.770+0000 7f4c88200640 1 Processor -- start 2024-03-06T07:31:55.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.770+0000 7f4c88200640 1 -- start start 2024-03-06T07:31:55.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.771+0000 7f4c88200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4c8015fe60 0x7f4c80071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.771+0000 7f4c88200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8006cef0 con 0x7f4c8015f9d0 2024-03-06T07:31:55.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.771+0000 7f4c88200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8006d1d0 con 0x7f4c80144c00 2024-03-06T07:31:55.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.771+0000 7f4c88200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8006d4b0 con 0x7f4c8015fe60 2024-03-06T07:31:55.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.772+0000 7f4c84e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4c8015f9d0 0x7f4c80070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36294/0 (socket says 172.21.15.114:36294) 2024-03-06T07:31:55.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.772+0000 7f4c84e00640 1 -- 172.21.15.114:0/2288403442 learned_addr learned my addr 172.21.15.114:0/2288403442 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.772+0000 7f4c86200640 1 --2- 172.21.15.114:0/2288403442 >> v2:172.21.15.177:3300/0 conn(0x7f4c8015fe60 0x7f4c80071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.772+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285216516 0 0) 0x7f4c8006cef0 con 0x7f4c8015f9d0 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.772+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c68003520 con 0x7f4c8015f9d0 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254771286 0 0) 0x7f4c68003520 con 0x7f4c8015f9d0 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 >> v2:172.21.15.177:3300/0 conn(0x7f4c8015fe60 msgr2=0x7f4c80071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 --2- 172.21.15.114:0/2288403442 >> v2:172.21.15.177:3300/0 conn(0x7f4c8015fe60 0x7f4c80071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 >> v1:172.21.15.177:6789/0 conn(0x7f4c80144c00 legacy=0x7f4c8006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8006d790 con 0x7f4c8015f9d0 2024-03-06T07:31:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f4c6c001670 con 0x7f4c8015f9d0 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 >> v1:172.21.15.114:6789/0 conn(0x7f4c8015f9d0 legacy=0x7f4c80070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c680039b0 0x7f4c68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c68004800 0x7f4c68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8006d4b0 con 0x7f4c680051c0 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6800c060 con 0x7f4c68004800 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6800c260 con 0x7f4c680039b0 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c86200640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.773+0000 7f4c85800640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c680039b0 0x7f4c68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.774+0000 7f4c84e00640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c68004800 0x7f4c68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.774+0000 7f4c86200640 1 -- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c680039b0 msgr2=0x7f4c68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.774+0000 7f4c86200640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c680039b0 0x7f4c68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.774+0000 7f4c86200640 1 -- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c68004800 msgr2=0x7f4c68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.774+0000 7f4c86200640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c68004800 0x7f4c68004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.774+0000 7f4c86200640 1 -- 172.21.15.114:0/2288403442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c68003520 con 0x7f4c680051c0 2024-03-06T07:31:55.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.774+0000 7f4c84e00640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c68004800 0x7f4c68004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.775+0000 7f4c86200640 1 -- 172.21.15.114:0/2288403442 reap_dead start 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.775+0000 7f4c86200640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c6800b9c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009fc0 tx=0x7f4c68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e42f7ed843ca108 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.775+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78009c10 con 0x7f4c680051c0 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.775+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7802cc60 con 0x7f4c680051c0 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.775+0000 7f4c7fe00640 1 -- 172.21.15.114:0/2288403442 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78003bd0 con 0x7f4c680051c0 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.776+0000 7f4c88200640 1 -- 172.21.15.114:0/2288403442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 msgr2=0x7f4c6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.776+0000 7f4c88200640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c6800b9c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009fc0 tx=0x7f4c68005830 comp rx=0 tx=0).stop 2024-03-06T07:31:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.776+0000 7f4c88200640 1 -- 172.21.15.114:0/2288403442 shutdown_connections 2024-03-06T07:31:55.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.776+0000 7f4c88200640 1 --2- 172.21.15.114:0/2288403442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c6800b9c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.776+0000 7f4c88200640 1 -- 172.21.15.114:0/2288403442 >> 172.21.15.114:0/2288403442 conn(0x7f4c800b1730 msgr2=0x7f4c800b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.776+0000 7f4c88200640 1 -- 172.21.15.114:0/2288403442 shutdown_connections 2024-03-06T07:31:55.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.776+0000 7f4c88200640 1 -- 172.21.15.114:0/2288403442 wait complete. 2024-03-06T07:31:55.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.778+0000 7f4c88200640 1 Processor -- start 2024-03-06T07:31:55.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.778+0000 7f4c88200640 1 -- start start 2024-03-06T07:31:55.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.780+0000 7f4c88200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c801e57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c88200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c800b2be0 0x7f4c8013c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c88200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c800b4420 0x7f4c8015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c88200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8006cb50 con 0x7f4c680051c0 2024-03-06T07:31:55.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c88200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c80160730 con 0x7f4c800b4420 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c88200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8006ccd0 con 0x7f4c800b2be0 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c801e57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c801e57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33320/0 (socket says 172.21.15.114:33320) 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 -- 172.21.15.114:0/2379978110 learned_addr learned my addr 172.21.15.114:0/2379978110 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c86200640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c800b4420 0x7f4c8015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c84e00640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c800b2be0 0x7f4c8013c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 -- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c800b2be0 msgr2=0x7f4c8013c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c800b2be0 0x7f4c8013c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 -- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c800b4420 msgr2=0x7f4c8015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c800b4420 0x7f4c8015a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.781+0000 7f4c85800640 1 -- 172.21.15.114:0/2379978110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c800b3060 con 0x7f4c680051c0 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.782+0000 7f4c85800640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c801e57d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400e4b0 tx=0x7f4c7400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.782+0000 7f4c7d600640 1 -- 172.21.15.114:0/2379978110 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740052e0 con 0x7f4c680051c0 2024-03-06T07:31:55.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.782+0000 7f4c7d600640 1 -- 172.21.15.114:0/2379978110 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74018070 con 0x7f4c680051c0 2024-03-06T07:31:55.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.782+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c801e6200 con 0x7f4c680051c0 2024-03-06T07:31:55.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.782+0000 7f4c7d600640 1 -- 172.21.15.114:0/2379978110 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74003830 con 0x7f4c680051c0 2024-03-06T07:31:55.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.782+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8013c6b0 con 0x7f4c680051c0 2024-03-06T07:31:55.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.784+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c80067830 con 0x7f4c680051c0 2024-03-06T07:31:55.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.784+0000 7f4c7d600640 1 -- 172.21.15.114:0/2379978110 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c740039d0 con 0x7f4c680051c0 2024-03-06T07:31:55.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.785+0000 7f4c7d600640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c6006b500 0x7f4c6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.785+0000 7f4c7d600640 1 -- 172.21.15.114:0/2379978110 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4c74089e40 con 0x7f4c680051c0 2024-03-06T07:31:55.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.786+0000 7f4c84e00640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c6006b500 0x7f4c6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.786+0000 7f4c84e00640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c6006b500 0x7f4c6006d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0017e0 tx=0x7f4c6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.790+0000 7f4c7d600640 1 -- 172.21.15.114:0/2379978110 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c80067830 con 0x7f4c680051c0 2024-03-06T07:31:55.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.986+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c8004f330 con 0x7f4c680051c0 2024-03-06T07:31:55.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.989+0000 7f4c7d600640 1 -- 172.21.15.114:0/2379978110 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16814 (secure 0 0 0) 0x7f4c74058c70 con 0x7f4c680051c0 2024-03-06T07:31:55.991 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:55.991 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":68,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:37.032494+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:18.351057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:19.518313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.20","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:55.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c6006b500 msgr2=0x7f4c6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c6006b500 0x7f4c6006d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0017e0 tx=0x7f4c6c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:55.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 msgr2=0x7f4c801e57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c801e57d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400e4b0 tx=0x7f4c7400e970 comp rx=0 tx=0).stop 2024-03-06T07:31:55.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 shutdown_connections 2024-03-06T07:31:55.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c800b4420 0x7f4c8015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c800b2be0 0x7f4c8013c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c6006b500 0x7f4c6006d9c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 --2- 172.21.15.114:0/2379978110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c680051c0 0x7f4c801e57d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 >> 172.21.15.114:0/2379978110 conn(0x7f4c800b1730 msgr2=0x7f4c800b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 shutdown_connections 2024-03-06T07:31:55.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:55.993+0000 7f4c88200640 1 -- 172.21.15.114:0/2379978110 wait complete. 2024-03-06T07:31:56.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:18.351057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:19.518313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:56.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:56.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.232+0000 7faeefe00640 1 Processor -- start 2024-03-06T07:31:56.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.233+0000 7faeefe00640 1 -- start start 2024-03-06T07:31:56.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.234+0000 7faeefe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faee815fc10 0x7faee8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.234+0000 7faeefe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee80b3f70 0x7faee80ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.234+0000 7faeefe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee80b46c0 con 0x7faee815fff0 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.234+0000 7faeefe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee80b49a0 con 0x7faee815fc10 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.234+0000 7faeefe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee8072c90 con 0x7faee80b3f70 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.234+0000 7faeed400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faee815fc10 0x7faee8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeed400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faee815fc10 0x7faee8070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33328/0 (socket says 172.21.15.114:33328) 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeed400640 1 -- 172.21.15.114:0/123858076 learned_addr learned my addr 172.21.15.114:0/123858076 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeeca00640 1 --1- 172.21.15.114:0/123858076 >> v1:172.21.15.114:6789/0 conn(0x7faee815fff0 0x7faee8160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36296/0 (socket says 172.21.15.114:36296) 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeede00640 1 --2- 172.21.15.114:0/123858076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee80b3f70 0x7faee80ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeed400640 1 -- 172.21.15.114:0/123858076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee80b3f70 msgr2=0x7faee80ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeed400640 1 --2- 172.21.15.114:0/123858076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee80b3f70 0x7faee80ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeed400640 1 -- 172.21.15.114:0/123858076 >> v1:172.21.15.114:6789/0 conn(0x7faee815fff0 legacy=0x7faee8160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faeed400640 1 -- 172.21.15.114:0/123858076 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee809dd80 con 0x7faee815fc10 2024-03-06T07:31:56.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.235+0000 7faee7e00640 1 -- 172.21.15.114:0/123858076 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59169675 0 0) 0x7faee80b46c0 con 0x7faee815fff0 2024-03-06T07:31:56.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.237+0000 7faeed400640 1 --2- 172.21.15.114:0/123858076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee815fc10 0x7faee8070ff0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faedc009870 tx=0x7faedc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9875c44a557f44f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.237+0000 7faee7e00640 1 -- 172.21.15.114:0/123858076 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc034070 con 0x7faee815fc10 2024-03-06T07:31:56.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.237+0000 7faee7e00640 1 -- 172.21.15.114:0/123858076 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faedc030840 con 0x7faee815fc10 2024-03-06T07:31:56.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.237+0000 7faee7e00640 1 -- 172.21.15.114:0/123858076 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc030c10 con 0x7faee815fc10 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 -- 172.21.15.114:0/123858076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee815fc10 msgr2=0x7faee8070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 --2- 172.21.15.114:0/123858076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee815fc10 0x7faee8070ff0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faedc009870 tx=0x7faedc02b980 comp rx=0 tx=0).stop 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 -- 172.21.15.114:0/123858076 shutdown_connections 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 --2- 172.21.15.114:0/123858076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee80b3f70 0x7faee80ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 --2- 172.21.15.114:0/123858076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee815fc10 0x7faee8070ff0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 -- 172.21.15.114:0/123858076 >> 172.21.15.114:0/123858076 conn(0x7faee80b1940 msgr2=0x7faee80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 -- 172.21.15.114:0/123858076 shutdown_connections 2024-03-06T07:31:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.238+0000 7faeefe00640 1 -- 172.21.15.114:0/123858076 wait complete. 2024-03-06T07:31:56.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.240+0000 7faeefe00640 1 Processor -- start 2024-03-06T07:31:56.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.240+0000 7faeefe00640 1 -- start start 2024-03-06T07:31:56.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeefe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee80b3f70 0x7faee81e6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeefe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee815fc10 0x7faee814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeefe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee815fff0 0x7faee80b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeefe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee815c460 con 0x7faee80b3f70 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeefe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee815c2e0 con 0x7faee815fc10 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeefe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80ce9e0 con 0x7faee815fff0 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeed400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee80b3f70 0x7faee81e6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeed400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee80b3f70 0x7faee81e6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33334/0 (socket says 172.21.15.114:33334) 2024-03-06T07:31:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeed400640 1 -- 172.21.15.114:0/1956843516 learned_addr learned my addr 172.21.15.114:0/1956843516 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:56.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeed400640 1 -- 172.21.15.114:0/1956843516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee815fff0 msgr2=0x7faee80b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeeca00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee815fc10 0x7faee814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.241+0000 7faeed400640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee815fff0 0x7faee80b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faeed400640 1 -- 172.21.15.114:0/1956843516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee815fc10 msgr2=0x7faee814e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faeed400640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee815fc10 0x7faee814e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faeed400640 1 -- 172.21.15.114:0/1956843516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee81f25c0 con 0x7faee80b3f70 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faeed400640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee80b3f70 0x7faee81e6b90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faedc000c00 tx=0x7faedc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faee5600640 1 -- 172.21.15.114:0/1956843516 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc048070 con 0x7faee80b3f70 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faee5600640 1 -- 172.21.15.114:0/1956843516 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faedc030640 con 0x7faee80b3f70 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee81f28a0 con 0x7faee80b3f70 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faee5600640 1 -- 172.21.15.114:0/1956843516 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc034040 con 0x7faee80b3f70 2024-03-06T07:31:56.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.242+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee81f2e60 con 0x7faee80b3f70 2024-03-06T07:31:56.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.244+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee804f330 con 0x7faee80b3f70 2024-03-06T07:31:56.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.245+0000 7faee5600640 1 -- 172.21.15.114:0/1956843516 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faedc04b070 con 0x7faee80b3f70 2024-03-06T07:31:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.245+0000 7faee5600640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faec806b4d0 0x7faec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.246+0000 7faee5600640 1 -- 172.21.15.114:0/1956843516 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faedc0ac340 con 0x7faee80b3f70 2024-03-06T07:31:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.246+0000 7faeeca00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faec806b4d0 0x7faec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.246+0000 7faeeca00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faec806b4d0 0x7faec806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7faee809d020 tx=0x7faed4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.250+0000 7faee5600640 1 -- 172.21.15.114:0/1956843516 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faedc07b290 con 0x7faee80b3f70 2024-03-06T07:31:56.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.465+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faee80b1fd0 con 0x7faee80b3f70 2024-03-06T07:31:56.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.466+0000 7faee5600640 1 -- 172.21.15.114:0/1956843516 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7faedc07ab90 con 0x7faee80b3f70 2024-03-06T07:31:56.468 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faec806b4d0 msgr2=0x7faec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faec806b4d0 0x7faec806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7faee809d020 tx=0x7faed4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee80b3f70 msgr2=0x7faee81e6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee80b3f70 0x7faee81e6b90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faedc000c00 tx=0x7faedc004730 comp rx=0 tx=0).stop 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 shutdown_connections 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faee815fff0 0x7faee80b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faee815fc10 0x7faee814e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faec806b4d0 0x7faec806d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 --2- 172.21.15.114:0/1956843516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee80b3f70 0x7faee81e6b90 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 >> 172.21.15.114:0/1956843516 conn(0x7faee80b1940 msgr2=0x7faee80add70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 shutdown_connections 2024-03-06T07:31:56.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.469+0000 7faeefe00640 1 -- 172.21.15.114:0/1956843516 wait complete. 2024-03-06T07:31:56.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:56.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.693+0000 7fef83600640 1 Processor -- start 2024-03-06T07:31:56.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.693+0000 7fef83600640 1 -- start start 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef83600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fef7c15fc10 0x7fef7c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef83600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef7c15fff0 0x7fef7c160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef83600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c0ce4a0 0x7fef7c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef83600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c0b46c0 con 0x7fef7c15fff0 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef83600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c0b49a0 con 0x7fef7c15fc10 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef83600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c072c90 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef7be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef7c15fff0 0x7fef7c160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef7be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef7c15fff0 0x7fef7c160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33336/0 (socket says 172.21.15.114:33336) 2024-03-06T07:31:56.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.695+0000 7fef7be00640 1 -- 172.21.15.114:0/2572617296 learned_addr learned my addr 172.21.15.114:0/2572617296 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:56.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef80c00640 1 --2- 172.21.15.114:0/2572617296 >> v2:172.21.15.177:3300/0 conn(0x7fef7c15fc10 0x7fef7c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef81600640 1 --2- 172.21.15.114:0/2572617296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c0ce4a0 0x7fef7c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef7be00640 1 -- 172.21.15.114:0/2572617296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c0ce4a0 msgr2=0x7fef7c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef7be00640 1 --2- 172.21.15.114:0/2572617296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c0ce4a0 0x7fef7c0ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef7be00640 1 -- 172.21.15.114:0/2572617296 >> v2:172.21.15.177:3300/0 conn(0x7fef7c15fc10 msgr2=0x7fef7c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef7be00640 1 --2- 172.21.15.114:0/2572617296 >> v2:172.21.15.177:3300/0 conn(0x7fef7c15fc10 0x7fef7c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef7be00640 1 -- 172.21.15.114:0/2572617296 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c160990 con 0x7fef7c15fff0 2024-03-06T07:31:56.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef80c00640 1 --2- 172.21.15.114:0/2572617296 >> v2:172.21.15.177:3300/0 conn(0x7fef7c15fc10 0x7fef7c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.696+0000 7fef81600640 1 --2- 172.21.15.114:0/2572617296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c0ce4a0 0x7fef7c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.697+0000 7fef7be00640 1 --2- 172.21.15.114:0/2572617296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c15fff0 0x7fef7c160450 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0cf770 tx=0x7fef6802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9720e58c84fba6ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.697+0000 7fef7b400640 1 -- 172.21.15.114:0/2572617296 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68035020 con 0x7fef7c15fff0 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.697+0000 7fef7b400640 1 -- 172.21.15.114:0/2572617296 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6802d040 con 0x7fef7c15fff0 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.697+0000 7fef7b400640 1 -- 172.21.15.114:0/2572617296 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68031a80 con 0x7fef7c15fff0 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 -- 172.21.15.114:0/2572617296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c15fff0 msgr2=0x7fef7c160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 --2- 172.21.15.114:0/2572617296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c15fff0 0x7fef7c160450 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0cf770 tx=0x7fef6802b900 comp rx=0 tx=0).stop 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 -- 172.21.15.114:0/2572617296 shutdown_connections 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 --2- 172.21.15.114:0/2572617296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c0ce4a0 0x7fef7c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 --2- 172.21.15.114:0/2572617296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c15fff0 0x7fef7c160450 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 --2- 172.21.15.114:0/2572617296 >> v2:172.21.15.177:3300/0 conn(0x7fef7c15fc10 0x7fef7c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 -- 172.21.15.114:0/2572617296 >> 172.21.15.114:0/2572617296 conn(0x7fef7c0b1940 msgr2=0x7fef7c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 -- 172.21.15.114:0/2572617296 shutdown_connections 2024-03-06T07:31:56.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.698+0000 7fef83600640 1 -- 172.21.15.114:0/2572617296 wait complete. 2024-03-06T07:31:56.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.700+0000 7fef83600640 1 Processor -- start 2024-03-06T07:31:56.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.701+0000 7fef83600640 1 -- start start 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef83600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c0ce4a0 0x7fef7c1f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef83600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c15fc10 0x7fef7c09ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef83600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef7c15fff0 0x7fef7c0c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef83600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c072a30 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef83600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c0728b0 con 0x7fef7c15fff0 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef83600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c15c2e0 con 0x7fef7c15fc10 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef80c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c0ce4a0 0x7fef7c1f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef80c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c0ce4a0 0x7fef7c1f8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33350/0 (socket says 172.21.15.114:33350) 2024-03-06T07:31:56.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.702+0000 7fef80c00640 1 -- 172.21.15.114:0/2732763020 learned_addr learned my addr 172.21.15.114:0/2732763020 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef81600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef7c15fff0 0x7fef7c0c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef7be00640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c15fc10 0x7fef7c09ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef80c00640 1 -- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c15fc10 msgr2=0x7fef7c09ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef80c00640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c15fc10 0x7fef7c09ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef80c00640 1 -- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef7c15fff0 msgr2=0x7fef7c0c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef80c00640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef7c15fff0 0x7fef7c0c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef80c00640 1 -- 172.21.15.114:0/2732763020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c09c190 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef81600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef7c15fff0 0x7fef7c0c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef80c00640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c0ce4a0 0x7fef7c1f8180 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c7f0 tx=0x7fef7000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef78c00640 1 -- 172.21.15.114:0/2732763020 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70015070 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef78c00640 1 -- 172.21.15.114:0/2732763020 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef70007a30 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c09c470 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef78c00640 1 -- 172.21.15.114:0/2732763020 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70007e00 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.703+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c09ca30 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.705+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c04f330 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.706+0000 7fef78c00640 1 -- 172.21.15.114:0/2732763020 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef70018070 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.707+0000 7fef78c00640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5c06b4d0 0x7fef5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.707+0000 7fef78c00640 1 -- 172.21.15.114:0/2732763020 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fef7008a250 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.708+0000 7fef7be00640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5c06b4d0 0x7fef5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.708+0000 7fef7be00640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5c06b4d0 0x7fef5c06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fef68036000 tx=0x7fef68008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.711+0000 7fef78c00640 1 -- 172.21.15.114:0/2732763020 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef70058ff0 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.920+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef7c0b43e0 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.921+0000 7fef78c00640 1 -- 172.21.15.114:0/2732763020 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v68) v1 ==== 104+0+11 (secure 0 0 0) 0x7fef700588f0 con 0x7fef7c0ce4a0 2024-03-06T07:31:56.922 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:31:56.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.923+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5c06b4d0 msgr2=0x7fef5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.923+0000 7fef83600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5c06b4d0 0x7fef5c06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fef68036000 tx=0x7fef68008030 comp rx=0 tx=0).stop 2024-03-06T07:31:56.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.923+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c0ce4a0 msgr2=0x7fef7c1f8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.923+0000 7fef83600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c0ce4a0 0x7fef7c1f8180 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c7f0 tx=0x7fef7000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 shutdown_connections 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef7c15fff0 0x7fef7c0c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef7c15fc10 0x7fef7c09ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef5c06b4d0 0x7fef5c06d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 --2- 172.21.15.114:0/2732763020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef7c0ce4a0 0x7fef7c1f8180 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 >> 172.21.15.114:0/2732763020 conn(0x7fef7c0b1940 msgr2=0x7fef7c15b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 shutdown_connections 2024-03-06T07:31:56.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:56.924+0000 7fef83600640 1 -- 172.21.15.114:0/2732763020 wait complete. 2024-03-06T07:31:56.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:57.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.150+0000 7fd83a400640 1 Processor -- start 2024-03-06T07:31:57.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.150+0000 7fd83a400640 1 -- start start 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd83a400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340d4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd83a400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd8340d23c0 0x7fd8340d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd83a400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd834161ed0 0x7fd834162360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd83a400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340d2d60 con 0x7fd8340d23c0 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd83a400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340d1e60 con 0x7fd834161ed0 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd83a400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340cee60 con 0x7fd8341616e0 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd833e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd8340d23c0 0x7fd8340d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd833e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd8340d23c0 0x7fd8340d2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33366/0 (socket says 172.21.15.114:33366) 2024-03-06T07:31:57.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.152+0000 7fd833e00640 1 -- 172.21.15.114:0/4051673152 learned_addr learned my addr 172.21.15.114:0/4051673152 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:57.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.153+0000 7fd839a00640 1 --2- 172.21.15.114:0/4051673152 >> v2:172.21.15.177:3300/0 conn(0x7fd834161ed0 0x7fd834162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.153+0000 7fd839000640 1 --2- 172.21.15.114:0/4051673152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340d4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.153+0000 7fd833e00640 1 -- 172.21.15.114:0/4051673152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 msgr2=0x7fd8340d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.153+0000 7fd833e00640 1 --2- 172.21.15.114:0/4051673152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340d4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.153+0000 7fd833e00640 1 -- 172.21.15.114:0/4051673152 >> v2:172.21.15.177:3300/0 conn(0x7fd834161ed0 msgr2=0x7fd834162360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.153+0000 7fd833e00640 1 --2- 172.21.15.114:0/4051673152 >> v2:172.21.15.177:3300/0 conn(0x7fd834161ed0 0x7fd834162360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.153+0000 7fd833e00640 1 -- 172.21.15.114:0/4051673152 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340cf140 con 0x7fd8340d23c0 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.154+0000 7fd833e00640 1 --2- 172.21.15.114:0/4051673152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8340d23c0 0x7fd8340d2820 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd834067cc0 tx=0x7fd82002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705237d60a586151 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.154+0000 7fd833400640 1 -- 172.21.15.114:0/4051673152 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820035020 con 0x7fd8340d23c0 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd833400640 1 -- 172.21.15.114:0/4051673152 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd820031870 con 0x7fd8340d23c0 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd833400640 1 -- 172.21.15.114:0/4051673152 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820031c40 con 0x7fd8340d23c0 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 -- 172.21.15.114:0/4051673152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8340d23c0 msgr2=0x7fd8340d2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 --2- 172.21.15.114:0/4051673152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8340d23c0 0x7fd8340d2820 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd834067cc0 tx=0x7fd82002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 -- 172.21.15.114:0/4051673152 shutdown_connections 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 --2- 172.21.15.114:0/4051673152 >> v2:172.21.15.177:3300/0 conn(0x7fd834161ed0 0x7fd834162360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 --2- 172.21.15.114:0/4051673152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8340d23c0 0x7fd8340d2820 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 --2- 172.21.15.114:0/4051673152 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340d4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 -- 172.21.15.114:0/4051673152 >> 172.21.15.114:0/4051673152 conn(0x7fd83409d050 msgr2=0x7fd8340d3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 -- 172.21.15.114:0/4051673152 shutdown_connections 2024-03-06T07:31:57.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.155+0000 7fd83a400640 1 -- 172.21.15.114:0/4051673152 wait complete. 2024-03-06T07:31:57.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.157+0000 7fd83a400640 1 Processor -- start 2024-03-06T07:31:57.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.157+0000 7fd83a400640 1 -- start start 2024-03-06T07:31:57.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd83a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8340d23c0 0x7fd8340caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd83a400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd83a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834161ed0 0x7fd83415d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd83a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340cf9e0 con 0x7fd834161ed0 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd83a400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340cf700 con 0x7fd8340d23c0 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd83a400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340d4c40 con 0x7fd8341616e0 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd839a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834161ed0 0x7fd83415d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd839a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834161ed0 0x7fd83415d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33378/0 (socket says 172.21.15.114:33378) 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.158+0000 7fd839a00640 1 -- 172.21.15.114:0/3008803394 learned_addr learned my addr 172.21.15.114:0/3008803394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839000640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8340d23c0 0x7fd8340caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd833e00640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839a00640 1 -- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 msgr2=0x7fd8340cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839a00640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340cb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839a00640 1 -- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8340d23c0 msgr2=0x7fd8340caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839a00640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8340d23c0 0x7fd8340caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839a00640 1 -- 172.21.15.114:0/3008803394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83415f2c0 con 0x7fd834161ed0 2024-03-06T07:31:57.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839000640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8340d23c0 0x7fd8340caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:57.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd839a00640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834161ed0 0x7fd83415d120 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd82c007e70 tx=0x7fd82c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd830c00640 1 -- 172.21.15.114:0/3008803394 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c014620 con 0x7fd834161ed0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd830c00640 1 -- 172.21.15.114:0/3008803394 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c019070 con 0x7fd834161ed0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.159+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8340cbb00 con 0x7fd834161ed0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.160+0000 7fd830c00640 1 -- 172.21.15.114:0/3008803394 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c014a20 con 0x7fd834161ed0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.160+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340cc0c0 con 0x7fd834161ed0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.161+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83404f330 con 0x7fd834161ed0 2024-03-06T07:31:57.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.162+0000 7fd830c00640 1 -- 172.21.15.114:0/3008803394 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd82c014bc0 con 0x7fd834161ed0 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.163+0000 7fd830c00640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd81406b500 0x7fd81406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.163+0000 7fd830c00640 1 -- 172.21.15.114:0/3008803394 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd82c08aaa0 con 0x7fd834161ed0 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.163+0000 7fd839000640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd81406b500 0x7fd81406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.164+0000 7fd839000640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd81406b500 0x7fd81406d9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd828006fd0 tx=0x7fd828008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.167+0000 7fd830c00640 1 -- 172.21.15.114:0/3008803394 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd82c0598d0 con 0x7fd834161ed0 2024-03-06T07:31:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.364+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8340d21a0 con 0x7fd834161ed0 2024-03-06T07:31:57.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.364+0000 7fd830c00640 1 -- 172.21.15.114:0/3008803394 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v68) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd82c0591d0 con 0x7fd834161ed0 2024-03-06T07:31:57.366 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd81406b500 msgr2=0x7fd81406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd81406b500 0x7fd81406d9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd828006fd0 tx=0x7fd828008040 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834161ed0 msgr2=0x7fd83415d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834161ed0 0x7fd83415d120 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd82c007e70 tx=0x7fd82c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 shutdown_connections 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd81406b500 0x7fd81406d9c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd834161ed0 0x7fd83415d120 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8341616e0 0x7fd8340cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 --2- 172.21.15.114:0/3008803394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8340d23c0 0x7fd8340caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 >> 172.21.15.114:0/3008803394 conn(0x7fd83409d050 msgr2=0x7fd8340d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 shutdown_connections 2024-03-06T07:31:57.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.367+0000 7fd83a400640 1 -- 172.21.15.114:0/3008803394 wait complete. 2024-03-06T07:31:57.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:57.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.588+0000 7fd3b5a00640 1 Processor -- start 2024-03-06T07:31:57.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.588+0000 7fd3b5a00640 1 -- start start 2024-03-06T07:31:57.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3b5a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd3b00c5040 0x7fd3b00cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3b5a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3b01474c0 0x7fd3b00cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3b5a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b00ae380 con 0x7fd3b01444e0 2024-03-06T07:31:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3b5a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b00ae660 con 0x7fd3b01474c0 2024-03-06T07:31:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3b5a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b00ae940 con 0x7fd3b00c5040 2024-03-06T07:31:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3aea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd3b01444e0 0x7fd3b00c4930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36304/0 (socket says 172.21.15.114:36304) 2024-03-06T07:31:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3aea00640 1 -- 172.21.15.114:0/772332682 learned_addr learned my addr 172.21.15.114:0/772332682 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:57.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3af400640 1 --2- 172.21.15.114:0/772332682 >> v2:172.21.15.114:3300/0 conn(0x7fd3b01474c0 0x7fd3b00cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.590+0000 7fd3ae000640 1 --2- 172.21.15.114:0/772332682 >> v2:172.21.15.177:3300/0 conn(0x7fd3b00c5040 0x7fd3b00cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.591+0000 7fd3ad600640 1 -- 172.21.15.114:0/772332682 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834201073 0 0) 0x7fd3b00ae380 con 0x7fd3b01444e0 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.591+0000 7fd3ad600640 1 -- 172.21.15.114:0/772332682 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd394003520 con 0x7fd3b01444e0 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.591+0000 7fd3af400640 1 -- 172.21.15.114:0/772332682 >> v2:172.21.15.177:3300/0 conn(0x7fd3b00c5040 msgr2=0x7fd3b00cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.591+0000 7fd3af400640 1 --2- 172.21.15.114:0/772332682 >> v2:172.21.15.177:3300/0 conn(0x7fd3b00c5040 0x7fd3b00cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.591+0000 7fd3af400640 1 -- 172.21.15.114:0/772332682 >> v1:172.21.15.114:6789/0 conn(0x7fd3b01444e0 legacy=0x7fd3b00c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.591+0000 7fd3af400640 1 -- 172.21.15.114:0/772332682 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b00aec20 con 0x7fd3b01474c0 2024-03-06T07:31:57.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.591+0000 7fd3ae000640 1 --2- 172.21.15.114:0/772332682 >> v2:172.21.15.177:3300/0 conn(0x7fd3b00c5040 0x7fd3b00cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.592+0000 7fd3af400640 1 --2- 172.21.15.114:0/772332682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cd920 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0068c90 tx=0x7fd3a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dcef0ef6320dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.592+0000 7fd3ad600640 1 -- 172.21.15.114:0/772332682 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8039070 con 0x7fd3b01474c0 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.592+0000 7fd3ad600640 1 -- 172.21.15.114:0/772332682 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a802cc80 con 0x7fd3b01474c0 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.592+0000 7fd3ad600640 1 -- 172.21.15.114:0/772332682 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8034660 con 0x7fd3b01474c0 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 -- 172.21.15.114:0/772332682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 msgr2=0x7fd3b00cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/772332682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cd920 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0068c90 tx=0x7fd3a802bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 -- 172.21.15.114:0/772332682 shutdown_connections 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/772332682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cd920 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/772332682 >> v2:172.21.15.177:3300/0 conn(0x7fd3b00c5040 0x7fd3b00cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 -- 172.21.15.114:0/772332682 >> 172.21.15.114:0/772332682 conn(0x7fd3b00cc4f0 msgr2=0x7fd3b00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 -- 172.21.15.114:0/772332682 shutdown_connections 2024-03-06T07:31:57.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.593+0000 7fd3b5a00640 1 -- 172.21.15.114:0/772332682 wait complete. 2024-03-06T07:31:57.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.595+0000 7fd3b5a00640 1 Processor -- start 2024-03-06T07:31:57.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.595+0000 7fd3b5a00640 1 -- start start 2024-03-06T07:31:57.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3b5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b00c5040 0x7fd3b015d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3b5a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b01444e0 0x7fd3b00c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3b5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3b5a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01448c0 con 0x7fd3b01474c0 2024-03-06T07:31:57.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3b5a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b0147cb0 con 0x7fd3b00c5040 2024-03-06T07:31:57.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3b5a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b0144a40 con 0x7fd3b01444e0 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33392/0 (socket says 172.21.15.114:33392) 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.596+0000 7fd3af400640 1 -- 172.21.15.114:0/1218804653 learned_addr learned my addr 172.21.15.114:0/1218804653 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3aea00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b00c5040 0x7fd3b015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3af400640 1 -- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b01444e0 msgr2=0x7fd3b00c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3af400640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b01444e0 0x7fd3b00c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3af400640 1 -- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b00c5040 msgr2=0x7fd3b015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3af400640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b00c5040 0x7fd3b015d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3af400640 1 -- 172.21.15.114:0/1218804653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b00ca100 con 0x7fd3b01474c0 2024-03-06T07:31:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3aea00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b00c5040 0x7fd3b015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3af400640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cb760 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd3a802c4e0 tx=0x7fd3a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3a6a00640 1 -- 172.21.15.114:0/1218804653 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8039050 con 0x7fd3b01474c0 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3a6a00640 1 -- 172.21.15.114:0/1218804653 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a8045070 con 0x7fd3b01474c0 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b00c6dc0 con 0x7fd3b01474c0 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.597+0000 7fd3a6a00640 1 -- 172.21.15.114:0/1218804653 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8009450 con 0x7fd3b01474c0 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.598+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b00c7380 con 0x7fd3b01474c0 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.599+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b00c6230 con 0x7fd3b01474c0 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.600+0000 7fd3a6a00640 1 -- 172.21.15.114:0/1218804653 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3a803e850 con 0x7fd3b01474c0 2024-03-06T07:31:57.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.600+0000 7fd3a6a00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd39006b4d0 0x7fd39006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.601+0000 7fd3a6a00640 1 -- 172.21.15.114:0/1218804653 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd3a80ad240 con 0x7fd3b01474c0 2024-03-06T07:31:57.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.601+0000 7fd3aea00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd39006b4d0 0x7fd39006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.601+0000 7fd3aea00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd39006b4d0 0x7fd39006d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0005880 tx=0x7fd3a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.605+0000 7fd3a6a00640 1 -- 172.21.15.114:0/1218804653 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3b00c6230 con 0x7fd3b01474c0 2024-03-06T07:31:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.814+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd3b01e88e0 con 0x7fd3b01474c0 2024-03-06T07:31:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.815+0000 7fd3a6a00640 1 -- 172.21.15.114:0/1218804653 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v68) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd3a807c100 con 0x7fd3b01474c0 2024-03-06T07:31:57.817 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:31:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.817+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd39006b4d0 msgr2=0x7fd39006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.817+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd39006b4d0 0x7fd39006d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0005880 tx=0x7fd3a0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.817+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 msgr2=0x7fd3b00cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.817+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cb760 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd3a802c4e0 tx=0x7fd3a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 shutdown_connections 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd39006b4d0 0x7fd39006d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b01474c0 0x7fd3b00cb760 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3b01444e0 0x7fd3b00c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 --2- 172.21.15.114:0/1218804653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3b00c5040 0x7fd3b015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 >> 172.21.15.114:0/1218804653 conn(0x7fd3b00cc4f0 msgr2=0x7fd3b00725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 shutdown_connections 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:57.818+0000 7fd3b5a00640 1 -- 172.21.15.114:0/1218804653 wait complete. 2024-03-06T07:31:57.837 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:31:57.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:57.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:57.947 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:31:57.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:57.947 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:57.999 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:31:57.999 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:58.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:58.038 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:31:58.038 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:58.078 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:58.078 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:31:58.078 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:58.127 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:58.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.315+0000 7f6083e00640 1 Processor -- start 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.315+0000 7f6083e00640 1 -- start start 2024-03-06T07:31:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.316+0000 7f6083e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f60840d27a0 0x7f60840b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.316+0000 7f6083e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6084161960 con 0x7f6084161580 2024-03-06T07:31:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.316+0000 7f6083e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60840cf6f0 con 0x7f60840d2360 2024-03-06T07:31:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.316+0000 7f6083e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60840cf9d0 con 0x7f60840d27a0 2024-03-06T07:31:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.316+0000 7f6083400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f60840d27a0 0x7f60840b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.317+0000 7f6082a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6084161580 0x7f60840d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44248/0 (socket says 172.21.15.114:44248) 2024-03-06T07:31:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.317+0000 7f6082a00640 1 -- 172.21.15.114:0/4256131218 learned_addr learned my addr 172.21.15.114:0/4256131218 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.317+0000 7f6083400640 1 -- 172.21.15.114:0/4256131218 >> v1:172.21.15.177:6789/0 conn(0x7f60840d2360 legacy=0x7f60840b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.317+0000 7f6083400640 1 -- 172.21.15.114:0/4256131218 >> v1:172.21.15.114:6789/0 conn(0x7f6084161580 legacy=0x7f60840d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.317+0000 7f6082000640 1 -- 172.21.15.114:0/4256131218 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854198469 0 0) 0x7f6084161960 con 0x7f6084161580 2024-03-06T07:31:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.317+0000 7f6083400640 1 -- 172.21.15.114:0/4256131218 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840cfcb0 con 0x7f60840d27a0 2024-03-06T07:31:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.318+0000 7f6083400640 1 --2- 172.21.15.114:0/4256131218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f60840b45d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f607c009d90 tx=0x7f607c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0d826285a38a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.318+0000 7f6082000640 1 -- 172.21.15.114:0/4256131218 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c039070 con 0x7f60840d27a0 2024-03-06T07:31:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.318+0000 7f6082000640 1 -- 172.21.15.114:0/4256131218 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c02cc60 con 0x7f60840d27a0 2024-03-06T07:31:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.318+0000 7f6082000640 1 -- 172.21.15.114:0/4256131218 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c034660 con 0x7f60840d27a0 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.319+0000 7f6083e00640 1 -- 172.21.15.114:0/4256131218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 msgr2=0x7f60840b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.319+0000 7f6083e00640 1 --2- 172.21.15.114:0/4256131218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f60840b45d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f607c009d90 tx=0x7f607c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.319+0000 7f6083e00640 1 -- 172.21.15.114:0/4256131218 shutdown_connections 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.319+0000 7f6083e00640 1 --2- 172.21.15.114:0/4256131218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f60840b45d0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.319+0000 7f6083e00640 1 -- 172.21.15.114:0/4256131218 >> 172.21.15.114:0/4256131218 conn(0x7f60840d3d80 msgr2=0x7f60840d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.319+0000 7f6083e00640 1 -- 172.21.15.114:0/4256131218 shutdown_connections 2024-03-06T07:31:58.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.319+0000 7f6083e00640 1 -- 172.21.15.114:0/4256131218 wait complete. 2024-03-06T07:31:58.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.321+0000 7f6083e00640 1 Processor -- start 2024-03-06T07:31:58.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.321+0000 7f6083e00640 1 -- start start 2024-03-06T07:31:58.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f6083e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60840d2360 0x7f60841f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f6083e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f608414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f6083e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6084161580 0x7f60840cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f6083e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840d2bf0 con 0x7f60840d27a0 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f6083e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840b39a0 con 0x7f60840d2360 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f6083e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60840cf3a0 con 0x7f6084161580 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f608414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f608414b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34002/0 (socket says 172.21.15.114:34002) 2024-03-06T07:31:58.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 -- 172.21.15.114:0/411337712 learned_addr learned my addr 172.21.15.114:0/411337712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:58.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f6083400640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6084161580 0x7f60840cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 -- 172.21.15.114:0/411337712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6084161580 msgr2=0x7f60840cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6084161580 0x7f60840cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 -- 172.21.15.114:0/411337712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60840d2360 msgr2=0x7f60841f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60840d2360 0x7f60841f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.322+0000 7f607be00640 1 -- 172.21.15.114:0/411337712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60841f3bb0 con 0x7f60840d27a0 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.323+0000 7f607be00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f608414b320 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f606c00b3f0 tx=0x7f606c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.323+0000 7f607aa00640 1 -- 172.21.15.114:0/411337712 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c015020 con 0x7f60840d27a0 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.323+0000 7f607aa00640 1 -- 172.21.15.114:0/411337712 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c011830 con 0x7f60840d27a0 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.323+0000 7f607aa00640 1 -- 172.21.15.114:0/411337712 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c011c20 con 0x7f60840d27a0 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.323+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60841f3e90 con 0x7f60840d27a0 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.323+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60841f4450 con 0x7f60840d27a0 2024-03-06T07:31:58.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.324+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608404f330 con 0x7f60840d27a0 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.325+0000 7f607aa00640 1 -- 172.21.15.114:0/411337712 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f606c018020 con 0x7f60840d27a0 2024-03-06T07:31:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.326+0000 7f607aa00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.326+0000 7f607aa00640 1 -- 172.21.15.114:0/411337712 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f606c089ef0 con 0x7f60840d27a0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.326+0000 7f6082a00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.327+0000 7f6082a00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6074007fd0 tx=0x7f6074007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.331+0000 7f607aa00640 1 -- 172.21.15.114:0/411337712 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f606c01f030 con 0x7f60840d27a0 2024-03-06T07:31:58.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.554+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60840d3b60 con 0x7f60840d27a0 2024-03-06T07:31:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.555+0000 7f607aa00640 1 -- 172.21.15.114:0/411337712 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f606c0172c0 con 0x7f60840d27a0 2024-03-06T07:31:58.557 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:58.557 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:31:58.557 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:31:58.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f606006b4d0 msgr2=0x7f606006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6074007fd0 tx=0x7f6074007920 comp rx=0 tx=0).stop 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 msgr2=0x7f608414b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f608414b320 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f606c00b3f0 tx=0x7f606c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 shutdown_connections 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6084161580 0x7f60840cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60840d27a0 0x7f608414b320 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 --2- 172.21.15.114:0/411337712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60840d2360 0x7f60841f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.558+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 >> 172.21.15.114:0/411337712 conn(0x7f60840d3d80 msgr2=0x7f60840ca650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.559+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 shutdown_connections 2024-03-06T07:31:58.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.559+0000 7f6083e00640 1 -- 172.21.15.114:0/411337712 wait complete. 2024-03-06T07:31:58.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:58.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.784+0000 7f23c5600640 1 Processor -- start 2024-03-06T07:31:58.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.784+0000 7f23c5600640 1 -- start start 2024-03-06T07:31:58.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c5600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f23c015e200 0x7f23c00d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c5600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23c00d0c00 0x7f23c00b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c5600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c00b5240 con 0x7f23c00b2e50 2024-03-06T07:31:58.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c5600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c00d2ec0 con 0x7f23c00d0c00 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c5600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c00cf4d0 con 0x7f23c015e200 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c4c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23c00d0c00 0x7f23c00b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c4c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23c00d0c00 0x7f23c00b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34016/0 (socket says 172.21.15.114:34016) 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c4c00640 1 -- 172.21.15.114:0/1846900948 learned_addr learned my addr 172.21.15.114:0/1846900948 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c4c00640 1 -- 172.21.15.114:0/1846900948 >> v2:172.21.15.177:3300/0 conn(0x7f23c015e200 msgr2=0x7f23c00d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c4c00640 1 --2- 172.21.15.114:0/1846900948 >> v2:172.21.15.177:3300/0 conn(0x7f23c015e200 0x7f23c00d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c4c00640 1 -- 172.21.15.114:0/1846900948 >> v1:172.21.15.177:6789/0 conn(0x7f23c00b2e50 legacy=0x7f23c00d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.786+0000 7f23c4c00640 1 -- 172.21.15.114:0/1846900948 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c00b5500 con 0x7f23c00d0c00 2024-03-06T07:31:58.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.787+0000 7f23c4c00640 1 --2- 172.21.15.114:0/1846900948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c00d0c00 0x7f23c00b3640 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f23c00d3d00 tx=0x7f23b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25dc49e5ab7e76f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.788+0000 7f23bf400640 1 -- 172.21.15.114:0/1846900948 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8034070 con 0x7f23c00d0c00 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.788+0000 7f23bf400640 1 -- 172.21.15.114:0/1846900948 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b8030860 con 0x7f23c00d0c00 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.788+0000 7f23bf400640 1 -- 172.21.15.114:0/1846900948 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8030c30 con 0x7f23c00d0c00 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.788+0000 7f23c5600640 1 -- 172.21.15.114:0/1846900948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c00d0c00 msgr2=0x7f23c00b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.788+0000 7f23c5600640 1 --2- 172.21.15.114:0/1846900948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c00d0c00 0x7f23c00b3640 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f23c00d3d00 tx=0x7f23b802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.789+0000 7f23c5600640 1 -- 172.21.15.114:0/1846900948 shutdown_connections 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.789+0000 7f23c5600640 1 --2- 172.21.15.114:0/1846900948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c00d0c00 0x7f23c00b3640 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.789+0000 7f23c5600640 1 --2- 172.21.15.114:0/1846900948 >> v2:172.21.15.177:3300/0 conn(0x7f23c015e200 0x7f23c00d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.789+0000 7f23c5600640 1 -- 172.21.15.114:0/1846900948 >> 172.21.15.114:0/1846900948 conn(0x7f23c00d3d30 msgr2=0x7f23c00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.789+0000 7f23c5600640 1 -- 172.21.15.114:0/1846900948 shutdown_connections 2024-03-06T07:31:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.789+0000 7f23c5600640 1 -- 172.21.15.114:0/1846900948 wait complete. 2024-03-06T07:31:58.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.790+0000 7f23c5600640 1 Processor -- start 2024-03-06T07:31:58.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.790+0000 7f23c5600640 1 -- start start 2024-03-06T07:31:58.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.791+0000 7f23c5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c00b2e50 0x7f23c00cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.791+0000 7f23c5600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23c00d0c00 0x7f23c00cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.791+0000 7f23c5600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015e200 0x7f23c0151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.791+0000 7f23c5600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c00cfd60 con 0x7f23c015e200 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.791+0000 7f23c5600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c00cfa80 con 0x7f23c00b2e50 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.791+0000 7f23c5600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c015e5e0 con 0x7f23c00d0c00 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015e200 0x7f23c0151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015e200 0x7f23c0151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34020/0 (socket says 172.21.15.114:34020) 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 -- 172.21.15.114:0/783062020 learned_addr learned my addr 172.21.15.114:0/783062020 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 -- 172.21.15.114:0/783062020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23c00d0c00 msgr2=0x7f23c00cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23c00d0c00 0x7f23c00cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 -- 172.21.15.114:0/783062020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c00b2e50 msgr2=0x7f23c00cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23bfe00640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c00b2e50 0x7f23c00cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c00b2e50 0x7f23c00cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 -- 172.21.15.114:0/783062020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c00b1ee0 con 0x7f23c015e200 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23c4c00640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015e200 0x7f23c0151da0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f23b802b9e0 tx=0x7f23b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.792+0000 7f23bcc00640 1 -- 172.21.15.114:0/783062020 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8048070 con 0x7f23c015e200 2024-03-06T07:31:58.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.793+0000 7f23bcc00640 1 -- 172.21.15.114:0/783062020 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b8030660 con 0x7f23c015e200 2024-03-06T07:31:58.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.793+0000 7f23bcc00640 1 -- 172.21.15.114:0/783062020 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8034040 con 0x7f23c015e200 2024-03-06T07:31:58.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.793+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c00b21c0 con 0x7f23c015e200 2024-03-06T07:31:58.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.793+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c00b2780 con 0x7f23c015e200 2024-03-06T07:31:58.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.794+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c00c8800 con 0x7f23c015e200 2024-03-06T07:31:58.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.795+0000 7f23bcc00640 1 -- 172.21.15.114:0/783062020 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23b804b070 con 0x7f23c015e200 2024-03-06T07:31:58.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.796+0000 7f23bcc00640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f239c06b500 0x7f239c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.796+0000 7f23bcc00640 1 -- 172.21.15.114:0/783062020 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f23b80ac4b0 con 0x7f23c015e200 2024-03-06T07:31:58.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.796+0000 7f23bfe00640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f239c06b500 0x7f239c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.796+0000 7f23bfe00640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f239c06b500 0x7f239c06d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007dc0 tx=0x7f23b00072f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.805+0000 7f23bcc00640 1 -- 172.21.15.114:0/783062020 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23b807b370 con 0x7f23c015e200 2024-03-06T07:31:59.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:58.999+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c008d4c0 con 0x7f23c015e200 2024-03-06T07:31:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.000+0000 7f23bcc00640 1 -- 172.21.15.114:0/783062020 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16814 (secure 0 0 0) 0x7f23b807ac70 con 0x7f23c015e200 2024-03-06T07:31:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:31:59.003 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":68,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:37.032494+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:18.351057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:19.518313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.20","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f239c06b500 msgr2=0x7f239c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f239c06b500 0x7f239c06d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007dc0 tx=0x7f23b00072f0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015e200 msgr2=0x7f23c0151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015e200 0x7f23c0151da0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f23b802b9e0 tx=0x7f23b8004730 comp rx=0 tx=0).stop 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 shutdown_connections 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f239c06b500 0x7f239c06d9c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015e200 0x7f23c0151da0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f23c00d0c00 0x7f23c00cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 --2- 172.21.15.114:0/783062020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23c00b2e50 0x7f23c00cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 >> 172.21.15.114:0/783062020 conn(0x7f23c00d3d30 msgr2=0x7f23c006b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 shutdown_connections 2024-03-06T07:31:59.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.003+0000 7f23c5600640 1 -- 172.21.15.114:0/783062020 wait complete. 2024-03-06T07:31:59.027 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:18.351057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:19.518313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:59.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:59.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.239+0000 7f46a5000640 1 Processor -- start 2024-03-06T07:31:59.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.239+0000 7f46a5000640 1 -- start start 2024-03-06T07:31:59.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.241+0000 7f46a5000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 0x7f46a00d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f46a5000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f46a00d1bf0 0x7f46a00cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f46a5000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a00b3730 con 0x7f46a015e450 2024-03-06T07:31:59.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f46a5000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a00d4930 con 0x7f46a00d1bf0 2024-03-06T07:31:59.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f46a5000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0067b20 con 0x7f46a00d07f0 2024-03-06T07:31:59.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f469d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f46a015e450 0x7f46a00b4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44254/0 (socket says 172.21.15.114:44254) 2024-03-06T07:31:59.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f469d400640 1 -- 172.21.15.114:0/1683369299 learned_addr learned my addr 172.21.15.114:0/1683369299 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f469de00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 0x7f46a00d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f469e800640 1 --2- 172.21.15.114:0/1683369299 >> v2:172.21.15.177:3300/0 conn(0x7f46a00d1bf0 0x7f46a00cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.242+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150204170 0 0) 0x7f46a00b3730 con 0x7f46a015e450 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4684003520 con 0x7f46a015e450 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124511755 0 0) 0x7f4684003520 con 0x7f46a015e450 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 msgr2=0x7f46a00d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 0x7f46a00d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 >> v2:172.21.15.177:3300/0 conn(0x7f46a00d1bf0 msgr2=0x7f46a00cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 --2- 172.21.15.114:0/1683369299 >> v2:172.21.15.177:3300/0 conn(0x7f46a00d1bf0 0x7f46a00cfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a00d1110 con 0x7f46a015e450 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f468c001670 con 0x7f46a015e450 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 >> v1:172.21.15.114:6789/0 conn(0x7f46a015e450 legacy=0x7f46a00b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46840039b0 0x7f4684003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 0x7f4684004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4684004d40 0x7f46840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00d4930 con 0x7f46840039b0 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0067b20 con 0x7f4684004d40 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468400ba70 con 0x7f46a00d07f0 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469e800640 1 --2- 172.21.15.114:0/1683369299 >> v2:172.21.15.177:3300/0 conn(0x7f46a00d1bf0 0x7f46a00cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.243+0000 7f469de00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46840039b0 0x7f4684003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469d400640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 0x7f4684004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469de00640 1 -- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 msgr2=0x7f4684004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469e800640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4684004d40 0x7f46840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469de00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 0x7f4684004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469de00640 1 -- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4684004d40 msgr2=0x7f46840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469de00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4684004d40 0x7f46840051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469de00640 1 -- 172.21.15.114:0/1683369299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684003520 con 0x7f46840039b0 2024-03-06T07:31:59.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.244+0000 7f469d400640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a00d07f0 0x7f4684004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.246+0000 7f469de00640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46840039b0 0x7f4684003dd0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4690009fc0 tx=0x7f4684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec69e62e0769d7ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.246+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690039070 con 0x7f46840039b0 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.246+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469002cbf0 con 0x7f46840039b0 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.246+0000 7f469ca00640 1 -- 172.21.15.114:0/1683369299 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690009c10 con 0x7f46840039b0 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.247+0000 7f46a5000640 1 -- 172.21.15.114:0/1683369299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46840039b0 msgr2=0x7f4684003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.247+0000 7f46a5000640 1 --2- 172.21.15.114:0/1683369299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46840039b0 0x7f4684003dd0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4690009fc0 tx=0x7f4684005830 comp rx=0 tx=0).stop 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.247+0000 7f469e800640 1 -- 172.21.15.114:0/1683369299 reap_dead start 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.247+0000 7f46a5000640 1 -- 172.21.15.114:0/1683369299 shutdown_connections 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.247+0000 7f46a5000640 1 -- 172.21.15.114:0/1683369299 >> 172.21.15.114:0/1683369299 conn(0x7f46a00d3e10 msgr2=0x7f46a00d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.247+0000 7f46a5000640 1 -- 172.21.15.114:0/1683369299 shutdown_connections 2024-03-06T07:31:59.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.247+0000 7f46a5000640 1 -- 172.21.15.114:0/1683369299 wait complete. 2024-03-06T07:31:59.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.250+0000 7f46a5000640 1 Processor -- start 2024-03-06T07:31:59.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.250+0000 7f46a5000640 1 -- start start 2024-03-06T07:31:59.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f46a5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46840039b0 0x7f46a014ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f46a5000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a014e120 0x7f46a00b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f46a5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46a00b24f0 0x7f46a01f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f46a5000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00b3c80 con 0x7f46a00b24f0 2024-03-06T07:31:59.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f46a5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00b39a0 con 0x7f46840039b0 2024-03-06T07:31:59.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f46a5000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00b3f60 con 0x7f46a014e120 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46a00b24f0 0x7f46a01f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46a00b24f0 0x7f46a01f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34036/0 (socket says 172.21.15.114:34036) 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 -- 172.21.15.114:0/2218160739 learned_addr learned my addr 172.21.15.114:0/2218160739 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469de00640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46840039b0 0x7f46a014ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469d400640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a014e120 0x7f46a00b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 -- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a014e120 msgr2=0x7f46a00b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a014e120 0x7f46a00b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 -- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46840039b0 msgr2=0x7f46a014ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46840039b0 0x7f46a014ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.252+0000 7f469e800640 1 -- 172.21.15.114:0/2218160739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a01f7de0 con 0x7f46a00b24f0 2024-03-06T07:31:59.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.253+0000 7f469e800640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46a00b24f0 0x7f46a01f7070 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f46a00b9ec0 tx=0x7f469800f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.253+0000 7f4696000640 1 -- 172.21.15.114:0/2218160739 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698018020 con 0x7f46a00b24f0 2024-03-06T07:31:59.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.253+0000 7f4696000640 1 -- 172.21.15.114:0/2218160739 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46980146a0 con 0x7f46a00b24f0 2024-03-06T07:31:59.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.253+0000 7f4696000640 1 -- 172.21.15.114:0/2218160739 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698014a90 con 0x7f46a00b24f0 2024-03-06T07:31:59.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.253+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a00cb130 con 0x7f46a00b24f0 2024-03-06T07:31:59.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.253+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a014e5a0 con 0x7f46a00b24f0 2024-03-06T07:31:59.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.254+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a004f330 con 0x7f46a00b24f0 2024-03-06T07:31:59.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.255+0000 7f4696000640 1 -- 172.21.15.114:0/2218160739 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f469801b020 con 0x7f46a00b24f0 2024-03-06T07:31:59.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.257+0000 7f4696000640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f468006b500 0x7f468006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.257+0000 7f4696000640 1 -- 172.21.15.114:0/2218160739 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4698089f60 con 0x7f46a00b24f0 2024-03-06T07:31:59.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.257+0000 7f469de00640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f468006b500 0x7f468006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.258+0000 7f469de00640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f468006b500 0x7f468006d9c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4690036000 tx=0x7f469002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.261+0000 7f4696000640 1 -- 172.21.15.114:0/2218160739 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4698022030 con 0x7f46a00b24f0 2024-03-06T07:31:59.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.458+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46a00ccdc0 con 0x7f46a00b24f0 2024-03-06T07:31:59.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.458+0000 7f4696000640 1 -- 172.21.15.114:0/2218160739 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4698058d90 con 0x7f46a00b24f0 2024-03-06T07:31:59.461 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f468006b500 msgr2=0x7f468006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f468006b500 0x7f468006d9c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4690036000 tx=0x7f469002e000 comp rx=0 tx=0).stop 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46a00b24f0 msgr2=0x7f46a01f7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46a00b24f0 0x7f46a01f7070 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f46a00b9ec0 tx=0x7f469800f670 comp rx=0 tx=0).stop 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 shutdown_connections 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f468006b500 0x7f468006d9c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46a00b24f0 0x7f46a01f7070 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f46a014e120 0x7f46a00b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 --2- 172.21.15.114:0/2218160739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46840039b0 0x7f46a014ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 >> 172.21.15.114:0/2218160739 conn(0x7f46a00d3e10 msgr2=0x7f46a00b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 shutdown_connections 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.461+0000 7f46a5000640 1 -- 172.21.15.114:0/2218160739 wait complete. 2024-03-06T07:31:59.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:59.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.682+0000 7f249b400640 1 Processor -- start 2024-03-06T07:31:59.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.682+0000 7f249b400640 1 -- start start 2024-03-06T07:31:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.684+0000 7f249b400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 0x7f2494165710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.684+0000 7f249b400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ce690 con 0x7f24940b3530 2024-03-06T07:31:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.684+0000 7f249b400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ce970 con 0x7f24940adf70 2024-03-06T07:31:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.684+0000 7f249b400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24940cec50 con 0x7f2494163320 2024-03-06T07:31:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.684+0000 7f2498a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f24940b3530 0x7f249415b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44256/0 (socket says 172.21.15.114:44256) 2024-03-06T07:31:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.684+0000 7f2498a00640 1 -- 172.21.15.114:0/3346397290 learned_addr learned my addr 172.21.15.114:0/3346397290 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:31:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.684+0000 7f2499400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 0x7f2494165710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126997101 0 0) 0x7f24940ce690 con 0x7f24940b3530 2024-03-06T07:31:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247c005730 con 0x7f24940b3530 2024-03-06T07:31:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389885881 0 0) 0x7f247c005730 con 0x7f24940b3530 2024-03-06T07:31:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 msgr2=0x7f2494165710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 0x7f2494165710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 >> v1:172.21.15.177:6789/0 conn(0x7f24940adf70 legacy=0x7f24940b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940cef30 con 0x7f24940b3530 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2488001670 con 0x7f24940b3530 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 >> v1:172.21.15.114:6789/0 conn(0x7f24940b3530 legacy=0x7f249415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 0x7f247c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f247c0061c0 0x7f247c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c006f50 0x7f247c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24940cec50 con 0x7f247c006f50 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c00dd10 con 0x7f247c0061c0 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.685+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c00df10 con 0x7f2494163320 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2499400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c006f50 0x7f247c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2498a00640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 0x7f247c005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2493e00640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f247c0061c0 0x7f247c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2499400640 1 -- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 msgr2=0x7f247c005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2499400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 0x7f247c005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2499400640 1 -- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f247c0061c0 msgr2=0x7f247c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2499400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f247c0061c0 0x7f247c006a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2499400640 1 -- 172.21.15.114:0/3346397290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c005730 con 0x7f247c006f50 2024-03-06T07:31:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.686+0000 7f2498a00640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2494163320 0x7f247c005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.687+0000 7f2499400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c006f50 0x7f247c0073d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f248c009fc0 tx=0x7f247c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0550cffbe62852 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.687+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c039070 con 0x7f247c006f50 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.687+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c02cc20 con 0x7f247c006f50 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.687+0000 7f2493400640 1 -- 172.21.15.114:0/3346397290 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c009c10 con 0x7f247c006f50 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.688+0000 7f249b400640 1 -- 172.21.15.114:0/3346397290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c006f50 msgr2=0x7f247c0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.688+0000 7f249b400640 1 --2- 172.21.15.114:0/3346397290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f247c006f50 0x7f247c0073d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f248c009fc0 tx=0x7f247c007a40 comp rx=0 tx=0).stop 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.688+0000 7f2499400640 1 -- 172.21.15.114:0/3346397290 reap_dead start 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.688+0000 7f249b400640 1 -- 172.21.15.114:0/3346397290 shutdown_connections 2024-03-06T07:31:59.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.688+0000 7f249b400640 1 -- 172.21.15.114:0/3346397290 >> 172.21.15.114:0/3346397290 conn(0x7f24940b1940 msgr2=0x7f24940b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.688+0000 7f249b400640 1 -- 172.21.15.114:0/3346397290 shutdown_connections 2024-03-06T07:31:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.688+0000 7f249b400640 1 -- 172.21.15.114:0/3346397290 wait complete. 2024-03-06T07:31:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.689+0000 7f249b400640 1 Processor -- start 2024-03-06T07:31:59.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.690+0000 7f249b400640 1 -- start start 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f249b400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f249409cf40 0x7f2494152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f249b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f249409d320 0x7f2494152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f249b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f249415d270 0x7f249414b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f249b400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249415abb0 con 0x7f249415d270 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f249b400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249415aa30 con 0x7f249409d320 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f249b400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24940af120 con 0x7f249409cf40 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f249415d270 0x7f249414b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f249415d270 0x7f249414b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34060/0 (socket says 172.21.15.114:34060) 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 -- 172.21.15.114:0/2606613049 learned_addr learned my addr 172.21.15.114:0/2606613049 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2498a00640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f249409cf40 0x7f2494152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 -- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f249409cf40 msgr2=0x7f2494152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f249409cf40 0x7f2494152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2493e00640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f249409d320 0x7f2494152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 -- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f249409d320 msgr2=0x7f2494152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f249409d320 0x7f2494152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2499400640 1 -- 172.21.15.114:0/2606613049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249415d810 con 0x7f249415d270 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.691+0000 7f2498a00640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f249409cf40 0x7f2494152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.692+0000 7f2499400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f249415d270 0x7f249414b3b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f248c002230 tx=0x7f248c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.692+0000 7f2490c00640 1 -- 172.21.15.114:0/2606613049 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c039030 con 0x7f249415d270 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.692+0000 7f2490c00640 1 -- 172.21.15.114:0/2606613049 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c03a040 con 0x7f249415d270 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.692+0000 7f2490c00640 1 -- 172.21.15.114:0/2606613049 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c009c10 con 0x7f249415d270 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.692+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249415daf0 con 0x7f249415d270 2024-03-06T07:31:59.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.692+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249414b8f0 con 0x7f249415d270 2024-03-06T07:31:59.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.694+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249404f330 con 0x7f249415d270 2024-03-06T07:31:59.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.694+0000 7f2490c00640 1 -- 172.21.15.114:0/2606613049 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f248c045020 con 0x7f249415d270 2024-03-06T07:31:59.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.695+0000 7f2490c00640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f247406b500 0x7f247406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.695+0000 7f2490c00640 1 -- 172.21.15.114:0/2606613049 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f248c0abbf0 con 0x7f249415d270 2024-03-06T07:31:59.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.695+0000 7f2498a00640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f247406b500 0x7f247406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.696+0000 7f2498a00640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f247406b500 0x7f247406d9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2488001eb0 tx=0x7f2488009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.699+0000 7f2490c00640 1 -- 172.21.15.114:0/2606613049 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f248c04c020 con 0x7f249415d270 2024-03-06T07:31:59.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.895+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2494071d30 con 0x7f249415d270 2024-03-06T07:31:59.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.897+0000 7f2490c00640 1 -- 172.21.15.114:0/2606613049 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v68) v1 ==== 104+0+11 (secure 0 0 0) 0x7f248c07aac0 con 0x7f249415d270 2024-03-06T07:31:59.900 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f247406b500 msgr2=0x7f247406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f247406b500 0x7f247406d9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2488001eb0 tx=0x7f2488009040 comp rx=0 tx=0).stop 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f249415d270 msgr2=0x7f249414b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f249415d270 0x7f249414b3b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f248c002230 tx=0x7f248c009750 comp rx=0 tx=0).stop 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 shutdown_connections 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f247406b500 0x7f247406d9c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f249415d270 0x7f249414b3b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f249409d320 0x7f2494152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 --2- 172.21.15.114:0/2606613049 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f249409cf40 0x7f2494152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 >> 172.21.15.114:0/2606613049 conn(0x7f24940b1940 msgr2=0x7f2494070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 shutdown_connections 2024-03-06T07:31:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:31:59.900+0000 7f249b400640 1 -- 172.21.15.114:0/2606613049 wait complete. 2024-03-06T07:31:59.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:00.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.128+0000 7febc8200640 1 Processor -- start 2024-03-06T07:32:00.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.128+0000 7febc8200640 1 -- start start 2024-03-06T07:32:00.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.130+0000 7febc8200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 0x7febc015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.130+0000 7febc8200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc006cba0 con 0x7febc006d960 2024-03-06T07:32:00.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.130+0000 7febc8200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc006ce80 con 0x7febc006e2f0 2024-03-06T07:32:00.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.130+0000 7febc8200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc015f700 con 0x7febc00b2c40 2024-03-06T07:32:00.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.130+0000 7febc4e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7febc006d960 0x7febc00b3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44270/0 (socket says 172.21.15.114:44270) 2024-03-06T07:32:00.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.130+0000 7febc4e00640 1 -- 172.21.15.114:0/1720623854 learned_addr learned my addr 172.21.15.114:0/1720623854 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febc5800640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 0x7febc015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830241275 0 0) 0x7febc006cba0 con 0x7febc006d960 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba8003520 con 0x7febc006d960 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254805771 0 0) 0x7feba8003520 con 0x7febc006d960 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 msgr2=0x7febc015f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 0x7febc015f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 >> v1:172.21.15.177:6789/0 conn(0x7febc006e2f0 legacy=0x7febc00cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc006d160 con 0x7febc006d960 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7febac001670 con 0x7febc006d960 2024-03-06T07:32:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 >> v1:172.21.15.114:6789/0 conn(0x7febc006d960 legacy=0x7febc00b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feba80039e0 0x7feba8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7feba8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 0x7feba80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba800ba70 con 0x7feba8004800 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.131+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba800bbf0 con 0x7feba80039e0 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba800bdf0 con 0x7febc00b2c40 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc4e00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7feba8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc6200640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 0x7feba80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc5800640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feba80039e0 0x7feba8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc4e00640 1 -- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 msgr2=0x7feba80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc4e00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc00b2c40 0x7feba80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc4e00640 1 -- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feba80039e0 msgr2=0x7feba8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc4e00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feba80039e0 0x7feba8003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.132+0000 7febc4e00640 1 -- 172.21.15.114:0/1720623854 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba8003520 con 0x7feba8004800 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.133+0000 7febc4e00640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7feba8004c80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7febac001eb0 tx=0x7feba8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8258fe873836361 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.133+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac007d30 con 0x7feba8004800 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febac030070 con 0x7feba8004800 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febbfe00640 1 -- 172.21.15.114:0/1720623854 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac005d30 con 0x7feba8004800 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febc8200640 1 -- 172.21.15.114:0/1720623854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 msgr2=0x7feba8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febc8200640 1 --2- 172.21.15.114:0/1720623854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7feba8004c80 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7febac001eb0 tx=0x7feba8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febc6200640 1 -- 172.21.15.114:0/1720623854 reap_dead start 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febc8200640 1 -- 172.21.15.114:0/1720623854 shutdown_connections 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febc8200640 1 -- 172.21.15.114:0/1720623854 >> 172.21.15.114:0/1720623854 conn(0x7febc00b1940 msgr2=0x7febc00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febc8200640 1 -- 172.21.15.114:0/1720623854 shutdown_connections 2024-03-06T07:32:00.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.134+0000 7febc8200640 1 -- 172.21.15.114:0/1720623854 wait complete. 2024-03-06T07:32:00.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.137+0000 7febc8200640 1 Processor -- start 2024-03-06T07:32:00.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.137+0000 7febc8200640 1 -- start start 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc8200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7febc0160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc8200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc0146cb0 0x7febc01f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc8200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febc01f4200 0x7febc01f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc8200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc015f1c0 con 0x7feba8004800 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc8200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc006d4d0 con 0x7febc01f4200 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc8200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc015f340 con 0x7febc0146cb0 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7febc0160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7febc0160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34072/0 (socket says 172.21.15.114:34072) 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.139+0000 7febc5800640 1 -- 172.21.15.114:0/95558479 learned_addr learned my addr 172.21.15.114:0/95558479 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc6200640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febc01f4200 0x7febc01f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc4e00640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc0146cb0 0x7febc01f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc5800640 1 -- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc0146cb0 msgr2=0x7febc01f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc5800640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc0146cb0 0x7febc01f3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc5800640 1 -- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febc01f4200 msgr2=0x7febc01f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc5800640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febc01f4200 0x7febc01f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc5800640 1 -- 172.21.15.114:0/95558479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc00acd20 con 0x7feba8004800 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc5800640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7febc0160cb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7febb4010860 tx=0x7febb4010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febbd600640 1 -- 172.21.15.114:0/95558479 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb400f040 con 0x7feba8004800 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febbd600640 1 -- 172.21.15.114:0/95558479 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb400eb60 con 0x7feba8004800 2024-03-06T07:32:00.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc01f4680 con 0x7feba8004800 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febbd600640 1 -- 172.21.15.114:0/95558479 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb401c400 con 0x7feba8004800 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.140+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc00acfc0 con 0x7feba8004800 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.142+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc004f330 con 0x7feba8004800 2024-03-06T07:32:00.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.143+0000 7febbd600640 1 -- 172.21.15.114:0/95558479 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febb401c5a0 con 0x7feba8004800 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.144+0000 7febbd600640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feba006b500 0x7feba006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.145+0000 7febbd600640 1 -- 172.21.15.114:0/95558479 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7febb408ab80 con 0x7feba8004800 2024-03-06T07:32:00.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.145+0000 7febc4e00640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feba006b500 0x7feba006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.145+0000 7febc4e00640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feba006b500 0x7feba006d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7febac034000 tx=0x7febac009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.147+0000 7febbd600640 1 -- 172.21.15.114:0/95558479 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febb4018030 con 0x7feba8004800 2024-03-06T07:32:00.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.344+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7febc0162bb0 con 0x7feba8004800 2024-03-06T07:32:00.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.345+0000 7febbd600640 1 -- 172.21.15.114:0/95558479 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v68) v1 ==== 100+0+11 (secure 0 0 0) 0x7febb4059a30 con 0x7feba8004800 2024-03-06T07:32:00.347 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:32:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feba006b500 msgr2=0x7feba006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feba006b500 0x7feba006d9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7febac034000 tx=0x7febac009030 comp rx=0 tx=0).stop 2024-03-06T07:32:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 msgr2=0x7febc0160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7febc0160cb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7febb4010860 tx=0x7febb4010d20 comp rx=0 tx=0).stop 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 shutdown_connections 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7febc01f4200 0x7febc01f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7febc0146cb0 0x7febc01f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feba006b500 0x7feba006d9c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 --2- 172.21.15.114:0/95558479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feba8004800 0x7febc0160cb0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 >> 172.21.15.114:0/95558479 conn(0x7febc00b1940 msgr2=0x7febc015d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 shutdown_connections 2024-03-06T07:32:00.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.347+0000 7febc8200640 1 -- 172.21.15.114:0/95558479 wait complete. 2024-03-06T07:32:00.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.575+0000 7f1605400640 1 Processor -- start 2024-03-06T07:32:00.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.575+0000 7f1605400640 1 -- start start 2024-03-06T07:32:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.577+0000 7f1605400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16001615c0 0x7f16000d4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.577+0000 7f1605400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16000d4980 con 0x7f16000d0600 2024-03-06T07:32:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.577+0000 7f1605400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16000d2a50 con 0x7f16000d0a80 2024-03-06T07:32:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.577+0000 7f1605400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16000d4c00 con 0x7f16001615c0 2024-03-06T07:32:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.577+0000 7f15ff400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f16000d0600 0x7f16000d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44286/0 (socket says 172.21.15.114:44286) 2024-03-06T07:32:00.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.577+0000 7f15ff400640 1 -- 172.21.15.114:0/3917840133 learned_addr learned my addr 172.21.15.114:0/3917840133 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15ffe00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16001615c0 0x7f16000d4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869331415 0 0) 0x7f15ec001110 con 0x7f16000d0600 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e4003520 con 0x7f16000d0600 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 849837648 0 0) 0x7f15e4003520 con 0x7f16000d0600 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16001615c0 msgr2=0x7f16000d4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16001615c0 0x7f16000d4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 >> v1:172.21.15.177:6789/0 conn(0x7f16000d0a80 legacy=0x7f16000d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160009d5e0 con 0x7f16000d0600 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f15ec001860 con 0x7f16000d0600 2024-03-06T07:32:00.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 >> v1:172.21.15.114:6789/0 conn(0x7f16000d0600 legacy=0x7f16000d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15e40039e0 0x7f15e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16001615c0 0x7f15e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4004d40 0x7f15e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16000d4c00 con 0x7f15e4004d40 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e400bba0 con 0x7f15e40039e0 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.578+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e400bda0 con 0x7f16001615c0 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f1604a00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4004d40 0x7f15e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f15ffe00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15e40039e0 0x7f15e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f1604a00640 1 -- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16001615c0 msgr2=0x7f15e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f1604a00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16001615c0 0x7f15e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f1604a00640 1 -- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15e40039e0 msgr2=0x7f15e4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f1604a00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15e40039e0 0x7f15e4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f1604a00640 1 -- 172.21.15.114:0/3917840133 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e4003520 con 0x7f15e4004d40 2024-03-06T07:32:00.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.579+0000 7f15ffe00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15e40039e0 0x7f15e4003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:00.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.580+0000 7f1604a00640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4004d40 0x7f15e40051c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f15f8001e50 tx=0x7f15e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8a416ea975d8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.580+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8001a70 con 0x7f15e4004d40 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.580+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8003b70 con 0x7f15e4004d40 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.580+0000 7f15fea00640 1 -- 172.21.15.114:0/3917840133 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80050d0 con 0x7f15e4004d40 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.581+0000 7f1605400640 1 -- 172.21.15.114:0/3917840133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4004d40 msgr2=0x7f15e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.581+0000 7f1605400640 1 --2- 172.21.15.114:0/3917840133 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4004d40 0x7f15e40051c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f15f8001e50 tx=0x7f15e4005830 comp rx=0 tx=0).stop 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.581+0000 7f1604a00640 1 -- 172.21.15.114:0/3917840133 reap_dead start 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.581+0000 7f1605400640 1 -- 172.21.15.114:0/3917840133 shutdown_connections 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.581+0000 7f1605400640 1 -- 172.21.15.114:0/3917840133 >> 172.21.15.114:0/3917840133 conn(0x7f160009d080 msgr2=0x7f16000d3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.581+0000 7f1605400640 1 -- 172.21.15.114:0/3917840133 shutdown_connections 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.581+0000 7f1605400640 1 -- 172.21.15.114:0/3917840133 wait complete. 2024-03-06T07:32:00.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.582+0000 7f1605400640 1 Processor -- start 2024-03-06T07:32:00.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.583+0000 7f1605400640 1 -- start start 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.584+0000 7f1605400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15e4004d40 0x7f160015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.584+0000 7f1605400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16000b86b0 0x7f1600160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.584+0000 7f1605400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16000ca5d0 0x7f1600160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.584+0000 7f1605400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16000b3a60 con 0x7f16000ca5d0 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.584+0000 7f1605400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16000b40b0 con 0x7f16000b86b0 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.584+0000 7f1605400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16000b3d40 con 0x7f15e4004d40 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16000ca5d0 0x7f1600160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16000ca5d0 0x7f1600160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34096/0 (socket says 172.21.15.114:34096) 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 -- 172.21.15.114:0/1438850553 learned_addr learned my addr 172.21.15.114:0/1438850553 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f15ffe00640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15e4004d40 0x7f160015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f15ff400640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16000b86b0 0x7f1600160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 -- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15e4004d40 msgr2=0x7f160015af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15e4004d40 0x7f160015af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 -- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16000b86b0 msgr2=0x7f1600160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16000b86b0 0x7f1600160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.585+0000 7f1604a00640 1 -- 172.21.15.114:0/1438850553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160015b4c0 con 0x7f16000ca5d0 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.586+0000 7f1604a00640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16000ca5d0 0x7f1600160690 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f15f802c060 tx=0x7f15f80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.586+0000 7f15f7e00640 1 -- 172.21.15.114:0/1438850553 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8007db0 con 0x7f16000ca5d0 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.586+0000 7f15f7e00640 1 -- 172.21.15.114:0/1438850553 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8003970 con 0x7f16000ca5d0 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.586+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16000b8c00 con 0x7f16000ca5d0 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.586+0000 7f15f7e00640 1 -- 172.21.15.114:0/1438850553 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8006d70 con 0x7f16000ca5d0 2024-03-06T07:32:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.586+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16001f7a30 con 0x7f16000ca5d0 2024-03-06T07:32:00.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.589+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160004f330 con 0x7f16000ca5d0 2024-03-06T07:32:00.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.589+0000 7f15f7e00640 1 -- 172.21.15.114:0/1438850553 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15f8007030 con 0x7f16000ca5d0 2024-03-06T07:32:00.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.590+0000 7f15f7e00640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e006b4d0 0x7f15e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.590+0000 7f15f7e00640 1 -- 172.21.15.114:0/1438850553 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f15f80ab350 con 0x7f16000ca5d0 2024-03-06T07:32:00.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.590+0000 7f15ffe00640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e006b4d0 0x7f15e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.591+0000 7f15ffe00640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e006b4d0 0x7f15e006d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f15f0006f00 tx=0x7f15f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.595+0000 7f15f7e00640 1 -- 172.21.15.114:0/1438850553 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15f8033020 con 0x7f16000ca5d0 2024-03-06T07:32:00.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.832+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16000cebc0 con 0x7f16000ca5d0 2024-03-06T07:32:00.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.833+0000 7f15f7e00640 1 -- 172.21.15.114:0/1438850553 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f15f8079b20 con 0x7f16000ca5d0 2024-03-06T07:32:00.835 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:00.836 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:32:00.836 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e006b4d0 msgr2=0x7f15e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e006b4d0 0x7f15e006d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f15f0006f00 tx=0x7f15f0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16000ca5d0 msgr2=0x7f1600160690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16000ca5d0 0x7f1600160690 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f15f802c060 tx=0x7f15f80014a0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 shutdown_connections 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f15e006b4d0 0x7f15e006d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16000ca5d0 0x7f1600160690 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16000b86b0 0x7f1600160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 --2- 172.21.15.114:0/1438850553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15e4004d40 0x7f160015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 >> 172.21.15.114:0/1438850553 conn(0x7f160009d080 msgr2=0x7f16000d1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 shutdown_connections 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:00.835+0000 7f1605400640 1 -- 172.21.15.114:0/1438850553 wait complete. 2024-03-06T07:32:00.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:01.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.054+0000 7fbaac000640 1 Processor -- start 2024-03-06T07:32:01.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.054+0000 7fbaac000640 1 -- start start 2024-03-06T07:32:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaac000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbaa415f9d0 0x7fbaa415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaac000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbaa4070b10 0x7fbaa4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaac000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa406cef0 con 0x7fbaa4144c00 2024-03-06T07:32:01.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaac000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa406d1d0 con 0x7fbaa415f9d0 2024-03-06T07:32:01.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaac000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa406d4b0 con 0x7fbaa4070b10 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaa8c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbaa415f9d0 0x7fbaa415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaa8c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbaa415f9d0 0x7fbaa415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34110/0 (socket says 172.21.15.114:34110) 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.056+0000 7fbaa8c00640 1 -- 172.21.15.114:0/3203185775 learned_addr learned my addr 172.21.15.114:0/3203185775 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.057+0000 7fbaaa000640 1 --2- 172.21.15.114:0/3203185775 >> v2:172.21.15.177:3300/0 conn(0x7fbaa4070b10 0x7fbaa4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.057+0000 7fbaa8c00640 1 -- 172.21.15.114:0/3203185775 >> v2:172.21.15.177:3300/0 conn(0x7fbaa4070b10 msgr2=0x7fbaa4070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.057+0000 7fbaa8c00640 1 --2- 172.21.15.114:0/3203185775 >> v2:172.21.15.177:3300/0 conn(0x7fbaa4070b10 0x7fbaa4070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.057+0000 7fbaa8c00640 1 -- 172.21.15.114:0/3203185775 >> v1:172.21.15.177:6789/0 conn(0x7fbaa4144c00 legacy=0x7fbaa406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.057+0000 7fbaa8c00640 1 -- 172.21.15.114:0/3203185775 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa406d790 con 0x7fbaa415f9d0 2024-03-06T07:32:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.058+0000 7fbaa8c00640 1 --2- 172.21.15.114:0/3203185775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa415f9d0 0x7fbaa415fe60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbaa406e500 tx=0x7fba9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2113a3601c47fefa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.059+0000 7fbaa3e00640 1 -- 172.21.15.114:0/3203185775 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90035020 con 0x7fbaa415f9d0 2024-03-06T07:32:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.059+0000 7fbaa3e00640 1 -- 172.21.15.114:0/3203185775 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba90031870 con 0x7fbaa415f9d0 2024-03-06T07:32:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.059+0000 7fbaa3e00640 1 -- 172.21.15.114:0/3203185775 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90031c40 con 0x7fbaa415f9d0 2024-03-06T07:32:01.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.059+0000 7fbaac000640 1 -- 172.21.15.114:0/3203185775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa415f9d0 msgr2=0x7fbaa415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.059+0000 7fbaac000640 1 --2- 172.21.15.114:0/3203185775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa415f9d0 0x7fbaa415fe60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbaa406e500 tx=0x7fba9002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:01.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.060+0000 7fbaac000640 1 -- 172.21.15.114:0/3203185775 shutdown_connections 2024-03-06T07:32:01.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.060+0000 7fbaac000640 1 --2- 172.21.15.114:0/3203185775 >> v2:172.21.15.177:3300/0 conn(0x7fbaa4070b10 0x7fbaa4070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.060+0000 7fbaac000640 1 --2- 172.21.15.114:0/3203185775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa415f9d0 0x7fbaa415fe60 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.060+0000 7fbaac000640 1 -- 172.21.15.114:0/3203185775 >> 172.21.15.114:0/3203185775 conn(0x7fbaa40b1730 msgr2=0x7fbaa40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.060+0000 7fbaac000640 1 -- 172.21.15.114:0/3203185775 shutdown_connections 2024-03-06T07:32:01.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.060+0000 7fbaac000640 1 -- 172.21.15.114:0/3203185775 wait complete. 2024-03-06T07:32:01.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.061+0000 7fbaac000640 1 Processor -- start 2024-03-06T07:32:01.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.062+0000 7fbaac000640 1 -- start start 2024-03-06T07:32:01.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaac000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa4070b10 0x7fbaa413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaac000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa4144c00 0x7fbaa413ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaac000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa415f9d0 0x7fbaa415d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaac000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa406ccd0 con 0x7fbaa4070b10 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaac000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa406cb50 con 0x7fbaa415f9d0 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaac000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa406df90 con 0x7fbaa4144c00 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa4070b10 0x7fbaa413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa4070b10 0x7fbaa413c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34112/0 (socket says 172.21.15.114:34112) 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 -- 172.21.15.114:0/1126526190 learned_addr learned my addr 172.21.15.114:0/1126526190 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa8c00640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa4144c00 0x7fbaa413ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 -- 172.21.15.114:0/1126526190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa4144c00 msgr2=0x7fbaa413ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa4144c00 0x7fbaa413ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 -- 172.21.15.114:0/1126526190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa415f9d0 msgr2=0x7fbaa415d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa415f9d0 0x7fbaa415d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.063+0000 7fbaa9600640 1 -- 172.21.15.114:0/1126526190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa41400e0 con 0x7fbaa4070b10 2024-03-06T07:32:01.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.064+0000 7fbaa9600640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa4070b10 0x7fbaa413c2e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c460 tx=0x7fba9800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.064+0000 7fbaa1600640 1 -- 172.21.15.114:0/1126526190 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98016020 con 0x7fbaa4070b10 2024-03-06T07:32:01.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.064+0000 7fbaa1600640 1 -- 172.21.15.114:0/1126526190 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba98012870 con 0x7fbaa4070b10 2024-03-06T07:32:01.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.064+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa413fb20 con 0x7fbaa4070b10 2024-03-06T07:32:01.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.064+0000 7fbaa1600640 1 -- 172.21.15.114:0/1126526190 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98012c20 con 0x7fbaa4070b10 2024-03-06T07:32:01.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.064+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa4140b60 con 0x7fbaa4070b10 2024-03-06T07:32:01.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.066+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa404f330 con 0x7fbaa4070b10 2024-03-06T07:32:01.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.067+0000 7fbaa1600640 1 -- 172.21.15.114:0/1126526190 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba98019020 con 0x7fbaa4070b10 2024-03-06T07:32:01.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.068+0000 7fbaa1600640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba8406b4d0 0x7fba8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.068+0000 7fbaa1600640 1 -- 172.21.15.114:0/1126526190 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba98089d30 con 0x7fbaa4070b10 2024-03-06T07:32:01.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.068+0000 7fbaa8c00640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba8406b4d0 0x7fba8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.069+0000 7fbaa8c00640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba8406b4d0 0x7fba8406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fba90036000 tx=0x7fba90008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.075+0000 7fbaa1600640 1 -- 172.21.15.114:0/1126526190 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba98020070 con 0x7fbaa4070b10 2024-03-06T07:32:01.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.282+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaa4072890 con 0x7fbaa4070b10 2024-03-06T07:32:01.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.284+0000 7fbaa1600640 1 -- 172.21.15.114:0/1126526190 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16814 (secure 0 0 0) 0x7fba9800e280 con 0x7fbaa4070b10 2024-03-06T07:32:01.287 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:01.287 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":68,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:37.032494+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:18.351057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:19.518313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.20","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba8406b4d0 msgr2=0x7fba8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba8406b4d0 0x7fba8406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fba90036000 tx=0x7fba90008030 comp rx=0 tx=0).stop 2024-03-06T07:32:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa4070b10 msgr2=0x7fbaa413c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa4070b10 0x7fbaa413c2e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c460 tx=0x7fba9800c920 comp rx=0 tx=0).stop 2024-03-06T07:32:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 shutdown_connections 2024-03-06T07:32:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbaa415f9d0 0x7fbaa415d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbaa4144c00 0x7fbaa413ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba8406b4d0 0x7fba8406d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 --2- 172.21.15.114:0/1126526190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaa4070b10 0x7fbaa413c2e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.287+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 >> 172.21.15.114:0/1126526190 conn(0x7fbaa40b1730 msgr2=0x7fbaa4145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.288+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 shutdown_connections 2024-03-06T07:32:01.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.288+0000 7fbaac000640 1 -- 172.21.15.114:0/1126526190 wait complete. 2024-03-06T07:32:01.309 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:01.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:01.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.554+0000 7fef47200640 1 Processor -- start 2024-03-06T07:32:01.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.554+0000 7fef47200640 1 -- start start 2024-03-06T07:32:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.555+0000 7fef47200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b40c0 0x7fef480b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.555+0000 7fef47200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef481620c0 0x7fef480b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.555+0000 7fef47200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef48161900 con 0x7fef480ae670 2024-03-06T07:32:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.555+0000 7fef47200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef480b49e0 con 0x7fef481620c0 2024-03-06T07:32:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.555+0000 7fef47200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4815ebd0 con 0x7fef480b40c0 2024-03-06T07:32:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef46800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef481620c0 0x7fef480b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef46800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef481620c0 0x7fef480b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34116/0 (socket says 172.21.15.114:34116) 2024-03-06T07:32:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef46800640 1 -- 172.21.15.114:0/2667001332 learned_addr learned my addr 172.21.15.114:0/2667001332 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef45e00640 1 --2- 172.21.15.114:0/2667001332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b40c0 0x7fef480b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef46800640 1 -- 172.21.15.114:0/2667001332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b40c0 msgr2=0x7fef480b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef46800640 1 --2- 172.21.15.114:0/2667001332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b40c0 0x7fef480b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef46800640 1 -- 172.21.15.114:0/2667001332 >> v1:172.21.15.177:6789/0 conn(0x7fef480ae670 legacy=0x7fef4815e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:01.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.556+0000 7fef46800640 1 -- 172.21.15.114:0/2667001332 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4815eeb0 con 0x7fef481620c0 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.557+0000 7fef46800640 1 --2- 172.21.15.114:0/2667001332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef481620c0 0x7fef480b28e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fef40009e50 tx=0x7fef4002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e77c37f37decb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef45400640 1 -- 172.21.15.114:0/2667001332 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4002ca70 con 0x7fef481620c0 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef45400640 1 -- 172.21.15.114:0/2667001332 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4002cc10 con 0x7fef481620c0 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef45400640 1 -- 172.21.15.114:0/2667001332 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef400343f0 con 0x7fef481620c0 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 -- 172.21.15.114:0/2667001332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef481620c0 msgr2=0x7fef480b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 --2- 172.21.15.114:0/2667001332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef481620c0 0x7fef480b28e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fef40009e50 tx=0x7fef4002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 -- 172.21.15.114:0/2667001332 shutdown_connections 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 --2- 172.21.15.114:0/2667001332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef481620c0 0x7fef480b28e0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 --2- 172.21.15.114:0/2667001332 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480b40c0 0x7fef480b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 -- 172.21.15.114:0/2667001332 >> 172.21.15.114:0/2667001332 conn(0x7fef4806afc0 msgr2=0x7fef4806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 -- 172.21.15.114:0/2667001332 shutdown_connections 2024-03-06T07:32:01.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.558+0000 7fef47200640 1 -- 172.21.15.114:0/2667001332 wait complete. 2024-03-06T07:32:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.561+0000 7fef47200640 1 Processor -- start 2024-03-06T07:32:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.561+0000 7fef47200640 1 -- start start 2024-03-06T07:32:01.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef47200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480ae670 0x7fef4814a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef47200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef480b40c0 0x7fef48159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef47200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef481620c0 0x7fef4815a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef47200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef48158660 con 0x7fef480b40c0 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef47200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef48161ba0 con 0x7fef481620c0 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef47200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef481587e0 con 0x7fef480ae670 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef3fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef480b40c0 0x7fef48159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef3fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef480b40c0 0x7fef48159b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34126/0 (socket says 172.21.15.114:34126) 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.563+0000 7fef3fe00640 1 -- 172.21.15.114:0/2069693061 learned_addr learned my addr 172.21.15.114:0/2069693061 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef45e00640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480ae670 0x7fef4814a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef46800640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef481620c0 0x7fef4815a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3fe00640 1 -- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480ae670 msgr2=0x7fef4814a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3fe00640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480ae670 0x7fef4814a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3fe00640 1 -- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef481620c0 msgr2=0x7fef4815a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3fe00640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef481620c0 0x7fef4815a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3fe00640 1 -- 172.21.15.114:0/2069693061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4815b7c0 con 0x7fef480b40c0 2024-03-06T07:32:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3fe00640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef480b40c0 0x7fef48159b80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b420 tx=0x7fef3000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3e000640 1 -- 172.21.15.114:0/2069693061 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30011830 con 0x7fef480b40c0 2024-03-06T07:32:01.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3e000640 1 -- 172.21.15.114:0/2069693061 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef30016070 con 0x7fef480b40c0 2024-03-06T07:32:01.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef3e000640 1 -- 172.21.15.114:0/2069693061 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30011c10 con 0x7fef480b40c0 2024-03-06T07:32:01.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4815a720 con 0x7fef480b40c0 2024-03-06T07:32:01.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.564+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4815ace0 con 0x7fef480b40c0 2024-03-06T07:32:01.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.565+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4804f330 con 0x7fef480b40c0 2024-03-06T07:32:01.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.566+0000 7fef3e000640 1 -- 172.21.15.114:0/2069693061 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef3001b460 con 0x7fef480b40c0 2024-03-06T07:32:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.567+0000 7fef3e000640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2406b4d0 0x7fef2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.567+0000 7fef3e000640 1 -- 172.21.15.114:0/2069693061 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fef30089c60 con 0x7fef480b40c0 2024-03-06T07:32:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.567+0000 7fef45e00640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2406b4d0 0x7fef2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.568+0000 7fef45e00640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2406b4d0 0x7fef2406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fef3800b7c0 tx=0x7fef38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.572+0000 7fef3e000640 1 -- 172.21.15.114:0/2069693061 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef30020020 con 0x7fef480b40c0 2024-03-06T07:32:01.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.806+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef480adf40 con 0x7fef480b40c0 2024-03-06T07:32:01.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.807+0000 7fef3e000640 1 -- 172.21.15.114:0/2069693061 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fef30058b10 con 0x7fef480b40c0 2024-03-06T07:32:01.810 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:01.810 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:32:01.810 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.809+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2406b4d0 msgr2=0x7fef2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.809+0000 7fef47200640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2406b4d0 0x7fef2406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fef3800b7c0 tx=0x7fef38008040 comp rx=0 tx=0).stop 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef480b40c0 msgr2=0x7fef48159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef480b40c0 0x7fef48159b80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fef3000b420 tx=0x7fef3000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 shutdown_connections 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef481620c0 0x7fef4815a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fef2406b4d0 0x7fef2406d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef480b40c0 0x7fef48159b80 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 --2- 172.21.15.114:0/2069693061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fef480ae670 0x7fef4814a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 >> 172.21.15.114:0/2069693061 conn(0x7fef4806afc0 msgr2=0x7fef48155370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 shutdown_connections 2024-03-06T07:32:01.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:01.810+0000 7fef47200640 1 -- 172.21.15.114:0/2069693061 wait complete. 2024-03-06T07:32:01.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:02.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.031+0000 7fe669a00640 1 Processor -- start 2024-03-06T07:32:02.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.031+0000 7fe669a00640 1 -- start start 2024-03-06T07:32:02.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.033+0000 7fe669a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 0x7fe66415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.033+0000 7fe669a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe664071d30 con 0x7fe66406e4d0 2024-03-06T07:32:02.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.033+0000 7fe669a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe664071490 con 0x7fe664070d20 2024-03-06T07:32:02.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.033+0000 7fe669a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe664071770 con 0x7fe6640af1c0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.033+0000 7fe663400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe66406e4d0 0x7fe6640b35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44290/0 (socket says 172.21.15.114:44290) 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.033+0000 7fe663400640 1 -- 172.21.15.114:0/3482111800 learned_addr learned my addr 172.21.15.114:0/3482111800 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.033+0000 7fe662a00640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 0x7fe66415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859041105 0 0) 0x7fe65c001490 con 0x7fe66406e4d0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe648003520 con 0x7fe66406e4d0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026297514 0 0) 0x7fe648003520 con 0x7fe66406e4d0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 msgr2=0x7fe66415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 0x7fe66415eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 >> v1:172.21.15.177:6789/0 conn(0x7fe664070d20 legacy=0x7fe66406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe664071a50 con 0x7fe66406e4d0 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fe65c001bc0 con 0x7fe66406e4d0 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 >> v1:172.21.15.114:6789/0 conn(0x7fe66406e4d0 legacy=0x7fe6640b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 0x7fe6480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe648003f60 0x7fe648004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe648004d40 0x7fe6480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe664071770 con 0x7fe648004d40 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64800baa0 con 0x7fe648003f60 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.034+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64800bca0 con 0x7fe6640af1c0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe663400640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe648004d40 0x7fe6480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe662000640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe648003f60 0x7fe648004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe662a00640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 0x7fe6480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe663400640 1 -- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 msgr2=0x7fe6480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe663400640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640af1c0 0x7fe6480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe663400640 1 -- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe648003f60 msgr2=0x7fe648004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe663400640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe648003f60 0x7fe648004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.035+0000 7fe663400640 1 -- 172.21.15.114:0/3482111800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648003520 con 0x7fe648004d40 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.036+0000 7fe663400640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe648004d40 0x7fe6480051c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00aa50 tx=0x7fe648005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d46a6cd9de26c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.036+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c038070 con 0x7fe648004d40 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.036+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c033650 con 0x7fe648004d40 2024-03-06T07:32:02.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.036+0000 7fe661600640 1 -- 172.21.15.114:0/3482111800 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00a670 con 0x7fe648004d40 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.037+0000 7fe669a00640 1 -- 172.21.15.114:0/3482111800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe648004d40 msgr2=0x7fe6480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.037+0000 7fe669a00640 1 --2- 172.21.15.114:0/3482111800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe648004d40 0x7fe6480051c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00aa50 tx=0x7fe648005830 comp rx=0 tx=0).stop 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.037+0000 7fe663400640 1 -- 172.21.15.114:0/3482111800 reap_dead start 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.037+0000 7fe669a00640 1 -- 172.21.15.114:0/3482111800 shutdown_connections 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.037+0000 7fe669a00640 1 -- 172.21.15.114:0/3482111800 >> 172.21.15.114:0/3482111800 conn(0x7fe6640b1730 msgr2=0x7fe6640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.037+0000 7fe669a00640 1 -- 172.21.15.114:0/3482111800 shutdown_connections 2024-03-06T07:32:02.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.037+0000 7fe669a00640 1 -- 172.21.15.114:0/3482111800 wait complete. 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.039+0000 7fe669a00640 1 Processor -- start 2024-03-06T07:32:02.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.039+0000 7fe669a00640 1 -- start start 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe669a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe664144bb0 0x7fe66415a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe669a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640cc430 0x7fe664147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe669a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe664147990 0x7fe6641453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe669a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66406d370 con 0x7fe664147990 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe669a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640711a0 con 0x7fe664144bb0 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe669a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640b3ce0 con 0x7fe6640cc430 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe663400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe664147990 0x7fe6641453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.040+0000 7fe663400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe664147990 0x7fe6641453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34142/0 (socket says 172.21.15.114:34142) 2024-03-06T07:32:02.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe663400640 1 -- 172.21.15.114:0/2151937284 learned_addr learned my addr 172.21.15.114:0/2151937284 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe663400640 1 -- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640cc430 msgr2=0x7fe664147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe662000640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640cc430 0x7fe664147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe662a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe664144bb0 0x7fe66415a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe663400640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640cc430 0x7fe664147450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe663400640 1 -- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe664144bb0 msgr2=0x7fe66415a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe663400640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe664144bb0 0x7fe66415a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe663400640 1 -- 172.21.15.114:0/2151937284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6640cc9a0 con 0x7fe664147990 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe662000640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640cc430 0x7fe664147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe662a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe664144bb0 0x7fe66415a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe663400640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe664147990 0x7fe6641453c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00a0b0 tx=0x7fe65c001f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.041+0000 7fe65aa00640 1 -- 172.21.15.114:0/2151937284 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c038040 con 0x7fe664147990 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.042+0000 7fe65aa00640 1 -- 172.21.15.114:0/2151937284 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c042070 con 0x7fe664147990 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.042+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe664159bc0 con 0x7fe664147990 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.042+0000 7fe65aa00640 1 -- 172.21.15.114:0/2151937284 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00a670 con 0x7fe664147990 2024-03-06T07:32:02.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.042+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66415a180 con 0x7fe664147990 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.043+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66404f330 con 0x7fe664147990 2024-03-06T07:32:02.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.044+0000 7fe65aa00640 1 -- 172.21.15.114:0/2151937284 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe65c033650 con 0x7fe664147990 2024-03-06T07:32:02.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.045+0000 7fe65aa00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe64406b500 0x7fe64406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.045+0000 7fe65aa00640 1 -- 172.21.15.114:0/2151937284 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe65c0abc90 con 0x7fe664147990 2024-03-06T07:32:02.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.045+0000 7fe662a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe64406b500 0x7fe64406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.046+0000 7fe662a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe64406b500 0x7fe64406d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe66415dd90 tx=0x7fe654008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.049+0000 7fe65aa00640 1 -- 172.21.15.114:0/2151937284 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe664071d30 con 0x7fe664147990 2024-03-06T07:32:02.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.285+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe66415a350 con 0x7fe664147990 2024-03-06T07:32:02.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.286+0000 7fe65aa00640 1 -- 172.21.15.114:0/2151937284 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fe65c07a9a0 con 0x7fe664147990 2024-03-06T07:32:02.291 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:02.291 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:20.504747+0000","modified":"2024-03-06T07:31:23.698546+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6354":{"gid":6354,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/3105808425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3105808425},{"type":"v1","addr":"172.21.15.114:6837","nonce":3105808425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6354,"qdb_cluster":[6354]},"id":5}]} 2024-03-06T07:32:02.292 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 44 2024-03-06T07:32:02.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe64406b500 msgr2=0x7fe64406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe64406b500 0x7fe64406d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe66415dd90 tx=0x7fe654008040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe664147990 msgr2=0x7fe6641453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe664147990 0x7fe6641453c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00a0b0 tx=0x7fe65c001f90 comp rx=0 tx=0).stop 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 shutdown_connections 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe64406b500 0x7fe64406d9c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe664147990 0x7fe6641453c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6640cc430 0x7fe664147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 --2- 172.21.15.114:0/2151937284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe664144bb0 0x7fe66415a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 >> 172.21.15.114:0/2151937284 conn(0x7fe6640b1730 msgr2=0x7fe6640cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 shutdown_connections 2024-03-06T07:32:02.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.289+0000 7fe669a00640 1 -- 172.21.15.114:0/2151937284 wait complete. 2024-03-06T07:32:02.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.506+0000 7fee19a00640 1 Processor -- start 2024-03-06T07:32:02.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.506+0000 7fee19a00640 1 -- start start 2024-03-06T07:32:02.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee19a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee1406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee19a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fee14072340 0x7fee1406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee19a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1415b090 con 0x7fee1415acb0 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee19a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee140b2a30 con 0x7fee14072340 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee19a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140b4360 con 0x7fee140b4680 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee13400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fee14072340 0x7fee1406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee13400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fee14072340 0x7fee1406ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34158/0 (socket says 172.21.15.114:34158) 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.509+0000 7fee13400640 1 -- 172.21.15.114:0/2352992692 learned_addr learned my addr 172.21.15.114:0/2352992692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.510+0000 7fee0be00640 1 --2- 172.21.15.114:0/2352992692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee1406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.510+0000 7fee13400640 1 -- 172.21.15.114:0/2352992692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 msgr2=0x7fee1406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.510+0000 7fee13400640 1 --2- 172.21.15.114:0/2352992692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee1406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.510+0000 7fee13400640 1 -- 172.21.15.114:0/2352992692 >> v1:172.21.15.177:6789/0 conn(0x7fee1415acb0 legacy=0x7fee140b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.510+0000 7fee13400640 1 -- 172.21.15.114:0/2352992692 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140b2cf0 con 0x7fee14072340 2024-03-06T07:32:02.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.511+0000 7fee13400640 1 --2- 172.21.15.114:0/2352992692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee1406ea30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009e50 tx=0x7fee0c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1f33b0b67a4ece server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.511+0000 7fee12000640 1 -- 172.21.15.114:0/2352992692 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c02cb00 con 0x7fee14072340 2024-03-06T07:32:02.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee12000640 1 -- 172.21.15.114:0/2352992692 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c02cca0 con 0x7fee14072340 2024-03-06T07:32:02.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee12000640 1 -- 172.21.15.114:0/2352992692 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c034610 con 0x7fee14072340 2024-03-06T07:32:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 -- 172.21.15.114:0/2352992692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 msgr2=0x7fee1406ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 --2- 172.21.15.114:0/2352992692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee1406ea30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009e50 tx=0x7fee0c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 -- 172.21.15.114:0/2352992692 shutdown_connections 2024-03-06T07:32:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 --2- 172.21.15.114:0/2352992692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee1406ea30 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 --2- 172.21.15.114:0/2352992692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee1406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 -- 172.21.15.114:0/2352992692 >> 172.21.15.114:0/2352992692 conn(0x7fee140b1730 msgr2=0x7fee140b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 -- 172.21.15.114:0/2352992692 shutdown_connections 2024-03-06T07:32:02.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.512+0000 7fee19a00640 1 -- 172.21.15.114:0/2352992692 wait complete. 2024-03-06T07:32:02.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.514+0000 7fee19a00640 1 Processor -- start 2024-03-06T07:32:02.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.514+0000 7fee19a00640 1 -- start start 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee19a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee14144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee19a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee14155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee19a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1415acb0 0x7fee1415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee19a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140b4010 con 0x7fee14072340 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee19a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee140b32b0 con 0x7fee1415acb0 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee19a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140ccfb0 con 0x7fee140b4680 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee12a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee14144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee12a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee14144e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34162/0 (socket says 172.21.15.114:34162) 2024-03-06T07:32:02.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.516+0000 7fee12a00640 1 -- 172.21.15.114:0/1804296005 learned_addr learned my addr 172.21.15.114:0/1804296005 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:02.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee0be00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee14155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee12a00640 1 -- 172.21.15.114:0/1804296005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 msgr2=0x7fee14155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee12a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee14155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee12a00640 1 -- 172.21.15.114:0/1804296005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1415acb0 msgr2=0x7fee1415e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee12a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1415acb0 0x7fee1415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee12a00640 1 -- 172.21.15.114:0/1804296005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee14147620 con 0x7fee14072340 2024-03-06T07:32:02.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee12a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee14144e20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c400 tx=0x7fee0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee0aa00640 1 -- 172.21.15.114:0/1804296005 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04017070 con 0x7fee14072340 2024-03-06T07:32:02.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee0aa00640 1 -- 172.21.15.114:0/1804296005 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee040052d0 con 0x7fee14072340 2024-03-06T07:32:02.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee0aa00640 1 -- 172.21.15.114:0/1804296005 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04003830 con 0x7fee14072340 2024-03-06T07:32:02.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee141478a0 con 0x7fee14072340 2024-03-06T07:32:02.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.517+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee14147e60 con 0x7fee14072340 2024-03-06T07:32:02.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.518+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee14148520 con 0x7fee14072340 2024-03-06T07:32:02.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.519+0000 7fee0aa00640 1 -- 172.21.15.114:0/1804296005 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee0401a070 con 0x7fee14072340 2024-03-06T07:32:02.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.520+0000 7fee0aa00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fedf006b4d0 0x7fedf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.520+0000 7fee0aa00640 1 -- 172.21.15.114:0/1804296005 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fee04089530 con 0x7fee14072340 2024-03-06T07:32:02.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.521+0000 7fee0be00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fedf006b4d0 0x7fedf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.521+0000 7fee0be00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fedf006b4d0 0x7fedf006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fee141f7670 tx=0x7fedfc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.525+0000 7fee0aa00640 1 -- 172.21.15.114:0/1804296005 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee14148520 con 0x7fee14072340 2024-03-06T07:32:02.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.706+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee1404f330 con 0x7fee14072340 2024-03-06T07:32:02.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.709+0000 7fee0aa00640 1 -- 172.21.15.114:0/1804296005 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16814 (secure 0 0 0) 0x7fee04087e20 con 0x7fee14072340 2024-03-06T07:32:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:02.713 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":68,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:31:37.032494+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:18.351057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:19.518313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.20","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fedf006b4d0 msgr2=0x7fedf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fedf006b4d0 0x7fedf006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fee141f7670 tx=0x7fedfc008040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 msgr2=0x7fee14144e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee14144e20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c400 tx=0x7fee0400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 shutdown_connections 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee1415acb0 0x7fee1415e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fee140b4680 0x7fee14155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fedf006b4d0 0x7fedf006d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 --2- 172.21.15.114:0/1804296005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee14072340 0x7fee14144e20 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 >> 172.21.15.114:0/1804296005 conn(0x7fee140b1730 msgr2=0x7fee14071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 shutdown_connections 2024-03-06T07:32:02.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.712+0000 7fee19a00640 1 -- 172.21.15.114:0/1804296005 wait complete. 2024-03-06T07:32:02.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:32:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.920+0000 7fa281800640 1 Processor -- start 2024-03-06T07:32:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.921+0000 7fa281800640 1 -- start start 2024-03-06T07:32:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.922+0000 7fa281800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa27c15e090 0x7fa27c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa281800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa27c0d2340 0x7fa27c0d2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa281800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27c0ce890 con 0x7fa27c0d1270 2024-03-06T07:32:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa281800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c15e470 con 0x7fa27c15e090 2024-03-06T07:32:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa281800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27c0d2180 con 0x7fa27c0d2340 2024-03-06T07:32:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa27be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa27c15e090 0x7fa27c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa273e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa27c0d1270 0x7fa27c0d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44300/0 (socket says 172.21.15.114:44300) 2024-03-06T07:32:02.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa273e00640 1 -- 172.21.15.114:0/2906622848 learned_addr learned my addr 172.21.15.114:0/2906622848 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:02.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa280e00640 1 --2- 172.21.15.114:0/2906622848 >> v2:172.21.15.177:3300/0 conn(0x7fa27c0d2340 0x7fa27c0d2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa27be00640 1 -- 172.21.15.114:0/2906622848 >> v2:172.21.15.177:3300/0 conn(0x7fa27c0d2340 msgr2=0x7fa27c0d2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa27be00640 1 --2- 172.21.15.114:0/2906622848 >> v2:172.21.15.177:3300/0 conn(0x7fa27c0d2340 0x7fa27c0d2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa27be00640 1 -- 172.21.15.114:0/2906622848 >> v1:172.21.15.114:6789/0 conn(0x7fa27c0d1270 legacy=0x7fa27c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa27be00640 1 -- 172.21.15.114:0/2906622848 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0ceb30 con 0x7fa27c15e090 2024-03-06T07:32:02.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.923+0000 7fa27b400640 1 -- 172.21.15.114:0/2906622848 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102481553 0 0) 0x7fa27c0ce890 con 0x7fa27c0d1270 2024-03-06T07:32:02.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.924+0000 7fa27be00640 1 --2- 172.21.15.114:0/2906622848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c15e090 0x7fa27c0cf8f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa27c08d180 tx=0x7fa26c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f33442a40b76c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa27b400640 1 -- 172.21.15.114:0/2906622848 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c035020 con 0x7fa27c15e090 2024-03-06T07:32:02.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa27b400640 1 -- 172.21.15.114:0/2906622848 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26c031830 con 0x7fa27c15e090 2024-03-06T07:32:02.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa27b400640 1 -- 172.21.15.114:0/2906622848 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c031c00 con 0x7fa27c15e090 2024-03-06T07:32:02.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa281800640 1 -- 172.21.15.114:0/2906622848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c15e090 msgr2=0x7fa27c0cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa281800640 1 --2- 172.21.15.114:0/2906622848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c15e090 0x7fa27c0cf8f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa27c08d180 tx=0x7fa26c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa281800640 1 -- 172.21.15.114:0/2906622848 shutdown_connections 2024-03-06T07:32:02.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa281800640 1 --2- 172.21.15.114:0/2906622848 >> v2:172.21.15.177:3300/0 conn(0x7fa27c0d2340 0x7fa27c0d2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa281800640 1 --2- 172.21.15.114:0/2906622848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c15e090 0x7fa27c0cf8f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.925+0000 7fa281800640 1 -- 172.21.15.114:0/2906622848 >> 172.21.15.114:0/2906622848 conn(0x7fa27c0d3d80 msgr2=0x7fa27c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.926+0000 7fa281800640 1 -- 172.21.15.114:0/2906622848 shutdown_connections 2024-03-06T07:32:02.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.926+0000 7fa281800640 1 -- 172.21.15.114:0/2906622848 wait complete. 2024-03-06T07:32:02.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.927+0000 7fa281800640 1 Processor -- start 2024-03-06T07:32:02.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.927+0000 7fa281800640 1 -- start start 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa281800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c0d1270 0x7fa27c0b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa281800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c0d2340 0x7fa27c0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa281800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa27c15e090 0x7fa27c160df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa281800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0cf460 con 0x7fa27c0d1270 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa281800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0d2cd0 con 0x7fa27c0d2340 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa281800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c0cf0f0 con 0x7fa27c15e090 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa27be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c0d1270 0x7fa27c0b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa27be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c0d1270 0x7fa27c0b8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34174/0 (socket says 172.21.15.114:34174) 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.928+0000 7fa27be00640 1 -- 172.21.15.114:0/4067270649 learned_addr learned my addr 172.21.15.114:0/4067270649 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa27be00640 1 -- 172.21.15.114:0/4067270649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa27c15e090 msgr2=0x7fa27c160df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa27be00640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa27c15e090 0x7fa27c160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa27be00640 1 -- 172.21.15.114:0/4067270649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c0d2340 msgr2=0x7fa27c0c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa27be00640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c0d2340 0x7fa27c0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa27be00640 1 -- 172.21.15.114:0/4067270649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c15f0c0 con 0x7fa27c0d1270 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa27be00640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c0d1270 0x7fa27c0b8e10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa26c000c00 tx=0x7fa26c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa278c00640 1 -- 172.21.15.114:0/4067270649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c040070 con 0x7fa27c0d1270 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa278c00640 1 -- 172.21.15.114:0/4067270649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26c039a20 con 0x7fa27c0d1270 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa278c00640 1 -- 172.21.15.114:0/4067270649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c035040 con 0x7fa27c0d1270 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c15f3a0 con 0x7fa27c0d1270 2024-03-06T07:32:02.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.929+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c0b9350 con 0x7fa27c0d1270 2024-03-06T07:32:02.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.931+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c04f330 con 0x7fa27c0d1270 2024-03-06T07:32:02.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.932+0000 7fa278c00640 1 -- 172.21.15.114:0/4067270649 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa26c04b060 con 0x7fa27c0d1270 2024-03-06T07:32:02.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.932+0000 7fa278c00640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa25806b4d0 0x7fa25806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.932+0000 7fa278c00640 1 -- 172.21.15.114:0/4067270649 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa26c0abbd0 con 0x7fa27c0d1270 2024-03-06T07:32:02.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.933+0000 7fa273e00640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa25806b4d0 0x7fa25806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.933+0000 7fa273e00640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa25806b4d0 0x7fa25806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa264007fd0 tx=0x7fa264007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:02.941+0000 7fa278c00640 1 -- 172.21.15.114:0/4067270649 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa26c07aaa0 con 0x7fa27c0d1270 2024-03-06T07:32:03.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.166+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa27c0d3b60 con 0x7fa27c0d1270 2024-03-06T07:32:03.169 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:32:03.167+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:03.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.169+0000 7fa278c00640 1 -- 172.21.15.114:0/4067270649 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa26c07a3a0 con 0x7fa27c0d1270 2024-03-06T07:32:03.172 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa25806b4d0 msgr2=0x7fa25806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa25806b4d0 0x7fa25806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa264007fd0 tx=0x7fa264007960 comp rx=0 tx=0).stop 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c0d1270 msgr2=0x7fa27c0b8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c0d1270 0x7fa27c0b8e10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa26c000c00 tx=0x7fa26c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 shutdown_connections 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa27c15e090 0x7fa27c160df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa27c0d2340 0x7fa27c0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa25806b4d0 0x7fa25806d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 --2- 172.21.15.114:0/4067270649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa27c0d1270 0x7fa27c0b8e10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.172+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 >> 172.21.15.114:0/4067270649 conn(0x7fa27c0d3d80 msgr2=0x7fa27c0b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.173+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 shutdown_connections 2024-03-06T07:32:03.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.173+0000 7fa281800640 1 -- 172.21.15.114:0/4067270649 wait complete. 2024-03-06T07:32:03.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:32:03.247 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:32:03.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.371+0000 7f2dfb200640 1 Processor -- start 2024-03-06T07:32:03.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.372+0000 7f2dfb200640 1 -- start start 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2dfb200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2df415be10 0x7f2df40ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2dfb200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2df40aef20 0x7f2df40ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2dfb200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df415c1f0 con 0x7f2df406d330 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2dfb200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df40cd200 con 0x7f2df40aef20 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2dfb200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df406cac0 con 0x7f2df415be10 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2df9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2df40aef20 0x7f2df40ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2df9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2df40aef20 0x7f2df40ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34194/0 (socket says 172.21.15.114:34194) 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2dfa800640 1 --1- 172.21.15.114:0/795721211 >> v1:172.21.15.114:6789/0 conn(0x7f2df406d330 0x7f2df4071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44310/0 (socket says 172.21.15.114:44310) 2024-03-06T07:32:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.373+0000 7f2df9400640 1 -- 172.21.15.114:0/795721211 learned_addr learned my addr 172.21.15.114:0/795721211 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.374+0000 7f2df9e00640 1 --2- 172.21.15.114:0/795721211 >> v2:172.21.15.177:3300/0 conn(0x7f2df415be10 0x7f2df40ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.374+0000 7f2df9400640 1 -- 172.21.15.114:0/795721211 >> v2:172.21.15.177:3300/0 conn(0x7f2df415be10 msgr2=0x7f2df40ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.374+0000 7f2df9400640 1 --2- 172.21.15.114:0/795721211 >> v2:172.21.15.177:3300/0 conn(0x7f2df415be10 0x7f2df40ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.374+0000 7f2df9400640 1 -- 172.21.15.114:0/795721211 >> v1:172.21.15.114:6789/0 conn(0x7f2df406d330 legacy=0x7f2df4071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.374+0000 7f2df9400640 1 -- 172.21.15.114:0/795721211 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df406cda0 con 0x7f2df40aef20 2024-03-06T07:32:03.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.375+0000 7f2df9400640 1 --2- 172.21.15.114:0/795721211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df40aef20 0x7f2df40ce2b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2df40d09c0 tx=0x7f2de402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7805ff4d43623843 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.375+0000 7f2df8a00640 1 -- 172.21.15.114:0/795721211 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4034070 con 0x7f2df40aef20 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.375+0000 7f2df8a00640 1 -- 172.21.15.114:0/795721211 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de4030860 con 0x7f2df40aef20 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.375+0000 7f2df8a00640 1 -- 172.21.15.114:0/795721211 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4030c30 con 0x7f2df40aef20 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 -- 172.21.15.114:0/795721211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df40aef20 msgr2=0x7f2df40ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 --2- 172.21.15.114:0/795721211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df40aef20 0x7f2df40ce2b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2df40d09c0 tx=0x7f2de402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 -- 172.21.15.114:0/795721211 shutdown_connections 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 --2- 172.21.15.114:0/795721211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df40aef20 0x7f2df40ce2b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 --2- 172.21.15.114:0/795721211 >> v2:172.21.15.177:3300/0 conn(0x7f2df415be10 0x7f2df40ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 -- 172.21.15.114:0/795721211 >> 172.21.15.114:0/795721211 conn(0x7f2df406afc0 msgr2=0x7f2df406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 -- 172.21.15.114:0/795721211 shutdown_connections 2024-03-06T07:32:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.376+0000 7f2dfb200640 1 -- 172.21.15.114:0/795721211 wait complete. 2024-03-06T07:32:03.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.378+0000 7f2dfb200640 1 Processor -- start 2024-03-06T07:32:03.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.378+0000 7f2dfb200640 1 -- start start 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2dfb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df406d330 0x7f2df41f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2dfb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df40aef20 0x7f2df415e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2dfb200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df415be10 0x7f2df415d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2dfb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df40cda90 con 0x7f2df406d330 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2dfb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df406d080 con 0x7f2df40aef20 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2dfb200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df406d7a0 con 0x7f2df415be10 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df406d330 0x7f2df41f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df406d330 0x7f2df41f4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34208/0 (socket says 172.21.15.114:34208) 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 -- 172.21.15.114:0/3085080407 learned_addr learned my addr 172.21.15.114:0/3085080407 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2dfa800640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df415be10 0x7f2df415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9400640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df40aef20 0x7f2df415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 -- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df415be10 msgr2=0x7f2df415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df415be10 0x7f2df415d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 -- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df40aef20 msgr2=0x7f2df415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df40aef20 0x7f2df415e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.379+0000 7f2df9e00640 1 -- 172.21.15.114:0/3085080407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df415ec80 con 0x7f2df406d330 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.380+0000 7f2df9e00640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df406d330 0x7f2df41f4840 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2de800d5f0 tx=0x7f2de800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.380+0000 7f2dee000640 1 -- 172.21.15.114:0/3085080407 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de80125f0 con 0x7f2df406d330 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.380+0000 7f2dee000640 1 -- 172.21.15.114:0/3085080407 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de8019070 con 0x7f2df406d330 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.380+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df415c930 con 0x7f2df406d330 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.380+0000 7f2dee000640 1 -- 172.21.15.114:0/3085080407 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de80129d0 con 0x7f2df406d330 2024-03-06T07:32:03.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.380+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df415cef0 con 0x7f2df406d330 2024-03-06T07:32:03.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.381+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df404f330 con 0x7f2df406d330 2024-03-06T07:32:03.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.382+0000 7f2dee000640 1 -- 172.21.15.114:0/3085080407 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2de8012b70 con 0x7f2df406d330 2024-03-06T07:32:03.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.383+0000 7f2dee000640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dd806b4d0 0x7f2dd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.383+0000 7f2dee000640 1 -- 172.21.15.114:0/3085080407 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2de808a6b0 con 0x7f2df406d330 2024-03-06T07:32:03.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.383+0000 7f2df9400640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dd806b4d0 0x7f2dd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.384+0000 7f2df9400640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dd806b4d0 0x7f2dd806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2de4036000 tx=0x7f2de403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.387+0000 7f2dee000640 1 -- 172.21.15.114:0/3085080407 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2de8059490 con 0x7f2df406d330 2024-03-06T07:32:03.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:03.627+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2df40723f0 con 0x7f2df406d330 2024-03-06T07:32:04.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.184+0000 7f2dee000640 1 -- 172.21.15.114:0/3085080407 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2de8058d90 con 0x7f2df406d330 2024-03-06T07:32:04.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dd806b4d0 msgr2=0x7f2dd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dd806b4d0 0x7f2dd806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2de4036000 tx=0x7f2de403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df406d330 msgr2=0x7f2df41f4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df406d330 0x7f2df41f4840 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2de800d5f0 tx=0x7f2de800dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 shutdown_connections 2024-03-06T07:32:04.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df415be10 0x7f2df415d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df40aef20 0x7f2df415e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dd806b4d0 0x7f2dd806d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 --2- 172.21.15.114:0/3085080407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df406d330 0x7f2df41f4840 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.188+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 >> 172.21.15.114:0/3085080407 conn(0x7f2df406afc0 msgr2=0x7f2df4153e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.189+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 shutdown_connections 2024-03-06T07:32:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.189+0000 7f2dfb200640 1 -- 172.21.15.114:0/3085080407 wait complete. 2024-03-06T07:32:04.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:04.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.421+0000 7fe635400640 1 Processor -- start 2024-03-06T07:32:04.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.421+0000 7fe635400640 1 -- start start 2024-03-06T07:32:04.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe635400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 0x7fe63006d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe635400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe63015c440 0x7fe63006db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe635400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe630072970 con 0x7fe630145750 2024-03-06T07:32:04.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe635400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300cd980 con 0x7fe63015c440 2024-03-06T07:32:04.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe635400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300cdbd0 con 0x7fe6300cd450 2024-03-06T07:32:04.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe62e800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe630145750 0x7fe630072260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44312/0 (socket says 172.21.15.114:44312) 2024-03-06T07:32:04.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe62e800640 1 -- 172.21.15.114:0/3004965094 learned_addr learned my addr 172.21.15.114:0/3004965094 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe62f200640 1 --2- 172.21.15.114:0/3004965094 >> v2:172.21.15.177:3300/0 conn(0x7fe63015c440 0x7fe63006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe625e00640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 0x7fe63006d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.423+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645297920 0 0) 0x7fe630072970 con 0x7fe630145750 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe614003520 con 0x7fe630145750 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3293165479 0 0) 0x7fe614003520 con 0x7fe630145750 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 msgr2=0x7fe63006d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 0x7fe63006d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 >> v2:172.21.15.177:3300/0 conn(0x7fe63015c440 msgr2=0x7fe63006db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 --2- 172.21.15.114:0/3004965094 >> v2:172.21.15.177:3300/0 conn(0x7fe63015c440 0x7fe63006db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6300cdeb0 con 0x7fe630145750 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fe620001670 con 0x7fe630145750 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 >> v1:172.21.15.114:6789/0 conn(0x7fe630145750 legacy=0x7fe630072260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe614003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 0x7fe614004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe614004d40 0x7fe6140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300cd980 con 0x7fe614003a20 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300cdbd0 con 0x7fe614004d40 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61400ba40 con 0x7fe6300cd450 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.424+0000 7fe62e800640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe614003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe625e00640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 0x7fe614004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe62f200640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe614004d40 0x7fe6140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe62e800640 1 -- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 msgr2=0x7fe614004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe62e800640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 0x7fe614004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe62e800640 1 -- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe614004d40 msgr2=0x7fe6140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe62e800640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe614004d40 0x7fe6140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe62e800640 1 -- 172.21.15.114:0/3004965094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe614003520 con 0x7fe614003a20 2024-03-06T07:32:04.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe625e00640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6300cd450 0x7fe614004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:04.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.425+0000 7fe62f200640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe614004d40 0x7fe6140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:04.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.426+0000 7fe62e800640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe614003e40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c290 tx=0x7fe614005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5b3186b2b971e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.426+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620007d50 con 0x7fe614003a20 2024-03-06T07:32:04.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.426+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe620030070 con 0x7fe614003a20 2024-03-06T07:32:04.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.426+0000 7fe62de00640 1 -- 172.21.15.114:0/3004965094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62002cd80 con 0x7fe614003a20 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.427+0000 7fe635400640 1 -- 172.21.15.114:0/3004965094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 msgr2=0x7fe614003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.427+0000 7fe635400640 1 --2- 172.21.15.114:0/3004965094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe614003e40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c290 tx=0x7fe614005830 comp rx=0 tx=0).stop 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.427+0000 7fe62f200640 1 -- 172.21.15.114:0/3004965094 reap_dead start 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.427+0000 7fe635400640 1 -- 172.21.15.114:0/3004965094 shutdown_connections 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.427+0000 7fe635400640 1 -- 172.21.15.114:0/3004965094 >> 172.21.15.114:0/3004965094 conn(0x7fe6300cfa80 msgr2=0x7fe6300cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.427+0000 7fe635400640 1 -- 172.21.15.114:0/3004965094 shutdown_connections 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.427+0000 7fe635400640 1 -- 172.21.15.114:0/3004965094 wait complete. 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.429+0000 7fe635400640 1 Processor -- start 2024-03-06T07:32:04.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.429+0000 7fe635400640 1 -- start start 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe635400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe630158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe635400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe630157530 0x7fe6301596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe635400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630159c30 0x7fe6300b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe635400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63006d620 con 0x7fe614003a20 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe635400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300ce470 con 0x7fe630159c30 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe635400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63006d7a0 con 0x7fe630157530 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe62e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe630158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe62e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe630158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34226/0 (socket says 172.21.15.114:34226) 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.430+0000 7fe62e800640 1 -- 172.21.15.114:0/3403428569 learned_addr learned my addr 172.21.15.114:0/3403428569 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:04.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe625e00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe630157530 0x7fe6301596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe62e800640 1 -- 172.21.15.114:0/3403428569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe630157530 msgr2=0x7fe6301596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe62e800640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe630157530 0x7fe6301596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe62e800640 1 -- 172.21.15.114:0/3403428569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630159c30 msgr2=0x7fe6300b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe62e800640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630159c30 0x7fe6300b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe62e800640 1 -- 172.21.15.114:0/3403428569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe630141130 con 0x7fe614003a20 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe62e800640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe630158d10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c290 tx=0x7fe6300ce080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe627400640 1 -- 172.21.15.114:0/3403428569 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620007d50 con 0x7fe614003a20 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe627400640 1 -- 172.21.15.114:0/3403428569 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe620004bf0 con 0x7fe614003a20 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe627400640 1 -- 172.21.15.114:0/3403428569 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620001c80 con 0x7fe614003a20 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe635400640 1 -- 172.21.15.114:0/3403428569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe630140370 con 0x7fe614003a20 2024-03-06T07:32:04.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.431+0000 7fe635400640 1 -- 172.21.15.114:0/3403428569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630140930 con 0x7fe614003a20 2024-03-06T07:32:04.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.433+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63015a050 con 0x7fe614003a20 2024-03-06T07:32:04.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.433+0000 7fe627400640 1 -- 172.21.15.114:0/3403428569 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe620047020 con 0x7fe614003a20 2024-03-06T07:32:04.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.434+0000 7fe627400640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe60c06b500 0x7fe60c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.434+0000 7fe627400640 1 -- 172.21.15.114:0/3403428569 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe6200ab320 con 0x7fe614003a20 2024-03-06T07:32:04.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.435+0000 7fe625e00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe60c06b500 0x7fe60c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.435+0000 7fe625e00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe60c06b500 0x7fe60c06d9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe6301f7720 tx=0x7fe618008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.439+0000 7fe627400640 1 -- 172.21.15.114:0/3403428569 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe62007a080 con 0x7fe614003a20 2024-03-06T07:32:04.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:04.634+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe630157910 con 0x7fe614003a20 2024-03-06T07:32:05.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.204+0000 7fe627400640 1 -- 172.21.15.114:0/3403428569 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v70) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe620034e80 con 0x7fe614003a20 2024-03-06T07:32:05.207 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:05.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.207+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe60c06b500 msgr2=0x7fe60c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.207+0000 7fe61fe00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe60c06b500 0x7fe60c06d9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe6301f7720 tx=0x7fe618008040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.207+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 msgr2=0x7fe630158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.207+0000 7fe61fe00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe630158d10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c290 tx=0x7fe6300ce080 comp rx=0 tx=0).stop 2024-03-06T07:32:05.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 shutdown_connections 2024-03-06T07:32:05.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe630159c30 0x7fe6300b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe630157530 0x7fe6301596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe60c06b500 0x7fe60c06d9c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 --2- 172.21.15.114:0/3403428569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614003a20 0x7fe630158d10 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 >> 172.21.15.114:0/3403428569 conn(0x7fe6300cfa80 msgr2=0x7fe630141ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 shutdown_connections 2024-03-06T07:32:05.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.208+0000 7fe61fe00640 1 -- 172.21.15.114:0/3403428569 wait complete. 2024-03-06T07:32:05.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:05.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.441+0000 7fa885800640 1 Processor -- start 2024-03-06T07:32:05.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.442+0000 7fa885800640 1 -- start start 2024-03-06T07:32:05.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.443+0000 7fa885800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa880145980 0x7fa88015fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.443+0000 7fa885800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa880160350 con 0x7fa8800ced20 2024-03-06T07:32:05.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.443+0000 7fa885800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8800cf1e0 con 0x7fa88015ff40 2024-03-06T07:32:05.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.443+0000 7fa885800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa880070e40 con 0x7fa880145980 2024-03-06T07:32:05.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.444+0000 7fa876000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa8800ced20 0x7fa8800712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44322/0 (socket says 172.21.15.114:44322) 2024-03-06T07:32:05.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.444+0000 7fa876000640 1 -- 172.21.15.114:0/2406512066 learned_addr learned my addr 172.21.15.114:0/2406512066 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.444+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103916300 0 0) 0x7fa868001110 con 0x7fa8800ced20 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.444+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa864003520 con 0x7fa8800ced20 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.444+0000 7fa87ea00640 1 --2- 172.21.15.114:0/2406512066 >> v2:172.21.15.177:3300/0 conn(0x7fa880145980 0x7fa88015fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1193066563 0 0) 0x7fa864003520 con 0x7fa8800ced20 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 >> v2:172.21.15.177:3300/0 conn(0x7fa880145980 msgr2=0x7fa88015fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 --2- 172.21.15.114:0/2406512066 >> v2:172.21.15.177:3300/0 conn(0x7fa880145980 0x7fa88015fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 >> v1:172.21.15.177:6789/0 conn(0x7fa88015ff40 legacy=0x7fa8800719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8800cf4c0 con 0x7fa8800ced20 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fa868001840 con 0x7fa8800ced20 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 >> v1:172.21.15.114:6789/0 conn(0x7fa8800ced20 legacy=0x7fa8800712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa864003a20 0x7fa864003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa864004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa864005230 0x7fa86400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa880070e40 con 0x7fa864004800 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86400c190 con 0x7fa864005230 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86400c390 con 0x7fa864003a20 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa876000640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa864004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa87ea00640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa864003a20 0x7fa864003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa876000640 1 -- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa864003a20 msgr2=0x7fa864003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.445+0000 7fa876000640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa864003a20 0x7fa864003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.446+0000 7fa876000640 1 -- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa864005230 msgr2=0x7fa86400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.446+0000 7fa87f400640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa864005230 0x7fa86400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.446+0000 7fa876000640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa864005230 0x7fa86400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.446+0000 7fa876000640 1 -- 172.21.15.114:0/2406512066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa864003520 con 0x7fa864004800 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.446+0000 7fa87f400640 1 -- 172.21.15.114:0/2406512066 reap_dead start 2024-03-06T07:32:05.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.447+0000 7fa876000640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa864004c80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c4c0 tx=0x7fa864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6863a5552d3e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.447+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86800c0e0 con 0x7fa864004800 2024-03-06T07:32:05.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.447+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868030070 con 0x7fa864004800 2024-03-06T07:32:05.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.447+0000 7fa87e000640 1 -- 172.21.15.114:0/2406512066 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8680058d0 con 0x7fa864004800 2024-03-06T07:32:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.447+0000 7fa885800640 1 -- 172.21.15.114:0/2406512066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 msgr2=0x7fa864004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.447+0000 7fa885800640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa864004c80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c4c0 tx=0x7fa864005830 comp rx=0 tx=0).stop 2024-03-06T07:32:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.448+0000 7fa885800640 1 -- 172.21.15.114:0/2406512066 shutdown_connections 2024-03-06T07:32:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.448+0000 7fa885800640 1 --2- 172.21.15.114:0/2406512066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa864004c80 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.448+0000 7fa885800640 1 -- 172.21.15.114:0/2406512066 >> 172.21.15.114:0/2406512066 conn(0x7fa8800cfa80 msgr2=0x7fa8800cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.448+0000 7fa885800640 1 -- 172.21.15.114:0/2406512066 shutdown_connections 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.448+0000 7fa885800640 1 -- 172.21.15.114:0/2406512066 wait complete. 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.449+0000 7fa885800640 1 Processor -- start 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.450+0000 7fa885800640 1 -- start start 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.450+0000 7fa885800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa8801507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa885800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8800ad860 0x7fa8801e61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa885800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8801e5750 0x7fa8801e6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa885800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa880160740 con 0x7fa864004800 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa885800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa880070ad0 con 0x7fa8800ad860 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa885800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8801608c0 con 0x7fa8801e5750 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa8801507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa8801507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34232/0 (socket says 172.21.15.114:34232) 2024-03-06T07:32:05.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 -- 172.21.15.114:0/1994800148 learned_addr learned my addr 172.21.15.114:0/1994800148 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa876000640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8800ad860 0x7fa8801e61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 -- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8801e5750 msgr2=0x7fa8801e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87f400640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8801e5750 0x7fa8801e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8801e5750 0x7fa8801e6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 -- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8800ad860 msgr2=0x7fa8801e61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8800ad860 0x7fa8801e61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa87ea00640 1 -- 172.21.15.114:0/1994800148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8800ae750 con 0x7fa864004800 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.451+0000 7fa876000640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8800ad860 0x7fa8801e61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.452+0000 7fa87ea00640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa8801507a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa87000e7e0 tx=0x7fa87000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.452+0000 7fa877400640 1 -- 172.21.15.114:0/1994800148 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870017070 con 0x7fa864004800 2024-03-06T07:32:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.452+0000 7fa877400640 1 -- 172.21.15.114:0/1994800148 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87000c670 con 0x7fa864004800 2024-03-06T07:32:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.452+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8800ae9d0 con 0x7fa864004800 2024-03-06T07:32:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.452+0000 7fa877400640 1 -- 172.21.15.114:0/1994800148 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87000f040 con 0x7fa864004800 2024-03-06T07:32:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.452+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8800aef90 con 0x7fa864004800 2024-03-06T07:32:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.453+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88004f330 con 0x7fa864004800 2024-03-06T07:32:05.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.454+0000 7fa877400640 1 -- 172.21.15.114:0/1994800148 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa87001a070 con 0x7fa864004800 2024-03-06T07:32:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.455+0000 7fa877400640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa85c06b4d0 0x7fa85c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.455+0000 7fa877400640 1 -- 172.21.15.114:0/1994800148 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5415+0+0 (secure 0 0 0) 0x7fa870089c70 con 0x7fa864004800 2024-03-06T07:32:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.455+0000 7fa876000640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa85c06b4d0 0x7fa85c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.456+0000 7fa876000640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa85c06b4d0 0x7fa85c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c490 tx=0x7fa868039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.460+0000 7fa877400640 1 -- 172.21.15.114:0/1994800148 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8800ae750 con 0x7fa864004800 2024-03-06T07:32:05.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:05.654+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8800af160 con 0x7fa864004800 2024-03-06T07:32:06.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.207+0000 7fa877400640 1 -- 172.21.15.114:0/1994800148 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v71) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa870058c00 con 0x7fa864004800 2024-03-06T07:32:06.210 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.211+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa85c06b4d0 msgr2=0x7fa85c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.211+0000 7fa885800640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa85c06b4d0 0x7fa85c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c490 tx=0x7fa868039040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.211+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 msgr2=0x7fa8801507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.211+0000 7fa885800640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa8801507a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa87000e7e0 tx=0x7fa87000eca0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.212+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 shutdown_connections 2024-03-06T07:32:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.212+0000 7fa885800640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8801e5750 0x7fa8801e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.212+0000 7fa885800640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8800ad860 0x7fa8801e61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.212+0000 7fa885800640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa85c06b4d0 0x7fa85c06d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.212+0000 7fa885800640 1 --2- 172.21.15.114:0/1994800148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa864004800 0x7fa8801507a0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.212+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 >> 172.21.15.114:0/1994800148 conn(0x7fa8800cfa80 msgr2=0x7fa8800afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.213+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 shutdown_connections 2024-03-06T07:32:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.213+0000 7fa885800640 1 -- 172.21.15.114:0/1994800148 wait complete. 2024-03-06T07:32:06.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.440+0000 7f97f1200640 1 Processor -- start 2024-03-06T07:32:06.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.440+0000 7f97f1200640 1 -- start start 2024-03-06T07:32:06.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.441+0000 7f97f1200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec15be10 0x7f97ec0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.441+0000 7f97f1200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f97ec0aef20 0x7f97ec0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.441+0000 7f97f1200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec15c1f0 con 0x7f97ec06d330 2024-03-06T07:32:06.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.442+0000 7f97f1200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec0cd200 con 0x7f97ec0aef20 2024-03-06T07:32:06.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.442+0000 7f97f1200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec06cac0 con 0x7f97ec15be10 2024-03-06T07:32:06.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.442+0000 7f97eb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec15be10 0x7f97ec0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.442+0000 7f97eaa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f97ec0aef20 0x7f97ec0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.442+0000 7f97eb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec15be10 0x7f97ec0ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59862/0 (socket says 172.21.15.114:59862) 2024-03-06T07:32:06.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.442+0000 7f97eb400640 1 -- 172.21.15.114:0/2745149812 learned_addr learned my addr 172.21.15.114:0/2745149812 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:06.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.442+0000 7f97ebe00640 1 --1- 172.21.15.114:0/2745149812 >> v1:172.21.15.177:6789/0 conn(0x7f97ec06d330 0x7f97ec071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:46716/0 (socket says 172.21.15.114:46716) 2024-03-06T07:32:06.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.443+0000 7f97eaa00640 1 -- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec15be10 msgr2=0x7f97ec0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.443+0000 7f97eaa00640 1 --2- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec15be10 0x7f97ec0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.443+0000 7f97eaa00640 1 -- 172.21.15.114:0/2745149812 >> v1:172.21.15.177:6789/0 conn(0x7f97ec06d330 legacy=0x7f97ec071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.443+0000 7f97eaa00640 1 -- 172.21.15.114:0/2745149812 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec06cda0 con 0x7f97ec0aef20 2024-03-06T07:32:06.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.443+0000 7f97eb400640 1 --2- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec15be10 0x7f97ec0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:06.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.445+0000 7f97eaa00640 1 --2- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec0aef20 0x7f97ec0ce2b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f97d80097a0 tx=0x7f97d802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c08121153944a505 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.445+0000 7f97ea000640 1 -- 172.21.15.114:0/2745149812 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8035020 con 0x7f97ec0aef20 2024-03-06T07:32:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.445+0000 7f97ea000640 1 -- 172.21.15.114:0/2745149812 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d8031830 con 0x7f97ec0aef20 2024-03-06T07:32:06.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.445+0000 7f97ea000640 1 -- 172.21.15.114:0/2745149812 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d8031c00 con 0x7f97ec0aef20 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 -- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec0aef20 msgr2=0x7f97ec0ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 --2- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec0aef20 0x7f97ec0ce2b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f97d80097a0 tx=0x7f97d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 -- 172.21.15.114:0/2745149812 shutdown_connections 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 --2- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec0aef20 0x7f97ec0ce2b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 --2- 172.21.15.114:0/2745149812 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec15be10 0x7f97ec0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 -- 172.21.15.114:0/2745149812 >> 172.21.15.114:0/2745149812 conn(0x7f97ec06afc0 msgr2=0x7f97ec06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 -- 172.21.15.114:0/2745149812 shutdown_connections 2024-03-06T07:32:06.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.446+0000 7f97f1200640 1 -- 172.21.15.114:0/2745149812 wait complete. 2024-03-06T07:32:06.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.448+0000 7f97f1200640 1 Processor -- start 2024-03-06T07:32:06.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.448+0000 7f97f1200640 1 -- start start 2024-03-06T07:32:06.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97f1200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec06d330 0x7f97ec152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97f1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97ec0aef20 0x7f97ec1508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97f1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec15be10 0x7f97ec0b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97f1200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0cda90 con 0x7f97ec0aef20 2024-03-06T07:32:06.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97f1200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec06d080 con 0x7f97ec15be10 2024-03-06T07:32:06.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97f1200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec06d7c0 con 0x7f97ec06d330 2024-03-06T07:32:06.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97ec0aef20 0x7f97ec1508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97ec0aef20 0x7f97ec1508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34242/0 (socket says 172.21.15.114:34242) 2024-03-06T07:32:06.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eaa00640 1 -- 172.21.15.114:0/2319732758 learned_addr learned my addr 172.21.15.114:0/2319732758 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:06.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eb400640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec06d330 0x7f97ec152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97ebe00640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec15be10 0x7f97ec0b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eaa00640 1 -- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec06d330 msgr2=0x7f97ec152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eaa00640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec06d330 0x7f97ec152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eaa00640 1 -- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec15be10 msgr2=0x7f97ec0b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.449+0000 7f97eaa00640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec15be10 0x7f97ec0b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97eaa00640 1 -- 172.21.15.114:0/2319732758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0b4980 con 0x7f97ec0aef20 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97eb400640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec06d330 0x7f97ec152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97eaa00640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97ec0aef20 0x7f97ec1508a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f97d80098c0 tx=0x7f97d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97e3400640 1 -- 172.21.15.114:0/2319732758 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d803b980 con 0x7f97ec0aef20 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97e3400640 1 -- 172.21.15.114:0/2319732758 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d803bb20 con 0x7f97ec0aef20 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec151010 con 0x7f97ec0aef20 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97e3400640 1 -- 172.21.15.114:0/2319732758 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80453f0 con 0x7f97ec0aef20 2024-03-06T07:32:06.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.450+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec1515d0 con 0x7f97ec0aef20 2024-03-06T07:32:06.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.451+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec04f330 con 0x7f97ec0aef20 2024-03-06T07:32:06.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.452+0000 7f97e3400640 1 -- 172.21.15.114:0/2319732758 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97d804c060 con 0x7f97ec0aef20 2024-03-06T07:32:06.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.453+0000 7f97e3400640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f97cc06b500 0x7f97cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.453+0000 7f97e3400640 1 -- 172.21.15.114:0/2319732758 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f97d80abcf0 con 0x7f97ec0aef20 2024-03-06T07:32:06.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.453+0000 7f97eb400640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f97cc06b500 0x7f97cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.454+0000 7f97eb400640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f97cc06b500 0x7f97cc06d9c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009a20 tx=0x7f97dc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.458+0000 7f97e3400640 1 -- 172.21.15.114:0/2319732758 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97d8040070 con 0x7f97ec0aef20 2024-03-06T07:32:06.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.653+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f97ec1f40b0 con 0x7f97ec0aef20 2024-03-06T07:32:06.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.658+0000 7f97e3400640 1 -- 172.21.15.114:0/2319732758 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f97d802f2e0 con 0x7f97ec0aef20 2024-03-06T07:32:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f97cc06b500 msgr2=0x7f97cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f97cc06b500 0x7f97cc06d9c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009a20 tx=0x7f97dc008040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97ec0aef20 msgr2=0x7f97ec1508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97ec0aef20 0x7f97ec1508a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f97d80098c0 tx=0x7f97d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 shutdown_connections 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97ec15be10 0x7f97ec0b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f97cc06b500 0x7f97cc06d9c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97ec0aef20 0x7f97ec1508a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.661+0000 7f97f1200640 1 --2- 172.21.15.114:0/2319732758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f97ec06d330 0x7f97ec152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.662+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 >> 172.21.15.114:0/2319732758 conn(0x7f97ec06afc0 msgr2=0x7f97ec071ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.662+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 shutdown_connections 2024-03-06T07:32:06.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.662+0000 7f97f1200640 1 -- 172.21.15.114:0/2319732758 wait complete. 2024-03-06T07:32:06.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:32:06.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.890+0000 7fa607200640 1 Processor -- start 2024-03-06T07:32:06.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.890+0000 7fa607200640 1 -- start start 2024-03-06T07:32:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa607200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6080d1ea0 0x7fa6080d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa607200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080c7970 0x7fa6080c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa607200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080d13e0 con 0x7fa60815e130 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa607200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6080d2840 con 0x7fa6080d1ea0 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa607200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080cf120 con 0x7fa6080c7970 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa5fd400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6080d1ea0 0x7fa6080d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa605e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa60815e130 0x7fa6080cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44336/0 (socket says 172.21.15.114:44336) 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.891+0000 7fa605e00640 1 -- 172.21.15.114:0/3025623804 learned_addr learned my addr 172.21.15.114:0/3025623804 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.892+0000 7fa606800640 1 --2- 172.21.15.114:0/3025623804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080c7970 0x7fa6080c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.892+0000 7fa5fd400640 1 -- 172.21.15.114:0/3025623804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080c7970 msgr2=0x7fa6080c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.892+0000 7fa5fd400640 1 --2- 172.21.15.114:0/3025623804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080c7970 0x7fa6080c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.892+0000 7fa5fd400640 1 -- 172.21.15.114:0/3025623804 >> v1:172.21.15.114:6789/0 conn(0x7fa60815e130 legacy=0x7fa6080cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.892+0000 7fa5fd400640 1 -- 172.21.15.114:0/3025623804 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6080cead0 con 0x7fa6080d1ea0 2024-03-06T07:32:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.892+0000 7fa605400640 1 -- 172.21.15.114:0/3025623804 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765700196 0 0) 0x7fa6080d13e0 con 0x7fa60815e130 2024-03-06T07:32:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.893+0000 7fa5fd400640 1 --2- 172.21.15.114:0/3025623804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080d1ea0 0x7fa6080d2300 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00097a0 tx=0x7fa5f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965b07648b1af652 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa605400640 1 -- 172.21.15.114:0/3025623804 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0035020 con 0x7fa6080d1ea0 2024-03-06T07:32:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa605400640 1 -- 172.21.15.114:0/3025623804 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f0031830 con 0x7fa6080d1ea0 2024-03-06T07:32:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa605400640 1 -- 172.21.15.114:0/3025623804 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0031c00 con 0x7fa6080d1ea0 2024-03-06T07:32:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 -- 172.21.15.114:0/3025623804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080d1ea0 msgr2=0x7fa6080d2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 --2- 172.21.15.114:0/3025623804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080d1ea0 0x7fa6080d2300 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00097a0 tx=0x7fa5f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 -- 172.21.15.114:0/3025623804 shutdown_connections 2024-03-06T07:32:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 --2- 172.21.15.114:0/3025623804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080c7970 0x7fa6080c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 --2- 172.21.15.114:0/3025623804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080d1ea0 0x7fa6080d2300 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 -- 172.21.15.114:0/3025623804 >> 172.21.15.114:0/3025623804 conn(0x7fa60809d070 msgr2=0x7fa6080d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 -- 172.21.15.114:0/3025623804 shutdown_connections 2024-03-06T07:32:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.894+0000 7fa607200640 1 -- 172.21.15.114:0/3025623804 wait complete. 2024-03-06T07:32:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.896+0000 7fa607200640 1 Processor -- start 2024-03-06T07:32:06.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.896+0000 7fa607200640 1 -- start start 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa607200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080c7970 0x7fa6081f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa607200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080d1ea0 0x7fa60806cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa607200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa60815e130 0x7fa6080b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa607200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080cfcc0 con 0x7fa6080c7970 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa607200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080c8340 con 0x7fa60815e130 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa607200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080d16c0 con 0x7fa6080d1ea0 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa605e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080c7970 0x7fa6081f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa605e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080c7970 0x7fa6081f7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34266/0 (socket says 172.21.15.114:34266) 2024-03-06T07:32:06.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.897+0000 7fa605e00640 1 -- 172.21.15.114:0/2168891844 learned_addr learned my addr 172.21.15.114:0/2168891844 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa605e00640 1 -- 172.21.15.114:0/2168891844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080d1ea0 msgr2=0x7fa60806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa605e00640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080d1ea0 0x7fa60806cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa605e00640 1 -- 172.21.15.114:0/2168891844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa60815e130 msgr2=0x7fa6080b4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa605e00640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa60815e130 0x7fa6080b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa605e00640 1 -- 172.21.15.114:0/2168891844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6081f81f0 con 0x7fa6080c7970 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa605e00640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080c7970 0x7fa6081f7cb0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800c4d0 tx=0x7fa5f800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa5fea00640 1 -- 172.21.15.114:0/2168891844 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8018070 con 0x7fa6080c7970 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa5fea00640 1 -- 172.21.15.114:0/2168891844 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f8007500 con 0x7fa6080c7970 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6081f40c0 con 0x7fa6080c7970 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa5fea00640 1 -- 172.21.15.114:0/2168891844 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8004a20 con 0x7fa6080c7970 2024-03-06T07:32:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.898+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6081f4680 con 0x7fa6080c7970 2024-03-06T07:32:06.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.900+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60804f330 con 0x7fa6080c7970 2024-03-06T07:32:06.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.900+0000 7fa5fea00640 1 -- 172.21.15.114:0/2168891844 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5f801b070 con 0x7fa6080c7970 2024-03-06T07:32:06.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.901+0000 7fa5fea00640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa5e406b4d0 0x7fa5e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.901+0000 7fa5fea00640 1 -- 172.21.15.114:0/2168891844 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa5f8089220 con 0x7fa6080c7970 2024-03-06T07:32:06.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.902+0000 7fa5fd400640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa5e406b4d0 0x7fa5e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.902+0000 7fa5fd400640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa5e406b4d0 0x7fa5e406d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0036000 tx=0x7fa5f0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:06.906+0000 7fa5fea00640 1 -- 172.21.15.114:0/2168891844 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6081f81f0 con 0x7fa6080c7970 2024-03-06T07:32:07.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.101+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fa60809d690 con 0x7fa6080c7970 2024-03-06T07:32:07.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.106+0000 7fa5fea00640 1 -- 172.21.15.114:0/2168891844 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v25) v1 ==== 108+0+0 (secure 0 0 0) 0x7fa5f8058390 con 0x7fa6080c7970 2024-03-06T07:32:07.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.109+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa5e406b4d0 msgr2=0x7fa5e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.109+0000 7fa607200640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa5e406b4d0 0x7fa5e406d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0036000 tx=0x7fa5f0030000 comp rx=0 tx=0).stop 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.109+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080c7970 msgr2=0x7fa6081f7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.109+0000 7fa607200640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080c7970 0x7fa6081f7cb0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800c4d0 tx=0x7fa5f800c990 comp rx=0 tx=0).stop 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 shutdown_connections 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa60815e130 0x7fa6080b4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6080d1ea0 0x7fa60806cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa5e406b4d0 0x7fa5e406d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 --2- 172.21.15.114:0/2168891844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6080c7970 0x7fa6081f7cb0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 >> 172.21.15.114:0/2168891844 conn(0x7fa60809d070 msgr2=0x7fa6080c8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 shutdown_connections 2024-03-06T07:32:07.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.110+0000 7fa607200640 1 -- 172.21.15.114:0/2168891844 wait complete. 2024-03-06T07:32:07.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:32:07.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.341+0000 7ff7a1a00640 1 Processor -- start 2024-03-06T07:32:07.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.342+0000 7ff7a1a00640 1 -- start start 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.343+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c0af1c0 0x7ff79c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.343+0000 7ff7a1a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff79c06e4d0 0x7ff79c06e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.343+0000 7ff7a1a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79c071d30 con 0x7ff79c070d20 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.343+0000 7ff7a1a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c071490 con 0x7ff79c06e4d0 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.343+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c071770 con 0x7ff79c0af1c0 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.343+0000 7ff79b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff79c06e4d0 0x7ff79c06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff79c06e4d0 0x7ff79c06e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34272/0 (socket says 172.21.15.114:34272) 2024-03-06T07:32:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79b400640 1 -- 172.21.15.114:0/1401995506 learned_addr learned my addr 172.21.15.114:0/1401995506 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79b400640 1 -- 172.21.15.114:0/1401995506 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c0af1c0 msgr2=0x7ff79c15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79aa00640 1 --2- 172.21.15.114:0/1401995506 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c0af1c0 0x7ff79c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79b400640 1 --2- 172.21.15.114:0/1401995506 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c0af1c0 0x7ff79c15eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79b400640 1 -- 172.21.15.114:0/1401995506 >> v1:172.21.15.177:6789/0 conn(0x7ff79c070d20 legacy=0x7ff79c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79b400640 1 -- 172.21.15.114:0/1401995506 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c071a50 con 0x7ff79c06e4d0 2024-03-06T07:32:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.344+0000 7ff79aa00640 1 --2- 172.21.15.114:0/1401995506 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c0af1c0 0x7ff79c15eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.345+0000 7ff79b400640 1 --2- 172.21.15.114:0/1401995506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c06e4d0 0x7ff79c06e960 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff794009870 tx=0x7ff79402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de73f23d227cf2f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.345+0000 7ff799600640 1 -- 172.21.15.114:0/1401995506 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794035020 con 0x7ff79c06e4d0 2024-03-06T07:32:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff799600640 1 -- 172.21.15.114:0/1401995506 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff794031640 con 0x7ff79c06e4d0 2024-03-06T07:32:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff799600640 1 -- 172.21.15.114:0/1401995506 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79402d040 con 0x7ff79c06e4d0 2024-03-06T07:32:07.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1401995506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c06e4d0 msgr2=0x7ff79c06e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1401995506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c06e4d0 0x7ff79c06e960 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff794009870 tx=0x7ff79402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1401995506 shutdown_connections 2024-03-06T07:32:07.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1401995506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c06e4d0 0x7ff79c06e960 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1401995506 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c0af1c0 0x7ff79c15eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.346+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1401995506 >> 172.21.15.114:0/1401995506 conn(0x7ff79c0b1730 msgr2=0x7ff79c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.347+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1401995506 shutdown_connections 2024-03-06T07:32:07.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.347+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1401995506 wait complete. 2024-03-06T07:32:07.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.349+0000 7ff7a1a00640 1 Processor -- start 2024-03-06T07:32:07.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.349+0000 7ff7a1a00640 1 -- start start 2024-03-06T07:32:07.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c06e4d0 0x7ff79c158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c070d20 0x7ff79c0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff7a1a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c0af1c0 0x7ff79c14b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c06d370 con 0x7ff79c0af1c0 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0711a0 con 0x7ff79c06e4d0 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff7a1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0b35d0 con 0x7ff79c070d20 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff79b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c0af1c0 0x7ff79c14b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff79b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c0af1c0 0x7ff79c14b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34278/0 (socket says 172.21.15.114:34278) 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.350+0000 7ff79b400640 1 -- 172.21.15.114:0/1107903997 learned_addr learned my addr 172.21.15.114:0/1107903997 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff79b400640 1 -- 172.21.15.114:0/1107903997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c070d20 msgr2=0x7ff79c0cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff79b400640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c070d20 0x7ff79c0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff79b400640 1 -- 172.21.15.114:0/1107903997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c06e4d0 msgr2=0x7ff79c158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff79b400640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c06e4d0 0x7ff79c158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff79b400640 1 -- 172.21.15.114:0/1107903997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c158cf0 con 0x7ff79c0af1c0 2024-03-06T07:32:07.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff79b400640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c0af1c0 0x7ff79c14b570 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff794009870 tx=0x7ff7940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff792a00640 1 -- 172.21.15.114:0/1107903997 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794040070 con 0x7ff79c0af1c0 2024-03-06T07:32:07.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff792a00640 1 -- 172.21.15.114:0/1107903997 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff794039c10 con 0x7ff79c0af1c0 2024-03-06T07:32:07.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff792a00640 1 -- 172.21.15.114:0/1107903997 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794035040 con 0x7ff79c0af1c0 2024-03-06T07:32:07.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c0ccb10 con 0x7ff79c0af1c0 2024-03-06T07:32:07.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.351+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c0cd0d0 con 0x7ff79c0af1c0 2024-03-06T07:32:07.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.352+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c04f330 con 0x7ff79c0af1c0 2024-03-06T07:32:07.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.353+0000 7ff792a00640 1 -- 172.21.15.114:0/1107903997 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff794038020 con 0x7ff79c0af1c0 2024-03-06T07:32:07.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.355+0000 7ff792a00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff77806b4d0 0x7ff77806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.355+0000 7ff792a00640 1 -- 172.21.15.114:0/1107903997 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff7940aba80 con 0x7ff79c0af1c0 2024-03-06T07:32:07.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.355+0000 7ff79aa00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff77806b4d0 0x7ff77806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.356+0000 7ff79aa00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff77806b4d0 0x7ff77806d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff784006fd0 tx=0x7ff784008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.359+0000 7ff792a00640 1 -- 172.21.15.114:0/1107903997 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7940ade20 con 0x7ff79c0af1c0 2024-03-06T07:32:07.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:07.620+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7ff79c0b1350 con 0x7ff79c0af1c0 2024-03-06T07:32:08.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.226+0000 7ff792a00640 1 -- 172.21.15.114:0/1107903997 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v236) v1 ==== 140+0+0 (secure 0 0 0) 0x7ff79c071d30 con 0x7ff79c0af1c0 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff77806b4d0 msgr2=0x7ff77806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff77806b4d0 0x7ff77806d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff784006fd0 tx=0x7ff784008040 comp rx=0 tx=0).stop 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c0af1c0 msgr2=0x7ff79c14b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c0af1c0 0x7ff79c14b570 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff794009870 tx=0x7ff7940046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 shutdown_connections 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff77806b4d0 0x7ff77806d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff79c0af1c0 0x7ff79c14b570 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff79c070d20 0x7ff79c0cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 --2- 172.21.15.114:0/1107903997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c06e4d0 0x7ff79c158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.229+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 >> 172.21.15.114:0/1107903997 conn(0x7ff79c0b1730 msgr2=0x7ff79c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.230+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 shutdown_connections 2024-03-06T07:32:08.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.230+0000 7ff7a1a00640 1 -- 172.21.15.114:0/1107903997 wait complete. 2024-03-06T07:32:08.253 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:32:08.254 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:32:08.254 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:08.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:08.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.459+0000 7fb85be00640 1 Processor -- start 2024-03-06T07:32:08.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.459+0000 7fb85be00640 1 -- start start 2024-03-06T07:32:08.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.460+0000 7fb85be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb85c1561f0 0x7fb85c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.460+0000 7fb85be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb85c0aebf0 0x7fb85c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.460+0000 7fb85be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 0x7fb85c0cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.460+0000 7fb85be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c15ce30 con 0x7fb85c1561f0 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.460+0000 7fb85be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85c15c240 con 0x7fb85c0aebf0 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.460+0000 7fb85be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c15c500 con 0x7fb85c0cc800 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.460+0000 7fb85aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb85c1561f0 0x7fb85c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb85c1561f0 0x7fb85c0ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50952/0 (socket says 172.21.15.114:50952) 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85aa00640 1 -- 172.21.15.114:0/785866771 learned_addr learned my addr 172.21.15.114:0/785866771 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb852000640 1 --2- 172.21.15.114:0/785866771 >> v2:172.21.15.177:3300/0 conn(0x7fb85c0aebf0 0x7fb85c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85b400640 1 --2- 172.21.15.114:0/785866771 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 0x7fb85c0cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85aa00640 1 -- 172.21.15.114:0/785866771 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 msgr2=0x7fb85c0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85aa00640 1 --2- 172.21.15.114:0/785866771 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 0x7fb85c0cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85aa00640 1 -- 172.21.15.114:0/785866771 >> v2:172.21.15.177:3300/0 conn(0x7fb85c0aebf0 msgr2=0x7fb85c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85aa00640 1 --2- 172.21.15.114:0/785866771 >> v2:172.21.15.177:3300/0 conn(0x7fb85c0aebf0 0x7fb85c0cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.461+0000 7fb85aa00640 1 -- 172.21.15.114:0/785866771 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c15c7e0 con 0x7fb85c1561f0 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.462+0000 7fb85aa00640 1 --2- 172.21.15.114:0/785866771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c1561f0 0x7fb85c0ae6b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb85c068400 tx=0x7fb84c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d309104923f7790f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.462+0000 7fb85a000640 1 -- 172.21.15.114:0/785866771 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c035020 con 0x7fb85c1561f0 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.462+0000 7fb85a000640 1 -- 172.21.15.114:0/785866771 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c031870 con 0x7fb85c1561f0 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.462+0000 7fb85a000640 1 -- 172.21.15.114:0/785866771 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c031c40 con 0x7fb85c1561f0 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 -- 172.21.15.114:0/785866771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c1561f0 msgr2=0x7fb85c0ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 --2- 172.21.15.114:0/785866771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c1561f0 0x7fb85c0ae6b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb85c068400 tx=0x7fb84c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 -- 172.21.15.114:0/785866771 shutdown_connections 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 --2- 172.21.15.114:0/785866771 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 0x7fb85c0cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 --2- 172.21.15.114:0/785866771 >> v2:172.21.15.177:3300/0 conn(0x7fb85c0aebf0 0x7fb85c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 --2- 172.21.15.114:0/785866771 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c1561f0 0x7fb85c0ae6b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 -- 172.21.15.114:0/785866771 >> 172.21.15.114:0/785866771 conn(0x7fb85c06afc0 msgr2=0x7fb85c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 -- 172.21.15.114:0/785866771 shutdown_connections 2024-03-06T07:32:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.463+0000 7fb85be00640 1 -- 172.21.15.114:0/785866771 wait complete. 2024-03-06T07:32:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.465+0000 7fb85be00640 1 Processor -- start 2024-03-06T07:32:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.465+0000 7fb85be00640 1 -- start start 2024-03-06T07:32:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c0aebf0 0x7fb85c14c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 0x7fb85c157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb85c1561f0 0x7fb85c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1565d0 con 0x7fb85c0aebf0 2024-03-06T07:32:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c0ccc90 con 0x7fb85c1561f0 2024-03-06T07:32:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c156750 con 0x7fb85c0cc800 2024-03-06T07:32:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c0aebf0 0x7fb85c14c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c0aebf0 0x7fb85c14c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50960/0 (socket says 172.21.15.114:50960) 2024-03-06T07:32:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 -- 172.21.15.114:0/1891839060 learned_addr learned my addr 172.21.15.114:0/1891839060 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85b400640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb85c1561f0 0x7fb85c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 -- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 msgr2=0x7fb85c157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 0x7fb85c157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 -- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb85c1561f0 msgr2=0x7fb85c157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb85c1561f0 0x7fb85c157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85aa00640 1 -- 172.21.15.114:0/1891839060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c159e60 con 0x7fb85c0aebf0 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.466+0000 7fb85b400640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb85c1561f0 0x7fb85c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.467+0000 7fb85aa00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c0aebf0 0x7fb85c14c6b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb84c02b950 tx=0x7fb84c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.467+0000 7fb853400640 1 -- 172.21.15.114:0/1891839060 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c040070 con 0x7fb85c0aebf0 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.467+0000 7fb853400640 1 -- 172.21.15.114:0/1891839060 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c03a420 con 0x7fb85c0aebf0 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.467+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c0cf090 con 0x7fb85c0aebf0 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.467+0000 7fb853400640 1 -- 172.21.15.114:0/1891839060 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c035040 con 0x7fb85c0aebf0 2024-03-06T07:32:08.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.467+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c15a120 con 0x7fb85c0aebf0 2024-03-06T07:32:08.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.468+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c04f330 con 0x7fb85c0aebf0 2024-03-06T07:32:08.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.469+0000 7fb853400640 1 -- 172.21.15.114:0/1891839060 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb84c04b060 con 0x7fb85c0aebf0 2024-03-06T07:32:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.470+0000 7fb853400640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb83806b4d0 0x7fb83806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.470+0000 7fb853400640 1 -- 172.21.15.114:0/1891839060 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb84c0abea0 con 0x7fb85c0aebf0 2024-03-06T07:32:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.470+0000 7fb852000640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb83806b4d0 0x7fb83806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.471+0000 7fb852000640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb83806b4d0 0x7fb83806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb85c1f7330 tx=0x7fb844008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.474+0000 7fb853400640 1 -- 172.21.15.114:0/1891839060 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb84c07b010 con 0x7fb85c0aebf0 2024-03-06T07:32:08.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.673+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb85c0b2b10 con 0x7fb85c0aebf0 2024-03-06T07:32:08.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.675+0000 7fb853400640 1 -- 172.21.15.114:0/1891839060 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+13071 (secure 0 0 0) 0x7fb84c07a910 con 0x7fb85c0aebf0 2024-03-06T07:32:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:08.678 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":71,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:06.205404+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/3105808425":"2024-03-07T07:32:03.168435+0000","172.21.15.114:6836/3105808425":"2024-03-07T07:32:03.168435+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:08.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb83806b4d0 msgr2=0x7fb83806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb83806b4d0 0x7fb83806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb85c1f7330 tx=0x7fb844008040 comp rx=0 tx=0).stop 2024-03-06T07:32:08.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c0aebf0 msgr2=0x7fb85c14c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c0aebf0 0x7fb85c14c6b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb84c02b950 tx=0x7fb84c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 shutdown_connections 2024-03-06T07:32:08.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb85c1561f0 0x7fb85c157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb85c0cc800 0x7fb85c157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb83806b4d0 0x7fb83806d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 --2- 172.21.15.114:0/1891839060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb85c0aebf0 0x7fb85c14c6b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 >> 172.21.15.114:0/1891839060 conn(0x7fb85c06afc0 msgr2=0x7fb85c155390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 shutdown_connections 2024-03-06T07:32:08.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.678+0000 7fb85be00640 1 -- 172.21.15.114:0/1891839060 wait complete. 2024-03-06T07:32:08.698 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:08.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:08.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.897+0000 7f744e200640 1 Processor -- start 2024-03-06T07:32:08.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.897+0000 7f744e200640 1 -- start start 2024-03-06T07:32:08.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.899+0000 7f744e200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7448160700 0x7f7448070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.899+0000 7f744e200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74480b46c0 con 0x7f744815fc10 2024-03-06T07:32:08.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.899+0000 7f744e200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74480b49a0 con 0x7f74480b3f70 2024-03-06T07:32:08.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.899+0000 7f744e200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7448072c90 con 0x7f7448160700 2024-03-06T07:32:08.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.899+0000 7f7447e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7448160700 0x7f7448070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.899+0000 7f7447e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7448160700 0x7f7448070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50974/0 (socket says 172.21.15.114:50974) 2024-03-06T07:32:08.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.899+0000 7f7447e00640 1 -- 172.21.15.114:0/177656980 learned_addr learned my addr 172.21.15.114:0/177656980 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:08.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.900+0000 7f7447e00640 1 -- 172.21.15.114:0/177656980 >> v1:172.21.15.177:6789/0 conn(0x7f74480b3f70 legacy=0x7f74480ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.900+0000 7f7447e00640 1 -- 172.21.15.114:0/177656980 >> v1:172.21.15.114:6789/0 conn(0x7f744815fc10 legacy=0x7f744815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.900+0000 7f7447e00640 1 -- 172.21.15.114:0/177656980 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744809dd80 con 0x7f7448160700 2024-03-06T07:32:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.901+0000 7f7447e00640 1 --2- 172.21.15.114:0/177656980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7448160700 0x7f7448070ff0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7440009e50 tx=0x7f744002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf5594f4ba199f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.901+0000 7f7446a00640 1 -- 172.21.15.114:0/177656980 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744002cb10 con 0x7f7448160700 2024-03-06T07:32:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.901+0000 7f7446a00640 1 -- 172.21.15.114:0/177656980 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744002ccb0 con 0x7f7448160700 2024-03-06T07:32:08.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.901+0000 7f7446a00640 1 -- 172.21.15.114:0/177656980 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744002ce60 con 0x7f7448160700 2024-03-06T07:32:08.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.902+0000 7f744e200640 1 -- 172.21.15.114:0/177656980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7448160700 msgr2=0x7f7448070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.902+0000 7f744e200640 1 --2- 172.21.15.114:0/177656980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7448160700 0x7f7448070ff0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7440009e50 tx=0x7f744002bf70 comp rx=0 tx=0).stop 2024-03-06T07:32:08.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.902+0000 7f744e200640 1 -- 172.21.15.114:0/177656980 shutdown_connections 2024-03-06T07:32:08.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.902+0000 7f744e200640 1 --2- 172.21.15.114:0/177656980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7448160700 0x7f7448070ff0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.902+0000 7f744e200640 1 -- 172.21.15.114:0/177656980 >> 172.21.15.114:0/177656980 conn(0x7f74480b1940 msgr2=0x7f74480b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.902+0000 7f744e200640 1 -- 172.21.15.114:0/177656980 shutdown_connections 2024-03-06T07:32:08.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.902+0000 7f744e200640 1 -- 172.21.15.114:0/177656980 wait complete. 2024-03-06T07:32:08.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.904+0000 7f744e200640 1 Processor -- start 2024-03-06T07:32:08.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.904+0000 7f744e200640 1 -- start start 2024-03-06T07:32:08.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f744e200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74480b3f70 0x7f744814a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f744e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744815fc10 0x7f744806d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f744e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7448160700 0x7f744806dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f744e200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744815c460 con 0x7f744815fc10 2024-03-06T07:32:08.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f744e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744815c2e0 con 0x7f7448160700 2024-03-06T07:32:08.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f744e200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74480cebb0 con 0x7f74480b3f70 2024-03-06T07:32:08.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744815fc10 0x7f744806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744815fc10 0x7f744806d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50984/0 (socket says 172.21.15.114:50984) 2024-03-06T07:32:08.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 -- 172.21.15.114:0/3940782741 learned_addr learned my addr 172.21.15.114:0/3940782741 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:08.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f7447e00640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7448160700 0x7f744806dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f7447400640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74480b3f70 0x7f744814a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 -- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74480b3f70 msgr2=0x7f744814a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74480b3f70 0x7f744814a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 -- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7448160700 msgr2=0x7f744806dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7448160700 0x7f744806dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f743fe00640 1 -- 172.21.15.114:0/3940782741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74480c9060 con 0x7f744815fc10 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.905+0000 7f7447400640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74480b3f70 0x7f744814a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.906+0000 7f743fe00640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744815fc10 0x7f744806d730 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f743000c400 tx=0x7f743000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.906+0000 7f743f400640 1 -- 172.21.15.114:0/3940782741 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430011790 con 0x7f744815fc10 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.906+0000 7f743f400640 1 -- 172.21.15.114:0/3940782741 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7430017070 con 0x7f744815fc10 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.906+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74480c9340 con 0x7f744815fc10 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.906+0000 7f743f400640 1 -- 172.21.15.114:0/3940782741 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7430011bb0 con 0x7f744815fc10 2024-03-06T07:32:08.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.906+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74480c9900 con 0x7f744815fc10 2024-03-06T07:32:08.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.907+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744804f330 con 0x7f744815fc10 2024-03-06T07:32:08.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.908+0000 7f743f400640 1 -- 172.21.15.114:0/3940782741 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f743001c460 con 0x7f744815fc10 2024-03-06T07:32:08.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.909+0000 7f743f400640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f742406b4d0 0x7f742406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.909+0000 7f743f400640 1 -- 172.21.15.114:0/3940782741 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7430089d40 con 0x7f744815fc10 2024-03-06T07:32:08.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.909+0000 7f7447400640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f742406b4d0 0x7f742406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.910+0000 7f7447400640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f742406b4d0 0x7f742406d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7438001680 tx=0x7f7438009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:08.913+0000 7f743f400640 1 -- 172.21.15.114:0/3940782741 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7430058ec0 con 0x7f744815fc10 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.108+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f744806ef30 con 0x7f744815fc10 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.109+0000 7f743f400640 1 -- 172.21.15.114:0/3940782741 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f74300587c0 con 0x7f744815fc10 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f742406b4d0 msgr2=0x7f742406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f742406b4d0 0x7f742406d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7438001680 tx=0x7f7438009040 comp rx=0 tx=0).stop 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744815fc10 msgr2=0x7f744806d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744815fc10 0x7f744806d730 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f743000c400 tx=0x7f743000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 shutdown_connections 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7448160700 0x7f744806dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f742406b4d0 0x7f742406d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744815fc10 0x7f744806d730 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 --2- 172.21.15.114:0/3940782741 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74480b3f70 0x7f744814a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 >> 172.21.15.114:0/3940782741 conn(0x7f74480b1940 msgr2=0x7f74480b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 shutdown_connections 2024-03-06T07:32:09.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.112+0000 7f744e200640 1 -- 172.21.15.114:0/3940782741 wait complete. 2024-03-06T07:32:09.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:32:09.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.333+0000 7f2895600640 1 Processor -- start 2024-03-06T07:32:09.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.333+0000 7f2895600640 1 -- start start 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f2895600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f289015acb0 0x7f28900b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f2895600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f289006ec00 0x7f28900ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f2895600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289015b090 con 0x7f28900b4680 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f2895600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28900b2a30 con 0x7f289015acb0 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f2895600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28900b4360 con 0x7f289006ec00 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f288ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f289015acb0 0x7f28900b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f288e000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f28900b4680 0x7f289006e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37434/0 (socket says 172.21.15.114:37434) 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f288e000640 1 -- 172.21.15.114:0/4276452961 learned_addr learned my addr 172.21.15.114:0/4276452961 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f288f400640 1 --2- 172.21.15.114:0/4276452961 >> v2:172.21.15.177:3300/0 conn(0x7f289006ec00 0x7f28900ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.335+0000 7f288ea00640 1 -- 172.21.15.114:0/4276452961 >> v2:172.21.15.177:3300/0 conn(0x7f289006ec00 msgr2=0x7f28900ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.336+0000 7f288ea00640 1 --2- 172.21.15.114:0/4276452961 >> v2:172.21.15.177:3300/0 conn(0x7f289006ec00 0x7f28900ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.336+0000 7f288ea00640 1 -- 172.21.15.114:0/4276452961 >> v1:172.21.15.114:6789/0 conn(0x7f28900b4680 legacy=0x7f289006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.336+0000 7f288ea00640 1 -- 172.21.15.114:0/4276452961 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28900b2cf0 con 0x7f289015acb0 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.336+0000 7f288d600640 1 -- 172.21.15.114:0/4276452961 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221464177 0 0) 0x7f289015b090 con 0x7f28900b4680 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.336+0000 7f288f400640 1 --2- 172.21.15.114:0/4276452961 >> v2:172.21.15.177:3300/0 conn(0x7f289006ec00 0x7f28900ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.337+0000 7f288ea00640 1 --2- 172.21.15.114:0/4276452961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28900b37d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f28900b4e60 tx=0x7f288002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c861f23e0b4cb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.337+0000 7f288d600640 1 -- 172.21.15.114:0/4276452961 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880035020 con 0x7f289015acb0 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.337+0000 7f288d600640 1 -- 172.21.15.114:0/4276452961 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2880031830 con 0x7f289015acb0 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.337+0000 7f288d600640 1 -- 172.21.15.114:0/4276452961 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880031c00 con 0x7f289015acb0 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 -- 172.21.15.114:0/4276452961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 msgr2=0x7f28900b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 --2- 172.21.15.114:0/4276452961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28900b37d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f28900b4e60 tx=0x7f288002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 -- 172.21.15.114:0/4276452961 shutdown_connections 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 --2- 172.21.15.114:0/4276452961 >> v2:172.21.15.177:3300/0 conn(0x7f289006ec00 0x7f28900ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 --2- 172.21.15.114:0/4276452961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28900b37d0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 -- 172.21.15.114:0/4276452961 >> 172.21.15.114:0/4276452961 conn(0x7f28900b1730 msgr2=0x7f28900b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 -- 172.21.15.114:0/4276452961 shutdown_connections 2024-03-06T07:32:09.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.338+0000 7f2895600640 1 -- 172.21.15.114:0/4276452961 wait complete. 2024-03-06T07:32:09.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.340+0000 7f2895600640 1 Processor -- start 2024-03-06T07:32:09.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.340+0000 7f2895600640 1 -- start start 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.340+0000 7f2895600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289006ec00 0x7f28900cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f2895600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28900b4680 0x7f2890146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f2895600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28901f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f2895600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28900b3d10 con 0x7f289015acb0 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f2895600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28900b32b0 con 0x7f289006ec00 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f2895600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28900b4010 con 0x7f28900b4680 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28901f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28901f66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51004/0 (socket says 172.21.15.114:51004) 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 -- 172.21.15.114:0/818287902 learned_addr learned my addr 172.21.15.114:0/818287902 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288ea00640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289006ec00 0x7f28900cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 -- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28900b4680 msgr2=0x7f2890146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28900b4680 0x7f2890146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 -- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289006ec00 msgr2=0x7f28900cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289006ec00 0x7f28900cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288f400640 1 -- 172.21.15.114:0/818287902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2890147730 con 0x7f289015acb0 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.341+0000 7f288ea00640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289006ec00 0x7f28900cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.342+0000 7f288f400640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28901f66d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f288800c7c0 tx=0x7f288800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.342+0000 7f2886a00640 1 -- 172.21.15.114:0/818287902 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888015020 con 0x7f289015acb0 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.342+0000 7f2886a00640 1 -- 172.21.15.114:0/818287902 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2888004230 con 0x7f289015acb0 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.342+0000 7f2886a00640 1 -- 172.21.15.114:0/818287902 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888007620 con 0x7f289015acb0 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.342+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28901f5630 con 0x7f289015acb0 2024-03-06T07:32:09.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.342+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28901f5bf0 con 0x7f289015acb0 2024-03-06T07:32:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.343+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289004f330 con 0x7f289015acb0 2024-03-06T07:32:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.344+0000 7f2886a00640 1 -- 172.21.15.114:0/818287902 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2888018020 con 0x7f289015acb0 2024-03-06T07:32:09.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.345+0000 7f2886a00640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f286c06b4d0 0x7f286c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.345+0000 7f2886a00640 1 -- 172.21.15.114:0/818287902 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2888089fa0 con 0x7f289015acb0 2024-03-06T07:32:09.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.346+0000 7f288ea00640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f286c06b4d0 0x7f286c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.346+0000 7f288ea00640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f286c06b4d0 0x7f286c06d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2880036000 tx=0x7f2880008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.350+0000 7f2886a00640 1 -- 172.21.15.114:0/818287902 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f288801f020 con 0x7f289015acb0 2024-03-06T07:32:09.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:09.616+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f2890072260 con 0x7f289015acb0 2024-03-06T07:32:10.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.238+0000 7f2886a00640 1 -- 172.21.15.114:0/818287902 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v238) v1 ==== 135+0+0 (secure 0 0 0) 0x7f2890147730 con 0x7f289015acb0 2024-03-06T07:32:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f286c06b4d0 msgr2=0x7f286c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f286c06b4d0 0x7f286c06d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2880036000 tx=0x7f2880008030 comp rx=0 tx=0).stop 2024-03-06T07:32:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 msgr2=0x7f28901f66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28901f66d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f288800c7c0 tx=0x7f288800cc80 comp rx=0 tx=0).stop 2024-03-06T07:32:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 shutdown_connections 2024-03-06T07:32:10.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f286c06b4d0 0x7f286c06d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f289015acb0 0x7f28901f66d0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f28900b4680 0x7f2890146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 --2- 172.21.15.114:0/818287902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f289006ec00 0x7f28900cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 >> 172.21.15.114:0/818287902 conn(0x7f28900b1730 msgr2=0x7f289014d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.242+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 shutdown_connections 2024-03-06T07:32:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.243+0000 7f2895600640 1 -- 172.21.15.114:0/818287902 wait complete. 2024-03-06T07:32:10.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:32:10.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.479+0000 7f45c0a00640 1 Processor -- start 2024-03-06T07:32:10.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.479+0000 7f45c0a00640 1 -- start start 2024-03-06T07:32:10.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.480+0000 7f45c0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45bc15cb90 0x7f45bc06d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45c0a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc163800 0x7f45bc0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45c0a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc15cf70 con 0x7f45bc06e060 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45c0a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc0b3890 con 0x7f45bc15cb90 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45c0a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc06c990 con 0x7f45bc163800 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45bc15cb90 0x7f45bc06d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45bc15cb90 0x7f45bc06d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51014/0 (socket says 172.21.15.114:51014) 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b9e00640 1 -- 172.21.15.114:0/3079022674 learned_addr learned my addr 172.21.15.114:0/3079022674 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b1400640 1 --2- 172.21.15.114:0/3079022674 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc163800 0x7f45bc0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b9e00640 1 -- 172.21.15.114:0/3079022674 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc163800 msgr2=0x7f45bc0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b9e00640 1 --2- 172.21.15.114:0/3079022674 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc163800 0x7f45bc0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b9e00640 1 -- 172.21.15.114:0/3079022674 >> v1:172.21.15.177:6789/0 conn(0x7f45bc06e060 legacy=0x7f45bc0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.481+0000 7f45b9e00640 1 -- 172.21.15.114:0/3079022674 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc0b35b0 con 0x7f45bc15cb90 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.482+0000 7f45b1400640 1 --2- 172.21.15.114:0/3079022674 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc163800 0x7f45bc0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:10.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.483+0000 7f45b9e00640 1 --2- 172.21.15.114:0/3079022674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc06d6c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009e50 tx=0x7f45ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a03649d018efdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.483+0000 7f45b9400640 1 -- 172.21.15.114:0/3079022674 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac02ca70 con 0x7f45bc15cb90 2024-03-06T07:32:10.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.483+0000 7f45b9400640 1 -- 172.21.15.114:0/3079022674 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac02cc10 con 0x7f45bc15cb90 2024-03-06T07:32:10.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.483+0000 7f45b9400640 1 -- 172.21.15.114:0/3079022674 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0343f0 con 0x7f45bc15cb90 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 -- 172.21.15.114:0/3079022674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 msgr2=0x7f45bc06d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 --2- 172.21.15.114:0/3079022674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc06d6c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009e50 tx=0x7f45ac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 -- 172.21.15.114:0/3079022674 shutdown_connections 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 --2- 172.21.15.114:0/3079022674 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc163800 0x7f45bc0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 --2- 172.21.15.114:0/3079022674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc06d6c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 -- 172.21.15.114:0/3079022674 >> 172.21.15.114:0/3079022674 conn(0x7f45bc0b1940 msgr2=0x7f45bc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 -- 172.21.15.114:0/3079022674 shutdown_connections 2024-03-06T07:32:10.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.484+0000 7f45c0a00640 1 -- 172.21.15.114:0/3079022674 wait complete. 2024-03-06T07:32:10.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.486+0000 7f45c0a00640 1 Processor -- start 2024-03-06T07:32:10.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.486+0000 7f45c0a00640 1 -- start start 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45c0a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc06e060 0x7f45bc15b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45c0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc164f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45c0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45bc163800 0x7f45bc09c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45c0a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc06cec0 con 0x7f45bc15cb90 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45c0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc06d230 con 0x7f45bc163800 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45c0a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc06dc00 con 0x7f45bc06e060 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc164f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc164f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51026/0 (socket says 172.21.15.114:51026) 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 -- 172.21.15.114:0/739476339 learned_addr learned my addr 172.21.15.114:0/739476339 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b9e00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc06e060 0x7f45bc15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45ba800640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45bc163800 0x7f45bc09c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 -- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc06e060 msgr2=0x7f45bc15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc06e060 0x7f45bc15b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 -- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45bc163800 msgr2=0x7f45bc09c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45bc163800 0x7f45bc09c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.487+0000 7f45b1400640 1 -- 172.21.15.114:0/739476339 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc1ed9c0 con 0x7f45bc15cb90 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.488+0000 7f45b1400640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc164f10 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c7f0 tx=0x7f45a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.488+0000 7f45b2a00640 1 -- 172.21.15.114:0/739476339 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4015070 con 0x7f45bc15cb90 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.488+0000 7f45b2a00640 1 -- 172.21.15.114:0/739476339 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a40077c0 con 0x7f45bc15cb90 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.488+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc1edc60 con 0x7f45bc15cb90 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.488+0000 7f45b2a00640 1 -- 172.21.15.114:0/739476339 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4007b70 con 0x7f45bc15cb90 2024-03-06T07:32:10.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.488+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc1ee220 con 0x7f45bc15cb90 2024-03-06T07:32:10.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.490+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc04f330 con 0x7f45bc15cb90 2024-03-06T07:32:10.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.490+0000 7f45b2a00640 1 -- 172.21.15.114:0/739476339 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45a4018070 con 0x7f45bc15cb90 2024-03-06T07:32:10.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.492+0000 7f45b2a00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459806b4d0 0x7f459806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.492+0000 7f45b2a00640 1 -- 172.21.15.114:0/739476339 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f45a4089fb0 con 0x7f45bc15cb90 2024-03-06T07:32:10.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.492+0000 7f45b9e00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459806b4d0 0x7f459806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.493+0000 7f45b9e00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459806b4d0 0x7f459806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f45ac035000 tx=0x7f45ac030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.497+0000 7f45b2a00640 1 -- 172.21.15.114:0/739476339 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45a4059130 con 0x7f45bc15cb90 2024-03-06T07:32:10.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.692+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f45bc0b2530 con 0x7f45bc15cb90 2024-03-06T07:32:10.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.698+0000 7f45b2a00640 1 -- 172.21.15.114:0/739476339 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f45a4058a30 con 0x7f45bc15cb90 2024-03-06T07:32:10.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.700+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459806b4d0 msgr2=0x7f459806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.700+0000 7f45c0a00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459806b4d0 0x7f459806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f45ac035000 tx=0x7f45ac030000 comp rx=0 tx=0).stop 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.700+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 msgr2=0x7f45bc164f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.700+0000 7f45c0a00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc164f10 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c7f0 tx=0x7f45a400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 shutdown_connections 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45bc163800 0x7f45bc09c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f459806b4d0 0x7f459806d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45bc15cb90 0x7f45bc164f10 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 --2- 172.21.15.114:0/739476339 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45bc06e060 0x7f45bc15b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 >> 172.21.15.114:0/739476339 conn(0x7f45bc0b1940 msgr2=0x7f45bc0cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 shutdown_connections 2024-03-06T07:32:10.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:10.701+0000 7f45c0a00640 1 -- 172.21.15.114:0/739476339 wait complete. 2024-03-06T07:32:10.721 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:10.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:10.823 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:10.948 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:10.948 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:10.948 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:11.005 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:11.005 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:32:11.005 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:11.060 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:11.060 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:11.060 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:11.118 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:11.118 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:32:11.118 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:11.172 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:11.172 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:32:11.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:11.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.384+0000 7f9a6de00640 1 Processor -- start 2024-03-06T07:32:11.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.384+0000 7f9a6de00640 1 -- start start 2024-03-06T07:32:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a6de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 0x7f9a6815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a6de00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9a68070b10 0x7f9a68070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a6de00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6806cef0 con 0x7f9a68144c00 2024-03-06T07:32:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a6de00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a68160370 con 0x7f9a68070b10 2024-03-06T07:32:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a6de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6806d1d0 con 0x7f9a6815f9d0 2024-03-06T07:32:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a66a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9a68144c00 0x7f9a6806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37448/0 (socket says 172.21.15.114:37448) 2024-03-06T07:32:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a66a00640 1 -- 172.21.15.114:0/1537736631 learned_addr learned my addr 172.21.15.114:0/1537736631 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a67400640 1 --2- 172.21.15.114:0/1537736631 >> v2:172.21.15.177:3300/0 conn(0x7f9a68070b10 0x7f9a68070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064241994 0 0) 0x7f9a6806cef0 con 0x7f9a68144c00 2024-03-06T07:32:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a66000640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 0x7f9a6815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.386+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c003520 con 0x7f9a68144c00 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745357252 0 0) 0x7f9a4c003520 con 0x7f9a68144c00 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 msgr2=0x7f9a6815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 0x7f9a6815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 >> v2:172.21.15.177:3300/0 conn(0x7f9a68070b10 msgr2=0x7f9a68070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 --2- 172.21.15.114:0/1537736631 >> v2:172.21.15.177:3300/0 conn(0x7f9a68070b10 0x7f9a68070f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6806d4b0 con 0x7f9a68144c00 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f9a58001670 con 0x7f9a68144c00 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 >> v1:172.21.15.114:6789/0 conn(0x7f9a68144c00 legacy=0x7f9a6806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0039b0 0x7f9a4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a4c004800 0x7f9a4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 0x7f9a4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6806d1d0 con 0x7f9a4c0039b0 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c00b9c0 con 0x7f9a4c004800 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c00bb40 con 0x7f9a6815f9d0 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.387+0000 7f9a66a00640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0039b0 0x7f9a4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.388+0000 7f9a67400640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 0x7f9a4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.388+0000 7f9a66a00640 1 -- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 msgr2=0x7f9a4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.388+0000 7f9a66a00640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a6815f9d0 0x7f9a4c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.388+0000 7f9a66a00640 1 -- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a4c004800 msgr2=0x7f9a4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.388+0000 7f9a66a00640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a4c004800 0x7f9a4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.388+0000 7f9a66a00640 1 -- 172.21.15.114:0/1537736631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c003520 con 0x7f9a4c0039b0 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.389+0000 7f9a66a00640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0039b0 0x7f9a4c003dd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c190 tx=0x7f9a4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2305ed5a282c513d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.389+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58007c80 con 0x7f9a4c0039b0 2024-03-06T07:32:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.389+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58030070 con 0x7f9a4c0039b0 2024-03-06T07:32:11.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.389+0000 7f9a65600640 1 -- 172.21.15.114:0/1537736631 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58005cf0 con 0x7f9a4c0039b0 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.389+0000 7f9a6de00640 1 -- 172.21.15.114:0/1537736631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0039b0 msgr2=0x7f9a4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.389+0000 7f9a6de00640 1 --2- 172.21.15.114:0/1537736631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0039b0 0x7f9a4c003dd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c190 tx=0x7f9a4c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.390+0000 7f9a67400640 1 -- 172.21.15.114:0/1537736631 reap_dead start 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.390+0000 7f9a6de00640 1 -- 172.21.15.114:0/1537736631 shutdown_connections 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.390+0000 7f9a6de00640 1 -- 172.21.15.114:0/1537736631 >> 172.21.15.114:0/1537736631 conn(0x7f9a680b1730 msgr2=0x7f9a680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.390+0000 7f9a6de00640 1 -- 172.21.15.114:0/1537736631 shutdown_connections 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.390+0000 7f9a6de00640 1 -- 172.21.15.114:0/1537736631 wait complete. 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.391+0000 7f9a6de00640 1 Processor -- start 2024-03-06T07:32:11.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.392+0000 7f9a6de00640 1 -- start start 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.392+0000 7f9a6de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a4c0039b0 0x7f9a680c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.392+0000 7f9a6de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a681f4a60 0x7f9a6815e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.392+0000 7f9a6de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a680c7300 0x7f9a680c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.392+0000 7f9a6de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6806cb50 con 0x7f9a680c7300 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.392+0000 7f9a6de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68160730 con 0x7f9a681f4a60 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.392+0000 7f9a6de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6806ccd0 con 0x7f9a4c0039b0 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a680c7300 0x7f9a680c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a680c7300 0x7f9a680c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51050/0 (socket says 172.21.15.114:51050) 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 -- 172.21.15.114:0/490963435 learned_addr learned my addr 172.21.15.114:0/490963435 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a66a00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a4c0039b0 0x7f9a680c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a66000640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a681f4a60 0x7f9a6815e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 -- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a4c0039b0 msgr2=0x7f9a680c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a4c0039b0 0x7f9a680c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 -- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a681f4a60 msgr2=0x7f9a6815e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a681f4a60 0x7f9a6815e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.393+0000 7f9a67400640 1 -- 172.21.15.114:0/490963435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a680c5020 con 0x7f9a680c7300 2024-03-06T07:32:11.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.394+0000 7f9a67400640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a680c7300 0x7f9a680c77b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9a6806e500 tx=0x7f9a6000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.394+0000 7f9a5f400640 1 -- 172.21.15.114:0/490963435 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6001c070 con 0x7f9a680c7300 2024-03-06T07:32:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.394+0000 7f9a5f400640 1 -- 172.21.15.114:0/490963435 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60017400 con 0x7f9a680c7300 2024-03-06T07:32:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.394+0000 7f9a5f400640 1 -- 172.21.15.114:0/490963435 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a600177f0 con 0x7f9a680c7300 2024-03-06T07:32:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.394+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a681f4f60 con 0x7f9a680c7300 2024-03-06T07:32:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.394+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a681f5520 con 0x7f9a680c7300 2024-03-06T07:32:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.395+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6804f330 con 0x7f9a680c7300 2024-03-06T07:32:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.396+0000 7f9a5f400640 1 -- 172.21.15.114:0/490963435 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a60005d10 con 0x7f9a680c7300 2024-03-06T07:32:11.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.397+0000 7f9a5f400640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a4806b500 0x7f9a4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.397+0000 7f9a5f400640 1 -- 172.21.15.114:0/490963435 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9a60089ff0 con 0x7f9a680c7300 2024-03-06T07:32:11.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.397+0000 7f9a66a00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a4806b500 0x7f9a4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.398+0000 7f9a66a00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a4806b500 0x7f9a4806d9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9a58034000 tx=0x7f9a58007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.402+0000 7f9a5f400640 1 -- 172.21.15.114:0/490963435 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a60012070 con 0x7f9a680c7300 2024-03-06T07:32:11.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.634+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6815be20 con 0x7f9a680c7300 2024-03-06T07:32:11.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.635+0000 7f9a5f400640 1 -- 172.21.15.114:0/490963435 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f9a60059110 con 0x7f9a680c7300 2024-03-06T07:32:11.638 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:11.638 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5338,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2397672662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2397672662},{"type":"v1","addr":"172.21.15.177:6833","nonce":2397672662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5341,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/165925173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":165925173},{"type":"v1","addr":"172.21.15.177:6835","nonce":165925173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6351,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2541907818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2541907818},{"type":"v1","addr":"172.21.15.114:6835","nonce":2541907818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/692030311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":692030311},{"type":"v1","addr":"172.21.15.114:6837","nonce":692030311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[]} 2024-03-06T07:32:11.639 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 46 2024-03-06T07:32:11.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a4806b500 msgr2=0x7f9a4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a4806b500 0x7f9a4806d9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9a58034000 tx=0x7f9a58007000 comp rx=0 tx=0).stop 2024-03-06T07:32:11.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a680c7300 msgr2=0x7f9a680c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a680c7300 0x7f9a680c77b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9a6806e500 tx=0x7f9a6000f4a0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 shutdown_connections 2024-03-06T07:32:11.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9a4806b500 0x7f9a4806d9c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a680c7300 0x7f9a680c77b0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a681f4a60 0x7f9a6815e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 --2- 172.21.15.114:0/490963435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9a4c0039b0 0x7f9a680c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 >> 172.21.15.114:0/490963435 conn(0x7f9a680b1730 msgr2=0x7f9a680adf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 shutdown_connections 2024-03-06T07:32:11.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.638+0000 7f9a6de00640 1 -- 172.21.15.114:0/490963435 wait complete. 2024-03-06T07:32:11.660 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:11.660 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:32:11.660 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:11.660 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:11.660 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:11.661 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:32:11.661 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:11.661 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:11.661 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:11.661 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:32:11.661 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:11.661 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:11.662 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:32:11.662 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:32:11.662 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:11.662 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:11.677 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:11.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:11.678 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:11.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:11.681 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:11.681 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:11.682 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:11.683 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:32:11.725 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:11.725 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:11.729 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:11.730 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:11.730 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:11.731 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:11.736 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:32:11.736 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:11.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:32:11.877 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:32:11.879 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:32:11.893 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:32:11.896 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:32:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.918+0000 7f4990a00640 1 Processor -- start 2024-03-06T07:32:11.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.918+0000 7f4990a00640 1 -- start start 2024-03-06T07:32:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4990a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f498c15dfc0 0x7f498c06d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4990a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f498c161cd0 0x7f498c0b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4990a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c0cf090 0x7f498c0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4990a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c0ad860 con 0x7f498c15dfc0 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4990a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f498c1621b0 con 0x7f498c161cd0 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4990a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c162490 con 0x7f498c0cf090 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4989e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f498c15dfc0 0x7f498c06d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4989e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f498c15dfc0 0x7f498c06d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51076/0 (socket says 172.21.15.114:51076) 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4989e00640 1 -- 172.21.15.114:0/1159517444 learned_addr learned my addr 172.21.15.114:0/1159517444 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f4989400640 1 --2- 172.21.15.114:0/1159517444 >> v2:172.21.15.177:3300/0 conn(0x7f498c161cd0 0x7f498c0b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.919+0000 7f498a800640 1 --2- 172.21.15.114:0/1159517444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c0cf090 0x7f498c0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.920+0000 7f4989e00640 1 -- 172.21.15.114:0/1159517444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c0cf090 msgr2=0x7f498c0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.920+0000 7f4989e00640 1 --2- 172.21.15.114:0/1159517444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c0cf090 0x7f498c0b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.920+0000 7f4989e00640 1 -- 172.21.15.114:0/1159517444 >> v2:172.21.15.177:3300/0 conn(0x7f498c161cd0 msgr2=0x7f498c0b1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.920+0000 7f4989e00640 1 --2- 172.21.15.114:0/1159517444 >> v2:172.21.15.177:3300/0 conn(0x7f498c161cd0 0x7f498c0b1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.920+0000 7f4989e00640 1 -- 172.21.15.114:0/1159517444 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c162770 con 0x7f498c15dfc0 2024-03-06T07:32:11.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.920+0000 7f4989400640 1 --2- 172.21.15.114:0/1159517444 >> v2:172.21.15.177:3300/0 conn(0x7f498c161cd0 0x7f498c0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:11.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.921+0000 7f4989e00640 1 --2- 172.21.15.114:0/1159517444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c15dfc0 0x7f498c06d070 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f498c0aeed0 tx=0x7f497c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1229c890a1b63f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.921+0000 7f4988a00640 1 -- 172.21.15.114:0/1159517444 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c035020 con 0x7f498c15dfc0 2024-03-06T07:32:11.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.921+0000 7f4988a00640 1 -- 172.21.15.114:0/1159517444 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c02d040 con 0x7f498c15dfc0 2024-03-06T07:32:11.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.921+0000 7f4988a00640 1 -- 172.21.15.114:0/1159517444 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c031a80 con 0x7f498c15dfc0 2024-03-06T07:32:11.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.921+0000 7f4990a00640 1 -- 172.21.15.114:0/1159517444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c15dfc0 msgr2=0x7f498c06d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.921+0000 7f4990a00640 1 --2- 172.21.15.114:0/1159517444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c15dfc0 0x7f498c06d070 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f498c0aeed0 tx=0x7f497c02b900 comp rx=0 tx=0).stop 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.922+0000 7f4990a00640 1 -- 172.21.15.114:0/1159517444 shutdown_connections 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.922+0000 7f4990a00640 1 --2- 172.21.15.114:0/1159517444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c0cf090 0x7f498c0b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.922+0000 7f4990a00640 1 --2- 172.21.15.114:0/1159517444 >> v2:172.21.15.177:3300/0 conn(0x7f498c161cd0 0x7f498c0b1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.922+0000 7f4990a00640 1 --2- 172.21.15.114:0/1159517444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c15dfc0 0x7f498c06d070 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.922+0000 7f4990a00640 1 -- 172.21.15.114:0/1159517444 >> 172.21.15.114:0/1159517444 conn(0x7f498c0cfa70 msgr2=0x7f498c0cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.922+0000 7f4990a00640 1 -- 172.21.15.114:0/1159517444 shutdown_connections 2024-03-06T07:32:11.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.922+0000 7f4990a00640 1 -- 172.21.15.114:0/1159517444 wait complete. 2024-03-06T07:32:11.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.923+0000 7f4990a00640 1 Processor -- start 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.924+0000 7f4990a00640 1 -- start start 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.924+0000 7f4990a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c0cf090 0x7f498c15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.924+0000 7f4990a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c15dfc0 0x7f498c146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4990a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c161cd0 0x7f498c1f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4990a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c06ccc0 con 0x7f498c0cf090 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4990a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c06d5b0 con 0x7f498c161cd0 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4990a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498c06ce40 con 0x7f498c15dfc0 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c0cf090 0x7f498c15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c0cf090 0x7f498c15adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51088/0 (socket says 172.21.15.114:51088) 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c15dfc0 0x7f498c146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 -- 172.21.15.114:0/2809613375 learned_addr learned my addr 172.21.15.114:0/2809613375 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f498a800640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c161cd0 0x7f498c1f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 -- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c15dfc0 msgr2=0x7f498c146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c15dfc0 0x7f498c146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 -- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c161cd0 msgr2=0x7f498c1f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c161cd0 0x7f498c1f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f4989e00640 1 -- 172.21.15.114:0/2809613375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c159ff0 con 0x7f498c0cf090 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.925+0000 7f498a800640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c161cd0 0x7f498c1f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.926+0000 7f4989e00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c0cf090 0x7f498c15adb0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f497c02b940 tx=0x7f497c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.926+0000 7f4982000640 1 -- 172.21.15.114:0/2809613375 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c040070 con 0x7f498c0cf090 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.926+0000 7f4982000640 1 -- 172.21.15.114:0/2809613375 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c0399f0 con 0x7f498c0cf090 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.926+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c15a2d0 con 0x7f498c0cf090 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.926+0000 7f4982000640 1 -- 172.21.15.114:0/2809613375 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c035040 con 0x7f498c0cf090 2024-03-06T07:32:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.926+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c15a890 con 0x7f498c0cf090 2024-03-06T07:32:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.927+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c04f330 con 0x7f498c0cf090 2024-03-06T07:32:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.928+0000 7f4982000640 1 -- 172.21.15.114:0/2809613375 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f497c038020 con 0x7f498c0cf090 2024-03-06T07:32:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.929+0000 7f4982000640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f496c06b4d0 0x7f496c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.929+0000 7f4982000640 1 -- 172.21.15.114:0/2809613375 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f497c0ab9b0 con 0x7f498c0cf090 2024-03-06T07:32:11.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.929+0000 7f4989400640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f496c06b4d0 0x7f496c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.930+0000 7f4989400640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f496c06b4d0 0x7f496c06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4978006fd0 tx=0x7f4978008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:11.933+0000 7f4982000640 1 -- 172.21.15.114:0/2809613375 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f497c07a440 con 0x7f498c0cf090 2024-03-06T07:32:12.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:12.139+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f498c0cf7e0 con 0x7f498c0cf090 2024-03-06T07:32:13.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.274+0000 7f4982000640 1 -- 172.21.15.114:0/2809613375 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v76) v1 ==== 97+0+0 (secure 0 0 0) 0x7f497c07e4e0 con 0x7f498c0cf090 2024-03-06T07:32:13.277 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:32:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f496c06b4d0 msgr2=0x7f496c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f496c06b4d0 0x7f496c06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4978006fd0 tx=0x7f4978008040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c0cf090 msgr2=0x7f498c15adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c0cf090 0x7f498c15adb0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f497c02b940 tx=0x7f497c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 shutdown_connections 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f498c161cd0 0x7f498c1f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f498c15dfc0 0x7f498c146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f496c06b4d0 0x7f496c06d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 --2- 172.21.15.114:0/2809613375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f498c0cf090 0x7f498c15adb0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.278+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 >> 172.21.15.114:0/2809613375 conn(0x7f498c0cfa70 msgr2=0x7f498c0cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.279+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 shutdown_connections 2024-03-06T07:32:13.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.279+0000 7f4990a00640 1 -- 172.21.15.114:0/2809613375 wait complete. 2024-03-06T07:32:13.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:32:13.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.507+0000 7f8b9fe00640 1 Processor -- start 2024-03-06T07:32:13.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.507+0000 7f8b9fe00640 1 -- start start 2024-03-06T07:32:13.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.508+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b9815f9d0 0x7f8b9815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.508+0000 7f8b9fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b98070b10 0x7f8b98070f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.508+0000 7f8b9fe00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9806cef0 con 0x7f8b98144c00 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.508+0000 7f8b9fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b98160370 con 0x7f8b98070b10 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.508+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9806d1d0 con 0x7f8b9815f9d0 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b98070b10 0x7f8b98070f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b98070b10 0x7f8b98070f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51090/0 (socket says 172.21.15.114:51090) 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9de00640 1 -- 172.21.15.114:0/432161609 learned_addr learned my addr 172.21.15.114:0/432161609 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9ca00640 1 --2- 172.21.15.114:0/432161609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b9815f9d0 0x7f8b9815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9de00640 1 -- 172.21.15.114:0/432161609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b9815f9d0 msgr2=0x7f8b9815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9de00640 1 --2- 172.21.15.114:0/432161609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b9815f9d0 0x7f8b9815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9de00640 1 -- 172.21.15.114:0/432161609 >> v1:172.21.15.177:6789/0 conn(0x7f8b98144c00 legacy=0x7f8b9806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.509+0000 7f8b9de00640 1 -- 172.21.15.114:0/432161609 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9806d4b0 con 0x7f8b98070b10 2024-03-06T07:32:13.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.511+0000 7f8b9de00640 1 --2- 172.21.15.114:0/432161609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b98070f60 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009e50 tx=0x7f8b9002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4e26c1c06de169 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.511+0000 7f8b97e00640 1 -- 172.21.15.114:0/432161609 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9002caf0 con 0x7f8b98070b10 2024-03-06T07:32:13.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.511+0000 7f8b97e00640 1 -- 172.21.15.114:0/432161609 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9002cc90 con 0x7f8b98070b10 2024-03-06T07:32:13.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.511+0000 7f8b97e00640 1 -- 172.21.15.114:0/432161609 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90034610 con 0x7f8b98070b10 2024-03-06T07:32:13.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 -- 172.21.15.114:0/432161609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 msgr2=0x7f8b98070f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/432161609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b98070f60 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009e50 tx=0x7f8b9002bf50 comp rx=0 tx=0).stop 2024-03-06T07:32:13.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 -- 172.21.15.114:0/432161609 shutdown_connections 2024-03-06T07:32:13.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/432161609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b98070f60 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/432161609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b9815f9d0 0x7f8b9815fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 -- 172.21.15.114:0/432161609 >> 172.21.15.114:0/432161609 conn(0x7f8b980b1730 msgr2=0x7f8b980b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 -- 172.21.15.114:0/432161609 shutdown_connections 2024-03-06T07:32:13.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.512+0000 7f8b9fe00640 1 -- 172.21.15.114:0/432161609 wait complete. 2024-03-06T07:32:13.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.513+0000 7f8b9fe00640 1 Processor -- start 2024-03-06T07:32:13.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.514+0000 7f8b9fe00640 1 -- start start 2024-03-06T07:32:13.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.514+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b9815b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.514+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b98144c00 0x7f8b98140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9815f9d0 0x7f8b98143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9806cb50 con 0x7f8b98070b10 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98160730 con 0x7f8b9815f9d0 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9806ccd0 con 0x7f8b98144c00 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b9815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b9815b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51094/0 (socket says 172.21.15.114:51094) 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 -- 172.21.15.114:0/1019387168 learned_addr learned my addr 172.21.15.114:0/1019387168 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9ca00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b98144c00 0x7f8b98140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9de00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9815f9d0 0x7f8b98143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 -- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b98144c00 msgr2=0x7f8b98140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b98144c00 0x7f8b98140740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 -- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9815f9d0 msgr2=0x7f8b98143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9815f9d0 0x7f8b98143050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 -- 172.21.15.114:0/1019387168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b981e8d30 con 0x7f8b98070b10 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9ca00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b98144c00 0x7f8b98140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.515+0000 7f8b9d400640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b9815b220 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00c400 tx=0x7f8b8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.516+0000 7f8b95600640 1 -- 172.21.15.114:0/1019387168 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0052d0 con 0x7f8b98070b10 2024-03-06T07:32:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.516+0000 7f8b95600640 1 -- 172.21.15.114:0/1019387168 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c005490 con 0x7f8b98070b10 2024-03-06T07:32:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.516+0000 7f8b95600640 1 -- 172.21.15.114:0/1019387168 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c007260 con 0x7f8b98070b10 2024-03-06T07:32:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.516+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b981e9010 con 0x7f8b98070b10 2024-03-06T07:32:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.516+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b981e95d0 con 0x7f8b98070b10 2024-03-06T07:32:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.517+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b981e9810 con 0x7f8b98070b10 2024-03-06T07:32:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.518+0000 7f8b95600640 1 -- 172.21.15.114:0/1019387168 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b8c007400 con 0x7f8b98070b10 2024-03-06T07:32:13.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.519+0000 7f8b95600640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b7806b4d0 0x7f8b7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.519+0000 7f8b95600640 1 -- 172.21.15.114:0/1019387168 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8b8c089370 con 0x7f8b98070b10 2024-03-06T07:32:13.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.519+0000 7f8b9ca00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b7806b4d0 0x7f8b7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.520+0000 7f8b9ca00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b7806b4d0 0x7f8b7806d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8b98142cf0 tx=0x7f8b84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.523+0000 7f8b95600640 1 -- 172.21.15.114:0/1019387168 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b981e9810 con 0x7f8b98070b10 2024-03-06T07:32:13.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.787+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8b9804f330 con 0x7f8b98070b10 2024-03-06T07:32:13.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.788+0000 7f8b95600640 1 -- 172.21.15.114:0/1019387168 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8b8c0585d0 con 0x7f8b98070b10 2024-03-06T07:32:13.791 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.793 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.795 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.796 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:32:13.798 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:32:13.799 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow r", 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:32:13.800 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:32:13.801 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.802 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:13.803 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:32:13.804 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:32:13.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b7806b4d0 msgr2=0x7f8b7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b7806b4d0 0x7f8b7806d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8b98142cf0 tx=0x7f8b84008040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 msgr2=0x7f8b9815b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b9815b220 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00c400 tx=0x7f8b8c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 shutdown_connections 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9815f9d0 0x7f8b98143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b98144c00 0x7f8b98140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b7806b4d0 0x7f8b7806d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 --2- 172.21.15.114:0/1019387168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b98070b10 0x7f8b9815b220 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 >> 172.21.15.114:0/1019387168 conn(0x7f8b980b1730 msgr2=0x7f8b980ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.791+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 shutdown_connections 2024-03-06T07:32:13.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:13.792+0000 7f8b9fe00640 1 -- 172.21.15.114:0/1019387168 wait complete. 2024-03-06T07:32:13.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:14.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.008+0000 7f6d17600640 1 Processor -- start 2024-03-06T07:32:14.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.008+0000 7f6d17600640 1 -- start start 2024-03-06T07:32:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d17600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6d10144c00 0x7f6d1006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d17600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d1015f9d0 0x7f6d1015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d17600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1006ee70 con 0x7f6d100ce0a0 2024-03-06T07:32:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d17600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d10160370 con 0x7f6d10144c00 2024-03-06T07:32:14.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d17600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1006cef0 con 0x7f6d1015f9d0 2024-03-06T07:32:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d1015f9d0 0x7f6d1015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d14c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6d10144c00 0x7f6d1006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d14c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6d10144c00 0x7f6d1006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:54012/0 (socket says 172.21.15.114:54012) 2024-03-06T07:32:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.010+0000 7f6d14c00640 1 -- 172.21.15.114:0/2867020954 learned_addr learned my addr 172.21.15.114:0/2867020954 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.011+0000 7f6d0fe00640 1 -- 172.21.15.114:0/2867020954 >> v2:172.21.15.177:3300/0 conn(0x7f6d10144c00 msgr2=0x7f6d1006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.011+0000 7f6d0fe00640 1 --2- 172.21.15.114:0/2867020954 >> v2:172.21.15.177:3300/0 conn(0x7f6d10144c00 0x7f6d1006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.011+0000 7f6d0fe00640 1 -- 172.21.15.114:0/2867020954 >> v1:172.21.15.177:6789/0 conn(0x7f6d100ce0a0 legacy=0x7f6d100ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.011+0000 7f6d0fe00640 1 -- 172.21.15.114:0/2867020954 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1006d1d0 con 0x7f6d1015f9d0 2024-03-06T07:32:14.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.011+0000 7f6d14c00640 1 --2- 172.21.15.114:0/2867020954 >> v2:172.21.15.177:3300/0 conn(0x7f6d10144c00 0x7f6d1006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:14.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.012+0000 7f6d0fe00640 1 --2- 172.21.15.114:0/2867020954 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d1015f9d0 0x7f6d1015fe30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0097a0 tx=0x7f6cfc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=556998eab9c7d324 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.013+0000 7f6d0f400640 1 -- 172.21.15.114:0/2867020954 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc035020 con 0x7f6d1015f9d0 2024-03-06T07:32:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.013+0000 7f6d0f400640 1 -- 172.21.15.114:0/2867020954 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc031830 con 0x7f6d1015f9d0 2024-03-06T07:32:14.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.013+0000 7f6d0f400640 1 -- 172.21.15.114:0/2867020954 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc031c00 con 0x7f6d1015f9d0 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 -- 172.21.15.114:0/2867020954 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d1015f9d0 msgr2=0x7f6d1015fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 --2- 172.21.15.114:0/2867020954 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d1015f9d0 0x7f6d1015fe30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0097a0 tx=0x7f6cfc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 -- 172.21.15.114:0/2867020954 shutdown_connections 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 --2- 172.21.15.114:0/2867020954 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d1015f9d0 0x7f6d1015fe30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 --2- 172.21.15.114:0/2867020954 >> v2:172.21.15.177:3300/0 conn(0x7f6d10144c00 0x7f6d1006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 -- 172.21.15.114:0/2867020954 >> 172.21.15.114:0/2867020954 conn(0x7f6d100b1730 msgr2=0x7f6d100b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 -- 172.21.15.114:0/2867020954 shutdown_connections 2024-03-06T07:32:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.014+0000 7f6d17600640 1 -- 172.21.15.114:0/2867020954 wait complete. 2024-03-06T07:32:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.016+0000 7f6d17600640 1 Processor -- start 2024-03-06T07:32:14.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.016+0000 7f6d17600640 1 -- start start 2024-03-06T07:32:14.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.016+0000 7f6d17600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d100ce0a0 0x7f6d1013d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.016+0000 7f6d17600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d10144c00 0x7f6d1013db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d17600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d1015f9d0 0x7f6d1013e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d17600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d10160730 con 0x7f6d10144c00 2024-03-06T07:32:14.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d17600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1006d790 con 0x7f6d1015f9d0 2024-03-06T07:32:14.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d17600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1006cb50 con 0x7f6d100ce0a0 2024-03-06T07:32:14.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d10144c00 0x7f6d1013db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d10144c00 0x7f6d1013db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51098/0 (socket says 172.21.15.114:51098) 2024-03-06T07:32:14.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 -- 172.21.15.114:0/3049997097 learned_addr learned my addr 172.21.15.114:0/3049997097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d14c00640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d100ce0a0 0x7f6d1013d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d15600640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d1015f9d0 0x7f6d1013e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 -- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d100ce0a0 msgr2=0x7f6d1013d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d100ce0a0 0x7f6d1013d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 -- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d1015f9d0 msgr2=0x7f6d1013e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d1015f9d0 0x7f6d1013e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.017+0000 7f6d0fe00640 1 -- 172.21.15.114:0/3049997097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10141ed0 con 0x7f6d10144c00 2024-03-06T07:32:14.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.018+0000 7f6d0fe00640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d10144c00 0x7f6d1013db00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc02bee0 tx=0x7f6cfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.018+0000 7f6d0cc00640 1 -- 172.21.15.114:0/3049997097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc041070 con 0x7f6d10144c00 2024-03-06T07:32:14.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.018+0000 7f6d0cc00640 1 -- 172.21.15.114:0/3049997097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc03f070 con 0x7f6d10144c00 2024-03-06T07:32:14.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.018+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d101421b0 con 0x7f6d10144c00 2024-03-06T07:32:14.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.018+0000 7f6d0cc00640 1 -- 172.21.15.114:0/3049997097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc035040 con 0x7f6d10144c00 2024-03-06T07:32:14.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.018+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d10142770 con 0x7f6d10144c00 2024-03-06T07:32:14.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.019+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d100ae870 con 0x7f6d10144c00 2024-03-06T07:32:14.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.020+0000 7f6d0cc00640 1 -- 172.21.15.114:0/3049997097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6cfc038020 con 0x7f6d10144c00 2024-03-06T07:32:14.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.021+0000 7f6d0cc00640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6cf006b4d0 0x7f6cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.021+0000 7f6d0cc00640 1 -- 172.21.15.114:0/3049997097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6cfc0ab9b0 con 0x7f6d10144c00 2024-03-06T07:32:14.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.021+0000 7f6d14c00640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6cf006b4d0 0x7f6cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.022+0000 7f6d14c00640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6cf006b4d0 0x7f6cf006d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009b70 tx=0x7f6d04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.027+0000 7f6d0cc00640 1 -- 172.21.15.114:0/3049997097 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d100ae870 con 0x7f6d10144c00 2024-03-06T07:32:14.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.222+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d10072890 con 0x7f6d10144c00 2024-03-06T07:32:14.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.224+0000 7f6d0cc00640 1 -- 172.21.15.114:0/3049997097 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+12940 (secure 0 0 0) 0x7f6cfc031800 con 0x7f6d10144c00 2024-03-06T07:32:14.227 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:14.227 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":76,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:13.272140+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:14.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6cf006b4d0 msgr2=0x7f6cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6cf006b4d0 0x7f6cf006d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009b70 tx=0x7f6d04008040 comp rx=0 tx=0).stop 2024-03-06T07:32:14.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d10144c00 msgr2=0x7f6d1013db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d10144c00 0x7f6d1013db00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc02bee0 tx=0x7f6cfc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 shutdown_connections 2024-03-06T07:32:14.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d1015f9d0 0x7f6d1013e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6cf006b4d0 0x7f6cf006d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d10144c00 0x7f6d1013db00 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 --2- 172.21.15.114:0/3049997097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6d100ce0a0 0x7f6d1013d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 >> 172.21.15.114:0/3049997097 conn(0x7f6d100b1730 msgr2=0x7f6d100ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 shutdown_connections 2024-03-06T07:32:14.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.227+0000 7f6d17600640 1 -- 172.21.15.114:0/3049997097 wait complete. 2024-03-06T07:32:14.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:14.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:14.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.464+0000 7fa15d800640 1 Processor -- start 2024-03-06T07:32:14.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.464+0000 7fa15d800640 1 -- start start 2024-03-06T07:32:14.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa15d800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa15815e000 0x7fa1580ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa15d800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1580d1b90 0x7fa1580d39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa15d800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa1580d3ee0 0x7fa1580d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa15d800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580cedd0 con 0x7fa1580d1b90 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa15d800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580d4370 con 0x7fa1580d3ee0 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa15d800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580cf020 con 0x7fa15815e000 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa14fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1580d1b90 0x7fa1580d39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa14fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1580d1b90 0x7fa1580d39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51114/0 (socket says 172.21.15.114:51114) 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.465+0000 7fa14fe00640 1 -- 172.21.15.114:0/2938867772 learned_addr learned my addr 172.21.15.114:0/2938867772 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.466+0000 7fa157e00640 1 --2- 172.21.15.114:0/2938867772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa15815e000 0x7fa1580ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.466+0000 7fa15ce00640 1 --2- 172.21.15.114:0/2938867772 >> v2:172.21.15.177:3300/0 conn(0x7fa1580d3ee0 0x7fa1580d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.466+0000 7fa14fe00640 1 -- 172.21.15.114:0/2938867772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa15815e000 msgr2=0x7fa1580ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.466+0000 7fa14fe00640 1 --2- 172.21.15.114:0/2938867772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa15815e000 0x7fa1580ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.466+0000 7fa14fe00640 1 -- 172.21.15.114:0/2938867772 >> v2:172.21.15.177:3300/0 conn(0x7fa1580d3ee0 msgr2=0x7fa1580d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.466+0000 7fa14fe00640 1 --2- 172.21.15.114:0/2938867772 >> v2:172.21.15.177:3300/0 conn(0x7fa1580d3ee0 0x7fa1580d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.466+0000 7fa14fe00640 1 -- 172.21.15.114:0/2938867772 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580cf300 con 0x7fa1580d1b90 2024-03-06T07:32:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.467+0000 7fa14fe00640 1 --2- 172.21.15.114:0/2938867772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d1b90 0x7fa1580d39a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa15808dc80 tx=0x7fa14002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca70622406cd31c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.467+0000 7fa157400640 1 -- 172.21.15.114:0/2938867772 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140035020 con 0x7fa1580d1b90 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.467+0000 7fa157400640 1 -- 172.21.15.114:0/2938867772 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa140031870 con 0x7fa1580d1b90 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.467+0000 7fa157400640 1 -- 172.21.15.114:0/2938867772 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140031c40 con 0x7fa1580d1b90 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 -- 172.21.15.114:0/2938867772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d1b90 msgr2=0x7fa1580d39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 --2- 172.21.15.114:0/2938867772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d1b90 0x7fa1580d39a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa15808dc80 tx=0x7fa14002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 -- 172.21.15.114:0/2938867772 shutdown_connections 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 --2- 172.21.15.114:0/2938867772 >> v2:172.21.15.177:3300/0 conn(0x7fa1580d3ee0 0x7fa1580d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 --2- 172.21.15.114:0/2938867772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d1b90 0x7fa1580d39a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 --2- 172.21.15.114:0/2938867772 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa15815e000 0x7fa1580ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 -- 172.21.15.114:0/2938867772 >> 172.21.15.114:0/2938867772 conn(0x7fa15808d740 msgr2=0x7fa15809d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 -- 172.21.15.114:0/2938867772 shutdown_connections 2024-03-06T07:32:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.468+0000 7fa15d800640 1 -- 172.21.15.114:0/2938867772 wait complete. 2024-03-06T07:32:14.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.470+0000 7fa15d800640 1 Processor -- start 2024-03-06T07:32:14.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.470+0000 7fa15d800640 1 -- start start 2024-03-06T07:32:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa15d800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1580d1b90 0x7fa158154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa15d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d3ee0 0x7fa1580c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa15d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15815e000 0x7fa158159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa15d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580cfba0 con 0x7fa1580d3ee0 2024-03-06T07:32:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa15d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580cf8c0 con 0x7fa15815e000 2024-03-06T07:32:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa15d800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580d2010 con 0x7fa1580d1b90 2024-03-06T07:32:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d3ee0 0x7fa1580c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d3ee0 0x7fa1580c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51126/0 (socket says 172.21.15.114:51126) 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 -- 172.21.15.114:0/134228439 learned_addr learned my addr 172.21.15.114:0/134228439 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 -- 172.21.15.114:0/134228439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1580d1b90 msgr2=0x7fa158154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1580d1b90 0x7fa158154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa15ce00640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15815e000 0x7fa158159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 -- 172.21.15.114:0/134228439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15815e000 msgr2=0x7fa158159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15815e000 0x7fa158159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.471+0000 7fa14fe00640 1 -- 172.21.15.114:0/134228439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580c85f0 con 0x7fa1580d3ee0 2024-03-06T07:32:14.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.472+0000 7fa14fe00640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d3ee0 0x7fa1580c7dc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa14002b950 tx=0x7fa1400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.472+0000 7fa154c00640 1 -- 172.21.15.114:0/134228439 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140040070 con 0x7fa1580d3ee0 2024-03-06T07:32:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.472+0000 7fa154c00640 1 -- 172.21.15.114:0/134228439 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa140030d00 con 0x7fa1580d3ee0 2024-03-06T07:32:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.472+0000 7fa154c00640 1 -- 172.21.15.114:0/134228439 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140035040 con 0x7fa1580d3ee0 2024-03-06T07:32:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.472+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1580c88d0 con 0x7fa1580d3ee0 2024-03-06T07:32:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.472+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1580c8e90 con 0x7fa1580d3ee0 2024-03-06T07:32:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.473+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15804f330 con 0x7fa1580d3ee0 2024-03-06T07:32:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.474+0000 7fa154c00640 1 -- 172.21.15.114:0/134228439 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa14004b060 con 0x7fa1580d3ee0 2024-03-06T07:32:14.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.475+0000 7fa154c00640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa13406b4d0 0x7fa13406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.475+0000 7fa154c00640 1 -- 172.21.15.114:0/134228439 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa1400abd60 con 0x7fa1580d3ee0 2024-03-06T07:32:14.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.475+0000 7fa157e00640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa13406b4d0 0x7fa13406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.476+0000 7fa157e00640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa13406b4d0 0x7fa13406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa148006fd0 tx=0x7fa148008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.480+0000 7fa154c00640 1 -- 172.21.15.114:0/134228439 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa14007ae90 con 0x7fa1580d3ee0 2024-03-06T07:32:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.679+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa1580d1270 con 0x7fa1580d3ee0 2024-03-06T07:32:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.680+0000 7fa154c00640 1 -- 172.21.15.114:0/134228439 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa14007a790 con 0x7fa1580d3ee0 2024-03-06T07:32:14.682 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:32:14.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.682+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa13406b4d0 msgr2=0x7fa13406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.682+0000 7fa15d800640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa13406b4d0 0x7fa13406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa148006fd0 tx=0x7fa148008040 comp rx=0 tx=0).stop 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.682+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d3ee0 msgr2=0x7fa1580c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.682+0000 7fa15d800640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d3ee0 0x7fa1580c7dc0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa14002b950 tx=0x7fa1400046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 shutdown_connections 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa15815e000 0x7fa158159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa13406b4d0 0x7fa13406d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d3ee0 0x7fa1580c7dc0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 --2- 172.21.15.114:0/134228439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa1580d1b90 0x7fa158154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 >> 172.21.15.114:0/134228439 conn(0x7fa15808d740 msgr2=0x7fa15815a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 shutdown_connections 2024-03-06T07:32:14.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.683+0000 7fa15d800640 1 -- 172.21.15.114:0/134228439 wait complete. 2024-03-06T07:32:14.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:32:14.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.907+0000 7f8d12000640 1 Processor -- start 2024-03-06T07:32:14.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.907+0000 7f8d12000640 1 -- start start 2024-03-06T07:32:14.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d12000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c151660 0x7f8d0c071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d12000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d0c15dff0 0x7f8d0c15e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d12000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c071a60 con 0x7f8d0c0cf430 2024-03-06T07:32:14.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d12000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0c0b3ee0 con 0x7f8d0c15dff0 2024-03-06T07:32:14.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d12000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0b35b0 con 0x7f8d0c151660 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d0c15dff0 0x7f8d0c15e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8d0c15dff0 0x7f8d0c15e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51130/0 (socket says 172.21.15.114:51130) 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0be00640 1 -- 172.21.15.114:0/1947352704 learned_addr learned my addr 172.21.15.114:0/1947352704 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0b400640 1 --2- 172.21.15.114:0/1947352704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c151660 0x7f8d0c071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0be00640 1 -- 172.21.15.114:0/1947352704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c151660 msgr2=0x7f8d0c071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0be00640 1 --2- 172.21.15.114:0/1947352704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c151660 0x7f8d0c071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0be00640 1 -- 172.21.15.114:0/1947352704 >> v1:172.21.15.177:6789/0 conn(0x7f8d0c0cf430 legacy=0x7f8d0c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.909+0000 7f8d0be00640 1 -- 172.21.15.114:0/1947352704 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c0b3890 con 0x7f8d0c15dff0 2024-03-06T07:32:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.910+0000 7f8d0be00640 1 --2- 172.21.15.114:0/1947352704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c15e480 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009e50 tx=0x7f8d0402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b737e81c91f276dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.910+0000 7f8d0aa00640 1 -- 172.21.15.114:0/1947352704 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0402c950 con 0x7f8d0c15dff0 2024-03-06T07:32:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.910+0000 7f8d0aa00640 1 -- 172.21.15.114:0/1947352704 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0402caf0 con 0x7f8d0c15dff0 2024-03-06T07:32:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.910+0000 7f8d0aa00640 1 -- 172.21.15.114:0/1947352704 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d040343f0 con 0x7f8d0c15dff0 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 -- 172.21.15.114:0/1947352704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 msgr2=0x7f8d0c15e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 --2- 172.21.15.114:0/1947352704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c15e480 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009e50 tx=0x7f8d0402bdb0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 -- 172.21.15.114:0/1947352704 shutdown_connections 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 --2- 172.21.15.114:0/1947352704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c15e480 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 --2- 172.21.15.114:0/1947352704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c151660 0x7f8d0c071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 -- 172.21.15.114:0/1947352704 >> 172.21.15.114:0/1947352704 conn(0x7f8d0c0b1730 msgr2=0x7f8d0c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 -- 172.21.15.114:0/1947352704 shutdown_connections 2024-03-06T07:32:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.911+0000 7f8d12000640 1 -- 172.21.15.114:0/1947352704 wait complete. 2024-03-06T07:32:14.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.913+0000 7f8d12000640 1 Processor -- start 2024-03-06T07:32:14.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.913+0000 7f8d12000640 1 -- start start 2024-03-06T07:32:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d12000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c0cf430 0x7f8d0c146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d12000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c151660 0x7f8d0c13f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d12000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d12000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c06daf0 con 0x7f8d0c15dff0 2024-03-06T07:32:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d12000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c071d30 con 0x7f8d0c151660 2024-03-06T07:32:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d12000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c06cb50 con 0x7f8d0c0cf430 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c148460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51142/0 (socket says 172.21.15.114:51142) 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d0be00640 1 -- 172.21.15.114:0/1669101504 learned_addr learned my addr 172.21.15.114:0/1669101504 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d0b400640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c0cf430 0x7f8d0c146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.914+0000 7f8d02a00640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c151660 0x7f8d0c13f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d0be00640 1 -- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c0cf430 msgr2=0x7f8d0c146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d0be00640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c0cf430 0x7f8d0c146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d0be00640 1 -- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c151660 msgr2=0x7f8d0c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d0be00640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c151660 0x7f8d0c13f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d0be00640 1 -- 172.21.15.114:0/1669101504 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c1473c0 con 0x7f8d0c15dff0 2024-03-06T07:32:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d0be00640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c148460 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8d04036040 tx=0x7f8d04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d03e00640 1 -- 172.21.15.114:0/1669101504 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04044020 con 0x7f8d0c15dff0 2024-03-06T07:32:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d03e00640 1 -- 172.21.15.114:0/1669101504 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04040690 con 0x7f8d0c15dff0 2024-03-06T07:32:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d03e00640 1 -- 172.21.15.114:0/1669101504 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04009510 con 0x7f8d0c15dff0 2024-03-06T07:32:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c1476a0 con 0x7f8d0c15dff0 2024-03-06T07:32:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.915+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c147c60 con 0x7f8d0c15dff0 2024-03-06T07:32:14.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.916+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c04f330 con 0x7f8d0c15dff0 2024-03-06T07:32:14.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.917+0000 7f8d03e00640 1 -- 172.21.15.114:0/1669101504 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d04047020 con 0x7f8d0c15dff0 2024-03-06T07:32:14.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.918+0000 7f8d03e00640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8ce806b4d0 0x7f8ce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.918+0000 7f8d03e00640 1 -- 172.21.15.114:0/1669101504 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8d040aca80 con 0x7f8d0c15dff0 2024-03-06T07:32:14.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.919+0000 7f8d0b400640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8ce806b4d0 0x7f8ce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.919+0000 7f8d0b400640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8ce806b4d0 0x7f8ce806d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc006fd0 tx=0x7f8cfc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:14.923+0000 7f8d03e00640 1 -- 172.21.15.114:0/1669101504 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d040aee20 con 0x7f8d0c15dff0 2024-03-06T07:32:15.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.124+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8d0c14cac0 con 0x7f8d0c15dff0 2024-03-06T07:32:15.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.125+0000 7f8d03e00640 1 -- 172.21.15.114:0/1669101504 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8d0404e030 con 0x7f8d0c15dff0 2024-03-06T07:32:15.126 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:15.126 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:32:15.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:15.127 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:32:15.127 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:32:15.127 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:15.127 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:32:15.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8ce806b4d0 msgr2=0x7f8ce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8ce806b4d0 0x7f8ce806d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc006fd0 tx=0x7f8cfc008040 comp rx=0 tx=0).stop 2024-03-06T07:32:15.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 msgr2=0x7f8d0c148460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c148460 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8d04036040 tx=0x7f8d04004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:15.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 shutdown_connections 2024-03-06T07:32:15.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8ce806b4d0 0x7f8ce806d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d0c15dff0 0x7f8d0c148460 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c151660 0x7f8d0c13f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 --2- 172.21.15.114:0/1669101504 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8d0c0cf430 0x7f8d0c146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.127+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 >> 172.21.15.114:0/1669101504 conn(0x7f8d0c0b1730 msgr2=0x7f8d0c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.128+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 shutdown_connections 2024-03-06T07:32:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.128+0000 7f8d12000640 1 -- 172.21.15.114:0/1669101504 wait complete. 2024-03-06T07:32:15.148 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:32:15.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:32:15.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.357+0000 7fbb7ca00640 1 Processor -- start 2024-03-06T07:32:15.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.357+0000 7fbb7ca00640 1 -- start start 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb7ca00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbb78070d30 0x7fbb78160470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb7ca00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb7806afc0 0x7fbb7806b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb7ca00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb78072090 con 0x7fbb78144940 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb7ca00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7815ffe0 con 0x7fbb7806afc0 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb7ca00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb78072370 con 0x7fbb78070d30 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb77200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb7806afc0 0x7fbb7806b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb77200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb7806afc0 0x7fbb7806b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51148/0 (socket says 172.21.15.114:51148) 2024-03-06T07:32:15.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb77200640 1 -- 172.21.15.114:0/832214197 learned_addr learned my addr 172.21.15.114:0/832214197 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:15.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.359+0000 7fbb6fe00640 1 --2- 172.21.15.114:0/832214197 >> v2:172.21.15.177:3300/0 conn(0x7fbb78070d30 0x7fbb78160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.360+0000 7fbb77200640 1 -- 172.21.15.114:0/832214197 >> v2:172.21.15.177:3300/0 conn(0x7fbb78070d30 msgr2=0x7fbb78160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.360+0000 7fbb77200640 1 --2- 172.21.15.114:0/832214197 >> v2:172.21.15.177:3300/0 conn(0x7fbb78070d30 0x7fbb78160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.360+0000 7fbb77200640 1 -- 172.21.15.114:0/832214197 >> v1:172.21.15.177:6789/0 conn(0x7fbb78144940 legacy=0x7fbb78071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.360+0000 7fbb77200640 1 -- 172.21.15.114:0/832214197 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7815f990 con 0x7fbb7806afc0 2024-03-06T07:32:15.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.360+0000 7fbb6fe00640 1 --2- 172.21.15.114:0/832214197 >> v2:172.21.15.177:3300/0 conn(0x7fbb78070d30 0x7fbb78160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.361+0000 7fbb77200640 1 --2- 172.21.15.114:0/832214197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7806afc0 0x7fbb7806b450 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fbb780cd4e0 tx=0x7fbb7002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1afbc24f213536d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.361+0000 7fbb75e00640 1 -- 172.21.15.114:0/832214197 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7002ca80 con 0x7fbb7806afc0 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.361+0000 7fbb75e00640 1 -- 172.21.15.114:0/832214197 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7002cc20 con 0x7fbb7806afc0 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.361+0000 7fbb75e00640 1 -- 172.21.15.114:0/832214197 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb700343f0 con 0x7fbb7806afc0 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 -- 172.21.15.114:0/832214197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7806afc0 msgr2=0x7fbb7806b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/832214197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7806afc0 0x7fbb7806b450 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fbb780cd4e0 tx=0x7fbb7002bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 -- 172.21.15.114:0/832214197 shutdown_connections 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/832214197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb7806afc0 0x7fbb7806b450 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/832214197 >> v2:172.21.15.177:3300/0 conn(0x7fbb78070d30 0x7fbb78160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 -- 172.21.15.114:0/832214197 >> 172.21.15.114:0/832214197 conn(0x7fbb780b18a0 msgr2=0x7fbb780b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 -- 172.21.15.114:0/832214197 shutdown_connections 2024-03-06T07:32:15.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.362+0000 7fbb7ca00640 1 -- 172.21.15.114:0/832214197 wait complete. 2024-03-06T07:32:15.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.364+0000 7fbb7ca00640 1 Processor -- start 2024-03-06T07:32:15.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.364+0000 7fbb7ca00640 1 -- start start 2024-03-06T07:32:15.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb7ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb7806afc0 0x7fbb7815abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb7ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb78070d30 0x7fbb781e9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb7ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb78144940 0x7fbb781e9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb7ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78144ea0 con 0x7fbb78144940 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb7ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78144d20 con 0x7fbb78070d30 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb7ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb780b3640 con 0x7fbb7806afc0 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb78144940 0x7fbb781e9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb78144940 0x7fbb781e9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51162/0 (socket says 172.21.15.114:51162) 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 -- 172.21.15.114:0/1511528985 learned_addr learned my addr 172.21.15.114:0/1511528985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb6fe00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb78070d30 0x7fbb781e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb76800640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb7806afc0 0x7fbb7815abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 -- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb7806afc0 msgr2=0x7fbb7815abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb7806afc0 0x7fbb7815abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 -- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb78070d30 msgr2=0x7fbb781e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb78070d30 0x7fbb781e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb77200640 1 -- 172.21.15.114:0/1511528985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb781e8ed0 con 0x7fbb78144940 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb6fe00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb78070d30 0x7fbb781e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.366+0000 7fbb76800640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb7806afc0 0x7fbb7815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.367+0000 7fbb77200640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb78144940 0x7fbb781e9660 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fbb70036040 tx=0x7fbb70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.367+0000 7fbb6ea00640 1 -- 172.21.15.114:0/1511528985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70043020 con 0x7fbb78144940 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.367+0000 7fbb6ea00640 1 -- 172.21.15.114:0/1511528985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7003e450 con 0x7fbb78144940 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.367+0000 7fbb6ea00640 1 -- 172.21.15.114:0/1511528985 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7003e860 con 0x7fbb78144940 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.367+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb781e8350 con 0x7fbb78144940 2024-03-06T07:32:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.367+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb781e8910 con 0x7fbb78144940 2024-03-06T07:32:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.368+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb780b4de0 con 0x7fbb78144940 2024-03-06T07:32:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.369+0000 7fbb6ea00640 1 -- 172.21.15.114:0/1511528985 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb70046020 con 0x7fbb78144940 2024-03-06T07:32:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.371+0000 7fbb6ea00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb5406b4d0 0x7fbb5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.371+0000 7fbb6ea00640 1 -- 172.21.15.114:0/1511528985 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbb700ac8c0 con 0x7fbb78144940 2024-03-06T07:32:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.371+0000 7fbb76800640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb5406b4d0 0x7fbb5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.372+0000 7fbb76800640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb5406b4d0 0x7fbb5406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800a830 tx=0x7fbb68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.375+0000 7fbb6ea00640 1 -- 172.21.15.114:0/1511528985 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb780b4de0 con 0x7fbb78144940 2024-03-06T07:32:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:15.573+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbb7815be00 con 0x7fbb78144940 2024-03-06T07:32:15.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:15.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.302+0000 7fbb6ea00640 1 -- 172.21.15.114:0/1511528985 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbb7815be00 con 0x7fbb78144940 2024-03-06T07:32:16.304 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:32:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.305+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb5406b4d0 msgr2=0x7fbb5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.305+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb5406b4d0 0x7fbb5406d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800a830 tx=0x7fbb68008040 comp rx=0 tx=0).stop 2024-03-06T07:32:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.305+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb78144940 msgr2=0x7fbb781e9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.305+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb78144940 0x7fbb781e9660 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fbb70036040 tx=0x7fbb70004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 shutdown_connections 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb5406b4d0 0x7fbb5406d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb78144940 0x7fbb781e9660 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb78070d30 0x7fbb781e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 --2- 172.21.15.114:0/1511528985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb7806afc0 0x7fbb7815abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 >> 172.21.15.114:0/1511528985 conn(0x7fbb780b18a0 msgr2=0x7fbb780af530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 shutdown_connections 2024-03-06T07:32:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.306+0000 7fbb7ca00640 1 -- 172.21.15.114:0/1511528985 wait complete. 2024-03-06T07:32:16.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:32:16.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.534+0000 7f8675800640 1 Processor -- start 2024-03-06T07:32:16.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.534+0000 7f8675800640 1 -- start start 2024-03-06T07:32:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f8675800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f86700b2d80 0x7f86700cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f8675800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f86700cf6f0 0x7f86700b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f8675800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86700b32f0 con 0x7f86701618e0 2024-03-06T07:32:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f8675800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86700d0d20 con 0x7f86700cf6f0 2024-03-06T07:32:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f8675800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86700d0f70 con 0x7f86700b2d80 2024-03-06T07:32:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f866f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f86700cf6f0 0x7f86700b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f866f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f86700cf6f0 0x7f86700b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51164/0 (socket says 172.21.15.114:51164) 2024-03-06T07:32:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f866f400640 1 -- 172.21.15.114:0/3194201349 learned_addr learned my addr 172.21.15.114:0/3194201349 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.536+0000 7f8666000640 1 --2- 172.21.15.114:0/3194201349 >> v2:172.21.15.177:3300/0 conn(0x7f86700b2d80 0x7f86700cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.537+0000 7f866f400640 1 -- 172.21.15.114:0/3194201349 >> v2:172.21.15.177:3300/0 conn(0x7f86700b2d80 msgr2=0x7f86700cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.537+0000 7f866f400640 1 --2- 172.21.15.114:0/3194201349 >> v2:172.21.15.177:3300/0 conn(0x7f86700b2d80 0x7f86700cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.537+0000 7f866f400640 1 -- 172.21.15.114:0/3194201349 >> v1:172.21.15.177:6789/0 conn(0x7f86701618e0 legacy=0x7f86700ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.537+0000 7f866f400640 1 -- 172.21.15.114:0/3194201349 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86700d1250 con 0x7f86700cf6f0 2024-03-06T07:32:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.537+0000 7f8666000640 1 --2- 172.21.15.114:0/3194201349 >> v2:172.21.15.177:3300/0 conn(0x7f86700b2d80 0x7f86700cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.538+0000 7f866f400640 1 --2- 172.21.15.114:0/3194201349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700cf6f0 0x7f86700b0c40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f867006b8d0 tx=0x7f866802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e5256c8464a007 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.538+0000 7f866e000640 1 -- 172.21.15.114:0/3194201349 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866802ca80 con 0x7f86700cf6f0 2024-03-06T07:32:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.538+0000 7f866e000640 1 -- 172.21.15.114:0/3194201349 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866802cc20 con 0x7f86700cf6f0 2024-03-06T07:32:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.538+0000 7f866e000640 1 -- 172.21.15.114:0/3194201349 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86680343f0 con 0x7f86700cf6f0 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 -- 172.21.15.114:0/3194201349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700cf6f0 msgr2=0x7f86700b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 --2- 172.21.15.114:0/3194201349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700cf6f0 0x7f86700b0c40 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f867006b8d0 tx=0x7f866802bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 -- 172.21.15.114:0/3194201349 shutdown_connections 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 --2- 172.21.15.114:0/3194201349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700cf6f0 0x7f86700b0c40 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 --2- 172.21.15.114:0/3194201349 >> v2:172.21.15.177:3300/0 conn(0x7f86700b2d80 0x7f86700cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 -- 172.21.15.114:0/3194201349 >> 172.21.15.114:0/3194201349 conn(0x7f86700d3e90 msgr2=0x7f86700d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 -- 172.21.15.114:0/3194201349 shutdown_connections 2024-03-06T07:32:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.539+0000 7f8675800640 1 -- 172.21.15.114:0/3194201349 wait complete. 2024-03-06T07:32:16.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.541+0000 7f8675800640 1 Processor -- start 2024-03-06T07:32:16.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.541+0000 7f8675800640 1 -- start start 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f8675800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700b2d80 0x7f86700cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f8675800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86700cf6f0 0x7f86700b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f8675800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86701618e0 0x7f867014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f8675800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86700d2da0 con 0x7f86700b2d80 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f8675800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8670161cc0 con 0x7f86700cf6f0 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f8675800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86700b1180 con 0x7f86701618e0 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f866ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700b2d80 0x7f86700cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f866ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700b2d80 0x7f86700cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51180/0 (socket says 172.21.15.114:51180) 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f866ea00640 1 -- 172.21.15.114:0/810180341 learned_addr learned my addr 172.21.15.114:0/810180341 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f8666000640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86700cf6f0 0x7f86700b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.543+0000 7f866f400640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86701618e0 0x7f867014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f866ea00640 1 -- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86701618e0 msgr2=0x7f867014abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f866ea00640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86701618e0 0x7f867014abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f866ea00640 1 -- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86700cf6f0 msgr2=0x7f86700b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f866ea00640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86700cf6f0 0x7f86700b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f866ea00640 1 -- 172.21.15.114:0/810180341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86700c74a0 con 0x7f86700b2d80 2024-03-06T07:32:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f866ea00640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700b2d80 0x7f86700cd3a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f866000b420 tx=0x7f866000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f8667400640 1 -- 172.21.15.114:0/810180341 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660011830 con 0x7f86700b2d80 2024-03-06T07:32:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f8667400640 1 -- 172.21.15.114:0/810180341 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8660016070 con 0x7f86700b2d80 2024-03-06T07:32:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f8667400640 1 -- 172.21.15.114:0/810180341 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660011c10 con 0x7f86700b2d80 2024-03-06T07:32:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86700c7780 con 0x7f86700b2d80 2024-03-06T07:32:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.544+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86700c8020 con 0x7f86700b2d80 2024-03-06T07:32:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.546+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86700c7950 con 0x7f86700b2d80 2024-03-06T07:32:16.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.547+0000 7f8667400640 1 -- 172.21.15.114:0/810180341 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f866001b460 con 0x7f86700b2d80 2024-03-06T07:32:16.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.548+0000 7f8667400640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f864c06b4d0 0x7f864c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.548+0000 7f8667400640 1 -- 172.21.15.114:0/810180341 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8660089ba0 con 0x7f86700b2d80 2024-03-06T07:32:16.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.548+0000 7f8666000640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f864c06b4d0 0x7f864c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.549+0000 7f8666000640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f864c06b4d0 0x7f864c06d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f86701f7550 tx=0x7f8658008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.553+0000 7f8667400640 1 -- 172.21.15.114:0/810180341 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8660020020 con 0x7f86700b2d80 2024-03-06T07:32:16.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:16.751+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f867004f330 con 0x7f86700b2d80 2024-03-06T07:32:17.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.308+0000 7f8667400640 1 -- 172.21.15.114:0/810180341 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8660058b60 con 0x7f86700b2d80 2024-03-06T07:32:17.310 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:32:17.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f864c06b4d0 msgr2=0x7f864c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f864c06b4d0 0x7f864c06d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f86701f7550 tx=0x7f8658008040 comp rx=0 tx=0).stop 2024-03-06T07:32:17.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700b2d80 msgr2=0x7f86700cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700b2d80 0x7f86700cd3a0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f866000b420 tx=0x7f866000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 shutdown_connections 2024-03-06T07:32:17.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86701618e0 0x7f867014abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86700cf6f0 0x7f86700b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f864c06b4d0 0x7f864c06d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 --2- 172.21.15.114:0/810180341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86700b2d80 0x7f86700cd3a0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.312+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 >> 172.21.15.114:0/810180341 conn(0x7f86700d3e90 msgr2=0x7f867006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.313+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 shutdown_connections 2024-03-06T07:32:17.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.313+0000 7f8675800640 1 -- 172.21.15.114:0/810180341 wait complete. 2024-03-06T07:32:17.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:32:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.535+0000 7f6c68000640 1 Processor -- start 2024-03-06T07:32:17.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.536+0000 7f6c68000640 1 -- start start 2024-03-06T07:32:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.537+0000 7f6c68000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6c600b3f70 0x7f6c60160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.537+0000 7f6c68000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c60070ff0 0x7f6c600ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.537+0000 7f6c68000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c600b46c0 con 0x7f6c6015fc10 2024-03-06T07:32:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.537+0000 7f6c68000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c60071480 con 0x7f6c600b3f70 2024-03-06T07:32:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.537+0000 7f6c68000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600b49a0 con 0x7f6c60070ff0 2024-03-06T07:32:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.538+0000 7f6c66000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c60070ff0 0x7f6c600ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.538+0000 7f6c66000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c60070ff0 0x7f6c600ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:51128/0 (socket says 172.21.15.114:51128) 2024-03-06T07:32:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.538+0000 7f6c66000640 1 -- 172.21.15.114:0/1387294503 learned_addr learned my addr 172.21.15.114:0/1387294503 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.538+0000 7f6c64c00640 1 --2- 172.21.15.114:0/1387294503 >> v2:172.21.15.177:3300/0 conn(0x7f6c600b3f70 0x7f6c60160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.539+0000 7f6c5fe00640 1 -- 172.21.15.114:0/1387294503 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797227304 0 0) 0x7f6c600b46c0 con 0x7f6c6015fc10 2024-03-06T07:32:17.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.539+0000 7f6c5fe00640 1 -- 172.21.15.114:0/1387294503 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c48003520 con 0x7f6c6015fc10 2024-03-06T07:32:17.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.539+0000 7f6c66000640 1 -- 172.21.15.114:0/1387294503 >> v2:172.21.15.177:3300/0 conn(0x7f6c600b3f70 msgr2=0x7f6c60160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.539+0000 7f6c66000640 1 --2- 172.21.15.114:0/1387294503 >> v2:172.21.15.177:3300/0 conn(0x7f6c600b3f70 0x7f6c60160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.539+0000 7f6c66000640 1 -- 172.21.15.114:0/1387294503 >> v1:172.21.15.177:6789/0 conn(0x7f6c6015fc10 legacy=0x7f6c6015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.539+0000 7f6c66000640 1 -- 172.21.15.114:0/1387294503 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60072c90 con 0x7f6c60070ff0 2024-03-06T07:32:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.540+0000 7f6c66000640 1 --2- 172.21.15.114:0/1387294503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c60070ff0 0x7f6c600ce4a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6c58009e50 tx=0x7f6c5802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bed1943a677da6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.541+0000 7f6c5fe00640 1 -- 172.21.15.114:0/1387294503 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5802cb00 con 0x7f6c60070ff0 2024-03-06T07:32:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.541+0000 7f6c5fe00640 1 -- 172.21.15.114:0/1387294503 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5802cca0 con 0x7f6c60070ff0 2024-03-06T07:32:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.541+0000 7f6c5fe00640 1 -- 172.21.15.114:0/1387294503 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c580345f0 con 0x7f6c60070ff0 2024-03-06T07:32:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.541+0000 7f6c68000640 1 -- 172.21.15.114:0/1387294503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c60070ff0 msgr2=0x7f6c600ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.541+0000 7f6c68000640 1 --2- 172.21.15.114:0/1387294503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c60070ff0 0x7f6c600ce4a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6c58009e50 tx=0x7f6c5802bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.542+0000 7f6c68000640 1 -- 172.21.15.114:0/1387294503 shutdown_connections 2024-03-06T07:32:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.542+0000 7f6c68000640 1 --2- 172.21.15.114:0/1387294503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c60070ff0 0x7f6c600ce4a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.542+0000 7f6c68000640 1 --2- 172.21.15.114:0/1387294503 >> v2:172.21.15.177:3300/0 conn(0x7f6c600b3f70 0x7f6c60160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.542+0000 7f6c68000640 1 -- 172.21.15.114:0/1387294503 >> 172.21.15.114:0/1387294503 conn(0x7f6c600b1940 msgr2=0x7f6c600b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.542+0000 7f6c68000640 1 -- 172.21.15.114:0/1387294503 shutdown_connections 2024-03-06T07:32:17.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.542+0000 7f6c68000640 1 -- 172.21.15.114:0/1387294503 wait complete. 2024-03-06T07:32:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.544+0000 7f6c68000640 1 Processor -- start 2024-03-06T07:32:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.544+0000 7f6c68000640 1 -- start start 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c68000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c60070ff0 0x7f6c6014aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c68000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c600b3f70 0x7f6c60153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c68000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c6015fc10 0x7f6c60153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c68000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c60072a30 con 0x7f6c600b3f70 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c68000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c600728b0 con 0x7f6c60070ff0 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c68000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6015c2e0 con 0x7f6c6015fc10 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c64c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c600b3f70 0x7f6c60153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c64c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c600b3f70 0x7f6c60153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51192/0 (socket says 172.21.15.114:51192) 2024-03-06T07:32:17.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c64c00640 1 -- 172.21.15.114:0/826956348 learned_addr learned my addr 172.21.15.114:0/826956348 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c66000640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c6015fc10 0x7f6c60153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.545+0000 7f6c65600640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c60070ff0 0x7f6c6014aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c64c00640 1 -- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c6015fc10 msgr2=0x7f6c60153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c64c00640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c6015fc10 0x7f6c60153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c64c00640 1 -- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c60070ff0 msgr2=0x7f6c6014aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c64c00640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c60070ff0 0x7f6c6014aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c64c00640 1 -- 172.21.15.114:0/826956348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60154df0 con 0x7f6c600b3f70 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c64c00640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c600b3f70 0x7f6c60153a00 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00e980 tx=0x7f6c4c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c5d600640 1 -- 172.21.15.114:0/826956348 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c017070 con 0x7f6c600b3f70 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c5d600640 1 -- 172.21.15.114:0/826956348 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c00ca00 con 0x7f6c600b3f70 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6014e670 con 0x7f6c600b3f70 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c5d600640 1 -- 172.21.15.114:0/826956348 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c00cdd0 con 0x7f6c600b3f70 2024-03-06T07:32:17.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.546+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c60154830 con 0x7f6c600b3f70 2024-03-06T07:32:17.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.547+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6004f330 con 0x7f6c600b3f70 2024-03-06T07:32:17.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.549+0000 7f6c5d600640 1 -- 172.21.15.114:0/826956348 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c4c01a070 con 0x7f6c600b3f70 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.549+0000 7f6c5d600640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.550+0000 7f6c5d600640 1 -- 172.21.15.114:0/826956348 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6c4c08a180 con 0x7f6c600b3f70 2024-03-06T07:32:17.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.550+0000 7f6c65600640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.550+0000 7f6c65600640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6c54007fd0 tx=0x7f6c54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.554+0000 7f6c5d600640 1 -- 172.21.15.114:0/826956348 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c60154df0 con 0x7f6c600b3f70 2024-03-06T07:32:17.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:17.780+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6c6006ef30 con 0x7f6c600b3f70 2024-03-06T07:32:18.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.346+0000 7f6c5d600640 1 -- 172.21.15.114:0/826956348 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v48) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6c4c059050 con 0x7f6c600b3f70 2024-03-06T07:32:18.353 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.359+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 msgr2=0x7f6c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.359+0000 7f6c68000640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6c54007fd0 tx=0x7f6c54009040 comp rx=0 tx=0).stop 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.359+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c600b3f70 msgr2=0x7f6c60153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.359+0000 7f6c68000640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c600b3f70 0x7f6c60153a00 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00e980 tx=0x7f6c4c00ee40 comp rx=0 tx=0).stop 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.360+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 shutdown_connections 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.360+0000 7f6c68000640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c6015fc10 0x7f6c60153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.360+0000 7f6c68000640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.360+0000 7f6c68000640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c600b3f70 0x7f6c60153a00 secure :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00e980 tx=0x7f6c4c00ee40 comp rx=0 tx=0).stop 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.360+0000 7f6c68000640 1 --2- 172.21.15.114:0/826956348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c60070ff0 0x7f6c6014aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.360+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 >> 172.21.15.114:0/826956348 conn(0x7f6c600b1940 msgr2=0x7f6c600ad9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.361+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 shutdown_connections 2024-03-06T07:32:18.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.361+0000 7f6c68000640 1 -- 172.21.15.114:0/826956348 wait complete. 2024-03-06T07:32:18.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:18.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.568+0000 7fd111600640 1 Processor -- start 2024-03-06T07:32:18.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.568+0000 7fd111600640 1 -- start start 2024-03-06T07:32:18.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.569+0000 7fd111600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.569+0000 7fd111600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd10c0b4680 0x7fd10c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.569+0000 7fd111600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10c15b090 con 0x7fd10c072340 2024-03-06T07:32:18.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.569+0000 7fd111600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd10c0b2a30 con 0x7fd10c0b4680 2024-03-06T07:32:18.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.569+0000 7fd111600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0b4360 con 0x7fd10c15acb0 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.570+0000 7fd10a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd10c0b4680 0x7fd10c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.570+0000 7fd10a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd10c0b4680 0x7fd10c06e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35020/0 (socket says 172.21.15.114:35020) 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.570+0000 7fd10a000640 1 -- 172.21.15.114:0/630402683 learned_addr learned my addr 172.21.15.114:0/630402683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.570+0000 7fd10a000640 1 -- 172.21.15.114:0/630402683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 msgr2=0x7fd10c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.570+0000 7fd10a000640 1 --2- 172.21.15.114:0/630402683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.570+0000 7fd10a000640 1 -- 172.21.15.114:0/630402683 >> v1:172.21.15.114:6789/0 conn(0x7fd10c072340 legacy=0x7fd10c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.570+0000 7fd10a000640 1 -- 172.21.15.114:0/630402683 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c0b2cf0 con 0x7fd10c0b4680 2024-03-06T07:32:18.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.571+0000 7fd10a000640 1 --2- 172.21.15.114:0/630402683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c06e4f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5c147efd1db60f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.571+0000 7fd109600640 1 -- 172.21.15.114:0/630402683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f402cb00 con 0x7fd10c0b4680 2024-03-06T07:32:18.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.571+0000 7fd109600640 1 -- 172.21.15.114:0/630402683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f402cca0 con 0x7fd10c0b4680 2024-03-06T07:32:18.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd109600640 1 -- 172.21.15.114:0/630402683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4034610 con 0x7fd10c0b4680 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 -- 172.21.15.114:0/630402683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 msgr2=0x7fd10c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 --2- 172.21.15.114:0/630402683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c06e4f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f402bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 -- 172.21.15.114:0/630402683 shutdown_connections 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 --2- 172.21.15.114:0/630402683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c06e4f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 --2- 172.21.15.114:0/630402683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 -- 172.21.15.114:0/630402683 >> 172.21.15.114:0/630402683 conn(0x7fd10c0b1730 msgr2=0x7fd10c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 -- 172.21.15.114:0/630402683 shutdown_connections 2024-03-06T07:32:18.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.572+0000 7fd111600640 1 -- 172.21.15.114:0/630402683 wait complete. 2024-03-06T07:32:18.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.574+0000 7fd111600640 1 Processor -- start 2024-03-06T07:32:18.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.574+0000 7fd111600640 1 -- start start 2024-03-06T07:32:18.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd111600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd10c072340 0x7fd10c15ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd111600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c15d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd111600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd111600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0b3d10 con 0x7fd10c0b4680 2024-03-06T07:32:18.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd111600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0b32b0 con 0x7fd10c072340 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd111600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0b4010 con 0x7fd10c15acb0 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd10a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c15d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd10a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c15d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35034/0 (socket says 172.21.15.114:35034) 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd10a000640 1 -- 172.21.15.114:0/1862290233 learned_addr learned my addr 172.21.15.114:0/1862290233 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd10b400640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.575+0000 7fd10a000640 1 -- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 msgr2=0x7fd10c15d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd10a000640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c15d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd10a000640 1 -- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd10c072340 msgr2=0x7fd10c15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd10a000640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd10c072340 0x7fd10c15ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd10a000640 1 -- 172.21.15.114:0/1862290233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c0cda20 con 0x7fd10c0b4680 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd10b400640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c15d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:18.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd10a000640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c15d3a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd102a00640 1 -- 172.21.15.114:0/1862290233 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4043020 con 0x7fd10c0b4680 2024-03-06T07:32:18.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd102a00640 1 -- 172.21.15.114:0/1862290233 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f402ce50 con 0x7fd10c0b4680 2024-03-06T07:32:18.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd102a00640 1 -- 172.21.15.114:0/1862290233 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f403e690 con 0x7fd10c0b4680 2024-03-06T07:32:18.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c0cdcc0 con 0x7fd10c0b4680 2024-03-06T07:32:18.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.576+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c0ce280 con 0x7fd10c0b4680 2024-03-06T07:32:18.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.577+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c04f330 con 0x7fd10c0b4680 2024-03-06T07:32:18.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.578+0000 7fd102a00640 1 -- 172.21.15.114:0/1862290233 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0f4046020 con 0x7fd10c0b4680 2024-03-06T07:32:18.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.580+0000 7fd102a00640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0e806b4d0 0x7fd0e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.580+0000 7fd102a00640 1 -- 172.21.15.114:0/1862290233 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0f40ac9c0 con 0x7fd10c0b4680 2024-03-06T07:32:18.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.580+0000 7fd10aa00640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0e806b4d0 0x7fd0e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.580+0000 7fd10aa00640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0e806b4d0 0x7fd0e806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd10c14e030 tx=0x7fd0fc006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.584+0000 7fd102a00640 1 -- 172.21.15.114:0/1862290233 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0f404d030 con 0x7fd10c0b4680 2024-03-06T07:32:18.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.780+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd10c06ce60 con 0x7fd10c0b4680 2024-03-06T07:32:18.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.783+0000 7fd102a00640 1 -- 172.21.15.114:0/1862290233 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16721 (secure 0 0 0) 0x7fd0f407b950 con 0x7fd10c0b4680 2024-03-06T07:32:18.785 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:18.785 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":79,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:17.781529+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:15.626899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:16.804201+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:18.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0e806b4d0 msgr2=0x7fd0e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0e806b4d0 0x7fd0e806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd10c14e030 tx=0x7fd0fc006c90 comp rx=0 tx=0).stop 2024-03-06T07:32:18.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 msgr2=0x7fd10c15d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c15d3a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:18.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 shutdown_connections 2024-03-06T07:32:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd10c15acb0 0x7fd10c15d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0e806b4d0 0x7fd0e806d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd10c0b4680 0x7fd10c15d3a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 --2- 172.21.15.114:0/1862290233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd10c072340 0x7fd10c15ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 >> 172.21.15.114:0/1862290233 conn(0x7fd10c0b1730 msgr2=0x7fd10c14d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 shutdown_connections 2024-03-06T07:32:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:18.786+0000 7fd111600640 1 -- 172.21.15.114:0/1862290233 wait complete. 2024-03-06T07:32:18.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:19.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.001+0000 7fd580c00640 1 Processor -- start 2024-03-06T07:32:19.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.001+0000 7fd580c00640 1 -- start start 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.002+0000 7fd580c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd57c0b4680 0x7fd57c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd580c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c072340 0x7fd57c06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd580c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c15b090 con 0x7fd57c15acb0 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd580c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c0b2a30 con 0x7fd57c0b4680 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd580c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0b4360 con 0x7fd57c072340 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd571400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd57c0b4680 0x7fd57c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd571400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd57c0b4680 0x7fd57c06e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35040/0 (socket says 172.21.15.114:35040) 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd571400640 1 -- 172.21.15.114:0/3132376590 learned_addr learned my addr 172.21.15.114:0/3132376590 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:19.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd57a800640 1 --2- 172.21.15.114:0/3132376590 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c072340 0x7fd57c06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd571400640 1 -- 172.21.15.114:0/3132376590 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c072340 msgr2=0x7fd57c06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd571400640 1 --2- 172.21.15.114:0/3132376590 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c072340 0x7fd57c06ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd571400640 1 -- 172.21.15.114:0/3132376590 >> v1:172.21.15.177:6789/0 conn(0x7fd57c15acb0 legacy=0x7fd57c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.003+0000 7fd571400640 1 -- 172.21.15.114:0/3132376590 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0b2cf0 con 0x7fd57c0b4680 2024-03-06T07:32:19.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.005+0000 7fd571400640 1 --2- 172.21.15.114:0/3132376590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c0b4680 0x7fd57c06e4f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd5640098a0 tx=0x7fd56402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5732e4183fa3e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.005+0000 7fd579400640 1 -- 172.21.15.114:0/3132376590 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564034070 con 0x7fd57c0b4680 2024-03-06T07:32:19.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.005+0000 7fd579400640 1 -- 172.21.15.114:0/3132376590 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564030840 con 0x7fd57c0b4680 2024-03-06T07:32:19.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.005+0000 7fd579400640 1 -- 172.21.15.114:0/3132376590 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564030c10 con 0x7fd57c0b4680 2024-03-06T07:32:19.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.005+0000 7fd580c00640 1 -- 172.21.15.114:0/3132376590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c0b4680 msgr2=0x7fd57c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.005+0000 7fd580c00640 1 --2- 172.21.15.114:0/3132376590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c0b4680 0x7fd57c06e4f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd5640098a0 tx=0x7fd56402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:19.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.006+0000 7fd580c00640 1 -- 172.21.15.114:0/3132376590 shutdown_connections 2024-03-06T07:32:19.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.006+0000 7fd580c00640 1 --2- 172.21.15.114:0/3132376590 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c072340 0x7fd57c06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.006+0000 7fd580c00640 1 --2- 172.21.15.114:0/3132376590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c0b4680 0x7fd57c06e4f0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.006+0000 7fd580c00640 1 -- 172.21.15.114:0/3132376590 >> 172.21.15.114:0/3132376590 conn(0x7fd57c0b1730 msgr2=0x7fd57c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.006+0000 7fd580c00640 1 -- 172.21.15.114:0/3132376590 shutdown_connections 2024-03-06T07:32:19.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.006+0000 7fd580c00640 1 -- 172.21.15.114:0/3132376590 wait complete. 2024-03-06T07:32:19.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.008+0000 7fd580c00640 1 Processor -- start 2024-03-06T07:32:19.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.008+0000 7fd580c00640 1 -- start start 2024-03-06T07:32:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd580c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c072340 0x7fd57c1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd580c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c0b4680 0x7fd57c156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd580c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c15acb0 0x7fd57c144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd580c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0b4010 con 0x7fd57c072340 2024-03-06T07:32:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd580c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0b32b0 con 0x7fd57c15acb0 2024-03-06T07:32:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd580c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c0ccfb0 con 0x7fd57c0b4680 2024-03-06T07:32:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c072340 0x7fd57c1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c072340 0x7fd57c1582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35048/0 (socket says 172.21.15.114:35048) 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 -- 172.21.15.114:0/2138129649 learned_addr learned my addr 172.21.15.114:0/2138129649 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd571400640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c0b4680 0x7fd57c156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd57a800640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c15acb0 0x7fd57c144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 -- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c0b4680 msgr2=0x7fd57c156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c0b4680 0x7fd57c156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 -- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c15acb0 msgr2=0x7fd57c144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c15acb0 0x7fd57c144670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.009+0000 7fd579e00640 1 -- 172.21.15.114:0/2138129649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c1454e0 con 0x7fd57c072340 2024-03-06T07:32:19.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.010+0000 7fd579e00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c072340 0x7fd57c1582a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00c730 tx=0x7fd56c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.010+0000 7fd572a00640 1 -- 172.21.15.114:0/2138129649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c016020 con 0x7fd57c072340 2024-03-06T07:32:19.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.010+0000 7fd572a00640 1 -- 172.21.15.114:0/2138129649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c006d50 con 0x7fd57c072340 2024-03-06T07:32:19.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.010+0000 7fd572a00640 1 -- 172.21.15.114:0/2138129649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c003bb0 con 0x7fd57c072340 2024-03-06T07:32:19.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.010+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c15a4e0 con 0x7fd57c072340 2024-03-06T07:32:19.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.010+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c14cb70 con 0x7fd57c072340 2024-03-06T07:32:19.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.012+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c0b2430 con 0x7fd57c072340 2024-03-06T07:32:19.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.012+0000 7fd572a00640 1 -- 172.21.15.114:0/2138129649 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd56c00a850 con 0x7fd57c072340 2024-03-06T07:32:19.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.013+0000 7fd572a00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd55806b4d0 0x7fd55806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.013+0000 7fd572a00640 1 -- 172.21.15.114:0/2138129649 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd56c089e20 con 0x7fd57c072340 2024-03-06T07:32:19.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.013+0000 7fd571400640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd55806b4d0 0x7fd55806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.014+0000 7fd571400640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd55806b4d0 0x7fd55806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd564036000 tx=0x7fd56403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.018+0000 7fd572a00640 1 -- 172.21.15.114:0/2138129649 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd57c0b2430 con 0x7fd57c072340 2024-03-06T07:32:19.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.214+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd57c04f330 con 0x7fd57c072340 2024-03-06T07:32:19.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.218+0000 7fd572a00640 1 -- 172.21.15.114:0/2138129649 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16721 (secure 0 0 0) 0x7fd56c020020 con 0x7fd57c072340 2024-03-06T07:32:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:19.220 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":79,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:17.781529+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:15.626899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:16.804201+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:19.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd55806b4d0 msgr2=0x7fd55806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd55806b4d0 0x7fd55806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd564036000 tx=0x7fd56403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:19.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c072340 msgr2=0x7fd57c1582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c072340 0x7fd57c1582a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00c730 tx=0x7fd56c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 shutdown_connections 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd57c15acb0 0x7fd57c144670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd57c0b4680 0x7fd57c156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd55806b4d0 0x7fd55806d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 --2- 172.21.15.114:0/2138129649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd57c072340 0x7fd57c1582a0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 >> 172.21.15.114:0/2138129649 conn(0x7fd57c0b1730 msgr2=0x7fd57c1512d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.220+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 shutdown_connections 2024-03-06T07:32:19.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.221+0000 7fd580c00640 1 -- 172.21.15.114:0/2138129649 wait complete. 2024-03-06T07:32:19.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:32:19.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.444+0000 7f6133000640 1 Processor -- start 2024-03-06T07:32:19.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.445+0000 7f6133000640 1 -- start start 2024-03-06T07:32:19.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.446+0000 7f6133000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 0x7f612c0d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.446+0000 7f6133000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c0b3730 con 0x7f612c0d1bf0 2024-03-06T07:32:19.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.446+0000 7f6133000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c067b20 con 0x7f612c15e450 2024-03-06T07:32:19.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.446+0000 7f6133000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0d4930 con 0x7f612c0d07f0 2024-03-06T07:32:19.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.446+0000 7f6131000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f612c0d1bf0 0x7f612c0b4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46248/0 (socket says 172.21.15.114:46248) 2024-03-06T07:32:19.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.446+0000 7f6131000640 1 -- 172.21.15.114:0/1342512563 learned_addr learned my addr 172.21.15.114:0/1342512563 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:19.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612be00640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 0x7f612c0d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643047239 0 0) 0x7f612c0b3730 con 0x7f612c0d1bf0 2024-03-06T07:32:19.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6110003520 con 0x7f612c0d1bf0 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2901908373 0 0) 0x7f6110003520 con 0x7f612c0d1bf0 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 msgr2=0x7f612c0d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 0x7f612c0d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 >> v1:172.21.15.177:6789/0 conn(0x7f612c15e450 legacy=0x7f612c0b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c0d1110 con 0x7f612c0d1bf0 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6124001670 con 0x7f612c0d1bf0 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 >> v1:172.21.15.114:6789/0 conn(0x7f612c0d1bf0 legacy=0x7f612c0b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61100039e0 0x7f6110003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 0x7f6110004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6110004d40 0x7f61100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0d4930 con 0x7f6110004d40 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611000bba0 con 0x7f61100039e0 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.447+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611000bda0 con 0x7f612c0d07f0 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f6131000640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6110004d40 0x7f61100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f612be00640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61100039e0 0x7f6110003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f6123400640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 0x7f6110004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f6131000640 1 -- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 msgr2=0x7f6110004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f6131000640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f612c0d07f0 0x7f6110004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f6131000640 1 -- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61100039e0 msgr2=0x7f6110003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f6131000640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61100039e0 0x7f6110003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f6131000640 1 -- 172.21.15.114:0/1342512563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6110003520 con 0x7f6110004d40 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.448+0000 7f612be00640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61100039e0 0x7f6110003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.449+0000 7f6131000640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6110004d40 0x7f61100051c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f612400c240 tx=0x7f6110005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765599fe028d3f0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.449+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124007d10 con 0x7f6110004d40 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.449+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6124030070 con 0x7f6110004d40 2024-03-06T07:32:19.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.449+0000 7f612b400640 1 -- 172.21.15.114:0/1342512563 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612402cd60 con 0x7f6110004d40 2024-03-06T07:32:19.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.450+0000 7f6133000640 1 -- 172.21.15.114:0/1342512563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6110004d40 msgr2=0x7f61100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.450+0000 7f6133000640 1 --2- 172.21.15.114:0/1342512563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6110004d40 0x7f61100051c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f612400c240 tx=0x7f6110005830 comp rx=0 tx=0).stop 2024-03-06T07:32:19.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.450+0000 7f6131000640 1 -- 172.21.15.114:0/1342512563 reap_dead start 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.450+0000 7f6133000640 1 -- 172.21.15.114:0/1342512563 shutdown_connections 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.450+0000 7f6133000640 1 -- 172.21.15.114:0/1342512563 >> 172.21.15.114:0/1342512563 conn(0x7f612c0d3e10 msgr2=0x7f612c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.450+0000 7f6133000640 1 -- 172.21.15.114:0/1342512563 shutdown_connections 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.450+0000 7f6133000640 1 -- 172.21.15.114:0/1342512563 wait complete. 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.452+0000 7f6133000640 1 Processor -- start 2024-03-06T07:32:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.452+0000 7f6133000640 1 -- start start 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6133000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6110004d40 0x7f612c0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6133000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612c0cc4e0 0x7f612c1f8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6133000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612c0c9ce0 0x7f612c0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6133000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0b3c80 con 0x7f612c0c9ce0 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6133000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0b39a0 con 0x7f612c0cc4e0 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6133000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0b3f60 con 0x7f6110004d40 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612c0c9ce0 0x7f612c0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612c0c9ce0 0x7f612c0ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35066/0 (socket says 172.21.15.114:35066) 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 -- 172.21.15.114:0/395217242 learned_addr learned my addr 172.21.15.114:0/395217242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:19.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f612be00640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6110004d40 0x7f612c0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6123400640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612c0cc4e0 0x7f612c1f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 -- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6110004d40 msgr2=0x7f612c0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6110004d40 0x7f612c0c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 -- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612c0cc4e0 msgr2=0x7f612c1f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612c0cc4e0 0x7f612c1f8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6131000640 1 -- 172.21.15.114:0/395217242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c1f8690 con 0x7f612c0c9ce0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.453+0000 7f6123400640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612c0cc4e0 0x7f612c1f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.454+0000 7f6131000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612c0c9ce0 0x7f612c0ca160 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f612400c240 tx=0x7f612402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.454+0000 7f6128c00640 1 -- 172.21.15.114:0/395217242 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124007d10 con 0x7f612c0c9ce0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.454+0000 7f6128c00640 1 -- 172.21.15.114:0/395217242 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6124004bf0 con 0x7f612c0c9ce0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.454+0000 7f6128c00640 1 -- 172.21.15.114:0/395217242 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612402de30 con 0x7f612c0c9ce0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.454+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c0ccaa0 con 0x7f612c0c9ce0 2024-03-06T07:32:19.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.454+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c0cd060 con 0x7f612c0c9ce0 2024-03-06T07:32:19.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.455+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c04f330 con 0x7f612c0c9ce0 2024-03-06T07:32:19.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.456+0000 7f6128c00640 1 -- 172.21.15.114:0/395217242 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6124007070 con 0x7f612c0c9ce0 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.457+0000 7f6128c00640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610806b500 0x7f610806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.457+0000 7f6128c00640 1 -- 172.21.15.114:0/395217242 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61240aad20 con 0x7f612c0c9ce0 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.457+0000 7f612be00640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610806b500 0x7f610806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.458+0000 7f612be00640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610806b500 0x7f610806d9c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f611c006fd0 tx=0x7f611c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.462+0000 7f6128c00640 1 -- 172.21.15.114:0/395217242 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f612c1f8690 con 0x7f612c0c9ce0 2024-03-06T07:32:19.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:19.697+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f612c0d21d0 con 0x7f612c0c9ce0 2024-03-06T07:32:20.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.367+0000 7f6128c00640 1 -- 172.21.15.114:0/395217242 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v51) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6124079b00 con 0x7f612c0c9ce0 2024-03-06T07:32:20.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610806b500 msgr2=0x7f610806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610806b500 0x7f610806d9c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f611c006fd0 tx=0x7f611c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612c0c9ce0 msgr2=0x7f612c0ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612c0c9ce0 0x7f612c0ca160 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f612400c240 tx=0x7f612402e070 comp rx=0 tx=0).stop 2024-03-06T07:32:20.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 shutdown_connections 2024-03-06T07:32:20.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f610806b500 0x7f610806d9c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f612c0c9ce0 0x7f612c0ca160 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f612c0cc4e0 0x7f612c1f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.370+0000 7f6133000640 1 --2- 172.21.15.114:0/395217242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6110004d40 0x7f612c0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.371+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 >> 172.21.15.114:0/395217242 conn(0x7f612c0d3e10 msgr2=0x7f612c0b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.371+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 shutdown_connections 2024-03-06T07:32:20.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.371+0000 7f6133000640 1 -- 172.21.15.114:0/395217242 wait complete. 2024-03-06T07:32:20.394 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:32:20.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:32:20.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.599+0000 7fc2d9a00640 1 Processor -- start 2024-03-06T07:32:20.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.599+0000 7fc2d9a00640 1 -- start start 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.600+0000 7fc2d9a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc2d41618e0 0x7fc2d40ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.600+0000 7fc2d9a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d40cf6f0 0x7fc2d40b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.600+0000 7fc2d9a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d40b32f0 con 0x7fc2d40b2d80 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.600+0000 7fc2d9a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d40d0d20 con 0x7fc2d41618e0 2024-03-06T07:32:20.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.600+0000 7fc2d9a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40d0f70 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.601+0000 7fc2d3400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d40cf6f0 0x7fc2d40b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.601+0000 7fc2d3400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d40cf6f0 0x7fc2d40b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:43036/0 (socket says 172.21.15.114:43036) 2024-03-06T07:32:20.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.601+0000 7fc2d3400640 1 -- 172.21.15.114:0/3332625644 learned_addr learned my addr 172.21.15.114:0/3332625644 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:20.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.601+0000 7fc2d2a00640 1 --2- 172.21.15.114:0/3332625644 >> v2:172.21.15.177:3300/0 conn(0x7fc2d41618e0 0x7fc2d40ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.602+0000 7fc2d3400640 1 -- 172.21.15.114:0/3332625644 >> v2:172.21.15.177:3300/0 conn(0x7fc2d41618e0 msgr2=0x7fc2d40ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.602+0000 7fc2d3400640 1 --2- 172.21.15.114:0/3332625644 >> v2:172.21.15.177:3300/0 conn(0x7fc2d41618e0 0x7fc2d40ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.602+0000 7fc2d3400640 1 -- 172.21.15.114:0/3332625644 >> v1:172.21.15.177:6789/0 conn(0x7fc2d40b2d80 legacy=0x7fc2d40cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.602+0000 7fc2d3400640 1 -- 172.21.15.114:0/3332625644 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d40d1250 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.603+0000 7fc2d3400640 1 --2- 172.21.15.114:0/3332625644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d40cf6f0 0x7fc2d40b0c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009e50 tx=0x7fc2cc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d73f7513b1f46c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d1600640 1 -- 172.21.15.114:0/3332625644 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc02cb00 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d1600640 1 -- 172.21.15.114:0/3332625644 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc02cca0 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d1600640 1 -- 172.21.15.114:0/3332625644 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc034610 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3332625644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d40cf6f0 msgr2=0x7fc2d40b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3332625644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d40cf6f0 0x7fc2d40b0c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009e50 tx=0x7fc2cc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3332625644 shutdown_connections 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3332625644 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d40cf6f0 0x7fc2d40b0c40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3332625644 >> v2:172.21.15.177:3300/0 conn(0x7fc2d41618e0 0x7fc2d40ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.604+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3332625644 >> 172.21.15.114:0/3332625644 conn(0x7fc2d40d3e90 msgr2=0x7fc2d40d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.605+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3332625644 shutdown_connections 2024-03-06T07:32:20.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.605+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3332625644 wait complete. 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.606+0000 7fc2d9a00640 1 Processor -- start 2024-03-06T07:32:20.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.606+0000 7fc2d9a00640 1 -- start start 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.607+0000 7fc2d9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d40b2d80 0x7fc2d4159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.607+0000 7fc2d9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d40cf6f0 0x7fc2d40cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.607+0000 7fc2d9a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d41618e0 0x7fc2d40cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.607+0000 7fc2d9a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40d2da0 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.607+0000 7fc2d9a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4161cc0 con 0x7fc2d40b2d80 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.607+0000 7fc2d9a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40b1180 con 0x7fc2d41618e0 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d40cf6f0 0x7fc2d40cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d40cf6f0 0x7fc2d40cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35074/0 (socket says 172.21.15.114:35074) 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 -- 172.21.15.114:0/3656875664 learned_addr learned my addr 172.21.15.114:0/3656875664 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d40b2d80 0x7fc2d4159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d3400640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d41618e0 0x7fc2d40cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 -- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d41618e0 msgr2=0x7fc2d40cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d41618e0 0x7fc2d40cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 -- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d40b2d80 msgr2=0x7fc2d4159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d40b2d80 0x7fc2d4159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.608+0000 7fc2d2000640 1 -- 172.21.15.114:0/3656875664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d41599c0 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.609+0000 7fc2d2000640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d40cf6f0 0x7fc2d40cd450 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00c400 tx=0x7fc2bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.609+0000 7fc2caa00640 1 -- 172.21.15.114:0/3656875664 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0052d0 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.609+0000 7fc2caa00640 1 -- 172.21.15.114:0/3656875664 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc005490 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.609+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d415b400 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.609+0000 7fc2caa00640 1 -- 172.21.15.114:0/3656875664 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc007260 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.609+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d40b6b50 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.610+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d404f330 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.611+0000 7fc2caa00640 1 -- 172.21.15.114:0/3656875664 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2bc007400 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.612+0000 7fc2caa00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc2b006b4d0 0x7fc2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.612+0000 7fc2caa00640 1 -- 172.21.15.114:0/3656875664 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2bc0896c0 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.612+0000 7fc2d2a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc2b006b4d0 0x7fc2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.613+0000 7fc2d2a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc2b006b4d0 0x7fc2b006d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4006fd0 tx=0x7fc2c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.617+0000 7fc2caa00640 1 -- 172.21.15.114:0/3656875664 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2d41599c0 con 0x7fc2d40cf6f0 2024-03-06T07:32:20.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:20.853+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc2d4160540 con 0x7fc2d40cf6f0 2024-03-06T07:32:21.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.377+0000 7fc2caa00640 1 -- 172.21.15.114:0/3656875664 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc2bc0585d0 con 0x7fc2d40cf6f0 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc2b006b4d0 msgr2=0x7fc2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc2b006b4d0 0x7fc2b006d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4006fd0 tx=0x7fc2c4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d40cf6f0 msgr2=0x7fc2d40cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d40cf6f0 0x7fc2d40cd450 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00c400 tx=0x7fc2bc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 shutdown_connections 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc2d41618e0 0x7fc2d40cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc2b006b4d0 0x7fc2b006d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2d40cf6f0 0x7fc2d40cd450 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 --2- 172.21.15.114:0/3656875664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2d40b2d80 0x7fc2d4159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 >> 172.21.15.114:0/3656875664 conn(0x7fc2d40d3e90 msgr2=0x7fc2d40ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 shutdown_connections 2024-03-06T07:32:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.381+0000 7fc2d9a00640 1 -- 172.21.15.114:0/3656875664 wait complete. 2024-03-06T07:32:21.410 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:32:21.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:21.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.596+0000 7f8e10000640 1 Processor -- start 2024-03-06T07:32:21.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.596+0000 7f8e10000640 1 -- start start 2024-03-06T07:32:21.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.597+0000 7f8e10000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8e080b4680 0x7f8e0806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.597+0000 7f8e10000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e08072340 0x7f8e0806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.597+0000 7f8e10000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0815b090 con 0x7f8e0815acb0 2024-03-06T07:32:21.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.597+0000 7f8e10000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e080b2a30 con 0x7f8e080b4680 2024-03-06T07:32:21.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.597+0000 7f8e10000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b4360 con 0x7f8e08072340 2024-03-06T07:32:21.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.598+0000 7f8e0d600640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8e0815acb0 0x7f8e080b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46250/0 (socket says 172.21.15.114:46250) 2024-03-06T07:32:21.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.598+0000 7f8e0d600640 1 -- 172.21.15.114:0/3597183448 learned_addr learned my addr 172.21.15.114:0/3597183448 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:21.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.598+0000 7f8e0cc00640 1 --2- 172.21.15.114:0/3597183448 >> v2:172.21.15.177:3300/0 conn(0x7f8e080b4680 0x7f8e0806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.598+0000 7f8e0e000640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e08072340 0x7f8e0806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.598+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354294011 0 0) 0x7f8e0815b090 con 0x7f8e0815acb0 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.598+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df0003520 con 0x7f8e0815acb0 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755142504 0 0) 0x7f8df0003520 con 0x7f8e0815acb0 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e08072340 msgr2=0x7f8e0806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e08072340 0x7f8e0806ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 >> v2:172.21.15.177:3300/0 conn(0x7f8e080b4680 msgr2=0x7f8e0806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 --2- 172.21.15.114:0/3597183448 >> v2:172.21.15.177:3300/0 conn(0x7f8e080b4680 0x7f8e0806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e080b2cf0 con 0x7f8e0815acb0 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8dfc0019f0 con 0x7f8e0815acb0 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 >> v1:172.21.15.114:6789/0 conn(0x7f8e0815acb0 legacy=0x7f8e080b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df00039b0 0x7f8df0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e08072340 0x7f8df0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8df0004d40 0x7f8df00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b2a30 con 0x7f8df00039b0 2024-03-06T07:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b4360 con 0x7f8df0004d40 2024-03-06T07:32:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df000b9c0 con 0x7f8e08072340 2024-03-06T07:32:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.599+0000 7f8e0d600640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df00039b0 0x7f8df0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.600+0000 7f8e0d600640 1 -- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e08072340 msgr2=0x7f8df0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.600+0000 7f8e0d600640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8e08072340 0x7f8df0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.600+0000 7f8e0d600640 1 -- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8df0004d40 msgr2=0x7f8df00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.600+0000 7f8e0d600640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8df0004d40 0x7f8df00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.600+0000 7f8e0d600640 1 -- 172.21.15.114:0/3597183448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df0003520 con 0x7f8df00039b0 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.601+0000 7f8e0d600640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df00039b0 0x7f8df0003dd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00a920 tx=0x7f8df0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4def9539209ea404 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.601+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc00a4c0 con 0x7f8df00039b0 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.601+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc034650 con 0x7f8df00039b0 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.601+0000 7f8e07e00640 1 -- 172.21.15.114:0/3597183448 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc034a40 con 0x7f8df00039b0 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.601+0000 7f8e10000640 1 -- 172.21.15.114:0/3597183448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df00039b0 msgr2=0x7f8df0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.601+0000 7f8e10000640 1 --2- 172.21.15.114:0/3597183448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8df00039b0 0x7f8df0003dd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00a920 tx=0x7f8df0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.601+0000 7f8e0e000640 1 -- 172.21.15.114:0/3597183448 reap_dead start 2024-03-06T07:32:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.602+0000 7f8e10000640 1 -- 172.21.15.114:0/3597183448 shutdown_connections 2024-03-06T07:32:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.602+0000 7f8e10000640 1 -- 172.21.15.114:0/3597183448 >> 172.21.15.114:0/3597183448 conn(0x7f8e080b1730 msgr2=0x7f8e080b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.602+0000 7f8e10000640 1 -- 172.21.15.114:0/3597183448 shutdown_connections 2024-03-06T07:32:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.602+0000 7f8e10000640 1 -- 172.21.15.114:0/3597183448 wait complete. 2024-03-06T07:32:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.603+0000 7f8e10000640 1 Processor -- start 2024-03-06T07:32:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.603+0000 7f8e10000640 1 -- start start 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.604+0000 7f8e10000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8df00039b0 0x7f8e0815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.604+0000 7f8e10000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e0815b2c0 0x7f8e0814f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.604+0000 7f8e10000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e081fbef0 0x7f8e081fc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.604+0000 7f8e10000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b4010 con 0x7f8e0815b2c0 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.604+0000 7f8e10000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080b32b0 con 0x7f8e081fbef0 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.604+0000 7f8e10000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0806ea30 con 0x7f8df00039b0 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e0815b2c0 0x7f8e0814f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e0815b2c0 0x7f8e0814f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35080/0 (socket says 172.21.15.114:35080) 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 -- 172.21.15.114:0/377609177 learned_addr learned my addr 172.21.15.114:0/377609177 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0d600640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8df00039b0 0x7f8e0815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0e000640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e081fbef0 0x7f8e081fc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 -- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8df00039b0 msgr2=0x7f8e0815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8df00039b0 0x7f8e0815ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 -- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e081fbef0 msgr2=0x7f8e081fc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e081fbef0 0x7f8e081fc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.605+0000 7f8e0cc00640 1 -- 172.21.15.114:0/377609177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e081529a0 con 0x7f8e0815b2c0 2024-03-06T07:32:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.606+0000 7f8e0cc00640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e0815b2c0 0x7f8e0814f3e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8df400e860 tx=0x7f8df400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.606+0000 7f8e05600640 1 -- 172.21.15.114:0/377609177 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4017070 con 0x7f8e0815b2c0 2024-03-06T07:32:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.606+0000 7f8e05600640 1 -- 172.21.15.114:0/377609177 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df400c940 con 0x7f8e0815b2c0 2024-03-06T07:32:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.606+0000 7f8e05600640 1 -- 172.21.15.114:0/377609177 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df400cd10 con 0x7f8e0815b2c0 2024-03-06T07:32:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.606+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0815b740 con 0x7f8e0815b2c0 2024-03-06T07:32:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.606+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e08152f20 con 0x7f8e0815b2c0 2024-03-06T07:32:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.607+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0804f330 con 0x7f8e0815b2c0 2024-03-06T07:32:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.608+0000 7f8e05600640 1 -- 172.21.15.114:0/377609177 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8df401a070 con 0x7f8e0815b2c0 2024-03-06T07:32:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.610+0000 7f8e05600640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8de806b4d0 0x7f8de806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.610+0000 7f8e05600640 1 -- 172.21.15.114:0/377609177 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8df408a1a0 con 0x7f8e0815b2c0 2024-03-06T07:32:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.610+0000 7f8e0d600640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8de806b4d0 0x7f8de806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.611+0000 7f8e0d600640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8de806b4d0 0x7f8de806d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc034000 tx=0x7f8dfc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.614+0000 7f8e05600640 1 -- 172.21.15.114:0/377609177 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8df4058f80 con 0x7f8e0815b2c0 2024-03-06T07:32:21.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.847+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e080b2700 con 0x7f8e0815b2c0 2024-03-06T07:32:21.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.847+0000 7f8e05600640 1 -- 172.21.15.114:0/377609177 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f8df4058880 con 0x7f8e0815b2c0 2024-03-06T07:32:21.849 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:21.849 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:17.782119+0000","modified":"2024-03-06T07:32:20.854485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2399499569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2399499569},{"type":"v1","addr":"172.21.15.114:6837","nonce":2399499569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:32:21.850 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 52 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8de806b4d0 msgr2=0x7f8de806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8de806b4d0 0x7f8de806d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc034000 tx=0x7f8dfc009030 comp rx=0 tx=0).stop 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e0815b2c0 msgr2=0x7f8e0814f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e0815b2c0 0x7f8e0814f3e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8df400e860 tx=0x7f8df400ed20 comp rx=0 tx=0).stop 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 shutdown_connections 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e081fbef0 0x7f8e081fc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8de806b4d0 0x7f8de806d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e0815b2c0 0x7f8e0814f3e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 --2- 172.21.15.114:0/377609177 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8df00039b0 0x7f8e0815ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 >> 172.21.15.114:0/377609177 conn(0x7f8e080b1730 msgr2=0x7f8e080b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 shutdown_connections 2024-03-06T07:32:21.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:21.850+0000 7f8e10000640 1 -- 172.21.15.114:0/377609177 wait complete. 2024-03-06T07:32:21.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.073+0000 7fbdfc400640 1 Processor -- start 2024-03-06T07:32:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.073+0000 7fbdfc400640 1 -- start start 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdfc400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdf415e0d0 0x7fbdf40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdfc400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 0x7fbdf40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdfc400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbdf40d0df0 0x7fbdf4160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdfc400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf40d1260 con 0x7fbdf415e0d0 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdfc400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf40d2f60 con 0x7fbdf40d0df0 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdfc400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40cea60 con 0x7fbdf415f470 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdf9a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdf415e0d0 0x7fbdf40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdf9a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdf415e0d0 0x7fbdf40d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35092/0 (socket says 172.21.15.114:35092) 2024-03-06T07:32:22.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.075+0000 7fbdf9a00640 1 -- 172.21.15.114:0/1687563568 learned_addr learned my addr 172.21.15.114:0/1687563568 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:22.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.076+0000 7fbdf9000640 1 --2- 172.21.15.114:0/1687563568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 0x7fbdf40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.076+0000 7fbdfa400640 1 --2- 172.21.15.114:0/1687563568 >> v2:172.21.15.177:3300/0 conn(0x7fbdf40d0df0 0x7fbdf4160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.076+0000 7fbdf9a00640 1 -- 172.21.15.114:0/1687563568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 msgr2=0x7fbdf40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.076+0000 7fbdf9a00640 1 --2- 172.21.15.114:0/1687563568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 0x7fbdf40d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.076+0000 7fbdf9a00640 1 -- 172.21.15.114:0/1687563568 >> v2:172.21.15.177:3300/0 conn(0x7fbdf40d0df0 msgr2=0x7fbdf4160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.076+0000 7fbdf9a00640 1 --2- 172.21.15.114:0/1687563568 >> v2:172.21.15.177:3300/0 conn(0x7fbdf40d0df0 0x7fbdf4160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.076+0000 7fbdf9a00640 1 -- 172.21.15.114:0/1687563568 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40d2350 con 0x7fbdf415e0d0 2024-03-06T07:32:22.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.077+0000 7fbdf9a00640 1 --2- 172.21.15.114:0/1687563568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf415e0d0 0x7fbdf40d1a90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbdf408cde0 tx=0x7fbde802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b773664994b5f71c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.078+0000 7fbdf3e00640 1 -- 172.21.15.114:0/1687563568 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8034070 con 0x7fbdf415e0d0 2024-03-06T07:32:22.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.078+0000 7fbdf3e00640 1 -- 172.21.15.114:0/1687563568 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8030860 con 0x7fbdf415e0d0 2024-03-06T07:32:22.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.078+0000 7fbdf3e00640 1 -- 172.21.15.114:0/1687563568 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8030c30 con 0x7fbdf415e0d0 2024-03-06T07:32:22.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 -- 172.21.15.114:0/1687563568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf415e0d0 msgr2=0x7fbdf40d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 --2- 172.21.15.114:0/1687563568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf415e0d0 0x7fbdf40d1a90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbdf408cde0 tx=0x7fbde802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 -- 172.21.15.114:0/1687563568 shutdown_connections 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 --2- 172.21.15.114:0/1687563568 >> v2:172.21.15.177:3300/0 conn(0x7fbdf40d0df0 0x7fbdf4160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 --2- 172.21.15.114:0/1687563568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 0x7fbdf40d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 --2- 172.21.15.114:0/1687563568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf415e0d0 0x7fbdf40d1a90 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 -- 172.21.15.114:0/1687563568 >> 172.21.15.114:0/1687563568 conn(0x7fbdf40d3ec0 msgr2=0x7fbdf40d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 -- 172.21.15.114:0/1687563568 shutdown_connections 2024-03-06T07:32:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.079+0000 7fbdfc400640 1 -- 172.21.15.114:0/1687563568 wait complete. 2024-03-06T07:32:22.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.081+0000 7fbdfc400640 1 Processor -- start 2024-03-06T07:32:22.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.081+0000 7fbdfc400640 1 -- start start 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdfc400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf40d0df0 0x7fbdf40b9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdfc400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdf415e0d0 0x7fbdf414e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdfc400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 0x7fbdf40c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdfc400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40d2bf0 con 0x7fbdf40d0df0 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdfc400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40d2910 con 0x7fbdf415e0d0 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdfc400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf415fb30 con 0x7fbdf415f470 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf40d0df0 0x7fbdf40b9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf40d0df0 0x7fbdf40b9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35096/0 (socket says 172.21.15.114:35096) 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 -- 172.21.15.114:0/32746776 learned_addr learned my addr 172.21.15.114:0/32746776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:22.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 -- 172.21.15.114:0/32746776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 msgr2=0x7fbdf40c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 0x7fbdf40c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 -- 172.21.15.114:0/32746776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdf415e0d0 msgr2=0x7fbdf414e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdf415e0d0 0x7fbdf414e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.082+0000 7fbdf9a00640 1 -- 172.21.15.114:0/32746776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40caf90 con 0x7fbdf40d0df0 2024-03-06T07:32:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.083+0000 7fbdf9a00640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf40d0df0 0x7fbdf40b9960 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbde8000c00 tx=0x7fbde8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.083+0000 7fbdf1600640 1 -- 172.21.15.114:0/32746776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8048070 con 0x7fbdf40d0df0 2024-03-06T07:32:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.083+0000 7fbdf1600640 1 -- 172.21.15.114:0/32746776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8030660 con 0x7fbdf40d0df0 2024-03-06T07:32:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.083+0000 7fbdf1600640 1 -- 172.21.15.114:0/32746776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8034040 con 0x7fbdf40d0df0 2024-03-06T07:32:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.083+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf40cb230 con 0x7fbdf40d0df0 2024-03-06T07:32:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.083+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf40cb7f0 con 0x7fbdf40d0df0 2024-03-06T07:32:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.084+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf404f330 con 0x7fbdf40d0df0 2024-03-06T07:32:22.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.085+0000 7fbdf1600640 1 -- 172.21.15.114:0/32746776 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbde804b070 con 0x7fbdf40d0df0 2024-03-06T07:32:22.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.086+0000 7fbdf1600640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbdd406b4d0 0x7fbdd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.086+0000 7fbdf1600640 1 -- 172.21.15.114:0/32746776 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbde80ac2a0 con 0x7fbdf40d0df0 2024-03-06T07:32:22.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.087+0000 7fbdf9000640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbdd406b4d0 0x7fbdd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.087+0000 7fbdf9000640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbdd406b4d0 0x7fbdd406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40cf120 tx=0x7fbde0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.094+0000 7fbdf1600640 1 -- 172.21.15.114:0/32746776 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbde807b240 con 0x7fbdf40d0df0 2024-03-06T07:32:22.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.288+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdf408d920 con 0x7fbdf40d0df0 2024-03-06T07:32:22.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.291+0000 7fbdf1600640 1 -- 172.21.15.114:0/32746776 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16708 (secure 0 0 0) 0x7fbde807ab40 con 0x7fbdf40d0df0 2024-03-06T07:32:22.293 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:22.293 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":80,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:19.344815+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:15.626899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:16.804201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:22.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbdd406b4d0 msgr2=0x7fbdd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbdd406b4d0 0x7fbdd406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40cf120 tx=0x7fbde0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf40d0df0 msgr2=0x7fbdf40b9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf40d0df0 0x7fbdf40b9960 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbde8000c00 tx=0x7fbde8004730 comp rx=0 tx=0).stop 2024-03-06T07:32:22.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 shutdown_connections 2024-03-06T07:32:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbdf415f470 0x7fbdf40c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdf415e0d0 0x7fbdf414e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbdd406b4d0 0x7fbdd406d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 --2- 172.21.15.114:0/32746776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdf40d0df0 0x7fbdf40b9960 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.293+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 >> 172.21.15.114:0/32746776 conn(0x7fbdf40d3ec0 msgr2=0x7fbdf40d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.294+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 shutdown_connections 2024-03-06T07:32:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.294+0000 7fbdfc400640 1 -- 172.21.15.114:0/32746776 wait complete. 2024-03-06T07:32:22.313 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:32:22.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.503+0000 7f59f0600640 1 Processor -- start 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.503+0000 7f59f0600640 1 -- start start 2024-03-06T07:32:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59f0600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f59e80b2dd0 0x7f59e806c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59f0600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 0x7f59e80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59f0600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80d1120 con 0x7f59e81617c0 2024-03-06T07:32:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59f0600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e80cc9d0 con 0x7f59e80b2dd0 2024-03-06T07:32:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59f0600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806c4e0 con 0x7f59e806ceb0 2024-03-06T07:32:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59edc00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f59e81617c0 0x7f59e80cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46266/0 (socket says 172.21.15.114:46266) 2024-03-06T07:32:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59edc00640 1 -- 172.21.15.114:0/2951241063 learned_addr learned my addr 172.21.15.114:0/2951241063 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:22.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59ed200640 1 --2- 172.21.15.114:0/2951241063 >> v2:172.21.15.177:3300/0 conn(0x7f59e80b2dd0 0x7f59e806c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59ee600640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 0x7f59e80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.505+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838747259 0 0) 0x7f59e80d1120 con 0x7f59e81617c0 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d0003520 con 0x7f59e81617c0 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2221576242 0 0) 0x7f59d0003520 con 0x7f59e81617c0 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 msgr2=0x7f59e80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 0x7f59e80ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 >> v2:172.21.15.177:3300/0 conn(0x7f59e80b2dd0 msgr2=0x7f59e806c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 --2- 172.21.15.114:0/2951241063 >> v2:172.21.15.177:3300/0 conn(0x7f59e80b2dd0 0x7f59e806c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80d0b60 con 0x7f59e81617c0 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59ed200640 1 --2- 172.21.15.114:0/2951241063 >> v2:172.21.15.177:3300/0 conn(0x7f59e80b2dd0 0x7f59e806c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f59dc001670 con 0x7f59e81617c0 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 >> v1:172.21.15.114:6789/0 conn(0x7f59e81617c0 legacy=0x7f59e80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0004800 0x7f59d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 0x7f59d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80cc9d0 con 0x7f59d00039b0 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806c4e0 con 0x7f59d0004800 2024-03-06T07:32:22.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d000ba40 con 0x7f59e806ceb0 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.506+0000 7f59edc00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.507+0000 7f59ed200640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0004800 0x7f59d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.507+0000 7f59ee600640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 0x7f59d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.507+0000 7f59edc00640 1 -- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 msgr2=0x7f59d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.507+0000 7f59edc00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e806ceb0 0x7f59d00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.507+0000 7f59edc00640 1 -- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0004800 msgr2=0x7f59d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.507+0000 7f59edc00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0004800 0x7f59d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.507+0000 7f59edc00640 1 -- 172.21.15.114:0/2951241063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0003520 con 0x7f59d00039b0 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.508+0000 7f59edc00640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59d0003dd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c190 tx=0x7f59d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251629c98fcada16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.508+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc007c80 con 0x7f59d00039b0 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.508+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc030070 con 0x7f59d00039b0 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.508+0000 7f59e7e00640 1 -- 172.21.15.114:0/2951241063 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc005cf0 con 0x7f59d00039b0 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.509+0000 7f59f0600640 1 -- 172.21.15.114:0/2951241063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 msgr2=0x7f59d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.509+0000 7f59f0600640 1 --2- 172.21.15.114:0/2951241063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59d0003dd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c190 tx=0x7f59d0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.509+0000 7f59ee600640 1 -- 172.21.15.114:0/2951241063 reap_dead start 2024-03-06T07:32:22.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.509+0000 7f59f0600640 1 -- 172.21.15.114:0/2951241063 shutdown_connections 2024-03-06T07:32:22.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.509+0000 7f59f0600640 1 -- 172.21.15.114:0/2951241063 >> 172.21.15.114:0/2951241063 conn(0x7f59e80b3800 msgr2=0x7f59e80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.509+0000 7f59f0600640 1 -- 172.21.15.114:0/2951241063 shutdown_connections 2024-03-06T07:32:22.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.509+0000 7f59f0600640 1 -- 172.21.15.114:0/2951241063 wait complete. 2024-03-06T07:32:22.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.511+0000 7f59f0600640 1 Processor -- start 2024-03-06T07:32:22.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.511+0000 7f59f0600640 1 -- start start 2024-03-06T07:32:22.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59f0600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59e815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59f0600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e81602d0 0x7f59e815dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59f0600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e815e150 0x7f59e815ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59f0600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80d0830 con 0x7f59d00039b0 2024-03-06T07:32:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59f0600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806c170 con 0x7f59e81602d0 2024-03-06T07:32:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59f0600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806d340 con 0x7f59e815e150 2024-03-06T07:32:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59e815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59e815bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35110/0 (socket says 172.21.15.114:35110) 2024-03-06T07:32:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 -- 172.21.15.114:0/2038493536 learned_addr learned my addr 172.21.15.114:0/2038493536 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 -- 172.21.15.114:0/2038493536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e815e150 msgr2=0x7f59e815ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e815e150 0x7f59e815ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 -- 172.21.15.114:0/2038493536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e81602d0 msgr2=0x7f59e815dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e81602d0 0x7f59e815dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.512+0000 7f59edc00640 1 -- 172.21.15.114:0/2038493536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8160840 con 0x7f59d00039b0 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.513+0000 7f59edc00640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59e815bd00 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f59dc02c4c0 tx=0x7f59dc001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.513+0000 7f59e5600640 1 -- 172.21.15.114:0/2038493536 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc045070 con 0x7f59d00039b0 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.513+0000 7f59e5600640 1 -- 172.21.15.114:0/2038493536 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc004bf0 con 0x7f59d00039b0 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.513+0000 7f59e5600640 1 -- 172.21.15.114:0/2038493536 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc007c80 con 0x7f59d00039b0 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.513+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e8160b20 con 0x7f59d00039b0 2024-03-06T07:32:22.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.513+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e815c240 con 0x7f59d00039b0 2024-03-06T07:32:22.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.515+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e806bbc0 con 0x7f59d00039b0 2024-03-06T07:32:22.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.515+0000 7f59e5600640 1 -- 172.21.15.114:0/2038493536 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59dc007070 con 0x7f59d00039b0 2024-03-06T07:32:22.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.516+0000 7f59e5600640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59c806b4d0 0x7f59c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.516+0000 7f59e5600640 1 -- 172.21.15.114:0/2038493536 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59dc0ab210 con 0x7f59d00039b0 2024-03-06T07:32:22.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.516+0000 7f59ed200640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59c806b4d0 0x7f59c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.517+0000 7f59ed200640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59c806b4d0 0x7f59c806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f59e809fe30 tx=0x7f59d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.520+0000 7f59e5600640 1 -- 172.21.15.114:0/2038493536 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59dc07a100 con 0x7f59d00039b0 2024-03-06T07:32:22.721 INFO:tasks.ceph.mds.c.smithi114.stderr:2024-03-06T07:32:22.719+0000 7f80e4e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:22.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.729+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59e804f330 con 0x7f59c806b4d0 2024-03-06T07:32:22.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.746+0000 7f59e5600640 1 -- 172.21.15.114:0/2038493536 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246753 (secure 0 0 0) 0x7f59e804f330 con 0x7f59c806b4d0 2024-03-06T07:32:22.751 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:22.753 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:32:22.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.750+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59c806b4d0 msgr2=0x7f59c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.750+0000 7f59f0600640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59c806b4d0 0x7f59c806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f59e809fe30 tx=0x7f59d4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.750+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 msgr2=0x7f59e815bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.750+0000 7f59f0600640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59e815bd00 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f59dc02c4c0 tx=0x7f59dc001970 comp rx=0 tx=0).stop 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 shutdown_connections 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59e815e150 0x7f59e815ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59e81602d0 0x7f59e815dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f59c806b4d0 0x7f59c806d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 --2- 172.21.15.114:0/2038493536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59d00039b0 0x7f59e815bd00 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 >> 172.21.15.114:0/2038493536 conn(0x7f59e80b3800 msgr2=0x7f59e8161ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 shutdown_connections 2024-03-06T07:32:22.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:22.751+0000 7f59f0600640 1 -- 172.21.15.114:0/2038493536 wait complete. 2024-03-06T07:32:22.768 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-03-06T07:32:21.180019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258096,"kb_used_data":8648,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716624,"statfs":{"total":773094113280,"available":772829822976,"internally_reserved":0,"allocated":8855552,"data_stored":4412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.874388"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332986+0000","last_change":"2024-03-06T07:32:17.332986+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332986+0000","last_undegraded":"2024-03-06T07:32:17.332986+0000","last_fullsized":"2024-03-06T07:32:17.332986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326645+0000","last_change":"2024-03-06T07:32:17.326645+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326645+0000","last_undegraded":"2024-03-06T07:32:17.326645+0000","last_fullsized":"2024-03-06T07:32:17.326645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317466+0000","last_change":"2024-03-06T07:32:17.317466+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317466+0000","last_undegraded":"2024-03-06T07:32:17.317466+0000","last_fullsized":"2024-03-06T07:32:17.317466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334707+0000","last_change":"2024-03-06T07:32:17.334707+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334707+0000","last_undegraded":"2024-03-06T07:32:17.334707+0000","last_fullsized":"2024-03-06T07:32:17.334707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322485+0000","last_change":"2024-03-06T07:32:17.322485+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322485+0000","last_undegraded":"2024-03-06T07:32:17.322485+0000","last_fullsized":"2024-03-06T07:32:17.322485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316077+0000","last_change":"2024-03-06T07:32:17.316077+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316077+0000","last_undegraded":"2024-03-06T07:32:17.316077+0000","last_fullsized":"2024-03-06T07:32:17.316077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317072+0000","last_change":"2024-03-06T07:32:17.317072+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317072+0000","last_undegraded":"2024-03-06T07:32:17.317072+0000","last_fullsized":"2024-03-06T07:32:17.317072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326283+0000","last_change":"2024-03-06T07:32:17.326283+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326283+0000","last_undegraded":"2024-03-06T07:32:17.326283+0000","last_fullsized":"2024-03-06T07:32:17.326283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331240+0000","last_change":"2024-03-06T07:32:17.331240+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331240+0000","last_undegraded":"2024-03-06T07:32:17.331240+0000","last_fullsized":"2024-03-06T07:32:17.331240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324388+0000","last_change":"2024-03-06T07:32:17.324388+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324388+0000","last_undegraded":"2024-03-06T07:32:17.324388+0000","last_fullsized":"2024-03-06T07:32:17.324388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332062+0000","last_change":"2024-03-06T07:32:17.332062+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332062+0000","last_undegraded":"2024-03-06T07:32:17.332062+0000","last_fullsized":"2024-03-06T07:32:17.332062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327857+0000","last_change":"2024-03-06T07:32:17.327857+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327857+0000","last_undegraded":"2024-03-06T07:32:17.327857+0000","last_fullsized":"2024-03-06T07:32:17.327857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315906+0000","last_change":"2024-03-06T07:32:17.315906+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315906+0000","last_undegraded":"2024-03-06T07:32:17.315906+0000","last_fullsized":"2024-03-06T07:32:17.315906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330155+0000","last_change":"2024-03-06T07:32:17.330155+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330155+0000","last_undegraded":"2024-03-06T07:32:17.330155+0000","last_fullsized":"2024-03-06T07:32:17.330155+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328356+0000","last_change":"2024-03-06T07:32:17.328356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328356+0000","last_undegraded":"2024-03-06T07:32:17.328356+0000","last_fullsized":"2024-03-06T07:32:17.328356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335599+0000","last_change":"2024-03-06T07:32:17.335599+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335599+0000","last_undegraded":"2024-03-06T07:32:17.335599+0000","last_fullsized":"2024-03-06T07:32:17.335599+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316017+0000","last_change":"2024-03-06T07:32:17.316017+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316017+0000","last_undegraded":"2024-03-06T07:32:17.316017+0000","last_fullsized":"2024-03-06T07:32:17.316017+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322552+0000","last_change":"2024-03-06T07:32:17.322552+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322552+0000","last_undegraded":"2024-03-06T07:32:17.322552+0000","last_fullsized":"2024-03-06T07:32:17.322552+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315799+0000","last_change":"2024-03-06T07:32:17.315799+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315799+0000","last_undegraded":"2024-03-06T07:32:17.315799+0000","last_fullsized":"2024-03-06T07:32:17.315799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316054+0000","last_change":"2024-03-06T07:32:17.316054+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316054+0000","last_undegraded":"2024-03-06T07:32:17.316054+0000","last_fullsized":"2024-03-06T07:32:17.316054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323894+0000","last_change":"2024-03-06T07:32:17.323894+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323894+0000","last_undegraded":"2024-03-06T07:32:17.323894+0000","last_fullsized":"2024-03-06T07:32:17.323894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330563+0000","last_change":"2024-03-06T07:32:17.330563+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330563+0000","last_undegraded":"2024-03-06T07:32:17.330563+0000","last_fullsized":"2024-03-06T07:32:17.330563+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315934+0000","last_change":"2024-03-06T07:32:17.315934+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315934+0000","last_undegraded":"2024-03-06T07:32:17.315934+0000","last_fullsized":"2024-03-06T07:32:17.315934+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322220+0000","last_change":"2024-03-06T07:32:17.322220+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322220+0000","last_undegraded":"2024-03-06T07:32:17.322220+0000","last_fullsized":"2024-03-06T07:32:17.322220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327299+0000","last_change":"2024-03-06T07:32:17.327299+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327299+0000","last_undegraded":"2024-03-06T07:32:17.327299+0000","last_fullsized":"2024-03-06T07:32:17.327299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.336090+0000","last_change":"2024-03-06T07:32:17.336090+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.336090+0000","last_undegraded":"2024-03-06T07:32:17.336090+0000","last_fullsized":"2024-03-06T07:32:17.336090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331216+0000","last_change":"2024-03-06T07:32:17.331216+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331216+0000","last_undegraded":"2024-03-06T07:32:17.331216+0000","last_fullsized":"2024-03-06T07:32:17.331216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322253+0000","last_change":"2024-03-06T07:32:17.322253+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322253+0000","last_undegraded":"2024-03-06T07:32:17.322253+0000","last_fullsized":"2024-03-06T07:32:17.322253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323539+0000","last_change":"2024-03-06T07:32:17.323539+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323539+0000","last_undegraded":"2024-03-06T07:32:17.323539+0000","last_fullsized":"2024-03-06T07:32:17.323539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332679+0000","last_change":"2024-03-06T07:32:17.332679+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332679+0000","last_undegraded":"2024-03-06T07:32:17.332679+0000","last_fullsized":"2024-03-06T07:32:17.332679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331770+0000","last_change":"2024-03-06T07:32:17.331770+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331770+0000","last_undegraded":"2024-03-06T07:32:17.331770+0000","last_fullsized":"2024-03-06T07:32:17.331770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338853+0000","last_change":"2024-03-06T07:32:17.338853+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338853+0000","last_undegraded":"2024-03-06T07:32:17.338853+0000","last_fullsized":"2024-03-06T07:32:17.338853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348012+0000","last_change":"2024-03-06T07:32:17.340541+0000","last_active":"2024-03-06T07:32:17.348012+0000","last_peered":"2024-03-06T07:32:17.348012+0000","last_clean":"2024-03-06T07:32:17.348012+0000","last_became_active":"2024-03-06T07:32:17.338327+0000","last_became_peered":"2024-03-06T07:32:17.338327+0000","last_unstale":"2024-03-06T07:32:17.348012+0000","last_undegraded":"2024-03-06T07:32:17.348012+0000","last_fullsized":"2024-03-06T07:32:17.348012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330344+0000","last_change":"2024-03-06T07:32:17.330344+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330344+0000","last_undegraded":"2024-03-06T07:32:17.330344+0000","last_fullsized":"2024-03-06T07:32:17.330344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349286+0000","last_change":"2024-03-06T07:32:17.346963+0000","last_active":"2024-03-06T07:32:17.349286+0000","last_peered":"2024-03-06T07:32:17.349286+0000","last_clean":"2024-03-06T07:32:17.349286+0000","last_became_active":"2024-03-06T07:32:17.346591+0000","last_became_peered":"2024-03-06T07:32:17.346591+0000","last_unstale":"2024-03-06T07:32:17.349286+0000","last_undegraded":"2024-03-06T07:32:17.349286+0000","last_fullsized":"2024-03-06T07:32:17.349286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329514+0000","last_change":"2024-03-06T07:32:17.329514+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329514+0000","last_undegraded":"2024-03-06T07:32:17.329514+0000","last_fullsized":"2024-03-06T07:32:17.329514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337377+0000","last_change":"2024-03-06T07:32:17.334402+0000","last_active":"2024-03-06T07:32:17.337377+0000","last_peered":"2024-03-06T07:32:17.337377+0000","last_clean":"2024-03-06T07:32:17.337377+0000","last_became_active":"2024-03-06T07:32:17.334125+0000","last_became_peered":"2024-03-06T07:32:17.334125+0000","last_unstale":"2024-03-06T07:32:17.337377+0000","last_undegraded":"2024-03-06T07:32:17.337377+0000","last_fullsized":"2024-03-06T07:32:17.337377+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:19.628620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324921+0000","last_change":"2024-03-06T07:32:17.324921+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324921+0000","last_undegraded":"2024-03-06T07:32:17.324921+0000","last_fullsized":"2024-03-06T07:32:17.324921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337396+0000","last_change":"2024-03-06T07:32:17.333923+0000","last_active":"2024-03-06T07:32:17.337396+0000","last_peered":"2024-03-06T07:32:17.337396+0000","last_clean":"2024-03-06T07:32:17.337396+0000","last_became_active":"2024-03-06T07:32:17.333343+0000","last_became_peered":"2024-03-06T07:32:17.333343+0000","last_unstale":"2024-03-06T07:32:17.337396+0000","last_undegraded":"2024-03-06T07:32:17.337396+0000","last_fullsized":"2024-03-06T07:32:17.337396+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.148717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323137+0000","last_change":"2024-03-06T07:32:17.323137+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323137+0000","last_undegraded":"2024-03-06T07:32:17.323137+0000","last_fullsized":"2024-03-06T07:32:17.323137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345919+0000","last_change":"2024-03-06T07:32:17.343520+0000","last_active":"2024-03-06T07:32:17.345919+0000","last_peered":"2024-03-06T07:32:17.345919+0000","last_clean":"2024-03-06T07:32:17.345919+0000","last_became_active":"2024-03-06T07:32:17.343250+0000","last_became_peered":"2024-03-06T07:32:17.343250+0000","last_unstale":"2024-03-06T07:32:17.345919+0000","last_undegraded":"2024-03-06T07:32:17.345919+0000","last_fullsized":"2024-03-06T07:32:17.345919+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334006+0000","last_change":"2024-03-06T07:32:17.334006+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334006+0000","last_undegraded":"2024-03-06T07:32:17.334006+0000","last_fullsized":"2024-03-06T07:32:17.334006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349283+0000","last_change":"2024-03-06T07:32:17.346884+0000","last_active":"2024-03-06T07:32:17.349283+0000","last_peered":"2024-03-06T07:32:17.349283+0000","last_clean":"2024-03-06T07:32:17.349283+0000","last_became_active":"2024-03-06T07:32:17.346080+0000","last_became_peered":"2024-03-06T07:32:17.346080+0000","last_unstale":"2024-03-06T07:32:17.349283+0000","last_undegraded":"2024-03-06T07:32:17.349283+0000","last_fullsized":"2024-03-06T07:32:17.349283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327997+0000","last_change":"2024-03-06T07:32:17.327997+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327997+0000","last_undegraded":"2024-03-06T07:32:17.327997+0000","last_fullsized":"2024-03-06T07:32:17.327997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.339181+0000","last_change":"2024-03-06T07:32:17.336331+0000","last_active":"2024-03-06T07:32:17.339181+0000","last_peered":"2024-03-06T07:32:17.339181+0000","last_clean":"2024-03-06T07:32:17.339181+0000","last_became_active":"2024-03-06T07:32:17.336045+0000","last_became_peered":"2024-03-06T07:32:17.336045+0000","last_unstale":"2024-03-06T07:32:17.339181+0000","last_undegraded":"2024-03-06T07:32:17.339181+0000","last_fullsized":"2024-03-06T07:32:17.339181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:45.515629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329661+0000","last_change":"2024-03-06T07:32:17.329661+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329661+0000","last_undegraded":"2024-03-06T07:32:17.329661+0000","last_fullsized":"2024-03-06T07:32:17.329661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343941+0000","last_change":"2024-03-06T07:32:17.342249+0000","last_active":"2024-03-06T07:32:17.343941+0000","last_peered":"2024-03-06T07:32:17.343941+0000","last_clean":"2024-03-06T07:32:17.343941+0000","last_became_active":"2024-03-06T07:32:17.341962+0000","last_became_peered":"2024-03-06T07:32:17.341962+0000","last_unstale":"2024-03-06T07:32:17.343941+0000","last_undegraded":"2024-03-06T07:32:17.343941+0000","last_fullsized":"2024-03-06T07:32:17.343941+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324876+0000","last_change":"2024-03-06T07:32:17.324876+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324876+0000","last_undegraded":"2024-03-06T07:32:17.324876+0000","last_fullsized":"2024-03-06T07:32:17.324876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343382+0000","last_change":"2024-03-06T07:32:17.337307+0000","last_active":"2024-03-06T07:32:17.343382+0000","last_peered":"2024-03-06T07:32:17.343382+0000","last_clean":"2024-03-06T07:32:17.343382+0000","last_became_active":"2024-03-06T07:32:17.335584+0000","last_became_peered":"2024-03-06T07:32:17.335584+0000","last_unstale":"2024-03-06T07:32:17.343382+0000","last_undegraded":"2024-03-06T07:32:17.343382+0000","last_fullsized":"2024-03-06T07:32:17.343382+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:44.602459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320172+0000","last_change":"2024-03-06T07:32:17.320172+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320172+0000","last_undegraded":"2024-03-06T07:32:17.320172+0000","last_fullsized":"2024-03-06T07:32:17.320172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348146+0000","last_change":"2024-03-06T07:32:17.343053+0000","last_active":"2024-03-06T07:32:17.348146+0000","last_peered":"2024-03-06T07:32:17.348146+0000","last_clean":"2024-03-06T07:32:17.348146+0000","last_became_active":"2024-03-06T07:32:17.342760+0000","last_became_peered":"2024-03-06T07:32:17.342760+0000","last_unstale":"2024-03-06T07:32:17.348146+0000","last_undegraded":"2024-03-06T07:32:17.348146+0000","last_fullsized":"2024-03-06T07:32:17.348146+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323212+0000","last_change":"2024-03-06T07:32:17.323212+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323212+0000","last_undegraded":"2024-03-06T07:32:17.323212+0000","last_fullsized":"2024-03-06T07:32:17.323212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343344+0000","last_change":"2024-03-06T07:32:17.337091+0000","last_active":"2024-03-06T07:32:17.343344+0000","last_peered":"2024-03-06T07:32:17.343344+0000","last_clean":"2024-03-06T07:32:17.343344+0000","last_became_active":"2024-03-06T07:32:17.335040+0000","last_became_peered":"2024-03-06T07:32:17.335040+0000","last_unstale":"2024-03-06T07:32:17.343344+0000","last_undegraded":"2024-03-06T07:32:17.343344+0000","last_fullsized":"2024-03-06T07:32:17.343344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:26.368598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335590+0000","last_change":"2024-03-06T07:32:17.335590+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335590+0000","last_undegraded":"2024-03-06T07:32:17.335590+0000","last_fullsized":"2024-03-06T07:32:17.335590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340689+0000","last_change":"2024-03-06T07:32:17.336112+0000","last_active":"2024-03-06T07:32:17.340689+0000","last_peered":"2024-03-06T07:32:17.340689+0000","last_clean":"2024-03-06T07:32:17.340689+0000","last_became_active":"2024-03-06T07:32:17.335798+0000","last_became_peered":"2024-03-06T07:32:17.335798+0000","last_unstale":"2024-03-06T07:32:17.340689+0000","last_undegraded":"2024-03-06T07:32:17.340689+0000","last_fullsized":"2024-03-06T07:32:17.340689+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:03.187760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325045+0000","last_change":"2024-03-06T07:32:17.325045+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325045+0000","last_undegraded":"2024-03-06T07:32:17.325045+0000","last_fullsized":"2024-03-06T07:32:17.325045+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.331418+0000","last_change":"2024-03-06T07:32:17.328646+0000","last_active":"2024-03-06T07:32:17.331418+0000","last_peered":"2024-03-06T07:32:17.331418+0000","last_clean":"2024-03-06T07:32:17.331418+0000","last_became_active":"2024-03-06T07:32:17.328345+0000","last_became_peered":"2024-03-06T07:32:17.328345+0000","last_unstale":"2024-03-06T07:32:17.331418+0000","last_undegraded":"2024-03-06T07:32:17.331418+0000","last_fullsized":"2024-03-06T07:32:17.331418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.428256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325656+0000","last_change":"2024-03-06T07:32:17.325656+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325656+0000","last_undegraded":"2024-03-06T07:32:17.325656+0000","last_fullsized":"2024-03-06T07:32:17.325656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337375+0000","last_change":"2024-03-06T07:32:17.336488+0000","last_active":"2024-03-06T07:32:17.337375+0000","last_peered":"2024-03-06T07:32:17.337375+0000","last_clean":"2024-03-06T07:32:17.337375+0000","last_became_active":"2024-03-06T07:32:17.336208+0000","last_became_peered":"2024-03-06T07:32:17.336208+0000","last_unstale":"2024-03-06T07:32:17.337375+0000","last_undegraded":"2024-03-06T07:32:17.337375+0000","last_fullsized":"2024-03-06T07:32:17.337375+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.320149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322676+0000","last_change":"2024-03-06T07:32:17.322676+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322676+0000","last_undegraded":"2024-03-06T07:32:17.322676+0000","last_fullsized":"2024-03-06T07:32:17.322676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.347792+0000","last_change":"2024-03-06T07:32:17.344364+0000","last_active":"2024-03-06T07:32:17.347792+0000","last_peered":"2024-03-06T07:32:17.347792+0000","last_clean":"2024-03-06T07:32:17.347792+0000","last_became_active":"2024-03-06T07:32:17.342047+0000","last_became_peered":"2024-03-06T07:32:17.342047+0000","last_unstale":"2024-03-06T07:32:17.347792+0000","last_undegraded":"2024-03-06T07:32:17.347792+0000","last_fullsized":"2024-03-06T07:32:17.347792+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:03.374987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323397+0000","last_change":"2024-03-06T07:32:17.323397+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323397+0000","last_undegraded":"2024-03-06T07:32:17.323397+0000","last_fullsized":"2024-03-06T07:32:17.323397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345205+0000","last_change":"2024-03-06T07:32:17.343791+0000","last_active":"2024-03-06T07:32:17.345205+0000","last_peered":"2024-03-06T07:32:17.345205+0000","last_clean":"2024-03-06T07:32:17.345205+0000","last_became_active":"2024-03-06T07:32:17.343078+0000","last_became_peered":"2024-03-06T07:32:17.343078+0000","last_unstale":"2024-03-06T07:32:17.345205+0000","last_undegraded":"2024-03-06T07:32:17.345205+0000","last_fullsized":"2024-03-06T07:32:17.345205+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:34.532984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325862+0000","last_change":"2024-03-06T07:32:17.325862+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325862+0000","last_undegraded":"2024-03-06T07:32:17.325862+0000","last_fullsized":"2024-03-06T07:32:17.325862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348133+0000","last_change":"2024-03-06T07:32:17.344136+0000","last_active":"2024-03-06T07:32:17.348133+0000","last_peered":"2024-03-06T07:32:17.348133+0000","last_clean":"2024-03-06T07:32:17.348133+0000","last_became_active":"2024-03-06T07:32:17.343838+0000","last_became_peered":"2024-03-06T07:32:17.343838+0000","last_unstale":"2024-03-06T07:32:17.348133+0000","last_undegraded":"2024-03-06T07:32:17.348133+0000","last_fullsized":"2024-03-06T07:32:17.348133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329627+0000","last_change":"2024-03-06T07:32:17.329627+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329627+0000","last_undegraded":"2024-03-06T07:32:17.329627+0000","last_fullsized":"2024-03-06T07:32:17.329627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345189+0000","last_change":"2024-03-06T07:32:17.343396+0000","last_active":"2024-03-06T07:32:17.345189+0000","last_peered":"2024-03-06T07:32:17.345189+0000","last_clean":"2024-03-06T07:32:17.345189+0000","last_became_active":"2024-03-06T07:32:17.343086+0000","last_became_peered":"2024-03-06T07:32:17.343086+0000","last_unstale":"2024-03-06T07:32:17.345189+0000","last_undegraded":"2024-03-06T07:32:17.345189+0000","last_fullsized":"2024-03-06T07:32:17.345189+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:27.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332356+0000","last_change":"2024-03-06T07:32:17.332356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332356+0000","last_undegraded":"2024-03-06T07:32:17.332356+0000","last_fullsized":"2024-03-06T07:32:17.332356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344401+0000","last_change":"2024-03-06T07:32:17.341403+0000","last_active":"2024-03-06T07:32:17.344401+0000","last_peered":"2024-03-06T07:32:17.344401+0000","last_clean":"2024-03-06T07:32:17.344401+0000","last_became_active":"2024-03-06T07:32:17.340813+0000","last_became_peered":"2024-03-06T07:32:17.340813+0000","last_unstale":"2024-03-06T07:32:17.344401+0000","last_undegraded":"2024-03-06T07:32:17.344401+0000","last_fullsized":"2024-03-06T07:32:17.344401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:02.671020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315236+0000","last_change":"2024-03-06T07:32:17.315236+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315236+0000","last_undegraded":"2024-03-06T07:32:17.315236+0000","last_fullsized":"2024-03-06T07:32:17.315236+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340114+0000","last_change":"2024-03-06T07:32:17.338528+0000","last_active":"2024-03-06T07:32:17.340114+0000","last_peered":"2024-03-06T07:32:17.340114+0000","last_clean":"2024-03-06T07:32:17.340114+0000","last_became_active":"2024-03-06T07:32:17.337976+0000","last_became_peered":"2024-03-06T07:32:17.337976+0000","last_unstale":"2024-03-06T07:32:17.340114+0000","last_undegraded":"2024-03-06T07:32:17.340114+0000","last_fullsized":"2024-03-06T07:32:17.340114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:33.417849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320713+0000","last_change":"2024-03-06T07:32:17.320713+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320713+0000","last_undegraded":"2024-03-06T07:32:17.320713+0000","last_fullsized":"2024-03-06T07:32:17.320713+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.336878+0000","last_change":"2024-03-06T07:32:17.333102+0000","last_active":"2024-03-06T07:32:17.336878+0000","last_peered":"2024-03-06T07:32:17.336878+0000","last_clean":"2024-03-06T07:32:17.336878+0000","last_became_active":"2024-03-06T07:32:17.332802+0000","last_became_peered":"2024-03-06T07:32:17.332802+0000","last_unstale":"2024-03-06T07:32:17.336878+0000","last_undegraded":"2024-03-06T07:32:17.336878+0000","last_fullsized":"2024-03-06T07:32:17.336878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:10.165881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.314948+0000","last_change":"2024-03-06T07:32:17.314948+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.314948+0000","last_undegraded":"2024-03-06T07:32:17.314948+0000","last_fullsized":"2024-03-06T07:32:17.314948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342804+0000","last_change":"2024-03-06T07:32:17.340789+0000","last_active":"2024-03-06T07:32:17.342804+0000","last_peered":"2024-03-06T07:32:17.342804+0000","last_clean":"2024-03-06T07:32:17.342804+0000","last_became_active":"2024-03-06T07:32:17.340454+0000","last_became_peered":"2024-03-06T07:32:17.340454+0000","last_unstale":"2024-03-06T07:32:17.342804+0000","last_undegraded":"2024-03-06T07:32:17.342804+0000","last_fullsized":"2024-03-06T07:32:17.342804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:19.107804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318226+0000","last_change":"2024-03-06T07:32:17.318226+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318226+0000","last_undegraded":"2024-03-06T07:32:17.318226+0000","last_fullsized":"2024-03-06T07:32:17.318226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.335552+0000","last_change":"2024-03-06T07:32:17.333757+0000","last_active":"2024-03-06T07:32:17.335552+0000","last_peered":"2024-03-06T07:32:17.335552+0000","last_clean":"2024-03-06T07:32:17.335552+0000","last_became_active":"2024-03-06T07:32:17.333402+0000","last_became_peered":"2024-03-06T07:32:17.333402+0000","last_unstale":"2024-03-06T07:32:17.335552+0000","last_undegraded":"2024-03-06T07:32:17.335552+0000","last_fullsized":"2024-03-06T07:32:17.335552+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:18.884596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338012+0000","last_change":"2024-03-06T07:32:17.338012+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338012+0000","last_undegraded":"2024-03-06T07:32:17.338012+0000","last_fullsized":"2024-03-06T07:32:17.338012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344412+0000","last_change":"2024-03-06T07:32:17.342979+0000","last_active":"2024-03-06T07:32:17.344412+0000","last_peered":"2024-03-06T07:32:17.344412+0000","last_clean":"2024-03-06T07:32:17.344412+0000","last_became_active":"2024-03-06T07:32:17.342679+0000","last_became_peered":"2024-03-06T07:32:17.342679+0000","last_unstale":"2024-03-06T07:32:17.344412+0000","last_undegraded":"2024-03-06T07:32:17.344412+0000","last_fullsized":"2024-03-06T07:32:17.344412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:57.890290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328756+0000","last_change":"2024-03-06T07:32:17.328756+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328756+0000","last_undegraded":"2024-03-06T07:32:17.328756+0000","last_fullsized":"2024-03-06T07:32:17.328756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.332778+0000","last_change":"2024-03-06T07:32:17.331176+0000","last_active":"2024-03-06T07:32:17.332778+0000","last_peered":"2024-03-06T07:32:17.332778+0000","last_clean":"2024-03-06T07:32:17.332778+0000","last_became_active":"2024-03-06T07:32:17.328312+0000","last_became_peered":"2024-03-06T07:32:17.328312+0000","last_unstale":"2024-03-06T07:32:17.332778+0000","last_undegraded":"2024-03-06T07:32:17.332778+0000","last_fullsized":"2024-03-06T07:32:17.332778+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.809897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343166+0000","last_change":"2024-03-06T07:32:17.335023+0000","last_active":"2024-03-06T07:32:17.343166+0000","last_peered":"2024-03-06T07:32:17.343166+0000","last_clean":"2024-03-06T07:32:17.343166+0000","last_became_active":"2024-03-06T07:32:17.334651+0000","last_became_peered":"2024-03-06T07:32:17.334651+0000","last_unstale":"2024-03-06T07:32:17.343166+0000","last_undegraded":"2024-03-06T07:32:17.343166+0000","last_fullsized":"2024-03-06T07:32:17.343166+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:14.380230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331816+0000","last_change":"2024-03-06T07:32:17.331816+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331816+0000","last_undegraded":"2024-03-06T07:32:17.331816+0000","last_fullsized":"2024-03-06T07:32:17.331816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348114+0000","last_change":"2024-03-06T07:32:17.342916+0000","last_active":"2024-03-06T07:32:17.348114+0000","last_peered":"2024-03-06T07:32:17.348114+0000","last_clean":"2024-03-06T07:32:17.348114+0000","last_became_active":"2024-03-06T07:32:17.340684+0000","last_became_peered":"2024-03-06T07:32:17.340684+0000","last_unstale":"2024-03-06T07:32:17.348114+0000","last_undegraded":"2024-03-06T07:32:17.348114+0000","last_fullsized":"2024-03-06T07:32:17.348114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318836+0000","last_change":"2024-03-06T07:32:17.318836+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318836+0000","last_undegraded":"2024-03-06T07:32:17.318836+0000","last_fullsized":"2024-03-06T07:32:17.318836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344424+0000","last_change":"2024-03-06T07:32:17.341757+0000","last_active":"2024-03-06T07:32:17.344424+0000","last_peered":"2024-03-06T07:32:17.344424+0000","last_clean":"2024-03-06T07:32:17.344424+0000","last_became_active":"2024-03-06T07:32:17.341395+0000","last_became_peered":"2024-03-06T07:32:17.341395+0000","last_unstale":"2024-03-06T07:32:17.344424+0000","last_undegraded":"2024-03-06T07:32:17.344424+0000","last_fullsized":"2024-03-06T07:32:17.344424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:43.814597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324328+0000","last_change":"2024-03-06T07:32:17.324328+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324328+0000","last_undegraded":"2024-03-06T07:32:17.324328+0000","last_fullsized":"2024-03-06T07:32:17.324328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343769+0000","last_change":"2024-03-06T07:32:17.341723+0000","last_active":"2024-03-06T07:32:17.343769+0000","last_peered":"2024-03-06T07:32:17.343769+0000","last_clean":"2024-03-06T07:32:17.343769+0000","last_became_active":"2024-03-06T07:32:17.341402+0000","last_became_peered":"2024-03-06T07:32:17.341402+0000","last_unstale":"2024-03-06T07:32:17.343769+0000","last_undegraded":"2024-03-06T07:32:17.343769+0000","last_fullsized":"2024-03-06T07:32:17.343769+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324264+0000","last_change":"2024-03-06T07:32:17.324264+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324264+0000","last_undegraded":"2024-03-06T07:32:17.324264+0000","last_fullsized":"2024-03-06T07:32:17.324264+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345951+0000","last_change":"2024-03-06T07:32:17.338127+0000","last_active":"2024-03-06T07:32:17.345951+0000","last_peered":"2024-03-06T07:32:17.345951+0000","last_clean":"2024-03-06T07:32:17.345951+0000","last_became_active":"2024-03-06T07:32:17.337773+0000","last_became_peered":"2024-03-06T07:32:17.337773+0000","last_unstale":"2024-03-06T07:32:17.345951+0000","last_undegraded":"2024-03-06T07:32:17.345951+0000","last_fullsized":"2024-03-06T07:32:17.345951+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327839+0000","last_change":"2024-03-06T07:32:17.327839+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327839+0000","last_undegraded":"2024-03-06T07:32:17.327839+0000","last_fullsized":"2024-03-06T07:32:17.327839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349318+0000","last_change":"2024-03-06T07:32:17.347045+0000","last_active":"2024-03-06T07:32:17.349318+0000","last_peered":"2024-03-06T07:32:17.349318+0000","last_clean":"2024-03-06T07:32:17.349318+0000","last_became_active":"2024-03-06T07:32:17.346721+0000","last_became_peered":"2024-03-06T07:32:17.346721+0000","last_unstale":"2024-03-06T07:32:17.349318+0000","last_undegraded":"2024-03-06T07:32:17.349318+0000","last_fullsized":"2024-03-06T07:32:17.349318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322723+0000","last_change":"2024-03-06T07:32:17.322723+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322723+0000","last_undegraded":"2024-03-06T07:32:17.322723+0000","last_fullsized":"2024-03-06T07:32:17.322723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342074+0000","last_change":"2024-03-06T07:32:17.339426+0000","last_active":"2024-03-06T07:32:17.342074+0000","last_peered":"2024-03-06T07:32:17.342074+0000","last_clean":"2024-03-06T07:32:17.342074+0000","last_became_active":"2024-03-06T07:32:17.339135+0000","last_became_peered":"2024-03-06T07:32:17.339135+0000","last_unstale":"2024-03-06T07:32:17.342074+0000","last_undegraded":"2024-03-06T07:32:17.342074+0000","last_fullsized":"2024-03-06T07:32:17.342074+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:30.314302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.328500+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:32:17.328500+0000","last_peered":"2024-03-06T07:32:17.328500+0000","last_clean":"2024-03-06T07:32:17.328500+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:32:17.328500+0000","last_undegraded":"2024-03-06T07:32:17.328500+0000","last_fullsized":"2024-03-06T07:32:17.328500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574905,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:22.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:23.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.007+0000 7fc930600640 1 Processor -- start 2024-03-06T07:32:23.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.007+0000 7fc930600640 1 -- start start 2024-03-06T07:32:23.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.008+0000 7fc930600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc92815fe60 0x7fc928071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.008+0000 7fc930600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92806cef0 con 0x7fc928144c00 2024-03-06T07:32:23.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc930600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92806d1d0 con 0x7fc92815f9d0 2024-03-06T07:32:23.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc930600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92806d4b0 con 0x7fc92815fe60 2024-03-06T07:32:23.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc92e600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc92815fe60 0x7fc928071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc92e600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc92815fe60 0x7fc928071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35116/0 (socket says 172.21.15.114:35116) 2024-03-06T07:32:23.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc92e600640 1 -- 172.21.15.114:0/3605226610 learned_addr learned my addr 172.21.15.114:0/3605226610 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:23.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc92dc00640 1 --1- 172.21.15.114:0/3605226610 >> v1:172.21.15.114:6789/0 conn(0x7fc928144c00 0x7fc92806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46278/0 (socket says 172.21.15.114:46278) 2024-03-06T07:32:23.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc92e600640 1 -- 172.21.15.114:0/3605226610 >> v1:172.21.15.177:6789/0 conn(0x7fc92815f9d0 legacy=0x7fc928070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc92e600640 1 -- 172.21.15.114:0/3605226610 >> v1:172.21.15.114:6789/0 conn(0x7fc928144c00 legacy=0x7fc92806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.009+0000 7fc92e600640 1 -- 172.21.15.114:0/3605226610 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92806d790 con 0x7fc92815fe60 2024-03-06T07:32:23.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.010+0000 7fc92e600640 1 --2- 172.21.15.114:0/3605226610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc92815fe60 0x7fc928071220 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc920009e50 tx=0x7fc92002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5df57827c6b19e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc927e00640 1 -- 172.21.15.114:0/3605226610 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92002cb10 con 0x7fc92815fe60 2024-03-06T07:32:23.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc927e00640 1 -- 172.21.15.114:0/3605226610 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92002ccb0 con 0x7fc92815fe60 2024-03-06T07:32:23.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc927e00640 1 -- 172.21.15.114:0/3605226610 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92002ce60 con 0x7fc92815fe60 2024-03-06T07:32:23.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc930600640 1 -- 172.21.15.114:0/3605226610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc92815fe60 msgr2=0x7fc928071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc930600640 1 --2- 172.21.15.114:0/3605226610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc92815fe60 0x7fc928071220 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc920009e50 tx=0x7fc92002bf70 comp rx=0 tx=0).stop 2024-03-06T07:32:23.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc930600640 1 -- 172.21.15.114:0/3605226610 shutdown_connections 2024-03-06T07:32:23.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc930600640 1 --2- 172.21.15.114:0/3605226610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc92815fe60 0x7fc928071220 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.011+0000 7fc930600640 1 -- 172.21.15.114:0/3605226610 >> 172.21.15.114:0/3605226610 conn(0x7fc9280b1730 msgr2=0x7fc9280b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.012+0000 7fc930600640 1 -- 172.21.15.114:0/3605226610 shutdown_connections 2024-03-06T07:32:23.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.012+0000 7fc930600640 1 -- 172.21.15.114:0/3605226610 wait complete. 2024-03-06T07:32:23.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.013+0000 7fc930600640 1 Processor -- start 2024-03-06T07:32:23.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.013+0000 7fc930600640 1 -- start start 2024-03-06T07:32:23.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.014+0000 7fc930600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc928144c00 0x7fc9280b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.014+0000 7fc930600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92815f9d0 0x7fc9280b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.014+0000 7fc930600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92815fe60 0x7fc9281e5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.014+0000 7fc930600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92806cb50 con 0x7fc928144c00 2024-03-06T07:32:23.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.014+0000 7fc930600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc928160730 con 0x7fc92815f9d0 2024-03-06T07:32:23.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.014+0000 7fc930600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92806ccd0 con 0x7fc92815fe60 2024-03-06T07:32:23.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.014+0000 7fc92dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc928144c00 0x7fc9280b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc928144c00 0x7fc9280b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35128/0 (socket says 172.21.15.114:35128) 2024-03-06T07:32:23.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 -- 172.21.15.114:0/1458365132 learned_addr learned my addr 172.21.15.114:0/1458365132 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 -- 172.21.15.114:0/1458365132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92815fe60 msgr2=0x7fc9281e5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92815fe60 0x7fc9281e5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 -- 172.21.15.114:0/1458365132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92815f9d0 msgr2=0x7fc9280b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92815f9d0 0x7fc9280b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 -- 172.21.15.114:0/1458365132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280b4b60 con 0x7fc928144c00 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.015+0000 7fc92dc00640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc928144c00 0x7fc9280b2b30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c3c0 tx=0x7fc91c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.016+0000 7fc925600640 1 -- 172.21.15.114:0/1458365132 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c0074e0 con 0x7fc928144c00 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.016+0000 7fc925600640 1 -- 172.21.15.114:0/1458365132 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c007680 con 0x7fc928144c00 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.016+0000 7fc925600640 1 -- 172.21.15.114:0/1458365132 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c004a00 con 0x7fc928144c00 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.016+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92814a620 con 0x7fc928144c00 2024-03-06T07:32:23.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.016+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92814aba0 con 0x7fc928144c00 2024-03-06T07:32:23.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.017+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92804f330 con 0x7fc928144c00 2024-03-06T07:32:23.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.019+0000 7fc925600640 1 -- 172.21.15.114:0/1458365132 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc91c01a020 con 0x7fc928144c00 2024-03-06T07:32:23.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.020+0000 7fc925600640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90806b500 0x7fc90806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.020+0000 7fc925600640 1 -- 172.21.15.114:0/1458365132 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc91c089a90 con 0x7fc928144c00 2024-03-06T07:32:23.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.020+0000 7fc92d200640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90806b500 0x7fc90806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.021+0000 7fc92d200640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90806b500 0x7fc90806d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc92813c360 tx=0x7fc9140079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.028+0000 7fc925600640 1 -- 172.21.15.114:0/1458365132 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9280b4b60 con 0x7fc928144c00 2024-03-06T07:32:23.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.220+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9280af880 con 0x7fc90806b500 2024-03-06T07:32:23.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.239+0000 7fc925600640 1 -- 172.21.15.114:0/1458365132 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246753 (secure 0 0 0) 0x7fc9280af880 con 0x7fc90806b500 2024-03-06T07:32:23.244 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:23.245 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:32:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.243+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90806b500 msgr2=0x7fc90806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.243+0000 7fc930600640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90806b500 0x7fc90806d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc92813c360 tx=0x7fc9140079b0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc928144c00 msgr2=0x7fc9280b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc928144c00 0x7fc9280b2b30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00c3c0 tx=0x7fc91c00c880 comp rx=0 tx=0).stop 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 shutdown_connections 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc92815fe60 0x7fc9281e5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92815f9d0 0x7fc9280b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc90806b500 0x7fc90806d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 --2- 172.21.15.114:0/1458365132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc928144c00 0x7fc9280b2b30 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 >> 172.21.15.114:0/1458365132 conn(0x7fc9280b1730 msgr2=0x7fc9280aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 shutdown_connections 2024-03-06T07:32:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.244+0000 7fc930600640 1 -- 172.21.15.114:0/1458365132 wait complete. 2024-03-06T07:32:23.263 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-03-06T07:32:23.180584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258096,"kb_used_data":8648,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716624,"statfs":{"total":773094113280,"available":772829822976,"internally_reserved":0,"allocated":8855552,"data_stored":4412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001539"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332986+0000","last_change":"2024-03-06T07:32:17.332986+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332986+0000","last_undegraded":"2024-03-06T07:32:17.332986+0000","last_fullsized":"2024-03-06T07:32:17.332986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326645+0000","last_change":"2024-03-06T07:32:17.326645+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326645+0000","last_undegraded":"2024-03-06T07:32:17.326645+0000","last_fullsized":"2024-03-06T07:32:17.326645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317466+0000","last_change":"2024-03-06T07:32:17.317466+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317466+0000","last_undegraded":"2024-03-06T07:32:17.317466+0000","last_fullsized":"2024-03-06T07:32:17.317466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334707+0000","last_change":"2024-03-06T07:32:17.334707+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334707+0000","last_undegraded":"2024-03-06T07:32:17.334707+0000","last_fullsized":"2024-03-06T07:32:17.334707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322485+0000","last_change":"2024-03-06T07:32:17.322485+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322485+0000","last_undegraded":"2024-03-06T07:32:17.322485+0000","last_fullsized":"2024-03-06T07:32:17.322485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316077+0000","last_change":"2024-03-06T07:32:17.316077+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316077+0000","last_undegraded":"2024-03-06T07:32:17.316077+0000","last_fullsized":"2024-03-06T07:32:17.316077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317072+0000","last_change":"2024-03-06T07:32:17.317072+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317072+0000","last_undegraded":"2024-03-06T07:32:17.317072+0000","last_fullsized":"2024-03-06T07:32:17.317072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326283+0000","last_change":"2024-03-06T07:32:17.326283+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326283+0000","last_undegraded":"2024-03-06T07:32:17.326283+0000","last_fullsized":"2024-03-06T07:32:17.326283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331240+0000","last_change":"2024-03-06T07:32:17.331240+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331240+0000","last_undegraded":"2024-03-06T07:32:17.331240+0000","last_fullsized":"2024-03-06T07:32:17.331240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324388+0000","last_change":"2024-03-06T07:32:17.324388+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324388+0000","last_undegraded":"2024-03-06T07:32:17.324388+0000","last_fullsized":"2024-03-06T07:32:17.324388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332062+0000","last_change":"2024-03-06T07:32:17.332062+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332062+0000","last_undegraded":"2024-03-06T07:32:17.332062+0000","last_fullsized":"2024-03-06T07:32:17.332062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327857+0000","last_change":"2024-03-06T07:32:17.327857+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327857+0000","last_undegraded":"2024-03-06T07:32:17.327857+0000","last_fullsized":"2024-03-06T07:32:17.327857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315906+0000","last_change":"2024-03-06T07:32:17.315906+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315906+0000","last_undegraded":"2024-03-06T07:32:17.315906+0000","last_fullsized":"2024-03-06T07:32:17.315906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330155+0000","last_change":"2024-03-06T07:32:17.330155+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330155+0000","last_undegraded":"2024-03-06T07:32:17.330155+0000","last_fullsized":"2024-03-06T07:32:17.330155+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328356+0000","last_change":"2024-03-06T07:32:17.328356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328356+0000","last_undegraded":"2024-03-06T07:32:17.328356+0000","last_fullsized":"2024-03-06T07:32:17.328356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335599+0000","last_change":"2024-03-06T07:32:17.335599+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335599+0000","last_undegraded":"2024-03-06T07:32:17.335599+0000","last_fullsized":"2024-03-06T07:32:17.335599+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316017+0000","last_change":"2024-03-06T07:32:17.316017+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316017+0000","last_undegraded":"2024-03-06T07:32:17.316017+0000","last_fullsized":"2024-03-06T07:32:17.316017+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322552+0000","last_change":"2024-03-06T07:32:17.322552+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322552+0000","last_undegraded":"2024-03-06T07:32:17.322552+0000","last_fullsized":"2024-03-06T07:32:17.322552+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315799+0000","last_change":"2024-03-06T07:32:17.315799+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315799+0000","last_undegraded":"2024-03-06T07:32:17.315799+0000","last_fullsized":"2024-03-06T07:32:17.315799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316054+0000","last_change":"2024-03-06T07:32:17.316054+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316054+0000","last_undegraded":"2024-03-06T07:32:17.316054+0000","last_fullsized":"2024-03-06T07:32:17.316054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323894+0000","last_change":"2024-03-06T07:32:17.323894+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323894+0000","last_undegraded":"2024-03-06T07:32:17.323894+0000","last_fullsized":"2024-03-06T07:32:17.323894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330563+0000","last_change":"2024-03-06T07:32:17.330563+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330563+0000","last_undegraded":"2024-03-06T07:32:17.330563+0000","last_fullsized":"2024-03-06T07:32:17.330563+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315934+0000","last_change":"2024-03-06T07:32:17.315934+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315934+0000","last_undegraded":"2024-03-06T07:32:17.315934+0000","last_fullsized":"2024-03-06T07:32:17.315934+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322220+0000","last_change":"2024-03-06T07:32:17.322220+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322220+0000","last_undegraded":"2024-03-06T07:32:17.322220+0000","last_fullsized":"2024-03-06T07:32:17.322220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327299+0000","last_change":"2024-03-06T07:32:17.327299+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327299+0000","last_undegraded":"2024-03-06T07:32:17.327299+0000","last_fullsized":"2024-03-06T07:32:17.327299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.336090+0000","last_change":"2024-03-06T07:32:17.336090+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.336090+0000","last_undegraded":"2024-03-06T07:32:17.336090+0000","last_fullsized":"2024-03-06T07:32:17.336090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331216+0000","last_change":"2024-03-06T07:32:17.331216+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331216+0000","last_undegraded":"2024-03-06T07:32:17.331216+0000","last_fullsized":"2024-03-06T07:32:17.331216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322253+0000","last_change":"2024-03-06T07:32:17.322253+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322253+0000","last_undegraded":"2024-03-06T07:32:17.322253+0000","last_fullsized":"2024-03-06T07:32:17.322253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323539+0000","last_change":"2024-03-06T07:32:17.323539+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323539+0000","last_undegraded":"2024-03-06T07:32:17.323539+0000","last_fullsized":"2024-03-06T07:32:17.323539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332679+0000","last_change":"2024-03-06T07:32:17.332679+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332679+0000","last_undegraded":"2024-03-06T07:32:17.332679+0000","last_fullsized":"2024-03-06T07:32:17.332679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331770+0000","last_change":"2024-03-06T07:32:17.331770+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331770+0000","last_undegraded":"2024-03-06T07:32:17.331770+0000","last_fullsized":"2024-03-06T07:32:17.331770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338853+0000","last_change":"2024-03-06T07:32:17.338853+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338853+0000","last_undegraded":"2024-03-06T07:32:17.338853+0000","last_fullsized":"2024-03-06T07:32:17.338853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348012+0000","last_change":"2024-03-06T07:32:17.340541+0000","last_active":"2024-03-06T07:32:17.348012+0000","last_peered":"2024-03-06T07:32:17.348012+0000","last_clean":"2024-03-06T07:32:17.348012+0000","last_became_active":"2024-03-06T07:32:17.338327+0000","last_became_peered":"2024-03-06T07:32:17.338327+0000","last_unstale":"2024-03-06T07:32:17.348012+0000","last_undegraded":"2024-03-06T07:32:17.348012+0000","last_fullsized":"2024-03-06T07:32:17.348012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330344+0000","last_change":"2024-03-06T07:32:17.330344+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330344+0000","last_undegraded":"2024-03-06T07:32:17.330344+0000","last_fullsized":"2024-03-06T07:32:17.330344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349286+0000","last_change":"2024-03-06T07:32:17.346963+0000","last_active":"2024-03-06T07:32:17.349286+0000","last_peered":"2024-03-06T07:32:17.349286+0000","last_clean":"2024-03-06T07:32:17.349286+0000","last_became_active":"2024-03-06T07:32:17.346591+0000","last_became_peered":"2024-03-06T07:32:17.346591+0000","last_unstale":"2024-03-06T07:32:17.349286+0000","last_undegraded":"2024-03-06T07:32:17.349286+0000","last_fullsized":"2024-03-06T07:32:17.349286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329514+0000","last_change":"2024-03-06T07:32:17.329514+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329514+0000","last_undegraded":"2024-03-06T07:32:17.329514+0000","last_fullsized":"2024-03-06T07:32:17.329514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337377+0000","last_change":"2024-03-06T07:32:17.334402+0000","last_active":"2024-03-06T07:32:17.337377+0000","last_peered":"2024-03-06T07:32:17.337377+0000","last_clean":"2024-03-06T07:32:17.337377+0000","last_became_active":"2024-03-06T07:32:17.334125+0000","last_became_peered":"2024-03-06T07:32:17.334125+0000","last_unstale":"2024-03-06T07:32:17.337377+0000","last_undegraded":"2024-03-06T07:32:17.337377+0000","last_fullsized":"2024-03-06T07:32:17.337377+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:19.628620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324921+0000","last_change":"2024-03-06T07:32:17.324921+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324921+0000","last_undegraded":"2024-03-06T07:32:17.324921+0000","last_fullsized":"2024-03-06T07:32:17.324921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337396+0000","last_change":"2024-03-06T07:32:17.333923+0000","last_active":"2024-03-06T07:32:17.337396+0000","last_peered":"2024-03-06T07:32:17.337396+0000","last_clean":"2024-03-06T07:32:17.337396+0000","last_became_active":"2024-03-06T07:32:17.333343+0000","last_became_peered":"2024-03-06T07:32:17.333343+0000","last_unstale":"2024-03-06T07:32:17.337396+0000","last_undegraded":"2024-03-06T07:32:17.337396+0000","last_fullsized":"2024-03-06T07:32:17.337396+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.148717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323137+0000","last_change":"2024-03-06T07:32:17.323137+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323137+0000","last_undegraded":"2024-03-06T07:32:17.323137+0000","last_fullsized":"2024-03-06T07:32:17.323137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345919+0000","last_change":"2024-03-06T07:32:17.343520+0000","last_active":"2024-03-06T07:32:17.345919+0000","last_peered":"2024-03-06T07:32:17.345919+0000","last_clean":"2024-03-06T07:32:17.345919+0000","last_became_active":"2024-03-06T07:32:17.343250+0000","last_became_peered":"2024-03-06T07:32:17.343250+0000","last_unstale":"2024-03-06T07:32:17.345919+0000","last_undegraded":"2024-03-06T07:32:17.345919+0000","last_fullsized":"2024-03-06T07:32:17.345919+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334006+0000","last_change":"2024-03-06T07:32:17.334006+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334006+0000","last_undegraded":"2024-03-06T07:32:17.334006+0000","last_fullsized":"2024-03-06T07:32:17.334006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349283+0000","last_change":"2024-03-06T07:32:17.346884+0000","last_active":"2024-03-06T07:32:17.349283+0000","last_peered":"2024-03-06T07:32:17.349283+0000","last_clean":"2024-03-06T07:32:17.349283+0000","last_became_active":"2024-03-06T07:32:17.346080+0000","last_became_peered":"2024-03-06T07:32:17.346080+0000","last_unstale":"2024-03-06T07:32:17.349283+0000","last_undegraded":"2024-03-06T07:32:17.349283+0000","last_fullsized":"2024-03-06T07:32:17.349283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327997+0000","last_change":"2024-03-06T07:32:17.327997+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327997+0000","last_undegraded":"2024-03-06T07:32:17.327997+0000","last_fullsized":"2024-03-06T07:32:17.327997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.339181+0000","last_change":"2024-03-06T07:32:17.336331+0000","last_active":"2024-03-06T07:32:17.339181+0000","last_peered":"2024-03-06T07:32:17.339181+0000","last_clean":"2024-03-06T07:32:17.339181+0000","last_became_active":"2024-03-06T07:32:17.336045+0000","last_became_peered":"2024-03-06T07:32:17.336045+0000","last_unstale":"2024-03-06T07:32:17.339181+0000","last_undegraded":"2024-03-06T07:32:17.339181+0000","last_fullsized":"2024-03-06T07:32:17.339181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:45.515629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329661+0000","last_change":"2024-03-06T07:32:17.329661+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329661+0000","last_undegraded":"2024-03-06T07:32:17.329661+0000","last_fullsized":"2024-03-06T07:32:17.329661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343941+0000","last_change":"2024-03-06T07:32:17.342249+0000","last_active":"2024-03-06T07:32:17.343941+0000","last_peered":"2024-03-06T07:32:17.343941+0000","last_clean":"2024-03-06T07:32:17.343941+0000","last_became_active":"2024-03-06T07:32:17.341962+0000","last_became_peered":"2024-03-06T07:32:17.341962+0000","last_unstale":"2024-03-06T07:32:17.343941+0000","last_undegraded":"2024-03-06T07:32:17.343941+0000","last_fullsized":"2024-03-06T07:32:17.343941+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324876+0000","last_change":"2024-03-06T07:32:17.324876+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324876+0000","last_undegraded":"2024-03-06T07:32:17.324876+0000","last_fullsized":"2024-03-06T07:32:17.324876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343382+0000","last_change":"2024-03-06T07:32:17.337307+0000","last_active":"2024-03-06T07:32:17.343382+0000","last_peered":"2024-03-06T07:32:17.343382+0000","last_clean":"2024-03-06T07:32:17.343382+0000","last_became_active":"2024-03-06T07:32:17.335584+0000","last_became_peered":"2024-03-06T07:32:17.335584+0000","last_unstale":"2024-03-06T07:32:17.343382+0000","last_undegraded":"2024-03-06T07:32:17.343382+0000","last_fullsized":"2024-03-06T07:32:17.343382+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:44.602459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320172+0000","last_change":"2024-03-06T07:32:17.320172+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320172+0000","last_undegraded":"2024-03-06T07:32:17.320172+0000","last_fullsized":"2024-03-06T07:32:17.320172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348146+0000","last_change":"2024-03-06T07:32:17.343053+0000","last_active":"2024-03-06T07:32:17.348146+0000","last_peered":"2024-03-06T07:32:17.348146+0000","last_clean":"2024-03-06T07:32:17.348146+0000","last_became_active":"2024-03-06T07:32:17.342760+0000","last_became_peered":"2024-03-06T07:32:17.342760+0000","last_unstale":"2024-03-06T07:32:17.348146+0000","last_undegraded":"2024-03-06T07:32:17.348146+0000","last_fullsized":"2024-03-06T07:32:17.348146+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323212+0000","last_change":"2024-03-06T07:32:17.323212+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323212+0000","last_undegraded":"2024-03-06T07:32:17.323212+0000","last_fullsized":"2024-03-06T07:32:17.323212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343344+0000","last_change":"2024-03-06T07:32:17.337091+0000","last_active":"2024-03-06T07:32:17.343344+0000","last_peered":"2024-03-06T07:32:17.343344+0000","last_clean":"2024-03-06T07:32:17.343344+0000","last_became_active":"2024-03-06T07:32:17.335040+0000","last_became_peered":"2024-03-06T07:32:17.335040+0000","last_unstale":"2024-03-06T07:32:17.343344+0000","last_undegraded":"2024-03-06T07:32:17.343344+0000","last_fullsized":"2024-03-06T07:32:17.343344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:26.368598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335590+0000","last_change":"2024-03-06T07:32:17.335590+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335590+0000","last_undegraded":"2024-03-06T07:32:17.335590+0000","last_fullsized":"2024-03-06T07:32:17.335590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340689+0000","last_change":"2024-03-06T07:32:17.336112+0000","last_active":"2024-03-06T07:32:17.340689+0000","last_peered":"2024-03-06T07:32:17.340689+0000","last_clean":"2024-03-06T07:32:17.340689+0000","last_became_active":"2024-03-06T07:32:17.335798+0000","last_became_peered":"2024-03-06T07:32:17.335798+0000","last_unstale":"2024-03-06T07:32:17.340689+0000","last_undegraded":"2024-03-06T07:32:17.340689+0000","last_fullsized":"2024-03-06T07:32:17.340689+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:03.187760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325045+0000","last_change":"2024-03-06T07:32:17.325045+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325045+0000","last_undegraded":"2024-03-06T07:32:17.325045+0000","last_fullsized":"2024-03-06T07:32:17.325045+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.331418+0000","last_change":"2024-03-06T07:32:17.328646+0000","last_active":"2024-03-06T07:32:17.331418+0000","last_peered":"2024-03-06T07:32:17.331418+0000","last_clean":"2024-03-06T07:32:17.331418+0000","last_became_active":"2024-03-06T07:32:17.328345+0000","last_became_peered":"2024-03-06T07:32:17.328345+0000","last_unstale":"2024-03-06T07:32:17.331418+0000","last_undegraded":"2024-03-06T07:32:17.331418+0000","last_fullsized":"2024-03-06T07:32:17.331418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.428256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325656+0000","last_change":"2024-03-06T07:32:17.325656+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325656+0000","last_undegraded":"2024-03-06T07:32:17.325656+0000","last_fullsized":"2024-03-06T07:32:17.325656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337375+0000","last_change":"2024-03-06T07:32:17.336488+0000","last_active":"2024-03-06T07:32:17.337375+0000","last_peered":"2024-03-06T07:32:17.337375+0000","last_clean":"2024-03-06T07:32:17.337375+0000","last_became_active":"2024-03-06T07:32:17.336208+0000","last_became_peered":"2024-03-06T07:32:17.336208+0000","last_unstale":"2024-03-06T07:32:17.337375+0000","last_undegraded":"2024-03-06T07:32:17.337375+0000","last_fullsized":"2024-03-06T07:32:17.337375+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.320149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322676+0000","last_change":"2024-03-06T07:32:17.322676+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322676+0000","last_undegraded":"2024-03-06T07:32:17.322676+0000","last_fullsized":"2024-03-06T07:32:17.322676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.347792+0000","last_change":"2024-03-06T07:32:17.344364+0000","last_active":"2024-03-06T07:32:17.347792+0000","last_peered":"2024-03-06T07:32:17.347792+0000","last_clean":"2024-03-06T07:32:17.347792+0000","last_became_active":"2024-03-06T07:32:17.342047+0000","last_became_peered":"2024-03-06T07:32:17.342047+0000","last_unstale":"2024-03-06T07:32:17.347792+0000","last_undegraded":"2024-03-06T07:32:17.347792+0000","last_fullsized":"2024-03-06T07:32:17.347792+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:03.374987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323397+0000","last_change":"2024-03-06T07:32:17.323397+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323397+0000","last_undegraded":"2024-03-06T07:32:17.323397+0000","last_fullsized":"2024-03-06T07:32:17.323397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345205+0000","last_change":"2024-03-06T07:32:17.343791+0000","last_active":"2024-03-06T07:32:17.345205+0000","last_peered":"2024-03-06T07:32:17.345205+0000","last_clean":"2024-03-06T07:32:17.345205+0000","last_became_active":"2024-03-06T07:32:17.343078+0000","last_became_peered":"2024-03-06T07:32:17.343078+0000","last_unstale":"2024-03-06T07:32:17.345205+0000","last_undegraded":"2024-03-06T07:32:17.345205+0000","last_fullsized":"2024-03-06T07:32:17.345205+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:34.532984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325862+0000","last_change":"2024-03-06T07:32:17.325862+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325862+0000","last_undegraded":"2024-03-06T07:32:17.325862+0000","last_fullsized":"2024-03-06T07:32:17.325862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348133+0000","last_change":"2024-03-06T07:32:17.344136+0000","last_active":"2024-03-06T07:32:17.348133+0000","last_peered":"2024-03-06T07:32:17.348133+0000","last_clean":"2024-03-06T07:32:17.348133+0000","last_became_active":"2024-03-06T07:32:17.343838+0000","last_became_peered":"2024-03-06T07:32:17.343838+0000","last_unstale":"2024-03-06T07:32:17.348133+0000","last_undegraded":"2024-03-06T07:32:17.348133+0000","last_fullsized":"2024-03-06T07:32:17.348133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329627+0000","last_change":"2024-03-06T07:32:17.329627+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329627+0000","last_undegraded":"2024-03-06T07:32:17.329627+0000","last_fullsized":"2024-03-06T07:32:17.329627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345189+0000","last_change":"2024-03-06T07:32:17.343396+0000","last_active":"2024-03-06T07:32:17.345189+0000","last_peered":"2024-03-06T07:32:17.345189+0000","last_clean":"2024-03-06T07:32:17.345189+0000","last_became_active":"2024-03-06T07:32:17.343086+0000","last_became_peered":"2024-03-06T07:32:17.343086+0000","last_unstale":"2024-03-06T07:32:17.345189+0000","last_undegraded":"2024-03-06T07:32:17.345189+0000","last_fullsized":"2024-03-06T07:32:17.345189+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:27.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332356+0000","last_change":"2024-03-06T07:32:17.332356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332356+0000","last_undegraded":"2024-03-06T07:32:17.332356+0000","last_fullsized":"2024-03-06T07:32:17.332356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344401+0000","last_change":"2024-03-06T07:32:17.341403+0000","last_active":"2024-03-06T07:32:17.344401+0000","last_peered":"2024-03-06T07:32:17.344401+0000","last_clean":"2024-03-06T07:32:17.344401+0000","last_became_active":"2024-03-06T07:32:17.340813+0000","last_became_peered":"2024-03-06T07:32:17.340813+0000","last_unstale":"2024-03-06T07:32:17.344401+0000","last_undegraded":"2024-03-06T07:32:17.344401+0000","last_fullsized":"2024-03-06T07:32:17.344401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:02.671020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315236+0000","last_change":"2024-03-06T07:32:17.315236+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315236+0000","last_undegraded":"2024-03-06T07:32:17.315236+0000","last_fullsized":"2024-03-06T07:32:17.315236+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340114+0000","last_change":"2024-03-06T07:32:17.338528+0000","last_active":"2024-03-06T07:32:17.340114+0000","last_peered":"2024-03-06T07:32:17.340114+0000","last_clean":"2024-03-06T07:32:17.340114+0000","last_became_active":"2024-03-06T07:32:17.337976+0000","last_became_peered":"2024-03-06T07:32:17.337976+0000","last_unstale":"2024-03-06T07:32:17.340114+0000","last_undegraded":"2024-03-06T07:32:17.340114+0000","last_fullsized":"2024-03-06T07:32:17.340114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:33.417849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320713+0000","last_change":"2024-03-06T07:32:17.320713+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320713+0000","last_undegraded":"2024-03-06T07:32:17.320713+0000","last_fullsized":"2024-03-06T07:32:17.320713+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.336878+0000","last_change":"2024-03-06T07:32:17.333102+0000","last_active":"2024-03-06T07:32:17.336878+0000","last_peered":"2024-03-06T07:32:17.336878+0000","last_clean":"2024-03-06T07:32:17.336878+0000","last_became_active":"2024-03-06T07:32:17.332802+0000","last_became_peered":"2024-03-06T07:32:17.332802+0000","last_unstale":"2024-03-06T07:32:17.336878+0000","last_undegraded":"2024-03-06T07:32:17.336878+0000","last_fullsized":"2024-03-06T07:32:17.336878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:10.165881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.314948+0000","last_change":"2024-03-06T07:32:17.314948+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.314948+0000","last_undegraded":"2024-03-06T07:32:17.314948+0000","last_fullsized":"2024-03-06T07:32:17.314948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342804+0000","last_change":"2024-03-06T07:32:17.340789+0000","last_active":"2024-03-06T07:32:17.342804+0000","last_peered":"2024-03-06T07:32:17.342804+0000","last_clean":"2024-03-06T07:32:17.342804+0000","last_became_active":"2024-03-06T07:32:17.340454+0000","last_became_peered":"2024-03-06T07:32:17.340454+0000","last_unstale":"2024-03-06T07:32:17.342804+0000","last_undegraded":"2024-03-06T07:32:17.342804+0000","last_fullsized":"2024-03-06T07:32:17.342804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:19.107804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318226+0000","last_change":"2024-03-06T07:32:17.318226+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318226+0000","last_undegraded":"2024-03-06T07:32:17.318226+0000","last_fullsized":"2024-03-06T07:32:17.318226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.335552+0000","last_change":"2024-03-06T07:32:17.333757+0000","last_active":"2024-03-06T07:32:17.335552+0000","last_peered":"2024-03-06T07:32:17.335552+0000","last_clean":"2024-03-06T07:32:17.335552+0000","last_became_active":"2024-03-06T07:32:17.333402+0000","last_became_peered":"2024-03-06T07:32:17.333402+0000","last_unstale":"2024-03-06T07:32:17.335552+0000","last_undegraded":"2024-03-06T07:32:17.335552+0000","last_fullsized":"2024-03-06T07:32:17.335552+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:18.884596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338012+0000","last_change":"2024-03-06T07:32:17.338012+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338012+0000","last_undegraded":"2024-03-06T07:32:17.338012+0000","last_fullsized":"2024-03-06T07:32:17.338012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344412+0000","last_change":"2024-03-06T07:32:17.342979+0000","last_active":"2024-03-06T07:32:17.344412+0000","last_peered":"2024-03-06T07:32:17.344412+0000","last_clean":"2024-03-06T07:32:17.344412+0000","last_became_active":"2024-03-06T07:32:17.342679+0000","last_became_peered":"2024-03-06T07:32:17.342679+0000","last_unstale":"2024-03-06T07:32:17.344412+0000","last_undegraded":"2024-03-06T07:32:17.344412+0000","last_fullsized":"2024-03-06T07:32:17.344412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:57.890290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328756+0000","last_change":"2024-03-06T07:32:17.328756+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328756+0000","last_undegraded":"2024-03-06T07:32:17.328756+0000","last_fullsized":"2024-03-06T07:32:17.328756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.332778+0000","last_change":"2024-03-06T07:32:17.331176+0000","last_active":"2024-03-06T07:32:17.332778+0000","last_peered":"2024-03-06T07:32:17.332778+0000","last_clean":"2024-03-06T07:32:17.332778+0000","last_became_active":"2024-03-06T07:32:17.328312+0000","last_became_peered":"2024-03-06T07:32:17.328312+0000","last_unstale":"2024-03-06T07:32:17.332778+0000","last_undegraded":"2024-03-06T07:32:17.332778+0000","last_fullsized":"2024-03-06T07:32:17.332778+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.809897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343166+0000","last_change":"2024-03-06T07:32:17.335023+0000","last_active":"2024-03-06T07:32:17.343166+0000","last_peered":"2024-03-06T07:32:17.343166+0000","last_clean":"2024-03-06T07:32:17.343166+0000","last_became_active":"2024-03-06T07:32:17.334651+0000","last_became_peered":"2024-03-06T07:32:17.334651+0000","last_unstale":"2024-03-06T07:32:17.343166+0000","last_undegraded":"2024-03-06T07:32:17.343166+0000","last_fullsized":"2024-03-06T07:32:17.343166+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:14.380230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331816+0000","last_change":"2024-03-06T07:32:17.331816+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331816+0000","last_undegraded":"2024-03-06T07:32:17.331816+0000","last_fullsized":"2024-03-06T07:32:17.331816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348114+0000","last_change":"2024-03-06T07:32:17.342916+0000","last_active":"2024-03-06T07:32:17.348114+0000","last_peered":"2024-03-06T07:32:17.348114+0000","last_clean":"2024-03-06T07:32:17.348114+0000","last_became_active":"2024-03-06T07:32:17.340684+0000","last_became_peered":"2024-03-06T07:32:17.340684+0000","last_unstale":"2024-03-06T07:32:17.348114+0000","last_undegraded":"2024-03-06T07:32:17.348114+0000","last_fullsized":"2024-03-06T07:32:17.348114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318836+0000","last_change":"2024-03-06T07:32:17.318836+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318836+0000","last_undegraded":"2024-03-06T07:32:17.318836+0000","last_fullsized":"2024-03-06T07:32:17.318836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344424+0000","last_change":"2024-03-06T07:32:17.341757+0000","last_active":"2024-03-06T07:32:17.344424+0000","last_peered":"2024-03-06T07:32:17.344424+0000","last_clean":"2024-03-06T07:32:17.344424+0000","last_became_active":"2024-03-06T07:32:17.341395+0000","last_became_peered":"2024-03-06T07:32:17.341395+0000","last_unstale":"2024-03-06T07:32:17.344424+0000","last_undegraded":"2024-03-06T07:32:17.344424+0000","last_fullsized":"2024-03-06T07:32:17.344424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:43.814597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324328+0000","last_change":"2024-03-06T07:32:17.324328+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324328+0000","last_undegraded":"2024-03-06T07:32:17.324328+0000","last_fullsized":"2024-03-06T07:32:17.324328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343769+0000","last_change":"2024-03-06T07:32:17.341723+0000","last_active":"2024-03-06T07:32:17.343769+0000","last_peered":"2024-03-06T07:32:17.343769+0000","last_clean":"2024-03-06T07:32:17.343769+0000","last_became_active":"2024-03-06T07:32:17.341402+0000","last_became_peered":"2024-03-06T07:32:17.341402+0000","last_unstale":"2024-03-06T07:32:17.343769+0000","last_undegraded":"2024-03-06T07:32:17.343769+0000","last_fullsized":"2024-03-06T07:32:17.343769+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324264+0000","last_change":"2024-03-06T07:32:17.324264+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324264+0000","last_undegraded":"2024-03-06T07:32:17.324264+0000","last_fullsized":"2024-03-06T07:32:17.324264+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345951+0000","last_change":"2024-03-06T07:32:17.338127+0000","last_active":"2024-03-06T07:32:17.345951+0000","last_peered":"2024-03-06T07:32:17.345951+0000","last_clean":"2024-03-06T07:32:17.345951+0000","last_became_active":"2024-03-06T07:32:17.337773+0000","last_became_peered":"2024-03-06T07:32:17.337773+0000","last_unstale":"2024-03-06T07:32:17.345951+0000","last_undegraded":"2024-03-06T07:32:17.345951+0000","last_fullsized":"2024-03-06T07:32:17.345951+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327839+0000","last_change":"2024-03-06T07:32:17.327839+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327839+0000","last_undegraded":"2024-03-06T07:32:17.327839+0000","last_fullsized":"2024-03-06T07:32:17.327839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349318+0000","last_change":"2024-03-06T07:32:17.347045+0000","last_active":"2024-03-06T07:32:17.349318+0000","last_peered":"2024-03-06T07:32:17.349318+0000","last_clean":"2024-03-06T07:32:17.349318+0000","last_became_active":"2024-03-06T07:32:17.346721+0000","last_became_peered":"2024-03-06T07:32:17.346721+0000","last_unstale":"2024-03-06T07:32:17.349318+0000","last_undegraded":"2024-03-06T07:32:17.349318+0000","last_fullsized":"2024-03-06T07:32:17.349318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322723+0000","last_change":"2024-03-06T07:32:17.322723+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322723+0000","last_undegraded":"2024-03-06T07:32:17.322723+0000","last_fullsized":"2024-03-06T07:32:17.322723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342074+0000","last_change":"2024-03-06T07:32:17.339426+0000","last_active":"2024-03-06T07:32:17.342074+0000","last_peered":"2024-03-06T07:32:17.342074+0000","last_clean":"2024-03-06T07:32:17.342074+0000","last_became_active":"2024-03-06T07:32:17.339135+0000","last_became_peered":"2024-03-06T07:32:17.339135+0000","last_unstale":"2024-03-06T07:32:17.342074+0000","last_undegraded":"2024-03-06T07:32:17.342074+0000","last_fullsized":"2024-03-06T07:32:17.342074+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:30.314302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.328500+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:32:17.328500+0000","last_peered":"2024-03-06T07:32:17.328500+0000","last_clean":"2024-03-06T07:32:17.328500+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:32:17.328500+0000","last_undegraded":"2024-03-06T07:32:17.328500+0000","last_fullsized":"2024-03-06T07:32:17.328500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574905,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:23.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:23.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.508+0000 7fde68a00640 1 Processor -- start 2024-03-06T07:32:23.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.508+0000 7fde68a00640 1 -- start start 2024-03-06T07:32:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.509+0000 7fde68a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde640b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.509+0000 7fde68a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fde640b4680 0x7fde6406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde68a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6415b090 con 0x7fde64072340 2024-03-06T07:32:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde68a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde640b2a30 con 0x7fde640b4680 2024-03-06T07:32:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde68a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde640b4360 con 0x7fde6415acb0 2024-03-06T07:32:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde5be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fde640b4680 0x7fde6406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde62800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fde64072340 0x7fde640ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46286/0 (socket says 172.21.15.114:46286) 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde5be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fde640b4680 0x7fde6406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35134/0 (socket says 172.21.15.114:35134) 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde62800640 1 -- 172.21.15.114:0/2016927387 learned_addr learned my addr 172.21.15.114:0/2016927387 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde61e00640 1 --2- 172.21.15.114:0/2016927387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde640b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde5be00640 1 -- 172.21.15.114:0/2016927387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 msgr2=0x7fde640b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde5be00640 1 --2- 172.21.15.114:0/2016927387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde640b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.510+0000 7fde5be00640 1 -- 172.21.15.114:0/2016927387 >> v1:172.21.15.114:6789/0 conn(0x7fde64072340 legacy=0x7fde640ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.511+0000 7fde5be00640 1 -- 172.21.15.114:0/2016927387 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640b2cf0 con 0x7fde640b4680 2024-03-06T07:32:23.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.511+0000 7fde61400640 1 -- 172.21.15.114:0/2016927387 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786843209 0 0) 0x7fde6415b090 con 0x7fde64072340 2024-03-06T07:32:23.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.512+0000 7fde5be00640 1 --2- 172.21.15.114:0/2016927387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde6406e4f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0097a0 tx=0x7fde4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f55ddd11f55cbdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.512+0000 7fde61400640 1 -- 172.21.15.114:0/2016927387 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c035020 con 0x7fde640b4680 2024-03-06T07:32:23.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.512+0000 7fde61400640 1 -- 172.21.15.114:0/2016927387 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c031830 con 0x7fde640b4680 2024-03-06T07:32:23.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.512+0000 7fde61400640 1 -- 172.21.15.114:0/2016927387 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c031c00 con 0x7fde640b4680 2024-03-06T07:32:23.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.512+0000 7fde68a00640 1 -- 172.21.15.114:0/2016927387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 msgr2=0x7fde6406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.512+0000 7fde68a00640 1 --2- 172.21.15.114:0/2016927387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde6406e4f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0097a0 tx=0x7fde4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.513+0000 7fde68a00640 1 -- 172.21.15.114:0/2016927387 shutdown_connections 2024-03-06T07:32:23.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.513+0000 7fde68a00640 1 --2- 172.21.15.114:0/2016927387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde6406e4f0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.513+0000 7fde68a00640 1 --2- 172.21.15.114:0/2016927387 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde640b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.513+0000 7fde68a00640 1 -- 172.21.15.114:0/2016927387 >> 172.21.15.114:0/2016927387 conn(0x7fde640b1730 msgr2=0x7fde640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.513+0000 7fde68a00640 1 -- 172.21.15.114:0/2016927387 shutdown_connections 2024-03-06T07:32:23.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.513+0000 7fde68a00640 1 -- 172.21.15.114:0/2016927387 wait complete. 2024-03-06T07:32:23.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.514+0000 7fde68a00640 1 Processor -- start 2024-03-06T07:32:23.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.514+0000 7fde68a00640 1 -- start start 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.515+0000 7fde68a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde64072340 0x7fde6414ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.515+0000 7fde68a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde641361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.515+0000 7fde68a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde641507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.515+0000 7fde68a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640b3d10 con 0x7fde640b4680 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.515+0000 7fde68a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640b32b0 con 0x7fde64072340 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.515+0000 7fde68a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde640b4010 con 0x7fde6415acb0 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde641361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde641361b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35140/0 (socket says 172.21.15.114:35140) 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 -- 172.21.15.114:0/1233071768 learned_addr learned my addr 172.21.15.114:0/1233071768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 -- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 msgr2=0x7fde641507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde62800640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde641507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde61e00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde64072340 0x7fde6414ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde641507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 -- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde64072340 msgr2=0x7fde6414ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde64072340 0x7fde6414ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.516+0000 7fde5be00640 1 -- 172.21.15.114:0/1233071768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64134300 con 0x7fde640b4680 2024-03-06T07:32:23.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.517+0000 7fde5be00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde641361b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fde4c02bee0 tx=0x7fde4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.517+0000 7fde5a000640 1 -- 172.21.15.114:0/1233071768 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c041070 con 0x7fde640b4680 2024-03-06T07:32:23.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.517+0000 7fde5a000640 1 -- 172.21.15.114:0/1233071768 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c03b920 con 0x7fde640b4680 2024-03-06T07:32:23.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.517+0000 7fde5a000640 1 -- 172.21.15.114:0/1233071768 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c035040 con 0x7fde640b4680 2024-03-06T07:32:23.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.517+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde64136c60 con 0x7fde640b4680 2024-03-06T07:32:23.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.517+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde641345c0 con 0x7fde640b4680 2024-03-06T07:32:23.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.518+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6404f330 con 0x7fde640b4680 2024-03-06T07:32:23.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.519+0000 7fde5a000640 1 -- 172.21.15.114:0/1233071768 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde4c04c060 con 0x7fde640b4680 2024-03-06T07:32:23.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.520+0000 7fde5a000640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fde4006b4d0 0x7fde4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.520+0000 7fde5a000640 1 -- 172.21.15.114:0/1233071768 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde4c0abca0 con 0x7fde640b4680 2024-03-06T07:32:23.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.520+0000 7fde61e00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fde4006b4d0 0x7fde4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.521+0000 7fde61e00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fde4006b4d0 0x7fde4006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fde54006fd0 tx=0x7fde54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.524+0000 7fde5a000640 1 -- 172.21.15.114:0/1233071768 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde4c07aa90 con 0x7fde640b4680 2024-03-06T07:32:23.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.735+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fde64137180 con 0x7fde4006b4d0 2024-03-06T07:32:23.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.753+0000 7fde5a000640 1 -- 172.21.15.114:0/1233071768 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246753 (secure 0 0 0) 0x7fde54009d60 con 0x7fde4006b4d0 2024-03-06T07:32:23.758 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:23.760 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:32:23.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fde4006b4d0 msgr2=0x7fde4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fde4006b4d0 0x7fde4006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fde54006fd0 tx=0x7fde54008040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 msgr2=0x7fde641361b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde641361b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fde4c02bee0 tx=0x7fde4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 shutdown_connections 2024-03-06T07:32:23.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde6415acb0 0x7fde641507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fde4006b4d0 0x7fde4006d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde640b4680 0x7fde641361b0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 --2- 172.21.15.114:0/1233071768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde64072340 0x7fde6414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 >> 172.21.15.114:0/1233071768 conn(0x7fde640b1730 msgr2=0x7fde640ae750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.759+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 shutdown_connections 2024-03-06T07:32:23.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:23.760+0000 7fde68a00640 1 -- 172.21.15.114:0/1233071768 wait complete. 2024-03-06T07:32:23.783 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-03-06T07:32:23.180584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258096,"kb_used_data":8648,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716624,"statfs":{"total":773094113280,"available":772829822976,"internally_reserved":0,"allocated":8855552,"data_stored":4412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001539"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332986+0000","last_change":"2024-03-06T07:32:17.332986+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332986+0000","last_undegraded":"2024-03-06T07:32:17.332986+0000","last_fullsized":"2024-03-06T07:32:17.332986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326645+0000","last_change":"2024-03-06T07:32:17.326645+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326645+0000","last_undegraded":"2024-03-06T07:32:17.326645+0000","last_fullsized":"2024-03-06T07:32:17.326645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317466+0000","last_change":"2024-03-06T07:32:17.317466+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317466+0000","last_undegraded":"2024-03-06T07:32:17.317466+0000","last_fullsized":"2024-03-06T07:32:17.317466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334707+0000","last_change":"2024-03-06T07:32:17.334707+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334707+0000","last_undegraded":"2024-03-06T07:32:17.334707+0000","last_fullsized":"2024-03-06T07:32:17.334707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322485+0000","last_change":"2024-03-06T07:32:17.322485+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322485+0000","last_undegraded":"2024-03-06T07:32:17.322485+0000","last_fullsized":"2024-03-06T07:32:17.322485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316077+0000","last_change":"2024-03-06T07:32:17.316077+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316077+0000","last_undegraded":"2024-03-06T07:32:17.316077+0000","last_fullsized":"2024-03-06T07:32:17.316077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317072+0000","last_change":"2024-03-06T07:32:17.317072+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317072+0000","last_undegraded":"2024-03-06T07:32:17.317072+0000","last_fullsized":"2024-03-06T07:32:17.317072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326283+0000","last_change":"2024-03-06T07:32:17.326283+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326283+0000","last_undegraded":"2024-03-06T07:32:17.326283+0000","last_fullsized":"2024-03-06T07:32:17.326283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331240+0000","last_change":"2024-03-06T07:32:17.331240+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331240+0000","last_undegraded":"2024-03-06T07:32:17.331240+0000","last_fullsized":"2024-03-06T07:32:17.331240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324388+0000","last_change":"2024-03-06T07:32:17.324388+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324388+0000","last_undegraded":"2024-03-06T07:32:17.324388+0000","last_fullsized":"2024-03-06T07:32:17.324388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332062+0000","last_change":"2024-03-06T07:32:17.332062+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332062+0000","last_undegraded":"2024-03-06T07:32:17.332062+0000","last_fullsized":"2024-03-06T07:32:17.332062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327857+0000","last_change":"2024-03-06T07:32:17.327857+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327857+0000","last_undegraded":"2024-03-06T07:32:17.327857+0000","last_fullsized":"2024-03-06T07:32:17.327857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315906+0000","last_change":"2024-03-06T07:32:17.315906+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315906+0000","last_undegraded":"2024-03-06T07:32:17.315906+0000","last_fullsized":"2024-03-06T07:32:17.315906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330155+0000","last_change":"2024-03-06T07:32:17.330155+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330155+0000","last_undegraded":"2024-03-06T07:32:17.330155+0000","last_fullsized":"2024-03-06T07:32:17.330155+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328356+0000","last_change":"2024-03-06T07:32:17.328356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328356+0000","last_undegraded":"2024-03-06T07:32:17.328356+0000","last_fullsized":"2024-03-06T07:32:17.328356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335599+0000","last_change":"2024-03-06T07:32:17.335599+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335599+0000","last_undegraded":"2024-03-06T07:32:17.335599+0000","last_fullsized":"2024-03-06T07:32:17.335599+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316017+0000","last_change":"2024-03-06T07:32:17.316017+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316017+0000","last_undegraded":"2024-03-06T07:32:17.316017+0000","last_fullsized":"2024-03-06T07:32:17.316017+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322552+0000","last_change":"2024-03-06T07:32:17.322552+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322552+0000","last_undegraded":"2024-03-06T07:32:17.322552+0000","last_fullsized":"2024-03-06T07:32:17.322552+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315799+0000","last_change":"2024-03-06T07:32:17.315799+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315799+0000","last_undegraded":"2024-03-06T07:32:17.315799+0000","last_fullsized":"2024-03-06T07:32:17.315799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316054+0000","last_change":"2024-03-06T07:32:17.316054+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316054+0000","last_undegraded":"2024-03-06T07:32:17.316054+0000","last_fullsized":"2024-03-06T07:32:17.316054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323894+0000","last_change":"2024-03-06T07:32:17.323894+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323894+0000","last_undegraded":"2024-03-06T07:32:17.323894+0000","last_fullsized":"2024-03-06T07:32:17.323894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330563+0000","last_change":"2024-03-06T07:32:17.330563+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330563+0000","last_undegraded":"2024-03-06T07:32:17.330563+0000","last_fullsized":"2024-03-06T07:32:17.330563+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315934+0000","last_change":"2024-03-06T07:32:17.315934+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315934+0000","last_undegraded":"2024-03-06T07:32:17.315934+0000","last_fullsized":"2024-03-06T07:32:17.315934+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322220+0000","last_change":"2024-03-06T07:32:17.322220+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322220+0000","last_undegraded":"2024-03-06T07:32:17.322220+0000","last_fullsized":"2024-03-06T07:32:17.322220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327299+0000","last_change":"2024-03-06T07:32:17.327299+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327299+0000","last_undegraded":"2024-03-06T07:32:17.327299+0000","last_fullsized":"2024-03-06T07:32:17.327299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.336090+0000","last_change":"2024-03-06T07:32:17.336090+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.336090+0000","last_undegraded":"2024-03-06T07:32:17.336090+0000","last_fullsized":"2024-03-06T07:32:17.336090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331216+0000","last_change":"2024-03-06T07:32:17.331216+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331216+0000","last_undegraded":"2024-03-06T07:32:17.331216+0000","last_fullsized":"2024-03-06T07:32:17.331216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322253+0000","last_change":"2024-03-06T07:32:17.322253+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322253+0000","last_undegraded":"2024-03-06T07:32:17.322253+0000","last_fullsized":"2024-03-06T07:32:17.322253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323539+0000","last_change":"2024-03-06T07:32:17.323539+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323539+0000","last_undegraded":"2024-03-06T07:32:17.323539+0000","last_fullsized":"2024-03-06T07:32:17.323539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332679+0000","last_change":"2024-03-06T07:32:17.332679+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332679+0000","last_undegraded":"2024-03-06T07:32:17.332679+0000","last_fullsized":"2024-03-06T07:32:17.332679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331770+0000","last_change":"2024-03-06T07:32:17.331770+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331770+0000","last_undegraded":"2024-03-06T07:32:17.331770+0000","last_fullsized":"2024-03-06T07:32:17.331770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338853+0000","last_change":"2024-03-06T07:32:17.338853+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338853+0000","last_undegraded":"2024-03-06T07:32:17.338853+0000","last_fullsized":"2024-03-06T07:32:17.338853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348012+0000","last_change":"2024-03-06T07:32:17.340541+0000","last_active":"2024-03-06T07:32:17.348012+0000","last_peered":"2024-03-06T07:32:17.348012+0000","last_clean":"2024-03-06T07:32:17.348012+0000","last_became_active":"2024-03-06T07:32:17.338327+0000","last_became_peered":"2024-03-06T07:32:17.338327+0000","last_unstale":"2024-03-06T07:32:17.348012+0000","last_undegraded":"2024-03-06T07:32:17.348012+0000","last_fullsized":"2024-03-06T07:32:17.348012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330344+0000","last_change":"2024-03-06T07:32:17.330344+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330344+0000","last_undegraded":"2024-03-06T07:32:17.330344+0000","last_fullsized":"2024-03-06T07:32:17.330344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349286+0000","last_change":"2024-03-06T07:32:17.346963+0000","last_active":"2024-03-06T07:32:17.349286+0000","last_peered":"2024-03-06T07:32:17.349286+0000","last_clean":"2024-03-06T07:32:17.349286+0000","last_became_active":"2024-03-06T07:32:17.346591+0000","last_became_peered":"2024-03-06T07:32:17.346591+0000","last_unstale":"2024-03-06T07:32:17.349286+0000","last_undegraded":"2024-03-06T07:32:17.349286+0000","last_fullsized":"2024-03-06T07:32:17.349286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329514+0000","last_change":"2024-03-06T07:32:17.329514+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329514+0000","last_undegraded":"2024-03-06T07:32:17.329514+0000","last_fullsized":"2024-03-06T07:32:17.329514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337377+0000","last_change":"2024-03-06T07:32:17.334402+0000","last_active":"2024-03-06T07:32:17.337377+0000","last_peered":"2024-03-06T07:32:17.337377+0000","last_clean":"2024-03-06T07:32:17.337377+0000","last_became_active":"2024-03-06T07:32:17.334125+0000","last_became_peered":"2024-03-06T07:32:17.334125+0000","last_unstale":"2024-03-06T07:32:17.337377+0000","last_undegraded":"2024-03-06T07:32:17.337377+0000","last_fullsized":"2024-03-06T07:32:17.337377+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:19.628620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324921+0000","last_change":"2024-03-06T07:32:17.324921+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324921+0000","last_undegraded":"2024-03-06T07:32:17.324921+0000","last_fullsized":"2024-03-06T07:32:17.324921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337396+0000","last_change":"2024-03-06T07:32:17.333923+0000","last_active":"2024-03-06T07:32:17.337396+0000","last_peered":"2024-03-06T07:32:17.337396+0000","last_clean":"2024-03-06T07:32:17.337396+0000","last_became_active":"2024-03-06T07:32:17.333343+0000","last_became_peered":"2024-03-06T07:32:17.333343+0000","last_unstale":"2024-03-06T07:32:17.337396+0000","last_undegraded":"2024-03-06T07:32:17.337396+0000","last_fullsized":"2024-03-06T07:32:17.337396+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.148717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323137+0000","last_change":"2024-03-06T07:32:17.323137+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323137+0000","last_undegraded":"2024-03-06T07:32:17.323137+0000","last_fullsized":"2024-03-06T07:32:17.323137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345919+0000","last_change":"2024-03-06T07:32:17.343520+0000","last_active":"2024-03-06T07:32:17.345919+0000","last_peered":"2024-03-06T07:32:17.345919+0000","last_clean":"2024-03-06T07:32:17.345919+0000","last_became_active":"2024-03-06T07:32:17.343250+0000","last_became_peered":"2024-03-06T07:32:17.343250+0000","last_unstale":"2024-03-06T07:32:17.345919+0000","last_undegraded":"2024-03-06T07:32:17.345919+0000","last_fullsized":"2024-03-06T07:32:17.345919+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334006+0000","last_change":"2024-03-06T07:32:17.334006+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334006+0000","last_undegraded":"2024-03-06T07:32:17.334006+0000","last_fullsized":"2024-03-06T07:32:17.334006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349283+0000","last_change":"2024-03-06T07:32:17.346884+0000","last_active":"2024-03-06T07:32:17.349283+0000","last_peered":"2024-03-06T07:32:17.349283+0000","last_clean":"2024-03-06T07:32:17.349283+0000","last_became_active":"2024-03-06T07:32:17.346080+0000","last_became_peered":"2024-03-06T07:32:17.346080+0000","last_unstale":"2024-03-06T07:32:17.349283+0000","last_undegraded":"2024-03-06T07:32:17.349283+0000","last_fullsized":"2024-03-06T07:32:17.349283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327997+0000","last_change":"2024-03-06T07:32:17.327997+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327997+0000","last_undegraded":"2024-03-06T07:32:17.327997+0000","last_fullsized":"2024-03-06T07:32:17.327997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.339181+0000","last_change":"2024-03-06T07:32:17.336331+0000","last_active":"2024-03-06T07:32:17.339181+0000","last_peered":"2024-03-06T07:32:17.339181+0000","last_clean":"2024-03-06T07:32:17.339181+0000","last_became_active":"2024-03-06T07:32:17.336045+0000","last_became_peered":"2024-03-06T07:32:17.336045+0000","last_unstale":"2024-03-06T07:32:17.339181+0000","last_undegraded":"2024-03-06T07:32:17.339181+0000","last_fullsized":"2024-03-06T07:32:17.339181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:45.515629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329661+0000","last_change":"2024-03-06T07:32:17.329661+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329661+0000","last_undegraded":"2024-03-06T07:32:17.329661+0000","last_fullsized":"2024-03-06T07:32:17.329661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343941+0000","last_change":"2024-03-06T07:32:17.342249+0000","last_active":"2024-03-06T07:32:17.343941+0000","last_peered":"2024-03-06T07:32:17.343941+0000","last_clean":"2024-03-06T07:32:17.343941+0000","last_became_active":"2024-03-06T07:32:17.341962+0000","last_became_peered":"2024-03-06T07:32:17.341962+0000","last_unstale":"2024-03-06T07:32:17.343941+0000","last_undegraded":"2024-03-06T07:32:17.343941+0000","last_fullsized":"2024-03-06T07:32:17.343941+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324876+0000","last_change":"2024-03-06T07:32:17.324876+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324876+0000","last_undegraded":"2024-03-06T07:32:17.324876+0000","last_fullsized":"2024-03-06T07:32:17.324876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343382+0000","last_change":"2024-03-06T07:32:17.337307+0000","last_active":"2024-03-06T07:32:17.343382+0000","last_peered":"2024-03-06T07:32:17.343382+0000","last_clean":"2024-03-06T07:32:17.343382+0000","last_became_active":"2024-03-06T07:32:17.335584+0000","last_became_peered":"2024-03-06T07:32:17.335584+0000","last_unstale":"2024-03-06T07:32:17.343382+0000","last_undegraded":"2024-03-06T07:32:17.343382+0000","last_fullsized":"2024-03-06T07:32:17.343382+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:44.602459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320172+0000","last_change":"2024-03-06T07:32:17.320172+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320172+0000","last_undegraded":"2024-03-06T07:32:17.320172+0000","last_fullsized":"2024-03-06T07:32:17.320172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348146+0000","last_change":"2024-03-06T07:32:17.343053+0000","last_active":"2024-03-06T07:32:17.348146+0000","last_peered":"2024-03-06T07:32:17.348146+0000","last_clean":"2024-03-06T07:32:17.348146+0000","last_became_active":"2024-03-06T07:32:17.342760+0000","last_became_peered":"2024-03-06T07:32:17.342760+0000","last_unstale":"2024-03-06T07:32:17.348146+0000","last_undegraded":"2024-03-06T07:32:17.348146+0000","last_fullsized":"2024-03-06T07:32:17.348146+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323212+0000","last_change":"2024-03-06T07:32:17.323212+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323212+0000","last_undegraded":"2024-03-06T07:32:17.323212+0000","last_fullsized":"2024-03-06T07:32:17.323212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343344+0000","last_change":"2024-03-06T07:32:17.337091+0000","last_active":"2024-03-06T07:32:17.343344+0000","last_peered":"2024-03-06T07:32:17.343344+0000","last_clean":"2024-03-06T07:32:17.343344+0000","last_became_active":"2024-03-06T07:32:17.335040+0000","last_became_peered":"2024-03-06T07:32:17.335040+0000","last_unstale":"2024-03-06T07:32:17.343344+0000","last_undegraded":"2024-03-06T07:32:17.343344+0000","last_fullsized":"2024-03-06T07:32:17.343344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:26.368598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335590+0000","last_change":"2024-03-06T07:32:17.335590+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335590+0000","last_undegraded":"2024-03-06T07:32:17.335590+0000","last_fullsized":"2024-03-06T07:32:17.335590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340689+0000","last_change":"2024-03-06T07:32:17.336112+0000","last_active":"2024-03-06T07:32:17.340689+0000","last_peered":"2024-03-06T07:32:17.340689+0000","last_clean":"2024-03-06T07:32:17.340689+0000","last_became_active":"2024-03-06T07:32:17.335798+0000","last_became_peered":"2024-03-06T07:32:17.335798+0000","last_unstale":"2024-03-06T07:32:17.340689+0000","last_undegraded":"2024-03-06T07:32:17.340689+0000","last_fullsized":"2024-03-06T07:32:17.340689+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:03.187760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325045+0000","last_change":"2024-03-06T07:32:17.325045+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325045+0000","last_undegraded":"2024-03-06T07:32:17.325045+0000","last_fullsized":"2024-03-06T07:32:17.325045+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.331418+0000","last_change":"2024-03-06T07:32:17.328646+0000","last_active":"2024-03-06T07:32:17.331418+0000","last_peered":"2024-03-06T07:32:17.331418+0000","last_clean":"2024-03-06T07:32:17.331418+0000","last_became_active":"2024-03-06T07:32:17.328345+0000","last_became_peered":"2024-03-06T07:32:17.328345+0000","last_unstale":"2024-03-06T07:32:17.331418+0000","last_undegraded":"2024-03-06T07:32:17.331418+0000","last_fullsized":"2024-03-06T07:32:17.331418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.428256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325656+0000","last_change":"2024-03-06T07:32:17.325656+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325656+0000","last_undegraded":"2024-03-06T07:32:17.325656+0000","last_fullsized":"2024-03-06T07:32:17.325656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337375+0000","last_change":"2024-03-06T07:32:17.336488+0000","last_active":"2024-03-06T07:32:17.337375+0000","last_peered":"2024-03-06T07:32:17.337375+0000","last_clean":"2024-03-06T07:32:17.337375+0000","last_became_active":"2024-03-06T07:32:17.336208+0000","last_became_peered":"2024-03-06T07:32:17.336208+0000","last_unstale":"2024-03-06T07:32:17.337375+0000","last_undegraded":"2024-03-06T07:32:17.337375+0000","last_fullsized":"2024-03-06T07:32:17.337375+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.320149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322676+0000","last_change":"2024-03-06T07:32:17.322676+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322676+0000","last_undegraded":"2024-03-06T07:32:17.322676+0000","last_fullsized":"2024-03-06T07:32:17.322676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.347792+0000","last_change":"2024-03-06T07:32:17.344364+0000","last_active":"2024-03-06T07:32:17.347792+0000","last_peered":"2024-03-06T07:32:17.347792+0000","last_clean":"2024-03-06T07:32:17.347792+0000","last_became_active":"2024-03-06T07:32:17.342047+0000","last_became_peered":"2024-03-06T07:32:17.342047+0000","last_unstale":"2024-03-06T07:32:17.347792+0000","last_undegraded":"2024-03-06T07:32:17.347792+0000","last_fullsized":"2024-03-06T07:32:17.347792+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:03.374987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323397+0000","last_change":"2024-03-06T07:32:17.323397+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323397+0000","last_undegraded":"2024-03-06T07:32:17.323397+0000","last_fullsized":"2024-03-06T07:32:17.323397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345205+0000","last_change":"2024-03-06T07:32:17.343791+0000","last_active":"2024-03-06T07:32:17.345205+0000","last_peered":"2024-03-06T07:32:17.345205+0000","last_clean":"2024-03-06T07:32:17.345205+0000","last_became_active":"2024-03-06T07:32:17.343078+0000","last_became_peered":"2024-03-06T07:32:17.343078+0000","last_unstale":"2024-03-06T07:32:17.345205+0000","last_undegraded":"2024-03-06T07:32:17.345205+0000","last_fullsized":"2024-03-06T07:32:17.345205+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:34.532984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325862+0000","last_change":"2024-03-06T07:32:17.325862+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325862+0000","last_undegraded":"2024-03-06T07:32:17.325862+0000","last_fullsized":"2024-03-06T07:32:17.325862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348133+0000","last_change":"2024-03-06T07:32:17.344136+0000","last_active":"2024-03-06T07:32:17.348133+0000","last_peered":"2024-03-06T07:32:17.348133+0000","last_clean":"2024-03-06T07:32:17.348133+0000","last_became_active":"2024-03-06T07:32:17.343838+0000","last_became_peered":"2024-03-06T07:32:17.343838+0000","last_unstale":"2024-03-06T07:32:17.348133+0000","last_undegraded":"2024-03-06T07:32:17.348133+0000","last_fullsized":"2024-03-06T07:32:17.348133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329627+0000","last_change":"2024-03-06T07:32:17.329627+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329627+0000","last_undegraded":"2024-03-06T07:32:17.329627+0000","last_fullsized":"2024-03-06T07:32:17.329627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345189+0000","last_change":"2024-03-06T07:32:17.343396+0000","last_active":"2024-03-06T07:32:17.345189+0000","last_peered":"2024-03-06T07:32:17.345189+0000","last_clean":"2024-03-06T07:32:17.345189+0000","last_became_active":"2024-03-06T07:32:17.343086+0000","last_became_peered":"2024-03-06T07:32:17.343086+0000","last_unstale":"2024-03-06T07:32:17.345189+0000","last_undegraded":"2024-03-06T07:32:17.345189+0000","last_fullsized":"2024-03-06T07:32:17.345189+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:27.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332356+0000","last_change":"2024-03-06T07:32:17.332356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332356+0000","last_undegraded":"2024-03-06T07:32:17.332356+0000","last_fullsized":"2024-03-06T07:32:17.332356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344401+0000","last_change":"2024-03-06T07:32:17.341403+0000","last_active":"2024-03-06T07:32:17.344401+0000","last_peered":"2024-03-06T07:32:17.344401+0000","last_clean":"2024-03-06T07:32:17.344401+0000","last_became_active":"2024-03-06T07:32:17.340813+0000","last_became_peered":"2024-03-06T07:32:17.340813+0000","last_unstale":"2024-03-06T07:32:17.344401+0000","last_undegraded":"2024-03-06T07:32:17.344401+0000","last_fullsized":"2024-03-06T07:32:17.344401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:02.671020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315236+0000","last_change":"2024-03-06T07:32:17.315236+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315236+0000","last_undegraded":"2024-03-06T07:32:17.315236+0000","last_fullsized":"2024-03-06T07:32:17.315236+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340114+0000","last_change":"2024-03-06T07:32:17.338528+0000","last_active":"2024-03-06T07:32:17.340114+0000","last_peered":"2024-03-06T07:32:17.340114+0000","last_clean":"2024-03-06T07:32:17.340114+0000","last_became_active":"2024-03-06T07:32:17.337976+0000","last_became_peered":"2024-03-06T07:32:17.337976+0000","last_unstale":"2024-03-06T07:32:17.340114+0000","last_undegraded":"2024-03-06T07:32:17.340114+0000","last_fullsized":"2024-03-06T07:32:17.340114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:33.417849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320713+0000","last_change":"2024-03-06T07:32:17.320713+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320713+0000","last_undegraded":"2024-03-06T07:32:17.320713+0000","last_fullsized":"2024-03-06T07:32:17.320713+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.336878+0000","last_change":"2024-03-06T07:32:17.333102+0000","last_active":"2024-03-06T07:32:17.336878+0000","last_peered":"2024-03-06T07:32:17.336878+0000","last_clean":"2024-03-06T07:32:17.336878+0000","last_became_active":"2024-03-06T07:32:17.332802+0000","last_became_peered":"2024-03-06T07:32:17.332802+0000","last_unstale":"2024-03-06T07:32:17.336878+0000","last_undegraded":"2024-03-06T07:32:17.336878+0000","last_fullsized":"2024-03-06T07:32:17.336878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:10.165881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.314948+0000","last_change":"2024-03-06T07:32:17.314948+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.314948+0000","last_undegraded":"2024-03-06T07:32:17.314948+0000","last_fullsized":"2024-03-06T07:32:17.314948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342804+0000","last_change":"2024-03-06T07:32:17.340789+0000","last_active":"2024-03-06T07:32:17.342804+0000","last_peered":"2024-03-06T07:32:17.342804+0000","last_clean":"2024-03-06T07:32:17.342804+0000","last_became_active":"2024-03-06T07:32:17.340454+0000","last_became_peered":"2024-03-06T07:32:17.340454+0000","last_unstale":"2024-03-06T07:32:17.342804+0000","last_undegraded":"2024-03-06T07:32:17.342804+0000","last_fullsized":"2024-03-06T07:32:17.342804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:19.107804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318226+0000","last_change":"2024-03-06T07:32:17.318226+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318226+0000","last_undegraded":"2024-03-06T07:32:17.318226+0000","last_fullsized":"2024-03-06T07:32:17.318226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.335552+0000","last_change":"2024-03-06T07:32:17.333757+0000","last_active":"2024-03-06T07:32:17.335552+0000","last_peered":"2024-03-06T07:32:17.335552+0000","last_clean":"2024-03-06T07:32:17.335552+0000","last_became_active":"2024-03-06T07:32:17.333402+0000","last_became_peered":"2024-03-06T07:32:17.333402+0000","last_unstale":"2024-03-06T07:32:17.335552+0000","last_undegraded":"2024-03-06T07:32:17.335552+0000","last_fullsized":"2024-03-06T07:32:17.335552+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:18.884596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338012+0000","last_change":"2024-03-06T07:32:17.338012+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338012+0000","last_undegraded":"2024-03-06T07:32:17.338012+0000","last_fullsized":"2024-03-06T07:32:17.338012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344412+0000","last_change":"2024-03-06T07:32:17.342979+0000","last_active":"2024-03-06T07:32:17.344412+0000","last_peered":"2024-03-06T07:32:17.344412+0000","last_clean":"2024-03-06T07:32:17.344412+0000","last_became_active":"2024-03-06T07:32:17.342679+0000","last_became_peered":"2024-03-06T07:32:17.342679+0000","last_unstale":"2024-03-06T07:32:17.344412+0000","last_undegraded":"2024-03-06T07:32:17.344412+0000","last_fullsized":"2024-03-06T07:32:17.344412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:57.890290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328756+0000","last_change":"2024-03-06T07:32:17.328756+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328756+0000","last_undegraded":"2024-03-06T07:32:17.328756+0000","last_fullsized":"2024-03-06T07:32:17.328756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.332778+0000","last_change":"2024-03-06T07:32:17.331176+0000","last_active":"2024-03-06T07:32:17.332778+0000","last_peered":"2024-03-06T07:32:17.332778+0000","last_clean":"2024-03-06T07:32:17.332778+0000","last_became_active":"2024-03-06T07:32:17.328312+0000","last_became_peered":"2024-03-06T07:32:17.328312+0000","last_unstale":"2024-03-06T07:32:17.332778+0000","last_undegraded":"2024-03-06T07:32:17.332778+0000","last_fullsized":"2024-03-06T07:32:17.332778+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.809897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343166+0000","last_change":"2024-03-06T07:32:17.335023+0000","last_active":"2024-03-06T07:32:17.343166+0000","last_peered":"2024-03-06T07:32:17.343166+0000","last_clean":"2024-03-06T07:32:17.343166+0000","last_became_active":"2024-03-06T07:32:17.334651+0000","last_became_peered":"2024-03-06T07:32:17.334651+0000","last_unstale":"2024-03-06T07:32:17.343166+0000","last_undegraded":"2024-03-06T07:32:17.343166+0000","last_fullsized":"2024-03-06T07:32:17.343166+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:14.380230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331816+0000","last_change":"2024-03-06T07:32:17.331816+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331816+0000","last_undegraded":"2024-03-06T07:32:17.331816+0000","last_fullsized":"2024-03-06T07:32:17.331816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348114+0000","last_change":"2024-03-06T07:32:17.342916+0000","last_active":"2024-03-06T07:32:17.348114+0000","last_peered":"2024-03-06T07:32:17.348114+0000","last_clean":"2024-03-06T07:32:17.348114+0000","last_became_active":"2024-03-06T07:32:17.340684+0000","last_became_peered":"2024-03-06T07:32:17.340684+0000","last_unstale":"2024-03-06T07:32:17.348114+0000","last_undegraded":"2024-03-06T07:32:17.348114+0000","last_fullsized":"2024-03-06T07:32:17.348114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318836+0000","last_change":"2024-03-06T07:32:17.318836+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318836+0000","last_undegraded":"2024-03-06T07:32:17.318836+0000","last_fullsized":"2024-03-06T07:32:17.318836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344424+0000","last_change":"2024-03-06T07:32:17.341757+0000","last_active":"2024-03-06T07:32:17.344424+0000","last_peered":"2024-03-06T07:32:17.344424+0000","last_clean":"2024-03-06T07:32:17.344424+0000","last_became_active":"2024-03-06T07:32:17.341395+0000","last_became_peered":"2024-03-06T07:32:17.341395+0000","last_unstale":"2024-03-06T07:32:17.344424+0000","last_undegraded":"2024-03-06T07:32:17.344424+0000","last_fullsized":"2024-03-06T07:32:17.344424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:43.814597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324328+0000","last_change":"2024-03-06T07:32:17.324328+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324328+0000","last_undegraded":"2024-03-06T07:32:17.324328+0000","last_fullsized":"2024-03-06T07:32:17.324328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343769+0000","last_change":"2024-03-06T07:32:17.341723+0000","last_active":"2024-03-06T07:32:17.343769+0000","last_peered":"2024-03-06T07:32:17.343769+0000","last_clean":"2024-03-06T07:32:17.343769+0000","last_became_active":"2024-03-06T07:32:17.341402+0000","last_became_peered":"2024-03-06T07:32:17.341402+0000","last_unstale":"2024-03-06T07:32:17.343769+0000","last_undegraded":"2024-03-06T07:32:17.343769+0000","last_fullsized":"2024-03-06T07:32:17.343769+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324264+0000","last_change":"2024-03-06T07:32:17.324264+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324264+0000","last_undegraded":"2024-03-06T07:32:17.324264+0000","last_fullsized":"2024-03-06T07:32:17.324264+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345951+0000","last_change":"2024-03-06T07:32:17.338127+0000","last_active":"2024-03-06T07:32:17.345951+0000","last_peered":"2024-03-06T07:32:17.345951+0000","last_clean":"2024-03-06T07:32:17.345951+0000","last_became_active":"2024-03-06T07:32:17.337773+0000","last_became_peered":"2024-03-06T07:32:17.337773+0000","last_unstale":"2024-03-06T07:32:17.345951+0000","last_undegraded":"2024-03-06T07:32:17.345951+0000","last_fullsized":"2024-03-06T07:32:17.345951+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327839+0000","last_change":"2024-03-06T07:32:17.327839+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327839+0000","last_undegraded":"2024-03-06T07:32:17.327839+0000","last_fullsized":"2024-03-06T07:32:17.327839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349318+0000","last_change":"2024-03-06T07:32:17.347045+0000","last_active":"2024-03-06T07:32:17.349318+0000","last_peered":"2024-03-06T07:32:17.349318+0000","last_clean":"2024-03-06T07:32:17.349318+0000","last_became_active":"2024-03-06T07:32:17.346721+0000","last_became_peered":"2024-03-06T07:32:17.346721+0000","last_unstale":"2024-03-06T07:32:17.349318+0000","last_undegraded":"2024-03-06T07:32:17.349318+0000","last_fullsized":"2024-03-06T07:32:17.349318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322723+0000","last_change":"2024-03-06T07:32:17.322723+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322723+0000","last_undegraded":"2024-03-06T07:32:17.322723+0000","last_fullsized":"2024-03-06T07:32:17.322723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342074+0000","last_change":"2024-03-06T07:32:17.339426+0000","last_active":"2024-03-06T07:32:17.342074+0000","last_peered":"2024-03-06T07:32:17.342074+0000","last_clean":"2024-03-06T07:32:17.342074+0000","last_became_active":"2024-03-06T07:32:17.339135+0000","last_became_peered":"2024-03-06T07:32:17.339135+0000","last_unstale":"2024-03-06T07:32:17.342074+0000","last_undegraded":"2024-03-06T07:32:17.342074+0000","last_fullsized":"2024-03-06T07:32:17.342074+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:30.314302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.328500+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:32:17.328500+0000","last_peered":"2024-03-06T07:32:17.328500+0000","last_clean":"2024-03-06T07:32:17.328500+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:32:17.328500+0000","last_undegraded":"2024-03-06T07:32:17.328500+0000","last_fullsized":"2024-03-06T07:32:17.328500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574905,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3f is not active+clean 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.332986+0000', 'last_change': '2024-03-06T07:32:17.332986+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.332986+0000', 'last_undegraded': '2024-03-06T07:32:17.332986+0000', 'last_fullsized': '2024-03-06T07:32:17.332986+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3e is not active+clean 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.326645+0000', 'last_change': '2024-03-06T07:32:17.326645+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.326645+0000', 'last_undegraded': '2024-03-06T07:32:17.326645+0000', 'last_fullsized': '2024-03-06T07:32:17.326645+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3d is not active+clean 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.325780+0000', 'last_change': '2024-03-06T07:32:17.325780+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.325780+0000', 'last_undegraded': '2024-03-06T07:32:17.325780+0000', 'last_fullsized': '2024-03-06T07:32:17.325780+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3c is not active+clean 2024-03-06T07:32:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.317466+0000', 'last_change': '2024-03-06T07:32:17.317466+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.317466+0000', 'last_undegraded': '2024-03-06T07:32:17.317466+0000', 'last_fullsized': '2024-03-06T07:32:17.317466+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3b is not active+clean 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.334707+0000', 'last_change': '2024-03-06T07:32:17.334707+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.334707+0000', 'last_undegraded': '2024-03-06T07:32:17.334707+0000', 'last_fullsized': '2024-03-06T07:32:17.334707+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3a is not active+clean 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.322485+0000', 'last_change': '2024-03-06T07:32:17.322485+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.322485+0000', 'last_undegraded': '2024-03-06T07:32:17.322485+0000', 'last_fullsized': '2024-03-06T07:32:17.322485+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.39 is not active+clean 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.316077+0000', 'last_change': '2024-03-06T07:32:17.316077+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.316077+0000', 'last_undegraded': '2024-03-06T07:32:17.316077+0000', 'last_fullsized': '2024-03-06T07:32:17.316077+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.38 is not active+clean 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.317072+0000', 'last_change': '2024-03-06T07:32:17.317072+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.317072+0000', 'last_undegraded': '2024-03-06T07:32:17.317072+0000', 'last_fullsized': '2024-03-06T07:32:17.317072+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.37 is not active+clean 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.326283+0000', 'last_change': '2024-03-06T07:32:17.326283+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.326283+0000', 'last_undegraded': '2024-03-06T07:32:17.326283+0000', 'last_fullsized': '2024-03-06T07:32:17.326283+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.36 is not active+clean 2024-03-06T07:32:23.788 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.331240+0000', 'last_change': '2024-03-06T07:32:17.331240+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.331240+0000', 'last_undegraded': '2024-03-06T07:32:17.331240+0000', 'last_fullsized': '2024-03-06T07:32:17.331240+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.35 is not active+clean 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.324388+0000', 'last_change': '2024-03-06T07:32:17.324388+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.324388+0000', 'last_undegraded': '2024-03-06T07:32:17.324388+0000', 'last_fullsized': '2024-03-06T07:32:17.324388+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.34 is not active+clean 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.332062+0000', 'last_change': '2024-03-06T07:32:17.332062+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.332062+0000', 'last_undegraded': '2024-03-06T07:32:17.332062+0000', 'last_fullsized': '2024-03-06T07:32:17.332062+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.33 is not active+clean 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.327857+0000', 'last_change': '2024-03-06T07:32:17.327857+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.327857+0000', 'last_undegraded': '2024-03-06T07:32:17.327857+0000', 'last_fullsized': '2024-03-06T07:32:17.327857+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.32 is not active+clean 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.315906+0000', 'last_change': '2024-03-06T07:32:17.315906+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.315906+0000', 'last_undegraded': '2024-03-06T07:32:17.315906+0000', 'last_fullsized': '2024-03-06T07:32:17.315906+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.31 is not active+clean 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.330155+0000', 'last_change': '2024-03-06T07:32:17.330155+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.330155+0000', 'last_undegraded': '2024-03-06T07:32:17.330155+0000', 'last_fullsized': '2024-03-06T07:32:17.330155+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.30 is not active+clean 2024-03-06T07:32:23.789 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.328356+0000', 'last_change': '2024-03-06T07:32:17.328356+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.328356+0000', 'last_undegraded': '2024-03-06T07:32:17.328356+0000', 'last_fullsized': '2024-03-06T07:32:17.328356+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2f is not active+clean 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.335599+0000', 'last_change': '2024-03-06T07:32:17.335599+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.335599+0000', 'last_undegraded': '2024-03-06T07:32:17.335599+0000', 'last_fullsized': '2024-03-06T07:32:17.335599+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2e is not active+clean 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.316017+0000', 'last_change': '2024-03-06T07:32:17.316017+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.316017+0000', 'last_undegraded': '2024-03-06T07:32:17.316017+0000', 'last_fullsized': '2024-03-06T07:32:17.316017+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2d is not active+clean 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.322552+0000', 'last_change': '2024-03-06T07:32:17.322552+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.322552+0000', 'last_undegraded': '2024-03-06T07:32:17.322552+0000', 'last_fullsized': '2024-03-06T07:32:17.322552+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2c is not active+clean 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.315799+0000', 'last_change': '2024-03-06T07:32:17.315799+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.315799+0000', 'last_undegraded': '2024-03-06T07:32:17.315799+0000', 'last_fullsized': '2024-03-06T07:32:17.315799+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2b is not active+clean 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.316054+0000', 'last_change': '2024-03-06T07:32:17.316054+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.316054+0000', 'last_undegraded': '2024-03-06T07:32:17.316054+0000', 'last_fullsized': '2024-03-06T07:32:17.316054+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2a is not active+clean 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.323894+0000', 'last_change': '2024-03-06T07:32:17.323894+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.323894+0000', 'last_undegraded': '2024-03-06T07:32:17.323894+0000', 'last_fullsized': '2024-03-06T07:32:17.323894+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.29 is not active+clean 2024-03-06T07:32:23.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.330563+0000', 'last_change': '2024-03-06T07:32:17.330563+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.330563+0000', 'last_undegraded': '2024-03-06T07:32:17.330563+0000', 'last_fullsized': '2024-03-06T07:32:17.330563+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.28 is not active+clean 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.315934+0000', 'last_change': '2024-03-06T07:32:17.315934+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.315934+0000', 'last_undegraded': '2024-03-06T07:32:17.315934+0000', 'last_fullsized': '2024-03-06T07:32:17.315934+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.27 is not active+clean 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.322220+0000', 'last_change': '2024-03-06T07:32:17.322220+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.322220+0000', 'last_undegraded': '2024-03-06T07:32:17.322220+0000', 'last_fullsized': '2024-03-06T07:32:17.322220+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.26 is not active+clean 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.327299+0000', 'last_change': '2024-03-06T07:32:17.327299+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.327299+0000', 'last_undegraded': '2024-03-06T07:32:17.327299+0000', 'last_fullsized': '2024-03-06T07:32:17.327299+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.25 is not active+clean 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.336090+0000', 'last_change': '2024-03-06T07:32:17.336090+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.336090+0000', 'last_undegraded': '2024-03-06T07:32:17.336090+0000', 'last_fullsized': '2024-03-06T07:32:17.336090+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.24 is not active+clean 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.331216+0000', 'last_change': '2024-03-06T07:32:17.331216+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.331216+0000', 'last_undegraded': '2024-03-06T07:32:17.331216+0000', 'last_fullsized': '2024-03-06T07:32:17.331216+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.23 is not active+clean 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.322253+0000', 'last_change': '2024-03-06T07:32:17.322253+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.322253+0000', 'last_undegraded': '2024-03-06T07:32:17.322253+0000', 'last_fullsized': '2024-03-06T07:32:17.322253+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.22 is not active+clean 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.323539+0000', 'last_change': '2024-03-06T07:32:17.323539+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.323539+0000', 'last_undegraded': '2024-03-06T07:32:17.323539+0000', 'last_fullsized': '2024-03-06T07:32:17.323539+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.21 is not active+clean 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.332679+0000', 'last_change': '2024-03-06T07:32:17.332679+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.332679+0000', 'last_undegraded': '2024-03-06T07:32:17.332679+0000', 'last_fullsized': '2024-03-06T07:32:17.332679+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.20 is not active+clean 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.331770+0000', 'last_change': '2024-03-06T07:32:17.331770+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.331770+0000', 'last_undegraded': '2024-03-06T07:32:17.331770+0000', 'last_fullsized': '2024-03-06T07:32:17.331770+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.338853+0000', 'last_change': '2024-03-06T07:32:17.338853+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.338853+0000', 'last_undegraded': '2024-03-06T07:32:17.338853+0000', 'last_fullsized': '2024-03-06T07:32:17.338853+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.330344+0000', 'last_change': '2024-03-06T07:32:17.330344+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.330344+0000', 'last_undegraded': '2024-03-06T07:32:17.330344+0000', 'last_fullsized': '2024-03-06T07:32:17.330344+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.329514+0000', 'last_change': '2024-03-06T07:32:17.329514+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.329514+0000', 'last_undegraded': '2024-03-06T07:32:17.329514+0000', 'last_fullsized': '2024-03-06T07:32:17.329514+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1d is not active+clean 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.324921+0000', 'last_change': '2024-03-06T07:32:17.324921+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.324921+0000', 'last_undegraded': '2024-03-06T07:32:17.324921+0000', 'last_fullsized': '2024-03-06T07:32:17.324921+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1a is not active+clean 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.323137+0000', 'last_change': '2024-03-06T07:32:17.323137+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.323137+0000', 'last_undegraded': '2024-03-06T07:32:17.323137+0000', 'last_fullsized': '2024-03-06T07:32:17.323137+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.334006+0000', 'last_change': '2024-03-06T07:32:17.334006+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.334006+0000', 'last_undegraded': '2024-03-06T07:32:17.334006+0000', 'last_fullsized': '2024-03-06T07:32:17.334006+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.18 is not active+clean 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.327997+0000', 'last_change': '2024-03-06T07:32:17.327997+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.327997+0000', 'last_undegraded': '2024-03-06T07:32:17.327997+0000', 'last_fullsized': '2024-03-06T07:32:17.327997+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.19 is not active+clean 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.329661+0000', 'last_change': '2024-03-06T07:32:17.329661+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.329661+0000', 'last_undegraded': '2024-03-06T07:32:17.329661+0000', 'last_fullsized': '2024-03-06T07:32:17.329661+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.16 is not active+clean 2024-03-06T07:32:23.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.324876+0000', 'last_change': '2024-03-06T07:32:17.324876+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.324876+0000', 'last_undegraded': '2024-03-06T07:32:17.324876+0000', 'last_fullsized': '2024-03-06T07:32:17.324876+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.320172+0000', 'last_change': '2024-03-06T07:32:17.320172+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.320172+0000', 'last_undegraded': '2024-03-06T07:32:17.320172+0000', 'last_fullsized': '2024-03-06T07:32:17.320172+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.14 is not active+clean 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.323212+0000', 'last_change': '2024-03-06T07:32:17.323212+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.323212+0000', 'last_undegraded': '2024-03-06T07:32:17.323212+0000', 'last_fullsized': '2024-03-06T07:32:17.323212+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.335590+0000', 'last_change': '2024-03-06T07:32:17.335590+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.335590+0000', 'last_undegraded': '2024-03-06T07:32:17.335590+0000', 'last_fullsized': '2024-03-06T07:32:17.335590+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.12 is not active+clean 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.325045+0000', 'last_change': '2024-03-06T07:32:17.325045+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.325045+0000', 'last_undegraded': '2024-03-06T07:32:17.325045+0000', 'last_fullsized': '2024-03-06T07:32:17.325045+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.13 is not active+clean 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.325656+0000', 'last_change': '2024-03-06T07:32:17.325656+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.325656+0000', 'last_undegraded': '2024-03-06T07:32:17.325656+0000', 'last_fullsized': '2024-03-06T07:32:17.325656+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.10 is not active+clean 2024-03-06T07:32:23.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.322676+0000', 'last_change': '2024-03-06T07:32:17.322676+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.322676+0000', 'last_undegraded': '2024-03-06T07:32:17.322676+0000', 'last_fullsized': '2024-03-06T07:32:17.322676+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.11 is not active+clean 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.323397+0000', 'last_change': '2024-03-06T07:32:17.323397+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.323397+0000', 'last_undegraded': '2024-03-06T07:32:17.323397+0000', 'last_fullsized': '2024-03-06T07:32:17.323397+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.e is not active+clean 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.325862+0000', 'last_change': '2024-03-06T07:32:17.325862+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.325862+0000', 'last_undegraded': '2024-03-06T07:32:17.325862+0000', 'last_fullsized': '2024-03-06T07:32:17.325862+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.329627+0000', 'last_change': '2024-03-06T07:32:17.329627+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.329627+0000', 'last_undegraded': '2024-03-06T07:32:17.329627+0000', 'last_fullsized': '2024-03-06T07:32:17.329627+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.332356+0000', 'last_change': '2024-03-06T07:32:17.332356+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.332356+0000', 'last_undegraded': '2024-03-06T07:32:17.332356+0000', 'last_fullsized': '2024-03-06T07:32:17.332356+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.a is not active+clean 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.315236+0000', 'last_change': '2024-03-06T07:32:17.315236+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.315236+0000', 'last_undegraded': '2024-03-06T07:32:17.315236+0000', 'last_fullsized': '2024-03-06T07:32:17.315236+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.320713+0000', 'last_change': '2024-03-06T07:32:17.320713+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.320713+0000', 'last_undegraded': '2024-03-06T07:32:17.320713+0000', 'last_fullsized': '2024-03-06T07:32:17.320713+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.8 is not active+clean 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.314948+0000', 'last_change': '2024-03-06T07:32:17.314948+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.314948+0000', 'last_undegraded': '2024-03-06T07:32:17.314948+0000', 'last_fullsized': '2024-03-06T07:32:17.314948+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.318226+0000', 'last_change': '2024-03-06T07:32:17.318226+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.318226+0000', 'last_undegraded': '2024-03-06T07:32:17.318226+0000', 'last_fullsized': '2024-03-06T07:32:17.318226+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.338012+0000', 'last_change': '2024-03-06T07:32:17.338012+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.338012+0000', 'last_undegraded': '2024-03-06T07:32:17.338012+0000', 'last_fullsized': '2024-03-06T07:32:17.338012+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.7 is not active+clean 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.328756+0000', 'last_change': '2024-03-06T07:32:17.328756+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.328756+0000', 'last_undegraded': '2024-03-06T07:32:17.328756+0000', 'last_fullsized': '2024-03-06T07:32:17.328756+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.325780+0000', 'last_change': '2024-03-06T07:32:17.325780+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.325780+0000', 'last_undegraded': '2024-03-06T07:32:17.325780+0000', 'last_fullsized': '2024-03-06T07:32:17.325780+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.331816+0000', 'last_change': '2024-03-06T07:32:17.331816+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.331816+0000', 'last_undegraded': '2024-03-06T07:32:17.331816+0000', 'last_fullsized': '2024-03-06T07:32:17.331816+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.318836+0000', 'last_change': '2024-03-06T07:32:17.318836+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.318836+0000', 'last_undegraded': '2024-03-06T07:32:17.318836+0000', 'last_fullsized': '2024-03-06T07:32:17.318836+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3 is not active+clean 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.324328+0000', 'last_change': '2024-03-06T07:32:17.324328+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.324328+0000', 'last_undegraded': '2024-03-06T07:32:17.324328+0000', 'last_fullsized': '2024-03-06T07:32:17.324328+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.0 is not active+clean 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.324264+0000', 'last_change': '2024-03-06T07:32:17.324264+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.324264+0000', 'last_undegraded': '2024-03-06T07:32:17.324264+0000', 'last_fullsized': '2024-03-06T07:32:17.324264+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.327839+0000', 'last_change': '2024-03-06T07:32:17.327839+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.327839+0000', 'last_undegraded': '2024-03-06T07:32:17.327839+0000', 'last_fullsized': '2024-03-06T07:32:17.327839+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.c is not active+clean 2024-03-06T07:32:23.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:32:17.322723+0000', 'last_change': '2024-03-06T07:32:17.322723+0000', 'last_active': '2024-03-06T07:32:17.302402+0000', 'last_peered': '2024-03-06T07:32:17.302402+0000', 'last_clean': '2024-03-06T07:32:17.302402+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:32:17.322723+0000', 'last_undegraded': '2024-03-06T07:32:17.322723+0000', 'last_fullsized': '2024-03-06T07:32:17.322723+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'last_clean_scrub_stamp': '2024-03-06T07:32:17.302402+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:32:23.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:32:24.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.018+0000 7feaaf600640 1 Processor -- start 2024-03-06T07:32:24.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.018+0000 7feaaf600640 1 -- start start 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaaf600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feaa8145850 0x7feaa806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaaf600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa80cfd60 0x7feaa80d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaaf600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa806e090 con 0x7feaa806afc0 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaaf600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa806e370 con 0x7feaa8145850 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaaf600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa806e650 con 0x7feaa80cfd60 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaacc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feaa8145850 0x7feaa806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaacc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feaa8145850 0x7feaa806f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35144/0 (socket says 172.21.15.114:35144) 2024-03-06T07:32:24.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.020+0000 7feaacc00640 1 -- 172.21.15.114:0/1469998926 learned_addr learned my addr 172.21.15.114:0/1469998926 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:24.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.021+0000 7fea9fe00640 1 --2- 172.21.15.114:0/1469998926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa80cfd60 0x7feaa80d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.021+0000 7feaacc00640 1 -- 172.21.15.114:0/1469998926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa80cfd60 msgr2=0x7feaa80d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.021+0000 7feaacc00640 1 --2- 172.21.15.114:0/1469998926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa80cfd60 0x7feaa80d01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.021+0000 7feaacc00640 1 -- 172.21.15.114:0/1469998926 >> v1:172.21.15.177:6789/0 conn(0x7feaa806afc0 legacy=0x7feaa806b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.021+0000 7feaacc00640 1 -- 172.21.15.114:0/1469998926 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa806e930 con 0x7feaa8145850 2024-03-06T07:32:24.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.022+0000 7feaacc00640 1 --2- 172.21.15.114:0/1469998926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa8145850 0x7feaa806f130 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fea98009870 tx=0x7fea9802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b09d83a3a87f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.022+0000 7feaa7e00640 1 -- 172.21.15.114:0/1469998926 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98035020 con 0x7feaa8145850 2024-03-06T07:32:24.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaa7e00640 1 -- 172.21.15.114:0/1469998926 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea98031640 con 0x7feaa8145850 2024-03-06T07:32:24.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaa7e00640 1 -- 172.21.15.114:0/1469998926 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9802d040 con 0x7feaa8145850 2024-03-06T07:32:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 -- 172.21.15.114:0/1469998926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa8145850 msgr2=0x7feaa806f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 --2- 172.21.15.114:0/1469998926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa8145850 0x7feaa806f130 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fea98009870 tx=0x7fea9802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 -- 172.21.15.114:0/1469998926 shutdown_connections 2024-03-06T07:32:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 --2- 172.21.15.114:0/1469998926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa80cfd60 0x7feaa80d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 --2- 172.21.15.114:0/1469998926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa8145850 0x7feaa806f130 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 -- 172.21.15.114:0/1469998926 >> 172.21.15.114:0/1469998926 conn(0x7feaa80b2840 msgr2=0x7feaa80b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 -- 172.21.15.114:0/1469998926 shutdown_connections 2024-03-06T07:32:24.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.023+0000 7feaaf600640 1 -- 172.21.15.114:0/1469998926 wait complete. 2024-03-06T07:32:24.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.025+0000 7feaaf600640 1 Processor -- start 2024-03-06T07:32:24.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.025+0000 7feaaf600640 1 -- start start 2024-03-06T07:32:24.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7feaaf600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa806afc0 0x7feaa80b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7feaaf600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa80cfd60 0x7feaa80b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7feaaf600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feaa8145850 0x7feaa8130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7feaaf600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa815c200 con 0x7feaa80cfd60 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7feaaf600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa806dd40 con 0x7feaa8145850 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7feaaf600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa815c380 con 0x7feaa806afc0 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7fea9fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa80cfd60 0x7feaa80b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7fea9fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa80cfd60 0x7feaa80b3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35150/0 (socket says 172.21.15.114:35150) 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.026+0000 7fea9fe00640 1 -- 172.21.15.114:0/3377886292 learned_addr learned my addr 172.21.15.114:0/3377886292 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7feaacc00640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa806afc0 0x7feaa80b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7feaad600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feaa8145850 0x7feaa8130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7fea9fe00640 1 -- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa806afc0 msgr2=0x7feaa80b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7fea9fe00640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa806afc0 0x7feaa80b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7fea9fe00640 1 -- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feaa8145850 msgr2=0x7feaa8130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7fea9fe00640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feaa8145850 0x7feaa8130370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7fea9fe00640 1 -- 172.21.15.114:0/3377886292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa80b3c60 con 0x7feaa80cfd60 2024-03-06T07:32:24.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.027+0000 7fea9fe00640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa80cfd60 0x7feaa80b3720 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fea9000d830 tx=0x7fea9000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.028+0000 7feaa5600640 1 -- 172.21.15.114:0/3377886292 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9000b6c0 con 0x7feaa80cfd60 2024-03-06T07:32:24.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.028+0000 7feaa5600640 1 -- 172.21.15.114:0/3377886292 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea90019070 con 0x7feaa80cfd60 2024-03-06T07:32:24.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.028+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa80b3f30 con 0x7feaa80cfd60 2024-03-06T07:32:24.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.028+0000 7feaa5600640 1 -- 172.21.15.114:0/3377886292 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9000ba80 con 0x7feaa80cfd60 2024-03-06T07:32:24.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.028+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa80b44f0 con 0x7feaa80cfd60 2024-03-06T07:32:24.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.029+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa80704c0 con 0x7feaa80cfd60 2024-03-06T07:32:24.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.030+0000 7feaa5600640 1 -- 172.21.15.114:0/3377886292 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea9001e460 con 0x7feaa80cfd60 2024-03-06T07:32:24.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.031+0000 7feaa5600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea8406b4d0 0x7fea8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.031+0000 7feaa5600640 1 -- 172.21.15.114:0/3377886292 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea9008a9e0 con 0x7feaa80cfd60 2024-03-06T07:32:24.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.031+0000 7feaacc00640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea8406b4d0 0x7fea8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.032+0000 7feaacc00640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea8406b4d0 0x7fea8406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fea98036000 tx=0x7fea9803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.036+0000 7feaa5600640 1 -- 172.21.15.114:0/3377886292 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feaa80704c0 con 0x7feaa80cfd60 2024-03-06T07:32:24.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.297+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7feaa804f330 con 0x7feaa80cfd60 2024-03-06T07:32:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.298+0000 7feaa5600640 1 -- 172.21.15.114:0/3377886292 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7fea90059730 con 0x7feaa80cfd60 2024-03-06T07:32:24.300 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:24.300 INFO:teuthology.orchestra.run.smithi114.stdout:{"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":294,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":80,"num_osds":8,"num_up_osds":8,"osd_up_since":1709710062,"num_in_osds":8,"osd_in_since":1709710053,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":264290304,"bytes_avail":772829822976,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":52,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":6756}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-06T07:32:03.175146+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea8406b4d0 msgr2=0x7fea8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea8406b4d0 0x7fea8406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fea98036000 tx=0x7fea9803c040 comp rx=0 tx=0).stop 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa80cfd60 msgr2=0x7feaa80b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa80cfd60 0x7feaa80b3720 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fea9000d830 tx=0x7fea9000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 shutdown_connections 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feaa8145850 0x7feaa8130370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea8406b4d0 0x7fea8406d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaa80cfd60 0x7feaa80b3720 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 --2- 172.21.15.114:0/3377886292 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feaa806afc0 0x7feaa80b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 >> 172.21.15.114:0/3377886292 conn(0x7feaa80b2840 msgr2=0x7feaa8141760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 shutdown_connections 2024-03-06T07:32:24.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.300+0000 7feaaf600640 1 -- 172.21.15.114:0/3377886292 wait complete. 2024-03-06T07:32:24.320 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:32:24.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:24.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.506+0000 7fab4cc00640 1 Processor -- start 2024-03-06T07:32:24.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.507+0000 7fab4cc00640 1 -- start start 2024-03-06T07:32:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.508+0000 7fab4cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fab4815fe60 0x7fab48071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.508+0000 7fab4cc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4806cef0 con 0x7fab4815f9d0 2024-03-06T07:32:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.508+0000 7fab4cc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4806d1d0 con 0x7fab48144c00 2024-03-06T07:32:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.508+0000 7fab4cc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4806d4b0 con 0x7fab4815fe60 2024-03-06T07:32:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.509+0000 7fab3d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fab4815f9d0 0x7fab48070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46294/0 (socket says 172.21.15.114:46294) 2024-03-06T07:32:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.509+0000 7fab3d400640 1 -- 172.21.15.114:0/842225211 learned_addr learned my addr 172.21.15.114:0/842225211 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.509+0000 7fab46800640 1 --2- 172.21.15.114:0/842225211 >> v2:172.21.15.114:3300/0 conn(0x7fab4815fe60 0x7fab48071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.509+0000 7fab46800640 1 -- 172.21.15.114:0/842225211 >> v1:172.21.15.177:6789/0 conn(0x7fab48144c00 legacy=0x7fab4806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.509+0000 7fab46800640 1 -- 172.21.15.114:0/842225211 >> v1:172.21.15.114:6789/0 conn(0x7fab4815f9d0 legacy=0x7fab48070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.509+0000 7fab46800640 1 -- 172.21.15.114:0/842225211 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4806d790 con 0x7fab4815fe60 2024-03-06T07:32:24.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.510+0000 7fab46800640 1 --2- 172.21.15.114:0/842225211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab4815fe60 0x7fab48071220 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fab400097a0 tx=0x7fab4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a283be0c645deef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.510+0000 7fab45400640 1 -- 172.21.15.114:0/842225211 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40035020 con 0x7fab4815fe60 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.510+0000 7fab45400640 1 -- 172.21.15.114:0/842225211 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab40031830 con 0x7fab4815fe60 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.510+0000 7fab45400640 1 -- 172.21.15.114:0/842225211 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40031c00 con 0x7fab4815fe60 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.511+0000 7fab4cc00640 1 -- 172.21.15.114:0/842225211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab4815fe60 msgr2=0x7fab48071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.511+0000 7fab4cc00640 1 --2- 172.21.15.114:0/842225211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab4815fe60 0x7fab48071220 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fab400097a0 tx=0x7fab4002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.511+0000 7fab4cc00640 1 -- 172.21.15.114:0/842225211 shutdown_connections 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.511+0000 7fab4cc00640 1 --2- 172.21.15.114:0/842225211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab4815fe60 0x7fab48071220 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.511+0000 7fab4cc00640 1 -- 172.21.15.114:0/842225211 >> 172.21.15.114:0/842225211 conn(0x7fab480b1730 msgr2=0x7fab480b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.511+0000 7fab4cc00640 1 -- 172.21.15.114:0/842225211 shutdown_connections 2024-03-06T07:32:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.511+0000 7fab4cc00640 1 -- 172.21.15.114:0/842225211 wait complete. 2024-03-06T07:32:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.513+0000 7fab4cc00640 1 Processor -- start 2024-03-06T07:32:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.513+0000 7fab4cc00640 1 -- start start 2024-03-06T07:32:24.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab4cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab48144c00 0x7fab4815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab4cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab4815f9d0 0x7fab480b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab4cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab4815fe60 0x7fab480b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab4cc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4806cb50 con 0x7fab48144c00 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab4cc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48160730 con 0x7fab4815f9d0 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab4cc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4806ccd0 con 0x7fab4815fe60 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab48144c00 0x7fab4815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab48144c00 0x7fab4815abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35164/0 (socket says 172.21.15.114:35164) 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 -- 172.21.15.114:0/3402376491 learned_addr learned my addr 172.21.15.114:0/3402376491 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab3d400640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab4815f9d0 0x7fab480b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 -- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab4815fe60 msgr2=0x7fab480b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab4815fe60 0x7fab480b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 -- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab4815f9d0 msgr2=0x7fab480b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab4815f9d0 0x7fab480b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab45e00640 1 -- 172.21.15.114:0/3402376491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab481e80e0 con 0x7fab48144c00 2024-03-06T07:32:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.514+0000 7fab3d400640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab4815f9d0 0x7fab480b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.515+0000 7fab45e00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab48144c00 0x7fab4815abb0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fab3800ba50 tx=0x7fab3800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.515+0000 7fab3ea00640 1 -- 172.21.15.114:0/3402376491 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38019070 con 0x7fab48144c00 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.515+0000 7fab3ea00640 1 -- 172.21.15.114:0/3402376491 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3800cc20 con 0x7fab48144c00 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.515+0000 7fab3ea00640 1 -- 172.21.15.114:0/3402376491 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38014680 con 0x7fab48144c00 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.515+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab481e83c0 con 0x7fab48144c00 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.515+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab481e8980 con 0x7fab48144c00 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.516+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4804f330 con 0x7fab48144c00 2024-03-06T07:32:24.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.517+0000 7fab3ea00640 1 -- 172.21.15.114:0/3402376491 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab3801c070 con 0x7fab48144c00 2024-03-06T07:32:24.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.518+0000 7fab3ea00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab2406b4d0 0x7fab2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.518+0000 7fab3ea00640 1 -- 172.21.15.114:0/3402376491 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab38089e90 con 0x7fab48144c00 2024-03-06T07:32:24.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.518+0000 7fab3d400640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab2406b4d0 0x7fab2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.519+0000 7fab3d400640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab2406b4d0 0x7fab2406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fab480af8b0 tx=0x7fab30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.524+0000 7fab3ea00640 1 -- 172.21.15.114:0/3402376491 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab380586b0 con 0x7fab48144c00 2024-03-06T07:32:24.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.715+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab48072890 con 0x7fab2406b4d0 2024-03-06T07:32:24.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.727+0000 7fab3ea00640 1 -- 172.21.15.114:0/3402376491 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246753 (secure 0 0 0) 0x7fab48072890 con 0x7fab2406b4d0 2024-03-06T07:32:24.732 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:24.733 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.732+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab2406b4d0 msgr2=0x7fab2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.732+0000 7fab4cc00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab2406b4d0 0x7fab2406d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fab480af8b0 tx=0x7fab30009040 comp rx=0 tx=0).stop 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.732+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab48144c00 msgr2=0x7fab4815abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.732+0000 7fab4cc00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab48144c00 0x7fab4815abb0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fab3800ba50 tx=0x7fab3800bf10 comp rx=0 tx=0).stop 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 shutdown_connections 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab4815fe60 0x7fab480b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab4815f9d0 0x7fab480b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab2406b4d0 0x7fab2406d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 --2- 172.21.15.114:0/3402376491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab48144c00 0x7fab4815abb0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 >> 172.21.15.114:0/3402376491 conn(0x7fab480b1730 msgr2=0x7fab480b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 shutdown_connections 2024-03-06T07:32:24.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:24.733+0000 7fab4cc00640 1 -- 172.21.15.114:0/3402376491 wait complete. 2024-03-06T07:32:24.753 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-03-06T07:32:23.180584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258096,"kb_used_data":8648,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716624,"statfs":{"total":773094113280,"available":772829822976,"internally_reserved":0,"allocated":8855552,"data_stored":4412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001539"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332986+0000","last_change":"2024-03-06T07:32:17.332986+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332986+0000","last_undegraded":"2024-03-06T07:32:17.332986+0000","last_fullsized":"2024-03-06T07:32:17.332986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326645+0000","last_change":"2024-03-06T07:32:17.326645+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326645+0000","last_undegraded":"2024-03-06T07:32:17.326645+0000","last_fullsized":"2024-03-06T07:32:17.326645+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317466+0000","last_change":"2024-03-06T07:32:17.317466+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317466+0000","last_undegraded":"2024-03-06T07:32:17.317466+0000","last_fullsized":"2024-03-06T07:32:17.317466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334707+0000","last_change":"2024-03-06T07:32:17.334707+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334707+0000","last_undegraded":"2024-03-06T07:32:17.334707+0000","last_fullsized":"2024-03-06T07:32:17.334707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322485+0000","last_change":"2024-03-06T07:32:17.322485+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322485+0000","last_undegraded":"2024-03-06T07:32:17.322485+0000","last_fullsized":"2024-03-06T07:32:17.322485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316077+0000","last_change":"2024-03-06T07:32:17.316077+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316077+0000","last_undegraded":"2024-03-06T07:32:17.316077+0000","last_fullsized":"2024-03-06T07:32:17.316077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.317072+0000","last_change":"2024-03-06T07:32:17.317072+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.317072+0000","last_undegraded":"2024-03-06T07:32:17.317072+0000","last_fullsized":"2024-03-06T07:32:17.317072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.326283+0000","last_change":"2024-03-06T07:32:17.326283+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.326283+0000","last_undegraded":"2024-03-06T07:32:17.326283+0000","last_fullsized":"2024-03-06T07:32:17.326283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331240+0000","last_change":"2024-03-06T07:32:17.331240+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331240+0000","last_undegraded":"2024-03-06T07:32:17.331240+0000","last_fullsized":"2024-03-06T07:32:17.331240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324388+0000","last_change":"2024-03-06T07:32:17.324388+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324388+0000","last_undegraded":"2024-03-06T07:32:17.324388+0000","last_fullsized":"2024-03-06T07:32:17.324388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332062+0000","last_change":"2024-03-06T07:32:17.332062+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332062+0000","last_undegraded":"2024-03-06T07:32:17.332062+0000","last_fullsized":"2024-03-06T07:32:17.332062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327857+0000","last_change":"2024-03-06T07:32:17.327857+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327857+0000","last_undegraded":"2024-03-06T07:32:17.327857+0000","last_fullsized":"2024-03-06T07:32:17.327857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315906+0000","last_change":"2024-03-06T07:32:17.315906+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315906+0000","last_undegraded":"2024-03-06T07:32:17.315906+0000","last_fullsized":"2024-03-06T07:32:17.315906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330155+0000","last_change":"2024-03-06T07:32:17.330155+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330155+0000","last_undegraded":"2024-03-06T07:32:17.330155+0000","last_fullsized":"2024-03-06T07:32:17.330155+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328356+0000","last_change":"2024-03-06T07:32:17.328356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328356+0000","last_undegraded":"2024-03-06T07:32:17.328356+0000","last_fullsized":"2024-03-06T07:32:17.328356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335599+0000","last_change":"2024-03-06T07:32:17.335599+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335599+0000","last_undegraded":"2024-03-06T07:32:17.335599+0000","last_fullsized":"2024-03-06T07:32:17.335599+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316017+0000","last_change":"2024-03-06T07:32:17.316017+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316017+0000","last_undegraded":"2024-03-06T07:32:17.316017+0000","last_fullsized":"2024-03-06T07:32:17.316017+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322552+0000","last_change":"2024-03-06T07:32:17.322552+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322552+0000","last_undegraded":"2024-03-06T07:32:17.322552+0000","last_fullsized":"2024-03-06T07:32:17.322552+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315799+0000","last_change":"2024-03-06T07:32:17.315799+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315799+0000","last_undegraded":"2024-03-06T07:32:17.315799+0000","last_fullsized":"2024-03-06T07:32:17.315799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.316054+0000","last_change":"2024-03-06T07:32:17.316054+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.316054+0000","last_undegraded":"2024-03-06T07:32:17.316054+0000","last_fullsized":"2024-03-06T07:32:17.316054+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323894+0000","last_change":"2024-03-06T07:32:17.323894+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323894+0000","last_undegraded":"2024-03-06T07:32:17.323894+0000","last_fullsized":"2024-03-06T07:32:17.323894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330563+0000","last_change":"2024-03-06T07:32:17.330563+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330563+0000","last_undegraded":"2024-03-06T07:32:17.330563+0000","last_fullsized":"2024-03-06T07:32:17.330563+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315934+0000","last_change":"2024-03-06T07:32:17.315934+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315934+0000","last_undegraded":"2024-03-06T07:32:17.315934+0000","last_fullsized":"2024-03-06T07:32:17.315934+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322220+0000","last_change":"2024-03-06T07:32:17.322220+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322220+0000","last_undegraded":"2024-03-06T07:32:17.322220+0000","last_fullsized":"2024-03-06T07:32:17.322220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327299+0000","last_change":"2024-03-06T07:32:17.327299+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327299+0000","last_undegraded":"2024-03-06T07:32:17.327299+0000","last_fullsized":"2024-03-06T07:32:17.327299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.336090+0000","last_change":"2024-03-06T07:32:17.336090+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.336090+0000","last_undegraded":"2024-03-06T07:32:17.336090+0000","last_fullsized":"2024-03-06T07:32:17.336090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331216+0000","last_change":"2024-03-06T07:32:17.331216+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331216+0000","last_undegraded":"2024-03-06T07:32:17.331216+0000","last_fullsized":"2024-03-06T07:32:17.331216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322253+0000","last_change":"2024-03-06T07:32:17.322253+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322253+0000","last_undegraded":"2024-03-06T07:32:17.322253+0000","last_fullsized":"2024-03-06T07:32:17.322253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323539+0000","last_change":"2024-03-06T07:32:17.323539+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323539+0000","last_undegraded":"2024-03-06T07:32:17.323539+0000","last_fullsized":"2024-03-06T07:32:17.323539+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332679+0000","last_change":"2024-03-06T07:32:17.332679+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332679+0000","last_undegraded":"2024-03-06T07:32:17.332679+0000","last_fullsized":"2024-03-06T07:32:17.332679+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331770+0000","last_change":"2024-03-06T07:32:17.331770+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331770+0000","last_undegraded":"2024-03-06T07:32:17.331770+0000","last_fullsized":"2024-03-06T07:32:17.331770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338853+0000","last_change":"2024-03-06T07:32:17.338853+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338853+0000","last_undegraded":"2024-03-06T07:32:17.338853+0000","last_fullsized":"2024-03-06T07:32:17.338853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348012+0000","last_change":"2024-03-06T07:32:17.340541+0000","last_active":"2024-03-06T07:32:17.348012+0000","last_peered":"2024-03-06T07:32:17.348012+0000","last_clean":"2024-03-06T07:32:17.348012+0000","last_became_active":"2024-03-06T07:32:17.338327+0000","last_became_peered":"2024-03-06T07:32:17.338327+0000","last_unstale":"2024-03-06T07:32:17.348012+0000","last_undegraded":"2024-03-06T07:32:17.348012+0000","last_fullsized":"2024-03-06T07:32:17.348012+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.330344+0000","last_change":"2024-03-06T07:32:17.330344+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.330344+0000","last_undegraded":"2024-03-06T07:32:17.330344+0000","last_fullsized":"2024-03-06T07:32:17.330344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349286+0000","last_change":"2024-03-06T07:32:17.346963+0000","last_active":"2024-03-06T07:32:17.349286+0000","last_peered":"2024-03-06T07:32:17.349286+0000","last_clean":"2024-03-06T07:32:17.349286+0000","last_became_active":"2024-03-06T07:32:17.346591+0000","last_became_peered":"2024-03-06T07:32:17.346591+0000","last_unstale":"2024-03-06T07:32:17.349286+0000","last_undegraded":"2024-03-06T07:32:17.349286+0000","last_fullsized":"2024-03-06T07:32:17.349286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329514+0000","last_change":"2024-03-06T07:32:17.329514+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329514+0000","last_undegraded":"2024-03-06T07:32:17.329514+0000","last_fullsized":"2024-03-06T07:32:17.329514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337377+0000","last_change":"2024-03-06T07:32:17.334402+0000","last_active":"2024-03-06T07:32:17.337377+0000","last_peered":"2024-03-06T07:32:17.337377+0000","last_clean":"2024-03-06T07:32:17.337377+0000","last_became_active":"2024-03-06T07:32:17.334125+0000","last_became_peered":"2024-03-06T07:32:17.334125+0000","last_unstale":"2024-03-06T07:32:17.337377+0000","last_undegraded":"2024-03-06T07:32:17.337377+0000","last_fullsized":"2024-03-06T07:32:17.337377+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:19.628620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324921+0000","last_change":"2024-03-06T07:32:17.324921+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324921+0000","last_undegraded":"2024-03-06T07:32:17.324921+0000","last_fullsized":"2024-03-06T07:32:17.324921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337396+0000","last_change":"2024-03-06T07:32:17.333923+0000","last_active":"2024-03-06T07:32:17.337396+0000","last_peered":"2024-03-06T07:32:17.337396+0000","last_clean":"2024-03-06T07:32:17.337396+0000","last_became_active":"2024-03-06T07:32:17.333343+0000","last_became_peered":"2024-03-06T07:32:17.333343+0000","last_unstale":"2024-03-06T07:32:17.337396+0000","last_undegraded":"2024-03-06T07:32:17.337396+0000","last_fullsized":"2024-03-06T07:32:17.337396+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.148717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323137+0000","last_change":"2024-03-06T07:32:17.323137+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323137+0000","last_undegraded":"2024-03-06T07:32:17.323137+0000","last_fullsized":"2024-03-06T07:32:17.323137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345919+0000","last_change":"2024-03-06T07:32:17.343520+0000","last_active":"2024-03-06T07:32:17.345919+0000","last_peered":"2024-03-06T07:32:17.345919+0000","last_clean":"2024-03-06T07:32:17.345919+0000","last_became_active":"2024-03-06T07:32:17.343250+0000","last_became_peered":"2024-03-06T07:32:17.343250+0000","last_unstale":"2024-03-06T07:32:17.345919+0000","last_undegraded":"2024-03-06T07:32:17.345919+0000","last_fullsized":"2024-03-06T07:32:17.345919+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:23.854444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.334006+0000","last_change":"2024-03-06T07:32:17.334006+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.334006+0000","last_undegraded":"2024-03-06T07:32:17.334006+0000","last_fullsized":"2024-03-06T07:32:17.334006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349283+0000","last_change":"2024-03-06T07:32:17.346884+0000","last_active":"2024-03-06T07:32:17.349283+0000","last_peered":"2024-03-06T07:32:17.349283+0000","last_clean":"2024-03-06T07:32:17.349283+0000","last_became_active":"2024-03-06T07:32:17.346080+0000","last_became_peered":"2024-03-06T07:32:17.346080+0000","last_unstale":"2024-03-06T07:32:17.349283+0000","last_undegraded":"2024-03-06T07:32:17.349283+0000","last_fullsized":"2024-03-06T07:32:17.349283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327997+0000","last_change":"2024-03-06T07:32:17.327997+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327997+0000","last_undegraded":"2024-03-06T07:32:17.327997+0000","last_fullsized":"2024-03-06T07:32:17.327997+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.339181+0000","last_change":"2024-03-06T07:32:17.336331+0000","last_active":"2024-03-06T07:32:17.339181+0000","last_peered":"2024-03-06T07:32:17.339181+0000","last_clean":"2024-03-06T07:32:17.339181+0000","last_became_active":"2024-03-06T07:32:17.336045+0000","last_became_peered":"2024-03-06T07:32:17.336045+0000","last_unstale":"2024-03-06T07:32:17.339181+0000","last_undegraded":"2024-03-06T07:32:17.339181+0000","last_fullsized":"2024-03-06T07:32:17.339181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:45.515629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329661+0000","last_change":"2024-03-06T07:32:17.329661+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329661+0000","last_undegraded":"2024-03-06T07:32:17.329661+0000","last_fullsized":"2024-03-06T07:32:17.329661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343941+0000","last_change":"2024-03-06T07:32:17.342249+0000","last_active":"2024-03-06T07:32:17.343941+0000","last_peered":"2024-03-06T07:32:17.343941+0000","last_clean":"2024-03-06T07:32:17.343941+0000","last_became_active":"2024-03-06T07:32:17.341962+0000","last_became_peered":"2024-03-06T07:32:17.341962+0000","last_unstale":"2024-03-06T07:32:17.343941+0000","last_undegraded":"2024-03-06T07:32:17.343941+0000","last_fullsized":"2024-03-06T07:32:17.343941+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324876+0000","last_change":"2024-03-06T07:32:17.324876+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324876+0000","last_undegraded":"2024-03-06T07:32:17.324876+0000","last_fullsized":"2024-03-06T07:32:17.324876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343382+0000","last_change":"2024-03-06T07:32:17.337307+0000","last_active":"2024-03-06T07:32:17.343382+0000","last_peered":"2024-03-06T07:32:17.343382+0000","last_clean":"2024-03-06T07:32:17.343382+0000","last_became_active":"2024-03-06T07:32:17.335584+0000","last_became_peered":"2024-03-06T07:32:17.335584+0000","last_unstale":"2024-03-06T07:32:17.343382+0000","last_undegraded":"2024-03-06T07:32:17.343382+0000","last_fullsized":"2024-03-06T07:32:17.343382+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:44.602459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320172+0000","last_change":"2024-03-06T07:32:17.320172+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320172+0000","last_undegraded":"2024-03-06T07:32:17.320172+0000","last_fullsized":"2024-03-06T07:32:17.320172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348146+0000","last_change":"2024-03-06T07:32:17.343053+0000","last_active":"2024-03-06T07:32:17.348146+0000","last_peered":"2024-03-06T07:32:17.348146+0000","last_clean":"2024-03-06T07:32:17.348146+0000","last_became_active":"2024-03-06T07:32:17.342760+0000","last_became_peered":"2024-03-06T07:32:17.342760+0000","last_unstale":"2024-03-06T07:32:17.348146+0000","last_undegraded":"2024-03-06T07:32:17.348146+0000","last_fullsized":"2024-03-06T07:32:17.348146+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:01.488492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323212+0000","last_change":"2024-03-06T07:32:17.323212+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323212+0000","last_undegraded":"2024-03-06T07:32:17.323212+0000","last_fullsized":"2024-03-06T07:32:17.323212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343344+0000","last_change":"2024-03-06T07:32:17.337091+0000","last_active":"2024-03-06T07:32:17.343344+0000","last_peered":"2024-03-06T07:32:17.343344+0000","last_clean":"2024-03-06T07:32:17.343344+0000","last_became_active":"2024-03-06T07:32:17.335040+0000","last_became_peered":"2024-03-06T07:32:17.335040+0000","last_unstale":"2024-03-06T07:32:17.343344+0000","last_undegraded":"2024-03-06T07:32:17.343344+0000","last_fullsized":"2024-03-06T07:32:17.343344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:26.368598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.335590+0000","last_change":"2024-03-06T07:32:17.335590+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.335590+0000","last_undegraded":"2024-03-06T07:32:17.335590+0000","last_fullsized":"2024-03-06T07:32:17.335590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340689+0000","last_change":"2024-03-06T07:32:17.336112+0000","last_active":"2024-03-06T07:32:17.340689+0000","last_peered":"2024-03-06T07:32:17.340689+0000","last_clean":"2024-03-06T07:32:17.340689+0000","last_became_active":"2024-03-06T07:32:17.335798+0000","last_became_peered":"2024-03-06T07:32:17.335798+0000","last_unstale":"2024-03-06T07:32:17.340689+0000","last_undegraded":"2024-03-06T07:32:17.340689+0000","last_fullsized":"2024-03-06T07:32:17.340689+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:03.187760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325045+0000","last_change":"2024-03-06T07:32:17.325045+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325045+0000","last_undegraded":"2024-03-06T07:32:17.325045+0000","last_fullsized":"2024-03-06T07:32:17.325045+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.331418+0000","last_change":"2024-03-06T07:32:17.328646+0000","last_active":"2024-03-06T07:32:17.331418+0000","last_peered":"2024-03-06T07:32:17.331418+0000","last_clean":"2024-03-06T07:32:17.331418+0000","last_became_active":"2024-03-06T07:32:17.328345+0000","last_became_peered":"2024-03-06T07:32:17.328345+0000","last_unstale":"2024-03-06T07:32:17.331418+0000","last_undegraded":"2024-03-06T07:32:17.331418+0000","last_fullsized":"2024-03-06T07:32:17.331418+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:58.428256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325656+0000","last_change":"2024-03-06T07:32:17.325656+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325656+0000","last_undegraded":"2024-03-06T07:32:17.325656+0000","last_fullsized":"2024-03-06T07:32:17.325656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.337375+0000","last_change":"2024-03-06T07:32:17.336488+0000","last_active":"2024-03-06T07:32:17.337375+0000","last_peered":"2024-03-06T07:32:17.337375+0000","last_clean":"2024-03-06T07:32:17.337375+0000","last_became_active":"2024-03-06T07:32:17.336208+0000","last_became_peered":"2024-03-06T07:32:17.336208+0000","last_unstale":"2024-03-06T07:32:17.337375+0000","last_undegraded":"2024-03-06T07:32:17.337375+0000","last_fullsized":"2024-03-06T07:32:17.337375+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:00.320149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322676+0000","last_change":"2024-03-06T07:32:17.322676+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322676+0000","last_undegraded":"2024-03-06T07:32:17.322676+0000","last_fullsized":"2024-03-06T07:32:17.322676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.347792+0000","last_change":"2024-03-06T07:32:17.344364+0000","last_active":"2024-03-06T07:32:17.347792+0000","last_peered":"2024-03-06T07:32:17.347792+0000","last_clean":"2024-03-06T07:32:17.347792+0000","last_became_active":"2024-03-06T07:32:17.342047+0000","last_became_peered":"2024-03-06T07:32:17.342047+0000","last_unstale":"2024-03-06T07:32:17.347792+0000","last_undegraded":"2024-03-06T07:32:17.347792+0000","last_fullsized":"2024-03-06T07:32:17.347792+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:03.374987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.323397+0000","last_change":"2024-03-06T07:32:17.323397+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.323397+0000","last_undegraded":"2024-03-06T07:32:17.323397+0000","last_fullsized":"2024-03-06T07:32:17.323397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345205+0000","last_change":"2024-03-06T07:32:17.343791+0000","last_active":"2024-03-06T07:32:17.345205+0000","last_peered":"2024-03-06T07:32:17.345205+0000","last_clean":"2024-03-06T07:32:17.345205+0000","last_became_active":"2024-03-06T07:32:17.343078+0000","last_became_peered":"2024-03-06T07:32:17.343078+0000","last_unstale":"2024-03-06T07:32:17.345205+0000","last_undegraded":"2024-03-06T07:32:17.345205+0000","last_fullsized":"2024-03-06T07:32:17.345205+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:34.532984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325862+0000","last_change":"2024-03-06T07:32:17.325862+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325862+0000","last_undegraded":"2024-03-06T07:32:17.325862+0000","last_fullsized":"2024-03-06T07:32:17.325862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348133+0000","last_change":"2024-03-06T07:32:17.344136+0000","last_active":"2024-03-06T07:32:17.348133+0000","last_peered":"2024-03-06T07:32:17.348133+0000","last_clean":"2024-03-06T07:32:17.348133+0000","last_became_active":"2024-03-06T07:32:17.343838+0000","last_became_peered":"2024-03-06T07:32:17.343838+0000","last_unstale":"2024-03-06T07:32:17.348133+0000","last_undegraded":"2024-03-06T07:32:17.348133+0000","last_fullsized":"2024-03-06T07:32:17.348133+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.329627+0000","last_change":"2024-03-06T07:32:17.329627+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.329627+0000","last_undegraded":"2024-03-06T07:32:17.329627+0000","last_fullsized":"2024-03-06T07:32:17.329627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345189+0000","last_change":"2024-03-06T07:32:17.343396+0000","last_active":"2024-03-06T07:32:17.345189+0000","last_peered":"2024-03-06T07:32:17.345189+0000","last_clean":"2024-03-06T07:32:17.345189+0000","last_became_active":"2024-03-06T07:32:17.343086+0000","last_became_peered":"2024-03-06T07:32:17.343086+0000","last_unstale":"2024-03-06T07:32:17.345189+0000","last_undegraded":"2024-03-06T07:32:17.345189+0000","last_fullsized":"2024-03-06T07:32:17.345189+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:27.118345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.332356+0000","last_change":"2024-03-06T07:32:17.332356+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.332356+0000","last_undegraded":"2024-03-06T07:32:17.332356+0000","last_fullsized":"2024-03-06T07:32:17.332356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344401+0000","last_change":"2024-03-06T07:32:17.341403+0000","last_active":"2024-03-06T07:32:17.344401+0000","last_peered":"2024-03-06T07:32:17.344401+0000","last_clean":"2024-03-06T07:32:17.344401+0000","last_became_active":"2024-03-06T07:32:17.340813+0000","last_became_peered":"2024-03-06T07:32:17.340813+0000","last_unstale":"2024-03-06T07:32:17.344401+0000","last_undegraded":"2024-03-06T07:32:17.344401+0000","last_fullsized":"2024-03-06T07:32:17.344401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:02.671020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.315236+0000","last_change":"2024-03-06T07:32:17.315236+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.315236+0000","last_undegraded":"2024-03-06T07:32:17.315236+0000","last_fullsized":"2024-03-06T07:32:17.315236+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.340114+0000","last_change":"2024-03-06T07:32:17.338528+0000","last_active":"2024-03-06T07:32:17.340114+0000","last_peered":"2024-03-06T07:32:17.340114+0000","last_clean":"2024-03-06T07:32:17.340114+0000","last_became_active":"2024-03-06T07:32:17.337976+0000","last_became_peered":"2024-03-06T07:32:17.337976+0000","last_unstale":"2024-03-06T07:32:17.340114+0000","last_undegraded":"2024-03-06T07:32:17.340114+0000","last_fullsized":"2024-03-06T07:32:17.340114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:33.417849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.320713+0000","last_change":"2024-03-06T07:32:17.320713+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.320713+0000","last_undegraded":"2024-03-06T07:32:17.320713+0000","last_fullsized":"2024-03-06T07:32:17.320713+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.336878+0000","last_change":"2024-03-06T07:32:17.333102+0000","last_active":"2024-03-06T07:32:17.336878+0000","last_peered":"2024-03-06T07:32:17.336878+0000","last_clean":"2024-03-06T07:32:17.336878+0000","last_became_active":"2024-03-06T07:32:17.332802+0000","last_became_peered":"2024-03-06T07:32:17.332802+0000","last_unstale":"2024-03-06T07:32:17.336878+0000","last_undegraded":"2024-03-06T07:32:17.336878+0000","last_fullsized":"2024-03-06T07:32:17.336878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:10.165881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.314948+0000","last_change":"2024-03-06T07:32:17.314948+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.314948+0000","last_undegraded":"2024-03-06T07:32:17.314948+0000","last_fullsized":"2024-03-06T07:32:17.314948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342804+0000","last_change":"2024-03-06T07:32:17.340789+0000","last_active":"2024-03-06T07:32:17.342804+0000","last_peered":"2024-03-06T07:32:17.342804+0000","last_clean":"2024-03-06T07:32:17.342804+0000","last_became_active":"2024-03-06T07:32:17.340454+0000","last_became_peered":"2024-03-06T07:32:17.340454+0000","last_unstale":"2024-03-06T07:32:17.342804+0000","last_undegraded":"2024-03-06T07:32:17.342804+0000","last_fullsized":"2024-03-06T07:32:17.342804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:19.107804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318226+0000","last_change":"2024-03-06T07:32:17.318226+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318226+0000","last_undegraded":"2024-03-06T07:32:17.318226+0000","last_fullsized":"2024-03-06T07:32:17.318226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.335552+0000","last_change":"2024-03-06T07:32:17.333757+0000","last_active":"2024-03-06T07:32:17.335552+0000","last_peered":"2024-03-06T07:32:17.335552+0000","last_clean":"2024-03-06T07:32:17.335552+0000","last_became_active":"2024-03-06T07:32:17.333402+0000","last_became_peered":"2024-03-06T07:32:17.333402+0000","last_unstale":"2024-03-06T07:32:17.335552+0000","last_undegraded":"2024-03-06T07:32:17.335552+0000","last_fullsized":"2024-03-06T07:32:17.335552+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:18.884596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.338012+0000","last_change":"2024-03-06T07:32:17.338012+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.338012+0000","last_undegraded":"2024-03-06T07:32:17.338012+0000","last_fullsized":"2024-03-06T07:32:17.338012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344412+0000","last_change":"2024-03-06T07:32:17.342979+0000","last_active":"2024-03-06T07:32:17.344412+0000","last_peered":"2024-03-06T07:32:17.344412+0000","last_clean":"2024-03-06T07:32:17.344412+0000","last_became_active":"2024-03-06T07:32:17.342679+0000","last_became_peered":"2024-03-06T07:32:17.342679+0000","last_unstale":"2024-03-06T07:32:17.344412+0000","last_undegraded":"2024-03-06T07:32:17.344412+0000","last_fullsized":"2024-03-06T07:32:17.344412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:57.890290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.328756+0000","last_change":"2024-03-06T07:32:17.328756+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.328756+0000","last_undegraded":"2024-03-06T07:32:17.328756+0000","last_fullsized":"2024-03-06T07:32:17.328756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.332778+0000","last_change":"2024-03-06T07:32:17.331176+0000","last_active":"2024-03-06T07:32:17.332778+0000","last_peered":"2024-03-06T07:32:17.332778+0000","last_clean":"2024-03-06T07:32:17.332778+0000","last_became_active":"2024-03-06T07:32:17.328312+0000","last_became_peered":"2024-03-06T07:32:17.328312+0000","last_unstale":"2024-03-06T07:32:17.332778+0000","last_undegraded":"2024-03-06T07:32:17.332778+0000","last_fullsized":"2024-03-06T07:32:17.332778+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.809897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.325780+0000","last_change":"2024-03-06T07:32:17.325780+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.325780+0000","last_undegraded":"2024-03-06T07:32:17.325780+0000","last_fullsized":"2024-03-06T07:32:17.325780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343166+0000","last_change":"2024-03-06T07:32:17.335023+0000","last_active":"2024-03-06T07:32:17.343166+0000","last_peered":"2024-03-06T07:32:17.343166+0000","last_clean":"2024-03-06T07:32:17.343166+0000","last_became_active":"2024-03-06T07:32:17.334651+0000","last_became_peered":"2024-03-06T07:32:17.334651+0000","last_unstale":"2024-03-06T07:32:17.343166+0000","last_undegraded":"2024-03-06T07:32:17.343166+0000","last_fullsized":"2024-03-06T07:32:17.343166+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:14.380230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.331816+0000","last_change":"2024-03-06T07:32:17.331816+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.331816+0000","last_undegraded":"2024-03-06T07:32:17.331816+0000","last_fullsized":"2024-03-06T07:32:17.331816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.348114+0000","last_change":"2024-03-06T07:32:17.342916+0000","last_active":"2024-03-06T07:32:17.348114+0000","last_peered":"2024-03-06T07:32:17.348114+0000","last_clean":"2024-03-06T07:32:17.348114+0000","last_became_active":"2024-03-06T07:32:17.340684+0000","last_became_peered":"2024-03-06T07:32:17.340684+0000","last_unstale":"2024-03-06T07:32:17.348114+0000","last_undegraded":"2024-03-06T07:32:17.348114+0000","last_fullsized":"2024-03-06T07:32:17.348114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:39.323379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.318836+0000","last_change":"2024-03-06T07:32:17.318836+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.318836+0000","last_undegraded":"2024-03-06T07:32:17.318836+0000","last_fullsized":"2024-03-06T07:32:17.318836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.344424+0000","last_change":"2024-03-06T07:32:17.341757+0000","last_active":"2024-03-06T07:32:17.344424+0000","last_peered":"2024-03-06T07:32:17.344424+0000","last_clean":"2024-03-06T07:32:17.344424+0000","last_became_active":"2024-03-06T07:32:17.341395+0000","last_became_peered":"2024-03-06T07:32:17.341395+0000","last_unstale":"2024-03-06T07:32:17.344424+0000","last_undegraded":"2024-03-06T07:32:17.344424+0000","last_fullsized":"2024-03-06T07:32:17.344424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:43.814597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324328+0000","last_change":"2024-03-06T07:32:17.324328+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324328+0000","last_undegraded":"2024-03-06T07:32:17.324328+0000","last_fullsized":"2024-03-06T07:32:17.324328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.343769+0000","last_change":"2024-03-06T07:32:17.341723+0000","last_active":"2024-03-06T07:32:17.343769+0000","last_peered":"2024-03-06T07:32:17.343769+0000","last_clean":"2024-03-06T07:32:17.343769+0000","last_became_active":"2024-03-06T07:32:17.341402+0000","last_became_peered":"2024-03-06T07:32:17.341402+0000","last_unstale":"2024-03-06T07:32:17.343769+0000","last_undegraded":"2024-03-06T07:32:17.343769+0000","last_fullsized":"2024-03-06T07:32:17.343769+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.324264+0000","last_change":"2024-03-06T07:32:17.324264+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.324264+0000","last_undegraded":"2024-03-06T07:32:17.324264+0000","last_fullsized":"2024-03-06T07:32:17.324264+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.345951+0000","last_change":"2024-03-06T07:32:17.338127+0000","last_active":"2024-03-06T07:32:17.345951+0000","last_peered":"2024-03-06T07:32:17.345951+0000","last_clean":"2024-03-06T07:32:17.345951+0000","last_became_active":"2024-03-06T07:32:17.337773+0000","last_became_peered":"2024-03-06T07:32:17.337773+0000","last_unstale":"2024-03-06T07:32:17.345951+0000","last_undegraded":"2024-03-06T07:32:17.345951+0000","last_fullsized":"2024-03-06T07:32:17.345951+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:30.919076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.327839+0000","last_change":"2024-03-06T07:32:17.327839+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.327839+0000","last_undegraded":"2024-03-06T07:32:17.327839+0000","last_fullsized":"2024-03-06T07:32:17.327839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.349318+0000","last_change":"2024-03-06T07:32:17.347045+0000","last_active":"2024-03-06T07:32:17.349318+0000","last_peered":"2024-03-06T07:32:17.349318+0000","last_clean":"2024-03-06T07:32:17.349318+0000","last_became_active":"2024-03-06T07:32:17.346721+0000","last_became_peered":"2024-03-06T07:32:17.346721+0000","last_unstale":"2024-03-06T07:32:17.349318+0000","last_undegraded":"2024-03-06T07:32:17.349318+0000","last_fullsized":"2024-03-06T07:32:17.349318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:17.643575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:32:17.322723+0000","last_change":"2024-03-06T07:32:17.322723+0000","last_active":"2024-03-06T07:32:17.302402+0000","last_peered":"2024-03-06T07:32:17.302402+0000","last_clean":"2024-03-06T07:32:17.302402+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:17.322723+0000","last_undegraded":"2024-03-06T07:32:17.322723+0000","last_fullsized":"2024-03-06T07:32:17.322723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.342074+0000","last_change":"2024-03-06T07:32:17.339426+0000","last_active":"2024-03-06T07:32:17.342074+0000","last_peered":"2024-03-06T07:32:17.342074+0000","last_clean":"2024-03-06T07:32:17.342074+0000","last_became_active":"2024-03-06T07:32:17.339135+0000","last_became_peered":"2024-03-06T07:32:17.339135+0000","last_unstale":"2024-03-06T07:32:17.342074+0000","last_undegraded":"2024-03-06T07:32:17.342074+0000","last_fullsized":"2024-03-06T07:32:17.342074+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:30.314302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:32:17.328500+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:32:17.328500+0000","last_peered":"2024-03-06T07:32:17.328500+0000","last_clean":"2024-03-06T07:32:17.328500+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:32:17.328500+0000","last_undegraded":"2024-03-06T07:32:17.328500+0000","last_fullsized":"2024-03-06T07:32:17.328500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574905,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":403960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32700,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339140,"statfs":{"total":96636764160,"available":96603279360,"internally_reserved":0,"allocated":1552384,"data_stored":994328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:27.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:27.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.970+0000 7fc1edc00640 1 Processor -- start 2024-03-06T07:32:27.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.971+0000 7fc1edc00640 1 -- start start 2024-03-06T07:32:27.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.972+0000 7fc1edc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 0x7fc1e80b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.972+0000 7fc1edc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e8161960 con 0x7fc1e80d2360 2024-03-06T07:32:27.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.972+0000 7fc1edc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e80cf6f0 con 0x7fc1e8161580 2024-03-06T07:32:27.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.972+0000 7fc1edc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80cf9d0 con 0x7fc1e80d27a0 2024-03-06T07:32:27.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.972+0000 7fc1e7400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc1e80d2360 0x7fc1e80b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35740/0 (socket says 172.21.15.114:35740) 2024-03-06T07:32:27.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.972+0000 7fc1e7400640 1 -- 172.21.15.114:0/1838615490 learned_addr learned my addr 172.21.15.114:0/1838615490 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:27.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.972+0000 7fc1ed200640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 0x7fc1e80b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146874249 0 0) 0x7fc1e8161960 con 0x7fc1e80d2360 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1cc003520 con 0x7fc1e80d2360 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860714999 0 0) 0x7fc1cc003520 con 0x7fc1e80d2360 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 msgr2=0x7fc1e80b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 0x7fc1e80b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 >> v1:172.21.15.177:6789/0 conn(0x7fc1e8161580 legacy=0x7fc1e80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e80cfcb0 con 0x7fc1e80d2360 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fc1d4001670 con 0x7fc1e80d2360 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 >> v1:172.21.15.114:6789/0 conn(0x7fc1e80d2360 legacy=0x7fc1e80b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1cc0039e0 0x7fc1cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 0x7fc1cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc004d40 0x7fc1cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80cf9d0 con 0x7fc1cc0039e0 2024-03-06T07:32:27.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc00ba70 con 0x7fc1cc004d40 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.973+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc00bc20 con 0x7fc1e80d27a0 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1e7e00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1cc0039e0 0x7fc1cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1e7400640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 0x7fc1cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1ed200640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc004d40 0x7fc1cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1e7e00640 1 -- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 msgr2=0x7fc1cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1e7e00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e80d27a0 0x7fc1cc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1e7e00640 1 -- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc004d40 msgr2=0x7fc1cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1e7e00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc004d40 0x7fc1cc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.974+0000 7fc1e7e00640 1 -- 172.21.15.114:0/1838615490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc003520 con 0x7fc1cc0039e0 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.975+0000 7fc1e7e00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1cc0039e0 0x7fc1cc003dc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8001860 tx=0x7fc1cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42598d8a0151d28c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.975+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d802f050 con 0x7fc1cc0039e0 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d8033650 con 0x7fc1cc0039e0 2024-03-06T07:32:27.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1e6a00640 1 -- 172.21.15.114:0/1838615490 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8001430 con 0x7fc1cc0039e0 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1edc00640 1 -- 172.21.15.114:0/1838615490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1cc0039e0 msgr2=0x7fc1cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1edc00640 1 --2- 172.21.15.114:0/1838615490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1cc0039e0 0x7fc1cc003dc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8001860 tx=0x7fc1cc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1ed200640 1 -- 172.21.15.114:0/1838615490 reap_dead start 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1edc00640 1 -- 172.21.15.114:0/1838615490 shutdown_connections 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1edc00640 1 -- 172.21.15.114:0/1838615490 >> 172.21.15.114:0/1838615490 conn(0x7fc1e80d3d80 msgr2=0x7fc1e80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1edc00640 1 -- 172.21.15.114:0/1838615490 shutdown_connections 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.976+0000 7fc1edc00640 1 -- 172.21.15.114:0/1838615490 wait complete. 2024-03-06T07:32:27.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.978+0000 7fc1edc00640 1 Processor -- start 2024-03-06T07:32:27.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.978+0000 7fc1edc00640 1 -- start start 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.979+0000 7fc1edc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc0039e0 0x7fc1e814e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.979+0000 7fc1edc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e815d7a0 0x7fc1e80ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.979+0000 7fc1edc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1e80c7680 0x7fc1e8153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.979+0000 7fc1edc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80d2c30 con 0x7fc1e80c7680 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.979+0000 7fc1edc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80b39a0 con 0x7fc1cc0039e0 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.979+0000 7fc1edc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80cf3a0 con 0x7fc1e815d7a0 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.979+0000 7fc1ed200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1e80c7680 0x7fc1e8153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1e80c7680 0x7fc1e8153b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58632/0 (socket says 172.21.15.114:58632) 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 -- 172.21.15.114:0/3591065402 learned_addr learned my addr 172.21.15.114:0/3591065402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1e7e00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc0039e0 0x7fc1e814e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1e7400640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e815d7a0 0x7fc1e80ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 -- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e815d7a0 msgr2=0x7fc1e80ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e815d7a0 0x7fc1e80ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 -- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc0039e0 msgr2=0x7fc1e814e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc0039e0 0x7fc1e814e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 -- 172.21.15.114:0/3591065402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e80caa80 con 0x7fc1e80c7680 2024-03-06T07:32:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.980+0000 7fc1ed200640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1e80c7680 0x7fc1e8153b60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc1e808dff0 tx=0x7fc1e000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.981+0000 7fc1dfe00640 1 -- 172.21.15.114:0/3591065402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e001c070 con 0x7fc1e80c7680 2024-03-06T07:32:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.981+0000 7fc1dfe00640 1 -- 172.21.15.114:0/3591065402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e0017400 con 0x7fc1e80c7680 2024-03-06T07:32:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.981+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e80b7db0 con 0x7fc1e80c7680 2024-03-06T07:32:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.981+0000 7fc1dfe00640 1 -- 172.21.15.114:0/3591065402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e00177f0 con 0x7fc1e80c7680 2024-03-06T07:32:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.981+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e80ca6c0 con 0x7fc1e80c7680 2024-03-06T07:32:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.982+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e80b7ba0 con 0x7fc1e80c7680 2024-03-06T07:32:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.983+0000 7fc1dfe00640 1 -- 172.21.15.114:0/3591065402 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1e001f070 con 0x7fc1e80c7680 2024-03-06T07:32:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.984+0000 7fc1dfe00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1c806b4d0 0x7fc1c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.984+0000 7fc1dfe00640 1 -- 172.21.15.114:0/3591065402 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1e008a150 con 0x7fc1e80c7680 2024-03-06T07:32:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.984+0000 7fc1e7e00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1c806b4d0 0x7fc1c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.985+0000 7fc1e7e00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1c806b4d0 0x7fc1c806d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8036000 tx=0x7fc1d8031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:27.993+0000 7fc1dfe00640 1 -- 172.21.15.114:0/3591065402 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1e0012070 con 0x7fc1e80c7680 2024-03-06T07:32:28.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.192+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1e804f330 con 0x7fc1c806b4d0 2024-03-06T07:32:28.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.210+0000 7fc1dfe00640 1 -- 172.21.15.114:0/3591065402 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252074 (secure 0 0 0) 0x7fc1e804f330 con 0x7fc1c806b4d0 2024-03-06T07:32:28.214 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:28.216 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1c806b4d0 msgr2=0x7fc1c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1c806b4d0 0x7fc1c806d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8036000 tx=0x7fc1d8031000 comp rx=0 tx=0).stop 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1e80c7680 msgr2=0x7fc1e8153b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1e80c7680 0x7fc1e8153b60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc1e808dff0 tx=0x7fc1e000f4a0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 shutdown_connections 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1c806b4d0 0x7fc1c806d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1e80c7680 0x7fc1e8153b60 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1e815d7a0 0x7fc1e80ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 --2- 172.21.15.114:0/3591065402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1cc0039e0 0x7fc1e814e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 >> 172.21.15.114:0/3591065402 conn(0x7fc1e80d3d80 msgr2=0x7fc1e815e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 shutdown_connections 2024-03-06T07:32:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.216+0000 7fc1edc00640 1 -- 172.21.15.114:0/3591065402 wait complete. 2024-03-06T07:32:28.242 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-03-06T07:32:27.182202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258376,"kb_used_data":8904,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716344,"statfs":{"total":773094113280,"available":772829536256,"internally_reserved":0,"allocated":9117696,"data_stored":4521836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.850963"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364687+0000","last_change":"2024-03-06T07:32:18.364516+0000","last_active":"2024-03-06T07:32:20.364687+0000","last_peered":"2024-03-06T07:32:20.364687+0000","last_clean":"2024-03-06T07:32:20.364687+0000","last_became_active":"2024-03-06T07:32:18.364145+0000","last_became_peered":"2024-03-06T07:32:18.364145+0000","last_unstale":"2024-03-06T07:32:20.364687+0000","last_undegraded":"2024-03-06T07:32:20.364687+0000","last_fullsized":"2024-03-06T07:32:20.364687+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:46.535835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.353620+0000","last_change":"2024-03-06T07:32:18.365340+0000","last_active":"2024-03-06T07:32:19.353620+0000","last_peered":"2024-03-06T07:32:19.353620+0000","last_clean":"2024-03-06T07:32:19.353620+0000","last_became_active":"2024-03-06T07:32:18.364393+0000","last_became_peered":"2024-03-06T07:32:18.364393+0000","last_unstale":"2024-03-06T07:32:19.353620+0000","last_undegraded":"2024-03-06T07:32:19.353620+0000","last_fullsized":"2024-03-06T07:32:19.353620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:04.202567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362954+0000","last_change":"2024-03-06T07:32:18.371614+0000","last_active":"2024-03-06T07:32:20.362954+0000","last_peered":"2024-03-06T07:32:20.362954+0000","last_clean":"2024-03-06T07:32:20.362954+0000","last_became_active":"2024-03-06T07:32:18.370824+0000","last_became_peered":"2024-03-06T07:32:18.370824+0000","last_unstale":"2024-03-06T07:32:20.362954+0000","last_undegraded":"2024-03-06T07:32:20.362954+0000","last_fullsized":"2024-03-06T07:32:20.362954+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:39.973458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364082+0000","last_change":"2024-03-06T07:32:18.370338+0000","last_active":"2024-03-06T07:32:20.364082+0000","last_peered":"2024-03-06T07:32:20.364082+0000","last_clean":"2024-03-06T07:32:20.364082+0000","last_became_active":"2024-03-06T07:32:18.369724+0000","last_became_peered":"2024-03-06T07:32:18.369724+0000","last_unstale":"2024-03-06T07:32:20.364082+0000","last_undegraded":"2024-03-06T07:32:20.364082+0000","last_fullsized":"2024-03-06T07:32:20.364082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:41.488379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355536+0000","last_change":"2024-03-06T07:32:18.350611+0000","last_active":"2024-03-06T07:32:19.355536+0000","last_peered":"2024-03-06T07:32:19.355536+0000","last_clean":"2024-03-06T07:32:19.355536+0000","last_became_active":"2024-03-06T07:32:18.350095+0000","last_became_peered":"2024-03-06T07:32:18.350095+0000","last_unstale":"2024-03-06T07:32:19.355536+0000","last_undegraded":"2024-03-06T07:32:19.355536+0000","last_fullsized":"2024-03-06T07:32:19.355536+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:16.997454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.665059+0000","last_change":"2024-03-06T07:32:18.373517+0000","last_active":"2024-03-06T07:32:19.665059+0000","last_peered":"2024-03-06T07:32:19.665059+0000","last_clean":"2024-03-06T07:32:19.665059+0000","last_became_active":"2024-03-06T07:32:18.372542+0000","last_became_peered":"2024-03-06T07:32:18.372542+0000","last_unstale":"2024-03-06T07:32:19.665059+0000","last_undegraded":"2024-03-06T07:32:19.665059+0000","last_fullsized":"2024-03-06T07:32:19.665059+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:51.584539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362907+0000","last_change":"2024-03-06T07:32:18.371589+0000","last_active":"2024-03-06T07:32:20.362907+0000","last_peered":"2024-03-06T07:32:20.362907+0000","last_clean":"2024-03-06T07:32:20.362907+0000","last_became_active":"2024-03-06T07:32:18.369535+0000","last_became_peered":"2024-03-06T07:32:18.369535+0000","last_unstale":"2024-03-06T07:32:20.362907+0000","last_undegraded":"2024-03-06T07:32:20.362907+0000","last_fullsized":"2024-03-06T07:32:20.362907+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:48.294087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.664946+0000","last_change":"2024-03-06T07:32:18.361961+0000","last_active":"2024-03-06T07:32:19.664946+0000","last_peered":"2024-03-06T07:32:19.664946+0000","last_clean":"2024-03-06T07:32:19.664946+0000","last_became_active":"2024-03-06T07:32:18.361456+0000","last_became_peered":"2024-03-06T07:32:18.361456+0000","last_unstale":"2024-03-06T07:32:19.664946+0000","last_undegraded":"2024-03-06T07:32:19.664946+0000","last_fullsized":"2024-03-06T07:32:19.664946+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:16.784844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.664852+0000","last_change":"2024-03-06T07:32:18.367546+0000","last_active":"2024-03-06T07:32:19.664852+0000","last_peered":"2024-03-06T07:32:19.664852+0000","last_clean":"2024-03-06T07:32:19.664852+0000","last_became_active":"2024-03-06T07:32:18.367284+0000","last_became_peered":"2024-03-06T07:32:18.367284+0000","last_unstale":"2024-03-06T07:32:19.664852+0000","last_undegraded":"2024-03-06T07:32:19.664852+0000","last_fullsized":"2024-03-06T07:32:19.664852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:48.610860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364395+0000","last_change":"2024-03-06T07:32:18.364660+0000","last_active":"2024-03-06T07:32:20.364395+0000","last_peered":"2024-03-06T07:32:20.364395+0000","last_clean":"2024-03-06T07:32:20.364395+0000","last_became_active":"2024-03-06T07:32:18.363060+0000","last_became_peered":"2024-03-06T07:32:18.363060+0000","last_unstale":"2024-03-06T07:32:20.364395+0000","last_undegraded":"2024-03-06T07:32:20.364395+0000","last_fullsized":"2024-03-06T07:32:20.364395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:34.432657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.353251+0000","last_change":"2024-03-06T07:32:18.364265+0000","last_active":"2024-03-06T07:32:19.353251+0000","last_peered":"2024-03-06T07:32:19.353251+0000","last_clean":"2024-03-06T07:32:19.353251+0000","last_became_active":"2024-03-06T07:32:18.363617+0000","last_became_peered":"2024-03-06T07:32:18.363617+0000","last_unstale":"2024-03-06T07:32:19.353251+0000","last_undegraded":"2024-03-06T07:32:19.353251+0000","last_fullsized":"2024-03-06T07:32:19.353251+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:33.494965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362694+0000","last_change":"2024-03-06T07:32:18.364345+0000","last_active":"2024-03-06T07:32:20.362694+0000","last_peered":"2024-03-06T07:32:20.362694+0000","last_clean":"2024-03-06T07:32:20.362694+0000","last_became_active":"2024-03-06T07:32:18.363796+0000","last_became_peered":"2024-03-06T07:32:18.363796+0000","last_unstale":"2024-03-06T07:32:20.362694+0000","last_undegraded":"2024-03-06T07:32:20.362694+0000","last_fullsized":"2024-03-06T07:32:20.362694+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:16.784844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362781+0000","last_change":"2024-03-06T07:32:18.367320+0000","last_active":"2024-03-06T07:32:20.362781+0000","last_peered":"2024-03-06T07:32:20.362781+0000","last_clean":"2024-03-06T07:32:20.362781+0000","last_became_active":"2024-03-06T07:32:18.366597+0000","last_became_peered":"2024-03-06T07:32:18.366597+0000","last_unstale":"2024-03-06T07:32:20.362781+0000","last_undegraded":"2024-03-06T07:32:20.362781+0000","last_fullsized":"2024-03-06T07:32:20.362781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:35.788638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363397+0000","last_change":"2024-03-06T07:32:18.370137+0000","last_active":"2024-03-06T07:32:20.363397+0000","last_peered":"2024-03-06T07:32:20.363397+0000","last_clean":"2024-03-06T07:32:20.363397+0000","last_became_active":"2024-03-06T07:32:18.369869+0000","last_became_peered":"2024-03-06T07:32:18.369869+0000","last_unstale":"2024-03-06T07:32:20.363397+0000","last_undegraded":"2024-03-06T07:32:20.363397+0000","last_fullsized":"2024-03-06T07:32:20.363397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:48.197213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.354267+0000","last_change":"2024-03-06T07:32:18.362065+0000","last_active":"2024-03-06T07:32:19.354267+0000","last_peered":"2024-03-06T07:32:19.354267+0000","last_clean":"2024-03-06T07:32:19.354267+0000","last_became_active":"2024-03-06T07:32:18.361752+0000","last_became_peered":"2024-03-06T07:32:18.361752+0000","last_unstale":"2024-03-06T07:32:19.354267+0000","last_undegraded":"2024-03-06T07:32:19.354267+0000","last_fullsized":"2024-03-06T07:32:19.354267+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:49.815791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.354470+0000","last_change":"2024-03-06T07:32:18.363311+0000","last_active":"2024-03-06T07:32:19.354470+0000","last_peered":"2024-03-06T07:32:19.354470+0000","last_clean":"2024-03-06T07:32:19.354470+0000","last_became_active":"2024-03-06T07:32:18.363001+0000","last_became_peered":"2024-03-06T07:32:18.363001+0000","last_unstale":"2024-03-06T07:32:19.354470+0000","last_undegraded":"2024-03-06T07:32:19.354470+0000","last_fullsized":"2024-03-06T07:32:19.354470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:50.208509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.356674+0000","last_change":"2024-03-06T07:32:18.374274+0000","last_active":"2024-03-06T07:32:19.356674+0000","last_peered":"2024-03-06T07:32:19.356674+0000","last_clean":"2024-03-06T07:32:19.356674+0000","last_became_active":"2024-03-06T07:32:18.370193+0000","last_became_peered":"2024-03-06T07:32:18.370193+0000","last_unstale":"2024-03-06T07:32:19.356674+0000","last_undegraded":"2024-03-06T07:32:19.356674+0000","last_fullsized":"2024-03-06T07:32:19.356674+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:01.334957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363876+0000","last_change":"2024-03-06T07:32:18.367015+0000","last_active":"2024-03-06T07:32:20.363876+0000","last_peered":"2024-03-06T07:32:20.363876+0000","last_clean":"2024-03-06T07:32:20.363876+0000","last_became_active":"2024-03-06T07:32:18.366743+0000","last_became_peered":"2024-03-06T07:32:18.366743+0000","last_unstale":"2024-03-06T07:32:20.363876+0000","last_undegraded":"2024-03-06T07:32:20.363876+0000","last_fullsized":"2024-03-06T07:32:20.363876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:59.443064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932784+0000","last_change":"2024-03-06T07:32:18.362846+0000","last_active":"2024-03-06T07:32:19.932784+0000","last_peered":"2024-03-06T07:32:19.932784+0000","last_clean":"2024-03-06T07:32:19.932784+0000","last_became_active":"2024-03-06T07:32:18.362560+0000","last_became_peered":"2024-03-06T07:32:18.362560+0000","last_unstale":"2024-03-06T07:32:19.932784+0000","last_undegraded":"2024-03-06T07:32:19.932784+0000","last_fullsized":"2024-03-06T07:32:19.932784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:34.432657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363930+0000","last_change":"2024-03-06T07:32:18.370080+0000","last_active":"2024-03-06T07:32:20.363930+0000","last_peered":"2024-03-06T07:32:20.363930+0000","last_clean":"2024-03-06T07:32:20.363930+0000","last_became_active":"2024-03-06T07:32:18.369316+0000","last_became_peered":"2024-03-06T07:32:18.369316+0000","last_unstale":"2024-03-06T07:32:20.363930+0000","last_undegraded":"2024-03-06T07:32:20.363930+0000","last_fullsized":"2024-03-06T07:32:20.363930+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:24.824705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363397+0000","last_change":"2024-03-06T07:32:18.378893+0000","last_active":"2024-03-06T07:32:20.363397+0000","last_peered":"2024-03-06T07:32:20.363397+0000","last_clean":"2024-03-06T07:32:20.363397+0000","last_became_active":"2024-03-06T07:32:18.378562+0000","last_became_peered":"2024-03-06T07:32:18.378562+0000","last_unstale":"2024-03-06T07:32:20.363397+0000","last_undegraded":"2024-03-06T07:32:20.363397+0000","last_fullsized":"2024-03-06T07:32:20.363397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:43.450037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.352087+0000","last_change":"2024-03-06T07:32:18.366196+0000","last_active":"2024-03-06T07:32:19.352087+0000","last_peered":"2024-03-06T07:32:19.352087+0000","last_clean":"2024-03-06T07:32:19.352087+0000","last_became_active":"2024-03-06T07:32:18.363764+0000","last_became_peered":"2024-03-06T07:32:18.363764+0000","last_unstale":"2024-03-06T07:32:19.352087+0000","last_undegraded":"2024-03-06T07:32:19.352087+0000","last_fullsized":"2024-03-06T07:32:19.352087+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:31.933884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362542+0000","last_change":"2024-03-06T07:32:18.361975+0000","last_active":"2024-03-06T07:32:20.362542+0000","last_peered":"2024-03-06T07:32:20.362542+0000","last_clean":"2024-03-06T07:32:20.362542+0000","last_became_active":"2024-03-06T07:32:18.361605+0000","last_became_peered":"2024-03-06T07:32:18.361605+0000","last_unstale":"2024-03-06T07:32:20.362542+0000","last_undegraded":"2024-03-06T07:32:20.362542+0000","last_fullsized":"2024-03-06T07:32:20.362542+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:48.610860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363138+0000","last_change":"2024-03-06T07:32:18.372938+0000","last_active":"2024-03-06T07:32:20.363138+0000","last_peered":"2024-03-06T07:32:20.363138+0000","last_clean":"2024-03-06T07:32:20.363138+0000","last_became_active":"2024-03-06T07:32:18.369595+0000","last_became_peered":"2024-03-06T07:32:18.369595+0000","last_unstale":"2024-03-06T07:32:20.363138+0000","last_undegraded":"2024-03-06T07:32:20.363138+0000","last_fullsized":"2024-03-06T07:32:20.363138+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:28.659930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.353792+0000","last_change":"2024-03-06T07:32:18.365113+0000","last_active":"2024-03-06T07:32:19.353792+0000","last_peered":"2024-03-06T07:32:19.353792+0000","last_clean":"2024-03-06T07:32:19.353792+0000","last_became_active":"2024-03-06T07:32:18.364811+0000","last_became_peered":"2024-03-06T07:32:18.364811+0000","last_unstale":"2024-03-06T07:32:19.353792+0000","last_undegraded":"2024-03-06T07:32:19.353792+0000","last_fullsized":"2024-03-06T07:32:19.353792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:30.015911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362689+0000","last_change":"2024-03-06T07:32:18.368294+0000","last_active":"2024-03-06T07:32:20.362689+0000","last_peered":"2024-03-06T07:32:20.362689+0000","last_clean":"2024-03-06T07:32:20.362689+0000","last_became_active":"2024-03-06T07:32:18.367782+0000","last_became_peered":"2024-03-06T07:32:18.367782+0000","last_unstale":"2024-03-06T07:32:20.362689+0000","last_undegraded":"2024-03-06T07:32:20.362689+0000","last_fullsized":"2024-03-06T07:32:20.362689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:46.535835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355845+0000","last_change":"2024-03-06T07:32:18.355094+0000","last_active":"2024-03-06T07:32:19.355845+0000","last_peered":"2024-03-06T07:32:19.355845+0000","last_clean":"2024-03-06T07:32:19.355845+0000","last_became_active":"2024-03-06T07:32:18.354563+0000","last_became_peered":"2024-03-06T07:32:18.354563+0000","last_unstale":"2024-03-06T07:32:19.355845+0000","last_undegraded":"2024-03-06T07:32:19.355845+0000","last_fullsized":"2024-03-06T07:32:19.355845+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:16.784844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363189+0000","last_change":"2024-03-06T07:32:18.369960+0000","last_active":"2024-03-06T07:32:20.363189+0000","last_peered":"2024-03-06T07:32:20.363189+0000","last_clean":"2024-03-06T07:32:20.363189+0000","last_became_active":"2024-03-06T07:32:18.369647+0000","last_became_peered":"2024-03-06T07:32:18.369647+0000","last_unstale":"2024-03-06T07:32:20.363189+0000","last_undegraded":"2024-03-06T07:32:20.363189+0000","last_fullsized":"2024-03-06T07:32:20.363189+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:26.865827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.352630+0000","last_change":"2024-03-06T07:32:18.348876+0000","last_active":"2024-03-06T07:32:19.352630+0000","last_peered":"2024-03-06T07:32:19.352630+0000","last_clean":"2024-03-06T07:32:19.352630+0000","last_became_active":"2024-03-06T07:32:18.348316+0000","last_became_peered":"2024-03-06T07:32:18.348316+0000","last_unstale":"2024-03-06T07:32:19.352630+0000","last_undegraded":"2024-03-06T07:32:19.352630+0000","last_fullsized":"2024-03-06T07:32:19.352630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:18.658383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.352369+0000","last_change":"2024-03-06T07:32:18.375876+0000","last_active":"2024-03-06T07:32:19.352369+0000","last_peered":"2024-03-06T07:32:19.352369+0000","last_clean":"2024-03-06T07:32:19.352369+0000","last_became_active":"2024-03-06T07:32:18.374612+0000","last_became_peered":"2024-03-06T07:32:18.374612+0000","last_unstale":"2024-03-06T07:32:19.352369+0000","last_undegraded":"2024-03-06T07:32:19.352369+0000","last_fullsized":"2024-03-06T07:32:19.352369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:40.338187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355205+0000","last_change":"2024-03-06T07:32:18.365100+0000","last_active":"2024-03-06T07:32:19.355205+0000","last_peered":"2024-03-06T07:32:19.355205+0000","last_clean":"2024-03-06T07:32:19.355205+0000","last_became_active":"2024-03-06T07:32:18.363928+0000","last_became_peered":"2024-03-06T07:32:18.363928+0000","last_unstale":"2024-03-06T07:32:19.355205+0000","last_undegraded":"2024-03-06T07:32:19.355205+0000","last_fullsized":"2024-03-06T07:32:19.355205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:10.667854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363046+0000","last_change":"2024-03-06T07:32:18.364701+0000","last_active":"2024-03-06T07:32:20.363046+0000","last_peered":"2024-03-06T07:32:20.363046+0000","last_clean":"2024-03-06T07:32:20.363046+0000","last_became_active":"2024-03-06T07:32:18.364053+0000","last_became_peered":"2024-03-06T07:32:18.364053+0000","last_unstale":"2024-03-06T07:32:20.363046+0000","last_undegraded":"2024-03-06T07:32:20.363046+0000","last_fullsized":"2024-03-06T07:32:20.363046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:46.703475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355767+0000","last_change":"2024-03-06T07:32:18.363619+0000","last_active":"2024-03-06T07:32:19.355767+0000","last_peered":"2024-03-06T07:32:19.355767+0000","last_clean":"2024-03-06T07:32:19.355767+0000","last_became_active":"2024-03-06T07:32:18.363284+0000","last_became_peered":"2024-03-06T07:32:18.363284+0000","last_unstale":"2024-03-06T07:32:19.355767+0000","last_undegraded":"2024-03-06T07:32:19.355767+0000","last_fullsized":"2024-03-06T07:32:19.355767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:48.610860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364972+0000","last_change":"2024-03-06T07:32:17.340541+0000","last_active":"2024-03-06T07:32:20.364972+0000","last_peered":"2024-03-06T07:32:20.364972+0000","last_clean":"2024-03-06T07:32:20.364972+0000","last_became_active":"2024-03-06T07:32:17.338327+0000","last_became_peered":"2024-03-06T07:32:17.338327+0000","last_unstale":"2024-03-06T07:32:20.364972+0000","last_undegraded":"2024-03-06T07:32:20.364972+0000","last_fullsized":"2024-03-06T07:32:20.364972+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:19.628620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364609+0000","last_change":"2024-03-06T07:32:18.368467+0000","last_active":"2024-03-06T07:32:20.364609+0000","last_peered":"2024-03-06T07:32:20.364609+0000","last_clean":"2024-03-06T07:32:20.364609+0000","last_became_active":"2024-03-06T07:32:18.366788+0000","last_became_peered":"2024-03-06T07:32:18.366788+0000","last_unstale":"2024-03-06T07:32:20.364609+0000","last_undegraded":"2024-03-06T07:32:20.364609+0000","last_fullsized":"2024-03-06T07:32:20.364609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:42.935188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.356366+0000","last_change":"2024-03-06T07:32:17.346963+0000","last_active":"2024-03-06T07:32:19.356366+0000","last_peered":"2024-03-06T07:32:19.356366+0000","last_clean":"2024-03-06T07:32:19.356366+0000","last_became_active":"2024-03-06T07:32:17.346591+0000","last_became_peered":"2024-03-06T07:32:17.346591+0000","last_unstale":"2024-03-06T07:32:19.356366+0000","last_undegraded":"2024-03-06T07:32:19.356366+0000","last_fullsized":"2024-03-06T07:32:19.356366+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:03.374987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363534+0000","last_change":"2024-03-06T07:32:18.362416+0000","last_active":"2024-03-06T07:32:20.363534+0000","last_peered":"2024-03-06T07:32:20.363534+0000","last_clean":"2024-03-06T07:32:20.363534+0000","last_became_active":"2024-03-06T07:32:18.361268+0000","last_became_peered":"2024-03-06T07:32:18.361268+0000","last_unstale":"2024-03-06T07:32:20.363534+0000","last_undegraded":"2024-03-06T07:32:20.363534+0000","last_fullsized":"2024-03-06T07:32:20.363534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:19.899404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932855+0000","last_change":"2024-03-06T07:32:17.334402+0000","last_active":"2024-03-06T07:32:19.932855+0000","last_peered":"2024-03-06T07:32:19.932855+0000","last_clean":"2024-03-06T07:32:19.932855+0000","last_became_active":"2024-03-06T07:32:17.334125+0000","last_became_peered":"2024-03-06T07:32:17.334125+0000","last_unstale":"2024-03-06T07:32:19.932855+0000","last_undegraded":"2024-03-06T07:32:19.932855+0000","last_fullsized":"2024-03-06T07:32:19.932855+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:34.681463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363354+0000","last_change":"2024-03-06T07:32:18.378817+0000","last_active":"2024-03-06T07:32:20.363354+0000","last_peered":"2024-03-06T07:32:20.363354+0000","last_clean":"2024-03-06T07:32:20.363354+0000","last_became_active":"2024-03-06T07:32:18.378466+0000","last_became_peered":"2024-03-06T07:32:18.378466+0000","last_unstale":"2024-03-06T07:32:20.363354+0000","last_undegraded":"2024-03-06T07:32:20.363354+0000","last_fullsized":"2024-03-06T07:32:20.363354+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:48.502146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932537+0000","last_change":"2024-03-06T07:32:17.333923+0000","last_active":"2024-03-06T07:32:19.932537+0000","last_peered":"2024-03-06T07:32:19.932537+0000","last_clean":"2024-03-06T07:32:19.932537+0000","last_became_active":"2024-03-06T07:32:17.333343+0000","last_became_peered":"2024-03-06T07:32:17.333343+0000","last_unstale":"2024-03-06T07:32:19.932537+0000","last_undegraded":"2024-03-06T07:32:19.932537+0000","last_fullsized":"2024-03-06T07:32:19.932537+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:09.653046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.352366+0000","last_change":"2024-03-06T07:32:18.376298+0000","last_active":"2024-03-06T07:32:19.352366+0000","last_peered":"2024-03-06T07:32:19.352366+0000","last_clean":"2024-03-06T07:32:19.352366+0000","last_became_active":"2024-03-06T07:32:18.374939+0000","last_became_peered":"2024-03-06T07:32:18.374939+0000","last_unstale":"2024-03-06T07:32:19.352366+0000","last_undegraded":"2024-03-06T07:32:19.352366+0000","last_fullsized":"2024-03-06T07:32:19.352366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:02.503300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.356380+0000","last_change":"2024-03-06T07:32:17.343520+0000","last_active":"2024-03-06T07:32:19.356380+0000","last_peered":"2024-03-06T07:32:19.356380+0000","last_clean":"2024-03-06T07:32:19.356380+0000","last_became_active":"2024-03-06T07:32:17.343250+0000","last_became_peered":"2024-03-06T07:32:17.343250+0000","last_unstale":"2024-03-06T07:32:19.356380+0000","last_undegraded":"2024-03-06T07:32:19.356380+0000","last_fullsized":"2024-03-06T07:32:19.356380+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:03.187760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355522+0000","last_change":"2024-03-06T07:32:18.371303+0000","last_active":"2024-03-06T07:32:19.355522+0000","last_peered":"2024-03-06T07:32:19.355522+0000","last_clean":"2024-03-06T07:32:19.355522+0000","last_became_active":"2024-03-06T07:32:18.368998+0000","last_became_peered":"2024-03-06T07:32:18.368998+0000","last_unstale":"2024-03-06T07:32:19.355522+0000","last_undegraded":"2024-03-06T07:32:19.355522+0000","last_fullsized":"2024-03-06T07:32:19.355522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:35.696271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355103+0000","last_change":"2024-03-06T07:32:17.346884+0000","last_active":"2024-03-06T07:32:19.355103+0000","last_peered":"2024-03-06T07:32:19.355103+0000","last_clean":"2024-03-06T07:32:19.355103+0000","last_became_active":"2024-03-06T07:32:17.346080+0000","last_became_peered":"2024-03-06T07:32:17.346080+0000","last_unstale":"2024-03-06T07:32:19.355103+0000","last_undegraded":"2024-03-06T07:32:19.355103+0000","last_fullsized":"2024-03-06T07:32:19.355103+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:13.325884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.361600+0000","last_change":"2024-03-06T07:32:18.370110+0000","last_active":"2024-03-06T07:32:20.361600+0000","last_peered":"2024-03-06T07:32:20.361600+0000","last_clean":"2024-03-06T07:32:20.361600+0000","last_became_active":"2024-03-06T07:32:18.369545+0000","last_became_peered":"2024-03-06T07:32:18.369545+0000","last_unstale":"2024-03-06T07:32:20.361600+0000","last_undegraded":"2024-03-06T07:32:20.361600+0000","last_fullsized":"2024-03-06T07:32:20.361600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:42.935188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.931862+0000","last_change":"2024-03-06T07:32:17.336331+0000","last_active":"2024-03-06T07:32:19.931862+0000","last_peered":"2024-03-06T07:32:19.931862+0000","last_clean":"2024-03-06T07:32:19.931862+0000","last_became_active":"2024-03-06T07:32:17.336045+0000","last_became_peered":"2024-03-06T07:32:17.336045+0000","last_unstale":"2024-03-06T07:32:19.931862+0000","last_undegraded":"2024-03-06T07:32:19.931862+0000","last_fullsized":"2024-03-06T07:32:19.931862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:30.314302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.354347+0000","last_change":"2024-03-06T07:32:18.363372+0000","last_active":"2024-03-06T07:32:19.354347+0000","last_peered":"2024-03-06T07:32:19.354347+0000","last_clean":"2024-03-06T07:32:19.354347+0000","last_became_active":"2024-03-06T07:32:18.363052+0000","last_became_peered":"2024-03-06T07:32:18.363052+0000","last_unstale":"2024-03-06T07:32:19.354347+0000","last_undegraded":"2024-03-06T07:32:19.354347+0000","last_fullsized":"2024-03-06T07:32:19.354347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:27.363269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.352081+0000","last_change":"2024-03-06T07:32:17.342249+0000","last_active":"2024-03-06T07:32:19.352081+0000","last_peered":"2024-03-06T07:32:19.352081+0000","last_clean":"2024-03-06T07:32:19.352081+0000","last_became_active":"2024-03-06T07:32:17.341962+0000","last_became_peered":"2024-03-06T07:32:17.341962+0000","last_unstale":"2024-03-06T07:32:19.352081+0000","last_undegraded":"2024-03-06T07:32:19.352081+0000","last_fullsized":"2024-03-06T07:32:19.352081+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:48.149538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355284+0000","last_change":"2024-03-06T07:32:18.348255+0000","last_active":"2024-03-06T07:32:19.355284+0000","last_peered":"2024-03-06T07:32:19.355284+0000","last_clean":"2024-03-06T07:32:19.355284+0000","last_became_active":"2024-03-06T07:32:18.347472+0000","last_became_peered":"2024-03-06T07:32:18.347472+0000","last_unstale":"2024-03-06T07:32:19.355284+0000","last_undegraded":"2024-03-06T07:32:19.355284+0000","last_fullsized":"2024-03-06T07:32:19.355284+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:30.214629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364983+0000","last_change":"2024-03-06T07:32:17.337307+0000","last_active":"2024-03-06T07:32:20.364983+0000","last_peered":"2024-03-06T07:32:20.364983+0000","last_clean":"2024-03-06T07:32:20.364983+0000","last_became_active":"2024-03-06T07:32:17.335584+0000","last_became_peered":"2024-03-06T07:32:17.335584+0000","last_unstale":"2024-03-06T07:32:20.364983+0000","last_undegraded":"2024-03-06T07:32:20.364983+0000","last_fullsized":"2024-03-06T07:32:20.364983+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:45.515629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.665165+0000","last_change":"2024-03-06T07:32:18.369541+0000","last_active":"2024-03-06T07:32:19.665165+0000","last_peered":"2024-03-06T07:32:19.665165+0000","last_clean":"2024-03-06T07:32:19.665165+0000","last_became_active":"2024-03-06T07:32:18.369104+0000","last_became_peered":"2024-03-06T07:32:18.369104+0000","last_unstale":"2024-03-06T07:32:19.665165+0000","last_undegraded":"2024-03-06T07:32:19.665165+0000","last_fullsized":"2024-03-06T07:32:19.665165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:46.703475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364390+0000","last_change":"2024-03-06T07:32:17.343053+0000","last_active":"2024-03-06T07:32:20.364390+0000","last_peered":"2024-03-06T07:32:20.364390+0000","last_clean":"2024-03-06T07:32:20.364390+0000","last_became_active":"2024-03-06T07:32:17.342760+0000","last_became_peered":"2024-03-06T07:32:17.342760+0000","last_unstale":"2024-03-06T07:32:20.364390+0000","last_undegraded":"2024-03-06T07:32:20.364390+0000","last_fullsized":"2024-03-06T07:32:20.364390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.148717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932436+0000","last_change":"2024-03-06T07:32:18.374186+0000","last_active":"2024-03-06T07:32:19.932436+0000","last_peered":"2024-03-06T07:32:19.932436+0000","last_clean":"2024-03-06T07:32:19.932436+0000","last_became_active":"2024-03-06T07:32:18.373933+0000","last_became_peered":"2024-03-06T07:32:18.373933+0000","last_unstale":"2024-03-06T07:32:19.932436+0000","last_undegraded":"2024-03-06T07:32:19.932436+0000","last_fullsized":"2024-03-06T07:32:19.932436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:01.334957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364043+0000","last_change":"2024-03-06T07:32:17.337091+0000","last_active":"2024-03-06T07:32:20.364043+0000","last_peered":"2024-03-06T07:32:20.364043+0000","last_clean":"2024-03-06T07:32:20.364043+0000","last_became_active":"2024-03-06T07:32:17.335040+0000","last_became_peered":"2024-03-06T07:32:17.335040+0000","last_unstale":"2024-03-06T07:32:20.364043+0000","last_undegraded":"2024-03-06T07:32:20.364043+0000","last_fullsized":"2024-03-06T07:32:20.364043+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:29.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355882+0000","last_change":"2024-03-06T07:32:18.374531+0000","last_active":"2024-03-06T07:32:19.355882+0000","last_peered":"2024-03-06T07:32:19.355882+0000","last_clean":"2024-03-06T07:32:19.355882+0000","last_became_active":"2024-03-06T07:32:18.374220+0000","last_became_peered":"2024-03-06T07:32:18.374220+0000","last_unstale":"2024-03-06T07:32:19.355882+0000","last_undegraded":"2024-03-06T07:32:19.355882+0000","last_fullsized":"2024-03-06T07:32:19.355882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:46.703475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.664831+0000","last_change":"2024-03-06T07:32:17.336112+0000","last_active":"2024-03-06T07:32:19.664831+0000","last_peered":"2024-03-06T07:32:19.664831+0000","last_clean":"2024-03-06T07:32:19.664831+0000","last_became_active":"2024-03-06T07:32:17.335798+0000","last_became_peered":"2024-03-06T07:32:17.335798+0000","last_unstale":"2024-03-06T07:32:19.664831+0000","last_undegraded":"2024-03-06T07:32:19.664831+0000","last_fullsized":"2024-03-06T07:32:19.664831+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:45.515629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932161+0000","last_change":"2024-03-06T07:32:18.361680+0000","last_active":"2024-03-06T07:32:19.932161+0000","last_peered":"2024-03-06T07:32:19.932161+0000","last_clean":"2024-03-06T07:32:19.932161+0000","last_became_active":"2024-03-06T07:32:18.361319+0000","last_became_peered":"2024-03-06T07:32:18.361319+0000","last_unstale":"2024-03-06T07:32:19.932161+0000","last_undegraded":"2024-03-06T07:32:19.932161+0000","last_fullsized":"2024-03-06T07:32:19.932161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:26.865827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.352761+0000","last_change":"2024-03-06T07:32:17.328646+0000","last_active":"2024-03-06T07:32:19.352761+0000","last_peered":"2024-03-06T07:32:19.352761+0000","last_clean":"2024-03-06T07:32:19.352761+0000","last_became_active":"2024-03-06T07:32:17.328345+0000","last_became_peered":"2024-03-06T07:32:17.328345+0000","last_unstale":"2024-03-06T07:32:19.352761+0000","last_undegraded":"2024-03-06T07:32:19.352761+0000","last_fullsized":"2024-03-06T07:32:19.352761+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:14.394216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362740+0000","last_change":"2024-03-06T07:32:18.373371+0000","last_active":"2024-03-06T07:32:20.362740+0000","last_peered":"2024-03-06T07:32:20.362740+0000","last_clean":"2024-03-06T07:32:20.362740+0000","last_became_active":"2024-03-06T07:32:18.372967+0000","last_became_peered":"2024-03-06T07:32:18.372967+0000","last_unstale":"2024-03-06T07:32:20.362740+0000","last_undegraded":"2024-03-06T07:32:20.362740+0000","last_fullsized":"2024-03-06T07:32:20.362740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:49.815791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.354556+0000","last_change":"2024-03-06T07:32:17.336488+0000","last_active":"2024-03-06T07:32:19.354556+0000","last_peered":"2024-03-06T07:32:19.354556+0000","last_clean":"2024-03-06T07:32:19.354556+0000","last_became_active":"2024-03-06T07:32:17.336208+0000","last_became_peered":"2024-03-06T07:32:17.336208+0000","last_unstale":"2024-03-06T07:32:19.354556+0000","last_undegraded":"2024-03-06T07:32:19.354556+0000","last_fullsized":"2024-03-06T07:32:19.354556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:34.773830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.352201+0000","last_change":"2024-03-06T07:32:18.369819+0000","last_active":"2024-03-06T07:32:19.352201+0000","last_peered":"2024-03-06T07:32:19.352201+0000","last_clean":"2024-03-06T07:32:19.352201+0000","last_became_active":"2024-03-06T07:32:18.369227+0000","last_became_peered":"2024-03-06T07:32:18.369227+0000","last_unstale":"2024-03-06T07:32:19.352201+0000","last_undegraded":"2024-03-06T07:32:19.352201+0000","last_fullsized":"2024-03-06T07:32:19.352201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:24.869252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.664503+0000","last_change":"2024-03-06T07:32:17.344364+0000","last_active":"2024-03-06T07:32:19.664503+0000","last_peered":"2024-03-06T07:32:19.664503+0000","last_clean":"2024-03-06T07:32:19.664503+0000","last_became_active":"2024-03-06T07:32:17.342047+0000","last_became_peered":"2024-03-06T07:32:17.342047+0000","last_unstale":"2024-03-06T07:32:19.664503+0000","last_undegraded":"2024-03-06T07:32:19.664503+0000","last_fullsized":"2024-03-06T07:32:19.664503+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:19.628620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.351920+0000","last_change":"2024-03-06T07:32:18.363092+0000","last_active":"2024-03-06T07:32:19.351920+0000","last_peered":"2024-03-06T07:32:19.351920+0000","last_clean":"2024-03-06T07:32:19.351920+0000","last_became_active":"2024-03-06T07:32:18.362759+0000","last_became_peered":"2024-03-06T07:32:18.362759+0000","last_unstale":"2024-03-06T07:32:19.351920+0000","last_undegraded":"2024-03-06T07:32:19.351920+0000","last_fullsized":"2024-03-06T07:32:19.351920+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:45.617267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.931661+0000","last_change":"2024-03-06T07:32:17.343791+0000","last_active":"2024-03-06T07:32:19.931661+0000","last_peered":"2024-03-06T07:32:19.931661+0000","last_clean":"2024-03-06T07:32:19.931661+0000","last_became_active":"2024-03-06T07:32:17.343078+0000","last_became_peered":"2024-03-06T07:32:17.343078+0000","last_unstale":"2024-03-06T07:32:19.931661+0000","last_undegraded":"2024-03-06T07:32:19.931661+0000","last_fullsized":"2024-03-06T07:32:19.931661+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:49.193701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363119+0000","last_change":"2024-03-06T07:32:18.372127+0000","last_active":"2024-03-06T07:32:20.363119+0000","last_peered":"2024-03-06T07:32:20.363119+0000","last_clean":"2024-03-06T07:32:20.363119+0000","last_became_active":"2024-03-06T07:32:18.371433+0000","last_became_peered":"2024-03-06T07:32:18.371433+0000","last_unstale":"2024-03-06T07:32:20.363119+0000","last_undegraded":"2024-03-06T07:32:20.363119+0000","last_fullsized":"2024-03-06T07:32:20.363119+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:02.728397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.365285+0000","last_change":"2024-03-06T07:32:17.344136+0000","last_active":"2024-03-06T07:32:20.365285+0000","last_peered":"2024-03-06T07:32:20.365285+0000","last_clean":"2024-03-06T07:32:20.365285+0000","last_became_active":"2024-03-06T07:32:17.343838+0000","last_became_peered":"2024-03-06T07:32:17.343838+0000","last_unstale":"2024-03-06T07:32:20.365285+0000","last_undegraded":"2024-03-06T07:32:20.365285+0000","last_fullsized":"2024-03-06T07:32:20.365285+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:02.671020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364849+0000","last_change":"2024-03-06T07:32:18.363263+0000","last_active":"2024-03-06T07:32:20.364849+0000","last_peered":"2024-03-06T07:32:20.364849+0000","last_clean":"2024-03-06T07:32:20.364849+0000","last_became_active":"2024-03-06T07:32:18.362947+0000","last_became_peered":"2024-03-06T07:32:18.362947+0000","last_unstale":"2024-03-06T07:32:20.364849+0000","last_undegraded":"2024-03-06T07:32:20.364849+0000","last_fullsized":"2024-03-06T07:32:20.364849+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:35.788638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932711+0000","last_change":"2024-03-06T07:32:17.343396+0000","last_active":"2024-03-06T07:32:19.932711+0000","last_peered":"2024-03-06T07:32:19.932711+0000","last_clean":"2024-03-06T07:32:19.932711+0000","last_became_active":"2024-03-06T07:32:17.343086+0000","last_became_peered":"2024-03-06T07:32:17.343086+0000","last_unstale":"2024-03-06T07:32:19.932711+0000","last_undegraded":"2024-03-06T07:32:19.932711+0000","last_fullsized":"2024-03-06T07:32:19.932711+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:59.737439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.356282+0000","last_change":"2024-03-06T07:32:18.355547+0000","last_active":"2024-03-06T07:32:19.356282+0000","last_peered":"2024-03-06T07:32:19.356282+0000","last_clean":"2024-03-06T07:32:19.356282+0000","last_became_active":"2024-03-06T07:32:18.355115+0000","last_became_peered":"2024-03-06T07:32:18.355115+0000","last_unstale":"2024-03-06T07:32:19.356282+0000","last_undegraded":"2024-03-06T07:32:19.356282+0000","last_fullsized":"2024-03-06T07:32:19.356282+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:26.865827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932579+0000","last_change":"2024-03-06T07:32:17.341403+0000","last_active":"2024-03-06T07:32:19.932579+0000","last_peered":"2024-03-06T07:32:19.932579+0000","last_clean":"2024-03-06T07:32:19.932579+0000","last_became_active":"2024-03-06T07:32:17.340813+0000","last_became_peered":"2024-03-06T07:32:17.340813+0000","last_unstale":"2024-03-06T07:32:19.932579+0000","last_undegraded":"2024-03-06T07:32:19.932579+0000","last_fullsized":"2024-03-06T07:32:19.932579+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:50.270318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363572+0000","last_change":"2024-03-06T07:32:18.370021+0000","last_active":"2024-03-06T07:32:20.363572+0000","last_peered":"2024-03-06T07:32:20.363572+0000","last_clean":"2024-03-06T07:32:20.363572+0000","last_became_active":"2024-03-06T07:32:18.369743+0000","last_became_peered":"2024-03-06T07:32:18.369743+0000","last_unstale":"2024-03-06T07:32:20.363572+0000","last_undegraded":"2024-03-06T07:32:20.363572+0000","last_fullsized":"2024-03-06T07:32:20.363572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:33.743263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.354903+0000","last_change":"2024-03-06T07:32:17.338528+0000","last_active":"2024-03-06T07:32:19.354903+0000","last_peered":"2024-03-06T07:32:19.354903+0000","last_clean":"2024-03-06T07:32:19.354903+0000","last_became_active":"2024-03-06T07:32:17.337976+0000","last_became_peered":"2024-03-06T07:32:17.337976+0000","last_unstale":"2024-03-06T07:32:19.354903+0000","last_undegraded":"2024-03-06T07:32:19.354903+0000","last_fullsized":"2024-03-06T07:32:19.354903+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:45.521027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.664581+0000","last_change":"2024-03-06T07:32:18.364888+0000","last_active":"2024-03-06T07:32:19.664581+0000","last_peered":"2024-03-06T07:32:19.664581+0000","last_clean":"2024-03-06T07:32:19.664581+0000","last_became_active":"2024-03-06T07:32:18.364560+0000","last_became_peered":"2024-03-06T07:32:18.364560+0000","last_unstale":"2024-03-06T07:32:19.664581+0000","last_undegraded":"2024-03-06T07:32:19.664581+0000","last_fullsized":"2024-03-06T07:32:19.664581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:35.696271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363558+0000","last_change":"2024-03-06T07:32:17.333102+0000","last_active":"2024-03-06T07:32:20.363558+0000","last_peered":"2024-03-06T07:32:20.363558+0000","last_clean":"2024-03-06T07:32:20.363558+0000","last_became_active":"2024-03-06T07:32:17.332802+0000","last_became_peered":"2024-03-06T07:32:17.332802+0000","last_unstale":"2024-03-06T07:32:20.363558+0000","last_undegraded":"2024-03-06T07:32:20.363558+0000","last_fullsized":"2024-03-06T07:32:20.363558+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:12.965022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363100+0000","last_change":"2024-03-06T07:32:18.365484+0000","last_active":"2024-03-06T07:32:20.363100+0000","last_peered":"2024-03-06T07:32:20.363100+0000","last_clean":"2024-03-06T07:32:20.363100+0000","last_became_active":"2024-03-06T07:32:18.365198+0000","last_became_peered":"2024-03-06T07:32:18.365198+0000","last_unstale":"2024-03-06T07:32:20.363100+0000","last_undegraded":"2024-03-06T07:32:20.363100+0000","last_fullsized":"2024-03-06T07:32:20.363100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:07.164298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362962+0000","last_change":"2024-03-06T07:32:17.340789+0000","last_active":"2024-03-06T07:32:20.362962+0000","last_peered":"2024-03-06T07:32:20.362962+0000","last_clean":"2024-03-06T07:32:20.362962+0000","last_became_active":"2024-03-06T07:32:17.340454+0000","last_became_peered":"2024-03-06T07:32:17.340454+0000","last_unstale":"2024-03-06T07:32:20.362962+0000","last_undegraded":"2024-03-06T07:32:20.362962+0000","last_fullsized":"2024-03-06T07:32:20.362962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:14.891876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.664532+0000","last_change":"2024-03-06T07:32:18.365772+0000","last_active":"2024-03-06T07:32:19.664532+0000","last_peered":"2024-03-06T07:32:19.664532+0000","last_clean":"2024-03-06T07:32:19.664532+0000","last_became_active":"2024-03-06T07:32:18.365372+0000","last_became_peered":"2024-03-06T07:32:18.365372+0000","last_unstale":"2024-03-06T07:32:19.664532+0000","last_undegraded":"2024-03-06T07:32:19.664532+0000","last_fullsized":"2024-03-06T07:32:19.664532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:10.667854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.354189+0000","last_change":"2024-03-06T07:32:17.333757+0000","last_active":"2024-03-06T07:32:19.354189+0000","last_peered":"2024-03-06T07:32:19.354189+0000","last_clean":"2024-03-06T07:32:19.354189+0000","last_became_active":"2024-03-06T07:32:17.333402+0000","last_became_peered":"2024-03-06T07:32:17.333402+0000","last_unstale":"2024-03-06T07:32:19.354189+0000","last_undegraded":"2024-03-06T07:32:19.354189+0000","last_fullsized":"2024-03-06T07:32:19.354189+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:41.920380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355874+0000","last_change":"2024-03-06T07:32:18.359196+0000","last_active":"2024-03-06T07:32:19.355874+0000","last_peered":"2024-03-06T07:32:19.355874+0000","last_clean":"2024-03-06T07:32:19.355874+0000","last_became_active":"2024-03-06T07:32:18.358836+0000","last_became_peered":"2024-03-06T07:32:18.358836+0000","last_unstale":"2024-03-06T07:32:19.355874+0000","last_undegraded":"2024-03-06T07:32:19.355874+0000","last_fullsized":"2024-03-06T07:32:19.355874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:51.584539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.353611+0000","last_change":"2024-03-06T07:32:17.342979+0000","last_active":"2024-03-06T07:32:19.353611+0000","last_peered":"2024-03-06T07:32:19.353611+0000","last_clean":"2024-03-06T07:32:19.353611+0000","last_became_active":"2024-03-06T07:32:17.342679+0000","last_became_peered":"2024-03-06T07:32:17.342679+0000","last_unstale":"2024-03-06T07:32:19.353611+0000","last_undegraded":"2024-03-06T07:32:19.353611+0000","last_fullsized":"2024-03-06T07:32:19.353611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:12.068932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363515+0000","last_change":"2024-03-06T07:32:18.375039+0000","last_active":"2024-03-06T07:32:20.363515+0000","last_peered":"2024-03-06T07:32:20.363515+0000","last_clean":"2024-03-06T07:32:20.363515+0000","last_became_active":"2024-03-06T07:32:18.371961+0000","last_became_peered":"2024-03-06T07:32:18.371961+0000","last_unstale":"2024-03-06T07:32:20.363515+0000","last_undegraded":"2024-03-06T07:32:20.363515+0000","last_fullsized":"2024-03-06T07:32:20.363515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:26.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.353792+0000","last_change":"2024-03-06T07:32:17.331176+0000","last_active":"2024-03-06T07:32:19.353792+0000","last_peered":"2024-03-06T07:32:19.353792+0000","last_clean":"2024-03-06T07:32:19.353792+0000","last_became_active":"2024-03-06T07:32:17.328312+0000","last_became_peered":"2024-03-06T07:32:17.328312+0000","last_unstale":"2024-03-06T07:32:19.353792+0000","last_undegraded":"2024-03-06T07:32:19.353792+0000","last_fullsized":"2024-03-06T07:32:19.353792+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:32.210436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363768+0000","last_change":"2024-03-06T07:32:18.368570+0000","last_active":"2024-03-06T07:32:20.363768+0000","last_peered":"2024-03-06T07:32:20.363768+0000","last_clean":"2024-03-06T07:32:20.363768+0000","last_became_active":"2024-03-06T07:32:18.367023+0000","last_became_peered":"2024-03-06T07:32:18.367023+0000","last_unstale":"2024-03-06T07:32:20.363768+0000","last_undegraded":"2024-03-06T07:32:20.363768+0000","last_fullsized":"2024-03-06T07:32:20.363768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:01.334957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.364327+0000","last_change":"2024-03-06T07:32:17.335023+0000","last_active":"2024-03-06T07:32:20.364327+0000","last_peered":"2024-03-06T07:32:20.364327+0000","last_clean":"2024-03-06T07:32:20.364327+0000","last_became_active":"2024-03-06T07:32:17.334651+0000","last_became_peered":"2024-03-06T07:32:17.334651+0000","last_unstale":"2024-03-06T07:32:20.364327+0000","last_undegraded":"2024-03-06T07:32:20.364327+0000","last_fullsized":"2024-03-06T07:32:20.364327+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:03.187760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.356326+0000","last_change":"2024-03-06T07:32:18.374627+0000","last_active":"2024-03-06T07:32:19.356326+0000","last_peered":"2024-03-06T07:32:19.356326+0000","last_clean":"2024-03-06T07:32:19.356326+0000","last_became_active":"2024-03-06T07:32:18.374340+0000","last_became_peered":"2024-03-06T07:32:18.374340+0000","last_unstale":"2024-03-06T07:32:19.356326+0000","last_undegraded":"2024-03-06T07:32:19.356326+0000","last_fullsized":"2024-03-06T07:32:19.356326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:19.899404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363946+0000","last_change":"2024-03-06T07:32:17.342916+0000","last_active":"2024-03-06T07:32:20.363946+0000","last_peered":"2024-03-06T07:32:20.363946+0000","last_clean":"2024-03-06T07:32:20.363946+0000","last_became_active":"2024-03-06T07:32:17.340684+0000","last_became_peered":"2024-03-06T07:32:17.340684+0000","last_unstale":"2024-03-06T07:32:20.363946+0000","last_undegraded":"2024-03-06T07:32:20.363946+0000","last_fullsized":"2024-03-06T07:32:20.363946+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:03.374987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.664712+0000","last_change":"2024-03-06T07:32:18.362738+0000","last_active":"2024-03-06T07:32:19.664712+0000","last_peered":"2024-03-06T07:32:19.664712+0000","last_clean":"2024-03-06T07:32:19.664712+0000","last_became_active":"2024-03-06T07:32:18.362445+0000","last_became_peered":"2024-03-06T07:32:18.362445+0000","last_unstale":"2024-03-06T07:32:19.664712+0000","last_undegraded":"2024-03-06T07:32:19.664712+0000","last_fullsized":"2024-03-06T07:32:19.664712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:16.997454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932309+0000","last_change":"2024-03-06T07:32:17.341757+0000","last_active":"2024-03-06T07:32:19.932309+0000","last_peered":"2024-03-06T07:32:19.932309+0000","last_clean":"2024-03-06T07:32:19.932309+0000","last_became_active":"2024-03-06T07:32:17.341395+0000","last_became_peered":"2024-03-06T07:32:17.341395+0000","last_unstale":"2024-03-06T07:32:19.932309+0000","last_undegraded":"2024-03-06T07:32:19.932309+0000","last_fullsized":"2024-03-06T07:32:19.932309+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:25.224946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.932262+0000","last_change":"2024-03-06T07:32:18.362968+0000","last_active":"2024-03-06T07:32:19.932262+0000","last_peered":"2024-03-06T07:32:19.932262+0000","last_clean":"2024-03-06T07:32:19.932262+0000","last_became_active":"2024-03-06T07:32:18.362635+0000","last_became_peered":"2024-03-06T07:32:18.362635+0000","last_unstale":"2024-03-06T07:32:19.932262+0000","last_undegraded":"2024-03-06T07:32:19.932262+0000","last_fullsized":"2024-03-06T07:32:19.932262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:19.899404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.355301+0000","last_change":"2024-03-06T07:32:17.341723+0000","last_active":"2024-03-06T07:32:19.355301+0000","last_peered":"2024-03-06T07:32:19.355301+0000","last_clean":"2024-03-06T07:32:19.355301+0000","last_became_active":"2024-03-06T07:32:17.341402+0000","last_became_peered":"2024-03-06T07:32:17.341402+0000","last_unstale":"2024-03-06T07:32:19.355301+0000","last_undegraded":"2024-03-06T07:32:19.355301+0000","last_fullsized":"2024-03-06T07:32:19.355301+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:29.001103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.361634+0000","last_change":"2024-03-06T07:32:18.368340+0000","last_active":"2024-03-06T07:32:20.361634+0000","last_peered":"2024-03-06T07:32:20.361634+0000","last_clean":"2024-03-06T07:32:20.361634+0000","last_became_active":"2024-03-06T07:32:18.366953+0000","last_became_peered":"2024-03-06T07:32:18.366953+0000","last_unstale":"2024-03-06T07:32:20.361634+0000","last_undegraded":"2024-03-06T07:32:20.361634+0000","last_fullsized":"2024-03-06T07:32:20.361634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:34.432657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363306+0000","last_change":"2024-03-06T07:32:17.338127+0000","last_active":"2024-03-06T07:32:20.363306+0000","last_peered":"2024-03-06T07:32:20.363306+0000","last_clean":"2024-03-06T07:32:20.363306+0000","last_became_active":"2024-03-06T07:32:17.337773+0000","last_became_peered":"2024-03-06T07:32:17.337773+0000","last_unstale":"2024-03-06T07:32:20.363306+0000","last_undegraded":"2024-03-06T07:32:20.363306+0000","last_fullsized":"2024-03-06T07:32:20.363306+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:32.480157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.362999+0000","last_change":"2024-03-06T07:32:18.370414+0000","last_active":"2024-03-06T07:32:20.362999+0000","last_peered":"2024-03-06T07:32:20.362999+0000","last_clean":"2024-03-06T07:32:20.362999+0000","last_became_active":"2024-03-06T07:32:18.369834+0000","last_became_peered":"2024-03-06T07:32:18.369834+0000","last_unstale":"2024-03-06T07:32:20.362999+0000","last_undegraded":"2024-03-06T07:32:20.362999+0000","last_fullsized":"2024-03-06T07:32:20.362999+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:51.584539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.354593+0000","last_change":"2024-03-06T07:32:17.347045+0000","last_active":"2024-03-06T07:32:19.354593+0000","last_peered":"2024-03-06T07:32:19.354593+0000","last_clean":"2024-03-06T07:32:19.354593+0000","last_became_active":"2024-03-06T07:32:17.346721+0000","last_became_peered":"2024-03-06T07:32:17.346721+0000","last_unstale":"2024-03-06T07:32:19.354593+0000","last_undegraded":"2024-03-06T07:32:19.354593+0000","last_fullsized":"2024-03-06T07:32:19.354593+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:32.480157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:19.353031+0000","last_change":"2024-03-06T07:32:18.365839+0000","last_active":"2024-03-06T07:32:19.353031+0000","last_peered":"2024-03-06T07:32:19.353031+0000","last_clean":"2024-03-06T07:32:19.353031+0000","last_became_active":"2024-03-06T07:32:18.361297+0000","last_became_peered":"2024-03-06T07:32:18.361297+0000","last_unstale":"2024-03-06T07:32:19.353031+0000","last_undegraded":"2024-03-06T07:32:19.353031+0000","last_fullsized":"2024-03-06T07:32:19.353031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:17.302402+0000","last_clean_scrub_stamp":"2024-03-06T07:32:17.302402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:14.340692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.363609+0000","last_change":"2024-03-06T07:32:17.339426+0000","last_active":"2024-03-06T07:32:20.363609+0000","last_peered":"2024-03-06T07:32:20.363609+0000","last_clean":"2024-03-06T07:32:20.363609+0000","last_became_active":"2024-03-06T07:32:17.339135+0000","last_became_peered":"2024-03-06T07:32:17.339135+0000","last_unstale":"2024-03-06T07:32:20.363609+0000","last_undegraded":"2024-03-06T07:32:20.363609+0000","last_fullsized":"2024-03-06T07:32:20.363609+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:16.287594+0000","last_clean_scrub_stamp":"2024-03-06T07:32:16.287594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:49.193701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":279,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:20.361832+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:32:20.361832+0000","last_peered":"2024-03-06T07:32:20.361832+0000","last_clean":"2024-03-06T07:32:20.361832+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:32:20.361832+0000","last_undegraded":"2024-03-06T07:32:20.361832+0000","last_fullsized":"2024-03-06T07:32:20.361832+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1581056,"data_stored":1007473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":417729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":418669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":417035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":995328,"data_stored":418647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":987136,"data_stored":417617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":995328,"data_stored":417229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1581056,"data_stored":1007437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:28.246 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:32:28.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:32:28.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.507+0000 7fed8fe00640 1 Processor -- start 2024-03-06T07:32:28.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.507+0000 7fed8fe00640 1 -- start start 2024-03-06T07:32:28.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.509+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 0x7fed880d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.509+0000 7fed8fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fed880b5490 0x7fed880c8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.509+0000 7fed8fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed880cecd0 con 0x7fed88161b50 2024-03-06T07:32:28.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.509+0000 7fed8fe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed880cef40 con 0x7fed880b5490 2024-03-06T07:32:28.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.509+0000 7fed8fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed880cfc80 con 0x7fed880d4a00 2024-03-06T07:32:28.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.509+0000 7fed8d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fed88161b50 0x7fed880d05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35750/0 (socket says 172.21.15.114:35750) 2024-03-06T07:32:28.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.509+0000 7fed8d400640 1 -- 172.21.15.114:0/3166302378 learned_addr learned my addr 172.21.15.114:0/3166302378 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:28.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed8ca00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 0x7fed880d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed8de00640 1 --2- 172.21.15.114:0/3166302378 >> v2:172.21.15.177:3300/0 conn(0x7fed880b5490 0x7fed880c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904564708 0 0) 0x7fed880cecd0 con 0x7fed88161b50 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed70003520 con 0x7fed88161b50 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162414574 0 0) 0x7fed70003520 con 0x7fed88161b50 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 msgr2=0x7fed880d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 0x7fed880d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 >> v2:172.21.15.177:3300/0 conn(0x7fed880b5490 msgr2=0x7fed880c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 --2- 172.21.15.114:0/3166302378 >> v2:172.21.15.177:3300/0 conn(0x7fed880b5490 0x7fed880c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed880c7b60 con 0x7fed88161b50 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fed7c001670 con 0x7fed88161b50 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 >> v1:172.21.15.114:6789/0 conn(0x7fed88161b50 legacy=0x7fed880d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed700039b0 0x7fed70003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.510+0000 7fed87e00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 0x7fed70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed87e00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880cef40 con 0x7fed70004d40 2024-03-06T07:32:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880cfc80 con 0x7fed700039b0 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7000b9c0 con 0x7fed880d4a00 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8de00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8d400640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed700039b0 0x7fed70003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8ca00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 0x7fed70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8de00640 1 -- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 msgr2=0x7fed70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8de00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880d4a00 0x7fed70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8de00640 1 -- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed700039b0 msgr2=0x7fed70003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8de00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed700039b0 0x7fed70003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.511+0000 7fed8de00640 1 -- 172.21.15.114:0/3166302378 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70003520 con 0x7fed70004d40 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.512+0000 7fed8de00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed700051c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fed80009fc0 tx=0x7fed70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc30d1776157f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80039070 con 0x7fed70004d40 2024-03-06T07:32:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8002cc60 con 0x7fed70004d40 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed87e00640 1 -- 172.21.15.114:0/3166302378 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80009c10 con 0x7fed70004d40 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed8fe00640 1 -- 172.21.15.114:0/3166302378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 msgr2=0x7fed700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed8fe00640 1 --2- 172.21.15.114:0/3166302378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed700051c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fed80009fc0 tx=0x7fed70005830 comp rx=0 tx=0).stop 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed8de00640 1 -- 172.21.15.114:0/3166302378 reap_dead start 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed8fe00640 1 -- 172.21.15.114:0/3166302378 shutdown_connections 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed8fe00640 1 -- 172.21.15.114:0/3166302378 >> 172.21.15.114:0/3166302378 conn(0x7fed880d3e70 msgr2=0x7fed880d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed8fe00640 1 -- 172.21.15.114:0/3166302378 shutdown_connections 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.513+0000 7fed8fe00640 1 -- 172.21.15.114:0/3166302378 wait complete. 2024-03-06T07:32:28.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.515+0000 7fed8fe00640 1 Processor -- start 2024-03-06T07:32:28.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.515+0000 7fed8fe00640 1 -- start start 2024-03-06T07:32:28.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed881f4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed880cdb60 0x7fed880cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880b8580 0x7fed880b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880d1040 con 0x7fed70004d40 2024-03-06T07:32:28.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880d0ec0 con 0x7fed880cdb60 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed880d11c0 con 0x7fed880b8580 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed881f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed881f4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58652/0 (socket says 172.21.15.114:58652) 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.516+0000 7fed8d400640 1 -- 172.21.15.114:0/2065788989 learned_addr learned my addr 172.21.15.114:0/2065788989 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8d400640 1 -- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880b8580 msgr2=0x7fed880b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8de00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880b8580 0x7fed880b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8d400640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880b8580 0x7fed880b8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8d400640 1 -- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed880cdb60 msgr2=0x7fed880cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8d400640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed880cdb60 0x7fed880cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8d400640 1 -- 172.21.15.114:0/2065788989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed881f4dc0 con 0x7fed70004d40 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8de00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880b8580 0x7fed880b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed8d400640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed881f4880 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00e730 tx=0x7fed7c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed85600640 1 -- 172.21.15.114:0/2065788989 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00ee40 con 0x7fed70004d40 2024-03-06T07:32:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed85600640 1 -- 172.21.15.114:0/2065788989 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c018070 con 0x7fed70004d40 2024-03-06T07:32:28.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.517+0000 7fed85600640 1 -- 172.21.15.114:0/2065788989 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00ca90 con 0x7fed70004d40 2024-03-06T07:32:28.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.518+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed880b6be0 con 0x7fed70004d40 2024-03-06T07:32:28.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.518+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed880b7180 con 0x7fed70004d40 2024-03-06T07:32:28.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.519+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8804f330 con 0x7fed70004d40 2024-03-06T07:32:28.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.520+0000 7fed85600640 1 -- 172.21.15.114:0/2065788989 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed7c00cc30 con 0x7fed70004d40 2024-03-06T07:32:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.521+0000 7fed85600640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fed6806b4d0 0x7fed6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.521+0000 7fed85600640 1 -- 172.21.15.114:0/2065788989 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed7c08a160 con 0x7fed70004d40 2024-03-06T07:32:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.521+0000 7fed8ca00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fed6806b4d0 0x7fed6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.522+0000 7fed8ca00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fed6806b4d0 0x7fed6806d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fed880b5460 tx=0x7fed74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.529+0000 7fed85600640 1 -- 172.21.15.114:0/2065788989 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed881f4dc0 con 0x7fed70004d40 2024-03-06T07:32:28.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.956+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fed880c7170 con 0x7fed70004d40 2024-03-06T07:32:28.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.961+0000 7fed85600640 1 -- 172.21.15.114:0/2065788989 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fed7c058f40 con 0x7fed70004d40 2024-03-06T07:32:28.963 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:32:28.963 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:32:28.963 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:32:28.963 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:32:28.963 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:28.964 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.963+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fed6806b4d0 msgr2=0x7fed6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.963+0000 7fed8fe00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fed6806b4d0 0x7fed6806d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fed880b5460 tx=0x7fed74008040 comp rx=0 tx=0).stop 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 msgr2=0x7fed881f4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed881f4880 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00e730 tx=0x7fed7c00ebf0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 shutdown_connections 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed880b8580 0x7fed880b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed880cdb60 0x7fed880cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fed6806b4d0 0x7fed6806d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 --2- 172.21.15.114:0/2065788989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed70004d40 0x7fed881f4880 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 >> 172.21.15.114:0/2065788989 conn(0x7fed880d3e70 msgr2=0x7fed880d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 shutdown_connections 2024-03-06T07:32:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:28.964+0000 7fed8fe00640 1 -- 172.21.15.114:0/2065788989 wait complete. 2024-03-06T07:32:28.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.194+0000 7f0284c00640 1 Processor -- start 2024-03-06T07:32:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.194+0000 7f0284c00640 1 -- start start 2024-03-06T07:32:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.195+0000 7f0284c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f028015a940 0x7f02800b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.195+0000 7f0284c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f02800cf370 0x7f028006d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.195+0000 7f0284c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02800b2b30 con 0x7f02800cc2a0 2024-03-06T07:32:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.195+0000 7f0284c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02800b28e0 con 0x7f02800cf370 2024-03-06T07:32:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.195+0000 7f0284c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02800b2e10 con 0x7f028015a940 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.196+0000 7f027ea00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f02800cf370 0x7f028006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.196+0000 7f027fe00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f02800cc2a0 0x7f02800cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:33530/0 (socket says 172.21.15.114:33530) 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.196+0000 7f027fe00640 1 -- 172.21.15.114:0/1410148782 learned_addr learned my addr 172.21.15.114:0/1410148782 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.196+0000 7f027f400640 1 --2- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f028015a940 0x7f02800b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.197+0000 7f027ea00640 1 -- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f028015a940 msgr2=0x7f02800b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.197+0000 7f027ea00640 1 --2- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f028015a940 0x7f02800b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.197+0000 7f027ea00640 1 -- 172.21.15.114:0/1410148782 >> v1:172.21.15.177:6789/0 conn(0x7f02800cc2a0 legacy=0x7f02800cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.197+0000 7f027ea00640 1 -- 172.21.15.114:0/1410148782 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028006ca50 con 0x7f02800cf370 2024-03-06T07:32:29.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.198+0000 7f027ea00640 1 --2- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 0x7f028006d250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f026c0097a0 tx=0x7f026c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a257dcf5e383b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.199+0000 7f027e000640 1 -- 172.21.15.114:0/1410148782 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c035020 con 0x7f02800cf370 2024-03-06T07:32:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.199+0000 7f027e000640 1 -- 172.21.15.114:0/1410148782 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c031810 con 0x7f02800cf370 2024-03-06T07:32:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.199+0000 7f027e000640 1 -- 172.21.15.114:0/1410148782 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c031be0 con 0x7f02800cf370 2024-03-06T07:32:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.199+0000 7f0284c00640 1 -- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 msgr2=0x7f028006d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.199+0000 7f0284c00640 1 --2- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 0x7f028006d250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f026c0097a0 tx=0x7f026c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.200+0000 7f0284c00640 1 -- 172.21.15.114:0/1410148782 shutdown_connections 2024-03-06T07:32:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.200+0000 7f0284c00640 1 --2- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 0x7f028006d250 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.200+0000 7f0284c00640 1 --2- 172.21.15.114:0/1410148782 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f028015a940 0x7f02800b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.200+0000 7f0284c00640 1 -- 172.21.15.114:0/1410148782 >> 172.21.15.114:0/1410148782 conn(0x7f028006afc0 msgr2=0x7f028006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.200+0000 7f0284c00640 1 -- 172.21.15.114:0/1410148782 shutdown_connections 2024-03-06T07:32:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.200+0000 7f0284c00640 1 -- 172.21.15.114:0/1410148782 wait complete. 2024-03-06T07:32:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.202+0000 7f0284c00640 1 Processor -- start 2024-03-06T07:32:29.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.202+0000 7f0284c00640 1 -- start start 2024-03-06T07:32:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.203+0000 7f0284c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02800cc2a0 0x7f028014c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f0284c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 0x7f02800c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f0284c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f028015a940 0x7f028015e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f0284c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02800b4b20 con 0x7f028015a940 2024-03-06T07:32:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f0284c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02800b30f0 con 0x7f02800cf370 2024-03-06T07:32:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f0284c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028006d790 con 0x7f02800cc2a0 2024-03-06T07:32:29.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f028015a940 0x7f028015e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f028015a940 0x7f028015e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58664/0 (socket says 172.21.15.114:58664) 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 -- 172.21.15.114:0/3995910415 learned_addr learned my addr 172.21.15.114:0/3995910415 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027ea00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 0x7f02800c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027f400640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02800cc2a0 0x7f028014c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 -- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02800cc2a0 msgr2=0x7f028014c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02800cc2a0 0x7f028014c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 -- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 msgr2=0x7f02800c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 0x7f02800c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027fe00640 1 -- 172.21.15.114:0/3995910415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02800b10a0 con 0x7f028015a940 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.204+0000 7f027f400640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02800cc2a0 0x7f028014c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:29.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.205+0000 7f027fe00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f028015a940 0x7f028015e6d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f027800ca80 tx=0x7f027800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.205+0000 7f0277400640 1 -- 172.21.15.114:0/3995910415 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278017070 con 0x7f028015a940 2024-03-06T07:32:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.205+0000 7f0277400640 1 -- 172.21.15.114:0/3995910415 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027800ac30 con 0x7f028015a940 2024-03-06T07:32:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.205+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02800b1380 con 0x7f028015a940 2024-03-06T07:32:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.205+0000 7f0277400640 1 -- 172.21.15.114:0/3995910415 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278012400 con 0x7f028015a940 2024-03-06T07:32:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.205+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02800b1940 con 0x7f028015a940 2024-03-06T07:32:29.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.206+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028004f330 con 0x7f028015a940 2024-03-06T07:32:29.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.207+0000 7f0277400640 1 -- 172.21.15.114:0/3995910415 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f027801a070 con 0x7f028015a940 2024-03-06T07:32:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.208+0000 7f0277400640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f026006b4d0 0x7f026006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.208+0000 7f0277400640 1 -- 172.21.15.114:0/3995910415 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0278089f00 con 0x7f028015a940 2024-03-06T07:32:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.208+0000 7f027f400640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f026006b4d0 0x7f026006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.209+0000 7f027f400640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f026006b4d0 0x7f026006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0270007ba0 tx=0x7f027000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.212+0000 7f0277400640 1 -- 172.21.15.114:0/3995910415 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0278058600 con 0x7f028015a940 2024-03-06T07:32:29.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.452+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02800b38e0 con 0x7f028015a940 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.453+0000 7f0277400640 1 -- 172.21.15.114:0/3995910415 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f027805c6a0 con 0x7f028015a940 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:17.782119+0000","modified":"2024-03-06T07:32:20.854485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2399499569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2399499569},{"type":"v1","addr":"172.21.15.114:6837","nonce":2399499569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 52 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.456+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f026006b4d0 msgr2=0x7f026006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.456+0000 7f0284c00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f026006b4d0 0x7f026006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0270007ba0 tx=0x7f027000d040 comp rx=0 tx=0).stop 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.456+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f028015a940 msgr2=0x7f028015e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.456+0000 7f0284c00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f028015a940 0x7f028015e6d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f027800ca80 tx=0x7f027800cf40 comp rx=0 tx=0).stop 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 shutdown_connections 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f026006b4d0 0x7f026006d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f028015a940 0x7f028015e6d0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02800cf370 0x7f02800c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 --2- 172.21.15.114:0/3995910415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02800cc2a0 0x7f028014c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 >> 172.21.15.114:0/3995910415 conn(0x7f028006afc0 msgr2=0x7f0280157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 shutdown_connections 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:29.457+0000 7f0284c00640 1 -- 172.21.15.114:0/3995910415 wait complete. 2024-03-06T07:32:29.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:17.782119+0000', 'modified': '2024-03-06T07:32:20.854485+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6756}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6756': {'gid': 6756, 'name': 'c', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/2399499569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 2399499569}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 2399499569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6756, 'qdb_cluster': [6756]} 2024-03-06T07:32:29.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:32:29.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:32:30.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.134+0000 7f7360c00640 1 Processor -- start 2024-03-06T07:32:30.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.135+0000 7f7360c00640 1 -- start start 2024-03-06T07:32:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f7360c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f735c0d1270 0x7f735c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f7360c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d2340 0x7f735c0d27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f7360c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c0ce890 con 0x7f735c15e090 2024-03-06T07:32:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f7360c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c15e470 con 0x7f735c0d1270 2024-03-06T07:32:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f7360c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0d2180 con 0x7f735c0d2340 2024-03-06T07:32:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f735aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f735c0d1270 0x7f735c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f735aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f735c0d1270 0x7f735c0cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58678/0 (socket says 172.21.15.114:58678) 2024-03-06T07:32:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.137+0000 7f735aa00640 1 -- 172.21.15.114:0/2624603402 learned_addr learned my addr 172.21.15.114:0/2624603402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.138+0000 7f735aa00640 1 -- 172.21.15.114:0/2624603402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d2340 msgr2=0x7f735c0d27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.138+0000 7f735aa00640 1 --2- 172.21.15.114:0/2624603402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d2340 0x7f735c0d27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.138+0000 7f735aa00640 1 -- 172.21.15.114:0/2624603402 >> v1:172.21.15.177:6789/0 conn(0x7f735c15e090 legacy=0x7f735c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.138+0000 7f735aa00640 1 -- 172.21.15.114:0/2624603402 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0ceb30 con 0x7f735c0d1270 2024-03-06T07:32:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.139+0000 7f735aa00640 1 --2- 172.21.15.114:0/2624603402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d1270 0x7f735c0cf8f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7348009870 tx=0x7f734802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a5eec15527230a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.139+0000 7f735a000640 1 -- 172.21.15.114:0/2624603402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348035020 con 0x7f735c0d1270 2024-03-06T07:32:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.139+0000 7f735a000640 1 -- 172.21.15.114:0/2624603402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7348031640 con 0x7f735c0d1270 2024-03-06T07:32:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.139+0000 7f735a000640 1 -- 172.21.15.114:0/2624603402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734802d040 con 0x7f735c0d1270 2024-03-06T07:32:30.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 -- 172.21.15.114:0/2624603402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d1270 msgr2=0x7f735c0cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 --2- 172.21.15.114:0/2624603402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d1270 0x7f735c0cf8f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7348009870 tx=0x7f734802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 -- 172.21.15.114:0/2624603402 shutdown_connections 2024-03-06T07:32:30.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 --2- 172.21.15.114:0/2624603402 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d2340 0x7f735c0d27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 --2- 172.21.15.114:0/2624603402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d1270 0x7f735c0cf8f0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 -- 172.21.15.114:0/2624603402 >> 172.21.15.114:0/2624603402 conn(0x7f735c0d3d80 msgr2=0x7f735c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 -- 172.21.15.114:0/2624603402 shutdown_connections 2024-03-06T07:32:30.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.140+0000 7f7360c00640 1 -- 172.21.15.114:0/2624603402 wait complete. 2024-03-06T07:32:30.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.142+0000 7f7360c00640 1 Processor -- start 2024-03-06T07:32:30.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.142+0000 7f7360c00640 1 -- start start 2024-03-06T07:32:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.143+0000 7f7360c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d1270 0x7f735c0b7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.143+0000 7f7360c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d2340 0x7f735c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.143+0000 7f7360c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c15e090 0x7f735c15c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.143+0000 7f7360c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0cf0f0 con 0x7f735c0d2340 2024-03-06T07:32:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.143+0000 7f7360c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0cf460 con 0x7f735c15e090 2024-03-06T07:32:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.143+0000 7f7360c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0b36b0 con 0x7f735c0d1270 2024-03-06T07:32:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d2340 0x7f735c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d2340 0x7f735c0cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58682/0 (socket says 172.21.15.114:58682) 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 -- 172.21.15.114:0/4108464922 learned_addr learned my addr 172.21.15.114:0/4108464922 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735b400640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d1270 0x7f735c0b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735be00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c15e090 0x7f735c15c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 -- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d1270 msgr2=0x7f735c0b7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d1270 0x7f735c0b7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 -- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c15e090 msgr2=0x7f735c15c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c15e090 0x7f735c15c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 -- 172.21.15.114:0/4108464922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0b7740 con 0x7f735c0d2340 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735be00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c15e090 0x7f735c15c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.144+0000 7f735aa00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d2340 0x7f735c0cd6d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f734802b930 tx=0x7f73480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.145+0000 7f7353400640 1 -- 172.21.15.114:0/4108464922 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734803ac60 con 0x7f735c0d2340 2024-03-06T07:32:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.145+0000 7f7353400640 1 -- 172.21.15.114:0/4108464922 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734803ae20 con 0x7f735c0d2340 2024-03-06T07:32:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.145+0000 7f7353400640 1 -- 172.21.15.114:0/4108464922 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348044630 con 0x7f735c0d2340 2024-03-06T07:32:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.145+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c1f7f90 con 0x7f735c0d2340 2024-03-06T07:32:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.145+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c1f8550 con 0x7f735c0d2340 2024-03-06T07:32:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.147+0000 7f7353400640 1 -- 172.21.15.114:0/4108464922 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f734804b060 con 0x7f735c0d2340 2024-03-06T07:32:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.148+0000 7f7353400640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f733c06b520 0x7f733c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.148+0000 7f7353400640 1 -- 172.21.15.114:0/4108464922 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73480abf10 con 0x7f735c0d2340 2024-03-06T07:32:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.148+0000 7f735b400640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f733c06b520 0x7f733c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.149+0000 7f735b400640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f733c06b520 0x7f733c06d9e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f734c006fd0 tx=0x7f734c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.149+0000 7f7364fd0740 1 Processor -- start 2024-03-06T07:32:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.149+0000 7f7364fd0740 1 -- start start 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee62930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8fd70 0x56111ee92720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee62e70 0x56111ee92c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f7364fd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111edcc600 con 0x56111ee8f990 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111ed75930 con 0x56111ee8fd70 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x56111edc86a0 con 0x56111ee62e70 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee62930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee62930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58684/0 (socket says 172.21.15.114:58684) 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 -- 172.21.15.114:0/1316942683 learned_addr learned my addr 172.21.15.114:0/1316942683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 -- 172.21.15.114:0/1316942683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee62e70 msgr2=0x56111ee92c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 --2- 172.21.15.114:0/1316942683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee62e70 0x56111ee92c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 -- 172.21.15.114:0/1316942683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8fd70 msgr2=0x56111ee92720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 --2- 172.21.15.114:0/1316942683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8fd70 0x56111ee92720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.151+0000 7f735b400640 1 -- 172.21.15.114:0/1316942683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56111eae12d0 con 0x56111ee8f990 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.152+0000 7f735b400640 1 --2- 172.21.15.114:0/1316942683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee62930 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f734c00d830 tx=0x7f734c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856031fc4764bb32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.152+0000 7f7347400640 1 -- 172.21.15.114:0/1316942683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c024070 con 0x56111ee8f990 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.152+0000 7f7347400640 1 -- 172.21.15.114:0/1316942683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c0149b0 con 0x56111ee8f990 2024-03-06T07:32:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.152+0000 7f7347400640 1 -- 172.21.15.114:0/1316942683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c014da0 con 0x56111ee8f990 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 -- 172.21.15.114:0/1316942683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 msgr2=0x56111ee62930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1316942683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee62930 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f734c00d830 tx=0x7f734c010000 comp rx=0 tx=0).stop 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 -- 172.21.15.114:0/1316942683 shutdown_connections 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1316942683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee62e70 0x56111ee92c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1316942683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8fd70 0x56111ee92720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1316942683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee62930 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 -- 172.21.15.114:0/1316942683 >> 172.21.15.114:0/1316942683 conn(0x56111ee64ba0 msgr2=0x56111eead720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 -- 172.21.15.114:0/1316942683 shutdown_connections 2024-03-06T07:32:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.153+0000 7f7364fd0740 1 -- 172.21.15.114:0/1316942683 wait complete. 2024-03-06T07:32:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.154+0000 7f7364fd0740 1 -- start start 2024-03-06T07:32:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.154+0000 7f7364fd0740 1 Processor -- start 2024-03-06T07:32:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8f990 0x56111ee84590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee7de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111eea4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f7364fd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111ee67140 con 0x56111ee8fd70 2024-03-06T07:32:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111ed88d90 con 0x56111ee81a60 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x56111ee66530 con 0x56111ee8f990 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f7364fd0740 10 client.0 _ll_register_callbacks cb 0x56111edc4350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f735aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee7de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f735aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee7de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58686/0 (socket says 172.21.15.114:58686) 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f735aa00640 1 -- 172.21.15.114:0/1647065362 learned_addr learned my addr 172.21.15.114:0/1647065362 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f735b400640 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8f990 0x56111ee84590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.155+0000 7f735be00640 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111eea4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f735aa00640 1 -- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8f990 msgr2=0x56111ee84590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f735aa00640 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8f990 0x56111ee84590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f735aa00640 1 -- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 msgr2=0x56111eea4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f735aa00640 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111eea4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f735aa00640 1 -- 172.21.15.114:0/1647065362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56111ebab6b0 con 0x56111ee8fd70 2024-03-06T07:32:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f735aa00640 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee7de30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f734807a400 tx=0x7f734807aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6aaa86cc1879122 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f733be00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f733be00640 1 -- 172.21.15.114:0/1647065362 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348092020 con 0x56111ee8fd70 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f733be00640 1 -- 172.21.15.114:0/1647065362 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73480921c0 con 0x56111ee8fd70 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f734c0149b0 con 0x56111ee8fd70 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.156+0000 7f733be00640 1 -- 172.21.15.114:0/1647065362 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7348082e10 con 0x56111ee8fd70 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 10 client.6912 fetch_fsmap learned FSMap version 52 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x56111ee823b0 con 0x56111ee8fd70 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f733be00640 1 -- 172.21.15.114:0/1647065362 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c0149b0 con 0x56111ee8fd70 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f733be00640 1 -- 172.21.15.114:0/1647065362 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f73480aa080 con 0x56111ee8fd70 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 10 client.6912 fetch_fsmap finished waiting for FSMap version 52 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 10 client.6912 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 20 client.6912 populate_metadata read hostname 'smithi114' 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111ee94640 0x56111ee94a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 4 client.6912 mds_command: new command op to 6756 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 --> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56111edd9b40 con 0x56111ee94640 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.157+0000 7f735b400640 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111ee94640 0x56111ee94a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.158+0000 7f733be00640 10 client.6912 ms_handle_connect on v2:172.21.15.114:6835/2399499569 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.158+0000 7f735b400640 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111ee94640 0x56111ee94a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e28e14afb1ab83c1 server_cookie=a9a1a3523f4ee6e4 in_seq=0 out_seq=0 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f733be00640 1 -- 172.21.15.114:0/1647065362 <== mds.0 v2:172.21.15.114:6835/2399499569 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56111edd9b40 con 0x56111ee94640 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f733be00640 10 client.6912 handle_command_reply: tid=0 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 client.6912 shutdown 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 msgr2=0x56111ee7de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee7de30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f734807a400 tx=0x7f734807aec0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 shutdown_connections 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111ee94640 0x56111ee94a20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111eea4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee7de30 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f733be00640 0 client.6912 ms_handle_reset on v2:172.21.15.114:6835/2399499569 2024-03-06T07:32:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 --2- 172.21.15.114:0/1647065362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8f990 0x56111ee84590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 >> 172.21.15.114:0/1647065362 conn(0x56111ee62e70 msgr2=0x56111eea7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.159+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 shutdown_connections 2024-03-06T07:32:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.160+0000 7f7364fd0740 1 -- 172.21.15.114:0/1647065362 wait complete. 2024-03-06T07:32:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.160+0000 7f7364fd0740 20 client.6912 trim_cache size 0 max 16384 2024-03-06T07:32:30.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.178+0000 7f7364fd0740 1 Processor -- start 2024-03-06T07:32:30.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.178+0000 7f7364fd0740 1 -- start start 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee81a60 0x56111eea8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8f990 0x56111eea8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee779c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f7364fd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c15e7e0 con 0x56111ee8fd70 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1602b0 con 0x56111ee8f990 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1608b0 con 0x56111ee81a60 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee779c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58702/0 (socket says 172.21.15.114:58702) 2024-03-06T07:32:30.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735be00640 1 -- 172.21.15.114:0/4287788494 learned_addr learned my addr 172.21.15.114:0/4287788494 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:30.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735be00640 1 -- 172.21.15.114:0/4287788494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee81a60 msgr2=0x56111eea8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735be00640 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee81a60 0x56111eea8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735aa00640 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8f990 0x56111eea8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735be00640 1 -- 172.21.15.114:0/4287788494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8f990 msgr2=0x56111eea8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.179+0000 7f735be00640 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8f990 0x56111eea8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.180+0000 7f735be00640 1 -- 172.21.15.114:0/4287788494 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56111ee5e0e0 con 0x56111ee8fd70 2024-03-06T07:32:30.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.180+0000 7f735be00640 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee779c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7354008d20 tx=0x7f735400d700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb936729066990a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.180+0000 7f7346a00640 1 -- 172.21.15.114:0/4287788494 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400f820 con 0x56111ee8fd70 2024-03-06T07:32:30.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.180+0000 7f7346a00640 1 -- 172.21.15.114:0/4287788494 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735400f9e0 con 0x56111ee8fd70 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.180+0000 7f7346a00640 1 -- 172.21.15.114:0/4287788494 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400fdd0 con 0x56111ee8fd70 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 -- 172.21.15.114:0/4287788494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 msgr2=0x56111ee779c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee779c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f7354008d20 tx=0x7f735400d700 comp rx=0 tx=0).stop 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 -- 172.21.15.114:0/4287788494 shutdown_connections 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8fd70 0x56111ee779c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee8f990 0x56111eea8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 --2- 172.21.15.114:0/4287788494 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee81a60 0x56111eea8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 -- 172.21.15.114:0/4287788494 >> 172.21.15.114:0/4287788494 conn(0x56111ee62e70 msgr2=0x56111ee95490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 -- 172.21.15.114:0/4287788494 shutdown_connections 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 -- 172.21.15.114:0/4287788494 wait complete. 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.181+0000 7f7364fd0740 1 -- start start 2024-03-06T07:32:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.182+0000 7f7364fd0740 1 Processor -- start 2024-03-06T07:32:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.182+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111ef17c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.182+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee779c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f7364fd0740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8fd70 0x56111ee77f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f7364fd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111ee829c0 con 0x56111ee8f990 2024-03-06T07:32:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x56111ee67140 con 0x56111ee81a60 2024-03-06T07:32:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f7364fd0740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x56111eea8220 con 0x56111ee8fd70 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f7364fd0740 10 client.0 _ll_register_callbacks cb 0x7f73480091c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee779c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58718/0 (socket says 172.21.15.114:58718) 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 -- 172.21.15.114:0/384284615 learned_addr learned my addr 172.21.15.114:0/384284615 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735b400640 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111ef17c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 -- 172.21.15.114:0/384284615 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8fd70 msgr2=0x56111ee77f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8fd70 0x56111ee77f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 -- 172.21.15.114:0/384284615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 msgr2=0x56111ef17c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111ef17c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 -- 172.21.15.114:0/384284615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56111ef21220 con 0x56111ee8f990 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f735aa00640 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee779c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7348076b60 tx=0x7f7348045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ed380f484589c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.183+0000 7f733be00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:32:30.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f733be00640 1 -- 172.21.15.114:0/384284615 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734809e060 con 0x56111ee8f990 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f733be00640 1 -- 172.21.15.114:0/384284615 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73480921e0 con 0x56111ee8f990 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f735400f9e0 con 0x56111ee8f990 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f733be00640 1 -- 172.21.15.114:0/384284615 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7348092040 con 0x56111ee8f990 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 10 client.6918 fetch_fsmap learned FSMap version 52 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x56111ef21220 con 0x56111ee8f990 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f733be00640 1 -- 172.21.15.114:0/384284615 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735400f9e0 con 0x56111ee8f990 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f733be00640 1 -- 172.21.15.114:0/384284615 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f73480aa080 con 0x56111ee8f990 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 10 client.6918 fetch_fsmap finished waiting for FSMap version 52 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 10 client.6918 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 20 client.6918 populate_metadata read hostname 'smithi114' 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111eeaf430 0x56111eeaf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 4 client.6918 mds_command: new command op to 6756 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.184+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 --> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56111edd3e90 con 0x56111eeaf430 2024-03-06T07:32:30.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f735b400640 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111eeaf430 0x56111eeaf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f735b400640 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111eeaf430 0x56111eeaf810 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a04c61176bc58743 server_cookie=6667e08cc1f07f66 in_seq=0 out_seq=0 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f733be00640 10 client.6918 ms_handle_connect on v2:172.21.15.114:6835/2399499569 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f733be00640 1 -- 172.21.15.114:0/384284615 <== mds.0 v2:172.21.15.114:6835/2399499569 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x56111edd3e90 con 0x56111eeaf430 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f733be00640 10 client.6918 handle_command_reply: tid=0 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f7364fd0740 1 client.6918 shutdown 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 msgr2=0x56111ee779c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.185+0000 7f7364fd0740 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee779c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7348076b60 tx=0x7f7348045000 comp rx=0 tx=0).stop 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 shutdown_connections 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:6835/2399499569,v1:172.21.15.114:6837/2399499569] conn(0x56111eeaf430 0x56111eeaf810 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x56111ee8fd70 0x56111ee77f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56111ee8f990 0x56111ee779c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f733be00640 0 client.6918 ms_handle_reset on v2:172.21.15.114:6835/2399499569 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 --2- 172.21.15.114:0/384284615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56111ee81a60 0x56111ef17c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 >> 172.21.15.114:0/384284615 conn(0x56111ee62e70 msgr2=0x56111ee93490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 shutdown_connections 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 1 -- 172.21.15.114:0/384284615 wait complete. 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.186+0000 7f7364fd0740 20 client.6918 trim_cache size 0 max 16384 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 6756, 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:32:30.192 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 11.838190907, 2024-03-06T07:32:30.193 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 52, 2024-03-06T07:32:30.193 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 80, 2024-03-06T07:32:30.193 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 80, 2024-03-06T07:32:30.193 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 18.303797689 2024-03-06T07:32:30.193 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:32:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f733c06b520 msgr2=0x7f733c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f733c06b520 0x7f733c06d9e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f734c006fd0 tx=0x7f734c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d2340 msgr2=0x7f735c0cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d2340 0x7f735c0cd6d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f734802b930 tx=0x7f73480046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 shutdown_connections 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c15e090 0x7f735c15c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f733c06b520 0x7f733c06d9e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f735c0d2340 0x7f735c0cd6d0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 --2- 172.21.15.114:0/4108464922 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f735c0d1270 0x7f735c0b7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 >> 172.21.15.114:0/4108464922 conn(0x7f735c0d3d80 msgr2=0x7f735c0b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 shutdown_connections 2024-03-06T07:32:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.187+0000 7f7360c00640 1 -- 172.21.15.114:0/4108464922 wait complete. 2024-03-06T07:32:30.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:30.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.391+0000 7f14af200640 1 Processor -- start 2024-03-06T07:32:30.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.391+0000 7f14af200640 1 -- start start 2024-03-06T07:32:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.393+0000 7f14af200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f14b00d27a0 0x7f14b00b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.393+0000 7f14af200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b0161960 con 0x7f14b0161580 2024-03-06T07:32:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.393+0000 7f14af200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b00cf6f0 con 0x7f14b00d2360 2024-03-06T07:32:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.393+0000 7f14af200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b00cf9d0 con 0x7f14b00d27a0 2024-03-06T07:32:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.393+0000 7f14ade00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f14b0161580 0x7f14b00d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35752/0 (socket says 172.21.15.114:35752) 2024-03-06T07:32:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.393+0000 7f14ade00640 1 -- 172.21.15.114:0/4023967165 learned_addr learned my addr 172.21.15.114:0/4023967165 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777271181 0 0) 0x7f14b0161960 con 0x7f14b0161580 2024-03-06T07:32:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ae800640 1 --2- 172.21.15.114:0/4023967165 >> v2:172.21.15.177:3300/0 conn(0x7f14b00d27a0 0x7f14b00b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1494003520 con 0x7f14b0161580 2024-03-06T07:32:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592353626 0 0) 0x7f1494003520 con 0x7f14b0161580 2024-03-06T07:32:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 >> v2:172.21.15.177:3300/0 conn(0x7f14b00d27a0 msgr2=0x7f14b00b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 --2- 172.21.15.114:0/4023967165 >> v2:172.21.15.177:3300/0 conn(0x7f14b00d27a0 0x7f14b00b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 >> v1:172.21.15.177:6789/0 conn(0x7f14b00d2360 legacy=0x7f14b00b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b00cfcb0 con 0x7f14b0161580 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f14a0001670 con 0x7f14b0161580 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 >> v1:172.21.15.114:6789/0 conn(0x7f14b0161580 legacy=0x7f14b00d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1494003a00 0x7f1494003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1494004800 0x7f1494004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14940051c0 0x7f149400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b00cf9d0 con 0x7f1494004800 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149400c0c0 con 0x7f1494003a00 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.394+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149400c2c0 con 0x7f14940051c0 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14a5400640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1494004800 0x7f1494004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14ade00640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1494003a00 0x7f1494003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14ae800640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14940051c0 0x7f149400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14a5400640 1 -- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14940051c0 msgr2=0x7f149400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14a5400640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f14940051c0 0x7f149400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14a5400640 1 -- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1494003a00 msgr2=0x7f1494003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14a5400640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1494003a00 0x7f1494003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14a5400640 1 -- 172.21.15.114:0/4023967165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1494003520 con 0x7f1494004800 2024-03-06T07:32:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.395+0000 7f14ae800640 1 -- 172.21.15.114:0/4023967165 reap_dead start 2024-03-06T07:32:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.396+0000 7f14a5400640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1494004800 0x7f1494004c80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1498001750 tx=0x7f1494005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c3b902a0406721 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.396+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498001370 con 0x7f1494004800 2024-03-06T07:32:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14980052d0 con 0x7f1494004800 2024-03-06T07:32:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14ad400640 1 -- 172.21.15.114:0/4023967165 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498003830 con 0x7f1494004800 2024-03-06T07:32:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14af200640 1 -- 172.21.15.114:0/4023967165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1494004800 msgr2=0x7f1494004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14af200640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1494004800 0x7f1494004c80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1498001750 tx=0x7f1494005830 comp rx=0 tx=0).stop 2024-03-06T07:32:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14af200640 1 -- 172.21.15.114:0/4023967165 shutdown_connections 2024-03-06T07:32:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14af200640 1 --2- 172.21.15.114:0/4023967165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1494004800 0x7f1494004c80 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14af200640 1 -- 172.21.15.114:0/4023967165 >> 172.21.15.114:0/4023967165 conn(0x7f14b00d3d80 msgr2=0x7f14b00d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14af200640 1 -- 172.21.15.114:0/4023967165 shutdown_connections 2024-03-06T07:32:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.397+0000 7f14af200640 1 -- 172.21.15.114:0/4023967165 wait complete. 2024-03-06T07:32:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.399+0000 7f14af200640 1 Processor -- start 2024-03-06T07:32:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.399+0000 7f14af200640 1 -- start start 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14af200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1494004800 0x7f14b00b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14af200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14b015b2a0 0x7f14b00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14af200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b015ca00 0x7f14b0151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14af200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b00d2bf0 con 0x7f14b015ca00 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14af200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b00b39a0 con 0x7f14b015b2a0 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14af200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b00cf3a0 con 0x7f1494004800 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14ae800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b015ca00 0x7f14b0151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14ae800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b015ca00 0x7f14b0151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58732/0 (socket says 172.21.15.114:58732) 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.400+0000 7f14ae800640 1 -- 172.21.15.114:0/262796773 learned_addr learned my addr 172.21.15.114:0/262796773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14a5400640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14b015b2a0 0x7f14b00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14ade00640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1494004800 0x7f14b00b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14ae800640 1 -- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1494004800 msgr2=0x7f14b00b73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14ae800640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1494004800 0x7f14b00b73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14ae800640 1 -- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14b015b2a0 msgr2=0x7f14b00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14ae800640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14b015b2a0 0x7f14b00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14ae800640 1 -- 172.21.15.114:0/262796773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b00cd000 con 0x7f14b015ca00 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14ae800640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b015ca00 0x7f14b0151df0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f14b008cb60 tx=0x7f14a800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.401+0000 7f14a6a00640 1 -- 172.21.15.114:0/262796773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8016620 con 0x7f14b015ca00 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.402+0000 7f14a6a00640 1 -- 172.21.15.114:0/262796773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a801b070 con 0x7f14b015ca00 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.402+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b00cd2e0 con 0x7f14b015ca00 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.402+0000 7f14a6a00640 1 -- 172.21.15.114:0/262796773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8016a20 con 0x7f14b015ca00 2024-03-06T07:32:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.402+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b00ccd20 con 0x7f14b015ca00 2024-03-06T07:32:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.403+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b004f330 con 0x7f14b015ca00 2024-03-06T07:32:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.404+0000 7f14a6a00640 1 -- 172.21.15.114:0/262796773 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14a8016bc0 con 0x7f14b015ca00 2024-03-06T07:32:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.405+0000 7f14a6a00640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f148c06b500 0x7f148c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.405+0000 7f14a6a00640 1 -- 172.21.15.114:0/262796773 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14a808b140 con 0x7f14b015ca00 2024-03-06T07:32:30.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.405+0000 7f14ade00640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f148c06b500 0x7f148c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.406+0000 7f14ade00640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f148c06b500 0x7f148c06d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f14a0001050 tx=0x7f14a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.409+0000 7f14a6a00640 1 -- 172.21.15.114:0/262796773 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14a8025020 con 0x7f14b015ca00 2024-03-06T07:32:30.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.652+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f14b00d3b60 con 0x7f14b015ca00 2024-03-06T07:32:30.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.652+0000 7f14a6a00640 1 -- 172.21.15.114:0/262796773 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) v1 ==== 53+0+73 (secure 0 0 0) 0x7f14a805a050 con 0x7f14b015ca00 2024-03-06T07:32:30.654 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f148c06b500 msgr2=0x7f148c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f148c06b500 0x7f148c06d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f14a0001050 tx=0x7f14a0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b015ca00 msgr2=0x7f14b0151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b015ca00 0x7f14b0151df0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f14b008cb60 tx=0x7f14a800f520 comp rx=0 tx=0).stop 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 shutdown_connections 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f148c06b500 0x7f148c06d9c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14b015ca00 0x7f14b0151df0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14b015b2a0 0x7f14b00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 --2- 172.21.15.114:0/262796773 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1494004800 0x7f14b00b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 >> 172.21.15.114:0/262796773 conn(0x7f14b00d3d80 msgr2=0x7f14b006d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 shutdown_connections 2024-03-06T07:32:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:30.655+0000 7f14af200640 1 -- 172.21.15.114:0/262796773 wait complete. 2024-03-06T07:32:30.676 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:32:30.676 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:30.676 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:32:30.677 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:30.677 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:30.677 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:32:30.677 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:32:30.677 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:30.677 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:30.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.677 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:32:30.726 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:30.726 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:30.727 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42305sec preferred_lft 42305sec 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:30.728 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:30.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.730 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:30.806 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:32:30.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.811 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:32:30.861 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:32:30.861 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:30.861 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:32:30.861 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:32:30.861 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:30.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.862 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:32:30.862 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:32:30.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:32:30.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:32:30.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:32:30.863 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:32:31.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.048 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:32:31.096 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:31.096 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:31.097 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:31.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:32:31.182 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:31.182 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:31.232 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:31.232 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:31.276 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:32:31.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:31.454 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:31.454 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:31.489 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:31.489 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:31.489 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:32:31.489 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:32:31.489 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:32:31.492 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:32:31.492 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:32:31.492 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:32:31.493 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:32:31.495 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:32:31.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:31.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.567 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:32:31.619 INFO:teuthology.orchestra.run.smithi114.stdout:client6927 2024-03-06T07:32:31.619 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:31.619 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:31.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.662 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:32:31.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.762 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:32:31.829 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:31.829 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:31.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:31.878 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:32:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:35+0 records in 2024-03-06T07:32:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:35+0 records out 2024-03-06T07:32:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.404115 s, 90.8 MB/s 2024-03-06T07:32:32.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:32.369 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:32:32.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:32.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.651+0000 7fa8e1800640 1 Processor -- start 2024-03-06T07:32:32.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.651+0000 7fa8e1800640 1 -- start start 2024-03-06T07:32:32.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.652+0000 7fa8e1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc0d4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.652+0000 7fa8e1800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa8dc0d0a80 0x7fa8dc0d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.652+0000 7fa8e1800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8dc0d4980 con 0x7fa8dc0d0600 2024-03-06T07:32:32.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.652+0000 7fa8e1800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8dc0d4c00 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.652+0000 7fa8e1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0b40b0 con 0x7fa8dc1615c0 2024-03-06T07:32:32.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.653+0000 7fa8dbe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc0d4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.653+0000 7fa8dbe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc0d4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:50076/0 (socket says 172.21.15.114:50076) 2024-03-06T07:32:32.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.653+0000 7fa8dbe00640 1 -- 172.21.15.114:0/36363386 learned_addr learned my addr 172.21.15.114:0/36363386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:32.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.653+0000 7fa8e0e00640 1 --2- 172.21.15.114:0/36363386 >> v2:172.21.15.177:3300/0 conn(0x7fa8dc0d0a80 0x7fa8dc0d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.653+0000 7fa8db400640 1 -- 172.21.15.114:0/36363386 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643448490 0 0) 0x7fa8c4001110 con 0x7fa8dc0d0600 2024-03-06T07:32:32.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.654+0000 7fa8db400640 1 -- 172.21.15.114:0/36363386 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c0003520 con 0x7fa8dc0d0600 2024-03-06T07:32:32.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.654+0000 7fa8dbe00640 1 -- 172.21.15.114:0/36363386 >> v2:172.21.15.177:3300/0 conn(0x7fa8dc0d0a80 msgr2=0x7fa8dc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.654+0000 7fa8dbe00640 1 --2- 172.21.15.114:0/36363386 >> v2:172.21.15.177:3300/0 conn(0x7fa8dc0d0a80 0x7fa8dc0d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.654+0000 7fa8dbe00640 1 -- 172.21.15.114:0/36363386 >> v1:172.21.15.177:6789/0 conn(0x7fa8dc0d0600 legacy=0x7fa8dc0d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.654+0000 7fa8dbe00640 1 -- 172.21.15.114:0/36363386 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc09d5e0 con 0x7fa8dc1615c0 2024-03-06T07:32:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.655+0000 7fa8dbe00640 1 --2- 172.21.15.114:0/36363386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc0d4440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009870 tx=0x7fa8cc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=152c75515ee2f56e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.655+0000 7fa8db400640 1 -- 172.21.15.114:0/36363386 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc035020 con 0x7fa8dc1615c0 2024-03-06T07:32:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.655+0000 7fa8db400640 1 -- 172.21.15.114:0/36363386 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc031640 con 0x7fa8dc1615c0 2024-03-06T07:32:32.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.655+0000 7fa8db400640 1 -- 172.21.15.114:0/36363386 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc02d040 con 0x7fa8dc1615c0 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 -- 172.21.15.114:0/36363386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 msgr2=0x7fa8dc0d4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 --2- 172.21.15.114:0/36363386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc0d4440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009870 tx=0x7fa8cc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 -- 172.21.15.114:0/36363386 shutdown_connections 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 --2- 172.21.15.114:0/36363386 >> v2:172.21.15.177:3300/0 conn(0x7fa8dc0d0a80 0x7fa8dc0d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 --2- 172.21.15.114:0/36363386 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc0d4440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 -- 172.21.15.114:0/36363386 >> 172.21.15.114:0/36363386 conn(0x7fa8dc09d080 msgr2=0x7fa8dc0d3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 -- 172.21.15.114:0/36363386 shutdown_connections 2024-03-06T07:32:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.656+0000 7fa8e1800640 1 -- 172.21.15.114:0/36363386 wait complete. 2024-03-06T07:32:32.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.658+0000 7fa8e1800640 1 Processor -- start 2024-03-06T07:32:32.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.658+0000 7fa8e1800640 1 -- start start 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8e1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8dc0d0600 0x7fa8dc1f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8e1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8dc0d0a80 0x7fa8dc1f7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8e1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc1f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8e1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0b3d40 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8e1800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0b3a60 con 0x7fa8dc0d0600 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8e1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8dc0cf850 con 0x7fa8dc1615c0 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8d3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8dc0d0a80 0x7fa8dc1f7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8d3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8dc0d0a80 0x7fa8dc1f7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58736/0 (socket says 172.21.15.114:58736) 2024-03-06T07:32:32.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.659+0000 7fa8d3400640 1 -- 172.21.15.114:0/2564491604 learned_addr learned my addr 172.21.15.114:0/2564491604 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:32.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8dbe00640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8dc0d0600 0x7fa8dc1f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8e0e00640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc1f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d3400640 1 -- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 msgr2=0x7fa8dc1f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d3400640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc1f7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d3400640 1 -- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8dc0d0600 msgr2=0x7fa8dc1f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d3400640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8dc0d0600 0x7fa8dc1f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d3400640 1 -- 172.21.15.114:0/2564491604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc0cbe20 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d3400640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8dc0d0a80 0x7fa8dc1f7440 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400c730 tx=0x7fa8c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d8c00640 1 -- 172.21.15.114:0/2564491604 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40050d0 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d8c00640 1 -- 172.21.15.114:0/2564491604 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c4005270 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.660+0000 7fa8d8c00640 1 -- 172.21.15.114:0/2564491604 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400a640 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.661+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8dc0cc100 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.661+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8dc0cc6c0 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.662+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8dc0ccaa0 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.663+0000 7fa8d8c00640 1 -- 172.21.15.114:0/2564491604 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8c400a7e0 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.664+0000 7fa8d8c00640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa8b806b4d0 0x7fa8b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.664+0000 7fa8d8c00640 1 -- 172.21.15.114:0/2564491604 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8c4089e60 con 0x7fa8dc0d0a80 2024-03-06T07:32:32.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.664+0000 7fa8dbe00640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa8b806b4d0 0x7fa8b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.665+0000 7fa8dbe00640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa8b806b4d0 0x7fa8b806d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc036000 tx=0x7fa8cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:32.668+0000 7fa8d8c00640 1 -- 172.21.15.114:0/2564491604 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8dc0ccaa0 con 0x7fa8dc0d0a80 2024-03-06T07:32:33.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.011+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8dc0b1140 con 0x7fa8dc0d0a80 2024-03-06T07:32:33.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.012+0000 7fa8d8c00640 1 -- 172.21.15.114:0/2564491604 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7fa8dc0d4980 con 0x7fa8dc0d0a80 2024-03-06T07:32:33.014 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:33.014 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:17.782119+0000","modified":"2024-03-06T07:32:20.854485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2399499569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2399499569},{"type":"v1","addr":"172.21.15.114:6837","nonce":2399499569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:32:33.015 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 52 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa8b806b4d0 msgr2=0x7fa8b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa8b806b4d0 0x7fa8b806d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc036000 tx=0x7fa8cc008030 comp rx=0 tx=0).stop 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8dc0d0a80 msgr2=0x7fa8dc1f7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8dc0d0a80 0x7fa8dc1f7440 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400c730 tx=0x7fa8c400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 shutdown_connections 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8dc1615c0 0x7fa8dc1f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa8b806b4d0 0x7fa8b806d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8dc0d0a80 0x7fa8dc1f7440 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 --2- 172.21.15.114:0/2564491604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8dc0d0600 0x7fa8dc1f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 >> 172.21.15.114:0/2564491604 conn(0x7fa8dc09d080 msgr2=0x7fa8dc0b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 shutdown_connections 2024-03-06T07:32:33.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.016+0000 7fa8e1800640 1 -- 172.21.15.114:0/2564491604 wait complete. 2024-03-06T07:32:33.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:32:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:32:33.279 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:32:33.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:32:33.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.480+0000 7fec37200640 1 Processor -- start 2024-03-06T07:32:33.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.480+0000 7fec37200640 1 -- start start 2024-03-06T07:32:33.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.481+0000 7fec37200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec3815e090 0x7fec380cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.481+0000 7fec37200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fec380d2340 0x7fec380d27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.481+0000 7fec37200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380ce890 con 0x7fec380d1270 2024-03-06T07:32:33.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.481+0000 7fec37200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3815e470 con 0x7fec380d2340 2024-03-06T07:32:33.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.481+0000 7fec37200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec380d2180 con 0x7fec3815e090 2024-03-06T07:32:33.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.482+0000 7fec35400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fec380d1270 0x7fec380d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35754/0 (socket says 172.21.15.114:35754) 2024-03-06T07:32:33.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.482+0000 7fec35400640 1 -- 172.21.15.114:0/2910422094 learned_addr learned my addr 172.21.15.114:0/2910422094 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.482+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348195549 0 0) 0x7fec380ce890 con 0x7fec380d1270 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.482+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec1c003520 con 0x7fec380d1270 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.482+0000 7fec35e00640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec3815e090 0x7fec380cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.482+0000 7fec36800640 1 --2- 172.21.15.114:0/2910422094 >> v2:172.21.15.177:3300/0 conn(0x7fec380d2340 0x7fec380d27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274798669 0 0) 0x7fec1c003520 con 0x7fec380d1270 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec3815e090 msgr2=0x7fec380cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec3815e090 0x7fec380cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 >> v2:172.21.15.177:3300/0 conn(0x7fec380d2340 msgr2=0x7fec380d27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 --2- 172.21.15.114:0/2910422094 >> v2:172.21.15.177:3300/0 conn(0x7fec380d2340 0x7fec380d27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380ceb30 con 0x7fec380d1270 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fec20001670 con 0x7fec380d1270 2024-03-06T07:32:33.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec36800640 1 --2- 172.21.15.114:0/2910422094 >> v2:172.21.15.177:3300/0 conn(0x7fec380d2340 0x7fec380d27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 >> v1:172.21.15.114:6789/0 conn(0x7fec380d1270 legacy=0x7fec380d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c0039b0 0x7fec1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec3815e090 0x7fec1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3815e470 con 0x7fec1c004800 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec380d2180 con 0x7fec1c0039b0 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c00ba70 con 0x7fec3815e090 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.483+0000 7fec35400640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.484+0000 7fec35e00640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c0039b0 0x7fec1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.484+0000 7fec35400640 1 -- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec3815e090 msgr2=0x7fec1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.484+0000 7fec35400640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec3815e090 0x7fec1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.484+0000 7fec35400640 1 -- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c0039b0 msgr2=0x7fec1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.484+0000 7fec35400640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c0039b0 0x7fec1c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.484+0000 7fec35400640 1 -- 172.21.15.114:0/2910422094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c003520 con 0x7fec1c004800 2024-03-06T07:32:33.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.485+0000 7fec35400640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fec20007fd0 tx=0x7fec1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d537610f78b0682d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.485+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20007d50 con 0x7fec1c004800 2024-03-06T07:32:33.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.485+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20030070 con 0x7fec1c004800 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.485+0000 7fec34a00640 1 -- 172.21.15.114:0/2910422094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20005d30 con 0x7fec1c004800 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.486+0000 7fec37200640 1 -- 172.21.15.114:0/2910422094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec1c004800 msgr2=0x7fec1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.486+0000 7fec37200640 1 --2- 172.21.15.114:0/2910422094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fec20007fd0 tx=0x7fec1c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.486+0000 7fec36800640 1 -- 172.21.15.114:0/2910422094 reap_dead start 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.486+0000 7fec37200640 1 -- 172.21.15.114:0/2910422094 shutdown_connections 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.486+0000 7fec37200640 1 -- 172.21.15.114:0/2910422094 >> 172.21.15.114:0/2910422094 conn(0x7fec380d3d80 msgr2=0x7fec380d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.486+0000 7fec37200640 1 -- 172.21.15.114:0/2910422094 shutdown_connections 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.486+0000 7fec37200640 1 -- 172.21.15.114:0/2910422094 wait complete. 2024-03-06T07:32:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.488+0000 7fec37200640 1 Processor -- start 2024-03-06T07:32:33.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.488+0000 7fec37200640 1 -- start start 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec37200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c004800 0x7fec3814aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec37200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec3814e330 0x7fec380b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec37200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec380b5da0 0x7fec381f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec37200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec380cf0f0 con 0x7fec3814e330 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec37200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec380cf460 con 0x7fec1c004800 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec37200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec380cf8f0 con 0x7fec380b5da0 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec35400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec3814e330 0x7fec380b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec35400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec3814e330 0x7fec380b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58758/0 (socket says 172.21.15.114:58758) 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec35400640 1 -- 172.21.15.114:0/3182439290 learned_addr learned my addr 172.21.15.114:0/3182439290 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.489+0000 7fec35e00640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c004800 0x7fec3814aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec35400640 1 -- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec380b5da0 msgr2=0x7fec381f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec35400640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec380b5da0 0x7fec381f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec35400640 1 -- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c004800 msgr2=0x7fec3814aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec35400640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c004800 0x7fec3814aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec35400640 1 -- 172.21.15.114:0/3182439290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec381f84d0 con 0x7fec3814e330 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec35e00640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c004800 0x7fec3814aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec35400640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec3814e330 0x7fec380b5860 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fec20000f60 tx=0x7fec20001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec2e000640 1 -- 172.21.15.114:0/3182439290 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20045070 con 0x7fec3814e330 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec2e000640 1 -- 172.21.15.114:0/3182439290 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20004c10 con 0x7fec3814e330 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec2e000640 1 -- 172.21.15.114:0/3182439290 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20007070 con 0x7fec3814e330 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3814d850 con 0x7fec3814e330 2024-03-06T07:32:33.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.490+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec381f7d70 con 0x7fec3814e330 2024-03-06T07:32:33.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.492+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3804f330 con 0x7fec3814e330 2024-03-06T07:32:33.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.492+0000 7fec2e000640 1 -- 172.21.15.114:0/3182439290 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec20007210 con 0x7fec3814e330 2024-03-06T07:32:33.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.493+0000 7fec2e000640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec1406b500 0x7fec1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.493+0000 7fec2e000640 1 -- 172.21.15.114:0/3182439290 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec200ab270 con 0x7fec3814e330 2024-03-06T07:32:33.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.493+0000 7fec35e00640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec1406b500 0x7fec1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.494+0000 7fec35e00640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec1406b500 0x7fec1406d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fec3815f680 tx=0x7fec28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.502+0000 7fec2e000640 1 -- 172.21.15.114:0/3182439290 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec20079a90 con 0x7fec3814e330 2024-03-06T07:32:33.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.779+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fec380d3b60 con 0x7fec3814e330 2024-03-06T07:32:33.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.780+0000 7fec2e000640 1 -- 172.21.15.114:0/3182439290 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1691 (secure 0 0 0) 0x7fec2007db30 con 0x7fec3814e330 2024-03-06T07:32:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:33.782 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:32:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772811718656, 2024-03-06T07:32:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 282394624, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 282394624, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00036527845077216625, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "num_osds": 8, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_osds": 8, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "stats_by_class": { 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "ssd": { 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772811718656, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 282394624, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 282394624, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00036527845077216625 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "name": ".mgr", 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 1, 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 590368, 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 2, 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 1160, 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 1187840, 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 1.6182445961021585e-06, 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367014379520 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_metadata", 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 12, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 2410, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 22, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 64, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 65536, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 8.9282600868045847e-08, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367014379520 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_data", 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 13, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 8912896, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 1, 2024-03-06T07:32:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 17408, 2024-03-06T07:32:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 17825792, 2024-03-06T07:32:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 2.4284279788844287e-05, 2024-03-06T07:32:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367014379520 2024-03-06T07:32:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:32:33.786 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec1406b500 msgr2=0x7fec1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec1406b500 0x7fec1406d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fec3815f680 tx=0x7fec28008040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec3814e330 msgr2=0x7fec380b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec3814e330 0x7fec380b5860 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fec20000f60 tx=0x7fec20001ab0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 shutdown_connections 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fec380b5da0 0x7fec381f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec1406b500 0x7fec1406d9c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec3814e330 0x7fec380b5860 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 --2- 172.21.15.114:0/3182439290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec1c004800 0x7fec3814aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 >> 172.21.15.114:0/3182439290 conn(0x7fec380d3d80 msgr2=0x7fec380b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 shutdown_connections 2024-03-06T07:32:33.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:33.783+0000 7fec37200640 1 -- 172.21.15.114:0/3182439290 wait complete. 2024-03-06T07:32:33.804 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-06T07:32:38.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.051+0000 7f1645600640 1 Processor -- start 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.051+0000 7f1645600640 1 -- start start 2024-03-06T07:32:39.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f1645600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16400d2360 0x7f16400d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f1645600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16400b3ec0 0x7f16400b4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f1645600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1640161960 con 0x7f1640161580 2024-03-06T07:32:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f1645600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16400d2ce0 con 0x7f16400b3ec0 2024-03-06T07:32:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f1645600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16400cf6f0 con 0x7f16400d2360 2024-03-06T07:32:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f1644c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16400b3ec0 0x7f16400b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f163fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1640161580 0x7f16400d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44572/0 (socket says 172.21.15.114:44572) 2024-03-06T07:32:39.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f163fe00640 1 -- 172.21.15.114:0/3304432166 learned_addr learned my addr 172.21.15.114:0/3304432166 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:39.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.053+0000 7f163f400640 1 --2- 172.21.15.114:0/3304432166 >> v2:172.21.15.177:3300/0 conn(0x7f16400d2360 0x7f16400d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.054+0000 7f163ea00640 1 -- 172.21.15.114:0/3304432166 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152573371 0 0) 0x7f1640161960 con 0x7f1640161580 2024-03-06T07:32:39.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.054+0000 7f1644c00640 1 -- 172.21.15.114:0/3304432166 >> v2:172.21.15.177:3300/0 conn(0x7f16400d2360 msgr2=0x7f16400d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.054+0000 7f1644c00640 1 --2- 172.21.15.114:0/3304432166 >> v2:172.21.15.177:3300/0 conn(0x7f16400d2360 0x7f16400d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.054+0000 7f1644c00640 1 -- 172.21.15.114:0/3304432166 >> v1:172.21.15.114:6789/0 conn(0x7f1640161580 legacy=0x7f16400d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.054+0000 7f1644c00640 1 -- 172.21.15.114:0/3304432166 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16400cf9d0 con 0x7f16400b3ec0 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.055+0000 7f1644c00640 1 --2- 172.21.15.114:0/3304432166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400b3ec0 0x7f16400b4310 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f16400d2e60 tx=0x7f163802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0ac7bc7356021d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.055+0000 7f163ea00640 1 -- 172.21.15.114:0/3304432166 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638039070 con 0x7f16400b3ec0 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.055+0000 7f163ea00640 1 -- 172.21.15.114:0/3304432166 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163802cc60 con 0x7f16400b3ec0 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.055+0000 7f163ea00640 1 -- 172.21.15.114:0/3304432166 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638034660 con 0x7f16400b3ec0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 -- 172.21.15.114:0/3304432166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400b3ec0 msgr2=0x7f16400b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 --2- 172.21.15.114:0/3304432166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400b3ec0 0x7f16400b4310 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f16400d2e60 tx=0x7f163802beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 -- 172.21.15.114:0/3304432166 shutdown_connections 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 --2- 172.21.15.114:0/3304432166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400b3ec0 0x7f16400b4310 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 --2- 172.21.15.114:0/3304432166 >> v2:172.21.15.177:3300/0 conn(0x7f16400d2360 0x7f16400d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 -- 172.21.15.114:0/3304432166 >> 172.21.15.114:0/3304432166 conn(0x7f16400d3d80 msgr2=0x7f16400d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 -- 172.21.15.114:0/3304432166 shutdown_connections 2024-03-06T07:32:39.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.056+0000 7f1645600640 1 -- 172.21.15.114:0/3304432166 wait complete. 2024-03-06T07:32:39.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.058+0000 7f1645600640 1 Processor -- start 2024-03-06T07:32:39.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.058+0000 7f1645600640 1 -- start start 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f1645600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16400b3ec0 0x7f164014f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f1645600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400d2360 0x7f16400cc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f1645600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1640161580 0x7f164006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f1645600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16400b39a0 con 0x7f16400d2360 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f1645600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16400b36c0 con 0x7f16400b3ec0 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f1645600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16400cf3a0 con 0x7f1640161580 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f163f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400d2360 0x7f16400cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f163f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400d2360 0x7f16400cc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51678/0 (socket says 172.21.15.114:51678) 2024-03-06T07:32:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.060+0000 7f163f400640 1 -- 172.21.15.114:0/322987947 learned_addr learned my addr 172.21.15.114:0/322987947 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:39.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163fe00640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16400b3ec0 0x7f164014f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f1644c00640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1640161580 0x7f164006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163f400640 1 -- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1640161580 msgr2=0x7f164006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163f400640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1640161580 0x7f164006c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163f400640 1 -- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16400b3ec0 msgr2=0x7f164014f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163f400640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16400b3ec0 0x7f164014f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163f400640 1 -- 172.21.15.114:0/322987947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16400cdc70 con 0x7f16400d2360 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163fe00640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16400b3ec0 0x7f164014f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f163f400640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400d2360 0x7f16400cc150 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f162c00d5f0 tx=0x7f162c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f1637e00640 1 -- 172.21.15.114:0/322987947 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0125f0 con 0x7f16400d2360 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f1637e00640 1 -- 172.21.15.114:0/322987947 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c019070 con 0x7f16400d2360 2024-03-06T07:32:39.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f1637e00640 1 -- 172.21.15.114:0/322987947 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0129d0 con 0x7f16400d2360 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.061+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16400ccbd0 con 0x7f16400d2360 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.062+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16400cd190 con 0x7f16400d2360 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.063+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164004f330 con 0x7f16400d2360 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.064+0000 7f1637e00640 1 -- 172.21.15.114:0/322987947 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f162c012b70 con 0x7f16400d2360 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.065+0000 7f1637e00640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f162006b4d0 0x7f162006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.065+0000 7f1637e00640 1 -- 172.21.15.114:0/322987947 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f162c08a680 con 0x7f16400d2360 2024-03-06T07:32:39.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.065+0000 7f163fe00640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f162006b4d0 0x7f162006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.065+0000 7f163fe00640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f162006b4d0 0x7f162006d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1630000f60 tx=0x7f1630009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.070+0000 7f1637e00640 1 -- 172.21.15.114:0/322987947 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f162c059440 con 0x7f16400d2360 2024-03-06T07:32:39.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.338+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f16400d3b60 con 0x7f16400d2360 2024-03-06T07:32:39.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.339+0000 7f1637e00640 1 -- 172.21.15.114:0/322987947 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1693 (secure 0 0 0) 0x7f162c058d40 con 0x7f16400d2360 2024-03-06T07:32:39.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772756180992, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 337932288, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 337932288, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00043711662874557078, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "num_osds": 8, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_osds": 8, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "stats_by_class": { 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "ssd": { 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:39.341 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772756180992, 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 337932288, 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 337932288, 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00043711662874557078 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "name": ".mgr", 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 1, 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 590368, 2024-03-06T07:32:39.342 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 2, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 1160, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 1187840, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 1.6182445961021585e-06, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367014379520 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_metadata", 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 12, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 17699, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 22, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 88, 2024-03-06T07:32:39.343 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 90112, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 1.2276356642360042e-07, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367014379520 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_data", 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 13, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 36700160, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 7, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 71680, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 73400320, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 9.9986522400286049e-05, 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367014379520 2024-03-06T07:32:39.344 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:39.345 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:39.345 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:32:39.345 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:32:39.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f162006b4d0 msgr2=0x7f162006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f162006b4d0 0x7f162006d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1630000f60 tx=0x7f1630009040 comp rx=0 tx=0).stop 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400d2360 msgr2=0x7f16400cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400d2360 0x7f16400cc150 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f162c00d5f0 tx=0x7f162c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 shutdown_connections 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1640161580 0x7f164006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f162006b4d0 0x7f162006d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16400d2360 0x7f16400cc150 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 --2- 172.21.15.114:0/322987947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16400b3ec0 0x7f164014f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 >> 172.21.15.114:0/322987947 conn(0x7f16400d3d80 msgr2=0x7f16400b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 shutdown_connections 2024-03-06T07:32:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:39.341+0000 7f1645600640 1 -- 172.21.15.114:0/322987947 wait complete. 2024-03-06T07:32:39.360 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-06T07:32:44.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:32:44.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.577+0000 7fddfaa00640 1 Processor -- start 2024-03-06T07:32:44.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.577+0000 7fddfaa00640 1 -- start start 2024-03-06T07:32:44.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.578+0000 7fddfaa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fddf4071d20 0x7fddf40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.578+0000 7fddfaa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf40b5310 0x7fddf40ce070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.578+0000 7fddfaa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4071280 con 0x7fddf41566a0 2024-03-06T07:32:44.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.578+0000 7fddfaa00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf40b5780 con 0x7fddf4071d20 2024-03-06T07:32:44.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.578+0000 7fddfaa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4072df0 con 0x7fddf40b5310 2024-03-06T07:32:44.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.578+0000 7fddf3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fddf41566a0 0x7fddf415ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44576/0 (socket says 172.21.15.114:44576) 2024-03-06T07:32:44.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.578+0000 7fddf3e00640 1 -- 172.21.15.114:0/4294731855 learned_addr learned my addr 172.21.15.114:0/4294731855 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddebe00640 1 --2- 172.21.15.114:0/4294731855 >> v2:172.21.15.177:3300/0 conn(0x7fddf4071d20 0x7fddf40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf8a00640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf40b5310 0x7fddf40ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399827207 0 0) 0x7fddf4071280 con 0x7fddf41566a0 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd8003520 con 0x7fddf41566a0 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547255002 0 0) 0x7fddd8003520 con 0x7fddf41566a0 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf40b5310 msgr2=0x7fddf40ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf40b5310 0x7fddf40ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 >> v2:172.21.15.177:3300/0 conn(0x7fddf4071d20 msgr2=0x7fddf40b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 --2- 172.21.15.114:0/4294731855 >> v2:172.21.15.177:3300/0 conn(0x7fddf4071d20 0x7fddf40b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40ccbd0 con 0x7fddf41566a0 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fdde4001670 con 0x7fddf41566a0 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 >> v1:172.21.15.114:6789/0 conn(0x7fddf41566a0 legacy=0x7fddf415ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.579+0000 7fddf3400640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3400640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddd8004800 0x7fddd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3400640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf40b5310 0x7fddd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40b5780 con 0x7fddd80039b0 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4072df0 con 0x7fddd8004800 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd800b9c0 con 0x7fddf40b5310 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3e00640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3e00640 1 -- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf40b5310 msgr2=0x7fddd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3e00640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf40b5310 0x7fddd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3e00640 1 -- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddd8004800 msgr2=0x7fddd8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3e00640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddd8004800 0x7fddd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.580+0000 7fddf3e00640 1 -- 172.21.15.114:0/4294731855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8003520 con 0x7fddd80039b0 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.581+0000 7fddf3e00640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddd8003dd0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fdde4001050 tx=0x7fddd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab0cda4b0573dba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4007d70 con 0x7fddd80039b0 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde4030070 con 0x7fddd80039b0 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddf3400640 1 -- 172.21.15.114:0/4294731855 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde402cdc0 con 0x7fddd80039b0 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddfaa00640 1 -- 172.21.15.114:0/4294731855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 msgr2=0x7fddd8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddfaa00640 1 --2- 172.21.15.114:0/4294731855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddd8003dd0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fdde4001050 tx=0x7fddd8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddf8a00640 1 -- 172.21.15.114:0/4294731855 reap_dead start 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddfaa00640 1 -- 172.21.15.114:0/4294731855 shutdown_connections 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddfaa00640 1 -- 172.21.15.114:0/4294731855 >> 172.21.15.114:0/4294731855 conn(0x7fddf40cfa30 msgr2=0x7fddf40cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddfaa00640 1 -- 172.21.15.114:0/4294731855 shutdown_connections 2024-03-06T07:32:44.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.582+0000 7fddfaa00640 1 -- 172.21.15.114:0/4294731855 wait complete. 2024-03-06T07:32:44.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.584+0000 7fddfaa00640 1 Processor -- start 2024-03-06T07:32:44.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.584+0000 7fddfaa00640 1 -- start start 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddfaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddf41e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddfaa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf41e48e0 0x7fddf414d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddfaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf41e4d40 0x7fddf413e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddfaa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4071800 con 0x7fddd80039b0 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddfaa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4071520 con 0x7fddf41e4d40 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddfaa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40cc4a0 con 0x7fddf41e48e0 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddf3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddf41e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddf3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddf41e6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51692/0 (socket says 172.21.15.114:51692) 2024-03-06T07:32:44.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.585+0000 7fddf3e00640 1 -- 172.21.15.114:0/3614906407 learned_addr learned my addr 172.21.15.114:0/3614906407 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf3e00640 1 -- 172.21.15.114:0/3614906407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf41e48e0 msgr2=0x7fddf414d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf3e00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf41e48e0 0x7fddf414d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf3e00640 1 -- 172.21.15.114:0/3614906407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf41e4d40 msgr2=0x7fddf413e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf8a00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf41e4d40 0x7fddf413e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf3e00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf41e4d40 0x7fddf413e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf3e00640 1 -- 172.21.15.114:0/3614906407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf414e1a0 con 0x7fddd80039b0 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf3e00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddf41e6580 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c300 tx=0x7fdde4000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf0c00640 1 -- 172.21.15.114:0/3614906407 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde40049f0 con 0x7fddd80039b0 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf0c00640 1 -- 172.21.15.114:0/3614906407 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde4004b90 con 0x7fddd80039b0 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddf0c00640 1 -- 172.21.15.114:0/3614906407 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4007070 con 0x7fddd80039b0 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.586+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf414e480 con 0x7fddd80039b0 2024-03-06T07:32:44.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.587+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf414ea40 con 0x7fddd80039b0 2024-03-06T07:32:44.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.588+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf404f330 con 0x7fddd80039b0 2024-03-06T07:32:44.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.588+0000 7fddf0c00640 1 -- 172.21.15.114:0/3614906407 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdde4048020 con 0x7fddd80039b0 2024-03-06T07:32:44.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.590+0000 7fddf0c00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddd006b4d0 0x7fddd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.590+0000 7fddf0c00640 1 -- 172.21.15.114:0/3614906407 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdde40ab8c0 con 0x7fddd80039b0 2024-03-06T07:32:44.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.590+0000 7fddebe00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddd006b4d0 0x7fddd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.591+0000 7fddebe00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddd006b4d0 0x7fddd006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdddc006e80 tx=0x7fdddc006dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.595+0000 7fddf0c00640 1 -- 172.21.15.114:0/3614906407 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdde407a0a0 con 0x7fddd80039b0 2024-03-06T07:32:44.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.865+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fddf4070e10 con 0x7fddd80039b0 2024-03-06T07:32:44.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.866+0000 7fddf0c00640 1 -- 172.21.15.114:0/3614906407 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1656 (secure 0 0 0) 0x7fdde407e140 con 0x7fddd80039b0 2024-03-06T07:32:44.867 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:44.867 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:32:44.867 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:44.867 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772829384704, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 264728576, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 264728576, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00034242734545841813, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "num_osds": 8, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_osds": 8, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "stats_by_class": { 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "ssd": { 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:44.868 INFO:teuthology.orchestra.run.smithi114.stdout: "total_avail_bytes": 772829384704, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_bytes": 264728576, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_bytes": 264728576, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "total_used_raw_ratio": 0.00034242734545841813 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "name": ".mgr", 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 1, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 590368, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 2, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 1160, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 1187840, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 1.6179308204300469e-06, 2024-03-06T07:32:44.869 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367085551616 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_metadata", 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 12, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 20942, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 23, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 104, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 106496, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 1.4505607737191895e-07, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367085551616 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:32:44.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephfs_data", 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 13, 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "stats": { 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "stored": 0, 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "objects": 0, 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "kb_used": 0, 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "bytes_used": 0, 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "percent_used": 0, 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: "max_avail": 367085551616 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:32:44.871 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:32:44.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.868+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddd006b4d0 msgr2=0x7fddd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.868+0000 7fddfaa00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddd006b4d0 0x7fddd006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdddc006e80 tx=0x7fdddc006dd0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.868+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 msgr2=0x7fddf41e6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.868+0000 7fddfaa00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddf41e6580 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c300 tx=0x7fdde4000fe0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.868+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 shutdown_connections 2024-03-06T07:32:44.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.869+0000 7fddfaa00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddf41e4d40 0x7fddf413e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.869+0000 7fddfaa00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddf41e48e0 0x7fddf414d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.869+0000 7fddfaa00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddd006b4d0 0x7fddd006d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.869+0000 7fddfaa00640 1 --2- 172.21.15.114:0/3614906407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddd80039b0 0x7fddf41e6580 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.869+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 >> 172.21.15.114:0/3614906407 conn(0x7fddf40cfa30 msgr2=0x7fddf406cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.869+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 shutdown_connections 2024-03-06T07:32:44.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:44.869+0000 7fddfaa00640 1 -- 172.21.15.114:0/3614906407 wait complete. 2024-03-06T07:32:44.890 DEBUG:tasks.ceph_test_case:wait_until_true: success in 10s and 0 retries 2024-03-06T07:32:44.890 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:44.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:44.991 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:45.114 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:45.114 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:45.114 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:45.168 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:45.168 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:45.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.467+0000 7f6cf2600640 1 Processor -- start 2024-03-06T07:32:45.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.467+0000 7f6cf2600640 1 -- start start 2024-03-06T07:32:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.468+0000 7f6cf2600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6cec144c00 0x7f6cec06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.468+0000 7f6cf2600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec06ee70 con 0x7f6cec15f9d0 2024-03-06T07:32:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.468+0000 7f6cf2600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec06cef0 con 0x7f6cec15fe60 2024-03-06T07:32:45.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.468+0000 7f6cf2600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec06d1d0 con 0x7f6cec144c00 2024-03-06T07:32:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.469+0000 7f6ceb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6cec144c00 0x7f6cec06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.469+0000 7f6ceaa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6cec15f9d0 0x7f6cec0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44586/0 (socket says 172.21.15.114:44586) 2024-03-06T07:32:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.469+0000 7f6ceaa00640 1 -- 172.21.15.114:0/3206190807 learned_addr learned my addr 172.21.15.114:0/3206190807 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.469+0000 7f6ceb400640 1 -- 172.21.15.114:0/3206190807 >> v1:172.21.15.177:6789/0 conn(0x7f6cec15fe60 legacy=0x7f6cec0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.469+0000 7f6ceb400640 1 -- 172.21.15.114:0/3206190807 >> v1:172.21.15.114:6789/0 conn(0x7f6cec15f9d0 legacy=0x7f6cec0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.469+0000 7f6ceb400640 1 -- 172.21.15.114:0/3206190807 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec06d4b0 con 0x7f6cec144c00 2024-03-06T07:32:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.470+0000 7f6ceb400640 1 --2- 172.21.15.114:0/3206190807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec144c00 0x7f6cec06e930 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0097a0 tx=0x7f6cdc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a916fdcca487bd3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cea000640 1 -- 172.21.15.114:0/3206190807 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc035020 con 0x7f6cec144c00 2024-03-06T07:32:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cea000640 1 -- 172.21.15.114:0/3206190807 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc031830 con 0x7f6cec144c00 2024-03-06T07:32:45.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cea000640 1 -- 172.21.15.114:0/3206190807 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc031c00 con 0x7f6cec144c00 2024-03-06T07:32:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cf2600640 1 -- 172.21.15.114:0/3206190807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec144c00 msgr2=0x7f6cec06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cf2600640 1 --2- 172.21.15.114:0/3206190807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec144c00 0x7f6cec06e930 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc0097a0 tx=0x7f6cdc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cf2600640 1 -- 172.21.15.114:0/3206190807 shutdown_connections 2024-03-06T07:32:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cf2600640 1 --2- 172.21.15.114:0/3206190807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec144c00 0x7f6cec06e930 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cf2600640 1 -- 172.21.15.114:0/3206190807 >> 172.21.15.114:0/3206190807 conn(0x7f6cec0b1730 msgr2=0x7f6cec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cf2600640 1 -- 172.21.15.114:0/3206190807 shutdown_connections 2024-03-06T07:32:45.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.471+0000 7f6cf2600640 1 -- 172.21.15.114:0/3206190807 wait complete. 2024-03-06T07:32:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.473+0000 7f6cf2600640 1 Processor -- start 2024-03-06T07:32:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.473+0000 7f6cf2600640 1 -- start start 2024-03-06T07:32:45.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cf2600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6cec144c00 0x7f6cec15d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cf2600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cec15f9d0 0x7f6cec0b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cf2600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec15fe60 0x7f6cec0b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cf2600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec160730 con 0x7f6cec15fe60 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cf2600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec160270 con 0x7f6cec15f9d0 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cf2600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0ceec0 con 0x7f6cec144c00 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cebe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec15fe60 0x7f6cec0b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cebe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec15fe60 0x7f6cec0b2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51702/0 (socket says 172.21.15.114:51702) 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.474+0000 7f6cebe00640 1 -- 172.21.15.114:0/2498627478 learned_addr learned my addr 172.21.15.114:0/2498627478 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6cebe00640 1 -- 172.21.15.114:0/2498627478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6cec144c00 msgr2=0x7f6cec15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6cebe00640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6cec144c00 0x7f6cec15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6cebe00640 1 -- 172.21.15.114:0/2498627478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cec15f9d0 msgr2=0x7f6cec0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6cebe00640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cec15f9d0 0x7f6cec0b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6cebe00640 1 -- 172.21.15.114:0/2498627478 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0b00d0 con 0x7f6cec15fe60 2024-03-06T07:32:45.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6cebe00640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec15fe60 0x7f6cec0b2c80 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400c780 tx=0x7f6ce400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6ce3400640 1 -- 172.21.15.114:0/2498627478 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4004b60 con 0x7f6cec15fe60 2024-03-06T07:32:45.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6ce3400640 1 -- 172.21.15.114:0/2498627478 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce4004d00 con 0x7f6cec15fe60 2024-03-06T07:32:45.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6ce3400640 1 -- 172.21.15.114:0/2498627478 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400aa90 con 0x7f6cec15fe60 2024-03-06T07:32:45.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.475+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec15da40 con 0x7f6cec15fe60 2024-03-06T07:32:45.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.476+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec15c890 con 0x7f6cec15fe60 2024-03-06T07:32:45.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.477+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec15dc40 con 0x7f6cec15fe60 2024-03-06T07:32:45.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.477+0000 7f6ce3400640 1 -- 172.21.15.114:0/2498627478 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ce400ac30 con 0x7f6cec15fe60 2024-03-06T07:32:45.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.478+0000 7f6ce3400640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ccc06b4d0 0x7f6ccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.478+0000 7f6ce3400640 1 -- 172.21.15.114:0/2498627478 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6ce408a490 con 0x7f6cec15fe60 2024-03-06T07:32:45.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.479+0000 7f6ceb400640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ccc06b4d0 0x7f6ccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.479+0000 7f6ceb400640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ccc06b4d0 0x7f6ccc06d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc036000 tx=0x7f6cdc030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.483+0000 7f6ce3400640 1 -- 172.21.15.114:0/2498627478 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6cec15dc40 con 0x7f6cec15fe60 2024-03-06T07:32:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.679+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cec04f330 con 0x7f6cec15fe60 2024-03-06T07:32:45.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.681+0000 7f6ce3400640 1 -- 172.21.15.114:0/2498627478 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16757 (secure 0 0 0) 0x7f6ce4020070 con 0x7f6cec15fe60 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:45.683 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":82,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:37.454277+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:15.626899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:16.804201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.c","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ccc06b4d0 msgr2=0x7f6ccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ccc06b4d0 0x7f6ccc06d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc036000 tx=0x7f6cdc030000 comp rx=0 tx=0).stop 2024-03-06T07:32:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec15fe60 msgr2=0x7f6cec0b2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec15fe60 0x7f6cec0b2c80 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400c780 tx=0x7f6ce400cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 shutdown_connections 2024-03-06T07:32:45.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ccc06b4d0 0x7f6ccc06d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cec15fe60 0x7f6cec0b2c80 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6cec15f9d0 0x7f6cec0b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 --2- 172.21.15.114:0/2498627478 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6cec144c00 0x7f6cec15d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 >> 172.21.15.114:0/2498627478 conn(0x7f6cec0b1730 msgr2=0x7f6cec0b08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.683+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 shutdown_connections 2024-03-06T07:32:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.684+0000 7f6cf2600640 1 -- 172.21.15.114:0/2498627478 wait complete. 2024-03-06T07:32:45.702 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:15.626899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:16.804201+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:45.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.905+0000 7fd1f9000640 1 Processor -- start 2024-03-06T07:32:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.905+0000 7fd1f9000640 1 -- start start 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1f415c2a0 0x7fd1f40af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f9000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 0x7fd1f40ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f9000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f40b2d80 con 0x7fd1f406afc0 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f9000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f40b0690 con 0x7fd1f415c2a0 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f9000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f40731b0 con 0x7fd1f4066e00 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f1e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1f415c2a0 0x7fd1f40af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f1e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1f415c2a0 0x7fd1f40af9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51712/0 (socket says 172.21.15.114:51712) 2024-03-06T07:32:45.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.907+0000 7fd1f1e00640 1 -- 172.21.15.114:0/1421467099 learned_addr learned my addr 172.21.15.114:0/1421467099 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.908+0000 7fd1f1e00640 1 -- 172.21.15.114:0/1421467099 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 msgr2=0x7fd1f40ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.908+0000 7fd1f1e00640 1 --2- 172.21.15.114:0/1421467099 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 0x7fd1f40ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.908+0000 7fd1f1e00640 1 -- 172.21.15.114:0/1421467099 >> v1:172.21.15.177:6789/0 conn(0x7fd1f406afc0 legacy=0x7fd1f40b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.908+0000 7fd1f1e00640 1 -- 172.21.15.114:0/1421467099 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f4073450 con 0x7fd1f415c2a0 2024-03-06T07:32:45.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.909+0000 7fd1f1e00640 1 --2- 172.21.15.114:0/1421467099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f40af9f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40098a0 tx=0x7fd1e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6560ba33d13d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.909+0000 7fd1f1400640 1 -- 172.21.15.114:0/1421467099 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4035020 con 0x7fd1f415c2a0 2024-03-06T07:32:45.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.909+0000 7fd1f1400640 1 -- 172.21.15.114:0/1421467099 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e4031640 con 0x7fd1f415c2a0 2024-03-06T07:32:45.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.909+0000 7fd1f1400640 1 -- 172.21.15.114:0/1421467099 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e402d040 con 0x7fd1f415c2a0 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 -- 172.21.15.114:0/1421467099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 msgr2=0x7fd1f40af9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 --2- 172.21.15.114:0/1421467099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f40af9f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40098a0 tx=0x7fd1e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 -- 172.21.15.114:0/1421467099 shutdown_connections 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 --2- 172.21.15.114:0/1421467099 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 0x7fd1f40ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 --2- 172.21.15.114:0/1421467099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f40af9f0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 -- 172.21.15.114:0/1421467099 >> 172.21.15.114:0/1421467099 conn(0x7fd1f40b1e80 msgr2=0x7fd1f40b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 -- 172.21.15.114:0/1421467099 shutdown_connections 2024-03-06T07:32:45.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.910+0000 7fd1f9000640 1 -- 172.21.15.114:0/1421467099 wait complete. 2024-03-06T07:32:45.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.912+0000 7fd1f9000640 1 Processor -- start 2024-03-06T07:32:45.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.912+0000 7fd1f9000640 1 -- start start 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f9000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 0x7fd1f40c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1f406afc0 0x7fd1f409d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f41f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f40b29a0 con 0x7fd1f415c2a0 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f415c680 con 0x7fd1f406afc0 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f9000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f40b2b20 con 0x7fd1f4066e00 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f41f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f41f1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51722/0 (socket says 172.21.15.114:51722) 2024-03-06T07:32:45.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f2800640 1 -- 172.21.15.114:0/450793381 learned_addr learned my addr 172.21.15.114:0/450793381 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1f1e00640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 0x7fd1f40c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.913+0000 7fd1e9400640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1f406afc0 0x7fd1f409d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f2800640 1 -- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 msgr2=0x7fd1f40c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f2800640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 0x7fd1f40c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f2800640 1 -- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1f406afc0 msgr2=0x7fd1f409d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f2800640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1f406afc0 0x7fd1f409d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f2800640 1 -- 172.21.15.114:0/450793381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f409a4c0 con 0x7fd1f415c2a0 2024-03-06T07:32:45.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f2800640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f41f1e60 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00c780 tx=0x7fd1ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1eaa00640 1 -- 172.21.15.114:0/450793381 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec004b60 con 0x7fd1f415c2a0 2024-03-06T07:32:45.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1eaa00640 1 -- 172.21.15.114:0/450793381 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ec004d00 con 0x7fd1f415c2a0 2024-03-06T07:32:45.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f409a7a0 con 0x7fd1f415c2a0 2024-03-06T07:32:45.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1eaa00640 1 -- 172.21.15.114:0/450793381 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec00aa90 con 0x7fd1f415c2a0 2024-03-06T07:32:45.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.914+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f409ad60 con 0x7fd1f415c2a0 2024-03-06T07:32:45.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.916+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f409af60 con 0x7fd1f415c2a0 2024-03-06T07:32:45.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:45.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.918+0000 7fd1eaa00640 1 -- 172.21.15.114:0/450793381 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1ec00ac30 con 0x7fd1f415c2a0 2024-03-06T07:32:45.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.919+0000 7fd1eaa00640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd1d006b4d0 0x7fd1d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.919+0000 7fd1eaa00640 1 -- 172.21.15.114:0/450793381 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd1ec08a590 con 0x7fd1f415c2a0 2024-03-06T07:32:45.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.919+0000 7fd1f1e00640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd1d006b4d0 0x7fd1d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.920+0000 7fd1f1e00640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd1d006b4d0 0x7fd1d006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4036000 tx=0x7fd1e403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:45.924+0000 7fd1eaa00640 1 -- 172.21.15.114:0/450793381 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1f409af60 con 0x7fd1f415c2a0 2024-03-06T07:32:46.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.122+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1f415f830 con 0x7fd1f415c2a0 2024-03-06T07:32:46.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.123+0000 7fd1eaa00640 1 -- 172.21.15.114:0/450793381 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1ec020070 con 0x7fd1f415c2a0 2024-03-06T07:32:46.124 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:32:46.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd1d006b4d0 msgr2=0x7fd1d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd1d006b4d0 0x7fd1d006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4036000 tx=0x7fd1e403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 msgr2=0x7fd1f41f1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f41f1e60 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00c780 tx=0x7fd1ec00cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 shutdown_connections 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd1d006b4d0 0x7fd1d006d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1f415c2a0 0x7fd1f41f1e60 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1f406afc0 0x7fd1f409d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 --2- 172.21.15.114:0/450793381 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1f4066e00 0x7fd1f40c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.125+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 >> 172.21.15.114:0/450793381 conn(0x7fd1f40b1e80 msgr2=0x7fd1f40b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.126+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 shutdown_connections 2024-03-06T07:32:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.126+0000 7fd1f9000640 1 -- 172.21.15.114:0/450793381 wait complete. 2024-03-06T07:32:46.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:46.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.339+0000 7f31a2a00640 1 Processor -- start 2024-03-06T07:32:46.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.339+0000 7f31a2a00640 1 -- start start 2024-03-06T07:32:46.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a2a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f319c15dcb0 0x7f319c0b5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a2a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f319c15f100 0x7f319c0b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a2a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319c06d060 con 0x7f319c15f100 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a2a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319c06cd80 con 0x7f319c15dcb0 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a2a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c15e620 con 0x7f319c15e090 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a0c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f319c15f100 0x7f319c0b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a0c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f319c15f100 0x7f319c0b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51734/0 (socket says 172.21.15.114:51734) 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.341+0000 7f31a0c00640 1 -- 172.21.15.114:0/1218913836 learned_addr learned my addr 172.21.15.114:0/1218913836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a2000640 1 --2- 172.21.15.114:0/1218913836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a1600640 1 --2- 172.21.15.114:0/1218913836 >> v2:172.21.15.177:3300/0 conn(0x7f319c15dcb0 0x7f319c0b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a0c00640 1 -- 172.21.15.114:0/1218913836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 msgr2=0x7f319c0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a0c00640 1 --2- 172.21.15.114:0/1218913836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a0c00640 1 -- 172.21.15.114:0/1218913836 >> v2:172.21.15.177:3300/0 conn(0x7f319c15dcb0 msgr2=0x7f319c0b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a0c00640 1 --2- 172.21.15.114:0/1218913836 >> v2:172.21.15.177:3300/0 conn(0x7f319c15dcb0 0x7f319c0b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a0c00640 1 -- 172.21.15.114:0/1218913836 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c15e900 con 0x7f319c15f100 2024-03-06T07:32:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.342+0000 7f31a1600640 1 --2- 172.21.15.114:0/1218913836 >> v2:172.21.15.177:3300/0 conn(0x7f319c15dcb0 0x7f319c0b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:46.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.343+0000 7f31a0c00640 1 --2- 172.21.15.114:0/1218913836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15f100 0x7f319c0b38f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f319c072500 tx=0x7f318802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385fe9e3e06a0508 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.343+0000 7f319be00640 1 -- 172.21.15.114:0/1218913836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188035020 con 0x7f319c15f100 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.343+0000 7f319be00640 1 -- 172.21.15.114:0/1218913836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3188031870 con 0x7f319c15f100 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.343+0000 7f319be00640 1 -- 172.21.15.114:0/1218913836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188031c40 con 0x7f319c15f100 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 -- 172.21.15.114:0/1218913836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15f100 msgr2=0x7f319c0b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 --2- 172.21.15.114:0/1218913836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15f100 0x7f319c0b38f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f319c072500 tx=0x7f318802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 -- 172.21.15.114:0/1218913836 shutdown_connections 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 --2- 172.21.15.114:0/1218913836 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 --2- 172.21.15.114:0/1218913836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15f100 0x7f319c0b38f0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 --2- 172.21.15.114:0/1218913836 >> v2:172.21.15.177:3300/0 conn(0x7f319c15dcb0 0x7f319c0b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 -- 172.21.15.114:0/1218913836 >> 172.21.15.114:0/1218913836 conn(0x7f319c0b25f0 msgr2=0x7f319c0b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 -- 172.21.15.114:0/1218913836 shutdown_connections 2024-03-06T07:32:46.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.344+0000 7f31a2a00640 1 -- 172.21.15.114:0/1218913836 wait complete. 2024-03-06T07:32:46.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.346+0000 7f31a2a00640 1 Processor -- start 2024-03-06T07:32:46.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.346+0000 7f31a2a00640 1 -- start start 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15dcb0 0x7f319c1575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f319c15f100 0x7f319c0aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a2a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c06ca70 con 0x7f319c15dcb0 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a2a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c06c630 con 0x7f319c15f100 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a2a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c0b5cc0 con 0x7f319c15e090 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15dcb0 0x7f319c1575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15dcb0 0x7f319c1575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51740/0 (socket says 172.21.15.114:51740) 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a1600640 1 -- 172.21.15.114:0/3978399163 learned_addr learned my addr 172.21.15.114:0/3978399163 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a0c00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.347+0000 7f31a2000640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f319c15f100 0x7f319c0aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a1600640 1 -- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 msgr2=0x7f319c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a1600640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a1600640 1 -- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f319c15f100 msgr2=0x7f319c0aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a1600640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f319c15f100 0x7f319c0aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a1600640 1 -- 172.21.15.114:0/3978399163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c157b00 con 0x7f319c15dcb0 2024-03-06T07:32:46.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a2000640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f319c15f100 0x7f319c0aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a1600640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15dcb0 0x7f319c1575c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f318c00c7f0 tx=0x7f318c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f3199600640 1 -- 172.21.15.114:0/3978399163 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c015070 con 0x7f319c15dcb0 2024-03-06T07:32:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f3199600640 1 -- 172.21.15.114:0/3978399163 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318c007a30 con 0x7f319c15dcb0 2024-03-06T07:32:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f3199600640 1 -- 172.21.15.114:0/3978399163 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c007e00 con 0x7f319c15dcb0 2024-03-06T07:32:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c1e8140 con 0x7f319c15dcb0 2024-03-06T07:32:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.348+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c1e7b80 con 0x7f319c15dcb0 2024-03-06T07:32:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.350+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c04f330 con 0x7f319c15dcb0 2024-03-06T07:32:46.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.351+0000 7f3199600640 1 -- 172.21.15.114:0/3978399163 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f318c018070 con 0x7f319c15dcb0 2024-03-06T07:32:46.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.351+0000 7f3199600640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f317c06b500 0x7f317c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.352+0000 7f3199600640 1 -- 172.21.15.114:0/3978399163 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f318c08a100 con 0x7f319c15dcb0 2024-03-06T07:32:46.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.352+0000 7f31a0c00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f317c06b500 0x7f317c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.352+0000 7f31a0c00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f317c06b500 0x7f317c06d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3188036000 tx=0x7f318803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.356+0000 7f3199600640 1 -- 172.21.15.114:0/3978399163 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f318c058f50 con 0x7f319c15dcb0 2024-03-06T07:32:46.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.553+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f319c0b4a80 con 0x7f319c15dcb0 2024-03-06T07:32:46.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.553+0000 7f3199600640 1 -- 172.21.15.114:0/3978399163 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f318c058850 con 0x7f319c15dcb0 2024-03-06T07:32:46.555 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f317c06b500 msgr2=0x7f317c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f317c06b500 0x7f317c06d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3188036000 tx=0x7f318803c040 comp rx=0 tx=0).stop 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15dcb0 msgr2=0x7f319c1575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15dcb0 0x7f319c1575c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f318c00c7f0 tx=0x7f318c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 shutdown_connections 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f319c15f100 0x7f319c0aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f319c15e090 0x7f319c159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f317c06b500 0x7f317c06d9c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 --2- 172.21.15.114:0/3978399163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f319c15dcb0 0x7f319c1575c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 >> 172.21.15.114:0/3978399163 conn(0x7f319c0b25f0 msgr2=0x7f319c0b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 shutdown_connections 2024-03-06T07:32:46.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.556+0000 7f31a2a00640 1 -- 172.21.15.114:0/3978399163 wait complete. 2024-03-06T07:32:46.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.778+0000 7f25e2600640 1 Processor -- start 2024-03-06T07:32:46.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.779+0000 7f25e2600640 1 -- start start 2024-03-06T07:32:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.780+0000 7f25e2600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f25dc15fc10 0x7f25dc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.780+0000 7f25e2600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0b46c0 con 0x7f25dc0b3f70 2024-03-06T07:32:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.780+0000 7f25e2600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0b49a0 con 0x7f25dc15fff0 2024-03-06T07:32:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.780+0000 7f25e2600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc072c90 con 0x7f25dc15fc10 2024-03-06T07:32:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.780+0000 7f25dbe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f25dc0b3f70 0x7f25dc0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44588/0 (socket says 172.21.15.114:44588) 2024-03-06T07:32:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.780+0000 7f25dbe00640 1 -- 172.21.15.114:0/118052921 learned_addr learned my addr 172.21.15.114:0/118052921 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25db400640 1 --2- 172.21.15.114:0/118052921 >> v2:172.21.15.177:3300/0 conn(0x7f25dc15fc10 0x7f25dc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392680220 0 0) 0x7f25dc0b46c0 con 0x7f25dc0b3f70 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25bc003520 con 0x7f25dc0b3f70 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3349944571 0 0) 0x7f25bc003520 con 0x7f25dc0b3f70 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 >> v2:172.21.15.177:3300/0 conn(0x7f25dc15fc10 msgr2=0x7f25dc070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 --2- 172.21.15.114:0/118052921 >> v2:172.21.15.177:3300/0 conn(0x7f25dc15fc10 0x7f25dc070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 >> v1:172.21.15.177:6789/0 conn(0x7f25dc15fff0 legacy=0x7f25dc160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc09dd80 con 0x7f25dc0b3f70 2024-03-06T07:32:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f25d40019f0 con 0x7f25dc0b3f70 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 >> v1:172.21.15.114:6789/0 conn(0x7f25dc0b3f70 legacy=0x7f25dc0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.781+0000 7f25daa00640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc0039b0 0x7f25bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25daa00640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25bc004800 0x7f25bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25daa00640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25bc0051c0 0x7f25bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc072c90 con 0x7f25bc004800 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc00c060 con 0x7f25bc0051c0 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc00c260 con 0x7f25bc0039b0 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25d2a00640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25bc004800 0x7f25bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25db400640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc0039b0 0x7f25bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25d2a00640 1 -- 172.21.15.114:0/118052921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc0039b0 msgr2=0x7f25bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25d2a00640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc0039b0 0x7f25bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25d2a00640 1 -- 172.21.15.114:0/118052921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25bc0051c0 msgr2=0x7f25bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25d2a00640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25bc0051c0 0x7f25bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25d2a00640 1 -- 172.21.15.114:0/118052921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc003520 con 0x7f25bc004800 2024-03-06T07:32:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.782+0000 7f25dbe00640 1 -- 172.21.15.114:0/118052921 reap_dead start 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.783+0000 7f25d2a00640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25bc004800 0x7f25bc004c80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f25c4001750 tx=0x7f25bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c794698ea2f6425 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4001370 con 0x7f25bc004800 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c40052d0 con 0x7f25bc004800 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25daa00640 1 -- 172.21.15.114:0/118052921 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4003830 con 0x7f25bc004800 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25e2600640 1 -- 172.21.15.114:0/118052921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25bc004800 msgr2=0x7f25bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25e2600640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25bc004800 0x7f25bc004c80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f25c4001750 tx=0x7f25bc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25e2600640 1 -- 172.21.15.114:0/118052921 shutdown_connections 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25e2600640 1 --2- 172.21.15.114:0/118052921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25bc004800 0x7f25bc004c80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.784+0000 7f25e2600640 1 -- 172.21.15.114:0/118052921 >> 172.21.15.114:0/118052921 conn(0x7f25dc0b1940 msgr2=0x7f25dc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.785+0000 7f25e2600640 1 -- 172.21.15.114:0/118052921 shutdown_connections 2024-03-06T07:32:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.785+0000 7f25e2600640 1 -- 172.21.15.114:0/118052921 wait complete. 2024-03-06T07:32:46.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.786+0000 7f25e2600640 1 Processor -- start 2024-03-06T07:32:46.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.786+0000 7f25e2600640 1 -- start start 2024-03-06T07:32:46.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.787+0000 7f25e2600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc004800 0x7f25dc155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.787+0000 7f25e2600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc159010 0x7f25dc06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.787+0000 7f25e2600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25dc06e7a0 0x7f25dc0cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.787+0000 7f25e2600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc15c460 con 0x7f25dc159010 2024-03-06T07:32:46.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.787+0000 7f25e2600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc15c2e0 con 0x7f25dc06e7a0 2024-03-06T07:32:46.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.787+0000 7f25e2600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc070ff0 con 0x7f25bc004800 2024-03-06T07:32:46.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc159010 0x7f25dc06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc159010 0x7f25dc06e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51768/0 (socket says 172.21.15.114:51768) 2024-03-06T07:32:46.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 -- 172.21.15.114:0/1781971826 learned_addr learned my addr 172.21.15.114:0/1781971826 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:46.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25db400640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc004800 0x7f25dc155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25dbe00640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25dc06e7a0 0x7f25dc0cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 -- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc004800 msgr2=0x7f25dc155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc004800 0x7f25dc155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 -- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25dc06e7a0 msgr2=0x7f25dc0cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25dc06e7a0 0x7f25dc0cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.788+0000 7f25d2a00640 1 -- 172.21.15.114:0/1781971826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc1564f0 con 0x7f25dc159010 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.789+0000 7f25d2a00640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc159010 0x7f25dc06e260 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f25c4005040 tx=0x7f25c4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.789+0000 7f25d3e00640 1 -- 172.21.15.114:0/1781971826 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4041070 con 0x7f25dc159010 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.789+0000 7f25d3e00640 1 -- 172.21.15.114:0/1781971826 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c4005480 con 0x7f25dc159010 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.789+0000 7f25d3e00640 1 -- 172.21.15.114:0/1781971826 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4001370 con 0x7f25dc159010 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.789+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc1595a0 con 0x7f25dc159010 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.789+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc159b60 con 0x7f25dc159010 2024-03-06T07:32:46.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.791+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc04f330 con 0x7f25dc159010 2024-03-06T07:32:46.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.792+0000 7f25d3e00640 1 -- 172.21.15.114:0/1781971826 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25c4044070 con 0x7f25dc159010 2024-03-06T07:32:46.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.793+0000 7f25d3e00640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f25b806b4d0 0x7f25b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.793+0000 7f25d3e00640 1 -- 172.21.15.114:0/1781971826 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f25c40abcd0 con 0x7f25dc159010 2024-03-06T07:32:46.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.794+0000 7f25db400640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f25b806b4d0 0x7f25b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.794+0000 7f25db400640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f25b806b4d0 0x7f25b806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009a90 tx=0x7f25cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.798+0000 7f25d3e00640 1 -- 172.21.15.114:0/1781971826 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25c407aa90 con 0x7f25dc159010 2024-03-06T07:32:46.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.996+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f25dc06ef30 con 0x7f25dc159010 2024-03-06T07:32:46.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.996+0000 7f25d3e00640 1 -- 172.21.15.114:0/1781971826 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f25c407a390 con 0x7f25dc159010 2024-03-06T07:32:46.998 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f25b806b4d0 msgr2=0x7f25b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f25b806b4d0 0x7f25b806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009a90 tx=0x7f25cc008040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc159010 msgr2=0x7f25dc06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc159010 0x7f25dc06e260 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f25c4005040 tx=0x7f25c4001120 comp rx=0 tx=0).stop 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 shutdown_connections 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25dc06e7a0 0x7f25dc0cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f25b806b4d0 0x7f25b806d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc159010 0x7f25dc06e260 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 --2- 172.21.15.114:0/1781971826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f25bc004800 0x7f25dc155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 >> 172.21.15.114:0/1781971826 conn(0x7f25dc0b1940 msgr2=0x7f25dc0adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 shutdown_connections 2024-03-06T07:32:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:46.999+0000 7f25e2600640 1 -- 172.21.15.114:0/1781971826 wait complete. 2024-03-06T07:32:47.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:47.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.201+0000 7fbc6e000640 1 Processor -- start 2024-03-06T07:32:47.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.202+0000 7fbc6e000640 1 -- start start 2024-03-06T07:32:47.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.204+0000 7fbc6e000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc681581d0 0x7fbc680cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.204+0000 7fbc6e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc6806e470 0x7fbc6806d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.204+0000 7fbc6e000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6815b9e0 con 0x7fbc6806d9b0 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.204+0000 7fbc6e000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc680cbff0 con 0x7fbc6806e470 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.204+0000 7fbc6e000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc680cc220 con 0x7fbc681581d0 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.204+0000 7fbc5fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc6806e470 0x7fbc6806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.205+0000 7fbc5fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc6806e470 0x7fbc6806d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51784/0 (socket says 172.21.15.114:51784) 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.205+0000 7fbc5fe00640 1 -- 172.21.15.114:0/1707705061 learned_addr learned my addr 172.21.15.114:0/1707705061 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.205+0000 7fbc67400640 1 --2- 172.21.15.114:0/1707705061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc681581d0 0x7fbc680cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.205+0000 7fbc5fe00640 1 -- 172.21.15.114:0/1707705061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc681581d0 msgr2=0x7fbc680cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.205+0000 7fbc5fe00640 1 --2- 172.21.15.114:0/1707705061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc681581d0 0x7fbc680cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.205+0000 7fbc5fe00640 1 -- 172.21.15.114:0/1707705061 >> v1:172.21.15.177:6789/0 conn(0x7fbc6806d9b0 legacy=0x7fbc6815e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.205+0000 7fbc5fe00640 1 -- 172.21.15.114:0/1707705061 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6815ac20 con 0x7fbc6806e470 2024-03-06T07:32:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.206+0000 7fbc5fe00640 1 --2- 172.21.15.114:0/1707705061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806e470 0x7fbc6806d470 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009870 tx=0x7fbc5002ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920618a154602483 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.206+0000 7fbc66a00640 1 -- 172.21.15.114:0/1707705061 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50030820 con 0x7fbc6806e470 2024-03-06T07:32:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.206+0000 7fbc66a00640 1 -- 172.21.15.114:0/1707705061 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc500309c0 con 0x7fbc6806e470 2024-03-06T07:32:47.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.206+0000 7fbc66a00640 1 -- 172.21.15.114:0/1707705061 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50030d90 con 0x7fbc6806e470 2024-03-06T07:32:47.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 -- 172.21.15.114:0/1707705061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806e470 msgr2=0x7fbc6806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 --2- 172.21.15.114:0/1707705061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806e470 0x7fbc6806d470 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009870 tx=0x7fbc5002ca00 comp rx=0 tx=0).stop 2024-03-06T07:32:47.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 -- 172.21.15.114:0/1707705061 shutdown_connections 2024-03-06T07:32:47.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 --2- 172.21.15.114:0/1707705061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806e470 0x7fbc6806d470 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 --2- 172.21.15.114:0/1707705061 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc681581d0 0x7fbc680cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 -- 172.21.15.114:0/1707705061 >> 172.21.15.114:0/1707705061 conn(0x7fbc680cc4f0 msgr2=0x7fbc680cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 -- 172.21.15.114:0/1707705061 shutdown_connections 2024-03-06T07:32:47.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.207+0000 7fbc6e000640 1 -- 172.21.15.114:0/1707705061 wait complete. 2024-03-06T07:32:47.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.209+0000 7fbc6e000640 1 Processor -- start 2024-03-06T07:32:47.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.209+0000 7fbc6e000640 1 -- start start 2024-03-06T07:32:47.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc6e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806d9b0 0x7fbc680b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc6e000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc6806e470 0x7fbc680b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc6e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc681581d0 0x7fbc680b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc6e000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6815b4c0 con 0x7fbc6806d9b0 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc6e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6815b1e0 con 0x7fbc681581d0 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc6e000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc680cb250 con 0x7fbc6806e470 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806d9b0 0x7fbc680b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806d9b0 0x7fbc680b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51796/0 (socket says 172.21.15.114:51796) 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.210+0000 7fbc67400640 1 -- 172.21.15.114:0/3262747776 learned_addr learned my addr 172.21.15.114:0/3262747776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc5fe00640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc6806e470 0x7fbc680b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67400640 1 -- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc6806e470 msgr2=0x7fbc680b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67400640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc6806e470 0x7fbc680b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67400640 1 -- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc681581d0 msgr2=0x7fbc680b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67e00640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc681581d0 0x7fbc680b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67400640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc681581d0 0x7fbc680b0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67400640 1 -- 172.21.15.114:0/3262747776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680b3720 con 0x7fbc6806d9b0 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67e00640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc681581d0 0x7fbc680b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc67400640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806d9b0 0x7fbc680b3040 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800d5f0 tx=0x7fbc5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc5f400640 1 -- 172.21.15.114:0/3262747776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc580125f0 con 0x7fbc6806d9b0 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc5f400640 1 -- 172.21.15.114:0/3262747776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc58019070 con 0x7fbc6806d9b0 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.211+0000 7fbc5f400640 1 -- 172.21.15.114:0/3262747776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc580129d0 con 0x7fbc6806d9b0 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.212+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc680b3a00 con 0x7fbc6806d9b0 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.212+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc680afb60 con 0x7fbc6806d9b0 2024-03-06T07:32:47.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.213+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6804f330 con 0x7fbc6806d9b0 2024-03-06T07:32:47.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.215+0000 7fbc5f400640 1 -- 172.21.15.114:0/3262747776 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc58012b70 con 0x7fbc6806d9b0 2024-03-06T07:32:47.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.216+0000 7fbc5f400640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc4406b4d0 0x7fbc4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.216+0000 7fbc5f400640 1 -- 172.21.15.114:0/3262747776 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbc5808a680 con 0x7fbc6806d9b0 2024-03-06T07:32:47.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.216+0000 7fbc5fe00640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc4406b4d0 0x7fbc4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.217+0000 7fbc5fe00640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc4406b4d0 0x7fbc4406d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbc50037000 tx=0x7fbc5003a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.220+0000 7fbc5f400640 1 -- 172.21.15.114:0/3262747776 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc58059440 con 0x7fbc6806d9b0 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.417+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbc680cd1c0 con 0x7fbc6806d9b0 2024-03-06T07:32:47.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.418+0000 7fbc5f400640 1 -- 172.21.15.114:0/3262747776 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbc58058d40 con 0x7fbc6806d9b0 2024-03-06T07:32:47.421 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:32:47.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.421+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc4406b4d0 msgr2=0x7fbc4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.421+0000 7fbc6e000640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc4406b4d0 0x7fbc4406d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fbc50037000 tx=0x7fbc5003a040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.421+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806d9b0 msgr2=0x7fbc680b3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.421+0000 7fbc6e000640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806d9b0 0x7fbc680b3040 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800d5f0 tx=0x7fbc5800dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 shutdown_connections 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc681581d0 0x7fbc680b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc6806e470 0x7fbc680b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc4406b4d0 0x7fbc4406d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 --2- 172.21.15.114:0/3262747776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc6806d9b0 0x7fbc680b3040 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 >> 172.21.15.114:0/3262747776 conn(0x7fbc680cc4f0 msgr2=0x7fbc68156580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 shutdown_connections 2024-03-06T07:32:47.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.422+0000 7fbc6e000640 1 -- 172.21.15.114:0/3262747776 wait complete. 2024-03-06T07:32:47.440 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:32:47.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:47.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:47.527 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:32:47.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:47.527 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:47.563 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:32:47.563 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:47.621 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:47.621 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:47.621 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:47.660 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:47.660 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:32:47.661 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:47.716 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:47.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:47.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.912+0000 7f0b80a00640 1 Processor -- start 2024-03-06T07:32:47.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.912+0000 7f0b80a00640 1 -- start start 2024-03-06T07:32:47.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.913+0000 7f0b80a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c0aedc0 0x7f0b7c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.913+0000 7f0b80a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0b7c0cc800 0x7f0b7c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.913+0000 7f0b80a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c15c240 con 0x7f0b7c1561f0 2024-03-06T07:32:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.913+0000 7f0b80a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c15ce30 con 0x7f0b7c0cc800 2024-03-06T07:32:47.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.913+0000 7f0b80a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c15c500 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.913+0000 7f0b72a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c0aedc0 0x7f0b7c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.914+0000 7f0b72a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c0aedc0 0x7f0b7c0cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58048/0 (socket says 172.21.15.114:58048) 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.914+0000 7f0b72a00640 1 -- 172.21.15.114:0/2712654910 learned_addr learned my addr 172.21.15.114:0/2712654910 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.914+0000 7f0b7be00640 1 --2- 172.21.15.114:0/2712654910 >> v2:172.21.15.177:3300/0 conn(0x7f0b7c0cc800 0x7f0b7c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.914+0000 7f0b72a00640 1 -- 172.21.15.114:0/2712654910 >> v2:172.21.15.177:3300/0 conn(0x7f0b7c0cc800 msgr2=0x7f0b7c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.914+0000 7f0b72a00640 1 --2- 172.21.15.114:0/2712654910 >> v2:172.21.15.177:3300/0 conn(0x7f0b7c0cc800 0x7f0b7c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.914+0000 7f0b72a00640 1 -- 172.21.15.114:0/2712654910 >> v1:172.21.15.177:6789/0 conn(0x7f0b7c1561f0 legacy=0x7f0b7c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.914+0000 7f0b72a00640 1 -- 172.21.15.114:0/2712654910 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c15c7e0 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.915+0000 7f0b72a00640 1 --2- 172.21.15.114:0/2712654910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c0aedc0 0x7f0b7c0cc2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009870 tx=0x7f0b6402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=468ea29f6277451 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.916+0000 7f0b7aa00640 1 -- 172.21.15.114:0/2712654910 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64034070 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.916+0000 7f0b7aa00640 1 -- 172.21.15.114:0/2712654910 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b64030840 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.916+0000 7f0b7aa00640 1 -- 172.21.15.114:0/2712654910 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64030c10 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.916+0000 7f0b80a00640 1 -- 172.21.15.114:0/2712654910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c0aedc0 msgr2=0x7f0b7c0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.916+0000 7f0b80a00640 1 --2- 172.21.15.114:0/2712654910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c0aedc0 0x7f0b7c0cc2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009870 tx=0x7f0b6402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:47.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.917+0000 7f0b80a00640 1 -- 172.21.15.114:0/2712654910 shutdown_connections 2024-03-06T07:32:47.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.917+0000 7f0b80a00640 1 --2- 172.21.15.114:0/2712654910 >> v2:172.21.15.177:3300/0 conn(0x7f0b7c0cc800 0x7f0b7c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.917+0000 7f0b80a00640 1 --2- 172.21.15.114:0/2712654910 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c0aedc0 0x7f0b7c0cc2c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.917+0000 7f0b80a00640 1 -- 172.21.15.114:0/2712654910 >> 172.21.15.114:0/2712654910 conn(0x7f0b7c06afc0 msgr2=0x7f0b7c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.917+0000 7f0b80a00640 1 -- 172.21.15.114:0/2712654910 shutdown_connections 2024-03-06T07:32:47.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.917+0000 7f0b80a00640 1 -- 172.21.15.114:0/2712654910 wait complete. 2024-03-06T07:32:47.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.918+0000 7f0b80a00640 1 Processor -- start 2024-03-06T07:32:47.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.918+0000 7f0b80a00640 1 -- start start 2024-03-06T07:32:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.919+0000 7f0b80a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b7c0aedc0 0x7f0b7c1f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.919+0000 7f0b80a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c0cc800 0x7f0b7c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.919+0000 7f0b80a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c1561f0 0x7f0b7c0adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.919+0000 7f0b80a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c1565d0 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.919+0000 7f0b80a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0ccc90 con 0x7f0b7c0cc800 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.919+0000 7f0b80a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c156750 con 0x7f0b7c1561f0 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b7c0aedc0 0x7f0b7c1f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b7c0aedc0 0x7f0b7c1f40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37394/0 (socket says 172.21.15.114:37394) 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 -- 172.21.15.114:0/1590978140 learned_addr learned my addr 172.21.15.114:0/1590978140 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 -- 172.21.15.114:0/1590978140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c1561f0 msgr2=0x7f0b7c0adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c1561f0 0x7f0b7c0adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 -- 172.21.15.114:0/1590978140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c0cc800 msgr2=0x7f0b7c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b72a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c0cc800 0x7f0b7c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c0cc800 0x7f0b7c0ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 -- 172.21.15.114:0/1590978140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c1f8020 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.920+0000 7f0b7b400640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b7c0aedc0 0x7f0b7c1f40c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c340 tx=0x7f0b6c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.921+0000 7f0b73e00640 1 -- 172.21.15.114:0/1590978140 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0052d0 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.921+0000 7f0b73e00640 1 -- 172.21.15.114:0/1590978140 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c005490 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.921+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c1f8300 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.921+0000 7f0b73e00640 1 -- 172.21.15.114:0/1590978140 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c007260 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.921+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c1f88c0 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.922+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c04f330 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.923+0000 7f0b73e00640 1 -- 172.21.15.114:0/1590978140 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b6c007400 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.924+0000 7f0b73e00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0b5806b4d0 0x7f0b5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.924+0000 7f0b73e00640 1 -- 172.21.15.114:0/1590978140 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0b6c089610 con 0x7f0b7c0aedc0 2024-03-06T07:32:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.924+0000 7f0b72a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0b5806b4d0 0x7f0b5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.925+0000 7f0b72a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0b5806b4d0 0x7f0b5806d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0b64036000 tx=0x7f0b6403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:47.929+0000 7f0b73e00640 1 -- 172.21.15.114:0/1590978140 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b6c058470 con 0x7f0b7c0aedc0 2024-03-06T07:32:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.161+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b7c0b2b10 con 0x7f0b7c0aedc0 2024-03-06T07:32:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.162+0000 7f0b73e00640 1 -- 172.21.15.114:0/1590978140 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f0b6c057d70 con 0x7f0b7c0aedc0 2024-03-06T07:32:48.165 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:48.165 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:17.782119+0000","modified":"2024-03-06T07:32:20.854485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2399499569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2399499569},{"type":"v1","addr":"172.21.15.114:6837","nonce":2399499569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:32:48.165 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 52 2024-03-06T07:32:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0b5806b4d0 msgr2=0x7f0b5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0b5806b4d0 0x7f0b5806d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0b64036000 tx=0x7f0b6403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b7c0aedc0 msgr2=0x7f0b7c1f40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b7c0aedc0 0x7f0b7c1f40c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c340 tx=0x7f0b6c00c800 comp rx=0 tx=0).stop 2024-03-06T07:32:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 shutdown_connections 2024-03-06T07:32:48.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0b7c1561f0 0x7f0b7c0adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b7c0cc800 0x7f0b7c0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0b5806b4d0 0x7f0b5806d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 --2- 172.21.15.114:0/1590978140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b7c0aedc0 0x7f0b7c1f40c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.164+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 >> 172.21.15.114:0/1590978140 conn(0x7f0b7c06afc0 msgr2=0x7f0b7c06e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.165+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 shutdown_connections 2024-03-06T07:32:48.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.165+0000 7f0b80a00640 1 -- 172.21.15.114:0/1590978140 wait complete. 2024-03-06T07:32:48.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.367+0000 7f6fb4a00640 1 Processor -- start 2024-03-06T07:32:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.367+0000 7f6fb4a00640 1 -- start start 2024-03-06T07:32:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fb4a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6fb006ee40 0x7f6fb00b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fb4a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb006f2f0 con 0x7f6fb0155230 2024-03-06T07:32:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fb4a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb015b3e0 con 0x7f6fb015bec0 2024-03-06T07:32:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fb4a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb006f5d0 con 0x7f6fb006ee40 2024-03-06T07:32:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fae800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6fb0155230 0x7f6fb006e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43750/0 (socket says 172.21.15.114:43750) 2024-03-06T07:32:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fae800640 1 -- 172.21.15.114:0/759453279 learned_addr learned my addr 172.21.15.114:0/759453279 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546448426 0 0) 0x7f6fb006f2f0 con 0x7f6fb0155230 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f94003520 con 0x7f6fb0155230 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.369+0000 7f6fa5e00640 1 --2- 172.21.15.114:0/759453279 >> v2:172.21.15.177:3300/0 conn(0x7f6fb006ee40 0x7f6fb00b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1064813629 0 0) 0x7f6f94003520 con 0x7f6fb0155230 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 >> v2:172.21.15.177:3300/0 conn(0x7f6fb006ee40 msgr2=0x7f6fb00b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 --2- 172.21.15.114:0/759453279 >> v2:172.21.15.177:3300/0 conn(0x7f6fb006ee40 0x7f6fb00b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 >> v1:172.21.15.177:6789/0 conn(0x7f6fb015bec0 legacy=0x7f6fb015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb015b6c0 con 0x7f6fb0155230 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6fa00019f0 con 0x7f6fb0155230 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 >> v1:172.21.15.114:6789/0 conn(0x7f6fb0155230 legacy=0x7f6fb006e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94003a00 0x7f6f94003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f94004800 0x7f6f94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f94005370 0x7f6f9400b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb006f5d0 con 0x7f6f94005370 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9400c2c0 con 0x7f6f94003a00 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9400c4c0 con 0x7f6f94004800 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6faf200640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f94005370 0x7f6f9400b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fae800640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94003a00 0x7f6f94003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6faf200640 1 -- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f94004800 msgr2=0x7f6f94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6fa5e00640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f94004800 0x7f6f94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6faf200640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6f94004800 0x7f6f94004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6faf200640 1 -- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94003a00 msgr2=0x7f6f94003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.370+0000 7f6faf200640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94003a00 0x7f6f94003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.371+0000 7f6faf200640 1 -- 172.21.15.114:0/759453279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94003520 con 0x7f6f94005370 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.372+0000 7f6faf200640 1 -- 172.21.15.114:0/759453279 reap_dead start 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.372+0000 7f6faf200640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f94005370 0x7f6f9400b9e0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8003db0 tx=0x7f6f94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db02842e9861b3ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.372+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8037070 con 0x7f6f94005370 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.372+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa8032600 con 0x7f6f94005370 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.372+0000 7f6fade00640 1 -- 172.21.15.114:0/759453279 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8003980 con 0x7f6f94005370 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.373+0000 7f6fb4a00640 1 -- 172.21.15.114:0/759453279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f94005370 msgr2=0x7f6f9400b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.373+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f94005370 0x7f6f9400b9e0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8003db0 tx=0x7f6f94005830 comp rx=0 tx=0).stop 2024-03-06T07:32:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.373+0000 7f6fb4a00640 1 -- 172.21.15.114:0/759453279 shutdown_connections 2024-03-06T07:32:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.373+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/759453279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f94005370 0x7f6f9400b9e0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.373+0000 7f6fb4a00640 1 -- 172.21.15.114:0/759453279 >> 172.21.15.114:0/759453279 conn(0x7f6fb006c0d0 msgr2=0x7f6fb006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.373+0000 7f6fb4a00640 1 -- 172.21.15.114:0/759453279 shutdown_connections 2024-03-06T07:32:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.373+0000 7f6fb4a00640 1 -- 172.21.15.114:0/759453279 wait complete. 2024-03-06T07:32:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.375+0000 7f6fb4a00640 1 Processor -- start 2024-03-06T07:32:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.375+0000 7f6fb4a00640 1 -- start start 2024-03-06T07:32:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.376+0000 7f6fb4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94005370 0x7f6fb00cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.376+0000 7f6fb4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb0158450 0x7f6fb00ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.376+0000 7f6fb4a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fb014d7f0 0x7f6fb00aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.376+0000 7f6fb4a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0155610 con 0x7f6fb0158450 2024-03-06T07:32:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.376+0000 7f6fb4a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb006e0c0 con 0x7f6f94005370 2024-03-06T07:32:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.376+0000 7f6fb4a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb006e730 con 0x7f6fb014d7f0 2024-03-06T07:32:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb0158450 0x7f6fb00ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb0158450 0x7f6fb00ae8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37418/0 (socket says 172.21.15.114:37418) 2024-03-06T07:32:48.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 -- 172.21.15.114:0/1606318366 learned_addr learned my addr 172.21.15.114:0/1606318366 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 -- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fb014d7f0 msgr2=0x7f6fb00aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6faf200640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fb014d7f0 0x7f6fb00aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fae800640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94005370 0x7f6fb00cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fb014d7f0 0x7f6fb00aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 -- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94005370 msgr2=0x7f6fb00cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94005370 0x7f6fb00cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 -- 172.21.15.114:0/1606318366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00cf500 con 0x7f6fb0158450 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.377+0000 7f6fa5e00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb0158450 0x7f6fb00ae8b0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800d630 tx=0x7f6f9800daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.378+0000 7f6fa7400640 1 -- 172.21.15.114:0/1606318366 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98016070 con 0x7f6fb0158450 2024-03-06T07:32:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.378+0000 7f6fa7400640 1 -- 172.21.15.114:0/1606318366 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f98012870 con 0x7f6fb0158450 2024-03-06T07:32:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.378+0000 7f6fa7400640 1 -- 172.21.15.114:0/1606318366 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98012c40 con 0x7f6fb0158450 2024-03-06T07:32:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.378+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb00cf7e0 con 0x7f6fb0158450 2024-03-06T07:32:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.378+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb00cfda0 con 0x7f6fb0158450 2024-03-06T07:32:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.379+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb004f330 con 0x7f6fb0158450 2024-03-06T07:32:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.380+0000 7f6fa7400640 1 -- 172.21.15.114:0/1606318366 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f98019070 con 0x7f6fb0158450 2024-03-06T07:32:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.380+0000 7f6fa7400640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6f8c06b500 0x7f6f8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.381+0000 7f6fa7400640 1 -- 172.21.15.114:0/1606318366 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6f9808a460 con 0x7f6fb0158450 2024-03-06T07:32:48.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.381+0000 7f6fae800640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6f8c06b500 0x7f6f8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.381+0000 7f6fae800640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6f8c06b500 0x7f6f8c06d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6fa00013d0 tx=0x7f6fa0001510 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.385+0000 7f6fa7400640 1 -- 172.21.15.114:0/1606318366 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f98059190 con 0x7f6fb0158450 2024-03-06T07:32:48.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.582+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fb00b0aa0 con 0x7f6fb0158450 2024-03-06T07:32:48.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.585+0000 7f6fa7400640 1 -- 172.21.15.114:0/1606318366 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16757 (secure 0 0 0) 0x7f6f98058a90 con 0x7f6fb0158450 2024-03-06T07:32:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:48.588 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":82,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:37.454277+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:15.626899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:16.804201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.c","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:48.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6f8c06b500 msgr2=0x7f6f8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6f8c06b500 0x7f6f8c06d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6fa00013d0 tx=0x7f6fa0001510 comp rx=0 tx=0).stop 2024-03-06T07:32:48.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb0158450 msgr2=0x7f6fb00ae8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb0158450 0x7f6fb00ae8b0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800d630 tx=0x7f6f9800daf0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 shutdown_connections 2024-03-06T07:32:48.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6fb014d7f0 0x7f6fb00aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6f8c06b500 0x7f6f8c06d9c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb0158450 0x7f6fb00ae8b0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 --2- 172.21.15.114:0/1606318366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f94005370 0x7f6fb00cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.587+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 >> 172.21.15.114:0/1606318366 conn(0x7f6fb006c0d0 msgr2=0x7f6fb015d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.588+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 shutdown_connections 2024-03-06T07:32:48.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.588+0000 7f6fb4a00640 1 -- 172.21.15.114:0/1606318366 wait complete. 2024-03-06T07:32:48.607 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:15.626899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:16.804201+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:48.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:48.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.809+0000 7f5162600640 1 Processor -- start 2024-03-06T07:32:48.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.809+0000 7f5162600640 1 -- start start 2024-03-06T07:32:48.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.811+0000 7f5162600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c15f490 0x7f515c0d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.811+0000 7f5162600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f515c0ced50 0x7f515c0cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.811+0000 7f5162600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515c0d4c40 con 0x7f515c0b4de0 2024-03-06T07:32:48.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.811+0000 7f5162600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515c0d49f0 con 0x7f515c0ced50 2024-03-06T07:32:48.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.811+0000 7f5162600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0d24b0 con 0x7f515c15f490 2024-03-06T07:32:48.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.811+0000 7f5153e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f515c0b4de0 0x7f515c0d0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43760/0 (socket says 172.21.15.114:43760) 2024-03-06T07:32:48.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.811+0000 7f5153e00640 1 -- 172.21.15.114:0/3008436828 learned_addr learned my addr 172.21.15.114:0/3008436828 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515b400640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c15f490 0x7f515c0d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515be00640 1 --2- 172.21.15.114:0/3008436828 >> v2:172.21.15.177:3300/0 conn(0x7f515c0ced50 0x7f515c0cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400146320 0 0) 0x7f515c0d4c40 con 0x7f515c0b4de0 2024-03-06T07:32:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5140003520 con 0x7f515c0b4de0 2024-03-06T07:32:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145638036 0 0) 0x7f5140003520 con 0x7f515c0b4de0 2024-03-06T07:32:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c15f490 msgr2=0x7f515c0d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c15f490 0x7f515c0d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 >> v2:172.21.15.177:3300/0 conn(0x7f515c0ced50 msgr2=0x7f515c0cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 --2- 172.21.15.114:0/3008436828 >> v2:172.21.15.177:3300/0 conn(0x7f515c0ced50 0x7f515c0cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c0d2700 con 0x7f515c0b4de0 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5144001670 con 0x7f515c0b4de0 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 >> v1:172.21.15.114:6789/0 conn(0x7f515c0b4de0 legacy=0x7f515c0d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51400039b0 0x7f5140003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f5140004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c15f490 0x7f51400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0d49f0 con 0x7f5140004800 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0d24b0 con 0x7f51400039b0 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.812+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514000b9c0 con 0x7f515c15f490 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.813+0000 7f5153e00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f5140004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.813+0000 7f5153e00640 1 -- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c15f490 msgr2=0x7f51400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.813+0000 7f5153e00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c15f490 0x7f51400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.813+0000 7f5153e00640 1 -- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51400039b0 msgr2=0x7f5140003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.813+0000 7f5153e00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51400039b0 0x7f5140003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.813+0000 7f5153e00640 1 -- 172.21.15.114:0/3008436828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5140003520 con 0x7f5140004800 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.814+0000 7f5153e00640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f5140004c80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f514400c210 tx=0x7f5140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496e049d062afe4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.814+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144007d50 con 0x7f5140004800 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.814+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5144030070 con 0x7f5140004800 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.814+0000 7f515aa00640 1 -- 172.21.15.114:0/3008436828 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514402cd20 con 0x7f5140004800 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.815+0000 7f5162600640 1 -- 172.21.15.114:0/3008436828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 msgr2=0x7f5140004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.815+0000 7f5162600640 1 --2- 172.21.15.114:0/3008436828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f5140004c80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f514400c210 tx=0x7f5140005830 comp rx=0 tx=0).stop 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.815+0000 7f515be00640 1 -- 172.21.15.114:0/3008436828 reap_dead start 2024-03-06T07:32:48.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.815+0000 7f5162600640 1 -- 172.21.15.114:0/3008436828 shutdown_connections 2024-03-06T07:32:48.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.815+0000 7f5162600640 1 -- 172.21.15.114:0/3008436828 >> 172.21.15.114:0/3008436828 conn(0x7f515c0d3ed0 msgr2=0x7f515c0d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.815+0000 7f5162600640 1 -- 172.21.15.114:0/3008436828 shutdown_connections 2024-03-06T07:32:48.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.815+0000 7f5162600640 1 -- 172.21.15.114:0/3008436828 wait complete. 2024-03-06T07:32:48.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.817+0000 7f5162600640 1 Processor -- start 2024-03-06T07:32:48.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.817+0000 7f5162600640 1 -- start start 2024-03-06T07:32:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.818+0000 7f5162600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f515c0b8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.818+0000 7f5162600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c14e3e0 0x7f515c14cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.818+0000 7f5162600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c14c280 0x7f515c0b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.818+0000 7f5162600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c15fd50 con 0x7f5140004800 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.818+0000 7f5162600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0d2cc0 con 0x7f515c14e3e0 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.818+0000 7f5162600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c15f870 con 0x7f515c14c280 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.818+0000 7f515b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f515c0b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f515c0b8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37438/0 (socket says 172.21.15.114:37438) 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 -- 172.21.15.114:0/2605183532 learned_addr learned my addr 172.21.15.114:0/2605183532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515be00640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c14c280 0x7f515c0b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f5153e00640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c14e3e0 0x7f515c14cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 -- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c14c280 msgr2=0x7f515c0b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c14c280 0x7f515c0b8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 -- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c14e3e0 msgr2=0x7f515c14cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c14e3e0 0x7f515c14cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 -- 172.21.15.114:0/2605183532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c14d340 con 0x7f5140004800 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.819+0000 7f515b400640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f515c0b8020 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f514c00e7d0 tx=0x7f514c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.820+0000 7f5153400640 1 -- 172.21.15.114:0/2605183532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c017070 con 0x7f5140004800 2024-03-06T07:32:48.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.820+0000 7f5153400640 1 -- 172.21.15.114:0/2605183532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c00c940 con 0x7f5140004800 2024-03-06T07:32:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.820+0000 7f5153400640 1 -- 172.21.15.114:0/2605183532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c00cd10 con 0x7f5140004800 2024-03-06T07:32:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.820+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c14d620 con 0x7f5140004800 2024-03-06T07:32:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.820+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c14dbe0 con 0x7f5140004800 2024-03-06T07:32:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.821+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c04f330 con 0x7f5140004800 2024-03-06T07:32:48.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.822+0000 7f5153400640 1 -- 172.21.15.114:0/2605183532 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f514c01a070 con 0x7f5140004800 2024-03-06T07:32:48.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.823+0000 7f5153400640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f513806b500 0x7f513806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.823+0000 7f5153400640 1 -- 172.21.15.114:0/2605183532 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f514c08a390 con 0x7f5140004800 2024-03-06T07:32:48.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.823+0000 7f5153e00640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f513806b500 0x7f513806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.824+0000 7f5153e00640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f513806b500 0x7f513806d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5144034000 tx=0x7f5144007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:48.827+0000 7f5153400640 1 -- 172.21.15.114:0/2605183532 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f514c0591f0 con 0x7f5140004800 2024-03-06T07:32:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.023+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f515c08d640 con 0x7f5140004800 2024-03-06T07:32:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.024+0000 7f5153400640 1 -- 172.21.15.114:0/2605183532 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f514c058af0 con 0x7f5140004800 2024-03-06T07:32:49.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f513806b500 msgr2=0x7f513806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f513806b500 0x7f513806d9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5144034000 tx=0x7f5144007000 comp rx=0 tx=0).stop 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 msgr2=0x7f515c0b8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f515c0b8020 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f514c00e7d0 tx=0x7f514c00ec90 comp rx=0 tx=0).stop 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 shutdown_connections 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f515c14c280 0x7f515c0b8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f515c14e3e0 0x7f515c14cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f513806b500 0x7f513806d9c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 --2- 172.21.15.114:0/2605183532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5140004800 0x7f515c0b8020 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.026+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 >> 172.21.15.114:0/2605183532 conn(0x7f515c0d3ed0 msgr2=0x7f515c06c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.027+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 shutdown_connections 2024-03-06T07:32:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.027+0000 7f5162600640 1 -- 172.21.15.114:0/2605183532 wait complete. 2024-03-06T07:32:49.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:49.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.258+0000 7f00a3a00640 1 Processor -- start 2024-03-06T07:32:49.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.258+0000 7f00a3a00640 1 -- start start 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f00a3a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fc10 0x7f009c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f00a3a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f009c15fff0 0x7f009c160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f00a3a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0b46c0 con 0x7f009c0ce4a0 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f00a3a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009c0b49a0 con 0x7f009c15fff0 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f00a3a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c072c90 con 0x7f009c15fc10 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f0093e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f009c15fff0 0x7f009c160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f0093e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f009c15fff0 0x7f009c160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37454/0 (socket says 172.21.15.114:37454) 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.260+0000 7f0093e00640 1 -- 172.21.15.114:0/3629717123 learned_addr learned my addr 172.21.15.114:0/3629717123 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.261+0000 7f00a1000640 1 --2- 172.21.15.114:0/3629717123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fc10 0x7f009c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.261+0000 7f0093e00640 1 -- 172.21.15.114:0/3629717123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fc10 msgr2=0x7f009c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.261+0000 7f0093e00640 1 --2- 172.21.15.114:0/3629717123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fc10 0x7f009c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.261+0000 7f0093e00640 1 -- 172.21.15.114:0/3629717123 >> v1:172.21.15.177:6789/0 conn(0x7f009c0ce4a0 legacy=0x7f009c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.261+0000 7f0093e00640 1 -- 172.21.15.114:0/3629717123 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c160990 con 0x7f009c15fff0 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.262+0000 7f0093e00640 1 --2- 172.21.15.114:0/3629717123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fff0 0x7f009c160450 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f008402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de4e958079544d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.262+0000 7f009be00640 1 -- 172.21.15.114:0/3629717123 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084035020 con 0x7f009c15fff0 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.262+0000 7f009be00640 1 -- 172.21.15.114:0/3629717123 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084031640 con 0x7f009c15fff0 2024-03-06T07:32:49.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.262+0000 7f009be00640 1 -- 172.21.15.114:0/3629717123 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008402d040 con 0x7f009c15fff0 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.263+0000 7f00a3a00640 1 -- 172.21.15.114:0/3629717123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fff0 msgr2=0x7f009c160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.263+0000 7f00a3a00640 1 --2- 172.21.15.114:0/3629717123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fff0 0x7f009c160450 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f008402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.263+0000 7f00a3a00640 1 -- 172.21.15.114:0/3629717123 shutdown_connections 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.263+0000 7f00a3a00640 1 --2- 172.21.15.114:0/3629717123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fff0 0x7f009c160450 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.263+0000 7f00a3a00640 1 --2- 172.21.15.114:0/3629717123 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fc10 0x7f009c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.263+0000 7f00a3a00640 1 -- 172.21.15.114:0/3629717123 >> 172.21.15.114:0/3629717123 conn(0x7f009c0b1940 msgr2=0x7f009c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.263+0000 7f00a3a00640 1 -- 172.21.15.114:0/3629717123 shutdown_connections 2024-03-06T07:32:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.264+0000 7f00a3a00640 1 -- 172.21.15.114:0/3629717123 wait complete. 2024-03-06T07:32:49.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.265+0000 7f00a3a00640 1 Processor -- start 2024-03-06T07:32:49.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.265+0000 7f00a3a00640 1 -- start start 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.266+0000 7f00a3a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009c0ce4a0 0x7f009c14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.266+0000 7f00a3a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fc10 0x7f009c150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.266+0000 7f00a3a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fff0 0x7f009c0b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.266+0000 7f00a3a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c072a30 con 0x7f009c15fc10 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.266+0000 7f00a3a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c0728b0 con 0x7f009c0ce4a0 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.266+0000 7f00a3a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c15c2e0 con 0x7f009c15fff0 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.266+0000 7f0093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fc10 0x7f009c150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fc10 0x7f009c150cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37470/0 (socket says 172.21.15.114:37470) 2024-03-06T07:32:49.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 -- 172.21.15.114:0/4009704638 learned_addr learned my addr 172.21.15.114:0/4009704638 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 -- 172.21.15.114:0/4009704638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fff0 msgr2=0x7f009c0b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fff0 0x7f009c0b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 -- 172.21.15.114:0/4009704638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009c0ce4a0 msgr2=0x7f009c14e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009c0ce4a0 0x7f009c14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 -- 172.21.15.114:0/4009704638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c14f0c0 con 0x7f009c15fc10 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.267+0000 7f0093e00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fc10 0x7f009c150cb0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f0084000c00 tx=0x7f00840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.268+0000 7f0099600640 1 -- 172.21.15.114:0/4009704638 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084040070 con 0x7f009c15fc10 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.268+0000 7f0099600640 1 -- 172.21.15.114:0/4009704638 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00840399f0 con 0x7f009c15fc10 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.268+0000 7f0099600640 1 -- 172.21.15.114:0/4009704638 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084035040 con 0x7f009c15fc10 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.268+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c14f3a0 con 0x7f009c15fc10 2024-03-06T07:32:49.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.268+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c14f960 con 0x7f009c15fc10 2024-03-06T07:32:49.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.269+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009c0b2a10 con 0x7f009c15fc10 2024-03-06T07:32:49.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.270+0000 7f0099600640 1 -- 172.21.15.114:0/4009704638 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f008404b060 con 0x7f009c15fc10 2024-03-06T07:32:49.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.271+0000 7f0099600640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007806b4d0 0x7f007806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.271+0000 7f0099600640 1 -- 172.21.15.114:0/4009704638 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f00840aba30 con 0x7f009c15fc10 2024-03-06T07:32:49.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.272+0000 7f00a1000640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007806b4d0 0x7f007806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.272+0000 7f00a1000640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007806b4d0 0x7f007806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f008c0098f0 tx=0x7f008c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.276+0000 7f0099600640 1 -- 172.21.15.114:0/4009704638 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f009c0b2a10 con 0x7f009c15fc10 2024-03-06T07:32:49.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.489+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f009c04f330 con 0x7f009c15fc10 2024-03-06T07:32:49.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.490+0000 7f0099600640 1 -- 172.21.15.114:0/4009704638 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f008407a920 con 0x7f009c15fc10 2024-03-06T07:32:49.493 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007806b4d0 msgr2=0x7f007806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007806b4d0 0x7f007806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f008c0098f0 tx=0x7f008c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fc10 msgr2=0x7f009c150cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fc10 0x7f009c150cb0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f0084000c00 tx=0x7f00840046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 shutdown_connections 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f009c15fff0 0x7f009c0b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007806b4d0 0x7f007806d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009c15fc10 0x7f009c150cb0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 --2- 172.21.15.114:0/4009704638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009c0ce4a0 0x7f009c14e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 >> 172.21.15.114:0/4009704638 conn(0x7f009c0b1940 msgr2=0x7f009c0acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 shutdown_connections 2024-03-06T07:32:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.493+0000 7f00a3a00640 1 -- 172.21.15.114:0/4009704638 wait complete. 2024-03-06T07:32:49.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:49.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.703+0000 7fd324c00640 1 Processor -- start 2024-03-06T07:32:49.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.703+0000 7fd324c00640 1 -- start start 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.705+0000 7fd324c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd32015e130 0x7fd3200cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd324c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd3200d25b0 0x7fd3200c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd324c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3200d13e0 con 0x7fd3200cfaf0 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd324c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200cf120 con 0x7fd32015e130 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd324c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200cead0 con 0x7fd3200d25b0 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd31f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd32015e130 0x7fd3200cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd31f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd32015e130 0x7fd3200cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37472/0 (socket says 172.21.15.114:37472) 2024-03-06T07:32:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd31f400640 1 -- 172.21.15.114:0/2740331446 learned_addr learned my addr 172.21.15.114:0/2740331446 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:49.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd31fe00640 1 --2- 172.21.15.114:0/2740331446 >> v2:172.21.15.177:3300/0 conn(0x7fd3200d25b0 0x7fd3200c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.706+0000 7fd31f400640 1 -- 172.21.15.114:0/2740331446 >> v2:172.21.15.177:3300/0 conn(0x7fd3200d25b0 msgr2=0x7fd3200c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.707+0000 7fd31f400640 1 --2- 172.21.15.114:0/2740331446 >> v2:172.21.15.177:3300/0 conn(0x7fd3200d25b0 0x7fd3200c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.707+0000 7fd31f400640 1 -- 172.21.15.114:0/2740331446 >> v1:172.21.15.177:6789/0 conn(0x7fd3200cfaf0 legacy=0x7fd3200d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.707+0000 7fd31f400640 1 -- 172.21.15.114:0/2740331446 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200cedb0 con 0x7fd32015e130 2024-03-06T07:32:49.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.707+0000 7fd31fe00640 1 --2- 172.21.15.114:0/2740331446 >> v2:172.21.15.177:3300/0 conn(0x7fd3200d25b0 0x7fd3200c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.708+0000 7fd31f400640 1 --2- 172.21.15.114:0/2740331446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015e130 0x7fd3200cf5b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fd3200b58c0 tx=0x7fd31002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f77e64b3381f8ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.708+0000 7fd31e000640 1 -- 172.21.15.114:0/2740331446 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31002ca80 con 0x7fd32015e130 2024-03-06T07:32:49.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.708+0000 7fd31e000640 1 -- 172.21.15.114:0/2740331446 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31002cc20 con 0x7fd32015e130 2024-03-06T07:32:49.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.708+0000 7fd31e000640 1 -- 172.21.15.114:0/2740331446 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100343f0 con 0x7fd32015e130 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 -- 172.21.15.114:0/2740331446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015e130 msgr2=0x7fd3200cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 --2- 172.21.15.114:0/2740331446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015e130 0x7fd3200cf5b0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fd3200b58c0 tx=0x7fd31002bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 -- 172.21.15.114:0/2740331446 shutdown_connections 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 --2- 172.21.15.114:0/2740331446 >> v2:172.21.15.177:3300/0 conn(0x7fd3200d25b0 0x7fd3200c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 --2- 172.21.15.114:0/2740331446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd32015e130 0x7fd3200cf5b0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 -- 172.21.15.114:0/2740331446 >> 172.21.15.114:0/2740331446 conn(0x7fd32009d070 msgr2=0x7fd3200d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 -- 172.21.15.114:0/2740331446 shutdown_connections 2024-03-06T07:32:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.709+0000 7fd324c00640 1 -- 172.21.15.114:0/2740331446 wait complete. 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.711+0000 7fd324c00640 1 Processor -- start 2024-03-06T07:32:49.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.711+0000 7fd324c00640 1 -- start start 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd324c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3200cfaf0 0x7fd3200a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd324c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200d25b0 0x7fd3200b8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd324c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd32015e130 0x7fd32009dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd324c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200c8030 con 0x7fd3200d25b0 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd324c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200c7eb0 con 0x7fd3200cfaf0 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd324c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200c81b0 con 0x7fd32015e130 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200d25b0 0x7fd3200b8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200d25b0 0x7fd3200b8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37474/0 (socket says 172.21.15.114:37474) 2024-03-06T07:32:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 -- 172.21.15.114:0/3430684128 learned_addr learned my addr 172.21.15.114:0/3430684128 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31f400640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3200cfaf0 0x7fd3200a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31fe00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd32015e130 0x7fd32009dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 -- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd32015e130 msgr2=0x7fd32009dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd32015e130 0x7fd32009dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 -- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3200cfaf0 msgr2=0x7fd3200a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3200cfaf0 0x7fd3200a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.712+0000 7fd31ea00640 1 -- 172.21.15.114:0/3430684128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200a1890 con 0x7fd3200d25b0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.713+0000 7fd31ea00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200d25b0 0x7fd3200b8f40 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00b3c0 tx=0x7fd30c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.713+0000 7fd317400640 1 -- 172.21.15.114:0/3430684128 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c011830 con 0x7fd3200d25b0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.713+0000 7fd317400640 1 -- 172.21.15.114:0/3430684128 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c016070 con 0x7fd3200d25b0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.713+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3200a0ff0 con 0x7fd3200d25b0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.713+0000 7fd317400640 1 -- 172.21.15.114:0/3430684128 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c011c50 con 0x7fd3200d25b0 2024-03-06T07:32:49.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.713+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3200a15b0 con 0x7fd3200d25b0 2024-03-06T07:32:49.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.714+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32004f330 con 0x7fd3200d25b0 2024-03-06T07:32:49.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.715+0000 7fd317400640 1 -- 172.21.15.114:0/3430684128 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd30c01b460 con 0x7fd3200d25b0 2024-03-06T07:32:49.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.716+0000 7fd317400640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd30006b4d0 0x7fd30006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.716+0000 7fd317400640 1 -- 172.21.15.114:0/3430684128 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd30c089cf0 con 0x7fd3200d25b0 2024-03-06T07:32:49.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.717+0000 7fd31f400640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd30006b4d0 0x7fd30006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.717+0000 7fd31f400640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd30006b4d0 0x7fd30006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd310035000 tx=0x7fd310030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.721+0000 7fd317400640 1 -- 172.21.15.114:0/3430684128 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd30c020020 con 0x7fd3200d25b0 2024-03-06T07:32:49.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.920+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd32009d690 con 0x7fd3200d25b0 2024-03-06T07:32:49.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.921+0000 7fd317400640 1 -- 172.21.15.114:0/3430684128 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd30c058b40 con 0x7fd3200d25b0 2024-03-06T07:32:49.923 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:32:49.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.923+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd30006b4d0 msgr2=0x7fd30006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.923+0000 7fd324c00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd30006b4d0 0x7fd30006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd310035000 tx=0x7fd310030000 comp rx=0 tx=0).stop 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.923+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200d25b0 msgr2=0x7fd3200b8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.923+0000 7fd324c00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200d25b0 0x7fd3200b8f40 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00b3c0 tx=0x7fd30c00b880 comp rx=0 tx=0).stop 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 shutdown_connections 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd32015e130 0x7fd32009dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd30006b4d0 0x7fd30006d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3200d25b0 0x7fd3200b8f40 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 --2- 172.21.15.114:0/3430684128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3200cfaf0 0x7fd3200a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 >> 172.21.15.114:0/3430684128 conn(0x7fd32009d070 msgr2=0x7fd3200d45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 shutdown_connections 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:49.924+0000 7fd324c00640 1 -- 172.21.15.114:0/3430684128 wait complete. 2024-03-06T07:32:49.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:50.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.139+0000 7feb50a00640 1 Processor -- start 2024-03-06T07:32:50.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.140+0000 7feb50a00640 1 -- start start 2024-03-06T07:32:50.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.141+0000 7feb50a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7feb4c1561f0 0x7feb4c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.141+0000 7feb50a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb4c0aebf0 0x7feb4c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.141+0000 7feb50a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c15c240 con 0x7feb4c0cc800 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.141+0000 7feb50a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c15ce30 con 0x7feb4c0aebf0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.141+0000 7feb50a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c15c500 con 0x7feb4c1561f0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.141+0000 7feb43e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb4c0aebf0 0x7feb4c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.141+0000 7feb4b200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7feb4c0cc800 0x7feb4c0cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43772/0 (socket says 172.21.15.114:43772) 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.142+0000 7feb4b200640 1 -- 172.21.15.114:0/1830579477 learned_addr learned my addr 172.21.15.114:0/1830579477 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.142+0000 7feb4a800640 1 --2- 172.21.15.114:0/1830579477 >> v2:172.21.15.177:3300/0 conn(0x7feb4c1561f0 0x7feb4c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.142+0000 7feb43e00640 1 -- 172.21.15.114:0/1830579477 >> v2:172.21.15.177:3300/0 conn(0x7feb4c1561f0 msgr2=0x7feb4c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.142+0000 7feb43e00640 1 --2- 172.21.15.114:0/1830579477 >> v2:172.21.15.177:3300/0 conn(0x7feb4c1561f0 0x7feb4c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.142+0000 7feb43e00640 1 -- 172.21.15.114:0/1830579477 >> v1:172.21.15.114:6789/0 conn(0x7feb4c0cc800 legacy=0x7feb4c0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.142+0000 7feb43e00640 1 -- 172.21.15.114:0/1830579477 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c15c7e0 con 0x7feb4c0aebf0 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.143+0000 7feb43e00640 1 --2- 172.21.15.114:0/1830579477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0aebf0 0x7feb4c0cc2c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0b4db0 tx=0x7feb3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421f28cd0ee3fcae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.143+0000 7feb49e00640 1 -- 172.21.15.114:0/1830579477 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34035020 con 0x7feb4c0aebf0 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.143+0000 7feb49e00640 1 -- 172.21.15.114:0/1830579477 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34031830 con 0x7feb4c0aebf0 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.143+0000 7feb49e00640 1 -- 172.21.15.114:0/1830579477 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34031c00 con 0x7feb4c0aebf0 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 -- 172.21.15.114:0/1830579477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0aebf0 msgr2=0x7feb4c0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 --2- 172.21.15.114:0/1830579477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0aebf0 0x7feb4c0cc2c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0b4db0 tx=0x7feb3402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 -- 172.21.15.114:0/1830579477 shutdown_connections 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 --2- 172.21.15.114:0/1830579477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0aebf0 0x7feb4c0cc2c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 --2- 172.21.15.114:0/1830579477 >> v2:172.21.15.177:3300/0 conn(0x7feb4c1561f0 0x7feb4c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 -- 172.21.15.114:0/1830579477 >> 172.21.15.114:0/1830579477 conn(0x7feb4c06afc0 msgr2=0x7feb4c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 -- 172.21.15.114:0/1830579477 shutdown_connections 2024-03-06T07:32:50.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.144+0000 7feb50a00640 1 -- 172.21.15.114:0/1830579477 wait complete. 2024-03-06T07:32:50.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.146+0000 7feb50a00640 1 Processor -- start 2024-03-06T07:32:50.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.146+0000 7feb50a00640 1 -- start start 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb50a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4c0aebf0 0x7feb4c149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb50a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0cc800 0x7feb4c0cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb50a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb4c1561f0 0x7feb4c156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb50a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c1565d0 con 0x7feb4c0cc800 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb50a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0ccc50 con 0x7feb4c0aebf0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb50a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c156750 con 0x7feb4c1561f0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb43e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0cc800 0x7feb4c0cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb43e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0cc800 0x7feb4c0cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37492/0 (socket says 172.21.15.114:37492) 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.147+0000 7feb43e00640 1 -- 172.21.15.114:0/2633677931 learned_addr learned my addr 172.21.15.114:0/2633677931 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb4a800640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4c0aebf0 0x7feb4c149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb4b200640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb4c1561f0 0x7feb4c156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb43e00640 1 -- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb4c1561f0 msgr2=0x7feb4c156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb43e00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb4c1561f0 0x7feb4c156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb43e00640 1 -- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4c0aebf0 msgr2=0x7feb4c149200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb43e00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4c0aebf0 0x7feb4c149200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb43e00640 1 -- 172.21.15.114:0/2633677931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0af5f0 con 0x7feb4c0cc800 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb4a800640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4c0aebf0 0x7feb4c149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb4b200640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb4c1561f0 0x7feb4c156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb43e00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0cc800 0x7feb4c0cde50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7feb3402bee0 tx=0x7feb340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb42a00640 1 -- 172.21.15.114:0/2633677931 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34041070 con 0x7feb4c0cc800 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb42a00640 1 -- 172.21.15.114:0/2633677931 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3403f070 con 0x7feb4c0cc800 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c0ce8a0 con 0x7feb4c0cc800 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.148+0000 7feb42a00640 1 -- 172.21.15.114:0/2633677931 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34035040 con 0x7feb4c0cc800 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.149+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c0af8b0 con 0x7feb4c0cc800 2024-03-06T07:32:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.150+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c151cb0 con 0x7feb4c0cc800 2024-03-06T07:32:50.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.150+0000 7feb42a00640 1 -- 172.21.15.114:0/2633677931 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb34038020 con 0x7feb4c0cc800 2024-03-06T07:32:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.152+0000 7feb42a00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb2806b500 0x7feb2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.152+0000 7feb42a00640 1 -- 172.21.15.114:0/2633677931 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7feb340ab940 con 0x7feb4c0cc800 2024-03-06T07:32:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.152+0000 7feb4a800640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb2806b500 0x7feb2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.153+0000 7feb4a800640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb2806b500 0x7feb2806d9c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7feb3c006fd0 tx=0x7feb3c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.156+0000 7feb42a00640 1 -- 172.21.15.114:0/2633677931 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb4c151cb0 con 0x7feb4c0cc800 2024-03-06T07:32:50.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.396+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb4c04f330 con 0x7feb4c0cc800 2024-03-06T07:32:50.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.397+0000 7feb42a00640 1 -- 172.21.15.114:0/2633677931 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7feb3407a700 con 0x7feb4c0cc800 2024-03-06T07:32:50.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:50.400 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:17.782119+0000","modified":"2024-03-06T07:32:20.854485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2399499569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2399499569},{"type":"v1","addr":"172.21.15.114:6837","nonce":2399499569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:32:50.400 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 52 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.399+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb2806b500 msgr2=0x7feb2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.399+0000 7feb50a00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb2806b500 0x7feb2806d9c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7feb3c006fd0 tx=0x7feb3c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.399+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0cc800 msgr2=0x7feb4c0cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.399+0000 7feb50a00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0cc800 0x7feb4c0cde50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7feb3402bee0 tx=0x7feb340046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 shutdown_connections 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7feb4c1561f0 0x7feb4c156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7feb2806b500 0x7feb2806d9c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c0cc800 0x7feb4c0cde50 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 --2- 172.21.15.114:0/2633677931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb4c0aebf0 0x7feb4c149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 >> 172.21.15.114:0/2633677931 conn(0x7feb4c06afc0 msgr2=0x7feb4c06d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 shutdown_connections 2024-03-06T07:32:50.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.400+0000 7feb50a00640 1 -- 172.21.15.114:0/2633677931 wait complete. 2024-03-06T07:32:50.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:50.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.620+0000 7f2475a00640 1 Processor -- start 2024-03-06T07:32:50.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.620+0000 7f2475a00640 1 -- start start 2024-03-06T07:32:50.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.622+0000 7f2475a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f24700af1c0 0x7f247015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.622+0000 7f2475a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2470070d20 0x7f247006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.622+0000 7f2475a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2470071d30 con 0x7f247006d1a0 2024-03-06T07:32:50.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.622+0000 7f2475a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2470071490 con 0x7f24700af1c0 2024-03-06T07:32:50.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.622+0000 7f2475a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2470071770 con 0x7f2470070d20 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f2467e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2470070d20 0x7f247006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f2467e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2470070d20 0x7f247006cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58122/0 (socket says 172.21.15.114:58122) 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f2467e00640 1 -- 172.21.15.114:0/3533681708 learned_addr learned my addr 172.21.15.114:0/3533681708 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f246ea00640 1 --2- 172.21.15.114:0/3533681708 >> v2:172.21.15.177:3300/0 conn(0x7f24700af1c0 0x7f247015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f2467e00640 1 -- 172.21.15.114:0/3533681708 >> v2:172.21.15.177:3300/0 conn(0x7f24700af1c0 msgr2=0x7f247015eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f2467e00640 1 --2- 172.21.15.114:0/3533681708 >> v2:172.21.15.177:3300/0 conn(0x7f24700af1c0 0x7f247015eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f2467e00640 1 -- 172.21.15.114:0/3533681708 >> v1:172.21.15.177:6789/0 conn(0x7f247006d1a0 legacy=0x7f247006e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.623+0000 7f2467e00640 1 -- 172.21.15.114:0/3533681708 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470071a50 con 0x7f2470070d20 2024-03-06T07:32:50.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.625+0000 7f2467e00640 1 --2- 172.21.15.114:0/3533681708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2470070d20 0x7f247006cc60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2458009870 tx=0x7f245802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed02a3195c3cf71f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.625+0000 7f246e000640 1 -- 172.21.15.114:0/3533681708 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458034070 con 0x7f2470070d20 2024-03-06T07:32:50.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.625+0000 7f246e000640 1 -- 172.21.15.114:0/3533681708 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2458030840 con 0x7f2470070d20 2024-03-06T07:32:50.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.625+0000 7f246e000640 1 -- 172.21.15.114:0/3533681708 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458030c10 con 0x7f2470070d20 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 -- 172.21.15.114:0/3533681708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2470070d20 msgr2=0x7f247006cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 --2- 172.21.15.114:0/3533681708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2470070d20 0x7f247006cc60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2458009870 tx=0x7f245802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 -- 172.21.15.114:0/3533681708 shutdown_connections 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 --2- 172.21.15.114:0/3533681708 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2470070d20 0x7f247006cc60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 --2- 172.21.15.114:0/3533681708 >> v2:172.21.15.177:3300/0 conn(0x7f24700af1c0 0x7f247015eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 -- 172.21.15.114:0/3533681708 >> 172.21.15.114:0/3533681708 conn(0x7f24700b1730 msgr2=0x7f24700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 -- 172.21.15.114:0/3533681708 shutdown_connections 2024-03-06T07:32:50.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.626+0000 7f2475a00640 1 -- 172.21.15.114:0/3533681708 wait complete. 2024-03-06T07:32:50.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.628+0000 7f2475a00640 1 Processor -- start 2024-03-06T07:32:50.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.628+0000 7f2475a00640 1 -- start start 2024-03-06T07:32:50.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.629+0000 7f2475a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247006d1a0 0x7f2470147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.629+0000 7f2475a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470070d20 0x7f2470144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.629+0000 7f2475a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24700af1c0 0x7f2470143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.629+0000 7f2475a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247006ebe0 con 0x7f24700af1c0 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.629+0000 7f2475a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24700711a0 con 0x7f2470070d20 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.629+0000 7f2475a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247006ed60 con 0x7f247006d1a0 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24700af1c0 0x7f2470143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24700af1c0 0x7f2470143ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37494/0 (socket says 172.21.15.114:37494) 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 -- 172.21.15.114:0/421715929 learned_addr learned my addr 172.21.15.114:0/421715929 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246ea00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247006d1a0 0x7f2470147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f2467e00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470070d20 0x7f2470144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 -- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247006d1a0 msgr2=0x7f2470147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247006d1a0 0x7f2470147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 -- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470070d20 msgr2=0x7f2470144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470070d20 0x7f2470144440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f246f400640 1 -- 172.21.15.114:0/421715929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2470144bb0 con 0x7f24700af1c0 2024-03-06T07:32:50.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.630+0000 7f2467e00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470070d20 0x7f2470144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.631+0000 7f246f400640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24700af1c0 0x7f2470143ba0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f246800c730 tx=0x7f246800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.631+0000 7f2466a00640 1 -- 172.21.15.114:0/421715929 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468003b70 con 0x7f24700af1c0 2024-03-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.631+0000 7f2466a00640 1 -- 172.21.15.114:0/421715929 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2468003d10 con 0x7f24700af1c0 2024-03-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.631+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2470144e90 con 0x7f24700af1c0 2024-03-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.631+0000 7f2466a00640 1 -- 172.21.15.114:0/421715929 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246800aa90 con 0x7f24700af1c0 2024-03-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.631+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2470145450 con 0x7f24700af1c0 2024-03-06T07:32:50.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.632+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247004f330 con 0x7f24700af1c0 2024-03-06T07:32:50.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.633+0000 7f2466a00640 1 -- 172.21.15.114:0/421715929 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f246800ac30 con 0x7f24700af1c0 2024-03-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.634+0000 7f2466a00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f244c06b4d0 0x7f244c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.634+0000 7f2466a00640 1 -- 172.21.15.114:0/421715929 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f246808a060 con 0x7f24700af1c0 2024-03-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.634+0000 7f246ea00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f244c06b4d0 0x7f244c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.635+0000 7f246ea00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f244c06b4d0 0x7f244c06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2460009aa0 tx=0x7f2460009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.643+0000 7f2466a00640 1 -- 172.21.15.114:0/421715929 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2470071d30 con 0x7f24700af1c0 2024-03-06T07:32:50.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.860+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24700b4230 con 0x7f24700af1c0 2024-03-06T07:32:50.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.863+0000 7f2466a00640 1 -- 172.21.15.114:0/421715929 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16757 (secure 0 0 0) 0x7f2468058ec0 con 0x7f24700af1c0 2024-03-06T07:32:50.866 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:50.866 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":82,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:37.454277+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:15.626899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:16.804201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.c","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:50.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.865+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f244c06b4d0 msgr2=0x7f244c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.865+0000 7f2475a00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f244c06b4d0 0x7f244c06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2460009aa0 tx=0x7f2460009190 comp rx=0 tx=0).stop 2024-03-06T07:32:50.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.865+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24700af1c0 msgr2=0x7f2470143ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.865+0000 7f2475a00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24700af1c0 0x7f2470143ba0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f246800c730 tx=0x7f246800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.865+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 shutdown_connections 2024-03-06T07:32:50.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.865+0000 7f2475a00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f244c06b4d0 0x7f244c06d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.865+0000 7f2475a00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24700af1c0 0x7f2470143ba0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.866+0000 7f2475a00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470070d20 0x7f2470144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.866+0000 7f2475a00640 1 --2- 172.21.15.114:0/421715929 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247006d1a0 0x7f2470147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.866+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 >> 172.21.15.114:0/421715929 conn(0x7f24700b1730 msgr2=0x7f247015beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.866+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 shutdown_connections 2024-03-06T07:32:50.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:50.866+0000 7f2475a00640 1 -- 172.21.15.114:0/421715929 wait complete. 2024-03-06T07:32:50.887 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:50.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.095+0000 7f4497200640 1 Processor -- start 2024-03-06T07:32:51.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.095+0000 7f4497200640 1 -- start start 2024-03-06T07:32:51.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.097+0000 7f4497200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 0x7f44980cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.097+0000 7f4497200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449815c240 con 0x7f44981561f0 2024-03-06T07:32:51.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.097+0000 7f4497200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449815c500 con 0x7f44980cc800 2024-03-06T07:32:51.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.097+0000 7f4497200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449815ce30 con 0x7f44980aedc0 2024-03-06T07:32:51.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.097+0000 7f4495e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f44981561f0 0x7f44980ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43780/0 (socket says 172.21.15.114:43780) 2024-03-06T07:32:51.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.097+0000 7f4495e00640 1 -- 172.21.15.114:0/4088833645 learned_addr learned my addr 172.21.15.114:0/4088833645 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f448fe00640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 0x7f44980cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750618998 0 0) 0x7f449815c240 con 0x7f44981561f0 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c003520 con 0x7f44981561f0 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 205670224 0 0) 0x7f447c003520 con 0x7f44981561f0 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 msgr2=0x7f44980cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 0x7f44980cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 >> v1:172.21.15.177:6789/0 conn(0x7f44980cc800 legacy=0x7f449815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449815c7e0 con 0x7f44981561f0 2024-03-06T07:32:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f4488001670 con 0x7f44981561f0 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.098+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 >> v1:172.21.15.114:6789/0 conn(0x7f44981561f0 legacy=0x7f44980ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4495400640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c0039e0 0x7f447c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4495400640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 0x7f447c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4495400640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f447c004d40 0x7f447c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449815ce30 con 0x7f447c004d40 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c00baa0 con 0x7f447c0039e0 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447c00bca0 con 0x7f44980aedc0 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4496800640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f447c004d40 0x7f447c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f448fe00640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 0x7f447c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4495e00640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c0039e0 0x7f447c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4496800640 1 -- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 msgr2=0x7f447c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4496800640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44980aedc0 0x7f447c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4496800640 1 -- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c0039e0 msgr2=0x7f447c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4496800640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c0039e0 0x7f447c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.099+0000 7f4496800640 1 -- 172.21.15.114:0/4088833645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c003520 con 0x7f447c004d40 2024-03-06T07:32:51.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.100+0000 7f4496800640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f447c004d40 0x7f447c0051c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4490001e20 tx=0x7f447c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ef2558d740d312 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490001a70 con 0x7f447c004d40 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4490003b70 con 0x7f447c004d40 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4495400640 1 -- 172.21.15.114:0/4088833645 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44900050d0 con 0x7f447c004d40 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4497200640 1 -- 172.21.15.114:0/4088833645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f447c004d40 msgr2=0x7f447c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4497200640 1 --2- 172.21.15.114:0/4088833645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f447c004d40 0x7f447c0051c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4490001e20 tx=0x7f447c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4496800640 1 -- 172.21.15.114:0/4088833645 reap_dead start 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4497200640 1 -- 172.21.15.114:0/4088833645 shutdown_connections 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4497200640 1 -- 172.21.15.114:0/4088833645 >> 172.21.15.114:0/4088833645 conn(0x7f449806afc0 msgr2=0x7f449806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4497200640 1 -- 172.21.15.114:0/4088833645 shutdown_connections 2024-03-06T07:32:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.101+0000 7f4497200640 1 -- 172.21.15.114:0/4088833645 wait complete. 2024-03-06T07:32:51.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.104+0000 7f4497200640 1 Processor -- start 2024-03-06T07:32:51.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.104+0000 7f4497200640 1 -- start start 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f4497200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f447c004d40 0x7f4498148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f4497200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f449814f550 0x7f4498159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f4497200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4498150170 0x7f44981505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f4497200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449806e360 con 0x7f449814f550 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f4497200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44980ccc90 con 0x7f4498150170 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f4497200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44981565d0 con 0x7f447c004d40 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f448fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f449814f550 0x7f4498159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f448fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f449814f550 0x7f4498159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37510/0 (socket says 172.21.15.114:37510) 2024-03-06T07:32:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.106+0000 7f448fe00640 1 -- 172.21.15.114:0/3527095778 learned_addr learned my addr 172.21.15.114:0/3527095778 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:51.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448fe00640 1 -- 172.21.15.114:0/3527095778 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f447c004d40 msgr2=0x7f4498148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448fe00640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f447c004d40 0x7f4498148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448fe00640 1 -- 172.21.15.114:0/3527095778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4498150170 msgr2=0x7f44981505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448fe00640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4498150170 0x7f44981505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448fe00640 1 -- 172.21.15.114:0/3527095778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4498148980 con 0x7f449814f550 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448fe00640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f449814f550 0x7f4498159e60 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4480011630 tx=0x7f4480011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448e000640 1 -- 172.21.15.114:0/3527095778 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480015870 con 0x7f449814f550 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448e000640 1 -- 172.21.15.114:0/3527095778 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4480015a10 con 0x7f449814f550 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f448e000640 1 -- 172.21.15.114:0/3527095778 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480015dc0 con 0x7f449814f550 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449815a6b0 con 0x7f449814f550 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.107+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449815ac70 con 0x7f449814f550 2024-03-06T07:32:51.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.108+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449804f330 con 0x7f449814f550 2024-03-06T07:32:51.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.109+0000 7f448e000640 1 -- 172.21.15.114:0/3527095778 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4480013430 con 0x7f449814f550 2024-03-06T07:32:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.111+0000 7f448e000640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f447406b500 0x7f447406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.111+0000 7f448e000640 1 -- 172.21.15.114:0/3527095778 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f448008a730 con 0x7f449814f550 2024-03-06T07:32:51.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.111+0000 7f4495e00640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f447406b500 0x7f447406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.112+0000 7f4495e00640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f447406b500 0x7f447406d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f44880079b0 tx=0x7f4488009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.115+0000 7f448e000640 1 -- 172.21.15.114:0/3527095778 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4480019030 con 0x7f449814f550 2024-03-06T07:32:51.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.350+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4498072c70 con 0x7f449814f550 2024-03-06T07:32:51.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.351+0000 7f448e000640 1 -- 172.21.15.114:0/3527095778 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f4480058ea0 con 0x7f449814f550 2024-03-06T07:32:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:51.354 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:17.782119+0000","modified":"2024-03-06T07:32:20.854485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2399499569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2399499569},{"type":"v1","addr":"172.21.15.114:6837","nonce":2399499569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:32:51.354 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 52 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f447406b500 msgr2=0x7f447406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f447406b500 0x7f447406d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f44880079b0 tx=0x7f4488009040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f449814f550 msgr2=0x7f4498159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f449814f550 0x7f4498159e60 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4480011630 tx=0x7f4480011af0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 shutdown_connections 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4498150170 0x7f44981505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f447406b500 0x7f447406d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f449814f550 0x7f4498159e60 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 --2- 172.21.15.114:0/3527095778 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f447c004d40 0x7f4498148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 >> 172.21.15.114:0/3527095778 conn(0x7f449806afc0 msgr2=0x7f44980720c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 shutdown_connections 2024-03-06T07:32:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.354+0000 7f4497200640 1 -- 172.21.15.114:0/3527095778 wait complete. 2024-03-06T07:32:51.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.577+0000 7f426a400640 1 Processor -- start 2024-03-06T07:32:51.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.578+0000 7f426a400640 1 -- start start 2024-03-06T07:32:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.580+0000 7f426a400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f426406ec00 0x7f42640ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.580+0000 7f426a400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426415b090 con 0x7f42640b4680 2024-03-06T07:32:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.581+0000 7f426a400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42640b2a30 con 0x7f426415acb0 2024-03-06T07:32:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.581+0000 7f426a400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42640b4360 con 0x7f426406ec00 2024-03-06T07:32:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.581+0000 7f4262a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f42640b4680 0x7f426406e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43790/0 (socket says 172.21.15.114:43790) 2024-03-06T07:32:51.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.581+0000 7f4262a00640 1 -- 172.21.15.114:0/3405169226 learned_addr learned my addr 172.21.15.114:0/3405169226 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.581+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212423123 0 0) 0x7f426415b090 con 0x7f42640b4680 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.581+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248005730 con 0x7f42640b4680 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.581+0000 7f4263e00640 1 --2- 172.21.15.114:0/3405169226 >> v2:172.21.15.177:3300/0 conn(0x7f426406ec00 0x7f42640ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825908208 0 0) 0x7f4248005730 con 0x7f42640b4680 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 >> v2:172.21.15.177:3300/0 conn(0x7f426406ec00 msgr2=0x7f42640ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 --2- 172.21.15.114:0/3405169226 >> v2:172.21.15.177:3300/0 conn(0x7f426406ec00 0x7f42640ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 >> v1:172.21.15.177:6789/0 conn(0x7f426415acb0 legacy=0x7f42640b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42640b2cf0 con 0x7f42640b4680 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f4250001670 con 0x7f42640b4680 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 >> v1:172.21.15.114:6789/0 conn(0x7f42640b4680 legacy=0x7f426406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4248006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4248006a10 0x7f4248006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4248007420 0x7f424800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42640b4360 con 0x7f4248005c10 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424800e370 con 0x7f4248007420 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424800e570 con 0x7f4248006a10 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4263400640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4248006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.582+0000 7f4263e00640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4248007420 0x7f424800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.583+0000 7f4262a00640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4248006a10 0x7f4248006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.583+0000 7f4263400640 1 -- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4248006a10 msgr2=0x7f4248006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.583+0000 7f4263400640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4248006a10 0x7f4248006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.583+0000 7f4263400640 1 -- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4248007420 msgr2=0x7f424800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.583+0000 7f4263400640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4248007420 0x7f424800dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.583+0000 7f4263400640 1 -- 172.21.15.114:0/3405169226 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4248005730 con 0x7f4248005c10 2024-03-06T07:32:51.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.583+0000 7f4263e00640 1 -- 172.21.15.114:0/3405169226 reap_dead start 2024-03-06T07:32:51.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.584+0000 7f4263400640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4248006030 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4254001750 tx=0x7f4248007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177d2fb9599c11de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254001370 con 0x7f4248005c10 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42540052d0 con 0x7f4248005c10 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f4262000640 1 -- 172.21.15.114:0/3405169226 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254003830 con 0x7f4248005c10 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f426a400640 1 -- 172.21.15.114:0/3405169226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 msgr2=0x7f4248006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f426a400640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4248006030 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4254001750 tx=0x7f4248007a40 comp rx=0 tx=0).stop 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f426a400640 1 -- 172.21.15.114:0/3405169226 shutdown_connections 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f426a400640 1 --2- 172.21.15.114:0/3405169226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4248006030 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f426a400640 1 -- 172.21.15.114:0/3405169226 >> 172.21.15.114:0/3405169226 conn(0x7f42640b1730 msgr2=0x7f42640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f426a400640 1 -- 172.21.15.114:0/3405169226 shutdown_connections 2024-03-06T07:32:51.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.585+0000 7f426a400640 1 -- 172.21.15.114:0/3405169226 wait complete. 2024-03-06T07:32:51.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.588+0000 7f426a400640 1 Processor -- start 2024-03-06T07:32:51.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.588+0000 7f426a400640 1 -- start start 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f426a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4264156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f426a400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f426415dad0 0x7f42641523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f426a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4264152910 0x7f4264159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f426a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42640b4010 con 0x7f4248005c10 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f426a400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42640b32b0 con 0x7f4264152910 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f426a400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4264072340 con 0x7f426415dad0 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f4263400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4264156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f4263400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4264156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37532/0 (socket says 172.21.15.114:37532) 2024-03-06T07:32:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.590+0000 7f4263400640 1 -- 172.21.15.114:0/136897056 learned_addr learned my addr 172.21.15.114:0/136897056 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f4263400640 1 -- 172.21.15.114:0/136897056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f426415dad0 msgr2=0x7f42641523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f4263400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f426415dad0 0x7f42641523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f4263400640 1 -- 172.21.15.114:0/136897056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4264152910 msgr2=0x7f4264159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f4263400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4264152910 0x7f4264159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f4263400640 1 -- 172.21.15.114:0/136897056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4264158700 con 0x7f4248005c10 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f4263400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4264156c40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4254005040 tx=0x7f4254001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f425b400640 1 -- 172.21.15.114:0/136897056 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254041070 con 0x7f4248005c10 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f425b400640 1 -- 172.21.15.114:0/136897056 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4254005480 con 0x7f4248005c10 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f425b400640 1 -- 172.21.15.114:0/136897056 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4254007a30 con 0x7f4248005c10 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.591+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42641589a0 con 0x7f4248005c10 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.592+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4264158f60 con 0x7f4248005c10 2024-03-06T07:32:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.593+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426404f330 con 0x7f4248005c10 2024-03-06T07:32:51.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.594+0000 7f425b400640 1 -- 172.21.15.114:0/136897056 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4254044070 con 0x7f4248005c10 2024-03-06T07:32:51.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.595+0000 7f425b400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f424406b500 0x7f424406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.595+0000 7f425b400640 1 -- 172.21.15.114:0/136897056 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f42540abd40 con 0x7f4248005c10 2024-03-06T07:32:51.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.595+0000 7f4262a00640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f424406b500 0x7f424406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.596+0000 7f4262a00640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f424406b500 0x7f424406d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4250000c00 tx=0x7f4250009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.599+0000 7f425b400640 1 -- 172.21.15.114:0/136897056 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f425407ab90 con 0x7f4248005c10 2024-03-06T07:32:51.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.835+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f426406ce60 con 0x7f4248005c10 2024-03-06T07:32:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.836+0000 7f425b400640 1 -- 172.21.15.114:0/136897056 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f425407a490 con 0x7f4248005c10 2024-03-06T07:32:51.839 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:51.839 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":52,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:17.782119+0000","modified":"2024-03-06T07:32:20.854485+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6756":{"gid":6756,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/2399499569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2399499569},{"type":"v1","addr":"172.21.15.114:6837","nonce":2399499569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6756,"qdb_cluster":[6756]},"id":6}]} 2024-03-06T07:32:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 52 2024-03-06T07:32:51.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f424406b500 msgr2=0x7f424406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f424406b500 0x7f424406d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4250000c00 tx=0x7f4250009040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 msgr2=0x7f4264156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4264156c40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4254005040 tx=0x7f4254001120 comp rx=0 tx=0).stop 2024-03-06T07:32:51.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 shutdown_connections 2024-03-06T07:32:51.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4264152910 0x7f4264159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f426415dad0 0x7f42641523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f424406b500 0x7f424406d9c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 --2- 172.21.15.114:0/136897056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4248005c10 0x7f4264156c40 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 >> 172.21.15.114:0/136897056 conn(0x7f42640b1730 msgr2=0x7f42640ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.838+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 shutdown_connections 2024-03-06T07:32:51.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:51.839+0000 7f426a400640 1 -- 172.21.15.114:0/136897056 wait complete. 2024-03-06T07:32:51.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.054+0000 7efd1da00640 1 Processor -- start 2024-03-06T07:32:52.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.054+0000 7efd1da00640 1 -- start start 2024-03-06T07:32:52.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.057+0000 7efd1da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 0x7efd18168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.057+0000 7efd1da00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1806f1f0 con 0x7efd1815e160 2024-03-06T07:32:52.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.057+0000 7efd1da00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd180afce0 con 0x7efd180af820 2024-03-06T07:32:52.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.057+0000 7efd1da00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806b0f0 con 0x7efd1806c590 2024-03-06T07:32:52.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.057+0000 7efd16a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7efd1815e160 0x7efd18162620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43800/0 (socket says 172.21.15.114:43800) 2024-03-06T07:32:52.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.057+0000 7efd16a00640 1 -- 172.21.15.114:0/909802916 learned_addr learned my addr 172.21.15.114:0/909802916 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:52.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd17400640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 0x7efd18168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947036469 0 0) 0x7efd1806f1f0 con 0x7efd1815e160 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcfc003520 con 0x7efd1815e160 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705863894 0 0) 0x7efd180afce0 con 0x7efd180af820 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1806f1f0 con 0x7efd180af820 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612145826 0 0) 0x7efcfc003520 con 0x7efd1815e160 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 msgr2=0x7efd18168fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 0x7efd18168fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 >> v1:172.21.15.177:6789/0 conn(0x7efd180af820 legacy=0x7efd180b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1806ec30 con 0x7efd1815e160 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7efd08001670 con 0x7efd1815e160 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.058+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 >> v1:172.21.15.114:6789/0 conn(0x7efd1815e160 legacy=0x7efd18162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd16000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 0x7efcfc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd16000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efcfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd16000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcfc004d40 0x7efcfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806b0f0 con 0x7efcfc004120 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc00bb80 con 0x7efcfc004d40 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc00bd80 con 0x7efd1806c590 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd0e000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efcfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd17400640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcfc004d40 0x7efcfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd16a00640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 0x7efcfc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd0e000640 1 -- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 msgr2=0x7efcfc003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd0e000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd1806c590 0x7efcfc003b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd0e000640 1 -- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcfc004d40 msgr2=0x7efcfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd0e000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcfc004d40 0x7efcfc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd0e000640 1 -- 172.21.15.114:0/909802916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc003520 con 0x7efcfc004120 2024-03-06T07:32:52.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.059+0000 7efd17400640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcfc004d40 0x7efcfc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:52.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.061+0000 7efd0e000640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efcfc004800 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7efd000018a0 tx=0x7efcfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1794261c2053b1ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.061+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd00037070 con 0x7efcfc004120 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.061+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd000052d0 con 0x7efcfc004120 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.061+0000 7efd16000640 1 -- 172.21.15.114:0/909802916 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000014f0 con 0x7efcfc004120 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.062+0000 7efd1da00640 1 -- 172.21.15.114:0/909802916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 msgr2=0x7efcfc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.062+0000 7efd1da00640 1 --2- 172.21.15.114:0/909802916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efcfc004800 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7efd000018a0 tx=0x7efcfc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.062+0000 7efd17400640 1 -- 172.21.15.114:0/909802916 reap_dead start 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.062+0000 7efd1da00640 1 -- 172.21.15.114:0/909802916 shutdown_connections 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.062+0000 7efd1da00640 1 -- 172.21.15.114:0/909802916 >> 172.21.15.114:0/909802916 conn(0x7efd180d0340 msgr2=0x7efd180d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.062+0000 7efd1da00640 1 -- 172.21.15.114:0/909802916 shutdown_connections 2024-03-06T07:32:52.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.062+0000 7efd1da00640 1 -- 172.21.15.114:0/909802916 wait complete. 2024-03-06T07:32:52.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.064+0000 7efd1da00640 1 Processor -- start 2024-03-06T07:32:52.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.064+0000 7efd1da00640 1 -- start start 2024-03-06T07:32:52.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd1da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efd18142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd1da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd18152c00 0x7efd18153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd1da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd18145660 0x7efd18145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd1da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd180b3ac0 con 0x7efcfc004120 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd1da00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd180b5620 con 0x7efd18152c00 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd1da00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd180b3c40 con 0x7efd18145660 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efd18142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efd18142940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37542/0 (socket says 172.21.15.114:37542) 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 -- 172.21.15.114:0/4175843844 learned_addr learned my addr 172.21.15.114:0/4175843844 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 -- 172.21.15.114:0/4175843844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd18145660 msgr2=0x7efd18145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd18145660 0x7efd18145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 -- 172.21.15.114:0/4175843844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd18152c00 msgr2=0x7efd18153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd18152c00 0x7efd18153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.065+0000 7efd16a00640 1 -- 172.21.15.114:0/4175843844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd181613e0 con 0x7efcfc004120 2024-03-06T07:32:52.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.066+0000 7efd16a00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efd18142940 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7efd0800d7d0 tx=0x7efd0800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.066+0000 7efd0f400640 1 -- 172.21.15.114:0/4175843844 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08019070 con 0x7efcfc004120 2024-03-06T07:32:52.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.066+0000 7efd0f400640 1 -- 172.21.15.114:0/4175843844 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd080072d0 con 0x7efcfc004120 2024-03-06T07:32:52.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.066+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd18160620 con 0x7efcfc004120 2024-03-06T07:32:52.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.066+0000 7efd0f400640 1 -- 172.21.15.114:0/4175843844 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd080047e0 con 0x7efcfc004120 2024-03-06T07:32:52.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.066+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd18160be0 con 0x7efcfc004120 2024-03-06T07:32:52.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.067+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd18074ae0 con 0x7efcfc004120 2024-03-06T07:32:52.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.068+0000 7efd0f400640 1 -- 172.21.15.114:0/4175843844 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd0801c070 con 0x7efcfc004120 2024-03-06T07:32:52.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.069+0000 7efd0f400640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.069+0000 7efd0f400640 1 -- 172.21.15.114:0/4175843844 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efd08089b70 con 0x7efcfc004120 2024-03-06T07:32:52.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.069+0000 7efd0e000640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.070+0000 7efd0e000640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcf406b4d0 0x7efcf406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7efd00034000 tx=0x7efd00009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.074+0000 7efd0f400640 1 -- 172.21.15.114:0/4175843844 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd08058930 con 0x7efcfc004120 2024-03-06T07:32:52.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.261+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd1804f330 con 0x7efcfc004120 2024-03-06T07:32:52.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.264+0000 7efd0f400640 1 -- 172.21.15.114:0/4175843844 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16757 (secure 0 0 0) 0x7efd08058230 con 0x7efcfc004120 2024-03-06T07:32:52.267 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:52.267 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":82,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:37.454277+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:15.626899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:16.804201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.c","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:52.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcf406b4d0 msgr2=0x7efcf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcf406b4d0 0x7efcf406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7efd00034000 tx=0x7efd00009030 comp rx=0 tx=0).stop 2024-03-06T07:32:52.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 msgr2=0x7efd18142940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efd18142940 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7efd0800d7d0 tx=0x7efd0800dc90 comp rx=0 tx=0).stop 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 shutdown_connections 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efd18145660 0x7efd18145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd18152c00 0x7efd18153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcf406b4d0 0x7efcf406d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 --2- 172.21.15.114:0/4175843844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcfc004120 0x7efd18142940 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.266+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 >> 172.21.15.114:0/4175843844 conn(0x7efd180d0340 msgr2=0x7efd180ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.267+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 shutdown_connections 2024-03-06T07:32:52.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.267+0000 7efd1da00640 1 -- 172.21.15.114:0/4175843844 wait complete. 2024-03-06T07:32:52.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:32:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.475+0000 7f1201c00640 1 Processor -- start 2024-03-06T07:32:52.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.475+0000 7f1201c00640 1 -- start start 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f1201c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f11fc161580 0x7f11fc0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f1201c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f11fc0d2a70 0x7f11fc0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f1201c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc161960 con 0x7f11fc0d1e90 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f1201c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc0cf6f0 con 0x7f11fc161580 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f1201c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc0cf9d0 con 0x7f11fc0d2a70 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f11fbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f11fc161580 0x7f11fc0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f11fbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f11fc161580 0x7f11fc0d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37554/0 (socket says 172.21.15.114:37554) 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f11fbe00640 1 -- 172.21.15.114:0/366336515 learned_addr learned my addr 172.21.15.114:0/366336515 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:52.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.477+0000 7f11fb400640 1 --1- 172.21.15.114:0/366336515 >> v1:172.21.15.114:6789/0 conn(0x7f11fc0d1e90 0x7f11fc0d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43804/0 (socket says 172.21.15.114:43804) 2024-03-06T07:32:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.478+0000 7f11fbe00640 1 -- 172.21.15.114:0/366336515 >> v2:172.21.15.177:3300/0 conn(0x7f11fc0d2a70 msgr2=0x7f11fc0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.478+0000 7f11fbe00640 1 --2- 172.21.15.114:0/366336515 >> v2:172.21.15.177:3300/0 conn(0x7f11fc0d2a70 0x7f11fc0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.478+0000 7f11fbe00640 1 -- 172.21.15.114:0/366336515 >> v1:172.21.15.114:6789/0 conn(0x7f11fc0d1e90 legacy=0x7f11fc0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.478+0000 7f11fbe00640 1 -- 172.21.15.114:0/366336515 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0cfcb0 con 0x7f11fc161580 2024-03-06T07:32:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.479+0000 7f11fbe00640 1 --2- 172.21.15.114:0/366336515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0d1950 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f11fc08d180 tx=0x7f11ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e854344c409572 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.479+0000 7f11faa00640 1 -- 172.21.15.114:0/366336515 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec034070 con 0x7f11fc161580 2024-03-06T07:32:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.479+0000 7f11faa00640 1 -- 172.21.15.114:0/366336515 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec030860 con 0x7f11fc161580 2024-03-06T07:32:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.479+0000 7f11faa00640 1 -- 172.21.15.114:0/366336515 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec030c30 con 0x7f11fc161580 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 -- 172.21.15.114:0/366336515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 msgr2=0x7f11fc0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 --2- 172.21.15.114:0/366336515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0d1950 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f11fc08d180 tx=0x7f11ec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 -- 172.21.15.114:0/366336515 shutdown_connections 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 --2- 172.21.15.114:0/366336515 >> v2:172.21.15.177:3300/0 conn(0x7f11fc0d2a70 0x7f11fc0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 --2- 172.21.15.114:0/366336515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0d1950 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 -- 172.21.15.114:0/366336515 >> 172.21.15.114:0/366336515 conn(0x7f11fc0d3d80 msgr2=0x7f11fc0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 -- 172.21.15.114:0/366336515 shutdown_connections 2024-03-06T07:32:52.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.480+0000 7f1201c00640 1 -- 172.21.15.114:0/366336515 wait complete. 2024-03-06T07:32:52.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.483+0000 7f1201c00640 1 Processor -- start 2024-03-06T07:32:52.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.483+0000 7f1201c00640 1 -- start start 2024-03-06T07:32:52.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.484+0000 7f1201c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11fc0d1e90 0x7f11fc0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.484+0000 7f1201c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11fc0d2a70 0x7f11fc1f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.484+0000 7f1201c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.484+0000 7f1201c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0cf3a0 con 0x7f11fc161580 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.484+0000 7f1201c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0b39a0 con 0x7f11fc0d2a70 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.484+0000 7f1201c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0b4400 con 0x7f11fc0d1e90 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37570/0 (socket says 172.21.15.114:37570) 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 -- 172.21.15.114:0/4015592809 learned_addr learned my addr 172.21.15.114:0/4015592809 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f11fb400640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11fc0d2a70 0x7f11fc1f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f11fbe00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11fc0d1e90 0x7f11fc0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 -- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11fc0d1e90 msgr2=0x7f11fc0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11fc0d1e90 0x7f11fc0c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 -- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11fc0d2a70 msgr2=0x7f11fc1f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11fc0d2a70 0x7f11fc1f7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 -- 172.21.15.114:0/4015592809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0c8620 con 0x7f11fc161580 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.485+0000 7f1201200640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0ca540 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f11f400b9e0 tx=0x7f11f400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.486+0000 7f11f3e00640 1 -- 172.21.15.114:0/4015592809 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4019070 con 0x7f11fc161580 2024-03-06T07:32:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.486+0000 7f11f3e00640 1 -- 172.21.15.114:0/4015592809 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f400cb40 con 0x7f11fc161580 2024-03-06T07:32:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.486+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc0c8900 con 0x7f11fc161580 2024-03-06T07:32:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.486+0000 7f11f3e00640 1 -- 172.21.15.114:0/4015592809 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4014440 con 0x7f11fc161580 2024-03-06T07:32:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.486+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc0c7ca0 con 0x7f11fc161580 2024-03-06T07:32:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.487+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc04f330 con 0x7f11fc161580 2024-03-06T07:32:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.488+0000 7f11f3e00640 1 -- 172.21.15.114:0/4015592809 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11f401c070 con 0x7f11fc161580 2024-03-06T07:32:52.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.489+0000 7f11f3e00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11dc06b500 0x7f11dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.489+0000 7f11f3e00640 1 -- 172.21.15.114:0/4015592809 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f11f4089f10 con 0x7f11fc161580 2024-03-06T07:32:52.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.489+0000 7f11fbe00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11dc06b500 0x7f11dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.490+0000 7f11fbe00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11dc06b500 0x7f11dc06d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f11ec036000 tx=0x7f11ec03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.492+0000 7f11f3e00640 1 -- 172.21.15.114:0/4015592809 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11f4058d60 con 0x7f11fc161580 2024-03-06T07:32:52.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.717+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f11fc0d3b60 con 0x7f11fc161580 2024-03-06T07:32:52.720 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:32:52.718+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:52.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.721+0000 7f11f3e00640 1 -- 172.21.15.114:0/4015592809 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v53) v1 ==== 161+0+0 (secure 0 0 0) 0x7f11f4058660 con 0x7f11fc161580 2024-03-06T07:32:52.723 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:32:52.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.723+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11dc06b500 msgr2=0x7f11dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.723+0000 7f1201c00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11dc06b500 0x7f11dc06d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f11ec036000 tx=0x7f11ec03b040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.723+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 msgr2=0x7f11fc0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.723+0000 7f1201c00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0ca540 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f11f400b9e0 tx=0x7f11f400bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.724+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 shutdown_connections 2024-03-06T07:32:52.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.724+0000 7f1201c00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f11dc06b500 0x7f11dc06d9c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.724+0000 7f1201c00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11fc161580 0x7f11fc0ca540 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.724+0000 7f1201c00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11fc0d2a70 0x7f11fc1f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.724+0000 7f1201c00640 1 --2- 172.21.15.114:0/4015592809 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f11fc0d1e90 0x7f11fc0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.725+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 >> 172.21.15.114:0/4015592809 conn(0x7f11fc0d3d80 msgr2=0x7f11fc06c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.725+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 shutdown_connections 2024-03-06T07:32:52.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.725+0000 7f1201c00640 1 -- 172.21.15.114:0/4015592809 wait complete. 2024-03-06T07:32:52.740 INFO:tasks.ceph.mds.c.smithi114.stderr: -4514> 2024-03-06T07:32:22.719+0000 7f80e4e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:52.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:32:52.799 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:32:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.930+0000 7fddce400640 1 Processor -- start 2024-03-06T07:32:52.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.930+0000 7fddce400640 1 -- start start 2024-03-06T07:32:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddce400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fddc815e000 0x7fddc806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddce400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 0x7fddc80d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddce400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc80d1260 con 0x7fddc806f7e0 2024-03-06T07:32:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddce400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc80d1020 con 0x7fddc815e000 2024-03-06T07:32:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddce400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc806c0d0 con 0x7fddc815e3e0 2024-03-06T07:32:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddc7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fddc806f7e0 0x7fddc80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43818/0 (socket says 172.21.15.114:43818) 2024-03-06T07:32:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddc7e00640 1 -- 172.21.15.114:0/3883103101 learned_addr learned my addr 172.21.15.114:0/3883103101 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:52.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6a00640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 0x7fddc80d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.932+0000 7fddc7400640 1 --2- 172.21.15.114:0/3883103101 >> v2:172.21.15.177:3300/0 conn(0x7fddc815e000 0x7fddc806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914980296 0 0) 0x7fddc80d1260 con 0x7fddc806f7e0 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8003520 con 0x7fddc806f7e0 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851696793 0 0) 0x7fdda8003520 con 0x7fddc806f7e0 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 msgr2=0x7fddc80d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 0x7fddc80d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 >> v2:172.21.15.177:3300/0 conn(0x7fddc815e000 msgr2=0x7fddc806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 --2- 172.21.15.114:0/3883103101 >> v2:172.21.15.177:3300/0 conn(0x7fddc815e000 0x7fddc806f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc806c320 con 0x7fddc806f7e0 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fddc0001670 con 0x7fddc806f7e0 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 >> v1:172.21.15.114:6789/0 conn(0x7fddc806f7e0 legacy=0x7fddc80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda80039b0 0x7fdda8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 0x7fdda8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8004d40 0x7fdda80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80d1020 con 0x7fdda8004d40 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc806c0d0 con 0x7fdda80039b0 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda800b9c0 con 0x7fddc815e3e0 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.933+0000 7fddc7e00640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8004d40 0x7fdda80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc7400640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda80039b0 0x7fdda8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc6a00640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 0x7fdda8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc7e00640 1 -- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 msgr2=0x7fdda8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc7e00640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 0x7fdda8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc7e00640 1 -- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda80039b0 msgr2=0x7fdda8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc7e00640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda80039b0 0x7fdda8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc7e00640 1 -- 172.21.15.114:0/3883103101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda8003520 con 0x7fdda8004d40 2024-03-06T07:32:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc7400640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda80039b0 0x7fdda8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.934+0000 7fddc6a00640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc815e3e0 0x7fdda8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.935+0000 7fddc7e00640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8004d40 0x7fdda80051c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fddc000c210 tx=0x7fdda8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b318507537fac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.935+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0007d50 con 0x7fdda8004d40 2024-03-06T07:32:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.935+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc0030070 con 0x7fdda8004d40 2024-03-06T07:32:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.935+0000 7fddc6000640 1 -- 172.21.15.114:0/3883103101 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc002cd90 con 0x7fdda8004d40 2024-03-06T07:32:52.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.935+0000 7fddce400640 1 -- 172.21.15.114:0/3883103101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8004d40 msgr2=0x7fdda80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.935+0000 7fddce400640 1 --2- 172.21.15.114:0/3883103101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8004d40 0x7fdda80051c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fddc000c210 tx=0x7fdda8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.936+0000 7fddc7e00640 1 -- 172.21.15.114:0/3883103101 reap_dead start 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.936+0000 7fddce400640 1 -- 172.21.15.114:0/3883103101 shutdown_connections 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.936+0000 7fddce400640 1 -- 172.21.15.114:0/3883103101 >> 172.21.15.114:0/3883103101 conn(0x7fddc80b3800 msgr2=0x7fddc80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.936+0000 7fddce400640 1 -- 172.21.15.114:0/3883103101 shutdown_connections 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.936+0000 7fddce400640 1 -- 172.21.15.114:0/3883103101 wait complete. 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.937+0000 7fddce400640 1 Processor -- start 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.937+0000 7fddce400640 1 -- start start 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.938+0000 7fddce400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda8004d40 0x7fddc81f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.938+0000 7fddce400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc80c7e30 0x7fddc8156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.938+0000 7fddce400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc815eed0 0x7fddc815f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.938+0000 7fddce400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80d0c60 con 0x7fddc815eed0 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.938+0000 7fddce400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80d0ae0 con 0x7fdda8004d40 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.938+0000 7fddce400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc80d0de0 con 0x7fddc80c7e30 2024-03-06T07:32:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc815eed0 0x7fddc815f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc815eed0 0x7fddc815f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37610/0 (socket says 172.21.15.114:37610) 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 -- 172.21.15.114:0/3656724585 learned_addr learned my addr 172.21.15.114:0/3656724585 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc6a00640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc80c7e30 0x7fddc8156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda8004d40 0x7fddc81f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 -- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc80c7e30 msgr2=0x7fddc8156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc80c7e30 0x7fddc8156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 -- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda8004d40 msgr2=0x7fddc81f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda8004d40 0x7fddc81f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 -- 172.21.15.114:0/3656724585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc80c6d90 con 0x7fddc815eed0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.939+0000 7fddc7e00640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc815eed0 0x7fddc815f350 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fddc002c600 tx=0x7fddc0001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.940+0000 7fddbf400640 1 -- 172.21.15.114:0/3656724585 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0007d50 con 0x7fddc815eed0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.940+0000 7fddbf400640 1 -- 172.21.15.114:0/3656724585 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc0004bf0 con 0x7fddc815eed0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.940+0000 7fddbf400640 1 -- 172.21.15.114:0/3656724585 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0001c90 con 0x7fddc815eed0 2024-03-06T07:32:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.940+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc80c7070 con 0x7fddc815eed0 2024-03-06T07:32:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.940+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc80c7630 con 0x7fddc815eed0 2024-03-06T07:32:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.941+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc804f330 con 0x7fddc815eed0 2024-03-06T07:32:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.942+0000 7fddbf400640 1 -- 172.21.15.114:0/3656724585 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddc0007070 con 0x7fddc815eed0 2024-03-06T07:32:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.943+0000 7fddbf400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdda406b4d0 0x7fdda406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.943+0000 7fddbf400640 1 -- 172.21.15.114:0/3656724585 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fddc00ab2d0 con 0x7fddc815eed0 2024-03-06T07:32:52.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.943+0000 7fddc7400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdda406b4d0 0x7fdda406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.944+0000 7fddc7400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdda406b4d0 0x7fdda406d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009a40 tx=0x7fddb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:52.947+0000 7fddbf400640 1 -- 172.21.15.114:0/3656724585 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddc007a170 con 0x7fddc815eed0 2024-03-06T07:32:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.176+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fddc80d1a10 con 0x7fddc815eed0 2024-03-06T07:32:53.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.724+0000 7fddbf400640 1 -- 172.21.15.114:0/3656724585 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7fddc0079a70 con 0x7fddc815eed0 2024-03-06T07:32:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.727+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdda406b4d0 msgr2=0x7fdda406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.727+0000 7fddce400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdda406b4d0 0x7fdda406d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009a40 tx=0x7fddb8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.727+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc815eed0 msgr2=0x7fddc815f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.727+0000 7fddce400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc815eed0 0x7fddc815f350 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fddc002c600 tx=0x7fddc0001ab0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 shutdown_connections 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdda406b4d0 0x7fdda406d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc815eed0 0x7fddc815f350 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fddc80c7e30 0x7fddc8156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 --2- 172.21.15.114:0/3656724585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdda8004d40 0x7fddc81f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 >> 172.21.15.114:0/3656724585 conn(0x7fddc80b3800 msgr2=0x7fddc806edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 shutdown_connections 2024-03-06T07:32:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.728+0000 7fddce400640 1 -- 172.21.15.114:0/3656724585 wait complete. 2024-03-06T07:32:53.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:53.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.951+0000 7f4939a00640 1 Processor -- start 2024-03-06T07:32:53.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.951+0000 7f4939a00640 1 -- start start 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4939a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4934165fe0 0x7f49341634b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4939a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f493406c9c0 0x7f49341663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4939a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4934166900 con 0x7f493406d620 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4939a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4934071aa0 con 0x7f493406c9c0 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4939a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49340b3f00 con 0x7f4934165fe0 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f493406c9c0 0x7f49341663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f493406c9c0 0x7f49341663c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37624/0 (socket says 172.21.15.114:37624) 2024-03-06T07:32:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932000640 1 -- 172.21.15.114:0/178731716 learned_addr learned my addr 172.21.15.114:0/178731716 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932a00640 1 --2- 172.21.15.114:0/178731716 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4934165fe0 0x7f49341634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932000640 1 -- 172.21.15.114:0/178731716 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4934165fe0 msgr2=0x7f49341634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932000640 1 --2- 172.21.15.114:0/178731716 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4934165fe0 0x7f49341634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932000640 1 -- 172.21.15.114:0/178731716 >> v1:172.21.15.114:6789/0 conn(0x7f493406d620 legacy=0x7f493406dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4932000640 1 -- 172.21.15.114:0/178731716 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49340b41e0 con 0x7f493406c9c0 2024-03-06T07:32:53.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.953+0000 7f4931600640 1 -- 172.21.15.114:0/178731716 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311083899 0 0) 0x7f4934166900 con 0x7f493406d620 2024-03-06T07:32:53.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4932000640 1 --2- 172.21.15.114:0/178731716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f493406c9c0 0x7f49341663c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4920009870 tx=0x7f492002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6261ec108c56500 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4931600640 1 -- 172.21.15.114:0/178731716 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920034070 con 0x7f493406c9c0 2024-03-06T07:32:53.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4931600640 1 -- 172.21.15.114:0/178731716 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4920030840 con 0x7f493406c9c0 2024-03-06T07:32:53.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4931600640 1 -- 172.21.15.114:0/178731716 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920030c10 con 0x7f493406c9c0 2024-03-06T07:32:53.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4939a00640 1 -- 172.21.15.114:0/178731716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f493406c9c0 msgr2=0x7f49341663c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4939a00640 1 --2- 172.21.15.114:0/178731716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f493406c9c0 0x7f49341663c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4920009870 tx=0x7f492002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4939a00640 1 -- 172.21.15.114:0/178731716 shutdown_connections 2024-03-06T07:32:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4939a00640 1 --2- 172.21.15.114:0/178731716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f493406c9c0 0x7f49341663c0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4939a00640 1 --2- 172.21.15.114:0/178731716 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4934165fe0 0x7f49341634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.955+0000 7f4939a00640 1 -- 172.21.15.114:0/178731716 >> 172.21.15.114:0/178731716 conn(0x7f49340cfa80 msgr2=0x7f49340cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.956+0000 7f4939a00640 1 -- 172.21.15.114:0/178731716 shutdown_connections 2024-03-06T07:32:53.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.956+0000 7f4939a00640 1 -- 172.21.15.114:0/178731716 wait complete. 2024-03-06T07:32:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.957+0000 7f4939a00640 1 Processor -- start 2024-03-06T07:32:53.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.957+0000 7f4939a00640 1 -- start start 2024-03-06T07:32:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.958+0000 7f4939a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f493406c9c0 0x7f4934143ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.958+0000 7f4939a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f493406d620 0x7f4934154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.958+0000 7f4939a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4934165fe0 0x7f493414ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.958+0000 7f4939a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493406ce20 con 0x7f4934165fe0 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.958+0000 7f4939a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49340b47a0 con 0x7f493406c9c0 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.958+0000 7f4939a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493414c0d0 con 0x7f493406d620 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4934165fe0 0x7f493414ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4934165fe0 0x7f493414ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37640/0 (socket says 172.21.15.114:37640) 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 -- 172.21.15.114:0/2836867223 learned_addr learned my addr 172.21.15.114:0/2836867223 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 -- 172.21.15.114:0/2836867223 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f493406d620 msgr2=0x7f4934154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f493406d620 0x7f4934154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 -- 172.21.15.114:0/2836867223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f493406c9c0 msgr2=0x7f4934143ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4932a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f493406c9c0 0x7f4934143ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f493406c9c0 0x7f4934143ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 -- 172.21.15.114:0/2836867223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49341441e0 con 0x7f4934165fe0 2024-03-06T07:32:53.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.959+0000 7f4933400640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4934165fe0 0x7f493414ad30 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f492c00c760 tx=0x7f492c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.960+0000 7f492aa00640 1 -- 172.21.15.114:0/2836867223 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c003da0 con 0x7f4934165fe0 2024-03-06T07:32:53.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.960+0000 7f492aa00640 1 -- 172.21.15.114:0/2836867223 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c005070 con 0x7f4934165fe0 2024-03-06T07:32:53.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.960+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493414b680 con 0x7f4934165fe0 2024-03-06T07:32:53.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.960+0000 7f492aa00640 1 -- 172.21.15.114:0/2836867223 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c005460 con 0x7f4934165fe0 2024-03-06T07:32:53.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.960+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4934144860 con 0x7f4934165fe0 2024-03-06T07:32:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.961+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493404f3a0 con 0x7f4934165fe0 2024-03-06T07:32:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.962+0000 7f492aa00640 1 -- 172.21.15.114:0/2836867223 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f492c005600 con 0x7f4934165fe0 2024-03-06T07:32:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.963+0000 7f492aa00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f491406b4d0 0x7f491406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.963+0000 7f492aa00640 1 -- 172.21.15.114:0/2836867223 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f492c08a180 con 0x7f4934165fe0 2024-03-06T07:32:53.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.963+0000 7f4932a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f491406b4d0 0x7f491406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.963+0000 7f4932a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f491406b4d0 0x7f491406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4924006fd0 tx=0x7f4924008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:53.967+0000 7f492aa00640 1 -- 172.21.15.114:0/2836867223 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49341441e0 con 0x7f4934165fe0 2024-03-06T07:32:54.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.165+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f49341668f0 con 0x7f4934165fe0 2024-03-06T07:32:54.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.734+0000 7f492aa00640 1 -- 172.21.15.114:0/2836867223 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7f492c058fa0 con 0x7f4934165fe0 2024-03-06T07:32:54.738 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.738+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f491406b4d0 msgr2=0x7f491406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.738+0000 7f4939a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f491406b4d0 0x7f491406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4924006fd0 tx=0x7f4924008040 comp rx=0 tx=0).stop 2024-03-06T07:32:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.738+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4934165fe0 msgr2=0x7f493414ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.738+0000 7f4939a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4934165fe0 0x7f493414ad30 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f492c00c760 tx=0x7f492c00cc20 comp rx=0 tx=0).stop 2024-03-06T07:32:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 shutdown_connections 2024-03-06T07:32:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f491406b4d0 0x7f491406d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4934165fe0 0x7f493414ad30 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f493406d620 0x7f4934154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 --2- 172.21.15.114:0/2836867223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f493406c9c0 0x7f4934143ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 >> 172.21.15.114:0/2836867223 conn(0x7f49340cfa80 msgr2=0x7f493406da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 shutdown_connections 2024-03-06T07:32:54.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.739+0000 7f4939a00640 1 -- 172.21.15.114:0/2836867223 wait complete. 2024-03-06T07:32:54.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.960+0000 7f5399200640 1 Processor -- start 2024-03-06T07:32:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.960+0000 7f5399200640 1 -- start start 2024-03-06T07:32:54.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5399200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 0x7f539415fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5399200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f53940ced20 0x7f539415ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5399200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940cf1e0 con 0x7f53940712d0 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5399200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5394070e40 con 0x7f53940ced20 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5399200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53940cf4c0 con 0x7f5394145980 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5392800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f53940712d0 0x7f5394071740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43836/0 (socket says 172.21.15.114:43836) 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5392800640 1 -- 172.21.15.114:0/2068798216 learned_addr learned my addr 172.21.15.114:0/2068798216 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5391400640 1 --2- 172.21.15.114:0/2068798216 >> v2:172.21.15.177:3300/0 conn(0x7f53940ced20 0x7f539415ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935702673 0 0) 0x7f53940cf1e0 con 0x7f53940712d0 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.962+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5378003520 con 0x7f53940712d0 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5391e00640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 0x7f539415fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835290117 0 0) 0x7f5378003520 con 0x7f53940712d0 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 msgr2=0x7f539415fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 0x7f539415fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 >> v2:172.21.15.177:3300/0 conn(0x7f53940ced20 msgr2=0x7f539415ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 --2- 172.21.15.114:0/2068798216 >> v2:172.21.15.177:3300/0 conn(0x7f53940ced20 0x7f539415ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940cf7a0 con 0x7f53940712d0 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f538c0019f0 con 0x7f53940712d0 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 >> v1:172.21.15.114:6789/0 conn(0x7f53940712d0 legacy=0x7f5394071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378003a20 0x7f5378003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 0x7f5378004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5378004d40 0x7f53780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5394070e40 con 0x7f5378004d40 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53940cf4c0 con 0x7f5378003a20 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537800ba40 con 0x7f5394145980 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.963+0000 7f5392800640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5378004d40 0x7f53780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.964+0000 7f5391400640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 0x7f5378004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.964+0000 7f5391e00640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378003a20 0x7f5378003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.964+0000 7f5392800640 1 -- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 msgr2=0x7f5378004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.964+0000 7f5392800640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5394145980 0x7f5378004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.964+0000 7f5392800640 1 -- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378003a20 msgr2=0x7f5378003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.964+0000 7f5392800640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378003a20 0x7f5378003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.964+0000 7f5392800640 1 -- 172.21.15.114:0/2068798216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5378003520 con 0x7f5378004d40 2024-03-06T07:32:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.965+0000 7f5392800640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5378004d40 0x7f53780051c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f538c00a8a0 tx=0x7f5378005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c4ad67fd99beb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.965+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c00a4c0 con 0x7f5378004d40 2024-03-06T07:32:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.965+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c034650 con 0x7f5378004d40 2024-03-06T07:32:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.965+0000 7f5390a00640 1 -- 172.21.15.114:0/2068798216 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c034a40 con 0x7f5378004d40 2024-03-06T07:32:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.966+0000 7f5399200640 1 -- 172.21.15.114:0/2068798216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5378004d40 msgr2=0x7f53780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.966+0000 7f5399200640 1 --2- 172.21.15.114:0/2068798216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5378004d40 0x7f53780051c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f538c00a8a0 tx=0x7f5378005830 comp rx=0 tx=0).stop 2024-03-06T07:32:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.966+0000 7f5392800640 1 -- 172.21.15.114:0/2068798216 reap_dead start 2024-03-06T07:32:54.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.966+0000 7f5399200640 1 -- 172.21.15.114:0/2068798216 shutdown_connections 2024-03-06T07:32:54.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.966+0000 7f5399200640 1 -- 172.21.15.114:0/2068798216 >> 172.21.15.114:0/2068798216 conn(0x7f53940cfa80 msgr2=0x7f53940cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.966+0000 7f5399200640 1 -- 172.21.15.114:0/2068798216 shutdown_connections 2024-03-06T07:32:54.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.966+0000 7f5399200640 1 -- 172.21.15.114:0/2068798216 wait complete. 2024-03-06T07:32:54.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.968+0000 7f5399200640 1 Processor -- start 2024-03-06T07:32:54.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.968+0000 7f5399200640 1 -- start start 2024-03-06T07:32:54.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5399200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378004d40 0x7f53940ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5399200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f539413d2b0 0x7f539413d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5399200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53940b39f0 0x7f53940b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5399200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53941608c0 con 0x7f53940b39f0 2024-03-06T07:32:54.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5399200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5394160740 con 0x7f5378004d40 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5399200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539415fa00 con 0x7f539413d2b0 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53940b39f0 0x7f53940b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53940b39f0 0x7f53940b3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37646/0 (socket says 172.21.15.114:37646) 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 -- 172.21.15.114:0/479125321 learned_addr learned my addr 172.21.15.114:0/479125321 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5391e00640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378004d40 0x7f53940ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5391400640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f539413d2b0 0x7f539413d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 -- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f539413d2b0 msgr2=0x7f539413d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f539413d2b0 0x7f539413d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 -- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378004d40 msgr2=0x7f53940ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378004d40 0x7f53940ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5392800640 1 -- 172.21.15.114:0/479125321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940ae8c0 con 0x7f53940b39f0 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.969+0000 7f5391e00640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378004d40 0x7f53940ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:54.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.970+0000 7f5392800640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53940b39f0 0x7f53940b3e50 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f538c00a9c0 tx=0x7f538c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.970+0000 7f538a000640 1 -- 172.21.15.114:0/479125321 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c043070 con 0x7f53940b39f0 2024-03-06T07:32:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.970+0000 7f538a000640 1 -- 172.21.15.114:0/479125321 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c0019b0 con 0x7f53940b39f0 2024-03-06T07:32:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.970+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53940aeba0 con 0x7f53940b39f0 2024-03-06T07:32:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.970+0000 7f538a000640 1 -- 172.21.15.114:0/479125321 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c039040 con 0x7f53940b39f0 2024-03-06T07:32:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.970+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53940af160 con 0x7f53940b39f0 2024-03-06T07:32:54.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.971+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53940d0390 con 0x7f53940b39f0 2024-03-06T07:32:54.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.972+0000 7f538a000640 1 -- 172.21.15.114:0/479125321 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f538c046070 con 0x7f53940b39f0 2024-03-06T07:32:54.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.973+0000 7f538a000640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.973+0000 7f538a000640 1 -- 172.21.15.114:0/479125321 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f538c0ab3d0 con 0x7f53940b39f0 2024-03-06T07:32:54.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.973+0000 7f5391e00640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.974+0000 7f5391e00640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f537406b4d0 0x7f537406d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5384006fd0 tx=0x7f5384008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:54.978+0000 7f538a000640 1 -- 172.21.15.114:0/479125321 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53940d0390 con 0x7f53940b39f0 2024-03-06T07:32:55.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.174+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f539404f330 con 0x7f53940b39f0 2024-03-06T07:32:55.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.749+0000 7f538a000640 1 -- 172.21.15.114:0/479125321 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f538c080080 con 0x7f53940b39f0 2024-03-06T07:32:55.752 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.754+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f537406b4d0 msgr2=0x7f537406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.754+0000 7f5399200640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f537406b4d0 0x7f537406d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5384006fd0 tx=0x7f5384008040 comp rx=0 tx=0).stop 2024-03-06T07:32:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.754+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53940b39f0 msgr2=0x7f53940b3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.754+0000 7f5399200640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53940b39f0 0x7f53940b3e50 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f538c00a9c0 tx=0x7f538c001dc0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.755+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 shutdown_connections 2024-03-06T07:32:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.755+0000 7f5399200640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.755+0000 7f5399200640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53940b39f0 0x7f53940b3e50 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.755+0000 7f5399200640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f539413d2b0 0x7f539413d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.755+0000 7f5399200640 1 --2- 172.21.15.114:0/479125321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378004d40 0x7f53940ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.755+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 >> 172.21.15.114:0/479125321 conn(0x7f53940cfa80 msgr2=0x7f53940b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.761+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 shutdown_connections 2024-03-06T07:32:55.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.761+0000 7f5399200640 1 -- 172.21.15.114:0/479125321 wait complete. 2024-03-06T07:32:55.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.968+0000 7f1a53200640 1 Processor -- start 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.968+0000 7f1a53200640 1 -- start start 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a53200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1a4c0b2c40 0x7f1a4c15f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a53200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a4c06d960 0x7f1a4c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a53200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c0b3560 con 0x7f1a4c06e2f0 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a53200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c15f700 con 0x7f1a4c06d960 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a53200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c06d4d0 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a43400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a4c06d960 0x7f1a4c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a43400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a4c06d960 0x7f1a4c0b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37658/0 (socket says 172.21.15.114:37658) 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.970+0000 7f1a43400640 1 -- 172.21.15.114:0/2497818399 learned_addr learned my addr 172.21.15.114:0/2497818399 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.971+0000 7f1a43400640 1 -- 172.21.15.114:0/2497818399 >> v2:172.21.15.177:3300/0 conn(0x7f1a4c0b2c40 msgr2=0x7f1a4c15f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.971+0000 7f1a43400640 1 --2- 172.21.15.114:0/2497818399 >> v2:172.21.15.177:3300/0 conn(0x7f1a4c0b2c40 0x7f1a4c15f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.971+0000 7f1a43400640 1 -- 172.21.15.114:0/2497818399 >> v1:172.21.15.177:6789/0 conn(0x7f1a4c06e2f0 legacy=0x7f1a4c161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.971+0000 7f1a43400640 1 -- 172.21.15.114:0/2497818399 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c06cba0 con 0x7f1a4c06d960 2024-03-06T07:32:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.972+0000 7f1a43400640 1 --2- 172.21.15.114:0/2497818399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c06d960 0x7f1a4c0b3020 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c06c6e0 tx=0x7f1a3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf78cd682a44e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.972+0000 7f1a4b400640 1 -- 172.21.15.114:0/2497818399 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34035020 con 0x7f1a4c06d960 2024-03-06T07:32:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.972+0000 7f1a4b400640 1 -- 172.21.15.114:0/2497818399 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a34031870 con 0x7f1a4c06d960 2024-03-06T07:32:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.972+0000 7f1a4b400640 1 -- 172.21.15.114:0/2497818399 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34031c40 con 0x7f1a4c06d960 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 -- 172.21.15.114:0/2497818399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c06d960 msgr2=0x7f1a4c0b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 --2- 172.21.15.114:0/2497818399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c06d960 0x7f1a4c0b3020 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c06c6e0 tx=0x7f1a3402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 -- 172.21.15.114:0/2497818399 shutdown_connections 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 --2- 172.21.15.114:0/2497818399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c06d960 0x7f1a4c0b3020 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 --2- 172.21.15.114:0/2497818399 >> v2:172.21.15.177:3300/0 conn(0x7f1a4c0b2c40 0x7f1a4c15f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 -- 172.21.15.114:0/2497818399 >> 172.21.15.114:0/2497818399 conn(0x7f1a4c0b1940 msgr2=0x7f1a4c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 -- 172.21.15.114:0/2497818399 shutdown_connections 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.973+0000 7f1a53200640 1 -- 172.21.15.114:0/2497818399 wait complete. 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.975+0000 7f1a53200640 1 Processor -- start 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.975+0000 7f1a53200640 1 -- start start 2024-03-06T07:32:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a53200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a4c06d960 0x7f1a4c09d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a53200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a4c06e2f0 0x7f1a4c14d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a53200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c0b2c40 0x7f1a4c09ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a53200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c06e780 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a53200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c06d160 con 0x7f1a4c06d960 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a53200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c06e900 con 0x7f1a4c06e2f0 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a51200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c0b2c40 0x7f1a4c09ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a51200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c0b2c40 0x7f1a4c09ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37674/0 (socket says 172.21.15.114:37674) 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a51200640 1 -- 172.21.15.114:0/4019745377 learned_addr learned my addr 172.21.15.114:0/4019745377 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a4be00640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a4c06d960 0x7f1a4c09d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a51200640 1 -- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a4c06e2f0 msgr2=0x7f1a4c14d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.976+0000 7f1a51200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a4c06e2f0 0x7f1a4c14d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a51200640 1 -- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a4c06d960 msgr2=0x7f1a4c09d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a51200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a4c06d960 0x7f1a4c09d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a51200640 1 -- 172.21.15.114:0/4019745377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c14dc10 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a4be00640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a4c06d960 0x7f1a4c09d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a51200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c0b2c40 0x7f1a4c09ac90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c780 tx=0x7f1a4400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a48c00640 1 -- 172.21.15.114:0/4019745377 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44017070 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a48c00640 1 -- 172.21.15.114:0/4019745377 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a44003da0 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c0ca770 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a48c00640 1 -- 172.21.15.114:0/4019745377 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a440052b0 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.977+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c0cad30 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.979+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4c04f330 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.979+0000 7f1a48c00640 1 -- 172.21.15.114:0/4019745377 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a4401a070 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.980+0000 7f1a48c00640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a2806b4d0 0x7f1a2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.980+0000 7f1a48c00640 1 -- 172.21.15.114:0/4019745377 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1a44089510 con 0x7f1a4c0b2c40 2024-03-06T07:32:55.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.980+0000 7f1a4be00640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a2806b4d0 0x7f1a2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.981+0000 7f1a4be00640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a2806b4d0 0x7f1a2806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c006fd0 tx=0x7f1a3c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:55.985+0000 7f1a48c00640 1 -- 172.21.15.114:0/4019745377 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a44058680 con 0x7f1a4c0b2c40 2024-03-06T07:32:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.181+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1a4c0b2530 con 0x7f1a4c0b2c40 2024-03-06T07:32:56.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.187+0000 7f1a48c00640 1 -- 172.21.15.114:0/4019745377 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1a44057f80 con 0x7f1a4c0b2c40 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a2806b4d0 msgr2=0x7f1a2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a2806b4d0 0x7f1a2806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c006fd0 tx=0x7f1a3c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c0b2c40 msgr2=0x7f1a4c09ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c0b2c40 0x7f1a4c09ac90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c780 tx=0x7f1a4400cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 shutdown_connections 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a2806b4d0 0x7f1a2806d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a4c0b2c40 0x7f1a4c09ac90 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a4c06e2f0 0x7f1a4c14d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 --2- 172.21.15.114:0/4019745377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a4c06d960 0x7f1a4c09d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 >> 172.21.15.114:0/4019745377 conn(0x7f1a4c0b1940 msgr2=0x7f1a4c1573b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 shutdown_connections 2024-03-06T07:32:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.190+0000 7f1a53200640 1 -- 172.21.15.114:0/4019745377 wait complete. 2024-03-06T07:32:56.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:32:56.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.419+0000 7f506f400640 1 Processor -- start 2024-03-06T07:32:56.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.419+0000 7f506f400640 1 -- start start 2024-03-06T07:32:56.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.420+0000 7f506f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.420+0000 7f506f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f506815fe30 0x7f50680ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506f400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506806ee70 con 0x7f506815f9d0 2024-03-06T07:32:56.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506f400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506806cef0 con 0x7f506815fe30 2024-03-06T07:32:56.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506f400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506806d1d0 con 0x7f5068144c00 2024-03-06T07:32:56.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f506815fe30 0x7f50680ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f5067e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f506815f9d0 0x7f50680ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43850/0 (socket says 172.21.15.114:43850) 2024-03-06T07:32:56.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f5067e00640 1 -- 172.21.15.114:0/128138042 learned_addr learned my addr 172.21.15.114:0/128138042 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:56.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506ca00640 1 --2- 172.21.15.114:0/128138042 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506d400640 1 -- 172.21.15.114:0/128138042 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 msgr2=0x7f506806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506d400640 1 --2- 172.21.15.114:0/128138042 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506d400640 1 -- 172.21.15.114:0/128138042 >> v1:172.21.15.114:6789/0 conn(0x7f506815f9d0 legacy=0x7f50680ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f506d400640 1 -- 172.21.15.114:0/128138042 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506806d4b0 con 0x7f506815fe30 2024-03-06T07:32:56.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.421+0000 7f5067400640 1 -- 172.21.15.114:0/128138042 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783185672 0 0) 0x7f5054001110 con 0x7f506815f9d0 2024-03-06T07:32:56.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f506d400640 1 --2- 172.21.15.114:0/128138042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815fe30 0x7f50680ce7b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f5060009d90 tx=0x7f506002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d6527a8a28d9d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f5067400640 1 -- 172.21.15.114:0/128138042 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060039070 con 0x7f506815fe30 2024-03-06T07:32:56.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f5067400640 1 -- 172.21.15.114:0/128138042 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506002cc50 con 0x7f506815fe30 2024-03-06T07:32:56.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f5067400640 1 -- 172.21.15.114:0/128138042 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060034440 con 0x7f506815fe30 2024-03-06T07:32:56.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f506f400640 1 -- 172.21.15.114:0/128138042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815fe30 msgr2=0x7f50680ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f506f400640 1 --2- 172.21.15.114:0/128138042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815fe30 0x7f50680ce7b0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f5060009d90 tx=0x7f506002bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f506f400640 1 -- 172.21.15.114:0/128138042 shutdown_connections 2024-03-06T07:32:56.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f506f400640 1 --2- 172.21.15.114:0/128138042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815fe30 0x7f50680ce7b0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f506f400640 1 --2- 172.21.15.114:0/128138042 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.423+0000 7f506f400640 1 -- 172.21.15.114:0/128138042 >> 172.21.15.114:0/128138042 conn(0x7f50680b1730 msgr2=0x7f50680b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.424+0000 7f506f400640 1 -- 172.21.15.114:0/128138042 shutdown_connections 2024-03-06T07:32:56.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.424+0000 7f506f400640 1 -- 172.21.15.114:0/128138042 wait complete. 2024-03-06T07:32:56.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.425+0000 7f506f400640 1 Processor -- start 2024-03-06T07:32:56.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.425+0000 7f506f400640 1 -- start start 2024-03-06T07:32:56.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.426+0000 7f506f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.426+0000 7f506f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815f9d0 0x7f506815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.426+0000 7f506f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f506815fe30 0x7f506815cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.426+0000 7f506f400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068160730 con 0x7f506815f9d0 2024-03-06T07:32:56.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.426+0000 7f506f400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068160280 con 0x7f506815fe30 2024-03-06T07:32:56.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.426+0000 7f506f400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506806cb50 con 0x7f5068144c00 2024-03-06T07:32:56.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815f9d0 0x7f506815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815f9d0 0x7f506815c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37694/0 (socket says 172.21.15.114:37694) 2024-03-06T07:32:56.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 -- 172.21.15.114:0/2050411977 learned_addr learned my addr 172.21.15.114:0/2050411977 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:56.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f506ca00640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f506d400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f506815fe30 0x7f506815cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 -- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 msgr2=0x7f506815b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506815b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 -- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f506815fe30 msgr2=0x7f506815cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f506815fe30 0x7f506815cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 -- 172.21.15.114:0/2050411977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506815d5d0 con 0x7f506815f9d0 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f506d400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f506815fe30 0x7f506815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.427+0000 7f5067e00640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815f9d0 0x7f506815c860 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f505400c780 tx=0x7f505400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.428+0000 7f5064c00640 1 -- 172.21.15.114:0/2050411977 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5054018070 con 0x7f506815f9d0 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.428+0000 7f5064c00640 1 -- 172.21.15.114:0/2050411977 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50540072d0 con 0x7f506815f9d0 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.428+0000 7f5064c00640 1 -- 172.21.15.114:0/2050411977 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50540047e0 con 0x7f506815f9d0 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.428+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506815d8b0 con 0x7f506815f9d0 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.428+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506815de70 con 0x7f506815f9d0 2024-03-06T07:32:56.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.429+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50680cd9a0 con 0x7f506815f9d0 2024-03-06T07:32:56.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.430+0000 7f5064c00640 1 -- 172.21.15.114:0/2050411977 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5054004980 con 0x7f506815f9d0 2024-03-06T07:32:56.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.431+0000 7f5064c00640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f504806b4d0 0x7f504806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.431+0000 7f5064c00640 1 -- 172.21.15.114:0/2050411977 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5054089360 con 0x7f506815f9d0 2024-03-06T07:32:56.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.431+0000 7f506ca00640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f504806b4d0 0x7f504806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.432+0000 7f506ca00640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f504806b4d0 0x7f504806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.436+0000 7f5064c00640 1 -- 172.21.15.114:0/2050411977 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50680cd9a0 con 0x7f506815f9d0 2024-03-06T07:32:56.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.710+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f5068072890 con 0x7f506815f9d0 2024-03-06T07:32:56.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.752+0000 7f5064c00640 1 -- 172.21.15.114:0/2050411977 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v280) v1 ==== 132+0+0 (secure 0 0 0) 0x7f506806ee70 con 0x7f506815f9d0 2024-03-06T07:32:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f504806b4d0 msgr2=0x7f504806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f504806b4d0 0x7f504806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815f9d0 msgr2=0x7f506815c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815f9d0 0x7f506815c860 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f505400c780 tx=0x7f505400cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 shutdown_connections 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f506815fe30 0x7f506815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f504806b4d0 0x7f504806d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f506815f9d0 0x7f506815c860 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 --2- 172.21.15.114:0/2050411977 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5068144c00 0x7f506815b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.755+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 >> 172.21.15.114:0/2050411977 conn(0x7f50680b1730 msgr2=0x7f50680ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.756+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 shutdown_connections 2024-03-06T07:32:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.756+0000 7f506f400640 1 -- 172.21.15.114:0/2050411977 wait complete. 2024-03-06T07:32:56.776 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:32:56.776 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:56.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:56.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.979+0000 7f2c86000640 1 Processor -- start 2024-03-06T07:32:56.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.979+0000 7f2c86000640 1 -- start start 2024-03-06T07:32:56.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.980+0000 7f2c86000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c800ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.980+0000 7f2c86000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2c800cc9d0 0x7f2c8015f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.980+0000 7f2c86000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8015c240 con 0x7f2c800aebf0 2024-03-06T07:32:56.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.980+0000 7f2c86000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8015ce30 con 0x7f2c800cc9d0 2024-03-06T07:32:56.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.980+0000 7f2c86000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8015c500 con 0x7f2c801561f0 2024-03-06T07:32:56.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.981+0000 7f2c84c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c800ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.981+0000 7f2c84c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c800ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58392/0 (socket says 172.21.15.114:58392) 2024-03-06T07:32:56.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.981+0000 7f2c84c00640 1 -- 172.21.15.114:0/608214831 learned_addr learned my addr 172.21.15.114:0/608214831 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:56.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.981+0000 7f2c85600640 1 --2- 172.21.15.114:0/608214831 >> v2:172.21.15.177:3300/0 conn(0x7f2c800cc9d0 0x7f2c8015f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.982+0000 7f2c7fe00640 1 -- 172.21.15.114:0/608214831 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198669160 0 0) 0x7f2c8015c240 con 0x7f2c800aebf0 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.982+0000 7f2c7fe00640 1 -- 172.21.15.114:0/608214831 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c64003520 con 0x7f2c800aebf0 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.982+0000 7f2c85600640 1 -- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 msgr2=0x7f2c800ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.982+0000 7f2c85600640 1 --2- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c800ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.982+0000 7f2c85600640 1 -- 172.21.15.114:0/608214831 >> v1:172.21.15.177:6789/0 conn(0x7f2c800aebf0 legacy=0x7f2c800cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.982+0000 7f2c85600640 1 -- 172.21.15.114:0/608214831 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8015c7e0 con 0x7f2c800cc9d0 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.982+0000 7f2c84c00640 1 --2- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c800ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.983+0000 7f2c85600640 1 --2- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015f830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009d90 tx=0x7f2c7802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aba93e8f3a3116 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.983+0000 7f2c7fe00640 1 -- 172.21.15.114:0/608214831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78039070 con 0x7f2c800cc9d0 2024-03-06T07:32:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c7fe00640 1 -- 172.21.15.114:0/608214831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7802cc50 con 0x7f2c800cc9d0 2024-03-06T07:32:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c7fe00640 1 -- 172.21.15.114:0/608214831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78034440 con 0x7f2c800cc9d0 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 -- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 msgr2=0x7f2c8015f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 --2- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015f830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009d90 tx=0x7f2c7802bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 -- 172.21.15.114:0/608214831 shutdown_connections 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 --2- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015f830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 --2- 172.21.15.114:0/608214831 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c800ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 -- 172.21.15.114:0/608214831 >> 172.21.15.114:0/608214831 conn(0x7f2c8006afc0 msgr2=0x7f2c8006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 -- 172.21.15.114:0/608214831 shutdown_connections 2024-03-06T07:32:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.984+0000 7f2c86000640 1 -- 172.21.15.114:0/608214831 wait complete. 2024-03-06T07:32:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.986+0000 7f2c86000640 1 Processor -- start 2024-03-06T07:32:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.986+0000 7f2c86000640 1 -- start start 2024-03-06T07:32:56.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c86000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c800aebf0 0x7f2c80145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c86000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c86000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c8015b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c86000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80156750 con 0x7f2c800aebf0 2024-03-06T07:32:56.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c86000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c801565d0 con 0x7f2c800cc9d0 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c86000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8015fd70 con 0x7f2c801561f0 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c84c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c800aebf0 0x7f2c80145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c84c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c800aebf0 0x7f2c80145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37708/0 (socket says 172.21.15.114:37708) 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.987+0000 7f2c84c00640 1 -- 172.21.15.114:0/1734996601 learned_addr learned my addr 172.21.15.114:0/1734996601 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c84c00640 1 -- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 msgr2=0x7f2c8015b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c84c00640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c8015b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c77e00640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c84c00640 1 -- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 msgr2=0x7f2c8015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c84c00640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c84c00640 1 -- 172.21.15.114:0/1734996601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c80156c30 con 0x7f2c800aebf0 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c77e00640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:56.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c84c00640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c800aebf0 0x7f2c80145ef0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000e9b0 tx=0x7f2c7000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c7d600640 1 -- 172.21.15.114:0/1734996601 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70017070 con 0x7f2c800aebf0 2024-03-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c7d600640 1 -- 172.21.15.114:0/1734996601 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7000c9a0 con 0x7f2c800aebf0 2024-03-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c7d600640 1 -- 172.21.15.114:0/1734996601 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000cd50 con 0x7f2c800aebf0 2024-03-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.988+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c80156f10 con 0x7f2c800aebf0 2024-03-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.989+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c801574d0 con 0x7f2c800aebf0 2024-03-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.990+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8004f330 con 0x7f2c800aebf0 2024-03-06T07:32:56.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.991+0000 7f2c7d600640 1 -- 172.21.15.114:0/1734996601 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c7001a070 con 0x7f2c800aebf0 2024-03-06T07:32:56.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.991+0000 7f2c7d600640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5c06b4d0 0x7f2c5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.992+0000 7f2c7d600640 1 -- 172.21.15.114:0/1734996601 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2c7008a1e0 con 0x7f2c800aebf0 2024-03-06T07:32:56.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.992+0000 7f2c77e00640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5c06b4d0 0x7f2c5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.993+0000 7f2c77e00640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5c06b4d0 0x7f2c5c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2c8015be70 tx=0x7f2c68009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:56.996+0000 7f2c7d600640 1 -- 172.21.15.114:0/1734996601 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c700592d0 con 0x7f2c800aebf0 2024-03-06T07:32:57.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.194+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c8006ce70 con 0x7f2c800aebf0 2024-03-06T07:32:57.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.197+0000 7f2c7d600640 1 -- 172.21.15.114:0/1734996601 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13071 (secure 0 0 0) 0x7f2c70058bd0 con 0x7f2c800aebf0 2024-03-06T07:32:57.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:32:57.200 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":85,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:32:55.737009+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6837/2399499569":"2024-03-07T07:32:52.719513+0000","172.21.15.114:6835/2399499569":"2024-03-07T07:32:52.719513+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:57.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5c06b4d0 msgr2=0x7f2c5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5c06b4d0 0x7f2c5c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2c8015be70 tx=0x7f2c68009040 comp rx=0 tx=0).stop 2024-03-06T07:32:57.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c800aebf0 msgr2=0x7f2c80145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c800aebf0 0x7f2c80145ef0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000e9b0 tx=0x7f2c7000ee70 comp rx=0 tx=0).stop 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 shutdown_connections 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c801561f0 0x7f2c8015b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c800cc9d0 0x7f2c8015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c5c06b4d0 0x7f2c5c06d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 --2- 172.21.15.114:0/1734996601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c800aebf0 0x7f2c80145ef0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 >> 172.21.15.114:0/1734996601 conn(0x7f2c8006afc0 msgr2=0x7f2c80155190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 shutdown_connections 2024-03-06T07:32:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.199+0000 7f2c86000640 1 -- 172.21.15.114:0/1734996601 wait complete. 2024-03-06T07:32:57.221 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:57.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:57.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.433+0000 7f08c2800640 1 Processor -- start 2024-03-06T07:32:57.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.433+0000 7f08c2800640 1 -- start start 2024-03-06T07:32:57.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.435+0000 7f08c2800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f08bc0b3f70 0x7f08bc160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.435+0000 7f08c2800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc071440 con 0x7f08bc070ff0 2024-03-06T07:32:57.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.435+0000 7f08c2800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0b46c0 con 0x7f08bc15fc10 2024-03-06T07:32:57.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.435+0000 7f08c2800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0b49a0 con 0x7f08bc0b3f70 2024-03-06T07:32:57.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.435+0000 7f08bbe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f08bc070ff0 0x7f08bc0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43860/0 (socket says 172.21.15.114:43860) 2024-03-06T07:32:57.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.435+0000 7f08bbe00640 1 -- 172.21.15.114:0/4065408347 learned_addr learned my addr 172.21.15.114:0/4065408347 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08b2a00640 1 --2- 172.21.15.114:0/4065408347 >> v2:172.21.15.177:3300/0 conn(0x7f08bc0b3f70 0x7f08bc160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349890229 0 0) 0x7f08bc071440 con 0x7f08bc070ff0 2024-03-06T07:32:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a0003520 con 0x7f08bc070ff0 2024-03-06T07:32:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915779957 0 0) 0x7f08a0003520 con 0x7f08bc070ff0 2024-03-06T07:32:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 >> v2:172.21.15.177:3300/0 conn(0x7f08bc0b3f70 msgr2=0x7f08bc160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 --2- 172.21.15.114:0/4065408347 >> v2:172.21.15.177:3300/0 conn(0x7f08bc0b3f70 0x7f08bc160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 >> v1:172.21.15.177:6789/0 conn(0x7f08bc15fc10 legacy=0x7f08bc15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc072c90 con 0x7f08bc070ff0 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f08b40019f0 con 0x7f08bc070ff0 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 >> v1:172.21.15.114:6789/0 conn(0x7f08bc070ff0 legacy=0x7f08bc0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08a0003a00 0x7f08a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a0004800 0x7f08a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0b49a0 con 0x7f08a00051c0 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a000c0c0 con 0x7f08a0004800 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.436+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a000c2c0 con 0x7f08a0003a00 2024-03-06T07:32:57.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bbe00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08b2a00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a0004800 0x7f08a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bb400640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08a0003a00 0x7f08a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bbe00640 1 -- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08a0003a00 msgr2=0x7f08a0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bbe00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08a0003a00 0x7f08a0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bbe00640 1 -- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a0004800 msgr2=0x7f08a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bbe00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a0004800 0x7f08a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bbe00640 1 -- 172.21.15.114:0/4065408347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a0003520 con 0x7f08a00051c0 2024-03-06T07:32:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.437+0000 7f08bb400640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08a0003a00 0x7f08a0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.438+0000 7f08bbe00640 1 -- 172.21.15.114:0/4065408347 reap_dead start 2024-03-06T07:32:57.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.438+0000 7f08bbe00640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08a000b9c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f08b400a7b0 tx=0x7f08a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f50ade7f67b429 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b400a3d0 con 0x7f08a00051c0 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b4034650 con 0x7f08a00051c0 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08baa00640 1 -- 172.21.15.114:0/4065408347 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b4034a40 con 0x7f08a00051c0 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08c2800640 1 -- 172.21.15.114:0/4065408347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 msgr2=0x7f08a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08c2800640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08a000b9c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f08b400a7b0 tx=0x7f08a0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08c2800640 1 -- 172.21.15.114:0/4065408347 shutdown_connections 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08c2800640 1 --2- 172.21.15.114:0/4065408347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08a000b9c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08c2800640 1 -- 172.21.15.114:0/4065408347 >> 172.21.15.114:0/4065408347 conn(0x7f08bc0b1940 msgr2=0x7f08bc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08c2800640 1 -- 172.21.15.114:0/4065408347 shutdown_connections 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.439+0000 7f08c2800640 1 -- 172.21.15.114:0/4065408347 wait complete. 2024-03-06T07:32:57.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.441+0000 7f08c2800640 1 Processor -- start 2024-03-06T07:32:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.441+0000 7f08c2800640 1 -- start start 2024-03-06T07:32:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08c2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08bc09bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08c2800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08bc13d8b0 0x7f08bc15c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08c2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08bc09c400 0x7f08bc15cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08c2800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc072a30 con 0x7f08a00051c0 2024-03-06T07:32:57.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08c2800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0728b0 con 0x7f08bc09c400 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08c2800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc15c2e0 con 0x7f08bc13d8b0 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08bc09bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08bc09bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37726/0 (socket says 172.21.15.114:37726) 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.442+0000 7f08bb400640 1 -- 172.21.15.114:0/2693358467 learned_addr learned my addr 172.21.15.114:0/2693358467 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08bb400640 1 -- 172.21.15.114:0/2693358467 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08bc13d8b0 msgr2=0x7f08bc15c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08b2a00640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08bc13d8b0 0x7f08bc15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08bb400640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08bc13d8b0 0x7f08bc15c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08bb400640 1 -- 172.21.15.114:0/2693358467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08bc09c400 msgr2=0x7f08bc15cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08bb400640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08bc09c400 0x7f08bc15cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08bb400640 1 -- 172.21.15.114:0/2693358467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc09ae20 con 0x7f08a00051c0 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08bb400640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08bc09bec0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d7d0 tx=0x7f08ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08b3e00640 1 -- 172.21.15.114:0/2693358467 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac005270 con 0x7f08a00051c0 2024-03-06T07:32:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08b3e00640 1 -- 172.21.15.114:0/2693358467 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac017070 con 0x7f08a00051c0 2024-03-06T07:32:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08b3e00640 1 -- 172.21.15.114:0/2693358467 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac00b6b0 con 0x7f08a00051c0 2024-03-06T07:32:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.443+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc09b100 con 0x7f08a00051c0 2024-03-06T07:32:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.444+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc09b6c0 con 0x7f08a00051c0 2024-03-06T07:32:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.445+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc04f330 con 0x7f08a00051c0 2024-03-06T07:32:57.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.445+0000 7f08b3e00640 1 -- 172.21.15.114:0/2693358467 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08ac00b850 con 0x7f08a00051c0 2024-03-06T07:32:57.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.446+0000 7f08b3e00640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f089806b500 0x7f089806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.446+0000 7f08b3e00640 1 -- 172.21.15.114:0/2693358467 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f08ac089a70 con 0x7f08a00051c0 2024-03-06T07:32:57.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.447+0000 7f08b2a00640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f089806b500 0x7f089806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.447+0000 7f08b2a00640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f089806b500 0x7f089806d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f08bc150590 tx=0x7f08a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.451+0000 7f08b3e00640 1 -- 172.21.15.114:0/2693358467 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08ac021020 con 0x7f08a00051c0 2024-03-06T07:32:57.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.645+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08bc0cc410 con 0x7f08a00051c0 2024-03-06T07:32:57.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.645+0000 7f08b3e00640 1 -- 172.21.15.114:0/2693358467 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08ac058cd0 con 0x7f08a00051c0 2024-03-06T07:32:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f089806b500 msgr2=0x7f089806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f089806b500 0x7f089806d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f08bc150590 tx=0x7f08a4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 msgr2=0x7f08bc09bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08bc09bec0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00d7d0 tx=0x7f08ac00dc90 comp rx=0 tx=0).stop 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 shutdown_connections 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08bc09c400 0x7f08bc15cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08bc13d8b0 0x7f08bc15c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f089806b500 0x7f089806d9c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 --2- 172.21.15.114:0/2693358467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a00051c0 0x7f08bc09bec0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 >> 172.21.15.114:0/2693358467 conn(0x7f08bc0b1940 msgr2=0x7f08bc15fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 shutdown_connections 2024-03-06T07:32:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.648+0000 7f08c2800640 1 -- 172.21.15.114:0/2693358467 wait complete. 2024-03-06T07:32:57.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:32:57.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.858+0000 7fe615200640 1 Processor -- start 2024-03-06T07:32:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.858+0000 7fe615200640 1 -- start start 2024-03-06T07:32:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.859+0000 7fe615200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe610144c00 0x7fe61006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.859+0000 7fe615200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe61015f9d0 0x7fe61015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.859+0000 7fe615200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6100ce0a0 0x7fe6100ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.859+0000 7fe615200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61006ee70 con 0x7fe610144c00 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.859+0000 7fe615200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61006cef0 con 0x7fe61015f9d0 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.859+0000 7fe615200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61006d1d0 con 0x7fe6100ce0a0 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe610144c00 0x7fe61006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe610144c00 0x7fe61006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41932/0 (socket says 172.21.15.114:41932) 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 -- 172.21.15.114:0/44640869 learned_addr learned my addr 172.21.15.114:0/44640869 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe605400640 1 --2- 172.21.15.114:0/44640869 >> v2:172.21.15.177:3300/0 conn(0x7fe61015f9d0 0x7fe61015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60e800640 1 --2- 172.21.15.114:0/44640869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6100ce0a0 0x7fe6100ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 -- 172.21.15.114:0/44640869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6100ce0a0 msgr2=0x7fe6100ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 --2- 172.21.15.114:0/44640869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6100ce0a0 0x7fe6100ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 -- 172.21.15.114:0/44640869 >> v2:172.21.15.177:3300/0 conn(0x7fe61015f9d0 msgr2=0x7fe61015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 --2- 172.21.15.114:0/44640869 >> v2:172.21.15.177:3300/0 conn(0x7fe61015f9d0 0x7fe61015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.860+0000 7fe60de00640 1 -- 172.21.15.114:0/44640869 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61006d4b0 con 0x7fe610144c00 2024-03-06T07:32:57.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.861+0000 7fe60de00640 1 --2- 172.21.15.114:0/44640869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe610144c00 0x7fe61006e930 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe6100b26f0 tx=0x7fe60002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10e0a675d65c1d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe60d400640 1 -- 172.21.15.114:0/44640869 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600035020 con 0x7fe610144c00 2024-03-06T07:32:57.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe60d400640 1 -- 172.21.15.114:0/44640869 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600031870 con 0x7fe610144c00 2024-03-06T07:32:57.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe60d400640 1 -- 172.21.15.114:0/44640869 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600031c40 con 0x7fe610144c00 2024-03-06T07:32:57.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe615200640 1 -- 172.21.15.114:0/44640869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe610144c00 msgr2=0x7fe61006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe615200640 1 --2- 172.21.15.114:0/44640869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe610144c00 0x7fe61006e930 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe6100b26f0 tx=0x7fe60002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe615200640 1 -- 172.21.15.114:0/44640869 shutdown_connections 2024-03-06T07:32:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe615200640 1 --2- 172.21.15.114:0/44640869 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6100ce0a0 0x7fe6100ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe615200640 1 --2- 172.21.15.114:0/44640869 >> v2:172.21.15.177:3300/0 conn(0x7fe61015f9d0 0x7fe61015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe615200640 1 --2- 172.21.15.114:0/44640869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe610144c00 0x7fe61006e930 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.862+0000 7fe615200640 1 -- 172.21.15.114:0/44640869 >> 172.21.15.114:0/44640869 conn(0x7fe6100b1730 msgr2=0x7fe6100b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.863+0000 7fe615200640 1 -- 172.21.15.114:0/44640869 shutdown_connections 2024-03-06T07:32:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.863+0000 7fe615200640 1 -- 172.21.15.114:0/44640869 wait complete. 2024-03-06T07:32:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.864+0000 7fe615200640 1 Processor -- start 2024-03-06T07:32:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.865+0000 7fe615200640 1 -- start start 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.865+0000 7fe615200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6100ce0a0 0x7fe6101f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.865+0000 7fe615200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe610144c00 0x7fe61015cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.865+0000 7fe615200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe61015f9d0 0x7fe61015d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.865+0000 7fe615200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61006cb50 con 0x7fe6100ce0a0 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe615200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe610160730 con 0x7fe610144c00 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe615200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61006ccd0 con 0x7fe61015f9d0 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6100ce0a0 0x7fe6101f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6100ce0a0 0x7fe6101f35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41944/0 (socket says 172.21.15.114:41944) 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 -- 172.21.15.114:0/818066905 learned_addr learned my addr 172.21.15.114:0/818066905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe605400640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe610144c00 0x7fe61015cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 -- 172.21.15.114:0/818066905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe61015f9d0 msgr2=0x7fe61015d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe61015f9d0 0x7fe61015d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 -- 172.21.15.114:0/818066905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe610144c00 msgr2=0x7fe61015cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe610144c00 0x7fe61015cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.866+0000 7fe60de00640 1 -- 172.21.15.114:0/818066905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6101f3ae0 con 0x7fe6100ce0a0 2024-03-06T07:32:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.867+0000 7fe60de00640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6100ce0a0 0x7fe6101f35a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe60002b950 tx=0x7fe6000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.867+0000 7fe606a00640 1 -- 172.21.15.114:0/818066905 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600040070 con 0x7fe6100ce0a0 2024-03-06T07:32:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.867+0000 7fe606a00640 1 -- 172.21.15.114:0/818066905 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600030d00 con 0x7fe6100ce0a0 2024-03-06T07:32:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.867+0000 7fe606a00640 1 -- 172.21.15.114:0/818066905 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600035040 con 0x7fe6100ce0a0 2024-03-06T07:32:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.867+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe610143080 con 0x7fe6100ce0a0 2024-03-06T07:32:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.867+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6101403f0 con 0x7fe6100ce0a0 2024-03-06T07:32:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.868+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61004f330 con 0x7fe6100ce0a0 2024-03-06T07:32:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.869+0000 7fe606a00640 1 -- 172.21.15.114:0/818066905 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe60004b060 con 0x7fe6100ce0a0 2024-03-06T07:32:57.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.870+0000 7fe606a00640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.870+0000 7fe606a00640 1 -- 172.21.15.114:0/818066905 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe6000abd30 con 0x7fe6100ce0a0 2024-03-06T07:32:57.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.871+0000 7fe605400640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.871+0000 7fe605400640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe6100cf480 tx=0x7fe5f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:57.875+0000 7fe606a00640 1 -- 172.21.15.114:0/818066905 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6101f3ae0 con 0x7fe6100ce0a0 2024-03-06T07:32:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.121+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fe610143b10 con 0x7fe6100ce0a0 2024-03-06T07:32:58.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.764+0000 7fe606a00640 1 -- 172.21.15.114:0/818066905 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v282) v1 ==== 137+0+0 (secure 0 0 0) 0x7fe60007ae10 con 0x7fe6100ce0a0 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.767+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5ec06b4d0 msgr2=0x7fe5ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.767+0000 7fe615200640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe6100cf480 tx=0x7fe5f8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6100ce0a0 msgr2=0x7fe6101f35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6100ce0a0 0x7fe6101f35a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe60002b950 tx=0x7fe6000046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 shutdown_connections 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe61015f9d0 0x7fe61015d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe610144c00 0x7fe61015cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5ec06b4d0 0x7fe5ec06d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 --2- 172.21.15.114:0/818066905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6100ce0a0 0x7fe6101f35a0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 >> 172.21.15.114:0/818066905 conn(0x7fe6100b1730 msgr2=0x7fe6100ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 shutdown_connections 2024-03-06T07:32:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.768+0000 7fe615200640 1 -- 172.21.15.114:0/818066905 wait complete. 2024-03-06T07:32:58.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:32:58.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.990+0000 7fecb6400640 1 Processor -- start 2024-03-06T07:32:58.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.990+0000 7fecb6400640 1 -- start start 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.992+0000 7fecb6400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fecb006da40 0x7fecb00ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.992+0000 7fecb6400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fecb00aef20 0x7fecb00ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.992+0000 7fecb6400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb015c1f0 con 0x7fecb015be10 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.992+0000 7fecb6400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb00cd200 con 0x7fecb006da40 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.992+0000 7fecb6400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb006cac0 con 0x7fecb00aef20 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7feca7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fecb006da40 0x7fecb00ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7feca7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fecb006da40 0x7fecb00ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41954/0 (socket says 172.21.15.114:41954) 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7feca7e00640 1 -- 172.21.15.114:0/3088787265 learned_addr learned my addr 172.21.15.114:0/3088787265 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7fecb5000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fecb015be10 0x7fecb006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50600/0 (socket says 172.21.15.114:50600) 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7fecb5a00640 1 --2- 172.21.15.114:0/3088787265 >> v2:172.21.15.177:3300/0 conn(0x7fecb00aef20 0x7fecb00ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7feca7e00640 1 -- 172.21.15.114:0/3088787265 >> v2:172.21.15.177:3300/0 conn(0x7fecb00aef20 msgr2=0x7fecb00ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7feca7e00640 1 --2- 172.21.15.114:0/3088787265 >> v2:172.21.15.177:3300/0 conn(0x7fecb00aef20 0x7fecb00ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7feca7e00640 1 -- 172.21.15.114:0/3088787265 >> v1:172.21.15.114:6789/0 conn(0x7fecb015be10 legacy=0x7fecb006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7feca7e00640 1 -- 172.21.15.114:0/3088787265 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb006cda0 con 0x7fecb006da40 2024-03-06T07:32:58.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.993+0000 7fecafe00640 1 -- 172.21.15.114:0/3088787265 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239004132 0 0) 0x7fecb015c1f0 con 0x7fecb015be10 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.994+0000 7feca7e00640 1 --2- 172.21.15.114:0/3088787265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb006da40 0x7fecb00ccc20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fecb00af490 tx=0x7fec9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788b209b89e8c357 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.994+0000 7fecafe00640 1 -- 172.21.15.114:0/3088787265 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98035020 con 0x7fecb006da40 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.994+0000 7fecafe00640 1 -- 172.21.15.114:0/3088787265 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec98031870 con 0x7fecb006da40 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.994+0000 7fecafe00640 1 -- 172.21.15.114:0/3088787265 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98031c40 con 0x7fecb006da40 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 -- 172.21.15.114:0/3088787265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb006da40 msgr2=0x7fecb00ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 --2- 172.21.15.114:0/3088787265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb006da40 0x7fecb00ccc20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fecb00af490 tx=0x7fec9802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 -- 172.21.15.114:0/3088787265 shutdown_connections 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 --2- 172.21.15.114:0/3088787265 >> v2:172.21.15.177:3300/0 conn(0x7fecb00aef20 0x7fecb00ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 --2- 172.21.15.114:0/3088787265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb006da40 0x7fecb00ccc20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 -- 172.21.15.114:0/3088787265 >> 172.21.15.114:0/3088787265 conn(0x7fecb006afc0 msgr2=0x7fecb006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 -- 172.21.15.114:0/3088787265 shutdown_connections 2024-03-06T07:32:58.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.995+0000 7fecb6400640 1 -- 172.21.15.114:0/3088787265 wait complete. 2024-03-06T07:32:58.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.997+0000 7fecb6400640 1 Processor -- start 2024-03-06T07:32:58.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.997+0000 7fecb6400640 1 -- start start 2024-03-06T07:32:59.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb6400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecb006da40 0x7fecb00ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb6400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb00aef20 0x7fecb01420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb6400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb015be10 0x7fecb00b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb6400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb00cda90 con 0x7fecb015be10 2024-03-06T07:32:59.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb6400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb006d080 con 0x7fecb00aef20 2024-03-06T07:32:59.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb6400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb0071ab0 con 0x7fecb006da40 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb015be10 0x7fecb00b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb015be10 0x7fecb00b0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41962/0 (socket says 172.21.15.114:41962) 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.998+0000 7fecb5a00640 1 -- 172.21.15.114:0/3210437838 learned_addr learned my addr 172.21.15.114:0/3210437838 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7feca7e00640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb00aef20 0x7fecb01420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5000640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecb006da40 0x7fecb00ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5a00640 1 -- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecb006da40 msgr2=0x7fecb00ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5a00640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecb006da40 0x7fecb00ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5a00640 1 -- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb00aef20 msgr2=0x7fecb01420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5a00640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb00aef20 0x7fecb01420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5a00640 1 -- 172.21.15.114:0/3210437838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb015e860 con 0x7fecb015be10 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5000640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecb006da40 0x7fecb00ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7feca7e00640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb00aef20 0x7fecb01420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb5a00640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb015be10 0x7fecb00b0660 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7feca800d830 tx=0x7feca800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecad600640 1 -- 172.21.15.114:0/3210437838 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800b6c0 con 0x7fecb015be10 2024-03-06T07:32:59.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecad600640 1 -- 172.21.15.114:0/3210437838 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca8019070 con 0x7fecb015be10 2024-03-06T07:32:59.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb015eb40 con 0x7fecb015be10 2024-03-06T07:32:59.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecad600640 1 -- 172.21.15.114:0/3210437838 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800bac0 con 0x7fecb015be10 2024-03-06T07:32:59.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:58.999+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb015f100 con 0x7fecb015be10 2024-03-06T07:32:59.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.001+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb004f330 con 0x7fecb015be10 2024-03-06T07:32:59.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.001+0000 7fecad600640 1 -- 172.21.15.114:0/3210437838 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feca801e460 con 0x7fecb015be10 2024-03-06T07:32:59.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.002+0000 7fecad600640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec8c06b500 0x7fec8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.002+0000 7fecad600640 1 -- 172.21.15.114:0/3210437838 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feca808a6a0 con 0x7fecb015be10 2024-03-06T07:32:59.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.003+0000 7fecb5000640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec8c06b500 0x7fec8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.003+0000 7fecb5000640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec8c06b500 0x7fec8c06d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7feca0007fd0 tx=0x7feca0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.006+0000 7fecad600640 1 -- 172.21.15.114:0/3210437838 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feca80596f0 con 0x7fecb015be10 2024-03-06T07:32:59.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.203+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fecb00cf900 con 0x7fecb015be10 2024-03-06T07:32:59.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.208+0000 7fecad600640 1 -- 172.21.15.114:0/3210437838 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7feca8058ff0 con 0x7fecb015be10 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec8c06b500 msgr2=0x7fec8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec8c06b500 0x7fec8c06d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7feca0007fd0 tx=0x7feca0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb015be10 msgr2=0x7fecb00b0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb015be10 0x7fecb00b0660 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7feca800d830 tx=0x7feca800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 shutdown_connections 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fec8c06b500 0x7fec8c06d9c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecb015be10 0x7fecb00b0660 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb00aef20 0x7fecb01420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 --2- 172.21.15.114:0/3210437838 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fecb006da40 0x7fecb00ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 >> 172.21.15.114:0/3210437838 conn(0x7fecb006afc0 msgr2=0x7fecb006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 shutdown_connections 2024-03-06T07:32:59.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.211+0000 7fecb6400640 1 -- 172.21.15.114:0/3210437838 wait complete. 2024-03-06T07:32:59.233 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:59.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:59.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:59.460 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:59.464 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:59.464 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:59.519 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:59.519 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:32:59.519 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:59.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:59.575 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:32:59.575 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:59.631 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:59.631 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:32:59.631 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:59.686 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:59.686 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:32:59.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:59.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.898+0000 7fe753a00640 1 Processor -- start 2024-03-06T07:32:59.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.898+0000 7fe753a00640 1 -- start start 2024-03-06T07:32:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.899+0000 7fe753a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe74c06ebe0 0x7fe74c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.899+0000 7fe753a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c071d30 con 0x7fe74c15eff0 2024-03-06T07:32:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe753a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c15f4a0 con 0x7fe74c0af1c0 2024-03-06T07:32:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe753a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74c071260 con 0x7fe74c06ebe0 2024-03-06T07:32:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe751a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe74c06ebe0 0x7fe74c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe74be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe74c15eff0 0x7fe74c06e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50612/0 (socket says 172.21.15.114:50612) 2024-03-06T07:32:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe74be00640 1 -- 172.21.15.114:0/3180942897 learned_addr learned my addr 172.21.15.114:0/3180942897 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:32:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe751a00640 1 --2- 172.21.15.114:0/3180942897 >> v2:172.21.15.114:3300/0 conn(0x7fe74c06ebe0 0x7fe74c070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41974/0 (socket says 172.21.15.114:41974) 2024-03-06T07:32:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe751a00640 1 -- 172.21.15.114:0/3180942897 >> v1:172.21.15.177:6789/0 conn(0x7fe74c0af1c0 legacy=0x7fe74c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe751a00640 1 -- 172.21.15.114:0/3180942897 >> v1:172.21.15.114:6789/0 conn(0x7fe74c15eff0 legacy=0x7fe74c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe751a00640 1 -- 172.21.15.114:0/3180942897 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c071490 con 0x7fe74c06ebe0 2024-03-06T07:32:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.900+0000 7fe74b400640 1 -- 172.21.15.114:0/3180942897 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685135133 0 0) 0x7fe738001110 con 0x7fe74c15eff0 2024-03-06T07:32:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.902+0000 7fe751a00640 1 --2- 172.21.15.114:0/3180942897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c06ebe0 0x7fe74c070d20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fe744009dc0 tx=0x7fe74402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0520668f10808e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.902+0000 7fe74b400640 1 -- 172.21.15.114:0/3180942897 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744039070 con 0x7fe74c06ebe0 2024-03-06T07:32:59.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.902+0000 7fe74b400640 1 -- 172.21.15.114:0/3180942897 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74402cc40 con 0x7fe74c06ebe0 2024-03-06T07:32:59.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.902+0000 7fe74b400640 1 -- 172.21.15.114:0/3180942897 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744034440 con 0x7fe74c06ebe0 2024-03-06T07:32:59.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.902+0000 7fe753a00640 1 -- 172.21.15.114:0/3180942897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c06ebe0 msgr2=0x7fe74c070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.902+0000 7fe753a00640 1 --2- 172.21.15.114:0/3180942897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c06ebe0 0x7fe74c070d20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fe744009dc0 tx=0x7fe74402be90 comp rx=0 tx=0).stop 2024-03-06T07:32:59.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.903+0000 7fe753a00640 1 -- 172.21.15.114:0/3180942897 shutdown_connections 2024-03-06T07:32:59.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.903+0000 7fe753a00640 1 --2- 172.21.15.114:0/3180942897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c06ebe0 0x7fe74c070d20 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.903+0000 7fe753a00640 1 -- 172.21.15.114:0/3180942897 >> 172.21.15.114:0/3180942897 conn(0x7fe74c0b1730 msgr2=0x7fe74c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.903+0000 7fe753a00640 1 -- 172.21.15.114:0/3180942897 shutdown_connections 2024-03-06T07:32:59.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.903+0000 7fe753a00640 1 -- 172.21.15.114:0/3180942897 wait complete. 2024-03-06T07:32:59.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.905+0000 7fe753a00640 1 Processor -- start 2024-03-06T07:32:59.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.905+0000 7fe753a00640 1 -- start start 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe753a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c06ebe0 0x7fe74c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe753a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c0af1c0 0x7fe74c1e7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe753a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c15eff0 0x7fe74c1e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe753a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c0b30b0 con 0x7fe74c0af1c0 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe753a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c071a50 con 0x7fe74c15eff0 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe753a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c06d370 con 0x7fe74c06ebe0 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c0af1c0 0x7fe74c1e7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c0af1c0 0x7fe74c1e7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41976/0 (socket says 172.21.15.114:41976) 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 -- 172.21.15.114:0/2705891357 learned_addr learned my addr 172.21.15.114:0/2705891357 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 -- 172.21.15.114:0/2705891357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c06ebe0 msgr2=0x7fe74c0ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c06ebe0 0x7fe74c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 -- 172.21.15.114:0/2705891357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c15eff0 msgr2=0x7fe74c1e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c15eff0 0x7fe74c1e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.906+0000 7fe74be00640 1 -- 172.21.15.114:0/2705891357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c1f42b0 con 0x7fe74c0af1c0 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.907+0000 7fe74be00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c0af1c0 0x7fe74c1e7ee0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe73800c430 tx=0x7fe73800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.907+0000 7fe748c00640 1 -- 172.21.15.114:0/2705891357 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7380074e0 con 0x7fe74c0af1c0 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.907+0000 7fe748c00640 1 -- 172.21.15.114:0/2705891357 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7380047e0 con 0x7fe74c0af1c0 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.907+0000 7fe748c00640 1 -- 172.21.15.114:0/2705891357 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738004bd0 con 0x7fe74c0af1c0 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.907+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74c15b340 con 0x7fe74c0af1c0 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.907+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c0cec40 con 0x7fe74c0af1c0 2024-03-06T07:32:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.908+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74c15fd90 con 0x7fe74c0af1c0 2024-03-06T07:32:59.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.909+0000 7fe748c00640 1 -- 172.21.15.114:0/2705891357 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe738011070 con 0x7fe74c0af1c0 2024-03-06T07:32:59.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.910+0000 7fe748c00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.911+0000 7fe748c00640 1 -- 172.21.15.114:0/2705891357 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe738089b70 con 0x7fe74c0af1c0 2024-03-06T07:32:59.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.911+0000 7fe751000640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.911+0000 7fe751000640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe72c06b4d0 0x7fe72c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0cf660 tx=0x7fe7400079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:32:59.915+0000 7fe748c00640 1 -- 172.21.15.114:0/2705891357 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe74c15fd90 con 0x7fe74c0af1c0 2024-03-06T07:33:00.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.148+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe74c04f330 con 0x7fe74c0af1c0 2024-03-06T07:33:00.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.149+0000 7fe748c00640 1 -- 172.21.15.114:0/2705891357 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+3711 (secure 0 0 0) 0x7fe74c071d30 con 0x7fe74c0af1c0 2024-03-06T07:33:00.153 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:00.153 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":54,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5159,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2745832276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2745832276},{"type":"v1","addr":"172.21.15.177:6833","nonce":2745832276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":5162,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/1727633959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":1727633959},{"type":"v1","addr":"172.21.15.177:6835","nonce":1727633959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1685583003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1685583003},{"type":"v1","addr":"172.21.15.114:6836","nonce":1685583003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":7056,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1359294453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1359294453},{"type":"v1","addr":"172.21.15.114:6837","nonce":1359294453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[]} 2024-03-06T07:33:00.153 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 54 2024-03-06T07:33:00.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe72c06b4d0 msgr2=0x7fe72c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe72c06b4d0 0x7fe72c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0cf660 tx=0x7fe7400079b0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c0af1c0 msgr2=0x7fe74c1e7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c0af1c0 0x7fe74c1e7ee0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe73800c430 tx=0x7fe73800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 shutdown_connections 2024-03-06T07:33:00.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe74c15eff0 0x7fe74c1e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe72c06b4d0 0x7fe72c06d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe74c0af1c0 0x7fe74c1e7ee0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 --2- 172.21.15.114:0/2705891357 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe74c06ebe0 0x7fe74c0ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 >> 172.21.15.114:0/2705891357 conn(0x7fe74c0b1730 msgr2=0x7fe74c15f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 shutdown_connections 2024-03-06T07:33:00.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.152+0000 7fe753a00640 1 -- 172.21.15.114:0/2705891357 wait complete. 2024-03-06T07:33:00.175 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:33:00.175 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:33:00.176 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:33:00.176 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:00.176 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:33:00.176 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:33:00.176 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:33:00.176 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:00.176 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:33:00.177 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:33:00.177 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:33:00.177 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:00.177 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:33:00.177 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:33:00.177 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:33:00.177 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:00.199 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:33:00.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:33:00.201 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:33:00.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:33:00.202 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:33:00.202 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:33:00.204 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:33:00.204 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:33:00.248 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:33:00.249 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:00.251 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:33:00.252 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:00.254 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:33:00.255 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:00.258 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:33:00.258 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:00.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:33:00.404 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:33:00.405 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:33:00.414 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:33:00.420 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:33:00.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.442+0000 7f2a5a400640 1 Processor -- start 2024-03-06T07:33:00.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.442+0000 7f2a5a400640 1 -- start start 2024-03-06T07:33:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.443+0000 7f2a5a400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2a5415b070 0x7f2a5406e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.443+0000 7f2a5a400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2a5415f120 0x7f2a540c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a5a400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a540cc000 con 0x7f2a5415ac90 2024-03-06T07:33:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a5a400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a540ce360 con 0x7f2a5415b070 2024-03-06T07:33:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a5a400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a540ce640 con 0x7f2a5415f120 2024-03-06T07:33:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a53400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2a5415ac90 0x7f2a5415ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50630/0 (socket says 172.21.15.114:50630) 2024-03-06T07:33:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a53400640 1 -- 172.21.15.114:0/3368581145 learned_addr learned my addr 172.21.15.114:0/3368581145 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a4aa00640 1 --2- 172.21.15.114:0/3368581145 >> v2:172.21.15.114:3300/0 conn(0x7f2a5415b070 0x7f2a5406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a53e00640 1 --2- 172.21.15.114:0/3368581145 >> v2:172.21.15.177:3300/0 conn(0x7f2a5415f120 0x7f2a540c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a4aa00640 1 -- 172.21.15.114:0/3368581145 >> v2:172.21.15.177:3300/0 conn(0x7f2a5415f120 msgr2=0x7f2a540c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a4aa00640 1 --2- 172.21.15.114:0/3368581145 >> v2:172.21.15.177:3300/0 conn(0x7f2a5415f120 0x7f2a540c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a4aa00640 1 -- 172.21.15.114:0/3368581145 >> v1:172.21.15.114:6789/0 conn(0x7f2a5415ac90 legacy=0x7f2a5415ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a4aa00640 1 -- 172.21.15.114:0/3368581145 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a540ce920 con 0x7f2a5415b070 2024-03-06T07:33:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.444+0000 7f2a52a00640 1 -- 172.21.15.114:0/3368581145 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170294355 0 0) 0x7f2a540cc000 con 0x7f2a5415ac90 2024-03-06T07:33:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.445+0000 7f2a4aa00640 1 --2- 172.21.15.114:0/3368581145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415b070 0x7f2a5406e580 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cf2e0 tx=0x7f2a3c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9e880f6574a895 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a52a00640 1 -- 172.21.15.114:0/3368581145 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0315a0 con 0x7f2a5415b070 2024-03-06T07:33:00.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a52a00640 1 -- 172.21.15.114:0/3368581145 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c037070 con 0x7f2a5415b070 2024-03-06T07:33:00.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a52a00640 1 -- 172.21.15.114:0/3368581145 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c031960 con 0x7f2a5415b070 2024-03-06T07:33:00.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 -- 172.21.15.114:0/3368581145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415b070 msgr2=0x7f2a5406e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 --2- 172.21.15.114:0/3368581145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415b070 0x7f2a5406e580 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cf2e0 tx=0x7f2a3c02b840 comp rx=0 tx=0).stop 2024-03-06T07:33:00.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 -- 172.21.15.114:0/3368581145 shutdown_connections 2024-03-06T07:33:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 --2- 172.21.15.114:0/3368581145 >> v2:172.21.15.177:3300/0 conn(0x7f2a5415f120 0x7f2a540c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 --2- 172.21.15.114:0/3368581145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415b070 0x7f2a5406e580 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 -- 172.21.15.114:0/3368581145 >> 172.21.15.114:0/3368581145 conn(0x7f2a540cc4f0 msgr2=0x7f2a540cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 -- 172.21.15.114:0/3368581145 shutdown_connections 2024-03-06T07:33:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.446+0000 7f2a5a400640 1 -- 172.21.15.114:0/3368581145 wait complete. 2024-03-06T07:33:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.448+0000 7f2a5a400640 1 Processor -- start 2024-03-06T07:33:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.448+0000 7f2a5a400640 1 -- start start 2024-03-06T07:33:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a5a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415ac90 0x7f2a5414f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a5a400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a5415b070 0x7f2a540aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a5a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a5415f120 0x7f2a540cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a5a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5415f5b0 con 0x7f2a5415ac90 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a5a400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540ceee0 con 0x7f2a5415f120 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a5a400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5415f730 con 0x7f2a5415b070 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a53400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415ac90 0x7f2a5414f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a53400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415ac90 0x7f2a5414f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42018/0 (socket says 172.21.15.114:42018) 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a53400640 1 -- 172.21.15.114:0/4024828905 learned_addr learned my addr 172.21.15.114:0/4024828905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a4aa00640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a5415b070 0x7f2a540aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.449+0000 7f2a53e00640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a5415f120 0x7f2a540cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a53400640 1 -- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a5415b070 msgr2=0x7f2a540aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a53400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a5415b070 0x7f2a540aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a53400640 1 -- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a5415f120 msgr2=0x7f2a540cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a53400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a5415f120 0x7f2a540cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a53400640 1 -- 172.21.15.114:0/4024828905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5415d870 con 0x7f2a5415ac90 2024-03-06T07:33:00.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a53400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415ac90 0x7f2a5414f260 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400c4d0 tx=0x7f2a4400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a4be00640 1 -- 172.21.15.114:0/4024828905 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44018070 con 0x7f2a5415ac90 2024-03-06T07:33:00.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a4be00640 1 -- 172.21.15.114:0/4024828905 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a44007500 con 0x7f2a5415ac90 2024-03-06T07:33:00.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a4be00640 1 -- 172.21.15.114:0/4024828905 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44004a20 con 0x7f2a5415ac90 2024-03-06T07:33:00.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.450+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a5415db50 con 0x7f2a5415ac90 2024-03-06T07:33:00.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.451+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a5415e110 con 0x7f2a5415ac90 2024-03-06T07:33:00.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.452+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5404f330 con 0x7f2a5415ac90 2024-03-06T07:33:00.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.453+0000 7f2a4be00640 1 -- 172.21.15.114:0/4024828905 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a4401b070 con 0x7f2a5415ac90 2024-03-06T07:33:00.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.454+0000 7f2a4be00640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a3006b4d0 0x7f2a3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.454+0000 7f2a4be00640 1 -- 172.21.15.114:0/4024828905 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2a44089230 con 0x7f2a5415ac90 2024-03-06T07:33:00.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.454+0000 7f2a4aa00640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a3006b4d0 0x7f2a3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.455+0000 7f2a4aa00640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a3006b4d0 0x7f2a3006d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c036000 tx=0x7f2a3c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.458+0000 7f2a4be00640 1 -- 172.21.15.114:0/4024828905 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a44057cc0 con 0x7f2a5415ac90 2024-03-06T07:33:00.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.665+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2a540cd1c0 con 0x7f2a5415ac90 2024-03-06T07:33:00.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.796+0000 7f2a4be00640 1 -- 172.21.15.114:0/4024828905 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v87) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2a4405bd60 con 0x7f2a5415ac90 2024-03-06T07:33:00.799 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:33:00.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a3006b4d0 msgr2=0x7f2a3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a3006b4d0 0x7f2a3006d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c036000 tx=0x7f2a3c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:00.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415ac90 msgr2=0x7f2a5414f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415ac90 0x7f2a5414f260 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400c4d0 tx=0x7f2a4400c990 comp rx=0 tx=0).stop 2024-03-06T07:33:00.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 shutdown_connections 2024-03-06T07:33:00.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a5415f120 0x7f2a540cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2a5415b070 0x7f2a540aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a3006b4d0 0x7f2a3006d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 --2- 172.21.15.114:0/4024828905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a5415ac90 0x7f2a5414f260 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 >> 172.21.15.114:0/4024828905 conn(0x7f2a540cc4f0 msgr2=0x7f2a5406d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 shutdown_connections 2024-03-06T07:33:00.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:00.799+0000 7f2a5a400640 1 -- 172.21.15.114:0/4024828905 wait complete. 2024-03-06T07:33:00.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:33:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.010+0000 7f567ea00640 1 Processor -- start 2024-03-06T07:33:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.011+0000 7f567ea00640 1 -- start start 2024-03-06T07:33:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f567ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f567815acb0 0x7f56780b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f567ea00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f567806ec00 0x7f56780ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f567ea00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567815b090 con 0x7f56780b4680 2024-03-06T07:33:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f567ea00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56780b2a30 con 0x7f567815acb0 2024-03-06T07:33:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f567ea00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56780b4360 con 0x7f567806ec00 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f5677400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f56780b4680 0x7f567806e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50642/0 (socket says 172.21.15.114:50642) 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f5677400640 1 -- 172.21.15.114:0/871000858 learned_addr learned my addr 172.21.15.114:0/871000858 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.012+0000 7f5677e00640 1 --2- 172.21.15.114:0/871000858 >> v2:172.21.15.114:3300/0 conn(0x7f567815acb0 0x7f56780b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.013+0000 7f567ca00640 1 --2- 172.21.15.114:0/871000858 >> v2:172.21.15.177:3300/0 conn(0x7f567806ec00 0x7f56780ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.013+0000 7f5676a00640 1 -- 172.21.15.114:0/871000858 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595463274 0 0) 0x7f567815b090 con 0x7f56780b4680 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.013+0000 7f5676a00640 1 -- 172.21.15.114:0/871000858 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f565c003520 con 0x7f56780b4680 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.013+0000 7f5677e00640 1 -- 172.21.15.114:0/871000858 >> v2:172.21.15.177:3300/0 conn(0x7f567806ec00 msgr2=0x7f56780ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.013+0000 7f5677e00640 1 --2- 172.21.15.114:0/871000858 >> v2:172.21.15.177:3300/0 conn(0x7f567806ec00 0x7f56780ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.013+0000 7f5677e00640 1 -- 172.21.15.114:0/871000858 >> v1:172.21.15.114:6789/0 conn(0x7f56780b4680 legacy=0x7f567806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.013+0000 7f5677e00640 1 -- 172.21.15.114:0/871000858 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56780b2cf0 con 0x7f567815acb0 2024-03-06T07:33:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.014+0000 7f5677e00640 1 --2- 172.21.15.114:0/871000858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f56780b37d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f56780b4e60 tx=0x7f566802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b97d27faa9ab6f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.014+0000 7f5676a00640 1 -- 172.21.15.114:0/871000858 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668035020 con 0x7f567815acb0 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.014+0000 7f5676a00640 1 -- 172.21.15.114:0/871000858 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668031830 con 0x7f567815acb0 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.014+0000 7f5676a00640 1 -- 172.21.15.114:0/871000858 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668031c00 con 0x7f567815acb0 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 -- 172.21.15.114:0/871000858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 msgr2=0x7f56780b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 --2- 172.21.15.114:0/871000858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f56780b37d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f56780b4e60 tx=0x7f566802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 -- 172.21.15.114:0/871000858 shutdown_connections 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 --2- 172.21.15.114:0/871000858 >> v2:172.21.15.177:3300/0 conn(0x7f567806ec00 0x7f56780ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 --2- 172.21.15.114:0/871000858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f56780b37d0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 -- 172.21.15.114:0/871000858 >> 172.21.15.114:0/871000858 conn(0x7f56780b1730 msgr2=0x7f56780b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 -- 172.21.15.114:0/871000858 shutdown_connections 2024-03-06T07:33:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.015+0000 7f567ea00640 1 -- 172.21.15.114:0/871000858 wait complete. 2024-03-06T07:33:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.016+0000 7f567ea00640 1 Processor -- start 2024-03-06T07:33:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.017+0000 7f567ea00640 1 -- start start 2024-03-06T07:33:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.017+0000 7f567ea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f567806ec00 0x7f5678144360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.017+0000 7f567ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780b4680 0x7f567806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.017+0000 7f567ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f567806d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.017+0000 7f567ea00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780b3d10 con 0x7f567815acb0 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.017+0000 7f567ea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780b32b0 con 0x7f56780b4680 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.017+0000 7f567ea00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56780b4010 con 0x7f567806ec00 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f567806d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f567806d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42036/0 (socket says 172.21.15.114:42036) 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 -- 172.21.15.114:0/3451351005 learned_addr learned my addr 172.21.15.114:0/3451351005 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f5677400640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780b4680 0x7f567806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 -- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f567806ec00 msgr2=0x7f5678144360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f5677e00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f567806ec00 0x7f5678144360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f567806ec00 0x7f5678144360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 -- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780b4680 msgr2=0x7f567806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780b4680 0x7f567806d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 -- 172.21.15.114:0/3451351005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56781f46a0 con 0x7f567815acb0 2024-03-06T07:33:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.018+0000 7f567ca00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f567806d920 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f56700070b0 tx=0x7f56700085d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.019+0000 7f566fe00640 1 -- 172.21.15.114:0/3451351005 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670008e00 con 0x7f567815acb0 2024-03-06T07:33:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.019+0000 7f566fe00640 1 -- 172.21.15.114:0/3451351005 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670009c60 con 0x7f567815acb0 2024-03-06T07:33:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.019+0000 7f566fe00640 1 -- 172.21.15.114:0/3451351005 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670003bd0 con 0x7f567815acb0 2024-03-06T07:33:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.019+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56781551d0 con 0x7f567815acb0 2024-03-06T07:33:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.019+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5678155790 con 0x7f567815acb0 2024-03-06T07:33:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.020+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567804f330 con 0x7f567815acb0 2024-03-06T07:33:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.021+0000 7f566fe00640 1 -- 172.21.15.114:0/3451351005 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5670004030 con 0x7f567815acb0 2024-03-06T07:33:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.021+0000 7f566fe00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565806b4d0 0x7f565806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.022+0000 7f566fe00640 1 -- 172.21.15.114:0/3451351005 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f567008a170 con 0x7f567815acb0 2024-03-06T07:33:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.022+0000 7f5677e00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565806b4d0 0x7f565806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.022+0000 7f5677e00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565806b4d0 0x7f565806d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5668036000 tx=0x7f5668008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.026+0000 7f566fe00640 1 -- 172.21.15.114:0/3451351005 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5670059330 con 0x7f567815acb0 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.290+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5678074be0 con 0x7f567815acb0 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.291+0000 7f566fe00640 1 -- 172.21.15.114:0/3451351005 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5670058c30 con 0x7f567815acb0 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow r", 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.293+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565806b4d0 msgr2=0x7f565806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565806b4d0 0x7f565806d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5668036000 tx=0x7f5668008030 comp rx=0 tx=0).stop 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 msgr2=0x7f567806d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f567806d920 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f56700070b0 tx=0x7f56700085d0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 shutdown_connections 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f565806b4d0 0x7f565806d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f567815acb0 0x7f567806d920 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56780b4680 0x7f567806d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 --2- 172.21.15.114:0/3451351005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f567806ec00 0x7f5678144360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 >> 172.21.15.114:0/3451351005 conn(0x7f56780b1730 msgr2=0x7f56780b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 shutdown_connections 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.294+0000 7f567ea00640 1 -- 172.21.15.114:0/3451351005 wait complete. 2024-03-06T07:33:01.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.505+0000 7fe8aec00640 1 Processor -- start 2024-03-06T07:33:01.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.505+0000 7fe8aec00640 1 -- start start 2024-03-06T07:33:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.506+0000 7fe8aec00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8a8144c00 0x7fe8a806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815fe60 0x7fe8a80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8aec00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a806ee70 con 0x7fe8a815f9d0 2024-03-06T07:33:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8aec00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a806cef0 con 0x7fe8a8144c00 2024-03-06T07:33:01.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8aec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a806d1d0 con 0x7fe8a815fe60 2024-03-06T07:33:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe89f400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe8a815f9d0 0x7fe8a80ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50644/0 (socket says 172.21.15.114:50644) 2024-03-06T07:33:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8a7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe8a8144c00 0x7fe8a806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe89f400640 1 -- 172.21.15.114:0/1682681715 learned_addr learned my addr 172.21.15.114:0/1682681715 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8acc00640 1 --2- 172.21.15.114:0/1682681715 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815fe60 0x7fe8a80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8a7e00640 1 -- 172.21.15.114:0/1682681715 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815fe60 msgr2=0x7fe8a80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8a7e00640 1 --2- 172.21.15.114:0/1682681715 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815fe60 0x7fe8a80ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8a7e00640 1 -- 172.21.15.114:0/1682681715 >> v1:172.21.15.114:6789/0 conn(0x7fe8a815f9d0 legacy=0x7fe8a80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8a7e00640 1 -- 172.21.15.114:0/1682681715 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a806d4b0 con 0x7fe8a8144c00 2024-03-06T07:33:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8a7400640 1 -- 172.21.15.114:0/1682681715 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034705610 0 0) 0x7fe890001110 con 0x7fe8a815f9d0 2024-03-06T07:33:01.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.507+0000 7fe8acc00640 1 --2- 172.21.15.114:0/1682681715 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815fe60 0x7fe8a80ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8a7e00640 1 --2- 172.21.15.114:0/1682681715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a806e930 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe8980097a0 tx=0x7fe89802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab311d8d1e546e8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8a7400640 1 -- 172.21.15.114:0/1682681715 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898035020 con 0x7fe8a8144c00 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8a7400640 1 -- 172.21.15.114:0/1682681715 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898031830 con 0x7fe8a8144c00 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8a7400640 1 -- 172.21.15.114:0/1682681715 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898031c00 con 0x7fe8a8144c00 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 -- 172.21.15.114:0/1682681715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 msgr2=0x7fe8a806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1682681715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a806e930 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe8980097a0 tx=0x7fe89802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 -- 172.21.15.114:0/1682681715 shutdown_connections 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1682681715 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815fe60 0x7fe8a80ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1682681715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a806e930 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 -- 172.21.15.114:0/1682681715 >> 172.21.15.114:0/1682681715 conn(0x7fe8a80b1730 msgr2=0x7fe8a80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 -- 172.21.15.114:0/1682681715 shutdown_connections 2024-03-06T07:33:01.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.509+0000 7fe8aec00640 1 -- 172.21.15.114:0/1682681715 wait complete. 2024-03-06T07:33:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.511+0000 7fe8aec00640 1 Processor -- start 2024-03-06T07:33:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.511+0000 7fe8aec00640 1 -- start start 2024-03-06T07:33:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a814cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815f9d0 0x7fe8a8140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8aec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8a815fe60 0x7fe8a813e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8aec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8160730 con 0x7fe8a8144c00 2024-03-06T07:33:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8aec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8160290 con 0x7fe8a815fe60 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8aec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a806cb50 con 0x7fe8a815f9d0 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8a7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a814cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8a7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a814cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42058/0 (socket says 172.21.15.114:42058) 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.512+0000 7fe8a7e00640 1 -- 172.21.15.114:0/1891147433 learned_addr learned my addr 172.21.15.114:0/1891147433 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a7e00640 1 -- 172.21.15.114:0/1891147433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815f9d0 msgr2=0x7fe8a8140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a7e00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815f9d0 0x7fe8a8140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a7e00640 1 -- 172.21.15.114:0/1891147433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8a815fe60 msgr2=0x7fe8a813e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a7e00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8a815fe60 0x7fe8a813e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a7e00640 1 -- 172.21.15.114:0/1891147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a813e6d0 con 0x7fe8a8144c00 2024-03-06T07:33:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a7e00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a814cab0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe898034040 tx=0x7fe8980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a4c00640 1 -- 172.21.15.114:0/1891147433 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898040070 con 0x7fe8a8144c00 2024-03-06T07:33:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a4c00640 1 -- 172.21.15.114:0/1891147433 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898039a20 con 0x7fe8a8144c00 2024-03-06T07:33:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8a4c00640 1 -- 172.21.15.114:0/1891147433 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898035040 con 0x7fe8a8144c00 2024-03-06T07:33:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a813e950 con 0x7fe8a8144c00 2024-03-06T07:33:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.513+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a814c2b0 con 0x7fe8a8144c00 2024-03-06T07:33:01.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.514+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a804f330 con 0x7fe8a8144c00 2024-03-06T07:33:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.515+0000 7fe8a4c00640 1 -- 172.21.15.114:0/1891147433 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe898038020 con 0x7fe8a8144c00 2024-03-06T07:33:01.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.516+0000 7fe8a4c00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe88406b4d0 0x7fe88406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.516+0000 7fe8a4c00640 1 -- 172.21.15.114:0/1891147433 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe8980ab9c0 con 0x7fe8a8144c00 2024-03-06T07:33:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.516+0000 7fe89f400640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe88406b4d0 0x7fe88406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.517+0000 7fe89f400640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe88406b4d0 0x7fe88406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8071f10 tx=0x7fe890001310 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.521+0000 7fe8a4c00640 1 -- 172.21.15.114:0/1891147433 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8a813e6d0 con 0x7fe8a8144c00 2024-03-06T07:33:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.713+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8a80b0400 con 0x7fe8a8144c00 2024-03-06T07:33:01.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.715+0000 7fe8a4c00640 1 -- 172.21.15.114:0/1891147433 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+12940 (secure 0 0 0) 0x7fe89807aaf0 con 0x7fe8a8144c00 2024-03-06T07:33:01.718 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:01.718 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:00.813944+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:01.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe88406b4d0 msgr2=0x7fe88406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe88406b4d0 0x7fe88406d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8071f10 tx=0x7fe890001310 comp rx=0 tx=0).stop 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 msgr2=0x7fe8a814cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a814cab0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fe898034040 tx=0x7fe8980046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 shutdown_connections 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8a815fe60 0x7fe8a813e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe8a815f9d0 0x7fe8a8140440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe88406b4d0 0x7fe88406d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 --2- 172.21.15.114:0/1891147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a8144c00 0x7fe8a814cab0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 >> 172.21.15.114:0/1891147433 conn(0x7fe8a80b1730 msgr2=0x7fe8a80b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 shutdown_connections 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.718+0000 7fe8aec00640 1 -- 172.21.15.114:0/1891147433 wait complete. 2024-03-06T07:33:01.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:01.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:01.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.944+0000 7f3b38400640 1 Processor -- start 2024-03-06T07:33:01.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.944+0000 7f3b38400640 1 -- start start 2024-03-06T07:33:01.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.946+0000 7f3b38400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 0x7f3b30070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.946+0000 7f3b38400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300b46c0 con 0x7f3b3015fc10 2024-03-06T07:33:01.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.946+0000 7f3b38400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300b49a0 con 0x7f3b300b3f70 2024-03-06T07:33:01.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.946+0000 7f3b38400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30072c90 con 0x7f3b30160700 2024-03-06T07:33:01.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.946+0000 7f3b35a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3b3015fc10 0x7f3b3015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50658/0 (socket says 172.21.15.114:50658) 2024-03-06T07:33:01.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.946+0000 7f3b35a00640 1 -- 172.21.15.114:0/1929107511 learned_addr learned my addr 172.21.15.114:0/1929107511 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:01.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715066540 0 0) 0x7f3b300b46c0 con 0x7f3b3015fc10 2024-03-06T07:33:01.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b18003520 con 0x7f3b3015fc10 2024-03-06T07:33:01.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b36400640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 0x7f3b30070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020741088 0 0) 0x7f3b18003520 con 0x7f3b3015fc10 2024-03-06T07:33:01.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 msgr2=0x7f3b30070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 0x7f3b30070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 >> v1:172.21.15.177:6789/0 conn(0x7f3b300b3f70 legacy=0x7f3b300ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3009dd80 con 0x7f3b3015fc10 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f3b24001670 con 0x7f3b3015fc10 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 >> v1:172.21.15.114:6789/0 conn(0x7f3b3015fc10 legacy=0x7f3b3015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b180039e0 0x7f3b18003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b18004800 0x7f3b18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 0x7f3b18005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30072c90 con 0x7f3b18004800 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1800baf0 con 0x7f3b180039e0 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.947+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1800bcf0 con 0x7f3b30160700 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35000640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b18004800 0x7f3b18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b36400640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 0x7f3b18005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35a00640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b180039e0 0x7f3b18003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35000640 1 -- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 msgr2=0x7f3b18005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35000640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b30160700 0x7f3b18005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35000640 1 -- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b180039e0 msgr2=0x7f3b18003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:01.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35000640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b180039e0 0x7f3b18003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35000640 1 -- 172.21.15.114:0/1929107511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18003520 con 0x7f3b18004800 2024-03-06T07:33:01.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.948+0000 7f3b35a00640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b180039e0 0x7f3b18003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.949+0000 7f3b35000640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b18004800 0x7f3b18004c80 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c001720 tx=0x7f3b18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8931c3fe9fb03b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.949+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c001370 con 0x7f3b18004800 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.949+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c0052d0 con 0x7f3b18004800 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.949+0000 7f3b2fe00640 1 -- 172.21.15.114:0/1929107511 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c003830 con 0x7f3b18004800 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.950+0000 7f3b38400640 1 -- 172.21.15.114:0/1929107511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b18004800 msgr2=0x7f3b18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.950+0000 7f3b38400640 1 --2- 172.21.15.114:0/1929107511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b18004800 0x7f3b18004c80 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c001720 tx=0x7f3b18005830 comp rx=0 tx=0).stop 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.950+0000 7f3b36400640 1 -- 172.21.15.114:0/1929107511 reap_dead start 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.950+0000 7f3b38400640 1 -- 172.21.15.114:0/1929107511 shutdown_connections 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.950+0000 7f3b38400640 1 -- 172.21.15.114:0/1929107511 >> 172.21.15.114:0/1929107511 conn(0x7f3b300b1940 msgr2=0x7f3b300b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.950+0000 7f3b38400640 1 -- 172.21.15.114:0/1929107511 shutdown_connections 2024-03-06T07:33:01.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.950+0000 7f3b38400640 1 -- 172.21.15.114:0/1929107511 wait complete. 2024-03-06T07:33:01.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.952+0000 7f3b38400640 1 Processor -- start 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.952+0000 7f3b38400640 1 -- start start 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b38400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18004800 0x7f3b3014cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b38400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3006de90 0x7f3b3015a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b38400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b3015a930 0x7f3b300af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b38400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3015c460 con 0x7f3b3006de90 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b38400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3015c2e0 con 0x7f3b18004800 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b38400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30070ff0 con 0x7f3b3015a930 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3006de90 0x7f3b3015a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3006de90 0x7f3b3015a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42076/0 (socket says 172.21.15.114:42076) 2024-03-06T07:33:01.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35000640 1 -- 172.21.15.114:0/1904394863 learned_addr learned my addr 172.21.15.114:0/1904394863 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b36400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b3015a930 0x7f3b300af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35a00640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18004800 0x7f3b3014cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35000640 1 -- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b3015a930 msgr2=0x7f3b300af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35000640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b3015a930 0x7f3b300af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35000640 1 -- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18004800 msgr2=0x7f3b3014cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.953+0000 7f3b35000640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18004800 0x7f3b3014cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b35000640 1 -- 172.21.15.114:0/1904394863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3006cdf0 con 0x7f3b3006de90 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b36400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b3015a930 0x7f3b300af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b35000640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3006de90 0x7f3b3015a3f0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c005040 tx=0x7f3b1c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b2d600640 1 -- 172.21.15.114:0/1904394863 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c041070 con 0x7f3b3006de90 2024-03-06T07:33:01.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b2d600640 1 -- 172.21.15.114:0/1904394863 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c005480 con 0x7f3b3006de90 2024-03-06T07:33:01.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3006d0d0 con 0x7f3b3006de90 2024-03-06T07:33:01.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b2d600640 1 -- 172.21.15.114:0/1904394863 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c007a30 con 0x7f3b3006de90 2024-03-06T07:33:01.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.954+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3006d690 con 0x7f3b3006de90 2024-03-06T07:33:01.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.955+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3004f330 con 0x7f3b3006de90 2024-03-06T07:33:01.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.956+0000 7f3b2d600640 1 -- 172.21.15.114:0/1904394863 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b1c044070 con 0x7f3b3006de90 2024-03-06T07:33:01.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.957+0000 7f3b2d600640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:01.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.957+0000 7f3b2d600640 1 -- 172.21.15.114:0/1904394863 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3b1c0abb30 con 0x7f3b3006de90 2024-03-06T07:33:01.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.957+0000 7f3b35a00640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:01.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.958+0000 7f3b35a00640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1006b500 0x7f3b1006d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3b24007930 tx=0x7f3b24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:01.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:01.961+0000 7f3b2d600640 1 -- 172.21.15.114:0/1904394863 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b1c07ad80 con 0x7f3b3006de90 2024-03-06T07:33:02.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.158+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b3015ad10 con 0x7f3b3006de90 2024-03-06T07:33:02.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.158+0000 7f3b2d600640 1 -- 172.21.15.114:0/1904394863 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b1c07a680 con 0x7f3b3006de90 2024-03-06T07:33:02.161 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:33:02.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1006b500 msgr2=0x7f3b1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1006b500 0x7f3b1006d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3b24007930 tx=0x7f3b24009040 comp rx=0 tx=0).stop 2024-03-06T07:33:02.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3006de90 msgr2=0x7f3b3015a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3006de90 0x7f3b3015a3f0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c005040 tx=0x7f3b1c001120 comp rx=0 tx=0).stop 2024-03-06T07:33:02.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 shutdown_connections 2024-03-06T07:33:02.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3b3015a930 0x7f3b300af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3b1006b500 0x7f3b1006d9c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3006de90 0x7f3b3015a3f0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 --2- 172.21.15.114:0/1904394863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18004800 0x7f3b3014cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 >> 172.21.15.114:0/1904394863 conn(0x7f3b300b1940 msgr2=0x7f3b300ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 shutdown_connections 2024-03-06T07:33:02.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.161+0000 7f3b38400640 1 -- 172.21.15.114:0/1904394863 wait complete. 2024-03-06T07:33:02.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:33:02.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.382+0000 7fd371800640 1 Processor -- start 2024-03-06T07:33:02.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.382+0000 7fd371800640 1 -- start start 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.382+0000 7fd371800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd36c15acb0 0x7fd36c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd371800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c06ec00 0x7fd36c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd371800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c15b090 con 0x7fd36c0b4680 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd371800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36c0b2a30 con 0x7fd36c15acb0 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd371800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c0b4360 con 0x7fd36c06ec00 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd36aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd36c15acb0 0x7fd36c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd36aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd36c15acb0 0x7fd36c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42086/0 (socket says 172.21.15.114:42086) 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd36aa00640 1 -- 172.21.15.114:0/544459917 learned_addr learned my addr 172.21.15.114:0/544459917 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:02.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.383+0000 7fd36b400640 1 --2- 172.21.15.114:0/544459917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c06ec00 0x7fd36c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.384+0000 7fd36aa00640 1 -- 172.21.15.114:0/544459917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c06ec00 msgr2=0x7fd36c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.384+0000 7fd36aa00640 1 --2- 172.21.15.114:0/544459917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c06ec00 0x7fd36c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.384+0000 7fd36aa00640 1 -- 172.21.15.114:0/544459917 >> v1:172.21.15.177:6789/0 conn(0x7fd36c0b4680 legacy=0x7fd36c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.384+0000 7fd36aa00640 1 -- 172.21.15.114:0/544459917 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c0b2cf0 con 0x7fd36c15acb0 2024-03-06T07:33:02.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.385+0000 7fd36aa00640 1 --2- 172.21.15.114:0/544459917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c15acb0 0x7fd36c0b37d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00d150 tx=0x7fd35c02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92badc6f08d2e723 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.385+0000 7fd36a000640 1 -- 172.21.15.114:0/544459917 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00b0d0 con 0x7fd36c15acb0 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.385+0000 7fd36a000640 1 -- 172.21.15.114:0/544459917 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c00b270 con 0x7fd36c15acb0 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.385+0000 7fd36a000640 1 -- 172.21.15.114:0/544459917 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c009600 con 0x7fd36c15acb0 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 -- 172.21.15.114:0/544459917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c15acb0 msgr2=0x7fd36c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 --2- 172.21.15.114:0/544459917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c15acb0 0x7fd36c0b37d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00d150 tx=0x7fd35c02e160 comp rx=0 tx=0).stop 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 -- 172.21.15.114:0/544459917 shutdown_connections 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 --2- 172.21.15.114:0/544459917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c06ec00 0x7fd36c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 --2- 172.21.15.114:0/544459917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c15acb0 0x7fd36c0b37d0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 -- 172.21.15.114:0/544459917 >> 172.21.15.114:0/544459917 conn(0x7fd36c0b1730 msgr2=0x7fd36c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 -- 172.21.15.114:0/544459917 shutdown_connections 2024-03-06T07:33:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.386+0000 7fd371800640 1 -- 172.21.15.114:0/544459917 wait complete. 2024-03-06T07:33:02.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.388+0000 7fd371800640 1 Processor -- start 2024-03-06T07:33:02.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.388+0000 7fd371800640 1 -- start start 2024-03-06T07:33:02.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd371800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c06ec00 0x7fd36c154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd371800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd36c0b4680 0x7fd36c144500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd371800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c15acb0 0x7fd36c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd371800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c0b3d10 con 0x7fd36c06ec00 2024-03-06T07:33:02.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd371800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c0b32b0 con 0x7fd36c0b4680 2024-03-06T07:33:02.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd371800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c0b4010 con 0x7fd36c15acb0 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c06ec00 0x7fd36c154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c06ec00 0x7fd36c154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42094/0 (socket says 172.21.15.114:42094) 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 -- 172.21.15.114:0/443709102 learned_addr learned my addr 172.21.15.114:0/443709102 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 -- 172.21.15.114:0/443709102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c15acb0 msgr2=0x7fd36c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c15acb0 0x7fd36c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 -- 172.21.15.114:0/443709102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd36c0b4680 msgr2=0x7fd36c144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd36c0b4680 0x7fd36c144500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.389+0000 7fd36aa00640 1 -- 172.21.15.114:0/443709102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c1531a0 con 0x7fd36c06ec00 2024-03-06T07:33:02.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.390+0000 7fd36aa00640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c06ec00 0x7fd36c154240 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd35c038040 tx=0x7fd35c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.390+0000 7fd363400640 1 -- 172.21.15.114:0/443709102 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00b0d0 con 0x7fd36c06ec00 2024-03-06T07:33:02.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.390+0000 7fd363400640 1 -- 172.21.15.114:0/443709102 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c045070 con 0x7fd36c06ec00 2024-03-06T07:33:02.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.390+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36c153480 con 0x7fd36c06ec00 2024-03-06T07:33:02.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.390+0000 7fd363400640 1 -- 172.21.15.114:0/443709102 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00b720 con 0x7fd36c06ec00 2024-03-06T07:33:02.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.390+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36c153a40 con 0x7fd36c06ec00 2024-03-06T07:33:02.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.392+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36c0b2430 con 0x7fd36c06ec00 2024-03-06T07:33:02.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.392+0000 7fd363400640 1 -- 172.21.15.114:0/443709102 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd35c02ed10 con 0x7fd36c06ec00 2024-03-06T07:33:02.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.393+0000 7fd363400640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd34806b4d0 0x7fd34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.393+0000 7fd363400640 1 -- 172.21.15.114:0/443709102 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd35c0acb70 con 0x7fd36c06ec00 2024-03-06T07:33:02.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.394+0000 7fd362000640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd34806b4d0 0x7fd34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.394+0000 7fd362000640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd34806b4d0 0x7fd34806d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd354000f60 tx=0x7fd3540079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.398+0000 7fd363400640 1 -- 172.21.15.114:0/443709102 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd36c0b2430 con 0x7fd36c06ec00 2024-03-06T07:33:02.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.590+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd36c04f330 con 0x7fd36c06ec00 2024-03-06T07:33:02.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.590+0000 7fd363400640 1 -- 172.21.15.114:0/443709102 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd35c07be50 con 0x7fd36c06ec00 2024-03-06T07:33:02.593 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:02.593 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:33:02.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:33:02.593 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:33:02.594 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:33:02.594 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:33:02.594 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:33:02.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd34806b4d0 msgr2=0x7fd34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd34806b4d0 0x7fd34806d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd354000f60 tx=0x7fd3540079d0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c06ec00 msgr2=0x7fd36c154240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c06ec00 0x7fd36c154240 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd35c038040 tx=0x7fd35c004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 shutdown_connections 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd36c15acb0 0x7fd36c155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd36c0b4680 0x7fd36c144500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd34806b4d0 0x7fd34806d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 --2- 172.21.15.114:0/443709102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd36c06ec00 0x7fd36c154240 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 >> 172.21.15.114:0/443709102 conn(0x7fd36c0b1730 msgr2=0x7fd36c0ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 shutdown_connections 2024-03-06T07:33:02.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.593+0000 7fd371800640 1 -- 172.21.15.114:0/443709102 wait complete. 2024-03-06T07:33:02.614 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:33:02.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:33:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.815+0000 7f02d6000640 1 Processor -- start 2024-03-06T07:33:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.815+0000 7f02d6000640 1 -- start start 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02d6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02d0145980 0x7f02d015fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02d6000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f02d00ced20 0x7f02d015ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02d6000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00712d0 0x7f02d0071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02d6000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d0070e40 con 0x7f02d0145980 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02d6000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d00cf1e0 con 0x7f02d00ced20 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02d6000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00cf4c0 con 0x7f02d00712d0 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02d0145980 0x7f02d015fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f02d0145980 0x7f02d015fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42098/0 (socket says 172.21.15.114:42098) 2024-03-06T07:33:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 -- 172.21.15.114:0/416641704 learned_addr learned my addr 172.21.15.114:0/416641704 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 -- 172.21.15.114:0/416641704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00712d0 msgr2=0x7f02d0071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 --2- 172.21.15.114:0/416641704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00712d0 0x7f02d0071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 -- 172.21.15.114:0/416641704 >> v2:172.21.15.177:3300/0 conn(0x7f02d00ced20 msgr2=0x7f02d015ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 --2- 172.21.15.114:0/416641704 >> v2:172.21.15.177:3300/0 conn(0x7f02d00ced20 0x7f02d015ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.817+0000 7f02cf400640 1 -- 172.21.15.114:0/416641704 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00cf7a0 con 0x7f02d0145980 2024-03-06T07:33:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.818+0000 7f02cf400640 1 --2- 172.21.15.114:0/416641704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d0145980 0x7f02d015fa00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f02d0070a60 tx=0x7f02c002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e0157255c01e96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.819+0000 7f02ce000640 1 -- 172.21.15.114:0/416641704 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0035020 con 0x7f02d0145980 2024-03-06T07:33:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.819+0000 7f02ce000640 1 -- 172.21.15.114:0/416641704 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c002d040 con 0x7f02d0145980 2024-03-06T07:33:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.819+0000 7f02ce000640 1 -- 172.21.15.114:0/416641704 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0031a80 con 0x7f02d0145980 2024-03-06T07:33:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.819+0000 7f02d6000640 1 -- 172.21.15.114:0/416641704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d0145980 msgr2=0x7f02d015fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.819+0000 7f02d6000640 1 --2- 172.21.15.114:0/416641704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d0145980 0x7f02d015fa00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f02d0070a60 tx=0x7f02c002b900 comp rx=0 tx=0).stop 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.820+0000 7f02d6000640 1 -- 172.21.15.114:0/416641704 shutdown_connections 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.820+0000 7f02d6000640 1 --2- 172.21.15.114:0/416641704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00712d0 0x7f02d0071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.820+0000 7f02d6000640 1 --2- 172.21.15.114:0/416641704 >> v2:172.21.15.177:3300/0 conn(0x7f02d00ced20 0x7f02d015ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.820+0000 7f02d6000640 1 --2- 172.21.15.114:0/416641704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d0145980 0x7f02d015fa00 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.820+0000 7f02d6000640 1 -- 172.21.15.114:0/416641704 >> 172.21.15.114:0/416641704 conn(0x7f02d00cfa80 msgr2=0x7f02d00cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.820+0000 7f02d6000640 1 -- 172.21.15.114:0/416641704 shutdown_connections 2024-03-06T07:33:02.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.820+0000 7f02d6000640 1 -- 172.21.15.114:0/416641704 wait complete. 2024-03-06T07:33:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.822+0000 7f02d6000640 1 Processor -- start 2024-03-06T07:33:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.822+0000 7f02d6000640 1 -- start start 2024-03-06T07:33:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.823+0000 7f02d6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d00712d0 0x7f02d015f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.823+0000 7f02d6000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00ced20 0x7f02d015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.823+0000 7f02d6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02d0145980 0x7f02d013f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.823+0000 7f02d6000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d01608c0 con 0x7f02d00712d0 2024-03-06T07:33:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.823+0000 7f02d6000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d0160740 con 0x7f02d0145980 2024-03-06T07:33:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.823+0000 7f02d6000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d0071c40 con 0x7f02d00ced20 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d00712d0 0x7f02d015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d00712d0 0x7f02d015f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42108/0 (socket says 172.21.15.114:42108) 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 -- 172.21.15.114:0/408794487 learned_addr learned my addr 172.21.15.114:0/408794487 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cea00640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00ced20 0x7f02d015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cfe00640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02d0145980 0x7f02d013f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 -- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00ced20 msgr2=0x7f02d015d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00ced20 0x7f02d015d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 -- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02d0145980 msgr2=0x7f02d013f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02d0145980 0x7f02d013f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 -- 172.21.15.114:0/408794487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d015d7c0 con 0x7f02d00712d0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.824+0000 7f02cf400640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d00712d0 0x7f02d015f360 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.825+0000 7f02c7400640 1 -- 172.21.15.114:0/408794487 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c003ac60 con 0x7f02d00712d0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.825+0000 7f02c7400640 1 -- 172.21.15.114:0/408794487 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c003ae00 con 0x7f02d00712d0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.825+0000 7f02c7400640 1 -- 172.21.15.114:0/408794487 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0044630 con 0x7f02d00712d0 2024-03-06T07:33:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.825+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d015daa0 con 0x7f02d00712d0 2024-03-06T07:33:02.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.825+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d015e060 con 0x7f02d00712d0 2024-03-06T07:33:02.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.826+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d0069050 con 0x7f02d00712d0 2024-03-06T07:33:02.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.827+0000 7f02c7400640 1 -- 172.21.15.114:0/408794487 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02c004b060 con 0x7f02d00712d0 2024-03-06T07:33:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.827+0000 7f02c7400640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02b006b4d0 0x7f02b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.828+0000 7f02c7400640 1 -- 172.21.15.114:0/408794487 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f02c00abf20 con 0x7f02d00712d0 2024-03-06T07:33:02.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.828+0000 7f02cea00640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02b006b4d0 0x7f02b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.828+0000 7f02cea00640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02b006b4d0 0x7f02b006d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f02d01f8e10 tx=0x7f02bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:02.833+0000 7f02c7400640 1 -- 172.21.15.114:0/408794487 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02d0069050 con 0x7f02d00712d0 2024-03-06T07:33:03.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.026+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f02d004f330 con 0x7f02d00712d0 2024-03-06T07:33:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.817+0000 7f02c7400640 1 -- 172.21.15.114:0/408794487 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v90) v1 ==== 138+0+0 (secure 0 0 0) 0x7f02c003f070 con 0x7f02d00712d0 2024-03-06T07:33:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02b006b4d0 msgr2=0x7f02b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02b006b4d0 0x7f02b006d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f02d01f8e10 tx=0x7f02bc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d00712d0 msgr2=0x7f02d015f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d00712d0 0x7f02d015f360 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f02c0000c00 tx=0x7f02c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 shutdown_connections 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02d0145980 0x7f02d013f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02d00ced20 0x7f02d015d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02b006b4d0 0x7f02b006d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 --2- 172.21.15.114:0/408794487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02d00712d0 0x7f02d015f360 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 >> 172.21.15.114:0/408794487 conn(0x7f02d00cfa80 msgr2=0x7f02d006db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 shutdown_connections 2024-03-06T07:33:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:03.821+0000 7f02d6000640 1 -- 172.21.15.114:0/408794487 wait complete. 2024-03-06T07:33:03.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:33:04.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.049+0000 7f6c69600640 1 Processor -- start 2024-03-06T07:33:04.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.049+0000 7f6c69600640 1 -- start start 2024-03-06T07:33:04.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.051+0000 7f6c69600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c640b3df0 0x7f6c640727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.051+0000 7f6c69600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6c640b3650 0x7f6c640b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.051+0000 7f6c69600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c640b4710 con 0x7f6c640cda90 2024-03-06T07:33:04.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.051+0000 7f6c69600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c64072310 con 0x7f6c640b3df0 2024-03-06T07:33:04.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.051+0000 7f6c69600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c640719e0 con 0x7f6c640b3650 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c62a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c640b3df0 0x7f6c640727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c63400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6c640cda90 0x7f6c6406c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50674/0 (socket says 172.21.15.114:50674) 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c63400640 1 -- 172.21.15.114:0/1839001329 learned_addr learned my addr 172.21.15.114:0/1839001329 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c5a000640 1 --2- 172.21.15.114:0/1839001329 >> v2:172.21.15.177:3300/0 conn(0x7f6c640b3650 0x7f6c640b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c62a00640 1 -- 172.21.15.114:0/1839001329 >> v2:172.21.15.177:3300/0 conn(0x7f6c640b3650 msgr2=0x7f6c640b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c62a00640 1 --2- 172.21.15.114:0/1839001329 >> v2:172.21.15.177:3300/0 conn(0x7f6c640b3650 0x7f6c640b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c62000640 1 -- 172.21.15.114:0/1839001329 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399355291 0 0) 0x7f6c640b4710 con 0x7f6c640cda90 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c62a00640 1 -- 172.21.15.114:0/1839001329 >> v1:172.21.15.114:6789/0 conn(0x7f6c640cda90 legacy=0x7f6c6406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.052+0000 7f6c62a00640 1 -- 172.21.15.114:0/1839001329 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c64071cc0 con 0x7f6c640b3df0 2024-03-06T07:33:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.053+0000 7f6c62a00640 1 --2- 172.21.15.114:0/1839001329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3df0 0x7f6c640727a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6c640b4c90 tx=0x7f6c5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14607a9f42ca61c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c62000640 1 -- 172.21.15.114:0/1839001329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54035020 con 0x7f6c640b3df0 2024-03-06T07:33:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c62000640 1 -- 172.21.15.114:0/1839001329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c54031830 con 0x7f6c640b3df0 2024-03-06T07:33:04.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c62000640 1 -- 172.21.15.114:0/1839001329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54031c00 con 0x7f6c640b3df0 2024-03-06T07:33:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 -- 172.21.15.114:0/1839001329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3df0 msgr2=0x7f6c640727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 --2- 172.21.15.114:0/1839001329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3df0 0x7f6c640727a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6c640b4c90 tx=0x7f6c5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 -- 172.21.15.114:0/1839001329 shutdown_connections 2024-03-06T07:33:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 --2- 172.21.15.114:0/1839001329 >> v2:172.21.15.177:3300/0 conn(0x7f6c640b3650 0x7f6c640b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 --2- 172.21.15.114:0/1839001329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3df0 0x7f6c640727a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 -- 172.21.15.114:0/1839001329 >> 172.21.15.114:0/1839001329 conn(0x7f6c640b1940 msgr2=0x7f6c640b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 -- 172.21.15.114:0/1839001329 shutdown_connections 2024-03-06T07:33:04.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.054+0000 7f6c69600640 1 -- 172.21.15.114:0/1839001329 wait complete. 2024-03-06T07:33:04.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.056+0000 7f6c69600640 1 Processor -- start 2024-03-06T07:33:04.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.056+0000 7f6c69600640 1 -- start start 2024-03-06T07:33:04.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c69600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3650 0x7f6c641549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c69600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c640b3df0 0x7f6c6415d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c69600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c640cda90 0x7f6c64157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c69600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6415b240 con 0x7f6c640b3650 2024-03-06T07:33:04.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c69600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c640b3b30 con 0x7f6c640cda90 2024-03-06T07:33:04.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c69600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6415b3c0 con 0x7f6c640b3df0 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3650 0x7f6c641549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3650 0x7f6c641549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42126/0 (socket says 172.21.15.114:42126) 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.057+0000 7f6c62a00640 1 -- 172.21.15.114:0/440080947 learned_addr learned my addr 172.21.15.114:0/440080947 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c5a000640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c640b3df0 0x7f6c6415d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c62a00640 1 -- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c640b3df0 msgr2=0x7f6c6415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c62a00640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c640b3df0 0x7f6c6415d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c63400640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c640cda90 0x7f6c64157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c62a00640 1 -- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c640cda90 msgr2=0x7f6c64157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c62a00640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c640cda90 0x7f6c64157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c62a00640 1 -- 172.21.15.114:0/440080947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6415c810 con 0x7f6c640b3650 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c62a00640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3650 0x7f6c641549b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009890 tx=0x7f6c540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.058+0000 7f6c5b400640 1 -- 172.21.15.114:0/440080947 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54035070 con 0x7f6c640b3650 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.059+0000 7f6c5b400640 1 -- 172.21.15.114:0/440080947 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c54004250 con 0x7f6c640b3650 2024-03-06T07:33:04.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.059+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6415caf0 con 0x7f6c640b3650 2024-03-06T07:33:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.059+0000 7f6c5b400640 1 -- 172.21.15.114:0/440080947 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5403a6f0 con 0x7f6c640b3650 2024-03-06T07:33:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.059+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6415d0b0 con 0x7f6c640b3650 2024-03-06T07:33:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.060+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c640cf510 con 0x7f6c640b3650 2024-03-06T07:33:04.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.061+0000 7f6c5b400640 1 -- 172.21.15.114:0/440080947 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c54038020 con 0x7f6c640b3650 2024-03-06T07:33:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.062+0000 7f6c5b400640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.062+0000 7f6c5b400640 1 -- 172.21.15.114:0/440080947 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f6c540abd90 con 0x7f6c640b3650 2024-03-06T07:33:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.062+0000 7f6c5a000640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.063+0000 7f6c5a000640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6c640cf190 tx=0x7f6c4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.067+0000 7f6c5b400640 1 -- 172.21.15.114:0/440080947 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c640cf510 con 0x7f6c640b3650 2024-03-06T07:33:04.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.263+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6c6404f330 con 0x7f6c640b3650 2024-03-06T07:33:04.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.824+0000 7f6c5b400640 1 -- 172.21.15.114:0/440080947 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v91) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6c54040070 con 0x7f6c640b3650 2024-03-06T07:33:04.826 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:33:04.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.828+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 msgr2=0x7f6c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.828+0000 7f6c69600640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6c640cf190 tx=0x7f6c4c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:04.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.828+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3650 msgr2=0x7f6c641549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.828+0000 7f6c69600640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3650 0x7f6c641549b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009890 tx=0x7f6c540046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.829+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 shutdown_connections 2024-03-06T07:33:04.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.829+0000 7f6c69600640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c640cda90 0x7f6c64157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.829+0000 7f6c69600640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c640b3df0 0x7f6c6415d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.829+0000 7f6c69600640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c4006b4d0 0x7f6c4006d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.829+0000 7f6c69600640 1 --2- 172.21.15.114:0/440080947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c640b3650 0x7f6c641549b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.829+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 >> 172.21.15.114:0/440080947 conn(0x7f6c640b1940 msgr2=0x7f6c640ce770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.832+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 shutdown_connections 2024-03-06T07:33:04.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:04.832+0000 7f6c69600640 1 -- 172.21.15.114:0/440080947 wait complete. 2024-03-06T07:33:04.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:33:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.050+0000 7f5b63200640 1 Processor -- start 2024-03-06T07:33:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.050+0000 7f5b63200640 1 -- start start 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.051+0000 7f5b63200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 0x7f5b640ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.051+0000 7f5b63200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5b640cfde0 0x7f5b640cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.051+0000 7f5b63200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6415b130 con 0x7f5b64158950 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.051+0000 7f5b63200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b640af170 con 0x7f5b640cfde0 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.051+0000 7f5b63200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6415b3f0 con 0x7f5b64158570 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b62800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5b640cfde0 0x7f5b640cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b62800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5b640cfde0 0x7f5b640cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42138/0 (socket says 172.21.15.114:42138) 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b62800640 1 -- 172.21.15.114:0/1802648095 learned_addr learned my addr 172.21.15.114:0/1802648095 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b61e00640 1 --2- 172.21.15.114:0/1802648095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 0x7f5b640ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b62800640 1 -- 172.21.15.114:0/1802648095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 msgr2=0x7f5b640ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b62800640 1 --2- 172.21.15.114:0/1802648095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 0x7f5b640ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b62800640 1 -- 172.21.15.114:0/1802648095 >> v1:172.21.15.177:6789/0 conn(0x7f5b64158950 legacy=0x7f5b640cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.052+0000 7f5b62800640 1 -- 172.21.15.114:0/1802648095 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6415b6d0 con 0x7f5b640cfde0 2024-03-06T07:33:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.053+0000 7f5b62800640 1 --2- 172.21.15.114:0/1802648095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b640cfde0 0x7f5b640cd7c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009e50 tx=0x7f5b5402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61da2176e82efc5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b60a00640 1 -- 172.21.15.114:0/1802648095 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5402c9e0 con 0x7f5b640cfde0 2024-03-06T07:33:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b60a00640 1 -- 172.21.15.114:0/1802648095 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5402cb80 con 0x7f5b640cfde0 2024-03-06T07:33:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b60a00640 1 -- 172.21.15.114:0/1802648095 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b540343f0 con 0x7f5b640cfde0 2024-03-06T07:33:05.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 -- 172.21.15.114:0/1802648095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b640cfde0 msgr2=0x7f5b640cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 --2- 172.21.15.114:0/1802648095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b640cfde0 0x7f5b640cd7c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009e50 tx=0x7f5b5402be40 comp rx=0 tx=0).stop 2024-03-06T07:33:05.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 -- 172.21.15.114:0/1802648095 shutdown_connections 2024-03-06T07:33:05.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 --2- 172.21.15.114:0/1802648095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b640cfde0 0x7f5b640cd7c0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 --2- 172.21.15.114:0/1802648095 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 0x7f5b640ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 -- 172.21.15.114:0/1802648095 >> 172.21.15.114:0/1802648095 conn(0x7f5b6406afc0 msgr2=0x7f5b6406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 -- 172.21.15.114:0/1802648095 shutdown_connections 2024-03-06T07:33:05.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.054+0000 7f5b63200640 1 -- 172.21.15.114:0/1802648095 wait complete. 2024-03-06T07:33:05.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.056+0000 7f5b63200640 1 Processor -- start 2024-03-06T07:33:05.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.056+0000 7f5b63200640 1 -- start start 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b63200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b640cfde0 0x7f5b640aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b63200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 0x7f5b640cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b63200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64158950 0x7f5b640cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b63200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6415bed0 con 0x7f5b64158950 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b63200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640b4950 con 0x7f5b640cfde0 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b63200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6415c050 con 0x7f5b64158570 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b62800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64158950 0x7f5b640cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b62800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64158950 0x7f5b640cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42140/0 (socket says 172.21.15.114:42140) 2024-03-06T07:33:05.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.057+0000 7f5b62800640 1 -- 172.21.15.114:0/676627700 learned_addr learned my addr 172.21.15.114:0/676627700 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:05.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b61e00640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b640cfde0 0x7f5b640aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b62800640 1 -- 172.21.15.114:0/676627700 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 msgr2=0x7f5b640cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b62800640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 0x7f5b640cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b62800640 1 -- 172.21.15.114:0/676627700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b640cfde0 msgr2=0x7f5b640aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b62800640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b640cfde0 0x7f5b640aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b62800640 1 -- 172.21.15.114:0/676627700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6414f3c0 con 0x7f5b64158950 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b62800640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64158950 0x7f5b640cc830 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009e50 tx=0x7f5b54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b5a000640 1 -- 172.21.15.114:0/676627700 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54043020 con 0x7f5b64158950 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b5a000640 1 -- 172.21.15.114:0/676627700 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5403e450 con 0x7f5b64158950 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b5a000640 1 -- 172.21.15.114:0/676627700 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54009510 con 0x7f5b64158950 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.058+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6414f6a0 con 0x7f5b64158950 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.059+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6414fc60 con 0x7f5b64158950 2024-03-06T07:33:05.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.060+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6404f330 con 0x7f5b64158950 2024-03-06T07:33:05.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.061+0000 7f5b5a000640 1 -- 172.21.15.114:0/676627700 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b54046020 con 0x7f5b64158950 2024-03-06T07:33:05.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.062+0000 7f5b5a000640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5b4006b4d0 0x7f5b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.062+0000 7f5b5a000640 1 -- 172.21.15.114:0/676627700 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f5b540ace60 con 0x7f5b64158950 2024-03-06T07:33:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.062+0000 7f5b61e00640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5b4006b4d0 0x7f5b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.063+0000 7f5b61e00640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5b4006b4d0 0x7f5b4006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c006fd0 tx=0x7f5b5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.066+0000 7f5b5a000640 1 -- 172.21.15.114:0/676627700 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b5404d030 con 0x7f5b64158950 2024-03-06T07:33:05.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.299+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5b64157660 con 0x7f5b64158950 2024-03-06T07:33:05.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.841+0000 7f5b5a000640 1 -- 172.21.15.114:0/676627700 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v57) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5b5407bd20 con 0x7f5b64158950 2024-03-06T07:33:05.843 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:33:05.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5b4006b4d0 msgr2=0x7f5b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5b4006b4d0 0x7f5b4006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c006fd0 tx=0x7f5b5c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:05.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64158950 msgr2=0x7f5b640cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64158950 0x7f5b640cc830 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009e50 tx=0x7f5b54004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:05.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 shutdown_connections 2024-03-06T07:33:05.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5b4006b4d0 0x7f5b4006d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64158950 0x7f5b640cc830 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5b64158570 0x7f5b640cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 --2- 172.21.15.114:0/676627700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b640cfde0 0x7f5b640aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.845+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 >> 172.21.15.114:0/676627700 conn(0x7f5b6406afc0 msgr2=0x7f5b640ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.847+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 shutdown_connections 2024-03-06T07:33:05.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:05.847+0000 7f5b63200640 1 -- 172.21.15.114:0/676627700 wait complete. 2024-03-06T07:33:05.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:06.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.060+0000 7f64eca00640 1 Processor -- start 2024-03-06T07:33:06.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.060+0000 7f64eca00640 1 -- start start 2024-03-06T07:33:06.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.061+0000 7f64eca00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f64e815e1d0 0x7f64e806db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.061+0000 7f64eca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 0x7f64e8167b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.061+0000 7f64eca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e806e160 con 0x7f64e806afc0 2024-03-06T07:33:06.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64eca00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e806ed70 con 0x7f64e815e1d0 2024-03-06T07:33:06.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64eca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e806e440 con 0x7f64e80aec50 2024-03-06T07:33:06.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64e5e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f64e806afc0 0x7f64e806b3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50678/0 (socket says 172.21.15.114:50678) 2024-03-06T07:33:06.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64e5e00640 1 -- 172.21.15.114:0/2393332735 learned_addr learned my addr 172.21.15.114:0/2393332735 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:06.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64e6800640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 0x7f64e8167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64dfe00640 1 --2- 172.21.15.114:0/2393332735 >> v2:172.21.15.177:3300/0 conn(0x7f64e815e1d0 0x7f64e806db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261568826 0 0) 0x7f64e806e160 con 0x7f64e806afc0 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.062+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc003520 con 0x7f64e806afc0 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812002180 0 0) 0x7f64cc003520 con 0x7f64e806afc0 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 msgr2=0x7f64e8167b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 0x7f64e8167b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 >> v2:172.21.15.177:3300/0 conn(0x7f64e815e1d0 msgr2=0x7f64e806db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 --2- 172.21.15.114:0/2393332735 >> v2:172.21.15.177:3300/0 conn(0x7f64e815e1d0 0x7f64e806db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e806e720 con 0x7f64e806afc0 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f64d8001670 con 0x7f64e806afc0 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 >> v1:172.21.15.114:6789/0 conn(0x7f64e806afc0 legacy=0x7f64e806b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 0x7f64cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64cc004d40 0x7f64cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e806ed70 con 0x7f64cc0039b0 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e806e440 con 0x7f64cc004d40 2024-03-06T07:33:06.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc00b9f0 con 0x7f64e80aec50 2024-03-06T07:33:06.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.063+0000 7f64e5e00640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.064+0000 7f64dfe00640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 0x7f64cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.064+0000 7f64e5e00640 1 -- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 msgr2=0x7f64cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.064+0000 7f64e5e00640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64e80aec50 0x7f64cc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.064+0000 7f64e5e00640 1 -- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64cc004d40 msgr2=0x7f64cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.064+0000 7f64e5e00640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64cc004d40 0x7f64cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.064+0000 7f64e5e00640 1 -- 172.21.15.114:0/2393332735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc003520 con 0x7f64cc0039b0 2024-03-06T07:33:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64e5e00640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c160 tx=0x7f64cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cf3cdf4812d75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8007c80 con 0x7f64cc0039b0 2024-03-06T07:33:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d8030070 con 0x7f64cc0039b0 2024-03-06T07:33:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64e5400640 1 -- 172.21.15.114:0/2393332735 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8005d30 con 0x7f64cc0039b0 2024-03-06T07:33:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64eca00640 1 -- 172.21.15.114:0/2393332735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64cc0039b0 msgr2=0x7f64cc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64eca00640 1 --2- 172.21.15.114:0/2393332735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c160 tx=0x7f64cc005830 comp rx=0 tx=0).stop 2024-03-06T07:33:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64e6800640 1 -- 172.21.15.114:0/2393332735 reap_dead start 2024-03-06T07:33:06.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64eca00640 1 -- 172.21.15.114:0/2393332735 shutdown_connections 2024-03-06T07:33:06.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.065+0000 7f64eca00640 1 -- 172.21.15.114:0/2393332735 >> 172.21.15.114:0/2393332735 conn(0x7f64e80d0b40 msgr2=0x7f64e80d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.066+0000 7f64eca00640 1 -- 172.21.15.114:0/2393332735 shutdown_connections 2024-03-06T07:33:06.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.066+0000 7f64eca00640 1 -- 172.21.15.114:0/2393332735 wait complete. 2024-03-06T07:33:06.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.067+0000 7f64eca00640 1 Processor -- start 2024-03-06T07:33:06.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.067+0000 7f64eca00640 1 -- start start 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64eca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64cc0039b0 0x7f64e815a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64eca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64e81e6780 0x7f64e80ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64e80ce590 0x7f64e80cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64eca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80d1630 con 0x7f64e80ce590 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64eca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80704b0 con 0x7f64e81e6780 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64eca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80d17b0 con 0x7f64cc0039b0 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64e6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64e80ce590 0x7f64e80cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64e6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64e80ce590 0x7f64e80cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42158/0 (socket says 172.21.15.114:42158) 2024-03-06T07:33:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.068+0000 7f64e6800640 1 -- 172.21.15.114:0/791405300 learned_addr learned my addr 172.21.15.114:0/791405300 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64e6800640 1 -- 172.21.15.114:0/791405300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64cc0039b0 msgr2=0x7f64e815a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64e6800640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64cc0039b0 0x7f64e815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64e6800640 1 -- 172.21.15.114:0/791405300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64e81e6780 msgr2=0x7f64e80ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64e6800640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64e81e6780 0x7f64e80ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64e6800640 1 -- 172.21.15.114:0/791405300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8159640 con 0x7f64e80ce590 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64e6800640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64e80ce590 0x7f64e80cea10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f64e000ee50 tx=0x7f64e000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64de000640 1 -- 172.21.15.114:0/791405300 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0018020 con 0x7f64e80ce590 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64de000640 1 -- 172.21.15.114:0/791405300 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e000bc50 con 0x7f64e80ce590 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64de000640 1 -- 172.21.15.114:0/791405300 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0014400 con 0x7f64e80ce590 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.069+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e815aee0 con 0x7f64e80ce590 2024-03-06T07:33:06.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.070+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e815b4a0 con 0x7f64e80ce590 2024-03-06T07:33:06.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.071+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e80ccd70 con 0x7f64e80ce590 2024-03-06T07:33:06.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.071+0000 7f64de000640 1 -- 172.21.15.114:0/791405300 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64e0005d10 con 0x7f64e80ce590 2024-03-06T07:33:06.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.072+0000 7f64de000640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.072+0000 7f64de000640 1 -- 172.21.15.114:0/791405300 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64e008a7c0 con 0x7f64e80ce590 2024-03-06T07:33:06.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.073+0000 7f64e5e00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.073+0000 7f64e5e00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f64c406b4d0 0x7f64c406d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f64d8034000 tx=0x7f64d8039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.077+0000 7f64de000640 1 -- 172.21.15.114:0/791405300 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64e80ccd70 con 0x7f64e80ce590 2024-03-06T07:33:06.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.257+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64e804f330 con 0x7f64e80ce590 2024-03-06T07:33:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.259+0000 7f64de000640 1 -- 172.21.15.114:0/791405300 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16669 (secure 0 0 0) 0x7f64e0022030 con 0x7f64e80ce590 2024-03-06T07:33:06.261 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:06.261 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":92,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:05.300911+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:03.077157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:04.314220+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f64c406b4d0 msgr2=0x7f64c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f64c406b4d0 0x7f64c406d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f64d8034000 tx=0x7f64d8039040 comp rx=0 tx=0).stop 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64e80ce590 msgr2=0x7f64e80cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64e80ce590 0x7f64e80cea10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f64e000ee50 tx=0x7f64e000b480 comp rx=0 tx=0).stop 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 shutdown_connections 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f64c406b4d0 0x7f64c406d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64e80ce590 0x7f64e80cea10 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64e81e6780 0x7f64e80ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 --2- 172.21.15.114:0/791405300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f64cc0039b0 0x7f64e815a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 >> 172.21.15.114:0/791405300 conn(0x7f64e80d0b40 msgr2=0x7f64e8072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 shutdown_connections 2024-03-06T07:33:06.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.262+0000 7f64eca00640 1 -- 172.21.15.114:0/791405300 wait complete. 2024-03-06T07:33:06.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.478+0000 7fb792200640 1 Processor -- start 2024-03-06T07:33:06.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.478+0000 7fb792200640 1 -- start start 2024-03-06T07:33:06.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.479+0000 7fb792200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb78c15acb0 0x7fb78c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.479+0000 7fb792200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 0x7fb78c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb792200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c15b090 con 0x7fb78c0b4680 2024-03-06T07:33:06.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb792200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c0b2a30 con 0x7fb78c15acb0 2024-03-06T07:33:06.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb792200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0b4360 con 0x7fb78c06ec00 2024-03-06T07:33:06.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb782a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb78c0b4680 0x7fb78c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50684/0 (socket says 172.21.15.114:50684) 2024-03-06T07:33:06.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb782a00640 1 -- 172.21.15.114:0/2357866125 learned_addr learned my addr 172.21.15.114:0/2357866125 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb78be00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 0x7fb78c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb78b400640 1 --2- 172.21.15.114:0/2357866125 >> v2:172.21.15.177:3300/0 conn(0x7fb78c15acb0 0x7fb78c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386318756 0 0) 0x7fb78c15b090 con 0x7fb78c0b4680 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.480+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb76c005730 con 0x7fb78c0b4680 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1873972038 0 0) 0x7fb76c005730 con 0x7fb78c0b4680 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 msgr2=0x7fb78c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 0x7fb78c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 >> v2:172.21.15.177:3300/0 conn(0x7fb78c15acb0 msgr2=0x7fb78c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 --2- 172.21.15.114:0/2357866125 >> v2:172.21.15.177:3300/0 conn(0x7fb78c15acb0 0x7fb78c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0b2cf0 con 0x7fb78c0b4680 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78b400640 1 --2- 172.21.15.114:0/2357866125 >> v2:172.21.15.177:3300/0 conn(0x7fb78c15acb0 0x7fb78c0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb774001670 con 0x7fb78c0b4680 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 >> v1:172.21.15.114:6789/0 conn(0x7fb78c0b4680 legacy=0x7fb78c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb76c005bc0 0x7fb76c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 0x7fb76c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb76c006f50 0x7fb76c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0b2a30 con 0x7fb76c006f50 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0b4360 con 0x7fb76c005bc0 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c00dcb0 con 0x7fb78c06ec00 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78be00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb76c006f50 0x7fb76c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb78b400640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb76c005bc0 0x7fb76c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.481+0000 7fb782a00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 0x7fb76c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.482+0000 7fb78be00640 1 -- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 msgr2=0x7fb76c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.482+0000 7fb78be00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c06ec00 0x7fb76c006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.482+0000 7fb78be00640 1 -- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb76c005bc0 msgr2=0x7fb76c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.482+0000 7fb78be00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb76c005bc0 0x7fb76c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.482+0000 7fb78be00640 1 -- 172.21.15.114:0/2357866125 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c005730 con 0x7fb76c006f50 2024-03-06T07:33:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb78be00640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb76c006f50 0x7fb76c0073d0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb784007330 tx=0x7fb76c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e378da34189b6a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784039070 con 0x7fb76c006f50 2024-03-06T07:33:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78402cc90 con 0x7fb76c006f50 2024-03-06T07:33:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb78aa00640 1 -- 172.21.15.114:0/2357866125 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784009be0 con 0x7fb76c006f50 2024-03-06T07:33:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb792200640 1 -- 172.21.15.114:0/2357866125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb76c006f50 msgr2=0x7fb76c0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb792200640 1 --2- 172.21.15.114:0/2357866125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb76c006f50 0x7fb76c0073d0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb784007330 tx=0x7fb76c007a40 comp rx=0 tx=0).stop 2024-03-06T07:33:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb78be00640 1 -- 172.21.15.114:0/2357866125 reap_dead start 2024-03-06T07:33:06.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb792200640 1 -- 172.21.15.114:0/2357866125 shutdown_connections 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.483+0000 7fb792200640 1 -- 172.21.15.114:0/2357866125 >> 172.21.15.114:0/2357866125 conn(0x7fb78c0b1730 msgr2=0x7fb78c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.484+0000 7fb792200640 1 -- 172.21.15.114:0/2357866125 shutdown_connections 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.484+0000 7fb792200640 1 -- 172.21.15.114:0/2357866125 wait complete. 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.485+0000 7fb792200640 1 Processor -- start 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.486+0000 7fb792200640 1 -- start start 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.486+0000 7fb792200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb78c159c50 0x7fb78c157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.486+0000 7fb792200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c1580f0 0x7fb78c1f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.486+0000 7fb792200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb78c15a030 0x7fb78c1f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.486+0000 7fb792200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0b3d10 con 0x7fb78c159c50 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.486+0000 7fb792200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0b32b0 con 0x7fb78c15a030 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.486+0000 7fb792200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0b4010 con 0x7fb78c1580f0 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb78c159c50 0x7fb78c157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb78c159c50 0x7fb78c157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42162/0 (socket says 172.21.15.114:42162) 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 -- 172.21.15.114:0/3735117584 learned_addr learned my addr 172.21.15.114:0/3735117584 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb782a00640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c1580f0 0x7fb78c1f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 -- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c1580f0 msgr2=0x7fb78c1f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c1580f0 0x7fb78c1f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 -- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb78c15a030 msgr2=0x7fb78c1f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78be00640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb78c15a030 0x7fb78c1f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb78c15a030 0x7fb78c1f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 -- 172.21.15.114:0/3735117584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c158590 con 0x7fb78c159c50 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.487+0000 7fb78b400640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb78c159c50 0x7fb78c157bb0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb77c010890 tx=0x7fb77c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.488+0000 7fb783e00640 1 -- 172.21.15.114:0/3735117584 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c01a020 con 0x7fb78c159c50 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.488+0000 7fb783e00640 1 -- 172.21.15.114:0/3735117584 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77c00eb60 con 0x7fb78c159c50 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.488+0000 7fb783e00640 1 -- 172.21.15.114:0/3735117584 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c00f040 con 0x7fb78c159c50 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.488+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c158870 con 0x7fb78c159c50 2024-03-06T07:33:06.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.488+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c158e30 con 0x7fb78c159c50 2024-03-06T07:33:06.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.489+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c04f330 con 0x7fb78c159c50 2024-03-06T07:33:06.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.490+0000 7fb783e00640 1 -- 172.21.15.114:0/3735117584 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb77c01d020 con 0x7fb78c159c50 2024-03-06T07:33:06.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.491+0000 7fb783e00640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb76806b4d0 0x7fb76806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.491+0000 7fb783e00640 1 -- 172.21.15.114:0/3735117584 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb77c08ad20 con 0x7fb78c159c50 2024-03-06T07:33:06.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.492+0000 7fb782a00640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb76806b4d0 0x7fb76806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.492+0000 7fb782a00640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb76806b4d0 0x7fb76806d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb774004840 tx=0x7fb774000fe0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.496+0000 7fb783e00640 1 -- 172.21.15.114:0/3735117584 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb77c024020 con 0x7fb78c159c50 2024-03-06T07:33:06.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.677+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb78c15f250 con 0x7fb78c159c50 2024-03-06T07:33:06.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.680+0000 7fb783e00640 1 -- 172.21.15.114:0/3735117584 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16669 (secure 0 0 0) 0x7fb77c059b00 con 0x7fb78c159c50 2024-03-06T07:33:06.682 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:06.682 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":92,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:05.300911+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:03.077157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:04.314220+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:06.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb76806b4d0 msgr2=0x7fb76806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb76806b4d0 0x7fb76806d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb774004840 tx=0x7fb774000fe0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb78c159c50 msgr2=0x7fb78c157bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb78c159c50 0x7fb78c157bb0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb77c010890 tx=0x7fb77c010d50 comp rx=0 tx=0).stop 2024-03-06T07:33:06.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 shutdown_connections 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb78c15a030 0x7fb78c1f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb78c1580f0 0x7fb78c1f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb76806b4d0 0x7fb76806d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 --2- 172.21.15.114:0/3735117584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb78c159c50 0x7fb78c157bb0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 >> 172.21.15.114:0/3735117584 conn(0x7fb78c0b1730 msgr2=0x7fb78c0ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.682+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 shutdown_connections 2024-03-06T07:33:06.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.683+0000 7fb792200640 1 -- 172.21.15.114:0/3735117584 wait complete. 2024-03-06T07:33:06.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:33:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.908+0000 7fbfdec00640 1 Processor -- start 2024-03-06T07:33:06.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.908+0000 7fbfdec00640 1 -- start start 2024-03-06T07:33:06.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.909+0000 7fbfdec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 0x7fbfd80d39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.909+0000 7fbfdec00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd80d4370 con 0x7fbfd80d3ee0 2024-03-06T07:33:06.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.909+0000 7fbfdec00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd80cf020 con 0x7fbfd815e000 2024-03-06T07:33:06.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.909+0000 7fbfdec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80cf300 con 0x7fbfd80ce890 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.909+0000 7fbfde200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbfd80d3ee0 0x7fbfd80b3de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50688/0 (socket says 172.21.15.114:50688) 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.909+0000 7fbfde200640 1 -- 172.21.15.114:0/4046636080 learned_addr learned my addr 172.21.15.114:0/4046636080 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfdce00640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 0x7fbfd80d39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59431645 0 0) 0x7fbfd80d4370 con 0x7fbfd80d3ee0 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0003520 con 0x7fbfd80d3ee0 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2478286819 0 0) 0x7fbfc0003520 con 0x7fbfd80d3ee0 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 msgr2=0x7fbfd80d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 0x7fbfd80d39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 >> v1:172.21.15.177:6789/0 conn(0x7fbfd815e000 legacy=0x7fbfd80d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80cf5e0 con 0x7fbfd80d3ee0 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fbfd00019f0 con 0x7fbfd80d3ee0 2024-03-06T07:33:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 >> v1:172.21.15.114:6789/0 conn(0x7fbfd80d3ee0 legacy=0x7fbfd80b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfc00039e0 0x7fbfc0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 0x7fbfc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc0004d40 0x7fbfc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80cf300 con 0x7fbfc0004d40 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc000bb50 con 0x7fbfc00039e0 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.910+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc000bd20 con 0x7fbfd80ce890 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfde200640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc0004d40 0x7fbfc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfdd800640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfc00039e0 0x7fbfc0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfdce00640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 0x7fbfc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfde200640 1 -- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 msgr2=0x7fbfc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfde200640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd80ce890 0x7fbfc0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfde200640 1 -- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfc00039e0 msgr2=0x7fbfc0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfde200640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfc00039e0 0x7fbfc0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.911+0000 7fbfde200640 1 -- 172.21.15.114:0/4046636080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc0003520 con 0x7fbfc0004d40 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.913+0000 7fbfde200640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc0004d40 0x7fbfc00051c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000a880 tx=0x7fbfc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be812cbdab3ef986 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.913+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000a4a0 con 0x7fbfc0004d40 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.913+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0034650 con 0x7fbfc0004d40 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.913+0000 7fbfd7e00640 1 -- 172.21.15.114:0/4046636080 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0034a40 con 0x7fbfc0004d40 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.913+0000 7fbfdec00640 1 -- 172.21.15.114:0/4046636080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc0004d40 msgr2=0x7fbfc00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.913+0000 7fbfdec00640 1 --2- 172.21.15.114:0/4046636080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc0004d40 0x7fbfc00051c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000a880 tx=0x7fbfc0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.913+0000 7fbfde200640 1 -- 172.21.15.114:0/4046636080 reap_dead start 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.914+0000 7fbfdec00640 1 -- 172.21.15.114:0/4046636080 shutdown_connections 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.914+0000 7fbfdec00640 1 -- 172.21.15.114:0/4046636080 >> 172.21.15.114:0/4046636080 conn(0x7fbfd808d740 msgr2=0x7fbfd809d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.914+0000 7fbfdec00640 1 -- 172.21.15.114:0/4046636080 shutdown_connections 2024-03-06T07:33:06.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.914+0000 7fbfdec00640 1 -- 172.21.15.114:0/4046636080 wait complete. 2024-03-06T07:33:06.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.915+0000 7fbfdec00640 1 Processor -- start 2024-03-06T07:33:06.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.915+0000 7fbfdec00640 1 -- start start 2024-03-06T07:33:06.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.916+0000 7fbfdec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd81f31f0 0x7fbfd80ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.916+0000 7fbfdec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd8148d30 0x7fbfd80b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.916+0000 7fbfdec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd806cb50 0x7fbfd806cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.916+0000 7fbfdec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80cecf0 con 0x7fbfd8148d30 2024-03-06T07:33:06.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.916+0000 7fbfdec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80cfba0 con 0x7fbfd806cb50 2024-03-06T07:33:06.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.916+0000 7fbfdec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80d39a0 con 0x7fbfd81f31f0 2024-03-06T07:33:06.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd8148d30 0x7fbfd80b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd8148d30 0x7fbfd80b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42180/0 (socket says 172.21.15.114:42180) 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 -- 172.21.15.114:0/1600324141 learned_addr learned my addr 172.21.15.114:0/1600324141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfde200640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd806cb50 0x7fbfd806cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdd800640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd81f31f0 0x7fbfd80ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 -- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd81f31f0 msgr2=0x7fbfd80ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd81f31f0 0x7fbfd80ca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 -- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd806cb50 msgr2=0x7fbfd806cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd806cb50 0x7fbfd806cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 -- 172.21.15.114:0/1600324141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80b2470 con 0x7fbfd8148d30 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfde200640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd806cb50 0x7fbfd806cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.917+0000 7fbfdce00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd8148d30 0x7fbfd80b3230 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400f580 tx=0x7fbfc400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.918+0000 7fbfd5600640 1 -- 172.21.15.114:0/1600324141 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4019020 con 0x7fbfd8148d30 2024-03-06T07:33:06.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.918+0000 7fbfd5600640 1 -- 172.21.15.114:0/1600324141 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc4014870 con 0x7fbfd8148d30 2024-03-06T07:33:06.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.918+0000 7fbfd5600640 1 -- 172.21.15.114:0/1600324141 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4014c60 con 0x7fbfd8148d30 2024-03-06T07:33:06.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.918+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd80b2750 con 0x7fbfd8148d30 2024-03-06T07:33:06.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.918+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd80b3770 con 0x7fbfd8148d30 2024-03-06T07:33:06.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.919+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd804f330 con 0x7fbfd8148d30 2024-03-06T07:33:06.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.920+0000 7fbfd5600640 1 -- 172.21.15.114:0/1600324141 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfc401c020 con 0x7fbfd8148d30 2024-03-06T07:33:06.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.922+0000 7fbfd5600640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbfb806b500 0x7fbfb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.922+0000 7fbfd5600640 1 -- 172.21.15.114:0/1600324141 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfc408aa30 con 0x7fbfd8148d30 2024-03-06T07:33:06.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.922+0000 7fbfdd800640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbfb806b500 0x7fbfb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.922+0000 7fbfdd800640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbfb806b500 0x7fbfb806d9c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc005590 tx=0x7fbfcc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:06.925+0000 7fbfd5600640 1 -- 172.21.15.114:0/1600324141 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfc4023020 con 0x7fbfd8148d30 2024-03-06T07:33:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.161+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbfd815ccb0 con 0x7fbfd8148d30 2024-03-06T07:33:07.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.889+0000 7fbfd5600640 1 -- 172.21.15.114:0/1600324141 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbfc40598c0 con 0x7fbfd8148d30 2024-03-06T07:33:07.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.892+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbfb806b500 msgr2=0x7fbfb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.892+0000 7fbfdec00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbfb806b500 0x7fbfb806d9c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc005590 tx=0x7fbfcc009040 comp rx=0 tx=0).stop 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.892+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd8148d30 msgr2=0x7fbfd80b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.892+0000 7fbfdec00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd8148d30 0x7fbfd80b3230 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400f580 tx=0x7fbfc400fa40 comp rx=0 tx=0).stop 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 shutdown_connections 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd806cb50 0x7fbfd806cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbfb806b500 0x7fbfb806d9c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd8148d30 0x7fbfd80b3230 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 --2- 172.21.15.114:0/1600324141 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbfd81f31f0 0x7fbfd80ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 >> 172.21.15.114:0/1600324141 conn(0x7fbfd808d740 msgr2=0x7fbfd815ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 shutdown_connections 2024-03-06T07:33:07.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:07.893+0000 7fbfdec00640 1 -- 172.21.15.114:0/1600324141 wait complete. 2024-03-06T07:33:07.919 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:33:07.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:33:08.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.120+0000 7f56b0000640 1 Processor -- start 2024-03-06T07:33:08.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.120+0000 7f56b0000640 1 -- start start 2024-03-06T07:33:08.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.121+0000 7f56b0000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56a80adf70 0x7f56a80b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56b0000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a8162310 0x7f56a8164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56b0000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80b3e50 con 0x7f56a80b3530 2024-03-06T07:33:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56b0000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a80ce690 con 0x7f56a80adf70 2024-03-06T07:33:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56b0000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80ce970 con 0x7f56a8162310 2024-03-06T07:33:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56ad600640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f56a80b3530 0x7f56a815b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53024/0 (socket says 172.21.15.114:53024) 2024-03-06T07:33:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56ad600640 1 -- 172.21.15.114:0/2315450526 learned_addr learned my addr 172.21.15.114:0/2315450526 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56acc00640 1 --2- 172.21.15.114:0/2315450526 >> v2:172.21.15.114:3300/0 conn(0x7f56a80adf70 0x7f56a80b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56ae000640 1 --2- 172.21.15.114:0/2315450526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a8162310 0x7f56a8164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56a7e00640 1 -- 172.21.15.114:0/2315450526 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076756158 0 0) 0x7f56a80b3e50 con 0x7f56a80b3530 2024-03-06T07:33:08.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56a7e00640 1 -- 172.21.15.114:0/2315450526 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5690003520 con 0x7f56a80b3530 2024-03-06T07:33:08.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56acc00640 1 -- 172.21.15.114:0/2315450526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a8162310 msgr2=0x7f56a8164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.122+0000 7f56acc00640 1 --2- 172.21.15.114:0/2315450526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a8162310 0x7f56a8164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.123+0000 7f56acc00640 1 -- 172.21.15.114:0/2315450526 >> v1:172.21.15.114:6789/0 conn(0x7f56a80b3530 legacy=0x7f56a815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.123+0000 7f56acc00640 1 -- 172.21.15.114:0/2315450526 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80cec50 con 0x7f56a80adf70 2024-03-06T07:33:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.124+0000 7f56acc00640 1 --2- 172.21.15.114:0/2315450526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80adf70 0x7f56a80b3910 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f56940097a0 tx=0x7f569402dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1c579377027aa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56a7e00640 1 -- 172.21.15.114:0/2315450526 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569402f8a0 con 0x7f56a80adf70 2024-03-06T07:33:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56a7e00640 1 -- 172.21.15.114:0/2315450526 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569402fa40 con 0x7f56a80adf70 2024-03-06T07:33:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56a7e00640 1 -- 172.21.15.114:0/2315450526 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569402fe10 con 0x7f56a80adf70 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 -- 172.21.15.114:0/2315450526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80adf70 msgr2=0x7f56a80b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 --2- 172.21.15.114:0/2315450526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80adf70 0x7f56a80b3910 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f56940097a0 tx=0x7f569402dae0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 -- 172.21.15.114:0/2315450526 shutdown_connections 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 --2- 172.21.15.114:0/2315450526 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a8162310 0x7f56a8164700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 --2- 172.21.15.114:0/2315450526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80adf70 0x7f56a80b3910 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 -- 172.21.15.114:0/2315450526 >> 172.21.15.114:0/2315450526 conn(0x7f56a80b1940 msgr2=0x7f56a80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 -- 172.21.15.114:0/2315450526 shutdown_connections 2024-03-06T07:33:08.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.125+0000 7f56b0000640 1 -- 172.21.15.114:0/2315450526 wait complete. 2024-03-06T07:33:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.127+0000 7f56b0000640 1 Processor -- start 2024-03-06T07:33:08.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.127+0000 7f56b0000640 1 -- start start 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.128+0000 7f56b0000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a80adf70 0x7f56a814a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.128+0000 7f56b0000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80b3530 0x7f56a809d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.128+0000 7f56b0000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8162310 0x7f56a81f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.128+0000 7f56b0000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a815aa30 con 0x7f56a80b3530 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.128+0000 7f56b0000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80cf210 con 0x7f56a8162310 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.128+0000 7f56b0000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a815abb0 con 0x7f56a80adf70 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80b3530 0x7f56a809d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80b3530 0x7f56a809d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60750/0 (socket says 172.21.15.114:60750) 2024-03-06T07:33:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 -- 172.21.15.114:0/887406000 learned_addr learned my addr 172.21.15.114:0/887406000 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56ae000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8162310 0x7f56a81f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 -- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a80adf70 msgr2=0x7f56a814a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56ad600640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a80adf70 0x7f56a814a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a80adf70 0x7f56a814a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 -- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8162310 msgr2=0x7f56a81f7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8162310 0x7f56a81f7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 -- 172.21.15.114:0/887406000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a815d070 con 0x7f56a80b3530 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56ae000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8162310 0x7f56a81f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:08.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.129+0000 7f56acc00640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80b3530 0x7f56a809d300 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f569402e0f0 tx=0x7f56940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.130+0000 7f56a5600640 1 -- 172.21.15.114:0/887406000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694047070 con 0x7f56a80b3530 2024-03-06T07:33:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.130+0000 7f56a5600640 1 -- 172.21.15.114:0/887406000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5694030ab0 con 0x7f56a80b3530 2024-03-06T07:33:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.130+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a815d350 con 0x7f56a80b3530 2024-03-06T07:33:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.130+0000 7f56a5600640 1 -- 172.21.15.114:0/887406000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694042410 con 0x7f56a80b3530 2024-03-06T07:33:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.130+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a815d910 con 0x7f56a80b3530 2024-03-06T07:33:08.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.131+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a804f330 con 0x7f56a80b3530 2024-03-06T07:33:08.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.132+0000 7f56a5600640 1 -- 172.21.15.114:0/887406000 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5694039400 con 0x7f56a80b3530 2024-03-06T07:33:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.133+0000 7f56a5600640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.133+0000 7f56a5600640 1 -- 172.21.15.114:0/887406000 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56940abcf0 con 0x7f56a80b3530 2024-03-06T07:33:08.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.134+0000 7f56ad600640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.134+0000 7f56ad600640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f569c000f60 tx=0x7f569c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.138+0000 7f56a5600640 1 -- 172.21.15.114:0/887406000 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5694035030 con 0x7f56a80b3530 2024-03-06T07:33:08.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.373+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f56a8071a10 con 0x7f56a80b3530 2024-03-06T07:33:08.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.911+0000 7f56a5600640 1 -- 172.21.15.114:0/887406000 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7f569407aad0 con 0x7f56a80b3530 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 msgr2=0x7f568806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f569c000f60 tx=0x7f569c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80b3530 msgr2=0x7f56a809d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80b3530 0x7f56a809d300 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f569402e0f0 tx=0x7f56940046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 shutdown_connections 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8162310 0x7f56a81f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a80b3530 0x7f56a809d300 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 --2- 172.21.15.114:0/887406000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a80adf70 0x7f56a814a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 >> 172.21.15.114:0/887406000 conn(0x7f56a80b1940 msgr2=0x7f56a80b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 shutdown_connections 2024-03-06T07:33:08.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:08.915+0000 7f56b0000640 1 -- 172.21.15.114:0/887406000 wait complete. 2024-03-06T07:33:08.937 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:33:08.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:09.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.138+0000 7f948b200640 1 Processor -- start 2024-03-06T07:33:09.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.139+0000 7f948b200640 1 -- start start 2024-03-06T07:33:09.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.140+0000 7f948b200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f948406b6d0 0x7f94840cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.140+0000 7f948b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948415d4f0 con 0x7f94840d0ca0 2024-03-06T07:33:09.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.140+0000 7f948b200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948415cc50 con 0x7f9484160530 2024-03-06T07:33:09.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.140+0000 7f948b200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94840b50a0 con 0x7f948406b6d0 2024-03-06T07:33:09.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.140+0000 7f9483e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f94840d0ca0 0x7f948406afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53026/0 (socket says 172.21.15.114:53026) 2024-03-06T07:33:09.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.140+0000 7f9483e00640 1 -- 172.21.15.114:0/3402475056 learned_addr learned my addr 172.21.15.114:0/3402475056 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:09.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9483400640 1 --2- 172.21.15.114:0/3402475056 >> v2:172.21.15.177:3300/0 conn(0x7f948406b6d0 0x7f94840cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869239414 0 0) 0x7f948415d4f0 con 0x7f94840d0ca0 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9468003520 con 0x7f94840d0ca0 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628710633 0 0) 0x7f9468003520 con 0x7f94840d0ca0 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 >> v2:172.21.15.177:3300/0 conn(0x7f948406b6d0 msgr2=0x7f94840cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 --2- 172.21.15.114:0/3402475056 >> v2:172.21.15.177:3300/0 conn(0x7f948406b6d0 0x7f94840cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 >> v1:172.21.15.177:6789/0 conn(0x7f9484160530 legacy=0x7f94840b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948415cf30 con 0x7f94840d0ca0 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f9474001670 con 0x7f94840d0ca0 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 >> v1:172.21.15.114:6789/0 conn(0x7f94840d0ca0 legacy=0x7f948406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.141+0000 7f9482a00640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9468003a00 0x7f9468003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9482a00640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468004800 0x7f9468004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9482a00640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9468005210 0x7f946800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94840b50a0 con 0x7f9468003a00 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946800c160 con 0x7f9468005210 2024-03-06T07:33:09.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946800c360 con 0x7f9468004800 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9483e00640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9468003a00 0x7f9468003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9483400640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468004800 0x7f9468004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9489200640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9468005210 0x7f946800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9483e00640 1 -- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468004800 msgr2=0x7f9468004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9483e00640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468004800 0x7f9468004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9483e00640 1 -- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9468005210 msgr2=0x7f946800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9483e00640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9468005210 0x7f946800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9483e00640 1 -- 172.21.15.114:0/3402475056 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468003520 con 0x7f9468003a00 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9489200640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9468005210 0x7f946800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:09.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.142+0000 7f9489200640 1 -- 172.21.15.114:0/3402475056 reap_dead start 2024-03-06T07:33:09.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.143+0000 7f9483e00640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9468003a00 0x7f9468003e20 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f947400c160 tx=0x7f9468005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d68a90e4ed4b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9474007c60 con 0x7f9468003a00 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9474030070 con 0x7f9468003a00 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f9482a00640 1 -- 172.21.15.114:0/3402475056 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9474005db0 con 0x7f9468003a00 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f948b200640 1 -- 172.21.15.114:0/3402475056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9468003a00 msgr2=0x7f9468003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f948b200640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9468003a00 0x7f9468003e20 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f947400c160 tx=0x7f9468005830 comp rx=0 tx=0).stop 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f948b200640 1 -- 172.21.15.114:0/3402475056 shutdown_connections 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f948b200640 1 --2- 172.21.15.114:0/3402475056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9468003a00 0x7f9468003e20 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f948b200640 1 -- 172.21.15.114:0/3402475056 >> 172.21.15.114:0/3402475056 conn(0x7f94840b1e10 msgr2=0x7f94840b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f948b200640 1 -- 172.21.15.114:0/3402475056 shutdown_connections 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.144+0000 7f948b200640 1 -- 172.21.15.114:0/3402475056 wait complete. 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.146+0000 7f948b200640 1 Processor -- start 2024-03-06T07:33:09.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.146+0000 7f948b200640 1 -- start start 2024-03-06T07:33:09.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f948b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468003a00 0x7f9484155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f948b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94841565f0 0x7f94841e6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f948b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94840cb780 0x7f94840cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f948b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94840d1080 con 0x7f94841565f0 2024-03-06T07:33:09.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f948b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9484160960 con 0x7f94840cb780 2024-03-06T07:33:09.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f948b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94840cfe50 con 0x7f9468003a00 2024-03-06T07:33:09.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f9483400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94841565f0 0x7f94841e6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f9483400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94841565f0 0x7f94841e6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60762/0 (socket says 172.21.15.114:60762) 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.147+0000 7f9483400640 1 -- 172.21.15.114:0/3613415602 learned_addr learned my addr 172.21.15.114:0/3613415602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f9483400640 1 -- 172.21.15.114:0/3613415602 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468003a00 msgr2=0x7f9484155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f9483400640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468003a00 0x7f9484155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f9483400640 1 -- 172.21.15.114:0/3613415602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94840cb780 msgr2=0x7f94840cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f9489200640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94840cb780 0x7f94840cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f9483400640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94840cb780 0x7f94840cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f9483400640 1 -- 172.21.15.114:0/3613415602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9484156330 con 0x7f94841565f0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f9483400640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94841565f0 0x7f94841e6200 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f9470010910 tx=0x7f9470010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f947be00640 1 -- 172.21.15.114:0/3613415602 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947000f040 con 0x7f94841565f0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f947be00640 1 -- 172.21.15.114:0/3613415602 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947000ec20 con 0x7f94841565f0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.148+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94841e6ac0 con 0x7f94841565f0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.149+0000 7f947be00640 1 -- 172.21.15.114:0/3613415602 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947001c400 con 0x7f94841565f0 2024-03-06T07:33:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.149+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94841e7080 con 0x7f94841565f0 2024-03-06T07:33:09.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.150+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948404f330 con 0x7f94841565f0 2024-03-06T07:33:09.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.151+0000 7f947be00640 1 -- 172.21.15.114:0/3613415602 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f947001c5a0 con 0x7f94841565f0 2024-03-06T07:33:09.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.152+0000 7f947be00640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f946406b550 0x7f946406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.152+0000 7f947be00640 1 -- 172.21.15.114:0/3613415602 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f947008a7d0 con 0x7f94841565f0 2024-03-06T07:33:09.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.152+0000 7f9483e00640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f946406b550 0x7f946406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.153+0000 7f9483e00640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f946406b550 0x7f946406da10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9468004780 tx=0x7f9474007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.156+0000 7f947be00640 1 -- 172.21.15.114:0/3613415602 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9470018030 con 0x7f94841565f0 2024-03-06T07:33:09.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.374+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94840b2ae0 con 0x7f94841565f0 2024-03-06T07:33:09.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.374+0000 7f947be00640 1 -- 172.21.15.114:0/3613415602 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f9470059630 con 0x7f94841565f0 2024-03-06T07:33:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:09.377 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1475876370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1475876370},{"type":"v1","addr":"172.21.15.114:6836","nonce":1475876370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5734,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2451255436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2451255436},{"type":"v1","addr":"172.21.15.177:6833","nonce":2451255436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5737,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3480882059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3480882059},{"type":"v1","addr":"172.21.15.177:6835","nonce":3480882059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:08.374434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/406262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":406262499},{"type":"v1","addr":"172.21.15.114:6837","nonce":406262499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":7}]} 2024-03-06T07:33:09.377 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 61 2024-03-06T07:33:09.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.376+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f946406b550 msgr2=0x7f946406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.376+0000 7f948b200640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f946406b550 0x7f946406da10 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9468004780 tx=0x7f9474007000 comp rx=0 tx=0).stop 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.376+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94841565f0 msgr2=0x7f94841e6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.376+0000 7f948b200640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94841565f0 0x7f94841e6200 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f9470010910 tx=0x7f9470010dd0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 shutdown_connections 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94840cb780 0x7f94840cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f946406b550 0x7f946406da10 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94841565f0 0x7f94841e6200 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 --2- 172.21.15.114:0/3613415602 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9468003a00 0x7f9484155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 >> 172.21.15.114:0/3613415602 conn(0x7f94840b1e10 msgr2=0x7f94840ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 shutdown_connections 2024-03-06T07:33:09.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.377+0000 7f948b200640 1 -- 172.21.15.114:0/3613415602 wait complete. 2024-03-06T07:33:09.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:09.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.580+0000 7fe856400640 1 Processor -- start 2024-03-06T07:33:09.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.580+0000 7fe856400640 1 -- start start 2024-03-06T07:33:09.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.581+0000 7fe856400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe850144c00 0x7fe85006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.581+0000 7fe856400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe85015fe60 0x7fe8500ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.581+0000 7fe856400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85006ee70 con 0x7fe85015f9d0 2024-03-06T07:33:09.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.581+0000 7fe856400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85006cef0 con 0x7fe85015fe60 2024-03-06T07:33:09.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.581+0000 7fe856400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85006d1d0 con 0x7fe850144c00 2024-03-06T07:33:09.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe85015fe60 0x7fe8500ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe85015fe60 0x7fe8500ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60772/0 (socket says 172.21.15.114:60772) 2024-03-06T07:33:09.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84fe00640 1 -- 172.21.15.114:0/3509832256 learned_addr learned my addr 172.21.15.114:0/3509832256 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:09.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84f400640 1 --2- 172.21.15.114:0/3509832256 >> v2:172.21.15.177:3300/0 conn(0x7fe850144c00 0x7fe85006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84fe00640 1 -- 172.21.15.114:0/3509832256 >> v2:172.21.15.177:3300/0 conn(0x7fe850144c00 msgr2=0x7fe85006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84fe00640 1 --2- 172.21.15.114:0/3509832256 >> v2:172.21.15.177:3300/0 conn(0x7fe850144c00 0x7fe85006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84fe00640 1 -- 172.21.15.114:0/3509832256 >> v1:172.21.15.177:6789/0 conn(0x7fe85015f9d0 legacy=0x7fe8500ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.582+0000 7fe84fe00640 1 -- 172.21.15.114:0/3509832256 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85006d4b0 con 0x7fe85015fe60 2024-03-06T07:33:09.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.583+0000 7fe84fe00640 1 --2- 172.21.15.114:0/3509832256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85015fe60 0x7fe8500ce7b0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fe8500b25f0 tx=0x7fe84802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c631b9033c0f5fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.584+0000 7fe84ea00640 1 -- 172.21.15.114:0/3509832256 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848035020 con 0x7fe85015fe60 2024-03-06T07:33:09.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.584+0000 7fe84ea00640 1 -- 172.21.15.114:0/3509832256 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe848031870 con 0x7fe85015fe60 2024-03-06T07:33:09.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.584+0000 7fe84ea00640 1 -- 172.21.15.114:0/3509832256 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848031c40 con 0x7fe85015fe60 2024-03-06T07:33:09.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.584+0000 7fe856400640 1 -- 172.21.15.114:0/3509832256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85015fe60 msgr2=0x7fe8500ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.584+0000 7fe856400640 1 --2- 172.21.15.114:0/3509832256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85015fe60 0x7fe8500ce7b0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fe8500b25f0 tx=0x7fe84802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:09.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.585+0000 7fe856400640 1 -- 172.21.15.114:0/3509832256 shutdown_connections 2024-03-06T07:33:09.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.585+0000 7fe856400640 1 --2- 172.21.15.114:0/3509832256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85015fe60 0x7fe8500ce7b0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.585+0000 7fe856400640 1 --2- 172.21.15.114:0/3509832256 >> v2:172.21.15.177:3300/0 conn(0x7fe850144c00 0x7fe85006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.585+0000 7fe856400640 1 -- 172.21.15.114:0/3509832256 >> 172.21.15.114:0/3509832256 conn(0x7fe8500b1730 msgr2=0x7fe8500b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.585+0000 7fe856400640 1 -- 172.21.15.114:0/3509832256 shutdown_connections 2024-03-06T07:33:09.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.585+0000 7fe856400640 1 -- 172.21.15.114:0/3509832256 wait complete. 2024-03-06T07:33:09.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.587+0000 7fe856400640 1 Processor -- start 2024-03-06T07:33:09.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.587+0000 7fe856400640 1 -- start start 2024-03-06T07:33:09.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe856400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe850144c00 0x7fe850139350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe856400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85015f9d0 0x7fe8500b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe856400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85015fe60 0x7fe85015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe856400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe850160730 con 0x7fe850144c00 2024-03-06T07:33:09.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe856400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe850160270 con 0x7fe85015f9d0 2024-03-06T07:33:09.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe856400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85006cb50 con 0x7fe85015fe60 2024-03-06T07:33:09.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe84f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe850144c00 0x7fe850139350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe84f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe850144c00 0x7fe850139350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60788/0 (socket says 172.21.15.114:60788) 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.589+0000 7fe84f400640 1 -- 172.21.15.114:0/3463113904 learned_addr learned my addr 172.21.15.114:0/3463113904 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe846a00640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85015f9d0 0x7fe8500b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe84f400640 1 -- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85015fe60 msgr2=0x7fe85015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe84f400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85015fe60 0x7fe85015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe84f400640 1 -- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85015f9d0 msgr2=0x7fe8500b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe84f400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85015f9d0 0x7fe8500b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe84f400640 1 -- 172.21.15.114:0/3463113904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500b3720 con 0x7fe850144c00 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe846a00640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85015f9d0 0x7fe8500b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe84f400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe850144c00 0x7fe850139350 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fe840007e40 tx=0x7fe84000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe847e00640 1 -- 172.21.15.114:0/3463113904 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840014620 con 0x7fe850144c00 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe847e00640 1 -- 172.21.15.114:0/3463113904 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840019070 con 0x7fe850144c00 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe847e00640 1 -- 172.21.15.114:0/3463113904 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840014a00 con 0x7fe850144c00 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.590+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8500b3a00 con 0x7fe850144c00 2024-03-06T07:33:09.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.591+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8500b3fc0 con 0x7fe850144c00 2024-03-06T07:33:09.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.592+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85004f330 con 0x7fe850144c00 2024-03-06T07:33:09.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.592+0000 7fe847e00640 1 -- 172.21.15.114:0/3463113904 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe840014ba0 con 0x7fe850144c00 2024-03-06T07:33:09.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.594+0000 7fe847e00640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe82c06b4d0 0x7fe82c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.594+0000 7fe847e00640 1 -- 172.21.15.114:0/3463113904 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe84008a940 con 0x7fe850144c00 2024-03-06T07:33:09.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.594+0000 7fe846a00640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe82c06b4d0 0x7fe82c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.595+0000 7fe846a00640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe82c06b4d0 0x7fe82c06d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe850140490 tx=0x7fe838009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.598+0000 7fe847e00640 1 -- 172.21.15.114:0/3463113904 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe840059850 con 0x7fe850144c00 2024-03-06T07:33:09.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.779+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe850072890 con 0x7fe850144c00 2024-03-06T07:33:09.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.780+0000 7fe847e00640 1 -- 172.21.15.114:0/3463113904 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16656 (secure 0 0 0) 0x7fe840059150 con 0x7fe850144c00 2024-03-06T07:33:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:09.782 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":93,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:06.841892+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:03.077157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:04.314220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe82c06b4d0 msgr2=0x7fe82c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe82c06b4d0 0x7fe82c06d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe850140490 tx=0x7fe838009040 comp rx=0 tx=0).stop 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe850144c00 msgr2=0x7fe850139350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe850144c00 0x7fe850139350 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fe840007e40 tx=0x7fe84000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 shutdown_connections 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85015fe60 0x7fe85015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85015f9d0 0x7fe8500b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe82c06b4d0 0x7fe82c06d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 --2- 172.21.15.114:0/3463113904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe850144c00 0x7fe850139350 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 >> 172.21.15.114:0/3463113904 conn(0x7fe8500b1730 msgr2=0x7fe8501434e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 shutdown_connections 2024-03-06T07:33:09.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.783+0000 7fe856400640 1 -- 172.21.15.114:0/3463113904 wait complete. 2024-03-06T07:33:09.803 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:33:09.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:09.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.997+0000 7fb6a7200640 1 Processor -- start 2024-03-06T07:33:09.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.998+0000 7fb6a7200640 1 -- start start 2024-03-06T07:33:10.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.999+0000 7fb6a7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a81561f0 0x7fb6a80ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.999+0000 7fb6a7200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a815c240 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.999+0000 7fb6a7200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a815c500 con 0x7fb6a80aebf0 2024-03-06T07:33:10.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.999+0000 7fb6a7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a815ce30 con 0x7fb6a81561f0 2024-03-06T07:33:10.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:09.999+0000 7fb6a5e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a81561f0 0x7fb6a80ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.000+0000 7fb6a6800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb6a80cc9d0 0x7fb6a815f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53028/0 (socket says 172.21.15.114:53028) 2024-03-06T07:33:10.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.000+0000 7fb6a6800640 1 -- 172.21.15.114:0/2831921562 learned_addr learned my addr 172.21.15.114:0/2831921562 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:10.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.000+0000 7fb6a5e00640 1 -- 172.21.15.114:0/2831921562 >> v1:172.21.15.177:6789/0 conn(0x7fb6a80aebf0 legacy=0x7fb6a80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.000+0000 7fb6a5e00640 1 -- 172.21.15.114:0/2831921562 >> v1:172.21.15.114:6789/0 conn(0x7fb6a80cc9d0 legacy=0x7fb6a815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.000+0000 7fb6a5e00640 1 -- 172.21.15.114:0/2831921562 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a815c7e0 con 0x7fb6a81561f0 2024-03-06T07:33:10.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.001+0000 7fb6a5e00640 1 --2- 172.21.15.114:0/2831921562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a81561f0 0x7fb6a80ae6b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb6980097a0 tx=0x7fb69802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5323cc9dda0f5b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.001+0000 7fb6a5400640 1 -- 172.21.15.114:0/2831921562 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698035020 con 0x7fb6a81561f0 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.001+0000 7fb6a5400640 1 -- 172.21.15.114:0/2831921562 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb698031830 con 0x7fb6a81561f0 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.001+0000 7fb6a5400640 1 -- 172.21.15.114:0/2831921562 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698031c00 con 0x7fb6a81561f0 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.002+0000 7fb6a7200640 1 -- 172.21.15.114:0/2831921562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a81561f0 msgr2=0x7fb6a80ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.002+0000 7fb6a7200640 1 --2- 172.21.15.114:0/2831921562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a81561f0 0x7fb6a80ae6b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fb6980097a0 tx=0x7fb69802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.002+0000 7fb6a7200640 1 -- 172.21.15.114:0/2831921562 shutdown_connections 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.002+0000 7fb6a7200640 1 --2- 172.21.15.114:0/2831921562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a81561f0 0x7fb6a80ae6b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.002+0000 7fb6a7200640 1 -- 172.21.15.114:0/2831921562 >> 172.21.15.114:0/2831921562 conn(0x7fb6a806afc0 msgr2=0x7fb6a806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.002+0000 7fb6a7200640 1 -- 172.21.15.114:0/2831921562 shutdown_connections 2024-03-06T07:33:10.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.002+0000 7fb6a7200640 1 -- 172.21.15.114:0/2831921562 wait complete. 2024-03-06T07:33:10.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.004+0000 7fb6a7200640 1 Processor -- start 2024-03-06T07:33:10.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.004+0000 7fb6a7200640 1 -- start start 2024-03-06T07:33:10.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb6a7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a80aebf0 0x7fb6a8147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb6a7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80cc9d0 0x7fb6a80a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb6a7200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a81561f0 0x7fb6a80aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb6a7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a81565d0 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb6a7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a806e360 con 0x7fb6a80aebf0 2024-03-06T07:33:10.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb6a7200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8156750 con 0x7fb6a81561f0 2024-03-06T07:33:10.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb69d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80cc9d0 0x7fb6a80a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb69d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80cc9d0 0x7fb6a80a9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60808/0 (socket says 172.21.15.114:60808) 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb69d400640 1 -- 172.21.15.114:0/4240020116 learned_addr learned my addr 172.21.15.114:0/4240020116 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.005+0000 7fb6a5e00640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a80aebf0 0x7fb6a8147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69d400640 1 -- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a81561f0 msgr2=0x7fb6a80aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb6a6800640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a81561f0 0x7fb6a80aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69d400640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a81561f0 0x7fb6a80aa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69d400640 1 -- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a80aebf0 msgr2=0x7fb6a8147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69d400640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a80aebf0 0x7fb6a8147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69d400640 1 -- 172.21.15.114:0/4240020116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a80ad920 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb6a6800640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a81561f0 0x7fb6a80aa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb6a5e00640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a80aebf0 0x7fb6a8147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69d400640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80cc9d0 0x7fb6a80a9d80 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c430 tx=0x7fb69000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69ea00640 1 -- 172.21.15.114:0/4240020116 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690011790 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69ea00640 1 -- 172.21.15.114:0/4240020116 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb690017070 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb69ea00640 1 -- 172.21.15.114:0/4240020116 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690011bb0 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a80adc00 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.006+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a80ae1c0 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.007+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a804f330 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.008+0000 7fb69ea00640 1 -- 172.21.15.114:0/4240020116 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb69001c460 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.010+0000 7fb69ea00640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.010+0000 7fb69ea00640 1 -- 172.21.15.114:0/4240020116 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb69008a100 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.010+0000 7fb6a5e00640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.011+0000 7fb6a5e00640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb698036000 tx=0x7fb698008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.014+0000 7fb69ea00640 1 -- 172.21.15.114:0/4240020116 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb690058f70 con 0x7fb6a80cc9d0 2024-03-06T07:33:10.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.212+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6a80b2b10 con 0x7fb68406b4d0 2024-03-06T07:33:10.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.222+0000 7fb69ea00640 1 -- 172.21.15.114:0/4240020116 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252085 (secure 0 0 0) 0x7fb6a80b2b10 con 0x7fb68406b4d0 2024-03-06T07:33:10.227 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:10.229 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:33:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 msgr2=0x7fb68406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb698036000 tx=0x7fb698008030 comp rx=0 tx=0).stop 2024-03-06T07:33:10.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80cc9d0 msgr2=0x7fb6a80a9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80cc9d0 0x7fb6a80a9d80 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c430 tx=0x7fb69000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 shutdown_connections 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a81561f0 0x7fb6a80aa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a80cc9d0 0x7fb6a80a9d80 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 --2- 172.21.15.114:0/4240020116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a80aebf0 0x7fb6a8147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.229+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 >> 172.21.15.114:0/4240020116 conn(0x7fb6a806afc0 msgr2=0x7fb6a8155220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.230+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 shutdown_connections 2024-03-06T07:33:10.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.230+0000 7fb6a7200640 1 -- 172.21.15.114:0/4240020116 wait complete. 2024-03-06T07:33:10.257 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-06T07:33:09.194890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259696,"kb_used_data":10216,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715024,"statfs":{"total":773094113280,"available":772828184576,"internally_reserved":0,"allocated":10461184,"data_stored":5123340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.373559"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004621+0000","last_change":"2024-03-06T07:33:05.877782+0000","last_active":"2024-03-06T07:33:07.004621+0000","last_peered":"2024-03-06T07:33:07.004621+0000","last_clean":"2024-03-06T07:33:07.004621+0000","last_became_active":"2024-03-06T07:33:05.875599+0000","last_became_peered":"2024-03-06T07:33:05.875599+0000","last_unstale":"2024-03-06T07:33:07.004621+0000","last_undegraded":"2024-03-06T07:33:07.004621+0000","last_fullsized":"2024-03-06T07:33:07.004621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:06.559368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851780+0000","last_change":"2024-03-06T07:33:05.879937+0000","last_active":"2024-03-06T07:33:06.851780+0000","last_peered":"2024-03-06T07:33:06.851780+0000","last_clean":"2024-03-06T07:33:06.851780+0000","last_became_active":"2024-03-06T07:33:05.879676+0000","last_became_peered":"2024-03-06T07:33:05.879676+0000","last_unstale":"2024-03-06T07:33:06.851780+0000","last_undegraded":"2024-03-06T07:33:06.851780+0000","last_fullsized":"2024-03-06T07:33:06.851780+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:26.856826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004478+0000","last_change":"2024-03-06T07:33:05.881262+0000","last_active":"2024-03-06T07:33:07.004478+0000","last_peered":"2024-03-06T07:33:07.004478+0000","last_clean":"2024-03-06T07:33:07.004478+0000","last_became_active":"2024-03-06T07:33:05.881000+0000","last_became_peered":"2024-03-06T07:33:05.881000+0000","last_unstale":"2024-03-06T07:33:07.004478+0000","last_undegraded":"2024-03-06T07:33:07.004478+0000","last_fullsized":"2024-03-06T07:33:07.004478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:37.331334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853865+0000","last_change":"2024-03-06T07:33:05.859166+0000","last_active":"2024-03-06T07:33:06.853865+0000","last_peered":"2024-03-06T07:33:06.853865+0000","last_clean":"2024-03-06T07:33:06.853865+0000","last_became_active":"2024-03-06T07:33:05.858881+0000","last_became_peered":"2024-03-06T07:33:05.858881+0000","last_unstale":"2024-03-06T07:33:06.853865+0000","last_undegraded":"2024-03-06T07:33:06.853865+0000","last_fullsized":"2024-03-06T07:33:06.853865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005554+0000","last_change":"2024-03-06T07:33:05.881488+0000","last_active":"2024-03-06T07:33:07.005554+0000","last_peered":"2024-03-06T07:33:07.005554+0000","last_clean":"2024-03-06T07:33:07.005554+0000","last_became_active":"2024-03-06T07:33:05.881104+0000","last_became_peered":"2024-03-06T07:33:05.881104+0000","last_unstale":"2024-03-06T07:33:07.005554+0000","last_undegraded":"2024-03-06T07:33:07.005554+0000","last_fullsized":"2024-03-06T07:33:07.005554+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006399+0000","last_change":"2024-03-06T07:33:05.874065+0000","last_active":"2024-03-06T07:33:07.006399+0000","last_peered":"2024-03-06T07:33:07.006399+0000","last_clean":"2024-03-06T07:33:07.006399+0000","last_became_active":"2024-03-06T07:33:05.873755+0000","last_became_peered":"2024-03-06T07:33:05.873755+0000","last_unstale":"2024-03-06T07:33:07.006399+0000","last_undegraded":"2024-03-06T07:33:07.006399+0000","last_fullsized":"2024-03-06T07:33:07.006399+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006345+0000","last_change":"2024-03-06T07:33:05.874759+0000","last_active":"2024-03-06T07:33:07.006345+0000","last_peered":"2024-03-06T07:33:07.006345+0000","last_clean":"2024-03-06T07:33:07.006345+0000","last_became_active":"2024-03-06T07:33:05.874462+0000","last_became_peered":"2024-03-06T07:33:05.874462+0000","last_unstale":"2024-03-06T07:33:07.006345+0000","last_undegraded":"2024-03-06T07:33:07.006345+0000","last_fullsized":"2024-03-06T07:33:07.006345+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:13.128225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.850870+0000","last_change":"2024-03-06T07:33:05.876477+0000","last_active":"2024-03-06T07:33:06.850870+0000","last_peered":"2024-03-06T07:33:06.850870+0000","last_clean":"2024-03-06T07:33:06.850870+0000","last_became_active":"2024-03-06T07:33:05.870488+0000","last_became_peered":"2024-03-06T07:33:05.870488+0000","last_unstale":"2024-03-06T07:33:06.850870+0000","last_undegraded":"2024-03-06T07:33:06.850870+0000","last_fullsized":"2024-03-06T07:33:06.850870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853135+0000","last_change":"2024-03-06T07:33:05.883628+0000","last_active":"2024-03-06T07:33:06.853135+0000","last_peered":"2024-03-06T07:33:06.853135+0000","last_clean":"2024-03-06T07:33:06.853135+0000","last_became_active":"2024-03-06T07:33:05.883240+0000","last_became_peered":"2024-03-06T07:33:05.883240+0000","last_unstale":"2024-03-06T07:33:06.853135+0000","last_undegraded":"2024-03-06T07:33:06.853135+0000","last_fullsized":"2024-03-06T07:33:06.853135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:26.301240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005033+0000","last_change":"2024-03-06T07:33:05.889679+0000","last_active":"2024-03-06T07:33:07.005033+0000","last_peered":"2024-03-06T07:33:07.005033+0000","last_clean":"2024-03-06T07:33:07.005033+0000","last_became_active":"2024-03-06T07:33:05.888726+0000","last_became_peered":"2024-03-06T07:33:05.888726+0000","last_unstale":"2024-03-06T07:33:07.005033+0000","last_undegraded":"2024-03-06T07:33:07.005033+0000","last_fullsized":"2024-03-06T07:33:07.005033+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.450793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007051+0000","last_change":"2024-03-06T07:33:05.886460+0000","last_active":"2024-03-06T07:33:07.007051+0000","last_peered":"2024-03-06T07:33:07.007051+0000","last_clean":"2024-03-06T07:33:07.007051+0000","last_became_active":"2024-03-06T07:33:05.884698+0000","last_became_peered":"2024-03-06T07:33:05.884698+0000","last_unstale":"2024-03-06T07:33:07.007051+0000","last_undegraded":"2024-03-06T07:33:07.007051+0000","last_fullsized":"2024-03-06T07:33:07.007051+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:34.637751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852903+0000","last_change":"2024-03-06T07:33:05.891586+0000","last_active":"2024-03-06T07:33:06.852903+0000","last_peered":"2024-03-06T07:33:06.852903+0000","last_clean":"2024-03-06T07:33:06.852903+0000","last_became_active":"2024-03-06T07:33:05.891280+0000","last_became_peered":"2024-03-06T07:33:05.891280+0000","last_unstale":"2024-03-06T07:33:06.852903+0000","last_undegraded":"2024-03-06T07:33:06.852903+0000","last_fullsized":"2024-03-06T07:33:06.852903+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:24.111569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853452+0000","last_change":"2024-03-06T07:33:05.891923+0000","last_active":"2024-03-06T07:33:06.853452+0000","last_peered":"2024-03-06T07:33:06.853452+0000","last_clean":"2024-03-06T07:33:06.853452+0000","last_became_active":"2024-03-06T07:33:05.891635+0000","last_became_peered":"2024-03-06T07:33:05.891635+0000","last_unstale":"2024-03-06T07:33:06.853452+0000","last_undegraded":"2024-03-06T07:33:06.853452+0000","last_fullsized":"2024-03-06T07:33:06.853452+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:27.034207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002355+0000","last_change":"2024-03-06T07:33:05.876219+0000","last_active":"2024-03-06T07:33:07.002355+0000","last_peered":"2024-03-06T07:33:07.002355+0000","last_clean":"2024-03-06T07:33:07.002355+0000","last_became_active":"2024-03-06T07:33:05.875879+0000","last_became_peered":"2024-03-06T07:33:05.875879+0000","last_unstale":"2024-03-06T07:33:07.002355+0000","last_undegraded":"2024-03-06T07:33:07.002355+0000","last_fullsized":"2024-03-06T07:33:07.002355+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:18.844653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853784+0000","last_change":"2024-03-06T07:33:05.879553+0000","last_active":"2024-03-06T07:33:06.853784+0000","last_peered":"2024-03-06T07:33:06.853784+0000","last_clean":"2024-03-06T07:33:06.853784+0000","last_became_active":"2024-03-06T07:33:05.879289+0000","last_became_peered":"2024-03-06T07:33:05.879289+0000","last_unstale":"2024-03-06T07:33:06.853784+0000","last_undegraded":"2024-03-06T07:33:06.853784+0000","last_fullsized":"2024-03-06T07:33:06.853784+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006250+0000","last_change":"2024-03-06T07:33:05.869374+0000","last_active":"2024-03-06T07:33:07.006250+0000","last_peered":"2024-03-06T07:33:07.006250+0000","last_clean":"2024-03-06T07:33:07.006250+0000","last_became_active":"2024-03-06T07:33:05.868221+0000","last_became_peered":"2024-03-06T07:33:05.868221+0000","last_unstale":"2024-03-06T07:33:07.006250+0000","last_undegraded":"2024-03-06T07:33:07.006250+0000","last_fullsized":"2024-03-06T07:33:07.006250+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.450793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851945+0000","last_change":"2024-03-06T07:33:05.875997+0000","last_active":"2024-03-06T07:33:06.851945+0000","last_peered":"2024-03-06T07:33:06.851945+0000","last_clean":"2024-03-06T07:33:06.851945+0000","last_became_active":"2024-03-06T07:33:05.873483+0000","last_became_peered":"2024-03-06T07:33:05.873483+0000","last_unstale":"2024-03-06T07:33:06.851945+0000","last_undegraded":"2024-03-06T07:33:06.851945+0000","last_fullsized":"2024-03-06T07:33:06.851945+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:48.795019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.855234+0000","last_change":"2024-03-06T07:33:05.858934+0000","last_active":"2024-03-06T07:33:06.855234+0000","last_peered":"2024-03-06T07:33:06.855234+0000","last_clean":"2024-03-06T07:33:06.855234+0000","last_became_active":"2024-03-06T07:33:05.858626+0000","last_became_peered":"2024-03-06T07:33:05.858626+0000","last_unstale":"2024-03-06T07:33:06.855234+0000","last_undegraded":"2024-03-06T07:33:06.855234+0000","last_fullsized":"2024-03-06T07:33:06.855234+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:26.856826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003636+0000","last_change":"2024-03-06T07:33:05.889630+0000","last_active":"2024-03-06T07:33:07.003636+0000","last_peered":"2024-03-06T07:33:07.003636+0000","last_clean":"2024-03-06T07:33:07.003636+0000","last_became_active":"2024-03-06T07:33:05.889067+0000","last_became_peered":"2024-03-06T07:33:05.889067+0000","last_unstale":"2024-03-06T07:33:07.003636+0000","last_undegraded":"2024-03-06T07:33:07.003636+0000","last_fullsized":"2024-03-06T07:33:07.003636+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:14.381370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853196+0000","last_change":"2024-03-06T07:33:05.881805+0000","last_active":"2024-03-06T07:33:06.853196+0000","last_peered":"2024-03-06T07:33:06.853196+0000","last_clean":"2024-03-06T07:33:06.853196+0000","last_became_active":"2024-03-06T07:33:05.881527+0000","last_became_peered":"2024-03-06T07:33:05.881527+0000","last_unstale":"2024-03-06T07:33:06.853196+0000","last_undegraded":"2024-03-06T07:33:06.853196+0000","last_fullsized":"2024-03-06T07:33:06.853196+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:33.682790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002692+0000","last_change":"2024-03-06T07:33:05.869384+0000","last_active":"2024-03-06T07:33:07.002692+0000","last_peered":"2024-03-06T07:33:07.002692+0000","last_clean":"2024-03-06T07:33:07.002692+0000","last_became_active":"2024-03-06T07:33:05.869031+0000","last_became_peered":"2024-03-06T07:33:05.869031+0000","last_unstale":"2024-03-06T07:33:07.002692+0000","last_undegraded":"2024-03-06T07:33:07.002692+0000","last_fullsized":"2024-03-06T07:33:07.002692+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:23.211814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851388+0000","last_change":"2024-03-06T07:33:05.874060+0000","last_active":"2024-03-06T07:33:06.851388+0000","last_peered":"2024-03-06T07:33:06.851388+0000","last_clean":"2024-03-06T07:33:06.851388+0000","last_became_active":"2024-03-06T07:33:05.873742+0000","last_became_peered":"2024-03-06T07:33:05.873742+0000","last_unstale":"2024-03-06T07:33:06.851388+0000","last_undegraded":"2024-03-06T07:33:06.851388+0000","last_fullsized":"2024-03-06T07:33:06.851388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853974+0000","last_change":"2024-03-06T07:33:05.874148+0000","last_active":"2024-03-06T07:33:06.853974+0000","last_peered":"2024-03-06T07:33:06.853974+0000","last_clean":"2024-03-06T07:33:06.853974+0000","last_became_active":"2024-03-06T07:33:05.873840+0000","last_became_peered":"2024-03-06T07:33:05.873840+0000","last_unstale":"2024-03-06T07:33:06.853974+0000","last_undegraded":"2024-03-06T07:33:06.853974+0000","last_fullsized":"2024-03-06T07:33:06.853974+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:34.637751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006263+0000","last_change":"2024-03-06T07:33:05.888716+0000","last_active":"2024-03-06T07:33:07.006263+0000","last_peered":"2024-03-06T07:33:07.006263+0000","last_clean":"2024-03-06T07:33:07.006263+0000","last_became_active":"2024-03-06T07:33:05.888433+0000","last_became_peered":"2024-03-06T07:33:05.888433+0000","last_unstale":"2024-03-06T07:33:07.006263+0000","last_undegraded":"2024-03-06T07:33:07.006263+0000","last_fullsized":"2024-03-06T07:33:07.006263+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:27.034207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005094+0000","last_change":"2024-03-06T07:33:05.891127+0000","last_active":"2024-03-06T07:33:07.005094+0000","last_peered":"2024-03-06T07:33:07.005094+0000","last_clean":"2024-03-06T07:33:07.005094+0000","last_became_active":"2024-03-06T07:33:05.890664+0000","last_became_peered":"2024-03-06T07:33:05.890664+0000","last_unstale":"2024-03-06T07:33:07.005094+0000","last_undegraded":"2024-03-06T07:33:07.005094+0000","last_fullsized":"2024-03-06T07:33:07.005094+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:27.034207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852169+0000","last_change":"2024-03-06T07:33:05.882431+0000","last_active":"2024-03-06T07:33:06.852169+0000","last_peered":"2024-03-06T07:33:06.852169+0000","last_clean":"2024-03-06T07:33:06.852169+0000","last_became_active":"2024-03-06T07:33:05.882173+0000","last_became_peered":"2024-03-06T07:33:05.882173+0000","last_unstale":"2024-03-06T07:33:06.852169+0000","last_undegraded":"2024-03-06T07:33:06.852169+0000","last_fullsized":"2024-03-06T07:33:06.852169+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:29.053384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007079+0000","last_change":"2024-03-06T07:33:05.858790+0000","last_active":"2024-03-06T07:33:07.007079+0000","last_peered":"2024-03-06T07:33:07.007079+0000","last_clean":"2024-03-06T07:33:07.007079+0000","last_became_active":"2024-03-06T07:33:05.858045+0000","last_became_peered":"2024-03-06T07:33:05.858045+0000","last_unstale":"2024-03-06T07:33:07.007079+0000","last_undegraded":"2024-03-06T07:33:07.007079+0000","last_fullsized":"2024-03-06T07:33:07.007079+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:26.856826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005284+0000","last_change":"2024-03-06T07:33:05.881239+0000","last_active":"2024-03-06T07:33:07.005284+0000","last_peered":"2024-03-06T07:33:07.005284+0000","last_clean":"2024-03-06T07:33:07.005284+0000","last_became_active":"2024-03-06T07:33:05.880967+0000","last_became_peered":"2024-03-06T07:33:05.880967+0000","last_unstale":"2024-03-06T07:33:07.005284+0000","last_undegraded":"2024-03-06T07:33:07.005284+0000","last_fullsized":"2024-03-06T07:33:07.005284+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:13.128225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852919+0000","last_change":"2024-03-06T07:33:05.886968+0000","last_active":"2024-03-06T07:33:06.852919+0000","last_peered":"2024-03-06T07:33:06.852919+0000","last_clean":"2024-03-06T07:33:06.852919+0000","last_became_active":"2024-03-06T07:33:05.886681+0000","last_became_peered":"2024-03-06T07:33:05.886681+0000","last_unstale":"2024-03-06T07:33:06.852919+0000","last_undegraded":"2024-03-06T07:33:06.852919+0000","last_fullsized":"2024-03-06T07:33:06.852919+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:05.060646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002713+0000","last_change":"2024-03-06T07:33:05.881474+0000","last_active":"2024-03-06T07:33:07.002713+0000","last_peered":"2024-03-06T07:33:07.002713+0000","last_clean":"2024-03-06T07:33:07.002713+0000","last_became_active":"2024-03-06T07:33:05.881165+0000","last_became_peered":"2024-03-06T07:33:05.881165+0000","last_unstale":"2024-03-06T07:33:07.002713+0000","last_undegraded":"2024-03-06T07:33:07.002713+0000","last_fullsized":"2024-03-06T07:33:07.002713+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:58.183397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853596+0000","last_change":"2024-03-06T07:33:05.872579+0000","last_active":"2024-03-06T07:33:06.853596+0000","last_peered":"2024-03-06T07:33:06.853596+0000","last_clean":"2024-03-06T07:33:06.853596+0000","last_became_active":"2024-03-06T07:33:05.872310+0000","last_became_peered":"2024-03-06T07:33:05.872310+0000","last_unstale":"2024-03-06T07:33:06.853596+0000","last_undegraded":"2024-03-06T07:33:06.853596+0000","last_fullsized":"2024-03-06T07:33:06.853596+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:13.128225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004161+0000","last_change":"2024-03-06T07:33:05.879741+0000","last_active":"2024-03-06T07:33:07.004161+0000","last_peered":"2024-03-06T07:33:07.004161+0000","last_clean":"2024-03-06T07:33:07.004161+0000","last_became_active":"2024-03-06T07:33:05.879428+0000","last_became_peered":"2024-03-06T07:33:05.879428+0000","last_unstale":"2024-03-06T07:33:07.004161+0000","last_undegraded":"2024-03-06T07:33:07.004161+0000","last_fullsized":"2024-03-06T07:33:07.004161+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:36.017689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003558+0000","last_change":"2024-03-06T07:33:05.857850+0000","last_active":"2024-03-06T07:33:07.003558+0000","last_peered":"2024-03-06T07:33:07.003558+0000","last_clean":"2024-03-06T07:33:07.003558+0000","last_became_active":"2024-03-06T07:33:05.857435+0000","last_became_peered":"2024-03-06T07:33:05.857435+0000","last_unstale":"2024-03-06T07:33:07.003558+0000","last_undegraded":"2024-03-06T07:33:07.003558+0000","last_fullsized":"2024-03-06T07:33:07.003558+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:34.219018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"92'4","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852198+0000","last_change":"2024-03-06T07:33:04.860428+0000","last_active":"2024-03-06T07:33:06.852198+0000","last_peered":"2024-03-06T07:33:06.852198+0000","last_clean":"2024-03-06T07:33:06.852198+0000","last_became_active":"2024-03-06T07:33:04.860099+0000","last_became_peered":"2024-03-06T07:33:04.860099+0000","last_unstale":"2024-03-06T07:33:06.852198+0000","last_undegraded":"2024-03-06T07:33:06.852198+0000","last_fullsized":"2024-03-06T07:33:06.852198+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:39.861022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.855180+0000","last_change":"2024-03-06T07:33:05.882275+0000","last_active":"2024-03-06T07:33:06.855180+0000","last_peered":"2024-03-06T07:33:06.855180+0000","last_clean":"2024-03-06T07:33:06.855180+0000","last_became_active":"2024-03-06T07:33:05.881829+0000","last_became_peered":"2024-03-06T07:33:05.881829+0000","last_unstale":"2024-03-06T07:33:06.855180+0000","last_undegraded":"2024-03-06T07:33:06.855180+0000","last_fullsized":"2024-03-06T07:33:06.855180+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:34.637751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007249+0000","last_change":"2024-03-06T07:33:04.859762+0000","last_active":"2024-03-06T07:33:07.007249+0000","last_peered":"2024-03-06T07:33:07.007249+0000","last_clean":"2024-03-06T07:33:07.007249+0000","last_became_active":"2024-03-06T07:33:04.859425+0000","last_became_peered":"2024-03-06T07:33:04.859425+0000","last_unstale":"2024-03-06T07:33:07.007249+0000","last_undegraded":"2024-03-06T07:33:07.007249+0000","last_fullsized":"2024-03-06T07:33:07.007249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:10.627589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004478+0000","last_change":"2024-03-06T07:33:05.893658+0000","last_active":"2024-03-06T07:33:07.004478+0000","last_peered":"2024-03-06T07:33:07.004478+0000","last_clean":"2024-03-06T07:33:07.004478+0000","last_became_active":"2024-03-06T07:33:05.893036+0000","last_became_peered":"2024-03-06T07:33:05.893036+0000","last_unstale":"2024-03-06T07:33:07.004478+0000","last_undegraded":"2024-03-06T07:33:07.004478+0000","last_fullsized":"2024-03-06T07:33:07.004478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:14.878812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853414+0000","last_change":"2024-03-06T07:33:04.852243+0000","last_active":"2024-03-06T07:33:06.853414+0000","last_peered":"2024-03-06T07:33:06.853414+0000","last_clean":"2024-03-06T07:33:06.853414+0000","last_became_active":"2024-03-06T07:33:04.851916+0000","last_became_peered":"2024-03-06T07:33:04.851916+0000","last_unstale":"2024-03-06T07:33:06.853414+0000","last_undegraded":"2024-03-06T07:33:06.853414+0000","last_fullsized":"2024-03-06T07:33:06.853414+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:46.245470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853159+0000","last_change":"2024-03-06T07:33:05.881439+0000","last_active":"2024-03-06T07:33:06.853159+0000","last_peered":"2024-03-06T07:33:06.853159+0000","last_clean":"2024-03-06T07:33:06.853159+0000","last_became_active":"2024-03-06T07:33:05.880961+0000","last_became_peered":"2024-03-06T07:33:05.880961+0000","last_unstale":"2024-03-06T07:33:06.853159+0000","last_undegraded":"2024-03-06T07:33:06.853159+0000","last_fullsized":"2024-03-06T07:33:06.853159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:36.945653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851785+0000","last_change":"2024-03-06T07:33:04.863645+0000","last_active":"2024-03-06T07:33:06.851785+0000","last_peered":"2024-03-06T07:33:06.851785+0000","last_clean":"2024-03-06T07:33:06.851785+0000","last_became_active":"2024-03-06T07:33:04.863353+0000","last_became_peered":"2024-03-06T07:33:04.863353+0000","last_unstale":"2024-03-06T07:33:06.851785+0000","last_undegraded":"2024-03-06T07:33:06.851785+0000","last_fullsized":"2024-03-06T07:33:06.851785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:29.115215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975235+0000","last_change":"2024-03-06T07:33:05.890060+0000","last_active":"2024-03-06T07:33:06.975235+0000","last_peered":"2024-03-06T07:33:06.975235+0000","last_clean":"2024-03-06T07:33:06.975235+0000","last_became_active":"2024-03-06T07:33:05.889719+0000","last_became_peered":"2024-03-06T07:33:05.889719+0000","last_unstale":"2024-03-06T07:33:06.975235+0000","last_undegraded":"2024-03-06T07:33:06.975235+0000","last_fullsized":"2024-03-06T07:33:06.975235+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:33.534315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004335+0000","last_change":"2024-03-06T07:33:04.856067+0000","last_active":"2024-03-06T07:33:07.004335+0000","last_peered":"2024-03-06T07:33:07.004335+0000","last_clean":"2024-03-06T07:33:07.004335+0000","last_became_active":"2024-03-06T07:33:04.855760+0000","last_became_peered":"2024-03-06T07:33:04.855760+0000","last_unstale":"2024-03-06T07:33:07.004335+0000","last_undegraded":"2024-03-06T07:33:07.004335+0000","last_fullsized":"2024-03-06T07:33:07.004335+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:38.085859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006935+0000","last_change":"2024-03-06T07:33:05.875298+0000","last_active":"2024-03-06T07:33:07.006935+0000","last_peered":"2024-03-06T07:33:07.006935+0000","last_clean":"2024-03-06T07:33:07.006935+0000","last_became_active":"2024-03-06T07:33:05.875000+0000","last_became_peered":"2024-03-06T07:33:05.875000+0000","last_unstale":"2024-03-06T07:33:07.006935+0000","last_undegraded":"2024-03-06T07:33:07.006935+0000","last_fullsized":"2024-03-06T07:33:07.006935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851716+0000","last_change":"2024-03-06T07:33:04.853605+0000","last_active":"2024-03-06T07:33:06.851716+0000","last_peered":"2024-03-06T07:33:06.851716+0000","last_clean":"2024-03-06T07:33:06.851716+0000","last_became_active":"2024-03-06T07:33:04.851072+0000","last_became_peered":"2024-03-06T07:33:04.851072+0000","last_unstale":"2024-03-06T07:33:06.851716+0000","last_undegraded":"2024-03-06T07:33:06.851716+0000","last_fullsized":"2024-03-06T07:33:06.851716+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:26.595055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852915+0000","last_change":"2024-03-06T07:33:05.883715+0000","last_active":"2024-03-06T07:33:06.852915+0000","last_peered":"2024-03-06T07:33:06.852915+0000","last_clean":"2024-03-06T07:33:06.852915+0000","last_became_active":"2024-03-06T07:33:05.883446+0000","last_became_peered":"2024-03-06T07:33:05.883446+0000","last_unstale":"2024-03-06T07:33:06.852915+0000","last_undegraded":"2024-03-06T07:33:06.852915+0000","last_fullsized":"2024-03-06T07:33:06.852915+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:00.556032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004628+0000","last_change":"2024-03-06T07:33:04.851991+0000","last_active":"2024-03-06T07:33:07.004628+0000","last_peered":"2024-03-06T07:33:07.004628+0000","last_clean":"2024-03-06T07:33:07.004628+0000","last_became_active":"2024-03-06T07:33:04.851633+0000","last_became_peered":"2024-03-06T07:33:04.851633+0000","last_unstale":"2024-03-06T07:33:07.004628+0000","last_undegraded":"2024-03-06T07:33:07.004628+0000","last_fullsized":"2024-03-06T07:33:07.004628+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:26.595055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852088+0000","last_change":"2024-03-06T07:33:05.883768+0000","last_active":"2024-03-06T07:33:06.852088+0000","last_peered":"2024-03-06T07:33:06.852088+0000","last_clean":"2024-03-06T07:33:06.852088+0000","last_became_active":"2024-03-06T07:33:05.883277+0000","last_became_peered":"2024-03-06T07:33:05.883277+0000","last_unstale":"2024-03-06T07:33:06.852088+0000","last_undegraded":"2024-03-06T07:33:06.852088+0000","last_fullsized":"2024-03-06T07:33:06.852088+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:01.440534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004197+0000","last_change":"2024-03-06T07:33:04.851214+0000","last_active":"2024-03-06T07:33:07.004197+0000","last_peered":"2024-03-06T07:33:07.004197+0000","last_clean":"2024-03-06T07:33:07.004197+0000","last_became_active":"2024-03-06T07:33:04.850896+0000","last_became_peered":"2024-03-06T07:33:04.850896+0000","last_unstale":"2024-03-06T07:33:07.004197+0000","last_undegraded":"2024-03-06T07:33:07.004197+0000","last_fullsized":"2024-03-06T07:33:07.004197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:33.204795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852546+0000","last_change":"2024-03-06T07:33:05.888484+0000","last_active":"2024-03-06T07:33:06.852546+0000","last_peered":"2024-03-06T07:33:06.852546+0000","last_clean":"2024-03-06T07:33:06.852546+0000","last_became_active":"2024-03-06T07:33:05.888152+0000","last_became_peered":"2024-03-06T07:33:05.888152+0000","last_unstale":"2024-03-06T07:33:06.852546+0000","last_undegraded":"2024-03-06T07:33:06.852546+0000","last_fullsized":"2024-03-06T07:33:06.852546+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:48.938608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004697+0000","last_change":"2024-03-06T07:33:04.859650+0000","last_active":"2024-03-06T07:33:07.004697+0000","last_peered":"2024-03-06T07:33:07.004697+0000","last_clean":"2024-03-06T07:33:07.004697+0000","last_became_active":"2024-03-06T07:33:04.859352+0000","last_became_peered":"2024-03-06T07:33:04.859352+0000","last_unstale":"2024-03-06T07:33:07.004697+0000","last_undegraded":"2024-03-06T07:33:07.004697+0000","last_fullsized":"2024-03-06T07:33:07.004697+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:33.037154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853305+0000","last_change":"2024-03-06T07:33:05.885234+0000","last_active":"2024-03-06T07:33:06.853305+0000","last_peered":"2024-03-06T07:33:06.853305+0000","last_clean":"2024-03-06T07:33:06.853305+0000","last_became_active":"2024-03-06T07:33:05.884633+0000","last_became_peered":"2024-03-06T07:33:05.884633+0000","last_unstale":"2024-03-06T07:33:06.853305+0000","last_undegraded":"2024-03-06T07:33:06.853305+0000","last_fullsized":"2024-03-06T07:33:06.853305+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:58.814629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.855253+0000","last_change":"2024-03-06T07:33:04.851647+0000","last_active":"2024-03-06T07:33:06.855253+0000","last_peered":"2024-03-06T07:33:06.855253+0000","last_clean":"2024-03-06T07:33:06.855253+0000","last_became_active":"2024-03-06T07:33:04.851380+0000","last_became_peered":"2024-03-06T07:33:04.851380+0000","last_unstale":"2024-03-06T07:33:06.855253+0000","last_undegraded":"2024-03-06T07:33:06.855253+0000","last_fullsized":"2024-03-06T07:33:06.855253+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:00.426311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004797+0000","last_change":"2024-03-06T07:33:05.871919+0000","last_active":"2024-03-06T07:33:07.004797+0000","last_peered":"2024-03-06T07:33:07.004797+0000","last_clean":"2024-03-06T07:33:07.004797+0000","last_became_active":"2024-03-06T07:33:05.871575+0000","last_became_peered":"2024-03-06T07:33:05.871575+0000","last_unstale":"2024-03-06T07:33:07.004797+0000","last_undegraded":"2024-03-06T07:33:07.004797+0000","last_fullsized":"2024-03-06T07:33:07.004797+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:00.556032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"92'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007512+0000","last_change":"2024-03-06T07:33:04.862688+0000","last_active":"2024-03-06T07:33:07.007512+0000","last_peered":"2024-03-06T07:33:07.007512+0000","last_clean":"2024-03-06T07:33:07.007512+0000","last_became_active":"2024-03-06T07:33:04.862306+0000","last_became_peered":"2024-03-06T07:33:04.862306+0000","last_unstale":"2024-03-06T07:33:07.007512+0000","last_undegraded":"2024-03-06T07:33:07.007512+0000","last_fullsized":"2024-03-06T07:33:07.007512+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:00.426311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007361+0000","last_change":"2024-03-06T07:33:05.869758+0000","last_active":"2024-03-06T07:33:07.007361+0000","last_peered":"2024-03-06T07:33:07.007361+0000","last_clean":"2024-03-06T07:33:07.007361+0000","last_became_active":"2024-03-06T07:33:05.868342+0000","last_became_peered":"2024-03-06T07:33:05.868342+0000","last_unstale":"2024-03-06T07:33:07.007361+0000","last_undegraded":"2024-03-06T07:33:07.007361+0000","last_fullsized":"2024-03-06T07:33:07.007361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:48.795019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"92'6","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853226+0000","last_change":"2024-03-06T07:33:04.862275+0000","last_active":"2024-03-06T07:33:06.853226+0000","last_peered":"2024-03-06T07:33:06.853226+0000","last_clean":"2024-03-06T07:33:06.853226+0000","last_became_active":"2024-03-06T07:33:04.861781+0000","last_became_peered":"2024-03-06T07:33:04.861781+0000","last_unstale":"2024-03-06T07:33:06.853226+0000","last_undegraded":"2024-03-06T07:33:06.853226+0000","last_fullsized":"2024-03-06T07:33:06.853226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:20.069348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853275+0000","last_change":"2024-03-06T07:33:05.866768+0000","last_active":"2024-03-06T07:33:06.853275+0000","last_peered":"2024-03-06T07:33:06.853275+0000","last_clean":"2024-03-06T07:33:06.853275+0000","last_became_active":"2024-03-06T07:33:05.866099+0000","last_became_peered":"2024-03-06T07:33:05.866099+0000","last_unstale":"2024-03-06T07:33:06.853275+0000","last_undegraded":"2024-03-06T07:33:06.853275+0000","last_fullsized":"2024-03-06T07:33:06.853275+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.450793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002967+0000","last_change":"2024-03-06T07:33:04.859375+0000","last_active":"2024-03-06T07:33:07.002967+0000","last_peered":"2024-03-06T07:33:07.002967+0000","last_clean":"2024-03-06T07:33:07.002967+0000","last_became_active":"2024-03-06T07:33:04.859083+0000","last_became_peered":"2024-03-06T07:33:04.859083+0000","last_unstale":"2024-03-06T07:33:07.002967+0000","last_undegraded":"2024-03-06T07:33:07.002967+0000","last_fullsized":"2024-03-06T07:33:07.002967+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:00.842012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004329+0000","last_change":"2024-03-06T07:33:05.880128+0000","last_active":"2024-03-06T07:33:07.004329+0000","last_peered":"2024-03-06T07:33:07.004329+0000","last_clean":"2024-03-06T07:33:07.004329+0000","last_became_active":"2024-03-06T07:33:05.879871+0000","last_became_peered":"2024-03-06T07:33:05.879871+0000","last_unstale":"2024-03-06T07:33:07.004329+0000","last_undegraded":"2024-03-06T07:33:07.004329+0000","last_fullsized":"2024-03-06T07:33:07.004329+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:14.389146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003270+0000","last_change":"2024-03-06T07:33:04.850271+0000","last_active":"2024-03-06T07:33:07.003270+0000","last_peered":"2024-03-06T07:33:07.003270+0000","last_clean":"2024-03-06T07:33:07.003270+0000","last_became_active":"2024-03-06T07:33:04.848130+0000","last_became_peered":"2024-03-06T07:33:04.848130+0000","last_unstale":"2024-03-06T07:33:07.003270+0000","last_undegraded":"2024-03-06T07:33:07.003270+0000","last_fullsized":"2024-03-06T07:33:07.003270+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:19.996285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852172+0000","last_change":"2024-03-06T07:33:05.884246+0000","last_active":"2024-03-06T07:33:06.852172+0000","last_peered":"2024-03-06T07:33:06.852172+0000","last_clean":"2024-03-06T07:33:06.852172+0000","last_became_active":"2024-03-06T07:33:05.883933+0000","last_became_peered":"2024-03-06T07:33:05.883933+0000","last_unstale":"2024-03-06T07:33:06.852172+0000","last_undegraded":"2024-03-06T07:33:06.852172+0000","last_fullsized":"2024-03-06T07:33:06.852172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:40.875245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852016+0000","last_change":"2024-03-06T07:33:04.859813+0000","last_active":"2024-03-06T07:33:06.852016+0000","last_peered":"2024-03-06T07:33:06.852016+0000","last_clean":"2024-03-06T07:33:06.852016+0000","last_became_active":"2024-03-06T07:33:04.859455+0000","last_became_peered":"2024-03-06T07:33:04.859455+0000","last_unstale":"2024-03-06T07:33:06.852016+0000","last_undegraded":"2024-03-06T07:33:06.852016+0000","last_fullsized":"2024-03-06T07:33:06.852016+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:58.219355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002700+0000","last_change":"2024-03-06T07:33:05.886186+0000","last_active":"2024-03-06T07:33:07.002700+0000","last_peered":"2024-03-06T07:33:07.002700+0000","last_clean":"2024-03-06T07:33:07.002700+0000","last_became_active":"2024-03-06T07:33:05.885480+0000","last_became_peered":"2024-03-06T07:33:05.885480+0000","last_unstale":"2024-03-06T07:33:07.002700+0000","last_undegraded":"2024-03-06T07:33:07.002700+0000","last_fullsized":"2024-03-06T07:33:07.002700+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:37.724052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"92'8","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004275+0000","last_change":"2024-03-06T07:33:04.859159+0000","last_active":"2024-03-06T07:33:07.004275+0000","last_peered":"2024-03-06T07:33:07.004275+0000","last_clean":"2024-03-06T07:33:07.004275+0000","last_became_active":"2024-03-06T07:33:04.857677+0000","last_became_peered":"2024-03-06T07:33:04.857677+0000","last_unstale":"2024-03-06T07:33:07.004275+0000","last_undegraded":"2024-03-06T07:33:07.004275+0000","last_fullsized":"2024-03-06T07:33:07.004275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:13.367147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005405+0000","last_change":"2024-03-06T07:33:05.891822+0000","last_active":"2024-03-06T07:33:07.005405+0000","last_peered":"2024-03-06T07:33:07.005405+0000","last_clean":"2024-03-06T07:33:07.005405+0000","last_became_active":"2024-03-06T07:33:05.891531+0000","last_became_peered":"2024-03-06T07:33:05.891531+0000","last_unstale":"2024-03-06T07:33:07.005405+0000","last_undegraded":"2024-03-06T07:33:07.005405+0000","last_fullsized":"2024-03-06T07:33:07.005405+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852776+0000","last_change":"2024-03-06T07:33:04.858922+0000","last_active":"2024-03-06T07:33:06.852776+0000","last_peered":"2024-03-06T07:33:06.852776+0000","last_clean":"2024-03-06T07:33:06.852776+0000","last_became_active":"2024-03-06T07:33:04.857859+0000","last_became_peered":"2024-03-06T07:33:04.857859+0000","last_unstale":"2024-03-06T07:33:06.852776+0000","last_undegraded":"2024-03-06T07:33:06.852776+0000","last_fullsized":"2024-03-06T07:33:06.852776+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:25.885524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975345+0000","last_change":"2024-03-06T07:33:05.875050+0000","last_active":"2024-03-06T07:33:06.975345+0000","last_peered":"2024-03-06T07:33:06.975345+0000","last_clean":"2024-03-06T07:33:06.975345+0000","last_became_active":"2024-03-06T07:33:05.874727+0000","last_became_peered":"2024-03-06T07:33:05.874727+0000","last_unstale":"2024-03-06T07:33:06.975345+0000","last_undegraded":"2024-03-06T07:33:06.975345+0000","last_fullsized":"2024-03-06T07:33:06.975345+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:02.689703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005322+0000","last_change":"2024-03-06T07:33:04.860247+0000","last_active":"2024-03-06T07:33:07.005322+0000","last_peered":"2024-03-06T07:33:07.005322+0000","last_clean":"2024-03-06T07:33:07.005322+0000","last_became_active":"2024-03-06T07:33:04.859606+0000","last_became_peered":"2024-03-06T07:33:04.859606+0000","last_unstale":"2024-03-06T07:33:07.005322+0000","last_undegraded":"2024-03-06T07:33:07.005322+0000","last_fullsized":"2024-03-06T07:33:07.005322+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:34.793545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007433+0000","last_change":"2024-03-06T07:33:05.885681+0000","last_active":"2024-03-06T07:33:07.007433+0000","last_peered":"2024-03-06T07:33:07.007433+0000","last_clean":"2024-03-06T07:33:07.007433+0000","last_became_active":"2024-03-06T07:33:05.885398+0000","last_became_peered":"2024-03-06T07:33:05.885398+0000","last_unstale":"2024-03-06T07:33:07.007433+0000","last_undegraded":"2024-03-06T07:33:07.007433+0000","last_fullsized":"2024-03-06T07:33:07.007433+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:29.053384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852050+0000","last_change":"2024-03-06T07:33:04.864317+0000","last_active":"2024-03-06T07:33:06.852050+0000","last_peered":"2024-03-06T07:33:06.852050+0000","last_clean":"2024-03-06T07:33:06.852050+0000","last_became_active":"2024-03-06T07:33:04.864003+0000","last_became_peered":"2024-03-06T07:33:04.864003+0000","last_unstale":"2024-03-06T07:33:06.852050+0000","last_undegraded":"2024-03-06T07:33:06.852050+0000","last_fullsized":"2024-03-06T07:33:06.852050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:10.627589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003224+0000","last_change":"2024-03-06T07:33:05.859977+0000","last_active":"2024-03-06T07:33:07.003224+0000","last_peered":"2024-03-06T07:33:07.003224+0000","last_clean":"2024-03-06T07:33:07.003224+0000","last_became_active":"2024-03-06T07:33:05.859661+0000","last_became_peered":"2024-03-06T07:33:05.859661+0000","last_unstale":"2024-03-06T07:33:07.003224+0000","last_undegraded":"2024-03-06T07:33:07.003224+0000","last_fullsized":"2024-03-06T07:33:07.003224+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:39.100082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853202+0000","last_change":"2024-03-06T07:33:04.854837+0000","last_active":"2024-03-06T07:33:06.853202+0000","last_peered":"2024-03-06T07:33:06.853202+0000","last_clean":"2024-03-06T07:33:06.853202+0000","last_became_active":"2024-03-06T07:33:04.854303+0000","last_became_peered":"2024-03-06T07:33:04.854303+0000","last_unstale":"2024-03-06T07:33:06.853202+0000","last_undegraded":"2024-03-06T07:33:06.853202+0000","last_fullsized":"2024-03-06T07:33:06.853202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:00.426311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002856+0000","last_change":"2024-03-06T07:33:05.868458+0000","last_active":"2024-03-06T07:33:07.002856+0000","last_peered":"2024-03-06T07:33:07.002856+0000","last_clean":"2024-03-06T07:33:07.002856+0000","last_became_active":"2024-03-06T07:33:05.866577+0000","last_became_peered":"2024-03-06T07:33:05.866577+0000","last_unstale":"2024-03-06T07:33:07.002856+0000","last_undegraded":"2024-03-06T07:33:07.002856+0000","last_fullsized":"2024-03-06T07:33:07.002856+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:36.126403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975191+0000","last_change":"2024-03-06T07:33:04.861361+0000","last_active":"2024-03-06T07:33:06.975191+0000","last_peered":"2024-03-06T07:33:06.975191+0000","last_clean":"2024-03-06T07:33:06.975191+0000","last_became_active":"2024-03-06T07:33:04.861055+0000","last_became_peered":"2024-03-06T07:33:04.861055+0000","last_unstale":"2024-03-06T07:33:06.975191+0000","last_undegraded":"2024-03-06T07:33:06.975191+0000","last_fullsized":"2024-03-06T07:33:06.975191+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:00.247995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002484+0000","last_change":"2024-03-06T07:33:05.859278+0000","last_active":"2024-03-06T07:33:07.002484+0000","last_peered":"2024-03-06T07:33:07.002484+0000","last_clean":"2024-03-06T07:33:07.002484+0000","last_became_active":"2024-03-06T07:33:05.858929+0000","last_became_peered":"2024-03-06T07:33:05.858929+0000","last_unstale":"2024-03-06T07:33:07.002484+0000","last_undegraded":"2024-03-06T07:33:07.002484+0000","last_fullsized":"2024-03-06T07:33:07.002484+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:38.800668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975116+0000","last_change":"2024-03-06T07:33:04.856628+0000","last_active":"2024-03-06T07:33:06.975116+0000","last_peered":"2024-03-06T07:33:06.975116+0000","last_clean":"2024-03-06T07:33:06.975116+0000","last_became_active":"2024-03-06T07:33:04.855981+0000","last_became_peered":"2024-03-06T07:33:04.855981+0000","last_unstale":"2024-03-06T07:33:06.975116+0000","last_undegraded":"2024-03-06T07:33:06.975116+0000","last_fullsized":"2024-03-06T07:33:06.975116+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:44.242932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004274+0000","last_change":"2024-03-06T07:33:05.884260+0000","last_active":"2024-03-06T07:33:07.004274+0000","last_peered":"2024-03-06T07:33:07.004274+0000","last_clean":"2024-03-06T07:33:07.004274+0000","last_became_active":"2024-03-06T07:33:05.883984+0000","last_became_peered":"2024-03-06T07:33:05.883984+0000","last_unstale":"2024-03-06T07:33:07.004274+0000","last_undegraded":"2024-03-06T07:33:07.004274+0000","last_fullsized":"2024-03-06T07:33:07.004274+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:58.814629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851018+0000","last_change":"2024-03-06T07:33:04.861157+0000","last_active":"2024-03-06T07:33:06.851018+0000","last_peered":"2024-03-06T07:33:06.851018+0000","last_clean":"2024-03-06T07:33:06.851018+0000","last_became_active":"2024-03-06T07:33:04.860313+0000","last_became_peered":"2024-03-06T07:33:04.860313+0000","last_unstale":"2024-03-06T07:33:06.851018+0000","last_undegraded":"2024-03-06T07:33:06.851018+0000","last_fullsized":"2024-03-06T07:33:06.851018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:25.287017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852922+0000","last_change":"2024-03-06T07:33:05.874379+0000","last_active":"2024-03-06T07:33:06.852922+0000","last_peered":"2024-03-06T07:33:06.852922+0000","last_clean":"2024-03-06T07:33:06.852922+0000","last_became_active":"2024-03-06T07:33:05.874052+0000","last_became_peered":"2024-03-06T07:33:05.874052+0000","last_unstale":"2024-03-06T07:33:06.852922+0000","last_undegraded":"2024-03-06T07:33:06.852922+0000","last_fullsized":"2024-03-06T07:33:06.852922+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:23.164088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975828+0000","last_change":"2024-03-06T07:33:04.864490+0000","last_active":"2024-03-06T07:33:06.975828+0000","last_peered":"2024-03-06T07:33:06.975828+0000","last_clean":"2024-03-06T07:33:06.975828+0000","last_became_active":"2024-03-06T07:33:04.863728+0000","last_became_peered":"2024-03-06T07:33:04.863728+0000","last_unstale":"2024-03-06T07:33:06.975828+0000","last_undegraded":"2024-03-06T07:33:06.975828+0000","last_fullsized":"2024-03-06T07:33:06.975828+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:16.375896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004729+0000","last_change":"2024-03-06T07:33:05.875908+0000","last_active":"2024-03-06T07:33:07.004729+0000","last_peered":"2024-03-06T07:33:07.004729+0000","last_clean":"2024-03-06T07:33:07.004729+0000","last_became_active":"2024-03-06T07:33:05.875264+0000","last_became_peered":"2024-03-06T07:33:05.875264+0000","last_unstale":"2024-03-06T07:33:07.004729+0000","last_undegraded":"2024-03-06T07:33:07.004729+0000","last_fullsized":"2024-03-06T07:33:07.004729+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.262218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"93'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.859274+0000","last_change":"2024-03-06T07:33:04.855353+0000","last_active":"2024-03-06T07:33:07.859274+0000","last_peered":"2024-03-06T07:33:07.859274+0000","last_clean":"2024-03-06T07:33:07.859274+0000","last_became_active":"2024-03-06T07:33:04.854865+0000","last_became_peered":"2024-03-06T07:33:04.854865+0000","last_unstale":"2024-03-06T07:33:07.859274+0000","last_undegraded":"2024-03-06T07:33:07.859274+0000","last_fullsized":"2024-03-06T07:33:07.859274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:23.097346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004568+0000","last_change":"2024-03-06T07:33:05.884234+0000","last_active":"2024-03-06T07:33:07.004568+0000","last_peered":"2024-03-06T07:33:07.004568+0000","last_clean":"2024-03-06T07:33:07.004568+0000","last_became_active":"2024-03-06T07:33:05.883983+0000","last_became_peered":"2024-03-06T07:33:05.883983+0000","last_unstale":"2024-03-06T07:33:07.004568+0000","last_undegraded":"2024-03-06T07:33:07.004568+0000","last_fullsized":"2024-03-06T07:33:07.004568+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:17.730172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853604+0000","last_change":"2024-03-06T07:33:04.862187+0000","last_active":"2024-03-06T07:33:06.853604+0000","last_peered":"2024-03-06T07:33:06.853604+0000","last_clean":"2024-03-06T07:33:06.853604+0000","last_became_active":"2024-03-06T07:33:04.861839+0000","last_became_peered":"2024-03-06T07:33:04.861839+0000","last_unstale":"2024-03-06T07:33:06.853604+0000","last_undegraded":"2024-03-06T07:33:06.853604+0000","last_fullsized":"2024-03-06T07:33:06.853604+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:34.793545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004791+0000","last_change":"2024-03-06T07:33:05.880230+0000","last_active":"2024-03-06T07:33:07.004791+0000","last_peered":"2024-03-06T07:33:07.004791+0000","last_clean":"2024-03-06T07:33:07.004791+0000","last_became_active":"2024-03-06T07:33:05.879898+0000","last_became_peered":"2024-03-06T07:33:05.879898+0000","last_unstale":"2024-03-06T07:33:07.004791+0000","last_undegraded":"2024-03-06T07:33:07.004791+0000","last_fullsized":"2024-03-06T07:33:07.004791+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:46.594997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005090+0000","last_change":"2024-03-06T07:33:04.856983+0000","last_active":"2024-03-06T07:33:07.005090+0000","last_peered":"2024-03-06T07:33:07.005090+0000","last_clean":"2024-03-06T07:33:07.005090+0000","last_became_active":"2024-03-06T07:33:04.856695+0000","last_became_peered":"2024-03-06T07:33:04.856695+0000","last_unstale":"2024-03-06T07:33:07.005090+0000","last_undegraded":"2024-03-06T07:33:07.005090+0000","last_fullsized":"2024-03-06T07:33:07.005090+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:29.115215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003011+0000","last_change":"2024-03-06T07:33:05.873762+0000","last_active":"2024-03-06T07:33:07.003011+0000","last_peered":"2024-03-06T07:33:07.003011+0000","last_clean":"2024-03-06T07:33:07.003011+0000","last_became_active":"2024-03-06T07:33:05.873467+0000","last_became_peered":"2024-03-06T07:33:05.873467+0000","last_unstale":"2024-03-06T07:33:07.003011+0000","last_undegraded":"2024-03-06T07:33:07.003011+0000","last_fullsized":"2024-03-06T07:33:07.003011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:04.300387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853423+0000","last_change":"2024-03-06T07:33:04.853282+0000","last_active":"2024-03-06T07:33:06.853423+0000","last_peered":"2024-03-06T07:33:06.853423+0000","last_clean":"2024-03-06T07:33:06.853423+0000","last_became_active":"2024-03-06T07:33:04.852975+0000","last_became_peered":"2024-03-06T07:33:04.852975+0000","last_unstale":"2024-03-06T07:33:06.853423+0000","last_undegraded":"2024-03-06T07:33:06.853423+0000","last_fullsized":"2024-03-06T07:33:06.853423+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:04.731706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852320+0000","last_change":"2024-03-06T07:33:05.883667+0000","last_active":"2024-03-06T07:33:06.852320+0000","last_peered":"2024-03-06T07:33:06.852320+0000","last_clean":"2024-03-06T07:33:06.852320+0000","last_became_active":"2024-03-06T07:33:05.883327+0000","last_became_peered":"2024-03-06T07:33:05.883327+0000","last_unstale":"2024-03-06T07:33:06.852320+0000","last_undegraded":"2024-03-06T07:33:06.852320+0000","last_fullsized":"2024-03-06T07:33:06.852320+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:05.745929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004186+0000","last_change":"2024-03-06T07:33:04.850498+0000","last_active":"2024-03-06T07:33:07.004186+0000","last_peered":"2024-03-06T07:33:07.004186+0000","last_clean":"2024-03-06T07:33:07.004186+0000","last_became_active":"2024-03-06T07:33:04.849848+0000","last_became_peered":"2024-03-06T07:33:04.849848+0000","last_unstale":"2024-03-06T07:33:07.004186+0000","last_undegraded":"2024-03-06T07:33:07.004186+0000","last_fullsized":"2024-03-06T07:33:07.004186+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:32.196543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002797+0000","last_change":"2024-03-06T07:33:05.885936+0000","last_active":"2024-03-06T07:33:07.002797+0000","last_peered":"2024-03-06T07:33:07.002797+0000","last_clean":"2024-03-06T07:33:07.002797+0000","last_became_active":"2024-03-06T07:33:05.885337+0000","last_became_peered":"2024-03-06T07:33:05.885337+0000","last_unstale":"2024-03-06T07:33:07.002797+0000","last_undegraded":"2024-03-06T07:33:07.002797+0000","last_fullsized":"2024-03-06T07:33:07.002797+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:04.512996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853815+0000","last_change":"2024-03-06T07:33:04.846351+0000","last_active":"2024-03-06T07:33:06.853815+0000","last_peered":"2024-03-06T07:33:06.853815+0000","last_clean":"2024-03-06T07:33:06.853815+0000","last_became_active":"2024-03-06T07:33:04.845704+0000","last_became_peered":"2024-03-06T07:33:04.845704+0000","last_unstale":"2024-03-06T07:33:06.853815+0000","last_undegraded":"2024-03-06T07:33:06.853815+0000","last_fullsized":"2024-03-06T07:33:06.853815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:46.245470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004275+0000","last_change":"2024-03-06T07:33:05.882402+0000","last_active":"2024-03-06T07:33:07.004275+0000","last_peered":"2024-03-06T07:33:07.004275+0000","last_clean":"2024-03-06T07:33:07.004275+0000","last_became_active":"2024-03-06T07:33:05.882047+0000","last_became_peered":"2024-03-06T07:33:05.882047+0000","last_unstale":"2024-03-06T07:33:07.004275+0000","last_undegraded":"2024-03-06T07:33:07.004275+0000","last_fullsized":"2024-03-06T07:33:07.004275+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.006500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002796+0000","last_change":"2024-03-06T07:33:04.852903+0000","last_active":"2024-03-06T07:33:07.002796+0000","last_peered":"2024-03-06T07:33:07.002796+0000","last_clean":"2024-03-06T07:33:07.002796+0000","last_became_active":"2024-03-06T07:33:04.851062+0000","last_became_peered":"2024-03-06T07:33:04.851062+0000","last_unstale":"2024-03-06T07:33:07.002796+0000","last_undegraded":"2024-03-06T07:33:07.002796+0000","last_fullsized":"2024-03-06T07:33:07.002796+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:05.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003828+0000","last_change":"2024-03-06T07:33:05.855680+0000","last_active":"2024-03-06T07:33:07.003828+0000","last_peered":"2024-03-06T07:33:07.003828+0000","last_clean":"2024-03-06T07:33:07.003828+0000","last_became_active":"2024-03-06T07:33:05.855371+0000","last_became_peered":"2024-03-06T07:33:05.855371+0000","last_unstale":"2024-03-06T07:33:07.003828+0000","last_undegraded":"2024-03-06T07:33:07.003828+0000","last_fullsized":"2024-03-06T07:33:07.003828+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:07.414947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007110+0000","last_change":"2024-03-06T07:33:04.861144+0000","last_active":"2024-03-06T07:33:07.007110+0000","last_peered":"2024-03-06T07:33:07.007110+0000","last_clean":"2024-03-06T07:33:07.007110+0000","last_became_active":"2024-03-06T07:33:04.860862+0000","last_became_peered":"2024-03-06T07:33:04.860862+0000","last_unstale":"2024-03-06T07:33:07.007110+0000","last_undegraded":"2024-03-06T07:33:07.007110+0000","last_fullsized":"2024-03-06T07:33:07.007110+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:46.245470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":303,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006395+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:33:07.006395+0000","last_peered":"2024-03-06T07:33:07.006395+0000","last_clean":"2024-03-06T07:33:07.006395+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:33:07.006395+0000","last_undegraded":"2024-03-06T07:33:07.006395+0000","last_fullsized":"2024-03-06T07:33:07.006395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1753088,"data_stored":1082703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1159168,"data_stored":492875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574915,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":492293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":493959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":493857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1155072,"data_stored":492257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1150976,"data_stored":492223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1748992,"data_stored":1083173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:10.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.506+0000 7fd274c00640 1 Processor -- start 2024-03-06T07:33:10.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.506+0000 7fd274c00640 1 -- start start 2024-03-06T07:33:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.507+0000 7fd274c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd270144c00 0x7fd27006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.507+0000 7fd274c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd27015f9d0 0x7fd27015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.507+0000 7fd274c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27006ee70 con 0x7fd2700ce0a0 2024-03-06T07:33:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.507+0000 7fd274c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd270160370 con 0x7fd27015f9d0 2024-03-06T07:33:10.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.507+0000 7fd274c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27006cef0 con 0x7fd270144c00 2024-03-06T07:33:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.508+0000 7fd26d400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd27015f9d0 0x7fd27015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.508+0000 7fd26de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd270144c00 0x7fd27006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.508+0000 7fd26d400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd27015f9d0 0x7fd27015fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:41688/0 (socket says 172.21.15.114:41688) 2024-03-06T07:33:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.508+0000 7fd26d400640 1 -- 172.21.15.114:0/1416554781 learned_addr learned my addr 172.21.15.114:0/1416554781 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.509+0000 7fd26d400640 1 -- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd270144c00 msgr2=0x7fd27006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:10.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.509+0000 7fd26d400640 1 --2- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd270144c00 0x7fd27006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.509+0000 7fd26d400640 1 -- 172.21.15.114:0/1416554781 >> v1:172.21.15.177:6789/0 conn(0x7fd2700ce0a0 legacy=0x7fd2700ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.509+0000 7fd26d400640 1 -- 172.21.15.114:0/1416554781 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27006d1d0 con 0x7fd27015f9d0 2024-03-06T07:33:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.509+0000 7fd26de00640 1 --2- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd270144c00 0x7fd27006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.511+0000 7fd26d400640 1 --2- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 0x7fd27015fe30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009870 tx=0x7fd25c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba6d83c16e16e6da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.511+0000 7fd26ca00640 1 -- 172.21.15.114:0/1416554781 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c034070 con 0x7fd27015f9d0 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.511+0000 7fd26ca00640 1 -- 172.21.15.114:0/1416554781 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c030840 con 0x7fd27015f9d0 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.511+0000 7fd26ca00640 1 -- 172.21.15.114:0/1416554781 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c030c10 con 0x7fd27015f9d0 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 -- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 msgr2=0x7fd27015fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 --2- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 0x7fd27015fe30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009870 tx=0x7fd25c02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 -- 172.21.15.114:0/1416554781 shutdown_connections 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 --2- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 0x7fd27015fe30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 --2- 172.21.15.114:0/1416554781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd270144c00 0x7fd27006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 -- 172.21.15.114:0/1416554781 >> 172.21.15.114:0/1416554781 conn(0x7fd2700b1730 msgr2=0x7fd2700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 -- 172.21.15.114:0/1416554781 shutdown_connections 2024-03-06T07:33:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.512+0000 7fd274c00640 1 -- 172.21.15.114:0/1416554781 wait complete. 2024-03-06T07:33:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.514+0000 7fd274c00640 1 Processor -- start 2024-03-06T07:33:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.514+0000 7fd274c00640 1 -- start start 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd274c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2700ce0a0 0x7fd27015b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd274c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd270144c00 0x7fd27013fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd274c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 0x7fd27015e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd274c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd270160730 con 0x7fd270144c00 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd274c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27006d790 con 0x7fd27015f9d0 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd274c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27006cb50 con 0x7fd2700ce0a0 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd270144c00 0x7fd27013fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd270144c00 0x7fd27013fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60824/0 (socket says 172.21.15.114:60824) 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 -- 172.21.15.114:0/270465310 learned_addr learned my addr 172.21.15.114:0/270465310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26de00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2700ce0a0 0x7fd27015b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26e800640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 0x7fd27015e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 -- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2700ce0a0 msgr2=0x7fd27015b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2700ce0a0 0x7fd27015b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 -- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 msgr2=0x7fd27015e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 0x7fd27015e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.515+0000 7fd26d400640 1 -- 172.21.15.114:0/270465310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27012eb50 con 0x7fd270144c00 2024-03-06T07:33:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.516+0000 7fd26d400640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd270144c00 0x7fd27013fd80 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd25c02b9c0 tx=0x7fd25c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.516+0000 7fd266000640 1 -- 172.21.15.114:0/270465310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c034040 con 0x7fd270144c00 2024-03-06T07:33:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.516+0000 7fd266000640 1 -- 172.21.15.114:0/270465310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c030640 con 0x7fd270144c00 2024-03-06T07:33:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.516+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27012ee30 con 0x7fd270144c00 2024-03-06T07:33:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.516+0000 7fd266000640 1 -- 172.21.15.114:0/270465310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c0423f0 con 0x7fd270144c00 2024-03-06T07:33:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.516+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27012f3f0 con 0x7fd270144c00 2024-03-06T07:33:10.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.518+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27004f330 con 0x7fd270144c00 2024-03-06T07:33:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.518+0000 7fd266000640 1 -- 172.21.15.114:0/270465310 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd25c037070 con 0x7fd270144c00 2024-03-06T07:33:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.520+0000 7fd266000640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd25006b4d0 0x7fd25006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.520+0000 7fd266000640 1 -- 172.21.15.114:0/270465310 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd25c0ac470 con 0x7fd270144c00 2024-03-06T07:33:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.521+0000 7fd26de00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd25006b4d0 0x7fd25006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:10.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.521+0000 7fd26de00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd25006b4d0 0x7fd25006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd2701f70e0 tx=0x7fd260008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:10.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.524+0000 7fd266000640 1 -- 172.21.15.114:0/270465310 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd25c07b370 con 0x7fd270144c00 2024-03-06T07:33:10.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.714+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2700b1d60 con 0x7fd25006b4d0 2024-03-06T07:33:10.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.732+0000 7fd266000640 1 -- 172.21.15.114:0/270465310 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252085 (secure 0 0 0) 0x7fd2700b1d60 con 0x7fd25006b4d0 2024-03-06T07:33:10.738 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd25006b4d0 msgr2=0x7fd25006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd25006b4d0 0x7fd25006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd2701f70e0 tx=0x7fd260008040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd270144c00 msgr2=0x7fd27013fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd270144c00 0x7fd27013fd80 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd25c02b9c0 tx=0x7fd25c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 shutdown_connections 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd27015f9d0 0x7fd27015e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd25006b4d0 0x7fd25006d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd270144c00 0x7fd27013fd80 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 --2- 172.21.15.114:0/270465310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd2700ce0a0 0x7fd27015b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 >> 172.21.15.114:0/270465310 conn(0x7fd2700b1730 msgr2=0x7fd2700ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.738+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 shutdown_connections 2024-03-06T07:33:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:10.739+0000 7fd274c00640 1 -- 172.21.15.114:0/270465310 wait complete. 2024-03-06T07:33:10.761 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-06T07:33:09.194890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259696,"kb_used_data":10216,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715024,"statfs":{"total":773094113280,"available":772828184576,"internally_reserved":0,"allocated":10461184,"data_stored":5123340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.373559"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004621+0000","last_change":"2024-03-06T07:33:05.877782+0000","last_active":"2024-03-06T07:33:07.004621+0000","last_peered":"2024-03-06T07:33:07.004621+0000","last_clean":"2024-03-06T07:33:07.004621+0000","last_became_active":"2024-03-06T07:33:05.875599+0000","last_became_peered":"2024-03-06T07:33:05.875599+0000","last_unstale":"2024-03-06T07:33:07.004621+0000","last_undegraded":"2024-03-06T07:33:07.004621+0000","last_fullsized":"2024-03-06T07:33:07.004621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:06.559368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851780+0000","last_change":"2024-03-06T07:33:05.879937+0000","last_active":"2024-03-06T07:33:06.851780+0000","last_peered":"2024-03-06T07:33:06.851780+0000","last_clean":"2024-03-06T07:33:06.851780+0000","last_became_active":"2024-03-06T07:33:05.879676+0000","last_became_peered":"2024-03-06T07:33:05.879676+0000","last_unstale":"2024-03-06T07:33:06.851780+0000","last_undegraded":"2024-03-06T07:33:06.851780+0000","last_fullsized":"2024-03-06T07:33:06.851780+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:26.856826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004478+0000","last_change":"2024-03-06T07:33:05.881262+0000","last_active":"2024-03-06T07:33:07.004478+0000","last_peered":"2024-03-06T07:33:07.004478+0000","last_clean":"2024-03-06T07:33:07.004478+0000","last_became_active":"2024-03-06T07:33:05.881000+0000","last_became_peered":"2024-03-06T07:33:05.881000+0000","last_unstale":"2024-03-06T07:33:07.004478+0000","last_undegraded":"2024-03-06T07:33:07.004478+0000","last_fullsized":"2024-03-06T07:33:07.004478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:37.331334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853865+0000","last_change":"2024-03-06T07:33:05.859166+0000","last_active":"2024-03-06T07:33:06.853865+0000","last_peered":"2024-03-06T07:33:06.853865+0000","last_clean":"2024-03-06T07:33:06.853865+0000","last_became_active":"2024-03-06T07:33:05.858881+0000","last_became_peered":"2024-03-06T07:33:05.858881+0000","last_unstale":"2024-03-06T07:33:06.853865+0000","last_undegraded":"2024-03-06T07:33:06.853865+0000","last_fullsized":"2024-03-06T07:33:06.853865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005554+0000","last_change":"2024-03-06T07:33:05.881488+0000","last_active":"2024-03-06T07:33:07.005554+0000","last_peered":"2024-03-06T07:33:07.005554+0000","last_clean":"2024-03-06T07:33:07.005554+0000","last_became_active":"2024-03-06T07:33:05.881104+0000","last_became_peered":"2024-03-06T07:33:05.881104+0000","last_unstale":"2024-03-06T07:33:07.005554+0000","last_undegraded":"2024-03-06T07:33:07.005554+0000","last_fullsized":"2024-03-06T07:33:07.005554+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006399+0000","last_change":"2024-03-06T07:33:05.874065+0000","last_active":"2024-03-06T07:33:07.006399+0000","last_peered":"2024-03-06T07:33:07.006399+0000","last_clean":"2024-03-06T07:33:07.006399+0000","last_became_active":"2024-03-06T07:33:05.873755+0000","last_became_peered":"2024-03-06T07:33:05.873755+0000","last_unstale":"2024-03-06T07:33:07.006399+0000","last_undegraded":"2024-03-06T07:33:07.006399+0000","last_fullsized":"2024-03-06T07:33:07.006399+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006345+0000","last_change":"2024-03-06T07:33:05.874759+0000","last_active":"2024-03-06T07:33:07.006345+0000","last_peered":"2024-03-06T07:33:07.006345+0000","last_clean":"2024-03-06T07:33:07.006345+0000","last_became_active":"2024-03-06T07:33:05.874462+0000","last_became_peered":"2024-03-06T07:33:05.874462+0000","last_unstale":"2024-03-06T07:33:07.006345+0000","last_undegraded":"2024-03-06T07:33:07.006345+0000","last_fullsized":"2024-03-06T07:33:07.006345+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:13.128225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.850870+0000","last_change":"2024-03-06T07:33:05.876477+0000","last_active":"2024-03-06T07:33:06.850870+0000","last_peered":"2024-03-06T07:33:06.850870+0000","last_clean":"2024-03-06T07:33:06.850870+0000","last_became_active":"2024-03-06T07:33:05.870488+0000","last_became_peered":"2024-03-06T07:33:05.870488+0000","last_unstale":"2024-03-06T07:33:06.850870+0000","last_undegraded":"2024-03-06T07:33:06.850870+0000","last_fullsized":"2024-03-06T07:33:06.850870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853135+0000","last_change":"2024-03-06T07:33:05.883628+0000","last_active":"2024-03-06T07:33:06.853135+0000","last_peered":"2024-03-06T07:33:06.853135+0000","last_clean":"2024-03-06T07:33:06.853135+0000","last_became_active":"2024-03-06T07:33:05.883240+0000","last_became_peered":"2024-03-06T07:33:05.883240+0000","last_unstale":"2024-03-06T07:33:06.853135+0000","last_undegraded":"2024-03-06T07:33:06.853135+0000","last_fullsized":"2024-03-06T07:33:06.853135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:26.301240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005033+0000","last_change":"2024-03-06T07:33:05.889679+0000","last_active":"2024-03-06T07:33:07.005033+0000","last_peered":"2024-03-06T07:33:07.005033+0000","last_clean":"2024-03-06T07:33:07.005033+0000","last_became_active":"2024-03-06T07:33:05.888726+0000","last_became_peered":"2024-03-06T07:33:05.888726+0000","last_unstale":"2024-03-06T07:33:07.005033+0000","last_undegraded":"2024-03-06T07:33:07.005033+0000","last_fullsized":"2024-03-06T07:33:07.005033+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.450793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007051+0000","last_change":"2024-03-06T07:33:05.886460+0000","last_active":"2024-03-06T07:33:07.007051+0000","last_peered":"2024-03-06T07:33:07.007051+0000","last_clean":"2024-03-06T07:33:07.007051+0000","last_became_active":"2024-03-06T07:33:05.884698+0000","last_became_peered":"2024-03-06T07:33:05.884698+0000","last_unstale":"2024-03-06T07:33:07.007051+0000","last_undegraded":"2024-03-06T07:33:07.007051+0000","last_fullsized":"2024-03-06T07:33:07.007051+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:34.637751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852903+0000","last_change":"2024-03-06T07:33:05.891586+0000","last_active":"2024-03-06T07:33:06.852903+0000","last_peered":"2024-03-06T07:33:06.852903+0000","last_clean":"2024-03-06T07:33:06.852903+0000","last_became_active":"2024-03-06T07:33:05.891280+0000","last_became_peered":"2024-03-06T07:33:05.891280+0000","last_unstale":"2024-03-06T07:33:06.852903+0000","last_undegraded":"2024-03-06T07:33:06.852903+0000","last_fullsized":"2024-03-06T07:33:06.852903+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:24.111569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853452+0000","last_change":"2024-03-06T07:33:05.891923+0000","last_active":"2024-03-06T07:33:06.853452+0000","last_peered":"2024-03-06T07:33:06.853452+0000","last_clean":"2024-03-06T07:33:06.853452+0000","last_became_active":"2024-03-06T07:33:05.891635+0000","last_became_peered":"2024-03-06T07:33:05.891635+0000","last_unstale":"2024-03-06T07:33:06.853452+0000","last_undegraded":"2024-03-06T07:33:06.853452+0000","last_fullsized":"2024-03-06T07:33:06.853452+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:27.034207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002355+0000","last_change":"2024-03-06T07:33:05.876219+0000","last_active":"2024-03-06T07:33:07.002355+0000","last_peered":"2024-03-06T07:33:07.002355+0000","last_clean":"2024-03-06T07:33:07.002355+0000","last_became_active":"2024-03-06T07:33:05.875879+0000","last_became_peered":"2024-03-06T07:33:05.875879+0000","last_unstale":"2024-03-06T07:33:07.002355+0000","last_undegraded":"2024-03-06T07:33:07.002355+0000","last_fullsized":"2024-03-06T07:33:07.002355+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:50:18.844653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853784+0000","last_change":"2024-03-06T07:33:05.879553+0000","last_active":"2024-03-06T07:33:06.853784+0000","last_peered":"2024-03-06T07:33:06.853784+0000","last_clean":"2024-03-06T07:33:06.853784+0000","last_became_active":"2024-03-06T07:33:05.879289+0000","last_became_peered":"2024-03-06T07:33:05.879289+0000","last_unstale":"2024-03-06T07:33:06.853784+0000","last_undegraded":"2024-03-06T07:33:06.853784+0000","last_fullsized":"2024-03-06T07:33:06.853784+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006250+0000","last_change":"2024-03-06T07:33:05.869374+0000","last_active":"2024-03-06T07:33:07.006250+0000","last_peered":"2024-03-06T07:33:07.006250+0000","last_clean":"2024-03-06T07:33:07.006250+0000","last_became_active":"2024-03-06T07:33:05.868221+0000","last_became_peered":"2024-03-06T07:33:05.868221+0000","last_unstale":"2024-03-06T07:33:07.006250+0000","last_undegraded":"2024-03-06T07:33:07.006250+0000","last_fullsized":"2024-03-06T07:33:07.006250+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.450793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851945+0000","last_change":"2024-03-06T07:33:05.875997+0000","last_active":"2024-03-06T07:33:06.851945+0000","last_peered":"2024-03-06T07:33:06.851945+0000","last_clean":"2024-03-06T07:33:06.851945+0000","last_became_active":"2024-03-06T07:33:05.873483+0000","last_became_peered":"2024-03-06T07:33:05.873483+0000","last_unstale":"2024-03-06T07:33:06.851945+0000","last_undegraded":"2024-03-06T07:33:06.851945+0000","last_fullsized":"2024-03-06T07:33:06.851945+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:48.795019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.855234+0000","last_change":"2024-03-06T07:33:05.858934+0000","last_active":"2024-03-06T07:33:06.855234+0000","last_peered":"2024-03-06T07:33:06.855234+0000","last_clean":"2024-03-06T07:33:06.855234+0000","last_became_active":"2024-03-06T07:33:05.858626+0000","last_became_peered":"2024-03-06T07:33:05.858626+0000","last_unstale":"2024-03-06T07:33:06.855234+0000","last_undegraded":"2024-03-06T07:33:06.855234+0000","last_fullsized":"2024-03-06T07:33:06.855234+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:26.856826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003636+0000","last_change":"2024-03-06T07:33:05.889630+0000","last_active":"2024-03-06T07:33:07.003636+0000","last_peered":"2024-03-06T07:33:07.003636+0000","last_clean":"2024-03-06T07:33:07.003636+0000","last_became_active":"2024-03-06T07:33:05.889067+0000","last_became_peered":"2024-03-06T07:33:05.889067+0000","last_unstale":"2024-03-06T07:33:07.003636+0000","last_undegraded":"2024-03-06T07:33:07.003636+0000","last_fullsized":"2024-03-06T07:33:07.003636+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:14.381370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853196+0000","last_change":"2024-03-06T07:33:05.881805+0000","last_active":"2024-03-06T07:33:06.853196+0000","last_peered":"2024-03-06T07:33:06.853196+0000","last_clean":"2024-03-06T07:33:06.853196+0000","last_became_active":"2024-03-06T07:33:05.881527+0000","last_became_peered":"2024-03-06T07:33:05.881527+0000","last_unstale":"2024-03-06T07:33:06.853196+0000","last_undegraded":"2024-03-06T07:33:06.853196+0000","last_fullsized":"2024-03-06T07:33:06.853196+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:33.682790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002692+0000","last_change":"2024-03-06T07:33:05.869384+0000","last_active":"2024-03-06T07:33:07.002692+0000","last_peered":"2024-03-06T07:33:07.002692+0000","last_clean":"2024-03-06T07:33:07.002692+0000","last_became_active":"2024-03-06T07:33:05.869031+0000","last_became_peered":"2024-03-06T07:33:05.869031+0000","last_unstale":"2024-03-06T07:33:07.002692+0000","last_undegraded":"2024-03-06T07:33:07.002692+0000","last_fullsized":"2024-03-06T07:33:07.002692+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:23.211814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851388+0000","last_change":"2024-03-06T07:33:05.874060+0000","last_active":"2024-03-06T07:33:06.851388+0000","last_peered":"2024-03-06T07:33:06.851388+0000","last_clean":"2024-03-06T07:33:06.851388+0000","last_became_active":"2024-03-06T07:33:05.873742+0000","last_became_peered":"2024-03-06T07:33:05.873742+0000","last_unstale":"2024-03-06T07:33:06.851388+0000","last_undegraded":"2024-03-06T07:33:06.851388+0000","last_fullsized":"2024-03-06T07:33:06.851388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853974+0000","last_change":"2024-03-06T07:33:05.874148+0000","last_active":"2024-03-06T07:33:06.853974+0000","last_peered":"2024-03-06T07:33:06.853974+0000","last_clean":"2024-03-06T07:33:06.853974+0000","last_became_active":"2024-03-06T07:33:05.873840+0000","last_became_peered":"2024-03-06T07:33:05.873840+0000","last_unstale":"2024-03-06T07:33:06.853974+0000","last_undegraded":"2024-03-06T07:33:06.853974+0000","last_fullsized":"2024-03-06T07:33:06.853974+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:34.637751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006263+0000","last_change":"2024-03-06T07:33:05.888716+0000","last_active":"2024-03-06T07:33:07.006263+0000","last_peered":"2024-03-06T07:33:07.006263+0000","last_clean":"2024-03-06T07:33:07.006263+0000","last_became_active":"2024-03-06T07:33:05.888433+0000","last_became_peered":"2024-03-06T07:33:05.888433+0000","last_unstale":"2024-03-06T07:33:07.006263+0000","last_undegraded":"2024-03-06T07:33:07.006263+0000","last_fullsized":"2024-03-06T07:33:07.006263+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:27.034207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005094+0000","last_change":"2024-03-06T07:33:05.891127+0000","last_active":"2024-03-06T07:33:07.005094+0000","last_peered":"2024-03-06T07:33:07.005094+0000","last_clean":"2024-03-06T07:33:07.005094+0000","last_became_active":"2024-03-06T07:33:05.890664+0000","last_became_peered":"2024-03-06T07:33:05.890664+0000","last_unstale":"2024-03-06T07:33:07.005094+0000","last_undegraded":"2024-03-06T07:33:07.005094+0000","last_fullsized":"2024-03-06T07:33:07.005094+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:27.034207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852169+0000","last_change":"2024-03-06T07:33:05.882431+0000","last_active":"2024-03-06T07:33:06.852169+0000","last_peered":"2024-03-06T07:33:06.852169+0000","last_clean":"2024-03-06T07:33:06.852169+0000","last_became_active":"2024-03-06T07:33:05.882173+0000","last_became_peered":"2024-03-06T07:33:05.882173+0000","last_unstale":"2024-03-06T07:33:06.852169+0000","last_undegraded":"2024-03-06T07:33:06.852169+0000","last_fullsized":"2024-03-06T07:33:06.852169+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:29.053384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007079+0000","last_change":"2024-03-06T07:33:05.858790+0000","last_active":"2024-03-06T07:33:07.007079+0000","last_peered":"2024-03-06T07:33:07.007079+0000","last_clean":"2024-03-06T07:33:07.007079+0000","last_became_active":"2024-03-06T07:33:05.858045+0000","last_became_peered":"2024-03-06T07:33:05.858045+0000","last_unstale":"2024-03-06T07:33:07.007079+0000","last_undegraded":"2024-03-06T07:33:07.007079+0000","last_fullsized":"2024-03-06T07:33:07.007079+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:26.856826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005284+0000","last_change":"2024-03-06T07:33:05.881239+0000","last_active":"2024-03-06T07:33:07.005284+0000","last_peered":"2024-03-06T07:33:07.005284+0000","last_clean":"2024-03-06T07:33:07.005284+0000","last_became_active":"2024-03-06T07:33:05.880967+0000","last_became_peered":"2024-03-06T07:33:05.880967+0000","last_unstale":"2024-03-06T07:33:07.005284+0000","last_undegraded":"2024-03-06T07:33:07.005284+0000","last_fullsized":"2024-03-06T07:33:07.005284+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:13.128225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852919+0000","last_change":"2024-03-06T07:33:05.886968+0000","last_active":"2024-03-06T07:33:06.852919+0000","last_peered":"2024-03-06T07:33:06.852919+0000","last_clean":"2024-03-06T07:33:06.852919+0000","last_became_active":"2024-03-06T07:33:05.886681+0000","last_became_peered":"2024-03-06T07:33:05.886681+0000","last_unstale":"2024-03-06T07:33:06.852919+0000","last_undegraded":"2024-03-06T07:33:06.852919+0000","last_fullsized":"2024-03-06T07:33:06.852919+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:12:05.060646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002713+0000","last_change":"2024-03-06T07:33:05.881474+0000","last_active":"2024-03-06T07:33:07.002713+0000","last_peered":"2024-03-06T07:33:07.002713+0000","last_clean":"2024-03-06T07:33:07.002713+0000","last_became_active":"2024-03-06T07:33:05.881165+0000","last_became_peered":"2024-03-06T07:33:05.881165+0000","last_unstale":"2024-03-06T07:33:07.002713+0000","last_undegraded":"2024-03-06T07:33:07.002713+0000","last_fullsized":"2024-03-06T07:33:07.002713+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:58.183397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853596+0000","last_change":"2024-03-06T07:33:05.872579+0000","last_active":"2024-03-06T07:33:06.853596+0000","last_peered":"2024-03-06T07:33:06.853596+0000","last_clean":"2024-03-06T07:33:06.853596+0000","last_became_active":"2024-03-06T07:33:05.872310+0000","last_became_peered":"2024-03-06T07:33:05.872310+0000","last_unstale":"2024-03-06T07:33:06.853596+0000","last_undegraded":"2024-03-06T07:33:06.853596+0000","last_fullsized":"2024-03-06T07:33:06.853596+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:13.128225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004161+0000","last_change":"2024-03-06T07:33:05.879741+0000","last_active":"2024-03-06T07:33:07.004161+0000","last_peered":"2024-03-06T07:33:07.004161+0000","last_clean":"2024-03-06T07:33:07.004161+0000","last_became_active":"2024-03-06T07:33:05.879428+0000","last_became_peered":"2024-03-06T07:33:05.879428+0000","last_unstale":"2024-03-06T07:33:07.004161+0000","last_undegraded":"2024-03-06T07:33:07.004161+0000","last_fullsized":"2024-03-06T07:33:07.004161+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:36.017689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003558+0000","last_change":"2024-03-06T07:33:05.857850+0000","last_active":"2024-03-06T07:33:07.003558+0000","last_peered":"2024-03-06T07:33:07.003558+0000","last_clean":"2024-03-06T07:33:07.003558+0000","last_became_active":"2024-03-06T07:33:05.857435+0000","last_became_peered":"2024-03-06T07:33:05.857435+0000","last_unstale":"2024-03-06T07:33:07.003558+0000","last_undegraded":"2024-03-06T07:33:07.003558+0000","last_fullsized":"2024-03-06T07:33:07.003558+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:34.219018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"92'4","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852198+0000","last_change":"2024-03-06T07:33:04.860428+0000","last_active":"2024-03-06T07:33:06.852198+0000","last_peered":"2024-03-06T07:33:06.852198+0000","last_clean":"2024-03-06T07:33:06.852198+0000","last_became_active":"2024-03-06T07:33:04.860099+0000","last_became_peered":"2024-03-06T07:33:04.860099+0000","last_unstale":"2024-03-06T07:33:06.852198+0000","last_undegraded":"2024-03-06T07:33:06.852198+0000","last_fullsized":"2024-03-06T07:33:06.852198+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:39.861022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.855180+0000","last_change":"2024-03-06T07:33:05.882275+0000","last_active":"2024-03-06T07:33:06.855180+0000","last_peered":"2024-03-06T07:33:06.855180+0000","last_clean":"2024-03-06T07:33:06.855180+0000","last_became_active":"2024-03-06T07:33:05.881829+0000","last_became_peered":"2024-03-06T07:33:05.881829+0000","last_unstale":"2024-03-06T07:33:06.855180+0000","last_undegraded":"2024-03-06T07:33:06.855180+0000","last_fullsized":"2024-03-06T07:33:06.855180+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:34.637751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007249+0000","last_change":"2024-03-06T07:33:04.859762+0000","last_active":"2024-03-06T07:33:07.007249+0000","last_peered":"2024-03-06T07:33:07.007249+0000","last_clean":"2024-03-06T07:33:07.007249+0000","last_became_active":"2024-03-06T07:33:04.859425+0000","last_became_peered":"2024-03-06T07:33:04.859425+0000","last_unstale":"2024-03-06T07:33:07.007249+0000","last_undegraded":"2024-03-06T07:33:07.007249+0000","last_fullsized":"2024-03-06T07:33:07.007249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:10.627589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004478+0000","last_change":"2024-03-06T07:33:05.893658+0000","last_active":"2024-03-06T07:33:07.004478+0000","last_peered":"2024-03-06T07:33:07.004478+0000","last_clean":"2024-03-06T07:33:07.004478+0000","last_became_active":"2024-03-06T07:33:05.893036+0000","last_became_peered":"2024-03-06T07:33:05.893036+0000","last_unstale":"2024-03-06T07:33:07.004478+0000","last_undegraded":"2024-03-06T07:33:07.004478+0000","last_fullsized":"2024-03-06T07:33:07.004478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:14.878812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853414+0000","last_change":"2024-03-06T07:33:04.852243+0000","last_active":"2024-03-06T07:33:06.853414+0000","last_peered":"2024-03-06T07:33:06.853414+0000","last_clean":"2024-03-06T07:33:06.853414+0000","last_became_active":"2024-03-06T07:33:04.851916+0000","last_became_peered":"2024-03-06T07:33:04.851916+0000","last_unstale":"2024-03-06T07:33:06.853414+0000","last_undegraded":"2024-03-06T07:33:06.853414+0000","last_fullsized":"2024-03-06T07:33:06.853414+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:46.245470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853159+0000","last_change":"2024-03-06T07:33:05.881439+0000","last_active":"2024-03-06T07:33:06.853159+0000","last_peered":"2024-03-06T07:33:06.853159+0000","last_clean":"2024-03-06T07:33:06.853159+0000","last_became_active":"2024-03-06T07:33:05.880961+0000","last_became_peered":"2024-03-06T07:33:05.880961+0000","last_unstale":"2024-03-06T07:33:06.853159+0000","last_undegraded":"2024-03-06T07:33:06.853159+0000","last_fullsized":"2024-03-06T07:33:06.853159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:36.945653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851785+0000","last_change":"2024-03-06T07:33:04.863645+0000","last_active":"2024-03-06T07:33:06.851785+0000","last_peered":"2024-03-06T07:33:06.851785+0000","last_clean":"2024-03-06T07:33:06.851785+0000","last_became_active":"2024-03-06T07:33:04.863353+0000","last_became_peered":"2024-03-06T07:33:04.863353+0000","last_unstale":"2024-03-06T07:33:06.851785+0000","last_undegraded":"2024-03-06T07:33:06.851785+0000","last_fullsized":"2024-03-06T07:33:06.851785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:29.115215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975235+0000","last_change":"2024-03-06T07:33:05.890060+0000","last_active":"2024-03-06T07:33:06.975235+0000","last_peered":"2024-03-06T07:33:06.975235+0000","last_clean":"2024-03-06T07:33:06.975235+0000","last_became_active":"2024-03-06T07:33:05.889719+0000","last_became_peered":"2024-03-06T07:33:05.889719+0000","last_unstale":"2024-03-06T07:33:06.975235+0000","last_undegraded":"2024-03-06T07:33:06.975235+0000","last_fullsized":"2024-03-06T07:33:06.975235+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:33.534315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004335+0000","last_change":"2024-03-06T07:33:04.856067+0000","last_active":"2024-03-06T07:33:07.004335+0000","last_peered":"2024-03-06T07:33:07.004335+0000","last_clean":"2024-03-06T07:33:07.004335+0000","last_became_active":"2024-03-06T07:33:04.855760+0000","last_became_peered":"2024-03-06T07:33:04.855760+0000","last_unstale":"2024-03-06T07:33:07.004335+0000","last_undegraded":"2024-03-06T07:33:07.004335+0000","last_fullsized":"2024-03-06T07:33:07.004335+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:38.085859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006935+0000","last_change":"2024-03-06T07:33:05.875298+0000","last_active":"2024-03-06T07:33:07.006935+0000","last_peered":"2024-03-06T07:33:07.006935+0000","last_clean":"2024-03-06T07:33:07.006935+0000","last_became_active":"2024-03-06T07:33:05.875000+0000","last_became_peered":"2024-03-06T07:33:05.875000+0000","last_unstale":"2024-03-06T07:33:07.006935+0000","last_undegraded":"2024-03-06T07:33:07.006935+0000","last_fullsized":"2024-03-06T07:33:07.006935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:50.923261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851716+0000","last_change":"2024-03-06T07:33:04.853605+0000","last_active":"2024-03-06T07:33:06.851716+0000","last_peered":"2024-03-06T07:33:06.851716+0000","last_clean":"2024-03-06T07:33:06.851716+0000","last_became_active":"2024-03-06T07:33:04.851072+0000","last_became_peered":"2024-03-06T07:33:04.851072+0000","last_unstale":"2024-03-06T07:33:06.851716+0000","last_undegraded":"2024-03-06T07:33:06.851716+0000","last_fullsized":"2024-03-06T07:33:06.851716+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:26.595055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852915+0000","last_change":"2024-03-06T07:33:05.883715+0000","last_active":"2024-03-06T07:33:06.852915+0000","last_peered":"2024-03-06T07:33:06.852915+0000","last_clean":"2024-03-06T07:33:06.852915+0000","last_became_active":"2024-03-06T07:33:05.883446+0000","last_became_peered":"2024-03-06T07:33:05.883446+0000","last_unstale":"2024-03-06T07:33:06.852915+0000","last_undegraded":"2024-03-06T07:33:06.852915+0000","last_fullsized":"2024-03-06T07:33:06.852915+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:00.556032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004628+0000","last_change":"2024-03-06T07:33:04.851991+0000","last_active":"2024-03-06T07:33:07.004628+0000","last_peered":"2024-03-06T07:33:07.004628+0000","last_clean":"2024-03-06T07:33:07.004628+0000","last_became_active":"2024-03-06T07:33:04.851633+0000","last_became_peered":"2024-03-06T07:33:04.851633+0000","last_unstale":"2024-03-06T07:33:07.004628+0000","last_undegraded":"2024-03-06T07:33:07.004628+0000","last_fullsized":"2024-03-06T07:33:07.004628+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:26.595055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852088+0000","last_change":"2024-03-06T07:33:05.883768+0000","last_active":"2024-03-06T07:33:06.852088+0000","last_peered":"2024-03-06T07:33:06.852088+0000","last_clean":"2024-03-06T07:33:06.852088+0000","last_became_active":"2024-03-06T07:33:05.883277+0000","last_became_peered":"2024-03-06T07:33:05.883277+0000","last_unstale":"2024-03-06T07:33:06.852088+0000","last_undegraded":"2024-03-06T07:33:06.852088+0000","last_fullsized":"2024-03-06T07:33:06.852088+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:01.440534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004197+0000","last_change":"2024-03-06T07:33:04.851214+0000","last_active":"2024-03-06T07:33:07.004197+0000","last_peered":"2024-03-06T07:33:07.004197+0000","last_clean":"2024-03-06T07:33:07.004197+0000","last_became_active":"2024-03-06T07:33:04.850896+0000","last_became_peered":"2024-03-06T07:33:04.850896+0000","last_unstale":"2024-03-06T07:33:07.004197+0000","last_undegraded":"2024-03-06T07:33:07.004197+0000","last_fullsized":"2024-03-06T07:33:07.004197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:33.204795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852546+0000","last_change":"2024-03-06T07:33:05.888484+0000","last_active":"2024-03-06T07:33:06.852546+0000","last_peered":"2024-03-06T07:33:06.852546+0000","last_clean":"2024-03-06T07:33:06.852546+0000","last_became_active":"2024-03-06T07:33:05.888152+0000","last_became_peered":"2024-03-06T07:33:05.888152+0000","last_unstale":"2024-03-06T07:33:06.852546+0000","last_undegraded":"2024-03-06T07:33:06.852546+0000","last_fullsized":"2024-03-06T07:33:06.852546+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:48.938608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004697+0000","last_change":"2024-03-06T07:33:04.859650+0000","last_active":"2024-03-06T07:33:07.004697+0000","last_peered":"2024-03-06T07:33:07.004697+0000","last_clean":"2024-03-06T07:33:07.004697+0000","last_became_active":"2024-03-06T07:33:04.859352+0000","last_became_peered":"2024-03-06T07:33:04.859352+0000","last_unstale":"2024-03-06T07:33:07.004697+0000","last_undegraded":"2024-03-06T07:33:07.004697+0000","last_fullsized":"2024-03-06T07:33:07.004697+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:33.037154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853305+0000","last_change":"2024-03-06T07:33:05.885234+0000","last_active":"2024-03-06T07:33:06.853305+0000","last_peered":"2024-03-06T07:33:06.853305+0000","last_clean":"2024-03-06T07:33:06.853305+0000","last_became_active":"2024-03-06T07:33:05.884633+0000","last_became_peered":"2024-03-06T07:33:05.884633+0000","last_unstale":"2024-03-06T07:33:06.853305+0000","last_undegraded":"2024-03-06T07:33:06.853305+0000","last_fullsized":"2024-03-06T07:33:06.853305+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:58.814629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.855253+0000","last_change":"2024-03-06T07:33:04.851647+0000","last_active":"2024-03-06T07:33:06.855253+0000","last_peered":"2024-03-06T07:33:06.855253+0000","last_clean":"2024-03-06T07:33:06.855253+0000","last_became_active":"2024-03-06T07:33:04.851380+0000","last_became_peered":"2024-03-06T07:33:04.851380+0000","last_unstale":"2024-03-06T07:33:06.855253+0000","last_undegraded":"2024-03-06T07:33:06.855253+0000","last_fullsized":"2024-03-06T07:33:06.855253+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:00.426311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004797+0000","last_change":"2024-03-06T07:33:05.871919+0000","last_active":"2024-03-06T07:33:07.004797+0000","last_peered":"2024-03-06T07:33:07.004797+0000","last_clean":"2024-03-06T07:33:07.004797+0000","last_became_active":"2024-03-06T07:33:05.871575+0000","last_became_peered":"2024-03-06T07:33:05.871575+0000","last_unstale":"2024-03-06T07:33:07.004797+0000","last_undegraded":"2024-03-06T07:33:07.004797+0000","last_fullsized":"2024-03-06T07:33:07.004797+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:00.556032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"92'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007512+0000","last_change":"2024-03-06T07:33:04.862688+0000","last_active":"2024-03-06T07:33:07.007512+0000","last_peered":"2024-03-06T07:33:07.007512+0000","last_clean":"2024-03-06T07:33:07.007512+0000","last_became_active":"2024-03-06T07:33:04.862306+0000","last_became_peered":"2024-03-06T07:33:04.862306+0000","last_unstale":"2024-03-06T07:33:07.007512+0000","last_undegraded":"2024-03-06T07:33:07.007512+0000","last_fullsized":"2024-03-06T07:33:07.007512+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:00.426311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007361+0000","last_change":"2024-03-06T07:33:05.869758+0000","last_active":"2024-03-06T07:33:07.007361+0000","last_peered":"2024-03-06T07:33:07.007361+0000","last_clean":"2024-03-06T07:33:07.007361+0000","last_became_active":"2024-03-06T07:33:05.868342+0000","last_became_peered":"2024-03-06T07:33:05.868342+0000","last_unstale":"2024-03-06T07:33:07.007361+0000","last_undegraded":"2024-03-06T07:33:07.007361+0000","last_fullsized":"2024-03-06T07:33:07.007361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:48.795019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"92'6","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853226+0000","last_change":"2024-03-06T07:33:04.862275+0000","last_active":"2024-03-06T07:33:06.853226+0000","last_peered":"2024-03-06T07:33:06.853226+0000","last_clean":"2024-03-06T07:33:06.853226+0000","last_became_active":"2024-03-06T07:33:04.861781+0000","last_became_peered":"2024-03-06T07:33:04.861781+0000","last_unstale":"2024-03-06T07:33:06.853226+0000","last_undegraded":"2024-03-06T07:33:06.853226+0000","last_fullsized":"2024-03-06T07:33:06.853226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:20.069348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853275+0000","last_change":"2024-03-06T07:33:05.866768+0000","last_active":"2024-03-06T07:33:06.853275+0000","last_peered":"2024-03-06T07:33:06.853275+0000","last_clean":"2024-03-06T07:33:06.853275+0000","last_became_active":"2024-03-06T07:33:05.866099+0000","last_became_peered":"2024-03-06T07:33:05.866099+0000","last_unstale":"2024-03-06T07:33:06.853275+0000","last_undegraded":"2024-03-06T07:33:06.853275+0000","last_fullsized":"2024-03-06T07:33:06.853275+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:43.450793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002967+0000","last_change":"2024-03-06T07:33:04.859375+0000","last_active":"2024-03-06T07:33:07.002967+0000","last_peered":"2024-03-06T07:33:07.002967+0000","last_clean":"2024-03-06T07:33:07.002967+0000","last_became_active":"2024-03-06T07:33:04.859083+0000","last_became_peered":"2024-03-06T07:33:04.859083+0000","last_unstale":"2024-03-06T07:33:07.002967+0000","last_undegraded":"2024-03-06T07:33:07.002967+0000","last_fullsized":"2024-03-06T07:33:07.002967+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:00.842012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004329+0000","last_change":"2024-03-06T07:33:05.880128+0000","last_active":"2024-03-06T07:33:07.004329+0000","last_peered":"2024-03-06T07:33:07.004329+0000","last_clean":"2024-03-06T07:33:07.004329+0000","last_became_active":"2024-03-06T07:33:05.879871+0000","last_became_peered":"2024-03-06T07:33:05.879871+0000","last_unstale":"2024-03-06T07:33:07.004329+0000","last_undegraded":"2024-03-06T07:33:07.004329+0000","last_fullsized":"2024-03-06T07:33:07.004329+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:14.389146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003270+0000","last_change":"2024-03-06T07:33:04.850271+0000","last_active":"2024-03-06T07:33:07.003270+0000","last_peered":"2024-03-06T07:33:07.003270+0000","last_clean":"2024-03-06T07:33:07.003270+0000","last_became_active":"2024-03-06T07:33:04.848130+0000","last_became_peered":"2024-03-06T07:33:04.848130+0000","last_unstale":"2024-03-06T07:33:07.003270+0000","last_undegraded":"2024-03-06T07:33:07.003270+0000","last_fullsized":"2024-03-06T07:33:07.003270+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:19.996285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852172+0000","last_change":"2024-03-06T07:33:05.884246+0000","last_active":"2024-03-06T07:33:06.852172+0000","last_peered":"2024-03-06T07:33:06.852172+0000","last_clean":"2024-03-06T07:33:06.852172+0000","last_became_active":"2024-03-06T07:33:05.883933+0000","last_became_peered":"2024-03-06T07:33:05.883933+0000","last_unstale":"2024-03-06T07:33:06.852172+0000","last_undegraded":"2024-03-06T07:33:06.852172+0000","last_fullsized":"2024-03-06T07:33:06.852172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:40.875245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852016+0000","last_change":"2024-03-06T07:33:04.859813+0000","last_active":"2024-03-06T07:33:06.852016+0000","last_peered":"2024-03-06T07:33:06.852016+0000","last_clean":"2024-03-06T07:33:06.852016+0000","last_became_active":"2024-03-06T07:33:04.859455+0000","last_became_peered":"2024-03-06T07:33:04.859455+0000","last_unstale":"2024-03-06T07:33:06.852016+0000","last_undegraded":"2024-03-06T07:33:06.852016+0000","last_fullsized":"2024-03-06T07:33:06.852016+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:58.219355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002700+0000","last_change":"2024-03-06T07:33:05.886186+0000","last_active":"2024-03-06T07:33:07.002700+0000","last_peered":"2024-03-06T07:33:07.002700+0000","last_clean":"2024-03-06T07:33:07.002700+0000","last_became_active":"2024-03-06T07:33:05.885480+0000","last_became_peered":"2024-03-06T07:33:05.885480+0000","last_unstale":"2024-03-06T07:33:07.002700+0000","last_undegraded":"2024-03-06T07:33:07.002700+0000","last_fullsized":"2024-03-06T07:33:07.002700+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:37.724052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"92'8","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004275+0000","last_change":"2024-03-06T07:33:04.859159+0000","last_active":"2024-03-06T07:33:07.004275+0000","last_peered":"2024-03-06T07:33:07.004275+0000","last_clean":"2024-03-06T07:33:07.004275+0000","last_became_active":"2024-03-06T07:33:04.857677+0000","last_became_peered":"2024-03-06T07:33:04.857677+0000","last_unstale":"2024-03-06T07:33:07.004275+0000","last_undegraded":"2024-03-06T07:33:07.004275+0000","last_fullsized":"2024-03-06T07:33:07.004275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:13.367147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005405+0000","last_change":"2024-03-06T07:33:05.891822+0000","last_active":"2024-03-06T07:33:07.005405+0000","last_peered":"2024-03-06T07:33:07.005405+0000","last_clean":"2024-03-06T07:33:07.005405+0000","last_became_active":"2024-03-06T07:33:05.891531+0000","last_became_peered":"2024-03-06T07:33:05.891531+0000","last_unstale":"2024-03-06T07:33:07.005405+0000","last_undegraded":"2024-03-06T07:33:07.005405+0000","last_fullsized":"2024-03-06T07:33:07.005405+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:58.140435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852776+0000","last_change":"2024-03-06T07:33:04.858922+0000","last_active":"2024-03-06T07:33:06.852776+0000","last_peered":"2024-03-06T07:33:06.852776+0000","last_clean":"2024-03-06T07:33:06.852776+0000","last_became_active":"2024-03-06T07:33:04.857859+0000","last_became_peered":"2024-03-06T07:33:04.857859+0000","last_unstale":"2024-03-06T07:33:06.852776+0000","last_undegraded":"2024-03-06T07:33:06.852776+0000","last_fullsized":"2024-03-06T07:33:06.852776+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:25.885524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975345+0000","last_change":"2024-03-06T07:33:05.875050+0000","last_active":"2024-03-06T07:33:06.975345+0000","last_peered":"2024-03-06T07:33:06.975345+0000","last_clean":"2024-03-06T07:33:06.975345+0000","last_became_active":"2024-03-06T07:33:05.874727+0000","last_became_peered":"2024-03-06T07:33:05.874727+0000","last_unstale":"2024-03-06T07:33:06.975345+0000","last_undegraded":"2024-03-06T07:33:06.975345+0000","last_fullsized":"2024-03-06T07:33:06.975345+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:02.689703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005322+0000","last_change":"2024-03-06T07:33:04.860247+0000","last_active":"2024-03-06T07:33:07.005322+0000","last_peered":"2024-03-06T07:33:07.005322+0000","last_clean":"2024-03-06T07:33:07.005322+0000","last_became_active":"2024-03-06T07:33:04.859606+0000","last_became_peered":"2024-03-06T07:33:04.859606+0000","last_unstale":"2024-03-06T07:33:07.005322+0000","last_undegraded":"2024-03-06T07:33:07.005322+0000","last_fullsized":"2024-03-06T07:33:07.005322+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:34.793545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007433+0000","last_change":"2024-03-06T07:33:05.885681+0000","last_active":"2024-03-06T07:33:07.007433+0000","last_peered":"2024-03-06T07:33:07.007433+0000","last_clean":"2024-03-06T07:33:07.007433+0000","last_became_active":"2024-03-06T07:33:05.885398+0000","last_became_peered":"2024-03-06T07:33:05.885398+0000","last_unstale":"2024-03-06T07:33:07.007433+0000","last_undegraded":"2024-03-06T07:33:07.007433+0000","last_fullsized":"2024-03-06T07:33:07.007433+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:29.053384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852050+0000","last_change":"2024-03-06T07:33:04.864317+0000","last_active":"2024-03-06T07:33:06.852050+0000","last_peered":"2024-03-06T07:33:06.852050+0000","last_clean":"2024-03-06T07:33:06.852050+0000","last_became_active":"2024-03-06T07:33:04.864003+0000","last_became_peered":"2024-03-06T07:33:04.864003+0000","last_unstale":"2024-03-06T07:33:06.852050+0000","last_undegraded":"2024-03-06T07:33:06.852050+0000","last_fullsized":"2024-03-06T07:33:06.852050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:10.627589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003224+0000","last_change":"2024-03-06T07:33:05.859977+0000","last_active":"2024-03-06T07:33:07.003224+0000","last_peered":"2024-03-06T07:33:07.003224+0000","last_clean":"2024-03-06T07:33:07.003224+0000","last_became_active":"2024-03-06T07:33:05.859661+0000","last_became_peered":"2024-03-06T07:33:05.859661+0000","last_unstale":"2024-03-06T07:33:07.003224+0000","last_undegraded":"2024-03-06T07:33:07.003224+0000","last_fullsized":"2024-03-06T07:33:07.003224+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:39.100082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853202+0000","last_change":"2024-03-06T07:33:04.854837+0000","last_active":"2024-03-06T07:33:06.853202+0000","last_peered":"2024-03-06T07:33:06.853202+0000","last_clean":"2024-03-06T07:33:06.853202+0000","last_became_active":"2024-03-06T07:33:04.854303+0000","last_became_peered":"2024-03-06T07:33:04.854303+0000","last_unstale":"2024-03-06T07:33:06.853202+0000","last_undegraded":"2024-03-06T07:33:06.853202+0000","last_fullsized":"2024-03-06T07:33:06.853202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:00.426311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002856+0000","last_change":"2024-03-06T07:33:05.868458+0000","last_active":"2024-03-06T07:33:07.002856+0000","last_peered":"2024-03-06T07:33:07.002856+0000","last_clean":"2024-03-06T07:33:07.002856+0000","last_became_active":"2024-03-06T07:33:05.866577+0000","last_became_peered":"2024-03-06T07:33:05.866577+0000","last_unstale":"2024-03-06T07:33:07.002856+0000","last_undegraded":"2024-03-06T07:33:07.002856+0000","last_fullsized":"2024-03-06T07:33:07.002856+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:36.126403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975191+0000","last_change":"2024-03-06T07:33:04.861361+0000","last_active":"2024-03-06T07:33:06.975191+0000","last_peered":"2024-03-06T07:33:06.975191+0000","last_clean":"2024-03-06T07:33:06.975191+0000","last_became_active":"2024-03-06T07:33:04.861055+0000","last_became_peered":"2024-03-06T07:33:04.861055+0000","last_unstale":"2024-03-06T07:33:06.975191+0000","last_undegraded":"2024-03-06T07:33:06.975191+0000","last_fullsized":"2024-03-06T07:33:06.975191+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:00.247995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002484+0000","last_change":"2024-03-06T07:33:05.859278+0000","last_active":"2024-03-06T07:33:07.002484+0000","last_peered":"2024-03-06T07:33:07.002484+0000","last_clean":"2024-03-06T07:33:07.002484+0000","last_became_active":"2024-03-06T07:33:05.858929+0000","last_became_peered":"2024-03-06T07:33:05.858929+0000","last_unstale":"2024-03-06T07:33:07.002484+0000","last_undegraded":"2024-03-06T07:33:07.002484+0000","last_fullsized":"2024-03-06T07:33:07.002484+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:38.800668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975116+0000","last_change":"2024-03-06T07:33:04.856628+0000","last_active":"2024-03-06T07:33:06.975116+0000","last_peered":"2024-03-06T07:33:06.975116+0000","last_clean":"2024-03-06T07:33:06.975116+0000","last_became_active":"2024-03-06T07:33:04.855981+0000","last_became_peered":"2024-03-06T07:33:04.855981+0000","last_unstale":"2024-03-06T07:33:06.975116+0000","last_undegraded":"2024-03-06T07:33:06.975116+0000","last_fullsized":"2024-03-06T07:33:06.975116+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:44.242932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004274+0000","last_change":"2024-03-06T07:33:05.884260+0000","last_active":"2024-03-06T07:33:07.004274+0000","last_peered":"2024-03-06T07:33:07.004274+0000","last_clean":"2024-03-06T07:33:07.004274+0000","last_became_active":"2024-03-06T07:33:05.883984+0000","last_became_peered":"2024-03-06T07:33:05.883984+0000","last_unstale":"2024-03-06T07:33:07.004274+0000","last_undegraded":"2024-03-06T07:33:07.004274+0000","last_fullsized":"2024-03-06T07:33:07.004274+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:58.814629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.851018+0000","last_change":"2024-03-06T07:33:04.861157+0000","last_active":"2024-03-06T07:33:06.851018+0000","last_peered":"2024-03-06T07:33:06.851018+0000","last_clean":"2024-03-06T07:33:06.851018+0000","last_became_active":"2024-03-06T07:33:04.860313+0000","last_became_peered":"2024-03-06T07:33:04.860313+0000","last_unstale":"2024-03-06T07:33:06.851018+0000","last_undegraded":"2024-03-06T07:33:06.851018+0000","last_fullsized":"2024-03-06T07:33:06.851018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:25.287017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852922+0000","last_change":"2024-03-06T07:33:05.874379+0000","last_active":"2024-03-06T07:33:06.852922+0000","last_peered":"2024-03-06T07:33:06.852922+0000","last_clean":"2024-03-06T07:33:06.852922+0000","last_became_active":"2024-03-06T07:33:05.874052+0000","last_became_peered":"2024-03-06T07:33:05.874052+0000","last_unstale":"2024-03-06T07:33:06.852922+0000","last_undegraded":"2024-03-06T07:33:06.852922+0000","last_fullsized":"2024-03-06T07:33:06.852922+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:23.164088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.975828+0000","last_change":"2024-03-06T07:33:04.864490+0000","last_active":"2024-03-06T07:33:06.975828+0000","last_peered":"2024-03-06T07:33:06.975828+0000","last_clean":"2024-03-06T07:33:06.975828+0000","last_became_active":"2024-03-06T07:33:04.863728+0000","last_became_peered":"2024-03-06T07:33:04.863728+0000","last_unstale":"2024-03-06T07:33:06.975828+0000","last_undegraded":"2024-03-06T07:33:06.975828+0000","last_fullsized":"2024-03-06T07:33:06.975828+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:16.375896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004729+0000","last_change":"2024-03-06T07:33:05.875908+0000","last_active":"2024-03-06T07:33:07.004729+0000","last_peered":"2024-03-06T07:33:07.004729+0000","last_clean":"2024-03-06T07:33:07.004729+0000","last_became_active":"2024-03-06T07:33:05.875264+0000","last_became_peered":"2024-03-06T07:33:05.875264+0000","last_unstale":"2024-03-06T07:33:07.004729+0000","last_undegraded":"2024-03-06T07:33:07.004729+0000","last_fullsized":"2024-03-06T07:33:07.004729+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:01.262218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"93'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.859274+0000","last_change":"2024-03-06T07:33:04.855353+0000","last_active":"2024-03-06T07:33:07.859274+0000","last_peered":"2024-03-06T07:33:07.859274+0000","last_clean":"2024-03-06T07:33:07.859274+0000","last_became_active":"2024-03-06T07:33:04.854865+0000","last_became_peered":"2024-03-06T07:33:04.854865+0000","last_unstale":"2024-03-06T07:33:07.859274+0000","last_undegraded":"2024-03-06T07:33:07.859274+0000","last_fullsized":"2024-03-06T07:33:07.859274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:23.097346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004568+0000","last_change":"2024-03-06T07:33:05.884234+0000","last_active":"2024-03-06T07:33:07.004568+0000","last_peered":"2024-03-06T07:33:07.004568+0000","last_clean":"2024-03-06T07:33:07.004568+0000","last_became_active":"2024-03-06T07:33:05.883983+0000","last_became_peered":"2024-03-06T07:33:05.883983+0000","last_unstale":"2024-03-06T07:33:07.004568+0000","last_undegraded":"2024-03-06T07:33:07.004568+0000","last_fullsized":"2024-03-06T07:33:07.004568+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:17.730172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853604+0000","last_change":"2024-03-06T07:33:04.862187+0000","last_active":"2024-03-06T07:33:06.853604+0000","last_peered":"2024-03-06T07:33:06.853604+0000","last_clean":"2024-03-06T07:33:06.853604+0000","last_became_active":"2024-03-06T07:33:04.861839+0000","last_became_peered":"2024-03-06T07:33:04.861839+0000","last_unstale":"2024-03-06T07:33:06.853604+0000","last_undegraded":"2024-03-06T07:33:06.853604+0000","last_fullsized":"2024-03-06T07:33:06.853604+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:34.793545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004791+0000","last_change":"2024-03-06T07:33:05.880230+0000","last_active":"2024-03-06T07:33:07.004791+0000","last_peered":"2024-03-06T07:33:07.004791+0000","last_clean":"2024-03-06T07:33:07.004791+0000","last_became_active":"2024-03-06T07:33:05.879898+0000","last_became_peered":"2024-03-06T07:33:05.879898+0000","last_unstale":"2024-03-06T07:33:07.004791+0000","last_undegraded":"2024-03-06T07:33:07.004791+0000","last_fullsized":"2024-03-06T07:33:07.004791+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:46.594997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.005090+0000","last_change":"2024-03-06T07:33:04.856983+0000","last_active":"2024-03-06T07:33:07.005090+0000","last_peered":"2024-03-06T07:33:07.005090+0000","last_clean":"2024-03-06T07:33:07.005090+0000","last_became_active":"2024-03-06T07:33:04.856695+0000","last_became_peered":"2024-03-06T07:33:04.856695+0000","last_unstale":"2024-03-06T07:33:07.005090+0000","last_undegraded":"2024-03-06T07:33:07.005090+0000","last_fullsized":"2024-03-06T07:33:07.005090+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:29.115215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003011+0000","last_change":"2024-03-06T07:33:05.873762+0000","last_active":"2024-03-06T07:33:07.003011+0000","last_peered":"2024-03-06T07:33:07.003011+0000","last_clean":"2024-03-06T07:33:07.003011+0000","last_became_active":"2024-03-06T07:33:05.873467+0000","last_became_peered":"2024-03-06T07:33:05.873467+0000","last_unstale":"2024-03-06T07:33:07.003011+0000","last_undegraded":"2024-03-06T07:33:07.003011+0000","last_fullsized":"2024-03-06T07:33:07.003011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:04.300387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853423+0000","last_change":"2024-03-06T07:33:04.853282+0000","last_active":"2024-03-06T07:33:06.853423+0000","last_peered":"2024-03-06T07:33:06.853423+0000","last_clean":"2024-03-06T07:33:06.853423+0000","last_became_active":"2024-03-06T07:33:04.852975+0000","last_became_peered":"2024-03-06T07:33:04.852975+0000","last_unstale":"2024-03-06T07:33:06.853423+0000","last_undegraded":"2024-03-06T07:33:06.853423+0000","last_fullsized":"2024-03-06T07:33:06.853423+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:04.731706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.852320+0000","last_change":"2024-03-06T07:33:05.883667+0000","last_active":"2024-03-06T07:33:06.852320+0000","last_peered":"2024-03-06T07:33:06.852320+0000","last_clean":"2024-03-06T07:33:06.852320+0000","last_became_active":"2024-03-06T07:33:05.883327+0000","last_became_peered":"2024-03-06T07:33:05.883327+0000","last_unstale":"2024-03-06T07:33:06.852320+0000","last_undegraded":"2024-03-06T07:33:06.852320+0000","last_fullsized":"2024-03-06T07:33:06.852320+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:05.745929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004186+0000","last_change":"2024-03-06T07:33:04.850498+0000","last_active":"2024-03-06T07:33:07.004186+0000","last_peered":"2024-03-06T07:33:07.004186+0000","last_clean":"2024-03-06T07:33:07.004186+0000","last_became_active":"2024-03-06T07:33:04.849848+0000","last_became_peered":"2024-03-06T07:33:04.849848+0000","last_unstale":"2024-03-06T07:33:07.004186+0000","last_undegraded":"2024-03-06T07:33:07.004186+0000","last_fullsized":"2024-03-06T07:33:07.004186+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:32.196543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002797+0000","last_change":"2024-03-06T07:33:05.885936+0000","last_active":"2024-03-06T07:33:07.002797+0000","last_peered":"2024-03-06T07:33:07.002797+0000","last_clean":"2024-03-06T07:33:07.002797+0000","last_became_active":"2024-03-06T07:33:05.885337+0000","last_became_peered":"2024-03-06T07:33:05.885337+0000","last_unstale":"2024-03-06T07:33:07.002797+0000","last_undegraded":"2024-03-06T07:33:07.002797+0000","last_fullsized":"2024-03-06T07:33:07.002797+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:04.512996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:06.853815+0000","last_change":"2024-03-06T07:33:04.846351+0000","last_active":"2024-03-06T07:33:06.853815+0000","last_peered":"2024-03-06T07:33:06.853815+0000","last_clean":"2024-03-06T07:33:06.853815+0000","last_became_active":"2024-03-06T07:33:04.845704+0000","last_became_peered":"2024-03-06T07:33:04.845704+0000","last_unstale":"2024-03-06T07:33:06.853815+0000","last_undegraded":"2024-03-06T07:33:06.853815+0000","last_fullsized":"2024-03-06T07:33:06.853815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:46.245470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.004275+0000","last_change":"2024-03-06T07:33:05.882402+0000","last_active":"2024-03-06T07:33:07.004275+0000","last_peered":"2024-03-06T07:33:07.004275+0000","last_clean":"2024-03-06T07:33:07.004275+0000","last_became_active":"2024-03-06T07:33:05.882047+0000","last_became_peered":"2024-03-06T07:33:05.882047+0000","last_unstale":"2024-03-06T07:33:07.004275+0000","last_undegraded":"2024-03-06T07:33:07.004275+0000","last_fullsized":"2024-03-06T07:33:07.004275+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:47.006500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.002796+0000","last_change":"2024-03-06T07:33:04.852903+0000","last_active":"2024-03-06T07:33:07.002796+0000","last_peered":"2024-03-06T07:33:07.002796+0000","last_clean":"2024-03-06T07:33:07.002796+0000","last_became_active":"2024-03-06T07:33:04.851062+0000","last_became_peered":"2024-03-06T07:33:04.851062+0000","last_unstale":"2024-03-06T07:33:07.002796+0000","last_undegraded":"2024-03-06T07:33:07.002796+0000","last_fullsized":"2024-03-06T07:33:07.002796+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:05.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.003828+0000","last_change":"2024-03-06T07:33:05.855680+0000","last_active":"2024-03-06T07:33:07.003828+0000","last_peered":"2024-03-06T07:33:07.003828+0000","last_clean":"2024-03-06T07:33:07.003828+0000","last_became_active":"2024-03-06T07:33:05.855371+0000","last_became_peered":"2024-03-06T07:33:05.855371+0000","last_unstale":"2024-03-06T07:33:07.003828+0000","last_undegraded":"2024-03-06T07:33:07.003828+0000","last_fullsized":"2024-03-06T07:33:07.003828+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:04.817945+0000","last_clean_scrub_stamp":"2024-03-06T07:33:04.817945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:07.414947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.007110+0000","last_change":"2024-03-06T07:33:04.861144+0000","last_active":"2024-03-06T07:33:07.007110+0000","last_peered":"2024-03-06T07:33:07.007110+0000","last_clean":"2024-03-06T07:33:07.007110+0000","last_became_active":"2024-03-06T07:33:04.860862+0000","last_became_peered":"2024-03-06T07:33:04.860862+0000","last_unstale":"2024-03-06T07:33:07.007110+0000","last_undegraded":"2024-03-06T07:33:07.007110+0000","last_fullsized":"2024-03-06T07:33:07.007110+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:03.803722+0000","last_clean_scrub_stamp":"2024-03-06T07:33:03.803722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:46.245470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":303,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:33:07.006395+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:33:07.006395+0000","last_peered":"2024-03-06T07:33:07.006395+0000","last_clean":"2024-03-06T07:33:07.006395+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:33:07.006395+0000","last_undegraded":"2024-03-06T07:33:07.006395+0000","last_fullsized":"2024-03-06T07:33:07.006395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1753088,"data_stored":1082703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1159168,"data_stored":492875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574915,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":492293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1171456,"data_stored":493959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1167360,"data_stored":493857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1155072,"data_stored":492257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1150976,"data_stored":492223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32892,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338948,"statfs":{"total":96636764160,"available":96603082752,"internally_reserved":0,"allocated":1748992,"data_stored":1083173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:10.764 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:33:10.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:33:11.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.025+0000 7f6328600640 1 Processor -- start 2024-03-06T07:33:11.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.025+0000 7f6328600640 1 -- start start 2024-03-06T07:33:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.027+0000 7f6328600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 0x7f632006f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.027+0000 7f6328600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63200d1260 con 0x7f632015e000 2024-03-06T07:33:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.027+0000 7f6328600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632006c320 con 0x7f632015e3e0 2024-03-06T07:33:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.027+0000 7f6328600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632006c0d0 con 0x7f632006fef0 2024-03-06T07:33:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.027+0000 7f6325c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f632015e000 0x7f632006f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53044/0 (socket says 172.21.15.114:53044) 2024-03-06T07:33:11.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.027+0000 7f6325c00640 1 -- 172.21.15.114:0/1209469695 learned_addr learned my addr 172.21.15.114:0/1209469695 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f6325200640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 0x7f632006f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731689108 0 0) 0x7f63200d1260 con 0x7f632015e000 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6308003520 con 0x7f632015e000 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2120095107 0 0) 0x7f6308003520 con 0x7f632015e000 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 msgr2=0x7f632006f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 0x7f632006f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 >> v1:172.21.15.177:6789/0 conn(0x7f632015e3e0 legacy=0x7f63200ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632006c600 con 0x7f632015e000 2024-03-06T07:33:11.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6314001670 con 0x7f632015e000 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 >> v1:172.21.15.114:6789/0 conn(0x7f632015e000 legacy=0x7f632006f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63080039e0 0x7f6308003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6308004800 0x7f6308004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 0x7f63080051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632006c0d0 con 0x7f63080039e0 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630800bb00 con 0x7f6308004800 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.028+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630800bd00 con 0x7f632006fef0 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6325c00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63080039e0 0x7f6308003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6326600640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 0x7f63080051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6325200640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6308004800 0x7f6308004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6325c00640 1 -- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 msgr2=0x7f63080051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6325c00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f632006fef0 0x7f63080051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6325c00640 1 -- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6308004800 msgr2=0x7f6308004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6325c00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6308004800 0x7f6308004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.029+0000 7f6325c00640 1 -- 172.21.15.114:0/1209469695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6308003520 con 0x7f63080039e0 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.030+0000 7f6325c00640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63080039e0 0x7f6308003e00 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f631400c0e0 tx=0x7f6308005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30efeba16aee76ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.030+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314007c00 con 0x7f63080039e0 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.030+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314030070 con 0x7f63080039e0 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.030+0000 7f631fe00640 1 -- 172.21.15.114:0/1209469695 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314005d10 con 0x7f63080039e0 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.031+0000 7f6328600640 1 -- 172.21.15.114:0/1209469695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63080039e0 msgr2=0x7f6308003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.031+0000 7f6328600640 1 --2- 172.21.15.114:0/1209469695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63080039e0 0x7f6308003e00 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f631400c0e0 tx=0x7f6308005830 comp rx=0 tx=0).stop 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.031+0000 7f6326600640 1 -- 172.21.15.114:0/1209469695 reap_dead start 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.031+0000 7f6328600640 1 -- 172.21.15.114:0/1209469695 shutdown_connections 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.031+0000 7f6328600640 1 -- 172.21.15.114:0/1209469695 >> 172.21.15.114:0/1209469695 conn(0x7f63200b3800 msgr2=0x7f63200b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.031+0000 7f6328600640 1 -- 172.21.15.114:0/1209469695 shutdown_connections 2024-03-06T07:33:11.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.031+0000 7f6328600640 1 -- 172.21.15.114:0/1209469695 wait complete. 2024-03-06T07:33:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.033+0000 7f6328600640 1 Processor -- start 2024-03-06T07:33:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.033+0000 7f6328600640 1 -- start start 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6328600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63080039e0 0x7f6320145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6328600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632015ea50 0x7f632015eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6328600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63201f7800 0x7f63201f7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6328600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632006f130 con 0x7f632015ea50 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6328600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632006e020 con 0x7f63080039e0 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6328600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632006f2b0 con 0x7f63201f7800 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6325200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632015ea50 0x7f632015eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6325200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632015ea50 0x7f632015eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60848/0 (socket says 172.21.15.114:60848) 2024-03-06T07:33:11.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.034+0000 7f6325200640 1 -- 172.21.15.114:0/4288802007 learned_addr learned my addr 172.21.15.114:0/4288802007 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325c00640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63080039e0 0x7f6320145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6326600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63201f7800 0x7f63201f7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325200640 1 -- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63201f7800 msgr2=0x7f63201f7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325200640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63201f7800 0x7f63201f7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325200640 1 -- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63080039e0 msgr2=0x7f6320145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325200640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63080039e0 0x7f6320145ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325200640 1 -- 172.21.15.114:0/4288802007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63201f3c90 con 0x7f632015ea50 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325c00640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63080039e0 0x7f6320145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6325200640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632015ea50 0x7f632015eed0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f630c010890 tx=0x7f630c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f631d600640 1 -- 172.21.15.114:0/4288802007 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c00f040 con 0x7f632015ea50 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f631d600640 1 -- 172.21.15.114:0/4288802007 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c00eb00 con 0x7f632015ea50 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f631d600640 1 -- 172.21.15.114:0/4288802007 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c01c400 con 0x7f632015ea50 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.035+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6320146480 con 0x7f632015ea50 2024-03-06T07:33:11.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.036+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6320146a40 con 0x7f632015ea50 2024-03-06T07:33:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.037+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632004f330 con 0x7f632015ea50 2024-03-06T07:33:11.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.038+0000 7f631d600640 1 -- 172.21.15.114:0/4288802007 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f630c01c5a0 con 0x7f632015ea50 2024-03-06T07:33:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.038+0000 7f631d600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f630006b500 0x7f630006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.039+0000 7f631d600640 1 -- 172.21.15.114:0/4288802007 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f630c08a6b0 con 0x7f632015ea50 2024-03-06T07:33:11.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.039+0000 7f6325c00640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f630006b500 0x7f630006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.039+0000 7f6325c00640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f630006b500 0x7f630006d9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6314034000 tx=0x7f6314007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.043+0000 7f631d600640 1 -- 172.21.15.114:0/4288802007 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f630c018030 con 0x7f632015ea50 2024-03-06T07:33:11.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.335+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f63200b7c50 con 0x7f632015ea50 2024-03-06T07:33:11.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.338+0000 7f631d600640 1 -- 172.21.15.114:0/4288802007 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f630c059520 con 0x7f632015ea50 2024-03-06T07:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:33:11.340 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:33:11.341 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:33:11.341 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:33:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f630006b500 msgr2=0x7f630006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f630006b500 0x7f630006d9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6314034000 tx=0x7f6314007000 comp rx=0 tx=0).stop 2024-03-06T07:33:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632015ea50 msgr2=0x7f632015eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632015ea50 0x7f632015eed0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f630c010890 tx=0x7f630c010d50 comp rx=0 tx=0).stop 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 shutdown_connections 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63201f7800 0x7f63201f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f630006b500 0x7f630006d9c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632015ea50 0x7f632015eed0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 --2- 172.21.15.114:0/4288802007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63080039e0 0x7f6320145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 >> 172.21.15.114:0/4288802007 conn(0x7f63200b3800 msgr2=0x7f63200aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 shutdown_connections 2024-03-06T07:33:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.342+0000 7f6328600640 1 -- 172.21.15.114:0/4288802007 wait complete. 2024-03-06T07:33:11.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:11.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.557+0000 7fb11a800640 1 Processor -- start 2024-03-06T07:33:11.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.558+0000 7fb11a800640 1 -- start start 2024-03-06T07:33:11.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb11a800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140d24c0 0x7fb1140d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb11a800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1140b2de0 0x7fb1140d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb11a800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140d2de0 con 0x7fb1140d07f0 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb11a800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1140d4a00 con 0x7fb1140b2de0 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb11a800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1140cf7e0 con 0x7fb1140d24c0 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb112a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1140b2de0 0x7fb1140d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb112a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1140b2de0 0x7fb1140d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60854/0 (socket says 172.21.15.114:60854) 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.559+0000 7fb112a00640 1 -- 172.21.15.114:0/2864583066 learned_addr learned my addr 172.21.15.114:0/2864583066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:11.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.560+0000 7fb113400640 1 --2- 172.21.15.114:0/2864583066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140d24c0 0x7fb1140d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.560+0000 7fb112a00640 1 -- 172.21.15.114:0/2864583066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140d24c0 msgr2=0x7fb1140d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.560+0000 7fb112a00640 1 --2- 172.21.15.114:0/2864583066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140d24c0 0x7fb1140d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.560+0000 7fb112a00640 1 -- 172.21.15.114:0/2864583066 >> v1:172.21.15.177:6789/0 conn(0x7fb1140d07f0 legacy=0x7fb1140d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.560+0000 7fb112a00640 1 -- 172.21.15.114:0/2864583066 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1140d4c50 con 0x7fb1140b2de0 2024-03-06T07:33:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.561+0000 7fb112a00640 1 --2- 172.21.15.114:0/2864583066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140b2de0 0x7fb1140d28a0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009e50 tx=0x7fb0fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5232aa422fc657fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.561+0000 7fb112000640 1 -- 172.21.15.114:0/2864583066 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc02c9e0 con 0x7fb1140b2de0 2024-03-06T07:33:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.561+0000 7fb112000640 1 -- 172.21.15.114:0/2864583066 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc02cb80 con 0x7fb1140b2de0 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.561+0000 7fb112000640 1 -- 172.21.15.114:0/2864583066 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0343f0 con 0x7fb1140b2de0 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 -- 172.21.15.114:0/2864583066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140b2de0 msgr2=0x7fb1140d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 --2- 172.21.15.114:0/2864583066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140b2de0 0x7fb1140d28a0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009e50 tx=0x7fb0fc02be40 comp rx=0 tx=0).stop 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 -- 172.21.15.114:0/2864583066 shutdown_connections 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 --2- 172.21.15.114:0/2864583066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140b2de0 0x7fb1140d28a0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 --2- 172.21.15.114:0/2864583066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140d24c0 0x7fb1140d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 -- 172.21.15.114:0/2864583066 >> 172.21.15.114:0/2864583066 conn(0x7fb1140d3ee0 msgr2=0x7fb1140d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 -- 172.21.15.114:0/2864583066 shutdown_connections 2024-03-06T07:33:11.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.562+0000 7fb11a800640 1 -- 172.21.15.114:0/2864583066 wait complete. 2024-03-06T07:33:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.564+0000 7fb11a800640 1 Processor -- start 2024-03-06T07:33:11.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.564+0000 7fb11a800640 1 -- start start 2024-03-06T07:33:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.565+0000 7fb11a800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140b2de0 0x7fb1140cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.565+0000 7fb11a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1140d07f0 0x7fb114151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.565+0000 7fb11a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140d24c0 0x7fb1140cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.565+0000 7fb11a800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1140cfd10 con 0x7fb1140d24c0 2024-03-06T07:33:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.565+0000 7fb11a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1140d0080 con 0x7fb1140d07f0 2024-03-06T07:33:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.565+0000 7fb11a800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11415e5b0 con 0x7fb1140b2de0 2024-03-06T07:33:11.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.565+0000 7fb113e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140d24c0 0x7fb1140cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140d24c0 0x7fb1140cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60858/0 (socket says 172.21.15.114:60858) 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 -- 172.21.15.114:0/2636481983 learned_addr learned my addr 172.21.15.114:0/2636481983 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113400640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140b2de0 0x7fb1140cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb112a00640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1140d07f0 0x7fb114151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 -- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140b2de0 msgr2=0x7fb1140cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140b2de0 0x7fb1140cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 -- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1140d07f0 msgr2=0x7fb114151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1140d07f0 0x7fb114151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 -- 172.21.15.114:0/2636481983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1140cd2b0 con 0x7fb1140d24c0 2024-03-06T07:33:11.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113400640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140b2de0 0x7fb1140cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.566+0000 7fb113e00640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140d24c0 0x7fb1140cc540 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb10c00c440 tx=0x7fb10c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.567+0000 7fb10b400640 1 -- 172.21.15.114:0/2636481983 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c011790 con 0x7fb1140d24c0 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.567+0000 7fb10b400640 1 -- 172.21.15.114:0/2636481983 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c017070 con 0x7fb1140d24c0 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.567+0000 7fb10b400640 1 -- 172.21.15.114:0/2636481983 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c011b70 con 0x7fb1140d24c0 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.567+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1140cba60 con 0x7fb1140d24c0 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.567+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1140cd590 con 0x7fb1140d24c0 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.568+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11404f330 con 0x7fb1140d24c0 2024-03-06T07:33:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.569+0000 7fb10b400640 1 -- 172.21.15.114:0/2636481983 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb10c011d10 con 0x7fb1140d24c0 2024-03-06T07:33:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.570+0000 7fb10b400640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0f006b500 0x7fb0f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.570+0000 7fb10b400640 1 -- 172.21.15.114:0/2636481983 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb10c089ff0 con 0x7fb1140d24c0 2024-03-06T07:33:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.570+0000 7fb113400640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0f006b500 0x7fb0f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.571+0000 7fb113400640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0f006b500 0x7fb0f006d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb104006fd0 tx=0x7fb104008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.575+0000 7fb10b400640 1 -- 172.21.15.114:0/2636481983 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb10c058e60 con 0x7fb1140d24c0 2024-03-06T07:33:11.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.808+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1140b50b0 con 0x7fb1140d24c0 2024-03-06T07:33:11.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T07:33:11.808+0000 7f31f0800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.808+0000 7fb10b400640 1 -- 172.21.15.114:0/2636481983 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fb10c058760 con 0x7fb1140d24c0 2024-03-06T07:33:11.811 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:11.811 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/1475876370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1475876370},{"type":"v1","addr":"172.21.15.114:6836","nonce":1475876370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5734,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/2451255436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":2451255436},{"type":"v1","addr":"172.21.15.177:6833","nonce":2451255436}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5737,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3480882059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3480882059},{"type":"v1","addr":"172.21.15.177:6835","nonce":3480882059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:08.374434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/406262499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":406262499},{"type":"v1","addr":"172.21.15.114:6837","nonce":406262499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":7}]} 2024-03-06T07:33:11.811 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 61 2024-03-06T07:33:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0f006b500 msgr2=0x7fb0f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0f006b500 0x7fb0f006d9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb104006fd0 tx=0x7fb104008040 comp rx=0 tx=0).stop 2024-03-06T07:33:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140d24c0 msgr2=0x7fb1140cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140d24c0 0x7fb1140cc540 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb10c00c440 tx=0x7fb10c00c900 comp rx=0 tx=0).stop 2024-03-06T07:33:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 shutdown_connections 2024-03-06T07:33:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb0f006b500 0x7fb0f006d9c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1140d24c0 0x7fb1140cc540 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1140d07f0 0x7fb114151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 --2- 172.21.15.114:0/2636481983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1140b2de0 0x7fb1140cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 >> 172.21.15.114:0/2636481983 conn(0x7fb1140d3ee0 msgr2=0x7fb1140b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 shutdown_connections 2024-03-06T07:33:11.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:11.811+0000 7fb11a800640 1 -- 172.21.15.114:0/2636481983 wait complete. 2024-03-06T07:33:12.233 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:05.301366+0000', 'modified': '2024-03-06T07:33:08.374434+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7143}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7143': {'gid': 7143, 'name': 'a', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/406262499', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 406262499}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 406262499}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7143, 'qdb_cluster': [7143]} 2024-03-06T07:33:12.234 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:33:12.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:33:12.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.451+0000 7f786fc00640 1 Processor -- start 2024-03-06T07:33:12.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.451+0000 7f786fc00640 1 -- start start 2024-03-06T07:33:12.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786fc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7868160400 0x7f786806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786fc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f786806b500 0x7f78681414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786fc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786815f940 con 0x7f786813ef20 2024-03-06T07:33:12.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786fc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786806e9c0 con 0x7f786806b500 2024-03-06T07:33:12.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786fc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786815ce80 con 0x7f7868160400 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f786806b500 0x7f78681414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f786806b500 0x7f78681414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60862/0 (socket says 172.21.15.114:60862) 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.453+0000 7f786dc00640 1 -- 172.21.15.114:0/1518069955 learned_addr learned my addr 172.21.15.114:0/1518069955 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.454+0000 7f7867e00640 1 --2- 172.21.15.114:0/1518069955 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7868160400 0x7f786806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.454+0000 7f786dc00640 1 -- 172.21.15.114:0/1518069955 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7868160400 msgr2=0x7f786806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.454+0000 7f786dc00640 1 --2- 172.21.15.114:0/1518069955 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7868160400 0x7f786806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.454+0000 7f786dc00640 1 -- 172.21.15.114:0/1518069955 >> v1:172.21.15.177:6789/0 conn(0x7f786813ef20 legacy=0x7f786815c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.454+0000 7f786dc00640 1 -- 172.21.15.114:0/1518069955 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786815d160 con 0x7f786806b500 2024-03-06T07:33:12.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.456+0000 7f786dc00640 1 --2- 172.21.15.114:0/1518069955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806b500 0x7f78681414b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f7860009e50 tx=0x7f786002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147b49bf8260a684 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.456+0000 7f7867400640 1 -- 172.21.15.114:0/1518069955 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786002cb00 con 0x7f786806b500 2024-03-06T07:33:12.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.456+0000 7f7867400640 1 -- 172.21.15.114:0/1518069955 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786002cca0 con 0x7f786806b500 2024-03-06T07:33:12.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.456+0000 7f7867400640 1 -- 172.21.15.114:0/1518069955 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860034610 con 0x7f786806b500 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 -- 172.21.15.114:0/1518069955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806b500 msgr2=0x7f78681414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 --2- 172.21.15.114:0/1518069955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806b500 0x7f78681414b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f7860009e50 tx=0x7f786002bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 -- 172.21.15.114:0/1518069955 shutdown_connections 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 --2- 172.21.15.114:0/1518069955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786806b500 0x7f78681414b0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 --2- 172.21.15.114:0/1518069955 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7868160400 0x7f786806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 -- 172.21.15.114:0/1518069955 >> 172.21.15.114:0/1518069955 conn(0x7f78680d0b90 msgr2=0x7f78680d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 -- 172.21.15.114:0/1518069955 shutdown_connections 2024-03-06T07:33:12.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.457+0000 7f786fc00640 1 -- 172.21.15.114:0/1518069955 wait complete. 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.459+0000 7f786fc00640 1 Processor -- start 2024-03-06T07:33:12.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.459+0000 7f786fc00640 1 -- start start 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786fc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f786806b500 0x7f786813a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786fc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f786813ef20 0x7f78681f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786fc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868160400 0x7f786813d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786fc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680700e0 con 0x7f7868160400 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786fc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786815fee0 con 0x7f786806b500 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786fc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786815ca60 con 0x7f786813ef20 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868160400 0x7f786813d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868160400 0x7f786813d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60872/0 (socket says 172.21.15.114:60872) 2024-03-06T07:33:12.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786dc00640 1 -- 172.21.15.114:0/4045644634 learned_addr learned my addr 172.21.15.114:0/4045644634 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786d200640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f786806b500 0x7f786813a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f7867e00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f786813ef20 0x7f78681f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786dc00640 1 -- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f786813ef20 msgr2=0x7f78681f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786dc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f786813ef20 0x7f78681f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786dc00640 1 -- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f786806b500 msgr2=0x7f786813a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.460+0000 7f786dc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f786806b500 0x7f786813a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.461+0000 7f786dc00640 1 -- 172.21.15.114:0/4045644634 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868143180 con 0x7f7868160400 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.461+0000 7f786dc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868160400 0x7f786813d270 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f7860036040 tx=0x7f7860004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.461+0000 7f7864c00640 1 -- 172.21.15.114:0/4045644634 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860043020 con 0x7f7868160400 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.461+0000 7f7864c00640 1 -- 172.21.15.114:0/4045644634 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786002ce50 con 0x7f7868160400 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.461+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7868143460 con 0x7f7868160400 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.461+0000 7f7864c00640 1 -- 172.21.15.114:0/4045644634 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860009510 con 0x7f7868160400 2024-03-06T07:33:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.461+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7868143a20 con 0x7f7868160400 2024-03-06T07:33:12.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.463+0000 7f7864c00640 1 -- 172.21.15.114:0/4045644634 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7860046020 con 0x7f7868160400 2024-03-06T07:33:12.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.465+0000 7f78729e3740 1 Processor -- start 2024-03-06T07:33:12.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.465+0000 7f7864c00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f784806b4d0 0x7f784806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.465+0000 7f78729e3740 1 -- start start 2024-03-06T07:33:12.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.465+0000 7f7864c00640 1 -- 172.21.15.114:0/4045644634 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78600ac7d0 con 0x7f7868160400 2024-03-06T07:33:12.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.465+0000 7f7867e00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f784806b4d0 0x7f784806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.466+0000 7f7867e00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f784806b4d0 0x7f784806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f786813e1f0 tx=0x7f7854008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.466+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd56750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.466+0000 7f78729e3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd86540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.466+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd56c90 0x55cf2cd86a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.466+0000 7f78729e3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2ccc0600 con 0x55cf2cd83b90 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.466+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2cc69930 con 0x55cf2cd837b0 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.466+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf2ccbc6a0 con 0x55cf2cd56c90 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd86540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd86540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60878/0 (socket says 172.21.15.114:60878) 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 -- 172.21.15.114:0/1671329255 learned_addr learned my addr 172.21.15.114:0/1671329255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f786d200640 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd56750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f786dc00640 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd56c90 0x55cf2cd86a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 -- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd56c90 msgr2=0x55cf2cd86a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd56c90 0x55cf2cd86a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 -- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 msgr2=0x55cf2cd56750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd56750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 -- 172.21.15.114:0/1671329255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf2c9d52d0 con 0x55cf2cd83b90 2024-03-06T07:33:12.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.467+0000 7f7867e00640 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd86540 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7854011cc0 tx=0x7f7854014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be583ec92da6132f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f7859600640 1 -- 172.21.15.114:0/1671329255 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854011cf0 con 0x55cf2cd83b90 2024-03-06T07:33:12.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f7859600640 1 -- 172.21.15.114:0/1671329255 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785401a030 con 0x55cf2cd83b90 2024-03-06T07:33:12.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f7859600640 1 -- 172.21.15.114:0/1671329255 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854011e90 con 0x55cf2cd83b90 2024-03-06T07:33:12.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f78729e3740 1 -- 172.21.15.114:0/1671329255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 msgr2=0x55cf2cd86540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f78729e3740 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd86540 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7854011cc0 tx=0x7f7854014000 comp rx=0 tx=0).stop 2024-03-06T07:33:12.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f78729e3740 1 -- 172.21.15.114:0/1671329255 shutdown_connections 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f78729e3740 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd56c90 0x55cf2cd86a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f78729e3740 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd86540 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f78729e3740 1 --2- 172.21.15.114:0/1671329255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd56750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.468+0000 7f78729e3740 1 -- 172.21.15.114:0/1671329255 >> 172.21.15.114:0/1671329255 conn(0x55cf2cd589c0 msgr2=0x55cf2cda1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.469+0000 7f78729e3740 1 -- 172.21.15.114:0/1671329255 shutdown_connections 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.469+0000 7f78729e3740 1 -- 172.21.15.114:0/1671329255 wait complete. 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.470+0000 7f78729e3740 1 -- start start 2024-03-06T07:33:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.470+0000 7f78729e3740 1 Processor -- start 2024-03-06T07:33:12.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd83b90 0x55cf2cd716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f78729e3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd9bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f78729e3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2cd5af60 con 0x55cf2cdb5660 2024-03-06T07:33:12.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2cc7cd90 con 0x55cf2cd837b0 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf2cd5a350 con 0x55cf2cd83b90 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f78729e3740 10 client.0 _ll_register_callbacks cb 0x55cf2ccb8350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd9bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd9bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60890/0 (socket says 172.21.15.114:60890) 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 -- 172.21.15.114:0/3521412208 learned_addr learned my addr 172.21.15.114:0/3521412208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f7867e00640 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd83b90 0x55cf2cd716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786d200640 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 -- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd83b90 msgr2=0x55cf2cd716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd83b90 0x55cf2cd716f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 -- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 msgr2=0x55cf2cd888b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd888b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 -- 172.21.15.114:0/3521412208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf2ca9f6b0 con 0x55cf2cdb5660 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.472+0000 7f786dc00640 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd9bb70 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f7860083090 tx=0x7f786007a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf84c6498e0e650 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f784e000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f784e000640 1 -- 172.21.15.114:0/3521412208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786007f000 con 0x55cf2cdb5660 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f784e000640 1 -- 172.21.15.114:0/3521412208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f786002cb10 con 0x55cf2cdb5660 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f785401a030 con 0x55cf2cdb5660 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f784e000640 1 -- 172.21.15.114:0/3521412208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78600835c0 con 0x55cf2cdb5660 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 10 client.7287 fetch_fsmap learned FSMap version 61 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55cf2cd896c0 con 0x55cf2cdb5660 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f784e000640 1 -- 172.21.15.114:0/3521412208 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785401a030 con 0x55cf2cdb5660 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f784e000640 1 -- 172.21.15.114:0/3521412208 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f786003c350 con 0x55cf2cdb5660 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 10 client.7287 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 10 client.7287 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 20 client.7287 populate_metadata read hostname 'smithi114' 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cdbd550 0x55cf2cdbf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 4 client.7287 mds_command: new command op to 7143 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:33:12.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.473+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 --> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf2cccdb40 con 0x55cf2cdbd550 2024-03-06T07:33:12.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.474+0000 7f786d200640 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cdbd550 0x55cf2cdbf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.474+0000 7f786d200640 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cdbd550 0x55cf2cdbf940 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77846e4ad0ba0585 server_cookie=39edc6316b275d87 in_seq=0 out_seq=0 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.474+0000 7f784e000640 10 client.7287 ms_handle_connect on v2:172.21.15.114:6834/406262499 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.475+0000 7f784e000640 1 -- 172.21.15.114:0/3521412208 <== mds.0 v2:172.21.15.114:6834/406262499 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cf2cccdb40 con 0x55cf2cdbd550 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.475+0000 7f784e000640 10 client.7287 handle_command_reply: tid=0 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.475+0000 7f78729e3740 1 client.7287 shutdown 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.475+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 msgr2=0x55cf2cd9bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.475+0000 7f78729e3740 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd9bb70 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f7860083090 tx=0x7f786007a810 comp rx=0 tx=0).stop 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.475+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 shutdown_connections 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cdbd550 0x55cf2cdbf940 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd9bb70 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd83b90 0x55cf2cd716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 1 --2- 172.21.15.114:0/3521412208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd888b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f784e000640 0 client.7287 ms_handle_reset on v2:172.21.15.114:6834/406262499 2024-03-06T07:33:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 >> 172.21.15.114:0/3521412208 conn(0x55cf2cd56c90 msgr2=0x55cf2cd86530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 shutdown_connections 2024-03-06T07:33:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 1 -- 172.21.15.114:0/3521412208 wait complete. 2024-03-06T07:33:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.476+0000 7f78729e3740 20 client.7287 trim_cache size 0 max 16384 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.494+0000 7f78729e3740 1 Processor -- start 2024-03-06T07:33:12.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.494+0000 7f78729e3740 1 -- start start 2024-03-06T07:33:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd9af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f78729e3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd9b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cdb5660 0x55cf2cd81f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f78729e3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7868200230 con 0x55cf2cd83b90 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680cfa10 con 0x55cf2cd837b0 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78681f8140 con 0x55cf2cdb5660 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd9b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd9b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60906/0 (socket says 172.21.15.114:60906) 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 -- 172.21.15.114:0/3641498373 learned_addr learned my addr 172.21.15.114:0/3641498373 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f786d200640 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd9af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 -- 172.21.15.114:0/3641498373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cdb5660 msgr2=0x55cf2cd81f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cdb5660 0x55cf2cd81f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 -- 172.21.15.114:0/3641498373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 msgr2=0x55cf2cd9af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd9af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.495+0000 7f7867e00640 1 -- 172.21.15.114:0/3641498373 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf2cd9bee0 con 0x55cf2cd83b90 2024-03-06T07:33:12.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.496+0000 7f7867e00640 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd9b4a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f7854015ca0 tx=0x7f7854018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adceec4f20bf7b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.496+0000 7f7858c00640 1 -- 172.21.15.114:0/3641498373 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854011cf0 con 0x55cf2cd83b90 2024-03-06T07:33:12.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.496+0000 7f7858c00640 1 -- 172.21.15.114:0/3641498373 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785401a040 con 0x55cf2cd83b90 2024-03-06T07:33:12.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.496+0000 7f7858c00640 1 -- 172.21.15.114:0/3641498373 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854011e90 con 0x55cf2cd83b90 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 -- 172.21.15.114:0/3641498373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 msgr2=0x55cf2cd9b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd9b4a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f7854015ca0 tx=0x7f7854018000 comp rx=0 tx=0).stop 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 -- 172.21.15.114:0/3641498373 shutdown_connections 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cdb5660 0x55cf2cd81f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cd83b90 0x55cf2cd9b4a0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 --2- 172.21.15.114:0/3641498373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd837b0 0x55cf2cd9af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 -- 172.21.15.114:0/3641498373 >> 172.21.15.114:0/3641498373 conn(0x55cf2cd56c90 msgr2=0x55cf2cd56060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 -- 172.21.15.114:0/3641498373 shutdown_connections 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 -- 172.21.15.114:0/3641498373 wait complete. 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.497+0000 7f78729e3740 1 -- start start 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.498+0000 7f78729e3740 1 Processor -- start 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd837b0 0x55cf2cd5a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f78729e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd83b90 0x55cf2cd5aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f78729e3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd5b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f78729e3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2cd8a1f0 con 0x55cf2cdb5660 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680cf410 con 0x55cf2cd83b90 2024-03-06T07:33:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f78729e3740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf2cd726b0 con 0x55cf2cd837b0 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f78729e3740 10 client.0 _ll_register_callbacks cb 0x7f7860007310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd5b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd5b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60920/0 (socket says 172.21.15.114:60920) 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 -- 172.21.15.114:0/2965531030 learned_addr learned my addr 172.21.15.114:0/2965531030 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f7867e00640 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd83b90 0x55cf2cd5aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786d200640 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd837b0 0x55cf2cd5a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 -- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd837b0 msgr2=0x55cf2cd5a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd837b0 0x55cf2cd5a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 -- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd83b90 msgr2=0x55cf2cd5aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd83b90 0x55cf2cd5aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.499+0000 7f786dc00640 1 -- 172.21.15.114:0/2965531030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf2cd9cca0 con 0x55cf2cdb5660 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f786dc00640 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd5b450 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f7860007fd0 tx=0x7f7860009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137b65d3c6c740eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f784e000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:33:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f784e000640 1 -- 172.21.15.114:0/2965531030 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860099070 con 0x55cf2cdb5660 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f784e000640 1 -- 172.21.15.114:0/2965531030 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7860083330 con 0x55cf2cdb5660 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f785401a040 con 0x55cf2cdb5660 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f784e000640 1 -- 172.21.15.114:0/2965531030 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f786007ac80 con 0x55cf2cdb5660 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f78729e3740 10 client.7293 fetch_fsmap learned FSMap version 61 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x55cf2cd80ff0 con 0x55cf2cdb5660 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f784e000640 1 -- 172.21.15.114:0/2965531030 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785401a040 con 0x55cf2cdb5660 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f784e000640 1 -- 172.21.15.114:0/2965531030 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f78600a4170 con 0x55cf2cdb5660 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f78729e3740 10 client.7293 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f78729e3740 10 client.7293 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.500+0000 7f78729e3740 20 client.7293 populate_metadata read hostname 'smithi114' 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.501+0000 7f78729e3740 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cd5c000 0x55cf2cdb36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.501+0000 7f78729e3740 4 client.7293 mds_command: new command op to 7143 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.501+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 --> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cf2ccc7e90 con 0x55cf2cd5c000 2024-03-06T07:33:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.501+0000 7f786d200640 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cd5c000 0x55cf2cdb36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.501+0000 7f786d200640 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cd5c000 0x55cf2cdb36d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d12e521f2b8fe14 server_cookie=2595e39e90ce912b in_seq=0 out_seq=0 2024-03-06T07:33:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.501+0000 7f784e000640 10 client.7293 ms_handle_connect on v2:172.21.15.114:6834/406262499 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f784e000640 1 -- 172.21.15.114:0/2965531030 <== mds.0 v2:172.21.15.114:6834/406262499 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55cf2ccc7e90 con 0x55cf2cd5c000 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f784e000640 10 client.7293 handle_command_reply: tid=0 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 client.7293 shutdown 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 msgr2=0x55cf2cd5b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd5b450 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f7860007fd0 tx=0x7f7860009810 comp rx=0 tx=0).stop 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 shutdown_connections 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:6834/406262499,v1:172.21.15.114:6837/406262499] conn(0x55cf2cd5c000 0x55cf2cdb36d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf2cdb5660 0x55cf2cd5b450 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cf2cd83b90 0x55cf2cd5aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 --2- 172.21.15.114:0/2965531030 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cf2cd837b0 0x55cf2cd5a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f784e000640 0 client.7293 ms_handle_reset on v2:172.21.15.114:6834/406262499 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 >> 172.21.15.114:0/2965531030 conn(0x55cf2cd56c90 msgr2=0x55cf2cd55620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 shutdown_connections 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 1 -- 172.21.15.114:0/2965531030 wait complete. 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.502+0000 7f78729e3740 20 client.7293 trim_cache size 0 max 16384 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 7143, 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:33:12.509 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:33:12.510 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:33:12.510 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 6.658323813, 2024-03-06T07:33:12.510 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 61, 2024-03-06T07:33:12.510 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 93, 2024-03-06T07:33:12.510 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-03-06T07:33:12.510 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 12.093462453000001 2024-03-06T07:33:12.510 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f784806b4d0 msgr2=0x7f784806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f784806b4d0 0x7f784806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f786813e1f0 tx=0x7f7854008040 comp rx=0 tx=0).stop 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868160400 msgr2=0x7f786813d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868160400 0x7f786813d270 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f7860036040 tx=0x7f7860004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 shutdown_connections 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f784806b4d0 0x7f784806d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868160400 0x7f786813d270 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f786813ef20 0x7f78681f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 --2- 172.21.15.114:0/4045644634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f786806b500 0x7f786813a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 >> 172.21.15.114:0/4045644634 conn(0x7f78680d0b90 msgr2=0x7f78680d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 shutdown_connections 2024-03-06T07:33:12.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.503+0000 7f786fc00640 1 -- 172.21.15.114:0/4045644634 wait complete. 2024-03-06T07:33:12.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:33:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.719+0000 7faccc800640 1 Processor -- start 2024-03-06T07:33:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.719+0000 7faccc800640 1 -- start start 2024-03-06T07:33:12.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.721+0000 7faccc800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facc415fe60 0x7facc4071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.721+0000 7faccc800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc406cef0 con 0x7facc415f9d0 2024-03-06T07:33:12.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.721+0000 7faccc800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc406d1d0 con 0x7facc4144c00 2024-03-06T07:33:12.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.721+0000 7faccc800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc406d4b0 con 0x7facc415fe60 2024-03-06T07:33:12.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.721+0000 7facc9400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7facc415f9d0 0x7facc4070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53048/0 (socket says 172.21.15.114:53048) 2024-03-06T07:33:12.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.721+0000 7facca800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facc415fe60 0x7facc4071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.721+0000 7facc9400640 1 -- 172.21.15.114:0/1866712773 learned_addr learned my addr 172.21.15.114:0/1866712773 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:12.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.722+0000 7facca800640 1 -- 172.21.15.114:0/1866712773 >> v1:172.21.15.177:6789/0 conn(0x7facc4144c00 legacy=0x7facc406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.722+0000 7facca800640 1 -- 172.21.15.114:0/1866712773 >> v1:172.21.15.114:6789/0 conn(0x7facc415f9d0 legacy=0x7facc4070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.722+0000 7facca800640 1 -- 172.21.15.114:0/1866712773 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc406d790 con 0x7facc415fe60 2024-03-06T07:33:12.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.723+0000 7facca800640 1 --2- 172.21.15.114:0/1866712773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc415fe60 0x7facc4071220 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7facc0009d90 tx=0x7facc002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0a5bd253fb9eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7facc8a00640 1 -- 172.21.15.114:0/1866712773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0039070 con 0x7facc415fe60 2024-03-06T07:33:12.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7facc8a00640 1 -- 172.21.15.114:0/1866712773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc002cd00 con 0x7facc415fe60 2024-03-06T07:33:12.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7facc8a00640 1 -- 172.21.15.114:0/1866712773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0034660 con 0x7facc415fe60 2024-03-06T07:33:12.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7faccc800640 1 -- 172.21.15.114:0/1866712773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc415fe60 msgr2=0x7facc4071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7faccc800640 1 --2- 172.21.15.114:0/1866712773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc415fe60 0x7facc4071220 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7facc0009d90 tx=0x7facc002bf50 comp rx=0 tx=0).stop 2024-03-06T07:33:12.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7faccc800640 1 -- 172.21.15.114:0/1866712773 shutdown_connections 2024-03-06T07:33:12.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7faccc800640 1 --2- 172.21.15.114:0/1866712773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc415fe60 0x7facc4071220 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.724+0000 7faccc800640 1 -- 172.21.15.114:0/1866712773 >> 172.21.15.114:0/1866712773 conn(0x7facc40b1730 msgr2=0x7facc40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:12.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.725+0000 7faccc800640 1 -- 172.21.15.114:0/1866712773 shutdown_connections 2024-03-06T07:33:12.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.725+0000 7faccc800640 1 -- 172.21.15.114:0/1866712773 wait complete. 2024-03-06T07:33:12.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.726+0000 7faccc800640 1 Processor -- start 2024-03-06T07:33:12.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.726+0000 7faccc800640 1 -- start start 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.727+0000 7faccc800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc4144c00 0x7facc413ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.727+0000 7faccc800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facc415f9d0 0x7facc41f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.727+0000 7faccc800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facc415fe60 0x7facc40ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.727+0000 7faccc800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc406cb50 con 0x7facc4144c00 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.727+0000 7faccc800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc4160730 con 0x7facc415f9d0 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.727+0000 7faccc800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc406ccd0 con 0x7facc415fe60 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc4144c00 0x7facc413ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc4144c00 0x7facc413ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60946/0 (socket says 172.21.15.114:60946) 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 -- 172.21.15.114:0/2145361233 learned_addr learned my addr 172.21.15.114:0/2145361233 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9400640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facc415f9d0 0x7facc41f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facca800640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facc415fe60 0x7facc40ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 -- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facc415fe60 msgr2=0x7facc40ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facc415fe60 0x7facc40ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 -- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facc415f9d0 msgr2=0x7facc41f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:12.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facc415f9d0 0x7facc41f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:12.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.728+0000 7facc9e00640 1 -- 172.21.15.114:0/2145361233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc415d850 con 0x7facc4144c00 2024-03-06T07:33:12.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.729+0000 7facc9e00640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc4144c00 0x7facc413ab10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7facb800c430 tx=0x7facb800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.729+0000 7facbe000640 1 -- 172.21.15.114:0/2145361233 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8017070 con 0x7facc4144c00 2024-03-06T07:33:12.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.729+0000 7facbe000640 1 -- 172.21.15.114:0/2145361233 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb80052d0 con 0x7facc4144c00 2024-03-06T07:33:12.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.729+0000 7facbe000640 1 -- 172.21.15.114:0/2145361233 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8003830 con 0x7facc4144c00 2024-03-06T07:33:12.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.729+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc415db30 con 0x7facc4144c00 2024-03-06T07:33:12.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.729+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc415e0f0 con 0x7facc4144c00 2024-03-06T07:33:12.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.730+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc415e5e0 con 0x7facc4144c00 2024-03-06T07:33:12.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.732+0000 7facbe000640 1 -- 172.21.15.114:0/2145361233 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facb801a070 con 0x7facc4144c00 2024-03-06T07:33:12.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.733+0000 7facbe000640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faca806b4d0 0x7faca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:12.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.733+0000 7facbe000640 1 -- 172.21.15.114:0/2145361233 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facb80894c0 con 0x7facc4144c00 2024-03-06T07:33:12.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.733+0000 7facc9400640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faca806b4d0 0x7faca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:12.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.734+0000 7facc9400640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faca806b4d0 0x7faca806d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7facc40b13e0 tx=0x7facb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:12.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:12.737+0000 7facbe000640 1 -- 172.21.15.114:0/2145361233 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7facb80583d0 con 0x7facc4144c00 2024-03-06T07:33:13.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.030+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7facc40b0910 con 0x7facc4144c00 2024-03-06T07:33:13.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.031+0000 7facbe000640 1 -- 172.21.15.114:0/2145361233 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) v1 ==== 53+0+73 (secure 0 0 0) 0x7facb8057cd0 con 0x7facc4144c00 2024-03-06T07:33:13.033 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:33:13.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.033+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faca806b4d0 msgr2=0x7faca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.033+0000 7faccc800640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faca806b4d0 0x7faca806d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7facc40b13e0 tx=0x7facb4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.033+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc4144c00 msgr2=0x7facc413ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.033+0000 7faccc800640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc4144c00 0x7facc413ab10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7facb800c430 tx=0x7facb800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 shutdown_connections 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facc415fe60 0x7facc40ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facc415f9d0 0x7facc41f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faca806b4d0 0x7faca806d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 --2- 172.21.15.114:0/2145361233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facc4144c00 0x7facc413ab10 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 >> 172.21.15.114:0/2145361233 conn(0x7facc40b1730 msgr2=0x7facc40afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 shutdown_connections 2024-03-06T07:33:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:13.034+0000 7faccc800640 1 -- 172.21.15.114:0/2145361233 wait complete. 2024-03-06T07:33:13.060 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:33:13.060 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:33:13.060 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:33:13.060 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:33:13.061 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:13.061 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:33:13.061 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:33:13.061 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:33:13.061 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:13.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.061 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:33:13.107 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42262sec preferred_lft 42262sec 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:13.109 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:13.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.110 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:33:13.230 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:33:13.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.234 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:33:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:33:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:33:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:33:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:33:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:33:13.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.281 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:33:13.281 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:33:13.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:33:13.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:33:13.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:33:13.282 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:33:13.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.496 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:33:13.544 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:33:13.544 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:33:13.545 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:13.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:33:13.635 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:33:13.635 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:13.685 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:33:13.685 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:13.731 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:33:13.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:13.909 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:13.909 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:33:13.947 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:13.947 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:33:13.947 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:33:13.947 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:33:13.947 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:33:13.949 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:33:13.950 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:33:13.950 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:33:13.950 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:33:13.950 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:33:13.955 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:33:13.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:13.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:14.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:14.030 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:33:14.089 INFO:teuthology.orchestra.run.smithi114.stdout:client5393 2024-03-06T07:33:14.089 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:33:14.089 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:14.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:14.183 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:14.339 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-06T07:33:14.339 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-06T07:33:14.339 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:14.339 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:14.340 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:14.341 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-06T07:33:14.342 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-06T07:33:14.343 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-03-06T07:33:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-06T07:33:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.347 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-06T07:33:14.348 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.349 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_files = 6 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:14.357 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:14.358 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-06T07:33:14.359 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on skipped message = False 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:14.360 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:14.361 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:14.362 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-06T07:33:14.363 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.364 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.365 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.366 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.367 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_files = 6 2024-03-06T07:33:14.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:14.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:14.570 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:14.571 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-06T07:33:14.572 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:14.573 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-03-06T07:33:14.574 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-06T07:33:14.575 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.576 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:14.577 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:33:14.578 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_files = 6 2024-03-06T07:33:14.579 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:14.580 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:14.581 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-06T07:33:14.582 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on skipped message = False 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.583 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:14.584 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:14.585 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.586 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:14.587 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-03-06T07:33:14.588 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:14.589 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_files = 6 2024-03-06T07:33:14.590 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:14.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:14.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.825+0000 7fc52f000640 1 Processor -- start 2024-03-06T07:33:14.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.826+0000 7fc52f000640 1 -- start start 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.827+0000 7fc52f000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc5280cfd80 0x7fc5281445c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.827+0000 7fc52f000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc528161ec0 0x7fc52806d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.827+0000 7fc52f000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc528071790 0x7fc528071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.827+0000 7fc52f000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5281418a0 con 0x7fc528071790 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.827+0000 7fc52f000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc528144b00 con 0x7fc5280cfd80 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.827+0000 7fc52f000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52806d8d0 con 0x7fc528161ec0 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc528071790 0x7fc528071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc528071790 0x7fc528071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60952/0 (socket says 172.21.15.114:60952) 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 -- 172.21.15.114:0/4120783103 learned_addr learned my addr 172.21.15.114:0/4120783103 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc51f400640 1 --2- 172.21.15.114:0/4120783103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc528161ec0 0x7fc52806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 -- 172.21.15.114:0/4120783103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc528161ec0 msgr2=0x7fc52806d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc527e00640 1 --2- 172.21.15.114:0/4120783103 >> v2:172.21.15.177:3300/0 conn(0x7fc5280cfd80 0x7fc5281445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 --2- 172.21.15.114:0/4120783103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc528161ec0 0x7fc52806d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 -- 172.21.15.114:0/4120783103 >> v2:172.21.15.177:3300/0 conn(0x7fc5280cfd80 msgr2=0x7fc5281445c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 --2- 172.21.15.114:0/4120783103 >> v2:172.21.15.177:3300/0 conn(0x7fc5280cfd80 0x7fc5281445c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc52d000640 1 -- 172.21.15.114:0/4120783103 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528162430 con 0x7fc528071790 2024-03-06T07:33:14.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.828+0000 7fc527e00640 1 --2- 172.21.15.114:0/4120783103 >> v2:172.21.15.177:3300/0 conn(0x7fc5280cfd80 0x7fc5281445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:14.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.829+0000 7fc52d000640 1 --2- 172.21.15.114:0/4120783103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528071790 0x7fc528071c20 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc520009e50 tx=0x7fc52002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dfd8b5c09a7e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.829+0000 7fc527400640 1 -- 172.21.15.114:0/4120783103 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52002ca80 con 0x7fc528071790 2024-03-06T07:33:14.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc527400640 1 -- 172.21.15.114:0/4120783103 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52002cc20 con 0x7fc528071790 2024-03-06T07:33:14.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc527400640 1 -- 172.21.15.114:0/4120783103 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5200343f0 con 0x7fc528071790 2024-03-06T07:33:14.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 -- 172.21.15.114:0/4120783103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528071790 msgr2=0x7fc528071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 --2- 172.21.15.114:0/4120783103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528071790 0x7fc528071c20 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc520009e50 tx=0x7fc52002bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 -- 172.21.15.114:0/4120783103 shutdown_connections 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 --2- 172.21.15.114:0/4120783103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528071790 0x7fc528071c20 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 --2- 172.21.15.114:0/4120783103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc528161ec0 0x7fc52806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 --2- 172.21.15.114:0/4120783103 >> v2:172.21.15.177:3300/0 conn(0x7fc5280cfd80 0x7fc5281445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 -- 172.21.15.114:0/4120783103 >> 172.21.15.114:0/4120783103 conn(0x7fc5280b2510 msgr2=0x7fc5280b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 -- 172.21.15.114:0/4120783103 shutdown_connections 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.830+0000 7fc52f000640 1 -- 172.21.15.114:0/4120783103 wait complete. 2024-03-06T07:33:14.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.832+0000 7fc52f000640 1 Processor -- start 2024-03-06T07:33:14.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.832+0000 7fc52f000640 1 -- start start 2024-03-06T07:33:14.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52f000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc528071790 0x7fc52815dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52f000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5280cfd80 0x7fc528144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52f000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528161ec0 0x7fc5281451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52f000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc528162cd0 con 0x7fc528161ec0 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52f000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5281629f0 con 0x7fc528071790 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52f000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280d0160 con 0x7fc5280cfd80 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528161ec0 0x7fc5281451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528161ec0 0x7fc5281451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60964/0 (socket says 172.21.15.114:60964) 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.833+0000 7fc52d000640 1 -- 172.21.15.114:0/318855457 learned_addr learned my addr 172.21.15.114:0/318855457 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc51f400640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5280cfd80 0x7fc528144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc527e00640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc528071790 0x7fc52815dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc52d000640 1 -- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5280cfd80 msgr2=0x7fc528144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc52d000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5280cfd80 0x7fc528144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc52d000640 1 -- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc528071790 msgr2=0x7fc52815dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc52d000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc528071790 0x7fc52815dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc52d000640 1 -- 172.21.15.114:0/318855457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528141a20 con 0x7fc528161ec0 2024-03-06T07:33:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc51f400640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5280cfd80 0x7fc528144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc527e00640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc528071790 0x7fc52815dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc52d000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528161ec0 0x7fc5281451c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc520009e50 tx=0x7fc520004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc524c00640 1 -- 172.21.15.114:0/318855457 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520043020 con 0x7fc528161ec0 2024-03-06T07:33:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc524c00640 1 -- 172.21.15.114:0/318855457 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52003e450 con 0x7fc528161ec0 2024-03-06T07:33:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.834+0000 7fc524c00640 1 -- 172.21.15.114:0/318855457 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520009510 con 0x7fc528161ec0 2024-03-06T07:33:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.835+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc528146ba0 con 0x7fc528161ec0 2024-03-06T07:33:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.835+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc528141cc0 con 0x7fc528161ec0 2024-03-06T07:33:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.836+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc528068050 con 0x7fc528161ec0 2024-03-06T07:33:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.837+0000 7fc524c00640 1 -- 172.21.15.114:0/318855457 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc520046020 con 0x7fc528161ec0 2024-03-06T07:33:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.838+0000 7fc524c00640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc50406b4d0 0x7fc50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.838+0000 7fc524c00640 1 -- 172.21.15.114:0/318855457 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5200acb40 con 0x7fc528161ec0 2024-03-06T07:33:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.838+0000 7fc527e00640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc50406b4d0 0x7fc50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.839+0000 7fc527e00640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc50406b4d0 0x7fc50406d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc518006fd0 tx=0x7fc518008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:14.843+0000 7fc524c00640 1 -- 172.21.15.114:0/318855457 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc52004d030 con 0x7fc528161ec0 2024-03-06T07:33:15.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.055+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc52806c580 con 0x7fc528161ec0 2024-03-06T07:33:15.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.055+0000 7fc524c00640 1 -- 172.21.15.114:0/318855457 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc52007b9a0 con 0x7fc528161ec0 2024-03-06T07:33:15.057 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc50406b4d0 msgr2=0x7fc50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc50406b4d0 0x7fc50406d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc518006fd0 tx=0x7fc518008040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528161ec0 msgr2=0x7fc5281451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528161ec0 0x7fc5281451c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc520009e50 tx=0x7fc520004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 shutdown_connections 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc50406b4d0 0x7fc50406d990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc528161ec0 0x7fc5281451c0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5280cfd80 0x7fc528144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 --2- 172.21.15.114:0/318855457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc528071790 0x7fc52815dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 >> 172.21.15.114:0/318855457 conn(0x7fc5280b2510 msgr2=0x7fc5280b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 shutdown_connections 2024-03-06T07:33:15.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.059+0000 7fc52f000640 1 -- 172.21.15.114:0/318855457 wait complete. 2024-03-06T07:33:15.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:15.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.235 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.236 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-03-06T07:33:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:15.241 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:15.242 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-06T07:33:15.243 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:33:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_files = 6 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:15.245 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.262 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:15.263 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-06T07:33:15.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on skipped message = False 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-06T07:33:15.265 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-06T07:33:15.266 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-06T07:33:15.267 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-06T07:33:15.268 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:15.269 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-06T07:33:15.270 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_files = 6 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:15.271 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:15.306 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:15.306 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:33:15.306 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:15.306 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:15.307 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:33:15.307 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:15.324 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:33:15.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:33:15.325 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:33:15.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:33:15.328 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:33:15.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:33:15.329 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:33:15.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:33:15.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.555+0000 7fc74ec00640 1 Processor -- start 2024-03-06T07:33:15.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.555+0000 7fc74ec00640 1 -- start start 2024-03-06T07:33:15.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.555+0000 7f501dc00640 1 Processor -- start 2024-03-06T07:33:15.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.555+0000 7f501dc00640 1 -- start start 2024-03-06T07:33:15.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.556+0000 7fc74ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74815be90 0x7fc7480b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.556+0000 7fc74ec00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc7480d0780 0x7fc74815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.556+0000 7fc74ec00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7480b5160 con 0x7fc7480acd60 2024-03-06T07:33:15.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.556+0000 7fc74ec00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7480b4f30 con 0x7fc7480d0780 2024-03-06T07:33:15.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.556+0000 7fc74ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480af240 con 0x7fc74815be90 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7f501dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f50180b5020 0x7f50180d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7f501dc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50180d21c0 con 0x7f50180b44f0 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7f501dc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50180b48d0 con 0x7f50180d2cc0 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7f501dc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50180d0000 con 0x7f50180b5020 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7f5016a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f50180b44f0 0x7f50180d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53062/0 (socket says 172.21.15.114:53062) 2024-03-06T07:33:15.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7f5016a00640 1 -- 172.21.15.114:0/4060704566 learned_addr learned my addr 172.21.15.114:0/4060704566 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:15.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7fc747e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74815be90 0x7fc7480b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7f5017400640 1 --2- 172.21.15.114:0/4060704566 >> v2:172.21.15.114:3300/0 conn(0x7f50180b5020 0x7f50180d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7fc747e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74815be90 0x7fc7480b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:42734/0 (socket says 172.21.15.114:42734) 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7fc747e00640 1 -- 172.21.15.114:0/2685806477 learned_addr learned my addr 172.21.15.114:0/2685806477 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.557+0000 7fc74cc00640 1 --2- 172.21.15.114:0/2685806477 >> v2:172.21.15.177:3300/0 conn(0x7fc7480d0780 0x7fc74815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7f5017400640 1 -- 172.21.15.114:0/4060704566 >> v1:172.21.15.177:6789/0 conn(0x7f50180d2cc0 legacy=0x7f50180d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7f5017400640 1 -- 172.21.15.114:0/4060704566 >> v1:172.21.15.114:6789/0 conn(0x7f50180b44f0 legacy=0x7f50180d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7f5017400640 1 -- 172.21.15.114:0/4060704566 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180d1ee0 con 0x7f50180b5020 2024-03-06T07:33:15.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7f5015600640 1 -- 172.21.15.114:0/4060704566 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846122333 0 0) 0x7f50180d21c0 con 0x7f50180b44f0 2024-03-06T07:33:15.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7fc747e00640 1 -- 172.21.15.114:0/2685806477 >> v2:172.21.15.177:3300/0 conn(0x7fc7480d0780 msgr2=0x7fc74815fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7fc747e00640 1 --2- 172.21.15.114:0/2685806477 >> v2:172.21.15.177:3300/0 conn(0x7fc7480d0780 0x7fc74815fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7fc747e00640 1 -- 172.21.15.114:0/2685806477 >> v1:172.21.15.177:6789/0 conn(0x7fc7480acd60 legacy=0x7fc7480d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7fc747e00640 1 -- 172.21.15.114:0/2685806477 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7480ae910 con 0x7fc74815be90 2024-03-06T07:33:15.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.558+0000 7fc746a00640 1 -- 172.21.15.114:0/2685806477 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329023419 0 0) 0x7fc7480b5160 con 0x7fc7480acd60 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.559+0000 7f5017400640 1 --2- 172.21.15.114:0/4060704566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b5020 0x7f50180d0da0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5010009d90 tx=0x7f501002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c999114b9ba18a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.559+0000 7f5015600640 1 -- 172.21.15.114:0/4060704566 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010039070 con 0x7f50180b5020 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7f5015600640 1 -- 172.21.15.114:0/4060704566 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501002cbd0 con 0x7f50180b5020 2024-03-06T07:33:15.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7f5015600640 1 -- 172.21.15.114:0/4060704566 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010034440 con 0x7f50180b5020 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7fc747e00640 1 --2- 172.21.15.114:0/2685806477 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74815be90 0x7fc7480b49f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc738009870 tx=0x7fc73802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7facde6e5df7345 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7fc746a00640 1 -- 172.21.15.114:0/2685806477 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738034070 con 0x7fc74815be90 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7fc746a00640 1 -- 172.21.15.114:0/2685806477 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc738030840 con 0x7fc74815be90 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7fc746a00640 1 -- 172.21.15.114:0/2685806477 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738030c10 con 0x7fc74815be90 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7f501dc00640 1 -- 172.21.15.114:0/4060704566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b5020 msgr2=0x7f50180d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.560+0000 7f501dc00640 1 --2- 172.21.15.114:0/4060704566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b5020 0x7f50180d0da0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5010009d90 tx=0x7f501002be20 comp rx=0 tx=0).stop 2024-03-06T07:33:15.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 -- 172.21.15.114:0/2685806477 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74815be90 msgr2=0x7fc7480b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2685806477 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74815be90 0x7fc7480b49f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc738009870 tx=0x7fc73802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:15.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7f501dc00640 1 -- 172.21.15.114:0/4060704566 shutdown_connections 2024-03-06T07:33:15.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7f501dc00640 1 --2- 172.21.15.114:0/4060704566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b5020 0x7f50180d0da0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7f501dc00640 1 -- 172.21.15.114:0/4060704566 >> 172.21.15.114:0/4060704566 conn(0x7f50180d3fd0 msgr2=0x7f50180d43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 -- 172.21.15.114:0/2685806477 shutdown_connections 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2685806477 >> v2:172.21.15.177:3300/0 conn(0x7fc7480d0780 0x7fc74815fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2685806477 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc74815be90 0x7fc7480b49f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 -- 172.21.15.114:0/2685806477 >> 172.21.15.114:0/2685806477 conn(0x7fc74806b360 msgr2=0x7fc74806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7f501dc00640 1 -- 172.21.15.114:0/4060704566 shutdown_connections 2024-03-06T07:33:15.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 -- 172.21.15.114:0/2685806477 shutdown_connections 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7f501dc00640 1 -- 172.21.15.114:0/4060704566 wait complete. 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.561+0000 7fc74ec00640 1 -- 172.21.15.114:0/2685806477 wait complete. 2024-03-06T07:33:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.564+0000 7f501dc00640 1 Processor -- start 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.564+0000 7fc74ec00640 1 Processor -- start 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.564+0000 7fc74ec00640 1 -- start start 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.564+0000 7f501dc00640 1 -- start start 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc74ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc7480acd60 0x7fc74809cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc74ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7480d0780 0x7fc7481ea550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f501dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b44f0 0x7f50180b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f501dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50180b5020 0x7f50180b9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f501dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50180d2cc0 0x7f50180b7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f501dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50180cfae0 con 0x7f50180b44f0 2024-03-06T07:33:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f501dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50180b4e40 con 0x7f50180d2cc0 2024-03-06T07:33:15.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f501dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50180b4ad0 con 0x7f50180b5020 2024-03-06T07:33:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc74ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74815be90 0x7fc7481eaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc747400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7480d0780 0x7fc7481ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc74ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480b2c80 con 0x7fc7480d0780 2024-03-06T07:33:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f5016a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b44f0 0x7f50180b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f5016a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b44f0 0x7f50180b1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60986/0 (socket says 172.21.15.114:60986) 2024-03-06T07:33:15.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7f5016a00640 1 -- 172.21.15.114:0/3681564300 learned_addr learned my addr 172.21.15.114:0/3681564300 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc747400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7480d0780 0x7fc7481ea550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32770/0 (socket says 172.21.15.114:32770) 2024-03-06T07:33:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc747400640 1 -- 172.21.15.114:0/2925408573 learned_addr learned my addr 172.21.15.114:0/2925408573 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480aeed0 con 0x7fc74815be90 2024-03-06T07:33:15.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480b2e00 con 0x7fc7480acd60 2024-03-06T07:33:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.566+0000 7fc747e00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc7480acd60 0x7fc74809cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5016000640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50180b5020 0x7f50180b9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5016a00640 1 -- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50180b5020 msgr2=0x7f50180b9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5016a00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50180b5020 0x7f50180b9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5017400640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50180d2cc0 0x7f50180b7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5016a00640 1 -- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50180d2cc0 msgr2=0x7f50180b7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7fc747400640 1 -- 172.21.15.114:0/2925408573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc7480acd60 msgr2=0x7fc74809cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7fc747400640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc7480acd60 0x7fc74809cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7fc747400640 1 -- 172.21.15.114:0/2925408573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74815be90 msgr2=0x7fc7481eaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7fc747400640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74815be90 0x7fc7481eaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7fc747400640 1 -- 172.21.15.114:0/2925408573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc748157dc0 con 0x7fc7480d0780 2024-03-06T07:33:15.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5016a00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50180d2cc0 0x7f50180b7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5016a00640 1 -- 172.21.15.114:0/3681564300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180b9e70 con 0x7f50180b44f0 2024-03-06T07:33:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5017400640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50180d2cc0 0x7f50180b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7fc747400640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7480d0780 0x7fc7481ea550 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c4b0 tx=0x7fc73400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.567+0000 7f5016a00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b44f0 0x7f50180b1c10 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f500800c500 tx=0x7f500800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7fc73fe00640 1 -- 172.21.15.114:0/2925408573 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734017070 con 0x7fc7480d0780 2024-03-06T07:33:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7fc73fe00640 1 -- 172.21.15.114:0/2925408573 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc734005480 con 0x7fc7480d0780 2024-03-06T07:33:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7fc73fe00640 1 -- 172.21.15.114:0/2925408573 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734003830 con 0x7fc7480d0780 2024-03-06T07:33:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7f500ea00640 1 -- 172.21.15.114:0/3681564300 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008018070 con 0x7f50180b44f0 2024-03-06T07:33:15.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7f500ea00640 1 -- 172.21.15.114:0/3681564300 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5008007580 con 0x7f50180b44f0 2024-03-06T07:33:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7f500ea00640 1 -- 172.21.15.114:0/3681564300 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008004a20 con 0x7f50180b44f0 2024-03-06T07:33:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50180b0e50 con 0x7f50180b44f0 2024-03-06T07:33:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7481580a0 con 0x7fc7480d0780 2024-03-06T07:33:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50180b1410 con 0x7f50180b44f0 2024-03-06T07:33:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.568+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc748158660 con 0x7fc7480d0780 2024-03-06T07:33:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.570+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5018068070 con 0x7f50180b44f0 2024-03-06T07:33:15.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.570+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc748068070 con 0x7fc7480d0780 2024-03-06T07:33:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.571+0000 7f500ea00640 1 -- 172.21.15.114:0/3681564300 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f500801b360 con 0x7f50180b44f0 2024-03-06T07:33:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.572+0000 7fc73fe00640 1 -- 172.21.15.114:0/2925408573 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc73401a070 con 0x7fc7480d0780 2024-03-06T07:33:15.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.573+0000 7f500ea00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.573+0000 7f500ea00640 1 -- 172.21.15.114:0/3681564300 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5008089770 con 0x7f50180b44f0 2024-03-06T07:33:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.573+0000 7fc73fe00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc72806b5a0 0x7fc72806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.573+0000 7fc73fe00640 1 -- 172.21.15.114:0/2925408573 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc734089960 con 0x7fc7480d0780 2024-03-06T07:33:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.573+0000 7f5016000640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.574+0000 7fc747e00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc72806b5a0 0x7fc72806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.574+0000 7fc747e00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc72806b5a0 0x7fc72806da60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc738036000 tx=0x7fc73803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.575+0000 7f5016000640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ff806b4d0 0x7f4ff806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.577+0000 7f500ea00640 1 -- 172.21.15.114:0/3681564300 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5008058660 con 0x7f50180b44f0 2024-03-06T07:33:15.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.582+0000 7fc73fe00640 1 -- 172.21.15.114:0/2925408573 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc734021020 con 0x7fc7480d0780 2024-03-06T07:33:15.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.605+0000 7f372f600640 1 Processor -- start 2024-03-06T07:33:15.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.606+0000 7f372f600640 1 -- start start 2024-03-06T07:33:15.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.607+0000 7f372f600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f37280d0780 0x7f37280b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.607+0000 7f372f600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280acd60 0x7f372815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.607+0000 7f372f600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280b5160 con 0x7f372815be90 2024-03-06T07:33:15.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.607+0000 7f372f600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37280b4f30 con 0x7f37280d0780 2024-03-06T07:33:15.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.607+0000 7f372f600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37280af240 con 0x7f37280acd60 2024-03-06T07:33:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.608+0000 7f372d600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280acd60 0x7f372815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.608+0000 7f372cc00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f372815be90 0x7f37280d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:41342/0 (socket says 172.21.15.114:41342) 2024-03-06T07:33:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.608+0000 7f372cc00640 1 -- 172.21.15.114:0/730746944 learned_addr learned my addr 172.21.15.114:0/730746944 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.608+0000 7f3727e00640 1 --2- 172.21.15.114:0/730746944 >> v2:172.21.15.177:3300/0 conn(0x7f37280d0780 0x7f37280b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.608+0000 7f3727400640 1 -- 172.21.15.114:0/730746944 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993248249 0 0) 0x7f37280b5160 con 0x7f372815be90 2024-03-06T07:33:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.608+0000 7f3727400640 1 -- 172.21.15.114:0/730746944 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3710003520 con 0x7f372815be90 2024-03-06T07:33:15.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.608+0000 7f3727e00640 1 -- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280acd60 msgr2=0x7f372815fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.609+0000 7f3727e00640 1 --2- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280acd60 0x7f372815fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.609+0000 7f3727e00640 1 -- 172.21.15.114:0/730746944 >> v1:172.21.15.177:6789/0 conn(0x7f372815be90 legacy=0x7f37280d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.609+0000 7f3727e00640 1 -- 172.21.15.114:0/730746944 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280ae910 con 0x7f37280d0780 2024-03-06T07:33:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.609+0000 7f372d600640 1 --2- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280acd60 0x7f372815fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.610+0000 7f3727e00640 1 --2- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280d0780 0x7f37280b49f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f37140097a0 tx=0x7f371402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6dd81545538bac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.610+0000 7f3727400640 1 -- 172.21.15.114:0/730746944 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714035020 con 0x7f37280d0780 2024-03-06T07:33:15.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.610+0000 7f3727400640 1 -- 172.21.15.114:0/730746944 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3714031810 con 0x7f37280d0780 2024-03-06T07:33:15.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.610+0000 7f3727400640 1 -- 172.21.15.114:0/730746944 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714031bc0 con 0x7f37280d0780 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 -- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280d0780 msgr2=0x7f37280b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 --2- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280d0780 0x7f37280b49f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f37140097a0 tx=0x7f371402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 -- 172.21.15.114:0/730746944 shutdown_connections 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 --2- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280acd60 0x7f372815fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 --2- 172.21.15.114:0/730746944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280d0780 0x7f37280b49f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 -- 172.21.15.114:0/730746944 >> 172.21.15.114:0/730746944 conn(0x7f372806b360 msgr2=0x7f372806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 -- 172.21.15.114:0/730746944 shutdown_connections 2024-03-06T07:33:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.611+0000 7f372f600640 1 -- 172.21.15.114:0/730746944 wait complete. 2024-03-06T07:33:15.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.613+0000 7f372f600640 1 Processor -- start 2024-03-06T07:33:15.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.613+0000 7f372f600640 1 -- start start 2024-03-06T07:33:15.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.614+0000 7f372f600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280acd60 0x7f372813dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.614+0000 7f372f600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280d0780 0x7f37280cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.614+0000 7f372f600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f372815be90 0x7f37280b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.614+0000 7f372f600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37280b2c80 con 0x7f372815be90 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.614+0000 7f372f600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37280aeed0 con 0x7f37280acd60 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.614+0000 7f372f600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37280b2e00 con 0x7f37280d0780 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f372815be90 0x7f37280b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f372815be90 0x7f37280b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32780/0 (socket says 172.21.15.114:32780) 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 -- 172.21.15.114:0/3811608777 learned_addr learned my addr 172.21.15.114:0/3811608777 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:15.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f3727e00640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280d0780 0x7f37280cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372cc00640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280acd60 0x7f372813dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 -- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280d0780 msgr2=0x7f37280cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280d0780 0x7f37280cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 -- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280acd60 msgr2=0x7f372813dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280acd60 0x7f372813dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 -- 172.21.15.114:0/3811608777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280cc820 con 0x7f372815be90 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372cc00640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280acd60 0x7f372813dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.615+0000 7f372d600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f372815be90 0x7f37280b1970 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3720009e50 tx=0x7f372000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.616+0000 7f3724c00640 1 -- 172.21.15.114:0/3811608777 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720018030 con 0x7f372815be90 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.616+0000 7f3724c00640 1 -- 172.21.15.114:0/3811608777 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372000cd90 con 0x7f372815be90 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.616+0000 7f3724c00640 1 -- 172.21.15.114:0/3811608777 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720003bb0 con 0x7f372815be90 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.616+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37280ccb00 con 0x7f372815be90 2024-03-06T07:33:15.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.616+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37280cd0c0 con 0x7f372815be90 2024-03-06T07:33:15.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.618+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3728068070 con 0x7f372815be90 2024-03-06T07:33:15.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.618+0000 7fd5dfa00640 1 Processor -- start 2024-03-06T07:33:15.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.618+0000 7fd5dfa00640 1 -- start start 2024-03-06T07:33:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.619+0000 7f3724c00640 1 -- 172.21.15.114:0/3811608777 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f372001b070 con 0x7f372815be90 2024-03-06T07:33:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dfa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5d815cc90 0x7fd5d806d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dfa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d80d0780 0x7fd5d8162f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dfa00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d81635a0 con 0x7fd5d8162b00 2024-03-06T07:33:15.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dfa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d806d940 con 0x7fd5d815cc90 2024-03-06T07:33:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dfa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d8163880 con 0x7fd5d80d0780 2024-03-06T07:33:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dd000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5d815cc90 0x7fd5d806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dd000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5d815cc90 0x7fd5d806d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32796/0 (socket says 172.21.15.114:32796) 2024-03-06T07:33:15.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dd000640 1 -- 172.21.15.114:0/2719928660 learned_addr learned my addr 172.21.15.114:0/2719928660 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.620+0000 7fd5dda00640 1 --2- 172.21.15.114:0/2719928660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d80d0780 0x7fd5d8162f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.621+0000 7f3724c00640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f370806b520 0x7f370806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.621+0000 7f3724c00640 1 -- 172.21.15.114:0/3811608777 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f372008abb0 con 0x7f372815be90 2024-03-06T07:33:15.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.621+0000 7f372cc00640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f370806b520 0x7f370806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.621+0000 7fd5dd000640 1 -- 172.21.15.114:0/2719928660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d80d0780 msgr2=0x7fd5d8162f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.621+0000 7fd5dd000640 1 --2- 172.21.15.114:0/2719928660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d80d0780 0x7fd5d8162f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.621+0000 7fd5dd000640 1 -- 172.21.15.114:0/2719928660 >> v1:172.21.15.177:6789/0 conn(0x7fd5d8162b00 legacy=0x7fd5d80d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.621+0000 7fd5dd000640 1 -- 172.21.15.114:0/2719928660 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d806cc00 con 0x7fd5d815cc90 2024-03-06T07:33:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.622+0000 7f372cc00640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f370806b520 0x7f370806d9e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f371c005880 tx=0x7f371c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.623+0000 7fd5dd000640 1 --2- 172.21.15.114:0/2719928660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d815cc90 0x7fd5d806d400 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009870 tx=0x7fd5cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49537becea8d6b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.625+0000 7fd5d7400640 1 -- 172.21.15.114:0/2719928660 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc034070 con 0x7fd5d815cc90 2024-03-06T07:33:15.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.625+0000 7fd5d7400640 1 -- 172.21.15.114:0/2719928660 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5cc030840 con 0x7fd5d815cc90 2024-03-06T07:33:15.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.625+0000 7fd5d7400640 1 -- 172.21.15.114:0/2719928660 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc030c10 con 0x7fd5d815cc90 2024-03-06T07:33:15.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.626+0000 7f3724c00640 1 -- 172.21.15.114:0/3811608777 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3720091080 con 0x7f372815be90 2024-03-06T07:33:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.626+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2719928660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d815cc90 msgr2=0x7fd5d806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.626+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2719928660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d815cc90 0x7fd5d806d400 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009870 tx=0x7fd5cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.626+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2719928660 shutdown_connections 2024-03-06T07:33:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.626+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2719928660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d80d0780 0x7fd5d8162f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.626+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2719928660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d815cc90 0x7fd5d806d400 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.626+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2719928660 >> 172.21.15.114:0/2719928660 conn(0x7fd5d806b360 msgr2=0x7fd5d806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.627+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2719928660 shutdown_connections 2024-03-06T07:33:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.627+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2719928660 wait complete. 2024-03-06T07:33:15.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.628+0000 7fd5dfa00640 1 Processor -- start 2024-03-06T07:33:15.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.628+0000 7fd5dfa00640 1 -- start start 2024-03-06T07:33:15.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.629+0000 7fd5dfa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5d80d0780 0x7fd5d8166b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.629+0000 7fd5dfa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d815cc90 0x7fd5d809c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.629+0000 7fd5dfa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8162b00 0x7fd5d8157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.629+0000 7fd5dfa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80b4fe0 con 0x7fd5d8162b00 2024-03-06T07:33:15.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.629+0000 7fd5dfa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80b4e60 con 0x7fd5d80d0780 2024-03-06T07:33:15.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.629+0000 7fd5dfa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80acd60 con 0x7fd5d815cc90 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.630+0000 7fd5dda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8162b00 0x7fd5d8157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.630+0000 7fd5d7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d815cc90 0x7fd5d809c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.630+0000 7fd5dd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5d80d0780 0x7fd5d8166b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.630+0000 7fd5d7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d815cc90 0x7fd5d809c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:42806/0 (socket says 172.21.15.114:42806) 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.630+0000 7fd5d7e00640 1 -- 172.21.15.114:0/2046923554 learned_addr learned my addr 172.21.15.114:0/2046923554 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.630+0000 7fd5dda00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8162b00 0x7fd5d8157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32806/0 (socket says 172.21.15.114:32806) 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.631+0000 7fd5dda00640 1 -- 172.21.15.114:0/2046923554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d815cc90 msgr2=0x7fd5d809c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.631+0000 7fd5dda00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d815cc90 0x7fd5d809c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.631+0000 7fd5dda00640 1 -- 172.21.15.114:0/2046923554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5d80d0780 msgr2=0x7fd5d8166b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.631+0000 7fd5dda00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5d80d0780 0x7fd5d8166b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.631+0000 7fd5dda00640 1 -- 172.21.15.114:0/2046923554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d809cc90 con 0x7fd5d8162b00 2024-03-06T07:33:15.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.631+0000 7fd5d7e00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d815cc90 0x7fd5d809c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:15.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.631+0000 7fd5dda00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8162b00 0x7fd5d8157aa0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009f60 tx=0x7fd5d0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.632+0000 7fd5d4c00640 1 -- 172.21.15.114:0/2046923554 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0018020 con 0x7fd5d8162b00 2024-03-06T07:33:15.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.632+0000 7fd5d4c00640 1 -- 172.21.15.114:0/2046923554 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d000cdd0 con 0x7fd5d8162b00 2024-03-06T07:33:15.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.632+0000 7fd5d4c00640 1 -- 172.21.15.114:0/2046923554 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00146c0 con 0x7fd5d8162b00 2024-03-06T07:33:15.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.632+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d809cf70 con 0x7fd5d8162b00 2024-03-06T07:33:15.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.632+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d809d530 con 0x7fd5d8162b00 2024-03-06T07:33:15.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.635+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d8068070 con 0x7fd5d8162b00 2024-03-06T07:33:15.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.635+0000 7fd5d4c00640 1 -- 172.21.15.114:0/2046923554 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5d0005d10 con 0x7fd5d8162b00 2024-03-06T07:33:15.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.636+0000 7fd5d4c00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.636+0000 7fd5d4c00640 1 -- 172.21.15.114:0/2046923554 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5d008ae00 con 0x7fd5d8162b00 2024-03-06T07:33:15.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.637+0000 7fd5dd000640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.637+0000 7fd5dd000640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd5b806b4d0 0x7fd5b806d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc036000 tx=0x7fd5cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.645+0000 7fd5d4c00640 1 -- 172.21.15.114:0/2046923554 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5d0022030 con 0x7fd5d8162b00 2024-03-06T07:33:15.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.864+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fc748071340 con 0x7fc7480d0780 2024-03-06T07:33:15.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.884+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f50180d2390 con 0x7f50180b44f0 2024-03-06T07:33:15.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.901+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f3728074520 con 0x7f372815be90 2024-03-06T07:33:15.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.935+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fd5d8156690 con 0x7fd5d8162b00 2024-03-06T07:33:15.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.964+0000 7fc73fe00640 1 -- 172.21.15.114:0/2925408573 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5734 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc734058170 con 0x7fc7480d0780 2024-03-06T07:33:15.965 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 5734 2024-03-06T07:33:15.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc72806b5a0 msgr2=0x7fc72806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc72806b5a0 0x7fc72806da60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc738036000 tx=0x7fc73803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7480d0780 msgr2=0x7fc7481ea550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7480d0780 0x7fc7481ea550 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc73400c4b0 tx=0x7fc73400c970 comp rx=0 tx=0).stop 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 shutdown_connections 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc74815be90 0x7fc7481eaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc72806b5a0 0x7fc72806da60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7480d0780 0x7fc7481ea550 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 --2- 172.21.15.114:0/2925408573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc7480acd60 0x7fc74809cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.967+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 >> 172.21.15.114:0/2925408573 conn(0x7fc74806b360 msgr2=0x7fc74806c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.968+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 shutdown_connections 2024-03-06T07:33:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.968+0000 7fc74ec00640 1 -- 172.21.15.114:0/2925408573 wait complete. 2024-03-06T07:33:15.970 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:33:15.968+0000 7fc8df800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.972+0000 7f500ea00640 1 -- 172.21.15.114:0/3681564300 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7143 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f5008057f60 con 0x7f50180b44f0 2024-03-06T07:33:15.973 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 7143 2024-03-06T07:33:15.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ff806b4d0 msgr2=0x7f4ff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ff806b4d0 0x7f4ff806d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004008040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b44f0 msgr2=0x7f50180b1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b44f0 0x7f50180b1c10 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f500800c500 tx=0x7f500800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 shutdown_connections 2024-03-06T07:33:15.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50180d2cc0 0x7f50180b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f50180b5020 0x7f50180b9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ff806b4d0 0x7f4ff806d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 --2- 172.21.15.114:0/3681564300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50180b44f0 0x7f50180b1c10 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.976+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 >> 172.21.15.114:0/3681564300 conn(0x7f50180d3fd0 msgr2=0x7f50180c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.977+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 shutdown_connections 2024-03-06T07:33:15.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.977+0000 7f501dc00640 1 -- 172.21.15.114:0/3681564300 wait complete. 2024-03-06T07:33:15.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.983+0000 7f3724c00640 1 -- 172.21.15.114:0/3811608777 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5737 v64) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3720010040 con 0x7f372815be90 2024-03-06T07:33:15.985 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 5737 2024-03-06T07:33:15.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f370806b520 msgr2=0x7f370806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f370806b520 0x7f370806d9e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f371c005880 tx=0x7f371c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f372815be90 msgr2=0x7f37280b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f372815be90 0x7f37280b1970 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3720009e50 tx=0x7f372000c490 comp rx=0 tx=0).stop 2024-03-06T07:33:15.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 shutdown_connections 2024-03-06T07:33:15.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f370806b520 0x7f370806d9e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f372815be90 0x7f37280b1970 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37280d0780 0x7f37280cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 --2- 172.21.15.114:0/3811608777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37280acd60 0x7f372813dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 >> 172.21.15.114:0/3811608777 conn(0x7f372806b360 msgr2=0x7f372809cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 shutdown_connections 2024-03-06T07:33:15.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:15.986+0000 7f372f600640 1 -- 172.21.15.114:0/3811608777 wait complete. 2024-03-06T07:33:15.993 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:33:15.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:33:16.000 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:33:16.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:33:16.001 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:33:16.001 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:16.007 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:33:16.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:33:16.014 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:33:16.014 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:16.053 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:33:16.053 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:16.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:16.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:16.155 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:33:16.161 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:33:16.218 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:33:16.976 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:33:16.974+0000 7fc8df800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:16.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.981+0000 7fd5d4c00640 1 -- 172.21.15.114:0/2046923554 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 5722 v65) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd5d0010040 con 0x7fd5d8162b00 2024-03-06T07:33:16.982 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 5722 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd5b806b4d0 msgr2=0x7fd5b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd5b806b4d0 0x7fd5b806d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc036000 tx=0x7fd5cc008030 comp rx=0 tx=0).stop 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8162b00 msgr2=0x7fd5d8157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8162b00 0x7fd5d8157aa0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009f60 tx=0x7fd5d0009f90 comp rx=0 tx=0).stop 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 shutdown_connections 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd5b806b4d0 0x7fd5b806d990 secure :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc036000 tx=0x7fd5cc008030 comp rx=0 tx=0).stop 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5d8162b00 0x7fd5d8157aa0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5d815cc90 0x7fd5d809c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 --2- 172.21.15.114:0/2046923554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5d80d0780 0x7fd5d8166b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.985+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 >> 172.21.15.114:0/2046923554 conn(0x7fd5d806b360 msgr2=0x7fd5d8072260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.986+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 shutdown_connections 2024-03-06T07:33:16.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:16.986+0000 7fd5dfa00640 1 -- 172.21.15.114:0/2046923554 wait complete. 2024-03-06T07:33:17.009 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:33:17.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:33:17.056 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:33:17.056 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:17.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:17.207 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:33:17.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.225+0000 7f9fd3e00640 1 Processor -- start 2024-03-06T07:33:17.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.225+0000 7f9fd3e00640 1 -- start start 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.226+0000 7f9fd3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd4145920 0x7f9fd406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.226+0000 7f9fd3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9fd415faa0 0x7f9fd40cd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.226+0000 7f9fd3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd40cdec0 con 0x7f9fd40cc490 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.226+0000 7f9fd3e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd41601f0 con 0x7f9fd415faa0 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.226+0000 7f9fd3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd41604d0 con 0x7f9fd4145920 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.226+0000 7f9fd2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9fd415faa0 0x7f9fd40cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9fd40cc490 0x7f9fd4071dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:53068/0 (socket says 172.21.15.114:53068) 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd3400640 1 -- 172.21.15.114:0/419031405 learned_addr learned my addr 172.21.15.114:0/419031405 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd2a00640 1 --2- 172.21.15.114:0/419031405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd4145920 0x7f9fd406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd2000640 1 -- 172.21.15.114:0/419031405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd4145920 msgr2=0x7f9fd406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd2000640 1 --2- 172.21.15.114:0/419031405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd4145920 0x7f9fd406c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd2000640 1 -- 172.21.15.114:0/419031405 >> v1:172.21.15.114:6789/0 conn(0x7f9fd40cc490 legacy=0x7f9fd4071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd2000640 1 -- 172.21.15.114:0/419031405 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd41607b0 con 0x7f9fd415faa0 2024-03-06T07:33:17.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd2a00640 1 --2- 172.21.15.114:0/419031405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd4145920 0x7f9fd406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:17.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.227+0000 7f9fd1600640 1 -- 172.21.15.114:0/419031405 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056079337 0 0) 0x7f9fd40cdec0 con 0x7f9fd40cc490 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.228+0000 7f9fd2000640 1 --2- 172.21.15.114:0/419031405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd415faa0 0x7f9fd40cd980 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00097a0 tx=0x7f9fc002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3f38b6024e478f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.228+0000 7f9fd1600640 1 -- 172.21.15.114:0/419031405 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0034070 con 0x7f9fd415faa0 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.228+0000 7f9fd1600640 1 -- 172.21.15.114:0/419031405 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc0030800 con 0x7f9fd415faa0 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.228+0000 7f9fd1600640 1 -- 172.21.15.114:0/419031405 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0030bd0 con 0x7f9fd415faa0 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 -- 172.21.15.114:0/419031405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd415faa0 msgr2=0x7f9fd40cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/419031405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd415faa0 0x7f9fd40cd980 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00097a0 tx=0x7f9fc002b940 comp rx=0 tx=0).stop 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 -- 172.21.15.114:0/419031405 shutdown_connections 2024-03-06T07:33:17.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/419031405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd415faa0 0x7f9fd40cd980 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/419031405 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd4145920 0x7f9fd406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 -- 172.21.15.114:0/419031405 >> 172.21.15.114:0/419031405 conn(0x7f9fd40b1770 msgr2=0x7f9fd40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 -- 172.21.15.114:0/419031405 shutdown_connections 2024-03-06T07:33:17.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.229+0000 7f9fd3e00640 1 -- 172.21.15.114:0/419031405 wait complete. 2024-03-06T07:33:17.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.230+0000 7f9fd3e00640 1 Processor -- start 2024-03-06T07:33:17.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.231+0000 7f9fd3e00640 1 -- start start 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.231+0000 7f9fd3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fd40cc490 0x7f9fd413b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.231+0000 7f9fd3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd4145920 0x7f9fd40ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd415faa0 0x7f9fd40ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40cc920 con 0x7f9fd4145920 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd415fec0 con 0x7f9fd40cc490 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40ccaa0 con 0x7f9fd415faa0 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd4145920 0x7f9fd40ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd4145920 0x7f9fd40ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32862/0 (socket says 172.21.15.114:32862) 2024-03-06T07:33:17.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 -- 172.21.15.114:0/443214754 learned_addr learned my addr 172.21.15.114:0/443214754 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 -- 172.21.15.114:0/443214754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd415faa0 msgr2=0x7f9fd40ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd415faa0 0x7f9fd40ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 -- 172.21.15.114:0/443214754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fd40cc490 msgr2=0x7f9fd413b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fd40cc490 0x7f9fd413b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 -- 172.21.15.114:0/443214754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd40c8df0 con 0x7f9fd4145920 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.232+0000 7f9fd2000640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd4145920 0x7f9fd40ce060 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000c00 tx=0x7f9fc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.233+0000 7f9fcaa00640 1 -- 172.21.15.114:0/443214754 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0047070 con 0x7f9fd4145920 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.233+0000 7f9fcaa00640 1 -- 172.21.15.114:0/443214754 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc0030600 con 0x7f9fd4145920 2024-03-06T07:33:17.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.233+0000 7f9fcaa00640 1 -- 172.21.15.114:0/443214754 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0034040 con 0x7f9fd4145920 2024-03-06T07:33:17.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.233+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd40c9090 con 0x7f9fd4145920 2024-03-06T07:33:17.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.233+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd40c9650 con 0x7f9fd4145920 2024-03-06T07:33:17.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.234+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd404f330 con 0x7f9fd4145920 2024-03-06T07:33:17.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.235+0000 7f9fcaa00640 1 -- 172.21.15.114:0/443214754 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fc0037070 con 0x7f9fd4145920 2024-03-06T07:33:17.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.236+0000 7f9fcaa00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fb406b4d0 0x7f9fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.236+0000 7f9fcaa00640 1 -- 172.21.15.114:0/443214754 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9fc00abc10 con 0x7f9fd4145920 2024-03-06T07:33:17.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.236+0000 7f9fd2a00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fb406b4d0 0x7f9fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.237+0000 7f9fd2a00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fb406b4d0 0x7f9fb406d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4006fd0 tx=0x7f9fc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.240+0000 7f9fcaa00640 1 -- 172.21.15.114:0/443214754 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fc007a940 con 0x7f9fd4145920 2024-03-06T07:33:17.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.458+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fd4144f30 con 0x7f9fd4145920 2024-03-06T07:33:17.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.459+0000 7f9fcaa00640 1 -- 172.21.15.114:0/443214754 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4191 (secure 0 0 0) 0x7f9fc007a240 con 0x7f9fd4145920 2024-03-06T07:33:17.461 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:17.461 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:16.987822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:33:17.461 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 66 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fb406b4d0 msgr2=0x7f9fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fb406b4d0 0x7f9fb406d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4006fd0 tx=0x7f9fc4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd4145920 msgr2=0x7f9fd40ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd4145920 0x7f9fd40ce060 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000c00 tx=0x7f9fc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 shutdown_connections 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9fd415faa0 0x7f9fd40ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fb406b4d0 0x7f9fb406d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd4145920 0x7f9fd40ce060 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 --2- 172.21.15.114:0/443214754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fd40cc490 0x7f9fd413b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 >> 172.21.15.114:0/443214754 conn(0x7f9fd40b1770 msgr2=0x7f9fd415ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 shutdown_connections 2024-03-06T07:33:17.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:17.462+0000 7f9fd3e00640 1 -- 172.21.15.114:0/443214754 wait complete. 2024-03-06T07:33:17.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:05.301366+0000', 'modified': '2024-03-06T07:33:16.987822+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7347}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7347': {'gid': 7347, 'name': 'a', 'rank': 0, 'incarnation': 66, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/1490119219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 1490119219}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 1490119219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:33:17.882 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7347:up:replay 2024-03-06T07:33:18.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.101+0000 7fd035e00640 1 Processor -- start 2024-03-06T07:33:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.101+0000 7fd035e00640 1 -- start start 2024-03-06T07:33:19.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.102+0000 7fd035e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd030155490 0x7fd0300ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd035e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0300d0070 0x7fd0300d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd035e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03015f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd035e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0300ad640 con 0x7fd0300d0070 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd035e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03015f980 con 0x7fd030155490 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd035e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd030155870 con 0x7fd03015ec10 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0300d0070 0x7fd0300d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0300d0070 0x7fd0300d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33390/0 (socket says 172.21.15.114:33390) 2024-03-06T07:33:19.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 -- 172.21.15.114:0/2740429655 learned_addr learned my addr 172.21.15.114:0/2740429655 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02ea00640 1 --2- 172.21.15.114:0/2740429655 >> v2:172.21.15.177:3300/0 conn(0x7fd030155490 0x7fd0300ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02f400640 1 --2- 172.21.15.114:0/2740429655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03015f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 -- 172.21.15.114:0/2740429655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 msgr2=0x7fd03015f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 --2- 172.21.15.114:0/2740429655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03015f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 -- 172.21.15.114:0/2740429655 >> v2:172.21.15.177:3300/0 conn(0x7fd030155490 msgr2=0x7fd0300ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 --2- 172.21.15.114:0/2740429655 >> v2:172.21.15.177:3300/0 conn(0x7fd030155490 0x7fd0300ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02e000640 1 -- 172.21.15.114:0/2740429655 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0300ad870 con 0x7fd0300d0070 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.103+0000 7fd02ea00640 1 --2- 172.21.15.114:0/2740429655 >> v2:172.21.15.177:3300/0 conn(0x7fd030155490 0x7fd0300ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:19.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.104+0000 7fd02e000640 1 --2- 172.21.15.114:0/2740429655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd0300d04f0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd03006ed50 tx=0x7fd01802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5573df40ff4b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd02d600640 1 -- 172.21.15.114:0/2740429655 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01803b070 con 0x7fd0300d0070 2024-03-06T07:33:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd02d600640 1 -- 172.21.15.114:0/2740429655 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd018036440 con 0x7fd0300d0070 2024-03-06T07:33:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd02d600640 1 -- 172.21.15.114:0/2740429655 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018036810 con 0x7fd0300d0070 2024-03-06T07:33:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 -- 172.21.15.114:0/2740429655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 msgr2=0x7fd0300d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 --2- 172.21.15.114:0/2740429655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd0300d04f0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd03006ed50 tx=0x7fd01802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 -- 172.21.15.114:0/2740429655 shutdown_connections 2024-03-06T07:33:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 --2- 172.21.15.114:0/2740429655 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03015f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 --2- 172.21.15.114:0/2740429655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd0300d04f0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 --2- 172.21.15.114:0/2740429655 >> v2:172.21.15.177:3300/0 conn(0x7fd030155490 0x7fd0300ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 -- 172.21.15.114:0/2740429655 >> 172.21.15.114:0/2740429655 conn(0x7fd03006b360 msgr2=0x7fd03006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.105+0000 7fd035e00640 1 -- 172.21.15.114:0/2740429655 shutdown_connections 2024-03-06T07:33:19.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.106+0000 7fd035e00640 1 -- 172.21.15.114:0/2740429655 wait complete. 2024-03-06T07:33:19.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.107+0000 7fd035e00640 1 Processor -- start 2024-03-06T07:33:19.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.107+0000 7fd035e00640 1 -- start start 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd035e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd03015d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd035e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd030155490 0x7fd0300b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd035e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03014eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd035e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0300ae110 con 0x7fd0300d0070 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd035e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0300ade30 con 0x7fd030155490 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd035e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03015f5c0 con 0x7fd03015ec10 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd02ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd03015d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd02ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd03015d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33392/0 (socket says 172.21.15.114:33392) 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.108+0000 7fd02ea00640 1 -- 172.21.15.114:0/230726090 learned_addr learned my addr 172.21.15.114:0/230726090 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02e000640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd030155490 0x7fd0300b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02f400640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03014eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02ea00640 1 -- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 msgr2=0x7fd03014eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02ea00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03014eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02ea00640 1 -- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd030155490 msgr2=0x7fd0300b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02ea00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd030155490 0x7fd0300b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02ea00640 1 -- 172.21.15.114:0/230726090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03015d680 con 0x7fd0300d0070 2024-03-06T07:33:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02e000640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd030155490 0x7fd0300b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd02ea00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd03015d0a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd02000cee0 tx=0x7fd02000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.109+0000 7fd027400640 1 -- 172.21.15.114:0/230726090 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02001a070 con 0x7fd0300d0070 2024-03-06T07:33:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.110+0000 7fd027400640 1 -- 172.21.15.114:0/230726090 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02000d840 con 0x7fd0300d0070 2024-03-06T07:33:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.110+0000 7fd027400640 1 -- 172.21.15.114:0/230726090 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02000dc10 con 0x7fd0300d0070 2024-03-06T07:33:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.110+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03015d960 con 0x7fd0300d0070 2024-03-06T07:33:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.110+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03015df20 con 0x7fd0300d0070 2024-03-06T07:33:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.111+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd030068070 con 0x7fd0300d0070 2024-03-06T07:33:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.112+0000 7fd027400640 1 -- 172.21.15.114:0/230726090 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd02001d070 con 0x7fd0300d0070 2024-03-06T07:33:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.112+0000 7fd027400640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd00c06b4d0 0x7fd00c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.113+0000 7fd027400640 1 -- 172.21.15.114:0/230726090 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd02008ab00 con 0x7fd0300d0070 2024-03-06T07:33:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.113+0000 7fd02e000640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd00c06b4d0 0x7fd00c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.113+0000 7fd02e000640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd00c06b4d0 0x7fd00c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd018038000 tx=0x7fd018032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.117+0000 7fd027400640 1 -- 172.21.15.114:0/230726090 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0200591e0 con 0x7fd0300d0070 2024-03-06T07:33:19.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.352+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd030071f00 con 0x7fd0300d0070 2024-03-06T07:33:19.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.353+0000 7fd027400640 1 -- 172.21.15.114:0/230726090 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd02005d280 con 0x7fd0300d0070 2024-03-06T07:33:19.355 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:19.355 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:18.015780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:33:19.355 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 68 2024-03-06T07:33:19.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.355+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd00c06b4d0 msgr2=0x7fd00c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.355+0000 7fd035e00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd00c06b4d0 0x7fd00c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd018038000 tx=0x7fd018032000 comp rx=0 tx=0).stop 2024-03-06T07:33:19.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.355+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 msgr2=0x7fd03015d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.355+0000 7fd035e00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd03015d0a0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd02000cee0 tx=0x7fd02000a710 comp rx=0 tx=0).stop 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 shutdown_connections 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd03015ec10 0x7fd03014eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd030155490 0x7fd0300b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd00c06b4d0 0x7fd00c06d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 --2- 172.21.15.114:0/230726090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0300d0070 0x7fd03015d0a0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 >> 172.21.15.114:0/230726090 conn(0x7fd03006b360 msgr2=0x7fd03006c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 shutdown_connections 2024-03-06T07:33:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:19.356+0000 7fd035e00640 1 -- 172.21.15.114:0/230726090 wait complete. 2024-03-06T07:33:19.778 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:05.301366+0000', 'modified': '2024-03-06T07:33:18.015780+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7347}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7347': {'gid': 7347, 'name': 'a', 'rank': 0, 'incarnation': 66, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.114:6835/1490119219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 1490119219}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 1490119219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:33:19.778 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7347:up:rejoin 2024-03-06T07:33:20.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.000+0000 7f247cc00640 1 Processor -- start 2024-03-06T07:33:21.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.000+0000 7f247cc00640 1 -- start start 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f247cc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f24780d0780 0x7f24780b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f247cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f24780acd60 0x7f247815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f247cc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24780b5160 con 0x7f247815be90 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f247cc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24780b4f30 con 0x7f24780acd60 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f247cc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24780af240 con 0x7f24780d0780 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f2475e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f247815be90 0x7f24780d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41162/0 (socket says 172.21.15.114:41162) 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f2475e00640 1 -- 172.21.15.114:0/2229876000 learned_addr learned my addr 172.21.15.114:0/2229876000 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.002+0000 7f2476800640 1 --2- 172.21.15.114:0/2229876000 >> v2:172.21.15.114:3300/0 conn(0x7f24780acd60 0x7f247815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.003+0000 7f2475400640 1 --2- 172.21.15.114:0/2229876000 >> v2:172.21.15.177:3300/0 conn(0x7f24780d0780 0x7f24780b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.003+0000 7f2474a00640 1 -- 172.21.15.114:0/2229876000 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023245525 0 0) 0x7f24780b5160 con 0x7f247815be90 2024-03-06T07:33:21.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.003+0000 7f2474a00640 1 -- 172.21.15.114:0/2229876000 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f245c003520 con 0x7f247815be90 2024-03-06T07:33:21.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.003+0000 7f2476800640 1 -- 172.21.15.114:0/2229876000 >> v2:172.21.15.177:3300/0 conn(0x7f24780d0780 msgr2=0x7f24780b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:21.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.003+0000 7f2476800640 1 --2- 172.21.15.114:0/2229876000 >> v2:172.21.15.177:3300/0 conn(0x7f24780d0780 0x7f24780b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.003+0000 7f2476800640 1 -- 172.21.15.114:0/2229876000 >> v1:172.21.15.114:6789/0 conn(0x7f247815be90 legacy=0x7f24780d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.003+0000 7f2476800640 1 -- 172.21.15.114:0/2229876000 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780ae910 con 0x7f24780acd60 2024-03-06T07:33:21.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.004+0000 7f2476800640 1 --2- 172.21.15.114:0/2229876000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780acd60 0x7f247815fb70 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f247806e040 tx=0x7f247002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdf1ad9793cd6ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.004+0000 7f2474a00640 1 -- 172.21.15.114:0/2229876000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470039070 con 0x7f24780acd60 2024-03-06T07:33:21.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.004+0000 7f2474a00640 1 -- 172.21.15.114:0/2229876000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247002cbe0 con 0x7f24780acd60 2024-03-06T07:33:21.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.004+0000 7f2474a00640 1 -- 172.21.15.114:0/2229876000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470034440 con 0x7f24780acd60 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 -- 172.21.15.114:0/2229876000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780acd60 msgr2=0x7f247815fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 --2- 172.21.15.114:0/2229876000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780acd60 0x7f247815fb70 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f247806e040 tx=0x7f247002be30 comp rx=0 tx=0).stop 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 -- 172.21.15.114:0/2229876000 shutdown_connections 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 --2- 172.21.15.114:0/2229876000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780acd60 0x7f247815fb70 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 --2- 172.21.15.114:0/2229876000 >> v2:172.21.15.177:3300/0 conn(0x7f24780d0780 0x7f24780b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 -- 172.21.15.114:0/2229876000 >> 172.21.15.114:0/2229876000 conn(0x7f247806b360 msgr2=0x7f247806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 -- 172.21.15.114:0/2229876000 shutdown_connections 2024-03-06T07:33:21.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.005+0000 7f247cc00640 1 -- 172.21.15.114:0/2229876000 wait complete. 2024-03-06T07:33:21.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.007+0000 7f247cc00640 1 Processor -- start 2024-03-06T07:33:21.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.007+0000 7f247cc00640 1 -- start start 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f247cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24780acd60 0x7f2478152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f247cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780d0780 0x7f2478158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f247cc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247815be90 0x7f247815d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f247cc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24780b2c80 con 0x7f24780d0780 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f247cc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24780aeed0 con 0x7f24780acd60 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f247cc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24780b2e00 con 0x7f247815be90 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f2475400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780d0780 0x7f2478158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f2475400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780d0780 0x7f2478158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33406/0 (socket says 172.21.15.114:33406) 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.008+0000 7f2475400640 1 -- 172.21.15.114:0/264813451 learned_addr learned my addr 172.21.15.114:0/264813451 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f2475400640 1 -- 172.21.15.114:0/264813451 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247815be90 msgr2=0x7f247815d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f2475400640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247815be90 0x7f247815d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f2475400640 1 -- 172.21.15.114:0/264813451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24780acd60 msgr2=0x7f2478152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f2475e00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24780acd60 0x7f2478152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f2475400640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24780acd60 0x7f2478152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f2475400640 1 -- 172.21.15.114:0/264813451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247815c370 con 0x7f24780d0780 2024-03-06T07:33:21.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f2475400640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780d0780 0x7f2478158910 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f246400d800 tx=0x7f246400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f246e000640 1 -- 172.21.15.114:0/264813451 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246400b6c0 con 0x7f24780d0780 2024-03-06T07:33:21.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f246e000640 1 -- 172.21.15.114:0/264813451 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464019070 con 0x7f24780d0780 2024-03-06T07:33:21.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f246e000640 1 -- 172.21.15.114:0/264813451 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246400ba80 con 0x7f24780d0780 2024-03-06T07:33:21.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.009+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247815de00 con 0x7f24780d0780 2024-03-06T07:33:21.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.010+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24780cd070 con 0x7f24780d0780 2024-03-06T07:33:21.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.011+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2478068070 con 0x7f24780d0780 2024-03-06T07:33:21.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.011+0000 7f246e000640 1 -- 172.21.15.114:0/264813451 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f246400bc20 con 0x7f24780d0780 2024-03-06T07:33:21.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.012+0000 7f246e000640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f245806b4d0 0x7f245806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:21.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.012+0000 7f246e000640 1 -- 172.21.15.114:0/264813451 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f246408a770 con 0x7f24780d0780 2024-03-06T07:33:21.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.013+0000 7f2475e00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f245806b4d0 0x7f245806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:21.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.013+0000 7f2475e00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f245806b4d0 0x7f245806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2468000f60 tx=0x7f2468009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:21.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.017+0000 7f246e000640 1 -- 172.21.15.114:0/264813451 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f247815c370 con 0x7f24780d0780 2024-03-06T07:33:21.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.254+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f247809c860 con 0x7f24780d0780 2024-03-06T07:33:21.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.255+0000 7f246e000640 1 -- 172.21.15.114:0/264813451 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f2464059530 con 0x7f24780d0780 2024-03-06T07:33:21.257 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:21.257 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:21.257 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:21.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f245806b4d0 msgr2=0x7f245806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f245806b4d0 0x7f245806d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2468000f60 tx=0x7f2468009040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780d0780 msgr2=0x7f2478158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780d0780 0x7f2478158910 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f246400d800 tx=0x7f246400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 shutdown_connections 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f247815be90 0x7f247815d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f245806b4d0 0x7f245806d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24780d0780 0x7f2478158910 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 --2- 172.21.15.114:0/264813451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24780acd60 0x7f2478152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.257+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 >> 172.21.15.114:0/264813451 conn(0x7f247806b360 msgr2=0x7f24780affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.258+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 shutdown_connections 2024-03-06T07:33:21.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.258+0000 7f247cc00640 1 -- 172.21.15.114:0/264813451 wait complete. 2024-03-06T07:33:21.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:05.301366+0000', 'modified': '2024-03-06T07:33:21.025972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7347}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7347': {'gid': 7347, 'name': 'a', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.114:6835/1490119219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 1490119219}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 1490119219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7347, 'qdb_cluster': [7347]} 2024-03-06T07:33:21.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:33:21.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:33:21.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.897+0000 7f694ac00640 1 Processor -- start 2024-03-06T07:33:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.897+0000 7f694ac00640 1 -- start start 2024-03-06T07:33:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.898+0000 7f694ac00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f69440d0780 0x7f69440ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.898+0000 7f694ac00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f694415ec10 0x7f694415f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.898+0000 7f694ac00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6944155870 con 0x7f6944155490 2024-03-06T07:33:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.898+0000 7f694ac00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69440ad640 con 0x7f69440d0780 2024-03-06T07:33:22.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.898+0000 7f694ac00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f694415f980 con 0x7f694415ec10 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f693b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f69440d0780 0x7f69440ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f693b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f69440d0780 0x7f69440ae630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33414/0 (socket says 172.21.15.114:33414) 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f693b400640 1 -- 172.21.15.114:0/2216238037 learned_addr learned my addr 172.21.15.114:0/2216238037 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f693b400640 1 -- 172.21.15.114:0/2216238037 >> v2:172.21.15.177:3300/0 conn(0x7f694415ec10 msgr2=0x7f694415f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f6948c00640 1 --2- 172.21.15.114:0/2216238037 >> v2:172.21.15.177:3300/0 conn(0x7f694415ec10 0x7f694415f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f693b400640 1 --2- 172.21.15.114:0/2216238037 >> v2:172.21.15.177:3300/0 conn(0x7f694415ec10 0x7f694415f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f693b400640 1 -- 172.21.15.114:0/2216238037 >> v1:172.21.15.177:6789/0 conn(0x7f6944155490 legacy=0x7f69440d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f693b400640 1 -- 172.21.15.114:0/2216238037 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69440ad870 con 0x7f69440d0780 2024-03-06T07:33:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.899+0000 7f6948c00640 1 --2- 172.21.15.114:0/2216238037 >> v2:172.21.15.177:3300/0 conn(0x7f694415ec10 0x7f694415f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:22.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.901+0000 7f693b400640 1 --2- 172.21.15.114:0/2216238037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69440d0780 0x7f69440ae630 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f69440cf860 tx=0x7f692c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe34017217e06a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.901+0000 7f6943400640 1 -- 172.21.15.114:0/2216238037 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c035020 con 0x7f69440d0780 2024-03-06T07:33:22.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.901+0000 7f6943400640 1 -- 172.21.15.114:0/2216238037 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c031870 con 0x7f69440d0780 2024-03-06T07:33:22.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.901+0000 7f6943400640 1 -- 172.21.15.114:0/2216238037 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c031c40 con 0x7f69440d0780 2024-03-06T07:33:22.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 -- 172.21.15.114:0/2216238037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69440d0780 msgr2=0x7f69440ae630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 --2- 172.21.15.114:0/2216238037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69440d0780 0x7f69440ae630 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f69440cf860 tx=0x7f692c02b910 comp rx=0 tx=0).stop 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 -- 172.21.15.114:0/2216238037 shutdown_connections 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 --2- 172.21.15.114:0/2216238037 >> v2:172.21.15.177:3300/0 conn(0x7f694415ec10 0x7f694415f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 --2- 172.21.15.114:0/2216238037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69440d0780 0x7f69440ae630 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 -- 172.21.15.114:0/2216238037 >> 172.21.15.114:0/2216238037 conn(0x7f694406b360 msgr2=0x7f694406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 -- 172.21.15.114:0/2216238037 shutdown_connections 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.902+0000 7f694ac00640 1 -- 172.21.15.114:0/2216238037 wait complete. 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.904+0000 7f694ac00640 1 Processor -- start 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.904+0000 7f694ac00640 1 -- start start 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f694ac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69440d0780 0x7f69440b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f694ac00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6944155490 0x7f69440b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f694ac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694415ec10 0x7f6944141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f694ac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69440ae110 con 0x7f694415ec10 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f694ac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69440ade30 con 0x7f69440d0780 2024-03-06T07:33:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f694ac00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694415f580 con 0x7f6944155490 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694415ec10 0x7f6944141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694415ec10 0x7f6944141410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33430/0 (socket says 172.21.15.114:33430) 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 -- 172.21.15.114:0/1592266805 learned_addr learned my addr 172.21.15.114:0/1592266805 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6943e00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69440d0780 0x7f69440b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f693b400640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6944155490 0x7f69440b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 -- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6944155490 msgr2=0x7f69440b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6944155490 0x7f69440b0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 -- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69440d0780 msgr2=0x7f69440b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69440d0780 0x7f69440b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.905+0000 7f6948c00640 1 -- 172.21.15.114:0/1592266805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69440b3d30 con 0x7f694415ec10 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.906+0000 7f6948c00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694415ec10 0x7f6944141410 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f693c00ccd0 tx=0x7f693c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.906+0000 7f6940c00640 1 -- 172.21.15.114:0/1592266805 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c00f680 con 0x7f694415ec10 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.906+0000 7f6940c00640 1 -- 172.21.15.114:0/1592266805 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693c00f840 con 0x7f694415ec10 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.906+0000 7f6940c00640 1 -- 172.21.15.114:0/1592266805 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c00fc30 con 0x7f694415ec10 2024-03-06T07:33:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.906+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69440b3320 con 0x7f694415ec10 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.906+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69440b2020 con 0x7f694415ec10 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.908+0000 7f6940c00640 1 -- 172.21.15.114:0/1592266805 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f693c0213f0 con 0x7f694415ec10 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.909+0000 7f6940c00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f692006b4d0 0x7f692006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.909+0000 7f6940c00640 1 -- 172.21.15.114:0/1592266805 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f693c08b0a0 con 0x7f694415ec10 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.909+0000 7f6943e00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f692006b4d0 0x7f692006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.910+0000 7f6943e00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f692006b4d0 0x7f692006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6934004ea0 tx=0x7f6934009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.911+0000 7f694d857740 1 Processor -- start 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.911+0000 7f694d857740 1 -- start start 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad62df750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad630f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f694d857740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad62dfc90 0x55cad630fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f694d857740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cad6249600 con 0x55cad62dfc90 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cad61f2930 con 0x55cad630cb90 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cad62456a0 con 0x55cad630c7b0 2024-03-06T07:33:22.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f6948c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad62dfc90 0x55cad630fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f6948c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad62dfc90 0x55cad630fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33438/0 (socket says 172.21.15.114:33438) 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f6948c00640 1 -- 172.21.15.114:0/4116253710 learned_addr learned my addr 172.21.15.114:0/4116253710 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.912+0000 7f6943e00640 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad62df750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6948c00640 1 -- 172.21.15.114:0/4116253710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 msgr2=0x55cad62df750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6948c00640 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad62df750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6948c00640 1 -- 172.21.15.114:0/4116253710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 msgr2=0x55cad630f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6948c00640 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad630f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6948c00640 1 -- 172.21.15.114:0/4116253710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cad5f5e2d0 con 0x55cad62dfc90 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6948c00640 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad62dfc90 0x55cad630fa80 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f693c059710 tx=0x7f693c0598c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511e91761746c7ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6933e00640 1 -- 172.21.15.114:0/4116253710 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c071020 con 0x55cad62dfc90 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6933e00640 1 -- 172.21.15.114:0/4116253710 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693c0711c0 con 0x55cad62dfc90 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.913+0000 7f6933e00640 1 -- 172.21.15.114:0/4116253710 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c055df0 con 0x55cad62dfc90 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 -- 172.21.15.114:0/4116253710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad62dfc90 msgr2=0x55cad630fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad62dfc90 0x55cad630fa80 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f693c059710 tx=0x7f693c0598c0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 -- 172.21.15.114:0/4116253710 shutdown_connections 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad62dfc90 0x55cad630fa80 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad630f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 --2- 172.21.15.114:0/4116253710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad62df750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 -- 172.21.15.114:0/4116253710 >> 172.21.15.114:0/4116253710 conn(0x55cad62e19c0 msgr2=0x55cad632a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 -- 172.21.15.114:0/4116253710 shutdown_connections 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.914+0000 7f694d857740 1 -- 172.21.15.114:0/4116253710 wait complete. 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.915+0000 7f694d857740 1 -- start start 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.915+0000 7f694d857740 1 Processor -- start 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630c7b0 0x55cad630ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f694d857740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad630cb90 0x55cad6308020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad6308560 0x55cad632aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f694d857740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cad62e3f60 con 0x55cad630cb90 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cad6205d90 con 0x55cad630c7b0 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cad62e3350 con 0x55cad6308560 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f694d857740 10 client.0 _ll_register_callbacks cb 0x55cad6241350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f693b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad630cb90 0x55cad6308020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f693b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad630cb90 0x55cad6308020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33448/0 (socket says 172.21.15.114:33448) 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.916+0000 7f693b400640 1 -- 172.21.15.114:0/1008012342 learned_addr learned my addr 172.21.15.114:0/1008012342 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f6943e00640 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630c7b0 0x55cad630ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f6948c00640 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad6308560 0x55cad632aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f693b400640 1 -- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad6308560 msgr2=0x55cad632aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f693b400640 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad6308560 0x55cad632aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f693b400640 1 -- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630c7b0 msgr2=0x55cad630ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f693b400640 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630c7b0 0x55cad630ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f693b400640 1 -- 172.21.15.114:0/1008012342 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cad60286b0 con 0x55cad630cb90 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f693b400640 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad630cb90 0x55cad6308020 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f692c02b950 tx=0x7f692c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11387ed5e33c7f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f6930c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f6930c00640 1 -- 172.21.15.114:0/1008012342 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c035040 con 0x55cad630cb90 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f693c0711c0 con 0x55cad630cb90 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.917+0000 7f6930c00640 1 -- 172.21.15.114:0/1008012342 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f692c03a830 con 0x55cad630cb90 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f6930c00640 1 -- 172.21.15.114:0/1008012342 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f692c04e050 con 0x55cad630cb90 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 10 client.7389 fetch_fsmap learned FSMap version 70 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55cad63059c0 con 0x55cad630cb90 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f6930c00640 1 -- 172.21.15.114:0/1008012342 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693c0711c0 con 0x55cad630cb90 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f6930c00640 1 -- 172.21.15.114:0/1008012342 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 70) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f692c051020 con 0x55cad630cb90 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 10 client.7389 fetch_fsmap finished waiting for FSMap version 70 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 10 client.7389 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 20 client.7389 populate_metadata read hostname 'smithi114' 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad6311920 0x55cad63486e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 4 client.7389 mds_command: new command op to 7347 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 --> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cad6256b40 con 0x55cad6311920 2024-03-06T07:33:22.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.918+0000 7f6943e00640 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad6311920 0x55cad63486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.919+0000 7f6943e00640 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad6311920 0x55cad63486e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9af8223ac540808f server_cookie=5c0bc0251eba205b in_seq=0 out_seq=0 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.919+0000 7f6930c00640 10 client.7389 ms_handle_connect on v2:172.21.15.114:6834/1490119219 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f6930c00640 1 -- 172.21.15.114:0/1008012342 <== mds.0 v2:172.21.15.114:6834/1490119219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cad6256b40 con 0x55cad6311920 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f6930c00640 10 client.7389 handle_command_reply: tid=0 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 client.7389 shutdown 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad630cb90 msgr2=0x55cad6308020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad630cb90 0x55cad6308020 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f692c02b950 tx=0x7f692c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 shutdown_connections 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad6311920 0x55cad63486e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad6308560 0x55cad632aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad630cb90 0x55cad6308020 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 --2- 172.21.15.114:0/1008012342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630c7b0 0x55cad630ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f6930c00640 0 client.7389 ms_handle_reset on v2:172.21.15.114:6834/1490119219 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 >> 172.21.15.114:0/1008012342 conn(0x55cad62dfc90 msgr2=0x55cad62deeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 shutdown_connections 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.920+0000 7f694d857740 1 -- 172.21.15.114:0/1008012342 wait complete. 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.921+0000 7f694d857740 20 client.7389 trim_cache size 0 max 16384 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.956+0000 7f694d857740 1 Processor -- start 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.957+0000 7f694d857740 1 -- start start 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f694d857740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad63394f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad632d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad6391e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f694d857740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441f36f0 con 0x55cad6308560 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944156780 con 0x55cad630cb90 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69441f30f0 con 0x55cad630c7b0 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f6943e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad63394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f6943e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad63394f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33460/0 (socket says 172.21.15.114:33460) 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.958+0000 7f6943e00640 1 -- 172.21.15.114:0/1402375821 learned_addr learned my addr 172.21.15.114:0/1402375821 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f6948c00640 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad6391e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f693b400640 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad632d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f6943e00640 1 -- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 msgr2=0x55cad632d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f6943e00640 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad632d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f6943e00640 1 -- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 msgr2=0x55cad6391e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f6943e00640 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad6391e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f6943e00640 1 -- 172.21.15.114:0/1402375821 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cad6310a50 con 0x55cad6308560 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.959+0000 7f6943e00640 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad63394f0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f6934007e50 tx=0x7f6934014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e89c447b36c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f6933400640 1 -- 172.21.15.114:0/1402375821 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693400ee80 con 0x55cad6308560 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f6933400640 1 -- 172.21.15.114:0/1402375821 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693402c070 con 0x55cad6308560 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f6933400640 1 -- 172.21.15.114:0/1402375821 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934026150 con 0x55cad6308560 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 -- 172.21.15.114:0/1402375821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 msgr2=0x55cad63394f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad63394f0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f6934007e50 tx=0x7f6934014000 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 -- 172.21.15.114:0/1402375821 shutdown_connections 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad6391e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad632d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 --2- 172.21.15.114:0/1402375821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad63394f0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 -- 172.21.15.114:0/1402375821 >> 172.21.15.114:0/1402375821 conn(0x55cad62dfc90 msgr2=0x55cad634d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 -- 172.21.15.114:0/1402375821 shutdown_connections 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.960+0000 7f694d857740 1 -- 172.21.15.114:0/1402375821 wait complete. 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.961+0000 7f694d857740 1 -- start start 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.961+0000 7f694d857740 1 Processor -- start 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.962+0000 7f694d857740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad6312110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.962+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad6312650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.962+0000 7f694d857740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad62de620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.962+0000 7f694d857740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cad63117a0 con 0x55cad6308560 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.962+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cad6305d00 con 0x55cad630cb90 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.962+0000 7f694d857740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cad6344fa0 con 0x55cad630c7b0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.962+0000 7f694d857740 10 client.0 _ll_register_callbacks cb 0x7f692c0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad6312110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad6312110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33470/0 (socket says 172.21.15.114:33470) 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 -- 172.21.15.114:0/2667947905 learned_addr learned my addr 172.21.15.114:0/2667947905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f693b400640 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad6312650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6948c00640 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad62de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 -- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 msgr2=0x55cad6312650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad6312650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 -- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 msgr2=0x55cad62de620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad62de620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 -- 172.21.15.114:0/2667947905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cad632c2d0 con 0x55cad6308560 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6943e00640 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad6312110 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f6934009010 tx=0x7f69340036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cfc3112ae2cb22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.963+0000 7f6930c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f6930c00640 1 -- 172.21.15.114:0/2667947905 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934003c10 con 0x55cad6308560 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f6930c00640 1 -- 172.21.15.114:0/2667947905 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6934003db0 con 0x55cad6308560 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f693402c070 con 0x55cad6308560 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f6930c00640 1 -- 172.21.15.114:0/2667947905 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6934004040 con 0x55cad6308560 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f694d857740 10 client.7395 fetch_fsmap learned FSMap version 70 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55cad62e52f0 con 0x55cad6308560 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f6930c00640 1 -- 172.21.15.114:0/2667947905 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693402c070 con 0x55cad6308560 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.964+0000 7f6930c00640 1 -- 172.21.15.114:0/2667947905 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 70) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6934040020 con 0x55cad6308560 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f694d857740 10 client.7395 fetch_fsmap finished waiting for FSMap version 70 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f694d857740 10 client.7395 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f694d857740 20 client.7395 populate_metadata read hostname 'smithi114' 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f694d857740 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad62df1d0 0x55cad63060d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f694d857740 4 client.7395 mds_command: new command op to 7347 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 --> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cad6250e90 con 0x55cad62df1d0 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f693b400640 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad62df1d0 0x55cad63060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f693b400640 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad62df1d0 0x55cad63060d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b96f21e57884854 server_cookie=9aed4f153813472d in_seq=0 out_seq=0 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.965+0000 7f6930c00640 10 client.7395 ms_handle_connect on v2:172.21.15.114:6834/1490119219 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f6930c00640 1 -- 172.21.15.114:0/2667947905 <== mds.0 v2:172.21.15.114:6834/1490119219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55cad6250e90 con 0x55cad62df1d0 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f6930c00640 10 client.7395 handle_command_reply: tid=0 2024-03-06T07:33:22.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 client.7395 shutdown 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 msgr2=0x55cad6312110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad6312110 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f6934009010 tx=0x7f69340036f0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 shutdown_connections 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cad630cb90 0x55cad62de620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:6834/1490119219,v1:172.21.15.114:6835/1490119219] conn(0x55cad62df1d0 0x55cad63060d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55cad630c7b0 0x55cad6312650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 --2- 172.21.15.114:0/2667947905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cad6308560 0x55cad6312110 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 >> 172.21.15.114:0/2667947905 conn(0x55cad62dfc90 msgr2=0x55cad634d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f6930c00640 0 client.7395 ms_handle_reset on v2:172.21.15.114:6834/1490119219 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 shutdown_connections 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 1 -- 172.21.15.114:0/2667947905 wait complete. 2024-03-06T07:33:22.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.966+0000 7f694d857740 20 client.7395 trim_cache size 0 max 16384 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 7347, 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 4.9692961540000002, 2024-03-06T07:33:22.106 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 70, 2024-03-06T07:33:22.107 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 97, 2024-03-06T07:33:22.107 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 97, 2024-03-06T07:33:22.107 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 5.7459884040000002 2024-03-06T07:33:22.107 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f692006b4d0 msgr2=0x7f692006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f692006b4d0 0x7f692006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6934004ea0 tx=0x7f6934009040 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694415ec10 msgr2=0x7f6944141410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694415ec10 0x7f6944141410 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f693c00ccd0 tx=0x7f693c00a650 comp rx=0 tx=0).stop 2024-03-06T07:33:22.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 shutdown_connections 2024-03-06T07:33:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f692006b4d0 0x7f692006d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694415ec10 0x7f6944141410 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6944155490 0x7f69440b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 --2- 172.21.15.114:0/1592266805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69440d0780 0x7f69440b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 >> 172.21.15.114:0/1592266805 conn(0x7f694406b360 msgr2=0x7f69440726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 shutdown_connections 2024-03-06T07:33:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:21.967+0000 7f694ac00640 1 -- 172.21.15.114:0/1592266805 wait complete. 2024-03-06T07:33:22.109 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> subdir = "delete_me" 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> size_unit = 1024 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> file_multiplier = 200 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> for i in range(0, file_multiplier): 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> f.write(size * '"'"'x'"'"') 2024-03-06T07:33:22.110 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:33:22.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T07:33:22.997+0000 7f3762e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:26.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:33:26.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_ops 2024-03-06T07:33:26.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:26.165 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:33:26.396 INFO:teuthology.orchestra.run.smithi177.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:33:26.396 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:33:26.397 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_files 2024-03-06T07:33:26.624 INFO:teuthology.orchestra.run.smithi177.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:33:26.624 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:33:26.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:26.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.834+0000 7f4e87e00640 1 Processor -- start 2024-03-06T07:33:26.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.834+0000 7f4e87e00640 1 -- start start 2024-03-06T07:33:26.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4e880af200 0x7f4e880af750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e8806b150 0x7f4e8806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e880cffa0 con 0x7f4e8815c020 2024-03-06T07:33:26.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8806bb00 con 0x7f4e8806b150 2024-03-06T07:33:26.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e880d05b0 con 0x7f4e880af200 2024-03-06T07:33:26.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e8806b150 0x7f4e8806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e8806b150 0x7f4e8806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33482/0 (socket says 172.21.15.114:33482) 2024-03-06T07:33:26.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87400640 1 -- 172.21.15.114:0/3501372008 learned_addr learned my addr 172.21.15.114:0/3501372008 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:26.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e86000640 1 --2- 172.21.15.114:0/3501372008 >> v2:172.21.15.177:3300/0 conn(0x7f4e880af200 0x7f4e880af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.836+0000 7f4e87400640 1 -- 172.21.15.114:0/3501372008 >> v2:172.21.15.177:3300/0 conn(0x7f4e880af200 msgr2=0x7f4e880af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.837+0000 7f4e87400640 1 --2- 172.21.15.114:0/3501372008 >> v2:172.21.15.177:3300/0 conn(0x7f4e880af200 0x7f4e880af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.837+0000 7f4e87400640 1 -- 172.21.15.114:0/3501372008 >> v1:172.21.15.177:6789/0 conn(0x7f4e8815c020 legacy=0x7f4e880d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.837+0000 7f4e87400640 1 -- 172.21.15.114:0/3501372008 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e88160710 con 0x7f4e8806b150 2024-03-06T07:33:26.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.838+0000 7f4e87400640 1 --2- 172.21.15.114:0/3501372008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e8806b5c0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f4e8806cd20 tx=0x7f4e8002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf3d961bcbb0f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.838+0000 7f4e85600640 1 -- 172.21.15.114:0/3501372008 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8002cb20 con 0x7f4e8806b150 2024-03-06T07:33:26.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.838+0000 7f4e85600640 1 -- 172.21.15.114:0/3501372008 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8002ccc0 con 0x7f4e8806b150 2024-03-06T07:33:26.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.838+0000 7f4e85600640 1 -- 172.21.15.114:0/3501372008 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e80034610 con 0x7f4e8806b150 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 -- 172.21.15.114:0/3501372008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 msgr2=0x7f4e8806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 --2- 172.21.15.114:0/3501372008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e8806b5c0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f4e8806cd20 tx=0x7f4e8002bf80 comp rx=0 tx=0).stop 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 -- 172.21.15.114:0/3501372008 shutdown_connections 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 --2- 172.21.15.114:0/3501372008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e8806b5c0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 --2- 172.21.15.114:0/3501372008 >> v2:172.21.15.177:3300/0 conn(0x7f4e880af200 0x7f4e880af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 -- 172.21.15.114:0/3501372008 >> 172.21.15.114:0/3501372008 conn(0x7f4e880725d0 msgr2=0x7f4e880729e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 -- 172.21.15.114:0/3501372008 shutdown_connections 2024-03-06T07:33:26.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.839+0000 7f4e87e00640 1 -- 172.21.15.114:0/3501372008 wait complete. 2024-03-06T07:33:26.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.841+0000 7f4e87e00640 1 Processor -- start 2024-03-06T07:33:26.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.841+0000 7f4e87e00640 1 -- start start 2024-03-06T07:33:26.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.842+0000 7f4e87e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e88156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.842+0000 7f4e87e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e880af200 0x7f4e880b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.842+0000 7f4e87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e8815c020 0x7f4e8806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.842+0000 7f4e87e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e881609f0 con 0x7f4e8806b150 2024-03-06T07:33:26.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.842+0000 7f4e87e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8815c400 con 0x7f4e8815c020 2024-03-06T07:33:26.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.842+0000 7f4e87e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e880b4f40 con 0x7f4e880af200 2024-03-06T07:33:26.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e88156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e88156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33494/0 (socket says 172.21.15.114:33494) 2024-03-06T07:33:26.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 -- 172.21.15.114:0/2592204256 learned_addr learned my addr 172.21.15.114:0/2592204256 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e87400640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e8815c020 0x7f4e8806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 -- 172.21.15.114:0/2592204256 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e880af200 msgr2=0x7f4e880b0030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e880af200 0x7f4e880b0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 -- 172.21.15.114:0/2592204256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e8815c020 msgr2=0x7f4e8806e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e8815c020 0x7f4e8806e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.843+0000 7f4e86a00640 1 -- 172.21.15.114:0/2592204256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e880b0570 con 0x7f4e8806b150 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.844+0000 7f4e86a00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e88156c90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800b420 tx=0x7f4e7800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.844+0000 7f4e7ea00640 1 -- 172.21.15.114:0/2592204256 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78011830 con 0x7f4e8806b150 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.844+0000 7f4e7ea00640 1 -- 172.21.15.114:0/2592204256 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e78016070 con 0x7f4e8806b150 2024-03-06T07:33:26.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.844+0000 7f4e7ea00640 1 -- 172.21.15.114:0/2592204256 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78011c10 con 0x7f4e8806b150 2024-03-06T07:33:26.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.844+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e88157550 con 0x7f4e8806b150 2024-03-06T07:33:26.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.845+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8815d0e0 con 0x7f4e8806b150 2024-03-06T07:33:26.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.846+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8815dcf0 con 0x7f4e8806b150 2024-03-06T07:33:26.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.847+0000 7f4e7ea00640 1 -- 172.21.15.114:0/2592204256 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e7801b460 con 0x7f4e8806b150 2024-03-06T07:33:26.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.848+0000 7f4e7ea00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.848+0000 7f4e7ea00640 1 -- 172.21.15.114:0/2592204256 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f4e78089ea0 con 0x7f4e8806b150 2024-03-06T07:33:26.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.848+0000 7f4e86000640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.850+0000 7f4e86000640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e6806b4d0 0x7f4e6806d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4e881f79e0 tx=0x7f4e74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:26.855+0000 7f4e7ea00640 1 -- 172.21.15.114:0/2592204256 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e78058d00 con 0x7f4e8806b150 2024-03-06T07:33:27.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.129+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e8804f330 con 0x7f4e8806b150 2024-03-06T07:33:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.130+0000 7f4e7ea00640 1 -- 172.21.15.114:0/2592204256 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f4e78058600 con 0x7f4e8806b150 2024-03-06T07:33:27.131 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:27.132 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e6806b4d0 msgr2=0x7f4e6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e6806b4d0 0x7f4e6806d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4e881f79e0 tx=0x7f4e74008040 comp rx=0 tx=0).stop 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 msgr2=0x7f4e88156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e88156c90 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800b420 tx=0x7f4e7800b8e0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 shutdown_connections 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e8815c020 0x7f4e8806e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e880af200 0x7f4e880b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 --2- 172.21.15.114:0/2592204256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e8806b150 0x7f4e88156c90 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 >> 172.21.15.114:0/2592204256 conn(0x7f4e880725d0 msgr2=0x7f4e880ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 shutdown_connections 2024-03-06T07:33:27.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:27.133+0000 7f4e87e00640 1 -- 172.21.15.114:0/2592204256 wait complete. 2024-03-06T07:33:27.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:27.442 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":25,"msgr_send_messages":22,"msgr_recv_bytes":108004,"msgr_send_bytes":119164,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.011619476,"msgr_running_send_time":0.002287067,"msgr_running_recv_time":0.007871183,"msgr_running_fast_dispatch_time":0.000272353,"msgr_send_messages_queue_lat":{"avgcount":22,"sum":0.005903887,"avgtime":0.000268358},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":107472,"msgr_send_encrypted_bytes":96528},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6244,"msgr_send_messages":12453,"msgr_recv_bytes":2261411,"msgr_send_bytes":7580576,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.307249550,"msgr_running_send_time":0.879948960,"msgr_running_recv_time":0.989027921,"msgr_running_fast_dispatch_time":0.003997047,"msgr_send_messages_queue_lat":{"avgcount":12452,"sum":0.459787505,"avgtime":0.000036924},"msgr_handle_ack_lat":{"avgcount":14297,"sum":0.007180339,"avgtime":0.000000502},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":38,"msgr_send_messages":40,"msgr_recv_bytes":16730,"msgr_send_bytes":9029652,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.047063087,"msgr_running_send_time":0.033311310,"msgr_running_recv_time":0.004630264,"msgr_running_fast_dispatch_time":0.008423860,"msgr_send_messages_queue_lat":{"avgcount":38,"sum":0.013384260,"avgtime":0.000352217},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":677,"complete_latency":{"avgcount":68,"sum":2.600804792,"avgtime":0.038247129}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000182847,"avgtime":0.000045711}},"mds":{"request":3206,"reply":3204,"reply_latency":{"avgcount":3204,"sum":691.186265401,"avgtime":0.215726050},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3011,"root_rbytes":20715520,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3199,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2994,"handle_client_caps_dirty":2992,"handle_client_cap_release":0,"process_request_cap_release":3},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6202,"evex":0,"evtrm":0,"ev":6205,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":17170625,"rdpos":4195244,"jlat":{"avgcount":11,"sum":0.800845741,"avgtime":0.072804158},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3211,"dn+":3211,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":99584,"heap":248164},"mds_server":{"dispatch_client_request":3206,"dispatch_server_request":0,"handle_client_request":3206,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":1,"sum":0.527307976,"avgtime":0.527307976},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3091,"average_load":3091,"avg_session_uptime":971,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18850340,"buffer_anon_items":10128,"buffer_meta_bytes":97328,"buffer_meta_items":1106,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12838836,"mds_co_items":125724,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":63,"op_send_bytes":13209447,"op_resend":0,"op_reply":63,"op_latency":{"avgcount":63,"sum":0.337992970,"avgtime":0.005364967},"op_inflight":0,"oplen_avg":{"avgcount":63,"sum":114},"op":63,"op_r":36,"op_w":27,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":11,"omap_rd":48,"omap_del":0},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":2412,"max":104857600,"get_started":0,"get":6303,"get_sum":1816213,"get_or_fail_fail":0,"get_or_fail_success":6303,"take":0,"take_sum":0,"put":6294,"put_sum":1813801,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":63,"take_sum":13210381,"put":63,"put_sum":13210381,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":63,"take_sum":63,"put":63,"put_sum":63,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":2621,"max":3758096384,"get_started":0,"get":6202,"get_sum":12975123,"get_or_fail_fail":0,"get_or_fail_success":6202,"take":0,"take_sum":0,"put":11,"put_sum":12972502,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:27.443 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 108004, "msgr_recv_encrypted_bytes": 107472, "msgr_recv_messages": 25, "msgr_running_fast_dispatch_time": 0.000272353, "msgr_running_recv_time": 0.007871183, "msgr_running_send_time": 0.002287067, "msgr_running_total_time": 0.011619476, "msgr_send_bytes": 119164, "msgr_send_encrypted_bytes": 96528, "msgr_send_messages": 22, "msgr_send_messages_queue_lat": { "avgcount": 22, "avgtime": 0.000268358, "sum": 0.005903887 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 14297, "avgtime": 5.02e-07, "sum": 0.007180339 }, "msgr_recv_bytes": 2261411, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 6244, "msgr_running_fast_dispatch_time": 0.003997047, "msgr_running_recv_time": 0.989027921, "msgr_running_send_time": 0.87994896, "msgr_running_total_time": 1.30724955, "msgr_send_bytes": 7580576, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 12453, "msgr_send_messages_queue_lat": { "avgcount": 12452, "avgtime": 3.6924e-05, "sum": 0.459787505 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 16730, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 38, "msgr_running_fast_dispatch_time": 0.00842386, "msgr_running_recv_time": 0.004630264, "msgr_running_send_time": 0.03331131, "msgr_running_total_time": 0.047063087, "msgr_send_bytes": 9029652, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 40, "msgr_send_messages_queue_lat": { "avgcount": 38, "avgtime": 0.000352217, "sum": 0.01338426 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 68, "avgtime": 0.038247129, "sum": 2.600804792 }, "queue_len": 677 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 4.5711e-05, "sum": 0.000182847 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3199, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2994, "handle_client_caps_dirty": 2992, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3, "q": 0, "reply": 3204, "reply_latency": { "avgcount": 3204, "avgtime": 0.21572605, "sum": 691.186265401 }, "request": 3206, "root_rbytes": 20715520, "root_rfiles": 3011, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6205, "evadd": 6202, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.072804158, "sum": 0.800845741 }, "rdpos": 4195244, "replayed": 3, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 17170625 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3211, "dn+": 3211, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 99584 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3206, "dispatch_server_request": 0, "handle_client_request": 3206, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 1, "avgtime": 0.527307976, "sum": 0.527307976 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3091, "avg_session_uptime": 971, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3091 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18850340, "buffer_anon_items": 10128, "buffer_meta_bytes": 97328, "buffer_meta_items": 1106, "mds_co_bytes": 12838836, "mds_co_items": 125724, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 0, "omap_rd": 48, "omap_wr": 11, "op": 63, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 63, "avgtime": 0.005364967, "sum": 0.33799297 }, "op_pg": 0, "op_r": 36, "op_reply": 63, "op_resend": 0, "op_rmw": 0, "op_send": 63, "op_send_bytes": 13209447, "op_w": 27, "oplen_avg": { "avgcount": 63, "sum": 114 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6303, "get_or_fail_fail": 0, "get_or_fail_success": 6303, "get_started": 0, "get_sum": 1816213, "max": 104857600, "put": 6294, "put_sum": 1813801, "take": 0, "take_sum": 0, "val": 2412, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 63, "put_sum": 13210381, "take": 63, "take_sum": 13210381, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 63, "put_sum": 63, "take": 63, "take_sum": 63, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 6202, "get_or_fail_fail": 0, "get_or_fail_success": 6202, "get_started": 0, "get_sum": 12975123, "max": 3758096384, "put": 11, "put_sum": 12972502, "take": 0, "take_sum": 0, "val": 2621, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:27.444 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/0, 0/0 2024-03-06T07:33:28.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:28.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.668+0000 7f6313200640 1 Processor -- start 2024-03-06T07:33:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.668+0000 7f6313200640 1 -- start start 2024-03-06T07:33:28.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.670+0000 7f6313200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f631415e230 0x7f63140d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.670+0000 7f6313200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f63140d13a0 0x7f63140d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.670+0000 7f6313200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63140b39d0 con 0x7f63140d2940 2024-03-06T07:33:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.670+0000 7f6313200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63140b3780 con 0x7f63140d13a0 2024-03-06T07:33:28.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.670+0000 7f6313200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63140b3cb0 con 0x7f631415e230 2024-03-06T07:33:28.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.671+0000 7f6311e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f631415e230 0x7f63140d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.671+0000 7f6311400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f63140d13a0 0x7f63140d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.671+0000 7f6312800640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f63140d2940 0x7f63140ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:44386/0 (socket says 172.21.15.114:44386) 2024-03-06T07:33:28.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.671+0000 7f6312800640 1 -- 172.21.15.114:0/2560902415 learned_addr learned my addr 172.21.15.114:0/2560902415 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:28.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.672+0000 7f6311400640 1 -- 172.21.15.114:0/2560902415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f631415e230 msgr2=0x7f63140d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.672+0000 7f6311400640 1 --2- 172.21.15.114:0/2560902415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f631415e230 0x7f63140d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.672+0000 7f6311400640 1 -- 172.21.15.114:0/2560902415 >> v1:172.21.15.177:6789/0 conn(0x7f63140d2940 legacy=0x7f63140ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.672+0000 7f6311400640 1 -- 172.21.15.114:0/2560902415 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63140b3f90 con 0x7f63140d13a0 2024-03-06T07:33:28.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.674+0000 7f6311400640 1 --2- 172.21.15.114:0/2560902415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d13a0 0x7f63140d2400 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f63000097a0 tx=0x7f630002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ecef2002dccf0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.674+0000 7f6310a00640 1 -- 172.21.15.114:0/2560902415 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300034070 con 0x7f63140d13a0 2024-03-06T07:33:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.674+0000 7f6310a00640 1 -- 172.21.15.114:0/2560902415 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6300030800 con 0x7f63140d13a0 2024-03-06T07:33:28.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.674+0000 7f6310a00640 1 -- 172.21.15.114:0/2560902415 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300030bd0 con 0x7f63140d13a0 2024-03-06T07:33:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 -- 172.21.15.114:0/2560902415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d13a0 msgr2=0x7f63140d2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 --2- 172.21.15.114:0/2560902415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d13a0 0x7f63140d2400 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f63000097a0 tx=0x7f630002b940 comp rx=0 tx=0).stop 2024-03-06T07:33:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 -- 172.21.15.114:0/2560902415 shutdown_connections 2024-03-06T07:33:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 --2- 172.21.15.114:0/2560902415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d13a0 0x7f63140d2400 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 --2- 172.21.15.114:0/2560902415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f631415e230 0x7f63140d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 -- 172.21.15.114:0/2560902415 >> 172.21.15.114:0/2560902415 conn(0x7f63140d3e20 msgr2=0x7f63140d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 -- 172.21.15.114:0/2560902415 shutdown_connections 2024-03-06T07:33:28.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.675+0000 7f6313200640 1 -- 172.21.15.114:0/2560902415 wait complete. 2024-03-06T07:33:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.677+0000 7f6313200640 1 Processor -- start 2024-03-06T07:33:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.677+0000 7f6313200640 1 -- start start 2024-03-06T07:33:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.678+0000 7f6313200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63140d13a0 0x7f63140ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.678+0000 7f6313200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d2940 0x7f631415e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.678+0000 7f6313200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f631415e230 0x7f631415ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.678+0000 7f6313200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63140d1d90 con 0x7f631415e230 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.678+0000 7f6313200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631415e610 con 0x7f63140d2940 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.678+0000 7f6313200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63140d1f10 con 0x7f63140d13a0 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f631415e230 0x7f631415ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f631415e230 0x7f631415ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54534/0 (socket says 172.21.15.114:54534) 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 -- 172.21.15.114:0/4047798483 learned_addr learned my addr 172.21.15.114:0/4047798483 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6311400640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d2940 0x7f631415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6311e00640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63140d13a0 0x7f63140ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 -- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63140d13a0 msgr2=0x7f63140ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63140d13a0 0x7f63140ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 -- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d2940 msgr2=0x7f631415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d2940 0x7f631415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6312800640 1 -- 172.21.15.114:0/4047798483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63140cb7f0 con 0x7f631415e230 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.679+0000 7f6311400640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d2940 0x7f631415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.680+0000 7f6312800640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f631415e230 0x7f631415ed60 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f630c00cbc0 tx=0x7f630c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.680+0000 7f630a000640 1 -- 172.21.15.114:0/4047798483 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c017070 con 0x7f631415e230 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.680+0000 7f630a000640 1 -- 172.21.15.114:0/4047798483 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c00ad40 con 0x7f631415e230 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.680+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63140cbad0 con 0x7f631415e230 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.680+0000 7f630a000640 1 -- 172.21.15.114:0/4047798483 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c012600 con 0x7f631415e230 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.680+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63140b8000 con 0x7f631415e230 2024-03-06T07:33:28.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.681+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631404f330 con 0x7f631415e230 2024-03-06T07:33:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.683+0000 7f630a000640 1 -- 172.21.15.114:0/4047798483 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f630c01a070 con 0x7f631415e230 2024-03-06T07:33:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.684+0000 7f630a000640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f62f406b4d0 0x7f62f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.684+0000 7f630a000640 1 -- 172.21.15.114:0/4047798483 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f630c089e80 con 0x7f631415e230 2024-03-06T07:33:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.684+0000 7f6311e00640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f62f406b4d0 0x7f62f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.685+0000 7f6311e00640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f62f406b4d0 0x7f62f406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6304009a30 tx=0x7f6304008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:28.689+0000 7f630a000640 1 -- 172.21.15.114:0/4047798483 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f630c058690 con 0x7f631415e230 2024-03-06T07:33:29.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.033+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f631415d610 con 0x7f631415e230 2024-03-06T07:33:29.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.034+0000 7f630a000640 1 -- 172.21.15.114:0/4047798483 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f630c05c730 con 0x7f631415e230 2024-03-06T07:33:29.037 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:29.037 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.036+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f62f406b4d0 msgr2=0x7f62f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.036+0000 7f6313200640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f62f406b4d0 0x7f62f406d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6304009a30 tx=0x7f6304008040 comp rx=0 tx=0).stop 2024-03-06T07:33:29.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.036+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f631415e230 msgr2=0x7f631415ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.036+0000 7f6313200640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f631415e230 0x7f631415ed60 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f630c00cbc0 tx=0x7f630c00a430 comp rx=0 tx=0).stop 2024-03-06T07:33:29.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 shutdown_connections 2024-03-06T07:33:29.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f62f406b4d0 0x7f62f406d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f631415e230 0x7f631415ed60 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63140d2940 0x7f631415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 --2- 172.21.15.114:0/4047798483 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f63140d13a0 0x7f63140ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 >> 172.21.15.114:0/4047798483 conn(0x7f63140d3e20 msgr2=0x7f631406bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 shutdown_connections 2024-03-06T07:33:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:29.037+0000 7f6313200640 1 -- 172.21.15.114:0/4047798483 wait complete. 2024-03-06T07:33:29.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:29.295 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":26,"msgr_send_messages":24,"msgr_recv_bytes":108276,"msgr_send_bytes":121580,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.011897572,"msgr_running_send_time":0.002462999,"msgr_running_recv_time":0.007914523,"msgr_running_fast_dispatch_time":0.000291262,"msgr_send_messages_queue_lat":{"avgcount":24,"sum":0.005970562,"avgtime":0.000248773},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":107744,"msgr_send_encrypted_bytes":98944},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7771,"msgr_send_messages":13627,"msgr_recv_bytes":2793898,"msgr_send_bytes":9738662,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.548657770,"msgr_running_send_time":1.024085861,"msgr_running_recv_time":1.250065623,"msgr_running_fast_dispatch_time":0.004127896,"msgr_send_messages_queue_lat":{"avgcount":13626,"sum":0.557731256,"avgtime":0.000040931},"msgr_handle_ack_lat":{"avgcount":16896,"sum":0.008266515,"avgtime":0.000000489},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":41,"msgr_send_messages":43,"msgr_recv_bytes":17560,"msgr_send_bytes":12884275,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.062697626,"msgr_running_send_time":0.046143397,"msgr_running_recv_time":0.004707256,"msgr_running_fast_dispatch_time":0.011119535,"msgr_send_messages_queue_lat":{"avgcount":41,"sum":0.014573900,"avgtime":0.000355460},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1418,"complete_latency":{"avgcount":73,"sum":3.004429019,"avgtime":0.041156561}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000182847,"avgtime":0.000045711}},"mds":{"request":4139,"reply":4137,"reply_latency":{"avgcount":4137,"sum":692.723997591,"avgtime":0.167445974},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":4140,"traverse_hit":4140,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3011,"root_rbytes":20715520,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":3266,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3594,"handle_client_caps_dirty":2996,"handle_client_cap_release":0,"process_request_cap_release":936},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":928,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":928,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7749,"evex":0,"evtrm":0,"ev":7752,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":21355008,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.857425093,"avgtime":0.071452091},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4139,"dn+":4139,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":103680,"heap":248164},"mds_server":{"dispatch_client_request":4140,"dispatch_server_request":0,"handle_client_request":4140,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":928,"sum":1.101942148,"avgtime":0.001187437},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3949,"average_load":3949,"avg_session_uptime":971,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18850340,"buffer_anon_items":10128,"buffer_meta_bytes":97328,"buffer_meta_items":1106,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12838836,"mds_co_items":125724,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":67,"op_send_bytes":17063032,"op_resend":0,"op_reply":67,"op_latency":{"avgcount":67,"sum":0.401991639,"avgtime":0.005999875},"op_inflight":0,"oplen_avg":{"avgcount":67,"sum":120},"op":67,"op_r":36,"op_w":31,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":12,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":9,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":14,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":7835,"get_sum":2212213,"get_or_fail_fail":0,"get_or_fail_success":7835,"take":0,"take_sum":0,"put":7834,"put_sum":2211960,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":67,"take_sum":17063966,"put":67,"put_sum":17063966,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":67,"take_sum":67,"put":67,"put_sum":67,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":383488,"max":3758096384,"get_started":0,"get":7749,"get_sum":17159506,"get_or_fail_fail":0,"get_or_fail_success":7749,"take":0,"take_sum":0,"put":12,"put_sum":16776018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:29.297 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 108276, "msgr_recv_encrypted_bytes": 107744, "msgr_recv_messages": 26, "msgr_running_fast_dispatch_time": 0.000291262, "msgr_running_recv_time": 0.007914523, "msgr_running_send_time": 0.002462999, "msgr_running_total_time": 0.011897572, "msgr_send_bytes": 121580, "msgr_send_encrypted_bytes": 98944, "msgr_send_messages": 24, "msgr_send_messages_queue_lat": { "avgcount": 24, "avgtime": 0.000248773, "sum": 0.005970562 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 16896, "avgtime": 4.89e-07, "sum": 0.008266515 }, "msgr_recv_bytes": 2793898, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 7771, "msgr_running_fast_dispatch_time": 0.004127896, "msgr_running_recv_time": 1.250065623, "msgr_running_send_time": 1.024085861, "msgr_running_total_time": 1.54865777, "msgr_send_bytes": 9738662, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 13627, "msgr_send_messages_queue_lat": { "avgcount": 13626, "avgtime": 4.0931e-05, "sum": 0.557731256 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 17560, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 41, "msgr_running_fast_dispatch_time": 0.011119535, "msgr_running_recv_time": 0.004707256, "msgr_running_send_time": 0.046143397, "msgr_running_total_time": 0.062697626, "msgr_send_bytes": 12884275, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 43, "msgr_send_messages_queue_lat": { "avgcount": 41, "avgtime": 0.00035546, "sum": 0.0145739 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 73, "avgtime": 0.041156561, "sum": 3.004429019 }, "queue_len": 1418 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 4.5711e-05, "sum": 0.000182847 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3266, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3594, "handle_client_caps_dirty": 2996, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 936, "q": 0, "reply": 4137, "reply_latency": { "avgcount": 4137, "avgtime": 0.167445974, "sum": 692.723997591 }, "request": 4139, "root_rbytes": 20715520, "root_rfiles": 3011, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4140, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4140, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 928, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 928, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7752, "evadd": 7749, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.071452091, "sum": 0.857425093 }, "rdpos": 4195244, "replayed": 3, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 21355008 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4139, "dn+": 4139, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 103680 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4140, "dispatch_server_request": 0, "handle_client_request": 4140, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 928, "avgtime": 0.001187437, "sum": 1.101942148 } }, "mds_sessions": { "average_load": 3949, "avg_session_uptime": 971, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3949 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18850340, "buffer_anon_items": 10128, "buffer_meta_bytes": 97328, "buffer_meta_items": 1106, "mds_co_bytes": 12838836, "mds_co_items": 125724, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 14, "op": 67, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 67, "avgtime": 0.005999875, "sum": 0.401991639 }, "op_pg": 0, "op_r": 36, "op_reply": 67, "op_resend": 0, "op_rmw": 0, "op_send": 67, "op_send_bytes": 17063032, "op_w": 31, "oplen_avg": { "avgcount": 67, "sum": 120 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 9, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 12, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7835, "get_or_fail_fail": 0, "get_or_fail_success": 7835, "get_started": 0, "get_sum": 2212213, "max": 104857600, "put": 7834, "put_sum": 2211960, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 67, "put_sum": 17063966, "take": 67, "take_sum": 17063966, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 67, "put_sum": 67, "take": 67, "take_sum": 67, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 7749, "get_or_fail_fail": 0, "get_or_fail_success": 7749, "get_started": 0, "get_sum": 17159506, "max": 3758096384, "put": 12, "put_sum": 16776018, "take": 0, "take_sum": 0, "val": 383488, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:29.297 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/928, 0/928 2024-03-06T07:33:30.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:30.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.532+0000 7f2aa4a00640 1 Processor -- start 2024-03-06T07:33:30.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.532+0000 7f2aa4a00640 1 -- start start 2024-03-06T07:33:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.533+0000 7f2aa4a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa015e2d0 0x7f2aa00d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.533+0000 7f2aa4a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00cebd0 con 0x7f2aa015e6b0 2024-03-06T07:33:30.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.534+0000 7f2aa4a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00d23e0 con 0x7f2aa00d2db0 2024-03-06T07:33:30.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.534+0000 7f2aa4a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d06f0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.534+0000 7f2a9d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2aa015e6b0 0x7f2aa00d26a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43228/0 (socket says 172.21.15.114:43228) 2024-03-06T07:33:30.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.534+0000 7f2a9d400640 1 -- 172.21.15.114:0/40637132 learned_addr learned my addr 172.21.15.114:0/40637132 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:30.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.534+0000 7f2a9de00640 1 --2- 172.21.15.114:0/40637132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa015e2d0 0x7f2aa00d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.534+0000 7f2a9ca00640 1 -- 172.21.15.114:0/40637132 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585202605 0 0) 0x7f2a8c001110 con 0x7f2aa015e6b0 2024-03-06T07:33:30.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.535+0000 7f2a9ca00640 1 -- 172.21.15.114:0/40637132 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a84003520 con 0x7f2aa015e6b0 2024-03-06T07:33:30.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.535+0000 7f2a9de00640 1 -- 172.21.15.114:0/40637132 >> v1:172.21.15.177:6789/0 conn(0x7f2aa00d2db0 legacy=0x7f2aa00cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.535+0000 7f2a9de00640 1 -- 172.21.15.114:0/40637132 >> v1:172.21.15.114:6789/0 conn(0x7f2aa015e6b0 legacy=0x7f2aa00d26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.535+0000 7f2a9de00640 1 -- 172.21.15.114:0/40637132 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00d09d0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.536+0000 7f2a9de00640 1 --2- 172.21.15.114:0/40637132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa015e2d0 0x7f2aa00d1ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2a900097d0 tx=0x7f2a9002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41aaff827c2eed2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.537+0000 7f2a9ca00640 1 -- 172.21.15.114:0/40637132 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90034070 con 0x7f2aa015e2d0 2024-03-06T07:33:30.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.537+0000 7f2a9ca00640 1 -- 172.21.15.114:0/40637132 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a90030800 con 0x7f2aa015e2d0 2024-03-06T07:33:30.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.537+0000 7f2a9ca00640 1 -- 172.21.15.114:0/40637132 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90030bd0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.538+0000 7f2aa4a00640 1 -- 172.21.15.114:0/40637132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa015e2d0 msgr2=0x7f2aa00d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.538+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/40637132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa015e2d0 0x7f2aa00d1ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2a900097d0 tx=0x7f2a9002b940 comp rx=0 tx=0).stop 2024-03-06T07:33:30.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.538+0000 7f2aa4a00640 1 -- 172.21.15.114:0/40637132 shutdown_connections 2024-03-06T07:33:30.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.538+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/40637132 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa015e2d0 0x7f2aa00d1ea0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.538+0000 7f2aa4a00640 1 -- 172.21.15.114:0/40637132 >> 172.21.15.114:0/40637132 conn(0x7f2aa009d390 msgr2=0x7f2aa00d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.538+0000 7f2aa4a00640 1 -- 172.21.15.114:0/40637132 shutdown_connections 2024-03-06T07:33:30.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.538+0000 7f2aa4a00640 1 -- 172.21.15.114:0/40637132 wait complete. 2024-03-06T07:33:30.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.540+0000 7f2aa4a00640 1 Processor -- start 2024-03-06T07:33:30.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.540+0000 7f2aa4a00640 1 -- start start 2024-03-06T07:33:30.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2aa4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aa00d2db0 0x7f2aa00cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2aa4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa015e2d0 0x7f2aa01e8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2aa4a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa00b2390 0x7f2aa0151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2aa4a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d1270 con 0x7f2aa015e2d0 2024-03-06T07:33:30.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2aa4a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00d0f90 con 0x7f2aa00d2db0 2024-03-06T07:33:30.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2aa4a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00b3660 con 0x7f2aa00b2390 2024-03-06T07:33:30.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2a9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa015e2d0 0x7f2aa01e8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2a9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa015e2d0 0x7f2aa01e8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54540/0 (socket says 172.21.15.114:54540) 2024-03-06T07:33:30.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.541+0000 7f2a9d400640 1 -- 172.21.15.114:0/944711309 learned_addr learned my addr 172.21.15.114:0/944711309 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:30.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9de00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aa00d2db0 0x7f2aa00cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9e800640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa00b2390 0x7f2aa0151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9d400640 1 -- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa00b2390 msgr2=0x7f2aa0151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9d400640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa00b2390 0x7f2aa0151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9d400640 1 -- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aa00d2db0 msgr2=0x7f2aa00cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9d400640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aa00d2db0 0x7f2aa00cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9d400640 1 -- 172.21.15.114:0/944711309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00cc4d0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9de00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aa00d2db0 0x7f2aa00cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.542+0000 7f2a9d400640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa015e2d0 0x7f2aa01e8010 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c730 tx=0x7f2a8c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.543+0000 7f2a96000640 1 -- 172.21.15.114:0/944711309 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c017070 con 0x7f2aa015e2d0 2024-03-06T07:33:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.543+0000 7f2a96000640 1 -- 172.21.15.114:0/944711309 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c005ab0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.543+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa00cc7b0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.543+0000 7f2a96000640 1 -- 172.21.15.114:0/944711309 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c007070 con 0x7f2aa015e2d0 2024-03-06T07:33:30.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.543+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa00ccd70 con 0x7f2aa015e2d0 2024-03-06T07:33:30.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.545+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa0068070 con 0x7f2aa015e2d0 2024-03-06T07:33:30.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.546+0000 7f2a96000640 1 -- 172.21.15.114:0/944711309 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a8c01a070 con 0x7f2aa015e2d0 2024-03-06T07:33:30.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.547+0000 7f2a96000640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8006b520 0x7f2a8006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.548+0000 7f2a96000640 1 -- 172.21.15.114:0/944711309 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f2a8c089ae0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.549+0000 7f2a9de00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8006b520 0x7f2a8006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.550+0000 7f2a9de00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8006b520 0x7f2a8006d9e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2a90036000 tx=0x7f2a9003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.554+0000 7f2a96000640 1 -- 172.21.15.114:0/944711309 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a8c021020 con 0x7f2aa015e2d0 2024-03-06T07:33:30.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.900+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa015d990 con 0x7f2aa015e2d0 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.901+0000 7f2a96000640 1 -- 172.21.15.114:0/944711309 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f2aa00cebd0 con 0x7f2aa015e2d0 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8006b520 msgr2=0x7f2a8006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8006b520 0x7f2a8006d9e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2a90036000 tx=0x7f2a9003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa015e2d0 msgr2=0x7f2aa01e8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa015e2d0 0x7f2aa01e8010 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c730 tx=0x7f2a8c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 shutdown_connections 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2aa00b2390 0x7f2aa0151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2a8006b520 0x7f2a8006d9e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa015e2d0 0x7f2aa01e8010 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 --2- 172.21.15.114:0/944711309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2aa00d2db0 0x7f2aa00cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 >> 172.21.15.114:0/944711309 conn(0x7f2aa009d390 msgr2=0x7f2aa00b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.904+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 shutdown_connections 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:30.905+0000 7f2aa4a00640 1 -- 172.21.15.114:0/944711309 wait complete. 2024-03-06T07:33:30.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:31.189 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":32,"msgr_send_messages":30,"msgr_recv_bytes":119953,"msgr_send_bytes":141769,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.012645948,"msgr_running_send_time":0.002833878,"msgr_running_recv_time":0.008076106,"msgr_running_fast_dispatch_time":0.000473206,"msgr_send_messages_queue_lat":{"avgcount":30,"sum":0.006392068,"avgtime":0.000213068},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":107744,"msgr_send_encrypted_bytes":98944},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8590,"msgr_send_messages":15629,"msgr_recv_bytes":3078480,"msgr_send_bytes":10484926,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.797116642,"msgr_running_send_time":1.167713238,"msgr_running_recv_time":1.383695095,"msgr_running_fast_dispatch_time":0.004322823,"msgr_send_messages_queue_lat":{"avgcount":15628,"sum":0.644264177,"avgtime":0.000041224},"msgr_handle_ack_lat":{"avgcount":19205,"sum":0.009140898,"avgtime":0.000000475},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":46,"msgr_send_messages":48,"msgr_recv_bytes":18836,"msgr_send_bytes":12887810,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.063143456,"msgr_running_send_time":0.046347156,"msgr_running_recv_time":0.004800126,"msgr_running_fast_dispatch_time":0.011246223,"msgr_send_messages_queue_lat":{"avgcount":46,"sum":0.014736157,"avgtime":0.000320351},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1419,"complete_latency":{"avgcount":73,"sum":3.004429019,"avgtime":0.041156561}},"finisher-PurgeQueue":{"queue_len":47,"complete_latency":{"avgcount":14,"sum":0.106658321,"avgtime":0.007618451}},"mds":{"request":4948,"reply":4945,"reply_latency":{"avgcount":4945,"sum":694.342333139,"avgtime":0.140413009},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4947,"inodes_top":0,"inodes_bottom":4947,"inodes_pin_tail":0,"inodes_pinned":4947,"inodes_expired":0,"inodes_with_caps":3021,"caps":3021,"subtrees":2,"traverse":4948,"traverse_hit":4948,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2852,"root_rbytes":21912576,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":3948,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3597,"handle_client_caps_dirty":2999,"handle_client_cap_release":2,"process_request_cap_release":1745},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1659,"num_strays_delayed":0,"num_strays_enqueuing":104,"strays_created":1736,"strays_enqueued":77,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":8667,"evex":0,"evtrm":0,"ev":8670,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":24148544,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.857425093,"avgtime":0.071452091},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4947,"dn+":4947,"dn-":0,"cap":3021,"cap+":3202,"cap-":181,"rss":107264,"heap":248164},"mds_server":{"dispatch_client_request":4949,"dispatch_server_request":0,"handle_client_request":4948,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1736,"sum":2.720277696,"avgtime":0.001566980},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4663,"average_load":4663,"avg_session_uptime":971,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":18850340,"buffer_anon_items":10128,"buffer_meta_bytes":97328,"buffer_meta_items":1106,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12838836,"mds_co_items":125724,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":80,"op_send_bytes":17083306,"op_resend":0,"op_reply":80,"op_latency":{"avgcount":80,"sum":0.442990780,"avgtime":0.005537384},"op_inflight":0,"oplen_avg":{"avgcount":80,"sum":134},"op":80,"op_r":38,"op_w":42,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":10,"osdop_write":15,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":15,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":16,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":254,"max":104857600,"get_started":0,"get":8664,"get_sum":2434795,"get_or_fail_fail":0,"get_or_fail_success":8664,"take":0,"take_sum":0,"put":8663,"put_sum":2434541,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":80,"take_sum":17094441,"put":80,"put_sum":17094441,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":80,"take_sum":80,"put":80,"put_sum":80,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":181,"get_sum":18281,"get_or_fail_fail":0,"get_or_fail_success":181,"take":0,"take_sum":0,"put":3,"put_sum":18281,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":3177024,"max":3758096384,"get_started":0,"get":8667,"get_sum":19953042,"get_or_fail_fail":0,"get_or_fail_success":8667,"take":0,"take_sum":0,"put":12,"put_sum":16776018,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:31.192 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 119953, "msgr_recv_encrypted_bytes": 107744, "msgr_recv_messages": 32, "msgr_running_fast_dispatch_time": 0.000473206, "msgr_running_recv_time": 0.008076106, "msgr_running_send_time": 0.002833878, "msgr_running_total_time": 0.012645948, "msgr_send_bytes": 141769, "msgr_send_encrypted_bytes": 98944, "msgr_send_messages": 30, "msgr_send_messages_queue_lat": { "avgcount": 30, "avgtime": 0.000213068, "sum": 0.006392068 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 19205, "avgtime": 4.75e-07, "sum": 0.009140898 }, "msgr_recv_bytes": 3078480, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 8590, "msgr_running_fast_dispatch_time": 0.004322823, "msgr_running_recv_time": 1.383695095, "msgr_running_send_time": 1.167713238, "msgr_running_total_time": 1.797116642, "msgr_send_bytes": 10484926, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 15629, "msgr_send_messages_queue_lat": { "avgcount": 15628, "avgtime": 4.1224e-05, "sum": 0.644264177 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 18836, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 46, "msgr_running_fast_dispatch_time": 0.011246223, "msgr_running_recv_time": 0.004800126, "msgr_running_send_time": 0.046347156, "msgr_running_total_time": 0.063143456, "msgr_send_bytes": 12887810, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 48, "msgr_send_messages_queue_lat": { "avgcount": 46, "avgtime": 0.000320351, "sum": 0.014736157 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 73, "avgtime": 0.041156561, "sum": 3.004429019 }, "queue_len": 1419 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 14, "avgtime": 0.007618451, "sum": 0.106658321 }, "queue_len": 47 }, "mds": { "caps": 3021, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3948, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 2, "handle_client_caps": 3597, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4947, "inodes_bottom": 4947, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4947, "inodes_top": 0, "inodes_with_caps": 3021, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1745, "q": 0, "reply": 4945, "reply_latency": { "avgcount": 4945, "avgtime": 0.140413009, "sum": 694.342333139 }, "request": 4948, "root_rbytes": 21912576, "root_rfiles": 2852, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4948, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4948, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1659, "num_strays_delayed": 0, "num_strays_enqueuing": 104, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1736, "strays_enqueued": 77, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8670, "evadd": 8667, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.071452091, "sum": 0.857425093 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 24148544 }, "mds_mem": { "cap": 3021, "cap+": 3202, "cap-": 181, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4947, "dn+": 4947, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 107264 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4949, "dispatch_server_request": 0, "handle_client_request": 4948, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1736, "avgtime": 0.00156698, "sum": 2.720277696 } }, "mds_sessions": { "average_load": 4663, "avg_session_uptime": 971, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4663 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 18850340, "buffer_anon_items": 10128, "buffer_meta_bytes": 97328, "buffer_meta_items": 1106, "mds_co_bytes": 12838836, "mds_co_items": 125724, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 16, "op": 80, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 80, "avgtime": 0.005537384, "sum": 0.44299078 }, "op_pg": 0, "op_r": 38, "op_reply": 80, "op_resend": 0, "op_rmw": 0, "op_send": 80, "op_send_bytes": 17083306, "op_w": 42, "oplen_avg": { "avgcount": 80, "sum": 134 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 15, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 10, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 15, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8664, "get_or_fail_fail": 0, "get_or_fail_success": 8664, "get_started": 0, "get_sum": 2434795, "max": 104857600, "put": 8663, "put_sum": 2434541, "take": 0, "take_sum": 0, "val": 254, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 80, "put_sum": 17094441, "take": 80, "take_sum": 17094441, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 80, "put_sum": 80, "take": 80, "take_sum": 80, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 181, "get_or_fail_fail": 0, "get_or_fail_success": 181, "get_started": 0, "get_sum": 18281, "max": 3758096384, "put": 3, "put_sum": 18281, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 8667, "get_or_fail_fail": 0, "get_or_fail_success": 8667, "get_started": 0, "get_sum": 19953042, "max": 3758096384, "put": 12, "put_sum": 16776018, "take": 0, "take_sum": 0, "val": 3177024, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:31.193 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1659, 0/1736 2024-03-06T07:33:32.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:32.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.435+0000 7f1ad9200640 1 Processor -- start 2024-03-06T07:33:32.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.435+0000 7f1ad9200640 1 -- start start 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.437+0000 7f1ad9200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ad40cd180 0x7f1ad415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.437+0000 7f1ad9200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1ad40722f0 0x7f1ad415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.437+0000 7f1ad9200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad415ec00 con 0x7f1ad4071f10 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.437+0000 7f1ad9200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad415f9c0 con 0x7f1ad40cd180 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.437+0000 7f1ad9200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad415c410 con 0x7f1ad40722f0 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad1e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1ad4071f10 0x7f1ad40cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43234/0 (socket says 172.21.15.114:43234) 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad1e00640 1 -- 172.21.15.114:0/3410868329 learned_addr learned my addr 172.21.15.114:0/3410868329 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:32.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad1400640 1 --2- 172.21.15.114:0/3410868329 >> v2:172.21.15.114:3300/0 conn(0x7f1ad40cd180 0x7f1ad415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad2800640 1 --2- 172.21.15.114:0/3410868329 >> v2:172.21.15.177:3300/0 conn(0x7f1ad40722f0 0x7f1ad415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad1400640 1 -- 172.21.15.114:0/3410868329 >> v2:172.21.15.177:3300/0 conn(0x7f1ad40722f0 msgr2=0x7f1ad415bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad1400640 1 --2- 172.21.15.114:0/3410868329 >> v2:172.21.15.177:3300/0 conn(0x7f1ad40722f0 0x7f1ad415bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad1400640 1 -- 172.21.15.114:0/3410868329 >> v1:172.21.15.114:6789/0 conn(0x7f1ad4071f10 legacy=0x7f1ad40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad1400640 1 -- 172.21.15.114:0/3410868329 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad415eee0 con 0x7f1ad40cd180 2024-03-06T07:33:32.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.438+0000 7f1ad0a00640 1 -- 172.21.15.114:0/3410868329 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116942213 0 0) 0x7f1ad415ec00 con 0x7f1ad4071f10 2024-03-06T07:33:32.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.441+0000 7f1ad1400640 1 --2- 172.21.15.114:0/3410868329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40cd180 0x7f1ad415b6a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00097d0 tx=0x7f1ac002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9da47366b96456 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.442+0000 7f1ad0a00640 1 -- 172.21.15.114:0/3410868329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0031780 con 0x7f1ad40cd180 2024-03-06T07:33:32.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.442+0000 7f1ad0a00640 1 -- 172.21.15.114:0/3410868329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac0037070 con 0x7f1ad40cd180 2024-03-06T07:33:32.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.442+0000 7f1ad0a00640 1 -- 172.21.15.114:0/3410868329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0031b40 con 0x7f1ad40cd180 2024-03-06T07:33:32.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.443+0000 7f1ad9200640 1 -- 172.21.15.114:0/3410868329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40cd180 msgr2=0x7f1ad415b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.443+0000 7f1ad9200640 1 --2- 172.21.15.114:0/3410868329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40cd180 0x7f1ad415b6a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00097d0 tx=0x7f1ac002b820 comp rx=0 tx=0).stop 2024-03-06T07:33:32.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.443+0000 7f1ad9200640 1 -- 172.21.15.114:0/3410868329 shutdown_connections 2024-03-06T07:33:32.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.443+0000 7f1ad9200640 1 --2- 172.21.15.114:0/3410868329 >> v2:172.21.15.177:3300/0 conn(0x7f1ad40722f0 0x7f1ad415bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.443+0000 7f1ad9200640 1 --2- 172.21.15.114:0/3410868329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40cd180 0x7f1ad415b6a0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.443+0000 7f1ad9200640 1 -- 172.21.15.114:0/3410868329 >> 172.21.15.114:0/3410868329 conn(0x7f1ad40d0d30 msgr2=0x7f1ad40d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.444+0000 7f1ad9200640 1 -- 172.21.15.114:0/3410868329 shutdown_connections 2024-03-06T07:33:32.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.444+0000 7f1ad9200640 1 -- 172.21.15.114:0/3410868329 wait complete. 2024-03-06T07:33:32.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.447+0000 7f1ad9200640 1 Processor -- start 2024-03-06T07:33:32.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.447+0000 7f1ad9200640 1 -- start start 2024-03-06T07:33:32.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.449+0000 7f1ad9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ad4071f10 0x7f1ad414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.449+0000 7f1ad9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40722f0 0x7f1ad40b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.449+0000 7f1ad9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ad41514a0 0x7f1ad40b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.449+0000 7f1ad9200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad415c120 con 0x7f1ad40722f0 2024-03-06T07:33:32.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.449+0000 7f1ad9200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad415f4a0 con 0x7f1ad41514a0 2024-03-06T07:33:32.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.449+0000 7f1ad9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad406af40 con 0x7f1ad4071f10 2024-03-06T07:33:32.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.450+0000 7f1ad1e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ad4071f10 0x7f1ad414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.450+0000 7f1ad1e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ad4071f10 0x7f1ad414f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:59686/0 (socket says 172.21.15.114:59686) 2024-03-06T07:33:32.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.450+0000 7f1ad1e00640 1 -- 172.21.15.114:0/1193687416 learned_addr learned my addr 172.21.15.114:0/1193687416 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:32.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.450+0000 7f1ad1400640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40722f0 0x7f1ad40b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.450+0000 7f1ad2800640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ad41514a0 0x7f1ad40b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.451+0000 7f1ad1400640 1 -- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ad4071f10 msgr2=0x7f1ad414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.451+0000 7f1ad1400640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ad4071f10 0x7f1ad414f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.451+0000 7f1ad1400640 1 -- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ad41514a0 msgr2=0x7f1ad40b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.451+0000 7f1ad1400640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ad41514a0 0x7f1ad40b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.451+0000 7f1ad1400640 1 -- 172.21.15.114:0/1193687416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad41500e0 con 0x7f1ad40722f0 2024-03-06T07:33:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.451+0000 7f1ad2800640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ad41514a0 0x7f1ad40b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.452+0000 7f1ad1400640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40722f0 0x7f1ad40b65d0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0033000 tx=0x7f1ac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.452+0000 7f1aca000640 1 -- 172.21.15.114:0/1193687416 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0032820 con 0x7f1ad40722f0 2024-03-06T07:33:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.452+0000 7f1aca000640 1 -- 172.21.15.114:0/1193687416 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac00329e0 con 0x7f1ad40722f0 2024-03-06T07:33:32.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.452+0000 7f1aca000640 1 -- 172.21.15.114:0/1193687416 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0032dd0 con 0x7f1ad40722f0 2024-03-06T07:33:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.452+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad40b39f0 con 0x7f1ad40722f0 2024-03-06T07:33:32.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.452+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad4151c90 con 0x7f1ad40722f0 2024-03-06T07:33:32.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.455+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad4068070 con 0x7f1ad40722f0 2024-03-06T07:33:32.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.458+0000 7f1aca000640 1 -- 172.21.15.114:0/1193687416 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ac003b3f0 con 0x7f1ad40722f0 2024-03-06T07:33:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.459+0000 7f1aca000640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ab406b550 0x7f1ab406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.459+0000 7f1aca000640 1 -- 172.21.15.114:0/1193687416 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f1ac00abef0 con 0x7f1ad40722f0 2024-03-06T07:33:32.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.459+0000 7f1ad1e00640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ab406b550 0x7f1ab406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.460+0000 7f1ad1e00640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ab406b550 0x7f1ab406da10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4007b10 tx=0x7f1ac4007a60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.471+0000 7f1aca000640 1 -- 172.21.15.114:0/1193687416 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ac004a060 con 0x7f1ad40722f0 2024-03-06T07:33:32.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.992+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ad4151ed0 con 0x7f1ad40722f0 2024-03-06T07:33:32.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.993+0000 7f1aca000640 1 -- 172.21.15.114:0/1193687416 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f1ac002f050 con 0x7f1ad40722f0 2024-03-06T07:33:32.995 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:32.995 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:32.995 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ab406b550 msgr2=0x7f1ab406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ab406b550 0x7f1ab406da10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4007b10 tx=0x7f1ac4007a60 comp rx=0 tx=0).stop 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40722f0 msgr2=0x7f1ad40b65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40722f0 0x7f1ad40b65d0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0033000 tx=0x7f1ac00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 shutdown_connections 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ad41514a0 0x7f1ad40b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ab406b550 0x7f1ab406da10 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ad40722f0 0x7f1ad40b65d0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 --2- 172.21.15.114:0/1193687416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ad4071f10 0x7f1ad414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 >> 172.21.15.114:0/1193687416 conn(0x7f1ad40d0d30 msgr2=0x7f1ad4159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.995+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 shutdown_connections 2024-03-06T07:33:32.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:32.996+0000 7f1ad9200640 1 -- 172.21.15.114:0/1193687416 wait complete. 2024-03-06T07:33:33.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:33.248 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":94,"msgr_send_messages":93,"msgr_recv_bytes":176520,"msgr_send_bytes":196792,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.022035283,"msgr_running_send_time":0.008298012,"msgr_running_recv_time":0.009928804,"msgr_running_fast_dispatch_time":0.002077587,"msgr_send_messages_queue_lat":{"avgcount":93,"sum":0.011040196,"avgtime":0.000118711},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108016,"msgr_send_encrypted_bytes":101360},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11128,"msgr_send_messages":18316,"msgr_recv_bytes":3969872,"msgr_send_bytes":11315951,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.108414659,"msgr_running_send_time":1.367699792,"msgr_running_recv_time":1.875523493,"msgr_running_fast_dispatch_time":0.008351813,"msgr_send_messages_queue_lat":{"avgcount":18315,"sum":1.073847271,"avgtime":0.000058632},"msgr_handle_ack_lat":{"avgcount":23151,"sum":0.010304464,"avgtime":0.000000445},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":199,"msgr_send_messages":201,"msgr_recv_bytes":57712,"msgr_send_bytes":17175710,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.100443409,"msgr_running_send_time":0.071874662,"msgr_running_recv_time":0.008581765,"msgr_running_fast_dispatch_time":0.017949639,"msgr_send_messages_queue_lat":{"avgcount":199,"sum":0.031844362,"avgtime":0.000160021},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1443,"complete_latency":{"avgcount":75,"sum":5.717674302,"avgtime":0.076235657}},"finisher-PurgeQueue":{"queue_len":77,"complete_latency":{"avgcount":632,"sum":0.807039522,"avgtime":0.001276961}},"mds":{"request":5490,"reply":5488,"reply_latency":{"avgcount":5488,"sum":695.656651103,"avgtime":0.126759593},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5490,"inodes_top":0,"inodes_bottom":5490,"inodes_pin_tail":0,"inodes_pinned":5490,"inodes_expired":0,"inodes_with_caps":2585,"caps":2585,"subtrees":2,"traverse":5491,"traverse_hit":5491,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1784,"root_rbytes":13787136,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":6,"ceph_cap_op_grant":4985,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3967,"handle_client_caps_dirty":2999,"handle_client_cap_release":6,"process_request_cap_release":2287},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1789,"num_strays_delayed":15,"num_strays_enqueuing":16,"strays_created":2279,"strays_enqueued":490,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10023,"evex":0,"evtrm":0,"ev":10026,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":12,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26970126,"rdpos":4195244,"jlat":{"avgcount":14,"sum":1.589924723,"avgtime":0.113566051},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":5490,"dn+":5490,"dn-":0,"cap":2585,"cap+":3202,"cap-":617,"rss":107904,"heap":248164},"mds_server":{"dispatch_client_request":5491,"dispatch_server_request":0,"handle_client_request":5490,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2279,"sum":4.034595660,"avgtime":0.001770335},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5120,"average_load":5120,"avg_session_uptime":976,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":20847797,"buffer_anon_items":9546,"buffer_meta_bytes":264,"buffer_meta_items":3,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16622434,"mds_co_items":214248,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":451,"op_send_bytes":21354323,"op_resend":0,"op_reply":451,"op_latency":{"avgcount":451,"sum":1.656965543,"avgtime":0.003673981},"op_inflight":0,"oplen_avg":{"avgcount":451,"sum":506},"op":451,"op_r":45,"op_w":406,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":17,"osdop_write":23,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":368,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":1,"pq_executing_ops_high_water":6,"pq_executing":1,"pq_executing_high_water":6,"pq_executed_ops":351,"pq_executed":351,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":389828,"max":104857600,"get_started":0,"get":11417,"get_sum":3173808,"get_or_fail_fail":0,"get_or_fail_success":11417,"take":0,"take_sum":0,"put":9954,"put_sum":2783980,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":451,"take_sum":21406363,"put":451,"put_sum":21406363,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":451,"take_sum":451,"put":451,"put_sum":451,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":506,"get_sum":51106,"get_or_fail_fail":0,"get_or_fail_success":506,"take":0,"take_sum":0,"put":9,"put_sum":51106,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":1804302,"max":3758096384,"get_started":0,"get":10023,"get_sum":22774624,"get_or_fail_fail":0,"get_or_fail_success":10023,"take":0,"take_sum":0,"put":14,"put_sum":20970322,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:33.250 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 176520, "msgr_recv_encrypted_bytes": 108016, "msgr_recv_messages": 94, "msgr_running_fast_dispatch_time": 0.002077587, "msgr_running_recv_time": 0.009928804, "msgr_running_send_time": 0.008298012, "msgr_running_total_time": 0.022035283, "msgr_send_bytes": 196792, "msgr_send_encrypted_bytes": 101360, "msgr_send_messages": 93, "msgr_send_messages_queue_lat": { "avgcount": 93, "avgtime": 0.000118711, "sum": 0.011040196 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 23151, "avgtime": 4.45e-07, "sum": 0.010304464 }, "msgr_recv_bytes": 3969872, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 11128, "msgr_running_fast_dispatch_time": 0.008351813, "msgr_running_recv_time": 1.875523493, "msgr_running_send_time": 1.367699792, "msgr_running_total_time": 2.108414659, "msgr_send_bytes": 11315951, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 18316, "msgr_send_messages_queue_lat": { "avgcount": 18315, "avgtime": 5.8632e-05, "sum": 1.073847271 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 57712, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 199, "msgr_running_fast_dispatch_time": 0.017949639, "msgr_running_recv_time": 0.008581765, "msgr_running_send_time": 0.071874662, "msgr_running_total_time": 0.100443409, "msgr_send_bytes": 17175710, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 201, "msgr_send_messages_queue_lat": { "avgcount": 199, "avgtime": 0.000160021, "sum": 0.031844362 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 75, "avgtime": 0.076235657, "sum": 5.717674302 }, "queue_len": 1443 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 632, "avgtime": 0.001276961, "sum": 0.807039522 }, "queue_len": 77 }, "mds": { "caps": 2585, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4985, "ceph_cap_op_revoke": 6, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 6, "handle_client_caps": 3967, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5490, "inodes_bottom": 5490, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5490, "inodes_top": 0, "inodes_with_caps": 2585, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2287, "q": 0, "reply": 5488, "reply_latency": { "avgcount": 5488, "avgtime": 0.126759593, "sum": 695.656651103 }, "request": 5490, "root_rbytes": 13787136, "root_rfiles": 1784, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5491, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5491, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1789, "num_strays_delayed": 15, "num_strays_enqueuing": 16, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2279, "strays_enqueued": 490, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10026, "evadd": 10023, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.113566051, "sum": 1.589924723 }, "rdpos": 4195244, "replayed": 3, "seg": 12, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26970126 }, "mds_mem": { "cap": 2585, "cap+": 3202, "cap-": 617, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5490, "dn+": 5490, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 107904 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5491, "dispatch_server_request": 0, "handle_client_request": 5490, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2279, "avgtime": 0.001770335, "sum": 4.03459566 } }, "mds_sessions": { "average_load": 5120, "avg_session_uptime": 976, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5120 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 20847797, "buffer_anon_items": 9546, "buffer_meta_bytes": 264, "buffer_meta_items": 3, "mds_co_bytes": 16622434, "mds_co_items": 214248, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 18, "op": 451, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 451, "avgtime": 0.003673981, "sum": 1.656965543 }, "op_pg": 0, "op_r": 45, "op_reply": 451, "op_resend": 0, "op_rmw": 0, "op_send": 451, "op_send_bytes": 21354323, "op_w": 406, "oplen_avg": { "avgcount": 451, "sum": 506 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 368, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 23, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 351, "pq_executed_ops": 351, "pq_executing": 1, "pq_executing_high_water": 6, "pq_executing_ops": 1, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11417, "get_or_fail_fail": 0, "get_or_fail_success": 11417, "get_started": 0, "get_sum": 3173808, "max": 104857600, "put": 9954, "put_sum": 2783980, "take": 0, "take_sum": 0, "val": 389828, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 451, "put_sum": 21406363, "take": 451, "take_sum": 21406363, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 451, "put_sum": 451, "take": 451, "take_sum": 451, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 506, "get_or_fail_fail": 0, "get_or_fail_success": 506, "get_started": 0, "get_sum": 51106, "max": 3758096384, "put": 9, "put_sum": 51106, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 10023, "get_or_fail_fail": 0, "get_or_fail_success": 10023, "get_started": 0, "get_sum": 22774624, "max": 3758096384, "put": 14, "put_sum": 20970322, "take": 0, "take_sum": 0, "val": 1804302, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:33.250 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 1/1789, 351/2279 2024-03-06T07:33:34.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:34.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.477+0000 7f4ad6e00640 1 Processor -- start 2024-03-06T07:33:34.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.477+0000 7f4ad6e00640 1 -- start start 2024-03-06T07:33:34.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.478+0000 7f4ad6e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4ad015e1f0 0x7f4ad00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.478+0000 7f4ad6e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad00cfe30 con 0x7f4ad00ca630 2024-03-06T07:33:34.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.479+0000 7f4ad6e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad00c7b40 con 0x7f4ad015e5d0 2024-03-06T07:33:34.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.479+0000 7f4ad6e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad00d00d0 con 0x7f4ad015e1f0 2024-03-06T07:33:34.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.479+0000 7f4ad5000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4ad00ca630 0x7f4ad00d0860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43240/0 (socket says 172.21.15.114:43240) 2024-03-06T07:33:34.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.479+0000 7f4ad5000640 1 -- 172.21.15.114:0/2754099965 learned_addr learned my addr 172.21.15.114:0/2754099965 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:34.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.479+0000 7f4ad5a00640 1 --2- 172.21.15.114:0/2754099965 >> v2:172.21.15.114:3300/0 conn(0x7f4ad015e1f0 0x7f4ad00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.479+0000 7f4ad5a00640 1 -- 172.21.15.114:0/2754099965 >> v1:172.21.15.177:6789/0 conn(0x7f4ad015e5d0 legacy=0x7f4ad0167f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.479+0000 7f4ad5a00640 1 -- 172.21.15.114:0/2754099965 >> v1:172.21.15.114:6789/0 conn(0x7f4ad00ca630 legacy=0x7f4ad00d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.480+0000 7f4acfe00640 1 -- 172.21.15.114:0/2754099965 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348628557 0 0) 0x7f4ad00cfe30 con 0x7f4ad00ca630 2024-03-06T07:33:34.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.480+0000 7f4ad5a00640 1 -- 172.21.15.114:0/2754099965 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad00c7e20 con 0x7f4ad015e1f0 2024-03-06T07:33:34.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.481+0000 7f4ad5a00640 1 --2- 172.21.15.114:0/2754099965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad015e1f0 0x7f4ad00cf590 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40097a0 tx=0x7f4ac402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc81898746d5ecd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.481+0000 7f4acfe00640 1 -- 172.21.15.114:0/2754099965 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4035020 con 0x7f4ad015e1f0 2024-03-06T07:33:34.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.481+0000 7f4acfe00640 1 -- 172.21.15.114:0/2754099965 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac4031830 con 0x7f4ad015e1f0 2024-03-06T07:33:34.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.481+0000 7f4acfe00640 1 -- 172.21.15.114:0/2754099965 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4031c00 con 0x7f4ad015e1f0 2024-03-06T07:33:34.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.482+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2754099965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad015e1f0 msgr2=0x7f4ad00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.482+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2754099965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad015e1f0 0x7f4ad00cf590 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40097a0 tx=0x7f4ac402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.482+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2754099965 shutdown_connections 2024-03-06T07:33:34.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.482+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2754099965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad015e1f0 0x7f4ad00cf590 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.482+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2754099965 >> 172.21.15.114:0/2754099965 conn(0x7f4ad00d3ee0 msgr2=0x7f4ad00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.482+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2754099965 shutdown_connections 2024-03-06T07:33:34.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.482+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2754099965 wait complete. 2024-03-06T07:33:34.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.484+0000 7f4ad6e00640 1 Processor -- start 2024-03-06T07:33:34.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.484+0000 7f4ad6e00640 1 -- start start 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad6e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad00ca630 0x7f4ad009dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad6e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ad015e1f0 0x7f4ad00cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad6e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ad015e5d0 0x7f4ad00a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad6e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad00d4a70 con 0x7f4ad00ca630 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad6e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad00cfad0 con 0x7f4ad015e5d0 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad6e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad00d4bf0 con 0x7f4ad015e1f0 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad00ca630 0x7f4ad009dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad00ca630 0x7f4ad009dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54554/0 (socket says 172.21.15.114:54554) 2024-03-06T07:33:34.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.485+0000 7f4ad5a00640 1 -- 172.21.15.114:0/2457448796 learned_addr learned my addr 172.21.15.114:0/2457448796 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad5000640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ad015e1f0 0x7f4ad00cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad6400640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ad015e5d0 0x7f4ad00a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad5a00640 1 -- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ad015e1f0 msgr2=0x7f4ad00cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad5a00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ad015e1f0 0x7f4ad00cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad5a00640 1 -- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ad015e5d0 msgr2=0x7f4ad00a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad5a00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ad015e5d0 0x7f4ad00a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad5a00640 1 -- 172.21.15.114:0/2457448796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad00ce020 con 0x7f4ad00ca630 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.486+0000 7f4ad6400640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ad015e5d0 0x7f4ad00a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.487+0000 7f4ad5a00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad00ca630 0x7f4ad009dbc0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4000c00 tx=0x7f4ac40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.487+0000 7f4acd600640 1 -- 172.21.15.114:0/2457448796 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4035050 con 0x7f4ad00ca630 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.487+0000 7f4acd600640 1 -- 172.21.15.114:0/2457448796 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac4004230 con 0x7f4ad00ca630 2024-03-06T07:33:34.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.487+0000 7f4acd600640 1 -- 172.21.15.114:0/2457448796 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac403b740 con 0x7f4ad00ca630 2024-03-06T07:33:34.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.487+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad009e230 con 0x7f4ad00ca630 2024-03-06T07:33:34.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.487+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad009e7f0 con 0x7f4ad00ca630 2024-03-06T07:33:34.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.488+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad004f330 con 0x7f4ad00ca630 2024-03-06T07:33:34.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.490+0000 7f4acd600640 1 -- 172.21.15.114:0/2457448796 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ac4038020 con 0x7f4ad00ca630 2024-03-06T07:33:34.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.491+0000 7f4acd600640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ab006b4d0 0x7f4ab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.491+0000 7f4acd600640 1 -- 172.21.15.114:0/2457448796 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f4ac40ac0e0 con 0x7f4ad00ca630 2024-03-06T07:33:34.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.491+0000 7f4ad5000640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ab006b4d0 0x7f4ab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.492+0000 7f4ad5000640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ab006b4d0 0x7f4ab006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4ad01f7070 tx=0x7f4abc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.496+0000 7f4acd600640 1 -- 172.21.15.114:0/2457448796 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ac4040070 con 0x7f4ad00ca630 2024-03-06T07:33:34.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.784+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ad00c9ef0 con 0x7f4ad00ca630 2024-03-06T07:33:34.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.784+0000 7f4acd600640 1 -- 172.21.15.114:0/2457448796 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f4ac407af40 con 0x7f4ad00ca630 2024-03-06T07:33:34.786 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:34.786 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:34.786 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ab006b4d0 msgr2=0x7f4ab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ab006b4d0 0x7f4ab006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4ad01f7070 tx=0x7f4abc009040 comp rx=0 tx=0).stop 2024-03-06T07:33:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad00ca630 msgr2=0x7f4ad009dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad00ca630 0x7f4ad009dbc0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4000c00 tx=0x7f4ac40046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 shutdown_connections 2024-03-06T07:33:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ad015e5d0 0x7f4ad00a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4ad015e1f0 0x7f4ad00cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4ab006b4d0 0x7f4ab006d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 --2- 172.21.15.114:0/2457448796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ad00ca630 0x7f4ad009dbc0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 >> 172.21.15.114:0/2457448796 conn(0x7f4ad00d3ee0 msgr2=0x7f4ad015d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 shutdown_connections 2024-03-06T07:33:34.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:34.787+0000 7f4ad6e00640 1 -- 172.21.15.114:0/2457448796 wait complete. 2024-03-06T07:33:34.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:35.038 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":127,"msgr_send_messages":126,"msgr_recv_bytes":188708,"msgr_send_bytes":211356,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.025391572,"msgr_running_send_time":0.009909488,"msgr_running_recv_time":0.010678242,"msgr_running_fast_dispatch_time":0.002832709,"msgr_send_messages_queue_lat":{"avgcount":126,"sum":0.013624877,"avgtime":0.000108133},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108016,"msgr_send_encrypted_bytes":101360},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11921,"msgr_send_messages":20772,"msgr_recv_bytes":4104910,"msgr_send_bytes":13397288,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.347301053,"msgr_running_send_time":1.519141514,"msgr_running_recv_time":1.975362394,"msgr_running_fast_dispatch_time":0.013656829,"msgr_send_messages_queue_lat":{"avgcount":20771,"sum":1.191936247,"avgtime":0.000057384},"msgr_handle_ack_lat":{"avgcount":25414,"sum":0.010999896,"avgtime":0.000000432},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":282,"msgr_send_messages":284,"msgr_recv_bytes":78794,"msgr_send_bytes":17202768,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.109152028,"msgr_running_send_time":0.075868784,"msgr_running_recv_time":0.010664386,"msgr_running_fast_dispatch_time":0.020008983,"msgr_send_messages_queue_lat":{"avgcount":282,"sum":0.036793976,"avgtime":0.000130475},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1477,"complete_latency":{"avgcount":77,"sum":8.011063479,"avgtime":0.104039785}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":982,"sum":0.959900078,"avgtime":0.000977494}},"mds":{"request":5520,"reply":5517,"reply_latency":{"avgcount":5517,"sum":697.991216012,"avgtime":0.126516442},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4986,"inodes_top":0,"inodes_bottom":4986,"inodes_pin_tail":0,"inodes_pinned":4986,"inodes_expired":0,"inodes_with_caps":979,"caps":979,"subtrees":2,"traverse":5521,"traverse_hit":5520,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1784,"root_rbytes":13787136,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":115,"ceph_cap_op_grant":6385,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5335,"handle_client_caps_dirty":3000,"handle_client_cap_release":776,"process_request_cap_release":2317},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1764,"num_strays_delayed":1420,"num_strays_enqueuing":0,"strays_created":2308,"strays_enqueued":544,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11140,"evex":0,"evtrm":0,"ev":11143,"evexg":0,"evexd":0,"segadd":12,"segex":0,"segtrm":0,"seg":14,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":29157990,"rdpos":4195244,"jlat":{"avgcount":16,"sum":2.183846875,"avgtime":0.136490429},"replayed":3},"mds_mem":{"ino":2708,"ino+":3214,"ino-":506,"dir":13,"dir+":13,"dir-":0,"dn":5013,"dn+":5519,"dn-":506,"cap":923,"cap+":3202,"cap-":2279,"rss":107904,"heap":248164},"mds_server":{"dispatch_client_request":5521,"dispatch_server_request":0,"handle_client_request":5520,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2308,"sum":6.369160569,"avgtime":0.002759601},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5012,"average_load":5012,"avg_session_uptime":976,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":20847797,"buffer_anon_items":9546,"buffer_meta_bytes":264,"buffer_meta_items":3,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":16622434,"mds_co_items":214248,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":653,"op_send_bytes":25241983,"op_resend":0,"op_reply":653,"op_latency":{"avgcount":653,"sum":2.344951238,"avgtime":0.003591043},"op_inflight":0,"oplen_avg":{"avgcount":653,"sum":708},"op":653,"op_r":47,"op_w":606,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":19,"osdop_write":28,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":560,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":544,"pq_executed":544,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":12326,"get_sum":3260306,"get_or_fail_fail":0,"get_or_fail_success":12326,"take":0,"take_sum":0,"put":12325,"put_sum":3260053,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":653,"take_sum":25297861,"put":653,"put_sum":25297861,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":653,"take_sum":653,"put":653,"put_sum":653,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":544,"get_sum":54944,"get_or_fail_fail":0,"get_or_fail_success":544,"take":0,"take_sum":0,"put":11,"put_sum":54944,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":108614,"max":3758096384,"get_started":0,"get":11140,"get_sum":24962488,"get_or_fail_fail":0,"get_or_fail_success":11140,"take":0,"take_sum":0,"put":17,"put_sum":24853874,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:35.039 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 188708, "msgr_recv_encrypted_bytes": 108016, "msgr_recv_messages": 127, "msgr_running_fast_dispatch_time": 0.002832709, "msgr_running_recv_time": 0.010678242, "msgr_running_send_time": 0.009909488, "msgr_running_total_time": 0.025391572, "msgr_send_bytes": 211356, "msgr_send_encrypted_bytes": 101360, "msgr_send_messages": 126, "msgr_send_messages_queue_lat": { "avgcount": 126, "avgtime": 0.000108133, "sum": 0.013624877 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 25414, "avgtime": 4.32e-07, "sum": 0.010999896 }, "msgr_recv_bytes": 4104910, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 11921, "msgr_running_fast_dispatch_time": 0.013656829, "msgr_running_recv_time": 1.975362394, "msgr_running_send_time": 1.519141514, "msgr_running_total_time": 2.347301053, "msgr_send_bytes": 13397288, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 20772, "msgr_send_messages_queue_lat": { "avgcount": 20771, "avgtime": 5.7384e-05, "sum": 1.191936247 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 78794, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 282, "msgr_running_fast_dispatch_time": 0.020008983, "msgr_running_recv_time": 0.010664386, "msgr_running_send_time": 0.075868784, "msgr_running_total_time": 0.109152028, "msgr_send_bytes": 17202768, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 284, "msgr_send_messages_queue_lat": { "avgcount": 282, "avgtime": 0.000130475, "sum": 0.036793976 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 77, "avgtime": 0.104039785, "sum": 8.011063479 }, "queue_len": 1477 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 982, "avgtime": 0.000977494, "sum": 0.959900078 }, "queue_len": 0 }, "mds": { "caps": 979, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6385, "ceph_cap_op_revoke": 115, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 776, "handle_client_caps": 5335, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4986, "inodes_bottom": 4986, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4986, "inodes_top": 0, "inodes_with_caps": 979, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2317, "q": 0, "reply": 5517, "reply_latency": { "avgcount": 5517, "avgtime": 0.126516442, "sum": 697.991216012 }, "request": 5520, "root_rbytes": 13787136, "root_rfiles": 1784, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5521, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5520, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1764, "num_strays_delayed": 1420, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2308, "strays_enqueued": 544, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11143, "evadd": 11140, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 16, "avgtime": 0.136490429, "sum": 2.183846875 }, "rdpos": 4195244, "replayed": 3, "seg": 14, "segadd": 12, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 29157990 }, "mds_mem": { "cap": 923, "cap+": 3202, "cap-": 2279, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5013, "dn+": 5519, "dn-": 506, "heap": 248164, "ino": 2708, "ino+": 3214, "ino-": 506, "rss": 107904 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5521, "dispatch_server_request": 0, "handle_client_request": 5520, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2308, "avgtime": 0.002759601, "sum": 6.369160569 } }, "mds_sessions": { "average_load": 5012, "avg_session_uptime": 976, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5012 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 20847797, "buffer_anon_items": 9546, "buffer_meta_bytes": 264, "buffer_meta_items": 3, "mds_co_bytes": 16622434, "mds_co_items": 214248, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 18, "op": 653, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 653, "avgtime": 0.003591043, "sum": 2.344951238 }, "op_pg": 0, "op_r": 47, "op_reply": 653, "op_resend": 0, "op_rmw": 0, "op_send": 653, "op_send_bytes": 25241983, "op_w": 606, "oplen_avg": { "avgcount": 653, "sum": 708 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 560, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 19, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 28, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 544, "pq_executed_ops": 544, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12326, "get_or_fail_fail": 0, "get_or_fail_success": 12326, "get_started": 0, "get_sum": 3260306, "max": 104857600, "put": 12325, "put_sum": 3260053, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 653, "put_sum": 25297861, "take": 653, "take_sum": 25297861, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 653, "put_sum": 653, "take": 653, "take_sum": 653, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 544, "get_or_fail_fail": 0, "get_or_fail_success": 544, "get_started": 0, "get_sum": 54944, "max": 3758096384, "put": 11, "put_sum": 54944, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 11140, "get_or_fail_fail": 0, "get_or_fail_success": 11140, "get_started": 0, "get_sum": 24962488, "max": 3758096384, "put": 17, "put_sum": 24853874, "take": 0, "take_sum": 0, "val": 108614, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:35.039 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1764, 544/2308 2024-03-06T07:33:36.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.259+0000 7f6083800640 1 Processor -- start 2024-03-06T07:33:36.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.259+0000 7f6083800640 1 -- start start 2024-03-06T07:33:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.260+0000 7f6083800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f607c0701f0 0x7f607c0cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.260+0000 7f6083800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c0cf4a0 con 0x7f607c0cf090 2024-03-06T07:33:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.260+0000 7f6083800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c0cc2d0 con 0x7f607c0cebe0 2024-03-06T07:33:36.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.260+0000 7f6083800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607c15f070 con 0x7f607c0701f0 2024-03-06T07:33:36.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.261+0000 7f6081800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f607c0cf090 0x7f607c06af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43244/0 (socket says 172.21.15.114:43244) 2024-03-06T07:33:36.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.261+0000 7f6081800640 1 -- 172.21.15.114:0/96574783 learned_addr learned my addr 172.21.15.114:0/96574783 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.261+0000 7f6080e00640 1 --2- 172.21.15.114:0/96574783 >> v2:172.21.15.177:3300/0 conn(0x7f607c0701f0 0x7f607c0cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.261+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217528532 0 0) 0x7f607c0cf4a0 con 0x7f607c0cf090 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.261+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6064003520 con 0x7f607c0cf090 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127905724 0 0) 0x7f6064003520 con 0x7f607c0cf090 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 >> v2:172.21.15.177:3300/0 conn(0x7f607c0701f0 msgr2=0x7f607c0cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 --2- 172.21.15.114:0/96574783 >> v2:172.21.15.177:3300/0 conn(0x7f607c0701f0 0x7f607c0cbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 >> v1:172.21.15.177:6789/0 conn(0x7f607c0cebe0 legacy=0x7f607c162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c15f350 con 0x7f607c0cf090 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6074001a10 con 0x7f607c0cf090 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 >> v1:172.21.15.114:6789/0 conn(0x7f607c0cf090 legacy=0x7f607c06af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640039b0 0x7f6064003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6064004800 0x7f6064004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640051c0 0x7f606400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c15f070 con 0x7f6064004800 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606400c060 con 0x7f60640051c0 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.262+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606400c260 con 0x7f60640039b0 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f607be00640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6064004800 0x7f6064004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f6080e00640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640039b0 0x7f6064003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f6081800640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640051c0 0x7f606400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f607be00640 1 -- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640039b0 msgr2=0x7f6064003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f607be00640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f60640039b0 0x7f6064003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f607be00640 1 -- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640051c0 msgr2=0x7f606400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f607be00640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f60640051c0 0x7f606400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f607be00640 1 -- 172.21.15.114:0/96574783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064003520 con 0x7f6064004800 2024-03-06T07:33:36.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.263+0000 7f6081800640 1 -- 172.21.15.114:0/96574783 reap_dead start 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.265+0000 7f607be00640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6064004800 0x7f6064004c80 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f6070001750 tx=0x7f6064005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce6587099d8777f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.265+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070001370 con 0x7f6064004800 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.265+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60700052d0 con 0x7f6064004800 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.265+0000 7f607b400640 1 -- 172.21.15.114:0/96574783 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070003830 con 0x7f6064004800 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.266+0000 7f6083800640 1 -- 172.21.15.114:0/96574783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6064004800 msgr2=0x7f6064004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.266+0000 7f6083800640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6064004800 0x7f6064004c80 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f6070001750 tx=0x7f6064005830 comp rx=0 tx=0).stop 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.266+0000 7f6083800640 1 -- 172.21.15.114:0/96574783 shutdown_connections 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.266+0000 7f6083800640 1 --2- 172.21.15.114:0/96574783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6064004800 0x7f6064004c80 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.266+0000 7f6083800640 1 -- 172.21.15.114:0/96574783 >> 172.21.15.114:0/96574783 conn(0x7f607c0d0d30 msgr2=0x7f607c0d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.266+0000 7f6083800640 1 -- 172.21.15.114:0/96574783 shutdown_connections 2024-03-06T07:33:36.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.266+0000 7f6083800640 1 -- 172.21.15.114:0/96574783 wait complete. 2024-03-06T07:33:36.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.269+0000 7f6083800640 1 Processor -- start 2024-03-06T07:33:36.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.269+0000 7f6083800640 1 -- start start 2024-03-06T07:33:36.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.270+0000 7f6083800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6064004800 0x7f607c158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.270+0000 7f6083800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c15df00 0x7f607c15e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.270+0000 7f6083800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f607c1601a0 0x7f607c160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.270+0000 7f6083800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c0705d0 con 0x7f607c15df00 2024-03-06T07:33:36.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.270+0000 7f6083800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c162000 con 0x7f607c1601a0 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.270+0000 7f6083800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607c0cbd90 con 0x7f6064004800 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c15df00 0x7f607c15e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c15df00 0x7f607c15e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54568/0 (socket says 172.21.15.114:54568) 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 -- 172.21.15.114:0/3153745519 learned_addr learned my addr 172.21.15.114:0/3153745519 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f6081800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f607c1601a0 0x7f607c160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f6080e00640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6064004800 0x7f607c158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 -- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6064004800 msgr2=0x7f607c158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6064004800 0x7f607c158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 -- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f607c1601a0 msgr2=0x7f607c160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f607c1601a0 0x7f607c160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f607be00640 1 -- 172.21.15.114:0/3153745519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c160b60 con 0x7f607c15df00 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.271+0000 7f6081800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f607c1601a0 0x7f607c160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:36.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.272+0000 7f607be00640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c15df00 0x7f607c15e360 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f6070000c00 tx=0x7f6070001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.272+0000 7f6078c00640 1 -- 172.21.15.114:0/3153745519 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070041020 con 0x7f607c15df00 2024-03-06T07:33:36.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.272+0000 7f6078c00640 1 -- 172.21.15.114:0/3153745519 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6070005480 con 0x7f607c15df00 2024-03-06T07:33:36.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.272+0000 7f6078c00640 1 -- 172.21.15.114:0/3153745519 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070007a30 con 0x7f607c15df00 2024-03-06T07:33:36.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.273+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c0c9f70 con 0x7f607c15df00 2024-03-06T07:33:36.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.273+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c15f9a0 con 0x7f607c15df00 2024-03-06T07:33:36.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.274+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c068070 con 0x7f607c15df00 2024-03-06T07:33:36.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.276+0000 7f6078c00640 1 -- 172.21.15.114:0/3153745519 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6070044020 con 0x7f607c15df00 2024-03-06T07:33:36.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.277+0000 7f6078c00640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f605c06b550 0x7f605c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.277+0000 7f6078c00640 1 -- 172.21.15.114:0/3153745519 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f60700abbb0 con 0x7f607c15df00 2024-03-06T07:33:36.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.277+0000 7f6080e00640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f605c06b550 0x7f605c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.279+0000 7f6080e00640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f605c06b550 0x7f605c06da10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f607c068720 tx=0x7f6068008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.281+0000 7f6078c00640 1 -- 172.21.15.114:0/3153745519 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60700b2080 con 0x7f607c15df00 2024-03-06T07:33:36.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.723+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f607c0d1c40 con 0x7f607c15df00 2024-03-06T07:33:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.725+0000 7f6078c00640 1 -- 172.21.15.114:0/3153745519 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f607004b030 con 0x7f607c15df00 2024-03-06T07:33:36.730 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:36.730 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.728+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f605c06b550 msgr2=0x7f605c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.728+0000 7f6083800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f605c06b550 0x7f605c06da10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f607c068720 tx=0x7f6068008040 comp rx=0 tx=0).stop 2024-03-06T07:33:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.728+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c15df00 msgr2=0x7f607c15e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.728+0000 7f6083800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c15df00 0x7f607c15e360 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f6070000c00 tx=0x7f6070001120 comp rx=0 tx=0).stop 2024-03-06T07:33:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.730+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 shutdown_connections 2024-03-06T07:33:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.730+0000 7f6083800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f607c1601a0 0x7f607c160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.730+0000 7f6083800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f605c06b550 0x7f605c06da10 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.730+0000 7f6083800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f607c15df00 0x7f607c15e360 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.730+0000 7f6083800640 1 --2- 172.21.15.114:0/3153745519 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6064004800 0x7f607c158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.730+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 >> 172.21.15.114:0/3153745519 conn(0x7f607c0d0d30 msgr2=0x7f607c06af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.730+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 shutdown_connections 2024-03-06T07:33:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:36.731+0000 7f6083800640 1 -- 172.21.15.114:0/3153745519 wait complete. 2024-03-06T07:33:36.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:37.067 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":326,"msgr_send_messages":326,"msgr_recv_bytes":421800,"msgr_send_bytes":459382,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.058348914,"msgr_running_send_time":0.026360801,"msgr_running_recv_time":0.017460628,"msgr_running_fast_dispatch_time":0.010954084,"msgr_send_messages_queue_lat":{"avgcount":326,"sum":0.041304641,"avgtime":0.000126701},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108288,"msgr_send_encrypted_bytes":101984},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13528,"msgr_send_messages":23188,"msgr_recv_bytes":4598300,"msgr_send_bytes":14619584,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.714615196,"msgr_running_send_time":1.752147880,"msgr_running_recv_time":2.106471190,"msgr_running_fast_dispatch_time":0.037829472,"msgr_send_messages_queue_lat":{"avgcount":23186,"sum":1.466411227,"avgtime":0.000063245},"msgr_handle_ack_lat":{"avgcount":27614,"sum":0.011758592,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":861,"msgr_send_messages":867,"msgr_recv_bytes":226068,"msgr_send_bytes":22061799,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.214365813,"msgr_running_send_time":0.134936421,"msgr_running_recv_time":0.025648895,"msgr_running_fast_dispatch_time":0.044979868,"msgr_send_messages_queue_lat":{"avgcount":865,"sum":0.115059607,"avgtime":0.000133016},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":2581,"complete_latency":{"avgcount":85,"sum":9.869790382,"avgtime":0.116115180}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3336,"sum":2.296855221,"avgtime":0.000688505}},"mds":{"request":6412,"reply":6409,"reply_latency":{"avgcount":6409,"sum":699.645047501,"avgtime":0.109166023},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5407,"inodes_top":0,"inodes_bottom":5407,"inodes_pin_tail":0,"inodes_pinned":5407,"inodes_expired":0,"inodes_with_caps":680,"caps":680,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":919,"root_rbytes":7228416,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":116,"ceph_cap_op_grant":6791,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5336,"handle_client_caps_dirty":3000,"handle_client_cap_release":778,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":848,"num_strays_delayed":170,"num_strays_enqueuing":0,"strays_created":3200,"strays_enqueued":2352,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":14498,"evex":0,"evtrm":0,"ev":14501,"evexg":0,"evexd":0,"segadd":18,"segex":0,"segtrm":0,"seg":20,"segmjr":3,"segexg":0,"segexd":0,"expos":4194304,"wrpos":33935135,"rdpos":4195244,"jlat":{"avgcount":20,"sum":3.011070148,"avgtime":0.150553507},"replayed":3},"mds_mem":{"ino":2260,"ino+":3214,"ino-":954,"dir":13,"dir+":13,"dir-":0,"dn":5458,"dn+":6412,"dn-":954,"cap":680,"cap+":3202,"cap-":2522,"rss":109932,"heap":248164},"mds_server":{"dispatch_client_request":6413,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.022992058,"avgtime":0.002507185},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5780,"average_load":5780,"avg_session_uptime":981,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":28033682,"buffer_anon_items":10114,"buffer_meta_bytes":352,"buffer_meta_items":4,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13051282,"mds_co_items":202416,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":2147,"op_send_bytes":30404132,"op_resend":0,"op_reply":2141,"op_latency":{"avgcount":2141,"sum":6.729860025,"avgtime":0.003143325},"op_inflight":5,"oplen_avg":{"avgcount":2147,"sum":2208},"op":2147,"op_r":52,"op_w":2095,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":24,"osdop_write":38,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2036,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":24,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":2012,"pq_executed":2012,"pq_item_in_journal":340},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14712,"get_sum":3919524,"get_or_fail_fail":0,"get_or_fail_success":14712,"take":0,"take_sum":0,"put":14712,"put_sum":3919524,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2147,"take_sum":30642618,"put":2141,"put_sum":30642618,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2147,"take_sum":2147,"put":2141,"put_sum":2141,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2352,"get_sum":237552,"get_or_fail_fail":0,"get_or_fail_success":2352,"take":0,"take_sum":0,"put":16,"put_sum":237552,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":0,"max":3758096384,"get_started":0,"get":14498,"get_sum":29739633,"get_or_fail_fail":0,"get_or_fail_success":14498,"take":0,"take_sum":0,"put":22,"put_sum":29739633,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:37.072 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 421800, "msgr_recv_encrypted_bytes": 108288, "msgr_recv_messages": 326, "msgr_running_fast_dispatch_time": 0.010954084, "msgr_running_recv_time": 0.017460628, "msgr_running_send_time": 0.026360801, "msgr_running_total_time": 0.058348914, "msgr_send_bytes": 459382, "msgr_send_encrypted_bytes": 101984, "msgr_send_messages": 326, "msgr_send_messages_queue_lat": { "avgcount": 326, "avgtime": 0.000126701, "sum": 0.041304641 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 27614, "avgtime": 4.25e-07, "sum": 0.011758592 }, "msgr_recv_bytes": 4598300, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 13528, "msgr_running_fast_dispatch_time": 0.037829472, "msgr_running_recv_time": 2.10647119, "msgr_running_send_time": 1.75214788, "msgr_running_total_time": 2.714615196, "msgr_send_bytes": 14619584, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 23188, "msgr_send_messages_queue_lat": { "avgcount": 23186, "avgtime": 6.3245e-05, "sum": 1.466411227 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 226068, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 861, "msgr_running_fast_dispatch_time": 0.044979868, "msgr_running_recv_time": 0.025648895, "msgr_running_send_time": 0.134936421, "msgr_running_total_time": 0.214365813, "msgr_send_bytes": 22061799, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 867, "msgr_send_messages_queue_lat": { "avgcount": 865, "avgtime": 0.000133016, "sum": 0.115059607 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 85, "avgtime": 0.11611518, "sum": 9.869790382 }, "queue_len": 2581 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3336, "avgtime": 0.000688505, "sum": 2.296855221 }, "queue_len": 0 }, "mds": { "caps": 680, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6791, "ceph_cap_op_revoke": 116, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 778, "handle_client_caps": 5336, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5407, "inodes_bottom": 5407, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5407, "inodes_top": 0, "inodes_with_caps": 680, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6409, "reply_latency": { "avgcount": 6409, "avgtime": 0.109166023, "sum": 699.645047501 }, "request": 6412, "root_rbytes": 7228416, "root_rfiles": 919, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 848, "num_strays_delayed": 170, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3200, "strays_enqueued": 2352, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 14501, "evadd": 14498, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 20, "avgtime": 0.150553507, "sum": 3.011070148 }, "rdpos": 4195244, "replayed": 3, "seg": 20, "segadd": 18, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 3, "segtrm": 0, "wrpos": 33935135 }, "mds_mem": { "cap": 680, "cap+": 3202, "cap-": 2522, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5458, "dn+": 6412, "dn-": 954, "heap": 248164, "ino": 2260, "ino+": 3214, "ino-": 954, "rss": 109932 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6413, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002507185, "sum": 8.022992058 } }, "mds_sessions": { "average_load": 5780, "avg_session_uptime": 981, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5780 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 28033682, "buffer_anon_items": 10114, "buffer_meta_bytes": 352, "buffer_meta_items": 4, "mds_co_bytes": 13051282, "mds_co_items": 202416, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 4, "omap_rd": 48, "omap_wr": 24, "op": 2147, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 2141, "avgtime": 0.003143325, "sum": 6.729860025 }, "op_pg": 0, "op_r": 52, "op_reply": 2141, "op_resend": 0, "op_rmw": 0, "op_send": 2147, "op_send_bytes": 30404132, "op_w": 2095, "oplen_avg": { "avgcount": 2147, "sum": 2208 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 2036, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 38, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2012, "pq_executed_ops": 2012, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 340 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14712, "get_or_fail_fail": 0, "get_or_fail_success": 14712, "get_started": 0, "get_sum": 3919524, "max": 104857600, "put": 14712, "put_sum": 3919524, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 2141, "put_sum": 30642618, "take": 2147, "take_sum": 30642618, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 2141, "put_sum": 2141, "take": 2147, "take_sum": 2147, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2352, "get_or_fail_fail": 0, "get_or_fail_success": 2352, "get_started": 0, "get_sum": 237552, "max": 3758096384, "put": 16, "put_sum": 237552, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 14498, "get_or_fail_fail": 0, "get_or_fail_success": 14498, "get_started": 0, "get_sum": 29739633, "max": 3758096384, "put": 22, "put_sum": 29739633, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:37.072 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/848, 2012/3200 2024-03-06T07:33:37.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:37.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.700+0000 7f4e71200640 1 Processor -- start 2024-03-06T07:33:37.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.700+0000 7f4e71200640 1 -- start start 2024-03-06T07:33:37.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.701+0000 7f4e71200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4e6c1617a0 0x7f4e6c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.701+0000 7f4e71200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c0cf070 con 0x7f4e6c0d0cf0 2024-03-06T07:33:37.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.701+0000 7f4e71200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c0cf350 con 0x7f4e6c0d1e10 2024-03-06T07:33:37.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.701+0000 7f4e71200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6c0b2120 con 0x7f4e6c1617a0 2024-03-06T07:33:37.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e63e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4e6c0d0cf0 0x7f4e6c06c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43260/0 (socket says 172.21.15.114:43260) 2024-03-06T07:33:37.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e63e00640 1 -- 172.21.15.114:0/2219385704 learned_addr learned my addr 172.21.15.114:0/2219385704 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572257027 0 0) 0x7f4e6c0cf070 con 0x7f4e6c0d0cf0 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e50003520 con 0x7f4e6c0d0cf0 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e69e00640 1 --2- 172.21.15.114:0/2219385704 >> v2:172.21.15.177:3300/0 conn(0x7f4e6c1617a0 0x7f4e6c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632765409 0 0) 0x7f4e50003520 con 0x7f4e6c0d0cf0 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 >> v2:172.21.15.177:3300/0 conn(0x7f4e6c1617a0 msgr2=0x7f4e6c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e69400640 1 --2- 172.21.15.114:0/2219385704 >> v2:172.21.15.177:3300/0 conn(0x7f4e6c1617a0 0x7f4e6c0ce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.702+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 >> v1:172.21.15.177:6789/0 conn(0x7f4e6c0d1e10 legacy=0x7f4e6c0b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0cf630 con 0x7f4e6c0d0cf0 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f4e54001670 con 0x7f4e6c0d0cf0 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 >> v1:172.21.15.114:6789/0 conn(0x7f4e6c0d0cf0 legacy=0x7f4e6c06c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e500039b0 0x7f4e50003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e50004800 0x7f4e50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e50005210 0x7f4e5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0b2120 con 0x7f4e500039b0 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5000c140 con 0x7f4e50004800 2024-03-06T07:33:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5000c340 con 0x7f4e50005210 2024-03-06T07:33:37.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69e00640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e500039b0 0x7f4e50003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e63e00640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e50004800 0x7f4e50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69e00640 1 -- 172.21.15.114:0/2219385704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e50005210 msgr2=0x7f4e5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69e00640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e50005210 0x7f4e5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69e00640 1 -- 172.21.15.114:0/2219385704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e50004800 msgr2=0x7f4e50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69e00640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e50004800 0x7f4e50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e69e00640 1 -- 172.21.15.114:0/2219385704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e50003520 con 0x7f4e500039b0 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.703+0000 7f4e6a800640 1 -- 172.21.15.114:0/2219385704 reap_dead start 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.704+0000 7f4e69e00640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e500039b0 0x7f4e50003df0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00ba30 tx=0x7f4e50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1403ce5bbbe618e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c036020 con 0x7f4e500039b0 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c032680 con 0x7f4e500039b0 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e69400640 1 -- 172.21.15.114:0/2219385704 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00b600 con 0x7f4e500039b0 2024-03-06T07:33:37.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e71200640 1 -- 172.21.15.114:0/2219385704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e500039b0 msgr2=0x7f4e50003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e71200640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e500039b0 0x7f4e50003df0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00ba30 tx=0x7f4e50005830 comp rx=0 tx=0).stop 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e71200640 1 -- 172.21.15.114:0/2219385704 shutdown_connections 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e71200640 1 --2- 172.21.15.114:0/2219385704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e500039b0 0x7f4e50003df0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e71200640 1 -- 172.21.15.114:0/2219385704 >> 172.21.15.114:0/2219385704 conn(0x7f4e6c09aa30 msgr2=0x7f4e6c09ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e71200640 1 -- 172.21.15.114:0/2219385704 shutdown_connections 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.705+0000 7f4e71200640 1 -- 172.21.15.114:0/2219385704 wait complete. 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.707+0000 7f4e71200640 1 Processor -- start 2024-03-06T07:33:37.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.707+0000 7f4e71200640 1 -- start start 2024-03-06T07:33:37.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e71200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e500039b0 0x7f4e6c1f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e71200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e6c15e570 0x7f4e6c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e71200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15dbf0 0x7f4e6c160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e71200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c161d00 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e71200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c161b80 con 0x7f4e500039b0 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e71200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0d22a0 con 0x7f4e6c15e570 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e6a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15dbf0 0x7f4e6c160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e6a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15dbf0 0x7f4e6c160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54578/0 (socket says 172.21.15.114:54578) 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.708+0000 7f4e6a800640 1 -- 172.21.15.114:0/1146254638 learned_addr learned my addr 172.21.15.114:0/1146254638 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e6a800640 1 -- 172.21.15.114:0/1146254638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e6c15e570 msgr2=0x7f4e6c0c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e6a800640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e6c15e570 0x7f4e6c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e6a800640 1 -- 172.21.15.114:0/1146254638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e500039b0 msgr2=0x7f4e6c1f72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e6a800640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e500039b0 0x7f4e6c1f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e6a800640 1 -- 172.21.15.114:0/1146254638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0b7340 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e6a800640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15dbf0 0x7f4e6c160160 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0c5960 tx=0x7f4e6400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e62000640 1 -- 172.21.15.114:0/1146254638 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64005d40 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e62000640 1 -- 172.21.15.114:0/1146254638 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6400a640 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e62000640 1 -- 172.21.15.114:0/1146254638 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6400aa30 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c0b7620 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.709+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c160e70 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.710+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c0b7860 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.711+0000 7f4e62000640 1 -- 172.21.15.114:0/1146254638 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e64011070 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.712+0000 7f4e62000640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e4806b4d0 0x7f4e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.712+0000 7f4e62000640 1 -- 172.21.15.114:0/1146254638 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f4e640895b0 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.712+0000 7f4e69e00640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e4806b4d0 0x7f4e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.713+0000 7f4e69e00640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e4806b4d0 0x7f4e4806d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00ba00 tx=0x7f4e5c032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.716+0000 7f4e62000640 1 -- 172.21.15.114:0/1146254638 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e6c0b7860 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.945+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e6c0d2d10 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.946+0000 7f4e62000640 1 -- 172.21.15.114:0/1146254638 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f4e64058410 con 0x7f4e6c15dbf0 2024-03-06T07:33:37.949 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:37.949 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:37.949 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:37.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e4806b4d0 msgr2=0x7f4e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e4806b4d0 0x7f4e4806d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00ba00 tx=0x7f4e5c032000 comp rx=0 tx=0).stop 2024-03-06T07:33:37.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15dbf0 msgr2=0x7f4e6c160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15dbf0 0x7f4e6c160160 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0c5960 tx=0x7f4e6400cb90 comp rx=0 tx=0).stop 2024-03-06T07:33:37.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 shutdown_connections 2024-03-06T07:33:37.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e4806b4d0 0x7f4e4806d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e6c15dbf0 0x7f4e6c160160 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e6c15e570 0x7f4e6c0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.948+0000 7f4e71200640 1 --2- 172.21.15.114:0/1146254638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e500039b0 0x7f4e6c1f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.949+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 >> 172.21.15.114:0/1146254638 conn(0x7f4e6c09aa30 msgr2=0x7f4e6c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.949+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 shutdown_connections 2024-03-06T07:33:37.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:37.949+0000 7f4e71200640 1 -- 172.21.15.114:0/1146254638 wait complete. 2024-03-06T07:33:37.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:33:38.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.311+0000 7f341e600640 1 Processor -- start 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.312+0000 7f341e600640 1 -- start start 2024-03-06T07:33:38.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.315+0000 7f341e600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f341815cc90 0x7f341806d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.315+0000 7f341e600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3418162b00 0x7f3418162f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.315+0000 7f341e600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34181635a0 con 0x7f34180d0070 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.315+0000 7f341e600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341806d940 con 0x7f3418162b00 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.315+0000 7f341e600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3418163880 con 0x7f341815cc90 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.316+0000 7f3417400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f341815cc90 0x7f341806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.316+0000 7f3417400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f341815cc90 0x7f341806d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:53670/0 (socket says 172.21.15.114:53670) 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.316+0000 7f3417400640 1 -- 172.21.15.114:0/165509082 learned_addr learned my addr 172.21.15.114:0/165509082 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.316+0000 7f3417400640 1 -- 172.21.15.114:0/165509082 >> v2:172.21.15.114:3300/0 conn(0x7f3418162b00 msgr2=0x7f3418162f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.316+0000 7f3417400640 1 --2- 172.21.15.114:0/165509082 >> v2:172.21.15.114:3300/0 conn(0x7f3418162b00 0x7f3418162f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.316+0000 7f3417400640 1 -- 172.21.15.114:0/165509082 >> v1:172.21.15.177:6789/0 conn(0x7f34180d0070 legacy=0x7f34180acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.316+0000 7f3417400640 1 -- 172.21.15.114:0/165509082 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341806cc00 con 0x7f341815cc90 2024-03-06T07:33:38.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.318+0000 7f3417400640 1 --2- 172.21.15.114:0/165509082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f341815cc90 0x7f341806d400 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3410009e50 tx=0x7f341002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a8ee7f82e1eb29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.318+0000 7f3416000640 1 -- 172.21.15.114:0/165509082 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341002cb00 con 0x7f341815cc90 2024-03-06T07:33:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.318+0000 7f3416000640 1 -- 172.21.15.114:0/165509082 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341002cca0 con 0x7f341815cc90 2024-03-06T07:33:38.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.318+0000 7f3416000640 1 -- 172.21.15.114:0/165509082 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410034610 con 0x7f341815cc90 2024-03-06T07:33:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.319+0000 7f341e600640 1 -- 172.21.15.114:0/165509082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f341815cc90 msgr2=0x7f341806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.319+0000 7f341e600640 1 --2- 172.21.15.114:0/165509082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f341815cc90 0x7f341806d400 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3410009e50 tx=0x7f341002bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.319+0000 7f341e600640 1 -- 172.21.15.114:0/165509082 shutdown_connections 2024-03-06T07:33:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.319+0000 7f341e600640 1 --2- 172.21.15.114:0/165509082 >> v2:172.21.15.114:3300/0 conn(0x7f3418162b00 0x7f3418162f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.319+0000 7f341e600640 1 --2- 172.21.15.114:0/165509082 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f341815cc90 0x7f341806d400 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.319+0000 7f341e600640 1 -- 172.21.15.114:0/165509082 >> 172.21.15.114:0/165509082 conn(0x7f341806b360 msgr2=0x7f341806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.320+0000 7f341e600640 1 -- 172.21.15.114:0/165509082 shutdown_connections 2024-03-06T07:33:38.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.320+0000 7f341e600640 1 -- 172.21.15.114:0/165509082 wait complete. 2024-03-06T07:33:38.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.323+0000 7f341e600640 1 Processor -- start 2024-03-06T07:33:38.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.323+0000 7f341e600640 1 -- start start 2024-03-06T07:33:38.326 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:33:38.327 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:33:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.324+0000 7f341e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180d0070 0x7f3418161830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.324+0000 7f341e600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f341815cc90 0x7f34181f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.324+0000 7f341e600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3418162b00 0x7f34181647b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.324+0000 7f341e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b4fe0 con 0x7f34180d0070 2024-03-06T07:33:38.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.324+0000 7f341e600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b4e60 con 0x7f341815cc90 2024-03-06T07:33:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.324+0000 7f341e600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34180d0500 con 0x7f3418162b00 2024-03-06T07:33:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.325+0000 7f3416a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f341815cc90 0x7f34181f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.325+0000 7f3416a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f341815cc90 0x7f34181f5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:37598/0 (socket says 172.21.15.114:37598) 2024-03-06T07:33:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.325+0000 7f3416a00640 1 -- 172.21.15.114:0/3071688192 learned_addr learned my addr 172.21.15.114:0/3071688192 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.325+0000 7f3417400640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180d0070 0x7f3418161830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.325+0000 7f3417e00640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3418162b00 0x7f34181647b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.326+0000 7f3417400640 1 -- 172.21.15.114:0/3071688192 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3418162b00 msgr2=0x7f34181647b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.326+0000 7f3417400640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3418162b00 0x7f34181647b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.326+0000 7f3417400640 1 -- 172.21.15.114:0/3071688192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f341815cc90 msgr2=0x7f34181f5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.326+0000 7f3417400640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f341815cc90 0x7f34181f5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.326+0000 7f3417400640 1 -- 172.21.15.114:0/3071688192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34181f48a0 con 0x7f34180d0070 2024-03-06T07:33:38.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.327+0000 7f3417400640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180d0070 0x7f3418161830 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3410032000 tx=0x7f3410004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.327+0000 7f340f400640 1 -- 172.21.15.114:0/3071688192 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410043020 con 0x7f34180d0070 2024-03-06T07:33:38.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.327+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34181f4b80 con 0x7f34180d0070 2024-03-06T07:33:38.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.328+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34181f5140 con 0x7f34180d0070 2024-03-06T07:33:38.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.328+0000 7f340f400640 1 -- 172.21.15.114:0/3071688192 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341002ce50 con 0x7f34180d0070 2024-03-06T07:33:38.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.328+0000 7f340f400640 1 -- 172.21.15.114:0/3071688192 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341003e450 con 0x7f34180d0070 2024-03-06T07:33:38.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.329+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f0005290 con 0x7f34180d0070 2024-03-06T07:33:38.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.331+0000 7f340f400640 1 -- 172.21.15.114:0/3071688192 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3410046020 con 0x7f34180d0070 2024-03-06T07:33:38.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.333+0000 7f340f400640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33f806b4d0 0x7f33f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.333+0000 7f340f400640 1 -- 172.21.15.114:0/3071688192 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f34100ac640 con 0x7f34180d0070 2024-03-06T07:33:38.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.334+0000 7f3416a00640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33f806b4d0 0x7f33f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.335+0000 7f3416a00640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33f806b4d0 0x7f33f806d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f34180adde0 tx=0x7f3404006f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.339+0000 7f340f400640 1 -- 172.21.15.114:0/3071688192 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f341004d030 con 0x7f34180d0070 2024-03-06T07:33:38.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.578+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33f0005580 con 0x7f34180d0070 2024-03-06T07:33:38.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.578+0000 7f340f400640 1 -- 172.21.15.114:0/3071688192 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f341007b370 con 0x7f34180d0070 2024-03-06T07:33:38.581 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:38.581 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:38.581 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:38.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.580+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33f806b4d0 msgr2=0x7f33f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.580+0000 7f341e600640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33f806b4d0 0x7f33f806d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f34180adde0 tx=0x7f3404006f90 comp rx=0 tx=0).stop 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.580+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180d0070 msgr2=0x7f3418161830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.580+0000 7f341e600640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180d0070 0x7f3418161830 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3410032000 tx=0x7f3410004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 shutdown_connections 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3418162b00 0x7f34181647b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f341815cc90 0x7f34181f5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f33f806b4d0 0x7f33f806d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 --2- 172.21.15.114:0/3071688192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34180d0070 0x7f3418161830 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 >> 172.21.15.114:0/3071688192 conn(0x7f341806b360 msgr2=0x7f34180b0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 shutdown_connections 2024-03-06T07:33:38.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:38.581+0000 7f341e600640 1 -- 172.21.15.114:0/3071688192 wait complete. 2024-03-06T07:33:38.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:38.856 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":501,"msgr_send_messages":501,"msgr_recv_bytes":536094,"msgr_send_bytes":589737,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.080116590,"msgr_running_send_time":0.037478798,"msgr_running_recv_time":0.022150311,"msgr_running_fast_dispatch_time":0.015688510,"msgr_send_messages_queue_lat":{"avgcount":501,"sum":0.064811864,"avgtime":0.000129364},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108288,"msgr_send_encrypted_bytes":103776},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14040,"msgr_send_messages":24727,"msgr_recv_bytes":4746745,"msgr_send_bytes":15156463,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.843602166,"msgr_running_send_time":1.835558932,"msgr_running_recv_time":2.124521213,"msgr_running_fast_dispatch_time":0.053042697,"msgr_send_messages_queue_lat":{"avgcount":24726,"sum":1.785119543,"avgtime":0.000072196},"msgr_handle_ack_lat":{"avgcount":28200,"sum":0.011995042,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1426,"msgr_send_messages":1428,"msgr_recv_bytes":377901,"msgr_send_bytes":22714959,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.275530725,"msgr_running_send_time":0.160681242,"msgr_running_recv_time":0.037446280,"msgr_running_fast_dispatch_time":0.064626378,"msgr_send_messages_queue_lat":{"avgcount":1426,"sum":0.222591237,"avgtime":0.000156094},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":117,"sum":11.240900442,"avgtime":0.096076072}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5017,"sum":2.464322679,"avgtime":0.000491194}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":700.166907368,"avgtime":0.109230406},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":11,"dir_split":0,"dir_merge":0,"inodes":3720,"inodes_top":0,"inodes_bottom":3720,"inodes_pin_tail":0,"inodes_pinned":3720,"inodes_expired":0,"inodes_with_caps":170,"caps":170,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":118,"ceph_cap_op_grant":7307,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5338,"handle_client_caps_dirty":3000,"handle_client_cap_release":781,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":169,"num_strays_delayed":169,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3032,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15182,"evex":15030,"evtrm":15030,"ev":15184,"evexg":0,"evexd":15029,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":34254966,"wrpos":34351906,"rdpos":4195244,"jlat":{"avgcount":25,"sum":4.069661213,"avgtime":0.162786448},"replayed":3},"mds_mem":{"ino":862,"ino+":3214,"ino-":2352,"dir":13,"dir+":13,"dir-":0,"dn":4060,"dn+":6412,"dn-":2352,"cap":170,"cap+":3202,"cap-":3032,"rss":117100,"heap":248164},"mds_server":{"dispatch_client_request":6416,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.521859867,"avgtime":0.521859867},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.022992058,"avgtime":0.002507185},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5746,"average_load":5746,"avg_session_uptime":981,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":28033682,"buffer_anon_items":10114,"buffer_meta_bytes":352,"buffer_meta_items":4,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13051282,"mds_co_items":202416,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3385,"op_send_bytes":31058312,"op_resend":0,"op_reply":3385,"op_latency":{"avgcount":3385,"sum":14.653695240,"avgtime":0.004329008},"op_inflight":0,"oplen_avg":{"avgcount":3385,"sum":3827},"op":3385,"op_r":58,"op_w":3327,"op_rmw":0,"op_pg":0,"osdop_stat":11,"osdop_create":181,"osdop_read":30,"osdop_write":47,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3057,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":362,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":45,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3032,"pq_executed":3032,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15963,"get_sum":4221252,"get_or_fail_fail":0,"get_or_fail_success":15963,"take":0,"take_sum":0,"put":15963,"put_sum":4221252,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":31365478,"put":3385,"put_sum":31365478,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":3385,"put":3385,"put_sum":3385,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3032,"get_sum":306232,"get_or_fail_fail":0,"get_or_fail_success":3032,"take":0,"take_sum":0,"put":22,"put_sum":306232,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":0,"max":3758096384,"get_started":0,"get":15182,"get_sum":30156404,"get_or_fail_fail":0,"get_or_fail_success":15182,"take":0,"take_sum":0,"put":25,"put_sum":30156404,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:38.858 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 536094, "msgr_recv_encrypted_bytes": 108288, "msgr_recv_messages": 501, "msgr_running_fast_dispatch_time": 0.01568851, "msgr_running_recv_time": 0.022150311, "msgr_running_send_time": 0.037478798, "msgr_running_total_time": 0.08011659, "msgr_send_bytes": 589737, "msgr_send_encrypted_bytes": 103776, "msgr_send_messages": 501, "msgr_send_messages_queue_lat": { "avgcount": 501, "avgtime": 0.000129364, "sum": 0.064811864 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28200, "avgtime": 4.25e-07, "sum": 0.011995042 }, "msgr_recv_bytes": 4746745, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 14040, "msgr_running_fast_dispatch_time": 0.053042697, "msgr_running_recv_time": 2.124521213, "msgr_running_send_time": 1.835558932, "msgr_running_total_time": 2.843602166, "msgr_send_bytes": 15156463, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 24727, "msgr_send_messages_queue_lat": { "avgcount": 24726, "avgtime": 7.2196e-05, "sum": 1.785119543 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 377901, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1426, "msgr_running_fast_dispatch_time": 0.064626378, "msgr_running_recv_time": 0.03744628, "msgr_running_send_time": 0.160681242, "msgr_running_total_time": 0.275530725, "msgr_send_bytes": 22714959, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1428, "msgr_send_messages_queue_lat": { "avgcount": 1426, "avgtime": 0.000156094, "sum": 0.222591237 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 117, "avgtime": 0.096076072, "sum": 11.240900442 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5017, "avgtime": 0.000491194, "sum": 2.464322679 }, "queue_len": 0 }, "mds": { "caps": 170, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7307, "ceph_cap_op_revoke": 118, "ceph_cap_op_trunc": 0, "dir_commit": 11, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 781, "handle_client_caps": 5338, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3720, "inodes_bottom": 3720, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3720, "inodes_top": 0, "inodes_with_caps": 170, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.109230406, "sum": 700.166907368 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 169, "num_strays_delayed": 169, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3032, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15184, "evadd": 15182, "evex": 15030, "evexd": 15029, "evexg": 0, "evlrg": 0, "evtrm": 15030, "expos": 34254966, "jlat": { "avgcount": 25, "avgtime": 0.162786448, "sum": 4.069661213 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 34351906 }, "mds_mem": { "cap": 170, "cap+": 3202, "cap-": 3032, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4060, "dn+": 6412, "dn-": 2352, "heap": 248164, "ino": 862, "ino+": 3214, "ino-": 2352, "rss": 117100 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6416, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.521859867, "sum": 0.521859867 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002507185, "sum": 8.022992058 } }, "mds_sessions": { "average_load": 5746, "avg_session_uptime": 981, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5746 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 28033682, "buffer_anon_items": 10114, "buffer_meta_bytes": 352, "buffer_meta_items": 4, "mds_co_bytes": 13051282, "mds_co_items": 202416, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 45, "op": 3385, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3385, "avgtime": 0.004329008, "sum": 14.65369524 }, "op_pg": 0, "op_r": 58, "op_reply": 3385, "op_resend": 0, "op_rmw": 0, "op_send": 3385, "op_send_bytes": 31058312, "op_w": 3327, "oplen_avg": { "avgcount": 3385, "sum": 3827 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 181, "osdop_delete": 3057, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 362, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 11, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 47, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3032, "pq_executed_ops": 3032, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15963, "get_or_fail_fail": 0, "get_or_fail_success": 15963, "get_started": 0, "get_sum": 4221252, "max": 104857600, "put": 15963, "put_sum": 4221252, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3385, "put_sum": 31365478, "take": 3385, "take_sum": 31365478, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3385, "put_sum": 3385, "take": 3385, "take_sum": 3385, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3032, "get_or_fail_fail": 0, "get_or_fail_success": 3032, "get_started": 0, "get_sum": 306232, "max": 3758096384, "put": 22, "put_sum": 306232, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 15182, "get_or_fail_fail": 0, "get_or_fail_success": 15182, "get_started": 0, "get_sum": 30156404, "max": 3758096384, "put": 25, "put_sum": 30156404, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:38.858 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/169, 3032/3201 2024-03-06T07:33:39.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:40.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.065+0000 7fd9b2800640 1 Processor -- start 2024-03-06T07:33:40.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.065+0000 7fd9b2800640 1 -- start start 2024-03-06T07:33:40.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.067+0000 7fd9b2800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.067+0000 7fd9b2800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd9ac0acd60 0x7fd9ac15fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.067+0000 7fd9b2800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0b5160 con 0x7fd9ac15be90 2024-03-06T07:33:40.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.067+0000 7fd9b2800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0b4f30 con 0x7fd9ac0acd60 2024-03-06T07:33:40.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.067+0000 7fd9b2800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0af240 con 0x7fd9ac0d0780 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9abe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd9ac0acd60 0x7fd9ac15fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac0b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:53686/0 (socket says 172.21.15.114:53686) 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9a3e00640 1 -- 172.21.15.114:0/3683204786 learned_addr learned my addr 172.21.15.114:0/3683204786 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9ab400640 1 --1- 172.21.15.114:0/3683204786 >> v1:172.21.15.177:6789/0 conn(0x7fd9ac15be90 0x7fd9ac0d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:35914/0 (socket says 172.21.15.114:35914) 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9a3e00640 1 -- 172.21.15.114:0/3683204786 >> v2:172.21.15.177:3300/0 conn(0x7fd9ac0acd60 msgr2=0x7fd9ac15fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9a3e00640 1 --2- 172.21.15.114:0/3683204786 >> v2:172.21.15.177:3300/0 conn(0x7fd9ac0acd60 0x7fd9ac15fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9a3e00640 1 -- 172.21.15.114:0/3683204786 >> v1:172.21.15.177:6789/0 conn(0x7fd9ac15be90 legacy=0x7fd9ac0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.068+0000 7fd9a3e00640 1 -- 172.21.15.114:0/3683204786 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac0ae910 con 0x7fd9ac0d0780 2024-03-06T07:33:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.070+0000 7fd9a3e00640 1 --2- 172.21.15.114:0/3683204786 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac0b49f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd994009870 tx=0x7fd99402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fac0e92d33430f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.070+0000 7fd9aaa00640 1 -- 172.21.15.114:0/3683204786 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994034070 con 0x7fd9ac0d0780 2024-03-06T07:33:40.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.070+0000 7fd9aaa00640 1 -- 172.21.15.114:0/3683204786 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd994030840 con 0x7fd9ac0d0780 2024-03-06T07:33:40.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.070+0000 7fd9aaa00640 1 -- 172.21.15.114:0/3683204786 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994030c10 con 0x7fd9ac0d0780 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 -- 172.21.15.114:0/3683204786 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 msgr2=0x7fd9ac0b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 --2- 172.21.15.114:0/3683204786 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac0b49f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd994009870 tx=0x7fd99402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 -- 172.21.15.114:0/3683204786 shutdown_connections 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 --2- 172.21.15.114:0/3683204786 >> v2:172.21.15.177:3300/0 conn(0x7fd9ac0acd60 0x7fd9ac15fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 --2- 172.21.15.114:0/3683204786 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac0b49f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 -- 172.21.15.114:0/3683204786 >> 172.21.15.114:0/3683204786 conn(0x7fd9ac06b360 msgr2=0x7fd9ac06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 -- 172.21.15.114:0/3683204786 shutdown_connections 2024-03-06T07:33:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.071+0000 7fd9b2800640 1 -- 172.21.15.114:0/3683204786 wait complete. 2024-03-06T07:33:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.073+0000 7fd9b2800640 1 Processor -- start 2024-03-06T07:33:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.073+0000 7fd9b2800640 1 -- start start 2024-03-06T07:33:40.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9ac0acd60 0x7fd9ac1465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9b2800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac09cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9b2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ac15be90 0x7fd9ac09d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9b2800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0b2c80 con 0x7fd9ac0acd60 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9b2800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0aeed0 con 0x7fd9ac15be90 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9b2800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac0b2e00 con 0x7fd9ac0d0780 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9ab400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9ac0acd60 0x7fd9ac1465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9ab400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9ac0acd60 0x7fd9ac1465a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51444/0 (socket says 172.21.15.114:51444) 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.074+0000 7fd9ab400640 1 -- 172.21.15.114:0/28770100 learned_addr learned my addr 172.21.15.114:0/28770100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9a3e00640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac09cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9abe00640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ac15be90 0x7fd9ac09d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9ab400640 1 -- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 msgr2=0x7fd9ac09cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9ab400640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac09cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9ab400640 1 -- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ac15be90 msgr2=0x7fd9ac09d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9ab400640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ac15be90 0x7fd9ac09d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9ab400640 1 -- 172.21.15.114:0/28770100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac148f80 con 0x7fd9ac0acd60 2024-03-06T07:33:40.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.075+0000 7fd9ab400640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9ac0acd60 0x7fd9ac1465a0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c3c0 tx=0x7fd99c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.076+0000 7fd9a3400640 1 -- 172.21.15.114:0/28770100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c0074e0 con 0x7fd9ac0acd60 2024-03-06T07:33:40.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.076+0000 7fd9a3400640 1 -- 172.21.15.114:0/28770100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c007680 con 0x7fd9ac0acd60 2024-03-06T07:33:40.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.076+0000 7fd9a3400640 1 -- 172.21.15.114:0/28770100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c004a00 con 0x7fd9ac0acd60 2024-03-06T07:33:40.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.076+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac1f8130 con 0x7fd9ac0acd60 2024-03-06T07:33:40.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.076+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac149520 con 0x7fd9ac0acd60 2024-03-06T07:33:40.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.077+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac068070 con 0x7fd9ac0acd60 2024-03-06T07:33:40.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.078+0000 7fd9a3400640 1 -- 172.21.15.114:0/28770100 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd99c01a020 con 0x7fd9ac0acd60 2024-03-06T07:33:40.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.079+0000 7fd9a3400640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd98806b4d0 0x7fd98806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.079+0000 7fd9a3400640 1 -- 172.21.15.114:0/28770100 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd99c089ac0 con 0x7fd9ac0acd60 2024-03-06T07:33:40.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.079+0000 7fd9a3e00640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd98806b4d0 0x7fd98806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.080+0000 7fd9a3e00640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd98806b4d0 0x7fd98806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd994036000 tx=0x7fd99403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.083+0000 7fd9a3400640 1 -- 172.21.15.114:0/28770100 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd99c0587f0 con 0x7fd9ac0acd60 2024-03-06T07:33:40.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.374+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9ac0b27e0 con 0x7fd9ac0acd60 2024-03-06T07:33:40.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.375+0000 7fd9a3400640 1 -- 172.21.15.114:0/28770100 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7fd99c0580f0 con 0x7fd9ac0acd60 2024-03-06T07:33:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:40.378 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:40.378 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:40.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.377+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd98806b4d0 msgr2=0x7fd98806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.377+0000 7fd9b2800640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd98806b4d0 0x7fd98806d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd994036000 tx=0x7fd99403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:40.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.377+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9ac0acd60 msgr2=0x7fd9ac1465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.377+0000 7fd9b2800640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9ac0acd60 0x7fd9ac1465a0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c3c0 tx=0x7fd99c00c880 comp rx=0 tx=0).stop 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 shutdown_connections 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ac15be90 0x7fd9ac09d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9ac0d0780 0x7fd9ac09cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd98806b4d0 0x7fd98806d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 --2- 172.21.15.114:0/28770100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9ac0acd60 0x7fd9ac1465a0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 >> 172.21.15.114:0/28770100 conn(0x7fd9ac06b360 msgr2=0x7fd9ac0b02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 shutdown_connections 2024-03-06T07:33:40.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:40.378+0000 7fd9b2800640 1 -- 172.21.15.114:0/28770100 wait complete. 2024-03-06T07:33:40.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:40.614 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":502,"msgr_send_messages":502,"msgr_recv_bytes":536366,"msgr_send_bytes":590361,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.080439914,"msgr_running_send_time":0.037648697,"msgr_running_recv_time":0.022229735,"msgr_running_fast_dispatch_time":0.015712982,"msgr_send_messages_queue_lat":{"avgcount":502,"sum":0.064851165,"avgtime":0.000129185},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108560,"msgr_send_encrypted_bytes":104400},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14041,"msgr_send_messages":24727,"msgr_recv_bytes":4747163,"msgr_send_bytes":15156463,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.843944193,"msgr_running_send_time":1.835712489,"msgr_running_recv_time":2.124582995,"msgr_running_fast_dispatch_time":0.053164578,"msgr_send_messages_queue_lat":{"avgcount":24726,"sum":1.785119543,"avgtime":0.000072196},"msgr_handle_ack_lat":{"avgcount":28201,"sum":0.011995298,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1426,"msgr_send_messages":1428,"msgr_recv_bytes":377901,"msgr_send_bytes":22714959,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.275530725,"msgr_running_send_time":0.160681242,"msgr_running_recv_time":0.037446280,"msgr_running_fast_dispatch_time":0.064626378,"msgr_send_messages_queue_lat":{"avgcount":1426,"sum":0.222591237,"avgtime":0.000156094},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":117,"sum":11.240900442,"avgtime":0.096076072}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5017,"sum":2.464322679,"avgtime":0.000491194}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":700.166907368,"avgtime":0.109230406},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":11,"dir_split":0,"dir_merge":0,"inodes":3720,"inodes_top":0,"inodes_bottom":3720,"inodes_pin_tail":0,"inodes_pinned":3720,"inodes_expired":0,"inodes_with_caps":170,"caps":170,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":118,"ceph_cap_op_grant":7307,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5338,"handle_client_caps_dirty":3000,"handle_client_cap_release":781,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":169,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3032,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15182,"evex":15030,"evtrm":15030,"ev":15184,"evexg":0,"evexd":15029,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":34254966,"wrpos":34351906,"rdpos":4195244,"jlat":{"avgcount":25,"sum":4.069661213,"avgtime":0.162786448},"replayed":3},"mds_mem":{"ino":182,"ino+":3214,"ino-":3032,"dir":13,"dir+":13,"dir-":0,"dn":180,"dn+":6412,"dn-":6232,"cap":170,"cap+":3202,"cap-":3032,"rss":117356,"heap":248164},"mds_server":{"dispatch_client_request":6416,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.521859867,"avgtime":0.521859867},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.022992058,"avgtime":0.002507185},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5746,"average_load":5746,"avg_session_uptime":981,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":28033682,"buffer_anon_items":10114,"buffer_meta_bytes":352,"buffer_meta_items":4,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13051282,"mds_co_items":202416,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3385,"op_send_bytes":31058312,"op_resend":0,"op_reply":3385,"op_latency":{"avgcount":3385,"sum":14.653695240,"avgtime":0.004329008},"op_inflight":0,"oplen_avg":{"avgcount":3385,"sum":3827},"op":3385,"op_r":58,"op_w":3327,"op_rmw":0,"op_pg":0,"osdop_stat":11,"osdop_create":181,"osdop_read":30,"osdop_write":47,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3057,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":362,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":45,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3032,"pq_executed":3032,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15965,"get_sum":4221710,"get_or_fail_fail":0,"get_or_fail_success":15965,"take":0,"take_sum":0,"put":15965,"put_sum":4221710,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":31365478,"put":3385,"put_sum":31365478,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":3385,"put":3385,"put_sum":3385,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3032,"get_sum":306232,"get_or_fail_fail":0,"get_or_fail_success":3032,"take":0,"take_sum":0,"put":22,"put_sum":306232,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":0,"max":3758096384,"get_started":0,"get":15182,"get_sum":30156404,"get_or_fail_fail":0,"get_or_fail_success":15182,"take":0,"take_sum":0,"put":25,"put_sum":30156404,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:40.616 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 536366, "msgr_recv_encrypted_bytes": 108560, "msgr_recv_messages": 502, "msgr_running_fast_dispatch_time": 0.015712982, "msgr_running_recv_time": 0.022229735, "msgr_running_send_time": 0.037648697, "msgr_running_total_time": 0.080439914, "msgr_send_bytes": 590361, "msgr_send_encrypted_bytes": 104400, "msgr_send_messages": 502, "msgr_send_messages_queue_lat": { "avgcount": 502, "avgtime": 0.000129185, "sum": 0.064851165 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28201, "avgtime": 4.25e-07, "sum": 0.011995298 }, "msgr_recv_bytes": 4747163, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 14041, "msgr_running_fast_dispatch_time": 0.053164578, "msgr_running_recv_time": 2.124582995, "msgr_running_send_time": 1.835712489, "msgr_running_total_time": 2.843944193, "msgr_send_bytes": 15156463, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 24727, "msgr_send_messages_queue_lat": { "avgcount": 24726, "avgtime": 7.2196e-05, "sum": 1.785119543 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 377901, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1426, "msgr_running_fast_dispatch_time": 0.064626378, "msgr_running_recv_time": 0.03744628, "msgr_running_send_time": 0.160681242, "msgr_running_total_time": 0.275530725, "msgr_send_bytes": 22714959, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1428, "msgr_send_messages_queue_lat": { "avgcount": 1426, "avgtime": 0.000156094, "sum": 0.222591237 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 117, "avgtime": 0.096076072, "sum": 11.240900442 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5017, "avgtime": 0.000491194, "sum": 2.464322679 }, "queue_len": 0 }, "mds": { "caps": 170, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7307, "ceph_cap_op_revoke": 118, "ceph_cap_op_trunc": 0, "dir_commit": 11, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 781, "handle_client_caps": 5338, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3720, "inodes_bottom": 3720, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3720, "inodes_top": 0, "inodes_with_caps": 170, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.109230406, "sum": 700.166907368 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 169, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3032, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15184, "evadd": 15182, "evex": 15030, "evexd": 15029, "evexg": 0, "evlrg": 0, "evtrm": 15030, "expos": 34254966, "jlat": { "avgcount": 25, "avgtime": 0.162786448, "sum": 4.069661213 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 34351906 }, "mds_mem": { "cap": 170, "cap+": 3202, "cap-": 3032, "dir": 13, "dir+": 13, "dir-": 0, "dn": 180, "dn+": 6412, "dn-": 6232, "heap": 248164, "ino": 182, "ino+": 3214, "ino-": 3032, "rss": 117356 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6416, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.521859867, "sum": 0.521859867 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002507185, "sum": 8.022992058 } }, "mds_sessions": { "average_load": 5746, "avg_session_uptime": 981, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5746 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 28033682, "buffer_anon_items": 10114, "buffer_meta_bytes": 352, "buffer_meta_items": 4, "mds_co_bytes": 13051282, "mds_co_items": 202416, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 45, "op": 3385, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3385, "avgtime": 0.004329008, "sum": 14.65369524 }, "op_pg": 0, "op_r": 58, "op_reply": 3385, "op_resend": 0, "op_rmw": 0, "op_send": 3385, "op_send_bytes": 31058312, "op_w": 3327, "oplen_avg": { "avgcount": 3385, "sum": 3827 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 181, "osdop_delete": 3057, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 362, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 11, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 47, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3032, "pq_executed_ops": 3032, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15965, "get_or_fail_fail": 0, "get_or_fail_success": 15965, "get_started": 0, "get_sum": 4221710, "max": 104857600, "put": 15965, "put_sum": 4221710, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3385, "put_sum": 31365478, "take": 3385, "take_sum": 31365478, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3385, "put_sum": 3385, "take": 3385, "take_sum": 3385, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3032, "get_or_fail_fail": 0, "get_or_fail_success": 3032, "get_started": 0, "get_sum": 306232, "max": 3758096384, "put": 22, "put_sum": 306232, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 15182, "get_or_fail_fail": 0, "get_or_fail_success": 15182, "get_started": 0, "get_sum": 30156404, "max": 3758096384, "put": 25, "put_sum": 30156404, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:40.616 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/169, 3032/3201 2024-03-06T07:33:41.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.826+0000 7f9eaae00640 1 Processor -- start 2024-03-06T07:33:41.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.827+0000 7f9eaae00640 1 -- start start 2024-03-06T07:33:41.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9eaae00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9ea415c250 0x7f9ea40b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9eaae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 0x7f9ea40cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9eaae00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9ea40cdaa0 0x7f9ea40d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9eaae00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea415c630 con 0x7f9ea415c250 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9eaae00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea40d0a50 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9eaae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40d0d30 con 0x7f9ea40b22b0 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9ea3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9ea415c250 0x7f9ea40b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9ea3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9ea415c250 0x7f9ea40b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51460/0 (socket says 172.21.15.114:51460) 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.829+0000 7f9ea3e00640 1 -- 172.21.15.114:0/1943283448 learned_addr learned my addr 172.21.15.114:0/1943283448 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.830+0000 7f9ea3e00640 1 -- 172.21.15.114:0/1943283448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 msgr2=0x7f9ea40cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.830+0000 7f9ea3e00640 1 --2- 172.21.15.114:0/1943283448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 0x7f9ea40cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.830+0000 7f9ea3e00640 1 -- 172.21.15.114:0/1943283448 >> v2:172.21.15.177:3300/0 conn(0x7f9ea40cdaa0 msgr2=0x7f9ea40d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.830+0000 7f9ea3e00640 1 --2- 172.21.15.114:0/1943283448 >> v2:172.21.15.177:3300/0 conn(0x7f9ea40cdaa0 0x7f9ea40d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.830+0000 7f9ea3e00640 1 -- 172.21.15.114:0/1943283448 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea40ccd60 con 0x7f9ea415c250 2024-03-06T07:33:41.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.831+0000 7f9ea3e00640 1 --2- 172.21.15.114:0/1943283448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea415c250 0x7f9ea40b1d70 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40c5730 tx=0x7f9e9402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6de2d9f8d959df7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.831+0000 7f9ea3400640 1 -- 172.21.15.114:0/1943283448 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e940317d0 con 0x7f9ea415c250 2024-03-06T07:33:41.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.831+0000 7f9ea3400640 1 -- 172.21.15.114:0/1943283448 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94037070 con 0x7f9ea415c250 2024-03-06T07:33:41.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.831+0000 7f9ea3400640 1 -- 172.21.15.114:0/1943283448 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94031b90 con 0x7f9ea415c250 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 -- 172.21.15.114:0/1943283448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea415c250 msgr2=0x7f9ea40b1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 --2- 172.21.15.114:0/1943283448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea415c250 0x7f9ea40b1d70 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40c5730 tx=0x7f9e9402b870 comp rx=0 tx=0).stop 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 -- 172.21.15.114:0/1943283448 shutdown_connections 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 --2- 172.21.15.114:0/1943283448 >> v2:172.21.15.177:3300/0 conn(0x7f9ea40cdaa0 0x7f9ea40d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 --2- 172.21.15.114:0/1943283448 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 0x7f9ea40cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 --2- 172.21.15.114:0/1943283448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea415c250 0x7f9ea40b1d70 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 -- 172.21.15.114:0/1943283448 >> 172.21.15.114:0/1943283448 conn(0x7f9ea409aa30 msgr2=0x7f9ea409ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 -- 172.21.15.114:0/1943283448 shutdown_connections 2024-03-06T07:33:41.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.832+0000 7f9eaae00640 1 -- 172.21.15.114:0/1943283448 wait complete. 2024-03-06T07:33:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.835+0000 7f9eaae00640 1 Processor -- start 2024-03-06T07:33:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.835+0000 7f9eaae00640 1 -- start start 2024-03-06T07:33:41.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.836+0000 7f9eaae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 0x7f9ea41558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.836+0000 7f9eaae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea40cdaa0 0x7f9ea415d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.836+0000 7f9eaae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ea415c250 0x7f9ea40b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.836+0000 7f9eaae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40d2310 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.836+0000 7f9eaae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40cca30 con 0x7f9ea415c250 2024-03-06T07:33:41.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.836+0000 7f9eaae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40d2490 con 0x7f9ea40b22b0 2024-03-06T07:33:41.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea40cdaa0 0x7f9ea415d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea40cdaa0 0x7f9ea415d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51462/0 (socket says 172.21.15.114:51462) 2024-03-06T07:33:41.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 -- 172.21.15.114:0/2762226355 learned_addr learned my addr 172.21.15.114:0/2762226355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9ea3e00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 0x7f9ea41558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9ea8e00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ea415c250 0x7f9ea40b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 -- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 msgr2=0x7f9ea41558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 0x7f9ea41558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 -- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ea415c250 msgr2=0x7f9ea40b4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ea415c250 0x7f9ea40b4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.837+0000 7f9e9b400640 1 -- 172.21.15.114:0/2762226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea415dd40 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.838+0000 7f9e9b400640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea40cdaa0 0x7f9ea415d800 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00b4d0 tx=0x7f9e8c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.838+0000 7f9ea0c00640 1 -- 172.21.15.114:0/2762226355 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c014070 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.838+0000 7f9ea0c00640 1 -- 172.21.15.114:0/2762226355 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c010870 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.838+0000 7f9ea0c00640 1 -- 172.21.15.114:0/2762226355 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c010c60 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.838+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea415dfe0 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.838+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea415e5a0 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.839+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea406fc00 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.840+0000 7f9ea0c00640 1 -- 172.21.15.114:0/2762226355 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e8c017070 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.841+0000 7f9ea0c00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e8006b4d0 0x7f9e8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.841+0000 7f9ea0c00640 1 -- 172.21.15.114:0/2762226355 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9e8c08a0e0 con 0x7f9ea40cdaa0 2024-03-06T07:33:41.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.842+0000 7f9ea3e00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e8006b4d0 0x7f9e8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.842+0000 7f9ea3e00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e8006b4d0 0x7f9e8006d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9e94036000 tx=0x7f9e9403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:41.846+0000 7f9ea0c00640 1 -- 172.21.15.114:0/2762226355 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ea406fc00 con 0x7f9ea40cdaa0 2024-03-06T07:33:42.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.107+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ea4062800 con 0x7f9ea40cdaa0 2024-03-06T07:33:42.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.108+0000 7f9ea0c00640 1 -- 172.21.15.114:0/2762226355 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f9e8c058ea0 con 0x7f9ea40cdaa0 2024-03-06T07:33:42.111 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:42.111 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:42.112 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e8006b4d0 msgr2=0x7f9e8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e8006b4d0 0x7f9e8006d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9e94036000 tx=0x7f9e9403d040 comp rx=0 tx=0).stop 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea40cdaa0 msgr2=0x7f9ea415d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea40cdaa0 0x7f9ea415d800 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00b4d0 tx=0x7f9e8c00b990 comp rx=0 tx=0).stop 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 shutdown_connections 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ea415c250 0x7f9ea40b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e8006b4d0 0x7f9e8006d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea40cdaa0 0x7f9ea415d800 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 --2- 172.21.15.114:0/2762226355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ea40b22b0 0x7f9ea41558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 >> 172.21.15.114:0/2762226355 conn(0x7f9ea409aa30 msgr2=0x7f9ea40cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 shutdown_connections 2024-03-06T07:33:42.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:42.111+0000 7f9eaae00640 1 -- 172.21.15.114:0/2762226355 wait complete. 2024-03-06T07:33:42.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:42.336 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":502,"msgr_send_messages":502,"msgr_recv_bytes":536366,"msgr_send_bytes":590361,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.080439914,"msgr_running_send_time":0.037648697,"msgr_running_recv_time":0.022229735,"msgr_running_fast_dispatch_time":0.015712982,"msgr_send_messages_queue_lat":{"avgcount":502,"sum":0.064851165,"avgtime":0.000129185},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108560,"msgr_send_encrypted_bytes":104400},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14044,"msgr_send_messages":24727,"msgr_recv_bytes":4752153,"msgr_send_bytes":15156463,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.844674529,"msgr_running_send_time":1.836089271,"msgr_running_recv_time":2.124844674,"msgr_running_fast_dispatch_time":0.053314874,"msgr_send_messages_queue_lat":{"avgcount":24726,"sum":1.785119543,"avgtime":0.000072196},"msgr_handle_ack_lat":{"avgcount":28204,"sum":0.011995824,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1426,"msgr_send_messages":1428,"msgr_recv_bytes":377901,"msgr_send_bytes":22714959,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.275530725,"msgr_running_send_time":0.160681242,"msgr_running_recv_time":0.037446280,"msgr_running_fast_dispatch_time":0.064626378,"msgr_send_messages_queue_lat":{"avgcount":1426,"sum":0.222591237,"avgtime":0.000156094},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":117,"sum":11.240900442,"avgtime":0.096076072}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5017,"sum":2.464322679,"avgtime":0.000491194}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":700.166907368,"avgtime":0.109230406},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":11,"dir_split":0,"dir_merge":0,"inodes":180,"inodes_top":0,"inodes_bottom":180,"inodes_pin_tail":0,"inodes_pinned":180,"inodes_expired":0,"inodes_with_caps":170,"caps":170,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":118,"ceph_cap_op_grant":7307,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5338,"handle_client_caps_dirty":3000,"handle_client_cap_release":782,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":169,"num_strays_delayed":169,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3032,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15182,"evex":15030,"evtrm":15030,"ev":15184,"evexg":0,"evexd":15029,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":34254966,"wrpos":34351906,"rdpos":4195244,"jlat":{"avgcount":25,"sum":4.069661213,"avgtime":0.162786448},"replayed":3},"mds_mem":{"ino":182,"ino+":3214,"ino-":3032,"dir":13,"dir+":13,"dir-":0,"dn":180,"dn+":6412,"dn-":6232,"cap":1,"cap+":3202,"cap-":3201,"rss":117356,"heap":248164},"mds_server":{"dispatch_client_request":6416,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.521859867,"avgtime":0.521859867},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.022992058,"avgtime":0.002507185},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5746,"average_load":5746,"avg_session_uptime":986,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248485,"buffer_anon_items":259,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":632001,"mds_co_items":12122,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3385,"op_send_bytes":31058312,"op_resend":0,"op_reply":3385,"op_latency":{"avgcount":3385,"sum":14.653695240,"avgtime":0.004329008},"op_inflight":0,"oplen_avg":{"avgcount":3385,"sum":3827},"op":3385,"op_r":58,"op_w":3327,"op_rmw":0,"op_pg":0,"osdop_stat":11,"osdop_create":181,"osdop_read":30,"osdop_write":47,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3057,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":362,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":45,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3032,"pq_executed":3032,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15968,"get_sum":4226430,"get_or_fail_fail":0,"get_or_fail_success":15968,"take":0,"take_sum":0,"put":15968,"put_sum":4226430,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":31365478,"put":3385,"put_sum":31365478,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":3385,"put":3385,"put_sum":3385,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3032,"get_sum":306232,"get_or_fail_fail":0,"get_or_fail_success":3032,"take":0,"take_sum":0,"put":22,"put_sum":306232,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":0,"max":3758096384,"get_started":0,"get":15182,"get_sum":30156404,"get_or_fail_fail":0,"get_or_fail_success":15182,"take":0,"take_sum":0,"put":25,"put_sum":30156404,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:42.338 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 536366, "msgr_recv_encrypted_bytes": 108560, "msgr_recv_messages": 502, "msgr_running_fast_dispatch_time": 0.015712982, "msgr_running_recv_time": 0.022229735, "msgr_running_send_time": 0.037648697, "msgr_running_total_time": 0.080439914, "msgr_send_bytes": 590361, "msgr_send_encrypted_bytes": 104400, "msgr_send_messages": 502, "msgr_send_messages_queue_lat": { "avgcount": 502, "avgtime": 0.000129185, "sum": 0.064851165 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28204, "avgtime": 4.25e-07, "sum": 0.011995824 }, "msgr_recv_bytes": 4752153, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 14044, "msgr_running_fast_dispatch_time": 0.053314874, "msgr_running_recv_time": 2.124844674, "msgr_running_send_time": 1.836089271, "msgr_running_total_time": 2.844674529, "msgr_send_bytes": 15156463, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 24727, "msgr_send_messages_queue_lat": { "avgcount": 24726, "avgtime": 7.2196e-05, "sum": 1.785119543 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 377901, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1426, "msgr_running_fast_dispatch_time": 0.064626378, "msgr_running_recv_time": 0.03744628, "msgr_running_send_time": 0.160681242, "msgr_running_total_time": 0.275530725, "msgr_send_bytes": 22714959, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1428, "msgr_send_messages_queue_lat": { "avgcount": 1426, "avgtime": 0.000156094, "sum": 0.222591237 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 117, "avgtime": 0.096076072, "sum": 11.240900442 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5017, "avgtime": 0.000491194, "sum": 2.464322679 }, "queue_len": 0 }, "mds": { "caps": 170, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7307, "ceph_cap_op_revoke": 118, "ceph_cap_op_trunc": 0, "dir_commit": 11, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 782, "handle_client_caps": 5338, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 180, "inodes_bottom": 180, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 180, "inodes_top": 0, "inodes_with_caps": 170, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.109230406, "sum": 700.166907368 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 169, "num_strays_delayed": 169, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3032, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15184, "evadd": 15182, "evex": 15030, "evexd": 15029, "evexg": 0, "evlrg": 0, "evtrm": 15030, "expos": 34254966, "jlat": { "avgcount": 25, "avgtime": 0.162786448, "sum": 4.069661213 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 34351906 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 180, "dn+": 6412, "dn-": 6232, "heap": 248164, "ino": 182, "ino+": 3214, "ino-": 3032, "rss": 117356 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6416, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.521859867, "sum": 0.521859867 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002507185, "sum": 8.022992058 } }, "mds_sessions": { "average_load": 5746, "avg_session_uptime": 986, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5746 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248485, "buffer_anon_items": 259, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 632001, "mds_co_items": 12122, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 45, "op": 3385, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3385, "avgtime": 0.004329008, "sum": 14.65369524 }, "op_pg": 0, "op_r": 58, "op_reply": 3385, "op_resend": 0, "op_rmw": 0, "op_send": 3385, "op_send_bytes": 31058312, "op_w": 3327, "oplen_avg": { "avgcount": 3385, "sum": 3827 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 181, "osdop_delete": 3057, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 362, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 11, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 47, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3032, "pq_executed_ops": 3032, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15968, "get_or_fail_fail": 0, "get_or_fail_success": 15968, "get_started": 0, "get_sum": 4226430, "max": 104857600, "put": 15968, "put_sum": 4226430, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3385, "put_sum": 31365478, "take": 3385, "take_sum": 31365478, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3385, "put_sum": 3385, "take": 3385, "take_sum": 3385, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3032, "get_or_fail_fail": 0, "get_or_fail_success": 3032, "get_started": 0, "get_sum": 306232, "max": 3758096384, "put": 22, "put_sum": 306232, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 15182, "get_or_fail_fail": 0, "get_or_fail_success": 15182, "get_started": 0, "get_sum": 30156404, "max": 3758096384, "put": 25, "put_sum": 30156404, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:42.338 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/169, 3032/3201 2024-03-06T07:33:43.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.527+0000 7fe3c6800640 1 Processor -- start 2024-03-06T07:33:43.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.527+0000 7fe3c6800640 1 -- start start 2024-03-06T07:33:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.528+0000 7fe3c6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3c015be90 0x7fe3c00b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.528+0000 7fe3c6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c015fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3c6800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c00b5160 con 0x7fe3c00acd60 2024-03-06T07:33:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3c6800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c00b4f30 con 0x7fe3c015be90 2024-03-06T07:33:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3c6800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00af240 con 0x7fe3c00d0780 2024-03-06T07:33:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3c015be90 0x7fe3c00b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3c015be90 0x7fe3c00b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51474/0 (socket says 172.21.15.114:51474) 2024-03-06T07:33:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bf400640 1 -- 172.21.15.114:0/1711476779 learned_addr learned my addr 172.21.15.114:0/1711476779 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe3c00acd60 0x7fe3c00d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33480/0 (socket says 172.21.15.114:33480) 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bfe00640 1 --2- 172.21.15.114:0/1711476779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c015fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bf400640 1 -- 172.21.15.114:0/1711476779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 msgr2=0x7fe3c015fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bf400640 1 --2- 172.21.15.114:0/1711476779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c015fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bf400640 1 -- 172.21.15.114:0/1711476779 >> v1:172.21.15.114:6789/0 conn(0x7fe3c00acd60 legacy=0x7fe3c00d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bf400640 1 -- 172.21.15.114:0/1711476779 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c00ae910 con 0x7fe3c015be90 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3be000640 1 -- 172.21.15.114:0/1711476779 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99525851 0 0) 0x7fe3c00b5160 con 0x7fe3c00acd60 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.529+0000 7fe3bfe00640 1 --2- 172.21.15.114:0/1711476779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c015fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.531+0000 7fe3bf400640 1 --2- 172.21.15.114:0/1711476779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00b49f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009870 tx=0x7fe3b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4fc392ec231e22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.531+0000 7fe3be000640 1 -- 172.21.15.114:0/1711476779 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0034070 con 0x7fe3c015be90 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.531+0000 7fe3be000640 1 -- 172.21.15.114:0/1711476779 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0030840 con 0x7fe3c015be90 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.531+0000 7fe3be000640 1 -- 172.21.15.114:0/1711476779 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0030c10 con 0x7fe3c015be90 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.531+0000 7fe3c6800640 1 -- 172.21.15.114:0/1711476779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 msgr2=0x7fe3c00b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.531+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1711476779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00b49f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009870 tx=0x7fe3b002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.532+0000 7fe3c6800640 1 -- 172.21.15.114:0/1711476779 shutdown_connections 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.532+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1711476779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c015fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.532+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1711476779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00b49f0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.532+0000 7fe3c6800640 1 -- 172.21.15.114:0/1711476779 >> 172.21.15.114:0/1711476779 conn(0x7fe3c006b360 msgr2=0x7fe3c006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.532+0000 7fe3c6800640 1 -- 172.21.15.114:0/1711476779 shutdown_connections 2024-03-06T07:33:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.532+0000 7fe3c6800640 1 -- 172.21.15.114:0/1711476779 wait complete. 2024-03-06T07:33:43.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.533+0000 7fe3c6800640 1 Processor -- start 2024-03-06T07:33:43.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.533+0000 7fe3c6800640 1 -- start start 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.534+0000 7fe3c6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3c00acd60 0x7fe3c0152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.534+0000 7fe3c6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.534+0000 7fe3c6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.534+0000 7fe3c6800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00b2c80 con 0x7fe3c015be90 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.534+0000 7fe3c6800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00aeed0 con 0x7fe3c00acd60 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.534+0000 7fe3c6800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c00b2e00 con 0x7fe3c00d0780 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51482/0 (socket says 172.21.15.114:51482) 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 -- 172.21.15.114:0/1008371255 learned_addr learned my addr 172.21.15.114:0/1008371255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bea00640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bf400640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3c00acd60 0x7fe3c0152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 -- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 msgr2=0x7fe3c00b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c00b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 -- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3c00acd60 msgr2=0x7fe3c0152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3c00acd60 0x7fe3c0152a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 -- 172.21.15.114:0/1008371255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c00cdc60 con 0x7fe3c015be90 2024-03-06T07:33:43.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.535+0000 7fe3bfe00640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00c9aa0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800ce00 tx=0x7fe3b800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.536+0000 7fe3b7400640 1 -- 172.21.15.114:0/1008371255 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b801a070 con 0x7fe3c015be90 2024-03-06T07:33:43.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.536+0000 7fe3b7400640 1 -- 172.21.15.114:0/1008371255 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b800d840 con 0x7fe3c015be90 2024-03-06T07:33:43.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.536+0000 7fe3b7400640 1 -- 172.21.15.114:0/1008371255 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800dc30 con 0x7fe3c015be90 2024-03-06T07:33:43.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.536+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c00cdf40 con 0x7fe3c015be90 2024-03-06T07:33:43.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.536+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c00ce500 con 0x7fe3c015be90 2024-03-06T07:33:43.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.537+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c00add90 con 0x7fe3c015be90 2024-03-06T07:33:43.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.538+0000 7fe3b7400640 1 -- 172.21.15.114:0/1008371255 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3b801d070 con 0x7fe3c015be90 2024-03-06T07:33:43.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.539+0000 7fe3b7400640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3a006b4d0 0x7fe3a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.539+0000 7fe3b7400640 1 -- 172.21.15.114:0/1008371255 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe3b808ab00 con 0x7fe3c015be90 2024-03-06T07:33:43.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.539+0000 7fe3bf400640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3a006b4d0 0x7fe3a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.540+0000 7fe3bf400640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3a006b4d0 0x7fe3a006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0036000 tx=0x7fe3b003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.548+0000 7fe3b7400640 1 -- 172.21.15.114:0/1008371255 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3c00add90 con 0x7fe3c015be90 2024-03-06T07:33:43.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.779+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3c0068070 con 0x7fe3c015be90 2024-03-06T07:33:43.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.780+0000 7fe3b7400640 1 -- 172.21.15.114:0/1008371255 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7fe3b808ae20 con 0x7fe3c015be90 2024-03-06T07:33:43.783 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:43.783 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:43.783 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:43.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.782+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3a006b4d0 msgr2=0x7fe3a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.782+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3a006b4d0 0x7fe3a006d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0036000 tx=0x7fe3b003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.782+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 msgr2=0x7fe3c00c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.782+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00c9aa0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800ce00 tx=0x7fe3b800a710 comp rx=0 tx=0).stop 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 shutdown_connections 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3a006b4d0 0x7fe3a006d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c015be90 0x7fe3c00c9aa0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3c00d0780 0x7fe3c00b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 --2- 172.21.15.114:0/1008371255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3c00acd60 0x7fe3c0152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 >> 172.21.15.114:0/1008371255 conn(0x7fe3c006b360 msgr2=0x7fe3c00725b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 shutdown_connections 2024-03-06T07:33:43.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:43.783+0000 7fe3c6800640 1 -- 172.21.15.114:0/1008371255 wait complete. 2024-03-06T07:33:43.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:44.016 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":523,"msgr_send_messages":524,"msgr_recv_bytes":558721,"msgr_send_bytes":616020,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.083899510,"msgr_running_send_time":0.039414791,"msgr_running_recv_time":0.022956507,"msgr_running_fast_dispatch_time":0.016426526,"msgr_send_messages_queue_lat":{"avgcount":524,"sum":0.066660861,"avgtime":0.000127215},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108560,"msgr_send_encrypted_bytes":106192},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14114,"msgr_send_messages":24795,"msgr_recv_bytes":4770253,"msgr_send_bytes":15178713,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.852980032,"msgr_running_send_time":1.839418702,"msgr_running_recv_time":2.126947334,"msgr_running_fast_dispatch_time":0.055602246,"msgr_send_messages_queue_lat":{"avgcount":24794,"sum":1.788907979,"avgtime":0.000072150},"msgr_handle_ack_lat":{"avgcount":28206,"sum":0.011996191,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1514,"msgr_send_messages":1516,"msgr_recv_bytes":400333,"msgr_send_bytes":22743895,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.286292212,"msgr_running_send_time":0.164477312,"msgr_running_recv_time":0.040471138,"msgr_running_fast_dispatch_time":0.067849158,"msgr_send_messages_queue_lat":{"avgcount":1514,"sum":0.227760922,"avgtime":0.000150436},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":118,"sum":11.240922582,"avgtime":0.095262055}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5330,"sum":2.494602481,"avgtime":0.000468030}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":700.166907368,"avgtime":0.109230406},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":11,"dir_split":0,"dir_merge":0,"inodes":180,"inodes_top":0,"inodes_bottom":180,"inodes_pin_tail":0,"inodes_pinned":180,"inodes_expired":0,"inodes_with_caps":170,"caps":170,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":118,"ceph_cap_op_grant":7307,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5338,"handle_client_caps_dirty":3000,"handle_client_cap_release":782,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15351,"evex":15030,"evtrm":15030,"ev":324,"evexg":0,"evexd":15029,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":34254966,"wrpos":34453986,"rdpos":4195244,"jlat":{"avgcount":25,"sum":4.069661213,"avgtime":0.162786448},"replayed":3},"mds_mem":{"ino":182,"ino+":3214,"ino-":3032,"dir":13,"dir+":13,"dir-":0,"dn":180,"dn+":6412,"dn-":6232,"cap":1,"cap+":3202,"cap-":3201,"rss":117356,"heap":248164},"mds_server":{"dispatch_client_request":6416,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.521859867,"avgtime":0.521859867},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.022992058,"avgtime":0.002507185},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5746,"average_load":5746,"avg_session_uptime":986,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248485,"buffer_anon_items":259,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":632001,"mds_co_items":12122,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3562,"op_send_bytes":31075647,"op_resend":0,"op_reply":3562,"op_latency":{"avgcount":3562,"sum":15.225683345,"avgtime":0.004274475},"op_inflight":0,"oplen_avg":{"avgcount":3562,"sum":4006},"op":3562,"op_r":61,"op_w":3501,"op_rmw":0,"op_pg":0,"osdop_stat":11,"osdop_create":181,"osdop_read":33,"osdop_write":50,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":362,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":47,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":16147,"get_sum":4273207,"get_or_fail_fail":0,"get_or_fail_success":16147,"take":0,"take_sum":0,"put":16147,"put_sum":4273207,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3562,"take_sum":31399882,"put":3562,"put_sum":31399882,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3562,"take_sum":3562,"put":3562,"put_sum":3562,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":25,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":102080,"max":3758096384,"get_started":0,"get":15351,"get_sum":30258484,"get_or_fail_fail":0,"get_or_fail_success":15351,"take":0,"take_sum":0,"put":25,"put_sum":30156404,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:44.017 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 558721, "msgr_recv_encrypted_bytes": 108560, "msgr_recv_messages": 523, "msgr_running_fast_dispatch_time": 0.016426526, "msgr_running_recv_time": 0.022956507, "msgr_running_send_time": 0.039414791, "msgr_running_total_time": 0.08389951, "msgr_send_bytes": 616020, "msgr_send_encrypted_bytes": 106192, "msgr_send_messages": 524, "msgr_send_messages_queue_lat": { "avgcount": 524, "avgtime": 0.000127215, "sum": 0.066660861 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28206, "avgtime": 4.25e-07, "sum": 0.011996191 }, "msgr_recv_bytes": 4770253, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 14114, "msgr_running_fast_dispatch_time": 0.055602246, "msgr_running_recv_time": 2.126947334, "msgr_running_send_time": 1.839418702, "msgr_running_total_time": 2.852980032, "msgr_send_bytes": 15178713, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 24795, "msgr_send_messages_queue_lat": { "avgcount": 24794, "avgtime": 7.215e-05, "sum": 1.788907979 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 400333, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1514, "msgr_running_fast_dispatch_time": 0.067849158, "msgr_running_recv_time": 0.040471138, "msgr_running_send_time": 0.164477312, "msgr_running_total_time": 0.286292212, "msgr_send_bytes": 22743895, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1516, "msgr_send_messages_queue_lat": { "avgcount": 1514, "avgtime": 0.000150436, "sum": 0.227760922 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 118, "avgtime": 0.095262055, "sum": 11.240922582 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5330, "avgtime": 0.00046803, "sum": 2.494602481 }, "queue_len": 0 }, "mds": { "caps": 170, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7307, "ceph_cap_op_revoke": 118, "ceph_cap_op_trunc": 0, "dir_commit": 11, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 782, "handle_client_caps": 5338, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 180, "inodes_bottom": 180, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 180, "inodes_top": 0, "inodes_with_caps": 170, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.109230406, "sum": 700.166907368 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 324, "evadd": 15351, "evex": 15030, "evexd": 15029, "evexg": 0, "evlrg": 0, "evtrm": 15030, "expos": 34254966, "jlat": { "avgcount": 25, "avgtime": 0.162786448, "sum": 4.069661213 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 34453986 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 180, "dn+": 6412, "dn-": 6232, "heap": 248164, "ino": 182, "ino+": 3214, "ino-": 3032, "rss": 117356 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6416, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.521859867, "sum": 0.521859867 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002507185, "sum": 8.022992058 } }, "mds_sessions": { "average_load": 5746, "avg_session_uptime": 986, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5746 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248485, "buffer_anon_items": 259, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 632001, "mds_co_items": 12122, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 47, "op": 3562, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3562, "avgtime": 0.004274475, "sum": 15.225683345 }, "op_pg": 0, "op_r": 61, "op_reply": 3562, "op_resend": 0, "op_rmw": 0, "op_send": 3562, "op_send_bytes": 31075647, "op_w": 3501, "oplen_avg": { "avgcount": 3562, "sum": 4006 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 181, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 362, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 11, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 50, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 16147, "get_or_fail_fail": 0, "get_or_fail_success": 16147, "get_started": 0, "get_sum": 4273207, "max": 104857600, "put": 16147, "put_sum": 4273207, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3562, "put_sum": 31399882, "take": 3562, "take_sum": 31399882, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3562, "put_sum": 3562, "take": 3562, "take_sum": 3562, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 25, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 15351, "get_or_fail_fail": 0, "get_or_fail_success": 15351, "get_started": 0, "get_sum": 30258484, "max": 3758096384, "put": 25, "put_sum": 30156404, "take": 0, "take_sum": 0, "val": 102080, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:44.017 INFO:tasks.cephfs.test_strays:Complete purge in 9 seconds 2024-03-06T07:33:44.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:44.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.217+0000 7f10e1a00640 1 Processor -- start 2024-03-06T07:33:44.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.217+0000 7f10e1a00640 1 -- start start 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.218+0000 7f10e1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc0d0780 0x7f10dc15e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.218+0000 7f10e1a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10dc06d600 0x7f10dc15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.218+0000 7f10e1a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc15f370 con 0x7f10dc0b3340 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.218+0000 7f10e1a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc15f650 con 0x7f10dc06d600 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.218+0000 7f10e1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06cb20 con 0x7f10dc0d0780 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10db400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10dc06d600 0x7f10dc15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10db400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f10dc06d600 0x7f10dc15ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51486/0 (socket says 172.21.15.114:51486) 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10db400640 1 -- 172.21.15.114:0/1236122118 learned_addr learned my addr 172.21.15.114:0/1236122118 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10daa00640 1 --1- 172.21.15.114:0/1236122118 >> v1:172.21.15.114:6789/0 conn(0x7f10dc0b3340 0x7f10dc0d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33486/0 (socket says 172.21.15.114:33486) 2024-03-06T07:33:44.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10d3e00640 1 --2- 172.21.15.114:0/1236122118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc0d0780 0x7f10dc15e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10db400640 1 -- 172.21.15.114:0/1236122118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc0d0780 msgr2=0x7f10dc15e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10db400640 1 --2- 172.21.15.114:0/1236122118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc0d0780 0x7f10dc15e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10da000640 1 -- 172.21.15.114:0/1236122118 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545937072 0 0) 0x7f10dc15f370 con 0x7f10dc0b3340 2024-03-06T07:33:44.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10db400640 1 -- 172.21.15.114:0/1236122118 >> v1:172.21.15.114:6789/0 conn(0x7f10dc0b3340 legacy=0x7f10dc0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10db400640 1 -- 172.21.15.114:0/1236122118 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc06ce00 con 0x7f10dc06d600 2024-03-06T07:33:44.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.219+0000 7f10d3e00640 1 --2- 172.21.15.114:0/1236122118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc0d0780 0x7f10dc15e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:44.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.221+0000 7f10db400640 1 --2- 172.21.15.114:0/1236122118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc06d600 0x7f10dc15ec60 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f10dc15f7d0 tx=0x7f10d402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8cc971f5c8e719 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.221+0000 7f10da000640 1 -- 172.21.15.114:0/1236122118 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d402c9c0 con 0x7f10dc06d600 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.221+0000 7f10da000640 1 -- 172.21.15.114:0/1236122118 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d402cb60 con 0x7f10dc06d600 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.221+0000 7f10da000640 1 -- 172.21.15.114:0/1236122118 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d40343f0 con 0x7f10dc06d600 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 -- 172.21.15.114:0/1236122118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc06d600 msgr2=0x7f10dc15ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 --2- 172.21.15.114:0/1236122118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc06d600 0x7f10dc15ec60 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f10dc15f7d0 tx=0x7f10d402be20 comp rx=0 tx=0).stop 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 -- 172.21.15.114:0/1236122118 shutdown_connections 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 --2- 172.21.15.114:0/1236122118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc06d600 0x7f10dc15ec60 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 --2- 172.21.15.114:0/1236122118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc0d0780 0x7f10dc15e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 -- 172.21.15.114:0/1236122118 >> 172.21.15.114:0/1236122118 conn(0x7f10dc06b360 msgr2=0x7f10dc06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 -- 172.21.15.114:0/1236122118 shutdown_connections 2024-03-06T07:33:44.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.222+0000 7f10e1a00640 1 -- 172.21.15.114:0/1236122118 wait complete. 2024-03-06T07:33:44.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.224+0000 7f10e1a00640 1 Processor -- start 2024-03-06T07:33:44.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.224+0000 7f10e1a00640 1 -- start start 2024-03-06T07:33:44.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10e1a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc06d600 0x7f10dc1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10e1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10dc0b3340 0x7f10dc161a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10e1a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc0d0780 0x7f10dc09bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10e1a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0acd60 con 0x7f10dc0d0780 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10e1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0b3720 con 0x7f10dc0b3340 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10e1a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0acee0 con 0x7f10dc06d600 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc0d0780 0x7f10dc09bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc0d0780 0x7f10dc09bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51496/0 (socket says 172.21.15.114:51496) 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10db400640 1 -- 172.21.15.114:0/4101086137 learned_addr learned my addr 172.21.15.114:0/4101086137 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.225+0000 7f10daa00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc06d600 0x7f10dc1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10d3e00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10dc0b3340 0x7f10dc161a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10db400640 1 -- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc06d600 msgr2=0x7f10dc1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10db400640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc06d600 0x7f10dc1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10db400640 1 -- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10dc0b3340 msgr2=0x7f10dc161a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10db400640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10dc0b3340 0x7f10dc161a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10db400640 1 -- 172.21.15.114:0/4101086137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc1609d0 con 0x7f10dc0d0780 2024-03-06T07:33:44.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10db400640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc0d0780 0x7f10dc09bff0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009f40 tx=0x7f10d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10d2a00640 1 -- 172.21.15.114:0/4101086137 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d4043030 con 0x7f10dc0d0780 2024-03-06T07:33:44.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10d2a00640 1 -- 172.21.15.114:0/4101086137 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d403e450 con 0x7f10dc0d0780 2024-03-06T07:33:44.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc160cb0 con 0x7f10dc0d0780 2024-03-06T07:33:44.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10d2a00640 1 -- 172.21.15.114:0/4101086137 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d403e660 con 0x7f10dc0d0780 2024-03-06T07:33:44.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.226+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc161270 con 0x7f10dc0d0780 2024-03-06T07:33:44.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.228+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc068070 con 0x7f10dc0d0780 2024-03-06T07:33:44.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.228+0000 7f10d2a00640 1 -- 172.21.15.114:0/4101086137 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10d4046070 con 0x7f10dc0d0780 2024-03-06T07:33:44.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.229+0000 7f10d2a00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.229+0000 7f10d2a00640 1 -- 172.21.15.114:0/4101086137 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f10d40ac8b0 con 0x7f10dc0d0780 2024-03-06T07:33:44.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.229+0000 7f10daa00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.230+0000 7f10daa00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f10b806b4d0 0x7f10b806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f10cc000f60 tx=0x7f10cc007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.234+0000 7f10d2a00640 1 -- 172.21.15.114:0/4101086137 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10d404d070 con 0x7f10dc0d0780 2024-03-06T07:33:44.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.469+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10dc06e6a0 con 0x7f10dc0d0780 2024-03-06T07:33:44.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.470+0000 7f10d2a00640 1 -- 172.21.15.114:0/4101086137 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f10d407b6f0 con 0x7f10dc0d0780 2024-03-06T07:33:44.473 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:44.473 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:44.473 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:44.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.472+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f10b806b4d0 msgr2=0x7f10b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.472+0000 7f10e1a00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f10b806b4d0 0x7f10b806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f10cc000f60 tx=0x7f10cc007b50 comp rx=0 tx=0).stop 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.472+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc0d0780 msgr2=0x7f10dc09bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.472+0000 7f10e1a00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc0d0780 0x7f10dc09bff0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009f40 tx=0x7f10d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 shutdown_connections 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10dc0d0780 0x7f10dc09bff0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10dc0b3340 0x7f10dc161a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 --2- 172.21.15.114:0/4101086137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f10dc06d600 0x7f10dc1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 >> 172.21.15.114:0/4101086137 conn(0x7f10dc06b360 msgr2=0x7f10dc0b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 shutdown_connections 2024-03-06T07:33:44.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:44.473+0000 7f10e1a00640 1 -- 172.21.15.114:0/4101086137 wait complete. 2024-03-06T07:33:44.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:33:44.702 INFO:teuthology.orchestra.run.smithi114.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":524,"msgr_send_messages":525,"msgr_recv_bytes":558993,"msgr_send_bytes":616644,"msgr_created_connections":8,"msgr_active_connections":4,"msgr_running_total_time":0.084077623,"msgr_running_send_time":0.039512485,"msgr_running_recv_time":0.022991214,"msgr_running_fast_dispatch_time":0.016458645,"msgr_send_messages_queue_lat":{"avgcount":525,"sum":0.066700788,"avgtime":0.000127049},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000171,"avgtime":0.000000085},"msgr_recv_encrypted_bytes":108832,"msgr_send_encrypted_bytes":106816},"AsyncMessenger::Worker-1":{"msgr_recv_messages":14114,"msgr_send_messages":24795,"msgr_recv_bytes":4770253,"msgr_send_bytes":15178713,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.852980032,"msgr_running_send_time":1.839418702,"msgr_running_recv_time":2.126947334,"msgr_running_fast_dispatch_time":0.055602246,"msgr_send_messages_queue_lat":{"avgcount":24794,"sum":1.788907979,"avgtime":0.000072150},"msgr_handle_ack_lat":{"avgcount":28206,"sum":0.011996191,"avgtime":0.000000425},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":78256},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1514,"msgr_send_messages":1516,"msgr_recv_bytes":400333,"msgr_send_bytes":22743895,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.286292212,"msgr_running_send_time":0.164477312,"msgr_running_recv_time":0.040471138,"msgr_running_fast_dispatch_time":0.067849158,"msgr_send_messages_queue_lat":{"avgcount":1514,"sum":0.227760922,"avgtime":0.000150436},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":118,"sum":11.240922582,"avgtime":0.095262055}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5330,"sum":2.494602481,"avgtime":0.000468030}},"mds":{"request":6412,"reply":6410,"reply_latency":{"avgcount":6410,"sum":700.166907368,"avgtime":0.109230406},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":11,"dir_split":0,"dir_merge":0,"inodes":180,"inodes_top":0,"inodes_bottom":180,"inodes_pin_tail":0,"inodes_pinned":180,"inodes_expired":0,"inodes_with_caps":170,"caps":170,"subtrees":2,"traverse":6413,"traverse_hit":6413,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":118,"ceph_cap_op_grant":7307,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5338,"handle_client_caps_dirty":3000,"handle_client_cap_release":782,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15351,"evex":15030,"evtrm":15030,"ev":324,"evexg":0,"evexd":15029,"segadd":19,"segex":20,"segtrm":20,"seg":1,"segmjr":1,"segexg":0,"segexd":19,"expos":34254966,"wrpos":34453986,"rdpos":4195244,"jlat":{"avgcount":25,"sum":4.069661213,"avgtime":0.162786448},"replayed":3},"mds_mem":{"ino":182,"ino+":3214,"ino-":3032,"dir":12,"dir+":13,"dir-":1,"dn":180,"dn+":6412,"dn-":6232,"cap":1,"cap+":3202,"cap-":3201,"rss":117356,"heap":248164},"mds_server":{"dispatch_client_request":6416,"dispatch_server_request":0,"handle_client_request":6412,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":690.044979716,"avgtime":0.215639056},"req_getattr_latency":{"avgcount":2,"sum":0.609168544,"avgtime":0.304584272},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.004809165,"avgtime":0.004809165},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.963098018,"avgtime":0.160516336},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.521859867,"avgtime":0.521859867},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":8.022992058,"avgtime":0.002507185},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5746,"average_load":5746,"avg_session_uptime":986,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248485,"buffer_anon_items":259,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":632001,"mds_co_items":12122,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3562,"op_send_bytes":31075647,"op_resend":0,"op_reply":3562,"op_latency":{"avgcount":3562,"sum":15.225683345,"avgtime":0.004274475},"op_inflight":0,"oplen_avg":{"avgcount":3562,"sum":4006},"op":3562,"op_r":61,"op_w":3501,"op_rmw":0,"op_pg":0,"osdop_stat":11,"osdop_create":181,"osdop_read":33,"osdop_write":50,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":362,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":47,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":16148,"get_sum":4273337,"get_or_fail_fail":0,"get_or_fail_success":16148,"take":0,"take_sum":0,"put":16148,"put_sum":4273337,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3562,"take_sum":31399882,"put":3562,"put_sum":31399882,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3562,"take_sum":3562,"put":3562,"put_sum":3562,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":25,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561c9037aaa0":{"val":102080,"max":3758096384,"get_started":0,"get":15351,"get_sum":30258484,"get_or_fail_fail":0,"get_or_fail_success":15351,"take":0,"take_sum":0,"put":25,"put_sum":30156404,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:44.704 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 8.5e-08, "sum": 1.71e-07 }, "msgr_recv_bytes": 558993, "msgr_recv_encrypted_bytes": 108832, "msgr_recv_messages": 524, "msgr_running_fast_dispatch_time": 0.016458645, "msgr_running_recv_time": 0.022991214, "msgr_running_send_time": 0.039512485, "msgr_running_total_time": 0.084077623, "msgr_send_bytes": 616644, "msgr_send_encrypted_bytes": 106816, "msgr_send_messages": 525, "msgr_send_messages_queue_lat": { "avgcount": 525, "avgtime": 0.000127049, "sum": 0.066700788 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 28206, "avgtime": 4.25e-07, "sum": 0.011996191 }, "msgr_recv_bytes": 4770253, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 14114, "msgr_running_fast_dispatch_time": 0.055602246, "msgr_running_recv_time": 2.126947334, "msgr_running_send_time": 1.839418702, "msgr_running_total_time": 2.852980032, "msgr_send_bytes": 15178713, "msgr_send_encrypted_bytes": 78256, "msgr_send_messages": 24795, "msgr_send_messages_queue_lat": { "avgcount": 24794, "avgtime": 7.215e-05, "sum": 1.788907979 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 400333, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1514, "msgr_running_fast_dispatch_time": 0.067849158, "msgr_running_recv_time": 0.040471138, "msgr_running_send_time": 0.164477312, "msgr_running_total_time": 0.286292212, "msgr_send_bytes": 22743895, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1516, "msgr_send_messages_queue_lat": { "avgcount": 1514, "avgtime": 0.000150436, "sum": 0.227760922 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 118, "avgtime": 0.095262055, "sum": 11.240922582 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5330, "avgtime": 0.00046803, "sum": 2.494602481 }, "queue_len": 0 }, "mds": { "caps": 170, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7307, "ceph_cap_op_revoke": 118, "ceph_cap_op_trunc": 0, "dir_commit": 11, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 782, "handle_client_caps": 5338, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 180, "inodes_bottom": 180, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 180, "inodes_top": 0, "inodes_with_caps": 170, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.109230406, "sum": 700.166907368 }, "request": 6412, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6413, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6413, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 324, "evadd": 15351, "evex": 15030, "evexd": 15029, "evexg": 0, "evlrg": 0, "evtrm": 15030, "expos": 34254966, "jlat": { "avgcount": 25, "avgtime": 0.162786448, "sum": 4.069661213 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 19, "segex": 20, "segexd": 19, "segexg": 0, "segmjr": 1, "segtrm": 20, "wrpos": 34453986 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 180, "dn+": 6412, "dn-": 6232, "heap": 248164, "ino": 182, "ino+": 3214, "ino-": 3032, "rss": 117356 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6416, "dispatch_server_request": 0, "handle_client_request": 6412, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.215639056, "sum": 690.044979716 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.304584272, "sum": 0.609168544 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.004809165, "sum": 0.004809165 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.160516336, "sum": 0.963098018 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.521859867, "sum": 0.521859867 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002507185, "sum": 8.022992058 } }, "mds_sessions": { "average_load": 5746, "avg_session_uptime": 986, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5746 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248485, "buffer_anon_items": 259, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 632001, "mds_co_items": 12122, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 47, "op": 3562, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3562, "avgtime": 0.004274475, "sum": 15.225683345 }, "op_pg": 0, "op_r": 61, "op_reply": 3562, "op_resend": 0, "op_rmw": 0, "op_send": 3562, "op_send_bytes": 31075647, "op_w": 3501, "oplen_avg": { "avgcount": 3562, "sum": 4006 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 181, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 362, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 11, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 50, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 16148, "get_or_fail_fail": 0, "get_or_fail_success": 16148, "get_started": 0, "get_sum": 4273337, "max": 104857600, "put": 16148, "put_sum": 4273337, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3562, "put_sum": 31399882, "take": 3562, "take_sum": 31399882, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3562, "put_sum": 3562, "take": 3562, "take_sum": 3562, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 25, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561c9037aaa0": { "get": 15351, "get_or_fail_fail": 0, "get_or_fail_success": 15351, "get_started": 0, "get_sum": 30258484, "max": 3758096384, "put": 25, "put_sum": 30156404, "take": 0, "take_sum": 0, "val": 102080, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:44.704 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:44.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:44.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:44.931 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:44.931 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:33:44.931 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:44.984 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:33:44.984 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:45.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:45.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.280+0000 7fa9aa000640 1 Processor -- start 2024-03-06T07:33:45.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.280+0000 7fa9aa000640 1 -- start start 2024-03-06T07:33:45.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9aa000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40d2cc0 0x7fa9a40d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9aa000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9a40b5020 0x7fa9a40d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9aa000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40d21c0 con 0x7fa9a40b44f0 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9aa000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a40d0000 con 0x7fa9a40b5020 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9aa000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40b4e40 con 0x7fa9a40d2cc0 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9a3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9a40b5020 0x7fa9a40d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9a3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9a40b5020 0x7fa9a40d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51498/0 (socket says 172.21.15.114:51498) 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9a3e00640 1 -- 172.21.15.114:0/503888090 learned_addr learned my addr 172.21.15.114:0/503888090 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.282+0000 7fa9a3400640 1 --1- 172.21.15.114:0/503888090 >> v1:172.21.15.114:6789/0 conn(0x7fa9a40b44f0 0x7fa9a40d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33494/0 (socket says 172.21.15.114:33494) 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.283+0000 7fa99aa00640 1 --2- 172.21.15.114:0/503888090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40d2cc0 0x7fa9a40d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.283+0000 7fa9a3e00640 1 -- 172.21.15.114:0/503888090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40d2cc0 msgr2=0x7fa9a40d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.283+0000 7fa9a3e00640 1 --2- 172.21.15.114:0/503888090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40d2cc0 0x7fa9a40d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.283+0000 7fa9a3e00640 1 -- 172.21.15.114:0/503888090 >> v1:172.21.15.114:6789/0 conn(0x7fa9a40b44f0 legacy=0x7fa9a40d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.283+0000 7fa9a3e00640 1 -- 172.21.15.114:0/503888090 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a40b48d0 con 0x7fa9a40b5020 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.283+0000 7fa99aa00640 1 --2- 172.21.15.114:0/503888090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40d2cc0 0x7fa9a40d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:45.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.283+0000 7fa9a2a00640 1 -- 172.21.15.114:0/503888090 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031150779 0 0) 0x7fa9a40d21c0 con 0x7fa9a40b44f0 2024-03-06T07:33:45.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.284+0000 7fa9a3e00640 1 --2- 172.21.15.114:0/503888090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a40d0bd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009e80 tx=0x7fa99c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c823b6bc20d86420 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.284+0000 7fa9a2a00640 1 -- 172.21.15.114:0/503888090 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c02ca50 con 0x7fa9a40b5020 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.284+0000 7fa9a2a00640 1 -- 172.21.15.114:0/503888090 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c02cbf0 con 0x7fa9a40b5020 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.284+0000 7fa9a2a00640 1 -- 172.21.15.114:0/503888090 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0343f0 con 0x7fa9a40b5020 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 -- 172.21.15.114:0/503888090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 msgr2=0x7fa9a40d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 --2- 172.21.15.114:0/503888090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a40d0bd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009e80 tx=0x7fa99c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 -- 172.21.15.114:0/503888090 shutdown_connections 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 --2- 172.21.15.114:0/503888090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a40d0bd0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 --2- 172.21.15.114:0/503888090 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40d2cc0 0x7fa9a40d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 -- 172.21.15.114:0/503888090 >> 172.21.15.114:0/503888090 conn(0x7fa9a40d3fd0 msgr2=0x7fa9a40d43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 -- 172.21.15.114:0/503888090 shutdown_connections 2024-03-06T07:33:45.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.285+0000 7fa9aa000640 1 -- 172.21.15.114:0/503888090 wait complete. 2024-03-06T07:33:45.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.287+0000 7fa9aa000640 1 Processor -- start 2024-03-06T07:33:45.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.287+0000 7fa9aa000640 1 -- start start 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa9aa000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40b44f0 0x7fa9a4156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa9aa000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a41f7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa9aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a40d2cc0 0x7fa9a41f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa9aa000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40cfae0 con 0x7fa9a40b5020 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa9aa000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40cf800 con 0x7fa9a40d2cc0 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa9aa000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40b4ad0 con 0x7fa9a40b44f0 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a41f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a41f7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51504/0 (socket says 172.21.15.114:51504) 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 -- 172.21.15.114:0/1753771069 learned_addr learned my addr 172.21.15.114:0/1753771069 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 -- 172.21.15.114:0/1753771069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40b44f0 msgr2=0x7fa9a4156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40b44f0 0x7fa9a4156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 -- 172.21.15.114:0/1753771069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a40d2cc0 msgr2=0x7fa9a41f7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a40d2cc0 0x7fa9a41f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.288+0000 7fa99aa00640 1 -- 172.21.15.114:0/1753771069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a40b6c50 con 0x7fa9a40b5020 2024-03-06T07:33:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.289+0000 7fa99aa00640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a41f7230 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00c7f0 tx=0x7fa98c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.289+0000 7fa99be00640 1 -- 172.21.15.114:0/1753771069 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c015070 con 0x7fa9a40b5020 2024-03-06T07:33:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.289+0000 7fa99be00640 1 -- 172.21.15.114:0/1753771069 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c007820 con 0x7fa9a40b5020 2024-03-06T07:33:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.289+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a409fa80 con 0x7fa9a40b5020 2024-03-06T07:33:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.289+0000 7fa99be00640 1 -- 172.21.15.114:0/1753771069 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c007bd0 con 0x7fa9a40b5020 2024-03-06T07:33:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.289+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a40b71d0 con 0x7fa9a40b5020 2024-03-06T07:33:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.291+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a408cf90 con 0x7fa9a40b5020 2024-03-06T07:33:45.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.291+0000 7fa99be00640 1 -- 172.21.15.114:0/1753771069 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98c018070 con 0x7fa9a40b5020 2024-03-06T07:33:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.292+0000 7fa99be00640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.292+0000 7fa99be00640 1 -- 172.21.15.114:0/1753771069 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fa98c08a3e0 con 0x7fa9a40b5020 2024-03-06T07:33:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.292+0000 7fa9a3400640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.293+0000 7fa9a3400640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa98006b4d0 0x7fa98006d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa994007fd0 tx=0x7fa994007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.297+0000 7fa99be00640 1 -- 172.21.15.114:0/1753771069 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9a408cf90 con 0x7fa9a40b5020 2024-03-06T07:33:45.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.495+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9a4068070 con 0x7fa9a40b5020 2024-03-06T07:33:45.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.497+0000 7fa99be00640 1 -- 172.21.15.114:0/1753771069 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16917 (secure 0 0 0) 0x7fa98c0591b0 con 0x7fa9a40b5020 2024-03-06T07:33:45.500 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:45.500 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":97,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:16.974876+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:03.077157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:04.314220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6836/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6835/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6837/406262499":"2024-03-07T07:33:15.968840+0000","172.21.15.114:6834/406262499":"2024-03-07T07:33:15.968840+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa98006b4d0 msgr2=0x7fa98006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa98006b4d0 0x7fa98006d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa994007fd0 tx=0x7fa994007920 comp rx=0 tx=0).stop 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 msgr2=0x7fa9a41f7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a41f7230 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00c7f0 tx=0x7fa98c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 shutdown_connections 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a40d2cc0 0x7fa9a41f7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa98006b4d0 0x7fa98006d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9a40b5020 0x7fa9a41f7230 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 --2- 172.21.15.114:0/1753771069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa9a40b44f0 0x7fa9a4156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 >> 172.21.15.114:0/1753771069 conn(0x7fa9a40d3fd0 msgr2=0x7fa9a40b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 shutdown_connections 2024-03-06T07:33:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.500+0000 7fa9aa000640 1 -- 172.21.15.114:0/1753771069 wait complete. 2024-03-06T07:33:45.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:33:03.077157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:33:04.314220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:45.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:45.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.714+0000 7f1505c00640 1 Processor -- start 2024-03-06T07:33:45.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.714+0000 7f1505c00640 1 -- start start 2024-03-06T07:33:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.715+0000 7f1505c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f15000d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f1505c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15000b4790 0x7f15000b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f1505c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15000b39d0 con 0x7f150015e230 2024-03-06T07:33:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f1505c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15000b3780 con 0x7f15000b4790 2024-03-06T07:33:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f1505c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15000d2d80 con 0x7f15000d2400 2024-03-06T07:33:45.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f1505200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15000b4790 0x7f15000b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f1505200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15000b4790 0x7f15000b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51508/0 (socket says 172.21.15.114:51508) 2024-03-06T07:33:45.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f1505200640 1 -- 172.21.15.114:0/2196707605 learned_addr learned my addr 172.21.15.114:0/2196707605 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:45.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.716+0000 7f14ff400640 1 --2- 172.21.15.114:0/2196707605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f15000d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.717+0000 7f1505200640 1 -- 172.21.15.114:0/2196707605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 msgr2=0x7f15000d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.717+0000 7f1505200640 1 --2- 172.21.15.114:0/2196707605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f15000d2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.717+0000 7f1505200640 1 -- 172.21.15.114:0/2196707605 >> v1:172.21.15.177:6789/0 conn(0x7f150015e230 legacy=0x7f15000d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.717+0000 7f1505200640 1 -- 172.21.15.114:0/2196707605 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000b3cb0 con 0x7f15000b4790 2024-03-06T07:33:45.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.718+0000 7f1505200640 1 --2- 172.21.15.114:0/2196707605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f15000b4c20 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009e50 tx=0x7f14f802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ea9f2fc74b5f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.718+0000 7f14fea00640 1 -- 172.21.15.114:0/2196707605 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f802cb90 con 0x7f15000b4790 2024-03-06T07:33:45.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.718+0000 7f14fea00640 1 -- 172.21.15.114:0/2196707605 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f802cd30 con 0x7f15000b4790 2024-03-06T07:33:45.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.718+0000 7f14fea00640 1 -- 172.21.15.114:0/2196707605 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8034610 con 0x7f15000b4790 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.718+0000 7f1505c00640 1 -- 172.21.15.114:0/2196707605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 msgr2=0x7f15000b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.719+0000 7f1505c00640 1 --2- 172.21.15.114:0/2196707605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f15000b4c20 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009e50 tx=0x7f14f802bff0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.719+0000 7f1505c00640 1 -- 172.21.15.114:0/2196707605 shutdown_connections 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.719+0000 7f1505c00640 1 --2- 172.21.15.114:0/2196707605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f15000b4c20 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.719+0000 7f1505c00640 1 --2- 172.21.15.114:0/2196707605 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f15000d2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.719+0000 7f1505c00640 1 -- 172.21.15.114:0/2196707605 >> 172.21.15.114:0/2196707605 conn(0x7f15000d3e20 msgr2=0x7f15000d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.719+0000 7f1505c00640 1 -- 172.21.15.114:0/2196707605 shutdown_connections 2024-03-06T07:33:45.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.719+0000 7f1505c00640 1 -- 172.21.15.114:0/2196707605 wait complete. 2024-03-06T07:33:45.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.720+0000 7f1505c00640 1 Processor -- start 2024-03-06T07:33:45.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.721+0000 7f1505c00640 1 -- start start 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f1505c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f1500159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f1505c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f150015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f1505c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f150015e230 0x7f15000cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f1505c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150015e610 con 0x7f15000b4790 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f1505c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15000b4270 con 0x7f150015e230 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f1505c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15000d1570 con 0x7f15000d2400 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f1500159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f1500159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51522/0 (socket says 172.21.15.114:51522) 2024-03-06T07:33:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 -- 172.21.15.114:0/2419449751 learned_addr learned my addr 172.21.15.114:0/2419449751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f1505200640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f150015e230 0x7f15000cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ff400640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f150015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 -- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 msgr2=0x7f150015a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f150015a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 -- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f150015e230 msgr2=0x7f15000cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f150015e230 0x7f15000cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ffe00640 1 -- 172.21.15.114:0/2419449751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000cc7d0 con 0x7f15000b4790 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.722+0000 7f14ff400640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f150015a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:45.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.723+0000 7f14ffe00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f1500159d20 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c3c0 tx=0x7f14f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.723+0000 7f14f7e00640 1 -- 172.21.15.114:0/2419449751 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f00052d0 con 0x7f15000b4790 2024-03-06T07:33:45.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.723+0000 7f14f7e00640 1 -- 172.21.15.114:0/2419449751 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f0005490 con 0x7f15000b4790 2024-03-06T07:33:45.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.723+0000 7f14f7e00640 1 -- 172.21.15.114:0/2419449751 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0007260 con 0x7f15000b4790 2024-03-06T07:33:45.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.723+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1500160030 con 0x7f15000b4790 2024-03-06T07:33:45.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.723+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150015a870 con 0x7f15000b4790 2024-03-06T07:33:45.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.724+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150004f330 con 0x7f15000b4790 2024-03-06T07:33:45.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.725+0000 7f14f7e00640 1 -- 172.21.15.114:0/2419449751 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14f0007400 con 0x7f15000b4790 2024-03-06T07:33:45.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.726+0000 7f14f7e00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14e006b500 0x7f14e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.726+0000 7f14f7e00640 1 -- 172.21.15.114:0/2419449751 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f14f00897a0 con 0x7f15000b4790 2024-03-06T07:33:45.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.726+0000 7f14ff400640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14e006b500 0x7f14e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.727+0000 7f14ff400640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14e006b500 0x7f14e006d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f15000d2c40 tx=0x7f14ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.731+0000 7f14f7e00640 1 -- 172.21.15.114:0/2419449751 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14f00584d0 con 0x7f15000b4790 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.933+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15000b3030 con 0x7f15000b4790 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.934+0000 7f14f7e00640 1 -- 172.21.15.114:0/2419449751 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14f0057dd0 con 0x7f15000b4790 2024-03-06T07:33:45.937 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.936+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14e006b500 msgr2=0x7f14e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.936+0000 7f1505c00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14e006b500 0x7f14e006d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f15000d2c40 tx=0x7f14ec008040 comp rx=0 tx=0).stop 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.936+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 msgr2=0x7f1500159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.936+0000 7f1505c00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f1500159d20 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c3c0 tx=0x7f14f000c880 comp rx=0 tx=0).stop 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.936+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 shutdown_connections 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.936+0000 7f1505c00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f150015e230 0x7f15000cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.936+0000 7f1505c00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15000d2400 0x7f150015a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.937+0000 7f1505c00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f14e006b500 0x7f14e006d9c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.937+0000 7f1505c00640 1 --2- 172.21.15.114:0/2419449751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15000b4790 0x7f1500159d20 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.937+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 >> 172.21.15.114:0/2419449751 conn(0x7f15000d3e20 msgr2=0x7f150006c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.937+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 shutdown_connections 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:45.937+0000 7f1505c00640 1 -- 172.21.15.114:0/2419449751 wait complete. 2024-03-06T07:33:45.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.165+0000 7fac57800640 1 Processor -- start 2024-03-06T07:33:46.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.165+0000 7fac57800640 1 -- start start 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.166+0000 7fac57800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac50071f10 0x7fac500cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.166+0000 7fac57800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac500722f0 0x7fac500ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.166+0000 7fac57800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5015ec00 con 0x7fac5015b6a0 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.166+0000 7fac57800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5015f9c0 con 0x7fac50071f10 2024-03-06T07:33:46.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.166+0000 7fac57800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5015c410 con 0x7fac500722f0 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.167+0000 7fac55800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fac5015b6a0 0x7fac5015bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33508/0 (socket says 172.21.15.114:33508) 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.167+0000 7fac55800640 1 -- 172.21.15.114:0/2195489201 learned_addr learned my addr 172.21.15.114:0/2195489201 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.167+0000 7fac54e00640 1 --2- 172.21.15.114:0/2195489201 >> v2:172.21.15.114:3300/0 conn(0x7fac50071f10 0x7fac500cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.167+0000 7fac4fe00640 1 --2- 172.21.15.114:0/2195489201 >> v2:172.21.15.177:3300/0 conn(0x7fac500722f0 0x7fac500ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.167+0000 7fac4f400640 1 -- 172.21.15.114:0/2195489201 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669495616 0 0) 0x7fac5015ec00 con 0x7fac5015b6a0 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.167+0000 7fac4f400640 1 -- 172.21.15.114:0/2195489201 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38003520 con 0x7fac5015b6a0 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.168+0000 7fac54e00640 1 -- 172.21.15.114:0/2195489201 >> v2:172.21.15.177:3300/0 conn(0x7fac500722f0 msgr2=0x7fac500ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.168+0000 7fac54e00640 1 --2- 172.21.15.114:0/2195489201 >> v2:172.21.15.177:3300/0 conn(0x7fac500722f0 0x7fac500ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.168+0000 7fac54e00640 1 -- 172.21.15.114:0/2195489201 >> v1:172.21.15.114:6789/0 conn(0x7fac5015b6a0 legacy=0x7fac5015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.168+0000 7fac4f400640 1 -- 172.21.15.114:0/2195489201 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285333309 0 0) 0x7fac38003520 con 0x7fac5015b6a0 2024-03-06T07:33:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.168+0000 7fac54e00640 1 -- 172.21.15.114:0/2195489201 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5015eee0 con 0x7fac50071f10 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.169+0000 7fac54e00640 1 --2- 172.21.15.114:0/2195489201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac50071f10 0x7fac500cca70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fac5015b4e0 tx=0x7fac4402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faee38e27022208 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.169+0000 7fac4f400640 1 -- 172.21.15.114:0/2195489201 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44037070 con 0x7fac50071f10 2024-03-06T07:33:46.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.169+0000 7fac4f400640 1 -- 172.21.15.114:0/2195489201 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac44030510 con 0x7fac50071f10 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.169+0000 7fac4f400640 1 -- 172.21.15.114:0/2195489201 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480019f0 con 0x7fac50071f10 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 -- 172.21.15.114:0/2195489201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac50071f10 msgr2=0x7fac500cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 --2- 172.21.15.114:0/2195489201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac50071f10 0x7fac500cca70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fac5015b4e0 tx=0x7fac4402b710 comp rx=0 tx=0).stop 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 -- 172.21.15.114:0/2195489201 shutdown_connections 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 --2- 172.21.15.114:0/2195489201 >> v2:172.21.15.177:3300/0 conn(0x7fac500722f0 0x7fac500ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 --2- 172.21.15.114:0/2195489201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac50071f10 0x7fac500cca70 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 -- 172.21.15.114:0/2195489201 >> 172.21.15.114:0/2195489201 conn(0x7fac500d0d30 msgr2=0x7fac500d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 -- 172.21.15.114:0/2195489201 shutdown_connections 2024-03-06T07:33:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.170+0000 7fac57800640 1 -- 172.21.15.114:0/2195489201 wait complete. 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.172+0000 7fac57800640 1 Processor -- start 2024-03-06T07:33:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.172+0000 7fac57800640 1 -- start start 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac57800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50071f10 0x7fac500c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac57800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500722f0 0x7fac5014e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac57800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5015b6a0 0x7fac500c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac57800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5006af40 con 0x7fac500722f0 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac57800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015f4a0 con 0x7fac50071f10 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac57800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5006b0c0 con 0x7fac5015b6a0 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500722f0 0x7fac5014e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500722f0 0x7fac5014e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51540/0 (socket says 172.21.15.114:51540) 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 -- 172.21.15.114:0/3882830420 learned_addr learned my addr 172.21.15.114:0/3882830420 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac55800640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5015b6a0 0x7fac500c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 -- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5015b6a0 msgr2=0x7fac500c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac54e00640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50071f10 0x7fac500c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5015b6a0 0x7fac500c6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 -- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50071f10 msgr2=0x7fac500c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50071f10 0x7fac500c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.173+0000 7fac4fe00640 1 -- 172.21.15.114:0/3882830420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac500c7060 con 0x7fac500722f0 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.174+0000 7fac4fe00640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500722f0 0x7fac5014e970 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00e9e0 tx=0x7fac3c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.174+0000 7fac4cc00640 1 -- 172.21.15.114:0/3882830420 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c017070 con 0x7fac500722f0 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.174+0000 7fac4cc00640 1 -- 172.21.15.114:0/3882830420 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c00c9d0 con 0x7fac500722f0 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.174+0000 7fac4cc00640 1 -- 172.21.15.114:0/3882830420 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c00cdc0 con 0x7fac500722f0 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.174+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac500c8f50 con 0x7fac500722f0 2024-03-06T07:33:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.174+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac500c7f30 con 0x7fac500722f0 2024-03-06T07:33:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.175+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac50068070 con 0x7fac500722f0 2024-03-06T07:33:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.176+0000 7fac4cc00640 1 -- 172.21.15.114:0/3882830420 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac3c01a070 con 0x7fac500722f0 2024-03-06T07:33:46.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.178+0000 7fac4cc00640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac3006b4d0 0x7fac3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.178+0000 7fac4cc00640 1 -- 172.21.15.114:0/3882830420 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fac3c08a350 con 0x7fac500722f0 2024-03-06T07:33:46.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.178+0000 7fac54e00640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac3006b4d0 0x7fac3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.179+0000 7fac54e00640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac3006b4d0 0x7fac3006d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fac44034000 tx=0x7fac44008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.181+0000 7fac4cc00640 1 -- 172.21.15.114:0/3882830420 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac3c059110 con 0x7fac500722f0 2024-03-06T07:33:46.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:46.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:46.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.464+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac5006fd60 con 0x7fac500722f0 2024-03-06T07:33:46.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.465+0000 7fac4cc00640 1 -- 172.21.15.114:0/3882830420 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7fac3c058a10 con 0x7fac500722f0 2024-03-06T07:33:46.468 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:33:46.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac3006b4d0 msgr2=0x7fac3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac3006b4d0 0x7fac3006d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fac44034000 tx=0x7fac44008030 comp rx=0 tx=0).stop 2024-03-06T07:33:46.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500722f0 msgr2=0x7fac5014e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500722f0 0x7fac5014e970 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00e9e0 tx=0x7fac3c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 shutdown_connections 2024-03-06T07:33:46.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac5015b6a0 0x7fac500c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac3006b4d0 0x7fac3006d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500722f0 0x7fac5014e970 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 --2- 172.21.15.114:0/3882830420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac50071f10 0x7fac500c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 >> 172.21.15.114:0/3882830420 conn(0x7fac500d0d30 msgr2=0x7fac500cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.468+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 shutdown_connections 2024-03-06T07:33:46.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.469+0000 7fac57800640 1 -- 172.21.15.114:0/3882830420 wait complete. 2024-03-06T07:33:46.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.683+0000 7fea4e200640 1 Processor -- start 2024-03-06T07:33:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.684+0000 7fea4e200640 1 -- start start 2024-03-06T07:33:46.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.685+0000 7fea4e200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea48155490 0x7fea480ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.685+0000 7fea4e200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea480d0070 0x7fea480d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.685+0000 7fea4e200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea48155870 con 0x7fea4815ec10 2024-03-06T07:33:46.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.685+0000 7fea4e200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea480ad640 con 0x7fea480d0070 2024-03-06T07:33:46.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.685+0000 7fea4e200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4815f980 con 0x7fea48155490 2024-03-06T07:33:46.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea3fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea480d0070 0x7fea480d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea3fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea480d0070 0x7fea480d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51554/0 (socket says 172.21.15.114:51554) 2024-03-06T07:33:46.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea3fe00640 1 -- 172.21.15.114:0/1210186789 learned_addr learned my addr 172.21.15.114:0/1210186789 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:46.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea47400640 1 --2- 172.21.15.114:0/1210186789 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea48155490 0x7fea480ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea3fe00640 1 -- 172.21.15.114:0/1210186789 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea48155490 msgr2=0x7fea480ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea3fe00640 1 --2- 172.21.15.114:0/1210186789 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea48155490 0x7fea480ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea3fe00640 1 -- 172.21.15.114:0/1210186789 >> v1:172.21.15.177:6789/0 conn(0x7fea4815ec10 legacy=0x7fea4815f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.686+0000 7fea3fe00640 1 -- 172.21.15.114:0/1210186789 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea480ad870 con 0x7fea480d0070 2024-03-06T07:33:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.687+0000 7fea3fe00640 1 --2- 172.21.15.114:0/1210186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea480d0070 0x7fea480d04f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fea30009870 tx=0x7fea3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59760d42c4ea80bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea46a00640 1 -- 172.21.15.114:0/1210186789 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30035020 con 0x7fea480d0070 2024-03-06T07:33:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea46a00640 1 -- 172.21.15.114:0/1210186789 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30031640 con 0x7fea480d0070 2024-03-06T07:33:46.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea46a00640 1 -- 172.21.15.114:0/1210186789 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3002d040 con 0x7fea480d0070 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 -- 172.21.15.114:0/1210186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea480d0070 msgr2=0x7fea480d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 --2- 172.21.15.114:0/1210186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea480d0070 0x7fea480d04f0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fea30009870 tx=0x7fea3002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 -- 172.21.15.114:0/1210186789 shutdown_connections 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 --2- 172.21.15.114:0/1210186789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea480d0070 0x7fea480d04f0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 --2- 172.21.15.114:0/1210186789 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea48155490 0x7fea480ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 -- 172.21.15.114:0/1210186789 >> 172.21.15.114:0/1210186789 conn(0x7fea4806b360 msgr2=0x7fea4806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 -- 172.21.15.114:0/1210186789 shutdown_connections 2024-03-06T07:33:46.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.688+0000 7fea4e200640 1 -- 172.21.15.114:0/1210186789 wait complete. 2024-03-06T07:33:46.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.690+0000 7fea4e200640 1 Processor -- start 2024-03-06T07:33:46.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.690+0000 7fea4e200640 1 -- start start 2024-03-06T07:33:46.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea4e200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea480d0070 0x7fea481415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea4e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea48155490 0x7fea48151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea4e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea4815ec10 0x7fea4815db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea4e200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea480ae110 con 0x7fea48155490 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea4e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea480ade30 con 0x7fea4815ec10 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea4e200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea48071960 con 0x7fea480d0070 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea3fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea48155490 0x7fea48151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea3fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea48155490 0x7fea48151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51566/0 (socket says 172.21.15.114:51566) 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.691+0000 7fea3fe00640 1 -- 172.21.15.114:0/1708642154 learned_addr learned my addr 172.21.15.114:0/1708642154 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.692+0000 7fea3fe00640 1 -- 172.21.15.114:0/1708642154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea480d0070 msgr2=0x7fea481415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.692+0000 7fea3fe00640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea480d0070 0x7fea481415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.692+0000 7fea3fe00640 1 -- 172.21.15.114:0/1708642154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea4815ec10 msgr2=0x7fea4815db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.692+0000 7fea3fe00640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea4815ec10 0x7fea4815db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.692+0000 7fea3fe00640 1 -- 172.21.15.114:0/1708642154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4815c900 con 0x7fea48155490 2024-03-06T07:33:46.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.692+0000 7fea3fe00640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea48155490 0x7fea48151fc0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fea30000c00 tx=0x7fea300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.693+0000 7fea3f400640 1 -- 172.21.15.114:0/1708642154 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30040070 con 0x7fea48155490 2024-03-06T07:33:46.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.693+0000 7fea3f400640 1 -- 172.21.15.114:0/1708642154 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea300399f0 con 0x7fea48155490 2024-03-06T07:33:46.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.693+0000 7fea3f400640 1 -- 172.21.15.114:0/1708642154 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30035040 con 0x7fea48155490 2024-03-06T07:33:46.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.693+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4815cbe0 con 0x7fea48155490 2024-03-06T07:33:46.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.693+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4815d1a0 con 0x7fea48155490 2024-03-06T07:33:46.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.695+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea48068070 con 0x7fea48155490 2024-03-06T07:33:46.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.695+0000 7fea3f400640 1 -- 172.21.15.114:0/1708642154 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea3004b060 con 0x7fea48155490 2024-03-06T07:33:46.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.696+0000 7fea3f400640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea2406b4d0 0x7fea2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.696+0000 7fea3f400640 1 -- 172.21.15.114:0/1708642154 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fea300abcd0 con 0x7fea48155490 2024-03-06T07:33:46.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.696+0000 7fea47400640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea2406b4d0 0x7fea2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.697+0000 7fea47400640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea2406b4d0 0x7fea2406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fea481f8ea0 tx=0x7fea38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.701+0000 7fea3f400640 1 -- 172.21.15.114:0/1708642154 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea3007aa00 con 0x7fea48155490 2024-03-06T07:33:46.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.942+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fea48071740 con 0x7fea48155490 2024-03-06T07:33:46.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.943+0000 7fea3f400640 1 -- 172.21.15.114:0/1708642154 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7fea3007a300 con 0x7fea48155490 2024-03-06T07:33:46.945 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:33:46.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea2406b4d0 msgr2=0x7fea2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea2406b4d0 0x7fea2406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fea481f8ea0 tx=0x7fea38008040 comp rx=0 tx=0).stop 2024-03-06T07:33:46.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea48155490 msgr2=0x7fea48151fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea48155490 0x7fea48151fc0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fea30000c00 tx=0x7fea300046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 shutdown_connections 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea4815ec10 0x7fea4815db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea2406b4d0 0x7fea2406d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea48155490 0x7fea48151fc0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.945+0000 7fea4e200640 1 --2- 172.21.15.114:0/1708642154 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea480d0070 0x7fea481415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.946+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 >> 172.21.15.114:0/1708642154 conn(0x7fea4806b360 msgr2=0x7fea4815f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.946+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 shutdown_connections 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:46.946+0000 7fea4e200640 1 -- 172.21.15.114:0/1708642154 wait complete. 2024-03-06T07:33:46.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:33:47.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.172+0000 7fd766c00640 1 Processor -- start 2024-03-06T07:33:47.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.172+0000 7fd766c00640 1 -- start start 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.173+0000 7fd766c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd760155490 0x7fd7600ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.173+0000 7fd766c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7600d0070 0x7fd7600d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd766c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd760155870 con 0x7fd76015ec10 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd766c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600ad640 con 0x7fd7600d0070 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd766c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76015f980 con 0x7fd760155490 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7600d0070 0x7fd7600d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7600d0070 0x7fd7600d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51580/0 (socket says 172.21.15.114:51580) 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75f400640 1 -- 172.21.15.114:0/1659280556 learned_addr learned my addr 172.21.15.114:0/1659280556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75fe00640 1 --2- 172.21.15.114:0/1659280556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd760155490 0x7fd7600ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75f400640 1 -- 172.21.15.114:0/1659280556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd760155490 msgr2=0x7fd7600ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75f400640 1 --2- 172.21.15.114:0/1659280556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd760155490 0x7fd7600ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75f400640 1 -- 172.21.15.114:0/1659280556 >> v1:172.21.15.177:6789/0 conn(0x7fd76015ec10 legacy=0x7fd76015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.174+0000 7fd75f400640 1 -- 172.21.15.114:0/1659280556 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600ad870 con 0x7fd7600d0070 2024-03-06T07:33:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.176+0000 7fd75f400640 1 --2- 172.21.15.114:0/1659280556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7600d0070 0x7fd7600d04f0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009870 tx=0x7fd74c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdd1314f8adaf84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.176+0000 7fd75ea00640 1 -- 172.21.15.114:0/1659280556 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c035020 con 0x7fd7600d0070 2024-03-06T07:33:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.176+0000 7fd75ea00640 1 -- 172.21.15.114:0/1659280556 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c031640 con 0x7fd7600d0070 2024-03-06T07:33:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.176+0000 7fd75ea00640 1 -- 172.21.15.114:0/1659280556 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c02d040 con 0x7fd7600d0070 2024-03-06T07:33:47.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.176+0000 7fd766c00640 1 -- 172.21.15.114:0/1659280556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7600d0070 msgr2=0x7fd7600d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.176+0000 7fd766c00640 1 --2- 172.21.15.114:0/1659280556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7600d0070 0x7fd7600d04f0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009870 tx=0x7fd74c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.177+0000 7fd766c00640 1 -- 172.21.15.114:0/1659280556 shutdown_connections 2024-03-06T07:33:47.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.177+0000 7fd766c00640 1 --2- 172.21.15.114:0/1659280556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7600d0070 0x7fd7600d04f0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.177+0000 7fd766c00640 1 --2- 172.21.15.114:0/1659280556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd760155490 0x7fd7600ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.177+0000 7fd766c00640 1 -- 172.21.15.114:0/1659280556 >> 172.21.15.114:0/1659280556 conn(0x7fd76006b360 msgr2=0x7fd76006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.177+0000 7fd766c00640 1 -- 172.21.15.114:0/1659280556 shutdown_connections 2024-03-06T07:33:47.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.177+0000 7fd766c00640 1 -- 172.21.15.114:0/1659280556 wait complete. 2024-03-06T07:33:47.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.178+0000 7fd766c00640 1 Processor -- start 2024-03-06T07:33:47.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.179+0000 7fd766c00640 1 -- start start 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.179+0000 7fd766c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7600d0070 0x7fd7601473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd766c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd760155490 0x7fd760150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd766c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76015ec10 0x7fd7600af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd766c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600ae110 con 0x7fd76015ec10 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd766c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600ade30 con 0x7fd760155490 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd766c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd760071960 con 0x7fd7600d0070 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76015ec10 0x7fd7600af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76015ec10 0x7fd7600af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51594/0 (socket says 172.21.15.114:51594) 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 -- 172.21.15.114:0/1753796921 learned_addr learned my addr 172.21.15.114:0/1753796921 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd75fe00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7600d0070 0x7fd7601473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd75f400640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd760155490 0x7fd760150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 -- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7600d0070 msgr2=0x7fd7601473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7600d0070 0x7fd7601473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 -- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd760155490 msgr2=0x7fd760150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd760155490 0x7fd760150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd764c00640 1 -- 172.21.15.114:0/1753796921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd760149d90 con 0x7fd76015ec10 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd75f400640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd760155490 0x7fd760150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.180+0000 7fd75fe00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7600d0070 0x7fd7601473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.181+0000 7fd764c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76015ec10 0x7fd7600af800 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd75800ba10 tx=0x7fd75800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.181+0000 7fd757e00640 1 -- 172.21.15.114:0/1753796921 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758019070 con 0x7fd76015ec10 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.181+0000 7fd757e00640 1 -- 172.21.15.114:0/1753796921 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75800cac0 con 0x7fd76015ec10 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.181+0000 7fd757e00640 1 -- 172.21.15.114:0/1753796921 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758014440 con 0x7fd76015ec10 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.181+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76014a070 con 0x7fd76015ec10 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.181+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76014a630 con 0x7fd76015ec10 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.182+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd760068070 con 0x7fd76015ec10 2024-03-06T07:33:47.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.183+0000 7fd757e00640 1 -- 172.21.15.114:0/1753796921 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd75801c070 con 0x7fd76015ec10 2024-03-06T07:33:47.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.184+0000 7fd757e00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd74006b4d0 0x7fd74006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.184+0000 7fd757e00640 1 -- 172.21.15.114:0/1753796921 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd758089e40 con 0x7fd76015ec10 2024-03-06T07:33:47.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.184+0000 7fd75fe00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd74006b4d0 0x7fd74006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.185+0000 7fd75fe00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd74006b4d0 0x7fd74006d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd750006fd0 tx=0x7fd750008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.189+0000 7fd757e00640 1 -- 172.21.15.114:0/1753796921 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd758058c00 con 0x7fd76015ec10 2024-03-06T07:33:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.402+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd76014aa70 con 0x7fd76015ec10 2024-03-06T07:33:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.402+0000 7fd757e00640 1 -- 172.21.15.114:0/1753796921 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v97) v1 ==== 80+0+246 (secure 0 0 0) 0x7fd758058500 con 0x7fd76015ec10 2024-03-06T07:33:47.405 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6836/1475876370 2024-03-07T07:33:16.974802+0000 2024-03-06T07:33:47.405 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6835/1475876370 2024-03-07T07:33:16.974802+0000 2024-03-06T07:33:47.405 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6837/406262499 2024-03-07T07:33:15.968840+0000 2024-03-06T07:33:47.405 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6834/406262499 2024-03-07T07:33:15.968840+0000 2024-03-06T07:33:47.406 INFO:teuthology.orchestra.run.smithi114.stderr:listed 4 entries 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd74006b4d0 msgr2=0x7fd74006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd74006b4d0 0x7fd74006d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd750006fd0 tx=0x7fd750008040 comp rx=0 tx=0).stop 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76015ec10 msgr2=0x7fd7600af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76015ec10 0x7fd7600af800 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd75800ba10 tx=0x7fd75800bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 shutdown_connections 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd74006b4d0 0x7fd74006d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76015ec10 0x7fd7600af800 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd760155490 0x7fd760150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 --2- 172.21.15.114:0/1753796921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7600d0070 0x7fd7601473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 >> 172.21.15.114:0/1753796921 conn(0x7fd76006b360 msgr2=0x7fd7600aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 shutdown_connections 2024-03-06T07:33:47.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.405+0000 7fd766c00640 1 -- 172.21.15.114:0/1753796921 wait complete. 2024-03-06T07:33:47.426 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:33:47.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:47.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:47.533 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:33:47.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:47.597 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:33:47.636 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:33:47.636 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:47.684 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:47.684 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:33:47.684 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:47.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:47.731 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:33:47.731 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:47.785 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:47.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:47.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.982+0000 7f534e800640 1 Processor -- start 2024-03-06T07:33:47.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.982+0000 7f534e800640 1 -- start start 2024-03-06T07:33:47.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.985+0000 7f534e800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f534806c9f0 0x7f5348070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.985+0000 7f534e800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f534806cdd0 0x7f534806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.985+0000 7f534e800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534815a9a0 con 0x7f53480b3a50 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.985+0000 7f534e800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5348071370 con 0x7f534806cdd0 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.985+0000 7f534e800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5348071650 con 0x7f534806c9f0 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.985+0000 7f5347e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f534806cdd0 0x7f534806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.986+0000 7f5347e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f534806cdd0 0x7f534806d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43012/0 (socket says 172.21.15.114:43012) 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.986+0000 7f5347e00640 1 -- 172.21.15.114:0/1284132539 learned_addr learned my addr 172.21.15.114:0/1284132539 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.986+0000 7f5347400640 1 --2- 172.21.15.114:0/1284132539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f534806c9f0 0x7f5348070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.986+0000 7f5347e00640 1 -- 172.21.15.114:0/1284132539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f534806c9f0 msgr2=0x7f5348070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.986+0000 7f5347e00640 1 --2- 172.21.15.114:0/1284132539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f534806c9f0 0x7f5348070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.986+0000 7f5347e00640 1 -- 172.21.15.114:0/1284132539 >> v1:172.21.15.177:6789/0 conn(0x7f53480b3a50 legacy=0x7f53480acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.986+0000 7f5347e00640 1 -- 172.21.15.114:0/1284132539 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5348071930 con 0x7f534806cdd0 2024-03-06T07:33:47.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.987+0000 7f5347e00640 1 --2- 172.21.15.114:0/1284132539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806cdd0 0x7f534806d210 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f534809e080 tx=0x7f534002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb2d53b8ed22618 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.987+0000 7f5346000640 1 -- 172.21.15.114:0/1284132539 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534002ca30 con 0x7f534806cdd0 2024-03-06T07:33:47.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.987+0000 7f5346000640 1 -- 172.21.15.114:0/1284132539 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534002cbd0 con 0x7f534806cdd0 2024-03-06T07:33:47.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.987+0000 7f5346000640 1 -- 172.21.15.114:0/1284132539 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53400343f0 con 0x7f534806cdd0 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 -- 172.21.15.114:0/1284132539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806cdd0 msgr2=0x7f534806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 --2- 172.21.15.114:0/1284132539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806cdd0 0x7f534806d210 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f534809e080 tx=0x7f534002be90 comp rx=0 tx=0).stop 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 -- 172.21.15.114:0/1284132539 shutdown_connections 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 --2- 172.21.15.114:0/1284132539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806cdd0 0x7f534806d210 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 --2- 172.21.15.114:0/1284132539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f534806c9f0 0x7f5348070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 -- 172.21.15.114:0/1284132539 >> 172.21.15.114:0/1284132539 conn(0x7f534806b360 msgr2=0x7f534806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 -- 172.21.15.114:0/1284132539 shutdown_connections 2024-03-06T07:33:47.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.988+0000 7f534e800640 1 -- 172.21.15.114:0/1284132539 wait complete. 2024-03-06T07:33:47.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.990+0000 7f534e800640 1 Processor -- start 2024-03-06T07:33:47.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.990+0000 7f534e800640 1 -- start start 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f534e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806c9f0 0x7f534814fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f534e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f534806cdd0 0x7f534815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f534e800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53480b3a50 0x7f534815b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f534e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53480ad5b0 con 0x7f534806c9f0 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f534e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5348071ef0 con 0x7f534806cdd0 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f534e800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534806d750 con 0x7f53480b3a50 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f5347400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806c9f0 0x7f534814fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f5347400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806c9f0 0x7f534814fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43018/0 (socket says 172.21.15.114:43018) 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f5347400640 1 -- 172.21.15.114:0/2301615370 learned_addr learned my addr 172.21.15.114:0/2301615370 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.991+0000 7f5346a00640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f534806cdd0 0x7f534815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5347e00640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53480b3a50 0x7f534815b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5347400640 1 -- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53480b3a50 msgr2=0x7f534815b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5347400640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53480b3a50 0x7f534815b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5347400640 1 -- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f534806cdd0 msgr2=0x7f534815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5347400640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f534806cdd0 0x7f534815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5347400640 1 -- 172.21.15.114:0/2301615370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534809abb0 con 0x7f534806c9f0 2024-03-06T07:33:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5346a00640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f534806cdd0 0x7f534815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f5347400640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806c9f0 0x7f534814fc50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f533800d800 tx=0x7f533800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f533f400640 1 -- 172.21.15.114:0/2301615370 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800b6c0 con 0x7f534806c9f0 2024-03-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f533f400640 1 -- 172.21.15.114:0/2301615370 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5338019070 con 0x7f534806c9f0 2024-03-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f533f400640 1 -- 172.21.15.114:0/2301615370 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800ba80 con 0x7f534806c9f0 2024-03-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534809ae70 con 0x7f534806c9f0 2024-03-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.992+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534815cd10 con 0x7f534806c9f0 2024-03-06T07:33:47.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.994+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534806e630 con 0x7f534806c9f0 2024-03-06T07:33:47.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.995+0000 7f533f400640 1 -- 172.21.15.114:0/2301615370 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f533800bc20 con 0x7f534806c9f0 2024-03-06T07:33:47.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.996+0000 7f533f400640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f532806b4d0 0x7f532806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.996+0000 7f533f400640 1 -- 172.21.15.114:0/2301615370 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f533808a7b0 con 0x7f534806c9f0 2024-03-06T07:33:47.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.996+0000 7f5346a00640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f532806b4d0 0x7f532806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:47.996+0000 7f5346a00640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f532806b4d0 0x7f532806d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f53480b1670 tx=0x7f5334008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.001+0000 7f533f400640 1 -- 172.21.15.114:0/2301615370 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f534806e630 con 0x7f534806c9f0 2024-03-06T07:33:48.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.242+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5348068070 con 0x7f534806c9f0 2024-03-06T07:33:48.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.243+0000 7f533f400640 1 -- 172.21.15.114:0/2301615370 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f5338059560 con 0x7f534806c9f0 2024-03-06T07:33:48.246 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:48.246 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:48.246 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:48.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.245+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f532806b4d0 msgr2=0x7f532806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.245+0000 7f534e800640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f532806b4d0 0x7f532806d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f53480b1670 tx=0x7f5334008040 comp rx=0 tx=0).stop 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.245+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806c9f0 msgr2=0x7f534814fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.245+0000 7f534e800640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806c9f0 0x7f534814fc50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f533800d800 tx=0x7f533800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 shutdown_connections 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53480b3a50 0x7f534815b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f534806cdd0 0x7f534815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f532806b4d0 0x7f532806d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 --2- 172.21.15.114:0/2301615370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f534806c9f0 0x7f534814fc50 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 >> 172.21.15.114:0/2301615370 conn(0x7f534806b360 msgr2=0x7f5348150890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 shutdown_connections 2024-03-06T07:33:48.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.246+0000 7f534e800640 1 -- 172.21.15.114:0/2301615370 wait complete. 2024-03-06T07:33:48.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:48.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.458+0000 7f9791400640 1 Processor -- start 2024-03-06T07:33:48.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.458+0000 7f9791400640 1 -- start start 2024-03-06T07:33:48.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.459+0000 7f9791400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f978c0afb20 0x7f978c071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.459+0000 7f9791400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c0cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.459+0000 7f9791400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c15c530 con 0x7f978c06bf50 2024-03-06T07:33:48.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.460+0000 7f9791400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c15c7f0 con 0x7f978c0afb20 2024-03-06T07:33:48.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.460+0000 7f9791400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c15cad0 con 0x7f978c0cf030 2024-03-06T07:33:48.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.460+0000 7f978b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c0cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.460+0000 7f978a800640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f978c06bf50 0x7f978c06c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:53616/0 (socket says 172.21.15.114:53616) 2024-03-06T07:33:48.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.460+0000 7f978a800640 1 -- 172.21.15.114:0/3021793568 learned_addr learned my addr 172.21.15.114:0/3021793568 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:48.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.460+0000 7f9789e00640 1 --2- 172.21.15.114:0/3021793568 >> v2:172.21.15.177:3300/0 conn(0x7f978c0afb20 0x7f978c071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.461+0000 7f978b200640 1 -- 172.21.15.114:0/3021793568 >> v2:172.21.15.177:3300/0 conn(0x7f978c0afb20 msgr2=0x7f978c071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.461+0000 7f978b200640 1 --2- 172.21.15.114:0/3021793568 >> v2:172.21.15.177:3300/0 conn(0x7f978c0afb20 0x7f978c071ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.461+0000 7f978b200640 1 -- 172.21.15.114:0/3021793568 >> v1:172.21.15.177:6789/0 conn(0x7f978c06bf50 legacy=0x7f978c06c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.461+0000 7f9789400640 1 -- 172.21.15.114:0/3021793568 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263658434 0 0) 0x7f978c15c530 con 0x7f978c06bf50 2024-03-06T07:33:48.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.461+0000 7f978b200640 1 -- 172.21.15.114:0/3021793568 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c15cdb0 con 0x7f978c0cf030 2024-03-06T07:33:48.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.463+0000 7f978b200640 1 --2- 172.21.15.114:0/3021793568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c0cf4e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9784009d90 tx=0x7f978402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdaa42b8135ed4a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.463+0000 7f9789400640 1 -- 172.21.15.114:0/3021793568 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784039070 con 0x7f978c0cf030 2024-03-06T07:33:48.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.463+0000 7f9789400640 1 -- 172.21.15.114:0/3021793568 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978402cc50 con 0x7f978c0cf030 2024-03-06T07:33:48.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.463+0000 7f9789400640 1 -- 172.21.15.114:0/3021793568 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784034440 con 0x7f978c0cf030 2024-03-06T07:33:48.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 -- 172.21.15.114:0/3021793568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 msgr2=0x7f978c0cf4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 --2- 172.21.15.114:0/3021793568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c0cf4e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9784009d90 tx=0x7f978402bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 -- 172.21.15.114:0/3021793568 shutdown_connections 2024-03-06T07:33:48.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 --2- 172.21.15.114:0/3021793568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c0cf4e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 --2- 172.21.15.114:0/3021793568 >> v2:172.21.15.177:3300/0 conn(0x7f978c0afb20 0x7f978c071ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 -- 172.21.15.114:0/3021793568 >> 172.21.15.114:0/3021793568 conn(0x7f978c0713d0 msgr2=0x7f978c0717e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 -- 172.21.15.114:0/3021793568 shutdown_connections 2024-03-06T07:33:48.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.464+0000 7f9791400640 1 -- 172.21.15.114:0/3021793568 wait complete. 2024-03-06T07:33:48.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.466+0000 7f9791400640 1 Processor -- start 2024-03-06T07:33:48.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.466+0000 7f9791400640 1 -- start start 2024-03-06T07:33:48.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.467+0000 7f9791400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c06bf50 0x7f978c148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.467+0000 7f9791400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f978c0afb20 0x7f978c1f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.467+0000 7f9791400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.467+0000 7f9791400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c156860 con 0x7f978c06bf50 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.467+0000 7f9791400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c072430 con 0x7f978c0afb20 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.467+0000 7f9791400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1569e0 con 0x7f978c0cf030 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c06bf50 0x7f978c148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c06bf50 0x7f978c148230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43032/0 (socket says 172.21.15.114:43032) 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 -- 172.21.15.114:0/2719319456 learned_addr learned my addr 172.21.15.114:0/2719319456 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f9789e00640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f978c0afb20 0x7f978c1f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978b200640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 -- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 msgr2=0x7f978c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 -- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f978c0afb20 msgr2=0x7f978c1f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f978c0afb20 0x7f978c1f3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 -- 172.21.15.114:0/2719319456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c148770 con 0x7f978c06bf50 2024-03-06T07:33:48.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.468+0000 7f978a800640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c06bf50 0x7f978c148230 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c4d0 tx=0x7f977c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.469+0000 7f9782a00640 1 -- 172.21.15.114:0/2719319456 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c018070 con 0x7f978c06bf50 2024-03-06T07:33:48.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.469+0000 7f9782a00640 1 -- 172.21.15.114:0/2719319456 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c007500 con 0x7f978c06bf50 2024-03-06T07:33:48.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.469+0000 7f9782a00640 1 -- 172.21.15.114:0/2719319456 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c004a20 con 0x7f978c06bf50 2024-03-06T07:33:48.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.469+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c145b70 con 0x7f978c06bf50 2024-03-06T07:33:48.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.469+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c146410 con 0x7f978c06bf50 2024-03-06T07:33:48.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.470+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c0683d0 con 0x7f978c06bf50 2024-03-06T07:33:48.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.471+0000 7f9782a00640 1 -- 172.21.15.114:0/2719319456 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f977c01b070 con 0x7f978c06bf50 2024-03-06T07:33:48.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.472+0000 7f9782a00640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f976c06b4d0 0x7f976c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.472+0000 7f9782a00640 1 -- 172.21.15.114:0/2719319456 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f977c0896f0 con 0x7f978c06bf50 2024-03-06T07:33:48.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.472+0000 7f9789e00640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f976c06b4d0 0x7f976c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.473+0000 7f9789e00640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f976c06b4d0 0x7f976c06d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f978c15bf70 tx=0x7f9778008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.476+0000 7f9782a00640 1 -- 172.21.15.114:0/2719319456 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f978c0683d0 con 0x7f978c06bf50 2024-03-06T07:33:48.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.671+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f978c068ea0 con 0x7f978c06bf50 2024-03-06T07:33:48.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.674+0000 7f9782a00640 1 -- 172.21.15.114:0/2719319456 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16917 (secure 0 0 0) 0x7f978c148770 con 0x7f978c06bf50 2024-03-06T07:33:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:48.677 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":97,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:16.974876+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:03.077157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:04.314220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6836/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6835/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6837/406262499":"2024-03-07T07:33:15.968840+0000","172.21.15.114:6834/406262499":"2024-03-07T07:33:15.968840+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:48.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f976c06b4d0 msgr2=0x7f976c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f976c06b4d0 0x7f976c06d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f978c15bf70 tx=0x7f9778008040 comp rx=0 tx=0).stop 2024-03-06T07:33:48.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c06bf50 msgr2=0x7f978c148230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c06bf50 0x7f978c148230 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c4d0 tx=0x7f977c00c990 comp rx=0 tx=0).stop 2024-03-06T07:33:48.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 shutdown_connections 2024-03-06T07:33:48.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f978c0cf030 0x7f978c156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f978c0afb20 0x7f978c1f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f976c06b4d0 0x7f976c06d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 --2- 172.21.15.114:0/2719319456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f978c06bf50 0x7f978c148230 secure :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c4d0 tx=0x7f977c00c990 comp rx=0 tx=0).stop 2024-03-06T07:33:48.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.676+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 >> 172.21.15.114:0/2719319456 conn(0x7f978c0713d0 msgr2=0x7f978c0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.677+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 shutdown_connections 2024-03-06T07:33:48.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.677+0000 7f9791400640 1 -- 172.21.15.114:0/2719319456 wait complete. 2024-03-06T07:33:48.699 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:33:03.077157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:33:04.314220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:48.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.906+0000 7fb1ce000640 1 Processor -- start 2024-03-06T07:33:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.906+0000 7fb1ce000640 1 -- start start 2024-03-06T07:33:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.909+0000 7fb1ce000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb1c815be90 0x7fb1c80b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1ce000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1c80d0780 0x7fb1c815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1ce000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c80b5160 con 0x7fb1c80acd60 2024-03-06T07:33:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1ce000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c80b4f30 con 0x7fb1c80d0780 2024-03-06T07:33:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1ce000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c80af240 con 0x7fb1c815be90 2024-03-06T07:33:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1c7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1c80d0780 0x7fb1c815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1c7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb1c80d0780 0x7fb1c815fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43036/0 (socket says 172.21.15.114:43036) 2024-03-06T07:33:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1bfe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb1c80acd60 0x7fb1c80d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38754/0 (socket says 172.21.15.114:38754) 2024-03-06T07:33:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3234918533 learned_addr learned my addr 172.21.15.114:0/3234918533 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3234918533 >> v2:172.21.15.177:3300/0 conn(0x7fb1c815be90 msgr2=0x7fb1c80b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1c7e00640 1 --2- 172.21.15.114:0/3234918533 >> v2:172.21.15.177:3300/0 conn(0x7fb1c815be90 0x7fb1c80b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3234918533 >> v1:172.21.15.114:6789/0 conn(0x7fb1c80acd60 legacy=0x7fb1c80d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.910+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3234918533 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c80ae910 con 0x7fb1c80d0780 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.912+0000 7fb1c7e00640 1 --2- 172.21.15.114:0/3234918533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c80d0780 0x7fb1c815fb70 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fb1c806ebe0 tx=0x7fb1c002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953971d507dd4b2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.912+0000 7fb1c6a00640 1 -- 172.21.15.114:0/3234918533 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c002cb20 con 0x7fb1c80d0780 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.912+0000 7fb1c6a00640 1 -- 172.21.15.114:0/3234918533 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c002ccc0 con 0x7fb1c80d0780 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.912+0000 7fb1c6a00640 1 -- 172.21.15.114:0/3234918533 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0034610 con 0x7fb1c80d0780 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.912+0000 7fb1ce000640 1 -- 172.21.15.114:0/3234918533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c80d0780 msgr2=0x7fb1c815fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.912+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3234918533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c80d0780 0x7fb1c815fb70 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fb1c806ebe0 tx=0x7fb1c002bf80 comp rx=0 tx=0).stop 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.913+0000 7fb1ce000640 1 -- 172.21.15.114:0/3234918533 shutdown_connections 2024-03-06T07:33:48.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.913+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3234918533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c80d0780 0x7fb1c815fb70 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.913+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3234918533 >> v2:172.21.15.177:3300/0 conn(0x7fb1c815be90 0x7fb1c80b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.913+0000 7fb1ce000640 1 -- 172.21.15.114:0/3234918533 >> 172.21.15.114:0/3234918533 conn(0x7fb1c806b360 msgr2=0x7fb1c806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.913+0000 7fb1ce000640 1 -- 172.21.15.114:0/3234918533 shutdown_connections 2024-03-06T07:33:48.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.913+0000 7fb1ce000640 1 -- 172.21.15.114:0/3234918533 wait complete. 2024-03-06T07:33:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.915+0000 7fb1ce000640 1 Processor -- start 2024-03-06T07:33:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.915+0000 7fb1ce000640 1 -- start start 2024-03-06T07:33:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.916+0000 7fb1ce000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c80acd60 0x7fb1c80c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.916+0000 7fb1ce000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c80d0780 0x7fb1c81f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.916+0000 7fb1ce000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c815be90 0x7fb1c81f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.916+0000 7fb1ce000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80b2c80 con 0x7fb1c815be90 2024-03-06T07:33:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.916+0000 7fb1ce000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80aeed0 con 0x7fb1c80acd60 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.916+0000 7fb1ce000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c80b2e00 con 0x7fb1c80d0780 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c815be90 0x7fb1c81f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c815be90 0x7fb1c81f5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43048/0 (socket says 172.21.15.114:43048) 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3916311646 learned_addr learned my addr 172.21.15.114:0/3916311646 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7400640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c80acd60 0x7fb1c80c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1bfe00640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c80d0780 0x7fb1c81f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c80d0780 msgr2=0x7fb1c81f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c80d0780 0x7fb1c81f5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c80acd60 msgr2=0x7fb1c80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c80acd60 0x7fb1c80c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 -- 172.21.15.114:0/3916311646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c815a840 con 0x7fb1c815be90 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.917+0000 7fb1c7e00640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c815be90 0x7fb1c81f5b90 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009e20 tx=0x7fb1c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.918+0000 7fb1bf400640 1 -- 172.21.15.114:0/3916311646 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0043020 con 0x7fb1c815be90 2024-03-06T07:33:48.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.918+0000 7fb1bf400640 1 -- 172.21.15.114:0/3916311646 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c003e450 con 0x7fb1c815be90 2024-03-06T07:33:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.918+0000 7fb1bf400640 1 -- 172.21.15.114:0/3916311646 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0009510 con 0x7fb1c815be90 2024-03-06T07:33:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.918+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c815ab20 con 0x7fb1c815be90 2024-03-06T07:33:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.918+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c815b0e0 con 0x7fb1c815be90 2024-03-06T07:33:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.919+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c8068070 con 0x7fb1c815be90 2024-03-06T07:33:48.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.920+0000 7fb1bf400640 1 -- 172.21.15.114:0/3916311646 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1c0046020 con 0x7fb1c815be90 2024-03-06T07:33:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.921+0000 7fb1bf400640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a406b4d0 0x7fb1a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.921+0000 7fb1bf400640 1 -- 172.21.15.114:0/3916311646 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fb1c00acb30 con 0x7fb1c815be90 2024-03-06T07:33:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.921+0000 7fb1c7400640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a406b4d0 0x7fb1a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.922+0000 7fb1c7400640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a406b4d0 0x7fb1a406d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800b810 tx=0x7fb1b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:48.926+0000 7fb1bf400640 1 -- 172.21.15.114:0/3916311646 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1c004d030 con 0x7fb1c815be90 2024-03-06T07:33:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.122+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1c80cfa30 con 0x7fb1c815be90 2024-03-06T07:33:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.123+0000 7fb1bf400640 1 -- 172.21.15.114:0/3916311646 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1c007bad0 con 0x7fb1c815be90 2024-03-06T07:33:49.126 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a406b4d0 msgr2=0x7fb1a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a406b4d0 0x7fb1a406d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800b810 tx=0x7fb1b8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c815be90 msgr2=0x7fb1c81f5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c815be90 0x7fb1c81f5b90 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009e20 tx=0x7fb1c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 shutdown_connections 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1a406b4d0 0x7fb1a406d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1c815be90 0x7fb1c81f5b90 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1c80d0780 0x7fb1c81f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 --2- 172.21.15.114:0/3916311646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1c80acd60 0x7fb1c80c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 >> 172.21.15.114:0/3916311646 conn(0x7fb1c806b360 msgr2=0x7fb1c80afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 shutdown_connections 2024-03-06T07:33:49.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.126+0000 7fb1ce000640 1 -- 172.21.15.114:0/3916311646 wait complete. 2024-03-06T07:33:49.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:49.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.347+0000 7f351fe00640 1 Processor -- start 2024-03-06T07:33:49.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.347+0000 7f351fe00640 1 -- start start 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35181617a0 0x7f35180ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f35180d1e10 0x7f35180d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180cf070 con 0x7f35180d0cf0 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180b2120 con 0x7f35181617a0 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351fe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180cf350 con 0x7f35180d1e10 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35181617a0 0x7f35180ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351ca00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f35180d0cf0 0x7f351806c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38764/0 (socket says 172.21.15.114:38764) 2024-03-06T07:33:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.349+0000 7f351ca00640 1 -- 172.21.15.114:0/3308479038 learned_addr learned my addr 172.21.15.114:0/3308479038 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:49.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.350+0000 7f351de00640 1 --2- 172.21.15.114:0/3308479038 >> v2:172.21.15.177:3300/0 conn(0x7f35180d1e10 0x7f35180d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.350+0000 7f3517e00640 1 -- 172.21.15.114:0/3308479038 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212789245 0 0) 0x7f35180cf070 con 0x7f35180d0cf0 2024-03-06T07:33:49.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.350+0000 7f3517e00640 1 -- 172.21.15.114:0/3308479038 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3500003520 con 0x7f35180d0cf0 2024-03-06T07:33:49.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.350+0000 7f351d400640 1 -- 172.21.15.114:0/3308479038 >> v2:172.21.15.177:3300/0 conn(0x7f35180d1e10 msgr2=0x7f35180d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.350+0000 7f351d400640 1 --2- 172.21.15.114:0/3308479038 >> v2:172.21.15.177:3300/0 conn(0x7f35180d1e10 0x7f35180d22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.350+0000 7f351d400640 1 -- 172.21.15.114:0/3308479038 >> v1:172.21.15.114:6789/0 conn(0x7f35180d0cf0 legacy=0x7f351806c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.350+0000 7f351d400640 1 -- 172.21.15.114:0/3308479038 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180cf630 con 0x7f35181617a0 2024-03-06T07:33:49.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.351+0000 7f351d400640 1 --2- 172.21.15.114:0/3308479038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f35180ce700 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f351809b860 tx=0x7f350c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2c23eeb12983dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.351+0000 7f3517e00640 1 -- 172.21.15.114:0/3308479038 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c039070 con 0x7f35181617a0 2024-03-06T07:33:49.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.351+0000 7f3517e00640 1 -- 172.21.15.114:0/3308479038 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c02cc80 con 0x7f35181617a0 2024-03-06T07:33:49.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.351+0000 7f3517e00640 1 -- 172.21.15.114:0/3308479038 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c034640 con 0x7f35181617a0 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 -- 172.21.15.114:0/3308479038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 msgr2=0x7f35180ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 --2- 172.21.15.114:0/3308479038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f35180ce700 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f351809b860 tx=0x7f350c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 -- 172.21.15.114:0/3308479038 shutdown_connections 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 --2- 172.21.15.114:0/3308479038 >> v2:172.21.15.177:3300/0 conn(0x7f35180d1e10 0x7f35180d22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 --2- 172.21.15.114:0/3308479038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f35180ce700 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 -- 172.21.15.114:0/3308479038 >> 172.21.15.114:0/3308479038 conn(0x7f351809aa30 msgr2=0x7f351809ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 -- 172.21.15.114:0/3308479038 shutdown_connections 2024-03-06T07:33:49.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.352+0000 7f351fe00640 1 -- 172.21.15.114:0/3308479038 wait complete. 2024-03-06T07:33:49.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.354+0000 7f351fe00640 1 Processor -- start 2024-03-06T07:33:49.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.354+0000 7f351fe00640 1 -- start start 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35180d0cf0 0x7f351814d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180d1e10 0x7f3518149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f351815d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351806cea0 con 0x7f35181617a0 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351806cd20 con 0x7f35180d1e10 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3518161b80 con 0x7f35180d0cf0 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f351815d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f351815d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43074/0 (socket says 172.21.15.114:43074) 2024-03-06T07:33:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.355+0000 7f351de00640 1 -- 172.21.15.114:0/990166682 learned_addr learned my addr 172.21.15.114:0/990166682 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:49.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351d400640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35180d0cf0 0x7f351814d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351de00640 1 -- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35180d0cf0 msgr2=0x7f351814d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351de00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35180d0cf0 0x7f351814d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351de00640 1 -- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180d1e10 msgr2=0x7f3518149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351ca00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180d1e10 0x7f3518149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351de00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180d1e10 0x7f3518149ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351de00640 1 -- 172.21.15.114:0/990166682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351814d710 con 0x7f35181617a0 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351de00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f351815d2d0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f351000d830 tx=0x7f351000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f3515600640 1 -- 172.21.15.114:0/990166682 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000b6c0 con 0x7f35181617a0 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f3515600640 1 -- 172.21.15.114:0/990166682 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3510019070 con 0x7f35181617a0 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351814a420 con 0x7f35181617a0 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.356+0000 7f3515600640 1 -- 172.21.15.114:0/990166682 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000bac0 con 0x7f35181617a0 2024-03-06T07:33:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.357+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351815fcd0 con 0x7f35181617a0 2024-03-06T07:33:49.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.358+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3518068070 con 0x7f35181617a0 2024-03-06T07:33:49.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.359+0000 7f3515600640 1 -- 172.21.15.114:0/990166682 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f351001e460 con 0x7f35181617a0 2024-03-06T07:33:49.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.360+0000 7f3515600640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f34f806b4d0 0x7f34f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.360+0000 7f3515600640 1 -- 172.21.15.114:0/990166682 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f351008a9d0 con 0x7f35181617a0 2024-03-06T07:33:49.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.360+0000 7f351d400640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f34f806b4d0 0x7f34f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.361+0000 7f351d400640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f34f806b4d0 0x7f34f806d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f350c036000 tx=0x7f350c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.364+0000 7f3515600640 1 -- 172.21.15.114:0/990166682 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f351814d710 con 0x7f35181617a0 2024-03-06T07:33:49.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.558+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f35180d08e0 con 0x7f35181617a0 2024-03-06T07:33:49.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.559+0000 7f3515600640 1 -- 172.21.15.114:0/990166682 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3510059700 con 0x7f35181617a0 2024-03-06T07:33:49.562 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:33:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f34f806b4d0 msgr2=0x7f34f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f34f806b4d0 0x7f34f806d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f350c036000 tx=0x7f350c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 msgr2=0x7f351815d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f351815d2d0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f351000d830 tx=0x7f351000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 shutdown_connections 2024-03-06T07:33:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f34f806b4d0 0x7f34f806d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35181617a0 0x7f351815d2d0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35180d1e10 0x7f3518149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 --2- 172.21.15.114:0/990166682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f35180d0cf0 0x7f351814d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 >> 172.21.15.114:0/990166682 conn(0x7f351809aa30 msgr2=0x7f35180af000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 shutdown_connections 2024-03-06T07:33:49.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.562+0000 7f351fe00640 1 -- 172.21.15.114:0/990166682 wait complete. 2024-03-06T07:33:49.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:49.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.782+0000 7f0136600640 1 Processor -- start 2024-03-06T07:33:49.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.782+0000 7f0136600640 1 -- start start 2024-03-06T07:33:49.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.783+0000 7f0136600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0130155490 0x7f01300ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.783+0000 7f0136600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01300d0070 0x7f01300d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.783+0000 7f0136600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0130155870 con 0x7f013015ec10 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.783+0000 7f0136600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01300ad640 con 0x7f01300d0070 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.783+0000 7f0136600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013015f750 con 0x7f0130155490 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f0126a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01300d0070 0x7f01300d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f0126a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f01300d0070 0x7f01300d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43078/0 (socket says 172.21.15.114:43078) 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f0126a00640 1 -- 172.21.15.114:0/1211114052 learned_addr learned my addr 172.21.15.114:0/1211114052 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f012f400640 1 --2- 172.21.15.114:0/1211114052 >> v2:172.21.15.177:3300/0 conn(0x7f0130155490 0x7f01300ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f0126a00640 1 -- 172.21.15.114:0/1211114052 >> v2:172.21.15.177:3300/0 conn(0x7f0130155490 msgr2=0x7f01300ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f0126a00640 1 --2- 172.21.15.114:0/1211114052 >> v2:172.21.15.177:3300/0 conn(0x7f0130155490 0x7f01300ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f0126a00640 1 -- 172.21.15.114:0/1211114052 >> v1:172.21.15.114:6789/0 conn(0x7f013015ec10 legacy=0x7f013015f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.785+0000 7f0126a00640 1 -- 172.21.15.114:0/1211114052 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300ad870 con 0x7f01300d0070 2024-03-06T07:33:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.784+0000 7f012ea00640 1 -- 172.21.15.114:0/1211114052 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384980939 0 0) 0x7f0130155870 con 0x7f013015ec10 2024-03-06T07:33:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.786+0000 7f0126a00640 1 --2- 172.21.15.114:0/1211114052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f01300d04f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f013006ed50 tx=0x7f011802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2bc99780d0f78ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.786+0000 7f012ea00640 1 -- 172.21.15.114:0/1211114052 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118034070 con 0x7f01300d0070 2024-03-06T07:33:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.786+0000 7f012ea00640 1 -- 172.21.15.114:0/1211114052 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0118030860 con 0x7f01300d0070 2024-03-06T07:33:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.786+0000 7f012ea00640 1 -- 172.21.15.114:0/1211114052 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118030c30 con 0x7f01300d0070 2024-03-06T07:33:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.786+0000 7f0136600640 1 -- 172.21.15.114:0/1211114052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 msgr2=0x7f01300d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.786+0000 7f0136600640 1 --2- 172.21.15.114:0/1211114052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f01300d04f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f013006ed50 tx=0x7f011802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.787+0000 7f0136600640 1 -- 172.21.15.114:0/1211114052 shutdown_connections 2024-03-06T07:33:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.787+0000 7f0136600640 1 --2- 172.21.15.114:0/1211114052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f01300d04f0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.787+0000 7f0136600640 1 --2- 172.21.15.114:0/1211114052 >> v2:172.21.15.177:3300/0 conn(0x7f0130155490 0x7f01300ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.787+0000 7f0136600640 1 -- 172.21.15.114:0/1211114052 >> 172.21.15.114:0/1211114052 conn(0x7f013006b360 msgr2=0x7f013006b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.787+0000 7f0136600640 1 -- 172.21.15.114:0/1211114052 shutdown_connections 2024-03-06T07:33:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.787+0000 7f0136600640 1 -- 172.21.15.114:0/1211114052 wait complete. 2024-03-06T07:33:49.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.788+0000 7f0136600640 1 Processor -- start 2024-03-06T07:33:49.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.788+0000 7f0136600640 1 -- start start 2024-03-06T07:33:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.789+0000 7f0136600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f0130146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.789+0000 7f0136600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0130155490 0x7f013015c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.789+0000 7f0136600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f013015ec10 0x7f0130155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.789+0000 7f0136600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01300ade30 con 0x7f01300d0070 2024-03-06T07:33:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.789+0000 7f0136600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013015f980 con 0x7f013015ec10 2024-03-06T07:33:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.789+0000 7f0136600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01300ae110 con 0x7f0130155490 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f0130146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f0130146860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43090/0 (socket says 172.21.15.114:43090) 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 -- 172.21.15.114:0/2024562326 learned_addr learned my addr 172.21.15.114:0/2024562326 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f0126a00640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0130155490 0x7f013015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012fe00640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f013015ec10 0x7f0130155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 -- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0130155490 msgr2=0x7f013015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0130155490 0x7f013015c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 -- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f013015ec10 msgr2=0x7f0130155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f013015ec10 0x7f0130155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 -- 172.21.15.114:0/2024562326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013015cfa0 con 0x7f01300d0070 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f0126a00640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0130155490 0x7f013015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012fe00640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f013015ec10 0x7f0130155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.790+0000 7f012f400640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f0130146860 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f012000d830 tx=0x7f012000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.791+0000 7f0127e00640 1 -- 172.21.15.114:0/2024562326 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012000b6c0 con 0x7f01300d0070 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.791+0000 7f0127e00640 1 -- 172.21.15.114:0/2024562326 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0120019070 con 0x7f01300d0070 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.791+0000 7f0127e00640 1 -- 172.21.15.114:0/2024562326 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012000ba80 con 0x7f01300d0070 2024-03-06T07:33:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.791+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013015d280 con 0x7f01300d0070 2024-03-06T07:33:49.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.791+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013015d840 con 0x7f01300d0070 2024-03-06T07:33:49.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.792+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0130068070 con 0x7f01300d0070 2024-03-06T07:33:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.793+0000 7f0127e00640 1 -- 172.21.15.114:0/2024562326 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f012001e460 con 0x7f01300d0070 2024-03-06T07:33:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.794+0000 7f0127e00640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.794+0000 7f0127e00640 1 -- 172.21.15.114:0/2024562326 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f012008a960 con 0x7f01300d0070 2024-03-06T07:33:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.794+0000 7f0126a00640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.795+0000 7f0126a00640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f010c06b4d0 0x7f010c06d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0118036000 tx=0x7f011803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.799+0000 7f0127e00640 1 -- 172.21.15.114:0/2024562326 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0120059730 con 0x7f01300d0070 2024-03-06T07:33:49.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.992+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f013015f300 con 0x7f01300d0070 2024-03-06T07:33:49.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.993+0000 7f0127e00640 1 -- 172.21.15.114:0/2024562326 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0120059030 con 0x7f01300d0070 2024-03-06T07:33:49.995 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f010c06b4d0 msgr2=0x7f010c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f010c06b4d0 0x7f010c06d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0118036000 tx=0x7f011803b040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 msgr2=0x7f0130146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f0130146860 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f012000d830 tx=0x7f012000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 shutdown_connections 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f013015ec10 0x7f0130155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0130155490 0x7f013015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f010c06b4d0 0x7f010c06d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 --2- 172.21.15.114:0/2024562326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01300d0070 0x7f0130146860 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 >> 172.21.15.114:0/2024562326 conn(0x7f013006b360 msgr2=0x7f013015a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 shutdown_connections 2024-03-06T07:33:49.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:49.995+0000 7f0136600640 1 -- 172.21.15.114:0/2024562326 wait complete. 2024-03-06T07:33:50.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:50.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.213+0000 7f72da200640 1 Processor -- start 2024-03-06T07:33:50.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.213+0000 7f72da200640 1 -- start start 2024-03-06T07:33:50.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.214+0000 7f72da200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f72d40d0780 0x7f72d40ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.214+0000 7f72da200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72d415ec10 0x7f72d415f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.214+0000 7f72da200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d4155870 con 0x7f72d4155490 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.214+0000 7f72da200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d40ad640 con 0x7f72d415ec10 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.214+0000 7f72da200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d415f980 con 0x7f72d40d0780 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72d415ec10 0x7f72d415f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f72d4155490 0x7f72d40d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38788/0 (socket says 172.21.15.114:38788) 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d3400640 1 -- 172.21.15.114:0/3356244304 learned_addr learned my addr 172.21.15.114:0/3356244304 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d2a00640 1 --2- 172.21.15.114:0/3356244304 >> v2:172.21.15.177:3300/0 conn(0x7f72d40d0780 0x7f72d40ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d3e00640 1 -- 172.21.15.114:0/3356244304 >> v2:172.21.15.177:3300/0 conn(0x7f72d40d0780 msgr2=0x7f72d40ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d3e00640 1 --2- 172.21.15.114:0/3356244304 >> v2:172.21.15.177:3300/0 conn(0x7f72d40d0780 0x7f72d40ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d3e00640 1 -- 172.21.15.114:0/3356244304 >> v1:172.21.15.114:6789/0 conn(0x7f72d4155490 legacy=0x7f72d40d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d2000640 1 -- 172.21.15.114:0/3356244304 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531508999 0 0) 0x7f72d4155870 con 0x7f72d4155490 2024-03-06T07:33:50.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.215+0000 7f72d3e00640 1 -- 172.21.15.114:0/3356244304 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d40ad870 con 0x7f72d415ec10 2024-03-06T07:33:50.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.217+0000 7f72d3e00640 1 --2- 172.21.15.114:0/3356244304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d415ec10 0x7f72d415f040 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f72d406eb90 tx=0x7f72cc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896082131877aaf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.217+0000 7f72d2000640 1 -- 172.21.15.114:0/3356244304 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc039070 con 0x7f72d415ec10 2024-03-06T07:33:50.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.217+0000 7f72d2000640 1 -- 172.21.15.114:0/3356244304 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc02cc70 con 0x7f72d415ec10 2024-03-06T07:33:50.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.217+0000 7f72d2000640 1 -- 172.21.15.114:0/3356244304 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc034660 con 0x7f72d415ec10 2024-03-06T07:33:50.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 -- 172.21.15.114:0/3356244304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d415ec10 msgr2=0x7f72d415f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 --2- 172.21.15.114:0/3356244304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d415ec10 0x7f72d415f040 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f72d406eb90 tx=0x7f72cc02bec0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 -- 172.21.15.114:0/3356244304 shutdown_connections 2024-03-06T07:33:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 --2- 172.21.15.114:0/3356244304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d415ec10 0x7f72d415f040 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 --2- 172.21.15.114:0/3356244304 >> v2:172.21.15.177:3300/0 conn(0x7f72d40d0780 0x7f72d40ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 -- 172.21.15.114:0/3356244304 >> 172.21.15.114:0/3356244304 conn(0x7f72d406b360 msgr2=0x7f72d406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 -- 172.21.15.114:0/3356244304 shutdown_connections 2024-03-06T07:33:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.218+0000 7f72da200640 1 -- 172.21.15.114:0/3356244304 wait complete. 2024-03-06T07:33:50.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.220+0000 7f72da200640 1 Processor -- start 2024-03-06T07:33:50.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.220+0000 7f72da200640 1 -- start start 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72da200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d40d0780 0x7f72d414ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72da200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d4155490 0x7f72d41597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72da200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d415ec10 0x7f72d4159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72da200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d40ae110 con 0x7f72d4155490 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72da200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d40ade30 con 0x7f72d40d0780 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72da200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d415f580 con 0x7f72d415ec10 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d4155490 0x7f72d41597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d4155490 0x7f72d41597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43106/0 (socket says 172.21.15.114:43106) 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 -- 172.21.15.114:0/3424670078 learned_addr learned my addr 172.21.15.114:0/3424670078 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d3400640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d40d0780 0x7f72d414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 -- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d415ec10 msgr2=0x7f72d4159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d3e00640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d415ec10 0x7f72d4159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d415ec10 0x7f72d4159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 -- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d40d0780 msgr2=0x7f72d414ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d40d0780 0x7f72d414ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d2a00640 1 -- 172.21.15.114:0/3424670078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d40b2770 con 0x7f72d4155490 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.221+0000 7f72d3400640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d40d0780 0x7f72d414ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.222+0000 7f72d2a00640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d4155490 0x7f72d41597b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f72c000d830 tx=0x7f72c000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.222+0000 7f72cb400640 1 -- 172.21.15.114:0/3424670078 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c000b6c0 con 0x7f72d4155490 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.222+0000 7f72cb400640 1 -- 172.21.15.114:0/3424670078 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c0019070 con 0x7f72d4155490 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.222+0000 7f72cb400640 1 -- 172.21.15.114:0/3424670078 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c000ba80 con 0x7f72d4155490 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.222+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d40b2a50 con 0x7f72d4155490 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.222+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d40afbb0 con 0x7f72d4155490 2024-03-06T07:33:50.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.224+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d4068070 con 0x7f72d4155490 2024-03-06T07:33:50.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.224+0000 7f72cb400640 1 -- 172.21.15.114:0/3424670078 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72c001e460 con 0x7f72d4155490 2024-03-06T07:33:50.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.225+0000 7f72cb400640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f72b406b4d0 0x7f72b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.225+0000 7f72cb400640 1 -- 172.21.15.114:0/3424670078 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f72c008a8d0 con 0x7f72d4155490 2024-03-06T07:33:50.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.225+0000 7f72d3400640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f72b406b4d0 0x7f72b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.226+0000 7f72d3400640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f72b406b4d0 0x7f72b406d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f72c4000f60 tx=0x7f72c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.235+0000 7f72cb400640 1 -- 172.21.15.114:0/3424670078 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72c0059730 con 0x7f72d4155490 2024-03-06T07:33:50.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.465+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72d4071740 con 0x7f72d4155490 2024-03-06T07:33:50.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.466+0000 7f72cb400640 1 -- 172.21.15.114:0/3424670078 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f72c0059030 con 0x7f72d4155490 2024-03-06T07:33:50.469 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:50.469 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:50.470 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f72b406b4d0 msgr2=0x7f72b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f72b406b4d0 0x7f72b406d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f72c4000f60 tx=0x7f72c4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d4155490 msgr2=0x7f72d41597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d4155490 0x7f72d41597b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f72c000d830 tx=0x7f72c000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 shutdown_connections 2024-03-06T07:33:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f72d415ec10 0x7f72d4159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f72b406b4d0 0x7f72b406d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72d4155490 0x7f72d41597b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 --2- 172.21.15.114:0/3424670078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f72d40d0780 0x7f72d414ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 >> 172.21.15.114:0/3424670078 conn(0x7f72d406b360 msgr2=0x7f72d40d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 shutdown_connections 2024-03-06T07:33:50.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.469+0000 7f72da200640 1 -- 172.21.15.114:0/3424670078 wait complete. 2024-03-06T07:33:50.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:50.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.679+0000 7ffa0be00640 1 Processor -- start 2024-03-06T07:33:50.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.679+0000 7ffa0be00640 1 -- start start 2024-03-06T07:33:50.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.680+0000 7ffa0be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c15e230 0x7ffa0c0d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.680+0000 7ffa0be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffa0c0d13a0 0x7ffa0c0d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.680+0000 7ffa0be00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0c0b39d0 con 0x7ffa0c0d2940 2024-03-06T07:33:50.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.680+0000 7ffa0be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa0c0b3780 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.680+0000 7ffa0be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c0b3cb0 con 0x7ffa0c15e230 2024-03-06T07:33:50.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.681+0000 7ffa0aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c15e230 0x7ffa0c0d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.681+0000 7ffa0aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c15e230 0x7ffa0c0d0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:54522/0 (socket says 172.21.15.114:54522) 2024-03-06T07:33:50.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.681+0000 7ffa0aa00640 1 -- 172.21.15.114:0/4286447683 learned_addr learned my addr 172.21.15.114:0/4286447683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:50.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.681+0000 7ffa0a000640 1 --2- 172.21.15.114:0/4286447683 >> v2:172.21.15.177:3300/0 conn(0x7ffa0c0d13a0 0x7ffa0c0d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.682+0000 7ffa0a000640 1 -- 172.21.15.114:0/4286447683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c15e230 msgr2=0x7ffa0c0d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.682+0000 7ffa0a000640 1 --2- 172.21.15.114:0/4286447683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c15e230 0x7ffa0c0d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.682+0000 7ffa0a000640 1 -- 172.21.15.114:0/4286447683 >> v1:172.21.15.177:6789/0 conn(0x7ffa0c0d2940 legacy=0x7ffa0c0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.682+0000 7ffa0a000640 1 -- 172.21.15.114:0/4286447683 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c0b3f90 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.683+0000 7ffa0a000640 1 --2- 172.21.15.114:0/4286447683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0d2400 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40097a0 tx=0x7ff9f402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73093afb79db7d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.683+0000 7ffa09600640 1 -- 172.21.15.114:0/4286447683 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4034070 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa09600640 1 -- 172.21.15.114:0/4286447683 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f4030800 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa09600640 1 -- 172.21.15.114:0/4286447683 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4030bd0 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 -- 172.21.15.114:0/4286447683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c0d13a0 msgr2=0x7ffa0c0d2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 --2- 172.21.15.114:0/4286447683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0d2400 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40097a0 tx=0x7ff9f402b940 comp rx=0 tx=0).stop 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 -- 172.21.15.114:0/4286447683 shutdown_connections 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 --2- 172.21.15.114:0/4286447683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0d2400 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 --2- 172.21.15.114:0/4286447683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c15e230 0x7ffa0c0d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 -- 172.21.15.114:0/4286447683 >> 172.21.15.114:0/4286447683 conn(0x7ffa0c0d3e20 msgr2=0x7ffa0c0d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 -- 172.21.15.114:0/4286447683 shutdown_connections 2024-03-06T07:33:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.684+0000 7ffa0be00640 1 -- 172.21.15.114:0/4286447683 wait complete. 2024-03-06T07:33:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.686+0000 7ffa0be00640 1 Processor -- start 2024-03-06T07:33:50.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.686+0000 7ffa0be00640 1 -- start start 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c0d2940 0x7ffa0c152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c15e230 0x7ffa0c0b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c0d1d90 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c15e610 con 0x7ffa0c15e230 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c0d1f10 con 0x7ffa0c0d2940 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43112/0 (socket says 172.21.15.114:43112) 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.687+0000 7ffa0aa00640 1 -- 172.21.15.114:0/2831570334 learned_addr learned my addr 172.21.15.114:0/2831570334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0aa00640 1 -- 172.21.15.114:0/2831570334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c0d2940 msgr2=0x7ffa0c152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0a000640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c0d2940 0x7ffa0c152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0aa00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c0d2940 0x7ffa0c152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0aa00640 1 -- 172.21.15.114:0/2831570334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c15e230 msgr2=0x7ffa0c0b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0aa00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c15e230 0x7ffa0c0b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0aa00640 1 -- 172.21.15.114:0/2831570334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c160240 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0aa00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0b6a40 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e840 tx=0x7ff9fc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa02a00640 1 -- 172.21.15.114:0/2831570334 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc017070 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa02a00640 1 -- 172.21.15.114:0/2831570334 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9fc00c8e0 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0c160b30 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa02a00640 1 -- 172.21.15.114:0/2831570334 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00ccd0 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.688+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0c1610f0 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.690+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0c04f330 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.691+0000 7ffa02a00640 1 -- 172.21.15.114:0/2831570334 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9fc01a070 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.691+0000 7ffa02a00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9e806b4d0 0x7ff9e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.691+0000 7ffa02a00640 1 -- 172.21.15.114:0/2831570334 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff9fc08a170 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.692+0000 7ffa0a000640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9e806b4d0 0x7ff9e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.692+0000 7ffa0a000640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9e806b4d0 0x7ff9e806d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4036000 tx=0x7ff9f403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.696+0000 7ffa02a00640 1 -- 172.21.15.114:0/2831570334 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9fc058f30 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.930+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa0c15d6f0 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.932+0000 7ffa02a00640 1 -- 172.21.15.114:0/2831570334 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16917 (secure 0 0 0) 0x7ff9fc058830 con 0x7ffa0c0d13a0 2024-03-06T07:33:50.936 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:50.936 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":97,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:16.974876+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:03.077157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:04.314220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6836/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6835/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6837/406262499":"2024-03-07T07:33:15.968840+0000","172.21.15.114:6834/406262499":"2024-03-07T07:33:15.968840+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9e806b4d0 msgr2=0x7ff9e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9e806b4d0 0x7ff9e806d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4036000 tx=0x7ff9f403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa0c0d13a0 msgr2=0x7ffa0c0b6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0b6a40 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00e840 tx=0x7ff9fc00ed00 comp rx=0 tx=0).stop 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 shutdown_connections 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa0c15e230 0x7ffa0c0b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffa0c0d2940 0x7ffa0c152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff9e806b4d0 0x7ff9e806d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 --2- 172.21.15.114:0/2831570334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa0c0d13a0 0x7ffa0c0b6a40 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 >> 172.21.15.114:0/2831570334 conn(0x7ffa0c0d3e20 msgr2=0x7ffa0c0d4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 shutdown_connections 2024-03-06T07:33:50.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:50.935+0000 7ffa0be00640 1 -- 172.21.15.114:0/2831570334 wait complete. 2024-03-06T07:33:50.956 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:50.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:51.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.136+0000 7fc6f6000640 1 Processor -- start 2024-03-06T07:33:51.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.136+0000 7fc6f6000640 1 -- start start 2024-03-06T07:33:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc6f01448a0 0x7fc6f00b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f6000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f00b2fb0 0x7fc6f00cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f6000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc6f015c190 0x7fc6f00cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f6000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f00cc490 con 0x7fc6f01448a0 2024-03-06T07:33:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f6000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f00b3430 con 0x7fc6f015c190 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f6000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f00cc6e0 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc6f01448a0 0x7fc6f00b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc6f01448a0 0x7fc6f00b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43114/0 (socket says 172.21.15.114:43114) 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 -- 172.21.15.114:0/3162982144 learned_addr learned my addr 172.21.15.114:0/3162982144 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 -- 172.21.15.114:0/3162982144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f00b2fb0 msgr2=0x7fc6f00cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 --2- 172.21.15.114:0/3162982144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f00b2fb0 0x7fc6f00cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 -- 172.21.15.114:0/3162982144 >> v2:172.21.15.177:3300/0 conn(0x7fc6f015c190 msgr2=0x7fc6f00cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 --2- 172.21.15.114:0/3162982144 >> v2:172.21.15.177:3300/0 conn(0x7fc6f015c190 0x7fc6f00cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.138+0000 7fc6f4c00640 1 -- 172.21.15.114:0/3162982144 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f00cc9c0 con 0x7fc6f01448a0 2024-03-06T07:33:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.139+0000 7fc6f4c00640 1 --2- 172.21.15.114:0/3162982144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f01448a0 0x7fc6f00b2a70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc6f00d0a90 tx=0x7fc6e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbf9e6a07776c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6ef400640 1 -- 172.21.15.114:0/3162982144 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4035020 con 0x7fc6f01448a0 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6ef400640 1 -- 172.21.15.114:0/3162982144 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e4031870 con 0x7fc6f01448a0 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6ef400640 1 -- 172.21.15.114:0/3162982144 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4031c40 con 0x7fc6f01448a0 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 -- 172.21.15.114:0/3162982144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f01448a0 msgr2=0x7fc6f00b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 --2- 172.21.15.114:0/3162982144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f01448a0 0x7fc6f00b2a70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc6f00d0a90 tx=0x7fc6e402b910 comp rx=0 tx=0).stop 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 -- 172.21.15.114:0/3162982144 shutdown_connections 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 --2- 172.21.15.114:0/3162982144 >> v2:172.21.15.177:3300/0 conn(0x7fc6f015c190 0x7fc6f00cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 --2- 172.21.15.114:0/3162982144 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f00b2fb0 0x7fc6f00cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 --2- 172.21.15.114:0/3162982144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f01448a0 0x7fc6f00b2a70 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 -- 172.21.15.114:0/3162982144 >> 172.21.15.114:0/3162982144 conn(0x7fc6f00b1770 msgr2=0x7fc6f00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 -- 172.21.15.114:0/3162982144 shutdown_connections 2024-03-06T07:33:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.140+0000 7fc6f6000640 1 -- 172.21.15.114:0/3162982144 wait complete. 2024-03-06T07:33:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 Processor -- start 2024-03-06T07:33:51.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 -- start start 2024-03-06T07:33:51.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f00b2fb0 0x7fc6f015aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6f01448a0 0x7fc6f013a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f015c190 0x7fc6f013a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f00cd1c0 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f015ea60 con 0x7fc6f01448a0 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.144+0000 7fc6f6000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f013c400 con 0x7fc6f015c190 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f00b2fb0 0x7fc6f015aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f00b2fb0 0x7fc6f015aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43130/0 (socket says 172.21.15.114:43130) 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 -- 172.21.15.114:0/1556968019 learned_addr learned my addr 172.21.15.114:0/1556968019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 -- 172.21.15.114:0/1556968019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f015c190 msgr2=0x7fc6f013a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f015c190 0x7fc6f013a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 -- 172.21.15.114:0/1556968019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6f01448a0 msgr2=0x7fc6f013a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6efe00640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6f01448a0 0x7fc6f013a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6f01448a0 0x7fc6f013a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 -- 172.21.15.114:0/1556968019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f013c6e0 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f4c00640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f00b2fb0 0x7fc6f015aae0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4000c00 tx=0x7fc6e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6ecc00640 1 -- 172.21.15.114:0/1556968019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e403ad60 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6ecc00640 1 -- 172.21.15.114:0/1556968019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e40443f0 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6ecc00640 1 -- 172.21.15.114:0/1556968019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40447e0 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f013c9c0 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.146+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f013cf80 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.147+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f004f330 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.148+0000 7fc6ecc00640 1 -- 172.21.15.114:0/1556968019 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6e404b060 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.149+0000 7fc6ecc00640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc6d006b4d0 0x7fc6d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.149+0000 7fc6ecc00640 1 -- 172.21.15.114:0/1556968019 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fc6e40ac530 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.149+0000 7fc6efe00640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc6d006b4d0 0x7fc6d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.150+0000 7fc6efe00640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc6d006b4d0 0x7fc6d006d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc6f015dea0 tx=0x7fc6dc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.153+0000 7fc6ecc00640 1 -- 172.21.15.114:0/1556968019 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6e403f070 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.382+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6f00cd9f0 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.382+0000 7fc6ecc00640 1 -- 172.21.15.114:0/1556968019 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7fc6e407b300 con 0x7fc6f00b2fb0 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:51.384 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:51.385 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc6d006b4d0 msgr2=0x7fc6d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc6d006b4d0 0x7fc6d006d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc6f015dea0 tx=0x7fc6dc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f00b2fb0 msgr2=0x7fc6f015aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f00b2fb0 0x7fc6f015aae0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4000c00 tx=0x7fc6e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 shutdown_connections 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6f015c190 0x7fc6f013a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6f01448a0 0x7fc6f013a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc6d006b4d0 0x7fc6d006d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 --2- 172.21.15.114:0/1556968019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6f00b2fb0 0x7fc6f015aae0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 >> 172.21.15.114:0/1556968019 conn(0x7fc6f00b1770 msgr2=0x7fc6f00ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 shutdown_connections 2024-03-06T07:33:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.385+0000 7fc6f6000640 1 -- 172.21.15.114:0/1556968019 wait complete. 2024-03-06T07:33:51.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.599+0000 7f2121800640 1 Processor -- start 2024-03-06T07:33:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.600+0000 7f2121800640 1 -- start start 2024-03-06T07:33:51.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.601+0000 7f2121800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 0x7f211c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.601+0000 7f2121800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c0d21c0 con 0x7f211c0b5020 2024-03-06T07:33:51.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.601+0000 7f2121800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c0b48d0 con 0x7f211c0d16a0 2024-03-06T07:33:51.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.601+0000 7f2121800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0d0000 con 0x7f211c0b44f0 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.601+0000 7f2112000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f211c0b5020 0x7f211c0c7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38794/0 (socket says 172.21.15.114:38794) 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.601+0000 7f2112000640 1 -- 172.21.15.114:0/638478613 learned_addr learned my addr 172.21.15.114:0/638478613 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211aa00640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 0x7f211c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237941698 0 0) 0x7f211c0d21c0 con 0x7f211c0b5020 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2100003520 con 0x7f211c0b5020 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266469311 0 0) 0x7f2100003520 con 0x7f211c0b5020 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 msgr2=0x7f211c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 0x7f211c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 >> v1:172.21.15.177:6789/0 conn(0x7f211c0d16a0 legacy=0x7f211c0c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c0d1ee0 con 0x7f211c0b5020 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2104001670 con 0x7f211c0b5020 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.602+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 >> v1:172.21.15.114:6789/0 conn(0x7f211c0b5020 legacy=0x7f211c0c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f211a000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 0x7f21000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f211a000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2100003f60 0x7f2100004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f211a000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2100004d40 0x7f21000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0d0000 con 0x7f2100003f60 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210000bb50 con 0x7f2100004d40 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210000bd50 con 0x7f211c0b44f0 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f2112000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2100003f60 0x7f2100004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f211aa00640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 0x7f21000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f2112000640 1 -- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 msgr2=0x7f21000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f2112000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f211c0b44f0 0x7f21000039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f2112000640 1 -- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2100004d40 msgr2=0x7f21000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f2112000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2100004d40 0x7f21000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.603+0000 7f2112000640 1 -- 172.21.15.114:0/638478613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2100003520 con 0x7f2100003f60 2024-03-06T07:33:51.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.604+0000 7f2112000640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2100003f60 0x7f2100004800 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f210400c240 tx=0x7f2100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28771ea0a71ba8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2104007d10 con 0x7f2100003f60 2024-03-06T07:33:51.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2104030070 con 0x7f2100003f60 2024-03-06T07:33:51.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f211a000640 1 -- 172.21.15.114:0/638478613 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210402cd60 con 0x7f2100003f60 2024-03-06T07:33:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f2121800640 1 -- 172.21.15.114:0/638478613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2100003f60 msgr2=0x7f2100004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f2121800640 1 --2- 172.21.15.114:0/638478613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2100003f60 0x7f2100004800 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f210400c240 tx=0x7f2100005830 comp rx=0 tx=0).stop 2024-03-06T07:33:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f211b400640 1 -- 172.21.15.114:0/638478613 reap_dead start 2024-03-06T07:33:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f2121800640 1 -- 172.21.15.114:0/638478613 shutdown_connections 2024-03-06T07:33:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f2121800640 1 -- 172.21.15.114:0/638478613 >> 172.21.15.114:0/638478613 conn(0x7f211c0d3fd0 msgr2=0x7f211c0d43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f2121800640 1 -- 172.21.15.114:0/638478613 shutdown_connections 2024-03-06T07:33:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.605+0000 7f2121800640 1 -- 172.21.15.114:0/638478613 wait complete. 2024-03-06T07:33:51.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.608+0000 7f2121800640 1 Processor -- start 2024-03-06T07:33:51.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.608+0000 7f2121800640 1 -- start start 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f2121800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2100003f60 0x7f211c0cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f2121800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f211c0b31a0 0x7f211c0cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f2121800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f211c0b3600 0x7f211c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f2121800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0cfae0 con 0x7f211c0b3600 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f2121800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0b4e40 con 0x7f211c0b31a0 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f2121800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0b4ad0 con 0x7f2100003f60 2024-03-06T07:33:51.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f211b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f211c0b3600 0x7f211c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f211b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f211c0b3600 0x7f211c0cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43148/0 (socket says 172.21.15.114:43148) 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.610+0000 7f211b400640 1 -- 172.21.15.114:0/160029050 learned_addr learned my addr 172.21.15.114:0/160029050 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f2112000640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f211c0b31a0 0x7f211c0cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f211aa00640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2100003f60 0x7f211c0cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f211b400640 1 -- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2100003f60 msgr2=0x7f211c0cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f211b400640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2100003f60 0x7f211c0cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f211b400640 1 -- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f211c0b31a0 msgr2=0x7f211c0cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f211b400640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f211c0b31a0 0x7f211c0cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f211b400640 1 -- 172.21.15.114:0/160029050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c150510 con 0x7f211c0b3600 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f211b400640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f211c0b3600 0x7f211c0cb5e0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f211c0d1ad0 tx=0x7f211400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f2113400640 1 -- 172.21.15.114:0/160029050 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114007280 con 0x7f211c0b3600 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f2113400640 1 -- 172.21.15.114:0/160029050 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2114007420 con 0x7f211c0b3600 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f2113400640 1 -- 172.21.15.114:0/160029050 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114004020 con 0x7f211c0b3600 2024-03-06T07:33:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.611+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c1507f0 con 0x7f211c0b3600 2024-03-06T07:33:51.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.612+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c0cbb20 con 0x7f211c0b3600 2024-03-06T07:33:51.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.613+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c150d10 con 0x7f211c0b3600 2024-03-06T07:33:51.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.614+0000 7f2113400640 1 -- 172.21.15.114:0/160029050 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2114011070 con 0x7f211c0b3600 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.615+0000 7f2113400640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f20f806b4d0 0x7f20f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.615+0000 7f2113400640 1 -- 172.21.15.114:0/160029050 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f2114089090 con 0x7f211c0b3600 2024-03-06T07:33:51.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.615+0000 7f211aa00640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f20f806b4d0 0x7f20f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.616+0000 7f211aa00640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f20f806b4d0 0x7f20f806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f210c0074a0 tx=0x7f210c00d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.619+0000 7f2113400640 1 -- 172.21.15.114:0/160029050 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f211c150d10 con 0x7f211c0b3600 2024-03-06T07:33:51.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.890+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f211c068070 con 0x7f211c0b3600 2024-03-06T07:33:51.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.891+0000 7f2113400640 1 -- 172.21.15.114:0/160029050 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f211401a030 con 0x7f211c0b3600 2024-03-06T07:33:51.893 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:51.893 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:05.301366+0000","modified":"2024-03-06T07:33:21.025972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7347":{"gid":7347,"name":"a","rank":0,"incarnation":66,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6835/1490119219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1490119219},{"type":"v1","addr":"172.21.15.114:6835","nonce":1490119219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7347,"qdb_cluster":[7347]},"id":7}]} 2024-03-06T07:33:51.893 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 70 2024-03-06T07:33:51.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f20f806b4d0 msgr2=0x7f20f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f20f806b4d0 0x7f20f806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f210c0074a0 tx=0x7f210c00d040 comp rx=0 tx=0).stop 2024-03-06T07:33:51.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f211c0b3600 msgr2=0x7f211c0cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f211c0b3600 0x7f211c0cb5e0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f211c0d1ad0 tx=0x7f211400c820 comp rx=0 tx=0).stop 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 shutdown_connections 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f20f806b4d0 0x7f20f806d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f211c0b3600 0x7f211c0cb5e0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f211c0b31a0 0x7f211c0cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 --2- 172.21.15.114:0/160029050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2100003f60 0x7f211c0cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.893+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 >> 172.21.15.114:0/160029050 conn(0x7f211c0d3fd0 msgr2=0x7f211c06d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.894+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 shutdown_connections 2024-03-06T07:33:51.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:51.894+0000 7f2121800640 1 -- 172.21.15.114:0/160029050 wait complete. 2024-03-06T07:33:51.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:52.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.116+0000 7f9201800640 1 Processor -- start 2024-03-06T07:33:52.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.116+0000 7f9201800640 1 -- start start 2024-03-06T07:33:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.119+0000 7f9201800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f91fc15be90 0x7f91fc0b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.119+0000 7f9201800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0d0780 0x7f91fc15fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.119+0000 7f9201800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91fc0b5160 con 0x7f91fc0acd60 2024-03-06T07:33:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.119+0000 7f9201800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91fc0b4f30 con 0x7f91fc15be90 2024-03-06T07:33:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.119+0000 7f9201800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0af240 con 0x7f91fc0d0780 2024-03-06T07:33:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.119+0000 7f91fa000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f91fc0acd60 0x7f91fc0d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38796/0 (socket says 172.21.15.114:38796) 2024-03-06T07:33:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.119+0000 7f91fa000640 1 -- 172.21.15.114:0/2542221986 learned_addr learned my addr 172.21.15.114:0/2542221986 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611650219 0 0) 0x7f91fc0b5160 con 0x7f91fc0acd60 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91faa00640 1 --2- 172.21.15.114:0/2542221986 >> v2:172.21.15.177:3300/0 conn(0x7f91fc15be90 0x7f91fc0b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91fb400640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0d0780 0x7f91fc15fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e0003520 con 0x7f91fc0acd60 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4267240822 0 0) 0x7f91e0003520 con 0x7f91fc0acd60 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0d0780 msgr2=0x7f91fc15fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0d0780 0x7f91fc15fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 >> v2:172.21.15.177:3300/0 conn(0x7f91fc15be90 msgr2=0x7f91fc0b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 --2- 172.21.15.114:0/2542221986 >> v2:172.21.15.177:3300/0 conn(0x7f91fc15be90 0x7f91fc0b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc0ae910 con 0x7f91fc0acd60 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f91e8001670 con 0x7f91fc0acd60 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 >> v1:172.21.15.114:6789/0 conn(0x7f91fc0acd60 legacy=0x7f91fc0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91e0004800 0x7f91e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00051c0 0x7f91e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0af240 con 0x7f91e00039b0 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e000bfe0 con 0x7f91e00051c0 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e000c1e0 con 0x7f91e0004800 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91fb400640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0d0780 0x7f91fc15fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.120+0000 7f91faa00640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.121+0000 7f91fa000640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91e0004800 0x7f91e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.121+0000 7f91faa00640 1 -- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91e0004800 msgr2=0x7f91e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.121+0000 7f91faa00640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91e0004800 0x7f91e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.121+0000 7f91faa00640 1 -- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00051c0 msgr2=0x7f91e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.121+0000 7f91faa00640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00051c0 0x7f91e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.121+0000 7f91faa00640 1 -- 172.21.15.114:0/2542221986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0003520 con 0x7f91e00039b0 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.123+0000 7f91faa00640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00ba30 tx=0x7f91e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519cac824391ff89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.123+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec036020 con 0x7f91e00039b0 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.123+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ec032680 con 0x7f91e00039b0 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.123+0000 7f91f9600640 1 -- 172.21.15.114:0/2542221986 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec00b600 con 0x7f91e00039b0 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.124+0000 7f9201800640 1 -- 172.21.15.114:0/2542221986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91e00039b0 msgr2=0x7f91e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.124+0000 7f9201800640 1 --2- 172.21.15.114:0/2542221986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00ba30 tx=0x7f91e0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.124+0000 7f91fb400640 1 -- 172.21.15.114:0/2542221986 reap_dead start 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.124+0000 7f9201800640 1 -- 172.21.15.114:0/2542221986 shutdown_connections 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.124+0000 7f9201800640 1 -- 172.21.15.114:0/2542221986 >> 172.21.15.114:0/2542221986 conn(0x7f91fc06b360 msgr2=0x7f91fc06b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.124+0000 7f9201800640 1 -- 172.21.15.114:0/2542221986 shutdown_connections 2024-03-06T07:33:52.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.124+0000 7f9201800640 1 -- 172.21.15.114:0/2542221986 wait complete. 2024-03-06T07:33:52.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.125+0000 7f9201800640 1 Processor -- start 2024-03-06T07:33:52.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.126+0000 7f9201800640 1 -- start start 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.126+0000 7f9201800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00039b0 0x7f91fc072600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f9201800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0c9a10 0x7f91fc0c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f9201800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91fc1590c0 0x7f91fc159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f9201800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0b2c80 con 0x7f91fc1590c0 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f9201800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0aeed0 con 0x7f91e00039b0 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f9201800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0b2e00 con 0x7f91fc0c9a10 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91fc1590c0 0x7f91fc159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91fc1590c0 0x7f91fc159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43174/0 (socket says 172.21.15.114:43174) 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 -- 172.21.15.114:0/1194025149 learned_addr learned my addr 172.21.15.114:0/1194025149 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91faa00640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00039b0 0x7f91fc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fa000640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0c9a10 0x7f91fc0c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 -- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0c9a10 msgr2=0x7f91fc0c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0c9a10 0x7f91fc0c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 -- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00039b0 msgr2=0x7f91fc072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00039b0 0x7f91fc072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.127+0000 7f91fb400640 1 -- 172.21.15.114:0/1194025149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc15a9e0 con 0x7f91fc1590c0 2024-03-06T07:33:52.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.128+0000 7f91fb400640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91fc1590c0 0x7f91fc159540 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f91fc0b4670 tx=0x7f91f400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.128+0000 7f91f2a00640 1 -- 172.21.15.114:0/1194025149 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f401a070 con 0x7f91fc1590c0 2024-03-06T07:33:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.128+0000 7f91f2a00640 1 -- 172.21.15.114:0/1194025149 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f4007e20 con 0x7f91fc1590c0 2024-03-06T07:33:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.128+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91fc071570 con 0x7f91fc1590c0 2024-03-06T07:33:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.128+0000 7f91f2a00640 1 -- 172.21.15.114:0/1194025149 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f4003bd0 con 0x7f91fc1590c0 2024-03-06T07:33:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.128+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91fc071b20 con 0x7f91fc1590c0 2024-03-06T07:33:52.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.130+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91fc0687e0 con 0x7f91fc1590c0 2024-03-06T07:33:52.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.131+0000 7f91f2a00640 1 -- 172.21.15.114:0/1194025149 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91f401d070 con 0x7f91fc1590c0 2024-03-06T07:33:52.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.132+0000 7f91f2a00640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91dc06b4d0 0x7f91dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.132+0000 7f91f2a00640 1 -- 172.21.15.114:0/1194025149 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f91f408a030 con 0x7f91fc1590c0 2024-03-06T07:33:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.132+0000 7f91faa00640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91dc06b4d0 0x7f91dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.133+0000 7f91faa00640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91dc06b4d0 0x7f91dc06d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00ba00 tx=0x7f91ec03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.137+0000 7f91f2a00640 1 -- 172.21.15.114:0/1194025149 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91fc0687e0 con 0x7f91fc1590c0 2024-03-06T07:33:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.335+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91fc068070 con 0x7f91fc1590c0 2024-03-06T07:33:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.338+0000 7f91f2a00640 1 -- 172.21.15.114:0/1194025149 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16917 (secure 0 0 0) 0x7f91f4010070 con 0x7f91fc1590c0 2024-03-06T07:33:52.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:52.340 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":97,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:16.974876+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:03.077157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:04.314220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6836/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6835/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6837/406262499":"2024-03-07T07:33:15.968840+0000","172.21.15.114:6834/406262499":"2024-03-07T07:33:15.968840+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91dc06b4d0 msgr2=0x7f91dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91dc06b4d0 0x7f91dc06d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00ba00 tx=0x7f91ec03d040 comp rx=0 tx=0).stop 2024-03-06T07:33:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91fc1590c0 msgr2=0x7f91fc159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91fc1590c0 0x7f91fc159540 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f91fc0b4670 tx=0x7f91f400d4a0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 shutdown_connections 2024-03-06T07:33:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f91dc06b4d0 0x7f91dc06d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91fc1590c0 0x7f91fc159540 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f91fc0c9a10 0x7f91fc0c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 --2- 172.21.15.114:0/1194025149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91e00039b0 0x7f91fc072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.341+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 >> 172.21.15.114:0/1194025149 conn(0x7f91fc06b360 msgr2=0x7f91fc0af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.342+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 shutdown_connections 2024-03-06T07:33:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.342+0000 7f9201800640 1 -- 172.21.15.114:0/1194025149 wait complete. 2024-03-06T07:33:52.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:33:52.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.562+0000 7f018ce00640 1 Processor -- start 2024-03-06T07:33:52.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.562+0000 7f018ce00640 1 -- start start 2024-03-06T07:33:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.563+0000 7f018ce00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f018806c9f0 0x7f0188070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.563+0000 7f018ce00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f01880b3a50 0x7f01880acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.563+0000 7f018ce00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018815a9a0 con 0x7f01880ad3e0 2024-03-06T07:33:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f018ce00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0188071370 con 0x7f018806c9f0 2024-03-06T07:33:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f018ce00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0188071650 con 0x7f01880b3a50 2024-03-06T07:33:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0186800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f01880ad3e0 0x7f018806cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38798/0 (socket says 172.21.15.114:38798) 2024-03-06T07:33:52.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0185e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f018806c9f0 0x7f0188070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0186800640 1 -- 172.21.15.114:0/2392158607 learned_addr learned my addr 172.21.15.114:0/2392158607 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0185400640 1 --2- 172.21.15.114:0/2392158607 >> v2:172.21.15.177:3300/0 conn(0x7f01880b3a50 0x7f01880acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0184a00640 1 -- 172.21.15.114:0/2392158607 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193678501 0 0) 0x7f018815a9a0 con 0x7f01880ad3e0 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0184a00640 1 -- 172.21.15.114:0/2392158607 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f016c003520 con 0x7f01880ad3e0 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0185e00640 1 -- 172.21.15.114:0/2392158607 >> v2:172.21.15.177:3300/0 conn(0x7f01880b3a50 msgr2=0x7f01880acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0185e00640 1 --2- 172.21.15.114:0/2392158607 >> v2:172.21.15.177:3300/0 conn(0x7f01880b3a50 0x7f01880acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.564+0000 7f0185e00640 1 -- 172.21.15.114:0/2392158607 >> v1:172.21.15.114:6789/0 conn(0x7f01880ad3e0 legacy=0x7f018806cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.565+0000 7f0185e00640 1 -- 172.21.15.114:0/2392158607 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188071930 con 0x7f018806c9f0 2024-03-06T07:33:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.565+0000 7f0185400640 1 --2- 172.21.15.114:0/2392158607 >> v2:172.21.15.177:3300/0 conn(0x7f01880b3a50 0x7f01880acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:52.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.566+0000 7f0185e00640 1 --2- 172.21.15.114:0/2392158607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018806c9f0 0x7f0188070ce0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f018809e080 tx=0x7f017802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1203e2953244970 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f0184a00640 1 -- 172.21.15.114:0/2392158607 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0178035020 con 0x7f018806c9f0 2024-03-06T07:33:52.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f0184a00640 1 -- 172.21.15.114:0/2392158607 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0178031830 con 0x7f018806c9f0 2024-03-06T07:33:52.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f0184a00640 1 -- 172.21.15.114:0/2392158607 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0178031c00 con 0x7f018806c9f0 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f018ce00640 1 -- 172.21.15.114:0/2392158607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018806c9f0 msgr2=0x7f0188070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f018ce00640 1 --2- 172.21.15.114:0/2392158607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018806c9f0 0x7f0188070ce0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f018809e080 tx=0x7f017802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f018ce00640 1 -- 172.21.15.114:0/2392158607 shutdown_connections 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f018ce00640 1 --2- 172.21.15.114:0/2392158607 >> v2:172.21.15.177:3300/0 conn(0x7f01880b3a50 0x7f01880acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f018ce00640 1 --2- 172.21.15.114:0/2392158607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018806c9f0 0x7f0188070ce0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f018ce00640 1 -- 172.21.15.114:0/2392158607 >> 172.21.15.114:0/2392158607 conn(0x7f018806b360 msgr2=0x7f018806b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.567+0000 7f018ce00640 1 -- 172.21.15.114:0/2392158607 shutdown_connections 2024-03-06T07:33:52.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.568+0000 7f018ce00640 1 -- 172.21.15.114:0/2392158607 wait complete. 2024-03-06T07:33:52.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.570+0000 7f018ce00640 1 Processor -- start 2024-03-06T07:33:52.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.570+0000 7f018ce00640 1 -- start start 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f018ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f018806c9f0 0x7f0188165120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f018ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01880ad3e0 0x7f0188147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f018ce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01880b3a50 0x7f0188162210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f018ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01880b3e60 con 0x7f01880ad3e0 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f018ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0188071ef0 con 0x7f018806c9f0 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f018ce00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01880b3fe0 con 0x7f01880b3a50 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f0185400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01880ad3e0 0x7f0188147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f0185400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01880ad3e0 0x7f0188147e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43188/0 (socket says 172.21.15.114:43188) 2024-03-06T07:33:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f0185400640 1 -- 172.21.15.114:0/870551995 learned_addr learned my addr 172.21.15.114:0/870551995 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f0185e00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f018806c9f0 0x7f0188165120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.572+0000 7f0186800640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01880b3a50 0x7f0188162210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f0185400640 1 -- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01880b3a50 msgr2=0x7f0188162210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f0185400640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01880b3a50 0x7f0188162210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f0185400640 1 -- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f018806c9f0 msgr2=0x7f0188165120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f0185400640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f018806c9f0 0x7f0188165120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f0185400640 1 -- 172.21.15.114:0/870551995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188165660 con 0x7f01880ad3e0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f0185400640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01880ad3e0 0x7f0188147e00 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f017400c7f0 tx=0x7f017400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f017e000640 1 -- 172.21.15.114:0/870551995 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174015070 con 0x7f01880ad3e0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f017e000640 1 -- 172.21.15.114:0/870551995 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01740077c0 con 0x7f01880ad3e0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f017e000640 1 -- 172.21.15.114:0/870551995 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174007b70 con 0x7f01880ad3e0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0188165910 con 0x7f01880ad3e0 2024-03-06T07:33:52.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.573+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0188165ed0 con 0x7f01880ad3e0 2024-03-06T07:33:52.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.574+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0188068070 con 0x7f01880ad3e0 2024-03-06T07:33:52.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.575+0000 7f017e000640 1 -- 172.21.15.114:0/870551995 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0174018070 con 0x7f01880ad3e0 2024-03-06T07:33:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.576+0000 7f017e000640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f016806b4d0 0x7f016806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.576+0000 7f017e000640 1 -- 172.21.15.114:0/870551995 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f017408a370 con 0x7f01880ad3e0 2024-03-06T07:33:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.576+0000 7f0185e00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f016806b4d0 0x7f016806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.577+0000 7f0185e00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f016806b4d0 0x7f016806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0178036000 tx=0x7f017803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.581+0000 7f017e000640 1 -- 172.21.15.114:0/870551995 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0174059130 con 0x7f01880ad3e0 2024-03-06T07:33:52.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.815+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f01880b27f0 con 0x7f01880ad3e0 2024-03-06T07:33:52.818 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:33:52.816+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:52.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.820+0000 7f017e000640 1 -- 172.21.15.114:0/870551995 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0174058a30 con 0x7f01880ad3e0 2024-03-06T07:33:52.822 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:33:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f016806b4d0 msgr2=0x7f016806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f016806b4d0 0x7f016806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0178036000 tx=0x7f017803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01880ad3e0 msgr2=0x7f0188147e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01880ad3e0 0x7f0188147e00 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f017400c7f0 tx=0x7f017400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 shutdown_connections 2024-03-06T07:33:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f01880b3a50 0x7f0188162210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f016806b4d0 0x7f016806d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01880ad3e0 0x7f0188147e00 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 --2- 172.21.15.114:0/870551995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f018806c9f0 0x7f0188165120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.823+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 >> 172.21.15.114:0/870551995 conn(0x7f018806b360 msgr2=0x7f0188072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.824+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 shutdown_connections 2024-03-06T07:33:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:52.824+0000 7f018ce00640 1 -- 172.21.15.114:0/870551995 wait complete. 2024-03-06T07:33:52.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:33:52.919 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:33:53.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.028+0000 7f5d7a000640 1 Processor -- start 2024-03-06T07:33:53.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.028+0000 7f5d7a000640 1 -- start start 2024-03-06T07:33:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.029+0000 7f5d7a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 0x7f5d740ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.029+0000 7f5d7a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5d7415f320 0x7f5d740d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.029+0000 7f5d7a000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d74155870 con 0x7f5d740d0070 2024-03-06T07:33:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d7a000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d740ad640 con 0x7f5d7415f320 2024-03-06T07:33:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d7a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7415f980 con 0x7f5d74155490 2024-03-06T07:33:53.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d6aa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5d740d0070 0x7f5d7415ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38802/0 (socket says 172.21.15.114:38802) 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d6aa00640 1 -- 172.21.15.114:0/298533861 learned_addr learned my addr 172.21.15.114:0/298533861 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364564979 0 0) 0x7f5d74155870 con 0x7f5d740d0070 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d73e00640 1 --2- 172.21.15.114:0/298533861 >> v2:172.21.15.114:3300/0 conn(0x7f5d7415f320 0x7f5d740d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d58005730 con 0x7f5d740d0070 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d73400640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 0x7f5d740ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1835452659 0 0) 0x7f5d58005730 con 0x7f5d740d0070 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 msgr2=0x7f5d740ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d72a00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 0x7f5d740ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 >> v2:172.21.15.114:3300/0 conn(0x7f5d7415f320 msgr2=0x7f5d740d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.030+0000 7f5d72a00640 1 --2- 172.21.15.114:0/298533861 >> v2:172.21.15.114:3300/0 conn(0x7f5d7415f320 0x7f5d740d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d740ad870 con 0x7f5d740d0070 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5d5c001670 con 0x7f5d740d0070 2024-03-06T07:33:53.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 >> v1:172.21.15.114:6789/0 conn(0x7f5d740d0070 legacy=0x7f5d7415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 0x7f5d58005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d58006140 0x7f5d58006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006f50 0x7f5d580073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7415f980 con 0x7f5d58006140 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5800dbd0 con 0x7f5d58006f50 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5800dda0 con 0x7f5d74155490 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d6aa00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d58006140 0x7f5d58006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d73400640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 0x7f5d58005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d6aa00640 1 -- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 msgr2=0x7f5d58005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d6aa00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d74155490 0x7f5d58005bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d73e00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006f50 0x7f5d580073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d6aa00640 1 -- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006f50 msgr2=0x7f5d580073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d6aa00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006f50 0x7f5d580073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.031+0000 7f5d6aa00640 1 -- 172.21.15.114:0/298533861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d58005730 con 0x7f5d58006140 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.032+0000 7f5d6aa00640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d58006140 0x7f5d58006a10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c240 tx=0x7f5d58007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105ad340b45b6492 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c007d30 con 0x7f5d58006140 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c030070 con 0x7f5d58006140 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d72a00640 1 -- 172.21.15.114:0/298533861 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c02cda0 con 0x7f5d58006140 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d7a000640 1 -- 172.21.15.114:0/298533861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d58006140 msgr2=0x7f5d58006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d7a000640 1 --2- 172.21.15.114:0/298533861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d58006140 0x7f5d58006a10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00c240 tx=0x7f5d58007a40 comp rx=0 tx=0).stop 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d73e00640 1 -- 172.21.15.114:0/298533861 reap_dead start 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d7a000640 1 -- 172.21.15.114:0/298533861 shutdown_connections 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d7a000640 1 -- 172.21.15.114:0/298533861 >> 172.21.15.114:0/298533861 conn(0x7f5d7406b360 msgr2=0x7f5d7406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d7a000640 1 -- 172.21.15.114:0/298533861 shutdown_connections 2024-03-06T07:33:53.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.033+0000 7f5d7a000640 1 -- 172.21.15.114:0/298533861 wait complete. 2024-03-06T07:33:53.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.035+0000 7f5d7a000640 1 Processor -- start 2024-03-06T07:33:53.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.035+0000 7f5d7a000640 1 -- start start 2024-03-06T07:33:53.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d7a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006140 0x7f5d740b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d7a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d74157f80 0x7f5d741f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d7a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7409b940 0x7f5d7409bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d7a000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740ae110 con 0x7f5d74157f80 2024-03-06T07:33:53.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d7a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740ade30 con 0x7f5d58006140 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d7a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740ae630 con 0x7f5d7409b940 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d6aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d74157f80 0x7f5d741f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d6aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d74157f80 0x7f5d741f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43230/0 (socket says 172.21.15.114:43230) 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.036+0000 7f5d6aa00640 1 -- 172.21.15.114:0/916082952 learned_addr learned my addr 172.21.15.114:0/916082952 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6aa00640 1 -- 172.21.15.114:0/916082952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7409b940 msgr2=0x7f5d7409bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6aa00640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7409b940 0x7f5d7409bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6aa00640 1 -- 172.21.15.114:0/916082952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006140 msgr2=0x7f5d740b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6aa00640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006140 0x7f5d740b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6aa00640 1 -- 172.21.15.114:0/916082952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d740af6f0 con 0x7f5d74157f80 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6aa00640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d74157f80 0x7f5d741f3b30 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c02c570 tx=0x7f5d74155a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6be00640 1 -- 172.21.15.114:0/916082952 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c001c70 con 0x7f5d74157f80 2024-03-06T07:33:53.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6be00640 1 -- 172.21.15.114:0/916082952 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c004bf0 con 0x7f5d74157f80 2024-03-06T07:33:53.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d6be00640 1 -- 172.21.15.114:0/916082952 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c007070 con 0x7f5d74157f80 2024-03-06T07:33:53.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d741f4420 con 0x7f5d74157f80 2024-03-06T07:33:53.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.037+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d740af960 con 0x7f5d74157f80 2024-03-06T07:33:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.038+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d74068070 con 0x7f5d74157f80 2024-03-06T07:33:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.039+0000 7f5d6be00640 1 -- 172.21.15.114:0/916082952 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d5c047020 con 0x7f5d74157f80 2024-03-06T07:33:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.040+0000 7f5d6be00640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5006b4d0 0x7f5d5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.040+0000 7f5d6be00640 1 -- 172.21.15.114:0/916082952 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5d5c0ab400 con 0x7f5d74157f80 2024-03-06T07:33:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.040+0000 7f5d73400640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5006b4d0 0x7f5d5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.041+0000 7f5d73400640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5006b4d0 0x7f5d5006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5d640070b0 tx=0x7f5d6400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.044+0000 7f5d6be00640 1 -- 172.21.15.114:0/916082952 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d5c07a170 con 0x7f5d74157f80 2024-03-06T07:33:53.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.283+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5d74071740 con 0x7f5d74157f80 2024-03-06T07:33:53.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.823+0000 7f5d6be00640 1 -- 172.21.15.114:0/916082952 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5d5c079a70 con 0x7f5d74157f80 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5006b4d0 msgr2=0x7f5d5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5006b4d0 0x7f5d5006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5d640070b0 tx=0x7f5d6400b040 comp rx=0 tx=0).stop 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d74157f80 msgr2=0x7f5d741f3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d74157f80 0x7f5d741f3b30 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c02c570 tx=0x7f5d74155a40 comp rx=0 tx=0).stop 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 shutdown_connections 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7409b940 0x7f5d7409bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5006b4d0 0x7f5d5006d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d74157f80 0x7f5d741f3b30 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 --2- 172.21.15.114:0/916082952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d58006140 0x7f5d740b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 >> 172.21.15.114:0/916082952 conn(0x7f5d7406b360 msgr2=0x7f5d7415b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 shutdown_connections 2024-03-06T07:33:53.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:53.827+0000 7f5d7a000640 1 -- 172.21.15.114:0/916082952 wait complete. 2024-03-06T07:33:53.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:33:54.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.053+0000 7f7dbb000640 1 Processor -- start 2024-03-06T07:33:54.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.053+0000 7f7dbb000640 1 -- start start 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7dbb000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7db4156610 0x7f7db40b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7dbb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40acd60 0x7f7db40d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7dbb000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db41569f0 con 0x7f7db40af2f0 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7dbb000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db415c850 con 0x7f7db4156610 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7dbb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db415cb30 con 0x7f7db40acd60 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7db4156610 0x7f7db40b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db9000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7db40af2f0 0x7f7db406cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38808/0 (socket says 172.21.15.114:38808) 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7db4156610 0x7f7db40b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43234/0 (socket says 172.21.15.114:43234) 2024-03-06T07:33:54.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db9000640 1 -- 172.21.15.114:0/1190737726 learned_addr learned my addr 172.21.15.114:0/1190737726 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:54.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db3e00640 1 -- 172.21.15.114:0/1190737726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40acd60 msgr2=0x7f7db40d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db3e00640 1 --2- 172.21.15.114:0/1190737726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40acd60 0x7f7db40d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db3e00640 1 -- 172.21.15.114:0/1190737726 >> v1:172.21.15.114:6789/0 conn(0x7f7db40af2f0 legacy=0x7f7db406cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.055+0000 7f7db3e00640 1 -- 172.21.15.114:0/1190737726 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db415ce10 con 0x7f7db4156610 2024-03-06T07:33:54.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.057+0000 7f7db3e00640 1 --2- 172.21.15.114:0/1190737726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db40b3ec0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f7da40097a0 tx=0x7f7da402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8255ff6fae96c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.057+0000 7f7db2a00640 1 -- 172.21.15.114:0/1190737726 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4035020 con 0x7f7db4156610 2024-03-06T07:33:54.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.057+0000 7f7db2a00640 1 -- 172.21.15.114:0/1190737726 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da4031830 con 0x7f7db4156610 2024-03-06T07:33:54.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.057+0000 7f7db2a00640 1 -- 172.21.15.114:0/1190737726 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4031c00 con 0x7f7db4156610 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 -- 172.21.15.114:0/1190737726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 msgr2=0x7f7db40b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1190737726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db40b3ec0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f7da40097a0 tx=0x7f7da402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 -- 172.21.15.114:0/1190737726 shutdown_connections 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1190737726 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40acd60 0x7f7db40d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1190737726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db40b3ec0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 -- 172.21.15.114:0/1190737726 >> 172.21.15.114:0/1190737726 conn(0x7f7db406b360 msgr2=0x7f7db406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 -- 172.21.15.114:0/1190737726 shutdown_connections 2024-03-06T07:33:54.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.058+0000 7f7dbb000640 1 -- 172.21.15.114:0/1190737726 wait complete. 2024-03-06T07:33:54.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.059+0000 7f7dbb000640 1 Processor -- start 2024-03-06T07:33:54.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.060+0000 7f7dbb000640 1 -- start start 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7dbb000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db40acd60 0x7f7db4159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7dbb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40af2f0 0x7f7db40af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7dbb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db413fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7dbb000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db415c540 con 0x7f7db4156610 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7dbb000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db415d3d0 con 0x7f7db40acd60 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7dbb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db40d05b0 con 0x7f7db40af2f0 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db413fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db413fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43242/0 (socket says 172.21.15.114:43242) 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 -- 172.21.15.114:0/1128581433 learned_addr learned my addr 172.21.15.114:0/1128581433 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db3e00640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db40acd60 0x7f7db4159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db3400640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40af2f0 0x7f7db40af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 -- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40af2f0 msgr2=0x7f7db40af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40af2f0 0x7f7db40af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 -- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db40acd60 msgr2=0x7f7db4159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db40acd60 0x7f7db4159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db9000640 1 -- 172.21.15.114:0/1128581433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db40afc80 con 0x7f7db4156610 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.061+0000 7f7db3400640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40af2f0 0x7f7db40af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.062+0000 7f7db9000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db413fff0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c450 tx=0x7f7dac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.062+0000 7f7dabe00640 1 -- 172.21.15.114:0/1128581433 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac017070 con 0x7f7db4156610 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.062+0000 7f7dabe00640 1 -- 172.21.15.114:0/1128581433 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac005ce0 con 0x7f7db4156610 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.062+0000 7f7dabe00640 1 -- 172.21.15.114:0/1128581433 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac007270 con 0x7f7db4156610 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.062+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db4140530 con 0x7f7db4156610 2024-03-06T07:33:54.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.062+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db40b22c0 con 0x7f7db4156610 2024-03-06T07:33:54.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.064+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db409b830 con 0x7f7db4156610 2024-03-06T07:33:54.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.064+0000 7f7dabe00640 1 -- 172.21.15.114:0/1128581433 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dac01a070 con 0x7f7db4156610 2024-03-06T07:33:54.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.065+0000 7f7dabe00640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9406b4d0 0x7f7d9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.065+0000 7f7dabe00640 1 -- 172.21.15.114:0/1128581433 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7dac0894d0 con 0x7f7db4156610 2024-03-06T07:33:54.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.065+0000 7f7db3e00640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9406b4d0 0x7f7d9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.066+0000 7f7db3e00640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9406b4d0 0x7f7d9406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7da4036000 tx=0x7f7da403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.070+0000 7f7dabe00640 1 -- 172.21.15.114:0/1128581433 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dac057b60 con 0x7f7db4156610 2024-03-06T07:33:54.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.267+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7db415f850 con 0x7f7db4156610 2024-03-06T07:33:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.836+0000 7f7dabe00640 1 -- 172.21.15.114:0/1128581433 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7dac05bc00 con 0x7f7db4156610 2024-03-06T07:33:54.838 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.841+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9406b4d0 msgr2=0x7f7d9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.841+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9406b4d0 0x7f7d9406d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7da4036000 tx=0x7f7da403c040 comp rx=0 tx=0).stop 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.841+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 msgr2=0x7f7db413fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.841+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db413fff0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c450 tx=0x7f7dac00c910 comp rx=0 tx=0).stop 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.842+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 shutdown_connections 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.842+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7d9406b4d0 0x7f7d9406d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.842+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7db4156610 0x7f7db413fff0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.842+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7db40af2f0 0x7f7db40af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.842+0000 7f7dbb000640 1 --2- 172.21.15.114:0/1128581433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7db40acd60 0x7f7db4159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.842+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 >> 172.21.15.114:0/1128581433 conn(0x7f7db406b360 msgr2=0x7f7db415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.842+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 shutdown_connections 2024-03-06T07:33:54.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:54.843+0000 7f7dbb000640 1 -- 172.21.15.114:0/1128581433 wait complete. 2024-03-06T07:33:54.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:33:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.071+0000 7f3e2ae00640 1 Processor -- start 2024-03-06T07:33:55.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.072+0000 7f3e2ae00640 1 -- start start 2024-03-06T07:33:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.073+0000 7f3e2ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 0x7f3e240ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.073+0000 7f3e2ae00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e24155870 con 0x7f3e2415ec10 2024-03-06T07:33:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.073+0000 7f3e2ae00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e240ad870 con 0x7f3e24155490 2024-03-06T07:33:55.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.073+0000 7f3e2ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240ad640 con 0x7f3e240d0780 2024-03-06T07:33:55.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.074+0000 7f3e28e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f3e2415ec10 0x7f3e2415f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38816/0 (socket says 172.21.15.114:38816) 2024-03-06T07:33:55.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.074+0000 7f3e28e00640 1 -- 172.21.15.114:0/1310244939 learned_addr learned my addr 172.21.15.114:0/1310244939 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:55.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.074+0000 7f3e23400640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 0x7f3e240ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.074+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851875309 0 0) 0x7f3e24155870 con 0x7f3e2415ec10 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.074+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e04003520 con 0x7f3e2415ec10 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.074+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048204846 0 0) 0x7f3e04003520 con 0x7f3e2415ec10 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 msgr2=0x7f3e240ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 0x7f3e240ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 >> v1:172.21.15.177:6789/0 conn(0x7f3e24155490 legacy=0x7f3e240d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240adb50 con 0x7f3e2415ec10 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f3e1c001670 con 0x7f3e2415ec10 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 >> v1:172.21.15.114:6789/0 conn(0x7f3e2415ec10 legacy=0x7f3e2415f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e040039e0 0x7f3e04003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e04004800 0x7f3e04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 0x7f3e040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0400dc80 con 0x7f3e04004800 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0400de00 con 0x7f3e040039e0 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e04005990 con 0x7f3e240d0780 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e23400640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e04004800 0x7f3e04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.075+0000 7f3e28e00640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 0x7f3e040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.076+0000 7f3e23400640 1 -- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 msgr2=0x7f3e040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.076+0000 7f3e23400640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e240d0780 0x7f3e040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.076+0000 7f3e23400640 1 -- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e040039e0 msgr2=0x7f3e04003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.076+0000 7f3e23e00640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e040039e0 0x7f3e04003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.076+0000 7f3e23400640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e040039e0 0x7f3e04003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.076+0000 7f3e23400640 1 -- 172.21.15.114:0/1310244939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04003520 con 0x7f3e04004800 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.077+0000 7f3e23400640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e04004800 0x7f3e04004c80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009f90 tx=0x7f3e04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542fd0224d552e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.077+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c039070 con 0x7f3e04004800 2024-03-06T07:33:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.077+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c02cbd0 con 0x7f3e04004800 2024-03-06T07:33:55.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.077+0000 7f3e22a00640 1 -- 172.21.15.114:0/1310244939 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c009c10 con 0x7f3e04004800 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.078+0000 7f3e2ae00640 1 -- 172.21.15.114:0/1310244939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e04004800 msgr2=0x7f3e04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.078+0000 7f3e2ae00640 1 --2- 172.21.15.114:0/1310244939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e04004800 0x7f3e04004c80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009f90 tx=0x7f3e04005830 comp rx=0 tx=0).stop 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.078+0000 7f3e28e00640 1 -- 172.21.15.114:0/1310244939 reap_dead start 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.078+0000 7f3e2ae00640 1 -- 172.21.15.114:0/1310244939 shutdown_connections 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.078+0000 7f3e2ae00640 1 -- 172.21.15.114:0/1310244939 >> 172.21.15.114:0/1310244939 conn(0x7f3e2406b360 msgr2=0x7f3e2406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.078+0000 7f3e2ae00640 1 -- 172.21.15.114:0/1310244939 shutdown_connections 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.078+0000 7f3e2ae00640 1 -- 172.21.15.114:0/1310244939 wait complete. 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.080+0000 7f3e2ae00640 1 Processor -- start 2024-03-06T07:33:55.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.080+0000 7f3e2ae00640 1 -- start start 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e2ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04004800 0x7f3e241581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e2ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e2415c920 0x7f3e24146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e2ae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2415e210 0x7f3e24153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e2ae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240ae110 con 0x7f3e2415e210 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e2ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2415f980 con 0x7f3e04004800 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e2ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240ae630 con 0x7f3e2415c920 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2415e210 0x7f3e24153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2415e210 0x7f3e24153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43268/0 (socket says 172.21.15.114:43268) 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 -- 172.21.15.114:0/3145285015 learned_addr learned my addr 172.21.15.114:0/3145285015 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:55.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e23e00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04004800 0x7f3e241581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e23400640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e2415c920 0x7f3e24146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 -- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e2415c920 msgr2=0x7f3e24146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e2415c920 0x7f3e24146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 -- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04004800 msgr2=0x7f3e241581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04004800 0x7f3e241581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e28e00640 1 -- 172.21.15.114:0/3145285015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2415d170 con 0x7f3e2415e210 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e23e00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04004800 0x7f3e241581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.081+0000 7f3e23400640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e2415c920 0x7f3e24146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.082+0000 7f3e28e00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2415e210 0x7f3e24153400 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c000c00 tx=0x7f3e1c00e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.082+0000 7f3e1be00640 1 -- 172.21.15.114:0/3145285015 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c010510 con 0x7f3e2415e210 2024-03-06T07:33:55.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.082+0000 7f3e1be00640 1 -- 172.21.15.114:0/3145285015 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c0106b0 con 0x7f3e2415e210 2024-03-06T07:33:55.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.082+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2415d450 con 0x7f3e2415e210 2024-03-06T07:33:55.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.082+0000 7f3e1be00640 1 -- 172.21.15.114:0/3145285015 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c010a80 con 0x7f3e2415e210 2024-03-06T07:33:55.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.082+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2415da10 con 0x7f3e2415e210 2024-03-06T07:33:55.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.083+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e24068070 con 0x7f3e2415e210 2024-03-06T07:33:55.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.084+0000 7f3e1be00640 1 -- 172.21.15.114:0/3145285015 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e1c010c20 con 0x7f3e2415e210 2024-03-06T07:33:55.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.085+0000 7f3e1be00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e0006b520 0x7f3e0006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.085+0000 7f3e1be00640 1 -- 172.21.15.114:0/3145285015 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5529+0+0 (secure 0 0 0) 0x7f3e1c089800 con 0x7f3e2415e210 2024-03-06T07:33:55.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.085+0000 7f3e23e00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e0006b520 0x7f3e0006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.086+0000 7f3e23e00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e0006b520 0x7f3e0006d9e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3e140016f0 tx=0x7f3e14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.090+0000 7f3e1be00640 1 -- 172.21.15.114:0/3145285015 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e1c058710 con 0x7f3e2415e210 2024-03-06T07:33:55.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.297+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e24071740 con 0x7f3e2415e210 2024-03-06T07:33:55.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.846+0000 7f3e1be00640 1 -- 172.21.15.114:0/3145285015 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3e1c058010 con 0x7f3e2415e210 2024-03-06T07:33:55.849 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e0006b520 msgr2=0x7f3e0006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e0006b520 0x7f3e0006d9e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3e140016f0 tx=0x7f3e14009040 comp rx=0 tx=0).stop 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2415e210 msgr2=0x7f3e24153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2415e210 0x7f3e24153400 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c000c00 tx=0x7f3e1c00e910 comp rx=0 tx=0).stop 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 shutdown_connections 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3e0006b520 0x7f3e0006d9e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e2415e210 0x7f3e24153400 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3e2415c920 0x7f3e24146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 --2- 172.21.15.114:0/3145285015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e04004800 0x7f3e241581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 >> 172.21.15.114:0/3145285015 conn(0x7f3e2406b360 msgr2=0x7f3e240b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.850+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 shutdown_connections 2024-03-06T07:33:55.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:55.851+0000 7f3e2ae00640 1 -- 172.21.15.114:0/3145285015 wait complete. 2024-03-06T07:33:55.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:55.885 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:56.036 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-06T07:33:56.036 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-06T07:33:56.036 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:56.036 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-06T07:33:56.036 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.036 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.037 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.038 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on skipped message = False 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-06T07:33:56.039 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.040 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.041 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:56.042 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.043 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.045 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.050 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-06T07:33:56.051 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:56.052 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:56.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:56.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:56.247 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:56.248 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:56.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-06T07:33:56.250 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-06T07:33:56.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-03-06T07:33:56.252 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.253 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.254 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.255 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:33:56.256 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:56.257 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.258 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi114.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.259 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:56.260 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-06T07:33:56.261 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on skipped message = False 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:56.262 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-03-06T07:33:56.263 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.264 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:56.265 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.266 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-06T07:33:56.267 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.268 INFO:teuthology.orchestra.run.smithi177.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:33:56.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:56.300 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:56.463 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:56.464 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.465 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.466 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:56.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:56.468 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-06T07:33:56.469 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.470 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-03-06T07:33:56.471 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-03-06T07:33:56.472 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.473 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.474 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:56.475 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.114,172.21.15.177,[v2:172.21.15.177:3301,v1:172.21.15.177:6790]" 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on skipped message = False 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 149c34a4-b335-4a4b-8a41-0fe21b0db808 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.476 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:56.477 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: bdev async discard = True 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: bdev enable discard = True 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore allocator = bitmap 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 1/20 2024-03-06T07:33:56.478 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 1/20 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 4/10 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:56.479 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.480 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: mon op complaint time = 120 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-03-06T07:33:56.481 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: client mount timeout = 600 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: debug client = 20 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = "" 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.a] 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.c] 2024-03-06T07:33:56.482 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.b] 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout:[mds.d] 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout:[mds] 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds = 20 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: debug mds balancer = 20 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug frag = True 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: mds debug scatterstat = True 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: mds op complaint time = 180 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: mds verify scatter = True 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: osd op complaint time = 180 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: rados mon op timeout = 900 2024-03-06T07:33:56.483 INFO:teuthology.orchestra.run.smithi177.stdout: rados osd op timeout = 900 2024-03-06T07:33:56.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:33:56.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.725+0000 7fe863200640 1 Processor -- start 2024-03-06T07:33:56.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.725+0000 7fe863200640 1 -- start start 2024-03-06T07:33:56.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.726+0000 7fe863200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe85c0b3f70 0x7fe85c160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.726+0000 7fe863200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c070ff0 0x7fe85c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe863200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c0b46c0 con 0x7fe85c15fc10 2024-03-06T07:33:56.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe863200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85c071480 con 0x7fe85c0b3f70 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe863200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0b49a0 con 0x7fe85c070ff0 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe85be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe85c15fc10 0x7fe85c15fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38830/0 (socket says 172.21.15.114:38830) 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe85be00640 1 -- 172.21.15.114:0/700993562 learned_addr learned my addr 172.21.15.114:0/700993562 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe853400640 1 --2- 172.21.15.114:0/700993562 >> v2:172.21.15.114:3300/0 conn(0x7fe85c0b3f70 0x7fe85c160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe861200640 1 --2- 172.21.15.114:0/700993562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c070ff0 0x7fe85c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe85b400640 1 -- 172.21.15.114:0/700993562 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732253498 0 0) 0x7fe85c0b46c0 con 0x7fe85c15fc10 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe85b400640 1 -- 172.21.15.114:0/700993562 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe840003520 con 0x7fe85c15fc10 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe853400640 1 -- 172.21.15.114:0/700993562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c070ff0 msgr2=0x7fe85c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe853400640 1 --2- 172.21.15.114:0/700993562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c070ff0 0x7fe85c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.727+0000 7fe853400640 1 -- 172.21.15.114:0/700993562 >> v1:172.21.15.114:6789/0 conn(0x7fe85c15fc10 legacy=0x7fe85c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.728+0000 7fe853400640 1 -- 172.21.15.114:0/700993562 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c072c90 con 0x7fe85c0b3f70 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.729+0000 7fe853400640 1 --2- 172.21.15.114:0/700993562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c160700 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fe8440097a0 tx=0x7fe84402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9794b112e7c7300 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.729+0000 7fe85b400640 1 -- 172.21.15.114:0/700993562 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844035020 con 0x7fe85c0b3f70 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.729+0000 7fe85b400640 1 -- 172.21.15.114:0/700993562 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe844031810 con 0x7fe85c0b3f70 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.729+0000 7fe85b400640 1 -- 172.21.15.114:0/700993562 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844031bc0 con 0x7fe85c0b3f70 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.729+0000 7fe863200640 1 -- 172.21.15.114:0/700993562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 msgr2=0x7fe85c160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.729+0000 7fe863200640 1 --2- 172.21.15.114:0/700993562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c160700 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fe8440097a0 tx=0x7fe84402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.730+0000 7fe863200640 1 -- 172.21.15.114:0/700993562 shutdown_connections 2024-03-06T07:33:56.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.730+0000 7fe863200640 1 --2- 172.21.15.114:0/700993562 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c070ff0 0x7fe85c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.730+0000 7fe863200640 1 --2- 172.21.15.114:0/700993562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c160700 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.730+0000 7fe863200640 1 -- 172.21.15.114:0/700993562 >> 172.21.15.114:0/700993562 conn(0x7fe85c0b1940 msgr2=0x7fe85c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.730+0000 7fe863200640 1 -- 172.21.15.114:0/700993562 shutdown_connections 2024-03-06T07:33:56.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.730+0000 7fe863200640 1 -- 172.21.15.114:0/700993562 wait complete. 2024-03-06T07:33:56.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.731+0000 7fe863200640 1 Processor -- start 2024-03-06T07:33:56.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.732+0000 7fe863200640 1 -- start start 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.732+0000 7fe863200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85c070ff0 0x7fe85c1f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.732+0000 7fe863200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.732+0000 7fe863200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c15fc10 0x7fe85c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe863200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c072a30 con 0x7fe85c0b3f70 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe863200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0728b0 con 0x7fe85c070ff0 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe863200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c15c2e0 con 0x7fe85c15fc10 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43286/0 (socket says 172.21.15.114:43286) 2024-03-06T07:33:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 -- 172.21.15.114:0/3780523323 learned_addr learned my addr 172.21.15.114:0/3780523323 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe85be00640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85c070ff0 0x7fe85c1f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe861200640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c15fc10 0x7fe85c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 -- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c15fc10 msgr2=0x7fe85c1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c15fc10 0x7fe85c1513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 -- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85c070ff0 msgr2=0x7fe85c1f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85c070ff0 0x7fe85c1f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe853400640 1 -- 172.21.15.114:0/3780523323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c158330 con 0x7fe85c0b3f70 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.733+0000 7fe85be00640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85c070ff0 0x7fe85c1f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.734+0000 7fe853400640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c157df0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fe8440096f0 tx=0x7fe8440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.734+0000 7fe858c00640 1 -- 172.21.15.114:0/3780523323 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844041070 con 0x7fe85c0b3f70 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.734+0000 7fe858c00640 1 -- 172.21.15.114:0/3780523323 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84403ad00 con 0x7fe85c0b3f70 2024-03-06T07:33:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.734+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c1f48f0 con 0x7fe85c0b3f70 2024-03-06T07:33:56.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.734+0000 7fe858c00640 1 -- 172.21.15.114:0/3780523323 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844035040 con 0x7fe85c0b3f70 2024-03-06T07:33:56.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.734+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c1f4eb0 con 0x7fe85c0b3f70 2024-03-06T07:33:56.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.735+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c04f330 con 0x7fe85c0b3f70 2024-03-06T07:33:56.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.736+0000 7fe858c00640 1 -- 172.21.15.114:0/3780523323 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe84404c060 con 0x7fe85c0b3f70 2024-03-06T07:33:56.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.737+0000 7fe858c00640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe83806b4d0 0x7fe83806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.737+0000 7fe858c00640 1 -- 172.21.15.114:0/3780523323 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fe8440abac0 con 0x7fe85c0b3f70 2024-03-06T07:33:56.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.737+0000 7fe85be00640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe83806b4d0 0x7fe83806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.738+0000 7fe85be00640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe83806b4d0 0x7fe83806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe84c005880 tx=0x7fe84c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.742+0000 7fe858c00640 1 -- 172.21.15.114:0/3780523323 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe84404b2c0 con 0x7fe85c0b3f70 2024-03-06T07:33:56.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.938+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe85c150c70 con 0x7fe85c0b3f70 2024-03-06T07:33:56.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.945+0000 7fe858c00640 1 -- 172.21.15.114:0/3780523323 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe84407ac20 con 0x7fe85c0b3f70 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.947+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe83806b4d0 msgr2=0x7fe83806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.947+0000 7fe863200640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe83806b4d0 0x7fe83806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe84c005880 tx=0x7fe84c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.947+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 msgr2=0x7fe85c157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.947+0000 7fe863200640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c157df0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fe8440096f0 tx=0x7fe8440046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 shutdown_connections 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe85c15fc10 0x7fe85c1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe83806b4d0 0x7fe83806d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe85c0b3f70 0x7fe85c157df0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 --2- 172.21.15.114:0/3780523323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe85c070ff0 0x7fe85c1f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 >> 172.21.15.114:0/3780523323 conn(0x7fe85c0b1940 msgr2=0x7fe85c0ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 shutdown_connections 2024-03-06T07:33:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:56.948+0000 7fe863200640 1 -- 172.21.15.114:0/3780523323 wait complete. 2024-03-06T07:33:56.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:33:57.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.168+0000 7f6af5c00640 1 Processor -- start 2024-03-06T07:33:57.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.168+0000 7f6af5c00640 1 -- start start 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6af5c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6af015f9d0 0x7f6af015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6af5c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0070b10 0x7f6af0070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6af5c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af006cef0 con 0x7f6af0144c00 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6af5c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af006d1d0 con 0x7f6af015f9d0 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6af5c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af006d4b0 con 0x7f6af0070b10 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6ae6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6af015f9d0 0x7f6af015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6ae6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6af015f9d0 0x7f6af015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43288/0 (socket says 172.21.15.114:43288) 2024-03-06T07:33:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.170+0000 7f6ae6000640 1 -- 172.21.15.114:0/661933045 learned_addr learned my addr 172.21.15.114:0/661933045 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:57.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.171+0000 7f6aef400640 1 --2- 172.21.15.114:0/661933045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0070b10 0x7f6af0070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.171+0000 7f6ae6000640 1 -- 172.21.15.114:0/661933045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0070b10 msgr2=0x7f6af0070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:57.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.171+0000 7f6ae6000640 1 --2- 172.21.15.114:0/661933045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0070b10 0x7f6af0070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.171+0000 7f6ae6000640 1 -- 172.21.15.114:0/661933045 >> v1:172.21.15.114:6789/0 conn(0x7f6af0144c00 legacy=0x7f6af006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.171+0000 7f6ae6000640 1 -- 172.21.15.114:0/661933045 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af006d790 con 0x7f6af015f9d0 2024-03-06T07:33:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.171+0000 7f6aef400640 1 --2- 172.21.15.114:0/661933045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0070b10 0x7f6af0070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:57.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.172+0000 7f6ae6000640 1 --2- 172.21.15.114:0/661933045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af015f9d0 0x7f6af015fe60 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cf95402762952d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.172+0000 7f6aee000640 1 -- 172.21.15.114:0/661933045 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8034070 con 0x7f6af015f9d0 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.172+0000 7f6aee000640 1 -- 172.21.15.114:0/661933045 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad8030840 con 0x7f6af015f9d0 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.172+0000 7f6aee000640 1 -- 172.21.15.114:0/661933045 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8030c10 con 0x7f6af015f9d0 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 -- 172.21.15.114:0/661933045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af015f9d0 msgr2=0x7f6af015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 --2- 172.21.15.114:0/661933045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af015f9d0 0x7f6af015fe60 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 -- 172.21.15.114:0/661933045 shutdown_connections 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 --2- 172.21.15.114:0/661933045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0070b10 0x7f6af0070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 --2- 172.21.15.114:0/661933045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af015f9d0 0x7f6af015fe60 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 -- 172.21.15.114:0/661933045 >> 172.21.15.114:0/661933045 conn(0x7f6af00b1730 msgr2=0x7f6af00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 -- 172.21.15.114:0/661933045 shutdown_connections 2024-03-06T07:33:57.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.173+0000 7f6af5c00640 1 -- 172.21.15.114:0/661933045 wait complete. 2024-03-06T07:33:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.175+0000 7f6af5c00640 1 Processor -- start 2024-03-06T07:33:57.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.175+0000 7f6af5c00640 1 -- start start 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6af5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af0070b10 0x7f6af0139d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6af5c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0144c00 0x7f6af0139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6af5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6af015f9d0 0x7f6af00ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6af5c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af006ccd0 con 0x7f6af0070b10 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6af5c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af006cb50 con 0x7f6af015f9d0 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6af5c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af006df90 con 0x7f6af0144c00 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af0070b10 0x7f6af0139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af0070b10 0x7f6af0139d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43300/0 (socket says 172.21.15.114:43300) 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 -- 172.21.15.114:0/4210652410 learned_addr learned my addr 172.21.15.114:0/4210652410 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 -- 172.21.15.114:0/4210652410 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0144c00 msgr2=0x7f6af0139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0144c00 0x7f6af0139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aef400640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6af015f9d0 0x7f6af00ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 -- 172.21.15.114:0/4210652410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6af015f9d0 msgr2=0x7f6af00ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6af015f9d0 0x7f6af00ae920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.176+0000 7f6aeea00640 1 -- 172.21.15.114:0/4210652410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af00b2e60 con 0x7f6af0070b10 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.177+0000 7f6aeea00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af0070b10 0x7f6af0139d90 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c400 tx=0x7f6ae000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.177+0000 7f6ae7400640 1 -- 172.21.15.114:0/4210652410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae00052d0 con 0x7f6af0070b10 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.177+0000 7f6ae7400640 1 -- 172.21.15.114:0/4210652410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae0005490 con 0x7f6af0070b10 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.177+0000 7f6ae7400640 1 -- 172.21.15.114:0/4210652410 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0007260 con 0x7f6af0070b10 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.177+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af00b3140 con 0x7f6af0070b10 2024-03-06T07:33:57.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.177+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af00b3700 con 0x7f6af0070b10 2024-03-06T07:33:57.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.178+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af004f330 con 0x7f6af0070b10 2024-03-06T07:33:57.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.179+0000 7f6ae7400640 1 -- 172.21.15.114:0/4210652410 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ae0007400 con 0x7f6af0070b10 2024-03-06T07:33:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.180+0000 7f6ae7400640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6acc06b4d0 0x7f6acc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.180+0000 7f6ae7400640 1 -- 172.21.15.114:0/4210652410 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f6ae0089510 con 0x7f6af0070b10 2024-03-06T07:33:57.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.180+0000 7f6ae6000640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6acc06b4d0 0x7f6acc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:57.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.181+0000 7f6ae6000640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6acc06b4d0 0x7f6acc06d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8036000 tx=0x7f6ad803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:57.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.185+0000 7f6ae7400640 1 -- 172.21.15.114:0/4210652410 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ae00585d0 con 0x7f6af0070b10 2024-03-06T07:33:57.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.453+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f6af00afa20 con 0x7f6af0070b10 2024-03-06T07:33:57.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.853+0000 7f6ae7400640 1 -- 172.21.15.114:0/4210652410 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v332) v1 ==== 134+0+0 (secure 0 0 0) 0x7f6ae0057ed0 con 0x7f6af0070b10 2024-03-06T07:33:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6acc06b4d0 msgr2=0x7f6acc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6acc06b4d0 0x7f6acc06d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8036000 tx=0x7f6ad803b040 comp rx=0 tx=0).stop 2024-03-06T07:33:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af0070b10 msgr2=0x7f6af0139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af0070b10 0x7f6af0139d90 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c400 tx=0x7f6ae000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 shutdown_connections 2024-03-06T07:33:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6af015f9d0 0x7f6af00ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6af0144c00 0x7f6af0139240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6acc06b4d0 0x7f6acc06d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.856+0000 7f6af5c00640 1 --2- 172.21.15.114:0/4210652410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6af0070b10 0x7f6af0139d90 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.857+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 >> 172.21.15.114:0/4210652410 conn(0x7f6af00b1730 msgr2=0x7f6af00cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.857+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 shutdown_connections 2024-03-06T07:33:57.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:57.857+0000 7f6af5c00640 1 -- 172.21.15.114:0/4210652410 wait complete. 2024-03-06T07:33:57.880 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:33:57.880 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:33:57.880 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:33:57.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:58.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.084+0000 7fe221200640 1 Processor -- start 2024-03-06T07:33:58.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.084+0000 7fe221200640 1 -- start start 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe221200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe21c15acb0 0x7fe21c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe221200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c15b090 con 0x7fe21c06ec00 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe221200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c0b2a30 con 0x7fe21c0b4680 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe221200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c0b4360 con 0x7fe21c15acb0 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe219e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe21c15acb0 0x7fe21c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe21a800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe21c06ec00 0x7fe21c0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50472/0 (socket says 172.21.15.114:50472) 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe21a800640 1 -- 172.21.15.114:0/4111020416 learned_addr learned my addr 172.21.15.114:0/4111020416 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.086+0000 7fe219e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe21c15acb0 0x7fe21c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41918/0 (socket says 172.21.15.114:41918) 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.087+0000 7fe219e00640 1 -- 172.21.15.114:0/4111020416 >> v1:172.21.15.177:6789/0 conn(0x7fe21c0b4680 legacy=0x7fe21c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.087+0000 7fe219e00640 1 -- 172.21.15.114:0/4111020416 >> v1:172.21.15.114:6789/0 conn(0x7fe21c06ec00 legacy=0x7fe21c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.087+0000 7fe219e00640 1 -- 172.21.15.114:0/4111020416 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c0b2cf0 con 0x7fe21c15acb0 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.088+0000 7fe219e00640 1 --2- 172.21.15.114:0/4111020416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c15acb0 0x7fe21c0b37d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009d90 tx=0x7fe20c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2b25fb42ec8306 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.088+0000 7fe218a00640 1 -- 172.21.15.114:0/4111020416 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c039070 con 0x7fe21c15acb0 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.088+0000 7fe218a00640 1 -- 172.21.15.114:0/4111020416 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20c02cc60 con 0x7fe21c15acb0 2024-03-06T07:33:58.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.088+0000 7fe218a00640 1 -- 172.21.15.114:0/4111020416 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c034660 con 0x7fe21c15acb0 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.089+0000 7fe221200640 1 -- 172.21.15.114:0/4111020416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c15acb0 msgr2=0x7fe21c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.089+0000 7fe221200640 1 --2- 172.21.15.114:0/4111020416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c15acb0 0x7fe21c0b37d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009d90 tx=0x7fe20c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.089+0000 7fe221200640 1 -- 172.21.15.114:0/4111020416 shutdown_connections 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.089+0000 7fe221200640 1 --2- 172.21.15.114:0/4111020416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c15acb0 0x7fe21c0b37d0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.089+0000 7fe221200640 1 -- 172.21.15.114:0/4111020416 >> 172.21.15.114:0/4111020416 conn(0x7fe21c0b1730 msgr2=0x7fe21c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.089+0000 7fe221200640 1 -- 172.21.15.114:0/4111020416 shutdown_connections 2024-03-06T07:33:58.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.089+0000 7fe221200640 1 -- 172.21.15.114:0/4111020416 wait complete. 2024-03-06T07:33:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.091+0000 7fe221200640 1 Processor -- start 2024-03-06T07:33:58.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.091+0000 7fe221200640 1 -- start start 2024-03-06T07:33:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe221200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe21c06ec00 0x7fe21c157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe221200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c0b4680 0x7fe21c0aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe221200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe21c15acb0 0x7fe21c0aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe221200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b3d10 con 0x7fe21c0b4680 2024-03-06T07:33:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe221200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b32b0 con 0x7fe21c06ec00 2024-03-06T07:33:58.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe221200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b4010 con 0x7fe21c15acb0 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe219400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c0b4680 0x7fe21c0aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe219400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c0b4680 0x7fe21c0aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41928/0 (socket says 172.21.15.114:41928) 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.092+0000 7fe219400640 1 -- 172.21.15.114:0/1391482865 learned_addr learned my addr 172.21.15.114:0/1391482865 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe219400640 1 -- 172.21.15.114:0/1391482865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe21c15acb0 msgr2=0x7fe21c0aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe219400640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe21c15acb0 0x7fe21c0aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe219e00640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe21c06ec00 0x7fe21c157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe219400640 1 -- 172.21.15.114:0/1391482865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe21c06ec00 msgr2=0x7fe21c157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe219400640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe21c06ec00 0x7fe21c157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe219400640 1 -- 172.21.15.114:0/1391482865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c0ae500 con 0x7fe21c0b4680 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe219400640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c0b4680 0x7fe21c0aa4f0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe20400c460 tx=0x7fe20400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe212000640 1 -- 172.21.15.114:0/1391482865 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204016020 con 0x7fe21c0b4680 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe212000640 1 -- 172.21.15.114:0/1391482865 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe204012870 con 0x7fe21c0b4680 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c0ab060 con 0x7fe21c0b4680 2024-03-06T07:33:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe212000640 1 -- 172.21.15.114:0/1391482865 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204012c60 con 0x7fe21c0b4680 2024-03-06T07:33:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.093+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c0ab620 con 0x7fe21c0b4680 2024-03-06T07:33:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.094+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c04f330 con 0x7fe21c0b4680 2024-03-06T07:33:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.095+0000 7fe212000640 1 -- 172.21.15.114:0/1391482865 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe204019020 con 0x7fe21c0b4680 2024-03-06T07:33:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.097+0000 7fe212000640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe1f806b4d0 0x7fe1f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.097+0000 7fe212000640 1 -- 172.21.15.114:0/1391482865 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fe204089d00 con 0x7fe21c0b4680 2024-03-06T07:33:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.097+0000 7fe219e00640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe1f806b4d0 0x7fe1f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.098+0000 7fe219e00640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe1f806b4d0 0x7fe1f806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe20c036000 tx=0x7fe20c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.101+0000 7fe212000640 1 -- 172.21.15.114:0/1391482865 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe204020030 con 0x7fe21c0b4680 2024-03-06T07:33:58.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.298+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe21c0aa2d0 con 0x7fe21c0b4680 2024-03-06T07:33:58.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.300+0000 7fe212000640 1 -- 172.21.15.114:0/1391482865 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13334 (secure 0 0 0) 0x7fe21c0ae500 con 0x7fe21c0b4680 2024-03-06T07:33:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:33:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":100,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:33:55.837179+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6835/1490119219":"2024-03-07T07:33:52.816871+0000","172.21.15.114:6834/1490119219":"2024-03-07T07:33:52.816871+0000","172.21.15.114:6836/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6835/1475876370":"2024-03-07T07:33:16.974802+0000","172.21.15.114:6837/406262499":"2024-03-07T07:33:15.968840+0000","172.21.15.114:6834/406262499":"2024-03-07T07:33:15.968840+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe1f806b4d0 msgr2=0x7fe1f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe1f806b4d0 0x7fe1f806d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe20c036000 tx=0x7fe20c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c0b4680 msgr2=0x7fe21c0aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c0b4680 0x7fe21c0aa4f0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe20400c460 tx=0x7fe20400c920 comp rx=0 tx=0).stop 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 shutdown_connections 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe21c15acb0 0x7fe21c0aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe1f806b4d0 0x7fe1f806d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe21c0b4680 0x7fe21c0aa4f0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 --2- 172.21.15.114:0/1391482865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe21c06ec00 0x7fe21c157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 >> 172.21.15.114:0/1391482865 conn(0x7fe21c0b1730 msgr2=0x7fe21c159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 shutdown_connections 2024-03-06T07:33:58.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.303+0000 7fe221200640 1 -- 172.21.15.114:0/1391482865 wait complete. 2024-03-06T07:33:58.323 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:58.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.528+0000 7f155fe00640 1 Processor -- start 2024-03-06T07:33:58.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.529+0000 7f155fe00640 1 -- start start 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580d0900 0x7f15580ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15580ceac0 0x7f155806edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155fe00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15580cf8a0 con 0x7f155815bfe0 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580cef10 con 0x7f15580ceac0 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580cfb40 con 0x7f15580d0900 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15580ceac0 0x7f155806edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15580ceac0 0x7f155806edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41934/0 (socket says 172.21.15.114:41934) 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.530+0000 7f155de00640 1 -- 172.21.15.114:0/2839988486 learned_addr learned my addr 172.21.15.114:0/2839988486 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.531+0000 7f155ca00640 1 --2- 172.21.15.114:0/2839988486 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580d0900 0x7f15580ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.531+0000 7f155de00640 1 -- 172.21.15.114:0/2839988486 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580d0900 msgr2=0x7f15580ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.531+0000 7f155de00640 1 --2- 172.21.15.114:0/2839988486 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580d0900 0x7f15580ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.531+0000 7f155de00640 1 -- 172.21.15.114:0/2839988486 >> v1:172.21.15.177:6789/0 conn(0x7f155815bfe0 legacy=0x7f15580cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.531+0000 7f155de00640 1 -- 172.21.15.114:0/2839988486 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580cfe20 con 0x7f15580ceac0 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.532+0000 7f155de00640 1 --2- 172.21.15.114:0/2839988486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15580ceac0 0x7f155806edd0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f1550009e50 tx=0x7f155002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8590823f0e6e37c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.532+0000 7f1557e00640 1 -- 172.21.15.114:0/2839988486 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155002cb00 con 0x7f15580ceac0 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.532+0000 7f1557e00640 1 -- 172.21.15.114:0/2839988486 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155002cca0 con 0x7f15580ceac0 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.532+0000 7f1557e00640 1 -- 172.21.15.114:0/2839988486 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550034610 con 0x7f15580ceac0 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 -- 172.21.15.114:0/2839988486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15580ceac0 msgr2=0x7f155806edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 --2- 172.21.15.114:0/2839988486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15580ceac0 0x7f155806edd0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f1550009e50 tx=0x7f155002bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 -- 172.21.15.114:0/2839988486 shutdown_connections 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 --2- 172.21.15.114:0/2839988486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15580ceac0 0x7f155806edd0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 --2- 172.21.15.114:0/2839988486 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580d0900 0x7f15580ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 -- 172.21.15.114:0/2839988486 >> 172.21.15.114:0/2839988486 conn(0x7f15580b3800 msgr2=0x7f15580b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 -- 172.21.15.114:0/2839988486 shutdown_connections 2024-03-06T07:33:58.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.533+0000 7f155fe00640 1 -- 172.21.15.114:0/2839988486 wait complete. 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.534+0000 7f155fe00640 1 Processor -- start 2024-03-06T07:33:58.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.535+0000 7f155fe00640 1 -- start start 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.535+0000 7f155fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580ceac0 0x7f15581f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.535+0000 7f155fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15580d0900 0x7f15580c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155815bfe0 0x7f15580ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580d0d20 con 0x7f155815bfe0 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580d03e0 con 0x7f15580d0900 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580b4540 con 0x7f15580ceac0 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155815bfe0 0x7f15580ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155815bfe0 0x7f15580ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41942/0 (socket says 172.21.15.114:41942) 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 -- 172.21.15.114:0/940944434 learned_addr learned my addr 172.21.15.114:0/940944434 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155ca00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15580d0900 0x7f15580c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155d400640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580ceac0 0x7f15581f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 -- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580ceac0 msgr2=0x7f15581f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580ceac0 0x7f15581f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 -- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15580d0900 msgr2=0x7f15580c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15580d0900 0x7f15580c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155de00640 1 -- 172.21.15.114:0/940944434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155815b530 con 0x7f155815bfe0 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.536+0000 7f155d400640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580ceac0 0x7f15581f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.537+0000 7f155de00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155815bfe0 0x7f15580ca330 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1550036040 tx=0x7f1550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.537+0000 7f1555600640 1 -- 172.21.15.114:0/940944434 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550043020 con 0x7f155815bfe0 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.537+0000 7f1555600640 1 -- 172.21.15.114:0/940944434 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155002ce50 con 0x7f155815bfe0 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.537+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15580ca960 con 0x7f155815bfe0 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.537+0000 7f1555600640 1 -- 172.21.15.114:0/940944434 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155003e650 con 0x7f155815bfe0 2024-03-06T07:33:58.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.537+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15580caf20 con 0x7f155815bfe0 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.538+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155804f330 con 0x7f155815bfe0 2024-03-06T07:33:58.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.539+0000 7f1555600640 1 -- 172.21.15.114:0/940944434 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1550046020 con 0x7f155815bfe0 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.540+0000 7f1555600640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f153c06b4d0 0x7f153c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.540+0000 7f1555600640 1 -- 172.21.15.114:0/940944434 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f15500ac6f0 con 0x7f155815bfe0 2024-03-06T07:33:58.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.541+0000 7f155d400640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f153c06b4d0 0x7f153c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.541+0000 7f155d400640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f153c06b4d0 0x7f153c06d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f154c00a830 tx=0x7f154c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.545+0000 7f1555600640 1 -- 172.21.15.114:0/940944434 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f155004d030 con 0x7f155815bfe0 2024-03-06T07:33:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.741+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15580b4320 con 0x7f155815bfe0 2024-03-06T07:33:58.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.742+0000 7f1555600640 1 -- 172.21.15.114:0/940944434 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f155007b7c0 con 0x7f155815bfe0 2024-03-06T07:33:58.744 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f153c06b4d0 msgr2=0x7f153c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f153c06b4d0 0x7f153c06d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f154c00a830 tx=0x7f154c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155815bfe0 msgr2=0x7f15580ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155815bfe0 0x7f15580ca330 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1550036040 tx=0x7f1550004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 shutdown_connections 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f153c06b4d0 0x7f153c06d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155815bfe0 0x7f15580ca330 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15580d0900 0x7f15580c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 --2- 172.21.15.114:0/940944434 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f15580ceac0 0x7f15581f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 >> 172.21.15.114:0/940944434 conn(0x7f15580b3800 msgr2=0x7f155806edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 shutdown_connections 2024-03-06T07:33:58.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.745+0000 7f155fe00640 1 -- 172.21.15.114:0/940944434 wait complete. 2024-03-06T07:33:58.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:33:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.973+0000 7fbc96e00640 1 Processor -- start 2024-03-06T07:33:58.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.973+0000 7fbc96e00640 1 -- start start 2024-03-06T07:33:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc96e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc900cf370 0x7fbc9006d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc96e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc900cc2a0 0x7fbc900cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc96e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc900b2b30 con 0x7fbc9015a940 2024-03-06T07:33:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc96e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc900b28e0 con 0x7fbc900cf370 2024-03-06T07:33:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc96e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900b2e10 con 0x7fbc900cc2a0 2024-03-06T07:33:58.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc900cf370 0x7fbc9006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbc9015a940 0x7fbc900b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50488/0 (socket says 172.21.15.114:50488) 2024-03-06T07:33:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbc900cf370 0x7fbc9006d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41946/0 (socket says 172.21.15.114:41946) 2024-03-06T07:33:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95a00640 1 -- 172.21.15.114:0/2814391756 learned_addr learned my addr 172.21.15.114:0/2814391756 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:33:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95000640 1 -- 172.21.15.114:0/2814391756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc900cc2a0 msgr2=0x7fbc900cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95000640 1 --2- 172.21.15.114:0/2814391756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc900cc2a0 0x7fbc900cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95000640 1 -- 172.21.15.114:0/2814391756 >> v1:172.21.15.114:6789/0 conn(0x7fbc9015a940 legacy=0x7fbc900b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.975+0000 7fbc95000640 1 -- 172.21.15.114:0/2814391756 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9006ca50 con 0x7fbc900cf370 2024-03-06T07:33:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.976+0000 7fbc95000640 1 --2- 172.21.15.114:0/2814391756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cf370 0x7fbc9006d250 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0097a0 tx=0x7fbc7c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf9cb06691a6852 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc8fe00640 1 -- 172.21.15.114:0/2814391756 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c035020 con 0x7fbc900cf370 2024-03-06T07:33:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc8fe00640 1 -- 172.21.15.114:0/2814391756 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c031810 con 0x7fbc900cf370 2024-03-06T07:33:58.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc8fe00640 1 -- 172.21.15.114:0/2814391756 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c031be0 con 0x7fbc900cf370 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 -- 172.21.15.114:0/2814391756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cf370 msgr2=0x7fbc9006d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2814391756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cf370 0x7fbc9006d250 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0097a0 tx=0x7fbc7c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 -- 172.21.15.114:0/2814391756 shutdown_connections 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2814391756 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc900cc2a0 0x7fbc900cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2814391756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cf370 0x7fbc9006d250 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 -- 172.21.15.114:0/2814391756 >> 172.21.15.114:0/2814391756 conn(0x7fbc9006afc0 msgr2=0x7fbc9006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 -- 172.21.15.114:0/2814391756 shutdown_connections 2024-03-06T07:33:58.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.977+0000 7fbc96e00640 1 -- 172.21.15.114:0/2814391756 wait complete. 2024-03-06T07:33:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.979+0000 7fbc96e00640 1 Processor -- start 2024-03-06T07:33:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.979+0000 7fbc96e00640 1 -- start start 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc96e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cc2a0 0x7fbc90152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc96e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc900cf370 0x7fbc901f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc96e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc9015a940 0x7fbc900adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc96e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9006d790 con 0x7fbc900cc2a0 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc96e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc900b30f0 con 0x7fbc900cf370 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc96e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc901e4490 con 0x7fbc9015a940 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc95a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cc2a0 0x7fbc90152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc95a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cc2a0 0x7fbc90152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41960/0 (socket says 172.21.15.114:41960) 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.980+0000 7fbc95a00640 1 -- 172.21.15.114:0/2189084958 learned_addr learned my addr 172.21.15.114:0/2189084958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc95a00640 1 -- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc9015a940 msgr2=0x7fbc900adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc96400640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc9015a940 0x7fbc900adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc95a00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc9015a940 0x7fbc900adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc95a00640 1 -- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc900cf370 msgr2=0x7fbc901f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc95a00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc900cf370 0x7fbc901f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc95a00640 1 -- 172.21.15.114:0/2189084958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc901534e0 con 0x7fbc900cc2a0 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc96400640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc9015a940 0x7fbc900adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc95a00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cc2a0 0x7fbc90152fa0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c3c0 tx=0x7fbc8400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc8d600640 1 -- 172.21.15.114:0/2189084958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84017070 con 0x7fbc900cc2a0 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc8d600640 1 -- 172.21.15.114:0/2189084958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc84005ce0 con 0x7fbc900cc2a0 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9015cbb0 con 0x7fbc900cc2a0 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc8d600640 1 -- 172.21.15.114:0/2189084958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc840072f0 con 0x7fbc900cc2a0 2024-03-06T07:33:58.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.981+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9015d170 con 0x7fbc900cc2a0 2024-03-06T07:33:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.983+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc900b41c0 con 0x7fbc900cc2a0 2024-03-06T07:33:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.984+0000 7fbc8d600640 1 -- 172.21.15.114:0/2189084958 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc8401a070 con 0x7fbc900cc2a0 2024-03-06T07:33:58.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.985+0000 7fbc8d600640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc7006b4d0 0x7fbc7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.985+0000 7fbc8d600640 1 -- 172.21.15.114:0/2189084958 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fbc84089300 con 0x7fbc900cc2a0 2024-03-06T07:33:58.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.985+0000 7fbc95000640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc7006b4d0 0x7fbc7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.986+0000 7fbc95000640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc7006b4d0 0x7fbc7006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c036000 tx=0x7fbc7c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:58.989+0000 7fbc8d600640 1 -- 172.21.15.114:0/2189084958 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc900b41c0 con 0x7fbc900cc2a0 2024-03-06T07:33:59.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.258+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7fbc9015eb20 con 0x7fbc900cc2a0 2024-03-06T07:33:59.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.865+0000 7fbc8d600640 1 -- 172.21.15.114:0/2189084958 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v334) v1 ==== 145+0+0 (secure 0 0 0) 0x7fbc9015eb20 con 0x7fbc900cc2a0 2024-03-06T07:33:59.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.868+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc7006b4d0 msgr2=0x7fbc7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.868+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc7006b4d0 0x7fbc7006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c036000 tx=0x7fbc7c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:59.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.868+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cc2a0 msgr2=0x7fbc90152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.868+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cc2a0 0x7fbc90152fa0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c3c0 tx=0x7fbc8400c880 comp rx=0 tx=0).stop 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 shutdown_connections 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbc9015a940 0x7fbc900adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc900cf370 0x7fbc901f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbc7006b4d0 0x7fbc7006d990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 --2- 172.21.15.114:0/2189084958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc900cc2a0 0x7fbc90152fa0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 >> 172.21.15.114:0/2189084958 conn(0x7fbc9006afc0 msgr2=0x7fbc900cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 shutdown_connections 2024-03-06T07:33:59.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:33:59.869+0000 7fbc96e00640 1 -- 172.21.15.114:0/2189084958 wait complete. 2024-03-06T07:33:59.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:34:00.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.088+0000 7f2f88c00640 1 Processor -- start 2024-03-06T07:34:00.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.088+0000 7f2f88c00640 1 -- start start 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.089+0000 7f2f88c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f840b3df0 0x7f2f840727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f88c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f840719e0 con 0x7f2f840b3650 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f88c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f84071cc0 con 0x7f2f840cda90 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f88c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84072310 con 0x7f2f840b3df0 2024-03-06T07:34:00.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f81400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2f840b3650 0x7f2f840b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50496/0 (socket says 172.21.15.114:50496) 2024-03-06T07:34:00.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f81400640 1 -- 172.21.15.114:0/2293682468 learned_addr learned my addr 172.21.15.114:0/2293682468 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:00.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f81e00640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f840b3df0 0x7f2f840727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104996495 0 0) 0x7f2f840719e0 con 0x7f2f840b3650 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.090+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f68003520 con 0x7f2f840b3650 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470194172 0 0) 0x7f2f68003520 con 0x7f2f840b3650 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f840b3df0 msgr2=0x7f2f840727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f840b3df0 0x7f2f840727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 >> v1:172.21.15.177:6789/0 conn(0x7f2f840cda90 legacy=0x7f2f8406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f84071fa0 con 0x7f2f840b3650 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2f70001670 con 0x7f2f840b3650 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 >> v1:172.21.15.114:6789/0 conn(0x7f2f840b3650 legacy=0x7f2f840b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f840b3df0 0x7f2f680039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f68003f60 0x7f2f68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68004d40 0x7f2f680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84072310 con 0x7f2f68003f60 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6800baa0 con 0x7f2f68004d40 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6800bca0 con 0x7f2f840b3df0 2024-03-06T07:34:00.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.091+0000 7f2f81400640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f68003f60 0x7f2f68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.092+0000 7f2f81400640 1 -- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f840b3df0 msgr2=0x7f2f680039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.092+0000 7f2f81400640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f840b3df0 0x7f2f680039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.092+0000 7f2f81400640 1 -- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68004d40 msgr2=0x7f2f680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.092+0000 7f2f81400640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f68004d40 0x7f2f680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.092+0000 7f2f81400640 1 -- 172.21.15.114:0/2293682468 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f68003520 con 0x7f2f68003f60 2024-03-06T07:34:00.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.093+0000 7f2f81400640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f68003f60 0x7f2f68004800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c210 tx=0x7f2f68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77b0b8ea07e9fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.093+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70007d30 con 0x7f2f68003f60 2024-03-06T07:34:00.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.093+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f70030070 con 0x7f2f68003f60 2024-03-06T07:34:00.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.093+0000 7f2f80a00640 1 -- 172.21.15.114:0/2293682468 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7002cda0 con 0x7f2f68003f60 2024-03-06T07:34:00.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.094+0000 7f2f88c00640 1 -- 172.21.15.114:0/2293682468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f68003f60 msgr2=0x7f2f68004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.094+0000 7f2f88c00640 1 --2- 172.21.15.114:0/2293682468 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f68003f60 0x7f2f68004800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c210 tx=0x7f2f68005830 comp rx=0 tx=0).stop 2024-03-06T07:34:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.094+0000 7f2f82800640 1 -- 172.21.15.114:0/2293682468 reap_dead start 2024-03-06T07:34:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.094+0000 7f2f88c00640 1 -- 172.21.15.114:0/2293682468 shutdown_connections 2024-03-06T07:34:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.094+0000 7f2f88c00640 1 -- 172.21.15.114:0/2293682468 >> 172.21.15.114:0/2293682468 conn(0x7f2f840b1940 msgr2=0x7f2f840b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.094+0000 7f2f88c00640 1 -- 172.21.15.114:0/2293682468 shutdown_connections 2024-03-06T07:34:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.094+0000 7f2f88c00640 1 -- 172.21.15.114:0/2293682468 wait complete. 2024-03-06T07:34:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.095+0000 7f2f88c00640 1 Processor -- start 2024-03-06T07:34:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.096+0000 7f2f88c00640 1 -- start start 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.096+0000 7f2f88c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f68003f60 0x7f2f840ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.096+0000 7f2f88c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f840aeb60 0x7f2f8409c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f88c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f8414bc40 0x7f2f84141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f88c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8415b3c0 con 0x7f2f8414bc40 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f88c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8415b240 con 0x7f2f840aeb60 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f88c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f840cf510 con 0x7f2f68003f60 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f8414bc40 0x7f2f84141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f8414bc40 0x7f2f84141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41964/0 (socket says 172.21.15.114:41964) 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 -- 172.21.15.114:0/1957542503 learned_addr learned my addr 172.21.15.114:0/1957542503 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f81e00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f68003f60 0x7f2f840ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f81400640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f840aeb60 0x7f2f8409c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 -- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f68003f60 msgr2=0x7f2f840ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f68003f60 0x7f2f840ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 -- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f840aeb60 msgr2=0x7f2f8409c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f840aeb60 0x7f2f8409c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f82800640 1 -- 172.21.15.114:0/1957542503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f840aefe0 con 0x7f2f8414bc40 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.097+0000 7f2f81e00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f68003f60 0x7f2f840ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.098+0000 7f2f82800640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f8414bc40 0x7f2f84141d70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f2f8406ed00 tx=0x7f2f7c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.098+0000 7f2f7a000640 1 -- 172.21.15.114:0/1957542503 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c017070 con 0x7f2f8414bc40 2024-03-06T07:34:00.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.098+0000 7f2f7a000640 1 -- 172.21.15.114:0/1957542503 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7c005d60 con 0x7f2f8414bc40 2024-03-06T07:34:00.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.098+0000 7f2f7a000640 1 -- 172.21.15.114:0/1957542503 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c00a850 con 0x7f2f8414bc40 2024-03-06T07:34:00.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.098+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8413e610 con 0x7f2f8414bc40 2024-03-06T07:34:00.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.098+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f840ae5e0 con 0x7f2f8414bc40 2024-03-06T07:34:00.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.099+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8404f330 con 0x7f2f8414bc40 2024-03-06T07:34:00.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.100+0000 7f2f7a000640 1 -- 172.21.15.114:0/1957542503 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f7c00a9f0 con 0x7f2f8414bc40 2024-03-06T07:34:00.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.101+0000 7f2f7a000640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f6406b500 0x7f2f6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.101+0000 7f2f7a000640 1 -- 172.21.15.114:0/1957542503 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f2f7c089170 con 0x7f2f8414bc40 2024-03-06T07:34:00.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.101+0000 7f2f81e00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f6406b500 0x7f2f6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.102+0000 7f2f81e00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f6406b500 0x7f2f6406d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2f740047b0 tx=0x7f2f74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.106+0000 7f2f7a000640 1 -- 172.21.15.114:0/1957542503 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f8413e610 con 0x7f2f8414bc40 2024-03-06T07:34:00.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.310+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2f840ae940 con 0x7f2f8414bc40 2024-03-06T07:34:00.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.317+0000 7f2f7a000640 1 -- 172.21.15.114:0/1957542503 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2f7c0582d0 con 0x7f2f8414bc40 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.319+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f6406b500 msgr2=0x7f2f6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.319+0000 7f2f88c00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f6406b500 0x7f2f6406d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2f740047b0 tx=0x7f2f74008040 comp rx=0 tx=0).stop 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.319+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f8414bc40 msgr2=0x7f2f84141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.319+0000 7f2f88c00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f8414bc40 0x7f2f84141d70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f2f8406ed00 tx=0x7f2f7c00cb90 comp rx=0 tx=0).stop 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 shutdown_connections 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f6406b500 0x7f2f6406d9c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f8414bc40 0x7f2f84141d70 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f840aeb60 0x7f2f8409c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 --2- 172.21.15.114:0/1957542503 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f68003f60 0x7f2f840ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 >> 172.21.15.114:0/1957542503 conn(0x7f2f840b1940 msgr2=0x7f2f840af890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 shutdown_connections 2024-03-06T07:34:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:00.320+0000 7f2f88c00640 1 -- 172.21.15.114:0/1957542503 wait complete. 2024-03-06T07:34:00.339 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:34:00.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:00.446 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:00.563 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:34:00.564 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:34:00.564 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:00.617 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:00.617 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:34:00.617 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:00.673 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:00.673 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:34:00.673 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:00.732 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:00.732 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:34:00.732 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:00.787 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:00.787 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:34:00.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.009+0000 7f37d5600640 1 Processor -- start 2024-03-06T07:34:01.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.009+0000 7f37d5600640 1 -- start start 2024-03-06T07:34:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37d5600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f37d00af1c0 0x7f37d015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37d5600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37d006e4d0 0x7f37d006e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37d5600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d0071d30 con 0x7f37d0070d20 2024-03-06T07:34:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37d5600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d006ee80 con 0x7f37d006e4d0 2024-03-06T07:34:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37d5600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d00711d0 con 0x7f37d00af1c0 2024-03-06T07:34:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37cf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37d006e4d0 0x7f37d006e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37cf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37d006e4d0 0x7f37d006e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41970/0 (socket says 172.21.15.114:41970) 2024-03-06T07:34:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.011+0000 7f37cf400640 1 -- 172.21.15.114:0/201470547 learned_addr learned my addr 172.21.15.114:0/201470547 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.012+0000 7f37cea00640 1 --2- 172.21.15.114:0/201470547 >> v2:172.21.15.177:3300/0 conn(0x7f37d00af1c0 0x7f37d015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.012+0000 7f37cf400640 1 -- 172.21.15.114:0/201470547 >> v2:172.21.15.177:3300/0 conn(0x7f37d00af1c0 msgr2=0x7f37d015eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.012+0000 7f37cf400640 1 --2- 172.21.15.114:0/201470547 >> v2:172.21.15.177:3300/0 conn(0x7f37d00af1c0 0x7f37d015eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.012+0000 7f37cf400640 1 -- 172.21.15.114:0/201470547 >> v1:172.21.15.177:6789/0 conn(0x7f37d0070d20 legacy=0x7f37d006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.012+0000 7f37cf400640 1 -- 172.21.15.114:0/201470547 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d0071490 con 0x7f37d006e4d0 2024-03-06T07:34:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.013+0000 7f37cf400640 1 --2- 172.21.15.114:0/201470547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d006e4d0 0x7f37d006e940 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009e50 tx=0x7f37c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32a743543c0610f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.013+0000 7f37cd600640 1 -- 172.21.15.114:0/201470547 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c802ca80 con 0x7f37d006e4d0 2024-03-06T07:34:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.013+0000 7f37cd600640 1 -- 172.21.15.114:0/201470547 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c802cc20 con 0x7f37d006e4d0 2024-03-06T07:34:01.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.013+0000 7f37cd600640 1 -- 172.21.15.114:0/201470547 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c80343f0 con 0x7f37d006e4d0 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 -- 172.21.15.114:0/201470547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d006e4d0 msgr2=0x7f37d006e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 --2- 172.21.15.114:0/201470547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d006e4d0 0x7f37d006e940 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009e50 tx=0x7f37c802bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 -- 172.21.15.114:0/201470547 shutdown_connections 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 --2- 172.21.15.114:0/201470547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d006e4d0 0x7f37d006e940 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 --2- 172.21.15.114:0/201470547 >> v2:172.21.15.177:3300/0 conn(0x7f37d00af1c0 0x7f37d015eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 -- 172.21.15.114:0/201470547 >> 172.21.15.114:0/201470547 conn(0x7f37d00b1730 msgr2=0x7f37d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 -- 172.21.15.114:0/201470547 shutdown_connections 2024-03-06T07:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.014+0000 7f37d5600640 1 -- 172.21.15.114:0/201470547 wait complete. 2024-03-06T07:34:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.016+0000 7f37d5600640 1 Processor -- start 2024-03-06T07:34:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.017+0000 7f37d5600640 1 -- start start 2024-03-06T07:34:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37d5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d006e4d0 0x7f37d0153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37d5600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37d0070d20 0x7f37d0154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37d5600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d00af1c0 0x7f37d01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37d5600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d00b30b0 con 0x7f37d00af1c0 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37d5600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d0071a50 con 0x7f37d006e4d0 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37d5600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d006d370 con 0x7f37d0070d20 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d00af1c0 0x7f37d01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d00af1c0 0x7f37d01554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41972/0 (socket says 172.21.15.114:41972) 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.018+0000 7f37cf400640 1 -- 172.21.15.114:0/3915161765 learned_addr learned my addr 172.21.15.114:0/3915161765 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37cea00640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d006e4d0 0x7f37d0153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37ce000640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37d0070d20 0x7f37d0154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37cf400640 1 -- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37d0070d20 msgr2=0x7f37d0154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37cf400640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37d0070d20 0x7f37d0154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37cf400640 1 -- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d006e4d0 msgr2=0x7f37d0153710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37cf400640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d006e4d0 0x7f37d0153710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37cf400640 1 -- 172.21.15.114:0/3915161765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d0155b20 con 0x7f37d00af1c0 2024-03-06T07:34:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37cf400640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d00af1c0 0x7f37d01554a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009e50 tx=0x7f37c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.019+0000 7f37c6a00640 1 -- 172.21.15.114:0/3915161765 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8043020 con 0x7f37d00af1c0 2024-03-06T07:34:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.020+0000 7f37c6a00640 1 -- 172.21.15.114:0/3915161765 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c803e450 con 0x7f37d00af1c0 2024-03-06T07:34:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.020+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d0155e00 con 0x7f37d00af1c0 2024-03-06T07:34:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.020+0000 7f37c6a00640 1 -- 172.21.15.114:0/3915161765 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c803e820 con 0x7f37d00af1c0 2024-03-06T07:34:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.020+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d015f100 con 0x7f37d00af1c0 2024-03-06T07:34:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.021+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d0156320 con 0x7f37d00af1c0 2024-03-06T07:34:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.022+0000 7f37c6a00640 1 -- 172.21.15.114:0/3915161765 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37c8046020 con 0x7f37d00af1c0 2024-03-06T07:34:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.023+0000 7f37c6a00640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37b006b4d0 0x7f37b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.023+0000 7f37c6a00640 1 -- 172.21.15.114:0/3915161765 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f37c80ac8d0 con 0x7f37d00af1c0 2024-03-06T07:34:01.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.023+0000 7f37cea00640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37b006b4d0 0x7f37b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.024+0000 7f37cea00640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37b006b4d0 0x7f37b006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f37c000a970 tx=0x7f37c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.027+0000 7f37c6a00640 1 -- 172.21.15.114:0/3915161765 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37d0156320 con 0x7f37d00af1c0 2024-03-06T07:34:01.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.276+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37d004f330 con 0x7f37d00af1c0 2024-03-06T07:34:01.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.277+0000 7f37c6a00640 1 -- 172.21.15.114:0/3915161765 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f37c804d030 con 0x7f37d00af1c0 2024-03-06T07:34:01.279 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:01.279 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5827,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1261595555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1261595555},{"type":"v1","addr":"172.21.15.177:6833","nonce":1261595555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":5830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3670404101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3670404101},{"type":"v1","addr":"172.21.15.177:6835","nonce":3670404101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/165502269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":165502269},{"type":"v1","addr":"172.21.15.114:6837","nonce":165502269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7584,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/420238277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":420238277},{"type":"v1","addr":"172.21.15.114:6835","nonce":420238277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72}],"filesystems":[]} 2024-03-06T07:34:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 72 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37b006b4d0 msgr2=0x7f37b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37b006b4d0 0x7f37b006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f37c000a970 tx=0x7f37c0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d00af1c0 msgr2=0x7f37d01554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d00af1c0 0x7f37d01554a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009e50 tx=0x7f37c8004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 shutdown_connections 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f37b006b4d0 0x7f37b006d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37d00af1c0 0x7f37d01554a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37d0070d20 0x7f37d0154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 --2- 172.21.15.114:0/3915161765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37d006e4d0 0x7f37d0153710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 >> 172.21.15.114:0/3915161765 conn(0x7f37d00b1730 msgr2=0x7f37d015beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 shutdown_connections 2024-03-06T07:34:01.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.280+0000 7f37d5600640 1 -- 172.21.15.114:0/3915161765 wait complete. 2024-03-06T07:34:01.299 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:34:01.299 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:34:01.299 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:01.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:01.300 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:34:01.300 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:34:01.300 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:01.300 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:01.300 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:34:01.300 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:34:01.301 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:34:01.301 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:01.301 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:34:01.301 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:34:01.301 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:34:01.301 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:01.314 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:01.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:34:01.315 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:01.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:34:01.320 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:34:01.321 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:34:01.326 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:34:01.326 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:34:01.360 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:34:01.360 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:01.365 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:34:01.365 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:01.373 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:34:01.373 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:01.379 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:34:01.379 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:01.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:34:01.506 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:34:01.506 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:34:01.530 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:34:01.531 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.561+0000 7fac0b200640 1 Processor -- start 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.561+0000 7fac0b200640 1 -- start start 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.562+0000 7fac0b200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fac0c0cc2a0 0x7fac0c0cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.562+0000 7fac0b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0b2b30 con 0x7fac0c0cf370 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.562+0000 7fac0b200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0b2e10 con 0x7fac0c15a940 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.562+0000 7fac0b200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0c0b28e0 con 0x7fac0c0cc2a0 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.562+0000 7fac09400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fac0c0cf370 0x7fac0c06d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50520/0 (socket says 172.21.15.114:50520) 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.562+0000 7fac09400640 1 -- 172.21.15.114:0/3725372077 learned_addr learned my addr 172.21.15.114:0/3725372077 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.563+0000 7fac0a800640 1 --2- 172.21.15.114:0/3725372077 >> v2:172.21.15.177:3300/0 conn(0x7fac0c0cc2a0 0x7fac0c0cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.563+0000 7fac08a00640 1 -- 172.21.15.114:0/3725372077 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781619458 0 0) 0x7fac0c0b2b30 con 0x7fac0c0cf370 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.563+0000 7fac08a00640 1 -- 172.21.15.114:0/3725372077 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf0005730 con 0x7fac0c0cf370 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.563+0000 7fac0a800640 1 -- 172.21.15.114:0/3725372077 >> v1:172.21.15.177:6789/0 conn(0x7fac0c15a940 legacy=0x7fac0c0b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.563+0000 7fac08a00640 1 -- 172.21.15.114:0/3725372077 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1191828212 0 0) 0x7fabf0005730 con 0x7fac0c0cf370 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.563+0000 7fac0a800640 1 -- 172.21.15.114:0/3725372077 >> v1:172.21.15.114:6789/0 conn(0x7fac0c0cf370 legacy=0x7fac0c06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.563+0000 7fac0a800640 1 -- 172.21.15.114:0/3725372077 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c06ca50 con 0x7fac0c0cc2a0 2024-03-06T07:34:01.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.564+0000 7fac0a800640 1 --2- 172.21.15.114:0/3725372077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c0cc2a0 0x7fac0c0cc720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fac04009d90 tx=0x7fac0402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88dd7346e202ff75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.564+0000 7fac08a00640 1 -- 172.21.15.114:0/3725372077 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac04039070 con 0x7fac0c0cc2a0 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.564+0000 7fac08a00640 1 -- 172.21.15.114:0/3725372077 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0402c940 con 0x7fac0c0cc2a0 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.564+0000 7fac08a00640 1 -- 172.21.15.114:0/3725372077 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8001670 con 0x7fac0c0cc2a0 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.565+0000 7fac0b200640 1 -- 172.21.15.114:0/3725372077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c0cc2a0 msgr2=0x7fac0c0cc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.565+0000 7fac0b200640 1 --2- 172.21.15.114:0/3725372077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c0cc2a0 0x7fac0c0cc720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fac04009d90 tx=0x7fac0402be10 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.565+0000 7fac0b200640 1 -- 172.21.15.114:0/3725372077 shutdown_connections 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.565+0000 7fac0b200640 1 --2- 172.21.15.114:0/3725372077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c0cc2a0 0x7fac0c0cc720 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.565+0000 7fac0b200640 1 -- 172.21.15.114:0/3725372077 >> 172.21.15.114:0/3725372077 conn(0x7fac0c06afc0 msgr2=0x7fac0c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.565+0000 7fac0b200640 1 -- 172.21.15.114:0/3725372077 shutdown_connections 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.565+0000 7fac0b200640 1 -- 172.21.15.114:0/3725372077 wait complete. 2024-03-06T07:34:01.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.567+0000 7fac0b200640 1 Processor -- start 2024-03-06T07:34:01.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.567+0000 7fac0b200640 1 -- start start 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.567+0000 7fac0b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac0c0cc2a0 0x7fac0c153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac0b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c0cf370 0x7fac0c15b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac0b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c15a940 0x7fac0c15d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac0b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c15ad20 con 0x7fac0c0cf370 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac0b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0b30f0 con 0x7fac0c15a940 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac0b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c15aea0 con 0x7fac0c0cc2a0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c0cf370 0x7fac0c15b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c0cf370 0x7fac0c15b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41990/0 (socket says 172.21.15.114:41990) 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 -- 172.21.15.114:0/2633473351 learned_addr learned my addr 172.21.15.114:0/2633473351 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09e00640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac0c0cc2a0 0x7fac0c153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac0a800640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c15a940 0x7fac0c15d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 -- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac0c0cc2a0 msgr2=0x7fac0c153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac0c0cc2a0 0x7fac0c153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 -- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c15a940 msgr2=0x7fac0c15d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c15a940 0x7fac0c15d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.568+0000 7fac09400640 1 -- 172.21.15.114:0/2633473351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c0afbc0 con 0x7fac0c0cf370 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.569+0000 7fac09400640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c0cf370 0x7fac0c15b020 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fabf800c6f0 tx=0x7fabf800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.569+0000 7fac02000640 1 -- 172.21.15.114:0/2633473351 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8005ad0 con 0x7fac0c0cf370 2024-03-06T07:34:01.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.569+0000 7fac02000640 1 -- 172.21.15.114:0/2633473351 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8005c70 con 0x7fac0c0cf370 2024-03-06T07:34:01.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.569+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0c0afea0 con 0x7fac0c0cf370 2024-03-06T07:34:01.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.569+0000 7fac02000640 1 -- 172.21.15.114:0/2633473351 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8007270 con 0x7fac0c0cf370 2024-03-06T07:34:01.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.569+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c0b0460 con 0x7fac0c0cf370 2024-03-06T07:34:01.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.570+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c04f330 con 0x7fac0c0cf370 2024-03-06T07:34:01.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.571+0000 7fac02000640 1 -- 172.21.15.114:0/2633473351 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabf8007410 con 0x7fac0c0cf370 2024-03-06T07:34:01.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.572+0000 7fac02000640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabec06b4d0 0x7fabec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.572+0000 7fac02000640 1 -- 172.21.15.114:0/2633473351 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fabf8089cc0 con 0x7fac0c0cf370 2024-03-06T07:34:01.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.572+0000 7fac09e00640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabec06b4d0 0x7fabec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.573+0000 7fac09e00640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabec06b4d0 0x7fabec06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fac0c156800 tx=0x7fabfc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.576+0000 7fac02000640 1 -- 172.21.15.114:0/2633473351 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabf8090080 con 0x7fac0c0cf370 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.799+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fac0c157cf0 con 0x7fac0c0cf370 2024-03-06T07:34:01.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.894+0000 7fac02000640 1 -- 172.21.15.114:0/2633473351 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v102) v1 ==== 97+0+0 (secure 0 0 0) 0x7fabf8058730 con 0x7fac0c0cf370 2024-03-06T07:34:01.896 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:34:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.897+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabec06b4d0 msgr2=0x7fabec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.897+0000 7fac0b200640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabec06b4d0 0x7fabec06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fac0c156800 tx=0x7fabfc009040 comp rx=0 tx=0).stop 2024-03-06T07:34:01.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.897+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c0cf370 msgr2=0x7fac0c15b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.897+0000 7fac0b200640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c0cf370 0x7fac0c15b020 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fabf800c6f0 tx=0x7fabf800cbb0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 shutdown_connections 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac0c15a940 0x7fac0c15d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fabec06b4d0 0x7fabec06d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c0cf370 0x7fac0c15b020 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 --2- 172.21.15.114:0/2633473351 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac0c0cc2a0 0x7fac0c153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 >> 172.21.15.114:0/2633473351 conn(0x7fac0c06afc0 msgr2=0x7fac0c15ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 shutdown_connections 2024-03-06T07:34:01.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:01.898+0000 7fac0b200640 1 -- 172.21.15.114:0/2633473351 wait complete. 2024-03-06T07:34:01.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:34:02.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.108+0000 7f8297e00640 1 Processor -- start 2024-03-06T07:34:02.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.108+0000 7f8297e00640 1 -- start start 2024-03-06T07:34:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.109+0000 7f8297e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82980aedc0 0x7f82980cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.109+0000 7f8297e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980cc800 0x7f829815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.109+0000 7f8297e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829815c240 con 0x7f82981561f0 2024-03-06T07:34:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.109+0000 7f8297e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829815ce30 con 0x7f82980aedc0 2024-03-06T07:34:02.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.109+0000 7f8297e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829815c500 con 0x7f82980cc800 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82980aedc0 0x7f82980cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f82981561f0 0x7f82980ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50530/0 (socket says 172.21.15.114:50530) 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82980aedc0 0x7f82980cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42000/0 (socket says 172.21.15.114:42000) 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296a00640 1 -- 172.21.15.114:0/3924353456 learned_addr learned my addr 172.21.15.114:0/3924353456 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8297400640 1 --2- 172.21.15.114:0/3924353456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980cc800 0x7f829815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296000640 1 -- 172.21.15.114:0/3924353456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980cc800 msgr2=0x7f829815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296000640 1 --2- 172.21.15.114:0/3924353456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980cc800 0x7f829815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296000640 1 -- 172.21.15.114:0/3924353456 >> v1:172.21.15.114:6789/0 conn(0x7f82981561f0 legacy=0x7f82980ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8296000640 1 -- 172.21.15.114:0/3924353456 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829815c7e0 con 0x7f82980aedc0 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.110+0000 7f8295600640 1 -- 172.21.15.114:0/3924353456 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551001792 0 0) 0x7f829815c240 con 0x7f82981561f0 2024-03-06T07:34:02.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.111+0000 7f8297400640 1 --2- 172.21.15.114:0/3924353456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980cc800 0x7f829815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8296000640 1 --2- 172.21.15.114:0/3924353456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82980aedc0 0x7f82980cc2c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f82840097a0 tx=0x7f828402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2267fdeda90095be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8295600640 1 -- 172.21.15.114:0/3924353456 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284035020 con 0x7f82980aedc0 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8295600640 1 -- 172.21.15.114:0/3924353456 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284031810 con 0x7f82980aedc0 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8295600640 1 -- 172.21.15.114:0/3924353456 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284031be0 con 0x7f82980aedc0 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8297e00640 1 -- 172.21.15.114:0/3924353456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82980aedc0 msgr2=0x7f82980cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8297e00640 1 --2- 172.21.15.114:0/3924353456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82980aedc0 0x7f82980cc2c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f82840097a0 tx=0x7f828402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8297e00640 1 -- 172.21.15.114:0/3924353456 shutdown_connections 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8297e00640 1 --2- 172.21.15.114:0/3924353456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980cc800 0x7f829815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8297e00640 1 --2- 172.21.15.114:0/3924353456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82980aedc0 0x7f82980cc2c0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.112+0000 7f8297e00640 1 -- 172.21.15.114:0/3924353456 >> 172.21.15.114:0/3924353456 conn(0x7f829806afc0 msgr2=0x7f829806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.113+0000 7f8297e00640 1 -- 172.21.15.114:0/3924353456 shutdown_connections 2024-03-06T07:34:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.113+0000 7f8297e00640 1 -- 172.21.15.114:0/3924353456 wait complete. 2024-03-06T07:34:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.114+0000 7f8297e00640 1 Processor -- start 2024-03-06T07:34:02.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.114+0000 7f8297e00640 1 -- start start 2024-03-06T07:34:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980aedc0 0x7f82980af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82980cc800 0x7f8298152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82981561f0 0x7f8298152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82981565d0 con 0x7f82981561f0 2024-03-06T07:34:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82980ccc70 con 0x7f82980cc800 2024-03-06T07:34:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8298156750 con 0x7f82980aedc0 2024-03-06T07:34:02.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82981561f0 0x7f8298152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.115+0000 7f8297400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82981561f0 0x7f8298152b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42010/0 (socket says 172.21.15.114:42010) 2024-03-06T07:34:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8297400640 1 -- 172.21.15.114:0/2660377683 learned_addr learned my addr 172.21.15.114:0/2660377683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8296a00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980aedc0 0x7f82980af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8296000640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82980cc800 0x7f8298152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8297400640 1 -- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980aedc0 msgr2=0x7f82980af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8297400640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980aedc0 0x7f82980af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8297400640 1 -- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82980cc800 msgr2=0x7f8298152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8297400640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82980cc800 0x7f8298152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8297400640 1 -- 172.21.15.114:0/2660377683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829815d480 con 0x7f82981561f0 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8296a00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980aedc0 0x7f82980af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8296000640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82980cc800 0x7f8298152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f8297400640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82981561f0 0x7f8298152b70 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f829000cef0 tx=0x7f829000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f828ea00640 1 -- 172.21.15.114:0/2660377683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829001a070 con 0x7f82981561f0 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f828ea00640 1 -- 172.21.15.114:0/2660377683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829000d860 con 0x7f82981561f0 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.116+0000 7f828ea00640 1 -- 172.21.15.114:0/2660377683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000dc50 con 0x7f82981561f0 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.117+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829815d6f0 con 0x7f82981561f0 2024-03-06T07:34:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.117+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829815dcb0 con 0x7f82981561f0 2024-03-06T07:34:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.118+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829804f330 con 0x7f82981561f0 2024-03-06T07:34:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.119+0000 7f828ea00640 1 -- 172.21.15.114:0/2660377683 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f829001d070 con 0x7f82981561f0 2024-03-06T07:34:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.119+0000 7f828ea00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.120+0000 7f828ea00640 1 -- 172.21.15.114:0/2660377683 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f829008a820 con 0x7f82981561f0 2024-03-06T07:34:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.120+0000 7f8296a00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.120+0000 7f8296a00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f827806b4d0 0x7f827806d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8288000f60 tx=0x7f8288009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.124+0000 7f828ea00640 1 -- 172.21.15.114:0/2660377683 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8290059a10 con 0x7f82981561f0 2024-03-06T07:34:02.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.389+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f829815de80 con 0x7f82981561f0 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.390+0000 7f828ea00640 1 -- 172.21.15.114:0/2660377683 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8290059310 con 0x7f82981561f0 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:34:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:34:02.394 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.395 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.396 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.397 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.397 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.397 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.397 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow r", 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.408 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:02.408 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:34:02.408 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.392+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f827806b4d0 msgr2=0x7f827806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f827806b4d0 0x7f827806d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8288000f60 tx=0x7f8288009040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82981561f0 msgr2=0x7f8298152b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82981561f0 0x7f8298152b70 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f829000cef0 tx=0x7f829000a710 comp rx=0 tx=0).stop 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 shutdown_connections 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f827806b4d0 0x7f827806d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82981561f0 0x7f8298152b70 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82980cc800 0x7f8298152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 --2- 172.21.15.114:0/2660377683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82980aedc0 0x7f82980af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 >> 172.21.15.114:0/2660377683 conn(0x7f829806afc0 msgr2=0x7f8298154c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 shutdown_connections 2024-03-06T07:34:02.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.393+0000 7f8297e00640 1 -- 172.21.15.114:0/2660377683 wait complete. 2024-03-06T07:34:02.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:02.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.610+0000 7ff643600640 1 Processor -- start 2024-03-06T07:34:02.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.610+0000 7ff643600640 1 -- start start 2024-03-06T07:34:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff643600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff63c15f9d0 0x7ff63c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff643600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff63c070b10 0x7ff63c070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff643600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c06cef0 con 0x7ff63c144c00 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff643600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c06d1d0 con 0x7ff63c070b10 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff643600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c06d4b0 con 0x7ff63c15f9d0 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff641600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff63c070b10 0x7ff63c070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff641600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff63c070b10 0x7ff63c070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42012/0 (socket says 172.21.15.114:42012) 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff641600640 1 -- 172.21.15.114:0/1976078673 learned_addr learned my addr 172.21.15.114:0/1976078673 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff641600640 1 -- 172.21.15.114:0/1976078673 >> v2:172.21.15.177:3300/0 conn(0x7ff63c15f9d0 msgr2=0x7ff63c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff633e00640 1 --2- 172.21.15.114:0/1976078673 >> v2:172.21.15.177:3300/0 conn(0x7ff63c15f9d0 0x7ff63c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff641600640 1 --2- 172.21.15.114:0/1976078673 >> v2:172.21.15.177:3300/0 conn(0x7ff63c15f9d0 0x7ff63c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff641600640 1 -- 172.21.15.114:0/1976078673 >> v1:172.21.15.177:6789/0 conn(0x7ff63c144c00 legacy=0x7ff63c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.612+0000 7ff641600640 1 -- 172.21.15.114:0/1976078673 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c06d790 con 0x7ff63c070b10 2024-03-06T07:34:02.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.613+0000 7ff633e00640 1 --2- 172.21.15.114:0/1976078673 >> v2:172.21.15.177:3300/0 conn(0x7ff63c15f9d0 0x7ff63c15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:02.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.614+0000 7ff641600640 1 --2- 172.21.15.114:0/1976078673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c070f20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0b26f0 tx=0x7ff63402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88075d651724212 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.614+0000 7ff63be00640 1 -- 172.21.15.114:0/1976078673 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634035020 con 0x7ff63c070b10 2024-03-06T07:34:02.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.614+0000 7ff63be00640 1 -- 172.21.15.114:0/1976078673 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff634031870 con 0x7ff63c070b10 2024-03-06T07:34:02.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.614+0000 7ff63be00640 1 -- 172.21.15.114:0/1976078673 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634031c40 con 0x7ff63c070b10 2024-03-06T07:34:02.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.614+0000 7ff643600640 1 -- 172.21.15.114:0/1976078673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 msgr2=0x7ff63c070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.614+0000 7ff643600640 1 --2- 172.21.15.114:0/1976078673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c070f20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0b26f0 tx=0x7ff63402b910 comp rx=0 tx=0).stop 2024-03-06T07:34:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.614+0000 7ff643600640 1 -- 172.21.15.114:0/1976078673 shutdown_connections 2024-03-06T07:34:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.615+0000 7ff643600640 1 --2- 172.21.15.114:0/1976078673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c070f20 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.615+0000 7ff643600640 1 --2- 172.21.15.114:0/1976078673 >> v2:172.21.15.177:3300/0 conn(0x7ff63c15f9d0 0x7ff63c15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.615+0000 7ff643600640 1 -- 172.21.15.114:0/1976078673 >> 172.21.15.114:0/1976078673 conn(0x7ff63c0b1730 msgr2=0x7ff63c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.615+0000 7ff643600640 1 -- 172.21.15.114:0/1976078673 shutdown_connections 2024-03-06T07:34:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.615+0000 7ff643600640 1 -- 172.21.15.114:0/1976078673 wait complete. 2024-03-06T07:34:02.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.616+0000 7ff643600640 1 Processor -- start 2024-03-06T07:34:02.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.616+0000 7ff643600640 1 -- start start 2024-03-06T07:34:02.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.617+0000 7ff643600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c13b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.617+0000 7ff643600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff63c144c00 0x7ff63c142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.617+0000 7ff643600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff63c15f9d0 0x7ff63c1431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.617+0000 7ff643600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c06ccd0 con 0x7ff63c070b10 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.617+0000 7ff643600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c06cb50 con 0x7ff63c15f9d0 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.617+0000 7ff643600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c06df90 con 0x7ff63c144c00 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.617+0000 7ff640c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c13b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c13b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42026/0 (socket says 172.21.15.114:42026) 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 -- 172.21.15.114:0/1753257621 learned_addr learned my addr 172.21.15.114:0/1753257621 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff633e00640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff63c144c00 0x7ff63c142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 -- 172.21.15.114:0/1753257621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff63c144c00 msgr2=0x7ff63c142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff63c144c00 0x7ff63c142c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 -- 172.21.15.114:0/1753257621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff63c15f9d0 msgr2=0x7ff63c1431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff63c15f9d0 0x7ff63c1431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 -- 172.21.15.114:0/1753257621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c13ba40 con 0x7ff63c070b10 2024-03-06T07:34:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff640c00640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c13b500 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00c7a0 tx=0x7ff62c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.618+0000 7ff639600640 1 -- 172.21.15.114:0/1753257621 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c017070 con 0x7ff63c070b10 2024-03-06T07:34:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.619+0000 7ff639600640 1 -- 172.21.15.114:0/1753257621 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c003da0 con 0x7ff63c070b10 2024-03-06T07:34:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.619+0000 7ff639600640 1 -- 172.21.15.114:0/1753257621 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0052b0 con 0x7ff63c070b10 2024-03-06T07:34:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.619+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c13c080 con 0x7ff63c070b10 2024-03-06T07:34:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.619+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c13c640 con 0x7ff63c070b10 2024-03-06T07:34:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.620+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c069050 con 0x7ff63c070b10 2024-03-06T07:34:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.621+0000 7ff639600640 1 -- 172.21.15.114:0/1753257621 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff62c005450 con 0x7ff63c070b10 2024-03-06T07:34:02.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.621+0000 7ff639600640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff61806b4d0 0x7ff61806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.622+0000 7ff639600640 1 -- 172.21.15.114:0/1753257621 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff62c0894e0 con 0x7ff63c070b10 2024-03-06T07:34:02.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.622+0000 7ff633e00640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff61806b4d0 0x7ff61806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.622+0000 7ff633e00640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff61806b4d0 0x7ff61806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff63c15f340 tx=0x7ff624008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.626+0000 7ff639600640 1 -- 172.21.15.114:0/1753257621 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff63c069050 con 0x7ff63c070b10 2024-03-06T07:34:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.819+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff63c072890 con 0x7ff63c070b10 2024-03-06T07:34:02.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.820+0000 7ff639600640 1 -- 172.21.15.114:0/1753257621 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12941 (secure 0 0 0) 0x7ff62c058610 con 0x7ff63c070b10 2024-03-06T07:34:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:02.822 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:01.909710+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.822+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff61806b4d0 msgr2=0x7ff61806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.822+0000 7ff643600640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff61806b4d0 0x7ff61806d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff63c15f340 tx=0x7ff624008040 comp rx=0 tx=0).stop 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.822+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 msgr2=0x7ff63c13b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.822+0000 7ff643600640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c13b500 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00c7a0 tx=0x7ff62c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 shutdown_connections 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff63c15f9d0 0x7ff63c1431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff63c144c00 0x7ff63c142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff61806b4d0 0x7ff61806d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 --2- 172.21.15.114:0/1753257621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63c070b10 0x7ff63c13b500 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 >> 172.21.15.114:0/1753257621 conn(0x7ff63c0b1730 msgr2=0x7ff63c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 shutdown_connections 2024-03-06T07:34:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:02.823+0000 7ff643600640 1 -- 172.21.15.114:0/1753257621 wait complete. 2024-03-06T07:34:02.843 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:02.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:03.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.049+0000 7f3f72600640 1 Processor -- start 2024-03-06T07:34:03.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.049+0000 7f3f72600640 1 -- start start 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f72600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fc10 0x7f3f6c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f72600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3f6c15fff0 0x7f3f6c160450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f72600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3f6c0ce4a0 0x7f3f6c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f72600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c0b46c0 con 0x7f3f6c15fff0 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f72600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c0b49a0 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f72600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c072c90 con 0x7f3f6c15fc10 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f63e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3f6c15fff0 0x7f3f6c160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f63e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3f6c15fff0 0x7f3f6c160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42040/0 (socket says 172.21.15.114:42040) 2024-03-06T07:34:03.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.051+0000 7f3f63e00640 1 -- 172.21.15.114:0/2787382450 learned_addr learned my addr 172.21.15.114:0/2787382450 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.052+0000 7f3f6b400640 1 --2- 172.21.15.114:0/2787382450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fc10 0x7f3f6c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.052+0000 7f3f6be00640 1 --2- 172.21.15.114:0/2787382450 >> v2:172.21.15.177:3300/0 conn(0x7f3f6c0ce4a0 0x7f3f6c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.052+0000 7f3f63e00640 1 -- 172.21.15.114:0/2787382450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fc10 msgr2=0x7f3f6c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.052+0000 7f3f63e00640 1 --2- 172.21.15.114:0/2787382450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fc10 0x7f3f6c070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.052+0000 7f3f63e00640 1 -- 172.21.15.114:0/2787382450 >> v2:172.21.15.177:3300/0 conn(0x7f3f6c0ce4a0 msgr2=0x7f3f6c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.052+0000 7f3f63e00640 1 --2- 172.21.15.114:0/2787382450 >> v2:172.21.15.177:3300/0 conn(0x7f3f6c0ce4a0 0x7f3f6c0ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.052+0000 7f3f63e00640 1 -- 172.21.15.114:0/2787382450 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c160990 con 0x7f3f6c15fff0 2024-03-06T07:34:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.053+0000 7f3f63e00640 1 --2- 172.21.15.114:0/2787382450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c15fff0 0x7f3f6c160450 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c15fb70 tx=0x7f3f5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270128ee5e484a1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.053+0000 7f3f6aa00640 1 -- 172.21.15.114:0/2787382450 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54035020 con 0x7f3f6c15fff0 2024-03-06T07:34:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.053+0000 7f3f6aa00640 1 -- 172.21.15.114:0/2787382450 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54031870 con 0x7f3f6c15fff0 2024-03-06T07:34:03.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.053+0000 7f3f6aa00640 1 -- 172.21.15.114:0/2787382450 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54031c40 con 0x7f3f6c15fff0 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 -- 172.21.15.114:0/2787382450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c15fff0 msgr2=0x7f3f6c160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 --2- 172.21.15.114:0/2787382450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c15fff0 0x7f3f6c160450 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c15fb70 tx=0x7f3f5402b910 comp rx=0 tx=0).stop 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 -- 172.21.15.114:0/2787382450 shutdown_connections 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 --2- 172.21.15.114:0/2787382450 >> v2:172.21.15.177:3300/0 conn(0x7f3f6c0ce4a0 0x7f3f6c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 --2- 172.21.15.114:0/2787382450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c15fff0 0x7f3f6c160450 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 --2- 172.21.15.114:0/2787382450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fc10 0x7f3f6c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 -- 172.21.15.114:0/2787382450 >> 172.21.15.114:0/2787382450 conn(0x7f3f6c0b1940 msgr2=0x7f3f6c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 -- 172.21.15.114:0/2787382450 shutdown_connections 2024-03-06T07:34:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.054+0000 7f3f72600640 1 -- 172.21.15.114:0/2787382450 wait complete. 2024-03-06T07:34:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.056+0000 7f3f72600640 1 Processor -- start 2024-03-06T07:34:03.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.056+0000 7f3f72600640 1 -- start start 2024-03-06T07:34:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f72600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c0ce4a0 0x7f3f6c158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f72600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f6c15fc10 0x7f3f6c0ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f72600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fff0 0x7f3f6c09b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f72600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c072a30 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f72600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0728b0 con 0x7f3f6c15fc10 2024-03-06T07:34:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f72600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c15c2e0 con 0x7f3f6c15fff0 2024-03-06T07:34:03.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c0ce4a0 0x7f3f6c158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c0ce4a0 0x7f3f6c158230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42048/0 (socket says 172.21.15.114:42048) 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 -- 172.21.15.114:0/3654631602 learned_addr learned my addr 172.21.15.114:0/3654631602 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 -- 172.21.15.114:0/3654631602 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fff0 msgr2=0x7f3f6c09b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fff0 0x7f3f6c09b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 -- 172.21.15.114:0/3654631602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f6c15fc10 msgr2=0x7f3f6c0ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f6c15fc10 0x7f3f6c0ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.057+0000 7f3f6b400640 1 -- 172.21.15.114:0/3654631602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c06ce00 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.058+0000 7f3f6b400640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c0ce4a0 0x7f3f6c158230 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00c7f0 tx=0x7f3f5c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.058+0000 7f3f63400640 1 -- 172.21.15.114:0/3654631602 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c015070 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.058+0000 7f3f63400640 1 -- 172.21.15.114:0/3654631602 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c004240 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.058+0000 7f3f63400640 1 -- 172.21.15.114:0/3654631602 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c007620 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.058+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c06d0e0 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.058+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c06d980 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.059+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c04f330 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.060+0000 7f3f63400640 1 -- 172.21.15.114:0/3654631602 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f5c0079d0 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.061+0000 7f3f63400640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3f4806b500 0x7f3f4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.061+0000 7f3f63400640 1 -- 172.21.15.114:0/3654631602 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3f5c089fc0 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.061+0000 7f3f63e00640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3f4806b500 0x7f3f4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.062+0000 7f3f63e00640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3f4806b500 0x7f3f4806d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3f54036000 tx=0x7f3f54030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.065+0000 7f3f63400640 1 -- 172.21.15.114:0/3654631602 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f5c059180 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.262+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f6c06ef30 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.262+0000 7f3f63400640 1 -- 172.21.15.114:0/3654631602 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f5c058a80 con 0x7f3f6c0ce4a0 2024-03-06T07:34:03.264 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:34:03.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3f4806b500 msgr2=0x7f3f4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3f4806b500 0x7f3f4806d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3f54036000 tx=0x7f3f54030000 comp rx=0 tx=0).stop 2024-03-06T07:34:03.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c0ce4a0 msgr2=0x7f3f6c158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c0ce4a0 0x7f3f6c158230 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00c7f0 tx=0x7f3f5c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 shutdown_connections 2024-03-06T07:34:03.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3f6c15fff0 0x7f3f6c09b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f6c15fc10 0x7f3f6c0ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3f4806b500 0x7f3f4806d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 --2- 172.21.15.114:0/3654631602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f6c0ce4a0 0x7f3f6c158230 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 >> 172.21.15.114:0/3654631602 conn(0x7f3f6c0b1940 msgr2=0x7f3f6c156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.265+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 shutdown_connections 2024-03-06T07:34:03.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.266+0000 7f3f72600640 1 -- 172.21.15.114:0/3654631602 wait complete. 2024-03-06T07:34:03.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:34:03.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.493+0000 7fc1dbc00640 1 Processor -- start 2024-03-06T07:34:03.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.493+0000 7fc1dbc00640 1 -- start start 2024-03-06T07:34:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.494+0000 7fc1dbc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d40b4680 0x7fc1d406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.494+0000 7fc1dbc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc1d4072340 0x7fc1d406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.494+0000 7fc1dbc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d415b090 con 0x7fc1d415acb0 2024-03-06T07:34:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.494+0000 7fc1dbc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d40b2a30 con 0x7fc1d4072340 2024-03-06T07:34:03.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.494+0000 7fc1dbc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40b4360 con 0x7fc1d40b4680 2024-03-06T07:34:03.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.495+0000 7fc1d9c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc1d4072340 0x7fc1d406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.495+0000 7fc1cbe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d40b4680 0x7fc1d406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.495+0000 7fc1d9c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc1d4072340 0x7fc1d406ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:54648/0 (socket says 172.21.15.114:54648) 2024-03-06T07:34:03.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.495+0000 7fc1d9c00640 1 -- 172.21.15.114:0/2972850431 learned_addr learned my addr 172.21.15.114:0/2972850431 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:03.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.495+0000 7fc1d9200640 1 --1- 172.21.15.114:0/2972850431 >> v1:172.21.15.177:6789/0 conn(0x7fc1d415acb0 0x7fc1d40b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:46946/0 (socket says 172.21.15.114:46946) 2024-03-06T07:34:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.496+0000 7fc1d3e00640 1 -- 172.21.15.114:0/2972850431 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263231182 0 0) 0x7fc1d415b090 con 0x7fc1d415acb0 2024-03-06T07:34:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.496+0000 7fc1d3e00640 1 -- 172.21.15.114:0/2972850431 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b8003520 con 0x7fc1d415acb0 2024-03-06T07:34:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.496+0000 7fc1cbe00640 1 -- 172.21.15.114:0/2972850431 >> v2:172.21.15.177:3300/0 conn(0x7fc1d4072340 msgr2=0x7fc1d406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.496+0000 7fc1cbe00640 1 --2- 172.21.15.114:0/2972850431 >> v2:172.21.15.177:3300/0 conn(0x7fc1d4072340 0x7fc1d406ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.496+0000 7fc1cbe00640 1 -- 172.21.15.114:0/2972850431 >> v1:172.21.15.177:6789/0 conn(0x7fc1d415acb0 legacy=0x7fc1d40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.496+0000 7fc1cbe00640 1 -- 172.21.15.114:0/2972850431 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40b2cf0 con 0x7fc1d40b4680 2024-03-06T07:34:03.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.498+0000 7fc1cbe00640 1 --2- 172.21.15.114:0/2972850431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d40b4680 0x7fc1d406e4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0097a0 tx=0x7fc1bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64ee2fca8f501fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.498+0000 7fc1d3e00640 1 -- 172.21.15.114:0/2972850431 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc035020 con 0x7fc1d40b4680 2024-03-06T07:34:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.498+0000 7fc1d3e00640 1 -- 172.21.15.114:0/2972850431 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc031830 con 0x7fc1d40b4680 2024-03-06T07:34:03.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.498+0000 7fc1d3e00640 1 -- 172.21.15.114:0/2972850431 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc031c00 con 0x7fc1d40b4680 2024-03-06T07:34:03.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 -- 172.21.15.114:0/2972850431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d40b4680 msgr2=0x7fc1d406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/2972850431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d40b4680 0x7fc1d406e4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0097a0 tx=0x7fc1bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 -- 172.21.15.114:0/2972850431 shutdown_connections 2024-03-06T07:34:03.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/2972850431 >> v2:172.21.15.177:3300/0 conn(0x7fc1d4072340 0x7fc1d406ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/2972850431 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d40b4680 0x7fc1d406e4f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 -- 172.21.15.114:0/2972850431 >> 172.21.15.114:0/2972850431 conn(0x7fc1d40b1730 msgr2=0x7fc1d40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 -- 172.21.15.114:0/2972850431 shutdown_connections 2024-03-06T07:34:03.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.499+0000 7fc1dbc00640 1 -- 172.21.15.114:0/2972850431 wait complete. 2024-03-06T07:34:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.501+0000 7fc1dbc00640 1 Processor -- start 2024-03-06T07:34:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.501+0000 7fc1dbc00640 1 -- start start 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.502+0000 7fc1dbc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1d4072340 0x7fc1d414e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.502+0000 7fc1dbc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1d40b4680 0x7fc1d414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.502+0000 7fc1dbc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d415acb0 0x7fc1d4152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.502+0000 7fc1dbc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40b4010 con 0x7fc1d4072340 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.502+0000 7fc1dbc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40b32b0 con 0x7fc1d40b4680 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.502+0000 7fc1dbc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40ccfb0 con 0x7fc1d415acb0 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1d4072340 0x7fc1d414e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1d4072340 0x7fc1d414e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42054/0 (socket says 172.21.15.114:42054) 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 -- 172.21.15.114:0/3144262112 learned_addr learned my addr 172.21.15.114:0/3144262112 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1cbe00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1d40b4680 0x7fc1d414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9c00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d415acb0 0x7fc1d4152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 -- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d415acb0 msgr2=0x7fc1d4152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d415acb0 0x7fc1d4152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 -- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1d40b4680 msgr2=0x7fc1d414fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1d40b4680 0x7fc1d414fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.503+0000 7fc1d9200640 1 -- 172.21.15.114:0/3144262112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d414ebe0 con 0x7fc1d4072340 2024-03-06T07:34:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.504+0000 7fc1d9200640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1d4072340 0x7fc1d414e570 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c3c0 tx=0x7fc1c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.504+0000 7fc1d1600640 1 -- 172.21.15.114:0/3144262112 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40074e0 con 0x7fc1d4072340 2024-03-06T07:34:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.504+0000 7fc1d1600640 1 -- 172.21.15.114:0/3144262112 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c40047e0 con 0x7fc1d4072340 2024-03-06T07:34:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.504+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d414eec0 con 0x7fc1d4072340 2024-03-06T07:34:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.504+0000 7fc1d1600640 1 -- 172.21.15.114:0/3144262112 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4004bd0 con 0x7fc1d4072340 2024-03-06T07:34:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.504+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d414f480 con 0x7fc1d4072340 2024-03-06T07:34:03.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.505+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d404f330 con 0x7fc1d4072340 2024-03-06T07:34:03.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.506+0000 7fc1d1600640 1 -- 172.21.15.114:0/3144262112 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1c401a020 con 0x7fc1d4072340 2024-03-06T07:34:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.507+0000 7fc1d1600640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1b006b4d0 0x7fc1b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.507+0000 7fc1d1600640 1 -- 172.21.15.114:0/3144262112 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc1c4089740 con 0x7fc1d4072340 2024-03-06T07:34:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.507+0000 7fc1cbe00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1b006b4d0 0x7fc1b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.508+0000 7fc1cbe00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1b006b4d0 0x7fc1b006d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc036000 tx=0x7fc1bc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.511+0000 7fc1d1600640 1 -- 172.21.15.114:0/3144262112 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1c4010300 con 0x7fc1d4072340 2024-03-06T07:34:03.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.710+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc1d406ce60 con 0x7fc1d4072340 2024-03-06T07:34:03.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.711+0000 7fc1d1600640 1 -- 172.21.15.114:0/3144262112 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc1c4058940 con 0x7fc1d4072340 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:03.713 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1b006b4d0 msgr2=0x7fc1b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1b006b4d0 0x7fc1b006d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc036000 tx=0x7fc1bc008030 comp rx=0 tx=0).stop 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1d4072340 msgr2=0x7fc1d414e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1d4072340 0x7fc1d414e570 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c3c0 tx=0x7fc1c400c880 comp rx=0 tx=0).stop 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 shutdown_connections 2024-03-06T07:34:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1d415acb0 0x7fc1d4152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1d40b4680 0x7fc1d414fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1b006b4d0 0x7fc1b006d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 --2- 172.21.15.114:0/3144262112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1d4072340 0x7fc1d414e570 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 >> 172.21.15.114:0/3144262112 conn(0x7fc1d40b1730 msgr2=0x7fc1d4071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 shutdown_connections 2024-03-06T07:34:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.713+0000 7fc1dbc00640 1 -- 172.21.15.114:0/3144262112 wait complete. 2024-03-06T07:34:03.732 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:34:03.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:34:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.938+0000 7f3162400640 1 Processor -- start 2024-03-06T07:34:03.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.938+0000 7f3162400640 1 -- start start 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f3162400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f315c145980 0x7f315c15fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f3162400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f315c0ced20 0x7f315c15ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f3162400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c0cf1e0 con 0x7f315c0712d0 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f3162400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c070e40 con 0x7f315c145980 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f3162400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c0cf4c0 con 0x7f315c0ced20 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f315b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f315c145980 0x7f315c15fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f315be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f315c0712d0 0x7f315c0716e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50532/0 (socket says 172.21.15.114:50532) 2024-03-06T07:34:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.940+0000 7f315be00640 1 -- 172.21.15.114:0/4271941334 learned_addr learned my addr 172.21.15.114:0/4271941334 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.941+0000 7f315b400640 1 -- 172.21.15.114:0/4271941334 >> v2:172.21.15.177:3300/0 conn(0x7f315c0ced20 msgr2=0x7f315c15ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.941+0000 7f315aa00640 1 --2- 172.21.15.114:0/4271941334 >> v2:172.21.15.177:3300/0 conn(0x7f315c0ced20 0x7f315c15ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.941+0000 7f315b400640 1 --2- 172.21.15.114:0/4271941334 >> v2:172.21.15.177:3300/0 conn(0x7f315c0ced20 0x7f315c15ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.941+0000 7f315b400640 1 -- 172.21.15.114:0/4271941334 >> v1:172.21.15.114:6789/0 conn(0x7f315c0712d0 legacy=0x7f315c0716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.941+0000 7f315a000640 1 -- 172.21.15.114:0/4271941334 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416781008 0 0) 0x7f315c0cf1e0 con 0x7f315c0712d0 2024-03-06T07:34:03.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.941+0000 7f315b400640 1 -- 172.21.15.114:0/4271941334 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0cf7a0 con 0x7f315c145980 2024-03-06T07:34:03.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.942+0000 7f315b400640 1 --2- 172.21.15.114:0/4271941334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c145980 0x7f315c15fa00 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f315c070a60 tx=0x7f314c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7da3a9a5114c9f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.942+0000 7f315a000640 1 -- 172.21.15.114:0/4271941334 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c035020 con 0x7f315c145980 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.942+0000 7f315a000640 1 -- 172.21.15.114:0/4271941334 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c031830 con 0x7f315c145980 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.942+0000 7f315a000640 1 -- 172.21.15.114:0/4271941334 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c031c00 con 0x7f315c145980 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 -- 172.21.15.114:0/4271941334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c145980 msgr2=0x7f315c15fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 --2- 172.21.15.114:0/4271941334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c145980 0x7f315c15fa00 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f315c070a60 tx=0x7f314c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 -- 172.21.15.114:0/4271941334 shutdown_connections 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 --2- 172.21.15.114:0/4271941334 >> v2:172.21.15.177:3300/0 conn(0x7f315c0ced20 0x7f315c15ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 --2- 172.21.15.114:0/4271941334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c145980 0x7f315c15fa00 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 -- 172.21.15.114:0/4271941334 >> 172.21.15.114:0/4271941334 conn(0x7f315c0cfa80 msgr2=0x7f315c0cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 -- 172.21.15.114:0/4271941334 shutdown_connections 2024-03-06T07:34:03.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.943+0000 7f3162400640 1 -- 172.21.15.114:0/4271941334 wait complete. 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.945+0000 7f3162400640 1 Processor -- start 2024-03-06T07:34:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.945+0000 7f3162400640 1 -- start start 2024-03-06T07:34:03.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f3162400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c0712d0 0x7f315c1331e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f3162400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f315c0ced20 0x7f315c15d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f3162400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f315c145980 0x7f315c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f3162400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c1608c0 con 0x7f315c0712d0 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f3162400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c160740 con 0x7f315c0ced20 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f3162400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c06dec0 con 0x7f315c145980 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c0712d0 0x7f315c1331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c0712d0 0x7f315c1331e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42072/0 (socket says 172.21.15.114:42072) 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 -- 172.21.15.114:0/3835648039 learned_addr learned my addr 172.21.15.114:0/3835648039 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 -- 172.21.15.114:0/3835648039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f315c145980 msgr2=0x7f315c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f315c145980 0x7f315c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 -- 172.21.15.114:0/3835648039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f315c0ced20 msgr2=0x7f315c15d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f315c0ced20 0x7f315c15d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.946+0000 7f315b400640 1 -- 172.21.15.114:0/3835648039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c15e2a0 con 0x7f315c0712d0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.947+0000 7f315b400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c0712d0 0x7f315c1331e0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f314c009890 tx=0x7f314c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.947+0000 7f3153400640 1 -- 172.21.15.114:0/3835648039 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c040070 con 0x7f315c0712d0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.947+0000 7f3153400640 1 -- 172.21.15.114:0/3835648039 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c039a00 con 0x7f315c0712d0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.947+0000 7f3153400640 1 -- 172.21.15.114:0/3835648039 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c035040 con 0x7f315c0712d0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.947+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c15e580 con 0x7f315c0712d0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.947+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c15eb40 con 0x7f315c0712d0 2024-03-06T07:34:03.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.948+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c04f330 con 0x7f315c0712d0 2024-03-06T07:34:03.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.949+0000 7f3153400640 1 -- 172.21.15.114:0/3835648039 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f314c038020 con 0x7f315c0712d0 2024-03-06T07:34:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.950+0000 7f3153400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f313c06b4d0 0x7f313c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.950+0000 7f3153400640 1 -- 172.21.15.114:0/3835648039 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f314c0aba40 con 0x7f315c0712d0 2024-03-06T07:34:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.950+0000 7f315aa00640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f313c06b4d0 0x7f313c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.951+0000 7f315aa00640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f313c06b4d0 0x7f313c06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3148006fd0 tx=0x7f3148008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:03.954+0000 7f3153400640 1 -- 172.21.15.114:0/3835648039 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f314c02f2e0 con 0x7f315c0712d0 2024-03-06T07:34:04.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.157+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f315c072920 con 0x7f315c0712d0 2024-03-06T07:34:04.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.918+0000 7f3153400640 1 -- 172.21.15.114:0/3835648039 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7f314c07ab70 con 0x7f315c0712d0 2024-03-06T07:34:04.920 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:34:04.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.922+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f313c06b4d0 msgr2=0x7f313c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.922+0000 7f3162400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f313c06b4d0 0x7f313c06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3148006fd0 tx=0x7f3148008040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.922+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c0712d0 msgr2=0x7f315c1331e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.922+0000 7f3162400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c0712d0 0x7f315c1331e0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f314c009890 tx=0x7f314c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.923+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 shutdown_connections 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.923+0000 7f3162400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f315c145980 0x7f315c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.923+0000 7f3162400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f315c0ced20 0x7f315c15d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.923+0000 7f3162400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f313c06b4d0 0x7f313c06d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.923+0000 7f3162400640 1 --2- 172.21.15.114:0/3835648039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f315c0712d0 0x7f315c1331e0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.923+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 >> 172.21.15.114:0/3835648039 conn(0x7f315c0cfa80 msgr2=0x7f315c0b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.923+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 shutdown_connections 2024-03-06T07:34:04.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:04.924+0000 7f3162400640 1 -- 172.21.15.114:0/3835648039 wait complete. 2024-03-06T07:34:04.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:34:05.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.151+0000 7f2344400640 1 Processor -- start 2024-03-06T07:34:05.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.151+0000 7f2344400640 1 -- start start 2024-03-06T07:34:05.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.153+0000 7f2344400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c06dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.153+0000 7f2344400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f233c06afc0 0x7f233c06b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.153+0000 7f2344400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233c071ea0 con 0x7f233c15efa0 2024-03-06T07:34:05.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.153+0000 7f2344400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233c15bf70 con 0x7f233c06afc0 2024-03-06T07:34:05.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.153+0000 7f2344400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233c06e380 con 0x7f233c15bb90 2024-03-06T07:34:05.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.154+0000 7f2342400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f233c06afc0 0x7f233c06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.154+0000 7f2341a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c06dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.154+0000 7f2342400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f233c06afc0 0x7f233c06b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:54694/0 (socket says 172.21.15.114:54694) 2024-03-06T07:34:05.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.154+0000 7f2342400640 1 -- 172.21.15.114:0/3840989650 learned_addr learned my addr 172.21.15.114:0/3840989650 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:05.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.154+0000 7f2341a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c06dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58260/0 (socket says 172.21.15.114:58260) 2024-03-06T07:34:05.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.155+0000 7f2342400640 1 -- 172.21.15.114:0/3840989650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 msgr2=0x7f233c06dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:05.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.155+0000 7f2342400640 1 --2- 172.21.15.114:0/3840989650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c06dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.155+0000 7f2342400640 1 -- 172.21.15.114:0/3840989650 >> v1:172.21.15.177:6789/0 conn(0x7f233c15efa0 legacy=0x7f233c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.155+0000 7f2342400640 1 -- 172.21.15.114:0/3840989650 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c06e660 con 0x7f233c06afc0 2024-03-06T07:34:05.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.156+0000 7f2342400640 1 --2- 172.21.15.114:0/3840989650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 0x7f233c06b450 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2334009e50 tx=0x7f233402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d6b438ec494c8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.156+0000 7f233be00640 1 -- 172.21.15.114:0/3840989650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233402cb00 con 0x7f233c06afc0 2024-03-06T07:34:05.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f233be00640 1 -- 172.21.15.114:0/3840989650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233402cca0 con 0x7f233c06afc0 2024-03-06T07:34:05.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f233be00640 1 -- 172.21.15.114:0/3840989650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334034610 con 0x7f233c06afc0 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 -- 172.21.15.114:0/3840989650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 msgr2=0x7f233c06b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 --2- 172.21.15.114:0/3840989650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 0x7f233c06b450 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2334009e50 tx=0x7f233402bf60 comp rx=0 tx=0).stop 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 -- 172.21.15.114:0/3840989650 shutdown_connections 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 --2- 172.21.15.114:0/3840989650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 0x7f233c06b450 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 --2- 172.21.15.114:0/3840989650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c06dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 -- 172.21.15.114:0/3840989650 >> 172.21.15.114:0/3840989650 conn(0x7f233c0b27d0 msgr2=0x7f233c0b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 -- 172.21.15.114:0/3840989650 shutdown_connections 2024-03-06T07:34:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.157+0000 7f2344400640 1 -- 172.21.15.114:0/3840989650 wait complete. 2024-03-06T07:34:05.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.159+0000 7f2344400640 1 Processor -- start 2024-03-06T07:34:05.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.159+0000 7f2344400640 1 -- start start 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2344400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 0x7f233c146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2344400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2344400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c15efa0 0x7f233c1579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2344400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c15f400 con 0x7f233c15efa0 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2344400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c06ec20 con 0x7f233c06afc0 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2344400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233c0723a0 con 0x7f233c15bb90 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2342400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c15efa0 0x7f233c1579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2342400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c15efa0 0x7f233c1579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42082/0 (socket says 172.21.15.114:42082) 2024-03-06T07:34:05.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.160+0000 7f2342400640 1 -- 172.21.15.114:0/4276059420 learned_addr learned my addr 172.21.15.114:0/4276059420 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2341000640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2341a00640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 0x7f233c146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2342400640 1 -- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 msgr2=0x7f233c145c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2342400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c145c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2342400640 1 -- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 msgr2=0x7f233c146610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2342400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 0x7f233c146610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2342400640 1 -- 172.21.15.114:0/4276059420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c0b6240 con 0x7f233c15efa0 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2341000640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.161+0000 7f2342400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c15efa0 0x7f233c1579b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f2334009e50 tx=0x7f2334004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.162+0000 7f2339600640 1 -- 172.21.15.114:0/4276059420 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334043020 con 0x7f233c15efa0 2024-03-06T07:34:05.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.162+0000 7f2339600640 1 -- 172.21.15.114:0/4276059420 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233402ce50 con 0x7f233c15efa0 2024-03-06T07:34:05.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.162+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233c159a30 con 0x7f233c15efa0 2024-03-06T07:34:05.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.162+0000 7f2339600640 1 -- 172.21.15.114:0/4276059420 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334009510 con 0x7f233c15efa0 2024-03-06T07:34:05.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.162+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233c159ff0 con 0x7f233c15efa0 2024-03-06T07:34:05.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.163+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233c04f330 con 0x7f233c15efa0 2024-03-06T07:34:05.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.165+0000 7f2339600640 1 -- 172.21.15.114:0/4276059420 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2334046020 con 0x7f233c15efa0 2024-03-06T07:34:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.165+0000 7f2339600640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f231c06b4d0 0x7f231c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.166+0000 7f2339600640 1 -- 172.21.15.114:0/4276059420 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f23340ac970 con 0x7f233c15efa0 2024-03-06T07:34:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.166+0000 7f2341a00640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f231c06b4d0 0x7f231c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:05.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.166+0000 7f2341a00640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f231c06b4d0 0x7f231c06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2330009ad0 tx=0x7f2330008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:05.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.170+0000 7f2339600640 1 -- 172.21.15.114:0/4276059420 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f233404d030 con 0x7f233c15efa0 2024-03-06T07:34:05.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.367+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f233c0b68b0 con 0x7f233c15efa0 2024-03-06T07:34:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.933+0000 7f2339600640 1 -- 172.21.15.114:0/4276059420 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7f23340473b0 con 0x7f233c15efa0 2024-03-06T07:34:05.935 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:34:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f231c06b4d0 msgr2=0x7f231c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f231c06b4d0 0x7f231c06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2330009ad0 tx=0x7f2330008040 comp rx=0 tx=0).stop 2024-03-06T07:34:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c15efa0 msgr2=0x7f233c1579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:05.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c15efa0 0x7f233c1579b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f2334009e50 tx=0x7f2334004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 shutdown_connections 2024-03-06T07:34:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f231c06b4d0 0x7f231c06d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c15efa0 0x7f233c1579b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f233c15bb90 0x7f233c145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 --2- 172.21.15.114:0/4276059420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f233c06afc0 0x7f233c146610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 >> 172.21.15.114:0/4276059420 conn(0x7f233c0b27d0 msgr2=0x7f233c15c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.937+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 shutdown_connections 2024-03-06T07:34:05.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:05.938+0000 7f2344400640 1 -- 172.21.15.114:0/4276059420 wait complete. 2024-03-06T07:34:05.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:34:06.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.157+0000 7f12c1c00640 1 Processor -- start 2024-03-06T07:34:06.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.157+0000 7f12c1c00640 1 -- start start 2024-03-06T07:34:06.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.158+0000 7f12c1c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 0x7f12bc0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.158+0000 7f12c1c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc15c1f0 con 0x7f12bc15be10 2024-03-06T07:34:06.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.158+0000 7f12c1c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc06cac0 con 0x7f12bc0aef20 2024-03-06T07:34:06.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.158+0000 7f12c1c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0cd200 con 0x7f12bc06da40 2024-03-06T07:34:06.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12bbe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f12bc15be10 0x7f12bc06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50542/0 (socket says 172.21.15.114:50542) 2024-03-06T07:34:06.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12bbe00640 1 -- 172.21.15.114:0/2813263476 learned_addr learned my addr 172.21.15.114:0/2813263476 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:06.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871817712 0 0) 0x7f12bc15c1f0 con 0x7f12bc15be10 2024-03-06T07:34:06.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12bb400640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 0x7f12bc0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0003520 con 0x7f12bc15be10 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 673663828 0 0) 0x7f12a0003520 con 0x7f12bc15be10 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 msgr2=0x7f12bc0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 0x7f12bc0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 >> v1:172.21.15.177:6789/0 conn(0x7f12bc0aef20 legacy=0x7f12bc071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc06cda0 con 0x7f12bc15be10 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.159+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f12ac001670 con 0x7f12bc15be10 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 >> v1:172.21.15.114:6789/0 conn(0x7f12bc15be10 legacy=0x7f12bc06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12baa00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12a00039e0 0x7f12a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12baa00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0004800 0x7f12a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12baa00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 0x7f12a0005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0cd200 con 0x7f12a00039e0 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000bb50 con 0x7f12a0004800 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000bd50 con 0x7f12bc06da40 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12bbe00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12a00039e0 0x7f12a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12bbe00640 1 -- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 msgr2=0x7f12a0005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12c1200640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 0x7f12a0005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12bbe00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc06da40 0x7f12a0005250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12bbe00640 1 -- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0004800 msgr2=0x7f12a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12bb400640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0004800 0x7f12a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12bbe00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0004800 0x7f12a0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.160+0000 7f12bbe00640 1 -- 172.21.15.114:0/2813263476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0003520 con 0x7f12a00039e0 2024-03-06T07:34:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.162+0000 7f12bbe00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12a00039e0 0x7f12a0003e00 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00c190 tx=0x7f12a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e99b27d7ec63f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.162+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac007c60 con 0x7f12a00039e0 2024-03-06T07:34:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.162+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac030070 con 0x7f12a00039e0 2024-03-06T07:34:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.162+0000 7f12baa00640 1 -- 172.21.15.114:0/2813263476 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac005d50 con 0x7f12a00039e0 2024-03-06T07:34:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.162+0000 7f12c1c00640 1 -- 172.21.15.114:0/2813263476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12a00039e0 msgr2=0x7f12a0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.162+0000 7f12c1c00640 1 --2- 172.21.15.114:0/2813263476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12a00039e0 0x7f12a0003e00 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00c190 tx=0x7f12a0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.162+0000 7f12c1200640 1 -- 172.21.15.114:0/2813263476 reap_dead start 2024-03-06T07:34:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.163+0000 7f12c1c00640 1 -- 172.21.15.114:0/2813263476 shutdown_connections 2024-03-06T07:34:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.163+0000 7f12c1c00640 1 -- 172.21.15.114:0/2813263476 >> 172.21.15.114:0/2813263476 conn(0x7f12bc06afc0 msgr2=0x7f12bc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.163+0000 7f12c1c00640 1 -- 172.21.15.114:0/2813263476 shutdown_connections 2024-03-06T07:34:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.163+0000 7f12c1c00640 1 -- 172.21.15.114:0/2813263476 wait complete. 2024-03-06T07:34:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.164+0000 7f12c1c00640 1 Processor -- start 2024-03-06T07:34:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.165+0000 7f12c1c00640 1 -- start start 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.165+0000 7f12c1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a00039e0 0x7f12bc153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.165+0000 7f12c1c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc141d50 0x7f12bc151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.165+0000 7f12c1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc151840 0x7f12bc0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.165+0000 7f12c1c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0cda90 con 0x7f12bc151840 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.165+0000 7f12c1c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc06d080 con 0x7f12a00039e0 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.165+0000 7f12c1c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0ccc20 con 0x7f12bc141d50 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc151840 0x7f12bc0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc151840 0x7f12bc0ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42100/0 (socket says 172.21.15.114:42100) 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 -- 172.21.15.114:0/386714214 learned_addr learned my addr 172.21.15.114:0/386714214 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12bb400640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc141d50 0x7f12bc151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12bbe00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a00039e0 0x7f12bc153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 -- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc141d50 msgr2=0x7f12bc151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc141d50 0x7f12bc151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 -- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a00039e0 msgr2=0x7f12bc153840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a00039e0 0x7f12bc153840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 -- 172.21.15.114:0/386714214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc0b2aa0 con 0x7f12bc151840 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.166+0000 7f12c1200640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc151840 0x7f12bc0ad860 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f12bc071110 tx=0x7f12b400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.167+0000 7f12b3e00640 1 -- 172.21.15.114:0/386714214 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4018070 con 0x7f12bc151840 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.167+0000 7f12b3e00640 1 -- 172.21.15.114:0/386714214 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4003b70 con 0x7f12bc151840 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.167+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12bc0b2d40 con 0x7f12bc151840 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.167+0000 7f12b3e00640 1 -- 172.21.15.114:0/386714214 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4005070 con 0x7f12bc151840 2024-03-06T07:34:06.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.167+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc0b3300 con 0x7f12bc151840 2024-03-06T07:34:06.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.168+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc04f330 con 0x7f12bc151840 2024-03-06T07:34:06.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.169+0000 7f12b3e00640 1 -- 172.21.15.114:0/386714214 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12b401b070 con 0x7f12bc151840 2024-03-06T07:34:06.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.170+0000 7f12b3e00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f129c06b4d0 0x7f129c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.170+0000 7f12b3e00640 1 -- 172.21.15.114:0/386714214 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f12b4089a50 con 0x7f12bc151840 2024-03-06T07:34:06.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.170+0000 7f12bbe00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f129c06b4d0 0x7f129c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.171+0000 7f12bbe00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f129c06b4d0 0x7f129c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f12ac034000 tx=0x7f12ac009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.175+0000 7f12b3e00640 1 -- 172.21.15.114:0/386714214 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12b4058890 con 0x7f12bc151840 2024-03-06T07:34:06.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.419+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f12bc156010 con 0x7f12bc151840 2024-03-06T07:34:06.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.937+0000 7f12b3e00640 1 -- 172.21.15.114:0/386714214 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v75) v1 ==== 174+0+0 (secure 0 0 0) 0x7f12b4058190 con 0x7f12bc151840 2024-03-06T07:34:06.939 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:34:06.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f129c06b4d0 msgr2=0x7f129c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f129c06b4d0 0x7f129c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f12ac034000 tx=0x7f12ac009030 comp rx=0 tx=0).stop 2024-03-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc151840 msgr2=0x7f12bc0ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc151840 0x7f12bc0ad860 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f12bc071110 tx=0x7f12b400dc00 comp rx=0 tx=0).stop 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 shutdown_connections 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f129c06b4d0 0x7f129c06d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc151840 0x7f12bc0ad860 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f12bc141d50 0x7f12bc151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 --2- 172.21.15.114:0/386714214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a00039e0 0x7f12bc153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 >> 172.21.15.114:0/386714214 conn(0x7f12bc06afc0 msgr2=0x7f12bc156d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 shutdown_connections 2024-03-06T07:34:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:06.941+0000 7f12c1c00640 1 -- 172.21.15.114:0/386714214 wait complete. 2024-03-06T07:34:06.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.161+0000 7f74a0a00640 1 Processor -- start 2024-03-06T07:34:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.162+0000 7f74a0a00640 1 -- start start 2024-03-06T07:34:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f74a0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f749c0ce580 0x7f749c165ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f74a0a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c06c910 con 0x7f749c15b040 2024-03-06T07:34:07.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f74a0a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c06b160 con 0x7f749c0cfcc0 2024-03-06T07:34:07.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f74a0a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c15b420 con 0x7f749c0ce580 2024-03-06T07:34:07.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f7499e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f749c15b040 0x7f749c06c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:50550/0 (socket says 172.21.15.114:50550) 2024-03-06T07:34:07.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f7499e00640 1 -- 172.21.15.114:0/4271615452 learned_addr learned my addr 172.21.15.114:0/4271615452 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:07.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f749a800640 1 --2- 172.21.15.114:0/4271615452 >> v2:172.21.15.114:3300/0 conn(0x7f749c0ce580 0x7f749c165ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204271936 0 0) 0x7f749c06c910 con 0x7f749c15b040 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.163+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c003520 con 0x7f749c15b040 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994987466 0 0) 0x7f747c003520 con 0x7f749c15b040 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 >> v2:172.21.15.114:3300/0 conn(0x7f749c0ce580 msgr2=0x7f749c165ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 --2- 172.21.15.114:0/4271615452 >> v2:172.21.15.114:3300/0 conn(0x7f749c0ce580 0x7f749c165ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 >> v1:172.21.15.177:6789/0 conn(0x7f749c0cfcc0 legacy=0x7f749c0cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c06b440 con 0x7f749c15b040 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f748c001670 con 0x7f749c15b040 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 >> v1:172.21.15.114:6789/0 conn(0x7f749c15b040 legacy=0x7f749c06c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0039b0 0x7f747c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f747c004800 0x7f747c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f747c0051c0 0x7f747c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c15b420 con 0x7f747c0051c0 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00c0f0 con 0x7f747c0039b0 2024-03-06T07:34:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00c2f0 con 0x7f747c004800 2024-03-06T07:34:07.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.164+0000 7f749a800640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f747c0051c0 0x7f747c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.165+0000 7f749a800640 1 -- 172.21.15.114:0/4271615452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f747c004800 msgr2=0x7f747c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.165+0000 7f749a800640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f747c004800 0x7f747c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.165+0000 7f749a800640 1 -- 172.21.15.114:0/4271615452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0039b0 msgr2=0x7f747c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.165+0000 7f749a800640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0039b0 0x7f747c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.165+0000 7f749a800640 1 -- 172.21.15.114:0/4271615452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c003520 con 0x7f747c0051c0 2024-03-06T07:34:07.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.166+0000 7f749a800640 1 -- 172.21.15.114:0/4271615452 reap_dead start 2024-03-06T07:34:07.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.166+0000 7f749a800640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f747c0051c0 0x7f747c00b9c0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f7494009ac0 tx=0x7f747c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b2c77d61c451b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.166+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494009690 con 0x7f747c0051c0 2024-03-06T07:34:07.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.166+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7494004020 con 0x7f747c0051c0 2024-03-06T07:34:07.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.166+0000 7f7498a00640 1 -- 172.21.15.114:0/4271615452 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494004410 con 0x7f747c0051c0 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.166+0000 7f74a0a00640 1 -- 172.21.15.114:0/4271615452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f747c0051c0 msgr2=0x7f747c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.166+0000 7f74a0a00640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f747c0051c0 0x7f747c00b9c0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f7494009ac0 tx=0x7f747c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.167+0000 7f74a0a00640 1 -- 172.21.15.114:0/4271615452 shutdown_connections 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.167+0000 7f74a0a00640 1 --2- 172.21.15.114:0/4271615452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f747c0051c0 0x7f747c00b9c0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.167+0000 7f74a0a00640 1 -- 172.21.15.114:0/4271615452 >> 172.21.15.114:0/4271615452 conn(0x7f749c0b1e10 msgr2=0x7f749c0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.167+0000 7f74a0a00640 1 -- 172.21.15.114:0/4271615452 shutdown_connections 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.167+0000 7f74a0a00640 1 -- 172.21.15.114:0/4271615452 wait complete. 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.169+0000 7f74a0a00640 1 Processor -- start 2024-03-06T07:34:07.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.169+0000 7f74a0a00640 1 -- start start 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f74a0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0051c0 0x7f749c0920b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f74a0a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f749c0ca940 0x7f749c0925f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f74a0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c169a40 0x7f749c0cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f74a0a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c06ba00 con 0x7f749c169a40 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f74a0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c06bd70 con 0x7f747c0051c0 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f74a0a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c06d5b0 con 0x7f749c0ca940 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f749a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c169a40 0x7f749c0cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f749a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c169a40 0x7f749c0cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42134/0 (socket says 172.21.15.114:42134) 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f749a800640 1 -- 172.21.15.114:0/2632592406 learned_addr learned my addr 172.21.15.114:0/2632592406 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f7499e00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0051c0 0x7f749c0920b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f749a800640 1 -- 172.21.15.114:0/2632592406 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f749c0ca940 msgr2=0x7f749c0925f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.170+0000 7f749a800640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f749c0ca940 0x7f749c0925f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f749a800640 1 -- 172.21.15.114:0/2632592406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0051c0 msgr2=0x7f749c0920b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f749a800640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0051c0 0x7f749c0920b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f749a800640 1 -- 172.21.15.114:0/2632592406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c0cd2f0 con 0x7f749c169a40 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f749a800640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c169a40 0x7f749c0cc040 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f749402c380 tx=0x7f7494004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f7492000640 1 -- 172.21.15.114:0/2632592406 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749402cc50 con 0x7f749c169a40 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f7492000640 1 -- 172.21.15.114:0/2632592406 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749402cdf0 con 0x7f749c169a40 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f7492000640 1 -- 172.21.15.114:0/2632592406 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494003850 con 0x7f749c169a40 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c0cd5d0 con 0x7f749c169a40 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.171+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c094000 con 0x7f749c169a40 2024-03-06T07:34:07.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.172+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c04f330 con 0x7f749c169a40 2024-03-06T07:34:07.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.173+0000 7f7492000640 1 -- 172.21.15.114:0/2632592406 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74940039f0 con 0x7f749c169a40 2024-03-06T07:34:07.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.174+0000 7f7492000640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f747806b500 0x7f747806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.174+0000 7f7492000640 1 -- 172.21.15.114:0/2632592406 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74940aaf00 con 0x7f749c169a40 2024-03-06T07:34:07.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.175+0000 7f7499e00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f747806b500 0x7f747806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.175+0000 7f7499e00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f747806b500 0x7f747806d9c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f748c000f60 tx=0x7f748c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.179+0000 7f7492000640 1 -- 172.21.15.114:0/2632592406 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7494079d60 con 0x7f749c169a40 2024-03-06T07:34:07.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.370+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f749c0b5050 con 0x7f749c169a40 2024-03-06T07:34:07.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.372+0000 7f7492000640 1 -- 172.21.15.114:0/2632592406 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16688 (secure 0 0 0) 0x7f7494079660 con 0x7f749c169a40 2024-03-06T07:34:07.374 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:07.374 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":107,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:06.420466+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f747806b500 msgr2=0x7f747806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f747806b500 0x7f747806d9c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f748c000f60 tx=0x7f748c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c169a40 msgr2=0x7f749c0cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c169a40 0x7f749c0cc040 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f749402c380 tx=0x7f7494004730 comp rx=0 tx=0).stop 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 shutdown_connections 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f747806b500 0x7f747806d9c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c169a40 0x7f749c0cc040 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f749c0ca940 0x7f749c0925f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 --2- 172.21.15.114:0/2632592406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747c0051c0 0x7f749c0920b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 >> 172.21.15.114:0/2632592406 conn(0x7f749c0b1e10 msgr2=0x7f749c0d0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 shutdown_connections 2024-03-06T07:34:07.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.375+0000 7f74a0a00640 1 -- 172.21.15.114:0/2632592406 wait complete. 2024-03-06T07:34:07.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:07.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.593+0000 7fa4c8c00640 1 Processor -- start 2024-03-06T07:34:07.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.594+0000 7fa4c8c00640 1 -- start start 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c8c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c415e230 0x7fa4c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c8c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4c40b4320 0x7fa4c40d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c8c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa4c40ce550 0x7fa4c40ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c8c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c415e610 con 0x7fa4c40b4320 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c8c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c40cf340 con 0x7fa4c40ce550 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c8c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40cfe60 con 0x7fa4c415e230 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c1400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4c40b4320 0x7fa4c40d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c1400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa4c40b4320 0x7fa4c40d0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42150/0 (socket says 172.21.15.114:42150) 2024-03-06T07:34:07.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.595+0000 7fa4c1400640 1 -- 172.21.15.114:0/2880951354 learned_addr learned my addr 172.21.15.114:0/2880951354 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:07.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.596+0000 7fa4c1e00640 1 --2- 172.21.15.114:0/2880951354 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c415e230 0x7fa4c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.596+0000 7fa4c2800640 1 --2- 172.21.15.114:0/2880951354 >> v2:172.21.15.177:3300/0 conn(0x7fa4c40ce550 0x7fa4c40ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.596+0000 7fa4c1400640 1 -- 172.21.15.114:0/2880951354 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c415e230 msgr2=0x7fa4c406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.596+0000 7fa4c1400640 1 --2- 172.21.15.114:0/2880951354 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c415e230 0x7fa4c406edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.596+0000 7fa4c1400640 1 -- 172.21.15.114:0/2880951354 >> v2:172.21.15.177:3300/0 conn(0x7fa4c40ce550 msgr2=0x7fa4c40ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.596+0000 7fa4c1400640 1 --2- 172.21.15.114:0/2880951354 >> v2:172.21.15.177:3300/0 conn(0x7fa4c40ce550 0x7fa4c40ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.596+0000 7fa4c1400640 1 -- 172.21.15.114:0/2880951354 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c406f390 con 0x7fa4c40b4320 2024-03-06T07:34:07.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.597+0000 7fa4c1400640 1 --2- 172.21.15.114:0/2880951354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40b4320 0x7fa4c40d0be0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40c57b0 tx=0x7fa4b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea32171b5451412a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c0a00640 1 -- 172.21.15.114:0/2880951354 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b002e070 con 0x7fa4c40b4320 2024-03-06T07:34:07.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c0a00640 1 -- 172.21.15.114:0/2880951354 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0037070 con 0x7fa4c40b4320 2024-03-06T07:34:07.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c0a00640 1 -- 172.21.15.114:0/2880951354 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0031a00 con 0x7fa4c40b4320 2024-03-06T07:34:07.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c8c00640 1 -- 172.21.15.114:0/2880951354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40b4320 msgr2=0x7fa4c40d0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/2880951354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40b4320 0x7fa4c40d0be0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40c57b0 tx=0x7fa4b002b880 comp rx=0 tx=0).stop 2024-03-06T07:34:07.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c8c00640 1 -- 172.21.15.114:0/2880951354 shutdown_connections 2024-03-06T07:34:07.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/2880951354 >> v2:172.21.15.177:3300/0 conn(0x7fa4c40ce550 0x7fa4c40ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/2880951354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40b4320 0x7fa4c40d0be0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/2880951354 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c415e230 0x7fa4c406edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.598+0000 7fa4c8c00640 1 -- 172.21.15.114:0/2880951354 >> 172.21.15.114:0/2880951354 conn(0x7fa4c40b3800 msgr2=0x7fa4c40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.599+0000 7fa4c8c00640 1 -- 172.21.15.114:0/2880951354 shutdown_connections 2024-03-06T07:34:07.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.599+0000 7fa4c8c00640 1 -- 172.21.15.114:0/2880951354 wait complete. 2024-03-06T07:34:07.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.600+0000 7fa4c8c00640 1 Processor -- start 2024-03-06T07:34:07.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.600+0000 7fa4c8c00640 1 -- start start 2024-03-06T07:34:07.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.601+0000 7fa4c8c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c40b4320 0x7fa4c40c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.601+0000 7fa4c8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40ce550 0x7fa4c4145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.601+0000 7fa4c8c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4c415e230 0x7fa4c40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.601+0000 7fa4c8c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40d06c0 con 0x7fa4c40ce550 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.601+0000 7fa4c8c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40d03e0 con 0x7fa4c415e230 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.601+0000 7fa4c8c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40d1120 con 0x7fa4c40b4320 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.601+0000 7fa4c1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40ce550 0x7fa4c4145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40ce550 0x7fa4c4145f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42164/0 (socket says 172.21.15.114:42164) 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 -- 172.21.15.114:0/3245565069 learned_addr learned my addr 172.21.15.114:0/3245565069 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c2800640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4c415e230 0x7fa4c40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 -- 172.21.15.114:0/3245565069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c40b4320 msgr2=0x7fa4c40c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c40b4320 0x7fa4c40c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 -- 172.21.15.114:0/3245565069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4c415e230 msgr2=0x7fa4c40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4c415e230 0x7fa4c40b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 -- 172.21.15.114:0/3245565069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c40c7150 con 0x7fa4c40ce550 2024-03-06T07:34:07.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.602+0000 7fa4c1400640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40ce550 0x7fa4c4145f80 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa4b002b8c0 tx=0x7fa4b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.603+0000 7fa4ba000640 1 -- 172.21.15.114:0/3245565069 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0032880 con 0x7fa4c40ce550 2024-03-06T07:34:07.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.603+0000 7fa4ba000640 1 -- 172.21.15.114:0/3245565069 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0032a20 con 0x7fa4c40ce550 2024-03-06T07:34:07.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.603+0000 7fa4ba000640 1 -- 172.21.15.114:0/3245565069 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0032e10 con 0x7fa4c40ce550 2024-03-06T07:34:07.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.603+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c40c7430 con 0x7fa4c40ce550 2024-03-06T07:34:07.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.603+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c40c79f0 con 0x7fa4c40ce550 2024-03-06T07:34:07.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.604+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c404f330 con 0x7fa4c40ce550 2024-03-06T07:34:07.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.605+0000 7fa4ba000640 1 -- 172.21.15.114:0/3245565069 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4b004c060 con 0x7fa4c40ce550 2024-03-06T07:34:07.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.606+0000 7fa4ba000640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4a406b4d0 0x7fa4a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.606+0000 7fa4ba000640 1 -- 172.21.15.114:0/3245565069 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4b00abe30 con 0x7fa4c40ce550 2024-03-06T07:34:07.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.606+0000 7fa4c1e00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4a406b4d0 0x7fa4a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.607+0000 7fa4c1e00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4a406b4d0 0x7fa4a406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4006fd0 tx=0x7fa4b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.610+0000 7fa4ba000640 1 -- 172.21.15.114:0/3245565069 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4b0040070 con 0x7fa4c40ce550 2024-03-06T07:34:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.806+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4c40d1740 con 0x7fa4c40ce550 2024-03-06T07:34:07.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.809+0000 7fa4ba000640 1 -- 172.21.15.114:0/3245565069 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16688 (secure 0 0 0) 0x7fa4b004c350 con 0x7fa4c40ce550 2024-03-06T07:34:07.811 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:07.811 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":107,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:06.420466+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.811+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4a406b4d0 msgr2=0x7fa4a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.811+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4a406b4d0 0x7fa4a406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4006fd0 tx=0x7fa4b4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.811+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40ce550 msgr2=0x7fa4c4145f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.811+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40ce550 0x7fa4c4145f80 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fa4b002b8c0 tx=0x7fa4b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 shutdown_connections 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4c415e230 0x7fa4c40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa4a406b4d0 0x7fa4a406d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa4c40ce550 0x7fa4c4145f80 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 --2- 172.21.15.114:0/3245565069 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa4c40b4320 0x7fa4c40c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 >> 172.21.15.114:0/3245565069 conn(0x7fa4c40b3800 msgr2=0x7fa4c40b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 shutdown_connections 2024-03-06T07:34:07.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:07.812+0000 7fa4c8c00640 1 -- 172.21.15.114:0/3245565069 wait complete. 2024-03-06T07:34:07.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:34:08.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.092+0000 7f316be00640 1 Processor -- start 2024-03-06T07:34:08.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.092+0000 7f316be00640 1 -- start start 2024-03-06T07:34:08.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.095+0000 7f316be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c15be10 0x7f316c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.095+0000 7f316be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f316c06da40 0x7f316c0ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.095+0000 7f316be00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316c15c1f0 con 0x7f316c0aef20 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.095+0000 7f316be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316c0cd200 con 0x7f316c06da40 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.095+0000 7f316be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c06cac0 con 0x7f316c15be10 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.095+0000 7f316aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c15be10 0x7f316c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.095+0000 7f316b400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f316c06da40 0x7f316c0ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.096+0000 7f316aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c15be10 0x7f316c0ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:39336/0 (socket says 172.21.15.114:39336) 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.096+0000 7f316aa00640 1 -- 172.21.15.114:0/3479719861 learned_addr learned my addr 172.21.15.114:0/3479719861 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.096+0000 7f316b400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f316c06da40 0x7f316c0ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:41920/0 (socket says 172.21.15.114:41920) 2024-03-06T07:34:08.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.096+0000 7f316aa00640 1 -- 172.21.15.114:0/3479719861 >> v2:172.21.15.177:3300/0 conn(0x7f316c06da40 msgr2=0x7f316c0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.096+0000 7f316aa00640 1 --2- 172.21.15.114:0/3479719861 >> v2:172.21.15.177:3300/0 conn(0x7f316c06da40 0x7f316c0ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.096+0000 7f316aa00640 1 -- 172.21.15.114:0/3479719861 >> v1:172.21.15.177:6789/0 conn(0x7f316c0aef20 legacy=0x7f316c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.096+0000 7f316aa00640 1 -- 172.21.15.114:0/3479719861 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c06cda0 con 0x7f316c15be10 2024-03-06T07:34:08.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.098+0000 7f316aa00640 1 --2- 172.21.15.114:0/3479719861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c15be10 0x7f316c0ccc20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f315c009870 tx=0x7f315c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b357a848b276839b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.098+0000 7f3169600640 1 -- 172.21.15.114:0/3479719861 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c034070 con 0x7f316c15be10 2024-03-06T07:34:08.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.098+0000 7f3169600640 1 -- 172.21.15.114:0/3479719861 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c030840 con 0x7f316c15be10 2024-03-06T07:34:08.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.098+0000 7f3169600640 1 -- 172.21.15.114:0/3479719861 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c030c10 con 0x7f316c15be10 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 -- 172.21.15.114:0/3479719861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c15be10 msgr2=0x7f316c0ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 --2- 172.21.15.114:0/3479719861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c15be10 0x7f316c0ccc20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f315c009870 tx=0x7f315c02b980 comp rx=0 tx=0).stop 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 -- 172.21.15.114:0/3479719861 shutdown_connections 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 --2- 172.21.15.114:0/3479719861 >> v2:172.21.15.177:3300/0 conn(0x7f316c06da40 0x7f316c0ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 --2- 172.21.15.114:0/3479719861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c15be10 0x7f316c0ccc20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 -- 172.21.15.114:0/3479719861 >> 172.21.15.114:0/3479719861 conn(0x7f316c06afc0 msgr2=0x7f316c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 -- 172.21.15.114:0/3479719861 shutdown_connections 2024-03-06T07:34:08.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.099+0000 7f316be00640 1 -- 172.21.15.114:0/3479719861 wait complete. 2024-03-06T07:34:08.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.101+0000 7f316be00640 1 Processor -- start 2024-03-06T07:34:08.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.101+0000 7f316be00640 1 -- start start 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f316c06da40 0x7f316c14d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c0aef20 0x7f316c1f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c15be10 0x7f316c159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0cda90 con 0x7f316c06da40 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c06d080 con 0x7f316c15be10 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c071ab0 con 0x7f316c0aef20 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f316c06da40 0x7f316c14d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f316c06da40 0x7f316c14d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41910/0 (socket says 172.21.15.114:41910) 2024-03-06T07:34:08.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.103+0000 7f316aa00640 1 -- 172.21.15.114:0/2450035593 learned_addr learned my addr 172.21.15.114:0/2450035593 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316a000640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c0aef20 0x7f316c1f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316b400640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c15be10 0x7f316c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316aa00640 1 -- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c0aef20 msgr2=0x7f316c1f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316aa00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c0aef20 0x7f316c1f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316aa00640 1 -- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c15be10 msgr2=0x7f316c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316aa00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c15be10 0x7f316c159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316aa00640 1 -- 172.21.15.114:0/2450035593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c1599f0 con 0x7f316c06da40 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316a000640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c0aef20 0x7f316c1f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316aa00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f316c06da40 0x7f316c14d690 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f3162a00640 1 -- 172.21.15.114:0/2450035593 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c034040 con 0x7f316c06da40 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f3162a00640 1 -- 172.21.15.114:0/2450035593 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c039d90 con 0x7f316c06da40 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f3162a00640 1 -- 172.21.15.114:0/2450035593 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c043630 con 0x7f316c06da40 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.104+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c152440 con 0x7f316c06da40 2024-03-06T07:34:08.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.105+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c152060 con 0x7f316c06da40 2024-03-06T07:34:08.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.106+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316c04f330 con 0x7f316c06da40 2024-03-06T07:34:08.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.107+0000 7f3162a00640 1 -- 172.21.15.114:0/2450035593 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f315c04a060 con 0x7f316c06da40 2024-03-06T07:34:08.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.107+0000 7f3162a00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f314c06b4d0 0x7f314c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.108+0000 7f3162a00640 1 -- 172.21.15.114:0/2450035593 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f315c0ac230 con 0x7f316c06da40 2024-03-06T07:34:08.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.108+0000 7f316a000640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f314c06b4d0 0x7f314c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.109+0000 7f316a000640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f314c06b4d0 0x7f314c06d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f316c1f5de0 tx=0x7f3158009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.112+0000 7f3162a00640 1 -- 172.21.15.114:0/2450035593 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f315c03e070 con 0x7f316c06da40 2024-03-06T07:34:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.346+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f316c0b2050 con 0x7f316c06da40 2024-03-06T07:34:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.978+0000 7f3162a00640 1 -- 172.21.15.114:0/2450035593 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v78) v1 ==== 118+0+0 (secure 0 0 0) 0x7f315c037e40 con 0x7f316c06da40 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.981+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f314c06b4d0 msgr2=0x7f314c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.981+0000 7f316be00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f314c06b4d0 0x7f314c06d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f316c1f5de0 tx=0x7f3158009040 comp rx=0 tx=0).stop 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.981+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f316c06da40 msgr2=0x7f316c14d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f316c06da40 0x7f316c14d690 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 shutdown_connections 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c15be10 0x7f316c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f316c0aef20 0x7f316c1f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f314c06b4d0 0x7f314c06d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 --2- 172.21.15.114:0/2450035593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f316c06da40 0x7f316c14d690 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 >> 172.21.15.114:0/2450035593 conn(0x7f316c06afc0 msgr2=0x7f316c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 shutdown_connections 2024-03-06T07:34:08.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:08.982+0000 7f316be00640 1 -- 172.21.15.114:0/2450035593 wait complete. 2024-03-06T07:34:09.003 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:34:09.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:34:09.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.196+0000 7f826be00640 1 Processor -- start 2024-03-06T07:34:09.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.196+0000 7f826be00640 1 -- start start 2024-03-06T07:34:09.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.197+0000 7f826be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f826c0d06d0 0x7f826c0d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.197+0000 7f826be00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0d0ff0 con 0x7f826c0d1db0 2024-03-06T07:34:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.198+0000 7f826be00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0cf2f0 con 0x7f826c0b4b90 2024-03-06T07:34:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.198+0000 7f826be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c0cf5d0 con 0x7f826c0d06d0 2024-03-06T07:34:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.198+0000 7f826b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f826c0d1db0 0x7f826c0d2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55356/0 (socket says 172.21.15.114:55356) 2024-03-06T07:34:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.198+0000 7f826b400640 1 -- 172.21.15.114:0/3882686412 learned_addr learned my addr 172.21.15.114:0/3882686412 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:09.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.198+0000 7f826aa00640 1 --2- 172.21.15.114:0/3882686412 >> v2:172.21.15.177:3300/0 conn(0x7f826c0d06d0 0x7f826c0d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.198+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236428638 0 0) 0x7f826c0d0ff0 con 0x7f826c0d1db0 2024-03-06T07:34:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.198+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8250003520 con 0x7f826c0d1db0 2024-03-06T07:34:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3375185985 0 0) 0x7f8250003520 con 0x7f826c0d1db0 2024-03-06T07:34:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 >> v2:172.21.15.177:3300/0 conn(0x7f826c0d06d0 msgr2=0x7f826c0d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 --2- 172.21.15.114:0/3882686412 >> v2:172.21.15.177:3300/0 conn(0x7f826c0d06d0 0x7f826c0d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 >> v1:172.21.15.177:6789/0 conn(0x7f826c0b4b90 legacy=0x7f826c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0cf8b0 con 0x7f826c0d1db0 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f82640019f0 con 0x7f826c0d1db0 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 >> v1:172.21.15.114:6789/0 conn(0x7f826c0d1db0 legacy=0x7f826c0d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500039b0 0x7f8250003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8250004800 0x7f8250004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82500051c0 0x7f825000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0cf5d0 con 0x7f82500051c0 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825000c060 con 0x7f82500039b0 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825000c260 con 0x7f8250004800 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.199+0000 7f826b400640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82500051c0 0x7f825000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826aa00640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500039b0 0x7f8250003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826b400640 1 -- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8250004800 msgr2=0x7f8250004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826a000640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8250004800 0x7f8250004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826b400640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8250004800 0x7f8250004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826b400640 1 -- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500039b0 msgr2=0x7f8250003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826b400640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500039b0 0x7f8250003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826b400640 1 -- 172.21.15.114:0/3882686412 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8250003520 con 0x7f82500051c0 2024-03-06T07:34:09.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826a000640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8250004800 0x7f8250004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.200+0000 7f826aa00640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500039b0 0x7f8250003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.201+0000 7f826b400640 1 -- 172.21.15.114:0/3882686412 reap_dead start 2024-03-06T07:34:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.201+0000 7f826b400640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82500051c0 0x7f825000b9c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f826400a840 tx=0x7f8250005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6819229073f5a24b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.201+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400a460 con 0x7f82500051c0 2024-03-06T07:34:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.201+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264034650 con 0x7f82500051c0 2024-03-06T07:34:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.201+0000 7f8269600640 1 -- 172.21.15.114:0/3882686412 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264034a40 con 0x7f82500051c0 2024-03-06T07:34:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.202+0000 7f826be00640 1 -- 172.21.15.114:0/3882686412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82500051c0 msgr2=0x7f825000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.202+0000 7f826be00640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82500051c0 0x7f825000b9c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f826400a840 tx=0x7f8250005830 comp rx=0 tx=0).stop 2024-03-06T07:34:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.202+0000 7f826be00640 1 -- 172.21.15.114:0/3882686412 shutdown_connections 2024-03-06T07:34:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.202+0000 7f826be00640 1 --2- 172.21.15.114:0/3882686412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82500051c0 0x7f825000b9c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.202+0000 7f826be00640 1 -- 172.21.15.114:0/3882686412 >> 172.21.15.114:0/3882686412 conn(0x7f826c09d070 msgr2=0x7f826c0d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.202+0000 7f826be00640 1 -- 172.21.15.114:0/3882686412 shutdown_connections 2024-03-06T07:34:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.202+0000 7f826be00640 1 -- 172.21.15.114:0/3882686412 wait complete. 2024-03-06T07:34:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.204+0000 7f826be00640 1 Processor -- start 2024-03-06T07:34:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.204+0000 7f826be00640 1 -- start start 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500051c0 0x7f826c14d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f826c14da50 0x7f826c15e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f826c15e980 0x7f826c14ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0d0ab0 con 0x7f826c14da50 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0b0960 con 0x7f82500051c0 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0d0c30 con 0x7f826c15e980 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f826c14da50 0x7f826c15e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f826c14da50 0x7f826c15e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41920/0 (socket says 172.21.15.114:41920) 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 -- 172.21.15.114:0/4268032905 learned_addr learned my addr 172.21.15.114:0/4268032905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826aa00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500051c0 0x7f826c14d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 -- 172.21.15.114:0/4268032905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f826c15e980 msgr2=0x7f826c14ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f826c15e980 0x7f826c14ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 -- 172.21.15.114:0/4268032905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500051c0 msgr2=0x7f826c14d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500051c0 0x7f826c14d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.205+0000 7f826a000640 1 -- 172.21.15.114:0/4268032905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c14dee0 con 0x7f826c14da50 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.206+0000 7f826a000640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f826c14da50 0x7f826c15e440 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f8258000f60 tx=0x7f825800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.206+0000 7f8263400640 1 -- 172.21.15.114:0/4268032905 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258005480 con 0x7f826c14da50 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.206+0000 7f8263400640 1 -- 172.21.15.114:0/4268032905 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8258017070 con 0x7f826c14da50 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.206+0000 7f8263400640 1 -- 172.21.15.114:0/4268032905 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258003810 con 0x7f826c14da50 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.206+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c14e1c0 con 0x7f826c14da50 2024-03-06T07:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.206+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c14e780 con 0x7f826c14da50 2024-03-06T07:34:09.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.207+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c04f330 con 0x7f826c14da50 2024-03-06T07:34:09.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.208+0000 7f8263400640 1 -- 172.21.15.114:0/4268032905 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82580039b0 con 0x7f826c14da50 2024-03-06T07:34:09.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.209+0000 7f8263400640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f824c06b4d0 0x7f824c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.209+0000 7f8263400640 1 -- 172.21.15.114:0/4268032905 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f825808a100 con 0x7f826c14da50 2024-03-06T07:34:09.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.209+0000 7f826aa00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f824c06b4d0 0x7f824c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.210+0000 7f826aa00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f824c06b4d0 0x7f824c06d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f826c157cc0 tx=0x7f825c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.218+0000 7f8263400640 1 -- 172.21.15.114:0/4268032905 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8258058ee0 con 0x7f826c14da50 2024-03-06T07:34:09.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.451+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f826c0b3540 con 0x7f826c14da50 2024-03-06T07:34:09.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.988+0000 7f8263400640 1 -- 172.21.15.114:0/4268032905 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v79) v1 ==== 122+0+0 (secure 0 0 0) 0x7f82580587e0 con 0x7f826c14da50 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.991+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f824c06b4d0 msgr2=0x7f824c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.991+0000 7f826be00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f824c06b4d0 0x7f824c06d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f826c157cc0 tx=0x7f825c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.991+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f826c14da50 msgr2=0x7f826c15e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.991+0000 7f826be00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f826c14da50 0x7f826c15e440 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f8258000f60 tx=0x7f825800d9f0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 shutdown_connections 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f826c15e980 0x7f826c14ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f824c06b4d0 0x7f824c06d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f826c14da50 0x7f826c15e440 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 --2- 172.21.15.114:0/4268032905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82500051c0 0x7f826c14d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 >> 172.21.15.114:0/4268032905 conn(0x7f826c09d070 msgr2=0x7f826c0b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 shutdown_connections 2024-03-06T07:34:09.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:09.992+0000 7f826be00640 1 -- 172.21.15.114:0/4268032905 wait complete. 2024-03-06T07:34:10.014 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:34:10.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:10.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.253+0000 7f9e37e00640 1 Processor -- start 2024-03-06T07:34:10.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.253+0000 7f9e37e00640 1 -- start start 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.255+0000 7f9e37e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e3815a940 0x7f9e380b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.255+0000 7f9e37e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e380cc2a0 0x7f9e380cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.255+0000 7f9e37e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e380b2b30 con 0x7f9e380cf370 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.255+0000 7f9e37e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e380b28e0 con 0x7f9e380cc2a0 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.255+0000 7f9e37e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380b2e10 con 0x7f9e3815a940 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e37400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e380cc2a0 0x7f9e380cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e37400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e380cc2a0 0x7f9e380cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41930/0 (socket says 172.21.15.114:41930) 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e37400640 1 -- 172.21.15.114:0/661892508 learned_addr learned my addr 172.21.15.114:0/661892508 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e36a00640 1 --2- 172.21.15.114:0/661892508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e3815a940 0x7f9e380b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e37400640 1 -- 172.21.15.114:0/661892508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e3815a940 msgr2=0x7f9e380b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e37400640 1 --2- 172.21.15.114:0/661892508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e3815a940 0x7f9e380b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e37400640 1 -- 172.21.15.114:0/661892508 >> v1:172.21.15.177:6789/0 conn(0x7f9e380cf370 legacy=0x7f9e3806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.256+0000 7f9e37400640 1 -- 172.21.15.114:0/661892508 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3806ca50 con 0x7f9e380cc2a0 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.257+0000 7f9e37400640 1 --2- 172.21.15.114:0/661892508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e380cc2a0 0x7f9e380cc750 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009e50 tx=0x7f9e3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd248fc1022be1e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.257+0000 7f9e35600640 1 -- 172.21.15.114:0/661892508 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3002ca70 con 0x7f9e380cc2a0 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e35600640 1 -- 172.21.15.114:0/661892508 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3002cc10 con 0x7f9e380cc2a0 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e35600640 1 -- 172.21.15.114:0/661892508 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e300343f0 con 0x7f9e380cc2a0 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 -- 172.21.15.114:0/661892508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e380cc2a0 msgr2=0x7f9e380cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 --2- 172.21.15.114:0/661892508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e380cc2a0 0x7f9e380cc750 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009e50 tx=0x7f9e3002bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 -- 172.21.15.114:0/661892508 shutdown_connections 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 --2- 172.21.15.114:0/661892508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e380cc2a0 0x7f9e380cc750 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 --2- 172.21.15.114:0/661892508 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e3815a940 0x7f9e380b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 -- 172.21.15.114:0/661892508 >> 172.21.15.114:0/661892508 conn(0x7f9e3806afc0 msgr2=0x7f9e3806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 -- 172.21.15.114:0/661892508 shutdown_connections 2024-03-06T07:34:10.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.258+0000 7f9e37e00640 1 -- 172.21.15.114:0/661892508 wait complete. 2024-03-06T07:34:10.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.261+0000 7f9e37e00640 1 Processor -- start 2024-03-06T07:34:10.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.261+0000 7f9e37e00640 1 -- start start 2024-03-06T07:34:10.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e380cc2a0 0x7f9e38158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e380cf370 0x7f9e381f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e3815a940 0x7f9e381f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380b4b20 con 0x7f9e3815a940 2024-03-06T07:34:10.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380b30f0 con 0x7f9e380cf370 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3815ad20 con 0x7f9e380cc2a0 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e3815a940 0x7f9e381f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e3815a940 0x7f9e381f5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41938/0 (socket says 172.21.15.114:41938) 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e37400640 1 -- 172.21.15.114:0/2646100750 learned_addr learned my addr 172.21.15.114:0/2646100750 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e36a00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e380cc2a0 0x7f9e38158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.263+0000 7f9e36000640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e380cf370 0x7f9e381f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37400640 1 -- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e380cc2a0 msgr2=0x7f9e38158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37400640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e380cc2a0 0x7f9e38158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37400640 1 -- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e380cf370 msgr2=0x7f9e381f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37400640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e380cf370 0x7f9e381f3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37400640 1 -- 172.21.15.114:0/2646100750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e381594c0 con 0x7f9e3815a940 2024-03-06T07:34:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37400640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e3815a940 0x7f9e381f5090 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009e50 tx=0x7f9e30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e2f400640 1 -- 172.21.15.114:0/2646100750 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30043020 con 0x7f9e3815a940 2024-03-06T07:34:10.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e2f400640 1 -- 172.21.15.114:0/2646100750 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3003e450 con 0x7f9e3815a940 2024-03-06T07:34:10.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e2f400640 1 -- 172.21.15.114:0/2646100750 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e30009510 con 0x7f9e3815a940 2024-03-06T07:34:10.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e381f4e20 con 0x7f9e3815a940 2024-03-06T07:34:10.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.264+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381f4b40 con 0x7f9e3815a940 2024-03-06T07:34:10.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.266+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3804f330 con 0x7f9e3815a940 2024-03-06T07:34:10.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.267+0000 7f9e2f400640 1 -- 172.21.15.114:0/2646100750 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e30046020 con 0x7f9e3815a940 2024-03-06T07:34:10.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.268+0000 7f9e2f400640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e1806b4d0 0x7f9e1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.268+0000 7f9e2f400640 1 -- 172.21.15.114:0/2646100750 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e300ac9b0 con 0x7f9e3815a940 2024-03-06T07:34:10.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.268+0000 7f9e36a00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e1806b4d0 0x7f9e1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.269+0000 7f9e36a00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e1806b4d0 0x7f9e1806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9e28006fd0 tx=0x7f9e28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.273+0000 7f9e2f400640 1 -- 172.21.15.114:0/2646100750 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e3004d030 con 0x7f9e3815a940 2024-03-06T07:34:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.509+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e380cf9a0 con 0x7f9e3815a940 2024-03-06T07:34:10.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.510+0000 7f9e2f400640 1 -- 172.21.15.114:0/2646100750 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f9e3007b830 con 0x7f9e3815a940 2024-03-06T07:34:10.513 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:10.513 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":79,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5983,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3305544049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3305544049},{"type":"v1","addr":"172.21.15.114:6836","nonce":3305544049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6001,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/3949834778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3949834778},{"type":"v1","addr":"172.21.15.177:6834","nonce":3949834778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6004,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2151340900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2151340900},{"type":"v1","addr":"172.21.15.177:6835","nonce":2151340900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:09.452793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7674},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7674":{"gid":7674,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/860883596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":860883596},{"type":"v1","addr":"172.21.15.114:6837","nonce":860883596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7674,"qdb_cluster":[7674]},"id":8}]} 2024-03-06T07:34:10.513 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 79 2024-03-06T07:34:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e1806b4d0 msgr2=0x7f9e1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e1806b4d0 0x7f9e1806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9e28006fd0 tx=0x7f9e28008040 comp rx=0 tx=0).stop 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e3815a940 msgr2=0x7f9e381f5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e3815a940 0x7f9e381f5090 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009e50 tx=0x7f9e30004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 shutdown_connections 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e1806b4d0 0x7f9e1806d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e3815a940 0x7f9e381f5090 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e380cf370 0x7f9e381f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 --2- 172.21.15.114:0/2646100750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e380cc2a0 0x7f9e38158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 >> 172.21.15.114:0/2646100750 conn(0x7f9e3806afc0 msgr2=0x7f9e380cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 shutdown_connections 2024-03-06T07:34:10.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.513+0000 7f9e37e00640 1 -- 172.21.15.114:0/2646100750 wait complete. 2024-03-06T07:34:10.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:10.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.731+0000 7f6ddb400640 1 Processor -- start 2024-03-06T07:34:10.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.731+0000 7f6ddb400640 1 -- start start 2024-03-06T07:34:10.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.733+0000 7f6ddb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6dd4144c00 0x7f6dd406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.733+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd415fe60 0x7f6dd40ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.733+0000 7f6ddb400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd406ee70 con 0x7f6dd415f9d0 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.733+0000 7f6ddb400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd406cef0 con 0x7f6dd4144c00 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.733+0000 7f6ddb400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd406d1d0 con 0x7f6dd415fe60 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.734+0000 7f6dd8a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6dd4144c00 0x7f6dd406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.734+0000 7f6dd8a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6dd4144c00 0x7f6dd406e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41954/0 (socket says 172.21.15.114:41954) 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.734+0000 7f6dd8a00640 1 -- 172.21.15.114:0/1870265846 learned_addr learned my addr 172.21.15.114:0/1870265846 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.734+0000 7f6dd9400640 1 --2- 172.21.15.114:0/1870265846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd415fe60 0x7f6dd40ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.734+0000 7f6dd8a00640 1 -- 172.21.15.114:0/1870265846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd415fe60 msgr2=0x7f6dd40ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:10.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.734+0000 7f6dd8a00640 1 --2- 172.21.15.114:0/1870265846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd415fe60 0x7f6dd40ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.735+0000 7f6dd8a00640 1 -- 172.21.15.114:0/1870265846 >> v1:172.21.15.177:6789/0 conn(0x7f6dd415f9d0 legacy=0x7f6dd40ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.735+0000 7f6dd8a00640 1 -- 172.21.15.114:0/1870265846 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd406d4b0 con 0x7f6dd4144c00 2024-03-06T07:34:10.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.736+0000 7f6dd8a00640 1 --2- 172.21.15.114:0/1870265846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd4144c00 0x7f6dd406e930 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009870 tx=0x7f6dc402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8cba73301969f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.736+0000 7f6dd3e00640 1 -- 172.21.15.114:0/1870265846 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4034070 con 0x7f6dd4144c00 2024-03-06T07:34:10.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6dd3e00640 1 -- 172.21.15.114:0/1870265846 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc4030840 con 0x7f6dd4144c00 2024-03-06T07:34:10.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6dd3e00640 1 -- 172.21.15.114:0/1870265846 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4030c10 con 0x7f6dd4144c00 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6ddb400640 1 -- 172.21.15.114:0/1870265846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd4144c00 msgr2=0x7f6dd406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1870265846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd4144c00 0x7f6dd406e930 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009870 tx=0x7f6dc402b980 comp rx=0 tx=0).stop 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6ddb400640 1 -- 172.21.15.114:0/1870265846 shutdown_connections 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1870265846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd415fe60 0x7f6dd40ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1870265846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd4144c00 0x7f6dd406e930 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6ddb400640 1 -- 172.21.15.114:0/1870265846 >> 172.21.15.114:0/1870265846 conn(0x7f6dd40b1730 msgr2=0x7f6dd40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.737+0000 7f6ddb400640 1 -- 172.21.15.114:0/1870265846 shutdown_connections 2024-03-06T07:34:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.738+0000 7f6ddb400640 1 -- 172.21.15.114:0/1870265846 wait complete. 2024-03-06T07:34:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.739+0000 7f6ddb400640 1 Processor -- start 2024-03-06T07:34:10.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.739+0000 7f6ddb400640 1 -- start start 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.740+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd4144c00 0x7f6dd4136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.740+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd415f9d0 0x7f6dd415e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.740+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dd415fe60 0x7f6dd415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.740+0000 7f6ddb400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4160730 con 0x7f6dd415f9d0 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.740+0000 7f6ddb400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4160290 con 0x7f6dd415fe60 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.740+0000 7f6ddb400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd406cb50 con 0x7f6dd4144c00 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd415f9d0 0x7f6dd415e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd415f9d0 0x7f6dd415e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41974/0 (socket says 172.21.15.114:41974) 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 -- 172.21.15.114:0/1421341076 learned_addr learned my addr 172.21.15.114:0/1421341076 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dd9400640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dd415fe60 0x7f6dd415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dd8a00640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd4144c00 0x7f6dd4136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 -- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd4144c00 msgr2=0x7f6dd4136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd4144c00 0x7f6dd4136180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 -- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dd415fe60 msgr2=0x7f6dd415e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dd415fe60 0x7f6dd415e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 -- 172.21.15.114:0/1421341076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd41e84a0 con 0x7f6dd415f9d0 2024-03-06T07:34:10.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.741+0000 7f6dcbe00640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd415f9d0 0x7f6dd415e160 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00c400 tx=0x7f6dbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.742+0000 7f6dd1600640 1 -- 172.21.15.114:0/1421341076 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc0052d0 con 0x7f6dd415f9d0 2024-03-06T07:34:10.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.742+0000 7f6dd1600640 1 -- 172.21.15.114:0/1421341076 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dbc005490 con 0x7f6dd415f9d0 2024-03-06T07:34:10.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.742+0000 7f6dd1600640 1 -- 172.21.15.114:0/1421341076 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc007260 con 0x7f6dd415f9d0 2024-03-06T07:34:10.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.742+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd41e8780 con 0x7f6dd415f9d0 2024-03-06T07:34:10.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.742+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd41e8d40 con 0x7f6dd415f9d0 2024-03-06T07:34:10.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.743+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd404f330 con 0x7f6dd415f9d0 2024-03-06T07:34:10.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.745+0000 7f6dd1600640 1 -- 172.21.15.114:0/1421341076 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6dbc007400 con 0x7f6dd415f9d0 2024-03-06T07:34:10.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.746+0000 7f6dd1600640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b4d0 0x7f6db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:10.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.746+0000 7f6dd1600640 1 -- 172.21.15.114:0/1421341076 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dbc0897e0 con 0x7f6dd415f9d0 2024-03-06T07:34:10.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.746+0000 7f6dd8a00640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b4d0 0x7f6db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:10.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.747+0000 7f6dd8a00640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b4d0 0x7f6db006d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4036000 tx=0x7f6dc403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:10.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.755+0000 7f6dd1600640 1 -- 172.21.15.114:0/1421341076 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6dd406ee70 con 0x7f6dd415f9d0 2024-03-06T07:34:10.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.954+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dd40cee80 con 0x7f6dd415f9d0 2024-03-06T07:34:10.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.956+0000 7f6dd1600640 1 -- 172.21.15.114:0/1421341076 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16675 (secure 0 0 0) 0x7f6dbc0585d0 con 0x7f6dd415f9d0 2024-03-06T07:34:10.958 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:10.958 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":108,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:07.954457+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:10.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.958+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b4d0 msgr2=0x7f6db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.958+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b4d0 0x7f6db006d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4036000 tx=0x7f6dc403b040 comp rx=0 tx=0).stop 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.958+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd415f9d0 msgr2=0x7f6dd415e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.958+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd415f9d0 0x7f6dd415e160 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00c400 tx=0x7f6dbc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.958+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 shutdown_connections 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.958+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dd415fe60 0x7f6dd415e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.959+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b4d0 0x7f6db006d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.959+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dd415f9d0 0x7f6dd415e160 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.959+0000 7f6ddb400640 1 --2- 172.21.15.114:0/1421341076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dd4144c00 0x7f6dd4136180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.959+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 >> 172.21.15.114:0/1421341076 conn(0x7f6dd40b1730 msgr2=0x7f6dd40ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.959+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 shutdown_connections 2024-03-06T07:34:10.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:10.959+0000 7f6ddb400640 1 -- 172.21.15.114:0/1421341076 wait complete. 2024-03-06T07:34:10.981 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:34:10.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:11.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.203+0000 7f165fe00640 1 Processor -- start 2024-03-06T07:34:11.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.203+0000 7f165fe00640 1 -- start start 2024-03-06T07:34:11.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.204+0000 7f165fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1660156320 0x7f166015f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.204+0000 7f165fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f166006afc0 0x7f166006b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.204+0000 7f165fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166015fed0 con 0x7f166015cd30 2024-03-06T07:34:11.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.204+0000 7f165fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1660071dc0 con 0x7f1660156320 2024-03-06T07:34:11.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.204+0000 7f165fe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166006b940 con 0x7f166006afc0 2024-03-06T07:34:11.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f165ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1660156320 0x7f166015f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f1657e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f166015cd30 0x7f16600ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55362/0 (socket says 172.21.15.114:55362) 2024-03-06T07:34:11.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f1657e00640 1 -- 172.21.15.114:0/1610601461 learned_addr learned my addr 172.21.15.114:0/1610601461 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:11.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f165f400640 1 --2- 172.21.15.114:0/1610601461 >> v2:172.21.15.177:3300/0 conn(0x7f166006afc0 0x7f166006b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f165ea00640 1 -- 172.21.15.114:0/1610601461 >> v2:172.21.15.177:3300/0 conn(0x7f166006afc0 msgr2=0x7f166006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f165ea00640 1 --2- 172.21.15.114:0/1610601461 >> v2:172.21.15.177:3300/0 conn(0x7f166006afc0 0x7f166006b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f165ea00640 1 -- 172.21.15.114:0/1610601461 >> v1:172.21.15.114:6789/0 conn(0x7f166015cd30 legacy=0x7f16600ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f165ea00640 1 -- 172.21.15.114:0/1610601461 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16601601b0 con 0x7f1660156320 2024-03-06T07:34:11.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.205+0000 7f165e000640 1 -- 172.21.15.114:0/1610601461 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150212381 0 0) 0x7f166015fed0 con 0x7f166015cd30 2024-03-06T07:34:11.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.206+0000 7f165ea00640 1 --2- 172.21.15.114:0/1610601461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f166015f8f0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f16600d1a40 tx=0x7f165002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608fb43672c364e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.207+0000 7f165e000640 1 -- 172.21.15.114:0/1610601461 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16500315a0 con 0x7f1660156320 2024-03-06T07:34:11.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.207+0000 7f165e000640 1 -- 172.21.15.114:0/1610601461 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1650037070 con 0x7f1660156320 2024-03-06T07:34:11.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.207+0000 7f165e000640 1 -- 172.21.15.114:0/1610601461 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650031960 con 0x7f1660156320 2024-03-06T07:34:11.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.207+0000 7f165fe00640 1 -- 172.21.15.114:0/1610601461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 msgr2=0x7f166015f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.207+0000 7f165fe00640 1 --2- 172.21.15.114:0/1610601461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f166015f8f0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f16600d1a40 tx=0x7f165002b840 comp rx=0 tx=0).stop 2024-03-06T07:34:11.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.208+0000 7f165fe00640 1 -- 172.21.15.114:0/1610601461 shutdown_connections 2024-03-06T07:34:11.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.208+0000 7f165fe00640 1 --2- 172.21.15.114:0/1610601461 >> v2:172.21.15.177:3300/0 conn(0x7f166006afc0 0x7f166006b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.208+0000 7f165fe00640 1 --2- 172.21.15.114:0/1610601461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f166015f8f0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.208+0000 7f165fe00640 1 -- 172.21.15.114:0/1610601461 >> 172.21.15.114:0/1610601461 conn(0x7f166006c0d0 msgr2=0x7f166006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.208+0000 7f165fe00640 1 -- 172.21.15.114:0/1610601461 shutdown_connections 2024-03-06T07:34:11.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.208+0000 7f165fe00640 1 -- 172.21.15.114:0/1610601461 wait complete. 2024-03-06T07:34:11.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.209+0000 7f165fe00640 1 Processor -- start 2024-03-06T07:34:11.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.209+0000 7f165fe00640 1 -- start start 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.210+0000 7f165fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166006afc0 0x7f166013bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.210+0000 7f165fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f16601e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.210+0000 7f165fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166015cd30 0x7f16601f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.210+0000 7f165fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1660156700 con 0x7f1660156320 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.210+0000 7f165fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1660160770 con 0x7f166015cd30 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.210+0000 7f165fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166015d1e0 con 0x7f166006afc0 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f16601e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f16601e9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41994/0 (socket says 172.21.15.114:41994) 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 -- 172.21.15.114:0/2289750895 learned_addr learned my addr 172.21.15.114:0/2289750895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f165f400640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166015cd30 0x7f16601f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 -- 172.21.15.114:0/2289750895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166006afc0 msgr2=0x7f166013bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166006afc0 0x7f166013bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 -- 172.21.15.114:0/2289750895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166015cd30 msgr2=0x7f16601f8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166015cd30 0x7f16601f8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 -- 172.21.15.114:0/2289750895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166013c5a0 con 0x7f1660156320 2024-03-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.211+0000 7f1657e00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f16601e9f70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f164800c430 tx=0x7f164800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.212+0000 7f1656a00640 1 -- 172.21.15.114:0/2289750895 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16480074e0 con 0x7f1660156320 2024-03-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.212+0000 7f1656a00640 1 -- 172.21.15.114:0/2289750895 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16480047e0 con 0x7f1660156320 2024-03-06T07:34:11.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.212+0000 7f1656a00640 1 -- 172.21.15.114:0/2289750895 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648004bd0 con 0x7f1660156320 2024-03-06T07:34:11.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.212+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166013c880 con 0x7f1660156320 2024-03-06T07:34:11.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.212+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16601f7a70 con 0x7f1660156320 2024-03-06T07:34:11.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.213+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166004f330 con 0x7f1660156320 2024-03-06T07:34:11.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.214+0000 7f1656a00640 1 -- 172.21.15.114:0/2289750895 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1648011070 con 0x7f1660156320 2024-03-06T07:34:11.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.215+0000 7f1656a00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f163c06b4d0 0x7f163c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.215+0000 7f1656a00640 1 -- 172.21.15.114:0/2289750895 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1648089e30 con 0x7f1660156320 2024-03-06T07:34:11.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.216+0000 7f165ea00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f163c06b4d0 0x7f163c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.216+0000 7f165ea00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f163c06b4d0 0x7f163c06d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1650036000 tx=0x7f1650030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.220+0000 7f1656a00640 1 -- 172.21.15.114:0/2289750895 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1648058d40 con 0x7f1660156320 2024-03-06T07:34:11.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.415+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16600cd3f0 con 0x7f163c06b4d0 2024-03-06T07:34:11.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.432+0000 7f1656a00640 1 -- 172.21.15.114:0/2289750895 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252480 (secure 0 0 0) 0x7f16600cd3f0 con 0x7f163c06b4d0 2024-03-06T07:34:11.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:11.439 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f163c06b4d0 msgr2=0x7f163c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f163c06b4d0 0x7f163c06d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1650036000 tx=0x7f1650030000 comp rx=0 tx=0).stop 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 msgr2=0x7f16601e9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f16601e9f70 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f164800c430 tx=0x7f164800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 shutdown_connections 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166015cd30 0x7f16601f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f163c06b4d0 0x7f163c06d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1660156320 0x7f16601e9f70 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 --2- 172.21.15.114:0/2289750895 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f166006afc0 0x7f166013bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 >> 172.21.15.114:0/2289750895 conn(0x7f166006c0d0 msgr2=0x7f16600d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 shutdown_connections 2024-03-06T07:34:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.437+0000 7f165fe00640 1 -- 172.21.15.114:0/2289750895 wait complete. 2024-03-06T07:34:11.457 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-03-06T07:34:11.212834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":294368,"kb_used_data":11624,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754680352,"statfs":{"total":773094113280,"available":772792680448,"internally_reserved":0,"allocated":11902976,"data_stored":5778380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":289394227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001133"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091068+0000","last_change":"2024-03-06T07:34:06.980348+0000","last_active":"2024-03-06T07:34:08.091068+0000","last_peered":"2024-03-06T07:34:08.091068+0000","last_clean":"2024-03-06T07:34:08.091068+0000","last_became_active":"2024-03-06T07:34:06.980057+0000","last_became_peered":"2024-03-06T07:34:06.980057+0000","last_unstale":"2024-03-06T07:34:08.091068+0000","last_undegraded":"2024-03-06T07:34:08.091068+0000","last_fullsized":"2024-03-06T07:34:08.091068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:59.404958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094610+0000","last_change":"2024-03-06T07:34:06.974427+0000","last_active":"2024-03-06T07:34:08.094610+0000","last_peered":"2024-03-06T07:34:08.094610+0000","last_clean":"2024-03-06T07:34:08.094610+0000","last_became_active":"2024-03-06T07:34:06.974095+0000","last_became_peered":"2024-03-06T07:34:06.974095+0000","last_unstale":"2024-03-06T07:34:08.094610+0000","last_undegraded":"2024-03-06T07:34:08.094610+0000","last_fullsized":"2024-03-06T07:34:08.094610+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:42.505055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093373+0000","last_change":"2024-03-06T07:34:06.980283+0000","last_active":"2024-03-06T07:34:08.093373+0000","last_peered":"2024-03-06T07:34:08.093373+0000","last_clean":"2024-03-06T07:34:08.093373+0000","last_became_active":"2024-03-06T07:34:06.979977+0000","last_became_peered":"2024-03-06T07:34:06.979977+0000","last_unstale":"2024-03-06T07:34:08.093373+0000","last_undegraded":"2024-03-06T07:34:08.093373+0000","last_fullsized":"2024-03-06T07:34:08.093373+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:00.835703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090946+0000","last_change":"2024-03-06T07:34:06.978020+0000","last_active":"2024-03-06T07:34:08.090946+0000","last_peered":"2024-03-06T07:34:08.090946+0000","last_clean":"2024-03-06T07:34:08.090946+0000","last_became_active":"2024-03-06T07:34:06.977678+0000","last_became_peered":"2024-03-06T07:34:06.977678+0000","last_unstale":"2024-03-06T07:34:08.090946+0000","last_undegraded":"2024-03-06T07:34:08.090946+0000","last_fullsized":"2024-03-06T07:34:08.090946+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:32.808970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975906+0000","last_change":"2024-03-06T07:34:06.964429+0000","last_active":"2024-03-06T07:34:07.975906+0000","last_peered":"2024-03-06T07:34:07.975906+0000","last_clean":"2024-03-06T07:34:07.975906+0000","last_became_active":"2024-03-06T07:34:06.963068+0000","last_became_peered":"2024-03-06T07:34:06.963068+0000","last_unstale":"2024-03-06T07:34:07.975906+0000","last_undegraded":"2024-03-06T07:34:07.975906+0000","last_fullsized":"2024-03-06T07:34:07.975906+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:34.784393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091110+0000","last_change":"2024-03-06T07:34:06.962728+0000","last_active":"2024-03-06T07:34:08.091110+0000","last_peered":"2024-03-06T07:34:08.091110+0000","last_clean":"2024-03-06T07:34:08.091110+0000","last_became_active":"2024-03-06T07:34:06.962447+0000","last_became_peered":"2024-03-06T07:34:06.962447+0000","last_unstale":"2024-03-06T07:34:08.091110+0000","last_undegraded":"2024-03-06T07:34:08.091110+0000","last_fullsized":"2024-03-06T07:34:08.091110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:27.958428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975608+0000","last_change":"2024-03-06T07:34:06.971628+0000","last_active":"2024-03-06T07:34:07.975608+0000","last_peered":"2024-03-06T07:34:07.975608+0000","last_clean":"2024-03-06T07:34:07.975608+0000","last_became_active":"2024-03-06T07:34:06.968306+0000","last_became_peered":"2024-03-06T07:34:06.968306+0000","last_unstale":"2024-03-06T07:34:07.975608+0000","last_undegraded":"2024-03-06T07:34:07.975608+0000","last_fullsized":"2024-03-06T07:34:07.975608+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:21.910276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092966+0000","last_change":"2024-03-06T07:34:06.970440+0000","last_active":"2024-03-06T07:34:08.092966+0000","last_peered":"2024-03-06T07:34:08.092966+0000","last_clean":"2024-03-06T07:34:08.092966+0000","last_became_active":"2024-03-06T07:34:06.969253+0000","last_became_peered":"2024-03-06T07:34:06.969253+0000","last_unstale":"2024-03-06T07:34:08.092966+0000","last_undegraded":"2024-03-06T07:34:08.092966+0000","last_fullsized":"2024-03-06T07:34:08.092966+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:21.032622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091240+0000","last_change":"2024-03-06T07:34:06.962825+0000","last_active":"2024-03-06T07:34:08.091240+0000","last_peered":"2024-03-06T07:34:08.091240+0000","last_clean":"2024-03-06T07:34:08.091240+0000","last_became_active":"2024-03-06T07:34:06.962520+0000","last_became_peered":"2024-03-06T07:34:06.962520+0000","last_unstale":"2024-03-06T07:34:08.091240+0000","last_undegraded":"2024-03-06T07:34:08.091240+0000","last_fullsized":"2024-03-06T07:34:08.091240+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:05.791347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090931+0000","last_change":"2024-03-06T07:34:06.962598+0000","last_active":"2024-03-06T07:34:08.090931+0000","last_peered":"2024-03-06T07:34:08.090931+0000","last_clean":"2024-03-06T07:34:08.090931+0000","last_became_active":"2024-03-06T07:34:06.962291+0000","last_became_peered":"2024-03-06T07:34:06.962291+0000","last_unstale":"2024-03-06T07:34:08.090931+0000","last_undegraded":"2024-03-06T07:34:08.090931+0000","last_fullsized":"2024-03-06T07:34:08.090931+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:51.588177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094332+0000","last_change":"2024-03-06T07:34:06.969957+0000","last_active":"2024-03-06T07:34:08.094332+0000","last_peered":"2024-03-06T07:34:08.094332+0000","last_clean":"2024-03-06T07:34:08.094332+0000","last_became_active":"2024-03-06T07:34:06.969600+0000","last_became_peered":"2024-03-06T07:34:06.969600+0000","last_unstale":"2024-03-06T07:34:08.094332+0000","last_undegraded":"2024-03-06T07:34:08.094332+0000","last_fullsized":"2024-03-06T07:34:08.094332+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:38.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091783+0000","last_change":"2024-03-06T07:34:06.961204+0000","last_active":"2024-03-06T07:34:08.091783+0000","last_peered":"2024-03-06T07:34:08.091783+0000","last_clean":"2024-03-06T07:34:08.091783+0000","last_became_active":"2024-03-06T07:34:06.960944+0000","last_became_peered":"2024-03-06T07:34:06.960944+0000","last_unstale":"2024-03-06T07:34:08.091783+0000","last_undegraded":"2024-03-06T07:34:08.091783+0000","last_fullsized":"2024-03-06T07:34:08.091783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:12.818634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094501+0000","last_change":"2024-03-06T07:34:06.975385+0000","last_active":"2024-03-06T07:34:08.094501+0000","last_peered":"2024-03-06T07:34:08.094501+0000","last_clean":"2024-03-06T07:34:08.094501+0000","last_became_active":"2024-03-06T07:34:06.973722+0000","last_became_peered":"2024-03-06T07:34:06.973722+0000","last_unstale":"2024-03-06T07:34:08.094501+0000","last_undegraded":"2024-03-06T07:34:08.094501+0000","last_fullsized":"2024-03-06T07:34:08.094501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:03.784636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092836+0000","last_change":"2024-03-06T07:34:06.971116+0000","last_active":"2024-03-06T07:34:08.092836+0000","last_peered":"2024-03-06T07:34:08.092836+0000","last_clean":"2024-03-06T07:34:08.092836+0000","last_became_active":"2024-03-06T07:34:06.970819+0000","last_became_peered":"2024-03-06T07:34:06.970819+0000","last_unstale":"2024-03-06T07:34:08.092836+0000","last_undegraded":"2024-03-06T07:34:08.092836+0000","last_fullsized":"2024-03-06T07:34:08.092836+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:42.747736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090905+0000","last_change":"2024-03-06T07:34:06.966120+0000","last_active":"2024-03-06T07:34:08.090905+0000","last_peered":"2024-03-06T07:34:08.090905+0000","last_clean":"2024-03-06T07:34:08.090905+0000","last_became_active":"2024-03-06T07:34:06.964755+0000","last_became_peered":"2024-03-06T07:34:06.964755+0000","last_unstale":"2024-03-06T07:34:08.090905+0000","last_undegraded":"2024-03-06T07:34:08.090905+0000","last_fullsized":"2024-03-06T07:34:08.090905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:35.739353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090659+0000","last_change":"2024-03-06T07:34:06.971995+0000","last_active":"2024-03-06T07:34:08.090659+0000","last_peered":"2024-03-06T07:34:08.090659+0000","last_clean":"2024-03-06T07:34:08.090659+0000","last_became_active":"2024-03-06T07:34:06.971630+0000","last_became_peered":"2024-03-06T07:34:06.971630+0000","last_unstale":"2024-03-06T07:34:08.090659+0000","last_undegraded":"2024-03-06T07:34:08.090659+0000","last_fullsized":"2024-03-06T07:34:08.090659+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:06.796524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094666+0000","last_change":"2024-03-06T07:34:06.978057+0000","last_active":"2024-03-06T07:34:08.094666+0000","last_peered":"2024-03-06T07:34:08.094666+0000","last_clean":"2024-03-06T07:34:08.094666+0000","last_became_active":"2024-03-06T07:34:06.977735+0000","last_became_peered":"2024-03-06T07:34:06.977735+0000","last_unstale":"2024-03-06T07:34:08.094666+0000","last_undegraded":"2024-03-06T07:34:08.094666+0000","last_fullsized":"2024-03-06T07:34:08.094666+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:02.686459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974591+0000","last_change":"2024-03-06T07:34:06.975554+0000","last_active":"2024-03-06T07:34:07.974591+0000","last_peered":"2024-03-06T07:34:07.974591+0000","last_clean":"2024-03-06T07:34:07.974591+0000","last_became_active":"2024-03-06T07:34:06.975134+0000","last_became_peered":"2024-03-06T07:34:06.975134+0000","last_unstale":"2024-03-06T07:34:07.974591+0000","last_undegraded":"2024-03-06T07:34:07.974591+0000","last_fullsized":"2024-03-06T07:34:07.974591+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:16.863083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093372+0000","last_change":"2024-03-06T07:34:06.982022+0000","last_active":"2024-03-06T07:34:08.093372+0000","last_peered":"2024-03-06T07:34:08.093372+0000","last_clean":"2024-03-06T07:34:08.093372+0000","last_became_active":"2024-03-06T07:34:06.981688+0000","last_became_peered":"2024-03-06T07:34:06.981688+0000","last_unstale":"2024-03-06T07:34:08.093372+0000","last_undegraded":"2024-03-06T07:34:08.093372+0000","last_fullsized":"2024-03-06T07:34:08.093372+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:07.771565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094217+0000","last_change":"2024-03-06T07:34:06.981255+0000","last_active":"2024-03-06T07:34:08.094217+0000","last_peered":"2024-03-06T07:34:08.094217+0000","last_clean":"2024-03-06T07:34:08.094217+0000","last_became_active":"2024-03-06T07:34:06.980870+0000","last_became_peered":"2024-03-06T07:34:06.980870+0000","last_unstale":"2024-03-06T07:34:08.094217+0000","last_undegraded":"2024-03-06T07:34:08.094217+0000","last_fullsized":"2024-03-06T07:34:08.094217+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:38.451040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093486+0000","last_change":"2024-03-06T07:34:06.975981+0000","last_active":"2024-03-06T07:34:08.093486+0000","last_peered":"2024-03-06T07:34:08.093486+0000","last_clean":"2024-03-06T07:34:08.093486+0000","last_became_active":"2024-03-06T07:34:06.970044+0000","last_became_peered":"2024-03-06T07:34:06.970044+0000","last_unstale":"2024-03-06T07:34:08.093486+0000","last_undegraded":"2024-03-06T07:34:08.093486+0000","last_fullsized":"2024-03-06T07:34:08.093486+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:16.863083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091821+0000","last_change":"2024-03-06T07:34:06.968727+0000","last_active":"2024-03-06T07:34:08.091821+0000","last_peered":"2024-03-06T07:34:08.091821+0000","last_clean":"2024-03-06T07:34:08.091821+0000","last_became_active":"2024-03-06T07:34:06.968452+0000","last_became_peered":"2024-03-06T07:34:06.968452+0000","last_unstale":"2024-03-06T07:34:08.091821+0000","last_undegraded":"2024-03-06T07:34:08.091821+0000","last_fullsized":"2024-03-06T07:34:08.091821+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:00.835703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092186+0000","last_change":"2024-03-06T07:34:06.975499+0000","last_active":"2024-03-06T07:34:08.092186+0000","last_peered":"2024-03-06T07:34:08.092186+0000","last_clean":"2024-03-06T07:34:08.092186+0000","last_became_active":"2024-03-06T07:34:06.975035+0000","last_became_peered":"2024-03-06T07:34:06.975035+0000","last_unstale":"2024-03-06T07:34:08.092186+0000","last_undegraded":"2024-03-06T07:34:08.092186+0000","last_fullsized":"2024-03-06T07:34:08.092186+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:00.320954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092292+0000","last_change":"2024-03-06T07:34:06.974754+0000","last_active":"2024-03-06T07:34:08.092292+0000","last_peered":"2024-03-06T07:34:08.092292+0000","last_clean":"2024-03-06T07:34:08.092292+0000","last_became_active":"2024-03-06T07:34:06.974499+0000","last_became_peered":"2024-03-06T07:34:06.974499+0000","last_unstale":"2024-03-06T07:34:08.092292+0000","last_undegraded":"2024-03-06T07:34:08.092292+0000","last_fullsized":"2024-03-06T07:34:08.092292+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:25.500332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.095013+0000","last_change":"2024-03-06T07:34:06.976030+0000","last_active":"2024-03-06T07:34:08.095013+0000","last_peered":"2024-03-06T07:34:08.095013+0000","last_clean":"2024-03-06T07:34:08.095013+0000","last_became_active":"2024-03-06T07:34:06.975614+0000","last_became_peered":"2024-03-06T07:34:06.975614+0000","last_unstale":"2024-03-06T07:34:08.095013+0000","last_undegraded":"2024-03-06T07:34:08.095013+0000","last_fullsized":"2024-03-06T07:34:08.095013+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:23.022586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091103+0000","last_change":"2024-03-06T07:34:06.965494+0000","last_active":"2024-03-06T07:34:08.091103+0000","last_peered":"2024-03-06T07:34:08.091103+0000","last_clean":"2024-03-06T07:34:08.091103+0000","last_became_active":"2024-03-06T07:34:06.965211+0000","last_became_peered":"2024-03-06T07:34:06.965211+0000","last_unstale":"2024-03-06T07:34:08.091103+0000","last_undegraded":"2024-03-06T07:34:08.091103+0000","last_fullsized":"2024-03-06T07:34:08.091103+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:24.119650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092307+0000","last_change":"2024-03-06T07:34:06.956224+0000","last_active":"2024-03-06T07:34:08.092307+0000","last_peered":"2024-03-06T07:34:08.092307+0000","last_clean":"2024-03-06T07:34:08.092307+0000","last_became_active":"2024-03-06T07:34:06.955939+0000","last_became_peered":"2024-03-06T07:34:06.955939+0000","last_unstale":"2024-03-06T07:34:08.092307+0000","last_undegraded":"2024-03-06T07:34:08.092307+0000","last_fullsized":"2024-03-06T07:34:08.092307+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:30.154986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092645+0000","last_change":"2024-03-06T07:34:06.982346+0000","last_active":"2024-03-06T07:34:08.092645+0000","last_peered":"2024-03-06T07:34:08.092645+0000","last_clean":"2024-03-06T07:34:08.092645+0000","last_became_active":"2024-03-06T07:34:06.981902+0000","last_became_peered":"2024-03-06T07:34:06.981902+0000","last_unstale":"2024-03-06T07:34:08.092645+0000","last_undegraded":"2024-03-06T07:34:08.092645+0000","last_fullsized":"2024-03-06T07:34:08.092645+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:03.784636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093909+0000","last_change":"2024-03-06T07:34:06.982356+0000","last_active":"2024-03-06T07:34:08.093909+0000","last_peered":"2024-03-06T07:34:08.093909+0000","last_clean":"2024-03-06T07:34:08.093909+0000","last_became_active":"2024-03-06T07:34:06.981943+0000","last_became_peered":"2024-03-06T07:34:06.981943+0000","last_unstale":"2024-03-06T07:34:08.093909+0000","last_undegraded":"2024-03-06T07:34:08.093909+0000","last_fullsized":"2024-03-06T07:34:08.093909+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:58.527981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.977222+0000","last_change":"2024-03-06T07:34:06.963610+0000","last_active":"2024-03-06T07:34:07.977222+0000","last_peered":"2024-03-06T07:34:07.977222+0000","last_clean":"2024-03-06T07:34:07.977222+0000","last_became_active":"2024-03-06T07:34:06.963345+0000","last_became_peered":"2024-03-06T07:34:06.963345+0000","last_unstale":"2024-03-06T07:34:07.977222+0000","last_undegraded":"2024-03-06T07:34:07.977222+0000","last_fullsized":"2024-03-06T07:34:07.977222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:47.221769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091341+0000","last_change":"2024-03-06T07:34:06.965421+0000","last_active":"2024-03-06T07:34:08.091341+0000","last_peered":"2024-03-06T07:34:08.091341+0000","last_clean":"2024-03-06T07:34:08.091341+0000","last_became_active":"2024-03-06T07:34:06.965152+0000","last_became_peered":"2024-03-06T07:34:06.965152+0000","last_unstale":"2024-03-06T07:34:08.091341+0000","last_undegraded":"2024-03-06T07:34:08.091341+0000","last_fullsized":"2024-03-06T07:34:08.091341+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:35.990446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090506+0000","last_change":"2024-03-06T07:34:06.962315+0000","last_active":"2024-03-06T07:34:08.090506+0000","last_peered":"2024-03-06T07:34:08.090506+0000","last_clean":"2024-03-06T07:34:08.090506+0000","last_became_active":"2024-03-06T07:34:06.961636+0000","last_became_peered":"2024-03-06T07:34:06.961636+0000","last_unstale":"2024-03-06T07:34:08.090506+0000","last_undegraded":"2024-03-06T07:34:08.090506+0000","last_fullsized":"2024-03-06T07:34:08.090506+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:52.024864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092452+0000","last_change":"2024-03-06T07:34:06.966572+0000","last_active":"2024-03-06T07:34:08.092452+0000","last_peered":"2024-03-06T07:34:08.092452+0000","last_clean":"2024-03-06T07:34:08.092452+0000","last_became_active":"2024-03-06T07:34:06.964082+0000","last_became_peered":"2024-03-06T07:34:06.964082+0000","last_unstale":"2024-03-06T07:34:08.092452+0000","last_undegraded":"2024-03-06T07:34:08.092452+0000","last_fullsized":"2024-03-06T07:34:08.092452+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:50.040210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976296+0000","last_change":"2024-03-06T07:34:05.969455+0000","last_active":"2024-03-06T07:34:07.976296+0000","last_peered":"2024-03-06T07:34:07.976296+0000","last_clean":"2024-03-06T07:34:07.976296+0000","last_became_active":"2024-03-06T07:34:05.969045+0000","last_became_peered":"2024-03-06T07:34:05.969045+0000","last_unstale":"2024-03-06T07:34:07.976296+0000","last_undegraded":"2024-03-06T07:34:07.976296+0000","last_fullsized":"2024-03-06T07:34:07.976296+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:51.009846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976755+0000","last_change":"2024-03-06T07:34:06.981148+0000","last_active":"2024-03-06T07:34:07.976755+0000","last_peered":"2024-03-06T07:34:07.976755+0000","last_clean":"2024-03-06T07:34:07.976755+0000","last_became_active":"2024-03-06T07:34:06.980715+0000","last_became_peered":"2024-03-06T07:34:06.980715+0000","last_unstale":"2024-03-06T07:34:07.976755+0000","last_undegraded":"2024-03-06T07:34:07.976755+0000","last_fullsized":"2024-03-06T07:34:07.976755+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:38.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094540+0000","last_change":"2024-03-06T07:34:05.968273+0000","last_active":"2024-03-06T07:34:08.094540+0000","last_peered":"2024-03-06T07:34:08.094540+0000","last_clean":"2024-03-06T07:34:08.094540+0000","last_became_active":"2024-03-06T07:34:05.967984+0000","last_became_peered":"2024-03-06T07:34:05.967984+0000","last_unstale":"2024-03-06T07:34:08.094540+0000","last_undegraded":"2024-03-06T07:34:08.094540+0000","last_fullsized":"2024-03-06T07:34:08.094540+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:43.892727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091859+0000","last_change":"2024-03-06T07:34:06.965490+0000","last_active":"2024-03-06T07:34:08.091859+0000","last_peered":"2024-03-06T07:34:08.091859+0000","last_clean":"2024-03-06T07:34:08.091859+0000","last_became_active":"2024-03-06T07:34:06.965229+0000","last_became_peered":"2024-03-06T07:34:06.965229+0000","last_unstale":"2024-03-06T07:34:08.091859+0000","last_undegraded":"2024-03-06T07:34:08.091859+0000","last_fullsized":"2024-03-06T07:34:08.091859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:49.896621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975792+0000","last_change":"2024-03-06T07:34:05.967179+0000","last_active":"2024-03-06T07:34:07.975792+0000","last_peered":"2024-03-06T07:34:07.975792+0000","last_clean":"2024-03-06T07:34:07.975792+0000","last_became_active":"2024-03-06T07:34:05.966765+0000","last_became_peered":"2024-03-06T07:34:05.966765+0000","last_unstale":"2024-03-06T07:34:07.975792+0000","last_undegraded":"2024-03-06T07:34:07.975792+0000","last_fullsized":"2024-03-06T07:34:07.975792+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:05.832514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093838+0000","last_change":"2024-03-06T07:34:06.976191+0000","last_active":"2024-03-06T07:34:08.093838+0000","last_peered":"2024-03-06T07:34:08.093838+0000","last_clean":"2024-03-06T07:34:08.093838+0000","last_became_active":"2024-03-06T07:34:06.975820+0000","last_became_peered":"2024-03-06T07:34:06.975820+0000","last_unstale":"2024-03-06T07:34:08.093838+0000","last_undegraded":"2024-03-06T07:34:08.093838+0000","last_fullsized":"2024-03-06T07:34:08.093838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:07.961805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975845+0000","last_change":"2024-03-06T07:34:05.969221+0000","last_active":"2024-03-06T07:34:07.975845+0000","last_peered":"2024-03-06T07:34:07.975845+0000","last_clean":"2024-03-06T07:34:07.975845+0000","last_became_active":"2024-03-06T07:34:05.968143+0000","last_became_peered":"2024-03-06T07:34:05.968143+0000","last_unstale":"2024-03-06T07:34:07.975845+0000","last_undegraded":"2024-03-06T07:34:07.975845+0000","last_fullsized":"2024-03-06T07:34:07.975845+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:11.728397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090632+0000","last_change":"2024-03-06T07:34:06.963406+0000","last_active":"2024-03-06T07:34:08.090632+0000","last_peered":"2024-03-06T07:34:08.090632+0000","last_clean":"2024-03-06T07:34:08.090632+0000","last_became_active":"2024-03-06T07:34:06.963156+0000","last_became_peered":"2024-03-06T07:34:06.963156+0000","last_unstale":"2024-03-06T07:34:08.090632+0000","last_undegraded":"2024-03-06T07:34:08.090632+0000","last_fullsized":"2024-03-06T07:34:08.090632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:21.865435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975298+0000","last_change":"2024-03-06T07:34:05.965076+0000","last_active":"2024-03-06T07:34:07.975298+0000","last_peered":"2024-03-06T07:34:07.975298+0000","last_clean":"2024-03-06T07:34:07.975298+0000","last_became_active":"2024-03-06T07:34:05.964531+0000","last_became_peered":"2024-03-06T07:34:05.964531+0000","last_unstale":"2024-03-06T07:34:07.975298+0000","last_undegraded":"2024-03-06T07:34:07.975298+0000","last_fullsized":"2024-03-06T07:34:07.975298+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:13.214810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092518+0000","last_change":"2024-03-06T07:34:06.976245+0000","last_active":"2024-03-06T07:34:08.092518+0000","last_peered":"2024-03-06T07:34:08.092518+0000","last_clean":"2024-03-06T07:34:08.092518+0000","last_became_active":"2024-03-06T07:34:06.975765+0000","last_became_peered":"2024-03-06T07:34:06.975765+0000","last_unstale":"2024-03-06T07:34:08.092518+0000","last_undegraded":"2024-03-06T07:34:08.092518+0000","last_fullsized":"2024-03-06T07:34:08.092518+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:12.422685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092101+0000","last_change":"2024-03-06T07:34:05.964609+0000","last_active":"2024-03-06T07:34:08.092101+0000","last_peered":"2024-03-06T07:34:08.092101+0000","last_clean":"2024-03-06T07:34:08.092101+0000","last_became_active":"2024-03-06T07:34:05.959755+0000","last_became_peered":"2024-03-06T07:34:05.959755+0000","last_unstale":"2024-03-06T07:34:08.092101+0000","last_undegraded":"2024-03-06T07:34:08.092101+0000","last_fullsized":"2024-03-06T07:34:08.092101+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:22.525554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090264+0000","last_change":"2024-03-06T07:34:06.960818+0000","last_active":"2024-03-06T07:34:08.090264+0000","last_peered":"2024-03-06T07:34:08.090264+0000","last_clean":"2024-03-06T07:34:08.090264+0000","last_became_active":"2024-03-06T07:34:06.960482+0000","last_became_peered":"2024-03-06T07:34:06.960482+0000","last_unstale":"2024-03-06T07:34:08.090264+0000","last_undegraded":"2024-03-06T07:34:08.090264+0000","last_fullsized":"2024-03-06T07:34:08.090264+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:00.293386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093848+0000","last_change":"2024-03-06T07:34:05.961648+0000","last_active":"2024-03-06T07:34:08.093848+0000","last_peered":"2024-03-06T07:34:08.093848+0000","last_clean":"2024-03-06T07:34:08.093848+0000","last_became_active":"2024-03-06T07:34:05.956450+0000","last_became_peered":"2024-03-06T07:34:05.956450+0000","last_unstale":"2024-03-06T07:34:08.093848+0000","last_undegraded":"2024-03-06T07:34:08.093848+0000","last_fullsized":"2024-03-06T07:34:08.093848+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:46.206751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975455+0000","last_change":"2024-03-06T07:34:06.968926+0000","last_active":"2024-03-06T07:34:07.975455+0000","last_peered":"2024-03-06T07:34:07.975455+0000","last_clean":"2024-03-06T07:34:07.975455+0000","last_became_active":"2024-03-06T07:34:06.967912+0000","last_became_peered":"2024-03-06T07:34:06.967912+0000","last_unstale":"2024-03-06T07:34:07.975455+0000","last_undegraded":"2024-03-06T07:34:07.975455+0000","last_fullsized":"2024-03-06T07:34:07.975455+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:28:43.430470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090776+0000","last_change":"2024-03-06T07:34:05.965174+0000","last_active":"2024-03-06T07:34:08.090776+0000","last_peered":"2024-03-06T07:34:08.090776+0000","last_clean":"2024-03-06T07:34:08.090776+0000","last_became_active":"2024-03-06T07:34:05.964635+0000","last_became_peered":"2024-03-06T07:34:05.964635+0000","last_unstale":"2024-03-06T07:34:08.090776+0000","last_undegraded":"2024-03-06T07:34:08.090776+0000","last_fullsized":"2024-03-06T07:34:08.090776+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:41.490037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976497+0000","last_change":"2024-03-06T07:34:06.963087+0000","last_active":"2024-03-06T07:34:07.976497+0000","last_peered":"2024-03-06T07:34:07.976497+0000","last_clean":"2024-03-06T07:34:07.976497+0000","last_became_active":"2024-03-06T07:34:06.962318+0000","last_became_peered":"2024-03-06T07:34:06.962318+0000","last_unstale":"2024-03-06T07:34:07.976497+0000","last_undegraded":"2024-03-06T07:34:07.976497+0000","last_fullsized":"2024-03-06T07:34:07.976497+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:06.221782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975843+0000","last_change":"2024-03-06T07:34:05.967458+0000","last_active":"2024-03-06T07:34:07.975843+0000","last_peered":"2024-03-06T07:34:07.975843+0000","last_clean":"2024-03-06T07:34:07.975843+0000","last_became_active":"2024-03-06T07:34:05.966747+0000","last_became_peered":"2024-03-06T07:34:05.966747+0000","last_unstale":"2024-03-06T07:34:07.975843+0000","last_undegraded":"2024-03-06T07:34:07.975843+0000","last_fullsized":"2024-03-06T07:34:07.975843+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:24.198154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976897+0000","last_change":"2024-03-06T07:34:06.970098+0000","last_active":"2024-03-06T07:34:07.976897+0000","last_peered":"2024-03-06T07:34:07.976897+0000","last_clean":"2024-03-06T07:34:07.976897+0000","last_became_active":"2024-03-06T07:34:06.965679+0000","last_became_peered":"2024-03-06T07:34:06.965679+0000","last_unstale":"2024-03-06T07:34:07.976897+0000","last_undegraded":"2024-03-06T07:34:07.976897+0000","last_fullsized":"2024-03-06T07:34:07.976897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:42.505055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092821+0000","last_change":"2024-03-06T07:34:05.959497+0000","last_active":"2024-03-06T07:34:08.092821+0000","last_peered":"2024-03-06T07:34:08.092821+0000","last_clean":"2024-03-06T07:34:08.092821+0000","last_became_active":"2024-03-06T07:34:05.958782+0000","last_became_peered":"2024-03-06T07:34:05.958782+0000","last_unstale":"2024-03-06T07:34:08.092821+0000","last_undegraded":"2024-03-06T07:34:08.092821+0000","last_fullsized":"2024-03-06T07:34:08.092821+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:49.191406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092646+0000","last_change":"2024-03-06T07:34:06.980693+0000","last_active":"2024-03-06T07:34:08.092646+0000","last_peered":"2024-03-06T07:34:08.092646+0000","last_clean":"2024-03-06T07:34:08.092646+0000","last_became_active":"2024-03-06T07:34:06.979142+0000","last_became_peered":"2024-03-06T07:34:06.979142+0000","last_unstale":"2024-03-06T07:34:08.092646+0000","last_undegraded":"2024-03-06T07:34:08.092646+0000","last_fullsized":"2024-03-06T07:34:08.092646+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.851707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091784+0000","last_change":"2024-03-06T07:34:05.957938+0000","last_active":"2024-03-06T07:34:08.091784+0000","last_peered":"2024-03-06T07:34:08.091784+0000","last_clean":"2024-03-06T07:34:08.091784+0000","last_became_active":"2024-03-06T07:34:05.957639+0000","last_became_peered":"2024-03-06T07:34:05.957639+0000","last_unstale":"2024-03-06T07:34:08.091784+0000","last_undegraded":"2024-03-06T07:34:08.091784+0000","last_fullsized":"2024-03-06T07:34:08.091784+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:59.518470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976186+0000","last_change":"2024-03-06T07:34:06.974394+0000","last_active":"2024-03-06T07:34:07.976186+0000","last_peered":"2024-03-06T07:34:07.976186+0000","last_clean":"2024-03-06T07:34:07.976186+0000","last_became_active":"2024-03-06T07:34:06.973996+0000","last_became_peered":"2024-03-06T07:34:06.973996+0000","last_unstale":"2024-03-06T07:34:07.976186+0000","last_undegraded":"2024-03-06T07:34:07.976186+0000","last_fullsized":"2024-03-06T07:34:07.976186+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:52.658511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092109+0000","last_change":"2024-03-06T07:34:05.953201+0000","last_active":"2024-03-06T07:34:08.092109+0000","last_peered":"2024-03-06T07:34:08.092109+0000","last_clean":"2024-03-06T07:34:08.092109+0000","last_became_active":"2024-03-06T07:34:05.952872+0000","last_became_peered":"2024-03-06T07:34:05.952872+0000","last_unstale":"2024-03-06T07:34:08.092109+0000","last_undegraded":"2024-03-06T07:34:08.092109+0000","last_fullsized":"2024-03-06T07:34:08.092109+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:20.017604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091981+0000","last_change":"2024-03-06T07:34:06.965398+0000","last_active":"2024-03-06T07:34:08.091981+0000","last_peered":"2024-03-06T07:34:08.091981+0000","last_clean":"2024-03-06T07:34:08.091981+0000","last_became_active":"2024-03-06T07:34:06.965098+0000","last_became_peered":"2024-03-06T07:34:06.965098+0000","last_unstale":"2024-03-06T07:34:08.091981+0000","last_undegraded":"2024-03-06T07:34:08.091981+0000","last_fullsized":"2024-03-06T07:34:08.091981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:40.040686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091396+0000","last_change":"2024-03-06T07:34:05.968178+0000","last_active":"2024-03-06T07:34:08.091396+0000","last_peered":"2024-03-06T07:34:08.091396+0000","last_clean":"2024-03-06T07:34:08.091396+0000","last_became_active":"2024-03-06T07:34:05.967804+0000","last_became_peered":"2024-03-06T07:34:05.967804+0000","last_unstale":"2024-03-06T07:34:08.091396+0000","last_undegraded":"2024-03-06T07:34:08.091396+0000","last_fullsized":"2024-03-06T07:34:08.091396+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:33.769374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092467+0000","last_change":"2024-03-06T07:34:06.983788+0000","last_active":"2024-03-06T07:34:08.092467+0000","last_peered":"2024-03-06T07:34:08.092467+0000","last_clean":"2024-03-06T07:34:08.092467+0000","last_became_active":"2024-03-06T07:34:06.983526+0000","last_became_peered":"2024-03-06T07:34:06.983526+0000","last_unstale":"2024-03-06T07:34:08.092467+0000","last_undegraded":"2024-03-06T07:34:08.092467+0000","last_fullsized":"2024-03-06T07:34:08.092467+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:07.961805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974738+0000","last_change":"2024-03-06T07:34:05.967071+0000","last_active":"2024-03-06T07:34:07.974738+0000","last_peered":"2024-03-06T07:34:07.974738+0000","last_clean":"2024-03-06T07:34:07.974738+0000","last_became_active":"2024-03-06T07:34:05.966768+0000","last_became_peered":"2024-03-06T07:34:05.966768+0000","last_unstale":"2024-03-06T07:34:07.974738+0000","last_undegraded":"2024-03-06T07:34:07.974738+0000","last_fullsized":"2024-03-06T07:34:07.974738+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:07.414354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091509+0000","last_change":"2024-03-06T07:34:06.981124+0000","last_active":"2024-03-06T07:34:08.091509+0000","last_peered":"2024-03-06T07:34:08.091509+0000","last_clean":"2024-03-06T07:34:08.091509+0000","last_became_active":"2024-03-06T07:34:06.980817+0000","last_became_peered":"2024-03-06T07:34:06.980817+0000","last_unstale":"2024-03-06T07:34:08.091509+0000","last_undegraded":"2024-03-06T07:34:08.091509+0000","last_fullsized":"2024-03-06T07:34:08.091509+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:58.738464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975580+0000","last_change":"2024-03-06T07:34:05.968981+0000","last_active":"2024-03-06T07:34:07.975580+0000","last_peered":"2024-03-06T07:34:07.975580+0000","last_clean":"2024-03-06T07:34:07.975580+0000","last_became_active":"2024-03-06T07:34:05.968157+0000","last_became_peered":"2024-03-06T07:34:05.968157+0000","last_unstale":"2024-03-06T07:34:07.975580+0000","last_undegraded":"2024-03-06T07:34:07.975580+0000","last_fullsized":"2024-03-06T07:34:07.975580+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:35.894352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976128+0000","last_change":"2024-03-06T07:34:06.965019+0000","last_active":"2024-03-06T07:34:07.976128+0000","last_peered":"2024-03-06T07:34:07.976128+0000","last_clean":"2024-03-06T07:34:07.976128+0000","last_became_active":"2024-03-06T07:34:06.964694+0000","last_became_peered":"2024-03-06T07:34:06.964694+0000","last_unstale":"2024-03-06T07:34:07.976128+0000","last_undegraded":"2024-03-06T07:34:07.976128+0000","last_fullsized":"2024-03-06T07:34:07.976128+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:59.916231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091536+0000","last_change":"2024-03-06T07:34:05.964919+0000","last_active":"2024-03-06T07:34:08.091536+0000","last_peered":"2024-03-06T07:34:08.091536+0000","last_clean":"2024-03-06T07:34:08.091536+0000","last_became_active":"2024-03-06T07:34:05.964586+0000","last_became_peered":"2024-03-06T07:34:05.964586+0000","last_unstale":"2024-03-06T07:34:08.091536+0000","last_undegraded":"2024-03-06T07:34:08.091536+0000","last_fullsized":"2024-03-06T07:34:08.091536+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:57.723446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092757+0000","last_change":"2024-03-06T07:34:06.964726+0000","last_active":"2024-03-06T07:34:08.092757+0000","last_peered":"2024-03-06T07:34:08.092757+0000","last_clean":"2024-03-06T07:34:08.092757+0000","last_became_active":"2024-03-06T07:34:06.964448+0000","last_became_peered":"2024-03-06T07:34:06.964448+0000","last_unstale":"2024-03-06T07:34:08.092757+0000","last_undegraded":"2024-03-06T07:34:08.092757+0000","last_fullsized":"2024-03-06T07:34:08.092757+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:06.162248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976091+0000","last_change":"2024-03-06T07:34:05.960026+0000","last_active":"2024-03-06T07:34:07.976091+0000","last_peered":"2024-03-06T07:34:07.976091+0000","last_clean":"2024-03-06T07:34:07.976091+0000","last_became_active":"2024-03-06T07:34:05.959759+0000","last_became_peered":"2024-03-06T07:34:05.959759+0000","last_unstale":"2024-03-06T07:34:07.976091+0000","last_undegraded":"2024-03-06T07:34:07.976091+0000","last_fullsized":"2024-03-06T07:34:07.976091+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:40.961829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094751+0000","last_change":"2024-03-06T07:34:06.985008+0000","last_active":"2024-03-06T07:34:08.094751+0000","last_peered":"2024-03-06T07:34:08.094751+0000","last_clean":"2024-03-06T07:34:08.094751+0000","last_became_active":"2024-03-06T07:34:06.984463+0000","last_became_peered":"2024-03-06T07:34:06.984463+0000","last_unstale":"2024-03-06T07:34:08.094751+0000","last_undegraded":"2024-03-06T07:34:08.094751+0000","last_fullsized":"2024-03-06T07:34:08.094751+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.851707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975573+0000","last_change":"2024-03-06T07:34:05.963002+0000","last_active":"2024-03-06T07:34:07.975573+0000","last_peered":"2024-03-06T07:34:07.975573+0000","last_clean":"2024-03-06T07:34:07.975573+0000","last_became_active":"2024-03-06T07:34:05.959894+0000","last_became_peered":"2024-03-06T07:34:05.959894+0000","last_unstale":"2024-03-06T07:34:07.975573+0000","last_undegraded":"2024-03-06T07:34:07.975573+0000","last_fullsized":"2024-03-06T07:34:07.975573+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:03.567830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093840+0000","last_change":"2024-03-06T07:34:06.983682+0000","last_active":"2024-03-06T07:34:08.093840+0000","last_peered":"2024-03-06T07:34:08.093840+0000","last_clean":"2024-03-06T07:34:08.093840+0000","last_became_active":"2024-03-06T07:34:06.981655+0000","last_became_peered":"2024-03-06T07:34:06.981655+0000","last_unstale":"2024-03-06T07:34:08.093840+0000","last_undegraded":"2024-03-06T07:34:08.093840+0000","last_fullsized":"2024-03-06T07:34:08.093840+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:25.500332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975522+0000","last_change":"2024-03-06T07:34:05.967095+0000","last_active":"2024-03-06T07:34:07.975522+0000","last_peered":"2024-03-06T07:34:07.975522+0000","last_clean":"2024-03-06T07:34:07.975522+0000","last_became_active":"2024-03-06T07:34:05.966654+0000","last_became_peered":"2024-03-06T07:34:05.966654+0000","last_unstale":"2024-03-06T07:34:07.975522+0000","last_undegraded":"2024-03-06T07:34:07.975522+0000","last_fullsized":"2024-03-06T07:34:07.975522+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:26.943410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093173+0000","last_change":"2024-03-06T07:34:06.980744+0000","last_active":"2024-03-06T07:34:08.093173+0000","last_peered":"2024-03-06T07:34:08.093173+0000","last_clean":"2024-03-06T07:34:08.093173+0000","last_became_active":"2024-03-06T07:34:06.980025+0000","last_became_peered":"2024-03-06T07:34:06.980025+0000","last_unstale":"2024-03-06T07:34:08.093173+0000","last_undegraded":"2024-03-06T07:34:08.093173+0000","last_fullsized":"2024-03-06T07:34:08.093173+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:58.738464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091506+0000","last_change":"2024-03-06T07:34:05.963203+0000","last_active":"2024-03-06T07:34:08.091506+0000","last_peered":"2024-03-06T07:34:08.091506+0000","last_clean":"2024-03-06T07:34:08.091506+0000","last_became_active":"2024-03-06T07:34:05.962888+0000","last_became_peered":"2024-03-06T07:34:05.962888+0000","last_unstale":"2024-03-06T07:34:08.091506+0000","last_undegraded":"2024-03-06T07:34:08.091506+0000","last_fullsized":"2024-03-06T07:34:08.091506+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:58.901213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090552+0000","last_change":"2024-03-06T07:34:06.976759+0000","last_active":"2024-03-06T07:34:08.090552+0000","last_peered":"2024-03-06T07:34:08.090552+0000","last_clean":"2024-03-06T07:34:08.090552+0000","last_became_active":"2024-03-06T07:34:06.976024+0000","last_became_peered":"2024-03-06T07:34:06.976024+0000","last_unstale":"2024-03-06T07:34:08.090552+0000","last_undegraded":"2024-03-06T07:34:08.090552+0000","last_fullsized":"2024-03-06T07:34:08.090552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:48.876510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975690+0000","last_change":"2024-03-06T07:34:05.968901+0000","last_active":"2024-03-06T07:34:07.975690+0000","last_peered":"2024-03-06T07:34:07.975690+0000","last_clean":"2024-03-06T07:34:07.975690+0000","last_became_active":"2024-03-06T07:34:05.968227+0000","last_became_peered":"2024-03-06T07:34:05.968227+0000","last_unstale":"2024-03-06T07:34:07.975690+0000","last_undegraded":"2024-03-06T07:34:07.975690+0000","last_fullsized":"2024-03-06T07:34:07.975690+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:01.527118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093054+0000","last_change":"2024-03-06T07:34:06.984129+0000","last_active":"2024-03-06T07:34:08.093054+0000","last_peered":"2024-03-06T07:34:08.093054+0000","last_clean":"2024-03-06T07:34:08.093054+0000","last_became_active":"2024-03-06T07:34:06.983778+0000","last_became_peered":"2024-03-06T07:34:06.983778+0000","last_unstale":"2024-03-06T07:34:08.093054+0000","last_undegraded":"2024-03-06T07:34:08.093054+0000","last_fullsized":"2024-03-06T07:34:08.093054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:00.533488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091002+0000","last_change":"2024-03-06T07:34:05.961806+0000","last_active":"2024-03-06T07:34:08.091002+0000","last_peered":"2024-03-06T07:34:08.091002+0000","last_clean":"2024-03-06T07:34:08.091002+0000","last_became_active":"2024-03-06T07:34:05.961553+0000","last_became_peered":"2024-03-06T07:34:05.961553+0000","last_unstale":"2024-03-06T07:34:08.091002+0000","last_undegraded":"2024-03-06T07:34:08.091002+0000","last_fullsized":"2024-03-06T07:34:08.091002+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:37.032238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091684+0000","last_change":"2024-03-06T07:34:06.969935+0000","last_active":"2024-03-06T07:34:08.091684+0000","last_peered":"2024-03-06T07:34:08.091684+0000","last_clean":"2024-03-06T07:34:08.091684+0000","last_became_active":"2024-03-06T07:34:06.969402+0000","last_became_peered":"2024-03-06T07:34:06.969402+0000","last_unstale":"2024-03-06T07:34:08.091684+0000","last_undegraded":"2024-03-06T07:34:08.091684+0000","last_fullsized":"2024-03-06T07:34:08.091684+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:07.771565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090899+0000","last_change":"2024-03-06T07:34:05.961371+0000","last_active":"2024-03-06T07:34:08.090899+0000","last_peered":"2024-03-06T07:34:08.090899+0000","last_clean":"2024-03-06T07:34:08.090899+0000","last_became_active":"2024-03-06T07:34:05.961031+0000","last_became_peered":"2024-03-06T07:34:05.961031+0000","last_unstale":"2024-03-06T07:34:08.090899+0000","last_undegraded":"2024-03-06T07:34:08.090899+0000","last_fullsized":"2024-03-06T07:34:08.090899+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:22.007568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974808+0000","last_change":"2024-03-06T07:34:06.967988+0000","last_active":"2024-03-06T07:34:07.974808+0000","last_peered":"2024-03-06T07:34:07.974808+0000","last_clean":"2024-03-06T07:34:07.974808+0000","last_became_active":"2024-03-06T07:34:06.967454+0000","last_became_peered":"2024-03-06T07:34:06.967454+0000","last_unstale":"2024-03-06T07:34:07.974808+0000","last_undegraded":"2024-03-06T07:34:07.974808+0000","last_fullsized":"2024-03-06T07:34:07.974808+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:58.527981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092942+0000","last_change":"2024-03-06T07:34:05.954142+0000","last_active":"2024-03-06T07:34:08.092942+0000","last_peered":"2024-03-06T07:34:08.092942+0000","last_clean":"2024-03-06T07:34:08.092942+0000","last_became_active":"2024-03-06T07:34:05.953849+0000","last_became_peered":"2024-03-06T07:34:05.953849+0000","last_unstale":"2024-03-06T07:34:08.092942+0000","last_undegraded":"2024-03-06T07:34:08.092942+0000","last_fullsized":"2024-03-06T07:34:08.092942+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:11.407667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091412+0000","last_change":"2024-03-06T07:34:06.979414+0000","last_active":"2024-03-06T07:34:08.091412+0000","last_peered":"2024-03-06T07:34:08.091412+0000","last_clean":"2024-03-06T07:34:08.091412+0000","last_became_active":"2024-03-06T07:34:06.979138+0000","last_became_peered":"2024-03-06T07:34:06.979138+0000","last_unstale":"2024-03-06T07:34:08.091412+0000","last_undegraded":"2024-03-06T07:34:08.091412+0000","last_fullsized":"2024-03-06T07:34:08.091412+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:20.886582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974896+0000","last_change":"2024-03-06T07:34:05.964059+0000","last_active":"2024-03-06T07:34:07.974896+0000","last_peered":"2024-03-06T07:34:07.974896+0000","last_clean":"2024-03-06T07:34:07.974896+0000","last_became_active":"2024-03-06T07:34:05.963731+0000","last_became_peered":"2024-03-06T07:34:05.963731+0000","last_unstale":"2024-03-06T07:34:07.974896+0000","last_undegraded":"2024-03-06T07:34:07.974896+0000","last_fullsized":"2024-03-06T07:34:07.974896+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:43.892727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091978+0000","last_change":"2024-03-06T07:34:06.963832+0000","last_active":"2024-03-06T07:34:08.091978+0000","last_peered":"2024-03-06T07:34:08.091978+0000","last_clean":"2024-03-06T07:34:08.091978+0000","last_became_active":"2024-03-06T07:34:06.963540+0000","last_became_peered":"2024-03-06T07:34:06.963540+0000","last_unstale":"2024-03-06T07:34:08.091978+0000","last_undegraded":"2024-03-06T07:34:08.091978+0000","last_fullsized":"2024-03-06T07:34:08.091978+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:35.999636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091874+0000","last_change":"2024-03-06T07:34:05.960922+0000","last_active":"2024-03-06T07:34:08.091874+0000","last_peered":"2024-03-06T07:34:08.091874+0000","last_clean":"2024-03-06T07:34:08.091874+0000","last_became_active":"2024-03-06T07:34:05.960593+0000","last_became_peered":"2024-03-06T07:34:05.960593+0000","last_unstale":"2024-03-06T07:34:08.091874+0000","last_undegraded":"2024-03-06T07:34:08.091874+0000","last_fullsized":"2024-03-06T07:34:08.091874+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:41.732718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975477+0000","last_change":"2024-03-06T07:34:06.956435+0000","last_active":"2024-03-06T07:34:07.975477+0000","last_peered":"2024-03-06T07:34:07.975477+0000","last_clean":"2024-03-06T07:34:07.975477+0000","last_became_active":"2024-03-06T07:34:06.955738+0000","last_became_peered":"2024-03-06T07:34:06.955738+0000","last_unstale":"2024-03-06T07:34:07.975477+0000","last_undegraded":"2024-03-06T07:34:07.975477+0000","last_fullsized":"2024-03-06T07:34:07.975477+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:38.451040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092773+0000","last_change":"2024-03-06T07:34:05.956937+0000","last_active":"2024-03-06T07:34:08.092773+0000","last_peered":"2024-03-06T07:34:08.092773+0000","last_clean":"2024-03-06T07:34:08.092773+0000","last_became_active":"2024-03-06T07:34:05.953600+0000","last_became_peered":"2024-03-06T07:34:05.953600+0000","last_unstale":"2024-03-06T07:34:08.092773+0000","last_undegraded":"2024-03-06T07:34:08.092773+0000","last_fullsized":"2024-03-06T07:34:08.092773+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:59.305936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092185+0000","last_change":"2024-03-06T07:34:06.977011+0000","last_active":"2024-03-06T07:34:08.092185+0000","last_peered":"2024-03-06T07:34:08.092185+0000","last_clean":"2024-03-06T07:34:08.092185+0000","last_became_active":"2024-03-06T07:34:06.976725+0000","last_became_peered":"2024-03-06T07:34:06.976725+0000","last_unstale":"2024-03-06T07:34:08.092185+0000","last_undegraded":"2024-03-06T07:34:08.092185+0000","last_fullsized":"2024-03-06T07:34:08.092185+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:23.540572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975685+0000","last_change":"2024-03-06T07:34:05.964722+0000","last_active":"2024-03-06T07:34:07.975685+0000","last_peered":"2024-03-06T07:34:07.975685+0000","last_clean":"2024-03-06T07:34:07.975685+0000","last_became_active":"2024-03-06T07:34:05.964398+0000","last_became_peered":"2024-03-06T07:34:05.964398+0000","last_unstale":"2024-03-06T07:34:07.975685+0000","last_undegraded":"2024-03-06T07:34:07.975685+0000","last_fullsized":"2024-03-06T07:34:07.975685+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:58.227020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092947+0000","last_change":"2024-03-06T07:34:06.976076+0000","last_active":"2024-03-06T07:34:08.092947+0000","last_peered":"2024-03-06T07:34:08.092947+0000","last_clean":"2024-03-06T07:34:08.092947+0000","last_became_active":"2024-03-06T07:34:06.975407+0000","last_became_peered":"2024-03-06T07:34:06.975407+0000","last_unstale":"2024-03-06T07:34:08.092947+0000","last_undegraded":"2024-03-06T07:34:08.092947+0000","last_fullsized":"2024-03-06T07:34:08.092947+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:08.631235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092244+0000","last_change":"2024-03-06T07:34:05.962313+0000","last_active":"2024-03-06T07:34:08.092244+0000","last_peered":"2024-03-06T07:34:08.092244+0000","last_clean":"2024-03-06T07:34:08.092244+0000","last_became_active":"2024-03-06T07:34:05.961834+0000","last_became_peered":"2024-03-06T07:34:05.961834+0000","last_unstale":"2024-03-06T07:34:08.092244+0000","last_undegraded":"2024-03-06T07:34:08.092244+0000","last_fullsized":"2024-03-06T07:34:08.092244+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:22.633316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093381+0000","last_change":"2024-03-06T07:34:06.976347+0000","last_active":"2024-03-06T07:34:08.093381+0000","last_peered":"2024-03-06T07:34:08.093381+0000","last_clean":"2024-03-06T07:34:08.093381+0000","last_became_active":"2024-03-06T07:34:06.975689+0000","last_became_peered":"2024-03-06T07:34:06.975689+0000","last_unstale":"2024-03-06T07:34:08.093381+0000","last_undegraded":"2024-03-06T07:34:08.093381+0000","last_fullsized":"2024-03-06T07:34:08.093381+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:40.040686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975590+0000","last_change":"2024-03-06T07:34:05.968243+0000","last_active":"2024-03-06T07:34:07.975590+0000","last_peered":"2024-03-06T07:34:07.975590+0000","last_clean":"2024-03-06T07:34:07.975590+0000","last_became_active":"2024-03-06T07:34:05.967956+0000","last_became_peered":"2024-03-06T07:34:05.967956+0000","last_unstale":"2024-03-06T07:34:07.975590+0000","last_undegraded":"2024-03-06T07:34:07.975590+0000","last_fullsized":"2024-03-06T07:34:07.975590+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:47.346278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091486+0000","last_change":"2024-03-06T07:34:06.968788+0000","last_active":"2024-03-06T07:34:08.091486+0000","last_peered":"2024-03-06T07:34:08.091486+0000","last_clean":"2024-03-06T07:34:08.091486+0000","last_became_active":"2024-03-06T07:34:06.968489+0000","last_became_peered":"2024-03-06T07:34:06.968489+0000","last_unstale":"2024-03-06T07:34:08.091486+0000","last_undegraded":"2024-03-06T07:34:08.091486+0000","last_fullsized":"2024-03-06T07:34:08.091486+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:22.894385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092027+0000","last_change":"2024-03-06T07:34:05.960733+0000","last_active":"2024-03-06T07:34:08.092027+0000","last_peered":"2024-03-06T07:34:08.092027+0000","last_clean":"2024-03-06T07:34:08.092027+0000","last_became_active":"2024-03-06T07:34:05.960445+0000","last_became_peered":"2024-03-06T07:34:05.960445+0000","last_unstale":"2024-03-06T07:34:08.092027+0000","last_undegraded":"2024-03-06T07:34:08.092027+0000","last_fullsized":"2024-03-06T07:34:08.092027+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.671441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091267+0000","last_change":"2024-03-06T07:34:06.971295+0000","last_active":"2024-03-06T07:34:08.091267+0000","last_peered":"2024-03-06T07:34:08.091267+0000","last_clean":"2024-03-06T07:34:08.091267+0000","last_became_active":"2024-03-06T07:34:06.970946+0000","last_became_peered":"2024-03-06T07:34:06.970946+0000","last_unstale":"2024-03-06T07:34:08.091267+0000","last_undegraded":"2024-03-06T07:34:08.091267+0000","last_fullsized":"2024-03-06T07:34:08.091267+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:00.533488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975747+0000","last_change":"2024-03-06T07:34:05.965338+0000","last_active":"2024-03-06T07:34:07.975747+0000","last_peered":"2024-03-06T07:34:07.975747+0000","last_clean":"2024-03-06T07:34:07.975747+0000","last_became_active":"2024-03-06T07:34:05.965019+0000","last_became_peered":"2024-03-06T07:34:05.965019+0000","last_unstale":"2024-03-06T07:34:07.975747+0000","last_undegraded":"2024-03-06T07:34:07.975747+0000","last_fullsized":"2024-03-06T07:34:07.975747+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:34.724335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":330,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093463+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:34:08.093463+0000","last_peered":"2024-03-06T07:34:08.093463+0000","last_clean":"2024-03-06T07:34:08.093463+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:34:08.093463+0000","last_undegraded":"2024-03-06T07:34:08.093463+0000","last_fullsized":"2024-03-06T07:34:08.093463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1933312,"data_stored":1165087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574929,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1335296,"data_stored":574173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36652,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335188,"statfs":{"total":96636764160,"available":96599232512,"internally_reserved":0,"allocated":1339392,"data_stored":574707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36640,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335200,"statfs":{"total":96636764160,"available":96599244800,"internally_reserved":0,"allocated":1331200,"data_stored":574103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1347584,"data_stored":575785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1331200,"data_stored":574103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36652,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335188,"statfs":{"total":96636764160,"available":96599232512,"internally_reserved":0,"allocated":1339392,"data_stored":574227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574929,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":1166195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:11.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:11.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.709+0000 7fd3e4a00640 1 Processor -- start 2024-03-06T07:34:11.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.709+0000 7fd3e4a00640 1 -- start start 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3e4a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3e015e230 0x7fd3e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3e4a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00b4320 0x7fd3e00d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3e4a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e00cf340 con 0x7fd3e00ce550 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3e4a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e015e610 con 0x7fd3e015e230 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3e4a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00cfe60 con 0x7fd3e00b4320 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3dde00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3e015e230 0x7fd3e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3dde00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd3e015e230 0x7fd3e006edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42008/0 (socket says 172.21.15.114:42008) 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.710+0000 7fd3dde00640 1 -- 172.21.15.114:0/317560426 learned_addr learned my addr 172.21.15.114:0/317560426 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.711+0000 7fd3d5400640 1 --2- 172.21.15.114:0/317560426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00b4320 0x7fd3e00d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.711+0000 7fd3dde00640 1 -- 172.21.15.114:0/317560426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00b4320 msgr2=0x7fd3e00d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.711+0000 7fd3dde00640 1 --2- 172.21.15.114:0/317560426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00b4320 0x7fd3e00d0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.711+0000 7fd3dde00640 1 -- 172.21.15.114:0/317560426 >> v1:172.21.15.177:6789/0 conn(0x7fd3e00ce550 legacy=0x7fd3e00cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.711+0000 7fd3dde00640 1 -- 172.21.15.114:0/317560426 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e006f390 con 0x7fd3e015e230 2024-03-06T07:34:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.711+0000 7fd3d5400640 1 --2- 172.21.15.114:0/317560426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00b4320 0x7fd3e00d0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.712+0000 7fd3dde00640 1 --2- 172.21.15.114:0/317560426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e015e230 0x7fd3e006edb0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009e50 tx=0x7fd3d002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd20190ab83c142 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.712+0000 7fd3dd400640 1 -- 172.21.15.114:0/317560426 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d002c9f0 con 0x7fd3e015e230 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.712+0000 7fd3dd400640 1 -- 172.21.15.114:0/317560426 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d002cb90 con 0x7fd3e015e230 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.712+0000 7fd3dd400640 1 -- 172.21.15.114:0/317560426 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00343f0 con 0x7fd3e015e230 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 -- 172.21.15.114:0/317560426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e015e230 msgr2=0x7fd3e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/317560426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e015e230 0x7fd3e006edb0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009e50 tx=0x7fd3d002be50 comp rx=0 tx=0).stop 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 -- 172.21.15.114:0/317560426 shutdown_connections 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/317560426 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00b4320 0x7fd3e00d0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/317560426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e015e230 0x7fd3e006edb0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 -- 172.21.15.114:0/317560426 >> 172.21.15.114:0/317560426 conn(0x7fd3e00b3800 msgr2=0x7fd3e00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 -- 172.21.15.114:0/317560426 shutdown_connections 2024-03-06T07:34:11.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.713+0000 7fd3e4a00640 1 -- 172.21.15.114:0/317560426 wait complete. 2024-03-06T07:34:11.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.715+0000 7fd3e4a00640 1 Processor -- start 2024-03-06T07:34:11.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.715+0000 7fd3e4a00640 1 -- start start 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3e4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e00b4320 0x7fd3e0158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3e4a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00ce550 0x7fd3e0156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3e4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3e015e230 0x7fd3e00c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3e4a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00d06c0 con 0x7fd3e00b4320 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3e4a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00d03e0 con 0x7fd3e015e230 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3e4a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00d1120 con 0x7fd3e00ce550 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e00b4320 0x7fd3e0158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e00b4320 0x7fd3e0158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42018/0 (socket says 172.21.15.114:42018) 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 -- 172.21.15.114:0/1167757000 learned_addr learned my addr 172.21.15.114:0/1167757000 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:11.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 -- 172.21.15.114:0/1167757000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00ce550 msgr2=0x7fd3e0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00ce550 0x7fd3e0156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 -- 172.21.15.114:0/1167757000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3e015e230 msgr2=0x7fd3e00c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3e015e230 0x7fd3e00c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.716+0000 7fd3dde00640 1 -- 172.21.15.114:0/1167757000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0159210 con 0x7fd3e00b4320 2024-03-06T07:34:11.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.717+0000 7fd3dde00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e00b4320 0x7fd3e0158ba0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009e50 tx=0x7fd3d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.717+0000 7fd3d6a00640 1 -- 172.21.15.114:0/1167757000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0043020 con 0x7fd3e00b4320 2024-03-06T07:34:11.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.717+0000 7fd3d6a00640 1 -- 172.21.15.114:0/1167757000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d003e450 con 0x7fd3e00b4320 2024-03-06T07:34:11.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.717+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e01594f0 con 0x7fd3e00b4320 2024-03-06T07:34:11.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.717+0000 7fd3d6a00640 1 -- 172.21.15.114:0/1167757000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d003e840 con 0x7fd3e00b4320 2024-03-06T07:34:11.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.717+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e0159ab0 con 0x7fd3e00b4320 2024-03-06T07:34:11.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.719+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e004f330 con 0x7fd3e00b4320 2024-03-06T07:34:11.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.720+0000 7fd3d6a00640 1 -- 172.21.15.114:0/1167757000 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3d0046020 con 0x7fd3e00b4320 2024-03-06T07:34:11.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.721+0000 7fd3d6a00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.721+0000 7fd3d6a00640 1 -- 172.21.15.114:0/1167757000 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3d00acb90 con 0x7fd3e00b4320 2024-03-06T07:34:11.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.721+0000 7fd3d5400640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.722+0000 7fd3d5400640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8006fd0 tx=0x7fd3c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.725+0000 7fd3d6a00640 1 -- 172.21.15.114:0/1167757000 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3d004d030 con 0x7fd3e00b4320 2024-03-06T07:34:11.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T07:34:11.887+0000 7f5451000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:34:11.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.953+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3e0156980 con 0x7fd3bc06b4d0 2024-03-06T07:34:11.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.972+0000 7fd3d6a00640 1 -- 172.21.15.114:0/1167757000 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252480 (secure 0 0 0) 0x7fd3c800c7a0 con 0x7fd3bc06b4d0 2024-03-06T07:34:11.980 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:11.982 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3bc06b4d0 msgr2=0x7fd3bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8006fd0 tx=0x7fd3c8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e00b4320 msgr2=0x7fd3e0158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e00b4320 0x7fd3e0158ba0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009e50 tx=0x7fd3d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 shutdown_connections 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd3e015e230 0x7fd3e00c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd3e00ce550 0x7fd3e0156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd3bc06b4d0 0x7fd3bc06d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 --2- 172.21.15.114:0/1167757000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3e00b4320 0x7fd3e0158ba0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 >> 172.21.15.114:0/1167757000 conn(0x7fd3e00b3800 msgr2=0x7fd3e01570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 shutdown_connections 2024-03-06T07:34:11.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:11.978+0000 7fd3e4a00640 1 -- 172.21.15.114:0/1167757000 wait complete. 2024-03-06T07:34:12.002 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-03-06T07:34:11.212834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":294368,"kb_used_data":11624,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754680352,"statfs":{"total":773094113280,"available":772792680448,"internally_reserved":0,"allocated":11902976,"data_stored":5778380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":289394227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001133"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091068+0000","last_change":"2024-03-06T07:34:06.980348+0000","last_active":"2024-03-06T07:34:08.091068+0000","last_peered":"2024-03-06T07:34:08.091068+0000","last_clean":"2024-03-06T07:34:08.091068+0000","last_became_active":"2024-03-06T07:34:06.980057+0000","last_became_peered":"2024-03-06T07:34:06.980057+0000","last_unstale":"2024-03-06T07:34:08.091068+0000","last_undegraded":"2024-03-06T07:34:08.091068+0000","last_fullsized":"2024-03-06T07:34:08.091068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:59.404958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094610+0000","last_change":"2024-03-06T07:34:06.974427+0000","last_active":"2024-03-06T07:34:08.094610+0000","last_peered":"2024-03-06T07:34:08.094610+0000","last_clean":"2024-03-06T07:34:08.094610+0000","last_became_active":"2024-03-06T07:34:06.974095+0000","last_became_peered":"2024-03-06T07:34:06.974095+0000","last_unstale":"2024-03-06T07:34:08.094610+0000","last_undegraded":"2024-03-06T07:34:08.094610+0000","last_fullsized":"2024-03-06T07:34:08.094610+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:42.505055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093373+0000","last_change":"2024-03-06T07:34:06.980283+0000","last_active":"2024-03-06T07:34:08.093373+0000","last_peered":"2024-03-06T07:34:08.093373+0000","last_clean":"2024-03-06T07:34:08.093373+0000","last_became_active":"2024-03-06T07:34:06.979977+0000","last_became_peered":"2024-03-06T07:34:06.979977+0000","last_unstale":"2024-03-06T07:34:08.093373+0000","last_undegraded":"2024-03-06T07:34:08.093373+0000","last_fullsized":"2024-03-06T07:34:08.093373+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:00.835703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090946+0000","last_change":"2024-03-06T07:34:06.978020+0000","last_active":"2024-03-06T07:34:08.090946+0000","last_peered":"2024-03-06T07:34:08.090946+0000","last_clean":"2024-03-06T07:34:08.090946+0000","last_became_active":"2024-03-06T07:34:06.977678+0000","last_became_peered":"2024-03-06T07:34:06.977678+0000","last_unstale":"2024-03-06T07:34:08.090946+0000","last_undegraded":"2024-03-06T07:34:08.090946+0000","last_fullsized":"2024-03-06T07:34:08.090946+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:32.808970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975906+0000","last_change":"2024-03-06T07:34:06.964429+0000","last_active":"2024-03-06T07:34:07.975906+0000","last_peered":"2024-03-06T07:34:07.975906+0000","last_clean":"2024-03-06T07:34:07.975906+0000","last_became_active":"2024-03-06T07:34:06.963068+0000","last_became_peered":"2024-03-06T07:34:06.963068+0000","last_unstale":"2024-03-06T07:34:07.975906+0000","last_undegraded":"2024-03-06T07:34:07.975906+0000","last_fullsized":"2024-03-06T07:34:07.975906+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:34.784393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091110+0000","last_change":"2024-03-06T07:34:06.962728+0000","last_active":"2024-03-06T07:34:08.091110+0000","last_peered":"2024-03-06T07:34:08.091110+0000","last_clean":"2024-03-06T07:34:08.091110+0000","last_became_active":"2024-03-06T07:34:06.962447+0000","last_became_peered":"2024-03-06T07:34:06.962447+0000","last_unstale":"2024-03-06T07:34:08.091110+0000","last_undegraded":"2024-03-06T07:34:08.091110+0000","last_fullsized":"2024-03-06T07:34:08.091110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:27.958428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975608+0000","last_change":"2024-03-06T07:34:06.971628+0000","last_active":"2024-03-06T07:34:07.975608+0000","last_peered":"2024-03-06T07:34:07.975608+0000","last_clean":"2024-03-06T07:34:07.975608+0000","last_became_active":"2024-03-06T07:34:06.968306+0000","last_became_peered":"2024-03-06T07:34:06.968306+0000","last_unstale":"2024-03-06T07:34:07.975608+0000","last_undegraded":"2024-03-06T07:34:07.975608+0000","last_fullsized":"2024-03-06T07:34:07.975608+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:21.910276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092966+0000","last_change":"2024-03-06T07:34:06.970440+0000","last_active":"2024-03-06T07:34:08.092966+0000","last_peered":"2024-03-06T07:34:08.092966+0000","last_clean":"2024-03-06T07:34:08.092966+0000","last_became_active":"2024-03-06T07:34:06.969253+0000","last_became_peered":"2024-03-06T07:34:06.969253+0000","last_unstale":"2024-03-06T07:34:08.092966+0000","last_undegraded":"2024-03-06T07:34:08.092966+0000","last_fullsized":"2024-03-06T07:34:08.092966+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:21.032622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091240+0000","last_change":"2024-03-06T07:34:06.962825+0000","last_active":"2024-03-06T07:34:08.091240+0000","last_peered":"2024-03-06T07:34:08.091240+0000","last_clean":"2024-03-06T07:34:08.091240+0000","last_became_active":"2024-03-06T07:34:06.962520+0000","last_became_peered":"2024-03-06T07:34:06.962520+0000","last_unstale":"2024-03-06T07:34:08.091240+0000","last_undegraded":"2024-03-06T07:34:08.091240+0000","last_fullsized":"2024-03-06T07:34:08.091240+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:05.791347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090931+0000","last_change":"2024-03-06T07:34:06.962598+0000","last_active":"2024-03-06T07:34:08.090931+0000","last_peered":"2024-03-06T07:34:08.090931+0000","last_clean":"2024-03-06T07:34:08.090931+0000","last_became_active":"2024-03-06T07:34:06.962291+0000","last_became_peered":"2024-03-06T07:34:06.962291+0000","last_unstale":"2024-03-06T07:34:08.090931+0000","last_undegraded":"2024-03-06T07:34:08.090931+0000","last_fullsized":"2024-03-06T07:34:08.090931+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:51.588177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094332+0000","last_change":"2024-03-06T07:34:06.969957+0000","last_active":"2024-03-06T07:34:08.094332+0000","last_peered":"2024-03-06T07:34:08.094332+0000","last_clean":"2024-03-06T07:34:08.094332+0000","last_became_active":"2024-03-06T07:34:06.969600+0000","last_became_peered":"2024-03-06T07:34:06.969600+0000","last_unstale":"2024-03-06T07:34:08.094332+0000","last_undegraded":"2024-03-06T07:34:08.094332+0000","last_fullsized":"2024-03-06T07:34:08.094332+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:38.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091783+0000","last_change":"2024-03-06T07:34:06.961204+0000","last_active":"2024-03-06T07:34:08.091783+0000","last_peered":"2024-03-06T07:34:08.091783+0000","last_clean":"2024-03-06T07:34:08.091783+0000","last_became_active":"2024-03-06T07:34:06.960944+0000","last_became_peered":"2024-03-06T07:34:06.960944+0000","last_unstale":"2024-03-06T07:34:08.091783+0000","last_undegraded":"2024-03-06T07:34:08.091783+0000","last_fullsized":"2024-03-06T07:34:08.091783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:12.818634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094501+0000","last_change":"2024-03-06T07:34:06.975385+0000","last_active":"2024-03-06T07:34:08.094501+0000","last_peered":"2024-03-06T07:34:08.094501+0000","last_clean":"2024-03-06T07:34:08.094501+0000","last_became_active":"2024-03-06T07:34:06.973722+0000","last_became_peered":"2024-03-06T07:34:06.973722+0000","last_unstale":"2024-03-06T07:34:08.094501+0000","last_undegraded":"2024-03-06T07:34:08.094501+0000","last_fullsized":"2024-03-06T07:34:08.094501+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:03.784636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092836+0000","last_change":"2024-03-06T07:34:06.971116+0000","last_active":"2024-03-06T07:34:08.092836+0000","last_peered":"2024-03-06T07:34:08.092836+0000","last_clean":"2024-03-06T07:34:08.092836+0000","last_became_active":"2024-03-06T07:34:06.970819+0000","last_became_peered":"2024-03-06T07:34:06.970819+0000","last_unstale":"2024-03-06T07:34:08.092836+0000","last_undegraded":"2024-03-06T07:34:08.092836+0000","last_fullsized":"2024-03-06T07:34:08.092836+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:42.747736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090905+0000","last_change":"2024-03-06T07:34:06.966120+0000","last_active":"2024-03-06T07:34:08.090905+0000","last_peered":"2024-03-06T07:34:08.090905+0000","last_clean":"2024-03-06T07:34:08.090905+0000","last_became_active":"2024-03-06T07:34:06.964755+0000","last_became_peered":"2024-03-06T07:34:06.964755+0000","last_unstale":"2024-03-06T07:34:08.090905+0000","last_undegraded":"2024-03-06T07:34:08.090905+0000","last_fullsized":"2024-03-06T07:34:08.090905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:35.739353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090659+0000","last_change":"2024-03-06T07:34:06.971995+0000","last_active":"2024-03-06T07:34:08.090659+0000","last_peered":"2024-03-06T07:34:08.090659+0000","last_clean":"2024-03-06T07:34:08.090659+0000","last_became_active":"2024-03-06T07:34:06.971630+0000","last_became_peered":"2024-03-06T07:34:06.971630+0000","last_unstale":"2024-03-06T07:34:08.090659+0000","last_undegraded":"2024-03-06T07:34:08.090659+0000","last_fullsized":"2024-03-06T07:34:08.090659+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:06.796524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094666+0000","last_change":"2024-03-06T07:34:06.978057+0000","last_active":"2024-03-06T07:34:08.094666+0000","last_peered":"2024-03-06T07:34:08.094666+0000","last_clean":"2024-03-06T07:34:08.094666+0000","last_became_active":"2024-03-06T07:34:06.977735+0000","last_became_peered":"2024-03-06T07:34:06.977735+0000","last_unstale":"2024-03-06T07:34:08.094666+0000","last_undegraded":"2024-03-06T07:34:08.094666+0000","last_fullsized":"2024-03-06T07:34:08.094666+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:02.686459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974591+0000","last_change":"2024-03-06T07:34:06.975554+0000","last_active":"2024-03-06T07:34:07.974591+0000","last_peered":"2024-03-06T07:34:07.974591+0000","last_clean":"2024-03-06T07:34:07.974591+0000","last_became_active":"2024-03-06T07:34:06.975134+0000","last_became_peered":"2024-03-06T07:34:06.975134+0000","last_unstale":"2024-03-06T07:34:07.974591+0000","last_undegraded":"2024-03-06T07:34:07.974591+0000","last_fullsized":"2024-03-06T07:34:07.974591+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:16.863083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093372+0000","last_change":"2024-03-06T07:34:06.982022+0000","last_active":"2024-03-06T07:34:08.093372+0000","last_peered":"2024-03-06T07:34:08.093372+0000","last_clean":"2024-03-06T07:34:08.093372+0000","last_became_active":"2024-03-06T07:34:06.981688+0000","last_became_peered":"2024-03-06T07:34:06.981688+0000","last_unstale":"2024-03-06T07:34:08.093372+0000","last_undegraded":"2024-03-06T07:34:08.093372+0000","last_fullsized":"2024-03-06T07:34:08.093372+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:07.771565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094217+0000","last_change":"2024-03-06T07:34:06.981255+0000","last_active":"2024-03-06T07:34:08.094217+0000","last_peered":"2024-03-06T07:34:08.094217+0000","last_clean":"2024-03-06T07:34:08.094217+0000","last_became_active":"2024-03-06T07:34:06.980870+0000","last_became_peered":"2024-03-06T07:34:06.980870+0000","last_unstale":"2024-03-06T07:34:08.094217+0000","last_undegraded":"2024-03-06T07:34:08.094217+0000","last_fullsized":"2024-03-06T07:34:08.094217+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:38.451040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093486+0000","last_change":"2024-03-06T07:34:06.975981+0000","last_active":"2024-03-06T07:34:08.093486+0000","last_peered":"2024-03-06T07:34:08.093486+0000","last_clean":"2024-03-06T07:34:08.093486+0000","last_became_active":"2024-03-06T07:34:06.970044+0000","last_became_peered":"2024-03-06T07:34:06.970044+0000","last_unstale":"2024-03-06T07:34:08.093486+0000","last_undegraded":"2024-03-06T07:34:08.093486+0000","last_fullsized":"2024-03-06T07:34:08.093486+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:16.863083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091821+0000","last_change":"2024-03-06T07:34:06.968727+0000","last_active":"2024-03-06T07:34:08.091821+0000","last_peered":"2024-03-06T07:34:08.091821+0000","last_clean":"2024-03-06T07:34:08.091821+0000","last_became_active":"2024-03-06T07:34:06.968452+0000","last_became_peered":"2024-03-06T07:34:06.968452+0000","last_unstale":"2024-03-06T07:34:08.091821+0000","last_undegraded":"2024-03-06T07:34:08.091821+0000","last_fullsized":"2024-03-06T07:34:08.091821+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:00.835703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092186+0000","last_change":"2024-03-06T07:34:06.975499+0000","last_active":"2024-03-06T07:34:08.092186+0000","last_peered":"2024-03-06T07:34:08.092186+0000","last_clean":"2024-03-06T07:34:08.092186+0000","last_became_active":"2024-03-06T07:34:06.975035+0000","last_became_peered":"2024-03-06T07:34:06.975035+0000","last_unstale":"2024-03-06T07:34:08.092186+0000","last_undegraded":"2024-03-06T07:34:08.092186+0000","last_fullsized":"2024-03-06T07:34:08.092186+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:00.320954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092292+0000","last_change":"2024-03-06T07:34:06.974754+0000","last_active":"2024-03-06T07:34:08.092292+0000","last_peered":"2024-03-06T07:34:08.092292+0000","last_clean":"2024-03-06T07:34:08.092292+0000","last_became_active":"2024-03-06T07:34:06.974499+0000","last_became_peered":"2024-03-06T07:34:06.974499+0000","last_unstale":"2024-03-06T07:34:08.092292+0000","last_undegraded":"2024-03-06T07:34:08.092292+0000","last_fullsized":"2024-03-06T07:34:08.092292+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:25.500332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.095013+0000","last_change":"2024-03-06T07:34:06.976030+0000","last_active":"2024-03-06T07:34:08.095013+0000","last_peered":"2024-03-06T07:34:08.095013+0000","last_clean":"2024-03-06T07:34:08.095013+0000","last_became_active":"2024-03-06T07:34:06.975614+0000","last_became_peered":"2024-03-06T07:34:06.975614+0000","last_unstale":"2024-03-06T07:34:08.095013+0000","last_undegraded":"2024-03-06T07:34:08.095013+0000","last_fullsized":"2024-03-06T07:34:08.095013+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:23.022586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091103+0000","last_change":"2024-03-06T07:34:06.965494+0000","last_active":"2024-03-06T07:34:08.091103+0000","last_peered":"2024-03-06T07:34:08.091103+0000","last_clean":"2024-03-06T07:34:08.091103+0000","last_became_active":"2024-03-06T07:34:06.965211+0000","last_became_peered":"2024-03-06T07:34:06.965211+0000","last_unstale":"2024-03-06T07:34:08.091103+0000","last_undegraded":"2024-03-06T07:34:08.091103+0000","last_fullsized":"2024-03-06T07:34:08.091103+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:24.119650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092307+0000","last_change":"2024-03-06T07:34:06.956224+0000","last_active":"2024-03-06T07:34:08.092307+0000","last_peered":"2024-03-06T07:34:08.092307+0000","last_clean":"2024-03-06T07:34:08.092307+0000","last_became_active":"2024-03-06T07:34:06.955939+0000","last_became_peered":"2024-03-06T07:34:06.955939+0000","last_unstale":"2024-03-06T07:34:08.092307+0000","last_undegraded":"2024-03-06T07:34:08.092307+0000","last_fullsized":"2024-03-06T07:34:08.092307+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:30.154986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092645+0000","last_change":"2024-03-06T07:34:06.982346+0000","last_active":"2024-03-06T07:34:08.092645+0000","last_peered":"2024-03-06T07:34:08.092645+0000","last_clean":"2024-03-06T07:34:08.092645+0000","last_became_active":"2024-03-06T07:34:06.981902+0000","last_became_peered":"2024-03-06T07:34:06.981902+0000","last_unstale":"2024-03-06T07:34:08.092645+0000","last_undegraded":"2024-03-06T07:34:08.092645+0000","last_fullsized":"2024-03-06T07:34:08.092645+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:03.784636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093909+0000","last_change":"2024-03-06T07:34:06.982356+0000","last_active":"2024-03-06T07:34:08.093909+0000","last_peered":"2024-03-06T07:34:08.093909+0000","last_clean":"2024-03-06T07:34:08.093909+0000","last_became_active":"2024-03-06T07:34:06.981943+0000","last_became_peered":"2024-03-06T07:34:06.981943+0000","last_unstale":"2024-03-06T07:34:08.093909+0000","last_undegraded":"2024-03-06T07:34:08.093909+0000","last_fullsized":"2024-03-06T07:34:08.093909+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:58.527981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.977222+0000","last_change":"2024-03-06T07:34:06.963610+0000","last_active":"2024-03-06T07:34:07.977222+0000","last_peered":"2024-03-06T07:34:07.977222+0000","last_clean":"2024-03-06T07:34:07.977222+0000","last_became_active":"2024-03-06T07:34:06.963345+0000","last_became_peered":"2024-03-06T07:34:06.963345+0000","last_unstale":"2024-03-06T07:34:07.977222+0000","last_undegraded":"2024-03-06T07:34:07.977222+0000","last_fullsized":"2024-03-06T07:34:07.977222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:47.221769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091341+0000","last_change":"2024-03-06T07:34:06.965421+0000","last_active":"2024-03-06T07:34:08.091341+0000","last_peered":"2024-03-06T07:34:08.091341+0000","last_clean":"2024-03-06T07:34:08.091341+0000","last_became_active":"2024-03-06T07:34:06.965152+0000","last_became_peered":"2024-03-06T07:34:06.965152+0000","last_unstale":"2024-03-06T07:34:08.091341+0000","last_undegraded":"2024-03-06T07:34:08.091341+0000","last_fullsized":"2024-03-06T07:34:08.091341+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:35.990446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090506+0000","last_change":"2024-03-06T07:34:06.962315+0000","last_active":"2024-03-06T07:34:08.090506+0000","last_peered":"2024-03-06T07:34:08.090506+0000","last_clean":"2024-03-06T07:34:08.090506+0000","last_became_active":"2024-03-06T07:34:06.961636+0000","last_became_peered":"2024-03-06T07:34:06.961636+0000","last_unstale":"2024-03-06T07:34:08.090506+0000","last_undegraded":"2024-03-06T07:34:08.090506+0000","last_fullsized":"2024-03-06T07:34:08.090506+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:52.024864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092452+0000","last_change":"2024-03-06T07:34:06.966572+0000","last_active":"2024-03-06T07:34:08.092452+0000","last_peered":"2024-03-06T07:34:08.092452+0000","last_clean":"2024-03-06T07:34:08.092452+0000","last_became_active":"2024-03-06T07:34:06.964082+0000","last_became_peered":"2024-03-06T07:34:06.964082+0000","last_unstale":"2024-03-06T07:34:08.092452+0000","last_undegraded":"2024-03-06T07:34:08.092452+0000","last_fullsized":"2024-03-06T07:34:08.092452+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:50.040210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976296+0000","last_change":"2024-03-06T07:34:05.969455+0000","last_active":"2024-03-06T07:34:07.976296+0000","last_peered":"2024-03-06T07:34:07.976296+0000","last_clean":"2024-03-06T07:34:07.976296+0000","last_became_active":"2024-03-06T07:34:05.969045+0000","last_became_peered":"2024-03-06T07:34:05.969045+0000","last_unstale":"2024-03-06T07:34:07.976296+0000","last_undegraded":"2024-03-06T07:34:07.976296+0000","last_fullsized":"2024-03-06T07:34:07.976296+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:51.009846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976755+0000","last_change":"2024-03-06T07:34:06.981148+0000","last_active":"2024-03-06T07:34:07.976755+0000","last_peered":"2024-03-06T07:34:07.976755+0000","last_clean":"2024-03-06T07:34:07.976755+0000","last_became_active":"2024-03-06T07:34:06.980715+0000","last_became_peered":"2024-03-06T07:34:06.980715+0000","last_unstale":"2024-03-06T07:34:07.976755+0000","last_undegraded":"2024-03-06T07:34:07.976755+0000","last_fullsized":"2024-03-06T07:34:07.976755+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:38.901944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094540+0000","last_change":"2024-03-06T07:34:05.968273+0000","last_active":"2024-03-06T07:34:08.094540+0000","last_peered":"2024-03-06T07:34:08.094540+0000","last_clean":"2024-03-06T07:34:08.094540+0000","last_became_active":"2024-03-06T07:34:05.967984+0000","last_became_peered":"2024-03-06T07:34:05.967984+0000","last_unstale":"2024-03-06T07:34:08.094540+0000","last_undegraded":"2024-03-06T07:34:08.094540+0000","last_fullsized":"2024-03-06T07:34:08.094540+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:43.892727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091859+0000","last_change":"2024-03-06T07:34:06.965490+0000","last_active":"2024-03-06T07:34:08.091859+0000","last_peered":"2024-03-06T07:34:08.091859+0000","last_clean":"2024-03-06T07:34:08.091859+0000","last_became_active":"2024-03-06T07:34:06.965229+0000","last_became_peered":"2024-03-06T07:34:06.965229+0000","last_unstale":"2024-03-06T07:34:08.091859+0000","last_undegraded":"2024-03-06T07:34:08.091859+0000","last_fullsized":"2024-03-06T07:34:08.091859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:49.896621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975792+0000","last_change":"2024-03-06T07:34:05.967179+0000","last_active":"2024-03-06T07:34:07.975792+0000","last_peered":"2024-03-06T07:34:07.975792+0000","last_clean":"2024-03-06T07:34:07.975792+0000","last_became_active":"2024-03-06T07:34:05.966765+0000","last_became_peered":"2024-03-06T07:34:05.966765+0000","last_unstale":"2024-03-06T07:34:07.975792+0000","last_undegraded":"2024-03-06T07:34:07.975792+0000","last_fullsized":"2024-03-06T07:34:07.975792+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:05.832514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093838+0000","last_change":"2024-03-06T07:34:06.976191+0000","last_active":"2024-03-06T07:34:08.093838+0000","last_peered":"2024-03-06T07:34:08.093838+0000","last_clean":"2024-03-06T07:34:08.093838+0000","last_became_active":"2024-03-06T07:34:06.975820+0000","last_became_peered":"2024-03-06T07:34:06.975820+0000","last_unstale":"2024-03-06T07:34:08.093838+0000","last_undegraded":"2024-03-06T07:34:08.093838+0000","last_fullsized":"2024-03-06T07:34:08.093838+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:07.961805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975845+0000","last_change":"2024-03-06T07:34:05.969221+0000","last_active":"2024-03-06T07:34:07.975845+0000","last_peered":"2024-03-06T07:34:07.975845+0000","last_clean":"2024-03-06T07:34:07.975845+0000","last_became_active":"2024-03-06T07:34:05.968143+0000","last_became_peered":"2024-03-06T07:34:05.968143+0000","last_unstale":"2024-03-06T07:34:07.975845+0000","last_undegraded":"2024-03-06T07:34:07.975845+0000","last_fullsized":"2024-03-06T07:34:07.975845+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:11.728397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090632+0000","last_change":"2024-03-06T07:34:06.963406+0000","last_active":"2024-03-06T07:34:08.090632+0000","last_peered":"2024-03-06T07:34:08.090632+0000","last_clean":"2024-03-06T07:34:08.090632+0000","last_became_active":"2024-03-06T07:34:06.963156+0000","last_became_peered":"2024-03-06T07:34:06.963156+0000","last_unstale":"2024-03-06T07:34:08.090632+0000","last_undegraded":"2024-03-06T07:34:08.090632+0000","last_fullsized":"2024-03-06T07:34:08.090632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:21.865435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975298+0000","last_change":"2024-03-06T07:34:05.965076+0000","last_active":"2024-03-06T07:34:07.975298+0000","last_peered":"2024-03-06T07:34:07.975298+0000","last_clean":"2024-03-06T07:34:07.975298+0000","last_became_active":"2024-03-06T07:34:05.964531+0000","last_became_peered":"2024-03-06T07:34:05.964531+0000","last_unstale":"2024-03-06T07:34:07.975298+0000","last_undegraded":"2024-03-06T07:34:07.975298+0000","last_fullsized":"2024-03-06T07:34:07.975298+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:13.214810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092518+0000","last_change":"2024-03-06T07:34:06.976245+0000","last_active":"2024-03-06T07:34:08.092518+0000","last_peered":"2024-03-06T07:34:08.092518+0000","last_clean":"2024-03-06T07:34:08.092518+0000","last_became_active":"2024-03-06T07:34:06.975765+0000","last_became_peered":"2024-03-06T07:34:06.975765+0000","last_unstale":"2024-03-06T07:34:08.092518+0000","last_undegraded":"2024-03-06T07:34:08.092518+0000","last_fullsized":"2024-03-06T07:34:08.092518+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:12.422685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092101+0000","last_change":"2024-03-06T07:34:05.964609+0000","last_active":"2024-03-06T07:34:08.092101+0000","last_peered":"2024-03-06T07:34:08.092101+0000","last_clean":"2024-03-06T07:34:08.092101+0000","last_became_active":"2024-03-06T07:34:05.959755+0000","last_became_peered":"2024-03-06T07:34:05.959755+0000","last_unstale":"2024-03-06T07:34:08.092101+0000","last_undegraded":"2024-03-06T07:34:08.092101+0000","last_fullsized":"2024-03-06T07:34:08.092101+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:22.525554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090264+0000","last_change":"2024-03-06T07:34:06.960818+0000","last_active":"2024-03-06T07:34:08.090264+0000","last_peered":"2024-03-06T07:34:08.090264+0000","last_clean":"2024-03-06T07:34:08.090264+0000","last_became_active":"2024-03-06T07:34:06.960482+0000","last_became_peered":"2024-03-06T07:34:06.960482+0000","last_unstale":"2024-03-06T07:34:08.090264+0000","last_undegraded":"2024-03-06T07:34:08.090264+0000","last_fullsized":"2024-03-06T07:34:08.090264+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:00.293386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093848+0000","last_change":"2024-03-06T07:34:05.961648+0000","last_active":"2024-03-06T07:34:08.093848+0000","last_peered":"2024-03-06T07:34:08.093848+0000","last_clean":"2024-03-06T07:34:08.093848+0000","last_became_active":"2024-03-06T07:34:05.956450+0000","last_became_peered":"2024-03-06T07:34:05.956450+0000","last_unstale":"2024-03-06T07:34:08.093848+0000","last_undegraded":"2024-03-06T07:34:08.093848+0000","last_fullsized":"2024-03-06T07:34:08.093848+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:46.206751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975455+0000","last_change":"2024-03-06T07:34:06.968926+0000","last_active":"2024-03-06T07:34:07.975455+0000","last_peered":"2024-03-06T07:34:07.975455+0000","last_clean":"2024-03-06T07:34:07.975455+0000","last_became_active":"2024-03-06T07:34:06.967912+0000","last_became_peered":"2024-03-06T07:34:06.967912+0000","last_unstale":"2024-03-06T07:34:07.975455+0000","last_undegraded":"2024-03-06T07:34:07.975455+0000","last_fullsized":"2024-03-06T07:34:07.975455+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:28:43.430470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090776+0000","last_change":"2024-03-06T07:34:05.965174+0000","last_active":"2024-03-06T07:34:08.090776+0000","last_peered":"2024-03-06T07:34:08.090776+0000","last_clean":"2024-03-06T07:34:08.090776+0000","last_became_active":"2024-03-06T07:34:05.964635+0000","last_became_peered":"2024-03-06T07:34:05.964635+0000","last_unstale":"2024-03-06T07:34:08.090776+0000","last_undegraded":"2024-03-06T07:34:08.090776+0000","last_fullsized":"2024-03-06T07:34:08.090776+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:41.490037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976497+0000","last_change":"2024-03-06T07:34:06.963087+0000","last_active":"2024-03-06T07:34:07.976497+0000","last_peered":"2024-03-06T07:34:07.976497+0000","last_clean":"2024-03-06T07:34:07.976497+0000","last_became_active":"2024-03-06T07:34:06.962318+0000","last_became_peered":"2024-03-06T07:34:06.962318+0000","last_unstale":"2024-03-06T07:34:07.976497+0000","last_undegraded":"2024-03-06T07:34:07.976497+0000","last_fullsized":"2024-03-06T07:34:07.976497+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:06.221782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975843+0000","last_change":"2024-03-06T07:34:05.967458+0000","last_active":"2024-03-06T07:34:07.975843+0000","last_peered":"2024-03-06T07:34:07.975843+0000","last_clean":"2024-03-06T07:34:07.975843+0000","last_became_active":"2024-03-06T07:34:05.966747+0000","last_became_peered":"2024-03-06T07:34:05.966747+0000","last_unstale":"2024-03-06T07:34:07.975843+0000","last_undegraded":"2024-03-06T07:34:07.975843+0000","last_fullsized":"2024-03-06T07:34:07.975843+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:24.198154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976897+0000","last_change":"2024-03-06T07:34:06.970098+0000","last_active":"2024-03-06T07:34:07.976897+0000","last_peered":"2024-03-06T07:34:07.976897+0000","last_clean":"2024-03-06T07:34:07.976897+0000","last_became_active":"2024-03-06T07:34:06.965679+0000","last_became_peered":"2024-03-06T07:34:06.965679+0000","last_unstale":"2024-03-06T07:34:07.976897+0000","last_undegraded":"2024-03-06T07:34:07.976897+0000","last_fullsized":"2024-03-06T07:34:07.976897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:42.505055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092821+0000","last_change":"2024-03-06T07:34:05.959497+0000","last_active":"2024-03-06T07:34:08.092821+0000","last_peered":"2024-03-06T07:34:08.092821+0000","last_clean":"2024-03-06T07:34:08.092821+0000","last_became_active":"2024-03-06T07:34:05.958782+0000","last_became_peered":"2024-03-06T07:34:05.958782+0000","last_unstale":"2024-03-06T07:34:08.092821+0000","last_undegraded":"2024-03-06T07:34:08.092821+0000","last_fullsized":"2024-03-06T07:34:08.092821+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:49.191406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092646+0000","last_change":"2024-03-06T07:34:06.980693+0000","last_active":"2024-03-06T07:34:08.092646+0000","last_peered":"2024-03-06T07:34:08.092646+0000","last_clean":"2024-03-06T07:34:08.092646+0000","last_became_active":"2024-03-06T07:34:06.979142+0000","last_became_peered":"2024-03-06T07:34:06.979142+0000","last_unstale":"2024-03-06T07:34:08.092646+0000","last_undegraded":"2024-03-06T07:34:08.092646+0000","last_fullsized":"2024-03-06T07:34:08.092646+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.851707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091784+0000","last_change":"2024-03-06T07:34:05.957938+0000","last_active":"2024-03-06T07:34:08.091784+0000","last_peered":"2024-03-06T07:34:08.091784+0000","last_clean":"2024-03-06T07:34:08.091784+0000","last_became_active":"2024-03-06T07:34:05.957639+0000","last_became_peered":"2024-03-06T07:34:05.957639+0000","last_unstale":"2024-03-06T07:34:08.091784+0000","last_undegraded":"2024-03-06T07:34:08.091784+0000","last_fullsized":"2024-03-06T07:34:08.091784+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:59.518470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976186+0000","last_change":"2024-03-06T07:34:06.974394+0000","last_active":"2024-03-06T07:34:07.976186+0000","last_peered":"2024-03-06T07:34:07.976186+0000","last_clean":"2024-03-06T07:34:07.976186+0000","last_became_active":"2024-03-06T07:34:06.973996+0000","last_became_peered":"2024-03-06T07:34:06.973996+0000","last_unstale":"2024-03-06T07:34:07.976186+0000","last_undegraded":"2024-03-06T07:34:07.976186+0000","last_fullsized":"2024-03-06T07:34:07.976186+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:52.658511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092109+0000","last_change":"2024-03-06T07:34:05.953201+0000","last_active":"2024-03-06T07:34:08.092109+0000","last_peered":"2024-03-06T07:34:08.092109+0000","last_clean":"2024-03-06T07:34:08.092109+0000","last_became_active":"2024-03-06T07:34:05.952872+0000","last_became_peered":"2024-03-06T07:34:05.952872+0000","last_unstale":"2024-03-06T07:34:08.092109+0000","last_undegraded":"2024-03-06T07:34:08.092109+0000","last_fullsized":"2024-03-06T07:34:08.092109+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:20.017604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091981+0000","last_change":"2024-03-06T07:34:06.965398+0000","last_active":"2024-03-06T07:34:08.091981+0000","last_peered":"2024-03-06T07:34:08.091981+0000","last_clean":"2024-03-06T07:34:08.091981+0000","last_became_active":"2024-03-06T07:34:06.965098+0000","last_became_peered":"2024-03-06T07:34:06.965098+0000","last_unstale":"2024-03-06T07:34:08.091981+0000","last_undegraded":"2024-03-06T07:34:08.091981+0000","last_fullsized":"2024-03-06T07:34:08.091981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:40.040686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091396+0000","last_change":"2024-03-06T07:34:05.968178+0000","last_active":"2024-03-06T07:34:08.091396+0000","last_peered":"2024-03-06T07:34:08.091396+0000","last_clean":"2024-03-06T07:34:08.091396+0000","last_became_active":"2024-03-06T07:34:05.967804+0000","last_became_peered":"2024-03-06T07:34:05.967804+0000","last_unstale":"2024-03-06T07:34:08.091396+0000","last_undegraded":"2024-03-06T07:34:08.091396+0000","last_fullsized":"2024-03-06T07:34:08.091396+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:33.769374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092467+0000","last_change":"2024-03-06T07:34:06.983788+0000","last_active":"2024-03-06T07:34:08.092467+0000","last_peered":"2024-03-06T07:34:08.092467+0000","last_clean":"2024-03-06T07:34:08.092467+0000","last_became_active":"2024-03-06T07:34:06.983526+0000","last_became_peered":"2024-03-06T07:34:06.983526+0000","last_unstale":"2024-03-06T07:34:08.092467+0000","last_undegraded":"2024-03-06T07:34:08.092467+0000","last_fullsized":"2024-03-06T07:34:08.092467+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:07.961805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974738+0000","last_change":"2024-03-06T07:34:05.967071+0000","last_active":"2024-03-06T07:34:07.974738+0000","last_peered":"2024-03-06T07:34:07.974738+0000","last_clean":"2024-03-06T07:34:07.974738+0000","last_became_active":"2024-03-06T07:34:05.966768+0000","last_became_peered":"2024-03-06T07:34:05.966768+0000","last_unstale":"2024-03-06T07:34:07.974738+0000","last_undegraded":"2024-03-06T07:34:07.974738+0000","last_fullsized":"2024-03-06T07:34:07.974738+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:07.414354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091509+0000","last_change":"2024-03-06T07:34:06.981124+0000","last_active":"2024-03-06T07:34:08.091509+0000","last_peered":"2024-03-06T07:34:08.091509+0000","last_clean":"2024-03-06T07:34:08.091509+0000","last_became_active":"2024-03-06T07:34:06.980817+0000","last_became_peered":"2024-03-06T07:34:06.980817+0000","last_unstale":"2024-03-06T07:34:08.091509+0000","last_undegraded":"2024-03-06T07:34:08.091509+0000","last_fullsized":"2024-03-06T07:34:08.091509+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:58.738464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975580+0000","last_change":"2024-03-06T07:34:05.968981+0000","last_active":"2024-03-06T07:34:07.975580+0000","last_peered":"2024-03-06T07:34:07.975580+0000","last_clean":"2024-03-06T07:34:07.975580+0000","last_became_active":"2024-03-06T07:34:05.968157+0000","last_became_peered":"2024-03-06T07:34:05.968157+0000","last_unstale":"2024-03-06T07:34:07.975580+0000","last_undegraded":"2024-03-06T07:34:07.975580+0000","last_fullsized":"2024-03-06T07:34:07.975580+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:35.894352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976128+0000","last_change":"2024-03-06T07:34:06.965019+0000","last_active":"2024-03-06T07:34:07.976128+0000","last_peered":"2024-03-06T07:34:07.976128+0000","last_clean":"2024-03-06T07:34:07.976128+0000","last_became_active":"2024-03-06T07:34:06.964694+0000","last_became_peered":"2024-03-06T07:34:06.964694+0000","last_unstale":"2024-03-06T07:34:07.976128+0000","last_undegraded":"2024-03-06T07:34:07.976128+0000","last_fullsized":"2024-03-06T07:34:07.976128+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:59.916231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091536+0000","last_change":"2024-03-06T07:34:05.964919+0000","last_active":"2024-03-06T07:34:08.091536+0000","last_peered":"2024-03-06T07:34:08.091536+0000","last_clean":"2024-03-06T07:34:08.091536+0000","last_became_active":"2024-03-06T07:34:05.964586+0000","last_became_peered":"2024-03-06T07:34:05.964586+0000","last_unstale":"2024-03-06T07:34:08.091536+0000","last_undegraded":"2024-03-06T07:34:08.091536+0000","last_fullsized":"2024-03-06T07:34:08.091536+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:57.723446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092757+0000","last_change":"2024-03-06T07:34:06.964726+0000","last_active":"2024-03-06T07:34:08.092757+0000","last_peered":"2024-03-06T07:34:08.092757+0000","last_clean":"2024-03-06T07:34:08.092757+0000","last_became_active":"2024-03-06T07:34:06.964448+0000","last_became_peered":"2024-03-06T07:34:06.964448+0000","last_unstale":"2024-03-06T07:34:08.092757+0000","last_undegraded":"2024-03-06T07:34:08.092757+0000","last_fullsized":"2024-03-06T07:34:08.092757+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:06.162248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.976091+0000","last_change":"2024-03-06T07:34:05.960026+0000","last_active":"2024-03-06T07:34:07.976091+0000","last_peered":"2024-03-06T07:34:07.976091+0000","last_clean":"2024-03-06T07:34:07.976091+0000","last_became_active":"2024-03-06T07:34:05.959759+0000","last_became_peered":"2024-03-06T07:34:05.959759+0000","last_unstale":"2024-03-06T07:34:07.976091+0000","last_undegraded":"2024-03-06T07:34:07.976091+0000","last_fullsized":"2024-03-06T07:34:07.976091+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:40.961829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.094751+0000","last_change":"2024-03-06T07:34:06.985008+0000","last_active":"2024-03-06T07:34:08.094751+0000","last_peered":"2024-03-06T07:34:08.094751+0000","last_clean":"2024-03-06T07:34:08.094751+0000","last_became_active":"2024-03-06T07:34:06.984463+0000","last_became_peered":"2024-03-06T07:34:06.984463+0000","last_unstale":"2024-03-06T07:34:08.094751+0000","last_undegraded":"2024-03-06T07:34:08.094751+0000","last_fullsized":"2024-03-06T07:34:08.094751+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:39.851707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975573+0000","last_change":"2024-03-06T07:34:05.963002+0000","last_active":"2024-03-06T07:34:07.975573+0000","last_peered":"2024-03-06T07:34:07.975573+0000","last_clean":"2024-03-06T07:34:07.975573+0000","last_became_active":"2024-03-06T07:34:05.959894+0000","last_became_peered":"2024-03-06T07:34:05.959894+0000","last_unstale":"2024-03-06T07:34:07.975573+0000","last_undegraded":"2024-03-06T07:34:07.975573+0000","last_fullsized":"2024-03-06T07:34:07.975573+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:03.567830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093840+0000","last_change":"2024-03-06T07:34:06.983682+0000","last_active":"2024-03-06T07:34:08.093840+0000","last_peered":"2024-03-06T07:34:08.093840+0000","last_clean":"2024-03-06T07:34:08.093840+0000","last_became_active":"2024-03-06T07:34:06.981655+0000","last_became_peered":"2024-03-06T07:34:06.981655+0000","last_unstale":"2024-03-06T07:34:08.093840+0000","last_undegraded":"2024-03-06T07:34:08.093840+0000","last_fullsized":"2024-03-06T07:34:08.093840+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:25.500332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975522+0000","last_change":"2024-03-06T07:34:05.967095+0000","last_active":"2024-03-06T07:34:07.975522+0000","last_peered":"2024-03-06T07:34:07.975522+0000","last_clean":"2024-03-06T07:34:07.975522+0000","last_became_active":"2024-03-06T07:34:05.966654+0000","last_became_peered":"2024-03-06T07:34:05.966654+0000","last_unstale":"2024-03-06T07:34:07.975522+0000","last_undegraded":"2024-03-06T07:34:07.975522+0000","last_fullsized":"2024-03-06T07:34:07.975522+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:26.943410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093173+0000","last_change":"2024-03-06T07:34:06.980744+0000","last_active":"2024-03-06T07:34:08.093173+0000","last_peered":"2024-03-06T07:34:08.093173+0000","last_clean":"2024-03-06T07:34:08.093173+0000","last_became_active":"2024-03-06T07:34:06.980025+0000","last_became_peered":"2024-03-06T07:34:06.980025+0000","last_unstale":"2024-03-06T07:34:08.093173+0000","last_undegraded":"2024-03-06T07:34:08.093173+0000","last_fullsized":"2024-03-06T07:34:08.093173+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:58.738464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091506+0000","last_change":"2024-03-06T07:34:05.963203+0000","last_active":"2024-03-06T07:34:08.091506+0000","last_peered":"2024-03-06T07:34:08.091506+0000","last_clean":"2024-03-06T07:34:08.091506+0000","last_became_active":"2024-03-06T07:34:05.962888+0000","last_became_peered":"2024-03-06T07:34:05.962888+0000","last_unstale":"2024-03-06T07:34:08.091506+0000","last_undegraded":"2024-03-06T07:34:08.091506+0000","last_fullsized":"2024-03-06T07:34:08.091506+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:58.901213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090552+0000","last_change":"2024-03-06T07:34:06.976759+0000","last_active":"2024-03-06T07:34:08.090552+0000","last_peered":"2024-03-06T07:34:08.090552+0000","last_clean":"2024-03-06T07:34:08.090552+0000","last_became_active":"2024-03-06T07:34:06.976024+0000","last_became_peered":"2024-03-06T07:34:06.976024+0000","last_unstale":"2024-03-06T07:34:08.090552+0000","last_undegraded":"2024-03-06T07:34:08.090552+0000","last_fullsized":"2024-03-06T07:34:08.090552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:48.876510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975690+0000","last_change":"2024-03-06T07:34:05.968901+0000","last_active":"2024-03-06T07:34:07.975690+0000","last_peered":"2024-03-06T07:34:07.975690+0000","last_clean":"2024-03-06T07:34:07.975690+0000","last_became_active":"2024-03-06T07:34:05.968227+0000","last_became_peered":"2024-03-06T07:34:05.968227+0000","last_unstale":"2024-03-06T07:34:07.975690+0000","last_undegraded":"2024-03-06T07:34:07.975690+0000","last_fullsized":"2024-03-06T07:34:07.975690+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:01.527118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093054+0000","last_change":"2024-03-06T07:34:06.984129+0000","last_active":"2024-03-06T07:34:08.093054+0000","last_peered":"2024-03-06T07:34:08.093054+0000","last_clean":"2024-03-06T07:34:08.093054+0000","last_became_active":"2024-03-06T07:34:06.983778+0000","last_became_peered":"2024-03-06T07:34:06.983778+0000","last_unstale":"2024-03-06T07:34:08.093054+0000","last_undegraded":"2024-03-06T07:34:08.093054+0000","last_fullsized":"2024-03-06T07:34:08.093054+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:00.533488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091002+0000","last_change":"2024-03-06T07:34:05.961806+0000","last_active":"2024-03-06T07:34:08.091002+0000","last_peered":"2024-03-06T07:34:08.091002+0000","last_clean":"2024-03-06T07:34:08.091002+0000","last_became_active":"2024-03-06T07:34:05.961553+0000","last_became_peered":"2024-03-06T07:34:05.961553+0000","last_unstale":"2024-03-06T07:34:08.091002+0000","last_undegraded":"2024-03-06T07:34:08.091002+0000","last_fullsized":"2024-03-06T07:34:08.091002+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:37.032238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091684+0000","last_change":"2024-03-06T07:34:06.969935+0000","last_active":"2024-03-06T07:34:08.091684+0000","last_peered":"2024-03-06T07:34:08.091684+0000","last_clean":"2024-03-06T07:34:08.091684+0000","last_became_active":"2024-03-06T07:34:06.969402+0000","last_became_peered":"2024-03-06T07:34:06.969402+0000","last_unstale":"2024-03-06T07:34:08.091684+0000","last_undegraded":"2024-03-06T07:34:08.091684+0000","last_fullsized":"2024-03-06T07:34:08.091684+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:07.771565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.090899+0000","last_change":"2024-03-06T07:34:05.961371+0000","last_active":"2024-03-06T07:34:08.090899+0000","last_peered":"2024-03-06T07:34:08.090899+0000","last_clean":"2024-03-06T07:34:08.090899+0000","last_became_active":"2024-03-06T07:34:05.961031+0000","last_became_peered":"2024-03-06T07:34:05.961031+0000","last_unstale":"2024-03-06T07:34:08.090899+0000","last_undegraded":"2024-03-06T07:34:08.090899+0000","last_fullsized":"2024-03-06T07:34:08.090899+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:22.007568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974808+0000","last_change":"2024-03-06T07:34:06.967988+0000","last_active":"2024-03-06T07:34:07.974808+0000","last_peered":"2024-03-06T07:34:07.974808+0000","last_clean":"2024-03-06T07:34:07.974808+0000","last_became_active":"2024-03-06T07:34:06.967454+0000","last_became_peered":"2024-03-06T07:34:06.967454+0000","last_unstale":"2024-03-06T07:34:07.974808+0000","last_undegraded":"2024-03-06T07:34:07.974808+0000","last_fullsized":"2024-03-06T07:34:07.974808+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:58.527981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092942+0000","last_change":"2024-03-06T07:34:05.954142+0000","last_active":"2024-03-06T07:34:08.092942+0000","last_peered":"2024-03-06T07:34:08.092942+0000","last_clean":"2024-03-06T07:34:08.092942+0000","last_became_active":"2024-03-06T07:34:05.953849+0000","last_became_peered":"2024-03-06T07:34:05.953849+0000","last_unstale":"2024-03-06T07:34:08.092942+0000","last_undegraded":"2024-03-06T07:34:08.092942+0000","last_fullsized":"2024-03-06T07:34:08.092942+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:11.407667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091412+0000","last_change":"2024-03-06T07:34:06.979414+0000","last_active":"2024-03-06T07:34:08.091412+0000","last_peered":"2024-03-06T07:34:08.091412+0000","last_clean":"2024-03-06T07:34:08.091412+0000","last_became_active":"2024-03-06T07:34:06.979138+0000","last_became_peered":"2024-03-06T07:34:06.979138+0000","last_unstale":"2024-03-06T07:34:08.091412+0000","last_undegraded":"2024-03-06T07:34:08.091412+0000","last_fullsized":"2024-03-06T07:34:08.091412+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:20.886582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.974896+0000","last_change":"2024-03-06T07:34:05.964059+0000","last_active":"2024-03-06T07:34:07.974896+0000","last_peered":"2024-03-06T07:34:07.974896+0000","last_clean":"2024-03-06T07:34:07.974896+0000","last_became_active":"2024-03-06T07:34:05.963731+0000","last_became_peered":"2024-03-06T07:34:05.963731+0000","last_unstale":"2024-03-06T07:34:07.974896+0000","last_undegraded":"2024-03-06T07:34:07.974896+0000","last_fullsized":"2024-03-06T07:34:07.974896+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:43.892727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091978+0000","last_change":"2024-03-06T07:34:06.963832+0000","last_active":"2024-03-06T07:34:08.091978+0000","last_peered":"2024-03-06T07:34:08.091978+0000","last_clean":"2024-03-06T07:34:08.091978+0000","last_became_active":"2024-03-06T07:34:06.963540+0000","last_became_peered":"2024-03-06T07:34:06.963540+0000","last_unstale":"2024-03-06T07:34:08.091978+0000","last_undegraded":"2024-03-06T07:34:08.091978+0000","last_fullsized":"2024-03-06T07:34:08.091978+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:35.999636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091874+0000","last_change":"2024-03-06T07:34:05.960922+0000","last_active":"2024-03-06T07:34:08.091874+0000","last_peered":"2024-03-06T07:34:08.091874+0000","last_clean":"2024-03-06T07:34:08.091874+0000","last_became_active":"2024-03-06T07:34:05.960593+0000","last_became_peered":"2024-03-06T07:34:05.960593+0000","last_unstale":"2024-03-06T07:34:08.091874+0000","last_undegraded":"2024-03-06T07:34:08.091874+0000","last_fullsized":"2024-03-06T07:34:08.091874+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:41.732718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975477+0000","last_change":"2024-03-06T07:34:06.956435+0000","last_active":"2024-03-06T07:34:07.975477+0000","last_peered":"2024-03-06T07:34:07.975477+0000","last_clean":"2024-03-06T07:34:07.975477+0000","last_became_active":"2024-03-06T07:34:06.955738+0000","last_became_peered":"2024-03-06T07:34:06.955738+0000","last_unstale":"2024-03-06T07:34:07.975477+0000","last_undegraded":"2024-03-06T07:34:07.975477+0000","last_fullsized":"2024-03-06T07:34:07.975477+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:38.451040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092773+0000","last_change":"2024-03-06T07:34:05.956937+0000","last_active":"2024-03-06T07:34:08.092773+0000","last_peered":"2024-03-06T07:34:08.092773+0000","last_clean":"2024-03-06T07:34:08.092773+0000","last_became_active":"2024-03-06T07:34:05.953600+0000","last_became_peered":"2024-03-06T07:34:05.953600+0000","last_unstale":"2024-03-06T07:34:08.092773+0000","last_undegraded":"2024-03-06T07:34:08.092773+0000","last_fullsized":"2024-03-06T07:34:08.092773+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:59.305936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092185+0000","last_change":"2024-03-06T07:34:06.977011+0000","last_active":"2024-03-06T07:34:08.092185+0000","last_peered":"2024-03-06T07:34:08.092185+0000","last_clean":"2024-03-06T07:34:08.092185+0000","last_became_active":"2024-03-06T07:34:06.976725+0000","last_became_peered":"2024-03-06T07:34:06.976725+0000","last_unstale":"2024-03-06T07:34:08.092185+0000","last_undegraded":"2024-03-06T07:34:08.092185+0000","last_fullsized":"2024-03-06T07:34:08.092185+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:23.540572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975685+0000","last_change":"2024-03-06T07:34:05.964722+0000","last_active":"2024-03-06T07:34:07.975685+0000","last_peered":"2024-03-06T07:34:07.975685+0000","last_clean":"2024-03-06T07:34:07.975685+0000","last_became_active":"2024-03-06T07:34:05.964398+0000","last_became_peered":"2024-03-06T07:34:05.964398+0000","last_unstale":"2024-03-06T07:34:07.975685+0000","last_undegraded":"2024-03-06T07:34:07.975685+0000","last_fullsized":"2024-03-06T07:34:07.975685+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:58.227020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092947+0000","last_change":"2024-03-06T07:34:06.976076+0000","last_active":"2024-03-06T07:34:08.092947+0000","last_peered":"2024-03-06T07:34:08.092947+0000","last_clean":"2024-03-06T07:34:08.092947+0000","last_became_active":"2024-03-06T07:34:06.975407+0000","last_became_peered":"2024-03-06T07:34:06.975407+0000","last_unstale":"2024-03-06T07:34:08.092947+0000","last_undegraded":"2024-03-06T07:34:08.092947+0000","last_fullsized":"2024-03-06T07:34:08.092947+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:08.631235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092244+0000","last_change":"2024-03-06T07:34:05.962313+0000","last_active":"2024-03-06T07:34:08.092244+0000","last_peered":"2024-03-06T07:34:08.092244+0000","last_clean":"2024-03-06T07:34:08.092244+0000","last_became_active":"2024-03-06T07:34:05.961834+0000","last_became_peered":"2024-03-06T07:34:05.961834+0000","last_unstale":"2024-03-06T07:34:08.092244+0000","last_undegraded":"2024-03-06T07:34:08.092244+0000","last_fullsized":"2024-03-06T07:34:08.092244+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:22.633316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093381+0000","last_change":"2024-03-06T07:34:06.976347+0000","last_active":"2024-03-06T07:34:08.093381+0000","last_peered":"2024-03-06T07:34:08.093381+0000","last_clean":"2024-03-06T07:34:08.093381+0000","last_became_active":"2024-03-06T07:34:06.975689+0000","last_became_peered":"2024-03-06T07:34:06.975689+0000","last_unstale":"2024-03-06T07:34:08.093381+0000","last_undegraded":"2024-03-06T07:34:08.093381+0000","last_fullsized":"2024-03-06T07:34:08.093381+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:40.040686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975590+0000","last_change":"2024-03-06T07:34:05.968243+0000","last_active":"2024-03-06T07:34:07.975590+0000","last_peered":"2024-03-06T07:34:07.975590+0000","last_clean":"2024-03-06T07:34:07.975590+0000","last_became_active":"2024-03-06T07:34:05.967956+0000","last_became_peered":"2024-03-06T07:34:05.967956+0000","last_unstale":"2024-03-06T07:34:07.975590+0000","last_undegraded":"2024-03-06T07:34:07.975590+0000","last_fullsized":"2024-03-06T07:34:07.975590+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:51:47.346278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091486+0000","last_change":"2024-03-06T07:34:06.968788+0000","last_active":"2024-03-06T07:34:08.091486+0000","last_peered":"2024-03-06T07:34:08.091486+0000","last_clean":"2024-03-06T07:34:08.091486+0000","last_became_active":"2024-03-06T07:34:06.968489+0000","last_became_peered":"2024-03-06T07:34:06.968489+0000","last_unstale":"2024-03-06T07:34:08.091486+0000","last_undegraded":"2024-03-06T07:34:08.091486+0000","last_fullsized":"2024-03-06T07:34:08.091486+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:22.894385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.092027+0000","last_change":"2024-03-06T07:34:05.960733+0000","last_active":"2024-03-06T07:34:08.092027+0000","last_peered":"2024-03-06T07:34:08.092027+0000","last_clean":"2024-03-06T07:34:08.092027+0000","last_became_active":"2024-03-06T07:34:05.960445+0000","last_became_peered":"2024-03-06T07:34:05.960445+0000","last_unstale":"2024-03-06T07:34:08.092027+0000","last_undegraded":"2024-03-06T07:34:08.092027+0000","last_fullsized":"2024-03-06T07:34:08.092027+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:01.671441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.091267+0000","last_change":"2024-03-06T07:34:06.971295+0000","last_active":"2024-03-06T07:34:08.091267+0000","last_peered":"2024-03-06T07:34:08.091267+0000","last_clean":"2024-03-06T07:34:08.091267+0000","last_became_active":"2024-03-06T07:34:06.970946+0000","last_became_peered":"2024-03-06T07:34:06.970946+0000","last_unstale":"2024-03-06T07:34:08.091267+0000","last_undegraded":"2024-03-06T07:34:08.091267+0000","last_fullsized":"2024-03-06T07:34:08.091267+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:05.919547+0000","last_clean_scrub_stamp":"2024-03-06T07:34:05.919547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:00.533488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:07.975747+0000","last_change":"2024-03-06T07:34:05.965338+0000","last_active":"2024-03-06T07:34:07.975747+0000","last_peered":"2024-03-06T07:34:07.975747+0000","last_clean":"2024-03-06T07:34:07.975747+0000","last_became_active":"2024-03-06T07:34:05.965019+0000","last_became_peered":"2024-03-06T07:34:05.965019+0000","last_unstale":"2024-03-06T07:34:07.975747+0000","last_undegraded":"2024-03-06T07:34:07.975747+0000","last_fullsized":"2024-03-06T07:34:07.975747+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:34:04.904529+0000","last_clean_scrub_stamp":"2024-03-06T07:34:04.904529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:34.724335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":330,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:34:08.093463+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:34:08.093463+0000","last_peered":"2024-03-06T07:34:08.093463+0000","last_clean":"2024-03-06T07:34:08.093463+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:34:08.093463+0000","last_undegraded":"2024-03-06T07:34:08.093463+0000","last_fullsized":"2024-03-06T07:34:08.093463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574928,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1933312,"data_stored":1165087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574929,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1335296,"data_stored":574173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574927,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36652,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335188,"statfs":{"total":96636764160,"available":96599232512,"internally_reserved":0,"allocated":1339392,"data_stored":574707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36640,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335200,"statfs":{"total":96636764160,"available":96599244800,"internally_reserved":0,"allocated":1331200,"data_stored":574103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1347584,"data_stored":575785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574928,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1331200,"data_stored":574103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36652,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335188,"statfs":{"total":96636764160,"available":96599232512,"internally_reserved":0,"allocated":1339392,"data_stored":574227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574929,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1945600,"data_stored":1166195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:12.006 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:34:12.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:34:12.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.270+0000 7f1390a00640 1 Processor -- start 2024-03-06T07:34:12.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.271+0000 7f1390a00640 1 -- start start 2024-03-06T07:34:12.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.273+0000 7f1390a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f138c0b50f0 0x7f138c0cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.273+0000 7f1390a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c0b3340 con 0x7f138c0d2900 2024-03-06T07:34:12.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.273+0000 7f1390a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c0d1ed0 con 0x7f138c15e270 2024-03-06T07:34:12.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.273+0000 7f1390a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f138c0b3600 con 0x7f138c0b50f0 2024-03-06T07:34:12.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f138b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f138c0b50f0 0x7f138c0cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f1389e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f138c0d2900 0x7f138c0ce960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55368/0 (socket says 172.21.15.114:55368) 2024-03-06T07:34:12.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f138b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f138c0b50f0 0x7f138c0cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42032/0 (socket says 172.21.15.114:42032) 2024-03-06T07:34:12.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f1389e00640 1 -- 172.21.15.114:0/2747189293 learned_addr learned my addr 172.21.15.114:0/2747189293 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:12.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f138b200640 1 -- 172.21.15.114:0/2747189293 >> v1:172.21.15.177:6789/0 conn(0x7f138c15e270 legacy=0x7f138c0d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f138b200640 1 -- 172.21.15.114:0/2747189293 >> v1:172.21.15.114:6789/0 conn(0x7f138c0d2900 legacy=0x7f138c0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f1389400640 1 -- 172.21.15.114:0/2747189293 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894078751 0 0) 0x7f138c0b3340 con 0x7f138c0d2900 2024-03-06T07:34:12.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.274+0000 7f138b200640 1 -- 172.21.15.114:0/2747189293 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0b38e0 con 0x7f138c0b50f0 2024-03-06T07:34:12.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.275+0000 7f138b200640 1 --2- 172.21.15.114:0/2747189293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c0cf070 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1384009d90 tx=0x7f138402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ec83ae1fc0c16b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1389400640 1 -- 172.21.15.114:0/2747189293 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384039070 con 0x7f138c0b50f0 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1389400640 1 -- 172.21.15.114:0/2747189293 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138402cb40 con 0x7f138c0b50f0 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1389400640 1 -- 172.21.15.114:0/2747189293 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384034440 con 0x7f138c0b50f0 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1390a00640 1 -- 172.21.15.114:0/2747189293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 msgr2=0x7f138c0cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1390a00640 1 --2- 172.21.15.114:0/2747189293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c0cf070 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f1384009d90 tx=0x7f138402bd90 comp rx=0 tx=0).stop 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1390a00640 1 -- 172.21.15.114:0/2747189293 shutdown_connections 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1390a00640 1 --2- 172.21.15.114:0/2747189293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c0cf070 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1390a00640 1 -- 172.21.15.114:0/2747189293 >> 172.21.15.114:0/2747189293 conn(0x7f138c0d3dc0 msgr2=0x7f138c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.276+0000 7f1390a00640 1 -- 172.21.15.114:0/2747189293 shutdown_connections 2024-03-06T07:34:12.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.277+0000 7f1390a00640 1 -- 172.21.15.114:0/2747189293 wait complete. 2024-03-06T07:34:12.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.278+0000 7f1390a00640 1 Processor -- start 2024-03-06T07:34:12.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.278+0000 7f1390a00640 1 -- start start 2024-03-06T07:34:12.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.279+0000 7f1390a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c160da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.279+0000 7f1390a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f138c0d2900 0x7f138c1f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.279+0000 7f1390a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f138c15e270 0x7f138c1f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.279+0000 7f1390a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15e650 con 0x7f138c0b50f0 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.279+0000 7f1390a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0d1ba0 con 0x7f138c15e270 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.279+0000 7f1390a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0d48e0 con 0x7f138c0d2900 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.279+0000 7f138a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42044/0 (socket says 172.21.15.114:42044) 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 -- 172.21.15.114:0/1291055525 learned_addr learned my addr 172.21.15.114:0/1291055525 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138b200640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f138c15e270 0x7f138c1f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f1389e00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f138c0d2900 0x7f138c1f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 -- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f138c0d2900 msgr2=0x7f138c1f4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f138c0d2900 0x7f138c1f4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 -- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f138c15e270 msgr2=0x7f138c1f0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f138c15e270 0x7f138c1f0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 -- 172.21.15.114:0/1291055525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0cc080 con 0x7f138c0b50f0 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138b200640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f138c15e270 0x7f138c1f0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.280+0000 7f138a800640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c160da0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f137c00b3f0 tx=0x7f137c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.281+0000 7f1382a00640 1 -- 172.21.15.114:0/1291055525 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c015020 con 0x7f138c0b50f0 2024-03-06T07:34:12.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.281+0000 7f1382a00640 1 -- 172.21.15.114:0/1291055525 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c011830 con 0x7f138c0b50f0 2024-03-06T07:34:12.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.281+0000 7f1382a00640 1 -- 172.21.15.114:0/1291055525 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c011c20 con 0x7f138c0b50f0 2024-03-06T07:34:12.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.281+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c0cc360 con 0x7f138c0b50f0 2024-03-06T07:34:12.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.281+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c0cc920 con 0x7f138c0b50f0 2024-03-06T07:34:12.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.282+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c075450 con 0x7f138c0b50f0 2024-03-06T07:34:12.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.283+0000 7f1382a00640 1 -- 172.21.15.114:0/1291055525 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f137c018020 con 0x7f138c0b50f0 2024-03-06T07:34:12.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.284+0000 7f1382a00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f136c06b4d0 0x7f136c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.284+0000 7f1382a00640 1 -- 172.21.15.114:0/1291055525 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f137c089db0 con 0x7f138c0b50f0 2024-03-06T07:34:12.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.284+0000 7f1389e00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f136c06b4d0 0x7f136c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.285+0000 7f1389e00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f136c06b4d0 0x7f136c06d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1378007fd0 tx=0x7f1378009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.289+0000 7f1382a00640 1 -- 172.21.15.114:0/1291055525 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f137c01f030 con 0x7f138c0b50f0 2024-03-06T07:34:12.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.557+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f138c0b3340 con 0x7f138c0b50f0 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.561+0000 7f1382a00640 1 -- 172.21.15.114:0/1291055525 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f137c058cc0 con 0x7f138c0b50f0 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f136c06b4d0 msgr2=0x7f136c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f136c06b4d0 0x7f136c06d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1378007fd0 tx=0x7f1378009040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 msgr2=0x7f138c160da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c160da0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f137c00b3f0 tx=0x7f137c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 shutdown_connections 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f138c15e270 0x7f138c1f0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f138c0d2900 0x7f138c1f4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f136c06b4d0 0x7f136c06d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 --2- 172.21.15.114:0/1291055525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f138c0b50f0 0x7f138c160da0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 >> 172.21.15.114:0/1291055525 conn(0x7f138c0d3dc0 msgr2=0x7f138c0d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 shutdown_connections 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.564+0000 7f1390a00640 1 -- 172.21.15.114:0/1291055525 wait complete. 2024-03-06T07:34:12.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:12.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.774+0000 7f4a37600640 1 Processor -- start 2024-03-06T07:34:12.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.775+0000 7f4a37600640 1 -- start start 2024-03-06T07:34:12.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.776+0000 7f4a37600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4a3015f9d0 0x7f4a3015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.776+0000 7f4a37600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4a30070b10 0x7f4a30070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.776+0000 7f4a37600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3006cef0 con 0x7f4a30144c00 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.776+0000 7f4a37600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3006d1d0 con 0x7f4a3015f9d0 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.776+0000 7f4a37600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3006d4b0 con 0x7f4a30070b10 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.776+0000 7f4a27e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4a3015f9d0 0x7f4a3015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.777+0000 7f4a27e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4a3015f9d0 0x7f4a3015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42056/0 (socket says 172.21.15.114:42056) 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.777+0000 7f4a27e00640 1 -- 172.21.15.114:0/2726342127 learned_addr learned my addr 172.21.15.114:0/2726342127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.777+0000 7f4a35600640 1 --2- 172.21.15.114:0/2726342127 >> v2:172.21.15.177:3300/0 conn(0x7f4a30070b10 0x7f4a30070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.777+0000 7f4a27e00640 1 -- 172.21.15.114:0/2726342127 >> v2:172.21.15.177:3300/0 conn(0x7f4a30070b10 msgr2=0x7f4a30070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.777+0000 7f4a27e00640 1 --2- 172.21.15.114:0/2726342127 >> v2:172.21.15.177:3300/0 conn(0x7f4a30070b10 0x7f4a30070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.777+0000 7f4a27e00640 1 -- 172.21.15.114:0/2726342127 >> v1:172.21.15.177:6789/0 conn(0x7f4a30144c00 legacy=0x7f4a3006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.777+0000 7f4a27e00640 1 -- 172.21.15.114:0/2726342127 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3006d790 con 0x7f4a3015f9d0 2024-03-06T07:34:12.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.778+0000 7f4a27e00640 1 --2- 172.21.15.114:0/2726342127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3015f9d0 0x7f4a3015fe60 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f4a3006e500 tx=0x7f4a1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2987be027a4832 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.778+0000 7f4a2fe00640 1 -- 172.21.15.114:0/2726342127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18035020 con 0x7f4a3015f9d0 2024-03-06T07:34:12.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a2fe00640 1 -- 172.21.15.114:0/2726342127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a18031870 con 0x7f4a3015f9d0 2024-03-06T07:34:12.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a2fe00640 1 -- 172.21.15.114:0/2726342127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18031c40 con 0x7f4a3015f9d0 2024-03-06T07:34:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 -- 172.21.15.114:0/2726342127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3015f9d0 msgr2=0x7f4a3015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 --2- 172.21.15.114:0/2726342127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3015f9d0 0x7f4a3015fe60 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f4a3006e500 tx=0x7f4a1802b910 comp rx=0 tx=0).stop 2024-03-06T07:34:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 -- 172.21.15.114:0/2726342127 shutdown_connections 2024-03-06T07:34:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 --2- 172.21.15.114:0/2726342127 >> v2:172.21.15.177:3300/0 conn(0x7f4a30070b10 0x7f4a30070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 --2- 172.21.15.114:0/2726342127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3015f9d0 0x7f4a3015fe60 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 -- 172.21.15.114:0/2726342127 >> 172.21.15.114:0/2726342127 conn(0x7f4a300b1730 msgr2=0x7f4a300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 -- 172.21.15.114:0/2726342127 shutdown_connections 2024-03-06T07:34:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.779+0000 7f4a37600640 1 -- 172.21.15.114:0/2726342127 wait complete. 2024-03-06T07:34:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.781+0000 7f4a37600640 1 Processor -- start 2024-03-06T07:34:12.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.781+0000 7f4a37600640 1 -- start start 2024-03-06T07:34:12.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a37600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a30070b10 0x7f4a3013c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a37600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a30144c00 0x7f4a3013ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a37600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a3015f9d0 0x7f4a3015d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a37600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3006ccd0 con 0x7f4a30070b10 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a37600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3006cb50 con 0x7f4a30144c00 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a37600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3006df90 con 0x7f4a3015f9d0 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a30070b10 0x7f4a3013c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a34c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a30070b10 0x7f4a3013c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42060/0 (socket says 172.21.15.114:42060) 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.782+0000 7f4a34c00640 1 -- 172.21.15.114:0/1615112284 learned_addr learned my addr 172.21.15.114:0/1615112284 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a34c00640 1 -- 172.21.15.114:0/1615112284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a3015f9d0 msgr2=0x7f4a3015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a27e00640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a30144c00 0x7f4a3013ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a34c00640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a3015f9d0 0x7f4a3015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a34c00640 1 -- 172.21.15.114:0/1615112284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a30144c00 msgr2=0x7f4a3013ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a34c00640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a30144c00 0x7f4a3013ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a34c00640 1 -- 172.21.15.114:0/1615112284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a301400e0 con 0x7f4a30070b10 2024-03-06T07:34:12.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a34c00640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a30070b10 0x7f4a3013c2e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c460 tx=0x7f4a2000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a2d600640 1 -- 172.21.15.114:0/1615112284 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20016020 con 0x7f4a30070b10 2024-03-06T07:34:12.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a2d600640 1 -- 172.21.15.114:0/1615112284 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a20012870 con 0x7f4a30070b10 2024-03-06T07:34:12.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.783+0000 7f4a2d600640 1 -- 172.21.15.114:0/1615112284 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20012c60 con 0x7f4a30070b10 2024-03-06T07:34:12.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.784+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3013fb20 con 0x7f4a30070b10 2024-03-06T07:34:12.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.784+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a30140b60 con 0x7f4a30070b10 2024-03-06T07:34:12.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.785+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3004f330 con 0x7f4a30070b10 2024-03-06T07:34:12.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.786+0000 7f4a2d600640 1 -- 172.21.15.114:0/1615112284 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a20019020 con 0x7f4a30070b10 2024-03-06T07:34:12.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.787+0000 7f4a2d600640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a0c06b4d0 0x7f4a0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.787+0000 7f4a2d600640 1 -- 172.21.15.114:0/1615112284 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a20089c80 con 0x7f4a30070b10 2024-03-06T07:34:12.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.787+0000 7f4a27e00640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a0c06b4d0 0x7f4a0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.788+0000 7f4a27e00640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a0c06b4d0 0x7f4a0c06d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:12.792+0000 7f4a2d600640 1 -- 172.21.15.114:0/1615112284 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a20020070 con 0x7f4a30070b10 2024-03-06T07:34:13.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.023+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a3006e550 con 0x7f4a30070b10 2024-03-06T07:34:13.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.024+0000 7f4a2d600640 1 -- 172.21.15.114:0/1615112284 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f4a20058b00 con 0x7f4a30070b10 2024-03-06T07:34:13.026 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:13.026 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":79,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5983,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3305544049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3305544049},{"type":"v1","addr":"172.21.15.114:6836","nonce":3305544049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6001,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/3949834778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3949834778},{"type":"v1","addr":"172.21.15.177:6834","nonce":3949834778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6004,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2151340900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2151340900},{"type":"v1","addr":"172.21.15.177:6835","nonce":2151340900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:09.452793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7674},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7674":{"gid":7674,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/860883596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":860883596},{"type":"v1","addr":"172.21.15.114:6837","nonce":860883596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7674,"qdb_cluster":[7674]},"id":8}]} 2024-03-06T07:34:13.027 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 79 2024-03-06T07:34:13.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a0c06b4d0 msgr2=0x7f4a0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a0c06b4d0 0x7f4a0c06d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a18008030 comp rx=0 tx=0).stop 2024-03-06T07:34:13.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a30070b10 msgr2=0x7f4a3013c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a30070b10 0x7f4a3013c2e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c460 tx=0x7f4a2000c920 comp rx=0 tx=0).stop 2024-03-06T07:34:13.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 shutdown_connections 2024-03-06T07:34:13.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a3015f9d0 0x7f4a3015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a30144c00 0x7f4a3013ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a0c06b4d0 0x7f4a0c06d990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 --2- 172.21.15.114:0/1615112284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a30070b10 0x7f4a3013c2e0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.027+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 >> 172.21.15.114:0/1615112284 conn(0x7f4a300b1730 msgr2=0x7f4a30145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.028+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 shutdown_connections 2024-03-06T07:34:13.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.028+0000 7f4a37600640 1 -- 172.21.15.114:0/1615112284 wait complete. 2024-03-06T07:34:13.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:06.421022+0000', 'modified': '2024-03-06T07:34:09.452793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7674}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7674': {'gid': 7674, 'name': 'a', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/860883596', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 860883596}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 860883596}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7674, 'qdb_cluster': [7674]} 2024-03-06T07:34:13.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:34:13.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:34:13.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.656+0000 7f832aa00640 1 Processor -- start 2024-03-06T07:34:13.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.656+0000 7f832aa00640 1 -- start start 2024-03-06T07:34:13.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.658+0000 7f832aa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f832415cc70 0x7f8324162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.658+0000 7f832aa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8324166ec0 con 0x7f8324145b70 2024-03-06T07:34:13.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.658+0000 7f832aa00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832406cbe0 con 0x7f83240cd5b0 2024-03-06T07:34:13.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.658+0000 7f832aa00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832406cec0 con 0x7f832415cc70 2024-03-06T07:34:13.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.658+0000 7f8323400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8324145b70 0x7f832415c560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55372/0 (socket says 172.21.15.114:55372) 2024-03-06T07:34:13.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.658+0000 7f8323400640 1 -- 172.21.15.114:0/1089204892 learned_addr learned my addr 172.21.15.114:0/1089204892 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8328a00640 1 --2- 172.21.15.114:0/1089204892 >> v2:172.21.15.177:3300/0 conn(0x7f832415cc70 0x7f8324162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074383236 0 0) 0x7f8324166ec0 con 0x7f8324145b70 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8308003520 con 0x7f8324145b70 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639087626 0 0) 0x7f8308003520 con 0x7f8324145b70 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 >> v2:172.21.15.177:3300/0 conn(0x7f832415cc70 msgr2=0x7f8324162fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8322a00640 1 --2- 172.21.15.114:0/1089204892 >> v2:172.21.15.177:3300/0 conn(0x7f832415cc70 0x7f8324162fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 >> v1:172.21.15.177:6789/0 conn(0x7f83240cd5b0 legacy=0x7f83240cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.659+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832406d1a0 con 0x7f8324145b70 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8310001670 con 0x7f8324145b70 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 >> v1:172.21.15.114:6789/0 conn(0x7f8324145b70 legacy=0x7f832415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83080039b0 0x7f8308003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308004800 0x7f8308004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83080051c0 0x7f830800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832406cec0 con 0x7f83080051c0 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830800c060 con 0x7f8308004800 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830800c260 con 0x7f83080039b0 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8328a00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83080051c0 0x7f830800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8323e00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83080039b0 0x7f8308003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8323400640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308004800 0x7f8308004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8328a00640 1 -- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83080039b0 msgr2=0x7f8308003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8328a00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83080039b0 0x7f8308003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8328a00640 1 -- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308004800 msgr2=0x7f8308004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8328a00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308004800 0x7f8308004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8328a00640 1 -- 172.21.15.114:0/1089204892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8308003520 con 0x7f83080051c0 2024-03-06T07:34:13.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.660+0000 7f8323400640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308004800 0x7f8308004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.661+0000 7f8328a00640 1 -- 172.21.15.114:0/1089204892 reap_dead start 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.662+0000 7f8328a00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83080051c0 0x7f830800b9c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f831c009ac0 tx=0x7f8308005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2328e81d989ad864 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.662+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c009690 con 0x7f83080051c0 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.662+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c004020 con 0x7f83080051c0 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.662+0000 7f8322a00640 1 -- 172.21.15.114:0/1089204892 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c0043d0 con 0x7f83080051c0 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.663+0000 7f832aa00640 1 -- 172.21.15.114:0/1089204892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83080051c0 msgr2=0x7f830800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.663+0000 7f832aa00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83080051c0 0x7f830800b9c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f831c009ac0 tx=0x7f8308005830 comp rx=0 tx=0).stop 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.663+0000 7f832aa00640 1 -- 172.21.15.114:0/1089204892 shutdown_connections 2024-03-06T07:34:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.663+0000 7f832aa00640 1 --2- 172.21.15.114:0/1089204892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83080051c0 0x7f830800b9c0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.663+0000 7f832aa00640 1 -- 172.21.15.114:0/1089204892 >> 172.21.15.114:0/1089204892 conn(0x7f83240cc4f0 msgr2=0x7f83240cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.663+0000 7f832aa00640 1 -- 172.21.15.114:0/1089204892 shutdown_connections 2024-03-06T07:34:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.663+0000 7f832aa00640 1 -- 172.21.15.114:0/1089204892 wait complete. 2024-03-06T07:34:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.664+0000 7f832aa00640 1 Processor -- start 2024-03-06T07:34:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.665+0000 7f832aa00640 1 -- start start 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.665+0000 7f832aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83080051c0 0x7f83241591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.665+0000 7f832aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83240af530 0x7f83240b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.665+0000 7f832aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83240b1480 0x7f83240b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.665+0000 7f832aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240cdb10 con 0x7f83240b1480 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.665+0000 7f832aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240cd990 con 0x7f83080051c0 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.665+0000 7f832aa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83240cdc90 con 0x7f83240af530 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83240b1480 0x7f83240b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83240b1480 0x7f83240b1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42084/0 (socket says 172.21.15.114:42084) 2024-03-06T07:34:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 -- 172.21.15.114:0/143514444 learned_addr learned my addr 172.21.15.114:0/143514444 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8323e00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83080051c0 0x7f83241591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8323400640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83240af530 0x7f83240b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 -- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83240af530 msgr2=0x7f83240b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83240af530 0x7f83240b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 -- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83080051c0 msgr2=0x7f83241591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83080051c0 0x7f83241591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8328a00640 1 -- 172.21.15.114:0/143514444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240b1ee0 con 0x7f83240b1480 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.666+0000 7f8323e00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83080051c0 0x7f83241591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:13.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.667+0000 7f8328a00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83240b1480 0x7f83240b1900 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f831c004650 tx=0x7f831c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.667+0000 7f831be00640 1 -- 172.21.15.114:0/143514444 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c009690 con 0x7f83240b1480 2024-03-06T07:34:13.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.667+0000 7f831be00640 1 -- 172.21.15.114:0/143514444 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c003610 con 0x7f83240b1480 2024-03-06T07:34:13.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.667+0000 7f831be00640 1 -- 172.21.15.114:0/143514444 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c003a00 con 0x7f83240b1480 2024-03-06T07:34:13.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.667+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83240b21c0 con 0x7f83240b1480 2024-03-06T07:34:13.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.667+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8324162d60 con 0x7f83240b1480 2024-03-06T07:34:13.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.670+0000 7f831be00640 1 -- 172.21.15.114:0/143514444 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f831c003ba0 con 0x7f83240b1480 2024-03-06T07:34:13.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.671+0000 7f831be00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f830406b4d0 0x7f830406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.671+0000 7f831be00640 1 -- 172.21.15.114:0/143514444 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f831c0ab0b0 con 0x7f83240b1480 2024-03-06T07:34:13.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.671+0000 7f8323e00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f830406b4d0 0x7f830406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.671+0000 7f832d816740 1 Processor -- start 2024-03-06T07:34:13.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f832d816740 1 -- start start 2024-03-06T07:34:13.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f8323e00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f830406b4d0 0x7f830406d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8314000f80 tx=0x7f8314009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f832d816740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9d3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da03540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3d9d3c90 0x564b3da03a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f832d816740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3d93d600 con 0x564b3da007b0 2024-03-06T07:34:13.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3d8e6930 con 0x564b3da00b90 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.672+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x564b3d9396a0 con 0x564b3d9d3c90 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9d3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42098/0 (socket says 172.21.15.114:42098) 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 -- 172.21.15.114:0/1318182115 learned_addr learned my addr 172.21.15.114:0/1318182115 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323400640 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da03540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8328a00640 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3d9d3c90 0x564b3da03a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 -- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3d9d3c90 msgr2=0x564b3da03a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3d9d3c90 0x564b3da03a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 -- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 msgr2=0x564b3da03540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da03540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 -- 172.21.15.114:0/1318182115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b3d6522d0 con 0x564b3da007b0 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.673+0000 7f8323e00640 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9d3750 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f831400fed0 tx=0x7f8314012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df00e24fea3733e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f830fe00640 1 -- 172.21.15.114:0/1318182115 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314026070 con 0x564b3da007b0 2024-03-06T07:34:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f830fe00640 1 -- 172.21.15.114:0/1318182115 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8314005650 con 0x564b3da007b0 2024-03-06T07:34:13.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f830fe00640 1 -- 172.21.15.114:0/1318182115 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83140135f0 con 0x564b3da007b0 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f832d816740 1 -- 172.21.15.114:0/1318182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 msgr2=0x564b3d9d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f832d816740 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9d3750 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f831400fed0 tx=0x7f8314012000 comp rx=0 tx=0).stop 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f832d816740 1 -- 172.21.15.114:0/1318182115 shutdown_connections 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f832d816740 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3d9d3c90 0x564b3da03a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f832d816740 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da03540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f832d816740 1 --2- 172.21.15.114:0/1318182115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9d3750 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.674+0000 7f832d816740 1 -- 172.21.15.114:0/1318182115 >> 172.21.15.114:0/1318182115 conn(0x564b3d9d59c0 msgr2=0x564b3da1e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.675+0000 7f832d816740 1 -- 172.21.15.114:0/1318182115 shutdown_connections 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.675+0000 7f832d816740 1 -- 172.21.15.114:0/1318182115 wait complete. 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.675+0000 7f832d816740 1 -- start start 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.675+0000 7f832d816740 1 Processor -- start 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.676+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.676+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3d9f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.676+0000 7f832d816740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da2e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.676+0000 7f832d816740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3d9d7f60 con 0x564b3da18b70 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.676+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3d8f9d90 con 0x564b3da00b90 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.676+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x564b3d9d7350 con 0x564b3da007b0 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f832d816740 10 client.0 _ll_register_callbacks cb 0x564b3d935350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da2e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da2e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42114/0 (socket says 172.21.15.114:42114) 2024-03-06T07:34:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 -- 172.21.15.114:0/2238429569 learned_addr learned my addr 172.21.15.114:0/2238429569 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8323e00640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8323400640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3d9f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 -- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 msgr2=0x564b3da17910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 -- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 msgr2=0x564b3d9f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3d9f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 -- 172.21.15.114:0/2238429569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b3d71c6b0 con 0x564b3da18b70 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8323400640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3d9f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f8328a00640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da2e440 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f831c081a10 tx=0x7f831c079850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfd06b331944659 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.677+0000 7f830cc00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f830cc00640 1 -- 172.21.15.114:0/2238429569 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c081f40 con 0x564b3da18b70 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f830cc00640 1 -- 172.21.15.114:0/2238429569 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f831c081ab0 con 0x564b3da18b70 2024-03-06T07:34:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8314005650 con 0x564b3da18b70 2024-03-06T07:34:13.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f830cc00640 1 -- 172.21.15.114:0/2238429569 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f831c081cb0 con 0x564b3da18b70 2024-03-06T07:34:13.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 10 client.7812 fetch_fsmap learned FSMap version 79 2024-03-06T07:34:13.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x564b3da173f0 con 0x564b3da18b70 2024-03-06T07:34:13.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f830cc00640 1 -- 172.21.15.114:0/2238429569 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8314005650 con 0x564b3da18b70 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f830cc00640 1 -- 172.21.15.114:0/2238429569 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 79) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f831c078ca0 con 0x564b3da18b70 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 10 client.7812 fetch_fsmap finished waiting for FSMap version 79 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 10 client.7812 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 20 client.7812 populate_metadata read hostname 'smithi114' 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da39f70 0x564b3da3c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 4 client.7812 mds_command: new command op to 7674 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.678+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 --> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b3d94ab40 con 0x564b3da39f70 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.679+0000 7f8323e00640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da39f70 0x564b3da3c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.679+0000 7f8323e00640 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da39f70 0x564b3da3c360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3049636d03183956 server_cookie=dbe74166e7540ed4 in_seq=0 out_seq=0 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.679+0000 7f830cc00640 10 client.7812 ms_handle_connect on v2:172.21.15.114:6835/860883596 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f830cc00640 1 -- 172.21.15.114:0/2238429569 <== mds.0 v2:172.21.15.114:6835/860883596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564b3d94ab40 con 0x564b3da39f70 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f830cc00640 10 client.7812 handle_command_reply: tid=0 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 client.7812 shutdown 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 msgr2=0x564b3da2e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da2e440 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f831c081a10 tx=0x7f831c079850 comp rx=0 tx=0).stop 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 shutdown_connections 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da39f70 0x564b3da3c360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da2e440 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3d9f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 --2- 172.21.15.114:0/2238429569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 >> 172.21.15.114:0/2238429569 conn(0x564b3d9d3c90 msgr2=0x564b3da03530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f830cc00640 0 client.7812 ms_handle_reset on v2:172.21.15.114:6835/860883596 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 shutdown_connections 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 1 -- 172.21.15.114:0/2238429569 wait complete. 2024-03-06T07:34:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.681+0000 7f832d816740 20 client.7812 trim_cache size 0 max 16384 2024-03-06T07:34:13.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.699+0000 7f832d816740 1 Processor -- start 2024-03-06T07:34:13.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.700+0000 7f832d816740 1 -- start start 2024-03-06T07:34:13.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.700+0000 7f832d816740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9f4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.700+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.700+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da18b70 0x564b3da37830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.700+0000 7f832d816740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324203e80 con 0x564b3da007b0 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.700+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324163f60 con 0x564b3da00b90 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.700+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8324164560 con 0x564b3da18b70 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9f4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42128/0 (socket says 172.21.15.114:42128) 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 -- 172.21.15.114:0/2922842352 learned_addr learned my addr 172.21.15.114:0/2922842352 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 -- 172.21.15.114:0/2922842352 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da18b70 msgr2=0x564b3da37830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 --2- 172.21.15.114:0/2922842352 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da18b70 0x564b3da37830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 -- 172.21.15.114:0/2922842352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 msgr2=0x564b3da372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 --2- 172.21.15.114:0/2922842352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 -- 172.21.15.114:0/2922842352 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b3da073a0 con 0x564b3da007b0 2024-03-06T07:34:13.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f8323e00640 1 --2- 172.21.15.114:0/2922842352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9f4220 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f831400fea0 tx=0x7f8314005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f27fda6785cedd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f830f400640 1 -- 172.21.15.114:0/2922842352 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314026040 con 0x564b3da007b0 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f830f400640 1 -- 172.21.15.114:0/2922842352 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831402d630 con 0x564b3da007b0 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.701+0000 7f830f400640 1 -- 172.21.15.114:0/2922842352 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831402da00 con 0x564b3da007b0 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 -- 172.21.15.114:0/2922842352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 msgr2=0x564b3d9f4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 --2- 172.21.15.114:0/2922842352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9f4220 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f831400fea0 tx=0x7f8314005000 comp rx=0 tx=0).stop 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 -- 172.21.15.114:0/2922842352 shutdown_connections 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 --2- 172.21.15.114:0/2922842352 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da18b70 0x564b3da37830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 --2- 172.21.15.114:0/2922842352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 --2- 172.21.15.114:0/2922842352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da007b0 0x564b3d9f4220 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 -- 172.21.15.114:0/2922842352 >> 172.21.15.114:0/2922842352 conn(0x564b3d9d3c90 msgr2=0x564b3d9fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 -- 172.21.15.114:0/2922842352 shutdown_connections 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.702+0000 7f832d816740 1 -- 172.21.15.114:0/2922842352 wait complete. 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.703+0000 7f832d816740 1 -- start start 2024-03-06T07:34:13.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.703+0000 7f832d816740 1 Processor -- start 2024-03-06T07:34:13.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f832d816740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da2d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f832d816740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da8d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f832d816740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3d9d7f60 con 0x564b3da18b70 2024-03-06T07:34:13.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b3d9d7350 con 0x564b3da00b90 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f832d816740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x564b3d8f9d90 con 0x564b3da007b0 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f832d816740 10 client.0 _ll_register_callbacks cb 0x7f831c02c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f8328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da8d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f8328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da8d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42136/0 (socket says 172.21.15.114:42136) 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.704+0000 7f8328a00640 1 -- 172.21.15.114:0/2550770750 learned_addr learned my addr 172.21.15.114:0/2550770750 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8323400640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da2d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8323e00640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8328a00640 1 -- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 msgr2=0x564b3da17c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8328a00640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8328a00640 1 -- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 msgr2=0x564b3da2d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8328a00640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da2d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8328a00640 1 -- 172.21.15.114:0/2550770750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b3d9f3b70 con 0x564b3da18b70 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8323e00640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f8328a00640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da8d290 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f831c009160 tx=0x7f831c0a94b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabdd9292104d5d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f830cc00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f830cc00640 1 -- 172.21.15.114:0/2550770750 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c081f40 con 0x564b3da18b70 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f830cc00640 1 -- 172.21.15.114:0/2550770750 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f831c081ab0 con 0x564b3da18b70 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f831402d630 con 0x564b3da18b70 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.705+0000 7f830cc00640 1 -- 172.21.15.114:0/2550770750 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f831c079390 con 0x564b3da18b70 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 10 client.7818 fetch_fsmap learned FSMap version 79 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x564b3d9f3b70 con 0x564b3da18b70 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f830cc00640 1 -- 172.21.15.114:0/2550770750 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831402d630 con 0x564b3da18b70 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f830cc00640 1 -- 172.21.15.114:0/2550770750 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 79) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f831c091040 con 0x564b3da18b70 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 10 client.7818 fetch_fsmap finished waiting for FSMap version 79 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 10 client.7818 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 20 client.7818 populate_metadata read hostname 'smithi114' 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da042b0 0x564b3da8d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 4 client.7818 mds_command: new command op to 7674 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 --> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564b3d944e90 con 0x564b3da042b0 2024-03-06T07:34:13.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.706+0000 7f8323e00640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da042b0 0x564b3da8d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f8323e00640 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da042b0 0x564b3da8d7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ad43be8fc06560 server_cookie=2607b163ee001d37 in_seq=0 out_seq=0 2024-03-06T07:34:13.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f830cc00640 10 client.7818 ms_handle_connect on v2:172.21.15.114:6835/860883596 2024-03-06T07:34:13.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f830cc00640 1 -- 172.21.15.114:0/2550770750 <== mds.0 v2:172.21.15.114:6835/860883596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x564b3d944e90 con 0x564b3da042b0 2024-03-06T07:34:13.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f830cc00640 10 client.7818 handle_command_reply: tid=0 2024-03-06T07:34:13.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 client.7818 shutdown 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 msgr2=0x564b3da8d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da8d290 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f831c009160 tx=0x7f831c0a94b0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 shutdown_connections 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:6835/860883596,v1:172.21.15.114:6837/860883596] conn(0x564b3da042b0 0x564b3da8d7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564b3da18b70 0x564b3da8d290 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b3da00b90 0x564b3da2d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 --2- 172.21.15.114:0/2550770750 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x564b3da007b0 0x564b3da17c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 >> 172.21.15.114:0/2550770750 conn(0x564b3d9d3c90 msgr2=0x564b3d9fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f830cc00640 0 client.7818 ms_handle_reset on v2:172.21.15.114:6835/860883596 2024-03-06T07:34:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 shutdown_connections 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.707+0000 7f832d816740 1 -- 172.21.15.114:0/2550770750 wait complete. 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832d816740 20 client.7818 trim_cache size 0 max 16384 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 7674, 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:34:13.713 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 6.7682538599999997, 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 79, 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 108, 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 12.198242299 2024-03-06T07:34:13.714 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f830406b4d0 msgr2=0x7f830406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f830406b4d0 0x7f830406d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8314000f80 tx=0x7f8314009040 comp rx=0 tx=0).stop 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83240b1480 msgr2=0x7f83240b1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83240b1480 0x7f83240b1900 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f831c004650 tx=0x7f831c004730 comp rx=0 tx=0).stop 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 shutdown_connections 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f830406b4d0 0x7f830406d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83240b1480 0x7f83240b1900 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f83240af530 0x7f83240b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 --2- 172.21.15.114:0/143514444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83080051c0 0x7f83241591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 >> 172.21.15.114:0/143514444 conn(0x7f83240cc4f0 msgr2=0x7f83240c5d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 shutdown_connections 2024-03-06T07:34:13.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.708+0000 7f832aa00640 1 -- 172.21.15.114:0/143514444 wait complete. 2024-03-06T07:34:13.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:34:13.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.913+0000 7f2b42400640 1 Processor -- start 2024-03-06T07:34:13.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.913+0000 7f2b42400640 1 -- start start 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b42400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0d22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b42400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2b3c0d0d90 0x7f2b3c0d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b42400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2b3c0b4d30 0x7f2b3c161e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b42400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c0d4800 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b42400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c0d1ac0 con 0x7f2b3c0b4d30 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b42400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0ca660 con 0x7f2b3c0ca280 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b3be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2b3c0d0d90 0x7f2b3c0d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b3be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2b3c0d0d90 0x7f2b3c0d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42148/0 (socket says 172.21.15.114:42148) 2024-03-06T07:34:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.915+0000 7f2b3be00640 1 -- 172.21.15.114:0/3651554742 learned_addr learned my addr 172.21.15.114:0/3651554742 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:13.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.916+0000 7f2b41a00640 1 --2- 172.21.15.114:0/3651554742 >> v2:172.21.15.177:3300/0 conn(0x7f2b3c0b4d30 0x7f2b3c161e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.916+0000 7f2b41000640 1 --2- 172.21.15.114:0/3651554742 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0d22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.916+0000 7f2b3be00640 1 -- 172.21.15.114:0/3651554742 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 msgr2=0x7f2b3c0d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.916+0000 7f2b3be00640 1 --2- 172.21.15.114:0/3651554742 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0d22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.916+0000 7f2b3be00640 1 -- 172.21.15.114:0/3651554742 >> v2:172.21.15.177:3300/0 conn(0x7f2b3c0b4d30 msgr2=0x7f2b3c161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:13.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.916+0000 7f2b3be00640 1 --2- 172.21.15.114:0/3651554742 >> v2:172.21.15.177:3300/0 conn(0x7f2b3c0b4d30 0x7f2b3c161e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.916+0000 7f2b3be00640 1 -- 172.21.15.114:0/3651554742 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c0b4980 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.917+0000 7f2b3be00640 1 --2- 172.21.15.114:0/3651554742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0d2800 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c06b410 tx=0x7f2b2802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a7032beee6c58e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.917+0000 7f2b3b400640 1 -- 172.21.15.114:0/3651554742 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28035020 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.917+0000 7f2b3b400640 1 -- 172.21.15.114:0/3651554742 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2802d040 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b3b400640 1 -- 172.21.15.114:0/3651554742 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28031a80 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 -- 172.21.15.114:0/3651554742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 msgr2=0x7f2b3c0d2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 --2- 172.21.15.114:0/3651554742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0d2800 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c06b410 tx=0x7f2b2802b900 comp rx=0 tx=0).stop 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 -- 172.21.15.114:0/3651554742 shutdown_connections 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 --2- 172.21.15.114:0/3651554742 >> v2:172.21.15.177:3300/0 conn(0x7f2b3c0b4d30 0x7f2b3c161e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 --2- 172.21.15.114:0/3651554742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0d2800 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 --2- 172.21.15.114:0/3651554742 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0d22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 -- 172.21.15.114:0/3651554742 >> 172.21.15.114:0/3651554742 conn(0x7f2b3c0d3ce0 msgr2=0x7f2b3c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 -- 172.21.15.114:0/3651554742 shutdown_connections 2024-03-06T07:34:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.918+0000 7f2b42400640 1 -- 172.21.15.114:0/3651554742 wait complete. 2024-03-06T07:34:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.920+0000 7f2b42400640 1 Processor -- start 2024-03-06T07:34:13.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.920+0000 7f2b42400640 1 -- start start 2024-03-06T07:34:13.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b42400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b3c0b4d30 0x7f2b3c0b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b42400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b42400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b42400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0d1630 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b42400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0d4a30 con 0x7f2b3c0b4d30 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b42400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c162380 con 0x7f2b3c0ca280 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b41a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b41a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42158/0 (socket says 172.21.15.114:42158) 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b41a00640 1 -- 172.21.15.114:0/1541694964 learned_addr learned my addr 172.21.15.114:0/1541694964 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b41000640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b3c0b4d30 0x7f2b3c0b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.922+0000 7f2b3be00640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b41a00640 1 -- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 msgr2=0x7f2b3c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b41a00640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b41a00640 1 -- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b3c0b4d30 msgr2=0x7f2b3c0b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b41a00640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b3c0b4d30 0x7f2b3c0b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b41a00640 1 -- 172.21.15.114:0/1541694964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c0b2780 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b3be00640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:13.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b41a00640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0b1c50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f2b34007e60 tx=0x7f2b3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b38c00640 1 -- 172.21.15.114:0/1541694964 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34014620 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b38c00640 1 -- 172.21.15.114:0/1541694964 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b34019070 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b38c00640 1 -- 172.21.15.114:0/1541694964 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34014a20 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c0b2a60 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.923+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c0b3020 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.925+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c04f330 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.925+0000 7f2b38c00640 1 -- 172.21.15.114:0/1541694964 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b34014bc0 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.926+0000 7f2b38c00640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:13.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.926+0000 7f2b38c00640 1 -- 172.21.15.114:0/1541694964 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b3408aaa0 con 0x7f2b3c0d0d90 2024-03-06T07:34:13.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.927+0000 7f2b41000640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:13.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.927+0000 7f2b41000640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c006fd0 tx=0x7f2b2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:13.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:13.934+0000 7f2b38c00640 1 -- 172.21.15.114:0/1541694964 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b34059910 con 0x7f2b3c0d0d90 2024-03-06T07:34:14.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.276+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2b3c0b9b50 con 0x7f2b3c0d0d90 2024-03-06T07:34:14.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.277+0000 7f2b38c00640 1 -- 172.21.15.114:0/1541694964 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v79) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2b34059210 con 0x7f2b3c0d0d90 2024-03-06T07:34:14.279 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.279+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2b1c06b4d0 msgr2=0x7f2b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.279+0000 7f2b42400640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c006fd0 tx=0x7f2b2c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.279+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 msgr2=0x7f2b3c0b1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0b1c50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f2b34007e60 tx=0x7f2b3400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 shutdown_connections 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b3c0d0d90 0x7f2b3c0b1c50 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2b3c0ca280 0x7f2b3c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 --2- 172.21.15.114:0/1541694964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b3c0b4d30 0x7f2b3c0b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 >> 172.21.15.114:0/1541694964 conn(0x7f2b3c0d3ce0 msgr2=0x7f2b3c15a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 shutdown_connections 2024-03-06T07:34:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:14.280+0000 7f2b42400640 1 -- 172.21.15.114:0/1541694964 wait complete. 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:34:14.301 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:14.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.301 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:34:14.343 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:14.344 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42201sec preferred_lft 42201sec 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:14.345 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:14.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.347 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:34:14.473 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:34:14.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.477 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:34:14.528 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:34:14.528 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:34:14.528 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:34:14.528 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:34:14.528 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:34:14.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.529 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:34:14.529 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:34:14.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:34:14.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:34:14.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:34:14.530 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:34:14.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.697 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:34:14.742 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:14.743 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:14.744 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:14.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:34:14.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:14.827 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:14.878 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:14.878 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:14.879 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:14.925 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:34:14.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:14.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:15.090 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:15.090 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:15.121 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:15.121 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:15.121 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:34:15.121 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:34:15.121 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:34:15.123 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:34:15.123 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:34:15.123 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:34:15.123 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:34:15.123 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:34:15.128 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:34:15.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:15.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.207 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:34:15.263 INFO:teuthology.orchestra.run.smithi114.stdout:client6082 2024-03-06T07:34:15.263 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:34:15.263 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:15.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.305 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:34:15.408 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:34:15.409 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:15.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.459 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:34:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:8+0 records in 2024-03-06T07:34:15.671 INFO:teuthology.orchestra.run.smithi114.stderr:8+0 records out 2024-03-06T07:34:15.672 INFO:teuthology.orchestra.run.smithi114.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.16337 s, 51.3 MB/s 2024-03-06T07:34:15.673 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:34:15.673 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:34:15.673 DEBUG:teuthology.orchestra.run.smithi114:> import stat 2024-03-06T07:34:15.673 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:34:15.673 DEBUG:teuthology.orchestra.run.smithi114:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:34:15.673 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:34:15.790 INFO:teuthology.orchestra.run.smithi114.stdout:1099511627777 2024-03-06T07:34:15.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:34:15.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.803 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:34:15.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:15.899 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:34:15.955 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:34:15.955 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:34:15.955 DEBUG:teuthology.orchestra.run.smithi114:> import stat 2024-03-06T07:34:15.955 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:34:15.955 DEBUG:teuthology.orchestra.run.smithi114:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:34:15.955 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:34:16.090 INFO:teuthology.orchestra.run.smithi114.stdout:1099511627777 2024-03-06T07:34:16.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:34:16.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:16.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:16.341 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:16.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.353+0000 7f585fa00640 1 Processor -- start 2024-03-06T07:34:16.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.353+0000 7f585fa00640 1 -- start start 2024-03-06T07:34:16.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.355+0000 7f585fa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f585815e0d0 0x7f58580d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.355+0000 7f585fa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f5858160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.355+0000 7f585fa00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58580cea60 con 0x7f585815f470 2024-03-06T07:34:16.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.355+0000 7f585fa00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58580d2f60 con 0x7f585815e0d0 2024-03-06T07:34:16.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.355+0000 7f585fa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580d2350 con 0x7f58580d0fc0 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585d000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f585815e0d0 0x7f58580d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585d000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f585815e0d0 0x7f58580d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:42110/0 (socket says 172.21.15.114:42110) 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f5858160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585d000640 1 -- 172.21.15.114:0/576316799 learned_addr learned my addr 172.21.15.114:0/576316799 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585d000640 1 -- 172.21.15.114:0/576316799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 msgr2=0x7f5858160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585d000640 1 --2- 172.21.15.114:0/576316799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f5858160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585d000640 1 -- 172.21.15.114:0/576316799 >> v1:172.21.15.177:6789/0 conn(0x7f585815f470 legacy=0x7f58580d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.356+0000 7f585d000640 1 -- 172.21.15.114:0/576316799 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580d2630 con 0x7f585815e0d0 2024-03-06T07:34:16.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.357+0000 7f585d000640 1 --2- 172.21.15.114:0/576316799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815e0d0 0x7f58580d1a90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f584c009870 tx=0x7f584c02ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36534694513267c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.358+0000 7f5857400640 1 -- 172.21.15.114:0/576316799 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c02f830 con 0x7f585815e0d0 2024-03-06T07:34:16.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.358+0000 7f5857400640 1 -- 172.21.15.114:0/576316799 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c02f9d0 con 0x7f585815e0d0 2024-03-06T07:34:16.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.358+0000 7f5857400640 1 -- 172.21.15.114:0/576316799 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c02fda0 con 0x7f585815e0d0 2024-03-06T07:34:16.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.358+0000 7f585fa00640 1 -- 172.21.15.114:0/576316799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815e0d0 msgr2=0x7f58580d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.358+0000 7f585fa00640 1 --2- 172.21.15.114:0/576316799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815e0d0 0x7f58580d1a90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f584c009870 tx=0x7f584c02ba10 comp rx=0 tx=0).stop 2024-03-06T07:34:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.359+0000 7f585fa00640 1 -- 172.21.15.114:0/576316799 shutdown_connections 2024-03-06T07:34:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.359+0000 7f585fa00640 1 --2- 172.21.15.114:0/576316799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f5858160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.359+0000 7f585fa00640 1 --2- 172.21.15.114:0/576316799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815e0d0 0x7f58580d1a90 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.359+0000 7f585fa00640 1 -- 172.21.15.114:0/576316799 >> 172.21.15.114:0/576316799 conn(0x7f58580d3ec0 msgr2=0x7f58580d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.359+0000 7f585fa00640 1 -- 172.21.15.114:0/576316799 shutdown_connections 2024-03-06T07:34:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.359+0000 7f585fa00640 1 -- 172.21.15.114:0/576316799 wait complete. 2024-03-06T07:34:16.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.360+0000 7f585fa00640 1 Processor -- start 2024-03-06T07:34:16.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.361+0000 7f585fa00640 1 -- start start 2024-03-06T07:34:16.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585fa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f58580b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585fa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585815e0d0 0x7f58580b8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585fa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815f470 0x7f5858155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585fa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585815fb30 con 0x7f585815e0d0 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585fa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58580d2bf0 con 0x7f585815f470 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585fa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580d1fd0 con 0x7f58580d0fc0 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f5857e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585815e0d0 0x7f58580b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f5857e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585815e0d0 0x7f58580b8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42174/0 (socket says 172.21.15.114:42174) 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f5857e00640 1 -- 172.21.15.114:0/2184022650 learned_addr learned my addr 172.21.15.114:0/2184022650 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585d000640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f58580b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.362+0000 7f585da00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815f470 0x7f5858155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5857e00640 1 -- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 msgr2=0x7f58580b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5857e00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f58580b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5857e00640 1 -- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815f470 msgr2=0x7f5858155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5857e00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815f470 0x7f5858155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5857e00640 1 -- 172.21.15.114:0/2184022650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580b8f30 con 0x7f585815e0d0 2024-03-06T07:34:16.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f585da00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815f470 0x7f5858155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:16.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5857e00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585815e0d0 0x7f58580b8700 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f584400c3f0 tx=0x7f584400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5854c00640 1 -- 172.21.15.114:0/2184022650 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58440052d0 con 0x7f585815e0d0 2024-03-06T07:34:16.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5854c00640 1 -- 172.21.15.114:0/2184022650 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5844005490 con 0x7f585815e0d0 2024-03-06T07:34:16.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f5854c00640 1 -- 172.21.15.114:0/2184022650 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844007260 con 0x7f585815e0d0 2024-03-06T07:34:16.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.363+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58580caf90 con 0x7f585815e0d0 2024-03-06T07:34:16.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.364+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58580cb550 con 0x7f585815e0d0 2024-03-06T07:34:16.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.365+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585804f330 con 0x7f585815e0d0 2024-03-06T07:34:16.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.366+0000 7f5854c00640 1 -- 172.21.15.114:0/2184022650 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5844007400 con 0x7f585815e0d0 2024-03-06T07:34:16.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.367+0000 7f5854c00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f583806b4d0 0x7f583806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.367+0000 7f5854c00640 1 -- 172.21.15.114:0/2184022650 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5844089630 con 0x7f585815e0d0 2024-03-06T07:34:16.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.367+0000 7f585d000640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f583806b4d0 0x7f583806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.368+0000 7f585d000640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f583806b4d0 0x7f583806d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f584c036000 tx=0x7f584c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.372+0000 7f5854c00640 1 -- 172.21.15.114:0/2184022650 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5844057e60 con 0x7f585815e0d0 2024-03-06T07:34:16.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.609+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f585808d920 con 0x7f585815e0d0 2024-03-06T07:34:16.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.610+0000 7f5854c00640 1 -- 172.21.15.114:0/2184022650 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f584405ce20 con 0x7f585815e0d0 2024-03-06T07:34:16.612 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:16.612 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":79,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5983,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3305544049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3305544049},{"type":"v1","addr":"172.21.15.114:6836","nonce":3305544049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6001,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/3949834778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3949834778},{"type":"v1","addr":"172.21.15.177:6834","nonce":3949834778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6004,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2151340900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2151340900},{"type":"v1","addr":"172.21.15.177:6835","nonce":2151340900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:09.452793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7674},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7674":{"gid":7674,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/860883596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":860883596},{"type":"v1","addr":"172.21.15.114:6837","nonce":860883596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7674,"qdb_cluster":[7674]},"id":8}]} 2024-03-06T07:34:16.612 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 79 2024-03-06T07:34:16.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f583806b4d0 msgr2=0x7f583806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f583806b4d0 0x7f583806d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f584c036000 tx=0x7f584c008030 comp rx=0 tx=0).stop 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585815e0d0 msgr2=0x7f58580b8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585815e0d0 0x7f58580b8700 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f584400c3f0 tx=0x7f584400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 shutdown_connections 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f585815f470 0x7f5858155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f583806b4d0 0x7f583806d990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585815e0d0 0x7f58580b8700 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 --2- 172.21.15.114:0/2184022650 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58580d0fc0 0x7f58580b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 >> 172.21.15.114:0/2184022650 conn(0x7f58580d3ec0 msgr2=0x7f58580b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 shutdown_connections 2024-03-06T07:34:16.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.613+0000 7f585fa00640 1 -- 172.21.15.114:0/2184022650 wait complete. 2024-03-06T07:34:16.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:34:16.869 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:34:16.869 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:16.869 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:34:16.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.952+0000 7f1dde012f40 1 Processor -- start 2024-03-06T07:34:16.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.952+0000 7f1dde012f40 1 -- start start 2024-03-06T07:34:16.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dde012f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edec990 0x555a8eeaa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dde012f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edecd70 0x555a8ee35fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dde012f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8eebf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dde012f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8edebd10 con 0x555a8eeaa9b0 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dde012f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8ed92fd0 con 0x555a8edec990 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dde012f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a8edebfb0 con 0x555a8edecd70 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8eebf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8eebf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42202/0 (socket says 172.21.15.114:42202) 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 -- 172.21.15.114:0/3522919348 learned_addr learned my addr 172.21.15.114:0/3522919348 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dda600640 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edec990 0x555a8eeaa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1dd3e00640 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edecd70 0x555a8ee35fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 -- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edecd70 msgr2=0x555a8ee35fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edecd70 0x555a8ee35fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 -- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edec990 msgr2=0x555a8eeaa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edec990 0x555a8eeaa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.953+0000 7f1ddb000640 1 -- 172.21.15.114:0/3522919348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a8ee1f020 con 0x555a8eeaa9b0 2024-03-06T07:34:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.954+0000 7f1ddb000640 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8eebf190 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400b600 tx=0x7f1dd400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c5e48ff12c8631 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.954+0000 7f1dd9200640 1 -- 172.21.15.114:0/3522919348 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4014030 con 0x555a8eeaa9b0 2024-03-06T07:34:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.954+0000 7f1dd9200640 1 -- 172.21.15.114:0/3522919348 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd4010620 con 0x555a8eeaa9b0 2024-03-06T07:34:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.954+0000 7f1dd9200640 1 -- 172.21.15.114:0/3522919348 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40109f0 con 0x555a8eeaa9b0 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.954+0000 7f1dde012f40 1 -- 172.21.15.114:0/3522919348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 msgr2=0x555a8eebf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.954+0000 7f1dde012f40 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8eebf190 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400b600 tx=0x7f1dd400bac0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 -- 172.21.15.114:0/3522919348 shutdown_connections 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8eebf190 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edecd70 0x555a8ee35fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 --2- 172.21.15.114:0/3522919348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edec990 0x555a8eeaa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 -- 172.21.15.114:0/3522919348 >> 172.21.15.114:0/3522919348 conn(0x555a8ee1bc50 msgr2=0x555a8eeaaee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 -- 172.21.15.114:0/3522919348 shutdown_connections 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 -- 172.21.15.114:0/3522919348 wait complete. 2024-03-06T07:34:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.955+0000 7f1dde012f40 1 Processor -- start 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.956+0000 7f1dde012f40 1 -- start start 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.956+0000 7f1dde012f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edec990 0x555a8eeba0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.956+0000 7f1dde012f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edecd70 0x555a8eec1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.956+0000 7f1dde012f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8ee2ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1dde012f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8ee1f2c0 con 0x555a8eeaa9b0 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1dde012f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a8ee1dfa0 con 0x555a8edecd70 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1dde012f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a8ee1e240 con 0x555a8edec990 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8ee2ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8ee2ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42212/0 (socket says 172.21.15.114:42212) 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 -- 172.21.15.114:0/2126513692 learned_addr learned my addr 172.21.15.114:0/2126513692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1dda600640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edec990 0x555a8eeba0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1dd3e00640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edecd70 0x555a8eec1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 -- 172.21.15.114:0/2126513692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edec990 msgr2=0x555a8eeba0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edec990 0x555a8eeba0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 -- 172.21.15.114:0/2126513692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edecd70 msgr2=0x555a8eec1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a8edecd70 0x555a8eec1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1ddb000640 1 -- 172.21.15.114:0/2126513692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a8eebdc60 con 0x555a8eeaa9b0 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.957+0000 7f1dda600640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555a8edec990 0x555a8eeba0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.958+0000 7f1ddb000640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8ee2ea80 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x555a8eeac8e0 tx=0x7f1dd40033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.958+0000 7f1dd2a00640 1 -- 172.21.15.114:0/2126513692 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4027070 con 0x555a8eeaa9b0 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.958+0000 7f1dd2a00640 1 -- 172.21.15.114:0/2126513692 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd401a9a0 con 0x555a8eeaa9b0 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.958+0000 7f1dd2a00640 1 -- 172.21.15.114:0/2126513692 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd401ad90 con 0x555a8eeaa9b0 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.958+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555a8eecd020 con 0x555a8eeaa9b0 2024-03-06T07:34:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.958+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a8eeba620 con 0x555a8eeaa9b0 2024-03-06T07:34:16.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.960+0000 7f1dd2a00640 1 -- 172.21.15.114:0/2126513692 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1dd402a070 con 0x555a8eeaa9b0 2024-03-06T07:34:16.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.961+0000 7f1dd2a00640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1db8069eb0 0x7f1db806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.961+0000 7f1dd2a00640 1 -- 172.21.15.114:0/2126513692 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1dd408a380 con 0x555a8eeaa9b0 2024-03-06T07:34:16.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.961+0000 7f1dda600640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1db8069eb0 0x7f1db806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.961+0000 7f1dde012f40 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555a8eec22e0 0x555a8ee31a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:16.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.961+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x555a8eebe910 con 0x555a8eec22e0 2024-03-06T07:34:16.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.962+0000 7f1dd3e00640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555a8eec22e0 0x555a8ee31a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:16.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.962+0000 7f1dda600640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1db8069eb0 0x7f1db806c370 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc054000 tx=0x7f1dcc04e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.962+0000 7f1dd3e00640 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555a8eec22e0 0x555a8ee31a20 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:16.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dd3e00640 1 -- 172.21.15.114:0/2126513692 <== osd.0 v2:172.21.15.114:6800/3895229755 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f1dc400e4b0 con 0x555a8eec22e0 2024-03-06T07:34:16.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555a8eec22e0 msgr2=0x555a8ee31a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555a8eec22e0 0x555a8ee31a20 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1db8069eb0 msgr2=0x7f1db806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1db8069eb0 0x7f1db806c370 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc054000 tx=0x7f1dcc04e000 comp rx=0 tx=0).stop 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 msgr2=0x555a8ee2ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 --2- 172.21.15.114:0/2126513692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a8eeaa9b0 0x555a8ee2ea80 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x555a8eeac8e0 tx=0x7f1dd40033a0 comp rx=0 tx=0).stop 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1ddb000640 1 -- 172.21.15.114:0/2126513692 reap_dead start 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 shutdown_connections 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 >> 172.21.15.114:0/2126513692 conn(0x555a8ee1bc50 msgr2=0x555a8ee1cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 shutdown_connections 2024-03-06T07:34:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:16.963+0000 7f1dde012f40 1 -- 172.21.15.114:0/2126513692 wait complete. 2024-03-06T07:34:16.969 INFO:teuthology.orchestra.run.smithi114.stdout:Wfile_adir_1 2024-03-06T07:34:16.969 DEBUG:teuthology.orchestra.run.smithi114:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 1099511627777, 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: "ancestors": [ 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 1099511627776, 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "file_a", 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 8 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 1, 2024-03-06T07:34:17.132 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "dir_1", 2024-03-06T07:34:17.133 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 15 2024-03-06T07:34:17.133 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:34:17.133 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-03-06T07:34:17.133 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 17, 2024-03-06T07:34:17.133 INFO:teuthology.orchestra.run.smithi114.stdout: "old_pools": [] 2024-03-06T07:34:17.133 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:34:17.133 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:17.153 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:34:17.153 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:17.202 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:34:17.202 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:17.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.516+0000 7f30b9400640 1 Processor -- start 2024-03-06T07:34:17.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.516+0000 7f30b9400640 1 -- start start 2024-03-06T07:34:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.517+0000 7f30b9400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f30b415f9d0 0x7f30b415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.517+0000 7f30b9400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b4070b10 0x7f30b4070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.517+0000 7f30b9400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b406cef0 con 0x7f30b4144c00 2024-03-06T07:34:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.517+0000 7f30b9400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30b406d1d0 con 0x7f30b415f9d0 2024-03-06T07:34:17.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.517+0000 7f30b9400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b406d4b0 con 0x7f30b4070b10 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.518+0000 7f30a9e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f30b415f9d0 0x7f30b415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.518+0000 7f30b3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b4070b10 0x7f30b4070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.518+0000 7f30a9e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f30b415f9d0 0x7f30b415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:42154/0 (socket says 172.21.15.114:42154) 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.518+0000 7f30a9e00640 1 -- 172.21.15.114:0/2517799318 learned_addr learned my addr 172.21.15.114:0/2517799318 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.518+0000 7f30b2800640 1 --1- 172.21.15.114:0/2517799318 >> v1:172.21.15.177:6789/0 conn(0x7f30b4144c00 0x7f30b406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:43182/0 (socket says 172.21.15.114:43182) 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.519+0000 7f30b1e00640 1 -- 172.21.15.114:0/2517799318 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58705674 0 0) 0x7f30b406cef0 con 0x7f30b4144c00 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.519+0000 7f30b1e00640 1 -- 172.21.15.114:0/2517799318 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3098003520 con 0x7f30b4144c00 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.519+0000 7f30a9e00640 1 -- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b4070b10 msgr2=0x7f30b4070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.519+0000 7f30a9e00640 1 --2- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b4070b10 0x7f30b4070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.519+0000 7f30a9e00640 1 -- 172.21.15.114:0/2517799318 >> v1:172.21.15.177:6789/0 conn(0x7f30b4144c00 legacy=0x7f30b406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.519+0000 7f30a9e00640 1 -- 172.21.15.114:0/2517799318 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b406d790 con 0x7f30b415f9d0 2024-03-06T07:34:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.519+0000 7f30b3200640 1 --2- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b4070b10 0x7f30b4070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:17.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.521+0000 7f30a9e00640 1 --2- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b415f9d0 0x7f30b415fe60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009870 tx=0x7f30a402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=305cf79b8d94da5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.521+0000 7f30b1e00640 1 -- 172.21.15.114:0/2517799318 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4034070 con 0x7f30b415f9d0 2024-03-06T07:34:17.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b1e00640 1 -- 172.21.15.114:0/2517799318 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a4030840 con 0x7f30b415f9d0 2024-03-06T07:34:17.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b1e00640 1 -- 172.21.15.114:0/2517799318 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4030bf0 con 0x7f30b415f9d0 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 -- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b415f9d0 msgr2=0x7f30b415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 --2- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b415f9d0 0x7f30b415fe60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009870 tx=0x7f30a402b980 comp rx=0 tx=0).stop 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 -- 172.21.15.114:0/2517799318 shutdown_connections 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 --2- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b4070b10 0x7f30b4070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 --2- 172.21.15.114:0/2517799318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b415f9d0 0x7f30b415fe60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 -- 172.21.15.114:0/2517799318 >> 172.21.15.114:0/2517799318 conn(0x7f30b40b1730 msgr2=0x7f30b40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 -- 172.21.15.114:0/2517799318 shutdown_connections 2024-03-06T07:34:17.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.522+0000 7f30b9400640 1 -- 172.21.15.114:0/2517799318 wait complete. 2024-03-06T07:34:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.525+0000 7f30b9400640 1 Processor -- start 2024-03-06T07:34:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.525+0000 7f30b9400640 1 -- start start 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30b9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b4070b10 0x7f30b4142570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30b9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30b4144c00 0x7f30b41e7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30b9400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b415f9d0 0x7f30b41e8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30b9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b406ccd0 con 0x7f30b4144c00 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30b9400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b406cb50 con 0x7f30b4070b10 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30b9400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b406df90 con 0x7f30b415f9d0 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30b4144c00 0x7f30b41e7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30a9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30b4144c00 0x7f30b41e7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42216/0 (socket says 172.21.15.114:42216) 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.526+0000 7f30a9e00640 1 -- 172.21.15.114:0/3410561333 learned_addr learned my addr 172.21.15.114:0/3410561333 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30b2800640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b4070b10 0x7f30b4142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30b3200640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b415f9d0 0x7f30b41e8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30a9e00640 1 -- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b415f9d0 msgr2=0x7f30b41e8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30a9e00640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b415f9d0 0x7f30b41e8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30a9e00640 1 -- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b4070b10 msgr2=0x7f30b4142570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30a9e00640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b4070b10 0x7f30b4142570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30a9e00640 1 -- 172.21.15.114:0/3410561333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b41e6610 con 0x7f30b4144c00 2024-03-06T07:34:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30b3200640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b415f9d0 0x7f30b41e8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:17.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.527+0000 7f30a9e00640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30b4144c00 0x7f30b41e7fd0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f30a402b9c0 tx=0x7f30a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.528+0000 7f30ab400640 1 -- 172.21.15.114:0/3410561333 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4047070 con 0x7f30b4144c00 2024-03-06T07:34:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.528+0000 7f30ab400640 1 -- 172.21.15.114:0/3410561333 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a4030e00 con 0x7f30b4144c00 2024-03-06T07:34:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.528+0000 7f30ab400640 1 -- 172.21.15.114:0/3410561333 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4034040 con 0x7f30b4144c00 2024-03-06T07:34:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.528+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b41e68b0 con 0x7f30b4144c00 2024-03-06T07:34:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.528+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b41e6e70 con 0x7f30b4144c00 2024-03-06T07:34:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.529+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b404f330 con 0x7f30b4144c00 2024-03-06T07:34:17.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.531+0000 7f30ab400640 1 -- 172.21.15.114:0/3410561333 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30a4037070 con 0x7f30b4144c00 2024-03-06T07:34:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.532+0000 7f30ab400640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f309006b4d0 0x7f309006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.532+0000 7f30ab400640 1 -- 172.21.15.114:0/3410561333 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30a40abaf0 con 0x7f30b4144c00 2024-03-06T07:34:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.532+0000 7f30b2800640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f309006b4d0 0x7f309006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.533+0000 7f30b2800640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f309006b4d0 0x7f309006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c007b10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.540+0000 7f30ab400640 1 -- 172.21.15.114:0/3410561333 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30a407aae0 con 0x7f30b4144c00 2024-03-06T07:34:17.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.735+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30b40cc800 con 0x7f30b4144c00 2024-03-06T07:34:17.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.737+0000 7f30ab400640 1 -- 172.21.15.114:0/3410561333 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16675 (secure 0 0 0) 0x7f30a407a3e0 con 0x7f30b4144c00 2024-03-06T07:34:17.740 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:17.740 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":108,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:07.954457+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f309006b4d0 msgr2=0x7f309006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f309006b4d0 0x7f309006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c007b10 comp rx=0 tx=0).stop 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30b4144c00 msgr2=0x7f30b41e7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30b4144c00 0x7f30b41e7fd0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f30a402b9c0 tx=0x7f30a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 shutdown_connections 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f30b415f9d0 0x7f30b41e8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f309006b4d0 0x7f309006d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30b4144c00 0x7f30b41e7fd0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 --2- 172.21.15.114:0/3410561333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30b4070b10 0x7f30b4142570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 >> 172.21.15.114:0/3410561333 conn(0x7f30b40b1730 msgr2=0x7f30b40af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 shutdown_connections 2024-03-06T07:34:17.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.740+0000 7f30b9400640 1 -- 172.21.15.114:0/3410561333 wait complete. 2024-03-06T07:34:17.759 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:04.207971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:05.420538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:17.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:17.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.961+0000 7fefdd400640 1 Processor -- start 2024-03-06T07:34:17.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.961+0000 7fefdd400640 1 -- start start 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefdd400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fefd80b3df0 0x7fefd80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefdd400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fefd80cda90 0x7fefd806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefdd400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd80719e0 con 0x7fefd80b3650 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefdd400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd8072310 con 0x7fefd80b3df0 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefdd400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd8071cc0 con 0x7fefd80cda90 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefd5e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fefd80b3650 0x7fefd80b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37768/0 (socket says 172.21.15.114:37768) 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefd5e00640 1 -- 172.21.15.114:0/4231442306 learned_addr learned my addr 172.21.15.114:0/4231442306 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:17.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.964+0000 7fefd6800640 1 --2- 172.21.15.114:0/4231442306 >> v2:172.21.15.114:3300/0 conn(0x7fefd80b3df0 0x7fefd80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd7200640 1 --2- 172.21.15.114:0/4231442306 >> v2:172.21.15.177:3300/0 conn(0x7fefd80cda90 0x7fefd806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd5400640 1 -- 172.21.15.114:0/4231442306 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083331887 0 0) 0x7fefd80719e0 con 0x7fefd80b3650 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd5400640 1 -- 172.21.15.114:0/4231442306 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefbc003520 con 0x7fefd80b3650 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd5400640 1 -- 172.21.15.114:0/4231442306 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 542255192 0 0) 0x7fefbc003520 con 0x7fefd80b3650 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd6800640 1 -- 172.21.15.114:0/4231442306 >> v2:172.21.15.177:3300/0 conn(0x7fefd80cda90 msgr2=0x7fefd806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd6800640 1 --2- 172.21.15.114:0/4231442306 >> v2:172.21.15.177:3300/0 conn(0x7fefd80cda90 0x7fefd806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd6800640 1 -- 172.21.15.114:0/4231442306 >> v1:172.21.15.114:6789/0 conn(0x7fefd80b3650 legacy=0x7fefd80b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.965+0000 7fefd6800640 1 -- 172.21.15.114:0/4231442306 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd8071fa0 con 0x7fefd80b3df0 2024-03-06T07:34:17.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.967+0000 7fefd6800640 1 --2- 172.21.15.114:0/4231442306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3df0 0x7fefd80727a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fefd806ed00 tx=0x7fefc802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd347edc8e1b98bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.967+0000 7fefd5400640 1 -- 172.21.15.114:0/4231442306 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8038070 con 0x7fefd80b3df0 2024-03-06T07:34:17.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.967+0000 7fefd5400640 1 -- 172.21.15.114:0/4231442306 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc8033440 con 0x7fefd80b3df0 2024-03-06T07:34:17.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.967+0000 7fefd5400640 1 -- 172.21.15.114:0/4231442306 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4001670 con 0x7fefd80b3df0 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 -- 172.21.15.114:0/4231442306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3df0 msgr2=0x7fefd80727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 --2- 172.21.15.114:0/4231442306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3df0 0x7fefd80727a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fefd806ed00 tx=0x7fefc802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 -- 172.21.15.114:0/4231442306 shutdown_connections 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 --2- 172.21.15.114:0/4231442306 >> v2:172.21.15.177:3300/0 conn(0x7fefd80cda90 0x7fefd806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 --2- 172.21.15.114:0/4231442306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3df0 0x7fefd80727a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 -- 172.21.15.114:0/4231442306 >> 172.21.15.114:0/4231442306 conn(0x7fefd80b1940 msgr2=0x7fefd80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 -- 172.21.15.114:0/4231442306 shutdown_connections 2024-03-06T07:34:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.968+0000 7fefdd400640 1 -- 172.21.15.114:0/4231442306 wait complete. 2024-03-06T07:34:17.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.970+0000 7fefdd400640 1 Processor -- start 2024-03-06T07:34:17.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.970+0000 7fefdd400640 1 -- start start 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefdd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3650 0x7fefd815a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefdd400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fefd80b3df0 0x7fefd815de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefdd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd80cda90 0x7fefd814ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefdd400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd815b3c0 con 0x7fefd80b3650 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefdd400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd815b240 con 0x7fefd80cda90 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefdd400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80cf510 con 0x7fefd80b3df0 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefd6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3650 0x7fefd815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefd6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3650 0x7fefd815a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38410/0 (socket says 172.21.15.114:38410) 2024-03-06T07:34:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.971+0000 7fefd6800640 1 -- 172.21.15.114:0/2229726026 learned_addr learned my addr 172.21.15.114:0/2229726026 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:17.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefd6800640 1 -- 172.21.15.114:0/2229726026 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fefd80b3df0 msgr2=0x7fefd815de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefd6800640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fefd80b3df0 0x7fefd815de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefd6800640 1 -- 172.21.15.114:0/2229726026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd80cda90 msgr2=0x7fefd814ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefd6800640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd80cda90 0x7fefd814ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefd6800640 1 -- 172.21.15.114:0/2229726026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd814cdc0 con 0x7fefd80b3650 2024-03-06T07:34:17.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefd6800640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3650 0x7fefd815a580 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fefc8000c00 tx=0x7fefc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefcea00640 1 -- 172.21.15.114:0/2229726026 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8043070 con 0x7fefd80b3650 2024-03-06T07:34:17.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefcea00640 1 -- 172.21.15.114:0/2229726026 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc803f890 con 0x7fefd80b3650 2024-03-06T07:34:17.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefcea00640 1 -- 172.21.15.114:0/2229726026 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8038040 con 0x7fefd80b3650 2024-03-06T07:34:17.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd8149bd0 con 0x7fefd80b3650 2024-03-06T07:34:17.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.972+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd814a190 con 0x7fefd80b3650 2024-03-06T07:34:17.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.974+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd815b5a0 con 0x7fefd80b3650 2024-03-06T07:34:17.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.975+0000 7fefcea00640 1 -- 172.21.15.114:0/2229726026 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefc8046070 con 0x7fefd80b3650 2024-03-06T07:34:17.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.975+0000 7fefcea00640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fefb806b4d0 0x7fefb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.976+0000 7fefcea00640 1 -- 172.21.15.114:0/2229726026 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefc80abff0 con 0x7fefd80b3650 2024-03-06T07:34:17.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.976+0000 7fefd5e00640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fefb806b4d0 0x7fefb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.976+0000 7fefd5e00640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fefb806b4d0 0x7fefb806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fefc40011e0 tx=0x7fefc40019a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:17.980+0000 7fefcea00640 1 -- 172.21.15.114:0/2229726026 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefd815b5a0 con 0x7fefd80b3650 2024-03-06T07:34:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.176+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fefd804f330 con 0x7fefd80b3650 2024-03-06T07:34:18.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.177+0000 7fefcea00640 1 -- 172.21.15.114:0/2229726026 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefd814cdc0 con 0x7fefd80b3650 2024-03-06T07:34:18.179 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:34:18.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.179+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fefb806b4d0 msgr2=0x7fefb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.179+0000 7fefdd400640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fefb806b4d0 0x7fefb806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fefc40011e0 tx=0x7fefc40019a0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.179+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3650 msgr2=0x7fefd815a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.179+0000 7fefdd400640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3650 0x7fefd815a580 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fefc8000c00 tx=0x7fefc80046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.179+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 shutdown_connections 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.179+0000 7fefdd400640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd80cda90 0x7fefd814ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.180+0000 7fefdd400640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fefd80b3df0 0x7fefd815de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.180+0000 7fefdd400640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fefb806b4d0 0x7fefb806d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.180+0000 7fefdd400640 1 --2- 172.21.15.114:0/2229726026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd80b3650 0x7fefd815a580 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.180+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 >> 172.21.15.114:0/2229726026 conn(0x7fefd80b1940 msgr2=0x7fefd81523f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.180+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 shutdown_connections 2024-03-06T07:34:18.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.180+0000 7fefdd400640 1 -- 172.21.15.114:0/2229726026 wait complete. 2024-03-06T07:34:18.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:18.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.402+0000 7fa53d400640 1 Processor -- start 2024-03-06T07:34:18.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.403+0000 7fa53d400640 1 -- start start 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa53d400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa5380b3f70 0x7fa538160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa53d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa538070ff0 0x7fa5380ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa53d400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa538071440 con 0x7fa53815fc10 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa53d400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5380b46c0 con 0x7fa538070ff0 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa53d400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5380b49a0 con 0x7fa5380b3f70 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa536800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa53815fc10 0x7fa53815fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37772/0 (socket says 172.21.15.114:37772) 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa536800640 1 -- 172.21.15.114:0/3881496231 learned_addr learned my addr 172.21.15.114:0/3881496231 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:18.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.404+0000 7fa537200640 1 --2- 172.21.15.114:0/3881496231 >> v2:172.21.15.114:3300/0 conn(0x7fa538070ff0 0x7fa5380ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.405+0000 7fa52de00640 1 --2- 172.21.15.114:0/3881496231 >> v2:172.21.15.177:3300/0 conn(0x7fa5380b3f70 0x7fa538160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.405+0000 7fa537200640 1 -- 172.21.15.114:0/3881496231 >> v2:172.21.15.177:3300/0 conn(0x7fa5380b3f70 msgr2=0x7fa538160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.405+0000 7fa537200640 1 --2- 172.21.15.114:0/3881496231 >> v2:172.21.15.177:3300/0 conn(0x7fa5380b3f70 0x7fa538160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.405+0000 7fa537200640 1 -- 172.21.15.114:0/3881496231 >> v1:172.21.15.114:6789/0 conn(0x7fa53815fc10 legacy=0x7fa53815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.405+0000 7fa537200640 1 -- 172.21.15.114:0/3881496231 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538072c90 con 0x7fa538070ff0 2024-03-06T07:34:18.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.405+0000 7fa535e00640 1 -- 172.21.15.114:0/3881496231 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874216113 0 0) 0x7fa538071440 con 0x7fa53815fc10 2024-03-06T07:34:18.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.406+0000 7fa537200640 1 --2- 172.21.15.114:0/3881496231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa538070ff0 0x7fa5380ce4a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fa538074a50 tx=0x7fa53002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca393b2c3dd5fea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.406+0000 7fa535e00640 1 -- 172.21.15.114:0/3881496231 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530039070 con 0x7fa538070ff0 2024-03-06T07:34:18.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa535e00640 1 -- 172.21.15.114:0/3881496231 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53002cc70 con 0x7fa538070ff0 2024-03-06T07:34:18.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa535e00640 1 -- 172.21.15.114:0/3881496231 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530034660 con 0x7fa538070ff0 2024-03-06T07:34:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 -- 172.21.15.114:0/3881496231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa538070ff0 msgr2=0x7fa5380ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 --2- 172.21.15.114:0/3881496231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa538070ff0 0x7fa5380ce4a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fa538074a50 tx=0x7fa53002bec0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 -- 172.21.15.114:0/3881496231 shutdown_connections 2024-03-06T07:34:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 --2- 172.21.15.114:0/3881496231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa538070ff0 0x7fa5380ce4a0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 --2- 172.21.15.114:0/3881496231 >> v2:172.21.15.177:3300/0 conn(0x7fa5380b3f70 0x7fa538160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 -- 172.21.15.114:0/3881496231 >> 172.21.15.114:0/3881496231 conn(0x7fa5380b1940 msgr2=0x7fa5380b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 -- 172.21.15.114:0/3881496231 shutdown_connections 2024-03-06T07:34:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.407+0000 7fa53d400640 1 -- 172.21.15.114:0/3881496231 wait complete. 2024-03-06T07:34:18.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.410+0000 7fa53d400640 1 Processor -- start 2024-03-06T07:34:18.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.410+0000 7fa53d400640 1 -- start start 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa53d400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa538070ff0 0x7fa53815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa53d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5380b3f70 0x7fa53815ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa53d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa53815fc10 0x7fa538151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa53d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa538072a30 con 0x7fa5380b3f70 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa53d400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380728b0 con 0x7fa53815fc10 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa53d400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53815c2e0 con 0x7fa538070ff0 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa52de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5380b3f70 0x7fa53815ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa52de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5380b3f70 0x7fa53815ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38432/0 (socket says 172.21.15.114:38432) 2024-03-06T07:34:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.412+0000 7fa52de00640 1 -- 172.21.15.114:0/4153931019 learned_addr learned my addr 172.21.15.114:0/4153931019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:18.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52de00640 1 -- 172.21.15.114:0/4153931019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa538070ff0 msgr2=0x7fa53815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52de00640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa538070ff0 0x7fa53815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52de00640 1 -- 172.21.15.114:0/4153931019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa53815fc10 msgr2=0x7fa538151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52de00640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa53815fc10 0x7fa538151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52de00640 1 -- 172.21.15.114:0/4153931019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53815d160 con 0x7fa5380b3f70 2024-03-06T07:34:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52de00640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5380b3f70 0x7fa53815ebb0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fa52000d7d0 tx=0x7fa52000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52f400640 1 -- 172.21.15.114:0/4153931019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52000b6c0 con 0x7fa5380b3f70 2024-03-06T07:34:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52f400640 1 -- 172.21.15.114:0/4153931019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa520019070 con 0x7fa5380b3f70 2024-03-06T07:34:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa52f400640 1 -- 172.21.15.114:0/4153931019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52000ba80 con 0x7fa5380b3f70 2024-03-06T07:34:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53815d420 con 0x7fa5380b3f70 2024-03-06T07:34:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.413+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53815d9e0 con 0x7fa5380b3f70 2024-03-06T07:34:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.415+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53804f330 con 0x7fa5380b3f70 2024-03-06T07:34:18.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.416+0000 7fa52f400640 1 -- 172.21.15.114:0/4153931019 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa52000bc20 con 0x7fa5380b3f70 2024-03-06T07:34:18.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.417+0000 7fa52f400640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa51406b4d0 0x7fa51406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.417+0000 7fa52f400640 1 -- 172.21.15.114:0/4153931019 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa52008a660 con 0x7fa5380b3f70 2024-03-06T07:34:18.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.417+0000 7fa536800640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa51406b4d0 0x7fa51406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.418+0000 7fa536800640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa51406b4d0 0x7fa51406d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa528007fd0 tx=0x7fa528007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.422+0000 7fa52f400640 1 -- 172.21.15.114:0/4153931019 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa52001c360 con 0x7fa5380b3f70 2024-03-06T07:34:18.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.616+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa53806ef30 con 0x7fa5380b3f70 2024-03-06T07:34:18.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.616+0000 7fa52f400640 1 -- 172.21.15.114:0/4153931019 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa5200594e0 con 0x7fa5380b3f70 2024-03-06T07:34:18.618 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:34:18.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa51406b4d0 msgr2=0x7fa51406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa51406b4d0 0x7fa51406d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa528007fd0 tx=0x7fa528007960 comp rx=0 tx=0).stop 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5380b3f70 msgr2=0x7fa53815ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5380b3f70 0x7fa53815ebb0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fa52000d7d0 tx=0x7fa52000dc90 comp rx=0 tx=0).stop 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 shutdown_connections 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa53815fc10 0x7fa538151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa51406b4d0 0x7fa51406d990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5380b3f70 0x7fa53815ebb0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.618+0000 7fa53d400640 1 --2- 172.21.15.114:0/4153931019 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa538070ff0 0x7fa53815e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.619+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 >> 172.21.15.114:0/4153931019 conn(0x7fa5380b1940 msgr2=0x7fa5380af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.619+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 shutdown_connections 2024-03-06T07:34:18.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.619+0000 7fa53d400640 1 -- 172.21.15.114:0/4153931019 wait complete. 2024-03-06T07:34:18.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.839+0000 7f935be00640 1 Processor -- start 2024-03-06T07:34:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.840+0000 7f935be00640 1 -- start start 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f935c15e130 0x7f935c0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0cfaf0 0x7f935c0d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f935c0d23e0 0x7f935c0c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0cf120 con 0x7f935c15e130 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0d13e0 con 0x7f935c0d23e0 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0cead0 con 0x7f935c0cfaf0 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f935c15e130 0x7f935c0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f935c15e130 0x7f935c0cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38446/0 (socket says 172.21.15.114:38446) 2024-03-06T07:34:18.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.841+0000 7f935aa00640 1 -- 172.21.15.114:0/433388371 learned_addr learned my addr 172.21.15.114:0/433388371 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.842+0000 7f935aa00640 1 -- 172.21.15.114:0/433388371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0cfaf0 msgr2=0x7f935c0d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.842+0000 7f935aa00640 1 --2- 172.21.15.114:0/433388371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0cfaf0 0x7f935c0d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.842+0000 7f935aa00640 1 -- 172.21.15.114:0/433388371 >> v2:172.21.15.177:3300/0 conn(0x7f935c0d23e0 msgr2=0x7f935c0c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.842+0000 7f935aa00640 1 --2- 172.21.15.114:0/433388371 >> v2:172.21.15.177:3300/0 conn(0x7f935c0d23e0 0x7f935c0c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.842+0000 7f935aa00640 1 -- 172.21.15.114:0/433388371 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0cedb0 con 0x7f935c15e130 2024-03-06T07:34:18.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.843+0000 7f935aa00640 1 --2- 172.21.15.114:0/433388371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c0cf5b0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f935c0d3460 tx=0x7f934c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6215d7cd07bff51d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.843+0000 7f935a000640 1 -- 172.21.15.114:0/433388371 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c02ca80 con 0x7f935c15e130 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.843+0000 7f935a000640 1 -- 172.21.15.114:0/433388371 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c02cc20 con 0x7f935c15e130 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.843+0000 7f935a000640 1 -- 172.21.15.114:0/433388371 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c0343f0 con 0x7f935c15e130 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 -- 172.21.15.114:0/433388371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 msgr2=0x7f935c0cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 --2- 172.21.15.114:0/433388371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c0cf5b0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f935c0d3460 tx=0x7f934c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 -- 172.21.15.114:0/433388371 shutdown_connections 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 --2- 172.21.15.114:0/433388371 >> v2:172.21.15.177:3300/0 conn(0x7f935c0d23e0 0x7f935c0c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 --2- 172.21.15.114:0/433388371 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0cfaf0 0x7f935c0d1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 --2- 172.21.15.114:0/433388371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c0cf5b0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 -- 172.21.15.114:0/433388371 >> 172.21.15.114:0/433388371 conn(0x7f935c09d070 msgr2=0x7f935c0d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 -- 172.21.15.114:0/433388371 shutdown_connections 2024-03-06T07:34:18.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.844+0000 7f935be00640 1 -- 172.21.15.114:0/433388371 wait complete. 2024-03-06T07:34:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.847+0000 7f935be00640 1 Processor -- start 2024-03-06T07:34:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.847+0000 7f935be00640 1 -- start start 2024-03-06T07:34:18.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.848+0000 7f935be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f935c0cfaf0 0x7f935c157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.848+0000 7f935be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0d23e0 0x7f935c15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.848+0000 7f935be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c15f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.848+0000 7f935be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c8030 con 0x7f935c15e130 2024-03-06T07:34:18.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.848+0000 7f935be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c7eb0 con 0x7f935c0cfaf0 2024-03-06T07:34:18.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.848+0000 7f935be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0c81b0 con 0x7f935c0d23e0 2024-03-06T07:34:18.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c15f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c15f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38450/0 (socket says 172.21.15.114:38450) 2024-03-06T07:34:18.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 -- 172.21.15.114:0/3088553135 learned_addr learned my addr 172.21.15.114:0/3088553135 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935aa00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f935c0cfaf0 0x7f935c157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 -- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0d23e0 msgr2=0x7f935c15ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0d23e0 0x7f935c15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 -- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f935c0cfaf0 msgr2=0x7f935c157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f935c0cfaf0 0x7f935c157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 -- 172.21.15.114:0/3088553135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0cb140 con 0x7f935c15e130 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935aa00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f935c0cfaf0 0x7f935c157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:18.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.849+0000 7f935b400640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c15f460 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f935400b440 tx=0x7f935400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.850+0000 7f9353400640 1 -- 172.21.15.114:0/3088553135 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354015020 con 0x7f935c15e130 2024-03-06T07:34:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.850+0000 7f9353400640 1 -- 172.21.15.114:0/3088553135 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9354011850 con 0x7f935c15e130 2024-03-06T07:34:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.850+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c0cb420 con 0x7f935c15e130 2024-03-06T07:34:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.850+0000 7f9353400640 1 -- 172.21.15.114:0/3088553135 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354011c40 con 0x7f935c15e130 2024-03-06T07:34:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.850+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c0cb9e0 con 0x7f935c15e130 2024-03-06T07:34:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.851+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c04f330 con 0x7f935c15e130 2024-03-06T07:34:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.852+0000 7f9353400640 1 -- 172.21.15.114:0/3088553135 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9354018020 con 0x7f935c15e130 2024-03-06T07:34:18.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.853+0000 7f9353400640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f933806b4d0 0x7f933806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.853+0000 7f9353400640 1 -- 172.21.15.114:0/3088553135 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f935408a230 con 0x7f935c15e130 2024-03-06T07:34:18.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.853+0000 7f935aa00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f933806b4d0 0x7f933806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.854+0000 7f935aa00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f933806b4d0 0x7f933806d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f934c035000 tx=0x7f934c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:18.857+0000 7f9353400640 1 -- 172.21.15.114:0/3088553135 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f935401f020 con 0x7f935c15e130 2024-03-06T07:34:19.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.097+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f935c09d690 con 0x7f935c15e130 2024-03-06T07:34:19.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.097+0000 7f9353400640 1 -- 172.21.15.114:0/3088553135 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93540172c0 con 0x7f935c15e130 2024-03-06T07:34:19.099 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:34:19.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f933806b4d0 msgr2=0x7f933806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f933806b4d0 0x7f933806d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f934c035000 tx=0x7f934c030000 comp rx=0 tx=0).stop 2024-03-06T07:34:19.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 msgr2=0x7f935c15f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c15f460 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f935400b440 tx=0x7f935400b900 comp rx=0 tx=0).stop 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 shutdown_connections 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f933806b4d0 0x7f933806d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f935c15e130 0x7f935c15f460 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f935c0d23e0 0x7f935c15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 --2- 172.21.15.114:0/3088553135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f935c0cfaf0 0x7f935c157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.100+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 >> 172.21.15.114:0/3088553135 conn(0x7f935c09d070 msgr2=0x7f935c06cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.101+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 shutdown_connections 2024-03-06T07:34:19.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.101+0000 7f935be00640 1 -- 172.21.15.114:0/3088553135 wait complete. 2024-03-06T07:34:19.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:34:19.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.321+0000 7f1a68e00640 1 Processor -- start 2024-03-06T07:34:19.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.321+0000 7f1a68e00640 1 -- start start 2024-03-06T07:34:19.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.322+0000 7f1a68e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415b070 0x7f1a6406e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a68e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a6415f120 0x7f1a640c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a68e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a640cc000 con 0x7f1a6415ac90 2024-03-06T07:34:19.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a68e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a640ce360 con 0x7f1a6415f120 2024-03-06T07:34:19.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a68e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640ce640 con 0x7f1a6415b070 2024-03-06T07:34:19.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a62800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a6415f120 0x7f1a640c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a62800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a6415f120 0x7f1a640c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38458/0 (socket says 172.21.15.114:38458) 2024-03-06T07:34:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a62800640 1 -- 172.21.15.114:0/62026776 learned_addr learned my addr 172.21.15.114:0/62026776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a61400640 1 --2- 172.21.15.114:0/62026776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415b070 0x7f1a6406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a62800640 1 -- 172.21.15.114:0/62026776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415b070 msgr2=0x7f1a6406e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a62800640 1 --2- 172.21.15.114:0/62026776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415b070 0x7f1a6406e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a62800640 1 -- 172.21.15.114:0/62026776 >> v1:172.21.15.177:6789/0 conn(0x7f1a6415ac90 legacy=0x7f1a6415ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.323+0000 7f1a62800640 1 -- 172.21.15.114:0/62026776 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a640ce920 con 0x7f1a6415f120 2024-03-06T07:34:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.325+0000 7f1a62800640 1 --2- 172.21.15.114:0/62026776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415f120 0x7f1a640c4a10 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f1a640cd500 tx=0x7f1a5c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b0494ad076f7f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.325+0000 7f1a60a00640 1 -- 172.21.15.114:0/62026776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c03b070 con 0x7f1a6415f120 2024-03-06T07:34:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.325+0000 7f1a60a00640 1 -- 172.21.15.114:0/62026776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c02edb0 con 0x7f1a6415f120 2024-03-06T07:34:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.325+0000 7f1a60a00640 1 -- 172.21.15.114:0/62026776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c036660 con 0x7f1a6415f120 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 -- 172.21.15.114:0/62026776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415f120 msgr2=0x7f1a640c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 --2- 172.21.15.114:0/62026776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415f120 0x7f1a640c4a10 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f1a640cd500 tx=0x7f1a5c02e000 comp rx=0 tx=0).stop 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 -- 172.21.15.114:0/62026776 shutdown_connections 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 --2- 172.21.15.114:0/62026776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415f120 0x7f1a640c4a10 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 --2- 172.21.15.114:0/62026776 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415b070 0x7f1a6406e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 -- 172.21.15.114:0/62026776 >> 172.21.15.114:0/62026776 conn(0x7f1a640cc4f0 msgr2=0x7f1a640cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 -- 172.21.15.114:0/62026776 shutdown_connections 2024-03-06T07:34:19.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.326+0000 7f1a68e00640 1 -- 172.21.15.114:0/62026776 wait complete. 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.327+0000 7f1a68e00640 1 Processor -- start 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.328+0000 7f1a68e00640 1 -- start start 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.328+0000 7f1a68e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6415ac90 0x7f1a640c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.328+0000 7f1a68e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415b070 0x7f1a641f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a68e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415f120 0x7f1a640723f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a68e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6415f5a0 con 0x7f1a6415b070 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a68e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640ceee0 con 0x7f1a6415ac90 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a68e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6415f720 con 0x7f1a6415f120 2024-03-06T07:34:19.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415b070 0x7f1a641f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415b070 0x7f1a641f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38468/0 (socket says 172.21.15.114:38468) 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 -- 172.21.15.114:0/1139384437 learned_addr learned my addr 172.21.15.114:0/1139384437 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a62800640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415f120 0x7f1a640723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6415ac90 0x7f1a640c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 -- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415f120 msgr2=0x7f1a640723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415f120 0x7f1a640723f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 -- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6415ac90 msgr2=0x7f1a640c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6415ac90 0x7f1a640c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.329+0000 7f1a61400640 1 -- 172.21.15.114:0/1139384437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a641f4c60 con 0x7f1a6415b070 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.330+0000 7f1a61400640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415b070 0x7f1a641f4720 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d830 tx=0x7f1a4c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.330+0000 7f1a5a000640 1 -- 172.21.15.114:0/1139384437 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00b6c0 con 0x7f1a6415b070 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.330+0000 7f1a5a000640 1 -- 172.21.15.114:0/1139384437 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c019070 con 0x7f1a6415b070 2024-03-06T07:34:19.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.330+0000 7f1a5a000640 1 -- 172.21.15.114:0/1139384437 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00ba80 con 0x7f1a6415b070 2024-03-06T07:34:19.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.330+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a640ade20 con 0x7f1a6415b070 2024-03-06T07:34:19.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.330+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a640ae3e0 con 0x7f1a6415b070 2024-03-06T07:34:19.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.331+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6404f330 con 0x7f1a6415b070 2024-03-06T07:34:19.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.332+0000 7f1a5a000640 1 -- 172.21.15.114:0/1139384437 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a4c01e460 con 0x7f1a6415b070 2024-03-06T07:34:19.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.333+0000 7f1a5a000640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a4006b4d0 0x7f1a4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.333+0000 7f1a5a000640 1 -- 172.21.15.114:0/1139384437 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a4c08a8c0 con 0x7f1a6415b070 2024-03-06T07:34:19.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.333+0000 7f1a61e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a4006b4d0 0x7f1a4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.334+0000 7f1a61e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a4006b4d0 0x7f1a4006d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1a54005520 tx=0x7f1a54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.337+0000 7f1a5a000640 1 -- 172.21.15.114:0/1139384437 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a641f4c60 con 0x7f1a6415b070 2024-03-06T07:34:19.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.548+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1a640cd1c0 con 0x7f1a6415b070 2024-03-06T07:34:19.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.548+0000 7f1a5a000640 1 -- 172.21.15.114:0/1139384437 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v108) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1a4c059730 con 0x7f1a6415b070 2024-03-06T07:34:19.550 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:34:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a4006b4d0 msgr2=0x7f1a4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a4006b4d0 0x7f1a4006d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1a54005520 tx=0x7f1a54009040 comp rx=0 tx=0).stop 2024-03-06T07:34:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415b070 msgr2=0x7f1a641f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415b070 0x7f1a641f4720 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00d830 tx=0x7f1a4c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 shutdown_connections 2024-03-06T07:34:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a6415f120 0x7f1a640723f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1a4006b4d0 0x7f1a4006d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a6415b070 0x7f1a641f4720 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 --2- 172.21.15.114:0/1139384437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a6415ac90 0x7f1a640c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 >> 172.21.15.114:0/1139384437 conn(0x7f1a640cc4f0 msgr2=0x7f1a6406dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.551+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 shutdown_connections 2024-03-06T07:34:19.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.552+0000 7f1a68e00640 1 -- 172.21.15.114:0/1139384437 wait complete. 2024-03-06T07:34:19.570 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:34:19.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:19.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:19.704 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:34:19.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:19.704 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:34:19.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:19.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.930+0000 7f7bd3e00640 1 Processor -- start 2024-03-06T07:34:19.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.930+0000 7f7bd3e00640 1 -- start start 2024-03-06T07:34:19.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd3e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7bcc151660 0x7f7bcc071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7bcc0cf430 0x7f7bcc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc071a60 con 0x7f7bcc06d920 2024-03-06T07:34:19.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd3e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bcc0b3ee0 con 0x7f7bcc0cf430 2024-03-06T07:34:19.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd3e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bcc0b35b0 con 0x7f7bcc151660 2024-03-06T07:34:19.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7bcc0cf430 0x7f7bcc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd1e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7bcc06d920 0x7f7bcc15dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37776/0 (socket says 172.21.15.114:37776) 2024-03-06T07:34:19.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd1e00640 1 -- 172.21.15.114:0/646858302 learned_addr learned my addr 172.21.15.114:0/646858302 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd1400640 1 --2- 172.21.15.114:0/646858302 >> v2:172.21.15.177:3300/0 conn(0x7f7bcc151660 0x7f7bcc071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd0a00640 1 -- 172.21.15.114:0/646858302 >> v2:172.21.15.177:3300/0 conn(0x7f7bcc151660 msgr2=0x7f7bcc071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.933+0000 7f7bd0a00640 1 --2- 172.21.15.114:0/646858302 >> v2:172.21.15.177:3300/0 conn(0x7f7bcc151660 0x7f7bcc071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.934+0000 7f7bd0a00640 1 -- 172.21.15.114:0/646858302 >> v1:172.21.15.114:6789/0 conn(0x7f7bcc06d920 legacy=0x7f7bcc15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.934+0000 7f7bd0a00640 1 -- 172.21.15.114:0/646858302 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0b3890 con 0x7f7bcc0cf430 2024-03-06T07:34:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.934+0000 7f7bcbe00640 1 -- 172.21.15.114:0/646858302 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131738305 0 0) 0x7f7bcc071a60 con 0x7f7bcc06d920 2024-03-06T07:34:19.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.935+0000 7f7bd0a00640 1 --2- 172.21.15.114:0/646858302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc0cf430 0x7f7bcc06d3e0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc072fc0 tx=0x7f7bb802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce29a7e668f8e017 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bcbe00640 1 -- 172.21.15.114:0/646858302 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8038070 con 0x7f7bcc0cf430 2024-03-06T07:34:19.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bcbe00640 1 -- 172.21.15.114:0/646858302 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb8031710 con 0x7f7bcc0cf430 2024-03-06T07:34:19.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bcbe00640 1 -- 172.21.15.114:0/646858302 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8031ae0 con 0x7f7bcc0cf430 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 -- 172.21.15.114:0/646858302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc0cf430 msgr2=0x7f7bcc06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/646858302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc0cf430 0x7f7bcc06d3e0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc072fc0 tx=0x7f7bb802b7b0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 -- 172.21.15.114:0/646858302 shutdown_connections 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/646858302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc0cf430 0x7f7bcc06d3e0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/646858302 >> v2:172.21.15.177:3300/0 conn(0x7f7bcc151660 0x7f7bcc071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 -- 172.21.15.114:0/646858302 >> 172.21.15.114:0/646858302 conn(0x7f7bcc0b1730 msgr2=0x7f7bcc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 -- 172.21.15.114:0/646858302 shutdown_connections 2024-03-06T07:34:19.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.936+0000 7f7bd3e00640 1 -- 172.21.15.114:0/646858302 wait complete. 2024-03-06T07:34:19.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.938+0000 7f7bd3e00640 1 Processor -- start 2024-03-06T07:34:19.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.938+0000 7f7bd3e00640 1 -- start start 2024-03-06T07:34:19.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bcc06d920 0x7f7bcc149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bcc0cf430 0x7f7bcc15c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc151660 0x7f7bcc152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc15e700 con 0x7f7bcc151660 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc071d30 con 0x7f7bcc06d920 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc15e880 con 0x7f7bcc0cf430 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc151660 0x7f7bcc152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc151660 0x7f7bcc152a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38494/0 (socket says 172.21.15.114:38494) 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.939+0000 7f7bd1e00640 1 -- 172.21.15.114:0/351350185 learned_addr learned my addr 172.21.15.114:0/351350185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd0a00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bcc0cf430 0x7f7bcc15c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd1e00640 1 -- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bcc0cf430 msgr2=0x7f7bcc15c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd1400640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bcc06d920 0x7f7bcc149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd1e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bcc0cf430 0x7f7bcc15c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd1e00640 1 -- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bcc06d920 msgr2=0x7f7bcc149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd1e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bcc06d920 0x7f7bcc149e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd1e00640 1 -- 172.21.15.114:0/351350185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc167540 con 0x7f7bcc151660 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd1e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc151660 0x7f7bcc152a90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400c7b0 tx=0x7f7bc400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bc9600640 1 -- 172.21.15.114:0/351350185 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4018070 con 0x7f7bcc151660 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bc9600640 1 -- 172.21.15.114:0/351350185 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc400a640 con 0x7f7bcc151660 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bcc153cc0 con 0x7f7bcc151660 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bc9600640 1 -- 172.21.15.114:0/351350185 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc400ce60 con 0x7f7bcc151660 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.940+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bcc14a3b0 con 0x7f7bcc151660 2024-03-06T07:34:19.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.942+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bcc04f330 con 0x7f7bcc151660 2024-03-06T07:34:19.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.942+0000 7f7bc9600640 1 -- 172.21.15.114:0/351350185 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bc401b070 con 0x7f7bcc151660 2024-03-06T07:34:19.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.944+0000 7f7bc9600640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bac06b4d0 0x7f7bac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:19.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.944+0000 7f7bc9600640 1 -- 172.21.15.114:0/351350185 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bc4089730 con 0x7f7bcc151660 2024-03-06T07:34:19.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.944+0000 7f7bd1400640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bac06b4d0 0x7f7bac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:19.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.945+0000 7f7bd1400640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bac06b4d0 0x7f7bac06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0006fd0 tx=0x7f7bc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:19.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:19.949+0000 7f7bc9600640 1 -- 172.21.15.114:0/351350185 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bc4058510 con 0x7f7bcc151660 2024-03-06T07:34:20.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.191+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bcc1678a0 con 0x7f7bcc151660 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.192+0000 7f7bc9600640 1 -- 172.21.15.114:0/351350185 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f7bc4057e10 con 0x7f7bcc151660 2024-03-06T07:34:20.194 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":79,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5983,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3305544049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3305544049},{"type":"v1","addr":"172.21.15.114:6836","nonce":3305544049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6001,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6834/3949834778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3949834778},{"type":"v1","addr":"172.21.15.177:6834","nonce":3949834778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6004,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2151340900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6833","nonce":2151340900},{"type":"v1","addr":"172.21.15.177:6835","nonce":2151340900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:09.452793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7674},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7674":{"gid":7674,"name":"a","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/860883596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":860883596},{"type":"v1","addr":"172.21.15.114:6837","nonce":860883596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7674,"qdb_cluster":[7674]},"id":8}]} 2024-03-06T07:34:20.195 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 79 2024-03-06T07:34:20.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bac06b4d0 msgr2=0x7f7bac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bac06b4d0 0x7f7bac06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0006fd0 tx=0x7f7bc0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc151660 msgr2=0x7f7bcc152a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc151660 0x7f7bcc152a90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400c7b0 tx=0x7f7bc400cc70 comp rx=0 tx=0).stop 2024-03-06T07:34:20.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 shutdown_connections 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bac06b4d0 0x7f7bac06d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc151660 0x7f7bcc152a90 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bcc0cf430 0x7f7bcc15c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.195+0000 7f7bd3e00640 1 --2- 172.21.15.114:0/351350185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bcc06d920 0x7f7bcc149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.196+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 >> 172.21.15.114:0/351350185 conn(0x7f7bcc0b1730 msgr2=0x7f7bcc15c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.196+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 shutdown_connections 2024-03-06T07:34:20.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.196+0000 7f7bd3e00640 1 -- 172.21.15.114:0/351350185 wait complete. 2024-03-06T07:34:20.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:34:20.455 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:34:20.455 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:20.456 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:20.456 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:20.457 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:20.457 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:20.457 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:34:20.457 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:20.457 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:34:20.458 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:20.475 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:20.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:34:20.477 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:20.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:34:20.478 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:34:20.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:34:20.479 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:34:20.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:34:20.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.704+0000 7f88b8600640 1 Processor -- start 2024-03-06T07:34:20.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.704+0000 7f88b8600640 1 -- start start 2024-03-06T07:34:20.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.705+0000 7f73f2400640 1 Processor -- start 2024-03-06T07:34:20.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.705+0000 7f73f2400640 1 -- start start 2024-03-06T07:34:20.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.706+0000 7f73f2400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73ec0cec10 0x7f73ec161d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.706+0000 7f73f2400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f73ec162270 0x7f73ec0d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.706+0000 7f88b8600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f88b015b070 0x7f88b006e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f88b8600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00cc000 con 0x7f88b015f120 2024-03-06T07:34:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f88b8600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00ce360 con 0x7f88b015ac90 2024-03-06T07:34:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f88b8600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b00ce640 con 0x7f88b015b070 2024-03-06T07:34:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.706+0000 7f73f2400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0cf2b0 con 0x7f73ec15e360 2024-03-06T07:34:20.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.706+0000 7f73f2400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ec1626c0 con 0x7f73ec0cec10 2024-03-06T07:34:20.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.706+0000 7f73f2400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ec0cf590 con 0x7f73ec162270 2024-03-06T07:34:20.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f88b5200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f88b015b070 0x7f88b006e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f88b5200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f88b015b070 0x7f88b006e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38514/0 (socket says 172.21.15.114:38514) 2024-03-06T07:34:20.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f73eaa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73ec0cec10 0x7f73ec161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f73eaa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f73ec0cec10 0x7f73ec161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38510/0 (socket says 172.21.15.114:38510) 2024-03-06T07:34:20.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f73eaa00640 1 -- 172.21.15.114:0/1622051408 learned_addr learned my addr 172.21.15.114:0/1622051408 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:20.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f88b5200640 1 -- 172.21.15.114:0/1341278252 learned_addr learned my addr 172.21.15.114:0/1341278252 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:20.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.707+0000 7f73ebe00640 1 --2- 172.21.15.114:0/1622051408 >> v2:172.21.15.177:3300/0 conn(0x7f73ec162270 0x7f73ec0d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f88b5200640 1 -- 172.21.15.114:0/1341278252 >> v1:172.21.15.177:6789/0 conn(0x7f88b015ac90 legacy=0x7f88b015ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f88b5200640 1 -- 172.21.15.114:0/1341278252 >> v1:172.21.15.114:6789/0 conn(0x7f88b015f120 legacy=0x7f88b00c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f73eaa00640 1 -- 172.21.15.114:0/1622051408 >> v2:172.21.15.177:3300/0 conn(0x7f73ec162270 msgr2=0x7f73ec0d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f73eaa00640 1 --2- 172.21.15.114:0/1622051408 >> v2:172.21.15.177:3300/0 conn(0x7f73ec162270 0x7f73ec0d1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f73eaa00640 1 -- 172.21.15.114:0/1622051408 >> v1:172.21.15.177:6789/0 conn(0x7f73ec15e360 legacy=0x7f73ec0d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f73eaa00640 1 -- 172.21.15.114:0/1622051408 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec0cf870 con 0x7f73ec0cec10 2024-03-06T07:34:20.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f88b5200640 1 -- 172.21.15.114:0/1341278252 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b00ce920 con 0x7f88b015b070 2024-03-06T07:34:20.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.708+0000 7f88afe00640 1 -- 172.21.15.114:0/1341278252 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217038600 0 0) 0x7f88b00cc000 con 0x7f88b015f120 2024-03-06T07:34:20.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.709+0000 7f73eaa00640 1 --2- 172.21.15.114:0/1622051408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec161d30 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0d0650 tx=0x7f73d802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4732d96299f18d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.709+0000 7f73ea000640 1 -- 172.21.15.114:0/1622051408 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d802f850 con 0x7f73ec0cec10 2024-03-06T07:34:20.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.709+0000 7f73ea000640 1 -- 172.21.15.114:0/1622051408 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d802f9f0 con 0x7f73ec0cec10 2024-03-06T07:34:20.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.709+0000 7f73ea000640 1 -- 172.21.15.114:0/1622051408 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d802fdc0 con 0x7f73ec0cec10 2024-03-06T07:34:20.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f88b5200640 1 --2- 172.21.15.114:0/1341278252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015b070 0x7f88b006e580 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f889c009870 tx=0x7f889c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d7cab7931d28a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f88afe00640 1 -- 172.21.15.114:0/1341278252 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c034070 con 0x7f88b015b070 2024-03-06T07:34:20.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f88afe00640 1 -- 172.21.15.114:0/1341278252 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c030860 con 0x7f88b015b070 2024-03-06T07:34:20.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f88afe00640 1 -- 172.21.15.114:0/1341278252 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c030c30 con 0x7f88b015b070 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f73f2400640 1 -- 172.21.15.114:0/1622051408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 msgr2=0x7f73ec161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f73f2400640 1 --2- 172.21.15.114:0/1622051408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec161d30 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0d0650 tx=0x7f73d802ba30 comp rx=0 tx=0).stop 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f73f2400640 1 -- 172.21.15.114:0/1622051408 shutdown_connections 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f73f2400640 1 --2- 172.21.15.114:0/1622051408 >> v2:172.21.15.177:3300/0 conn(0x7f73ec162270 0x7f73ec0d1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f73f2400640 1 --2- 172.21.15.114:0/1622051408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec161d30 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f73f2400640 1 -- 172.21.15.114:0/1622051408 >> 172.21.15.114:0/1622051408 conn(0x7f73ec0d3f50 msgr2=0x7f73ec0d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.710+0000 7f73f2400640 1 -- 172.21.15.114:0/1622051408 shutdown_connections 2024-03-06T07:34:20.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f73f2400640 1 -- 172.21.15.114:0/1622051408 wait complete. 2024-03-06T07:34:20.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f88b8600640 1 -- 172.21.15.114:0/1341278252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015b070 msgr2=0x7f88b006e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f88b8600640 1 --2- 172.21.15.114:0/1341278252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015b070 0x7f88b006e580 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f889c009870 tx=0x7f889c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f88b8600640 1 -- 172.21.15.114:0/1341278252 shutdown_connections 2024-03-06T07:34:20.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f88b8600640 1 --2- 172.21.15.114:0/1341278252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015b070 0x7f88b006e580 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f88b8600640 1 -- 172.21.15.114:0/1341278252 >> 172.21.15.114:0/1341278252 conn(0x7f88b00cc4f0 msgr2=0x7f88b00cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f88b8600640 1 -- 172.21.15.114:0/1341278252 shutdown_connections 2024-03-06T07:34:20.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7f88b8600640 1 -- 172.21.15.114:0/1341278252 wait complete. 2024-03-06T07:34:20.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.711+0000 7fc617a00640 1 Processor -- start 2024-03-06T07:34:20.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.712+0000 7fc617a00640 1 -- start start 2024-03-06T07:34:20.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.712+0000 7f73f2400640 1 Processor -- start 2024-03-06T07:34:20.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.712+0000 7f73f2400640 1 -- start start 2024-03-06T07:34:20.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.713+0000 7f73f2400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.713+0000 7f73f2400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ec15e360 0x7f73ec0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.713+0000 7f73f2400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73ec162270 0x7f73ec15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.713+0000 7f73f2400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec0d21d0 con 0x7f73ec0cec10 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.713+0000 7f73f2400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec0cfe30 con 0x7f73ec15e360 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.713+0000 7f73f2400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec0d2350 con 0x7f73ec162270 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7f73eb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7f73eb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec150900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38518/0 (socket says 172.21.15.114:38518) 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7f73eb400640 1 -- 172.21.15.114:0/4205415942 learned_addr learned my addr 172.21.15.114:0/4205415942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:20.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7fc617a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7fc617a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc6100c5960 0x7fc610165b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7f88b8600640 1 Processor -- start 2024-03-06T07:34:20.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7fc617a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc610145b30 con 0x7fc6100c4930 2024-03-06T07:34:20.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7fc617a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6100c68c0 con 0x7fc6100c5960 2024-03-06T07:34:20.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7fc617a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc610142ce0 con 0x7fc6100c6460 2024-03-06T07:34:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7f73eaa00640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ec15e360 0x7f73ec0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7f73ebe00640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73ec162270 0x7f73ec15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.714+0000 7f88b8600640 1 -- start start 2024-03-06T07:34:20.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7f73eb400640 1 -- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73ec162270 msgr2=0x7f73ec15a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc615a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc6100c5960 0x7fc610165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc615000640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fc6100c4930 0x7fc6100c4d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:35180/0 (socket says 172.21.15.114:35180) 2024-03-06T07:34:20.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc615000640 1 -- 172.21.15.114:0/921020252 learned_addr learned my addr 172.21.15.114:0/921020252 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:20.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc60fe00640 1 --2- 172.21.15.114:0/921020252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7f73eb400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73ec162270 0x7f73ec15a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7f73eb400640 1 -- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ec15e360 msgr2=0x7f73ec0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7f73eb400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ec15e360 0x7f73ec0cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7f73eb400640 1 -- 172.21.15.114:0/4205415942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec0cc1a0 con 0x7f73ec0cec10 2024-03-06T07:34:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7f73eb400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec150900 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00c430 tx=0x7f73dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc615a00640 1 -- 172.21.15.114:0/921020252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 msgr2=0x7fc6100c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7f73e3400640 1 -- 172.21.15.114:0/4205415942 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc0052d0 con 0x7f73ec0cec10 2024-03-06T07:34:20.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc60f400640 1 -- 172.21.15.114:0/921020252 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373249987 0 0) 0x7fc604001110 con 0x7fc6100c4930 2024-03-06T07:34:20.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc615a00640 1 --2- 172.21.15.114:0/921020252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc615a00640 1 -- 172.21.15.114:0/921020252 >> v1:172.21.15.177:6789/0 conn(0x7fc6100c4930 legacy=0x7fc6100c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.715+0000 7fc615a00640 1 -- 172.21.15.114:0/921020252 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6100cbca0 con 0x7fc6100c5960 2024-03-06T07:34:20.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f73e3400640 1 -- 172.21.15.114:0/4205415942 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc005490 con 0x7f73ec0cec10 2024-03-06T07:34:20.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ec0cc480 con 0x7f73ec0cec10 2024-03-06T07:34:20.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f73e3400640 1 -- 172.21.15.114:0/4205415942 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc007260 con 0x7f73ec0cec10 2024-03-06T07:34:20.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ec0cca40 con 0x7f73ec0cec10 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f88b8600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015ac90 0x7f88b014b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f88b8600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88b015b070 0x7f88b01fba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f88b8600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88b015f120 0x7f88b01fbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f88b8600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b015f5b0 con 0x7f88b015ac90 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f88b8600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00ceee0 con 0x7f88b015f120 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.716+0000 7f88b8600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b015f730 con 0x7f88b015b070 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.717+0000 7f88b6600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88b015f120 0x7f88b01fbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.717+0000 7f88b6600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88b015f120 0x7f88b01fbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.114:45678/0 (socket says 172.21.15.114:45678) 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.717+0000 7f88b6600640 1 -- 172.21.15.114:0/407361921 learned_addr learned my addr 172.21.15.114:0/407361921 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:20.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.717+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ec04f330 con 0x7f73ec0cec10 2024-03-06T07:34:20.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.717+0000 7fc615a00640 1 --2- 172.21.15.114:0/921020252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc610165b30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc608009d90 tx=0x7fc60802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac1ebda37c20b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.717+0000 7f88b5200640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88b015b070 0x7f88b01fba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7fc60f400640 1 -- 172.21.15.114:0/921020252 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608039070 con 0x7fc6100c5960 2024-03-06T07:34:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7fc60f400640 1 -- 172.21.15.114:0/921020252 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60802cbc0 con 0x7fc6100c5960 2024-03-06T07:34:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7fc60f400640 1 -- 172.21.15.114:0/921020252 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608034440 con 0x7fc6100c5960 2024-03-06T07:34:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7f88b5c00640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015ac90 0x7f88b014b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7f88b6600640 1 -- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88b015b070 msgr2=0x7f88b01fba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7f88b6600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88b015b070 0x7f88b01fba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7f88b6600640 1 -- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015ac90 msgr2=0x7f88b014b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7f88b6600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015ac90 0x7f88b014b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.718+0000 7f88b6600640 1 -- 172.21.15.114:0/407361921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b014bdd0 con 0x7f88b015f120 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7f88b6600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88b015f120 0x7f88b01fbf70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c780 tx=0x7f88a800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7f88ad600640 1 -- 172.21.15.114:0/407361921 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8003b70 con 0x7f88b015f120 2024-03-06T07:34:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7f73e3400640 1 -- 172.21.15.114:0/4205415942 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73dc007400 con 0x7f73ec0cec10 2024-03-06T07:34:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7fc617a00640 1 -- 172.21.15.114:0/921020252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 msgr2=0x7fc610165b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7fc617a00640 1 --2- 172.21.15.114:0/921020252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc610165b30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc608009d90 tx=0x7fc60802be10 comp rx=0 tx=0).stop 2024-03-06T07:34:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7f88ad600640 1 -- 172.21.15.114:0/407361921 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a8003d10 con 0x7f88b015f120 2024-03-06T07:34:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7f88ad600640 1 -- 172.21.15.114:0/407361921 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8005290 con 0x7f88b015f120 2024-03-06T07:34:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.719+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b00b5230 con 0x7f88b015f120 2024-03-06T07:34:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b01583b0 con 0x7f88b015f120 2024-03-06T07:34:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7f7b9ba00640 1 Processor -- start 2024-03-06T07:34:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7fc617a00640 1 -- 172.21.15.114:0/921020252 shutdown_connections 2024-03-06T07:34:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7fc617a00640 1 --2- 172.21.15.114:0/921020252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc610165b30 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7fc617a00640 1 --2- 172.21.15.114:0/921020252 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7fc617a00640 1 -- 172.21.15.114:0/921020252 >> 172.21.15.114:0/921020252 conn(0x7fc6100cc4f0 msgr2=0x7fc6100cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7f73e3400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73cc06b520 0x7f73cc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7fc617a00640 1 -- 172.21.15.114:0/921020252 shutdown_connections 2024-03-06T07:34:20.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7fc617a00640 1 -- 172.21.15.114:0/921020252 wait complete. 2024-03-06T07:34:20.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.720+0000 7f73e3400640 1 -- 172.21.15.114:0/4205415942 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73dc089650 con 0x7f73ec0cec10 2024-03-06T07:34:20.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.721+0000 7f7b9ba00640 1 -- start start 2024-03-06T07:34:20.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.721+0000 7f73eaa00640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73cc06b520 0x7f73cc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.721+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b004f330 con 0x7f88b015f120 2024-03-06T07:34:20.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.722+0000 7f73eaa00640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73cc06b520 0x7f73cc06d9e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f73d8036000 tx=0x7f73d8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.722+0000 7f88ad600640 1 -- 172.21.15.114:0/407361921 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88a8005430 con 0x7f88b015f120 2024-03-06T07:34:20.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.722+0000 7fc617a00640 1 Processor -- start 2024-03-06T07:34:20.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.723+0000 7fc617a00640 1 -- start start 2024-03-06T07:34:20.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc617a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6100c4930 0x7fc61013c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc617a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc6100c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc617a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc617a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6100ca3b0 con 0x7fc6100c5960 2024-03-06T07:34:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc617a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6100ca0d0 con 0x7fc6100c4930 2024-03-06T07:34:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc617a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61006e600 con 0x7fc6100c6460 2024-03-06T07:34:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f88ad600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f7b9ba00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940d0590 0x7f7b94140510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f88ad600640 1 -- 172.21.15.114:0/407361921 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88a808a940 con 0x7f88b015f120 2024-03-06T07:34:20.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc60fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc6100c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc60fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc6100c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38544/0 (socket says 172.21.15.114:38544) 2024-03-06T07:34:20.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc60fe00640 1 -- 172.21.15.114:0/2549106190 learned_addr learned my addr 172.21.15.114:0/2549106190 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:20.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7fc615000640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6100c4930 0x7fc61013c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f7b9ba00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b940c5e90 con 0x7f7b940b1650 2024-03-06T07:34:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f7b9ba00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9406db20 con 0x7f7b940cdd30 2024-03-06T07:34:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f7b9ba00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9406d690 con 0x7f7b940d0590 2024-03-06T07:34:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f7b93e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7b940b1650 0x7f7b940cd620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37796/0 (socket says 172.21.15.114:37796) 2024-03-06T07:34:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.724+0000 7f7b93e00640 1 -- 172.21.15.114:0/470448903 learned_addr learned my addr 172.21.15.114:0/470448903 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7fc615a00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7f88b5c00640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7fc60fe00640 1 -- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 msgr2=0x7fc6100cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7fc60fe00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7fc60fe00640 1 -- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6100c4930 msgr2=0x7fc61013c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7fc60fe00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6100c4930 0x7fc61013c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7fc60fe00640 1 -- 172.21.15.114:0/2549106190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61013dfe0 con 0x7fc6100c5960 2024-03-06T07:34:20.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7f7b99000640 1 --2- 172.21.15.114:0/470448903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940d0590 0x7f7b94140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.725+0000 7fc60fe00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc6100c8670 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00e9b0 tx=0x7fc5fc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.726+0000 7f7b99000640 1 -- 172.21.15.114:0/470448903 >> v1:172.21.15.177:6789/0 conn(0x7f7b940cdd30 legacy=0x7f7b94165060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.726+0000 7f88b5c00640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f889006b4d0 0x7f889006d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f88b01fb990 tx=0x7f88a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.726+0000 7f7b99000640 1 -- 172.21.15.114:0/470448903 >> v1:172.21.15.114:6789/0 conn(0x7f7b940b1650 legacy=0x7f7b940cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.726+0000 7f7b99000640 1 -- 172.21.15.114:0/470448903 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9406cd60 con 0x7f7b940d0590 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.727+0000 7fc60cc00640 1 -- 172.21.15.114:0/2549106190 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc017070 con 0x7fc6100c5960 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.727+0000 7fc60cc00640 1 -- 172.21.15.114:0/2549106190 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc00c9a0 con 0x7fc6100c5960 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.727+0000 7fc60cc00640 1 -- 172.21.15.114:0/2549106190 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc00cd90 con 0x7fc6100c5960 2024-03-06T07:34:20.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.727+0000 7f7b99000640 1 --2- 172.21.15.114:0/470448903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940d0590 0x7f7b94140510 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b880097d0 tx=0x7f7b8802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355d9c635416d94d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.727+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61013c9f0 con 0x7fc6100c5960 2024-03-06T07:34:20.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.728+0000 7f73e3400640 1 -- 172.21.15.114:0/4205415942 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73dc0585b0 con 0x7f73ec0cec10 2024-03-06T07:34:20.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.728+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61013e560 con 0x7fc6100c5960 2024-03-06T07:34:20.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.728+0000 7f7b93400640 1 -- 172.21.15.114:0/470448903 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88034070 con 0x7f7b940d0590 2024-03-06T07:34:20.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.728+0000 7f7b93400640 1 -- 172.21.15.114:0/470448903 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88030800 con 0x7f7b940d0590 2024-03-06T07:34:20.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.728+0000 7f7b93400640 1 -- 172.21.15.114:0/470448903 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88030bd0 con 0x7f7b940d0590 2024-03-06T07:34:20.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.729+0000 7f7b9ba00640 1 -- 172.21.15.114:0/470448903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940d0590 msgr2=0x7f7b94140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.729+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/470448903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940d0590 0x7f7b94140510 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b880097d0 tx=0x7f7b8802b940 comp rx=0 tx=0).stop 2024-03-06T07:34:20.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.729+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6100cb530 con 0x7fc6100c5960 2024-03-06T07:34:20.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.729+0000 7f7b9ba00640 1 -- 172.21.15.114:0/470448903 shutdown_connections 2024-03-06T07:34:20.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.729+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/470448903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940d0590 0x7f7b94140510 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b880097d0 tx=0x7f7b8802b940 comp rx=0 tx=0).stop 2024-03-06T07:34:20.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.729+0000 7f7b9ba00640 1 -- 172.21.15.114:0/470448903 >> 172.21.15.114:0/470448903 conn(0x7f7b940cfa70 msgr2=0x7f7b940cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.730+0000 7f7b9ba00640 1 -- 172.21.15.114:0/470448903 shutdown_connections 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.730+0000 7f7b9ba00640 1 -- 172.21.15.114:0/470448903 wait complete. 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.730+0000 7fc60cc00640 1 -- 172.21.15.114:0/2549106190 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5fc01a070 con 0x7fc6100c5960 2024-03-06T07:34:20.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.731+0000 7fc60cc00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f006b520 0x7fc5f006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.731+0000 7fc60cc00640 1 -- 172.21.15.114:0/2549106190 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5fc08a1b0 con 0x7fc6100c5960 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.732+0000 7fc615000640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f006b520 0x7fc5f006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.732+0000 7f7b9ba00640 1 Processor -- start 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.732+0000 7f7b9ba00640 1 -- start start 2024-03-06T07:34:20.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.733+0000 7fc615000640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f006b520 0x7fc5f006d9e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc604007fd0 tx=0x7fc604007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.733+0000 7f7b9ba00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b940b1650 0x7f7b940727a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.733+0000 7f7b9ba00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940cdd30 0x7f7b9413b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.733+0000 7f7b9ba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b94071200 0x7f7b941e1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.733+0000 7f7b9ba00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9406d320 con 0x7f7b940b1650 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.733+0000 7f7b9ba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940ce390 con 0x7f7b94071200 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.733+0000 7f7b9ba00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940b1ad0 con 0x7f7b940cdd30 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b940b1650 0x7f7b940727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b940b1650 0x7f7b940727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38550/0 (socket says 172.21.15.114:38550) 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 -- 172.21.15.114:0/3518389275 learned_addr learned my addr 172.21.15.114:0/3518389275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99a00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b94071200 0x7f7b941e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 -- 172.21.15.114:0/3518389275 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940cdd30 msgr2=0x7f7b9413b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940cdd30 0x7f7b9413b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 -- 172.21.15.114:0/3518389275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b94071200 msgr2=0x7f7b941e1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b94071200 0x7f7b941e1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.734+0000 7f7b99000640 1 -- 172.21.15.114:0/3518389275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9413ba00 con 0x7f7b940b1650 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.735+0000 7f7b99000640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b940b1650 0x7f7b940727a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f7b880098f0 tx=0x7f7b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.736+0000 7f7b90c00640 1 -- 172.21.15.114:0/3518389275 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88034040 con 0x7f7b940b1650 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.736+0000 7f7b90c00640 1 -- 172.21.15.114:0/3518389275 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88039c90 con 0x7f7b940b1650 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.736+0000 7f7b90c00640 1 -- 172.21.15.114:0/3518389275 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88042610 con 0x7f7b940b1650 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.736+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b94071700 con 0x7f7b940b1650 2024-03-06T07:34:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.736+0000 7fc60cc00640 1 -- 172.21.15.114:0/2549106190 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6100cb530 con 0x7fc6100c5960 2024-03-06T07:34:20.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.737+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b94071cc0 con 0x7f7b940b1650 2024-03-06T07:34:20.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.738+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9404f330 con 0x7f7b940b1650 2024-03-06T07:34:20.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.738+0000 7f88ad600640 1 -- 172.21.15.114:0/407361921 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88a8039030 con 0x7f88b015f120 2024-03-06T07:34:20.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.739+0000 7f7b90c00640 1 -- 172.21.15.114:0/3518389275 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b88037070 con 0x7f7b940b1650 2024-03-06T07:34:20.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.739+0000 7f7b90c00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.739+0000 7f7b90c00640 1 -- 172.21.15.114:0/3518389275 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b880ac370 con 0x7f7b940b1650 2024-03-06T07:34:20.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.740+0000 7f7b93e00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.741+0000 7f7b93e00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7b7406b500 0x7f7b7406d9c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7b9406c7a0 tx=0x7f7b80007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:20.744+0000 7f7b90c00640 1 -- 172.21.15.114:0/3518389275 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b9413ba00 con 0x7f7b940b1650 2024-03-06T07:34:21.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.017+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f73ec08d9b0 con 0x7f73ec0cec10 2024-03-06T07:34:21.019 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:34:21.018+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:21.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.020+0000 7f73e3400640 1 -- 172.21.15.114:0/4205415942 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7674 v80) v1 ==== 95+0+0 (secure 0 0 0) 0x7f73dc057eb0 con 0x7f73ec0cec10 2024-03-06T07:34:21.022 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 7674 2024-03-06T07:34:21.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73cc06b520 msgr2=0x7f73cc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73cc06b520 0x7f73cc06d9e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f73d8036000 tx=0x7f73d8008030 comp rx=0 tx=0).stop 2024-03-06T07:34:21.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 msgr2=0x7f73ec150900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec150900 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00c430 tx=0x7f73dc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 shutdown_connections 2024-03-06T07:34:21.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f73ec162270 0x7f73ec15a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73ec15e360 0x7f73ec0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f73cc06b520 0x7f73cc06d9e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 --2- 172.21.15.114:0/4205415942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73ec0cec10 0x7f73ec150900 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.023+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 >> 172.21.15.114:0/4205415942 conn(0x7f73ec0d3f50 msgr2=0x7f73ec1510b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.024+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 shutdown_connections 2024-03-06T07:34:21.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.024+0000 7f73f2400640 1 -- 172.21.15.114:0/4205415942 wait complete. 2024-03-06T07:34:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.033+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fc6100cc250 con 0x7fc6100c5960 2024-03-06T07:34:21.045 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:34:21.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:34:21.049 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:34:21.049 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:21.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.053+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f7b9406e6e0 con 0x7f7b940b1650 2024-03-06T07:34:21.055 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:34:21.053+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:21.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.053+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f88b00cd1c0 con 0x7f88b015f120 2024-03-06T07:34:21.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.056+0000 7fc60cc00640 1 -- 172.21.15.114:0/2549106190 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6001 v82) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc5fc0589c0 con 0x7fc6100c5960 2024-03-06T07:34:21.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.056+0000 7f7b90c00640 1 -- 172.21.15.114:0/3518389275 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6004 v82) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7b88037360 con 0x7f7b940b1650 2024-03-06T07:34:21.058 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 6001 2024-03-06T07:34:21.058 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 6004 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7b7406b500 msgr2=0x7f7b7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7b7406b500 0x7f7b7406d9c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7b9406c7a0 tx=0x7f7b80007a40 comp rx=0 tx=0).stop 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b940b1650 msgr2=0x7f7b940727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b940b1650 0x7f7b940727a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f7b880098f0 tx=0x7f7b880046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 shutdown_connections 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b94071200 0x7f7b941e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7b940cdd30 0x7f7b9413b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 --2- 172.21.15.114:0/3518389275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b940b1650 0x7f7b940727a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 >> 172.21.15.114:0/3518389275 conn(0x7f7b940cfa70 msgr2=0x7f7b9413c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f006b520 msgr2=0x7fc5f006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7fc617a00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f006b520 0x7fc5f006d9e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc604007fd0 tx=0x7fc604007de0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 msgr2=0x7fc6100c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:21.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7fc617a00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc6100c8670 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00e9b0 tx=0x7fc5fc00ee70 comp rx=0 tx=0).stop 2024-03-06T07:34:21.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 shutdown_connections 2024-03-06T07:34:21.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.059+0000 7f7b9ba00640 1 -- 172.21.15.114:0/3518389275 wait complete. 2024-03-06T07:34:21.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 shutdown_connections 2024-03-06T07:34:21.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc6100c6460 0x7fc6100cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5f006b520 0x7fc5f006d9e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6100c5960 0x7fc6100c8670 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 --2- 172.21.15.114:0/2549106190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6100c4930 0x7fc61013c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:21.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 >> 172.21.15.114:0/2549106190 conn(0x7fc6100cc4f0 msgr2=0x7fc6100ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:21.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 shutdown_connections 2024-03-06T07:34:21.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:21.060+0000 7fc617a00640 1 -- 172.21.15.114:0/2549106190 wait complete. 2024-03-06T07:34:21.081 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:34:21.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:34:21.081 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:34:21.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:34:21.088 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:34:21.089 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:21.094 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:34:21.094 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:21.195 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:34:21.245 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:34:21.250 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:34:22.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.066+0000 7f88ad600640 1 -- 172.21.15.114:0/407361921 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 5983 v83) v1 ==== 95+0+0 (secure 0 0 0) 0x7f88b014bdd0 con 0x7f88b015f120 2024-03-06T07:34:22.068 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 5983 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.070+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f889006b4d0 msgr2=0x7f889006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.070+0000 7f88b8600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f889006b4d0 0x7f889006d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f88b01fb990 tx=0x7f88a4009040 comp rx=0 tx=0).stop 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.070+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88b015f120 msgr2=0x7f88b01fbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.070+0000 7f88b8600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88b015f120 0x7f88b01fbf70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c780 tx=0x7f88a800cc40 comp rx=0 tx=0).stop 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 shutdown_connections 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88b015f120 0x7f88b01fbf70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f88b015b070 0x7f88b01fba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 --2- 172.21.15.114:0/407361921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b015ac90 0x7f88b014b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 >> 172.21.15.114:0/407361921 conn(0x7f88b00cc4f0 msgr2=0x7f88b0071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 shutdown_connections 2024-03-06T07:34:22.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.071+0000 7f88b8600640 1 -- 172.21.15.114:0/407361921 wait complete. 2024-03-06T07:34:22.096 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:34:22.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:34:22.100 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:34:22.100 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:22.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:22.242 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:34:22.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.267+0000 7fae2be00640 1 Processor -- start 2024-03-06T07:34:22.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.267+0000 7fae2be00640 1 -- start start 2024-03-06T07:34:22.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae2c15a940 0x7fae2c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fae2c0cf370 0x7fae2c06d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c0cc2a0 0x7fae2c0cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2c0b28e0 con 0x7fae2c15a940 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae2c0b2b30 con 0x7fae2c0cf370 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c0b2e10 con 0x7fae2c0cc2a0 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae2c15a940 0x7fae2c0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae2c15a940 0x7fae2c0b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38602/0 (socket says 172.21.15.114:38602) 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 -- 172.21.15.114:0/3166370140 learned_addr learned my addr 172.21.15.114:0/3166370140 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 -- 172.21.15.114:0/3166370140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c0cc2a0 msgr2=0x7fae2c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 --2- 172.21.15.114:0/3166370140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c0cc2a0 0x7fae2c0cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 -- 172.21.15.114:0/3166370140 >> v2:172.21.15.177:3300/0 conn(0x7fae2c0cf370 msgr2=0x7fae2c06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 --2- 172.21.15.114:0/3166370140 >> v2:172.21.15.177:3300/0 conn(0x7fae2c0cf370 0x7fae2c06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.269+0000 7fae2aa00640 1 -- 172.21.15.114:0/3166370140 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c06ca50 con 0x7fae2c15a940 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.270+0000 7fae2aa00640 1 --2- 172.21.15.114:0/3166370140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c15a940 0x7fae2c0b45e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fae2c068400 tx=0x7fae1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f883bd4d3ff731f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2a000640 1 -- 172.21.15.114:0/3166370140 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c035020 con 0x7fae2c15a940 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2a000640 1 -- 172.21.15.114:0/3166370140 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c031870 con 0x7fae2c15a940 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2a000640 1 -- 172.21.15.114:0/3166370140 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c031c40 con 0x7fae2c15a940 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 -- 172.21.15.114:0/3166370140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c15a940 msgr2=0x7fae2c0b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 --2- 172.21.15.114:0/3166370140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c15a940 0x7fae2c0b45e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fae2c068400 tx=0x7fae1c02b910 comp rx=0 tx=0).stop 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 -- 172.21.15.114:0/3166370140 shutdown_connections 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 --2- 172.21.15.114:0/3166370140 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c0cc2a0 0x7fae2c0cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 --2- 172.21.15.114:0/3166370140 >> v2:172.21.15.177:3300/0 conn(0x7fae2c0cf370 0x7fae2c06d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 --2- 172.21.15.114:0/3166370140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c15a940 0x7fae2c0b45e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 -- 172.21.15.114:0/3166370140 >> 172.21.15.114:0/3166370140 conn(0x7fae2c06afc0 msgr2=0x7fae2c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 -- 172.21.15.114:0/3166370140 shutdown_connections 2024-03-06T07:34:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.271+0000 7fae2be00640 1 -- 172.21.15.114:0/3166370140 wait complete. 2024-03-06T07:34:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.273+0000 7fae2be00640 1 Processor -- start 2024-03-06T07:34:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.273+0000 7fae2be00640 1 -- start start 2024-03-06T07:34:22.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae2c0cc2a0 0x7fae2c146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae2be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c0cf370 0x7fae2c154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae2be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c15a940 0x7fae2c159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae2be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c0b4b20 con 0x7fae2c0cf370 2024-03-06T07:34:22.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae2be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c0b30f0 con 0x7fae2c0cc2a0 2024-03-06T07:34:22.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae2be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c06d790 con 0x7fae2c15a940 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae22000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c0cf370 0x7fae2c154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae22000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c0cf370 0x7fae2c154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38610/0 (socket says 172.21.15.114:38610) 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.274+0000 7fae22000640 1 -- 172.21.15.114:0/622341917 learned_addr learned my addr 172.21.15.114:0/622341917 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae22000640 1 -- 172.21.15.114:0/622341917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c15a940 msgr2=0x7fae2c159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae22000640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c15a940 0x7fae2c159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae22000640 1 -- 172.21.15.114:0/622341917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae2c0cc2a0 msgr2=0x7fae2c146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae22000640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae2c0cc2a0 0x7fae2c146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae22000640 1 -- 172.21.15.114:0/622341917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c154bc0 con 0x7fae2c0cf370 2024-03-06T07:34:22.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae22000640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c0cf370 0x7fae2c154250 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b4d0 tx=0x7fae1400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae23400640 1 -- 172.21.15.114:0/622341917 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14014070 con 0x7fae2c0cf370 2024-03-06T07:34:22.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae23400640 1 -- 172.21.15.114:0/622341917 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae14010870 con 0x7fae2c0cf370 2024-03-06T07:34:22.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae23400640 1 -- 172.21.15.114:0/622341917 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14010c60 con 0x7fae2c0cf370 2024-03-06T07:34:22.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c154ea0 con 0x7fae2c0cf370 2024-03-06T07:34:22.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.275+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c155460 con 0x7fae2c0cf370 2024-03-06T07:34:22.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.276+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c04f330 con 0x7fae2c0cf370 2024-03-06T07:34:22.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.277+0000 7fae23400640 1 -- 172.21.15.114:0/622341917 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae14017070 con 0x7fae2c0cf370 2024-03-06T07:34:22.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.278+0000 7fae23400640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b500 0x7fae0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.278+0000 7fae23400640 1 -- 172.21.15.114:0/622341917 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fae1408a100 con 0x7fae2c0cf370 2024-03-06T07:34:22.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.278+0000 7fae2aa00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b500 0x7fae0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.279+0000 7fae2aa00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b500 0x7fae0806d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fae1c036000 tx=0x7fae1c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.282+0000 7fae23400640 1 -- 172.21.15.114:0/622341917 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae14058e30 con 0x7fae2c0cf370 2024-03-06T07:34:22.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.500+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae2c167930 con 0x7fae2c0cf370 2024-03-06T07:34:22.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.502+0000 7fae23400640 1 -- 172.21.15.114:0/622341917 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4189 (secure 0 0 0) 0x7fae14058730 con 0x7fae2c0cf370 2024-03-06T07:34:22.504 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:22.504 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:22.064034+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:34:22.504 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 84 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b500 msgr2=0x7fae0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b500 0x7fae0806d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fae1c036000 tx=0x7fae1c030000 comp rx=0 tx=0).stop 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c0cf370 msgr2=0x7fae2c154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c0cf370 0x7fae2c154250 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b4d0 tx=0x7fae1400b990 comp rx=0 tx=0).stop 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 shutdown_connections 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae2c15a940 0x7fae2c159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b500 0x7fae0806d9c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae2c0cf370 0x7fae2c154250 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 --2- 172.21.15.114:0/622341917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae2c0cc2a0 0x7fae2c146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 >> 172.21.15.114:0/622341917 conn(0x7fae2c06afc0 msgr2=0x7fae2c0cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 shutdown_connections 2024-03-06T07:34:22.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:22.504+0000 7fae2be00640 1 -- 172.21.15.114:0/622341917 wait complete. 2024-03-06T07:34:22.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:06.421022+0000', 'modified': '2024-03-06T07:34:22.064034+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7911}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7911': {'gid': 7911, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/2704122879', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 2704122879}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 2704122879}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:22.927 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7911:up:replay 2024-03-06T07:34:23.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:24.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.148+0000 7fcce1c00640 1 Processor -- start 2024-03-06T07:34:24.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.148+0000 7fcce1c00640 1 -- start start 2024-03-06T07:34:24.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.149+0000 7fcce1c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 0x7fccdc070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.149+0000 7fcce1c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fccdc071260 0x7fccdc06cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fcce1c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccdc0717c0 con 0x7fccdc06d090 2024-03-06T07:34:24.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fcce1c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccdc06d4c0 con 0x7fccdc071260 2024-03-06T07:34:24.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fcce1c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc071aa0 con 0x7fccdc0cf430 2024-03-06T07:34:24.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fccdb400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fccdc06d090 0x7fccdc165f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37818/0 (socket says 172.21.15.114:37818) 2024-03-06T07:34:24.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fccdb400640 1 -- 172.21.15.114:0/3323328808 learned_addr learned my addr 172.21.15.114:0/3323328808 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fccd2000640 1 --2- 172.21.15.114:0/3323328808 >> v2:172.21.15.177:3300/0 conn(0x7fccdc071260 0x7fccdc06cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fccdaa00640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 0x7fccdc070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843461015 0 0) 0x7fccdc0717c0 con 0x7fccdc06d090 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.150+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc0003520 con 0x7fccdc06d090 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016852926 0 0) 0x7fccc0003520 con 0x7fccdc06d090 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 msgr2=0x7fccdc070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 0x7fccdc070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 >> v2:172.21.15.177:3300/0 conn(0x7fccdc071260 msgr2=0x7fccdc06cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 --2- 172.21.15.114:0/3323328808 >> v2:172.21.15.177:3300/0 conn(0x7fccdc071260 0x7fccdc06cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc071d80 con 0x7fccdc06d090 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fccd4001670 con 0x7fccdc06d090 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 >> v1:172.21.15.114:6789/0 conn(0x7fccdc06d090 legacy=0x7fccdc165f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 0x7fccc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccc0004d40 0x7fccc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc06d4c0 con 0x7fccc00039b0 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc071aa0 con 0x7fccc0004d40 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc000b9c0 con 0x7fccdc0cf430 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.151+0000 7fccdaa00640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccd2000640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 0x7fccc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccdb400640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccc0004d40 0x7fccc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccdaa00640 1 -- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 msgr2=0x7fccc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccdaa00640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc0cf430 0x7fccc0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccdaa00640 1 -- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccc0004d40 msgr2=0x7fccc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccdaa00640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccc0004d40 0x7fccc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccdaa00640 1 -- 172.21.15.114:0/3323328808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc0003520 con 0x7fccc00039b0 2024-03-06T07:34:24.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.152+0000 7fccdb400640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccc0004d40 0x7fccc00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.153+0000 7fccdaa00640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccc0003dd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fcccc007390 tx=0x7fccc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbccf3be2a2691b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.153+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc039070 con 0x7fccc00039b0 2024-03-06T07:34:24.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.153+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc02ccb0 con 0x7fccc00039b0 2024-03-06T07:34:24.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.153+0000 7fccda000640 1 -- 172.21.15.114:0/3323328808 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc009be0 con 0x7fccc00039b0 2024-03-06T07:34:24.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.154+0000 7fcce1c00640 1 -- 172.21.15.114:0/3323328808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 msgr2=0x7fccc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.154+0000 7fcce1c00640 1 --2- 172.21.15.114:0/3323328808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccc0003dd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fcccc007390 tx=0x7fccc0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:24.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.154+0000 7fccdb400640 1 -- 172.21.15.114:0/3323328808 reap_dead start 2024-03-06T07:34:24.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.154+0000 7fcce1c00640 1 -- 172.21.15.114:0/3323328808 shutdown_connections 2024-03-06T07:34:24.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.154+0000 7fcce1c00640 1 -- 172.21.15.114:0/3323328808 >> 172.21.15.114:0/3323328808 conn(0x7fccdc0b1730 msgr2=0x7fccdc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.154+0000 7fcce1c00640 1 -- 172.21.15.114:0/3323328808 shutdown_connections 2024-03-06T07:34:24.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.154+0000 7fcce1c00640 1 -- 172.21.15.114:0/3323328808 wait complete. 2024-03-06T07:34:24.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.155+0000 7fcce1c00640 1 Processor -- start 2024-03-06T07:34:24.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.156+0000 7fcce1c00640 1 -- start start 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.156+0000 7fcce1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccdc0aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.156+0000 7fcce1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccdc1541b0 0x7fccdc154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fcce1c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc1524a0 0x7fccdc152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fcce1c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc162b70 con 0x7fccc00039b0 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fcce1c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc072340 con 0x7fccdc1541b0 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fcce1c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc162cf0 con 0x7fccdc1524a0 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccdc0aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccdc0aed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38636/0 (socket says 172.21.15.114:38636) 2024-03-06T07:34:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 -- 172.21.15.114:0/2566766489 learned_addr learned my addr 172.21.15.114:0/2566766489 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 -- 172.21.15.114:0/2566766489 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc1524a0 msgr2=0x7fccdc152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc1524a0 0x7fccdc152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 -- 172.21.15.114:0/2566766489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccdc1541b0 msgr2=0x7fccdc154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccdc1541b0 0x7fccdc154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 -- 172.21.15.114:0/2566766489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc0af2d0 con 0x7fccc00039b0 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.157+0000 7fccdaa00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccdc0aed90 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fcccc000f80 tx=0x7fcccc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.158+0000 7fccd3400640 1 -- 172.21.15.114:0/2566766489 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc042030 con 0x7fccc00039b0 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.158+0000 7fccd3400640 1 -- 172.21.15.114:0/2566766489 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcccc03a040 con 0x7fccc00039b0 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.158+0000 7fccd3400640 1 -- 172.21.15.114:0/2566766489 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcccc039070 con 0x7fccc00039b0 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.158+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc152e60 con 0x7fccc00039b0 2024-03-06T07:34:24.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.158+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc153400 con 0x7fccc00039b0 2024-03-06T07:34:24.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.159+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc04f330 con 0x7fccc00039b0 2024-03-06T07:34:24.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.160+0000 7fccd3400640 1 -- 172.21.15.114:0/2566766489 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcccc045070 con 0x7fccc00039b0 2024-03-06T07:34:24.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.161+0000 7fccd3400640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccb806b500 0x7fccb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.161+0000 7fccd3400640 1 -- 172.21.15.114:0/2566766489 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fcccc0ac230 con 0x7fccc00039b0 2024-03-06T07:34:24.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.161+0000 7fccd2000640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccb806b500 0x7fccb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.162+0000 7fccd2000640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccb806b500 0x7fccb806d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fccdc165390 tx=0x7fccc4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.165+0000 7fccd3400640 1 -- 172.21.15.114:0/2566766489 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccdc0af2d0 con 0x7fccc00039b0 2024-03-06T07:34:24.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.418+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccdc0b4a40 con 0x7fccc00039b0 2024-03-06T07:34:24.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.419+0000 7fccd3400640 1 -- 172.21.15.114:0/2566766489 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fcccc07b090 con 0x7fccc00039b0 2024-03-06T07:34:24.422 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:24.422 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":86,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:23.085292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:34:24.422 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 86 2024-03-06T07:34:24.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.421+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccb806b500 msgr2=0x7fccb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.421+0000 7fcce1c00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccb806b500 0x7fccb806d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fccdc165390 tx=0x7fccc4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:24.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.421+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 msgr2=0x7fccdc0aed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.421+0000 7fcce1c00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccdc0aed90 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fcccc000f80 tx=0x7fcccc004ab0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.421+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 shutdown_connections 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.421+0000 7fcce1c00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fccdc1524a0 0x7fccdc152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.421+0000 7fcce1c00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fccdc1541b0 0x7fccdc154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.422+0000 7fcce1c00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fccb806b500 0x7fccb806d9c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.422+0000 7fcce1c00640 1 --2- 172.21.15.114:0/2566766489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc00039b0 0x7fccdc0aed90 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.422+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 >> 172.21.15.114:0/2566766489 conn(0x7fccdc0b1730 msgr2=0x7fccdc165e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.422+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 shutdown_connections 2024-03-06T07:34:24.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:24.422+0000 7fcce1c00640 1 -- 172.21.15.114:0/2566766489 wait complete. 2024-03-06T07:34:24.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:06.421022+0000', 'modified': '2024-03-06T07:34:23.085292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7911}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7911': {'gid': 7911, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.114:6835/2704122879', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 2704122879}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 2704122879}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:24.844 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7911:up:rejoin 2024-03-06T07:34:25.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:26.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.053+0000 7f2e9fc00640 1 Processor -- start 2024-03-06T07:34:26.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.053+0000 7f2e9fc00640 1 -- start start 2024-03-06T07:34:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.055+0000 7f2e9fc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 0x7f2e9806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.055+0000 7f2e9fc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2e9815fe30 0x7f2e980ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.055+0000 7f2e9fc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9806ee70 con 0x7f2e9815f9d0 2024-03-06T07:34:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.055+0000 7f2e9fc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9806cef0 con 0x7f2e9815fe30 2024-03-06T07:34:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.055+0000 7f2e9fc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9806d1d0 con 0x7f2e98144c00 2024-03-06T07:34:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2e9815f9d0 0x7f2e980ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37820/0 (socket says 172.21.15.114:37820) 2024-03-06T07:34:26.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97e00640 1 -- 172.21.15.114:0/1323681861 learned_addr learned my addr 172.21.15.114:0/1323681861 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e9dc00640 1 --2- 172.21.15.114:0/1323681861 >> v2:172.21.15.177:3300/0 conn(0x7f2e9815fe30 0x7f2e980ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e9d200640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 0x7f2e9806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652174224 0 0) 0x7f2e84001110 con 0x7f2e9815f9d0 2024-03-06T07:34:26.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e80003520 con 0x7f2e9815f9d0 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131062993 0 0) 0x7f2e80003520 con 0x7f2e9815f9d0 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 msgr2=0x7f2e9806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 0x7f2e9806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 >> v2:172.21.15.177:3300/0 conn(0x7f2e9815fe30 msgr2=0x7f2e980ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 --2- 172.21.15.114:0/1323681861 >> v2:172.21.15.177:3300/0 conn(0x7f2e9815fe30 0x7f2e980ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9806d4b0 con 0x7f2e9815f9d0 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.056+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2e84001840 con 0x7f2e9815f9d0 2024-03-06T07:34:26.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 >> v1:172.21.15.114:6789/0 conn(0x7f2e9815f9d0 legacy=0x7f2e980ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97400640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e800039b0 0x7f2e80003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97400640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e80004800 0x7f2e80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97400640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 0x7f2e800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9806cef0 con 0x7f2e80004800 2024-03-06T07:34:26.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9806d1d0 con 0x7f2e800039b0 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8000b9c0 con 0x7f2e98144c00 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97e00640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e80004800 0x7f2e80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e9d200640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e800039b0 0x7f2e80003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e9dc00640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 0x7f2e800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97e00640 1 -- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 msgr2=0x7f2e800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97e00640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e98144c00 0x7f2e800051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97e00640 1 -- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e800039b0 msgr2=0x7f2e80003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97e00640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e800039b0 0x7f2e80003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.057+0000 7f2e97e00640 1 -- 172.21.15.114:0/1323681861 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e80003520 con 0x7f2e80004800 2024-03-06T07:34:26.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.058+0000 7f2e97e00640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e80004800 0x7f2e80004c80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400c3e0 tx=0x7f2e80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847b57ca699611d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8400c0e0 con 0x7f2e80004800 2024-03-06T07:34:26.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e84030070 con 0x7f2e80004800 2024-03-06T07:34:26.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e97400640 1 -- 172.21.15.114:0/1323681861 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840058d0 con 0x7f2e80004800 2024-03-06T07:34:26.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e9fc00640 1 -- 172.21.15.114:0/1323681861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e80004800 msgr2=0x7f2e80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e9fc00640 1 --2- 172.21.15.114:0/1323681861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e80004800 0x7f2e80004c80 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400c3e0 tx=0x7f2e80005830 comp rx=0 tx=0).stop 2024-03-06T07:34:26.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e9dc00640 1 -- 172.21.15.114:0/1323681861 reap_dead start 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e9fc00640 1 -- 172.21.15.114:0/1323681861 shutdown_connections 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e9fc00640 1 -- 172.21.15.114:0/1323681861 >> 172.21.15.114:0/1323681861 conn(0x7f2e980b1730 msgr2=0x7f2e980b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e9fc00640 1 -- 172.21.15.114:0/1323681861 shutdown_connections 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.059+0000 7f2e9fc00640 1 -- 172.21.15.114:0/1323681861 wait complete. 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.061+0000 7f2e9fc00640 1 Processor -- start 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.061+0000 7f2e9fc00640 1 -- start start 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e9fc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e80004800 0x7f2e981e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e9fc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e9813adb0 0x7f2e9815ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e9fc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e9815c340 0x7f2e981e5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e9fc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e98160730 con 0x7f2e9813adb0 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e9fc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e98160280 con 0x7f2e80004800 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e9fc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9806cb50 con 0x7f2e9815c340 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e97e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e9813adb0 0x7f2e9815ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e97e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e9813adb0 0x7f2e9815ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38646/0 (socket says 172.21.15.114:38646) 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.062+0000 7f2e97e00640 1 -- 172.21.15.114:0/2762951556 learned_addr learned my addr 172.21.15.114:0/2762951556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e9d200640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e80004800 0x7f2e981e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e9dc00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e9815c340 0x7f2e981e5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e97e00640 1 -- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e9815c340 msgr2=0x7f2e981e5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e97e00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e9815c340 0x7f2e981e5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e97e00640 1 -- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e80004800 msgr2=0x7f2e981e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e97e00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e80004800 0x7f2e981e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e97e00640 1 -- 172.21.15.114:0/2762951556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e981e5b00 con 0x7f2e9813adb0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e9d200640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e80004800 0x7f2e981e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e97e00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e9813adb0 0x7f2e9815ae60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400c3e0 tx=0x7f2e84001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e94c00640 1 -- 172.21.15.114:0/2762951556 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84004bf0 con 0x7f2e9813adb0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e94c00640 1 -- 172.21.15.114:0/2762951556 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e84004d90 con 0x7f2e9813adb0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e94c00640 1 -- 172.21.15.114:0/2762951556 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84007270 con 0x7f2e9813adb0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.063+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9813b240 con 0x7f2e9813adb0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.064+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9813b800 con 0x7f2e9813adb0 2024-03-06T07:34:26.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.065+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9804f330 con 0x7f2e9813adb0 2024-03-06T07:34:26.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.066+0000 7f2e94c00640 1 -- 172.21.15.114:0/2762951556 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e84007410 con 0x7f2e9813adb0 2024-03-06T07:34:26.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.066+0000 7f2e94c00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2e7806b4d0 0x7f2e7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.066+0000 7f2e94c00640 1 -- 172.21.15.114:0/2762951556 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f2e840aba10 con 0x7f2e9813adb0 2024-03-06T07:34:26.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.067+0000 7f2e9d200640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2e7806b4d0 0x7f2e7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.067+0000 7f2e9d200640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2e7806b4d0 0x7f2e7806d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2e98144210 tx=0x7f2e8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.071+0000 7f2e94c00640 1 -- 172.21.15.114:0/2762951556 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e9806ee70 con 0x7f2e9813adb0 2024-03-06T07:34:26.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.382+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e98145380 con 0x7f2e9813adb0 2024-03-06T07:34:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.383+0000 7f2e94c00640 1 -- 172.21.15.114:0/2762951556 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f2e8407a740 con 0x7f2e9813adb0 2024-03-06T07:34:26.386 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:26.386 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:34:26.386 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:34:26.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2e7806b4d0 msgr2=0x7f2e7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2e7806b4d0 0x7f2e7806d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2e98144210 tx=0x7f2e8c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:26.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e9813adb0 msgr2=0x7f2e9815ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e9813adb0 0x7f2e9815ae60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400c3e0 tx=0x7f2e84001c10 comp rx=0 tx=0).stop 2024-03-06T07:34:26.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 shutdown_connections 2024-03-06T07:34:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e9815c340 0x7f2e981e5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2e7806b4d0 0x7f2e7806d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e9813adb0 0x7f2e9815ae60 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 --2- 172.21.15.114:0/2762951556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e80004800 0x7f2e981e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 >> 172.21.15.114:0/2762951556 conn(0x7f2e980b1730 msgr2=0x7f2e980ceec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.386+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 shutdown_connections 2024-03-06T07:34:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:26.387+0000 7f2e9fc00640 1 -- 172.21.15.114:0/2762951556 wait complete. 2024-03-06T07:34:26.817 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:34:06.421022+0000', 'modified': '2024-03-06T07:34:25.093846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7911}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7911': {'gid': 7911, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.114:6835/2704122879', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 2704122879}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 2704122879}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7911, 'qdb_cluster': [7911]} 2024-03-06T07:34:26.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:34:26.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:34:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.026+0000 7f908f200640 1 Processor -- start 2024-03-06T07:34:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.026+0000 7f908f200640 1 -- start start 2024-03-06T07:34:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9088145540 0x7f908806e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908f200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f908815f950 0x7f908815cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908f200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908806e5e0 con 0x7f908806afc0 2024-03-06T07:34:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908f200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90880ce960 con 0x7f9088145540 2024-03-06T07:34:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908f200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908815d2f0 con 0x7f908815f950 2024-03-06T07:34:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9088145540 0x7f908806e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9088145540 0x7f908806e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38654/0 (socket says 172.21.15.114:38654) 2024-03-06T07:34:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.028+0000 7f908de00640 1 -- 172.21.15.114:0/781202046 learned_addr learned my addr 172.21.15.114:0/781202046 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.029+0000 7f908de00640 1 -- 172.21.15.114:0/781202046 >> v2:172.21.15.177:3300/0 conn(0x7f908815f950 msgr2=0x7f908815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.029+0000 7f908de00640 1 --2- 172.21.15.114:0/781202046 >> v2:172.21.15.177:3300/0 conn(0x7f908815f950 0x7f908815cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.029+0000 7f908de00640 1 -- 172.21.15.114:0/781202046 >> v1:172.21.15.177:6789/0 conn(0x7f908806afc0 legacy=0x7f90880cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.029+0000 7f908de00640 1 -- 172.21.15.114:0/781202046 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908815fed0 con 0x7f9088145540 2024-03-06T07:34:27.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.030+0000 7f908de00640 1 --2- 172.21.15.114:0/781202046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9088145540 0x7f908806e0a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f90880b5fd0 tx=0x7f907c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45212530719ddf07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.030+0000 7f908ca00640 1 -- 172.21.15.114:0/781202046 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c035020 con 0x7f9088145540 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.030+0000 7f908ca00640 1 -- 172.21.15.114:0/781202046 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c031870 con 0x7f9088145540 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.030+0000 7f908ca00640 1 -- 172.21.15.114:0/781202046 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c031c40 con 0x7f9088145540 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 -- 172.21.15.114:0/781202046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9088145540 msgr2=0x7f908806e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 --2- 172.21.15.114:0/781202046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9088145540 0x7f908806e0a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f90880b5fd0 tx=0x7f907c02b910 comp rx=0 tx=0).stop 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 -- 172.21.15.114:0/781202046 shutdown_connections 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 --2- 172.21.15.114:0/781202046 >> v2:172.21.15.177:3300/0 conn(0x7f908815f950 0x7f908815cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 --2- 172.21.15.114:0/781202046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9088145540 0x7f908806e0a0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 -- 172.21.15.114:0/781202046 >> 172.21.15.114:0/781202046 conn(0x7f908806c0d0 msgr2=0x7f908806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 -- 172.21.15.114:0/781202046 shutdown_connections 2024-03-06T07:34:27.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.031+0000 7f908f200640 1 -- 172.21.15.114:0/781202046 wait complete. 2024-03-06T07:34:27.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.033+0000 7f908f200640 1 Processor -- start 2024-03-06T07:34:27.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.033+0000 7f908f200640 1 -- start start 2024-03-06T07:34:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908806afc0 0x7f90881e7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9088145540 0x7f90881f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908815f950 0x7f908815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9088160770 con 0x7f908815f950 2024-03-06T07:34:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9088160490 con 0x7f908806afc0 2024-03-06T07:34:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9088145920 con 0x7f9088145540 2024-03-06T07:34:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908815f950 0x7f908815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908815f950 0x7f908815ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38660/0 (socket says 172.21.15.114:38660) 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.034+0000 7f908e800640 1 -- 172.21.15.114:0/859923758 learned_addr learned my addr 172.21.15.114:0/859923758 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908de00640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908806afc0 0x7f90881e7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908e800640 1 -- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9088145540 msgr2=0x7f90881f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908d400640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9088145540 0x7f90881f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908e800640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9088145540 0x7f90881f2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908e800640 1 -- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908806afc0 msgr2=0x7f90881e7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908e800640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908806afc0 0x7f90881e7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908e800640 1 -- 172.21.15.114:0/859923758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908815f210 con 0x7f908815f950 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908de00640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908806afc0 0x7f90881e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908e800640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908815f950 0x7f908815ecd0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f908400c7b0 tx=0x7f908400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f9082000640 1 -- 172.21.15.114:0/859923758 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084017070 con 0x7f908815f950 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f9082000640 1 -- 172.21.15.114:0/859923758 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084003da0 con 0x7f908815f950 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90881f2d30 con 0x7f908815f950 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f9082000640 1 -- 172.21.15.114:0/859923758 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840052b0 con 0x7f908815f950 2024-03-06T07:34:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.035+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90881f32f0 con 0x7f908815f950 2024-03-06T07:34:27.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.039+0000 7f9082000640 1 -- 172.21.15.114:0/859923758 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f908401a070 con 0x7f908815f950 2024-03-06T07:34:27.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.039+0000 7f909359c740 1 Processor -- start 2024-03-06T07:34:27.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.039+0000 7f909359c740 1 -- start start 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f9082000640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b4d0 0x7f906c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f9082000640 1 -- 172.21.15.114:0/859923758 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9084089860 con 0x7f908815f950 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f908d400640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b4d0 0x7f906c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9c990 0x560f26d6f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f909359c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9cd70 0x560f26d9f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d6fe70 0x560f26d9fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f909359c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f26cd9600 con 0x560f26d9cd70 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f26c82930 con 0x560f26d6fe70 2024-03-06T07:34:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f26cd56a0 con 0x560f26d9c990 2024-03-06T07:34:27.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f908d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9cd70 0x560f26d9f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f908d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9cd70 0x560f26d9f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38670/0 (socket says 172.21.15.114:38670) 2024-03-06T07:34:27.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.040+0000 7f908d400640 1 -- 172.21.15.114:0/403913565 learned_addr learned my addr 172.21.15.114:0/403913565 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908de00640 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9c990 0x560f26d6f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908e800640 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d6fe70 0x560f26d9fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908d400640 1 -- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9c990 msgr2=0x560f26d6f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908d400640 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9c990 0x560f26d6f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908d400640 1 -- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d6fe70 msgr2=0x560f26d9fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908d400640 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d6fe70 0x560f26d9fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908d400640 1 -- 172.21.15.114:0/403913565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f269ee2d0 con 0x560f26d9cd70 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908d400640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b4d0 0x7f906c06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f907800ac90 tx=0x7f9078013650 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.041+0000 7f908d400640 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9cd70 0x560f26d9f720 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f907800af60 tx=0x7f907801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0dea932f105e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f9076600640 1 -- 172.21.15.114:0/403913565 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078026070 con 0x560f26d9cd70 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f9076600640 1 -- 172.21.15.114:0/403913565 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9078021660 con 0x560f26d9cd70 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f9076600640 1 -- 172.21.15.114:0/403913565 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907800f040 con 0x560f26d9cd70 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 -- 172.21.15.114:0/403913565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9cd70 msgr2=0x560f26d9f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9cd70 0x560f26d9f720 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f907800af60 tx=0x7f907801b000 comp rx=0 tx=0).stop 2024-03-06T07:34:27.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 -- 172.21.15.114:0/403913565 shutdown_connections 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d6fe70 0x560f26d9fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9cd70 0x560f26d9f720 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 --2- 172.21.15.114:0/403913565 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9c990 0x560f26d6f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 -- 172.21.15.114:0/403913565 >> 172.21.15.114:0/403913565 conn(0x560f26d71ba0 msgr2=0x560f26dba720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 -- 172.21.15.114:0/403913565 shutdown_connections 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.042+0000 7f909359c740 1 -- 172.21.15.114:0/403913565 wait complete. 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.044+0000 7f909359c740 1 -- start start 2024-03-06T07:34:27.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.044+0000 7f909359c740 1 Processor -- start 2024-03-06T07:34:27.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.045+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26db5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.045+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26db3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f909359c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26d9b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f909359c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f26d74140 con 0x560f26db3770 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f26c95d90 con 0x560f26d9c990 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f26d73530 con 0x560f26d9cd70 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f909359c740 10 client.0 _ll_register_callbacks cb 0x560f26cd1350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26d9b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26d9b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38686/0 (socket says 172.21.15.114:38686) 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 -- 172.21.15.114:0/3431464796 learned_addr learned my addr 172.21.15.114:0/3431464796 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908de00640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26db5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908d400640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26db3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 -- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 msgr2=0x560f26db3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26db3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 -- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 msgr2=0x560f26db5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26db5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908e800640 1 -- 172.21.15.114:0/3431464796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f26ab86b0 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.046+0000 7f908d400640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26db3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f908e800640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26d9b3b0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f9084000f80 tx=0x7f9084058100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5d918b8889d8fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f906aa00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f906aa00640 1 -- 172.21.15.114:0/3431464796 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084060440 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f906aa00640 1 -- 172.21.15.114:0/3431464796 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9084057410 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9078021660 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f906aa00640 1 -- 172.21.15.114:0/3431464796 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9084057610 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 10 client.7953 fetch_fsmap learned FSMap version 87 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x560f26da1620 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f906aa00640 1 -- 172.21.15.114:0/3431464796 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9078021660 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f906aa00640 1 -- 172.21.15.114:0/3431464796 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 87) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f908405a040 con 0x560f26db3770 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 10 client.7953 fetch_fsmap finished waiting for FSMap version 87 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 10 client.7953 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:34:27.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 20 client.7953 populate_metadata read hostname 'smithi114' 2024-03-06T07:34:27.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26dce6e0 0x560f26da23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 4 client.7953 mds_command: new command op to 7911 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:34:27.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.047+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 --> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f26ce6b40 con 0x560f26dce6e0 2024-03-06T07:34:27.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.048+0000 7f908de00640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26dce6e0 0x560f26da23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.048+0000 7f906aa00640 10 client.7953 ms_handle_connect on v2:172.21.15.114:6834/2704122879 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.048+0000 7f908de00640 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26dce6e0 0x560f26da23d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=996c046c9f7f776d server_cookie=9d280b31eb503f33 in_seq=0 out_seq=0 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f906aa00640 1 -- 172.21.15.114:0/3431464796 <== mds.0 v2:172.21.15.114:6834/2704122879 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560f26ce6b40 con 0x560f26dce6e0 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f906aa00640 10 client.7953 handle_command_reply: tid=0 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f909359c740 1 client.7953 shutdown 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 msgr2=0x560f26d9b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f909359c740 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26d9b3b0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f9084000f80 tx=0x7f9084058100 comp rx=0 tx=0).stop 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 shutdown_connections 2024-03-06T07:34:27.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f909359c740 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26d9b3b0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f909359c740 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26db3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.049+0000 7f909359c740 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26dce6e0 0x560f26da23d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.050+0000 7f909359c740 1 --2- 172.21.15.114:0/3431464796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26db5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.050+0000 7f906aa00640 0 client.7953 ms_handle_reset on v2:172.21.15.114:6834/2704122879 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.050+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 >> 172.21.15.114:0/3431464796 conn(0x560f26d6fe70 msgr2=0x560f26d6f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.050+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 shutdown_connections 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.050+0000 7f909359c740 1 -- 172.21.15.114:0/3431464796 wait complete. 2024-03-06T07:34:27.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.050+0000 7f909359c740 20 client.7953 trim_cache size 0 max 16384 2024-03-06T07:34:27.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.068+0000 7f909359c740 1 Processor -- start 2024-03-06T07:34:27.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.068+0000 7f909359c740 1 -- start start 2024-03-06T07:34:27.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f909359c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9c990 0x560f26dba4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9cd70 0x560f26dbd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26db3770 0x560f26dbde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f909359c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9088158ff0 con 0x560f26d9c990 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908815b2d0 con 0x560f26d9cd70 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908815b8a0 con 0x560f26db3770 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f908de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9c990 0x560f26dba4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f908de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9c990 0x560f26dba4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38698/0 (socket says 172.21.15.114:38698) 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.069+0000 7f908de00640 1 -- 172.21.15.114:0/3635279240 learned_addr learned my addr 172.21.15.114:0/3635279240 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f908de00640 1 -- 172.21.15.114:0/3635279240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26db3770 msgr2=0x560f26dbde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f908de00640 1 --2- 172.21.15.114:0/3635279240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26db3770 0x560f26dbde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f908de00640 1 -- 172.21.15.114:0/3635279240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9cd70 msgr2=0x560f26dbd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f908de00640 1 --2- 172.21.15.114:0/3635279240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9cd70 0x560f26dbd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f908de00640 1 -- 172.21.15.114:0/3635279240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f26db3c90 con 0x560f26d9c990 2024-03-06T07:34:27.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f908de00640 1 --2- 172.21.15.114:0/3635279240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9c990 0x560f26dba4c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f907c009630 tx=0x7f907c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11df65007b6e5fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f9075c00640 1 -- 172.21.15.114:0/3635279240 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c04e070 con 0x560f26d9c990 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f9075c00640 1 -- 172.21.15.114:0/3635279240 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c049170 con 0x560f26d9c990 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.070+0000 7f9075c00640 1 -- 172.21.15.114:0/3635279240 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c035040 con 0x560f26d9c990 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 -- 172.21.15.114:0/3635279240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9c990 msgr2=0x560f26dba4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 --2- 172.21.15.114:0/3635279240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9c990 0x560f26dba4c0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f907c009630 tx=0x7f907c030000 comp rx=0 tx=0).stop 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 -- 172.21.15.114:0/3635279240 shutdown_connections 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 --2- 172.21.15.114:0/3635279240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26db3770 0x560f26dbde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 --2- 172.21.15.114:0/3635279240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9cd70 0x560f26dbd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 --2- 172.21.15.114:0/3635279240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26d9c990 0x560f26dba4c0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 -- 172.21.15.114:0/3635279240 >> 172.21.15.114:0/3635279240 conn(0x560f26d6fe70 msgr2=0x560f26d9f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 -- 172.21.15.114:0/3635279240 shutdown_connections 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.071+0000 7f909359c740 1 -- 172.21.15.114:0/3635279240 wait complete. 2024-03-06T07:34:27.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.072+0000 7f909359c740 1 -- start start 2024-03-06T07:34:27.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.072+0000 7f909359c740 1 Processor -- start 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26e2e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f909359c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26da2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f909359c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26e2f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f909359c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f26dce560 con 0x560f26db3770 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f26da1960 con 0x560f26d9c990 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f909359c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f26da1ef0 con 0x560f26d9cd70 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f909359c740 10 client.0 _ll_register_callbacks cb 0x7f9084054320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f908e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26e2f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f908e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26e2f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38702/0 (socket says 172.21.15.114:38702) 2024-03-06T07:34:27.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.073+0000 7f908e800640 1 -- 172.21.15.114:0/305585982 learned_addr learned my addr 172.21.15.114:0/305585982 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908d400640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26da2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908de00640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26e2e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908e800640 1 -- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 msgr2=0x560f26da2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908e800640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26da2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908e800640 1 -- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 msgr2=0x560f26e2e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908e800640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26e2e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908e800640 1 -- 172.21.15.114:0/305585982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f26dca370 con 0x560f26db3770 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908d400640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26da2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f908e800640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26e2f2a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f9084054420 tx=0x7f9084003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba14ad3e5ec780d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f906aa00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f906aa00640 1 -- 172.21.15.114:0/305585982 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084006b10 con 0x560f26db3770 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f906aa00640 1 -- 172.21.15.114:0/305585982 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f9084006cb0 con 0x560f26db3770 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.074+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f907c049170 con 0x560f26db3770 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f906aa00640 1 -- 172.21.15.114:0/305585982 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9084053d10 con 0x560f26db3770 2024-03-06T07:34:27.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 10 client.7959 fetch_fsmap learned FSMap version 87 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x560f26e2fbd0 con 0x560f26db3770 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f906aa00640 1 -- 172.21.15.114:0/305585982 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c049170 con 0x560f26db3770 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f906aa00640 1 -- 172.21.15.114:0/305585982 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 87) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f908405a040 con 0x560f26db3770 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 10 client.7959 fetch_fsmap finished waiting for FSMap version 87 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 10 client.7959 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 20 client.7959 populate_metadata read hostname 'smithi114' 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26d748b0 0x560f26db9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 4 client.7959 mds_command: new command op to 7911 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 --> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560f26ce0e90 con 0x560f26d748b0 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.075+0000 7f908de00640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26d748b0 0x560f26db9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f908de00640 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26d748b0 0x560f26db9320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fc6ddf5f3116d5b server_cookie=68777e37d82295f1 in_seq=0 out_seq=0 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f906aa00640 10 client.7959 ms_handle_connect on v2:172.21.15.114:6834/2704122879 2024-03-06T07:34:27.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f906aa00640 1 -- 172.21.15.114:0/305585982 <== mds.0 v2:172.21.15.114:6834/2704122879 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x560f26ce0e90 con 0x560f26d748b0 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f906aa00640 10 client.7959 handle_command_reply: tid=0 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 client.7959 shutdown 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 msgr2=0x560f26e2f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26e2f2a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f9084054420 tx=0x7f9084003f50 comp rx=0 tx=0).stop 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 shutdown_connections 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f26db3770 0x560f26e2f2a0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x560f26d9cd70 0x560f26da2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.114:6834/2704122879,v1:172.21.15.114:6835/2704122879] conn(0x560f26d748b0 0x560f26db9320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 --2- 172.21.15.114:0/305585982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f26d9c990 0x560f26e2e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 >> 172.21.15.114:0/305585982 conn(0x560f26d6fe70 msgr2=0x560f26dbd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f906aa00640 0 client.7959 ms_handle_reset on v2:172.21.15.114:6834/2704122879 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 shutdown_connections 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.076+0000 7f909359c740 1 -- 172.21.15.114:0/305585982 wait complete. 2024-03-06T07:34:27.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f909359c740 20 client.7959 trim_cache size 0 max 16384 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 7911, 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 5.0057180949999998, 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 87, 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 112, 2024-03-06T07:34:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 112, 2024-03-06T07:34:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 5.8803277490000001 2024-03-06T07:34:27.088 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:34:27.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b4d0 msgr2=0x7f906c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b4d0 0x7f906c06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f907800ac90 tx=0x7f9078013650 comp rx=0 tx=0).stop 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908815f950 msgr2=0x7f908815ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908815f950 0x7f908815ecd0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f908400c7b0 tx=0x7f908400cc70 comp rx=0 tx=0).stop 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 shutdown_connections 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f906c06b4d0 0x7f906c06d990 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f908815f950 0x7f908815ecd0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9088145540 0x7f90881f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 --2- 172.21.15.114:0/859923758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f908806afc0 0x7f90881e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.077+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 >> 172.21.15.114:0/859923758 conn(0x7f908806c0d0 msgr2=0x7f9088143490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.078+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 shutdown_connections 2024-03-06T07:34:27.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.078+0000 7f908f200640 1 -- 172.21.15.114:0/859923758 wait complete. 2024-03-06T07:34:27.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:34:27.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.307+0000 7f009b800640 1 Processor -- start 2024-03-06T07:34:27.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.307+0000 7f009b800640 1 -- start start 2024-03-06T07:34:27.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f009b800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0094144c00 0x7f009406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f009b800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f009415f9d0 0x7f009415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f009b800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f00940ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f009b800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009406ee70 con 0x7f009415f9d0 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f009b800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009406cef0 con 0x7f0094144c00 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f009b800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009406d1d0 con 0x7f00940ce0a0 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f0093e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f009415f9d0 0x7f009415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f0093e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f009415f9d0 0x7f009415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38706/0 (socket says 172.21.15.114:38706) 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f0093e00640 1 -- 172.21.15.114:0/1501981724 learned_addr learned my addr 172.21.15.114:0/1501981724 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.309+0000 7f0099800640 1 --2- 172.21.15.114:0/1501981724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f00940ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.310+0000 7f0098e00640 1 --2- 172.21.15.114:0/1501981724 >> v2:172.21.15.177:3300/0 conn(0x7f0094144c00 0x7f009406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.310+0000 7f0093e00640 1 -- 172.21.15.114:0/1501981724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 msgr2=0x7f00940ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.310+0000 7f0093e00640 1 --2- 172.21.15.114:0/1501981724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f00940ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.310+0000 7f0093e00640 1 -- 172.21.15.114:0/1501981724 >> v2:172.21.15.177:3300/0 conn(0x7f0094144c00 msgr2=0x7f009406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.310+0000 7f0093e00640 1 --2- 172.21.15.114:0/1501981724 >> v2:172.21.15.177:3300/0 conn(0x7f0094144c00 0x7f009406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.310+0000 7f0093e00640 1 -- 172.21.15.114:0/1501981724 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009406d4b0 con 0x7f009415f9d0 2024-03-06T07:34:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.310+0000 7f0098e00640 1 --2- 172.21.15.114:0/1501981724 >> v2:172.21.15.177:3300/0 conn(0x7f0094144c00 0x7f009406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.311+0000 7f0093e00640 1 --2- 172.21.15.114:0/1501981724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009415f9d0 0x7f009415fe60 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f00940b25f0 tx=0x7f008002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfaa6eb03deb7460 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.311+0000 7f0093400640 1 -- 172.21.15.114:0/1501981724 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080035020 con 0x7f009415f9d0 2024-03-06T07:34:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.311+0000 7f0093400640 1 -- 172.21.15.114:0/1501981724 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0080031870 con 0x7f009415f9d0 2024-03-06T07:34:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.311+0000 7f0093400640 1 -- 172.21.15.114:0/1501981724 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080031c40 con 0x7f009415f9d0 2024-03-06T07:34:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 -- 172.21.15.114:0/1501981724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009415f9d0 msgr2=0x7f009415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 --2- 172.21.15.114:0/1501981724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009415f9d0 0x7f009415fe60 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f00940b25f0 tx=0x7f008002b910 comp rx=0 tx=0).stop 2024-03-06T07:34:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 -- 172.21.15.114:0/1501981724 shutdown_connections 2024-03-06T07:34:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 --2- 172.21.15.114:0/1501981724 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f00940ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 --2- 172.21.15.114:0/1501981724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f009415f9d0 0x7f009415fe60 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 --2- 172.21.15.114:0/1501981724 >> v2:172.21.15.177:3300/0 conn(0x7f0094144c00 0x7f009406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 -- 172.21.15.114:0/1501981724 >> 172.21.15.114:0/1501981724 conn(0x7f00940b1730 msgr2=0x7f00940b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 -- 172.21.15.114:0/1501981724 shutdown_connections 2024-03-06T07:34:27.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.312+0000 7f009b800640 1 -- 172.21.15.114:0/1501981724 wait complete. 2024-03-06T07:34:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.314+0000 7f009b800640 1 Processor -- start 2024-03-06T07:34:27.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.314+0000 7f009b800640 1 -- start start 2024-03-06T07:34:27.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f009b800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f0094138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f009b800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0094144c00 0x7f00940719d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f009b800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009415f9d0 0x7f0094071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f009b800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009406cb50 con 0x7f0094144c00 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f009b800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0094160730 con 0x7f009415f9d0 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f009b800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009406ccd0 con 0x7f00940ce0a0 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0094144c00 0x7f00940719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0094144c00 0x7f00940719d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38712/0 (socket says 172.21.15.114:38712) 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 -- 172.21.15.114:0/1966229501 learned_addr learned my addr 172.21.15.114:0/1966229501 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0098e00640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f0094138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 -- 172.21.15.114:0/1966229501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 msgr2=0x7f0094138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f0094138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 -- 172.21.15.114:0/1966229501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009415f9d0 msgr2=0x7f0094071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009415f9d0 0x7f0094071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.315+0000 7f0093e00640 1 -- 172.21.15.114:0/1966229501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00941e62d0 con 0x7f0094144c00 2024-03-06T07:34:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.316+0000 7f0093e00640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0094144c00 0x7f00940719d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f008002b950 tx=0x7f00800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.316+0000 7f0090c00640 1 -- 172.21.15.114:0/1966229501 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080040070 con 0x7f0094144c00 2024-03-06T07:34:27.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.316+0000 7f0090c00640 1 -- 172.21.15.114:0/1966229501 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0080030d00 con 0x7f0094144c00 2024-03-06T07:34:27.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.316+0000 7f0090c00640 1 -- 172.21.15.114:0/1966229501 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080035040 con 0x7f0094144c00 2024-03-06T07:34:27.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.316+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00941e65b0 con 0x7f0094144c00 2024-03-06T07:34:27.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.316+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00941e6b70 con 0x7f0094144c00 2024-03-06T07:34:27.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.318+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009404f330 con 0x7f0094144c00 2024-03-06T07:34:27.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.318+0000 7f0090c00640 1 -- 172.21.15.114:0/1966229501 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f008004b060 con 0x7f0094144c00 2024-03-06T07:34:27.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.319+0000 7f0090c00640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007406b4d0 0x7f007406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.319+0000 7f0090c00640 1 -- 172.21.15.114:0/1966229501 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f00800ac160 con 0x7f0094144c00 2024-03-06T07:34:27.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.320+0000 7f0098e00640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007406b4d0 0x7f007406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.320+0000 7f0098e00640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007406b4d0 0x7f007406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0088008850 tx=0x7f0088008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.324+0000 7f0090c00640 1 -- 172.21.15.114:0/1966229501 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f008007ae90 con 0x7f0094144c00 2024-03-06T07:34:27.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.642+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f00940af9b0 con 0x7f0094144c00 2024-03-06T07:34:27.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.643+0000 7f0090c00640 1 -- 172.21.15.114:0/1966229501 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v87) v1 ==== 53+0+73 (secure 0 0 0) 0x7f008007a790 con 0x7f0094144c00 2024-03-06T07:34:27.646 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:34:27.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.645+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007406b4d0 msgr2=0x7f007406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.645+0000 7f009b800640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007406b4d0 0x7f007406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0088008850 tx=0x7f0088008040 comp rx=0 tx=0).stop 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0094144c00 msgr2=0x7f00940719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0094144c00 0x7f00940719d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f008002b950 tx=0x7f00800046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 shutdown_connections 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f009415f9d0 0x7f0094071f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f007406b4d0 0x7f007406d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0094144c00 0x7f00940719d0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 --2- 172.21.15.114:0/1966229501 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f00940ce0a0 0x7f0094138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 >> 172.21.15.114:0/1966229501 conn(0x7f00940b1730 msgr2=0x7f00940ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 shutdown_connections 2024-03-06T07:34:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:27.646+0000 7f009b800640 1 -- 172.21.15.114:0/1966229501 wait complete. 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:34:27.667 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:27.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:27.668 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:34:27.709 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:27.710 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42188sec preferred_lft 42188sec 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:27.711 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:34:27.712 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:27.712 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:27.712 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:27.712 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:34:27.712 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:27.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:27.712 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:34:27.835 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:34:27.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:27.839 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:34:27.883 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:34:27.883 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:34:27.884 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:34:27.884 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:34:27.884 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:34:27.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:27.884 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:34:27.884 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:34:27.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:34:27.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:34:27.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:34:27.884 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:34:28.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T07:34:28.071+0000 7f4c28800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:34:28.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:28.095 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:34:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:28.142 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:28.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:28.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:34:28.282 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:28.283 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:28.333 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:28.333 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:28.333 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:28.381 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:34:28.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:28.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:28.550 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:28.550 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:28.583 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:28.583 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:28.583 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:34:28.583 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:34:28.583 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:34:28.586 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:34:28.586 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:34:28.586 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:34:28.586 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:34:28.586 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:34:28.591 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:34:28.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:28.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:28.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:28.664 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:34:28.715 INFO:teuthology.orchestra.run.smithi114.stdout:client6175 2024-03-06T07:34:28.715 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:34:28.715 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:28.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:28.760 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:34:28.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:29.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.058+0000 7eff47600640 1 Processor -- start 2024-03-06T07:34:29.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.058+0000 7eff47600640 1 -- start start 2024-03-06T07:34:29.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.060+0000 7eff47600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7eff4006ec00 0x7eff400ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.060+0000 7eff47600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4015b090 con 0x7eff4015acb0 2024-03-06T07:34:29.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.060+0000 7eff47600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff400b2a30 con 0x7eff400b4680 2024-03-06T07:34:29.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.060+0000 7eff47600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff400b4360 con 0x7eff4006ec00 2024-03-06T07:34:29.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.060+0000 7eff44c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7eff4015acb0 0x7eff400b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:60576/0 (socket says 172.21.15.114:60576) 2024-03-06T07:34:29.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.060+0000 7eff44c00640 1 -- 172.21.15.114:0/1409361491 learned_addr learned my addr 172.21.15.114:0/1409361491 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff45600640 1 --2- 172.21.15.114:0/1409361491 >> v2:172.21.15.177:3300/0 conn(0x7eff4006ec00 0x7eff400ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139768389 0 0) 0x7eff4015b090 con 0x7eff4015acb0 2024-03-06T07:34:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff28005730 con 0x7eff4015acb0 2024-03-06T07:34:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4035444916 0 0) 0x7eff28005730 con 0x7eff4015acb0 2024-03-06T07:34:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 >> v2:172.21.15.177:3300/0 conn(0x7eff4006ec00 msgr2=0x7eff400ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 --2- 172.21.15.114:0/1409361491 >> v2:172.21.15.177:3300/0 conn(0x7eff4006ec00 0x7eff400ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 >> v1:172.21.15.177:6789/0 conn(0x7eff400b4680 legacy=0x7eff4006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff400b2cf0 con 0x7eff4015acb0 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7eff34001670 con 0x7eff4015acb0 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 >> v1:172.21.15.114:6789/0 conn(0x7eff4015acb0 legacy=0x7eff400b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff28005c10 0x7eff28006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff28006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff280073d0 0x7eff2800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff400b4360 con 0x7eff28006a10 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2800e2d0 con 0x7eff28005c10 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.061+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2800e4d0 con 0x7eff280073d0 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff3fe00640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff28006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff45600640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff280073d0 0x7eff2800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff3fe00640 1 -- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff280073d0 msgr2=0x7eff2800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff3fe00640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff280073d0 0x7eff2800dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff3fe00640 1 -- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff28005c10 msgr2=0x7eff28006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff44c00640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff28005c10 0x7eff28006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff3fe00640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff28005c10 0x7eff28006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff45600640 1 -- 172.21.15.114:0/1409361491 reap_dead start 2024-03-06T07:34:29.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff3fe00640 1 -- 172.21.15.114:0/1409361491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff28005730 con 0x7eff28006a10 2024-03-06T07:34:29.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.062+0000 7eff44c00640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff28005c10 0x7eff28006030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.063+0000 7eff3fe00640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff28006e90 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001750 tx=0x7eff28007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f2170037c0534c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.063+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c001370 con 0x7eff28006a10 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.063+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c0052d0 con 0x7eff28006a10 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.063+0000 7eff3f400640 1 -- 172.21.15.114:0/1409361491 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c003830 con 0x7eff28006a10 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.064+0000 7eff47600640 1 -- 172.21.15.114:0/1409361491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 msgr2=0x7eff28006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.064+0000 7eff47600640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff28006e90 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001750 tx=0x7eff28007a40 comp rx=0 tx=0).stop 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.064+0000 7eff47600640 1 -- 172.21.15.114:0/1409361491 shutdown_connections 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.064+0000 7eff47600640 1 --2- 172.21.15.114:0/1409361491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff28006e90 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.064+0000 7eff47600640 1 -- 172.21.15.114:0/1409361491 >> 172.21.15.114:0/1409361491 conn(0x7eff400b1730 msgr2=0x7eff400b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.064+0000 7eff47600640 1 -- 172.21.15.114:0/1409361491 shutdown_connections 2024-03-06T07:34:29.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.064+0000 7eff47600640 1 -- 172.21.15.114:0/1409361491 wait complete. 2024-03-06T07:34:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.067+0000 7eff47600640 1 Processor -- start 2024-03-06T07:34:29.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.067+0000 7eff47600640 1 -- start start 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff47600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff401f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff47600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff401f6c40 0x7eff40155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff47600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40156130 0x7eff401f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff47600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff400b4010 con 0x7eff28006a10 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff47600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff400b32b0 con 0x7eff40156130 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff47600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff40072340 con 0x7eff401f6c40 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff44c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff401f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff44c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff401f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42744/0 (socket says 172.21.15.114:42744) 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.069+0000 7eff44c00640 1 -- 172.21.15.114:0/742589394 learned_addr learned my addr 172.21.15.114:0/742589394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff44c00640 1 -- 172.21.15.114:0/742589394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff401f6c40 msgr2=0x7eff40155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff44c00640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff401f6c40 0x7eff40155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff44c00640 1 -- 172.21.15.114:0/742589394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40156130 msgr2=0x7eff401f3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff44c00640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40156130 0x7eff401f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff44c00640 1 -- 172.21.15.114:0/742589394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff401f49d0 con 0x7eff28006a10 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff44c00640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff401f4f70 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7eff3400d7a0 tx=0x7eff3400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff3cc00640 1 -- 172.21.15.114:0/742589394 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34019070 con 0x7eff28006a10 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff3cc00640 1 -- 172.21.15.114:0/742589394 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff340072d0 con 0x7eff28006a10 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff3cc00640 1 -- 172.21.15.114:0/742589394 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff340047e0 con 0x7eff28006a10 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.070+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff400adb10 con 0x7eff28006a10 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.071+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff400cd220 con 0x7eff28006a10 2024-03-06T07:34:29.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.072+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff40156530 con 0x7eff28006a10 2024-03-06T07:34:29.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.073+0000 7eff3cc00640 1 -- 172.21.15.114:0/742589394 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff34004980 con 0x7eff28006a10 2024-03-06T07:34:29.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.073+0000 7eff3cc00640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7eff2006b4d0 0x7eff2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.074+0000 7eff3cc00640 1 -- 172.21.15.114:0/742589394 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7eff34089a50 con 0x7eff28006a10 2024-03-06T07:34:29.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.074+0000 7eff3fe00640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7eff2006b4d0 0x7eff2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.074+0000 7eff3fe00640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7eff2006b4d0 0x7eff2006d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7eff2c033000 tx=0x7eff2c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.082+0000 7eff3cc00640 1 -- 172.21.15.114:0/742589394 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff34058820 con 0x7eff28006a10 2024-03-06T07:34:29.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.322+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff4004f330 con 0x7eff28006a10 2024-03-06T07:34:29.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.323+0000 7eff3cc00640 1 -- 172.21.15.114:0/742589394 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7eff34058120 con 0x7eff28006a10 2024-03-06T07:34:29.328 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:29.328 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7eff2006b4d0 msgr2=0x7eff2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7eff2006b4d0 0x7eff2006d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7eff2c033000 tx=0x7eff2c02f000 comp rx=0 tx=0).stop 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 msgr2=0x7eff401f4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff401f4f70 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7eff3400d7a0 tx=0x7eff3400dc60 comp rx=0 tx=0).stop 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 shutdown_connections 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff40156130 0x7eff401f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7eff401f6c40 0x7eff40155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7eff2006b4d0 0x7eff2006d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 --2- 172.21.15.114:0/742589394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff28006a10 0x7eff401f4f70 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 >> 172.21.15.114:0/742589394 conn(0x7eff400b1730 msgr2=0x7eff40071540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 shutdown_connections 2024-03-06T07:34:29.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.326+0000 7eff47600640 1 -- 172.21.15.114:0/742589394 wait complete. 2024-03-06T07:34:29.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:34:29.586 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:34:29.587 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:34:29.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.776+0000 7fc938800640 1 Processor -- start 2024-03-06T07:34:29.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.776+0000 7fc938800640 1 -- start start 2024-03-06T07:34:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.777+0000 7fc938800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc93015acb0 0x7fc9300b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.777+0000 7fc938800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 0x7fc9300ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.777+0000 7fc938800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93015b090 con 0x7fc9300b4680 2024-03-06T07:34:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.777+0000 7fc938800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9300b2a30 con 0x7fc93015acb0 2024-03-06T07:34:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.777+0000 7fc938800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300b4360 con 0x7fc93006ec00 2024-03-06T07:34:29.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.778+0000 7fc935400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc9300b4680 0x7fc93006e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:60590/0 (socket says 172.21.15.114:60590) 2024-03-06T07:34:29.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.778+0000 7fc935400640 1 -- 172.21.15.114:0/287459243 learned_addr learned my addr 172.21.15.114:0/287459243 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:34:29.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.778+0000 7fc936800640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 0x7fc9300ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.778+0000 7fc935e00640 1 --2- 172.21.15.114:0/287459243 >> v2:172.21.15.177:3300/0 conn(0x7fc93015acb0 0x7fc9300b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.778+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563143963 0 0) 0x7fc93015b090 con 0x7fc9300b4680 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc918003520 con 0x7fc9300b4680 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3922639363 0 0) 0x7fc918003520 con 0x7fc9300b4680 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 msgr2=0x7fc9300ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 0x7fc9300ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 >> v2:172.21.15.177:3300/0 conn(0x7fc93015acb0 msgr2=0x7fc9300b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 --2- 172.21.15.114:0/287459243 >> v2:172.21.15.177:3300/0 conn(0x7fc93015acb0 0x7fc9300b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9300b2cf0 con 0x7fc9300b4680 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fc920001670 con 0x7fc9300b4680 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 >> v1:172.21.15.114:6789/0 conn(0x7fc9300b4680 legacy=0x7fc93006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc918003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 0x7fc918004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc918004d40 0x7fc9180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300b2a30 con 0x7fc9180039b0 2024-03-06T07:34:29.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300b4360 con 0x7fc918004d40 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91800b9c0 con 0x7fc93006ec00 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.779+0000 7fc935e00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc918003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.780+0000 7fc935400640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 0x7fc918004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.780+0000 7fc935e00640 1 -- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 msgr2=0x7fc918004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.780+0000 7fc935e00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93006ec00 0x7fc918004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.780+0000 7fc935e00640 1 -- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc918004d40 msgr2=0x7fc9180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.780+0000 7fc935e00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc918004d40 0x7fc9180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.780+0000 7fc935e00640 1 -- 172.21.15.114:0/287459243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc918003520 con 0x7fc9180039b0 2024-03-06T07:34:29.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.781+0000 7fc935e00640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc918003dd0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc924009af0 tx=0x7fc918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6a944596b02304 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.781+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924039070 con 0x7fc9180039b0 2024-03-06T07:34:29.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.781+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92402c710 con 0x7fc9180039b0 2024-03-06T07:34:29.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.781+0000 7fc934a00640 1 -- 172.21.15.114:0/287459243 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924009690 con 0x7fc9180039b0 2024-03-06T07:34:29.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.782+0000 7fc938800640 1 -- 172.21.15.114:0/287459243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 msgr2=0x7fc918003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.782+0000 7fc938800640 1 --2- 172.21.15.114:0/287459243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc918003dd0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fc924009af0 tx=0x7fc918005830 comp rx=0 tx=0).stop 2024-03-06T07:34:29.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.782+0000 7fc936800640 1 -- 172.21.15.114:0/287459243 reap_dead start 2024-03-06T07:34:29.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.782+0000 7fc938800640 1 -- 172.21.15.114:0/287459243 shutdown_connections 2024-03-06T07:34:29.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.782+0000 7fc938800640 1 -- 172.21.15.114:0/287459243 >> 172.21.15.114:0/287459243 conn(0x7fc9300b1730 msgr2=0x7fc9300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.782+0000 7fc938800640 1 -- 172.21.15.114:0/287459243 shutdown_connections 2024-03-06T07:34:29.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.782+0000 7fc938800640 1 -- 172.21.15.114:0/287459243 wait complete. 2024-03-06T07:34:29.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.784+0000 7fc938800640 1 Processor -- start 2024-03-06T07:34:29.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.784+0000 7fc938800640 1 -- start start 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc938800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc93015cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc938800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9301686e0 0x7fc930146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc938800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93014bf80 0x7fc93014c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc938800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300b3d10 con 0x7fc9180039b0 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc938800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300b32b0 con 0x7fc9301686e0 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc938800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300b4010 con 0x7fc93014bf80 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc93015cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc93015cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42764/0 (socket says 172.21.15.114:42764) 2024-03-06T07:34:29.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 -- 172.21.15.114:0/2891545595 learned_addr learned my addr 172.21.15.114:0/2891545595 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:34:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 -- 172.21.15.114:0/2891545595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93014bf80 msgr2=0x7fc93014c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93014bf80 0x7fc93014c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 -- 172.21.15.114:0/2891545595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9301686e0 msgr2=0x7fc930146440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9301686e0 0x7fc930146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.786+0000 7fc935e00640 1 -- 172.21.15.114:0/2891545595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9300c8f30 con 0x7fc9180039b0 2024-03-06T07:34:29.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.787+0000 7fc935e00640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc93015cb10 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fc924037040 tx=0x7fc924004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.787+0000 7fc92a000640 1 -- 172.21.15.114:0/2891545595 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924039050 con 0x7fc9180039b0 2024-03-06T07:34:29.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.787+0000 7fc92a000640 1 -- 172.21.15.114:0/2891545595 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc924045070 con 0x7fc9180039b0 2024-03-06T07:34:29.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.787+0000 7fc92a000640 1 -- 172.21.15.114:0/2891545595 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924003a50 con 0x7fc9180039b0 2024-03-06T07:34:29.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.787+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc930168b60 con 0x7fc9180039b0 2024-03-06T07:34:29.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.787+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9300c91f0 con 0x7fc9180039b0 2024-03-06T07:34:29.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.789+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93004f330 con 0x7fc9180039b0 2024-03-06T07:34:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.790+0000 7fc92a000640 1 -- 172.21.15.114:0/2891545595 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc924003bf0 con 0x7fc9180039b0 2024-03-06T07:34:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.790+0000 7fc92a000640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc91406b500 0x7fc91406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.791+0000 7fc92a000640 1 -- 172.21.15.114:0/2891545595 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fc9240ac2b0 con 0x7fc9180039b0 2024-03-06T07:34:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.791+0000 7fc935400640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc91406b500 0x7fc91406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.791+0000 7fc935400640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc91406b500 0x7fc91406d9c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc920000f60 tx=0x7fc920000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:29.795+0000 7fc92a000640 1 -- 172.21.15.114:0/2891545595 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc92407b110 con 0x7fc9180039b0 2024-03-06T07:34:30.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.030+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc93006ce60 con 0x7fc9180039b0 2024-03-06T07:34:30.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.032+0000 7fc92a000640 1 -- 172.21.15.114:0/2891545595 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fc92407aa10 con 0x7fc9180039b0 2024-03-06T07:34:30.034 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:34:30.034 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:34:30.035 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:34:30.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc91406b500 msgr2=0x7fc91406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc91406b500 0x7fc91406d9c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc920000f60 tx=0x7fc920000ed0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 msgr2=0x7fc93015cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc93015cb10 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fc924037040 tx=0x7fc924004730 comp rx=0 tx=0).stop 2024-03-06T07:34:30.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 shutdown_connections 2024-03-06T07:34:30.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc93014bf80 0x7fc93014c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9301686e0 0x7fc930146440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc91406b500 0x7fc91406d9c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 --2- 172.21.15.114:0/2891545595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9180039b0 0x7fc93015cb10 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.034+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 >> 172.21.15.114:0/2891545595 conn(0x7fc9300b1730 msgr2=0x7fc9300ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.035+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 shutdown_connections 2024-03-06T07:34:30.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:34:30.035+0000 7fc938800640 1 -- 172.21.15.114:0/2891545595 wait complete. 2024-03-06T07:34:30.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:34:30.278 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:34:30.278 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:34:46.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:46.490 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:00.279 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:35:00.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.405+0000 7ff09bc0ff40 1 Processor -- start 2024-03-06T07:35:00.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.405+0000 7ff09bc0ff40 1 -- start start 2024-03-06T07:35:00.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff09bc0ff40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3021e5f0 0x559c302da530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff09bc0ff40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c30269b30 0x559c302f1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff09bc0ff40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c302f5250 0x559c3024dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff09bc0ff40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c30222560 con 0x559c30269b30 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff09bc0ff40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c301c6fd0 con 0x559c3021e5f0 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff09bc0ff40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c30222800 con 0x559c302f5250 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff097800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c30269b30 0x559c302f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff097800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c30269b30 0x559c302f1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34500/0 (socket says 172.21.15.114:34500) 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff097800640 1 -- 172.21.15.114:0/3133174563 learned_addr learned my addr 172.21.15.114:0/3133174563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff098c00640 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c302f5250 0x559c3024dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.406+0000 7ff098200640 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3021e5f0 0x559c302da530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff097800640 1 -- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c302f5250 msgr2=0x559c3024dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff097800640 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c302f5250 0x559c3024dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff097800640 1 -- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3021e5f0 msgr2=0x559c302da530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff097800640 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3021e5f0 0x559c302da530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff097800640 1 -- 172.21.15.114:0/3133174563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c3024fed0 con 0x559c30269b30 2024-03-06T07:35:00.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff098c00640 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c302f5250 0x559c3024dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:00.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff098200640 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3021e5f0 0x559c302da530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:00.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff097800640 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c30269b30 0x559c302f1710 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff0840098b0 tx=0x7ff084005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d318c768b33786ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff096400640 1 -- 172.21.15.114:0/3133174563 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084008930 con 0x559c30269b30 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff096400640 1 -- 172.21.15.114:0/3133174563 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff084018070 con 0x559c30269b30 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.407+0000 7ff096400640 1 -- 172.21.15.114:0/3133174563 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084008cd0 con 0x559c30269b30 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/3133174563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c30269b30 msgr2=0x559c302f1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c30269b30 0x559c302f1710 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7ff0840098b0 tx=0x7ff084005a00 comp rx=0 tx=0).stop 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/3133174563 shutdown_connections 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c302f5250 0x559c3024dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c30269b30 0x559c302f1710 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/3133174563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c3021e5f0 0x559c302da530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/3133174563 >> 172.21.15.114:0/3133174563 conn(0x559c3024dbe0 msgr2=0x559c302ec160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/3133174563 shutdown_connections 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.408+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/3133174563 wait complete. 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.409+0000 7ff09bc0ff40 1 Processor -- start 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.409+0000 7ff09bc0ff40 1 -- start start 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff09bc0ff40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c3021e5f0 0x559c302d7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff09bc0ff40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c30269b30 0x559c302e5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff09bc0ff40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c302f5250 0x559c302f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff09bc0ff40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c30250170 con 0x559c3021e5f0 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff09bc0ff40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c302daa70 con 0x559c302f5250 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff09bc0ff40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c302f11f0 con 0x559c30269b30 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff098200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c3021e5f0 0x559c302d7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff098200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c3021e5f0 0x559c302d7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34514/0 (socket says 172.21.15.114:34514) 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.410+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 learned_addr learned my addr 172.21.15.114:0/575701694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c30269b30 0x559c302e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c30269b30 msgr2=0x559c302e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559c30269b30 0x559c302e5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c302f5250 msgr2=0x559c302f19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559c302f5250 0x559c302f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c3024f8d0 con 0x559c3021e5f0 2024-03-06T07:35:00.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c3021e5f0 0x559c302d7200 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff08801cf40 tx=0x7ff088002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.411+0000 7ff095000640 1 -- 172.21.15.114:0/575701694 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088061020 con 0x559c3021e5f0 2024-03-06T07:35:00.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.412+0000 7ff095000640 1 -- 172.21.15.114:0/575701694 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0880529f0 con 0x559c3021e5f0 2024-03-06T07:35:00.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.412+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559c302e8cf0 con 0x559c3021e5f0 2024-03-06T07:35:00.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.412+0000 7ff095000640 1 -- 172.21.15.114:0/575701694 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff088057040 con 0x559c3021e5f0 2024-03-06T07:35:00.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.412+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c302e92b0 con 0x559c3021e5f0 2024-03-06T07:35:00.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.414+0000 7ff095000640 1 -- 172.21.15.114:0/575701694 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff088064020 con 0x559c3021e5f0 2024-03-06T07:35:00.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.415+0000 7ff095000640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff078069eb0 0x7ff07806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.415+0000 7ff095000640 1 -- 172.21.15.114:0/575701694 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff0880ca930 con 0x559c3021e5f0 2024-03-06T07:35:00.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.415+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff078069eb0 0x7ff07806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.415+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559c302f1ee0 0x559c302e9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.415+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.416+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff078069eb0 0x7ff07806c370 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x559c302f24c0 tx=0x7ff08400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.416+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559c302f1ee0 0x559c302e9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.417+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559c302f1ee0 0x559c302e9d90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.418+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.418+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7ff090003c30 0x7ff09000f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.418+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090003c30 2024-03-06T07:35:00.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.418+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7ff090003c30 0x7ff09000f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.419+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7ff090003c30 0x7ff09000f240 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.419+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.1 v2:172.21.15.114:6816/1955604845 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff090003c30 2024-03-06T07:35:00.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.419+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7ff088066020 0x7ff088097f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.419+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.420+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7ff088066020 0x7ff088097f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.420+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7ff088066020 0x7ff088097f10 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.421+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.421+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7ff08401f040 0x7ff084013420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.421+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08401f040 2024-03-06T07:35:00.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.422+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7ff08401f040 0x7ff084013420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.423+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7ff08401f040 0x7ff084013420 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.423+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.5 v2:172.21.15.177:6816/2170499622 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff08401f040 2024-03-06T07:35:00.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.423+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.424+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.424+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.425+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.425+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7ff08402d440 0x7ff08402d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.425+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.425+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7ff08402d440 0x7ff08402d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.426+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7ff08402d440 0x7ff08402d8c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.426+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 1 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.426+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.427+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.427+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.428+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 2 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.428+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.428+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.428+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.429+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.429+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.429+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 3 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.429+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.430+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.430+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7ff090019450 0x7ff0900198d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.430+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.431+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7ff090019450 0x7ff0900198d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.432+0000 7ff097800640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7ff090019450 0x7ff0900198d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.433+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.433+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090003c30 2024-03-06T07:35:00.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.433+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.1 v2:172.21.15.114:6816/1955604845 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff090003c30 2024-03-06T07:35:00.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.433+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.434+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 4 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.434+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.435+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.435+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08401f040 2024-03-06T07:35:00.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.436+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.5 v2:172.21.15.177:6816/2170499622 2 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff08401f040 2024-03-06T07:35:00.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.436+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.436+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.436+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.437+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.437+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.438+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.438+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.438+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.438+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.439+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 5 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.439+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090003c30 2024-03-06T07:35:00.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.439+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.1 v2:172.21.15.114:6816/1955604845 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff090003c30 2024-03-06T07:35:00.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.439+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7ff0880992b0 0x7ff0880ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.439+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff0880992b0 2024-03-06T07:35:00.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.440+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7ff0880992b0 0x7ff0880ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.441+0000 7ff098c00640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7ff0880992b0 0x7ff0880ca200 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.442+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.6 v2:172.21.15.177:6808/4041662010 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff0880992b0 2024-03-06T07:35:00.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.442+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.442+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 6 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.442+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08401f040 2024-03-06T07:35:00.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.443+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.5 v2:172.21.15.177:6816/2170499622 3 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff08401f040 2024-03-06T07:35:00.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.443+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.444+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 7 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.444+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7ff0880a12d0 0x7ff088097610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.444+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff0880a12d0 2024-03-06T07:35:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.444+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7ff0880a12d0 0x7ff088097610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.445+0000 7ff098200640 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7ff0880a12d0 0x7ff088097610 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.445+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.2 v2:172.21.15.114:6808/1832993461 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff0880a12d0 2024-03-06T07:35:00.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.445+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff0880992b0 2024-03-06T07:35:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.446+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.6 v2:172.21.15.177:6808/4041662010 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff0880992b0 2024-03-06T07:35:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.446+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090003c30 2024-03-06T07:35:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.447+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.1 v2:172.21.15.114:6816/1955604845 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff090003c30 2024-03-06T07:35:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.447+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.448+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.448+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.448+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.448+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.449+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.449+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.449+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.449+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090003c30 2024-03-06T07:35:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.450+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.1 v2:172.21.15.114:6816/1955604845 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff090003c30 2024-03-06T07:35:00.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.450+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.450+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.450+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.451+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.451+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08401f040 2024-03-06T07:35:00.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.452+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.5 v2:172.21.15.177:6816/2170499622 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff08401f040 2024-03-06T07:35:00.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.452+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08401f040 2024-03-06T07:35:00.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.453+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.5 v2:172.21.15.177:6816/2170499622 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff08401f040 2024-03-06T07:35:00.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.453+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff0880992b0 2024-03-06T07:35:00.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.454+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.6 v2:172.21.15.177:6808/4041662010 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff0880992b0 2024-03-06T07:35:00.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.454+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090003c30 2024-03-06T07:35:00.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.455+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.1 v2:172.21.15.114:6816/1955604845 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff090003c30 2024-03-06T07:35:00.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.455+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08401f040 2024-03-06T07:35:00.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.456+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.5 v2:172.21.15.177:6816/2170499622 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff08401f040 2024-03-06T07:35:00.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.456+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.457+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.457+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.458+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.458+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.459+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.459+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.460+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.460+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.461+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.462+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.463+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.463+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.464+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.464+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff088066020 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.464+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.3 v2:172.21.15.114:6818/735520392 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff088066020 2024-03-06T07:35:00.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.464+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x559c302f1ee0 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.465+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.7 v2:172.21.15.177:6824/1593615297 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x559c302f1ee0 2024-03-06T07:35:00.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.465+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.466+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.466+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.467+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.467+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.468+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.468+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff0880a12d0 2024-03-06T07:35:00.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.469+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.2 v2:172.21.15.114:6808/1832993461 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff0880a12d0 2024-03-06T07:35:00.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.469+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.470+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 11 ==== osd_op_reply(57 [pgnls start_epoch 114 out=81b] v108'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.470+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.471+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.471+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff0880992b0 2024-03-06T07:35:00.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.472+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.6 v2:172.21.15.177:6808/4041662010 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff0880992b0 2024-03-06T07:35:00.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.472+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.473+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 12 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.473+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090003c30 2024-03-06T07:35:00.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.473+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.1 v2:172.21.15.114:6816/1955604845 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff090003c30 2024-03-06T07:35:00.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.473+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff090019450 2024-03-06T07:35:00.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.474+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 <== osd.4 v2:172.21.15.177:6800/4111454897 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=81b] v108'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ff084008c70 con 0x7ff090019450 2024-03-06T07:35:00.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.474+0000 7ff097800640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff0880992b0 2024-03-06T07:35:00.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.475+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 <== osd.6 v2:172.21.15.177:6808/4041662010 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff0900014a0 con 0x7ff0880992b0 2024-03-06T07:35:00.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.475+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559c302ea2d0 con 0x7ff08402d440 2024-03-06T07:35:00.478 INFO:teuthology.orchestra.run.smithi114.stdout:10000000001.00000000 2024-03-06T07:35:00.478 INFO:teuthology.orchestra.run.smithi114.stdout:10000000001.00000001 2024-03-06T07:35:00.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff098200640 1 -- 172.21.15.114:0/575701694 <== osd.0 v2:172.21.15.114:6800/3895229755 13 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff08805a040 con 0x7ff08402d440 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7ff08402d440 msgr2=0x7ff08402d8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7ff08402d440 0x7ff08402d8c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7ff090003c30 msgr2=0x7ff09000f240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7ff090003c30 0x7ff09000f240 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7ff0880a12d0 msgr2=0x7ff088097610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7ff0880a12d0 0x7ff088097610 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7ff088066020 msgr2=0x7ff088097f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7ff088066020 0x7ff088097f10 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7ff090019450 msgr2=0x7ff0900198d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7ff090019450 0x7ff0900198d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7ff08401f040 msgr2=0x7ff084013420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7ff08401f040 0x7ff084013420 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7ff0880992b0 msgr2=0x7ff0880ca200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7ff0880992b0 0x7ff0880ca200 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 reap_dead start 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559c302f1ee0 msgr2=0x559c302e9d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559c302f1ee0 0x559c302e9d90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 reap_dead start 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff098c00640 1 -- 172.21.15.114:0/575701694 reap_dead start 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff078069eb0 msgr2=0x7ff07806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff078069eb0 0x7ff07806c370 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x559c302f24c0 tx=0x7ff08400b040 comp rx=0 tx=0).stop 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c3021e5f0 msgr2=0x559c302d7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.476+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c3021e5f0 0x559c302d7200 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7ff08801cf40 tx=0x7ff088002fb0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.477+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 shutdown_connections 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.477+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff078069eb0 0x7ff07806c370 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.477+0000 7ff09bc0ff40 1 --2- 172.21.15.114:0/575701694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559c3021e5f0 0x559c302d7200 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.477+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 >> 172.21.15.114:0/575701694 conn(0x559c3024dbe0 msgr2=0x559c30221470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.477+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 shutdown_connections 2024-03-06T07:35:00.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.477+0000 7ff09bc0ff40 1 -- 172.21.15.114:0/575701694 wait complete. 2024-03-06T07:35:00.486 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:35:00.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:00.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.682+0000 7fb461600640 1 Processor -- start 2024-03-06T07:35:00.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.683+0000 7fb461600640 1 -- start start 2024-03-06T07:35:00.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.684+0000 7fb461600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb45c15acb0 0x7fb45c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.684+0000 7fb461600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c15b090 con 0x7fb45c06ec00 2024-03-06T07:35:00.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.684+0000 7fb461600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0b2a30 con 0x7fb45c0b4680 2024-03-06T07:35:00.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.684+0000 7fb461600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45c0b4360 con 0x7fb45c15acb0 2024-03-06T07:35:00.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb45aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb45c15acb0 0x7fb45c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb45b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb45c06ec00 0x7fb45c0ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59950/0 (socket says 172.21.15.114:59950) 2024-03-06T07:35:00.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb45b400640 1 -- 172.21.15.114:0/521253073 learned_addr learned my addr 172.21.15.114:0/521253073 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:00.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb459600640 1 -- 172.21.15.114:0/521253073 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166565836 0 0) 0x7fb45c15b090 con 0x7fb45c06ec00 2024-03-06T07:35:00.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb459600640 1 -- 172.21.15.114:0/521253073 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb43c003520 con 0x7fb45c06ec00 2024-03-06T07:35:00.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb45aa00640 1 -- 172.21.15.114:0/521253073 >> v1:172.21.15.177:6789/0 conn(0x7fb45c0b4680 legacy=0x7fb45c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb45aa00640 1 -- 172.21.15.114:0/521253073 >> v1:172.21.15.114:6789/0 conn(0x7fb45c06ec00 legacy=0x7fb45c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.685+0000 7fb45aa00640 1 -- 172.21.15.114:0/521253073 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0b2cf0 con 0x7fb45c15acb0 2024-03-06T07:35:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb45aa00640 1 --2- 172.21.15.114:0/521253073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c0b37d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ba40 tx=0x7fb44c02e130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac0c33d1fa4bba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb459600640 1 -- 172.21.15.114:0/521253073 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c03b070 con 0x7fb45c15acb0 2024-03-06T07:35:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb459600640 1 -- 172.21.15.114:0/521253073 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c036440 con 0x7fb45c15acb0 2024-03-06T07:35:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb459600640 1 -- 172.21.15.114:0/521253073 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44c0367f0 con 0x7fb45c15acb0 2024-03-06T07:35:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb461600640 1 -- 172.21.15.114:0/521253073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 msgr2=0x7fb45c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb461600640 1 --2- 172.21.15.114:0/521253073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c0b37d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00ba40 tx=0x7fb44c02e130 comp rx=0 tx=0).stop 2024-03-06T07:35:00.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb461600640 1 -- 172.21.15.114:0/521253073 shutdown_connections 2024-03-06T07:35:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb461600640 1 --2- 172.21.15.114:0/521253073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c0b37d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.687+0000 7fb461600640 1 -- 172.21.15.114:0/521253073 >> 172.21.15.114:0/521253073 conn(0x7fb45c0b1730 msgr2=0x7fb45c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.688+0000 7fb461600640 1 -- 172.21.15.114:0/521253073 shutdown_connections 2024-03-06T07:35:00.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.688+0000 7fb461600640 1 -- 172.21.15.114:0/521253073 wait complete. 2024-03-06T07:35:00.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.690+0000 7fb461600640 1 Processor -- start 2024-03-06T07:35:00.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.691+0000 7fb461600640 1 -- start start 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb461600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb45c06ec00 0x7fb45c159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb461600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb45c0b4680 0x7fb45c06d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb461600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb461600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c0b3d10 con 0x7fb45c15acb0 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb461600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c0b32b0 con 0x7fb45c0b4680 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb461600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c0b4010 con 0x7fb45c06ec00 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb45b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb45b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34532/0 (socket says 172.21.15.114:34532) 2024-03-06T07:35:00.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.692+0000 7fb45b400640 1 -- 172.21.15.114:0/3464455822 learned_addr learned my addr 172.21.15.114:0/3464455822 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45aa00640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb45c06ec00 0x7fb45c159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45b400640 1 -- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb45c06ec00 msgr2=0x7fb45c159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45b400640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb45c06ec00 0x7fb45c159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45a000640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb45c0b4680 0x7fb45c06d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45b400640 1 -- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb45c0b4680 msgr2=0x7fb45c06d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45b400640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb45c0b4680 0x7fb45c06d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45b400640 1 -- 172.21.15.114:0/3464455822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c15cfd0 con 0x7fb45c15acb0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb45b400640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c157270 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb45400c390 tx=0x7fb45400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb452a00640 1 -- 172.21.15.114:0/3464455822 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454017070 con 0x7fb45c15acb0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb452a00640 1 -- 172.21.15.114:0/3464455822 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb454005ce0 con 0x7fb45c15acb0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb452a00640 1 -- 172.21.15.114:0/3464455822 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454007310 con 0x7fb45c15acb0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.693+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c15d2b0 con 0x7fb45c15acb0 2024-03-06T07:35:00.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.694+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c15d870 con 0x7fb45c15acb0 2024-03-06T07:35:00.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.695+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c0b2040 con 0x7fb45c15acb0 2024-03-06T07:35:00.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.696+0000 7fb452a00640 1 -- 172.21.15.114:0/3464455822 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb45401a070 con 0x7fb45c15acb0 2024-03-06T07:35:00.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.697+0000 7fb452a00640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb43806b4d0 0x7fb43806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:00.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.697+0000 7fb452a00640 1 -- 172.21.15.114:0/3464455822 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb4540897b0 con 0x7fb45c15acb0 2024-03-06T07:35:00.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.698+0000 7fb45aa00640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb43806b4d0 0x7fb43806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:00.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.698+0000 7fb45aa00640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb43806b4d0 0x7fb43806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb44c038000 tx=0x7fb44c032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:00.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.701+0000 7fb452a00640 1 -- 172.21.15.114:0/3464455822 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb45c0b2040 con 0x7fb45c15acb0 2024-03-06T07:35:00.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.952+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb45c04f330 con 0x7fb45c15acb0 2024-03-06T07:35:00.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.953+0000 7fb452a00640 1 -- 172.21.15.114:0/3464455822 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fb4540584a0 con 0x7fb45c15acb0 2024-03-06T07:35:00.955 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:00.955 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:00.956 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:00.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb43806b4d0 msgr2=0x7fb43806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb43806b4d0 0x7fb43806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb44c038000 tx=0x7fb44c032000 comp rx=0 tx=0).stop 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 msgr2=0x7fb45c157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c157270 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb45400c390 tx=0x7fb45400c850 comp rx=0 tx=0).stop 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 shutdown_connections 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb43806b4d0 0x7fb43806d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb45c15acb0 0x7fb45c157270 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb45c0b4680 0x7fb45c06d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 --2- 172.21.15.114:0/3464455822 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb45c06ec00 0x7fb45c159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.956+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 >> 172.21.15.114:0/3464455822 conn(0x7fb45c0b1730 msgr2=0x7fb45c071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.957+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 shutdown_connections 2024-03-06T07:35:00.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:00.957+0000 7fb461600640 1 -- 172.21.15.114:0/3464455822 wait complete. 2024-03-06T07:35:00.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:01.195 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:35:01.195 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:35:01.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:01.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.396+0000 7f6dd4800640 1 Processor -- start 2024-03-06T07:35:01.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.396+0000 7f6dd4800640 1 -- start start 2024-03-06T07:35:01.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.398+0000 7f6dd4800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6dcc144c00 0x7f6dcc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.398+0000 7f6dd4800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc06ee70 con 0x7f6dcc15fe60 2024-03-06T07:35:01.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.398+0000 7f6dd4800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc06cef0 con 0x7f6dcc15f9d0 2024-03-06T07:35:01.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.398+0000 7f6dd4800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dcc06d1d0 con 0x7f6dcc144c00 2024-03-06T07:35:01.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.398+0000 7f6dd2800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6dcc15fe60 0x7f6dcc0ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59962/0 (socket says 172.21.15.114:59962) 2024-03-06T07:35:01.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.398+0000 7f6dd2800640 1 -- 172.21.15.114:0/3001188171 learned_addr learned my addr 172.21.15.114:0/3001188171 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:01.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.398+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87483227 0 0) 0x7f6dc8001490 con 0x7f6dcc15fe60 2024-03-06T07:35:01.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db4003520 con 0x7f6dcc15fe60 2024-03-06T07:35:01.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd1e00640 1 --2- 172.21.15.114:0/3001188171 >> v2:172.21.15.177:3300/0 conn(0x7f6dcc144c00 0x7f6dcc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751298294 0 0) 0x7f6db4003520 con 0x7f6dcc15fe60 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 >> v2:172.21.15.177:3300/0 conn(0x7f6dcc144c00 msgr2=0x7f6dcc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 --2- 172.21.15.114:0/3001188171 >> v2:172.21.15.177:3300/0 conn(0x7f6dcc144c00 0x7f6dcc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 >> v1:172.21.15.177:6789/0 conn(0x7f6dcc15f9d0 legacy=0x7f6dcc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc06d4b0 con 0x7f6dcc15fe60 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6dc8001bc0 con 0x7f6dcc15fe60 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 >> v1:172.21.15.114:6789/0 conn(0x7f6dcc15fe60 legacy=0x7f6dcc0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6db40039b0 0x7f6db4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6db4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6db40051c0 0x7f6db400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc06d1d0 con 0x7f6db4004800 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db400c110 con 0x7f6db40039b0 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db400c310 con 0x7f6db40051c0 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.399+0000 7f6dd1400640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6db4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd1e00640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6db40039b0 0x7f6db4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd1400640 1 -- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6db40051c0 msgr2=0x7f6db400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd2800640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6db40051c0 0x7f6db400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd1400640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6db40051c0 0x7f6db400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd1400640 1 -- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6db40039b0 msgr2=0x7f6db4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd1400640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6db40039b0 0x7f6db4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd1400640 1 -- 172.21.15.114:0/3001188171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db4003520 con 0x7f6db4004800 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd1e00640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6db40039b0 0x7f6db4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.400+0000 7f6dd2800640 1 -- 172.21.15.114:0/3001188171 reap_dead start 2024-03-06T07:35:01.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.401+0000 7f6dd1400640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6db4004c80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc001750 tx=0x7f6db4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2fe5dea87d01cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.401+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc001370 con 0x7f6db4004800 2024-03-06T07:35:01.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.401+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dbc0052d0 con 0x7f6db4004800 2024-03-06T07:35:01.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.401+0000 7f6dd0a00640 1 -- 172.21.15.114:0/3001188171 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc003830 con 0x7f6db4004800 2024-03-06T07:35:01.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.402+0000 7f6dd4800640 1 -- 172.21.15.114:0/3001188171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 msgr2=0x7f6db4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.402+0000 7f6dd4800640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6db4004c80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc001750 tx=0x7f6db4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:01.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.402+0000 7f6dd4800640 1 -- 172.21.15.114:0/3001188171 shutdown_connections 2024-03-06T07:35:01.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.402+0000 7f6dd4800640 1 --2- 172.21.15.114:0/3001188171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6db4004c80 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.402+0000 7f6dd4800640 1 -- 172.21.15.114:0/3001188171 >> 172.21.15.114:0/3001188171 conn(0x7f6dcc0b1730 msgr2=0x7f6dcc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.402+0000 7f6dd4800640 1 -- 172.21.15.114:0/3001188171 shutdown_connections 2024-03-06T07:35:01.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.402+0000 7f6dd4800640 1 -- 172.21.15.114:0/3001188171 wait complete. 2024-03-06T07:35:01.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.404+0000 7f6dd4800640 1 Processor -- start 2024-03-06T07:35:01.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.404+0000 7f6dd4800640 1 -- start start 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd4800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6dcc1f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd4800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc15dda0 0x7f6dcc15e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd4800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dcc13cdc0 0x7f6dcc13d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd4800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc160730 con 0x7f6db4004800 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd4800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc160270 con 0x7f6dcc15dda0 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd4800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc06cb50 con 0x7f6dcc13cdc0 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6dcc1f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6dcc1f3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34550/0 (socket says 172.21.15.114:34550) 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 -- 172.21.15.114:0/4203294034 learned_addr learned my addr 172.21.15.114:0/4203294034 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1400640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc15dda0 0x7f6dcc15e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd2800640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dcc13cdc0 0x7f6dcc13d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 -- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dcc13cdc0 msgr2=0x7f6dcc13d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dcc13cdc0 0x7f6dcc13d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 -- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc15dda0 msgr2=0x7f6dcc15e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc15dda0 0x7f6dcc15e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.405+0000 7f6dd1e00640 1 -- 172.21.15.114:0/4203294034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc15e860 con 0x7f6db4004800 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.406+0000 7f6dd1e00640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6dcc1f3e20 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000e540 tx=0x7f6dc000ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.406+0000 7f6dc6000640 1 -- 172.21.15.114:0/4203294034 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0017070 con 0x7f6db4004800 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.406+0000 7f6dc6000640 1 -- 172.21.15.114:0/4203294034 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc0013610 con 0x7f6db4004800 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.406+0000 7f6dc6000640 1 -- 172.21.15.114:0/4203294034 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000f040 con 0x7f6db4004800 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.406+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc15eb40 con 0x7f6db4004800 2024-03-06T07:35:01.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.406+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc1412e0 con 0x7f6db4004800 2024-03-06T07:35:01.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.407+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc04f330 con 0x7f6db4004800 2024-03-06T07:35:01.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.408+0000 7f6dc6000640 1 -- 172.21.15.114:0/4203294034 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6dc001a070 con 0x7f6db4004800 2024-03-06T07:35:01.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.409+0000 7f6dc6000640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b500 0x7f6db006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:01.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.409+0000 7f6dc6000640 1 -- 172.21.15.114:0/4203294034 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f6dc0089f30 con 0x7f6db4004800 2024-03-06T07:35:01.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.409+0000 7f6dd1400640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b500 0x7f6db006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:01.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.410+0000 7f6dd1400640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b500 0x7f6db006d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc033000 tx=0x7f6dbc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:01.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.413+0000 7f6dc6000640 1 -- 172.21.15.114:0/4203294034 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6dc0058b90 con 0x7f6db4004800 2024-03-06T07:35:01.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.646+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dcc15ed10 con 0x7f6db4004800 2024-03-06T07:35:01.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.647+0000 7f6dc6000640 1 -- 172.21.15.114:0/4203294034 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f6dc0058490 con 0x7f6db4004800 2024-03-06T07:35:01.649 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:01.649 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:01.649 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b500 msgr2=0x7f6db006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b500 0x7f6db006d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc033000 tx=0x7f6dbc009030 comp rx=0 tx=0).stop 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 msgr2=0x7f6dcc1f3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6dcc1f3e20 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000e540 tx=0x7f6dc000ea00 comp rx=0 tx=0).stop 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 shutdown_connections 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6dcc13cdc0 0x7f6dcc13d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc15dda0 0x7f6dcc15e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6db006b500 0x7f6db006d9c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 --2- 172.21.15.114:0/4203294034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6db4004800 0x7f6dcc1f3e20 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 >> 172.21.15.114:0/4203294034 conn(0x7f6dcc0b1730 msgr2=0x7f6dcc0719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 shutdown_connections 2024-03-06T07:35:01.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:01.651+0000 7f6dd4800640 1 -- 172.21.15.114:0/4203294034 wait complete. 2024-03-06T07:35:01.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:35:01.918 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:01.918 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:35:02.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.007+0000 7fc01bc35f40 1 Processor -- start 2024-03-06T07:35:02.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.007+0000 7fc01bc35f40 1 -- start start 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc01bc35f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf049f990 0x55adf05730b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc01bc35f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049fd70 0x55adf055f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc01bc35f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf055d9b0 0x55adf0573c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc01bc35f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf049ed10 con 0x55adf055d9b0 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc01bc35f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf0445fd0 con 0x55adf049f990 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc01bc35f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55adf049efb0 con 0x55adf049fd70 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc019000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf055d9b0 0x55adf0573c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc019000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf055d9b0 0x55adf0573c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34578/0 (socket says 172.21.15.114:34578) 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.008+0000 7fc019000640 1 -- 172.21.15.114:0/3304907452 learned_addr learned my addr 172.21.15.114:0/3304907452 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc018600640 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf049f990 0x55adf05730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc019000640 1 -- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049fd70 msgr2=0x55adf055f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc00fe00640 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049fd70 0x55adf055f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc019000640 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049fd70 0x55adf055f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc019000640 1 -- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf049f990 msgr2=0x55adf05730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc019000640 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf049f990 0x55adf05730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc019000640 1 -- 172.21.15.114:0/3304907452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adf055ef90 con 0x55adf055d9b0 2024-03-06T07:35:02.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc019000640 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf055d9b0 0x55adf0573c30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc01000a4e0 tx=0x7fc01000a9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253966dca23b3ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc017200640 1 -- 172.21.15.114:0/3304907452 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000f7b0 con 0x55adf055d9b0 2024-03-06T07:35:02.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc017200640 1 -- 172.21.15.114:0/3304907452 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc010015070 con 0x55adf055d9b0 2024-03-06T07:35:02.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.009+0000 7fc017200640 1 -- 172.21.15.114:0/3304907452 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000fb70 con 0x55adf055d9b0 2024-03-06T07:35:02.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 -- 172.21.15.114:0/3304907452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf055d9b0 msgr2=0x55adf0573c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf055d9b0 0x55adf0573c30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc01000a4e0 tx=0x7fc01000a9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 -- 172.21.15.114:0/3304907452 shutdown_connections 2024-03-06T07:35:02.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf055d9b0 0x55adf0573c30 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049fd70 0x55adf055f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/3304907452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf049f990 0x55adf05730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 -- 172.21.15.114:0/3304907452 >> 172.21.15.114:0/3304907452 conn(0x55adf04cec50 msgr2=0x55adf04d10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 -- 172.21.15.114:0/3304907452 shutdown_connections 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.010+0000 7fc01bc35f40 1 -- 172.21.15.114:0/3304907452 wait complete. 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.011+0000 7fc01bc35f40 1 Processor -- start 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.011+0000 7fc01bc35f40 1 -- start start 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc01bc35f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049f990 0x55adf05709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc01bc35f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf049fd70 0x55adf04d0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc01bc35f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf055d9b0 0x55adf056f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc01bc35f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf0573670 con 0x55adf049fd70 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc01bc35f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf05738c0 con 0x55adf055d9b0 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc01bc35f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x55adf04d1870 con 0x55adf049f990 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc00fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf049fd70 0x55adf04d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc00fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf049fd70 0x55adf04d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34594/0 (socket says 172.21.15.114:34594) 2024-03-06T07:35:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.012+0000 7fc00fe00640 1 -- 172.21.15.114:0/7604581 learned_addr learned my addr 172.21.15.114:0/7604581 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc018600640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049f990 0x55adf05709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc00fe00640 1 -- 172.21.15.114:0/7604581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049f990 msgr2=0x55adf05709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc00fe00640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55adf049f990 0x55adf05709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc00fe00640 1 -- 172.21.15.114:0/7604581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf055d9b0 msgr2=0x55adf056f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc00fe00640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adf055d9b0 0x55adf056f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc00fe00640 1 -- 172.21.15.114:0/7604581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adf04cdaf0 con 0x55adf049fd70 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc00fe00640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf049fd70 0x55adf04d0530 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc0000094d0 tx=0x7fc000009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc015e00640 1 -- 172.21.15.114:0/7604581 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000005900 con 0x55adf049fd70 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc015e00640 1 -- 172.21.15.114:0/7604581 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc000005aa0 con 0x55adf049fd70 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55adf0575e10 con 0x55adf049fd70 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc015e00640 1 -- 172.21.15.114:0/7604581 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000014440 con 0x55adf049fd70 2024-03-06T07:35:02.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.013+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55adf04cd810 con 0x55adf049fd70 2024-03-06T07:35:02.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.015+0000 7fc015e00640 1 -- 172.21.15.114:0/7604581 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc00001d440 con 0x55adf049fd70 2024-03-06T07:35:02.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.016+0000 7fc015e00640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbff4069eb0 0x7fbff406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.016+0000 7fc015e00640 1 -- 172.21.15.114:0/7604581 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc0000893d0 con 0x55adf049fd70 2024-03-06T07:35:02.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.017+0000 7fc018600640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbff4069eb0 0x7fbff406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.017+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x55adf05704d0 0x55adf04e2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.017+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x55adf04d0a70 con 0x55adf05704d0 2024-03-06T07:35:02.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.017+0000 7fc019000640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x55adf05704d0 0x55adf04e2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.017+0000 7fc018600640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbff4069eb0 0x7fbff406c370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc008054000 tx=0x7fc00804e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc019000640 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x55adf05704d0 0x55adf04e2ad0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc019000640 1 -- 172.21.15.114:0/7604581 <== osd.0 v2:172.21.15.114:6800/3895229755 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7fc01000e070 con 0x55adf05704d0 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x55adf05704d0 msgr2=0x55adf04e2ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x55adf05704d0 0x55adf04e2ad0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbff4069eb0 msgr2=0x7fbff406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbff4069eb0 0x7fbff406c370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc008054000 tx=0x7fc00804e000 comp rx=0 tx=0).stop 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf049fd70 msgr2=0x55adf04d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 --2- 172.21.15.114:0/7604581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55adf049fd70 0x55adf04d0530 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc0000094d0 tx=0x7fc000009990 comp rx=0 tx=0).stop 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc019000640 1 -- 172.21.15.114:0/7604581 reap_dead start 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 shutdown_connections 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 >> 172.21.15.114:0/7604581 conn(0x55adf04cec50 msgr2=0x55adf0571690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 shutdown_connections 2024-03-06T07:35:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.018+0000 7fc01bc35f40 1 -- 172.21.15.114:0/7604581 wait complete. 2024-03-06T07:35:02.027 INFO:teuthology.orchestra.run.smithi114.stdout:] 10000000001 stray0 2024-03-06T07:35:02.028 DEBUG:teuthology.orchestra.run.smithi114:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 1099511627777, 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "ancestors": [ 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 1536, 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "10000000001", 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 13 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 256, 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "stray0", 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 5 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:02.116 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-03-06T07:35:02.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 17, 2024-03-06T07:35:02.117 INFO:teuthology.orchestra.run.smithi114.stdout: "old_pools": [] 2024-03-06T07:35:02.117 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:35:02.117 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:02.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:02.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.315+0000 7f1eccc00640 1 Processor -- start 2024-03-06T07:35:02.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.315+0000 7f1eccc00640 1 -- start start 2024-03-06T07:35:02.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.316+0000 7f1eccc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ec815e0d0 0x7f1ec80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.316+0000 7f1eccc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1ec815f470 0x7f1ec80d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.316+0000 7f1eccc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.316+0000 7f1eccc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec80d2f60 con 0x7f1ec815e0d0 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1eccc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec80d1240 con 0x7f1ec815f470 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1eccc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80cea60 con 0x7f1ec80d0df0 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ec815e0d0 0x7f1ec80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ec815e0d0 0x7f1ec80d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34602/0 (socket says 172.21.15.114:34602) 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 -- 172.21.15.114:0/255167905 learned_addr learned my addr 172.21.15.114:0/255167905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec6800640 1 --2- 172.21.15.114:0/255167905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5400640 1 --2- 172.21.15.114:0/255167905 >> v2:172.21.15.177:3300/0 conn(0x7f1ec815f470 0x7f1ec80d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 -- 172.21.15.114:0/255167905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 msgr2=0x7f1ec8160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 --2- 172.21.15.114:0/255167905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8160dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 -- 172.21.15.114:0/255167905 >> v2:172.21.15.177:3300/0 conn(0x7f1ec815f470 msgr2=0x7f1ec80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 --2- 172.21.15.114:0/255167905 >> v2:172.21.15.177:3300/0 conn(0x7f1ec815f470 0x7f1ec80d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.317+0000 7f1ec5e00640 1 -- 172.21.15.114:0/255167905 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80d2350 con 0x7f1ec815e0d0 2024-03-06T07:35:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.318+0000 7f1ec5e00640 1 --2- 172.21.15.114:0/255167905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815e0d0 0x7f1ec80d1a90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80c77f0 tx=0x7f1eb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767657db4d2c79b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1ec4a00640 1 -- 172.21.15.114:0/255167905 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8034070 con 0x7f1ec815e0d0 2024-03-06T07:35:02.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1ec4a00640 1 -- 172.21.15.114:0/255167905 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8030860 con 0x7f1ec815e0d0 2024-03-06T07:35:02.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1ec4a00640 1 -- 172.21.15.114:0/255167905 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8030c30 con 0x7f1ec815e0d0 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 -- 172.21.15.114:0/255167905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815e0d0 msgr2=0x7f1ec80d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 --2- 172.21.15.114:0/255167905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815e0d0 0x7f1ec80d1a90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80c77f0 tx=0x7f1eb802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 -- 172.21.15.114:0/255167905 shutdown_connections 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 --2- 172.21.15.114:0/255167905 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 --2- 172.21.15.114:0/255167905 >> v2:172.21.15.177:3300/0 conn(0x7f1ec815f470 0x7f1ec80d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 --2- 172.21.15.114:0/255167905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815e0d0 0x7f1ec80d1a90 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 -- 172.21.15.114:0/255167905 >> 172.21.15.114:0/255167905 conn(0x7f1ec80d3ec0 msgr2=0x7f1ec80d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.319+0000 7f1eccc00640 1 -- 172.21.15.114:0/255167905 shutdown_connections 2024-03-06T07:35:02.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.320+0000 7f1eccc00640 1 -- 172.21.15.114:0/255167905 wait complete. 2024-03-06T07:35:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.321+0000 7f1eccc00640 1 Processor -- start 2024-03-06T07:35:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.321+0000 7f1eccc00640 1 -- start start 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.322+0000 7f1eccc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.322+0000 7f1eccc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec815e0d0 0x7f1ec80caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.322+0000 7f1eccc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815f470 0x7f1ec80c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.322+0000 7f1eccc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80d2bf0 con 0x7f1ec815f470 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.322+0000 7f1eccc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80d2910 con 0x7f1ec815e0d0 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.322+0000 7f1eccc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec815fb30 con 0x7f1ec80d0df0 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815f470 0x7f1ec80c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815f470 0x7f1ec80c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34606/0 (socket says 172.21.15.114:34606) 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 -- 172.21.15.114:0/1583289868 learned_addr learned my addr 172.21.15.114:0/1583289868 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec5400640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec815e0d0 0x7f1ec80caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec5e00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 -- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 msgr2=0x7f1ec8155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 -- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec815e0d0 msgr2=0x7f1ec80caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec815e0d0 0x7f1ec80caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.323+0000 7f1ec6800640 1 -- 172.21.15.114:0/1583289868 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80cb9e0 con 0x7f1ec815f470 2024-03-06T07:35:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.324+0000 7f1ec6800640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815f470 0x7f1ec80c8e70 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000cc30 tx=0x7f1ec000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.324+0000 7f1ebe000640 1 -- 172.21.15.114:0/1583289868 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000ce40 con 0x7f1ec815f470 2024-03-06T07:35:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.324+0000 7f1ebe000640 1 -- 172.21.15.114:0/1583289868 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec0003990 con 0x7f1ec815f470 2024-03-06T07:35:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.324+0000 7f1ebe000640 1 -- 172.21.15.114:0/1583289868 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0003d40 con 0x7f1ec815f470 2024-03-06T07:35:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.324+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec80cbcc0 con 0x7f1ec815f470 2024-03-06T07:35:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.324+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec80cc280 con 0x7f1ec815f470 2024-03-06T07:35:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.325+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec804f330 con 0x7f1ec815f470 2024-03-06T07:35:02.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.326+0000 7f1ebe000640 1 -- 172.21.15.114:0/1583289868 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ec0003ee0 con 0x7f1ec815f470 2024-03-06T07:35:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.327+0000 7f1ebe000640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ea806b4d0 0x7f1ea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.327+0000 7f1ebe000640 1 -- 172.21.15.114:0/1583289868 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1ec008a8c0 con 0x7f1ec815f470 2024-03-06T07:35:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.327+0000 7f1ec5e00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ea806b4d0 0x7f1ea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.328+0000 7f1ec5e00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ea806b4d0 0x7f1ea806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8036000 tx=0x7f1eb803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.331+0000 7f1ebe000640 1 -- 172.21.15.114:0/1583289868 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ec0014030 con 0x7f1ec815f470 2024-03-06T07:35:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.568+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ec808d920 con 0x7f1ec815f470 2024-03-06T07:35:02.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.569+0000 7f1ebe000640 1 -- 172.21.15.114:0/1583289868 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f1ec0059520 con 0x7f1ec815f470 2024-03-06T07:35:02.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:02.571 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:02.571 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ea806b4d0 msgr2=0x7f1ea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ea806b4d0 0x7f1ea806d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8036000 tx=0x7f1eb803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815f470 msgr2=0x7f1ec80c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815f470 0x7f1ec80c8e70 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000cc30 tx=0x7f1ec000a5c0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 shutdown_connections 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1ea806b4d0 0x7f1ea806d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec815f470 0x7f1ec80c8e70 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec815e0d0 0x7f1ec80caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 --2- 172.21.15.114:0/1583289868 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1ec80d0df0 0x7f1ec8155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 >> 172.21.15.114:0/1583289868 conn(0x7f1ec80d3ec0 msgr2=0x7f1ec80d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 shutdown_connections 2024-03-06T07:35:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:02.572+0000 7f1eccc00640 1 -- 172.21.15.114:0/1583289868 wait complete. 2024-03-06T07:35:02.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:35:02.808 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:35:02.809 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:35:02.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:02.809 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:35:02.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.067+0000 7f7470800640 1 Processor -- start 2024-03-06T07:35:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.067+0000 7f7470800640 1 -- start start 2024-03-06T07:35:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.069+0000 7f7470800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 0x7f7468071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.069+0000 7f7470800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f74680cf3e0 0x7f74680724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.069+0000 7f7470800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74680cabf0 con 0x7f74680c9ec0 2024-03-06T07:35:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.069+0000 7f7470800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74680ce900 con 0x7f74680cf3e0 2024-03-06T07:35:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.069+0000 7f7470800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74680cebe0 con 0x7f7468144660 2024-03-06T07:35:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.069+0000 7f746e800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f74680c9ec0 0x7f74680ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59964/0 (socket says 172.21.15.114:59964) 2024-03-06T07:35:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.069+0000 7f746e800640 1 -- 172.21.15.114:0/3166462613 learned_addr learned my addr 172.21.15.114:0/3166462613 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746de00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 0x7f7468071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746d400640 1 --2- 172.21.15.114:0/3166462613 >> v2:172.21.15.177:3300/0 conn(0x7f74680cf3e0 0x7f74680724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596409768 0 0) 0x7f74680cabf0 con 0x7f74680c9ec0 2024-03-06T07:35:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7450003520 con 0x7f74680c9ec0 2024-03-06T07:35:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419424965 0 0) 0x7f7450003520 con 0x7f74680c9ec0 2024-03-06T07:35:03.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 msgr2=0x7f7468071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 0x7f7468071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 >> v2:172.21.15.177:3300/0 conn(0x7f74680cf3e0 msgr2=0x7f74680724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 --2- 172.21.15.114:0/3166462613 >> v2:172.21.15.177:3300/0 conn(0x7f74680cf3e0 0x7f74680724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74680ceec0 con 0x7f74680c9ec0 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f74640019f0 con 0x7f74680c9ec0 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 >> v1:172.21.15.114:6789/0 conn(0x7f74680c9ec0 legacy=0x7f74680ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f7450003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7450004800 0x7f7450004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 0x7f74500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74680ce900 con 0x7f74500039b0 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74680cebe0 con 0x7f7450004800 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.070+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745000b9c0 con 0x7f7468144660 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746de00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f7450003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746d400640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7450004800 0x7f7450004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746e800640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 0x7f74500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746de00640 1 -- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 msgr2=0x7f74500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746de00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7468144660 0x7f74500051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746de00640 1 -- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7450004800 msgr2=0x7f7450004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746de00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7450004800 0x7f7450004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746de00640 1 -- 172.21.15.114:0/3166462613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7450003520 con 0x7f74500039b0 2024-03-06T07:35:03.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.071+0000 7f746d400640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7450004800 0x7f7450004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f746de00640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f7450003dd0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f7454009ac0 tx=0x7f7450005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959597de49f726cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454039070 con 0x7f74500039b0 2024-03-06T07:35:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745402c6b0 con 0x7f74500039b0 2024-03-06T07:35:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f746ca00640 1 -- 172.21.15.114:0/3166462613 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454009690 con 0x7f74500039b0 2024-03-06T07:35:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f7470800640 1 -- 172.21.15.114:0/3166462613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 msgr2=0x7f7450003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f7470800640 1 --2- 172.21.15.114:0/3166462613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f7450003dd0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f7454009ac0 tx=0x7f7450005830 comp rx=0 tx=0).stop 2024-03-06T07:35:03.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f746e800640 1 -- 172.21.15.114:0/3166462613 reap_dead start 2024-03-06T07:35:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f7470800640 1 -- 172.21.15.114:0/3166462613 shutdown_connections 2024-03-06T07:35:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.073+0000 7f7470800640 1 -- 172.21.15.114:0/3166462613 >> 172.21.15.114:0/3166462613 conn(0x7f74680d0b90 msgr2=0x7f74680d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.074+0000 7f7470800640 1 -- 172.21.15.114:0/3166462613 shutdown_connections 2024-03-06T07:35:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.074+0000 7f7470800640 1 -- 172.21.15.114:0/3166462613 wait complete. 2024-03-06T07:35:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.075+0000 7f7470800640 1 Processor -- start 2024-03-06T07:35:03.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.075+0000 7f7470800640 1 -- start start 2024-03-06T07:35:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.076+0000 7f7470800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f746813e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.076+0000 7f7470800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f746806f570 0x7f74681606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.076+0000 7f7470800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f746806f9f0 0x7f746815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.076+0000 7f7470800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74680720f0 con 0x7f74500039b0 2024-03-06T07:35:03.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.076+0000 7f7470800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7468071f70 con 0x7f746806f570 2024-03-06T07:35:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.076+0000 7f7470800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7468072270 con 0x7f746806f9f0 2024-03-06T07:35:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f746813e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f746813e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34630/0 (socket says 172.21.15.114:34630) 2024-03-06T07:35:03.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 -- 172.21.15.114:0/3489992829 learned_addr learned my addr 172.21.15.114:0/3489992829 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 -- 172.21.15.114:0/3489992829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f746806f9f0 msgr2=0x7f746815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f746806f9f0 0x7f746815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 -- 172.21.15.114:0/3489992829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f746806f570 msgr2=0x7f74681606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746d400640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f746806f570 0x7f74681606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f746806f570 0x7f74681606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.077+0000 7f746de00640 1 -- 172.21.15.114:0/3489992829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746813deb0 con 0x7f74500039b0 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.078+0000 7f746de00640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f746813e950 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f7454009ac0 tx=0x7f7454004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.078+0000 7f7462000640 1 -- 172.21.15.114:0/3489992829 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454039050 con 0x7f74500039b0 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.078+0000 7f7462000640 1 -- 172.21.15.114:0/3489992829 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7454045070 con 0x7f74500039b0 2024-03-06T07:35:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.078+0000 7f7462000640 1 -- 172.21.15.114:0/3489992829 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454003a50 con 0x7f74500039b0 2024-03-06T07:35:03.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.078+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7468161130 con 0x7f74500039b0 2024-03-06T07:35:03.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.078+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746813e150 con 0x7f74500039b0 2024-03-06T07:35:03.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.079+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746804f330 con 0x7f74500039b0 2024-03-06T07:35:03.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.080+0000 7f7462000640 1 -- 172.21.15.114:0/3489992829 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7454003bf0 con 0x7f74500039b0 2024-03-06T07:35:03.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.081+0000 7f7462000640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f744806b4d0 0x7f744806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.081+0000 7f7462000640 1 -- 172.21.15.114:0/3489992829 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f74540abef0 con 0x7f74500039b0 2024-03-06T07:35:03.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.081+0000 7f746d400640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f744806b4d0 0x7f744806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.082+0000 7f746d400640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f744806b4d0 0x7f744806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7468143700 tx=0x7f745c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.085+0000 7f7462000640 1 -- 172.21.15.114:0/3489992829 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f745404e020 con 0x7f74500039b0 2024-03-06T07:35:03.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.375+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74681619a0 con 0x7f74500039b0 2024-03-06T07:35:03.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.376+0000 7f7462000640 1 -- 172.21.15.114:0/3489992829 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f745407adb0 con 0x7f74500039b0 2024-03-06T07:35:03.378 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:03.378 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:03.378 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f744806b4d0 msgr2=0x7f744806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f744806b4d0 0x7f744806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7468143700 tx=0x7f745c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 msgr2=0x7f746813e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f746813e950 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f7454009ac0 tx=0x7f7454004730 comp rx=0 tx=0).stop 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 shutdown_connections 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f746806f9f0 0x7f746815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f746806f570 0x7f74681606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f744806b4d0 0x7f744806d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 --2- 172.21.15.114:0/3489992829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74500039b0 0x7f746813e950 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 >> 172.21.15.114:0/3489992829 conn(0x7f74680d0b90 msgr2=0x7f7468143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 shutdown_connections 2024-03-06T07:35:03.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.379+0000 7f7470800640 1 -- 172.21.15.114:0/3489992829 wait complete. 2024-03-06T07:35:03.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:03.616 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:35:03.616 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:35:03.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:03.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.824+0000 7f93fd800640 1 Processor -- start 2024-03-06T07:35:03.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.824+0000 7f93fd800640 1 -- start start 2024-03-06T07:35:03.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.825+0000 7f93fd800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f93f80cff20 0x7f93f80b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.825+0000 7f93fd800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 0x7f93f80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.825+0000 7f93fd800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80b5e00 con 0x7f93f80b7860 2024-03-06T07:35:03.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.825+0000 7f93fd800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f80b6a10 con 0x7f93f80cff20 2024-03-06T07:35:03.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.825+0000 7f93fd800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80b60e0 con 0x7f93f80b73e0 2024-03-06T07:35:03.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.826+0000 7f93f7400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f93f80b7860 0x7f93f80d0840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59976/0 (socket says 172.21.15.114:59976) 2024-03-06T07:35:03.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.826+0000 7f93f7400640 1 -- 172.21.15.114:0/1041006287 learned_addr learned my addr 172.21.15.114:0/1041006287 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:03.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.826+0000 7f93f6a00640 1 --2- 172.21.15.114:0/1041006287 >> v2:172.21.15.177:3300/0 conn(0x7f93f80cff20 0x7f93f80b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.826+0000 7f93f6000640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 0x7f93f80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.826+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922384379 0 0) 0x7f93f80b5e00 con 0x7f93f80b7860 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.826+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93dc003520 con 0x7f93f80b7860 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567847830 0 0) 0x7f93dc003520 con 0x7f93f80b7860 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 msgr2=0x7f93f80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 0x7f93f80d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 >> v2:172.21.15.177:3300/0 conn(0x7f93f80cff20 msgr2=0x7f93f80b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 --2- 172.21.15.114:0/1041006287 >> v2:172.21.15.177:3300/0 conn(0x7f93f80cff20 0x7f93f80b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80b63c0 con 0x7f93f80b7860 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f93f0001670 con 0x7f93f80b7860 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 >> v1:172.21.15.114:6789/0 conn(0x7f93f80b7860 legacy=0x7f93f80d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 0x7f93dc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93dc003f30 0x7f93dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc004d40 0x7f93dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80b6a10 con 0x7f93dc003f30 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80b60e0 con 0x7f93dc004d40 2024-03-06T07:35:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc00b9c0 con 0x7f93f80b73e0 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6000640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93dc003f30 0x7f93dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f7400640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc004d40 0x7f93dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6a00640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 0x7f93dc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6000640 1 -- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 msgr2=0x7f93dc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6000640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 0x7f93dc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6000640 1 -- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc004d40 msgr2=0x7f93dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6000640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc004d40 0x7f93dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6000640 1 -- 172.21.15.114:0/1041006287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc003520 con 0x7f93dc003f30 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f7400640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc004d40 0x7f93dc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.827+0000 7f93f6a00640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80b73e0 0x7f93dc0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.828+0000 7f93f6000640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93dc003f30 0x7f93dc004800 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009ac0 tx=0x7f93dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0a39c96c892ae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0039070 con 0x7f93dc003f30 2024-03-06T07:35:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e002c6b0 con 0x7f93dc003f30 2024-03-06T07:35:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93f5600640 1 -- 172.21.15.114:0/1041006287 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0009690 con 0x7f93dc003f30 2024-03-06T07:35:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93fd800640 1 -- 172.21.15.114:0/1041006287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93dc003f30 msgr2=0x7f93dc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93fd800640 1 --2- 172.21.15.114:0/1041006287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93dc003f30 0x7f93dc004800 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009ac0 tx=0x7f93dc005830 comp rx=0 tx=0).stop 2024-03-06T07:35:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93f7400640 1 -- 172.21.15.114:0/1041006287 reap_dead start 2024-03-06T07:35:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93fd800640 1 -- 172.21.15.114:0/1041006287 shutdown_connections 2024-03-06T07:35:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93fd800640 1 -- 172.21.15.114:0/1041006287 >> 172.21.15.114:0/1041006287 conn(0x7f93f80d1c90 msgr2=0x7f93f80d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.829+0000 7f93fd800640 1 -- 172.21.15.114:0/1041006287 shutdown_connections 2024-03-06T07:35:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.830+0000 7f93fd800640 1 -- 172.21.15.114:0/1041006287 wait complete. 2024-03-06T07:35:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.831+0000 7f93fd800640 1 Processor -- start 2024-03-06T07:35:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.831+0000 7f93fd800640 1 -- start start 2024-03-06T07:35:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.832+0000 7f93fd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc003f30 0x7f93f806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.832+0000 7f93fd800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80c8d60 0x7f93f814bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.832+0000 7f93fd800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f80c7940 0x7f93f80c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.832+0000 7f93fd800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f8161a50 con 0x7f93f80c7940 2024-03-06T07:35:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.832+0000 7f93fd800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80b5af0 con 0x7f93dc003f30 2024-03-06T07:35:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.832+0000 7f93fd800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80d0f50 con 0x7f93f80c8d60 2024-03-06T07:35:03.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f80c7940 0x7f93f80c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f80c7940 0x7f93f80c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34646/0 (socket says 172.21.15.114:34646) 2024-03-06T07:35:03.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 -- 172.21.15.114:0/3410537535 learned_addr learned my addr 172.21.15.114:0/3410537535 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f6a00640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc003f30 0x7f93f806fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f6000640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80c8d60 0x7f93f814bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 -- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80c8d60 msgr2=0x7f93f814bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80c8d60 0x7f93f814bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 -- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc003f30 msgr2=0x7f93f806fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc003f30 0x7f93f806fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 -- 172.21.15.114:0/3410537535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80c9420 con 0x7f93f80c7940 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f6000640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80c8d60 0x7f93f814bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.833+0000 7f93f7400640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f80c7940 0x7f93f80c7dc0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f93f8071f90 tx=0x7f93f000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.834+0000 7f93eea00640 1 -- 172.21.15.114:0/3410537535 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0007230 con 0x7f93f80c7940 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.834+0000 7f93eea00640 1 -- 172.21.15.114:0/3410537535 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f00073f0 con 0x7f93f80c7940 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.834+0000 7f93eea00640 1 -- 172.21.15.114:0/3410537535 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00047e0 con 0x7f93f80c7940 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.834+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f80c9700 con 0x7f93f80c7940 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.834+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f80c9cc0 con 0x7f93f80c7940 2024-03-06T07:35:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.835+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f80c8300 con 0x7f93f80c7940 2024-03-06T07:35:03.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.837+0000 7f93eea00640 1 -- 172.21.15.114:0/3410537535 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93f001b020 con 0x7f93f80c7940 2024-03-06T07:35:03.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.838+0000 7f93eea00640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f93d406b500 0x7f93d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.838+0000 7f93eea00640 1 -- 172.21.15.114:0/3410537535 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f93f0089e00 con 0x7f93f80c7940 2024-03-06T07:35:03.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.838+0000 7f93f6a00640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f93d406b500 0x7f93d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.839+0000 7f93f6a00640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f93d406b500 0x7f93d406d9c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f93e8003be0 tx=0x7f93e800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:03.843+0000 7f93eea00640 1 -- 172.21.15.114:0/3410537535 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93f001b310 con 0x7f93f80c7940 2024-03-06T07:35:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.075+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93f80684a0 con 0x7f93f80c7940 2024-03-06T07:35:04.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.077+0000 7f93eea00640 1 -- 172.21.15.114:0/3410537535 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f93f0058af0 con 0x7f93f80c7940 2024-03-06T07:35:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:04.079 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:04.079 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.079+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f93d406b500 msgr2=0x7f93d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.079+0000 7f93fd800640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f93d406b500 0x7f93d406d9c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f93e8003be0 tx=0x7f93e800b040 comp rx=0 tx=0).stop 2024-03-06T07:35:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.079+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f80c7940 msgr2=0x7f93f80c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.079+0000 7f93fd800640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f80c7940 0x7f93f80c7dc0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f93f8071f90 tx=0x7f93f000db80 comp rx=0 tx=0).stop 2024-03-06T07:35:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 shutdown_connections 2024-03-06T07:35:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f93d406b500 0x7f93d406d9c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f80c7940 0x7f93f80c7dc0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f93f80c8d60 0x7f93f814bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 --2- 172.21.15.114:0/3410537535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93dc003f30 0x7f93f806fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 >> 172.21.15.114:0/3410537535 conn(0x7f93f80d1c90 msgr2=0x7f93f80cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 shutdown_connections 2024-03-06T07:35:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.080+0000 7f93fd800640 1 -- 172.21.15.114:0/3410537535 wait complete. 2024-03-06T07:35:04.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:35:04.323 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:35:04.323 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:35:04.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:04.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.522+0000 7f7e49c00640 1 Processor -- start 2024-03-06T07:35:04.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.522+0000 7f7e49c00640 1 -- start start 2024-03-06T07:35:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.523+0000 7f7e49c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7e44144660 0x7f7e440b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e49c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440cfe30 con 0x7f7e440cf170 2024-03-06T07:35:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e49c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440cfb50 con 0x7f7e44072310 2024-03-06T07:35:04.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e49c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4406ed90 con 0x7f7e44144660 2024-03-06T07:35:04.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e42000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7e440cf170 0x7f7e44071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59984/0 (socket says 172.21.15.114:59984) 2024-03-06T07:35:04.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e42000640 1 -- 172.21.15.114:0/4010520892 learned_addr learned my addr 172.21.15.114:0/4010520892 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:04.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625880432 0 0) 0x7f7e440cfe30 con 0x7f7e440cf170 2024-03-06T07:35:04.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e28003520 con 0x7f7e440cf170 2024-03-06T07:35:04.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.524+0000 7f7e42a00640 1 --2- 172.21.15.114:0/4010520892 >> v2:172.21.15.177:3300/0 conn(0x7f7e44144660 0x7f7e440b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1206741172 0 0) 0x7f7e28003520 con 0x7f7e440cf170 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 >> v2:172.21.15.177:3300/0 conn(0x7f7e44144660 msgr2=0x7f7e440b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 --2- 172.21.15.114:0/4010520892 >> v2:172.21.15.177:3300/0 conn(0x7f7e44144660 0x7f7e440b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 >> v1:172.21.15.177:6789/0 conn(0x7f7e44072310 legacy=0x7f7e4406f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4406f070 con 0x7f7e440cf170 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7e30001670 con 0x7f7e440cf170 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 >> v1:172.21.15.114:6789/0 conn(0x7f7e440cf170 legacy=0x7f7e44071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e280039b0 0x7f7e28003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e28004800 0x7f7e28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4406ed90 con 0x7f7e280051c0 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2800c0f0 con 0x7f7e28004800 2024-03-06T07:35:04.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2800c2f0 con 0x7f7e280039b0 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e43400640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.525+0000 7f7e42a00640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e280039b0 0x7f7e28003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.526+0000 7f7e42000640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e28004800 0x7f7e28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.526+0000 7f7e43400640 1 -- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e280039b0 msgr2=0x7f7e28003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.526+0000 7f7e43400640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e280039b0 0x7f7e28003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.526+0000 7f7e43400640 1 -- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e28004800 msgr2=0x7f7e28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.526+0000 7f7e43400640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e28004800 0x7f7e28004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.526+0000 7f7e43400640 1 -- 172.21.15.114:0/4010520892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e28003520 con 0x7f7e280051c0 2024-03-06T07:35:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.527+0000 7f7e43400640 1 -- 172.21.15.114:0/4010520892 reap_dead start 2024-03-06T07:35:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.527+0000 7f7e43400640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e2800b9c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c004480 tx=0x7f7e28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ede7796c3b7d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.527+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c034070 con 0x7f7e280051c0 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.527+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c006b50 con 0x7f7e280051c0 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.527+0000 7f7e41600640 1 -- 172.21.15.114:0/4010520892 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c004080 con 0x7f7e280051c0 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.528+0000 7f7e49c00640 1 -- 172.21.15.114:0/4010520892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 msgr2=0x7f7e2800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.528+0000 7f7e49c00640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e2800b9c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c004480 tx=0x7f7e28005830 comp rx=0 tx=0).stop 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.528+0000 7f7e49c00640 1 -- 172.21.15.114:0/4010520892 shutdown_connections 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.528+0000 7f7e49c00640 1 --2- 172.21.15.114:0/4010520892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e2800b9c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.528+0000 7f7e49c00640 1 -- 172.21.15.114:0/4010520892 >> 172.21.15.114:0/4010520892 conn(0x7f7e440d0b90 msgr2=0x7f7e440d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.528+0000 7f7e49c00640 1 -- 172.21.15.114:0/4010520892 shutdown_connections 2024-03-06T07:35:04.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.528+0000 7f7e49c00640 1 -- 172.21.15.114:0/4010520892 wait complete. 2024-03-06T07:35:04.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.531+0000 7f7e49c00640 1 Processor -- start 2024-03-06T07:35:04.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.531+0000 7f7e49c00640 1 -- start start 2024-03-06T07:35:04.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.532+0000 7f7e49c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e44137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e49c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e441f4440 0x7f7e441f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e49c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e44135d20 0x7f7e440ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e49c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440cc6d0 con 0x7f7e280051c0 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e49c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440cc550 con 0x7f7e44135d20 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e49c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e440cf5e0 con 0x7f7e441f4440 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e44137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e44137d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34664/0 (socket says 172.21.15.114:34664) 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 -- 172.21.15.114:0/3227144136 learned_addr learned my addr 172.21.15.114:0/3227144136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42000640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e441f4440 0x7f7e441f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 -- 172.21.15.114:0/3227144136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e441f4440 msgr2=0x7f7e441f0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e441f4440 0x7f7e441f0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 -- 172.21.15.114:0/3227144136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e44135d20 msgr2=0x7f7e440ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e44135d20 0x7f7e440ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.533+0000 7f7e42a00640 1 -- 172.21.15.114:0/3227144136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e44138370 con 0x7f7e280051c0 2024-03-06T07:35:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.534+0000 7f7e42a00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e44137d70 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400f580 tx=0x7f7e3400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.534+0000 7f7e3aa00640 1 -- 172.21.15.114:0/3227144136 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34013810 con 0x7f7e280051c0 2024-03-06T07:35:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.534+0000 7f7e3aa00640 1 -- 172.21.15.114:0/3227144136 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3401a070 con 0x7f7e280051c0 2024-03-06T07:35:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.534+0000 7f7e3aa00640 1 -- 172.21.15.114:0/3227144136 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34013bb0 con 0x7f7e280051c0 2024-03-06T07:35:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.534+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e44138650 con 0x7f7e280051c0 2024-03-06T07:35:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.534+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e441f13a0 con 0x7f7e280051c0 2024-03-06T07:35:04.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.535+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4404f330 con 0x7f7e280051c0 2024-03-06T07:35:04.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.536+0000 7f7e3aa00640 1 -- 172.21.15.114:0/3227144136 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e34012070 con 0x7f7e280051c0 2024-03-06T07:35:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.537+0000 7f7e3aa00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e2406b4d0 0x7f7e2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.537+0000 7f7e3aa00640 1 -- 172.21.15.114:0/3227144136 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7e3408a2d0 con 0x7f7e280051c0 2024-03-06T07:35:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.537+0000 7f7e42000640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e2406b4d0 0x7f7e2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.538+0000 7f7e42000640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e2406b4d0 0x7f7e2406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7e30001200 tx=0x7f7e30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.541+0000 7f7e3aa00640 1 -- 172.21.15.114:0/3227144136 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e3401c310 con 0x7f7e280051c0 2024-03-06T07:35:04.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.777+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e441e3430 con 0x7f7e280051c0 2024-03-06T07:35:04.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.778+0000 7f7e3aa00640 1 -- 172.21.15.114:0/3227144136 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f7e34059070 con 0x7f7e280051c0 2024-03-06T07:35:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:04.780 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:04.780 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e2406b4d0 msgr2=0x7f7e2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e2406b4d0 0x7f7e2406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7e30001200 tx=0x7f7e30009040 comp rx=0 tx=0).stop 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 msgr2=0x7f7e44137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e44137d70 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400f580 tx=0x7f7e3400fa40 comp rx=0 tx=0).stop 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 shutdown_connections 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e44135d20 0x7f7e440ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e441f4440 0x7f7e441f0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e2406b4d0 0x7f7e2406d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 --2- 172.21.15.114:0/3227144136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e280051c0 0x7f7e44137d70 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 >> 172.21.15.114:0/3227144136 conn(0x7f7e440d0b90 msgr2=0x7f7e440cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 shutdown_connections 2024-03-06T07:35:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:04.781+0000 7f7e49c00640 1 -- 172.21.15.114:0/3227144136 wait complete. 2024-03-06T07:35:04.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:35:05.037 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:35:05.037 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:05.037 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:35:05.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.125+0000 7fb781636f40 1 Processor -- start 2024-03-06T07:35:05.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.126+0000 7fb781636f40 1 -- start start 2024-03-06T07:35:05.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb781636f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef31f6990 0x555ef32cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb781636f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef31f6d70 0x555ef32c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb781636f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef32b49b0 0x555ef32c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb781636f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef31f5d10 con 0x555ef31f6d70 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb781636f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef319cfd0 con 0x555ef31f6990 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb781636f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ef31f5fb0 con 0x555ef32b49b0 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb77d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef31f6d70 0x555ef32c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb77d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef31f6d70 0x555ef32c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34688/0 (socket says 172.21.15.114:34688) 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.127+0000 7fb77d600640 1 -- 172.21.15.114:0/667558946 learned_addr learned my addr 172.21.15.114:0/667558946 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb77d600640 1 -- 172.21.15.114:0/667558946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef32b49b0 msgr2=0x555ef32c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb77d600640 1 --2- 172.21.15.114:0/667558946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef32b49b0 0x555ef32c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb77d600640 1 -- 172.21.15.114:0/667558946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef31f6990 msgr2=0x555ef32cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb77d600640 1 --2- 172.21.15.114:0/667558946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef31f6990 0x555ef32cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb77d600640 1 -- 172.21.15.114:0/667558946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ef32cb040 con 0x555ef31f6d70 2024-03-06T07:35:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb77d600640 1 --2- 172.21.15.114:0/667558946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef31f6d70 0x555ef32c9510 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb770001db0 tx=0x7fb77000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3943dc044734a4d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb777e00640 1 -- 172.21.15.114:0/667558946 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77000bde0 con 0x555ef31f6d70 2024-03-06T07:35:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb777e00640 1 -- 172.21.15.114:0/667558946 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77000f400 con 0x555ef31f6d70 2024-03-06T07:35:05.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.128+0000 7fb777e00640 1 -- 172.21.15.114:0/667558946 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77000f7f0 con 0x555ef31f6d70 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 -- 172.21.15.114:0/667558946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef31f6d70 msgr2=0x555ef32c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 --2- 172.21.15.114:0/667558946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef31f6d70 0x555ef32c9510 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb770001db0 tx=0x7fb77000b5b0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 -- 172.21.15.114:0/667558946 shutdown_connections 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 --2- 172.21.15.114:0/667558946 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef32b49b0 0x555ef32c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 --2- 172.21.15.114:0/667558946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef31f6d70 0x555ef32c9510 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 --2- 172.21.15.114:0/667558946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef31f6990 0x555ef32cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 -- 172.21.15.114:0/667558946 >> 172.21.15.114:0/667558946 conn(0x555ef3225c50 msgr2=0x555ef32402a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 -- 172.21.15.114:0/667558946 shutdown_connections 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.129+0000 7fb781636f40 1 -- 172.21.15.114:0/667558946 wait complete. 2024-03-06T07:35:05.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.130+0000 7fb781636f40 1 Processor -- start 2024-03-06T07:35:05.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.130+0000 7fb781636f40 1 -- start start 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb781636f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef31f6990 0x555ef3227e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb781636f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef31f6d70 0x555ef3228370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb781636f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef32b49b0 0x555ef3238a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb781636f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef32ca150 con 0x555ef32b49b0 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb781636f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ef32276f0 con 0x555ef31f6990 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb781636f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ef32ca3f0 con 0x555ef31f6d70 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef32b49b0 0x555ef3238a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef32b49b0 0x555ef3238a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34696/0 (socket says 172.21.15.114:34696) 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 -- 172.21.15.114:0/587543002 learned_addr learned my addr 172.21.15.114:0/587543002 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77d600640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef31f6d70 0x555ef3228370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 -- 172.21.15.114:0/587543002 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef31f6d70 msgr2=0x555ef3228370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x555ef31f6d70 0x555ef3228370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 -- 172.21.15.114:0/587543002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef31f6990 msgr2=0x555ef3227e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555ef31f6990 0x555ef3227e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.132+0000 7fb77ea00640 1 -- 172.21.15.114:0/587543002 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ef323a810 con 0x555ef32b49b0 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.133+0000 7fb77ea00640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef32b49b0 0x555ef3238a80 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fb7780091a0 tx=0x7fb778009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.133+0000 7fb776a00640 1 -- 172.21.15.114:0/587543002 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778014070 con 0x555ef32b49b0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.133+0000 7fb776a00640 1 -- 172.21.15.114:0/587543002 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb778010800 con 0x555ef32b49b0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.133+0000 7fb776a00640 1 -- 172.21.15.114:0/587543002 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778010bf0 con 0x555ef32b49b0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.133+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555ef323aaf0 con 0x555ef32b49b0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.133+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ef323b0b0 con 0x555ef32b49b0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.135+0000 7fb776a00640 1 -- 172.21.15.114:0/587543002 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb778017070 con 0x555ef32b49b0 2024-03-06T07:35:05.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.136+0000 7fb776a00640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb760069eb0 0x7fb76006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.136+0000 7fb776a00640 1 -- 172.21.15.114:0/587543002 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb778088d00 con 0x555ef32b49b0 2024-03-06T07:35:05.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.136+0000 7fb77e000640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb760069eb0 0x7fb76006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.136+0000 7fb781636f40 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555ef32b2650 0x555ef32394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.136+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x555ef32c93c0 con 0x555ef32b2650 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.137+0000 7fb77d600640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555ef32b2650 0x555ef32394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.137+0000 7fb77e000640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb760069eb0 0x7fb76006c370 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb76c054000 tx=0x7fb76c04c070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.137+0000 7fb77d600640 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555ef32b2650 0x555ef32394d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.137+0000 7fb77d600640 1 -- 172.21.15.114:0/587543002 <== osd.0 v2:172.21.15.114:6800/3895229755 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fb77000f5b0 con 0x555ef32b2650 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.137+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555ef32b2650 msgr2=0x555ef32394d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.137+0000 7fb781636f40 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x555ef32b2650 0x555ef32394d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb760069eb0 msgr2=0x7fb76006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb760069eb0 0x7fb76006c370 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb76c054000 tx=0x7fb76c04c070 comp rx=0 tx=0).stop 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef32b49b0 msgr2=0x555ef3238a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 --2- 172.21.15.114:0/587543002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555ef32b49b0 0x555ef3238a80 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fb7780091a0 tx=0x7fb778009660 comp rx=0 tx=0).stop 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb77ea00640 1 -- 172.21.15.114:0/587543002 reap_dead start 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 shutdown_connections 2024-03-06T07:35:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 >> 172.21.15.114:0/587543002 conn(0x555ef3225c50 msgr2=0x555ef3223ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 shutdown_connections 2024-03-06T07:35:05.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.138+0000 7fb781636f40 1 -- 172.21.15.114:0/587543002 wait complete. 2024-03-06T07:35:05.144 INFO:teuthology.orchestra.run.smithi114.stdout:W�file_cdir_2! 2024-03-06T07:35:05.144 DEBUG:teuthology.orchestra.run.smithi114:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 1099511627777, 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "ancestors": [ 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 1099511628278, 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "file_c", 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 17 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 1, 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "dir_2", 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 33 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 17, 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "old_pools": [] 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:35:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:05.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:05.231 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:35:05.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:05.327 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:35:05.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:05.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.577+0000 7f986f200640 1 Processor -- start 2024-03-06T07:35:05.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.577+0000 7f986f200640 1 -- start start 2024-03-06T07:35:05.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.579+0000 7f986f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f98700d4830 0x7f98700cf520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.579+0000 7f986f200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987015f030 con 0x7f98700b2a60 2024-03-06T07:35:05.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.579+0000 7f986f200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987015f310 con 0x7f98700d0600 2024-03-06T07:35:05.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.579+0000 7f986f200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98700d4cc0 con 0x7f98700d4830 2024-03-06T07:35:05.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.579+0000 7f986e800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f98700d4830 0x7f98700cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.579+0000 7f986d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f98700b2a60 0x7f98700d18e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:60006/0 (socket says 172.21.15.114:60006) 2024-03-06T07:35:05.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.579+0000 7f986d400640 1 -- 172.21.15.114:0/2156960158 learned_addr learned my addr 172.21.15.114:0/2156960158 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:05.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.580+0000 7f986e800640 1 -- 172.21.15.114:0/2156960158 >> v1:172.21.15.177:6789/0 conn(0x7f98700d0600 legacy=0x7f98700d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.580+0000 7f986e800640 1 -- 172.21.15.114:0/2156960158 >> v1:172.21.15.114:6789/0 conn(0x7f98700b2a60 legacy=0x7f98700d18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.580+0000 7f986e800640 1 -- 172.21.15.114:0/2156960158 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700cea40 con 0x7f98700d4830 2024-03-06T07:35:05.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.580+0000 7f986ca00640 1 -- 172.21.15.114:0/2156960158 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860929890 0 0) 0x7f987015f030 con 0x7f98700b2a60 2024-03-06T07:35:05.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.581+0000 7f986e800640 1 --2- 172.21.15.114:0/2156960158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d4830 0x7f98700cf520 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9868009d90 tx=0x7f986802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe29ca1adac265 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.581+0000 7f986ca00640 1 -- 172.21.15.114:0/2156960158 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868039070 con 0x7f98700d4830 2024-03-06T07:35:05.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.581+0000 7f986ca00640 1 -- 172.21.15.114:0/2156960158 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986802cd00 con 0x7f98700d4830 2024-03-06T07:35:05.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.581+0000 7f986ca00640 1 -- 172.21.15.114:0/2156960158 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868034660 con 0x7f98700d4830 2024-03-06T07:35:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.582+0000 7f986f200640 1 -- 172.21.15.114:0/2156960158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d4830 msgr2=0x7f98700cf520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.582+0000 7f986f200640 1 --2- 172.21.15.114:0/2156960158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d4830 0x7f98700cf520 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9868009d90 tx=0x7f986802bf50 comp rx=0 tx=0).stop 2024-03-06T07:35:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.582+0000 7f986f200640 1 -- 172.21.15.114:0/2156960158 shutdown_connections 2024-03-06T07:35:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.582+0000 7f986f200640 1 --2- 172.21.15.114:0/2156960158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d4830 0x7f98700cf520 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.582+0000 7f986f200640 1 -- 172.21.15.114:0/2156960158 >> 172.21.15.114:0/2156960158 conn(0x7f98700d3d10 msgr2=0x7f98700d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.582+0000 7f986f200640 1 -- 172.21.15.114:0/2156960158 shutdown_connections 2024-03-06T07:35:05.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.582+0000 7f986f200640 1 -- 172.21.15.114:0/2156960158 wait complete. 2024-03-06T07:35:05.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.585+0000 7f986f200640 1 Processor -- start 2024-03-06T07:35:05.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.585+0000 7f986f200640 1 -- start start 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.586+0000 7f986f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f98700b2a60 0x7f98700b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.586+0000 7f986f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d0600 0x7f9870159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.586+0000 7f986f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98700d4830 0x7f987015d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.586+0000 7f986f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700b3250 con 0x7f98700d0600 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.586+0000 7f986f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700cf090 con 0x7f98700d4830 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.586+0000 7f986f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987015ecf0 con 0x7f98700b2a60 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d0600 0x7f9870159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d0600 0x7f9870159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34712/0 (socket says 172.21.15.114:34712) 2024-03-06T07:35:05.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 -- 172.21.15.114:0/1891125242 learned_addr learned my addr 172.21.15.114:0/1891125242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986de00640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f98700b2a60 0x7f98700b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986e800640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98700d4830 0x7f987015d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 -- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f98700b2a60 msgr2=0x7f98700b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f98700b2a60 0x7f98700b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 -- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98700d4830 msgr2=0x7f987015d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98700d4830 0x7f987015d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986d400640 1 -- 172.21.15.114:0/1891125242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700b9850 con 0x7f98700d0600 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.587+0000 7f986e800640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98700d4830 0x7f987015d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.588+0000 7f986d400640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d0600 0x7f9870159060 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f985c00c4d0 tx=0x7f985c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.588+0000 7f9866000640 1 -- 172.21.15.114:0/1891125242 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c018070 con 0x7f98700d0600 2024-03-06T07:35:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.588+0000 7f9866000640 1 -- 172.21.15.114:0/1891125242 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c0074e0 con 0x7f98700d0600 2024-03-06T07:35:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.588+0000 7f9866000640 1 -- 172.21.15.114:0/1891125242 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c004a20 con 0x7f98700d0600 2024-03-06T07:35:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.588+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98700b7f40 con 0x7f98700d0600 2024-03-06T07:35:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.588+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98700b7180 con 0x7f98700d0600 2024-03-06T07:35:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.589+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987004f330 con 0x7f98700d0600 2024-03-06T07:35:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.590+0000 7f9866000640 1 -- 172.21.15.114:0/1891125242 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f985c004bc0 con 0x7f98700d0600 2024-03-06T07:35:05.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.591+0000 7f9866000640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.591+0000 7f9866000640 1 -- 172.21.15.114:0/1891125242 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f985c089650 con 0x7f98700d0600 2024-03-06T07:35:05.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.591+0000 7f986de00640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.592+0000 7f986de00640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9860000f60 tx=0x7f9860009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.596+0000 7f9866000640 1 -- 172.21.15.114:0/1891125242 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f985c0583d0 con 0x7f98700d0600 2024-03-06T07:35:05.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.914+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f987008d7e0 con 0x7f98700d0600 2024-03-06T07:35:05.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.915+0000 7f9866000640 1 -- 172.21.15.114:0/1891125242 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f985c05e0d0 con 0x7f98700d0600 2024-03-06T07:35:05.917 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:05.917 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:05.917 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f985006b4d0 msgr2=0x7f985006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9860000f60 tx=0x7f9860009040 comp rx=0 tx=0).stop 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d0600 msgr2=0x7f9870159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d0600 0x7f9870159060 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f985c00c4d0 tx=0x7f985c00c990 comp rx=0 tx=0).stop 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 shutdown_connections 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98700d4830 0x7f987015d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98700d0600 0x7f9870159060 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.917+0000 7f986f200640 1 --2- 172.21.15.114:0/1891125242 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f98700b2a60 0x7f98700b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.918+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 >> 172.21.15.114:0/1891125242 conn(0x7f98700d3d10 msgr2=0x7f98700b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.918+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 shutdown_connections 2024-03-06T07:35:05.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:05.918+0000 7f986f200640 1 -- 172.21.15.114:0/1891125242 wait complete. 2024-03-06T07:35:05.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:06.166 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:35:06.166 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:35:06.166 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:35:11.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:11.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.373+0000 7f52dbe00640 1 Processor -- start 2024-03-06T07:35:11.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.373+0000 7f52dbe00640 1 -- start start 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52dbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52d40afa90 0x7f52d40afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52dbe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f52d40d1880 0x7f52d40c7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52dbe00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d406b160 con 0x7f52d40721f0 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52dbe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d4071d60 con 0x7f52d40afa90 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52dbe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d406b9c0 con 0x7f52d40d1880 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52d9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52d40afa90 0x7f52d40afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52d9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52d40afa90 0x7f52d40afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36188/0 (socket says 172.21.15.114:36188) 2024-03-06T07:35:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.375+0000 7f52d9400640 1 -- 172.21.15.114:0/2286695782 learned_addr learned my addr 172.21.15.114:0/2286695782 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.376+0000 7f52d9400640 1 -- 172.21.15.114:0/2286695782 >> v2:172.21.15.177:3300/0 conn(0x7f52d40d1880 msgr2=0x7f52d40c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.376+0000 7f52d8a00640 1 --2- 172.21.15.114:0/2286695782 >> v2:172.21.15.177:3300/0 conn(0x7f52d40d1880 0x7f52d40c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.376+0000 7f52d9400640 1 --2- 172.21.15.114:0/2286695782 >> v2:172.21.15.177:3300/0 conn(0x7f52d40d1880 0x7f52d40c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.376+0000 7f52d9400640 1 -- 172.21.15.114:0/2286695782 >> v1:172.21.15.177:6789/0 conn(0x7f52d40721f0 legacy=0x7f52d415e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.376+0000 7f52d9400640 1 -- 172.21.15.114:0/2286695782 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d40b03b0 con 0x7f52d40afa90 2024-03-06T07:35:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.376+0000 7f52d8a00640 1 --2- 172.21.15.114:0/2286695782 >> v2:172.21.15.177:3300/0 conn(0x7f52d40d1880 0x7f52d40c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.377+0000 7f52d9400640 1 --2- 172.21.15.114:0/2286695782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40afa90 0x7f52d40afe70 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f52d40cd5f0 tx=0x7f52c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d8a67e7bd49810 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.377+0000 7f52d3e00640 1 -- 172.21.15.114:0/2286695782 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c802e070 con 0x7f52d40afa90 2024-03-06T07:35:11.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.377+0000 7f52d3e00640 1 -- 172.21.15.114:0/2286695782 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c8037070 con 0x7f52d40afa90 2024-03-06T07:35:11.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.377+0000 7f52d3e00640 1 -- 172.21.15.114:0/2286695782 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8031a00 con 0x7f52d40afa90 2024-03-06T07:35:11.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 -- 172.21.15.114:0/2286695782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40afa90 msgr2=0x7f52d40afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2286695782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40afa90 0x7f52d40afe70 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f52d40cd5f0 tx=0x7f52c802b880 comp rx=0 tx=0).stop 2024-03-06T07:35:11.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 -- 172.21.15.114:0/2286695782 shutdown_connections 2024-03-06T07:35:11.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2286695782 >> v2:172.21.15.177:3300/0 conn(0x7f52d40d1880 0x7f52d40c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2286695782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40afa90 0x7f52d40afe70 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 -- 172.21.15.114:0/2286695782 >> 172.21.15.114:0/2286695782 conn(0x7f52d40d0ba0 msgr2=0x7f52d40d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 -- 172.21.15.114:0/2286695782 shutdown_connections 2024-03-06T07:35:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.378+0000 7f52dbe00640 1 -- 172.21.15.114:0/2286695782 wait complete. 2024-03-06T07:35:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.380+0000 7f52dbe00640 1 Processor -- start 2024-03-06T07:35:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.380+0000 7f52dbe00640 1 -- start start 2024-03-06T07:35:11.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.381+0000 7f52dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d40721f0 0x7f52d4157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.381+0000 7f52dbe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52d40afa90 0x7f52d4158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.381+0000 7f52dbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40d1880 0x7f52d4158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.381+0000 7f52dbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d40c7850 con 0x7f52d40d1880 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.381+0000 7f52dbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d406b6e0 con 0x7f52d40721f0 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.381+0000 7f52dbe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d40c79d0 con 0x7f52d40afa90 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.381+0000 7f52d9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40d1880 0x7f52d4158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40d1880 0x7f52d4158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36204/0 (socket says 172.21.15.114:36204) 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 -- 172.21.15.114:0/2685487530 learned_addr learned my addr 172.21.15.114:0/2685487530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d8a00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52d40afa90 0x7f52d4158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9400640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d40721f0 0x7f52d4157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 -- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52d40afa90 msgr2=0x7f52d4158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52d40afa90 0x7f52d4158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 -- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d40721f0 msgr2=0x7f52d4157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d40721f0 0x7f52d4157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 -- 172.21.15.114:0/2685487530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d40b5a00 con 0x7f52d40d1880 2024-03-06T07:35:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d8a00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52d40afa90 0x7f52d4158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9400640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d40721f0 0x7f52d4157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d9e00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40d1880 0x7f52d4158aa0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f52cc00b970 tx=0x7f52cc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.382+0000 7f52d1600640 1 -- 172.21.15.114:0/2685487530 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc019070 con 0x7f52d40d1880 2024-03-06T07:35:11.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.383+0000 7f52d1600640 1 -- 172.21.15.114:0/2685487530 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc00cb40 con 0x7f52d40d1880 2024-03-06T07:35:11.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.383+0000 7f52d1600640 1 -- 172.21.15.114:0/2685487530 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc014440 con 0x7f52d40d1880 2024-03-06T07:35:11.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.383+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d40b5ce0 con 0x7f52d40d1880 2024-03-06T07:35:11.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.383+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d40b62a0 con 0x7f52d40d1880 2024-03-06T07:35:11.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.384+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d404f330 con 0x7f52d40d1880 2024-03-06T07:35:11.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.385+0000 7f52d1600640 1 -- 172.21.15.114:0/2685487530 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52cc01c070 con 0x7f52d40d1880 2024-03-06T07:35:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.386+0000 7f52d1600640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52b406b500 0x7f52b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.387+0000 7f52d1600640 1 -- 172.21.15.114:0/2685487530 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f52cc089ed0 con 0x7f52d40d1880 2024-03-06T07:35:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.387+0000 7f52d9400640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52b406b500 0x7f52b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.387+0000 7f52d9400640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52b406b500 0x7f52b406d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f52c8036000 tx=0x7f52c8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.391+0000 7f52d1600640 1 -- 172.21.15.114:0/2685487530 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52cc058c50 con 0x7f52d40d1880 2024-03-06T07:35:11.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.723+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52d415af10 con 0x7f52d40d1880 2024-03-06T07:35:11.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.723+0000 7f52d1600640 1 -- 172.21.15.114:0/2685487530 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f52cc058550 con 0x7f52d40d1880 2024-03-06T07:35:11.726 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:11.727 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:11.727 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:11.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52b406b500 msgr2=0x7f52b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52b406b500 0x7f52b406d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f52c8036000 tx=0x7f52c8008030 comp rx=0 tx=0).stop 2024-03-06T07:35:11.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40d1880 msgr2=0x7f52d4158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40d1880 0x7f52d4158aa0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f52cc00b970 tx=0x7f52cc00be30 comp rx=0 tx=0).stop 2024-03-06T07:35:11.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 shutdown_connections 2024-03-06T07:35:11.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f52b406b500 0x7f52b406d9c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52d40d1880 0x7f52d4158aa0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52d40afa90 0x7f52d4158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 --2- 172.21.15.114:0/2685487530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d40721f0 0x7f52d4157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 >> 172.21.15.114:0/2685487530 conn(0x7f52d40d0ba0 msgr2=0x7f52d406dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 shutdown_connections 2024-03-06T07:35:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:11.726+0000 7f52dbe00640 1 -- 172.21.15.114:0/2685487530 wait complete. 2024-03-06T07:35:11.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:11.956 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:35:11.956 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:35:11.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:12.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.146+0000 7f8cbea00640 1 Processor -- start 2024-03-06T07:35:12.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.146+0000 7f8cbea00640 1 -- start start 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.148+0000 7f8cbea00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8cb8144c00 0x7f8cb806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.148+0000 7f8cbea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 0x7f8cb815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.148+0000 7f8cbea00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb806ee70 con 0x7f8cb80ce0a0 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.148+0000 7f8cbea00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb8160370 con 0x7f8cb8144c00 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.148+0000 7f8cbea00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb806cef0 con 0x7f8cb815f9d0 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.148+0000 7f8cbca00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8cb80ce0a0 0x7f8cb80ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42680/0 (socket says 172.21.15.114:42680) 2024-03-06T07:35:12.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.148+0000 7f8cbca00640 1 -- 172.21.15.114:0/3875638672 learned_addr learned my addr 172.21.15.114:0/3875638672 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb7400640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 0x7f8cb815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784127247 0 0) 0x7f8cb0001490 con 0x7f8cb80ce0a0 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c003520 con 0x7f8cb80ce0a0 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3875638672 >> v2:172.21.15.177:3300/0 conn(0x7f8cb8144c00 0x7f8cb806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1395578581 0 0) 0x7f8c9c003520 con 0x7f8cb80ce0a0 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 msgr2=0x7f8cb815fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 0x7f8cb815fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 >> v2:172.21.15.177:3300/0 conn(0x7f8cb8144c00 msgr2=0x7f8cb806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 --2- 172.21.15.114:0/3875638672 >> v2:172.21.15.177:3300/0 conn(0x7f8cb8144c00 0x7f8cb806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb806d1d0 con 0x7f8cb80ce0a0 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8cb0001bc0 con 0x7f8cb80ce0a0 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 >> v1:172.21.15.114:6789/0 conn(0x7f8cb80ce0a0 legacy=0x7f8cb80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c9c0039b0 0x7f8c9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c004800 0x7f8c9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 0x7f8c9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb806cef0 con 0x7f8c9c0039b0 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c00b9c0 con 0x7f8c9c004800 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.149+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c00bb70 con 0x7f8cb815f9d0 2024-03-06T07:35:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c9c0039b0 0x7f8c9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cbca00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 0x7f8c9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cb7400640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c004800 0x7f8c9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cb7e00640 1 -- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 msgr2=0x7f8c9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 0x7f8c9c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cb7e00640 1 -- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c004800 msgr2=0x7f8c9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c004800 0x7f8c9c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cb7e00640 1 -- 172.21.15.114:0/3875638672 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c003520 con 0x7f8c9c0039b0 2024-03-06T07:35:12.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.150+0000 7f8cbca00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb815f9d0 0x7f8c9c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:12.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.151+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c9c0039b0 0x7f8c9c003dd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ba30 tx=0x7f8c9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298afd24a1b787f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.151+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8036020 con 0x7f8c9c0039b0 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.151+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca8032680 con 0x7f8c9c0039b0 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.151+0000 7f8cb6a00640 1 -- 172.21.15.114:0/3875638672 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800b600 con 0x7f8c9c0039b0 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.152+0000 7f8cbea00640 1 -- 172.21.15.114:0/3875638672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c9c0039b0 msgr2=0x7f8c9c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.152+0000 7f8cbea00640 1 --2- 172.21.15.114:0/3875638672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c9c0039b0 0x7f8c9c003dd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ba30 tx=0x7f8c9c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.152+0000 7f8cbca00640 1 -- 172.21.15.114:0/3875638672 reap_dead start 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.152+0000 7f8cbea00640 1 -- 172.21.15.114:0/3875638672 shutdown_connections 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.152+0000 7f8cbea00640 1 -- 172.21.15.114:0/3875638672 >> 172.21.15.114:0/3875638672 conn(0x7f8cb80b1730 msgr2=0x7f8cb80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.152+0000 7f8cbea00640 1 -- 172.21.15.114:0/3875638672 shutdown_connections 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.152+0000 7f8cbea00640 1 -- 172.21.15.114:0/3875638672 wait complete. 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.154+0000 7f8cbea00640 1 Processor -- start 2024-03-06T07:35:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.154+0000 7f8cbea00640 1 -- start start 2024-03-06T07:35:12.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cbea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c0039b0 0x7f8cb80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cbea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cb813fc10 0x7f8cb80cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cbea00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb80aeeb0 0x7f8cb814c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cbea00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb8160730 con 0x7f8cb813fc10 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cbea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb806d790 con 0x7f8c9c0039b0 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cbea00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb806cb50 con 0x7f8cb80aeeb0 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cb7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cb813fc10 0x7f8cb80cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cb7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cb813fc10 0x7f8cb80cea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36216/0 (socket says 172.21.15.114:36216) 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.155+0000 7f8cb7400640 1 -- 172.21.15.114:0/3496088304 learned_addr learned my addr 172.21.15.114:0/3496088304 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c0039b0 0x7f8cb80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cb7400640 1 -- 172.21.15.114:0/3496088304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb80aeeb0 msgr2=0x7f8cb814c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cb7400640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb80aeeb0 0x7f8cb814c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cb7400640 1 -- 172.21.15.114:0/3496088304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c0039b0 msgr2=0x7f8cb80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cb7400640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c0039b0 0x7f8cb80ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cb7400640 1 -- 172.21.15.114:0/3496088304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb813ee50 con 0x7f8cb813fc10 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cb7400640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cb813fc10 0x7f8cb80cea30 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4010860 tx=0x7f8ca4010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cafe00640 1 -- 172.21.15.114:0/3496088304 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca400f040 con 0x7f8cb813fc10 2024-03-06T07:35:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cafe00640 1 -- 172.21.15.114:0/3496088304 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca400eb60 con 0x7f8cb813fc10 2024-03-06T07:35:12.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb813f130 con 0x7f8cb813fc10 2024-03-06T07:35:12.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cafe00640 1 -- 172.21.15.114:0/3496088304 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca401c400 con 0x7f8cb813fc10 2024-03-06T07:35:12.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.156+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb813f6f0 con 0x7f8cb813fc10 2024-03-06T07:35:12.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.158+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb804f330 con 0x7f8cb813fc10 2024-03-06T07:35:12.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.159+0000 7f8cafe00640 1 -- 172.21.15.114:0/3496088304 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ca401c5a0 con 0x7f8cb813fc10 2024-03-06T07:35:12.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.160+0000 7f8cafe00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c9806b500 0x7f8c9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.160+0000 7f8cafe00640 1 -- 172.21.15.114:0/3496088304 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f8ca408a990 con 0x7f8cb813fc10 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.160+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c9806b500 0x7f8c9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.161+0000 7f8cb7e00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c9806b500 0x7f8c9806d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ba00 tx=0x7f8ca8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.165+0000 7f8cafe00640 1 -- 172.21.15.114:0/3496088304 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ca4018030 con 0x7f8cb813fc10 2024-03-06T07:35:12.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.402+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb8072890 con 0x7f8cb813fc10 2024-03-06T07:35:12.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.403+0000 7f8cafe00640 1 -- 172.21.15.114:0/3496088304 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f8ca4059680 con 0x7f8cb813fc10 2024-03-06T07:35:12.406 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:12.406 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:12.406 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:12.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.406+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c9806b500 msgr2=0x7f8c9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.406+0000 7f8cbea00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c9806b500 0x7f8c9806d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ba00 tx=0x7f8ca8030000 comp rx=0 tx=0).stop 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.406+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cb813fc10 msgr2=0x7f8cb80cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.406+0000 7f8cbea00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cb813fc10 0x7f8cb80cea30 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4010860 tx=0x7f8ca4010d20 comp rx=0 tx=0).stop 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.406+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 shutdown_connections 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.406+0000 7f8cbea00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8cb80aeeb0 0x7f8cb814c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.407+0000 7f8cbea00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8c9806b500 0x7f8c9806d9c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.407+0000 7f8cbea00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cb813fc10 0x7f8cb80cea30 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.407+0000 7f8cbea00640 1 --2- 172.21.15.114:0/3496088304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c0039b0 0x7f8cb80ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.407+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 >> 172.21.15.114:0/3496088304 conn(0x7f8cb80b1730 msgr2=0x7f8cb80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.407+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 shutdown_connections 2024-03-06T07:35:12.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.407+0000 7f8cbea00640 1 -- 172.21.15.114:0/3496088304 wait complete. 2024-03-06T07:35:12.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:35:12.641 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:35:12.642 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:35:12.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.837+0000 7fa563200640 1 Processor -- start 2024-03-06T07:35:12.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.837+0000 7fa563200640 1 -- start start 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa563200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c151660 0x7fa55c071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa563200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa55c15dff0 0x7fa55c15e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa563200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55c071a60 con 0x7fa55c0cf430 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa563200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa55c0b3ee0 con 0x7fa55c15dff0 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa563200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c0b35b0 con 0x7fa55c151660 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa55b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa55c0cf430 0x7fa55c06d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42696/0 (socket says 172.21.15.114:42696) 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa55b400640 1 -- 172.21.15.114:0/3665619145 learned_addr learned my addr 172.21.15.114:0/3665619145 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:12.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.840+0000 7fa561200640 1 --2- 172.21.15.114:0/3665619145 >> v2:172.21.15.114:3300/0 conn(0x7fa55c15dff0 0x7fa55c15e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.841+0000 7fa55be00640 1 --2- 172.21.15.114:0/3665619145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c151660 0x7fa55c071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.841+0000 7fa561200640 1 -- 172.21.15.114:0/3665619145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c151660 msgr2=0x7fa55c071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.841+0000 7fa561200640 1 --2- 172.21.15.114:0/3665619145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c151660 0x7fa55c071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.841+0000 7fa55aa00640 1 -- 172.21.15.114:0/3665619145 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170361525 0 0) 0x7fa55c071a60 con 0x7fa55c0cf430 2024-03-06T07:35:12.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.841+0000 7fa561200640 1 -- 172.21.15.114:0/3665619145 >> v1:172.21.15.114:6789/0 conn(0x7fa55c0cf430 legacy=0x7fa55c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.841+0000 7fa561200640 1 -- 172.21.15.114:0/3665619145 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c0b3890 con 0x7fa55c15dff0 2024-03-06T07:35:12.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.842+0000 7fa561200640 1 --2- 172.21.15.114:0/3665619145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c15dff0 0x7fa55c15e480 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fa554009d90 tx=0x7fa55402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7f2649eb59b3d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.842+0000 7fa55aa00640 1 -- 172.21.15.114:0/3665619145 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554039070 con 0x7fa55c15dff0 2024-03-06T07:35:12.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.842+0000 7fa55aa00640 1 -- 172.21.15.114:0/3665619145 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55402cb30 con 0x7fa55c15dff0 2024-03-06T07:35:12.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.842+0000 7fa55aa00640 1 -- 172.21.15.114:0/3665619145 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554034440 con 0x7fa55c15dff0 2024-03-06T07:35:12.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 -- 172.21.15.114:0/3665619145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c15dff0 msgr2=0x7fa55c15e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 --2- 172.21.15.114:0/3665619145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c15dff0 0x7fa55c15e480 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fa554009d90 tx=0x7fa55402bd80 comp rx=0 tx=0).stop 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 -- 172.21.15.114:0/3665619145 shutdown_connections 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 --2- 172.21.15.114:0/3665619145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c15dff0 0x7fa55c15e480 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 --2- 172.21.15.114:0/3665619145 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c151660 0x7fa55c071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 -- 172.21.15.114:0/3665619145 >> 172.21.15.114:0/3665619145 conn(0x7fa55c0b1730 msgr2=0x7fa55c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 -- 172.21.15.114:0/3665619145 shutdown_connections 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.843+0000 7fa563200640 1 -- 172.21.15.114:0/3665619145 wait complete. 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.845+0000 7fa563200640 1 Processor -- start 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.845+0000 7fa563200640 1 -- start start 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa563200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa55c0cf430 0x7fa55c154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa563200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c151660 0x7fa55c15cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa563200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c15dff0 0x7fa55c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa563200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c06daf0 con 0x7fa55c151660 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa563200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c071d30 con 0x7fa55c0cf430 2024-03-06T07:35:12.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa563200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c06cb50 con 0x7fa55c15dff0 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa55b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c151660 0x7fa55c15cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa55b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c151660 0x7fa55c15cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36228/0 (socket says 172.21.15.114:36228) 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa55b400640 1 -- 172.21.15.114:0/1649111511 learned_addr learned my addr 172.21.15.114:0/1649111511 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa561200640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c15dff0 0x7fa55c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa55b400640 1 -- 172.21.15.114:0/1649111511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c15dff0 msgr2=0x7fa55c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa55b400640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c15dff0 0x7fa55c15d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa55b400640 1 -- 172.21.15.114:0/1649111511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa55c0cf430 msgr2=0x7fa55c154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.846+0000 7fa55b400640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa55c0cf430 0x7fa55c154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.847+0000 7fa55b400640 1 -- 172.21.15.114:0/1649111511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c146b50 con 0x7fa55c151660 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.847+0000 7fa55b400640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c151660 0x7fa55c15cc70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa54800c4d0 tx=0x7fa54800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.847+0000 7fa553e00640 1 -- 172.21.15.114:0/1649111511 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548018070 con 0x7fa55c151660 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.847+0000 7fa553e00640 1 -- 172.21.15.114:0/1649111511 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa548007500 con 0x7fa55c151660 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.847+0000 7fa553e00640 1 -- 172.21.15.114:0/1649111511 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548004a20 con 0x7fa55c151660 2024-03-06T07:35:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.847+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55c146e90 con 0x7fa55c151660 2024-03-06T07:35:12.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.847+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55c152120 con 0x7fa55c151660 2024-03-06T07:35:12.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.849+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa55c152380 con 0x7fa55c151660 2024-03-06T07:35:12.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.849+0000 7fa553e00640 1 -- 172.21.15.114:0/1649111511 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa54801b070 con 0x7fa55c151660 2024-03-06T07:35:12.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.850+0000 7fa553e00640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa53c06b4d0 0x7fa53c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.850+0000 7fa553e00640 1 -- 172.21.15.114:0/1649111511 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa5480896d0 con 0x7fa55c151660 2024-03-06T07:35:12.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.850+0000 7fa55be00640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa53c06b4d0 0x7fa53c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.851+0000 7fa55be00640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa53c06b4d0 0x7fa53c06d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0098f0 tx=0x7fa54c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:12.855+0000 7fa553e00640 1 -- 172.21.15.114:0/1649111511 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa55c146e90 con 0x7fa55c151660 2024-03-06T07:35:13.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.090+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa55c147070 con 0x7fa55c151660 2024-03-06T07:35:13.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.091+0000 7fa553e00640 1 -- 172.21.15.114:0/1649111511 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fa5480583c0 con 0x7fa55c151660 2024-03-06T07:35:13.094 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:13.094 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:13.094 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa53c06b4d0 msgr2=0x7fa53c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa53c06b4d0 0x7fa53c06d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0098f0 tx=0x7fa54c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c151660 msgr2=0x7fa55c15cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c151660 0x7fa55c15cc70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fa54800c4d0 tx=0x7fa54800c990 comp rx=0 tx=0).stop 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 shutdown_connections 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa55c15dff0 0x7fa55c15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa53c06b4d0 0x7fa53c06d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa55c151660 0x7fa55c15cc70 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 --2- 172.21.15.114:0/1649111511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa55c0cf430 0x7fa55c154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 >> 172.21.15.114:0/1649111511 conn(0x7fa55c0b1730 msgr2=0x7fa55c166c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 shutdown_connections 2024-03-06T07:35:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.094+0000 7fa563200640 1 -- 172.21.15.114:0/1649111511 wait complete. 2024-03-06T07:35:13.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:35:13.342 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:35:13.342 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:13.343 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:35:13.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.444+0000 7fbe8ea39f40 1 Processor -- start 2024-03-06T07:35:13.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.444+0000 7fbe8ea39f40 1 -- start start 2024-03-06T07:35:13.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.445+0000 7fbe8ea39f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe700022d0 0x558752dd1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.445+0000 7fbe8ea39f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdbc0 0x558752f0b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.445+0000 7fbe8ea39f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558752dcdfa0 0x558752f0b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.445+0000 7fbe8ea39f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x558752d45fd0 con 0x558752dcdfa0 2024-03-06T07:35:13.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.445+0000 7fbe8ea39f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558752d49f30 con 0x558752dcdbc0 2024-03-06T07:35:13.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.445+0000 7fbe8ea39f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x558752de9ba0 con 0x7fbe700022d0 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.445+0000 7fbe8be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558752dcdfa0 0x558752f0b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558752dcdfa0 0x558752f0b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36238/0 (socket says 172.21.15.114:36238) 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 -- 172.21.15.114:0/3403437686 learned_addr learned my addr 172.21.15.114:0/3403437686 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe83e00640 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdbc0 0x558752f0b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8b400640 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe700022d0 0x558752dd1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 -- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe700022d0 msgr2=0x558752dd1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe700022d0 0x558752dd1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 -- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdbc0 msgr2=0x558752f0b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdbc0 0x558752f0b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 -- 172.21.15.114:0/3403437686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558752e5cd50 con 0x558752dcdfa0 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8be00640 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558752dcdfa0 0x558752f0b630 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x558752e60330 tx=0x7fbe84001940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d3951c7677672c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8a000640 1 -- 172.21.15.114:0/3403437686 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84013660 con 0x558752dcdfa0 2024-03-06T07:35:13.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8a000640 1 -- 172.21.15.114:0/3403437686 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84013800 con 0x558752dcdfa0 2024-03-06T07:35:13.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.446+0000 7fbe8a000640 1 -- 172.21.15.114:0/3403437686 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84013bf0 con 0x558752dcdfa0 2024-03-06T07:35:13.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/3403437686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558752dcdfa0 msgr2=0x558752f0b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558752dcdfa0 0x558752f0b630 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x558752e60330 tx=0x7fbe84001940 comp rx=0 tx=0).stop 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/3403437686 shutdown_connections 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558752dcdfa0 0x558752f0b630 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdbc0 0x558752f0b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/3403437686 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbe700022d0 0x558752dd1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/3403437686 >> 172.21.15.114:0/3403437686 conn(0x558752dcec50 msgr2=0x558752e5ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/3403437686 shutdown_connections 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.447+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/3403437686 wait complete. 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.448+0000 7fbe8ea39f40 1 Processor -- start 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.448+0000 7fbe8ea39f40 1 -- start start 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8ea39f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x558752dcdbc0 0x558752e758c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8ea39f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdfa0 0x558752e6fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8ea39f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe700022d0 0x558752e5b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8ea39f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x558752dd2300 con 0x7fbe700022d0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8ea39f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x558752e7b7f0 con 0x558752dcdfa0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8ea39f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x558752e7ba40 con 0x558752dcdbc0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe700022d0 0x558752e5b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe700022d0 0x558752e5b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36254/0 (socket says 172.21.15.114:36254) 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.449+0000 7fbe8be00640 1 -- 172.21.15.114:0/4223388439 learned_addr learned my addr 172.21.15.114:0/4223388439 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8b400640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x558752dcdbc0 0x558752e758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe83e00640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdfa0 0x558752e6fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8be00640 1 -- 172.21.15.114:0/4223388439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x558752dcdbc0 msgr2=0x558752e758c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8be00640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x558752dcdbc0 0x558752e758c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8be00640 1 -- 172.21.15.114:0/4223388439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdfa0 msgr2=0x558752e6fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8be00640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558752dcdfa0 0x558752e6fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8be00640 1 -- 172.21.15.114:0/4223388439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558752e7bd20 con 0x7fbe700022d0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8be00640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe700022d0 0x558752e5b350 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fbe84001e40 tx=0x7fbe840014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe88c00640 1 -- 172.21.15.114:0/4223388439 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400a070 con 0x7fbe700022d0 2024-03-06T07:35:13.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe88c00640 1 -- 172.21.15.114:0/4223388439 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84004630 con 0x7fbe700022d0 2024-03-06T07:35:13.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.450+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558752e597f0 con 0x7fbe700022d0 2024-03-06T07:35:13.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.451+0000 7fbe88c00640 1 -- 172.21.15.114:0/4223388439 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84004a20 con 0x7fbe700022d0 2024-03-06T07:35:13.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.451+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558752e70290 con 0x7fbe700022d0 2024-03-06T07:35:13.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.453+0000 7fbe88c00640 1 -- 172.21.15.114:0/4223388439 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe84004bc0 con 0x7fbe700022d0 2024-03-06T07:35:13.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.454+0000 7fbe88c00640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbe68069eb0 0x7fbe6806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.454+0000 7fbe88c00640 1 -- 172.21.15.114:0/4223388439 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fbe84089df0 con 0x7fbe700022d0 2024-03-06T07:35:13.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.454+0000 7fbe8b400640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbe68069eb0 0x7fbe6806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.454+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x558752e56080 0x558752e564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.454+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x558752e56a00 con 0x558752e56080 2024-03-06T07:35:13.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.455+0000 7fbe83e00640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x558752e56080 0x558752e564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.455+0000 7fbe8b400640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbe68069eb0 0x7fbe6806c370 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c054000 tx=0x7fbe7c04e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.455+0000 7fbe83e00640 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x558752e56080 0x558752e564c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe83e00640 1 -- 172.21.15.114:0/4223388439 <== osd.0 v2:172.21.15.114:6800/3895229755 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fbe740075f0 con 0x558752e56080 2024-03-06T07:35:13.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x558752e56080 msgr2=0x558752e564c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x558752e56080 0x558752e564c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbe68069eb0 msgr2=0x7fbe6806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbe68069eb0 0x7fbe6806c370 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c054000 tx=0x7fbe7c04e000 comp rx=0 tx=0).stop 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe700022d0 msgr2=0x558752e5b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 --2- 172.21.15.114:0/4223388439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe700022d0 0x558752e5b350 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fbe84001e40 tx=0x7fbe840014a0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8be00640 1 -- 172.21.15.114:0/4223388439 reap_dead start 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 shutdown_connections 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 >> 172.21.15.114:0/4223388439 conn(0x558752dcec50 msgr2=0x558752e5edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 shutdown_connections 2024-03-06T07:35:13.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.456+0000 7fbe8ea39f40 1 -- 172.21.15.114:0/4223388439 wait complete. 2024-03-06T07:35:13.463 INFO:teuthology.orchestra.run.smithi114.stdout:W�file_ddir_2, 2024-03-06T07:35:13.464 DEBUG:teuthology.orchestra.run.smithi114:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:35:13.553 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:35:13.553 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 1099511627777, 2024-03-06T07:35:13.553 INFO:teuthology.orchestra.run.smithi114.stdout: "ancestors": [ 2024-03-06T07:35:13.553 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 1099511628278, 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "file_d", 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 27 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "dirino": 1, 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "dname": "dir_2", 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 44 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 17, 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: "old_pools": [] 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:35:13.554 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:13.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:13.559 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:35:13.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:13.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.795+0000 7fd95f800640 1 Processor -- start 2024-03-06T07:35:13.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.796+0000 7fd95f800640 1 -- start start 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.797+0000 7fd95f800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd95815f9d0 0x7fd95815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.797+0000 7fd95f800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd958070b10 0x7fd958070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.797+0000 7fd95f800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95806cef0 con 0x7fd958144c00 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.797+0000 7fd95f800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95806d1d0 con 0x7fd95815f9d0 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.797+0000 7fd95f800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95806d4b0 con 0x7fd958070b10 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.797+0000 7fd957e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd95815f9d0 0x7fd95815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd95ce00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd958144c00 0x7fd95806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42712/0 (socket says 172.21.15.114:42712) 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd95ce00640 1 -- 172.21.15.114:0/2660905856 learned_addr learned my addr 172.21.15.114:0/2660905856 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd95d800640 1 --2- 172.21.15.114:0/2660905856 >> v2:172.21.15.177:3300/0 conn(0x7fd958070b10 0x7fd958070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd957e00640 1 -- 172.21.15.114:0/2660905856 >> v2:172.21.15.177:3300/0 conn(0x7fd958070b10 msgr2=0x7fd958070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd957e00640 1 --2- 172.21.15.114:0/2660905856 >> v2:172.21.15.177:3300/0 conn(0x7fd958070b10 0x7fd958070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd957e00640 1 -- 172.21.15.114:0/2660905856 >> v1:172.21.15.114:6789/0 conn(0x7fd958144c00 legacy=0x7fd95806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd957e00640 1 -- 172.21.15.114:0/2660905856 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95806d790 con 0x7fd95815f9d0 2024-03-06T07:35:13.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.798+0000 7fd957400640 1 -- 172.21.15.114:0/2660905856 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665296112 0 0) 0x7fd95806cef0 con 0x7fd958144c00 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.799+0000 7fd957e00640 1 --2- 172.21.15.114:0/2660905856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd95815f9d0 0x7fd95815fe60 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fd95806e500 tx=0x7fd94402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e51066a99d3707 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.799+0000 7fd957400640 1 -- 172.21.15.114:0/2660905856 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944035020 con 0x7fd95815f9d0 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.799+0000 7fd957400640 1 -- 172.21.15.114:0/2660905856 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd944031830 con 0x7fd95815f9d0 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.799+0000 7fd957400640 1 -- 172.21.15.114:0/2660905856 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944031c00 con 0x7fd95815f9d0 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 -- 172.21.15.114:0/2660905856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd95815f9d0 msgr2=0x7fd95815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 --2- 172.21.15.114:0/2660905856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd95815f9d0 0x7fd95815fe60 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fd95806e500 tx=0x7fd94402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 -- 172.21.15.114:0/2660905856 shutdown_connections 2024-03-06T07:35:13.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 --2- 172.21.15.114:0/2660905856 >> v2:172.21.15.177:3300/0 conn(0x7fd958070b10 0x7fd958070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 --2- 172.21.15.114:0/2660905856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd95815f9d0 0x7fd95815fe60 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 -- 172.21.15.114:0/2660905856 >> 172.21.15.114:0/2660905856 conn(0x7fd9580b1730 msgr2=0x7fd9580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 -- 172.21.15.114:0/2660905856 shutdown_connections 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.800+0000 7fd95f800640 1 -- 172.21.15.114:0/2660905856 wait complete. 2024-03-06T07:35:13.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.802+0000 7fd95f800640 1 Processor -- start 2024-03-06T07:35:13.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.802+0000 7fd95f800640 1 -- start start 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd95f800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd958070b10 0x7fd9580b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd95f800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd958144c00 0x7fd9580afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd95f800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd95815f9d0 0x7fd9580b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd95f800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95806ccd0 con 0x7fd958144c00 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd95f800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95806cb50 con 0x7fd95815f9d0 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd95f800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95806df90 con 0x7fd958070b10 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd957e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd958144c00 0x7fd9580afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd957e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd958144c00 0x7fd9580afe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36270/0 (socket says 172.21.15.114:36270) 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.803+0000 7fd957e00640 1 -- 172.21.15.114:0/257480619 learned_addr learned my addr 172.21.15.114:0/257480619 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd95ce00640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd958070b10 0x7fd9580b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd95d800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd95815f9d0 0x7fd9580b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd957e00640 1 -- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd958070b10 msgr2=0x7fd9580b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd957e00640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd958070b10 0x7fd9580b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd957e00640 1 -- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd95815f9d0 msgr2=0x7fd9580b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd957e00640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd95815f9d0 0x7fd9580b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd957e00640 1 -- 172.21.15.114:0/257480619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9580b4020 con 0x7fd958144c00 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd95d800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd95815f9d0 0x7fd9580b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.804+0000 7fd957e00640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd958144c00 0x7fd9580afe10 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fd944009890 tx=0x7fd9440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.805+0000 7fd954c00640 1 -- 172.21.15.114:0/257480619 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944040070 con 0x7fd958144c00 2024-03-06T07:35:13.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.805+0000 7fd954c00640 1 -- 172.21.15.114:0/257480619 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94403e070 con 0x7fd958144c00 2024-03-06T07:35:13.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.805+0000 7fd954c00640 1 -- 172.21.15.114:0/257480619 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944035040 con 0x7fd958144c00 2024-03-06T07:35:13.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.805+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9580b34a0 con 0x7fd958144c00 2024-03-06T07:35:13.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.805+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9580b3a60 con 0x7fd958144c00 2024-03-06T07:35:13.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.806+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95804f330 con 0x7fd958144c00 2024-03-06T07:35:13.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.807+0000 7fd954c00640 1 -- 172.21.15.114:0/257480619 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd944038020 con 0x7fd958144c00 2024-03-06T07:35:13.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.808+0000 7fd954c00640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd93806b500 0x7fd93806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.808+0000 7fd954c00640 1 -- 172.21.15.114:0/257480619 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd9440abe10 con 0x7fd958144c00 2024-03-06T07:35:13.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.808+0000 7fd95ce00640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd93806b500 0x7fd93806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.809+0000 7fd95ce00640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd93806b500 0x7fd93806d9c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd94c000f60 tx=0x7fd94c007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:13.813+0000 7fd954c00640 1 -- 172.21.15.114:0/257480619 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9440af030 con 0x7fd958144c00 2024-03-06T07:35:14.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.047+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9580b1250 con 0x7fd958144c00 2024-03-06T07:35:14.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.048+0000 7fd954c00640 1 -- 172.21.15.114:0/257480619 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fd94407aa70 con 0x7fd958144c00 2024-03-06T07:35:14.051 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:14.051 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:14.051 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:14.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd93806b500 msgr2=0x7fd93806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd93806b500 0x7fd93806d9c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd94c000f60 tx=0x7fd94c007b50 comp rx=0 tx=0).stop 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd958144c00 msgr2=0x7fd9580afe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd958144c00 0x7fd9580afe10 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fd944009890 tx=0x7fd9440046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 shutdown_connections 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd95815f9d0 0x7fd9580b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd93806b500 0x7fd93806d9c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd958144c00 0x7fd9580afe10 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 --2- 172.21.15.114:0/257480619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd958070b10 0x7fd9580b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.050+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 >> 172.21.15.114:0/257480619 conn(0x7fd9580b1730 msgr2=0x7fd9580cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.051+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 shutdown_connections 2024-03-06T07:35:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:14.051+0000 7fd95f800640 1 -- 172.21.15.114:0/257480619 wait complete. 2024-03-06T07:35:14.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:14.299 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:35:14.300 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:35:14.300 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:35:16.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:16.677 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:19.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:19.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.529+0000 7f2e14c00640 1 Processor -- start 2024-03-06T07:35:19.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.529+0000 7f2e14c00640 1 -- start start 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.530+0000 7f2e14c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2e10066f10 0x7f2e1006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.530+0000 7f2e14c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2e100ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.530+0000 7f2e14c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1006e5b0 con 0x7f2e1015f930 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.530+0000 7f2e14c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e100ccd90 con 0x7f2e10066f10 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.530+0000 7f2e14c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100afc00 con 0x7f2e100ce8f0 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.530+0000 7f2e0de00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2e1015f930 0x7f2e1015fd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34902/0 (socket says 172.21.15.114:34902) 2024-03-06T07:35:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.530+0000 7f2e0de00640 1 -- 172.21.15.114:0/4158738172 learned_addr learned my addr 172.21.15.114:0/4158738172 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e07e00640 1 --2- 172.21.15.114:0/4158738172 >> v2:172.21.15.177:3300/0 conn(0x7f2e10066f10 0x7f2e1006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0e800640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2e100ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567234624 0 0) 0x7f2e1006e5b0 con 0x7f2e1015f930 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df4003520 con 0x7f2e1015f930 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356439063 0 0) 0x7f2df4003520 con 0x7f2e1015f930 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 msgr2=0x7f2e100ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2e100ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 >> v2:172.21.15.177:3300/0 conn(0x7f2e10066f10 msgr2=0x7f2e1006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 --2- 172.21.15.114:0/4158738172 >> v2:172.21.15.177:3300/0 conn(0x7f2e10066f10 0x7f2e1006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e100ccfe0 con 0x7f2e1015f930 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2e00001670 con 0x7f2e1015f930 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0e800640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2e100ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 >> v1:172.21.15.114:6789/0 conn(0x7f2e1015f930 legacy=0x7f2e1015fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2df40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df4003ef0 0x7f2df4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df4004d40 0x7f2df40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100ccd90 con 0x7f2df4004d40 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100afc00 con 0x7f2df4003ef0 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.531+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df400ba70 con 0x7f2e100ce8f0 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0e800640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df4004d40 0x7f2df40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0de00640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2df40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0e800640 1 -- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 msgr2=0x7f2df40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e07e00640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df4003ef0 0x7f2df4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0e800640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2df40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0e800640 1 -- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df4003ef0 msgr2=0x7f2df4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0e800640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2df4003ef0 0x7f2df4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0e800640 1 -- 172.21.15.114:0/4158738172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df4003520 con 0x7f2df4004d40 2024-03-06T07:35:19.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.532+0000 7f2e0de00640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e100ce8f0 0x7f2df40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.533+0000 7f2e0e800640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df4004d40 0x7f2df40051c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f2e080073e0 tx=0x7f2df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b790a46013f7a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.533+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08039070 con 0x7f2df4004d40 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.533+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0802cd40 con 0x7f2df4004d40 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.533+0000 7f2e0d400640 1 -- 172.21.15.114:0/4158738172 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08009c80 con 0x7f2df4004d40 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.534+0000 7f2e14c00640 1 -- 172.21.15.114:0/4158738172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df4004d40 msgr2=0x7f2df40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.534+0000 7f2e14c00640 1 --2- 172.21.15.114:0/4158738172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df4004d40 0x7f2df40051c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f2e080073e0 tx=0x7f2df4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.534+0000 7f2e0e800640 1 -- 172.21.15.114:0/4158738172 reap_dead start 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.534+0000 7f2e14c00640 1 -- 172.21.15.114:0/4158738172 shutdown_connections 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.534+0000 7f2e14c00640 1 -- 172.21.15.114:0/4158738172 >> 172.21.15.114:0/4158738172 conn(0x7f2e100d0340 msgr2=0x7f2e100d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.534+0000 7f2e14c00640 1 -- 172.21.15.114:0/4158738172 shutdown_connections 2024-03-06T07:35:19.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.534+0000 7f2e14c00640 1 -- 172.21.15.114:0/4158738172 wait complete. 2024-03-06T07:35:19.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.537+0000 7f2e14c00640 1 Processor -- start 2024-03-06T07:35:19.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.537+0000 7f2e14c00640 1 -- start start 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e14c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df4004d40 0x7f2e10143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e14c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e1014e710 0x7f2e1014eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e14c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e101f1c80 0x7f2e101f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e14c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1006e090 con 0x7f2e101f1c80 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e14c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1006ddb0 con 0x7f2e1014e710 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e14c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1006c780 con 0x7f2df4004d40 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e101f1c80 0x7f2e101f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e101f1c80 0x7f2e101f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38666/0 (socket says 172.21.15.114:38666) 2024-03-06T07:35:19.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 -- 172.21.15.114:0/765410902 learned_addr learned my addr 172.21.15.114:0/765410902 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0de00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df4004d40 0x7f2e10143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e07e00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e1014e710 0x7f2e1014eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 -- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df4004d40 msgr2=0x7f2e10143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df4004d40 0x7f2e10143c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 -- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e1014e710 msgr2=0x7f2e1014eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e1014e710 0x7f2e1014eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.539+0000 7f2e0e800640 1 -- 172.21.15.114:0/765410902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e101441b0 con 0x7f2e101f1c80 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e0de00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df4004d40 0x7f2e10143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e07e00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e1014e710 0x7f2e1014eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e0e800640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e101f1c80 0x7f2e101f2100 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2e08037040 tx=0x7f2e08004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e06000640 1 -- 172.21.15.114:0/765410902 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08039030 con 0x7f2e101f1c80 2024-03-06T07:35:19.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e06000640 1 -- 172.21.15.114:0/765410902 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0803a040 con 0x7f2e101f1c80 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1014f290 con 0x7f2e101f1c80 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e06000640 1 -- 172.21.15.114:0/765410902 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08009c80 con 0x7f2e101f1c80 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.540+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1014f850 con 0x7f2e101f1c80 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.542+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e10074ae0 con 0x7f2e101f1c80 2024-03-06T07:35:19.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.542+0000 7f2e06000640 1 -- 172.21.15.114:0/765410902 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e08045020 con 0x7f2e101f1c80 2024-03-06T07:35:19.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.544+0000 7f2e06000640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dec06b500 0x7f2dec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.544+0000 7f2e06000640 1 -- 172.21.15.114:0/765410902 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f2e080abcb0 con 0x7f2e101f1c80 2024-03-06T07:35:19.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.544+0000 7f2e0de00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dec06b500 0x7f2dec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.544+0000 7f2e0de00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dec06b500 0x7f2dec06d9c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2e00001200 tx=0x7f2e00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.548+0000 7f2e06000640 1 -- 172.21.15.114:0/765410902 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e0807a2c0 con 0x7f2e101f1c80 2024-03-06T07:35:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.814+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e1015be10 con 0x7f2e101f1c80 2024-03-06T07:35:19.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.815+0000 7f2e06000640 1 -- 172.21.15.114:0/765410902 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f2e0807e360 con 0x7f2e101f1c80 2024-03-06T07:35:19.818 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dec06b500 msgr2=0x7f2dec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dec06b500 0x7f2dec06d9c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2e00001200 tx=0x7f2e00009040 comp rx=0 tx=0).stop 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e101f1c80 msgr2=0x7f2e101f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e101f1c80 0x7f2e101f2100 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2e08037040 tx=0x7f2e08004ab0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 shutdown_connections 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2dec06b500 0x7f2dec06d9c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e101f1c80 0x7f2e101f2100 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e1014e710 0x7f2e1014eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 --2- 172.21.15.114:0/765410902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2df4004d40 0x7f2e10143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 >> 172.21.15.114:0/765410902 conn(0x7f2e100d0340 msgr2=0x7f2e1015ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 shutdown_connections 2024-03-06T07:35:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:19.818+0000 7f2e14c00640 1 -- 172.21.15.114:0/765410902 wait complete. 2024-03-06T07:35:19.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:20.057 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:35:20.057 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:35:20.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:20.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.293+0000 7f9b34a00640 1 Processor -- start 2024-03-06T07:35:20.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.293+0000 7f9b34a00640 1 -- start start 2024-03-06T07:35:20.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.296+0000 7f9b34a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9b300b40c0 0x7f9b300b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.296+0000 7f9b34a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b30161900 con 0x7f9b301620c0 2024-03-06T07:35:20.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.296+0000 7f9b34a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300b49e0 con 0x7f9b300ae670 2024-03-06T07:35:20.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.296+0000 7f9b34a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3015ebd0 con 0x7f9b300b40c0 2024-03-06T07:35:20.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9b301620c0 0x7f9b300b28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34914/0 (socket says 172.21.15.114:34914) 2024-03-06T07:35:20.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2fe00640 1 -- 172.21.15.114:0/2025214477 learned_addr learned my addr 172.21.15.114:0/2025214477 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:20.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2f400640 1 --2- 172.21.15.114:0/2025214477 >> v2:172.21.15.114:3300/0 conn(0x7f9b300b40c0 0x7f9b300b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2ea00640 1 -- 172.21.15.114:0/2025214477 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959228753 0 0) 0x7f9b30161900 con 0x7f9b301620c0 2024-03-06T07:35:20.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2ea00640 1 -- 172.21.15.114:0/2025214477 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b14003520 con 0x7f9b301620c0 2024-03-06T07:35:20.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2f400640 1 -- 172.21.15.114:0/2025214477 >> v1:172.21.15.177:6789/0 conn(0x7f9b300ae670 legacy=0x7f9b3015e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2f400640 1 -- 172.21.15.114:0/2025214477 >> v1:172.21.15.114:6789/0 conn(0x7f9b301620c0 legacy=0x7f9b300b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.297+0000 7f9b2f400640 1 -- 172.21.15.114:0/2025214477 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3015eeb0 con 0x7f9b300b40c0 2024-03-06T07:35:20.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.298+0000 7f9b2f400640 1 --2- 172.21.15.114:0/2025214477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b300b44a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097a0 tx=0x7f9b2002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e43e6ccdee4316 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b2ea00640 1 -- 172.21.15.114:0/2025214477 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20034070 con 0x7f9b300b40c0 2024-03-06T07:35:20.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b2ea00640 1 -- 172.21.15.114:0/2025214477 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b20030610 con 0x7f9b300b40c0 2024-03-06T07:35:20.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b2ea00640 1 -- 172.21.15.114:0/2025214477 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b200309e0 con 0x7f9b300b40c0 2024-03-06T07:35:20.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b34a00640 1 -- 172.21.15.114:0/2025214477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 msgr2=0x7f9b300b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b34a00640 1 --2- 172.21.15.114:0/2025214477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b300b44a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097a0 tx=0x7f9b2002b960 comp rx=0 tx=0).stop 2024-03-06T07:35:20.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b34a00640 1 -- 172.21.15.114:0/2025214477 shutdown_connections 2024-03-06T07:35:20.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b34a00640 1 --2- 172.21.15.114:0/2025214477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b300b44a0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.299+0000 7f9b34a00640 1 -- 172.21.15.114:0/2025214477 >> 172.21.15.114:0/2025214477 conn(0x7f9b3006afc0 msgr2=0x7f9b3006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.300+0000 7f9b34a00640 1 -- 172.21.15.114:0/2025214477 shutdown_connections 2024-03-06T07:35:20.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.300+0000 7f9b34a00640 1 -- 172.21.15.114:0/2025214477 wait complete. 2024-03-06T07:35:20.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.302+0000 7f9b34a00640 1 Processor -- start 2024-03-06T07:35:20.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.302+0000 7f9b34a00640 1 -- start start 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b34a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b300ae670 0x7f9b30158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b34a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b30160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b34a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9b301620c0 0x7f9b3015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b34a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30158660 con 0x7f9b300b40c0 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b34a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30161ba0 con 0x7f9b300ae670 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b34a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301587e0 con 0x7f9b301620c0 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b26a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b30160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b26a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b30160c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38682/0 (socket says 172.21.15.114:38682) 2024-03-06T07:35:20.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.304+0000 7f9b26a00640 1 -- 172.21.15.114:0/1925047436 learned_addr learned my addr 172.21.15.114:0/1925047436 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:20.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b26a00640 1 -- 172.21.15.114:0/1925047436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9b301620c0 msgr2=0x7f9b3015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b26a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9b301620c0 0x7f9b3015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b26a00640 1 -- 172.21.15.114:0/1925047436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b300ae670 msgr2=0x7f9b30158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b2f400640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b300ae670 0x7f9b30158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b26a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b300ae670 0x7f9b30158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b26a00640 1 -- 172.21.15.114:0/1925047436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b30161200 con 0x7f9b300b40c0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b26a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b30160c90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800c440 tx=0x7f9b18001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b27e00640 1 -- 172.21.15.114:0/1925047436 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b180052d0 con 0x7f9b300b40c0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b27e00640 1 -- 172.21.15.114:0/1925047436 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b18005470 con 0x7f9b300b40c0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b27e00640 1 -- 172.21.15.114:0/1925047436 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18007260 con 0x7f9b300b40c0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b30159290 con 0x7f9b300b40c0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.305+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b30159850 con 0x7f9b300b40c0 2024-03-06T07:35:20.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.307+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3004f330 con 0x7f9b300b40c0 2024-03-06T07:35:20.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.307+0000 7f9b27e00640 1 -- 172.21.15.114:0/1925047436 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b18007400 con 0x7f9b300b40c0 2024-03-06T07:35:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.308+0000 7f9b27e00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b0c06b4d0 0x7f9b0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.308+0000 7f9b27e00640 1 -- 172.21.15.114:0/1925047436 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f9b180898d0 con 0x7f9b300b40c0 2024-03-06T07:35:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.309+0000 7f9b2f400640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b0c06b4d0 0x7f9b0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:20.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.309+0000 7f9b2f400640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b0c06b4d0 0x7f9b0c06d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9b20036000 tx=0x7f9b2003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:20.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.313+0000 7f9b27e00640 1 -- 172.21.15.114:0/1925047436 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b180585c0 con 0x7f9b300b40c0 2024-03-06T07:35:20.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.630+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b30159010 con 0x7f9b300b40c0 2024-03-06T07:35:20.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.631+0000 7f9b27e00640 1 -- 172.21.15.114:0/1925047436 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f9b30159010 con 0x7f9b300b40c0 2024-03-06T07:35:20.634 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:20.634 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:20.634 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.633+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b0c06b4d0 msgr2=0x7f9b0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.633+0000 7f9b34a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b0c06b4d0 0x7f9b0c06d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9b20036000 tx=0x7f9b2003b040 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.633+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 msgr2=0x7f9b30160c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.633+0000 7f9b34a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b30160c90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800c440 tx=0x7f9b18001e70 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 shutdown_connections 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9b301620c0 0x7f9b3015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b0c06b4d0 0x7f9b0c06d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b300b40c0 0x7f9b30160c90 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 --2- 172.21.15.114:0/1925047436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b300ae670 0x7f9b30158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 >> 172.21.15.114:0/1925047436 conn(0x7f9b3006afc0 msgr2=0x7f9b301575e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 shutdown_connections 2024-03-06T07:35:20.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:20.634+0000 7f9b34a00640 1 -- 172.21.15.114:0/1925047436 wait complete. 2024-03-06T07:35:20.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:35:20.899 INFO:teuthology.orchestra.run.smithi114.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:35:20.899 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:35:20.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:21.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.095+0000 7fd100c00640 1 Processor -- start 2024-03-06T07:35:21.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.095+0000 7fd100c00640 1 -- start start 2024-03-06T07:35:21.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.096+0000 7fd100c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0fc15a940 0x7fd0fc0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.096+0000 7fd100c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc06d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.096+0000 7fd100c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc0b2b30 con 0x7fd0fc0cc2a0 2024-03-06T07:35:21.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.096+0000 7fd100c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc0b28e0 con 0x7fd0fc15a940 2024-03-06T07:35:21.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.096+0000 7fd100c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0b2e10 con 0x7fd0fc0cf370 2024-03-06T07:35:21.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.097+0000 7fd0fb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0fc15a940 0x7fd0fc0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.097+0000 7fd0fb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0fc15a940 0x7fd0fc0b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38696/0 (socket says 172.21.15.114:38696) 2024-03-06T07:35:21.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.097+0000 7fd0fb400640 1 -- 172.21.15.114:0/3401234710 learned_addr learned my addr 172.21.15.114:0/3401234710 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:21.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.097+0000 7fd0faa00640 1 --2- 172.21.15.114:0/3401234710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.097+0000 7fd0fb400640 1 -- 172.21.15.114:0/3401234710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 msgr2=0x7fd0fc06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.097+0000 7fd0fb400640 1 --2- 172.21.15.114:0/3401234710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc06d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.097+0000 7fd0fb400640 1 -- 172.21.15.114:0/3401234710 >> v1:172.21.15.114:6789/0 conn(0x7fd0fc0cc2a0 legacy=0x7fd0fc0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.098+0000 7fd0fb400640 1 -- 172.21.15.114:0/3401234710 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc06ca50 con 0x7fd0fc15a940 2024-03-06T07:35:21.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.099+0000 7fd0fb400640 1 --2- 172.21.15.114:0/3401234710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc0b45e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009870 tx=0x7fd0ec02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a392fe5780bfe0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd0fa000640 1 -- 172.21.15.114:0/3401234710 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec037070 con 0x7fd0fc15a940 2024-03-06T07:35:21.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd0fa000640 1 -- 172.21.15.114:0/3401234710 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec004230 con 0x7fd0fc15a940 2024-03-06T07:35:21.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd0fa000640 1 -- 172.21.15.114:0/3401234710 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec003610 con 0x7fd0fc15a940 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd100c00640 1 -- 172.21.15.114:0/3401234710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 msgr2=0x7fd0fc0b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd100c00640 1 --2- 172.21.15.114:0/3401234710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc0b45e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009870 tx=0x7fd0ec02b7e0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd100c00640 1 -- 172.21.15.114:0/3401234710 shutdown_connections 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd100c00640 1 --2- 172.21.15.114:0/3401234710 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc06d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd100c00640 1 --2- 172.21.15.114:0/3401234710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc0b45e0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd100c00640 1 -- 172.21.15.114:0/3401234710 >> 172.21.15.114:0/3401234710 conn(0x7fd0fc06afc0 msgr2=0x7fd0fc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.100+0000 7fd100c00640 1 -- 172.21.15.114:0/3401234710 shutdown_connections 2024-03-06T07:35:21.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.101+0000 7fd100c00640 1 -- 172.21.15.114:0/3401234710 wait complete. 2024-03-06T07:35:21.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.103+0000 7fd100c00640 1 Processor -- start 2024-03-06T07:35:21.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.103+0000 7fd100c00640 1 -- start start 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd100c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0fc0cc2a0 0x7fd0fc152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd100c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd100c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc070990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd100c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0b4b20 con 0x7fd0fc15a940 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd100c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0b30f0 con 0x7fd0fc0cc2a0 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd100c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc06d790 con 0x7fd0fc0cf370 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd0fbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd0fbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38708/0 (socket says 172.21.15.114:38708) 2024-03-06T07:35:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.105+0000 7fd0fbe00640 1 -- 172.21.15.114:0/1285960149 learned_addr learned my addr 172.21.15.114:0/1285960149 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fb400640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0fc0cc2a0 0x7fd0fc152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0faa00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fbe00640 1 -- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 msgr2=0x7fd0fc152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fbe00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fbe00640 1 -- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0fc0cc2a0 msgr2=0x7fd0fc152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fbe00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0fc0cc2a0 0x7fd0fc152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fbe00640 1 -- 172.21.15.114:0/1285960149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc153920 con 0x7fd0fc15a940 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fb400640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0fc0cc2a0 0x7fd0fc152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0fbe00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc070990 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400caf0 tx=0x7fd0f40020d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0f3400640 1 -- 172.21.15.114:0/1285960149 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4017070 con 0x7fd0fc15a940 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.106+0000 7fd0f3400640 1 -- 172.21.15.114:0/1285960149 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f4003da0 con 0x7fd0fc15a940 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.107+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc153c00 con 0x7fd0fc15a940 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.107+0000 7fd0f3400640 1 -- 172.21.15.114:0/1285960149 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f400a920 con 0x7fd0fc15a940 2024-03-06T07:35:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.107+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc1541c0 con 0x7fd0fc15a940 2024-03-06T07:35:21.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.108+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc04f330 con 0x7fd0fc15a940 2024-03-06T07:35:21.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.109+0000 7fd0f3400640 1 -- 172.21.15.114:0/1285960149 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0f401a070 con 0x7fd0fc15a940 2024-03-06T07:35:21.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.110+0000 7fd0f3400640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.110+0000 7fd0f3400640 1 -- 172.21.15.114:0/1285960149 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd0f4089b80 con 0x7fd0fc15a940 2024-03-06T07:35:21.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.110+0000 7fd0fb400640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.111+0000 7fd0fb400640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec034000 tx=0x7fd0ec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.114+0000 7fd0f3400640 1 -- 172.21.15.114:0/1285960149 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0f4058900 con 0x7fd0fc15a940 2024-03-06T07:35:21.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.351+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0fc06e180 con 0x7fd0fc15a940 2024-03-06T07:35:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.352+0000 7fd0f3400640 1 -- 172.21.15.114:0/1285960149 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fd0f4058200 con 0x7fd0fc15a940 2024-03-06T07:35:21.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:21.355 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0dc06b4d0 msgr2=0x7fd0dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec034000 tx=0x7fd0ec008030 comp rx=0 tx=0).stop 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 msgr2=0x7fd0fc070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc070990 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400caf0 tx=0x7fd0f40020d0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 shutdown_connections 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd0dc06b4d0 0x7fd0dc06d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0fc15a940 0x7fd0fc070990 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd0fc0cf370 0x7fd0fc152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 --2- 172.21.15.114:0/1285960149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0fc0cc2a0 0x7fd0fc152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 >> 172.21.15.114:0/1285960149 conn(0x7fd0fc06afc0 msgr2=0x7fd0fc0cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 shutdown_connections 2024-03-06T07:35:21.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.354+0000 7fd100c00640 1 -- 172.21.15.114:0/1285960149 wait complete. 2024-03-06T07:35:21.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache 2024-03-06T07:35:21.604 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:35:21.604 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:35:21.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:21.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.810+0000 7f4f43600640 1 Processor -- start 2024-03-06T07:35:21.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.810+0000 7f4f43600640 1 -- start start 2024-03-06T07:35:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.811+0000 7f4f43600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c15eff0 0x7f4f3c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.811+0000 7f4f43600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4f3c06e4d0 0x7f4f3c06e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.811+0000 7f4f43600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c071d30 con 0x7f4f3c0af1c0 2024-03-06T07:35:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.812+0000 7f4f43600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3c071260 con 0x7f4f3c06e4d0 2024-03-06T07:35:21.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.812+0000 7f4f43600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c15f470 con 0x7f4f3c15eff0 2024-03-06T07:35:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.812+0000 7f4f3be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c15eff0 0x7f4f3c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.812+0000 7f4f3be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c15eff0 0x7f4f3c070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:39366/0 (socket says 172.21.15.114:39366) 2024-03-06T07:35:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.812+0000 7f4f3be00640 1 -- 172.21.15.114:0/135146327 learned_addr learned my addr 172.21.15.114:0/135146327 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:21.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.813+0000 7f4f41600640 1 --2- 172.21.15.114:0/135146327 >> v2:172.21.15.177:3300/0 conn(0x7f4f3c06e4d0 0x7f4f3c06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.813+0000 7f4f3be00640 1 -- 172.21.15.114:0/135146327 >> v2:172.21.15.177:3300/0 conn(0x7f4f3c06e4d0 msgr2=0x7f4f3c06e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.813+0000 7f4f3be00640 1 --2- 172.21.15.114:0/135146327 >> v2:172.21.15.177:3300/0 conn(0x7f4f3c06e4d0 0x7f4f3c06e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.813+0000 7f4f3be00640 1 -- 172.21.15.114:0/135146327 >> v1:172.21.15.177:6789/0 conn(0x7f4f3c0af1c0 legacy=0x7f4f3c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.813+0000 7f4f3be00640 1 -- 172.21.15.114:0/135146327 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c071490 con 0x7f4f3c15eff0 2024-03-06T07:35:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.813+0000 7f4f3b400640 1 -- 172.21.15.114:0/135146327 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792686846 0 0) 0x7f4f2c001490 con 0x7f4f3c0af1c0 2024-03-06T07:35:21.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.815+0000 7f4f3be00640 1 --2- 172.21.15.114:0/135146327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c15eff0 0x7f4f3c070d20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009870 tx=0x7f4f2802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf86f1096a5da33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.815+0000 7f4f3b400640 1 -- 172.21.15.114:0/135146327 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28034070 con 0x7f4f3c15eff0 2024-03-06T07:35:21.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.815+0000 7f4f3b400640 1 -- 172.21.15.114:0/135146327 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f28030840 con 0x7f4f3c15eff0 2024-03-06T07:35:21.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.815+0000 7f4f3b400640 1 -- 172.21.15.114:0/135146327 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28030c10 con 0x7f4f3c15eff0 2024-03-06T07:35:21.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 -- 172.21.15.114:0/135146327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c15eff0 msgr2=0x7f4f3c070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 --2- 172.21.15.114:0/135146327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c15eff0 0x7f4f3c070d20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009870 tx=0x7f4f2802b980 comp rx=0 tx=0).stop 2024-03-06T07:35:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 -- 172.21.15.114:0/135146327 shutdown_connections 2024-03-06T07:35:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 --2- 172.21.15.114:0/135146327 >> v2:172.21.15.177:3300/0 conn(0x7f4f3c06e4d0 0x7f4f3c06e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 --2- 172.21.15.114:0/135146327 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c15eff0 0x7f4f3c070d20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 -- 172.21.15.114:0/135146327 >> 172.21.15.114:0/135146327 conn(0x7f4f3c0b1730 msgr2=0x7f4f3c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 -- 172.21.15.114:0/135146327 shutdown_connections 2024-03-06T07:35:21.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.816+0000 7f4f43600640 1 -- 172.21.15.114:0/135146327 wait complete. 2024-03-06T07:35:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.818+0000 7f4f43600640 1 Processor -- start 2024-03-06T07:35:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.818+0000 7f4f43600640 1 -- start start 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f43600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3c06e4d0 0x7f4f3c1e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f43600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c0af1c0 0x7f4f3c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f43600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f3c15eff0 0x7f4f3c1f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f43600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0b30b0 con 0x7f4f3c15eff0 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f43600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c071a50 con 0x7f4f3c06e4d0 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f43600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c06d370 con 0x7f4f3c0af1c0 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f41600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f3c15eff0 0x7f4f3c1f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f41600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f3c15eff0 0x7f4f3c1f6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38724/0 (socket says 172.21.15.114:38724) 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f41600640 1 -- 172.21.15.114:0/955844281 learned_addr learned my addr 172.21.15.114:0/955844281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f40c00640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3c06e4d0 0x7f4f3c1e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.819+0000 7f4f3be00640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c0af1c0 0x7f4f3c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f41600640 1 -- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c0af1c0 msgr2=0x7f4f3c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f41600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c0af1c0 0x7f4f3c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f41600640 1 -- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3c06e4d0 msgr2=0x7f4f3c1e77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f41600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3c06e4d0 0x7f4f3c1e77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f41600640 1 -- 172.21.15.114:0/955844281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0ce670 con 0x7f4f3c15eff0 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f40c00640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3c06e4d0 0x7f4f3c1e77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f41600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f3c15eff0 0x7f4f3c1f6a10 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f4f3400e930 tx=0x7f4f3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f38c00640 1 -- 172.21.15.114:0/955844281 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f34017070 con 0x7f4f3c15eff0 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f38c00640 1 -- 172.21.15.114:0/955844281 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3400c980 con 0x7f4f3c15eff0 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f38c00640 1 -- 172.21.15.114:0/955844281 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3400cd70 con 0x7f4f3c15eff0 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.820+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c0ce940 con 0x7f4f3c15eff0 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.821+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c0cef00 con 0x7f4f3c15eff0 2024-03-06T07:35:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.822+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c04f330 con 0x7f4f3c15eff0 2024-03-06T07:35:21.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.823+0000 7f4f38c00640 1 -- 172.21.15.114:0/955844281 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f3401a070 con 0x7f4f3c15eff0 2024-03-06T07:35:21.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.823+0000 7f4f38c00640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.824+0000 7f4f38c00640 1 -- 172.21.15.114:0/955844281 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f4f3408a240 con 0x7f4f3c15eff0 2024-03-06T07:35:21.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.824+0000 7f4f40c00640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.824+0000 7f4f40c00640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0012e0 tx=0x7f4f2c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:21.828+0000 7f4f38c00640 1 -- 172.21.15.114:0/955844281 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f34059120 con 0x7f4f3c15eff0 2024-03-06T07:35:22.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.063+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3c074be0 con 0x7f4f3c15eff0 2024-03-06T07:35:22.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.064+0000 7f4f38c00640 1 -- 172.21.15.114:0/955844281 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f4f34058a20 con 0x7f4f3c15eff0 2024-03-06T07:35:22.067 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:22.067 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:22.067 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:22.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4f1c06b4d0 msgr2=0x7f4f1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0012e0 tx=0x7f4f2c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:22.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f3c15eff0 msgr2=0x7f4f3c1f6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f3c15eff0 0x7f4f3c1f6a10 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f4f3400e930 tx=0x7f4f3400edf0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 shutdown_connections 2024-03-06T07:35:22.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4f1c06b4d0 0x7f4f1c06d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f3c15eff0 0x7f4f3c1f6a10 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4f3c0af1c0 0x7f4f3c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 --2- 172.21.15.114:0/955844281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f3c06e4d0 0x7f4f3c1e77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.066+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 >> 172.21.15.114:0/955844281 conn(0x7f4f3c0b1730 msgr2=0x7f4f3c160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.067+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 shutdown_connections 2024-03-06T07:35:22.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.067+0000 7f4f43600640 1 -- 172.21.15.114:0/955844281 wait complete. 2024-03-06T07:35:22.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue 2024-03-06T07:35:22.319 INFO:teuthology.orchestra.run.smithi114.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:35:22.320 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:35:22.320 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:35:22.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.418+0000 7f0a1fe36f40 1 Processor -- start 2024-03-06T07:35:22.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.418+0000 7f0a1fe36f40 1 -- start start 2024-03-06T07:35:22.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1fe36f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51bb3f0 0x559ad527dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1fe36f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 0x559ad51ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1fe36f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51ec1a0 0x559ad51eac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1fe36f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ad52077c0 con 0x559ad51ec1a0 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1fe36f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ad5160fd0 con 0x559ad51bb3f0 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1fe36f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ad528c310 con 0x559ad51ebdc0 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51ec1a0 0x559ad51eac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51ec1a0 0x559ad51eac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38754/0 (socket says 172.21.15.114:38754) 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 -- 172.21.15.114:0/2687478211 learned_addr learned my addr 172.21.15.114:0/2687478211 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1c800640 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51bb3f0 0x559ad527dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 -- 172.21.15.114:0/2687478211 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 msgr2=0x559ad51ea6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 0x559ad51ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 -- 172.21.15.114:0/2687478211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51bb3f0 msgr2=0x559ad527dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51bb3f0 0x559ad527dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.419+0000 7f0a1d200640 1 -- 172.21.15.114:0/2687478211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ad528c540 con 0x559ad51ec1a0 2024-03-06T07:35:22.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1d200640 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51ec1a0 0x559ad51eac20 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0a14001660 tx=0x7f0a14001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e32b570ad9b8cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1aa00640 1 -- 172.21.15.114:0/2687478211 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14003650 con 0x559ad51ec1a0 2024-03-06T07:35:22.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1aa00640 1 -- 172.21.15.114:0/2687478211 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a14003830 con 0x559ad51ec1a0 2024-03-06T07:35:22.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1aa00640 1 -- 172.21.15.114:0/2687478211 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14003c60 con 0x559ad51ec1a0 2024-03-06T07:35:22.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/2687478211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51ec1a0 msgr2=0x559ad51eac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51ec1a0 0x559ad51eac20 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0a14001660 tx=0x7f0a14001b20 comp rx=0 tx=0).stop 2024-03-06T07:35:22.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/2687478211 shutdown_connections 2024-03-06T07:35:22.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51ec1a0 0x559ad51eac20 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 0x559ad51ea6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/2687478211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51bb3f0 0x559ad527dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.420+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/2687478211 >> 172.21.15.114:0/2687478211 conn(0x559ad5207a60 msgr2=0x559ad51eb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.421+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/2687478211 shutdown_connections 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.421+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/2687478211 wait complete. 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.421+0000 7f0a1fe36f40 1 Processor -- start 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.422+0000 7f0a1fe36f40 1 -- start start 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.422+0000 7f0a1fe36f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51bb3f0 0x559ad5267dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.422+0000 7f0a1fe36f40 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 0x559ad52695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.422+0000 7f0a1fe36f40 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51ec1a0 0x559ad527cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.422+0000 7f0a1fe36f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ad528fdb0 con 0x559ad51bb3f0 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.422+0000 7f0a1fe36f40 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ad5290260 con 0x559ad51ec1a0 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.422+0000 7f0a1fe36f40 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ad5290500 con 0x559ad51ebdc0 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51bb3f0 0x559ad5267dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51bb3f0 0x559ad5267dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38756/0 (socket says 172.21.15.114:38756) 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 learned_addr learned my addr 172.21.15.114:0/3060381828 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51ec1a0 0x559ad527cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 0x559ad52695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 msgr2=0x559ad52695e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x559ad51ebdc0 0x559ad52695e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51ec1a0 msgr2=0x559ad527cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ad51ec1a0 0x559ad527cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.423+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ad51eca70 con 0x559ad51bb3f0 2024-03-06T07:35:22.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.424+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51bb3f0 0x559ad5267dd0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f0a10055040 tx=0x559ad51ba3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.424+0000 7f0a19600640 1 -- 172.21.15.114:0/3060381828 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10062020 con 0x559ad51bb3f0 2024-03-06T07:35:22.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.424+0000 7f0a19600640 1 -- 172.21.15.114:0/3060381828 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a10052a70 con 0x559ad51bb3f0 2024-03-06T07:35:22.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.424+0000 7f0a19600640 1 -- 172.21.15.114:0/3060381828 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10057040 con 0x559ad51bb3f0 2024-03-06T07:35:22.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.424+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559ad5268540 con 0x559ad51bb3f0 2024-03-06T07:35:22.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.424+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ad5268b00 con 0x559ad51bb3f0 2024-03-06T07:35:22.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.427+0000 7f0a19600640 1 -- 172.21.15.114:0/3060381828 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a10065020 con 0x559ad51bb3f0 2024-03-06T07:35:22.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.428+0000 7f0a19600640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09fc069eb0 0x7f09fc06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.428+0000 7f0a19600640 1 -- 172.21.15.114:0/3060381828 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f0a100cabf0 con 0x559ad51bb3f0 2024-03-06T07:35:22.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.428+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09fc069eb0 0x7f09fc06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.428+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559ad51ba970 0x559ad5292b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.428+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.429+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09fc069eb0 0x7f09fc06c370 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x559ad51bb020 tx=0x7f0a08005820 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.429+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559ad51ba970 0x559ad5292b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.430+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559ad51ba970 0x559ad5292b20 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.431+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.431+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f0a14013930 0x7f0a140123c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.431+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14013930 2024-03-06T07:35:22.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.431+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f0a14013930 0x7f0a140123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.431+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f0a14013930 0x7f0a140123c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.432+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.1 v2:172.21.15.114:6816/1955604845 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1005a040 con 0x7f0a14013930 2024-03-06T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.432+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f0a10067020 0x7f0a100981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.432+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.432+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f0a10067020 0x7f0a100981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.433+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f0a10067020 0x7f0a100981d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.433+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.433+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f0a08017040 0x7f0a08019950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.433+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017040 2024-03-06T07:35:22.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.434+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f0a08017040 0x7f0a08019950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.435+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f0a08017040 0x7f0a08019950 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.435+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.5 v2:172.21.15.177:6816/2170499622 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a08017040 2024-03-06T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.435+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.436+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.436+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.436+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.436+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f0a08017930 0x7f0a080163b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.437+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.437+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f0a08017930 0x7f0a080163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.437+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f0a08017930 0x7f0a080163b0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.438+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 1 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1005a040 con 0x7f0a08017930 2024-03-06T07:35:22.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.438+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.439+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.439+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.439+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 2 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1005a040 con 0x7f0a08017930 2024-03-06T07:35:22.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.439+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.439+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.440+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.440+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.440+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.440+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 3 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.440+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.441+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.441+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7f0a14022450 0x7f0a140228b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.441+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.442+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7f0a14022450 0x7f0a140228b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.443+0000 7f0a1be00640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7f0a14022450 0x7f0a140228b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.444+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.444+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14013930 2024-03-06T07:35:22.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.444+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.1 v2:172.21.15.114:6816/1955604845 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a14013930 2024-03-06T07:35:22.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.444+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.445+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 4 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.445+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.446+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.446+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017040 2024-03-06T07:35:22.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.447+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.5 v2:172.21.15.177:6816/2170499622 2 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a08017040 2024-03-06T07:35:22.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.447+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.447+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.447+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.448+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.448+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.449+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.449+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.449+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.449+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.450+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 5 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.450+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14013930 2024-03-06T07:35:22.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.450+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.1 v2:172.21.15.114:6816/1955604845 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a14013930 2024-03-06T07:35:22.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.450+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f0a10099570 0x7f0a100c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.450+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10099570 2024-03-06T07:35:22.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.451+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f0a10099570 0x7f0a100c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.452+0000 7f0a1d200640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f0a10099570 0x7f0a100c2a20 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.453+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.6 v2:172.21.15.177:6808/4041662010 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a10099570 2024-03-06T07:35:22.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.453+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.453+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 6 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.453+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017040 2024-03-06T07:35:22.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.454+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.5 v2:172.21.15.177:6816/2170499622 3 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a08017040 2024-03-06T07:35:22.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.454+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.455+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 7 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.455+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f0a100a1590 0x7f0a100c2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.455+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a100a1590 2024-03-06T07:35:22.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.455+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f0a100a1590 0x7f0a100c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.456+0000 7f0a1c800640 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f0a100a1590 0x7f0a100c2130 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.456+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.2 v2:172.21.15.114:6808/1832993461 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a100a1590 2024-03-06T07:35:22.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.456+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10099570 2024-03-06T07:35:22.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.457+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.6 v2:172.21.15.177:6808/4041662010 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a10099570 2024-03-06T07:35:22.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.457+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14013930 2024-03-06T07:35:22.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.458+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.1 v2:172.21.15.114:6816/1955604845 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a14013930 2024-03-06T07:35:22.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.458+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.459+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.459+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.459+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.459+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.459+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.459+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.460+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.460+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14013930 2024-03-06T07:35:22.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.460+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.1 v2:172.21.15.114:6816/1955604845 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a14013930 2024-03-06T07:35:22.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.460+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.461+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.461+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.462+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.462+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017040 2024-03-06T07:35:22.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.463+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.5 v2:172.21.15.177:6816/2170499622 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a08017040 2024-03-06T07:35:22.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.463+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017040 2024-03-06T07:35:22.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.463+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.5 v2:172.21.15.177:6816/2170499622 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a08017040 2024-03-06T07:35:22.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.463+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10099570 2024-03-06T07:35:22.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.464+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.6 v2:172.21.15.177:6808/4041662010 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a10099570 2024-03-06T07:35:22.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.464+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14013930 2024-03-06T07:35:22.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.465+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.1 v2:172.21.15.114:6816/1955604845 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a14013930 2024-03-06T07:35:22.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.465+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017040 2024-03-06T07:35:22.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.466+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.5 v2:172.21.15.177:6816/2170499622 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a08017040 2024-03-06T07:35:22.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.466+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.467+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.467+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.468+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.468+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.469+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.469+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.470+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.470+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.471+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.471+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.472+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.472+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.472+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.473+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10067020 2024-03-06T07:35:22.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.473+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.3 v2:172.21.15.114:6818/735520392 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a10067020 2024-03-06T07:35:22.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.473+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x559ad51ba970 2024-03-06T07:35:22.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.474+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.7 v2:172.21.15.177:6824/1593615297 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x559ad51ba970 2024-03-06T07:35:22.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.474+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.475+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.475+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.476+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.476+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.476+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.477+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a100a1590 2024-03-06T07:35:22.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.477+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.2 v2:172.21.15.114:6808/1832993461 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a100a1590 2024-03-06T07:35:22.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.477+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.478+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 11 ==== osd_op_reply(57 [pgnls start_epoch 114 out=49b] v114'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.478+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.479+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.479+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10099570 2024-03-06T07:35:22.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.480+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.6 v2:172.21.15.177:6808/4041662010 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a10099570 2024-03-06T07:35:22.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.480+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.480+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 12 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.480+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14013930 2024-03-06T07:35:22.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.481+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.1 v2:172.21.15.114:6816/1955604845 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a14013930 2024-03-06T07:35:22.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.481+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a14022450 2024-03-06T07:35:22.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.482+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 <== osd.4 v2:172.21.15.177:6800/4111454897 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=49b] v114'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a08014070 con 0x7f0a14022450 2024-03-06T07:35:22.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.482+0000 7f0a1be00640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a10099570 2024-03-06T07:35:22.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.483+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 <== osd.6 v2:172.21.15.177:6808/4041662010 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a1400f030 con 0x7f0a10099570 2024-03-06T07:35:22.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.483+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 --> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x559ad5293060 con 0x7f0a08017930 2024-03-06T07:35:22.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.483+0000 7f0a1c800640 1 -- 172.21.15.114:0/3060381828 <== osd.0 v2:172.21.15.114:6800/3895229755 13 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0a100b6040 con 0x7f0a08017930 2024-03-06T07:35:22.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.483+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f0a08017930 msgr2=0x7f0a080163b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.483+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6800/3895229755,v1:172.21.15.114:6801/3895229755] conn(0x7f0a08017930 0x7f0a080163b0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f0a14013930 msgr2=0x7f0a140123c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6816/1955604845,v1:172.21.15.114:6817/1955604845] conn(0x7f0a14013930 0x7f0a140123c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f0a100a1590 msgr2=0x7f0a100c2130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6808/1832993461,v1:172.21.15.114:6809/1832993461] conn(0x7f0a100a1590 0x7f0a100c2130 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f0a10067020 msgr2=0x7f0a100981d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6818/735520392,v1:172.21.15.114:6820/735520392] conn(0x7f0a10067020 0x7f0a100981d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7f0a14022450 msgr2=0x7f0a140228b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6800/4111454897,v1:172.21.15.177:6801/4111454897] conn(0x7f0a14022450 0x7f0a140228b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 reap_dead start 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f0a08017040 msgr2=0x7f0a08019950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6816/2170499622,v1:172.21.15.177:6817/2170499622] conn(0x7f0a08017040 0x7f0a08019950 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1d200640 1 -- 172.21.15.114:0/3060381828 reap_dead start 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f0a10099570 msgr2=0x7f0a100c2a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f0a10099570 0x7f0a100c2a20 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559ad51ba970 msgr2=0x559ad5292b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559ad51ba970 0x559ad5292b20 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09fc069eb0 msgr2=0x7f09fc06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09fc069eb0 0x7f09fc06c370 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x559ad51bb020 tx=0x7f0a08005820 comp rx=0 tx=0).stop 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51bb3f0 msgr2=0x559ad5267dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51bb3f0 0x559ad5267dd0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f0a10055040 tx=0x559ad51ba3f0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 shutdown_connections 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6808/4041662010,v1:172.21.15.177:6809/4041662010] conn(0x7f0a10099570 0x7f0a100c2a20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.177:6824/1593615297,v1:172.21.15.177:6825/1593615297] conn(0x559ad51ba970 0x559ad5292b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f09fc069eb0 0x7f09fc06c370 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 --2- 172.21.15.114:0/3060381828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ad51bb3f0 0x559ad5267dd0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.484+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 >> 172.21.15.114:0/3060381828 conn(0x559ad5207a60 msgr2=0x559ad51eb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.485+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 shutdown_connections 2024-03-06T07:35:22.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.485+0000 7f0a1fe36f40 1 -- 172.21.15.114:0/3060381828 wait complete. 2024-03-06T07:35:22.495 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:35:22.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:22.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.695+0000 7f8605a00640 1 Processor -- start 2024-03-06T07:35:22.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.695+0000 7f8605a00640 1 -- start start 2024-03-06T07:35:22.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f8605a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f860015f9d0 0x7f860015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f8605a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8600070b10 0x7f8600070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f8605a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860006cef0 con 0x7f8600144c00 2024-03-06T07:35:22.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f8605a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860006d1d0 con 0x7f860015f9d0 2024-03-06T07:35:22.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f8605a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860006d4b0 con 0x7f8600070b10 2024-03-06T07:35:22.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f85fea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8600144c00 0x7f860006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34958/0 (socket says 172.21.15.114:34958) 2024-03-06T07:35:22.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f85fea00640 1 -- 172.21.15.114:0/1304308198 learned_addr learned my addr 172.21.15.114:0/1304308198 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f85ff400640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8600070b10 0x7f8600070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f85fe000640 1 --2- 172.21.15.114:0/1304308198 >> v2:172.21.15.177:3300/0 conn(0x7f860015f9d0 0x7f860015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049074829 0 0) 0x7f860006cef0 con 0x7f8600144c00 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.697+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e4003520 con 0x7f8600144c00 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1569140704 0 0) 0x7f85e4003520 con 0x7f8600144c00 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8600070b10 msgr2=0x7f8600070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8600070b10 0x7f8600070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 >> v2:172.21.15.177:3300/0 conn(0x7f860015f9d0 msgr2=0x7f860015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 --2- 172.21.15.114:0/1304308198 >> v2:172.21.15.177:3300/0 conn(0x7f860015f9d0 0x7f860015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860006d790 con 0x7f8600144c00 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f85f0001670 con 0x7f8600144c00 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 >> v1:172.21.15.114:6789/0 conn(0x7f8600144c00 legacy=0x7f860006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f85e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e4004800 0x7f85e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8600070b10 0x7f85e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860006d1d0 con 0x7f85e40039b0 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860006d4b0 con 0x7f85e4004800 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e400b9c0 con 0x7f8600070b10 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.698+0000 7f85fea00640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f85e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.699+0000 7f85fea00640 1 -- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8600070b10 msgr2=0x7f85e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.699+0000 7f85fea00640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8600070b10 0x7f85e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.699+0000 7f85fea00640 1 -- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e4004800 msgr2=0x7f85e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.699+0000 7f85fea00640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e4004800 0x7f85e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.699+0000 7f85fea00640 1 -- 172.21.15.114:0/1304308198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e4003520 con 0x7f85e40039b0 2024-03-06T07:35:22.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.700+0000 7f85fea00640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f85e4003dd0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f85f0001050 tx=0x7f85e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21a4188cb227e6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.700+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0007ca0 con 0x7f85e40039b0 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.700+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f0030070 con 0x7f85e40039b0 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.700+0000 7f85fd600640 1 -- 172.21.15.114:0/1304308198 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0005d10 con 0x7f85e40039b0 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.700+0000 7f8605a00640 1 -- 172.21.15.114:0/1304308198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 msgr2=0x7f85e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.700+0000 7f8605a00640 1 --2- 172.21.15.114:0/1304308198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f85e4003dd0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f85f0001050 tx=0x7f85e4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.700+0000 7f85ff400640 1 -- 172.21.15.114:0/1304308198 reap_dead start 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.701+0000 7f8605a00640 1 -- 172.21.15.114:0/1304308198 shutdown_connections 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.701+0000 7f8605a00640 1 -- 172.21.15.114:0/1304308198 >> 172.21.15.114:0/1304308198 conn(0x7f86000b1730 msgr2=0x7f86000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.701+0000 7f8605a00640 1 -- 172.21.15.114:0/1304308198 shutdown_connections 2024-03-06T07:35:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.701+0000 7f8605a00640 1 -- 172.21.15.114:0/1304308198 wait complete. 2024-03-06T07:35:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.702+0000 7f8605a00640 1 Processor -- start 2024-03-06T07:35:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.703+0000 7f8605a00640 1 -- start start 2024-03-06T07:35:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.703+0000 7f8605a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f860013ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.703+0000 7f8605a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86001430b0 0x7f860013c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.703+0000 7f8605a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f860013e6b0 0x7f860013eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.703+0000 7f8605a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860006ccd0 con 0x7f85e40039b0 2024-03-06T07:35:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.703+0000 7f8605a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860006cb50 con 0x7f860013e6b0 2024-03-06T07:35:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.703+0000 7f8605a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860006df90 con 0x7f86001430b0 2024-03-06T07:35:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f860013ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f860013ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38770/0 (socket says 172.21.15.114:38770) 2024-03-06T07:35:22.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 -- 172.21.15.114:0/2938925884 learned_addr learned my addr 172.21.15.114:0/2938925884 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fe000640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86001430b0 0x7f860013c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85ff400640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f860013e6b0 0x7f860013eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 -- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86001430b0 msgr2=0x7f860013c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86001430b0 0x7f860013c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 -- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f860013e6b0 msgr2=0x7f860013eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f860013e6b0 0x7f860013eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fea00640 1 -- 172.21.15.114:0/2938925884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860013f380 con 0x7f85e40039b0 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.704+0000 7f85fe000640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86001430b0 0x7f860013c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.705+0000 7f85fea00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f860013ce30 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f85f002c500 tx=0x7f85f0000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.705+0000 7f85f6a00640 1 -- 172.21.15.114:0/2938925884 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0004bf0 con 0x7f85e40039b0 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.705+0000 7f85f6a00640 1 -- 172.21.15.114:0/2938925884 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f0004dd0 con 0x7f85e40039b0 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.705+0000 7f85f6a00640 1 -- 172.21.15.114:0/2938925884 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f00072b0 con 0x7f85e40039b0 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.705+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860013f660 con 0x7f85e40039b0 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.705+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860013fc20 con 0x7f85e40039b0 2024-03-06T07:35:22.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.706+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860004f330 con 0x7f85e40039b0 2024-03-06T07:35:22.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.708+0000 7f85f6a00640 1 -- 172.21.15.114:0/2938925884 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85f0007450 con 0x7f85e40039b0 2024-03-06T07:35:22.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.709+0000 7f85f6a00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.709+0000 7f85f6a00640 1 -- 172.21.15.114:0/2938925884 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f85f00abac0 con 0x7f85e40039b0 2024-03-06T07:35:22.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.709+0000 7f85fe000640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.710+0000 7f85fe000640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f860015c090 tx=0x7f85ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.714+0000 7f85f6a00640 1 -- 172.21.15.114:0/2938925884 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85f0032320 con 0x7f85e40039b0 2024-03-06T07:35:22.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.935+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86000b4b00 con 0x7f85e40039b0 2024-03-06T07:35:22.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.936+0000 7f85f6a00640 1 -- 172.21.15.114:0/2938925884 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f85f0080020 con 0x7f85e40039b0 2024-03-06T07:35:22.938 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:22.939 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:22.939 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:22.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f85e006b4d0 msgr2=0x7f85e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f85e006b4d0 0x7f85e006d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f860015c090 tx=0x7f85ec008040 comp rx=0 tx=0).stop 2024-03-06T07:35:22.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 msgr2=0x7f860013ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f860013ce30 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f85f002c500 tx=0x7f85f0000fe0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 shutdown_connections 2024-03-06T07:35:22.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f860013e6b0 0x7f860013eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f86001430b0 0x7f860013c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f85e006b4d0 0x7f85e006d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 --2- 172.21.15.114:0/2938925884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e40039b0 0x7f860013ce30 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 >> 172.21.15.114:0/2938925884 conn(0x7f86000b1730 msgr2=0x7f8600071780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 shutdown_connections 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:22.938+0000 7f8605a00640 1 -- 172.21.15.114:0/2938925884 wait complete. 2024-03-06T07:35:22.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:35:23.179 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:35:23.179 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:35:23.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.385+0000 7f1699600640 1 Processor -- start 2024-03-06T07:35:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.385+0000 7f1699600640 1 -- start start 2024-03-06T07:35:23.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.387+0000 7f1699600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f16940af820 0x7f16940b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.387+0000 7f1699600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16940b3da0 con 0x7f169415e160 2024-03-06T07:35:23.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.387+0000 7f1699600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169406f1f0 con 0x7f16940b38f0 2024-03-06T07:35:23.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.387+0000 7f1699600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169406b0f0 con 0x7f16940af820 2024-03-06T07:35:23.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.387+0000 7f1692a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f169415e160 0x7f1694162620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34970/0 (socket says 172.21.15.114:34970) 2024-03-06T07:35:23.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.387+0000 7f1692a00640 1 -- 172.21.15.114:0/603701281 learned_addr learned my addr 172.21.15.114:0/603701281 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091653500 0 0) 0x7f1684001490 con 0x7f169415e160 2024-03-06T07:35:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678003520 con 0x7f169415e160 2024-03-06T07:35:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1692000640 1 --2- 172.21.15.114:0/603701281 >> v2:172.21.15.177:3300/0 conn(0x7f16940af820 0x7f16940b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 492428224 0 0) 0x7f1678003520 con 0x7f169415e160 2024-03-06T07:35:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 >> v2:172.21.15.177:3300/0 conn(0x7f16940af820 msgr2=0x7f16940b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 --2- 172.21.15.114:0/603701281 >> v2:172.21.15.177:3300/0 conn(0x7f16940af820 0x7f16940b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 >> v1:172.21.15.177:6789/0 conn(0x7f16940b38f0 legacy=0x7f169406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16940afce0 con 0x7f169415e160 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f1684001bc0 con 0x7f169415e160 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 >> v1:172.21.15.114:6789/0 conn(0x7f169415e160 legacy=0x7f1694162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16780039b0 0x7f1678003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678004800 0x7f1678004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780052e0 0x7f167800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169406b0f0 con 0x7f16780039b0 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167800c1b0 con 0x7f16780052e0 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.388+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167800c3b0 con 0x7f1678004800 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1692a00640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16780039b0 0x7f1678003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1692000640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678004800 0x7f1678004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1692a00640 1 -- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678004800 msgr2=0x7f1678004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1693400640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780052e0 0x7f167800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1692a00640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1678004800 0x7f1678004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1692a00640 1 -- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780052e0 msgr2=0x7f167800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1692a00640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16780052e0 0x7f167800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1692a00640 1 -- 172.21.15.114:0/603701281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1678003520 con 0x7f16780039b0 2024-03-06T07:35:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.389+0000 7f1693400640 1 -- 172.21.15.114:0/603701281 reap_dead start 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.390+0000 7f1692a00640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16780039b0 0x7f1678003dd0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f168400aa00 tx=0x7f1678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51beb5a5b8158cc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.390+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168400a650 con 0x7f16780039b0 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.390+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684034640 con 0x7f16780039b0 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.390+0000 7f1691600640 1 -- 172.21.15.114:0/603701281 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684034a70 con 0x7f16780039b0 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.391+0000 7f1699600640 1 -- 172.21.15.114:0/603701281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16780039b0 msgr2=0x7f1678003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.391+0000 7f1699600640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16780039b0 0x7f1678003dd0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f168400aa00 tx=0x7f1678005830 comp rx=0 tx=0).stop 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.391+0000 7f1699600640 1 -- 172.21.15.114:0/603701281 shutdown_connections 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.391+0000 7f1699600640 1 --2- 172.21.15.114:0/603701281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16780039b0 0x7f1678003dd0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.391+0000 7f1699600640 1 -- 172.21.15.114:0/603701281 >> 172.21.15.114:0/603701281 conn(0x7f16940d0340 msgr2=0x7f16940d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.391+0000 7f1699600640 1 -- 172.21.15.114:0/603701281 shutdown_connections 2024-03-06T07:35:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.391+0000 7f1699600640 1 -- 172.21.15.114:0/603701281 wait complete. 2024-03-06T07:35:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.393+0000 7f1699600640 1 Processor -- start 2024-03-06T07:35:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.393+0000 7f1699600640 1 -- start start 2024-03-06T07:35:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1699600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16780039b0 0x7f1694160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1699600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1694143790 0x7f1694142f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1699600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1694144df0 0x7f1694145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1699600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169406ef10 con 0x7f1694144df0 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1699600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16940b5620 con 0x7f1694143790 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1699600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16940b33b0 con 0x7f16780039b0 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1693400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1694144df0 0x7f1694145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1693400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1694144df0 0x7f1694145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38798/0 (socket says 172.21.15.114:38798) 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1693400640 1 -- 172.21.15.114:0/2922907323 learned_addr learned my addr 172.21.15.114:0/2922907323 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1692a00640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16780039b0 0x7f1694160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.394+0000 7f1692000640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1694143790 0x7f1694142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1693400640 1 -- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16780039b0 msgr2=0x7f1694160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1693400640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16780039b0 0x7f1694160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1693400640 1 -- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1694143790 msgr2=0x7f1694142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1693400640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1694143790 0x7f1694142f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1693400640 1 -- 172.21.15.114:0/2922907323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1694161460 con 0x7f1694144df0 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1692000640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1694143790 0x7f1694142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1693400640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1694144df0 0x7f1694145820 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f16940b2040 tx=0x7f168c00d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f168aa00640 1 -- 172.21.15.114:0/2922907323 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c018070 con 0x7f1694144df0 2024-03-06T07:35:23.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f168aa00640 1 -- 172.21.15.114:0/2922907323 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c0052d0 con 0x7f1694144df0 2024-03-06T07:35:23.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1694143460 con 0x7f1694144df0 2024-03-06T07:35:23.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f168aa00640 1 -- 172.21.15.114:0/2922907323 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c003810 con 0x7f1694144df0 2024-03-06T07:35:23.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.395+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1694144030 con 0x7f1694144df0 2024-03-06T07:35:23.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.397+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1694074ae0 con 0x7f1694144df0 2024-03-06T07:35:23.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.397+0000 7f168aa00640 1 -- 172.21.15.114:0/2922907323 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f168c01b070 con 0x7f1694144df0 2024-03-06T07:35:23.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.398+0000 7f168aa00640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f167406b500 0x7f167406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.398+0000 7f168aa00640 1 -- 172.21.15.114:0/2922907323 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f168c089760 con 0x7f1694144df0 2024-03-06T07:35:23.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.399+0000 7f1692a00640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f167406b500 0x7f167406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.399+0000 7f1692a00640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f167406b500 0x7f167406d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1684035000 tx=0x7f1684030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.403+0000 7f168aa00640 1 -- 172.21.15.114:0/2922907323 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f168c0584e0 con 0x7f1694144df0 2024-03-06T07:35:23.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.680+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f169406bba0 con 0x7f1694144df0 2024-03-06T07:35:23.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.681+0000 7f168aa00640 1 -- 172.21.15.114:0/2922907323 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f168c057de0 con 0x7f1694144df0 2024-03-06T07:35:23.684 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:23.684 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:23.685 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f167406b500 msgr2=0x7f167406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f167406b500 0x7f167406d9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1684035000 tx=0x7f1684030000 comp rx=0 tx=0).stop 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1694144df0 msgr2=0x7f1694145820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1694144df0 0x7f1694145820 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f16940b2040 tx=0x7f168c00d5e0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 shutdown_connections 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f167406b500 0x7f167406d9c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1694144df0 0x7f1694145820 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1694143790 0x7f1694142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 --2- 172.21.15.114:0/2922907323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f16780039b0 0x7f1694160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 >> 172.21.15.114:0/2922907323 conn(0x7f16940d0340 msgr2=0x7f169406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 shutdown_connections 2024-03-06T07:35:23.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:23.684+0000 7f1699600640 1 -- 172.21.15.114:0/2922907323 wait complete. 2024-03-06T07:35:23.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:23.932 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:35:23.933 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:35:23.933 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:23.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:24.035 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:24.156 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:24.156 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:35:24.156 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:24.210 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:35:24.210 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:24.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.501+0000 7f0abbe00640 1 Processor -- start 2024-03-06T07:35:24.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.501+0000 7f0abbe00640 1 -- start start 2024-03-06T07:35:24.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0abbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0abc0aedc0 0x7f0abc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0abbe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc15c240 con 0x7f0abc1561f0 2024-03-06T07:35:24.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0abbe00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc15c500 con 0x7f0abc0cc800 2024-03-06T07:35:24.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0abbe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc15ce30 con 0x7f0abc0aedc0 2024-03-06T07:35:24.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0aba000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0abc0aedc0 0x7f0abc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0aba000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0abc0aedc0 0x7f0abc0cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38800/0 (socket says 172.21.15.114:38800) 2024-03-06T07:35:24.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0abaa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0abc1561f0 0x7f0abc0ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34976/0 (socket says 172.21.15.114:34976) 2024-03-06T07:35:24.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.503+0000 7f0aba000640 1 -- 172.21.15.114:0/3140816102 learned_addr learned my addr 172.21.15.114:0/3140816102 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:24.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.504+0000 7f0aba000640 1 -- 172.21.15.114:0/3140816102 >> v1:172.21.15.177:6789/0 conn(0x7f0abc0cc800 legacy=0x7f0abc15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.504+0000 7f0aba000640 1 -- 172.21.15.114:0/3140816102 >> v1:172.21.15.114:6789/0 conn(0x7f0abc1561f0 legacy=0x7f0abc0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.504+0000 7f0ab9600640 1 -- 172.21.15.114:0/3140816102 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696720420 0 0) 0x7f0abc15c240 con 0x7f0abc1561f0 2024-03-06T07:35:24.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.504+0000 7f0aba000640 1 -- 172.21.15.114:0/3140816102 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc15c7e0 con 0x7f0abc0aedc0 2024-03-06T07:35:24.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.505+0000 7f0aba000640 1 --2- 172.21.15.114:0/3140816102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc0cc2c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f0abc15cfb0 tx=0x7f0aa802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd3529d185df05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.505+0000 7f0ab9600640 1 -- 172.21.15.114:0/3140816102 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8034070 con 0x7f0abc0aedc0 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.505+0000 7f0ab9600640 1 -- 172.21.15.114:0/3140816102 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8030840 con 0x7f0abc0aedc0 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.505+0000 7f0ab9600640 1 -- 172.21.15.114:0/3140816102 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8030c10 con 0x7f0abc0aedc0 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.506+0000 7f0abbe00640 1 -- 172.21.15.114:0/3140816102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 msgr2=0x7f0abc0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.506+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3140816102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc0cc2c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f0abc15cfb0 tx=0x7f0aa802b980 comp rx=0 tx=0).stop 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.506+0000 7f0abbe00640 1 -- 172.21.15.114:0/3140816102 shutdown_connections 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.506+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3140816102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc0cc2c0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.506+0000 7f0abbe00640 1 -- 172.21.15.114:0/3140816102 >> 172.21.15.114:0/3140816102 conn(0x7f0abc06afc0 msgr2=0x7f0abc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.506+0000 7f0abbe00640 1 -- 172.21.15.114:0/3140816102 shutdown_connections 2024-03-06T07:35:24.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.506+0000 7f0abbe00640 1 -- 172.21.15.114:0/3140816102 wait complete. 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.507+0000 7f0abbe00640 1 Processor -- start 2024-03-06T07:35:24.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.508+0000 7f0abbe00640 1 -- start start 2024-03-06T07:35:24.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc1f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0abc0cc800 0x7f0abc14c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abbe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0abc1561f0 0x7f0abc152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0ccc30 con 0x7f0abc0aedc0 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc06e360 con 0x7f0abc0cc800 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abbe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc1565d0 con 0x7f0abc1561f0 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc1f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc1f4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38804/0 (socket says 172.21.15.114:38804) 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 -- 172.21.15.114:0/3034821890 learned_addr learned my addr 172.21.15.114:0/3034821890 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0aba000640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0abc0cc800 0x7f0abc14c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 -- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0abc1561f0 msgr2=0x7f0abc152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abb400640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0abc1561f0 0x7f0abc152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0abc1561f0 0x7f0abc152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 -- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0abc0cc800 msgr2=0x7f0abc14c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0abc0cc800 0x7f0abc14c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.509+0000 7f0abaa00640 1 -- 172.21.15.114:0/3034821890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc157690 con 0x7f0abc0aedc0 2024-03-06T07:35:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.510+0000 7f0abaa00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc1f4c50 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00c430 tx=0x7f0aac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.510+0000 7f0ab2a00640 1 -- 172.21.15.114:0/3034821890 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac0074e0 con 0x7f0abc0aedc0 2024-03-06T07:35:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.510+0000 7f0ab2a00640 1 -- 172.21.15.114:0/3034821890 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac0047e0 con 0x7f0abc0aedc0 2024-03-06T07:35:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.510+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc1f4170 con 0x7f0abc0aedc0 2024-03-06T07:35:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.510+0000 7f0ab2a00640 1 -- 172.21.15.114:0/3034821890 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac004bd0 con 0x7f0abc0aedc0 2024-03-06T07:35:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.510+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc1572b0 con 0x7f0abc0aedc0 2024-03-06T07:35:24.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.511+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc04f330 con 0x7f0abc0aedc0 2024-03-06T07:35:24.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.512+0000 7f0ab2a00640 1 -- 172.21.15.114:0/3034821890 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0aac011070 con 0x7f0abc0aedc0 2024-03-06T07:35:24.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.514+0000 7f0ab2a00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.514+0000 7f0ab2a00640 1 -- 172.21.15.114:0/3034821890 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f0aac089f20 con 0x7f0abc0aedc0 2024-03-06T07:35:24.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.514+0000 7f0aba000640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.514+0000 7f0aba000640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8036000 tx=0x7f0aa8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.518+0000 7f0ab2a00640 1 -- 172.21.15.114:0/3034821890 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0aac058d40 con 0x7f0abc0aedc0 2024-03-06T07:35:24.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.713+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0abc0b2b10 con 0x7f0abc0aedc0 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.715+0000 7f0ab2a00640 1 -- 172.21.15.114:0/3034821890 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+17089 (secure 0 0 0) 0x7f0aac058640 con 0x7f0abc0aedc0 2024-03-06T07:35:24.718 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:24.718 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":114,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:38.169073+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.177:6833/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.114:6837/860883596":"2024-03-07T07:34:21.018930+0000","172.21.15.114:6835/860883596":"2024-03-07T07:34:21.018930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:24.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9c06b4d0 msgr2=0x7f0a9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8036000 tx=0x7f0aa8008030 comp rx=0 tx=0).stop 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 msgr2=0x7f0abc1f4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc1f4c50 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00c430 tx=0x7f0aac00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 shutdown_connections 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0abc1561f0 0x7f0abc152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0abc0cc800 0x7f0abc14c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 --2- 172.21.15.114:0/3034821890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0abc0aedc0 0x7f0abc1f4c50 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.717+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 >> 172.21.15.114:0/3034821890 conn(0x7f0abc06afc0 msgr2=0x7f0abc0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.718+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 shutdown_connections 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.718+0000 7f0abbe00640 1 -- 172.21.15.114:0/3034821890 wait complete. 2024-03-06T07:35:24.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:04.207971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:05.420538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:24.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:24.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.944+0000 7fa479000640 1 Processor -- start 2024-03-06T07:35:24.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.944+0000 7fa479000640 1 -- start start 2024-03-06T07:35:24.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.945+0000 7fa479000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 0x7fa47406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.945+0000 7fa479000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4740d1260 con 0x7fa47406fef0 2024-03-06T07:35:24.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.946+0000 7fa479000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47406c320 con 0x7fa47415e000 2024-03-06T07:35:24.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.946+0000 7fa479000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47406c0d0 con 0x7fa47415e3e0 2024-03-06T07:35:24.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.946+0000 7fa46be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa47406fef0 0x7fa4740ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34988/0 (socket says 172.21.15.114:34988) 2024-03-06T07:35:24.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.946+0000 7fa46be00640 1 -- 172.21.15.114:0/1376041200 learned_addr learned my addr 172.21.15.114:0/1376041200 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:24.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.946+0000 7fa472800640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 0x7fa47406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.946+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748317506 0 0) 0x7fa4740d1260 con 0x7fa47406fef0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.946+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa458003520 con 0x7fa47406fef0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4061789920 0 0) 0x7fa458003520 con 0x7fa47406fef0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 msgr2=0x7fa47406f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 0x7fa47406f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 >> v1:172.21.15.177:6789/0 conn(0x7fa47415e000 legacy=0x7fa47406f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47406c600 con 0x7fa47406fef0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fa45c001670 con 0x7fa47406fef0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 >> v1:172.21.15.114:6789/0 conn(0x7fa47406fef0 legacy=0x7fa4740ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4580039e0 0x7fa458003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 0x7fa458004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa458004d40 0x7fa4580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47406c0d0 con 0x7fa458004d40 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45800baa0 con 0x7fa4580039e0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45800bca0 con 0x7fa47415e3e0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa472800640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa458004d40 0x7fa4580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.947+0000 7fa471e00640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4580039e0 0x7fa458003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.948+0000 7fa46be00640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 0x7fa458004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.948+0000 7fa472800640 1 -- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 msgr2=0x7fa458004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.948+0000 7fa472800640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa47415e3e0 0x7fa458004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.948+0000 7fa472800640 1 -- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4580039e0 msgr2=0x7fa458003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.948+0000 7fa472800640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4580039e0 0x7fa458003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.948+0000 7fa472800640 1 -- 172.21.15.114:0/1376041200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa458003520 con 0x7fa458004d40 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.949+0000 7fa472800640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa458004d40 0x7fa4580051c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa47415e860 tx=0x7fa458005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108b0de0f04c78a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.949+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c039070 con 0x7fa458004d40 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.949+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c02cb90 con 0x7fa458004d40 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.949+0000 7fa471400640 1 -- 172.21.15.114:0/1376041200 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c009c10 con 0x7fa458004d40 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.950+0000 7fa479000640 1 -- 172.21.15.114:0/1376041200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa458004d40 msgr2=0x7fa4580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.950+0000 7fa479000640 1 --2- 172.21.15.114:0/1376041200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa458004d40 0x7fa4580051c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa47415e860 tx=0x7fa458005830 comp rx=0 tx=0).stop 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.950+0000 7fa472800640 1 -- 172.21.15.114:0/1376041200 reap_dead start 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.950+0000 7fa479000640 1 -- 172.21.15.114:0/1376041200 shutdown_connections 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.950+0000 7fa479000640 1 -- 172.21.15.114:0/1376041200 >> 172.21.15.114:0/1376041200 conn(0x7fa4740b3800 msgr2=0x7fa4740b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.950+0000 7fa479000640 1 -- 172.21.15.114:0/1376041200 shutdown_connections 2024-03-06T07:35:24.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.950+0000 7fa479000640 1 -- 172.21.15.114:0/1376041200 wait complete. 2024-03-06T07:35:24.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.951+0000 7fa479000640 1 Processor -- start 2024-03-06T07:35:24.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.952+0000 7fa479000640 1 -- start start 2024-03-06T07:35:24.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.952+0000 7fa479000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa458004d40 0x7fa4741516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.952+0000 7fa479000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa474152220 0x7fa47415f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa479000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa47415f910 0x7fa4741f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa479000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47406f130 con 0x7fa47415f910 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa479000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47406e060 con 0x7fa458004d40 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa479000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47406f2b0 con 0x7fa474152220 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa47415f910 0x7fa4741f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa47415f910 0x7fa4741f79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38830/0 (socket says 172.21.15.114:38830) 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 -- 172.21.15.114:0/3653225415 learned_addr learned my addr 172.21.15.114:0/3653225415 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa471e00640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa458004d40 0x7fa4741516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa46be00640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa474152220 0x7fa47415f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 -- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa474152220 msgr2=0x7fa47415f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa474152220 0x7fa47415f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 -- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa458004d40 msgr2=0x7fa4741516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa458004d40 0x7fa4741516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.953+0000 7fa472800640 1 -- 172.21.15.114:0/3653225415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4740c8610 con 0x7fa47415f910 2024-03-06T07:35:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.954+0000 7fa472800640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa47415f910 0x7fa4741f79f0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00e230 tx=0x7fa46c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.954+0000 7fa46a000640 1 -- 172.21.15.114:0/3653225415 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c042020 con 0x7fa47415f910 2024-03-06T07:35:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.954+0000 7fa46a000640 1 -- 172.21.15.114:0/3653225415 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c03a040 con 0x7fa47415f910 2024-03-06T07:35:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.954+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa474151d40 con 0x7fa47415f910 2024-03-06T07:35:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.954+0000 7fa46a000640 1 -- 172.21.15.114:0/3653225415 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c039070 con 0x7fa47415f910 2024-03-06T07:35:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.954+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47415ebd0 con 0x7fa47415f910 2024-03-06T07:35:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.955+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47404f330 con 0x7fa47415f910 2024-03-06T07:35:24.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.956+0000 7fa46a000640 1 -- 172.21.15.114:0/3653225415 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa46c045020 con 0x7fa47415f910 2024-03-06T07:35:24.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.958+0000 7fa46a000640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.959+0000 7fa46a000640 1 -- 172.21.15.114:0/3653225415 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fa46c0ac360 con 0x7fa47415f910 2024-03-06T07:35:24.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.959+0000 7fa471e00640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.959+0000 7fa471e00640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa45006b4d0 0x7fa45006d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa464000c00 tx=0x7fa464009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:24.962+0000 7fa46a000640 1 -- 172.21.15.114:0/3653225415 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa46c07b0e0 con 0x7fa47415f910 2024-03-06T07:35:25.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.159+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa4740b7c50 con 0x7fa47415f910 2024-03-06T07:35:25.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.160+0000 7fa46a000640 1 -- 172.21.15.114:0/3653225415 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa46c07a9e0 con 0x7fa47415f910 2024-03-06T07:35:25.163 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:35:25.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa45006b4d0 msgr2=0x7fa45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa45006b4d0 0x7fa45006d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa464000c00 tx=0x7fa464009040 comp rx=0 tx=0).stop 2024-03-06T07:35:25.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa47415f910 msgr2=0x7fa4741f79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa47415f910 0x7fa4741f79f0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00e230 tx=0x7fa46c009750 comp rx=0 tx=0).stop 2024-03-06T07:35:25.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 shutdown_connections 2024-03-06T07:35:25.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa47415f910 0x7fa4741f79f0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa474152220 0x7fa47415f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 --2- 172.21.15.114:0/3653225415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa458004d40 0x7fa4741516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.162+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 >> 172.21.15.114:0/3653225415 conn(0x7fa4740b3800 msgr2=0x7fa4740d1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.163+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 shutdown_connections 2024-03-06T07:35:25.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.163+0000 7fa479000640 1 -- 172.21.15.114:0/3653225415 wait complete. 2024-03-06T07:35:25.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:25.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.378+0000 7ff8b9200640 1 Processor -- start 2024-03-06T07:35:25.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.378+0000 7ff8b9200640 1 -- start start 2024-03-06T07:35:25.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.380+0000 7ff8b9200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8b40b3530 0x7ff8b415b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.380+0000 7ff8b9200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff8b40b3910 0x7ff8b40acf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.380+0000 7ff8b9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40ad4a0 0x7ff8b40ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.380+0000 7ff8b9200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b415bcc0 con 0x7ff8b40b3530 2024-03-06T07:35:25.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.380+0000 7ff8b9200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b40ce690 con 0x7ff8b40b3910 2024-03-06T07:35:25.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.380+0000 7ff8b9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40ce970 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8b40b3530 0x7ff8b415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8b40b3530 0x7ff8b415b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38832/0 (socket says 172.21.15.114:38832) 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1079406660 learned_addr learned my addr 172.21.15.114:0/1079406660 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b2800640 1 --2- 172.21.15.114:0/1079406660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40ad4a0 0x7ff8b40ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8a9400640 1 --2- 172.21.15.114:0/1079406660 >> v2:172.21.15.177:3300/0 conn(0x7ff8b40b3910 0x7ff8b40acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1079406660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40ad4a0 msgr2=0x7ff8b40ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 --2- 172.21.15.114:0/1079406660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40ad4a0 0x7ff8b40ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1079406660 >> v2:172.21.15.177:3300/0 conn(0x7ff8b40b3910 msgr2=0x7ff8b40acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 --2- 172.21.15.114:0/1079406660 >> v2:172.21.15.177:3300/0 conn(0x7ff8b40b3910 0x7ff8b40acf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1079406660 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b40cec50 con 0x7ff8b40b3530 2024-03-06T07:35:25.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.381+0000 7ff8a9400640 1 --2- 172.21.15.114:0/1079406660 >> v2:172.21.15.177:3300/0 conn(0x7ff8b40b3910 0x7ff8b40acf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.382+0000 7ff8b1e00640 1 --2- 172.21.15.114:0/1079406660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40b3530 0x7ff8b415b780 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4074ae0 tx=0x7ff8a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48cc411b909b030 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b1400640 1 -- 172.21.15.114:0/1079406660 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4035020 con 0x7ff8b40b3530 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b1400640 1 -- 172.21.15.114:0/1079406660 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a4031870 con 0x7ff8b40b3530 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b1400640 1 -- 172.21.15.114:0/1079406660 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4031c40 con 0x7ff8b40b3530 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 -- 172.21.15.114:0/1079406660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40b3530 msgr2=0x7ff8b415b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1079406660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40b3530 0x7ff8b415b780 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4074ae0 tx=0x7ff8a402b910 comp rx=0 tx=0).stop 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 -- 172.21.15.114:0/1079406660 shutdown_connections 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1079406660 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40ad4a0 0x7ff8b40ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1079406660 >> v2:172.21.15.177:3300/0 conn(0x7ff8b40b3910 0x7ff8b40acf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1079406660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40b3530 0x7ff8b415b780 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 -- 172.21.15.114:0/1079406660 >> 172.21.15.114:0/1079406660 conn(0x7ff8b40b1940 msgr2=0x7ff8b40b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 -- 172.21.15.114:0/1079406660 shutdown_connections 2024-03-06T07:35:25.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.383+0000 7ff8b9200640 1 -- 172.21.15.114:0/1079406660 wait complete. 2024-03-06T07:35:25.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.385+0000 7ff8b9200640 1 Processor -- start 2024-03-06T07:35:25.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.385+0000 7ff8b9200640 1 -- start start 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40ad4a0 0x7ff8b414d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40b3530 0x7ff8b4072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b40b3910 0x7ff8b41f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b9200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b415aa30 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b9200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40cf210 con 0x7ff8b40b3910 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b415abb0 con 0x7ff8b40b3530 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40ad4a0 0x7ff8b414d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40ad4a0 0x7ff8b414d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38834/0 (socket says 172.21.15.114:38834) 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.386+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1138789490 learned_addr learned my addr 172.21.15.114:0/1138789490 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b2800640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b40b3910 0x7ff8b41f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8a9400640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40b3530 0x7ff8b4072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40b3530 msgr2=0x7ff8b4072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b1e00640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40b3530 0x7ff8b4072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b40b3910 msgr2=0x7ff8b41f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b1e00640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b40b3910 0x7ff8b41f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b1e00640 1 -- 172.21.15.114:0/1138789490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4071fb0 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b2800640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b40b3910 0x7ff8b41f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8b1e00640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40ad4a0 0x7ff8b414d2a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000c00 tx=0x7ff8a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.387+0000 7ff8aaa00640 1 -- 172.21.15.114:0/1138789490 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a403ad60 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.388+0000 7ff8aaa00640 1 -- 172.21.15.114:0/1138789490 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a40443f0 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.388+0000 7ff8aaa00640 1 -- 172.21.15.114:0/1138789490 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a40447c0 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.388+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b41f7c40 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.388+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b4072270 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.389+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b404f330 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.390+0000 7ff8aaa00640 1 -- 172.21.15.114:0/1138789490 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8a404b060 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.390+0000 7ff8aaa00640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.391+0000 7ff8aaa00640 1 -- 172.21.15.114:0/1138789490 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7ff8a40ac440 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.391+0000 7ff8a9400640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.391+0000 7ff8a9400640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff89006b4d0 0x7ff89006d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff8b41f70e0 tx=0x7ff89c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.395+0000 7ff8aaa00640 1 -- 172.21.15.114:0/1138789490 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8a403f070 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.591+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8b40cc070 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.592+0000 7ff8aaa00640 1 -- 172.21.15.114:0/1138789490 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8a402f320 con 0x7ff8b40ad4a0 2024-03-06T07:35:25.595 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:35:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff89006b4d0 msgr2=0x7ff89006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff89006b4d0 0x7ff89006d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff8b41f70e0 tx=0x7ff89c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40ad4a0 msgr2=0x7ff8b414d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40ad4a0 0x7ff8b414d2a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000c00 tx=0x7ff8a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 shutdown_connections 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8b40b3910 0x7ff8b41f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8b40b3530 0x7ff8b4072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff89006b4d0 0x7ff89006d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 --2- 172.21.15.114:0/1138789490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8b40ad4a0 0x7ff8b414d2a0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 >> 172.21.15.114:0/1138789490 conn(0x7ff8b40b1940 msgr2=0x7ff8b41542e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 shutdown_connections 2024-03-06T07:35:25.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.595+0000 7ff8b9200640 1 -- 172.21.15.114:0/1138789490 wait complete. 2024-03-06T07:35:25.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:25.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.868+0000 7fdb35800640 1 Processor -- start 2024-03-06T07:35:25.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.868+0000 7fdb35800640 1 -- start start 2024-03-06T07:35:25.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb35800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdb300aef20 0x7fdb300ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb35800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3015c1f0 con 0x7fdb3015be10 2024-03-06T07:35:25.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb35800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3006cac0 con 0x7fdb3006da40 2024-03-06T07:35:25.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb35800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb300cd200 con 0x7fdb300aef20 2024-03-06T07:35:25.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb2fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdb3015be10 0x7fdb3006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:34996/0 (socket says 172.21.15.114:34996) 2024-03-06T07:35:25.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb2fe00640 1 -- 172.21.15.114:0/2770728219 learned_addr learned my addr 172.21.15.114:0/2770728219 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:25.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791285025 0 0) 0x7fdb3015c1f0 con 0x7fdb3015be10 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb10003520 con 0x7fdb3015be10 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.870+0000 7fdb34e00640 1 --2- 172.21.15.114:0/2770728219 >> v2:172.21.15.177:3300/0 conn(0x7fdb300aef20 0x7fdb300ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219179902 0 0) 0x7fdb10003520 con 0x7fdb3015be10 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 >> v2:172.21.15.177:3300/0 conn(0x7fdb300aef20 msgr2=0x7fdb300ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 --2- 172.21.15.114:0/2770728219 >> v2:172.21.15.177:3300/0 conn(0x7fdb300aef20 0x7fdb300ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 >> v1:172.21.15.177:6789/0 conn(0x7fdb3006da40 legacy=0x7fdb30071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3006cda0 con 0x7fdb3015be10 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fdb20001670 con 0x7fdb3015be10 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb34e00640 1 --2- 172.21.15.114:0/2770728219 >> v2:172.21.15.177:3300/0 conn(0x7fdb300aef20 0x7fdb300ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 >> v1:172.21.15.114:6789/0 conn(0x7fdb3015be10 legacy=0x7fdb3006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100039b0 0x7fdb10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb10004800 0x7fdb10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb100052e0 0x7fdb1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300cd200 con 0x7fdb100052e0 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1000c1b0 con 0x7fdb100039b0 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1000c3b0 con 0x7fdb10004800 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb34e00640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb100052e0 0x7fdb1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb2fe00640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100039b0 0x7fdb10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb34e00640 1 -- 172.21.15.114:0/2770728219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb10004800 msgr2=0x7fdb10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.871+0000 7fdb34e00640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb10004800 0x7fdb10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.872+0000 7fdb34e00640 1 -- 172.21.15.114:0/2770728219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100039b0 msgr2=0x7fdb10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.872+0000 7fdb34e00640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100039b0 0x7fdb10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.872+0000 7fdb34e00640 1 -- 172.21.15.114:0/2770728219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb10003520 con 0x7fdb100052e0 2024-03-06T07:35:25.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.872+0000 7fdb34e00640 1 -- 172.21.15.114:0/2770728219 reap_dead start 2024-03-06T07:35:25.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.873+0000 7fdb34e00640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb100052e0 0x7fdb1000b9c0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fdb280073b0 tx=0x7fdb10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77fef5f627ccfb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.873+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28039070 con 0x7fdb100052e0 2024-03-06T07:35:25.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.873+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2802cd90 con 0x7fdb100052e0 2024-03-06T07:35:25.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.873+0000 7fdb2f400640 1 -- 172.21.15.114:0/2770728219 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28009c80 con 0x7fdb100052e0 2024-03-06T07:35:25.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.873+0000 7fdb35800640 1 -- 172.21.15.114:0/2770728219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb100052e0 msgr2=0x7fdb1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.873+0000 7fdb35800640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb100052e0 0x7fdb1000b9c0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fdb280073b0 tx=0x7fdb10005830 comp rx=0 tx=0).stop 2024-03-06T07:35:25.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.873+0000 7fdb35800640 1 -- 172.21.15.114:0/2770728219 shutdown_connections 2024-03-06T07:35:25.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.874+0000 7fdb35800640 1 --2- 172.21.15.114:0/2770728219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb100052e0 0x7fdb1000b9c0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.874+0000 7fdb35800640 1 -- 172.21.15.114:0/2770728219 >> 172.21.15.114:0/2770728219 conn(0x7fdb3006afc0 msgr2=0x7fdb3006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.874+0000 7fdb35800640 1 -- 172.21.15.114:0/2770728219 shutdown_connections 2024-03-06T07:35:25.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.874+0000 7fdb35800640 1 -- 172.21.15.114:0/2770728219 wait complete. 2024-03-06T07:35:25.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.876+0000 7fdb35800640 1 Processor -- start 2024-03-06T07:35:25.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.877+0000 7fdb35800640 1 -- start start 2024-03-06T07:35:25.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.878+0000 7fdb35800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100052e0 0x7fdb30152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.878+0000 7fdb35800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb30143290 0x7fdb300b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.878+0000 7fdb35800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb300b2eb0 0x7fdb300b3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.878+0000 7fdb35800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300cda90 con 0x7fdb30143290 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.878+0000 7fdb35800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3006d080 con 0x7fdb100052e0 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.878+0000 7fdb35800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300ccc20 con 0x7fdb300b2eb0 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.878+0000 7fdb27400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb30143290 0x7fdb300b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb30143290 0x7fdb300b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38860/0 (socket says 172.21.15.114:38860) 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 -- 172.21.15.114:0/3409179712 learned_addr learned my addr 172.21.15.114:0/3409179712 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb2fe00640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100052e0 0x7fdb30152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb34e00640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb300b2eb0 0x7fdb300b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 -- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb300b2eb0 msgr2=0x7fdb300b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb300b2eb0 0x7fdb300b3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 -- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100052e0 msgr2=0x7fdb30152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100052e0 0x7fdb30152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 -- 172.21.15.114:0/3409179712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb30155e40 con 0x7fdb30143290 2024-03-06T07:35:25.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb34e00640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb300b2eb0 0x7fdb300b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:25.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.879+0000 7fdb27400640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb30143290 0x7fdb300b2970 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800c530 tx=0x7fdb1800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.880+0000 7fdb2cc00640 1 -- 172.21.15.114:0/3409179712 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb180052d0 con 0x7fdb30143290 2024-03-06T07:35:25.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.880+0000 7fdb2cc00640 1 -- 172.21.15.114:0/3409179712 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb18005470 con 0x7fdb30143290 2024-03-06T07:35:25.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.880+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb30155880 con 0x7fdb30143290 2024-03-06T07:35:25.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.880+0000 7fdb2cc00640 1 -- 172.21.15.114:0/3409179712 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18012870 con 0x7fdb30143290 2024-03-06T07:35:25.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.880+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3014fb40 con 0x7fdb30143290 2024-03-06T07:35:25.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.881+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb301555f0 con 0x7fdb30143290 2024-03-06T07:35:25.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.882+0000 7fdb2cc00640 1 -- 172.21.15.114:0/3409179712 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb18012a10 con 0x7fdb30143290 2024-03-06T07:35:25.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.883+0000 7fdb2cc00640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:25.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.883+0000 7fdb2cc00640 1 -- 172.21.15.114:0/3409179712 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fdb18089fb0 con 0x7fdb30143290 2024-03-06T07:35:25.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.883+0000 7fdb2fe00640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:25.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.884+0000 7fdb2fe00640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fdb20001050 tx=0x7fdb20009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:25.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:25.887+0000 7fdb2cc00640 1 -- 172.21.15.114:0/3409179712 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb18058d30 con 0x7fdb30143290 2024-03-06T07:35:26.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.088+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdb3004f330 con 0x7fdb30143290 2024-03-06T07:35:26.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.089+0000 7fdb2cc00640 1 -- 172.21.15.114:0/3409179712 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdb18058630 con 0x7fdb30143290 2024-03-06T07:35:26.092 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:35:26.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.091+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0c06b4d0 msgr2=0x7fdb0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.091+0000 7fdb35800640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fdb20001050 tx=0x7fdb20009040 comp rx=0 tx=0).stop 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.091+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb30143290 msgr2=0x7fdb300b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.091+0000 7fdb35800640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb30143290 0x7fdb300b2970 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800c530 tx=0x7fdb1800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 shutdown_connections 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb300b2eb0 0x7fdb300b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb0c06b4d0 0x7fdb0c06d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb30143290 0x7fdb300b2970 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 --2- 172.21.15.114:0/3409179712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb100052e0 0x7fdb30152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 >> 172.21.15.114:0/3409179712 conn(0x7fdb3006afc0 msgr2=0x7fdb300cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 shutdown_connections 2024-03-06T07:35:26.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.092+0000 7fdb35800640 1 -- 172.21.15.114:0/3409179712 wait complete. 2024-03-06T07:35:26.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:35:26.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.302+0000 7f82b6a00640 1 Processor -- start 2024-03-06T07:35:26.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.302+0000 7f82b6a00640 1 -- start start 2024-03-06T07:35:26.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82b015e200 0x7f82b00d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b6a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b00d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b6a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b00b5240 con 0x7f82b015e5e0 2024-03-06T07:35:26.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b6a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b00d2ec0 con 0x7f82b015e200 2024-03-06T07:35:26.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b6a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00cf4d0 con 0x7f82b00b2e50 2024-03-06T07:35:26.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b5600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82b015e200 0x7f82b00d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b5600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82b015e200 0x7f82b00d4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38862/0 (socket says 172.21.15.114:38862) 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b5600640 1 -- 172.21.15.114:0/3727108863 learned_addr learned my addr 172.21.15.114:0/3727108863 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.304+0000 7f82b4c00640 1 --2- 172.21.15.114:0/3727108863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b00d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.305+0000 7f82b5600640 1 -- 172.21.15.114:0/3727108863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 msgr2=0x7f82b00d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.305+0000 7f82b5600640 1 --2- 172.21.15.114:0/3727108863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b00d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.305+0000 7f82b5600640 1 -- 172.21.15.114:0/3727108863 >> v1:172.21.15.177:6789/0 conn(0x7f82b015e5e0 legacy=0x7f82b00d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.305+0000 7f82b5600640 1 -- 172.21.15.114:0/3727108863 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00b5500 con 0x7f82b015e200 2024-03-06T07:35:26.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.307+0000 7f82b5600640 1 --2- 172.21.15.114:0/3727108863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e200 0x7f82b00d4850 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009870 tx=0x7f82a402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9473f8464429d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.308+0000 7f82afe00640 1 -- 172.21.15.114:0/3727108863 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a402f840 con 0x7f82b015e200 2024-03-06T07:35:26.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.308+0000 7f82afe00640 1 -- 172.21.15.114:0/3727108863 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a402f9e0 con 0x7f82b015e200 2024-03-06T07:35:26.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.308+0000 7f82afe00640 1 -- 172.21.15.114:0/3727108863 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a402fdb0 con 0x7f82b015e200 2024-03-06T07:35:26.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.308+0000 7f82b6a00640 1 -- 172.21.15.114:0/3727108863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e200 msgr2=0x7f82b00d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.308+0000 7f82b6a00640 1 --2- 172.21.15.114:0/3727108863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e200 0x7f82b00d4850 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009870 tx=0x7f82a402ba20 comp rx=0 tx=0).stop 2024-03-06T07:35:26.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.309+0000 7f82b6a00640 1 -- 172.21.15.114:0/3727108863 shutdown_connections 2024-03-06T07:35:26.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.309+0000 7f82b6a00640 1 --2- 172.21.15.114:0/3727108863 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b00d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.309+0000 7f82b6a00640 1 --2- 172.21.15.114:0/3727108863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e200 0x7f82b00d4850 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.309+0000 7f82b6a00640 1 -- 172.21.15.114:0/3727108863 >> 172.21.15.114:0/3727108863 conn(0x7f82b00d3d30 msgr2=0x7f82b00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.309+0000 7f82b6a00640 1 -- 172.21.15.114:0/3727108863 shutdown_connections 2024-03-06T07:35:26.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.309+0000 7f82b6a00640 1 -- 172.21.15.114:0/3727108863 wait complete. 2024-03-06T07:35:26.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.310+0000 7f82b6a00640 1 Processor -- start 2024-03-06T07:35:26.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.310+0000 7f82b6a00640 1 -- start start 2024-03-06T07:35:26.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.311+0000 7f82b6a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b01f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.311+0000 7f82b6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82b015e200 0x7f82b015be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.311+0000 7f82b6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e5e0 0x7f82b00c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.311+0000 7f82b6a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00cfd60 con 0x7f82b015e5e0 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.311+0000 7f82b6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00cfa80 con 0x7f82b015e200 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.311+0000 7f82b6a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00b4650 con 0x7f82b00b2e50 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e5e0 0x7f82b00c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e5e0 0x7f82b00c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38866/0 (socket says 172.21.15.114:38866) 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 -- 172.21.15.114:0/2562793682 learned_addr learned my addr 172.21.15.114:0/2562793682 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b5600640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b01f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b4c00640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82b015e200 0x7f82b015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 -- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 msgr2=0x7f82b01f5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b01f5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 -- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82b015e200 msgr2=0x7f82b015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82b015e200 0x7f82b015be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 -- 172.21.15.114:0/2562793682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b015c340 con 0x7f82b015e5e0 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.312+0000 7f82b6000640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e5e0 0x7f82b00c9990 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c6b0 tx=0x7f82a800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.313+0000 7f82ad600640 1 -- 172.21.15.114:0/2562793682 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8003b70 con 0x7f82b015e5e0 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.313+0000 7f82ad600640 1 -- 172.21.15.114:0/2562793682 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a8003d30 con 0x7f82b015e5e0 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.313+0000 7f82ad600640 1 -- 172.21.15.114:0/2562793682 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8005310 con 0x7f82b015e5e0 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.313+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b00ca480 con 0x7f82b015e5e0 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.313+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b00c7d90 con 0x7f82b015e5e0 2024-03-06T07:35:26.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.314+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b004f330 con 0x7f82b015e5e0 2024-03-06T07:35:26.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.315+0000 7f82ad600640 1 -- 172.21.15.114:0/2562793682 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82a80054b0 con 0x7f82b015e5e0 2024-03-06T07:35:26.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.316+0000 7f82ad600640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f829006b500 0x7f829006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.316+0000 7f82ad600640 1 -- 172.21.15.114:0/2562793682 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f82a808a120 con 0x7f82b015e5e0 2024-03-06T07:35:26.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.316+0000 7f82b5600640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f829006b500 0x7f829006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.317+0000 7f82b5600640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f829006b500 0x7f829006d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f82a4036000 tx=0x7f82a4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.324+0000 7f82ad600640 1 -- 172.21.15.114:0/2562793682 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82a8058ea0 con 0x7f82b015e5e0 2024-03-06T07:35:26.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.533+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f82b008d4c0 con 0x7f82b015e5e0 2024-03-06T07:35:26.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.534+0000 7f82ad600640 1 -- 172.21.15.114:0/2562793682 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v114) v1 ==== 80+0+246 (secure 0 0 0) 0x7f82a80587a0 con 0x7f82b015e5e0 2024-03-06T07:35:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.177:6835/2151340900 2024-03-07T07:34:21.054096+0000 2024-03-06T07:35:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.177:6833/2151340900 2024-03-07T07:34:21.054096+0000 2024-03-06T07:35:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6837/860883596 2024-03-07T07:34:21.018930+0000 2024-03-06T07:35:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6835/860883596 2024-03-07T07:34:21.018930+0000 2024-03-06T07:35:26.538 INFO:teuthology.orchestra.run.smithi114.stderr:listed 4 entries 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f829006b500 msgr2=0x7f829006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f829006b500 0x7f829006d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f82a4036000 tx=0x7f82a4008030 comp rx=0 tx=0).stop 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e5e0 msgr2=0x7f82b00c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e5e0 0x7f82b00c9990 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f82a800c6b0 tx=0x7f82a800cb70 comp rx=0 tx=0).stop 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 shutdown_connections 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f829006b500 0x7f829006d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b015e5e0 0x7f82b00c9990 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82b015e200 0x7f82b015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 --2- 172.21.15.114:0/2562793682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f82b00b2e50 0x7f82b01f5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 >> 172.21.15.114:0/2562793682 conn(0x7f82b00d3d30 msgr2=0x7f82b00d0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 shutdown_connections 2024-03-06T07:35:26.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:26.537+0000 7f82b6a00640 1 -- 172.21.15.114:0/2562793682 wait complete. 2024-03-06T07:35:26.558 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:35:26.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:26.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:26.650 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-03-06T07:35:26.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:26.651 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:35:26.697 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:35:26.697 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:26.755 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:26.755 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:35:26.755 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:26.803 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:26.803 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:35:26.804 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:26.859 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:26.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:27.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.111+0000 7f503be00640 1 Processor -- start 2024-03-06T07:35:27.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.111+0000 7f503be00640 1 -- start start 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.111+0000 7f503be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f503c0d2360 0x7f503c0d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.111+0000 7f503be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.111+0000 7f503be00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c161960 con 0x7f503c161580 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.111+0000 7f503be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503c0d2ce0 con 0x7f503c0d2360 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.111+0000 7f503be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0cf6f0 con 0x7f503c0b3ec0 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.112+0000 7f5032000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f503c0d2360 0x7f503c0d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.112+0000 7f5032000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f503c0d2360 0x7f503c0d27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38874/0 (socket says 172.21.15.114:38874) 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.112+0000 7f5032000640 1 -- 172.21.15.114:0/2594896751 learned_addr learned my addr 172.21.15.114:0/2594896751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.112+0000 7f5032000640 1 -- 172.21.15.114:0/2594896751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 msgr2=0x7f503c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.112+0000 7f5032000640 1 --2- 172.21.15.114:0/2594896751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.112+0000 7f5032000640 1 -- 172.21.15.114:0/2594896751 >> v1:172.21.15.177:6789/0 conn(0x7f503c161580 legacy=0x7f503c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.112+0000 7f5032000640 1 -- 172.21.15.114:0/2594896751 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0cf9d0 con 0x7f503c0d2360 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.113+0000 7f5032000640 1 --2- 172.21.15.114:0/2594896751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c0d2360 0x7f503c0d27a0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5024009e50 tx=0x7f502402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8e0c681d69cf12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.113+0000 7f503a000640 1 -- 172.21.15.114:0/2594896751 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502402ca70 con 0x7f503c0d2360 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.113+0000 7f503a000640 1 -- 172.21.15.114:0/2594896751 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502402cc10 con 0x7f503c0d2360 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.113+0000 7f503a000640 1 -- 172.21.15.114:0/2594896751 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50240343f0 con 0x7f503c0d2360 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 -- 172.21.15.114:0/2594896751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c0d2360 msgr2=0x7f503c0d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 --2- 172.21.15.114:0/2594896751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c0d2360 0x7f503c0d27a0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5024009e50 tx=0x7f502402bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 -- 172.21.15.114:0/2594896751 shutdown_connections 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 --2- 172.21.15.114:0/2594896751 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 --2- 172.21.15.114:0/2594896751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c0d2360 0x7f503c0d27a0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 -- 172.21.15.114:0/2594896751 >> 172.21.15.114:0/2594896751 conn(0x7f503c0d3d80 msgr2=0x7f503c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 -- 172.21.15.114:0/2594896751 shutdown_connections 2024-03-06T07:35:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.114+0000 7f503be00640 1 -- 172.21.15.114:0/2594896751 wait complete. 2024-03-06T07:35:27.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.116+0000 7f503be00640 1 Processor -- start 2024-03-06T07:35:27.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.116+0000 7f503be00640 1 -- start start 2024-03-06T07:35:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f503c0d2360 0x7f503c15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c161580 0x7f503c0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0b39a0 con 0x7f503c161580 2024-03-06T07:35:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0b36c0 con 0x7f503c0d2360 2024-03-06T07:35:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0cf3a0 con 0x7f503c0b3ec0 2024-03-06T07:35:27.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c161580 0x7f503c0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c161580 0x7f503c0cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38876/0 (socket says 172.21.15.114:38876) 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.117+0000 7f503b400640 1 -- 172.21.15.114:0/50949911 learned_addr learned my addr 172.21.15.114:0/50949911 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503aa00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503b400640 1 -- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 msgr2=0x7f503c0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f5032000640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f503c0d2360 0x7f503c15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503b400640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503b400640 1 -- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f503c0d2360 msgr2=0x7f503c15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503b400640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f503c0d2360 0x7f503c15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503b400640 1 -- 172.21.15.114:0/50949911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0caff0 con 0x7f503c161580 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503aa00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503b400640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c161580 0x7f503c0cba90 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f5034009fd0 tx=0x7f503400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f5033e00640 1 -- 172.21.15.114:0/50949911 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034011830 con 0x7f503c161580 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f5033e00640 1 -- 172.21.15.114:0/50949911 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5034016070 con 0x7f503c161580 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f5033e00640 1 -- 172.21.15.114:0/50949911 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034011bf0 con 0x7f503c161580 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.118+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c0c9af0 con 0x7f503c161580 2024-03-06T07:35:27.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.119+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c1568b0 con 0x7f503c161580 2024-03-06T07:35:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.120+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c04f330 con 0x7f503c161580 2024-03-06T07:35:27.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.121+0000 7f5033e00640 1 -- 172.21.15.114:0/50949911 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f503401b460 con 0x7f503c161580 2024-03-06T07:35:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.121+0000 7f5033e00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f501806b500 0x7f501806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.122+0000 7f5033e00640 1 -- 172.21.15.114:0/50949911 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f503408a020 con 0x7f503c161580 2024-03-06T07:35:27.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.122+0000 7f503aa00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f501806b500 0x7f501806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.122+0000 7f503aa00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f501806b500 0x7f501806d9c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f503c0b6830 tx=0x7f502c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.131+0000 7f5033e00640 1 -- 172.21.15.114:0/50949911 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5034058db0 con 0x7f503c161580 2024-03-06T07:35:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.353+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f503c0d3b60 con 0x7f503c161580 2024-03-06T07:35:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.354+0000 7f5033e00640 1 -- 172.21.15.114:0/50949911 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f50340586b0 con 0x7f503c161580 2024-03-06T07:35:27.357 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:27.357 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:27.357 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:27.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.356+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f501806b500 msgr2=0x7f501806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.356+0000 7f503be00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f501806b500 0x7f501806d9c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f503c0b6830 tx=0x7f502c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.356+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c161580 msgr2=0x7f503c0cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.356+0000 7f503be00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c161580 0x7f503c0cba90 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f5034009fd0 tx=0x7f503400b8e0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.356+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 shutdown_connections 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.357+0000 7f503be00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f501806b500 0x7f501806d9c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.357+0000 7f503be00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f503c161580 0x7f503c0cba90 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.357+0000 7f503be00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f503c0d2360 0x7f503c15bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.357+0000 7f503be00640 1 --2- 172.21.15.114:0/50949911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f503c0b3ec0 0x7f503c0c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.357+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 >> 172.21.15.114:0/50949911 conn(0x7f503c0d3d80 msgr2=0x7f503c15fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.357+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 shutdown_connections 2024-03-06T07:35:27.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.357+0000 7f503be00640 1 -- 172.21.15.114:0/50949911 wait complete. 2024-03-06T07:35:27.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:27.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.580+0000 7fb392600640 1 Processor -- start 2024-03-06T07:35:27.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.580+0000 7fb392600640 1 -- start start 2024-03-06T07:35:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb392600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb38c0d0600 0x7fb38c0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb392600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb38c0d18e0 0x7fb38c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb392600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c15f030 con 0x7fb38c0b2a60 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb392600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38c0cf090 con 0x7fb38c0d0600 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb392600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38c15f310 con 0x7fb38c0d18e0 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb391200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb38c0d0600 0x7fb38c0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb391200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb38c0d0600 0x7fb38c0d4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38888/0 (socket says 172.21.15.114:38888) 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.582+0000 7fb391200640 1 -- 172.21.15.114:0/1768644026 learned_addr learned my addr 172.21.15.114:0/1768644026 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.583+0000 7fb391200640 1 -- 172.21.15.114:0/1768644026 >> v2:172.21.15.177:3300/0 conn(0x7fb38c0d18e0 msgr2=0x7fb38c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.583+0000 7fb391200640 1 --2- 172.21.15.114:0/1768644026 >> v2:172.21.15.177:3300/0 conn(0x7fb38c0d18e0 0x7fb38c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.583+0000 7fb391200640 1 -- 172.21.15.114:0/1768644026 >> v1:172.21.15.177:6789/0 conn(0x7fb38c0b2a60 legacy=0x7fb38c0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.583+0000 7fb391200640 1 -- 172.21.15.114:0/1768644026 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0cea40 con 0x7fb38c0d0600 2024-03-06T07:35:27.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.584+0000 7fb391200640 1 --2- 172.21.15.114:0/1768644026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0d0600 0x7fb38c0d4830 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fb38c08dc30 tx=0x7fb38002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1ae1b87f4a80a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.584+0000 7fb38b400640 1 -- 172.21.15.114:0/1768644026 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380035020 con 0x7fb38c0d0600 2024-03-06T07:35:27.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.584+0000 7fb38b400640 1 -- 172.21.15.114:0/1768644026 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb380031870 con 0x7fb38c0d0600 2024-03-06T07:35:27.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.584+0000 7fb38b400640 1 -- 172.21.15.114:0/1768644026 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380031c40 con 0x7fb38c0d0600 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 -- 172.21.15.114:0/1768644026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0d0600 msgr2=0x7fb38c0d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 --2- 172.21.15.114:0/1768644026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0d0600 0x7fb38c0d4830 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fb38c08dc30 tx=0x7fb38002b910 comp rx=0 tx=0).stop 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 -- 172.21.15.114:0/1768644026 shutdown_connections 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 --2- 172.21.15.114:0/1768644026 >> v2:172.21.15.177:3300/0 conn(0x7fb38c0d18e0 0x7fb38c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 --2- 172.21.15.114:0/1768644026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0d0600 0x7fb38c0d4830 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 -- 172.21.15.114:0/1768644026 >> 172.21.15.114:0/1768644026 conn(0x7fb38c0d3d10 msgr2=0x7fb38c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 -- 172.21.15.114:0/1768644026 shutdown_connections 2024-03-06T07:35:27.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.585+0000 7fb392600640 1 -- 172.21.15.114:0/1768644026 wait complete. 2024-03-06T07:35:27.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.587+0000 7fb392600640 1 Processor -- start 2024-03-06T07:35:27.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.587+0000 7fb392600640 1 -- start start 2024-03-06T07:35:27.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb392600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0b2a60 0x7fb38c14adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb392600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb38c0d0600 0x7fb38c0c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb392600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb38c0d18e0 0x7fb38c0c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb392600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c15ecf0 con 0x7fb38c0b2a60 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb392600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0b3250 con 0x7fb38c0d0600 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb392600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0cf520 con 0x7fb38c0d18e0 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb391200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0b2a60 0x7fb38c14adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb391200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0b2a60 0x7fb38c14adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38902/0 (socket says 172.21.15.114:38902) 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.588+0000 7fb391200640 1 -- 172.21.15.114:0/1294589224 learned_addr learned my addr 172.21.15.114:0/1294589224 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb38be00640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb38c0d0600 0x7fb38c0c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb391c00640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb38c0d18e0 0x7fb38c0c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb391200640 1 -- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb38c0d18e0 msgr2=0x7fb38c0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb391200640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb38c0d18e0 0x7fb38c0c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb391200640 1 -- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb38c0d0600 msgr2=0x7fb38c0c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb391200640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb38c0d0600 0x7fb38c0c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb391200640 1 -- 172.21.15.114:0/1294589224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c14d870 con 0x7fb38c0b2a60 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb38be00640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb38c0d0600 0x7fb38c0c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb391200640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0b2a60 0x7fb38c14adf0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fb38002b950 tx=0x7fb3800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb388c00640 1 -- 172.21.15.114:0/1294589224 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3800443f0 con 0x7fb38c0b2a60 2024-03-06T07:35:27.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb388c00640 1 -- 172.21.15.114:0/1294589224 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb380044590 con 0x7fb38c0b2a60 2024-03-06T07:35:27.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38c14db10 con 0x7fb38c0b2a60 2024-03-06T07:35:27.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.589+0000 7fb388c00640 1 -- 172.21.15.114:0/1294589224 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380044980 con 0x7fb38c0b2a60 2024-03-06T07:35:27.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.590+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38c14e0d0 con 0x7fb38c0b2a60 2024-03-06T07:35:27.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.591+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38c04f330 con 0x7fb38c0b2a60 2024-03-06T07:35:27.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.592+0000 7fb388c00640 1 -- 172.21.15.114:0/1294589224 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb380038020 con 0x7fb38c0b2a60 2024-03-06T07:35:27.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.593+0000 7fb388c00640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb36c06b4d0 0x7fb36c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.593+0000 7fb388c00640 1 -- 172.21.15.114:0/1294589224 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fb3800ac630 con 0x7fb38c0b2a60 2024-03-06T07:35:27.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.593+0000 7fb38be00640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb36c06b4d0 0x7fb36c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.594+0000 7fb38be00640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb36c06b4d0 0x7fb36c06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb378007fd0 tx=0x7fb378009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.597+0000 7fb388c00640 1 -- 172.21.15.114:0/1294589224 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb38003f070 con 0x7fb38c0b2a60 2024-03-06T07:35:27.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.794+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb38c08d7e0 con 0x7fb38c0b2a60 2024-03-06T07:35:27.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.798+0000 7fb388c00640 1 -- 172.21.15.114:0/1294589224 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+17089 (secure 0 0 0) 0x7fb38007b3b0 con 0x7fb38c0b2a60 2024-03-06T07:35:27.800 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:27.800 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":114,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:38.169073+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.177:6833/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.114:6837/860883596":"2024-03-07T07:34:21.018930+0000","172.21.15.114:6835/860883596":"2024-03-07T07:34:21.018930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:27.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb36c06b4d0 msgr2=0x7fb36c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb36c06b4d0 0x7fb36c06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb378007fd0 tx=0x7fb378009040 comp rx=0 tx=0).stop 2024-03-06T07:35:27.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0b2a60 msgr2=0x7fb38c14adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0b2a60 0x7fb38c14adf0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fb38002b950 tx=0x7fb3800046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 shutdown_connections 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb38c0d18e0 0x7fb38c0c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb38c0d0600 0x7fb38c0c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb36c06b4d0 0x7fb36c06d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 --2- 172.21.15.114:0/1294589224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb38c0b2a60 0x7fb38c14adf0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.799+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 >> 172.21.15.114:0/1294589224 conn(0x7fb38c0d3d10 msgr2=0x7fb38c06b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.801+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 shutdown_connections 2024-03-06T07:35:27.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:27.801+0000 7fb392600640 1 -- 172.21.15.114:0/1294589224 wait complete. 2024-03-06T07:35:27.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:34:04.207971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:34:05.420538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:27.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.031+0000 7efe28800640 1 Processor -- start 2024-03-06T07:35:28.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.031+0000 7efe28800640 1 -- start start 2024-03-06T07:35:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.033+0000 7efe28800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b5490 0x7efe200cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.033+0000 7efe28800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200b3730 con 0x7efe200d07f0 2024-03-06T07:35:28.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.033+0000 7efe28800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe20067b20 con 0x7efe2015e450 2024-03-06T07:35:28.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.033+0000 7efe28800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe200d4930 con 0x7efe200b5490 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.033+0000 7efe25e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7efe200d07f0 0x7efe200d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58902/0 (socket says 172.21.15.114:58902) 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.033+0000 7efe25e00640 1 -- 172.21.15.114:0/3789536163 learned_addr learned my addr 172.21.15.114:0/3789536163 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.033+0000 7efe26800640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b5490 0x7efe200cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092579988 0 0) 0x7efe200b3730 con 0x7efe200d07f0 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe08003520 con 0x7efe200d07f0 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094975579 0 0) 0x7efe08003520 con 0x7efe200d07f0 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b5490 msgr2=0x7efe200cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b5490 0x7efe200cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 >> v1:172.21.15.177:6789/0 conn(0x7efe2015e450 legacy=0x7efe200cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe200d4b80 con 0x7efe200d07f0 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7efe14001670 con 0x7efe200d07f0 2024-03-06T07:35:28.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 >> v1:172.21.15.114:6789/0 conn(0x7efe200d07f0 legacy=0x7efe200d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe08003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe08004800 0x7efe08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b5490 0x7efe080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200d4930 con 0x7efe080039e0 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0800bac0 con 0x7efe08004800 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.034+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0800bcc0 con 0x7efe200b5490 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.035+0000 7efe25e00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe08003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.035+0000 7efe25e00640 1 -- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b5490 msgr2=0x7efe080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.035+0000 7efe25e00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b5490 0x7efe080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.035+0000 7efe25e00640 1 -- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe08004800 msgr2=0x7efe08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.035+0000 7efe25e00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe08004800 0x7efe08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.035+0000 7efe25e00640 1 -- 172.21.15.114:0/3789536163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe08003520 con 0x7efe080039e0 2024-03-06T07:35:28.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.036+0000 7efe25e00640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe08003dc0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c190 tx=0x7efe08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57adcc5ac7d103d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.036+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14007c60 con 0x7efe080039e0 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.036+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14030070 con 0x7efe080039e0 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.036+0000 7efe24a00640 1 -- 172.21.15.114:0/3789536163 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14005d10 con 0x7efe080039e0 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.037+0000 7efe28800640 1 -- 172.21.15.114:0/3789536163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 msgr2=0x7efe08003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.037+0000 7efe28800640 1 --2- 172.21.15.114:0/3789536163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe08003dc0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c190 tx=0x7efe08005830 comp rx=0 tx=0).stop 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.037+0000 7efe26800640 1 -- 172.21.15.114:0/3789536163 reap_dead start 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.037+0000 7efe28800640 1 -- 172.21.15.114:0/3789536163 shutdown_connections 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.037+0000 7efe28800640 1 -- 172.21.15.114:0/3789536163 >> 172.21.15.114:0/3789536163 conn(0x7efe200d3e10 msgr2=0x7efe200d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.037+0000 7efe28800640 1 -- 172.21.15.114:0/3789536163 shutdown_connections 2024-03-06T07:35:28.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.037+0000 7efe28800640 1 -- 172.21.15.114:0/3789536163 wait complete. 2024-03-06T07:35:28.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.039+0000 7efe28800640 1 Processor -- start 2024-03-06T07:35:28.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.039+0000 7efe28800640 1 -- start start 2024-03-06T07:35:28.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe28800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe200caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe28800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b7f60 0x7efe2014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe28800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe200b94e0 0x7efe20160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe28800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200b3f60 con 0x7efe080039e0 2024-03-06T07:35:28.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe28800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200b3c80 con 0x7efe200b94e0 2024-03-06T07:35:28.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe28800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe200d1bf0 con 0x7efe200b7f60 2024-03-06T07:35:28.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe200caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe200caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48942/0 (socket says 172.21.15.114:48942) 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 -- 172.21.15.114:0/743203484 learned_addr learned my addr 172.21.15.114:0/743203484 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 -- 172.21.15.114:0/743203484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b7f60 msgr2=0x7efe2014f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b7f60 0x7efe2014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 -- 172.21.15.114:0/743203484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe200b94e0 msgr2=0x7efe20160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe200b94e0 0x7efe20160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.040+0000 7efe25e00640 1 -- 172.21.15.114:0/743203484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe200b8440 con 0x7efe080039e0 2024-03-06T07:35:28.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.041+0000 7efe25e00640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe200caee0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c280 tx=0x7efe200b09e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.041+0000 7efe1a000640 1 -- 172.21.15.114:0/743203484 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14045070 con 0x7efe080039e0 2024-03-06T07:35:28.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.041+0000 7efe1a000640 1 -- 172.21.15.114:0/743203484 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe140049e0 con 0x7efe080039e0 2024-03-06T07:35:28.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.041+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe200b8720 con 0x7efe080039e0 2024-03-06T07:35:28.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.041+0000 7efe1a000640 1 -- 172.21.15.114:0/743203484 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14004dd0 con 0x7efe080039e0 2024-03-06T07:35:28.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.041+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe200b8ce0 con 0x7efe080039e0 2024-03-06T07:35:28.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.043+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2004f330 con 0x7efe080039e0 2024-03-06T07:35:28.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.043+0000 7efe1a000640 1 -- 172.21.15.114:0/743203484 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe14007070 con 0x7efe080039e0 2024-03-06T07:35:28.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.044+0000 7efe1a000640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efe0406b500 0x7efe0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.044+0000 7efe1a000640 1 -- 172.21.15.114:0/743203484 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7efe140ab0f0 con 0x7efe080039e0 2024-03-06T07:35:28.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.045+0000 7efe25400640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efe0406b500 0x7efe0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.045+0000 7efe25400640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efe0406b500 0x7efe0406d9c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efe200cb420 tx=0x7efe10007930 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.049+0000 7efe1a000640 1 -- 172.21.15.114:0/743203484 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe14079ff0 con 0x7efe080039e0 2024-03-06T07:35:28.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.247+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe200d0d00 con 0x7efe080039e0 2024-03-06T07:35:28.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.248+0000 7efe1a000640 1 -- 172.21.15.114:0/743203484 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe1402de20 con 0x7efe080039e0 2024-03-06T07:35:28.250 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efe0406b500 msgr2=0x7efe0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efe0406b500 0x7efe0406d9c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efe200cb420 tx=0x7efe10007930 comp rx=0 tx=0).stop 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 msgr2=0x7efe200caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe200caee0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c280 tx=0x7efe200b09e0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 shutdown_connections 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe200b94e0 0x7efe20160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efe200b7f60 0x7efe2014f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efe0406b500 0x7efe0406d9c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 --2- 172.21.15.114:0/743203484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe080039e0 0x7efe200caee0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 >> 172.21.15.114:0/743203484 conn(0x7efe200d3e10 msgr2=0x7efe2006d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 shutdown_connections 2024-03-06T07:35:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.251+0000 7efe28800640 1 -- 172.21.15.114:0/743203484 wait complete. 2024-03-06T07:35:28.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.470+0000 7fafc0c00640 1 Processor -- start 2024-03-06T07:35:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.470+0000 7fafc0c00640 1 -- start start 2024-03-06T07:35:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.471+0000 7fafc0c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafbc0b3f70 0x7fafbc160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.471+0000 7fafc0c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafbc071440 con 0x7fafbc15fc10 2024-03-06T07:35:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.471+0000 7fafc0c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafbc0b46c0 con 0x7fafbc070ff0 2024-03-06T07:35:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.471+0000 7fafc0c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc0b49a0 con 0x7fafbc0b3f70 2024-03-06T07:35:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.471+0000 7fafb9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafbc0b3f70 0x7fafbc160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.472+0000 7fafb9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fafbc0b3f70 0x7fafbc160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48948/0 (socket says 172.21.15.114:48948) 2024-03-06T07:35:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.472+0000 7fafb9400640 1 -- 172.21.15.114:0/3289124731 learned_addr learned my addr 172.21.15.114:0/3289124731 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.472+0000 7fafb9400640 1 -- 172.21.15.114:0/3289124731 >> v1:172.21.15.177:6789/0 conn(0x7fafbc070ff0 legacy=0x7fafbc0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.472+0000 7fafb9400640 1 -- 172.21.15.114:0/3289124731 >> v1:172.21.15.114:6789/0 conn(0x7fafbc15fc10 legacy=0x7fafbc15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.472+0000 7fafb9400640 1 -- 172.21.15.114:0/3289124731 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc072c90 con 0x7fafbc0b3f70 2024-03-06T07:35:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.473+0000 7fafb9400640 1 --2- 172.21.15.114:0/3289124731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc0b3f70 0x7fafbc160700 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009870 tx=0x7fafa802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7340503b39a868d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.473+0000 7fafb8a00640 1 -- 172.21.15.114:0/3289124731 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8034070 con 0x7fafbc0b3f70 2024-03-06T07:35:28.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.473+0000 7fafb8a00640 1 -- 172.21.15.114:0/3289124731 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa8030850 con 0x7fafbc0b3f70 2024-03-06T07:35:28.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.473+0000 7fafb8a00640 1 -- 172.21.15.114:0/3289124731 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8030c20 con 0x7fafbc0b3f70 2024-03-06T07:35:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.474+0000 7fafc0c00640 1 -- 172.21.15.114:0/3289124731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc0b3f70 msgr2=0x7fafbc160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.474+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3289124731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc0b3f70 0x7fafbc160700 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009870 tx=0x7fafa802b990 comp rx=0 tx=0).stop 2024-03-06T07:35:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.474+0000 7fafc0c00640 1 -- 172.21.15.114:0/3289124731 shutdown_connections 2024-03-06T07:35:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.474+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3289124731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc0b3f70 0x7fafbc160700 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.474+0000 7fafc0c00640 1 -- 172.21.15.114:0/3289124731 >> 172.21.15.114:0/3289124731 conn(0x7fafbc0b1940 msgr2=0x7fafbc0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.474+0000 7fafc0c00640 1 -- 172.21.15.114:0/3289124731 shutdown_connections 2024-03-06T07:35:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.474+0000 7fafc0c00640 1 -- 172.21.15.114:0/3289124731 wait complete. 2024-03-06T07:35:28.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.476+0000 7fafc0c00640 1 Processor -- start 2024-03-06T07:35:28.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.476+0000 7fafc0c00640 1 -- start start 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafc0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc070ff0 0x7fafbc15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafc0c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafbc0b3f70 0x7fafbc09ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafc0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafbc15fc10 0x7fafbc09bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafc0c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc072a30 con 0x7fafbc070ff0 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafc0c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0728b0 con 0x7fafbc15fc10 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafc0c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc15c2e0 con 0x7fafbc0b3f70 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc070ff0 0x7fafbc15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc070ff0 0x7fafbc15b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48954/0 (socket says 172.21.15.114:48954) 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 -- 172.21.15.114:0/3855266690 learned_addr learned my addr 172.21.15.114:0/3855266690 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:28.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 -- 172.21.15.114:0/3855266690 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafbc0b3f70 msgr2=0x7fafbc09ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafbc0b3f70 0x7fafbc09ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 -- 172.21.15.114:0/3855266690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafbc15fc10 msgr2=0x7fafbc09bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafbc15fc10 0x7fafbc09bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.477+0000 7fafb9e00640 1 -- 172.21.15.114:0/3855266690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc1f7ae0 con 0x7fafbc070ff0 2024-03-06T07:35:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.478+0000 7fafb9e00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc070ff0 0x7fafbc15b320 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c3c0 tx=0x7fafac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.478+0000 7fafb2000640 1 -- 172.21.15.114:0/3855266690 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac0052d0 con 0x7fafbc070ff0 2024-03-06T07:35:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.478+0000 7fafb2000640 1 -- 172.21.15.114:0/3855266690 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac005490 con 0x7fafbc070ff0 2024-03-06T07:35:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.478+0000 7fafb2000640 1 -- 172.21.15.114:0/3855266690 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac007260 con 0x7fafbc070ff0 2024-03-06T07:35:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.478+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc1f7dc0 con 0x7fafbc070ff0 2024-03-06T07:35:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.478+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc1f8380 con 0x7fafbc070ff0 2024-03-06T07:35:28.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.479+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc04f330 con 0x7fafbc070ff0 2024-03-06T07:35:28.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.480+0000 7fafb2000640 1 -- 172.21.15.114:0/3855266690 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafac007400 con 0x7fafbc070ff0 2024-03-06T07:35:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.481+0000 7fafb2000640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf9c06b4d0 0x7faf9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.482+0000 7fafb2000640 1 -- 172.21.15.114:0/3855266690 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fafac089790 con 0x7fafbc070ff0 2024-03-06T07:35:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.482+0000 7fafb9400640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf9c06b4d0 0x7faf9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.482+0000 7fafb9400640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf9c06b4d0 0x7faf9c06d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fafa8036000 tx=0x7fafa803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.486+0000 7fafb2000640 1 -- 172.21.15.114:0/3855266690 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafac058520 con 0x7fafbc070ff0 2024-03-06T07:35:28.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.694+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fafbc06c990 con 0x7fafbc070ff0 2024-03-06T07:35:28.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.695+0000 7fafb2000640 1 -- 172.21.15.114:0/3855266690 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7fafac057e20 con 0x7fafbc070ff0 2024-03-06T07:35:28.697 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:35:28.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf9c06b4d0 msgr2=0x7faf9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf9c06b4d0 0x7faf9c06d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fafa8036000 tx=0x7fafa803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc070ff0 msgr2=0x7fafbc15b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc070ff0 0x7fafbc15b320 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c3c0 tx=0x7fafac00c880 comp rx=0 tx=0).stop 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 shutdown_connections 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafbc15fc10 0x7fafbc09bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fafbc0b3f70 0x7fafbc09ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf9c06b4d0 0x7faf9c06d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 --2- 172.21.15.114:0/3855266690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fafbc070ff0 0x7fafbc15b320 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.698+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 >> 172.21.15.114:0/3855266690 conn(0x7fafbc0b1940 msgr2=0x7fafbc0acf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.699+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 shutdown_connections 2024-03-06T07:35:28.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.699+0000 7fafc0c00640 1 -- 172.21.15.114:0/3855266690 wait complete. 2024-03-06T07:35:28.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.911+0000 7f4e04a00640 1 Processor -- start 2024-03-06T07:35:28.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.912+0000 7f4e04a00640 1 -- start start 2024-03-06T07:35:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.913+0000 7f4e04a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4e0015f1c0 0x7f4e001690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.913+0000 7f4e04a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0006cba0 con 0x7f4e000b2c40 2024-03-06T07:35:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.913+0000 7f4e04a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0006ce80 con 0x7f4e0006d960 2024-03-06T07:35:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.913+0000 7f4e04a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0006ea00 con 0x7f4e0015f1c0 2024-03-06T07:35:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.913+0000 7f4dfde00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4e000b2c40 0x7f4e000b3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58912/0 (socket says 172.21.15.114:58912) 2024-03-06T07:35:28.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfde00640 1 -- 172.21.15.114:0/2382812017 learned_addr learned my addr 172.21.15.114:0/2382812017 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770790692 0 0) 0x7f4e0006cba0 con 0x7f4e000b2c40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de4003520 con 0x7f4e000b2c40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfe800640 1 --2- 172.21.15.114:0/2382812017 >> v2:172.21.15.177:3300/0 conn(0x7f4e0015f1c0 0x7f4e001690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62626089 0 0) 0x7f4de4003520 con 0x7f4e000b2c40 2024-03-06T07:35:28.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 >> v2:172.21.15.177:3300/0 conn(0x7f4e0015f1c0 msgr2=0x7f4e001690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 --2- 172.21.15.114:0/2382812017 >> v2:172.21.15.177:3300/0 conn(0x7f4e0015f1c0 0x7f4e001690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 >> v1:172.21.15.177:6789/0 conn(0x7f4e0006d960 legacy=0x7f4e0006e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0006d160 con 0x7f4e000b2c40 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f4df0001670 con 0x7f4e000b2c40 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.914+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 >> v1:172.21.15.114:6789/0 conn(0x7f4e000b2c40 legacy=0x7f4e000b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfca00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4de40039b0 0x7f4de4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfca00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de4004800 0x7f4de4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfca00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4de40051c0 0x7f4de400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0006ea00 con 0x7f4de40039b0 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de400c0f0 con 0x7f4de4004800 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de400c2f0 con 0x7f4de40051c0 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfde00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4de40039b0 0x7f4de4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfde00640 1 -- 172.21.15.114:0/2382812017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4de40051c0 msgr2=0x7f4de400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfde00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4de40051c0 0x7f4de400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfde00640 1 -- 172.21.15.114:0/2382812017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de4004800 msgr2=0x7f4de4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfde00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de4004800 0x7f4de4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfde00640 1 -- 172.21.15.114:0/2382812017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4003520 con 0x7f4de40039b0 2024-03-06T07:35:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.915+0000 7f4dfe800640 1 -- 172.21.15.114:0/2382812017 reap_dead start 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.916+0000 7f4dfde00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4de40039b0 0x7f4de4003db0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4df0007fd0 tx=0x7f4de4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77426d5b772b3186 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0007d50 con 0x7f4de40039b0 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df0030070 con 0x7f4de40039b0 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4dfca00640 1 -- 172.21.15.114:0/2382812017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0005d70 con 0x7f4de40039b0 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4e04a00640 1 -- 172.21.15.114:0/2382812017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4de40039b0 msgr2=0x7f4de4003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4e04a00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4de40039b0 0x7f4de4003db0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4df0007fd0 tx=0x7f4de4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4e04a00640 1 -- 172.21.15.114:0/2382812017 shutdown_connections 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4e04a00640 1 --2- 172.21.15.114:0/2382812017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4de40039b0 0x7f4de4003db0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4e04a00640 1 -- 172.21.15.114:0/2382812017 >> 172.21.15.114:0/2382812017 conn(0x7f4e000b1940 msgr2=0x7f4e000b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4e04a00640 1 -- 172.21.15.114:0/2382812017 shutdown_connections 2024-03-06T07:35:28.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.917+0000 7f4e04a00640 1 -- 172.21.15.114:0/2382812017 wait complete. 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.919+0000 7f4e04a00640 1 Processor -- start 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.919+0000 7f4e04a00640 1 -- start start 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4e04a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de40039b0 0x7f4e000aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4e04a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e00152140 0x7f4e001619a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4e04a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e000add70 0x7f4e000ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4e04a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0006e2f0 con 0x7f4e000add70 2024-03-06T07:35:28.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4e04a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0006d4d0 con 0x7f4de40039b0 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4e04a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0006e470 con 0x7f4e00152140 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e000add70 0x7f4e000ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e000add70 0x7f4e000ae1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48974/0 (socket says 172.21.15.114:48974) 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 -- 172.21.15.114:0/3368684638 learned_addr learned my addr 172.21.15.114:0/3368684638 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 -- 172.21.15.114:0/3368684638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e00152140 msgr2=0x7f4e001619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e00152140 0x7f4e001619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 -- 172.21.15.114:0/3368684638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de40039b0 msgr2=0x7f4e000aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de40039b0 0x7f4e000aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.920+0000 7f4dfe800640 1 -- 172.21.15.114:0/3368684638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e001622d0 con 0x7f4e000add70 2024-03-06T07:35:28.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.921+0000 7f4dfe800640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e000add70 0x7f4e000ae1f0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4e00073b30 tx=0x7f4df800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.921+0000 7f4df6000640 1 -- 172.21.15.114:0/3368684638 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df800edc0 con 0x7f4e000add70 2024-03-06T07:35:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.921+0000 7f4df6000640 1 -- 172.21.15.114:0/3368684638 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df8019070 con 0x7f4e000add70 2024-03-06T07:35:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.921+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e001625b0 con 0x7f4e000add70 2024-03-06T07:35:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.921+0000 7f4df6000640 1 -- 172.21.15.114:0/3368684638 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8014660 con 0x7f4e000add70 2024-03-06T07:35:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.921+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0015fb40 con 0x7f4e000add70 2024-03-06T07:35:28.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.922+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e000690e0 con 0x7f4e000add70 2024-03-06T07:35:28.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.923+0000 7f4df6000640 1 -- 172.21.15.114:0/3368684638 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4df8014800 con 0x7f4e000add70 2024-03-06T07:35:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.924+0000 7f4df6000640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4de006b500 0x7f4de006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.924+0000 7f4df6000640 1 -- 172.21.15.114:0/3368684638 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f4df808a440 con 0x7f4e000add70 2024-03-06T07:35:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.924+0000 7f4dfde00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4de006b500 0x7f4de006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.925+0000 7f4dfde00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4de006b500 0x7f4de006d9c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4df0034000 tx=0x7f4df0007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:28.929+0000 7f4df6000640 1 -- 172.21.15.114:0/3368684638 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e000690e0 con 0x7f4e000add70 2024-03-06T07:35:29.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.116+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4e0004f330 con 0x7f4e000add70 2024-03-06T07:35:29.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.117+0000 7f4df6000640 1 -- 172.21.15.114:0/3368684638 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4df8023020 con 0x7f4e000add70 2024-03-06T07:35:29.119 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:35:29.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4de006b500 msgr2=0x7f4de006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4de006b500 0x7f4de006d9c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4df0034000 tx=0x7f4df0007000 comp rx=0 tx=0).stop 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e000add70 msgr2=0x7f4e000ae1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e000add70 0x7f4e000ae1f0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4e00073b30 tx=0x7f4df800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 shutdown_connections 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4de006b500 0x7f4de006d9c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e000add70 0x7f4e000ae1f0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e00152140 0x7f4e001619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 --2- 172.21.15.114:0/3368684638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de40039b0 0x7f4e000aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.119+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 >> 172.21.15.114:0/3368684638 conn(0x7f4e000b1940 msgr2=0x7f4e000715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.120+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 shutdown_connections 2024-03-06T07:35:29.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.120+0000 7f4e04a00640 1 -- 172.21.15.114:0/3368684638 wait complete. 2024-03-06T07:35:29.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:29.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.351+0000 7f0081c00640 1 Processor -- start 2024-03-06T07:35:29.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.351+0000 7f0081c00640 1 -- start start 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f0081c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f007c0af1c0 0x7f007c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f0081c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f007c070d20 0x7f007c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f0081c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c06d1a0 0x7f007c06e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f0081c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f007c071d30 con 0x7f007c070d20 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f0081c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f007c0711d0 con 0x7f007c0af1c0 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f0081c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c071490 con 0x7f007c06d1a0 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f007a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f007c070d20 0x7f007c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f007a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f007c070d20 0x7f007c06cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48990/0 (socket says 172.21.15.114:48990) 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f007a000640 1 -- 172.21.15.114:0/3087593972 learned_addr learned my addr 172.21.15.114:0/3087593972 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f007b400640 1 --2- 172.21.15.114:0/3087593972 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c06d1a0 0x7f007c06e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.353+0000 7f007aa00640 1 --2- 172.21.15.114:0/3087593972 >> v2:172.21.15.177:3300/0 conn(0x7f007c0af1c0 0x7f007c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.354+0000 7f007a000640 1 -- 172.21.15.114:0/3087593972 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c06d1a0 msgr2=0x7f007c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.354+0000 7f007a000640 1 --2- 172.21.15.114:0/3087593972 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c06d1a0 0x7f007c06e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.354+0000 7f007a000640 1 -- 172.21.15.114:0/3087593972 >> v2:172.21.15.177:3300/0 conn(0x7f007c0af1c0 msgr2=0x7f007c15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.354+0000 7f007a000640 1 --2- 172.21.15.114:0/3087593972 >> v2:172.21.15.177:3300/0 conn(0x7f007c0af1c0 0x7f007c15eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.354+0000 7f007a000640 1 -- 172.21.15.114:0/3087593972 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c071770 con 0x7f007c070d20 2024-03-06T07:35:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.355+0000 7f007a000640 1 --2- 172.21.15.114:0/3087593972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c06cc60 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0068009870 tx=0x7f006802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b158efbdc6871e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.355+0000 7f0079600640 1 -- 172.21.15.114:0/3087593972 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068035020 con 0x7f007c070d20 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.355+0000 7f0079600640 1 -- 172.21.15.114:0/3087593972 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0068031640 con 0x7f007c070d20 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0079600640 1 -- 172.21.15.114:0/3087593972 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006802d040 con 0x7f007c070d20 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 -- 172.21.15.114:0/3087593972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 msgr2=0x7f007c06cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 --2- 172.21.15.114:0/3087593972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c06cc60 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0068009870 tx=0x7f006802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 -- 172.21.15.114:0/3087593972 shutdown_connections 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 --2- 172.21.15.114:0/3087593972 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c06d1a0 0x7f007c06e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 --2- 172.21.15.114:0/3087593972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c06cc60 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 --2- 172.21.15.114:0/3087593972 >> v2:172.21.15.177:3300/0 conn(0x7f007c0af1c0 0x7f007c15eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 -- 172.21.15.114:0/3087593972 >> 172.21.15.114:0/3087593972 conn(0x7f007c0b1730 msgr2=0x7f007c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 -- 172.21.15.114:0/3087593972 shutdown_connections 2024-03-06T07:35:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.356+0000 7f0081c00640 1 -- 172.21.15.114:0/3087593972 wait complete. 2024-03-06T07:35:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.358+0000 7f0081c00640 1 Processor -- start 2024-03-06T07:35:29.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.358+0000 7f0081c00640 1 -- start start 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f0081c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f007c06d1a0 0x7f007c14b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f0081c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c15e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f0081c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c0af1c0 0x7f007c0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f0081c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c06ea10 con 0x7f007c070d20 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f0081c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0b30b0 con 0x7f007c06d1a0 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f0081c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c06eb90 con 0x7f007c0af1c0 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f007a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c15e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f007a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c15e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48996/0 (socket says 172.21.15.114:48996) 2024-03-06T07:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.359+0000 7f007a000640 1 -- 172.21.15.114:0/3717863603 learned_addr learned my addr 172.21.15.114:0/3717863603 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007b400640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c0af1c0 0x7f007c0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007aa00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f007c06d1a0 0x7f007c14b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007a000640 1 -- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c0af1c0 msgr2=0x7f007c0cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007a000640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c0af1c0 0x7f007c0cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007a000640 1 -- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f007c06d1a0 msgr2=0x7f007c14b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007a000640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f007c06d1a0 0x7f007c14b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007a000640 1 -- 172.21.15.114:0/3717863603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c15f540 con 0x7f007c070d20 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007aa00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f007c06d1a0 0x7f007c14b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f007a000640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c15e4c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f006802b930 tx=0x7f00680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f0072a00640 1 -- 172.21.15.114:0/3717863603 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068035040 con 0x7f007c070d20 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f0072a00640 1 -- 172.21.15.114:0/3717863603 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00680399f0 con 0x7f007c070d20 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f0072a00640 1 -- 172.21.15.114:0/3717863603 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068039de0 con 0x7f007c070d20 2024-03-06T07:35:29.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.360+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c15ef80 con 0x7f007c070d20 2024-03-06T07:35:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.361+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c15fa40 con 0x7f007c070d20 2024-03-06T07:35:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.362+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c06c780 con 0x7f007c070d20 2024-03-06T07:35:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.363+0000 7f0072a00640 1 -- 172.21.15.114:0/3717863603 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0068038020 con 0x7f007c070d20 2024-03-06T07:35:29.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.364+0000 7f0072a00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f005c06b4d0 0x7f005c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.364+0000 7f0072a00640 1 -- 172.21.15.114:0/3717863603 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f00680ac4c0 con 0x7f007c070d20 2024-03-06T07:35:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.364+0000 7f007aa00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f005c06b4d0 0x7f005c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.365+0000 7f007aa00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f005c06b4d0 0x7f005c06d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f006c0099a0 tx=0x7f006c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.368+0000 7f0072a00640 1 -- 172.21.15.114:0/3717863603 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f007c06c780 con 0x7f007c070d20 2024-03-06T07:35:29.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.620+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f007c0b1f90 con 0x7f007c070d20 2024-03-06T07:35:29.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.621+0000 7f0072a00640 1 -- 172.21.15.114:0/3717863603 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f0068054080 con 0x7f007c070d20 2024-03-06T07:35:29.623 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:29.623 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:29.623 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f005c06b4d0 msgr2=0x7f005c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f005c06b4d0 0x7f005c06d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f006c0099a0 tx=0x7f006c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 msgr2=0x7f007c15e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c15e4c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f006802b930 tx=0x7f00680046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 shutdown_connections 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f007c0af1c0 0x7f007c0cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f005c06b4d0 0x7f005c06d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f007c070d20 0x7f007c15e4c0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 --2- 172.21.15.114:0/3717863603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f007c06d1a0 0x7f007c14b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 >> 172.21.15.114:0/3717863603 conn(0x7f007c0b1730 msgr2=0x7f007c0b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 shutdown_connections 2024-03-06T07:35:29.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.624+0000 7f0081c00640 1 -- 172.21.15.114:0/3717863603 wait complete. 2024-03-06T07:35:29.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.836+0000 7f7bdd400640 1 Processor -- start 2024-03-06T07:35:29.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.836+0000 7f7bdd400640 1 -- start start 2024-03-06T07:35:29.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.838+0000 7f7bdd400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7bd80cc2a0 0x7f7bd80cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.838+0000 7f7bdd400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd80b2b30 con 0x7f7bd80cf370 2024-03-06T07:35:29.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.838+0000 7f7bdd400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd80b2e10 con 0x7f7bd815a940 2024-03-06T07:35:29.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.838+0000 7f7bdd400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd80b28e0 con 0x7f7bd80cc2a0 2024-03-06T07:35:29.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.838+0000 7f7bd7400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7bd80cf370 0x7f7bd806d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58922/0 (socket says 172.21.15.114:58922) 2024-03-06T07:35:29.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.838+0000 7f7bd7400640 1 -- 172.21.15.114:0/4190420516 learned_addr learned my addr 172.21.15.114:0/4190420516 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.838+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547470045 0 0) 0x7f7bd80b2b30 con 0x7f7bd80cf370 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bbc003520 con 0x7f7bd80cf370 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bdca00640 1 --2- 172.21.15.114:0/4190420516 >> v2:172.21.15.177:3300/0 conn(0x7f7bd80cc2a0 0x7f7bd80cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023917790 0 0) 0x7f7bbc003520 con 0x7f7bd80cf370 2024-03-06T07:35:29.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 >> v2:172.21.15.177:3300/0 conn(0x7f7bd80cc2a0 msgr2=0x7f7bd80cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 --2- 172.21.15.114:0/4190420516 >> v2:172.21.15.177:3300/0 conn(0x7f7bd80cc2a0 0x7f7bd80cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 >> v1:172.21.15.177:6789/0 conn(0x7f7bd815a940 legacy=0x7f7bd80b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd806ca50 con 0x7f7bd80cf370 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7bc0001670 con 0x7f7bd80cf370 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bdca00640 1 --2- 172.21.15.114:0/4190420516 >> v2:172.21.15.177:3300/0 conn(0x7f7bd80cc2a0 0x7f7bd80cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 >> v1:172.21.15.114:6789/0 conn(0x7f7bd80cf370 legacy=0x7f7bd806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0039b0 0x7f7bbc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc004800 0x7f7bbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bbc0052e0 0x7f7bbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd80b28e0 con 0x7f7bbc0039b0 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc00c1e0 con 0x7f7bbc004800 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc00c3e0 con 0x7f7bbc0052e0 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.839+0000 7f7bd7e00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0039b0 0x7f7bbc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bd7400640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc004800 0x7f7bbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bdca00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bbc0052e0 0x7f7bbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bd7e00640 1 -- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bbc0052e0 msgr2=0x7f7bbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bd7e00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bbc0052e0 0x7f7bbc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bd7e00640 1 -- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc004800 msgr2=0x7f7bbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bd7e00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc004800 0x7f7bbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bd7e00640 1 -- 172.21.15.114:0/4190420516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc003520 con 0x7f7bbc0039b0 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bd7400640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc004800 0x7f7bbc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.840+0000 7f7bdca00640 1 -- 172.21.15.114:0/4190420516 reap_dead start 2024-03-06T07:35:29.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.841+0000 7f7bd7e00640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0039b0 0x7f7bbc003db0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8001d20 tx=0x7f7bbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d6fa55f43eb3d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.841+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8001920 con 0x7f7bbc0039b0 2024-03-06T07:35:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.841+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc8035070 con 0x7f7bbc0039b0 2024-03-06T07:35:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.841+0000 7f7bd6a00640 1 -- 172.21.15.114:0/4190420516 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8003df0 con 0x7f7bbc0039b0 2024-03-06T07:35:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.842+0000 7f7bdd400640 1 -- 172.21.15.114:0/4190420516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0039b0 msgr2=0x7f7bbc003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.842+0000 7f7bdd400640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0039b0 0x7f7bbc003db0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8001d20 tx=0x7f7bbc005830 comp rx=0 tx=0).stop 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.842+0000 7f7bdd400640 1 -- 172.21.15.114:0/4190420516 shutdown_connections 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.842+0000 7f7bdd400640 1 --2- 172.21.15.114:0/4190420516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc0039b0 0x7f7bbc003db0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.842+0000 7f7bdd400640 1 -- 172.21.15.114:0/4190420516 >> 172.21.15.114:0/4190420516 conn(0x7f7bd806afc0 msgr2=0x7f7bd806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.842+0000 7f7bdd400640 1 -- 172.21.15.114:0/4190420516 shutdown_connections 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.842+0000 7f7bdd400640 1 -- 172.21.15.114:0/4190420516 wait complete. 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.843+0000 7f7bdd400640 1 Processor -- start 2024-03-06T07:35:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.844+0000 7f7bdd400640 1 -- start start 2024-03-06T07:35:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.844+0000 7f7bdd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc0039b0 0x7f7bd80afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.844+0000 7f7bdd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd814a360 0x7f7bd8158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bdd400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bd80ad860 0x7f7bd80adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bdd400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd815ad20 con 0x7f7bd814a360 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bdd400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd80b30f0 con 0x7f7bbc0039b0 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bdd400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd815aea0 con 0x7f7bd80ad860 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd814a360 0x7f7bd8158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd814a360 0x7f7bd8158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49028/0 (socket says 172.21.15.114:49028) 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 -- 172.21.15.114:0/1746331230 learned_addr learned my addr 172.21.15.114:0/1746331230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7e00640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc0039b0 0x7f7bd80afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 -- 172.21.15.114:0/1746331230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bd80ad860 msgr2=0x7f7bd80adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bd80ad860 0x7f7bd80adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 -- 172.21.15.114:0/1746331230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc0039b0 msgr2=0x7f7bd80afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc0039b0 0x7f7bd80afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 -- 172.21.15.114:0/1746331230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd81593d0 con 0x7f7bd814a360 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.845+0000 7f7bd7400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd814a360 0x7f7bd8158c60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000d800 tx=0x7f7bc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.846+0000 7f7bcfe00640 1 -- 172.21.15.114:0/1746331230 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0019070 con 0x7f7bd814a360 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.846+0000 7f7bcfe00640 1 -- 172.21.15.114:0/1746331230 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc00074e0 con 0x7f7bd814a360 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.846+0000 7f7bcfe00640 1 -- 172.21.15.114:0/1746331230 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0004a20 con 0x7f7bd814a360 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.846+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd81596b0 con 0x7f7bd814a360 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.846+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd8159c70 con 0x7f7bd814a360 2024-03-06T07:35:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.847+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd804f330 con 0x7f7bd814a360 2024-03-06T07:35:29.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.848+0000 7f7bcfe00640 1 -- 172.21.15.114:0/1746331230 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bc001c070 con 0x7f7bd814a360 2024-03-06T07:35:29.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.849+0000 7f7bcfe00640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bb406b470 0x7f7bb406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.849+0000 7f7bcfe00640 1 -- 172.21.15.114:0/1746331230 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f7bc0089f20 con 0x7f7bd814a360 2024-03-06T07:35:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.849+0000 7f7bd7e00640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bb406b470 0x7f7bb406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.850+0000 7f7bd7e00640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bb406b470 0x7f7bb406d930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8034000 tx=0x7f7bc8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:29.853+0000 7f7bcfe00640 1 -- 172.21.15.114:0/1746331230 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bc0058b80 con 0x7f7bd814a360 2024-03-06T07:35:30.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.048+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bd806e180 con 0x7f7bd814a360 2024-03-06T07:35:30.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.050+0000 7f7bcfe00640 1 -- 172.21.15.114:0/1746331230 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+17089 (secure 0 0 0) 0x7f7bc0058480 con 0x7f7bd814a360 2024-03-06T07:35:30.052 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:30.052 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":114,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:38.169073+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.177:6833/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.114:6837/860883596":"2024-03-07T07:34:21.018930+0000","172.21.15.114:6835/860883596":"2024-03-07T07:34:21.018930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.052+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bb406b470 msgr2=0x7f7bb406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.052+0000 7f7bdd400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bb406b470 0x7f7bb406d930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8034000 tx=0x7f7bc8009030 comp rx=0 tx=0).stop 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.052+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd814a360 msgr2=0x7f7bd8158c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.052+0000 7f7bdd400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd814a360 0x7f7bd8158c60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000d800 tx=0x7f7bc000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 shutdown_connections 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7bd80ad860 0x7f7bd80adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7bb406b470 0x7f7bb406d930 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd814a360 0x7f7bd8158c60 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 --2- 172.21.15.114:0/1746331230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bbc0039b0 0x7f7bd80afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 >> 172.21.15.114:0/1746331230 conn(0x7f7bd806afc0 msgr2=0x7f7bd815e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 shutdown_connections 2024-03-06T07:35:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.053+0000 7f7bdd400640 1 -- 172.21.15.114:0/1746331230 wait complete. 2024-03-06T07:35:30.072 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:35:30.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.319+0000 7fd099c00640 1 Processor -- start 2024-03-06T07:35:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.319+0000 7fd099c00640 1 -- start start 2024-03-06T07:35:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.320+0000 7fd099c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd094144c00 0x7fd09406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.320+0000 7fd099c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd0940ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.320+0000 7fd099c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09406ee70 con 0x7fd09415f9d0 2024-03-06T07:35:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.320+0000 7fd099c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09406cef0 con 0x7fd094144c00 2024-03-06T07:35:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.320+0000 7fd099c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09406d1d0 con 0x7fd09415fe60 2024-03-06T07:35:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.321+0000 7fd093400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd0940ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.321+0000 7fd092a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd094144c00 0x7fd09406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.321+0000 7fd093400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd0940ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58486/0 (socket says 172.21.15.114:58486) 2024-03-06T07:35:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.321+0000 7fd093400640 1 -- 172.21.15.114:0/3415917816 learned_addr learned my addr 172.21.15.114:0/3415917816 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.322+0000 7fd092a00640 1 -- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 msgr2=0x7fd0940ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.322+0000 7fd092a00640 1 --2- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd0940ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.322+0000 7fd092a00640 1 -- 172.21.15.114:0/3415917816 >> v1:172.21.15.177:6789/0 conn(0x7fd09415f9d0 legacy=0x7fd0940ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.322+0000 7fd092a00640 1 -- 172.21.15.114:0/3415917816 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09406d4b0 con 0x7fd094144c00 2024-03-06T07:35:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.322+0000 7fd093400640 1 --2- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd0940ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.323+0000 7fd092a00640 1 --2- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd094144c00 0x7fd09406e930 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd0840097a0 tx=0x7fd08402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f5f0c57d4cb4d54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.324+0000 7fd091600640 1 -- 172.21.15.114:0/3415917816 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084035020 con 0x7fd094144c00 2024-03-06T07:35:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.324+0000 7fd091600640 1 -- 172.21.15.114:0/3415917816 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd084031810 con 0x7fd094144c00 2024-03-06T07:35:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.324+0000 7fd091600640 1 -- 172.21.15.114:0/3415917816 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084031be0 con 0x7fd094144c00 2024-03-06T07:35:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 -- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd094144c00 msgr2=0x7fd09406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 --2- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd094144c00 0x7fd09406e930 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd0840097a0 tx=0x7fd08402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 -- 172.21.15.114:0/3415917816 shutdown_connections 2024-03-06T07:35:30.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 --2- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd0940ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 --2- 172.21.15.114:0/3415917816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd094144c00 0x7fd09406e930 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 -- 172.21.15.114:0/3415917816 >> 172.21.15.114:0/3415917816 conn(0x7fd0940b1730 msgr2=0x7fd0940b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 -- 172.21.15.114:0/3415917816 shutdown_connections 2024-03-06T07:35:30.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.325+0000 7fd099c00640 1 -- 172.21.15.114:0/3415917816 wait complete. 2024-03-06T07:35:30.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.326+0000 7fd099c00640 1 Processor -- start 2024-03-06T07:35:30.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.327+0000 7fd099c00640 1 -- start start 2024-03-06T07:35:30.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.328+0000 7fd099c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd094144c00 0x7fd0940b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.328+0000 7fd099c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09415f9d0 0x7fd09415b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.328+0000 7fd099c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd09415b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.328+0000 7fd099c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd094160730 con 0x7fd094144c00 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.328+0000 7fd099c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd094160290 con 0x7fd09415f9d0 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.328+0000 7fd099c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09406cb50 con 0x7fd09415fe60 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.328+0000 7fd092a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd094144c00 0x7fd0940b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd094144c00 0x7fd0940b3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49042/0 (socket says 172.21.15.114:49042) 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 -- 172.21.15.114:0/1371760344 learned_addr learned my addr 172.21.15.114:0/1371760344 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd093400640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd09415b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092000640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09415f9d0 0x7fd09415b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 -- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 msgr2=0x7fd09415b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd09415b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 -- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09415f9d0 msgr2=0x7fd09415b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09415f9d0 0x7fd09415b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 -- 172.21.15.114:0/1371760344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0941e8dd0 con 0x7fd094144c00 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.329+0000 7fd092a00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd094144c00 0x7fd0940b3fa0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd084034040 tx=0x7fd0840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.330+0000 7fd08aa00640 1 -- 172.21.15.114:0/1371760344 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084040070 con 0x7fd094144c00 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.330+0000 7fd08aa00640 1 -- 172.21.15.114:0/1371760344 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd084039cf0 con 0x7fd094144c00 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.330+0000 7fd08aa00640 1 -- 172.21.15.114:0/1371760344 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084035040 con 0x7fd094144c00 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.330+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0941e90b0 con 0x7fd094144c00 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.330+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0941e9670 con 0x7fd094144c00 2024-03-06T07:35:30.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.331+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09404f330 con 0x7fd094144c00 2024-03-06T07:35:30.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.332+0000 7fd08aa00640 1 -- 172.21.15.114:0/1371760344 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd084038020 con 0x7fd094144c00 2024-03-06T07:35:30.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.333+0000 7fd08aa00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd07406b4d0 0x7fd07406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.333+0000 7fd08aa00640 1 -- 172.21.15.114:0/1371760344 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fd0840abf80 con 0x7fd094144c00 2024-03-06T07:35:30.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.333+0000 7fd092000640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd07406b4d0 0x7fd07406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.334+0000 7fd092000640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd07406b4d0 0x7fd07406d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd0940cc9f0 tx=0x7fd080009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.337+0000 7fd08aa00640 1 -- 172.21.15.114:0/1371760344 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd08407ac70 con 0x7fd094144c00 2024-03-06T07:35:30.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.585+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd094072890 con 0x7fd094144c00 2024-03-06T07:35:30.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.586+0000 7fd08aa00640 1 -- 172.21.15.114:0/1371760344 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fd08407a570 con 0x7fd094144c00 2024-03-06T07:35:30.588 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:30.588 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:30.588 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:30.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.588+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd07406b4d0 msgr2=0x7fd07406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.588+0000 7fd099c00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd07406b4d0 0x7fd07406d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd0940cc9f0 tx=0x7fd080009040 comp rx=0 tx=0).stop 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.588+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd094144c00 msgr2=0x7fd0940b3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.588+0000 7fd099c00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd094144c00 0x7fd0940b3fa0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fd084034040 tx=0x7fd0840046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.588+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 shutdown_connections 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.589+0000 7fd099c00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd09415fe60 0x7fd09415b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.589+0000 7fd099c00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09415f9d0 0x7fd09415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.589+0000 7fd099c00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd07406b4d0 0x7fd07406d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.589+0000 7fd099c00640 1 --2- 172.21.15.114:0/1371760344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd094144c00 0x7fd0940b3fa0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.589+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 >> 172.21.15.114:0/1371760344 conn(0x7fd0940b1730 msgr2=0x7fd094070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.589+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 shutdown_connections 2024-03-06T07:35:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.589+0000 7fd099c00640 1 -- 172.21.15.114:0/1371760344 wait complete. 2024-03-06T07:35:30.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:30.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.806+0000 7fcc5b000640 1 Processor -- start 2024-03-06T07:35:30.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.806+0000 7fcc5b000640 1 -- start start 2024-03-06T07:35:30.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc5b000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcc5415acb0 0x7fcc540b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc5b000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc540b4680 0x7fcc5406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc5b000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5415b090 con 0x7fcc54072340 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc5b000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc540b2a30 con 0x7fcc540b4680 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc5b000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc540b4360 con 0x7fcc5415acb0 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc4b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc540b4680 0x7fcc5406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc4b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcc540b4680 0x7fcc5406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49056/0 (socket says 172.21.15.114:49056) 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.808+0000 7fcc4b400640 1 -- 172.21.15.114:0/1343081423 learned_addr learned my addr 172.21.15.114:0/1343081423 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.809+0000 7fcc53e00640 1 --2- 172.21.15.114:0/1343081423 >> v2:172.21.15.177:3300/0 conn(0x7fcc5415acb0 0x7fcc540b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.809+0000 7fcc4b400640 1 -- 172.21.15.114:0/1343081423 >> v2:172.21.15.177:3300/0 conn(0x7fcc5415acb0 msgr2=0x7fcc540b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.809+0000 7fcc4b400640 1 --2- 172.21.15.114:0/1343081423 >> v2:172.21.15.177:3300/0 conn(0x7fcc5415acb0 0x7fcc540b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.809+0000 7fcc4b400640 1 -- 172.21.15.114:0/1343081423 >> v1:172.21.15.177:6789/0 conn(0x7fcc54072340 legacy=0x7fcc540ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.809+0000 7fcc4b400640 1 -- 172.21.15.114:0/1343081423 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc540b2cf0 con 0x7fcc540b4680 2024-03-06T07:35:30.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.811+0000 7fcc4b400640 1 --2- 172.21.15.114:0/1343081423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc540b4680 0x7fcc5406e4f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fcc540b27f0 tx=0x7fcc3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9fb44f736f0168a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.811+0000 7fcc53400640 1 -- 172.21.15.114:0/1343081423 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c035020 con 0x7fcc540b4680 2024-03-06T07:35:30.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.811+0000 7fcc53400640 1 -- 172.21.15.114:0/1343081423 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c031870 con 0x7fcc540b4680 2024-03-06T07:35:30.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.811+0000 7fcc53400640 1 -- 172.21.15.114:0/1343081423 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c031c40 con 0x7fcc540b4680 2024-03-06T07:35:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.811+0000 7fcc5b000640 1 -- 172.21.15.114:0/1343081423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc540b4680 msgr2=0x7fcc5406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.811+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1343081423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc540b4680 0x7fcc5406e4f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fcc540b27f0 tx=0x7fcc3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:35:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.812+0000 7fcc5b000640 1 -- 172.21.15.114:0/1343081423 shutdown_connections 2024-03-06T07:35:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.812+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1343081423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc540b4680 0x7fcc5406e4f0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.812+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1343081423 >> v2:172.21.15.177:3300/0 conn(0x7fcc5415acb0 0x7fcc540b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.812+0000 7fcc5b000640 1 -- 172.21.15.114:0/1343081423 >> 172.21.15.114:0/1343081423 conn(0x7fcc540b1730 msgr2=0x7fcc540b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.812+0000 7fcc5b000640 1 -- 172.21.15.114:0/1343081423 shutdown_connections 2024-03-06T07:35:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.812+0000 7fcc5b000640 1 -- 172.21.15.114:0/1343081423 wait complete. 2024-03-06T07:35:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.813+0000 7fcc5b000640 1 Processor -- start 2024-03-06T07:35:30.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.813+0000 7fcc5b000640 1 -- start start 2024-03-06T07:35:30.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.814+0000 7fcc5b000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc54072340 0x7fcc54147be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.814+0000 7fcc5b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc540b4680 0x7fcc540cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.814+0000 7fcc5b000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc5415acb0 0x7fcc540cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.814+0000 7fcc5b000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540b3d10 con 0x7fcc5415acb0 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.814+0000 7fcc5b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540b32b0 con 0x7fcc540b4680 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.814+0000 7fcc5b000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540b4010 con 0x7fcc54072340 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc5415acb0 0x7fcc540cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc5415acb0 0x7fcc540cde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49064/0 (socket says 172.21.15.114:49064) 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 -- 172.21.15.114:0/1822930295 learned_addr learned my addr 172.21.15.114:0/1822930295 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc4b400640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc540b4680 0x7fcc540cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc53e00640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc54072340 0x7fcc54147be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 -- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc54072340 msgr2=0x7fcc54147be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc54072340 0x7fcc54147be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 -- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc540b4680 msgr2=0x7fcc540cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc540b4680 0x7fcc540cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.815+0000 7fcc59000640 1 -- 172.21.15.114:0/1822930295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5414af10 con 0x7fcc5415acb0 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.816+0000 7fcc59000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc5415acb0 0x7fcc540cde90 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b410 tx=0x7fcc4c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.816+0000 7fcc50c00640 1 -- 172.21.15.114:0/1822930295 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c015030 con 0x7fcc5415acb0 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.816+0000 7fcc50c00640 1 -- 172.21.15.114:0/1822930295 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c004020 con 0x7fcc5415acb0 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.816+0000 7fcc50c00640 1 -- 172.21.15.114:0/1822930295 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c004450 con 0x7fcc5415acb0 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.816+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc540ce480 con 0x7fcc5415acb0 2024-03-06T07:35:30.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.816+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5414b1b0 con 0x7fcc5415acb0 2024-03-06T07:35:30.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.817+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5404f330 con 0x7fcc5415acb0 2024-03-06T07:35:30.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.819+0000 7fcc50c00640 1 -- 172.21.15.114:0/1822930295 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc4c018070 con 0x7fcc5415acb0 2024-03-06T07:35:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.820+0000 7fcc50c00640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc3006b500 0x7fcc3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.820+0000 7fcc50c00640 1 -- 172.21.15.114:0/1822930295 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fcc4c089d50 con 0x7fcc5415acb0 2024-03-06T07:35:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.820+0000 7fcc53e00640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc3006b500 0x7fcc3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.821+0000 7fcc53e00640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc3006b500 0x7fcc3006d9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009a60 tx=0x7fcc44003990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:30.825+0000 7fcc50c00640 1 -- 172.21.15.114:0/1822930295 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc4c058ad0 con 0x7fcc5415acb0 2024-03-06T07:35:31.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.048+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc5406ce60 con 0x7fcc5415acb0 2024-03-06T07:35:31.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.048+0000 7fcc50c00640 1 -- 172.21.15.114:0/1822930295 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4987 (secure 0 0 0) 0x7fcc4c0583d0 con 0x7fcc5415acb0 2024-03-06T07:35:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:31.050 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:34:06.421022+0000","modified":"2024-03-06T07:34:25.093846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7911":{"gid":7911,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.114:6835/2704122879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2704122879},{"type":"v1","addr":"172.21.15.114:6835","nonce":2704122879}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7911,"qdb_cluster":[7911]},"id":8}]} 2024-03-06T07:35:31.051 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 87 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc3006b500 msgr2=0x7fcc3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc3006b500 0x7fcc3006d9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009a60 tx=0x7fcc44003990 comp rx=0 tx=0).stop 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc5415acb0 msgr2=0x7fcc540cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc5415acb0 0x7fcc540cde90 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b410 tx=0x7fcc4c00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 shutdown_connections 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcc3006b500 0x7fcc3006d9c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc5415acb0 0x7fcc540cde90 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc540b4680 0x7fcc540cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 --2- 172.21.15.114:0/1822930295 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcc54072340 0x7fcc54147be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 >> 172.21.15.114:0/1822930295 conn(0x7fcc540b1730 msgr2=0x7fcc54155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 shutdown_connections 2024-03-06T07:35:31.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.051+0000 7fcc5b000640 1 -- 172.21.15.114:0/1822930295 wait complete. 2024-03-06T07:35:31.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:31.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.268+0000 7f0ebf200640 1 Processor -- start 2024-03-06T07:35:31.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.268+0000 7f0ebf200640 1 -- start start 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebf200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0eb80d0c00 0x7f0eb80d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebf200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0eb80b2e50 0x7f0eb80b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebf200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb80b5240 con 0x7f0eb815e200 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebf200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb80d2ec0 con 0x7f0eb80d0c00 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebf200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb80cf4d0 con 0x7f0eb80b2e50 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebde00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0eb815e200 0x7f0eb80d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58930/0 (socket says 172.21.15.114:58930) 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebde00640 1 -- 172.21.15.114:0/3972919300 learned_addr learned my addr 172.21.15.114:0/3972919300 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:31.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebd400640 1 --2- 172.21.15.114:0/3972919300 >> v2:172.21.15.114:3300/0 conn(0x7f0eb80d0c00 0x7f0eb80d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.270+0000 7f0ebe800640 1 --2- 172.21.15.114:0/3972919300 >> v2:172.21.15.177:3300/0 conn(0x7f0eb80b2e50 0x7f0eb80b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.271+0000 7f0ebd400640 1 -- 172.21.15.114:0/3972919300 >> v2:172.21.15.177:3300/0 conn(0x7f0eb80b2e50 msgr2=0x7f0eb80b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.271+0000 7f0ebd400640 1 --2- 172.21.15.114:0/3972919300 >> v2:172.21.15.177:3300/0 conn(0x7f0eb80b2e50 0x7f0eb80b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.271+0000 7f0ebd400640 1 -- 172.21.15.114:0/3972919300 >> v1:172.21.15.114:6789/0 conn(0x7f0eb815e200 legacy=0x7f0eb80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.271+0000 7f0ebca00640 1 -- 172.21.15.114:0/3972919300 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286584863 0 0) 0x7f0eb80b5240 con 0x7f0eb815e200 2024-03-06T07:35:31.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.271+0000 7f0ebd400640 1 -- 172.21.15.114:0/3972919300 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb80b5500 con 0x7f0eb80d0c00 2024-03-06T07:35:31.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.272+0000 7f0ebd400640 1 --2- 172.21.15.114:0/3972919300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80d0c00 0x7f0eb80d4850 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0eb80cad90 tx=0x7f0ea802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8102f141cdd3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.272+0000 7f0ebca00640 1 -- 172.21.15.114:0/3972919300 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8034070 con 0x7f0eb80d0c00 2024-03-06T07:35:31.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.272+0000 7f0ebca00640 1 -- 172.21.15.114:0/3972919300 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea8030610 con 0x7f0eb80d0c00 2024-03-06T07:35:31.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.272+0000 7f0ebca00640 1 -- 172.21.15.114:0/3972919300 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80309e0 con 0x7f0eb80d0c00 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.272+0000 7f0ebf200640 1 -- 172.21.15.114:0/3972919300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80d0c00 msgr2=0x7f0eb80d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.272+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3972919300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80d0c00 0x7f0eb80d4850 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0eb80cad90 tx=0x7f0ea802b960 comp rx=0 tx=0).stop 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.273+0000 7f0ebf200640 1 -- 172.21.15.114:0/3972919300 shutdown_connections 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.273+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3972919300 >> v2:172.21.15.177:3300/0 conn(0x7f0eb80b2e50 0x7f0eb80b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.273+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3972919300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80d0c00 0x7f0eb80d4850 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.273+0000 7f0ebf200640 1 -- 172.21.15.114:0/3972919300 >> 172.21.15.114:0/3972919300 conn(0x7f0eb80d3d30 msgr2=0x7f0eb80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.273+0000 7f0ebf200640 1 -- 172.21.15.114:0/3972919300 shutdown_connections 2024-03-06T07:35:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.273+0000 7f0ebf200640 1 -- 172.21.15.114:0/3972919300 wait complete. 2024-03-06T07:35:31.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.275+0000 7f0ebf200640 1 Processor -- start 2024-03-06T07:35:31.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.275+0000 7f0ebf200640 1 -- start start 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80b2e50 0x7f0eb80b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebf200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb80d0c00 0x7f0eb80b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebf200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb815e200 0x7f0eb80b7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebf200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80cfd60 con 0x7f0eb80b2e50 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebf200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80cfa80 con 0x7f0eb815e200 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebf200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb815e5e0 con 0x7f0eb80d0c00 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80b2e50 0x7f0eb80b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80b2e50 0x7f0eb80b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49074/0 (socket says 172.21.15.114:49074) 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebde00640 1 -- 172.21.15.114:0/3048798282 learned_addr learned my addr 172.21.15.114:0/3048798282 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebe800640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb815e200 0x7f0eb80b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.276+0000 7f0ebd400640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb80d0c00 0x7f0eb80b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebde00640 1 -- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb80d0c00 msgr2=0x7f0eb80b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebde00640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb80d0c00 0x7f0eb80b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebde00640 1 -- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb815e200 msgr2=0x7f0eb80b7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebde00640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb815e200 0x7f0eb80b7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebde00640 1 -- 172.21.15.114:0/3048798282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb80b1f60 con 0x7f0eb80b2e50 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebd400640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb80d0c00 0x7f0eb80b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebe800640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb815e200 0x7f0eb80b7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebde00640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80b2e50 0x7f0eb80b6b90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c460 tx=0x7f0eac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0eb2000640 1 -- 172.21.15.114:0/3048798282 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac018070 con 0x7f0eb80b2e50 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0eb2000640 1 -- 172.21.15.114:0/3048798282 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac007500 con 0x7f0eb80b2e50 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb80b22a0 con 0x7f0eb80b2e50 2024-03-06T07:35:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0eb2000640 1 -- 172.21.15.114:0/3048798282 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac004a20 con 0x7f0eb80b2e50 2024-03-06T07:35:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.277+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb80cbc70 con 0x7f0eb80b2e50 2024-03-06T07:35:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.279+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb804f330 con 0x7f0eb80b2e50 2024-03-06T07:35:31.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.279+0000 7f0eb2000640 1 -- 172.21.15.114:0/3048798282 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0eac01b070 con 0x7f0eb80b2e50 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.280+0000 7f0eb2000640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e9c06b4d0 0x7f0e9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.280+0000 7f0eb2000640 1 -- 172.21.15.114:0/3048798282 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f0eac089530 con 0x7f0eb80b2e50 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.281+0000 7f0ebd400640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e9c06b4d0 0x7f0e9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.281+0000 7f0ebd400640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e9c06b4d0 0x7f0e9c06d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8036000 tx=0x7f0ea8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.285+0000 7f0eb2000640 1 -- 172.21.15.114:0/3048798282 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0eb80b22a0 con 0x7f0eb80b2e50 2024-03-06T07:35:31.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.474+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0eb80b3c00 con 0x7f0eb80b2e50 2024-03-06T07:35:31.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.477+0000 7f0eb2000640 1 -- 172.21.15.114:0/3048798282 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+17089 (secure 0 0 0) 0x7f0eac058350 con 0x7f0eb80b2e50 2024-03-06T07:35:31.479 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:31.479 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":114,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:34:38.169073+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:34:04.207971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:34:05.420538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6835/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.177:6833/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.114:6837/860883596":"2024-03-07T07:34:21.018930+0000","172.21.15.114:6835/860883596":"2024-03-07T07:34:21.018930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.479+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e9c06b4d0 msgr2=0x7f0e9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e9c06b4d0 0x7f0e9c06d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8036000 tx=0x7f0ea8008030 comp rx=0 tx=0).stop 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80b2e50 msgr2=0x7f0eb80b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80b2e50 0x7f0eb80b6b90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c460 tx=0x7f0eac00c920 comp rx=0 tx=0).stop 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 shutdown_connections 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eb815e200 0x7f0eb80b7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0eb80d0c00 0x7f0eb80b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0e9c06b4d0 0x7f0e9c06d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 --2- 172.21.15.114:0/3048798282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0eb80b2e50 0x7f0eb80b6b90 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 >> 172.21.15.114:0/3048798282 conn(0x7f0eb80d3d30 msgr2=0x7f0eb80d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 shutdown_connections 2024-03-06T07:35:31.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.480+0000 7f0ebf200640 1 -- 172.21.15.114:0/3048798282 wait complete. 2024-03-06T07:35:31.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:35:31.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.689+0000 7fdf3f600640 1 Processor -- start 2024-03-06T07:35:31.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.689+0000 7fdf3f600640 1 -- start start 2024-03-06T07:35:31.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3f600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdf3815e3d0 0x7fdf3806c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3f600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380d0980 0x7fdf3806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3f600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf380d10f0 con 0x7fdf3806ccd0 2024-03-06T07:35:31.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3f600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf380b12c0 con 0x7fdf3815e3d0 2024-03-06T07:35:31.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3f600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380cc4d0 con 0x7fdf380d0980 2024-03-06T07:35:31.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdf3815e3d0 0x7fdf3806c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3cc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdf3815e3d0 0x7fdf3806c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49090/0 (socket says 172.21.15.114:49090) 2024-03-06T07:35:31.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf3cc00640 1 -- 172.21.15.114:0/22666353 learned_addr learned my addr 172.21.15.114:0/22666353 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:31.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.691+0000 7fdf37e00640 1 --2- 172.21.15.114:0/22666353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380d0980 0x7fdf3806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.692+0000 7fdf3cc00640 1 -- 172.21.15.114:0/22666353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380d0980 msgr2=0x7fdf3806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.692+0000 7fdf3cc00640 1 --2- 172.21.15.114:0/22666353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380d0980 0x7fdf3806c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.692+0000 7fdf3cc00640 1 -- 172.21.15.114:0/22666353 >> v1:172.21.15.177:6789/0 conn(0x7fdf3806ccd0 legacy=0x7fdf380ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.692+0000 7fdf3cc00640 1 -- 172.21.15.114:0/22666353 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380cc7b0 con 0x7fdf3815e3d0 2024-03-06T07:35:31.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.693+0000 7fdf3cc00640 1 --2- 172.21.15.114:0/22666353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf3815e3d0 0x7fdf3806c140 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009870 tx=0x7fdf2802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7cf79504d17ffa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf37400640 1 -- 172.21.15.114:0/22666353 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28034070 con 0x7fdf3815e3d0 2024-03-06T07:35:31.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf37400640 1 -- 172.21.15.114:0/22666353 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf28030840 con 0x7fdf3815e3d0 2024-03-06T07:35:31.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf37400640 1 -- 172.21.15.114:0/22666353 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28030c10 con 0x7fdf3815e3d0 2024-03-06T07:35:31.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 -- 172.21.15.114:0/22666353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf3815e3d0 msgr2=0x7fdf3806c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 --2- 172.21.15.114:0/22666353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf3815e3d0 0x7fdf3806c140 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009870 tx=0x7fdf2802b980 comp rx=0 tx=0).stop 2024-03-06T07:35:31.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 -- 172.21.15.114:0/22666353 shutdown_connections 2024-03-06T07:35:31.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 --2- 172.21.15.114:0/22666353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380d0980 0x7fdf3806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 --2- 172.21.15.114:0/22666353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf3815e3d0 0x7fdf3806c140 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 -- 172.21.15.114:0/22666353 >> 172.21.15.114:0/22666353 conn(0x7fdf380b3870 msgr2=0x7fdf380b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 -- 172.21.15.114:0/22666353 shutdown_connections 2024-03-06T07:35:31.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.694+0000 7fdf3f600640 1 -- 172.21.15.114:0/22666353 wait complete. 2024-03-06T07:35:31.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.696+0000 7fdf3f600640 1 Processor -- start 2024-03-06T07:35:31.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.696+0000 7fdf3f600640 1 -- start start 2024-03-06T07:35:31.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf3f600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3806ccd0 0x7fdf38152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf3f600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf380d0980 0x7fdf380b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf3f600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380b6cd0 0x7fdf380c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf3f600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380cdfc0 con 0x7fdf380d0980 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf3f600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380d0de0 con 0x7fdf3806ccd0 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf3f600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380ce140 con 0x7fdf380b6cd0 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf37e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf380d0980 0x7fdf380b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf37e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf380d0980 0x7fdf380b7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49100/0 (socket says 172.21.15.114:49100) 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.697+0000 7fdf37e00640 1 -- 172.21.15.114:0/439127261 learned_addr learned my addr 172.21.15.114:0/439127261 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf37e00640 1 -- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380b6cd0 msgr2=0x7fdf380c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf3cc00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3806ccd0 0x7fdf38152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf3d600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380b6cd0 0x7fdf380c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf37e00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380b6cd0 0x7fdf380c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf37e00640 1 -- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3806ccd0 msgr2=0x7fdf38152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf37e00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3806ccd0 0x7fdf38152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf37e00640 1 -- 172.21.15.114:0/439127261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380c6a50 con 0x7fdf380d0980 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf3d600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380b6cd0 0x7fdf380c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf3cc00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3806ccd0 0x7fdf38152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf37e00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf380d0980 0x7fdf380b7190 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fdf24007ef0 tx=0x7fdf2400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf34c00640 1 -- 172.21.15.114:0/439127261 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24014620 con 0x7fdf380d0980 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf34c00640 1 -- 172.21.15.114:0/439127261 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf24019070 con 0x7fdf380d0980 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf34c00640 1 -- 172.21.15.114:0/439127261 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24014a00 con 0x7fdf380d0980 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.698+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf380c59b0 con 0x7fdf380d0980 2024-03-06T07:35:31.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.699+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf380c5f70 con 0x7fdf380d0980 2024-03-06T07:35:31.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.700+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3804f330 con 0x7fdf380d0980 2024-03-06T07:35:31.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.700+0000 7fdf34c00640 1 -- 172.21.15.114:0/439127261 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf24014ba0 con 0x7fdf380d0980 2024-03-06T07:35:31.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.702+0000 7fdf34c00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf1806b4d0 0x7fdf1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.702+0000 7fdf34c00640 1 -- 172.21.15.114:0/439127261 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fdf2408ad40 con 0x7fdf380d0980 2024-03-06T07:35:31.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.702+0000 7fdf3cc00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf1806b4d0 0x7fdf1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.703+0000 7fdf3cc00640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf1806b4d0 0x7fdf1806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdf28036000 tx=0x7fdf2803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.706+0000 7fdf34c00640 1 -- 172.21.15.114:0/439127261 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf24059a30 con 0x7fdf380d0980 2024-03-06T07:35:31.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.939+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdf3806f180 con 0x7fdf380d0980 2024-03-06T07:35:31.941 INFO:tasks.ceph.mon.a.smithi114.stderr:2024-03-06T07:35:31.940+0000 7fc8e2a00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:35:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.943+0000 7fdf34c00640 1 -- 172.21.15.114:0/439127261 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdf24059330 con 0x7fdf380d0980 2024-03-06T07:35:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:35:31.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf1806b4d0 msgr2=0x7fdf1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf1806b4d0 0x7fdf1806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdf28036000 tx=0x7fdf2803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:31.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf380d0980 msgr2=0x7fdf380b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf380d0980 0x7fdf380b7190 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fdf24007ef0 tx=0x7fdf2400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 shutdown_connections 2024-03-06T07:35:31.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf380b6cd0 0x7fdf380c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf1806b4d0 0x7fdf1806d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf380d0980 0x7fdf380b7190 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 --2- 172.21.15.114:0/439127261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3806ccd0 0x7fdf38152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 >> 172.21.15.114:0/439127261 conn(0x7fdf380b3870 msgr2=0x7fdf380b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 shutdown_connections 2024-03-06T07:35:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:31.946+0000 7fdf3f600640 1 -- 172.21.15.114:0/439127261 wait complete. 2024-03-06T07:35:31.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:35:31.965 INFO:tasks.ceph.mds.a.smithi114.stderr: -8183> 2024-03-06T07:34:28.071+0000 7f4c28800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:32.026 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:35:32.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.149+0000 7fab3fc00640 1 Processor -- start 2024-03-06T07:35:32.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.149+0000 7fab3fc00640 1 -- start start 2024-03-06T07:35:32.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.150+0000 7fab3fc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380b3530 0x7fab3815b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.150+0000 7fab3fc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fab380b3910 0x7fab380acf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.151+0000 7fab3fc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3815bcc0 con 0x7fab380ad4a0 2024-03-06T07:35:32.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.151+0000 7fab3fc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab380ce690 con 0x7fab380b3910 2024-03-06T07:35:32.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.151+0000 7fab3fc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab380ce970 con 0x7fab380b3530 2024-03-06T07:35:32.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.151+0000 7fab3d200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380b3530 0x7fab3815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.151+0000 7fab3d200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380b3530 0x7fab3815b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:58558/0 (socket says 172.21.15.114:58558) 2024-03-06T07:35:32.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.151+0000 7fab3d200640 1 -- 172.21.15.114:0/1109254333 learned_addr learned my addr 172.21.15.114:0/1109254333 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:32.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.152+0000 7fab2fe00640 1 --2- 172.21.15.114:0/1109254333 >> v2:172.21.15.177:3300/0 conn(0x7fab380b3910 0x7fab380acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.152+0000 7fab3d200640 1 -- 172.21.15.114:0/1109254333 >> v2:172.21.15.177:3300/0 conn(0x7fab380b3910 msgr2=0x7fab380acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.152+0000 7fab3d200640 1 --2- 172.21.15.114:0/1109254333 >> v2:172.21.15.177:3300/0 conn(0x7fab380b3910 0x7fab380acf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.152+0000 7fab3d200640 1 -- 172.21.15.114:0/1109254333 >> v1:172.21.15.177:6789/0 conn(0x7fab380ad4a0 legacy=0x7fab380ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.152+0000 7fab3d200640 1 -- 172.21.15.114:0/1109254333 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab380cec50 con 0x7fab380b3530 2024-03-06T07:35:32.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.153+0000 7fab3d200640 1 --2- 172.21.15.114:0/1109254333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380b3530 0x7fab3815b780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fab2800aa40 tx=0x7fab2802cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4594720f134b8a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab37e00640 1 -- 172.21.15.114:0/1109254333 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2803a070 con 0x7fab380b3530 2024-03-06T07:35:32.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab37e00640 1 -- 172.21.15.114:0/1109254333 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2802db60 con 0x7fab380b3530 2024-03-06T07:35:32.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab37e00640 1 -- 172.21.15.114:0/1109254333 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2800b040 con 0x7fab380b3530 2024-03-06T07:35:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 -- 172.21.15.114:0/1109254333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380b3530 msgr2=0x7fab3815b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 --2- 172.21.15.114:0/1109254333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380b3530 0x7fab3815b780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fab2800aa40 tx=0x7fab2802cdb0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 -- 172.21.15.114:0/1109254333 shutdown_connections 2024-03-06T07:35:32.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 --2- 172.21.15.114:0/1109254333 >> v2:172.21.15.177:3300/0 conn(0x7fab380b3910 0x7fab380acf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 --2- 172.21.15.114:0/1109254333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380b3530 0x7fab3815b780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 -- 172.21.15.114:0/1109254333 >> 172.21.15.114:0/1109254333 conn(0x7fab380b1940 msgr2=0x7fab380b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 -- 172.21.15.114:0/1109254333 shutdown_connections 2024-03-06T07:35:32.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.154+0000 7fab3fc00640 1 -- 172.21.15.114:0/1109254333 wait complete. 2024-03-06T07:35:32.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.156+0000 7fab3fc00640 1 Processor -- start 2024-03-06T07:35:32.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.156+0000 7fab3fc00640 1 -- start start 2024-03-06T07:35:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3fc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380ad4a0 0x7fab38154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3fc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab380b3530 0x7fab381595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3fc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab380b3910 0x7fab38159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3fc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab380b3d90 con 0x7fab380b3910 2024-03-06T07:35:32.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3fc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab380cf210 con 0x7fab380b3530 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3fc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3815aa30 con 0x7fab380ad4a0 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab380b3910 0x7fab38159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab380b3910 0x7fab38159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49126/0 (socket says 172.21.15.114:49126) 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.157+0000 7fab3dc00640 1 -- 172.21.15.114:0/2896143362 learned_addr learned my addr 172.21.15.114:0/2896143362 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3d200640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380ad4a0 0x7fab38154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3dc00640 1 -- 172.21.15.114:0/2896143362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380ad4a0 msgr2=0x7fab38154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3dc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380ad4a0 0x7fab38154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3dc00640 1 -- 172.21.15.114:0/2896143362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab380b3530 msgr2=0x7fab381595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3dc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab380b3530 0x7fab381595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3dc00640 1 -- 172.21.15.114:0/2896143362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3815a100 con 0x7fab380b3910 2024-03-06T07:35:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3dc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab380b3910 0x7fab38159af0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fab3000c780 tx=0x7fab3000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab35600640 1 -- 172.21.15.114:0/2896143362 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30004b60 con 0x7fab380b3910 2024-03-06T07:35:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab35600640 1 -- 172.21.15.114:0/2896143362 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30004d00 con 0x7fab380b3910 2024-03-06T07:35:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab35600640 1 -- 172.21.15.114:0/2896143362 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3000aa90 con 0x7fab380b3910 2024-03-06T07:35:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.158+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab381f4440 con 0x7fab380b3910 2024-03-06T07:35:32.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.159+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab381f4a00 con 0x7fab380b3910 2024-03-06T07:35:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.160+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3804f330 con 0x7fab380b3910 2024-03-06T07:35:32.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.161+0000 7fab35600640 1 -- 172.21.15.114:0/2896143362 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab3000ac30 con 0x7fab380b3910 2024-03-06T07:35:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.162+0000 7fab35600640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab1406b4d0 0x7fab1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.162+0000 7fab35600640 1 -- 172.21.15.114:0/2896143362 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fab3008a590 con 0x7fab380b3910 2024-03-06T07:35:32.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.162+0000 7fab3d200640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab1406b4d0 0x7fab1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.162+0000 7fab3d200640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab1406b4d0 0x7fab1406d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fab28037000 tx=0x7fab2800c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.166+0000 7fab35600640 1 -- 172.21.15.114:0/2896143362 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab30020070 con 0x7fab380b3910 2024-03-06T07:35:32.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.402+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fab380ae2a0 con 0x7fab380b3910 2024-03-06T07:35:32.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.946+0000 7fab35600640 1 -- 172.21.15.114:0/2896143362 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) v1 ==== 104+0+0 (secure 0 0 0) 0x7fab30059360 con 0x7fab380b3910 2024-03-06T07:35:32.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.949+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab1406b4d0 msgr2=0x7fab1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.949+0000 7fab3fc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab1406b4d0 0x7fab1406d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fab28037000 tx=0x7fab2800c040 comp rx=0 tx=0).stop 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.949+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab380b3910 msgr2=0x7fab38159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.949+0000 7fab3fc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab380b3910 0x7fab38159af0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fab3000c780 tx=0x7fab3000cc40 comp rx=0 tx=0).stop 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 shutdown_connections 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fab1406b4d0 0x7fab1406d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab380b3910 0x7fab38159af0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab380b3530 0x7fab381595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 --2- 172.21.15.114:0/2896143362 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fab380ad4a0 0x7fab38154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 >> 172.21.15.114:0/2896143362 conn(0x7fab380b1940 msgr2=0x7fab380cb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 shutdown_connections 2024-03-06T07:35:32.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:32.950+0000 7fab3fc00640 1 -- 172.21.15.114:0/2896143362 wait complete. 2024-03-06T07:35:32.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:35:33.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.188+0000 7f2f9a400640 1 Processor -- start 2024-03-06T07:35:33.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.188+0000 7f2f9a400640 1 -- start start 2024-03-06T07:35:33.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.189+0000 7f2f9a400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2f9415df30 0x7f2f940cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.189+0000 7f2f9a400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2f940d47b0 0x7f2f940d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f9a400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f940cfa00 con 0x7f2f940d06a0 2024-03-06T07:35:33.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f9a400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f940d1860 con 0x7f2f9415df30 2024-03-06T07:35:33.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f9a400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f940d1ab0 con 0x7f2f940d47b0 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2f9415df30 0x7f2f940cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2f9415df30 0x7f2f940cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49142/0 (socket says 172.21.15.114:49142) 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99000640 1 -- 172.21.15.114:0/1839943840 learned_addr learned my addr 172.21.15.114:0/1839943840 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99a00640 1 --1- 172.21.15.114:0/1839943840 >> v1:172.21.15.114:6789/0 conn(0x7f2f940d06a0 0x7f2f940d0ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58940/0 (socket says 172.21.15.114:58940) 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99000640 1 -- 172.21.15.114:0/1839943840 >> v2:172.21.15.177:3300/0 conn(0x7f2f940d47b0 msgr2=0x7f2f940d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f93e00640 1 --2- 172.21.15.114:0/1839943840 >> v2:172.21.15.177:3300/0 conn(0x7f2f940d47b0 0x7f2f940d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99000640 1 --2- 172.21.15.114:0/1839943840 >> v2:172.21.15.177:3300/0 conn(0x7f2f940d47b0 0x7f2f940d2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99000640 1 -- 172.21.15.114:0/1839943840 >> v1:172.21.15.114:6789/0 conn(0x7f2f940d06a0 legacy=0x7f2f940d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f99000640 1 -- 172.21.15.114:0/1839943840 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940d1d90 con 0x7f2f9415df30 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.191+0000 7f2f93e00640 1 --2- 172.21.15.114:0/1839943840 >> v2:172.21.15.177:3300/0 conn(0x7f2f940d47b0 0x7f2f940d2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:33.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.190+0000 7f2f93400640 1 -- 172.21.15.114:0/1839943840 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304673486 0 0) 0x7f2f940cfa00 con 0x7f2f940d06a0 2024-03-06T07:35:33.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.192+0000 7f2f99000640 1 --2- 172.21.15.114:0/1839943840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f9415df30 0x7f2f940cf2b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f2f940d1380 tx=0x7f2f8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d8d24f504ce67f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.192+0000 7f2f93400640 1 -- 172.21.15.114:0/1839943840 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f88035020 con 0x7f2f9415df30 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.192+0000 7f2f93400640 1 -- 172.21.15.114:0/1839943840 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f88031870 con 0x7f2f9415df30 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.192+0000 7f2f93400640 1 -- 172.21.15.114:0/1839943840 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f88031c40 con 0x7f2f9415df30 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 -- 172.21.15.114:0/1839943840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f9415df30 msgr2=0x7f2f940cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 --2- 172.21.15.114:0/1839943840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f9415df30 0x7f2f940cf2b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f2f940d1380 tx=0x7f2f8802b910 comp rx=0 tx=0).stop 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 -- 172.21.15.114:0/1839943840 shutdown_connections 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 --2- 172.21.15.114:0/1839943840 >> v2:172.21.15.177:3300/0 conn(0x7f2f940d47b0 0x7f2f940d2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 --2- 172.21.15.114:0/1839943840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f9415df30 0x7f2f940cf2b0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 -- 172.21.15.114:0/1839943840 >> 172.21.15.114:0/1839943840 conn(0x7f2f940d3c90 msgr2=0x7f2f940d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 -- 172.21.15.114:0/1839943840 shutdown_connections 2024-03-06T07:35:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.193+0000 7f2f9a400640 1 -- 172.21.15.114:0/1839943840 wait complete. 2024-03-06T07:35:33.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.194+0000 7f2f9a400640 1 Processor -- start 2024-03-06T07:35:33.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.195+0000 7f2f9a400640 1 -- start start 2024-03-06T07:35:33.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.195+0000 7f2f9a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f940d06a0 0x7f2f941521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.195+0000 7f2f9a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f940d47b0 0x7f2f940b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.195+0000 7f2f9a400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f9415df30 0x7f2f940b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.195+0000 7f2f9a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940d2c50 con 0x7f2f940d47b0 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.195+0000 7f2f9a400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940d2ad0 con 0x7f2f940d06a0 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.195+0000 7f2f9a400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9415e310 con 0x7f2f9415df30 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f940d47b0 0x7f2f940b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f940d47b0 0x7f2f940b67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49152/0 (socket says 172.21.15.114:49152) 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 -- 172.21.15.114:0/3671954022 learned_addr learned my addr 172.21.15.114:0/3671954022 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f99a00640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f9415df30 0x7f2f940b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 -- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f9415df30 msgr2=0x7f2f940b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f9415df30 0x7f2f940b2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 -- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f940d06a0 msgr2=0x7f2f941521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f99000640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f940d06a0 0x7f2f941521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f940d06a0 0x7f2f941521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.196+0000 7f2f93e00640 1 -- 172.21.15.114:0/3671954022 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940b71d0 con 0x7f2f940d47b0 2024-03-06T07:35:33.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.197+0000 7f2f93e00640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f940d47b0 0x7f2f940b67f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000c7f0 tx=0x7f2f8000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.197+0000 7f2f90c00640 1 -- 172.21.15.114:0/3671954022 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80015070 con 0x7f2f940d47b0 2024-03-06T07:35:33.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.197+0000 7f2f90c00640 1 -- 172.21.15.114:0/3671954022 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f80007a30 con 0x7f2f940d47b0 2024-03-06T07:35:33.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.197+0000 7f2f90c00640 1 -- 172.21.15.114:0/3671954022 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80007e00 con 0x7f2f940d47b0 2024-03-06T07:35:33.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.197+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f940b74b0 con 0x7f2f940d47b0 2024-03-06T07:35:33.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.197+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f940b7a70 con 0x7f2f940d47b0 2024-03-06T07:35:33.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.198+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9404f330 con 0x7f2f940d47b0 2024-03-06T07:35:33.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.199+0000 7f2f90c00640 1 -- 172.21.15.114:0/3671954022 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f80018070 con 0x7f2f940d47b0 2024-03-06T07:35:33.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.200+0000 7f2f90c00640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f7406b4d0 0x7f2f7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.200+0000 7f2f90c00640 1 -- 172.21.15.114:0/3671954022 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f2f8008a390 con 0x7f2f940d47b0 2024-03-06T07:35:33.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.200+0000 7f2f99000640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f7406b4d0 0x7f2f7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.201+0000 7f2f99000640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f7406b4d0 0x7f2f7406d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2f88036000 tx=0x7f2f88008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.204+0000 7f2f90c00640 1 -- 172.21.15.114:0/3671954022 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f80059030 con 0x7f2f940d47b0 2024-03-06T07:35:33.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.399+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f9408d430 con 0x7f2f940d47b0 2024-03-06T07:35:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.959+0000 7f2f90c00640 1 -- 172.21.15.114:0/3671954022 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2f80058930 con 0x7f2f940d47b0 2024-03-06T07:35:33.961 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f7406b4d0 msgr2=0x7f2f7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f7406b4d0 0x7f2f7406d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2f88036000 tx=0x7f2f88008030 comp rx=0 tx=0).stop 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f940d47b0 msgr2=0x7f2f940b67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f940d47b0 0x7f2f940b67f0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000c7f0 tx=0x7f2f8000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 shutdown_connections 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2f9415df30 0x7f2f940b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2f7406b4d0 0x7f2f7406d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f940d47b0 0x7f2f940b67f0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 --2- 172.21.15.114:0/3671954022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2f940d06a0 0x7f2f941521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.963+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 >> 172.21.15.114:0/3671954022 conn(0x7f2f940d3c90 msgr2=0x7f2f940cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.964+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 shutdown_connections 2024-03-06T07:35:33.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:33.964+0000 7f2f9a400640 1 -- 172.21.15.114:0/3671954022 wait complete. 2024-03-06T07:35:33.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:35:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.191+0000 7f8800e00640 1 Processor -- start 2024-03-06T07:35:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.191+0000 7f8800e00640 1 -- start start 2024-03-06T07:35:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f8800e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f87fc145980 0x7f87fc15fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f8800e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87fc0ced20 0x7f87fc15ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f8800e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0cf1e0 con 0x7f87fc0712d0 2024-03-06T07:35:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f8800e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc070e40 con 0x7f87fc0ced20 2024-03-06T07:35:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f8800e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc0cf4c0 con 0x7f87fc145980 2024-03-06T07:35:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f87f1400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87fc0ced20 0x7f87fc15ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f87f1400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87fc0ced20 0x7f87fc15ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49156/0 (socket says 172.21.15.114:49156) 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f87f1400640 1 -- 172.21.15.114:0/3145105785 learned_addr learned my addr 172.21.15.114:0/3145105785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f87f9e00640 1 --2- 172.21.15.114:0/3145105785 >> v2:172.21.15.177:3300/0 conn(0x7f87fc145980 0x7f87fc15fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f87f1400640 1 -- 172.21.15.114:0/3145105785 >> v2:172.21.15.177:3300/0 conn(0x7f87fc145980 msgr2=0x7f87fc15fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.193+0000 7f87f1400640 1 --2- 172.21.15.114:0/3145105785 >> v2:172.21.15.177:3300/0 conn(0x7f87fc145980 0x7f87fc15fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.194+0000 7f87f1400640 1 -- 172.21.15.114:0/3145105785 >> v1:172.21.15.114:6789/0 conn(0x7f87fc0712d0 legacy=0x7f87fc0716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.194+0000 7f87f1400640 1 -- 172.21.15.114:0/3145105785 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc0cf7a0 con 0x7f87fc0ced20 2024-03-06T07:35:34.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.194+0000 7f87f9e00640 1 --2- 172.21.15.114:0/3145105785 >> v2:172.21.15.177:3300/0 conn(0x7f87fc145980 0x7f87fc15fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:34.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.195+0000 7f87f1400640 1 --2- 172.21.15.114:0/3145105785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0ced20 0x7f87fc15ff40 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f87fc074fb0 tx=0x7f87e402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34402c85f18d0a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.195+0000 7f87f9400640 1 -- 172.21.15.114:0/3145105785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4034070 con 0x7f87fc0ced20 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.195+0000 7f87f9400640 1 -- 172.21.15.114:0/3145105785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e4030870 con 0x7f87fc0ced20 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.195+0000 7f87f9400640 1 -- 172.21.15.114:0/3145105785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4030c40 con 0x7f87fc0ced20 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.195+0000 7f8800e00640 1 -- 172.21.15.114:0/3145105785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0ced20 msgr2=0x7f87fc15ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.195+0000 7f8800e00640 1 --2- 172.21.15.114:0/3145105785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0ced20 0x7f87fc15ff40 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f87fc074fb0 tx=0x7f87e402b9b0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.196+0000 7f8800e00640 1 -- 172.21.15.114:0/3145105785 shutdown_connections 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.196+0000 7f8800e00640 1 --2- 172.21.15.114:0/3145105785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0ced20 0x7f87fc15ff40 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.196+0000 7f8800e00640 1 --2- 172.21.15.114:0/3145105785 >> v2:172.21.15.177:3300/0 conn(0x7f87fc145980 0x7f87fc15fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.196+0000 7f8800e00640 1 -- 172.21.15.114:0/3145105785 >> 172.21.15.114:0/3145105785 conn(0x7f87fc0cfa80 msgr2=0x7f87fc0cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.196+0000 7f8800e00640 1 -- 172.21.15.114:0/3145105785 shutdown_connections 2024-03-06T07:35:34.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.196+0000 7f8800e00640 1 -- 172.21.15.114:0/3145105785 wait complete. 2024-03-06T07:35:34.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.197+0000 7f8800e00640 1 Processor -- start 2024-03-06T07:35:34.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.198+0000 7f8800e00640 1 -- start start 2024-03-06T07:35:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.198+0000 7f8800e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0712d0 0x7f87fc15d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.198+0000 7f8800e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0ced20 0x7f87fc138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.198+0000 7f8800e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc145980 0x7f87fc13bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.198+0000 7f8800e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc1608c0 con 0x7f87fc0712d0 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.198+0000 7f8800e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc160740 con 0x7f87fc0ced20 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.198+0000 7f8800e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc06dec0 con 0x7f87fc145980 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0712d0 0x7f87fc15d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0712d0 0x7f87fc15d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49162/0 (socket says 172.21.15.114:49162) 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 -- 172.21.15.114:0/1504804842 learned_addr learned my addr 172.21.15.114:0/1504804842 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 -- 172.21.15.114:0/1504804842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc145980 msgr2=0x7f87fc13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc145980 0x7f87fc13bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f1400640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0ced20 0x7f87fc138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 -- 172.21.15.114:0/1504804842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0ced20 msgr2=0x7f87fc138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0ced20 0x7f87fc138d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 -- 172.21.15.114:0/1504804842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc15d880 con 0x7f87fc0712d0 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.199+0000 7f87f9e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0712d0 0x7f87fc15d270 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00c7f0 tx=0x7f87ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.200+0000 7f87f2a00640 1 -- 172.21.15.114:0/1504804842 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec015070 con 0x7f87fc0712d0 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.200+0000 7f87f2a00640 1 -- 172.21.15.114:0/1504804842 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec007820 con 0x7f87fc0712d0 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.200+0000 7f87f2a00640 1 -- 172.21.15.114:0/1504804842 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec007bd0 con 0x7f87fc0712d0 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.200+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc15db60 con 0x7f87fc0712d0 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.200+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc15e120 con 0x7f87fc0712d0 2024-03-06T07:35:34.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.201+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc04f330 con 0x7f87fc0712d0 2024-03-06T07:35:34.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.202+0000 7f87f2a00640 1 -- 172.21.15.114:0/1504804842 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87ec018070 con 0x7f87fc0712d0 2024-03-06T07:35:34.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.203+0000 7f87f2a00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.204+0000 7f87f2a00640 1 -- 172.21.15.114:0/1504804842 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5587+0+0 (secure 0 0 0) 0x7f87ec08a350 con 0x7f87fc0712d0 2024-03-06T07:35:34.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.204+0000 7f87f1400640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.204+0000 7f87f1400640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f87e4036000 tx=0x7f87e403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.208+0000 7f87f2a00640 1 -- 172.21.15.114:0/1504804842 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87ec0591a0 con 0x7f87fc0712d0 2024-03-06T07:35:34.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.403+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f87fc15f620 con 0x7f87fc0712d0 2024-03-06T07:35:34.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.962+0000 7f87f2a00640 1 -- 172.21.15.114:0/1504804842 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) v1 ==== 169+0+0 (secure 0 0 0) 0x7f87ec058aa0 con 0x7f87fc0712d0 2024-03-06T07:35:34.964 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 msgr2=0x7f87d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f87e4036000 tx=0x7f87e403b040 comp rx=0 tx=0).stop 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0712d0 msgr2=0x7f87fc15d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0712d0 0x7f87fc15d270 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00c7f0 tx=0x7f87ec00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 shutdown_connections 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87fc145980 0x7f87fc13bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87fc0ced20 0x7f87fc138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 --2- 172.21.15.114:0/1504804842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87fc0712d0 0x7f87fc15d270 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.968+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 >> 172.21.15.114:0/1504804842 conn(0x7f87fc0cfa80 msgr2=0x7f87fc0af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.969+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 shutdown_connections 2024-03-06T07:35:34.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:34.969+0000 7f8800e00640 1 -- 172.21.15.114:0/1504804842 wait complete. 2024-03-06T07:35:35.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:35:35.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.188+0000 7f9ffb400640 1 Processor -- start 2024-03-06T07:35:35.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.188+0000 7f9ffb400640 1 -- start start 2024-03-06T07:35:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.189+0000 7f9ffb400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9ff40b3df0 0x7f9ff40727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.189+0000 7f9ffb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9ff40b3650 0x7f9ff40b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.189+0000 7f9ffb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40cda90 0x7f9ff406c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.189+0000 7f9ffb400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff4072310 con 0x7f9ff40b3650 2024-03-06T07:35:35.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ffb400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff40b4710 con 0x7f9ff40b3df0 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ffb400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40719e0 con 0x7f9ff40cda90 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9ff40b3650 0x7f9ff40b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9ff40b3650 0x7f9ff40b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49166/0 (socket says 172.21.15.114:49166) 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 -- 172.21.15.114:0/1109404156 learned_addr learned my addr 172.21.15.114:0/1109404156 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff8a00640 1 --2- 172.21.15.114:0/1109404156 >> v2:172.21.15.177:3300/0 conn(0x7f9ff40b3df0 0x7f9ff40727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff9400640 1 --2- 172.21.15.114:0/1109404156 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40cda90 0x7f9ff406c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 -- 172.21.15.114:0/1109404156 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40cda90 msgr2=0x7f9ff406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 --2- 172.21.15.114:0/1109404156 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40cda90 0x7f9ff406c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 -- 172.21.15.114:0/1109404156 >> v2:172.21.15.177:3300/0 conn(0x7f9ff40b3df0 msgr2=0x7f9ff40727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 --2- 172.21.15.114:0/1109404156 >> v2:172.21.15.177:3300/0 conn(0x7f9ff40b3df0 0x7f9ff40727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff3e00640 1 -- 172.21.15.114:0/1109404156 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4071cc0 con 0x7f9ff40b3650 2024-03-06T07:35:35.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.190+0000 7f9ff8a00640 1 --2- 172.21.15.114:0/1109404156 >> v2:172.21.15.177:3300/0 conn(0x7f9ff40b3df0 0x7f9ff40727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.191+0000 7f9ff3e00640 1 --2- 172.21.15.114:0/1109404156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3650 0x7f9ff40b41d0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40b3dc0 tx=0x7f9fdc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62b1b951795efb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ff3400640 1 -- 172.21.15.114:0/1109404156 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc035020 con 0x7f9ff40b3650 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ff3400640 1 -- 172.21.15.114:0/1109404156 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc02d040 con 0x7f9ff40b3650 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ff3400640 1 -- 172.21.15.114:0/1109404156 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc031a80 con 0x7f9ff40b3650 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ffb400640 1 -- 172.21.15.114:0/1109404156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3650 msgr2=0x7f9ff40b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ffb400640 1 --2- 172.21.15.114:0/1109404156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3650 0x7f9ff40b41d0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40b3dc0 tx=0x7f9fdc02b900 comp rx=0 tx=0).stop 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ffb400640 1 -- 172.21.15.114:0/1109404156 shutdown_connections 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ffb400640 1 --2- 172.21.15.114:0/1109404156 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40cda90 0x7f9ff406c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ffb400640 1 --2- 172.21.15.114:0/1109404156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3650 0x7f9ff40b41d0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ffb400640 1 --2- 172.21.15.114:0/1109404156 >> v2:172.21.15.177:3300/0 conn(0x7f9ff40b3df0 0x7f9ff40727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.192+0000 7f9ffb400640 1 -- 172.21.15.114:0/1109404156 >> 172.21.15.114:0/1109404156 conn(0x7f9ff40b1940 msgr2=0x7f9ff40b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.193+0000 7f9ffb400640 1 -- 172.21.15.114:0/1109404156 shutdown_connections 2024-03-06T07:35:35.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.193+0000 7f9ffb400640 1 -- 172.21.15.114:0/1109404156 wait complete. 2024-03-06T07:35:35.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.194+0000 7f9ffb400640 1 Processor -- start 2024-03-06T07:35:35.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.194+0000 7f9ffb400640 1 -- start start 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.195+0000 7f9ffb400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40b3650 0x7f9ff414dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.195+0000 7f9ffb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3df0 0x7f9ff414f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.195+0000 7f9ffb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ff40cda90 0x7f9ff4144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.195+0000 7f9ffb400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff415b240 con 0x7f9ff40b3df0 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.195+0000 7f9ffb400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40b3b30 con 0x7f9ff40cda90 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.195+0000 7f9ffb400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff415b3c0 con 0x7f9ff40b3650 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3df0 0x7f9ff414f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3df0 0x7f9ff414f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49170/0 (socket says 172.21.15.114:49170) 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 -- 172.21.15.114:0/3936191463 learned_addr learned my addr 172.21.15.114:0/3936191463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:35.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff9400640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ff40cda90 0x7f9ff4144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff8a00640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40b3650 0x7f9ff414dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 -- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40b3650 msgr2=0x7f9ff414dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40b3650 0x7f9ff414dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 -- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ff40cda90 msgr2=0x7f9ff4144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ff40cda90 0x7f9ff4144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 -- 172.21.15.114:0/3936191463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4148b80 con 0x7f9ff40b3df0 2024-03-06T07:35:35.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.196+0000 7f9ff3e00640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3df0 0x7f9ff414f260 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc02b940 tx=0x7f9fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.197+0000 7f9ff0c00640 1 -- 172.21.15.114:0/3936191463 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc040070 con 0x7f9ff40b3df0 2024-03-06T07:35:35.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.197+0000 7f9ff0c00640 1 -- 172.21.15.114:0/3936191463 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc039c30 con 0x7f9ff40b3df0 2024-03-06T07:35:35.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.197+0000 7f9ff0c00640 1 -- 172.21.15.114:0/3936191463 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc035040 con 0x7f9ff40b3df0 2024-03-06T07:35:35.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.197+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff414e230 con 0x7f9ff40b3df0 2024-03-06T07:35:35.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.197+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff4149100 con 0x7f9ff40b3df0 2024-03-06T07:35:35.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.198+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff40b0d70 con 0x7f9ff40b3df0 2024-03-06T07:35:35.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.199+0000 7f9ff0c00640 1 -- 172.21.15.114:0/3936191463 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fdc038020 con 0x7f9ff40b3df0 2024-03-06T07:35:35.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.200+0000 7f9ff0c00640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fd006b4d0 0x7f9fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.200+0000 7f9ff0c00640 1 -- 172.21.15.114:0/3936191463 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f9fdc0abe40 con 0x7f9ff40b3df0 2024-03-06T07:35:35.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.200+0000 7f9ff8a00640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fd006b4d0 0x7f9fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.201+0000 7f9ff8a00640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fd006b4d0 0x7f9fd006d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4008f10 tx=0x7f9fe4008460 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.205+0000 7f9ff0c00640 1 -- 172.21.15.114:0/3936191463 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ff40b0d70 con 0x7f9ff40b3df0 2024-03-06T07:35:35.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.402+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9ff404f330 con 0x7f9ff40b3df0 2024-03-06T07:35:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.407+0000 7f9ff0c00640 1 -- 172.21.15.114:0/3936191463 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9fdc07ade0 con 0x7f9ff40b3df0 2024-03-06T07:35:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fd006b4d0 msgr2=0x7f9fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fd006b4d0 0x7f9fd006d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4008f10 tx=0x7f9fe4008460 comp rx=0 tx=0).stop 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3df0 msgr2=0x7f9ff414f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3df0 0x7f9ff414f260 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc02b940 tx=0x7f9fdc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 shutdown_connections 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ff40cda90 0x7f9ff4144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9fd006b4d0 0x7f9fd006d990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ff40b3df0 0x7f9ff414f260 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 --2- 172.21.15.114:0/3936191463 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ff40b3650 0x7f9ff414dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 >> 172.21.15.114:0/3936191463 conn(0x7f9ff40b1940 msgr2=0x7f9ff40ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 shutdown_connections 2024-03-06T07:35:35.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.411+0000 7f9ffb400640 1 -- 172.21.15.114:0/3936191463 wait complete. 2024-03-06T07:35:35.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:35:35.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.623+0000 7fbab9800640 1 Processor -- start 2024-03-06T07:35:35.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.623+0000 7fbab9800640 1 -- start start 2024-03-06T07:35:35.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.625+0000 7fbab9800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 0x7fbab40ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.625+0000 7fbab9800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab415b090 con 0x7fbab40b4680 2024-03-06T07:35:35.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.625+0000 7fbab9800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40b2a30 con 0x7fbab415acb0 2024-03-06T07:35:35.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.625+0000 7fbab9800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40b4360 con 0x7fbab406ec00 2024-03-06T07:35:35.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.625+0000 7fbaaa000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbab40b4680 0x7fbab406e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58952/0 (socket says 172.21.15.114:58952) 2024-03-06T07:35:35.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.625+0000 7fbaaa000640 1 -- 172.21.15.114:0/610371705 learned_addr learned my addr 172.21.15.114:0/610371705 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:35.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab3400640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 0x7fbab40ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932877928 0 0) 0x7fbab415b090 con 0x7fbab40b4680 2024-03-06T07:35:35.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba98003520 con 0x7fbab40b4680 2024-03-06T07:35:35.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3116251956 0 0) 0x7fba98003520 con 0x7fbab40b4680 2024-03-06T07:35:35.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 msgr2=0x7fbab40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 0x7fbab40ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 >> v1:172.21.15.177:6789/0 conn(0x7fbab415acb0 legacy=0x7fbab40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40b2cf0 con 0x7fbab40b4680 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fba9c001670 con 0x7fbab40b4680 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 >> v1:172.21.15.114:6789/0 conn(0x7fbab40b4680 legacy=0x7fbab406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 0x7fba980039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98003f60 0x7fba98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba98004d40 0x7fba980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40b4360 con 0x7fba98004d40 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9800baa0 con 0x7fba98003f60 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.626+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9800bca0 con 0x7fbab406ec00 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab3400640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba98004d40 0x7fba980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab2a00640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 0x7fba980039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab3400640 1 -- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 msgr2=0x7fba980039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab3400640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 0x7fba980039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab3400640 1 -- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98003f60 msgr2=0x7fba98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab3400640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98003f60 0x7fba98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab3400640 1 -- 172.21.15.114:0/610371705 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98003520 con 0x7fba98004d40 2024-03-06T07:35:35.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.627+0000 7fbab2a00640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab406ec00 0x7fba980039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.628+0000 7fbab3400640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba98004d40 0x7fba980051c0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009650 tx=0x7fba98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913e6567e9efe897 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.628+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac03b070 con 0x7fba98004d40 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.628+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac02e890 con 0x7fba98004d40 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.628+0000 7fbab2000640 1 -- 172.21.15.114:0/610371705 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac009690 con 0x7fba98004d40 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.629+0000 7fbab9800640 1 -- 172.21.15.114:0/610371705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba98004d40 msgr2=0x7fba980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.629+0000 7fbab9800640 1 --2- 172.21.15.114:0/610371705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba98004d40 0x7fba980051c0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009650 tx=0x7fba98005830 comp rx=0 tx=0).stop 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.629+0000 7fbab3400640 1 -- 172.21.15.114:0/610371705 reap_dead start 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.629+0000 7fbab9800640 1 -- 172.21.15.114:0/610371705 shutdown_connections 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.629+0000 7fbab9800640 1 -- 172.21.15.114:0/610371705 >> 172.21.15.114:0/610371705 conn(0x7fbab40b1730 msgr2=0x7fbab40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.629+0000 7fbab9800640 1 -- 172.21.15.114:0/610371705 shutdown_connections 2024-03-06T07:35:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.629+0000 7fbab9800640 1 -- 172.21.15.114:0/610371705 wait complete. 2024-03-06T07:35:35.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.631+0000 7fbab9800640 1 Processor -- start 2024-03-06T07:35:35.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.631+0000 7fbab9800640 1 -- start start 2024-03-06T07:35:35.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98004d40 0x7fbab4147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab9800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab40ce000 0x7fbab40ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab415b100 0x7fbab415b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab9800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40b4010 con 0x7fbab415b100 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40b32b0 con 0x7fba98004d40 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab9800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4072340 con 0x7fbab40ce000 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab415b100 0x7fbab415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab415b100 0x7fbab415b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49180/0 (socket says 172.21.15.114:49180) 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 -- 172.21.15.114:0/1033319620 learned_addr learned my addr 172.21.15.114:0/1033319620 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab2a00640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98004d40 0x7fbab4147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 -- 172.21.15.114:0/1033319620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab40ce000 msgr2=0x7fbab40ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab40ce000 0x7fbab40ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 -- 172.21.15.114:0/1033319620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98004d40 msgr2=0x7fbab4147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98004d40 0x7fbab4147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.632+0000 7fbab3400640 1 -- 172.21.15.114:0/1033319620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab415be70 con 0x7fbab415b100 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.633+0000 7fbab3400640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab415b100 0x7fbab415b580 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009b50 tx=0x7fbaac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.633+0000 7fbaab400640 1 -- 172.21.15.114:0/1033319620 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac044070 con 0x7fbab415b100 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.633+0000 7fbaab400640 1 -- 172.21.15.114:0/1033319620 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac036640 con 0x7fbab415b100 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.633+0000 7fbaab400640 1 -- 172.21.15.114:0/1033319620 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac03b040 con 0x7fbab415b100 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.633+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab415c150 con 0x7fbab415b100 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.633+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab415c710 con 0x7fbab415b100 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.634+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab404f330 con 0x7fbab415b100 2024-03-06T07:35:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.635+0000 7fbaab400640 1 -- 172.21.15.114:0/1033319620 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaac047070 con 0x7fbab415b100 2024-03-06T07:35:35.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.636+0000 7fbaab400640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9006b4d0 0x7fba9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.636+0000 7fbaab400640 1 -- 172.21.15.114:0/1033319620 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fbaac0aaa40 con 0x7fbab415b100 2024-03-06T07:35:35.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.636+0000 7fbab2a00640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9006b4d0 0x7fba9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.637+0000 7fbab2a00640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9006b4d0 0x7fba9006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4001d20 tx=0x7fbaa4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.640+0000 7fbaab400640 1 -- 172.21.15.114:0/1033319620 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaac0794b0 con 0x7fbab415b100 2024-03-06T07:35:35.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.903+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7fbab406ce60 con 0x7fbab415b100 2024-03-06T07:35:35.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.987+0000 7fbaab400640 1 -- 172.21.15.114:0/1033319620 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v408) v1 ==== 142+0+0 (secure 0 0 0) 0x7fbaac07d550 con 0x7fbab415b100 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9006b4d0 msgr2=0x7fba9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9006b4d0 0x7fba9006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4001d20 tx=0x7fbaa4009040 comp rx=0 tx=0).stop 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab415b100 msgr2=0x7fbab415b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab415b100 0x7fbab415b580 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009b50 tx=0x7fbaac004730 comp rx=0 tx=0).stop 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 shutdown_connections 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba9006b4d0 0x7fba9006d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab415b100 0x7fbab415b580 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbab40ce000 0x7fbab40ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 --2- 172.21.15.114:0/1033319620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba98004d40 0x7fbab4147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 >> 172.21.15.114:0/1033319620 conn(0x7fbab40b1730 msgr2=0x7fbab4156310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 shutdown_connections 2024-03-06T07:35:35.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:35.990+0000 7fbab9800640 1 -- 172.21.15.114:0/1033319620 wait complete. 2024-03-06T07:35:36.011 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:35:36.011 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:35:36.011 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:35:36.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:36.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.214+0000 7f4e9d200640 1 Processor -- start 2024-03-06T07:35:36.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.214+0000 7f4e9d200640 1 -- start start 2024-03-06T07:35:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.216+0000 7f4e9d200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e98144c00 0x7f4e9806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.216+0000 7f4e9d200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9806ee70 con 0x7f4e9815f9d0 2024-03-06T07:35:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.216+0000 7f4e9d200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9806cef0 con 0x7f4e9815fe60 2024-03-06T07:35:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.216+0000 7f4e9d200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9806d1d0 con 0x7f4e98144c00 2024-03-06T07:35:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.216+0000 7f4e8d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4e9815f9d0 0x7f4e980ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58962/0 (socket says 172.21.15.114:58962) 2024-03-06T07:35:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.216+0000 7f4e8d400640 1 -- 172.21.15.114:0/2339775053 learned_addr learned my addr 172.21.15.114:0/2339775053 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.216+0000 7f4e95e00640 1 --2- 172.21.15.114:0/2339775053 >> v2:172.21.15.114:3300/0 conn(0x7f4e98144c00 0x7f4e9806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.217+0000 7f4e95400640 1 -- 172.21.15.114:0/2339775053 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991726454 0 0) 0x7f4e80001110 con 0x7f4e9815f9d0 2024-03-06T07:35:36.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.217+0000 7f4e95400640 1 -- 172.21.15.114:0/2339775053 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e7c003520 con 0x7f4e9815f9d0 2024-03-06T07:35:36.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.217+0000 7f4e95e00640 1 -- 172.21.15.114:0/2339775053 >> v1:172.21.15.177:6789/0 conn(0x7f4e9815fe60 legacy=0x7f4e980ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.217+0000 7f4e95e00640 1 -- 172.21.15.114:0/2339775053 >> v1:172.21.15.114:6789/0 conn(0x7f4e9815f9d0 legacy=0x7f4e980ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.217+0000 7f4e95e00640 1 -- 172.21.15.114:0/2339775053 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9806d4b0 con 0x7f4e98144c00 2024-03-06T07:35:36.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.218+0000 7f4e95e00640 1 --2- 172.21.15.114:0/2339775053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e98144c00 0x7f4e9806e930 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097a0 tx=0x7f4e8802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7730c28e67d4837 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.218+0000 7f4e95400640 1 -- 172.21.15.114:0/2339775053 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88034070 con 0x7f4e98144c00 2024-03-06T07:35:36.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.218+0000 7f4e95400640 1 -- 172.21.15.114:0/2339775053 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e88030610 con 0x7f4e98144c00 2024-03-06T07:35:36.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.218+0000 7f4e95400640 1 -- 172.21.15.114:0/2339775053 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880309e0 con 0x7f4e98144c00 2024-03-06T07:35:36.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.219+0000 7f4e9d200640 1 -- 172.21.15.114:0/2339775053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e98144c00 msgr2=0x7f4e9806e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.219+0000 7f4e9d200640 1 --2- 172.21.15.114:0/2339775053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e98144c00 0x7f4e9806e930 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097a0 tx=0x7f4e8802b960 comp rx=0 tx=0).stop 2024-03-06T07:35:36.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.219+0000 7f4e9d200640 1 -- 172.21.15.114:0/2339775053 shutdown_connections 2024-03-06T07:35:36.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.219+0000 7f4e9d200640 1 --2- 172.21.15.114:0/2339775053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e98144c00 0x7f4e9806e930 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.219+0000 7f4e9d200640 1 -- 172.21.15.114:0/2339775053 >> 172.21.15.114:0/2339775053 conn(0x7f4e980b1730 msgr2=0x7f4e980b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.219+0000 7f4e9d200640 1 -- 172.21.15.114:0/2339775053 shutdown_connections 2024-03-06T07:35:36.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.219+0000 7f4e9d200640 1 -- 172.21.15.114:0/2339775053 wait complete. 2024-03-06T07:35:36.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.221+0000 7f4e9d200640 1 Processor -- start 2024-03-06T07:35:36.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.221+0000 7f4e9d200640 1 -- start start 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e9d200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e98144c00 0x7f4e9813b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e9d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e9815f9d0 0x7f4e9814eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e9d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e9815fe60 0x7f4e9815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e9d200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98160730 con 0x7f4e9815fe60 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e9d200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98160270 con 0x7f4e9815f9d0 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e9d200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980ceec0 con 0x7f4e98144c00 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e96800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e9815fe60 0x7f4e9815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e96800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e9815fe60 0x7f4e9815eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49212/0 (socket says 172.21.15.114:49212) 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e96800640 1 -- 172.21.15.114:0/970418051 learned_addr learned my addr 172.21.15.114:0/970418051 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:36.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.222+0000 7f4e8d400640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e9815f9d0 0x7f4e9814eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e96800640 1 -- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e98144c00 msgr2=0x7f4e9813b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e95e00640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e98144c00 0x7f4e9813b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e96800640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e98144c00 0x7f4e9813b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e96800640 1 -- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e9815f9d0 msgr2=0x7f4e9814eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e96800640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e9815f9d0 0x7f4e9814eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e96800640 1 -- 172.21.15.114:0/970418051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9813d720 con 0x7f4e9815fe60 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e8d400640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e9815f9d0 0x7f4e9814eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e96800640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e9815fe60 0x7f4e9815eec0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000c7d0 tx=0x7f4e9000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e8ea00640 1 -- 172.21.15.114:0/970418051 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90003b70 con 0x7f4e9815fe60 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e8ea00640 1 -- 172.21.15.114:0/970418051 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e90003d30 con 0x7f4e9815fe60 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e8ea00640 1 -- 172.21.15.114:0/970418051 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000aa90 con 0x7f4e9815fe60 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9813da00 con 0x7f4e9815fe60 2024-03-06T07:35:36.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.223+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9813dfc0 con 0x7f4e9815fe60 2024-03-06T07:35:36.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.224+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e980cd9a0 con 0x7f4e9815fe60 2024-03-06T07:35:36.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.225+0000 7f4e8ea00640 1 -- 172.21.15.114:0/970418051 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e9000ac30 con 0x7f4e9815fe60 2024-03-06T07:35:36.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.226+0000 7f4e8ea00640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e7406b4d0 0x7f4e7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.226+0000 7f4e8ea00640 1 -- 172.21.15.114:0/970418051 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4e90089f80 con 0x7f4e9815fe60 2024-03-06T07:35:36.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.226+0000 7f4e95e00640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e7406b4d0 0x7f4e7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.227+0000 7f4e95e00640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e7406b4d0 0x7f4e7406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4e88036000 tx=0x7f4e88008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.231+0000 7f4e8ea00640 1 -- 172.21.15.114:0/970418051 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e980cd9a0 con 0x7f4e9815fe60 2024-03-06T07:35:36.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.410+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e980cd0f0 con 0x7f4e9815fe60 2024-03-06T07:35:36.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.412+0000 7f4e8ea00640 1 -- 172.21.15.114:0/970418051 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13342 (secure 0 0 0) 0x7f4e90058fb0 con 0x7f4e9815fe60 2024-03-06T07:35:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:36.415 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":117,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:35:34.959411+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6835/2704122879":"2024-03-07T07:35:31.940459+0000","172.21.15.114:6834/2704122879":"2024-03-07T07:35:31.940459+0000","172.21.15.177:6835/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.177:6833/2151340900":"2024-03-07T07:34:21.054096+0000","172.21.15.114:6837/860883596":"2024-03-07T07:34:21.018930+0000","172.21.15.114:6835/860883596":"2024-03-07T07:34:21.018930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e7406b4d0 msgr2=0x7f4e7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e7406b4d0 0x7f4e7406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4e88036000 tx=0x7f4e88008030 comp rx=0 tx=0).stop 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e9815fe60 msgr2=0x7f4e9815eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e9815fe60 0x7f4e9815eec0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000c7d0 tx=0x7f4e9000cc90 comp rx=0 tx=0).stop 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 shutdown_connections 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4e7406b4d0 0x7f4e7406d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e9815fe60 0x7f4e9815eec0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e9815f9d0 0x7f4e9814eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 --2- 172.21.15.114:0/970418051 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4e98144c00 0x7f4e9813b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 >> 172.21.15.114:0/970418051 conn(0x7f4e980b1730 msgr2=0x7f4e980713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 shutdown_connections 2024-03-06T07:35:36.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.415+0000 7f4e9d200640 1 -- 172.21.15.114:0/970418051 wait complete. 2024-03-06T07:35:36.435 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:36.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:36.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.639+0000 7f99ef800640 1 Processor -- start 2024-03-06T07:35:36.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.640+0000 7f99ef800640 1 -- start start 2024-03-06T07:35:36.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.641+0000 7f99ef800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 0x7f99e80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.641+0000 7f99ef800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e80719e0 con 0x7f99e80b3650 2024-03-06T07:35:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.641+0000 7f99ef800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e8071cc0 con 0x7f99e80cda90 2024-03-06T07:35:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.641+0000 7f99ef800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8072310 con 0x7f99e80b3df0 2024-03-06T07:35:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.641+0000 7f99e7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f99e80b3650 0x7f99e80b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58966/0 (socket says 172.21.15.114:58966) 2024-03-06T07:35:36.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.641+0000 7f99e7e00640 1 -- 172.21.15.114:0/3574149998 learned_addr learned my addr 172.21.15.114:0/3574149998 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99ece00640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 0x7f99e80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106900364 0 0) 0x7f99e80719e0 con 0x7f99e80b3650 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c8003520 con 0x7f99e80b3650 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 240154085 0 0) 0x7f99c8003520 con 0x7f99e80b3650 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 msgr2=0x7f99e80727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 0x7f99e80727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 >> v1:172.21.15.177:6789/0 conn(0x7f99e80cda90 legacy=0x7f99e806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8071fa0 con 0x7f99e80b3650 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f99d00019f0 con 0x7f99e80b3650 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 >> v1:172.21.15.114:6789/0 conn(0x7f99e80b3650 legacy=0x7f99e80b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c80039e0 0x7f99c8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99c8004800 0x7f99c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 0x7f99c8005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8072310 con 0x7f99c8004800 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c800ff60 con 0x7f99c80039e0 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8010160 con 0x7f99e80b3df0 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.642+0000 7f99e7e00640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99c8004800 0x7f99c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99ece00640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c80039e0 0x7f99c8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99ed800640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 0x7f99c8005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99e7e00640 1 -- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 msgr2=0x7f99c8005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99e7e00640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e80b3df0 0x7f99c8005250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99e7e00640 1 -- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c80039e0 msgr2=0x7f99c8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99e7e00640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c80039e0 0x7f99c8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99e7e00640 1 -- 172.21.15.114:0/3574149998 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8003520 con 0x7f99c8004800 2024-03-06T07:35:36.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.643+0000 7f99ece00640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c80039e0 0x7f99c8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:36.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.644+0000 7f99e7e00640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99c8004800 0x7f99c8004c80 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f99d000a7b0 tx=0x7f99c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a35a388466c9e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.644+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000a3d0 con 0x7f99c8004800 2024-03-06T07:35:36.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.644+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d0034630 con 0x7f99c8004800 2024-03-06T07:35:36.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.644+0000 7f99e7400640 1 -- 172.21.15.114:0/3574149998 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0034a60 con 0x7f99c8004800 2024-03-06T07:35:36.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.644+0000 7f99ef800640 1 -- 172.21.15.114:0/3574149998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99c8004800 msgr2=0x7f99c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.644+0000 7f99ef800640 1 --2- 172.21.15.114:0/3574149998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99c8004800 0x7f99c8004c80 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f99d000a7b0 tx=0x7f99c8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.644+0000 7f99ed800640 1 -- 172.21.15.114:0/3574149998 reap_dead start 2024-03-06T07:35:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.645+0000 7f99ef800640 1 -- 172.21.15.114:0/3574149998 shutdown_connections 2024-03-06T07:35:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.645+0000 7f99ef800640 1 -- 172.21.15.114:0/3574149998 >> 172.21.15.114:0/3574149998 conn(0x7f99e80b1940 msgr2=0x7f99e80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.645+0000 7f99ef800640 1 -- 172.21.15.114:0/3574149998 shutdown_connections 2024-03-06T07:35:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.645+0000 7f99ef800640 1 -- 172.21.15.114:0/3574149998 wait complete. 2024-03-06T07:35:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.646+0000 7f99ef800640 1 Processor -- start 2024-03-06T07:35:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.646+0000 7f99ef800640 1 -- start start 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99ef800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c8004800 0x7f99e809cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99ef800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99e8142cd0 0x7f99e8144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99ef800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e8143150 0x7f99e8142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99ef800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e815b3c0 con 0x7f99e8142cd0 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99ef800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e815b240 con 0x7f99c8004800 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99ef800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80cf510 con 0x7f99e8143150 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99e7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99e8142cd0 0x7f99e8144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99e7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99e8142cd0 0x7f99e8144d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49228/0 (socket says 172.21.15.114:49228) 2024-03-06T07:35:36.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.647+0000 7f99e7e00640 1 -- 172.21.15.114:0/3006694436 learned_addr learned my addr 172.21.15.114:0/3006694436 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99ece00640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c8004800 0x7f99e809cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e7e00640 1 -- 172.21.15.114:0/3006694436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e8143150 msgr2=0x7f99e8142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e7e00640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e8143150 0x7f99e8142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e7e00640 1 -- 172.21.15.114:0/3006694436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c8004800 msgr2=0x7f99e809cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e7e00640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c8004800 0x7f99e809cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e7e00640 1 -- 172.21.15.114:0/3006694436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8145820 con 0x7f99e8142cd0 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e7e00640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99e8142cd0 0x7f99e8144d20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f99d000a7b0 tx=0x7f99d0001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e4c00640 1 -- 172.21.15.114:0/3006694436 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0034430 con 0x7f99e8142cd0 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e4c00640 1 -- 172.21.15.114:0/3006694436 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d0004d70 con 0x7f99e8142cd0 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99e4c00640 1 -- 172.21.15.114:0/3006694436 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0034a00 con 0x7f99e8142cd0 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.648+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e8145b00 con 0x7f99e8142cd0 2024-03-06T07:35:36.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.649+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e8152340 con 0x7f99e8142cd0 2024-03-06T07:35:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.650+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e8151e50 con 0x7f99e8142cd0 2024-03-06T07:35:36.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.650+0000 7f99e4c00640 1 -- 172.21.15.114:0/3006694436 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99d0034ba0 con 0x7f99e8142cd0 2024-03-06T07:35:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.651+0000 7f99e4c00640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99c406b500 0x7f99c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.651+0000 7f99e4c00640 1 -- 172.21.15.114:0/3006694436 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f99d00acaa0 con 0x7f99e8142cd0 2024-03-06T07:35:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.652+0000 7f99ece00640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99c406b500 0x7f99c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.652+0000 7f99ece00640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99c406b500 0x7f99c406d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f99d8006fd0 tx=0x7f99d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.656+0000 7f99e4c00640 1 -- 172.21.15.114:0/3006694436 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99d007bad0 con 0x7f99e8142cd0 2024-03-06T07:35:36.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.834+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f99e804f330 con 0x7f99e8142cd0 2024-03-06T07:35:36.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.835+0000 7f99e4c00640 1 -- 172.21.15.114:0/3006694436 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7f99d007b3d0 con 0x7f99e8142cd0 2024-03-06T07:35:36.837 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99c406b500 msgr2=0x7f99c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99c406b500 0x7f99c406d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f99d8006fd0 tx=0x7f99d8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99e8142cd0 msgr2=0x7f99e8144d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99e8142cd0 0x7f99e8144d20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f99d000a7b0 tx=0x7f99d0001cf0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 shutdown_connections 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f99e8143150 0x7f99e8142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f99c406b500 0x7f99c406d9c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99e8142cd0 0x7f99e8144d20 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 --2- 172.21.15.114:0/3006694436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99c8004800 0x7f99e809cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 >> 172.21.15.114:0/3006694436 conn(0x7f99e80b1940 msgr2=0x7f99e806c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 shutdown_connections 2024-03-06T07:35:36.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:36.837+0000 7f99ef800640 1 -- 172.21.15.114:0/3006694436 wait complete. 2024-03-06T07:35:36.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:35:37.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.061+0000 7fdaea600640 1 Processor -- start 2024-03-06T07:35:37.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.061+0000 7fdaea600640 1 -- start start 2024-03-06T07:35:37.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.063+0000 7fdaea600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdae415f9d0 0x7fdae415fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.063+0000 7fdaea600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae406cef0 con 0x7fdae4144c00 2024-03-06T07:35:37.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.063+0000 7fdaea600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae406d1d0 con 0x7fdae4070b10 2024-03-06T07:35:37.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.063+0000 7fdaea600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae4160370 con 0x7fdae415f9d0 2024-03-06T07:35:37.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.063+0000 7fdae3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdae4144c00 0x7fdae406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:58980/0 (socket says 172.21.15.114:58980) 2024-03-06T07:35:37.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.063+0000 7fdae3400640 1 -- 172.21.15.114:0/3804976202 learned_addr learned my addr 172.21.15.114:0/3804976202 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdadaa00640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdae415f9d0 0x7fdae415fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626330884 0 0) 0x7fdae406cef0 con 0x7fdae4144c00 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac8003520 con 0x7fdae4144c00 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555571391 0 0) 0x7fdac8003520 con 0x7fdae4144c00 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdae415f9d0 msgr2=0x7fdae415fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdae415f9d0 0x7fdae415fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 >> v1:172.21.15.177:6789/0 conn(0x7fdae4070b10 legacy=0x7fdae4070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae406d4b0 con 0x7fdae4144c00 2024-03-06T07:35:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fdad4001670 con 0x7fdae4144c00 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 >> v1:172.21.15.114:6789/0 conn(0x7fdae4144c00 legacy=0x7fdae406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdac80039e0 0x7fdac8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdac8004800 0x7fdac8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdae415f9d0 0x7fdac80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac800ba70 con 0x7fdac80039e0 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac800bc20 con 0x7fdac8004800 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.064+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac800be20 con 0x7fdae415f9d0 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdae3400640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdac80039e0 0x7fdac8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdadaa00640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdac8004800 0x7fdac8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdae3400640 1 -- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdae415f9d0 msgr2=0x7fdac80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdae3400640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdae415f9d0 0x7fdac80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdae3400640 1 -- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdac8004800 msgr2=0x7fdac8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdae3400640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdac8004800 0x7fdac8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdae3400640 1 -- 172.21.15.114:0/3804976202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac8003520 con 0x7fdac80039e0 2024-03-06T07:35:37.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.065+0000 7fdadaa00640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdac8004800 0x7fdac8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.066+0000 7fdae3400640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdac80039e0 0x7fdac8003e00 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fdad4001eb0 tx=0x7fdac8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a5100cbaf06142 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.066+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4007d30 con 0x7fdac80039e0 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.066+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad4030070 con 0x7fdac80039e0 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.066+0000 7fdae2a00640 1 -- 172.21.15.114:0/3804976202 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4005d30 con 0x7fdac80039e0 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.067+0000 7fdaea600640 1 -- 172.21.15.114:0/3804976202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdac80039e0 msgr2=0x7fdac8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.067+0000 7fdaea600640 1 --2- 172.21.15.114:0/3804976202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdac80039e0 0x7fdac8003e00 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fdad4001eb0 tx=0x7fdac8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.067+0000 7fdae3e00640 1 -- 172.21.15.114:0/3804976202 reap_dead start 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.067+0000 7fdaea600640 1 -- 172.21.15.114:0/3804976202 shutdown_connections 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.067+0000 7fdaea600640 1 -- 172.21.15.114:0/3804976202 >> 172.21.15.114:0/3804976202 conn(0x7fdae40b1730 msgr2=0x7fdae40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.067+0000 7fdaea600640 1 -- 172.21.15.114:0/3804976202 shutdown_connections 2024-03-06T07:35:37.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.067+0000 7fdaea600640 1 -- 172.21.15.114:0/3804976202 wait complete. 2024-03-06T07:35:37.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.069+0000 7fdaea600640 1 Processor -- start 2024-03-06T07:35:37.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.069+0000 7fdaea600640 1 -- start start 2024-03-06T07:35:37.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdaea600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdac80039e0 0x7fdae415e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdaea600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae4139470 0x7fdae415c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdaea600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdae41f5050 0x7fdae41f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdaea600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae406cb50 con 0x7fdae4139470 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdaea600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae4160730 con 0x7fdae41f5050 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdaea600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae406ccd0 con 0x7fdac80039e0 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae4139470 0x7fdae415c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae4139470 0x7fdae415c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49244/0 (socket says 172.21.15.114:49244) 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 -- 172.21.15.114:0/4272160952 learned_addr learned my addr 172.21.15.114:0/4272160952 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdae3400640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdac80039e0 0x7fdae415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 -- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdac80039e0 msgr2=0x7fdae415e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdae3e00640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdae41f5050 0x7fdae41f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdac80039e0 0x7fdae415e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 -- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdae41f5050 msgr2=0x7fdae41f54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdae41f5050 0x7fdae41f54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.070+0000 7fdadaa00640 1 -- 172.21.15.114:0/4272160952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae415f000 con 0x7fdae4139470 2024-03-06T07:35:37.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.071+0000 7fdadaa00640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae4139470 0x7fdae415c420 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0108f0 tx=0x7fdacc010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.071+0000 7fdadbe00640 1 -- 172.21.15.114:0/4272160952 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc00f040 con 0x7fdae4139470 2024-03-06T07:35:37.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.071+0000 7fdadbe00640 1 -- 172.21.15.114:0/4272160952 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdacc00eb60 con 0x7fdae4139470 2024-03-06T07:35:37.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.071+0000 7fdadbe00640 1 -- 172.21.15.114:0/4272160952 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc01c400 con 0x7fdae4139470 2024-03-06T07:35:37.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.071+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae415f2e0 con 0x7fdae4139470 2024-03-06T07:35:37.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.071+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae413ad80 con 0x7fdae4139470 2024-03-06T07:35:37.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.072+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae4143e60 con 0x7fdae4139470 2024-03-06T07:35:37.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.073+0000 7fdadbe00640 1 -- 172.21.15.114:0/4272160952 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdacc01c5a0 con 0x7fdae4139470 2024-03-06T07:35:37.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.074+0000 7fdadbe00640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdac006b4d0 0x7fdac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.074+0000 7fdadbe00640 1 -- 172.21.15.114:0/4272160952 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fdacc08a510 con 0x7fdae4139470 2024-03-06T07:35:37.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.074+0000 7fdae3400640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdac006b4d0 0x7fdac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.075+0000 7fdae3400640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdac006b4d0 0x7fdac006d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdad4034000 tx=0x7fdad4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.078+0000 7fdadbe00640 1 -- 172.21.15.114:0/4272160952 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdae4143e60 con 0x7fdae4139470 2024-03-06T07:35:37.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:37.344+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7fdae404f330 con 0x7fdae4139470 2024-03-06T07:35:38.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.000+0000 7fdadbe00640 1 -- 172.21.15.114:0/4272160952 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v410) v1 ==== 143+0+0 (secure 0 0 0) 0x7fdacc018030 con 0x7fdae4139470 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.003+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdac006b4d0 msgr2=0x7fdac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.003+0000 7fdaea600640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdac006b4d0 0x7fdac006d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdad4034000 tx=0x7fdad4009030 comp rx=0 tx=0).stop 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.003+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae4139470 msgr2=0x7fdae415c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.003+0000 7fdaea600640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae4139470 0x7fdae415c420 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0108f0 tx=0x7fdacc010db0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 shutdown_connections 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdae41f5050 0x7fdae41f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdac006b4d0 0x7fdac006d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae4139470 0x7fdae415c420 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 --2- 172.21.15.114:0/4272160952 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdac80039e0 0x7fdae415e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 >> 172.21.15.114:0/4272160952 conn(0x7fdae40b1730 msgr2=0x7fdae40af990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 shutdown_connections 2024-03-06T07:35:38.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.004+0000 7fdaea600640 1 -- 172.21.15.114:0/4272160952 wait complete. 2024-03-06T07:35:38.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:35:38.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.233+0000 7f0fde000640 1 Processor -- start 2024-03-06T07:35:38.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.233+0000 7f0fde000640 1 -- start start 2024-03-06T07:35:38.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.234+0000 7f0fde000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0fd806da40 0x7f0fd80ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.234+0000 7f0fde000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd815c1f0 con 0x7f0fd815be10 2024-03-06T07:35:38.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.234+0000 7f0fde000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd806cac0 con 0x7f0fd80aef20 2024-03-06T07:35:38.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.234+0000 7f0fde000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80cd200 con 0x7f0fd806da40 2024-03-06T07:35:38.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fdcc00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0fd815be10 0x7f0fd806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51778/0 (socket says 172.21.15.114:51778) 2024-03-06T07:35:38.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fdcc00640 1 -- 172.21.15.114:0/2237136779 learned_addr learned my addr 172.21.15.114:0/2237136779 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:38.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475179043 0 0) 0x7f0fd815c1f0 con 0x7f0fd815be10 2024-03-06T07:35:38.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc0003520 con 0x7f0fd815be10 2024-03-06T07:35:38.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7e00640 1 --2- 172.21.15.114:0/2237136779 >> v2:172.21.15.177:3300/0 conn(0x7f0fd806da40 0x7f0fd80ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1697923814 0 0) 0x7f0fc0003520 con 0x7f0fd815be10 2024-03-06T07:35:38.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 >> v2:172.21.15.177:3300/0 conn(0x7f0fd806da40 msgr2=0x7f0fd80ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 --2- 172.21.15.114:0/2237136779 >> v2:172.21.15.177:3300/0 conn(0x7f0fd806da40 0x7f0fd80ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 >> v1:172.21.15.177:6789/0 conn(0x7f0fd80aef20 legacy=0x7f0fd8071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd806cda0 con 0x7f0fd815be10 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f0fcc001670 con 0x7f0fd815be10 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 >> v1:172.21.15.114:6789/0 conn(0x7f0fd815be10 legacy=0x7f0fd806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc00039b0 0x7f0fc0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fc0004800 0x7f0fc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.235+0000 7f0fd7400640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fc00052e0 0x7f0fc000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80cd200 con 0x7f0fc00039b0 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc000c190 con 0x7f0fc00052e0 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc000c390 con 0x7f0fc0004800 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdcc00640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc00039b0 0x7f0fc0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdcc00640 1 -- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fc0004800 msgr2=0x7f0fc0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdcc00640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fc0004800 0x7f0fc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdcc00640 1 -- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fc00052e0 msgr2=0x7f0fc000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdd600640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fc00052e0 0x7f0fc000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdcc00640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fc00052e0 0x7f0fc000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdcc00640 1 -- 172.21.15.114:0/2237136779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc0003520 con 0x7f0fc00039b0 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdd600640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fc00052e0 0x7f0fc000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.236+0000 7f0fdd600640 1 -- 172.21.15.114:0/2237136779 reap_dead start 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.237+0000 7f0fdcc00640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc00039b0 0x7f0fc0003db0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00c210 tx=0x7f0fc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb76fb8df34b6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc007d30 con 0x7f0fc00039b0 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc030070 con 0x7f0fc00039b0 2024-03-06T07:35:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fd7400640 1 -- 172.21.15.114:0/2237136779 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc005d70 con 0x7f0fc00039b0 2024-03-06T07:35:38.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fde000640 1 -- 172.21.15.114:0/2237136779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc00039b0 msgr2=0x7f0fc0003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fde000640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc00039b0 0x7f0fc0003db0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00c210 tx=0x7f0fc0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:38.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fde000640 1 -- 172.21.15.114:0/2237136779 shutdown_connections 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fde000640 1 --2- 172.21.15.114:0/2237136779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc00039b0 0x7f0fc0003db0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fde000640 1 -- 172.21.15.114:0/2237136779 >> 172.21.15.114:0/2237136779 conn(0x7f0fd806afc0 msgr2=0x7f0fd806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fde000640 1 -- 172.21.15.114:0/2237136779 shutdown_connections 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.238+0000 7f0fde000640 1 -- 172.21.15.114:0/2237136779 wait complete. 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.240+0000 7f0fde000640 1 Processor -- start 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.240+0000 7f0fde000640 1 -- start start 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fde000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fc00039b0 0x7f0fd81463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fde000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fd80af400 0x7f0fd815d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fde000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fd815d820 0x7f0fd81f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fde000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80cda90 con 0x7f0fd815d820 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fde000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd806d080 con 0x7f0fd80af400 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fde000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80ccc20 con 0x7f0fc00039b0 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fdd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fd815d820 0x7f0fd81f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fdd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fd815d820 0x7f0fd81f41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48052/0 (socket says 172.21.15.114:48052) 2024-03-06T07:35:38.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.241+0000 7f0fdd600640 1 -- 172.21.15.114:0/2484685635 learned_addr learned my addr 172.21.15.114:0/2484685635 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:38.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fdd600640 1 -- 172.21.15.114:0/2484685635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fc00039b0 msgr2=0x7f0fd81463e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fdd600640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fc00039b0 0x7f0fd81463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fdd600640 1 -- 172.21.15.114:0/2484685635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fd80af400 msgr2=0x7f0fd815d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fdd600640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fd80af400 0x7f0fd815d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fdd600640 1 -- 172.21.15.114:0/2484685635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd81f4f40 con 0x7f0fd815d820 2024-03-06T07:35:38.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fdd600640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fd815d820 0x7f0fd81f41d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8072e00 tx=0x7f0fd000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fd4c00640 1 -- 172.21.15.114:0/2484685635 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0018070 con 0x7f0fd815d820 2024-03-06T07:35:38.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fd4c00640 1 -- 172.21.15.114:0/2484685635 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd0003d80 con 0x7f0fd815d820 2024-03-06T07:35:38.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fd4c00640 1 -- 172.21.15.114:0/2484685635 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00052b0 con 0x7f0fd815d820 2024-03-06T07:35:38.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd80b3060 con 0x7f0fd815d820 2024-03-06T07:35:38.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.242+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd80b3620 con 0x7f0fd815d820 2024-03-06T07:35:38.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.243+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd804f330 con 0x7f0fd815d820 2024-03-06T07:35:38.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.244+0000 7f0fd4c00640 1 -- 172.21.15.114:0/2484685635 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fd0005450 con 0x7f0fd815d820 2024-03-06T07:35:38.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.245+0000 7f0fd4c00640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0fb806b500 0x7f0fb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.245+0000 7f0fd4c00640 1 -- 172.21.15.114:0/2484685635 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f0fd00893b0 con 0x7f0fd815d820 2024-03-06T07:35:38.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.245+0000 7f0fdcc00640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0fb806b500 0x7f0fb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.246+0000 7f0fdcc00640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0fb806b500 0x7f0fb806d9c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc034000 tx=0x7f0fcc007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.249+0000 7f0fd4c00640 1 -- 172.21.15.114:0/2484685635 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fd00583e0 con 0x7f0fd815d820 2024-03-06T07:35:38.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.432+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0fd815f6f0 con 0x7f0fd815d820 2024-03-06T07:35:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.439+0000 7f0fd4c00640 1 -- 172.21.15.114:0/2484685635 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0fd0057ce0 con 0x7f0fd815d820 2024-03-06T07:35:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.441+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0fb806b500 msgr2=0x7f0fb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.441+0000 7f0fde000640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0fb806b500 0x7f0fb806d9c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc034000 tx=0x7f0fcc007000 comp rx=0 tx=0).stop 2024-03-06T07:35:38.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.441+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fd815d820 msgr2=0x7f0fd81f41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.441+0000 7f0fde000640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fd815d820 0x7f0fd81f41d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8072e00 tx=0x7f0fd000d9d0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 shutdown_connections 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0fb806b500 0x7f0fb806d9c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fd815d820 0x7f0fd81f41d0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fd80af400 0x7f0fd815d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 --2- 172.21.15.114:0/2484685635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0fc00039b0 0x7f0fd81463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 >> 172.21.15.114:0/2484685635 conn(0x7f0fd806afc0 msgr2=0x7f0fd80ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 shutdown_connections 2024-03-06T07:35:38.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:38.442+0000 7f0fde000640 1 -- 172.21.15.114:0/2484685635 wait complete. 2024-03-06T07:35:38.462 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:38.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:38.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:38.681 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:38.681 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:35:38.681 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:38.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:38.738 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:35:38.738 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:38.794 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:38.794 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:35:38.795 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:38.849 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053417,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:38.849 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-06T07:35:38.849 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:38.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053387,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249148k,nr_inodes=812287,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498300k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:38.903 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:35:38.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:39.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.121+0000 7f673fe00640 1 Processor -- start 2024-03-06T07:35:39.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.122+0000 7f673fe00640 1 -- start start 2024-03-06T07:35:39.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.123+0000 7f673fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67400cc2a0 0x7f67400cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.123+0000 7f673fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67400b2b30 con 0x7f674015a940 2024-03-06T07:35:39.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.123+0000 7f673fe00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67400b2e10 con 0x7f67400cf370 2024-03-06T07:35:39.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.123+0000 7f673fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67400b28e0 con 0x7f67400cc2a0 2024-03-06T07:35:39.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673ea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f674015a940 0x7f67400b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51780/0 (socket says 172.21.15.114:51780) 2024-03-06T07:35:39.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673ea00640 1 -- 172.21.15.114:0/601717456 learned_addr learned my addr 172.21.15.114:0/601717456 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:39.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307257467 0 0) 0x7f67400b2b30 con 0x7f674015a940 2024-03-06T07:35:39.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6724003520 con 0x7f674015a940 2024-03-06T07:35:39.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673f400640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67400cc2a0 0x7f67400cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2080496847 0 0) 0x7f6724003520 con 0x7f674015a940 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67400cc2a0 msgr2=0x7f67400cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67400cc2a0 0x7f67400cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 >> v1:172.21.15.177:6789/0 conn(0x7f67400cf370 legacy=0x7f674006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674006ca50 con 0x7f674015a940 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6730001670 con 0x7f674015a940 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.124+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 >> v1:172.21.15.114:6789/0 conn(0x7f674015a940 legacy=0x7f67400b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673e000640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67240039e0 0x7f6724003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673e000640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6724004800 0x7f6724004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673e000640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67400cc2a0 0x7f6724005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67400b28e0 con 0x7f6724004800 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672400baf0 con 0x7f67240039e0 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672400bcf0 con 0x7f67400cc2a0 2024-03-06T07:35:39.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f6737e00640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6724004800 0x7f6724004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673ea00640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67240039e0 0x7f6724003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f6737e00640 1 -- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67400cc2a0 msgr2=0x7f6724005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f6737e00640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f67400cc2a0 0x7f6724005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f6737e00640 1 -- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67240039e0 msgr2=0x7f6724003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f6737e00640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67240039e0 0x7f6724003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f6737e00640 1 -- 172.21.15.114:0/601717456 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6724003520 con 0x7f6724004800 2024-03-06T07:35:39.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.125+0000 7f673ea00640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67240039e0 0x7f6724003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:39.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.126+0000 7f6737e00640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6724004800 0x7f6724004c80 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6728001330 tx=0x7f6724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc96a390d941edac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.126+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728001370 con 0x7f6724004800 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672802f040 con 0x7f6724004800 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673e000640 1 -- 172.21.15.114:0/601717456 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67280035f0 con 0x7f6724004800 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673fe00640 1 -- 172.21.15.114:0/601717456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6724004800 msgr2=0x7f6724004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673fe00640 1 --2- 172.21.15.114:0/601717456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6724004800 0x7f6724004c80 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f6728001330 tx=0x7f6724005830 comp rx=0 tx=0).stop 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673f400640 1 -- 172.21.15.114:0/601717456 reap_dead start 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673fe00640 1 -- 172.21.15.114:0/601717456 shutdown_connections 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673fe00640 1 -- 172.21.15.114:0/601717456 >> 172.21.15.114:0/601717456 conn(0x7f674006afc0 msgr2=0x7f674006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673fe00640 1 -- 172.21.15.114:0/601717456 shutdown_connections 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.127+0000 7f673fe00640 1 -- 172.21.15.114:0/601717456 wait complete. 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.129+0000 7f673fe00640 1 Processor -- start 2024-03-06T07:35:39.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.129+0000 7f673fe00640 1 -- start start 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6724004800 0x7f6740151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6740153020 0x7f6740153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6740159430 0x7f67401598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674015ad20 con 0x7f6740159430 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67400b30f0 con 0x7f6740153020 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674015aea0 con 0x7f6724004800 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6740159430 0x7f67401598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6740159430 0x7f67401598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48074/0 (socket says 172.21.15.114:48074) 2024-03-06T07:35:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.130+0000 7f673f400640 1 -- 172.21.15.114:0/1358887329 learned_addr learned my addr 172.21.15.114:0/1358887329 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673ea00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6724004800 0x7f6740151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673f400640 1 -- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6724004800 msgr2=0x7f6740151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f6737e00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6740153020 0x7f6740153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673f400640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6724004800 0x7f6740151660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673f400640 1 -- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6740153020 msgr2=0x7f6740153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673f400640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6740153020 0x7f6740153480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673f400640 1 -- 172.21.15.114:0/1358887329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674015a160 con 0x7f6740159430 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673f400640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6740159430 0x7f67401598b0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f6740070960 tx=0x7f673800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f6736a00640 1 -- 172.21.15.114:0/1358887329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673800ebc0 con 0x7f6740159430 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f6736a00640 1 -- 172.21.15.114:0/1358887329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673800ed80 con 0x7f6740159430 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f6736a00640 1 -- 172.21.15.114:0/1358887329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738003bd0 con 0x7f6740159430 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6740151ba0 con 0x7f6740159430 2024-03-06T07:35:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.131+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6740071a60 con 0x7f6740159430 2024-03-06T07:35:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.133+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674004f330 con 0x7f6740159430 2024-03-06T07:35:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.134+0000 7f6736a00640 1 -- 172.21.15.114:0/1358887329 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f673801b020 con 0x7f6740159430 2024-03-06T07:35:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.135+0000 7f6736a00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f671c06b4d0 0x7f671c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.135+0000 7f6736a00640 1 -- 172.21.15.114:0/1358887329 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f6738089ef0 con 0x7f6740159430 2024-03-06T07:35:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.135+0000 7f673ea00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f671c06b4d0 0x7f671c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.136+0000 7f673ea00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f671c06b4d0 0x7f671c06d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6730007950 tx=0x7f6730009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.139+0000 7f6736a00640 1 -- 172.21.15.114:0/1358887329 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6738058f20 con 0x7f6740159430 2024-03-06T07:35:39.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.374+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67400d05c0 con 0x7f6740159430 2024-03-06T07:35:39.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.375+0000 7f6736a00640 1 -- 172.21.15.114:0/1358887329 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f6738058820 con 0x7f6740159430 2024-03-06T07:35:39.377 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:39.378 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5627,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2490788512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2490788512},{"type":"v1","addr":"172.21.15.177:6835","nonce":2490788512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":6154,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/190819130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":190819130},{"type":"v1","addr":"172.21.15.177:6833","nonce":190819130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7923,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/650618622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":650618622},{"type":"v1","addr":"172.21.15.114:6837","nonce":650618622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":8250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1452189267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1452189267},{"type":"v1","addr":"172.21.15.114:6835","nonce":1452189267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":89}],"filesystems":[]} 2024-03-06T07:35:39.378 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 89 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f671c06b4d0 msgr2=0x7f671c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f671c06b4d0 0x7f671c06d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6730007950 tx=0x7f6730009040 comp rx=0 tx=0).stop 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6740159430 msgr2=0x7f67401598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6740159430 0x7f67401598b0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f6740070960 tx=0x7f673800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 shutdown_connections 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f671c06b4d0 0x7f671c06d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6740159430 0x7f67401598b0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6740153020 0x7f6740153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 --2- 172.21.15.114:0/1358887329 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6724004800 0x7f6740151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 >> 172.21.15.114:0/1358887329 conn(0x7f674006afc0 msgr2=0x7f67400cd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:39.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 shutdown_connections 2024-03-06T07:35:39.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.378+0000 7f673fe00640 1 -- 172.21.15.114:0/1358887329 wait complete. 2024-03-06T07:35:39.399 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:35:39.400 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:35:39.400 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:35:39.400 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:39.400 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:35:39.400 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:35:39.400 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:35:39.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:39.401 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:35:39.401 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:35:39.401 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:35:39.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:39.401 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:35:39.402 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:35:39.402 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:35:39.402 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:39.417 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:35:39.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:35:39.423 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:35:39.423 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:35:39.424 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:35:39.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:35:39.431 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:35:39.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:35:39.471 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:35:39.471 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:39.477 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:35:39.477 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:39.478 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:35:39.478 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:39.484 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:35:39.485 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:39.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:35:39.622 INFO:tasks.ceph.mds.c.smithi114.stdout:starting mds.c at 2024-03-06T07:35:39.622 INFO:tasks.ceph.mds.a.smithi114.stdout:starting mds.a at 2024-03-06T07:35:39.624 INFO:tasks.ceph.mds.c.smithi114.stderr:2024-03-06T07:35:39.623+0000 7f5558000640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:35:39.639 INFO:tasks.ceph.mds.d.smithi177.stdout:starting mds.d at 2024-03-06T07:35:39.646 INFO:tasks.ceph.mds.b.smithi177.stdout:starting mds.b at 2024-03-06T07:35:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.656+0000 7fbbcbc00640 1 Processor -- start 2024-03-06T07:35:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.656+0000 7fbbcbc00640 1 -- start start 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbcbc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd5b0 0x7fbbc4145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbcbc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbbc40cd990 0x7fbbc415c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbcbc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc4166ec0 con 0x7fbbc415caa0 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbcbc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc406cbe0 con 0x7fbbc40cd990 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbcbc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc406cec0 con 0x7fbbc40cd5b0 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbc9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd5b0 0x7fbbc4145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbc9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd5b0 0x7fbbc4145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:33886/0 (socket says 172.21.15.114:33886) 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbc3e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbbc40cd990 0x7fbbc415c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbc9200640 1 -- 172.21.15.114:0/1854802009 learned_addr learned my addr 172.21.15.114:0/1854802009 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:39.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbc3e00640 1 -- 172.21.15.114:0/1854802009 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd5b0 msgr2=0x7fbbc4145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.658+0000 7fbbc3e00640 1 --2- 172.21.15.114:0/1854802009 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd5b0 0x7fbbc4145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.659+0000 7fbbc3e00640 1 -- 172.21.15.114:0/1854802009 >> v1:172.21.15.177:6789/0 conn(0x7fbbc415caa0 legacy=0x7fbbc4162fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.659+0000 7fbbc3400640 1 -- 172.21.15.114:0/1854802009 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958864881 0 0) 0x7fbbc4166ec0 con 0x7fbbc415caa0 2024-03-06T07:35:39.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.659+0000 7fbbc3e00640 1 -- 172.21.15.114:0/1854802009 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc406d1a0 con 0x7fbbc40cd990 2024-03-06T07:35:39.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.660+0000 7fbbc3e00640 1 --2- 172.21.15.114:0/1854802009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd990 0x7fbbc415c560 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0097a0 tx=0x7fbbac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82eef89f224f8ccf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.660+0000 7fbbc3400640 1 -- 172.21.15.114:0/1854802009 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac035020 con 0x7fbbc40cd990 2024-03-06T07:35:39.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.660+0000 7fbbc3400640 1 -- 172.21.15.114:0/1854802009 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac031830 con 0x7fbbc40cd990 2024-03-06T07:35:39.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.660+0000 7fbbc3400640 1 -- 172.21.15.114:0/1854802009 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac031c00 con 0x7fbbc40cd990 2024-03-06T07:35:39.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.660+0000 7fbbcbc00640 1 -- 172.21.15.114:0/1854802009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd990 msgr2=0x7fbbc415c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.660+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/1854802009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd990 0x7fbbc415c560 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0097a0 tx=0x7fbbac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.661+0000 7fbbcbc00640 1 -- 172.21.15.114:0/1854802009 shutdown_connections 2024-03-06T07:35:39.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.661+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/1854802009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd990 0x7fbbc415c560 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.661+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/1854802009 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd5b0 0x7fbbc4145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.661+0000 7fbbcbc00640 1 -- 172.21.15.114:0/1854802009 >> 172.21.15.114:0/1854802009 conn(0x7fbbc40cc4f0 msgr2=0x7fbbc40cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:39.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.661+0000 7fbbcbc00640 1 -- 172.21.15.114:0/1854802009 shutdown_connections 2024-03-06T07:35:39.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.661+0000 7fbbcbc00640 1 -- 172.21.15.114:0/1854802009 wait complete. 2024-03-06T07:35:39.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.662+0000 7fbbcbc00640 1 Processor -- start 2024-03-06T07:35:39.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.662+0000 7fbbcbc00640 1 -- start start 2024-03-06T07:35:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.663+0000 7fbbcbc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd5b0 0x7fbbc415e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.663+0000 7fbbcbc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd990 0x7fbbc4071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.663+0000 7fbbcbc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbc415caa0 0x7fbbc40afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.663+0000 7fbbcbc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40cde10 con 0x7fbbc415caa0 2024-03-06T07:35:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.663+0000 7fbbcbc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc415cef0 con 0x7fbbc40cd5b0 2024-03-06T07:35:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.663+0000 7fbbcbc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc41636c0 con 0x7fbbc40cd990 2024-03-06T07:35:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.663+0000 7fbbc9c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbc415caa0 0x7fbbc40afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbc415caa0 0x7fbbc40afcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48122/0 (socket says 172.21.15.114:48122) 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 -- 172.21.15.114:0/2281458070 learned_addr learned my addr 172.21.15.114:0/2281458070 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9200640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd5b0 0x7fbbc415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc3e00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd990 0x7fbbc4071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 -- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd990 msgr2=0x7fbbc4071d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd990 0x7fbbc4071d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 -- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd5b0 msgr2=0x7fbbc415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd5b0 0x7fbbc415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 -- 172.21.15.114:0/2281458070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc415ecc0 con 0x7fbbc415caa0 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc3e00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd990 0x7fbbc4071d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9200640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd5b0 0x7fbbc415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc9c00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbc415caa0 0x7fbbc40afcf0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00c3e0 tx=0x7fbbbc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc0c00640 1 -- 172.21.15.114:0/2281458070 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc017070 con 0x7fbbc415caa0 2024-03-06T07:35:39.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc0c00640 1 -- 172.21.15.114:0/2281458070 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc0035f0 con 0x7fbbc415caa0 2024-03-06T07:35:39.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc415d110 con 0x7fbbc415caa0 2024-03-06T07:35:39.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.664+0000 7fbbc0c00640 1 -- 172.21.15.114:0/2281458070 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc003a20 con 0x7fbbc415caa0 2024-03-06T07:35:39.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.665+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc415d6d0 con 0x7fbbc415caa0 2024-03-06T07:35:39.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.665+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc404f330 con 0x7fbbc415caa0 2024-03-06T07:35:39.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.667+0000 7fbbc0c00640 1 -- 172.21.15.114:0/2281458070 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbbc01a070 con 0x7fbbc415caa0 2024-03-06T07:35:39.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.667+0000 7fbbc0c00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbba006b4d0 0x7fbba006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.667+0000 7fbbc0c00640 1 -- 172.21.15.114:0/2281458070 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fbbbc089420 con 0x7fbbc415caa0 2024-03-06T07:35:39.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.668+0000 7fbbc9200640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbba006b4d0 0x7fbba006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.668+0000 7fbbc9200640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbba006b4d0 0x7fbba006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009d20 tx=0x7fbbb400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.672+0000 7fbbc0c00640 1 -- 172.21.15.114:0/2281458070 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbbc058450 con 0x7fbbc415caa0 2024-03-06T07:35:39.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:39.873+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbbc40c4b80 con 0x7fbbc415caa0 2024-03-06T07:35:40.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.027+0000 7fbbc0c00640 1 -- 172.21.15.114:0/2281458070 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v119) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbbbc057d50 con 0x7fbbc415caa0 2024-03-06T07:35:40.029 INFO:teuthology.orchestra.run.smithi114.stderr: removed all blocklist entries 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbba006b4d0 msgr2=0x7fbba006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbba006b4d0 0x7fbba006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009d20 tx=0x7fbbb400b040 comp rx=0 tx=0).stop 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbc415caa0 msgr2=0x7fbbc40afcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbc415caa0 0x7fbbc40afcf0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00c3e0 tx=0x7fbbbc00c8a0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 shutdown_connections 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbba006b4d0 0x7fbba006d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbc415caa0 0x7fbbc40afcf0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbbc40cd990 0x7fbbc4071d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 --2- 172.21.15.114:0/2281458070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbc40cd5b0 0x7fbbc415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 >> 172.21.15.114:0/2281458070 conn(0x7fbbc40cc4f0 msgr2=0x7fbbc4162b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 shutdown_connections 2024-03-06T07:35:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.031+0000 7fbbcbc00640 1 -- 172.21.15.114:0/2281458070 wait complete. 2024-03-06T07:35:40.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:35:40.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.246+0000 7f3d05800640 1 Processor -- start 2024-03-06T07:35:40.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.246+0000 7f3d05800640 1 -- start start 2024-03-06T07:35:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.247+0000 7f3d05800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d00144c00 0x7f3d0006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3d05800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d0015fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3d05800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0006ee70 con 0x7f3d000ce0a0 2024-03-06T07:35:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3d05800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d00160370 con 0x7f3d00144c00 2024-03-06T07:35:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3d05800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006cef0 con 0x7f3d0015f9d0 2024-03-06T07:35:40.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3cfea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d00144c00 0x7f3d0006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3cfea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d00144c00 0x7f3d0006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48124/0 (socket says 172.21.15.114:48124) 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3cfea00640 1 -- 172.21.15.114:0/2592883396 learned_addr learned my addr 172.21.15.114:0/2592883396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3cfea00640 1 -- 172.21.15.114:0/2592883396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 msgr2=0x7f3d0015fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.248+0000 7f3cfe000640 1 --2- 172.21.15.114:0/2592883396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d0015fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.249+0000 7f3cfea00640 1 --2- 172.21.15.114:0/2592883396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d0015fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.249+0000 7f3cfea00640 1 -- 172.21.15.114:0/2592883396 >> v1:172.21.15.177:6789/0 conn(0x7f3d000ce0a0 legacy=0x7f3d000ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.249+0000 7f3cfea00640 1 -- 172.21.15.114:0/2592883396 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0006d1d0 con 0x7f3d00144c00 2024-03-06T07:35:40.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.249+0000 7f3cfe000640 1 --2- 172.21.15.114:0/2592883396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d0015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.250+0000 7f3cfea00640 1 --2- 172.21.15.114:0/2592883396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d00144c00 0x7f3d0006e930 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009870 tx=0x7f3cf002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa892bf29d6423c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3cfd600640 1 -- 172.21.15.114:0/2592883396 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0035020 con 0x7f3d00144c00 2024-03-06T07:35:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3cfd600640 1 -- 172.21.15.114:0/2592883396 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf0031640 con 0x7f3d00144c00 2024-03-06T07:35:40.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3cfd600640 1 -- 172.21.15.114:0/2592883396 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf002d040 con 0x7f3d00144c00 2024-03-06T07:35:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3d05800640 1 -- 172.21.15.114:0/2592883396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d00144c00 msgr2=0x7f3d0006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3d05800640 1 --2- 172.21.15.114:0/2592883396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d00144c00 0x7f3d0006e930 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009870 tx=0x7f3cf002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3d05800640 1 -- 172.21.15.114:0/2592883396 shutdown_connections 2024-03-06T07:35:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3d05800640 1 --2- 172.21.15.114:0/2592883396 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d0015fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3d05800640 1 --2- 172.21.15.114:0/2592883396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d00144c00 0x7f3d0006e930 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3d05800640 1 -- 172.21.15.114:0/2592883396 >> 172.21.15.114:0/2592883396 conn(0x7f3d000b1730 msgr2=0x7f3d000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.251+0000 7f3d05800640 1 -- 172.21.15.114:0/2592883396 shutdown_connections 2024-03-06T07:35:40.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.252+0000 7f3d05800640 1 -- 172.21.15.114:0/2592883396 wait complete. 2024-03-06T07:35:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.253+0000 7f3d05800640 1 Processor -- start 2024-03-06T07:35:40.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.253+0000 7f3d05800640 1 -- start start 2024-03-06T07:35:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.254+0000 7f3d05800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d000ce0a0 0x7f3d001e9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.254+0000 7f3d05800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d00144c00 0x7f3d0015e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.254+0000 7f3d05800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d000b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.254+0000 7f3d05800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00160730 con 0x7f3d000ce0a0 2024-03-06T07:35:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.254+0000 7f3d05800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006d790 con 0x7f3d00144c00 2024-03-06T07:35:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.254+0000 7f3d05800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006cb50 con 0x7f3d0015f9d0 2024-03-06T07:35:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d000ce0a0 0x7f3d001e9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d000ce0a0 0x7f3d001e9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48130/0 (socket says 172.21.15.114:48130) 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 -- 172.21.15.114:0/1105206972 learned_addr learned my addr 172.21.15.114:0/1105206972 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 -- 172.21.15.114:0/1105206972 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 msgr2=0x7f3d000b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d000b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 -- 172.21.15.114:0/1105206972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d00144c00 msgr2=0x7f3d0015e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d00144c00 0x7f3d0015e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 -- 172.21.15.114:0/1105206972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0015e880 con 0x7f3d000ce0a0 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.255+0000 7f3cfea00640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d000ce0a0 0x7f3d001e9640 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f3cf002b930 tx=0x7f3cf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.256+0000 7f3cf6a00640 1 -- 172.21.15.114:0/1105206972 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0040070 con 0x7f3d000ce0a0 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.256+0000 7f3cf6a00640 1 -- 172.21.15.114:0/1105206972 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf0039bf0 con 0x7f3d000ce0a0 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.256+0000 7f3cf6a00640 1 -- 172.21.15.114:0/1105206972 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0035040 con 0x7f3d000ce0a0 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.256+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d001f7940 con 0x7f3d000ce0a0 2024-03-06T07:35:40.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.256+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d001f7f00 con 0x7f3d000ce0a0 2024-03-06T07:35:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.257+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00067830 con 0x7f3d000ce0a0 2024-03-06T07:35:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.258+0000 7f3cf6a00640 1 -- 172.21.15.114:0/1105206972 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cf004b060 con 0x7f3d000ce0a0 2024-03-06T07:35:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.258+0000 7f3cf6a00640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ce006b4d0 0x7f3ce006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.259+0000 7f3cf6a00640 1 -- 172.21.15.114:0/1105206972 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3cf00abb00 con 0x7f3d000ce0a0 2024-03-06T07:35:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.259+0000 7f3cfe000640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ce006b4d0 0x7f3ce006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.259+0000 7f3cfe000640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ce006b4d0 0x7f3ce006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3cec006fd0 tx=0x7f3cec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.263+0000 7f3cf6a00640 1 -- 172.21.15.114:0/1105206972 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d00067830 con 0x7f3d000ce0a0 2024-03-06T07:35:40.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.522+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3d0004f330 con 0x7f3d000ce0a0 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.523+0000 7f3cf6a00640 1 -- 172.21.15.114:0/1105206972 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3d0015e880 con 0x7f3d000ce0a0 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_dump": [ 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.a", 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlZWH/ERAAeYBxXFPtLRKg+/3/r3nZDg==", 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:35:40.525 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.b", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlzrM2MxAAQFA6yBziB2wnRgt2XbvTAA==", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.c", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlTV8KIhAAWLT8sE/t43CWagU7lPVEuQ==", 2024-03-06T07:35:40.526 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mds.d", 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlLSMbCBAAoIIrLXXXX5d+xknHddcc+w==", 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.527 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.0", 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC9GuhlzQVTMxAATg70uUzjoxxPUp4Lg37Rfw==", 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.1", 2024-03-06T07:35:40.528 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDBGuhlUIaeDxAAHqDfSU61ozmUeicDN7upFw==", 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.2", 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDEGuhlLAQqJxAAtfdWsAToYtY3YGKSDW3fnA==", 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.529 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.3", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDIGuhlyufTBBAA599OVKql6ymJJGkIDqOTww==", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.4", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDPGuhle157AxAAvPv7khCwwVcxfnlLa7TilQ==", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.530 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.5", 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDSGuhl1XwJIhAADWADBBzb+O36gxXJdh9Epg==", 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.6", 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDVGuhl1DkjOhAAO0b5zvKpWtOBqR7/kMkaXw==", 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.531 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "osd.7", 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQDZGuhlx+fyBxAAPAdEEPaDCkomQEOR2Qon/A==", 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile osd", 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.532 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.0", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA==", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow r", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.1", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlvazvKRAAWQcCFhQjJEDwq8CWUNV+Lw==", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.2", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlyNlRIBAAANQcqh6OmUzYL0LIBush+Q==", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.3", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC5GuhlSwdQMxAAqMmmsoDshlrq2b5dGjhu6w==", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow r", 2024-03-06T07:35:40.534 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow rw", 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow rwx" 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "client.admin", 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlWVw1BBAAjEsjGcdx5wfezdGM+7HfRg==", 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": "allow *", 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow *", 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.535 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.x", 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC4GuhlscdsBhAARoBhtXVnE1YAHIfPUWmT4A==", 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": "mgr.y", 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "AQC3GuhlBh/aNBAAKgwf6zb0kKyRMy4FeS/sXA==", 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": { 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": "allow *", 2024-03-06T07:35:40.536 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:40.537 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": "allow *" 2024-03-06T07:35:40.537 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.537 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:40.537 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-06T07:35:40.537 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:35:40.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.525+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ce006b4d0 msgr2=0x7f3ce006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.525+0000 7f3d05800640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ce006b4d0 0x7f3ce006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3cec006fd0 tx=0x7f3cec008040 comp rx=0 tx=0).stop 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.525+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d000ce0a0 msgr2=0x7f3d001e9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.525+0000 7f3d05800640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d000ce0a0 0x7f3d001e9640 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f3cf002b930 tx=0x7f3cf00046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 shutdown_connections 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f3d0015f9d0 0x7f3d000b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d00144c00 0x7f3d0015e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f3ce006b4d0 0x7f3ce006d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 --2- 172.21.15.114:0/1105206972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d000ce0a0 0x7f3d001e9640 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 >> 172.21.15.114:0/1105206972 conn(0x7f3d000b1730 msgr2=0x7f3d00070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 shutdown_connections 2024-03-06T07:35:40.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.526+0000 7f3d05800640 1 -- 172.21.15.114:0/1105206972 wait complete. 2024-03-06T07:35:40.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.747+0000 7f4777200640 1 Processor -- start 2024-03-06T07:35:40.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.747+0000 7f4777200640 1 -- start start 2024-03-06T07:35:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f4777200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47781561f0 0x7f47780ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f4777200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47780aebf0 0x7f47780cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f4777200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477815c240 con 0x7f47780cc800 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f4777200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477815ce30 con 0x7f47780aebf0 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f4777200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477815c500 con 0x7f47781561f0 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f476d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47780aebf0 0x7f47780cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f476d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f47780aebf0 0x7f47780cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48146/0 (socket says 172.21.15.114:48146) 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.749+0000 7f476d400640 1 -- 172.21.15.114:0/1661353908 learned_addr learned my addr 172.21.15.114:0/1661353908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.750+0000 7f476d400640 1 -- 172.21.15.114:0/1661353908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47781561f0 msgr2=0x7f47780ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.750+0000 7f476d400640 1 --2- 172.21.15.114:0/1661353908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47781561f0 0x7f47780ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.750+0000 7f476d400640 1 -- 172.21.15.114:0/1661353908 >> v1:172.21.15.177:6789/0 conn(0x7f47780cc800 legacy=0x7f47780cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.750+0000 7f476d400640 1 -- 172.21.15.114:0/1661353908 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477815c7e0 con 0x7f47780aebf0 2024-03-06T07:35:40.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.751+0000 7f476d400640 1 --2- 172.21.15.114:0/1661353908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780aebf0 0x7f47780cc2c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f4760009870 tx=0x7f476002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afb0ad3631d119e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.751+0000 7f4775400640 1 -- 172.21.15.114:0/1661353908 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760035020 con 0x7f47780aebf0 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.751+0000 7f4775400640 1 -- 172.21.15.114:0/1661353908 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4760031640 con 0x7f47780aebf0 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.751+0000 7f4775400640 1 -- 172.21.15.114:0/1661353908 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476002d040 con 0x7f47780aebf0 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 -- 172.21.15.114:0/1661353908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780aebf0 msgr2=0x7f47780cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 --2- 172.21.15.114:0/1661353908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780aebf0 0x7f47780cc2c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f4760009870 tx=0x7f476002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 -- 172.21.15.114:0/1661353908 shutdown_connections 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 --2- 172.21.15.114:0/1661353908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780aebf0 0x7f47780cc2c0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 --2- 172.21.15.114:0/1661353908 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47781561f0 0x7f47780ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 -- 172.21.15.114:0/1661353908 >> 172.21.15.114:0/1661353908 conn(0x7f477806afc0 msgr2=0x7f477806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 -- 172.21.15.114:0/1661353908 shutdown_connections 2024-03-06T07:35:40.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.752+0000 7f4777200640 1 -- 172.21.15.114:0/1661353908 wait complete. 2024-03-06T07:35:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.753+0000 7f4777200640 1 Processor -- start 2024-03-06T07:35:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.754+0000 7f4777200640 1 -- start start 2024-03-06T07:35:40.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.754+0000 7f4777200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47780aebf0 0x7f47781e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.754+0000 7f4777200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780cc800 0x7f47781f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.754+0000 7f4777200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47781561f0 0x7f4778158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.754+0000 7f4777200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47781565d0 con 0x7f47780cc800 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.754+0000 7f4777200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47780ccc90 con 0x7f47781561f0 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.754+0000 7f4777200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4778156750 con 0x7f47780aebf0 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780cc800 0x7f47781f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780cc800 0x7f47781f4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48152/0 (socket says 172.21.15.114:48152) 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 -- 172.21.15.114:0/383781680 learned_addr learned my addr 172.21.15.114:0/383781680 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f4775e00640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47780aebf0 0x7f47781e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 -- 172.21.15.114:0/383781680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47780aebf0 msgr2=0x7f47781e5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47780aebf0 0x7f47781e5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 -- 172.21.15.114:0/383781680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47781561f0 msgr2=0x7f4778158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47781561f0 0x7f4778158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 -- 172.21.15.114:0/383781680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47781f5140 con 0x7f47780cc800 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.755+0000 7f476d400640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780cc800 0x7f47781f4c00 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f476002b930 tx=0x7f47600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.756+0000 7f476ea00640 1 -- 172.21.15.114:0/383781680 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476003ac60 con 0x7f47780cc800 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.756+0000 7f476ea00640 1 -- 172.21.15.114:0/383781680 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476003ae20 con 0x7f47780cc800 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.756+0000 7f476ea00640 1 -- 172.21.15.114:0/383781680 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760044630 con 0x7f47780cc800 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.756+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47781f7b20 con 0x7f47780cc800 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.756+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47781f80e0 con 0x7f47780cc800 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.757+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477804f330 con 0x7f47780cc800 2024-03-06T07:35:40.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.758+0000 7f476ea00640 1 -- 172.21.15.114:0/383781680 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f476004b060 con 0x7f47780cc800 2024-03-06T07:35:40.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.759+0000 7f476ea00640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f475406b4d0 0x7f475406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.759+0000 7f476ea00640 1 -- 172.21.15.114:0/383781680 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f47600abdc0 con 0x7f47780cc800 2024-03-06T07:35:40.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.759+0000 7f4775e00640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f475406b4d0 0x7f475406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.760+0000 7f4775e00640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f475406b4d0 0x7f475406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4778159770 tx=0x7f4768008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.763+0000 7f476ea00640 1 -- 172.21.15.114:0/383781680 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f476003f070 con 0x7f47780cc800 2024-03-06T07:35:40.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.960+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47780b2b10 con 0x7f47780cc800 2024-03-06T07:35:40.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.961+0000 7f476ea00640 1 -- 172.21.15.114:0/383781680 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+12949 (secure 0 0 0) 0x7f476002f300 con 0x7f47780cc800 2024-03-06T07:35:40.963 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:40.963 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":121,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:35:40.039718+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f475406b4d0 msgr2=0x7f475406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f475406b4d0 0x7f475406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4778159770 tx=0x7f4768008040 comp rx=0 tx=0).stop 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780cc800 msgr2=0x7f47781f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780cc800 0x7f47781f4c00 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f476002b930 tx=0x7f47600046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 shutdown_connections 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47781561f0 0x7f4778158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f475406b4d0 0x7f475406d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47780cc800 0x7f47781f4c00 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 --2- 172.21.15.114:0/383781680 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f47780aebf0 0x7f47781e5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 >> 172.21.15.114:0/383781680 conn(0x7f477806afc0 msgr2=0x7f4778155290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 shutdown_connections 2024-03-06T07:35:40.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:40.964+0000 7f4777200640 1 -- 172.21.15.114:0/383781680 wait complete. 2024-03-06T07:35:40.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:40.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.189+0000 7f7002c00640 1 Processor -- start 2024-03-06T07:35:41.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.189+0000 7f7002c00640 1 -- start start 2024-03-06T07:35:41.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.191+0000 7f7002c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6ffc154dc0 0x7f6ffc161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.191+0000 7f7002c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc1551a0 0x7f6ffc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.191+0000 7f7002c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc0cfb30 con 0x7f6ffc0aee20 2024-03-06T07:35:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.191+0000 7f7002c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0af270 con 0x7f6ffc154dc0 2024-03-06T07:35:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.191+0000 7f7002c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc15e540 con 0x7f6ffc1551a0 2024-03-06T07:35:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.192+0000 7f7000e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc1551a0 0x7f6ffc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.192+0000 7f7000e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc1551a0 0x7f6ffc0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:33952/0 (socket says 172.21.15.114:33952) 2024-03-06T07:35:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.192+0000 7f7000e00640 1 -- 172.21.15.114:0/1377983050 learned_addr learned my addr 172.21.15.114:0/1377983050 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.192+0000 7f7001800640 1 --2- 172.21.15.114:0/1377983050 >> v2:172.21.15.177:3300/0 conn(0x7f6ffc154dc0 0x7f6ffc161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.193+0000 7f7000e00640 1 -- 172.21.15.114:0/1377983050 >> v2:172.21.15.177:3300/0 conn(0x7f6ffc154dc0 msgr2=0x7f6ffc161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.193+0000 7f7000e00640 1 --2- 172.21.15.114:0/1377983050 >> v2:172.21.15.177:3300/0 conn(0x7f6ffc154dc0 0x7f6ffc161b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.193+0000 7f7000e00640 1 -- 172.21.15.114:0/1377983050 >> v1:172.21.15.177:6789/0 conn(0x7f6ffc0aee20 legacy=0x7f6ffc0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.193+0000 7f7000e00640 1 -- 172.21.15.114:0/1377983050 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc0cfe10 con 0x7f6ffc1551a0 2024-03-06T07:35:41.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.194+0000 7f7000e00640 1 --2- 172.21.15.114:0/1377983050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc1551a0 0x7f6ffc0ae8e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea11eb02d494cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.194+0000 7f6ffbe00640 1 -- 172.21.15.114:0/1377983050 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec034070 con 0x7f6ffc1551a0 2024-03-06T07:35:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.194+0000 7f6ffbe00640 1 -- 172.21.15.114:0/1377983050 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec030840 con 0x7f6ffc1551a0 2024-03-06T07:35:41.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.194+0000 7f6ffbe00640 1 -- 172.21.15.114:0/1377983050 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec030c10 con 0x7f6ffc1551a0 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 -- 172.21.15.114:0/1377983050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc1551a0 msgr2=0x7f6ffc0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 --2- 172.21.15.114:0/1377983050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc1551a0 0x7f6ffc0ae8e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec02b980 comp rx=0 tx=0).stop 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 -- 172.21.15.114:0/1377983050 shutdown_connections 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 --2- 172.21.15.114:0/1377983050 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc1551a0 0x7f6ffc0ae8e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 --2- 172.21.15.114:0/1377983050 >> v2:172.21.15.177:3300/0 conn(0x7f6ffc154dc0 0x7f6ffc161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 -- 172.21.15.114:0/1377983050 >> 172.21.15.114:0/1377983050 conn(0x7f6ffc06afc0 msgr2=0x7f6ffc06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 -- 172.21.15.114:0/1377983050 shutdown_connections 2024-03-06T07:35:41.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.195+0000 7f7002c00640 1 -- 172.21.15.114:0/1377983050 wait complete. 2024-03-06T07:35:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.197+0000 7f7002c00640 1 Processor -- start 2024-03-06T07:35:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.197+0000 7f7002c00640 1 -- start start 2024-03-06T07:35:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7002c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ffc0aee20 0x7f6ffc157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7002c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc154dc0 0x7f6ffc1491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7002c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ffc1551a0 0x7f6ffc0b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7002c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0cf820 con 0x7f6ffc0aee20 2024-03-06T07:35:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7002c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0d03d0 con 0x7f6ffc1551a0 2024-03-06T07:35:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7002c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc162070 con 0x7f6ffc154dc0 2024-03-06T07:35:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ffc0aee20 0x7f6ffc157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ffc0aee20 0x7f6ffc157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48162/0 (socket says 172.21.15.114:48162) 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 -- 172.21.15.114:0/4168078184 learned_addr learned my addr 172.21.15.114:0/4168078184 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 -- 172.21.15.114:0/4168078184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc154dc0 msgr2=0x7f6ffc1491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc154dc0 0x7f6ffc1491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 -- 172.21.15.114:0/4168078184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ffc1551a0 msgr2=0x7f6ffc0b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ffc1551a0 0x7f6ffc0b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.198+0000 7f7001800640 1 -- 172.21.15.114:0/4168078184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc156730 con 0x7f6ffc0aee20 2024-03-06T07:35:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.199+0000 7f7001800640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ffc0aee20 0x7f6ffc157a50 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000e9b0 tx=0x7f6ff000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.199+0000 7f6ff9600640 1 -- 172.21.15.114:0/4168078184 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0017070 con 0x7f6ffc0aee20 2024-03-06T07:35:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.199+0000 7f6ff9600640 1 -- 172.21.15.114:0/4168078184 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff000c9a0 con 0x7f6ffc0aee20 2024-03-06T07:35:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.199+0000 7f6ff9600640 1 -- 172.21.15.114:0/4168078184 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff000cd90 con 0x7f6ffc0aee20 2024-03-06T07:35:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.199+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc156a10 con 0x7f6ffc0aee20 2024-03-06T07:35:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.199+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc0b2310 con 0x7f6ffc0aee20 2024-03-06T07:35:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.200+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc04f330 con 0x7f6ffc0aee20 2024-03-06T07:35:41.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.201+0000 7f6ff9600640 1 -- 172.21.15.114:0/4168078184 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ff001a070 con 0x7f6ffc0aee20 2024-03-06T07:35:41.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.203+0000 7f6ff9600640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6fe006b500 0x7f6fe006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.203+0000 7f6ff9600640 1 -- 172.21.15.114:0/4168078184 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ff008a200 con 0x7f6ffc0aee20 2024-03-06T07:35:41.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.203+0000 7f7000e00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6fe006b500 0x7f6fe006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.203+0000 7f7000e00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6fe006b500 0x7f6fe006d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6fec036000 tx=0x7f6fec030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.207+0000 7f6ff9600640 1 -- 172.21.15.114:0/4168078184 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ff0059460 con 0x7f6ffc0aee20 2024-03-06T07:35:41.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.403+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ffc0b0940 con 0x7f6ffc0aee20 2024-03-06T07:35:41.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.404+0000 7f6ff9600640 1 -- 172.21.15.114:0/4168078184 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ff0058d60 con 0x7f6ffc0aee20 2024-03-06T07:35:41.406 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:35:41.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6fe006b500 msgr2=0x7f6fe006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6fe006b500 0x7f6fe006d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6fec036000 tx=0x7f6fec030000 comp rx=0 tx=0).stop 2024-03-06T07:35:41.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ffc0aee20 msgr2=0x7f6ffc157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ffc0aee20 0x7f6ffc157a50 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000e9b0 tx=0x7f6ff000ee70 comp rx=0 tx=0).stop 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 shutdown_connections 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6ffc1551a0 0x7f6ffc0b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6ffc154dc0 0x7f6ffc1491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6fe006b500 0x7f6fe006d9c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 --2- 172.21.15.114:0/4168078184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ffc0aee20 0x7f6ffc157a50 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 >> 172.21.15.114:0/4168078184 conn(0x7f6ffc06afc0 msgr2=0x7f6ffc0ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 shutdown_connections 2024-03-06T07:35:41.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.407+0000 7f7002c00640 1 -- 172.21.15.114:0/4168078184 wait complete. 2024-03-06T07:35:41.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:35:41.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.629+0000 7fde1c000640 1 Processor -- start 2024-03-06T07:35:41.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.629+0000 7fde1c000640 1 -- start start 2024-03-06T07:35:41.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde1c000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fde140cff20 0x7fde140b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde1c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde140b7860 0x7fde140d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde1c000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde140b5e00 con 0x7fde140b73e0 2024-03-06T07:35:41.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde1c000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde140b6a10 con 0x7fde140cff20 2024-03-06T07:35:41.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde1c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde140b60e0 con 0x7fde140b7860 2024-03-06T07:35:41.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde18c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fde140b73e0 0x7fde140d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51792/0 (socket says 172.21.15.114:51792) 2024-03-06T07:35:41.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde18c00640 1 -- 172.21.15.114:0/628902712 learned_addr learned my addr 172.21.15.114:0/628902712 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde19600640 1 --2- 172.21.15.114:0/628902712 >> v2:172.21.15.177:3300/0 conn(0x7fde140cff20 0x7fde140b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde1a000640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde140b7860 0x7fde140d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.631+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630860174 0 0) 0x7fde140b5e00 con 0x7fde140b73e0 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddfc003520 con 0x7fde140b73e0 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1211268116 0 0) 0x7fddfc003520 con 0x7fde140b73e0 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde140b7860 msgr2=0x7fde140d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde140b7860 0x7fde140d0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 >> v2:172.21.15.177:3300/0 conn(0x7fde140cff20 msgr2=0x7fde140b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 --2- 172.21.15.114:0/628902712 >> v2:172.21.15.177:3300/0 conn(0x7fde140cff20 0x7fde140b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde140b63c0 con 0x7fde140b73e0 2024-03-06T07:35:41.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fde00001670 con 0x7fde140b73e0 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 >> v1:172.21.15.114:6789/0 conn(0x7fde140b73e0 legacy=0x7fde140d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc0039b0 0x7fddfc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde140b7860 0x7fddfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc004d40 0x7fddfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140b6a10 con 0x7fddfc004d40 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140b60e0 con 0x7fddfc0039b0 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc00b9c0 con 0x7fde140b7860 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.632+0000 7fde1a000640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc004d40 0x7fddfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.633+0000 7fde1a000640 1 -- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde140b7860 msgr2=0x7fddfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.633+0000 7fde1a000640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde140b7860 0x7fddfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.633+0000 7fde19600640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc0039b0 0x7fddfc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.633+0000 7fde1a000640 1 -- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc0039b0 msgr2=0x7fddfc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:41.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.633+0000 7fde1a000640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddfc0039b0 0x7fddfc003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.633+0000 7fde1a000640 1 -- 172.21.15.114:0/628902712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc003520 con 0x7fddfc004d40 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.634+0000 7fde1a000640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc004d40 0x7fddfc0051c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00e170 tx=0x7fddfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2df09edb6baad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.634+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c039070 con 0x7fddfc004d40 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.634+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c02cc20 con 0x7fddfc004d40 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.634+0000 7fde13e00640 1 -- 172.21.15.114:0/628902712 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c009c10 con 0x7fddfc004d40 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.635+0000 7fde1c000640 1 -- 172.21.15.114:0/628902712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc004d40 msgr2=0x7fddfc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.635+0000 7fde1c000640 1 --2- 172.21.15.114:0/628902712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc004d40 0x7fddfc0051c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00e170 tx=0x7fddfc005830 comp rx=0 tx=0).stop 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.635+0000 7fde1a000640 1 -- 172.21.15.114:0/628902712 reap_dead start 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.635+0000 7fde1c000640 1 -- 172.21.15.114:0/628902712 shutdown_connections 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.635+0000 7fde1c000640 1 -- 172.21.15.114:0/628902712 >> 172.21.15.114:0/628902712 conn(0x7fde140d1c90 msgr2=0x7fde140d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.635+0000 7fde1c000640 1 -- 172.21.15.114:0/628902712 shutdown_connections 2024-03-06T07:35:41.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.635+0000 7fde1c000640 1 -- 172.21.15.114:0/628902712 wait complete. 2024-03-06T07:35:41.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.636+0000 7fde1c000640 1 Processor -- start 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.637+0000 7fde1c000640 1 -- start start 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.637+0000 7fde1c000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde14158c00 0x7fde140c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.637+0000 7fde1c000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde140c74b0 0x7fde14148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde1c000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14148780 0x7fde1415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde1c000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde14161a50 con 0x7fde140c74b0 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde1c000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140b5af0 con 0x7fde14148780 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde1c000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde140d0a10 con 0x7fde14158c00 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde140c74b0 0x7fde14148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde140c74b0 0x7fde14148240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48192/0 (socket says 172.21.15.114:48192) 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 -- 172.21.15.114:0/2460133025 learned_addr learned my addr 172.21.15.114:0/2460133025 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde19600640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde14158c00 0x7fde140c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 -- 172.21.15.114:0/2460133025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde14158c00 msgr2=0x7fde140c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde14158c00 0x7fde140c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 -- 172.21.15.114:0/2460133025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14148780 msgr2=0x7fde1415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14148780 0x7fde1415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 -- 172.21.15.114:0/2460133025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14148ee0 con 0x7fde140c74b0 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.638+0000 7fde18c00640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde140c74b0 0x7fde14148240 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fde0000c760 tx=0x7fde0000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.639+0000 7fde11600640 1 -- 172.21.15.114:0/2460133025 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00018070 con 0x7fde140c74b0 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.639+0000 7fde11600640 1 -- 172.21.15.114:0/2460133025 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde00004cb0 con 0x7fde140c74b0 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.639+0000 7fde11600640 1 -- 172.21.15.114:0/2460133025 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00004e60 con 0x7fde140c74b0 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.639+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde141462e0 con 0x7fde140c74b0 2024-03-06T07:35:41.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.639+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde141468a0 con 0x7fde140c74b0 2024-03-06T07:35:41.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.640+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde140c7930 con 0x7fde140c74b0 2024-03-06T07:35:41.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.641+0000 7fde11600640 1 -- 172.21.15.114:0/2460133025 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde00007310 con 0x7fde140c74b0 2024-03-06T07:35:41.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.642+0000 7fde11600640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddf406b4d0 0x7fddf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:41.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.642+0000 7fde11600640 1 -- 172.21.15.114:0/2460133025 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fde000893b0 con 0x7fde140c74b0 2024-03-06T07:35:41.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.642+0000 7fde19600640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddf406b4d0 0x7fddf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:41.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.642+0000 7fde19600640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddf406b4d0 0x7fddf406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fde080099b0 tx=0x7fde08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:41.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.646+0000 7fde11600640 1 -- 172.21.15.114:0/2460133025 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde00058570 con 0x7fde140c74b0 2024-03-06T07:35:41.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.832+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fde1404f330 con 0x7fde140c74b0 2024-03-06T07:35:41.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.833+0000 7fde11600640 1 -- 172.21.15.114:0/2460133025 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+65 (secure 0 0 0) 0x7fde00057e70 con 0x7fde140c74b0 2024-03-06T07:35:41.834 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:41.835 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-03-06T07:35:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-03-06T07:35:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-03-06T07:35:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-03-06T07:35:41.835 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-06T07:35:41.835 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-03-06T07:35:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddf406b4d0 msgr2=0x7fddf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddf406b4d0 0x7fddf406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fde080099b0 tx=0x7fde08008040 comp rx=0 tx=0).stop 2024-03-06T07:35:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde140c74b0 msgr2=0x7fde14148240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde140c74b0 0x7fde14148240 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fde0000c760 tx=0x7fde0000cc20 comp rx=0 tx=0).stop 2024-03-06T07:35:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 shutdown_connections 2024-03-06T07:35:41.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14148780 0x7fde1415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fddf406b4d0 0x7fddf406d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde140c74b0 0x7fde14148240 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 --2- 172.21.15.114:0/2460133025 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fde14158c00 0x7fde140c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:41.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 >> 172.21.15.114:0/2460133025 conn(0x7fde140d1c90 msgr2=0x7fde140c7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:41.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.835+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 shutdown_connections 2024-03-06T07:35:41.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:41.836+0000 7fde1c000640 1 -- 172.21.15.114:0/2460133025 wait complete. 2024-03-06T07:35:41.853 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:35:41.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:35:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.050+0000 7fc5dfa00640 1 Processor -- start 2024-03-06T07:35:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.050+0000 7fc5dfa00640 1 -- start start 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.051+0000 7fc5dfa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d80b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.051+0000 7fc5dfa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5d815e770 0x7fc5d8150c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.051+0000 7fc5dfa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80b3bd0 con 0x7fc5d80cf3a0 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.051+0000 7fc5dfa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d80b47e0 con 0x7fc5d815e770 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.051+0000 7fc5dfa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d80b3eb0 con 0x7fc5d806e850 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.051+0000 7fc5dda00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5d815e770 0x7fc5d8150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dda00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5d815e770 0x7fc5d8150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48206/0 (socket says 172.21.15.114:48206) 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dda00640 1 -- 172.21.15.114:0/3081643077 learned_addr learned my addr 172.21.15.114:0/3081643077 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5cfe00640 1 --1- 172.21.15.114:0/3081643077 >> v1:172.21.15.114:6789/0 conn(0x7fc5d80cf3a0 0x7fc5d815e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51800/0 (socket says 172.21.15.114:51800) 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dd000640 1 --2- 172.21.15.114:0/3081643077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d80b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dda00640 1 -- 172.21.15.114:0/3081643077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 msgr2=0x7fc5d80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:42.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dda00640 1 --2- 172.21.15.114:0/3081643077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d80b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dda00640 1 -- 172.21.15.114:0/3081643077 >> v1:172.21.15.114:6789/0 conn(0x7fc5d80cf3a0 legacy=0x7fc5d815e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dda00640 1 -- 172.21.15.114:0/3081643077 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d80b4190 con 0x7fc5d815e770 2024-03-06T07:35:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5d7e00640 1 -- 172.21.15.114:0/3081643077 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730084641 0 0) 0x7fc5d80b3bd0 con 0x7fc5d80cf3a0 2024-03-06T07:35:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.052+0000 7fc5dd000640 1 --2- 172.21.15.114:0/3081643077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d80b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.053+0000 7fc5dda00640 1 --2- 172.21.15.114:0/3081643077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d8150c90 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009e50 tx=0x7fc5d002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cab1b8a38bd538 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.053+0000 7fc5d7e00640 1 -- 172.21.15.114:0/3081643077 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d002cb00 con 0x7fc5d815e770 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.053+0000 7fc5d7e00640 1 -- 172.21.15.114:0/3081643077 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d002cca0 con 0x7fc5d815e770 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.053+0000 7fc5d7e00640 1 -- 172.21.15.114:0/3081643077 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0034610 con 0x7fc5d815e770 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3081643077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 msgr2=0x7fc5d8150c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3081643077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d8150c90 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009e50 tx=0x7fc5d002bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3081643077 shutdown_connections 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3081643077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d8150c90 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3081643077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d80b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3081643077 >> 172.21.15.114:0/3081643077 conn(0x7fc5d80cfa80 msgr2=0x7fc5d80cfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3081643077 shutdown_connections 2024-03-06T07:35:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3081643077 wait complete. 2024-03-06T07:35:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.056+0000 7fc5dfa00640 1 Processor -- start 2024-03-06T07:35:42.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.056+0000 7fc5dfa00640 1 -- start start 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dfa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d8144230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dfa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5d80cf3a0 0x7fc5d81f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dfa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d81f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dfa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d80b4c70 con 0x7fc5d815e770 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dfa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d80cf7e0 con 0x7fc5d80cf3a0 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dfa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d814a550 con 0x7fc5d806e850 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d81f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d81f7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48212/0 (socket says 172.21.15.114:48212) 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 -- 172.21.15.114:0/3908125235 learned_addr learned my addr 172.21.15.114:0/3908125235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dd000640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d8144230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5cfe00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5d80cf3a0 0x7fc5d81f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 -- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 msgr2=0x7fc5d8144230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d8144230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 -- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5d80cf3a0 msgr2=0x7fc5d81f70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5d80cf3a0 0x7fc5d81f70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.057+0000 7fc5dda00640 1 -- 172.21.15.114:0/3908125235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8141760 con 0x7fc5d815e770 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.058+0000 7fc5dda00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d81f7610 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009e50 tx=0x7fc5d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.058+0000 7fc5d5600640 1 -- 172.21.15.114:0/3908125235 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0044030 con 0x7fc5d815e770 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.058+0000 7fc5d5600640 1 -- 172.21.15.114:0/3908125235 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d002ce50 con 0x7fc5d815e770 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.058+0000 7fc5d5600640 1 -- 172.21.15.114:0/3908125235 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d00406d0 con 0x7fc5d815e770 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.058+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d8141ce0 con 0x7fc5d815e770 2024-03-06T07:35:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.058+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d8141a00 con 0x7fc5d815e770 2024-03-06T07:35:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.059+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d815e520 con 0x7fc5d815e770 2024-03-06T07:35:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.060+0000 7fc5d5600640 1 -- 172.21.15.114:0/3908125235 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5d0047070 con 0x7fc5d815e770 2024-03-06T07:35:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.061+0000 7fc5d5600640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5b406b4d0 0x7fc5b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:42.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.061+0000 7fc5d5600640 1 -- 172.21.15.114:0/3908125235 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc5d00ac550 con 0x7fc5d815e770 2024-03-06T07:35:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.061+0000 7fc5dd000640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5b406b4d0 0x7fc5b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:42.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.062+0000 7fc5dd000640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5b406b4d0 0x7fc5b406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800a990 tx=0x7fc5c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.065+0000 7fc5d5600640 1 -- 172.21.15.114:0/3908125235 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5d815e520 con 0x7fc5d815e770 2024-03-06T07:35:42.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:42.248+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc5d804f330 con 0x7fc5d815e770 2024-03-06T07:35:43.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.049+0000 7fc5d5600640 1 -- 172.21.15.114:0/3908125235 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v122) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc5d004e070 con 0x7fc5d815e770 2024-03-06T07:35:43.052 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.053+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5b406b4d0 msgr2=0x7fc5b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.053+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5b406b4d0 0x7fc5b406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800a990 tx=0x7fc5c8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.053+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 msgr2=0x7fc5d81f7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.053+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d81f7610 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009e50 tx=0x7fc5d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 shutdown_connections 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc5b406b4d0 0x7fc5b406d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5d815e770 0x7fc5d81f7610 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5d80cf3a0 0x7fc5d81f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 --2- 172.21.15.114:0/3908125235 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc5d806e850 0x7fc5d8144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 >> 172.21.15.114:0/3908125235 conn(0x7fc5d80cfa80 msgr2=0x7fc5d8159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:43.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 shutdown_connections 2024-03-06T07:35:43.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.054+0000 7fc5dfa00640 1 -- 172.21.15.114:0/3908125235 wait complete. 2024-03-06T07:35:43.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:35:43.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.275+0000 7f1173200640 1 Processor -- start 2024-03-06T07:35:43.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.275+0000 7f1173200640 1 -- start start 2024-03-06T07:35:43.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f1173200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c0b3f70 0x7f116c160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f1173200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116c0b46c0 con 0x7f116c15fc10 2024-03-06T07:35:43.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f1173200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116c0b49a0 con 0x7f116c070ff0 2024-03-06T07:35:43.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f1173200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116c071480 con 0x7f116c0b3f70 2024-03-06T07:35:43.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f116be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f116c15fc10 0x7f116c15fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51804/0 (socket says 172.21.15.114:51804) 2024-03-06T07:35:43.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f116be00640 1 -- 172.21.15.114:0/3609107243 learned_addr learned my addr 172.21.15.114:0/3609107243 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f116b400640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c0b3f70 0x7f116c160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.277+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989577536 0 0) 0x7f116c0b46c0 con 0x7f116c15fc10 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150003520 con 0x7f116c15fc10 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 326825175 0 0) 0x7f1150003520 con 0x7f116c15fc10 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c0b3f70 msgr2=0x7f116c160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c0b3f70 0x7f116c160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 >> v1:172.21.15.177:6789/0 conn(0x7f116c070ff0 legacy=0x7f116c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c072c90 con 0x7f116c15fc10 2024-03-06T07:35:43.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f115c001670 con 0x7f116c15fc10 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 >> v1:172.21.15.114:6789/0 conn(0x7f116c15fc10 legacy=0x7f116c15fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11500039e0 0x7f1150003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150004800 0x7f1150004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c0b3f70 0x7f1150005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c071480 con 0x7f1150004800 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115000baf0 con 0x7f11500039e0 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115000bcf0 con 0x7f116c0b3f70 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.278+0000 7f116b400640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150004800 0x7f1150004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.279+0000 7f116be00640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11500039e0 0x7f1150003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.279+0000 7f116b400640 1 -- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c0b3f70 msgr2=0x7f1150005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.279+0000 7f116b400640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c0b3f70 0x7f1150005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.279+0000 7f116b400640 1 -- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11500039e0 msgr2=0x7f1150003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.279+0000 7f116b400640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11500039e0 0x7f1150003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.279+0000 7f116b400640 1 -- 172.21.15.114:0/3609107243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150003520 con 0x7f1150004800 2024-03-06T07:35:43.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.279+0000 7f116be00640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11500039e0 0x7f1150003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.280+0000 7f116b400640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150004800 0x7f1150004c80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f115800ba00 tx=0x7f1150005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6447adfe2ac2f487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.280+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158036020 con 0x7f1150004800 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.280+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1158032640 con 0x7f1150004800 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.280+0000 7f116aa00640 1 -- 172.21.15.114:0/3609107243 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115800b600 con 0x7f1150004800 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.281+0000 7f1173200640 1 -- 172.21.15.114:0/3609107243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150004800 msgr2=0x7f1150004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.281+0000 7f1173200640 1 --2- 172.21.15.114:0/3609107243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150004800 0x7f1150004c80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f115800ba00 tx=0x7f1150005830 comp rx=0 tx=0).stop 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.281+0000 7f1171200640 1 -- 172.21.15.114:0/3609107243 reap_dead start 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.281+0000 7f1173200640 1 -- 172.21.15.114:0/3609107243 shutdown_connections 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.281+0000 7f1173200640 1 -- 172.21.15.114:0/3609107243 >> 172.21.15.114:0/3609107243 conn(0x7f116c0b1940 msgr2=0x7f116c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.281+0000 7f1173200640 1 -- 172.21.15.114:0/3609107243 shutdown_connections 2024-03-06T07:35:43.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.281+0000 7f1173200640 1 -- 172.21.15.114:0/3609107243 wait complete. 2024-03-06T07:35:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.283+0000 7f1173200640 1 Processor -- start 2024-03-06T07:35:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.283+0000 7f1173200640 1 -- start start 2024-03-06T07:35:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f1173200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f116c140a30 0x7f116c14f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f1173200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c150860 0x7f116c15ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f1173200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c140f50 0x7f116c1413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f1173200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c072a30 con 0x7f116c140a30 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f1173200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c0728b0 con 0x7f116c150860 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f1173200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116c15c2e0 con 0x7f116c140f50 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f116c140a30 0x7f116c14f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f116c140a30 0x7f116c14f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48228/0 (socket says 172.21.15.114:48228) 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116be00640 1 -- 172.21.15.114:0/297642316 learned_addr learned my addr 172.21.15.114:0/297642316 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116b400640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c150860 0x7f116c15ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f1171200640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c140f50 0x7f116c1413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116be00640 1 -- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c140f50 msgr2=0x7f116c1413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116be00640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c140f50 0x7f116c1413b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116be00640 1 -- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c150860 msgr2=0x7f116c15ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.284+0000 7f116be00640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c150860 0x7f116c15ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f116be00640 1 -- 172.21.15.114:0/297642316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c15f450 con 0x7f116c140a30 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f116b400640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c150860 0x7f116c15ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:43.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f116be00640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f116c140a30 0x7f116c14f1c0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f115c00eaa0 tx=0x7f115c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f1163e00640 1 -- 172.21.15.114:0/297642316 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c00c960 con 0x7f116c140a30 2024-03-06T07:35:43.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f1163e00640 1 -- 172.21.15.114:0/297642316 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c018070 con 0x7f116c140a30 2024-03-06T07:35:43.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c14f7c0 con 0x7f116c140a30 2024-03-06T07:35:43.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f1163e00640 1 -- 172.21.15.114:0/297642316 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c00cd00 con 0x7f116c140a30 2024-03-06T07:35:43.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.285+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116c14fd80 con 0x7f116c140a30 2024-03-06T07:35:43.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.286+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116c04f330 con 0x7f116c140a30 2024-03-06T07:35:43.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.288+0000 7f1163e00640 1 -- 172.21.15.114:0/297642316 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f115c026430 con 0x7f116c140a30 2024-03-06T07:35:43.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.288+0000 7f1163e00640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f114c06b500 0x7f114c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:43.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.289+0000 7f1163e00640 1 -- 172.21.15.114:0/297642316 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f115c08a880 con 0x7f116c140a30 2024-03-06T07:35:43.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.289+0000 7f116b400640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f114c06b500 0x7f114c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:43.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.289+0000 7f116b400640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f114c06b500 0x7f114c06d9c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f116c0b43b0 tx=0x7f1158030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:43.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.292+0000 7f1163e00640 1 -- 172.21.15.114:0/297642316 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f115c059950 con 0x7f116c140a30 2024-03-06T07:35:43.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:43.476+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f116c06ef30 con 0x7f116c140a30 2024-03-06T07:35:44.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.063+0000 7f1163e00640 1 -- 172.21.15.114:0/297642316 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v123) v1 ==== 170+0+0 (secure 0 0 0) 0x7f115c059250 con 0x7f116c140a30 2024-03-06T07:35:44.065 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-03-06T07:35:44.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.066+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f114c06b500 msgr2=0x7f114c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f114c06b500 0x7f114c06d9c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f116c0b43b0 tx=0x7f1158030000 comp rx=0 tx=0).stop 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f116c140a30 msgr2=0x7f116c14f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f116c140a30 0x7f116c14f1c0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f115c00eaa0 tx=0x7f115c00ef60 comp rx=0 tx=0).stop 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 shutdown_connections 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f116c140f50 0x7f116c1413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f116c150860 0x7f116c15ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f114c06b500 0x7f114c06d9c0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 --2- 172.21.15.114:0/297642316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f116c140a30 0x7f116c14f1c0 secure :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f115c00eaa0 tx=0x7f115c00ef60 comp rx=0 tx=0).stop 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.067+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 >> 172.21.15.114:0/297642316 conn(0x7f116c0b1940 msgr2=0x7f116c0ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:44.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.068+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 shutdown_connections 2024-03-06T07:35:44.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.068+0000 7f1173200640 1 -- 172.21.15.114:0/297642316 wait complete. 2024-03-06T07:35:44.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:35:44.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.286+0000 7f9f57e00640 1 Processor -- start 2024-03-06T07:35:44.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.286+0000 7f9f57e00640 1 -- start start 2024-03-06T07:35:44.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f57e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f580d1ea0 0x7f9f580d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f57e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580d13e0 con 0x7f9f5815e130 2024-03-06T07:35:44.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f57e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580cead0 con 0x7f9f580c7970 2024-03-06T07:35:44.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f57e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f580d2840 con 0x7f9f580d1ea0 2024-03-06T07:35:44.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f56000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f580d1ea0 0x7f9f580d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f56000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f580d1ea0 0x7f9f580d2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48236/0 (socket says 172.21.15.114:48236) 2024-03-06T07:35:44.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f56000640 1 -- 172.21.15.114:0/2161569373 learned_addr learned my addr 172.21.15.114:0/2161569373 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:44.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f56000640 1 -- 172.21.15.114:0/2161569373 >> v1:172.21.15.177:6789/0 conn(0x7f9f580c7970 legacy=0x7f9f580d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:44.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f56000640 1 -- 172.21.15.114:0/2161569373 >> v1:172.21.15.114:6789/0 conn(0x7f9f5815e130 legacy=0x7f9f580cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.288+0000 7f9f56000640 1 -- 172.21.15.114:0/2161569373 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f580cedb0 con 0x7f9f580d1ea0 2024-03-06T07:35:44.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.290+0000 7f9f56000640 1 --2- 172.21.15.114:0/2161569373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580d1ea0 0x7f9f580d2300 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009e50 tx=0x7f9f4002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6bdbdc1cafd482 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.290+0000 7f9f55600640 1 -- 172.21.15.114:0/2161569373 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4002cb10 con 0x7f9f580d1ea0 2024-03-06T07:35:44.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.290+0000 7f9f55600640 1 -- 172.21.15.114:0/2161569373 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4002ccb0 con 0x7f9f580d1ea0 2024-03-06T07:35:44.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.290+0000 7f9f55600640 1 -- 172.21.15.114:0/2161569373 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4002ce60 con 0x7f9f580d1ea0 2024-03-06T07:35:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.291+0000 7f9f57e00640 1 -- 172.21.15.114:0/2161569373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580d1ea0 msgr2=0x7f9f580d2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.291+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2161569373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580d1ea0 0x7f9f580d2300 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009e50 tx=0x7f9f4002bf70 comp rx=0 tx=0).stop 2024-03-06T07:35:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.291+0000 7f9f57e00640 1 -- 172.21.15.114:0/2161569373 shutdown_connections 2024-03-06T07:35:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.291+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2161569373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580d1ea0 0x7f9f580d2300 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.291+0000 7f9f57e00640 1 -- 172.21.15.114:0/2161569373 >> 172.21.15.114:0/2161569373 conn(0x7f9f5809d070 msgr2=0x7f9f580d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.291+0000 7f9f57e00640 1 -- 172.21.15.114:0/2161569373 shutdown_connections 2024-03-06T07:35:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.291+0000 7f9f57e00640 1 -- 172.21.15.114:0/2161569373 wait complete. 2024-03-06T07:35:44.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.293+0000 7f9f57e00640 1 Processor -- start 2024-03-06T07:35:44.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.293+0000 7f9f57e00640 1 -- start start 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f57e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580c7970 0x7f9f580c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f57e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580d1ea0 0x7f9f580b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f57e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f5815e130 0x7f9f580b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f57e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580d16c0 con 0x7f9f580c7970 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f57e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580cf120 con 0x7f9f580d1ea0 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f57e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580d0dc0 con 0x7f9f5815e130 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580c7970 0x7f9f580c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580c7970 0x7f9f580c6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48248/0 (socket says 172.21.15.114:48248) 2024-03-06T07:35:44.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.294+0000 7f9f56a00640 1 -- 172.21.15.114:0/2193376054 learned_addr learned my addr 172.21.15.114:0/2193376054 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:44.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f56a00640 1 -- 172.21.15.114:0/2193376054 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f5815e130 msgr2=0x7f9f580b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f56a00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f5815e130 0x7f9f580b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f56a00640 1 -- 172.21.15.114:0/2193376054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580d1ea0 msgr2=0x7f9f580b88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f56a00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580d1ea0 0x7f9f580b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f56a00640 1 -- 172.21.15.114:0/2193376054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f580c70d0 con 0x7f9f580c7970 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f56a00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580c7970 0x7f9f580c6b90 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800c3c0 tx=0x7f9f4800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f4ea00640 1 -- 172.21.15.114:0/2193376054 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f480052d0 con 0x7f9f580c7970 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f4ea00640 1 -- 172.21.15.114:0/2193376054 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f48005490 con 0x7f9f580c7970 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f580b94a0 con 0x7f9f580c7970 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f4ea00640 1 -- 172.21.15.114:0/2193376054 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48007260 con 0x7f9f580c7970 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.295+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f580b9a60 con 0x7f9f580c7970 2024-03-06T07:35:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.297+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5804f330 con 0x7f9f580c7970 2024-03-06T07:35:44.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.298+0000 7f9f4ea00640 1 -- 172.21.15.114:0/2193376054 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f48007400 con 0x7f9f580c7970 2024-03-06T07:35:44.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.298+0000 7f9f4ea00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:44.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.299+0000 7f9f4ea00640 1 -- 172.21.15.114:0/2193376054 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9f48089650 con 0x7f9f580c7970 2024-03-06T07:35:44.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.299+0000 7f9f56000640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:44.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.299+0000 7f9f56000640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f3406b4d0 0x7f9f3406d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9f40035000 tx=0x7f9f4002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:44.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.304+0000 7f9f4ea00640 1 -- 172.21.15.114:0/2193376054 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f580c70d0 con 0x7f9f580c7970 2024-03-06T07:35:44.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:44.541+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9f5809d690 con 0x7f9f580c7970 2024-03-06T07:35:45.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.090+0000 7f9f4ea00640 1 -- 172.21.15.114:0/2193376054 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v92) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9f48058520 con 0x7f9f580c7970 2024-03-06T07:35:45.094 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:35:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f3406b4d0 msgr2=0x7f9f3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f3406b4d0 0x7f9f3406d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9f40035000 tx=0x7f9f4002e000 comp rx=0 tx=0).stop 2024-03-06T07:35:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580c7970 msgr2=0x7f9f580c6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580c7970 0x7f9f580c6b90 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800c3c0 tx=0x7f9f4800c880 comp rx=0 tx=0).stop 2024-03-06T07:35:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 shutdown_connections 2024-03-06T07:35:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9f5815e130 0x7f9f580b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f580d1ea0 0x7f9f580b88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 --2- 172.21.15.114:0/2193376054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f580c7970 0x7f9f580c6b90 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.104+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 >> 172.21.15.114:0/2193376054 conn(0x7f9f5809d070 msgr2=0x7f9f580d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.105+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 shutdown_connections 2024-03-06T07:35:45.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.105+0000 7f9f57e00640 1 -- 172.21.15.114:0/2193376054 wait complete. 2024-03-06T07:35:45.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.307+0000 7ff5b9200640 1 Processor -- start 2024-03-06T07:35:45.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.308+0000 7ff5b9200640 1 -- start start 2024-03-06T07:35:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.309+0000 7ff5b9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 0x7ff5b415dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.309+0000 7ff5b9200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b406db60 con 0x7ff5b4151660 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.309+0000 7ff5b9200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b40b35b0 con 0x7ff5b415e530 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.309+0000 7ff5b9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40b3ee0 con 0x7ff5b40cf430 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.309+0000 7ff5b1e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff5b4151660 0x7ff5b4071520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51814/0 (socket says 172.21.15.114:51814) 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.309+0000 7ff5b1e00640 1 -- 172.21.15.114:0/2151171296 learned_addr learned my addr 172.21.15.114:0/2151171296 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789563366 0 0) 0x7ff5b406db60 con 0x7ff5b4151660 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff594005730 con 0x7ff5b4151660 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5abe00640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 0x7ff5b415dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2400397506 0 0) 0x7ff594005730 con 0x7ff5b4151660 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 msgr2=0x7ff5b415dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 0x7ff5b415dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 >> v1:172.21.15.177:6789/0 conn(0x7ff5b415e530 legacy=0x7ff5b40b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b40b3890 con 0x7ff5b4151660 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7ff5a4001670 con 0x7ff5b4151660 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 >> v1:172.21.15.114:6789/0 conn(0x7ff5b4151660 legacy=0x7ff5b4071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff594005bf0 0x7ff594006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 0x7ff594006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff594007070 0x7ff59400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40b3ee0 con 0x7ff594005bf0 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff594007830 con 0x7ff594007070 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59400e110 con 0x7ff5b40cf430 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.310+0000 7ff5b1e00640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff594005bf0 0x7ff594006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.311+0000 7ff5abe00640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 0x7ff594006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.311+0000 7ff5b1e00640 1 -- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 msgr2=0x7ff594006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.311+0000 7ff5b1e00640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 0x7ff594006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.311+0000 7ff5b1e00640 1 -- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff594007070 msgr2=0x7ff59400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.311+0000 7ff5b1e00640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff594007070 0x7ff59400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.311+0000 7ff5b1e00640 1 -- 172.21.15.114:0/2151171296 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594005730 con 0x7ff594005bf0 2024-03-06T07:35:45.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.311+0000 7ff5abe00640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5b40cf430 0x7ff594006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.312+0000 7ff5b1e00640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff594005bf0 0x7ff594006010 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c190 tx=0x7ff594007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ef36298ddfa72d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.312+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4007c60 con 0x7ff594005bf0 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.312+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a4030070 con 0x7ff594005bf0 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.312+0000 7ff5b1400640 1 -- 172.21.15.114:0/2151171296 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4005d70 con 0x7ff594005bf0 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.312+0000 7ff5b9200640 1 -- 172.21.15.114:0/2151171296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff594005bf0 msgr2=0x7ff594006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.312+0000 7ff5b9200640 1 --2- 172.21.15.114:0/2151171296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff594005bf0 0x7ff594006010 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c190 tx=0x7ff594007a40 comp rx=0 tx=0).stop 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.312+0000 7ff5b2800640 1 -- 172.21.15.114:0/2151171296 reap_dead start 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.313+0000 7ff5b9200640 1 -- 172.21.15.114:0/2151171296 shutdown_connections 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.313+0000 7ff5b9200640 1 -- 172.21.15.114:0/2151171296 >> 172.21.15.114:0/2151171296 conn(0x7ff5b40b1730 msgr2=0x7ff5b40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.313+0000 7ff5b9200640 1 -- 172.21.15.114:0/2151171296 shutdown_connections 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.313+0000 7ff5b9200640 1 -- 172.21.15.114:0/2151171296 wait complete. 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.314+0000 7ff5b9200640 1 Processor -- start 2024-03-06T07:35:45.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.314+0000 7ff5b9200640 1 -- start start 2024-03-06T07:35:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.315+0000 7ff5b9200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff594005bf0 0x7ff5b41e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.315+0000 7ff5b9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b41f7c30 0x7ff5b414ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.315+0000 7ff5b9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b414f350 0x7ff5b41592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.315+0000 7ff5b9200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40b4a80 con 0x7ff5b414f350 2024-03-06T07:35:45.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.315+0000 7ff5b9200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b4071d30 con 0x7ff5b41f7c30 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.315+0000 7ff5b9200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40b4c00 con 0x7ff594005bf0 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.315+0000 7ff5b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b414f350 0x7ff5b41592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b414f350 0x7ff5b41592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48262/0 (socket says 172.21.15.114:48262) 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 -- 172.21.15.114:0/1357881158 learned_addr learned my addr 172.21.15.114:0/1357881158 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 -- 172.21.15.114:0/1357881158 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff594005bf0 msgr2=0x7ff5b41e54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b1e00640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff594005bf0 0x7ff5b41e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff594005bf0 0x7ff5b41e54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 -- 172.21.15.114:0/1357881158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b41f7c30 msgr2=0x7ff5b414ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b41f7c30 0x7ff5b414ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 -- 172.21.15.114:0/1357881158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b41f8110 con 0x7ff5b414f350 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b2800640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b414f350 0x7ff5b41592d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7ff594005bc0 tx=0x7ff5ac00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5aa000640 1 -- 172.21.15.114:0/1357881158 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac017070 con 0x7ff5b414f350 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5aa000640 1 -- 172.21.15.114:0/1357881158 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ac005d60 con 0x7ff5b414f350 2024-03-06T07:35:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5aa000640 1 -- 172.21.15.114:0/1357881158 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00a850 con 0x7ff5b414f350 2024-03-06T07:35:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.316+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b41f83f0 con 0x7ff5b414f350 2024-03-06T07:35:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.317+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b41f89b0 con 0x7ff5b414f350 2024-03-06T07:35:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.318+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b404f330 con 0x7ff5b414f350 2024-03-06T07:35:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.319+0000 7ff5aa000640 1 -- 172.21.15.114:0/1357881158 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5ac01a070 con 0x7ff5b414f350 2024-03-06T07:35:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.319+0000 7ff5aa000640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff59006b4d0 0x7ff59006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.320+0000 7ff5aa000640 1 -- 172.21.15.114:0/1357881158 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5ac089390 con 0x7ff5b414f350 2024-03-06T07:35:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.320+0000 7ff5b1e00640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff59006b4d0 0x7ff59006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.320+0000 7ff5b1e00640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff59006b4d0 0x7ff59006d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c160 tx=0x7ff5a4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.324+0000 7ff5aa000640 1 -- 172.21.15.114:0/1357881158 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5ac0582a0 con 0x7ff5b414f350 2024-03-06T07:35:45.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.519+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b40cf0d0 con 0x7ff5b414f350 2024-03-06T07:35:45.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.520+0000 7ff5aa000640 1 -- 172.21.15.114:0/1357881158 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16740 (secure 0 0 0) 0x7ff5ac057ba0 con 0x7ff5b414f350 2024-03-06T07:35:45.523 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:45.523 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":124,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:35:44.543354+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:42.299180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:43.529461+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff59006b4d0 msgr2=0x7ff59006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff59006b4d0 0x7ff59006d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff5a400c160 tx=0x7ff5a4009030 comp rx=0 tx=0).stop 2024-03-06T07:35:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b414f350 msgr2=0x7ff5b41592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b414f350 0x7ff5b41592d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7ff594005bc0 tx=0x7ff5ac00cb90 comp rx=0 tx=0).stop 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 shutdown_connections 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff59006b4d0 0x7ff59006d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5b414f350 0x7ff5b41592d0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b41f7c30 0x7ff5b414ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 --2- 172.21.15.114:0/1357881158 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff594005bf0 0x7ff5b41e54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 >> 172.21.15.114:0/1357881158 conn(0x7ff5b40b1730 msgr2=0x7ff5b40af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 shutdown_connections 2024-03-06T07:35:45.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.523+0000 7ff5b9200640 1 -- 172.21.15.114:0/1357881158 wait complete. 2024-03-06T07:35:45.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:45.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.750+0000 7fed0ac00640 1 Processor -- start 2024-03-06T07:35:45.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.750+0000 7fed0ac00640 1 -- start start 2024-03-06T07:35:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed0ac00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0415e160 0x7fed04162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed0ac00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fed040b3ac0 0x7fed0406b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed0ac00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed04162b60 con 0x7fed040af820 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed0ac00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0406b0f0 con 0x7fed040b3ac0 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed0ac00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed040b5620 con 0x7fed0415e160 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed08c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fed040b3ac0 0x7fed0406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed08c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fed040b3ac0 0x7fed0406b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48266/0 (socket says 172.21.15.114:48266) 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed08c00640 1 -- 172.21.15.114:0/2977775841 learned_addr learned my addr 172.21.15.114:0/2977775841 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.752+0000 7fed03e00640 1 --2- 172.21.15.114:0/2977775841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0415e160 0x7fed04162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.753+0000 7fed08c00640 1 -- 172.21.15.114:0/2977775841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0415e160 msgr2=0x7fed04162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.753+0000 7fed08c00640 1 --2- 172.21.15.114:0/2977775841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0415e160 0x7fed04162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.753+0000 7fed08c00640 1 -- 172.21.15.114:0/2977775841 >> v1:172.21.15.114:6789/0 conn(0x7fed040af820 legacy=0x7fed040b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.753+0000 7fed08c00640 1 -- 172.21.15.114:0/2977775841 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0406f1f0 con 0x7fed040b3ac0 2024-03-06T07:35:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.753+0000 7fed02a00640 1 -- 172.21.15.114:0/2977775841 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318082242 0 0) 0x7fecf0001110 con 0x7fed040af820 2024-03-06T07:35:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.754+0000 7fed08c00640 1 --2- 172.21.15.114:0/2977775841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed040b3ac0 0x7fed0406b580 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009e50 tx=0x7fecfc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7280931ee4a0784 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.754+0000 7fed02a00640 1 -- 172.21.15.114:0/2977775841 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc02ca70 con 0x7fed040b3ac0 2024-03-06T07:35:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.754+0000 7fed02a00640 1 -- 172.21.15.114:0/2977775841 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc02cc10 con 0x7fed040b3ac0 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.754+0000 7fed02a00640 1 -- 172.21.15.114:0/2977775841 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc0343f0 con 0x7fed040b3ac0 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 -- 172.21.15.114:0/2977775841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed040b3ac0 msgr2=0x7fed0406b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 --2- 172.21.15.114:0/2977775841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed040b3ac0 0x7fed0406b580 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009e50 tx=0x7fecfc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 -- 172.21.15.114:0/2977775841 shutdown_connections 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 --2- 172.21.15.114:0/2977775841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed040b3ac0 0x7fed0406b580 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 --2- 172.21.15.114:0/2977775841 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed0415e160 0x7fed04162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 -- 172.21.15.114:0/2977775841 >> 172.21.15.114:0/2977775841 conn(0x7fed040d0340 msgr2=0x7fed040d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 -- 172.21.15.114:0/2977775841 shutdown_connections 2024-03-06T07:35:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.755+0000 7fed0ac00640 1 -- 172.21.15.114:0/2977775841 wait complete. 2024-03-06T07:35:45.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.757+0000 7fed0ac00640 1 Processor -- start 2024-03-06T07:35:45.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.757+0000 7fed0ac00640 1 -- start start 2024-03-06T07:35:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.758+0000 7fed0ac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed040af820 0x7fed04160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.758+0000 7fed0ac00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed040b3ac0 0x7fed040cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.758+0000 7fed0ac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0415e160 0x7fed0415e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.758+0000 7fed0ac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0406ef10 con 0x7fed0415e160 2024-03-06T07:35:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.758+0000 7fed0ac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0406ec30 con 0x7fed040af820 2024-03-06T07:35:45.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.758+0000 7fed0ac00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0406bac0 con 0x7fed040b3ac0 2024-03-06T07:35:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0415e160 0x7fed0415e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0415e160 0x7fed0415e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48280/0 (socket says 172.21.15.114:48280) 2024-03-06T07:35:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 -- 172.21.15.114:0/1543071066 learned_addr learned my addr 172.21.15.114:0/1543071066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 -- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed040b3ac0 msgr2=0x7fed040cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed03e00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed040af820 0x7fed04160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed03400640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed040b3ac0 0x7fed040cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed040b3ac0 0x7fed040cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 -- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed040af820 msgr2=0x7fed04160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed040af820 0x7fed04160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 -- 172.21.15.114:0/1543071066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed040cd580 con 0x7fed0415e160 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.759+0000 7fed08c00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0415e160 0x7fed0415e690 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fecfc036040 tx=0x7fecfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.760+0000 7fecfbe00640 1 -- 172.21.15.114:0/1543071066 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc043020 con 0x7fed0415e160 2024-03-06T07:35:45.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.760+0000 7fecfbe00640 1 -- 172.21.15.114:0/1543071066 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecfc03e450 con 0x7fed0415e160 2024-03-06T07:35:45.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.760+0000 7fecfbe00640 1 -- 172.21.15.114:0/1543071066 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc009510 con 0x7fed0415e160 2024-03-06T07:35:45.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.760+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0415ebd0 con 0x7fed0415e160 2024-03-06T07:35:45.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.760+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed041e7d10 con 0x7fed0415e160 2024-03-06T07:35:45.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.761+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed04074ae0 con 0x7fed0415e160 2024-03-06T07:35:45.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.762+0000 7fecfbe00640 1 -- 172.21.15.114:0/1543071066 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecfc046020 con 0x7fed0415e160 2024-03-06T07:35:45.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.763+0000 7fecfbe00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece406b4d0 0x7fece406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.763+0000 7fecfbe00640 1 -- 172.21.15.114:0/1543071066 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecfc0ac880 con 0x7fed0415e160 2024-03-06T07:35:45.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.763+0000 7fed03e00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece406b4d0 0x7fece406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.764+0000 7fed03e00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece406b4d0 0x7fece406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fecf4006fd0 tx=0x7fecf4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.768+0000 7fecfbe00640 1 -- 172.21.15.114:0/1543071066 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecfc04d030 con 0x7fed0415e160 2024-03-06T07:35:45.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.962+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed0404f330 con 0x7fed0415e160 2024-03-06T07:35:45.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.964+0000 7fecfbe00640 1 -- 172.21.15.114:0/1543071066 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16740 (secure 0 0 0) 0x7fed0415ebd0 con 0x7fed0415e160 2024-03-06T07:35:45.967 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:45.967 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":124,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:35:44.543354+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:42.299180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:43.529461+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece406b4d0 msgr2=0x7fece406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece406b4d0 0x7fece406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fecf4006fd0 tx=0x7fecf4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0415e160 msgr2=0x7fed0415e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0415e160 0x7fed0415e690 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fecfc036040 tx=0x7fecfc004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 shutdown_connections 2024-03-06T07:35:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fece406b4d0 0x7fece406d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed0415e160 0x7fed0415e690 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fed040b3ac0 0x7fed040cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 --2- 172.21.15.114:0/1543071066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed040af820 0x7fed04160af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.967+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 >> 172.21.15.114:0/1543071066 conn(0x7fed040d0340 msgr2=0x7fed0415c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.968+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 shutdown_connections 2024-03-06T07:35:45.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:45.968+0000 7fed0ac00640 1 -- 172.21.15.114:0/1543071066 wait complete. 2024-03-06T07:35:45.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:35:46.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.320+0000 7faa64400640 1 Processor -- start 2024-03-06T07:35:46.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.321+0000 7faa64400640 1 -- start start 2024-03-06T07:35:46.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.322+0000 7faa64400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.322+0000 7faa64400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faa5c0b3f70 0x7faa5c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.322+0000 7faa64400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c0b46c0 con 0x7faa5c15fff0 2024-03-06T07:35:46.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.322+0000 7faa64400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa5c0b49a0 con 0x7faa5c0b3f70 2024-03-06T07:35:46.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.322+0000 7faa64400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c072c90 con 0x7faa5c15fc10 2024-03-06T07:35:46.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.323+0000 7faa61000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7faa5c15fff0 0x7faa5c160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51828/0 (socket says 172.21.15.114:51828) 2024-03-06T07:35:46.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.323+0000 7faa61000640 1 -- 172.21.15.114:0/3673623552 learned_addr learned my addr 172.21.15.114:0/3673623552 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:46.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.323+0000 7faa62400640 1 --2- 172.21.15.114:0/3673623552 >> v2:172.21.15.114:3300/0 conn(0x7faa5c0b3f70 0x7faa5c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.323+0000 7faa61a00640 1 --2- 172.21.15.114:0/3673623552 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.323+0000 7faa5be00640 1 -- 172.21.15.114:0/3673623552 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508952440 0 0) 0x7faa5c0b46c0 con 0x7faa5c15fff0 2024-03-06T07:35:46.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.323+0000 7faa5be00640 1 -- 172.21.15.114:0/3673623552 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa44003520 con 0x7faa5c15fff0 2024-03-06T07:35:46.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.324+0000 7faa62400640 1 -- 172.21.15.114:0/3673623552 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 msgr2=0x7faa5c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:46.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.324+0000 7faa62400640 1 --2- 172.21.15.114:0/3673623552 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.324+0000 7faa62400640 1 -- 172.21.15.114:0/3673623552 >> v1:172.21.15.114:6789/0 conn(0x7faa5c15fff0 legacy=0x7faa5c160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.324+0000 7faa62400640 1 -- 172.21.15.114:0/3673623552 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c09dd80 con 0x7faa5c0b3f70 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.325+0000 7faa62400640 1 --2- 172.21.15.114:0/3673623552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0ce4a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7faa54009d90 tx=0x7faa5402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8275a4bf6951ad92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.325+0000 7faa5be00640 1 -- 172.21.15.114:0/3673623552 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54039070 con 0x7faa5c0b3f70 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.325+0000 7faa5be00640 1 -- 172.21.15.114:0/3673623552 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5402cc50 con 0x7faa5c0b3f70 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.325+0000 7faa5be00640 1 -- 172.21.15.114:0/3673623552 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54034440 con 0x7faa5c0b3f70 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.325+0000 7faa64400640 1 -- 172.21.15.114:0/3673623552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c0b3f70 msgr2=0x7faa5c0ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.325+0000 7faa64400640 1 --2- 172.21.15.114:0/3673623552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0ce4a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7faa54009d90 tx=0x7faa5402bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.326+0000 7faa64400640 1 -- 172.21.15.114:0/3673623552 shutdown_connections 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.326+0000 7faa64400640 1 --2- 172.21.15.114:0/3673623552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0ce4a0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.326+0000 7faa64400640 1 --2- 172.21.15.114:0/3673623552 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.326+0000 7faa64400640 1 -- 172.21.15.114:0/3673623552 >> 172.21.15.114:0/3673623552 conn(0x7faa5c0b1940 msgr2=0x7faa5c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.326+0000 7faa64400640 1 -- 172.21.15.114:0/3673623552 shutdown_connections 2024-03-06T07:35:46.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.326+0000 7faa64400640 1 -- 172.21.15.114:0/3673623552 wait complete. 2024-03-06T07:35:46.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.327+0000 7faa64400640 1 Processor -- start 2024-03-06T07:35:46.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.327+0000 7faa64400640 1 -- start start 2024-03-06T07:35:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.328+0000 7faa64400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.328+0000 7faa64400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c1483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.328+0000 7faa64400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c15fff0 0x7faa5c0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.328+0000 7faa64400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c15c460 con 0x7faa5c15fff0 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.328+0000 7faa64400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c15c2e0 con 0x7faa5c0b3f70 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.328+0000 7faa64400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c0ce9e0 con 0x7faa5c15fc10 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c15fff0 0x7faa5c0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c15fff0 0x7faa5c0b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48290/0 (socket says 172.21.15.114:48290) 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 -- 172.21.15.114:0/1165089865 learned_addr learned my addr 172.21.15.114:0/1165089865 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa61a00640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa61000640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c1483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 -- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 msgr2=0x7faa5c1483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c1483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 -- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa5c0b3f70 msgr2=0x7faa5c0cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa62400640 1 -- 172.21.15.114:0/1165089865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c09d7d0 con 0x7faa5c15fff0 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa61a00640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.329+0000 7faa61000640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c1483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.330+0000 7faa62400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c15fff0 0x7faa5c0b2f60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7faa5402c4b0 tx=0x7faa54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.330+0000 7faa59600640 1 -- 172.21.15.114:0/1165089865 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54039050 con 0x7faa5c15fff0 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.330+0000 7faa59600640 1 -- 172.21.15.114:0/1165089865 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa54045070 con 0x7faa5c15fff0 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.330+0000 7faa59600640 1 -- 172.21.15.114:0/1165089865 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54009450 con 0x7faa5c15fff0 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.330+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c09d210 con 0x7faa5c15fff0 2024-03-06T07:35:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.330+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5c0c9060 con 0x7faa5c15fff0 2024-03-06T07:35:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.331+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5c04f330 con 0x7faa5c15fff0 2024-03-06T07:35:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.332+0000 7faa59600640 1 -- 172.21.15.114:0/1165089865 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa5403e610 con 0x7faa5c15fff0 2024-03-06T07:35:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.333+0000 7faa59600640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa3c06b4d0 0x7faa3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.333+0000 7faa59600640 1 -- 172.21.15.114:0/1165089865 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa540acce0 con 0x7faa5c15fff0 2024-03-06T07:35:46.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.333+0000 7faa61a00640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa3c06b4d0 0x7faa3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.334+0000 7faa61a00640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa3c06b4d0 0x7faa3c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faa50009ac0 tx=0x7faa50008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.337+0000 7faa59600640 1 -- 172.21.15.114:0/1165089865 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa5407baf0 con 0x7faa5c15fff0 2024-03-06T07:35:46.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:46.575+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7faa5c06ef30 con 0x7faa5c15fff0 2024-03-06T07:35:46.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:46.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:47.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.120+0000 7faa59600640 1 -- 172.21.15.114:0/1165089865 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v95) v1 ==== 118+0+0 (secure 0 0 0) 0x7faa5407b3f0 con 0x7faa5c15fff0 2024-03-06T07:35:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.124+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa3c06b4d0 msgr2=0x7faa3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.124+0000 7faa64400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa3c06b4d0 0x7faa3c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faa50009ac0 tx=0x7faa50008040 comp rx=0 tx=0).stop 2024-03-06T07:35:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.124+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c15fff0 msgr2=0x7faa5c0b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.124+0000 7faa64400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c15fff0 0x7faa5c0b2f60 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7faa5402c4b0 tx=0x7faa54004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 shutdown_connections 2024-03-06T07:35:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faa3c06b4d0 0x7faa3c06d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa5c15fff0 0x7faa5c0b2f60 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa5c15fc10 0x7faa5c1483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 --2- 172.21.15.114:0/1165089865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa5c0b3f70 0x7faa5c0cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 >> 172.21.15.114:0/1165089865 conn(0x7faa5c0b1940 msgr2=0x7faa5c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 shutdown_connections 2024-03-06T07:35:47.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.125+0000 7faa64400640 1 -- 172.21.15.114:0/1165089865 wait complete. 2024-03-06T07:35:47.151 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:35:47.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:35:47.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.356+0000 7ff0e5e00640 1 Processor -- start 2024-03-06T07:35:47.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.356+0000 7ff0e5e00640 1 -- start start 2024-03-06T07:35:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.357+0000 7ff0e5e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.357+0000 7ff0e5e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff0e00d25b0 0x7ff0e00c7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.357+0000 7ff0e5e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e00d13e0 con 0x7ff0e00cfaf0 2024-03-06T07:35:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.357+0000 7ff0e5e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e00cf120 con 0x7ff0e00d25b0 2024-03-06T07:35:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.357+0000 7ff0e5e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00cead0 con 0x7ff0e015e130 2024-03-06T07:35:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.358+0000 7ff0e4a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.358+0000 7ff0e5400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff0e00d25b0 0x7ff0e00c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.358+0000 7ff0e4a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:34128/0 (socket says 172.21.15.114:34128) 2024-03-06T07:35:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.358+0000 7ff0e4a00640 1 -- 172.21.15.114:0/1792833368 learned_addr learned my addr 172.21.15.114:0/1792833368 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.359+0000 7ff0e5400640 1 -- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 msgr2=0x7ff0e00cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.359+0000 7ff0e5400640 1 --2- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.359+0000 7ff0e5400640 1 -- 172.21.15.114:0/1792833368 >> v1:172.21.15.177:6789/0 conn(0x7ff0e00cfaf0 legacy=0x7ff0e00d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.359+0000 7ff0e5400640 1 -- 172.21.15.114:0/1792833368 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e00cedb0 con 0x7ff0e00d25b0 2024-03-06T07:35:47.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.359+0000 7ff0e4a00640 1 --2- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:47.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.360+0000 7ff0e5400640 1 --2- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00c7970 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80097a0 tx=0x7ff0d802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46aa28d28014d36e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.360+0000 7ff0df400640 1 -- 172.21.15.114:0/1792833368 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8035020 con 0x7ff0e00d25b0 2024-03-06T07:35:47.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.360+0000 7ff0df400640 1 -- 172.21.15.114:0/1792833368 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8031810 con 0x7ff0e00d25b0 2024-03-06T07:35:47.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.360+0000 7ff0df400640 1 -- 172.21.15.114:0/1792833368 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8031be0 con 0x7ff0e00d25b0 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 -- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00d25b0 msgr2=0x7ff0e00c7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00c7970 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80097a0 tx=0x7ff0d802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 -- 172.21.15.114:0/1792833368 shutdown_connections 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00c7970 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/1792833368 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 -- 172.21.15.114:0/1792833368 >> 172.21.15.114:0/1792833368 conn(0x7ff0e009d070 msgr2=0x7ff0e00d3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 -- 172.21.15.114:0/1792833368 shutdown_connections 2024-03-06T07:35:47.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.361+0000 7ff0e5e00640 1 -- 172.21.15.114:0/1792833368 wait complete. 2024-03-06T07:35:47.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.363+0000 7ff0e5e00640 1 Processor -- start 2024-03-06T07:35:47.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.363+0000 7ff0e5e00640 1 -- start start 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.364+0000 7ff0e5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00cfaf0 0x7ff0e00cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00b8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e5e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00b9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e5e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00c8030 con 0x7ff0e00d25b0 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e5e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00c7eb0 con 0x7ff0e00cfaf0 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e5e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00c81b0 con 0x7ff0e015e130 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00b8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00b8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48294/0 (socket says 172.21.15.114:48294) 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 -- 172.21.15.114:0/3012555799 learned_addr learned my addr 172.21.15.114:0/3012555799 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e4a00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00cfaf0 0x7ff0e00cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e5400640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 -- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 msgr2=0x7ff0e00b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 -- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00cfaf0 msgr2=0x7ff0e00cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00cfaf0 0x7ff0e00cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0dfe00640 1 -- 172.21.15.114:0/3012555799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e00b7310 con 0x7ff0e00d25b0 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e5400640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.365+0000 7ff0e4a00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00cfaf0 0x7ff0e00cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.366+0000 7ff0dfe00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00b8f10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00c470 tx=0x7ff0cc00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.366+0000 7ff0dcc00640 1 -- 172.21.15.114:0/3012555799 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc0052d0 con 0x7ff0e00d25b0 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.366+0000 7ff0dcc00640 1 -- 172.21.15.114:0/3012555799 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc005490 con 0x7ff0e00d25b0 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.366+0000 7ff0dcc00640 1 -- 172.21.15.114:0/3012555799 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc007260 con 0x7ff0e00d25b0 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.366+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e00b75b0 con 0x7ff0e00d25b0 2024-03-06T07:35:47.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.366+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e01e88e0 con 0x7ff0e00d25b0 2024-03-06T07:35:47.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.367+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e004f330 con 0x7ff0e00d25b0 2024-03-06T07:35:47.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.368+0000 7ff0dcc00640 1 -- 172.21.15.114:0/3012555799 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0cc007400 con 0x7ff0e00d25b0 2024-03-06T07:35:47.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.369+0000 7ff0dcc00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff0c006b4d0 0x7ff0c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.369+0000 7ff0dcc00640 1 -- 172.21.15.114:0/3012555799 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0cc089690 con 0x7ff0e00d25b0 2024-03-06T07:35:47.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.369+0000 7ff0e4a00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff0c006b4d0 0x7ff0c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.370+0000 7ff0e4a00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff0c006b4d0 0x7ff0c006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4009240 tx=0x7ff0d400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.377+0000 7ff0dcc00640 1 -- 172.21.15.114:0/3012555799 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0cc058500 con 0x7ff0e00d25b0 2024-03-06T07:35:47.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:47.608+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff0e009d690 con 0x7ff0e00d25b0 2024-03-06T07:35:48.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.131+0000 7ff0dcc00640 1 -- 172.21.15.114:0/3012555799 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v96) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff0cc057e00 con 0x7ff0e00d25b0 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff0c006b4d0 msgr2=0x7ff0c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff0c006b4d0 0x7ff0c006d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4009240 tx=0x7ff0d400b040 comp rx=0 tx=0).stop 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e00d25b0 msgr2=0x7ff0e00b8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00b8f10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00c470 tx=0x7ff0cc00c930 comp rx=0 tx=0).stop 2024-03-06T07:35:48.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 shutdown_connections 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff0e015e130 0x7ff0e00b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff0c006b4d0 0x7ff0c006d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e00d25b0 0x7ff0e00b8f10 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 --2- 172.21.15.114:0/3012555799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0e00cfaf0 0x7ff0e00cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 >> 172.21.15.114:0/3012555799 conn(0x7ff0e009d070 msgr2=0x7ff0e006c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 shutdown_connections 2024-03-06T07:35:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.135+0000 7ff0e5e00640 1 -- 172.21.15.114:0/3012555799 wait complete. 2024-03-06T07:35:48.165 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:35:48.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:48.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.366+0000 7f0861200640 1 Processor -- start 2024-03-06T07:35:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.367+0000 7f0861200640 1 -- start start 2024-03-06T07:35:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.368+0000 7f0861200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f085c15f9d0 0x7f085c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.368+0000 7f0861200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f085c070b10 0x7f085c070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.368+0000 7f0861200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f085c06cef0 con 0x7f085c144c00 2024-03-06T07:35:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.368+0000 7f0861200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085c06d1d0 con 0x7f085c070b10 2024-03-06T07:35:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.368+0000 7f0861200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085c06d4b0 con 0x7f085c15f9d0 2024-03-06T07:35:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.368+0000 7f085a800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f085c070b10 0x7f085c070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f085a800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f085c070b10 0x7f085c070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39632/0 (socket says 172.21.15.114:39632) 2024-03-06T07:35:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f085a800640 1 -- 172.21.15.114:0/1198168303 learned_addr learned my addr 172.21.15.114:0/1198168303 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f085a800640 1 -- 172.21.15.114:0/1198168303 >> v2:172.21.15.177:3300/0 conn(0x7f085c15f9d0 msgr2=0x7f085c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f0853e00640 1 --2- 172.21.15.114:0/1198168303 >> v2:172.21.15.177:3300/0 conn(0x7f085c15f9d0 0x7f085c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f085a800640 1 --2- 172.21.15.114:0/1198168303 >> v2:172.21.15.177:3300/0 conn(0x7f085c15f9d0 0x7f085c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f085a800640 1 -- 172.21.15.114:0/1198168303 >> v1:172.21.15.177:6789/0 conn(0x7f085c144c00 legacy=0x7f085c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f085a800640 1 -- 172.21.15.114:0/1198168303 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085c06d790 con 0x7f085c070b10 2024-03-06T07:35:48.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.369+0000 7f0853e00640 1 --2- 172.21.15.114:0/1198168303 >> v2:172.21.15.177:3300/0 conn(0x7f085c15f9d0 0x7f085c15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:48.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.370+0000 7f085a800640 1 --2- 172.21.15.114:0/1198168303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c070b10 0x7f085c070f20 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f085c0b26f0 tx=0x7f085402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8451b1a68d87e0e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.370+0000 7f0859400640 1 -- 172.21.15.114:0/1198168303 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085402ca80 con 0x7f085c070b10 2024-03-06T07:35:48.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0859400640 1 -- 172.21.15.114:0/1198168303 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085402cc20 con 0x7f085c070b10 2024-03-06T07:35:48.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0859400640 1 -- 172.21.15.114:0/1198168303 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08540343f0 con 0x7f085c070b10 2024-03-06T07:35:48.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 -- 172.21.15.114:0/1198168303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c070b10 msgr2=0x7f085c070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 --2- 172.21.15.114:0/1198168303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c070b10 0x7f085c070f20 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f085c0b26f0 tx=0x7f085402bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 -- 172.21.15.114:0/1198168303 shutdown_connections 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 --2- 172.21.15.114:0/1198168303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c070b10 0x7f085c070f20 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 --2- 172.21.15.114:0/1198168303 >> v2:172.21.15.177:3300/0 conn(0x7f085c15f9d0 0x7f085c15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 -- 172.21.15.114:0/1198168303 >> 172.21.15.114:0/1198168303 conn(0x7f085c0b1730 msgr2=0x7f085c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 -- 172.21.15.114:0/1198168303 shutdown_connections 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.371+0000 7f0861200640 1 -- 172.21.15.114:0/1198168303 wait complete. 2024-03-06T07:35:48.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.373+0000 7f0861200640 1 Processor -- start 2024-03-06T07:35:48.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.373+0000 7f0861200640 1 -- start start 2024-03-06T07:35:48.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f0861200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f085c070b10 0x7f085c13cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f0861200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f085c144c00 0x7f085c139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f0861200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c15f9d0 0x7f085c140770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f0861200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c06ccd0 con 0x7f085c15f9d0 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f0861200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c06cb50 con 0x7f085c070b10 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f0861200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085c06df90 con 0x7f085c144c00 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f085a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c15f9d0 0x7f085c140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f085a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c15f9d0 0x7f085c140770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39648/0 (socket says 172.21.15.114:39648) 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.374+0000 7f085a800640 1 -- 172.21.15.114:0/2262839355 learned_addr learned my addr 172.21.15.114:0/2262839355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f085a800640 1 -- 172.21.15.114:0/2262839355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f085c144c00 msgr2=0x7f085c139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f085a800640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f085c144c00 0x7f085c139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f085a800640 1 -- 172.21.15.114:0/2262839355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f085c070b10 msgr2=0x7f085c13cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f085a800640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f085c070b10 0x7f085c13cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f085a800640 1 -- 172.21.15.114:0/2262839355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085c13fef0 con 0x7f085c15f9d0 2024-03-06T07:35:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f085a800640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c15f9d0 0x7f085c140770 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f0854009f40 tx=0x7f0854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f0852000640 1 -- 172.21.15.114:0/2262839355 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854043020 con 0x7f085c15f9d0 2024-03-06T07:35:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f0852000640 1 -- 172.21.15.114:0/2262839355 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085403e450 con 0x7f085c15f9d0 2024-03-06T07:35:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f0852000640 1 -- 172.21.15.114:0/2262839355 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854009510 con 0x7f085c15f9d0 2024-03-06T07:35:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.375+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085c1401d0 con 0x7f085c15f9d0 2024-03-06T07:35:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.376+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085c1399b0 con 0x7f085c15f9d0 2024-03-06T07:35:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.377+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085c04f330 con 0x7f085c15f9d0 2024-03-06T07:35:48.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.378+0000 7f0852000640 1 -- 172.21.15.114:0/2262839355 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0854046020 con 0x7f085c15f9d0 2024-03-06T07:35:48.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.378+0000 7f0852000640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f083806b4d0 0x7f083806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.379+0000 7f0852000640 1 -- 172.21.15.114:0/2262839355 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08540ace70 con 0x7f085c15f9d0 2024-03-06T07:35:48.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.379+0000 7f0859e00640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f083806b4d0 0x7f083806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.380+0000 7f0859e00640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f083806b4d0 0x7f083806d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f084c003cc0 tx=0x7f084c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.384+0000 7f0852000640 1 -- 172.21.15.114:0/2262839355 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f085404d020 con 0x7f085c15f9d0 2024-03-06T07:35:48.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.621+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f085c072890 con 0x7f085c15f9d0 2024-03-06T07:35:48.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.622+0000 7f0852000640 1 -- 172.21.15.114:0/2262839355 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f085407bc50 con 0x7f085c15f9d0 2024-03-06T07:35:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:48.624 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1958627905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1958627905},{"type":"v1","addr":"172.21.15.177:6833","nonce":1958627905}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":5765,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/7846601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":7846601},{"type":"v1","addr":"172.21.15.177:6835","nonce":7846601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":8328,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:47.609493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:35:48.624 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 96 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f083806b4d0 msgr2=0x7f083806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f083806b4d0 0x7f083806d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f084c003cc0 tx=0x7f084c0079b0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c15f9d0 msgr2=0x7f085c140770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c15f9d0 0x7f085c140770 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f0854009f40 tx=0x7f0854004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 shutdown_connections 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f083806b4d0 0x7f083806d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f085c15f9d0 0x7f085c140770 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f085c144c00 0x7f085c139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 --2- 172.21.15.114:0/2262839355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f085c070b10 0x7f085c13cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 >> 172.21.15.114:0/2262839355 conn(0x7f085c0b1730 msgr2=0x7f085c0aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 shutdown_connections 2024-03-06T07:35:48.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.625+0000 7f0861200640 1 -- 172.21.15.114:0/2262839355 wait complete. 2024-03-06T07:35:48.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:48.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.842+0000 7f36d0000640 1 Processor -- start 2024-03-06T07:35:48.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.842+0000 7f36d0000640 1 -- start start 2024-03-06T07:35:48.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.843+0000 7f36d0000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f36c8144c00 0x7f36c806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.843+0000 7f36d0000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f36c815fe60 0x7f36c80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36d0000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c806ee70 con 0x7f36c815f9d0 2024-03-06T07:35:48.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36d0000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c806cef0 con 0x7f36c815fe60 2024-03-06T07:35:48.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36d0000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c806d1d0 con 0x7f36c8144c00 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36ce000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f36c815fe60 0x7f36c80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36ce000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f36c815fe60 0x7f36c80ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39654/0 (socket says 172.21.15.114:39654) 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36ce000640 1 -- 172.21.15.114:0/754422599 learned_addr learned my addr 172.21.15.114:0/754422599 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36cd600640 1 --2- 172.21.15.114:0/754422599 >> v2:172.21.15.177:3300/0 conn(0x7f36c8144c00 0x7f36c806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36ce000640 1 -- 172.21.15.114:0/754422599 >> v2:172.21.15.177:3300/0 conn(0x7f36c8144c00 msgr2=0x7f36c806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36ce000640 1 --2- 172.21.15.114:0/754422599 >> v2:172.21.15.177:3300/0 conn(0x7f36c8144c00 0x7f36c806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36ce000640 1 -- 172.21.15.114:0/754422599 >> v1:172.21.15.177:6789/0 conn(0x7f36c815f9d0 legacy=0x7f36c80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.844+0000 7f36ce000640 1 -- 172.21.15.114:0/754422599 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c806d4b0 con 0x7f36c815fe60 2024-03-06T07:35:48.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.845+0000 7f36ce000640 1 --2- 172.21.15.114:0/754422599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c815fe60 0x7f36c80ce7b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f36c80b25f0 tx=0x7f36c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aaf90e04de4887c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36c7e00640 1 -- 172.21.15.114:0/754422599 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c002ca80 con 0x7f36c815fe60 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36c7e00640 1 -- 172.21.15.114:0/754422599 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c002cc20 con 0x7f36c815fe60 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36c7e00640 1 -- 172.21.15.114:0/754422599 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c00343f0 con 0x7f36c815fe60 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36d0000640 1 -- 172.21.15.114:0/754422599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c815fe60 msgr2=0x7f36c80ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36d0000640 1 --2- 172.21.15.114:0/754422599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c815fe60 0x7f36c80ce7b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f36c80b25f0 tx=0x7f36c002bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36d0000640 1 -- 172.21.15.114:0/754422599 shutdown_connections 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36d0000640 1 --2- 172.21.15.114:0/754422599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c815fe60 0x7f36c80ce7b0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36d0000640 1 --2- 172.21.15.114:0/754422599 >> v2:172.21.15.177:3300/0 conn(0x7f36c8144c00 0x7f36c806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.846+0000 7f36d0000640 1 -- 172.21.15.114:0/754422599 >> 172.21.15.114:0/754422599 conn(0x7f36c80b1730 msgr2=0x7f36c80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.847+0000 7f36d0000640 1 -- 172.21.15.114:0/754422599 shutdown_connections 2024-03-06T07:35:48.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.847+0000 7f36d0000640 1 -- 172.21.15.114:0/754422599 wait complete. 2024-03-06T07:35:48.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.848+0000 7f36d0000640 1 Processor -- start 2024-03-06T07:35:48.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.848+0000 7f36d0000640 1 -- start start 2024-03-06T07:35:48.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.849+0000 7f36d0000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c8144c00 0x7f36c8133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.849+0000 7f36d0000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36c815f9d0 0x7f36c80b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.849+0000 7f36d0000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36c815fe60 0x7f36c81390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.849+0000 7f36d0000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8160730 con 0x7f36c8144c00 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.849+0000 7f36d0000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c8160270 con 0x7f36c815fe60 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.849+0000 7f36d0000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c806cb50 con 0x7f36c815f9d0 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c8144c00 0x7f36c8133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c8144c00 0x7f36c8133430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39658/0 (socket says 172.21.15.114:39658) 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 -- 172.21.15.114:0/3172689572 learned_addr learned my addr 172.21.15.114:0/3172689572 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36ce000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36c815fe60 0x7f36c81390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36ccc00640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36c815f9d0 0x7f36c80b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 -- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36c815f9d0 msgr2=0x7f36c80b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36c815f9d0 0x7f36c80b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 -- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36c815fe60 msgr2=0x7f36c81390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36c815fe60 0x7f36c81390f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36cd600640 1 -- 172.21.15.114:0/3172689572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80b2720 con 0x7f36c8144c00 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36ccc00640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36c815f9d0 0x7f36c80b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.850+0000 7f36ce000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36c815fe60 0x7f36c81390f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.851+0000 7f36cd600640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c8144c00 0x7f36c8133430 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c7f0 tx=0x7f36bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.851+0000 7f36c5600640 1 -- 172.21.15.114:0/3172689572 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc015070 con 0x7f36c8144c00 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.851+0000 7f36c5600640 1 -- 172.21.15.114:0/3172689572 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc004240 con 0x7f36c8144c00 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.851+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c80b2a00 con 0x7f36c8144c00 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.851+0000 7f36c5600640 1 -- 172.21.15.114:0/3172689572 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc007620 con 0x7f36c8144c00 2024-03-06T07:35:48.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.851+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c80b2fc0 con 0x7f36c8144c00 2024-03-06T07:35:48.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.852+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c804f330 con 0x7f36c8144c00 2024-03-06T07:35:48.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.854+0000 7f36c5600640 1 -- 172.21.15.114:0/3172689572 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36bc0079d0 con 0x7f36c8144c00 2024-03-06T07:35:48.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.854+0000 7f36c5600640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36a806b500 0x7f36a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.855+0000 7f36c5600640 1 -- 172.21.15.114:0/3172689572 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36bc08a3a0 con 0x7f36c8144c00 2024-03-06T07:35:48.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.855+0000 7f36ccc00640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36a806b500 0x7f36a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.855+0000 7f36ccc00640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36a806b500 0x7f36a806d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f36c81327d0 tx=0x7f36b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:48.863+0000 7f36c5600640 1 -- 172.21.15.114:0/3172689572 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36bc059180 con 0x7f36c8144c00 2024-03-06T07:35:49.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.059+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36c8072890 con 0x7f36c8144c00 2024-03-06T07:35:49.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.061+0000 7f36c5600640 1 -- 172.21.15.114:0/3172689572 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16727 (secure 0 0 0) 0x7f36bc058a80 con 0x7f36c8144c00 2024-03-06T07:35:49.063 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:49.064 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":125,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:35:46.091695+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:42.299180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:43.529461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36a806b500 msgr2=0x7f36a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36a806b500 0x7f36a806d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f36c81327d0 tx=0x7f36b4009040 comp rx=0 tx=0).stop 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c8144c00 msgr2=0x7f36c8133430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c8144c00 0x7f36c8133430 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c7f0 tx=0x7f36bc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 shutdown_connections 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f36c815fe60 0x7f36c81390f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f36c815f9d0 0x7f36c80b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f36a806b500 0x7f36a806d9c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 --2- 172.21.15.114:0/3172689572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36c8144c00 0x7f36c8133430 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 >> 172.21.15.114:0/3172689572 conn(0x7f36c80b1730 msgr2=0x7f36c8145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 shutdown_connections 2024-03-06T07:35:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.064+0000 7f36d0000640 1 -- 172.21.15.114:0/3172689572 wait complete. 2024-03-06T07:35:49.083 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:35:49.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.280+0000 7f7f4f200640 1 Processor -- start 2024-03-06T07:35:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.281+0000 7f7f4f200640 1 -- start start 2024-03-06T07:35:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.282+0000 7f7f4f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500d2360 0x7f7f500d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.282+0000 7f7f4f200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7f500b3ec0 0x7f7f500b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.282+0000 7f7f4f200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f50161960 con 0x7f7f50161580 2024-03-06T07:35:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.282+0000 7f7f4f200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f500d2ce0 con 0x7f7f500b3ec0 2024-03-06T07:35:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.282+0000 7f7f4f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500cf6f0 con 0x7f7f500d2360 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f45400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500d2360 0x7f7f500d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f45400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500d2360 0x7f7f500d27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:47786/0 (socket says 172.21.15.114:47786) 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f45400640 1 -- 172.21.15.114:0/243347349 learned_addr learned my addr 172.21.15.114:0/243347349 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f4e800640 1 --2- 172.21.15.114:0/243347349 >> v2:172.21.15.177:3300/0 conn(0x7f7f500b3ec0 0x7f7f500b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f45400640 1 -- 172.21.15.114:0/243347349 >> v2:172.21.15.177:3300/0 conn(0x7f7f500b3ec0 msgr2=0x7f7f500b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f45400640 1 --2- 172.21.15.114:0/243347349 >> v2:172.21.15.177:3300/0 conn(0x7f7f500b3ec0 0x7f7f500b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f45400640 1 -- 172.21.15.114:0/243347349 >> v1:172.21.15.177:6789/0 conn(0x7f7f50161580 legacy=0x7f7f500d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.283+0000 7f7f45400640 1 -- 172.21.15.114:0/243347349 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500cf9d0 con 0x7f7f500d2360 2024-03-06T07:35:49.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.285+0000 7f7f45400640 1 --2- 172.21.15.114:0/243347349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500d2360 0x7f7f500d27a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009870 tx=0x7f7f3802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd6df733074b5d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.285+0000 7f7f4d400640 1 -- 172.21.15.114:0/243347349 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38034070 con 0x7f7f500d2360 2024-03-06T07:35:49.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.285+0000 7f7f4d400640 1 -- 172.21.15.114:0/243347349 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f38030840 con 0x7f7f500d2360 2024-03-06T07:35:49.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.285+0000 7f7f4d400640 1 -- 172.21.15.114:0/243347349 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38030c10 con 0x7f7f500d2360 2024-03-06T07:35:49.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 -- 172.21.15.114:0/243347349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500d2360 msgr2=0x7f7f500d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 --2- 172.21.15.114:0/243347349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500d2360 0x7f7f500d27a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009870 tx=0x7f7f3802b980 comp rx=0 tx=0).stop 2024-03-06T07:35:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 -- 172.21.15.114:0/243347349 shutdown_connections 2024-03-06T07:35:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 --2- 172.21.15.114:0/243347349 >> v2:172.21.15.177:3300/0 conn(0x7f7f500b3ec0 0x7f7f500b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 --2- 172.21.15.114:0/243347349 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500d2360 0x7f7f500d27a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 -- 172.21.15.114:0/243347349 >> 172.21.15.114:0/243347349 conn(0x7f7f500d3d80 msgr2=0x7f7f500d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 -- 172.21.15.114:0/243347349 shutdown_connections 2024-03-06T07:35:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.286+0000 7f7f4f200640 1 -- 172.21.15.114:0/243347349 wait complete. 2024-03-06T07:35:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.288+0000 7f7f4f200640 1 Processor -- start 2024-03-06T07:35:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.288+0000 7f7f4f200640 1 -- start start 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500b3ec0 0x7f7f5014f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f500d2360 0x7f7f500b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50161580 0x7f7f5015fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500b39a0 con 0x7f7f50161580 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500b36c0 con 0x7f7f500d2360 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500cf3a0 con 0x7f7f500b3ec0 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50161580 0x7f7f5015fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50161580 0x7f7f5015fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39664/0 (socket says 172.21.15.114:39664) 2024-03-06T07:35:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4e800640 1 -- 172.21.15.114:0/3248756785 learned_addr learned my addr 172.21.15.114:0/3248756785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f45400640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f500d2360 0x7f7f500b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.289+0000 7f7f4de00640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500b3ec0 0x7f7f5014f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f4e800640 1 -- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500b3ec0 msgr2=0x7f7f5014f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f4e800640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500b3ec0 0x7f7f5014f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f4e800640 1 -- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f500d2360 msgr2=0x7f7f500b7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f4e800640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f500d2360 0x7f7f500b7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f4e800640 1 -- 172.21.15.114:0/3248756785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5014f710 con 0x7f7f50161580 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f4e800640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50161580 0x7f7f5015fbe0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800d600 tx=0x7f7f4800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f46a00640 1 -- 172.21.15.114:0/3248756785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f480125f0 con 0x7f7f50161580 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f46a00640 1 -- 172.21.15.114:0/3248756785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f48019070 con 0x7f7f50161580 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f500b6b90 con 0x7f7f50161580 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.290+0000 7f7f46a00640 1 -- 172.21.15.114:0/3248756785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f480129f0 con 0x7f7f50161580 2024-03-06T07:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.291+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f500b7150 con 0x7f7f50161580 2024-03-06T07:35:49.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.292+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5004f330 con 0x7f7f50161580 2024-03-06T07:35:49.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.292+0000 7f7f46a00640 1 -- 172.21.15.114:0/3248756785 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f48012b90 con 0x7f7f50161580 2024-03-06T07:35:49.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.293+0000 7f7f46a00640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.294+0000 7f7f46a00640 1 -- 172.21.15.114:0/3248756785 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f4808a610 con 0x7f7f50161580 2024-03-06T07:35:49.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.294+0000 7f7f4de00640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.295+0000 7f7f4de00640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7f40001d40 tx=0x7f7f40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.301+0000 7f7f46a00640 1 -- 172.21.15.114:0/3248756785 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f5014f710 con 0x7f7f50161580 2024-03-06T07:35:49.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.495+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f500b6940 con 0x7f7f2c06b500 2024-03-06T07:35:49.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.513+0000 7f7f46a00640 1 -- 172.21.15.114:0/3248756785 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252480 (secure 0 0 0) 0x7f7f500b6940 con 0x7f7f2c06b500 2024-03-06T07:35:49.519 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:35:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f2c06b500 msgr2=0x7f7f2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7f40001d40 tx=0x7f7f40009040 comp rx=0 tx=0).stop 2024-03-06T07:35:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50161580 msgr2=0x7f7f5015fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50161580 0x7f7f5015fbe0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800d600 tx=0x7f7f4800dac0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 shutdown_connections 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f2c06b500 0x7f7f2c06d9c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50161580 0x7f7f5015fbe0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f500d2360 0x7f7f500b7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 --2- 172.21.15.114:0/3248756785 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f500b3ec0 0x7f7f5014f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 >> 172.21.15.114:0/3248756785 conn(0x7f7f500d3d80 msgr2=0x7f7f500b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 shutdown_connections 2024-03-06T07:35:49.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.519+0000 7f7f4f200640 1 -- 172.21.15.114:0/3248756785 wait complete. 2024-03-06T07:35:49.539 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-03-06T07:35:49.245571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":296160,"kb_used_data":13416,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754678560,"statfs":{"total":773094113280,"available":772790845440,"internally_reserved":0,"allocated":13737984,"data_stored":6635612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":289394227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001827"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113335+0000","last_change":"2024-03-06T07:35:45.106118+0000","last_active":"2024-03-06T07:35:46.113335+0000","last_peered":"2024-03-06T07:35:46.113335+0000","last_clean":"2024-03-06T07:35:46.113335+0000","last_became_active":"2024-03-06T07:35:45.105818+0000","last_became_peered":"2024-03-06T07:35:45.105818+0000","last_unstale":"2024-03-06T07:35:46.113335+0000","last_undegraded":"2024-03-06T07:35:46.113335+0000","last_fullsized":"2024-03-06T07:35:46.113335+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:17.983404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487338+0000","last_change":"2024-03-06T07:35:45.103855+0000","last_active":"2024-03-06T07:35:46.487338+0000","last_peered":"2024-03-06T07:35:46.487338+0000","last_clean":"2024-03-06T07:35:46.487338+0000","last_became_active":"2024-03-06T07:35:45.103134+0000","last_became_peered":"2024-03-06T07:35:45.103134+0000","last_unstale":"2024-03-06T07:35:46.487338+0000","last_undegraded":"2024-03-06T07:35:46.487338+0000","last_fullsized":"2024-03-06T07:35:46.487338+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:43.923044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491796+0000","last_change":"2024-03-06T07:35:45.107594+0000","last_active":"2024-03-06T07:35:46.491796+0000","last_peered":"2024-03-06T07:35:46.491796+0000","last_clean":"2024-03-06T07:35:46.491796+0000","last_became_active":"2024-03-06T07:35:45.106136+0000","last_became_peered":"2024-03-06T07:35:45.106136+0000","last_unstale":"2024-03-06T07:35:46.491796+0000","last_undegraded":"2024-03-06T07:35:46.491796+0000","last_fullsized":"2024-03-06T07:35:46.491796+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:22.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111621+0000","last_change":"2024-03-06T07:35:45.092984+0000","last_active":"2024-03-06T07:35:46.111621+0000","last_peered":"2024-03-06T07:35:46.111621+0000","last_clean":"2024-03-06T07:35:46.111621+0000","last_became_active":"2024-03-06T07:35:45.092724+0000","last_became_peered":"2024-03-06T07:35:45.092724+0000","last_unstale":"2024-03-06T07:35:46.111621+0000","last_undegraded":"2024-03-06T07:35:46.111621+0000","last_fullsized":"2024-03-06T07:35:46.111621+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:38.967400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118665+0000","last_change":"2024-03-06T07:35:45.111311+0000","last_active":"2024-03-06T07:35:46.118665+0000","last_peered":"2024-03-06T07:35:46.118665+0000","last_clean":"2024-03-06T07:35:46.118665+0000","last_became_active":"2024-03-06T07:35:45.111037+0000","last_became_peered":"2024-03-06T07:35:45.111037+0000","last_unstale":"2024-03-06T07:35:46.118665+0000","last_undegraded":"2024-03-06T07:35:46.118665+0000","last_fullsized":"2024-03-06T07:35:46.118665+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:52.584633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125513+0000","last_change":"2024-03-06T07:35:45.114837+0000","last_active":"2024-03-06T07:35:46.125513+0000","last_peered":"2024-03-06T07:35:46.125513+0000","last_clean":"2024-03-06T07:35:46.125513+0000","last_became_active":"2024-03-06T07:35:45.114456+0000","last_became_peered":"2024-03-06T07:35:45.114456+0000","last_unstale":"2024-03-06T07:35:46.125513+0000","last_undegraded":"2024-03-06T07:35:46.125513+0000","last_fullsized":"2024-03-06T07:35:46.125513+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:21.534275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125048+0000","last_change":"2024-03-06T07:35:45.105165+0000","last_active":"2024-03-06T07:35:46.125048+0000","last_peered":"2024-03-06T07:35:46.125048+0000","last_clean":"2024-03-06T07:35:46.125048+0000","last_became_active":"2024-03-06T07:35:45.104839+0000","last_became_peered":"2024-03-06T07:35:45.104839+0000","last_unstale":"2024-03-06T07:35:46.125048+0000","last_undegraded":"2024-03-06T07:35:46.125048+0000","last_fullsized":"2024-03-06T07:35:46.125048+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:16.271827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490081+0000","last_change":"2024-03-06T07:35:45.104922+0000","last_active":"2024-03-06T07:35:46.490081+0000","last_peered":"2024-03-06T07:35:46.490081+0000","last_clean":"2024-03-06T07:35:46.490081+0000","last_became_active":"2024-03-06T07:35:45.103996+0000","last_became_peered":"2024-03-06T07:35:45.103996+0000","last_unstale":"2024-03-06T07:35:46.490081+0000","last_undegraded":"2024-03-06T07:35:46.490081+0000","last_fullsized":"2024-03-06T07:35:46.490081+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:16.271827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491932+0000","last_change":"2024-03-06T07:35:45.110563+0000","last_active":"2024-03-06T07:35:46.491932+0000","last_peered":"2024-03-06T07:35:46.491932+0000","last_clean":"2024-03-06T07:35:46.491932+0000","last_became_active":"2024-03-06T07:35:45.110276+0000","last_became_peered":"2024-03-06T07:35:45.110276+0000","last_unstale":"2024-03-06T07:35:46.491932+0000","last_undegraded":"2024-03-06T07:35:46.491932+0000","last_fullsized":"2024-03-06T07:35:46.491932+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:44.897420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487376+0000","last_change":"2024-03-06T07:35:45.104313+0000","last_active":"2024-03-06T07:35:46.487376+0000","last_peered":"2024-03-06T07:35:46.487376+0000","last_clean":"2024-03-06T07:35:46.487376+0000","last_became_active":"2024-03-06T07:35:45.102785+0000","last_became_peered":"2024-03-06T07:35:45.102785+0000","last_unstale":"2024-03-06T07:35:46.487376+0000","last_undegraded":"2024-03-06T07:35:46.487376+0000","last_fullsized":"2024-03-06T07:35:46.487376+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:14.122144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110809+0000","last_change":"2024-03-06T07:35:45.111820+0000","last_active":"2024-03-06T07:35:46.110809+0000","last_peered":"2024-03-06T07:35:46.110809+0000","last_clean":"2024-03-06T07:35:46.110809+0000","last_became_active":"2024-03-06T07:35:45.111563+0000","last_became_peered":"2024-03-06T07:35:45.111563+0000","last_unstale":"2024-03-06T07:35:46.110809+0000","last_undegraded":"2024-03-06T07:35:46.110809+0000","last_fullsized":"2024-03-06T07:35:46.110809+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:05.947118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490015+0000","last_change":"2024-03-06T07:35:45.111119+0000","last_active":"2024-03-06T07:35:46.490015+0000","last_peered":"2024-03-06T07:35:46.490015+0000","last_clean":"2024-03-06T07:35:46.490015+0000","last_became_active":"2024-03-06T07:35:45.110799+0000","last_became_peered":"2024-03-06T07:35:45.110799+0000","last_unstale":"2024-03-06T07:35:46.490015+0000","last_undegraded":"2024-03-06T07:35:46.490015+0000","last_fullsized":"2024-03-06T07:35:46.490015+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:27.160436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111472+0000","last_change":"2024-03-06T07:35:45.103530+0000","last_active":"2024-03-06T07:35:46.111472+0000","last_peered":"2024-03-06T07:35:46.111472+0000","last_clean":"2024-03-06T07:35:46.111472+0000","last_became_active":"2024-03-06T07:35:45.103223+0000","last_became_peered":"2024-03-06T07:35:45.103223+0000","last_unstale":"2024-03-06T07:35:46.111472+0000","last_undegraded":"2024-03-06T07:35:46.111472+0000","last_fullsized":"2024-03-06T07:35:46.111472+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:36.870162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486985+0000","last_change":"2024-03-06T07:35:45.112706+0000","last_active":"2024-03-06T07:35:46.486985+0000","last_peered":"2024-03-06T07:35:46.486985+0000","last_clean":"2024-03-06T07:35:46.486985+0000","last_became_active":"2024-03-06T07:35:45.112406+0000","last_became_peered":"2024-03-06T07:35:45.112406+0000","last_unstale":"2024-03-06T07:35:46.486985+0000","last_undegraded":"2024-03-06T07:35:46.486985+0000","last_fullsized":"2024-03-06T07:35:46.486985+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:02.251348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110748+0000","last_change":"2024-03-06T07:35:45.106268+0000","last_active":"2024-03-06T07:35:46.110748+0000","last_peered":"2024-03-06T07:35:46.110748+0000","last_clean":"2024-03-06T07:35:46.110748+0000","last_became_active":"2024-03-06T07:35:45.104563+0000","last_became_peered":"2024-03-06T07:35:45.104563+0000","last_unstale":"2024-03-06T07:35:46.110748+0000","last_undegraded":"2024-03-06T07:35:46.110748+0000","last_fullsized":"2024-03-06T07:35:46.110748+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:20.879433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112789+0000","last_change":"2024-03-06T07:35:45.096501+0000","last_active":"2024-03-06T07:35:46.112789+0000","last_peered":"2024-03-06T07:35:46.112789+0000","last_clean":"2024-03-06T07:35:46.112789+0000","last_became_active":"2024-03-06T07:35:45.095905+0000","last_became_peered":"2024-03-06T07:35:45.095905+0000","last_unstale":"2024-03-06T07:35:46.112789+0000","last_undegraded":"2024-03-06T07:35:46.112789+0000","last_fullsized":"2024-03-06T07:35:46.112789+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:52.866649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.126453+0000","last_change":"2024-03-06T07:35:45.115680+0000","last_active":"2024-03-06T07:35:46.126453+0000","last_peered":"2024-03-06T07:35:46.126453+0000","last_clean":"2024-03-06T07:35:46.126453+0000","last_became_active":"2024-03-06T07:35:45.115299+0000","last_became_peered":"2024-03-06T07:35:45.115299+0000","last_unstale":"2024-03-06T07:35:46.126453+0000","last_undegraded":"2024-03-06T07:35:46.126453+0000","last_fullsized":"2024-03-06T07:35:46.126453+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:38.789296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113049+0000","last_change":"2024-03-06T07:35:45.106696+0000","last_active":"2024-03-06T07:35:46.113049+0000","last_peered":"2024-03-06T07:35:46.113049+0000","last_clean":"2024-03-06T07:35:46.113049+0000","last_became_active":"2024-03-06T07:35:45.106360+0000","last_became_peered":"2024-03-06T07:35:45.106360+0000","last_unstale":"2024-03-06T07:35:46.113049+0000","last_undegraded":"2024-03-06T07:35:46.113049+0000","last_fullsized":"2024-03-06T07:35:46.113049+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:52.584633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110959+0000","last_change":"2024-03-06T07:35:45.107168+0000","last_active":"2024-03-06T07:35:46.110959+0000","last_peered":"2024-03-06T07:35:46.110959+0000","last_clean":"2024-03-06T07:35:46.110959+0000","last_became_active":"2024-03-06T07:35:45.106251+0000","last_became_peered":"2024-03-06T07:35:45.106251+0000","last_unstale":"2024-03-06T07:35:46.110959+0000","last_undegraded":"2024-03-06T07:35:46.110959+0000","last_fullsized":"2024-03-06T07:35:46.110959+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:53.166054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113019+0000","last_change":"2024-03-06T07:35:45.101496+0000","last_active":"2024-03-06T07:35:46.113019+0000","last_peered":"2024-03-06T07:35:46.113019+0000","last_clean":"2024-03-06T07:35:46.113019+0000","last_became_active":"2024-03-06T07:35:45.100736+0000","last_became_peered":"2024-03-06T07:35:45.100736+0000","last_unstale":"2024-03-06T07:35:46.113019+0000","last_undegraded":"2024-03-06T07:35:46.113019+0000","last_fullsized":"2024-03-06T07:35:46.113019+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.224692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118966+0000","last_change":"2024-03-06T07:35:45.112515+0000","last_active":"2024-03-06T07:35:46.118966+0000","last_peered":"2024-03-06T07:35:46.118966+0000","last_clean":"2024-03-06T07:35:46.118966+0000","last_became_active":"2024-03-06T07:35:45.112255+0000","last_became_peered":"2024-03-06T07:35:45.112255+0000","last_unstale":"2024-03-06T07:35:46.118966+0000","last_undegraded":"2024-03-06T07:35:46.118966+0000","last_fullsized":"2024-03-06T07:35:46.118966+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:48.144647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109975+0000","last_change":"2024-03-06T07:35:45.102694+0000","last_active":"2024-03-06T07:35:46.109975+0000","last_peered":"2024-03-06T07:35:46.109975+0000","last_clean":"2024-03-06T07:35:46.109975+0000","last_became_active":"2024-03-06T07:35:45.102404+0000","last_became_peered":"2024-03-06T07:35:45.102404+0000","last_unstale":"2024-03-06T07:35:46.109975+0000","last_undegraded":"2024-03-06T07:35:46.109975+0000","last_fullsized":"2024-03-06T07:35:46.109975+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:59.714613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491307+0000","last_change":"2024-03-06T07:35:45.110337+0000","last_active":"2024-03-06T07:35:46.491307+0000","last_peered":"2024-03-06T07:35:46.491307+0000","last_clean":"2024-03-06T07:35:46.491307+0000","last_became_active":"2024-03-06T07:35:45.110044+0000","last_became_peered":"2024-03-06T07:35:45.110044+0000","last_unstale":"2024-03-06T07:35:46.491307+0000","last_undegraded":"2024-03-06T07:35:46.491307+0000","last_fullsized":"2024-03-06T07:35:46.491307+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:59.228810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109238+0000","last_change":"2024-03-06T07:35:45.108130+0000","last_active":"2024-03-06T07:35:46.109238+0000","last_peered":"2024-03-06T07:35:46.109238+0000","last_clean":"2024-03-06T07:35:46.109238+0000","last_became_active":"2024-03-06T07:35:45.107721+0000","last_became_peered":"2024-03-06T07:35:45.107721+0000","last_unstale":"2024-03-06T07:35:46.109238+0000","last_undegraded":"2024-03-06T07:35:46.109238+0000","last_fullsized":"2024-03-06T07:35:46.109238+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:10.056863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487403+0000","last_change":"2024-03-06T07:35:45.112091+0000","last_active":"2024-03-06T07:35:46.487403+0000","last_peered":"2024-03-06T07:35:46.487403+0000","last_clean":"2024-03-06T07:35:46.487403+0000","last_became_active":"2024-03-06T07:35:45.108958+0000","last_became_peered":"2024-03-06T07:35:45.108958+0000","last_unstale":"2024-03-06T07:35:46.487403+0000","last_undegraded":"2024-03-06T07:35:46.487403+0000","last_fullsized":"2024-03-06T07:35:46.487403+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:59.018280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113223+0000","last_change":"2024-03-06T07:35:45.096842+0000","last_active":"2024-03-06T07:35:46.113223+0000","last_peered":"2024-03-06T07:35:46.113223+0000","last_clean":"2024-03-06T07:35:46.113223+0000","last_became_active":"2024-03-06T07:35:45.096281+0000","last_became_peered":"2024-03-06T07:35:45.096281+0000","last_unstale":"2024-03-06T07:35:46.113223+0000","last_undegraded":"2024-03-06T07:35:46.113223+0000","last_fullsized":"2024-03-06T07:35:46.113223+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:13.729625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111811+0000","last_change":"2024-03-06T07:35:45.104616+0000","last_active":"2024-03-06T07:35:46.111811+0000","last_peered":"2024-03-06T07:35:46.111811+0000","last_clean":"2024-03-06T07:35:46.111811+0000","last_became_active":"2024-03-06T07:35:45.104305+0000","last_became_peered":"2024-03-06T07:35:45.104305+0000","last_unstale":"2024-03-06T07:35:46.111811+0000","last_undegraded":"2024-03-06T07:35:46.111811+0000","last_fullsized":"2024-03-06T07:35:46.111811+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:03.632029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489911+0000","last_change":"2024-03-06T07:35:45.110178+0000","last_active":"2024-03-06T07:35:46.489911+0000","last_peered":"2024-03-06T07:35:46.489911+0000","last_clean":"2024-03-06T07:35:46.489911+0000","last_became_active":"2024-03-06T07:35:45.109226+0000","last_became_peered":"2024-03-06T07:35:45.109226+0000","last_unstale":"2024-03-06T07:35:46.489911+0000","last_undegraded":"2024-03-06T07:35:46.489911+0000","last_fullsized":"2024-03-06T07:35:46.489911+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.008208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111616+0000","last_change":"2024-03-06T07:35:45.094889+0000","last_active":"2024-03-06T07:35:46.111616+0000","last_peered":"2024-03-06T07:35:46.111616+0000","last_clean":"2024-03-06T07:35:46.111616+0000","last_became_active":"2024-03-06T07:35:45.094583+0000","last_became_peered":"2024-03-06T07:35:45.094583+0000","last_unstale":"2024-03-06T07:35:46.111616+0000","last_undegraded":"2024-03-06T07:35:46.111616+0000","last_fullsized":"2024-03-06T07:35:46.111616+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:45.903263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110904+0000","last_change":"2024-03-06T07:35:45.105555+0000","last_active":"2024-03-06T07:35:46.110904+0000","last_peered":"2024-03-06T07:35:46.110904+0000","last_clean":"2024-03-06T07:35:46.110904+0000","last_became_active":"2024-03-06T07:35:45.101044+0000","last_became_peered":"2024-03-06T07:35:45.101044+0000","last_unstale":"2024-03-06T07:35:46.110904+0000","last_undegraded":"2024-03-06T07:35:46.110904+0000","last_fullsized":"2024-03-06T07:35:46.110904+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:59.164319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118551+0000","last_change":"2024-03-06T07:35:45.113042+0000","last_active":"2024-03-06T07:35:46.118551+0000","last_peered":"2024-03-06T07:35:46.118551+0000","last_clean":"2024-03-06T07:35:46.118551+0000","last_became_active":"2024-03-06T07:35:45.112120+0000","last_became_peered":"2024-03-06T07:35:45.112120+0000","last_unstale":"2024-03-06T07:35:46.118551+0000","last_undegraded":"2024-03-06T07:35:46.118551+0000","last_fullsized":"2024-03-06T07:35:46.118551+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.224692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125071+0000","last_change":"2024-03-06T07:35:45.107896+0000","last_active":"2024-03-06T07:35:46.125071+0000","last_peered":"2024-03-06T07:35:46.125071+0000","last_clean":"2024-03-06T07:35:46.125071+0000","last_became_active":"2024-03-06T07:35:45.107597+0000","last_became_peered":"2024-03-06T07:35:45.107597+0000","last_unstale":"2024-03-06T07:35:46.125071+0000","last_undegraded":"2024-03-06T07:35:46.125071+0000","last_fullsized":"2024-03-06T07:35:46.125071+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:55.954006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113405+0000","last_change":"2024-03-06T07:35:45.110812+0000","last_active":"2024-03-06T07:35:46.113405+0000","last_peered":"2024-03-06T07:35:46.113405+0000","last_clean":"2024-03-06T07:35:46.113405+0000","last_became_active":"2024-03-06T07:35:45.110421+0000","last_became_peered":"2024-03-06T07:35:45.110421+0000","last_unstale":"2024-03-06T07:35:46.113405+0000","last_undegraded":"2024-03-06T07:35:46.113405+0000","last_fullsized":"2024-03-06T07:35:46.113405+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:26.423755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"124'4","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119335+0000","last_change":"2024-03-06T07:35:44.101676+0000","last_active":"2024-03-06T07:35:46.119335+0000","last_peered":"2024-03-06T07:35:46.119335+0000","last_clean":"2024-03-06T07:35:46.119335+0000","last_became_active":"2024-03-06T07:35:44.101371+0000","last_became_peered":"2024-03-06T07:35:44.101371+0000","last_unstale":"2024-03-06T07:35:46.119335+0000","last_undegraded":"2024-03-06T07:35:46.119335+0000","last_fullsized":"2024-03-06T07:35:46.119335+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:54.938493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.492273+0000","last_change":"2024-03-06T07:35:45.095771+0000","last_active":"2024-03-06T07:35:46.492273+0000","last_peered":"2024-03-06T07:35:46.492273+0000","last_clean":"2024-03-06T07:35:46.492273+0000","last_became_active":"2024-03-06T07:35:45.094903+0000","last_became_peered":"2024-03-06T07:35:45.094903+0000","last_unstale":"2024-03-06T07:35:46.492273+0000","last_undegraded":"2024-03-06T07:35:46.492273+0000","last_fullsized":"2024-03-06T07:35:46.492273+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:38.789296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489922+0000","last_change":"2024-03-06T07:35:44.094615+0000","last_active":"2024-03-06T07:35:46.489922+0000","last_peered":"2024-03-06T07:35:46.489922+0000","last_clean":"2024-03-06T07:35:46.489922+0000","last_became_active":"2024-03-06T07:35:44.094269+0000","last_became_peered":"2024-03-06T07:35:44.094269+0000","last_unstale":"2024-03-06T07:35:46.489922+0000","last_undegraded":"2024-03-06T07:35:46.489922+0000","last_fullsized":"2024-03-06T07:35:46.489922+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:28.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119234+0000","last_change":"2024-03-06T07:35:45.102981+0000","last_active":"2024-03-06T07:35:46.119234+0000","last_peered":"2024-03-06T07:35:46.119234+0000","last_clean":"2024-03-06T07:35:46.119234+0000","last_became_active":"2024-03-06T07:35:45.102685+0000","last_became_peered":"2024-03-06T07:35:45.102685+0000","last_unstale":"2024-03-06T07:35:46.119234+0000","last_undegraded":"2024-03-06T07:35:46.119234+0000","last_fullsized":"2024-03-06T07:35:46.119234+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:26.423755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111016+0000","last_change":"2024-03-06T07:35:44.088919+0000","last_active":"2024-03-06T07:35:46.111016+0000","last_peered":"2024-03-06T07:35:46.111016+0000","last_clean":"2024-03-06T07:35:46.111016+0000","last_became_active":"2024-03-06T07:35:44.088563+0000","last_became_peered":"2024-03-06T07:35:44.088563+0000","last_unstale":"2024-03-06T07:35:46.111016+0000","last_undegraded":"2024-03-06T07:35:46.111016+0000","last_fullsized":"2024-03-06T07:35:46.111016+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:21.261283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119359+0000","last_change":"2024-03-06T07:35:45.102746+0000","last_active":"2024-03-06T07:35:46.119359+0000","last_peered":"2024-03-06T07:35:46.119359+0000","last_clean":"2024-03-06T07:35:46.119359+0000","last_became_active":"2024-03-06T07:35:45.101705+0000","last_became_peered":"2024-03-06T07:35:45.101705+0000","last_unstale":"2024-03-06T07:35:46.119359+0000","last_undegraded":"2024-03-06T07:35:46.119359+0000","last_fullsized":"2024-03-06T07:35:46.119359+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.589604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487179+0000","last_change":"2024-03-06T07:35:44.093888+0000","last_active":"2024-03-06T07:35:46.487179+0000","last_peered":"2024-03-06T07:35:46.487179+0000","last_clean":"2024-03-06T07:35:46.487179+0000","last_became_active":"2024-03-06T07:35:44.093606+0000","last_became_peered":"2024-03-06T07:35:44.093606+0000","last_unstale":"2024-03-06T07:35:46.487179+0000","last_undegraded":"2024-03-06T07:35:46.487179+0000","last_fullsized":"2024-03-06T07:35:46.487179+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:28.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113505+0000","last_change":"2024-03-06T07:35:45.109719+0000","last_active":"2024-03-06T07:35:46.113505+0000","last_peered":"2024-03-06T07:35:46.113505+0000","last_clean":"2024-03-06T07:35:46.113505+0000","last_became_active":"2024-03-06T07:35:45.107177+0000","last_became_peered":"2024-03-06T07:35:45.107177+0000","last_unstale":"2024-03-06T07:35:46.113505+0000","last_undegraded":"2024-03-06T07:35:46.113505+0000","last_fullsized":"2024-03-06T07:35:46.113505+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.589604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486852+0000","last_change":"2024-03-06T07:35:44.087950+0000","last_active":"2024-03-06T07:35:46.486852+0000","last_peered":"2024-03-06T07:35:46.486852+0000","last_clean":"2024-03-06T07:35:46.486852+0000","last_became_active":"2024-03-06T07:35:44.087583+0000","last_became_peered":"2024-03-06T07:35:44.087583+0000","last_unstale":"2024-03-06T07:35:46.486852+0000","last_undegraded":"2024-03-06T07:35:46.486852+0000","last_fullsized":"2024-03-06T07:35:46.486852+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:17.156870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125766+0000","last_change":"2024-03-06T07:35:45.109898+0000","last_active":"2024-03-06T07:35:46.125766+0000","last_peered":"2024-03-06T07:35:46.125766+0000","last_clean":"2024-03-06T07:35:46.125766+0000","last_became_active":"2024-03-06T07:35:45.109230+0000","last_became_peered":"2024-03-06T07:35:45.109230+0000","last_unstale":"2024-03-06T07:35:46.125766+0000","last_undegraded":"2024-03-06T07:35:46.125766+0000","last_fullsized":"2024-03-06T07:35:46.125766+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:59.228810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113019+0000","last_change":"2024-03-06T07:35:44.084588+0000","last_active":"2024-03-06T07:35:46.113019+0000","last_peered":"2024-03-06T07:35:46.113019+0000","last_clean":"2024-03-06T07:35:46.113019+0000","last_became_active":"2024-03-06T07:35:44.084253+0000","last_became_peered":"2024-03-06T07:35:44.084253+0000","last_unstale":"2024-03-06T07:35:46.113019+0000","last_undegraded":"2024-03-06T07:35:46.113019+0000","last_fullsized":"2024-03-06T07:35:46.113019+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:58.213297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490412+0000","last_change":"2024-03-06T07:35:45.105155+0000","last_active":"2024-03-06T07:35:46.490412+0000","last_peered":"2024-03-06T07:35:46.490412+0000","last_clean":"2024-03-06T07:35:46.490412+0000","last_became_active":"2024-03-06T07:35:45.104549+0000","last_became_peered":"2024-03-06T07:35:45.104549+0000","last_unstale":"2024-03-06T07:35:46.490412+0000","last_undegraded":"2024-03-06T07:35:46.490412+0000","last_fullsized":"2024-03-06T07:35:46.490412+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:55.954006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125045+0000","last_change":"2024-03-06T07:35:44.095063+0000","last_active":"2024-03-06T07:35:46.125045+0000","last_peered":"2024-03-06T07:35:46.125045+0000","last_clean":"2024-03-06T07:35:46.125045+0000","last_became_active":"2024-03-06T07:35:44.094514+0000","last_became_peered":"2024-03-06T07:35:44.094514+0000","last_unstale":"2024-03-06T07:35:46.125045+0000","last_undegraded":"2024-03-06T07:35:46.125045+0000","last_fullsized":"2024-03-06T07:35:46.125045+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:34.548623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486715+0000","last_change":"2024-03-06T07:35:45.105058+0000","last_active":"2024-03-06T07:35:46.486715+0000","last_peered":"2024-03-06T07:35:46.486715+0000","last_clean":"2024-03-06T07:35:46.486715+0000","last_became_active":"2024-03-06T07:35:45.104763+0000","last_became_peered":"2024-03-06T07:35:45.104763+0000","last_unstale":"2024-03-06T07:35:46.486715+0000","last_undegraded":"2024-03-06T07:35:46.486715+0000","last_fullsized":"2024-03-06T07:35:46.486715+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:10.940668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486660+0000","last_change":"2024-03-06T07:35:44.094814+0000","last_active":"2024-03-06T07:35:46.486660+0000","last_peered":"2024-03-06T07:35:46.486660+0000","last_clean":"2024-03-06T07:35:46.486660+0000","last_became_active":"2024-03-06T07:35:44.094471+0000","last_became_peered":"2024-03-06T07:35:44.094471+0000","last_unstale":"2024-03-06T07:35:46.486660+0000","last_undegraded":"2024-03-06T07:35:46.486660+0000","last_fullsized":"2024-03-06T07:35:46.486660+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:45.077989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112101+0000","last_change":"2024-03-06T07:35:45.113274+0000","last_active":"2024-03-06T07:35:46.112101+0000","last_peered":"2024-03-06T07:35:46.112101+0000","last_clean":"2024-03-06T07:35:46.112101+0000","last_became_active":"2024-03-06T07:35:45.112921+0000","last_became_peered":"2024-03-06T07:35:45.112921+0000","last_unstale":"2024-03-06T07:35:46.112101+0000","last_undegraded":"2024-03-06T07:35:46.112101+0000","last_fullsized":"2024-03-06T07:35:46.112101+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:46.093502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491037+0000","last_change":"2024-03-06T07:35:44.096933+0000","last_active":"2024-03-06T07:35:46.491037+0000","last_peered":"2024-03-06T07:35:46.491037+0000","last_clean":"2024-03-06T07:35:46.491037+0000","last_became_active":"2024-03-06T07:35:44.096653+0000","last_became_peered":"2024-03-06T07:35:44.096653+0000","last_unstale":"2024-03-06T07:35:46.491037+0000","last_undegraded":"2024-03-06T07:35:46.491037+0000","last_fullsized":"2024-03-06T07:35:46.491037+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:50.612890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490901+0000","last_change":"2024-03-06T07:35:45.111188+0000","last_active":"2024-03-06T07:35:46.490901+0000","last_peered":"2024-03-06T07:35:46.490901+0000","last_clean":"2024-03-06T07:35:46.490901+0000","last_became_active":"2024-03-06T07:35:45.110869+0000","last_became_peered":"2024-03-06T07:35:45.110869+0000","last_unstale":"2024-03-06T07:35:46.490901+0000","last_undegraded":"2024-03-06T07:35:46.490901+0000","last_fullsized":"2024-03-06T07:35:46.490901+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:59.997132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113321+0000","last_change":"2024-03-06T07:35:44.094715+0000","last_active":"2024-03-06T07:35:46.113321+0000","last_peered":"2024-03-06T07:35:46.113321+0000","last_clean":"2024-03-06T07:35:46.113321+0000","last_became_active":"2024-03-06T07:35:44.094414+0000","last_became_peered":"2024-03-06T07:35:44.094414+0000","last_unstale":"2024-03-06T07:35:46.113321+0000","last_undegraded":"2024-03-06T07:35:46.113321+0000","last_fullsized":"2024-03-06T07:35:46.113321+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:21.395719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119431+0000","last_change":"2024-03-06T07:35:45.102282+0000","last_active":"2024-03-06T07:35:46.119431+0000","last_peered":"2024-03-06T07:35:46.119431+0000","last_clean":"2024-03-06T07:35:46.119431+0000","last_became_active":"2024-03-06T07:35:45.101846+0000","last_became_peered":"2024-03-06T07:35:45.101846+0000","last_unstale":"2024-03-06T07:35:46.119431+0000","last_undegraded":"2024-03-06T07:35:46.119431+0000","last_fullsized":"2024-03-06T07:35:46.119431+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:22.276797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"124'3","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489297+0000","last_change":"2024-03-06T07:35:44.092115+0000","last_active":"2024-03-06T07:35:46.489297+0000","last_peered":"2024-03-06T07:35:46.489297+0000","last_clean":"2024-03-06T07:35:46.489297+0000","last_became_active":"2024-03-06T07:35:44.091693+0000","last_became_peered":"2024-03-06T07:35:44.091693+0000","last_unstale":"2024-03-06T07:35:46.489297+0000","last_undegraded":"2024-03-06T07:35:46.489297+0000","last_fullsized":"2024-03-06T07:35:46.489297+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:16.967891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119005+0000","last_change":"2024-03-06T07:35:45.099167+0000","last_active":"2024-03-06T07:35:46.119005+0000","last_peered":"2024-03-06T07:35:46.119005+0000","last_clean":"2024-03-06T07:35:46.119005+0000","last_became_active":"2024-03-06T07:35:45.098599+0000","last_became_peered":"2024-03-06T07:35:45.098599+0000","last_unstale":"2024-03-06T07:35:46.119005+0000","last_undegraded":"2024-03-06T07:35:46.119005+0000","last_fullsized":"2024-03-06T07:35:46.119005+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:13.729625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"124'6","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491338+0000","last_change":"2024-03-06T07:35:44.100745+0000","last_active":"2024-03-06T07:35:46.491338+0000","last_peered":"2024-03-06T07:35:46.491338+0000","last_clean":"2024-03-06T07:35:46.491338+0000","last_became_active":"2024-03-06T07:35:44.100441+0000","last_became_peered":"2024-03-06T07:35:44.100441+0000","last_unstale":"2024-03-06T07:35:46.491338+0000","last_undegraded":"2024-03-06T07:35:46.491338+0000","last_fullsized":"2024-03-06T07:35:46.491338+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:49.540666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111797+0000","last_change":"2024-03-06T07:35:45.114340+0000","last_active":"2024-03-06T07:35:46.111797+0000","last_peered":"2024-03-06T07:35:46.111797+0000","last_clean":"2024-03-06T07:35:46.111797+0000","last_became_active":"2024-03-06T07:35:45.111068+0000","last_became_peered":"2024-03-06T07:35:45.111068+0000","last_unstale":"2024-03-06T07:35:46.111797+0000","last_undegraded":"2024-03-06T07:35:46.111797+0000","last_fullsized":"2024-03-06T07:35:46.111797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:18.172383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111095+0000","last_change":"2024-03-06T07:35:44.093573+0000","last_active":"2024-03-06T07:35:46.111095+0000","last_peered":"2024-03-06T07:35:46.111095+0000","last_clean":"2024-03-06T07:35:46.111095+0000","last_became_active":"2024-03-06T07:35:44.093251+0000","last_became_peered":"2024-03-06T07:35:44.093251+0000","last_unstale":"2024-03-06T07:35:46.111095+0000","last_undegraded":"2024-03-06T07:35:46.111095+0000","last_fullsized":"2024-03-06T07:35:46.111095+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:22.023929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486589+0000","last_change":"2024-03-06T07:35:45.099602+0000","last_active":"2024-03-06T07:35:46.486589+0000","last_peered":"2024-03-06T07:35:46.486589+0000","last_clean":"2024-03-06T07:35:46.486589+0000","last_became_active":"2024-03-06T07:35:45.099292+0000","last_became_peered":"2024-03-06T07:35:45.099292+0000","last_unstale":"2024-03-06T07:35:46.486589+0000","last_undegraded":"2024-03-06T07:35:46.486589+0000","last_fullsized":"2024-03-06T07:35:46.486589+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:44.928222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486796+0000","last_change":"2024-03-06T07:35:44.099478+0000","last_active":"2024-03-06T07:35:46.486796+0000","last_peered":"2024-03-06T07:35:46.486796+0000","last_clean":"2024-03-06T07:35:46.486796+0000","last_became_active":"2024-03-06T07:35:44.099239+0000","last_became_peered":"2024-03-06T07:35:44.099239+0000","last_unstale":"2024-03-06T07:35:46.486796+0000","last_undegraded":"2024-03-06T07:35:46.486796+0000","last_fullsized":"2024-03-06T07:35:46.486796+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:16.967891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491013+0000","last_change":"2024-03-06T07:35:45.108125+0000","last_active":"2024-03-06T07:35:46.491013+0000","last_peered":"2024-03-06T07:35:46.491013+0000","last_clean":"2024-03-06T07:35:46.491013+0000","last_became_active":"2024-03-06T07:35:45.107844+0000","last_became_peered":"2024-03-06T07:35:45.107844+0000","last_unstale":"2024-03-06T07:35:46.491013+0000","last_undegraded":"2024-03-06T07:35:46.491013+0000","last_fullsized":"2024-03-06T07:35:46.491013+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:21.534275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"124'8","reported_seq":22,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490516+0000","last_change":"2024-03-06T07:35:44.087035+0000","last_active":"2024-03-06T07:35:46.490516+0000","last_peered":"2024-03-06T07:35:46.490516+0000","last_clean":"2024-03-06T07:35:46.490516+0000","last_became_active":"2024-03-06T07:35:44.078949+0000","last_became_peered":"2024-03-06T07:35:44.078949+0000","last_unstale":"2024-03-06T07:35:46.490516+0000","last_undegraded":"2024-03-06T07:35:46.490516+0000","last_fullsized":"2024-03-06T07:35:46.490516+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:14.967838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491797+0000","last_change":"2024-03-06T07:35:45.093320+0000","last_active":"2024-03-06T07:35:46.491797+0000","last_peered":"2024-03-06T07:35:46.491797+0000","last_clean":"2024-03-06T07:35:46.491797+0000","last_became_active":"2024-03-06T07:35:45.092437+0000","last_became_peered":"2024-03-06T07:35:45.092437+0000","last_unstale":"2024-03-06T07:35:46.491797+0000","last_undegraded":"2024-03-06T07:35:46.491797+0000","last_fullsized":"2024-03-06T07:35:46.491797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.899873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491188+0000","last_change":"2024-03-06T07:35:44.100724+0000","last_active":"2024-03-06T07:35:46.491188+0000","last_peered":"2024-03-06T07:35:46.491188+0000","last_clean":"2024-03-06T07:35:46.491188+0000","last_became_active":"2024-03-06T07:35:44.100041+0000","last_became_peered":"2024-03-06T07:35:44.100041+0000","last_unstale":"2024-03-06T07:35:46.491188+0000","last_undegraded":"2024-03-06T07:35:46.491188+0000","last_fullsized":"2024-03-06T07:35:46.491188+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:32.927988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491062+0000","last_change":"2024-03-06T07:35:45.114907+0000","last_active":"2024-03-06T07:35:46.491062+0000","last_peered":"2024-03-06T07:35:46.491062+0000","last_clean":"2024-03-06T07:35:46.491062+0000","last_became_active":"2024-03-06T07:35:45.114612+0000","last_became_peered":"2024-03-06T07:35:45.114612+0000","last_unstale":"2024-03-06T07:35:46.491062+0000","last_undegraded":"2024-03-06T07:35:46.491062+0000","last_fullsized":"2024-03-06T07:35:46.491062+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:44.928222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110524+0000","last_change":"2024-03-06T07:35:44.098645+0000","last_active":"2024-03-06T07:35:46.110524+0000","last_peered":"2024-03-06T07:35:46.110524+0000","last_clean":"2024-03-06T07:35:46.110524+0000","last_became_active":"2024-03-06T07:35:44.097545+0000","last_became_peered":"2024-03-06T07:35:44.097545+0000","last_unstale":"2024-03-06T07:35:46.110524+0000","last_undegraded":"2024-03-06T07:35:46.110524+0000","last_fullsized":"2024-03-06T07:35:46.110524+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:25.408242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113168+0000","last_change":"2024-03-06T07:35:45.093230+0000","last_active":"2024-03-06T07:35:46.113168+0000","last_peered":"2024-03-06T07:35:46.113168+0000","last_clean":"2024-03-06T07:35:46.113168+0000","last_became_active":"2024-03-06T07:35:45.092374+0000","last_became_peered":"2024-03-06T07:35:45.092374+0000","last_unstale":"2024-03-06T07:35:46.113168+0000","last_undegraded":"2024-03-06T07:35:46.113168+0000","last_fullsized":"2024-03-06T07:35:46.113168+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:20.076733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125987+0000","last_change":"2024-03-06T07:35:44.095319+0000","last_active":"2024-03-06T07:35:46.125987+0000","last_peered":"2024-03-06T07:35:46.125987+0000","last_clean":"2024-03-06T07:35:46.125987+0000","last_became_active":"2024-03-06T07:35:44.095024+0000","last_became_peered":"2024-03-06T07:35:44.095024+0000","last_unstale":"2024-03-06T07:35:46.125987+0000","last_undegraded":"2024-03-06T07:35:46.125987+0000","last_fullsized":"2024-03-06T07:35:46.125987+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:51.582903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487449+0000","last_change":"2024-03-06T07:35:45.105371+0000","last_active":"2024-03-06T07:35:46.487449+0000","last_peered":"2024-03-06T07:35:46.487449+0000","last_clean":"2024-03-06T07:35:46.487449+0000","last_became_active":"2024-03-06T07:35:45.105007+0000","last_became_peered":"2024-03-06T07:35:45.105007+0000","last_unstale":"2024-03-06T07:35:46.487449+0000","last_undegraded":"2024-03-06T07:35:46.487449+0000","last_fullsized":"2024-03-06T07:35:46.487449+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:01.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489922+0000","last_change":"2024-03-06T07:35:44.095909+0000","last_active":"2024-03-06T07:35:46.489922+0000","last_peered":"2024-03-06T07:35:46.489922+0000","last_clean":"2024-03-06T07:35:46.489922+0000","last_became_active":"2024-03-06T07:35:44.095505+0000","last_became_peered":"2024-03-06T07:35:44.095505+0000","last_unstale":"2024-03-06T07:35:46.489922+0000","last_undegraded":"2024-03-06T07:35:46.489922+0000","last_fullsized":"2024-03-06T07:35:46.489922+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:40.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111045+0000","last_change":"2024-03-06T07:35:45.106202+0000","last_active":"2024-03-06T07:35:46.111045+0000","last_peered":"2024-03-06T07:35:46.111045+0000","last_clean":"2024-03-06T07:35:46.111045+0000","last_became_active":"2024-03-06T07:35:45.104241+0000","last_became_peered":"2024-03-06T07:35:45.104241+0000","last_unstale":"2024-03-06T07:35:46.111045+0000","last_undegraded":"2024-03-06T07:35:46.111045+0000","last_fullsized":"2024-03-06T07:35:46.111045+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:38.665186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489523+0000","last_change":"2024-03-06T07:35:44.090901+0000","last_active":"2024-03-06T07:35:46.489523+0000","last_peered":"2024-03-06T07:35:46.489523+0000","last_clean":"2024-03-06T07:35:46.489523+0000","last_became_active":"2024-03-06T07:35:44.090078+0000","last_became_peered":"2024-03-06T07:35:44.090078+0000","last_unstale":"2024-03-06T07:35:46.489523+0000","last_undegraded":"2024-03-06T07:35:46.489523+0000","last_fullsized":"2024-03-06T07:35:46.489523+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:05.572754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.126076+0000","last_change":"2024-03-06T07:35:45.105420+0000","last_active":"2024-03-06T07:35:46.126076+0000","last_peered":"2024-03-06T07:35:46.126076+0000","last_clean":"2024-03-06T07:35:46.126076+0000","last_became_active":"2024-03-06T07:35:45.105110+0000","last_became_peered":"2024-03-06T07:35:45.105110+0000","last_unstale":"2024-03-06T07:35:46.126076+0000","last_undegraded":"2024-03-06T07:35:46.126076+0000","last_fullsized":"2024-03-06T07:35:46.126076+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:22.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113351+0000","last_change":"2024-03-06T07:35:44.095536+0000","last_active":"2024-03-06T07:35:46.113351+0000","last_peered":"2024-03-06T07:35:46.113351+0000","last_clean":"2024-03-06T07:35:46.113351+0000","last_became_active":"2024-03-06T07:35:44.095234+0000","last_became_peered":"2024-03-06T07:35:44.095234+0000","last_unstale":"2024-03-06T07:35:46.113351+0000","last_undegraded":"2024-03-06T07:35:46.113351+0000","last_fullsized":"2024-03-06T07:35:46.113351+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:10.884360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487035+0000","last_change":"2024-03-06T07:35:45.100958+0000","last_active":"2024-03-06T07:35:46.487035+0000","last_peered":"2024-03-06T07:35:46.487035+0000","last_clean":"2024-03-06T07:35:46.487035+0000","last_became_active":"2024-03-06T07:35:45.100594+0000","last_became_peered":"2024-03-06T07:35:45.100594+0000","last_unstale":"2024-03-06T07:35:46.487035+0000","last_undegraded":"2024-03-06T07:35:46.487035+0000","last_fullsized":"2024-03-06T07:35:46.487035+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:29.719875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112839+0000","last_change":"2024-03-06T07:35:44.090384+0000","last_active":"2024-03-06T07:35:46.112839+0000","last_peered":"2024-03-06T07:35:46.112839+0000","last_clean":"2024-03-06T07:35:46.112839+0000","last_became_active":"2024-03-06T07:35:44.090007+0000","last_became_peered":"2024-03-06T07:35:44.090007+0000","last_unstale":"2024-03-06T07:35:46.112839+0000","last_undegraded":"2024-03-06T07:35:46.112839+0000","last_fullsized":"2024-03-06T07:35:46.112839+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:54.938493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118141+0000","last_change":"2024-03-06T07:35:45.099126+0000","last_active":"2024-03-06T07:35:46.118141+0000","last_peered":"2024-03-06T07:35:46.118141+0000","last_clean":"2024-03-06T07:35:46.118141+0000","last_became_active":"2024-03-06T07:35:45.097006+0000","last_became_peered":"2024-03-06T07:35:45.097006+0000","last_unstale":"2024-03-06T07:35:46.118141+0000","last_undegraded":"2024-03-06T07:35:46.118141+0000","last_fullsized":"2024-03-06T07:35:46.118141+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:55.635913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109310+0000","last_change":"2024-03-06T07:35:44.097593+0000","last_active":"2024-03-06T07:35:46.109310+0000","last_peered":"2024-03-06T07:35:46.109310+0000","last_clean":"2024-03-06T07:35:46.109310+0000","last_became_active":"2024-03-06T07:35:44.097308+0000","last_became_peered":"2024-03-06T07:35:44.097308+0000","last_unstale":"2024-03-06T07:35:46.109310+0000","last_undegraded":"2024-03-06T07:35:46.109310+0000","last_fullsized":"2024-03-06T07:35:46.109310+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:19.061220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.126405+0000","last_change":"2024-03-06T07:35:45.113261+0000","last_active":"2024-03-06T07:35:46.126405+0000","last_peered":"2024-03-06T07:35:46.126405+0000","last_clean":"2024-03-06T07:35:46.126405+0000","last_became_active":"2024-03-06T07:35:45.111663+0000","last_became_peered":"2024-03-06T07:35:45.111663+0000","last_unstale":"2024-03-06T07:35:46.126405+0000","last_undegraded":"2024-03-06T07:35:46.126405+0000","last_fullsized":"2024-03-06T07:35:46.126405+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:44.897420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113476+0000","last_change":"2024-03-06T07:35:44.098719+0000","last_active":"2024-03-06T07:35:46.113476+0000","last_peered":"2024-03-06T07:35:46.113476+0000","last_clean":"2024-03-06T07:35:46.113476+0000","last_became_active":"2024-03-06T07:35:44.097979+0000","last_became_peered":"2024-03-06T07:35:44.097979+0000","last_unstale":"2024-03-06T07:35:46.113476+0000","last_undegraded":"2024-03-06T07:35:46.113476+0000","last_fullsized":"2024-03-06T07:35:46.113476+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:41.699032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112429+0000","last_change":"2024-03-06T07:35:45.101142+0000","last_active":"2024-03-06T07:35:46.112429+0000","last_peered":"2024-03-06T07:35:46.112429+0000","last_clean":"2024-03-06T07:35:46.112429+0000","last_became_active":"2024-03-06T07:35:45.098224+0000","last_became_peered":"2024-03-06T07:35:45.098224+0000","last_unstale":"2024-03-06T07:35:46.112429+0000","last_undegraded":"2024-03-06T07:35:46.112429+0000","last_fullsized":"2024-03-06T07:35:46.112429+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:41.916333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"125'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118971+0000","last_change":"2024-03-06T07:35:44.102511+0000","last_active":"2024-03-06T07:35:46.118971+0000","last_peered":"2024-03-06T07:35:46.118971+0000","last_clean":"2024-03-06T07:35:46.118971+0000","last_became_active":"2024-03-06T07:35:44.102126+0000","last_became_peered":"2024-03-06T07:35:44.102126+0000","last_unstale":"2024-03-06T07:35:46.118971+0000","last_undegraded":"2024-03-06T07:35:46.118971+0000","last_fullsized":"2024-03-06T07:35:46.118971+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:20.518762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487471+0000","last_change":"2024-03-06T07:35:45.109652+0000","last_active":"2024-03-06T07:35:46.487471+0000","last_peered":"2024-03-06T07:35:46.487471+0000","last_clean":"2024-03-06T07:35:46.487471+0000","last_became_active":"2024-03-06T07:35:45.109327+0000","last_became_peered":"2024-03-06T07:35:45.109327+0000","last_unstale":"2024-03-06T07:35:46.487471+0000","last_undegraded":"2024-03-06T07:35:46.487471+0000","last_fullsized":"2024-03-06T07:35:46.487471+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:50.950332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487255+0000","last_change":"2024-03-06T07:35:44.098795+0000","last_active":"2024-03-06T07:35:46.487255+0000","last_peered":"2024-03-06T07:35:46.487255+0000","last_clean":"2024-03-06T07:35:46.487255+0000","last_became_active":"2024-03-06T07:35:44.098533+0000","last_became_peered":"2024-03-06T07:35:44.098533+0000","last_unstale":"2024-03-06T07:35:46.487255+0000","last_undegraded":"2024-03-06T07:35:46.487255+0000","last_fullsized":"2024-03-06T07:35:46.487255+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:40.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125828+0000","last_change":"2024-03-06T07:35:45.114794+0000","last_active":"2024-03-06T07:35:46.125828+0000","last_peered":"2024-03-06T07:35:46.125828+0000","last_clean":"2024-03-06T07:35:46.125828+0000","last_became_active":"2024-03-06T07:35:45.114513+0000","last_became_peered":"2024-03-06T07:35:45.114513+0000","last_unstale":"2024-03-06T07:35:46.125828+0000","last_undegraded":"2024-03-06T07:35:46.125828+0000","last_fullsized":"2024-03-06T07:35:46.125828+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.899873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110470+0000","last_change":"2024-03-06T07:35:44.090419+0000","last_active":"2024-03-06T07:35:46.110470+0000","last_peered":"2024-03-06T07:35:46.110470+0000","last_clean":"2024-03-06T07:35:46.110470+0000","last_became_active":"2024-03-06T07:35:44.090139+0000","last_became_peered":"2024-03-06T07:35:44.090139+0000","last_unstale":"2024-03-06T07:35:46.110470+0000","last_undegraded":"2024-03-06T07:35:46.110470+0000","last_fullsized":"2024-03-06T07:35:46.110470+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:10.574091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118676+0000","last_change":"2024-03-06T07:35:45.103565+0000","last_active":"2024-03-06T07:35:46.118676+0000","last_peered":"2024-03-06T07:35:46.118676+0000","last_clean":"2024-03-06T07:35:46.118676+0000","last_became_active":"2024-03-06T07:35:45.103297+0000","last_became_peered":"2024-03-06T07:35:45.103297+0000","last_unstale":"2024-03-06T07:35:46.118676+0000","last_undegraded":"2024-03-06T07:35:46.118676+0000","last_fullsized":"2024-03-06T07:35:46.118676+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:20.076733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486711+0000","last_change":"2024-03-06T07:35:44.098343+0000","last_active":"2024-03-06T07:35:46.486711+0000","last_peered":"2024-03-06T07:35:46.486711+0000","last_clean":"2024-03-06T07:35:46.486711+0000","last_became_active":"2024-03-06T07:35:44.098064+0000","last_became_peered":"2024-03-06T07:35:44.098064+0000","last_unstale":"2024-03-06T07:35:46.486711+0000","last_undegraded":"2024-03-06T07:35:46.486711+0000","last_fullsized":"2024-03-06T07:35:46.486711+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:02.616516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489746+0000","last_change":"2024-03-06T07:35:45.114015+0000","last_active":"2024-03-06T07:35:46.489746+0000","last_peered":"2024-03-06T07:35:46.489746+0000","last_clean":"2024-03-06T07:35:46.489746+0000","last_became_active":"2024-03-06T07:35:45.113727+0000","last_became_peered":"2024-03-06T07:35:45.113727+0000","last_unstale":"2024-03-06T07:35:46.489746+0000","last_undegraded":"2024-03-06T07:35:46.489746+0000","last_fullsized":"2024-03-06T07:35:46.489746+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:38.425084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118391+0000","last_change":"2024-03-06T07:35:44.099342+0000","last_active":"2024-03-06T07:35:46.118391+0000","last_peered":"2024-03-06T07:35:46.118391+0000","last_clean":"2024-03-06T07:35:46.118391+0000","last_became_active":"2024-03-06T07:35:44.099043+0000","last_became_peered":"2024-03-06T07:35:44.099043+0000","last_unstale":"2024-03-06T07:35:46.118391+0000","last_undegraded":"2024-03-06T07:35:46.118391+0000","last_fullsized":"2024-03-06T07:35:46.118391+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:26.144923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112941+0000","last_change":"2024-03-06T07:35:45.092834+0000","last_active":"2024-03-06T07:35:46.112941+0000","last_peered":"2024-03-06T07:35:46.112941+0000","last_clean":"2024-03-06T07:35:46.112941+0000","last_became_active":"2024-03-06T07:35:45.091939+0000","last_became_peered":"2024-03-06T07:35:45.091939+0000","last_unstale":"2024-03-06T07:35:46.112941+0000","last_undegraded":"2024-03-06T07:35:46.112941+0000","last_fullsized":"2024-03-06T07:35:46.112941+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:48.144647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112960+0000","last_change":"2024-03-06T07:35:44.094658+0000","last_active":"2024-03-06T07:35:46.112960+0000","last_peered":"2024-03-06T07:35:46.112960+0000","last_clean":"2024-03-06T07:35:46.112960+0000","last_became_active":"2024-03-06T07:35:44.094381+0000","last_became_peered":"2024-03-06T07:35:44.094381+0000","last_unstale":"2024-03-06T07:35:46.112960+0000","last_undegraded":"2024-03-06T07:35:46.112960+0000","last_fullsized":"2024-03-06T07:35:46.112960+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:20.518762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486870+0000","last_change":"2024-03-06T07:35:45.113950+0000","last_active":"2024-03-06T07:35:46.486870+0000","last_peered":"2024-03-06T07:35:46.486870+0000","last_clean":"2024-03-06T07:35:46.486870+0000","last_became_active":"2024-03-06T07:35:45.113633+0000","last_became_peered":"2024-03-06T07:35:45.113633+0000","last_unstale":"2024-03-06T07:35:46.486870+0000","last_undegraded":"2024-03-06T07:35:46.486870+0000","last_fullsized":"2024-03-06T07:35:46.486870+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:37.536656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109408+0000","last_change":"2024-03-06T07:35:44.093728+0000","last_active":"2024-03-06T07:35:46.109408+0000","last_peered":"2024-03-06T07:35:46.109408+0000","last_clean":"2024-03-06T07:35:46.109408+0000","last_became_active":"2024-03-06T07:35:44.092689+0000","last_became_peered":"2024-03-06T07:35:44.092689+0000","last_unstale":"2024-03-06T07:35:46.109408+0000","last_undegraded":"2024-03-06T07:35:46.109408+0000","last_fullsized":"2024-03-06T07:35:46.109408+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:12.714112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118244+0000","last_change":"2024-03-06T07:35:45.099419+0000","last_active":"2024-03-06T07:35:46.118244+0000","last_peered":"2024-03-06T07:35:46.118244+0000","last_clean":"2024-03-06T07:35:46.118244+0000","last_became_active":"2024-03-06T07:35:45.099007+0000","last_became_peered":"2024-03-06T07:35:45.099007+0000","last_unstale":"2024-03-06T07:35:46.118244+0000","last_undegraded":"2024-03-06T07:35:46.118244+0000","last_fullsized":"2024-03-06T07:35:46.118244+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:52.866649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118895+0000","last_change":"2024-03-06T07:35:44.101844+0000","last_active":"2024-03-06T07:35:46.118895+0000","last_peered":"2024-03-06T07:35:46.118895+0000","last_clean":"2024-03-06T07:35:46.118895+0000","last_became_active":"2024-03-06T07:35:44.101424+0000","last_became_peered":"2024-03-06T07:35:44.101424+0000","last_unstale":"2024-03-06T07:35:46.118895+0000","last_undegraded":"2024-03-06T07:35:46.118895+0000","last_fullsized":"2024-03-06T07:35:46.118895+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:15.256314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":360,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.117892+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:35:46.117892+0000","last_peered":"2024-03-06T07:35:46.117892+0000","last_clean":"2024-03-06T07:35:46.117892+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:35:46.117892+0000","last_undegraded":"2024-03-06T07:35:46.117892+0000","last_fullsized":"2024-03-06T07:35:46.117892+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574948,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37472,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334368,"statfs":{"total":96636764160,"available":96598392832,"internally_reserved":0,"allocated":2179072,"data_stored":1273875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36880,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334960,"statfs":{"total":96636764160,"available":96598999040,"internally_reserved":0,"allocated":1572864,"data_stored":682869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574947,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36876,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334964,"statfs":{"total":96636764160,"available":96599003136,"internally_reserved":0,"allocated":1568768,"data_stored":681369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574947,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1564672,"data_stored":681327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36876,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334964,"statfs":{"total":96636764160,"available":96599003136,"internally_reserved":0,"allocated":1568768,"data_stored":681417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574948,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36876,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334964,"statfs":{"total":96636764160,"available":96599003136,"internally_reserved":0,"allocated":1568768,"data_stored":681873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":681257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37444,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334396,"statfs":{"total":96636764160,"available":96598421504,"internally_reserved":0,"allocated":2154496,"data_stored":1271625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:49.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:49.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.785+0000 7fb047400640 1 Processor -- start 2024-03-06T07:35:49.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.785+0000 7fb047400640 1 -- start start 2024-03-06T07:35:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.786+0000 7fb047400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb0400d24c0 0x7fb0400d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.786+0000 7fb047400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb0400b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.786+0000 7fb047400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0400d4c50 con 0x7fb0400b2de0 2024-03-06T07:35:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.786+0000 7fb047400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0400d4a00 con 0x7fb0400d24c0 2024-03-06T07:35:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.786+0000 7fb047400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0400cf7e0 con 0x7fb0400b0aa0 2024-03-06T07:35:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.787+0000 7fb045400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb0400b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.787+0000 7fb045400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb0400b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:47798/0 (socket says 172.21.15.114:47798) 2024-03-06T07:35:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.787+0000 7fb045400640 1 -- 172.21.15.114:0/602963981 learned_addr learned my addr 172.21.15.114:0/602963981 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.787+0000 7fb044a00640 1 --2- 172.21.15.114:0/602963981 >> v2:172.21.15.177:3300/0 conn(0x7fb0400d24c0 0x7fb0400d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.788+0000 7fb045400640 1 -- 172.21.15.114:0/602963981 >> v2:172.21.15.177:3300/0 conn(0x7fb0400d24c0 msgr2=0x7fb0400d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.788+0000 7fb045400640 1 --2- 172.21.15.114:0/602963981 >> v2:172.21.15.177:3300/0 conn(0x7fb0400d24c0 0x7fb0400d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.788+0000 7fb045400640 1 -- 172.21.15.114:0/602963981 >> v1:172.21.15.177:6789/0 conn(0x7fb0400b2de0 legacy=0x7fb0400d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.788+0000 7fb045400640 1 -- 172.21.15.114:0/602963981 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0400cfa30 con 0x7fb0400b0aa0 2024-03-06T07:35:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.789+0000 7fb045400640 1 --2- 172.21.15.114:0/602963981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb0400b0f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb038009e50 tx=0x7fb03802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b0f77c8dcc6312a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb03f400640 1 -- 172.21.15.114:0/602963981 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03802ca70 con 0x7fb0400b0aa0 2024-03-06T07:35:49.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb03f400640 1 -- 172.21.15.114:0/602963981 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03802cc10 con 0x7fb0400b0aa0 2024-03-06T07:35:49.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb03f400640 1 -- 172.21.15.114:0/602963981 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0380343f0 con 0x7fb0400b0aa0 2024-03-06T07:35:49.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb047400640 1 -- 172.21.15.114:0/602963981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 msgr2=0x7fb0400b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb047400640 1 --2- 172.21.15.114:0/602963981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb0400b0f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb038009e50 tx=0x7fb03802bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb047400640 1 -- 172.21.15.114:0/602963981 shutdown_connections 2024-03-06T07:35:49.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb047400640 1 --2- 172.21.15.114:0/602963981 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb0400b0f30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb047400640 1 --2- 172.21.15.114:0/602963981 >> v2:172.21.15.177:3300/0 conn(0x7fb0400d24c0 0x7fb0400d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb047400640 1 -- 172.21.15.114:0/602963981 >> 172.21.15.114:0/602963981 conn(0x7fb0400d3ee0 msgr2=0x7fb0400d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.790+0000 7fb047400640 1 -- 172.21.15.114:0/602963981 shutdown_connections 2024-03-06T07:35:49.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.791+0000 7fb047400640 1 -- 172.21.15.114:0/602963981 wait complete. 2024-03-06T07:35:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.792+0000 7fb047400640 1 Processor -- start 2024-03-06T07:35:49.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.792+0000 7fb047400640 1 -- start start 2024-03-06T07:35:49.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.793+0000 7fb047400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb040153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.793+0000 7fb047400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0400b2de0 0x7fb0400b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.793+0000 7fb047400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0400d24c0 0x7fb04015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.793+0000 7fb047400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04015e5b0 con 0x7fb0400b2de0 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.793+0000 7fb047400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0400d0080 con 0x7fb0400d24c0 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.793+0000 7fb047400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04015e730 con 0x7fb0400b0aa0 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0400b2de0 0x7fb0400b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0400b2de0 0x7fb0400b74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39680/0 (socket says 172.21.15.114:39680) 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 -- 172.21.15.114:0/3964829572 learned_addr learned my addr 172.21.15.114:0/3964829572 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb044a00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb040153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb045400640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0400d24c0 0x7fb04015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 -- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 msgr2=0x7fb040153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb040153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 -- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0400d24c0 msgr2=0x7fb04015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0400d24c0 0x7fb04015f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb03fe00640 1 -- 172.21.15.114:0/3964829572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0400c7ab0 con 0x7fb0400b2de0 2024-03-06T07:35:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.794+0000 7fb044a00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb040153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:49.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.795+0000 7fb03fe00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0400b2de0 0x7fb0400b74d0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00c3f0 tx=0x7fb02c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.795+0000 7fb03cc00640 1 -- 172.21.15.114:0/3964829572 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0052d0 con 0x7fb0400b2de0 2024-03-06T07:35:49.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.795+0000 7fb03cc00640 1 -- 172.21.15.114:0/3964829572 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c005490 con 0x7fb0400b2de0 2024-03-06T07:35:49.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.795+0000 7fb03cc00640 1 -- 172.21.15.114:0/3964829572 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c007260 con 0x7fb0400b2de0 2024-03-06T07:35:49.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.795+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0400c7d20 con 0x7fb0400b2de0 2024-03-06T07:35:49.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.795+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0400c82e0 con 0x7fb0400b2de0 2024-03-06T07:35:49.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.796+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04015dde0 con 0x7fb0400b2de0 2024-03-06T07:35:49.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.798+0000 7fb03cc00640 1 -- 172.21.15.114:0/3964829572 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb02c007400 con 0x7fb0400b2de0 2024-03-06T07:35:49.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.799+0000 7fb03cc00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb02006b4d0 0x7fb02006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.799+0000 7fb03cc00640 1 -- 172.21.15.114:0/3964829572 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb02c089650 con 0x7fb0400b2de0 2024-03-06T07:35:49.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.799+0000 7fb044a00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb02006b4d0 0x7fb02006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.800+0000 7fb044a00640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb02006b4d0 0x7fb02006d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb034009af0 tx=0x7fb034008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:49.808+0000 7fb03cc00640 1 -- 172.21.15.114:0/3964829572 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb04015dde0 con 0x7fb0400b2de0 2024-03-06T07:35:50.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.000+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 --> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0400c84b0 con 0x7fb02006b4d0 2024-03-06T07:35:50.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.017+0000 7fb03cc00640 1 -- 172.21.15.114:0/3964829572 <== mgr.4107 v2:172.21.15.114:6832/3967440666 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252480 (secure 0 0 0) 0x7fb0400c84b0 con 0x7fb02006b4d0 2024-03-06T07:35:50.022 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:50.023 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb02006b4d0 msgr2=0x7fb02006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb02006b4d0 0x7fb02006d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb034009af0 tx=0x7fb034008040 comp rx=0 tx=0).stop 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0400b2de0 msgr2=0x7fb0400b74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0400b2de0 0x7fb0400b74d0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00c3f0 tx=0x7fb02c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 shutdown_connections 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0400d24c0 0x7fb04015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb02006b4d0 0x7fb02006d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0400b2de0 0x7fb0400b74d0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 --2- 172.21.15.114:0/3964829572 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb0400b0aa0 0x7fb040153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 >> 172.21.15.114:0/3964829572 conn(0x7fb0400d3ee0 msgr2=0x7fb0400d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 shutdown_connections 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.024+0000 7fb047400640 1 -- 172.21.15.114:0/3964829572 wait complete. 2024-03-06T07:35:50.051 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-03-06T07:35:49.245571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":296160,"kb_used_data":13416,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754678560,"statfs":{"total":773094113280,"available":772790845440,"internally_reserved":0,"allocated":13737984,"data_stored":6635612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":289394227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001827"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113335+0000","last_change":"2024-03-06T07:35:45.106118+0000","last_active":"2024-03-06T07:35:46.113335+0000","last_peered":"2024-03-06T07:35:46.113335+0000","last_clean":"2024-03-06T07:35:46.113335+0000","last_became_active":"2024-03-06T07:35:45.105818+0000","last_became_peered":"2024-03-06T07:35:45.105818+0000","last_unstale":"2024-03-06T07:35:46.113335+0000","last_undegraded":"2024-03-06T07:35:46.113335+0000","last_fullsized":"2024-03-06T07:35:46.113335+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:17.983404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487338+0000","last_change":"2024-03-06T07:35:45.103855+0000","last_active":"2024-03-06T07:35:46.487338+0000","last_peered":"2024-03-06T07:35:46.487338+0000","last_clean":"2024-03-06T07:35:46.487338+0000","last_became_active":"2024-03-06T07:35:45.103134+0000","last_became_peered":"2024-03-06T07:35:45.103134+0000","last_unstale":"2024-03-06T07:35:46.487338+0000","last_undegraded":"2024-03-06T07:35:46.487338+0000","last_fullsized":"2024-03-06T07:35:46.487338+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:43.923044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491796+0000","last_change":"2024-03-06T07:35:45.107594+0000","last_active":"2024-03-06T07:35:46.491796+0000","last_peered":"2024-03-06T07:35:46.491796+0000","last_clean":"2024-03-06T07:35:46.491796+0000","last_became_active":"2024-03-06T07:35:45.106136+0000","last_became_peered":"2024-03-06T07:35:45.106136+0000","last_unstale":"2024-03-06T07:35:46.491796+0000","last_undegraded":"2024-03-06T07:35:46.491796+0000","last_fullsized":"2024-03-06T07:35:46.491796+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:22.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111621+0000","last_change":"2024-03-06T07:35:45.092984+0000","last_active":"2024-03-06T07:35:46.111621+0000","last_peered":"2024-03-06T07:35:46.111621+0000","last_clean":"2024-03-06T07:35:46.111621+0000","last_became_active":"2024-03-06T07:35:45.092724+0000","last_became_peered":"2024-03-06T07:35:45.092724+0000","last_unstale":"2024-03-06T07:35:46.111621+0000","last_undegraded":"2024-03-06T07:35:46.111621+0000","last_fullsized":"2024-03-06T07:35:46.111621+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:38.967400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118665+0000","last_change":"2024-03-06T07:35:45.111311+0000","last_active":"2024-03-06T07:35:46.118665+0000","last_peered":"2024-03-06T07:35:46.118665+0000","last_clean":"2024-03-06T07:35:46.118665+0000","last_became_active":"2024-03-06T07:35:45.111037+0000","last_became_peered":"2024-03-06T07:35:45.111037+0000","last_unstale":"2024-03-06T07:35:46.118665+0000","last_undegraded":"2024-03-06T07:35:46.118665+0000","last_fullsized":"2024-03-06T07:35:46.118665+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:52.584633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125513+0000","last_change":"2024-03-06T07:35:45.114837+0000","last_active":"2024-03-06T07:35:46.125513+0000","last_peered":"2024-03-06T07:35:46.125513+0000","last_clean":"2024-03-06T07:35:46.125513+0000","last_became_active":"2024-03-06T07:35:45.114456+0000","last_became_peered":"2024-03-06T07:35:45.114456+0000","last_unstale":"2024-03-06T07:35:46.125513+0000","last_undegraded":"2024-03-06T07:35:46.125513+0000","last_fullsized":"2024-03-06T07:35:46.125513+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:21.534275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125048+0000","last_change":"2024-03-06T07:35:45.105165+0000","last_active":"2024-03-06T07:35:46.125048+0000","last_peered":"2024-03-06T07:35:46.125048+0000","last_clean":"2024-03-06T07:35:46.125048+0000","last_became_active":"2024-03-06T07:35:45.104839+0000","last_became_peered":"2024-03-06T07:35:45.104839+0000","last_unstale":"2024-03-06T07:35:46.125048+0000","last_undegraded":"2024-03-06T07:35:46.125048+0000","last_fullsized":"2024-03-06T07:35:46.125048+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:16.271827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490081+0000","last_change":"2024-03-06T07:35:45.104922+0000","last_active":"2024-03-06T07:35:46.490081+0000","last_peered":"2024-03-06T07:35:46.490081+0000","last_clean":"2024-03-06T07:35:46.490081+0000","last_became_active":"2024-03-06T07:35:45.103996+0000","last_became_peered":"2024-03-06T07:35:45.103996+0000","last_unstale":"2024-03-06T07:35:46.490081+0000","last_undegraded":"2024-03-06T07:35:46.490081+0000","last_fullsized":"2024-03-06T07:35:46.490081+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:16.271827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491932+0000","last_change":"2024-03-06T07:35:45.110563+0000","last_active":"2024-03-06T07:35:46.491932+0000","last_peered":"2024-03-06T07:35:46.491932+0000","last_clean":"2024-03-06T07:35:46.491932+0000","last_became_active":"2024-03-06T07:35:45.110276+0000","last_became_peered":"2024-03-06T07:35:45.110276+0000","last_unstale":"2024-03-06T07:35:46.491932+0000","last_undegraded":"2024-03-06T07:35:46.491932+0000","last_fullsized":"2024-03-06T07:35:46.491932+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:44.897420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487376+0000","last_change":"2024-03-06T07:35:45.104313+0000","last_active":"2024-03-06T07:35:46.487376+0000","last_peered":"2024-03-06T07:35:46.487376+0000","last_clean":"2024-03-06T07:35:46.487376+0000","last_became_active":"2024-03-06T07:35:45.102785+0000","last_became_peered":"2024-03-06T07:35:45.102785+0000","last_unstale":"2024-03-06T07:35:46.487376+0000","last_undegraded":"2024-03-06T07:35:46.487376+0000","last_fullsized":"2024-03-06T07:35:46.487376+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:14.122144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110809+0000","last_change":"2024-03-06T07:35:45.111820+0000","last_active":"2024-03-06T07:35:46.110809+0000","last_peered":"2024-03-06T07:35:46.110809+0000","last_clean":"2024-03-06T07:35:46.110809+0000","last_became_active":"2024-03-06T07:35:45.111563+0000","last_became_peered":"2024-03-06T07:35:45.111563+0000","last_unstale":"2024-03-06T07:35:46.110809+0000","last_undegraded":"2024-03-06T07:35:46.110809+0000","last_fullsized":"2024-03-06T07:35:46.110809+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:05.947118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490015+0000","last_change":"2024-03-06T07:35:45.111119+0000","last_active":"2024-03-06T07:35:46.490015+0000","last_peered":"2024-03-06T07:35:46.490015+0000","last_clean":"2024-03-06T07:35:46.490015+0000","last_became_active":"2024-03-06T07:35:45.110799+0000","last_became_peered":"2024-03-06T07:35:45.110799+0000","last_unstale":"2024-03-06T07:35:46.490015+0000","last_undegraded":"2024-03-06T07:35:46.490015+0000","last_fullsized":"2024-03-06T07:35:46.490015+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:27.160436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111472+0000","last_change":"2024-03-06T07:35:45.103530+0000","last_active":"2024-03-06T07:35:46.111472+0000","last_peered":"2024-03-06T07:35:46.111472+0000","last_clean":"2024-03-06T07:35:46.111472+0000","last_became_active":"2024-03-06T07:35:45.103223+0000","last_became_peered":"2024-03-06T07:35:45.103223+0000","last_unstale":"2024-03-06T07:35:46.111472+0000","last_undegraded":"2024-03-06T07:35:46.111472+0000","last_fullsized":"2024-03-06T07:35:46.111472+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:36.870162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486985+0000","last_change":"2024-03-06T07:35:45.112706+0000","last_active":"2024-03-06T07:35:46.486985+0000","last_peered":"2024-03-06T07:35:46.486985+0000","last_clean":"2024-03-06T07:35:46.486985+0000","last_became_active":"2024-03-06T07:35:45.112406+0000","last_became_peered":"2024-03-06T07:35:45.112406+0000","last_unstale":"2024-03-06T07:35:46.486985+0000","last_undegraded":"2024-03-06T07:35:46.486985+0000","last_fullsized":"2024-03-06T07:35:46.486985+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:02.251348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110748+0000","last_change":"2024-03-06T07:35:45.106268+0000","last_active":"2024-03-06T07:35:46.110748+0000","last_peered":"2024-03-06T07:35:46.110748+0000","last_clean":"2024-03-06T07:35:46.110748+0000","last_became_active":"2024-03-06T07:35:45.104563+0000","last_became_peered":"2024-03-06T07:35:45.104563+0000","last_unstale":"2024-03-06T07:35:46.110748+0000","last_undegraded":"2024-03-06T07:35:46.110748+0000","last_fullsized":"2024-03-06T07:35:46.110748+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:20.879433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112789+0000","last_change":"2024-03-06T07:35:45.096501+0000","last_active":"2024-03-06T07:35:46.112789+0000","last_peered":"2024-03-06T07:35:46.112789+0000","last_clean":"2024-03-06T07:35:46.112789+0000","last_became_active":"2024-03-06T07:35:45.095905+0000","last_became_peered":"2024-03-06T07:35:45.095905+0000","last_unstale":"2024-03-06T07:35:46.112789+0000","last_undegraded":"2024-03-06T07:35:46.112789+0000","last_fullsized":"2024-03-06T07:35:46.112789+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:52.866649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.126453+0000","last_change":"2024-03-06T07:35:45.115680+0000","last_active":"2024-03-06T07:35:46.126453+0000","last_peered":"2024-03-06T07:35:46.126453+0000","last_clean":"2024-03-06T07:35:46.126453+0000","last_became_active":"2024-03-06T07:35:45.115299+0000","last_became_peered":"2024-03-06T07:35:45.115299+0000","last_unstale":"2024-03-06T07:35:46.126453+0000","last_undegraded":"2024-03-06T07:35:46.126453+0000","last_fullsized":"2024-03-06T07:35:46.126453+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:38.789296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113049+0000","last_change":"2024-03-06T07:35:45.106696+0000","last_active":"2024-03-06T07:35:46.113049+0000","last_peered":"2024-03-06T07:35:46.113049+0000","last_clean":"2024-03-06T07:35:46.113049+0000","last_became_active":"2024-03-06T07:35:45.106360+0000","last_became_peered":"2024-03-06T07:35:45.106360+0000","last_unstale":"2024-03-06T07:35:46.113049+0000","last_undegraded":"2024-03-06T07:35:46.113049+0000","last_fullsized":"2024-03-06T07:35:46.113049+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:52.584633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110959+0000","last_change":"2024-03-06T07:35:45.107168+0000","last_active":"2024-03-06T07:35:46.110959+0000","last_peered":"2024-03-06T07:35:46.110959+0000","last_clean":"2024-03-06T07:35:46.110959+0000","last_became_active":"2024-03-06T07:35:45.106251+0000","last_became_peered":"2024-03-06T07:35:45.106251+0000","last_unstale":"2024-03-06T07:35:46.110959+0000","last_undegraded":"2024-03-06T07:35:46.110959+0000","last_fullsized":"2024-03-06T07:35:46.110959+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:53.166054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113019+0000","last_change":"2024-03-06T07:35:45.101496+0000","last_active":"2024-03-06T07:35:46.113019+0000","last_peered":"2024-03-06T07:35:46.113019+0000","last_clean":"2024-03-06T07:35:46.113019+0000","last_became_active":"2024-03-06T07:35:45.100736+0000","last_became_peered":"2024-03-06T07:35:45.100736+0000","last_unstale":"2024-03-06T07:35:46.113019+0000","last_undegraded":"2024-03-06T07:35:46.113019+0000","last_fullsized":"2024-03-06T07:35:46.113019+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.224692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118966+0000","last_change":"2024-03-06T07:35:45.112515+0000","last_active":"2024-03-06T07:35:46.118966+0000","last_peered":"2024-03-06T07:35:46.118966+0000","last_clean":"2024-03-06T07:35:46.118966+0000","last_became_active":"2024-03-06T07:35:45.112255+0000","last_became_peered":"2024-03-06T07:35:45.112255+0000","last_unstale":"2024-03-06T07:35:46.118966+0000","last_undegraded":"2024-03-06T07:35:46.118966+0000","last_fullsized":"2024-03-06T07:35:46.118966+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:48.144647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109975+0000","last_change":"2024-03-06T07:35:45.102694+0000","last_active":"2024-03-06T07:35:46.109975+0000","last_peered":"2024-03-06T07:35:46.109975+0000","last_clean":"2024-03-06T07:35:46.109975+0000","last_became_active":"2024-03-06T07:35:45.102404+0000","last_became_peered":"2024-03-06T07:35:45.102404+0000","last_unstale":"2024-03-06T07:35:46.109975+0000","last_undegraded":"2024-03-06T07:35:46.109975+0000","last_fullsized":"2024-03-06T07:35:46.109975+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:59.714613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491307+0000","last_change":"2024-03-06T07:35:45.110337+0000","last_active":"2024-03-06T07:35:46.491307+0000","last_peered":"2024-03-06T07:35:46.491307+0000","last_clean":"2024-03-06T07:35:46.491307+0000","last_became_active":"2024-03-06T07:35:45.110044+0000","last_became_peered":"2024-03-06T07:35:45.110044+0000","last_unstale":"2024-03-06T07:35:46.491307+0000","last_undegraded":"2024-03-06T07:35:46.491307+0000","last_fullsized":"2024-03-06T07:35:46.491307+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:59.228810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109238+0000","last_change":"2024-03-06T07:35:45.108130+0000","last_active":"2024-03-06T07:35:46.109238+0000","last_peered":"2024-03-06T07:35:46.109238+0000","last_clean":"2024-03-06T07:35:46.109238+0000","last_became_active":"2024-03-06T07:35:45.107721+0000","last_became_peered":"2024-03-06T07:35:45.107721+0000","last_unstale":"2024-03-06T07:35:46.109238+0000","last_undegraded":"2024-03-06T07:35:46.109238+0000","last_fullsized":"2024-03-06T07:35:46.109238+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:10.056863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487403+0000","last_change":"2024-03-06T07:35:45.112091+0000","last_active":"2024-03-06T07:35:46.487403+0000","last_peered":"2024-03-06T07:35:46.487403+0000","last_clean":"2024-03-06T07:35:46.487403+0000","last_became_active":"2024-03-06T07:35:45.108958+0000","last_became_peered":"2024-03-06T07:35:45.108958+0000","last_unstale":"2024-03-06T07:35:46.487403+0000","last_undegraded":"2024-03-06T07:35:46.487403+0000","last_fullsized":"2024-03-06T07:35:46.487403+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:59.018280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113223+0000","last_change":"2024-03-06T07:35:45.096842+0000","last_active":"2024-03-06T07:35:46.113223+0000","last_peered":"2024-03-06T07:35:46.113223+0000","last_clean":"2024-03-06T07:35:46.113223+0000","last_became_active":"2024-03-06T07:35:45.096281+0000","last_became_peered":"2024-03-06T07:35:45.096281+0000","last_unstale":"2024-03-06T07:35:46.113223+0000","last_undegraded":"2024-03-06T07:35:46.113223+0000","last_fullsized":"2024-03-06T07:35:46.113223+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:13.729625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111811+0000","last_change":"2024-03-06T07:35:45.104616+0000","last_active":"2024-03-06T07:35:46.111811+0000","last_peered":"2024-03-06T07:35:46.111811+0000","last_clean":"2024-03-06T07:35:46.111811+0000","last_became_active":"2024-03-06T07:35:45.104305+0000","last_became_peered":"2024-03-06T07:35:45.104305+0000","last_unstale":"2024-03-06T07:35:46.111811+0000","last_undegraded":"2024-03-06T07:35:46.111811+0000","last_fullsized":"2024-03-06T07:35:46.111811+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:03.632029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489911+0000","last_change":"2024-03-06T07:35:45.110178+0000","last_active":"2024-03-06T07:35:46.489911+0000","last_peered":"2024-03-06T07:35:46.489911+0000","last_clean":"2024-03-06T07:35:46.489911+0000","last_became_active":"2024-03-06T07:35:45.109226+0000","last_became_peered":"2024-03-06T07:35:45.109226+0000","last_unstale":"2024-03-06T07:35:46.489911+0000","last_undegraded":"2024-03-06T07:35:46.489911+0000","last_fullsized":"2024-03-06T07:35:46.489911+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.008208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111616+0000","last_change":"2024-03-06T07:35:45.094889+0000","last_active":"2024-03-06T07:35:46.111616+0000","last_peered":"2024-03-06T07:35:46.111616+0000","last_clean":"2024-03-06T07:35:46.111616+0000","last_became_active":"2024-03-06T07:35:45.094583+0000","last_became_peered":"2024-03-06T07:35:45.094583+0000","last_unstale":"2024-03-06T07:35:46.111616+0000","last_undegraded":"2024-03-06T07:35:46.111616+0000","last_fullsized":"2024-03-06T07:35:46.111616+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:45.903263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110904+0000","last_change":"2024-03-06T07:35:45.105555+0000","last_active":"2024-03-06T07:35:46.110904+0000","last_peered":"2024-03-06T07:35:46.110904+0000","last_clean":"2024-03-06T07:35:46.110904+0000","last_became_active":"2024-03-06T07:35:45.101044+0000","last_became_peered":"2024-03-06T07:35:45.101044+0000","last_unstale":"2024-03-06T07:35:46.110904+0000","last_undegraded":"2024-03-06T07:35:46.110904+0000","last_fullsized":"2024-03-06T07:35:46.110904+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:59.164319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118551+0000","last_change":"2024-03-06T07:35:45.113042+0000","last_active":"2024-03-06T07:35:46.118551+0000","last_peered":"2024-03-06T07:35:46.118551+0000","last_clean":"2024-03-06T07:35:46.118551+0000","last_became_active":"2024-03-06T07:35:45.112120+0000","last_became_peered":"2024-03-06T07:35:45.112120+0000","last_unstale":"2024-03-06T07:35:46.118551+0000","last_undegraded":"2024-03-06T07:35:46.118551+0000","last_fullsized":"2024-03-06T07:35:46.118551+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:07.224692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125071+0000","last_change":"2024-03-06T07:35:45.107896+0000","last_active":"2024-03-06T07:35:46.125071+0000","last_peered":"2024-03-06T07:35:46.125071+0000","last_clean":"2024-03-06T07:35:46.125071+0000","last_became_active":"2024-03-06T07:35:45.107597+0000","last_became_peered":"2024-03-06T07:35:45.107597+0000","last_unstale":"2024-03-06T07:35:46.125071+0000","last_undegraded":"2024-03-06T07:35:46.125071+0000","last_fullsized":"2024-03-06T07:35:46.125071+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:55.954006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113405+0000","last_change":"2024-03-06T07:35:45.110812+0000","last_active":"2024-03-06T07:35:46.113405+0000","last_peered":"2024-03-06T07:35:46.113405+0000","last_clean":"2024-03-06T07:35:46.113405+0000","last_became_active":"2024-03-06T07:35:45.110421+0000","last_became_peered":"2024-03-06T07:35:45.110421+0000","last_unstale":"2024-03-06T07:35:46.113405+0000","last_undegraded":"2024-03-06T07:35:46.113405+0000","last_fullsized":"2024-03-06T07:35:46.113405+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:26.423755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"124'4","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119335+0000","last_change":"2024-03-06T07:35:44.101676+0000","last_active":"2024-03-06T07:35:46.119335+0000","last_peered":"2024-03-06T07:35:46.119335+0000","last_clean":"2024-03-06T07:35:46.119335+0000","last_became_active":"2024-03-06T07:35:44.101371+0000","last_became_peered":"2024-03-06T07:35:44.101371+0000","last_unstale":"2024-03-06T07:35:46.119335+0000","last_undegraded":"2024-03-06T07:35:46.119335+0000","last_fullsized":"2024-03-06T07:35:46.119335+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:54.938493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.492273+0000","last_change":"2024-03-06T07:35:45.095771+0000","last_active":"2024-03-06T07:35:46.492273+0000","last_peered":"2024-03-06T07:35:46.492273+0000","last_clean":"2024-03-06T07:35:46.492273+0000","last_became_active":"2024-03-06T07:35:45.094903+0000","last_became_peered":"2024-03-06T07:35:45.094903+0000","last_unstale":"2024-03-06T07:35:46.492273+0000","last_undegraded":"2024-03-06T07:35:46.492273+0000","last_fullsized":"2024-03-06T07:35:46.492273+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:38.789296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489922+0000","last_change":"2024-03-06T07:35:44.094615+0000","last_active":"2024-03-06T07:35:46.489922+0000","last_peered":"2024-03-06T07:35:46.489922+0000","last_clean":"2024-03-06T07:35:46.489922+0000","last_became_active":"2024-03-06T07:35:44.094269+0000","last_became_peered":"2024-03-06T07:35:44.094269+0000","last_unstale":"2024-03-06T07:35:46.489922+0000","last_undegraded":"2024-03-06T07:35:46.489922+0000","last_fullsized":"2024-03-06T07:35:46.489922+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:28.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119234+0000","last_change":"2024-03-06T07:35:45.102981+0000","last_active":"2024-03-06T07:35:46.119234+0000","last_peered":"2024-03-06T07:35:46.119234+0000","last_clean":"2024-03-06T07:35:46.119234+0000","last_became_active":"2024-03-06T07:35:45.102685+0000","last_became_peered":"2024-03-06T07:35:45.102685+0000","last_unstale":"2024-03-06T07:35:46.119234+0000","last_undegraded":"2024-03-06T07:35:46.119234+0000","last_fullsized":"2024-03-06T07:35:46.119234+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:26.423755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111016+0000","last_change":"2024-03-06T07:35:44.088919+0000","last_active":"2024-03-06T07:35:46.111016+0000","last_peered":"2024-03-06T07:35:46.111016+0000","last_clean":"2024-03-06T07:35:46.111016+0000","last_became_active":"2024-03-06T07:35:44.088563+0000","last_became_peered":"2024-03-06T07:35:44.088563+0000","last_unstale":"2024-03-06T07:35:46.111016+0000","last_undegraded":"2024-03-06T07:35:46.111016+0000","last_fullsized":"2024-03-06T07:35:46.111016+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:21.261283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119359+0000","last_change":"2024-03-06T07:35:45.102746+0000","last_active":"2024-03-06T07:35:46.119359+0000","last_peered":"2024-03-06T07:35:46.119359+0000","last_clean":"2024-03-06T07:35:46.119359+0000","last_became_active":"2024-03-06T07:35:45.101705+0000","last_became_peered":"2024-03-06T07:35:45.101705+0000","last_unstale":"2024-03-06T07:35:46.119359+0000","last_undegraded":"2024-03-06T07:35:46.119359+0000","last_fullsized":"2024-03-06T07:35:46.119359+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.589604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487179+0000","last_change":"2024-03-06T07:35:44.093888+0000","last_active":"2024-03-06T07:35:46.487179+0000","last_peered":"2024-03-06T07:35:46.487179+0000","last_clean":"2024-03-06T07:35:46.487179+0000","last_became_active":"2024-03-06T07:35:44.093606+0000","last_became_peered":"2024-03-06T07:35:44.093606+0000","last_unstale":"2024-03-06T07:35:46.487179+0000","last_undegraded":"2024-03-06T07:35:46.487179+0000","last_fullsized":"2024-03-06T07:35:46.487179+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:28.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113505+0000","last_change":"2024-03-06T07:35:45.109719+0000","last_active":"2024-03-06T07:35:46.113505+0000","last_peered":"2024-03-06T07:35:46.113505+0000","last_clean":"2024-03-06T07:35:46.113505+0000","last_became_active":"2024-03-06T07:35:45.107177+0000","last_became_peered":"2024-03-06T07:35:45.107177+0000","last_unstale":"2024-03-06T07:35:46.113505+0000","last_undegraded":"2024-03-06T07:35:46.113505+0000","last_fullsized":"2024-03-06T07:35:46.113505+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.589604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486852+0000","last_change":"2024-03-06T07:35:44.087950+0000","last_active":"2024-03-06T07:35:46.486852+0000","last_peered":"2024-03-06T07:35:46.486852+0000","last_clean":"2024-03-06T07:35:46.486852+0000","last_became_active":"2024-03-06T07:35:44.087583+0000","last_became_peered":"2024-03-06T07:35:44.087583+0000","last_unstale":"2024-03-06T07:35:46.486852+0000","last_undegraded":"2024-03-06T07:35:46.486852+0000","last_fullsized":"2024-03-06T07:35:46.486852+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:17.156870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125766+0000","last_change":"2024-03-06T07:35:45.109898+0000","last_active":"2024-03-06T07:35:46.125766+0000","last_peered":"2024-03-06T07:35:46.125766+0000","last_clean":"2024-03-06T07:35:46.125766+0000","last_became_active":"2024-03-06T07:35:45.109230+0000","last_became_peered":"2024-03-06T07:35:45.109230+0000","last_unstale":"2024-03-06T07:35:46.125766+0000","last_undegraded":"2024-03-06T07:35:46.125766+0000","last_fullsized":"2024-03-06T07:35:46.125766+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:59.228810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113019+0000","last_change":"2024-03-06T07:35:44.084588+0000","last_active":"2024-03-06T07:35:46.113019+0000","last_peered":"2024-03-06T07:35:46.113019+0000","last_clean":"2024-03-06T07:35:46.113019+0000","last_became_active":"2024-03-06T07:35:44.084253+0000","last_became_peered":"2024-03-06T07:35:44.084253+0000","last_unstale":"2024-03-06T07:35:46.113019+0000","last_undegraded":"2024-03-06T07:35:46.113019+0000","last_fullsized":"2024-03-06T07:35:46.113019+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:58.213297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490412+0000","last_change":"2024-03-06T07:35:45.105155+0000","last_active":"2024-03-06T07:35:46.490412+0000","last_peered":"2024-03-06T07:35:46.490412+0000","last_clean":"2024-03-06T07:35:46.490412+0000","last_became_active":"2024-03-06T07:35:45.104549+0000","last_became_peered":"2024-03-06T07:35:45.104549+0000","last_unstale":"2024-03-06T07:35:46.490412+0000","last_undegraded":"2024-03-06T07:35:46.490412+0000","last_fullsized":"2024-03-06T07:35:46.490412+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:55.954006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125045+0000","last_change":"2024-03-06T07:35:44.095063+0000","last_active":"2024-03-06T07:35:46.125045+0000","last_peered":"2024-03-06T07:35:46.125045+0000","last_clean":"2024-03-06T07:35:46.125045+0000","last_became_active":"2024-03-06T07:35:44.094514+0000","last_became_peered":"2024-03-06T07:35:44.094514+0000","last_unstale":"2024-03-06T07:35:46.125045+0000","last_undegraded":"2024-03-06T07:35:46.125045+0000","last_fullsized":"2024-03-06T07:35:46.125045+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:34.548623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486715+0000","last_change":"2024-03-06T07:35:45.105058+0000","last_active":"2024-03-06T07:35:46.486715+0000","last_peered":"2024-03-06T07:35:46.486715+0000","last_clean":"2024-03-06T07:35:46.486715+0000","last_became_active":"2024-03-06T07:35:45.104763+0000","last_became_peered":"2024-03-06T07:35:45.104763+0000","last_unstale":"2024-03-06T07:35:46.486715+0000","last_undegraded":"2024-03-06T07:35:46.486715+0000","last_fullsized":"2024-03-06T07:35:46.486715+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:10.940668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486660+0000","last_change":"2024-03-06T07:35:44.094814+0000","last_active":"2024-03-06T07:35:46.486660+0000","last_peered":"2024-03-06T07:35:46.486660+0000","last_clean":"2024-03-06T07:35:46.486660+0000","last_became_active":"2024-03-06T07:35:44.094471+0000","last_became_peered":"2024-03-06T07:35:44.094471+0000","last_unstale":"2024-03-06T07:35:46.486660+0000","last_undegraded":"2024-03-06T07:35:46.486660+0000","last_fullsized":"2024-03-06T07:35:46.486660+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:45.077989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112101+0000","last_change":"2024-03-06T07:35:45.113274+0000","last_active":"2024-03-06T07:35:46.112101+0000","last_peered":"2024-03-06T07:35:46.112101+0000","last_clean":"2024-03-06T07:35:46.112101+0000","last_became_active":"2024-03-06T07:35:45.112921+0000","last_became_peered":"2024-03-06T07:35:45.112921+0000","last_unstale":"2024-03-06T07:35:46.112101+0000","last_undegraded":"2024-03-06T07:35:46.112101+0000","last_fullsized":"2024-03-06T07:35:46.112101+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:46.093502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491037+0000","last_change":"2024-03-06T07:35:44.096933+0000","last_active":"2024-03-06T07:35:46.491037+0000","last_peered":"2024-03-06T07:35:46.491037+0000","last_clean":"2024-03-06T07:35:46.491037+0000","last_became_active":"2024-03-06T07:35:44.096653+0000","last_became_peered":"2024-03-06T07:35:44.096653+0000","last_unstale":"2024-03-06T07:35:46.491037+0000","last_undegraded":"2024-03-06T07:35:46.491037+0000","last_fullsized":"2024-03-06T07:35:46.491037+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:50.612890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490901+0000","last_change":"2024-03-06T07:35:45.111188+0000","last_active":"2024-03-06T07:35:46.490901+0000","last_peered":"2024-03-06T07:35:46.490901+0000","last_clean":"2024-03-06T07:35:46.490901+0000","last_became_active":"2024-03-06T07:35:45.110869+0000","last_became_peered":"2024-03-06T07:35:45.110869+0000","last_unstale":"2024-03-06T07:35:46.490901+0000","last_undegraded":"2024-03-06T07:35:46.490901+0000","last_fullsized":"2024-03-06T07:35:46.490901+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:40:59.997132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113321+0000","last_change":"2024-03-06T07:35:44.094715+0000","last_active":"2024-03-06T07:35:46.113321+0000","last_peered":"2024-03-06T07:35:46.113321+0000","last_clean":"2024-03-06T07:35:46.113321+0000","last_became_active":"2024-03-06T07:35:44.094414+0000","last_became_peered":"2024-03-06T07:35:44.094414+0000","last_unstale":"2024-03-06T07:35:46.113321+0000","last_undegraded":"2024-03-06T07:35:46.113321+0000","last_fullsized":"2024-03-06T07:35:46.113321+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:21.395719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119431+0000","last_change":"2024-03-06T07:35:45.102282+0000","last_active":"2024-03-06T07:35:46.119431+0000","last_peered":"2024-03-06T07:35:46.119431+0000","last_clean":"2024-03-06T07:35:46.119431+0000","last_became_active":"2024-03-06T07:35:45.101846+0000","last_became_peered":"2024-03-06T07:35:45.101846+0000","last_unstale":"2024-03-06T07:35:46.119431+0000","last_undegraded":"2024-03-06T07:35:46.119431+0000","last_fullsized":"2024-03-06T07:35:46.119431+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:22.276797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"124'3","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489297+0000","last_change":"2024-03-06T07:35:44.092115+0000","last_active":"2024-03-06T07:35:46.489297+0000","last_peered":"2024-03-06T07:35:46.489297+0000","last_clean":"2024-03-06T07:35:46.489297+0000","last_became_active":"2024-03-06T07:35:44.091693+0000","last_became_peered":"2024-03-06T07:35:44.091693+0000","last_unstale":"2024-03-06T07:35:46.489297+0000","last_undegraded":"2024-03-06T07:35:46.489297+0000","last_fullsized":"2024-03-06T07:35:46.489297+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:16.967891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.119005+0000","last_change":"2024-03-06T07:35:45.099167+0000","last_active":"2024-03-06T07:35:46.119005+0000","last_peered":"2024-03-06T07:35:46.119005+0000","last_clean":"2024-03-06T07:35:46.119005+0000","last_became_active":"2024-03-06T07:35:45.098599+0000","last_became_peered":"2024-03-06T07:35:45.098599+0000","last_unstale":"2024-03-06T07:35:46.119005+0000","last_undegraded":"2024-03-06T07:35:46.119005+0000","last_fullsized":"2024-03-06T07:35:46.119005+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:13.729625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"124'6","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491338+0000","last_change":"2024-03-06T07:35:44.100745+0000","last_active":"2024-03-06T07:35:46.491338+0000","last_peered":"2024-03-06T07:35:46.491338+0000","last_clean":"2024-03-06T07:35:46.491338+0000","last_became_active":"2024-03-06T07:35:44.100441+0000","last_became_peered":"2024-03-06T07:35:44.100441+0000","last_unstale":"2024-03-06T07:35:46.491338+0000","last_undegraded":"2024-03-06T07:35:46.491338+0000","last_fullsized":"2024-03-06T07:35:46.491338+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:49.540666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111797+0000","last_change":"2024-03-06T07:35:45.114340+0000","last_active":"2024-03-06T07:35:46.111797+0000","last_peered":"2024-03-06T07:35:46.111797+0000","last_clean":"2024-03-06T07:35:46.111797+0000","last_became_active":"2024-03-06T07:35:45.111068+0000","last_became_peered":"2024-03-06T07:35:45.111068+0000","last_unstale":"2024-03-06T07:35:46.111797+0000","last_undegraded":"2024-03-06T07:35:46.111797+0000","last_fullsized":"2024-03-06T07:35:46.111797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:45:18.172383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111095+0000","last_change":"2024-03-06T07:35:44.093573+0000","last_active":"2024-03-06T07:35:46.111095+0000","last_peered":"2024-03-06T07:35:46.111095+0000","last_clean":"2024-03-06T07:35:46.111095+0000","last_became_active":"2024-03-06T07:35:44.093251+0000","last_became_peered":"2024-03-06T07:35:44.093251+0000","last_unstale":"2024-03-06T07:35:46.111095+0000","last_undegraded":"2024-03-06T07:35:46.111095+0000","last_fullsized":"2024-03-06T07:35:46.111095+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:22.023929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486589+0000","last_change":"2024-03-06T07:35:45.099602+0000","last_active":"2024-03-06T07:35:46.486589+0000","last_peered":"2024-03-06T07:35:46.486589+0000","last_clean":"2024-03-06T07:35:46.486589+0000","last_became_active":"2024-03-06T07:35:45.099292+0000","last_became_peered":"2024-03-06T07:35:45.099292+0000","last_unstale":"2024-03-06T07:35:46.486589+0000","last_undegraded":"2024-03-06T07:35:46.486589+0000","last_fullsized":"2024-03-06T07:35:46.486589+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:44.928222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486796+0000","last_change":"2024-03-06T07:35:44.099478+0000","last_active":"2024-03-06T07:35:46.486796+0000","last_peered":"2024-03-06T07:35:46.486796+0000","last_clean":"2024-03-06T07:35:46.486796+0000","last_became_active":"2024-03-06T07:35:44.099239+0000","last_became_peered":"2024-03-06T07:35:44.099239+0000","last_unstale":"2024-03-06T07:35:46.486796+0000","last_undegraded":"2024-03-06T07:35:46.486796+0000","last_fullsized":"2024-03-06T07:35:46.486796+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:16.967891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491013+0000","last_change":"2024-03-06T07:35:45.108125+0000","last_active":"2024-03-06T07:35:46.491013+0000","last_peered":"2024-03-06T07:35:46.491013+0000","last_clean":"2024-03-06T07:35:46.491013+0000","last_became_active":"2024-03-06T07:35:45.107844+0000","last_became_peered":"2024-03-06T07:35:45.107844+0000","last_unstale":"2024-03-06T07:35:46.491013+0000","last_undegraded":"2024-03-06T07:35:46.491013+0000","last_fullsized":"2024-03-06T07:35:46.491013+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:21.534275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"124'8","reported_seq":22,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.490516+0000","last_change":"2024-03-06T07:35:44.087035+0000","last_active":"2024-03-06T07:35:46.490516+0000","last_peered":"2024-03-06T07:35:46.490516+0000","last_clean":"2024-03-06T07:35:46.490516+0000","last_became_active":"2024-03-06T07:35:44.078949+0000","last_became_peered":"2024-03-06T07:35:44.078949+0000","last_unstale":"2024-03-06T07:35:46.490516+0000","last_undegraded":"2024-03-06T07:35:46.490516+0000","last_fullsized":"2024-03-06T07:35:46.490516+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:14.967838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491797+0000","last_change":"2024-03-06T07:35:45.093320+0000","last_active":"2024-03-06T07:35:46.491797+0000","last_peered":"2024-03-06T07:35:46.491797+0000","last_clean":"2024-03-06T07:35:46.491797+0000","last_became_active":"2024-03-06T07:35:45.092437+0000","last_became_peered":"2024-03-06T07:35:45.092437+0000","last_unstale":"2024-03-06T07:35:46.491797+0000","last_undegraded":"2024-03-06T07:35:46.491797+0000","last_fullsized":"2024-03-06T07:35:46.491797+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.899873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491188+0000","last_change":"2024-03-06T07:35:44.100724+0000","last_active":"2024-03-06T07:35:46.491188+0000","last_peered":"2024-03-06T07:35:46.491188+0000","last_clean":"2024-03-06T07:35:46.491188+0000","last_became_active":"2024-03-06T07:35:44.100041+0000","last_became_peered":"2024-03-06T07:35:44.100041+0000","last_unstale":"2024-03-06T07:35:46.491188+0000","last_undegraded":"2024-03-06T07:35:46.491188+0000","last_fullsized":"2024-03-06T07:35:46.491188+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:32.927988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.491062+0000","last_change":"2024-03-06T07:35:45.114907+0000","last_active":"2024-03-06T07:35:46.491062+0000","last_peered":"2024-03-06T07:35:46.491062+0000","last_clean":"2024-03-06T07:35:46.491062+0000","last_became_active":"2024-03-06T07:35:45.114612+0000","last_became_peered":"2024-03-06T07:35:45.114612+0000","last_unstale":"2024-03-06T07:35:46.491062+0000","last_undegraded":"2024-03-06T07:35:46.491062+0000","last_fullsized":"2024-03-06T07:35:46.491062+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:44.928222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110524+0000","last_change":"2024-03-06T07:35:44.098645+0000","last_active":"2024-03-06T07:35:46.110524+0000","last_peered":"2024-03-06T07:35:46.110524+0000","last_clean":"2024-03-06T07:35:46.110524+0000","last_became_active":"2024-03-06T07:35:44.097545+0000","last_became_peered":"2024-03-06T07:35:44.097545+0000","last_unstale":"2024-03-06T07:35:46.110524+0000","last_undegraded":"2024-03-06T07:35:46.110524+0000","last_fullsized":"2024-03-06T07:35:46.110524+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:25.408242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113168+0000","last_change":"2024-03-06T07:35:45.093230+0000","last_active":"2024-03-06T07:35:46.113168+0000","last_peered":"2024-03-06T07:35:46.113168+0000","last_clean":"2024-03-06T07:35:46.113168+0000","last_became_active":"2024-03-06T07:35:45.092374+0000","last_became_peered":"2024-03-06T07:35:45.092374+0000","last_unstale":"2024-03-06T07:35:46.113168+0000","last_undegraded":"2024-03-06T07:35:46.113168+0000","last_fullsized":"2024-03-06T07:35:46.113168+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:20.076733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"124'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125987+0000","last_change":"2024-03-06T07:35:44.095319+0000","last_active":"2024-03-06T07:35:46.125987+0000","last_peered":"2024-03-06T07:35:46.125987+0000","last_clean":"2024-03-06T07:35:46.125987+0000","last_became_active":"2024-03-06T07:35:44.095024+0000","last_became_peered":"2024-03-06T07:35:44.095024+0000","last_unstale":"2024-03-06T07:35:46.125987+0000","last_undegraded":"2024-03-06T07:35:46.125987+0000","last_fullsized":"2024-03-06T07:35:46.125987+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:51.582903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487449+0000","last_change":"2024-03-06T07:35:45.105371+0000","last_active":"2024-03-06T07:35:46.487449+0000","last_peered":"2024-03-06T07:35:46.487449+0000","last_clean":"2024-03-06T07:35:46.487449+0000","last_became_active":"2024-03-06T07:35:45.105007+0000","last_became_peered":"2024-03-06T07:35:45.105007+0000","last_unstale":"2024-03-06T07:35:46.487449+0000","last_undegraded":"2024-03-06T07:35:46.487449+0000","last_fullsized":"2024-03-06T07:35:46.487449+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:01.026083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489922+0000","last_change":"2024-03-06T07:35:44.095909+0000","last_active":"2024-03-06T07:35:46.489922+0000","last_peered":"2024-03-06T07:35:46.489922+0000","last_clean":"2024-03-06T07:35:46.489922+0000","last_became_active":"2024-03-06T07:35:44.095505+0000","last_became_peered":"2024-03-06T07:35:44.095505+0000","last_unstale":"2024-03-06T07:35:46.489922+0000","last_undegraded":"2024-03-06T07:35:46.489922+0000","last_fullsized":"2024-03-06T07:35:46.489922+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:40.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.111045+0000","last_change":"2024-03-06T07:35:45.106202+0000","last_active":"2024-03-06T07:35:46.111045+0000","last_peered":"2024-03-06T07:35:46.111045+0000","last_clean":"2024-03-06T07:35:46.111045+0000","last_became_active":"2024-03-06T07:35:45.104241+0000","last_became_peered":"2024-03-06T07:35:45.104241+0000","last_unstale":"2024-03-06T07:35:46.111045+0000","last_undegraded":"2024-03-06T07:35:46.111045+0000","last_fullsized":"2024-03-06T07:35:46.111045+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:38.665186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489523+0000","last_change":"2024-03-06T07:35:44.090901+0000","last_active":"2024-03-06T07:35:46.489523+0000","last_peered":"2024-03-06T07:35:46.489523+0000","last_clean":"2024-03-06T07:35:46.489523+0000","last_became_active":"2024-03-06T07:35:44.090078+0000","last_became_peered":"2024-03-06T07:35:44.090078+0000","last_unstale":"2024-03-06T07:35:46.489523+0000","last_undegraded":"2024-03-06T07:35:46.489523+0000","last_fullsized":"2024-03-06T07:35:46.489523+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:05.572754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.126076+0000","last_change":"2024-03-06T07:35:45.105420+0000","last_active":"2024-03-06T07:35:46.126076+0000","last_peered":"2024-03-06T07:35:46.126076+0000","last_clean":"2024-03-06T07:35:46.126076+0000","last_became_active":"2024-03-06T07:35:45.105110+0000","last_became_peered":"2024-03-06T07:35:45.105110+0000","last_unstale":"2024-03-06T07:35:46.126076+0000","last_undegraded":"2024-03-06T07:35:46.126076+0000","last_fullsized":"2024-03-06T07:35:46.126076+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:22.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113351+0000","last_change":"2024-03-06T07:35:44.095536+0000","last_active":"2024-03-06T07:35:46.113351+0000","last_peered":"2024-03-06T07:35:46.113351+0000","last_clean":"2024-03-06T07:35:46.113351+0000","last_became_active":"2024-03-06T07:35:44.095234+0000","last_became_peered":"2024-03-06T07:35:44.095234+0000","last_unstale":"2024-03-06T07:35:46.113351+0000","last_undegraded":"2024-03-06T07:35:46.113351+0000","last_fullsized":"2024-03-06T07:35:46.113351+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:10.884360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487035+0000","last_change":"2024-03-06T07:35:45.100958+0000","last_active":"2024-03-06T07:35:46.487035+0000","last_peered":"2024-03-06T07:35:46.487035+0000","last_clean":"2024-03-06T07:35:46.487035+0000","last_became_active":"2024-03-06T07:35:45.100594+0000","last_became_peered":"2024-03-06T07:35:45.100594+0000","last_unstale":"2024-03-06T07:35:46.487035+0000","last_undegraded":"2024-03-06T07:35:46.487035+0000","last_fullsized":"2024-03-06T07:35:46.487035+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:29.719875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112839+0000","last_change":"2024-03-06T07:35:44.090384+0000","last_active":"2024-03-06T07:35:46.112839+0000","last_peered":"2024-03-06T07:35:46.112839+0000","last_clean":"2024-03-06T07:35:46.112839+0000","last_became_active":"2024-03-06T07:35:44.090007+0000","last_became_peered":"2024-03-06T07:35:44.090007+0000","last_unstale":"2024-03-06T07:35:46.112839+0000","last_undegraded":"2024-03-06T07:35:46.112839+0000","last_fullsized":"2024-03-06T07:35:46.112839+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:54.938493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118141+0000","last_change":"2024-03-06T07:35:45.099126+0000","last_active":"2024-03-06T07:35:46.118141+0000","last_peered":"2024-03-06T07:35:46.118141+0000","last_clean":"2024-03-06T07:35:46.118141+0000","last_became_active":"2024-03-06T07:35:45.097006+0000","last_became_peered":"2024-03-06T07:35:45.097006+0000","last_unstale":"2024-03-06T07:35:46.118141+0000","last_undegraded":"2024-03-06T07:35:46.118141+0000","last_fullsized":"2024-03-06T07:35:46.118141+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:55.635913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109310+0000","last_change":"2024-03-06T07:35:44.097593+0000","last_active":"2024-03-06T07:35:46.109310+0000","last_peered":"2024-03-06T07:35:46.109310+0000","last_clean":"2024-03-06T07:35:46.109310+0000","last_became_active":"2024-03-06T07:35:44.097308+0000","last_became_peered":"2024-03-06T07:35:44.097308+0000","last_unstale":"2024-03-06T07:35:46.109310+0000","last_undegraded":"2024-03-06T07:35:46.109310+0000","last_fullsized":"2024-03-06T07:35:46.109310+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:19.061220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.126405+0000","last_change":"2024-03-06T07:35:45.113261+0000","last_active":"2024-03-06T07:35:46.126405+0000","last_peered":"2024-03-06T07:35:46.126405+0000","last_clean":"2024-03-06T07:35:46.126405+0000","last_became_active":"2024-03-06T07:35:45.111663+0000","last_became_peered":"2024-03-06T07:35:45.111663+0000","last_unstale":"2024-03-06T07:35:46.126405+0000","last_undegraded":"2024-03-06T07:35:46.126405+0000","last_fullsized":"2024-03-06T07:35:46.126405+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:44.897420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.113476+0000","last_change":"2024-03-06T07:35:44.098719+0000","last_active":"2024-03-06T07:35:46.113476+0000","last_peered":"2024-03-06T07:35:46.113476+0000","last_clean":"2024-03-06T07:35:46.113476+0000","last_became_active":"2024-03-06T07:35:44.097979+0000","last_became_peered":"2024-03-06T07:35:44.097979+0000","last_unstale":"2024-03-06T07:35:46.113476+0000","last_undegraded":"2024-03-06T07:35:46.113476+0000","last_fullsized":"2024-03-06T07:35:46.113476+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:41.699032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112429+0000","last_change":"2024-03-06T07:35:45.101142+0000","last_active":"2024-03-06T07:35:46.112429+0000","last_peered":"2024-03-06T07:35:46.112429+0000","last_clean":"2024-03-06T07:35:46.112429+0000","last_became_active":"2024-03-06T07:35:45.098224+0000","last_became_peered":"2024-03-06T07:35:45.098224+0000","last_unstale":"2024-03-06T07:35:46.112429+0000","last_undegraded":"2024-03-06T07:35:46.112429+0000","last_fullsized":"2024-03-06T07:35:46.112429+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:41.916333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"125'1","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118971+0000","last_change":"2024-03-06T07:35:44.102511+0000","last_active":"2024-03-06T07:35:46.118971+0000","last_peered":"2024-03-06T07:35:46.118971+0000","last_clean":"2024-03-06T07:35:46.118971+0000","last_became_active":"2024-03-06T07:35:44.102126+0000","last_became_peered":"2024-03-06T07:35:44.102126+0000","last_unstale":"2024-03-06T07:35:46.118971+0000","last_undegraded":"2024-03-06T07:35:46.118971+0000","last_fullsized":"2024-03-06T07:35:46.118971+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:20.518762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487471+0000","last_change":"2024-03-06T07:35:45.109652+0000","last_active":"2024-03-06T07:35:46.487471+0000","last_peered":"2024-03-06T07:35:46.487471+0000","last_clean":"2024-03-06T07:35:46.487471+0000","last_became_active":"2024-03-06T07:35:45.109327+0000","last_became_peered":"2024-03-06T07:35:45.109327+0000","last_unstale":"2024-03-06T07:35:46.487471+0000","last_undegraded":"2024-03-06T07:35:46.487471+0000","last_fullsized":"2024-03-06T07:35:46.487471+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:50.950332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.487255+0000","last_change":"2024-03-06T07:35:44.098795+0000","last_active":"2024-03-06T07:35:46.487255+0000","last_peered":"2024-03-06T07:35:46.487255+0000","last_clean":"2024-03-06T07:35:46.487255+0000","last_became_active":"2024-03-06T07:35:44.098533+0000","last_became_peered":"2024-03-06T07:35:44.098533+0000","last_unstale":"2024-03-06T07:35:46.487255+0000","last_undegraded":"2024-03-06T07:35:46.487255+0000","last_fullsized":"2024-03-06T07:35:46.487255+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:40.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.125828+0000","last_change":"2024-03-06T07:35:45.114794+0000","last_active":"2024-03-06T07:35:46.125828+0000","last_peered":"2024-03-06T07:35:46.125828+0000","last_clean":"2024-03-06T07:35:46.125828+0000","last_became_active":"2024-03-06T07:35:45.114513+0000","last_became_peered":"2024-03-06T07:35:45.114513+0000","last_unstale":"2024-03-06T07:35:46.125828+0000","last_undegraded":"2024-03-06T07:35:46.125828+0000","last_fullsized":"2024-03-06T07:35:46.125828+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:11.899873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.110470+0000","last_change":"2024-03-06T07:35:44.090419+0000","last_active":"2024-03-06T07:35:46.110470+0000","last_peered":"2024-03-06T07:35:46.110470+0000","last_clean":"2024-03-06T07:35:46.110470+0000","last_became_active":"2024-03-06T07:35:44.090139+0000","last_became_peered":"2024-03-06T07:35:44.090139+0000","last_unstale":"2024-03-06T07:35:46.110470+0000","last_undegraded":"2024-03-06T07:35:46.110470+0000","last_fullsized":"2024-03-06T07:35:46.110470+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:10.574091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118676+0000","last_change":"2024-03-06T07:35:45.103565+0000","last_active":"2024-03-06T07:35:46.118676+0000","last_peered":"2024-03-06T07:35:46.118676+0000","last_clean":"2024-03-06T07:35:46.118676+0000","last_became_active":"2024-03-06T07:35:45.103297+0000","last_became_peered":"2024-03-06T07:35:45.103297+0000","last_unstale":"2024-03-06T07:35:46.118676+0000","last_undegraded":"2024-03-06T07:35:46.118676+0000","last_fullsized":"2024-03-06T07:35:46.118676+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:20.076733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486711+0000","last_change":"2024-03-06T07:35:44.098343+0000","last_active":"2024-03-06T07:35:46.486711+0000","last_peered":"2024-03-06T07:35:46.486711+0000","last_clean":"2024-03-06T07:35:46.486711+0000","last_became_active":"2024-03-06T07:35:44.098064+0000","last_became_peered":"2024-03-06T07:35:44.098064+0000","last_unstale":"2024-03-06T07:35:46.486711+0000","last_undegraded":"2024-03-06T07:35:46.486711+0000","last_fullsized":"2024-03-06T07:35:46.486711+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:02.616516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.489746+0000","last_change":"2024-03-06T07:35:45.114015+0000","last_active":"2024-03-06T07:35:46.489746+0000","last_peered":"2024-03-06T07:35:46.489746+0000","last_clean":"2024-03-06T07:35:46.489746+0000","last_became_active":"2024-03-06T07:35:45.113727+0000","last_became_peered":"2024-03-06T07:35:45.113727+0000","last_unstale":"2024-03-06T07:35:46.489746+0000","last_undegraded":"2024-03-06T07:35:46.489746+0000","last_fullsized":"2024-03-06T07:35:46.489746+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:38.425084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118391+0000","last_change":"2024-03-06T07:35:44.099342+0000","last_active":"2024-03-06T07:35:46.118391+0000","last_peered":"2024-03-06T07:35:46.118391+0000","last_clean":"2024-03-06T07:35:46.118391+0000","last_became_active":"2024-03-06T07:35:44.099043+0000","last_became_peered":"2024-03-06T07:35:44.099043+0000","last_unstale":"2024-03-06T07:35:46.118391+0000","last_undegraded":"2024-03-06T07:35:46.118391+0000","last_fullsized":"2024-03-06T07:35:46.118391+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:33:26.144923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112941+0000","last_change":"2024-03-06T07:35:45.092834+0000","last_active":"2024-03-06T07:35:46.112941+0000","last_peered":"2024-03-06T07:35:46.112941+0000","last_clean":"2024-03-06T07:35:46.112941+0000","last_became_active":"2024-03-06T07:35:45.091939+0000","last_became_peered":"2024-03-06T07:35:45.091939+0000","last_unstale":"2024-03-06T07:35:46.112941+0000","last_undegraded":"2024-03-06T07:35:46.112941+0000","last_fullsized":"2024-03-06T07:35:46.112941+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:48.144647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.112960+0000","last_change":"2024-03-06T07:35:44.094658+0000","last_active":"2024-03-06T07:35:46.112960+0000","last_peered":"2024-03-06T07:35:46.112960+0000","last_clean":"2024-03-06T07:35:46.112960+0000","last_became_active":"2024-03-06T07:35:44.094381+0000","last_became_peered":"2024-03-06T07:35:44.094381+0000","last_unstale":"2024-03-06T07:35:46.112960+0000","last_undegraded":"2024-03-06T07:35:46.112960+0000","last_fullsized":"2024-03-06T07:35:46.112960+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:20.518762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.486870+0000","last_change":"2024-03-06T07:35:45.113950+0000","last_active":"2024-03-06T07:35:46.486870+0000","last_peered":"2024-03-06T07:35:46.486870+0000","last_clean":"2024-03-06T07:35:46.486870+0000","last_became_active":"2024-03-06T07:35:45.113633+0000","last_became_peered":"2024-03-06T07:35:45.113633+0000","last_unstale":"2024-03-06T07:35:46.486870+0000","last_undegraded":"2024-03-06T07:35:46.486870+0000","last_fullsized":"2024-03-06T07:35:46.486870+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:37.536656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.109408+0000","last_change":"2024-03-06T07:35:44.093728+0000","last_active":"2024-03-06T07:35:46.109408+0000","last_peered":"2024-03-06T07:35:46.109408+0000","last_clean":"2024-03-06T07:35:46.109408+0000","last_became_active":"2024-03-06T07:35:44.092689+0000","last_became_peered":"2024-03-06T07:35:44.092689+0000","last_unstale":"2024-03-06T07:35:46.109408+0000","last_undegraded":"2024-03-06T07:35:46.109408+0000","last_fullsized":"2024-03-06T07:35:46.109408+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:12.714112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118244+0000","last_change":"2024-03-06T07:35:45.099419+0000","last_active":"2024-03-06T07:35:46.118244+0000","last_peered":"2024-03-06T07:35:46.118244+0000","last_clean":"2024-03-06T07:35:46.118244+0000","last_became_active":"2024-03-06T07:35:45.099007+0000","last_became_peered":"2024-03-06T07:35:45.099007+0000","last_unstale":"2024-03-06T07:35:46.118244+0000","last_undegraded":"2024-03-06T07:35:46.118244+0000","last_fullsized":"2024-03-06T07:35:46.118244+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:44.051245+0000","last_clean_scrub_stamp":"2024-03-06T07:35:44.051245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:52.866649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"124'2","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.118895+0000","last_change":"2024-03-06T07:35:44.101844+0000","last_active":"2024-03-06T07:35:46.118895+0000","last_peered":"2024-03-06T07:35:46.118895+0000","last_clean":"2024-03-06T07:35:46.118895+0000","last_became_active":"2024-03-06T07:35:44.101424+0000","last_became_peered":"2024-03-06T07:35:44.101424+0000","last_unstale":"2024-03-06T07:35:46.118895+0000","last_undegraded":"2024-03-06T07:35:46.118895+0000","last_fullsized":"2024-03-06T07:35:46.118895+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:43.035732+0000","last_clean_scrub_stamp":"2024-03-06T07:35:43.035732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:15.256314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":360,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:46.117892+0000","last_change":"2024-03-06T07:27:44.878933+0000","last_active":"2024-03-06T07:35:46.117892+0000","last_peered":"2024-03-06T07:35:46.117892+0000","last_clean":"2024-03-06T07:35:46.117892+0000","last_became_active":"2024-03-06T07:27:44.876224+0000","last_became_peered":"2024-03-06T07:27:44.876224+0000","last_unstale":"2024-03-06T07:35:46.117892+0000","last_undegraded":"2024-03-06T07:35:46.117892+0000","last_fullsized":"2024-03-06T07:35:46.117892+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:43.366481+0000","last_clean_scrub_stamp":"2024-03-06T07:27:43.366481+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:47.188896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574948,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37472,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334368,"statfs":{"total":96636764160,"available":96598392832,"internally_reserved":0,"allocated":2179072,"data_stored":1273875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36880,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334960,"statfs":{"total":96636764160,"available":96598999040,"internally_reserved":0,"allocated":1572864,"data_stored":682869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574947,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36876,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334964,"statfs":{"total":96636764160,"available":96599003136,"internally_reserved":0,"allocated":1568768,"data_stored":681369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574947,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1564672,"data_stored":681327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36876,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334964,"statfs":{"total":96636764160,"available":96599003136,"internally_reserved":0,"allocated":1568768,"data_stored":681417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574948,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36876,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334964,"statfs":{"total":96636764160,"available":96599003136,"internally_reserved":0,"allocated":1568768,"data_stored":681873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1560576,"data_stored":681257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37444,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334396,"statfs":{"total":96636764160,"available":96598421504,"internally_reserved":0,"allocated":2154496,"data_stored":1271625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:50.055 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:35:50.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:35:50.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.297+0000 7f5cef800640 1 Processor -- start 2024-03-06T07:35:50.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.297+0000 7f5cef800640 1 -- start start 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.298+0000 7f5cef800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80b3df0 0x7f5ce80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.298+0000 7f5cef800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ce80b3650 0x7f5ce80b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.298+0000 7f5cef800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80b4710 con 0x7f5ce80cda90 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.298+0000 7f5cef800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce8072310 con 0x7f5ce80b3650 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5cef800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80719e0 con 0x7f5ce80b3df0 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ce7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ce80b3650 0x7f5ce80b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ced800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5ce80cda90 0x7f5ce806c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59612/0 (socket says 172.21.15.114:59612) 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ced800640 1 -- 172.21.15.114:0/50886128 learned_addr learned my addr 172.21.15.114:0/50886128 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ce7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ce80b3650 0x7f5ce80b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39694/0 (socket says 172.21.15.114:39694) 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5cece00640 1 --2- 172.21.15.114:0/50886128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80b3df0 0x7f5ce80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ce7e00640 1 -- 172.21.15.114:0/50886128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80b3df0 msgr2=0x7f5ce80727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ce7e00640 1 --2- 172.21.15.114:0/50886128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80b3df0 0x7f5ce80727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ce7e00640 1 -- 172.21.15.114:0/50886128 >> v1:172.21.15.114:6789/0 conn(0x7f5ce80cda90 legacy=0x7f5ce806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.299+0000 7f5ce7e00640 1 -- 172.21.15.114:0/50886128 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8071cc0 con 0x7f5ce80b3650 2024-03-06T07:35:50.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.300+0000 7f5ce7e00640 1 --2- 172.21.15.114:0/50886128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3650 0x7f5ce80b41d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8916649095f15b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5ce7400640 1 -- 172.21.15.114:0/50886128 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4035020 con 0x7f5ce80b3650 2024-03-06T07:35:50.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5ce7400640 1 -- 172.21.15.114:0/50886128 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd4031810 con 0x7f5ce80b3650 2024-03-06T07:35:50.302 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5ce7400640 1 -- 172.21.15.114:0/50886128 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4031be0 con 0x7f5ce80b3650 2024-03-06T07:35:50.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5cef800640 1 -- 172.21.15.114:0/50886128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3650 msgr2=0x7f5ce80b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5cef800640 1 --2- 172.21.15.114:0/50886128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3650 0x7f5ce80b41d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5cef800640 1 -- 172.21.15.114:0/50886128 shutdown_connections 2024-03-06T07:35:50.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5cef800640 1 --2- 172.21.15.114:0/50886128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3650 0x7f5ce80b41d0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.301+0000 7f5cef800640 1 --2- 172.21.15.114:0/50886128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80b3df0 0x7f5ce80727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.302+0000 7f5cef800640 1 -- 172.21.15.114:0/50886128 >> 172.21.15.114:0/50886128 conn(0x7f5ce80b1940 msgr2=0x7f5ce80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.302+0000 7f5cef800640 1 -- 172.21.15.114:0/50886128 shutdown_connections 2024-03-06T07:35:50.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.302+0000 7f5cef800640 1 -- 172.21.15.114:0/50886128 wait complete. 2024-03-06T07:35:50.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.304+0000 7f5cef800640 1 Processor -- start 2024-03-06T07:35:50.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.304+0000 7f5cef800640 1 -- start start 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5cef800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ce80b3650 0x7f5ce81504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5cef800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3df0 0x7f5ce8166f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5cef800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80cda90 0x7f5ce814c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5cef800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce815b240 con 0x7f5ce80b3df0 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5cef800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80b3b30 con 0x7f5ce80b3650 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5cef800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce815b3c0 con 0x7f5ce80cda90 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5ce7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3df0 0x7f5ce8166f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5ce7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3df0 0x7f5ce8166f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39706/0 (socket says 172.21.15.114:39706) 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.306+0000 7f5ce7e00640 1 -- 172.21.15.114:0/3503591784 learned_addr learned my addr 172.21.15.114:0/3503591784 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ced800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80cda90 0x7f5ce814c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5cece00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ce80b3650 0x7f5ce81504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ce7e00640 1 -- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80cda90 msgr2=0x7f5ce814c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ce7e00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80cda90 0x7f5ce814c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ce7e00640 1 -- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ce80b3650 msgr2=0x7f5ce81504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ce7e00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ce80b3650 0x7f5ce81504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ce7e00640 1 -- 172.21.15.114:0/3503591784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce8141170 con 0x7f5ce80b3df0 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ced800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80cda90 0x7f5ce814c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5cece00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ce80b3650 0x7f5ce81504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ce7e00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3df0 0x7f5ce8166f80 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f5cd402bec0 tx=0x7f5cd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.307+0000 7f5ce4c00640 1 -- 172.21.15.114:0/3503591784 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4041070 con 0x7f5ce80b3df0 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.308+0000 7f5ce4c00640 1 -- 172.21.15.114:0/3503591784 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd403ad00 con 0x7f5ce80b3df0 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.308+0000 7f5ce4c00640 1 -- 172.21.15.114:0/3503591784 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4035040 con 0x7f5ce80b3df0 2024-03-06T07:35:50.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.308+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce8141450 con 0x7f5ce80b3df0 2024-03-06T07:35:50.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.308+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce8141a10 con 0x7f5ce80b3df0 2024-03-06T07:35:50.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.309+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce804f330 con 0x7f5ce80b3df0 2024-03-06T07:35:50.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.310+0000 7f5ce4c00640 1 -- 172.21.15.114:0/3503591784 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cd404c060 con 0x7f5ce80b3df0 2024-03-06T07:35:50.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.311+0000 7f5ce4c00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.311+0000 7f5ce4c00640 1 -- 172.21.15.114:0/3503591784 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cd40abe50 con 0x7f5ce80b3df0 2024-03-06T07:35:50.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.311+0000 7f5cece00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.312+0000 7f5cece00640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5cc806b4d0 0x7f5cc806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc006fd0 tx=0x7f5cdc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.316+0000 7f5ce4c00640 1 -- 172.21.15.114:0/3503591784 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cd407acc0 con 0x7f5ce80b3df0 2024-03-06T07:35:50.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.582+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5ce80cc0e0 con 0x7f5ce80b3df0 2024-03-06T07:35:50.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.587+0000 7f5ce4c00640 1 -- 172.21.15.114:0/3503591784 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5cd407a5c0 con 0x7f5ce80b3df0 2024-03-06T07:35:50.588 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-06T07:35:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQC5GuhlRd46FRAAp/IjAkNCnVTPohV559YNBA== 2024-03-06T07:35:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: caps mds = "allow" 2024-03-06T07:35:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: caps mon = "allow r" 2024-03-06T07:35:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:50.589 INFO:teuthology.orchestra.run.smithi114.stderr:updated caps for client.0 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5cc806b4d0 msgr2=0x7f5cc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5cc806b4d0 0x7f5cc806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc006fd0 tx=0x7f5cdc008040 comp rx=0 tx=0).stop 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3df0 msgr2=0x7f5ce8166f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3df0 0x7f5ce8166f80 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f5cd402bec0 tx=0x7f5cd40046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 shutdown_connections 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5ce80cda90 0x7f5ce814c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5cc806b4d0 0x7f5cc806d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80b3df0 0x7f5ce8166f80 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 --2- 172.21.15.114:0/3503591784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ce80b3650 0x7f5ce81504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 >> 172.21.15.114:0/3503591784 conn(0x7f5ce80b1940 msgr2=0x7f5ce80ce8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 shutdown_connections 2024-03-06T07:35:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.590+0000 7f5cef800640 1 -- 172.21.15.114:0/3503591784 wait complete. 2024-03-06T07:35:50.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.810+0000 7f45a8400640 1 Processor -- start 2024-03-06T07:35:50.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.810+0000 7f45a8400640 1 -- start start 2024-03-06T07:35:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.812+0000 7f45a8400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f45a006ec00 0x7f45a00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.812+0000 7f45a8400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a015b090 con 0x7f45a00b4680 2024-03-06T07:35:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.812+0000 7f45a8400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00b2a30 con 0x7f45a015acb0 2024-03-06T07:35:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.812+0000 7f45a8400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a00b4360 con 0x7f45a006ec00 2024-03-06T07:35:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.812+0000 7f45a5000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f45a00b4680 0x7f45a006e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59620/0 (socket says 172.21.15.114:59620) 2024-03-06T07:35:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.812+0000 7f45a5000640 1 -- 172.21.15.114:0/2636099827 learned_addr learned my addr 172.21.15.114:0/2636099827 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f45a6400640 1 --2- 172.21.15.114:0/2636099827 >> v2:172.21.15.177:3300/0 conn(0x7f45a006ec00 0x7f45a00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506727336 0 0) 0x7f45a015b090 con 0x7f45a00b4680 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588005730 con 0x7f45a00b4680 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3709992949 0 0) 0x7f4588005730 con 0x7f45a00b4680 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 >> v2:172.21.15.177:3300/0 conn(0x7f45a006ec00 msgr2=0x7f45a00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 --2- 172.21.15.114:0/2636099827 >> v2:172.21.15.177:3300/0 conn(0x7f45a006ec00 0x7f45a00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 >> v1:172.21.15.177:6789/0 conn(0x7f45a015acb0 legacy=0x7f45a00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a00b2cf0 con 0x7f45a00b4680 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f458c001670 con 0x7f45a00b4680 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 >> v1:172.21.15.114:6789/0 conn(0x7f45a00b4680 legacy=0x7f45a006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.813+0000 7f459fe00640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4588005bc0 0x7f4588005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f459fe00640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4588006a10 0x7f4588006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f459fe00640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880073d0 0x7f458800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00b4360 con 0x7f45880073d0 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458800e270 con 0x7f4588005bc0 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458800e470 con 0x7f4588006a10 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a6400640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880073d0 0x7f458800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a5a00640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4588005bc0 0x7f4588005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a5000640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4588006a10 0x7f4588006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a6400640 1 -- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4588006a10 msgr2=0x7f4588006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a6400640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4588006a10 0x7f4588006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a6400640 1 -- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4588005bc0 msgr2=0x7f4588005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a6400640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4588005bc0 0x7f4588005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.814+0000 7f45a6400640 1 -- 172.21.15.114:0/2636099827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588005730 con 0x7f45880073d0 2024-03-06T07:35:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.815+0000 7f45a6400640 1 -- 172.21.15.114:0/2636099827 reap_dead start 2024-03-06T07:35:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.815+0000 7f45a6400640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880073d0 0x7f458800dbd0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f4598009fc0 tx=0x7f4588007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff266dd6eb549d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598009c10 con 0x7f45880073d0 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459802cca0 con 0x7f45880073d0 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f459fe00640 1 -- 172.21.15.114:0/2636099827 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4598003bd0 con 0x7f45880073d0 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f45a8400640 1 -- 172.21.15.114:0/2636099827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880073d0 msgr2=0x7f458800dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f45a8400640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880073d0 0x7f458800dbd0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f4598009fc0 tx=0x7f4588007a40 comp rx=0 tx=0).stop 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f45a8400640 1 -- 172.21.15.114:0/2636099827 shutdown_connections 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f45a8400640 1 --2- 172.21.15.114:0/2636099827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880073d0 0x7f458800dbd0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f45a8400640 1 -- 172.21.15.114:0/2636099827 >> 172.21.15.114:0/2636099827 conn(0x7f45a00b1730 msgr2=0x7f45a00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f45a8400640 1 -- 172.21.15.114:0/2636099827 shutdown_connections 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.816+0000 7f45a8400640 1 -- 172.21.15.114:0/2636099827 wait complete. 2024-03-06T07:35:50.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.818+0000 7f45a8400640 1 Processor -- start 2024-03-06T07:35:50.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.818+0000 7f45a8400640 1 -- start start 2024-03-06T07:35:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.819+0000 7f45a8400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45880073d0 0x7f45a00cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.819+0000 7f45a8400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a01569f0 0x7f45a0156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.819+0000 7f45a8400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45a0157390 0x7f45a0159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.819+0000 7f45a8400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00b4010 con 0x7f45a01569f0 2024-03-06T07:35:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.819+0000 7f45a8400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00b32b0 con 0x7f45880073d0 2024-03-06T07:35:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.819+0000 7f45a8400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0072340 con 0x7f45a0157390 2024-03-06T07:35:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.819+0000 7f45a5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a01569f0 0x7f45a0156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a01569f0 0x7f45a0156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39732/0 (socket says 172.21.15.114:39732) 2024-03-06T07:35:50.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 -- 172.21.15.114:0/3001034240 learned_addr learned my addr 172.21.15.114:0/3001034240 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:50.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 -- 172.21.15.114:0/3001034240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45a0157390 msgr2=0x7f45a0159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45a0157390 0x7f45a0159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 -- 172.21.15.114:0/3001034240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45880073d0 msgr2=0x7f45a00cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45880073d0 0x7f45a00cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 -- 172.21.15.114:0/3001034240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0157810 con 0x7f45a01569f0 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f45a5000640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a01569f0 0x7f45a0156e50 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f458c000fb0 tx=0x7f458c00eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f459d600640 1 -- 172.21.15.114:0/3001034240 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c00c630 con 0x7f45a01569f0 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.820+0000 7f459d600640 1 -- 172.21.15.114:0/3001034240 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c018070 con 0x7f45a01569f0 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.821+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a01478a0 con 0x7f45a01569f0 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.821+0000 7f459d600640 1 -- 172.21.15.114:0/3001034240 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c00ca10 con 0x7f45a01569f0 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.821+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a0158a60 con 0x7f45a01569f0 2024-03-06T07:35:50.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.822+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a0147dc0 con 0x7f45a01569f0 2024-03-06T07:35:50.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.823+0000 7f459d600640 1 -- 172.21.15.114:0/3001034240 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f458c00cbb0 con 0x7f45a01569f0 2024-03-06T07:35:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.824+0000 7f459d600640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458006b4d0 0x7f458006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.824+0000 7f459d600640 1 -- 172.21.15.114:0/3001034240 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f458c089fd0 con 0x7f45a01569f0 2024-03-06T07:35:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.824+0000 7f45a5a00640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458006b4d0 0x7f458006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.825+0000 7f45a5a00640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458006b4d0 0x7f458006d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4594007fd0 tx=0x7f4594009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:50.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:50.828+0000 7f459d600640 1 -- 172.21.15.114:0/3001034240 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45a0147dc0 con 0x7f45a01569f0 2024-03-06T07:35:51.029 INFO:tasks.ceph.mds.c.smithi114.stderr:2024-03-06T07:35:51.027+0000 7f5550800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:51.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.059+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45a004f330 con 0x7f45a01569f0 2024-03-06T07:35:51.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.059+0000 7f459d600640 1 -- 172.21.15.114:0/3001034240 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f458c058e50 con 0x7f45a01569f0 2024-03-06T07:35:51.061 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:51.061 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":96,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1958627905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1958627905},{"type":"v1","addr":"172.21.15.177:6833","nonce":1958627905}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":5765,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/7846601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":7846601},{"type":"v1","addr":"172.21.15.177:6835","nonce":7846601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":8328,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:47.609493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8331},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:35:51.062 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 96 2024-03-06T07:35:51.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458006b4d0 msgr2=0x7f458006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458006b4d0 0x7f458006d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4594007fd0 tx=0x7f4594009040 comp rx=0 tx=0).stop 2024-03-06T07:35:51.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a01569f0 msgr2=0x7f45a0156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a01569f0 0x7f45a0156e50 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f458c000fb0 tx=0x7f458c00eba0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 shutdown_connections 2024-03-06T07:35:51.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f45a0157390 0x7f45a0159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f458006b4d0 0x7f458006d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a01569f0 0x7f45a0156e50 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 --2- 172.21.15.114:0/3001034240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45880073d0 0x7f45a00cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 >> 172.21.15.114:0/3001034240 conn(0x7f45a00b1730 msgr2=0x7f45a00cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 shutdown_connections 2024-03-06T07:35:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.062+0000 7f45a8400640 1 -- 172.21.15.114:0/3001034240 wait complete. 2024-03-06T07:35:51.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:35:47.609493+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8331}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:35:51.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:35:51.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:35:51.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.707+0000 7fea9ae00640 1 Processor -- start 2024-03-06T07:35:51.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.708+0000 7fea9ae00640 1 -- start start 2024-03-06T07:35:51.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.710+0000 7fea9ae00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fea9415e2c0 0x7fea940cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.710+0000 7fea9ae00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea940cc860 0x7fea940ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.710+0000 7fea9ae00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea940b4570 con 0x7fea9406c0d0 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.710+0000 7fea9ae00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea940b4320 con 0x7fea940cc860 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.710+0000 7fea9ae00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea940d05d0 con 0x7fea9415e2c0 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea98e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea940cc860 0x7fea940ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea98e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea940cc860 0x7fea940ce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39744/0 (socket says 172.21.15.114:39744) 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea98e00640 1 -- 172.21.15.114:0/2911891389 learned_addr learned my addr 172.21.15.114:0/2911891389 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea93e00640 1 --2- 172.21.15.114:0/2911891389 >> v2:172.21.15.177:3300/0 conn(0x7fea9415e2c0 0x7fea940cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea98e00640 1 -- 172.21.15.114:0/2911891389 >> v2:172.21.15.177:3300/0 conn(0x7fea9415e2c0 msgr2=0x7fea940cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea98e00640 1 --2- 172.21.15.114:0/2911891389 >> v2:172.21.15.177:3300/0 conn(0x7fea9415e2c0 0x7fea940cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea98e00640 1 -- 172.21.15.114:0/2911891389 >> v1:172.21.15.114:6789/0 conn(0x7fea9406c0d0 legacy=0x7fea940d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.711+0000 7fea98e00640 1 -- 172.21.15.114:0/2911891389 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940d0260 con 0x7fea940cc860 2024-03-06T07:35:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.712+0000 7fea98e00640 1 --2- 172.21.15.114:0/2911891389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea940cc860 0x7fea940ce550 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fea94067ce0 tx=0x7fea8c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc89ee9f54fe0b50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.712+0000 7fea93400640 1 -- 172.21.15.114:0/2911891389 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c02cb20 con 0x7fea940cc860 2024-03-06T07:35:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea93400640 1 -- 172.21.15.114:0/2911891389 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c02ccc0 con 0x7fea940cc860 2024-03-06T07:35:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea93400640 1 -- 172.21.15.114:0/2911891389 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c034610 con 0x7fea940cc860 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 -- 172.21.15.114:0/2911891389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea940cc860 msgr2=0x7fea940ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2911891389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea940cc860 0x7fea940ce550 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fea94067ce0 tx=0x7fea8c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 -- 172.21.15.114:0/2911891389 shutdown_connections 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2911891389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea940cc860 0x7fea940ce550 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2911891389 >> v2:172.21.15.177:3300/0 conn(0x7fea9415e2c0 0x7fea940cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 -- 172.21.15.114:0/2911891389 >> 172.21.15.114:0/2911891389 conn(0x7fea940b3800 msgr2=0x7fea940b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 -- 172.21.15.114:0/2911891389 shutdown_connections 2024-03-06T07:35:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.713+0000 7fea9ae00640 1 -- 172.21.15.114:0/2911891389 wait complete. 2024-03-06T07:35:51.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.715+0000 7fea9ae00640 1 Processor -- start 2024-03-06T07:35:51.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.715+0000 7fea9ae00640 1 -- start start 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea9ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9406c0d0 0x7fea941f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea9ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea940cc860 0x7fea941f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea9ae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea9415e2c0 0x7fea940b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea9ae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea940d1170 con 0x7fea9415e2c0 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea9ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea940b0e20 con 0x7fea9406c0d0 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea9ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940d12f0 con 0x7fea940cc860 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea98e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea9415e2c0 0x7fea940b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea98e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea9415e2c0 0x7fea940b48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39746/0 (socket says 172.21.15.114:39746) 2024-03-06T07:35:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.716+0000 7fea98e00640 1 -- 172.21.15.114:0/2394495736 learned_addr learned my addr 172.21.15.114:0/2394495736 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea93e00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9406c0d0 0x7fea941f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea8b400640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea940cc860 0x7fea941f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea98e00640 1 -- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea940cc860 msgr2=0x7fea941f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea98e00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea940cc860 0x7fea941f7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea98e00640 1 -- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9406c0d0 msgr2=0x7fea941f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea98e00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9406c0d0 0x7fea941f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea98e00640 1 -- 172.21.15.114:0/2394495736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea941f86e0 con 0x7fea9415e2c0 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea93e00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9406c0d0 0x7fea941f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea8b400640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea940cc860 0x7fea941f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea98e00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea9415e2c0 0x7fea940b48d0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fea8c036040 tx=0x7fea8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea90c00640 1 -- 172.21.15.114:0/2394495736 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c043020 con 0x7fea9415e2c0 2024-03-06T07:35:51.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea90c00640 1 -- 172.21.15.114:0/2394495736 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c03e450 con 0x7fea9415e2c0 2024-03-06T07:35:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.717+0000 7fea90c00640 1 -- 172.21.15.114:0/2394495736 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c009510 con 0x7fea9415e2c0 2024-03-06T07:35:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.718+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea941f89c0 con 0x7fea9415e2c0 2024-03-06T07:35:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.718+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea94146db0 con 0x7fea9415e2c0 2024-03-06T07:35:51.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.720+0000 7fea90c00640 1 -- 172.21.15.114:0/2394495736 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea8c046020 con 0x7fea9415e2c0 2024-03-06T07:35:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.720+0000 7fea90c00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.721+0000 7fea90c00640 1 -- 172.21.15.114:0/2394495736 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea8c0acad0 con 0x7fea9415e2c0 2024-03-06T07:35:51.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.721+0000 7fea93e00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.722+0000 7fea93e00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b4d0 0x7fea7006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fea84006fd0 tx=0x7fea84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.722+0000 7fea9dc1c740 1 Processor -- start 2024-03-06T07:35:51.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.722+0000 7fea9dc1c740 1 -- start start 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409b7b0 0x55757406e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 0x55757409e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757406ec90 0x55757409ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557573fd8600 con 0x55757409b7b0 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557573f81930 con 0x55757409bb90 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x557573fd46a0 con 0x55757406ec90 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409b7b0 0x55757406e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409b7b0 0x55757406e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39750/0 (socket says 172.21.15.114:39750) 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 -- 172.21.15.114:0/3817716716 learned_addr learned my addr 172.21.15.114:0/3817716716 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 -- 172.21.15.114:0/3817716716 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757406ec90 msgr2=0x55757409ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 --2- 172.21.15.114:0/3817716716 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757406ec90 0x55757409ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 -- 172.21.15.114:0/3817716716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 msgr2=0x55757409e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 --2- 172.21.15.114:0/3817716716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 0x55757409e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.723+0000 7fea93e00640 1 -- 172.21.15.114:0/3817716716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557573ced2d0 con 0x55757409b7b0 2024-03-06T07:35:51.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.724+0000 7fea93e00640 1 --2- 172.21.15.114:0/3817716716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409b7b0 0x55757406e750 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fea84004380 tx=0x7fea84011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91cb358ca126c134 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.724+0000 7fea88c00640 1 -- 172.21.15.114:0/3817716716 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8400f040 con 0x55757409b7b0 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.724+0000 7fea88c00640 1 -- 172.21.15.114:0/3817716716 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea840043b0 con 0x55757409b7b0 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.724+0000 7fea88c00640 1 -- 172.21.15.114:0/3817716716 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8400ee80 con 0x55757409b7b0 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.724+0000 7fea9dc1c740 1 -- 172.21.15.114:0/3817716716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409b7b0 msgr2=0x55757406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.724+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/3817716716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409b7b0 0x55757406e750 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fea84004380 tx=0x7fea84011000 comp rx=0 tx=0).stop 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 -- 172.21.15.114:0/3817716716 shutdown_connections 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/3817716716 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757406ec90 0x55757409ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/3817716716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 0x55757409e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/3817716716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409b7b0 0x55757406e750 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 -- 172.21.15.114:0/3817716716 >> 172.21.15.114:0/3817716716 conn(0x5575740709c0 msgr2=0x5575740b9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 -- 172.21.15.114:0/3817716716 shutdown_connections 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 -- 172.21.15.114:0/3817716716 wait complete. 2024-03-06T07:35:51.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.725+0000 7fea9dc1c740 1 -- start start 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.726+0000 7fea9dc1c740 1 Processor -- start 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.726+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 0x5575740b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.726+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409bb90 0x5575740b7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.726+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5575740b48f0 0x55757408c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.726+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557574072f60 con 0x55757409bb90 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.726+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x557573f94d90 con 0x5575740b48f0 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.726+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x557574072350 con 0x55757409b7b0 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea9dc1c740 10 client.0 _ll_register_callbacks cb 0x557573fd0350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409bb90 0x5575740b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409bb90 0x5575740b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39766/0 (socket says 172.21.15.114:39766) 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 -- 172.21.15.114:0/1999061647 learned_addr learned my addr 172.21.15.114:0/1999061647 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea93e00640 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 0x5575740b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea98e00640 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5575740b48f0 0x55757408c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 -- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 msgr2=0x5575740b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 0x5575740b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 -- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5575740b48f0 msgr2=0x55757408c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5575740b48f0 0x55757408c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.727+0000 7fea8b400640 1 -- 172.21.15.114:0/1999061647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557573db76b0 con 0x55757409bb90 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea8b400640 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409bb90 0x5575740b7790 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fea7c001c60 tx=0x7fea7c00d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7a4f6507fe0d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea81600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea81600640 1 -- 172.21.15.114:0/1999061647 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c018060 con 0x55757409bb90 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea81600640 1 -- 172.21.15.114:0/1999061647 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea7c0052d0 con 0x55757409bb90 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea840043b0 con 0x55757409bb90 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea81600640 1 -- 172.21.15.114:0/1999061647 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea7c00de40 con 0x55757409bb90 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea9dc1c740 10 client.8472 fetch_fsmap learned FSMap version 96 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) v3 -- 0x55757408cf00 con 0x55757409bb90 2024-03-06T07:35:51.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.728+0000 7fea81600640 1 -- 172.21.15.114:0/1999061647 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea840043b0 con 0x55757409bb90 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea81600640 1 -- 172.21.15.114:0/1999061647 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 96) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fea7c0067d0 con 0x55757409bb90 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea9dc1c740 10 client.8472 fetch_fsmap finished waiting for FSMap version 96 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea9dc1c740 10 client.8472 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea9dc1c740 20 client.8472 populate_metadata read hostname 'smithi114' 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x5575740a1950 0x5575740cd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea9dc1c740 4 client.8472 mds_command: new command op to 8331 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 --> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557573fe5b40 con 0x5575740a1950 2024-03-06T07:35:51.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea93e00640 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x5575740a1950 0x5575740cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea93e00640 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x5575740a1950 0x5575740cd320 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9556c49f11b497b0 server_cookie=d736183fe40f2fbe in_seq=0 out_seq=0 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.729+0000 7fea81600640 10 client.8472 ms_handle_connect on v2:172.21.15.114:6835/1465597546 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.730+0000 7fea81600640 1 -- 172.21.15.114:0/1999061647 <== mds.0 v2:172.21.15.114:6835/1465597546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557573fe5b40 con 0x5575740a1950 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.730+0000 7fea81600640 10 client.8472 handle_command_reply: tid=0 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.730+0000 7fea9dc1c740 1 client.8472 shutdown 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.730+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409bb90 msgr2=0x5575740b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.730+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409bb90 0x5575740b7790 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fea7c001c60 tx=0x7fea7c00d520 comp rx=0 tx=0).stop 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 shutdown_connections 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x5575740a1950 0x5575740cd320 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5575740b48f0 0x55757408c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55757409bb90 0x5575740b7790 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/1999061647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 0x5575740b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea81600640 0 client.8472 ms_handle_reset on v2:172.21.15.114:6835/1465597546 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 >> 172.21.15.114:0/1999061647 conn(0x55757406ec90 msgr2=0x5575740bca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 shutdown_connections 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 1 -- 172.21.15.114:0/1999061647 wait complete. 2024-03-06T07:35:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.731+0000 7fea9dc1c740 20 client.8472 trim_cache size 0 max 16384 2024-03-06T07:35:51.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.749+0000 7fea9dc1c740 1 Processor -- start 2024-03-06T07:35:51.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.749+0000 7fea9dc1c740 1 -- start start 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 0x5575740bca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 0x5575740bcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea940b7240 con 0x5575740b48f0 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea94149d80 con 0x55757409bb90 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940b6c40 con 0x55757409b7b0 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740722b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39782/0 (socket says 172.21.15.114:39782) 2024-03-06T07:35:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 -- 172.21.15.114:0/768709245 learned_addr learned my addr 172.21.15.114:0/768709245 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 -- 172.21.15.114:0/768709245 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 msgr2=0x5575740bca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 --2- 172.21.15.114:0/768709245 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 0x5575740bca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 -- 172.21.15.114:0/768709245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 msgr2=0x5575740bcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 --2- 172.21.15.114:0/768709245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 0x5575740bcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.750+0000 7fea98e00640 1 -- 172.21.15.114:0/768709245 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575740a25a0 con 0x5575740b48f0 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.751+0000 7fea98e00640 1 --2- 172.21.15.114:0/768709245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740722b0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fea8c07aeb0 tx=0x7fea8c07adb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aace19a0d6b387ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.751+0000 7fea83e00640 1 -- 172.21.15.114:0/768709245 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c090070 con 0x5575740b48f0 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.751+0000 7fea83e00640 1 -- 172.21.15.114:0/768709245 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c07b1b0 con 0x5575740b48f0 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.751+0000 7fea83e00640 1 -- 172.21.15.114:0/768709245 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c07f300 con 0x5575740b48f0 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 -- 172.21.15.114:0/768709245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 msgr2=0x5575740722b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/768709245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740722b0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fea8c07aeb0 tx=0x7fea8c07adb0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 -- 172.21.15.114:0/768709245 shutdown_connections 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/768709245 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740722b0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/768709245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409bb90 0x5575740bcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/768709245 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409b7b0 0x5575740bca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 -- 172.21.15.114:0/768709245 >> 172.21.15.114:0/768709245 conn(0x55757406ec90 msgr2=0x5575740b7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 -- 172.21.15.114:0/768709245 shutdown_connections 2024-03-06T07:35:51.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 -- 172.21.15.114:0/768709245 wait complete. 2024-03-06T07:35:51.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.752+0000 7fea9dc1c740 1 -- start start 2024-03-06T07:35:51.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.753+0000 7fea9dc1c740 1 Processor -- start 2024-03-06T07:35:51.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409b7b0 0x557574128690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409bb90 0x557574073f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea9dc1c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740bd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575740a17d0 con 0x5575740b48f0 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575740a1440 con 0x55757409b7b0 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea9dc1c740 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x5575740a1dd0 con 0x55757409bb90 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea9dc1c740 10 client.0 _ll_register_callbacks cb 0x7fea7c007ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740bd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740bd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39784/0 (socket says 172.21.15.114:39784) 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 -- 172.21.15.114:0/2596079682 learned_addr learned my addr 172.21.15.114:0/2596079682 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 -- 172.21.15.114:0/2596079682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409bb90 msgr2=0x557574073f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409bb90 0x557574073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 -- 172.21.15.114:0/2596079682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409b7b0 msgr2=0x557574128690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409b7b0 0x557574128690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 -- 172.21.15.114:0/2596079682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575740b9940 con 0x5575740b48f0 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea98e00640 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740bd200 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fea8c07aeb0 tx=0x7fea8c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1ec617b44761bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.754+0000 7fea81600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea81600640 1 -- 172.21.15.114:0/2596079682 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c097060 con 0x5575740b48f0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea81600640 1 -- 172.21.15.114:0/2596079682 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea8c083430 con 0x5575740b48f0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea8c07b1b0 con 0x5575740b48f0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea81600640 1 -- 172.21.15.114:0/2596079682 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea8c090040 con 0x5575740b48f0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 10 client.8478 fetch_fsmap learned FSMap version 96 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) v3 -- 0x5575740a0150 con 0x5575740b48f0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea81600640 1 -- 172.21.15.114:0/2596079682 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c07b1b0 con 0x5575740b48f0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea81600640 1 -- 172.21.15.114:0/2596079682 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 96) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fea8c02cb30 con 0x5575740b48f0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 10 client.8478 fetch_fsmap finished waiting for FSMap version 96 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 10 client.8478 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 20 client.8478 populate_metadata read hostname 'smithi114' 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x55757409f610 0x55757409f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 4 client.8478 mds_command: new command op to 8331 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.755+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 --> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557573fdfe90 con 0x55757409f610 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea93e00640 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x55757409f610 0x55757409f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea93e00640 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x55757409f610 0x55757409f9f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93aabaeb8d02ed96 server_cookie=611952a3069cf7f6 in_seq=0 out_seq=0 2024-03-06T07:35:51.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea81600640 10 client.8478 ms_handle_connect on v2:172.21.15.114:6835/1465597546 2024-03-06T07:35:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea81600640 1 -- 172.21.15.114:0/2596079682 <== mds.0 v2:172.21.15.114:6835/1465597546 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x557573fdfe90 con 0x55757409f610 2024-03-06T07:35:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea81600640 10 client.8478 handle_command_reply: tid=0 2024-03-06T07:35:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 client.8478 shutdown 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 msgr2=0x5575740bd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740bd200 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fea8c07aeb0 tx=0x7fea8c009810 comp rx=0 tx=0).stop 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 shutdown_connections 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:6835/1465597546,v1:172.21.15.114:6837/1465597546] conn(0x55757409f610 0x55757409f9f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575740b48f0 0x5575740bd200 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x55757409bb90 0x557574073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 --2- 172.21.15.114:0/2596079682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55757409b7b0 0x557574128690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 >> 172.21.15.114:0/2596079682 conn(0x55757406ec90 msgr2=0x5575740cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.756+0000 7fea81600640 0 client.8478 ms_handle_reset on v2:172.21.15.114:6835/1465597546 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 shutdown_connections 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9dc1c740 1 -- 172.21.15.114:0/2596079682 wait complete. 2024-03-06T07:35:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9dc1c740 20 client.8478 trim_cache size 0 max 16384 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "149c34a4-b335-4a4b-8a41-0fe21b0db808", 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 8331, 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 6.6604195549999998, 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 96, 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 125, 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 125, 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 12.130346100000001 2024-03-06T07:35:51.762 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-06T07:35:51.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b4d0 msgr2=0x7fea7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b4d0 0x7fea7006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fea84006fd0 tx=0x7fea84008040 comp rx=0 tx=0).stop 2024-03-06T07:35:51.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea9415e2c0 msgr2=0x7fea940b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:51.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea9415e2c0 0x7fea940b48d0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fea8c036040 tx=0x7fea8c004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:51.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 shutdown_connections 2024-03-06T07:35:51.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea9415e2c0 0x7fea940b48d0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fea940cc860 0x7fea941f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 --2- 172.21.15.114:0/2394495736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea9406c0d0 0x7fea941f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:51.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.757+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 >> 172.21.15.114:0/2394495736 conn(0x7fea940b3800 msgr2=0x7fea9415d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:51.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.758+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 shutdown_connections 2024-03-06T07:35:51.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.758+0000 7fea9ae00640 1 -- 172.21.15.114:0/2394495736 wait complete. 2024-03-06T07:35:51.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:35:51.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.994+0000 7ff0b7e00640 1 Processor -- start 2024-03-06T07:35:51.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.995+0000 7ff0b7e00640 1 -- start start 2024-03-06T07:35:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.997+0000 7ff0b7e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff0b80cc9d0 0x7ff0b806d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.997+0000 7ff0b7e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b815c240 con 0x7ff0b81561f0 2024-03-06T07:35:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.997+0000 7ff0b7e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b815c500 con 0x7ff0b80aedc0 2024-03-06T07:35:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.997+0000 7ff0b7e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b815ce30 con 0x7ff0b80cc9d0 2024-03-06T07:35:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.997+0000 7ff0b6a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff0b81561f0 0x7ff0b80ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59630/0 (socket says 172.21.15.114:59630) 2024-03-06T07:35:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.997+0000 7ff0b6a00640 1 -- 172.21.15.114:0/3925210919 learned_addr learned my addr 172.21.15.114:0/3925210919 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:52.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456483289 0 0) 0x7ff0b815c240 con 0x7ff0b81561f0 2024-03-06T07:35:52.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09c003520 con 0x7ff0b81561f0 2024-03-06T07:35:52.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b7400640 1 --2- 172.21.15.114:0/3925210919 >> v2:172.21.15.177:3300/0 conn(0x7ff0b80cc9d0 0x7ff0b806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1380688881 0 0) 0x7ff09c003520 con 0x7ff0b81561f0 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 >> v2:172.21.15.177:3300/0 conn(0x7ff0b80cc9d0 msgr2=0x7ff0b806d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 --2- 172.21.15.114:0/3925210919 >> v2:172.21.15.177:3300/0 conn(0x7ff0b80cc9d0 0x7ff0b806d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 >> v1:172.21.15.177:6789/0 conn(0x7ff0b80aedc0 legacy=0x7ff0b80cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b815c7e0 con 0x7ff0b81561f0 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7ff0a80019f0 con 0x7ff0b81561f0 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 >> v1:172.21.15.114:6789/0 conn(0x7ff0b81561f0 legacy=0x7ff0b80ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c003a00 0x7ff09c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09c004800 0x7ff09c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.998+0000 7ff0b6000640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff09c005230 0x7ff09c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b815ce30 con 0x7ff09c004800 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c00c190 con 0x7ff09c005230 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c00c390 con 0x7ff09c003a00 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0ae000640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09c004800 0x7ff09c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0b6a00640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c003a00 0x7ff09c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0ae000640 1 -- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c003a00 msgr2=0x7ff09c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0ae000640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c003a00 0x7ff09c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0ae000640 1 -- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff09c005230 msgr2=0x7ff09c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0b7400640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff09c005230 0x7ff09c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0ae000640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff09c005230 0x7ff09c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0ae000640 1 -- 172.21.15.114:0/3925210919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c003520 con 0x7ff09c004800 2024-03-06T07:35:52.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0b7400640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff09c005230 0x7ff09c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:52.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:51.999+0000 7ff0b7400640 1 -- 172.21.15.114:0/3925210919 reap_dead start 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.001+0000 7ff0ae000640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09c004800 0x7ff09c004c80 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0001750 tx=0x7ff09c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4819fc78a3b5f4e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.001+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0001370 con 0x7ff09c004800 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.001+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a00052d0 con 0x7ff09c004800 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.001+0000 7ff0b6000640 1 -- 172.21.15.114:0/3925210919 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0003830 con 0x7ff09c004800 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.002+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3925210919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09c004800 msgr2=0x7ff09c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.002+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09c004800 0x7ff09c004c80 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0001750 tx=0x7ff09c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.002+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3925210919 shutdown_connections 2024-03-06T07:35:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.002+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3925210919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff09c004800 0x7ff09c004c80 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.002+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3925210919 >> 172.21.15.114:0/3925210919 conn(0x7ff0b806afc0 msgr2=0x7ff0b806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.002+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3925210919 shutdown_connections 2024-03-06T07:35:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.002+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3925210919 wait complete. 2024-03-06T07:35:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.004+0000 7ff0b7e00640 1 Processor -- start 2024-03-06T07:35:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.004+0000 7ff0b7e00640 1 -- start start 2024-03-06T07:35:52.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c004800 0x7ff0b8155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0b81579b0 0x7ff0b8158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0b8159120 0x7ff0b80b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8156750 con 0x7ff0b8159120 2024-03-06T07:35:52.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81565d0 con 0x7ff0b81579b0 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b80cc2c0 con 0x7ff09c004800 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0b8159120 0x7ff0b80b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0b8159120 0x7ff0b80b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39792/0 (socket says 172.21.15.114:39792) 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b7400640 1 -- 172.21.15.114:0/3914430894 learned_addr learned my addr 172.21.15.114:0/3914430894 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.005+0000 7ff0b6a00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c004800 0x7ff0b8155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0b7400640 1 -- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c004800 msgr2=0x7ff0b8155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0ae000640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0b81579b0 0x7ff0b8158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0b7400640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c004800 0x7ff0b8155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0b7400640 1 -- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0b81579b0 msgr2=0x7ff0b8158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0b7400640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0b81579b0 0x7ff0b8158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0b7400640 1 -- 172.21.15.114:0/3914430894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b8155a50 con 0x7ff0b8159120 2024-03-06T07:35:52.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0b7400640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0b8159120 0x7ff0b80b04b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0006fd0 tx=0x7ff0b000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:52.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0afe00640 1 -- 172.21.15.114:0/3914430894 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b000c070 con 0x7ff0b8159120 2024-03-06T07:35:52.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0afe00640 1 -- 172.21.15.114:0/3914430894 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b0018070 con 0x7ff0b8159120 2024-03-06T07:35:52.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0afe00640 1 -- 172.21.15.114:0/3914430894 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b000f040 con 0x7ff0b8159120 2024-03-06T07:35:52.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.006+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b8156c30 con 0x7ff0b8159120 2024-03-06T07:35:52.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.007+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b81571b0 con 0x7ff0b8159120 2024-03-06T07:35:52.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.008+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b804f330 con 0x7ff0b8159120 2024-03-06T07:35:52.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.008+0000 7ff0afe00640 1 -- 172.21.15.114:0/3914430894 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0b0003610 con 0x7ff0b8159120 2024-03-06T07:35:52.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.009+0000 7ff0afe00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff09406b4d0 0x7ff09406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.009+0000 7ff0afe00640 1 -- 172.21.15.114:0/3914430894 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0b0089b20 con 0x7ff0b8159120 2024-03-06T07:35:52.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.010+0000 7ff0b6a00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff09406b4d0 0x7ff09406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.010+0000 7ff0b6a00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff09406b4d0 0x7ff09406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80013d0 tx=0x7ff0a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:52.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.015+0000 7ff0afe00640 1 -- 172.21.15.114:0/3914430894 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0b8155a50 con 0x7ff0b8159120 2024-03-06T07:35:52.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.279+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff0b806ce70 con 0x7ff0b8159120 2024-03-06T07:35:52.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.280+0000 7ff0afe00640 1 -- 172.21.15.114:0/3914430894 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v96) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff0b0058900 con 0x7ff0b8159120 2024-03-06T07:35:52.281 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:35:52.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff09406b4d0 msgr2=0x7ff09406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff09406b4d0 0x7ff09406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff0a80013d0 tx=0x7ff0a8009040 comp rx=0 tx=0).stop 2024-03-06T07:35:52.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0b8159120 msgr2=0x7ff0b80b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0b8159120 0x7ff0b80b04b0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0006fd0 tx=0x7ff0b000e830 comp rx=0 tx=0).stop 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 shutdown_connections 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff09406b4d0 0x7ff09406d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0b8159120 0x7ff0b80b04b0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff0b81579b0 0x7ff0b8158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 --2- 172.21.15.114:0/3914430894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff09c004800 0x7ff0b8155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 >> 172.21.15.114:0/3914430894 conn(0x7ff0b806afc0 msgr2=0x7ff0b806d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.282+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 shutdown_connections 2024-03-06T07:35:52.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:52.283+0000 7ff0b7e00640 1 -- 172.21.15.114:0/3914430894 wait complete. 2024-03-06T07:35:52.303 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:35:52.303 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:35:52.303 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-03-06T07:35:52.303 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:35:52.303 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:52.304 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:35:52.304 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:35:52.304 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:35:52.304 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:52.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.304 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-03-06T07:35:52.351 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42103sec preferred_lft 42103sec 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link proto kernel_ll 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::e4ca:6dff:fe69:cdb1/64 scope link proto kernel_ll 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether a6:34:e1:e7:c4:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:35:52.352 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::a434:e1ff:fee7:c486/64 scope link proto kernel_ll 2024-03-06T07:35:52.353 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:52.353 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:52.353 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 72:50:2a:6f:c0:bb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:35:52.353 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::7050:2aff:fe6f:c0bb/64 scope link proto kernel_ll 2024-03-06T07:35:52.353 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:52.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.354 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:35:52.480 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-03-06T07:35:52.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.484 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-03-06T07:35:52.530 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-03-06T07:35:52.530 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:35:52.530 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:35:52.530 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:35:52.530 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:35:52.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.530 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-03-06T07:35:52.530 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-03-06T07:35:52.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:35:52.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:35:52.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:35:52.531 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:35:52.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.750 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-03-06T07:35:52.798 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:35:52.799 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:35:52.799 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:52.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-03-06T07:35:52.879 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:52.879 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:52.929 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:52.929 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:52.929 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:52.973 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:35:52.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:52.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:53.144 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:53.144 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:53.176 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.177:3300,172.21.15.177:3301" 2024-03-06T07:35:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 0@149c34a4-b335-4a4b-8a41-0fe21b0db808.cephfs=/ 2024-03-06T07:35:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.114:3300/172.21.15.177:3300/172.21.15.177:3301" will pass to kernel 2024-03-06T07:35:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:35:53.179 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:35:53.179 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:35:53.179 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:35:53.179 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:35:53.183 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:35:53.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:53.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:53.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:53.260 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:35:53.333 INFO:teuthology.orchestra.run.smithi114.stdout:client8487 2024-03-06T07:35:53.333 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:35:53.333 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:53.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:35:53.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.628+0000 7f1f9ce00640 1 Processor -- start 2024-03-06T07:35:53.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.628+0000 7f1f9ce00640 1 -- start start 2024-03-06T07:35:53.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.629+0000 7f1f9ce00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f9815e450 0x7f1f980cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.629+0000 7f1f9ce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f980b5490 0x7f1f980cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.629+0000 7f1f9ce00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f980b3730 con 0x7f1f980d07f0 2024-03-06T07:35:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.630+0000 7f1f9ce00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f980d4930 con 0x7f1f9815e450 2024-03-06T07:35:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.630+0000 7f1f9ce00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f98067b20 con 0x7f1f980b5490 2024-03-06T07:35:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.630+0000 7f1f95400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f9815e450 0x7f1f980cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.630+0000 7f1f95e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f1f980d07f0 0x7f1f980d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59638/0 (socket says 172.21.15.114:59638) 2024-03-06T07:35:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.630+0000 7f1f95e00640 1 -- 172.21.15.114:0/2917187151 learned_addr learned my addr 172.21.15.114:0/2917187151 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.630+0000 7f1f96800640 1 --2- 172.21.15.114:0/2917187151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f980b5490 0x7f1f980cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.631+0000 7f1f95400640 1 -- 172.21.15.114:0/2917187151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f980b5490 msgr2=0x7f1f980cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.631+0000 7f1f95400640 1 --2- 172.21.15.114:0/2917187151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f980b5490 0x7f1f980cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.631+0000 7f1f95400640 1 -- 172.21.15.114:0/2917187151 >> v1:172.21.15.114:6789/0 conn(0x7f1f980d07f0 legacy=0x7f1f980d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.631+0000 7f1f94a00640 1 -- 172.21.15.114:0/2917187151 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160886579 0 0) 0x7f1f980b3730 con 0x7f1f980d07f0 2024-03-06T07:35:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.631+0000 7f1f95400640 1 -- 172.21.15.114:0/2917187151 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980d4b80 con 0x7f1f9815e450 2024-03-06T07:35:53.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.632+0000 7f1f95400640 1 --2- 172.21.15.114:0/2917187151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f9815e450 0x7f1f980cea20 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f1f840097a0 tx=0x7f1f8402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c473c3428664d2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.632+0000 7f1f94a00640 1 -- 172.21.15.114:0/2917187151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84031780 con 0x7f1f9815e450 2024-03-06T07:35:53.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.632+0000 7f1f94a00640 1 -- 172.21.15.114:0/2917187151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f84037070 con 0x7f1f9815e450 2024-03-06T07:35:53.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.632+0000 7f1f94a00640 1 -- 172.21.15.114:0/2917187151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84031b40 con 0x7f1f9815e450 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 -- 172.21.15.114:0/2917187151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f9815e450 msgr2=0x7f1f980cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/2917187151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f9815e450 0x7f1f980cea20 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f1f840097a0 tx=0x7f1f8402b820 comp rx=0 tx=0).stop 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 -- 172.21.15.114:0/2917187151 shutdown_connections 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/2917187151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f980b5490 0x7f1f980cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/2917187151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f9815e450 0x7f1f980cea20 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 -- 172.21.15.114:0/2917187151 >> 172.21.15.114:0/2917187151 conn(0x7f1f980d3e10 msgr2=0x7f1f980d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 -- 172.21.15.114:0/2917187151 shutdown_connections 2024-03-06T07:35:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.633+0000 7f1f9ce00640 1 -- 172.21.15.114:0/2917187151 wait complete. 2024-03-06T07:35:53.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.636+0000 7f1f9ce00640 1 Processor -- start 2024-03-06T07:35:53.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.636+0000 7f1f9ce00640 1 -- start start 2024-03-06T07:35:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f9ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f980b5490 0x7f1f9815c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f9ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f980d07f0 0x7f1f9815cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f9ce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f9815e450 0x7f1f9815d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f9ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980b3f60 con 0x7f1f980b5490 2024-03-06T07:35:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f9ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980b3c80 con 0x7f1f980d07f0 2024-03-06T07:35:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f9ce00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980d1bf0 con 0x7f1f9815e450 2024-03-06T07:35:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f980b5490 0x7f1f9815c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f980b5490 0x7f1f9815c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39810/0 (socket says 172.21.15.114:39810) 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 -- 172.21.15.114:0/3914664053 learned_addr learned my addr 172.21.15.114:0/3914664053 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 -- 172.21.15.114:0/3914664053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f9815e450 msgr2=0x7f1f9815d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f96800640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f9815e450 0x7f1f9815d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f9815e450 0x7f1f9815d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 -- 172.21.15.114:0/3914664053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f980d07f0 msgr2=0x7f1f9815cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f980d07f0 0x7f1f9815cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.638+0000 7f1f95e00640 1 -- 172.21.15.114:0/3914664053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9815fed0 con 0x7f1f980b5490 2024-03-06T07:35:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.639+0000 7f1f95e00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f980b5490 0x7f1f9815c640 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800c4d0 tx=0x7f1f8800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.639+0000 7f1f8e000640 1 -- 172.21.15.114:0/3914664053 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f88018070 con 0x7f1f980b5490 2024-03-06T07:35:53.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.639+0000 7f1f8e000640 1 -- 172.21.15.114:0/3914664053 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f88007500 con 0x7f1f980b5490 2024-03-06T07:35:53.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.639+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f981601b0 con 0x7f1f980b5490 2024-03-06T07:35:53.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.639+0000 7f1f8e000640 1 -- 172.21.15.114:0/3914664053 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f88004a20 con 0x7f1f980b5490 2024-03-06T07:35:53.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.639+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f98160770 con 0x7f1f980b5490 2024-03-06T07:35:53.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.641+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9804f330 con 0x7f1f980b5490 2024-03-06T07:35:53.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.641+0000 7f1f8e000640 1 -- 172.21.15.114:0/3914664053 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f8801b070 con 0x7f1f980b5490 2024-03-06T07:35:53.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.642+0000 7f1f8e000640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f7806b500 0x7f1f7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:53.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.642+0000 7f1f8e000640 1 -- 172.21.15.114:0/3914664053 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f880894f0 con 0x7f1f980b5490 2024-03-06T07:35:53.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.642+0000 7f1f95400640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f7806b500 0x7f1f7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:53.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.643+0000 7f1f95400640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f7806b500 0x7f1f7806d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1f84036000 tx=0x7f1f84030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:53.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:53.651+0000 7f1f8e000640 1 -- 172.21.15.114:0/3914664053 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f88058360 con 0x7f1f980b5490 2024-03-06T07:35:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.037+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f1f980cfef0 con 0x7f1f980b5490 2024-03-06T07:35:54.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.156+0000 7f1f8e000640 1 -- 172.21.15.114:0/3914664053 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v97) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1f88057c60 con 0x7f1f980b5490 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.159+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f7806b500 msgr2=0x7f1f7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.159+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f7806b500 0x7f1f7806d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1f84036000 tx=0x7f1f84030000 comp rx=0 tx=0).stop 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.159+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f980b5490 msgr2=0x7f1f9815c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.159+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f980b5490 0x7f1f9815c640 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f1f8800c4d0 tx=0x7f1f8800c990 comp rx=0 tx=0).stop 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 shutdown_connections 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1f9815e450 0x7f1f9815d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f980d07f0 0x7f1f9815cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f7806b500 0x7f1f7806d9c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 --2- 172.21.15.114:0/3914664053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f980b5490 0x7f1f9815c640 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 >> 172.21.15.114:0/3914664053 conn(0x7f1f980d3e10 msgr2=0x7f1f9806cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 shutdown_connections 2024-03-06T07:35:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.160+0000 7f1f9ce00640 1 -- 172.21.15.114:0/3914664053 wait complete. 2024-03-06T07:35:54.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:54.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.363+0000 7f58d9000640 1 Processor -- start 2024-03-06T07:35:54.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.363+0000 7f58d9000640 1 -- start start 2024-03-06T07:35:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.364+0000 7f58d9000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f58d40d24c0 0x7f58d40d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.364+0000 7f58d9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58d40b0aa0 0x7f58d40b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.364+0000 7f58d9000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d40d4c50 con 0x7f58d40b2de0 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.364+0000 7f58d9000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d40d4a00 con 0x7f58d40b0aa0 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.364+0000 7f58d9000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d40cf7e0 con 0x7f58d40d24c0 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.364+0000 7f58d2800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58d40b0aa0 0x7f58d40b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.365+0000 7f58d2800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58d40b0aa0 0x7f58d40b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39814/0 (socket says 172.21.15.114:39814) 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.365+0000 7f58d2800640 1 -- 172.21.15.114:0/2818156792 learned_addr learned my addr 172.21.15.114:0/2818156792 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.365+0000 7f58d2800640 1 -- 172.21.15.114:0/2818156792 >> v2:172.21.15.177:3300/0 conn(0x7f58d40d24c0 msgr2=0x7f58d40d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.365+0000 7f58d2800640 1 --2- 172.21.15.114:0/2818156792 >> v2:172.21.15.177:3300/0 conn(0x7f58d40d24c0 0x7f58d40d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.365+0000 7f58d2800640 1 -- 172.21.15.114:0/2818156792 >> v1:172.21.15.177:6789/0 conn(0x7f58d40b2de0 legacy=0x7f58d40d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.365+0000 7f58d2800640 1 -- 172.21.15.114:0/2818156792 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40cfa30 con 0x7f58d40b0aa0 2024-03-06T07:35:54.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.366+0000 7f58d2800640 1 --2- 172.21.15.114:0/2818156792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40b0aa0 0x7f58d40b0f30 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f58d408d300 tx=0x7f58cc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85b61eb8d89a176 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.366+0000 7f58d0a00640 1 -- 172.21.15.114:0/2818156792 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc02c9f0 con 0x7f58d40b0aa0 2024-03-06T07:35:54.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.366+0000 7f58d0a00640 1 -- 172.21.15.114:0/2818156792 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc02cb90 con 0x7f58d40b0aa0 2024-03-06T07:35:54.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.366+0000 7f58d0a00640 1 -- 172.21.15.114:0/2818156792 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0343f0 con 0x7f58d40b0aa0 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 -- 172.21.15.114:0/2818156792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40b0aa0 msgr2=0x7f58d40b0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 --2- 172.21.15.114:0/2818156792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40b0aa0 0x7f58d40b0f30 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f58d408d300 tx=0x7f58cc02be50 comp rx=0 tx=0).stop 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 -- 172.21.15.114:0/2818156792 shutdown_connections 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 --2- 172.21.15.114:0/2818156792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40b0aa0 0x7f58d40b0f30 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 --2- 172.21.15.114:0/2818156792 >> v2:172.21.15.177:3300/0 conn(0x7f58d40d24c0 0x7f58d40d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 -- 172.21.15.114:0/2818156792 >> 172.21.15.114:0/2818156792 conn(0x7f58d40d3ee0 msgr2=0x7f58d40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 -- 172.21.15.114:0/2818156792 shutdown_connections 2024-03-06T07:35:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.367+0000 7f58d9000640 1 -- 172.21.15.114:0/2818156792 wait complete. 2024-03-06T07:35:54.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.370+0000 7f58d9000640 1 Processor -- start 2024-03-06T07:35:54.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.370+0000 7f58d9000640 1 -- start start 2024-03-06T07:35:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.371+0000 7f58d9000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b0aa0 0x7f58d41531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b2de0 0x7f58d40c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40d24c0 0x7f58d40ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d415e5b0 con 0x7f58d40d24c0 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40d0080 con 0x7f58d40b2de0 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d9000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d415e730 con 0x7f58d40b0aa0 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40d24c0 0x7f58d40ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40d24c0 0x7f58d40ca390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39828/0 (socket says 172.21.15.114:39828) 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 -- 172.21.15.114:0/3277154500 learned_addr learned my addr 172.21.15.114:0/3277154500 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 -- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b0aa0 msgr2=0x7f58d41531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d1e00640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b0aa0 0x7f58d41531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d1400640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b2de0 0x7f58d40c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b0aa0 0x7f58d41531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 -- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b2de0 msgr2=0x7f58d40c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b2de0 0x7f58d40c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d2800640 1 -- 172.21.15.114:0/3277154500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40ca9c0 con 0x7f58d40d24c0 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.372+0000 7f58d1400640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b2de0 0x7f58d40c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.373+0000 7f58d2800640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40d24c0 0x7f58d40ca390 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009f40 tx=0x7f58cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.373+0000 7f58ca000640 1 -- 172.21.15.114:0/3277154500 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc043020 con 0x7f58d40d24c0 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.373+0000 7f58ca000640 1 -- 172.21.15.114:0/3277154500 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc03e450 con 0x7f58d40d24c0 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.373+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d40b6bf0 con 0x7f58d40d24c0 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.373+0000 7f58ca000640 1 -- 172.21.15.114:0/3277154500 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc03e820 con 0x7f58d40d24c0 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.373+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d40b7170 con 0x7f58d40d24c0 2024-03-06T07:35:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.374+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d404f330 con 0x7f58d40d24c0 2024-03-06T07:35:54.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.375+0000 7f58ca000640 1 -- 172.21.15.114:0/3277154500 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58cc046020 con 0x7f58d40d24c0 2024-03-06T07:35:54.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.376+0000 7f58ca000640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:54.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.376+0000 7f58ca000640 1 -- 172.21.15.114:0/3277154500 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58cc0acb80 con 0x7f58d40d24c0 2024-03-06T07:35:54.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.376+0000 7f58d1e00640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:54.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.377+0000 7f58d1e00640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f58c400a770 tx=0x7f58c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:54.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.381+0000 7f58ca000640 1 -- 172.21.15.114:0/3277154500 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58cc04d030 con 0x7f58d40d24c0 2024-03-06T07:35:54.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.641+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58d40b61f0 con 0x7f58d40d24c0 2024-03-06T07:35:54.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.642+0000 7f58ca000640 1 -- 172.21.15.114:0/3277154500 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4998 (secure 0 0 0) 0x7f58cc07b9f0 con 0x7f58d40d24c0 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1958627905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1958627905},{"type":"v1","addr":"172.21.15.177:6833","nonce":1958627905}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":5765,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/7846601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":7846601},{"type":"v1","addr":"172.21.15.177:6835","nonce":7846601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:54.161208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:creating","state_seq":1,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:35:54.644 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 98 2024-03-06T07:35:54.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 msgr2=0x7f58b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f58c400a770 tx=0x7f58c4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:54.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40d24c0 msgr2=0x7f58d40ca390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40d24c0 0x7f58d40ca390 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009f40 tx=0x7f58cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 shutdown_connections 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d40d24c0 0x7f58d40ca390 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d40b2de0 0x7f58d40c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 --2- 172.21.15.114:0/3277154500 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d40b0aa0 0x7f58d41531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.644+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 >> 172.21.15.114:0/3277154500 conn(0x7f58d40d3ee0 msgr2=0x7f58d40b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.645+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 shutdown_connections 2024-03-06T07:35:54.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:54.645+0000 7f58d9000640 1 -- 172.21.15.114:0/3277154500 wait complete. 2024-03-06T07:35:54.664 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:35:56.029 INFO:tasks.ceph.mds.c.smithi114.stderr:2024-03-06T07:35:56.027+0000 7f5550800640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:35:59.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.890+0000 7ff5a5200640 1 Processor -- start 2024-03-06T07:35:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.890+0000 7ff5a5200640 1 -- start start 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.893+0000 7ff5a5200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff5a006da30 0x7ff5a00d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.893+0000 7ff5a5200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff5a00b3960 0x7ff5a00b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.893+0000 7ff5a5200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a00cf410 con 0x7ff5a0144630 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.893+0000 7ff5a5200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a0144a10 con 0x7ff5a00b3960 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.893+0000 7ff5a5200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a006e050 con 0x7ff5a006da30 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.893+0000 7ff59f400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff5a0144630 0x7ff5a00d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42126/0 (socket says 172.21.15.114:42126) 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.893+0000 7ff59f400640 1 -- 172.21.15.114:0/1384590876 learned_addr learned my addr 172.21.15.114:0/1384590876 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:35:59.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59fe00640 1 --2- 172.21.15.114:0/1384590876 >> v2:172.21.15.114:3300/0 conn(0x7ff5a00b3960 0x7ff5a00b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59ea00640 1 --2- 172.21.15.114:0/1384590876 >> v2:172.21.15.177:3300/0 conn(0x7ff5a006da30 0x7ff5a00d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59e000640 1 -- 172.21.15.114:0/1384590876 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010521420 0 0) 0x7ff5a00cf410 con 0x7ff5a0144630 2024-03-06T07:35:59.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59e000640 1 -- 172.21.15.114:0/1384590876 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff584003520 con 0x7ff5a0144630 2024-03-06T07:35:59.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59fe00640 1 -- 172.21.15.114:0/1384590876 >> v2:172.21.15.177:3300/0 conn(0x7ff5a006da30 msgr2=0x7ff5a00d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:59.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59fe00640 1 --2- 172.21.15.114:0/1384590876 >> v2:172.21.15.177:3300/0 conn(0x7ff5a006da30 0x7ff5a00d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59fe00640 1 -- 172.21.15.114:0/1384590876 >> v1:172.21.15.114:6789/0 conn(0x7ff5a0144630 legacy=0x7ff5a00d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.894+0000 7ff59fe00640 1 -- 172.21.15.114:0/1384590876 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a00cf6f0 con 0x7ff5a00b3960 2024-03-06T07:35:59.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.896+0000 7ff59fe00640 1 --2- 172.21.15.114:0/1384590876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a00b3e10 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00cceb0 tx=0x7ff59802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cc58a97c7221a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.896+0000 7ff59e000640 1 -- 172.21.15.114:0/1384590876 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598039070 con 0x7ff5a00b3960 2024-03-06T07:35:59.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.896+0000 7ff59e000640 1 -- 172.21.15.114:0/1384590876 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59802cbf0 con 0x7ff5a00b3960 2024-03-06T07:35:59.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.896+0000 7ff59e000640 1 -- 172.21.15.114:0/1384590876 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598034440 con 0x7ff5a00b3960 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 -- 172.21.15.114:0/1384590876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 msgr2=0x7ff5a00b3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 --2- 172.21.15.114:0/1384590876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a00b3e10 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00cceb0 tx=0x7ff59802be40 comp rx=0 tx=0).stop 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 -- 172.21.15.114:0/1384590876 shutdown_connections 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 --2- 172.21.15.114:0/1384590876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a00b3e10 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 --2- 172.21.15.114:0/1384590876 >> v2:172.21.15.177:3300/0 conn(0x7ff5a006da30 0x7ff5a00d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 -- 172.21.15.114:0/1384590876 >> 172.21.15.114:0/1384590876 conn(0x7ff5a006c0d0 msgr2=0x7ff5a006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 -- 172.21.15.114:0/1384590876 shutdown_connections 2024-03-06T07:35:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.897+0000 7ff5a5200640 1 -- 172.21.15.114:0/1384590876 wait complete. 2024-03-06T07:35:59.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.899+0000 7ff5a5200640 1 Processor -- start 2024-03-06T07:35:59.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.899+0000 7ff5a5200640 1 -- start start 2024-03-06T07:35:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff5a5200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5a006da30 0x7ff5a00cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff5a5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a01f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff5a5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a0144630 0x7ff5a01f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff5a5200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a006f850 con 0x7ff5a00b3960 2024-03-06T07:35:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff5a5200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a00cfd40 con 0x7ff5a0144630 2024-03-06T07:35:59.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff5a5200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a00d0fa0 con 0x7ff5a006da30 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a01f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a01f1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40244/0 (socket says 172.21.15.114:40244) 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 -- 172.21.15.114:0/837108005 learned_addr learned my addr 172.21.15.114:0/837108005 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 -- 172.21.15.114:0/837108005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5a006da30 msgr2=0x7ff5a00cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5a006da30 0x7ff5a00cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 -- 172.21.15.114:0/837108005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a0144630 msgr2=0x7ff5a01f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a0144630 0x7ff5a01f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.900+0000 7ff59ea00640 1 -- 172.21.15.114:0/837108005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a01f2310 con 0x7ff5a00b3960 2024-03-06T07:35:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.901+0000 7ff59ea00640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a01f1dd0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00d7d0 tx=0x7ff58c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.901+0000 7ff597400640 1 -- 172.21.15.114:0/837108005 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c004020 con 0x7ff5a00b3960 2024-03-06T07:35:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.901+0000 7ff597400640 1 -- 172.21.15.114:0/837108005 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c0041e0 con 0x7ff5a00b3960 2024-03-06T07:35:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.901+0000 7ff597400640 1 -- 172.21.15.114:0/837108005 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0043d0 con 0x7ff5a00b3960 2024-03-06T07:35:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.901+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a013c880 con 0x7ff5a00b3960 2024-03-06T07:35:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.901+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a013ce40 con 0x7ff5a00b3960 2024-03-06T07:35:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.902+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a004f330 con 0x7ff5a00b3960 2024-03-06T07:35:59.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.903+0000 7ff597400640 1 -- 172.21.15.114:0/837108005 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff58c01b020 con 0x7ff5a00b3960 2024-03-06T07:35:59.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.904+0000 7ff597400640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff58006b4d0 0x7ff58006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.904+0000 7ff597400640 1 -- 172.21.15.114:0/837108005 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff58c08a0b0 con 0x7ff5a00b3960 2024-03-06T07:35:59.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.904+0000 7ff59f400640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff58006b4d0 0x7ff58006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.905+0000 7ff59f400640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff58006b4d0 0x7ff58006d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff590000f60 tx=0x7ff5900078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:35:59.909+0000 7ff597400640 1 -- 172.21.15.114:0/837108005 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff58c058e90 con 0x7ff5a00b3960 2024-03-06T07:36:00.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.147+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5a013fb80 con 0x7ff5a00b3960 2024-03-06T07:36:00.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.148+0000 7ff597400640 1 -- 172.21.15.114:0/837108005 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5001 (secure 0 0 0) 0x7ff58c058790 con 0x7ff5a00b3960 2024-03-06T07:36:00.151 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:00.151 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1958627905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1958627905},{"type":"v1","addr":"172.21.15.177:6833","nonce":1958627905}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":5765,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/7846601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":7846601},{"type":"v1","addr":"172.21.15.177:6835","nonce":7846601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:55.163774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8328,8331]},"id":9}]} 2024-03-06T07:36:00.151 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 99 2024-03-06T07:36:00.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff58006b4d0 msgr2=0x7ff58006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff58006b4d0 0x7ff58006d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff590000f60 tx=0x7ff5900078f0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 msgr2=0x7ff5a01f1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a01f1dd0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00d7d0 tx=0x7ff58c00dc90 comp rx=0 tx=0).stop 2024-03-06T07:36:00.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 shutdown_connections 2024-03-06T07:36:00.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5a0144630 0x7ff5a01f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff58006b4d0 0x7ff58006d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5a00b3960 0x7ff5a01f1dd0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 --2- 172.21.15.114:0/837108005 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5a006da30 0x7ff5a00cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.150+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 >> 172.21.15.114:0/837108005 conn(0x7ff5a006c0d0 msgr2=0x7ff5a01412f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.151+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 shutdown_connections 2024-03-06T07:36:00.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.151+0000 7ff5a5200640 1 -- 172.21.15.114:0/837108005 wait complete. 2024-03-06T07:36:00.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.355+0000 7f5771600640 1 Processor -- start 2024-03-06T07:36:00.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.355+0000 7f5771600640 1 -- start start 2024-03-06T07:36:00.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.357+0000 7f5771600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f576c15f9d0 0x7f576c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.357+0000 7f5771600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c06cef0 con 0x7f576c144c00 2024-03-06T07:36:00.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.357+0000 7f5771600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c06d1d0 con 0x7f576c070b10 2024-03-06T07:36:00.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.357+0000 7f5771600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c06d4b0 con 0x7f576c15f9d0 2024-03-06T07:36:00.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.357+0000 7f576aa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f576c144c00 0x7f576c06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42130/0 (socket says 172.21.15.114:42130) 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.357+0000 7f576aa00640 1 -- 172.21.15.114:0/2131586898 learned_addr learned my addr 172.21.15.114:0/2131586898 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.357+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125964421 0 0) 0x7f576c06cef0 con 0x7f576c144c00 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5750003520 con 0x7f576c144c00 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f5763e00640 1 --2- 172.21.15.114:0/2131586898 >> v2:172.21.15.177:3300/0 conn(0x7f576c15f9d0 0x7f576c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2889374508 0 0) 0x7f5750003520 con 0x7f576c144c00 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 >> v2:172.21.15.177:3300/0 conn(0x7f576c15f9d0 msgr2=0x7f576c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 --2- 172.21.15.114:0/2131586898 >> v2:172.21.15.177:3300/0 conn(0x7f576c15f9d0 0x7f576c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 >> v1:172.21.15.177:6789/0 conn(0x7f576c070b10 legacy=0x7f576c070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c06d790 con 0x7f576c144c00 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f575c001670 con 0x7f576c144c00 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 >> v1:172.21.15.114:6789/0 conn(0x7f576c144c00 legacy=0x7f576c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5750003a00 0x7f5750003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750004800 0x7f5750004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750005210 0x7f575000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c06d4b0 con 0x7f5750003a00 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575000c160 con 0x7f5750004800 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575000c360 con 0x7f5750005210 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.358+0000 7f576aa00640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5750003a00 0x7f5750003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f5763e00640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750004800 0x7f5750004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f576b400640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750005210 0x7f575000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f576aa00640 1 -- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750005210 msgr2=0x7f575000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f576aa00640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750005210 0x7f575000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f576aa00640 1 -- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750004800 msgr2=0x7f5750004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f576aa00640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750004800 0x7f5750004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f576aa00640 1 -- 172.21.15.114:0/2131586898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5750003520 con 0x7f5750003a00 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f576b400640 1 -- 172.21.15.114:0/2131586898 reap_dead start 2024-03-06T07:36:00.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.359+0000 7f5763e00640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750004800 0x7f5750004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.360+0000 7f576aa00640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5750003a00 0x7f5750003e20 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f575c00c190 tx=0x7f5750005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd727b6c58430261 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c007c60 con 0x7f5750003a00 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c030070 con 0x7f5750003a00 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f576a000640 1 -- 172.21.15.114:0/2131586898 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c005d90 con 0x7f5750003a00 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f5771600640 1 -- 172.21.15.114:0/2131586898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5750003a00 msgr2=0x7f5750003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f5771600640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5750003a00 0x7f5750003e20 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f575c00c190 tx=0x7f5750005830 comp rx=0 tx=0).stop 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f5771600640 1 -- 172.21.15.114:0/2131586898 shutdown_connections 2024-03-06T07:36:00.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f5771600640 1 --2- 172.21.15.114:0/2131586898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5750003a00 0x7f5750003e20 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f5771600640 1 -- 172.21.15.114:0/2131586898 >> 172.21.15.114:0/2131586898 conn(0x7f576c0b1730 msgr2=0x7f576c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f5771600640 1 -- 172.21.15.114:0/2131586898 shutdown_connections 2024-03-06T07:36:00.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.361+0000 7f5771600640 1 -- 172.21.15.114:0/2131586898 wait complete. 2024-03-06T07:36:00.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.363+0000 7f5771600640 1 Processor -- start 2024-03-06T07:36:00.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.363+0000 7f5771600640 1 -- start start 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f5771600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750003a00 0x7f576c13c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f5771600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c13daa0 0x7f576c15e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f5771600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f576c13df20 0x7f576c15ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f5771600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c06ccd0 con 0x7f576c13df20 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f5771600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c06cb50 con 0x7f576c13daa0 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f5771600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c06df90 con 0x7f5750003a00 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f576b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f576c13df20 0x7f576c15ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f576b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f576c13df20 0x7f576c15ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40262/0 (socket says 172.21.15.114:40262) 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.364+0000 7f576b400640 1 -- 172.21.15.114:0/583099627 learned_addr learned my addr 172.21.15.114:0/583099627 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f576b400640 1 -- 172.21.15.114:0/583099627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750003a00 msgr2=0x7f576c13c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f576b400640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750003a00 0x7f576c13c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f5763e00640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c13daa0 0x7f576c15e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f576b400640 1 -- 172.21.15.114:0/583099627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c13daa0 msgr2=0x7f576c15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f576b400640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c13daa0 0x7f576c15e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f576b400640 1 -- 172.21.15.114:0/583099627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c14b0f0 con 0x7f576c13df20 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f576b400640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f576c13df20 0x7f576c15ea30 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f576c0737b0 tx=0x7f576400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f5762a00640 1 -- 172.21.15.114:0/583099627 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764006b50 con 0x7f576c13df20 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f5762a00640 1 -- 172.21.15.114:0/583099627 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5764006cf0 con 0x7f576c13df20 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c14b3d0 con 0x7f576c13df20 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.365+0000 7f5762a00640 1 -- 172.21.15.114:0/583099627 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764003970 con 0x7f576c13df20 2024-03-06T07:36:00.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.366+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c14b990 con 0x7f576c13df20 2024-03-06T07:36:00.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.367+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c04f330 con 0x7f576c13df20 2024-03-06T07:36:00.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.367+0000 7f5762a00640 1 -- 172.21.15.114:0/583099627 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f576401a020 con 0x7f576c13df20 2024-03-06T07:36:00.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.368+0000 7f5762a00640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f574806b4d0 0x7f574806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.368+0000 7f5762a00640 1 -- 172.21.15.114:0/583099627 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5764089920 con 0x7f576c13df20 2024-03-06T07:36:00.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.369+0000 7f576aa00640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f574806b4d0 0x7f574806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.369+0000 7f576aa00640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f574806b4d0 0x7f574806d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f575c00c160 tx=0x7f575c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.373+0000 7f5762a00640 1 -- 172.21.15.114:0/583099627 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5764058700 con 0x7f576c13df20 2024-03-06T07:36:00.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.662+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f576c071d40 con 0x7f576c13df20 2024-03-06T07:36:00.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.663+0000 7f5762a00640 1 -- 172.21.15.114:0/583099627 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5001 (secure 0 0 0) 0x7f5764058000 con 0x7f576c13df20 2024-03-06T07:36:00.666 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:00.666 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5762,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1958627905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1958627905},{"type":"v1","addr":"172.21.15.177:6833","nonce":1958627905}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":5765,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/7846601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":7846601},{"type":"v1","addr":"172.21.15.177:6835","nonce":7846601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:55.163774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8328,8331]},"id":9}]} 2024-03-06T07:36:00.666 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 99 2024-03-06T07:36:00.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f574806b4d0 msgr2=0x7f574806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f574806b4d0 0x7f574806d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f575c00c160 tx=0x7f575c007000 comp rx=0 tx=0).stop 2024-03-06T07:36:00.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f576c13df20 msgr2=0x7f576c15ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f576c13df20 0x7f576c15ea30 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f576c0737b0 tx=0x7f576400ec10 comp rx=0 tx=0).stop 2024-03-06T07:36:00.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 shutdown_connections 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f574806b4d0 0x7f574806d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f576c13df20 0x7f576c15ea30 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c13daa0 0x7f576c15e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 --2- 172.21.15.114:0/583099627 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5750003a00 0x7f576c13c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.665+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 >> 172.21.15.114:0/583099627 conn(0x7f576c0b1730 msgr2=0x7f576c06e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.666+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 shutdown_connections 2024-03-06T07:36:00.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.666+0000 7f5771600640 1 -- 172.21.15.114:0/583099627 wait complete. 2024-03-06T07:36:00.687 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are c and a 2024-03-06T07:36:00.687 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:36:00.687 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:00.719 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:36:00.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:36:00.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.917+0000 7f8149400640 1 Processor -- start 2024-03-06T07:36:00.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.917+0000 7f8149400640 1 -- start start 2024-03-06T07:36:00.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.918+0000 7f8149400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440c6460 0x7f81440c5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.918+0000 7f8149400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8144145b30 con 0x7f81440c4930 2024-03-06T07:36:00.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.919+0000 7f8149400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8144142ce0 con 0x7f81440c5960 2024-03-06T07:36:00.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.919+0000 7f8149400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81440c68c0 con 0x7f81440c6460 2024-03-06T07:36:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.919+0000 7f8142800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f81440c4930 0x7f81440c4d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42138/0 (socket says 172.21.15.114:42138) 2024-03-06T07:36:00.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.919+0000 7f8142800640 1 -- 172.21.15.114:0/1847686507 learned_addr learned my addr 172.21.15.114:0/1847686507 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.919+0000 7f813be00640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440c6460 0x7f81440c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157648090 0 0) 0x7f8134001110 con 0x7f81440c4930 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8128003520 con 0x7f81440c4930 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814253338 0 0) 0x7f8144142ce0 con 0x7f81440c5960 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8134001110 con 0x7f81440c5960 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 364605405 0 0) 0x7f8128003520 con 0x7f81440c4930 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440c6460 msgr2=0x7f81440c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440c6460 0x7f81440c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 >> v1:172.21.15.177:6789/0 conn(0x7f81440c5960 legacy=0x7f8144165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81440cbca0 con 0x7f81440c4930 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8134001840 con 0x7f81440c4930 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 >> v1:172.21.15.114:6789/0 conn(0x7f81440c4930 legacy=0x7f81440c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8128003b70 0x7f8128003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8128004800 0x7f8128004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81280051c0 0x7f812800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81440c68c0 con 0x7f81280051c0 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812800c090 con 0x7f8128003b70 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812800c290 con 0x7f8128004800 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f813be00640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440c6460 0x7f81440c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.920+0000 7f8143200640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81280051c0 0x7f812800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.921+0000 7f8142800640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8128003b70 0x7f8128003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.921+0000 7f8143200640 1 -- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8128004800 msgr2=0x7f8128004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.921+0000 7f8143200640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8128004800 0x7f8128004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.921+0000 7f8143200640 1 -- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8128003b70 msgr2=0x7f8128003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.921+0000 7f8143200640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8128003b70 0x7f8128003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.921+0000 7f8143200640 1 -- 172.21.15.114:0/1847686507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128003520 con 0x7f81280051c0 2024-03-06T07:36:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.921+0000 7f8142800640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8128003b70 0x7f8128003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:00.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.922+0000 7f8143200640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81280051c0 0x7f812800b9c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f813c001fc0 tx=0x7f8128005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a7d3ddc1f0a2c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.922+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c001be0 con 0x7f81280051c0 2024-03-06T07:36:00.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.922+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c003b90 con 0x7f81280051c0 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.922+0000 7f8141e00640 1 -- 172.21.15.114:0/1847686507 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c0050d0 con 0x7f81280051c0 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.923+0000 7f8149400640 1 -- 172.21.15.114:0/1847686507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81280051c0 msgr2=0x7f812800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.923+0000 7f8149400640 1 --2- 172.21.15.114:0/1847686507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81280051c0 0x7f812800b9c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f813c001fc0 tx=0x7f8128005830 comp rx=0 tx=0).stop 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.923+0000 7f8143200640 1 -- 172.21.15.114:0/1847686507 reap_dead start 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.923+0000 7f8149400640 1 -- 172.21.15.114:0/1847686507 shutdown_connections 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.923+0000 7f8149400640 1 -- 172.21.15.114:0/1847686507 >> 172.21.15.114:0/1847686507 conn(0x7f81440cc4f0 msgr2=0x7f81440cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.923+0000 7f8149400640 1 -- 172.21.15.114:0/1847686507 shutdown_connections 2024-03-06T07:36:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.923+0000 7f8149400640 1 -- 172.21.15.114:0/1847686507 wait complete. 2024-03-06T07:36:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.925+0000 7f8149400640 1 Processor -- start 2024-03-06T07:36:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.925+0000 7f8149400640 1 -- start start 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.926+0000 7f8149400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81280051c0 0x7f814413e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.926+0000 7f8149400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814413de00 0x7f8144141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.926+0000 7f8149400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440af530 0x7f81440af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.926+0000 7f8149400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81440ca3b0 con 0x7f814413de00 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.926+0000 7f8149400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81440ca0d0 con 0x7f81280051c0 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.926+0000 7f8149400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814406e600 con 0x7f81440af530 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814413de00 0x7f8144141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814413de00 0x7f8144141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40274/0 (socket says 172.21.15.114:40274) 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 -- 172.21.15.114:0/317641323 learned_addr learned my addr 172.21.15.114:0/317641323 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f8142800640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81280051c0 0x7f814413e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f8143200640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440af530 0x7f81440af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 -- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440af530 msgr2=0x7f81440af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440af530 0x7f81440af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 -- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81280051c0 msgr2=0x7f814413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81280051c0 0x7f814413e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 -- 172.21.15.114:0/317641323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8144140a40 con 0x7f814413de00 2024-03-06T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.927+0000 7f813be00640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814413de00 0x7f8144141ae0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f812c004790 tx=0x7f812c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.928+0000 7f813aa00640 1 -- 172.21.15.114:0/317641323 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c004030 con 0x7f814413de00 2024-03-06T07:36:00.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.928+0000 7f813aa00640 1 -- 172.21.15.114:0/317641323 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c004210 con 0x7f814413de00 2024-03-06T07:36:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.928+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8144140d20 con 0x7f814413de00 2024-03-06T07:36:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.928+0000 7f813aa00640 1 -- 172.21.15.114:0/317641323 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00c830 con 0x7f814413de00 2024-03-06T07:36:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.928+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81441412e0 con 0x7f814413de00 2024-03-06T07:36:00.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.929+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814404f330 con 0x7f814413de00 2024-03-06T07:36:00.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.930+0000 7f813aa00640 1 -- 172.21.15.114:0/317641323 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f812c00c9d0 con 0x7f814413de00 2024-03-06T07:36:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.932+0000 7f813aa00640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f812006b500 0x7f812006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:00.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.932+0000 7f813aa00640 1 -- 172.21.15.114:0/317641323 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f812c08a5c0 con 0x7f814413de00 2024-03-06T07:36:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.932+0000 7f8142800640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f812006b500 0x7f812006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.933+0000 7f8142800640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f812006b500 0x7f812006d9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8134001eb0 tx=0x7f8134009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:00.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:00.936+0000 7f813aa00640 1 -- 172.21.15.114:0/317641323 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f812c017020 con 0x7f814413de00 2024-03-06T07:36:01.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.307+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f814415e080 con 0x7f814413de00 2024-03-06T07:36:01.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.362+0000 7f813aa00640 1 -- 172.21.15.114:0/317641323 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5765 v100) v1 ==== 95+0+0 (secure 0 0 0) 0x7f812c0593a0 con 0x7f814413de00 2024-03-06T07:36:01.365 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 5765 2024-03-06T07:36:01.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.364+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f812006b500 msgr2=0x7f812006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.364+0000 7f8149400640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f812006b500 0x7f812006d9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8134001eb0 tx=0x7f8134009040 comp rx=0 tx=0).stop 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.364+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814413de00 msgr2=0x7f8144141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.364+0000 7f8149400640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814413de00 0x7f8144141ae0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f812c004790 tx=0x7f812c00ed60 comp rx=0 tx=0).stop 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 shutdown_connections 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f81440af530 0x7f81440af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f812006b500 0x7f812006d9c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f814413de00 0x7f8144141ae0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 --2- 172.21.15.114:0/317641323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81280051c0 0x7f814413e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 >> 172.21.15.114:0/317641323 conn(0x7f81440cc4f0 msgr2=0x7f8144144460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 shutdown_connections 2024-03-06T07:36:01.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.365+0000 7f8149400640 1 -- 172.21.15.114:0/317641323 wait complete. 2024-03-06T07:36:01.390 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:36:01.390 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:36:01.420 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:36:01.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:36:01.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.628+0000 7f75dd000640 1 Processor -- start 2024-03-06T07:36:01.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.628+0000 7f75dd000640 1 -- start start 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75dd000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f75d80d2360 0x7f75d80d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75dd000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75dd000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d8161960 con 0x7f75d8161580 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75dd000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d80d2ce0 con 0x7f75d80d2360 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75dd000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80cf6f0 con 0x7f75d80b3ec0 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75cfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f75d80d2360 0x7f75d80d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75d7400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f75d8161580 0x7f75d80d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42144/0 (socket says 172.21.15.114:42144) 2024-03-06T07:36:01.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.630+0000 7f75d7400640 1 -- 172.21.15.114:0/3549185637 learned_addr learned my addr 172.21.15.114:0/3549185637 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:01.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.631+0000 7f75d7e00640 1 --2- 172.21.15.114:0/3549185637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.631+0000 7f75cfe00640 1 -- 172.21.15.114:0/3549185637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 msgr2=0x7f75d80b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:01.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.631+0000 7f75cfe00640 1 --2- 172.21.15.114:0/3549185637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.631+0000 7f75cfe00640 1 -- 172.21.15.114:0/3549185637 >> v1:172.21.15.114:6789/0 conn(0x7f75d8161580 legacy=0x7f75d80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.631+0000 7f75cfe00640 1 -- 172.21.15.114:0/3549185637 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80cf9d0 con 0x7f75d80d2360 2024-03-06T07:36:01.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.631+0000 7f75d6a00640 1 -- 172.21.15.114:0/3549185637 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746404103 0 0) 0x7f75d8161960 con 0x7f75d8161580 2024-03-06T07:36:01.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.632+0000 7f75cfe00640 1 --2- 172.21.15.114:0/3549185637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d80d2360 0x7f75d80d27a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f75c00097a0 tx=0x7f75c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e79774797485d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:01.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.632+0000 7f75d6a00640 1 -- 172.21.15.114:0/3549185637 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0035020 con 0x7f75d80d2360 2024-03-06T07:36:01.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.632+0000 7f75d6a00640 1 -- 172.21.15.114:0/3549185637 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c0031830 con 0x7f75d80d2360 2024-03-06T07:36:01.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.632+0000 7f75d6a00640 1 -- 172.21.15.114:0/3549185637 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0031c00 con 0x7f75d80d2360 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 -- 172.21.15.114:0/3549185637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d80d2360 msgr2=0x7f75d80d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 --2- 172.21.15.114:0/3549185637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d80d2360 0x7f75d80d27a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f75c00097a0 tx=0x7f75c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 -- 172.21.15.114:0/3549185637 shutdown_connections 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 --2- 172.21.15.114:0/3549185637 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 --2- 172.21.15.114:0/3549185637 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d80d2360 0x7f75d80d27a0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 -- 172.21.15.114:0/3549185637 >> 172.21.15.114:0/3549185637 conn(0x7f75d80d3d80 msgr2=0x7f75d80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 -- 172.21.15.114:0/3549185637 shutdown_connections 2024-03-06T07:36:01.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.633+0000 7f75dd000640 1 -- 172.21.15.114:0/3549185637 wait complete. 2024-03-06T07:36:01.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.635+0000 7f75dd000640 1 Processor -- start 2024-03-06T07:36:01.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.635+0000 7f75dd000640 1 -- start start 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75dd000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75dd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75d80d2360 0x7f75d81f1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75dd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d8161580 0x7f75d80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75dd000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80b39a0 con 0x7f75d8161580 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75dd000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80b36c0 con 0x7f75d80d2360 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75dd000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80cf3a0 con 0x7f75d80b3ec0 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d8161580 0x7f75d80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d8161580 0x7f75d80c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40306/0 (socket says 172.21.15.114:40306) 2024-03-06T07:36:01.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7e00640 1 -- 172.21.15.114:0/4046640394 learned_addr learned my addr 172.21.15.114:0/4046640394 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75cfe00640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75d80d2360 0x7f75d81f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7400640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7e00640 1 -- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 msgr2=0x7f75d80c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7e00640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7e00640 1 -- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75d80d2360 msgr2=0x7f75d81f1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.636+0000 7f75d7e00640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75d80d2360 0x7f75d81f1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75d7e00640 1 -- 172.21.15.114:0/4046640394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80c7c60 con 0x7f75d8161580 2024-03-06T07:36:01.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75cfe00640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75d80d2360 0x7f75d81f1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:01.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75d7e00640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d8161580 0x7f75d80c95e0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f75d0007e40 tx=0x7f75d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:01.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75cf400640 1 -- 172.21.15.114:0/4046640394 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d0014620 con 0x7f75d8161580 2024-03-06T07:36:01.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75cf400640 1 -- 172.21.15.114:0/4046640394 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d0019070 con 0x7f75d8161580 2024-03-06T07:36:01.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75cf400640 1 -- 172.21.15.114:0/4046640394 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d0014a20 con 0x7f75d8161580 2024-03-06T07:36:01.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d80c7f40 con 0x7f75d8161580 2024-03-06T07:36:01.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.637+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d80c8500 con 0x7f75d8161580 2024-03-06T07:36:01.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.639+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d804f330 con 0x7f75d8161580 2024-03-06T07:36:01.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.639+0000 7f75cf400640 1 -- 172.21.15.114:0/4046640394 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75d0014bc0 con 0x7f75d8161580 2024-03-06T07:36:01.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.640+0000 7f75cf400640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.640+0000 7f75cf400640 1 -- 172.21.15.114:0/4046640394 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75d008aae0 con 0x7f75d8161580 2024-03-06T07:36:01.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.640+0000 7f75d7400640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.642+0000 7f75d7400640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75b406b4d0 0x7f75b406d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f75c8000f60 tx=0x7f75c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.645+0000 7f75cf400640 1 -- 172.21.15.114:0/4046640394 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75d0059950 con 0x7f75d8161580 2024-03-06T07:36:01.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:01.919+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f75d80d3b60 con 0x7f75d8161580 2024-03-06T07:36:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.364+0000 7f75cf400640 1 -- 172.21.15.114:0/4046640394 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5762 v101) v1 ==== 95+0+0 (secure 0 0 0) 0x7f75d0059250 con 0x7f75d8161580 2024-03-06T07:36:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:failed mds gid 5762 2024-03-06T07:36:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75b406b4d0 msgr2=0x7f75b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75b406b4d0 0x7f75b406d990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f75c8000f60 tx=0x7f75c8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d8161580 msgr2=0x7f75d80c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d8161580 0x7f75d80c95e0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f75d0007e40 tx=0x7f75d000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 shutdown_connections 2024-03-06T07:36:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75d8161580 0x7f75d80c95e0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75d80d2360 0x7f75d81f1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 --2- 172.21.15.114:0/4046640394 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f75d80b3ec0 0x7f75d80c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.368+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 >> 172.21.15.114:0/4046640394 conn(0x7f75d80d3d80 msgr2=0x7f75d80d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:02.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.369+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 shutdown_connections 2024-03-06T07:36:02.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.369+0000 7f75dd000640 1 -- 172.21.15.114:0/4046640394 wait complete. 2024-03-06T07:36:02.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:02.389 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:36:02.389 DEBUG:teuthology.orchestra.run.smithi114:> mkdir pin 2024-03-06T07:36:02.389 DEBUG:teuthology.orchestra.run.smithi114:> touch pin/placeholder 2024-03-06T07:36:02.389 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-03-06T07:36:02.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:02.458 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:36:02.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:02.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.723+0000 7fe3dee00640 1 Processor -- start 2024-03-06T07:36:02.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.723+0000 7fe3dee00640 1 -- start start 2024-03-06T07:36:02.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.724+0000 7fe3dee00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3d80b4680 0x7fe3d806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.724+0000 7fe3dee00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe3d8072340 0x7fe3d806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.724+0000 7fe3dee00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d815b090 con 0x7fe3d815acb0 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.724+0000 7fe3dee00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d80b2a30 con 0x7fe3d80b4680 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.724+0000 7fe3dee00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d80b4360 con 0x7fe3d8072340 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.725+0000 7fe3d7400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3d80b4680 0x7fe3d806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.725+0000 7fe3d7400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3d80b4680 0x7fe3d806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40316/0 (socket says 172.21.15.114:40316) 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.725+0000 7fe3d7400640 1 -- 172.21.15.114:0/3554715111 learned_addr learned my addr 172.21.15.114:0/3554715111 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.725+0000 7fe3d7400640 1 -- 172.21.15.114:0/3554715111 >> v2:172.21.15.177:3300/0 conn(0x7fe3d8072340 msgr2=0x7fe3d806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.725+0000 7fe3d7400640 1 --2- 172.21.15.114:0/3554715111 >> v2:172.21.15.177:3300/0 conn(0x7fe3d8072340 0x7fe3d806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.725+0000 7fe3d7400640 1 -- 172.21.15.114:0/3554715111 >> v1:172.21.15.177:6789/0 conn(0x7fe3d815acb0 legacy=0x7fe3d80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:02.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.725+0000 7fe3d7400640 1 -- 172.21.15.114:0/3554715111 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d80b2cf0 con 0x7fe3d80b4680 2024-03-06T07:36:02.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.726+0000 7fe3d7400640 1 --2- 172.21.15.114:0/3554715111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d80b4680 0x7fe3d806e4f0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fe3d80b4e60 tx=0x7fe3c402db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882d459607b43696 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.726+0000 7fe3d6a00640 1 -- 172.21.15.114:0/3554715111 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c402f8e0 con 0x7fe3d80b4680 2024-03-06T07:36:02.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.726+0000 7fe3d6a00640 1 -- 172.21.15.114:0/3554715111 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c402fa80 con 0x7fe3d80b4680 2024-03-06T07:36:02.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3d6a00640 1 -- 172.21.15.114:0/3554715111 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c402fe50 con 0x7fe3d80b4680 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 -- 172.21.15.114:0/3554715111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d80b4680 msgr2=0x7fe3d806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3554715111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d80b4680 0x7fe3d806e4f0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fe3d80b4e60 tx=0x7fe3c402db20 comp rx=0 tx=0).stop 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 -- 172.21.15.114:0/3554715111 shutdown_connections 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3554715111 >> v2:172.21.15.177:3300/0 conn(0x7fe3d8072340 0x7fe3d806ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3554715111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d80b4680 0x7fe3d806e4f0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 -- 172.21.15.114:0/3554715111 >> 172.21.15.114:0/3554715111 conn(0x7fe3d80b1730 msgr2=0x7fe3d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 -- 172.21.15.114:0/3554715111 shutdown_connections 2024-03-06T07:36:02.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.727+0000 7fe3dee00640 1 -- 172.21.15.114:0/3554715111 wait complete. 2024-03-06T07:36:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.730+0000 7fe3dee00640 1 Processor -- start 2024-03-06T07:36:02.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.730+0000 7fe3dee00640 1 -- start start 2024-03-06T07:36:02.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d8072340 0x7fe3d8146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dee00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3d80b4680 0x7fe3d8159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d815acb0 0x7fe3d80ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d80b4010 con 0x7fe3d815acb0 2024-03-06T07:36:02.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d80b32b0 con 0x7fe3d8072340 2024-03-06T07:36:02.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dee00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d80ccfb0 con 0x7fe3d80b4680 2024-03-06T07:36:02.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d815acb0 0x7fe3d80ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d815acb0 0x7fe3d80ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40326/0 (socket says 172.21.15.114:40326) 2024-03-06T07:36:02.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3dce00640 1 -- 172.21.15.114:0/3680498619 learned_addr learned my addr 172.21.15.114:0/3680498619 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:02.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3d7e00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d8072340 0x7fe3d8146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.732+0000 7fe3d7400640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3d80b4680 0x7fe3d8159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dce00640 1 -- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3d80b4680 msgr2=0x7fe3d8159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dce00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3d80b4680 0x7fe3d8159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dce00640 1 -- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d8072340 msgr2=0x7fe3d8146520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dce00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d8072340 0x7fe3d8146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dce00640 1 -- 172.21.15.114:0/3680498619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8146a60 con 0x7fe3d815acb0 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dce00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d815acb0 0x7fe3d80ad860 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000b9a0 tx=0x7fe3d000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3cfe00640 1 -- 172.21.15.114:0/3680498619 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0019070 con 0x7fe3d815acb0 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3cfe00640 1 -- 172.21.15.114:0/3680498619 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d000cbc0 con 0x7fe3d815acb0 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d81f4c60 con 0x7fe3d815acb0 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3cfe00640 1 -- 172.21.15.114:0/3680498619 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0014440 con 0x7fe3d815acb0 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.733+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d8159460 con 0x7fe3d815acb0 2024-03-06T07:36:02.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.735+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d804f330 con 0x7fe3d815acb0 2024-03-06T07:36:02.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.736+0000 7fe3cfe00640 1 -- 172.21.15.114:0/3680498619 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3d001c070 con 0x7fe3d815acb0 2024-03-06T07:36:02.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.736+0000 7fe3cfe00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3b806b4d0 0x7fe3b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:02.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.737+0000 7fe3cfe00640 1 -- 172.21.15.114:0/3680498619 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3d0089fd0 con 0x7fe3d815acb0 2024-03-06T07:36:02.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.737+0000 7fe3d7e00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3b806b4d0 0x7fe3b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:02.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.738+0000 7fe3d7e00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3b806b4d0 0x7fe3b806d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800a730 tx=0x7fe3c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:02.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:02.741+0000 7fe3cfe00640 1 -- 172.21.15.114:0/3680498619 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3d0058db0 con 0x7fe3d815acb0 2024-03-06T07:36:03.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.065+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3d80b3c30 con 0x7fe3d815acb0 2024-03-06T07:36:03.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.066+0000 7fe3cfe00640 1 -- 172.21.15.114:0/3680498619 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3424 (secure 0 0 0) 0x7fe3d00586b0 con 0x7fe3d815acb0 2024-03-06T07:36:03.068 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:03.069 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:55.163774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8328,8331]},"id":9}]} 2024-03-06T07:36:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 101 2024-03-06T07:36:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3b806b4d0 msgr2=0x7fe3b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3b806b4d0 0x7fe3b806d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800a730 tx=0x7fe3c8008040 comp rx=0 tx=0).stop 2024-03-06T07:36:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d815acb0 msgr2=0x7fe3d80ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d815acb0 0x7fe3d80ad860 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000b9a0 tx=0x7fe3d000be60 comp rx=0 tx=0).stop 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 shutdown_connections 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe3b806b4d0 0x7fe3b806d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3d815acb0 0x7fe3d80ad860 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe3d80b4680 0x7fe3d8159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.068+0000 7fe3dee00640 1 --2- 172.21.15.114:0/3680498619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3d8072340 0x7fe3d8146520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.069+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 >> 172.21.15.114:0/3680498619 conn(0x7fe3d80b1730 msgr2=0x7fe3d80af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.069+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 shutdown_connections 2024-03-06T07:36:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:03.069+0000 7fe3dee00640 1 -- 172.21.15.114:0/3680498619 wait complete. 2024-03-06T07:36:03.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:36:03.327 INFO:teuthology.orchestra.run.smithi114.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:36:03.327 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:03.327 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:36:05.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:05.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.542+0000 7fa3fd400640 1 Processor -- start 2024-03-06T07:36:05.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.542+0000 7fa3fd400640 1 -- start start 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.543+0000 7fa3fd400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa3f815f9d0 0x7fa3f815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.543+0000 7fa3fd400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3f8070b10 0x7fa3f8070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.543+0000 7fa3fd400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f806cef0 con 0x7fa3f8144c00 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.543+0000 7fa3fd400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f806d1d0 con 0x7fa3f8070b10 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.543+0000 7fa3fd400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f806d4b0 con 0x7fa3f815f9d0 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3f8070b10 0x7fa3f8070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f6800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa3f8144c00 0x7fa3f806e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42152/0 (socket says 172.21.15.114:42152) 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f6800640 1 -- 172.21.15.114:0/683653313 learned_addr learned my addr 172.21.15.114:0/683653313 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f5e00640 1 --2- 172.21.15.114:0/683653313 >> v2:172.21.15.177:3300/0 conn(0x7fa3f815f9d0 0x7fa3f815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f7200640 1 -- 172.21.15.114:0/683653313 >> v2:172.21.15.177:3300/0 conn(0x7fa3f815f9d0 msgr2=0x7fa3f815fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f7200640 1 --2- 172.21.15.114:0/683653313 >> v2:172.21.15.177:3300/0 conn(0x7fa3f815f9d0 0x7fa3f815fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f7200640 1 -- 172.21.15.114:0/683653313 >> v1:172.21.15.114:6789/0 conn(0x7fa3f8144c00 legacy=0x7fa3f806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.544+0000 7fa3f7200640 1 -- 172.21.15.114:0/683653313 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f806d790 con 0x7fa3f8070b10 2024-03-06T07:36:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.546+0000 7fa3f7200640 1 --2- 172.21.15.114:0/683653313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f8070b10 0x7fa3f8070f20 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80b26f0 tx=0x7fa3f002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188c3087555bb880 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.546+0000 7fa3f5400640 1 -- 172.21.15.114:0/683653313 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0039070 con 0x7fa3f8070b10 2024-03-06T07:36:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3f5400640 1 -- 172.21.15.114:0/683653313 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f002cc70 con 0x7fa3f8070b10 2024-03-06T07:36:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3f5400640 1 -- 172.21.15.114:0/683653313 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0034660 con 0x7fa3f8070b10 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 -- 172.21.15.114:0/683653313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f8070b10 msgr2=0x7fa3f8070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 --2- 172.21.15.114:0/683653313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f8070b10 0x7fa3f8070f20 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80b26f0 tx=0x7fa3f002bec0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 -- 172.21.15.114:0/683653313 shutdown_connections 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 --2- 172.21.15.114:0/683653313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f8070b10 0x7fa3f8070f20 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 --2- 172.21.15.114:0/683653313 >> v2:172.21.15.177:3300/0 conn(0x7fa3f815f9d0 0x7fa3f815fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 -- 172.21.15.114:0/683653313 >> 172.21.15.114:0/683653313 conn(0x7fa3f80b1730 msgr2=0x7fa3f80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 -- 172.21.15.114:0/683653313 shutdown_connections 2024-03-06T07:36:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.547+0000 7fa3fd400640 1 -- 172.21.15.114:0/683653313 wait complete. 2024-03-06T07:36:05.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.549+0000 7fa3fd400640 1 Processor -- start 2024-03-06T07:36:05.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.549+0000 7fa3fd400640 1 -- start start 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.550+0000 7fa3fd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3f8070b10 0x7fa3f81533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.550+0000 7fa3fd400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3f8144c00 0x7fa3f8141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.550+0000 7fa3fd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f815f9d0 0x7fa3f81417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.550+0000 7fa3fd400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f806ccd0 con 0x7fa3f815f9d0 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.550+0000 7fa3fd400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f806cb50 con 0x7fa3f8070b10 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.550+0000 7fa3fd400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f806df90 con 0x7fa3f8144c00 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.550+0000 7fa3f7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f815f9d0 0x7fa3f81417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f815f9d0 0x7fa3f81417c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40336/0 (socket says 172.21.15.114:40336) 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 -- 172.21.15.114:0/1084451475 learned_addr learned my addr 172.21.15.114:0/1084451475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f6800640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3f8070b10 0x7fa3f81533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f5e00640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3f8144c00 0x7fa3f8141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 -- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3f8144c00 msgr2=0x7fa3f8141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3f8144c00 0x7fa3f8141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 -- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3f8070b10 msgr2=0x7fa3f81533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3f8070b10 0x7fa3f81533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 -- 172.21.15.114:0/1084451475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f813e060 con 0x7fa3f815f9d0 2024-03-06T07:36:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.551+0000 7fa3f7200640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f815f9d0 0x7fa3f81417c0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009e80 tx=0x7fa3f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.552+0000 7fa3eea00640 1 -- 172.21.15.114:0/1084451475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0039050 con 0x7fa3f815f9d0 2024-03-06T07:36:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.552+0000 7fa3eea00640 1 -- 172.21.15.114:0/1084451475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f0045070 con 0x7fa3f815f9d0 2024-03-06T07:36:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.552+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f8140340 con 0x7fa3f815f9d0 2024-03-06T07:36:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.552+0000 7fa3eea00640 1 -- 172.21.15.114:0/1084451475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f003e830 con 0x7fa3f815f9d0 2024-03-06T07:36:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.552+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f8140900 con 0x7fa3f815f9d0 2024-03-06T07:36:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.553+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f804f330 con 0x7fa3f815f9d0 2024-03-06T07:36:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.555+0000 7fa3eea00640 1 -- 172.21.15.114:0/1084451475 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3f003e9d0 con 0x7fa3f815f9d0 2024-03-06T07:36:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.556+0000 7fa3eea00640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa3d806b4d0 0x7fa3d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.556+0000 7fa3eea00640 1 -- 172.21.15.114:0/1084451475 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3f00ad070 con 0x7fa3f815f9d0 2024-03-06T07:36:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.556+0000 7fa3f6800640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa3d806b4d0 0x7fa3d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.557+0000 7fa3f6800640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa3d806b4d0 0x7fa3d806d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8000f60 tx=0x7fa3e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:05.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.562+0000 7fa3eea00640 1 -- 172.21.15.114:0/1084451475 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3f813e060 con 0x7fa3f815f9d0 2024-03-06T07:36:05.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.960+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3f8160010 con 0x7fa3f815f9d0 2024-03-06T07:36:05.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.960+0000 7fa3eea00640 1 -- 172.21.15.114:0/1084451475 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3424 (secure 0 0 0) 0x7fa3f002ce20 con 0x7fa3f815f9d0 2024-03-06T07:36:05.963 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:05.963 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:55.163774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8328,8331]},"id":9}]} 2024-03-06T07:36:05.964 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 101 2024-03-06T07:36:05.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.963+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa3d806b4d0 msgr2=0x7fa3d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.963+0000 7fa3fd400640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa3d806b4d0 0x7fa3d806d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8000f60 tx=0x7fa3e8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:05.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.963+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f815f9d0 msgr2=0x7fa3f81417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.963+0000 7fa3fd400640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f815f9d0 0x7fa3f81417c0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009e80 tx=0x7fa3f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 shutdown_connections 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa3d806b4d0 0x7fa3d806d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3f815f9d0 0x7fa3f81417c0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa3f8144c00 0x7fa3f8141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 --2- 172.21.15.114:0/1084451475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3f8070b10 0x7fa3f81533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 >> 172.21.15.114:0/1084451475 conn(0x7fa3f80b1730 msgr2=0x7fa3f8145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 shutdown_connections 2024-03-06T07:36:05.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:05.964+0000 7fa3fd400640 1 -- 172.21.15.114:0/1084451475 wait complete. 2024-03-06T07:36:05.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:36:06.227 INFO:teuthology.orchestra.run.smithi114.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:36:06.227 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:06.227 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:36:08.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:08.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.434+0000 7f8fad600640 1 Processor -- start 2024-03-06T07:36:08.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.434+0000 7f8fad600640 1 -- start start 2024-03-06T07:36:08.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.435+0000 7f8fad600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fa815be20 0x7f8fa80b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fad600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8fa80af5a0 0x7f8fa80b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fad600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa8066f50 con 0x7f8fa8072670 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fad600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa8066d00 con 0x7f8fa815be20 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fad600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa8067230 con 0x7f8fa80af5a0 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fa7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fa815be20 0x7f8fa80b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fa7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fa815be20 0x7f8fa80b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38684/0 (socket says 172.21.15.114:38684) 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fa7e00640 1 -- 172.21.15.114:0/1397894047 learned_addr learned my addr 172.21.15.114:0/1397894047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:08.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fa7e00640 1 -- 172.21.15.114:0/1397894047 >> v2:172.21.15.177:3300/0 conn(0x7f8fa80af5a0 msgr2=0x7f8fa80b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fa7e00640 1 --2- 172.21.15.114:0/1397894047 >> v2:172.21.15.177:3300/0 conn(0x7f8fa80af5a0 0x7f8fa80b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fa7e00640 1 -- 172.21.15.114:0/1397894047 >> v1:172.21.15.177:6789/0 conn(0x7f8fa8072670 legacy=0x7f8fa815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.436+0000 7f8fa7e00640 1 -- 172.21.15.114:0/1397894047 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80b0ea0 con 0x7f8fa815be20 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.438+0000 7f8fa7e00640 1 --2- 172.21.15.114:0/1397894047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa815be20 0x7f8fa80b5db0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f8fa806b4c0 tx=0x7f8f9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44992e2336e48c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.438+0000 7f8fa6a00640 1 -- 172.21.15.114:0/1397894047 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98035020 con 0x7f8fa815be20 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.438+0000 7f8fa6a00640 1 -- 172.21.15.114:0/1397894047 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f98031870 con 0x7f8fa815be20 2024-03-06T07:36:08.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fa6a00640 1 -- 172.21.15.114:0/1397894047 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98031c40 con 0x7f8fa815be20 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 -- 172.21.15.114:0/1397894047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa815be20 msgr2=0x7f8fa80b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 --2- 172.21.15.114:0/1397894047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa815be20 0x7f8fa80b5db0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f8fa806b4c0 tx=0x7f8f9802b910 comp rx=0 tx=0).stop 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 -- 172.21.15.114:0/1397894047 shutdown_connections 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 --2- 172.21.15.114:0/1397894047 >> v2:172.21.15.177:3300/0 conn(0x7f8fa80af5a0 0x7f8fa80b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 --2- 172.21.15.114:0/1397894047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa815be20 0x7f8fa80b5db0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 -- 172.21.15.114:0/1397894047 >> 172.21.15.114:0/1397894047 conn(0x7f8fa80b4fe0 msgr2=0x7f8fa80b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 -- 172.21.15.114:0/1397894047 shutdown_connections 2024-03-06T07:36:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.439+0000 7f8fad600640 1 -- 172.21.15.114:0/1397894047 wait complete. 2024-03-06T07:36:08.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.441+0000 7f8fad600640 1 Processor -- start 2024-03-06T07:36:08.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.441+0000 7f8fad600640 1 -- start start 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fad600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa8072670 0x7f8fa81f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fad600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa80af5a0 0x7f8fa8141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fad600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa815be20 0x7f8fa8141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fad600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa80b0b50 con 0x7f8fa80af5a0 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fad600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa80b1be0 con 0x7f8fa815be20 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fad600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa8072ac0 con 0x7f8fa8072670 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fa7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa80af5a0 0x7f8fa8141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fa7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa80af5a0 0x7f8fa8141300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38698/0 (socket says 172.21.15.114:38698) 2024-03-06T07:36:08.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.442+0000 7f8fa7400640 1 -- 172.21.15.114:0/2575862427 learned_addr learned my addr 172.21.15.114:0/2575862427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:08.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8facc00640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa815be20 0x7f8fa8141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8fa7e00640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa8072670 0x7f8fa81f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8fa7400640 1 -- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa8072670 msgr2=0x7f8fa81f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8fa7400640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa8072670 0x7f8fa81f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8fa7400640 1 -- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa815be20 msgr2=0x7f8fa8141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8fa7400640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa815be20 0x7f8fa8141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8fa7400640 1 -- 172.21.15.114:0/2575862427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa81f8330 con 0x7f8fa80af5a0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.443+0000 7f8fa7400640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa80af5a0 0x7f8fa8141300 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400b4a0 tx=0x7f8f9400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.444+0000 7f8f9fe00640 1 -- 172.21.15.114:0/2575862427 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94014070 con 0x7f8fa80af5a0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.444+0000 7f8f9fe00640 1 -- 172.21.15.114:0/2575862427 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f94010610 con 0x7f8fa80af5a0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.444+0000 7f8f9fe00640 1 -- 172.21.15.114:0/2575862427 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94010a00 con 0x7f8fa80af5a0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.444+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa8151090 con 0x7f8fa80af5a0 2024-03-06T07:36:08.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.444+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa8151650 con 0x7f8fa80af5a0 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.445+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa804f330 con 0x7f8fa80af5a0 2024-03-06T07:36:08.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.446+0000 7f8f9fe00640 1 -- 172.21.15.114:0/2575862427 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f94017070 con 0x7f8fa80af5a0 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.447+0000 7f8f9fe00640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.447+0000 7f8f9fe00640 1 -- 172.21.15.114:0/2575862427 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f94089c90 con 0x7f8fa80af5a0 2024-03-06T07:36:08.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.447+0000 7f8fa7e00640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.448+0000 7f8fa7e00640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f8806b4d0 0x7f8f8806d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8f98036000 tx=0x7f8f9803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.451+0000 7f8f9fe00640 1 -- 172.21.15.114:0/2575862427 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fa81f8330 con 0x7f8fa80af5a0 2024-03-06T07:36:08.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.769+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa806f020 con 0x7f8fa80af5a0 2024-03-06T07:36:08.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.770+0000 7f8f9fe00640 1 -- 172.21.15.114:0/2575862427 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3424 (secure 0 0 0) 0x7f8f94058a70 con 0x7f8fa80af5a0 2024-03-06T07:36:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:08.773 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:35:55.163774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8328,8331]},"id":9}]} 2024-03-06T07:36:08.773 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 101 2024-03-06T07:36:08.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f8806b4d0 msgr2=0x7f8f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f8806b4d0 0x7f8f8806d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8f98036000 tx=0x7f8f9803c040 comp rx=0 tx=0).stop 2024-03-06T07:36:08.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa80af5a0 msgr2=0x7f8fa8141300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa80af5a0 0x7f8fa8141300 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400b4a0 tx=0x7f8f9400b960 comp rx=0 tx=0).stop 2024-03-06T07:36:08.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 shutdown_connections 2024-03-06T07:36:08.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fa815be20 0x7f8fa8141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fa80af5a0 0x7f8fa8141300 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 --2- 172.21.15.114:0/2575862427 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fa8072670 0x7f8fa81f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 >> 172.21.15.114:0/2575862427 conn(0x7f8fa80b4fe0 msgr2=0x7f8fa81537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 shutdown_connections 2024-03-06T07:36:08.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:08.772+0000 7f8fad600640 1 -- 172.21.15.114:0/2575862427 wait complete. 2024-03-06T07:36:08.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:36:08.995 INFO:teuthology.orchestra.run.smithi114.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:36:08.995 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:08.996 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:36:10.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:11.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.220+0000 7fb6a9400640 1 Processor -- start 2024-03-06T07:36:11.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.220+0000 7fb6a9400640 1 -- start start 2024-03-06T07:36:11.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.221+0000 7fb6a9400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a40b40c0 0x7fb6a40b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.221+0000 7fb6a9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a40ae670 0x7fb6a415e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.221+0000 7fb6a9400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb6a41620c0 0x7fb6a40b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.221+0000 7fb6a9400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a4161900 con 0x7fb6a40ae670 2024-03-06T07:36:11.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.221+0000 7fb6a9400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a40b49e0 con 0x7fb6a41620c0 2024-03-06T07:36:11.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.221+0000 7fb6a9400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a415ebd0 con 0x7fb6a40b40c0 2024-03-06T07:36:11.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a40ae670 0x7fb6a415e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a40ae670 0x7fb6a415e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38700/0 (socket says 172.21.15.114:38700) 2024-03-06T07:36:11.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 -- 172.21.15.114:0/2873382548 learned_addr learned my addr 172.21.15.114:0/2873382548 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:11.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 -- 172.21.15.114:0/2873382548 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a40b40c0 msgr2=0x7fb6a40b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:11.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 --2- 172.21.15.114:0/2873382548 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a40b40c0 0x7fb6a40b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 -- 172.21.15.114:0/2873382548 >> v2:172.21.15.177:3300/0 conn(0x7fb6a41620c0 msgr2=0x7fb6a40b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:11.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 --2- 172.21.15.114:0/2873382548 >> v2:172.21.15.177:3300/0 conn(0x7fb6a41620c0 0x7fb6a40b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.222+0000 7fb6a3400640 1 -- 172.21.15.114:0/2873382548 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a415eeb0 con 0x7fb6a40ae670 2024-03-06T07:36:11.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.224+0000 7fb6a3400640 1 --2- 172.21.15.114:0/2873382548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40ae670 0x7fb6a415e2c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4070800 tx=0x7fb69002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1540b988456aab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.224+0000 7fb6a2a00640 1 -- 172.21.15.114:0/2873382548 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690035020 con 0x7fb6a40ae670 2024-03-06T07:36:11.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.224+0000 7fb6a2a00640 1 -- 172.21.15.114:0/2873382548 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb690031870 con 0x7fb6a40ae670 2024-03-06T07:36:11.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.224+0000 7fb6a2a00640 1 -- 172.21.15.114:0/2873382548 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690031c40 con 0x7fb6a40ae670 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 -- 172.21.15.114:0/2873382548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40ae670 msgr2=0x7fb6a415e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 --2- 172.21.15.114:0/2873382548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40ae670 0x7fb6a415e2c0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4070800 tx=0x7fb69002b910 comp rx=0 tx=0).stop 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 -- 172.21.15.114:0/2873382548 shutdown_connections 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 --2- 172.21.15.114:0/2873382548 >> v2:172.21.15.177:3300/0 conn(0x7fb6a41620c0 0x7fb6a40b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 --2- 172.21.15.114:0/2873382548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40ae670 0x7fb6a415e2c0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 --2- 172.21.15.114:0/2873382548 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a40b40c0 0x7fb6a40b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 -- 172.21.15.114:0/2873382548 >> 172.21.15.114:0/2873382548 conn(0x7fb6a406afc0 msgr2=0x7fb6a406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:11.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 -- 172.21.15.114:0/2873382548 shutdown_connections 2024-03-06T07:36:11.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.225+0000 7fb6a9400640 1 -- 172.21.15.114:0/2873382548 wait complete. 2024-03-06T07:36:11.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.227+0000 7fb6a9400640 1 Processor -- start 2024-03-06T07:36:11.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.227+0000 7fb6a9400640 1 -- start start 2024-03-06T07:36:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a40ae670 0x7fb6a4158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40b40c0 0x7fb6a415ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a9400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a41620c0 0x7fb6a415fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a415e800 con 0x7fb6a40b40c0 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a9400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4161ba0 con 0x7fb6a40ae670 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a9400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a415e980 con 0x7fb6a41620c0 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40b40c0 0x7fb6a415ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40b40c0 0x7fb6a415ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38712/0 (socket says 172.21.15.114:38712) 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a3400640 1 -- 172.21.15.114:0/3515528860 learned_addr learned my addr 172.21.15.114:0/3515528860 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.228+0000 7fb6a3e00640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a40ae670 0x7fb6a4158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a8a00640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a41620c0 0x7fb6a415fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a3400640 1 -- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a41620c0 msgr2=0x7fb6a415fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a3400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a41620c0 0x7fb6a415fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a3400640 1 -- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a40ae670 msgr2=0x7fb6a4158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a3400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a40ae670 0x7fb6a4158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a3400640 1 -- 172.21.15.114:0/3515528860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4158eb0 con 0x7fb6a40b40c0 2024-03-06T07:36:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a8a00640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a41620c0 0x7fb6a415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a3400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40b40c0 0x7fb6a415ae20 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb6900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb69be00640 1 -- 172.21.15.114:0/3515528860 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69003ad60 con 0x7fb6a40b40c0 2024-03-06T07:36:11.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb69be00640 1 -- 172.21.15.114:0/3515528860 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6900443f0 con 0x7fb6a40b40c0 2024-03-06T07:36:11.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb69be00640 1 -- 172.21.15.114:0/3515528860 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6900447e0 con 0x7fb6a40b40c0 2024-03-06T07:36:11.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.229+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a4159190 con 0x7fb6a40b40c0 2024-03-06T07:36:11.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.230+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a415f4b0 con 0x7fb6a40b40c0 2024-03-06T07:36:11.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.231+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a404f330 con 0x7fb6a40b40c0 2024-03-06T07:36:11.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.233+0000 7fb69be00640 1 -- 172.21.15.114:0/3515528860 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb69004b060 con 0x7fb6a40b40c0 2024-03-06T07:36:11.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.233+0000 7fb69be00640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.234+0000 7fb69be00640 1 -- 172.21.15.114:0/3515528860 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6900ac4a0 con 0x7fb6a40b40c0 2024-03-06T07:36:11.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.234+0000 7fb6a3e00640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.235+0000 7fb6a3e00640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb694009890 tx=0x7fb694008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.238+0000 7fb69be00640 1 -- 172.21.15.114:0/3515528860 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb69003f070 con 0x7fb6a40b40c0 2024-03-06T07:36:11.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.541+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6a40adf40 con 0x7fb6a40b40c0 2024-03-06T07:36:11.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.542+0000 7fb69be00640 1 -- 172.21.15.114:0/3515528860 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3426 (secure 0 0 0) 0x7fb69002f320 con 0x7fb6a40b40c0 2024-03-06T07:36:11.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:11.545 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":102,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:09.631804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331,8328]},"id":9}]} 2024-03-06T07:36:11.546 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 102 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 msgr2=0x7fb68406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb694009890 tx=0x7fb694008040 comp rx=0 tx=0).stop 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40b40c0 msgr2=0x7fb6a415ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40b40c0 0x7fb6a415ae20 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb690000c00 tx=0x7fb6900046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 shutdown_connections 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb6a41620c0 0x7fb6a415fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb68406b4d0 0x7fb68406d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a40b40c0 0x7fb6a415ae20 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 --2- 172.21.15.114:0/3515528860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a40ae670 0x7fb6a4158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 >> 172.21.15.114:0/3515528860 conn(0x7fb6a406afc0 msgr2=0x7fb6a4155450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 shutdown_connections 2024-03-06T07:36:11.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:11.545+0000 7fb6a9400640 1 -- 172.21.15.114:0/3515528860 wait complete. 2024-03-06T07:36:11.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:36:11.784 INFO:teuthology.orchestra.run.smithi114.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:36:11.785 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:36:11.785 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:36:11.785 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:36:11.785 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> stdin-killer -- python3 -c ' 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> import fcntl 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> import sys 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> import time 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:36:11.829 DEBUG:teuthology.orchestra.run.smithi114:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> while True: 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> try: 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> if os.read(0, 4096) == b"": 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> break 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> except BlockingIOError: 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> pass 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> time.sleep(2) 2024-03-06T07:36:11.830 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-03-06T07:36:11.876 DEBUG:teuthology.orchestra.run.smithi114:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:36:11.919 INFO:teuthology.orchestra.run.smithi114.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:36:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:36:11.993 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:12.920 DEBUG:teuthology.orchestra.run.smithi114:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:36:12.984 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 29h/41d Inode: 2199023255552 Links: 2 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-06 07:36:11.987751579 +0000 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-06 07:36:11.987751579 +0000 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-06 07:36:11.987751579 +0000 2024-03-06T07:36:12.985 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-06 07:36:11.987751579 +0000 2024-03-06T07:36:12.985 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:36:12.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:36:12.986 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:36:13.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:36:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:36:13.279 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:36:13.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:36:13.584 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:36:13.584 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:36:13.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:36:13.784 INFO:teuthology.orchestra.run.smithi114.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:36:13.784 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:36:13.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:36:13.994 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:14.018 INFO:teuthology.orchestra.run.smithi114.stdout:{"message":"","return_code":0} 2024-03-06T07:36:14.018 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:36:14.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json cache drop 2024-03-06T07:36:14.247 INFO:teuthology.orchestra.run.smithi114.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":378,"bytes":65362}},"duration":0.012832013999999999} 2024-03-06T07:36:14.248 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65362, "items": 378 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.012832014, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:36:14.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:36:14.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.464+0000 7ff270800640 1 Processor -- start 2024-03-06T07:36:14.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.464+0000 7ff270800640 1 -- start start 2024-03-06T07:36:14.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff270800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff2681618e0 0x7ff2680ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff270800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff2680cf6f0 0x7ff2680b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff270800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2680b32f0 con 0x7ff2680b2d80 2024-03-06T07:36:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff270800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2680d0d20 con 0x7ff2681618e0 2024-03-06T07:36:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff270800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2680d0f70 con 0x7ff2680cf6f0 2024-03-06T07:36:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff26de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff2681618e0 0x7ff2680ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff26d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff2680b2d80 0x7ff2680cefe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38940/0 (socket says 172.21.15.114:38940) 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.466+0000 7ff26d400640 1 -- 172.21.15.114:0/3969407648 learned_addr learned my addr 172.21.15.114:0/3969407648 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.467+0000 7ff26de00640 1 -- 172.21.15.114:0/3969407648 >> v2:172.21.15.177:3300/0 conn(0x7ff2680cf6f0 msgr2=0x7ff2680b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.467+0000 7ff26e800640 1 --2- 172.21.15.114:0/3969407648 >> v2:172.21.15.177:3300/0 conn(0x7ff2680cf6f0 0x7ff2680b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.467+0000 7ff26de00640 1 --2- 172.21.15.114:0/3969407648 >> v2:172.21.15.177:3300/0 conn(0x7ff2680cf6f0 0x7ff2680b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.467+0000 7ff26ca00640 1 -- 172.21.15.114:0/3969407648 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626311274 0 0) 0x7ff2680b32f0 con 0x7ff2680b2d80 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.467+0000 7ff26de00640 1 -- 172.21.15.114:0/3969407648 >> v1:172.21.15.114:6789/0 conn(0x7ff2680b2d80 legacy=0x7ff2680cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.467+0000 7ff26de00640 1 -- 172.21.15.114:0/3969407648 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2680d1250 con 0x7ff2681618e0 2024-03-06T07:36:14.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.467+0000 7ff26e800640 1 --2- 172.21.15.114:0/3969407648 >> v2:172.21.15.177:3300/0 conn(0x7ff2680cf6f0 0x7ff2680b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.468+0000 7ff26de00640 1 --2- 172.21.15.114:0/3969407648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2681618e0 0x7ff2680ceaa0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7ff26808da90 tx=0x7ff25c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98754a6016f907ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff26ca00640 1 -- 172.21.15.114:0/3969407648 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c035020 con 0x7ff2681618e0 2024-03-06T07:36:14.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff26ca00640 1 -- 172.21.15.114:0/3969407648 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c031830 con 0x7ff2681618e0 2024-03-06T07:36:14.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff26ca00640 1 -- 172.21.15.114:0/3969407648 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c031c00 con 0x7ff2681618e0 2024-03-06T07:36:14.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff270800640 1 -- 172.21.15.114:0/3969407648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2681618e0 msgr2=0x7ff2680ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff270800640 1 --2- 172.21.15.114:0/3969407648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2681618e0 0x7ff2680ceaa0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7ff26808da90 tx=0x7ff25c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff270800640 1 -- 172.21.15.114:0/3969407648 shutdown_connections 2024-03-06T07:36:14.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff270800640 1 --2- 172.21.15.114:0/3969407648 >> v2:172.21.15.177:3300/0 conn(0x7ff2680cf6f0 0x7ff2680b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff270800640 1 --2- 172.21.15.114:0/3969407648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2681618e0 0x7ff2680ceaa0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.469+0000 7ff270800640 1 -- 172.21.15.114:0/3969407648 >> 172.21.15.114:0/3969407648 conn(0x7ff2680d3e90 msgr2=0x7ff2680d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:14.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.470+0000 7ff270800640 1 -- 172.21.15.114:0/3969407648 shutdown_connections 2024-03-06T07:36:14.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.470+0000 7ff270800640 1 -- 172.21.15.114:0/3969407648 wait complete. 2024-03-06T07:36:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.471+0000 7ff270800640 1 Processor -- start 2024-03-06T07:36:14.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.471+0000 7ff270800640 1 -- start start 2024-03-06T07:36:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.472+0000 7ff270800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2680b2d80 0x7ff2680cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.472+0000 7ff270800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2680cf6f0 0x7ff2680c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.472+0000 7ff270800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2681618e0 0x7ff26814ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.472+0000 7ff270800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680d2da0 con 0x7ff2680b2d80 2024-03-06T07:36:14.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.472+0000 7ff270800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff268161cc0 con 0x7ff2680cf6f0 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.472+0000 7ff270800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680b1180 con 0x7ff2681618e0 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2680b2d80 0x7ff2680cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2680b2d80 0x7ff2680cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38734/0 (socket says 172.21.15.114:38734) 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 -- 172.21.15.114:0/2747186499 learned_addr learned my addr 172.21.15.114:0/2747186499 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26d400640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2680cf6f0 0x7ff2680c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26e800640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2681618e0 0x7ff26814ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 -- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2681618e0 msgr2=0x7ff26814ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2681618e0 0x7ff26814ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 -- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2680cf6f0 msgr2=0x7ff2680c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2680cf6f0 0x7ff2680c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 -- 172.21.15.114:0/2747186499 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2680c75d0 con 0x7ff2680b2d80 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26d400640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2680cf6f0 0x7ff2680c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.473+0000 7ff26de00640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2680b2d80 0x7ff2680cd1e0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff25c034040 tx=0x7ff25c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.474+0000 7ff262000640 1 -- 172.21.15.114:0/2747186499 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c041070 con 0x7ff2680b2d80 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.474+0000 7ff262000640 1 -- 172.21.15.114:0/2747186499 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c004230 con 0x7ff2680b2d80 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.474+0000 7ff262000640 1 -- 172.21.15.114:0/2747186499 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c035040 con 0x7ff2680b2d80 2024-03-06T07:36:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.474+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2680c78b0 con 0x7ff2680b2d80 2024-03-06T07:36:14.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.474+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2680c7e70 con 0x7ff2680b2d80 2024-03-06T07:36:14.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.475+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26804f330 con 0x7ff2680b2d80 2024-03-06T07:36:14.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.476+0000 7ff262000640 1 -- 172.21.15.114:0/2747186499 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff25c038020 con 0x7ff2680b2d80 2024-03-06T07:36:14.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.477+0000 7ff262000640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff24c06b4d0 0x7ff24c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:14.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.477+0000 7ff262000640 1 -- 172.21.15.114:0/2747186499 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff25c0abd70 con 0x7ff2680b2d80 2024-03-06T07:36:14.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.477+0000 7ff26d400640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff24c06b4d0 0x7ff24c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:14.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.478+0000 7ff26d400640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff24c06b4d0 0x7ff24c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff2680b6740 tx=0x7ff258009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:14.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.481+0000 7ff262000640 1 -- 172.21.15.114:0/2747186499 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff25c07abe0 con 0x7ff2680b2d80 2024-03-06T07:36:14.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.716+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7ff26815ff20 con 0x7ff2680b2d80 2024-03-06T07:36:14.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.921+0000 7ff262000640 1 -- 172.21.15.114:0/2747186499 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v103) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff25c07a4e0 con 0x7ff2680b2d80 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff24c06b4d0 msgr2=0x7ff24c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff24c06b4d0 0x7ff24c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff2680b6740 tx=0x7ff258009040 comp rx=0 tx=0).stop 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2680b2d80 msgr2=0x7ff2680cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2680b2d80 0x7ff2680cd1e0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7ff25c034040 tx=0x7ff25c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 shutdown_connections 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2681618e0 0x7ff26814ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff2680cf6f0 0x7ff2680c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff24c06b4d0 0x7ff24c06d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 --2- 172.21.15.114:0/2747186499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2680b2d80 0x7ff2680cd1e0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:14.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 >> 172.21.15.114:0/2747186499 conn(0x7ff2680d3e90 msgr2=0x7ff268160720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:14.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 shutdown_connections 2024-03-06T07:36:14.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:14.925+0000 7ff270800640 1 -- 172.21.15.114:0/2747186499 wait complete. 2024-03-06T07:36:14.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.140+0000 7fe545000640 1 Processor -- start 2024-03-06T07:36:15.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.140+0000 7fe545000640 1 -- start start 2024-03-06T07:36:15.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.142+0000 7fe545000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe54015f9d0 0x7fe54015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.142+0000 7fe545000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 0x7fe540070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.142+0000 7fe545000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54006cef0 con 0x7fe540144c00 2024-03-06T07:36:15.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.142+0000 7fe545000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54006d1d0 con 0x7fe54015f9d0 2024-03-06T07:36:15.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.142+0000 7fe545000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54006d4b0 con 0x7fe540070b10 2024-03-06T07:36:15.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.142+0000 7fe53de00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe540144c00 0x7fe54006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38948/0 (socket says 172.21.15.114:38948) 2024-03-06T07:36:15.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.142+0000 7fe53de00640 1 -- 172.21.15.114:0/2183110959 learned_addr learned my addr 172.21.15.114:0/2183110959 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:15.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173228930 0 0) 0x7fe54006cef0 con 0x7fe540144c00 2024-03-06T07:36:15.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe535400640 1 --2- 172.21.15.114:0/2183110959 >> v2:172.21.15.177:3300/0 conn(0x7fe54015f9d0 0x7fe54015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53e800640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 0x7fe540070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe524003520 con 0x7fe540144c00 2024-03-06T07:36:15.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 712404960 0 0) 0x7fe524003520 con 0x7fe540144c00 2024-03-06T07:36:15.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 msgr2=0x7fe540070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 0x7fe540070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 >> v2:172.21.15.177:3300/0 conn(0x7fe54015f9d0 msgr2=0x7fe54015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 --2- 172.21.15.114:0/2183110959 >> v2:172.21.15.177:3300/0 conn(0x7fe54015f9d0 0x7fe54015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54006d790 con 0x7fe540144c00 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fe528001670 con 0x7fe540144c00 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 >> v1:172.21.15.114:6789/0 conn(0x7fe540144c00 legacy=0x7fe54006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 0x7fe5240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe524003f80 0x7fe524004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524004d40 0x7fe5240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54006d1d0 con 0x7fe524003f80 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54006d4b0 con 0x7fe524004d40 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52400ba70 con 0x7fe540070b10 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.143+0000 7fe535400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe524003f80 0x7fe524004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe53de00640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 0x7fe5240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe535400640 1 -- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 msgr2=0x7fe5240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe53e800640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524004d40 0x7fe5240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe535400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe540070b10 0x7fe5240039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe535400640 1 -- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524004d40 msgr2=0x7fe5240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe535400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524004d40 0x7fe5240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe535400640 1 -- 172.21.15.114:0/2183110959 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524003520 con 0x7fe524003f80 2024-03-06T07:36:15.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.144+0000 7fe53e800640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524004d40 0x7fe5240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.145+0000 7fe535400640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe524003f80 0x7fe524004800 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fe530009ac0 tx=0x7fe524005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e76a5f9a53b5994 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.145+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530039070 con 0x7fe524003f80 2024-03-06T07:36:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.145+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53002c710 con 0x7fe524003f80 2024-03-06T07:36:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.145+0000 7fe53d400640 1 -- 172.21.15.114:0/2183110959 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530009690 con 0x7fe524003f80 2024-03-06T07:36:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.146+0000 7fe545000640 1 -- 172.21.15.114:0/2183110959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe524003f80 msgr2=0x7fe524004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.146+0000 7fe545000640 1 --2- 172.21.15.114:0/2183110959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe524003f80 0x7fe524004800 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fe530009ac0 tx=0x7fe524005830 comp rx=0 tx=0).stop 2024-03-06T07:36:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.146+0000 7fe53e800640 1 -- 172.21.15.114:0/2183110959 reap_dead start 2024-03-06T07:36:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.146+0000 7fe545000640 1 -- 172.21.15.114:0/2183110959 shutdown_connections 2024-03-06T07:36:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.146+0000 7fe545000640 1 -- 172.21.15.114:0/2183110959 >> 172.21.15.114:0/2183110959 conn(0x7fe5400b1730 msgr2=0x7fe5400b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.146+0000 7fe545000640 1 -- 172.21.15.114:0/2183110959 shutdown_connections 2024-03-06T07:36:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.146+0000 7fe545000640 1 -- 172.21.15.114:0/2183110959 wait complete. 2024-03-06T07:36:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.148+0000 7fe545000640 1 Processor -- start 2024-03-06T07:36:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.148+0000 7fe545000640 1 -- start start 2024-03-06T07:36:15.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.149+0000 7fe545000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524003f80 0x7fe54014de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe545000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54013e360 0x7fe54013f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe545000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe54006e590 0x7fe5401380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe545000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54006ccd0 con 0x7fe54013e360 2024-03-06T07:36:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe545000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54006cb50 con 0x7fe524003f80 2024-03-06T07:36:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe545000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54006df90 con 0x7fe54006e590 2024-03-06T07:36:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54013e360 0x7fe54013f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54013e360 0x7fe54013f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38764/0 (socket says 172.21.15.114:38764) 2024-03-06T07:36:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 -- 172.21.15.114:0/2258062585 learned_addr learned my addr 172.21.15.114:0/2258062585 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe53de00640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524003f80 0x7fe54014de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe53e800640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe54006e590 0x7fe5401380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 -- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe54006e590 msgr2=0x7fe5401380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe54006e590 0x7fe5401380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 -- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524003f80 msgr2=0x7fe54014de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524003f80 0x7fe54014de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe535400640 1 -- 172.21.15.114:0/2258062585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54014e340 con 0x7fe54013e360 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.150+0000 7fe53de00640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524003f80 0x7fe54014de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.151+0000 7fe535400640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54013e360 0x7fe54013f9c0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe530000c00 tx=0x7fe530004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.151+0000 7fe536a00640 1 -- 172.21.15.114:0/2258062585 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530039050 con 0x7fe54013e360 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.151+0000 7fe536a00640 1 -- 172.21.15.114:0/2258062585 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe530045070 con 0x7fe54013e360 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.151+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54013e920 con 0x7fe54013e360 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.151+0000 7fe536a00640 1 -- 172.21.15.114:0/2258062585 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5300044e0 con 0x7fe54013e360 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.151+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54013eee0 con 0x7fe54013e360 2024-03-06T07:36:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.152+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54004f330 con 0x7fe54013e360 2024-03-06T07:36:15.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.154+0000 7fe536a00640 1 -- 172.21.15.114:0/2258062585 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe530003a50 con 0x7fe54013e360 2024-03-06T07:36:15.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.155+0000 7fe536a00640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe51c06b4d0 0x7fe51c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.155+0000 7fe536a00640 1 -- 172.21.15.114:0/2258062585 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5300ab980 con 0x7fe54013e360 2024-03-06T07:36:15.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.155+0000 7fe53de00640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe51c06b4d0 0x7fe51c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.156+0000 7fe53de00640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe51c06b4d0 0x7fe51c06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe528005990 tx=0x7fe528000f70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.160+0000 7fe536a00640 1 -- 172.21.15.114:0/2258062585 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe53007a760 con 0x7fe54013e360 2024-03-06T07:36:15.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.392+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe540072890 con 0x7fe54013e360 2024-03-06T07:36:15.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.393+0000 7fe536a00640 1 -- 172.21.15.114:0/2258062585 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fe53007a060 con 0x7fe54013e360 2024-03-06T07:36:15.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:15.396 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:15.396 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe51c06b4d0 msgr2=0x7fe51c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe51c06b4d0 0x7fe51c06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe528005990 tx=0x7fe528000f70 comp rx=0 tx=0).stop 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54013e360 msgr2=0x7fe54013f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54013e360 0x7fe54013f9c0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fe530000c00 tx=0x7fe530004730 comp rx=0 tx=0).stop 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 shutdown_connections 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe54006e590 0x7fe5401380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe51c06b4d0 0x7fe51c06d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54013e360 0x7fe54013f9c0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 --2- 172.21.15.114:0/2258062585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe524003f80 0x7fe54014de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 >> 172.21.15.114:0/2258062585 conn(0x7fe5400b1730 msgr2=0x7fe5400b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 shutdown_connections 2024-03-06T07:36:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:15.396+0000 7fe545000640 1 -- 172.21.15.114:0/2258062585 wait complete. 2024-03-06T07:36:15.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:15.622 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:15.996 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:16.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:16.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.834+0000 7fa730a00640 1 Processor -- start 2024-03-06T07:36:16.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.834+0000 7fa730a00640 1 -- start start 2024-03-06T07:36:16.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.837+0000 7fa730a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa72c161580 0x7fa72c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.837+0000 7fa730a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d1e90 0x7fa72c0d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.837+0000 7fa730a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c161960 con 0x7fa72c0d28a0 2024-03-06T07:36:16.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.837+0000 7fa730a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72c0cf6f0 con 0x7fa72c161580 2024-03-06T07:36:16.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.837+0000 7fa730a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c0cf9d0 con 0x7fa72c0d1e90 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.837+0000 7fa72a800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa72c161580 0x7fa72c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.837+0000 7fa72b200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa72c0d28a0 0x7fa72c0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38956/0 (socket says 172.21.15.114:38956) 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.838+0000 7fa72b200640 1 -- 172.21.15.114:0/2827546925 learned_addr learned my addr 172.21.15.114:0/2827546925 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.838+0000 7fa721e00640 1 --2- 172.21.15.114:0/2827546925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d1e90 0x7fa72c0d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.838+0000 7fa72a800640 1 -- 172.21.15.114:0/2827546925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d1e90 msgr2=0x7fa72c0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.838+0000 7fa72a800640 1 --2- 172.21.15.114:0/2827546925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d1e90 0x7fa72c0d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.838+0000 7fa72a800640 1 -- 172.21.15.114:0/2827546925 >> v1:172.21.15.114:6789/0 conn(0x7fa72c0d28a0 legacy=0x7fa72c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.838+0000 7fa72a800640 1 -- 172.21.15.114:0/2827546925 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c0cfcb0 con 0x7fa72c161580 2024-03-06T07:36:16.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.839+0000 7fa72a800640 1 --2- 172.21.15.114:0/2827546925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c0d1950 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0097a0 tx=0x7fa71c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274adf4eabea63cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:16.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.839+0000 7fa729e00640 1 -- 172.21.15.114:0/2827546925 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c035020 con 0x7fa72c161580 2024-03-06T07:36:16.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa729e00640 1 -- 172.21.15.114:0/2827546925 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71c031810 con 0x7fa72c161580 2024-03-06T07:36:16.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa729e00640 1 -- 172.21.15.114:0/2827546925 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c031be0 con 0x7fa72c161580 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 -- 172.21.15.114:0/2827546925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 msgr2=0x7fa72c0d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 --2- 172.21.15.114:0/2827546925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c0d1950 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0097a0 tx=0x7fa71c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 -- 172.21.15.114:0/2827546925 shutdown_connections 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 --2- 172.21.15.114:0/2827546925 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d1e90 0x7fa72c0d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 --2- 172.21.15.114:0/2827546925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c0d1950 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 -- 172.21.15.114:0/2827546925 >> 172.21.15.114:0/2827546925 conn(0x7fa72c0d3d80 msgr2=0x7fa72c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 -- 172.21.15.114:0/2827546925 shutdown_connections 2024-03-06T07:36:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.840+0000 7fa730a00640 1 -- 172.21.15.114:0/2827546925 wait complete. 2024-03-06T07:36:16.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.843+0000 7fa730a00640 1 Processor -- start 2024-03-06T07:36:16.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.843+0000 7fa730a00640 1 -- start start 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa730a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa72c0d1e90 0x7fa72c0ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa730a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d28a0 0x7fa72c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa730a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa730a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c0cf3a0 con 0x7fa72c161580 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa730a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c0b39a0 con 0x7fa72c0d1e90 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa730a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c0b4ed0 con 0x7fa72c0d28a0 2024-03-06T07:36:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa72b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa72b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c15ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38788/0 (socket says 172.21.15.114:38788) 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa72b200640 1 -- 172.21.15.114:0/2201352048 learned_addr learned my addr 172.21.15.114:0/2201352048 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa72a800640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa72c0d1e90 0x7fa72c0ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.845+0000 7fa721e00640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d28a0 0x7fa72c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa72b200640 1 -- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d28a0 msgr2=0x7fa72c0c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa72b200640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d28a0 0x7fa72c0c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa72b200640 1 -- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa72c0d1e90 msgr2=0x7fa72c0ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa72b200640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa72c0d1e90 0x7fa72c0ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa72b200640 1 -- 172.21.15.114:0/2201352048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c0c7b80 con 0x7fa72c161580 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa72b200640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c15ac10 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa72400cbf0 tx=0x7fa72400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa723400640 1 -- 172.21.15.114:0/2201352048 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724017070 con 0x7fa72c161580 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa723400640 1 -- 172.21.15.114:0/2201352048 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72400ad40 con 0x7fa72c161580 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c0c7e60 con 0x7fa72c161580 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa723400640 1 -- 172.21.15.114:0/2201352048 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724012600 con 0x7fa72c161580 2024-03-06T07:36:16.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.846+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72c15b150 con 0x7fa72c161580 2024-03-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.848+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72c0d0130 con 0x7fa72c161580 2024-03-06T07:36:16.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.848+0000 7fa723400640 1 -- 172.21.15.114:0/2201352048 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa72401a070 con 0x7fa72c161580 2024-03-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.849+0000 7fa723400640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.849+0000 7fa723400640 1 -- 172.21.15.114:0/2201352048 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa724089ec0 con 0x7fa72c161580 2024-03-06T07:36:16.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.849+0000 7fa72a800640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.850+0000 7fa72a800640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa70806b4d0 0x7fa70806d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa71c036000 tx=0x7fa71c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:16.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:16.854+0000 7fa723400640 1 -- 172.21.15.114:0/2201352048 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa72c0d0130 con 0x7fa72c161580 2024-03-06T07:36:16.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:16.978 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:17.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.101+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa72c04f330 con 0x7fa72c161580 2024-03-06T07:36:17.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.102+0000 7fa723400640 1 -- 172.21.15.114:0/2201352048 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fa7240586e0 con 0x7fa72c161580 2024-03-06T07:36:17.103 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:17.103 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:17.104 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:17.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.105+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa70806b4d0 msgr2=0x7fa70806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:17.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.105+0000 7fa730a00640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa70806b4d0 0x7fa70806d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa71c036000 tx=0x7fa71c03c040 comp rx=0 tx=0).stop 2024-03-06T07:36:17.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.105+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 msgr2=0x7fa72c15ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.105+0000 7fa730a00640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c15ac10 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa72400cbf0 tx=0x7fa72400a430 comp rx=0 tx=0).stop 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 shutdown_connections 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa72c161580 0x7fa72c15ac10 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa72c0d28a0 0x7fa72c0c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 --2- 172.21.15.114:0/2201352048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa72c0d1e90 0x7fa72c0ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 >> 172.21.15.114:0/2201352048 conn(0x7fa72c0d3d80 msgr2=0x7fa72c0d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 shutdown_connections 2024-03-06T07:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:17.106+0000 7fa730a00640 1 -- 172.21.15.114:0/2201352048 wait complete. 2024-03-06T07:36:17.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:17.330 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:17.999 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:18.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:18.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.547+0000 7fb205e00640 1 Processor -- start 2024-03-06T07:36:18.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.548+0000 7fb205e00640 1 -- start start 2024-03-06T07:36:18.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.549+0000 7fb205e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb20006afc0 0x7fb20006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.549+0000 7fb205e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20006ea00 con 0x7fb2000cc470 2024-03-06T07:36:18.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.549+0000 7fb205e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20006b8e0 con 0x7fb20006f1b0 2024-03-06T07:36:18.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.549+0000 7fb205e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb200071ba0 con 0x7fb20006afc0 2024-03-06T07:36:18.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1ff400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb2000cc470 0x7fb20006e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49096/0 (socket says 172.21.15.114:49096) 2024-03-06T07:36:18.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1ff400640 1 -- 172.21.15.114:0/845014731 learned_addr learned my addr 172.21.15.114:0/845014731 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:18.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1fea00640 1 --2- 172.21.15.114:0/845014731 >> v2:172.21.15.114:3300/0 conn(0x7fb20006afc0 0x7fb20006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1fe000640 1 -- 172.21.15.114:0/845014731 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699975390 0 0) 0x7fb20006ea00 con 0x7fb2000cc470 2024-03-06T07:36:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1fe000640 1 -- 172.21.15.114:0/845014731 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4003520 con 0x7fb2000cc470 2024-03-06T07:36:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1fea00640 1 -- 172.21.15.114:0/845014731 >> v1:172.21.15.177:6789/0 conn(0x7fb20006f1b0 legacy=0x7fb20006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1fea00640 1 -- 172.21.15.114:0/845014731 >> v1:172.21.15.114:6789/0 conn(0x7fb2000cc470 legacy=0x7fb20006e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.550+0000 7fb1fea00640 1 -- 172.21.15.114:0/845014731 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb200071df0 con 0x7fb20006afc0 2024-03-06T07:36:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.551+0000 7fb1fea00640 1 --2- 172.21.15.114:0/845014731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb20006afc0 0x7fb20006b3a0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00097a0 tx=0x7fb1f002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b303d59eb56ee1c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.552+0000 7fb1fe000640 1 -- 172.21.15.114:0/845014731 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0034070 con 0x7fb20006afc0 2024-03-06T07:36:18.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.552+0000 7fb1fe000640 1 -- 172.21.15.114:0/845014731 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f0030610 con 0x7fb20006afc0 2024-03-06T07:36:18.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.552+0000 7fb1fe000640 1 -- 172.21.15.114:0/845014731 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f00309c0 con 0x7fb20006afc0 2024-03-06T07:36:18.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.552+0000 7fb205e00640 1 -- 172.21.15.114:0/845014731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb20006afc0 msgr2=0x7fb20006b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.552+0000 7fb205e00640 1 --2- 172.21.15.114:0/845014731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb20006afc0 0x7fb20006b3a0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00097a0 tx=0x7fb1f002b960 comp rx=0 tx=0).stop 2024-03-06T07:36:18.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.553+0000 7fb205e00640 1 -- 172.21.15.114:0/845014731 shutdown_connections 2024-03-06T07:36:18.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.553+0000 7fb205e00640 1 --2- 172.21.15.114:0/845014731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb20006afc0 0x7fb20006b3a0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.553+0000 7fb205e00640 1 -- 172.21.15.114:0/845014731 >> 172.21.15.114:0/845014731 conn(0x7fb2000d0ba0 msgr2=0x7fb2000d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:18.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.553+0000 7fb205e00640 1 -- 172.21.15.114:0/845014731 shutdown_connections 2024-03-06T07:36:18.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.553+0000 7fb205e00640 1 -- 172.21.15.114:0/845014731 wait complete. 2024-03-06T07:36:18.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.555+0000 7fb205e00640 1 Processor -- start 2024-03-06T07:36:18.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.556+0000 7fb205e00640 1 -- start start 2024-03-06T07:36:18.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.557+0000 7fb205e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb20006afc0 0x7fb20015af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.557+0000 7fb205e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb20006f1b0 0x7fb2000c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.557+0000 7fb205e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2000cc470 0x7fb2000c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.557+0000 7fb205e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000723b0 con 0x7fb2000cc470 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.557+0000 7fb205e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200072a00 con 0x7fb20006afc0 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.557+0000 7fb205e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb200072690 con 0x7fb20006f1b0 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.557+0000 7fb1ff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2000cc470 0x7fb2000c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2000cc470 0x7fb2000c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57404/0 (socket says 172.21.15.114:57404) 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 -- 172.21.15.114:0/2692805222 learned_addr learned my addr 172.21.15.114:0/2692805222 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1fea00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb20006afc0 0x7fb20015af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1f6000640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb20006f1b0 0x7fb2000c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 -- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb20006f1b0 msgr2=0x7fb2000c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb20006f1b0 0x7fb2000c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 -- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb20006afc0 msgr2=0x7fb20015af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb20006afc0 0x7fb20015af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 -- 172.21.15.114:0/2692805222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000b65d0 con 0x7fb2000cc470 2024-03-06T07:36:18.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1fea00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb20006afc0 0x7fb20015af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:18.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1f6000640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb20006f1b0 0x7fb2000c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:18.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1ff400640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2000cc470 0x7fb2000c9200 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800c390 tx=0x7fb1f800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1f7e00640 1 -- 172.21.15.114:0/2692805222 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8017070 con 0x7fb2000cc470 2024-03-06T07:36:18.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.558+0000 7fb1f7e00640 1 -- 172.21.15.114:0/2692805222 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f8005ce0 con 0x7fb2000cc470 2024-03-06T07:36:18.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.559+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2000b68b0 con 0x7fb2000cc470 2024-03-06T07:36:18.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.559+0000 7fb1f7e00640 1 -- 172.21.15.114:0/2692805222 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f80072d0 con 0x7fb2000cc470 2024-03-06T07:36:18.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.559+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20015c9e0 con 0x7fb2000cc470 2024-03-06T07:36:18.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.560+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20004f330 con 0x7fb2000cc470 2024-03-06T07:36:18.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.561+0000 7fb1f7e00640 1 -- 172.21.15.114:0/2692805222 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1f801a070 con 0x7fb2000cc470 2024-03-06T07:36:18.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.561+0000 7fb1f7e00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1dc06b4d0 0x7fb1dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.561+0000 7fb1f7e00640 1 -- 172.21.15.114:0/2692805222 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1f8089460 con 0x7fb2000cc470 2024-03-06T07:36:18.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.562+0000 7fb1fea00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1dc06b4d0 0x7fb1dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.563+0000 7fb1fea00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1dc06b4d0 0x7fb1dc06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0036000 tx=0x7fb1f0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.566+0000 7fb1f7e00640 1 -- 172.21.15.114:0/2692805222 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1f8057c80 con 0x7fb2000cc470 2024-03-06T07:36:18.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.798+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2000cdf40 con 0x7fb2000cc470 2024-03-06T07:36:18.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.798+0000 7fb1f7e00640 1 -- 172.21.15.114:0/2692805222 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fb1f805bd20 con 0x7fb2000cc470 2024-03-06T07:36:18.800 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:18.800 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:18.801 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1dc06b4d0 msgr2=0x7fb1dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1dc06b4d0 0x7fb1dc06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0036000 tx=0x7fb1f0008030 comp rx=0 tx=0).stop 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2000cc470 msgr2=0x7fb2000c9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2000cc470 0x7fb2000c9200 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800c390 tx=0x7fb1f800c850 comp rx=0 tx=0).stop 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 shutdown_connections 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb1dc06b4d0 0x7fb1dc06d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2000cc470 0x7fb2000c9200 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb20006f1b0 0x7fb2000c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 --2- 172.21.15.114:0/2692805222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb20006afc0 0x7fb20015af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 >> 172.21.15.114:0/2692805222 conn(0x7fb2000d0ba0 msgr2=0x7fb2000ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 shutdown_connections 2024-03-06T07:36:18.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:18.801+0000 7fb205e00640 1 -- 172.21.15.114:0/2692805222 wait complete. 2024-03-06T07:36:19.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:19.022 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:20.001 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:20.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.240+0000 7f2e1ae00640 1 Processor -- start 2024-03-06T07:36:20.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.240+0000 7f2e1ae00640 1 -- start start 2024-03-06T07:36:20.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.241+0000 7f2e1ae00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2e14071ba0 0x7f2e140c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e1ae00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2e140cf910 0x7f2e140cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e1ae00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e14144a20 con 0x7f2e14071f80 2024-03-06T07:36:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e1ae00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e140cf480 con 0x7f2e140cf910 2024-03-06T07:36:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e1ae00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1415bcd0 con 0x7f2e14071ba0 2024-03-06T07:36:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e18e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2e140cf910 0x7f2e140cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e18e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2e140cf910 0x7f2e140cfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57412/0 (socket says 172.21.15.114:57412) 2024-03-06T07:36:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e18e00640 1 -- 172.21.15.114:0/670745475 learned_addr learned my addr 172.21.15.114:0/670745475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e18e00640 1 -- 172.21.15.114:0/670745475 >> v2:172.21.15.177:3300/0 conn(0x7f2e14071ba0 msgr2=0x7f2e140c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e18e00640 1 --2- 172.21.15.114:0/670745475 >> v2:172.21.15.177:3300/0 conn(0x7f2e14071ba0 0x7f2e140c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.242+0000 7f2e18e00640 1 -- 172.21.15.114:0/670745475 >> v1:172.21.15.177:6789/0 conn(0x7f2e14071f80 legacy=0x7f2e1415b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.243+0000 7f2e18e00640 1 -- 172.21.15.114:0/670745475 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1415bfb0 con 0x7f2e140cf910 2024-03-06T07:36:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.244+0000 7f2e18e00640 1 --2- 172.21.15.114:0/670745475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e140cf910 0x7f2e140cfd60 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f2e140b3810 tx=0x7f2e0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be380107a4bd487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.245+0000 7f2e13400640 1 -- 172.21.15.114:0/670745475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c035020 con 0x7f2e140cf910 2024-03-06T07:36:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.245+0000 7f2e13400640 1 -- 172.21.15.114:0/670745475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c031870 con 0x7f2e140cf910 2024-03-06T07:36:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.245+0000 7f2e13400640 1 -- 172.21.15.114:0/670745475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c031c40 con 0x7f2e140cf910 2024-03-06T07:36:20.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.245+0000 7f2e1ae00640 1 -- 172.21.15.114:0/670745475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e140cf910 msgr2=0x7f2e140cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.245+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/670745475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e140cf910 0x7f2e140cfd60 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f2e140b3810 tx=0x7f2e0c02b910 comp rx=0 tx=0).stop 2024-03-06T07:36:20.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.246+0000 7f2e1ae00640 1 -- 172.21.15.114:0/670745475 shutdown_connections 2024-03-06T07:36:20.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.246+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/670745475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e140cf910 0x7f2e140cfd60 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.246+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/670745475 >> v2:172.21.15.177:3300/0 conn(0x7f2e14071ba0 0x7f2e140c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.246+0000 7f2e1ae00640 1 -- 172.21.15.114:0/670745475 >> 172.21.15.114:0/670745475 conn(0x7f2e140d0ba0 msgr2=0x7f2e140d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.246+0000 7f2e1ae00640 1 -- 172.21.15.114:0/670745475 shutdown_connections 2024-03-06T07:36:20.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.246+0000 7f2e1ae00640 1 -- 172.21.15.114:0/670745475 wait complete. 2024-03-06T07:36:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.248+0000 7f2e1ae00640 1 Processor -- start 2024-03-06T07:36:20.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.249+0000 7f2e1ae00640 1 -- start start 2024-03-06T07:36:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e1ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e14071ba0 0x7f2e1415a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e1ae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14071f80 0x7f2e14143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e1ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e140cf910 0x7f2e1415e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e1ae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140723e0 con 0x7f2e14071f80 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e1ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140cb1f0 con 0x7f2e140cf910 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e1ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140cf010 con 0x7f2e14071ba0 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e0b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14071f80 0x7f2e14143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e0b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14071f80 0x7f2e14143ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57420/0 (socket says 172.21.15.114:57420) 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.250+0000 7f2e0b400640 1 -- 172.21.15.114:0/460857877 learned_addr learned my addr 172.21.15.114:0/460857877 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e18e00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e140cf910 0x7f2e1415e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e13e00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e14071ba0 0x7f2e1415a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e0b400640 1 -- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e14071ba0 msgr2=0x7f2e1415a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e0b400640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e14071ba0 0x7f2e1415a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e0b400640 1 -- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e140cf910 msgr2=0x7f2e1415e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e0b400640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e140cf910 0x7f2e1415e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e0b400640 1 -- 172.21.15.114:0/460857877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140c9570 con 0x7f2e14071f80 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e13e00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e14071ba0 0x7f2e1415a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:20.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e18e00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e140cf910 0x7f2e1415e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e0b400640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14071f80 0x7f2e14143ac0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c430 tx=0x7f2dfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e10c00640 1 -- 172.21.15.114:0/460857877 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc011790 con 0x7f2e14071f80 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e10c00640 1 -- 172.21.15.114:0/460857877 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dfc017070 con 0x7f2e14071f80 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e10c00640 1 -- 172.21.15.114:0/460857877 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc011bb0 con 0x7f2e14071f80 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.251+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14144000 con 0x7f2e14071f80 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.252+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e140c8a90 con 0x7f2e14071f80 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.253+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1404f330 con 0x7f2e14071f80 2024-03-06T07:36:20.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.254+0000 7f2e10c00640 1 -- 172.21.15.114:0/460857877 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dfc01c460 con 0x7f2e14071f80 2024-03-06T07:36:20.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.255+0000 7f2e10c00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2df006b500 0x7f2df006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.255+0000 7f2e10c00640 1 -- 172.21.15.114:0/460857877 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dfc08a130 con 0x7f2e14071f80 2024-03-06T07:36:20.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.255+0000 7f2e13e00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2df006b500 0x7f2df006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.256+0000 7f2e13e00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2df006b500 0x7f2df006d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2e04007eb0 tx=0x7f2e040073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.260+0000 7f2e10c00640 1 -- 172.21.15.114:0/460857877 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dfc058f10 con 0x7f2e14071f80 2024-03-06T07:36:20.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.494+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e140ccff0 con 0x7f2e14071f80 2024-03-06T07:36:20.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.495+0000 7f2e10c00640 1 -- 172.21.15.114:0/460857877 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f2dfc058810 con 0x7f2e14071f80 2024-03-06T07:36:20.497 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:20.497 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:20.497 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:20.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2df006b500 msgr2=0x7f2df006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2df006b500 0x7f2df006d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2e04007eb0 tx=0x7f2e040073c0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14071f80 msgr2=0x7f2e14143ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14071f80 0x7f2e14143ac0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c430 tx=0x7f2dfc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 shutdown_connections 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e140cf910 0x7f2e1415e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2df006b500 0x7f2df006d9c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e14071f80 0x7f2e14143ac0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 --2- 172.21.15.114:0/460857877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2e14071ba0 0x7f2e1415a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.498+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 >> 172.21.15.114:0/460857877 conn(0x7f2e140d0ba0 msgr2=0x7f2e140cbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.499+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 shutdown_connections 2024-03-06T07:36:20.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:20.499+0000 7f2e1ae00640 1 -- 172.21.15.114:0/460857877 wait complete. 2024-03-06T07:36:20.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:20.721 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:21.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.932+0000 7efdbe400640 1 Processor -- start 2024-03-06T07:36:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.932+0000 7efdbe400640 1 -- start start 2024-03-06T07:36:21.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.933+0000 7efdbe400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efdb80cff20 0x7efdb80b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.933+0000 7efdbe400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb80d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.933+0000 7efdbe400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb80b5e00 con 0x7efdb80b73e0 2024-03-06T07:36:21.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.934+0000 7efdbe400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb80b6a10 con 0x7efdb80cff20 2024-03-06T07:36:21.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.934+0000 7efdbe400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80b60e0 con 0x7efdb80b7860 2024-03-06T07:36:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.934+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb80d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.934+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb80d0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:43548/0 (socket says 172.21.15.114:43548) 2024-03-06T07:36:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.934+0000 7efdb7e00640 1 -- 172.21.15.114:0/2826142853 learned_addr learned my addr 172.21.15.114:0/2826142853 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.934+0000 7efdb7400640 1 --2- 172.21.15.114:0/2826142853 >> v2:172.21.15.177:3300/0 conn(0x7efdb80cff20 0x7efdb80b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.935+0000 7efdb7e00640 1 -- 172.21.15.114:0/2826142853 >> v2:172.21.15.177:3300/0 conn(0x7efdb80cff20 msgr2=0x7efdb80b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.935+0000 7efdb7e00640 1 --2- 172.21.15.114:0/2826142853 >> v2:172.21.15.177:3300/0 conn(0x7efdb80cff20 0x7efdb80b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.935+0000 7efdb7e00640 1 -- 172.21.15.114:0/2826142853 >> v1:172.21.15.177:6789/0 conn(0x7efdb80b73e0 legacy=0x7efdb80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.935+0000 7efdb7e00640 1 -- 172.21.15.114:0/2826142853 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb80b63c0 con 0x7efdb80b7860 2024-03-06T07:36:21.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.936+0000 7efdb7e00640 1 --2- 172.21.15.114:0/2826142853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb80d0a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009e50 tx=0x7efdb002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19091806e3ece25b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.937+0000 7efdb6000640 1 -- 172.21.15.114:0/2826142853 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb002cb00 con 0x7efdb80b7860 2024-03-06T07:36:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.937+0000 7efdb6000640 1 -- 172.21.15.114:0/2826142853 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb002cca0 con 0x7efdb80b7860 2024-03-06T07:36:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.937+0000 7efdb6000640 1 -- 172.21.15.114:0/2826142853 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0034610 con 0x7efdb80b7860 2024-03-06T07:36:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.937+0000 7efdbe400640 1 -- 172.21.15.114:0/2826142853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 msgr2=0x7efdb80d0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.937+0000 7efdbe400640 1 --2- 172.21.15.114:0/2826142853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb80d0a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009e50 tx=0x7efdb002bf60 comp rx=0 tx=0).stop 2024-03-06T07:36:21.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.938+0000 7efdbe400640 1 -- 172.21.15.114:0/2826142853 shutdown_connections 2024-03-06T07:36:21.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.938+0000 7efdbe400640 1 --2- 172.21.15.114:0/2826142853 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb80d0a10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.938+0000 7efdbe400640 1 --2- 172.21.15.114:0/2826142853 >> v2:172.21.15.177:3300/0 conn(0x7efdb80cff20 0x7efdb80b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.938+0000 7efdbe400640 1 -- 172.21.15.114:0/2826142853 >> 172.21.15.114:0/2826142853 conn(0x7efdb80d1c90 msgr2=0x7efdb80d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:21.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.938+0000 7efdbe400640 1 -- 172.21.15.114:0/2826142853 shutdown_connections 2024-03-06T07:36:21.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.938+0000 7efdbe400640 1 -- 172.21.15.114:0/2826142853 wait complete. 2024-03-06T07:36:21.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.940+0000 7efdbe400640 1 Processor -- start 2024-03-06T07:36:21.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.940+0000 7efdbe400640 1 -- start start 2024-03-06T07:36:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdbe400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efdb80b73e0 0x7efdb815cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdbe400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb8146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdbe400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdb80cff20 0x7efdb8147220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdbe400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb8161a50 con 0x7efdb80cff20 2024-03-06T07:36:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdbe400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80b5af0 con 0x7efdb80b73e0 2024-03-06T07:36:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdbe400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80d0f50 con 0x7efdb80b7860 2024-03-06T07:36:21.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdb80cff20 0x7efdb8147220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdb80cff20 0x7efdb8147220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57422/0 (socket says 172.21.15.114:57422) 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdb7e00640 1 -- 172.21.15.114:0/3599559690 learned_addr learned my addr 172.21.15.114:0/3599559690 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdb7400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efdb80b73e0 0x7efdb815cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdb6a00640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb8146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdb7e00640 1 -- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 msgr2=0x7efdb8146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.941+0000 7efdb7e00640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb8146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdb7e00640 1 -- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efdb80b73e0 msgr2=0x7efdb815cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdb7e00640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efdb80b73e0 0x7efdb815cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:21.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdb7e00640 1 -- 172.21.15.114:0/3599559690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb809c770 con 0x7efdb80cff20 2024-03-06T07:36:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdb7e00640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdb80cff20 0x7efdb8147220 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7efdb0036040 tx=0x7efdb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdaf400640 1 -- 172.21.15.114:0/3599559690 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0043020 con 0x7efdb80cff20 2024-03-06T07:36:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdaf400640 1 -- 172.21.15.114:0/3599559690 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb002ce50 con 0x7efdb80cff20 2024-03-06T07:36:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdaf400640 1 -- 172.21.15.114:0/3599559690 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0009510 con 0x7efdb80cff20 2024-03-06T07:36:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb81479d0 con 0x7efdb80cff20 2024-03-06T07:36:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.942+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb8147f90 con 0x7efdb80cff20 2024-03-06T07:36:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.943+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb804f330 con 0x7efdb80cff20 2024-03-06T07:36:21.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.945+0000 7efdaf400640 1 -- 172.21.15.114:0/3599559690 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdb0046020 con 0x7efdb80cff20 2024-03-06T07:36:21.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.946+0000 7efdaf400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efd9806b4d0 0x7efd9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:21.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.946+0000 7efdaf400640 1 -- 172.21.15.114:0/3599559690 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdb00ac780 con 0x7efdb80cff20 2024-03-06T07:36:21.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.947+0000 7efdb7400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efd9806b4d0 0x7efd9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:21.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.947+0000 7efdb7400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efd9806b4d0 0x7efd9806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efda8009a30 tx=0x7efda8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:21.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:21.951+0000 7efdaf400640 1 -- 172.21.15.114:0/3599559690 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdb004d030 con 0x7efdb80cff20 2024-03-06T07:36:22.002 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:22.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.188+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdb80684a0 con 0x7efdb80cff20 2024-03-06T07:36:22.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.188+0000 7efdaf400640 1 -- 172.21.15.114:0/3599559690 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7efdb003c350 con 0x7efdb80cff20 2024-03-06T07:36:22.190 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:22.190 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:22.191 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efd9806b4d0 msgr2=0x7efd9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efd9806b4d0 0x7efd9806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efda8009a30 tx=0x7efda8008040 comp rx=0 tx=0).stop 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdb80cff20 msgr2=0x7efdb8147220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdb80cff20 0x7efdb8147220 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7efdb0036040 tx=0x7efdb0004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 shutdown_connections 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efd9806b4d0 0x7efd9806d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efdb80cff20 0x7efdb8147220 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efdb80b7860 0x7efdb8146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 --2- 172.21.15.114:0/3599559690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efdb80b73e0 0x7efdb815cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 >> 172.21.15.114:0/3599559690 conn(0x7efdb80d1c90 msgr2=0x7efdb80c6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 shutdown_connections 2024-03-06T07:36:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:22.191+0000 7efdbe400640 1 -- 172.21.15.114:0/3599559690 wait complete. 2024-03-06T07:36:22.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:22.412 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:23.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.629+0000 7fd69e400640 1 Processor -- start 2024-03-06T07:36:23.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.629+0000 7fd69e400640 1 -- start start 2024-03-06T07:36:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69e400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd698156320 0x7fd69815f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69e400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69815cd30 0x7fd6980ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69e400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69815fed0 con 0x7fd69806afc0 2024-03-06T07:36:23.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69e400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd698071dc0 con 0x7fd698156320 2024-03-06T07:36:23.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69e400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6981601b0 con 0x7fd69815cd30 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69d000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd698156320 0x7fd69815f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69d000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd698156320 0x7fd69815f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57438/0 (socket says 172.21.15.114:57438) 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.631+0000 7fd69d000640 1 -- 172.21.15.114:0/4109722065 learned_addr learned my addr 172.21.15.114:0/4109722065 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.632+0000 7fd69d000640 1 -- 172.21.15.114:0/4109722065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69815cd30 msgr2=0x7fd6980ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.632+0000 7fd68fe00640 1 --2- 172.21.15.114:0/4109722065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69815cd30 0x7fd6980ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.632+0000 7fd69d000640 1 --2- 172.21.15.114:0/4109722065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69815cd30 0x7fd6980ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.632+0000 7fd69d000640 1 -- 172.21.15.114:0/4109722065 >> v1:172.21.15.177:6789/0 conn(0x7fd69806afc0 legacy=0x7fd698072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.632+0000 7fd69d000640 1 -- 172.21.15.114:0/4109722065 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698160490 con 0x7fd698156320 2024-03-06T07:36:23.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.632+0000 7fd68fe00640 1 --2- 172.21.15.114:0/4109722065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69815cd30 0x7fd6980ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.633+0000 7fd69d000640 1 --2- 172.21.15.114:0/4109722065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698156320 0x7fd69815f8f0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fd688009870 tx=0x7fd68802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df83d80113b9d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.633+0000 7fd697e00640 1 -- 172.21.15.114:0/4109722065 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6880317c0 con 0x7fd698156320 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.633+0000 7fd697e00640 1 -- 172.21.15.114:0/4109722065 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd688037070 con 0x7fd698156320 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.633+0000 7fd697e00640 1 -- 172.21.15.114:0/4109722065 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688031b80 con 0x7fd698156320 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 -- 172.21.15.114:0/4109722065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698156320 msgr2=0x7fd69815f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 --2- 172.21.15.114:0/4109722065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698156320 0x7fd69815f8f0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fd688009870 tx=0x7fd68802b860 comp rx=0 tx=0).stop 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 -- 172.21.15.114:0/4109722065 shutdown_connections 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 --2- 172.21.15.114:0/4109722065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69815cd30 0x7fd6980ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 --2- 172.21.15.114:0/4109722065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd698156320 0x7fd69815f8f0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 -- 172.21.15.114:0/4109722065 >> 172.21.15.114:0/4109722065 conn(0x7fd69806c0d0 msgr2=0x7fd69806c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 -- 172.21.15.114:0/4109722065 shutdown_connections 2024-03-06T07:36:23.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.634+0000 7fd69e400640 1 -- 172.21.15.114:0/4109722065 wait complete. 2024-03-06T07:36:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.635+0000 7fd69e400640 1 Processor -- start 2024-03-06T07:36:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.636+0000 7fd69e400640 1 -- start start 2024-03-06T07:36:23.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.636+0000 7fd69e400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69806afc0 0x7fd698151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.636+0000 7fd69e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd698156320 0x7fd69806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.636+0000 7fd69e400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd69815cd30 0x7fd69806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.636+0000 7fd69e400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69815d1e0 con 0x7fd69815cd30 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.636+0000 7fd69e400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd698156700 con 0x7fd698156320 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.636+0000 7fd69e400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980cee10 con 0x7fd69806afc0 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd69815cd30 0x7fd69806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd69815cd30 0x7fd69806eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57454/0 (socket says 172.21.15.114:57454) 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 -- 172.21.15.114:0/1551089774 learned_addr learned my addr 172.21.15.114:0/1551089774 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69d000640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69806afc0 0x7fd698151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 -- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69806afc0 msgr2=0x7fd698151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd68fe00640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd698156320 0x7fd69806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69806afc0 0x7fd698151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 -- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd698156320 msgr2=0x7fd69806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd698156320 0x7fd69806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 -- 172.21.15.114:0/1551089774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6981e65b0 con 0x7fd69815cd30 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69d000640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69806afc0 0x7fd698151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd68fe00640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd698156320 0x7fd69806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.637+0000 7fd69da00640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd69815cd30 0x7fd69806eeb0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fd69000c7b0 tx=0x7fd69000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.638+0000 7fd695600640 1 -- 172.21.15.114:0/1551089774 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690004b60 con 0x7fd69815cd30 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.638+0000 7fd695600640 1 -- 172.21.15.114:0/1551089774 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd690004d00 con 0x7fd69815cd30 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.638+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6981e6890 con 0x7fd69815cd30 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.638+0000 7fd695600640 1 -- 172.21.15.114:0/1551089774 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69000aa90 con 0x7fd69815cd30 2024-03-06T07:36:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.638+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6981e6e50 con 0x7fd69815cd30 2024-03-06T07:36:23.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.639+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69804f330 con 0x7fd69815cd30 2024-03-06T07:36:23.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.640+0000 7fd695600640 1 -- 172.21.15.114:0/1551089774 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd69000ac30 con 0x7fd69815cd30 2024-03-06T07:36:23.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.641+0000 7fd695600640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd67406b4d0 0x7fd67406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.641+0000 7fd695600640 1 -- 172.21.15.114:0/1551089774 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd69008a570 con 0x7fd69815cd30 2024-03-06T07:36:23.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.641+0000 7fd69d000640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd67406b4d0 0x7fd67406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.642+0000 7fd69d000640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd67406b4d0 0x7fd67406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd688036000 tx=0x7fd688008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:23.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.645+0000 7fd695600640 1 -- 172.21.15.114:0/1551089774 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd690020070 con 0x7fd69815cd30 2024-03-06T07:36:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.880+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6980cd3f0 con 0x7fd69815cd30 2024-03-06T07:36:23.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.881+0000 7fd695600640 1 -- 172.21.15.114:0/1551089774 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fd690088e20 con 0x7fd69815cd30 2024-03-06T07:36:23.883 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:23.883 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:23.883 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd67406b4d0 msgr2=0x7fd67406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd67406b4d0 0x7fd67406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd688036000 tx=0x7fd688008030 comp rx=0 tx=0).stop 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd69815cd30 msgr2=0x7fd69806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd69815cd30 0x7fd69806eeb0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fd69000c7b0 tx=0x7fd69000cc70 comp rx=0 tx=0).stop 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 shutdown_connections 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd67406b4d0 0x7fd67406d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd69815cd30 0x7fd69806eeb0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd698156320 0x7fd69806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 --2- 172.21.15.114:0/1551089774 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd69806afc0 0x7fd698151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 >> 172.21.15.114:0/1551089774 conn(0x7fd69806c0d0 msgr2=0x7fd6980cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:23.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 shutdown_connections 2024-03-06T07:36:23.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:23.884+0000 7fd69e400640 1 -- 172.21.15.114:0/1551089774 wait complete. 2024-03-06T07:36:24.004 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:24.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:24.105 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:25.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.314+0000 7fb7ab400640 1 Processor -- start 2024-03-06T07:36:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.315+0000 7fb7ab400640 1 -- start start 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7ab400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb7a415f9d0 0x7fb7a415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7ab400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7a4070b10 0x7fb7a4070f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7ab400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a406cef0 con 0x7fb7a4144c00 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7ab400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a406d1d0 con 0x7fb7a4070b10 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7ab400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a406d4b0 con 0x7fb7a415f9d0 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7a9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7a4070b10 0x7fb7a4070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7a9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7a4070b10 0x7fb7a4070f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57460/0 (socket says 172.21.15.114:57460) 2024-03-06T07:36:25.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.316+0000 7fb7a9400640 1 -- 172.21.15.114:0/2805982471 learned_addr learned my addr 172.21.15.114:0/2805982471 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.317+0000 7fb7a9400640 1 -- 172.21.15.114:0/2805982471 >> v2:172.21.15.177:3300/0 conn(0x7fb7a415f9d0 msgr2=0x7fb7a415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.317+0000 7fb7a9400640 1 --2- 172.21.15.114:0/2805982471 >> v2:172.21.15.177:3300/0 conn(0x7fb7a415f9d0 0x7fb7a415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.317+0000 7fb7a9400640 1 -- 172.21.15.114:0/2805982471 >> v1:172.21.15.177:6789/0 conn(0x7fb7a4144c00 legacy=0x7fb7a406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.317+0000 7fb7a9400640 1 -- 172.21.15.114:0/2805982471 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a406d790 con 0x7fb7a4070b10 2024-03-06T07:36:25.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.318+0000 7fb7a9400640 1 --2- 172.21.15.114:0/2805982471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4070f20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40b26f0 tx=0x7fb79c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6658c431020f7a64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.318+0000 7fb7a3e00640 1 -- 172.21.15.114:0/2805982471 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c02ca80 con 0x7fb7a4070b10 2024-03-06T07:36:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.318+0000 7fb7a3e00640 1 -- 172.21.15.114:0/2805982471 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c02cc20 con 0x7fb7a4070b10 2024-03-06T07:36:25.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.318+0000 7fb7a3e00640 1 -- 172.21.15.114:0/2805982471 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0343f0 con 0x7fb7a4070b10 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 -- 172.21.15.114:0/2805982471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 msgr2=0x7fb7a4070f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 --2- 172.21.15.114:0/2805982471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4070f20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40b26f0 tx=0x7fb79c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 -- 172.21.15.114:0/2805982471 shutdown_connections 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 --2- 172.21.15.114:0/2805982471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4070f20 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 --2- 172.21.15.114:0/2805982471 >> v2:172.21.15.177:3300/0 conn(0x7fb7a415f9d0 0x7fb7a415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 -- 172.21.15.114:0/2805982471 >> 172.21.15.114:0/2805982471 conn(0x7fb7a40b1730 msgr2=0x7fb7a40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 -- 172.21.15.114:0/2805982471 shutdown_connections 2024-03-06T07:36:25.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.319+0000 7fb7ab400640 1 -- 172.21.15.114:0/2805982471 wait complete. 2024-03-06T07:36:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.321+0000 7fb7ab400640 1 Processor -- start 2024-03-06T07:36:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.321+0000 7fb7ab400640 1 -- start start 2024-03-06T07:36:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7ab400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7ab400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4144c00 0x7fb7a413d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7ab400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7a415f9d0 0x7fb7a413d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7ab400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a406ccd0 con 0x7fb7a4070b10 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7ab400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a406cb50 con 0x7fb7a4144c00 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7ab400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a406df90 con 0x7fb7a415f9d0 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7a8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7a8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4138ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57470/0 (socket says 172.21.15.114:57470) 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.322+0000 7fb7a8a00640 1 -- 172.21.15.114:0/1636193435 learned_addr learned my addr 172.21.15.114:0/1636193435 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a8a00640 1 -- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7a415f9d0 msgr2=0x7fb7a413d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a8a00640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7a415f9d0 0x7fb7a413d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a8a00640 1 -- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4144c00 msgr2=0x7fb7a413d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb79be00640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4144c00 0x7fb7a413d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a8a00640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4144c00 0x7fb7a413d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a8a00640 1 -- 172.21.15.114:0/1636193435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a415e0a0 con 0x7fb7a4070b10 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb79be00640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4144c00 0x7fb7a413d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a8a00640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4138ee0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c410 tx=0x7fb79400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a1600640 1 -- 172.21.15.114:0/1636193435 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794017070 con 0x7fb7a4070b10 2024-03-06T07:36:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a1600640 1 -- 172.21.15.114:0/1636193435 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7940035f0 con 0x7fb7a4070b10 2024-03-06T07:36:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7a1600640 1 -- 172.21.15.114:0/1636193435 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794003a20 con 0x7fb7a4070b10 2024-03-06T07:36:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.323+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a415e380 con 0x7fb7a4070b10 2024-03-06T07:36:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.324+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a415e940 con 0x7fb7a4070b10 2024-03-06T07:36:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.325+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a404f330 con 0x7fb7a4070b10 2024-03-06T07:36:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.326+0000 7fb7a1600640 1 -- 172.21.15.114:0/1636193435 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb794003bc0 con 0x7fb7a4070b10 2024-03-06T07:36:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.327+0000 7fb7a1600640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb78006b4d0 0x7fb78006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.327+0000 7fb7a1600640 1 -- 172.21.15.114:0/1636193435 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb794089570 con 0x7fb7a4070b10 2024-03-06T07:36:25.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.327+0000 7fb79be00640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb78006b4d0 0x7fb78006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.328+0000 7fb79be00640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb78006b4d0 0x7fb78006d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4140720 tx=0x7fb78c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.331+0000 7fb7a1600640 1 -- 172.21.15.114:0/1636193435 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7940583e0 con 0x7fb7a4070b10 2024-03-06T07:36:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.576+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7a4074be0 con 0x7fb7a4070b10 2024-03-06T07:36:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.577+0000 7fb7a1600640 1 -- 172.21.15.114:0/1636193435 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fb794057ce0 con 0x7fb7a4070b10 2024-03-06T07:36:25.579 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:25.580 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb78006b4d0 msgr2=0x7fb78006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb78006b4d0 0x7fb78006d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4140720 tx=0x7fb78c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 msgr2=0x7fb7a4138ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4138ee0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb79400c410 tx=0x7fb79400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 shutdown_connections 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb7a415f9d0 0x7fb7a413d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4144c00 0x7fb7a413d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb78006b4d0 0x7fb78006d990 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 --2- 172.21.15.114:0/1636193435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4070b10 0x7fb7a4138ee0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 >> 172.21.15.114:0/1636193435 conn(0x7fb7a40b1730 msgr2=0x7fb7a40b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.580+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 shutdown_connections 2024-03-06T07:36:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:25.581+0000 7fb7ab400640 1 -- 172.21.15.114:0/1636193435 wait complete. 2024-03-06T07:36:25.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:25.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:26.006 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:26.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:27.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.025+0000 7fba48a00640 1 Processor -- start 2024-03-06T07:36:27.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.025+0000 7fba48a00640 1 -- start start 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.026+0000 7fba48a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba44161580 0x7fba440d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.026+0000 7fba48a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba440d1e90 0x7fba440d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba48a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba44161960 con 0x7fba440d28a0 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba48a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba440cf6f0 con 0x7fba44161580 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba48a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba440cf9d0 con 0x7fba440d1e90 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba43400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba44161580 0x7fba440d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba43e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fba440d28a0 0x7fba440d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49104/0 (socket says 172.21.15.114:49104) 2024-03-06T07:36:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba43e00640 1 -- 172.21.15.114:0/2720242231 learned_addr learned my addr 172.21.15.114:0/2720242231 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:27.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba42a00640 1 --2- 172.21.15.114:0/2720242231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba440d1e90 0x7fba440d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba43400640 1 -- 172.21.15.114:0/2720242231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba440d1e90 msgr2=0x7fba440d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:27.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba43400640 1 --2- 172.21.15.114:0/2720242231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba440d1e90 0x7fba440d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba43400640 1 -- 172.21.15.114:0/2720242231 >> v1:172.21.15.114:6789/0 conn(0x7fba440d28a0 legacy=0x7fba440d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.027+0000 7fba43400640 1 -- 172.21.15.114:0/2720242231 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba440cfcb0 con 0x7fba44161580 2024-03-06T07:36:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.029+0000 7fba43400640 1 --2- 172.21.15.114:0/2720242231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba44161580 0x7fba440d1950 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fba34009d90 tx=0x7fba3402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c09cb94de45c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.029+0000 7fba42000640 1 -- 172.21.15.114:0/2720242231 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34039070 con 0x7fba44161580 2024-03-06T07:36:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.029+0000 7fba42000640 1 -- 172.21.15.114:0/2720242231 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3402cc60 con 0x7fba44161580 2024-03-06T07:36:27.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.029+0000 7fba42000640 1 -- 172.21.15.114:0/2720242231 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34034660 con 0x7fba44161580 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.029+0000 7fba48a00640 1 -- 172.21.15.114:0/2720242231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba44161580 msgr2=0x7fba440d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.029+0000 7fba48a00640 1 --2- 172.21.15.114:0/2720242231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba44161580 0x7fba440d1950 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fba34009d90 tx=0x7fba3402beb0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.030+0000 7fba48a00640 1 -- 172.21.15.114:0/2720242231 shutdown_connections 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.030+0000 7fba48a00640 1 --2- 172.21.15.114:0/2720242231 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba440d1e90 0x7fba440d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.030+0000 7fba48a00640 1 --2- 172.21.15.114:0/2720242231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba44161580 0x7fba440d1950 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.030+0000 7fba48a00640 1 -- 172.21.15.114:0/2720242231 >> 172.21.15.114:0/2720242231 conn(0x7fba440d3d80 msgr2=0x7fba440d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.030+0000 7fba48a00640 1 -- 172.21.15.114:0/2720242231 shutdown_connections 2024-03-06T07:36:27.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.030+0000 7fba48a00640 1 -- 172.21.15.114:0/2720242231 wait complete. 2024-03-06T07:36:27.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.031+0000 7fba48a00640 1 Processor -- start 2024-03-06T07:36:27.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.031+0000 7fba48a00640 1 -- start start 2024-03-06T07:36:27.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.032+0000 7fba48a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba440d1e90 0x7fba440b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.032+0000 7fba48a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba440d28a0 0x7fba440b7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.032+0000 7fba48a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba44161580 0x7fba4415fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.032+0000 7fba48a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba440cf3a0 con 0x7fba440d28a0 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.032+0000 7fba48a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba440b39a0 con 0x7fba440d1e90 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.032+0000 7fba48a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba440b4ed0 con 0x7fba44161580 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba440d28a0 0x7fba440b7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba440d28a0 0x7fba440b7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57488/0 (socket says 172.21.15.114:57488) 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 -- 172.21.15.114:0/196779209 learned_addr learned my addr 172.21.15.114:0/196779209 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 -- 172.21.15.114:0/196779209 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba44161580 msgr2=0x7fba4415fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba44161580 0x7fba4415fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 -- 172.21.15.114:0/196779209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba440d1e90 msgr2=0x7fba440b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba440d1e90 0x7fba440b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 -- 172.21.15.114:0/196779209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba440b8bc0 con 0x7fba440d28a0 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.033+0000 7fba42a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba440d28a0 0x7fba440b7510 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00d800 tx=0x7fba2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.034+0000 7fba3b400640 1 -- 172.21.15.114:0/196779209 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00b6c0 con 0x7fba440d28a0 2024-03-06T07:36:27.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.034+0000 7fba3b400640 1 -- 172.21.15.114:0/196779209 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c019070 con 0x7fba440d28a0 2024-03-06T07:36:27.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.034+0000 7fba3b400640 1 -- 172.21.15.114:0/196779209 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00ba80 con 0x7fba440d28a0 2024-03-06T07:36:27.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.034+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba440cae50 con 0x7fba440d28a0 2024-03-06T07:36:27.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.034+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba440cb410 con 0x7fba440d28a0 2024-03-06T07:36:27.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.035+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4404f330 con 0x7fba440d28a0 2024-03-06T07:36:27.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.036+0000 7fba3b400640 1 -- 172.21.15.114:0/196779209 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba2c00bc20 con 0x7fba440d28a0 2024-03-06T07:36:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.037+0000 7fba3b400640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba2006b4d0 0x7fba2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.037+0000 7fba3b400640 1 -- 172.21.15.114:0/196779209 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba2c08a6c0 con 0x7fba440d28a0 2024-03-06T07:36:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.037+0000 7fba43400640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba2006b4d0 0x7fba2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.038+0000 7fba43400640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba2006b4d0 0x7fba2006d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fba34036000 tx=0x7fba3402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.041+0000 7fba3b400640 1 -- 172.21.15.114:0/196779209 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba2c059530 con 0x7fba440d28a0 2024-03-06T07:36:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.277+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba440b97e0 con 0x7fba440d28a0 2024-03-06T07:36:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.278+0000 7fba3b400640 1 -- 172.21.15.114:0/196779209 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fba2c058e30 con 0x7fba440d28a0 2024-03-06T07:36:27.280 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:27.280 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba2006b4d0 msgr2=0x7fba2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba2006b4d0 0x7fba2006d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fba34036000 tx=0x7fba3402e000 comp rx=0 tx=0).stop 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba440d28a0 msgr2=0x7fba440b7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba440d28a0 0x7fba440b7510 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00d800 tx=0x7fba2c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 shutdown_connections 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fba44161580 0x7fba4415fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fba2006b4d0 0x7fba2006d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba440d28a0 0x7fba440b7510 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 --2- 172.21.15.114:0/196779209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba440d1e90 0x7fba440b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 >> 172.21.15.114:0/196779209 conn(0x7fba440d3d80 msgr2=0x7fba4406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 shutdown_connections 2024-03-06T07:36:27.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:27.281+0000 7fba48a00640 1 -- 172.21.15.114:0/196779209 wait complete. 2024-03-06T07:36:27.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:27.501 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:28.008 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:28.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.712+0000 7fd5a6600640 1 Processor -- start 2024-03-06T07:36:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.712+0000 7fd5a6600640 1 -- start start 2024-03-06T07:36:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.713+0000 7fd5a6600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a0144c00 0x7fd5a006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.713+0000 7fd5a6600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd5a015fe30 0x7fd5a00ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.713+0000 7fd5a6600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a006ee70 con 0x7fd5a015f9d0 2024-03-06T07:36:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.713+0000 7fd5a6600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a006cef0 con 0x7fd5a015fe30 2024-03-06T07:36:28.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.713+0000 7fd5a6600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a006d1d0 con 0x7fd5a0144c00 2024-03-06T07:36:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.714+0000 7fd59f400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a0144c00 0x7fd5a006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.714+0000 7fd59fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd5a015fe30 0x7fd5a00ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.714+0000 7fd597e00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fd5a015f9d0 0x7fd5a00ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:36322/0 (socket says 172.21.15.114:36322) 2024-03-06T07:36:28.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.714+0000 7fd597e00640 1 -- 172.21.15.114:0/1815383511 learned_addr learned my addr 172.21.15.114:0/1815383511 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.715+0000 7fd59fe00640 1 -- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a0144c00 msgr2=0x7fd5a006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.715+0000 7fd59fe00640 1 --2- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a0144c00 0x7fd5a006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.715+0000 7fd59fe00640 1 -- 172.21.15.114:0/1815383511 >> v1:172.21.15.177:6789/0 conn(0x7fd5a015f9d0 legacy=0x7fd5a00ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.715+0000 7fd59fe00640 1 -- 172.21.15.114:0/1815383511 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a006d4b0 con 0x7fd5a015fe30 2024-03-06T07:36:28.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.715+0000 7fd59f400640 1 --2- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a0144c00 0x7fd5a006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:36:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.717+0000 7fd59fe00640 1 --2- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a00ce7b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=134d3ccc1a704c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.717+0000 7fd59ea00640 1 -- 172.21.15.114:0/1815383511 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598039070 con 0x7fd5a015fe30 2024-03-06T07:36:28.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.717+0000 7fd59ea00640 1 -- 172.21.15.114:0/1815383511 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59802cc60 con 0x7fd5a015fe30 2024-03-06T07:36:28.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.717+0000 7fd59ea00640 1 -- 172.21.15.114:0/1815383511 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598034660 con 0x7fd5a015fe30 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 -- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 msgr2=0x7fd5a00ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 --2- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a00ce7b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59802beb0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 -- 172.21.15.114:0/1815383511 shutdown_connections 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 --2- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a00ce7b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 --2- 172.21.15.114:0/1815383511 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a0144c00 0x7fd5a006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 -- 172.21.15.114:0/1815383511 >> 172.21.15.114:0/1815383511 conn(0x7fd5a00b1730 msgr2=0x7fd5a00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 -- 172.21.15.114:0/1815383511 shutdown_connections 2024-03-06T07:36:28.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.718+0000 7fd5a6600640 1 -- 172.21.15.114:0/1815383511 wait complete. 2024-03-06T07:36:28.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.720+0000 7fd5a6600640 1 Processor -- start 2024-03-06T07:36:28.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.720+0000 7fd5a6600640 1 -- start start 2024-03-06T07:36:28.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd5a6600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a0144c00 0x7fd5a0143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd5a6600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a015f9d0 0x7fd5a013c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd5a6600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a0140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd5a6600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0160730 con 0x7fd5a0144c00 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd5a6600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0160280 con 0x7fd5a015fe30 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd5a6600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a006cb50 con 0x7fd5a015f9d0 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd59f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a0144c00 0x7fd5a0143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd59f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a0144c00 0x7fd5a0143080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48126/0 (socket says 172.21.15.114:48126) 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd59f400640 1 -- 172.21.15.114:0/3589308348 learned_addr learned my addr 172.21.15.114:0/3589308348 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd59fe00640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a0140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.721+0000 7fd597e00640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a015f9d0 0x7fd5a013c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd59f400640 1 -- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a015f9d0 msgr2=0x7fd5a013c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd59f400640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a015f9d0 0x7fd5a013c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd59f400640 1 -- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 msgr2=0x7fd5a0140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd59f400640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a0140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd59f400640 1 -- 172.21.15.114:0/3589308348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a01435c0 con 0x7fd5a0144c00 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd59fe00640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a0140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd59f400640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a0144c00 0x7fd5a0143080 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd59000e960 tx=0x7fd59000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd597400640 1 -- 172.21.15.114:0/3589308348 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590017070 con 0x7fd5a0144c00 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd597400640 1 -- 172.21.15.114:0/3589308348 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59000ca00 con 0x7fd5a0144c00 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd597400640 1 -- 172.21.15.114:0/3589308348 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59000cdd0 con 0x7fd5a0144c00 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.722+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a015bb40 con 0x7fd5a0144c00 2024-03-06T07:36:28.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.723+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a013f830 con 0x7fd5a0144c00 2024-03-06T07:36:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.724+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a004f330 con 0x7fd5a0144c00 2024-03-06T07:36:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.725+0000 7fd597400640 1 -- 172.21.15.114:0/3589308348 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd59001a070 con 0x7fd5a0144c00 2024-03-06T07:36:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.725+0000 7fd597400640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd57c06b4d0 0x7fd57c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.726+0000 7fd597400640 1 -- 172.21.15.114:0/3589308348 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd59008a510 con 0x7fd5a0144c00 2024-03-06T07:36:28.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.726+0000 7fd597e00640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd57c06b4d0 0x7fd57c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.726+0000 7fd597e00640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd57c06b4d0 0x7fd57c06d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd588000f60 tx=0x7fd588009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.730+0000 7fd597400640 1 -- 172.21.15.114:0/3589308348 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5900592f0 con 0x7fd5a0144c00 2024-03-06T07:36:28.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.961+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5a0140700 con 0x7fd5a0144c00 2024-03-06T07:36:28.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.962+0000 7fd597400640 1 -- 172.21.15.114:0/3589308348 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fd590058bf0 con 0x7fd5a0144c00 2024-03-06T07:36:28.964 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:28.964 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:28.965 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:28.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd57c06b4d0 msgr2=0x7fd57c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd57c06b4d0 0x7fd57c06d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd588000f60 tx=0x7fd588009040 comp rx=0 tx=0).stop 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a0144c00 msgr2=0x7fd5a0143080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a0144c00 0x7fd5a0143080 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd59000e960 tx=0x7fd59000ee20 comp rx=0 tx=0).stop 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 shutdown_connections 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd5a015fe30 0x7fd5a0140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd5a015f9d0 0x7fd5a013c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd57c06b4d0 0x7fd57c06d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 --2- 172.21.15.114:0/3589308348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5a0144c00 0x7fd5a0143080 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 >> 172.21.15.114:0/3589308348 conn(0x7fd5a00b1730 msgr2=0x7fd5a0071910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 shutdown_connections 2024-03-06T07:36:28.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:28.965+0000 7fd5a6600640 1 -- 172.21.15.114:0/3589308348 wait complete. 2024-03-06T07:36:29.185 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:29.185 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:30.010 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:30.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.397+0000 7ffb12c00640 1 Processor -- start 2024-03-06T07:36:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.397+0000 7ffb12c00640 1 -- start start 2024-03-06T07:36:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb12c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ffb0c0d2360 0x7ffb0c0d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb12c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b3ec0 0x7ffb0c0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb12c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0c161960 con 0x7ffb0c161580 2024-03-06T07:36:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb12c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c0d2ce0 con 0x7ffb0c0d2360 2024-03-06T07:36:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb12c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0cf6f0 con 0x7ffb0c0b3ec0 2024-03-06T07:36:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb11800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ffb0c161580 0x7ffb0c0d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39948/0 (socket says 172.21.15.114:39948) 2024-03-06T07:36:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb11800640 1 -- 172.21.15.114:0/1127804062 learned_addr learned my addr 172.21.15.114:0/1127804062 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb12200640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b3ec0 0x7ffb0c0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb10e00640 1 --2- 172.21.15.114:0/1127804062 >> v2:172.21.15.177:3300/0 conn(0x7ffb0c0d2360 0x7ffb0c0d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483214131 0 0) 0x7ffb0c161960 con 0x7ffb0c161580 2024-03-06T07:36:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.399+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf4003520 con 0x7ffb0c161580 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188297542 0 0) 0x7ffaf4003520 con 0x7ffb0c161580 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b3ec0 msgr2=0x7ffb0c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b3ec0 0x7ffb0c0b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 >> v2:172.21.15.177:3300/0 conn(0x7ffb0c0d2360 msgr2=0x7ffb0c0d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 --2- 172.21.15.114:0/1127804062 >> v2:172.21.15.177:3300/0 conn(0x7ffb0c0d2360 0x7ffb0c0d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0cf9d0 con 0x7ffb0c161580 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7ffb000019f0 con 0x7ffb0c161580 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 >> v1:172.21.15.114:6789/0 conn(0x7ffb0c161580 legacy=0x7ffb0c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffaf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b3ec0 0x7ffaf4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf4004d40 0x7ffaf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0d2ce0 con 0x7ffaf40039b0 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0cf6f0 con 0x7ffaf4004d40 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf400b9c0 con 0x7ffb0c0b3ec0 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.400+0000 7ffb11800640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffaf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.401+0000 7ffb11800640 1 -- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b3ec0 msgr2=0x7ffaf4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.401+0000 7ffb11800640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b3ec0 0x7ffaf4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.401+0000 7ffb11800640 1 -- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf4004d40 msgr2=0x7ffaf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.401+0000 7ffb11800640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffaf4004d40 0x7ffaf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.401+0000 7ffb11800640 1 -- 172.21.15.114:0/1127804062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf4003520 con 0x7ffaf40039b0 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.402+0000 7ffb11800640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffaf4003dd0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000a7d0 tx=0x7ffaf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce9e8e1edca4cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.402+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0000a3f0 con 0x7ffaf40039b0 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.402+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00034630 con 0x7ffaf40039b0 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.402+0000 7ffb0be00640 1 -- 172.21.15.114:0/1127804062 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0002ce60 con 0x7ffaf40039b0 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.402+0000 7ffb12c00640 1 -- 172.21.15.114:0/1127804062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 msgr2=0x7ffaf4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.402+0000 7ffb12c00640 1 --2- 172.21.15.114:0/1127804062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffaf4003dd0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000a7d0 tx=0x7ffaf4005830 comp rx=0 tx=0).stop 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.403+0000 7ffb12200640 1 -- 172.21.15.114:0/1127804062 reap_dead start 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.403+0000 7ffb12c00640 1 -- 172.21.15.114:0/1127804062 shutdown_connections 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.403+0000 7ffb12c00640 1 -- 172.21.15.114:0/1127804062 >> 172.21.15.114:0/1127804062 conn(0x7ffb0c0d3d80 msgr2=0x7ffb0c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:30.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.403+0000 7ffb12c00640 1 -- 172.21.15.114:0/1127804062 shutdown_connections 2024-03-06T07:36:30.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.403+0000 7ffb12c00640 1 -- 172.21.15.114:0/1127804062 wait complete. 2024-03-06T07:36:30.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.404+0000 7ffb12c00640 1 Processor -- start 2024-03-06T07:36:30.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.405+0000 7ffb12c00640 1 -- start start 2024-03-06T07:36:30.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.405+0000 7ffb12c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffb0c152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb12c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b6970 0x7ffb0c0b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb12c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0c15faf0 0x7ffb0c15ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb12c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0b39a0 con 0x7ffaf40039b0 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb12c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0b36c0 con 0x7ffb0c15faf0 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb12c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0cf3a0 con 0x7ffb0c0b6970 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffb0c152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffb0c152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48132/0 (socket says 172.21.15.114:48132) 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 -- 172.21.15.114:0/1755585161 learned_addr learned my addr 172.21.15.114:0/1755585161 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb10e00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b6970 0x7ffb0c0b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb12200640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0c15faf0 0x7ffb0c15ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 -- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b6970 msgr2=0x7ffb0c0b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b6970 0x7ffb0c0b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 -- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0c15faf0 msgr2=0x7ffb0c15ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0c15faf0 0x7ffb0c15ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb11800640 1 -- 172.21.15.114:0/1755585161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0b74e0 con 0x7ffaf40039b0 2024-03-06T07:36:30.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.406+0000 7ffb12200640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0c15faf0 0x7ffb0c15ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.407+0000 7ffb11800640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffb0c152170 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000a8f0 tx=0x7ffb00001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.407+0000 7ffb09600640 1 -- 172.21.15.114:0/1755585161 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0002ce60 con 0x7ffaf40039b0 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.407+0000 7ffb09600640 1 -- 172.21.15.114:0/1755585161 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00045070 con 0x7ffaf40039b0 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.407+0000 7ffb09600640 1 -- 172.21.15.114:0/1755585161 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb000349f0 con 0x7ffaf40039b0 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.407+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c0b77c0 con 0x7ffaf40039b0 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.407+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c0b7d80 con 0x7ffaf40039b0 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.408+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c04f330 con 0x7ffaf40039b0 2024-03-06T07:36:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.409+0000 7ffb09600640 1 -- 172.21.15.114:0/1755585161 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb00034b90 con 0x7ffaf40039b0 2024-03-06T07:36:30.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.411+0000 7ffb09600640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffaec06b500 0x7ffaec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.411+0000 7ffb09600640 1 -- 172.21.15.114:0/1755585161 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb000ac9e0 con 0x7ffaf40039b0 2024-03-06T07:36:30.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.411+0000 7ffb10e00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffaec06b500 0x7ffaec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.412+0000 7ffb10e00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffaec06b500 0x7ffaec06d9c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c15b500 tx=0x7ffaf8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.415+0000 7ffb09600640 1 -- 172.21.15.114:0/1755585161 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb0003e350 con 0x7ffaf40039b0 2024-03-06T07:36:30.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.668+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb0c0d3b60 con 0x7ffaf40039b0 2024-03-06T07:36:30.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.669+0000 7ffb09600640 1 -- 172.21.15.114:0/1755585161 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7ffb0007b7c0 con 0x7ffaf40039b0 2024-03-06T07:36:30.671 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:30.671 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:30.671 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffaec06b500 msgr2=0x7ffaec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffaec06b500 0x7ffaec06d9c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c15b500 tx=0x7ffaf8008040 comp rx=0 tx=0).stop 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 msgr2=0x7ffb0c152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffb0c152170 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000a8f0 tx=0x7ffb00001cf0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 shutdown_connections 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0c15faf0 0x7ffb0c15ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ffb0c0b6970 0x7ffb0c0b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ffaec06b500 0x7ffaec06d9c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 --2- 172.21.15.114:0/1755585161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaf40039b0 0x7ffb0c152170 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 >> 172.21.15.114:0/1755585161 conn(0x7ffb0c0d3d80 msgr2=0x7ffb0c06c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 shutdown_connections 2024-03-06T07:36:30.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:30.672+0000 7ffb12c00640 1 -- 172.21.15.114:0/1755585161 wait complete. 2024-03-06T07:36:30.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:30.896 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:31.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:32.012 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:32.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.105+0000 7fcf3ca00640 1 Processor -- start 2024-03-06T07:36:32.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.105+0000 7fcf3ca00640 1 -- start start 2024-03-06T07:36:32.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.107+0000 7fcf3ca00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf3815e0d0 0x7fcf380d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf3ca00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcf3815f470 0x7fcf380d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf3ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf380d0df0 0x7fcf38160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf3ca00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf380d2f60 con 0x7fcf3815e0d0 2024-03-06T07:36:32.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf3ca00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf380d1240 con 0x7fcf3815f470 2024-03-06T07:36:32.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf3ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380cea60 con 0x7fcf380d0df0 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf3815e0d0 0x7fcf380d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf3815e0d0 0x7fcf380d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48134/0 (socket says 172.21.15.114:48134) 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 -- 172.21.15.114:0/1072731328 learned_addr learned my addr 172.21.15.114:0/1072731328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35400640 1 --2- 172.21.15.114:0/1072731328 >> v2:172.21.15.177:3300/0 conn(0x7fcf3815f470 0x7fcf380d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf36800640 1 --2- 172.21.15.114:0/1072731328 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf380d0df0 0x7fcf38160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 -- 172.21.15.114:0/1072731328 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf380d0df0 msgr2=0x7fcf38160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 --2- 172.21.15.114:0/1072731328 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf380d0df0 0x7fcf38160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 -- 172.21.15.114:0/1072731328 >> v2:172.21.15.177:3300/0 conn(0x7fcf3815f470 msgr2=0x7fcf380d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 --2- 172.21.15.114:0/1072731328 >> v2:172.21.15.177:3300/0 conn(0x7fcf3815f470 0x7fcf380d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35e00640 1 -- 172.21.15.114:0/1072731328 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf380d2350 con 0x7fcf3815e0d0 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf36800640 1 --2- 172.21.15.114:0/1072731328 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf380d0df0 0x7fcf38160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:32.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.108+0000 7fcf35400640 1 --2- 172.21.15.114:0/1072731328 >> v2:172.21.15.177:3300/0 conn(0x7fcf3815f470 0x7fcf380d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.110+0000 7fcf35e00640 1 --2- 172.21.15.114:0/1072731328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815e0d0 0x7fcf380d1a90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fcf380c77f0 tx=0x7fcf2802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4d15b8606985c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.110+0000 7fcf34a00640 1 -- 172.21.15.114:0/1072731328 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28034070 con 0x7fcf3815e0d0 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.110+0000 7fcf34a00640 1 -- 172.21.15.114:0/1072731328 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28030850 con 0x7fcf3815e0d0 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.110+0000 7fcf34a00640 1 -- 172.21.15.114:0/1072731328 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28030c20 con 0x7fcf3815e0d0 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.110+0000 7fcf3ca00640 1 -- 172.21.15.114:0/1072731328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815e0d0 msgr2=0x7fcf380d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.110+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/1072731328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815e0d0 0x7fcf380d1a90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fcf380c77f0 tx=0x7fcf2802b990 comp rx=0 tx=0).stop 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.111+0000 7fcf3ca00640 1 -- 172.21.15.114:0/1072731328 shutdown_connections 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.111+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/1072731328 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf380d0df0 0x7fcf38160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.111+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/1072731328 >> v2:172.21.15.177:3300/0 conn(0x7fcf3815f470 0x7fcf380d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.111+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/1072731328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815e0d0 0x7fcf380d1a90 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.111+0000 7fcf3ca00640 1 -- 172.21.15.114:0/1072731328 >> 172.21.15.114:0/1072731328 conn(0x7fcf380d3ec0 msgr2=0x7fcf380d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.111+0000 7fcf3ca00640 1 -- 172.21.15.114:0/1072731328 shutdown_connections 2024-03-06T07:36:32.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.111+0000 7fcf3ca00640 1 -- 172.21.15.114:0/1072731328 wait complete. 2024-03-06T07:36:32.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.112+0000 7fcf3ca00640 1 Processor -- start 2024-03-06T07:36:32.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.113+0000 7fcf3ca00640 1 -- start start 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.113+0000 7fcf3ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf380d0df0 0x7fcf380c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.113+0000 7fcf3ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf3815e0d0 0x7fcf380cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.113+0000 7fcf3ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815f470 0x7fcf38155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.113+0000 7fcf3ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380d2bf0 con 0x7fcf3815f470 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.113+0000 7fcf3ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380d2910 con 0x7fcf380d0df0 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.113+0000 7fcf3ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3815fb30 con 0x7fcf3815e0d0 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815f470 0x7fcf38155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815f470 0x7fcf38155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48138/0 (socket says 172.21.15.114:48138) 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 -- 172.21.15.114:0/452979958 learned_addr learned my addr 172.21.15.114:0/452979958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf35e00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf380d0df0 0x7fcf380c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf35400640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf3815e0d0 0x7fcf380cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 -- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf3815e0d0 msgr2=0x7fcf380cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf3815e0d0 0x7fcf380cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 -- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf380d0df0 msgr2=0x7fcf380c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf380d0df0 0x7fcf380c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.114+0000 7fcf36800640 1 -- 172.21.15.114:0/452979958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38155cd0 con 0x7fcf3815f470 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.115+0000 7fcf36800640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815f470 0x7fcf38155790 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000c7c0 tx=0x7fcf3000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.115+0000 7fcf2e000640 1 -- 172.21.15.114:0/452979958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30015020 con 0x7fcf3815f470 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.115+0000 7fcf2e000640 1 -- 172.21.15.114:0/452979958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf30004230 con 0x7fcf3815f470 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.115+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf380c8660 con 0x7fcf3815f470 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.115+0000 7fcf2e000640 1 -- 172.21.15.114:0/452979958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30007620 con 0x7fcf3815f470 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.115+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf380c8c20 con 0x7fcf3815f470 2024-03-06T07:36:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.116+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3804f330 con 0x7fcf3815f470 2024-03-06T07:36:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.118+0000 7fcf2e000640 1 -- 172.21.15.114:0/452979958 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf30018020 con 0x7fcf3815f470 2024-03-06T07:36:32.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.119+0000 7fcf2e000640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.119+0000 7fcf2e000640 1 -- 172.21.15.114:0/452979958 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf3008a2f0 con 0x7fcf3815f470 2024-03-06T07:36:32.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.119+0000 7fcf35e00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.120+0000 7fcf35e00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf1406b4d0 0x7fcf1406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fcf28036000 tx=0x7fcf2803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.124+0000 7fcf2e000640 1 -- 172.21.15.114:0/452979958 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf3001f020 con 0x7fcf3815f470 2024-03-06T07:36:32.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.376+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf3808d920 con 0x7fcf3815f470 2024-03-06T07:36:32.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.377+0000 7fcf2e000640 1 -- 172.21.15.114:0/452979958 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fcf30059220 con 0x7fcf3815f470 2024-03-06T07:36:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:32.380 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf1406b4d0 msgr2=0x7fcf1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf1406b4d0 0x7fcf1406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fcf28036000 tx=0x7fcf2803b040 comp rx=0 tx=0).stop 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815f470 msgr2=0x7fcf38155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815f470 0x7fcf38155790 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000c7c0 tx=0x7fcf3000cc80 comp rx=0 tx=0).stop 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 shutdown_connections 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf3815f470 0x7fcf38155790 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fcf3815e0d0 0x7fcf380cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 --2- 172.21.15.114:0/452979958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf380d0df0 0x7fcf380c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 >> 172.21.15.114:0/452979958 conn(0x7fcf380d3ec0 msgr2=0x7fcf380d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.380+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 shutdown_connections 2024-03-06T07:36:32.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:32.381+0000 7fcf3ca00640 1 -- 172.21.15.114:0/452979958 wait complete. 2024-03-06T07:36:32.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:32.603 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:33.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:33.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.817+0000 7f5fe0e00640 1 Processor -- start 2024-03-06T07:36:33.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.817+0000 7f5fe0e00640 1 -- start start 2024-03-06T07:36:33.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.818+0000 7f5fe0e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5fdc144c00 0x7f5fdc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.818+0000 7f5fe0e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15fe60 0x7f5fdc0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fe0e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc06ee70 con 0x7f5fdc15f9d0 2024-03-06T07:36:33.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fe0e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc06cef0 con 0x7f5fdc144c00 2024-03-06T07:36:33.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fe0e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc06d1d0 con 0x7f5fdc15fe60 2024-03-06T07:36:33.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fd1400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5fdc15f9d0 0x7f5fdc0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39960/0 (socket says 172.21.15.114:39960) 2024-03-06T07:36:33.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fd1400640 1 -- 172.21.15.114:0/904896312 learned_addr learned my addr 172.21.15.114:0/904896312 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:33.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fda800640 1 --2- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15fe60 0x7f5fdc0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fd9e00640 1 --2- 172.21.15.114:0/904896312 >> v2:172.21.15.177:3300/0 conn(0x7f5fdc144c00 0x7f5fdc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.819+0000 7f5fd9400640 1 -- 172.21.15.114:0/904896312 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444219716 0 0) 0x7f5fcc001110 con 0x7f5fdc15f9d0 2024-03-06T07:36:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.820+0000 7f5fd9400640 1 -- 172.21.15.114:0/904896312 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fbc003520 con 0x7f5fdc15f9d0 2024-03-06T07:36:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.820+0000 7f5fd9400640 1 -- 172.21.15.114:0/904896312 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361392026 0 0) 0x7f5fbc003520 con 0x7f5fdc15f9d0 2024-03-06T07:36:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.820+0000 7f5fd9e00640 1 -- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15fe60 msgr2=0x7f5fdc0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.820+0000 7f5fd9e00640 1 --2- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15fe60 0x7f5fdc0ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.820+0000 7f5fd9e00640 1 -- 172.21.15.114:0/904896312 >> v1:172.21.15.114:6789/0 conn(0x7f5fdc15f9d0 legacy=0x7f5fdc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.820+0000 7f5fd9e00640 1 -- 172.21.15.114:0/904896312 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc06d4b0 con 0x7f5fdc144c00 2024-03-06T07:36:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.821+0000 7f5fd9e00640 1 --2- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc144c00 0x7f5fdc06e930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40097a0 tx=0x7f5fc402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b10a537b75d82bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.822+0000 7f5fd9400640 1 -- 172.21.15.114:0/904896312 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4037070 con 0x7f5fdc144c00 2024-03-06T07:36:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.822+0000 7f5fd9400640 1 -- 172.21.15.114:0/904896312 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4030510 con 0x7f5fdc144c00 2024-03-06T07:36:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.822+0000 7f5fd9400640 1 -- 172.21.15.114:0/904896312 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc001840 con 0x7f5fdc144c00 2024-03-06T07:36:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 -- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc144c00 msgr2=0x7f5fdc06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc144c00 0x7f5fdc06e930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40097a0 tx=0x7f5fc402b790 comp rx=0 tx=0).stop 2024-03-06T07:36:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 -- 172.21.15.114:0/904896312 shutdown_connections 2024-03-06T07:36:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15fe60 0x7f5fdc0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/904896312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc144c00 0x7f5fdc06e930 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 -- 172.21.15.114:0/904896312 >> 172.21.15.114:0/904896312 conn(0x7f5fdc0b1730 msgr2=0x7f5fdc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 -- 172.21.15.114:0/904896312 shutdown_connections 2024-03-06T07:36:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.823+0000 7f5fe0e00640 1 -- 172.21.15.114:0/904896312 wait complete. 2024-03-06T07:36:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.824+0000 7f5fe0e00640 1 Processor -- start 2024-03-06T07:36:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.825+0000 7f5fe0e00640 1 -- start start 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.825+0000 7f5fe0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fdc144c00 0x7f5fdc14fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.825+0000 7f5fe0e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15f9d0 0x7f5fdc1394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.825+0000 7f5fe0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc15fe60 0x7f5fdc13bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.825+0000 7f5fe0e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc160730 con 0x7f5fdc144c00 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.825+0000 7f5fe0e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc160290 con 0x7f5fdc15fe60 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.825+0000 7f5fe0e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc06cb50 con 0x7f5fdc15f9d0 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fdc144c00 0x7f5fdc14fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fdc144c00 0x7f5fdc14fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48148/0 (socket says 172.21.15.114:48148) 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 -- 172.21.15.114:0/3969033781 learned_addr learned my addr 172.21.15.114:0/3969033781 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd1400640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15f9d0 0x7f5fdc1394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fda800640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc15fe60 0x7f5fdc13bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 -- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15f9d0 msgr2=0x7f5fdc1394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15f9d0 0x7f5fdc1394e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 -- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc15fe60 msgr2=0x7f5fdc13bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc15fe60 0x7f5fdc13bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.826+0000 7f5fd9e00640 1 -- 172.21.15.114:0/3969033781 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc141a80 con 0x7f5fdc144c00 2024-03-06T07:36:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.827+0000 7f5fd9e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fdc144c00 0x7f5fdc14fac0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f5fc402b7d0 tx=0x7f5fc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:33.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.827+0000 7f5fd2a00640 1 -- 172.21.15.114:0/3969033781 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4040020 con 0x7f5fdc144c00 2024-03-06T07:36:33.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.827+0000 7f5fd2a00640 1 -- 172.21.15.114:0/3969033781 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4038040 con 0x7f5fdc144c00 2024-03-06T07:36:33.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.827+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc141d60 con 0x7f5fdc144c00 2024-03-06T07:36:33.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.827+0000 7f5fd2a00640 1 -- 172.21.15.114:0/3969033781 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4037070 con 0x7f5fdc144c00 2024-03-06T07:36:33.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.827+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc150000 con 0x7f5fdc144c00 2024-03-06T07:36:33.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.828+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc04f330 con 0x7f5fdc144c00 2024-03-06T07:36:33.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.829+0000 7f5fd2a00640 1 -- 172.21.15.114:0/3969033781 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fc4043020 con 0x7f5fdc144c00 2024-03-06T07:36:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.830+0000 7f5fd2a00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5fb806b500 0x7f5fb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.830+0000 7f5fd2a00640 1 -- 172.21.15.114:0/3969033781 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fc40ab6d0 con 0x7f5fdc144c00 2024-03-06T07:36:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.830+0000 7f5fd1400640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5fb806b500 0x7f5fb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.831+0000 7f5fd1400640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5fb806b500 0x7f5fb806d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc1e6ac0 tx=0x7f5fcc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:33.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:33.834+0000 7f5fd2a00640 1 -- 172.21.15.114:0/3969033781 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fc404a030 con 0x7f5fdc144c00 2024-03-06T07:36:34.014 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.073+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fdc072890 con 0x7f5fdc144c00 2024-03-06T07:36:34.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.074+0000 7f5fd2a00640 1 -- 172.21.15.114:0/3969033781 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f5fc407ee20 con 0x7f5fdc144c00 2024-03-06T07:36:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:34.076 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:34.076 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5fb806b500 msgr2=0x7f5fb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5fb806b500 0x7f5fb806d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc1e6ac0 tx=0x7f5fcc009040 comp rx=0 tx=0).stop 2024-03-06T07:36:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fdc144c00 msgr2=0x7f5fdc14fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fdc144c00 0x7f5fdc14fac0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f5fc402b7d0 tx=0x7f5fc40046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 shutdown_connections 2024-03-06T07:36:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fdc15fe60 0x7f5fdc13bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5fdc15f9d0 0x7f5fdc1394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5fb806b500 0x7f5fb806d9c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 --2- 172.21.15.114:0/3969033781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fdc144c00 0x7f5fdc14fac0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 >> 172.21.15.114:0/3969033781 conn(0x7f5fdc0b1730 msgr2=0x7f5fdc13d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.078+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 shutdown_connections 2024-03-06T07:36:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:34.079+0000 7f5fe0e00640 1 -- 172.21.15.114:0/3969033781 wait complete. 2024-03-06T07:36:34.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:34.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:35.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:35.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.515+0000 7fb602a00640 1 Processor -- start 2024-03-06T07:36:35.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.515+0000 7fb602a00640 1 -- start start 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.517+0000 7fb602a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb5fc144c00 0x7fb5fc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.517+0000 7fb602a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb5fc15fe60 0x7fb5fc0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.517+0000 7fb602a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc06ee70 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.517+0000 7fb602a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5fc06cef0 con 0x7fb5fc15fe60 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.517+0000 7fb602a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5fc06d1d0 con 0x7fb5fc144c00 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.517+0000 7fb600a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb5fc15fe60 0x7fb5fc0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.517+0000 7fb5fb400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb5fc15f9d0 0x7fb5fc0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39976/0 (socket says 172.21.15.114:39976) 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.518+0000 7fb5fb400640 1 -- 172.21.15.114:0/569617113 learned_addr learned my addr 172.21.15.114:0/569617113 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.518+0000 7fb5fbe00640 1 --2- 172.21.15.114:0/569617113 >> v2:172.21.15.177:3300/0 conn(0x7fb5fc144c00 0x7fb5fc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.518+0000 7fb600a00640 1 -- 172.21.15.114:0/569617113 >> v2:172.21.15.177:3300/0 conn(0x7fb5fc144c00 msgr2=0x7fb5fc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.518+0000 7fb600a00640 1 --2- 172.21.15.114:0/569617113 >> v2:172.21.15.177:3300/0 conn(0x7fb5fc144c00 0x7fb5fc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.518+0000 7fb600a00640 1 -- 172.21.15.114:0/569617113 >> v1:172.21.15.114:6789/0 conn(0x7fb5fc15f9d0 legacy=0x7fb5fc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.518+0000 7fb600a00640 1 -- 172.21.15.114:0/569617113 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc06d4b0 con 0x7fb5fc15fe60 2024-03-06T07:36:35.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.518+0000 7fb5faa00640 1 -- 172.21.15.114:0/569617113 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767673198 0 0) 0x7fb5e8001110 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.519+0000 7fb600a00640 1 --2- 172.21.15.114:0/569617113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15fe60 0x7fb5fc0ce7b0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0b25f0 tx=0x7fb5f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dc1d8f76c45259 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.519+0000 7fb5faa00640 1 -- 172.21.15.114:0/569617113 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4035020 con 0x7fb5fc15fe60 2024-03-06T07:36:35.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.519+0000 7fb5faa00640 1 -- 172.21.15.114:0/569617113 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f4031830 con 0x7fb5fc15fe60 2024-03-06T07:36:35.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.519+0000 7fb5faa00640 1 -- 172.21.15.114:0/569617113 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4031c00 con 0x7fb5fc15fe60 2024-03-06T07:36:35.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 -- 172.21.15.114:0/569617113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15fe60 msgr2=0x7fb5fc0ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 --2- 172.21.15.114:0/569617113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15fe60 0x7fb5fc0ce7b0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0b25f0 tx=0x7fb5f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 -- 172.21.15.114:0/569617113 shutdown_connections 2024-03-06T07:36:35.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 --2- 172.21.15.114:0/569617113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15fe60 0x7fb5fc0ce7b0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 --2- 172.21.15.114:0/569617113 >> v2:172.21.15.177:3300/0 conn(0x7fb5fc144c00 0x7fb5fc06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 -- 172.21.15.114:0/569617113 >> 172.21.15.114:0/569617113 conn(0x7fb5fc0b1730 msgr2=0x7fb5fc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:35.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 -- 172.21.15.114:0/569617113 shutdown_connections 2024-03-06T07:36:35.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.520+0000 7fb602a00640 1 -- 172.21.15.114:0/569617113 wait complete. 2024-03-06T07:36:35.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.523+0000 7fb602a00640 1 Processor -- start 2024-03-06T07:36:35.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.523+0000 7fb602a00640 1 -- start start 2024-03-06T07:36:35.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb602a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5fc144c00 0x7fb5fc15e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb602a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15f9d0 0x7fb5fc15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb602a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5fc15fe60 0x7fb5fc13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb602a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc160730 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb602a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc160270 con 0x7fb5fc144c00 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb602a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc06cb50 con 0x7fb5fc15fe60 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb5fb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15f9d0 0x7fb5fc15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb5fb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15f9d0 0x7fb5fc15c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48166/0 (socket says 172.21.15.114:48166) 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb5fb400640 1 -- 172.21.15.114:0/3618421912 learned_addr learned my addr 172.21.15.114:0/3618421912 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb5fbe00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5fc144c00 0x7fb5fc15e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.525+0000 7fb600a00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5fc15fe60 0x7fb5fc13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5fb400640 1 -- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5fc15fe60 msgr2=0x7fb5fc13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5fb400640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5fc15fe60 0x7fb5fc13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5fb400640 1 -- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5fc144c00 msgr2=0x7fb5fc15e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5fb400640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5fc144c00 0x7fb5fc15e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5fb400640 1 -- 172.21.15.114:0/3618421912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc15cc20 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5fb400640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15f9d0 0x7fb5fc15c340 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c7b0 tx=0x7fb5e800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5f3e00640 1 -- 172.21.15.114:0/3618421912 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8018070 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5f3e00640 1 -- 172.21.15.114:0/3618421912 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e80072d0 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb5f3e00640 1 -- 172.21.15.114:0/3618421912 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e80047e0 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.526+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc15cf00 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.527+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc15d4c0 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.528+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc04f330 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.528+0000 7fb5f3e00640 1 -- 172.21.15.114:0/3618421912 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5e8004980 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.529+0000 7fb5f3e00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:35.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.529+0000 7fb5f3e00640 1 -- 172.21.15.114:0/3618421912 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5e8089530 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.530+0000 7fb5fbe00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:35.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.530+0000 7fb5fbe00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0099c0 tx=0x7fb5ec003990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:35.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.534+0000 7fb5f3e00640 1 -- 172.21.15.114:0/3618421912 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5fc06ee70 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.785+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5fc1438e0 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.786+0000 7fb5f3e00640 1 -- 172.21.15.114:0/3618421912 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fb5e80583a0 con 0x7fb5fc15f9d0 2024-03-06T07:36:35.788 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:35.788 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:35.789 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb5dc06b4d0 msgr2=0x7fb5dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0099c0 tx=0x7fb5ec003990 comp rx=0 tx=0).stop 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15f9d0 msgr2=0x7fb5fc15c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15f9d0 0x7fb5fc15c340 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c7b0 tx=0x7fb5e800cc70 comp rx=0 tx=0).stop 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 shutdown_connections 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb5fc15fe60 0x7fb5fc13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5fc15f9d0 0x7fb5fc15c340 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 --2- 172.21.15.114:0/3618421912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5fc144c00 0x7fb5fc15e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 >> 172.21.15.114:0/3618421912 conn(0x7fb5fc0b1730 msgr2=0x7fb5fc0719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:35.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 shutdown_connections 2024-03-06T07:36:35.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:35.790+0000 7fb602a00640 1 -- 172.21.15.114:0/3618421912 wait complete. 2024-03-06T07:36:36.012 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:36.013 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:37.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:37.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.220+0000 7f6b10600640 1 Processor -- start 2024-03-06T07:36:37.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.220+0000 7f6b10600640 1 -- start start 2024-03-06T07:36:37.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.223+0000 7f6b10600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6b080b4680 0x7f6b0806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.223+0000 7f6b10600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0815b090 con 0x7f6b0815acb0 2024-03-06T07:36:37.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.223+0000 7f6b10600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080b2a30 con 0x7f6b08072340 2024-03-06T07:36:37.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.223+0000 7f6b10600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b080b4360 con 0x7f6b080b4680 2024-03-06T07:36:37.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.223+0000 7f6b0dc00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6b0815acb0 0x7f6b080b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39984/0 (socket says 172.21.15.114:39984) 2024-03-06T07:36:37.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.223+0000 7f6b0dc00640 1 -- 172.21.15.114:0/378341210 learned_addr learned my addr 172.21.15.114:0/378341210 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:37.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438387729 0 0) 0x7f6b0815b090 con 0x7f6b0815acb0 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aec003520 con 0x7f6b0815acb0 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b0d200640 1 --2- 172.21.15.114:0/378341210 >> v2:172.21.15.177:3300/0 conn(0x7f6b080b4680 0x7f6b0806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409265571 0 0) 0x7f6aec003520 con 0x7f6b0815acb0 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 >> v2:172.21.15.177:3300/0 conn(0x7f6b080b4680 msgr2=0x7f6b0806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 --2- 172.21.15.114:0/378341210 >> v2:172.21.15.177:3300/0 conn(0x7f6b080b4680 0x7f6b0806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 >> v1:172.21.15.177:6789/0 conn(0x7f6b08072340 legacy=0x7f6b080ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080b2cf0 con 0x7f6b0815acb0 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6af8001670 con 0x7f6b0815acb0 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 >> v1:172.21.15.114:6789/0 conn(0x7f6b0815acb0 legacy=0x7f6b080b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec003a00 0x7f6aec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec004800 0x7f6aec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aec005210 0x7f6aec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080b4360 con 0x7f6aec004800 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec00c140 con 0x7f6aec003a00 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec00c340 con 0x7f6aec005210 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.224+0000 7f6b0d200640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec004800 0x7f6aec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0dc00640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec003a00 0x7f6aec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0d200640 1 -- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aec005210 msgr2=0x7f6aec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0d200640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6aec005210 0x7f6aec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0d200640 1 -- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec003a00 msgr2=0x7f6aec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0d200640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec003a00 0x7f6aec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0d200640 1 -- 172.21.15.114:0/378341210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec003520 con 0x7f6aec004800 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0dc00640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec003a00 0x7f6aec003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.225+0000 7f6b0e600640 1 -- 172.21.15.114:0/378341210 reap_dead start 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.226+0000 7f6b0d200640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec004800 0x7f6aec004c80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f6af40073b0 tx=0x7f6aec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2d316ba55dad7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.226+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4038020 con 0x7f6aec004800 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.226+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af4034600 con 0x7f6aec004800 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.226+0000 7f6b07e00640 1 -- 172.21.15.114:0/378341210 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4009c70 con 0x7f6aec004800 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.227+0000 7f6b10600640 1 -- 172.21.15.114:0/378341210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec004800 msgr2=0x7f6aec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.227+0000 7f6b10600640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec004800 0x7f6aec004c80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f6af40073b0 tx=0x7f6aec005830 comp rx=0 tx=0).stop 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.227+0000 7f6b10600640 1 -- 172.21.15.114:0/378341210 shutdown_connections 2024-03-06T07:36:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.227+0000 7f6b10600640 1 --2- 172.21.15.114:0/378341210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6aec004800 0x7f6aec004c80 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.227+0000 7f6b10600640 1 -- 172.21.15.114:0/378341210 >> 172.21.15.114:0/378341210 conn(0x7f6b080b1730 msgr2=0x7f6b080b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.227+0000 7f6b10600640 1 -- 172.21.15.114:0/378341210 shutdown_connections 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.227+0000 7f6b10600640 1 -- 172.21.15.114:0/378341210 wait complete. 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.230+0000 7f6b10600640 1 Processor -- start 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.230+0000 7f6b10600640 1 -- start start 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b10600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec004800 0x7f6b08157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b10600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b08158e50 0x7f6b0815cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b10600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6b08144290 0x7f6b08144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b10600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080b4010 con 0x7f6b08158e50 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b10600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080b32b0 con 0x7f6aec004800 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b10600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080b37d0 con 0x7f6b08144290 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b08158e50 0x7f6b0815cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b08158e50 0x7f6b0815cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48188/0 (socket says 172.21.15.114:48188) 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 -- 172.21.15.114:0/222655658 learned_addr learned my addr 172.21.15.114:0/222655658 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:37.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0dc00640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec004800 0x7f6b08157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 -- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6b08144290 msgr2=0x7f6b08144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0e600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6b08144290 0x7f6b08144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6b08144290 0x7f6b08144710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 -- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec004800 msgr2=0x7f6b08157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec004800 0x7f6b08157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.232+0000 7f6b0d200640 1 -- 172.21.15.114:0/222655658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b081595c0 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.233+0000 7f6b0d200640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b08158e50 0x7f6b0815cc30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f6af40074d0 tx=0x7f6af4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.233+0000 7f6b05600640 1 -- 172.21.15.114:0/222655658 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4038040 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.233+0000 7f6b05600640 1 -- 172.21.15.114:0/222655658 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af4003990 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.233+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b081598a0 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.233+0000 7f6b05600640 1 -- 172.21.15.114:0/222655658 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4009c70 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.233+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b081592b0 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.234+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0804f330 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.235+0000 7f6b05600640 1 -- 172.21.15.114:0/222655658 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6af403b020 con 0x7f6b08158e50 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.236+0000 7f6b05600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ae806b4d0 0x7f6ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.236+0000 7f6b05600640 1 -- 172.21.15.114:0/222655658 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6af40ace70 con 0x7f6b08158e50 2024-03-06T07:36:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.237+0000 7f6b0dc00640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ae806b4d0 0x7f6ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.237+0000 7f6b0dc00640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ae806b4d0 0x7f6ae806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6af8000fd0 tx=0x7f6af8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.241+0000 7f6b05600640 1 -- 172.21.15.114:0/222655658 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6af407bbc0 con 0x7f6b08158e50 2024-03-06T07:36:37.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.478+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b081e4620 con 0x7f6b08158e50 2024-03-06T07:36:37.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.479+0000 7f6b05600640 1 -- 172.21.15.114:0/222655658 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f6af407b4c0 con 0x7f6b08158e50 2024-03-06T07:36:37.481 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:37.481 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ae806b4d0 msgr2=0x7f6ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ae806b4d0 0x7f6ae806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6af8000fd0 tx=0x7f6af8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b08158e50 msgr2=0x7f6b0815cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b08158e50 0x7f6b0815cc30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f6af40074d0 tx=0x7f6af4004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 shutdown_connections 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6b08144290 0x7f6b08144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6ae806b4d0 0x7f6ae806d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b08158e50 0x7f6b0815cc30 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 --2- 172.21.15.114:0/222655658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6aec004800 0x7f6b08157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 >> 172.21.15.114:0/222655658 conn(0x7f6b080b1730 msgr2=0x7f6b080cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 shutdown_connections 2024-03-06T07:36:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:37.482+0000 7f6b10600640 1 -- 172.21.15.114:0/222655658 wait complete. 2024-03-06T07:36:37.705 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:37.705 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:38.019 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:38.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:38.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.978+0000 7f2aecc00640 1 Processor -- start 2024-03-06T07:36:38.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.978+0000 7f2aecc00640 1 -- start start 2024-03-06T07:36:38.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2aecc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2ae806d4c0 0x7f2ae806dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2aecc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae806d900 con 0x7f2ae806e2c0 2024-03-06T07:36:38.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2aecc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae80af030 con 0x7f2ae815a330 2024-03-06T07:36:38.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2aecc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae80af310 con 0x7f2ae806d4c0 2024-03-06T07:36:38.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2ae7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2ae806e2c0 0x7f2ae80709b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39302/0 (socket says 172.21.15.114:39302) 2024-03-06T07:36:38.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2ae7e00640 1 -- 172.21.15.114:0/2460543957 learned_addr learned my addr 172.21.15.114:0/2460543957 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2adfe00640 1 --2- 172.21.15.114:0/2460543957 >> v2:172.21.15.177:3300/0 conn(0x7f2ae806d4c0 0x7f2ae806dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78245544 0 0) 0x7f2ae0001490 con 0x7f2ae806e2c0 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.980+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2acc003520 con 0x7f2ae806e2c0 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692839576 0 0) 0x7f2acc003520 con 0x7f2ae806e2c0 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 >> v2:172.21.15.177:3300/0 conn(0x7f2ae806d4c0 msgr2=0x7f2ae806dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 --2- 172.21.15.114:0/2460543957 >> v2:172.21.15.177:3300/0 conn(0x7f2ae806d4c0 0x7f2ae806dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 >> v1:172.21.15.177:6789/0 conn(0x7f2ae815a330 legacy=0x7f2ae80ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae806c9e0 con 0x7f2ae806e2c0 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f2ae0001bc0 con 0x7f2ae806e2c0 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 >> v1:172.21.15.114:6789/0 conn(0x7f2ae806e2c0 legacy=0x7f2ae80709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc003a00 0x7f2acc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2acc004800 0x7f2acc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc0051c0 0x7f2acc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae80af310 con 0x7f2acc003a00 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc00e2b0 con 0x7f2acc0051c0 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.981+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc00e4b0 con 0x7f2acc004800 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7400640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc003a00 0x7f2acc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2adfe00640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2acc004800 0x7f2acc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7e00640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc0051c0 0x7f2acc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7400640 1 -- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2acc004800 msgr2=0x7f2acc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7400640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2acc004800 0x7f2acc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7400640 1 -- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc0051c0 msgr2=0x7f2acc00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:38.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7400640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc0051c0 0x7f2acc00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7400640 1 -- 172.21.15.114:0/2460543957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc003520 con 0x7f2acc003a00 2024-03-06T07:36:38.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.982+0000 7f2ae7e00640 1 -- 172.21.15.114:0/2460543957 reap_dead start 2024-03-06T07:36:38.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.983+0000 7f2ae7400640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc003a00 0x7f2acc003e20 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8001750 tx=0x7f2acc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e5f437c07407f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.983+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8001370 con 0x7f2acc003a00 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.983+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad80052d0 con 0x7f2acc003a00 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.983+0000 7f2ae6a00640 1 -- 172.21.15.114:0/2460543957 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8003830 con 0x7f2acc003a00 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.984+0000 7f2aecc00640 1 -- 172.21.15.114:0/2460543957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc003a00 msgr2=0x7f2acc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.984+0000 7f2aecc00640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc003a00 0x7f2acc003e20 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8001750 tx=0x7f2acc005830 comp rx=0 tx=0).stop 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.984+0000 7f2aecc00640 1 -- 172.21.15.114:0/2460543957 shutdown_connections 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.984+0000 7f2aecc00640 1 --2- 172.21.15.114:0/2460543957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc003a00 0x7f2acc003e20 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.984+0000 7f2aecc00640 1 -- 172.21.15.114:0/2460543957 >> 172.21.15.114:0/2460543957 conn(0x7f2ae806afc0 msgr2=0x7f2ae806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.984+0000 7f2aecc00640 1 -- 172.21.15.114:0/2460543957 shutdown_connections 2024-03-06T07:36:38.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.984+0000 7f2aecc00640 1 -- 172.21.15.114:0/2460543957 wait complete. 2024-03-06T07:36:38.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.986+0000 7f2aecc00640 1 Processor -- start 2024-03-06T07:36:38.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.986+0000 7f2aecc00640 1 -- start start 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2aecc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc003a00 0x7f2ae81f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2aecc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2ae814e770 0x7f2ae814df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2aecc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ae8158300 0x7f2ae8158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2aecc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae8072a20 con 0x7f2ae8158300 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2aecc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae806cfa0 con 0x7f2acc003a00 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2aecc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae815a710 con 0x7f2ae814e770 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2ae7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ae8158300 0x7f2ae8158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2ae7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ae8158300 0x7f2ae8158780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49528/0 (socket says 172.21.15.114:49528) 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.987+0000 7f2ae7e00640 1 -- 172.21.15.114:0/4016642794 learned_addr learned my addr 172.21.15.114:0/4016642794 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2ae7400640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc003a00 0x7f2ae81f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2adfe00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2ae814e770 0x7f2ae814df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2ae7e00640 1 -- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2ae814e770 msgr2=0x7f2ae814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2ae7e00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2ae814e770 0x7f2ae814df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2ae7e00640 1 -- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc003a00 msgr2=0x7f2ae81f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2ae7e00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc003a00 0x7f2ae81f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2ae7e00640 1 -- 172.21.15.114:0/4016642794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae814ee50 con 0x7f2ae8158300 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2adfe00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2ae814e770 0x7f2ae814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2ae7e00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ae8158300 0x7f2ae8158780 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f2ae815de40 tx=0x7f2ae0007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2adf400640 1 -- 172.21.15.114:0/4016642794 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae000eaa0 con 0x7f2ae8158300 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2adf400640 1 -- 172.21.15.114:0/4016642794 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae0018070 con 0x7f2ae8158300 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2adf400640 1 -- 172.21.15.114:0/4016642794 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae000ee80 con 0x7f2ae8158300 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.988+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae814f130 con 0x7f2ae8158300 2024-03-06T07:36:38.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.989+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae814f6f0 con 0x7f2ae8158300 2024-03-06T07:36:38.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.990+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae804f330 con 0x7f2ae8158300 2024-03-06T07:36:38.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.991+0000 7f2adf400640 1 -- 172.21.15.114:0/4016642794 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ae001d460 con 0x7f2ae8158300 2024-03-06T07:36:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.991+0000 7f2adf400640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ac406b4d0 0x7f2ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.991+0000 7f2adf400640 1 -- 172.21.15.114:0/4016642794 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ae008a500 con 0x7f2ae8158300 2024-03-06T07:36:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.992+0000 7f2ae7400640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ac406b4d0 0x7f2ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:38.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.992+0000 7f2ae7400640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ac406b4d0 0x7f2ac406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8033000 tx=0x7f2ad8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:38.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:38.996+0000 7f2adf400640 1 -- 172.21.15.114:0/4016642794 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ae806d900 con 0x7f2ae8158300 2024-03-06T07:36:39.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.238+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ae80ae6b0 con 0x7f2ae8158300 2024-03-06T07:36:39.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.239+0000 7f2adf400640 1 -- 172.21.15.114:0/4016642794 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f2ae0059330 con 0x7f2ae8158300 2024-03-06T07:36:39.241 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:39.241 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:39.241 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ac406b4d0 msgr2=0x7f2ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ac406b4d0 0x7f2ac406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8033000 tx=0x7f2ad8009030 comp rx=0 tx=0).stop 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ae8158300 msgr2=0x7f2ae8158780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ae8158300 0x7f2ae8158780 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f2ae815de40 tx=0x7f2ae0007bc0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 shutdown_connections 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2ac406b4d0 0x7f2ac406d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ae8158300 0x7f2ae8158780 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2ae814e770 0x7f2ae814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 --2- 172.21.15.114:0/4016642794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2acc003a00 0x7f2ae81f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 >> 172.21.15.114:0/4016642794 conn(0x7f2ae806afc0 msgr2=0x7f2ae80ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 shutdown_connections 2024-03-06T07:36:39.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:39.241+0000 7f2aecc00640 1 -- 172.21.15.114:0/4016642794 wait complete. 2024-03-06T07:36:39.462 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:39.463 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:40.021 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:40.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:40.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.675+0000 7f0711000640 1 Processor -- start 2024-03-06T07:36:40.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.675+0000 7f0711000640 1 -- start start 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.676+0000 7f0711000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f070c1583d0 0x7f070c072820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.676+0000 7f0711000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c161a80 0x7f070c161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.676+0000 7f0711000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c072d60 con 0x7f070c06afc0 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.676+0000 7f0711000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f070c071a80 con 0x7f070c1583d0 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.676+0000 7f0711000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070c1627d0 con 0x7f070c161a80 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070b400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f070c1583d0 0x7f070c072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070aa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f070c06afc0 0x7f070c06b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39310/0 (socket says 172.21.15.114:39310) 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070aa00640 1 -- 172.21.15.114:0/1375721219 learned_addr learned my addr 172.21.15.114:0/1375721219 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070be00640 1 --2- 172.21.15.114:0/1375721219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c161a80 0x7f070c161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070b400640 1 -- 172.21.15.114:0/1375721219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c161a80 msgr2=0x7f070c161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070b400640 1 --2- 172.21.15.114:0/1375721219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c161a80 0x7f070c161f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070b400640 1 -- 172.21.15.114:0/1375721219 >> v1:172.21.15.114:6789/0 conn(0x7f070c06afc0 legacy=0x7f070c06b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.677+0000 7f070b400640 1 -- 172.21.15.114:0/1375721219 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c071d40 con 0x7f070c1583d0 2024-03-06T07:36:40.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.678+0000 7f070b400640 1 --2- 172.21.15.114:0/1375721219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c1583d0 0x7f070c072820 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0097a0 tx=0x7f06fc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6197dcd36be77a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f070a000640 1 -- 172.21.15.114:0/1375721219 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc035020 con 0x7f070c1583d0 2024-03-06T07:36:40.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f070a000640 1 -- 172.21.15.114:0/1375721219 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc031810 con 0x7f070c1583d0 2024-03-06T07:36:40.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f070a000640 1 -- 172.21.15.114:0/1375721219 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc031be0 con 0x7f070c1583d0 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 -- 172.21.15.114:0/1375721219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c1583d0 msgr2=0x7f070c072820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 --2- 172.21.15.114:0/1375721219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c1583d0 0x7f070c072820 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0097a0 tx=0x7f06fc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 -- 172.21.15.114:0/1375721219 shutdown_connections 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 --2- 172.21.15.114:0/1375721219 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c161a80 0x7f070c161f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 --2- 172.21.15.114:0/1375721219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c1583d0 0x7f070c072820 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 -- 172.21.15.114:0/1375721219 >> 172.21.15.114:0/1375721219 conn(0x7f070c06c0d0 msgr2=0x7f070c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 -- 172.21.15.114:0/1375721219 shutdown_connections 2024-03-06T07:36:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.679+0000 7f0711000640 1 -- 172.21.15.114:0/1375721219 wait complete. 2024-03-06T07:36:40.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.681+0000 7f0711000640 1 Processor -- start 2024-03-06T07:36:40.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.681+0000 7f0711000640 1 -- start start 2024-03-06T07:36:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.682+0000 7f0711000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c06afc0 0x7f070c15e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.682+0000 7f0711000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c1583d0 0x7f070c0adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.682+0000 7f0711000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f070c161a80 0x7f070c15bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.682+0000 7f0711000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c1587b0 con 0x7f070c06afc0 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.682+0000 7f0711000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070c072300 con 0x7f070c161a80 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.682+0000 7f0711000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070c162450 con 0x7f070c1583d0 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.682+0000 7f070b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c06afc0 0x7f070c15e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c06afc0 0x7f070c15e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49556/0 (socket says 172.21.15.114:49556) 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 -- 172.21.15.114:0/2957054170 learned_addr learned my addr 172.21.15.114:0/2957054170 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070aa00640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c1583d0 0x7f070c0adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070be00640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f070c161a80 0x7f070c15bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 -- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c1583d0 msgr2=0x7f070c0adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c1583d0 0x7f070c0adaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 -- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f070c161a80 msgr2=0x7f070c15bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f070c161a80 0x7f070c15bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 -- 172.21.15.114:0/2957054170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c0ae560 con 0x7f070c06afc0 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070aa00640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c1583d0 0x7f070c0adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070be00640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f070c161a80 0x7f070c15bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.683+0000 7f070b400640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c06afc0 0x7f070c15e5f0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f06fc02bec0 tx=0x7f06fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.684+0000 7f0703400640 1 -- 172.21.15.114:0/2957054170 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc041070 con 0x7f070c06afc0 2024-03-06T07:36:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.684+0000 7f0703400640 1 -- 172.21.15.114:0/2957054170 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc03ad00 con 0x7f070c06afc0 2024-03-06T07:36:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.684+0000 7f0703400640 1 -- 172.21.15.114:0/2957054170 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc035040 con 0x7f070c06afc0 2024-03-06T07:36:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.684+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070c0ae840 con 0x7f070c06afc0 2024-03-06T07:36:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.684+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070c0af0e0 con 0x7f070c06afc0 2024-03-06T07:36:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.685+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070c0aea10 con 0x7f070c06afc0 2024-03-06T07:36:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.687+0000 7f0703400640 1 -- 172.21.15.114:0/2957054170 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06fc04c060 con 0x7f070c06afc0 2024-03-06T07:36:40.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.688+0000 7f0703400640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:40.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.688+0000 7f0703400640 1 -- 172.21.15.114:0/2957054170 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f06fc0abf00 con 0x7f070c06afc0 2024-03-06T07:36:40.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.688+0000 7f070aa00640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:40.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.689+0000 7f070aa00640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06ec06b4d0 0x7f06ec06d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f070c1578e0 tx=0x7f06f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:40.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.693+0000 7f0703400640 1 -- 172.21.15.114:0/2957054170 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06fc07aca0 con 0x7f070c06afc0 2024-03-06T07:36:40.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.968+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f070c04f330 con 0x7f070c06afc0 2024-03-06T07:36:40.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.969+0000 7f0703400640 1 -- 172.21.15.114:0/2957054170 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7f06fc07a5a0 con 0x7f070c06afc0 2024-03-06T07:36:40.971 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:40.971 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:40.971 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:40.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06ec06b4d0 msgr2=0x7f06ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06ec06b4d0 0x7f06ec06d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f070c1578e0 tx=0x7f06f8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c06afc0 msgr2=0x7f070c15e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c06afc0 0x7f070c15e5f0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f06fc02bec0 tx=0x7f06fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 shutdown_connections 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f070c161a80 0x7f070c15bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f070c1583d0 0x7f070c0adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f06ec06b4d0 0x7f06ec06d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 --2- 172.21.15.114:0/2957054170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f070c06afc0 0x7f070c15e5f0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 >> 172.21.15.114:0/2957054170 conn(0x7f070c06c0d0 msgr2=0x7f070c160db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 shutdown_connections 2024-03-06T07:36:40.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:40.972+0000 7f0711000640 1 -- 172.21.15.114:0/2957054170 wait complete. 2024-03-06T07:36:41.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:41.193 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:42.023 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:42.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:42.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.405+0000 7fdf70a00640 1 Processor -- start 2024-03-06T07:36:42.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.405+0000 7fdf70a00640 1 -- start start 2024-03-06T07:36:42.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.406+0000 7fdf70a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fdf6c15dfd0 0x7fdf6c162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf70a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf70a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0cef70 con 0x7fdf6c0affe0 2024-03-06T07:36:42.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf70a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c162270 con 0x7fdf6c15dfd0 2024-03-06T07:36:42.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf70a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c15e3b0 con 0x7fdf6c06b6d0 2024-03-06T07:36:42.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf6be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf6be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c0d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:43844/0 (socket says 172.21.15.114:43844) 2024-03-06T07:36:42.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf6be00640 1 -- 172.21.15.114:0/3140671333 learned_addr learned my addr 172.21.15.114:0/3140671333 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:42.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.407+0000 7fdf6b400640 1 --2- 172.21.15.114:0/3140671333 >> v2:172.21.15.177:3300/0 conn(0x7fdf6c15dfd0 0x7fdf6c162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.408+0000 7fdf6be00640 1 -- 172.21.15.114:0/3140671333 >> v2:172.21.15.177:3300/0 conn(0x7fdf6c15dfd0 msgr2=0x7fdf6c162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:42.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.408+0000 7fdf6be00640 1 --2- 172.21.15.114:0/3140671333 >> v2:172.21.15.177:3300/0 conn(0x7fdf6c15dfd0 0x7fdf6c162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.408+0000 7fdf6be00640 1 -- 172.21.15.114:0/3140671333 >> v1:172.21.15.177:6789/0 conn(0x7fdf6c0affe0 legacy=0x7fdf6c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.408+0000 7fdf6aa00640 1 -- 172.21.15.114:0/3140671333 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000397887 0 0) 0x7fdf6c0cef70 con 0x7fdf6c0affe0 2024-03-06T07:36:42.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.408+0000 7fdf6be00640 1 -- 172.21.15.114:0/3140671333 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c161c60 con 0x7fdf6c06b6d0 2024-03-06T07:36:42.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.410+0000 7fdf6be00640 1 --2- 172.21.15.114:0/3140671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c0d06f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009e50 tx=0x7fdf6402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2232e18b6c1ff98e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:42.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf6aa00640 1 -- 172.21.15.114:0/3140671333 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6402c9e0 con 0x7fdf6c06b6d0 2024-03-06T07:36:42.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf6aa00640 1 -- 172.21.15.114:0/3140671333 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6402cb80 con 0x7fdf6c06b6d0 2024-03-06T07:36:42.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf6aa00640 1 -- 172.21.15.114:0/3140671333 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf640343f0 con 0x7fdf6c06b6d0 2024-03-06T07:36:42.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 -- 172.21.15.114:0/3140671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 msgr2=0x7fdf6c0d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 --2- 172.21.15.114:0/3140671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c0d06f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009e50 tx=0x7fdf6402be40 comp rx=0 tx=0).stop 2024-03-06T07:36:42.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 -- 172.21.15.114:0/3140671333 shutdown_connections 2024-03-06T07:36:42.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 --2- 172.21.15.114:0/3140671333 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c0d06f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 --2- 172.21.15.114:0/3140671333 >> v2:172.21.15.177:3300/0 conn(0x7fdf6c15dfd0 0x7fdf6c162700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 -- 172.21.15.114:0/3140671333 >> 172.21.15.114:0/3140671333 conn(0x7fdf6c06c0d0 msgr2=0x7fdf6c06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:42.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 -- 172.21.15.114:0/3140671333 shutdown_connections 2024-03-06T07:36:42.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.411+0000 7fdf70a00640 1 -- 172.21.15.114:0/3140671333 wait complete. 2024-03-06T07:36:42.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.413+0000 7fdf70a00640 1 Processor -- start 2024-03-06T07:36:42.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.413+0000 7fdf70a00640 1 -- start start 2024-03-06T07:36:42.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf70a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c160c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf70a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf6c0affe0 0x7fdf6c1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf70a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf6c15dfd0 0x7fdf6c0b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf70a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c0ce8d0 con 0x7fdf6c0affe0 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf70a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c161f00 con 0x7fdf6c15dfd0 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf70a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c0cea50 con 0x7fdf6c06b6d0 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf63e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf6c0affe0 0x7fdf6c1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf63e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf6c0affe0 0x7fdf6c1524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49558/0 (socket says 172.21.15.114:49558) 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.414+0000 7fdf63e00640 1 -- 172.21.15.114:0/1936375229 learned_addr learned my addr 172.21.15.114:0/1936375229 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf6be00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf6c15dfd0 0x7fdf6c0b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf6b400640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c160c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf63e00640 1 -- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 msgr2=0x7fdf6c160c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf63e00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c160c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf63e00640 1 -- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf6c15dfd0 msgr2=0x7fdf6c0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf63e00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf6c15dfd0 0x7fdf6c0b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf63e00640 1 -- 172.21.15.114:0/1936375229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c1529e0 con 0x7fdf6c0affe0 2024-03-06T07:36:42.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf6b400640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c160c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:42.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf63e00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf6c0affe0 0x7fdf6c1524a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c4b0 tx=0x7fdf5400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:42.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.415+0000 7fdf63400640 1 -- 172.21.15.114:0/1936375229 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54017070 con 0x7fdf6c0affe0 2024-03-06T07:36:42.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.416+0000 7fdf63400640 1 -- 172.21.15.114:0/1936375229 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf54005480 con 0x7fdf6c0affe0 2024-03-06T07:36:42.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.416+0000 7fdf63400640 1 -- 172.21.15.114:0/1936375229 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54003830 con 0x7fdf6c0affe0 2024-03-06T07:36:42.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.416+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c15fb90 con 0x7fdf6c0affe0 2024-03-06T07:36:42.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.416+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c160150 con 0x7fdf6c0affe0 2024-03-06T07:36:42.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.417+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c04f330 con 0x7fdf6c0affe0 2024-03-06T07:36:42.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.419+0000 7fdf63400640 1 -- 172.21.15.114:0/1936375229 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf5401a070 con 0x7fdf6c0affe0 2024-03-06T07:36:42.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.420+0000 7fdf63400640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf4806b4d0 0x7fdf4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:42.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.420+0000 7fdf63400640 1 -- 172.21.15.114:0/1936375229 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf54089630 con 0x7fdf6c0affe0 2024-03-06T07:36:42.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.420+0000 7fdf6b400640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf4806b4d0 0x7fdf4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:42.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.421+0000 7fdf6b400640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf4806b4d0 0x7fdf4806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009a20 tx=0x7fdf5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:42.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.425+0000 7fdf63400640 1 -- 172.21.15.114:0/1936375229 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf540583d0 con 0x7fdf6c0affe0 2024-03-06T07:36:42.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.663+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf6c0cdea0 con 0x7fdf6c0affe0 2024-03-06T07:36:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.663+0000 7fdf63400640 1 -- 172.21.15.114:0/1936375229 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7fdf5405e0d0 con 0x7fdf6c0affe0 2024-03-06T07:36:42.665 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:42.665 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:42.666 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:42.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.666+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf4806b4d0 msgr2=0x7fdf4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.666+0000 7fdf70a00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf4806b4d0 0x7fdf4806d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009a20 tx=0x7fdf5c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.666+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf6c0affe0 msgr2=0x7fdf6c1524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.666+0000 7fdf70a00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf6c0affe0 0x7fdf6c1524a0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400c4b0 tx=0x7fdf5400c970 comp rx=0 tx=0).stop 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 shutdown_connections 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf6c15dfd0 0x7fdf6c0b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdf4806b4d0 0x7fdf4806d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf6c0affe0 0x7fdf6c1524a0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 --2- 172.21.15.114:0/1936375229 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdf6c06b6d0 0x7fdf6c160c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 >> 172.21.15.114:0/1936375229 conn(0x7fdf6c06c0d0 msgr2=0x7fdf6c0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 shutdown_connections 2024-03-06T07:36:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:42.667+0000 7fdf70a00640 1 -- 172.21.15.114:0/1936375229 wait complete. 2024-03-06T07:36:42.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:42.889 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:43.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:44.024 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:44.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.100+0000 7ff853e00640 1 Processor -- start 2024-03-06T07:36:44.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.100+0000 7ff853e00640 1 -- start start 2024-03-06T07:36:44.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.103+0000 7ff853e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8540af5a0 0x7ff8540b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.103+0000 7ff853e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff854066f50 con 0x7ff85415be20 2024-03-06T07:36:44.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.103+0000 7ff853e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff854067230 con 0x7ff854072670 2024-03-06T07:36:44.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.103+0000 7ff853e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff854066d00 con 0x7ff8540af5a0 2024-03-06T07:36:44.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.103+0000 7ff852000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8540af5a0 0x7ff8540b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.103+0000 7ff852a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff85415be20 0x7ff8540b16a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39312/0 (socket says 172.21.15.114:39312) 2024-03-06T07:36:44.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.103+0000 7ff852a00640 1 -- 172.21.15.114:0/3108846567 learned_addr learned my addr 172.21.15.114:0/3108846567 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:44.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.104+0000 7ff852000640 1 -- 172.21.15.114:0/3108846567 >> v1:172.21.15.177:6789/0 conn(0x7ff854072670 legacy=0x7ff85415c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.104+0000 7ff852000640 1 -- 172.21.15.114:0/3108846567 >> v1:172.21.15.114:6789/0 conn(0x7ff85415be20 legacy=0x7ff8540b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.104+0000 7ff852000640 1 -- 172.21.15.114:0/3108846567 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540b0ea0 con 0x7ff8540af5a0 2024-03-06T07:36:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.105+0000 7ff852000640 1 --2- 172.21.15.114:0/3108846567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8540af5a0 0x7ff8540b5db0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff854066e80 tx=0x7ff84002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7c9bf8dfd6d608 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.106+0000 7ff851600640 1 -- 172.21.15.114:0/3108846567 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840035020 con 0x7ff8540af5a0 2024-03-06T07:36:44.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.106+0000 7ff851600640 1 -- 172.21.15.114:0/3108846567 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff840031810 con 0x7ff8540af5a0 2024-03-06T07:36:44.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.106+0000 7ff851600640 1 -- 172.21.15.114:0/3108846567 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840031be0 con 0x7ff8540af5a0 2024-03-06T07:36:44.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.106+0000 7ff853e00640 1 -- 172.21.15.114:0/3108846567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8540af5a0 msgr2=0x7ff8540b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.106+0000 7ff853e00640 1 --2- 172.21.15.114:0/3108846567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8540af5a0 0x7ff8540b5db0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff854066e80 tx=0x7ff84002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.107+0000 7ff853e00640 1 -- 172.21.15.114:0/3108846567 shutdown_connections 2024-03-06T07:36:44.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.107+0000 7ff853e00640 1 --2- 172.21.15.114:0/3108846567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8540af5a0 0x7ff8540b5db0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.107+0000 7ff853e00640 1 -- 172.21.15.114:0/3108846567 >> 172.21.15.114:0/3108846567 conn(0x7ff8540b4fe0 msgr2=0x7ff8540b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:44.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.107+0000 7ff853e00640 1 -- 172.21.15.114:0/3108846567 shutdown_connections 2024-03-06T07:36:44.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.107+0000 7ff853e00640 1 -- 172.21.15.114:0/3108846567 wait complete. 2024-03-06T07:36:44.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.109+0000 7ff853e00640 1 Processor -- start 2024-03-06T07:36:44.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.109+0000 7ff853e00640 1 -- start start 2024-03-06T07:36:44.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff853e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff854072670 0x7ff8541f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff853e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8540af5a0 0x7ff854149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff853e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff85415be20 0x7ff854153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff853e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854072ac0 con 0x7ff854072670 2024-03-06T07:36:44.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff853e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540b0b50 con 0x7ff85415be20 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff853e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff854072c40 con 0x7ff8540af5a0 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff854072670 0x7ff8541f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff854072670 0x7ff8541f7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49572/0 (socket says 172.21.15.114:49572) 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852a00640 1 -- 172.21.15.114:0/2249247931 learned_addr learned my addr 172.21.15.114:0/2249247931 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852000640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8540af5a0 0x7ff854149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff853400640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff85415be20 0x7ff854153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852a00640 1 -- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8540af5a0 msgr2=0x7ff854149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852a00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8540af5a0 0x7ff854149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852a00640 1 -- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff85415be20 msgr2=0x7ff854153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.110+0000 7ff852a00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff85415be20 0x7ff854153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.111+0000 7ff852a00640 1 -- 172.21.15.114:0/2249247931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85414a720 con 0x7ff854072670 2024-03-06T07:36:44.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.111+0000 7ff852a00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff854072670 0x7ff8541f7030 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7ff84400c3f0 tx=0x7ff84400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.111+0000 7ff84aa00640 1 -- 172.21.15.114:0/2249247931 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844017070 con 0x7ff854072670 2024-03-06T07:36:44.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.111+0000 7ff84aa00640 1 -- 172.21.15.114:0/2249247931 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844005ce0 con 0x7ff854072670 2024-03-06T07:36:44.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.111+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85414aa00 con 0x7ff854072670 2024-03-06T07:36:44.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.111+0000 7ff84aa00640 1 -- 172.21.15.114:0/2249247931 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8440072f0 con 0x7ff854072670 2024-03-06T07:36:44.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.111+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85414afc0 con 0x7ff854072670 2024-03-06T07:36:44.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.113+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85414c9c0 con 0x7ff854072670 2024-03-06T07:36:44.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.113+0000 7ff84aa00640 1 -- 172.21.15.114:0/2249247931 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff84401a070 con 0x7ff854072670 2024-03-06T07:36:44.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.114+0000 7ff84aa00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.114+0000 7ff84aa00640 1 -- 172.21.15.114:0/2249247931 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff844089550 con 0x7ff854072670 2024-03-06T07:36:44.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.114+0000 7ff852000640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.115+0000 7ff852000640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff840036000 tx=0x7ff840008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.119+0000 7ff84aa00640 1 -- 172.21.15.114:0/2249247931 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff85414c9c0 con 0x7ff854072670 2024-03-06T07:36:44.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.354+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff85404f330 con 0x7ff854072670 2024-03-06T07:36:44.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.354+0000 7ff84aa00640 1 -- 172.21.15.114:0/2249247931 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3423 (secure 0 0 0) 0x7ff844058420 con 0x7ff854072670 2024-03-06T07:36:44.356 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:44.356 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":103,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:14.916463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:44.356 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 103 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff83406b4d0 msgr2=0x7ff83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff840036000 tx=0x7ff840008030 comp rx=0 tx=0).stop 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff854072670 msgr2=0x7ff8541f7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff854072670 0x7ff8541f7030 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7ff84400c3f0 tx=0x7ff84400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 shutdown_connections 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff85415be20 0x7ff854153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff8540af5a0 0x7ff854149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 --2- 172.21.15.114:0/2249247931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff854072670 0x7ff8541f7030 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 >> 172.21.15.114:0/2249247931 conn(0x7ff8540b4fe0 msgr2=0x7ff8541547d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 shutdown_connections 2024-03-06T07:36:44.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:44.357+0000 7ff853e00640 1 -- 172.21.15.114:0/2249247931 wait complete. 2024-03-06T07:36:44.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:14.916463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:44.580 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:45.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:45.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.792+0000 7f51d6400640 1 Processor -- start 2024-03-06T07:36:45.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.792+0000 7f51d6400640 1 -- start start 2024-03-06T07:36:45.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51d6400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f51d015e0d0 0x7f51d00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51d6400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d015f470 0x7f51d00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51d6400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d00cea60 con 0x7f51d00d0df0 2024-03-06T07:36:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51d6400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d00d1260 con 0x7f51d015e0d0 2024-03-06T07:36:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51d6400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00d2f60 con 0x7f51d015f470 2024-03-06T07:36:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51cfe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f51d00d0df0 0x7f51d0160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39324/0 (socket says 172.21.15.114:39324) 2024-03-06T07:36:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51cfe00640 1 -- 172.21.15.114:0/1567798312 learned_addr learned my addr 172.21.15.114:0/1567798312 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.794+0000 7f51cf400640 1 --2- 172.21.15.114:0/1567798312 >> v2:172.21.15.177:3300/0 conn(0x7f51d015e0d0 0x7f51d00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691981013 0 0) 0x7f51d00cea60 con 0x7f51d00d0df0 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51c6a00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d015f470 0x7f51d00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b4003520 con 0x7f51d00d0df0 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714963202 0 0) 0x7f51b4003520 con 0x7f51d00d0df0 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d015f470 msgr2=0x7f51d00d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d015f470 0x7f51d00d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 >> v2:172.21.15.177:3300/0 conn(0x7f51d015e0d0 msgr2=0x7f51d00d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 --2- 172.21.15.114:0/1567798312 >> v2:172.21.15.177:3300/0 conn(0x7f51d015e0d0 0x7f51d00d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d00d2350 con 0x7f51d00d0df0 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f51c8001670 con 0x7f51d00d0df0 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 >> v1:172.21.15.114:6789/0 conn(0x7f51d00d0df0 legacy=0x7f51d0160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d015f470 0x7f51b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51b4004d40 0x7f51b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00d1260 con 0x7f51b4003f30 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b400b9c0 con 0x7f51b4004d40 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b400bb70 con 0x7f51d015f470 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.795+0000 7f51c6a00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.796+0000 7f51c6a00640 1 -- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d015f470 msgr2=0x7f51b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.796+0000 7f51c6a00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d015f470 0x7f51b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.796+0000 7f51c6a00640 1 -- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51b4004d40 msgr2=0x7f51b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.796+0000 7f51c6a00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51b4004d40 0x7f51b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.796+0000 7f51c6a00640 1 -- 172.21.15.114:0/1567798312 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b4003520 con 0x7f51b4003f30 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.797+0000 7f51c6a00640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51b4004800 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009ac0 tx=0x7f51b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b6afe86ed69ed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.797+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8039070 con 0x7f51b4003f30 2024-03-06T07:36:45.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.797+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b802c710 con 0x7f51b4003f30 2024-03-06T07:36:45.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.797+0000 7f51cea00640 1 -- 172.21.15.114:0/1567798312 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8009690 con 0x7f51b4003f30 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.798+0000 7f51d6400640 1 -- 172.21.15.114:0/1567798312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 msgr2=0x7f51b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.798+0000 7f51d6400640 1 --2- 172.21.15.114:0/1567798312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51b4004800 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009ac0 tx=0x7f51b4005830 comp rx=0 tx=0).stop 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.798+0000 7f51cfe00640 1 -- 172.21.15.114:0/1567798312 reap_dead start 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.798+0000 7f51d6400640 1 -- 172.21.15.114:0/1567798312 shutdown_connections 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.798+0000 7f51d6400640 1 -- 172.21.15.114:0/1567798312 >> 172.21.15.114:0/1567798312 conn(0x7f51d00d3ec0 msgr2=0x7f51d00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.798+0000 7f51d6400640 1 -- 172.21.15.114:0/1567798312 shutdown_connections 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.798+0000 7f51d6400640 1 -- 172.21.15.114:0/1567798312 wait complete. 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.800+0000 7f51d6400640 1 Processor -- start 2024-03-06T07:36:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.800+0000 7f51d6400640 1 -- start start 2024-03-06T07:36:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51d6400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51d015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51d6400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d00ca0d0 0x7f51d015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51d6400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d015c930 0x7f51d01600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51d6400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00d2bf0 con 0x7f51b4003f30 2024-03-06T07:36:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51d6400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00d2910 con 0x7f51d015c930 2024-03-06T07:36:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51d6400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d015fb30 con 0x7f51d00ca0d0 2024-03-06T07:36:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51d015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51d015a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49588/0 (socket says 172.21.15.114:49588) 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 -- 172.21.15.114:0/96552653 learned_addr learned my addr 172.21.15.114:0/96552653 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 -- 172.21.15.114:0/96552653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d00ca0d0 msgr2=0x7f51d015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d00ca0d0 0x7f51d015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 -- 172.21.15.114:0/96552653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d015c930 msgr2=0x7f51d01600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d015c930 0x7f51d01600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.801+0000 7f51cf400640 1 -- 172.21.15.114:0/96552653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d00c9030 con 0x7f51b4003f30 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.802+0000 7f51cf400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51d015a120 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009e10 tx=0x7f51c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.802+0000 7f51c7e00640 1 -- 172.21.15.114:0/96552653 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0018020 con 0x7f51b4003f30 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.802+0000 7f51c7e00640 1 -- 172.21.15.114:0/96552653 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c0014480 con 0x7f51b4003f30 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.802+0000 7f51c7e00640 1 -- 172.21.15.114:0/96552653 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0014870 con 0x7f51b4003f30 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.802+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d00c9310 con 0x7f51b4003f30 2024-03-06T07:36:45.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.802+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d00c98d0 con 0x7f51b4003f30 2024-03-06T07:36:45.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.803+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d015cd50 con 0x7f51b4003f30 2024-03-06T07:36:45.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.804+0000 7f51c7e00640 1 -- 172.21.15.114:0/96552653 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51c0005d10 con 0x7f51b4003f30 2024-03-06T07:36:45.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.805+0000 7f51c7e00640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51ac06b4d0 0x7f51ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:45.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.806+0000 7f51c7e00640 1 -- 172.21.15.114:0/96552653 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51c008b120 con 0x7f51b4003f30 2024-03-06T07:36:45.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.806+0000 7f51c6a00640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51ac06b4d0 0x7f51ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:45.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.806+0000 7f51c6a00640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51ac06b4d0 0x7f51ac06d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f51b8036000 tx=0x7f51b802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:45.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:45.810+0000 7f51c7e00640 1 -- 172.21.15.114:0/96552653 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51c0022030 con 0x7f51b4003f30 2024-03-06T07:36:46.026 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:46.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.127+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51d004f330 con 0x7f51b4003f30 2024-03-06T07:36:46.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.128+0000 7f51c7e00640 1 -- 172.21.15.114:0/96552653 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f51c0010040 con 0x7f51b4003f30 2024-03-06T07:36:46.129 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:46.130 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:46.130 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:46.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51ac06b4d0 msgr2=0x7f51ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51ac06b4d0 0x7f51ac06d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f51b8036000 tx=0x7f51b802e000 comp rx=0 tx=0).stop 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 msgr2=0x7f51d015a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51d015a120 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009e10 tx=0x7f51c000c490 comp rx=0 tx=0).stop 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 shutdown_connections 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51d015c930 0x7f51d01600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f51d00ca0d0 0x7f51d015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f51ac06b4d0 0x7f51ac06d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 --2- 172.21.15.114:0/96552653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51b4003f30 0x7f51d015a120 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 >> 172.21.15.114:0/96552653 conn(0x7f51d00d3ec0 msgr2=0x7f51d006c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.130+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 shutdown_connections 2024-03-06T07:36:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:46.131+0000 7f51d6400640 1 -- 172.21.15.114:0/96552653 wait complete. 2024-03-06T07:36:46.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:46.358 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:47.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:47.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:47.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:47.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.574+0000 7f8036400640 1 Processor -- start 2024-03-06T07:36:47.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.574+0000 7f8036400640 1 -- start start 2024-03-06T07:36:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.576+0000 7f8036400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f803015fe60 0x7f8030071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.576+0000 7f8036400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803006cef0 con 0x7f803015f9d0 2024-03-06T07:36:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.576+0000 7f8036400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803006d1d0 con 0x7f8030144c00 2024-03-06T07:36:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.576+0000 7f8036400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803006d4b0 con 0x7f803015fe60 2024-03-06T07:36:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.576+0000 7f802ea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f803015f9d0 0x7f8030070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39332/0 (socket says 172.21.15.114:39332) 2024-03-06T07:36:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.576+0000 7f802ea00640 1 -- 172.21.15.114:0/1246698985 learned_addr learned my addr 172.21.15.114:0/1246698985 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:47.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.576+0000 7f802fe00640 1 --2- 172.21.15.114:0/1246698985 >> v2:172.21.15.114:3300/0 conn(0x7f803015fe60 0x7f8030071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.577+0000 7f802fe00640 1 -- 172.21.15.114:0/1246698985 >> v1:172.21.15.177:6789/0 conn(0x7f8030144c00 legacy=0x7f803006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.577+0000 7f802fe00640 1 -- 172.21.15.114:0/1246698985 >> v1:172.21.15.114:6789/0 conn(0x7f803015f9d0 legacy=0x7f8030070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.577+0000 7f802fe00640 1 -- 172.21.15.114:0/1246698985 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803006d790 con 0x7f803015fe60 2024-03-06T07:36:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.578+0000 7f802fe00640 1 --2- 172.21.15.114:0/1246698985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015fe60 0x7f8030071220 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f8028009d90 tx=0x7f802802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a3842ccbe9a41f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.578+0000 7f802e000640 1 -- 172.21.15.114:0/1246698985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028039070 con 0x7f803015fe60 2024-03-06T07:36:47.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.578+0000 7f802e000640 1 -- 172.21.15.114:0/1246698985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802802cc60 con 0x7f803015fe60 2024-03-06T07:36:47.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.578+0000 7f802e000640 1 -- 172.21.15.114:0/1246698985 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028034660 con 0x7f803015fe60 2024-03-06T07:36:47.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.579+0000 7f8036400640 1 -- 172.21.15.114:0/1246698985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015fe60 msgr2=0x7f8030071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.579+0000 7f8036400640 1 --2- 172.21.15.114:0/1246698985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015fe60 0x7f8030071220 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f8028009d90 tx=0x7f802802beb0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.579+0000 7f8036400640 1 -- 172.21.15.114:0/1246698985 shutdown_connections 2024-03-06T07:36:47.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.579+0000 7f8036400640 1 --2- 172.21.15.114:0/1246698985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015fe60 0x7f8030071220 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.579+0000 7f8036400640 1 -- 172.21.15.114:0/1246698985 >> 172.21.15.114:0/1246698985 conn(0x7f80300b1730 msgr2=0x7f80300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:47.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.579+0000 7f8036400640 1 -- 172.21.15.114:0/1246698985 shutdown_connections 2024-03-06T07:36:47.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.579+0000 7f8036400640 1 -- 172.21.15.114:0/1246698985 wait complete. 2024-03-06T07:36:47.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.581+0000 7f8036400640 1 Processor -- start 2024-03-06T07:36:47.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.581+0000 7f8036400640 1 -- start start 2024-03-06T07:36:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.582+0000 7f8036400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8030144c00 0x7f803013b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.582+0000 7f8036400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015f9d0 0x7f80300b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.582+0000 7f8036400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f803015fe60 0x7f803013fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.582+0000 7f8036400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803006cb50 con 0x7f803015f9d0 2024-03-06T07:36:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.582+0000 7f8036400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8030160730 con 0x7f803015fe60 2024-03-06T07:36:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.582+0000 7f8036400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803006ccd0 con 0x7f8030144c00 2024-03-06T07:36:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015f9d0 0x7f80300b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015f9d0 0x7f80300b38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49604/0 (socket says 172.21.15.114:49604) 2024-03-06T07:36:47.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 -- 172.21.15.114:0/1519374676 learned_addr learned my addr 172.21.15.114:0/1519374676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:47.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802fe00640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f803015fe60 0x7f803013fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 -- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8030144c00 msgr2=0x7f803013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802f400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8030144c00 0x7f803013b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8030144c00 0x7f803013b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 -- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f803015fe60 msgr2=0x7f803013fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f803015fe60 0x7f803013fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 -- 172.21.15.114:0/1519374676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80300b2820 con 0x7f803015f9d0 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802fe00640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f803015fe60 0x7f803013fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802f400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8030144c00 0x7f803013b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.583+0000 7f802ea00640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015f9d0 0x7f80300b38c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c430 tx=0x7f801c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.584+0000 7f8027400640 1 -- 172.21.15.114:0/1519374676 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c017070 con 0x7f803015f9d0 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.584+0000 7f8027400640 1 -- 172.21.15.114:0/1519374676 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c005ce0 con 0x7f803015f9d0 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.584+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80300b2b00 con 0x7f803015f9d0 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.584+0000 7f8027400640 1 -- 172.21.15.114:0/1519374676 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c0072f0 con 0x7f803015f9d0 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.584+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80300b30c0 con 0x7f803015f9d0 2024-03-06T07:36:47.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.585+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803004f330 con 0x7f803015f9d0 2024-03-06T07:36:47.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.586+0000 7f8027400640 1 -- 172.21.15.114:0/1519374676 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f801c01a070 con 0x7f803015f9d0 2024-03-06T07:36:47.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.587+0000 7f8027400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f801006b4d0 0x7f801006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:47.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.588+0000 7f8027400640 1 -- 172.21.15.114:0/1519374676 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f801c089660 con 0x7f803015f9d0 2024-03-06T07:36:47.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.588+0000 7f802f400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f801006b4d0 0x7f801006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:47.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.589+0000 7f802f400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f801006b4d0 0x7f801006d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f802000a830 tx=0x7f8020008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:47.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.592+0000 7f8027400640 1 -- 172.21.15.114:0/1519374676 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f801c0584a0 con 0x7f803015f9d0 2024-03-06T07:36:47.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.830+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8030072890 con 0x7f803015f9d0 2024-03-06T07:36:47.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.831+0000 7f8027400640 1 -- 172.21.15.114:0/1519374676 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f801c057da0 con 0x7f803015f9d0 2024-03-06T07:36:47.833 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:47.833 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:47.833 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f801006b4d0 msgr2=0x7f801006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f801006b4d0 0x7f801006d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f802000a830 tx=0x7f8020008040 comp rx=0 tx=0).stop 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015f9d0 msgr2=0x7f80300b38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015f9d0 0x7f80300b38c0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c430 tx=0x7f801c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 shutdown_connections 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f803015fe60 0x7f803013fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f801006b4d0 0x7f801006d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f803015f9d0 0x7f80300b38c0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 --2- 172.21.15.114:0/1519374676 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8030144c00 0x7f803013b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 >> 172.21.15.114:0/1519374676 conn(0x7f80300b1730 msgr2=0x7f80300cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 shutdown_connections 2024-03-06T07:36:47.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:47.834+0000 7f8036400640 1 -- 172.21.15.114:0/1519374676 wait complete. 2024-03-06T07:36:48.028 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:48.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:48.058 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:49.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.275+0000 7f278ce00640 1 Processor -- start 2024-03-06T07:36:49.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.275+0000 7f278ce00640 1 -- start start 2024-03-06T07:36:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.276+0000 7f278ce00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f278806afc0 0x7f27881632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f278ce00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880b59f0 con 0x7f2788169ec0 2024-03-06T07:36:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f278ce00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278806dd20 con 0x7f27880d1630 2024-03-06T07:36:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f278ce00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278806e760 con 0x7f278806afc0 2024-03-06T07:36:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f2786800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2788169ec0 0x7f278816c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:60320/0 (socket says 172.21.15.114:60320) 2024-03-06T07:36:49.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f2786800640 1 -- 172.21.15.114:0/1884922420 learned_addr learned my addr 172.21.15.114:0/1884922420 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:49.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799283881 0 0) 0x7f27880b59f0 con 0x7f2788169ec0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f276c003520 con 0x7f2788169ec0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.277+0000 7f2785e00640 1 --2- 172.21.15.114:0/1884922420 >> v2:172.21.15.177:3300/0 conn(0x7f278806afc0 0x7f27881632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3251254211 0 0) 0x7f276c003520 con 0x7f2788169ec0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 >> v2:172.21.15.177:3300/0 conn(0x7f278806afc0 msgr2=0x7f27881632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 --2- 172.21.15.114:0/1884922420 >> v2:172.21.15.177:3300/0 conn(0x7f278806afc0 0x7f27881632e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 >> v1:172.21.15.177:6789/0 conn(0x7f27880d1630 legacy=0x7f278806b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278806e000 con 0x7f2788169ec0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f27800019f0 con 0x7f2788169ec0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 >> v1:172.21.15.114:6789/0 conn(0x7f2788169ec0 legacy=0x7f278816c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f276c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f276c004800 0x7f276c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f276c0053c0 0x7f276c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278806e760 con 0x7f276c0039b0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c00c250 con 0x7f276c0053c0 2024-03-06T07:36:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276c00c450 con 0x7f276c004800 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.278+0000 7f2785e00640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f276c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2785e00640 1 -- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f276c004800 msgr2=0x7f276c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f277fe00640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f276c004800 0x7f276c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2786800640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f276c0053c0 0x7f276c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2785e00640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f276c004800 0x7f276c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2785e00640 1 -- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f276c0053c0 msgr2=0x7f276c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2785e00640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f276c0053c0 0x7f276c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2785e00640 1 -- 172.21.15.114:0/1884922420 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c003520 con 0x7f276c0039b0 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2786800640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f276c0053c0 0x7f276c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:49.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.279+0000 7f2786800640 1 -- 172.21.15.114:0/1884922420 reap_dead start 2024-03-06T07:36:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.280+0000 7f2785e00640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f276c003dd0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f277800ba30 tx=0x7f276c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1044236691d00177 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.280+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778036020 con 0x7f276c0039b0 2024-03-06T07:36:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.280+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2778032640 con 0x7f276c0039b0 2024-03-06T07:36:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.280+0000 7f2785400640 1 -- 172.21.15.114:0/1884922420 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800b600 con 0x7f276c0039b0 2024-03-06T07:36:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.281+0000 7f278ce00640 1 -- 172.21.15.114:0/1884922420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 msgr2=0x7f276c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.281+0000 7f278ce00640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f276c003dd0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f277800ba30 tx=0x7f276c005830 comp rx=0 tx=0).stop 2024-03-06T07:36:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.281+0000 7f278ce00640 1 -- 172.21.15.114:0/1884922420 shutdown_connections 2024-03-06T07:36:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.281+0000 7f278ce00640 1 --2- 172.21.15.114:0/1884922420 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f276c003dd0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.281+0000 7f278ce00640 1 -- 172.21.15.114:0/1884922420 >> 172.21.15.114:0/1884922420 conn(0x7f27880d0b40 msgr2=0x7f27880d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.281+0000 7f278ce00640 1 -- 172.21.15.114:0/1884922420 shutdown_connections 2024-03-06T07:36:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.281+0000 7f278ce00640 1 -- 172.21.15.114:0/1884922420 wait complete. 2024-03-06T07:36:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.283+0000 7f278ce00640 1 Processor -- start 2024-03-06T07:36:49.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.283+0000 7f278ce00640 1 -- start start 2024-03-06T07:36:49.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f278ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f278815d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f278ce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2788167ed0 0x7f27881666d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f278ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27880aa6a0 0x7f27880aab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f278ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278806ed70 con 0x7f276c0039b0 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f278ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278806ebf0 con 0x7f27880aa6a0 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f278ce00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27881632e0 con 0x7f2788167ed0 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f2785e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f278815d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f2785e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f278815d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32806/0 (socket says 172.21.15.114:32806) 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.284+0000 7f2785e00640 1 -- 172.21.15.114:0/1929014193 learned_addr learned my addr 172.21.15.114:0/1929014193 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2785e00640 1 -- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2788167ed0 msgr2=0x7f27881666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f277fe00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2788167ed0 0x7f27881666d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2786800640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27880aa6a0 0x7f27880aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2785e00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2788167ed0 0x7f27881666d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2785e00640 1 -- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27880aa6a0 msgr2=0x7f27880aab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2785e00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27880aa6a0 0x7f27880aab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2785e00640 1 -- 172.21.15.114:0/1929014193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788166d60 con 0x7f276c0039b0 2024-03-06T07:36:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2786800640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27880aa6a0 0x7f27880aab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f2785e00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f278815d0d0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f2778035040 tx=0x7f2778004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f277e000640 1 -- 172.21.15.114:0/1929014193 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778041070 con 0x7f276c0039b0 2024-03-06T07:36:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f277e000640 1 -- 172.21.15.114:0/1929014193 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2778030a50 con 0x7f276c0039b0 2024-03-06T07:36:49.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f277e000640 1 -- 172.21.15.114:0/1929014193 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778036040 con 0x7f276c0039b0 2024-03-06T07:36:49.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.285+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2788167040 con 0x7f276c0039b0 2024-03-06T07:36:49.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.286+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27881f85d0 con 0x7f276c0039b0 2024-03-06T07:36:49.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.287+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278804f330 con 0x7f276c0039b0 2024-03-06T07:36:49.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.288+0000 7f277e000640 1 -- 172.21.15.114:0/1929014193 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2778039020 con 0x7f276c0039b0 2024-03-06T07:36:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.288+0000 7f277e000640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f276406b500 0x7f276406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.289+0000 7f277e000640 1 -- 172.21.15.114:0/1929014193 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f27780ac430 con 0x7f276c0039b0 2024-03-06T07:36:49.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.289+0000 7f277fe00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f276406b500 0x7f276406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:49.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.289+0000 7f277fe00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f276406b500 0x7f276406d9c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f27880b3850 tx=0x7f2770009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:49.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.298+0000 7f277e000640 1 -- 172.21.15.114:0/1929014193 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f277807b260 con 0x7f276c0039b0 2024-03-06T07:36:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.532+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2788163bb0 con 0x7f276c0039b0 2024-03-06T07:36:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.533+0000 7f277e000640 1 -- 172.21.15.114:0/1929014193 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f277807ab60 con 0x7f276c0039b0 2024-03-06T07:36:49.536 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:49.536 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:49.536 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:49.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f276406b500 msgr2=0x7f276406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f276406b500 0x7f276406d9c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f27880b3850 tx=0x7f2770009040 comp rx=0 tx=0).stop 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 msgr2=0x7f278815d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f278815d0d0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f2778035040 tx=0x7f2778004730 comp rx=0 tx=0).stop 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 shutdown_connections 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27880aa6a0 0x7f27880aab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2788167ed0 0x7f27881666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f276406b500 0x7f276406d9c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 --2- 172.21.15.114:0/1929014193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f276c0039b0 0x7f278815d0d0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.535+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 >> 172.21.15.114:0/1929014193 conn(0x7f27880d0b40 msgr2=0x7f2788161df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.536+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 shutdown_connections 2024-03-06T07:36:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:49.536+0000 7f278ce00640 1 -- 172.21.15.114:0/1929014193 wait complete. 2024-03-06T07:36:49.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:49.758 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:50.031 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:50.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:50.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.965+0000 7f9269000640 1 Processor -- start 2024-03-06T07:36:50.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.965+0000 7f9269000640 1 -- start start 2024-03-06T07:36:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.968+0000 7f9269000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9264161580 0x7f92640d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.968+0000 7f9269000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f92640d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.968+0000 7f9269000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9264161960 con 0x7f92640d28a0 2024-03-06T07:36:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.968+0000 7f9269000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92640cf6f0 con 0x7f9264161580 2024-03-06T07:36:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.968+0000 7f9269000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92640cf9d0 con 0x7f92640d1e90 2024-03-06T07:36:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f92640d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f92640d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:55502/0 (socket says 172.21.15.114:55502) 2024-03-06T07:36:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262a00640 1 -- 172.21.15.114:0/3741735355 learned_addr learned my addr 172.21.15.114:0/3741735355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9263400640 1 --2- 172.21.15.114:0/3741735355 >> v2:172.21.15.177:3300/0 conn(0x7f9264161580 0x7f92640d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262000640 1 -- 172.21.15.114:0/3741735355 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651260583 0 0) 0x7f9264161960 con 0x7f92640d28a0 2024-03-06T07:36:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262000640 1 -- 172.21.15.114:0/3741735355 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9248003520 con 0x7f92640d28a0 2024-03-06T07:36:50.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262a00640 1 -- 172.21.15.114:0/3741735355 >> v2:172.21.15.177:3300/0 conn(0x7f9264161580 msgr2=0x7f92640d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262a00640 1 --2- 172.21.15.114:0/3741735355 >> v2:172.21.15.177:3300/0 conn(0x7f9264161580 0x7f92640d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.969+0000 7f9262a00640 1 -- 172.21.15.114:0/3741735355 >> v1:172.21.15.177:6789/0 conn(0x7f92640d28a0 legacy=0x7f92640d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.970+0000 7f9262a00640 1 -- 172.21.15.114:0/3741735355 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92640cfcb0 con 0x7f92640d1e90 2024-03-06T07:36:50.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.971+0000 7f9262a00640 1 --2- 172.21.15.114:0/3741735355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f92640d2360 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9250009870 tx=0x7f925002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=178c51621c350622 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.971+0000 7f9262000640 1 -- 172.21.15.114:0/3741735355 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250034070 con 0x7f92640d1e90 2024-03-06T07:36:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.971+0000 7f9262000640 1 -- 172.21.15.114:0/3741735355 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250030840 con 0x7f92640d1e90 2024-03-06T07:36:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.971+0000 7f9262000640 1 -- 172.21.15.114:0/3741735355 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250030bf0 con 0x7f92640d1e90 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 -- 172.21.15.114:0/3741735355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 msgr2=0x7f92640d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 --2- 172.21.15.114:0/3741735355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f92640d2360 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9250009870 tx=0x7f925002b980 comp rx=0 tx=0).stop 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 -- 172.21.15.114:0/3741735355 shutdown_connections 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 --2- 172.21.15.114:0/3741735355 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f92640d2360 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 --2- 172.21.15.114:0/3741735355 >> v2:172.21.15.177:3300/0 conn(0x7f9264161580 0x7f92640d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 -- 172.21.15.114:0/3741735355 >> 172.21.15.114:0/3741735355 conn(0x7f92640d3d80 msgr2=0x7f92640d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 -- 172.21.15.114:0/3741735355 shutdown_connections 2024-03-06T07:36:50.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.972+0000 7f9269000640 1 -- 172.21.15.114:0/3741735355 wait complete. 2024-03-06T07:36:50.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.976+0000 7f9269000640 1 Processor -- start 2024-03-06T07:36:50.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.976+0000 7f9269000640 1 -- start start 2024-03-06T07:36:50.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9269000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f926415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9269000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92640d28a0 0x7f92640ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9269000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9264161580 0x7f92640cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9269000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92640cf3a0 con 0x7f9264161580 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9269000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92640b39a0 con 0x7f92640d28a0 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9269000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92640b4ed0 con 0x7f92640d1e90 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9263e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9264161580 0x7f92640cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9263e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9264161580 0x7f92640cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32816/0 (socket says 172.21.15.114:32816) 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.978+0000 7f9263e00640 1 -- 172.21.15.114:0/1033845066 learned_addr learned my addr 172.21.15.114:0/1033845066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9262a00640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92640d28a0 0x7f92640ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263400640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f926415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263e00640 1 -- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 msgr2=0x7f926415ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263e00640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f926415ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263e00640 1 -- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92640d28a0 msgr2=0x7f92640ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263e00640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92640d28a0 0x7f92640ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263e00640 1 -- 172.21.15.114:0/1033845066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926415d1f0 con 0x7f9264161580 2024-03-06T07:36:50.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263400640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f926415ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:50.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9263e00640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9264161580 0x7f92640cae50 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c790 tx=0x7f925c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f925b400640 1 -- 172.21.15.114:0/1033845066 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c003b70 con 0x7f9264161580 2024-03-06T07:36:50.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f925b400640 1 -- 172.21.15.114:0/1033845066 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c003d10 con 0x7f9264161580 2024-03-06T07:36:50.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f925b400640 1 -- 172.21.15.114:0/1033845066 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c00aa50 con 0x7f9264161580 2024-03-06T07:36:50.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.979+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92640c7cf0 con 0x7f9264161580 2024-03-06T07:36:50.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.980+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92640cdb90 con 0x7f9264161580 2024-03-06T07:36:50.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.981+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926404f330 con 0x7f9264161580 2024-03-06T07:36:50.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.983+0000 7f925b400640 1 -- 172.21.15.114:0/1033845066 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f925c00abf0 con 0x7f9264161580 2024-03-06T07:36:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.984+0000 7f925b400640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f924406b4d0 0x7f924406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.984+0000 7f925b400640 1 -- 172.21.15.114:0/1033845066 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f925c08a240 con 0x7f9264161580 2024-03-06T07:36:50.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.984+0000 7f9263400640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f924406b4d0 0x7f924406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.985+0000 7f9263400640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f924406b4d0 0x7f924406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9254009a20 tx=0x7f9254008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:50.989+0000 7f925b400640 1 -- 172.21.15.114:0/1033845066 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f925c059070 con 0x7f9264161580 2024-03-06T07:36:51.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.225+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92640cac10 con 0x7f9264161580 2024-03-06T07:36:51.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.226+0000 7f925b400640 1 -- 172.21.15.114:0/1033845066 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f925c058970 con 0x7f9264161580 2024-03-06T07:36:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:51.229 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:51.229 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f924406b4d0 msgr2=0x7f924406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f924406b4d0 0x7f924406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9254009a20 tx=0x7f9254008040 comp rx=0 tx=0).stop 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9264161580 msgr2=0x7f92640cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9264161580 0x7f92640cae50 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c790 tx=0x7f925c00cc50 comp rx=0 tx=0).stop 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 shutdown_connections 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f924406b4d0 0x7f924406d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9264161580 0x7f92640cae50 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92640d28a0 0x7f92640ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 --2- 172.21.15.114:0/1033845066 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f92640d1e90 0x7f926415ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 >> 172.21.15.114:0/1033845066 conn(0x7f92640d3d80 msgr2=0x7f92640b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 shutdown_connections 2024-03-06T07:36:51.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:51.229+0000 7f9269000640 1 -- 172.21.15.114:0/1033845066 wait complete. 2024-03-06T07:36:51.452 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:51.452 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:52.033 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:52.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:52.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.665+0000 7f26a3e00640 1 Processor -- start 2024-03-06T07:36:52.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.665+0000 7f26a3e00640 1 -- start start 2024-03-06T07:36:52.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.668+0000 7f26a3e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f26a40d2360 0x7f26a40d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.668+0000 7f26a3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a4161960 con 0x7f26a40b3ec0 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.668+0000 7f26a3e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a40cf6f0 con 0x7f26a4161580 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.668+0000 7f26a3e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a40d2ce0 con 0x7f26a40d2360 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.668+0000 7f26a3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f26a40b3ec0 0x7f26a40b4310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:60336/0 (socket says 172.21.15.114:60336) 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.668+0000 7f26a3400640 1 -- 172.21.15.114:0/898360182 learned_addr learned my addr 172.21.15.114:0/898360182 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243254616 0 0) 0x7f26a4161960 con 0x7f26a40b3ec0 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a2000640 1 --2- 172.21.15.114:0/898360182 >> v2:172.21.15.177:3300/0 conn(0x7f26a40d2360 0x7f26a40d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2684003520 con 0x7f26a40b3ec0 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3199322341 0 0) 0x7f2684003520 con 0x7f26a40b3ec0 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 >> v2:172.21.15.177:3300/0 conn(0x7f26a40d2360 msgr2=0x7f26a40d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 --2- 172.21.15.114:0/898360182 >> v2:172.21.15.177:3300/0 conn(0x7f26a40d2360 0x7f26a40d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 >> v1:172.21.15.177:6789/0 conn(0x7f26a4161580 legacy=0x7f26a40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a40cf9d0 con 0x7f26a40b3ec0 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f269c0019f0 con 0x7f26a40b3ec0 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 >> v1:172.21.15.114:6789/0 conn(0x7f26a40b3ec0 legacy=0x7f26a40b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26840039b0 0x7f2684003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2684004800 0x7f2684004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f268400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40d2ce0 con 0x7f26840051c0 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268400c110 con 0x7f26840039b0 2024-03-06T07:36:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268400c310 con 0x7f2684004800 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.669+0000 7f26a3400640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f268400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a2000640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2684004800 0x7f2684004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a2a00640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26840039b0 0x7f2684003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a3400640 1 -- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2684004800 msgr2=0x7f2684004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a3400640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2684004800 0x7f2684004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a3400640 1 -- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26840039b0 msgr2=0x7f2684003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a3400640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26840039b0 0x7f2684003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a3400640 1 -- 172.21.15.114:0/898360182 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684003520 con 0x7f26840051c0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.670+0000 7f26a2a00640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26840039b0 0x7f2684003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.671+0000 7f26a3400640 1 -- 172.21.15.114:0/898360182 reap_dead start 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.671+0000 7f26a3400640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f268400b9c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f269c00a860 tx=0x7f2684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79604b655c989c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.671+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c00a480 con 0x7f26840051c0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.671+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c034630 con 0x7f26840051c0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.671+0000 7f26a1600640 1 -- 172.21.15.114:0/898360182 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c034a00 con 0x7f26840051c0 2024-03-06T07:36:52.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.672+0000 7f26a3e00640 1 -- 172.21.15.114:0/898360182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 msgr2=0x7f268400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.672+0000 7f26a3e00640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f268400b9c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f269c00a860 tx=0x7f2684005830 comp rx=0 tx=0).stop 2024-03-06T07:36:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.672+0000 7f26a3e00640 1 -- 172.21.15.114:0/898360182 shutdown_connections 2024-03-06T07:36:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.672+0000 7f26a3e00640 1 --2- 172.21.15.114:0/898360182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f268400b9c0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.672+0000 7f26a3e00640 1 -- 172.21.15.114:0/898360182 >> 172.21.15.114:0/898360182 conn(0x7f26a40d3d80 msgr2=0x7f26a40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.672+0000 7f26a3e00640 1 -- 172.21.15.114:0/898360182 shutdown_connections 2024-03-06T07:36:52.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.672+0000 7f26a3e00640 1 -- 172.21.15.114:0/898360182 wait complete. 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.674+0000 7f26a3e00640 1 Processor -- start 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.674+0000 7f26a3e00640 1 -- start start 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f26a414dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a3e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a40cd7d0 0x7f26a40c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a40c7410 0x7f26a40c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40b39a0 con 0x7f26840051c0 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40b36c0 con 0x7f26a40c7410 2024-03-06T07:36:52.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a3e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40cf3a0 con 0x7f26a40cd7d0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f26a414dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f26a414dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32838/0 (socket says 172.21.15.114:32838) 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 -- 172.21.15.114:0/2649279196 learned_addr learned my addr 172.21.15.114:0/2649279196 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2000640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a40cd7d0 0x7f26a40c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a3400640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a40c7410 0x7f26a40c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 -- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a40cd7d0 msgr2=0x7f26a40c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a40cd7d0 0x7f26a40c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 -- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a40c7410 msgr2=0x7f26a40c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a40c7410 0x7f26a40c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.675+0000 7f26a2a00640 1 -- 172.21.15.114:0/2649279196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a40c9ab0 con 0x7f26840051c0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.676+0000 7f26a2a00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f26a414dd70 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f269400d770 tx=0x7f269400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.676+0000 7f269aa00640 1 -- 172.21.15.114:0/2649279196 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694005270 con 0x7f26840051c0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.676+0000 7f269aa00640 1 -- 172.21.15.114:0/2649279196 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2694017070 con 0x7f26840051c0 2024-03-06T07:36:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.676+0000 7f269aa00640 1 -- 172.21.15.114:0/2649279196 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269400b650 con 0x7f26840051c0 2024-03-06T07:36:52.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.676+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a40c9d90 con 0x7f26840051c0 2024-03-06T07:36:52.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.676+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a40ca350 con 0x7f26840051c0 2024-03-06T07:36:52.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.677+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a404f330 con 0x7f26840051c0 2024-03-06T07:36:52.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.678+0000 7f269aa00640 1 -- 172.21.15.114:0/2649279196 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f269400b7f0 con 0x7f26840051c0 2024-03-06T07:36:52.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.679+0000 7f269aa00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b520 0x7f268006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:52.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.679+0000 7f269aa00640 1 -- 172.21.15.114:0/2649279196 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2694089eb0 con 0x7f26840051c0 2024-03-06T07:36:52.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.679+0000 7f26a2000640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b520 0x7f268006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:52.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.680+0000 7f26a2000640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b520 0x7f268006d9e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f26a40d0b80 tx=0x7f268c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:52.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.684+0000 7f269aa00640 1 -- 172.21.15.114:0/2649279196 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2694058e60 con 0x7f26840051c0 2024-03-06T07:36:52.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.922+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a40d3b60 con 0x7f26840051c0 2024-03-06T07:36:52.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.923+0000 7f269aa00640 1 -- 172.21.15.114:0/2649279196 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f2694001230 con 0x7f26840051c0 2024-03-06T07:36:52.926 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:52.926 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:52.926 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:52.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b520 msgr2=0x7f268006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b520 0x7f268006d9e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f26a40d0b80 tx=0x7f268c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:52.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 msgr2=0x7f26a414dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f26a414dd70 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f269400d770 tx=0x7f269400dc30 comp rx=0 tx=0).stop 2024-03-06T07:36:52.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 shutdown_connections 2024-03-06T07:36:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26a40c7410 0x7f26a40c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a40cd7d0 0x7f26a40c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b520 0x7f268006d9e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 --2- 172.21.15.114:0/2649279196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26840051c0 0x7f26a414dd70 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.925+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 >> 172.21.15.114:0/2649279196 conn(0x7f26a40d3d80 msgr2=0x7f26a40cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.926+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 shutdown_connections 2024-03-06T07:36:52.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:52.926+0000 7f26a3e00640 1 -- 172.21.15.114:0/2649279196 wait complete. 2024-03-06T07:36:53.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:53.155 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:54.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:54.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.360+0000 7fd01b000640 1 Processor -- start 2024-03-06T07:36:54.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.360+0000 7fd01b000640 1 -- start start 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.362+0000 7fd01b000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd01415f930 0x7fd014066f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.362+0000 7fd01b000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd0140ce8f0 0x7fd0140ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.362+0000 7fd01b000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01406e5b0 con 0x7fd01406ed70 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.362+0000 7fd01b000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01406f220 con 0x7fd01415f930 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.362+0000 7fd01b000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0140ccd90 con 0x7fd0140ce8f0 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.362+0000 7fd013e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd01415f930 0x7fd014066f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd013e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd01415f930 0x7fd014066f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32854/0 (socket says 172.21.15.114:32854) 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd013e00640 1 -- 172.21.15.114:0/808774493 learned_addr learned my addr 172.21.15.114:0/808774493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd019000640 1 --2- 172.21.15.114:0/808774493 >> v2:172.21.15.177:3300/0 conn(0x7fd0140ce8f0 0x7fd0140ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd013e00640 1 -- 172.21.15.114:0/808774493 >> v2:172.21.15.177:3300/0 conn(0x7fd0140ce8f0 msgr2=0x7fd0140ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd013e00640 1 --2- 172.21.15.114:0/808774493 >> v2:172.21.15.177:3300/0 conn(0x7fd0140ce8f0 0x7fd0140ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd013e00640 1 -- 172.21.15.114:0/808774493 >> v1:172.21.15.177:6789/0 conn(0x7fd01406ed70 legacy=0x7fd01415fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd013e00640 1 -- 172.21.15.114:0/808774493 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0140ccfe0 con 0x7fd01415f930 2024-03-06T07:36:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.363+0000 7fd019000640 1 --2- 172.21.15.114:0/808774493 >> v2:172.21.15.177:3300/0 conn(0x7fd0140ce8f0 0x7fd0140ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.364+0000 7fd013e00640 1 --2- 172.21.15.114:0/808774493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd014066f10 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fd0140b3460 tx=0x7fd00402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899a8d64dd09e6e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.365+0000 7fd013400640 1 -- 172.21.15.114:0/808774493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00402e070 con 0x7fd01415f930 2024-03-06T07:36:54.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.365+0000 7fd013400640 1 -- 172.21.15.114:0/808774493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd004037070 con 0x7fd01415f930 2024-03-06T07:36:54.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.365+0000 7fd013400640 1 -- 172.21.15.114:0/808774493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd004031a00 con 0x7fd01415f930 2024-03-06T07:36:54.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.365+0000 7fd01b000640 1 -- 172.21.15.114:0/808774493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 msgr2=0x7fd014066f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.365+0000 7fd01b000640 1 --2- 172.21.15.114:0/808774493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd014066f10 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fd0140b3460 tx=0x7fd00402b880 comp rx=0 tx=0).stop 2024-03-06T07:36:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.366+0000 7fd01b000640 1 -- 172.21.15.114:0/808774493 shutdown_connections 2024-03-06T07:36:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.366+0000 7fd01b000640 1 --2- 172.21.15.114:0/808774493 >> v2:172.21.15.177:3300/0 conn(0x7fd0140ce8f0 0x7fd0140ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.366+0000 7fd01b000640 1 --2- 172.21.15.114:0/808774493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd014066f10 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.366+0000 7fd01b000640 1 -- 172.21.15.114:0/808774493 >> 172.21.15.114:0/808774493 conn(0x7fd0140d0340 msgr2=0x7fd0140d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.366+0000 7fd01b000640 1 -- 172.21.15.114:0/808774493 shutdown_connections 2024-03-06T07:36:54.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.366+0000 7fd01b000640 1 -- 172.21.15.114:0/808774493 wait complete. 2024-03-06T07:36:54.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.367+0000 7fd01b000640 1 Processor -- start 2024-03-06T07:36:54.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.368+0000 7fd01b000640 1 -- start start 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.368+0000 7fd01b000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd01406ed70 0x7fd0140b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.368+0000 7fd01b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0140ce8f0 0x7fd0141f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.368+0000 7fd01b000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd0141eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd01b000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01406ddb0 con 0x7fd01415f930 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd01b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0140afc00 con 0x7fd0140ce8f0 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd01b000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01406e090 con 0x7fd01406ed70 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd0141eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd0141eda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32866/0 (socket says 172.21.15.114:32866) 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 -- 172.21.15.114:0/1144955311 learned_addr learned my addr 172.21.15.114:0/1144955311 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd013e00640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd01406ed70 0x7fd0140b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 -- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd01406ed70 msgr2=0x7fd0140b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd00be00640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0140ce8f0 0x7fd0141f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd01406ed70 0x7fd0140b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 -- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0140ce8f0 msgr2=0x7fd0141f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0140ce8f0 0x7fd0141f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd019000640 1 -- 172.21.15.114:0/1144955311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0140b4020 con 0x7fd01415f930 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.369+0000 7fd00be00640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0140ce8f0 0x7fd0141f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.370+0000 7fd019000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd0141eda80 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00ce40 tx=0x7fd00c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.370+0000 7fd010c00640 1 -- 172.21.15.114:0/1144955311 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c01a070 con 0x7fd01415f930 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.370+0000 7fd010c00640 1 -- 172.21.15.114:0/1144955311 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c00d840 con 0x7fd01415f930 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.370+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0140b4300 con 0x7fd01415f930 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.370+0000 7fd010c00640 1 -- 172.21.15.114:0/1144955311 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c00dc30 con 0x7fd01415f930 2024-03-06T07:36:54.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.370+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0140b48c0 con 0x7fd01415f930 2024-03-06T07:36:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.371+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01404f330 con 0x7fd01415f930 2024-03-06T07:36:54.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.372+0000 7fd010c00640 1 -- 172.21.15.114:0/1144955311 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd00c01d070 con 0x7fd01415f930 2024-03-06T07:36:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.373+0000 7fd010c00640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcff006b500 0x7fcff006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.373+0000 7fd010c00640 1 -- 172.21.15.114:0/1144955311 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd00c08aab0 con 0x7fd01415f930 2024-03-06T07:36:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.374+0000 7fd013e00640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcff006b500 0x7fcff006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:54.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.374+0000 7fd013e00640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcff006b500 0x7fcff006d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd004036000 tx=0x7fd00403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:54.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.378+0000 7fd010c00640 1 -- 172.21.15.114:0/1144955311 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd00c059850 con 0x7fd01415f930 2024-03-06T07:36:54.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.619+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0140cd8c0 con 0x7fd01415f930 2024-03-06T07:36:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.620+0000 7fd010c00640 1 -- 172.21.15.114:0/1144955311 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fd00c059150 con 0x7fd01415f930 2024-03-06T07:36:54.623 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:54.623 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:54.623 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcff006b500 msgr2=0x7fcff006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcff006b500 0x7fcff006d9c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd004036000 tx=0x7fd00403d040 comp rx=0 tx=0).stop 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 msgr2=0x7fd0141eda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd0141eda80 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00ce40 tx=0x7fd00c00a710 comp rx=0 tx=0).stop 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 shutdown_connections 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fcff006b500 0x7fcff006d9c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd01415f930 0x7fd0141eda80 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0140ce8f0 0x7fd0141f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 --2- 172.21.15.114:0/1144955311 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd01406ed70 0x7fd0140b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 >> 172.21.15.114:0/1144955311 conn(0x7fd0140d0340 msgr2=0x7fd01415fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 shutdown_connections 2024-03-06T07:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:54.623+0000 7fd01b000640 1 -- 172.21.15.114:0/1144955311 wait complete. 2024-03-06T07:36:54.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:54.848 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:55.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:56.036 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.060+0000 7f2758e00640 1 Processor -- start 2024-03-06T07:36:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.061+0000 7f2758e00640 1 -- start start 2024-03-06T07:36:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.062+0000 7f2758e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f27540d24c0 0x7f27540d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.062+0000 7f2758e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27540b2de0 0x7f27540d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.062+0000 7f2758e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.062+0000 7f2758e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27540d4a00 con 0x7f27540b2de0 2024-03-06T07:36:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.062+0000 7f2758e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27540cf7e0 con 0x7f27540d24c0 2024-03-06T07:36:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.062+0000 7f2758e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27540d0080 con 0x7f27540d07f0 2024-03-06T07:36:56.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.062+0000 7f2749400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27540b2de0 0x7f27540d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2752800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2751e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f27540d24c0 0x7f27540d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2749400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27540b2de0 0x7f27540d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32882/0 (socket says 172.21.15.114:32882) 2024-03-06T07:36:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2749400640 1 -- 172.21.15.114:0/2188440553 learned_addr learned my addr 172.21.15.114:0/2188440553 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2749400640 1 -- 172.21.15.114:0/2188440553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 msgr2=0x7f27540d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2749400640 1 --2- 172.21.15.114:0/2188440553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540d0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2749400640 1 -- 172.21.15.114:0/2188440553 >> v2:172.21.15.177:3300/0 conn(0x7f27540d24c0 msgr2=0x7f27540d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:56.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2749400640 1 --2- 172.21.15.114:0/2188440553 >> v2:172.21.15.177:3300/0 conn(0x7f27540d24c0 0x7f27540d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.063+0000 7f2749400640 1 -- 172.21.15.114:0/2188440553 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27540d2de0 con 0x7f27540b2de0 2024-03-06T07:36:56.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.066+0000 7f2749400640 1 --2- 172.21.15.114:0/2188440553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540b2de0 0x7f27540d28a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f275408d300 tx=0x7f274402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7019d0f99d67cfda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.067+0000 7f2751400640 1 -- 172.21.15.114:0/2188440553 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274402e070 con 0x7f27540b2de0 2024-03-06T07:36:56.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.067+0000 7f2751400640 1 -- 172.21.15.114:0/2188440553 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2744037070 con 0x7f27540b2de0 2024-03-06T07:36:56.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.067+0000 7f2751400640 1 -- 172.21.15.114:0/2188440553 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2744031a00 con 0x7f27540b2de0 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 -- 172.21.15.114:0/2188440553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540b2de0 msgr2=0x7f27540d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 --2- 172.21.15.114:0/2188440553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540b2de0 0x7f27540d28a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f275408d300 tx=0x7f274402b880 comp rx=0 tx=0).stop 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 -- 172.21.15.114:0/2188440553 shutdown_connections 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 --2- 172.21.15.114:0/2188440553 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540d0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 --2- 172.21.15.114:0/2188440553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540b2de0 0x7f27540d28a0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 --2- 172.21.15.114:0/2188440553 >> v2:172.21.15.177:3300/0 conn(0x7f27540d24c0 0x7f27540d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 -- 172.21.15.114:0/2188440553 >> 172.21.15.114:0/2188440553 conn(0x7f27540d3ee0 msgr2=0x7f27540d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 -- 172.21.15.114:0/2188440553 shutdown_connections 2024-03-06T07:36:56.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.068+0000 7f2758e00640 1 -- 172.21.15.114:0/2188440553 wait complete. 2024-03-06T07:36:56.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.071+0000 7f2758e00640 1 Processor -- start 2024-03-06T07:36:56.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.071+0000 7f2758e00640 1 -- start start 2024-03-06T07:36:56.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2758e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27540b2de0 0x7f27540c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2758e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2758e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540d24c0 0x7f2754158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2758e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27540cfd10 con 0x7f27540d24c0 2024-03-06T07:36:56.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2758e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27540cfa30 con 0x7f27540b2de0 2024-03-06T07:36:56.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2758e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275415e5b0 con 0x7f27540d07f0 2024-03-06T07:36:56.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2752800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540d24c0 0x7f2754158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2752800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540d24c0 0x7f2754158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32884/0 (socket says 172.21.15.114:32884) 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.072+0000 7f2752800640 1 -- 172.21.15.114:0/1722561683 learned_addr learned my addr 172.21.15.114:0/1722561683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2749400640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2751e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27540b2de0 0x7f27540c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2752800640 1 -- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 msgr2=0x7f27540ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2752800640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2752800640 1 -- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27540b2de0 msgr2=0x7f27540c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2752800640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27540b2de0 0x7f27540c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2752800640 1 -- 172.21.15.114:0/1722561683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27541f4290 con 0x7f27540d24c0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2751e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27540b2de0 0x7f27540c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2752800640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540d24c0 0x7f2754158a10 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f274c00eed0 tx=0x7f274c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f274aa00640 1 -- 172.21.15.114:0/1722561683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c018020 con 0x7f27540d24c0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f274aa00640 1 -- 172.21.15.114:0/1722561683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c00bd20 con 0x7f27540d24c0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f274aa00640 1 -- 172.21.15.114:0/1722561683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c014640 con 0x7f27540d24c0 2024-03-06T07:36:56.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.073+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27541f4570 con 0x7f27540d24c0 2024-03-06T07:36:56.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.074+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27541f4b30 con 0x7f27540d24c0 2024-03-06T07:36:56.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.075+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275404f330 con 0x7f27540d24c0 2024-03-06T07:36:56.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.076+0000 7f274aa00640 1 -- 172.21.15.114:0/1722561683 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f274c005d10 con 0x7f27540d24c0 2024-03-06T07:36:56.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.076+0000 7f274aa00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f273006b4d0 0x7f273006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:56.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.077+0000 7f274aa00640 1 -- 172.21.15.114:0/1722561683 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f274c08a960 con 0x7f27540d24c0 2024-03-06T07:36:56.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.077+0000 7f2751e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f273006b4d0 0x7f273006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:56.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.078+0000 7f2751e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f273006b4d0 0x7f273006d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f27540b2b70 tx=0x7f273c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:56.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.081+0000 7f274aa00640 1 -- 172.21.15.114:0/1722561683 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f274c022070 con 0x7f27540d24c0 2024-03-06T07:36:56.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.325+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27540d22a0 con 0x7f27540d24c0 2024-03-06T07:36:56.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.326+0000 7f274aa00640 1 -- 172.21.15.114:0/1722561683 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f274c059700 con 0x7f27540d24c0 2024-03-06T07:36:56.329 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:56.329 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:56.329 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:56.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f273006b4d0 msgr2=0x7f273006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f273006b4d0 0x7f273006d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f27540b2b70 tx=0x7f273c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540d24c0 msgr2=0x7f2754158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540d24c0 0x7f2754158a10 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f274c00eed0 tx=0x7f274c00b480 comp rx=0 tx=0).stop 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 shutdown_connections 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f273006b4d0 0x7f273006d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27540d24c0 0x7f2754158a10 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f27540d07f0 0x7f27540ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 --2- 172.21.15.114:0/1722561683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27540b2de0 0x7f27540c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 >> 172.21.15.114:0/1722561683 conn(0x7f27540d3ee0 msgr2=0x7f27540d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 shutdown_connections 2024-03-06T07:36:56.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:56.328+0000 7f2758e00640 1 -- 172.21.15.114:0/1722561683 wait complete. 2024-03-06T07:36:56.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:56.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:57.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:57.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.760+0000 7fe4aca00640 1 Processor -- start 2024-03-06T07:36:57.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.760+0000 7fe4aca00640 1 -- start start 2024-03-06T07:36:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.761+0000 7fe4aca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a8161580 0x7fe4a80d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.761+0000 7fe4aca00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4a80d1e90 0x7fe4a80d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.761+0000 7fe4aca00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe4a80d28a0 0x7fe4a80d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.761+0000 7fe4aca00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a8161960 con 0x7fe4a80d1e90 2024-03-06T07:36:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.761+0000 7fe4aca00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a80cf6f0 con 0x7fe4a80d28a0 2024-03-06T07:36:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.761+0000 7fe4aca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80cf9d0 con 0x7fe4a8161580 2024-03-06T07:36:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4a80d1e90 0x7fe4a80d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4a80d1e90 0x7fe4a80d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32896/0 (socket says 172.21.15.114:32896) 2024-03-06T07:36:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 -- 172.21.15.114:0/2057819701 learned_addr learned my addr 172.21.15.114:0/2057819701 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:57.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a7400640 1 --2- 172.21.15.114:0/2057819701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a8161580 0x7fe4a80d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a7e00640 1 --2- 172.21.15.114:0/2057819701 >> v2:172.21.15.177:3300/0 conn(0x7fe4a80d28a0 0x7fe4a80d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 -- 172.21.15.114:0/2057819701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a8161580 msgr2=0x7fe4a80d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:57.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 --2- 172.21.15.114:0/2057819701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a8161580 0x7fe4a80d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 -- 172.21.15.114:0/2057819701 >> v2:172.21.15.177:3300/0 conn(0x7fe4a80d28a0 msgr2=0x7fe4a80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:57.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 --2- 172.21.15.114:0/2057819701 >> v2:172.21.15.177:3300/0 conn(0x7fe4a80d28a0 0x7fe4a80d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.762+0000 7fe4a6a00640 1 -- 172.21.15.114:0/2057819701 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80cfcb0 con 0x7fe4a80d1e90 2024-03-06T07:36:57.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.764+0000 7fe4a6a00640 1 --2- 172.21.15.114:0/2057819701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80d2360 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80b6880 tx=0x7fe49002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194b593ade7157a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:57.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.764+0000 7fe4a6000640 1 -- 172.21.15.114:0/2057819701 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490035020 con 0x7fe4a80d1e90 2024-03-06T07:36:57.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.764+0000 7fe4a6000640 1 -- 172.21.15.114:0/2057819701 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe490031870 con 0x7fe4a80d1e90 2024-03-06T07:36:57.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.764+0000 7fe4a6000640 1 -- 172.21.15.114:0/2057819701 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe490031c40 con 0x7fe4a80d1e90 2024-03-06T07:36:57.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 -- 172.21.15.114:0/2057819701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 msgr2=0x7fe4a80d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 --2- 172.21.15.114:0/2057819701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80d2360 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80b6880 tx=0x7fe49002b910 comp rx=0 tx=0).stop 2024-03-06T07:36:57.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 -- 172.21.15.114:0/2057819701 shutdown_connections 2024-03-06T07:36:57.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 --2- 172.21.15.114:0/2057819701 >> v2:172.21.15.177:3300/0 conn(0x7fe4a80d28a0 0x7fe4a80d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 --2- 172.21.15.114:0/2057819701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80d2360 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 --2- 172.21.15.114:0/2057819701 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a8161580 0x7fe4a80d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 -- 172.21.15.114:0/2057819701 >> 172.21.15.114:0/2057819701 conn(0x7fe4a80d3d80 msgr2=0x7fe4a80d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:57.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 -- 172.21.15.114:0/2057819701 shutdown_connections 2024-03-06T07:36:57.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.765+0000 7fe4aca00640 1 -- 172.21.15.114:0/2057819701 wait complete. 2024-03-06T07:36:57.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.768+0000 7fe4aca00640 1 Processor -- start 2024-03-06T07:36:57.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.768+0000 7fe4aca00640 1 -- start start 2024-03-06T07:36:57.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4aca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80b9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4aca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a80d28a0 0x7fe4a815dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4aca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8161580 0x7fe4a815c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4aca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80cf3a0 con 0x7fe4a80d1e90 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4aca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80b39a0 con 0x7fe4a8161580 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4aca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80d0c20 con 0x7fe4a80d28a0 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80b9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32912/0 (socket says 172.21.15.114:32912) 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 -- 172.21.15.114:0/795691554 learned_addr learned my addr 172.21.15.114:0/795691554 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a6a00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a80d28a0 0x7fe4a815dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 -- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a80d28a0 msgr2=0x7fe4a815dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a80d28a0 0x7fe4a815dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 -- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8161580 msgr2=0x7fe4a815c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7e00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8161580 0x7fe4a815c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8161580 0x7fe4a815c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.770+0000 7fe4a7400640 1 -- 172.21.15.114:0/795691554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80b9a80 con 0x7fe4a80d1e90 2024-03-06T07:36:57.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.771+0000 7fe4a7400640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80b9540 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fe498007e40 tx=0x7fe49800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:57.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.771+0000 7fe49f400640 1 -- 172.21.15.114:0/795691554 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498014620 con 0x7fe4a80d1e90 2024-03-06T07:36:57.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.771+0000 7fe49f400640 1 -- 172.21.15.114:0/795691554 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe498019070 con 0x7fe4a80d1e90 2024-03-06T07:36:57.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.771+0000 7fe49f400640 1 -- 172.21.15.114:0/795691554 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498014a00 con 0x7fe4a80d1e90 2024-03-06T07:36:57.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.771+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a815cb50 con 0x7fe4a80d1e90 2024-03-06T07:36:57.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.771+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a815d110 con 0x7fe4a80d1e90 2024-03-06T07:36:57.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.772+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a804f330 con 0x7fe4a80d1e90 2024-03-06T07:36:57.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.773+0000 7fe49f400640 1 -- 172.21.15.114:0/795691554 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe498014ba0 con 0x7fe4a80d1e90 2024-03-06T07:36:57.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.774+0000 7fe49f400640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe48406b500 0x7fe48406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.774+0000 7fe49f400640 1 -- 172.21.15.114:0/795691554 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe49808aa40 con 0x7fe4a80d1e90 2024-03-06T07:36:57.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.775+0000 7fe4a6a00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe48406b500 0x7fe48406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.775+0000 7fe4a6a00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe48406b500 0x7fe48406d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe490036000 tx=0x7fe490008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:57.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:57.783+0000 7fe49f400640 1 -- 172.21.15.114:0/795691554 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4a80b9a80 con 0x7fe4a80d1e90 2024-03-06T07:36:58.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.019+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4a80d3b60 con 0x7fe4a80d1e90 2024-03-06T07:36:58.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.020+0000 7fe49f400640 1 -- 172.21.15.114:0/795691554 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fe498059870 con 0x7fe4a80d1e90 2024-03-06T07:36:58.022 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:58.023 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:58.023 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:58.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe48406b500 msgr2=0x7fe48406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe48406b500 0x7fe48406d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe490036000 tx=0x7fe490008030 comp rx=0 tx=0).stop 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 msgr2=0x7fe4a80b9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80b9540 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fe498007e40 tx=0x7fe49800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 shutdown_connections 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8161580 0x7fe4a815c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe4a80d28a0 0x7fe4a815dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe48406b500 0x7fe48406d9c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 --2- 172.21.15.114:0/795691554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4a80d1e90 0x7fe4a80b9540 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 >> 172.21.15.114:0/795691554 conn(0x7fe4a80d3d80 msgr2=0x7fe4a80d2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 shutdown_connections 2024-03-06T07:36:58.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:58.022+0000 7fe4aca00640 1 -- 172.21.15.114:0/795691554 wait complete. 2024-03-06T07:36:58.038 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:58.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:58.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:36:59.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:59.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.483+0000 7fb363a00640 1 Processor -- start 2024-03-06T07:36:59.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.483+0000 7fb363a00640 1 -- start start 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb363a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb35c15acb0 0x7fb35c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb363a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb35c0b4680 0x7fb35c06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb363a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c15b090 con 0x7fb35c072340 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb363a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35c0b2a30 con 0x7fb35c0b4680 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb363a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35c0b4360 con 0x7fb35c15acb0 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb353e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb35c0b4680 0x7fb35c06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb353e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb35c0b4680 0x7fb35c06e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56466/0 (socket says 172.21.15.114:56466) 2024-03-06T07:36:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.486+0000 7fb353e00640 1 -- 172.21.15.114:0/2989157011 learned_addr learned my addr 172.21.15.114:0/2989157011 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:59.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.487+0000 7fb353e00640 1 -- 172.21.15.114:0/2989157011 >> v2:172.21.15.177:3300/0 conn(0x7fb35c15acb0 msgr2=0x7fb35c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.487+0000 7fb353e00640 1 --2- 172.21.15.114:0/2989157011 >> v2:172.21.15.177:3300/0 conn(0x7fb35c15acb0 0x7fb35c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.487+0000 7fb353e00640 1 -- 172.21.15.114:0/2989157011 >> v1:172.21.15.177:6789/0 conn(0x7fb35c072340 legacy=0x7fb35c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:59.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.487+0000 7fb353e00640 1 -- 172.21.15.114:0/2989157011 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0b2cf0 con 0x7fb35c0b4680 2024-03-06T07:36:59.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.488+0000 7fb353e00640 1 --2- 172.21.15.114:0/2989157011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c0b4680 0x7fb35c06e4f0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b27f0 tx=0x7fb34402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ff0020e2c9b570 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb35be00640 1 -- 172.21.15.114:0/2989157011 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb344035020 con 0x7fb35c0b4680 2024-03-06T07:36:59.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb35be00640 1 -- 172.21.15.114:0/2989157011 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb344031870 con 0x7fb35c0b4680 2024-03-06T07:36:59.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb35be00640 1 -- 172.21.15.114:0/2989157011 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb344031c40 con 0x7fb35c0b4680 2024-03-06T07:36:59.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb363a00640 1 -- 172.21.15.114:0/2989157011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c0b4680 msgr2=0x7fb35c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb363a00640 1 --2- 172.21.15.114:0/2989157011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c0b4680 0x7fb35c06e4f0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b27f0 tx=0x7fb34402b910 comp rx=0 tx=0).stop 2024-03-06T07:36:59.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb363a00640 1 -- 172.21.15.114:0/2989157011 shutdown_connections 2024-03-06T07:36:59.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb363a00640 1 --2- 172.21.15.114:0/2989157011 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c0b4680 0x7fb35c06e4f0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.489+0000 7fb363a00640 1 --2- 172.21.15.114:0/2989157011 >> v2:172.21.15.177:3300/0 conn(0x7fb35c15acb0 0x7fb35c0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.490+0000 7fb363a00640 1 -- 172.21.15.114:0/2989157011 >> 172.21.15.114:0/2989157011 conn(0x7fb35c0b1730 msgr2=0x7fb35c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.490+0000 7fb363a00640 1 -- 172.21.15.114:0/2989157011 shutdown_connections 2024-03-06T07:36:59.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.490+0000 7fb363a00640 1 -- 172.21.15.114:0/2989157011 wait complete. 2024-03-06T07:36:59.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.491+0000 7fb363a00640 1 Processor -- start 2024-03-06T07:36:59.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.491+0000 7fb363a00640 1 -- start start 2024-03-06T07:36:59.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.492+0000 7fb363a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c072340 0x7fb35c158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.492+0000 7fb363a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35c0b4680 0x7fb35c159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.492+0000 7fb363a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb35c15acb0 0x7fb35c147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.492+0000 7fb363a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c0b3d10 con 0x7fb35c072340 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.492+0000 7fb363a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c0b32b0 con 0x7fb35c0b4680 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.492+0000 7fb363a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c0b4010 con 0x7fb35c15acb0 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c072340 0x7fb35c158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c072340 0x7fb35c158570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56468/0 (socket says 172.21.15.114:56468) 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 -- 172.21.15.114:0/2263700416 learned_addr learned my addr 172.21.15.114:0/2263700416 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb353e00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35c0b4680 0x7fb35c159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361a00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb35c15acb0 0x7fb35c147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 -- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb35c15acb0 msgr2=0x7fb35c147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb35c15acb0 0x7fb35c147ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 -- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35c0b4680 msgr2=0x7fb35c159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35c0b4680 0x7fb35c159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.493+0000 7fb361000640 1 -- 172.21.15.114:0/2263700416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c15d5a0 con 0x7fb35c072340 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.494+0000 7fb361000640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c072340 0x7fb35c158570 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00b4a0 tx=0x7fb34c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.494+0000 7fb359600640 1 -- 172.21.15.114:0/2263700416 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c014070 con 0x7fb35c072340 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.494+0000 7fb359600640 1 -- 172.21.15.114:0/2263700416 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c010610 con 0x7fb35c072340 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.494+0000 7fb359600640 1 -- 172.21.15.114:0/2263700416 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c010a00 con 0x7fb35c072340 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.494+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c15d880 con 0x7fb35c072340 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.494+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c158ab0 con 0x7fb35c072340 2024-03-06T07:36:59.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.495+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c04f330 con 0x7fb35c072340 2024-03-06T07:36:59.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.496+0000 7fb359600640 1 -- 172.21.15.114:0/2263700416 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb34c017070 con 0x7fb35c072340 2024-03-06T07:36:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.497+0000 7fb359600640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb33806b4d0 0x7fb33806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.497+0000 7fb359600640 1 -- 172.21.15.114:0/2263700416 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb34c089bf0 con 0x7fb35c072340 2024-03-06T07:36:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.497+0000 7fb353e00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb33806b4d0 0x7fb33806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:59.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.498+0000 7fb353e00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb33806b4d0 0x7fb33806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb344036000 tx=0x7fb34403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:59.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.506+0000 7fb359600640 1 -- 172.21.15.114:0/2263700416 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb34c058ac0 con 0x7fb35c072340 2024-03-06T07:36:59.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.738+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35c06ce60 con 0x7fb35c072340 2024-03-06T07:36:59.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.739+0000 7fb359600640 1 -- 172.21.15.114:0/2263700416 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fb34c0583c0 con 0x7fb35c072340 2024-03-06T07:36:59.742 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:36:59.742 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:36:59.743 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:36:59.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.742+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb33806b4d0 msgr2=0x7fb33806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb33806b4d0 0x7fb33806d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb344036000 tx=0x7fb34403c040 comp rx=0 tx=0).stop 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c072340 msgr2=0x7fb35c158570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c072340 0x7fb35c158570 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00b4a0 tx=0x7fb34c00b960 comp rx=0 tx=0).stop 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 shutdown_connections 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb35c15acb0 0x7fb35c147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb35c0b4680 0x7fb35c159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb33806b4d0 0x7fb33806d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 --2- 172.21.15.114:0/2263700416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35c072340 0x7fb35c158570 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 >> 172.21.15.114:0/2263700416 conn(0x7fb35c0b1730 msgr2=0x7fb35c157100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 shutdown_connections 2024-03-06T07:36:59.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:36:59.743+0000 7fb363a00640 1 -- 172.21.15.114:0/2263700416 wait complete. 2024-03-06T07:36:59.969 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:36:59.969 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:00.040 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:00.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.174+0000 7f26a3200640 1 Processor -- start 2024-03-06T07:37:01.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.174+0000 7f26a3200640 1 -- start start 2024-03-06T07:37:01.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.176+0000 7f26a3200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f26a4161580 0x7f26a40d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.176+0000 7f26a3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a4161960 con 0x7f26a40d2a70 2024-03-06T07:37:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.176+0000 7f26a3200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a40cf6f0 con 0x7f26a40d1e90 2024-03-06T07:37:01.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.176+0000 7f26a3200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a40cf9d0 con 0x7f26a4161580 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.176+0000 7f26a2800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f26a40d2a70 0x7f26a40b3ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42558/0 (socket says 172.21.15.114:42558) 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.176+0000 7f26a2800640 1 -- 172.21.15.114:0/2848691076 learned_addr learned my addr 172.21.15.114:0/2848691076 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1e00640 1 --2- 172.21.15.114:0/2848691076 >> v2:172.21.15.177:3300/0 conn(0x7f26a4161580 0x7f26a40d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195092053 0 0) 0x7f26a4161960 con 0x7f26a40d2a70 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2688003520 con 0x7f26a40d2a70 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1655705682 0 0) 0x7f2688003520 con 0x7f26a40d2a70 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 >> v2:172.21.15.177:3300/0 conn(0x7f26a4161580 msgr2=0x7f26a40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 --2- 172.21.15.114:0/2848691076 >> v2:172.21.15.177:3300/0 conn(0x7f26a4161580 0x7f26a40d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 >> v1:172.21.15.177:6789/0 conn(0x7f26a40d1e90 legacy=0x7f26a40d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a40cfcb0 con 0x7f26a40d2a70 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f269c001670 con 0x7f26a40d2a70 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 >> v1:172.21.15.114:6789/0 conn(0x7f26a40d2a70 legacy=0x7f26a40b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.177+0000 7f26a1400640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2688004800 0x7f2688004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1400640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26880051c0 0x7f268800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40cf9d0 con 0x7f26880039b0 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268800c060 con 0x7f2688004800 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268800c260 con 0x7f26880051c0 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1e00640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1e00640 1 -- 172.21.15.114:0/2848691076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26880051c0 msgr2=0x7f268800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1e00640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26880051c0 0x7f268800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1e00640 1 -- 172.21.15.114:0/2848691076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2688004800 msgr2=0x7f2688004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1e00640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2688004800 0x7f2688004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a1e00640 1 -- 172.21.15.114:0/2848691076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688003520 con 0x7f26880039b0 2024-03-06T07:37:01.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.178+0000 7f26a2800640 1 -- 172.21.15.114:0/2848691076 reap_dead start 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.179+0000 7f26a1e00640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f2694009fc0 tx=0x7f2688005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f25608e413746b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694039070 con 0x7f26880039b0 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269402cc20 con 0x7f26880039b0 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a1400640 1 -- 172.21.15.114:0/2848691076 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694009c10 con 0x7f26880039b0 2024-03-06T07:37:01.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a3200640 1 -- 172.21.15.114:0/2848691076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26880039b0 msgr2=0x7f2688003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a3200640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f2694009fc0 tx=0x7f2688005830 comp rx=0 tx=0).stop 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a3200640 1 -- 172.21.15.114:0/2848691076 shutdown_connections 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a3200640 1 --2- 172.21.15.114:0/2848691076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26880039b0 0x7f2688003dd0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a3200640 1 -- 172.21.15.114:0/2848691076 >> 172.21.15.114:0/2848691076 conn(0x7f26a40d3d80 msgr2=0x7f26a40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a3200640 1 -- 172.21.15.114:0/2848691076 shutdown_connections 2024-03-06T07:37:01.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.180+0000 7f26a3200640 1 -- 172.21.15.114:0/2848691076 wait complete. 2024-03-06T07:37:01.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.182+0000 7f26a3200640 1 Processor -- start 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.182+0000 7f26a3200640 1 -- start start 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.183+0000 7f26a3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26880039b0 0x7f26a41522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.183+0000 7f26a3200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a41f3ea0 0x7f26a40c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.183+0000 7f26a3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26a40b94c0 0x7f26a40b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.183+0000 7f26a3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40cf3a0 con 0x7f26a40b94c0 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.183+0000 7f26a3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40b39a0 con 0x7f26880039b0 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.183+0000 7f26a3200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a40d1950 con 0x7f26a41f3ea0 2024-03-06T07:37:01.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26a40b94c0 0x7f26a40b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26a40b94c0 0x7f26a40b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56480/0 (socket says 172.21.15.114:56480) 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 -- 172.21.15.114:0/3825664717 learned_addr learned my addr 172.21.15.114:0/3825664717 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f2699400640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a41f3ea0 0x7f26a40c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a1e00640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26880039b0 0x7f26a41522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 -- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a41f3ea0 msgr2=0x7f26a40c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a41f3ea0 0x7f26a40c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 -- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26880039b0 msgr2=0x7f26a41522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26880039b0 0x7f26a41522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a2800640 1 -- 172.21.15.114:0/3825664717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a40c8360 con 0x7f26a40b94c0 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.184+0000 7f26a1e00640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26880039b0 0x7f26a41522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.185+0000 7f26a2800640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26a40b94c0 0x7f26a40b1a00 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f26a408dff0 tx=0x7f269c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.185+0000 7f269aa00640 1 -- 172.21.15.114:0/3825664717 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c004bf0 con 0x7f26a40b94c0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.185+0000 7f269aa00640 1 -- 172.21.15.114:0/3825664717 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c004d90 con 0x7f26a40b94c0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.185+0000 7f269aa00640 1 -- 172.21.15.114:0/3825664717 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c007250 con 0x7f26a40b94c0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.185+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a40b99a0 con 0x7f26a40b94c0 2024-03-06T07:37:01.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.185+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a40c8b40 con 0x7f26a40b94c0 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.186+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a404f330 con 0x7f26a40b94c0 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.187+0000 7f269aa00640 1 -- 172.21.15.114:0/3825664717 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f269c0073f0 con 0x7f26a40b94c0 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.188+0000 7f269aa00640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b4d0 0x7f268006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.188+0000 7f269aa00640 1 -- 172.21.15.114:0/3825664717 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f269c089af0 con 0x7f26a40b94c0 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.188+0000 7f26a1e00640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b4d0 0x7f268006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:01.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.189+0000 7f26a1e00640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b4d0 0x7f268006d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2694036000 tx=0x7f269402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:01.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.198+0000 7f269aa00640 1 -- 172.21.15.114:0/3825664717 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26a40c8360 con 0x7f26a40b94c0 2024-03-06T07:37:01.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.432+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a40d3b60 con 0x7f26a40b94c0 2024-03-06T07:37:01.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.433+0000 7f269aa00640 1 -- 172.21.15.114:0/3825664717 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f269c058930 con 0x7f26a40b94c0 2024-03-06T07:37:01.437 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:01.437 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:01.437 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b4d0 msgr2=0x7f268006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b4d0 0x7f268006d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2694036000 tx=0x7f269402e000 comp rx=0 tx=0).stop 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26a40b94c0 msgr2=0x7f26a40b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26a40b94c0 0x7f26a40b1a00 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f26a408dff0 tx=0x7f269c00c890 comp rx=0 tx=0).stop 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 shutdown_connections 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f268006b4d0 0x7f268006d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26a40b94c0 0x7f26a40b1a00 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f26a41f3ea0 0x7f26a40c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 --2- 172.21.15.114:0/3825664717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26880039b0 0x7f26a41522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 >> 172.21.15.114:0/3825664717 conn(0x7f26a40d3d80 msgr2=0x7f26a415e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:01.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 shutdown_connections 2024-03-06T07:37:01.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:01.436+0000 7f26a3200640 1 -- 172.21.15.114:0/3825664717 wait complete. 2024-03-06T07:37:01.658 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:01.658 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:02.042 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:02.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.871+0000 7f44eb800640 1 Processor -- start 2024-03-06T07:37:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.871+0000 7f44eb800640 1 -- start start 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.872+0000 7f44eb800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f44e40af1c0 0x7f44e415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.872+0000 7f44eb800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f44e4070d20 0x7f44e406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44eb800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e4071d30 con 0x7f44e406d1a0 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44eb800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e40711d0 con 0x7f44e4070d20 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44eb800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e4071490 con 0x7f44e40af1c0 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44e3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f44e4070d20 0x7f44e406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44e3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f44e4070d20 0x7f44e406cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56484/0 (socket says 172.21.15.114:56484) 2024-03-06T07:37:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44e3e00640 1 -- 172.21.15.114:0/3123132279 learned_addr learned my addr 172.21.15.114:0/3123132279 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44e3e00640 1 -- 172.21.15.114:0/3123132279 >> v2:172.21.15.177:3300/0 conn(0x7f44e40af1c0 msgr2=0x7f44e415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44e3e00640 1 --2- 172.21.15.114:0/3123132279 >> v2:172.21.15.177:3300/0 conn(0x7f44e40af1c0 0x7f44e415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44e3e00640 1 -- 172.21.15.114:0/3123132279 >> v1:172.21.15.177:6789/0 conn(0x7f44e406d1a0 legacy=0x7f44e406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.873+0000 7f44e3e00640 1 -- 172.21.15.114:0/3123132279 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4071770 con 0x7f44e4070d20 2024-03-06T07:37:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.874+0000 7f44e3e00640 1 --2- 172.21.15.114:0/3123132279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e406cc60 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae99ce692d24f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44e3400640 1 -- 172.21.15.114:0/3123132279 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0035020 con 0x7f44e4070d20 2024-03-06T07:37:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44e3400640 1 -- 172.21.15.114:0/3123132279 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0031870 con 0x7f44e4070d20 2024-03-06T07:37:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44e3400640 1 -- 172.21.15.114:0/3123132279 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0031c40 con 0x7f44e4070d20 2024-03-06T07:37:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44eb800640 1 -- 172.21.15.114:0/3123132279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 msgr2=0x7f44e406cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44eb800640 1 --2- 172.21.15.114:0/3123132279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e406cc60 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d002b910 comp rx=0 tx=0).stop 2024-03-06T07:37:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44eb800640 1 -- 172.21.15.114:0/3123132279 shutdown_connections 2024-03-06T07:37:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44eb800640 1 --2- 172.21.15.114:0/3123132279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e406cc60 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44eb800640 1 --2- 172.21.15.114:0/3123132279 >> v2:172.21.15.177:3300/0 conn(0x7f44e40af1c0 0x7f44e415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44eb800640 1 -- 172.21.15.114:0/3123132279 >> 172.21.15.114:0/3123132279 conn(0x7f44e40b1730 msgr2=0x7f44e40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.875+0000 7f44eb800640 1 -- 172.21.15.114:0/3123132279 shutdown_connections 2024-03-06T07:37:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.876+0000 7f44eb800640 1 -- 172.21.15.114:0/3123132279 wait complete. 2024-03-06T07:37:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.877+0000 7f44eb800640 1 Processor -- start 2024-03-06T07:37:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.877+0000 7f44eb800640 1 -- start start 2024-03-06T07:37:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.878+0000 7f44eb800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44e406d1a0 0x7f44e4143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.878+0000 7f44eb800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e4160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.878+0000 7f44eb800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e40af1c0 0x7f44e4160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.878+0000 7f44eb800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e406ebe0 con 0x7f44e4070d20 2024-03-06T07:37:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.878+0000 7f44eb800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e40b30b0 con 0x7f44e40af1c0 2024-03-06T07:37:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.878+0000 7f44eb800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e406ed60 con 0x7f44e406d1a0 2024-03-06T07:37:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e4160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e4160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56496/0 (socket says 172.21.15.114:56496) 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 -- 172.21.15.114:0/2905607435 learned_addr learned my addr 172.21.15.114:0/2905607435 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e8e00640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44e406d1a0 0x7f44e4143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 -- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44e406d1a0 msgr2=0x7f44e4143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e9800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e40af1c0 0x7f44e4160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44e406d1a0 0x7f44e4143330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 -- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e40af1c0 msgr2=0x7f44e4160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e40af1c0 0x7f44e4160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 -- 172.21.15.114:0/2905607435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4143e40 con 0x7f44e4070d20 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e9800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e40af1c0 0x7f44e4160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.879+0000 7f44e3e00640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e4160210 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.880+0000 7f44e0c00640 1 -- 172.21.15.114:0/2905607435 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d003ad60 con 0x7f44e4070d20 2024-03-06T07:37:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.880+0000 7f44e0c00640 1 -- 172.21.15.114:0/2905607435 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d00443f0 con 0x7f44e4070d20 2024-03-06T07:37:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.880+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e40ceef0 con 0x7f44e4070d20 2024-03-06T07:37:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.880+0000 7f44e0c00640 1 -- 172.21.15.114:0/2905607435 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d00447e0 con 0x7f44e4070d20 2024-03-06T07:37:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.880+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e40cf4b0 con 0x7f44e4070d20 2024-03-06T07:37:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.881+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e4144010 con 0x7f44e4070d20 2024-03-06T07:37:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.882+0000 7f44e0c00640 1 -- 172.21.15.114:0/2905607435 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44d004b060 con 0x7f44e4070d20 2024-03-06T07:37:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.883+0000 7f44e0c00640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44c406b4d0 0x7f44c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.883+0000 7f44e0c00640 1 -- 172.21.15.114:0/2905607435 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f44d00ac370 con 0x7f44e4070d20 2024-03-06T07:37:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.883+0000 7f44e8e00640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44c406b4d0 0x7f44c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.884+0000 7f44e8e00640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44c406b4d0 0x7f44c406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f44d800a810 tx=0x7f44d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:02.887+0000 7f44e0c00640 1 -- 172.21.15.114:0/2905607435 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44d003f070 con 0x7f44e4070d20 2024-03-06T07:37:03.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.156+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44e404f330 con 0x7f44e4070d20 2024-03-06T07:37:03.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.157+0000 7f44e0c00640 1 -- 172.21.15.114:0/2905607435 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f44e4071d30 con 0x7f44e4070d20 2024-03-06T07:37:03.159 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:03.159 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:03.160 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44c406b4d0 msgr2=0x7f44c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44c406b4d0 0x7f44c406d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f44d800a810 tx=0x7f44d8008040 comp rx=0 tx=0).stop 2024-03-06T07:37:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 msgr2=0x7f44e4160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e4160210 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 shutdown_connections 2024-03-06T07:37:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e40af1c0 0x7f44e4160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44c406b4d0 0x7f44c406d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44e4070d20 0x7f44e4160210 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 --2- 172.21.15.114:0/2905607435 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44e406d1a0 0x7f44e4143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:03.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 >> 172.21.15.114:0/2905607435 conn(0x7f44e40b1730 msgr2=0x7f44e40b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:03.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 shutdown_connections 2024-03-06T07:37:03.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:03.159+0000 7f44eb800640 1 -- 172.21.15.114:0/2905607435 wait complete. 2024-03-06T07:37:03.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:03.383 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:04.045 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:04.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:04.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.591+0000 7f58d8400640 1 Processor -- start 2024-03-06T07:37:04.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.591+0000 7f58d8400640 1 -- start start 2024-03-06T07:37:04.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.593+0000 7f58d8400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d015e0d0 0x7f58d00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.593+0000 7f58d8400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f58d00d0fc0 0x7f58d0160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.593+0000 7f58d8400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d00cea60 con 0x7f58d015f470 2024-03-06T07:37:04.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.593+0000 7f58d8400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d00d2f60 con 0x7f58d00d0fc0 2024-03-06T07:37:04.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.593+0000 7f58d8400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00d2350 con 0x7f58d015e0d0 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d5a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d015e0d0 0x7f58d00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d5a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d015e0d0 0x7f58d00d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:49164/0 (socket says 172.21.15.114:49164) 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d5a00640 1 -- 172.21.15.114:0/1199037510 learned_addr learned my addr 172.21.15.114:0/1199037510 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:04.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d6400640 1 --2- 172.21.15.114:0/1199037510 >> v2:172.21.15.177:3300/0 conn(0x7f58d00d0fc0 0x7f58d0160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d5a00640 1 -- 172.21.15.114:0/1199037510 >> v2:172.21.15.177:3300/0 conn(0x7f58d00d0fc0 msgr2=0x7f58d0160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d5a00640 1 --2- 172.21.15.114:0/1199037510 >> v2:172.21.15.177:3300/0 conn(0x7f58d00d0fc0 0x7f58d0160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d5a00640 1 -- 172.21.15.114:0/1199037510 >> v1:172.21.15.177:6789/0 conn(0x7f58d015f470 legacy=0x7f58d00d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58d5a00640 1 -- 172.21.15.114:0/1199037510 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00d2630 con 0x7f58d015e0d0 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.594+0000 7f58cfe00640 1 -- 172.21.15.114:0/1199037510 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921994580 0 0) 0x7f58d00cea60 con 0x7f58d015f470 2024-03-06T07:37:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.596+0000 7f58d5a00640 1 --2- 172.21.15.114:0/1199037510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d015e0d0 0x7f58d00d1a90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009870 tx=0x7f58c402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f4257c9dc54879 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.596+0000 7f58cfe00640 1 -- 172.21.15.114:0/1199037510 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c402f840 con 0x7f58d015e0d0 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.596+0000 7f58cfe00640 1 -- 172.21.15.114:0/1199037510 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c402f9e0 con 0x7f58d015e0d0 2024-03-06T07:37:04.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.596+0000 7f58cfe00640 1 -- 172.21.15.114:0/1199037510 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c402fdb0 con 0x7f58d015e0d0 2024-03-06T07:37:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 -- 172.21.15.114:0/1199037510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d015e0d0 msgr2=0x7f58d00d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 --2- 172.21.15.114:0/1199037510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d015e0d0 0x7f58d00d1a90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009870 tx=0x7f58c402ba20 comp rx=0 tx=0).stop 2024-03-06T07:37:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 -- 172.21.15.114:0/1199037510 shutdown_connections 2024-03-06T07:37:04.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 --2- 172.21.15.114:0/1199037510 >> v2:172.21.15.177:3300/0 conn(0x7f58d00d0fc0 0x7f58d0160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 --2- 172.21.15.114:0/1199037510 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d015e0d0 0x7f58d00d1a90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 -- 172.21.15.114:0/1199037510 >> 172.21.15.114:0/1199037510 conn(0x7f58d00d3ec0 msgr2=0x7f58d00d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 -- 172.21.15.114:0/1199037510 shutdown_connections 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.597+0000 7f58d8400640 1 -- 172.21.15.114:0/1199037510 wait complete. 2024-03-06T07:37:04.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.599+0000 7f58d8400640 1 Processor -- start 2024-03-06T07:37:04.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.599+0000 7f58d8400640 1 -- start start 2024-03-06T07:37:04.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d8400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d00d0fc0 0x7f58d01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d8400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d015e0d0 0x7f58d00b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d8400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d015f470 0x7f58d00c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d8400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d015fb30 con 0x7f58d015e0d0 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d8400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00d2bf0 con 0x7f58d015f470 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d8400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d00d1fd0 con 0x7f58d00d0fc0 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d015e0d0 0x7f58d00b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d015e0d0 0x7f58d00b8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56502/0 (socket says 172.21.15.114:56502) 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.600+0000 7f58d5000640 1 -- 172.21.15.114:0/1642304586 learned_addr learned my addr 172.21.15.114:0/1642304586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d6400640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d015f470 0x7f58d00c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d5a00640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d00d0fc0 0x7f58d01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d5000640 1 -- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d00d0fc0 msgr2=0x7f58d01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d5000640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d00d0fc0 0x7f58d01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d5000640 1 -- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d015f470 msgr2=0x7f58d00c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d5000640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d015f470 0x7f58d00c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d5000640 1 -- 172.21.15.114:0/1642304586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d00c89a0 con 0x7f58d015e0d0 2024-03-06T07:37:04.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58d5000640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d015e0d0 0x7f58d00b8a30 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c4e0 tx=0x7f58bc00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.601+0000 7f58cd600640 1 -- 172.21.15.114:0/1642304586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc017070 con 0x7f58d015e0d0 2024-03-06T07:37:04.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.602+0000 7f58cd600640 1 -- 172.21.15.114:0/1642304586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc0035f0 con 0x7f58d015e0d0 2024-03-06T07:37:04.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.602+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d00c8c80 con 0x7f58d015e0d0 2024-03-06T07:37:04.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.602+0000 7f58cd600640 1 -- 172.21.15.114:0/1642304586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0039e0 con 0x7f58d015e0d0 2024-03-06T07:37:04.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.602+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d00c9240 con 0x7f58d015e0d0 2024-03-06T07:37:04.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.603+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d004f330 con 0x7f58d015e0d0 2024-03-06T07:37:04.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.604+0000 7f58cd600640 1 -- 172.21.15.114:0/1642304586 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58bc01a070 con 0x7f58d015e0d0 2024-03-06T07:37:04.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.605+0000 7f58cd600640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.605+0000 7f58cd600640 1 -- 172.21.15.114:0/1642304586 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f58bc089640 con 0x7f58d015e0d0 2024-03-06T07:37:04.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.605+0000 7f58d5a00640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.606+0000 7f58d5a00640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f58c4036000 tx=0x7f58c4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.609+0000 7f58cd600640 1 -- 172.21.15.114:0/1642304586 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58bc057d10 con 0x7f58d015e0d0 2024-03-06T07:37:04.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.838+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58d008d920 con 0x7f58d015e0d0 2024-03-06T07:37:04.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.839+0000 7f58cd600640 1 -- 172.21.15.114:0/1642304586 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f58bc05bdb0 con 0x7f58d015e0d0 2024-03-06T07:37:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:04.842 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:04.842 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:04.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 msgr2=0x7f58b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f58c4036000 tx=0x7f58c4008030 comp rx=0 tx=0).stop 2024-03-06T07:37:04.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d015e0d0 msgr2=0x7f58d00b8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d015e0d0 0x7f58d00b8a30 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c4e0 tx=0x7f58bc00c9a0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 shutdown_connections 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d015f470 0x7f58d00c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b006b4d0 0x7f58b006d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d015e0d0 0x7f58d00b8a30 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 --2- 172.21.15.114:0/1642304586 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d00d0fc0 0x7f58d01566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 >> 172.21.15.114:0/1642304586 conn(0x7f58d00d3ec0 msgr2=0x7f58d00b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 shutdown_connections 2024-03-06T07:37:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:04.842+0000 7f58d8400640 1 -- 172.21.15.114:0/1642304586 wait complete. 2024-03-06T07:37:05.070 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:05.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:06.046 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:06.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:06.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.286+0000 7f8b1a600640 1 Processor -- start 2024-03-06T07:37:06.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.286+0000 7f8b1a600640 1 -- start start 2024-03-06T07:37:06.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.287+0000 7f8b1a600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 0x7f8b140d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.287+0000 7f8b1a600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b140d1e90 0x7f8b140d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b1a600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b14161960 con 0x7f8b140d28a0 2024-03-06T07:37:06.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b1a600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140cf6f0 con 0x7f8b140d1e90 2024-03-06T07:37:06.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b1a600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140cf9d0 con 0x7f8b14161580 2024-03-06T07:37:06.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b0be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b140d1e90 0x7f8b140d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b0be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8b140d1e90 0x7f8b140d2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56514/0 (socket says 172.21.15.114:56514) 2024-03-06T07:37:06.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b0be00640 1 -- 172.21.15.114:0/3299224428 learned_addr learned my addr 172.21.15.114:0/3299224428 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b0be00640 1 -- 172.21.15.114:0/3299224428 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 msgr2=0x7f8b140d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b0be00640 1 --2- 172.21.15.114:0/3299224428 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 0x7f8b140d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b0be00640 1 -- 172.21.15.114:0/3299224428 >> v1:172.21.15.177:6789/0 conn(0x7f8b140d28a0 legacy=0x7f8b140d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.288+0000 7f8b0be00640 1 -- 172.21.15.114:0/3299224428 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b140cfcb0 con 0x7f8b140d1e90 2024-03-06T07:37:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.289+0000 7f8b0be00640 1 --2- 172.21.15.114:0/3299224428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d1e90 0x7f8b140d2360 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009870 tx=0x7f8afc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32112dcd4368e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.290+0000 7f8b13e00640 1 -- 172.21.15.114:0/3299224428 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc035020 con 0x7f8b140d1e90 2024-03-06T07:37:06.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.290+0000 7f8b13e00640 1 -- 172.21.15.114:0/3299224428 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc031640 con 0x7f8b140d1e90 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.290+0000 7f8b13e00640 1 -- 172.21.15.114:0/3299224428 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc02d040 con 0x7f8b140d1e90 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.290+0000 7f8b1a600640 1 -- 172.21.15.114:0/3299224428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d1e90 msgr2=0x7f8b140d2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.290+0000 7f8b1a600640 1 --2- 172.21.15.114:0/3299224428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d1e90 0x7f8b140d2360 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009870 tx=0x7f8afc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.291+0000 7f8b1a600640 1 -- 172.21.15.114:0/3299224428 shutdown_connections 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.291+0000 7f8b1a600640 1 --2- 172.21.15.114:0/3299224428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d1e90 0x7f8b140d2360 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.291+0000 7f8b1a600640 1 --2- 172.21.15.114:0/3299224428 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 0x7f8b140d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.291+0000 7f8b1a600640 1 -- 172.21.15.114:0/3299224428 >> 172.21.15.114:0/3299224428 conn(0x7f8b140d3d80 msgr2=0x7f8b140d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.291+0000 7f8b1a600640 1 -- 172.21.15.114:0/3299224428 shutdown_connections 2024-03-06T07:37:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.291+0000 7f8b1a600640 1 -- 172.21.15.114:0/3299224428 wait complete. 2024-03-06T07:37:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.293+0000 7f8b1a600640 1 Processor -- start 2024-03-06T07:37:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.293+0000 7f8b1a600640 1 -- start start 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b1a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b140d1e90 0x7f8b140b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b1a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d28a0 0x7f8b14151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b1a600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 0x7f8b140c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b1a600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140cf3a0 con 0x7f8b140d28a0 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b1a600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140b39a0 con 0x7f8b140d1e90 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b1a600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140b4ed0 con 0x7f8b14161580 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d28a0 0x7f8b14151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d28a0 0x7f8b14151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56522/0 (socket says 172.21.15.114:56522) 2024-03-06T07:37:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b0be00640 1 -- 172.21.15.114:0/1242500488 learned_addr learned my addr 172.21.15.114:0/1242500488 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b19200640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b140d1e90 0x7f8b140b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b19c00640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 0x7f8b140c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b0be00640 1 -- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 msgr2=0x7f8b140c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b0be00640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 0x7f8b140c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b0be00640 1 -- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b140d1e90 msgr2=0x7f8b140b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.294+0000 7f8b0be00640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b140d1e90 0x7f8b140b6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b0be00640 1 -- 172.21.15.114:0/1242500488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b140b8310 con 0x7f8b140d28a0 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b19200640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b140d1e90 0x7f8b140b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b0be00640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d28a0 0x7f8b14151180 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f8afc02b930 tx=0x7f8afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b11600640 1 -- 172.21.15.114:0/1242500488 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc03ac60 con 0x7f8b140d28a0 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b11600640 1 -- 172.21.15.114:0/1242500488 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc03ae20 con 0x7f8b140d28a0 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b140b7270 con 0x7f8b140d28a0 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b11600640 1 -- 172.21.15.114:0/1242500488 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc044630 con 0x7f8b140d28a0 2024-03-06T07:37:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.295+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b140b7830 con 0x7f8b140d28a0 2024-03-06T07:37:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.296+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1404f330 con 0x7f8b140d28a0 2024-03-06T07:37:06.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.298+0000 7f8b11600640 1 -- 172.21.15.114:0/1242500488 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8afc04b060 con 0x7f8b140d28a0 2024-03-06T07:37:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.298+0000 7f8b11600640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8af006b4d0 0x7f8af006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.299+0000 7f8b11600640 1 -- 172.21.15.114:0/1242500488 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8afc0ac1c0 con 0x7f8b140d28a0 2024-03-06T07:37:06.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.299+0000 7f8b19200640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8af006b4d0 0x7f8af006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.300+0000 7f8b19200640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8af006b4d0 0x7f8af006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8b0400a870 tx=0x7f8b04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:06.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.304+0000 7f8b11600640 1 -- 172.21.15.114:0/1242500488 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8afc03f070 con 0x7f8b140d28a0 2024-03-06T07:37:06.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.539+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b140d3b60 con 0x7f8b140d28a0 2024-03-06T07:37:06.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.540+0000 7f8b11600640 1 -- 172.21.15.114:0/1242500488 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f8afc02f300 con 0x7f8b140d28a0 2024-03-06T07:37:06.542 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:06.542 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:06.543 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8af006b4d0 msgr2=0x7f8af006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8af006b4d0 0x7f8af006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8b0400a870 tx=0x7f8b04008040 comp rx=0 tx=0).stop 2024-03-06T07:37:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d28a0 msgr2=0x7f8b14151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d28a0 0x7f8b14151180 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f8afc02b930 tx=0x7f8afc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 shutdown_connections 2024-03-06T07:37:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8b14161580 0x7f8b140c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8af006b4d0 0x7f8af006d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b140d28a0 0x7f8b14151180 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 --2- 172.21.15.114:0/1242500488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b140d1e90 0x7f8b140b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 >> 172.21.15.114:0/1242500488 conn(0x7f8b140d3d80 msgr2=0x7f8b140b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:06.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.543+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 shutdown_connections 2024-03-06T07:37:06.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:06.544+0000 7f8b1a600640 1 -- 172.21.15.114:0/1242500488 wait complete. 2024-03-06T07:37:06.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:06.766 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:07.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:07.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.980+0000 7fe60a000640 1 Processor -- start 2024-03-06T07:37:07.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.980+0000 7fe60a000640 1 -- start start 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.982+0000 7fe60a000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe60415f9d0 0x7fe60415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.982+0000 7fe60a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.982+0000 7fe60a000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60406cef0 con 0x7fe604144c00 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.982+0000 7fe60a000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60406d1d0 con 0x7fe60415f9d0 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.982+0000 7fe60a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60406d4b0 con 0x7fe604070b10 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.982+0000 7fe5faa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe60415f9d0 0x7fe60415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe5faa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe60415f9d0 0x7fe60415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48492/0 (socket says 172.21.15.114:48492) 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe5faa00640 1 -- 172.21.15.114:0/3804128825 learned_addr learned my addr 172.21.15.114:0/3804128825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe603400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe604144c00 0x7fe60406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59952/0 (socket says 172.21.15.114:59952) 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe603e00640 1 --2- 172.21.15.114:0/3804128825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe5faa00640 1 -- 172.21.15.114:0/3804128825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 msgr2=0x7fe604070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe5faa00640 1 --2- 172.21.15.114:0/3804128825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe5faa00640 1 -- 172.21.15.114:0/3804128825 >> v1:172.21.15.114:6789/0 conn(0x7fe604144c00 legacy=0x7fe60406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.983+0000 7fe5faa00640 1 -- 172.21.15.114:0/3804128825 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60406d790 con 0x7fe60415f9d0 2024-03-06T07:37:07.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.984+0000 7fe5faa00640 1 --2- 172.21.15.114:0/3804128825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe60415f9d0 0x7fe60415fe60 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009870 tx=0x7fe5ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deadd012cfbb5365 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.985+0000 7fe602a00640 1 -- 172.21.15.114:0/3804128825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec034070 con 0x7fe60415f9d0 2024-03-06T07:37:07.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.985+0000 7fe602a00640 1 -- 172.21.15.114:0/3804128825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec030840 con 0x7fe60415f9d0 2024-03-06T07:37:07.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.985+0000 7fe602a00640 1 -- 172.21.15.114:0/3804128825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec030c10 con 0x7fe60415f9d0 2024-03-06T07:37:07.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.985+0000 7fe60a000640 1 -- 172.21.15.114:0/3804128825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe60415f9d0 msgr2=0x7fe60415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.985+0000 7fe60a000640 1 --2- 172.21.15.114:0/3804128825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe60415f9d0 0x7fe60415fe60 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009870 tx=0x7fe5ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:37:07.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.986+0000 7fe60a000640 1 -- 172.21.15.114:0/3804128825 shutdown_connections 2024-03-06T07:37:07.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.986+0000 7fe60a000640 1 --2- 172.21.15.114:0/3804128825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.986+0000 7fe60a000640 1 --2- 172.21.15.114:0/3804128825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe60415f9d0 0x7fe60415fe60 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.986+0000 7fe60a000640 1 -- 172.21.15.114:0/3804128825 >> 172.21.15.114:0/3804128825 conn(0x7fe6040b1730 msgr2=0x7fe6040b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:07.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.986+0000 7fe60a000640 1 -- 172.21.15.114:0/3804128825 shutdown_connections 2024-03-06T07:37:07.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.986+0000 7fe60a000640 1 -- 172.21.15.114:0/3804128825 wait complete. 2024-03-06T07:37:07.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.987+0000 7fe60a000640 1 Processor -- start 2024-03-06T07:37:07.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.987+0000 7fe60a000640 1 -- start start 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.988+0000 7fe60a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.988+0000 7fe60a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe604144c00 0x7fe6041423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.988+0000 7fe60a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60415f9d0 0x7fe604139c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.988+0000 7fe60a000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60406ccd0 con 0x7fe604144c00 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.988+0000 7fe60a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60406cb50 con 0x7fe60415f9d0 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.988+0000 7fe60a000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60406df90 con 0x7fe604070b10 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe604144c00 0x7fe6041423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe604144c00 0x7fe6041423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48500/0 (socket says 172.21.15.114:48500) 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 -- 172.21.15.114:0/1996053736 learned_addr learned my addr 172.21.15.114:0/1996053736 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe603400640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe603e00640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60415f9d0 0x7fe604139c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 -- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 msgr2=0x7fe604143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 -- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60415f9d0 msgr2=0x7fe604139c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60415f9d0 0x7fe604139c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.989+0000 7fe5faa00640 1 -- 172.21.15.114:0/1996053736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6041f7a90 con 0x7fe604144c00 2024-03-06T07:37:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.990+0000 7fe5faa00640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe604144c00 0x7fe6041423c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec000c00 tx=0x7fe5ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.990+0000 7fe5fbe00640 1 -- 172.21.15.114:0/1996053736 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec048070 con 0x7fe604144c00 2024-03-06T07:37:07.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.990+0000 7fe5fbe00640 1 -- 172.21.15.114:0/1996053736 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec030640 con 0x7fe604144c00 2024-03-06T07:37:07.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.990+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6041f7d70 con 0x7fe604144c00 2024-03-06T07:37:07.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.990+0000 7fe5fbe00640 1 -- 172.21.15.114:0/1996053736 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec034040 con 0x7fe604144c00 2024-03-06T07:37:07.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.990+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6041f8330 con 0x7fe604144c00 2024-03-06T07:37:07.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.991+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60404f330 con 0x7fe604144c00 2024-03-06T07:37:07.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.992+0000 7fe5fbe00640 1 -- 172.21.15.114:0/1996053736 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5ec04b070 con 0x7fe604144c00 2024-03-06T07:37:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.993+0000 7fe5fbe00640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.993+0000 7fe5fbe00640 1 -- 172.21.15.114:0/1996053736 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe5ec0ac1d0 con 0x7fe604144c00 2024-03-06T07:37:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.993+0000 7fe603400640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.994+0000 7fe603400640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5e006b4d0 0x7fe5e006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4000f60 tx=0x7fe5f4007cc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:07.998+0000 7fe5fbe00640 1 -- 172.21.15.114:0/1996053736 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5ec07b120 con 0x7fe604144c00 2024-03-06T07:37:08.048 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:08.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.235+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe60415f7b0 con 0x7fe604144c00 2024-03-06T07:37:08.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.236+0000 7fe5fbe00640 1 -- 172.21.15.114:0/1996053736 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fe5ec030810 con 0x7fe604144c00 2024-03-06T07:37:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:08.238 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:08.239 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5e006b4d0 msgr2=0x7fe5e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5e006b4d0 0x7fe5e006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4000f60 tx=0x7fe5f4007cc0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe604144c00 msgr2=0x7fe6041423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe604144c00 0x7fe6041423c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec000c00 tx=0x7fe5ec004730 comp rx=0 tx=0).stop 2024-03-06T07:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 shutdown_connections 2024-03-06T07:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60415f9d0 0x7fe604139c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe5e006b4d0 0x7fe5e006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe604144c00 0x7fe6041423c0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 --2- 172.21.15.114:0/1996053736 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe604070b10 0x7fe604143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 >> 172.21.15.114:0/1996053736 conn(0x7fe6040b1730 msgr2=0x7fe60415e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.239+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 shutdown_connections 2024-03-06T07:37:08.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:08.240+0000 7fe60a000640 1 -- 172.21.15.114:0/1996053736 wait complete. 2024-03-06T07:37:08.460 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:08.461 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:09.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:09.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.660+0000 7fbb5bc00640 1 Processor -- start 2024-03-06T07:37:09.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.661+0000 7fbb5bc00640 1 -- start start 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.662+0000 7fbb5bc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbb5415acb0 0x7fbb540b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.662+0000 7fbb5bc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb540b4680 0x7fbb5406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.662+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.662+0000 7fbb5bc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5415b090 con 0x7fbb540b4680 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.662+0000 7fbb5bc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb540b2a30 con 0x7fbb5415acb0 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.662+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540b4360 con 0x7fbb54072340 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb540b4680 0x7fbb5406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb540b4680 0x7fbb5406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48506/0 (socket says 172.21.15.114:48506) 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 -- 172.21.15.114:0/3976825846 learned_addr learned my addr 172.21.15.114:0/3976825846 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb59200640 1 --2- 172.21.15.114:0/3976825846 >> v2:172.21.15.177:3300/0 conn(0x7fbb5415acb0 0x7fbb540b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb59c00640 1 --2- 172.21.15.114:0/3976825846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 -- 172.21.15.114:0/3976825846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 msgr2=0x7fbb5406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:09.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 --2- 172.21.15.114:0/3976825846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5406ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 -- 172.21.15.114:0/3976825846 >> v2:172.21.15.177:3300/0 conn(0x7fbb5415acb0 msgr2=0x7fbb540b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:09.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 --2- 172.21.15.114:0/3976825846 >> v2:172.21.15.177:3300/0 conn(0x7fbb5415acb0 0x7fbb540b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.663+0000 7fbb4be00640 1 -- 172.21.15.114:0/3976825846 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb540b2cf0 con 0x7fbb540b4680 2024-03-06T07:37:09.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.664+0000 7fbb4be00640 1 --2- 172.21.15.114:0/3976825846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb540b4680 0x7fbb5406e4f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fbb540b27f0 tx=0x7fbb3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5747d77efe46a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb53e00640 1 -- 172.21.15.114:0/3976825846 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c035020 con 0x7fbb540b4680 2024-03-06T07:37:09.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb53e00640 1 -- 172.21.15.114:0/3976825846 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c031870 con 0x7fbb540b4680 2024-03-06T07:37:09.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb53e00640 1 -- 172.21.15.114:0/3976825846 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c031c40 con 0x7fbb540b4680 2024-03-06T07:37:09.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3976825846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb540b4680 msgr2=0x7fbb5406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3976825846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb540b4680 0x7fbb5406e4f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fbb540b27f0 tx=0x7fbb3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:37:09.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3976825846 shutdown_connections 2024-03-06T07:37:09.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3976825846 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5406ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3976825846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb540b4680 0x7fbb5406e4f0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3976825846 >> v2:172.21.15.177:3300/0 conn(0x7fbb5415acb0 0x7fbb540b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3976825846 >> 172.21.15.114:0/3976825846 conn(0x7fbb540b1730 msgr2=0x7fbb540b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:09.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3976825846 shutdown_connections 2024-03-06T07:37:09.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.665+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3976825846 wait complete. 2024-03-06T07:37:09.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.667+0000 7fbb5bc00640 1 Processor -- start 2024-03-06T07:37:09.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.667+0000 7fbb5bc00640 1 -- start start 2024-03-06T07:37:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5414e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb540b4680 0x7fbb54159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb5415acb0 0x7fbb540b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540b3d10 con 0x7fbb5415acb0 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540b32b0 con 0x7fbb540b4680 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540b4010 con 0x7fbb54072340 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb59c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb5415acb0 0x7fbb540b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb59c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb5415acb0 0x7fbb540b0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48520/0 (socket says 172.21.15.114:48520) 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.668+0000 7fbb59c00640 1 -- 172.21.15.114:0/3512258936 learned_addr learned my addr 172.21.15.114:0/3512258936 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb4be00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb540b4680 0x7fbb54159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb59c00640 1 -- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 msgr2=0x7fbb5414e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb59200640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5414e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb59c00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5414e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb59c00640 1 -- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb540b4680 msgr2=0x7fbb54159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb59c00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb540b4680 0x7fbb54159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb59c00640 1 -- 172.21.15.114:0/3512258936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb54159960 con 0x7fbb5415acb0 2024-03-06T07:37:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb4be00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb540b4680 0x7fbb54159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb59c00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb5415acb0 0x7fbb540b0570 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00d800 tx=0x7fbb4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb51600640 1 -- 172.21.15.114:0/3512258936 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c018020 con 0x7fbb5415acb0 2024-03-06T07:37:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb51600640 1 -- 172.21.15.114:0/3512258936 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4c0096c0 con 0x7fbb5415acb0 2024-03-06T07:37:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5415cfd0 con 0x7fbb5415acb0 2024-03-06T07:37:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.669+0000 7fbb51600640 1 -- 172.21.15.114:0/3512258936 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c009a90 con 0x7fbb5415acb0 2024-03-06T07:37:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.670+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5415d590 con 0x7fbb5415acb0 2024-03-06T07:37:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.671+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5415e4a0 con 0x7fbb5415acb0 2024-03-06T07:37:09.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.672+0000 7fbb51600640 1 -- 172.21.15.114:0/3512258936 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb4c01b020 con 0x7fbb5415acb0 2024-03-06T07:37:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.672+0000 7fbb51600640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb3006b4d0 0x7fbb3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.673+0000 7fbb51600640 1 -- 172.21.15.114:0/3512258936 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbb4c08a8a0 con 0x7fbb5415acb0 2024-03-06T07:37:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.673+0000 7fbb59200640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb3006b4d0 0x7fbb3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.674+0000 7fbb59200640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb3006b4d0 0x7fbb3006d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbb5415a6c0 tx=0x7fbb4400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.677+0000 7fbb51600640 1 -- 172.21.15.114:0/3512258936 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb5415e4a0 con 0x7fbb5415acb0 2024-03-06T07:37:09.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.907+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb5404f330 con 0x7fbb5415acb0 2024-03-06T07:37:09.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.908+0000 7fbb51600640 1 -- 172.21.15.114:0/3512258936 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fbb4c022020 con 0x7fbb5415acb0 2024-03-06T07:37:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:09.910 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:09.910 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:09.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb3006b4d0 msgr2=0x7fbb3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb3006b4d0 0x7fbb3006d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbb5415a6c0 tx=0x7fbb4400b040 comp rx=0 tx=0).stop 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb5415acb0 msgr2=0x7fbb540b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb5415acb0 0x7fbb540b0570 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00d800 tx=0x7fbb4c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 shutdown_connections 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbb3006b4d0 0x7fbb3006d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb5415acb0 0x7fbb540b0570 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb540b4680 0x7fbb54159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 --2- 172.21.15.114:0/3512258936 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb54072340 0x7fbb5414e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.910+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 >> 172.21.15.114:0/3512258936 conn(0x7fbb540b1730 msgr2=0x7fbb5406e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.911+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 shutdown_connections 2024-03-06T07:37:09.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:09.911+0000 7fbb5bc00640 1 -- 172.21.15.114:0/3512258936 wait complete. 2024-03-06T07:37:10.050 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:10.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:10.132 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:11.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:11.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.344+0000 7f7e9de00640 1 Processor -- start 2024-03-06T07:37:11.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.344+0000 7f7e9de00640 1 -- start start 2024-03-06T07:37:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.346+0000 7f7e9de00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7e9815e0d0 0x7f7e980d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.346+0000 7f7e9de00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e980cea60 con 0x7f7e9815f470 2024-03-06T07:37:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.346+0000 7f7e9de00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e980d2350 con 0x7f7e980d0fc0 2024-03-06T07:37:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.346+0000 7f7e9de00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e980d2f60 con 0x7f7e9815e0d0 2024-03-06T07:37:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.346+0000 7f7e96000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7e9815f470 0x7f7e980d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59968/0 (socket says 172.21.15.114:59968) 2024-03-06T07:37:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.346+0000 7f7e96000640 1 -- 172.21.15.114:0/3053549450 learned_addr learned my addr 172.21.15.114:0/3053549450 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:11.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e96a00640 1 --2- 172.21.15.114:0/3053549450 >> v2:172.21.15.177:3300/0 conn(0x7f7e9815e0d0 0x7f7e980d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713839321 0 0) 0x7f7e980cea60 con 0x7f7e9815f470 2024-03-06T07:37:11.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e7c003520 con 0x7f7e9815f470 2024-03-06T07:37:11.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1073938193 0 0) 0x7f7e7c003520 con 0x7f7e9815f470 2024-03-06T07:37:11.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 >> v2:172.21.15.177:3300/0 conn(0x7f7e9815e0d0 msgr2=0x7f7e980d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 --2- 172.21.15.114:0/3053549450 >> v2:172.21.15.177:3300/0 conn(0x7f7e9815e0d0 0x7f7e980d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 >> v1:172.21.15.177:6789/0 conn(0x7f7e980d0fc0 legacy=0x7f7e98160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980d2630 con 0x7f7e9815f470 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.347+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7e84001670 con 0x7f7e9815f470 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 >> v1:172.21.15.114:6789/0 conn(0x7f7e9815f470 legacy=0x7f7e980d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e95600640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e7c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e95600640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c004800 0x7f7e7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e95600640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e7c0051c0 0x7f7e7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c00c070 con 0x7f7e7c003a00 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c00c220 con 0x7f7e7c004800 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c00c420 con 0x7f7e7c0051c0 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96a00640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e7c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96000640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c004800 0x7f7e7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96a00640 1 -- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e7c0051c0 msgr2=0x7f7e7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e97400640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e7c0051c0 0x7f7e7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96a00640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e7c0051c0 0x7f7e7c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96a00640 1 -- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c004800 msgr2=0x7f7e7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:11.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96a00640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c004800 0x7f7e7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96a00640 1 -- 172.21.15.114:0/3053549450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c003520 con 0x7f7e7c003a00 2024-03-06T07:37:11.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e96000640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e7c004800 0x7f7e7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:11.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.348+0000 7f7e97400640 1 -- 172.21.15.114:0/3053549450 reap_dead start 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.350+0000 7f7e96a00640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e7c003e20 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009fc0 tx=0x7f7e7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565da01de00101dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.350+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90039070 con 0x7f7e7c003a00 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.350+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9002cbf0 con 0x7f7e7c003a00 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.350+0000 7f7e95600640 1 -- 172.21.15.114:0/3053549450 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90009c10 con 0x7f7e7c003a00 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.350+0000 7f7e9de00640 1 -- 172.21.15.114:0/3053549450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 msgr2=0x7f7e7c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.350+0000 7f7e9de00640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e7c003e20 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009fc0 tx=0x7f7e7c005830 comp rx=0 tx=0).stop 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.351+0000 7f7e9de00640 1 -- 172.21.15.114:0/3053549450 shutdown_connections 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.351+0000 7f7e9de00640 1 --2- 172.21.15.114:0/3053549450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e7c003e20 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.351+0000 7f7e9de00640 1 -- 172.21.15.114:0/3053549450 >> 172.21.15.114:0/3053549450 conn(0x7f7e980d3ec0 msgr2=0x7f7e980d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.351+0000 7f7e9de00640 1 -- 172.21.15.114:0/3053549450 shutdown_connections 2024-03-06T07:37:11.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.351+0000 7f7e9de00640 1 -- 172.21.15.114:0/3053549450 wait complete. 2024-03-06T07:37:11.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.352+0000 7f7e9de00640 1 Processor -- start 2024-03-06T07:37:11.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.352+0000 7f7e9de00640 1 -- start start 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.353+0000 7f7e9de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e981520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.353+0000 7f7e9de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e980caf90 0x7f7e98162450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.353+0000 7f7e9de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e98162990 0x7f7e980b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.353+0000 7f7e9de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9815fb30 con 0x7f7e7c003a00 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.353+0000 7f7e9de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e980d2bf0 con 0x7f7e98162990 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.353+0000 7f7e9de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9815e4b0 con 0x7f7e980caf90 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e981520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e981520a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48530/0 (socket says 172.21.15.114:48530) 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 -- 172.21.15.114:0/917495207 learned_addr learned my addr 172.21.15.114:0/917495207 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e97400640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e98162990 0x7f7e980b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 -- 172.21.15.114:0/917495207 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e980caf90 msgr2=0x7f7e98162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e980caf90 0x7f7e98162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 -- 172.21.15.114:0/917495207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e98162990 msgr2=0x7f7e980b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e98162990 0x7f7e980b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 -- 172.21.15.114:0/917495207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980c81b0 con 0x7f7e7c003a00 2024-03-06T07:37:11.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.354+0000 7f7e96a00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e981520a0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7e90037040 tx=0x7f7e90009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.355+0000 7f7e8f400640 1 -- 172.21.15.114:0/917495207 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90039050 con 0x7f7e7c003a00 2024-03-06T07:37:11.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.355+0000 7f7e8f400640 1 -- 172.21.15.114:0/917495207 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9003a040 con 0x7f7e7c003a00 2024-03-06T07:37:11.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.355+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e980c8490 con 0x7f7e7c003a00 2024-03-06T07:37:11.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.355+0000 7f7e8f400640 1 -- 172.21.15.114:0/917495207 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90009c10 con 0x7f7e7c003a00 2024-03-06T07:37:11.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.355+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e980c8a50 con 0x7f7e7c003a00 2024-03-06T07:37:11.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.356+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9804f330 con 0x7f7e7c003a00 2024-03-06T07:37:11.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.357+0000 7f7e8f400640 1 -- 172.21.15.114:0/917495207 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e90047020 con 0x7f7e7c003a00 2024-03-06T07:37:11.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.358+0000 7f7e8f400640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e7806b4d0 0x7f7e7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:11.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.358+0000 7f7e8f400640 1 -- 172.21.15.114:0/917495207 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e900ac4d0 con 0x7f7e7c003a00 2024-03-06T07:37:11.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.358+0000 7f7e96000640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e7806b4d0 0x7f7e7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:11.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.359+0000 7f7e96000640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e7806b4d0 0x7f7e7806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000c00 tx=0x7f7e84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:11.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.363+0000 7f7e8f400640 1 -- 172.21.15.114:0/917495207 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e9004e020 con 0x7f7e7c003a00 2024-03-06T07:37:11.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.596+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e9808d920 con 0x7f7e7c003a00 2024-03-06T07:37:11.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.597+0000 7f7e8f400640 1 -- 172.21.15.114:0/917495207 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f7e9007b270 con 0x7f7e7c003a00 2024-03-06T07:37:11.599 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:11.599 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:11.599 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e7806b4d0 msgr2=0x7f7e7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e7806b4d0 0x7f7e7806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000c00 tx=0x7f7e84009040 comp rx=0 tx=0).stop 2024-03-06T07:37:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 msgr2=0x7f7e981520a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e981520a0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7e90037040 tx=0x7f7e90009750 comp rx=0 tx=0).stop 2024-03-06T07:37:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 shutdown_connections 2024-03-06T07:37:11.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e98162990 0x7f7e980b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e980caf90 0x7f7e98162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7e7806b4d0 0x7f7e7806d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 --2- 172.21.15.114:0/917495207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7c003a00 0x7f7e981520a0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.599+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 >> 172.21.15.114:0/917495207 conn(0x7f7e980d3ec0 msgr2=0x7f7e980d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.600+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 shutdown_connections 2024-03-06T07:37:11.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:11.600+0000 7f7e9de00640 1 -- 172.21.15.114:0/917495207 wait complete. 2024-03-06T07:37:11.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:11.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:12.052 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:12.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:13.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.031+0000 7f61d6800640 1 Processor -- start 2024-03-06T07:37:13.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.032+0000 7f61d6800640 1 -- start start 2024-03-06T07:37:13.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61d6800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f61d00cec00 0x7f61d0168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61d6800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d006cc80 con 0x7f61d015e540 2024-03-06T07:37:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61d6800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d006cf60 con 0x7f61d015e160 2024-03-06T07:37:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61d6800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d006d240 con 0x7f61d00cec00 2024-03-06T07:37:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61c7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f61d015e540 0x7f61d00ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59984/0 (socket says 172.21.15.114:59984) 2024-03-06T07:37:13.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61c7e00640 1 -- 172.21.15.114:0/2086134766 learned_addr learned my addr 172.21.15.114:0/2086134766 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61cfe00640 1 --2- 172.21.15.114:0/2086134766 >> v2:172.21.15.177:3300/0 conn(0x7f61d00cec00 0x7f61d0168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786722558 0 0) 0x7f61d006cc80 con 0x7f61d015e540 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.034+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b4003520 con 0x7f61d015e540 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4265360951 0 0) 0x7f61b4003520 con 0x7f61d015e540 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 >> v2:172.21.15.177:3300/0 conn(0x7f61d00cec00 msgr2=0x7f61d0168ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 --2- 172.21.15.114:0/2086134766 >> v2:172.21.15.177:3300/0 conn(0x7f61d00cec00 0x7f61d0168ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 >> v1:172.21.15.177:6789/0 conn(0x7f61d015e160 legacy=0x7f61d00b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00cf170 con 0x7f61d015e540 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f61b8001670 con 0x7f61d015e540 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 >> v1:172.21.15.114:6789/0 conn(0x7f61d015e540 legacy=0x7f61d00ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61b4003a00 0x7f61b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4004800 0x7f61b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b40051c0 0x7f61b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d006d240 con 0x7f61b4004800 2024-03-06T07:37:13.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b400c0c0 con 0x7f61b4003a00 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b400c2c0 con 0x7f61b40051c0 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.035+0000 7f61c7e00640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4004800 0x7f61b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.036+0000 7f61cf400640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61b4003a00 0x7f61b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.036+0000 7f61c7e00640 1 -- 172.21.15.114:0/2086134766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b40051c0 msgr2=0x7f61b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.036+0000 7f61c7e00640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b40051c0 0x7f61b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.036+0000 7f61c7e00640 1 -- 172.21.15.114:0/2086134766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61b4003a00 msgr2=0x7f61b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.036+0000 7f61c7e00640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61b4003a00 0x7f61b4003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.036+0000 7f61cfe00640 1 -- 172.21.15.114:0/2086134766 reap_dead start 2024-03-06T07:37:13.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.036+0000 7f61c7e00640 1 -- 172.21.15.114:0/2086134766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4003520 con 0x7f61b4004800 2024-03-06T07:37:13.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.037+0000 7f61c7e00640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4004800 0x7f61b4004c80 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f61b800c190 tx=0x7f61b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e375f2911d90c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.037+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8007c60 con 0x7f61b4004800 2024-03-06T07:37:13.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.037+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b8030070 con 0x7f61b4004800 2024-03-06T07:37:13.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.037+0000 7f61cea00640 1 -- 172.21.15.114:0/2086134766 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8005d90 con 0x7f61b4004800 2024-03-06T07:37:13.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.038+0000 7f61d6800640 1 -- 172.21.15.114:0/2086134766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4004800 msgr2=0x7f61b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.038+0000 7f61d6800640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4004800 0x7f61b4004c80 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f61b800c190 tx=0x7f61b4005830 comp rx=0 tx=0).stop 2024-03-06T07:37:13.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.038+0000 7f61d6800640 1 -- 172.21.15.114:0/2086134766 shutdown_connections 2024-03-06T07:37:13.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.038+0000 7f61d6800640 1 --2- 172.21.15.114:0/2086134766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4004800 0x7f61b4004c80 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.038+0000 7f61d6800640 1 -- 172.21.15.114:0/2086134766 >> 172.21.15.114:0/2086134766 conn(0x7f61d00b3800 msgr2=0x7f61d00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:13.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.038+0000 7f61d6800640 1 -- 172.21.15.114:0/2086134766 shutdown_connections 2024-03-06T07:37:13.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.038+0000 7f61d6800640 1 -- 172.21.15.114:0/2086134766 wait complete. 2024-03-06T07:37:13.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.040+0000 7f61d6800640 1 Processor -- start 2024-03-06T07:37:13.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.040+0000 7f61d6800640 1 -- start start 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61d6800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b4004800 0x7f61d00b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61d6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d01f7810 0x7f61d00cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61d6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61d00c7770 0x7f61d00c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61d6800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00ce880 con 0x7f61d00c7770 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61d6800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00ce700 con 0x7f61d01f7810 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61d6800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00cea00 con 0x7f61b4004800 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61d00c7770 0x7f61d00c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61d00c7770 0x7f61d00c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48536/0 (socket says 172.21.15.114:48536) 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 -- 172.21.15.114:0/3589756045 learned_addr learned my addr 172.21.15.114:0/3589756045 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cf400640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b4004800 0x7f61d00b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61c7e00640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d01f7810 0x7f61d00cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 -- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b4004800 msgr2=0x7f61d00b75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b4004800 0x7f61d00b75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 -- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d01f7810 msgr2=0x7f61d00cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d01f7810 0x7f61d00cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.041+0000 7f61cfe00640 1 -- 172.21.15.114:0/3589756045 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d01f7d90 con 0x7f61d00c7770 2024-03-06T07:37:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.042+0000 7f61cfe00640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61d00c7770 0x7f61d00c7bd0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f61d00cfd90 tx=0x7f61c800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.042+0000 7f61c7400640 1 -- 172.21.15.114:0/3589756045 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8018020 con 0x7f61d00c7770 2024-03-06T07:37:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.042+0000 7f61c7400640 1 -- 172.21.15.114:0/3589756045 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c800bcb0 con 0x7f61d00c7770 2024-03-06T07:37:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.042+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d01f8070 con 0x7f61d00c7770 2024-03-06T07:37:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.042+0000 7f61c7400640 1 -- 172.21.15.114:0/3589756045 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c800be60 con 0x7f61d00c7770 2024-03-06T07:37:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.042+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d01f8630 con 0x7f61d00c7770 2024-03-06T07:37:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.043+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d00c8110 con 0x7f61d00c7770 2024-03-06T07:37:13.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.044+0000 7f61c7400640 1 -- 172.21.15.114:0/3589756045 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61c801b020 con 0x7f61d00c7770 2024-03-06T07:37:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.045+0000 7f61c7400640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f61ac06b4d0 0x7f61ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.045+0000 7f61c7400640 1 -- 172.21.15.114:0/3589756045 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f61c808aa50 con 0x7f61d00c7770 2024-03-06T07:37:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.046+0000 7f61cf400640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f61ac06b4d0 0x7f61ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.046+0000 7f61cf400640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f61ac06b4d0 0x7f61ac06d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f61d00686c0 tx=0x7f61c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:13.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.054+0000 7f61c7400640 1 -- 172.21.15.114:0/3589756045 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61c8022020 con 0x7f61d00c7770 2024-03-06T07:37:13.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.297+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61d004f330 con 0x7f61d00c7770 2024-03-06T07:37:13.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.298+0000 7f61c7400640 1 -- 172.21.15.114:0/3589756045 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f61c8059880 con 0x7f61d00c7770 2024-03-06T07:37:13.300 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:13.300 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:13.300 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f61ac06b4d0 msgr2=0x7f61ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f61ac06b4d0 0x7f61ac06d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f61d00686c0 tx=0x7f61c0009040 comp rx=0 tx=0).stop 2024-03-06T07:37:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61d00c7770 msgr2=0x7f61d00c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61d00c7770 0x7f61d00c7bd0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f61d00cfd90 tx=0x7f61c800b480 comp rx=0 tx=0).stop 2024-03-06T07:37:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 shutdown_connections 2024-03-06T07:37:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f61ac06b4d0 0x7f61ac06d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61d00c7770 0x7f61d00c7bd0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d01f7810 0x7f61d00cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 --2- 172.21.15.114:0/3589756045 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f61b4004800 0x7f61d00b75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:13.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 >> 172.21.15.114:0/3589756045 conn(0x7f61d00b3800 msgr2=0x7f61d006c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:13.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 shutdown_connections 2024-03-06T07:37:13.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:13.301+0000 7f61d6800640 1 -- 172.21.15.114:0/3589756045 wait complete. 2024-03-06T07:37:13.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:13.523 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:14.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:14.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.734+0000 7fae2fe00640 1 Processor -- start 2024-03-06T07:37:14.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.734+0000 7fae2fe00640 1 -- start start 2024-03-06T07:37:14.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.735+0000 7fae2fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae28072680 0x7fae280c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.735+0000 7fae2fe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fae280cf3e0 0x7fae280ca400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.735+0000 7fae2fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280ca940 con 0x7fae28144660 2024-03-06T07:37:14.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.735+0000 7fae2fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae280cabf0 con 0x7fae28072680 2024-03-06T07:37:14.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.735+0000 7fae2fe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae280ce900 con 0x7fae280cf3e0 2024-03-06T07:37:14.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2ca00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae28072680 0x7fae280c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2d400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fae28144660 0x7fae28071f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59988/0 (socket says 172.21.15.114:59988) 2024-03-06T07:37:14.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2d400640 1 -- 172.21.15.114:0/2107934111 learned_addr learned my addr 172.21.15.114:0/2107934111 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:14.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2de00640 1 --2- 172.21.15.114:0/2107934111 >> v2:172.21.15.177:3300/0 conn(0x7fae280cf3e0 0x7fae280ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2ca00640 1 -- 172.21.15.114:0/2107934111 >> v2:172.21.15.177:3300/0 conn(0x7fae280cf3e0 msgr2=0x7fae280ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:14.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2ca00640 1 --2- 172.21.15.114:0/2107934111 >> v2:172.21.15.177:3300/0 conn(0x7fae280cf3e0 0x7fae280ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2ca00640 1 -- 172.21.15.114:0/2107934111 >> v1:172.21.15.114:6789/0 conn(0x7fae28144660 legacy=0x7fae28071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae27e00640 1 -- 172.21.15.114:0/2107934111 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704975730 0 0) 0x7fae280ca940 con 0x7fae28144660 2024-03-06T07:37:14.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2ca00640 1 -- 172.21.15.114:0/2107934111 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280cebe0 con 0x7fae28072680 2024-03-06T07:37:14.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.736+0000 7fae2de00640 1 --2- 172.21.15.114:0/2107934111 >> v2:172.21.15.177:3300/0 conn(0x7fae280cf3e0 0x7fae280ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:14.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.738+0000 7fae2ca00640 1 --2- 172.21.15.114:0/2107934111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280c9ec0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fae2806bbe0 tx=0x7fae1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bb5ae46faef570 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.738+0000 7fae27e00640 1 -- 172.21.15.114:0/2107934111 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14035020 con 0x7fae28072680 2024-03-06T07:37:14.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.738+0000 7fae27e00640 1 -- 172.21.15.114:0/2107934111 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae14031830 con 0x7fae28072680 2024-03-06T07:37:14.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.738+0000 7fae27e00640 1 -- 172.21.15.114:0/2107934111 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14031c00 con 0x7fae28072680 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.738+0000 7fae2fe00640 1 -- 172.21.15.114:0/2107934111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 msgr2=0x7fae280c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.738+0000 7fae2fe00640 1 --2- 172.21.15.114:0/2107934111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280c9ec0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fae2806bbe0 tx=0x7fae1402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.739+0000 7fae2fe00640 1 -- 172.21.15.114:0/2107934111 shutdown_connections 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.739+0000 7fae2fe00640 1 --2- 172.21.15.114:0/2107934111 >> v2:172.21.15.177:3300/0 conn(0x7fae280cf3e0 0x7fae280ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.739+0000 7fae2fe00640 1 --2- 172.21.15.114:0/2107934111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280c9ec0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.739+0000 7fae2fe00640 1 -- 172.21.15.114:0/2107934111 >> 172.21.15.114:0/2107934111 conn(0x7fae280d0b90 msgr2=0x7fae280d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.739+0000 7fae2fe00640 1 -- 172.21.15.114:0/2107934111 shutdown_connections 2024-03-06T07:37:14.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.739+0000 7fae2fe00640 1 -- 172.21.15.114:0/2107934111 wait complete. 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.740+0000 7fae2fe00640 1 Processor -- start 2024-03-06T07:37:14.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.741+0000 7fae2fe00640 1 -- start start 2024-03-06T07:37:14.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.741+0000 7fae2fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.741+0000 7fae2fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae280cf3e0 0x7fae280b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.741+0000 7fae2fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae28144660 0x7fae280cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.741+0000 7fae2fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae280cc0f0 con 0x7fae28072680 2024-03-06T07:37:14.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.741+0000 7fae2fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae28144a40 con 0x7fae280cf3e0 2024-03-06T07:37:14.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.741+0000 7fae2fe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae280cc270 con 0x7fae28144660 2024-03-06T07:37:14.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48544/0 (socket says 172.21.15.114:48544) 2024-03-06T07:37:14.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 -- 172.21.15.114:0/3284751106 learned_addr learned my addr 172.21.15.114:0/3284751106 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:14.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 -- 172.21.15.114:0/3284751106 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae28144660 msgr2=0x7fae280cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae28144660 0x7fae280cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 -- 172.21.15.114:0/3284751106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae280cf3e0 msgr2=0x7fae280b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2ca00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae280cf3e0 0x7fae280b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae280cf3e0 0x7fae280b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 -- 172.21.15.114:0/3284751106 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280b2650 con 0x7fae28072680 2024-03-06T07:37:14.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.742+0000 7fae2d400640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280b2110 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00c430 tx=0x7fae1c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.743+0000 7fae25600640 1 -- 172.21.15.114:0/3284751106 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c0074e0 con 0x7fae28072680 2024-03-06T07:37:14.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.743+0000 7fae25600640 1 -- 172.21.15.114:0/3284751106 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c0047e0 con 0x7fae28072680 2024-03-06T07:37:14.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.743+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae280a97c0 con 0x7fae28072680 2024-03-06T07:37:14.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.743+0000 7fae25600640 1 -- 172.21.15.114:0/3284751106 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c004bd0 con 0x7fae28072680 2024-03-06T07:37:14.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.743+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae280a9d80 con 0x7fae28072680 2024-03-06T07:37:14.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.744+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2804f330 con 0x7fae28072680 2024-03-06T07:37:14.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.746+0000 7fae25600640 1 -- 172.21.15.114:0/3284751106 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae1c011070 con 0x7fae28072680 2024-03-06T07:37:14.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.747+0000 7fae25600640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b4d0 0x7fae0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.747+0000 7fae25600640 1 -- 172.21.15.114:0/3284751106 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fae1c089ed0 con 0x7fae28072680 2024-03-06T07:37:14.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.747+0000 7fae2ca00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b4d0 0x7fae0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.748+0000 7fae2ca00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b4d0 0x7fae0806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fae14036000 tx=0x7fae14008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.752+0000 7fae25600640 1 -- 172.21.15.114:0/3284751106 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae1c058cf0 con 0x7fae28072680 2024-03-06T07:37:14.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.984+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae281430b0 con 0x7fae28072680 2024-03-06T07:37:14.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.985+0000 7fae25600640 1 -- 172.21.15.114:0/3284751106 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fae1c0585f0 con 0x7fae28072680 2024-03-06T07:37:14.987 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:14.987 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:14.988 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:14.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b4d0 msgr2=0x7fae0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b4d0 0x7fae0806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fae14036000 tx=0x7fae14008030 comp rx=0 tx=0).stop 2024-03-06T07:37:14.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 msgr2=0x7fae280b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280b2110 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00c430 tx=0x7fae1c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 shutdown_connections 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae28144660 0x7fae280cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae280cf3e0 0x7fae280b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae0806b4d0 0x7fae0806d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 --2- 172.21.15.114:0/3284751106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae28072680 0x7fae280b2110 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 >> 172.21.15.114:0/3284751106 conn(0x7fae280d0b90 msgr2=0x7fae28143600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 shutdown_connections 2024-03-06T07:37:14.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:14.988+0000 7fae2fe00640 1 -- 172.21.15.114:0/3284751106 wait complete. 2024-03-06T07:37:15.213 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:15.213 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:16.056 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:16.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:16.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.426+0000 7f8fdbc00640 1 Processor -- start 2024-03-06T07:37:16.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.426+0000 7f8fdbc00640 1 -- start start 2024-03-06T07:37:16.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.429+0000 7f8fdbc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8fd415e160 0x7f8fd4162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.429+0000 7f8fdbc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd4162b60 con 0x7f8fd40af820 2024-03-06T07:37:16.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.429+0000 7f8fdbc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd406f1f0 con 0x7f8fd40b3ac0 2024-03-06T07:37:16.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.429+0000 7f8fdbc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd406b0f0 con 0x7f8fd415e160 2024-03-06T07:37:16.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.430+0000 7f8fd3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8fd40af820 0x7f8fd40b33b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59998/0 (socket says 172.21.15.114:59998) 2024-03-06T07:37:16.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.430+0000 7f8fd3e00640 1 -- 172.21.15.114:0/601480514 learned_addr learned my addr 172.21.15.114:0/601480514 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:16.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.430+0000 7f8fd9200640 1 --2- 172.21.15.114:0/601480514 >> v2:172.21.15.177:3300/0 conn(0x7f8fd415e160 0x7f8fd4162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.430+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144861469 0 0) 0x7f8fc0001110 con 0x7f8fd40af820 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.430+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fbc003520 con 0x7f8fd40af820 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 995861772 0 0) 0x7f8fbc003520 con 0x7f8fd40af820 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 >> v2:172.21.15.177:3300/0 conn(0x7f8fd415e160 msgr2=0x7f8fd4162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 --2- 172.21.15.114:0/601480514 >> v2:172.21.15.177:3300/0 conn(0x7f8fd415e160 0x7f8fd4162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 >> v1:172.21.15.177:6789/0 conn(0x7f8fd40b3ac0 legacy=0x7f8fd406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40afce0 con 0x7f8fd40af820 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8fc0001840 con 0x7f8fd40af820 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 >> v1:172.21.15.114:6789/0 conn(0x7f8fd40af820 legacy=0x7f8fd40b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fbc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc004800 0x7f8fbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fbc0051c0 0x7f8fbc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd406b0f0 con 0x7f8fbc0039b0 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc00c160 con 0x7f8fbc004800 2024-03-06T07:37:16.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc00c360 con 0x7f8fbc0051c0 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd9200640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fbc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd9200640 1 -- 172.21.15.114:0/601480514 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fbc0051c0 msgr2=0x7f8fbc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd9200640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fbc0051c0 0x7f8fbc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd9200640 1 -- 172.21.15.114:0/601480514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc004800 msgr2=0x7f8fbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd3e00640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc004800 0x7f8fbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd9200640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc004800 0x7f8fbc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd9200640 1 -- 172.21.15.114:0/601480514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc003520 con 0x7f8fbc0039b0 2024-03-06T07:37:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.431+0000 7f8fd9c00640 1 -- 172.21.15.114:0/601480514 reap_dead start 2024-03-06T07:37:16.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.433+0000 7f8fd9200640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fbc003dd0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009ac0 tx=0x7f8fbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2971925098b42a94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:16.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.433+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8039070 con 0x7f8fbc0039b0 2024-03-06T07:37:16.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.433+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc802c710 con 0x7f8fbc0039b0 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.433+0000 7f8fd3400640 1 -- 172.21.15.114:0/601480514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8009690 con 0x7f8fbc0039b0 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.433+0000 7f8fdbc00640 1 -- 172.21.15.114:0/601480514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 msgr2=0x7f8fbc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.433+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fbc003dd0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009ac0 tx=0x7f8fbc005830 comp rx=0 tx=0).stop 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.434+0000 7f8fdbc00640 1 -- 172.21.15.114:0/601480514 shutdown_connections 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.434+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/601480514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fbc003dd0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.434+0000 7f8fdbc00640 1 -- 172.21.15.114:0/601480514 >> 172.21.15.114:0/601480514 conn(0x7f8fd40d0340 msgr2=0x7f8fd40d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.434+0000 7f8fdbc00640 1 -- 172.21.15.114:0/601480514 shutdown_connections 2024-03-06T07:37:16.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.434+0000 7f8fdbc00640 1 -- 172.21.15.114:0/601480514 wait complete. 2024-03-06T07:37:16.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.436+0000 7f8fdbc00640 1 Processor -- start 2024-03-06T07:37:16.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.437+0000 7f8fdbc00640 1 -- start start 2024-03-06T07:37:16.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.438+0000 7f8fdbc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fd406c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.438+0000 7f8fdbc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fd4160bc0 0x7f8fd4161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.438+0000 7f8fdbc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd414fd80 0x7f8fd4150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.438+0000 7f8fdbc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd406ef10 con 0x7f8fbc0039b0 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.438+0000 7f8fdbc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b5620 con 0x7f8fd414fd80 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.438+0000 7f8fdbc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4162620 con 0x7f8fd4160bc0 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fd406c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fd406c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48572/0 (socket says 172.21.15.114:48572) 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 -- 172.21.15.114:0/3164014799 learned_addr learned my addr 172.21.15.114:0/3164014799 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd3e00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fd4160bc0 0x7f8fd4161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 -- 172.21.15.114:0/3164014799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fd4160bc0 msgr2=0x7f8fd4161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fd4160bc0 0x7f8fd4161040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 -- 172.21.15.114:0/3164014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd414fd80 msgr2=0x7f8fd4150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd414fd80 0x7f8fd4150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 -- 172.21.15.114:0/3164014799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd41f7760 con 0x7f8fbc0039b0 2024-03-06T07:37:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.439+0000 7f8fd9200640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fd406c070 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009ac0 tx=0x7f8fc8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.440+0000 7f8fd0c00640 1 -- 172.21.15.114:0/3164014799 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8039050 con 0x7f8fbc0039b0 2024-03-06T07:37:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.440+0000 7f8fd0c00640 1 -- 172.21.15.114:0/3164014799 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc8045070 con 0x7f8fbc0039b0 2024-03-06T07:37:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.440+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd41f7a00 con 0x7f8fbc0039b0 2024-03-06T07:37:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.440+0000 7f8fd0c00640 1 -- 172.21.15.114:0/3164014799 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8003a50 con 0x7f8fbc0039b0 2024-03-06T07:37:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.440+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd41f7fc0 con 0x7f8fbc0039b0 2024-03-06T07:37:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.441+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4074ae0 con 0x7f8fbc0039b0 2024-03-06T07:37:16.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.443+0000 7f8fd0c00640 1 -- 172.21.15.114:0/3164014799 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fc8003bf0 con 0x7f8fbc0039b0 2024-03-06T07:37:16.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.444+0000 7f8fd0c00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.444+0000 7f8fd0c00640 1 -- 172.21.15.114:0/3164014799 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8fc80ab970 con 0x7f8fbc0039b0 2024-03-06T07:37:16.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.444+0000 7f8fd3e00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.445+0000 7f8fd3e00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fb406b4d0 0x7f8fb406d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00012b0 tx=0x7f8fc0001e00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:16.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.448+0000 7f8fd0c00640 1 -- 172.21.15.114:0/3164014799 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fc807a7d0 con 0x7f8fbc0039b0 2024-03-06T07:37:16.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.686+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fd404f330 con 0x7f8fbc0039b0 2024-03-06T07:37:16.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.687+0000 7f8fd0c00640 1 -- 172.21.15.114:0/3164014799 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f8fc807a0d0 con 0x7f8fbc0039b0 2024-03-06T07:37:16.688 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:16.688 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:16.689 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fb406b4d0 msgr2=0x7f8fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fb406b4d0 0x7f8fb406d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00012b0 tx=0x7f8fc0001e00 comp rx=0 tx=0).stop 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 msgr2=0x7f8fd406c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fd406c070 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009ac0 tx=0x7f8fc8004730 comp rx=0 tx=0).stop 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 shutdown_connections 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fd414fd80 0x7f8fd4150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fd4160bc0 0x7f8fd4161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.689+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.690+0000 7f8fdbc00640 1 --2- 172.21.15.114:0/3164014799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fbc0039b0 0x7f8fd406c070 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.690+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 >> 172.21.15.114:0/3164014799 conn(0x7f8fd40d0340 msgr2=0x7f8fd4161ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.690+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 shutdown_connections 2024-03-06T07:37:16.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:16.690+0000 7f8fdbc00640 1 -- 172.21.15.114:0/3164014799 wait complete. 2024-03-06T07:37:16.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:16.909 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:17.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:17.272 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:17.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:18.058 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:18.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.119+0000 7f57e6200640 1 Processor -- start 2024-03-06T07:37:18.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.119+0000 7f57e6200640 1 -- start start 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57e6200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57e6200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f57e015e4b0 0x7f57e00af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57e6200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e00b1440 con 0x7f57e00b5db0 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57e6200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e006b500 con 0x7f57e015e4b0 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57e6200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e006c0d0 con 0x7f57e015e0d0 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57d7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f57e015e4b0 0x7f57e00af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57e5800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f57e00b5db0 0x7f57e0072670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40446/0 (socket says 172.21.15.114:40446) 2024-03-06T07:37:18.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.121+0000 7f57e5800640 1 -- 172.21.15.114:0/3677475599 learned_addr learned my addr 172.21.15.114:0/3677475599 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:18.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.122+0000 7f57e4e00640 1 --2- 172.21.15.114:0/3677475599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.122+0000 7f57d7e00640 1 -- 172.21.15.114:0/3677475599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 msgr2=0x7f57e006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:18.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.122+0000 7f57d7e00640 1 --2- 172.21.15.114:0/3677475599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e006afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.122+0000 7f57d7e00640 1 -- 172.21.15.114:0/3677475599 >> v1:172.21.15.114:6789/0 conn(0x7f57e00b5db0 legacy=0x7f57e0072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.122+0000 7f57d7e00640 1 -- 172.21.15.114:0/3677475599 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e00b16e0 con 0x7f57e015e4b0 2024-03-06T07:37:18.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.122+0000 7f57e4e00640 1 --2- 172.21.15.114:0/3677475599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:18.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.123+0000 7f57d7e00640 1 --2- 172.21.15.114:0/3677475599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e015e4b0 0x7f57e00af240 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f57c80097a0 tx=0x7f57c802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e90ad713ece831 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57dfe00640 1 -- 172.21.15.114:0/3677475599 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8034070 con 0x7f57e015e4b0 2024-03-06T07:37:18.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57dfe00640 1 -- 172.21.15.114:0/3677475599 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c8030800 con 0x7f57e015e4b0 2024-03-06T07:37:18.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57dfe00640 1 -- 172.21.15.114:0/3677475599 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8030bd0 con 0x7f57e015e4b0 2024-03-06T07:37:18.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 -- 172.21.15.114:0/3677475599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e015e4b0 msgr2=0x7f57e00af240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 --2- 172.21.15.114:0/3677475599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e015e4b0 0x7f57e00af240 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f57c80097a0 tx=0x7f57c802b940 comp rx=0 tx=0).stop 2024-03-06T07:37:18.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 -- 172.21.15.114:0/3677475599 shutdown_connections 2024-03-06T07:37:18.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 --2- 172.21.15.114:0/3677475599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e015e4b0 0x7f57e00af240 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 --2- 172.21.15.114:0/3677475599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 -- 172.21.15.114:0/3677475599 >> 172.21.15.114:0/3677475599 conn(0x7f57e00b4fe0 msgr2=0x7f57e00b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 -- 172.21.15.114:0/3677475599 shutdown_connections 2024-03-06T07:37:18.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.124+0000 7f57e6200640 1 -- 172.21.15.114:0/3677475599 wait complete. 2024-03-06T07:37:18.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.126+0000 7f57e6200640 1 Processor -- start 2024-03-06T07:37:18.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.126+0000 7f57e6200640 1 -- start start 2024-03-06T07:37:18.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e6200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e00b5db0 0x7f57e0155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e6200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e014e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57e015e4b0 0x7f57e0151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e6200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e006c320 con 0x7f57e00b5db0 2024-03-06T07:37:18.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e6200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e006c690 con 0x7f57e015e4b0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e6200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e00ae170 con 0x7f57e015e0d0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e00b5db0 0x7f57e0155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e00b5db0 0x7f57e0155730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59514/0 (socket says 172.21.15.114:59514) 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.127+0000 7f57e4e00640 1 -- 172.21.15.114:0/3344771403 learned_addr learned my addr 172.21.15.114:0/3344771403 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57e4e00640 1 -- 172.21.15.114:0/3344771403 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 msgr2=0x7f57e014e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57e4e00640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e014e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57e4e00640 1 -- 172.21.15.114:0/3344771403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57e015e4b0 msgr2=0x7f57e0151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57e4e00640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57e015e4b0 0x7f57e0151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57e4e00640 1 -- 172.21.15.114:0/3344771403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e0150050 con 0x7f57e00b5db0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57e4e00640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e00b5db0 0x7f57e0155730 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f57d000c7f0 tx=0x7f57d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57dd600640 1 -- 172.21.15.114:0/3344771403 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0015070 con 0x7f57e00b5db0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57dd600640 1 -- 172.21.15.114:0/3344771403 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d0007a30 con 0x7f57e00b5db0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e0150330 con 0x7f57e00b5db0 2024-03-06T07:37:18.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.128+0000 7f57dd600640 1 -- 172.21.15.114:0/3344771403 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0007e00 con 0x7f57e00b5db0 2024-03-06T07:37:18.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.129+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e015f700 con 0x7f57e00b5db0 2024-03-06T07:37:18.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.130+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e004f330 con 0x7f57e00b5db0 2024-03-06T07:37:18.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.131+0000 7f57dd600640 1 -- 172.21.15.114:0/3344771403 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57d0018070 con 0x7f57e00b5db0 2024-03-06T07:37:18.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.132+0000 7f57dd600640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f57bc06b4d0 0x7f57bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:18.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.132+0000 7f57dd600640 1 -- 172.21.15.114:0/3344771403 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f57d008a200 con 0x7f57e00b5db0 2024-03-06T07:37:18.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.132+0000 7f57d7e00640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f57bc06b4d0 0x7f57bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:18.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.133+0000 7f57d7e00640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f57bc06b4d0 0x7f57bc06d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f57c8036000 tx=0x7f57c8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:18.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.137+0000 7f57dd600640 1 -- 172.21.15.114:0/3344771403 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57d0059030 con 0x7f57e00b5db0 2024-03-06T07:37:18.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.376+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57e006ebc0 con 0x7f57e00b5db0 2024-03-06T07:37:18.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.377+0000 7f57dd600640 1 -- 172.21.15.114:0/3344771403 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f57d0058930 con 0x7f57e00b5db0 2024-03-06T07:37:18.379 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:18.379 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:18.379 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f57bc06b4d0 msgr2=0x7f57bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f57bc06b4d0 0x7f57bc06d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f57c8036000 tx=0x7f57c8030000 comp rx=0 tx=0).stop 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e00b5db0 msgr2=0x7f57e0155730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e00b5db0 0x7f57e0155730 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f57d000c7f0 tx=0x7f57d000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 shutdown_connections 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57e015e4b0 0x7f57e0151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f57e015e0d0 0x7f57e014e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f57bc06b4d0 0x7f57bc06d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 --2- 172.21.15.114:0/3344771403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e00b5db0 0x7f57e0155730 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 >> 172.21.15.114:0/3344771403 conn(0x7f57e00b4fe0 msgr2=0x7f57e015cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 shutdown_connections 2024-03-06T07:37:18.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:18.381+0000 7f57e6200640 1 -- 172.21.15.114:0/3344771403 wait complete. 2024-03-06T07:37:18.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:18.602 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:19.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:19.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.812+0000 7fe332200640 1 Processor -- start 2024-03-06T07:37:19.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.812+0000 7fe332200640 1 -- start start 2024-03-06T07:37:19.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.814+0000 7fe332200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.814+0000 7fe332200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe32c06d260 0x7fe32c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.814+0000 7fe332200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c0717c0 con 0x7fe32c071260 2024-03-06T07:37:19.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.814+0000 7fe332200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c071aa0 con 0x7fe32c06d260 2024-03-06T07:37:19.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.814+0000 7fe332200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c071d80 con 0x7fe32c0cf430 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.815+0000 7fe32b400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.815+0000 7fe32be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe32c06d260 0x7fe32c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.815+0000 7fe32b400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:35274/0 (socket says 172.21.15.114:35274) 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.815+0000 7fe32b400640 1 -- 172.21.15.114:0/2661223385 learned_addr learned my addr 172.21.15.114:0/2661223385 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.816+0000 7fe32b400640 1 -- 172.21.15.114:0/2661223385 >> v2:172.21.15.177:3300/0 conn(0x7fe32c06d260 msgr2=0x7fe32c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.816+0000 7fe32b400640 1 --2- 172.21.15.114:0/2661223385 >> v2:172.21.15.177:3300/0 conn(0x7fe32c06d260 0x7fe32c0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.816+0000 7fe32b400640 1 -- 172.21.15.114:0/2661223385 >> v1:172.21.15.177:6789/0 conn(0x7fe32c071260 legacy=0x7fe32c06cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.816+0000 7fe32b400640 1 -- 172.21.15.114:0/2661223385 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c072060 con 0x7fe32c0cf430 2024-03-06T07:37:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.817+0000 7fe32b400640 1 --2- 172.21.15.114:0/2661223385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c070d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe314009870 tx=0x7fe31402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52b67c19ca4288f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.818+0000 7fe32aa00640 1 -- 172.21.15.114:0/2661223385 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314034070 con 0x7fe32c0cf430 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.818+0000 7fe32aa00640 1 -- 172.21.15.114:0/2661223385 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe314030840 con 0x7fe32c0cf430 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.818+0000 7fe32aa00640 1 -- 172.21.15.114:0/2661223385 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314030c10 con 0x7fe32c0cf430 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.818+0000 7fe332200640 1 -- 172.21.15.114:0/2661223385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 msgr2=0x7fe32c070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.818+0000 7fe332200640 1 --2- 172.21.15.114:0/2661223385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c070d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe314009870 tx=0x7fe31402b980 comp rx=0 tx=0).stop 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.819+0000 7fe332200640 1 -- 172.21.15.114:0/2661223385 shutdown_connections 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.819+0000 7fe332200640 1 --2- 172.21.15.114:0/2661223385 >> v2:172.21.15.177:3300/0 conn(0x7fe32c06d260 0x7fe32c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.819+0000 7fe332200640 1 --2- 172.21.15.114:0/2661223385 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c070d20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.819+0000 7fe332200640 1 -- 172.21.15.114:0/2661223385 >> 172.21.15.114:0/2661223385 conn(0x7fe32c0b1730 msgr2=0x7fe32c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:19.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.819+0000 7fe332200640 1 -- 172.21.15.114:0/2661223385 shutdown_connections 2024-03-06T07:37:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.819+0000 7fe332200640 1 -- 172.21.15.114:0/2661223385 wait complete. 2024-03-06T07:37:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.820+0000 7fe332200640 1 Processor -- start 2024-03-06T07:37:19.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.821+0000 7fe332200640 1 -- start start 2024-03-06T07:37:19.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.821+0000 7fe332200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe32c06d260 0x7fe32c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.821+0000 7fe332200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe32c071260 0x7fe32c147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.821+0000 7fe332200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c1447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.821+0000 7fe332200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c162cf0 con 0x7fe32c071260 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.821+0000 7fe332200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c162b70 con 0x7fe32c06d260 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.821+0000 7fe332200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0b3250 con 0x7fe32c0cf430 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe32c071260 0x7fe32c147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe32c071260 0x7fe32c147140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59530/0 (socket says 172.21.15.114:59530) 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 -- 172.21.15.114:0/2075853695 learned_addr learned my addr 172.21.15.114:0/2075853695 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe32be00640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c1447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe32b400640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe32c06d260 0x7fe32c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 -- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 msgr2=0x7fe32c1447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c1447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 -- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe32c06d260 msgr2=0x7fe32c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe32c06d260 0x7fe32c15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe322a00640 1 -- 172.21.15.114:0/2075853695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c147680 con 0x7fe32c071260 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.822+0000 7fe32be00640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c1447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.823+0000 7fe322a00640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe32c071260 0x7fe32c147140 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00c400 tx=0x7fe31c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.823+0000 7fe323e00640 1 -- 172.21.15.114:0/2075853695 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0052d0 con 0x7fe32c071260 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.823+0000 7fe323e00640 1 -- 172.21.15.114:0/2075853695 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c005490 con 0x7fe32c071260 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.823+0000 7fe323e00640 1 -- 172.21.15.114:0/2075853695 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c007260 con 0x7fe32c071260 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.823+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c147950 con 0x7fe32c071260 2024-03-06T07:37:19.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.823+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c147f10 con 0x7fe32c071260 2024-03-06T07:37:19.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.824+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c0b1550 con 0x7fe32c071260 2024-03-06T07:37:19.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.825+0000 7fe323e00640 1 -- 172.21.15.114:0/2075853695 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe31c007400 con 0x7fe32c071260 2024-03-06T07:37:19.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.826+0000 7fe323e00640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe30806b4d0 0x7fe30806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.826+0000 7fe323e00640 1 -- 172.21.15.114:0/2075853695 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe31c089780 con 0x7fe32c071260 2024-03-06T07:37:19.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.827+0000 7fe32b400640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe30806b4d0 0x7fe30806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.827+0000 7fe32b400640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe30806b4d0 0x7fe30806d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe314036000 tx=0x7fe31403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:19.831+0000 7fe323e00640 1 -- 172.21.15.114:0/2075853695 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe32c0b1550 con 0x7fe32c071260 2024-03-06T07:37:20.060 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:20.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.064+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe32c158e80 con 0x7fe32c071260 2024-03-06T07:37:20.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.065+0000 7fe323e00640 1 -- 172.21.15.114:0/2075853695 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fe31c057ed0 con 0x7fe32c071260 2024-03-06T07:37:20.068 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:20.068 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:20.068 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:20.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe30806b4d0 msgr2=0x7fe30806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe30806b4d0 0x7fe30806d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe314036000 tx=0x7fe31403b040 comp rx=0 tx=0).stop 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe32c071260 msgr2=0x7fe32c147140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe32c071260 0x7fe32c147140 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00c400 tx=0x7fe31c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 shutdown_connections 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe32c0cf430 0x7fe32c1447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe30806b4d0 0x7fe30806d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe32c071260 0x7fe32c147140 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 --2- 172.21.15.114:0/2075853695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe32c06d260 0x7fe32c15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.068+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 >> 172.21.15.114:0/2075853695 conn(0x7fe32c0b1730 msgr2=0x7fe32c163000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.069+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 shutdown_connections 2024-03-06T07:37:20.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:20.069+0000 7fe332200640 1 -- 172.21.15.114:0/2075853695 wait complete. 2024-03-06T07:37:20.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:20.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:21.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:21.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.509+0000 7ff5ef200640 1 Processor -- start 2024-03-06T07:37:21.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.509+0000 7ff5ef200640 1 -- start start 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.510+0000 7ff5ef200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff5e80b4680 0x7ff5e806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.510+0000 7ff5ef200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff5e8072340 0x7ff5e806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.510+0000 7ff5ef200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e815b090 con 0x7ff5e815acb0 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.510+0000 7ff5ef200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e80b2a30 con 0x7ff5e8072340 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.510+0000 7ff5ef200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e80b4360 con 0x7ff5e80b4680 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5ed200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff5e8072340 0x7ff5e806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5ed200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff5e8072340 0x7ff5e806ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59540/0 (socket says 172.21.15.114:59540) 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5ed200640 1 -- 172.21.15.114:0/4080276331 learned_addr learned my addr 172.21.15.114:0/4080276331 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:21.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5e7400640 1 --2- 172.21.15.114:0/4080276331 >> v2:172.21.15.177:3300/0 conn(0x7ff5e80b4680 0x7ff5e806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5ed200640 1 -- 172.21.15.114:0/4080276331 >> v2:172.21.15.177:3300/0 conn(0x7ff5e80b4680 msgr2=0x7ff5e806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5ed200640 1 --2- 172.21.15.114:0/4080276331 >> v2:172.21.15.177:3300/0 conn(0x7ff5e80b4680 0x7ff5e806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5ed200640 1 -- 172.21.15.114:0/4080276331 >> v1:172.21.15.177:6789/0 conn(0x7ff5e815acb0 legacy=0x7ff5e80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5ed200640 1 -- 172.21.15.114:0/4080276331 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e80b2cf0 con 0x7ff5e8072340 2024-03-06T07:37:21.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.511+0000 7ff5e7400640 1 --2- 172.21.15.114:0/4080276331 >> v2:172.21.15.177:3300/0 conn(0x7ff5e80b4680 0x7ff5e806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.512+0000 7ff5ed200640 1 --2- 172.21.15.114:0/4080276331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e8072340 0x7ff5e806ea30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8068560 tx=0x7ff5e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f332a955394f7ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5e6a00640 1 -- 172.21.15.114:0/4080276331 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e002ca80 con 0x7ff5e8072340 2024-03-06T07:37:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5e6a00640 1 -- 172.21.15.114:0/4080276331 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e002cc20 con 0x7ff5e8072340 2024-03-06T07:37:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5e6a00640 1 -- 172.21.15.114:0/4080276331 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e00343f0 con 0x7ff5e8072340 2024-03-06T07:37:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 -- 172.21.15.114:0/4080276331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e8072340 msgr2=0x7ff5e806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 --2- 172.21.15.114:0/4080276331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e8072340 0x7ff5e806ea30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8068560 tx=0x7ff5e002bee0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 -- 172.21.15.114:0/4080276331 shutdown_connections 2024-03-06T07:37:21.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 --2- 172.21.15.114:0/4080276331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e8072340 0x7ff5e806ea30 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 --2- 172.21.15.114:0/4080276331 >> v2:172.21.15.177:3300/0 conn(0x7ff5e80b4680 0x7ff5e806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 -- 172.21.15.114:0/4080276331 >> 172.21.15.114:0/4080276331 conn(0x7ff5e80b1730 msgr2=0x7ff5e80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:21.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 -- 172.21.15.114:0/4080276331 shutdown_connections 2024-03-06T07:37:21.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.513+0000 7ff5ef200640 1 -- 172.21.15.114:0/4080276331 wait complete. 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.515+0000 7ff5ef200640 1 Processor -- start 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.515+0000 7ff5ef200640 1 -- start start 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ef200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5e8072340 0x7ff5e8152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ef200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5e80b4680 0x7ff5e81512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ef200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e815acb0 0x7ff5e81582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ef200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80b4010 con 0x7ff5e815acb0 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ef200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80b32b0 con 0x7ff5e8072340 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ef200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80ccfb0 con 0x7ff5e80b4680 2024-03-06T07:37:21.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ed200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e815acb0 0x7ff5e81582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ed200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e815acb0 0x7ff5e81582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59554/0 (socket says 172.21.15.114:59554) 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.516+0000 7ff5ed200640 1 -- 172.21.15.114:0/994729628 learned_addr learned my addr 172.21.15.114:0/994729628 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5e7400640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5e80b4680 0x7ff5e81512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5e7e00640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5e8072340 0x7ff5e8152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5ed200640 1 -- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5e80b4680 msgr2=0x7ff5e81512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5ed200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5e80b4680 0x7ff5e81512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5ed200640 1 -- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5e8072340 msgr2=0x7ff5e8152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5ed200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5e8072340 0x7ff5e8152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5ed200640 1 -- 172.21.15.114:0/994729628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e814b2e0 con 0x7ff5e815acb0 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5e7e00640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5e8072340 0x7ff5e8152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5ed200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e815acb0 0x7ff5e81582a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0036040 tx=0x7ff5e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5dfe00640 1 -- 172.21.15.114:0/994729628 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0043020 con 0x7ff5e815acb0 2024-03-06T07:37:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5dfe00640 1 -- 172.21.15.114:0/994729628 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e003e450 con 0x7ff5e815acb0 2024-03-06T07:37:21.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.517+0000 7ff5dfe00640 1 -- 172.21.15.114:0/994729628 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e003e840 con 0x7ff5e815acb0 2024-03-06T07:37:21.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.518+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e814b580 con 0x7ff5e815acb0 2024-03-06T07:37:21.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.518+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e814bb40 con 0x7ff5e815acb0 2024-03-06T07:37:21.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.519+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e804f330 con 0x7ff5e815acb0 2024-03-06T07:37:21.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.520+0000 7ff5dfe00640 1 -- 172.21.15.114:0/994729628 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5e0005d10 con 0x7ff5e815acb0 2024-03-06T07:37:21.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.521+0000 7ff5dfe00640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff5c406b4d0 0x7ff5c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.521+0000 7ff5dfe00640 1 -- 172.21.15.114:0/994729628 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff5e00acde0 con 0x7ff5e815acb0 2024-03-06T07:37:21.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.522+0000 7ff5e7e00640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff5c406b4d0 0x7ff5c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.522+0000 7ff5e7e00640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff5c406b4d0 0x7ff5c406d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff5e814ee70 tx=0x7ff5d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.531+0000 7ff5dfe00640 1 -- 172.21.15.114:0/994729628 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5e007bcb0 con 0x7ff5e815acb0 2024-03-06T07:37:21.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.763+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5e806ce60 con 0x7ff5e815acb0 2024-03-06T07:37:21.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.765+0000 7ff5dfe00640 1 -- 172.21.15.114:0/994729628 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7ff5e007b5b0 con 0x7ff5e815acb0 2024-03-06T07:37:21.767 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:21.768 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:21.768 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.767+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff5c406b4d0 msgr2=0x7ff5c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.767+0000 7ff5ef200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff5c406b4d0 0x7ff5c406d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff5e814ee70 tx=0x7ff5d8008040 comp rx=0 tx=0).stop 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.767+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e815acb0 msgr2=0x7ff5e81582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.767+0000 7ff5ef200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e815acb0 0x7ff5e81582a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0036040 tx=0x7ff5e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 shutdown_connections 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff5c406b4d0 0x7ff5c406d990 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5e815acb0 0x7ff5e81582a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff5e80b4680 0x7ff5e81512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 --2- 172.21.15.114:0/994729628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5e8072340 0x7ff5e8152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 >> 172.21.15.114:0/994729628 conn(0x7ff5e80b1730 msgr2=0x7ff5e8071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:21.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 shutdown_connections 2024-03-06T07:37:21.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:21.768+0000 7ff5ef200640 1 -- 172.21.15.114:0/994729628 wait complete. 2024-03-06T07:37:21.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:21.992 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:22.062 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:22.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:23.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.207+0000 7f0846c00640 1 Processor -- start 2024-03-06T07:37:23.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.207+0000 7f0846c00640 1 -- start start 2024-03-06T07:37:23.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f0846c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0840144660 0x7f0840071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f0846c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f08400cf3e0 0x7f08400724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f0846c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08400cabf0 con 0x7f08400c9ec0 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f0846c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08400ca9b0 con 0x7f0840144660 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f0846c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08400ce900 con 0x7f08400cf3e0 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f083fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0840144660 0x7f0840071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f083fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0840144660 0x7f0840071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59566/0 (socket says 172.21.15.114:59566) 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.209+0000 7f083fe00640 1 -- 172.21.15.114:0/1122806150 learned_addr learned my addr 172.21.15.114:0/1122806150 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.210+0000 7f083fe00640 1 -- 172.21.15.114:0/1122806150 >> v2:172.21.15.177:3300/0 conn(0x7f08400cf3e0 msgr2=0x7f08400724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.210+0000 7f083fe00640 1 --2- 172.21.15.114:0/1122806150 >> v2:172.21.15.177:3300/0 conn(0x7f08400cf3e0 0x7f08400724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.210+0000 7f083fe00640 1 -- 172.21.15.114:0/1122806150 >> v1:172.21.15.177:6789/0 conn(0x7f08400c9ec0 legacy=0x7f08400ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:23.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.210+0000 7f083fe00640 1 -- 172.21.15.114:0/1122806150 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08400cebe0 con 0x7f0840144660 2024-03-06T07:37:23.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.211+0000 7f083fe00640 1 --2- 172.21.15.114:0/1122806150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f0840071f70 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f0840161b30 tx=0x7f083002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1794debaff026ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.211+0000 7f083f400640 1 -- 172.21.15.114:0/1122806150 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830035020 con 0x7f0840144660 2024-03-06T07:37:23.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.211+0000 7f083f400640 1 -- 172.21.15.114:0/1122806150 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0830031870 con 0x7f0840144660 2024-03-06T07:37:23.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.211+0000 7f083f400640 1 -- 172.21.15.114:0/1122806150 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830031c40 con 0x7f0840144660 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 -- 172.21.15.114:0/1122806150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 msgr2=0x7f0840071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 --2- 172.21.15.114:0/1122806150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f0840071f70 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f0840161b30 tx=0x7f083002b910 comp rx=0 tx=0).stop 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 -- 172.21.15.114:0/1122806150 shutdown_connections 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 --2- 172.21.15.114:0/1122806150 >> v2:172.21.15.177:3300/0 conn(0x7f08400cf3e0 0x7f08400724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 --2- 172.21.15.114:0/1122806150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f0840071f70 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 -- 172.21.15.114:0/1122806150 >> 172.21.15.114:0/1122806150 conn(0x7f08400d0b90 msgr2=0x7f08400d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 -- 172.21.15.114:0/1122806150 shutdown_connections 2024-03-06T07:37:23.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.212+0000 7f0846c00640 1 -- 172.21.15.114:0/1122806150 wait complete. 2024-03-06T07:37:23.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.214+0000 7f0846c00640 1 Processor -- start 2024-03-06T07:37:23.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.214+0000 7f0846c00640 1 -- start start 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0846c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08400c9ec0 0x7f084013f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0846c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08400cf3e0 0x7f084014dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0846c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f08400b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0846c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400cc0f0 con 0x7f0840144660 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0846c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840144a40 con 0x7f08400cf3e0 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0846c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08400cc270 con 0x7f08400c9ec0 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0844c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f08400b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0844c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f08400b62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59580/0 (socket says 172.21.15.114:59580) 2024-03-06T07:37:23.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0844c00640 1 -- 172.21.15.114:0/2624448711 learned_addr learned my addr 172.21.15.114:0/2624448711 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0837400640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08400cf3e0 0x7f084014dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f083fe00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08400c9ec0 0x7f084013f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.215+0000 7f0844c00640 1 -- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08400c9ec0 msgr2=0x7f084013f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0844c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08400c9ec0 0x7f084013f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0844c00640 1 -- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08400cf3e0 msgr2=0x7f084014dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0844c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08400cf3e0 0x7f084014dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0844c00640 1 -- 172.21.15.114:0/2624448711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084014e290 con 0x7f0840144660 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0837400640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08400cf3e0 0x7f084014dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0844c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f08400b62f0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f083800c780 tx=0x7f083800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f083cc00640 1 -- 172.21.15.114:0/2624448711 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838017070 con 0x7f0840144660 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f083cc00640 1 -- 172.21.15.114:0/2624448711 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838003da0 con 0x7f0840144660 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f083cc00640 1 -- 172.21.15.114:0/2624448711 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08380052b0 con 0x7f0840144660 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084015f8d0 con 0x7f0840144660 2024-03-06T07:37:23.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.216+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084015fe90 con 0x7f0840144660 2024-03-06T07:37:23.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.217+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084004f330 con 0x7f0840144660 2024-03-06T07:37:23.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.218+0000 7f083cc00640 1 -- 172.21.15.114:0/2624448711 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0838005450 con 0x7f0840144660 2024-03-06T07:37:23.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.219+0000 7f083cc00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.219+0000 7f083cc00640 1 -- 172.21.15.114:0/2624448711 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0838089800 con 0x7f0840144660 2024-03-06T07:37:23.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.220+0000 7f083fe00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.222 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.220+0000 7f083fe00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f081c06b4d0 0x7f081c06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0830036000 tx=0x7f083003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.225+0000 7f083cc00640 1 -- 172.21.15.114:0/2624448711 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0838058630 con 0x7f0840144660 2024-03-06T07:37:23.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.463+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0840072ba0 con 0x7f0840144660 2024-03-06T07:37:23.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.464+0000 7f083cc00640 1 -- 172.21.15.114:0/2624448711 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f0838057f30 con 0x7f0840144660 2024-03-06T07:37:23.466 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:23.466 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:23.466 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:23.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f081c06b4d0 msgr2=0x7f081c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f081c06b4d0 0x7f081c06d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0830036000 tx=0x7f083003c040 comp rx=0 tx=0).stop 2024-03-06T07:37:23.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 msgr2=0x7f08400b62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f08400b62f0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f083800c780 tx=0x7f083800cc40 comp rx=0 tx=0).stop 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 shutdown_connections 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0840144660 0x7f08400b62f0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08400cf3e0 0x7f084014dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 --2- 172.21.15.114:0/2624448711 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08400c9ec0 0x7f084013f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.466+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 >> 172.21.15.114:0/2624448711 conn(0x7f08400d0b90 msgr2=0x7f0840072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.467+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 shutdown_connections 2024-03-06T07:37:23.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:23.467+0000 7f0846c00640 1 -- 172.21.15.114:0/2624448711 wait complete. 2024-03-06T07:37:23.696 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:23.696 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:24.064 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:24.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:24.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.907+0000 7f7e25c00640 1 Processor -- start 2024-03-06T07:37:24.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.908+0000 7f7e25c00640 1 -- start start 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.909+0000 7f7e25c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 0x7f7e2006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.909+0000 7f7e25c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7e2015fe30 0x7f7e200ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.909+0000 7f7e25c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2006ee70 con 0x7f7e2015f9d0 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.909+0000 7f7e25c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2006cef0 con 0x7f7e2015fe30 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.909+0000 7f7e25c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2006d1d0 con 0x7f7e20144c00 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.909+0000 7f7e16000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7e2015f9d0 0x7f7e200ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40454/0 (socket says 172.21.15.114:40454) 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.909+0000 7f7e16000640 1 -- 172.21.15.114:0/1301482882 learned_addr learned my addr 172.21.15.114:0/1301482882 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1f400640 1 --2- 172.21.15.114:0/1301482882 >> v2:172.21.15.177:3300/0 conn(0x7f7e2015fe30 0x7f7e200ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1ea00640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 0x7f7e2006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543158192 0 0) 0x7f7e08001110 con 0x7f7e2015f9d0 2024-03-06T07:37:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e04003520 con 0x7f7e2015f9d0 2024-03-06T07:37:24.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1780408695 0 0) 0x7f7e04003520 con 0x7f7e2015f9d0 2024-03-06T07:37:24.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 msgr2=0x7f7e2006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:24.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 0x7f7e2006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 >> v2:172.21.15.177:3300/0 conn(0x7f7e2015fe30 msgr2=0x7f7e200ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 --2- 172.21.15.114:0/1301482882 >> v2:172.21.15.177:3300/0 conn(0x7f7e2015fe30 0x7f7e200ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2006d4b0 con 0x7f7e2015f9d0 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7e08001840 con 0x7f7e2015f9d0 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.910+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 >> v1:172.21.15.114:6789/0 conn(0x7f7e2015f9d0 legacy=0x7f7e200ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1e000640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 0x7f7e040039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1e000640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04003f30 0x7f7e04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1e000640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2006cef0 con 0x7f7e04004d40 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2006d1d0 con 0x7f7e04003f30 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0400b9c0 con 0x7f7e20144c00 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1f400640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1ea00640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 0x7f7e040039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e16000640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04003f30 0x7f7e04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1f400640 1 -- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 msgr2=0x7f7e040039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1f400640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e20144c00 0x7f7e040039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1f400640 1 -- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04003f30 msgr2=0x7f7e04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1f400640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04003f30 0x7f7e04004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.911+0000 7f7e1f400640 1 -- 172.21.15.114:0/1301482882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04003520 con 0x7f7e04004d40 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e1f400640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009fc0 tx=0x7f7e04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb0ad6c67d260a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18039070 con 0x7f7e04004d40 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1802cc40 con 0x7f7e04004d40 2024-03-06T07:37:24.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e1e000640 1 -- 172.21.15.114:0/1301482882 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18009c10 con 0x7f7e04004d40 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e25c00640 1 -- 172.21.15.114:0/1301482882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e04004d40 msgr2=0x7f7e040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e25c00640 1 --2- 172.21.15.114:0/1301482882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009fc0 tx=0x7f7e04005830 comp rx=0 tx=0).stop 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e1f400640 1 -- 172.21.15.114:0/1301482882 reap_dead start 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e25c00640 1 -- 172.21.15.114:0/1301482882 shutdown_connections 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.913+0000 7f7e25c00640 1 -- 172.21.15.114:0/1301482882 >> 172.21.15.114:0/1301482882 conn(0x7f7e200b1730 msgr2=0x7f7e200b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.914+0000 7f7e25c00640 1 -- 172.21.15.114:0/1301482882 shutdown_connections 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.914+0000 7f7e25c00640 1 -- 172.21.15.114:0/1301482882 wait complete. 2024-03-06T07:37:24.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.915+0000 7f7e25c00640 1 Processor -- start 2024-03-06T07:37:24.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.916+0000 7f7e25c00640 1 -- start start 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.916+0000 7f7e25c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04004d40 0x7f7e20138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e25c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e200b3fd0 0x7f7e2013aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e25c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e2015ab80 0x7f7e2015b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e25c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e20160730 con 0x7f7e200b3fd0 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e25c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e20160280 con 0x7f7e04004d40 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e25c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2006cb50 con 0x7f7e2015ab80 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e200b3fd0 0x7f7e2013aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e200b3fd0 0x7f7e2013aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59604/0 (socket says 172.21.15.114:59604) 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 -- 172.21.15.114:0/1303315991 learned_addr learned my addr 172.21.15.114:0/1303315991 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e1ea00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04004d40 0x7f7e20138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 -- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e2015ab80 msgr2=0x7f7e2015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e2015ab80 0x7f7e2015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 -- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04004d40 msgr2=0x7f7e20138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04004d40 0x7f7e20138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e16000640 1 -- 172.21.15.114:0/1303315991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e200b4430 con 0x7f7e200b3fd0 2024-03-06T07:37:24.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.917+0000 7f7e1ea00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04004d40 0x7f7e20138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:24.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.918+0000 7f7e16000640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e200b3fd0 0x7f7e2013aa60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800e870 tx=0x7f7e0800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.918+0000 7f7e17400640 1 -- 172.21.15.114:0/1303315991 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0800c850 con 0x7f7e200b3fd0 2024-03-06T07:37:24.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.918+0000 7f7e17400640 1 -- 172.21.15.114:0/1303315991 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08018070 con 0x7f7e200b3fd0 2024-03-06T07:37:24.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.918+0000 7f7e17400640 1 -- 172.21.15.114:0/1303315991 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0800cc50 con 0x7f7e200b3fd0 2024-03-06T07:37:24.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.918+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e200b2c50 con 0x7f7e200b3fd0 2024-03-06T07:37:24.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.918+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e200b3210 con 0x7f7e200b3fd0 2024-03-06T07:37:24.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.919+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2004f330 con 0x7f7e200b3fd0 2024-03-06T07:37:24.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.921+0000 7f7e17400640 1 -- 172.21.15.114:0/1303315991 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e0801d460 con 0x7f7e200b3fd0 2024-03-06T07:37:24.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.922+0000 7f7e17400640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.922+0000 7f7e17400640 1 -- 172.21.15.114:0/1303315991 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e0808a250 con 0x7f7e200b3fd0 2024-03-06T07:37:24.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.922+0000 7f7e1ea00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.923+0000 7f7e1ea00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009a00 tx=0x7f7e10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:24.931+0000 7f7e17400640 1 -- 172.21.15.114:0/1303315991 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e2006ee70 con 0x7f7e200b3fd0 2024-03-06T07:37:25.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.164+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e20139ae0 con 0x7f7e200b3fd0 2024-03-06T07:37:25.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.165+0000 7f7e17400640 1 -- 172.21.15.114:0/1303315991 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f7e080590b0 con 0x7f7e200b3fd0 2024-03-06T07:37:25.167 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:25.167 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:25.168 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7dfc06b4d0 msgr2=0x7f7dfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009a00 tx=0x7f7e10008040 comp rx=0 tx=0).stop 2024-03-06T07:37:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e200b3fd0 msgr2=0x7f7e2013aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e200b3fd0 0x7f7e2013aa60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800e870 tx=0x7f7e0800ed30 comp rx=0 tx=0).stop 2024-03-06T07:37:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 shutdown_connections 2024-03-06T07:37:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7e2015ab80 0x7f7e2015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e200b3fd0 0x7f7e2013aa60 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 --2- 172.21.15.114:0/1303315991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e04004d40 0x7f7e20138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 >> 172.21.15.114:0/1303315991 conn(0x7f7e200b1730 msgr2=0x7f7e20143590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:25.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 shutdown_connections 2024-03-06T07:37:25.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:25.168+0000 7f7e25c00640 1 -- 172.21.15.114:0/1303315991 wait complete. 2024-03-06T07:37:25.389 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:25.389 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:26.066 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:26.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:26.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.599+0000 7f9e29600640 1 Processor -- start 2024-03-06T07:37:26.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.600+0000 7f9e29600640 1 -- start start 2024-03-06T07:37:26.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e29600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9e2415acb0 0x7f9e240b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e29600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e240b4680 0x7f9e2406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e29600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2415b090 con 0x7f9e24072340 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e29600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240b2a30 con 0x7f9e240b4680 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e29600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240b4360 con 0x7f9e2415acb0 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e23400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9e24072340 0x7f9e240ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40458/0 (socket says 172.21.15.114:40458) 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e23400640 1 -- 172.21.15.114:0/3721901825 learned_addr learned my addr 172.21.15.114:0/3721901825 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.602+0000 7f9e1be00640 1 --2- 172.21.15.114:0/3721901825 >> v2:172.21.15.114:3300/0 conn(0x7f9e240b4680 0x7f9e2406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.603+0000 7f9e22a00640 1 --2- 172.21.15.114:0/3721901825 >> v2:172.21.15.177:3300/0 conn(0x7f9e2415acb0 0x7f9e240b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.603+0000 7f9e22000640 1 -- 172.21.15.114:0/3721901825 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454292080 0 0) 0x7f9e2415b090 con 0x7f9e24072340 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.603+0000 7f9e1be00640 1 -- 172.21.15.114:0/3721901825 >> v2:172.21.15.177:3300/0 conn(0x7f9e2415acb0 msgr2=0x7f9e240b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.603+0000 7f9e1be00640 1 --2- 172.21.15.114:0/3721901825 >> v2:172.21.15.177:3300/0 conn(0x7f9e2415acb0 0x7f9e240b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.603+0000 7f9e1be00640 1 -- 172.21.15.114:0/3721901825 >> v1:172.21.15.114:6789/0 conn(0x7f9e24072340 legacy=0x7f9e240ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.603+0000 7f9e1be00640 1 -- 172.21.15.114:0/3721901825 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240b2cf0 con 0x7f9e240b4680 2024-03-06T07:37:26.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.604+0000 7f9e1be00640 1 --2- 172.21.15.114:0/3721901825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e240b4680 0x7f9e2406e4f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b27f0 tx=0x7f9e0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5e929144589853 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.604+0000 7f9e22000640 1 -- 172.21.15.114:0/3721901825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c035020 con 0x7f9e240b4680 2024-03-06T07:37:26.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.604+0000 7f9e22000640 1 -- 172.21.15.114:0/3721901825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c031830 con 0x7f9e240b4680 2024-03-06T07:37:26.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.604+0000 7f9e22000640 1 -- 172.21.15.114:0/3721901825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c031c00 con 0x7f9e240b4680 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 -- 172.21.15.114:0/3721901825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e240b4680 msgr2=0x7f9e2406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 --2- 172.21.15.114:0/3721901825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e240b4680 0x7f9e2406e4f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b27f0 tx=0x7f9e0c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 -- 172.21.15.114:0/3721901825 shutdown_connections 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 --2- 172.21.15.114:0/3721901825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e240b4680 0x7f9e2406e4f0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 --2- 172.21.15.114:0/3721901825 >> v2:172.21.15.177:3300/0 conn(0x7f9e2415acb0 0x7f9e240b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 -- 172.21.15.114:0/3721901825 >> 172.21.15.114:0/3721901825 conn(0x7f9e240b1730 msgr2=0x7f9e240b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 -- 172.21.15.114:0/3721901825 shutdown_connections 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.605+0000 7f9e29600640 1 -- 172.21.15.114:0/3721901825 wait complete. 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.607+0000 7f9e29600640 1 Processor -- start 2024-03-06T07:37:26.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.607+0000 7f9e29600640 1 -- start start 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e29600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e24072340 0x7f9e24143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e29600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e240b4680 0x7f9e24159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e29600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e2415acb0 0x7f9e24159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e29600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b3d10 con 0x7f9e24072340 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e29600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b32b0 con 0x7f9e240b4680 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e29600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240b4010 con 0x7f9e2415acb0 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e22a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e24072340 0x7f9e24143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e22a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e24072340 0x7f9e24143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59630/0 (socket says 172.21.15.114:59630) 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e22a00640 1 -- 172.21.15.114:0/2986271888 learned_addr learned my addr 172.21.15.114:0/2986271888 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e1be00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e240b4680 0x7f9e24159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e22a00640 1 -- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e2415acb0 msgr2=0x7f9e24159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.608+0000 7f9e23400640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e2415acb0 0x7f9e24159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e22a00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e2415acb0 0x7f9e24159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e22a00640 1 -- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e240b4680 msgr2=0x7f9e24159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e22a00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e240b4680 0x7f9e24159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e22a00640 1 -- 172.21.15.114:0/2986271888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e241439c0 con 0x7f9e24072340 2024-03-06T07:37:26.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e1be00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e240b4680 0x7f9e24159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:26.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e22a00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e24072340 0x7f9e24143480 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400d830 tx=0x7f9e1400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e1aa00640 1 -- 172.21.15.114:0/2986271888 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400b6c0 con 0x7f9e24072340 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e1aa00640 1 -- 172.21.15.114:0/2986271888 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14019070 con 0x7f9e24072340 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e1aa00640 1 -- 172.21.15.114:0/2986271888 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400ba80 con 0x7f9e24072340 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e241f4e00 con 0x7f9e24072340 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.609+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e241f57b0 con 0x7f9e24072340 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.611+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2404f330 con 0x7f9e24072340 2024-03-06T07:37:26.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.612+0000 7f9e1aa00640 1 -- 172.21.15.114:0/2986271888 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e1401e460 con 0x7f9e24072340 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.613+0000 7f9e1aa00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e0006b4d0 0x7f9e0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.613+0000 7f9e1aa00640 1 -- 172.21.15.114:0/2986271888 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e1408a990 con 0x7f9e24072340 2024-03-06T07:37:26.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.613+0000 7f9e1be00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e0006b4d0 0x7f9e0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.613+0000 7f9e1be00640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e0006b4d0 0x7f9e0006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c036000 tx=0x7f9e0c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.622+0000 7f9e1aa00640 1 -- 172.21.15.114:0/2986271888 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e241439c0 con 0x7f9e24072340 2024-03-06T07:37:26.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.947+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e2406eaf0 con 0x7f9e24072340 2024-03-06T07:37:26.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.948+0000 7f9e1aa00640 1 -- 172.21.15.114:0/2986271888 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f9e14059730 con 0x7f9e24072340 2024-03-06T07:37:26.950 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:26.950 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:26.950 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:26.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.950+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e0006b4d0 msgr2=0x7f9e0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.950+0000 7f9e29600640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e0006b4d0 0x7f9e0006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c036000 tx=0x7f9e0c008030 comp rx=0 tx=0).stop 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.950+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e24072340 msgr2=0x7f9e24143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.950+0000 7f9e29600640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e24072340 0x7f9e24143480 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400d830 tx=0x7f9e1400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 shutdown_connections 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e2415acb0 0x7f9e24159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e240b4680 0x7f9e24159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e0006b4d0 0x7f9e0006d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 --2- 172.21.15.114:0/2986271888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e24072340 0x7f9e24143480 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 >> 172.21.15.114:0/2986271888 conn(0x7f9e240b1730 msgr2=0x7f9e240ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 shutdown_connections 2024-03-06T07:37:26.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:26.951+0000 7f9e29600640 1 -- 172.21.15.114:0/2986271888 wait complete. 2024-03-06T07:37:27.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:27.173 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:28.068 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:28.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:28.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.386+0000 7f8758c00640 1 Processor -- start 2024-03-06T07:37:28.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.386+0000 7f8758c00640 1 -- start start 2024-03-06T07:37:28.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.387+0000 7f8758c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8754161580 0x7f87540d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.387+0000 7f8758c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8754161960 con 0x7f87540d1e90 2024-03-06T07:37:28.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.387+0000 7f8758c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87540cf6f0 con 0x7f87540d2a70 2024-03-06T07:37:28.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.387+0000 7f8758c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87540cf9d0 con 0x7f8754161580 2024-03-06T07:37:28.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.387+0000 7f8752a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f87540d1e90 0x7f87540d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38456/0 (socket says 172.21.15.114:38456) 2024-03-06T07:37:28.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.387+0000 7f8752a00640 1 -- 172.21.15.114:0/700241814 learned_addr learned my addr 172.21.15.114:0/700241814 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:28.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794008852 0 0) 0x7f8754161960 con 0x7f87540d1e90 2024-03-06T07:37:28.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8738003520 con 0x7f87540d1e90 2024-03-06T07:37:28.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8753400640 1 --2- 172.21.15.114:0/700241814 >> v2:172.21.15.177:3300/0 conn(0x7f8754161580 0x7f87540d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203353137 0 0) 0x7f8738003520 con 0x7f87540d1e90 2024-03-06T07:37:28.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 >> v2:172.21.15.177:3300/0 conn(0x7f8754161580 msgr2=0x7f87540d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:28.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 --2- 172.21.15.114:0/700241814 >> v2:172.21.15.177:3300/0 conn(0x7f8754161580 0x7f87540d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 >> v1:172.21.15.177:6789/0 conn(0x7f87540d2a70 legacy=0x7f87540b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87540cfcb0 con 0x7f87540d1e90 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8740001670 con 0x7f87540d1e90 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 >> v1:172.21.15.114:6789/0 conn(0x7f87540d1e90 legacy=0x7f87540d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8738004800 0x7f8738004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87380051c0 0x7f873800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87540cf9d0 con 0x7f87380039b0 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873800c060 con 0x7f8738004800 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.388+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873800c260 con 0x7f87380051c0 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8753400640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8752a00640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8738004800 0x7f8738004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8753400640 1 -- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87380051c0 msgr2=0x7f873800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8753400640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87380051c0 0x7f873800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8753400640 1 -- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8738004800 msgr2=0x7f8738004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8753400640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8738004800 0x7f8738004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8753400640 1 -- 172.21.15.114:0/700241814 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738003520 con 0x7f87380039b0 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8753e00640 1 -- 172.21.15.114:0/700241814 reap_dead start 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.389+0000 7f8752a00640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8738004800 0x7f8738004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.390+0000 7f8753400640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8744009ac0 tx=0x7f8738005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a86c2170072060 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.390+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8744039070 con 0x7f87380039b0 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.390+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874402c710 con 0x7f87380039b0 2024-03-06T07:37:28.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.390+0000 7f8752000640 1 -- 172.21.15.114:0/700241814 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8744009690 con 0x7f87380039b0 2024-03-06T07:37:28.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.391+0000 7f8758c00640 1 -- 172.21.15.114:0/700241814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87380039b0 msgr2=0x7f8738003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.391+0000 7f8758c00640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8744009ac0 tx=0x7f8738005830 comp rx=0 tx=0).stop 2024-03-06T07:37:28.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.391+0000 7f8758c00640 1 -- 172.21.15.114:0/700241814 shutdown_connections 2024-03-06T07:37:28.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.391+0000 7f8758c00640 1 --2- 172.21.15.114:0/700241814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.391+0000 7f8758c00640 1 -- 172.21.15.114:0/700241814 >> 172.21.15.114:0/700241814 conn(0x7f87540d3d80 msgr2=0x7f87540d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.391+0000 7f8758c00640 1 -- 172.21.15.114:0/700241814 shutdown_connections 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.391+0000 7f8758c00640 1 -- 172.21.15.114:0/700241814 wait complete. 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.393+0000 7f8758c00640 1 Processor -- start 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.393+0000 7f8758c00640 1 -- start start 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8758c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87380039b0 0x7f87541f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8758c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f875415bee0 0x7f87540b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8758c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87540b78c0 0x7f87540b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8758c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87540cf3a0 con 0x7f875415bee0 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8758c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87540b39a0 con 0x7f87380039b0 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8758c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87540d1950 con 0x7f87540b78c0 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8752a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f875415bee0 0x7f87540b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8752a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f875415bee0 0x7f87540b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44504/0 (socket says 172.21.15.114:44504) 2024-03-06T07:37:28.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.394+0000 7f8752a00640 1 -- 172.21.15.114:0/4183883543 learned_addr learned my addr 172.21.15.114:0/4183883543 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8753400640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87380039b0 0x7f87541f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8752a00640 1 -- 172.21.15.114:0/4183883543 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87540b78c0 msgr2=0x7f87540b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8752a00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87540b78c0 0x7f87540b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8752a00640 1 -- 172.21.15.114:0/4183883543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87380039b0 msgr2=0x7f87541f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8752a00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87380039b0 0x7f87541f4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8752a00640 1 -- 172.21.15.114:0/4183883543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87540cb060 con 0x7f875415bee0 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8752a00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f875415bee0 0x7f87540b7380 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f874000d770 tx=0x7f874000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f874b400640 1 -- 172.21.15.114:0/4183883543 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8740019070 con 0x7f875415bee0 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f874b400640 1 -- 172.21.15.114:0/4183883543 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87400074e0 con 0x7f875415bee0 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f874b400640 1 -- 172.21.15.114:0/4183883543 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87400049e0 con 0x7f875415bee0 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87540cb340 con 0x7f875415bee0 2024-03-06T07:37:28.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.395+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875415c360 con 0x7f875415bee0 2024-03-06T07:37:28.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.397+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875404f330 con 0x7f875415bee0 2024-03-06T07:37:28.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.398+0000 7f874b400640 1 -- 172.21.15.114:0/4183883543 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8740004b80 con 0x7f875415bee0 2024-03-06T07:37:28.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.399+0000 7f874b400640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f873406b500 0x7f873406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.399+0000 7f874b400640 1 -- 172.21.15.114:0/4183883543 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8740089b20 con 0x7f875415bee0 2024-03-06T07:37:28.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.399+0000 7f8753400640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f873406b500 0x7f873406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.399+0000 7f8753400640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f873406b500 0x7f873406d9c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8744036000 tx=0x7f8744030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.407+0000 7f874b400640 1 -- 172.21.15.114:0/4183883543 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87400588c0 con 0x7f875415bee0 2024-03-06T07:37:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.706+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87540d3b60 con 0x7f875415bee0 2024-03-06T07:37:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.707+0000 7f874b400640 1 -- 172.21.15.114:0/4183883543 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f874000de20 con 0x7f875415bee0 2024-03-06T07:37:28.709 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:28.709 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:28.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f873406b500 msgr2=0x7f873406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f873406b500 0x7f873406d9c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8744036000 tx=0x7f8744030000 comp rx=0 tx=0).stop 2024-03-06T07:37:28.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f875415bee0 msgr2=0x7f87540b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f875415bee0 0x7f87540b7380 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f874000d770 tx=0x7f874000dc30 comp rx=0 tx=0).stop 2024-03-06T07:37:28.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 shutdown_connections 2024-03-06T07:37:28.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87540b78c0 0x7f87540b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f873406b500 0x7f873406d9c0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f875415bee0 0x7f87540b7380 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 --2- 172.21.15.114:0/4183883543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87380039b0 0x7f87541f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.709+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 >> 172.21.15.114:0/4183883543 conn(0x7f87540d3d80 msgr2=0x7f875406ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.710+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 shutdown_connections 2024-03-06T07:37:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:28.710+0000 7f8758c00640 1 -- 172.21.15.114:0/4183883543 wait complete. 2024-03-06T07:37:28.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:28.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:29.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:30.070 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.149+0000 7f5535200640 1 Processor -- start 2024-03-06T07:37:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.149+0000 7f5535200640 1 -- start start 2024-03-06T07:37:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.150+0000 7f5535200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 0x7f553015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.150+0000 7f5535200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5530071d30 con 0x7f5530070d20 2024-03-06T07:37:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.150+0000 7f5535200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5530071490 con 0x7f553006e4d0 2024-03-06T07:37:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.150+0000 7f5535200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5530071770 con 0x7f55300af1c0 2024-03-06T07:37:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f5525400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5530070d20 0x7f553006cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38470/0 (socket says 172.21.15.114:38470) 2024-03-06T07:37:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f5525400640 1 -- 172.21.15.114:0/647931962 learned_addr learned my addr 172.21.15.114:0/647931962 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552de00640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 0x7f553015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007165661 0 0) 0x7f5518001110 con 0x7f5530070d20 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5514003520 con 0x7f5530070d20 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116920349 0 0) 0x7f5514003520 con 0x7f5530070d20 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 msgr2=0x7f553015eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552d400640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 0x7f553015eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 >> v1:172.21.15.177:6789/0 conn(0x7f553006e4d0 legacy=0x7f55300b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.151+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5530071a50 con 0x7f5530070d20 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5518001840 con 0x7f5530070d20 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 >> v1:172.21.15.114:6789/0 conn(0x7f5530070d20 legacy=0x7f553006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 0x7f55140039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5514003fb0 0x7f5514004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5530071770 con 0x7f5514004d40 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551400bb00 con 0x7f5514003fb0 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551400bd00 con 0x7f55300af1c0 2024-03-06T07:37:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552e800640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f552de00640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 0x7f55140039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.152+0000 7f5525400640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5514003fb0 0x7f5514004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.153+0000 7f552e800640 1 -- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 msgr2=0x7f55140039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.153+0000 7f552e800640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 0x7f55140039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.153+0000 7f552e800640 1 -- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5514003fb0 msgr2=0x7f5514004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.153+0000 7f552e800640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5514003fb0 0x7f5514004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.153+0000 7f552e800640 1 -- 172.21.15.114:0/647931962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5514003520 con 0x7f5514004d40 2024-03-06T07:37:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.153+0000 7f552de00640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f55300af1c0 0x7f55140039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.154+0000 7f552e800640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55140051c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f5528001750 tx=0x7f5514005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fce92e5a709a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.154+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528001370 con 0x7f5514004d40 2024-03-06T07:37:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.154+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55280052d0 con 0x7f5514004d40 2024-03-06T07:37:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f552d400640 1 -- 172.21.15.114:0/647931962 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528003830 con 0x7f5514004d40 2024-03-06T07:37:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f5535200640 1 -- 172.21.15.114:0/647931962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 msgr2=0x7f55140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f5535200640 1 --2- 172.21.15.114:0/647931962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55140051c0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f5528001750 tx=0x7f5514005830 comp rx=0 tx=0).stop 2024-03-06T07:37:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f552e800640 1 -- 172.21.15.114:0/647931962 reap_dead start 2024-03-06T07:37:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f5535200640 1 -- 172.21.15.114:0/647931962 shutdown_connections 2024-03-06T07:37:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f5535200640 1 -- 172.21.15.114:0/647931962 >> 172.21.15.114:0/647931962 conn(0x7f55300b1730 msgr2=0x7f55300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f5535200640 1 -- 172.21.15.114:0/647931962 shutdown_connections 2024-03-06T07:37:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.155+0000 7f5535200640 1 -- 172.21.15.114:0/647931962 wait complete. 2024-03-06T07:37:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.157+0000 7f5535200640 1 Processor -- start 2024-03-06T07:37:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.157+0000 7f5535200640 1 -- start start 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f5535200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55301e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f5535200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5530148cf0 0x7f55301483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f5535200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f553015ff40 0x7f55301603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f5535200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553006d370 con 0x7f5514004d40 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f5535200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55300711a0 con 0x7f5530148cf0 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f5535200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55300b3ce0 con 0x7f553015ff40 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f552de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55301e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f552de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55301e67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44520/0 (socket says 172.21.15.114:44520) 2024-03-06T07:37:30.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.158+0000 7f552de00640 1 -- 172.21.15.114:0/706512498 learned_addr learned my addr 172.21.15.114:0/706512498 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f552de00640 1 -- 172.21.15.114:0/706512498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f553015ff40 msgr2=0x7f55301603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f552de00640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f553015ff40 0x7f55301603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f552de00640 1 -- 172.21.15.114:0/706512498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5530148cf0 msgr2=0x7f55301483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f552de00640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5530148cf0 0x7f55301483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f552de00640 1 -- 172.21.15.114:0/706512498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55301491d0 con 0x7f5514004d40 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f552de00640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55301e67e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f55200101c0 tx=0x7f5520010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f5526a00640 1 -- 172.21.15.114:0/706512498 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552001a030 con 0x7f5514004d40 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f5526a00640 1 -- 172.21.15.114:0/706512498 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5520003990 con 0x7f5514004d40 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f5526a00640 1 -- 172.21.15.114:0/706512498 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520003d60 con 0x7f5514004d40 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.159+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55301494b0 con 0x7f5514004d40 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.160+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5530149a70 con 0x7f5514004d40 2024-03-06T07:37:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.161+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553004f330 con 0x7f5514004d40 2024-03-06T07:37:30.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.162+0000 7f5526a00640 1 -- 172.21.15.114:0/706512498 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f552001d070 con 0x7f5514004d40 2024-03-06T07:37:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.163+0000 7f5526a00640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f550c06b500 0x7f550c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.163+0000 7f5526a00640 1 -- 172.21.15.114:0/706512498 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f552008b1b0 con 0x7f5514004d40 2024-03-06T07:37:30.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.164+0000 7f5525400640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f550c06b500 0x7f550c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:30.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.164+0000 7f5525400640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f550c06b500 0x7f550c06d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f55180013d0 tx=0x7f5518000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.169+0000 7f5526a00640 1 -- 172.21.15.114:0/706512498 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5520059f50 con 0x7f5514004d40 2024-03-06T07:37:30.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.391+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55300b3600 con 0x7f5514004d40 2024-03-06T07:37:30.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.391+0000 7f5526a00640 1 -- 172.21.15.114:0/706512498 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f5520059850 con 0x7f5514004d40 2024-03-06T07:37:30.393 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:30.393 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:30.393 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f550c06b500 msgr2=0x7f550c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f550c06b500 0x7f550c06d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f55180013d0 tx=0x7f5518000ed0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 msgr2=0x7f55301e67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55301e67e0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f55200101c0 tx=0x7f5520010680 comp rx=0 tx=0).stop 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 shutdown_connections 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f553015ff40 0x7f55301603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5530148cf0 0x7f55301483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f550c06b500 0x7f550c06d9c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 --2- 172.21.15.114:0/706512498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514004d40 0x7f55301e67e0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 >> 172.21.15.114:0/706512498 conn(0x7f55300b1730 msgr2=0x7f55300b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 shutdown_connections 2024-03-06T07:37:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:30.394+0000 7f5535200640 1 -- 172.21.15.114:0/706512498 wait complete. 2024-03-06T07:37:30.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:30.621 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:31.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:31.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.835+0000 7faa1fa00640 1 Processor -- start 2024-03-06T07:37:31.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.835+0000 7faa1fa00640 1 -- start start 2024-03-06T07:37:31.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.837+0000 7faa1fa00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7faa1815dff0 0x7faa1815e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.837+0000 7faa1fa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa180b35b0 con 0x7faa180cf430 2024-03-06T07:37:31.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.837+0000 7faa1fa00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa180b3890 con 0x7faa18151660 2024-03-06T07:37:31.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.837+0000 7faa1fa00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa180b3ee0 con 0x7faa1815dff0 2024-03-06T07:37:31.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.837+0000 7faa17e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7faa180cf430 0x7faa1806d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38474/0 (socket says 172.21.15.114:38474) 2024-03-06T07:37:31.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.837+0000 7faa17e00640 1 -- 172.21.15.114:0/1945160496 learned_addr learned my addr 172.21.15.114:0/1945160496 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:31.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182638474 0 0) 0x7faa180b35b0 con 0x7faa180cf430 2024-03-06T07:37:31.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa1da00640 1 --2- 172.21.15.114:0/1945160496 >> v2:172.21.15.177:3300/0 conn(0x7faa1815dff0 0x7faa1815e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa00003440 con 0x7faa180cf430 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565043124 0 0) 0x7faa00003440 con 0x7faa180cf430 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 >> v2:172.21.15.177:3300/0 conn(0x7faa1815dff0 msgr2=0x7faa1815e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 --2- 172.21.15.114:0/1945160496 >> v2:172.21.15.177:3300/0 conn(0x7faa1815dff0 0x7faa1815e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 >> v1:172.21.15.177:6789/0 conn(0x7faa18151660 legacy=0x7faa18071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180b3b70 con 0x7faa180cf430 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7faa04001670 con 0x7faa180cf430 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 >> v1:172.21.15.114:6789/0 conn(0x7faa180cf430 legacy=0x7faa1806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa00003aa0 0x7faa00003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa000088f0 0x7faa0000ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa0000dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180b3ee0 con 0x7faa0000b600 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0000e3a0 con 0x7faa000088f0 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.838+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0000e5a0 con 0x7faa00003aa0 2024-03-06T07:37:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.839+0000 7faa1da00640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa0000dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.839+0000 7faa1da00640 1 -- 172.21.15.114:0/1945160496 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa00003aa0 msgr2=0x7faa00003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.839+0000 7faa1da00640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa00003aa0 0x7faa00003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.839+0000 7faa1da00640 1 -- 172.21.15.114:0/1945160496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa000088f0 msgr2=0x7faa0000ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.839+0000 7faa1da00640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa000088f0 0x7faa0000ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.839+0000 7faa1da00640 1 -- 172.21.15.114:0/1945160496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa00003440 con 0x7faa0000b600 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.840+0000 7faa1da00640 1 -- 172.21.15.114:0/1945160496 reap_dead start 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.840+0000 7faa1da00640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa0000dac0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7faa10009fc0 tx=0x7faa00003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b00c4fc330683d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.840+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10009c10 con 0x7faa0000b600 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.840+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1002cc60 con 0x7faa0000b600 2024-03-06T07:37:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.840+0000 7faa17400640 1 -- 172.21.15.114:0/1945160496 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10003bd0 con 0x7faa0000b600 2024-03-06T07:37:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.841+0000 7faa1fa00640 1 -- 172.21.15.114:0/1945160496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 msgr2=0x7faa0000dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.841+0000 7faa1fa00640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa0000dac0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7faa10009fc0 tx=0x7faa00003730 comp rx=0 tx=0).stop 2024-03-06T07:37:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.841+0000 7faa1fa00640 1 -- 172.21.15.114:0/1945160496 shutdown_connections 2024-03-06T07:37:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.841+0000 7faa1fa00640 1 --2- 172.21.15.114:0/1945160496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa0000dac0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.841+0000 7faa1fa00640 1 -- 172.21.15.114:0/1945160496 >> 172.21.15.114:0/1945160496 conn(0x7faa180b1730 msgr2=0x7faa180b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.841+0000 7faa1fa00640 1 -- 172.21.15.114:0/1945160496 shutdown_connections 2024-03-06T07:37:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.841+0000 7faa1fa00640 1 -- 172.21.15.114:0/1945160496 wait complete. 2024-03-06T07:37:31.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.843+0000 7faa1fa00640 1 Processor -- start 2024-03-06T07:37:31.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.843+0000 7faa1fa00640 1 -- start start 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1fa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa180cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1fa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa181479f0 0x7faa181efae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1fa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa181f0020 0x7faa1814e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1fa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1815e600 con 0x7faa0000b600 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1fa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1815e480 con 0x7faa181479f0 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1fa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1815e780 con 0x7faa181f0020 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa180cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa180cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44546/0 (socket says 172.21.15.114:44546) 2024-03-06T07:37:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.844+0000 7faa1d000640 1 -- 172.21.15.114:0/4272358801 learned_addr learned my addr 172.21.15.114:0/4272358801 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1d000640 1 -- 172.21.15.114:0/4272358801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa181f0020 msgr2=0x7faa1814e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1d000640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa181f0020 0x7faa1814e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1d000640 1 -- 172.21.15.114:0/4272358801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa181479f0 msgr2=0x7faa181efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1d000640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa181479f0 0x7faa181efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1d000640 1 -- 172.21.15.114:0/4272358801 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa181f0570 con 0x7faa0000b600 2024-03-06T07:37:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1d000640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa180cd120 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c480 tx=0x7faa0c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa14c00640 1 -- 172.21.15.114:0/4272358801 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0052d0 con 0x7faa0000b600 2024-03-06T07:37:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa14c00640 1 -- 172.21.15.114:0/4272358801 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c005470 con 0x7faa0000b600 2024-03-06T07:37:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa14c00640 1 -- 172.21.15.114:0/4272358801 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c012870 con 0x7faa0000b600 2024-03-06T07:37:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa181f0850 con 0x7faa0000b600 2024-03-06T07:37:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.845+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa180ccc00 con 0x7faa0000b600 2024-03-06T07:37:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.846+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1804f330 con 0x7faa0000b600 2024-03-06T07:37:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.847+0000 7faa14c00640 1 -- 172.21.15.114:0/4272358801 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa0c012a10 con 0x7faa0000b600 2024-03-06T07:37:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.849+0000 7faa14c00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9f806b500 0x7fa9f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.849+0000 7faa14c00640 1 -- 172.21.15.114:0/4272358801 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faa0c089ea0 con 0x7faa0000b600 2024-03-06T07:37:31.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.849+0000 7faa17e00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9f806b500 0x7fa9f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.850+0000 7faa17e00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9f806b500 0x7fa9f806d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7faa04001eb0 tx=0x7faa04001a60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:31.852+0000 7faa14c00640 1 -- 172.21.15.114:0/4272358801 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa0c058cd0 con 0x7faa0000b600 2024-03-06T07:37:32.072 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:32.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.126+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa180cbff0 con 0x7faa0000b600 2024-03-06T07:37:32.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.127+0000 7faa14c00640 1 -- 172.21.15.114:0/4272358801 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7faa0c0585d0 con 0x7faa0000b600 2024-03-06T07:37:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:32.129 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:32.130 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:32.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9f806b500 msgr2=0x7fa9f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:32.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9f806b500 0x7fa9f806d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7faa04001eb0 tx=0x7faa04001a60 comp rx=0 tx=0).stop 2024-03-06T07:37:32.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 msgr2=0x7faa180cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:32.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa180cd120 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c480 tx=0x7faa0c00c940 comp rx=0 tx=0).stop 2024-03-06T07:37:32.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 shutdown_connections 2024-03-06T07:37:32.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faa181f0020 0x7faa1814e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa181479f0 0x7faa181efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa9f806b500 0x7fa9f806d9c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 --2- 172.21.15.114:0/4272358801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa0000b600 0x7faa180cd120 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:32.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 >> 172.21.15.114:0/4272358801 conn(0x7faa180b1730 msgr2=0x7faa180b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:32.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 shutdown_connections 2024-03-06T07:37:32.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:32.130+0000 7faa1fa00640 1 -- 172.21.15.114:0/4272358801 wait complete. 2024-03-06T07:37:32.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:32.350 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:33.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:33.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.566+0000 7f592be00640 1 Processor -- start 2024-03-06T07:37:33.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.566+0000 7f592be00640 1 -- start start 2024-03-06T07:37:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.567+0000 7f592be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f592c161580 0x7f592c0d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.567+0000 7f592be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f592c0d1e90 0x7f592c0d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.567+0000 7f592be00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592c161960 con 0x7f592c0d28a0 2024-03-06T07:37:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.567+0000 7f592be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592c0cf6f0 con 0x7f592c161580 2024-03-06T07:37:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.567+0000 7f592be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0cf9d0 con 0x7f592c0d1e90 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.568+0000 7f592b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f592c0d28a0 0x7f592c0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38478/0 (socket says 172.21.15.114:38478) 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.568+0000 7f592b400640 1 -- 172.21.15.114:0/985136613 learned_addr learned my addr 172.21.15.114:0/985136613 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.568+0000 7f5922000640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f592c0d1e90 0x7f592c0d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.568+0000 7f592aa00640 1 --2- 172.21.15.114:0/985136613 >> v2:172.21.15.177:3300/0 conn(0x7f592c161580 0x7f592c0d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.568+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063181078 0 0) 0x7f592c161960 con 0x7f592c0d28a0 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.568+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5910003520 con 0x7f592c0d28a0 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.568+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3662379229 0 0) 0x7f5910003520 con 0x7f592c0d28a0 2024-03-06T07:37:33.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f592c0d1e90 msgr2=0x7f592c0d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f592c0d1e90 0x7f592c0d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 >> v2:172.21.15.177:3300/0 conn(0x7f592c161580 msgr2=0x7f592c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 --2- 172.21.15.114:0/985136613 >> v2:172.21.15.177:3300/0 conn(0x7f592c161580 0x7f592c0d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0cfcb0 con 0x7f592c0d28a0 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5924001670 con 0x7f592c0d28a0 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 >> v1:172.21.15.114:6789/0 conn(0x7f592c0d28a0 legacy=0x7f592c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59100039b0 0x7f5910003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f592c0d1e90 0x7f5910004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5910004d40 0x7f59100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0cf6f0 con 0x7f59100039b0 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0cf9d0 con 0x7f5910004d40 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591000b9c0 con 0x7f592c0d1e90 2024-03-06T07:37:33.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592aa00640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59100039b0 0x7f5910003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592aa00640 1 -- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f592c0d1e90 msgr2=0x7f5910004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:33.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592aa00640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f592c0d1e90 0x7f5910004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592aa00640 1 -- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5910004d40 msgr2=0x7f59100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:33.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592aa00640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5910004d40 0x7f59100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.569+0000 7f592aa00640 1 -- 172.21.15.114:0/985136613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910003520 con 0x7f59100039b0 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.570+0000 7f592aa00640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59100039b0 0x7f5910003dd0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f591c00e170 tx=0x7f5910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20621f09923f79c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c039070 con 0x7f59100039b0 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c02cc20 con 0x7f59100039b0 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592a000640 1 -- 172.21.15.114:0/985136613 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c009c10 con 0x7f59100039b0 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592be00640 1 -- 172.21.15.114:0/985136613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59100039b0 msgr2=0x7f5910003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592be00640 1 --2- 172.21.15.114:0/985136613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59100039b0 0x7f5910003dd0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f591c00e170 tx=0x7f5910005830 comp rx=0 tx=0).stop 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592b400640 1 -- 172.21.15.114:0/985136613 reap_dead start 2024-03-06T07:37:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592be00640 1 -- 172.21.15.114:0/985136613 shutdown_connections 2024-03-06T07:37:33.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592be00640 1 -- 172.21.15.114:0/985136613 >> 172.21.15.114:0/985136613 conn(0x7f592c0d3d80 msgr2=0x7f592c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:33.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592be00640 1 -- 172.21.15.114:0/985136613 shutdown_connections 2024-03-06T07:37:33.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.571+0000 7f592be00640 1 -- 172.21.15.114:0/985136613 wait complete. 2024-03-06T07:37:33.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.573+0000 7f592be00640 1 Processor -- start 2024-03-06T07:37:33.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.573+0000 7f592be00640 1 -- start start 2024-03-06T07:37:33.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59100039b0 0x7f592c160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f592c157bc0 0x7f592c160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f592c0b8220 0x7f592c0b86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0cf3a0 con 0x7f592c0b8220 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0b39a0 con 0x7f592c157bc0 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592c0b4ed0 con 0x7f59100039b0 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f592c0b8220 0x7f592c0b86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f592c0b8220 0x7f592c0b86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44564/0 (socket says 172.21.15.114:44564) 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.574+0000 7f592b400640 1 -- 172.21.15.114:0/534345557 learned_addr learned my addr 172.21.15.114:0/534345557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f5922000640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f592c157bc0 0x7f592c160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f592b400640 1 -- 172.21.15.114:0/534345557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59100039b0 msgr2=0x7f592c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f592b400640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59100039b0 0x7f592c160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f592b400640 1 -- 172.21.15.114:0/534345557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f592c157bc0 msgr2=0x7f592c160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f592b400640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f592c157bc0 0x7f592c160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f592b400640 1 -- 172.21.15.114:0/534345557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0b8be0 con 0x7f592c0b8220 2024-03-06T07:37:33.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f592b400640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f592c0b8220 0x7f592c0b86a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f592c08cb60 tx=0x7f592400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:33.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f5923400640 1 -- 172.21.15.114:0/534345557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924007230 con 0x7f592c0b8220 2024-03-06T07:37:33.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f5923400640 1 -- 172.21.15.114:0/534345557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59240073d0 con 0x7f592c0b8220 2024-03-06T07:37:33.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f5923400640 1 -- 172.21.15.114:0/534345557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59240047e0 con 0x7f592c0b8220 2024-03-06T07:37:33.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.575+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592c0ccd10 con 0x7f592c0b8220 2024-03-06T07:37:33.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.576+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c0cd2d0 con 0x7f592c0b8220 2024-03-06T07:37:33.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.577+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592c04f330 con 0x7f592c0b8220 2024-03-06T07:37:33.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.578+0000 7f5923400640 1 -- 172.21.15.114:0/534345557 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5924004980 con 0x7f592c0b8220 2024-03-06T07:37:33.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.579+0000 7f5923400640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f590806b500 0x7f590806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.579+0000 7f5923400640 1 -- 172.21.15.114:0/534345557 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f59240894b0 con 0x7f592c0b8220 2024-03-06T07:37:33.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.579+0000 7f592aa00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f590806b500 0x7f590806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.580+0000 7f592aa00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f590806b500 0x7f590806d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f591c036000 tx=0x7f591c02e070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:33.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.584+0000 7f5923400640 1 -- 172.21.15.114:0/534345557 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f592401a030 con 0x7f592c0b8220 2024-03-06T07:37:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.821+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f592c0d3b60 con 0x7f592c0b8220 2024-03-06T07:37:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.821+0000 7f5923400640 1 -- 172.21.15.114:0/534345557 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f5924058250 con 0x7f592c0b8220 2024-03-06T07:37:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:33.823 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f590806b500 msgr2=0x7f590806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f590806b500 0x7f590806d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f591c036000 tx=0x7f591c02e070 comp rx=0 tx=0).stop 2024-03-06T07:37:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f592c0b8220 msgr2=0x7f592c0b86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f592c0b8220 0x7f592c0b86a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f592c08cb60 tx=0x7f592400cb90 comp rx=0 tx=0).stop 2024-03-06T07:37:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 shutdown_connections 2024-03-06T07:37:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f590806b500 0x7f590806d9c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f592c0b8220 0x7f592c0b86a0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f592c157bc0 0x7f592c160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 --2- 172.21.15.114:0/534345557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f59100039b0 0x7f592c160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 >> 172.21.15.114:0/534345557 conn(0x7f592c0d3d80 msgr2=0x7f592c06cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 shutdown_connections 2024-03-06T07:37:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:33.824+0000 7f592be00640 1 -- 172.21.15.114:0/534345557 wait complete. 2024-03-06T07:37:34.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:34.053 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:34.074 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:35.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:35.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.272+0000 7fcedac00640 1 Processor -- start 2024-03-06T07:37:35.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.272+0000 7fcedac00640 1 -- start start 2024-03-06T07:37:35.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.273+0000 7fcedac00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fced415acb0 0x7fced40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.273+0000 7fcedac00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fced40b4680 0x7fced406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.273+0000 7fcedac00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced415b090 con 0x7fced4072340 2024-03-06T07:37:35.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.273+0000 7fcedac00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced40b2a30 con 0x7fced40b4680 2024-03-06T07:37:35.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.273+0000 7fcedac00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced40b4360 con 0x7fced415acb0 2024-03-06T07:37:35.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fced8c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fced4072340 0x7fced40ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38484/0 (socket says 172.21.15.114:38484) 2024-03-06T07:37:35.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fcecb400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fced40b4680 0x7fced406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fced8c00640 1 -- 172.21.15.114:0/73590266 learned_addr learned my addr 172.21.15.114:0/73590266 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:35.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fced3e00640 1 --2- 172.21.15.114:0/73590266 >> v2:172.21.15.177:3300/0 conn(0x7fced415acb0 0x7fced40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fcecb400640 1 -- 172.21.15.114:0/73590266 >> v2:172.21.15.177:3300/0 conn(0x7fced415acb0 msgr2=0x7fced40b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fcecb400640 1 --2- 172.21.15.114:0/73590266 >> v2:172.21.15.177:3300/0 conn(0x7fced415acb0 0x7fced40b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fcecb400640 1 -- 172.21.15.114:0/73590266 >> v1:172.21.15.114:6789/0 conn(0x7fced4072340 legacy=0x7fced40ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fcecb400640 1 -- 172.21.15.114:0/73590266 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced40b2cf0 con 0x7fced40b4680 2024-03-06T07:37:35.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.274+0000 7fced3400640 1 -- 172.21.15.114:0/73590266 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626179563 0 0) 0x7fced415b090 con 0x7fced4072340 2024-03-06T07:37:35.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.276+0000 7fcecb400640 1 --2- 172.21.15.114:0/73590266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced40b4680 0x7fced406e4f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fced40b27f0 tx=0x7fcebc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468b290f1a3dda4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.276+0000 7fced3400640 1 -- 172.21.15.114:0/73590266 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc035020 con 0x7fced40b4680 2024-03-06T07:37:35.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.276+0000 7fced3400640 1 -- 172.21.15.114:0/73590266 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc031830 con 0x7fced40b4680 2024-03-06T07:37:35.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.276+0000 7fced3400640 1 -- 172.21.15.114:0/73590266 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc031c00 con 0x7fced40b4680 2024-03-06T07:37:35.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.276+0000 7fcedac00640 1 -- 172.21.15.114:0/73590266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced40b4680 msgr2=0x7fced406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.276+0000 7fcedac00640 1 --2- 172.21.15.114:0/73590266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced40b4680 0x7fced406e4f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fced40b27f0 tx=0x7fcebc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.277+0000 7fcedac00640 1 -- 172.21.15.114:0/73590266 shutdown_connections 2024-03-06T07:37:35.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.277+0000 7fcedac00640 1 --2- 172.21.15.114:0/73590266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced40b4680 0x7fced406e4f0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.277+0000 7fcedac00640 1 --2- 172.21.15.114:0/73590266 >> v2:172.21.15.177:3300/0 conn(0x7fced415acb0 0x7fced40b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.277+0000 7fcedac00640 1 -- 172.21.15.114:0/73590266 >> 172.21.15.114:0/73590266 conn(0x7fced40b1730 msgr2=0x7fced40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:35.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.277+0000 7fcedac00640 1 -- 172.21.15.114:0/73590266 shutdown_connections 2024-03-06T07:37:35.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.277+0000 7fcedac00640 1 -- 172.21.15.114:0/73590266 wait complete. 2024-03-06T07:37:35.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.279+0000 7fcedac00640 1 Processor -- start 2024-03-06T07:37:35.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.279+0000 7fcedac00640 1 -- start start 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fcedac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced4072340 0x7fced41521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fcedac00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fced40b4680 0x7fced41f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fcedac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced415acb0 0x7fced41534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fcedac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced40b3d10 con 0x7fced415acb0 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fcedac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced40b32b0 con 0x7fced4072340 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fcedac00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced40b4010 con 0x7fced40b4680 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fced8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced415acb0 0x7fced41534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fced8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced415acb0 0x7fced41534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44580/0 (socket says 172.21.15.114:44580) 2024-03-06T07:37:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.280+0000 7fced8c00640 1 -- 172.21.15.114:0/1923593186 learned_addr learned my addr 172.21.15.114:0/1923593186 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:35.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced8c00640 1 -- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fced40b4680 msgr2=0x7fced41f0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fcecb400640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fced40b4680 0x7fced41f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced3e00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced4072340 0x7fced41521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced8c00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fced40b4680 0x7fced41f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced8c00640 1 -- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced4072340 msgr2=0x7fced41521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced8c00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced4072340 0x7fced41521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced8c00640 1 -- 172.21.15.114:0/1923593186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4152840 con 0x7fced415acb0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced8c00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced415acb0 0x7fced41534d0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00c410 tx=0x7fcecc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced0c00640 1 -- 172.21.15.114:0/1923593186 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc0074e0 con 0x7fced415acb0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced0c00640 1 -- 172.21.15.114:0/1923593186 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc0047e0 con 0x7fced415acb0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced4152b20 con 0x7fced415acb0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.281+0000 7fced0c00640 1 -- 172.21.15.114:0/1923593186 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc004bd0 con 0x7fced415acb0 2024-03-06T07:37:35.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.282+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced415b5c0 con 0x7fced415acb0 2024-03-06T07:37:35.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.283+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced404f330 con 0x7fced415acb0 2024-03-06T07:37:35.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.284+0000 7fced0c00640 1 -- 172.21.15.114:0/1923593186 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcecc011070 con 0x7fced415acb0 2024-03-06T07:37:35.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.285+0000 7fced0c00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fceb006b4d0 0x7fceb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.285+0000 7fced0c00640 1 -- 172.21.15.114:0/1923593186 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcecc089d60 con 0x7fced415acb0 2024-03-06T07:37:35.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.286+0000 7fced3e00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fceb006b4d0 0x7fceb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.286+0000 7fced3e00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fceb006b4d0 0x7fceb006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009f30 tx=0x7fcec400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.290+0000 7fced0c00640 1 -- 172.21.15.114:0/1923593186 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcecc058c30 con 0x7fced415acb0 2024-03-06T07:37:35.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.527+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fced406ce60 con 0x7fced415acb0 2024-03-06T07:37:35.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.528+0000 7fced0c00640 1 -- 172.21.15.114:0/1923593186 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fcecc058530 con 0x7fced415acb0 2024-03-06T07:37:35.529 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:35.530 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:35.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fceb006b4d0 msgr2=0x7fceb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fceb006b4d0 0x7fceb006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009f30 tx=0x7fcec400b040 comp rx=0 tx=0).stop 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced415acb0 msgr2=0x7fced41534d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced415acb0 0x7fced41534d0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00c410 tx=0x7fcecc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 shutdown_connections 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fceb006b4d0 0x7fceb006d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced415acb0 0x7fced41534d0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fced40b4680 0x7fced41f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 --2- 172.21.15.114:0/1923593186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced4072340 0x7fced41521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.530+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 >> 172.21.15.114:0/1923593186 conn(0x7fced40b1730 msgr2=0x7fced4071a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.531+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 shutdown_connections 2024-03-06T07:37:35.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:35.531+0000 7fcedac00640 1 -- 172.21.15.114:0/1923593186 wait complete. 2024-03-06T07:37:35.752 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:35.752 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:36.076 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:36.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:36.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.966+0000 7f1321400640 1 Processor -- start 2024-03-06T07:37:36.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.967+0000 7f1321400640 1 -- start start 2024-03-06T07:37:36.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.969+0000 7f1321400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c15e160 0x7f131c162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.969+0000 7f1321400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f131c0af820 0x7f131c0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.969+0000 7f1321400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c162b60 con 0x7f131c0b38f0 2024-03-06T07:37:36.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f1321400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131c06b0f0 con 0x7f131c0af820 2024-03-06T07:37:36.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f1321400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0b5620 con 0x7f131c15e160 2024-03-06T07:37:36.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f131b200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f131c0b38f0 0x7f131c06d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38492/0 (socket says 172.21.15.114:38492) 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f131b200640 1 -- 172.21.15.114:0/3854737007 learned_addr learned my addr 172.21.15.114:0/3854737007 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f1319e00640 1 --2- 172.21.15.114:0/3854737007 >> v2:172.21.15.114:3300/0 conn(0x7f131c0af820 0x7f131c0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f131a800640 1 --2- 172.21.15.114:0/3854737007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c15e160 0x7f131c162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f1319400640 1 -- 172.21.15.114:0/3854737007 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205338401 0 0) 0x7f1314001110 con 0x7f131c0b38f0 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.970+0000 7f1319400640 1 -- 172.21.15.114:0/3854737007 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12fc003520 con 0x7f131c0b38f0 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.971+0000 7f1319e00640 1 -- 172.21.15.114:0/3854737007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c15e160 msgr2=0x7f131c162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.971+0000 7f1319e00640 1 --2- 172.21.15.114:0/3854737007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c15e160 0x7f131c162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.971+0000 7f1319e00640 1 -- 172.21.15.114:0/3854737007 >> v1:172.21.15.114:6789/0 conn(0x7f131c0b38f0 legacy=0x7f131c06d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.971+0000 7f1319e00640 1 -- 172.21.15.114:0/3854737007 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c0b3d70 con 0x7f131c0af820 2024-03-06T07:37:36.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.972+0000 7f1319e00640 1 --2- 172.21.15.114:0/3854737007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c0b33b0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f1304009d90 tx=0x7f130402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d28e4f65b12c02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:36.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.972+0000 7f1319400640 1 -- 172.21.15.114:0/3854737007 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304039070 con 0x7f131c0af820 2024-03-06T07:37:36.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.972+0000 7f1319400640 1 -- 172.21.15.114:0/3854737007 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130402cbc0 con 0x7f131c0af820 2024-03-06T07:37:36.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.972+0000 7f1319400640 1 -- 172.21.15.114:0/3854737007 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304034440 con 0x7f131c0af820 2024-03-06T07:37:36.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 -- 172.21.15.114:0/3854737007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 msgr2=0x7f131c0b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 --2- 172.21.15.114:0/3854737007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c0b33b0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f1304009d90 tx=0x7f130402be10 comp rx=0 tx=0).stop 2024-03-06T07:37:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 -- 172.21.15.114:0/3854737007 shutdown_connections 2024-03-06T07:37:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 --2- 172.21.15.114:0/3854737007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c0b33b0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 --2- 172.21.15.114:0/3854737007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c15e160 0x7f131c162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 -- 172.21.15.114:0/3854737007 >> 172.21.15.114:0/3854737007 conn(0x7f131c0d0340 msgr2=0x7f131c0d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 -- 172.21.15.114:0/3854737007 shutdown_connections 2024-03-06T07:37:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.973+0000 7f1321400640 1 -- 172.21.15.114:0/3854737007 wait complete. 2024-03-06T07:37:36.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.975+0000 7f1321400640 1 Processor -- start 2024-03-06T07:37:36.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.975+0000 7f1321400640 1 -- start start 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.976+0000 7f1321400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c14ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.976+0000 7f1321400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c0b38f0 0x7f131c0b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.976+0000 7f1321400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f131c15e160 0x7f131c0b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.976+0000 7f1321400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c06ec30 con 0x7f131c0af820 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.976+0000 7f1321400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0afce0 con 0x7f131c15e160 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.976+0000 7f1321400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131c06ef10 con 0x7f131c0b38f0 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c14ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c14ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44592/0 (socket says 172.21.15.114:44592) 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 -- 172.21.15.114:0/3902564536 learned_addr learned my addr 172.21.15.114:0/3902564536 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f1319e00640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c0b38f0 0x7f131c0b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131b200640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f131c15e160 0x7f131c0b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 -- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c0b38f0 msgr2=0x7f131c0b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c0b38f0 0x7f131c0b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 -- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f131c15e160 msgr2=0x7f131c0b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f131c15e160 0x7f131c0b1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.977+0000 7f131a800640 1 -- 172.21.15.114:0/3902564536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c0cfa60 con 0x7f131c0af820 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.978+0000 7f131a800640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c14ada0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f130c00e9b0 tx=0x7f130c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.978+0000 7f1312a00640 1 -- 172.21.15.114:0/3902564536 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c017070 con 0x7f131c0af820 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.978+0000 7f1312a00640 1 -- 172.21.15.114:0/3902564536 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c00c9a0 con 0x7f131c0af820 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.978+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c15e670 con 0x7f131c0af820 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.978+0000 7f1312a00640 1 -- 172.21.15.114:0/3902564536 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c00cd90 con 0x7f131c0af820 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.978+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c15ef10 con 0x7f131c0af820 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.979+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c15e840 con 0x7f131c0af820 2024-03-06T07:37:36.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.980+0000 7f1312a00640 1 -- 172.21.15.114:0/3902564536 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f130c01a070 con 0x7f131c0af820 2024-03-06T07:37:36.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.981+0000 7f1312a00640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f12f806b4d0 0x7f12f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.981+0000 7f1312a00640 1 -- 172.21.15.114:0/3902564536 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f130c08a250 con 0x7f131c0af820 2024-03-06T07:37:36.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.981+0000 7f1319e00640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f12f806b4d0 0x7f12f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.982+0000 7f1319e00640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f12f806b4d0 0x7f12f806d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1304036000 tx=0x7f1304030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:36.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:36.986+0000 7f1312a00640 1 -- 172.21.15.114:0/3902564536 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f130c059080 con 0x7f131c0af820 2024-03-06T07:37:37.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.230+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f131c04f330 con 0x7f131c0af820 2024-03-06T07:37:37.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.231+0000 7f1312a00640 1 -- 172.21.15.114:0/3902564536 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f130c058980 con 0x7f131c0af820 2024-03-06T07:37:37.233 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:37.233 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:37.233 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f12f806b4d0 msgr2=0x7f12f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f12f806b4d0 0x7f12f806d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1304036000 tx=0x7f1304030000 comp rx=0 tx=0).stop 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 msgr2=0x7f131c14ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c14ada0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f130c00e9b0 tx=0x7f130c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 shutdown_connections 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f131c15e160 0x7f131c0b1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f131c0b38f0 0x7f131c0b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f12f806b4d0 0x7f12f806d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 --2- 172.21.15.114:0/3902564536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c0af820 0x7f131c14ada0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 >> 172.21.15.114:0/3902564536 conn(0x7f131c0d0340 msgr2=0x7f131c071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 shutdown_connections 2024-03-06T07:37:37.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:37.234+0000 7f1321400640 1 -- 172.21.15.114:0/3902564536 wait complete. 2024-03-06T07:37:37.454 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:37.454 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:38.079 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:38.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.695+0000 7fbd5d800640 1 Processor -- start 2024-03-06T07:37:38.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.695+0000 7fbd5d800640 1 -- start start 2024-03-06T07:37:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.697+0000 7fbd5d800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd581561f0 0x7fbd580ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.697+0000 7fbd5d800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbd580cc9d0 0x7fbd5815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.697+0000 7fbd5d800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5815c240 con 0x7fbd580aebf0 2024-03-06T07:37:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.697+0000 7fbd5d800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5815ce30 con 0x7fbd580cc9d0 2024-03-06T07:37:38.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.697+0000 7fbd5d800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5815c500 con 0x7fbd581561f0 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.698+0000 7fbd57e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd581561f0 0x7fbd580ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.698+0000 7fbd5ce00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbd580cc9d0 0x7fbd5815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.698+0000 7fbd57400640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fbd580aebf0 0x7fbd580cc2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:48522/0 (socket says 172.21.15.114:48522) 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.698+0000 7fbd57400640 1 -- 172.21.15.114:0/2119929159 learned_addr learned my addr 172.21.15.114:0/2119929159 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.698+0000 7fbd5ce00640 1 -- 172.21.15.114:0/2119929159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd581561f0 msgr2=0x7fbd580ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.698+0000 7fbd5ce00640 1 --2- 172.21.15.114:0/2119929159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd581561f0 0x7fbd580ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.698+0000 7fbd5ce00640 1 -- 172.21.15.114:0/2119929159 >> v1:172.21.15.177:6789/0 conn(0x7fbd580aebf0 legacy=0x7fbd580cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.699+0000 7fbd5ce00640 1 -- 172.21.15.114:0/2119929159 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5815c7e0 con 0x7fbd580cc9d0 2024-03-06T07:37:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.700+0000 7fbd5ce00640 1 --2- 172.21.15.114:0/2119929159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd580cc9d0 0x7fbd5815f830 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009d90 tx=0x7fbd5002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfe9285a490ecbb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.700+0000 7fbd56a00640 1 -- 172.21.15.114:0/2119929159 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50039070 con 0x7fbd580cc9d0 2024-03-06T07:37:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.700+0000 7fbd56a00640 1 -- 172.21.15.114:0/2119929159 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5002cc50 con 0x7fbd580cc9d0 2024-03-06T07:37:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.700+0000 7fbd56a00640 1 -- 172.21.15.114:0/2119929159 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50034440 con 0x7fbd580cc9d0 2024-03-06T07:37:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 -- 172.21.15.114:0/2119929159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd580cc9d0 msgr2=0x7fbd5815f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 --2- 172.21.15.114:0/2119929159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd580cc9d0 0x7fbd5815f830 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009d90 tx=0x7fbd5002bea0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 -- 172.21.15.114:0/2119929159 shutdown_connections 2024-03-06T07:37:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 --2- 172.21.15.114:0/2119929159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd580cc9d0 0x7fbd5815f830 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 --2- 172.21.15.114:0/2119929159 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd581561f0 0x7fbd580ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 -- 172.21.15.114:0/2119929159 >> 172.21.15.114:0/2119929159 conn(0x7fbd5806afc0 msgr2=0x7fbd5806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 -- 172.21.15.114:0/2119929159 shutdown_connections 2024-03-06T07:37:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.701+0000 7fbd5d800640 1 -- 172.21.15.114:0/2119929159 wait complete. 2024-03-06T07:37:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.703+0000 7fbd5d800640 1 Processor -- start 2024-03-06T07:37:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.703+0000 7fbd5d800640 1 -- start start 2024-03-06T07:37:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd5d800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd580aebf0 0x7fbd581493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd5d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd580cc9d0 0x7fbd58149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd5d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd581561f0 0x7fbd58157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd5d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd58156750 con 0x7fbd580cc9d0 2024-03-06T07:37:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd5d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd581565d0 con 0x7fbd581561f0 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd5d800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5815fd70 con 0x7fbd580aebf0 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd580cc9d0 0x7fbd58149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd580cc9d0 0x7fbd58149c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44562/0 (socket says 172.21.15.114:44562) 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 -- 172.21.15.114:0/1895714802 learned_addr learned my addr 172.21.15.114:0/1895714802 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd5ce00640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd581561f0 0x7fbd58157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57e00640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd580aebf0 0x7fbd581493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 -- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd580aebf0 msgr2=0x7fbd581493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd580aebf0 0x7fbd581493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 -- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd581561f0 msgr2=0x7fbd58157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd581561f0 0x7fbd58157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57400640 1 -- 172.21.15.114:0/1895714802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5814a1c0 con 0x7fbd580cc9d0 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.704+0000 7fbd57e00640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd580aebf0 0x7fbd581493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.705+0000 7fbd57400640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd580cc9d0 0x7fbd58149c80 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c400 tx=0x7fbd4400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.705+0000 7fbd4fe00640 1 -- 172.21.15.114:0/1895714802 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44017070 con 0x7fbd580cc9d0 2024-03-06T07:37:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.705+0000 7fbd4fe00640 1 -- 172.21.15.114:0/1895714802 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd44005ce0 con 0x7fbd580cc9d0 2024-03-06T07:37:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.705+0000 7fbd4fe00640 1 -- 172.21.15.114:0/1895714802 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd440072f0 con 0x7fbd580cc9d0 2024-03-06T07:37:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.705+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd580afa30 con 0x7fbd580cc9d0 2024-03-06T07:37:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.705+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd580afff0 con 0x7fbd580cc9d0 2024-03-06T07:37:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.707+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5804f330 con 0x7fbd580cc9d0 2024-03-06T07:37:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.707+0000 7fbd4fe00640 1 -- 172.21.15.114:0/1895714802 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd4401a070 con 0x7fbd580cc9d0 2024-03-06T07:37:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.708+0000 7fbd4fe00640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbd3806b4d0 0x7fbd3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.708+0000 7fbd4fe00640 1 -- 172.21.15.114:0/1895714802 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd440896d0 con 0x7fbd580cc9d0 2024-03-06T07:37:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.708+0000 7fbd57e00640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbd3806b4d0 0x7fbd3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.709+0000 7fbd57e00640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbd3806b4d0 0x7fbd3806d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbd580b2070 tx=0x7fbd48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.712+0000 7fbd4fe00640 1 -- 172.21.15.114:0/1895714802 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd44058470 con 0x7fbd580cc9d0 2024-03-06T07:37:38.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.985+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd580b2b10 con 0x7fbd580cc9d0 2024-03-06T07:37:38.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.986+0000 7fbd4fe00640 1 -- 172.21.15.114:0/1895714802 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fbd44057d70 con 0x7fbd580cc9d0 2024-03-06T07:37:38.990 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:38.990 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:38.990 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.989+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbd3806b4d0 msgr2=0x7fbd3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.989+0000 7fbd5d800640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbd3806b4d0 0x7fbd3806d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbd580b2070 tx=0x7fbd48008040 comp rx=0 tx=0).stop 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.989+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd580cc9d0 msgr2=0x7fbd58149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.989+0000 7fbd5d800640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd580cc9d0 0x7fbd58149c80 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c400 tx=0x7fbd4400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 shutdown_connections 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd581561f0 0x7fbd58157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbd3806b4d0 0x7fbd3806d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd580cc9d0 0x7fbd58149c80 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 --2- 172.21.15.114:0/1895714802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbd580aebf0 0x7fbd581493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 >> 172.21.15.114:0/1895714802 conn(0x7fbd5806afc0 msgr2=0x7fbd58071ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 shutdown_connections 2024-03-06T07:37:38.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:38.990+0000 7fbd5d800640 1 -- 172.21.15.114:0/1895714802 wait complete. 2024-03-06T07:37:39.211 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:39.211 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:40.081 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:40.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:40.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.424+0000 7f4c5de00640 1 Processor -- start 2024-03-06T07:37:40.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.424+0000 7f4c5de00640 1 -- start start 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c5de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c580cf430 0x7f4c58070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c5de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4c5806d260 0x7f4c580b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c5de00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c580717c0 con 0x7f4c58071260 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c5de00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c58071aa0 con 0x7f4c5806d260 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c5de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58071d80 con 0x7f4c580cf430 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c57400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4c5806d260 0x7f4c580b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c56000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4c58071260 0x7f4c5806cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35392/0 (socket says 172.21.15.114:35392) 2024-03-06T07:37:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.426+0000 7f4c56000640 1 -- 172.21.15.114:0/1692480134 learned_addr learned my addr 172.21.15.114:0/1692480134 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:40.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.427+0000 7f4c56a00640 1 --2- 172.21.15.114:0/1692480134 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c580cf430 0x7f4c58070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.427+0000 7f4c57400640 1 -- 172.21.15.114:0/1692480134 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c580cf430 msgr2=0x7f4c58070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:40.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.427+0000 7f4c57400640 1 --2- 172.21.15.114:0/1692480134 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c580cf430 0x7f4c58070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.427+0000 7f4c57400640 1 -- 172.21.15.114:0/1692480134 >> v1:172.21.15.114:6789/0 conn(0x7f4c58071260 legacy=0x7f4c5806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.427+0000 7f4c57400640 1 -- 172.21.15.114:0/1692480134 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58072060 con 0x7f4c5806d260 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.428+0000 7f4c57400640 1 --2- 172.21.15.114:0/1692480134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c5806d260 0x7f4c580b2d10 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009d90 tx=0x7f4c5002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42eb03019aad799 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.428+0000 7f4c55600640 1 -- 172.21.15.114:0/1692480134 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50039070 con 0x7f4c5806d260 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.428+0000 7f4c55600640 1 -- 172.21.15.114:0/1692480134 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5002cc60 con 0x7f4c5806d260 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.428+0000 7f4c55600640 1 -- 172.21.15.114:0/1692480134 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50034660 con 0x7f4c5806d260 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 -- 172.21.15.114:0/1692480134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c5806d260 msgr2=0x7f4c580b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 --2- 172.21.15.114:0/1692480134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c5806d260 0x7f4c580b2d10 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009d90 tx=0x7f4c5002beb0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 -- 172.21.15.114:0/1692480134 shutdown_connections 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 --2- 172.21.15.114:0/1692480134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c5806d260 0x7f4c580b2d10 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 --2- 172.21.15.114:0/1692480134 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c580cf430 0x7f4c58070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 -- 172.21.15.114:0/1692480134 >> 172.21.15.114:0/1692480134 conn(0x7f4c580b1730 msgr2=0x7f4c580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 -- 172.21.15.114:0/1692480134 shutdown_connections 2024-03-06T07:37:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.429+0000 7f4c5de00640 1 -- 172.21.15.114:0/1692480134 wait complete. 2024-03-06T07:37:40.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.431+0000 7f4c5de00640 1 Processor -- start 2024-03-06T07:37:40.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.431+0000 7f4c5de00640 1 -- start start 2024-03-06T07:37:40.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c5de00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c5806d260 0x7f4c581ec990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c5de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c58071260 0x7f4c5815bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c5de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580cf430 0x7f4c5815c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c5de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58162cf0 con 0x7f4c580cf430 2024-03-06T07:37:40.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c5de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58162b70 con 0x7f4c58071260 2024-03-06T07:37:40.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c5de00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c580b3250 con 0x7f4c5806d260 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580cf430 0x7f4c5815c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580cf430 0x7f4c5815c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44574/0 (socket says 172.21.15.114:44574) 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c57400640 1 -- 172.21.15.114:0/3062593178 learned_addr learned my addr 172.21.15.114:0/3062593178 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c56000640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c58071260 0x7f4c5815bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c57400640 1 -- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c5806d260 msgr2=0x7f4c581ec990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c56a00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c5806d260 0x7f4c581ec990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.432+0000 7f4c57400640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c5806d260 0x7f4c581ec990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c57400640 1 -- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c58071260 msgr2=0x7f4c5815bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c57400640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c58071260 0x7f4c5815bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c57400640 1 -- 172.21.15.114:0/3062593178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58164bb0 con 0x7f4c580cf430 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c56000640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c58071260 0x7f4c5815bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c57400640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580cf430 0x7f4c5815c0d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009eb0 tx=0x7f4c50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c4f400640 1 -- 172.21.15.114:0/3062593178 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50039050 con 0x7f4c580cf430 2024-03-06T07:37:40.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c4f400640 1 -- 172.21.15.114:0/3062593178 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c50045070 con 0x7f4c580cf430 2024-03-06T07:37:40.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c4f400640 1 -- 172.21.15.114:0/3062593178 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5003e830 con 0x7f4c580cf430 2024-03-06T07:37:40.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c581648d0 con 0x7f4c580cf430 2024-03-06T07:37:40.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.433+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581ec190 con 0x7f4c580cf430 2024-03-06T07:37:40.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.434+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5804f330 con 0x7f4c580cf430 2024-03-06T07:37:40.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.435+0000 7f4c4f400640 1 -- 172.21.15.114:0/3062593178 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c5003e9d0 con 0x7f4c580cf430 2024-03-06T07:37:40.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.436+0000 7f4c4f400640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.436+0000 7f4c4f400640 1 -- 172.21.15.114:0/3062593178 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4c500ad070 con 0x7f4c580cf430 2024-03-06T07:37:40.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.436+0000 7f4c56a00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.437+0000 7f4c56a00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c3806b4d0 0x7f4c3806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800b7c0 tx=0x7f4c48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.441+0000 7f4c4f400640 1 -- 172.21.15.114:0/3062593178 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c5007bea0 con 0x7f4c580cf430 2024-03-06T07:37:40.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.679+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c580b4a40 con 0x7f4c580cf430 2024-03-06T07:37:40.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.680+0000 7f4c4f400640 1 -- 172.21.15.114:0/3062593178 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f4c5007b7a0 con 0x7f4c580cf430 2024-03-06T07:37:40.683 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:40.683 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:40.683 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:40.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.682+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c3806b4d0 msgr2=0x7f4c3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.682+0000 7f4c5de00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c3806b4d0 0x7f4c3806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800b7c0 tx=0x7f4c48008040 comp rx=0 tx=0).stop 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.682+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580cf430 msgr2=0x7f4c5815c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.682+0000 7f4c5de00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580cf430 0x7f4c5815c0d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009eb0 tx=0x7f4c50004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.682+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 shutdown_connections 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.683+0000 7f4c5de00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4c3806b4d0 0x7f4c3806d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.683+0000 7f4c5de00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c580cf430 0x7f4c5815c0d0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.683+0000 7f4c5de00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c58071260 0x7f4c5815bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.683+0000 7f4c5de00640 1 --2- 172.21.15.114:0/3062593178 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4c5806d260 0x7f4c581ec990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.683+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 >> 172.21.15.114:0/3062593178 conn(0x7f4c580b1730 msgr2=0x7f4c580b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.683+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 shutdown_connections 2024-03-06T07:37:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:40.683+0000 7f4c5de00640 1 -- 172.21.15.114:0/3062593178 wait complete. 2024-03-06T07:37:40.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:40.906 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:41.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:42.083 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.133+0000 7fa2f1200640 1 Processor -- start 2024-03-06T07:37:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.133+0000 7fa2f1200640 1 -- start start 2024-03-06T07:37:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.135+0000 7fa2f1200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa2ec06ebe0 0x7fa2ec070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.135+0000 7fa2f1200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ec071d30 con 0x7fa2ec0af1c0 2024-03-06T07:37:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.135+0000 7fa2f1200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ec15f4a0 con 0x7fa2ec15eff0 2024-03-06T07:37:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.135+0000 7fa2f1200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ec071260 con 0x7fa2ec06ebe0 2024-03-06T07:37:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.135+0000 7fa2e9e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa2ec0af1c0 0x7fa2ec06cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35408/0 (socket says 172.21.15.114:35408) 2024-03-06T07:37:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.135+0000 7fa2e9e00640 1 -- 172.21.15.114:0/2967311652 learned_addr learned my addr 172.21.15.114:0/2967311652 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761177806 0 0) 0x7fa2dc001110 con 0x7fa2ec0af1c0 2024-03-06T07:37:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2ea800640 1 --2- 172.21.15.114:0/2967311652 >> v2:172.21.15.177:3300/0 conn(0x7fa2ec06ebe0 0x7fa2ec070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d0003520 con 0x7fa2ec0af1c0 2024-03-06T07:37:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1115757514 0 0) 0x7fa2d0003520 con 0x7fa2ec0af1c0 2024-03-06T07:37:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 >> v2:172.21.15.177:3300/0 conn(0x7fa2ec06ebe0 msgr2=0x7fa2ec070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 --2- 172.21.15.114:0/2967311652 >> v2:172.21.15.177:3300/0 conn(0x7fa2ec06ebe0 0x7fa2ec070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 >> v1:172.21.15.177:6789/0 conn(0x7fa2ec15eff0 legacy=0x7fa2ec06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec071490 con 0x7fa2ec0af1c0 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fa2dc001840 con 0x7fa2ec0af1c0 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 >> v1:172.21.15.114:6789/0 conn(0x7fa2ec0af1c0 legacy=0x7fa2ec06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d0003a00 0x7fa2d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d000c090 con 0x7fa2d00051c0 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d000c240 con 0x7fa2d0004800 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.136+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d000c440 con 0x7fa2d0003a00 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2ea800640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2e9e00640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d0003a00 0x7fa2d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2e9400640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2ea800640 1 -- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d0003a00 msgr2=0x7fa2d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2ea800640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d0003a00 0x7fa2d0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2ea800640 1 -- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2d0004800 msgr2=0x7fa2d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2ea800640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2ea800640 1 -- 172.21.15.114:0/2967311652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0003520 con 0x7fa2d00051c0 2024-03-06T07:37:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.137+0000 7fa2e9e00640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d0003a00 0x7fa2d0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.138+0000 7fa2ea800640 1 -- 172.21.15.114:0/2967311652 reap_dead start 2024-03-06T07:37:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.138+0000 7fa2ea800640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009f90 tx=0x7fa2d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d5d8aa47b1a2ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.138+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4009c10 con 0x7fa2d00051c0 2024-03-06T07:37:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.138+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e402cc20 con 0x7fa2d00051c0 2024-03-06T07:37:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.138+0000 7fa2e8a00640 1 -- 172.21.15.114:0/2967311652 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4003990 con 0x7fa2d00051c0 2024-03-06T07:37:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.139+0000 7fa2f1200640 1 -- 172.21.15.114:0/2967311652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d00051c0 msgr2=0x7fa2d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.139+0000 7fa2f1200640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009f90 tx=0x7fa2d0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.139+0000 7fa2f1200640 1 -- 172.21.15.114:0/2967311652 shutdown_connections 2024-03-06T07:37:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.139+0000 7fa2f1200640 1 --2- 172.21.15.114:0/2967311652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.139+0000 7fa2f1200640 1 -- 172.21.15.114:0/2967311652 >> 172.21.15.114:0/2967311652 conn(0x7fa2ec0b1730 msgr2=0x7fa2ec0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.139+0000 7fa2f1200640 1 -- 172.21.15.114:0/2967311652 shutdown_connections 2024-03-06T07:37:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.139+0000 7fa2f1200640 1 -- 172.21.15.114:0/2967311652 wait complete. 2024-03-06T07:37:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.142+0000 7fa2f1200640 1 Processor -- start 2024-03-06T07:37:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.143+0000 7fa2f1200640 1 -- start start 2024-03-06T07:37:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.144+0000 7fa2f1200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d00051c0 0x7fa2ec1e71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.144+0000 7fa2f1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec1f3490 0x7fa2ec0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2f1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec0b34d0 0x7fa2ec0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2f1200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec0b30b0 con 0x7fa2ec1f3490 2024-03-06T07:37:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2f1200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec071a50 con 0x7fa2ec0b34d0 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2f1200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec070d20 con 0x7fa2d00051c0 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec1f3490 0x7fa2ec0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec1f3490 0x7fa2ec0cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44600/0 (socket says 172.21.15.114:44600) 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 -- 172.21.15.114:0/3282335304 learned_addr learned my addr 172.21.15.114:0/3282335304 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9e00640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d00051c0 0x7fa2ec1e71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2ea800640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec0b34d0 0x7fa2ec0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 -- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d00051c0 msgr2=0x7fa2ec1e71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d00051c0 0x7fa2ec1e71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 -- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec0b34d0 msgr2=0x7fa2ec0b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec0b34d0 0x7fa2ec0b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9400640 1 -- 172.21.15.114:0/3282335304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec0cdf40 con 0x7fa2ec1f3490 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2e9e00640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d00051c0 0x7fa2ec1e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.145+0000 7fa2ea800640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec0b34d0 0x7fa2ec0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.146+0000 7fa2e9400640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec1f3490 0x7fa2ec0cd680 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800e480 tx=0x7fa2d800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.146+0000 7fa2e2000640 1 -- 172.21.15.114:0/3282335304 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d80052e0 con 0x7fa2ec1f3490 2024-03-06T07:37:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.146+0000 7fa2e2000640 1 -- 172.21.15.114:0/3282335304 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d8018070 con 0x7fa2ec1f3490 2024-03-06T07:37:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.146+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ec0ce220 con 0x7fa2ec1f3490 2024-03-06T07:37:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.146+0000 7fa2e2000640 1 -- 172.21.15.114:0/3282335304 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8003830 con 0x7fa2ec1f3490 2024-03-06T07:37:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.146+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ec0ce7e0 con 0x7fa2ec1f3490 2024-03-06T07:37:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.148+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec04f330 con 0x7fa2ec1f3490 2024-03-06T07:37:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.149+0000 7fa2e2000640 1 -- 172.21.15.114:0/3282335304 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2d80039d0 con 0x7fa2ec1f3490 2024-03-06T07:37:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.151+0000 7fa2e2000640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2cc06b500 0x7fa2cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.151+0000 7fa2e2000640 1 -- 172.21.15.114:0/3282335304 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa2d808a960 con 0x7fa2ec1f3490 2024-03-06T07:37:42.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.151+0000 7fa2e9e00640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2cc06b500 0x7fa2cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.152+0000 7fa2e9e00640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2cc06b500 0x7fa2cc06d9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec1f6fd0 tx=0x7fa2dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.153+0000 7fa2e2000640 1 -- 172.21.15.114:0/3282335304 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2ec15f4a0 con 0x7fa2ec1f3490 2024-03-06T07:37:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.387+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2ec074440 con 0x7fa2ec1f3490 2024-03-06T07:37:42.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.388+0000 7fa2e2000640 1 -- 172.21.15.114:0/3282335304 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fa2d8059810 con 0x7fa2ec1f3490 2024-03-06T07:37:42.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:42.391 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:42.391 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:42.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2cc06b500 msgr2=0x7fa2cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2cc06b500 0x7fa2cc06d9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec1f6fd0 tx=0x7fa2dc009040 comp rx=0 tx=0).stop 2024-03-06T07:37:42.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec1f3490 msgr2=0x7fa2ec0cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec1f3490 0x7fa2ec0cd680 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800e480 tx=0x7fa2d800e940 comp rx=0 tx=0).stop 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 shutdown_connections 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec0b34d0 0x7fa2ec0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa2cc06b500 0x7fa2cc06d9c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2ec1f3490 0x7fa2ec0cd680 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 --2- 172.21.15.114:0/3282335304 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa2d00051c0 0x7fa2ec1e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.390+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 >> 172.21.15.114:0/3282335304 conn(0x7fa2ec0b1730 msgr2=0x7fa2ec1611d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.391+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 shutdown_connections 2024-03-06T07:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:42.391+0000 7fa2f1200640 1 -- 172.21.15.114:0/3282335304 wait complete. 2024-03-06T07:37:42.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:42.613 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:43.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:43.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.824+0000 7ff237c00640 1 Processor -- start 2024-03-06T07:37:43.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.824+0000 7ff237c00640 1 -- start start 2024-03-06T07:37:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.825+0000 7ff237c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff2300cf430 0x7ff23006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.825+0000 7ff237c00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff23006d920 0x7ff23015dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.825+0000 7ff237c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300b35b0 con 0x7ff230151660 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.825+0000 7ff237c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2300b3ee0 con 0x7ff2300cf430 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.825+0000 7ff237c00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2300b3890 con 0x7ff23006d920 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff22fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff2300cf430 0x7ff23006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff22fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff2300cf430 0x7ff23006d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44608/0 (socket says 172.21.15.114:44608) 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff22fe00640 1 -- 172.21.15.114:0/1572071534 learned_addr learned my addr 172.21.15.114:0/1572071534 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff235c00640 1 --2- 172.21.15.114:0/1572071534 >> v2:172.21.15.177:3300/0 conn(0x7ff23006d920 0x7ff23015dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff22fe00640 1 -- 172.21.15.114:0/1572071534 >> v2:172.21.15.177:3300/0 conn(0x7ff23006d920 msgr2=0x7ff23015dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff22fe00640 1 --2- 172.21.15.114:0/1572071534 >> v2:172.21.15.177:3300/0 conn(0x7ff23006d920 0x7ff23015dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff22fe00640 1 -- 172.21.15.114:0/1572071534 >> v1:172.21.15.177:6789/0 conn(0x7ff230151660 legacy=0x7ff230071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff22fe00640 1 -- 172.21.15.114:0/1572071534 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300b3b70 con 0x7ff2300cf430 2024-03-06T07:37:43.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.826+0000 7ff235c00640 1 --2- 172.21.15.114:0/1572071534 >> v2:172.21.15.177:3300/0 conn(0x7ff23006d920 0x7ff23015dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.827+0000 7ff22fe00640 1 --2- 172.21.15.114:0/1572071534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2300cf430 0x7ff23006d3e0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff230154fc0 tx=0x7ff21c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ab9a291d821f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff22f400640 1 -- 172.21.15.114:0/1572071534 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c038070 con 0x7ff2300cf430 2024-03-06T07:37:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff22f400640 1 -- 172.21.15.114:0/1572071534 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c02f050 con 0x7ff2300cf430 2024-03-06T07:37:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff22f400640 1 -- 172.21.15.114:0/1572071534 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c031970 con 0x7ff2300cf430 2024-03-06T07:37:43.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff237c00640 1 -- 172.21.15.114:0/1572071534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2300cf430 msgr2=0x7ff23006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff237c00640 1 --2- 172.21.15.114:0/1572071534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2300cf430 0x7ff23006d3e0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff230154fc0 tx=0x7ff21c02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff237c00640 1 -- 172.21.15.114:0/1572071534 shutdown_connections 2024-03-06T07:37:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff237c00640 1 --2- 172.21.15.114:0/1572071534 >> v2:172.21.15.177:3300/0 conn(0x7ff23006d920 0x7ff23015dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff237c00640 1 --2- 172.21.15.114:0/1572071534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2300cf430 0x7ff23006d3e0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.828+0000 7ff237c00640 1 -- 172.21.15.114:0/1572071534 >> 172.21.15.114:0/1572071534 conn(0x7ff2300b1730 msgr2=0x7ff2300b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.829+0000 7ff237c00640 1 -- 172.21.15.114:0/1572071534 shutdown_connections 2024-03-06T07:37:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.829+0000 7ff237c00640 1 -- 172.21.15.114:0/1572071534 wait complete. 2024-03-06T07:37:43.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.831+0000 7ff237c00640 1 Processor -- start 2024-03-06T07:37:43.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.831+0000 7ff237c00640 1 -- start start 2024-03-06T07:37:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff237c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23006d920 0x7ff23014bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff237c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2300cf430 0x7ff2301541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff237c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff230151660 0x7ff230145230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff237c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23015e6b0 con 0x7ff23006d920 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff237c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23015e530 con 0x7ff230151660 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff237c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23015e830 con 0x7ff2300cf430 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff235200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23006d920 0x7ff23014bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff235200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23006d920 0x7ff23014bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44612/0 (socket says 172.21.15.114:44612) 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.833+0000 7ff235200640 1 -- 172.21.15.114:0/3018625232 learned_addr learned my addr 172.21.15.114:0/3018625232 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff235200640 1 -- 172.21.15.114:0/3018625232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2300cf430 msgr2=0x7ff2301541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff235200640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2300cf430 0x7ff2301541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff235200640 1 -- 172.21.15.114:0/3018625232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff230151660 msgr2=0x7ff230145230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff235200640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff230151660 0x7ff230145230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff235200640 1 -- 172.21.15.114:0/3018625232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2301f4b10 con 0x7ff23006d920 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff235200640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23006d920 0x7ff23014bef0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c460 tx=0x7ff22400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff22cc00640 1 -- 172.21.15.114:0/3018625232 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224016020 con 0x7ff23006d920 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff22cc00640 1 -- 172.21.15.114:0/3018625232 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff224012870 con 0x7ff23006d920 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff22cc00640 1 -- 172.21.15.114:0/3018625232 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224012c60 con 0x7ff23006d920 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2301f4df0 con 0x7ff23006d920 2024-03-06T07:37:43.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.834+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff230152830 con 0x7ff23006d920 2024-03-06T07:37:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.835+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23004f330 con 0x7ff23006d920 2024-03-06T07:37:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.836+0000 7ff22cc00640 1 -- 172.21.15.114:0/3018625232 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff224019020 con 0x7ff23006d920 2024-03-06T07:37:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.837+0000 7ff22cc00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff21006b4d0 0x7ff21006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.837+0000 7ff22cc00640 1 -- 172.21.15.114:0/3018625232 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff224089fe0 con 0x7ff23006d920 2024-03-06T07:37:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.837+0000 7ff22fe00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff21006b4d0 0x7ff21006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.838+0000 7ff22fe00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff21006b4d0 0x7ff21006d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff21c035000 tx=0x7ff21c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:43.841+0000 7ff22cc00640 1 -- 172.21.15.114:0/3018625232 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff224020030 con 0x7ff23006d920 2024-03-06T07:37:44.084 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:44.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.100+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff230074be0 con 0x7ff23006d920 2024-03-06T07:37:44.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.101+0000 7ff22cc00640 1 -- 172.21.15.114:0/3018625232 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7ff224058da0 con 0x7ff23006d920 2024-03-06T07:37:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:44.104 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:44.104 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:44.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff21006b4d0 msgr2=0x7ff21006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff21006b4d0 0x7ff21006d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff21c035000 tx=0x7ff21c030000 comp rx=0 tx=0).stop 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23006d920 msgr2=0x7ff23014bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23006d920 0x7ff23014bef0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c460 tx=0x7ff22400c920 comp rx=0 tx=0).stop 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 shutdown_connections 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff230151660 0x7ff230145230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff2300cf430 0x7ff2301541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff21006b4d0 0x7ff21006d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 --2- 172.21.15.114:0/3018625232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23006d920 0x7ff23014bef0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.103+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 >> 172.21.15.114:0/3018625232 conn(0x7ff2300b1730 msgr2=0x7ff23006e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.104+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 shutdown_connections 2024-03-06T07:37:44.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:44.104+0000 7ff237c00640 1 -- 172.21.15.114:0/3018625232 wait complete. 2024-03-06T07:37:44.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:44.325 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:45.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.546+0000 7f7502a00640 1 Processor -- start 2024-03-06T07:37:45.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.546+0000 7f7502a00640 1 -- start start 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.548+0000 7f7502a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74fc15dfd0 0x7f74fc162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7502a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc0affe0 0x7f74fc06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7502a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74fc0cef70 con 0x7f74fc06b500 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7502a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74fc162270 con 0x7f74fc15dfd0 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7502a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc15e3b0 con 0x7f74fc0affe0 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7501600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74fc15dfd0 0x7f74fc162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7501600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74fc15dfd0 0x7f74fc162700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44618/0 (socket says 172.21.15.114:44618) 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7501600640 1 -- 172.21.15.114:0/759720373 learned_addr learned my addr 172.21.15.114:0/759720373 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:45.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7500c00640 1 --2- 172.21.15.114:0/759720373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc0affe0 0x7f74fc06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7501600640 1 -- 172.21.15.114:0/759720373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc0affe0 msgr2=0x7f74fc06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7501600640 1 --2- 172.21.15.114:0/759720373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc0affe0 0x7f74fc06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7501600640 1 -- 172.21.15.114:0/759720373 >> v1:172.21.15.177:6789/0 conn(0x7f74fc06b500 legacy=0x7f74fc0d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.549+0000 7f7501600640 1 -- 172.21.15.114:0/759720373 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc161c60 con 0x7f74fc15dfd0 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.550+0000 7f7501600640 1 --2- 172.21.15.114:0/759720373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc15dfd0 0x7f74fc162700 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009870 tx=0x7f74f002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c122391c573112f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f74fbe00640 1 -- 172.21.15.114:0/759720373 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0038070 con 0x7f74fc15dfd0 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f74fbe00640 1 -- 172.21.15.114:0/759720373 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f0031740 con 0x7f74fc15dfd0 2024-03-06T07:37:45.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f74fbe00640 1 -- 172.21.15.114:0/759720373 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f002f050 con 0x7f74fc15dfd0 2024-03-06T07:37:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f7502a00640 1 -- 172.21.15.114:0/759720373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc15dfd0 msgr2=0x7f74fc162700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f7502a00640 1 --2- 172.21.15.114:0/759720373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc15dfd0 0x7f74fc162700 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009870 tx=0x7f74f002b7e0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f7502a00640 1 -- 172.21.15.114:0/759720373 shutdown_connections 2024-03-06T07:37:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f7502a00640 1 --2- 172.21.15.114:0/759720373 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc0affe0 0x7f74fc06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.551+0000 7f7502a00640 1 --2- 172.21.15.114:0/759720373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc15dfd0 0x7f74fc162700 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.552+0000 7f7502a00640 1 -- 172.21.15.114:0/759720373 >> 172.21.15.114:0/759720373 conn(0x7f74fc06c0d0 msgr2=0x7f74fc06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.552+0000 7f7502a00640 1 -- 172.21.15.114:0/759720373 shutdown_connections 2024-03-06T07:37:45.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.552+0000 7f7502a00640 1 -- 172.21.15.114:0/759720373 wait complete. 2024-03-06T07:37:45.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.554+0000 7f7502a00640 1 Processor -- start 2024-03-06T07:37:45.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.555+0000 7f7502a00640 1 -- start start 2024-03-06T07:37:45.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7502a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74fc06b500 0x7f74fc14ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7502a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc0affe0 0x7f74fc14f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7502a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc15dfd0 0x7f74fc0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7502a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc0ce8d0 con 0x7f74fc0affe0 2024-03-06T07:37:45.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7502a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc161f00 con 0x7f74fc06b500 2024-03-06T07:37:45.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7502a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc0cea50 con 0x7f74fc15dfd0 2024-03-06T07:37:45.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7500c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc0affe0 0x7f74fc14f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7500c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc0affe0 0x7f74fc14f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44632/0 (socket says 172.21.15.114:44632) 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.556+0000 7f7500c00640 1 -- 172.21.15.114:0/850245974 learned_addr learned my addr 172.21.15.114:0/850245974 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7501600640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74fc06b500 0x7f74fc14ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7502000640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc15dfd0 0x7f74fc0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7500c00640 1 -- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc15dfd0 msgr2=0x7f74fc0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7500c00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc15dfd0 0x7f74fc0cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7500c00640 1 -- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74fc06b500 msgr2=0x7f74fc14ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7500c00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74fc06b500 0x7f74fc14ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7500c00640 1 -- 172.21.15.114:0/850245974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc149410 con 0x7f74fc0affe0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7501600640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74fc06b500 0x7f74fc14ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f7500c00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc0affe0 0x7f74fc14f130 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f74e800d800 tx=0x7f74e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f74f9600640 1 -- 172.21.15.114:0/850245974 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e800b6c0 con 0x7f74fc0affe0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f74f9600640 1 -- 172.21.15.114:0/850245974 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e8019070 con 0x7f74fc0affe0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.557+0000 7f74f9600640 1 -- 172.21.15.114:0/850245974 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e800ba80 con 0x7f74fc0affe0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.558+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74fc0b4b80 con 0x7f74fc0affe0 2024-03-06T07:37:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.558+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc149930 con 0x7f74fc0affe0 2024-03-06T07:37:45.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.559+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc149740 con 0x7f74fc0affe0 2024-03-06T07:37:45.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.560+0000 7f74f9600640 1 -- 172.21.15.114:0/850245974 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74e800bc20 con 0x7f74fc0affe0 2024-03-06T07:37:45.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.561+0000 7f74f9600640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74dc06b4d0 0x7f74dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.561+0000 7f74f9600640 1 -- 172.21.15.114:0/850245974 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f74e808a710 con 0x7f74fc0affe0 2024-03-06T07:37:45.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.561+0000 7f7501600640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74dc06b4d0 0x7f74dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.562+0000 7f7501600640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74dc06b4d0 0x7f74dc06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f74f0035000 tx=0x7f74f0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:45.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.567+0000 7f74f9600640 1 -- 172.21.15.114:0/850245974 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74e8059530 con 0x7f74fc0affe0 2024-03-06T07:37:45.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.807+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74fc0b5e60 con 0x7f74fc0affe0 2024-03-06T07:37:45.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.808+0000 7f74f9600640 1 -- 172.21.15.114:0/850245974 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f74e8058e30 con 0x7f74fc0affe0 2024-03-06T07:37:45.811 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:45.811 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:45.812 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:45.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.811+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74dc06b4d0 msgr2=0x7f74dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.811+0000 7f7502a00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74dc06b4d0 0x7f74dc06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f74f0035000 tx=0x7f74f0008030 comp rx=0 tx=0).stop 2024-03-06T07:37:45.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.811+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc0affe0 msgr2=0x7f74fc14f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.811+0000 7f7502a00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc0affe0 0x7f74fc14f130 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f74e800d800 tx=0x7f74e800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 shutdown_connections 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f74fc15dfd0 0x7f74fc0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f74dc06b4d0 0x7f74dc06d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74fc0affe0 0x7f74fc14f130 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 --2- 172.21.15.114:0/850245974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74fc06b500 0x7f74fc14ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 >> 172.21.15.114:0/850245974 conn(0x7f74fc06c0d0 msgr2=0x7f74fc0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 shutdown_connections 2024-03-06T07:37:45.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:45.812+0000 7f7502a00640 1 -- 172.21.15.114:0/850245974 wait complete. 2024-03-06T07:37:46.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:46.035 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:46.086 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:47.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.241+0000 7fb13da00640 1 Processor -- start 2024-03-06T07:37:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.242+0000 7fb13da00640 1 -- start start 2024-03-06T07:37:47.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.244+0000 7fb13da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 0x7fb13815fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.244+0000 7fb13da00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380b3040 con 0x7fb1380cf9a0 2024-03-06T07:37:47.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.244+0000 7fb13da00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380cd3e0 con 0x7fb1380b4390 2024-03-06T07:37:47.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.244+0000 7fb13da00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13815fb50 con 0x7fb13815f770 2024-03-06T07:37:47.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb137400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb1380cf9a0 0x7fb1380cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35416/0 (socket says 172.21.15.114:35416) 2024-03-06T07:37:47.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb137400640 1 -- 172.21.15.114:0/404780218 learned_addr learned my addr 172.21.15.114:0/404780218 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:47.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb136a00640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 0x7fb13815fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639458734 0 0) 0x7fb1380b3040 con 0x7fb1380cf9a0 2024-03-06T07:37:47.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb11c003520 con 0x7fb1380cf9a0 2024-03-06T07:37:47.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081861523 0 0) 0x7fb11c003520 con 0x7fb1380cf9a0 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 msgr2=0x7fb13815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb135600640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 0x7fb13815fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.245+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 >> v1:172.21.15.177:6789/0 conn(0x7fb1380b4390 legacy=0x7fb138160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380cd6c0 con 0x7fb1380cf9a0 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb130001670 con 0x7fb1380cf9a0 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 >> v1:172.21.15.114:6789/0 conn(0x7fb1380cf9a0 legacy=0x7fb1380cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb11c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 0x7fb11c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb11c004d40 0x7fb11c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c00bac0 con 0x7fb11c0039e0 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c00bc70 con 0x7fb11c004d40 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c00be70 con 0x7fb13815f770 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb136a00640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb11c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb136000640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 0x7fb11c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb136a00640 1 -- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 msgr2=0x7fb11c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb136a00640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb13815f770 0x7fb11c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb137400640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb11c004d40 0x7fb11c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.246+0000 7fb136a00640 1 -- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb11c004d40 msgr2=0x7fb11c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.247+0000 7fb136a00640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb11c004d40 0x7fb11c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.247+0000 7fb136a00640 1 -- 172.21.15.114:0/404780218 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c003520 con 0x7fb11c0039e0 2024-03-06T07:37:47.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.247+0000 7fb137400640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb11c004d40 0x7fb11c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:47.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.248+0000 7fb136a00640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb11c003e00 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fb128009fc0 tx=0x7fb11c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e7cae0bfd59149 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.248+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128039070 con 0x7fb11c0039e0 2024-03-06T07:37:47.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.248+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12802cb40 con 0x7fb11c0039e0 2024-03-06T07:37:47.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.248+0000 7fb135600640 1 -- 172.21.15.114:0/404780218 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128009c10 con 0x7fb11c0039e0 2024-03-06T07:37:47.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.249+0000 7fb13da00640 1 -- 172.21.15.114:0/404780218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 msgr2=0x7fb11c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.249+0000 7fb13da00640 1 --2- 172.21.15.114:0/404780218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb11c003e00 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fb128009fc0 tx=0x7fb11c005830 comp rx=0 tx=0).stop 2024-03-06T07:37:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.249+0000 7fb137400640 1 -- 172.21.15.114:0/404780218 reap_dead start 2024-03-06T07:37:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.249+0000 7fb13da00640 1 -- 172.21.15.114:0/404780218 shutdown_connections 2024-03-06T07:37:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.249+0000 7fb13da00640 1 -- 172.21.15.114:0/404780218 >> 172.21.15.114:0/404780218 conn(0x7fb1380b3870 msgr2=0x7fb1380b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.249+0000 7fb13da00640 1 -- 172.21.15.114:0/404780218 shutdown_connections 2024-03-06T07:37:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.249+0000 7fb13da00640 1 -- 172.21.15.114:0/404780218 wait complete. 2024-03-06T07:37:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.251+0000 7fb13da00640 1 Processor -- start 2024-03-06T07:37:47.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.251+0000 7fb13da00640 1 -- start start 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb13da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb1381f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb13da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138150710 0x7fb1380b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb13da00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1380b53a0 0x7fb1380b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb13da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380b34a0 con 0x7fb11c0039e0 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb13da00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380b3320 con 0x7fb138150710 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb13da00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1381609a0 con 0x7fb1380b53a0 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb1381f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb1381f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44646/0 (socket says 172.21.15.114:44646) 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 -- 172.21.15.114:0/1135707788 learned_addr learned my addr 172.21.15.114:0/1135707788 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:47.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 -- 172.21.15.114:0/1135707788 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1380b53a0 msgr2=0x7fb1380b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1380b53a0 0x7fb1380b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 -- 172.21.15.114:0/1135707788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138150710 msgr2=0x7fb1380b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138150710 0x7fb1380b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.252+0000 7fb136a00640 1 -- 172.21.15.114:0/1135707788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1381f4450 con 0x7fb11c0039e0 2024-03-06T07:37:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.253+0000 7fb136a00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb1381f3e40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb128037040 tx=0x7fb128009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.253+0000 7fb12ea00640 1 -- 172.21.15.114:0/1135707788 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128039050 con 0x7fb11c0039e0 2024-03-06T07:37:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.253+0000 7fb12ea00640 1 -- 172.21.15.114:0/1135707788 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12803a040 con 0x7fb11c0039e0 2024-03-06T07:37:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.253+0000 7fb12ea00640 1 -- 172.21.15.114:0/1135707788 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128009c10 con 0x7fb11c0039e0 2024-03-06T07:37:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.253+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1381f4730 con 0x7fb11c0039e0 2024-03-06T07:37:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.253+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1381f4cf0 con 0x7fb11c0039e0 2024-03-06T07:37:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.254+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13804f330 con 0x7fb11c0039e0 2024-03-06T07:37:47.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.255+0000 7fb12ea00640 1 -- 172.21.15.114:0/1135707788 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb128047030 con 0x7fb11c0039e0 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.256+0000 7fb12ea00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.256+0000 7fb12ea00640 1 -- 172.21.15.114:0/1135707788 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1280ac590 con 0x7fb11c0039e0 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.256+0000 7fb136000640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.257+0000 7fb136000640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb124007fd0 tx=0x7fb124009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.261+0000 7fb12ea00640 1 -- 172.21.15.114:0/1135707788 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb12804e060 con 0x7fb11c0039e0 2024-03-06T07:37:47.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:47.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:47.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.499+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1380b7cc0 con 0x7fb11c0039e0 2024-03-06T07:37:47.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.500+0000 7fb12ea00640 1 -- 172.21.15.114:0/1135707788 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fb12807b460 con 0x7fb11c0039e0 2024-03-06T07:37:47.503 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:47.503 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:47.503 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:47.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 msgr2=0x7fb11806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb124007fd0 tx=0x7fb124009040 comp rx=0 tx=0).stop 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 msgr2=0x7fb1381f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb1381f3e40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb128037040 tx=0x7fb128009750 comp rx=0 tx=0).stop 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 shutdown_connections 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb1380b53a0 0x7fb1380b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb138150710 0x7fb1380b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb11806b4d0 0x7fb11806d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 --2- 172.21.15.114:0/1135707788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb11c0039e0 0x7fb1381f3e40 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.502+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 >> 172.21.15.114:0/1135707788 conn(0x7fb1380b3870 msgr2=0x7fb138159e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.503+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 shutdown_connections 2024-03-06T07:37:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:47.503+0000 7fb13da00640 1 -- 172.21.15.114:0/1135707788 wait complete. 2024-03-06T07:37:47.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:47.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:48.088 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:48.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:48.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.943+0000 7f6355600640 1 Processor -- start 2024-03-06T07:37:48.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.943+0000 7f6355600640 1 -- start start 2024-03-06T07:37:48.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.944+0000 7f6355600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f635015fe60 0x7f6350071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.945+0000 7f6355600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635006cef0 con 0x7f635015f9d0 2024-03-06T07:37:48.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.945+0000 7f6355600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635006d1d0 con 0x7f6350144c00 2024-03-06T07:37:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.945+0000 7f6355600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635006d4b0 con 0x7f635015fe60 2024-03-06T07:37:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.945+0000 7f6346000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f635015f9d0 0x7f6350070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40552/0 (socket says 172.21.15.114:40552) 2024-03-06T07:37:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.945+0000 7f6346000640 1 -- 172.21.15.114:0/4090554567 learned_addr learned my addr 172.21.15.114:0/4090554567 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.945+0000 7f634f400640 1 --2- 172.21.15.114:0/4090554567 >> v2:172.21.15.177:3300/0 conn(0x7f635015fe60 0x7f6350071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.945+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589906212 0 0) 0x7f635006cef0 con 0x7f635015f9d0 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6334003520 con 0x7f635015f9d0 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244609811 0 0) 0x7f6334003520 con 0x7f635015f9d0 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 >> v2:172.21.15.177:3300/0 conn(0x7f635015fe60 msgr2=0x7f6350071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 --2- 172.21.15.114:0/4090554567 >> v2:172.21.15.177:3300/0 conn(0x7f635015fe60 0x7f6350071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 >> v1:172.21.15.177:6789/0 conn(0x7f6350144c00 legacy=0x7f635006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635006d790 con 0x7f635015f9d0 2024-03-06T07:37:48.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6338001670 con 0x7f635015f9d0 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 >> v1:172.21.15.114:6789/0 conn(0x7f635015f9d0 legacy=0x7f6350070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334003a00 0x7f6334003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6334004800 0x7f6334004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635006d4b0 con 0x7f6334003a00 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633400c0c0 con 0x7f63340051c0 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633400c2c0 con 0x7f6334004800 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.946+0000 7f634ea00640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334003a00 0x7f6334003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f6346000640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6334004800 0x7f6334004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f634f400640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f634ea00640 1 -- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6334004800 msgr2=0x7f6334004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f634ea00640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6334004800 0x7f6334004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f634ea00640 1 -- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63340051c0 msgr2=0x7f633400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f634ea00640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f634ea00640 1 -- 172.21.15.114:0/4090554567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334003520 con 0x7f6334003a00 2024-03-06T07:37:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.947+0000 7f634f400640 1 -- 172.21.15.114:0/4090554567 reap_dead start 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.949+0000 7f634ea00640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334003a00 0x7f6334003e20 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f6340001750 tx=0x7f6334005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daef7d985682c8a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.949+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340001370 con 0x7f6334003a00 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.949+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63400052d0 con 0x7f6334003a00 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.949+0000 7f634e000640 1 -- 172.21.15.114:0/4090554567 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6340003830 con 0x7f6334003a00 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.949+0000 7f6355600640 1 -- 172.21.15.114:0/4090554567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334003a00 msgr2=0x7f6334003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.949+0000 7f6355600640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334003a00 0x7f6334003e20 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f6340001750 tx=0x7f6334005830 comp rx=0 tx=0).stop 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.950+0000 7f6355600640 1 -- 172.21.15.114:0/4090554567 shutdown_connections 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.950+0000 7f6355600640 1 --2- 172.21.15.114:0/4090554567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334003a00 0x7f6334003e20 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.950+0000 7f6355600640 1 -- 172.21.15.114:0/4090554567 >> 172.21.15.114:0/4090554567 conn(0x7f63500b1730 msgr2=0x7f63500b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.950+0000 7f6355600640 1 -- 172.21.15.114:0/4090554567 shutdown_connections 2024-03-06T07:37:48.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.950+0000 7f6355600640 1 -- 172.21.15.114:0/4090554567 wait complete. 2024-03-06T07:37:48.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.951+0000 7f6355600640 1 Processor -- start 2024-03-06T07:37:48.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.951+0000 7f6355600640 1 -- start start 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.952+0000 7f6355600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6334003a00 0x7f635013f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.952+0000 7f6355600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635014d4b0 0x7f6350142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.952+0000 7f6355600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635015e9d0 0x7f635015ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.952+0000 7f6355600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635006cb50 con 0x7f635015e9d0 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.952+0000 7f6355600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6350160730 con 0x7f6334003a00 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.952+0000 7f6355600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635006ccd0 con 0x7f635014d4b0 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635015e9d0 0x7f635015ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635015e9d0 0x7f635015ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54026/0 (socket says 172.21.15.114:54026) 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 -- 172.21.15.114:0/3241749091 learned_addr learned my addr 172.21.15.114:0/3241749091 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:48.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f6346000640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635014d4b0 0x7f6350142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634ea00640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6334003a00 0x7f635013f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 -- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635014d4b0 msgr2=0x7f6350142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635014d4b0 0x7f6350142990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 -- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6334003a00 msgr2=0x7f635013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6334003a00 0x7f635013f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 -- 172.21.15.114:0/3241749091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635013ff80 con 0x7f635015e9d0 2024-03-06T07:37:48.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.953+0000 7f634f400640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635015e9d0 0x7f635015ee50 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f63500d0ac0 tx=0x7f634800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.954+0000 7f6347400640 1 -- 172.21.15.114:0/3241749091 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348016420 con 0x7f635015e9d0 2024-03-06T07:37:48.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.954+0000 7f6347400640 1 -- 172.21.15.114:0/3241749091 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634801b070 con 0x7f635015e9d0 2024-03-06T07:37:48.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.954+0000 7f6347400640 1 -- 172.21.15.114:0/3241749091 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348016800 con 0x7f635015e9d0 2024-03-06T07:37:48.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.954+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635015dc10 con 0x7f635015e9d0 2024-03-06T07:37:48.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.954+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635015e1d0 con 0x7f635015e9d0 2024-03-06T07:37:48.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.955+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635006cef0 con 0x7f635015e9d0 2024-03-06T07:37:48.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.956+0000 7f6347400640 1 -- 172.21.15.114:0/3241749091 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63480169a0 con 0x7f635015e9d0 2024-03-06T07:37:48.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.957+0000 7f6347400640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f632c06b4d0 0x7f632c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.957+0000 7f6347400640 1 -- 172.21.15.114:0/3241749091 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f634808b200 con 0x7f635015e9d0 2024-03-06T07:37:48.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.957+0000 7f634ea00640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f632c06b4d0 0x7f632c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.958+0000 7f634ea00640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f632c06b4d0 0x7f632c06d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6340033000 tx=0x7f6340009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:48.966+0000 7f6347400640 1 -- 172.21.15.114:0/3241749091 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f635006cef0 con 0x7f635015e9d0 2024-03-06T07:37:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.199+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f635004f330 con 0x7f635015e9d0 2024-03-06T07:37:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.200+0000 7f6347400640 1 -- 172.21.15.114:0/3241749091 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f6348025020 con 0x7f635015e9d0 2024-03-06T07:37:49.203 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:49.203 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:49.203 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.203+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f632c06b4d0 msgr2=0x7f632c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.203+0000 7f6355600640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f632c06b4d0 0x7f632c06d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6340033000 tx=0x7f6340009030 comp rx=0 tx=0).stop 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.203+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635015e9d0 msgr2=0x7f635015ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.203+0000 7f6355600640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635015e9d0 0x7f635015ee50 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f63500d0ac0 tx=0x7f634800f520 comp rx=0 tx=0).stop 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 shutdown_connections 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f632c06b4d0 0x7f632c06d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f635015e9d0 0x7f635015ee50 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f635014d4b0 0x7f6350142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 --2- 172.21.15.114:0/3241749091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6334003a00 0x7f635013f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 >> 172.21.15.114:0/3241749091 conn(0x7f63500b1730 msgr2=0x7f63500b0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 shutdown_connections 2024-03-06T07:37:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:49.204+0000 7f6355600640 1 -- 172.21.15.114:0/3241749091 wait complete. 2024-03-06T07:37:49.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:49.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:50.090 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:50.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:50.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.639+0000 7f999aa00640 1 Processor -- start 2024-03-06T07:37:50.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.639+0000 7f999aa00640 1 -- start start 2024-03-06T07:37:50.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.641+0000 7f999aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9994066d00 0x7f99940b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.641+0000 7f999aa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940afc20 con 0x7f999415c120 2024-03-06T07:37:50.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.641+0000 7f999aa00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940b5080 con 0x7f999406b6d0 2024-03-06T07:37:50.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.641+0000 7f999aa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99940d1130 con 0x7f9994066d00 2024-03-06T07:37:50.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.641+0000 7f9993e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f999415c120 0x7f999406afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40566/0 (socket says 172.21.15.114:40566) 2024-03-06T07:37:50.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.641+0000 7f9993e00640 1 -- 172.21.15.114:0/1871507861 learned_addr learned my addr 172.21.15.114:0/1871507861 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:50.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.641+0000 7f9998a00640 1 --2- 172.21.15.114:0/1871507861 >> v2:172.21.15.114:3300/0 conn(0x7f9994066d00 0x7f99940b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.642+0000 7f9993400640 1 -- 172.21.15.114:0/1871507861 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614364547 0 0) 0x7f99940afc20 con 0x7f999415c120 2024-03-06T07:37:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.642+0000 7f9993400640 1 -- 172.21.15.114:0/1871507861 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9978003520 con 0x7f999415c120 2024-03-06T07:37:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.642+0000 7f9998a00640 1 -- 172.21.15.114:0/1871507861 >> v1:172.21.15.177:6789/0 conn(0x7f999406b6d0 legacy=0x7f99940ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.642+0000 7f9998a00640 1 -- 172.21.15.114:0/1871507861 >> v1:172.21.15.114:6789/0 conn(0x7f999415c120 legacy=0x7f999406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.642+0000 7f9998a00640 1 -- 172.21.15.114:0/1871507861 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99940b5360 con 0x7f9994066d00 2024-03-06T07:37:50.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.643+0000 7f9998a00640 1 --2- 172.21.15.114:0/1871507861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9994066d00 0x7f99940b5e40 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f998c009d90 tx=0x7f998c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3624174d1e5bde17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:50.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.644+0000 7f9993400640 1 -- 172.21.15.114:0/1871507861 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c039070 con 0x7f9994066d00 2024-03-06T07:37:50.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.644+0000 7f9993400640 1 -- 172.21.15.114:0/1871507861 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c02cd00 con 0x7f9994066d00 2024-03-06T07:37:50.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.644+0000 7f9993400640 1 -- 172.21.15.114:0/1871507861 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c034640 con 0x7f9994066d00 2024-03-06T07:37:50.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.644+0000 7f999aa00640 1 -- 172.21.15.114:0/1871507861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9994066d00 msgr2=0x7f99940b5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.644+0000 7f999aa00640 1 --2- 172.21.15.114:0/1871507861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9994066d00 0x7f99940b5e40 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f998c009d90 tx=0x7f998c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:37:50.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.645+0000 7f999aa00640 1 -- 172.21.15.114:0/1871507861 shutdown_connections 2024-03-06T07:37:50.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.645+0000 7f999aa00640 1 --2- 172.21.15.114:0/1871507861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9994066d00 0x7f99940b5e40 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.645+0000 7f999aa00640 1 -- 172.21.15.114:0/1871507861 >> 172.21.15.114:0/1871507861 conn(0x7f99940b1e80 msgr2=0x7f99940b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:50.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.645+0000 7f999aa00640 1 -- 172.21.15.114:0/1871507861 shutdown_connections 2024-03-06T07:37:50.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.645+0000 7f999aa00640 1 -- 172.21.15.114:0/1871507861 wait complete. 2024-03-06T07:37:50.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.647+0000 7f999aa00640 1 Processor -- start 2024-03-06T07:37:50.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.647+0000 7f999aa00640 1 -- start start 2024-03-06T07:37:50.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f999aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9994066d00 0x7f999415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f999aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f999406b6d0 0x7f99941f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f999aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999415c120 0x7f99941f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f999aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999415c500 con 0x7f999406b6d0 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f999aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99940b59b0 con 0x7f999415c120 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f999aa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99940d0a70 con 0x7f9994066d00 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f998be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f999406b6d0 0x7f99941f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f998be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f999406b6d0 0x7f99941f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54048/0 (socket says 172.21.15.114:54048) 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f998be00640 1 -- 172.21.15.114:0/3796344208 learned_addr learned my addr 172.21.15.114:0/3796344208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.648+0000 7f9993e00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9994066d00 0x7f999415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f9998a00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999415c120 0x7f99941f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f998be00640 1 -- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9994066d00 msgr2=0x7f999415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f998be00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9994066d00 0x7f999415eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f998be00640 1 -- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999415c120 msgr2=0x7f99941f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f998be00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999415c120 0x7f99941f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f998be00640 1 -- 172.21.15.114:0/3796344208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99941f3f00 con 0x7f999406b6d0 2024-03-06T07:37:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f9998a00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999415c120 0x7f99941f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f998be00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f999406b6d0 0x7f99941f3480 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c400 tx=0x7f997c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f9990c00640 1 -- 172.21.15.114:0/3796344208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c0052d0 con 0x7f999406b6d0 2024-03-06T07:37:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f9990c00640 1 -- 172.21.15.114:0/3796344208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c005490 con 0x7f999406b6d0 2024-03-06T07:37:50.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999406ef80 con 0x7f999406b6d0 2024-03-06T07:37:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.649+0000 7f9990c00640 1 -- 172.21.15.114:0/3796344208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c007260 con 0x7f999406b6d0 2024-03-06T07:37:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.650+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999406e010 con 0x7f999406b6d0 2024-03-06T07:37:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.651+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999404f330 con 0x7f999406b6d0 2024-03-06T07:37:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.652+0000 7f9990c00640 1 -- 172.21.15.114:0/3796344208 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f997c007400 con 0x7f999406b6d0 2024-03-06T07:37:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.652+0000 7f9990c00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f997006b4d0 0x7f997006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.653+0000 7f9990c00640 1 -- 172.21.15.114:0/3796344208 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f997c0897a0 con 0x7f999406b6d0 2024-03-06T07:37:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.653+0000 7f9993e00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f997006b4d0 0x7f997006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.653+0000 7f9993e00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f997006b4d0 0x7f997006d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f99840011e0 tx=0x7f9984009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:50.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.657+0000 7f9990c00640 1 -- 172.21.15.114:0/3796344208 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99941f3f00 con 0x7f999406b6d0 2024-03-06T07:37:50.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.892+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99940b16e0 con 0x7f999406b6d0 2024-03-06T07:37:50.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.893+0000 7f9990c00640 1 -- 172.21.15.114:0/3796344208 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f997c0585d0 con 0x7f999406b6d0 2024-03-06T07:37:50.896 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:50.896 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:50.896 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f997006b4d0 msgr2=0x7f997006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f997006b4d0 0x7f997006d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f99840011e0 tx=0x7f9984009040 comp rx=0 tx=0).stop 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f999406b6d0 msgr2=0x7f99941f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f999406b6d0 0x7f99941f3480 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c400 tx=0x7f997c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 shutdown_connections 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f999415c120 0x7f99941f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f997006b4d0 0x7f997006d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f999406b6d0 0x7f99941f3480 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 --2- 172.21.15.114:0/3796344208 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9994066d00 0x7f999415eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 >> 172.21.15.114:0/3796344208 conn(0x7f99940b1e80 msgr2=0x7f99940ccbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:50.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 shutdown_connections 2024-03-06T07:37:50.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:50.895+0000 7f999aa00640 1 -- 172.21.15.114:0/3796344208 wait complete. 2024-03-06T07:37:51.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:51.117 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:52.092 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:52.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:52.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.328+0000 7efce0200640 1 Processor -- start 2024-03-06T07:37:52.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.329+0000 7efce0200640 1 -- start start 2024-03-06T07:37:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.330+0000 7efce0200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efcd80cf3e0 0x7efcd80ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.330+0000 7efce0200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd80cabf0 con 0x7efcd8072680 2024-03-06T07:37:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.330+0000 7efce0200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd80ce900 con 0x7efcd8144660 2024-03-06T07:37:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.330+0000 7efce0200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd80cebe0 con 0x7efcd80cf3e0 2024-03-06T07:37:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.330+0000 7efcde200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efcd80cf3e0 0x7efcd80ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.330+0000 7efcdce00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7efcd8072680 0x7efcd80c9ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40574/0 (socket says 172.21.15.114:40574) 2024-03-06T07:37:52.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.330+0000 7efcdce00640 1 -- 172.21.15.114:0/3635024589 learned_addr learned my addr 172.21.15.114:0/3635024589 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:52.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.331+0000 7efcd7e00640 1 -- 172.21.15.114:0/3635024589 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798973319 0 0) 0x7efcd80cabf0 con 0x7efcd8072680 2024-03-06T07:37:52.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.331+0000 7efcd7e00640 1 -- 172.21.15.114:0/3635024589 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc0003520 con 0x7efcd8072680 2024-03-06T07:37:52.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.331+0000 7efcde200640 1 -- 172.21.15.114:0/3635024589 >> v1:172.21.15.177:6789/0 conn(0x7efcd8144660 legacy=0x7efcd8071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.331+0000 7efcde200640 1 -- 172.21.15.114:0/3635024589 >> v1:172.21.15.114:6789/0 conn(0x7efcd8072680 legacy=0x7efcd80c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.331+0000 7efcde200640 1 -- 172.21.15.114:0/3635024589 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd80ceec0 con 0x7efcd80cf3e0 2024-03-06T07:37:52.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.332+0000 7efcde200640 1 --2- 172.21.15.114:0/3635024589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd80cf3e0 0x7efcd80ca5d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009d90 tx=0x7efcd002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02ba5b7826e3390 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.332+0000 7efcd7e00640 1 -- 172.21.15.114:0/3635024589 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0039070 con 0x7efcd80cf3e0 2024-03-06T07:37:52.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.332+0000 7efcd7e00640 1 -- 172.21.15.114:0/3635024589 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd002ccf0 con 0x7efcd80cf3e0 2024-03-06T07:37:52.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.332+0000 7efcd7e00640 1 -- 172.21.15.114:0/3635024589 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0034660 con 0x7efcd80cf3e0 2024-03-06T07:37:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.333+0000 7efce0200640 1 -- 172.21.15.114:0/3635024589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd80cf3e0 msgr2=0x7efcd80ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.333+0000 7efce0200640 1 --2- 172.21.15.114:0/3635024589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd80cf3e0 0x7efcd80ca5d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009d90 tx=0x7efcd002bf40 comp rx=0 tx=0).stop 2024-03-06T07:37:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.333+0000 7efce0200640 1 -- 172.21.15.114:0/3635024589 shutdown_connections 2024-03-06T07:37:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.333+0000 7efce0200640 1 --2- 172.21.15.114:0/3635024589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd80cf3e0 0x7efcd80ca5d0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.333+0000 7efce0200640 1 -- 172.21.15.114:0/3635024589 >> 172.21.15.114:0/3635024589 conn(0x7efcd80d0b90 msgr2=0x7efcd80d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.333+0000 7efce0200640 1 -- 172.21.15.114:0/3635024589 shutdown_connections 2024-03-06T07:37:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.333+0000 7efce0200640 1 -- 172.21.15.114:0/3635024589 wait complete. 2024-03-06T07:37:52.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.335+0000 7efce0200640 1 Processor -- start 2024-03-06T07:37:52.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.335+0000 7efce0200640 1 -- start start 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efce0200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcd8072680 0x7efcd815cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efce0200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcd80cf3e0 0x7efcd8135ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efce0200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd8144660 0x7efcd8136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efce0200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8168ed0 con 0x7efcd8144660 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efce0200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8168d50 con 0x7efcd8072680 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efce0200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8169050 con 0x7efcd80cf3e0 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efcde200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd8144660 0x7efcd8136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efcde200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd8144660 0x7efcd8136410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54066/0 (socket says 172.21.15.114:54066) 2024-03-06T07:37:52.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.336+0000 7efcde200640 1 -- 172.21.15.114:0/3727584825 learned_addr learned my addr 172.21.15.114:0/3727584825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcde200640 1 -- 172.21.15.114:0/3727584825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcd80cf3e0 msgr2=0x7efcd8135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcde200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcd80cf3e0 0x7efcd8135ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcde200640 1 -- 172.21.15.114:0/3727584825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcd8072680 msgr2=0x7efcd815cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcde200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcd8072680 0x7efcd815cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcde200640 1 -- 172.21.15.114:0/3727584825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd813ad50 con 0x7efcd8144660 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcde200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd8144660 0x7efcd8136410 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009eb0 tx=0x7efcd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcd5600640 1 -- 172.21.15.114:0/3727584825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0039050 con 0x7efcd8144660 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcd5600640 1 -- 172.21.15.114:0/3727584825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0045070 con 0x7efcd8144660 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd813b030 con 0x7efcd8144660 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.337+0000 7efcd5600640 1 -- 172.21.15.114:0/3727584825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0009450 con 0x7efcd8144660 2024-03-06T07:37:52.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.338+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd8136f30 con 0x7efcd8144660 2024-03-06T07:37:52.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.339+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd813b200 con 0x7efcd8144660 2024-03-06T07:37:52.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.339+0000 7efcd5600640 1 -- 172.21.15.114:0/3727584825 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcd003e830 con 0x7efcd8144660 2024-03-06T07:37:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.340+0000 7efcd5600640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.340+0000 7efcd5600640 1 -- 172.21.15.114:0/3727584825 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efcd00ace70 con 0x7efcd8144660 2024-03-06T07:37:52.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.341+0000 7efcdd800640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.342+0000 7efcdd800640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7efccc000f60 tx=0x7efccc0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.345+0000 7efcd5600640 1 -- 172.21.15.114:0/3727584825 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcd0032e20 con 0x7efcd8144660 2024-03-06T07:37:52.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.578+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcd804f330 con 0x7efcd8144660 2024-03-06T07:37:52.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.579+0000 7efcd5600640 1 -- 172.21.15.114:0/3727584825 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7efcd007bd20 con 0x7efcd8144660 2024-03-06T07:37:52.582 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:52.582 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:52.583 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 msgr2=0x7efcb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7efccc000f60 tx=0x7efccc0079b0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd8144660 msgr2=0x7efcd8136410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd8144660 0x7efcd8136410 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009eb0 tx=0x7efcd0004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 shutdown_connections 2024-03-06T07:37:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7efcb806b4d0 0x7efcb806d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcd8144660 0x7efcd8136410 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7efcd80cf3e0 0x7efcd8135ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 --2- 172.21.15.114:0/3727584825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efcd8072680 0x7efcd815cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 >> 172.21.15.114:0/3727584825 conn(0x7efcd80d0b90 msgr2=0x7efcd806df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 shutdown_connections 2024-03-06T07:37:52.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:52.582+0000 7efce0200640 1 -- 172.21.15.114:0/3727584825 wait complete. 2024-03-06T07:37:52.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:52.803 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:53.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:54.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.021+0000 7f3052600640 1 Processor -- start 2024-03-06T07:37:54.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.022+0000 7f3052600640 1 -- start start 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.023+0000 7f3052600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f304c144c00 0x7f304c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.023+0000 7f3052600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f304c15f9d0 0x7f304c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.023+0000 7f3052600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c0ce0a0 0x7f304c0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.023+0000 7f3052600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304c06ee70 con 0x7f304c15f9d0 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.023+0000 7f3052600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304c06cef0 con 0x7f304c144c00 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.023+0000 7f3052600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c06d1d0 con 0x7f304c0ce0a0 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f304c15f9d0 0x7f304c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f304c15f9d0 0x7f304c15fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54076/0 (socket says 172.21.15.114:54076) 2024-03-06T07:37:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 -- 172.21.15.114:0/265869801 learned_addr learned my addr 172.21.15.114:0/265869801 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f304be00640 1 --2- 172.21.15.114:0/265869801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c0ce0a0 0x7f304c0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f304b400640 1 --2- 172.21.15.114:0/265869801 >> v2:172.21.15.177:3300/0 conn(0x7f304c144c00 0x7f304c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 -- 172.21.15.114:0/265869801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c0ce0a0 msgr2=0x7f304c0ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 --2- 172.21.15.114:0/265869801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c0ce0a0 0x7f304c0ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 -- 172.21.15.114:0/265869801 >> v2:172.21.15.177:3300/0 conn(0x7f304c144c00 msgr2=0x7f304c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 --2- 172.21.15.114:0/265869801 >> v2:172.21.15.177:3300/0 conn(0x7f304c144c00 0x7f304c06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f3042a00640 1 -- 172.21.15.114:0/265869801 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c06d4b0 con 0x7f304c15f9d0 2024-03-06T07:37:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.024+0000 7f304b400640 1 --2- 172.21.15.114:0/265869801 >> v2:172.21.15.177:3300/0 conn(0x7f304c144c00 0x7f304c06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:54.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.025+0000 7f3042a00640 1 --2- 172.21.15.114:0/265869801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c15f9d0 0x7f304c15fe60 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f304c0b25f0 tx=0x7f303402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8d2a4dd374d485 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:54.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f304aa00640 1 -- 172.21.15.114:0/265869801 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3034035020 con 0x7f304c15f9d0 2024-03-06T07:37:54.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f304aa00640 1 -- 172.21.15.114:0/265869801 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303402d040 con 0x7f304c15f9d0 2024-03-06T07:37:54.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f304aa00640 1 -- 172.21.15.114:0/265869801 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3034031a80 con 0x7f304c15f9d0 2024-03-06T07:37:54.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f3052600640 1 -- 172.21.15.114:0/265869801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c15f9d0 msgr2=0x7f304c15fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f3052600640 1 --2- 172.21.15.114:0/265869801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c15f9d0 0x7f304c15fe60 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f304c0b25f0 tx=0x7f303402b900 comp rx=0 tx=0).stop 2024-03-06T07:37:54.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f3052600640 1 -- 172.21.15.114:0/265869801 shutdown_connections 2024-03-06T07:37:54.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f3052600640 1 --2- 172.21.15.114:0/265869801 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c0ce0a0 0x7f304c0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f3052600640 1 --2- 172.21.15.114:0/265869801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c15f9d0 0x7f304c15fe60 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f3052600640 1 --2- 172.21.15.114:0/265869801 >> v2:172.21.15.177:3300/0 conn(0x7f304c144c00 0x7f304c06e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.026+0000 7f3052600640 1 -- 172.21.15.114:0/265869801 >> 172.21.15.114:0/265869801 conn(0x7f304c0b1730 msgr2=0x7f304c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:54.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.027+0000 7f3052600640 1 -- 172.21.15.114:0/265869801 shutdown_connections 2024-03-06T07:37:54.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.027+0000 7f3052600640 1 -- 172.21.15.114:0/265869801 wait complete. 2024-03-06T07:37:54.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.028+0000 7f3052600640 1 Processor -- start 2024-03-06T07:37:54.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.028+0000 7f3052600640 1 -- start start 2024-03-06T07:37:54.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.029+0000 7f3052600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c0ce0a0 0x7f304c13c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.029+0000 7f3052600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c144c00 0x7f304c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.029+0000 7f3052600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c15f9d0 0x7f304c15d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.029+0000 7f3052600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c06cb50 con 0x7f304c0ce0a0 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.029+0000 7f3052600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c160730 con 0x7f304c15f9d0 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.029+0000 7f3052600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c06ccd0 con 0x7f304c144c00 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.029+0000 7f304b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c0ce0a0 0x7f304c13c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c0ce0a0 0x7f304c13c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54088/0 (socket says 172.21.15.114:54088) 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 -- 172.21.15.114:0/2293641087 learned_addr learned my addr 172.21.15.114:0/2293641087 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304be00640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c15f9d0 0x7f304c15d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f3042a00640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c144c00 0x7f304c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 -- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c144c00 msgr2=0x7f304c15d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c144c00 0x7f304c15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 -- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c15f9d0 msgr2=0x7f304c15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c15f9d0 0x7f304c15d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 -- 172.21.15.114:0/2293641087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c13cf10 con 0x7f304c0ce0a0 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304be00640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c15f9d0 0x7f304c15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.030+0000 7f304b400640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c0ce0a0 0x7f304c13c270 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f303c00c7f0 tx=0x7f303c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.031+0000 7f3043e00640 1 -- 172.21.15.114:0/2293641087 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c015070 con 0x7f304c0ce0a0 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.031+0000 7f3043e00640 1 -- 172.21.15.114:0/2293641087 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303c007a30 con 0x7f304c0ce0a0 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.031+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304c140b90 con 0x7f304c0ce0a0 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.031+0000 7f3043e00640 1 -- 172.21.15.114:0/2293641087 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c007e00 con 0x7f304c0ce0a0 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.031+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304c141150 con 0x7f304c0ce0a0 2024-03-06T07:37:54.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.032+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304c04f330 con 0x7f304c0ce0a0 2024-03-06T07:37:54.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.034+0000 7f3043e00640 1 -- 172.21.15.114:0/2293641087 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f303c018070 con 0x7f304c0ce0a0 2024-03-06T07:37:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.035+0000 7f3043e00640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f302806b4d0 0x7f302806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.035+0000 7f3043e00640 1 -- 172.21.15.114:0/2293641087 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f303c08a230 con 0x7f304c0ce0a0 2024-03-06T07:37:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.036+0000 7f3042a00640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f302806b4d0 0x7f302806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.036+0000 7f3042a00640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f302806b4d0 0x7f302806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3034036000 tx=0x7f3034008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:54.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.040+0000 7f3043e00640 1 -- 172.21.15.114:0/2293641087 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f303c058fd0 con 0x7f304c0ce0a0 2024-03-06T07:37:54.094 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.305+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f304c072890 con 0x7f304c0ce0a0 2024-03-06T07:37:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.306+0000 7f3043e00640 1 -- 172.21.15.114:0/2293641087 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f303c0588d0 con 0x7f304c0ce0a0 2024-03-06T07:37:54.308 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:54.308 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.308+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f302806b4d0 msgr2=0x7f302806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f302806b4d0 0x7f302806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3034036000 tx=0x7f3034008030 comp rx=0 tx=0).stop 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c0ce0a0 msgr2=0x7f304c13c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c0ce0a0 0x7f304c13c270 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f303c00c7f0 tx=0x7f303c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 shutdown_connections 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c15f9d0 0x7f304c15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f304c144c00 0x7f304c15d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f302806b4d0 0x7f302806d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 --2- 172.21.15.114:0/2293641087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f304c0ce0a0 0x7f304c13c270 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 >> 172.21.15.114:0/2293641087 conn(0x7f304c0b1730 msgr2=0x7f304c071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 shutdown_connections 2024-03-06T07:37:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:54.309+0000 7f3052600640 1 -- 172.21.15.114:0/2293641087 wait complete. 2024-03-06T07:37:54.531 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:54.531 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:55.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:55.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.753+0000 7f376aa00640 1 Processor -- start 2024-03-06T07:37:55.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.754+0000 7f376aa00640 1 -- start start 2024-03-06T07:37:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.755+0000 7f376aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37640cc980 0x7f376415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.755+0000 7f376aa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37640ccdd0 con 0x7f37640ce9b0 2024-03-06T07:37:55.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.755+0000 7f376aa00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37640cd060 con 0x7f376415dfa0 2024-03-06T07:37:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.755+0000 7f376aa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640cd340 con 0x7f37640cc980 2024-03-06T07:37:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.755+0000 7f3768c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f37640ce9b0 0x7f37640cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40588/0 (socket says 172.21.15.114:40588) 2024-03-06T07:37:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.755+0000 7f3768c00640 1 -- 172.21.15.114:0/298902654 learned_addr learned my addr 172.21.15.114:0/298902654 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:55.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.755+0000 7f376a000640 1 --2- 172.21.15.114:0/298902654 >> v2:172.21.15.114:3300/0 conn(0x7f37640cc980 0x7f376415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.756+0000 7f376a000640 1 -- 172.21.15.114:0/298902654 >> v1:172.21.15.177:6789/0 conn(0x7f376415dfa0 legacy=0x7f37640ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.756+0000 7f376a000640 1 -- 172.21.15.114:0/298902654 >> v1:172.21.15.114:6789/0 conn(0x7f37640ce9b0 legacy=0x7f37640cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.756+0000 7f376a000640 1 -- 172.21.15.114:0/298902654 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640cd620 con 0x7f37640cc980 2024-03-06T07:37:55.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.757+0000 7f376a000640 1 --2- 172.21.15.114:0/298902654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640cc980 0x7f376415f0f0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b38a91afceb6908 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.757+0000 7f3763e00640 1 -- 172.21.15.114:0/298902654 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c039070 con 0x7f37640cc980 2024-03-06T07:37:55.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.757+0000 7f3763e00640 1 -- 172.21.15.114:0/298902654 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c02ccf0 con 0x7f37640cc980 2024-03-06T07:37:55.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.757+0000 7f3763e00640 1 -- 172.21.15.114:0/298902654 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c034660 con 0x7f37640cc980 2024-03-06T07:37:55.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.758+0000 7f376aa00640 1 -- 172.21.15.114:0/298902654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640cc980 msgr2=0x7f376415f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.758+0000 7f376aa00640 1 --2- 172.21.15.114:0/298902654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640cc980 0x7f376415f0f0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f375c009d90 tx=0x7f375c02bf40 comp rx=0 tx=0).stop 2024-03-06T07:37:55.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.758+0000 7f376aa00640 1 -- 172.21.15.114:0/298902654 shutdown_connections 2024-03-06T07:37:55.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.758+0000 7f376aa00640 1 --2- 172.21.15.114:0/298902654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640cc980 0x7f376415f0f0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.758+0000 7f376aa00640 1 -- 172.21.15.114:0/298902654 >> 172.21.15.114:0/298902654 conn(0x7f37640b3620 msgr2=0x7f37640b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.758+0000 7f376aa00640 1 -- 172.21.15.114:0/298902654 shutdown_connections 2024-03-06T07:37:55.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.758+0000 7f376aa00640 1 -- 172.21.15.114:0/298902654 wait complete. 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.760+0000 7f376aa00640 1 Processor -- start 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.760+0000 7f376aa00640 1 -- start start 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f376aa00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37640cc980 0x7f37640cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f376aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640ce9b0 0x7f376415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f376aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376415dfa0 0x7f376415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f376aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640cfe00 con 0x7f37640ce9b0 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f376aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640cdbe0 con 0x7f376415dfa0 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f376aa00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640cff80 con 0x7f37640cc980 2024-03-06T07:37:55.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f3768c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640ce9b0 0x7f376415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f3768c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640ce9b0 0x7f376415c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54098/0 (socket says 172.21.15.114:54098) 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.761+0000 7f3768c00640 1 -- 172.21.15.114:0/1437546804 learned_addr learned my addr 172.21.15.114:0/1437546804 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f376a000640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376415dfa0 0x7f376415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f3769600640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37640cc980 0x7f37640cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f3768c00640 1 -- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37640cc980 msgr2=0x7f37640cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f3768c00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37640cc980 0x7f37640cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f3768c00640 1 -- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376415dfa0 msgr2=0x7f376415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f3768c00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376415dfa0 0x7f376415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f3768c00640 1 -- 172.21.15.114:0/1437546804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640cb800 con 0x7f37640ce9b0 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.762+0000 7f3768c00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640ce9b0 0x7f376415c310 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f375000c430 tx=0x7f375000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.763+0000 7f3761600640 1 -- 172.21.15.114:0/1437546804 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750017070 con 0x7f37640ce9b0 2024-03-06T07:37:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.763+0000 7f3761600640 1 -- 172.21.15.114:0/1437546804 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3750005ce0 con 0x7f37640ce9b0 2024-03-06T07:37:55.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.763+0000 7f3761600640 1 -- 172.21.15.114:0/1437546804 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37500072f0 con 0x7f37640ce9b0 2024-03-06T07:37:55.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.763+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37640c9260 con 0x7f37640ce9b0 2024-03-06T07:37:55.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.763+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37641562b0 con 0x7f37640ce9b0 2024-03-06T07:37:55.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.764+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376404f330 con 0x7f37640ce9b0 2024-03-06T07:37:55.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.766+0000 7f3761600640 1 -- 172.21.15.114:0/1437546804 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f375001a070 con 0x7f37640ce9b0 2024-03-06T07:37:55.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.767+0000 7f3761600640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f374406b500 0x7f374406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.767+0000 7f3761600640 1 -- 172.21.15.114:0/1437546804 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3750089660 con 0x7f37640ce9b0 2024-03-06T07:37:55.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.767+0000 7f3769600640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f374406b500 0x7f374406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.768+0000 7f3769600640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f374406b500 0x7f374406d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f37641f7430 tx=0x7f3758008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:55.776+0000 7f3761600640 1 -- 172.21.15.114:0/1437546804 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37500584a0 con 0x7f37640ce9b0 2024-03-06T07:37:56.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.012+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3764072e40 con 0x7f37640ce9b0 2024-03-06T07:37:56.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.013+0000 7f3761600640 1 -- 172.21.15.114:0/1437546804 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f3750057da0 con 0x7f37640ce9b0 2024-03-06T07:37:56.016 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:56.016 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:56.016 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:56.097 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:56.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f374406b500 msgr2=0x7f374406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f374406b500 0x7f374406d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f37641f7430 tx=0x7f3758008040 comp rx=0 tx=0).stop 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640ce9b0 msgr2=0x7f376415c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640ce9b0 0x7f376415c310 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f375000c430 tx=0x7f375000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 shutdown_connections 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f376415dfa0 0x7f376415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f374406b500 0x7f374406d9c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37640ce9b0 0x7f376415c310 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 --2- 172.21.15.114:0/1437546804 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f37640cc980 0x7f37640cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 >> 172.21.15.114:0/1437546804 conn(0x7f37640b3620 msgr2=0x7f37640ce120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 shutdown_connections 2024-03-06T07:37:56.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:56.016+0000 7f376aa00640 1 -- 172.21.15.114:0/1437546804 wait complete. 2024-03-06T07:37:56.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:56.453 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:57.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:57.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.664+0000 7f96fae00640 1 Processor -- start 2024-03-06T07:37:57.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.664+0000 7f96fae00640 1 -- start start 2024-03-06T07:37:57.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96fae00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96f415a940 0x7f96f40b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96fae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f40cc2a0 0x7f96f40cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96fae00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f40b2b30 con 0x7f96f40cf370 2024-03-06T07:37:57.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96fae00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f40b28e0 con 0x7f96f415a940 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96fae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f40b2e10 con 0x7f96f40cc2a0 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96f9a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96f415a940 0x7f96f40b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96f9a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96f415a940 0x7f96f40b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54108/0 (socket says 172.21.15.114:54108) 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.666+0000 7f96f9a00640 1 -- 172.21.15.114:0/3566400521 learned_addr learned my addr 172.21.15.114:0/3566400521 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.667+0000 7f96f9a00640 1 -- 172.21.15.114:0/3566400521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f40cc2a0 msgr2=0x7f96f40cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.667+0000 7f96f9a00640 1 --2- 172.21.15.114:0/3566400521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f40cc2a0 0x7f96f40cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.667+0000 7f96f9a00640 1 -- 172.21.15.114:0/3566400521 >> v1:172.21.15.177:6789/0 conn(0x7f96f40cf370 legacy=0x7f96f406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.667+0000 7f96f9a00640 1 -- 172.21.15.114:0/3566400521 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f406ca50 con 0x7f96f415a940 2024-03-06T07:37:57.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.668+0000 7f96f9a00640 1 --2- 172.21.15.114:0/3566400521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f415a940 0x7f96f40b45e0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009870 tx=0x7f96e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d623940c9b0e9d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.668+0000 7f96f3e00640 1 -- 172.21.15.114:0/3566400521 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e4035020 con 0x7f96f415a940 2024-03-06T07:37:57.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.668+0000 7f96f3e00640 1 -- 172.21.15.114:0/3566400521 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e4031640 con 0x7f96f415a940 2024-03-06T07:37:57.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.668+0000 7f96f3e00640 1 -- 172.21.15.114:0/3566400521 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e402d040 con 0x7f96f415a940 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 -- 172.21.15.114:0/3566400521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f415a940 msgr2=0x7f96f40b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 --2- 172.21.15.114:0/3566400521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f415a940 0x7f96f40b45e0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009870 tx=0x7f96e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 -- 172.21.15.114:0/3566400521 shutdown_connections 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 --2- 172.21.15.114:0/3566400521 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f40cc2a0 0x7f96f40cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 --2- 172.21.15.114:0/3566400521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f415a940 0x7f96f40b45e0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 -- 172.21.15.114:0/3566400521 >> 172.21.15.114:0/3566400521 conn(0x7f96f406afc0 msgr2=0x7f96f406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 -- 172.21.15.114:0/3566400521 shutdown_connections 2024-03-06T07:37:57.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.669+0000 7f96fae00640 1 -- 172.21.15.114:0/3566400521 wait complete. 2024-03-06T07:37:57.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.672+0000 7f96fae00640 1 Processor -- start 2024-03-06T07:37:57.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.672+0000 7f96fae00640 1 -- start start 2024-03-06T07:37:57.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.673+0000 7f96fae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96f40cc2a0 0x7f96f4149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.673+0000 7f96fae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f40cf370 0x7f96f4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.673+0000 7f96fae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f415a940 0x7f96f40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.673+0000 7f96fae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f40b4b20 con 0x7f96f40cf370 2024-03-06T07:37:57.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.673+0000 7f96fae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f40b30f0 con 0x7f96f40cc2a0 2024-03-06T07:37:57.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.673+0000 7f96fae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f415ad20 con 0x7f96f415a940 2024-03-06T07:37:57.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f40cf370 0x7f96f4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f40cf370 0x7f96f4070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54118/0 (socket says 172.21.15.114:54118) 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 -- 172.21.15.114:0/3081812599 learned_addr learned my addr 172.21.15.114:0/3081812599 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96fa400640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f415a940 0x7f96f40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9a00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96f40cc2a0 0x7f96f4149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 -- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f415a940 msgr2=0x7f96f40712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f415a940 0x7f96f40712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 -- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96f40cc2a0 msgr2=0x7f96f4149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96f40cc2a0 0x7f96f4149fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 -- 172.21.15.114:0/3081812599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f414aa80 con 0x7f96f40cf370 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96fa400640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f415a940 0x7f96f40712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.674+0000 7f96f9000640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f40cf370 0x7f96f4070d80 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f96e000c410 tx=0x7f96e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.675+0000 7f96f1600640 1 -- 172.21.15.114:0/3081812599 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0017070 con 0x7f96f40cf370 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.675+0000 7f96f1600640 1 -- 172.21.15.114:0/3081812599 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e00035f0 con 0x7f96f40cf370 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.675+0000 7f96f1600640 1 -- 172.21.15.114:0/3081812599 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0003a20 con 0x7f96f40cf370 2024-03-06T07:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.675+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f414ad60 con 0x7f96f40cf370 2024-03-06T07:37:57.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.675+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f414b320 con 0x7f96f40cf370 2024-03-06T07:37:57.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.676+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f404f330 con 0x7f96f40cf370 2024-03-06T07:37:57.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.677+0000 7f96f1600640 1 -- 172.21.15.114:0/3081812599 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96e0003bc0 con 0x7f96f40cf370 2024-03-06T07:37:57.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.678+0000 7f96f1600640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f96d406b4d0 0x7f96d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.678+0000 7f96f1600640 1 -- 172.21.15.114:0/3081812599 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f96e00896f0 con 0x7f96f40cf370 2024-03-06T07:37:57.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.678+0000 7f96f9a00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f96d406b4d0 0x7f96d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.679+0000 7f96f9a00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f96d406b4d0 0x7f96d406d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f96e4036000 tx=0x7f96e403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.682+0000 7f96f1600640 1 -- 172.21.15.114:0/3081812599 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96e0058490 con 0x7f96f40cf370 2024-03-06T07:37:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.921+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96f414b4f0 con 0x7f96f40cf370 2024-03-06T07:37:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.922+0000 7f96f1600640 1 -- 172.21.15.114:0/3081812599 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f96e00010c0 con 0x7f96f40cf370 2024-03-06T07:37:57.924 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:57.924 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.924+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f96d406b4d0 msgr2=0x7f96d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.924+0000 7f96fae00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f96d406b4d0 0x7f96d406d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f96e4036000 tx=0x7f96e403c040 comp rx=0 tx=0).stop 2024-03-06T07:37:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.924+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f40cf370 msgr2=0x7f96f4070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.924+0000 7f96fae00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f40cf370 0x7f96f4070d80 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f96e000c410 tx=0x7f96e000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 shutdown_connections 2024-03-06T07:37:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f96f415a940 0x7f96f40712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f96d406b4d0 0x7f96d406d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f40cf370 0x7f96f4070d80 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 --2- 172.21.15.114:0/3081812599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96f40cc2a0 0x7f96f4149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 >> 172.21.15.114:0/3081812599 conn(0x7f96f406afc0 msgr2=0x7f96f4071f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 shutdown_connections 2024-03-06T07:37:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:57.925+0000 7f96fae00640 1 -- 172.21.15.114:0/3081812599 wait complete. 2024-03-06T07:37:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:58.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:58.147 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:37:59.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:59.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.379+0000 7f8107c00640 1 Processor -- start 2024-03-06T07:37:59.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.379+0000 7f8107c00640 1 -- start start 2024-03-06T07:37:59.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.381+0000 7f8107c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f810015e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.381+0000 7f8107c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000b35b0 con 0x7f81000cf430 2024-03-06T07:37:59.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.381+0000 7f8107c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000b3890 con 0x7f8100151660 2024-03-06T07:37:59.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.381+0000 7f8107c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81000b3ee0 con 0x7f810015dff0 2024-03-06T07:37:59.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.381+0000 7f80f7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f81000cf430 0x7f810006d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48566/0 (socket says 172.21.15.114:48566) 2024-03-06T07:37:59.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.381+0000 7f80f7e00640 1 -- 172.21.15.114:0/4159144763 learned_addr learned my addr 172.21.15.114:0/4159144763 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:37:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.381+0000 7f8105c00640 1 --2- 172.21.15.114:0/4159144763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f810015e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.382+0000 7f80ffe00640 1 -- 172.21.15.114:0/4159144763 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45873900 0 0) 0x7f81000b35b0 con 0x7f81000cf430 2024-03-06T07:37:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.382+0000 7f80ffe00640 1 -- 172.21.15.114:0/4159144763 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e4003520 con 0x7f81000cf430 2024-03-06T07:37:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.382+0000 7f80ffe00640 1 -- 172.21.15.114:0/4159144763 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456228697 0 0) 0x7f80e4003520 con 0x7f81000cf430 2024-03-06T07:37:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.382+0000 7f8105c00640 1 -- 172.21.15.114:0/4159144763 >> v1:172.21.15.177:6789/0 conn(0x7f8100151660 legacy=0x7f8100071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.382+0000 7f8105c00640 1 -- 172.21.15.114:0/4159144763 >> v1:172.21.15.114:6789/0 conn(0x7f81000cf430 legacy=0x7f810006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.382+0000 7f8105c00640 1 -- 172.21.15.114:0/4159144763 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000b3b70 con 0x7f810015dff0 2024-03-06T07:37:59.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.384+0000 7f8105c00640 1 --2- 172.21.15.114:0/4159144763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f810015e480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f80f800aed0 tx=0x7f80f802cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6deb776cc42ec3db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.384+0000 7f80ffe00640 1 -- 172.21.15.114:0/4159144763 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8001670 con 0x7f810015dff0 2024-03-06T07:37:59.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.384+0000 7f80ffe00640 1 -- 172.21.15.114:0/4159144763 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f802d550 con 0x7f810015dff0 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.384+0000 7f80ffe00640 1 -- 172.21.15.114:0/4159144763 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f802d900 con 0x7f810015dff0 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.385+0000 7f8107c00640 1 -- 172.21.15.114:0/4159144763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 msgr2=0x7f810015e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.385+0000 7f8107c00640 1 --2- 172.21.15.114:0/4159144763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f810015e480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f80f800aed0 tx=0x7f80f802cc30 comp rx=0 tx=0).stop 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.385+0000 7f8107c00640 1 -- 172.21.15.114:0/4159144763 shutdown_connections 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.385+0000 7f8107c00640 1 --2- 172.21.15.114:0/4159144763 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f810015e480 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.385+0000 7f8107c00640 1 -- 172.21.15.114:0/4159144763 >> 172.21.15.114:0/4159144763 conn(0x7f81000b1730 msgr2=0x7f81000b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.385+0000 7f8107c00640 1 -- 172.21.15.114:0/4159144763 shutdown_connections 2024-03-06T07:37:59.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.385+0000 7f8107c00640 1 -- 172.21.15.114:0/4159144763 wait complete. 2024-03-06T07:37:59.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.386+0000 7f8107c00640 1 Processor -- start 2024-03-06T07:37:59.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.387+0000 7f8107c00640 1 -- start start 2024-03-06T07:37:59.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.387+0000 7f8107c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81000cf430 0x7f810014b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.387+0000 7f8107c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8100151660 0x7f810014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.387+0000 7f8107c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f8100148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.387+0000 7f8107c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810006cb50 con 0x7f8100151660 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.387+0000 7f8107c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810006daf0 con 0x7f81000cf430 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.387+0000 7f8107c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810006ccd0 con 0x7f810015dff0 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8100151660 0x7f810014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8100151660 0x7f810014e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47130/0 (socket says 172.21.15.114:47130) 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 -- 172.21.15.114:0/4127592353 learned_addr learned my addr 172.21.15.114:0/4127592353 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f8105c00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f8100148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 -- 172.21.15.114:0/4127592353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 msgr2=0x7f8100148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f8100148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 -- 172.21.15.114:0/4127592353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81000cf430 msgr2=0x7f810014b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81000cf430 0x7f810014b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 -- 172.21.15.114:0/4127592353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810014d420 con 0x7f8100151660 2024-03-06T07:37:59.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.388+0000 7f80f7e00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8100151660 0x7f810014e900 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f80e800c750 tx=0x7f80e800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.389+0000 7f80fd600640 1 -- 172.21.15.114:0/4127592353 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8018070 con 0x7f8100151660 2024-03-06T07:37:59.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.389+0000 7f80fd600640 1 -- 172.21.15.114:0/4127592353 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e80072d0 con 0x7f8100151660 2024-03-06T07:37:59.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.389+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81000cc8b0 con 0x7f8100151660 2024-03-06T07:37:59.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.389+0000 7f80fd600640 1 -- 172.21.15.114:0/4127592353 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e80047e0 con 0x7f8100151660 2024-03-06T07:37:59.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.389+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81000cce70 con 0x7f8100151660 2024-03-06T07:37:59.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.390+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810004f330 con 0x7f8100151660 2024-03-06T07:37:59.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.391+0000 7f80fd600640 1 -- 172.21.15.114:0/4127592353 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80e801b070 con 0x7f8100151660 2024-03-06T07:37:59.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.392+0000 7f80fd600640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f80dc06b4d0 0x7f80dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.392+0000 7f80fd600640 1 -- 172.21.15.114:0/4127592353 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f80e8089780 con 0x7f8100151660 2024-03-06T07:37:59.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.392+0000 7f8105200640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f80dc06b4d0 0x7f80dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.393+0000 7f8105200640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f80dc06b4d0 0x7f80dc06d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000f60 tx=0x7f80f0007990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.400+0000 7f80fd600640 1 -- 172.21.15.114:0/4127592353 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80e80585b0 con 0x7f8100151660 2024-03-06T07:37:59.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.637+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81000722c0 con 0x7f8100151660 2024-03-06T07:37:59.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.638+0000 7f80fd600640 1 -- 172.21.15.114:0/4127592353 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f80e8006e20 con 0x7f8100151660 2024-03-06T07:37:59.640 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:37:59.640 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:37:59.640 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f80dc06b4d0 msgr2=0x7f80dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f80dc06b4d0 0x7f80dc06d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000f60 tx=0x7f80f0007990 comp rx=0 tx=0).stop 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8100151660 msgr2=0x7f810014e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8100151660 0x7f810014e900 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f80e800c750 tx=0x7f80e800cc10 comp rx=0 tx=0).stop 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 shutdown_connections 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f810015dff0 0x7f8100148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f80dc06b4d0 0x7f80dc06d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8100151660 0x7f810014e900 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 --2- 172.21.15.114:0/4127592353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81000cf430 0x7f810014b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 >> 172.21.15.114:0/4127592353 conn(0x7f81000b1730 msgr2=0x7f8100150400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 shutdown_connections 2024-03-06T07:37:59.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:37:59.641+0000 7f8107c00640 1 -- 172.21.15.114:0/4127592353 wait complete. 2024-03-06T07:37:59.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:37:59.870 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:00.100 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:00.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:01.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.090+0000 7f9bc1e00640 1 Processor -- start 2024-03-06T07:38:01.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.090+0000 7f9bc1e00640 1 -- start start 2024-03-06T07:38:01.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.091+0000 7f9bc1e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 0x7f9bbc0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.092+0000 7f9bc1e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bbc15c3d0 con 0x7f9bbc15fac0 2024-03-06T07:38:01.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.092+0000 7f9bc1e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bbc0cdb50 con 0x7f9bbc15bff0 2024-03-06T07:38:01.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.092+0000 7f9bc1e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0cf8c0 con 0x7f9bbc1606d0 2024-03-06T07:38:01.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.092+0000 7f9bbb400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9bbc15fac0 0x7f9bbc06ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48580/0 (socket says 172.21.15.114:48580) 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.092+0000 7f9bbb400640 1 -- 172.21.15.114:0/3081580813 learned_addr learned my addr 172.21.15.114:0/3081580813 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.092+0000 7f9bb2000640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 0x7f9bbc0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787973885 0 0) 0x7f9bbc15c3d0 con 0x7f9bbc15fac0 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba0003520 con 0x7f9bbc15fac0 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309638151 0 0) 0x7f9bbc0cdb50 con 0x7f9bbc15bff0 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bbc15c3d0 con 0x7f9bbc15bff0 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3435424372 0 0) 0x7f9ba0003520 con 0x7f9bbc15fac0 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 msgr2=0x7f9bbc0cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 0x7f9bbc0cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 >> v1:172.21.15.177:6789/0 conn(0x7f9bbc15bff0 legacy=0x7f9bbc0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0cfba0 con 0x7f9bbc15fac0 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f9bb40019f0 con 0x7f9bbc15fac0 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 >> v1:172.21.15.114:6789/0 conn(0x7f9bbc15fac0 legacy=0x7f9bbc06ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0003b70 0x7f9ba0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 0x7f9ba0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0004d40 0x7f9ba00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0cf8c0 con 0x7f9ba0004d40 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba000baa0 con 0x7f9ba0003b70 2024-03-06T07:38:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.093+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00100b0 con 0x7f9bbc1606d0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bbb400640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0004d40 0x7f9ba00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bb2000640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 0x7f9ba0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bbaa00640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0003b70 0x7f9ba0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bbb400640 1 -- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 msgr2=0x7f9ba0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bbb400640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bbc1606d0 0x7f9ba0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bbb400640 1 -- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0003b70 msgr2=0x7f9ba0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bbb400640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0003b70 0x7f9ba0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.094+0000 7f9bbb400640 1 -- 172.21.15.114:0/3081580813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba0003520 con 0x7f9ba0004d40 2024-03-06T07:38:01.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.095+0000 7f9bbb400640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0004d40 0x7f9ba00051c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400a760 tx=0x7f9ba0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4516d996a9f43678 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.095+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400a3b0 con 0x7f9ba0004d40 2024-03-06T07:38:01.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.095+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb4034630 con 0x7f9ba0004d40 2024-03-06T07:38:01.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.095+0000 7f9bba000640 1 -- 172.21.15.114:0/3081580813 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4034a60 con 0x7f9ba0004d40 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.096+0000 7f9bc1e00640 1 -- 172.21.15.114:0/3081580813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0004d40 msgr2=0x7f9ba00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.096+0000 7f9bc1e00640 1 --2- 172.21.15.114:0/3081580813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ba0004d40 0x7f9ba00051c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400a760 tx=0x7f9ba0005830 comp rx=0 tx=0).stop 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.096+0000 7f9bbb400640 1 -- 172.21.15.114:0/3081580813 reap_dead start 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.096+0000 7f9bc1e00640 1 -- 172.21.15.114:0/3081580813 shutdown_connections 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.096+0000 7f9bc1e00640 1 -- 172.21.15.114:0/3081580813 >> 172.21.15.114:0/3081580813 conn(0x7f9bbc0d1c50 msgr2=0x7f9bbc0d2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.096+0000 7f9bc1e00640 1 -- 172.21.15.114:0/3081580813 shutdown_connections 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.096+0000 7f9bc1e00640 1 -- 172.21.15.114:0/3081580813 wait complete. 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.098+0000 7f9bc1e00640 1 Processor -- start 2024-03-06T07:38:01.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.098+0000 7f9bc1e00640 1 -- start start 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bc1e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ba0004d40 0x7f9bbc1f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bc1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bbc14d1a0 0x7f9bbc0ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bc1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc14d620 0x7f9bbc0ced20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bc1e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc06f430 con 0x7f9bbc14d620 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bc1e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0d0160 con 0x7f9bbc14d1a0 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bc1e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc06f5b0 con 0x7f9ba0004d40 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bbb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc14d620 0x7f9bbc0ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bbb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc14d620 0x7f9bbc0ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47146/0 (socket says 172.21.15.114:47146) 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.100+0000 7f9bbb400640 1 -- 172.21.15.114:0/20687945 learned_addr learned my addr 172.21.15.114:0/20687945 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bbaa00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ba0004d40 0x7f9bbc1f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bb2000640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bbc14d1a0 0x7f9bbc0ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bbb400640 1 -- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ba0004d40 msgr2=0x7f9bbc1f39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bbb400640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ba0004d40 0x7f9bbc1f39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bbb400640 1 -- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bbc14d1a0 msgr2=0x7f9bbc0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bbb400640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bbc14d1a0 0x7f9bbc0ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bbb400640 1 -- 172.21.15.114:0/20687945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0c8380 con 0x7f9bbc14d620 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bbb400640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc14d620 0x7f9bbc0ced20 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400a760 tx=0x7f9bb4001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.101+0000 7f9bb3e00640 1 -- 172.21.15.114:0/20687945 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400a3b0 con 0x7f9bbc14d620 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.102+0000 7f9bb3e00640 1 -- 172.21.15.114:0/20687945 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb4044070 con 0x7f9bbc14d620 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.102+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bbc0b0d40 con 0x7f9bbc14d620 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.102+0000 7f9bb3e00640 1 -- 172.21.15.114:0/20687945 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4034bf0 con 0x7f9bbc14d620 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.102+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bbc0c8030 con 0x7f9bbc14d620 2024-03-06T07:38:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.103+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bbc04f330 con 0x7f9bbc14d620 2024-03-06T07:38:01.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.104+0000 7f9bb3e00640 1 -- 172.21.15.114:0/20687945 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bb4049460 con 0x7f9bbc14d620 2024-03-06T07:38:01.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.106+0000 7f9bb3e00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b9806b4d0 0x7f9b9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.106+0000 7f9bb3e00640 1 -- 172.21.15.114:0/20687945 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9bb40ac900 con 0x7f9bbc14d620 2024-03-06T07:38:01.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.106+0000 7f9bbaa00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b9806b4d0 0x7f9b9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.107+0000 7f9bbaa00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b9806b4d0 0x7f9b9806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9bac007ae0 tx=0x7f9bac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.112+0000 7f9bb3e00640 1 -- 172.21.15.114:0/20687945 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bb407b850 con 0x7f9bbc14d620 2024-03-06T07:38:01.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.375+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bbc1586b0 con 0x7f9bbc14d620 2024-03-06T07:38:01.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.375+0000 7f9bb3e00640 1 -- 172.21.15.114:0/20687945 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f9bb407b150 con 0x7f9bbc14d620 2024-03-06T07:38:01.377 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:01.377 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:01.378 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b9806b4d0 msgr2=0x7f9b9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b9806b4d0 0x7f9b9806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9bac007ae0 tx=0x7f9bac009040 comp rx=0 tx=0).stop 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc14d620 msgr2=0x7f9bbc0ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc14d620 0x7f9bbc0ced20 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400a760 tx=0x7f9bb4001cf0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 shutdown_connections 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9b9806b4d0 0x7f9b9806d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bbc14d620 0x7f9bbc0ced20 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bbc14d1a0 0x7f9bbc0ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 --2- 172.21.15.114:0/20687945 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9ba0004d40 0x7f9bbc1f39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 >> 172.21.15.114:0/20687945 conn(0x7f9bbc0d1c50 msgr2=0x7f9bbc15a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 shutdown_connections 2024-03-06T07:38:01.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:01.378+0000 7f9bc1e00640 1 -- 172.21.15.114:0/20687945 wait complete. 2024-03-06T07:38:01.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:01.602 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:02.102 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:02.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:02.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.813+0000 7fe0e1800640 1 Processor -- start 2024-03-06T07:38:02.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.813+0000 7fe0e1800640 1 -- start start 2024-03-06T07:38:02.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0e1800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe0dc15f470 0x7fe0dc0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0e1800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0dc0cea60 con 0x7fe0dc15e0d0 2024-03-06T07:38:02.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0e1800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0dc0d2f60 con 0x7fe0dc15f470 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0e1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc0d1240 con 0x7fe0dc0d0df0 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0daa00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe0dc15e0d0 0x7fe0dc0d1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48588/0 (socket says 172.21.15.114:48588) 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0daa00640 1 -- 172.21.15.114:0/1556364383 learned_addr learned my addr 172.21.15.114:0/1556364383 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0d2000640 1 --2- 172.21.15.114:0/1556364383 >> v2:172.21.15.114:3300/0 conn(0x7fe0dc15f470 0x7fe0dc0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0db400640 1 --2- 172.21.15.114:0/1556364383 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0da000640 1 -- 172.21.15.114:0/1556364383 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105927572 0 0) 0x7fe0dc0cea60 con 0x7fe0dc15e0d0 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.815+0000 7fe0da000640 1 -- 172.21.15.114:0/1556364383 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c0003520 con 0x7fe0dc15e0d0 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.816+0000 7fe0d2000640 1 -- 172.21.15.114:0/1556364383 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 msgr2=0x7fe0dc160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.816+0000 7fe0d2000640 1 --2- 172.21.15.114:0/1556364383 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.816+0000 7fe0d2000640 1 -- 172.21.15.114:0/1556364383 >> v1:172.21.15.114:6789/0 conn(0x7fe0dc15e0d0 legacy=0x7fe0dc0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.816+0000 7fe0d2000640 1 -- 172.21.15.114:0/1556364383 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc0d2350 con 0x7fe0dc15f470 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.817+0000 7fe0d2000640 1 --2- 172.21.15.114:0/1556364383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0d08b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40097a0 tx=0x7fe0c402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bf652f6e1d477d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.817+0000 7fe0da000640 1 -- 172.21.15.114:0/1556364383 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4034070 con 0x7fe0dc15f470 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.817+0000 7fe0da000640 1 -- 172.21.15.114:0/1556364383 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c4030800 con 0x7fe0dc15f470 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.817+0000 7fe0da000640 1 -- 172.21.15.114:0/1556364383 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4030bb0 con 0x7fe0dc15f470 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 -- 172.21.15.114:0/1556364383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 msgr2=0x7fe0dc0d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1556364383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0d08b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40097a0 tx=0x7fe0c402b940 comp rx=0 tx=0).stop 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 -- 172.21.15.114:0/1556364383 shutdown_connections 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1556364383 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1556364383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0d08b0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 -- 172.21.15.114:0/1556364383 >> 172.21.15.114:0/1556364383 conn(0x7fe0dc0d3ec0 msgr2=0x7fe0dc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 -- 172.21.15.114:0/1556364383 shutdown_connections 2024-03-06T07:38:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.818+0000 7fe0e1800640 1 -- 172.21.15.114:0/1556364383 wait complete. 2024-03-06T07:38:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.820+0000 7fe0e1800640 1 Processor -- start 2024-03-06T07:38:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.820+0000 7fe0e1800640 1 -- start start 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc0cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0dc15e0d0 0x7fe0dc0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0e1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0e1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc0d2bf0 con 0x7fe0dc15f470 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0e1800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc0d2910 con 0x7fe0dc15e0d0 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0e1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc15fb30 con 0x7fe0dc0d0df0 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47170/0 (socket says 172.21.15.114:47170) 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 -- 172.21.15.114:0/1102809894 learned_addr learned my addr 172.21.15.114:0/1102809894 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0daa00640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc0cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0d2000640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0dc15e0d0 0x7fe0dc0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 -- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 msgr2=0x7fe0dc0cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc0cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 -- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0dc15e0d0 msgr2=0x7fe0dc0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0dc15e0d0 0x7fe0dc0c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0db400640 1 -- 172.21.15.114:0/1102809894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc0c9df0 con 0x7fe0dc15f470 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.821+0000 7fe0daa00640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc0cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.822+0000 7fe0db400640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0c98b0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400d600 tx=0x7fe0d400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.822+0000 7fe0d3400640 1 -- 172.21.15.114:0/1102809894 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40125f0 con 0x7fe0dc15f470 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.822+0000 7fe0d3400640 1 -- 172.21.15.114:0/1102809894 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d4019070 con 0x7fe0dc15f470 2024-03-06T07:38:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.822+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc0b7620 con 0x7fe0dc15f470 2024-03-06T07:38:02.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.822+0000 7fe0d3400640 1 -- 172.21.15.114:0/1102809894 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40129f0 con 0x7fe0dc15f470 2024-03-06T07:38:02.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.822+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc0b7be0 con 0x7fe0dc15f470 2024-03-06T07:38:02.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.824+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc0b7f10 con 0x7fe0dc15f470 2024-03-06T07:38:02.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.824+0000 7fe0d3400640 1 -- 172.21.15.114:0/1102809894 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0d4012b90 con 0x7fe0dc15f470 2024-03-06T07:38:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.826+0000 7fe0d3400640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe0b806b500 0x7fe0b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:02.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.826+0000 7fe0d3400640 1 -- 172.21.15.114:0/1102809894 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe0d408a650 con 0x7fe0dc15f470 2024-03-06T07:38:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.826+0000 7fe0daa00640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe0b806b500 0x7fe0b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:02.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.827+0000 7fe0daa00640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe0b806b500 0x7fe0b806d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005c00 tx=0x7fe0cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:02.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:02.830+0000 7fe0d3400640 1 -- 172.21.15.114:0/1102809894 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0d4059400 con 0x7fe0dc15f470 2024-03-06T07:38:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.063+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0dc04f330 con 0x7fe0dc15f470 2024-03-06T07:38:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.063+0000 7fe0d3400640 1 -- 172.21.15.114:0/1102809894 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fe0d4058d00 con 0x7fe0dc15f470 2024-03-06T07:38:03.065 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:03.065 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:03.066 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe0b806b500 msgr2=0x7fe0b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe0b806b500 0x7fe0b806d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005c00 tx=0x7fe0cc009040 comp rx=0 tx=0).stop 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 msgr2=0x7fe0dc0c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0c98b0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400d600 tx=0x7fe0d400dac0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 shutdown_connections 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe0b806b500 0x7fe0b806d9c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0dc15f470 0x7fe0dc0c98b0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0dc15e0d0 0x7fe0dc0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 --2- 172.21.15.114:0/1102809894 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe0dc0d0df0 0x7fe0dc0cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 >> 172.21.15.114:0/1102809894 conn(0x7fe0dc0d3ec0 msgr2=0x7fe0dc06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:03.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 shutdown_connections 2024-03-06T07:38:03.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:03.067+0000 7fe0e1800640 1 -- 172.21.15.114:0/1102809894 wait complete. 2024-03-06T07:38:03.290 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:03.290 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:04.105 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:04.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.504+0000 7f870ec00640 1 Processor -- start 2024-03-06T07:38:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.504+0000 7f870ec00640 1 -- start start 2024-03-06T07:38:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f870ec00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f87080d24c0 0x7f87080d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f870ec00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87080b2de0 0x7f87080d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f870ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f870ec00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87080d4a00 con 0x7f87080b2de0 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f870ec00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87080cf7e0 con 0x7f87080d24c0 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f870ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87080d0080 con 0x7f87080d07f0 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f86ffe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87080b2de0 0x7f87080d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f86ffe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87080b2de0 0x7f87080d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47176/0 (socket says 172.21.15.114:47176) 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f86ffe00640 1 -- 172.21.15.114:0/2634005635 learned_addr learned my addr 172.21.15.114:0/2634005635 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f8707e00640 1 --2- 172.21.15.114:0/2634005635 >> v2:172.21.15.177:3300/0 conn(0x7f87080d24c0 0x7f87080d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.506+0000 7f870cc00640 1 --2- 172.21.15.114:0/2634005635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.507+0000 7f86ffe00640 1 -- 172.21.15.114:0/2634005635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 msgr2=0x7f87080d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.507+0000 7f86ffe00640 1 --2- 172.21.15.114:0/2634005635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080d0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.507+0000 7f86ffe00640 1 -- 172.21.15.114:0/2634005635 >> v2:172.21.15.177:3300/0 conn(0x7f87080d24c0 msgr2=0x7f87080d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.507+0000 7f86ffe00640 1 --2- 172.21.15.114:0/2634005635 >> v2:172.21.15.177:3300/0 conn(0x7f87080d24c0 0x7f87080d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.507+0000 7f86ffe00640 1 -- 172.21.15.114:0/2634005635 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87080d2de0 con 0x7f87080b2de0 2024-03-06T07:38:04.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.507+0000 7f8707e00640 1 --2- 172.21.15.114:0/2634005635 >> v2:172.21.15.177:3300/0 conn(0x7f87080d24c0 0x7f87080d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:04.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.508+0000 7f86ffe00640 1 --2- 172.21.15.114:0/2634005635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080d28a0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f870808d300 tx=0x7f86f002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641d412de8d0ce96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f8707400640 1 -- 172.21.15.114:0/2634005635 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f002e070 con 0x7f87080b2de0 2024-03-06T07:38:04.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f8707400640 1 -- 172.21.15.114:0/2634005635 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f0037070 con 0x7f87080b2de0 2024-03-06T07:38:04.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f8707400640 1 -- 172.21.15.114:0/2634005635 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0031a00 con 0x7f87080b2de0 2024-03-06T07:38:04.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f870ec00640 1 -- 172.21.15.114:0/2634005635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 msgr2=0x7f87080d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f870ec00640 1 --2- 172.21.15.114:0/2634005635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080d28a0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f870808d300 tx=0x7f86f002b880 comp rx=0 tx=0).stop 2024-03-06T07:38:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f870ec00640 1 -- 172.21.15.114:0/2634005635 shutdown_connections 2024-03-06T07:38:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f870ec00640 1 --2- 172.21.15.114:0/2634005635 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080d0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f870ec00640 1 --2- 172.21.15.114:0/2634005635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080d28a0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f870ec00640 1 --2- 172.21.15.114:0/2634005635 >> v2:172.21.15.177:3300/0 conn(0x7f87080d24c0 0x7f87080d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.509+0000 7f870ec00640 1 -- 172.21.15.114:0/2634005635 >> 172.21.15.114:0/2634005635 conn(0x7f87080d3ee0 msgr2=0x7f87080d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.510+0000 7f870ec00640 1 -- 172.21.15.114:0/2634005635 shutdown_connections 2024-03-06T07:38:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.510+0000 7f870ec00640 1 -- 172.21.15.114:0/2634005635 wait complete. 2024-03-06T07:38:04.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.512+0000 7f870ec00640 1 Processor -- start 2024-03-06T07:38:04.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.513+0000 7f870ec00640 1 -- start start 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.513+0000 7f870ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.513+0000 7f870ec00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.513+0000 7f870ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87080d24c0 0x7f8708158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.513+0000 7f870ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87080cfd10 con 0x7f87080b2de0 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.513+0000 7f870ec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87080cfa30 con 0x7f87080d24c0 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.513+0000 7f870ec00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870815e5b0 con 0x7f87080d07f0 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080c8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47180/0 (socket says 172.21.15.114:47180) 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 -- 172.21.15.114:0/351389006 learned_addr learned my addr 172.21.15.114:0/351389006 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f86ffe00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f870cc00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87080d24c0 0x7f8708158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 -- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 msgr2=0x7f87080ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 -- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87080d24c0 msgr2=0x7f8708158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87080d24c0 0x7f8708158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f8707e00640 1 -- 172.21.15.114:0/351389006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87081f4290 con 0x7f87080b2de0 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.514+0000 7f870cc00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87080d24c0 0x7f8708158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:04.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.515+0000 7f8707e00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080c8400 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f86f800e7e0 tx=0x7f86f800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.515+0000 7f8704c00640 1 -- 172.21.15.114:0/351389006 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f800f040 con 0x7f87080b2de0 2024-03-06T07:38:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.515+0000 7f8704c00640 1 -- 172.21.15.114:0/351389006 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f8004030 con 0x7f87080b2de0 2024-03-06T07:38:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.515+0000 7f8704c00640 1 -- 172.21.15.114:0/351389006 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8004400 con 0x7f87080b2de0 2024-03-06T07:38:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.515+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87081f4570 con 0x7f87080b2de0 2024-03-06T07:38:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.515+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87081f4b30 con 0x7f87080b2de0 2024-03-06T07:38:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.516+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870804f330 con 0x7f87080b2de0 2024-03-06T07:38:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.518+0000 7f8704c00640 1 -- 172.21.15.114:0/351389006 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86f800c6d0 con 0x7f87080b2de0 2024-03-06T07:38:04.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.519+0000 7f8704c00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f86e406b4d0 0x7f86e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.520+0000 7f8704c00640 1 -- 172.21.15.114:0/351389006 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f86f808a0c0 con 0x7f87080b2de0 2024-03-06T07:38:04.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.520+0000 7f86ffe00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f86e406b4d0 0x7f86e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.520+0000 7f86ffe00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f86e406b4d0 0x7f86e406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036000 tx=0x7f86f0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.524+0000 7f8704c00640 1 -- 172.21.15.114:0/351389006 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86f8017020 con 0x7f87080b2de0 2024-03-06T07:38:04.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.800+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87080d22a0 con 0x7f87080b2de0 2024-03-06T07:38:04.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.801+0000 7f8704c00640 1 -- 172.21.15.114:0/351389006 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f86f8058ee0 con 0x7f87080b2de0 2024-03-06T07:38:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:04.803 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:04.803 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:04.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f86e406b4d0 msgr2=0x7f86e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f86e406b4d0 0x7f86e406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036000 tx=0x7f86f0008030 comp rx=0 tx=0).stop 2024-03-06T07:38:04.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 msgr2=0x7f87080c8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080c8400 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f86f800e7e0 tx=0x7f86f800eca0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 shutdown_connections 2024-03-06T07:38:04.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87080d24c0 0x7f8708158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87080d07f0 0x7f87080ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f86e406b4d0 0x7f86e406d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 --2- 172.21.15.114:0/351389006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87080b2de0 0x7f87080c8400 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 >> 172.21.15.114:0/351389006 conn(0x7f87080d3ee0 msgr2=0x7f87080d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:04.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 shutdown_connections 2024-03-06T07:38:04.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:04.804+0000 7f870ec00640 1 -- 172.21.15.114:0/351389006 wait complete. 2024-03-06T07:38:05.028 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:05.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:06.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:06.106 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:06.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.244+0000 7f8fc7600640 1 Processor -- start 2024-03-06T07:38:06.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.244+0000 7f8fc7600640 1 -- start start 2024-03-06T07:38:06.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.246+0000 7f8fc7600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc015acb0 0x7f8fc00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.246+0000 7f8fc7600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8fc00b4680 0x7f8fc006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.246+0000 7f8fc7600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc015b090 con 0x7f8fc0072340 2024-03-06T07:38:06.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.246+0000 7f8fc7600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc00b2a30 con 0x7f8fc00b4680 2024-03-06T07:38:06.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.246+0000 7f8fc7600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00b4360 con 0x7f8fc015acb0 2024-03-06T07:38:06.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.247+0000 7f8fc4c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc015acb0 0x7f8fc00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.247+0000 7f8fc4c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc015acb0 0x7f8fc00b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:41070/0 (socket says 172.21.15.114:41070) 2024-03-06T07:38:06.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.247+0000 7f8fc4c00640 1 -- 172.21.15.114:0/4055666310 learned_addr learned my addr 172.21.15.114:0/4055666310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:06.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.247+0000 7f8fb7e00640 1 --2- 172.21.15.114:0/4055666310 >> v2:172.21.15.177:3300/0 conn(0x7f8fc00b4680 0x7f8fc006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.248+0000 7f8fc4c00640 1 -- 172.21.15.114:0/4055666310 >> v2:172.21.15.177:3300/0 conn(0x7f8fc00b4680 msgr2=0x7f8fc006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:06.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.248+0000 7f8fc4c00640 1 --2- 172.21.15.114:0/4055666310 >> v2:172.21.15.177:3300/0 conn(0x7f8fc00b4680 0x7f8fc006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.248+0000 7f8fc4c00640 1 -- 172.21.15.114:0/4055666310 >> v1:172.21.15.177:6789/0 conn(0x7f8fc0072340 legacy=0x7f8fc00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.248+0000 7f8fc4c00640 1 -- 172.21.15.114:0/4055666310 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc00b2cf0 con 0x7f8fc015acb0 2024-03-06T07:38:06.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.249+0000 7f8fc4c00640 1 --2- 172.21.15.114:0/4055666310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc015acb0 0x7f8fc00b37d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0009870 tx=0x7f8fb002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf9260a7e7d0ff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.249+0000 7f8fbfe00640 1 -- 172.21.15.114:0/4055666310 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0034070 con 0x7f8fc015acb0 2024-03-06T07:38:06.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.249+0000 7f8fbfe00640 1 -- 172.21.15.114:0/4055666310 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb0030840 con 0x7f8fc015acb0 2024-03-06T07:38:06.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.249+0000 7f8fbfe00640 1 -- 172.21.15.114:0/4055666310 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0030c10 con 0x7f8fc015acb0 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 -- 172.21.15.114:0/4055666310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc015acb0 msgr2=0x7f8fc00b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 --2- 172.21.15.114:0/4055666310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc015acb0 0x7f8fc00b37d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0009870 tx=0x7f8fb002b980 comp rx=0 tx=0).stop 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 -- 172.21.15.114:0/4055666310 shutdown_connections 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 --2- 172.21.15.114:0/4055666310 >> v2:172.21.15.177:3300/0 conn(0x7f8fc00b4680 0x7f8fc006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 --2- 172.21.15.114:0/4055666310 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc015acb0 0x7f8fc00b37d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 -- 172.21.15.114:0/4055666310 >> 172.21.15.114:0/4055666310 conn(0x7f8fc00b1730 msgr2=0x7f8fc00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 -- 172.21.15.114:0/4055666310 shutdown_connections 2024-03-06T07:38:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.250+0000 7f8fc7600640 1 -- 172.21.15.114:0/4055666310 wait complete. 2024-03-06T07:38:06.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.252+0000 7f8fc7600640 1 Processor -- start 2024-03-06T07:38:06.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.252+0000 7f8fc7600640 1 -- start start 2024-03-06T07:38:06.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc7600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0072340 0x7f8fc0158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc7600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc00b4680 0x7f8fc0146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc7600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fc015acb0 0x7f8fc015d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc7600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00b3d10 con 0x7f8fc0072340 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc7600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00b32b0 con 0x7f8fc015acb0 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc7600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00b4010 con 0x7f8fc00b4680 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0072340 0x7f8fc0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0072340 0x7f8fc0158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47190/0 (socket says 172.21.15.114:47190) 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.253+0000 7f8fc4c00640 1 -- 172.21.15.114:0/1848275230 learned_addr learned my addr 172.21.15.114:0/1848275230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fb7e00640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc00b4680 0x7f8fc0146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fc4c00640 1 -- 172.21.15.114:0/1848275230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc00b4680 msgr2=0x7f8fc0146440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fc4c00640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc00b4680 0x7f8fc0146440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fc4c00640 1 -- 172.21.15.114:0/1848275230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fc015acb0 msgr2=0x7f8fc015d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fc4c00640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fc015acb0 0x7f8fc015d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fc4c00640 1 -- 172.21.15.114:0/1848275230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc0157350 con 0x7f8fc0072340 2024-03-06T07:38:06.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fc4c00640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0072340 0x7f8fc0158110 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00090b0 tx=0x7f8fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fbd600640 1 -- 172.21.15.114:0/1848275230 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0034040 con 0x7f8fc0072340 2024-03-06T07:38:06.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fbd600640 1 -- 172.21.15.114:0/1848275230 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb0030640 con 0x7f8fc0072340 2024-03-06T07:38:06.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc0157630 con 0x7f8fc0072340 2024-03-06T07:38:06.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.254+0000 7f8fbd600640 1 -- 172.21.15.114:0/1848275230 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb00423f0 con 0x7f8fc0072340 2024-03-06T07:38:06.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.255+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc0146980 con 0x7f8fc0072340 2024-03-06T07:38:06.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.256+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc004f330 con 0x7f8fc0072340 2024-03-06T07:38:06.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.258+0000 7f8fbd600640 1 -- 172.21.15.114:0/1848275230 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fb0037070 con 0x7f8fc0072340 2024-03-06T07:38:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.259+0000 7f8fbd600640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f9c06b500 0x7f8f9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.259+0000 7f8fbd600640 1 -- 172.21.15.114:0/1848275230 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8fb00ac540 con 0x7f8fc0072340 2024-03-06T07:38:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.259+0000 7f8fb7e00640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f9c06b500 0x7f8f9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.260+0000 7f8fb7e00640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f9c06b500 0x7f8f9c06d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00cc6f0 tx=0x7f8fa8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.264+0000 7f8fbd600640 1 -- 172.21.15.114:0/1848275230 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fb007b370 con 0x7f8fc0072340 2024-03-06T07:38:06.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.496+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fc006ce60 con 0x7f8fc0072340 2024-03-06T07:38:06.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.497+0000 7f8fbd600640 1 -- 172.21.15.114:0/1848275230 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f8fb007ac70 con 0x7f8fc0072340 2024-03-06T07:38:06.499 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:06.499 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:06.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.499+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f9c06b500 msgr2=0x7f8f9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.499+0000 7f8fc7600640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f9c06b500 0x7f8f9c06d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00cc6f0 tx=0x7f8fa8008040 comp rx=0 tx=0).stop 2024-03-06T07:38:06.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.499+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0072340 msgr2=0x7f8fc0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.499+0000 7f8fc7600640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0072340 0x7f8fc0158110 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00090b0 tx=0x7f8fb00046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 shutdown_connections 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fc015acb0 0x7f8fc015d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8fc00b4680 0x7f8fc0146440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8f9c06b500 0x7f8f9c06d9c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 --2- 172.21.15.114:0/1848275230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc0072340 0x7f8fc0158110 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 >> 172.21.15.114:0/1848275230 conn(0x7f8fc00b1730 msgr2=0x7f8fc006e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 shutdown_connections 2024-03-06T07:38:06.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:06.500+0000 7f8fc7600640 1 -- 172.21.15.114:0/1848275230 wait complete. 2024-03-06T07:38:06.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:06.722 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:07.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:07.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.934+0000 7f19ffe00640 1 Processor -- start 2024-03-06T07:38:07.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.934+0000 7f19ffe00640 1 -- start start 2024-03-06T07:38:07.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.935+0000 7f19ffe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 0x7f1a000d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.935+0000 7f19ffe00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1a000d1e90 0x7f1a000d2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.935+0000 7f19ffe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a000d28a0 0x7f1a000d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.935+0000 7f19ffe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a00161960 con 0x7f1a000d28a0 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.935+0000 7f19ffe00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a000cf6f0 con 0x7f1a000d1e90 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.935+0000 7f19ffe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000cf9d0 con 0x7f1a00161580 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a000d28a0 0x7f1a000d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a000d28a0 0x7f1a000d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42472/0 (socket says 172.21.15.114:42472) 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 -- 172.21.15.114:0/249869707 learned_addr learned my addr 172.21.15.114:0/249869707 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19fea00640 1 --2- 172.21.15.114:0/249869707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 0x7f1a000d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19fe000640 1 --2- 172.21.15.114:0/249869707 >> v2:172.21.15.177:3300/0 conn(0x7f1a000d1e90 0x7f1a000d2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 -- 172.21.15.114:0/249869707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 msgr2=0x7f1a000d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:07.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 --2- 172.21.15.114:0/249869707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 0x7f1a000d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 -- 172.21.15.114:0/249869707 >> v2:172.21.15.177:3300/0 conn(0x7f1a000d1e90 msgr2=0x7f1a000d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:07.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 --2- 172.21.15.114:0/249869707 >> v2:172.21.15.177:3300/0 conn(0x7f1a000d1e90 0x7f1a000d2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.936+0000 7f19ff400640 1 -- 172.21.15.114:0/249869707 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a000cfcb0 con 0x7f1a000d28a0 2024-03-06T07:38:07.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.938+0000 7f19ff400640 1 --2- 172.21.15.114:0/249869707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d28a0 0x7f1a000d06e0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f1a0008c8e0 tx=0x7f19f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a968ec9e296bfaa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:07.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.938+0000 7f19fd600640 1 -- 172.21.15.114:0/249869707 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f802ca70 con 0x7f1a000d28a0 2024-03-06T07:38:07.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.938+0000 7f19fd600640 1 -- 172.21.15.114:0/249869707 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f802cc10 con 0x7f1a000d28a0 2024-03-06T07:38:07.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.938+0000 7f19fd600640 1 -- 172.21.15.114:0/249869707 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f80343f0 con 0x7f1a000d28a0 2024-03-06T07:38:07.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 -- 172.21.15.114:0/249869707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d28a0 msgr2=0x7f1a000d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 --2- 172.21.15.114:0/249869707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d28a0 0x7f1a000d06e0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f1a0008c8e0 tx=0x7f19f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 -- 172.21.15.114:0/249869707 shutdown_connections 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 --2- 172.21.15.114:0/249869707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d28a0 0x7f1a000d06e0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 --2- 172.21.15.114:0/249869707 >> v2:172.21.15.177:3300/0 conn(0x7f1a000d1e90 0x7f1a000d2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 --2- 172.21.15.114:0/249869707 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 0x7f1a000d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 -- 172.21.15.114:0/249869707 >> 172.21.15.114:0/249869707 conn(0x7f1a000d3d80 msgr2=0x7f1a000d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 -- 172.21.15.114:0/249869707 shutdown_connections 2024-03-06T07:38:07.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.939+0000 7f19ffe00640 1 -- 172.21.15.114:0/249869707 wait complete. 2024-03-06T07:38:07.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.941+0000 7f19ffe00640 1 Processor -- start 2024-03-06T07:38:07.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.941+0000 7f19ffe00640 1 -- start start 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19ffe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d1e90 0x7f1a00150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19ffe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a000d28a0 0x7f1a0014e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19ffe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 0x7f1a0014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19ffe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000cf3a0 con 0x7f1a000d1e90 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19ffe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000b39a0 con 0x7f1a000d28a0 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19ffe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000d0c20 con 0x7f1a00161580 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d1e90 0x7f1a00150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d1e90 0x7f1a00150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42488/0 (socket says 172.21.15.114:42488) 2024-03-06T07:38:07.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 -- 172.21.15.114:0/1999053059 learned_addr learned my addr 172.21.15.114:0/1999053059 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:07.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 -- 172.21.15.114:0/1999053059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 msgr2=0x7f1a0014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 0x7f1a0014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 -- 172.21.15.114:0/1999053059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a000d28a0 msgr2=0x7f1a0014e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:07.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a000d28a0 0x7f1a0014e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:07.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.943+0000 7f19fea00640 1 -- 172.21.15.114:0/1999053059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0014f380 con 0x7f1a000d1e90 2024-03-06T07:38:07.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.944+0000 7f19fea00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d1e90 0x7f1a00150420 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f19f000d5f0 tx=0x7f19f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:07.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.944+0000 7f19f6a00640 1 -- 172.21.15.114:0/1999053059 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f00125f0 con 0x7f1a000d1e90 2024-03-06T07:38:07.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.944+0000 7f19f6a00640 1 -- 172.21.15.114:0/1999053059 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f0019070 con 0x7f1a000d1e90 2024-03-06T07:38:07.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.944+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0014f660 con 0x7f1a000d1e90 2024-03-06T07:38:07.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.944+0000 7f19f6a00640 1 -- 172.21.15.114:0/1999053059 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f00129d0 con 0x7f1a000d1e90 2024-03-06T07:38:07.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.944+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0014fc20 con 0x7f1a000d1e90 2024-03-06T07:38:07.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.945+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0004f330 con 0x7f1a000d1e90 2024-03-06T07:38:07.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.946+0000 7f19f6a00640 1 -- 172.21.15.114:0/1999053059 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19f0012b70 con 0x7f1a000d1e90 2024-03-06T07:38:07.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.947+0000 7f19f6a00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f19e006b4d0 0x7f19e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:07.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.947+0000 7f19f6a00640 1 -- 172.21.15.114:0/1999053059 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f19f008a610 con 0x7f1a000d1e90 2024-03-06T07:38:07.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.947+0000 7f19fe000640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f19e006b4d0 0x7f19e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:07.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.948+0000 7f19fe000640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f19e006b4d0 0x7f19e006d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1a000b87f0 tx=0x7f19ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:07.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:07.952+0000 7f19f6a00640 1 -- 172.21.15.114:0/1999053059 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19f0059450 con 0x7f1a000d1e90 2024-03-06T07:38:08.108 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.187+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a000d3b60 con 0x7f1a000d1e90 2024-03-06T07:38:08.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.188+0000 7f19f6a00640 1 -- 172.21.15.114:0/1999053059 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f19f0058d50 con 0x7f1a000d1e90 2024-03-06T07:38:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:08.190 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:08.191 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f19e006b4d0 msgr2=0x7f19e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f19e006b4d0 0x7f19e006d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1a000b87f0 tx=0x7f19ec008040 comp rx=0 tx=0).stop 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d1e90 msgr2=0x7f1a00150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d1e90 0x7f1a00150420 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f19f000d5f0 tx=0x7f19f000dab0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 shutdown_connections 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1a00161580 0x7f1a0014e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a000d28a0 0x7f1a0014e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f19e006b4d0 0x7f19e006d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 --2- 172.21.15.114:0/1999053059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a000d1e90 0x7f1a00150420 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 >> 172.21.15.114:0/1999053059 conn(0x7f1a000d3d80 msgr2=0x7f1a000cfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 shutdown_connections 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:08.191+0000 7f19ffe00640 1 -- 172.21.15.114:0/1999053059 wait complete. 2024-03-06T07:38:08.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:08.419 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:09.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:09.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.634+0000 7f44cae00640 1 Processor -- start 2024-03-06T07:38:09.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.634+0000 7f44cae00640 1 -- start start 2024-03-06T07:38:09.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.636+0000 7f44cae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 0x7f44c40b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.636+0000 7f44cae00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c40cf340 con 0x7f44c415e230 2024-03-06T07:38:09.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.636+0000 7f44cae00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c40cfe60 con 0x7f44c406edb0 2024-03-06T07:38:09.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.636+0000 7f44cae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c415e610 con 0x7f44c40cec60 2024-03-06T07:38:09.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.636+0000 7f44c3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f44c415e230 0x7f44c40d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39000/0 (socket says 172.21.15.114:39000) 2024-03-06T07:38:09.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.636+0000 7f44c3e00640 1 -- 172.21.15.114:0/2390885137 learned_addr learned my addr 172.21.15.114:0/2390885137 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:09.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381277527 0 0) 0x7f44c40cf340 con 0x7f44c415e230 2024-03-06T07:38:09.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a8003520 con 0x7f44c415e230 2024-03-06T07:38:09.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c8e00640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 0x7f44c40b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225754182 0 0) 0x7f44a8003520 con 0x7f44c415e230 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 msgr2=0x7f44c40b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 0x7f44c40b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 >> v1:172.21.15.177:6789/0 conn(0x7f44c406edb0 legacy=0x7f44c40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c406f390 con 0x7f44c415e230 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f44b4001670 con 0x7f44c415e230 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 >> v1:172.21.15.114:6789/0 conn(0x7f44c415e230 legacy=0x7f44c40d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a80039e0 0x7f44a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a8004800 0x7f44a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 0x7f44a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c415e610 con 0x7f44a80039e0 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8005550 con 0x7f44a8004800 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.637+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a800bf00 con 0x7f44c40cec60 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44c3e00640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a80039e0 0x7f44a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44c8e00640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 0x7f44a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44bb400640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a8004800 0x7f44a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44c3e00640 1 -- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 msgr2=0x7f44a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44c3e00640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44c40cec60 0x7f44a800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44c3e00640 1 -- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a8004800 msgr2=0x7f44a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44c3e00640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a8004800 0x7f44a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.638+0000 7f44c3e00640 1 -- 172.21.15.114:0/2390885137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a8003520 con 0x7f44a80039e0 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.639+0000 7f44c3e00640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a80039e0 0x7f44a8003e00 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f44b400c210 tx=0x7f44a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246ddae2bec3d5a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4007d30 con 0x7f44a80039e0 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b4030070 con 0x7f44a80039e0 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44c3400640 1 -- 172.21.15.114:0/2390885137 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b402cd60 con 0x7f44a80039e0 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44cae00640 1 -- 172.21.15.114:0/2390885137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a80039e0 msgr2=0x7f44a8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44cae00640 1 --2- 172.21.15.114:0/2390885137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a80039e0 0x7f44a8003e00 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f44b400c210 tx=0x7f44a8005830 comp rx=0 tx=0).stop 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44c8e00640 1 -- 172.21.15.114:0/2390885137 reap_dead start 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44cae00640 1 -- 172.21.15.114:0/2390885137 shutdown_connections 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.641+0000 7f44cae00640 1 -- 172.21.15.114:0/2390885137 >> 172.21.15.114:0/2390885137 conn(0x7f44c40b3800 msgr2=0x7f44c40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:09.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.642+0000 7f44cae00640 1 -- 172.21.15.114:0/2390885137 shutdown_connections 2024-03-06T07:38:09.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.642+0000 7f44cae00640 1 -- 172.21.15.114:0/2390885137 wait complete. 2024-03-06T07:38:09.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.643+0000 7f44cae00640 1 Processor -- start 2024-03-06T07:38:09.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.643+0000 7f44cae00640 1 -- start start 2024-03-06T07:38:09.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44cae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44a80039e0 0x7f44c40c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44cae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4156be0 0x7f44c41e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44cae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c41e7ca0 0x7f44c40b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44cae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40d06c0 con 0x7f44c4156be0 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44cae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40d03e0 con 0x7f44c41e7ca0 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44cae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40b4320 con 0x7f44a80039e0 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4156be0 0x7f44c41e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44bb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4156be0 0x7f44c41e7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42498/0 (socket says 172.21.15.114:42498) 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.645+0000 7f44bb400640 1 -- 172.21.15.114:0/1638509658 learned_addr learned my addr 172.21.15.114:0/1638509658 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44c3e00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44a80039e0 0x7f44c40c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44bb400640 1 -- 172.21.15.114:0/1638509658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44a80039e0 msgr2=0x7f44c40c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44bb400640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44a80039e0 0x7f44c40c5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44bb400640 1 -- 172.21.15.114:0/1638509658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c41e7ca0 msgr2=0x7f44c40b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44bb400640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c41e7ca0 0x7f44c40b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44bb400640 1 -- 172.21.15.114:0/1638509658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c4144270 con 0x7f44c4156be0 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44bb400640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4156be0 0x7f44c41e7760 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00e550 tx=0x7f44ac00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44c0c00640 1 -- 172.21.15.114:0/1638509658 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0052e0 con 0x7f44c4156be0 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44c0c00640 1 -- 172.21.15.114:0/1638509658 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ac005480 con 0x7f44c4156be0 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44c0c00640 1 -- 172.21.15.114:0/1638509658 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac007020 con 0x7f44c4156be0 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c4144550 con 0x7f44c4156be0 2024-03-06T07:38:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.646+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c4144b10 con 0x7f44c4156be0 2024-03-06T07:38:09.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.648+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c41e80c0 con 0x7f44c4156be0 2024-03-06T07:38:09.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.648+0000 7f44c0c00640 1 -- 172.21.15.114:0/1638509658 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44ac01a020 con 0x7f44c4156be0 2024-03-06T07:38:09.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.649+0000 7f44c0c00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44a006b500 0x7f44a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.649+0000 7f44c0c00640 1 -- 172.21.15.114:0/1638509658 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f44ac089800 con 0x7f44c4156be0 2024-03-06T07:38:09.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.650+0000 7f44c3e00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44a006b500 0x7f44a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.650+0000 7f44c3e00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44a006b500 0x7f44a006d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f44b4034000 tx=0x7f44b4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.658+0000 7f44c0c00640 1 -- 172.21.15.114:0/1638509658 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44ac0585a0 con 0x7f44c4156be0 2024-03-06T07:38:09.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.896+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44c404f330 con 0x7f44c4156be0 2024-03-06T07:38:09.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.897+0000 7f44c0c00640 1 -- 172.21.15.114:0/1638509658 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f44ac057ea0 con 0x7f44c4156be0 2024-03-06T07:38:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:09.899 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:09.899 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:09.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.900+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44a006b500 msgr2=0x7f44a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.900+0000 7f44cae00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44a006b500 0x7f44a006d9c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f44b4034000 tx=0x7f44b4009030 comp rx=0 tx=0).stop 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.900+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4156be0 msgr2=0x7f44c41e7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.900+0000 7f44cae00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4156be0 0x7f44c41e7760 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00e550 tx=0x7f44ac00ea10 comp rx=0 tx=0).stop 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 shutdown_connections 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44c41e7ca0 0x7f44c40b67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f44a006b500 0x7f44a006d9c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4156be0 0x7f44c41e7760 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 --2- 172.21.15.114:0/1638509658 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f44a80039e0 0x7f44c40c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 >> 172.21.15.114:0/1638509658 conn(0x7f44c40b3800 msgr2=0x7f44c406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 shutdown_connections 2024-03-06T07:38:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:09.901+0000 7f44cae00640 1 -- 172.21.15.114:0/1638509658 wait complete. 2024-03-06T07:38:10.110 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:10.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:10.124 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:11.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:11.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.333+0000 7fa80d200640 1 Processor -- start 2024-03-06T07:38:11.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.334+0000 7fa80d200640 1 -- start start 2024-03-06T07:38:11.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa80d200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa80806edb0 0x7fa8080b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa80d200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa8080ce550 0x7fa8080ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa80d200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8080cf340 con 0x7fa80815e230 2024-03-06T07:38:11.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa80d200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80815e610 con 0x7fa80806edb0 2024-03-06T07:38:11.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa80d200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8080ceee0 con 0x7fa8080ce550 2024-03-06T07:38:11.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa805e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa80815e230 0x7fa8080d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39014/0 (socket says 172.21.15.114:39014) 2024-03-06T07:38:11.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa805e00640 1 -- 172.21.15.114:0/2667082570 learned_addr learned my addr 172.21.15.114:0/2667082570 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.335+0000 7fa805400640 1 --2- 172.21.15.114:0/2667082570 >> v2:172.21.15.114:3300/0 conn(0x7fa80806edb0 0x7fa8080b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.336+0000 7fa806800640 1 --2- 172.21.15.114:0/2667082570 >> v2:172.21.15.177:3300/0 conn(0x7fa8080ce550 0x7fa8080ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.336+0000 7fa804a00640 1 -- 172.21.15.114:0/2667082570 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639323930 0 0) 0x7fa8080cf340 con 0x7fa80815e230 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.336+0000 7fa804a00640 1 -- 172.21.15.114:0/2667082570 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ec003520 con 0x7fa80815e230 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.336+0000 7fa805400640 1 -- 172.21.15.114:0/2667082570 >> v2:172.21.15.177:3300/0 conn(0x7fa8080ce550 msgr2=0x7fa8080ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.336+0000 7fa805400640 1 --2- 172.21.15.114:0/2667082570 >> v2:172.21.15.177:3300/0 conn(0x7fa8080ce550 0x7fa8080ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.336+0000 7fa805400640 1 -- 172.21.15.114:0/2667082570 >> v1:172.21.15.114:6789/0 conn(0x7fa80815e230 legacy=0x7fa8080d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.336+0000 7fa805400640 1 -- 172.21.15.114:0/2667082570 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8080cfe60 con 0x7fa80806edb0 2024-03-06T07:38:11.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.337+0000 7fa805400640 1 --2- 172.21.15.114:0/2667082570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa80806edb0 0x7fa8080b4320 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fa808072ca0 tx=0x7fa7f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f594c2369c65d54b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa804a00640 1 -- 172.21.15.114:0/2667082570 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40315a0 con 0x7fa80806edb0 2024-03-06T07:38:11.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa804a00640 1 -- 172.21.15.114:0/2667082570 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f4037070 con 0x7fa80806edb0 2024-03-06T07:38:11.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa804a00640 1 -- 172.21.15.114:0/2667082570 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4031940 con 0x7fa80806edb0 2024-03-06T07:38:11.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa80d200640 1 -- 172.21.15.114:0/2667082570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa80806edb0 msgr2=0x7fa8080b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa80d200640 1 --2- 172.21.15.114:0/2667082570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa80806edb0 0x7fa8080b4320 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fa808072ca0 tx=0x7fa7f402b840 comp rx=0 tx=0).stop 2024-03-06T07:38:11.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa80d200640 1 -- 172.21.15.114:0/2667082570 shutdown_connections 2024-03-06T07:38:11.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa80d200640 1 --2- 172.21.15.114:0/2667082570 >> v2:172.21.15.177:3300/0 conn(0x7fa8080ce550 0x7fa8080ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa80d200640 1 --2- 172.21.15.114:0/2667082570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa80806edb0 0x7fa8080b4320 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.338+0000 7fa80d200640 1 -- 172.21.15.114:0/2667082570 >> 172.21.15.114:0/2667082570 conn(0x7fa8080b3800 msgr2=0x7fa8080b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:11.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.339+0000 7fa80d200640 1 -- 172.21.15.114:0/2667082570 shutdown_connections 2024-03-06T07:38:11.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.339+0000 7fa80d200640 1 -- 172.21.15.114:0/2667082570 wait complete. 2024-03-06T07:38:11.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.340+0000 7fa80d200640 1 Processor -- start 2024-03-06T07:38:11.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.340+0000 7fa80d200640 1 -- start start 2024-03-06T07:38:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.341+0000 7fa80d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa80806edb0 0x7fa8080b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.341+0000 7fa80d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8080ce550 0x7fa8081f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.341+0000 7fa80d200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa80815e230 0x7fa8081f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.341+0000 7fa80d200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080d03e0 con 0x7fa8080ce550 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.341+0000 7fa80d200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080d0100 con 0x7fa80806edb0 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.341+0000 7fa80d200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080d06c0 con 0x7fa80815e230 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8080ce550 0x7fa8081f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8080ce550 0x7fa8081f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42510/0 (socket says 172.21.15.114:42510) 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 -- 172.21.15.114:0/149366163 learned_addr learned my addr 172.21.15.114:0/149366163 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805e00640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa80806edb0 0x7fa8080b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa806800640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa80815e230 0x7fa8081f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 -- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa80815e230 msgr2=0x7fa8081f77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa80815e230 0x7fa8081f77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 -- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa80806edb0 msgr2=0x7fa8080b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa80806edb0 0x7fa8080b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 -- 172.21.15.114:0/149366163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8080b52b0 con 0x7fa8080ce550 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.342+0000 7fa805400640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8080ce550 0x7fa8081f7270 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4000c00 tx=0x7fa7f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.343+0000 7fa7fe000640 1 -- 172.21.15.114:0/149366163 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4032630 con 0x7fa8080ce550 2024-03-06T07:38:11.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.343+0000 7fa7fe000640 1 -- 172.21.15.114:0/149366163 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f40327d0 con 0x7fa8080ce550 2024-03-06T07:38:11.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.343+0000 7fa7fe000640 1 -- 172.21.15.114:0/149366163 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4032b80 con 0x7fa8080ce550 2024-03-06T07:38:11.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.343+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8080b5590 con 0x7fa8080ce550 2024-03-06T07:38:11.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.343+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8080b5b50 con 0x7fa8080ce550 2024-03-06T07:38:11.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.344+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80804f330 con 0x7fa8080ce550 2024-03-06T07:38:11.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.345+0000 7fa7fe000640 1 -- 172.21.15.114:0/149366163 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7f404c020 con 0x7fa8080ce550 2024-03-06T07:38:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.346+0000 7fa7fe000640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa7e806b4d0 0x7fa7e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.346+0000 7fa7fe000640 1 -- 172.21.15.114:0/149366163 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa7f402f020 con 0x7fa8080ce550 2024-03-06T07:38:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.346+0000 7fa805e00640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa7e806b4d0 0x7fa7e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.347+0000 7fa805e00640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa7e806b4d0 0x7fa7e806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8000f60 tx=0x7fa7f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.351+0000 7fa7fe000640 1 -- 172.21.15.114:0/149366163 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7f4040070 con 0x7fa8080ce550 2024-03-06T07:38:11.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.699+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8080d1740 con 0x7fa8080ce550 2024-03-06T07:38:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.699+0000 7fa7fe000640 1 -- 172.21.15.114:0/149366163 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7fa7f407ac50 con 0x7fa8080ce550 2024-03-06T07:38:11.701 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:11.701 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:11.702 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:11.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa7e806b4d0 msgr2=0x7fa7e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa7e806b4d0 0x7fa7e806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8000f60 tx=0x7fa7f8009040 comp rx=0 tx=0).stop 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8080ce550 msgr2=0x7fa8081f7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8080ce550 0x7fa8081f7270 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4000c00 tx=0x7fa7f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 shutdown_connections 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa80815e230 0x7fa8081f77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa7e806b4d0 0x7fa7e806d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8080ce550 0x7fa8081f7270 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 --2- 172.21.15.114:0/149366163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa80806edb0 0x7fa8080b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.702+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 >> 172.21.15.114:0/149366163 conn(0x7fa8080b3800 msgr2=0x7fa8080b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.703+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 shutdown_connections 2024-03-06T07:38:11.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:11.703+0000 7fa80d200640 1 -- 172.21.15.114:0/149366163 wait complete. 2024-03-06T07:38:11.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:11.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:12.112 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:12.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.141+0000 7f4cdf400640 1 Processor -- start 2024-03-06T07:38:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.142+0000 7f4cdf400640 1 -- start start 2024-03-06T07:38:13.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.144+0000 7f4cdf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4cd815e160 0x7f4cd8162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.144+0000 7f4cdf400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4cd80b3ac0 0x7f4cd806b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.144+0000 7f4cdf400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd8162b60 con 0x7f4cd80af820 2024-03-06T07:38:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.144+0000 7f4cdf400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd806b0f0 con 0x7f4cd815e160 2024-03-06T07:38:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.144+0000 7f4cdf400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd80b5620 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cd7e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4cd80af820 0x7f4cd80b33b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39026/0 (socket says 172.21.15.114:39026) 2024-03-06T07:38:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cd7e00640 1 -- 172.21.15.114:0/1469188511 learned_addr learned my addr 172.21.15.114:0/1469188511 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cdca00640 1 --2- 172.21.15.114:0/1469188511 >> v2:172.21.15.114:3300/0 conn(0x7f4cd815e160 0x7f4cd8162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cdd400640 1 --2- 172.21.15.114:0/1469188511 >> v2:172.21.15.177:3300/0 conn(0x7f4cd80b3ac0 0x7f4cd806b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cd7400640 1 -- 172.21.15.114:0/1469188511 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383670323 0 0) 0x7f4cc4001110 con 0x7f4cd80af820 2024-03-06T07:38:13.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cd7400640 1 -- 172.21.15.114:0/1469188511 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc0003520 con 0x7f4cd80af820 2024-03-06T07:38:13.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cdca00640 1 -- 172.21.15.114:0/1469188511 >> v2:172.21.15.177:3300/0 conn(0x7f4cd80b3ac0 msgr2=0x7f4cd806b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cdca00640 1 --2- 172.21.15.114:0/1469188511 >> v2:172.21.15.177:3300/0 conn(0x7f4cd80b3ac0 0x7f4cd806b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cdca00640 1 -- 172.21.15.114:0/1469188511 >> v1:172.21.15.114:6789/0 conn(0x7f4cd80af820 legacy=0x7f4cd80b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.145+0000 7f4cdca00640 1 -- 172.21.15.114:0/1469188511 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd806f1f0 con 0x7f4cd815e160 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.146+0000 7f4cdca00640 1 --2- 172.21.15.114:0/1469188511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd815e160 0x7f4cd8162620 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f4cd806e460 tx=0x7f4ccc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d6baaf8f9b26f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cd7400640 1 -- 172.21.15.114:0/1469188511 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc0315a0 con 0x7f4cd815e160 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cd7400640 1 -- 172.21.15.114:0/1469188511 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc037070 con 0x7f4cd815e160 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cd7400640 1 -- 172.21.15.114:0/1469188511 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc031940 con 0x7f4cd815e160 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cdf400640 1 -- 172.21.15.114:0/1469188511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd815e160 msgr2=0x7f4cd8162620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1469188511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd815e160 0x7f4cd8162620 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f4cd806e460 tx=0x7f4ccc02b840 comp rx=0 tx=0).stop 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cdf400640 1 -- 172.21.15.114:0/1469188511 shutdown_connections 2024-03-06T07:38:13.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1469188511 >> v2:172.21.15.177:3300/0 conn(0x7f4cd80b3ac0 0x7f4cd806b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1469188511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd815e160 0x7f4cd8162620 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.147+0000 7f4cdf400640 1 -- 172.21.15.114:0/1469188511 >> 172.21.15.114:0/1469188511 conn(0x7f4cd80d0340 msgr2=0x7f4cd80d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.148+0000 7f4cdf400640 1 -- 172.21.15.114:0/1469188511 shutdown_connections 2024-03-06T07:38:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.148+0000 7f4cdf400640 1 -- 172.21.15.114:0/1469188511 wait complete. 2024-03-06T07:38:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.149+0000 7f4cdf400640 1 Processor -- start 2024-03-06T07:38:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.149+0000 7f4cdf400640 1 -- start start 2024-03-06T07:38:13.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.150+0000 7f4cdf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd80af820 0x7f4cd8154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.150+0000 7f4cdf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd80b3ac0 0x7f4cd80cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.150+0000 7f4cdf400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4cd815e160 0x7f4cd8151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.150+0000 7f4cdf400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd806ef10 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.150+0000 7f4cdf400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd806ec30 con 0x7f4cd80af820 2024-03-06T07:38:13.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.150+0000 7f4cdf400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd806bac0 con 0x7f4cd815e160 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd80b3ac0 0x7f4cd80cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd80b3ac0 0x7f4cd80cee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42520/0 (socket says 172.21.15.114:42520) 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 -- 172.21.15.114:0/1901202260 learned_addr learned my addr 172.21.15.114:0/1901202260 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cdd400640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4cd815e160 0x7f4cd8151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 -- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4cd815e160 msgr2=0x7f4cd8151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4cd815e160 0x7f4cd8151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cdca00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd80af820 0x7f4cd8154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 -- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd80af820 msgr2=0x7f4cd8154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd80af820 0x7f4cd8154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cd7e00640 1 -- 172.21.15.114:0/1901202260 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8153ef0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.151+0000 7f4cdca00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd80af820 0x7f4cd8154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.152+0000 7f4cd7e00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd80b3ac0 0x7f4cd80cee90 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c730 tx=0x7f4cc400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.152+0000 7f4cd4c00640 1 -- 172.21.15.114:0/1901202260 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4017070 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.152+0000 7f4cd4c00640 1 -- 172.21.15.114:0/1901202260 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc40058b0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.152+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd80cfac0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.152+0000 7f4cd4c00640 1 -- 172.21.15.114:0/1901202260 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4005ce0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.152+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd8154170 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.153+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd8074ae0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.154+0000 7f4cd4c00640 1 -- 172.21.15.114:0/1901202260 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cc401a070 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.155+0000 7f4cd4c00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4cb806b4d0 0x7f4cb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.156+0000 7f4cd4c00640 1 -- 172.21.15.114:0/1901202260 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4cc40897e0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.156+0000 7f4cdca00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4cb806b4d0 0x7f4cb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.157+0000 7f4cdca00640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4cb806b4d0 0x7f4cb806d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc036000 tx=0x7f4ccc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.160+0000 7f4cd4c00640 1 -- 172.21.15.114:0/1901202260 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cd8153ef0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.402+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd80d11e0 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.403+0000 7f4cd4c00640 1 -- 172.21.15.114:0/1901202260 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+3471 (secure 0 0 0) 0x7f4cd8162b60 con 0x7f4cd80b3ac0 2024-03-06T07:38:13.405 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:13.405 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:36:44.929793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:13.405 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 104 2024-03-06T07:38:13.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4cb806b4d0 msgr2=0x7f4cb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4cb806b4d0 0x7f4cb806d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc036000 tx=0x7f4ccc008030 comp rx=0 tx=0).stop 2024-03-06T07:38:13.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd80b3ac0 msgr2=0x7f4cd80cee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd80b3ac0 0x7f4cd80cee90 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c730 tx=0x7f4cc400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 shutdown_connections 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4cd815e160 0x7f4cd8151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4cb806b4d0 0x7f4cb806d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cd80b3ac0 0x7f4cd80cee90 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 --2- 172.21.15.114:0/1901202260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4cd80af820 0x7f4cd8154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 >> 172.21.15.114:0/1901202260 conn(0x7f4cd80d0340 msgr2=0x7f4cd815d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 shutdown_connections 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:13.405+0000 7f4cdf400640 1 -- 172.21.15.114:0/1901202260 wait complete. 2024-03-06T07:38:13.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:36:44.929793+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:13.635 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:14.114 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:14.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.842+0000 7f8ba7e00640 1 Processor -- start 2024-03-06T07:38:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.842+0000 7f8ba7e00640 1 -- start start 2024-03-06T07:38:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ba0144c00 0x7f8ba006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba7e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8ba015f9d0 0x7f8ba015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba00ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba7e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba006ee70 con 0x7f8ba0144c00 2024-03-06T07:38:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba7e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba006cef0 con 0x7f8ba015f9d0 2024-03-06T07:38:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba006d1d0 con 0x7f8ba00ce0a0 2024-03-06T07:38:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ba0144c00 0x7f8ba006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ba0144c00 0x7f8ba006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42532/0 (socket says 172.21.15.114:42532) 2024-03-06T07:38:14.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.844+0000 7f8ba5400640 1 -- 172.21.15.114:0/710126016 learned_addr learned my addr 172.21.15.114:0/710126016 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.845+0000 7f8ba4a00640 1 --2- 172.21.15.114:0/710126016 >> v2:172.21.15.177:3300/0 conn(0x7f8ba015f9d0 0x7f8ba015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.845+0000 7f8ba5e00640 1 --2- 172.21.15.114:0/710126016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba00ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.845+0000 7f8ba5400640 1 -- 172.21.15.114:0/710126016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 msgr2=0x7f8ba00ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.845+0000 7f8ba5400640 1 --2- 172.21.15.114:0/710126016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba00ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.845+0000 7f8ba5400640 1 -- 172.21.15.114:0/710126016 >> v2:172.21.15.177:3300/0 conn(0x7f8ba015f9d0 msgr2=0x7f8ba015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.845+0000 7f8ba5400640 1 --2- 172.21.15.114:0/710126016 >> v2:172.21.15.177:3300/0 conn(0x7f8ba015f9d0 0x7f8ba015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.845+0000 7f8ba5400640 1 -- 172.21.15.114:0/710126016 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba006d4b0 con 0x7f8ba0144c00 2024-03-06T07:38:14.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.846+0000 7f8ba5400640 1 --2- 172.21.15.114:0/710126016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba006e930 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00b26f0 tx=0x7f8b9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71b0a98a440d8c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.846+0000 7f8b9fe00640 1 -- 172.21.15.114:0/710126016 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94035020 con 0x7f8ba0144c00 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.846+0000 7f8b9fe00640 1 -- 172.21.15.114:0/710126016 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b94031870 con 0x7f8ba0144c00 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.846+0000 7f8b9fe00640 1 -- 172.21.15.114:0/710126016 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94031c40 con 0x7f8ba0144c00 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 -- 172.21.15.114:0/710126016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 msgr2=0x7f8ba006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/710126016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba006e930 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00b26f0 tx=0x7f8b9402b910 comp rx=0 tx=0).stop 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 -- 172.21.15.114:0/710126016 shutdown_connections 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/710126016 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba00ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/710126016 >> v2:172.21.15.177:3300/0 conn(0x7f8ba015f9d0 0x7f8ba015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/710126016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba006e930 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 -- 172.21.15.114:0/710126016 >> 172.21.15.114:0/710126016 conn(0x7f8ba00b1730 msgr2=0x7f8ba00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 -- 172.21.15.114:0/710126016 shutdown_connections 2024-03-06T07:38:14.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.847+0000 7f8ba7e00640 1 -- 172.21.15.114:0/710126016 wait complete. 2024-03-06T07:38:14.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.849+0000 7f8ba7e00640 1 Processor -- start 2024-03-06T07:38:14.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.849+0000 7f8ba7e00640 1 -- start start 2024-03-06T07:38:14.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.850+0000 7f8ba7e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba015a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.850+0000 7f8ba7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba015aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.850+0000 7f8ba7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba015f9d0 0x7f8ba0138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.850+0000 7f8ba7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba006cb50 con 0x7f8ba0144c00 2024-03-06T07:38:14.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.850+0000 7f8ba7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba0160730 con 0x7f8ba015f9d0 2024-03-06T07:38:14.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.850+0000 7f8ba7e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba006ccd0 con 0x7f8ba00ce0a0 2024-03-06T07:38:14.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.850+0000 7f8ba4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba015aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba015aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42548/0 (socket says 172.21.15.114:42548) 2024-03-06T07:38:14.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 -- 172.21.15.114:0/1304692264 learned_addr learned my addr 172.21.15.114:0/1304692264 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:14.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 -- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 msgr2=0x7f8ba015a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba5400640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba015a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba015a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 -- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba015f9d0 msgr2=0x7f8ba0138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:14.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba5e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba015f9d0 0x7f8ba0138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba015f9d0 0x7f8ba0138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 -- 172.21.15.114:0/1304692264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba0139f00 con 0x7f8ba0144c00 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba5e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba015f9d0 0x7f8ba0138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.851+0000 7f8ba4a00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba015aaf0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00d830 tx=0x7f8b8c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.852+0000 7f8b9d600640 1 -- 172.21.15.114:0/1304692264 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00b6c0 con 0x7f8ba0144c00 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.852+0000 7f8b9d600640 1 -- 172.21.15.114:0/1304692264 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c019070 con 0x7f8ba0144c00 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.852+0000 7f8b9d600640 1 -- 172.21.15.114:0/1304692264 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00ba80 con 0x7f8ba0144c00 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.852+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba00b46b0 con 0x7f8ba0144c00 2024-03-06T07:38:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.852+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba013a180 con 0x7f8ba0144c00 2024-03-06T07:38:14.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.853+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba004f330 con 0x7f8ba0144c00 2024-03-06T07:38:14.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.854+0000 7f8b9d600640 1 -- 172.21.15.114:0/1304692264 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b8c01e460 con 0x7f8ba0144c00 2024-03-06T07:38:14.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.855+0000 7f8b9d600640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b8006b4d0 0x7f8b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:14.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.855+0000 7f8b9d600640 1 -- 172.21.15.114:0/1304692264 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8b8c08a990 con 0x7f8ba0144c00 2024-03-06T07:38:14.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.856+0000 7f8ba5400640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b8006b4d0 0x7f8b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:14.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.856+0000 7f8ba5400640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b8006b4d0 0x7f8b8006d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8b94036000 tx=0x7f8b94008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:14.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:14.859+0000 7f8b9d600640 1 -- 172.21.15.114:0/1304692264 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ba0139f00 con 0x7f8ba0144c00 2024-03-06T07:38:15.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.095+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ba0074be0 con 0x7f8ba0144c00 2024-03-06T07:38:15.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.095+0000 7f8b9d600640 1 -- 172.21.15.114:0/1304692264 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f8b8c059730 con 0x7f8ba0144c00 2024-03-06T07:38:15.097 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:15.097 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:15.098 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b8006b4d0 msgr2=0x7f8b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b8006b4d0 0x7f8b8006d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8b94036000 tx=0x7f8b94008030 comp rx=0 tx=0).stop 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 msgr2=0x7f8ba015aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba015aaf0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00d830 tx=0x7f8b8c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 shutdown_connections 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba015f9d0 0x7f8ba0138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8b8006b4d0 0x7f8b8006d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ba0144c00 0x7f8ba015aaf0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 --2- 172.21.15.114:0/1304692264 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8ba00ce0a0 0x7f8ba015a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 >> 172.21.15.114:0/1304692264 conn(0x7f8ba00b1730 msgr2=0x7f8ba0070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:15.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 shutdown_connections 2024-03-06T07:38:15.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:15.098+0000 7f8ba7e00640 1 -- 172.21.15.114:0/1304692264 wait complete. 2024-03-06T07:38:15.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:15.320 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:16.116 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:16.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:16.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.533+0000 7f0d17200640 1 Processor -- start 2024-03-06T07:38:16.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.533+0000 7f0d17200640 1 -- start start 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d17200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d180b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d17200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0d180ae670 0x7f0d1815e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d17200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d18161900 con 0x7f0d181620c0 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d17200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d180b49e0 con 0x7f0d180ae670 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d17200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1815ebd0 con 0x7f0d180b40c0 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d15400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0d180ae670 0x7f0d1815e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d16800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0d181620c0 0x7f0d180b28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39030/0 (socket says 172.21.15.114:39030) 2024-03-06T07:38:16.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d16800640 1 -- 172.21.15.114:0/1605797720 learned_addr learned my addr 172.21.15.114:0/1605797720 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.535+0000 7f0d15e00640 1 --2- 172.21.15.114:0/1605797720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d180b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.536+0000 7f0d14a00640 1 -- 172.21.15.114:0/1605797720 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70053109 0 0) 0x7f0d18161900 con 0x7f0d181620c0 2024-03-06T07:38:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.536+0000 7f0d14a00640 1 -- 172.21.15.114:0/1605797720 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cfc003520 con 0x7f0d181620c0 2024-03-06T07:38:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.536+0000 7f0d15400640 1 -- 172.21.15.114:0/1605797720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 msgr2=0x7f0d180b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.536+0000 7f0d15400640 1 --2- 172.21.15.114:0/1605797720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d180b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.536+0000 7f0d15400640 1 -- 172.21.15.114:0/1605797720 >> v1:172.21.15.114:6789/0 conn(0x7f0d181620c0 legacy=0x7f0d180b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.536+0000 7f0d15400640 1 -- 172.21.15.114:0/1605797720 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1815eeb0 con 0x7f0d180ae670 2024-03-06T07:38:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.537+0000 7f0d15400640 1 --2- 172.21.15.114:0/1605797720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d180ae670 0x7f0d1815e2c0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f0d040097a0 tx=0x7f0d0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea018cec6fa807f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.537+0000 7f0d14a00640 1 -- 172.21.15.114:0/1605797720 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04035020 con 0x7f0d180ae670 2024-03-06T07:38:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.537+0000 7f0d14a00640 1 -- 172.21.15.114:0/1605797720 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d04031830 con 0x7f0d180ae670 2024-03-06T07:38:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.537+0000 7f0d14a00640 1 -- 172.21.15.114:0/1605797720 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04031c00 con 0x7f0d180ae670 2024-03-06T07:38:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 -- 172.21.15.114:0/1605797720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d180ae670 msgr2=0x7f0d1815e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 --2- 172.21.15.114:0/1605797720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d180ae670 0x7f0d1815e2c0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f0d040097a0 tx=0x7f0d0402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 -- 172.21.15.114:0/1605797720 shutdown_connections 2024-03-06T07:38:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 --2- 172.21.15.114:0/1605797720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d180ae670 0x7f0d1815e2c0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 --2- 172.21.15.114:0/1605797720 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d180b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 -- 172.21.15.114:0/1605797720 >> 172.21.15.114:0/1605797720 conn(0x7f0d1806afc0 msgr2=0x7f0d1806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 -- 172.21.15.114:0/1605797720 shutdown_connections 2024-03-06T07:38:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.538+0000 7f0d17200640 1 -- 172.21.15.114:0/1605797720 wait complete. 2024-03-06T07:38:16.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.540+0000 7f0d17200640 1 Processor -- start 2024-03-06T07:38:16.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.540+0000 7f0d17200640 1 -- start start 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.541+0000 7f0d17200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d180ae670 0x7f0d1814b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.541+0000 7f0d17200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d18156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.541+0000 7f0d17200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d181620c0 0x7f0d181594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.541+0000 7f0d17200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1815e800 con 0x7f0d181620c0 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.541+0000 7f0d17200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d18161ba0 con 0x7f0d180ae670 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.541+0000 7f0d17200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1815e980 con 0x7f0d180b40c0 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.541+0000 7f0d16800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d181620c0 0x7f0d181594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d181620c0 0x7f0d181594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42560/0 (socket says 172.21.15.114:42560) 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 -- 172.21.15.114:0/3582901900 learned_addr learned my addr 172.21.15.114:0/3582901900 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d15e00640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d180ae670 0x7f0d1814b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d15400640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d18156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 -- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 msgr2=0x7f0d18156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d18156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 -- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d180ae670 msgr2=0x7f0d1814b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d180ae670 0x7f0d1814b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 -- 172.21.15.114:0/3582901900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d18070a30 con 0x7f0d181620c0 2024-03-06T07:38:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d15400640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d18156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d16800640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d181620c0 0x7f0d181594f0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000cb40 tx=0x7f0d1000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.542+0000 7f0d0e000640 1 -- 172.21.15.114:0/3582901900 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10017070 con 0x7f0d181620c0 2024-03-06T07:38:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.543+0000 7f0d0e000640 1 -- 172.21.15.114:0/3582901900 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1000aca0 con 0x7f0d181620c0 2024-03-06T07:38:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.543+0000 7f0d0e000640 1 -- 172.21.15.114:0/3582901900 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10012620 con 0x7f0d181620c0 2024-03-06T07:38:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.543+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d181578b0 con 0x7f0d181620c0 2024-03-06T07:38:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.543+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d18070cf0 con 0x7f0d181620c0 2024-03-06T07:38:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.544+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1804f330 con 0x7f0d181620c0 2024-03-06T07:38:16.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.545+0000 7f0d0e000640 1 -- 172.21.15.114:0/3582901900 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d1001a070 con 0x7f0d181620c0 2024-03-06T07:38:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.545+0000 7f0d0e000640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.546+0000 7f0d0e000640 1 -- 172.21.15.114:0/3582901900 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0d1008a110 con 0x7f0d181620c0 2024-03-06T07:38:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.546+0000 7f0d15e00640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.546+0000 7f0d15e00640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cf806b4d0 0x7f0cf806d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0d08006fd0 tx=0x7f0d08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.551+0000 7f0d0e000640 1 -- 172.21.15.114:0/3582901900 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d10058eb0 con 0x7f0d181620c0 2024-03-06T07:38:16.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.829+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d18070ec0 con 0x7f0d181620c0 2024-03-06T07:38:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.830+0000 7f0d0e000640 1 -- 172.21.15.114:0/3582901900 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0d100587b0 con 0x7f0d181620c0 2024-03-06T07:38:16.832 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:16.832 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:16.832 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.832+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cf806b4d0 msgr2=0x7f0cf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.832+0000 7f0d17200640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cf806b4d0 0x7f0cf806d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0d08006fd0 tx=0x7f0d08008040 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d181620c0 msgr2=0x7f0d181594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d181620c0 0x7f0d181594f0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000cb40 tx=0x7f0d1000a060 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 shutdown_connections 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d181620c0 0x7f0d181594f0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f0d180b40c0 0x7f0d18156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 --2- 172.21.15.114:0/3582901900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d180ae670 0x7f0d1814b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 >> 172.21.15.114:0/3582901900 conn(0x7f0d1806afc0 msgr2=0x7f0d1806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 shutdown_connections 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:16.833+0000 7f0d17200640 1 -- 172.21.15.114:0/3582901900 wait complete. 2024-03-06T07:38:17.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:17.054 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:17.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:17.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:18.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:18.118 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.270+0000 7fb579e00640 1 Processor -- start 2024-03-06T07:38:18.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.270+0000 7fb579e00640 1 -- start start 2024-03-06T07:38:18.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.271+0000 7fb579e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.271+0000 7fb579e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb5740b4680 0x7fb57406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.271+0000 7fb579e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57415b090 con 0x7fb574072340 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.271+0000 7fb579e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5740b2a30 con 0x7fb5740b4680 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.271+0000 7fb579e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740b4360 con 0x7fb57415acb0 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.272+0000 7fb573400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb574072340 0x7fb5740ccfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56766/0 (socket says 172.21.15.114:56766) 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.272+0000 7fb573400640 1 -- 172.21.15.114:0/100942675 learned_addr learned my addr 172.21.15.114:0/100942675 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.272+0000 7fb572a00640 1 --2- 172.21.15.114:0/100942675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.272+0000 7fb572000640 1 --2- 172.21.15.114:0/100942675 >> v2:172.21.15.177:3300/0 conn(0x7fb5740b4680 0x7fb57406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.272+0000 7fb571600640 1 -- 172.21.15.114:0/100942675 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719979303 0 0) 0x7fb57415b090 con 0x7fb574072340 2024-03-06T07:38:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.272+0000 7fb571600640 1 -- 172.21.15.114:0/100942675 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb558003520 con 0x7fb574072340 2024-03-06T07:38:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.273+0000 7fb572a00640 1 -- 172.21.15.114:0/100942675 >> v2:172.21.15.177:3300/0 conn(0x7fb5740b4680 msgr2=0x7fb57406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.273+0000 7fb572a00640 1 --2- 172.21.15.114:0/100942675 >> v2:172.21.15.177:3300/0 conn(0x7fb5740b4680 0x7fb57406e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.273+0000 7fb572a00640 1 -- 172.21.15.114:0/100942675 >> v1:172.21.15.114:6789/0 conn(0x7fb574072340 legacy=0x7fb5740ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.273+0000 7fb572a00640 1 -- 172.21.15.114:0/100942675 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5740b2cf0 con 0x7fb57415acb0 2024-03-06T07:38:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.274+0000 7fb572a00640 1 --2- 172.21.15.114:0/100942675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740b37d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb5640097a0 tx=0x7fb56402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7e9f044df958cc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.274+0000 7fb571600640 1 -- 172.21.15.114:0/100942675 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564035020 con 0x7fb57415acb0 2024-03-06T07:38:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.274+0000 7fb571600640 1 -- 172.21.15.114:0/100942675 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb564031830 con 0x7fb57415acb0 2024-03-06T07:38:18.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.274+0000 7fb571600640 1 -- 172.21.15.114:0/100942675 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564031be0 con 0x7fb57415acb0 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 -- 172.21.15.114:0/100942675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 msgr2=0x7fb5740b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 --2- 172.21.15.114:0/100942675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740b37d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb5640097a0 tx=0x7fb56402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 -- 172.21.15.114:0/100942675 shutdown_connections 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 --2- 172.21.15.114:0/100942675 >> v2:172.21.15.177:3300/0 conn(0x7fb5740b4680 0x7fb57406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 --2- 172.21.15.114:0/100942675 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740b37d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 -- 172.21.15.114:0/100942675 >> 172.21.15.114:0/100942675 conn(0x7fb5740b1730 msgr2=0x7fb5740b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 -- 172.21.15.114:0/100942675 shutdown_connections 2024-03-06T07:38:18.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.275+0000 7fb579e00640 1 -- 172.21.15.114:0/100942675 wait complete. 2024-03-06T07:38:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.277+0000 7fb579e00640 1 Processor -- start 2024-03-06T07:38:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.277+0000 7fb579e00640 1 -- start start 2024-03-06T07:38:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb579e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb574072340 0x7fb57414eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb579e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740b4680 0x7fb574150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb579e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb579e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740b3d10 con 0x7fb5740b4680 2024-03-06T07:38:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb579e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740b32b0 con 0x7fb574072340 2024-03-06T07:38:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb579e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740b4010 con 0x7fb57415acb0 2024-03-06T07:38:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb572000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740b4680 0x7fb574150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb572000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740b4680 0x7fb574150340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47216/0 (socket says 172.21.15.114:47216) 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.278+0000 7fb572000640 1 -- 172.21.15.114:0/3529892620 learned_addr learned my addr 172.21.15.114:0/3529892620 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb573400640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb572000640 1 -- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 msgr2=0x7fb5740caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb572a00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb574072340 0x7fb57414eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb572000640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb572000640 1 -- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb574072340 msgr2=0x7fb57414eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb572000640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb574072340 0x7fb57414eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb572000640 1 -- 172.21.15.114:0/3529892620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57414f400 con 0x7fb5740b4680 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb573400640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb572000640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740b4680 0x7fb574150340 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00efc0 tx=0x7fb55c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb56b400640 1 -- 172.21.15.114:0/3529892620 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c018020 con 0x7fb5740b4680 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb56b400640 1 -- 172.21.15.114:0/3529892620 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c014480 con 0x7fb5740b4680 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.279+0000 7fb56b400640 1 -- 172.21.15.114:0/3529892620 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c014850 con 0x7fb5740b4680 2024-03-06T07:38:18.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.280+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57414fc00 con 0x7fb5740b4680 2024-03-06T07:38:18.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.280+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5741f71d0 con 0x7fb5740b4680 2024-03-06T07:38:18.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.281+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57404f330 con 0x7fb5740b4680 2024-03-06T07:38:18.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.282+0000 7fb56b400640 1 -- 172.21.15.114:0/3529892620 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb55c005d10 con 0x7fb5740b4680 2024-03-06T07:38:18.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.283+0000 7fb56b400640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.283+0000 7fb56b400640 1 -- 172.21.15.114:0/3529892620 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb55c08ad40 con 0x7fb5740b4680 2024-03-06T07:38:18.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.283+0000 7fb572a00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.284+0000 7fb572a00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb564036000 tx=0x7fb564008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.287+0000 7fb56b400640 1 -- 172.21.15.114:0/3529892620 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb55c022070 con 0x7fb5740b4680 2024-03-06T07:38:18.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.520+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb57406ce60 con 0x7fb5740b4680 2024-03-06T07:38:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.521+0000 7fb56b400640 1 -- 172.21.15.114:0/3529892620 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fb55c010040 con 0x7fb5740b4680 2024-03-06T07:38:18.523 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:18.523 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:18.523 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.523+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 msgr2=0x7fb55006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.523+0000 7fb579e00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb564036000 tx=0x7fb564008030 comp rx=0 tx=0).stop 2024-03-06T07:38:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.523+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740b4680 msgr2=0x7fb574150340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.523+0000 7fb579e00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740b4680 0x7fb574150340 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00efc0 tx=0x7fb55c00c490 comp rx=0 tx=0).stop 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.523+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 shutdown_connections 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.523+0000 7fb579e00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb57415acb0 0x7fb5740caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.524+0000 7fb579e00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb55006b4d0 0x7fb55006d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.524+0000 7fb579e00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5740b4680 0x7fb574150340 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.524+0000 7fb579e00640 1 --2- 172.21.15.114:0/3529892620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb574072340 0x7fb57414eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.524+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 >> 172.21.15.114:0/3529892620 conn(0x7fb5740b1730 msgr2=0x7fb5740cdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.524+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 shutdown_connections 2024-03-06T07:38:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:18.524+0000 7fb579e00640 1 -- 172.21.15.114:0/3529892620 wait complete. 2024-03-06T07:38:18.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:18.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:19.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:19.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.952+0000 7f5d82000640 1 Processor -- start 2024-03-06T07:38:19.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.952+0000 7f5d82000640 1 -- start start 2024-03-06T07:38:19.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.953+0000 7f5d82000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5d7c144c00 0x7f5d7c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d82000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15fe60 0x7f5d7c0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d82000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06ee70 con 0x7f5d7c15f9d0 2024-03-06T07:38:19.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d82000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c06cef0 con 0x7f5d7c144c00 2024-03-06T07:38:19.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d82000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c06d1d0 con 0x7f5d7c15fe60 2024-03-06T07:38:19.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d73e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5d7c15f9d0 0x7f5d7c0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56770/0 (socket says 172.21.15.114:56770) 2024-03-06T07:38:19.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d73e00640 1 -- 172.21.15.114:0/3264528164 learned_addr learned my addr 172.21.15.114:0/3264528164 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:19.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d7b400640 1 --2- 172.21.15.114:0/3264528164 >> v2:172.21.15.114:3300/0 conn(0x7f5d7c144c00 0x7f5d7c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.954+0000 7f5d7be00640 1 --2- 172.21.15.114:0/3264528164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15fe60 0x7f5d7c0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.955+0000 7f5d7b400640 1 -- 172.21.15.114:0/3264528164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15fe60 msgr2=0x7f5d7c0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:19.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.955+0000 7f5d7b400640 1 --2- 172.21.15.114:0/3264528164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15fe60 0x7f5d7c0ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.955+0000 7f5d7b400640 1 -- 172.21.15.114:0/3264528164 >> v1:172.21.15.114:6789/0 conn(0x7f5d7c15f9d0 legacy=0x7f5d7c0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.955+0000 7f5d7b400640 1 -- 172.21.15.114:0/3264528164 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c06d4b0 con 0x7f5d7c144c00 2024-03-06T07:38:19.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.956+0000 7f5d7b400640 1 --2- 172.21.15.114:0/3264528164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c144c00 0x7f5d7c06e930 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0097a0 tx=0x7f5d6c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a749624ce8868a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:19.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d7aa00640 1 -- 172.21.15.114:0/3264528164 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c035020 con 0x7f5d7c144c00 2024-03-06T07:38:19.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d7aa00640 1 -- 172.21.15.114:0/3264528164 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c031810 con 0x7f5d7c144c00 2024-03-06T07:38:19.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d7aa00640 1 -- 172.21.15.114:0/3264528164 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c031be0 con 0x7f5d7c144c00 2024-03-06T07:38:19.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 -- 172.21.15.114:0/3264528164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c144c00 msgr2=0x7f5d7c06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 --2- 172.21.15.114:0/3264528164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c144c00 0x7f5d7c06e930 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0097a0 tx=0x7f5d6c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 -- 172.21.15.114:0/3264528164 shutdown_connections 2024-03-06T07:38:19.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 --2- 172.21.15.114:0/3264528164 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15fe60 0x7f5d7c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 --2- 172.21.15.114:0/3264528164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c144c00 0x7f5d7c06e930 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 -- 172.21.15.114:0/3264528164 >> 172.21.15.114:0/3264528164 conn(0x7f5d7c0b1730 msgr2=0x7f5d7c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:19.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 -- 172.21.15.114:0/3264528164 shutdown_connections 2024-03-06T07:38:19.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.957+0000 7f5d82000640 1 -- 172.21.15.114:0/3264528164 wait complete. 2024-03-06T07:38:19.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.959+0000 7f5d82000640 1 Processor -- start 2024-03-06T07:38:19.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.959+0000 7f5d82000640 1 -- start start 2024-03-06T07:38:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.960+0000 7f5d82000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d7c144c00 0x7f5d7c13df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.960+0000 7f5d82000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15f9d0 0x7f5d7c1394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.960+0000 7f5d82000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c15fe60 0x7f5d7c13d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.960+0000 7f5d82000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c160730 con 0x7f5d7c15fe60 2024-03-06T07:38:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.960+0000 7f5d82000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c160290 con 0x7f5d7c144c00 2024-03-06T07:38:19.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.960+0000 7f5d82000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c06cb50 con 0x7f5d7c15f9d0 2024-03-06T07:38:19.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c15fe60 0x7f5d7c13d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c15fe60 0x7f5d7c13d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47242/0 (socket says 172.21.15.114:47242) 2024-03-06T07:38:19.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 -- 172.21.15.114:0/552830911 learned_addr learned my addr 172.21.15.114:0/552830911 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d73e00640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15f9d0 0x7f5d7c1394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 -- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15f9d0 msgr2=0x7f5d7c1394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15f9d0 0x7f5d7c1394e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 -- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d7c144c00 msgr2=0x7f5d7c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7b400640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d7c144c00 0x7f5d7c13df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d7c144c00 0x7f5d7c13df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 -- 172.21.15.114:0/552830911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c13dcd0 con 0x7f5d7c15fe60 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.961+0000 7f5d7be00640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c15fe60 0x7f5d7c13d790 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f5d74007e70 tx=0x7f5d7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:19.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.962+0000 7f5d73400640 1 -- 172.21.15.114:0/552830911 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74014620 con 0x7f5d7c15fe60 2024-03-06T07:38:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.962+0000 7f5d73400640 1 -- 172.21.15.114:0/552830911 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d74019070 con 0x7f5d7c15fe60 2024-03-06T07:38:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.962+0000 7f5d73400640 1 -- 172.21.15.114:0/552830911 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74014a20 con 0x7f5d7c15fe60 2024-03-06T07:38:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.962+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c13e550 con 0x7f5d7c15fe60 2024-03-06T07:38:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.962+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c13eb10 con 0x7f5d7c15fe60 2024-03-06T07:38:19.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.963+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c04f330 con 0x7f5d7c15fe60 2024-03-06T07:38:19.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.965+0000 7f5d73400640 1 -- 172.21.15.114:0/552830911 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d74014bc0 con 0x7f5d7c15fe60 2024-03-06T07:38:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.966+0000 7f5d73400640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5806b4d0 0x7f5d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.966+0000 7f5d73400640 1 -- 172.21.15.114:0/552830911 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5d7408ab40 con 0x7f5d7c15fe60 2024-03-06T07:38:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.966+0000 7f5d7b400640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5806b4d0 0x7f5d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:19.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.967+0000 7f5d7b400640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5806b4d0 0x7f5d5806d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c036000 tx=0x7f5d6c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:19.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:19.971+0000 7f5d73400640 1 -- 172.21.15.114:0/552830911 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d74059980 con 0x7f5d7c15fe60 2024-03-06T07:38:20.120 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:20.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.209+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d7c072890 con 0x7f5d7c15fe60 2024-03-06T07:38:20.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.210+0000 7f5d73400640 1 -- 172.21.15.114:0/552830911 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f5d74059280 con 0x7f5d7c15fe60 2024-03-06T07:38:20.212 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:20.212 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:20.212 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:20.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5806b4d0 msgr2=0x7f5d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5806b4d0 0x7f5d5806d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c036000 tx=0x7f5d6c008030 comp rx=0 tx=0).stop 2024-03-06T07:38:20.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c15fe60 msgr2=0x7f5d7c13d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c15fe60 0x7f5d7c13d790 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f5d74007e70 tx=0x7f5d7400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 shutdown_connections 2024-03-06T07:38:20.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5806b4d0 0x7f5d5806d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c15fe60 0x7f5d7c13d790 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c15f9d0 0x7f5d7c1394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 --2- 172.21.15.114:0/552830911 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d7c144c00 0x7f5d7c13df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 >> 172.21.15.114:0/552830911 conn(0x7f5d7c0b1730 msgr2=0x7f5d7c145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:20.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.213+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 shutdown_connections 2024-03-06T07:38:20.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:20.214+0000 7f5d82000640 1 -- 172.21.15.114:0/552830911 wait complete. 2024-03-06T07:38:20.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:20.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:21.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.648+0000 7f7976000640 1 Processor -- start 2024-03-06T07:38:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.648+0000 7f7976000640 1 -- start start 2024-03-06T07:38:21.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.649+0000 7f7976000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 0x7f79700b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.649+0000 7f7976000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7970161960 con 0x7f7970161580 2024-03-06T07:38:21.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.649+0000 7f7976000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79700cf6f0 con 0x7f79700d2360 2024-03-06T07:38:21.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.649+0000 7f7976000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79700cf9d0 con 0x7f79700d27a0 2024-03-06T07:38:21.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.649+0000 7f7974c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7970161580 0x7f79700d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56778/0 (socket says 172.21.15.114:56778) 2024-03-06T07:38:21.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.649+0000 7f7974c00640 1 -- 172.21.15.114:0/210571691 learned_addr learned my addr 172.21.15.114:0/210571691 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f7975600640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 0x7f79700b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280463476 0 0) 0x7f7970161960 con 0x7f7970161580 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7954003520 con 0x7f7970161580 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549695727 0 0) 0x7f7954003520 con 0x7f7970161580 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 msgr2=0x7f79700b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 0x7f79700b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 >> v1:172.21.15.177:6789/0 conn(0x7f79700d2360 legacy=0x7f79700b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79700cfcb0 con 0x7f7970161580 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7960001670 con 0x7f7970161580 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 >> v1:172.21.15.114:6789/0 conn(0x7f7970161580 legacy=0x7f79700d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79540039e0 0x7f7954003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 0x7f7954004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7954004d40 0x7f79540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79700cf9d0 con 0x7f79540039e0 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795400ba70 con 0x7f7954004d40 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.650+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795400bc70 con 0x7f79700d27a0 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7974c00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79540039e0 0x7f7954003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7967e00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 0x7f7954004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7975600640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7954004d40 0x7f79540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7974c00640 1 -- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 msgr2=0x7f7954004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7974c00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79700d27a0 0x7f7954004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7974c00640 1 -- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7954004d40 msgr2=0x7f79540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7974c00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7954004d40 0x7f79540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7974c00640 1 -- 172.21.15.114:0/210571691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7954003520 con 0x7f79540039e0 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.651+0000 7f7975600640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7954004d40 0x7f79540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:21.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.652+0000 7f7974c00640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79540039e0 0x7f7954003e00 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f7960001eb0 tx=0x7f7954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52842d13fd479f05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960007d30 con 0x7f79540039e0 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7960030070 con 0x7f79540039e0 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f796fe00640 1 -- 172.21.15.114:0/210571691 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960005d30 con 0x7f79540039e0 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f7976000640 1 -- 172.21.15.114:0/210571691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79540039e0 msgr2=0x7f7954003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f7976000640 1 --2- 172.21.15.114:0/210571691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79540039e0 0x7f7954003e00 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f7960001eb0 tx=0x7f7954005830 comp rx=0 tx=0).stop 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f7975600640 1 -- 172.21.15.114:0/210571691 reap_dead start 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f7976000640 1 -- 172.21.15.114:0/210571691 shutdown_connections 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f7976000640 1 -- 172.21.15.114:0/210571691 >> 172.21.15.114:0/210571691 conn(0x7f79700d3d80 msgr2=0x7f79700d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f7976000640 1 -- 172.21.15.114:0/210571691 shutdown_connections 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.653+0000 7f7976000640 1 -- 172.21.15.114:0/210571691 wait complete. 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.655+0000 7f7976000640 1 Processor -- start 2024-03-06T07:38:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.655+0000 7f7976000640 1 -- start start 2024-03-06T07:38:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.656+0000 7f7976000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79540039e0 0x7f79700c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.656+0000 7f7976000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79700c8100 0x7f79700cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.656+0000 7f7976000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797014f160 0x7f797015fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.656+0000 7f7976000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79700d2c30 con 0x7f797014f160 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.656+0000 7f7976000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79700b39a0 con 0x7f79700c8100 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.656+0000 7f7976000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79700cf3a0 con 0x7f79540039e0 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.656+0000 7f7975600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797014f160 0x7f797015fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797014f160 0x7f797015fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47268/0 (socket says 172.21.15.114:47268) 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 -- 172.21.15.114:0/3581149442 learned_addr learned my addr 172.21.15.114:0/3581149442 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7974c00640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79540039e0 0x7f79700c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7967e00640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79700c8100 0x7f79700cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 -- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79540039e0 msgr2=0x7f79700c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79540039e0 0x7f79700c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 -- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79700c8100 msgr2=0x7f79700cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79700c8100 0x7f79700cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 -- 172.21.15.114:0/3581149442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79700cb390 con 0x7f797014f160 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7974c00640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79540039e0 0x7f79700c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f7975600640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797014f160 0x7f797015fbf0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f797008cb60 tx=0x7f796800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.657+0000 7f796d600640 1 -- 172.21.15.114:0/3581149442 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796801c070 con 0x7f797014f160 2024-03-06T07:38:21.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.658+0000 7f796d600640 1 -- 172.21.15.114:0/3581149442 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7968017400 con 0x7f797014f160 2024-03-06T07:38:21.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.658+0000 7f796d600640 1 -- 172.21.15.114:0/3581149442 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79680177f0 con 0x7f797014f160 2024-03-06T07:38:21.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.658+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79700c8d40 con 0x7f797014f160 2024-03-06T07:38:21.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.658+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79700c7900 con 0x7f797014f160 2024-03-06T07:38:21.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.659+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79700c7410 con 0x7f797014f160 2024-03-06T07:38:21.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.660+0000 7f796d600640 1 -- 172.21.15.114:0/3581149442 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7968005d10 con 0x7f797014f160 2024-03-06T07:38:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.661+0000 7f796d600640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f794c06b500 0x7f794c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.661+0000 7f796d600640 1 -- 172.21.15.114:0/3581149442 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f796808a310 con 0x7f797014f160 2024-03-06T07:38:21.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.661+0000 7f7974c00640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f794c06b500 0x7f794c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.662+0000 7f7974c00640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f794c06b500 0x7f794c06d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7960034000 tx=0x7f7960009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.665+0000 7f796d600640 1 -- 172.21.15.114:0/3581149442 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7968012070 con 0x7f797014f160 2024-03-06T07:38:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.902+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f797004f330 con 0x7f797014f160 2024-03-06T07:38:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.903+0000 7f796d600640 1 -- 172.21.15.114:0/3581149442 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f79680590b0 con 0x7f797014f160 2024-03-06T07:38:21.905 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:21.905 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:21.905 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f794c06b500 msgr2=0x7f794c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f794c06b500 0x7f794c06d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7960034000 tx=0x7f7960009030 comp rx=0 tx=0).stop 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797014f160 msgr2=0x7f797015fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797014f160 0x7f797015fbf0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f797008cb60 tx=0x7f796800f4a0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 shutdown_connections 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f794c06b500 0x7f794c06d9c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f797014f160 0x7f797015fbf0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f79700c8100 0x7f79700cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 --2- 172.21.15.114:0/3581149442 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f79540039e0 0x7f79700c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 >> 172.21.15.114:0/3581149442 conn(0x7f79700d3d80 msgr2=0x7f79700d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 shutdown_connections 2024-03-06T07:38:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:21.906+0000 7f7976000640 1 -- 172.21.15.114:0/3581149442 wait complete. 2024-03-06T07:38:22.122 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:22.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:22.127 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:23.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:23.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.381+0000 7fd806000640 1 Processor -- start 2024-03-06T07:38:23.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.381+0000 7fd806000640 1 -- start start 2024-03-06T07:38:23.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.382+0000 7fd806000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 0x7fd80006d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.382+0000 7fd806000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd8000cc2a0 0x7fd8000cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.382+0000 7fd806000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8000b2b30 con 0x7fd80015a940 2024-03-06T07:38:23.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.382+0000 7fd806000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8000b28e0 con 0x7fd8000cc2a0 2024-03-06T07:38:23.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.382+0000 7fd806000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000b2e10 con 0x7fd8000cf370 2024-03-06T07:38:23.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.383+0000 7fd804c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd80015a940 0x7fd8000b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:56780/0 (socket says 172.21.15.114:56780) 2024-03-06T07:38:23.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.383+0000 7fd804c00640 1 -- 172.21.15.114:0/2751166077 learned_addr learned my addr 172.21.15.114:0/2751166077 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.383+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754861834 0 0) 0x7fd8000b2b30 con 0x7fd80015a940 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.383+0000 7fd805600640 1 --2- 172.21.15.114:0/2751166077 >> v2:172.21.15.177:3300/0 conn(0x7fd8000cc2a0 0x7fd8000cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.383+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e4005730 con 0x7fd80015a940 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.383+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 0x7fd80006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.383+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1452318833 0 0) 0x7fd7e4005730 con 0x7fd80015a940 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 msgr2=0x7fd80006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 0x7fd80006d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 >> v2:172.21.15.177:3300/0 conn(0x7fd8000cc2a0 msgr2=0x7fd8000cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 --2- 172.21.15.114:0/2751166077 >> v2:172.21.15.177:3300/0 conn(0x7fd8000cc2a0 0x7fd8000cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80006ca50 con 0x7fd80015a940 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fd7f0001670 con 0x7fd80015a940 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd805600640 1 --2- 172.21.15.114:0/2751166077 >> v2:172.21.15.177:3300/0 conn(0x7fd8000cc2a0 0x7fd8000cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 >> v1:172.21.15.114:6789/0 conn(0x7fd80015a940 legacy=0x7fd8000b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7e4005bc0 0x7fd7e4005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7e4006a10 0x7fd7e4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 0x7fd7e40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000b28e0 con 0x7fd7e4006a10 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000b2e10 con 0x7fd7e4005bc0 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e400dcd0 con 0x7fd8000cf370 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7e4006a10 0x7fd7e4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd804c00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7e4005bc0 0x7fd7e4005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.384+0000 7fd805600640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 0x7fd7e40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.385+0000 7fd7f7e00640 1 -- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 msgr2=0x7fd7e40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.385+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd8000cf370 0x7fd7e40073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.385+0000 7fd7f7e00640 1 -- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7e4005bc0 msgr2=0x7fd7e4005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.385+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd7e4005bc0 0x7fd7e4005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.385+0000 7fd7f7e00640 1 -- 172.21.15.114:0/2751166077 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e4005730 con 0x7fd7e4006a10 2024-03-06T07:38:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.386+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7e4006a10 0x7fd7e4006e90 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800ba00 tx=0x7fd7e4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bea22b0c1a4c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.386+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8036020 con 0x7fd7e4006a10 2024-03-06T07:38:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.386+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e80325f0 con 0x7fd7e4006a10 2024-03-06T07:38:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.386+0000 7fd7ffe00640 1 -- 172.21.15.114:0/2751166077 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e800b600 con 0x7fd7e4006a10 2024-03-06T07:38:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.387+0000 7fd806000640 1 -- 172.21.15.114:0/2751166077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7e4006a10 msgr2=0x7fd7e4006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.387+0000 7fd806000640 1 --2- 172.21.15.114:0/2751166077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7e4006a10 0x7fd7e4006e90 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800ba00 tx=0x7fd7e4007a40 comp rx=0 tx=0).stop 2024-03-06T07:38:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.387+0000 7fd805600640 1 -- 172.21.15.114:0/2751166077 reap_dead start 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.387+0000 7fd806000640 1 -- 172.21.15.114:0/2751166077 shutdown_connections 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.387+0000 7fd806000640 1 -- 172.21.15.114:0/2751166077 >> 172.21.15.114:0/2751166077 conn(0x7fd80006afc0 msgr2=0x7fd80006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.387+0000 7fd806000640 1 -- 172.21.15.114:0/2751166077 shutdown_connections 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.387+0000 7fd806000640 1 -- 172.21.15.114:0/2751166077 wait complete. 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.389+0000 7fd806000640 1 Processor -- start 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.389+0000 7fd806000640 1 -- start start 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd806000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7e4006a10 0x7fd80014e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd806000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd800149ce0 0x7fd8000afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd806000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd80014fd70 0x7fd8000ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd806000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80015ad20 con 0x7fd800149ce0 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd806000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000b30f0 con 0x7fd80014fd70 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd806000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80015aea0 con 0x7fd7e4006a10 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd7f7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd800149ce0 0x7fd8000afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd7f7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd800149ce0 0x7fd8000afd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47280/0 (socket says 172.21.15.114:47280) 2024-03-06T07:38:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.390+0000 7fd7f7e00640 1 -- 172.21.15.114:0/3103449065 learned_addr learned my addr 172.21.15.114:0/3103449065 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7f7e00640 1 -- 172.21.15.114:0/3103449065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7e4006a10 msgr2=0x7fd80014e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7e4006a10 0x7fd80014e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7f7e00640 1 -- 172.21.15.114:0/3103449065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd80014fd70 msgr2=0x7fd8000ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd80014fd70 0x7fd8000ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7f7e00640 1 -- 172.21.15.114:0/3103449065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80014ecd0 con 0x7fd800149ce0 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7f7e00640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd800149ce0 0x7fd8000afd70 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800bb20 tx=0x7fd7e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7fd600640 1 -- 172.21.15.114:0/3103449065 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8041070 con 0x7fd800149ce0 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7fd600640 1 -- 172.21.15.114:0/3103449065 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8032e50 con 0x7fd800149ce0 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd7fd600640 1 -- 172.21.15.114:0/3103449065 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8036040 con 0x7fd800149ce0 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.391+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80014efb0 con 0x7fd800149ce0 2024-03-06T07:38:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.392+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80014f850 con 0x7fd800149ce0 2024-03-06T07:38:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.393+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80014f180 con 0x7fd800149ce0 2024-03-06T07:38:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.394+0000 7fd7fd600640 1 -- 172.21.15.114:0/3103449065 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7e8039020 con 0x7fd800149ce0 2024-03-06T07:38:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.395+0000 7fd7fd600640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7dc06b4d0 0x7fd7dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.395+0000 7fd7fd600640 1 -- 172.21.15.114:0/3103449065 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7e80ab9d0 con 0x7fd800149ce0 2024-03-06T07:38:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.395+0000 7fd804c00640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7dc06b4d0 0x7fd7dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.396+0000 7fd804c00640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7dc06b4d0 0x7fd7dc06d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00058d0 tx=0x7fd7f0000f50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.400+0000 7fd7fd600640 1 -- 172.21.15.114:0/3103449065 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7e802d040 con 0x7fd800149ce0 2024-03-06T07:38:23.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.635+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd80006b3a0 con 0x7fd800149ce0 2024-03-06T07:38:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.636+0000 7fd7fd600640 1 -- 172.21.15.114:0/3103449065 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd80006b3a0 con 0x7fd800149ce0 2024-03-06T07:38:23.638 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:23.638 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.638+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7dc06b4d0 msgr2=0x7fd7dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7dc06b4d0 0x7fd7dc06d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00058d0 tx=0x7fd7f0000f50 comp rx=0 tx=0).stop 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd800149ce0 msgr2=0x7fd8000afd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd800149ce0 0x7fd8000afd70 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800bb20 tx=0x7fd7e8004730 comp rx=0 tx=0).stop 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 shutdown_connections 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd80014fd70 0x7fd8000ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd7dc06b4d0 0x7fd7dc06d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd800149ce0 0x7fd8000afd70 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 --2- 172.21.15.114:0/3103449065 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd7e4006a10 0x7fd80014e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 >> 172.21.15.114:0/3103449065 conn(0x7fd80006afc0 msgr2=0x7fd800154690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 shutdown_connections 2024-03-06T07:38:23.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:23.639+0000 7fd806000640 1 -- 172.21.15.114:0/3103449065 wait complete. 2024-03-06T07:38:23.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:23.861 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:24.124 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:24.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.063+0000 7f2c4be00640 1 Processor -- start 2024-03-06T07:38:25.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.063+0000 7f2c4be00640 1 -- start start 2024-03-06T07:38:25.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.065+0000 7f2c4be00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2c4415eff0 0x7f2c44070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.065+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c4406e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.065+0000 7f2c4be00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c44071d30 con 0x7f2c440af1c0 2024-03-06T07:38:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.065+0000 7f2c4be00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c44071260 con 0x7f2c4415eff0 2024-03-06T07:38:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.065+0000 7f2c4be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4415f4a0 con 0x7f2c4406e4d0 2024-03-06T07:38:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.065+0000 7f2c49e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c4406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c49e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c4406e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:40372/0 (socket says 172.21.15.114:40372) 2024-03-06T07:38:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c49e00640 1 -- 172.21.15.114:0/623853539 learned_addr learned my addr 172.21.15.114:0/623853539 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c49400640 1 --1- 172.21.15.114:0/623853539 >> v1:172.21.15.177:6789/0 conn(0x7f2c440af1c0 0x7f2c4406cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:32996/0 (socket says 172.21.15.114:32996) 2024-03-06T07:38:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c48a00640 1 --2- 172.21.15.114:0/623853539 >> v2:172.21.15.177:3300/0 conn(0x7f2c4415eff0 0x7f2c44070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c49e00640 1 -- 172.21.15.114:0/623853539 >> v2:172.21.15.177:3300/0 conn(0x7f2c4415eff0 msgr2=0x7f2c44070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c49e00640 1 --2- 172.21.15.114:0/623853539 >> v2:172.21.15.177:3300/0 conn(0x7f2c4415eff0 0x7f2c44070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c49e00640 1 -- 172.21.15.114:0/623853539 >> v1:172.21.15.177:6789/0 conn(0x7f2c440af1c0 legacy=0x7f2c4406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.066+0000 7f2c49e00640 1 -- 172.21.15.114:0/623853539 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c44071490 con 0x7f2c4406e4d0 2024-03-06T07:38:25.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.068+0000 7f2c49e00640 1 --2- 172.21.15.114:0/623853539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c4406e960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009e80 tx=0x7f2c3c02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6970c16ead9529d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.068+0000 7f2c43e00640 1 -- 172.21.15.114:0/623853539 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c02caf0 con 0x7f2c4406e4d0 2024-03-06T07:38:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.068+0000 7f2c43e00640 1 -- 172.21.15.114:0/623853539 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c02cc90 con 0x7f2c4406e4d0 2024-03-06T07:38:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.068+0000 7f2c43e00640 1 -- 172.21.15.114:0/623853539 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c034610 con 0x7f2c4406e4d0 2024-03-06T07:38:25.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 -- 172.21.15.114:0/623853539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 msgr2=0x7f2c4406e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 --2- 172.21.15.114:0/623853539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c4406e960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009e80 tx=0x7f2c3c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:38:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 -- 172.21.15.114:0/623853539 shutdown_connections 2024-03-06T07:38:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 --2- 172.21.15.114:0/623853539 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c4406e960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 --2- 172.21.15.114:0/623853539 >> v2:172.21.15.177:3300/0 conn(0x7f2c4415eff0 0x7f2c44070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 -- 172.21.15.114:0/623853539 >> 172.21.15.114:0/623853539 conn(0x7f2c440b1730 msgr2=0x7f2c440b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 -- 172.21.15.114:0/623853539 shutdown_connections 2024-03-06T07:38:25.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.069+0000 7f2c4be00640 1 -- 172.21.15.114:0/623853539 wait complete. 2024-03-06T07:38:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.071+0000 7f2c4be00640 1 Processor -- start 2024-03-06T07:38:25.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.071+0000 7f2c4be00640 1 -- start start 2024-03-06T07:38:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c440ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c440af1c0 0x7f2c441e7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c4415eff0 0x7f2c441f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c4be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c440b30b0 con 0x7f2c4415eff0 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c4be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c44071a50 con 0x7f2c440af1c0 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c4be00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4406d370 con 0x7f2c4406e4d0 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c49e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c4415eff0 0x7f2c441f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c49e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c4415eff0 0x7f2c441f6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47294/0 (socket says 172.21.15.114:47294) 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.072+0000 7f2c49e00640 1 -- 172.21.15.114:0/3876153860 learned_addr learned my addr 172.21.15.114:0/3876153860 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c49400640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c440ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c48a00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c440af1c0 0x7f2c441e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c49e00640 1 -- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 msgr2=0x7f2c440ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c49e00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c440ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c49e00640 1 -- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c440af1c0 msgr2=0x7f2c441e7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c49e00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c440af1c0 0x7f2c441e7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c49e00640 1 -- 172.21.15.114:0/3876153860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c440ce8d0 con 0x7f2c4415eff0 2024-03-06T07:38:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c49e00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c4415eff0 0x7f2c441f6970 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c036040 tx=0x7f2c3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.073+0000 7f2c41600640 1 -- 172.21.15.114:0/3876153860 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c043020 con 0x7f2c4415eff0 2024-03-06T07:38:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.074+0000 7f2c41600640 1 -- 172.21.15.114:0/3876153860 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c03e450 con 0x7f2c4415eff0 2024-03-06T07:38:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.074+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c440cebb0 con 0x7f2c4415eff0 2024-03-06T07:38:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.074+0000 7f2c41600640 1 -- 172.21.15.114:0/3876153860 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c009510 con 0x7f2c4415eff0 2024-03-06T07:38:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.074+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c440cf170 con 0x7f2c4415eff0 2024-03-06T07:38:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.075+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4404f330 con 0x7f2c4415eff0 2024-03-06T07:38:25.077 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.076+0000 7f2c41600640 1 -- 172.21.15.114:0/3876153860 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c3c046020 con 0x7f2c4415eff0 2024-03-06T07:38:25.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.077+0000 7f2c41600640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c2406b500 0x7f2c2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.077+0000 7f2c41600640 1 -- 172.21.15.114:0/3876153860 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2c3c0aca00 con 0x7f2c4415eff0 2024-03-06T07:38:25.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.077+0000 7f2c49400640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c2406b500 0x7f2c2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.078+0000 7f2c49400640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c2406b500 0x7f2c2406d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2c440b12c0 tx=0x7f2c38007ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.082+0000 7f2c41600640 1 -- 172.21.15.114:0/3876153860 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c3c04d030 con 0x7f2c4415eff0 2024-03-06T07:38:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.321+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c44074be0 con 0x7f2c4415eff0 2024-03-06T07:38:25.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.322+0000 7f2c41600640 1 -- 172.21.15.114:0/3876153860 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2c3c07b7a0 con 0x7f2c4415eff0 2024-03-06T07:38:25.324 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:25.324 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:25.324 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c2406b500 msgr2=0x7f2c2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c2406b500 0x7f2c2406d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2c440b12c0 tx=0x7f2c38007ce0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c4415eff0 msgr2=0x7f2c441f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c4415eff0 0x7f2c441f6970 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c036040 tx=0x7f2c3c004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 shutdown_connections 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2c2406b500 0x7f2c2406d9c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c4415eff0 0x7f2c441f6970 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2c440af1c0 0x7f2c441e7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.324+0000 7f2c4be00640 1 --2- 172.21.15.114:0/3876153860 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2c4406e4d0 0x7f2c440ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.325+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 >> 172.21.15.114:0/3876153860 conn(0x7f2c440b1730 msgr2=0x7f2c440ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.325+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 shutdown_connections 2024-03-06T07:38:25.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:25.325+0000 7f2c4be00640 1 -- 172.21.15.114:0/3876153860 wait complete. 2024-03-06T07:38:25.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:25.546 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:26.127 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:26.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:26.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.766+0000 7f7f0b200640 1 Processor -- start 2024-03-06T07:38:26.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.766+0000 7f7f0b200640 1 -- start start 2024-03-06T07:38:26.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.767+0000 7f7f0b200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7f0c1561f0 0x7f7f0c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.767+0000 7f7f0b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c0aebf0 0x7f7f0c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.767+0000 7f7f0b200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0c15c240 con 0x7f7f0c0cc800 2024-03-06T07:38:26.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.767+0000 7f7f0b200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f0c15ce30 con 0x7f7f0c1561f0 2024-03-06T07:38:26.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.767+0000 7f7f0b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c15c500 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.768+0000 7f7f01400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c0aebf0 0x7f7f0c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.768+0000 7f7f09e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7f0c1561f0 0x7f7f0c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.768+0000 7f7f0a800640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f7f0c0cc800 0x7f7f0c0cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:32998/0 (socket says 172.21.15.114:32998) 2024-03-06T07:38:26.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.768+0000 7f7f0a800640 1 -- 172.21.15.114:0/3259817802 learned_addr learned my addr 172.21.15.114:0/3259817802 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:26.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.769+0000 7f7f09400640 1 -- 172.21.15.114:0/3259817802 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481583276 0 0) 0x7f7f0c15c240 con 0x7f7f0c0cc800 2024-03-06T07:38:26.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.769+0000 7f7f01400640 1 -- 172.21.15.114:0/3259817802 >> v2:172.21.15.177:3300/0 conn(0x7f7f0c1561f0 msgr2=0x7f7f0c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:26.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.769+0000 7f7f01400640 1 --2- 172.21.15.114:0/3259817802 >> v2:172.21.15.177:3300/0 conn(0x7f7f0c1561f0 0x7f7f0c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.769+0000 7f7f01400640 1 -- 172.21.15.114:0/3259817802 >> v1:172.21.15.177:6789/0 conn(0x7f7f0c0cc800 legacy=0x7f7f0c0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.769+0000 7f7f01400640 1 -- 172.21.15.114:0/3259817802 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c15c7e0 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.769+0000 7f7f09e00640 1 --2- 172.21.15.114:0/3259817802 >> v2:172.21.15.177:3300/0 conn(0x7f7f0c1561f0 0x7f7f0c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:38:26.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.770+0000 7f7f01400640 1 --2- 172.21.15.114:0/3259817802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c0aebf0 0x7f7f0c0cc2c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40097a0 tx=0x7f7ef402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44b1f6236c665c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:26.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f09400640 1 -- 172.21.15.114:0/3259817802 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4035020 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f09400640 1 -- 172.21.15.114:0/3259817802 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef4031810 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f09400640 1 -- 172.21.15.114:0/3259817802 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4031be0 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f0b200640 1 -- 172.21.15.114:0/3259817802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c0aebf0 msgr2=0x7f7f0c0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f0b200640 1 --2- 172.21.15.114:0/3259817802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c0aebf0 0x7f7f0c0cc2c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40097a0 tx=0x7f7ef402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f0b200640 1 -- 172.21.15.114:0/3259817802 shutdown_connections 2024-03-06T07:38:26.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f0b200640 1 --2- 172.21.15.114:0/3259817802 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c0aebf0 0x7f7f0c0cc2c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f0b200640 1 --2- 172.21.15.114:0/3259817802 >> v2:172.21.15.177:3300/0 conn(0x7f7f0c1561f0 0x7f7f0c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.771+0000 7f7f0b200640 1 -- 172.21.15.114:0/3259817802 >> 172.21.15.114:0/3259817802 conn(0x7f7f0c06afc0 msgr2=0x7f7f0c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.772+0000 7f7f0b200640 1 -- 172.21.15.114:0/3259817802 shutdown_connections 2024-03-06T07:38:26.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.772+0000 7f7f0b200640 1 -- 172.21.15.114:0/3259817802 wait complete. 2024-03-06T07:38:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.773+0000 7f7f0b200640 1 Processor -- start 2024-03-06T07:38:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.773+0000 7f7f0b200640 1 -- start start 2024-03-06T07:38:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.774+0000 7f7f0b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f0c0aebf0 0x7f7f0c1555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.774+0000 7f7f0b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0c0cc800 0x7f7f0c1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.774+0000 7f7f0b200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c1561f0 0x7f7f0c156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.774+0000 7f7f0b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c1565d0 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.774+0000 7f7f0b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c0ccc90 con 0x7f7f0c0cc800 2024-03-06T07:38:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.774+0000 7f7f0b200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c156750 con 0x7f7f0c1561f0 2024-03-06T07:38:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f0c0aebf0 0x7f7f0c1555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f0c0aebf0 0x7f7f0c1555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47300/0 (socket says 172.21.15.114:47300) 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 -- 172.21.15.114:0/1095326536 learned_addr learned my addr 172.21.15.114:0/1095326536 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f0a800640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c1561f0 0x7f7f0c156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f01400640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0c0cc800 0x7f7f0c1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 -- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c1561f0 msgr2=0x7f7f0c156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c1561f0 0x7f7f0c156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 -- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0c0cc800 msgr2=0x7f7f0c1568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0c0cc800 0x7f7f0c1568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f09e00640 1 -- 172.21.15.114:0/1095326536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c157440 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.775+0000 7f7f01400640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0c0cc800 0x7f7f0c1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.776+0000 7f7f09e00640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f0c0aebf0 0x7f7f0c1555a0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00e960 tx=0x7f7efc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:26.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.776+0000 7f7f02a00640 1 -- 172.21.15.114:0/1095326536 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc017070 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.776+0000 7f7f02a00640 1 -- 172.21.15.114:0/1095326536 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc00ca00 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.776+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0c157720 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.776+0000 7f7f02a00640 1 -- 172.21.15.114:0/1095326536 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc00cdd0 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.776+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0c1efd90 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.777+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c1ef8a0 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.779+0000 7f7f02a00640 1 -- 172.21.15.114:0/1095326536 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7efc01a070 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.780+0000 7f7f02a00640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee806b4d0 0x7f7ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:26.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.780+0000 7f7f02a00640 1 -- 172.21.15.114:0/1095326536 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7efc08a550 con 0x7f7f0c0aebf0 2024-03-06T07:38:26.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.780+0000 7f7f01400640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee806b4d0 0x7f7ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.781+0000 7f7f01400640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee806b4d0 0x7f7ee806d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4036000 tx=0x7f7ef4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:26.785+0000 7f7f02a00640 1 -- 172.21.15.114:0/1095326536 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7efc0592f0 con 0x7f7f0c0aebf0 2024-03-06T07:38:27.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.019+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f0c04f330 con 0x7f7f0c0aebf0 2024-03-06T07:38:27.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.020+0000 7f7f02a00640 1 -- 172.21.15.114:0/1095326536 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f7efc058bf0 con 0x7f7f0c0aebf0 2024-03-06T07:38:27.023 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:27.023 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:27.023 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:27.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee806b4d0 msgr2=0x7f7ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee806b4d0 0x7f7ee806d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4036000 tx=0x7f7ef4008030 comp rx=0 tx=0).stop 2024-03-06T07:38:27.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f0c0aebf0 msgr2=0x7f7f0c1555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f0c0aebf0 0x7f7f0c1555a0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00e960 tx=0x7f7efc00ee20 comp rx=0 tx=0).stop 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 shutdown_connections 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f0c1561f0 0x7f7f0c156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f0c0cc800 0x7f7f0c1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7ee806b4d0 0x7f7ee806d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 --2- 172.21.15.114:0/1095326536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f0c0aebf0 0x7f7f0c1555a0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 >> 172.21.15.114:0/1095326536 conn(0x7f7f0c06afc0 msgr2=0x7f7f0c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 shutdown_connections 2024-03-06T07:38:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:27.022+0000 7f7f0b200640 1 -- 172.21.15.114:0/1095326536 wait complete. 2024-03-06T07:38:27.246 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:27.246 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:28.129 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:28.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.456+0000 7fac50600640 1 Processor -- start 2024-03-06T07:38:28.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.456+0000 7fac50600640 1 -- start start 2024-03-06T07:38:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac50600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac48071ba0 0x7fac48071f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac50600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac4806dbe0 0x7fac4815e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac50600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480724c0 con 0x7fac4815b3c0 2024-03-06T07:38:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac50600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4815f770 con 0x7fac48071ba0 2024-03-06T07:38:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac50600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4815f1b0 con 0x7fac4806dbe0 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac4dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac48071ba0 0x7fac48071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac4dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac48071ba0 0x7fac48071f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44152/0 (socket says 172.21.15.114:44152) 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.458+0000 7fac4dc00640 1 -- 172.21.15.114:0/2786941185 learned_addr learned my addr 172.21.15.114:0/2786941185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.459+0000 7fac4d200640 1 --2- 172.21.15.114:0/2786941185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac4806dbe0 0x7fac4815e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.459+0000 7fac4dc00640 1 -- 172.21.15.114:0/2786941185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac4806dbe0 msgr2=0x7fac4815e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.459+0000 7fac4dc00640 1 --2- 172.21.15.114:0/2786941185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac4806dbe0 0x7fac4815e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.459+0000 7fac4dc00640 1 -- 172.21.15.114:0/2786941185 >> v1:172.21.15.177:6789/0 conn(0x7fac4815b3c0 legacy=0x7fac480c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.459+0000 7fac4dc00640 1 -- 172.21.15.114:0/2786941185 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4815f490 con 0x7fac48071ba0 2024-03-06T07:38:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.460+0000 7fac4dc00640 1 --2- 172.21.15.114:0/2786941185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac48071ba0 0x7fac48071f80 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00bb10 tx=0x7fac3c02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f928ad7e568db3c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.461+0000 7fac47e00640 1 -- 172.21.15.114:0/2786941185 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c03b070 con 0x7fac48071ba0 2024-03-06T07:38:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.461+0000 7fac47e00640 1 -- 172.21.15.114:0/2786941185 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c036440 con 0x7fac48071ba0 2024-03-06T07:38:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.461+0000 7fac47e00640 1 -- 172.21.15.114:0/2786941185 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c036810 con 0x7fac48071ba0 2024-03-06T07:38:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 -- 172.21.15.114:0/2786941185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac48071ba0 msgr2=0x7fac48071f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 --2- 172.21.15.114:0/2786941185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac48071ba0 0x7fac48071f80 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00bb10 tx=0x7fac3c02e150 comp rx=0 tx=0).stop 2024-03-06T07:38:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 -- 172.21.15.114:0/2786941185 shutdown_connections 2024-03-06T07:38:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 --2- 172.21.15.114:0/2786941185 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac4806dbe0 0x7fac4815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 --2- 172.21.15.114:0/2786941185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac48071ba0 0x7fac48071f80 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 -- 172.21.15.114:0/2786941185 >> 172.21.15.114:0/2786941185 conn(0x7fac480d0ba0 msgr2=0x7fac480d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 -- 172.21.15.114:0/2786941185 shutdown_connections 2024-03-06T07:38:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.462+0000 7fac50600640 1 -- 172.21.15.114:0/2786941185 wait complete. 2024-03-06T07:38:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.463+0000 7fac50600640 1 Processor -- start 2024-03-06T07:38:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.464+0000 7fac50600640 1 -- start start 2024-03-06T07:38:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.464+0000 7fac50600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac4806dbe0 0x7fac4806b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.464+0000 7fac50600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac48071ba0 0x7fac480c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.464+0000 7fac50600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4815b3c0 0x7fac480b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.464+0000 7fac50600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806fae0 con 0x7fac4815b3c0 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.464+0000 7fac50600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806f960 con 0x7fac4806dbe0 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac50600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4815ee10 con 0x7fac48071ba0 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4815b3c0 0x7fac480b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4815b3c0 0x7fac480b5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44154/0 (socket says 172.21.15.114:44154) 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 -- 172.21.15.114:0/219489184 learned_addr learned my addr 172.21.15.114:0/219489184 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4dc00640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac4806dbe0 0x7fac4806b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4d200640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac48071ba0 0x7fac480c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 -- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac48071ba0 msgr2=0x7fac480c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac48071ba0 0x7fac480c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 -- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac4806dbe0 msgr2=0x7fac4806b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac4806dbe0 0x7fac4806b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4e600640 1 -- 172.21.15.114:0/219489184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac481578e0 con 0x7fac4815b3c0 2024-03-06T07:38:28.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.465+0000 7fac4d200640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac48071ba0 0x7fac480c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.466+0000 7fac4e600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4815b3c0 0x7fac480b5dc0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fac4000c7a0 tx=0x7fac4000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.466+0000 7fac45600640 1 -- 172.21.15.114:0/219489184 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40003b70 con 0x7fac4815b3c0 2024-03-06T07:38:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.466+0000 7fac45600640 1 -- 172.21.15.114:0/219489184 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac40003d30 con 0x7fac4815b3c0 2024-03-06T07:38:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.466+0000 7fac45600640 1 -- 172.21.15.114:0/219489184 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4000aa90 con 0x7fac4815b3c0 2024-03-06T07:38:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.466+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac48157bc0 con 0x7fac4815b3c0 2024-03-06T07:38:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.466+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac48158180 con 0x7fac4815b3c0 2024-03-06T07:38:28.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.467+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4804f330 con 0x7fac4815b3c0 2024-03-06T07:38:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.468+0000 7fac45600640 1 -- 172.21.15.114:0/219489184 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac4000ac30 con 0x7fac4815b3c0 2024-03-06T07:38:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.469+0000 7fac45600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac2c06b4d0 0x7fac2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.469+0000 7fac45600640 1 -- 172.21.15.114:0/219489184 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fac4008a230 con 0x7fac4815b3c0 2024-03-06T07:38:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.470+0000 7fac4dc00640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac2c06b4d0 0x7fac2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.470+0000 7fac4dc00640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac2c06b4d0 0x7fac2c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fac3c038000 tx=0x7fac3c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.474+0000 7fac45600640 1 -- 172.21.15.114:0/219489184 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac40058fd0 con 0x7fac4815b3c0 2024-03-06T07:38:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.750+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac4806beb0 con 0x7fac4815b3c0 2024-03-06T07:38:28.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.756+0000 7fac45600640 1 -- 172.21.15.114:0/219489184 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fac400588d0 con 0x7fac4815b3c0 2024-03-06T07:38:28.759 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:28.759 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:28.759 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.758+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac2c06b4d0 msgr2=0x7fac2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.758+0000 7fac50600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac2c06b4d0 0x7fac2c06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fac3c038000 tx=0x7fac3c008040 comp rx=0 tx=0).stop 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.758+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4815b3c0 msgr2=0x7fac480b5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.758+0000 7fac50600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4815b3c0 0x7fac480b5dc0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fac4000c7a0 tx=0x7fac4000cc60 comp rx=0 tx=0).stop 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 shutdown_connections 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fac2c06b4d0 0x7fac2c06d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac4815b3c0 0x7fac480b5dc0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fac48071ba0 0x7fac480c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 --2- 172.21.15.114:0/219489184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac4806dbe0 0x7fac4806b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 >> 172.21.15.114:0/219489184 conn(0x7fac480d0ba0 msgr2=0x7fac481517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 shutdown_connections 2024-03-06T07:38:28.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:28.759+0000 7fac50600640 1 -- 172.21.15.114:0/219489184 wait complete. 2024-03-06T07:38:28.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:28.983 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:29.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:30.130 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.181+0000 7fc9fb000640 1 Processor -- start 2024-03-06T07:38:30.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.181+0000 7fc9fb000640 1 -- start start 2024-03-06T07:38:30.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9fb000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9f4161580 0x7fc9f40d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9fb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f40d2a70 0x7fc9f40b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9fb000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f4161960 con 0x7fc9f40d1e90 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9fb000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f40cf6f0 con 0x7fc9f4161580 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9fb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f40cf9d0 con 0x7fc9f40d2a70 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9f9c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9f4161580 0x7fc9f40d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9f9c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc9f4161580 0x7fc9f40d1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44164/0 (socket says 172.21.15.114:44164) 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.183+0000 7fc9f9c00640 1 -- 172.21.15.114:0/2719595031 learned_addr learned my addr 172.21.15.114:0/2719595031 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.184+0000 7fc9fa600640 1 --2- 172.21.15.114:0/2719595031 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f40d2a70 0x7fc9f40b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.184+0000 7fc9f9c00640 1 -- 172.21.15.114:0/2719595031 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f40d2a70 msgr2=0x7fc9f40b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.184+0000 7fc9f9c00640 1 --2- 172.21.15.114:0/2719595031 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f40d2a70 0x7fc9f40b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.184+0000 7fc9f9c00640 1 -- 172.21.15.114:0/2719595031 >> v1:172.21.15.177:6789/0 conn(0x7fc9f40d1e90 legacy=0x7fc9f40d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.184+0000 7fc9f9c00640 1 -- 172.21.15.114:0/2719595031 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f40cfcb0 con 0x7fc9f4161580 2024-03-06T07:38:30.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.184+0000 7fc9fa600640 1 --2- 172.21.15.114:0/2719595031 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f40d2a70 0x7fc9f40b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.185+0000 7fc9f9c00640 1 --2- 172.21.15.114:0/2719595031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f4161580 0x7fc9f40d1950 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4073740 tx=0x7fc9e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7992c2b1be0f2267 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.185+0000 7fc9f3e00640 1 -- 172.21.15.114:0/2719595031 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8035020 con 0x7fc9f4161580 2024-03-06T07:38:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.185+0000 7fc9f3e00640 1 -- 172.21.15.114:0/2719595031 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e8031830 con 0x7fc9f4161580 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.185+0000 7fc9f3e00640 1 -- 172.21.15.114:0/2719595031 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8031c00 con 0x7fc9f4161580 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 -- 172.21.15.114:0/2719595031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f4161580 msgr2=0x7fc9f40d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 --2- 172.21.15.114:0/2719595031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f4161580 0x7fc9f40d1950 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4073740 tx=0x7fc9e802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 -- 172.21.15.114:0/2719595031 shutdown_connections 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 --2- 172.21.15.114:0/2719595031 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f40d2a70 0x7fc9f40b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 --2- 172.21.15.114:0/2719595031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f4161580 0x7fc9f40d1950 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 -- 172.21.15.114:0/2719595031 >> 172.21.15.114:0/2719595031 conn(0x7fc9f40d3d80 msgr2=0x7fc9f40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 -- 172.21.15.114:0/2719595031 shutdown_connections 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.186+0000 7fc9fb000640 1 -- 172.21.15.114:0/2719595031 wait complete. 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.188+0000 7fc9fb000640 1 Processor -- start 2024-03-06T07:38:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.188+0000 7fc9fb000640 1 -- start start 2024-03-06T07:38:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.189+0000 7fc9fb000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f40d1e90 0x7fc9f4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.189+0000 7fc9fb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f40d2a70 0x7fc9f40b7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.189+0000 7fc9fb000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f4161580 0x7fc9f415ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.189+0000 7fc9fb000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f40cf3a0 con 0x7fc9f40d2a70 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9fb000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f40b39a0 con 0x7fc9f40d1e90 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9fb000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f40b4400 con 0x7fc9f4161580 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f40d2a70 0x7fc9f40b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f40d2a70 0x7fc9f40b7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44180/0 (socket says 172.21.15.114:44180) 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 -- 172.21.15.114:0/750608080 learned_addr learned my addr 172.21.15.114:0/750608080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9c00640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f40d1e90 0x7fc9f4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9fa600640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f4161580 0x7fc9f415ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 -- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f4161580 msgr2=0x7fc9f415ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f4161580 0x7fc9f415ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 -- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f40d1e90 msgr2=0x7fc9f4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f40d1e90 0x7fc9f4158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.190+0000 7fc9f9200640 1 -- 172.21.15.114:0/750608080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f40b8210 con 0x7fc9f40d2a70 2024-03-06T07:38:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.191+0000 7fc9f9200640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f40d2a70 0x7fc9f40b7350 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c430 tx=0x7fc9e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.191+0000 7fc9f1600640 1 -- 172.21.15.114:0/750608080 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0011790 con 0x7fc9f40d2a70 2024-03-06T07:38:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.191+0000 7fc9f1600640 1 -- 172.21.15.114:0/750608080 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e0017070 con 0x7fc9f40d2a70 2024-03-06T07:38:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.191+0000 7fc9f1600640 1 -- 172.21.15.114:0/750608080 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0011b70 con 0x7fc9f40d2a70 2024-03-06T07:38:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.191+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f40b84f0 con 0x7fc9f40d2a70 2024-03-06T07:38:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.191+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f40b8ab0 con 0x7fc9f40d2a70 2024-03-06T07:38:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.192+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f40b8cb0 con 0x7fc9f40d2a70 2024-03-06T07:38:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.194+0000 7fc9f1600640 1 -- 172.21.15.114:0/750608080 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9e0011d10 con 0x7fc9f40d2a70 2024-03-06T07:38:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.195+0000 7fc9f1600640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d406b4d0 0x7fc9d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.196+0000 7fc9f1600640 1 -- 172.21.15.114:0/750608080 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc9e008a0a0 con 0x7fc9f40d2a70 2024-03-06T07:38:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.196+0000 7fc9f9c00640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d406b4d0 0x7fc9d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.196+0000 7fc9f9c00640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d406b4d0 0x7fc9d406d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8036000 tx=0x7fc9e803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.202+0000 7fc9f1600640 1 -- 172.21.15.114:0/750608080 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9f40b8cb0 con 0x7fc9f40d2a70 2024-03-06T07:38:30.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.438+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9f404f330 con 0x7fc9f40d2a70 2024-03-06T07:38:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.438+0000 7fc9f1600640 1 -- 172.21.15.114:0/750608080 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fc9e0058e40 con 0x7fc9f40d2a70 2024-03-06T07:38:30.441 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:30.441 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:30.442 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d406b4d0 msgr2=0x7fc9d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d406b4d0 0x7fc9d406d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8036000 tx=0x7fc9e803c040 comp rx=0 tx=0).stop 2024-03-06T07:38:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f40d2a70 msgr2=0x7fc9f40b7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f40d2a70 0x7fc9f40b7350 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c430 tx=0x7fc9e000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 shutdown_connections 2024-03-06T07:38:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc9f4161580 0x7fc9f415ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc9d406b4d0 0x7fc9d406d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f40d2a70 0x7fc9f40b7350 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 --2- 172.21.15.114:0/750608080 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc9f40d1e90 0x7fc9f4158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 >> 172.21.15.114:0/750608080 conn(0x7fc9f40d3d80 msgr2=0x7fc9f406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.441+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 shutdown_connections 2024-03-06T07:38:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:30.442+0000 7fc9fb000640 1 -- 172.21.15.114:0/750608080 wait complete. 2024-03-06T07:38:30.662 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:30.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:31.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:31.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.882+0000 7f9bd4200640 1 Processor -- start 2024-03-06T07:38:31.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.882+0000 7f9bd4200640 1 -- start start 2024-03-06T07:38:31.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.884+0000 7f9bd4200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9bcc0d0fc0 0x7f9bcc160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.884+0000 7f9bd4200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0cea60 con 0x7f9bcc15f470 2024-03-06T07:38:31.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.884+0000 7f9bd4200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0d2350 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.884+0000 7f9bd4200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0d2f60 con 0x7f9bcc0d0fc0 2024-03-06T07:38:31.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.884+0000 7f9bd0e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9bcc15f470 0x7f9bcc0d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47458/0 (socket says 172.21.15.114:47458) 2024-03-06T07:38:31.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.884+0000 7f9bd0e00640 1 -- 172.21.15.114:0/1869946542 learned_addr learned my addr 172.21.15.114:0/1869946542 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:31.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.884+0000 7f9bd2200640 1 --2- 172.21.15.114:0/1869946542 >> v2:172.21.15.114:3300/0 conn(0x7f9bcc0d0fc0 0x7f9bcc160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.885+0000 7f9bcbe00640 1 -- 172.21.15.114:0/1869946542 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869076525 0 0) 0x7f9bcc0cea60 con 0x7f9bcc15f470 2024-03-06T07:38:31.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.885+0000 7f9bcbe00640 1 -- 172.21.15.114:0/1869946542 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb0003520 con 0x7f9bcc15f470 2024-03-06T07:38:31.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.885+0000 7f9bd2200640 1 -- 172.21.15.114:0/1869946542 >> v1:172.21.15.177:6789/0 conn(0x7f9bcc15e0d0 legacy=0x7f9bcc0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.885+0000 7f9bd2200640 1 -- 172.21.15.114:0/1869946542 >> v1:172.21.15.114:6789/0 conn(0x7f9bcc15f470 legacy=0x7f9bcc0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.885+0000 7f9bd2200640 1 -- 172.21.15.114:0/1869946542 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0d2630 con 0x7f9bcc0d0fc0 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.886+0000 7f9bd2200640 1 --2- 172.21.15.114:0/1869946542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc0d0fc0 0x7f9bcc160dc0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009d90 tx=0x7f9bc402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93af1b2e4b8909f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.886+0000 7f9bcbe00640 1 -- 172.21.15.114:0/1869946542 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4039070 con 0x7f9bcc0d0fc0 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.886+0000 7f9bcbe00640 1 -- 172.21.15.114:0/1869946542 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc402cd80 con 0x7f9bcc0d0fc0 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.886+0000 7f9bcbe00640 1 -- 172.21.15.114:0/1869946542 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4034660 con 0x7f9bcc0d0fc0 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.887+0000 7f9bd4200640 1 -- 172.21.15.114:0/1869946542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc0d0fc0 msgr2=0x7f9bcc160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.887+0000 7f9bd4200640 1 --2- 172.21.15.114:0/1869946542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc0d0fc0 0x7f9bcc160dc0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009d90 tx=0x7f9bc402bfd0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.887+0000 7f9bd4200640 1 -- 172.21.15.114:0/1869946542 shutdown_connections 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.887+0000 7f9bd4200640 1 --2- 172.21.15.114:0/1869946542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc0d0fc0 0x7f9bcc160dc0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.887+0000 7f9bd4200640 1 -- 172.21.15.114:0/1869946542 >> 172.21.15.114:0/1869946542 conn(0x7f9bcc0d3ec0 msgr2=0x7f9bcc0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.887+0000 7f9bd4200640 1 -- 172.21.15.114:0/1869946542 shutdown_connections 2024-03-06T07:38:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.887+0000 7f9bd4200640 1 -- 172.21.15.114:0/1869946542 wait complete. 2024-03-06T07:38:31.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.888+0000 7f9bd4200640 1 Processor -- start 2024-03-06T07:38:31.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.889+0000 7f9bd4200640 1 -- start start 2024-03-06T07:38:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.889+0000 7f9bd4200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bcc0d0fc0 0x7f9bcc0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd4200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc15e0d0 0x7f9bcc155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd4200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bcc15f470 0x7f9bcc0caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd4200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0cff10 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd4200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc0d2bf0 con 0x7f9bcc0d0fc0 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd4200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc15fb30 con 0x7f9bcc15f470 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc15e0d0 0x7f9bcc155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc15e0d0 0x7f9bcc155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44198/0 (socket says 172.21.15.114:44198) 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 -- 172.21.15.114:0/3864063993 learned_addr learned my addr 172.21.15.114:0/3864063993 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd1800640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bcc0d0fc0 0x7f9bcc0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd2200640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bcc15f470 0x7f9bcc0caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 -- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bcc15f470 msgr2=0x7f9bcc0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bcc15f470 0x7f9bcc0caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 -- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bcc0d0fc0 msgr2=0x7f9bcc0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bcc0d0fc0 0x7f9bcc0c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.890+0000 7f9bd0e00640 1 -- 172.21.15.114:0/3864063993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0cb8e0 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.891+0000 7f9bd0e00640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc15e0d0 0x7f9bcc155500 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800c530 tx=0x7f9bb800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.891+0000 7f9bc9600640 1 -- 172.21.15.114:0/3864063993 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8017070 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.891+0000 7f9bc9600640 1 -- 172.21.15.114:0/3864063993 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb8005ce0 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.891+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc0c9320 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.891+0000 7f9bc9600640 1 -- 172.21.15.114:0/3864063993 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb80072b0 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.891+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc0c98e0 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.893+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc04f330 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.893+0000 7f9bc9600640 1 -- 172.21.15.114:0/3864063993 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bb801a070 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.894+0000 7f9bc9600640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:31.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.894+0000 7f9bc9600640 1 -- 172.21.15.114:0/3864063993 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9bb8089760 con 0x7f9bcc15e0d0 2024-03-06T07:38:31.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.894+0000 7f9bd1800640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:31.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.895+0000 7f9bd1800640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9bac06b4d0 0x7f9bac06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc000f60 tx=0x7f9bbc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:31.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:31.898+0000 7f9bc9600640 1 -- 172.21.15.114:0/3864063993 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bb80585c0 con 0x7f9bcc15e0d0 2024-03-06T07:38:32.132 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:32.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.174+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bcc08d920 con 0x7f9bcc15e0d0 2024-03-06T07:38:32.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.175+0000 7f9bc9600640 1 -- 172.21.15.114:0/3864063993 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9bb8057ec0 con 0x7f9bcc15e0d0 2024-03-06T07:38:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:32.178 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:32.179 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9bac06b4d0 msgr2=0x7f9bac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9bac06b4d0 0x7f9bac06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc000f60 tx=0x7f9bbc009040 comp rx=0 tx=0).stop 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc15e0d0 msgr2=0x7f9bcc155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc15e0d0 0x7f9bcc155500 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800c530 tx=0x7f9bb800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 shutdown_connections 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9bcc15f470 0x7f9bcc0caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bcc15e0d0 0x7f9bcc155500 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 --2- 172.21.15.114:0/3864063993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bcc0d0fc0 0x7f9bcc0c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 >> 172.21.15.114:0/3864063993 conn(0x7f9bcc0d3ec0 msgr2=0x7f9bcc0b60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 shutdown_connections 2024-03-06T07:38:32.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:32.179+0000 7f9bd4200640 1 -- 172.21.15.114:0/3864063993 wait complete. 2024-03-06T07:38:32.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:32.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:33.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:33.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.619+0000 7fe6d2c00640 1 Processor -- start 2024-03-06T07:38:33.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.619+0000 7fe6d2c00640 1 -- start start 2024-03-06T07:38:33.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.621+0000 7fe6d2c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 0x7fe6cc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.621+0000 7fe6d2c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc06ee70 con 0x7fe6cc15fe30 2024-03-06T07:38:33.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.621+0000 7fe6d2c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc06cef0 con 0x7fe6cc15f9d0 2024-03-06T07:38:33.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.621+0000 7fe6d2c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc06d1d0 con 0x7fe6cc144c00 2024-03-06T07:38:33.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.621+0000 7fe6d0c00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe6cc15fe30 0x7fe6cc0ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47474/0 (socket says 172.21.15.114:47474) 2024-03-06T07:38:33.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.621+0000 7fe6d0c00640 1 -- 172.21.15.114:0/422621609 learned_addr learned my addr 172.21.15.114:0/422621609 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:33.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.621+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 0x7fe6cc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228229758 0 0) 0x7fe6c4001490 con 0x7fe6cc15fe30 2024-03-06T07:38:33.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b0003520 con 0x7fe6cc15fe30 2024-03-06T07:38:33.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944232936 0 0) 0x7fe6b0003520 con 0x7fe6cc15fe30 2024-03-06T07:38:33.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 msgr2=0x7fe6cc06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 0x7fe6cc06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 >> v1:172.21.15.177:6789/0 conn(0x7fe6cc15f9d0 legacy=0x7fe6cc0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc06d4b0 con 0x7fe6cc15fe30 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fe6c4001bc0 con 0x7fe6cc15fe30 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 >> v1:172.21.15.114:6789/0 conn(0x7fe6cc15fe30 legacy=0x7fe6cc0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 0x7fe6b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6b0004d40 0x7fe6b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc06d1d0 con 0x7fe6b00039e0 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b000ba70 con 0x7fe6b0004d40 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.622+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b000bc20 con 0x7fe6cc144c00 2024-03-06T07:38:33.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cb400640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 0x7fe6b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6d0c00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6b0004d40 0x7fe6b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cbe00640 1 -- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 msgr2=0x7fe6b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 0x7fe6b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cbe00640 1 -- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6b0004d40 msgr2=0x7fe6b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6b0004d40 0x7fe6b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cbe00640 1 -- 172.21.15.114:0/422621609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0003520 con 0x7fe6b00039e0 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6cb400640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc144c00 0x7fe6b0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.623+0000 7fe6d0c00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6b0004d40 0x7fe6b00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.624+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6b0003dc0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009ac0 tx=0x7fe6b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=411a97f7ed74dd6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc039070 con 0x7fe6b00039e0 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc02c6b0 con 0x7fe6b00039e0 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6caa00640 1 -- 172.21.15.114:0/422621609 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc009690 con 0x7fe6b00039e0 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6d2c00640 1 -- 172.21.15.114:0/422621609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 msgr2=0x7fe6b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6d2c00640 1 --2- 172.21.15.114:0/422621609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6b0003dc0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009ac0 tx=0x7fe6b0005830 comp rx=0 tx=0).stop 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6d0c00640 1 -- 172.21.15.114:0/422621609 reap_dead start 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6d2c00640 1 -- 172.21.15.114:0/422621609 shutdown_connections 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6d2c00640 1 -- 172.21.15.114:0/422621609 >> 172.21.15.114:0/422621609 conn(0x7fe6cc0b1730 msgr2=0x7fe6cc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6d2c00640 1 -- 172.21.15.114:0/422621609 shutdown_connections 2024-03-06T07:38:33.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.625+0000 7fe6d2c00640 1 -- 172.21.15.114:0/422621609 wait complete. 2024-03-06T07:38:33.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.627+0000 7fe6d2c00640 1 Processor -- start 2024-03-06T07:38:33.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.627+0000 7fe6d2c00640 1 -- start start 2024-03-06T07:38:33.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6d2c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6cc15eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6d2c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6cc14d640 0x7fe6cc13f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6d2c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc15c890 0x7fe6cc13fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6d2c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc160730 con 0x7fe6b00039e0 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6d2c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc160280 con 0x7fe6cc14d640 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6d2c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc0ceec0 con 0x7fe6cc15c890 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6cbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6cc15eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6cbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6cc15eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44214/0 (socket says 172.21.15.114:44214) 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.628+0000 7fe6cbe00640 1 -- 172.21.15.114:0/3272373624 learned_addr learned my addr 172.21.15.114:0/3272373624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6cbe00640 1 -- 172.21.15.114:0/3272373624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc15c890 msgr2=0x7fe6cc13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc15c890 0x7fe6cc13fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6cbe00640 1 -- 172.21.15.114:0/3272373624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6cc14d640 msgr2=0x7fe6cc13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6cc14d640 0x7fe6cc13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6cbe00640 1 -- 172.21.15.114:0/3272373624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc15f710 con 0x7fe6b00039e0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6cbe00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6cc15eb30 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc000c00 tx=0x7fe6bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6c3e00640 1 -- 172.21.15.114:0/3272373624 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc042020 con 0x7fe6b00039e0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6c3e00640 1 -- 172.21.15.114:0/3272373624 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc03a040 con 0x7fe6b00039e0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6c3e00640 1 -- 172.21.15.114:0/3272373624 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc039070 con 0x7fe6b00039e0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.629+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc15f430 con 0x7fe6b00039e0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.630+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc0b2ac0 con 0x7fe6b00039e0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.631+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc04f330 con 0x7fe6b00039e0 2024-03-06T07:38:33.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.632+0000 7fe6c3e00640 1 -- 172.21.15.114:0/3272373624 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6bc045020 con 0x7fe6b00039e0 2024-03-06T07:38:33.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.633+0000 7fe6c3e00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:33.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.633+0000 7fe6c3e00640 1 -- 172.21.15.114:0/3272373624 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe6bc0ab6d0 con 0x7fe6b00039e0 2024-03-06T07:38:33.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.633+0000 7fe6cb400640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:33.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.634+0000 7fe6cb400640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0b1080 tx=0x7fe6b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:33.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.638+0000 7fe6c3e00640 1 -- 172.21.15.114:0/3272373624 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6bc04c070 con 0x7fe6b00039e0 2024-03-06T07:38:33.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.873+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6cc144320 con 0x7fe6b00039e0 2024-03-06T07:38:33.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.874+0000 7fe6c3e00640 1 -- 172.21.15.114:0/3272373624 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fe6bc07a470 con 0x7fe6b00039e0 2024-03-06T07:38:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:33.878 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:33.878 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:33.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe6ac06b500 msgr2=0x7fe6ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0b1080 tx=0x7fe6b8009040 comp rx=0 tx=0).stop 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 msgr2=0x7fe6cc15eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6cc15eb30 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc000c00 tx=0x7fe6bc004730 comp rx=0 tx=0).stop 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 shutdown_connections 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fe6cc15c890 0x7fe6cc13fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6cc14d640 0x7fe6cc13f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 --2- 172.21.15.114:0/3272373624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b00039e0 0x7fe6cc15eb30 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.877+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 >> 172.21.15.114:0/3272373624 conn(0x7fe6cc0b1730 msgr2=0x7fe6cc0afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.878+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 shutdown_connections 2024-03-06T07:38:33.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:33.878+0000 7fe6d2c00640 1 -- 172.21.15.114:0/3272373624 wait complete. 2024-03-06T07:38:34.102 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:34.102 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:34.134 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:35.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:35.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.317+0000 7f3381200640 1 Processor -- start 2024-03-06T07:38:35.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.317+0000 7f3381200640 1 -- start start 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3381200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f337c144c00 0x7f337c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3381200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f337c15fe60 0x7f337c0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3381200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c06ee70 con 0x7f337c15f9d0 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3381200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337c06cef0 con 0x7f337c144c00 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3381200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337c06d1d0 con 0x7f337c15fe60 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3379e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f337c144c00 0x7f337c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3379400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f337c15f9d0 0x7f337c0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47484/0 (socket says 172.21.15.114:47484) 2024-03-06T07:38:35.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f3379400640 1 -- 172.21.15.114:0/3243188151 learned_addr learned my addr 172.21.15.114:0/3243188151 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.319+0000 7f337a800640 1 --2- 172.21.15.114:0/3243188151 >> v2:172.21.15.177:3300/0 conn(0x7f337c15fe60 0x7f337c0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.320+0000 7f3379e00640 1 -- 172.21.15.114:0/3243188151 >> v2:172.21.15.177:3300/0 conn(0x7f337c15fe60 msgr2=0x7f337c0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.320+0000 7f3379e00640 1 --2- 172.21.15.114:0/3243188151 >> v2:172.21.15.177:3300/0 conn(0x7f337c15fe60 0x7f337c0ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.320+0000 7f3379e00640 1 -- 172.21.15.114:0/3243188151 >> v1:172.21.15.114:6789/0 conn(0x7f337c15f9d0 legacy=0x7f337c0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.320+0000 7f3379e00640 1 -- 172.21.15.114:0/3243188151 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c06d4b0 con 0x7f337c144c00 2024-03-06T07:38:35.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.320+0000 7f3378a00640 1 -- 172.21.15.114:0/3243188151 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377956514 0 0) 0x7f3368001110 con 0x7f337c15f9d0 2024-03-06T07:38:35.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.320+0000 7f337a800640 1 --2- 172.21.15.114:0/3243188151 >> v2:172.21.15.177:3300/0 conn(0x7f337c15fe60 0x7f337c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:35.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.321+0000 7f3379e00640 1 --2- 172.21.15.114:0/3243188151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c06e930 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f337c06e500 tx=0x7f336c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7910ef42aadb3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.321+0000 7f3378a00640 1 -- 172.21.15.114:0/3243188151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c035020 con 0x7f337c144c00 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.321+0000 7f3378a00640 1 -- 172.21.15.114:0/3243188151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c031830 con 0x7f337c144c00 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.321+0000 7f3378a00640 1 -- 172.21.15.114:0/3243188151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c031c00 con 0x7f337c144c00 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 -- 172.21.15.114:0/3243188151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 msgr2=0x7f337c06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 --2- 172.21.15.114:0/3243188151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c06e930 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f337c06e500 tx=0x7f336c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 -- 172.21.15.114:0/3243188151 shutdown_connections 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 --2- 172.21.15.114:0/3243188151 >> v2:172.21.15.177:3300/0 conn(0x7f337c15fe60 0x7f337c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 --2- 172.21.15.114:0/3243188151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c06e930 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 -- 172.21.15.114:0/3243188151 >> 172.21.15.114:0/3243188151 conn(0x7f337c0b1730 msgr2=0x7f337c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 -- 172.21.15.114:0/3243188151 shutdown_connections 2024-03-06T07:38:35.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.322+0000 7f3381200640 1 -- 172.21.15.114:0/3243188151 wait complete. 2024-03-06T07:38:35.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.324+0000 7f3381200640 1 Processor -- start 2024-03-06T07:38:35.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.324+0000 7f3381200640 1 -- start start 2024-03-06T07:38:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3381200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c13c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3381200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f337c15f9d0 0x7f337c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3381200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f337c15fe60 0x7f337c1e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3381200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c160730 con 0x7f337c144c00 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3381200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c160270 con 0x7f337c15f9d0 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3381200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c06cb50 con 0x7f337c15fe60 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c13c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c13c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44238/0 (socket says 172.21.15.114:44238) 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379e00640 1 -- 172.21.15.114:0/3095359825 learned_addr learned my addr 172.21.15.114:0/3095359825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f337a800640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f337c15fe60 0x7f337c1e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379400640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f337c15f9d0 0x7f337c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379e00640 1 -- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f337c15fe60 msgr2=0x7f337c1e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379e00640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f337c15fe60 0x7f337c1e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379e00640 1 -- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f337c15f9d0 msgr2=0x7f337c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.326+0000 7f3379e00640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f337c15f9d0 0x7f337c15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.327+0000 7f3379e00640 1 -- 172.21.15.114:0/3095359825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c1e6df0 con 0x7f337c144c00 2024-03-06T07:38:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.327+0000 7f3379e00640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c13c680 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f336c000c00 tx=0x7f336c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.327+0000 7f3372000640 1 -- 172.21.15.114:0/3095359825 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c041070 con 0x7f337c144c00 2024-03-06T07:38:35.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.327+0000 7f3372000640 1 -- 172.21.15.114:0/3095359825 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c004230 con 0x7f337c144c00 2024-03-06T07:38:35.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.327+0000 7f3372000640 1 -- 172.21.15.114:0/3095359825 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c035040 con 0x7f337c144c00 2024-03-06T07:38:35.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.327+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c1e7380 con 0x7f337c144c00 2024-03-06T07:38:35.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.327+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c1e7940 con 0x7f337c144c00 2024-03-06T07:38:35.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.329+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c04f330 con 0x7f337c144c00 2024-03-06T07:38:35.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.329+0000 7f3372000640 1 -- 172.21.15.114:0/3095359825 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f336c038020 con 0x7f337c144c00 2024-03-06T07:38:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.331+0000 7f3372000640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f335c06b4d0 0x7f335c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.331+0000 7f3372000640 1 -- 172.21.15.114:0/3095359825 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f336c0abdb0 con 0x7f337c144c00 2024-03-06T07:38:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.331+0000 7f3379400640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f335c06b4d0 0x7f335c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.332+0000 7f3379400640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f335c06b4d0 0x7f335c06d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f337c13afc0 tx=0x7f3368007970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.336+0000 7f3372000640 1 -- 172.21.15.114:0/3095359825 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f336c0b0070 con 0x7f337c144c00 2024-03-06T07:38:35.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.570+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337c072890 con 0x7f337c144c00 2024-03-06T07:38:35.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.571+0000 7f3372000640 1 -- 172.21.15.114:0/3095359825 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f336c07abe0 con 0x7f337c144c00 2024-03-06T07:38:35.574 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:35.574 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:35.574 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:35.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f335c06b4d0 msgr2=0x7f335c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f335c06b4d0 0x7f335c06d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f337c13afc0 tx=0x7f3368007970 comp rx=0 tx=0).stop 2024-03-06T07:38:35.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 msgr2=0x7f337c13c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c13c680 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f336c000c00 tx=0x7f336c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 shutdown_connections 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f337c15fe60 0x7f337c1e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f337c15f9d0 0x7f337c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f335c06b4d0 0x7f335c06d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 --2- 172.21.15.114:0/3095359825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f337c144c00 0x7f337c13c680 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.573+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 >> 172.21.15.114:0/3095359825 conn(0x7f337c0b1730 msgr2=0x7f337c142d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.574+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 shutdown_connections 2024-03-06T07:38:35.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:35.574+0000 7f3381200640 1 -- 172.21.15.114:0/3095359825 wait complete. 2024-03-06T07:38:35.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:35.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:36.136 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:36.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:37.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.006+0000 7f58dce00640 1 Processor -- start 2024-03-06T07:38:37.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.006+0000 7f58dce00640 1 -- start start 2024-03-06T07:38:37.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.007+0000 7f58dce00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f58d815acb0 0x7f58d80b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58dce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58dce00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d815b090 con 0x7f58d8072340 2024-03-06T07:38:37.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58dce00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d80b2a30 con 0x7f58d815acb0 2024-03-06T07:38:37.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58dce00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d80b4360 con 0x7f58d80b4680 2024-03-06T07:38:37.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58d5400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58d5400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d806e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:45300/0 (socket says 172.21.15.114:45300) 2024-03-06T07:38:37.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58d5400640 1 -- 172.21.15.114:0/2871352459 learned_addr learned my addr 172.21.15.114:0/2871352459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:37.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.008+0000 7f58d5e00640 1 --2- 172.21.15.114:0/2871352459 >> v2:172.21.15.177:3300/0 conn(0x7f58d815acb0 0x7f58d80b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.009+0000 7f58d5400640 1 -- 172.21.15.114:0/2871352459 >> v2:172.21.15.177:3300/0 conn(0x7f58d815acb0 msgr2=0x7f58d80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:37.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.009+0000 7f58d5400640 1 --2- 172.21.15.114:0/2871352459 >> v2:172.21.15.177:3300/0 conn(0x7f58d815acb0 0x7f58d80b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.009+0000 7f58d5400640 1 -- 172.21.15.114:0/2871352459 >> v1:172.21.15.177:6789/0 conn(0x7f58d8072340 legacy=0x7f58d80ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.009+0000 7f58d5400640 1 -- 172.21.15.114:0/2871352459 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d80b2cf0 con 0x7f58d80b4680 2024-03-06T07:38:37.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.011+0000 7f58d5400640 1 --2- 172.21.15.114:0/2871352459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d806e4f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009870 tx=0x7f58c002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f3b9efee999ca95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.011+0000 7f58d4a00640 1 -- 172.21.15.114:0/2871352459 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c0034070 con 0x7f58d80b4680 2024-03-06T07:38:37.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.011+0000 7f58d4a00640 1 -- 172.21.15.114:0/2871352459 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c0030840 con 0x7f58d80b4680 2024-03-06T07:38:37.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.011+0000 7f58d4a00640 1 -- 172.21.15.114:0/2871352459 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c0030c10 con 0x7f58d80b4680 2024-03-06T07:38:37.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 -- 172.21.15.114:0/2871352459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 msgr2=0x7f58d806e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 --2- 172.21.15.114:0/2871352459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d806e4f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009870 tx=0x7f58c002b980 comp rx=0 tx=0).stop 2024-03-06T07:38:37.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 -- 172.21.15.114:0/2871352459 shutdown_connections 2024-03-06T07:38:37.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 --2- 172.21.15.114:0/2871352459 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d806e4f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 --2- 172.21.15.114:0/2871352459 >> v2:172.21.15.177:3300/0 conn(0x7f58d815acb0 0x7f58d80b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 -- 172.21.15.114:0/2871352459 >> 172.21.15.114:0/2871352459 conn(0x7f58d80b1730 msgr2=0x7f58d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 -- 172.21.15.114:0/2871352459 shutdown_connections 2024-03-06T07:38:37.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.012+0000 7f58dce00640 1 -- 172.21.15.114:0/2871352459 wait complete. 2024-03-06T07:38:37.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.013+0000 7f58dce00640 1 Processor -- start 2024-03-06T07:38:37.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.014+0000 7f58dce00640 1 -- start start 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58dce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d8072340 0x7f58d814ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58dce00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d8159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58dce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d815acb0 0x7f58d8167be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58dce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d80b3d10 con 0x7f58d8072340 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58dce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d80b32b0 con 0x7f58d815acb0 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58dce00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d80b4010 con 0x7f58d80b4680 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d8072340 0x7f58d814ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d8072340 0x7f58d814ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44250/0 (socket says 172.21.15.114:44250) 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 -- 172.21.15.114:0/3021477904 learned_addr learned my addr 172.21.15.114:0/3021477904 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5400640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d8159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d6800640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d815acb0 0x7f58d8167be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 -- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 msgr2=0x7f58d8159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d8159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 -- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d815acb0 msgr2=0x7f58d8167be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d815acb0 0x7f58d8167be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5e00640 1 -- 172.21.15.114:0/3021477904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d814b9e0 con 0x7f58d8072340 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d5400640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d8159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.015+0000 7f58d6800640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d815acb0 0x7f58d8167be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:37.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.016+0000 7f58d5e00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d8072340 0x7f58d814ca80 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009db0 tx=0x7f58c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.016+0000 7f58ce000640 1 -- 172.21.15.114:0/3021477904 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8018020 con 0x7f58d8072340 2024-03-06T07:38:37.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.016+0000 7f58ce000640 1 -- 172.21.15.114:0/3021477904 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c8014480 con 0x7f58d8072340 2024-03-06T07:38:37.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.016+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d814bcc0 con 0x7f58d8072340 2024-03-06T07:38:37.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.016+0000 7f58ce000640 1 -- 172.21.15.114:0/3021477904 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8014850 con 0x7f58d8072340 2024-03-06T07:38:37.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.016+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d814c280 con 0x7f58d8072340 2024-03-06T07:38:37.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.017+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d804f330 con 0x7f58d8072340 2024-03-06T07:38:37.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.018+0000 7f58ce000640 1 -- 172.21.15.114:0/3021477904 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58c8005d10 con 0x7f58d8072340 2024-03-06T07:38:37.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.019+0000 7f58ce000640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.019+0000 7f58ce000640 1 -- 172.21.15.114:0/3021477904 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f58c808aef0 con 0x7f58d8072340 2024-03-06T07:38:37.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.019+0000 7f58d5400640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.020+0000 7f58d5400640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f58c0036000 tx=0x7f58c003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.024+0000 7f58ce000640 1 -- 172.21.15.114:0/3021477904 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58c8022070 con 0x7f58d8072340 2024-03-06T07:38:37.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.259+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58d80d05d0 con 0x7f58d8072340 2024-03-06T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.260+0000 7f58ce000640 1 -- 172.21.15.114:0/3021477904 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f58c8010040 con 0x7f58d8072340 2024-03-06T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:37.263 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 msgr2=0x7f58b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f58c0036000 tx=0x7f58c003b040 comp rx=0 tx=0).stop 2024-03-06T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d8072340 msgr2=0x7f58d814ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d8072340 0x7f58d814ca80 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009db0 tx=0x7f58c800c490 comp rx=0 tx=0).stop 2024-03-06T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 shutdown_connections 2024-03-06T07:38:37.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f58d815acb0 0x7f58d8167be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f58d80b4680 0x7f58d8159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 --2- 172.21.15.114:0/3021477904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58d8072340 0x7f58d814ca80 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 >> 172.21.15.114:0/3021477904 conn(0x7f58d80b1730 msgr2=0x7f58d80af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 shutdown_connections 2024-03-06T07:38:37.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:37.263+0000 7f58dce00640 1 -- 172.21.15.114:0/3021477904 wait complete. 2024-03-06T07:38:37.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:37.486 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:38.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.698+0000 7f7f44600640 1 Processor -- start 2024-03-06T07:38:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.698+0000 7f7f44600640 1 -- start start 2024-03-06T07:38:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f44600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 0x7f7f3c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f44600640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7f3c15f470 0x7f7f3c0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f44600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3c0cea60 con 0x7f7f3c0d0df0 2024-03-06T07:38:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f44600640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3c0d1260 con 0x7f7f3c15f470 2024-03-06T07:38:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f44600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0d2f60 con 0x7f7f3c15e0d0 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f42600640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7f3c0d0df0 0x7f7f3c160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39272/0 (socket says 172.21.15.114:39272) 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f42600640 1 -- 172.21.15.114:0/2956102903 learned_addr learned my addr 172.21.15.114:0/2956102903 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.700+0000 7f7f41200640 1 --2- 172.21.15.114:0/2956102903 >> v2:172.21.15.177:3300/0 conn(0x7f7f3c15f470 0x7f7f3c0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f41c00640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 0x7f7f3c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585145457 0 0) 0x7f7f3c0cea60 con 0x7f7f3c0d0df0 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f24003520 con 0x7f7f3c0d0df0 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4196941715 0 0) 0x7f7f24003520 con 0x7f7f3c0d0df0 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 msgr2=0x7f7f3c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 0x7f7f3c0d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 >> v2:172.21.15.177:3300/0 conn(0x7f7f3c15f470 msgr2=0x7f7f3c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 --2- 172.21.15.114:0/2956102903 >> v2:172.21.15.177:3300/0 conn(0x7f7f3c15f470 0x7f7f3c0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0d2350 con 0x7f7f3c0d0df0 2024-03-06T07:38:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f7f340019f0 con 0x7f7f3c0d0df0 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 >> v1:172.21.15.114:6789/0 conn(0x7f7f3c0d0df0 legacy=0x7f7f3c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 0x7f7f240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24003f30 0x7f7f24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f24004d40 0x7f7f240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0d1260 con 0x7f7f24004d40 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2400b9c0 con 0x7f7f24003f30 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.701+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2400bb70 con 0x7f7f3c15e0d0 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f42600640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f24004d40 0x7f7f240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f41c00640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 0x7f7f240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f42600640 1 -- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 msgr2=0x7f7f240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f41200640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24003f30 0x7f7f24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f42600640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c15e0d0 0x7f7f240039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f42600640 1 -- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24003f30 msgr2=0x7f7f24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f42600640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24003f30 0x7f7f24004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.702+0000 7f7f42600640 1 -- 172.21.15.114:0/2956102903 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f24003520 con 0x7f7f24004d40 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.703+0000 7f7f42600640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f24004d40 0x7f7f240051c0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400a8a0 tx=0x7f7f24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94a027c0ead8ace server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.703+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3400a4c0 con 0x7f7f24004d40 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.703+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f34034650 con 0x7f7f24004d40 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.703+0000 7f7f3be00640 1 -- 172.21.15.114:0/2956102903 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34034a80 con 0x7f7f24004d40 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.704+0000 7f7f44600640 1 -- 172.21.15.114:0/2956102903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f24004d40 msgr2=0x7f7f240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.704+0000 7f7f44600640 1 --2- 172.21.15.114:0/2956102903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f24004d40 0x7f7f240051c0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400a8a0 tx=0x7f7f24005830 comp rx=0 tx=0).stop 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.704+0000 7f7f42600640 1 -- 172.21.15.114:0/2956102903 reap_dead start 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.704+0000 7f7f44600640 1 -- 172.21.15.114:0/2956102903 shutdown_connections 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.704+0000 7f7f44600640 1 -- 172.21.15.114:0/2956102903 >> 172.21.15.114:0/2956102903 conn(0x7f7f3c0d3ec0 msgr2=0x7f7f3c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.704+0000 7f7f44600640 1 -- 172.21.15.114:0/2956102903 shutdown_connections 2024-03-06T07:38:38.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.704+0000 7f7f44600640 1 -- 172.21.15.114:0/2956102903 wait complete. 2024-03-06T07:38:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.706+0000 7f7f44600640 1 Processor -- start 2024-03-06T07:38:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.706+0000 7f7f44600640 1 -- start start 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f44600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24004d40 0x7f7f3c1f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f44600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1f7d20 0x7f7f3c0b9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f44600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c0b1ad0 0x7f7f3c0b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f44600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0d2bf0 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f44600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c0d2910 con 0x7f7f24004d40 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f44600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3c15fb30 con 0x7f7f3c0b1ad0 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f41200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1f7d20 0x7f7f3c0b9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f41200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1f7d20 0x7f7f3c0b9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60548/0 (socket says 172.21.15.114:60548) 2024-03-06T07:38:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.707+0000 7f7f41200640 1 -- 172.21.15.114:0/421276353 learned_addr learned my addr 172.21.15.114:0/421276353 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f41200640 1 -- 172.21.15.114:0/421276353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c0b1ad0 msgr2=0x7f7f3c0b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f41200640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c0b1ad0 0x7f7f3c0b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f41200640 1 -- 172.21.15.114:0/421276353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24004d40 msgr2=0x7f7f3c1f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f41c00640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24004d40 0x7f7f3c1f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f41200640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24004d40 0x7f7f3c1f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f41200640 1 -- 172.21.15.114:0/421276353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0c94f0 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f41200640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1f7d20 0x7f7f3c0b9710 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f7f28010860 tx=0x7f7f28010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f39600640 1 -- 172.21.15.114:0/421276353 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2800f040 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f39600640 1 -- 172.21.15.114:0/421276353 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2800eb60 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f39600640 1 -- 172.21.15.114:0/421276353 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2801c400 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c1f81e0 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.708+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3c1f87a0 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.710+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3c04f330 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.710+0000 7f7f39600640 1 -- 172.21.15.114:0/421276353 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f2801c5a0 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.711+0000 7f7f39600640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f1c06b4d0 0x7f7f1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.711+0000 7f7f39600640 1 -- 172.21.15.114:0/421276353 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f2808a860 con 0x7f7f3c1f7d20 2024-03-06T07:38:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.711+0000 7f7f41c00640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f1c06b4d0 0x7f7f1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.712+0000 7f7f41c00640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f1c06b4d0 0x7f7f1c06d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009950 tx=0x7f7f30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:38.716+0000 7f7f39600640 1 -- 172.21.15.114:0/421276353 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f28018030 con 0x7f7f3c1f7d20 2024-03-06T07:38:39.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.006+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f3c0ce5d0 con 0x7f7f3c1f7d20 2024-03-06T07:38:39.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.007+0000 7f7f39600640 1 -- 172.21.15.114:0/421276353 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f7f28059690 con 0x7f7f3c1f7d20 2024-03-06T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:39.010 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.010+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f1c06b4d0 msgr2=0x7f7f1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.010+0000 7f7f44600640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f1c06b4d0 0x7f7f1c06d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009950 tx=0x7f7f30008040 comp rx=0 tx=0).stop 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.010+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1f7d20 msgr2=0x7f7f3c0b9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.010+0000 7f7f44600640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1f7d20 0x7f7f3c0b9710 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f7f28010860 tx=0x7f7f28010d20 comp rx=0 tx=0).stop 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 shutdown_connections 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f7f3c0b1ad0 0x7f7f3c0b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f7f1c06b4d0 0x7f7f1c06d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f3c1f7d20 0x7f7f3c0b9710 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 --2- 172.21.15.114:0/421276353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7f24004d40 0x7f7f3c1f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 >> 172.21.15.114:0/421276353 conn(0x7f7f3c0d3ec0 msgr2=0x7f7f3c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 shutdown_connections 2024-03-06T07:38:39.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:39.011+0000 7f7f44600640 1 -- 172.21.15.114:0/421276353 wait complete. 2024-03-06T07:38:39.234 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:39.234 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:40.141 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:40.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:40.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.442+0000 7fa6e6000640 1 Processor -- start 2024-03-06T07:38:40.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.442+0000 7fa6e6000640 1 -- start start 2024-03-06T07:38:40.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.444+0000 7fa6e6000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa6e0066f10 0x7fa6e006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.444+0000 7fa6e6000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.444+0000 7fa6e6000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e006e5b0 con 0x7fa6e015f930 2024-03-06T07:38:40.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.444+0000 7fa6e6000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e00ccd90 con 0x7fa6e0066f10 2024-03-06T07:38:40.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.444+0000 7fa6e6000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00afc00 con 0x7fa6e00ce8f0 2024-03-06T07:38:40.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.445+0000 7fa6dfe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.445+0000 7fa6dfe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e00ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:38936/0 (socket says 172.21.15.114:38936) 2024-03-06T07:38:40.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.445+0000 7fa6df400640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7fa6e015f930 0x7fa6e015fd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:53926/0 (socket says 172.21.15.114:53926) 2024-03-06T07:38:40.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.445+0000 7fa6dfe00640 1 -- 172.21.15.114:0/1235511076 learned_addr learned my addr 172.21.15.114:0/1235511076 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:40.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.445+0000 7fa6dea00640 1 --2- 172.21.15.114:0/1235511076 >> v2:172.21.15.177:3300/0 conn(0x7fa6e0066f10 0x7fa6e006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.445+0000 7fa6de000640 1 -- 172.21.15.114:0/1235511076 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593208608 0 0) 0x7fa6e006e5b0 con 0x7fa6e015f930 2024-03-06T07:38:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.446+0000 7fa6de000640 1 -- 172.21.15.114:0/1235511076 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4003520 con 0x7fa6e015f930 2024-03-06T07:38:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.446+0000 7fa6dfe00640 1 -- 172.21.15.114:0/1235511076 >> v2:172.21.15.177:3300/0 conn(0x7fa6e0066f10 msgr2=0x7fa6e006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.446+0000 7fa6dfe00640 1 --2- 172.21.15.114:0/1235511076 >> v2:172.21.15.177:3300/0 conn(0x7fa6e0066f10 0x7fa6e006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.446+0000 7fa6dfe00640 1 -- 172.21.15.114:0/1235511076 >> v1:172.21.15.177:6789/0 conn(0x7fa6e015f930 legacy=0x7fa6e015fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.446+0000 7fa6dfe00640 1 -- 172.21.15.114:0/1235511076 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e00ccfe0 con 0x7fa6e00ce8f0 2024-03-06T07:38:40.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.446+0000 7fa6dea00640 1 --2- 172.21.15.114:0/1235511076 >> v2:172.21.15.177:3300/0 conn(0x7fa6e0066f10 0x7fa6e006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:38:40.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.448+0000 7fa6dfe00640 1 --2- 172.21.15.114:0/1235511076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e00ced80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009e50 tx=0x7fa6d802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69b0f3c019e36a28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6de000640 1 -- 172.21.15.114:0/1235511076 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d802ca70 con 0x7fa6e00ce8f0 2024-03-06T07:38:40.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6de000640 1 -- 172.21.15.114:0/1235511076 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d802cc10 con 0x7fa6e00ce8f0 2024-03-06T07:38:40.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6de000640 1 -- 172.21.15.114:0/1235511076 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d80343f0 con 0x7fa6e00ce8f0 2024-03-06T07:38:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 -- 172.21.15.114:0/1235511076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 msgr2=0x7fa6e00ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 --2- 172.21.15.114:0/1235511076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e00ced80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009e50 tx=0x7fa6d802bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 -- 172.21.15.114:0/1235511076 shutdown_connections 2024-03-06T07:38:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 --2- 172.21.15.114:0/1235511076 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e00ced80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 --2- 172.21.15.114:0/1235511076 >> v2:172.21.15.177:3300/0 conn(0x7fa6e0066f10 0x7fa6e006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 -- 172.21.15.114:0/1235511076 >> 172.21.15.114:0/1235511076 conn(0x7fa6e00d0340 msgr2=0x7fa6e00d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:40.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 -- 172.21.15.114:0/1235511076 shutdown_connections 2024-03-06T07:38:40.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.449+0000 7fa6e6000640 1 -- 172.21.15.114:0/1235511076 wait complete. 2024-03-06T07:38:40.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.451+0000 7fa6e6000640 1 Processor -- start 2024-03-06T07:38:40.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.451+0000 7fa6e6000640 1 -- start start 2024-03-06T07:38:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.452+0000 7fa6e6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e0066f10 0x7fa6e0145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.452+0000 7fa6e6000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e014e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.452+0000 7fa6e6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6e015f930 0x7fa6e015ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.452+0000 7fa6e6000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e006e090 con 0x7fa6e015f930 2024-03-06T07:38:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.452+0000 7fa6e6000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e006ddb0 con 0x7fa6e0066f10 2024-03-06T07:38:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.452+0000 7fa6e6000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e006c780 con 0x7fa6e00ce8f0 2024-03-06T07:38:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6e015f930 0x7fa6e015ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6e015f930 0x7fa6e015ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60560/0 (socket says 172.21.15.114:60560) 2024-03-06T07:38:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 -- 172.21.15.114:0/2411366983 learned_addr learned my addr 172.21.15.114:0/2411366983 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6df400640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e0066f10 0x7fa6e0145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dea00640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e014e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 -- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 msgr2=0x7fa6e014e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e014e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 -- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e0066f10 msgr2=0x7fa6e0145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e0066f10 0x7fa6e0145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6dfe00640 1 -- 172.21.15.114:0/2411366983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e0151460 con 0x7fa6e015f930 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.453+0000 7fa6df400640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e0066f10 0x7fa6e0145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.454+0000 7fa6dfe00640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6e015f930 0x7fa6e015ae10 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8036040 tx=0x7fa6d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.454+0000 7fa6d7400640 1 -- 172.21.15.114:0/2411366983 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8043020 con 0x7fa6e015f930 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.454+0000 7fa6d7400640 1 -- 172.21.15.114:0/2411366983 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d803e450 con 0x7fa6e015f930 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.454+0000 7fa6d7400640 1 -- 172.21.15.114:0/2411366983 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8009510 con 0x7fa6e015f930 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.454+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e0151740 con 0x7fa6e015f930 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.454+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e014ee30 con 0x7fa6e015f930 2024-03-06T07:38:40.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.455+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e004f330 con 0x7fa6e015f930 2024-03-06T07:38:40.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.456+0000 7fa6d7400640 1 -- 172.21.15.114:0/2411366983 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6d8046020 con 0x7fa6e015f930 2024-03-06T07:38:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.458+0000 7fa6d7400640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.458+0000 7fa6d7400640 1 -- 172.21.15.114:0/2411366983 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa6d80ac7e0 con 0x7fa6e015f930 2024-03-06T07:38:40.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.458+0000 7fa6df400640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:40.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.459+0000 7fa6df400640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0000f60 tx=0x7fa6d0007b90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:40.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.462+0000 7fa6d7400640 1 -- 172.21.15.114:0/2411366983 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6d804d030 con 0x7fa6e015f930 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.694+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6e00d0fa0 con 0x7fa6e015f930 2024-03-06T07:38:40.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.695+0000 7fa6d7400640 1 -- 172.21.15.114:0/2411366983 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa6d807b610 con 0x7fa6e015f930 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:40.698 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6bc06b4d0 msgr2=0x7fa6bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0000f60 tx=0x7fa6d0007b90 comp rx=0 tx=0).stop 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6e015f930 msgr2=0x7fa6e015ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6e015f930 0x7fa6e015ae10 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8036040 tx=0x7fa6d8004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 shutdown_connections 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa6bc06b4d0 0x7fa6bc06d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6e015f930 0x7fa6e015ae10 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa6e00ce8f0 0x7fa6e014e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 --2- 172.21.15.114:0/2411366983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6e0066f10 0x7fa6e0145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 >> 172.21.15.114:0/2411366983 conn(0x7fa6e00d0340 msgr2=0x7fa6e006b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 shutdown_connections 2024-03-06T07:38:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:40.697+0000 7fa6e6000640 1 -- 172.21.15.114:0/2411366983 wait complete. 2024-03-06T07:38:40.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:40.918 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:41.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:42.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.128+0000 7f0307200640 1 Processor -- start 2024-03-06T07:38:42.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.128+0000 7f0307200640 1 -- start start 2024-03-06T07:38:42.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.130+0000 7f0307200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f030815a4b0 0x7f0308071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.130+0000 7f0307200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f03080cbb60 0x7f03080b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.130+0000 7f0307200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030806c940 con 0x7f03080b42d0 2024-03-06T07:38:42.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.130+0000 7f0307200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03080ce680 con 0x7f03080cbb60 2024-03-06T07:38:42.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.130+0000 7f0307200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03080cec40 con 0x7f030815a4b0 2024-03-06T07:38:42.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.130+0000 7f0306800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f03080b42d0 0x7f03080cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39288/0 (socket says 172.21.15.114:39288) 2024-03-06T07:38:42.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.130+0000 7f0306800640 1 -- 172.21.15.114:0/4216144114 learned_addr learned my addr 172.21.15.114:0/4216144114 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305e00640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f030815a4b0 0x7f0308071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f02fd400640 1 --2- 172.21.15.114:0/4216144114 >> v2:172.21.15.177:3300/0 conn(0x7f03080cbb60 0x7f03080b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938029880 0 0) 0x7f0300001490 con 0x7f03080b42d0 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec003520 con 0x7f03080b42d0 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4833509 0 0) 0x7f02ec003520 con 0x7f03080b42d0 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f030815a4b0 msgr2=0x7f0308071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f030815a4b0 0x7f0308071380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 >> v2:172.21.15.177:3300/0 conn(0x7f03080cbb60 msgr2=0x7f03080b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 --2- 172.21.15.114:0/4216144114 >> v2:172.21.15.177:3300/0 conn(0x7f03080cbb60 0x7f03080b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0308071980 con 0x7f03080b42d0 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f0300001bc0 con 0x7f03080b42d0 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 >> v1:172.21.15.114:6789/0 conn(0x7f03080b42d0 legacy=0x7f03080cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02ec0039b0 0x7f02ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02ec004800 0x7f02ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f030815a4b0 0x7f02ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03080ce680 con 0x7f02ec004800 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03080cec40 con 0x7f02ec0039b0 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.131+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec00dbd0 con 0x7f030815a4b0 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.132+0000 7f02fd400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02ec004800 0x7f02ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.132+0000 7f02fd400640 1 -- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f030815a4b0 msgr2=0x7f02ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.132+0000 7f02fd400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f030815a4b0 0x7f02ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.132+0000 7f02fd400640 1 -- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02ec0039b0 msgr2=0x7f02ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.132+0000 7f02fd400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02ec0039b0 0x7f02ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.132+0000 7f02fd400640 1 -- 172.21.15.114:0/4216144114 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec003520 con 0x7f02ec004800 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.133+0000 7f02fd400640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02ec004800 0x7f02ec004c80 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009af0 tx=0x7f02ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d96e7300612971 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.133+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0039070 con 0x7f02ec004800 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.133+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f002c710 con 0x7f02ec004800 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.133+0000 7f0305400640 1 -- 172.21.15.114:0/4216144114 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0009690 con 0x7f02ec004800 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.134+0000 7f0307200640 1 -- 172.21.15.114:0/4216144114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02ec004800 msgr2=0x7f02ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.134+0000 7f0307200640 1 --2- 172.21.15.114:0/4216144114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02ec004800 0x7f02ec004c80 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009af0 tx=0x7f02ec005830 comp rx=0 tx=0).stop 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.134+0000 7f0306800640 1 -- 172.21.15.114:0/4216144114 reap_dead start 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.134+0000 7f0307200640 1 -- 172.21.15.114:0/4216144114 shutdown_connections 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.134+0000 7f0307200640 1 -- 172.21.15.114:0/4216144114 >> 172.21.15.114:0/4216144114 conn(0x7f030806afc0 msgr2=0x7f030806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.134+0000 7f0307200640 1 -- 172.21.15.114:0/4216144114 shutdown_connections 2024-03-06T07:38:42.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.134+0000 7f0307200640 1 -- 172.21.15.114:0/4216144114 wait complete. 2024-03-06T07:38:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.136+0000 7f0307200640 1 Processor -- start 2024-03-06T07:38:42.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.136+0000 7f0307200640 1 -- start start 2024-03-06T07:38:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0307200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02ec004800 0x7f030813f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0307200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03081eedd0 0x7f03081544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0307200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03081549f0 0x7f03080cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0307200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03080b4760 con 0x7f03081549f0 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0307200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03080ce8d0 con 0x7f03081eedd0 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0307200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03080d00e0 con 0x7f02ec004800 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0306800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03081549f0 0x7f03080cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0306800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03081549f0 0x7f03080cefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60590/0 (socket says 172.21.15.114:60590) 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.137+0000 7f0306800640 1 -- 172.21.15.114:0/742133992 learned_addr learned my addr 172.21.15.114:0/742133992 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0305e00640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02ec004800 0x7f030813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f02fd400640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03081eedd0 0x7f03081544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0306800640 1 -- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02ec004800 msgr2=0x7f030813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0306800640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02ec004800 0x7f030813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0306800640 1 -- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03081eedd0 msgr2=0x7f03081544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0306800640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03081eedd0 0x7f03081544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0306800640 1 -- 172.21.15.114:0/742133992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030813ebd0 con 0x7f03081549f0 2024-03-06T07:38:42.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0306800640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03081549f0 0x7f03080cefd0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f0308068400 tx=0x7f030000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f02fea00640 1 -- 172.21.15.114:0/742133992 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300007110 con 0x7f03081549f0 2024-03-06T07:38:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f02fea00640 1 -- 172.21.15.114:0/742133992 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03000072b0 con 0x7f03081549f0 2024-03-06T07:38:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f02fea00640 1 -- 172.21.15.114:0/742133992 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03000076a0 con 0x7f03081549f0 2024-03-06T07:38:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.138+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0308142de0 con 0x7f03081549f0 2024-03-06T07:38:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.139+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030813ee70 con 0x7f03081549f0 2024-03-06T07:38:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.140+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030804f330 con 0x7f03081549f0 2024-03-06T07:38:42.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.141+0000 7f02fea00640 1 -- 172.21.15.114:0/742133992 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0300007840 con 0x7f03081549f0 2024-03-06T07:38:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.142+0000 7f02fea00640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02e406b500 0x7f02e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.142+0000 7f02fea00640 1 -- 172.21.15.114:0/742133992 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f030008a0b0 con 0x7f03081549f0 2024-03-06T07:38:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.142+0000 7f0305e00640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02e406b500 0x7f02e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.143+0000 7f0305e00640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02e406b500 0x7f02e406d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f02f80047b0 tx=0x7f02f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.146+0000 7f02fea00640 1 -- 172.21.15.114:0/742133992 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f030806c940 con 0x7f03081549f0 2024-03-06T07:38:42.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.383+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0308155300 con 0x7f03081549f0 2024-03-06T07:38:42.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.384+0000 7f02fea00640 1 -- 172.21.15.114:0/742133992 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f0300058ee0 con 0x7f03081549f0 2024-03-06T07:38:42.386 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:42.386 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:42.386 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02e406b500 msgr2=0x7f02e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02e406b500 0x7f02e406d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f02f80047b0 tx=0x7f02f8008040 comp rx=0 tx=0).stop 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03081549f0 msgr2=0x7f03080cefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03081549f0 0x7f03080cefd0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f0308068400 tx=0x7f030000cf00 comp rx=0 tx=0).stop 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 shutdown_connections 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f02e406b500 0x7f02e406d9c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03081549f0 0x7f03080cefd0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03081eedd0 0x7f03081544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 --2- 172.21.15.114:0/742133992 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f02ec004800 0x7f030813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 >> 172.21.15.114:0/742133992 conn(0x7f030806afc0 msgr2=0x7f0308155ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 shutdown_connections 2024-03-06T07:38:42.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:42.387+0000 7f0307200640 1 -- 172.21.15.114:0/742133992 wait complete. 2024-03-06T07:38:42.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:42.612 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:43.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.861+0000 7fd995200640 1 Processor -- start 2024-03-06T07:38:43.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.861+0000 7fd995200640 1 -- start start 2024-03-06T07:38:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.862+0000 7fd995200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd9900aedc0 0x7fd9900cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:43.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.862+0000 7fd995200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd9900cc800 0x7fd99015f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.862+0000 7fd995200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99015c240 con 0x7fd9901561f0 2024-03-06T07:38:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.862+0000 7fd995200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99015ce30 con 0x7fd9900aedc0 2024-03-06T07:38:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.862+0000 7fd995200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99015c500 con 0x7fd9900cc800 2024-03-06T07:38:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd9900aedc0 0x7fd9900cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98ea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd9900aedc0 0x7fd9900cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60592/0 (socket says 172.21.15.114:60592) 2024-03-06T07:38:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98ea00640 1 -- 172.21.15.114:0/2640937038 learned_addr learned my addr 172.21.15.114:0/2640937038 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98ea00640 1 -- 172.21.15.114:0/2640937038 >> v2:172.21.15.177:3300/0 conn(0x7fd9900cc800 msgr2=0x7fd99015f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98fe00640 1 --2- 172.21.15.114:0/2640937038 >> v2:172.21.15.177:3300/0 conn(0x7fd9900cc800 0x7fd99015f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98ea00640 1 --2- 172.21.15.114:0/2640937038 >> v2:172.21.15.177:3300/0 conn(0x7fd9900cc800 0x7fd99015f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98ea00640 1 -- 172.21.15.114:0/2640937038 >> v1:172.21.15.177:6789/0 conn(0x7fd9901561f0 legacy=0x7fd9900ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:43.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.863+0000 7fd98ea00640 1 -- 172.21.15.114:0/2640937038 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99015c7e0 con 0x7fd9900aedc0 2024-03-06T07:38:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.865+0000 7fd98ea00640 1 --2- 172.21.15.114:0/2640937038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd9900cc2c0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fd9900cc1f0 tx=0x7fd97802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779af2db5d1248e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.865+0000 7fd98e000640 1 -- 172.21.15.114:0/2640937038 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978035020 con 0x7fd9900aedc0 2024-03-06T07:38:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.865+0000 7fd98e000640 1 -- 172.21.15.114:0/2640937038 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd978031870 con 0x7fd9900aedc0 2024-03-06T07:38:43.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.865+0000 7fd98e000640 1 -- 172.21.15.114:0/2640937038 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd978031c40 con 0x7fd9900aedc0 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 -- 172.21.15.114:0/2640937038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 msgr2=0x7fd9900cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 --2- 172.21.15.114:0/2640937038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd9900cc2c0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fd9900cc1f0 tx=0x7fd97802b910 comp rx=0 tx=0).stop 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 -- 172.21.15.114:0/2640937038 shutdown_connections 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 --2- 172.21.15.114:0/2640937038 >> v2:172.21.15.177:3300/0 conn(0x7fd9900cc800 0x7fd99015f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 --2- 172.21.15.114:0/2640937038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd9900cc2c0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 -- 172.21.15.114:0/2640937038 >> 172.21.15.114:0/2640937038 conn(0x7fd99006afc0 msgr2=0x7fd99006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 -- 172.21.15.114:0/2640937038 shutdown_connections 2024-03-06T07:38:43.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.866+0000 7fd995200640 1 -- 172.21.15.114:0/2640937038 wait complete. 2024-03-06T07:38:43.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.868+0000 7fd995200640 1 Processor -- start 2024-03-06T07:38:43.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.868+0000 7fd995200640 1 -- start start 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd995200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd990159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd995200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9900cc800 0x7fd990157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd995200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9901561f0 0x7fd9901474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd995200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9901565d0 con 0x7fd9900aedc0 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd995200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9900ccc30 con 0x7fd9900cc800 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd995200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd990156750 con 0x7fd9901561f0 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd98f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd990159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd98f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd990159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60600/0 (socket says 172.21.15.114:60600) 2024-03-06T07:38:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.869+0000 7fd98f400640 1 -- 172.21.15.114:0/4211547826 learned_addr learned my addr 172.21.15.114:0/4211547826 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd98f400640 1 -- 172.21.15.114:0/4211547826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9901561f0 msgr2=0x7fd9901474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd98f400640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9901561f0 0x7fd9901474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd98f400640 1 -- 172.21.15.114:0/4211547826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9900cc800 msgr2=0x7fd990157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd98f400640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9900cc800 0x7fd990157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd98f400640 1 -- 172.21.15.114:0/4211547826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd990157d50 con 0x7fd9900aedc0 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd98f400640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd990159700 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd98000b3f0 tx=0x7fd98000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd987400640 1 -- 172.21.15.114:0/4211547826 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980015020 con 0x7fd9900aedc0 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd987400640 1 -- 172.21.15.114:0/4211547826 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd980011830 con 0x7fd9900aedc0 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd990157ff0 con 0x7fd9900aedc0 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.870+0000 7fd987400640 1 -- 172.21.15.114:0/4211547826 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd980011c20 con 0x7fd9900aedc0 2024-03-06T07:38:43.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.871+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9901524d0 con 0x7fd9900aedc0 2024-03-06T07:38:43.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.872+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99004f330 con 0x7fd9900aedc0 2024-03-06T07:38:43.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.873+0000 7fd987400640 1 -- 172.21.15.114:0/4211547826 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd980018020 con 0x7fd9900aedc0 2024-03-06T07:38:43.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.874+0000 7fd987400640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd96c06b4d0 0x7fd96c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:43.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.874+0000 7fd987400640 1 -- 172.21.15.114:0/4211547826 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd980089f00 con 0x7fd9900aedc0 2024-03-06T07:38:43.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.874+0000 7fd98ea00640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd96c06b4d0 0x7fd96c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:43.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.875+0000 7fd98ea00640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd96c06b4d0 0x7fd96c06d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd978036000 tx=0x7fd978030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:43.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:43.879+0000 7fd987400640 1 -- 172.21.15.114:0/4211547826 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd98001f030 con 0x7fd9900aedc0 2024-03-06T07:38:44.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.131+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9900b2b10 con 0x7fd9900aedc0 2024-03-06T07:38:44.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.131+0000 7fd987400640 1 -- 172.21.15.114:0/4211547826 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd9800172c0 con 0x7fd9900aedc0 2024-03-06T07:38:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:44.133 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:44.134 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd96c06b4d0 msgr2=0x7fd96c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd96c06b4d0 0x7fd96c06d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd978036000 tx=0x7fd978030000 comp rx=0 tx=0).stop 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 msgr2=0x7fd990159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd990159700 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd98000b3f0 tx=0x7fd98000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 shutdown_connections 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd9901561f0 0x7fd9901474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9900cc800 0x7fd990157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd96c06b4d0 0x7fd96c06d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 --2- 172.21.15.114:0/4211547826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9900aedc0 0x7fd990159700 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.135+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 >> 172.21.15.114:0/4211547826 conn(0x7fd99006afc0 msgr2=0x7fd99006d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.136+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 shutdown_connections 2024-03-06T07:38:44.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:44.136+0000 7fd995200640 1 -- 172.21.15.114:0/4211547826 wait complete. 2024-03-06T07:38:44.144 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:44.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:44.358 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:45.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:45.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.572+0000 7fc1fa600640 1 Processor -- start 2024-03-06T07:38:45.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.572+0000 7fc1fa600640 1 -- start start 2024-03-06T07:38:45.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.573+0000 7fc1fa600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f4144660 0x7fc1f4071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.573+0000 7fc1fa600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc1f40cf3e0 0x7fc1f40724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.573+0000 7fc1fa600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f40cabf0 con 0x7fc1f40c9ec0 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.573+0000 7fc1fa600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f40ce900 con 0x7fc1f40cf3e0 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.573+0000 7fc1fa600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40cebe0 con 0x7fc1f4144660 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.574+0000 7fc1f3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc1f40c9ec0 0x7fc1f40ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39298/0 (socket says 172.21.15.114:39298) 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.574+0000 7fc1f3e00640 1 -- 172.21.15.114:0/2315565021 learned_addr learned my addr 172.21.15.114:0/2315565021 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.574+0000 7fc1f2a00640 1 --2- 172.21.15.114:0/2315565021 >> v2:172.21.15.114:3300/0 conn(0x7fc1f40cf3e0 0x7fc1f40724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.574+0000 7fc1f3400640 1 --2- 172.21.15.114:0/2315565021 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f4144660 0x7fc1f4071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.574+0000 7fc1f2000640 1 -- 172.21.15.114:0/2315565021 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752751868 0 0) 0x7fc1f40cabf0 con 0x7fc1f40c9ec0 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.574+0000 7fc1f2000640 1 -- 172.21.15.114:0/2315565021 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d8003520 con 0x7fc1f40c9ec0 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.574+0000 7fc1f2a00640 1 -- 172.21.15.114:0/2315565021 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f4144660 msgr2=0x7fc1f4071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.575+0000 7fc1f2a00640 1 --2- 172.21.15.114:0/2315565021 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f4144660 0x7fc1f4071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.575+0000 7fc1f2a00640 1 -- 172.21.15.114:0/2315565021 >> v1:172.21.15.114:6789/0 conn(0x7fc1f40c9ec0 legacy=0x7fc1f40ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.575+0000 7fc1f2a00640 1 -- 172.21.15.114:0/2315565021 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f40ceec0 con 0x7fc1f40cf3e0 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.576+0000 7fc1f2a00640 1 --2- 172.21.15.114:0/2315565021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f40cf3e0 0x7fc1f40724b0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00097a0 tx=0x7fc1e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46971c57bb5a96d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.576+0000 7fc1f2000640 1 -- 172.21.15.114:0/2315565021 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0035020 con 0x7fc1f40cf3e0 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.576+0000 7fc1f2000640 1 -- 172.21.15.114:0/2315565021 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e0031830 con 0x7fc1f40cf3e0 2024-03-06T07:38:45.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.576+0000 7fc1f2000640 1 -- 172.21.15.114:0/2315565021 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0031c00 con 0x7fc1f40cf3e0 2024-03-06T07:38:45.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 -- 172.21.15.114:0/2315565021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f40cf3e0 msgr2=0x7fc1f40724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 --2- 172.21.15.114:0/2315565021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f40cf3e0 0x7fc1f40724b0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00097a0 tx=0x7fc1e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 -- 172.21.15.114:0/2315565021 shutdown_connections 2024-03-06T07:38:45.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 --2- 172.21.15.114:0/2315565021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f40cf3e0 0x7fc1f40724b0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 --2- 172.21.15.114:0/2315565021 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f4144660 0x7fc1f4071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 -- 172.21.15.114:0/2315565021 >> 172.21.15.114:0/2315565021 conn(0x7fc1f40d0b90 msgr2=0x7fc1f40d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:45.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 -- 172.21.15.114:0/2315565021 shutdown_connections 2024-03-06T07:38:45.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.577+0000 7fc1fa600640 1 -- 172.21.15.114:0/2315565021 wait complete. 2024-03-06T07:38:45.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.580+0000 7fc1fa600640 1 Processor -- start 2024-03-06T07:38:45.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.580+0000 7fc1fa600640 1 -- start start 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.581+0000 7fc1fa600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f40c9ec0 0x7fc1f40aff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.581+0000 7fc1fa600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f40cf3e0 0x7fc1f414e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.581+0000 7fc1fa600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4144660 0x7fc1f40cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.581+0000 7fc1fa600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40cc270 con 0x7fc1f4144660 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.581+0000 7fc1fa600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40cc0f0 con 0x7fc1f40cf3e0 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.581+0000 7fc1fa600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40cc3f0 con 0x7fc1f40c9ec0 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4144660 0x7fc1f40cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4144660 0x7fc1f40cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60610/0 (socket says 172.21.15.114:60610) 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 -- 172.21.15.114:0/3150394228 learned_addr learned my addr 172.21.15.114:0/3150394228 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 -- 172.21.15.114:0/3150394228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f40c9ec0 msgr2=0x7fc1f40aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f40c9ec0 0x7fc1f40aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 -- 172.21.15.114:0/3150394228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f40cf3e0 msgr2=0x7fc1f414e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f2a00640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f40cf3e0 0x7fc1f414e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f40cf3e0 0x7fc1f414e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 -- 172.21.15.114:0/3150394228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f414ef40 con 0x7fc1f4144660 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.582+0000 7fc1f3e00640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4144660 0x7fc1f40cd210 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00cb70 tx=0x7fc1ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.583+0000 7fc1eb400640 1 -- 172.21.15.114:0/3150394228 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec017070 con 0x7fc1f4144660 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.583+0000 7fc1eb400640 1 -- 172.21.15.114:0/3150394228 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec00ad40 con 0x7fc1f4144660 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.583+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f414f220 con 0x7fc1f4144660 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.583+0000 7fc1eb400640 1 -- 172.21.15.114:0/3150394228 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec012620 con 0x7fc1f4144660 2024-03-06T07:38:45.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.583+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f414c900 con 0x7fc1f4144660 2024-03-06T07:38:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.584+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f404f330 con 0x7fc1f4144660 2024-03-06T07:38:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.586+0000 7fc1eb400640 1 -- 172.21.15.114:0/3150394228 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1ec01a070 con 0x7fc1f4144660 2024-03-06T07:38:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.587+0000 7fc1eb400640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1d406b4d0 0x7fc1d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:45.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.587+0000 7fc1eb400640 1 -- 172.21.15.114:0/3150394228 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc1ec08a0d0 con 0x7fc1f4144660 2024-03-06T07:38:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.587+0000 7fc1f3400640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1d406b4d0 0x7fc1d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:45.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.588+0000 7fc1f3400640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1d406b4d0 0x7fc1d406d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4006a90 tx=0x7fc1e4006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:45.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.592+0000 7fc1eb400640 1 -- 172.21.15.114:0/3150394228 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1ec058f00 con 0x7fc1f4144660 2024-03-06T07:38:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.824+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f4070560 con 0x7fc1f4144660 2024-03-06T07:38:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.825+0000 7fc1eb400640 1 -- 172.21.15.114:0/3150394228 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fc1ec058800 con 0x7fc1f4144660 2024-03-06T07:38:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:45.827 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:45.827 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:45.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.828+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1d406b4d0 msgr2=0x7fc1d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.828+0000 7fc1fa600640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1d406b4d0 0x7fc1d406d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4006a90 tx=0x7fc1e4006a20 comp rx=0 tx=0).stop 2024-03-06T07:38:45.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.828+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4144660 msgr2=0x7fc1f40cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.828+0000 7fc1fa600640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4144660 0x7fc1f40cd210 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00cb70 tx=0x7fc1ec00a430 comp rx=0 tx=0).stop 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 shutdown_connections 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc1d406b4d0 0x7fc1d406d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4144660 0x7fc1f40cd210 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f40cf3e0 0x7fc1f414e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 --2- 172.21.15.114:0/3150394228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1f40c9ec0 0x7fc1f40aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 >> 172.21.15.114:0/3150394228 conn(0x7fc1f40d0b90 msgr2=0x7fc1f40b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 shutdown_connections 2024-03-06T07:38:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:45.829+0000 7fc1fa600640 1 -- 172.21.15.114:0/3150394228 wait complete. 2024-03-06T07:38:46.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:46.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:46.146 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:47.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:47.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.263+0000 7fbb21a00640 1 Processor -- start 2024-03-06T07:38:47.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.263+0000 7fbb21a00640 1 -- start start 2024-03-06T07:38:47.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.264+0000 7fbb21a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 0x7fbb1c0d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.264+0000 7fbb21a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1c161960 con 0x7fbb1c0b3ec0 2024-03-06T07:38:47.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.264+0000 7fbb21a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1c0cf6f0 con 0x7fbb1c161580 2024-03-06T07:38:47.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.264+0000 7fbb21a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0d2ce0 con 0x7fbb1c0d2360 2024-03-06T07:38:47.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.265+0000 7fbb21000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbb1c0b3ec0 0x7fbb1c0b4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:39312/0 (socket says 172.21.15.114:39312) 2024-03-06T07:38:47.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.265+0000 7fbb21000640 1 -- 172.21.15.114:0/2389260093 learned_addr learned my addr 172.21.15.114:0/2389260093 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:47.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.265+0000 7fbb1b400640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 0x7fbb1c0d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.265+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099466204 0 0) 0x7fbb1c161960 con 0x7fbb1c0b3ec0 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.265+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb00003520 con 0x7fbb1c0b3ec0 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003392470 0 0) 0x7fbb00003520 con 0x7fbb1c0b3ec0 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 msgr2=0x7fbb1c0d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 0x7fbb1c0d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 >> v1:172.21.15.177:6789/0 conn(0x7fbb1c161580 legacy=0x7fbb1c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c0cf9d0 con 0x7fbb1c0b3ec0 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fbb14001670 con 0x7fbb1c0b3ec0 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 >> v1:172.21.15.114:6789/0 conn(0x7fbb1c0b3ec0 legacy=0x7fbb1c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 0x7fbb000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb00004d40 0x7fbb000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0d2ce0 con 0x7fbb00003fb0 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0000bb00 con 0x7fbb00004d40 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0000bd00 con 0x7fbb1c0d2360 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1b400640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb21000640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb00004d40 0x7fbb000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1b400640 1 -- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 msgr2=0x7fbb000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:47.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1be00640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 0x7fbb000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.266+0000 7fbb1b400640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c0d2360 0x7fbb000039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.267+0000 7fbb1b400640 1 -- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb00004d40 msgr2=0x7fbb000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:47.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.267+0000 7fbb1b400640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb00004d40 0x7fbb000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.267+0000 7fbb1b400640 1 -- 172.21.15.114:0/2389260093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb00003520 con 0x7fbb00003fb0 2024-03-06T07:38:47.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.267+0000 7fbb21000640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb00004d40 0x7fbb000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:47.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb1b400640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb00004800 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009ac0 tx=0x7fbb00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e1f91c81cd4670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08039070 con 0x7fbb00003fb0 2024-03-06T07:38:47.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0802c600 con 0x7fbb00003fb0 2024-03-06T07:38:47.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb1aa00640 1 -- 172.21.15.114:0/2389260093 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08009690 con 0x7fbb00003fb0 2024-03-06T07:38:47.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb21a00640 1 -- 172.21.15.114:0/2389260093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 msgr2=0x7fbb00004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb21a00640 1 --2- 172.21.15.114:0/2389260093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb00004800 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009ac0 tx=0x7fbb00005830 comp rx=0 tx=0).stop 2024-03-06T07:38:47.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb21000640 1 -- 172.21.15.114:0/2389260093 reap_dead start 2024-03-06T07:38:47.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb21a00640 1 -- 172.21.15.114:0/2389260093 shutdown_connections 2024-03-06T07:38:47.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.268+0000 7fbb21a00640 1 -- 172.21.15.114:0/2389260093 >> 172.21.15.114:0/2389260093 conn(0x7fbb1c0d3d80 msgr2=0x7fbb1c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:47.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.269+0000 7fbb21a00640 1 -- 172.21.15.114:0/2389260093 shutdown_connections 2024-03-06T07:38:47.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.269+0000 7fbb21a00640 1 -- 172.21.15.114:0/2389260093 wait complete. 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.270+0000 7fbb21a00640 1 Processor -- start 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.270+0000 7fbb21a00640 1 -- start start 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.271+0000 7fbb21a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb1c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.271+0000 7fbb21a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c1534c0 0x7fbb1c0b7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.271+0000 7fbb21a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb1c153940 0x7fbb1c0b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.271+0000 7fbb21a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0b39a0 con 0x7fbb00003fb0 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.271+0000 7fbb21a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0b36c0 con 0x7fbb1c153940 2024-03-06T07:38:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.271+0000 7fbb21a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0cf3a0 con 0x7fbb1c1534c0 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb1c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb1c0c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60624/0 (socket says 172.21.15.114:60624) 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 -- 172.21.15.114:0/4035293877 learned_addr learned my addr 172.21.15.114:0/4035293877 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1b400640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c1534c0 0x7fbb1c0b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb21000640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb1c153940 0x7fbb1c0b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 -- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c1534c0 msgr2=0x7fbb1c0b7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c1534c0 0x7fbb1c0b7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 -- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb1c153940 msgr2=0x7fbb1c0b7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb1c153940 0x7fbb1c0b7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 -- 172.21.15.114:0/4035293877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c0ce1e0 con 0x7fbb00003fb0 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb21000640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb1c153940 0x7fbb1c0b7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.272+0000 7fbb1be00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb1c0c7410 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00d860 tx=0x7fbb0c00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.273+0000 7fbb13e00640 1 -- 172.21.15.114:0/4035293877 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c003da0 con 0x7fbb00003fb0 2024-03-06T07:38:47.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.273+0000 7fbb13e00640 1 -- 172.21.15.114:0/4035293877 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c017070 con 0x7fbb00003fb0 2024-03-06T07:38:47.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.273+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c0cdc60 con 0x7fbb00003fb0 2024-03-06T07:38:47.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.273+0000 7fbb13e00640 1 -- 172.21.15.114:0/4035293877 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00b8f0 con 0x7fbb00003fb0 2024-03-06T07:38:47.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.273+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c0cb4e0 con 0x7fbb00003fb0 2024-03-06T07:38:47.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.274+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c04f330 con 0x7fbb00003fb0 2024-03-06T07:38:47.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.275+0000 7fbb13e00640 1 -- 172.21.15.114:0/4035293877 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb0c00ba90 con 0x7fbb00003fb0 2024-03-06T07:38:47.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.276+0000 7fbb13e00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbafc06b4d0 0x7fbafc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:47.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.276+0000 7fbb13e00640 1 -- 172.21.15.114:0/4035293877 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbb0c08a3a0 con 0x7fbb00003fb0 2024-03-06T07:38:47.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.277+0000 7fbb1b400640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbafc06b4d0 0x7fbafc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:47.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.277+0000 7fbb1b400640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbafc06b4d0 0x7fbafc06d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fbb08036000 tx=0x7fbb08030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:47.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.280+0000 7fbb13e00640 1 -- 172.21.15.114:0/4035293877 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb0c0591d0 con 0x7fbb00003fb0 2024-03-06T07:38:47.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.623+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb1c0d3b60 con 0x7fbb00003fb0 2024-03-06T07:38:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.624+0000 7fbb13e00640 1 -- 172.21.15.114:0/4035293877 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fbb0c058ad0 con 0x7fbb00003fb0 2024-03-06T07:38:47.626 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:47.626 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbafc06b4d0 msgr2=0x7fbafc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbafc06b4d0 0x7fbafc06d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fbb08036000 tx=0x7fbb08030000 comp rx=0 tx=0).stop 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 msgr2=0x7fbb1c0c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb1c0c7410 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00d860 tx=0x7fbb0c00dd20 comp rx=0 tx=0).stop 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 shutdown_connections 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbb1c153940 0x7fbb1c0b7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbb1c1534c0 0x7fbb1c0b7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbafc06b4d0 0x7fbafc06d990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 --2- 172.21.15.114:0/4035293877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb00003fb0 0x7fbb1c0c7410 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 >> 172.21.15.114:0/4035293877 conn(0x7fbb1c0d3d80 msgr2=0x7fbb1c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:47.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 shutdown_connections 2024-03-06T07:38:47.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:47.627+0000 7fbb21a00640 1 -- 172.21.15.114:0/4035293877 wait complete. 2024-03-06T07:38:47.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:47.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:47.854 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:47.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:48.148 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:48.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.063+0000 7f7891c00640 1 Processor -- start 2024-03-06T07:38:49.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.064+0000 7f7891c00640 1 -- start start 2024-03-06T07:38:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.065+0000 7f7891c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f788c15e0d0 0x7f788c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.065+0000 7f7891c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c0cea60 con 0x7f788c15f470 2024-03-06T07:38:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.065+0000 7f7891c00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c0d2350 con 0x7f788c0d0fc0 2024-03-06T07:38:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.065+0000 7f7891c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c0d2f60 con 0x7f788c15e0d0 2024-03-06T07:38:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.065+0000 7f788a000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f788c15f470 0x7f788c0d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51050/0 (socket says 172.21.15.114:51050) 2024-03-06T07:38:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.065+0000 7f788a000640 1 -- 172.21.15.114:0/3233869506 learned_addr learned my addr 172.21.15.114:0/3233869506 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.065+0000 7f788aa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f788c15e0d0 0x7f788c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.066+0000 7f788aa00640 1 -- 172.21.15.114:0/3233869506 >> v1:172.21.15.177:6789/0 conn(0x7f788c0d0fc0 legacy=0x7f788c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:49.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.066+0000 7f788aa00640 1 -- 172.21.15.114:0/3233869506 >> v1:172.21.15.114:6789/0 conn(0x7f788c15f470 legacy=0x7f788c0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.066+0000 7f788aa00640 1 -- 172.21.15.114:0/3233869506 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0d2630 con 0x7f788c15e0d0 2024-03-06T07:38:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.067+0000 7f788aa00640 1 --2- 172.21.15.114:0/3233869506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15e0d0 0x7f788c0d1a90 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f788c0b6a80 tx=0x7f787c02bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c8005fc402aac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.067+0000 7f7889600640 1 -- 172.21.15.114:0/3233869506 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c039070 con 0x7f788c15e0d0 2024-03-06T07:38:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.067+0000 7f7889600640 1 -- 172.21.15.114:0/3233869506 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c02ccc0 con 0x7f788c15e0d0 2024-03-06T07:38:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.067+0000 7f7889600640 1 -- 172.21.15.114:0/3233869506 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c034660 con 0x7f788c15e0d0 2024-03-06T07:38:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.068+0000 7f7891c00640 1 -- 172.21.15.114:0/3233869506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15e0d0 msgr2=0x7f788c0d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.068+0000 7f7891c00640 1 --2- 172.21.15.114:0/3233869506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15e0d0 0x7f788c0d1a90 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f788c0b6a80 tx=0x7f787c02bf10 comp rx=0 tx=0).stop 2024-03-06T07:38:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.068+0000 7f7891c00640 1 -- 172.21.15.114:0/3233869506 shutdown_connections 2024-03-06T07:38:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.068+0000 7f7891c00640 1 --2- 172.21.15.114:0/3233869506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15e0d0 0x7f788c0d1a90 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.068+0000 7f7891c00640 1 -- 172.21.15.114:0/3233869506 >> 172.21.15.114:0/3233869506 conn(0x7f788c0d3ec0 msgr2=0x7f788c0d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.068+0000 7f7891c00640 1 -- 172.21.15.114:0/3233869506 shutdown_connections 2024-03-06T07:38:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.068+0000 7f7891c00640 1 -- 172.21.15.114:0/3233869506 wait complete. 2024-03-06T07:38:49.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.070+0000 7f7891c00640 1 Processor -- start 2024-03-06T07:38:49.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.070+0000 7f7891c00640 1 -- start start 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f7891c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f788c0d0fc0 0x7f788c0b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f7891c00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f788c15e0d0 0x7f788c152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f7891c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15f470 0x7f788c06bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f7891c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0cff10 con 0x7f788c15f470 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f7891c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0d2bf0 con 0x7f788c0d0fc0 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f7891c00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c15fb30 con 0x7f788c15e0d0 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f788b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15f470 0x7f788c06bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f788b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15f470 0x7f788c06bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33970/0 (socket says 172.21.15.114:33970) 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.071+0000 7f788b400640 1 -- 172.21.15.114:0/3663960151 learned_addr learned my addr 172.21.15.114:0/3663960151 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788aa00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f788c0d0fc0 0x7f788c0b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788a000640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f788c15e0d0 0x7f788c152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788b400640 1 -- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f788c15e0d0 msgr2=0x7f788c152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788b400640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f788c15e0d0 0x7f788c152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788b400640 1 -- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f788c0d0fc0 msgr2=0x7f788c0b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788b400640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f788c0d0fc0 0x7f788c0b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788b400640 1 -- 172.21.15.114:0/3663960151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c150bd0 con 0x7f788c15f470 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788a000640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f788c15e0d0 0x7f788c152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f788b400640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15f470 0x7f788c06bfc0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f788400b410 tx=0x7f788400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f7882a00640 1 -- 172.21.15.114:0/3663960151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884015030 con 0x7f788c15f470 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f7882a00640 1 -- 172.21.15.114:0/3663960151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7884004020 con 0x7f788c15f470 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c0b6f90 con 0x7f788c15f470 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.072+0000 7f7882a00640 1 -- 172.21.15.114:0/3663960151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884004410 con 0x7f788c15f470 2024-03-06T07:38:49.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.073+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c1f7d90 con 0x7f788c15f470 2024-03-06T07:38:49.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.074+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c1f8730 con 0x7f788c15f470 2024-03-06T07:38:49.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.075+0000 7f7882a00640 1 -- 172.21.15.114:0/3663960151 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7884018070 con 0x7f788c15f470 2024-03-06T07:38:49.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.076+0000 7f7882a00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f786806b500 0x7f786806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.076+0000 7f7882a00640 1 -- 172.21.15.114:0/3663960151 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7884089e00 con 0x7f788c15f470 2024-03-06T07:38:49.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.076+0000 7f788aa00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f786806b500 0x7f786806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.077+0000 7f788aa00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f786806b500 0x7f786806d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f787c036000 tx=0x7f787c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.080+0000 7f7882a00640 1 -- 172.21.15.114:0/3663960151 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f788c1f8730 con 0x7f788c15f470 2024-03-06T07:38:49.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.322+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f788c04f330 con 0x7f788c15f470 2024-03-06T07:38:49.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.322+0000 7f7882a00640 1 -- 172.21.15.114:0/3663960151 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f7884058b20 con 0x7f788c15f470 2024-03-06T07:38:49.324 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:49.325 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:49.325 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:49.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f786806b500 msgr2=0x7f786806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f786806b500 0x7f786806d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f787c036000 tx=0x7f787c030000 comp rx=0 tx=0).stop 2024-03-06T07:38:49.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15f470 msgr2=0x7f788c06bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15f470 0x7f788c06bfc0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f788400b410 tx=0x7f788400b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 shutdown_connections 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f786806b500 0x7f786806d9c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f788c15f470 0x7f788c06bfc0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f788c15e0d0 0x7f788c152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 --2- 172.21.15.114:0/3663960151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f788c0d0fc0 0x7f788c0b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 >> 172.21.15.114:0/3663960151 conn(0x7f788c0d3ec0 msgr2=0x7f788c06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 shutdown_connections 2024-03-06T07:38:49.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:49.325+0000 7f7891c00640 1 -- 172.21.15.114:0/3663960151 wait complete. 2024-03-06T07:38:49.547 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:49.547 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:50.150 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:50.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:50.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.759+0000 7f2d75400640 1 Processor -- start 2024-03-06T07:38:50.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.759+0000 7f2d75400640 1 -- start start 2024-03-06T07:38:50.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.761+0000 7f2d75400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2d7015fe60 0x7f2d70071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.761+0000 7f2d75400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7006cef0 con 0x7f2d7015f9d0 2024-03-06T07:38:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.761+0000 7f2d75400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7006d1d0 con 0x7f2d70144c00 2024-03-06T07:38:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.761+0000 7f2d75400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7006d4b0 con 0x7f2d7015fe60 2024-03-06T07:38:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.761+0000 7f2d6f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2d7015fe60 0x7f2d70071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.761+0000 7f2d6de00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2d7015f9d0 0x7f2d70070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51060/0 (socket says 172.21.15.114:51060) 2024-03-06T07:38:50.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.761+0000 7f2d6de00640 1 -- 172.21.15.114:0/1011283988 learned_addr learned my addr 172.21.15.114:0/1011283988 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.762+0000 7f2d6d400640 1 -- 172.21.15.114:0/1011283988 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958497540 0 0) 0x7f2d7006cef0 con 0x7f2d7015f9d0 2024-03-06T07:38:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.762+0000 7f2d6d400640 1 -- 172.21.15.114:0/1011283988 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54003520 con 0x7f2d7015f9d0 2024-03-06T07:38:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.762+0000 7f2d6f200640 1 -- 172.21.15.114:0/1011283988 >> v1:172.21.15.177:6789/0 conn(0x7f2d70144c00 legacy=0x7f2d7006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.762+0000 7f2d6f200640 1 -- 172.21.15.114:0/1011283988 >> v1:172.21.15.114:6789/0 conn(0x7f2d7015f9d0 legacy=0x7f2d70070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.762+0000 7f2d6f200640 1 -- 172.21.15.114:0/1011283988 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7006d790 con 0x7f2d7015fe60 2024-03-06T07:38:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.763+0000 7f2d6f200640 1 --2- 172.21.15.114:0/1011283988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7015fe60 0x7f2d70071220 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009d90 tx=0x7f2d6802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a2200299414ca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.763+0000 7f2d6d400640 1 -- 172.21.15.114:0/1011283988 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68039070 con 0x7f2d7015fe60 2024-03-06T07:38:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.763+0000 7f2d6d400640 1 -- 172.21.15.114:0/1011283988 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6802cd00 con 0x7f2d7015fe60 2024-03-06T07:38:50.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.763+0000 7f2d6d400640 1 -- 172.21.15.114:0/1011283988 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68034640 con 0x7f2d7015fe60 2024-03-06T07:38:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.764+0000 7f2d75400640 1 -- 172.21.15.114:0/1011283988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7015fe60 msgr2=0x7f2d70071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.764+0000 7f2d75400640 1 --2- 172.21.15.114:0/1011283988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7015fe60 0x7f2d70071220 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009d90 tx=0x7f2d6802bf50 comp rx=0 tx=0).stop 2024-03-06T07:38:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.764+0000 7f2d75400640 1 -- 172.21.15.114:0/1011283988 shutdown_connections 2024-03-06T07:38:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.764+0000 7f2d75400640 1 --2- 172.21.15.114:0/1011283988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7015fe60 0x7f2d70071220 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.764+0000 7f2d75400640 1 -- 172.21.15.114:0/1011283988 >> 172.21.15.114:0/1011283988 conn(0x7f2d700b1730 msgr2=0x7f2d700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.764+0000 7f2d75400640 1 -- 172.21.15.114:0/1011283988 shutdown_connections 2024-03-06T07:38:50.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.764+0000 7f2d75400640 1 -- 172.21.15.114:0/1011283988 wait complete. 2024-03-06T07:38:50.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.766+0000 7f2d75400640 1 Processor -- start 2024-03-06T07:38:50.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.766+0000 7f2d75400640 1 -- start start 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d75400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d70144c00 0x7f2d701e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d75400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7015f9d0 0x7f2d701e6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d75400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d7015fe60 0x7f2d70139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d75400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7006cb50 con 0x7f2d70144c00 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d75400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d70160730 con 0x7f2d7015f9d0 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d75400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7006ccd0 con 0x7f2d7015fe60 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d6e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d70144c00 0x7f2d701e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d6e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d70144c00 0x7f2d701e55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33976/0 (socket says 172.21.15.114:33976) 2024-03-06T07:38:50.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.767+0000 7f2d6e800640 1 -- 172.21.15.114:0/2779232241 learned_addr learned my addr 172.21.15.114:0/2779232241 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6f200640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d7015fe60 0x7f2d70139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6de00640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7015f9d0 0x7f2d701e6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6e800640 1 -- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d7015fe60 msgr2=0x7f2d70139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6e800640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d7015fe60 0x7f2d70139470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6e800640 1 -- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7015f9d0 msgr2=0x7f2d701e6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6e800640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7015f9d0 0x7f2d701e6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6e800640 1 -- 172.21.15.114:0/2779232241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7013b1f0 con 0x7f2d70144c00 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6f200640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d7015fe60 0x7f2d70139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6de00640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7015f9d0 0x7f2d701e6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d6e800640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d70144c00 0x7f2d701e55c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000c400 tx=0x7f2d6000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d66a00640 1 -- 172.21.15.114:0/2779232241 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60017070 con 0x7f2d70144c00 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d66a00640 1 -- 172.21.15.114:0/2779232241 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d600052d0 con 0x7f2d70144c00 2024-03-06T07:38:50.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d66a00640 1 -- 172.21.15.114:0/2779232241 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60003830 con 0x7f2d70144c00 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.768+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7013b4d0 con 0x7f2d70144c00 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.769+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7013ba90 con 0x7f2d70144c00 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.770+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7004f330 con 0x7f2d70144c00 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.771+0000 7f2d66a00640 1 -- 172.21.15.114:0/2779232241 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d6001a070 con 0x7f2d70144c00 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.771+0000 7f2d66a00640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2d5006b4d0 0x7f2d5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.772+0000 7f2d66a00640 1 -- 172.21.15.114:0/2779232241 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2d60089530 con 0x7f2d70144c00 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.772+0000 7f2d6de00640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2d5006b4d0 0x7f2d5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:50.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.773+0000 7f2d6de00640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2d5006b4d0 0x7f2d5006d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2d7014eae0 tx=0x7f2d5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:50.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:50.778+0000 7f2d66a00640 1 -- 172.21.15.114:0/2779232241 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d60058400 con 0x7f2d70144c00 2024-03-06T07:38:51.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.016+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d700cd3b0 con 0x7f2d70144c00 2024-03-06T07:38:51.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.017+0000 7f2d66a00640 1 -- 172.21.15.114:0/2779232241 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2d60057d00 con 0x7f2d70144c00 2024-03-06T07:38:51.019 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:51.019 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:51.020 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2d5006b4d0 msgr2=0x7f2d5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2d5006b4d0 0x7f2d5006d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2d7014eae0 tx=0x7f2d5c009040 comp rx=0 tx=0).stop 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d70144c00 msgr2=0x7f2d701e55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d70144c00 0x7f2d701e55c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000c400 tx=0x7f2d6000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 shutdown_connections 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f2d7015fe60 0x7f2d70139470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d7015f9d0 0x7f2d701e6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f2d5006b4d0 0x7f2d5006d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 --2- 172.21.15.114:0/2779232241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d70144c00 0x7f2d701e55c0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.020+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 >> 172.21.15.114:0/2779232241 conn(0x7f2d700b1730 msgr2=0x7f2d700b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.021+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 shutdown_connections 2024-03-06T07:38:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:51.021+0000 7f2d75400640 1 -- 172.21.15.114:0/2779232241 wait complete. 2024-03-06T07:38:51.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:51.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:52.152 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:52.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:52.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.445+0000 7f56afe00640 1 Processor -- start 2024-03-06T07:38:52.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.445+0000 7f56afe00640 1 -- start start 2024-03-06T07:38:52.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56afe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56a815fe60 0x7f56a8071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56afe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a806cef0 con 0x7f56a815f9d0 2024-03-06T07:38:52.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56afe00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a806d1d0 con 0x7f56a8144c00 2024-03-06T07:38:52.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56afe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a806d4b0 con 0x7f56a815fe60 2024-03-06T07:38:52.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56ade00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f56a815fe60 0x7f56a8071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56aca00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f56a815f9d0 0x7f56a8070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51072/0 (socket says 172.21.15.114:51072) 2024-03-06T07:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56aca00640 1 -- 172.21.15.114:0/3523460258 learned_addr learned my addr 172.21.15.114:0/3523460258 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56ade00640 1 -- 172.21.15.114:0/3523460258 >> v1:172.21.15.177:6789/0 conn(0x7f56a8144c00 legacy=0x7f56a806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56ade00640 1 -- 172.21.15.114:0/3523460258 >> v1:172.21.15.114:6789/0 conn(0x7f56a815f9d0 legacy=0x7f56a8070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.447+0000 7f56a7e00640 1 -- 172.21.15.114:0/3523460258 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742326754 0 0) 0x7f56a806cef0 con 0x7f56a815f9d0 2024-03-06T07:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.448+0000 7f56ade00640 1 -- 172.21.15.114:0/3523460258 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a806d790 con 0x7f56a815fe60 2024-03-06T07:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.449+0000 7f56ade00640 1 --2- 172.21.15.114:0/3523460258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a8071220 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009d90 tx=0x7f56a002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5b9f3b3ef99498 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.449+0000 7f56a7e00640 1 -- 172.21.15.114:0/3523460258 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0039070 con 0x7f56a815fe60 2024-03-06T07:38:52.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.449+0000 7f56a7e00640 1 -- 172.21.15.114:0/3523460258 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a002cc60 con 0x7f56a815fe60 2024-03-06T07:38:52.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.449+0000 7f56a7e00640 1 -- 172.21.15.114:0/3523460258 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0034660 con 0x7f56a815fe60 2024-03-06T07:38:52.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.449+0000 7f56afe00640 1 -- 172.21.15.114:0/3523460258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 msgr2=0x7f56a8071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.449+0000 7f56afe00640 1 --2- 172.21.15.114:0/3523460258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a8071220 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009d90 tx=0x7f56a002beb0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.450+0000 7f56afe00640 1 -- 172.21.15.114:0/3523460258 shutdown_connections 2024-03-06T07:38:52.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.450+0000 7f56afe00640 1 --2- 172.21.15.114:0/3523460258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a8071220 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.450+0000 7f56afe00640 1 -- 172.21.15.114:0/3523460258 >> 172.21.15.114:0/3523460258 conn(0x7f56a80b1730 msgr2=0x7f56a80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:52.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.450+0000 7f56afe00640 1 -- 172.21.15.114:0/3523460258 shutdown_connections 2024-03-06T07:38:52.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.450+0000 7f56afe00640 1 -- 172.21.15.114:0/3523460258 wait complete. 2024-03-06T07:38:52.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.452+0000 7f56afe00640 1 Processor -- start 2024-03-06T07:38:52.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.452+0000 7f56afe00640 1 -- start start 2024-03-06T07:38:52.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56afe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8144c00 0x7f56a815b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56afe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a815f9d0 0x7f56a80ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56afe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a80ac570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56afe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a806cb50 con 0x7f56a815fe60 2024-03-06T07:38:52.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56afe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a8160730 con 0x7f56a8144c00 2024-03-06T07:38:52.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56afe00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a806ccd0 con 0x7f56a815f9d0 2024-03-06T07:38:52.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56ade00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a80ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56ade00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a80ac570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33986/0 (socket says 172.21.15.114:33986) 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.454+0000 7f56ade00640 1 -- 172.21.15.114:0/1952084962 learned_addr learned my addr 172.21.15.114:0/1952084962 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ad400640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8144c00 0x7f56a815b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56aca00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a815f9d0 0x7f56a80ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ade00640 1 -- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a815f9d0 msgr2=0x7f56a80ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ade00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a815f9d0 0x7f56a80ac030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ade00640 1 -- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8144c00 msgr2=0x7f56a815b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ade00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8144c00 0x7f56a815b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ade00640 1 -- 172.21.15.114:0/1952084962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a815bbb0 con 0x7f56a815fe60 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ad400640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8144c00 0x7f56a815b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56ade00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a80ac570 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009eb0 tx=0x7f56a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56a5600640 1 -- 172.21.15.114:0/1952084962 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0044030 con 0x7f56a815fe60 2024-03-06T07:38:52.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56a5600640 1 -- 172.21.15.114:0/1952084962 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a002ca60 con 0x7f56a815fe60 2024-03-06T07:38:52.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56a5600640 1 -- 172.21.15.114:0/1952084962 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0039040 con 0x7f56a815fe60 2024-03-06T07:38:52.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.455+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a81e7470 con 0x7f56a815fe60 2024-03-06T07:38:52.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.456+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a80a98f0 con 0x7f56a815fe60 2024-03-06T07:38:52.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.457+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a80cd3e0 con 0x7f56a815fe60 2024-03-06T07:38:52.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.458+0000 7f56a5600640 1 -- 172.21.15.114:0/1952084962 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56a0047070 con 0x7f56a815fe60 2024-03-06T07:38:52.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.459+0000 7f56a5600640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:52.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.459+0000 7f56a5600640 1 -- 172.21.15.114:0/1952084962 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f56a00acca0 con 0x7f56a815fe60 2024-03-06T07:38:52.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.459+0000 7f56ad400640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:52.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.460+0000 7f56ad400640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f569c00a830 tx=0x7f569c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:52.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.463+0000 7f56a5600640 1 -- 172.21.15.114:0/1952084962 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56a80cd3e0 con 0x7f56a815fe60 2024-03-06T07:38:52.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.823+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56a804f330 con 0x7f56a815fe60 2024-03-06T07:38:52.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.824+0000 7f56a5600640 1 -- 172.21.15.114:0/1952084962 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f56a004e070 con 0x7f56a815fe60 2024-03-06T07:38:52.826 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 msgr2=0x7f568806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f569c00a830 tx=0x7f569c008040 comp rx=0 tx=0).stop 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 msgr2=0x7f56a80ac570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a80ac570 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009eb0 tx=0x7f56a0004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 shutdown_connections 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f568806b4d0 0x7f568806d990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56a815fe60 0x7f56a80ac570 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f56a815f9d0 0x7f56a80ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 --2- 172.21.15.114:0/1952084962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56a8144c00 0x7f56a815b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:52.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 >> 172.21.15.114:0/1952084962 conn(0x7f56a80b1730 msgr2=0x7f56a815ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:52.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 shutdown_connections 2024-03-06T07:38:52.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:52.827+0000 7f56afe00640 1 -- 172.21.15.114:0/1952084962 wait complete. 2024-03-06T07:38:53.055 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:53.055 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:54.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:54.154 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:54.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.267+0000 7fae66000640 1 Processor -- start 2024-03-06T07:38:54.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.268+0000 7fae66000640 1 -- start start 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae66000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6015f770 0x7fae6015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae66000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae600cf9a0 0x7fae600cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae66000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae600b3040 con 0x7fae600b4390 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae66000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae6015fb50 con 0x7fae600cf9a0 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae66000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae600cd3e0 con 0x7fae6015f770 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae56a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fae600b4390 0x7fae60160290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51084/0 (socket says 172.21.15.114:51084) 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae56a00640 1 -- 172.21.15.114:0/568353899 learned_addr learned my addr 172.21.15.114:0/568353899 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.269+0000 7fae5fe00640 1 --2- 172.21.15.114:0/568353899 >> v2:172.21.15.114:3300/0 conn(0x7fae600cf9a0 0x7fae600cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5f400640 1 --2- 172.21.15.114:0/568353899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6015f770 0x7fae6015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5ea00640 1 -- 172.21.15.114:0/568353899 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221765171 0 0) 0x7fae600b3040 con 0x7fae600b4390 2024-03-06T07:38:54.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5ea00640 1 -- 172.21.15.114:0/568353899 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae44003520 con 0x7fae600b4390 2024-03-06T07:38:54.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5fe00640 1 -- 172.21.15.114:0/568353899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6015f770 msgr2=0x7fae6015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:54.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5fe00640 1 --2- 172.21.15.114:0/568353899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6015f770 0x7fae6015fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5fe00640 1 -- 172.21.15.114:0/568353899 >> v1:172.21.15.114:6789/0 conn(0x7fae600b4390 legacy=0x7fae60160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5ea00640 1 -- 172.21.15.114:0/568353899 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261904837 0 0) 0x7fae44003520 con 0x7fae600b4390 2024-03-06T07:38:54.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.270+0000 7fae5fe00640 1 -- 172.21.15.114:0/568353899 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae600cd6c0 con 0x7fae600cf9a0 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.271+0000 7fae5fe00640 1 --2- 172.21.15.114:0/568353899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae600cf9a0 0x7fae600cfe30 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fae580097a0 tx=0x7fae5802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56faaad569e20208 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae5ea00640 1 -- 172.21.15.114:0/568353899 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58037070 con 0x7fae600cf9a0 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae5ea00640 1 -- 172.21.15.114:0/568353899 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae58030510 con 0x7fae600cf9a0 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae5ea00640 1 -- 172.21.15.114:0/568353899 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48001670 con 0x7fae600cf9a0 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 -- 172.21.15.114:0/568353899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae600cf9a0 msgr2=0x7fae600cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 --2- 172.21.15.114:0/568353899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae600cf9a0 0x7fae600cfe30 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fae580097a0 tx=0x7fae5802b710 comp rx=0 tx=0).stop 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 -- 172.21.15.114:0/568353899 shutdown_connections 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 --2- 172.21.15.114:0/568353899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae600cf9a0 0x7fae600cfe30 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 --2- 172.21.15.114:0/568353899 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae6015f770 0x7fae6015fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 -- 172.21.15.114:0/568353899 >> 172.21.15.114:0/568353899 conn(0x7fae600b3870 msgr2=0x7fae600b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:54.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 -- 172.21.15.114:0/568353899 shutdown_connections 2024-03-06T07:38:54.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.272+0000 7fae66000640 1 -- 172.21.15.114:0/568353899 wait complete. 2024-03-06T07:38:54.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.274+0000 7fae66000640 1 Processor -- start 2024-03-06T07:38:54.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.274+0000 7fae66000640 1 -- start start 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.275+0000 7fae66000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae600b4390 0x7fae600c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.275+0000 7fae66000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae600cf9a0 0x7fae60154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.275+0000 7fae66000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6015f770 0x7fae601556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.275+0000 7fae66000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600b34a0 con 0x7fae6015f770 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.275+0000 7fae66000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae600b3320 con 0x7fae600b4390 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.275+0000 7fae66000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae601609a0 con 0x7fae600cf9a0 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6015f770 0x7fae601556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6015f770 0x7fae601556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34012/0 (socket says 172.21.15.114:34012) 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 -- 172.21.15.114:0/470498746 learned_addr learned my addr 172.21.15.114:0/470498746 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:54.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5f400640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae600b4390 0x7fae600c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae56a00640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae600cf9a0 0x7fae60154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 -- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae600cf9a0 msgr2=0x7fae60154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae600cf9a0 0x7fae60154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 -- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae600b4390 msgr2=0x7fae600c5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae600b4390 0x7fae600c5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 -- 172.21.15.114:0/470498746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae60152d80 con 0x7fae6015f770 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae56a00640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae600cf9a0 0x7fae60154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:54.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.276+0000 7fae5fe00640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6015f770 0x7fae601556a0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fae580098c0 tx=0x7fae580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.277+0000 7fae57e00640 1 -- 172.21.15.114:0/470498746 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58041020 con 0x7fae6015f770 2024-03-06T07:38:54.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.277+0000 7fae57e00640 1 -- 172.21.15.114:0/470498746 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5803c890 con 0x7fae6015f770 2024-03-06T07:38:54.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.277+0000 7fae57e00640 1 -- 172.21.15.114:0/470498746 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58037040 con 0x7fae6015f770 2024-03-06T07:38:54.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.277+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae60154880 con 0x7fae6015f770 2024-03-06T07:38:54.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.277+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae60154e40 con 0x7fae6015f770 2024-03-06T07:38:54.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.278+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6004f330 con 0x7fae6015f770 2024-03-06T07:38:54.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.280+0000 7fae57e00640 1 -- 172.21.15.114:0/470498746 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae58044020 con 0x7fae6015f770 2024-03-06T07:38:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.281+0000 7fae57e00640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b520 0x7fae3c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.281+0000 7fae57e00640 1 -- 172.21.15.114:0/470498746 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fae580abb30 con 0x7fae6015f770 2024-03-06T07:38:54.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.281+0000 7fae5f400640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b520 0x7fae3c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.282+0000 7fae5f400640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b520 0x7fae3c06d9e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fae500073f0 tx=0x7fae5000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.285+0000 7fae57e00640 1 -- 172.21.15.114:0/470498746 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae5804b030 con 0x7fae6015f770 2024-03-06T07:38:54.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.521+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae600ce130 con 0x7fae6015f770 2024-03-06T07:38:54.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.522+0000 7fae57e00640 1 -- 172.21.15.114:0/470498746 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fae5803ae20 con 0x7fae6015f770 2024-03-06T07:38:54.524 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:54.524 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b520 msgr2=0x7fae3c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b520 0x7fae3c06d9e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fae500073f0 tx=0x7fae5000b040 comp rx=0 tx=0).stop 2024-03-06T07:38:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6015f770 msgr2=0x7fae601556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6015f770 0x7fae601556a0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fae580098c0 tx=0x7fae580046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 shutdown_connections 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fae3c06b520 0x7fae3c06d9e0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae6015f770 0x7fae601556a0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fae600cf9a0 0x7fae60154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 --2- 172.21.15.114:0/470498746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae600b4390 0x7fae600c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 >> 172.21.15.114:0/470498746 conn(0x7fae600b3870 msgr2=0x7fae60070670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.525+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 shutdown_connections 2024-03-06T07:38:54.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:54.526+0000 7fae66000640 1 -- 172.21.15.114:0/470498746 wait complete. 2024-03-06T07:38:54.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:54.747 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:55.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:56.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.081+0000 7fdb71600640 1 Processor -- start 2024-03-06T07:38:56.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.082+0000 7fdb71600640 1 -- start start 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.083+0000 7fdb71600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdb6c15a940 0x7fdb6c0b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.083+0000 7fdb71600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c06d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.083+0000 7fdb71600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c0b2b30 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.083+0000 7fdb71600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c0b28e0 con 0x7fdb6c15a940 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.083+0000 7fdb71600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0b2e10 con 0x7fdb6c0cf370 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.083+0000 7fdb6be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdb6c15a940 0x7fdb6c0b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.084+0000 7fdb6be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdb6c15a940 0x7fdb6c0b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34024/0 (socket says 172.21.15.114:34024) 2024-03-06T07:38:56.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.084+0000 7fdb6be00640 1 -- 172.21.15.114:0/246416570 learned_addr learned my addr 172.21.15.114:0/246416570 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.084+0000 7fdb63400640 1 --2- 172.21.15.114:0/246416570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.084+0000 7fdb6be00640 1 -- 172.21.15.114:0/246416570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 msgr2=0x7fdb6c06d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.084+0000 7fdb6be00640 1 --2- 172.21.15.114:0/246416570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c06d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.084+0000 7fdb6be00640 1 -- 172.21.15.114:0/246416570 >> v1:172.21.15.177:6789/0 conn(0x7fdb6c0cc2a0 legacy=0x7fdb6c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.084+0000 7fdb6be00640 1 -- 172.21.15.114:0/246416570 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c06ca50 con 0x7fdb6c15a940 2024-03-06T07:38:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.085+0000 7fdb6be00640 1 --2- 172.21.15.114:0/246416570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0b45e0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009870 tx=0x7fdb5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50d72e917b328d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.085+0000 7fdb6b400640 1 -- 172.21.15.114:0/246416570 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c035020 con 0x7fdb6c15a940 2024-03-06T07:38:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.085+0000 7fdb6b400640 1 -- 172.21.15.114:0/246416570 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c031640 con 0x7fdb6c15a940 2024-03-06T07:38:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.085+0000 7fdb6b400640 1 -- 172.21.15.114:0/246416570 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c02d040 con 0x7fdb6c15a940 2024-03-06T07:38:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 -- 172.21.15.114:0/246416570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c15a940 msgr2=0x7fdb6c0b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 --2- 172.21.15.114:0/246416570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0b45e0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009870 tx=0x7fdb5c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 -- 172.21.15.114:0/246416570 shutdown_connections 2024-03-06T07:38:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 --2- 172.21.15.114:0/246416570 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c06d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 --2- 172.21.15.114:0/246416570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0b45e0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 -- 172.21.15.114:0/246416570 >> 172.21.15.114:0/246416570 conn(0x7fdb6c06afc0 msgr2=0x7fdb6c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 -- 172.21.15.114:0/246416570 shutdown_connections 2024-03-06T07:38:56.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.086+0000 7fdb71600640 1 -- 172.21.15.114:0/246416570 wait complete. 2024-03-06T07:38:56.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.088+0000 7fdb71600640 1 Processor -- start 2024-03-06T07:38:56.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.088+0000 7fdb71600640 1 -- start start 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb71600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c0cc2a0 0x7fdb6c148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb71600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c0b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb71600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb71600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0b4b20 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb71600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0b30f0 con 0x7fdb6c15a940 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb71600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c06d790 con 0x7fdb6c0cf370 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb6be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c0cc2a0 0x7fdb6c148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb6be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c0cc2a0 0x7fdb6c148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34038/0 (socket says 172.21.15.114:34038) 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.089+0000 7fdb6be00640 1 -- 172.21.15.114:0/409246926 learned_addr learned my addr 172.21.15.114:0/409246926 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb63400640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c0b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb6be00640 1 -- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 msgr2=0x7fdb6c0b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb6be00640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c0b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb6be00640 1 -- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb6c15a940 msgr2=0x7fdb6c0aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb70c00640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb6be00640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb6be00640 1 -- 172.21.15.114:0/409246926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c148880 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb63400640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c0b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb70c00640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:56.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.090+0000 7fdb6be00640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c0cc2a0 0x7fdb6c148340 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c02b930 tx=0x7fdb5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.091+0000 7fdb68c00640 1 -- 172.21.15.114:0/409246926 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c040070 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.091+0000 7fdb68c00640 1 -- 172.21.15.114:0/409246926 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c0399f0 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.091+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c0af820 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.091+0000 7fdb68c00640 1 -- 172.21.15.114:0/409246926 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c035040 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.091+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c0afde0 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.092+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c04f330 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.093+0000 7fdb68c00640 1 -- 172.21.15.114:0/409246926 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb5c04b060 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.094+0000 7fdb68c00640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb4806b4d0 0x7fdb4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.094+0000 7fdb68c00640 1 -- 172.21.15.114:0/409246926 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdb5c0abb40 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.094+0000 7fdb63400640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb4806b4d0 0x7fdb4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.095+0000 7fdb63400640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb4806b4d0 0x7fdb4806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c074590 tx=0x7fdb54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.100+0000 7fdb68c00640 1 -- 172.21.15.114:0/409246926 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb6c148880 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.156 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:56.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.355+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb6c0742e0 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.356+0000 7fdb68c00640 1 -- 172.21.15.114:0/409246926 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fdb5c07a970 con 0x7fdb6c0cc2a0 2024-03-06T07:38:56.358 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:56.358 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:56.358 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:56.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb4806b4d0 msgr2=0x7fdb4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb4806b4d0 0x7fdb4806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c074590 tx=0x7fdb54008040 comp rx=0 tx=0).stop 2024-03-06T07:38:56.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c0cc2a0 msgr2=0x7fdb6c148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c0cc2a0 0x7fdb6c148340 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c02b930 tx=0x7fdb5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 shutdown_connections 2024-03-06T07:38:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb6c15a940 0x7fdb6c0aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdb6c0cf370 0x7fdb6c0b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdb4806b4d0 0x7fdb4806d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 --2- 172.21.15.114:0/409246926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb6c0cc2a0 0x7fdb6c148340 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.358+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 >> 172.21.15.114:0/409246926 conn(0x7fdb6c06afc0 msgr2=0x7fdb6c070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.359+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 shutdown_connections 2024-03-06T07:38:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:56.359+0000 7fdb71600640 1 -- 172.21.15.114:0/409246926 wait complete. 2024-03-06T07:38:56.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:56.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:57.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:57.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.786+0000 7fbac8200640 1 Processor -- start 2024-03-06T07:38:57.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.786+0000 7fbac8200640 1 -- start start 2024-03-06T07:38:57.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac8200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac0144c00 0x7fbac006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac8200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbac015f9d0 0x7fbac015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac8200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac006ee70 con 0x7fbac00ce0a0 2024-03-06T07:38:57.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac8200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac006cef0 con 0x7fbac0144c00 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac8200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac006d1d0 con 0x7fbac015f9d0 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac5800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac0144c00 0x7fbac006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac5800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac0144c00 0x7fbac006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34052/0 (socket says 172.21.15.114:34052) 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac5800640 1 -- 172.21.15.114:0/2122304933 learned_addr learned my addr 172.21.15.114:0/2122304933 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac5800640 1 -- 172.21.15.114:0/2122304933 >> v2:172.21.15.177:3300/0 conn(0x7fbac015f9d0 msgr2=0x7fbac015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac5800640 1 --2- 172.21.15.114:0/2122304933 >> v2:172.21.15.177:3300/0 conn(0x7fbac015f9d0 0x7fbac015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.788+0000 7fbac5800640 1 -- 172.21.15.114:0/2122304933 >> v1:172.21.15.177:6789/0 conn(0x7fbac00ce0a0 legacy=0x7fbac00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:57.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.789+0000 7fbac5800640 1 -- 172.21.15.114:0/2122304933 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac006d4b0 con 0x7fbac0144c00 2024-03-06T07:38:57.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.790+0000 7fbac5800640 1 --2- 172.21.15.114:0/2122304933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac006e930 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fbac00b26f0 tx=0x7fbab402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d0a372b1f58ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.790+0000 7fbabfe00640 1 -- 172.21.15.114:0/2122304933 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4035020 con 0x7fbac0144c00 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.790+0000 7fbabfe00640 1 -- 172.21.15.114:0/2122304933 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4031870 con 0x7fbac0144c00 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.790+0000 7fbabfe00640 1 -- 172.21.15.114:0/2122304933 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4031c40 con 0x7fbac0144c00 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.790+0000 7fbac8200640 1 -- 172.21.15.114:0/2122304933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 msgr2=0x7fbac006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.790+0000 7fbac8200640 1 --2- 172.21.15.114:0/2122304933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac006e930 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fbac00b26f0 tx=0x7fbab402b910 comp rx=0 tx=0).stop 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.791+0000 7fbac8200640 1 -- 172.21.15.114:0/2122304933 shutdown_connections 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.791+0000 7fbac8200640 1 --2- 172.21.15.114:0/2122304933 >> v2:172.21.15.177:3300/0 conn(0x7fbac015f9d0 0x7fbac015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.791+0000 7fbac8200640 1 --2- 172.21.15.114:0/2122304933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac006e930 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.791+0000 7fbac8200640 1 -- 172.21.15.114:0/2122304933 >> 172.21.15.114:0/2122304933 conn(0x7fbac00b1730 msgr2=0x7fbac00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.791+0000 7fbac8200640 1 -- 172.21.15.114:0/2122304933 shutdown_connections 2024-03-06T07:38:57.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.791+0000 7fbac8200640 1 -- 172.21.15.114:0/2122304933 wait complete. 2024-03-06T07:38:57.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.792+0000 7fbac8200640 1 Processor -- start 2024-03-06T07:38:57.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.793+0000 7fbac8200640 1 -- start start 2024-03-06T07:38:57.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.793+0000 7fbac8200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac00ce0a0 0x7fbac013e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.793+0000 7fbac8200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac01e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac8200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac015f9d0 0x7fbac00ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac8200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac006cb50 con 0x7fbac0144c00 2024-03-06T07:38:57.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac8200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0160730 con 0x7fbac015f9d0 2024-03-06T07:38:57.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac8200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac006ccd0 con 0x7fbac00ce0a0 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac01e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac01e6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34066/0 (socket says 172.21.15.114:34066) 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 -- 172.21.15.114:0/299617316 learned_addr learned my addr 172.21.15.114:0/299617316 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac5800640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac00ce0a0 0x7fbac013e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac6200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac015f9d0 0x7fbac00ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 -- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac00ce0a0 msgr2=0x7fbac013e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac00ce0a0 0x7fbac013e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 -- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac015f9d0 msgr2=0x7fbac00ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac015f9d0 0x7fbac00ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac4e00640 1 -- 172.21.15.114:0/299617316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac015e790 con 0x7fbac0144c00 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.794+0000 7fbac6200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac015f9d0 0x7fbac00ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.795+0000 7fbac4e00640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac01e6890 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b4d0 tx=0x7fbaac00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.795+0000 7fbabd600640 1 -- 172.21.15.114:0/299617316 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac014070 con 0x7fbac0144c00 2024-03-06T07:38:57.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.795+0000 7fbabd600640 1 -- 172.21.15.114:0/299617316 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac010870 con 0x7fbac0144c00 2024-03-06T07:38:57.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.795+0000 7fbabd600640 1 -- 172.21.15.114:0/299617316 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac010c60 con 0x7fbac0144c00 2024-03-06T07:38:57.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.795+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac015ea70 con 0x7fbac0144c00 2024-03-06T07:38:57.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.795+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac015f030 con 0x7fbac0144c00 2024-03-06T07:38:57.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.796+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac004f330 con 0x7fbac0144c00 2024-03-06T07:38:57.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.797+0000 7fbabd600640 1 -- 172.21.15.114:0/299617316 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaac017070 con 0x7fbac0144c00 2024-03-06T07:38:57.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.799+0000 7fbabd600640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:57.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.799+0000 7fbabd600640 1 -- 172.21.15.114:0/299617316 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbaac08a130 con 0x7fbac0144c00 2024-03-06T07:38:57.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.799+0000 7fbac5800640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:57.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.800+0000 7fbac5800640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbab4036000 tx=0x7fbab403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:57.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:57.803+0000 7fbabd600640 1 -- 172.21.15.114:0/299617316 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaac058e40 con 0x7fbac0144c00 2024-03-06T07:38:58.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.037+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac00b4b20 con 0x7fbac0144c00 2024-03-06T07:38:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.038+0000 7fbabd600640 1 -- 172.21.15.114:0/299617316 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fbaac058740 con 0x7fbac0144c00 2024-03-06T07:38:58.040 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:58.040 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:58.040 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 msgr2=0x7fbaa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fbab4036000 tx=0x7fbab403c040 comp rx=0 tx=0).stop 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 msgr2=0x7fbac01e6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac01e6890 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00b4d0 tx=0x7fbaac00b990 comp rx=0 tx=0).stop 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 shutdown_connections 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbac015f9d0 0x7fbac00ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fbaa006b4d0 0x7fbaa006d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac0144c00 0x7fbac01e6890 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 --2- 172.21.15.114:0/299617316 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fbac00ce0a0 0x7fbac013e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 >> 172.21.15.114:0/299617316 conn(0x7fbac00b1730 msgr2=0x7fbac0142d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:58.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 shutdown_connections 2024-03-06T07:38:58.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:58.041+0000 7fbac8200640 1 -- 172.21.15.114:0/299617316 wait complete. 2024-03-06T07:38:58.158 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:58.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:58.266 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:38:59.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.474+0000 7f243d800640 1 Processor -- start 2024-03-06T07:38:59.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.475+0000 7f243d800640 1 -- start start 2024-03-06T07:38:59.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.476+0000 7f243d800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 0x7f24380d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.476+0000 7f243d800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24380cea60 con 0x7f24380d0fc0 2024-03-06T07:38:59.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.476+0000 7f243d800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24380d2350 con 0x7f243815f470 2024-03-06T07:38:59.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.476+0000 7f243d800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24380d2f60 con 0x7f243815e0d0 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.476+0000 7f2437400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f24380d0fc0 0x7f2438160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54200/0 (socket says 172.21.15.114:54200) 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.476+0000 7f2437400640 1 -- 172.21.15.114:0/3126362668 learned_addr learned my addr 172.21.15.114:0/3126362668 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436a00640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 0x7f24380d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218967778 0 0) 0x7f24380cea60 con 0x7f24380d0fc0 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c003520 con 0x7f24380d0fc0 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1320009929 0 0) 0x7f241c003520 con 0x7f24380d0fc0 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 msgr2=0x7f24380d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 0x7f24380d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 >> v1:172.21.15.177:6789/0 conn(0x7f243815f470 legacy=0x7f24380d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380d2630 con 0x7f24380d0fc0 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f24300019f0 con 0x7f24380d0fc0 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 >> v1:172.21.15.114:6789/0 conn(0x7f24380d0fc0 legacy=0x7f2438160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f241c0039e0 0x7f241c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f241c004800 0x7f241c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.477+0000 7f2436000640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 0x7f241c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c00bb00 con 0x7f241c0039e0 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c00bc80 con 0x7f241c004800 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c00be80 con 0x7f243815e0d0 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436a00640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f241c0039e0 0x7f241c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2437400640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 0x7f241c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436a00640 1 -- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 msgr2=0x7f241c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f242fe00640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f241c004800 0x7f241c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436a00640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f243815e0d0 0x7f241c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436a00640 1 -- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f241c004800 msgr2=0x7f241c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:59.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436a00640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f241c004800 0x7f241c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.478+0000 7f2436a00640 1 -- 172.21.15.114:0/3126362668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c003520 con 0x7f241c0039e0 2024-03-06T07:38:59.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.479+0000 7f2436a00640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f241c0039e0 0x7f241c003e00 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f2428009a90 tx=0x7f241c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1d1c3b03dd0dae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428039070 con 0x7f241c0039e0 2024-03-06T07:38:59.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242802c680 con 0x7f241c0039e0 2024-03-06T07:38:59.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f2436000640 1 -- 172.21.15.114:0/3126362668 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428009690 con 0x7f241c0039e0 2024-03-06T07:38:59.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f243d800640 1 -- 172.21.15.114:0/3126362668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f241c0039e0 msgr2=0x7f241c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f243d800640 1 --2- 172.21.15.114:0/3126362668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f241c0039e0 0x7f241c003e00 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f2428009a90 tx=0x7f241c005830 comp rx=0 tx=0).stop 2024-03-06T07:38:59.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f2437400640 1 -- 172.21.15.114:0/3126362668 reap_dead start 2024-03-06T07:38:59.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f243d800640 1 -- 172.21.15.114:0/3126362668 shutdown_connections 2024-03-06T07:38:59.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f243d800640 1 -- 172.21.15.114:0/3126362668 >> 172.21.15.114:0/3126362668 conn(0x7f24380d3ec0 msgr2=0x7f24380d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:59.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f243d800640 1 -- 172.21.15.114:0/3126362668 shutdown_connections 2024-03-06T07:38:59.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.480+0000 7f243d800640 1 -- 172.21.15.114:0/3126362668 wait complete. 2024-03-06T07:38:59.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.482+0000 7f243d800640 1 Processor -- start 2024-03-06T07:38:59.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.482+0000 7f243d800640 1 -- start start 2024-03-06T07:38:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.483+0000 7f243d800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f241c0039e0 0x7f2438154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.483+0000 7f243d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24380cb0b0 0x7f24380cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.483+0000 7f243d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24380cb530 0x7f24380cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.483+0000 7f243d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24380cff10 con 0x7f24380cb0b0 2024-03-06T07:38:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.483+0000 7f243d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24380d2bf0 con 0x7f24380cb530 2024-03-06T07:38:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.483+0000 7f243d800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243815fb30 con 0x7f241c0039e0 2024-03-06T07:38:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24380cb0b0 0x7f24380cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24380cb0b0 0x7f24380cf1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38684/0 (socket says 172.21.15.114:38684) 2024-03-06T07:38:59.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 -- 172.21.15.114:0/1543147103 learned_addr learned my addr 172.21.15.114:0/1543147103 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f2436a00640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f241c0039e0 0x7f2438154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 -- 172.21.15.114:0/1543147103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f241c0039e0 msgr2=0x7f2438154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f241c0039e0 0x7f2438154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 -- 172.21.15.114:0/1543147103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24380cb530 msgr2=0x7f24380cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24380cb530 0x7f24380cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 -- 172.21.15.114:0/1543147103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380b8400 con 0x7f24380cb0b0 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242fe00640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24380cb0b0 0x7f24380cf1e0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f242000d570 tx=0x7f242000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242ea00640 1 -- 172.21.15.114:0/1543147103 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420005480 con 0x7f24380cb0b0 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242ea00640 1 -- 172.21.15.114:0/1543147103 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420017070 con 0x7f24380cb0b0 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.484+0000 7f242ea00640 1 -- 172.21.15.114:0/1543147103 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420003810 con 0x7f24380cb0b0 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.485+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24380cdd90 con 0x7f24380cb0b0 2024-03-06T07:38:59.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.485+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24380ce070 con 0x7f24380cb0b0 2024-03-06T07:38:59.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.486+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243804f330 con 0x7f24380cb0b0 2024-03-06T07:38:59.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.487+0000 7f242ea00640 1 -- 172.21.15.114:0/1543147103 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24200039b0 con 0x7f24380cb0b0 2024-03-06T07:38:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.487+0000 7f242ea00640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f241406b500 0x7f241406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.488+0000 7f242ea00640 1 -- 172.21.15.114:0/1543147103 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f242008a160 con 0x7f24380cb0b0 2024-03-06T07:38:59.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.488+0000 7f2436a00640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f241406b500 0x7f241406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:59.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.488+0000 7f2436a00640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f241406b500 0x7f241406d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2428036000 tx=0x7f2428030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:59.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.492+0000 7f242ea00640 1 -- 172.21.15.114:0/1543147103 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2420058f00 con 0x7f24380cb0b0 2024-03-06T07:38:59.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.719+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f243808d920 con 0x7f24380cb0b0 2024-03-06T07:38:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.720+0000 7f242ea00640 1 -- 172.21.15.114:0/1543147103 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f2420058800 con 0x7f24380cb0b0 2024-03-06T07:38:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:38:59.722 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:38:59.722 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:38:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.722+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f241406b500 msgr2=0x7f241406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.722+0000 7f243d800640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f241406b500 0x7f241406d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2428036000 tx=0x7f2428030000 comp rx=0 tx=0).stop 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.722+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24380cb0b0 msgr2=0x7f24380cf1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.722+0000 7f243d800640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24380cb0b0 0x7f24380cf1e0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f242000d570 tx=0x7f242000da30 comp rx=0 tx=0).stop 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 shutdown_connections 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24380cb530 0x7f24380cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f241406b500 0x7f241406d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24380cb0b0 0x7f24380cf1e0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 --2- 172.21.15.114:0/1543147103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f241c0039e0 0x7f2438154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 >> 172.21.15.114:0/1543147103 conn(0x7f24380d3ec0 msgr2=0x7f24380b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 shutdown_connections 2024-03-06T07:38:59.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:38:59.723+0000 7f243d800640 1 -- 172.21.15.114:0/1543147103 wait complete. 2024-03-06T07:38:59.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:38:59.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:00.161 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:00.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:01.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.158+0000 7f8082800640 1 Processor -- start 2024-03-06T07:39:01.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.158+0000 7f8082800640 1 -- start start 2024-03-06T07:39:01.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.159+0000 7f8082800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f807c144660 0x7f807c071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.159+0000 7f8082800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 0x7f807c0724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.159+0000 7f8082800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c0cabf0 con 0x7f807c0c9ec0 2024-03-06T07:39:01.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.159+0000 7f8082800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c0ce900 con 0x7f807c144660 2024-03-06T07:39:01.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.159+0000 7f8082800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0cebe0 con 0x7f807c0cf3e0 2024-03-06T07:39:01.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f807c0c9ec0 0x7f807c0ca310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54202/0 (socket says 172.21.15.114:54202) 2024-03-06T07:39:01.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807be00640 1 -- 172.21.15.114:0/3829428930 learned_addr learned my addr 172.21.15.114:0/3829428930 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:01.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f8073e00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 0x7f807c0724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807b400640 1 --2- 172.21.15.114:0/3829428930 >> v2:172.21.15.177:3300/0 conn(0x7f807c144660 0x7f807c071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144783180 0 0) 0x7f807c0cabf0 con 0x7f807c0c9ec0 2024-03-06T07:39:01.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8060003520 con 0x7f807c0c9ec0 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2807396950 0 0) 0x7f8060003520 con 0x7f807c0c9ec0 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 msgr2=0x7f807c0724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 0x7f807c0724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 >> v2:172.21.15.177:3300/0 conn(0x7f807c144660 msgr2=0x7f807c071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 --2- 172.21.15.114:0/3829428930 >> v2:172.21.15.177:3300/0 conn(0x7f807c144660 0x7f807c071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.160+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0ceec0 con 0x7f807c0c9ec0 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f80740019f0 con 0x7f807c0c9ec0 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 >> v1:172.21.15.114:6789/0 conn(0x7f807c0c9ec0 legacy=0x7f807c0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f8060003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8060004800 0x7f8060004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 0x7f80600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0ce900 con 0x7f80600039b0 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0cebe0 con 0x7f8060004800 2024-03-06T07:39:01.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806000b9c0 con 0x7f807c0cf3e0 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807b400640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f8060003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807be00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 0x7f80600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f8073e00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8060004800 0x7f8060004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807b400640 1 -- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 msgr2=0x7f80600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807b400640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c0cf3e0 0x7f80600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807b400640 1 -- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8060004800 msgr2=0x7f8060004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807b400640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8060004800 0x7f8060004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f807b400640 1 -- 172.21.15.114:0/3829428930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8060003520 con 0x7f80600039b0 2024-03-06T07:39:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.161+0000 7f8073e00640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8060004800 0x7f8060004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f807b400640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f8060003dd0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f806c009ac0 tx=0x7f8060005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6aed0da23827e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c039070 con 0x7f80600039b0 2024-03-06T07:39:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c02c710 con 0x7f80600039b0 2024-03-06T07:39:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f807aa00640 1 -- 172.21.15.114:0/3829428930 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c009690 con 0x7f80600039b0 2024-03-06T07:39:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f8082800640 1 -- 172.21.15.114:0/3829428930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 msgr2=0x7f8060003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f8082800640 1 --2- 172.21.15.114:0/3829428930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f8060003dd0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f806c009ac0 tx=0x7f8060005830 comp rx=0 tx=0).stop 2024-03-06T07:39:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f807be00640 1 -- 172.21.15.114:0/3829428930 reap_dead start 2024-03-06T07:39:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f8082800640 1 -- 172.21.15.114:0/3829428930 shutdown_connections 2024-03-06T07:39:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f8082800640 1 -- 172.21.15.114:0/3829428930 >> 172.21.15.114:0/3829428930 conn(0x7f807c0d0b90 msgr2=0x7f807c0d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.163+0000 7f8082800640 1 -- 172.21.15.114:0/3829428930 shutdown_connections 2024-03-06T07:39:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.164+0000 7f8082800640 1 -- 172.21.15.114:0/3829428930 wait complete. 2024-03-06T07:39:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.165+0000 7f8082800640 1 Processor -- start 2024-03-06T07:39:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.165+0000 7f8082800640 1 -- start start 2024-03-06T07:39:01.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f8082800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f807c0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f8082800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c1f4df0 0x7f807c1f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f8082800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807c0b4ae0 0x7f807c0b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f8082800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c072630 con 0x7f80600039b0 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f8082800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0724b0 con 0x7f807c0b4ae0 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f8082800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0727b0 con 0x7f807c1f4df0 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f807b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f807c0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.166+0000 7f807b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f807c0b6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38696/0 (socket says 172.21.15.114:38696) 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807b400640 1 -- 172.21.15.114:0/2499612557 learned_addr learned my addr 172.21.15.114:0/2499612557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807b400640 1 -- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c1f4df0 msgr2=0x7f807c1f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f8073e00640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c1f4df0 0x7f807c1f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807be00640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807c0b4ae0 0x7f807c0b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807b400640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c1f4df0 0x7f807c1f5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807b400640 1 -- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807c0b4ae0 msgr2=0x7f807c0b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807b400640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807c0b4ae0 0x7f807c0b4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807b400640 1 -- 172.21.15.114:0/2499612557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0b3a40 con 0x7f80600039b0 2024-03-06T07:39:01.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.167+0000 7f807b400640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f807c0b6240 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f806c000c00 tx=0x7f806c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.168+0000 7f8073400640 1 -- 172.21.15.114:0/2499612557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c039050 con 0x7f80600039b0 2024-03-06T07:39:01.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.168+0000 7f8073400640 1 -- 172.21.15.114:0/2499612557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c045070 con 0x7f80600039b0 2024-03-06T07:39:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.168+0000 7f8073400640 1 -- 172.21.15.114:0/2499612557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c0044e0 con 0x7f80600039b0 2024-03-06T07:39:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.168+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c0b3d20 con 0x7f80600039b0 2024-03-06T07:39:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.168+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c0b42e0 con 0x7f80600039b0 2024-03-06T07:39:01.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.169+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c04f330 con 0x7f80600039b0 2024-03-06T07:39:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.171+0000 7f8073400640 1 -- 172.21.15.114:0/2499612557 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f806c003a50 con 0x7f80600039b0 2024-03-06T07:39:01.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.172+0000 7f8073400640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f805806b4d0 0x7f805806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.172+0000 7f8073400640 1 -- 172.21.15.114:0/2499612557 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f806c0abd50 con 0x7f80600039b0 2024-03-06T07:39:01.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.172+0000 7f8073e00640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f805806b4d0 0x7f805806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.173+0000 7f8073e00640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f805806b4d0 0x7f805806d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f807c15d440 tx=0x7f8064008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.177+0000 7f8073400640 1 -- 172.21.15.114:0/2499612557 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f806c04e020 con 0x7f80600039b0 2024-03-06T07:39:01.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.423+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807c0d0140 con 0x7f80600039b0 2024-03-06T07:39:01.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.424+0000 7f8073400640 1 -- 172.21.15.114:0/2499612557 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f806c07aca0 con 0x7f80600039b0 2024-03-06T07:39:01.427 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:01.427 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:01.427 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:01.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f805806b4d0 msgr2=0x7f805806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f805806b4d0 0x7f805806d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f807c15d440 tx=0x7f8064008040 comp rx=0 tx=0).stop 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 msgr2=0x7f807c0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f807c0b6240 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f806c000c00 tx=0x7f806c004730 comp rx=0 tx=0).stop 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 shutdown_connections 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f807c0b4ae0 0x7f807c0b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f807c1f4df0 0x7f807c1f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f805806b4d0 0x7f805806d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 --2- 172.21.15.114:0/2499612557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80600039b0 0x7f807c0b6240 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 >> 172.21.15.114:0/2499612557 conn(0x7f807c0d0b90 msgr2=0x7f807c143890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 shutdown_connections 2024-03-06T07:39:01.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:01.427+0000 7f8082800640 1 -- 172.21.15.114:0/2499612557 wait complete. 2024-03-06T07:39:01.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:01.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:02.163 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:02.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:02.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.865+0000 7f84a3400640 1 Processor -- start 2024-03-06T07:39:02.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.865+0000 7f84a3400640 1 -- start start 2024-03-06T07:39:02.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.868+0000 7f84a3400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f849c15acb0 0x7f849c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.868+0000 7f84a3400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c06ec00 0x7f849c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.868+0000 7f84a3400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849c15b090 con 0x7f849c0b4680 2024-03-06T07:39:02.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.868+0000 7f84a3400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849c0b2a30 con 0x7f849c15acb0 2024-03-06T07:39:02.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.868+0000 7f84a3400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0b4360 con 0x7f849c06ec00 2024-03-06T07:39:02.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.869+0000 7f849be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f849c0b4680 0x7f849c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54212/0 (socket says 172.21.15.114:54212) 2024-03-06T07:39:02.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.869+0000 7f849be00640 1 -- 172.21.15.114:0/173136341 learned_addr learned my addr 172.21.15.114:0/173136341 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.869+0000 7f84a0a00640 1 --2- 172.21.15.114:0/173136341 >> v2:172.21.15.177:3300/0 conn(0x7f849c15acb0 0x7f849c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.869+0000 7f84a1400640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c06ec00 0x7f849c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.869+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425714075 0 0) 0x7f849c15b090 con 0x7f849c0b4680 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.869+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484003520 con 0x7f849c0b4680 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4029176669 0 0) 0x7f8484003520 con 0x7f849c0b4680 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c06ec00 msgr2=0x7f849c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c06ec00 0x7f849c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 >> v2:172.21.15.177:3300/0 conn(0x7f849c15acb0 msgr2=0x7f849c0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 --2- 172.21.15.114:0/173136341 >> v2:172.21.15.177:3300/0 conn(0x7f849c15acb0 0x7f849c0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c0b2cf0 con 0x7f849c0b4680 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8488001670 con 0x7f849c0b4680 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 >> v1:172.21.15.114:6789/0 conn(0x7f849c0b4680 legacy=0x7f849c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84840039b0 0x7f8484003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c06ec00 0x7f8484004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8484004d40 0x7f84840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0b2a30 con 0x7f84840039b0 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0b4360 con 0x7f8484004d40 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848400b9c0 con 0x7f849c06ec00 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.870+0000 7f84a0a00640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84840039b0 0x7f8484003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.871+0000 7f84a0a00640 1 -- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c06ec00 msgr2=0x7f8484004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.871+0000 7f84a0a00640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c06ec00 0x7f8484004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.871+0000 7f84a0a00640 1 -- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8484004d40 msgr2=0x7f84840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.871+0000 7f84a0a00640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8484004d40 0x7f84840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.871+0000 7f84a0a00640 1 -- 172.21.15.114:0/173136341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484003520 con 0x7f84840039b0 2024-03-06T07:39:02.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.872+0000 7f84a0a00640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84840039b0 0x7f8484003dd0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f8490009af0 tx=0x7f8484005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62261936ddebda2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:02.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.872+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490039070 con 0x7f84840039b0 2024-03-06T07:39:02.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.872+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849002c710 con 0x7f84840039b0 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.872+0000 7f849b400640 1 -- 172.21.15.114:0/173136341 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490009690 con 0x7f84840039b0 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.873+0000 7f84a3400640 1 -- 172.21.15.114:0/173136341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84840039b0 msgr2=0x7f8484003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.873+0000 7f84a3400640 1 --2- 172.21.15.114:0/173136341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84840039b0 0x7f8484003dd0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f8490009af0 tx=0x7f8484005830 comp rx=0 tx=0).stop 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.873+0000 7f84a1400640 1 -- 172.21.15.114:0/173136341 reap_dead start 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.873+0000 7f84a3400640 1 -- 172.21.15.114:0/173136341 shutdown_connections 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.873+0000 7f84a3400640 1 -- 172.21.15.114:0/173136341 >> 172.21.15.114:0/173136341 conn(0x7f849c0b1730 msgr2=0x7f849c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.873+0000 7f84a3400640 1 -- 172.21.15.114:0/173136341 shutdown_connections 2024-03-06T07:39:02.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.873+0000 7f84a3400640 1 -- 172.21.15.114:0/173136341 wait complete. 2024-03-06T07:39:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.875+0000 7f84a3400640 1 Processor -- start 2024-03-06T07:39:02.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.875+0000 7f84a3400640 1 -- start start 2024-03-06T07:39:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f84a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84840039b0 0x7f849c1f1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f84a3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c152210 0x7f849c157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f84a3400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c1533a0 0x7f849c153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f84a3400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0b3d10 con 0x7f849c152210 2024-03-06T07:39:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f84a3400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0b32b0 con 0x7f84840039b0 2024-03-06T07:39:02.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f84a3400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0b4010 con 0x7f849c1533a0 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c152210 0x7f849c157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c152210 0x7f849c157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38718/0 (socket says 172.21.15.114:38718) 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 -- 172.21.15.114:0/2510238569 learned_addr learned my addr 172.21.15.114:0/2510238569 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f84a0a00640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84840039b0 0x7f849c1f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 -- 172.21.15.114:0/2510238569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c1533a0 msgr2=0x7f849c153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c1533a0 0x7f849c153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 -- 172.21.15.114:0/2510238569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84840039b0 msgr2=0x7f849c1f1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84840039b0 0x7f849c1f1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.876+0000 7f849be00640 1 -- 172.21.15.114:0/2510238569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c1f1a40 con 0x7f849c152210 2024-03-06T07:39:02.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.877+0000 7f849be00640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c152210 0x7f849c157f70 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f848800c7d0 tx=0x7f848800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.877+0000 7f8498c00640 1 -- 172.21.15.114:0/2510238569 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488018070 con 0x7f849c152210 2024-03-06T07:39:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.877+0000 7f8498c00640 1 -- 172.21.15.114:0/2510238569 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488007500 con 0x7f849c152210 2024-03-06T07:39:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.877+0000 7f8498c00640 1 -- 172.21.15.114:0/2510238569 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488004a20 con 0x7f849c152210 2024-03-06T07:39:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.877+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c151170 con 0x7f849c152210 2024-03-06T07:39:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.877+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c151730 con 0x7f849c152210 2024-03-06T07:39:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.878+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c04f330 con 0x7f849c152210 2024-03-06T07:39:02.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.880+0000 7f8498c00640 1 -- 172.21.15.114:0/2510238569 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8488004bc0 con 0x7f849c152210 2024-03-06T07:39:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.881+0000 7f8498c00640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.881+0000 7f8498c00640 1 -- 172.21.15.114:0/2510238569 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8488089740 con 0x7f849c152210 2024-03-06T07:39:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.881+0000 7f84a0a00640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.882+0000 7f84a0a00640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847c06b4d0 0x7f847c06d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8490036000 tx=0x7f8490030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:02.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:02.886+0000 7f8498c00640 1 -- 172.21.15.114:0/2510238569 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8488058580 con 0x7f849c152210 2024-03-06T07:39:03.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.118+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f849c06ce60 con 0x7f849c152210 2024-03-06T07:39:03.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.119+0000 7f8498c00640 1 -- 172.21.15.114:0/2510238569 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f8488006e20 con 0x7f849c152210 2024-03-06T07:39:03.121 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:03.121 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:03.122 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847c06b4d0 msgr2=0x7f847c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847c06b4d0 0x7f847c06d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8490036000 tx=0x7f8490030000 comp rx=0 tx=0).stop 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c152210 msgr2=0x7f849c157f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c152210 0x7f849c157f70 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f848800c7d0 tx=0x7f848800cc90 comp rx=0 tx=0).stop 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 shutdown_connections 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f849c1533a0 0x7f849c153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f849c152210 0x7f849c157f70 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 --2- 172.21.15.114:0/2510238569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84840039b0 0x7f849c1f1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 >> 172.21.15.114:0/2510238569 conn(0x7f849c0b1730 msgr2=0x7f849c06d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:03.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 shutdown_connections 2024-03-06T07:39:03.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:03.122+0000 7f84a3400640 1 -- 172.21.15.114:0/2510238569 wait complete. 2024-03-06T07:39:03.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:03.347 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:04.165 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:04.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:04.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.559+0000 7f6c57200640 1 Processor -- start 2024-03-06T07:39:04.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.559+0000 7f6c57200640 1 -- start start 2024-03-06T07:39:04.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c57200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c5015fe60 0x7f6c50071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c57200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5006cef0 con 0x7f6c50144c00 2024-03-06T07:39:04.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c57200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5006d1d0 con 0x7f6c5015f9d0 2024-03-06T07:39:04.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c57200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5006d4b0 con 0x7f6c5015fe60 2024-03-06T07:39:04.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c4fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6c50144c00 0x7f6c5006e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54222/0 (socket says 172.21.15.114:54222) 2024-03-06T07:39:04.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c4fe00640 1 -- 172.21.15.114:0/1936654826 learned_addr learned my addr 172.21.15.114:0/1936654826 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:04.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958896800 0 0) 0x7f6c5006cef0 con 0x7f6c50144c00 2024-03-06T07:39:04.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c55200640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c5015fe60 0x7f6c50071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.561+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c34003520 con 0x7f6c50144c00 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102775503 0 0) 0x7f6c34003520 con 0x7f6c50144c00 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c5015fe60 msgr2=0x7f6c50071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c5015fe60 0x7f6c50071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 >> v1:172.21.15.177:6789/0 conn(0x7f6c5015f9d0 legacy=0x7f6c50070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5006d790 con 0x7f6c50144c00 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f6c40001670 con 0x7f6c50144c00 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 >> v1:172.21.15.114:6789/0 conn(0x7f6c50144c00 legacy=0x7f6c5006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c340039e0 0x7f6c34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34004800 0x7f6c34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c340051c0 0x7f6c3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5006d4b0 con 0x7f6c340051c0 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3400c0c0 con 0x7f6c34004800 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3400c2c0 con 0x7f6c340039e0 2024-03-06T07:39:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.562+0000 7f6c55200640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c340051c0 0x7f6c3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.563+0000 7f6c4fe00640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c340039e0 0x7f6c34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.563+0000 7f6c4f400640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34004800 0x7f6c34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.563+0000 7f6c55200640 1 -- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c340039e0 msgr2=0x7f6c34003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:04.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.563+0000 7f6c55200640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c340039e0 0x7f6c34003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.563+0000 7f6c55200640 1 -- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34004800 msgr2=0x7f6c34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:04.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.563+0000 7f6c55200640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c34004800 0x7f6c34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.563+0000 7f6c55200640 1 -- 172.21.15.114:0/1936654826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c34003520 con 0x7f6c340051c0 2024-03-06T07:39:04.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.566+0000 7f6c55200640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c340051c0 0x7f6c3400b9c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f6c50160240 tx=0x7f6c34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89383491ed284080 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48039070 con 0x7f6c340051c0 2024-03-06T07:39:04.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4802cbb0 con 0x7f6c340051c0 2024-03-06T07:39:04.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c4ea00640 1 -- 172.21.15.114:0/1936654826 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c48009b20 con 0x7f6c340051c0 2024-03-06T07:39:04.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c57200640 1 -- 172.21.15.114:0/1936654826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c340051c0 msgr2=0x7f6c3400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c57200640 1 --2- 172.21.15.114:0/1936654826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c340051c0 0x7f6c3400b9c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f6c50160240 tx=0x7f6c34005830 comp rx=0 tx=0).stop 2024-03-06T07:39:04.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c55200640 1 -- 172.21.15.114:0/1936654826 reap_dead start 2024-03-06T07:39:04.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c57200640 1 -- 172.21.15.114:0/1936654826 shutdown_connections 2024-03-06T07:39:04.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c57200640 1 -- 172.21.15.114:0/1936654826 >> 172.21.15.114:0/1936654826 conn(0x7f6c500b1730 msgr2=0x7f6c500b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:04.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c57200640 1 -- 172.21.15.114:0/1936654826 shutdown_connections 2024-03-06T07:39:04.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.567+0000 7f6c57200640 1 -- 172.21.15.114:0/1936654826 wait complete. 2024-03-06T07:39:04.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.569+0000 7f6c57200640 1 Processor -- start 2024-03-06T07:39:04.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.569+0000 7f6c57200640 1 -- start start 2024-03-06T07:39:04.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c57200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c5015b700 0x7f6c501e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c57200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c500ad860 0x7f6c500ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c57200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c500ce5e0 0x7f6c500b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c57200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5006cb50 con 0x7f6c5015b700 2024-03-06T07:39:04.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c57200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c50160730 con 0x7f6c500ce5e0 2024-03-06T07:39:04.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c57200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5006ccd0 con 0x7f6c500ad860 2024-03-06T07:39:04.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c4fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c5015b700 0x7f6c501e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.570+0000 7f6c4fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c5015b700 0x7f6c501e5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38738/0 (socket says 172.21.15.114:38738) 2024-03-06T07:39:04.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4fe00640 1 -- 172.21.15.114:0/1054540653 learned_addr learned my addr 172.21.15.114:0/1054540653 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c55200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c500ce5e0 0x7f6c500b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4f400640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c500ad860 0x7f6c500ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4fe00640 1 -- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c500ad860 msgr2=0x7f6c500ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4fe00640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c500ad860 0x7f6c500ce0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4fe00640 1 -- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c500ce5e0 msgr2=0x7f6c500b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4fe00640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c500ce5e0 0x7f6c500b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4fe00640 1 -- 172.21.15.114:0/1054540653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c500adce0 con 0x7f6c5015b700 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c55200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c500ce5e0 0x7f6c500b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c4fe00640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c5015b700 0x7f6c501e5f80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f6c400072f0 tx=0x7f6c400077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.571+0000 7f6c47e00640 1 -- 172.21.15.114:0/1054540653 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4001a070 con 0x7f6c5015b700 2024-03-06T07:39:04.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.572+0000 7f6c47e00640 1 -- 172.21.15.114:0/1054540653 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4000f920 con 0x7f6c5015b700 2024-03-06T07:39:04.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.572+0000 7f6c47e00640 1 -- 172.21.15.114:0/1054540653 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4000fd10 con 0x7f6c5015b700 2024-03-06T07:39:04.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.572+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c500adf60 con 0x7f6c5015b700 2024-03-06T07:39:04.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.572+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c500ae520 con 0x7f6c5015b700 2024-03-06T07:39:04.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.573+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c500cf290 con 0x7f6c5015b700 2024-03-06T07:39:04.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.574+0000 7f6c47e00640 1 -- 172.21.15.114:0/1054540653 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c4001d360 con 0x7f6c5015b700 2024-03-06T07:39:04.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.575+0000 7f6c47e00640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c3006b4d0 0x7f6c3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:04.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.576+0000 7f6c47e00640 1 -- 172.21.15.114:0/1054540653 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6c40089eb0 con 0x7f6c5015b700 2024-03-06T07:39:04.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.576+0000 7f6c4f400640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c3006b4d0 0x7f6c3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:04.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.576+0000 7f6c4f400640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c3006b4d0 0x7f6c3006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6c5015f9a0 tx=0x7f6c3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.582+0000 7f6c47e00640 1 -- 172.21.15.114:0/1054540653 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c500cf290 con 0x7f6c5015b700 2024-03-06T07:39:04.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.818+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c5004f330 con 0x7f6c5015b700 2024-03-06T07:39:04.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.818+0000 7f6c47e00640 1 -- 172.21.15.114:0/1054540653 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f6c500adce0 con 0x7f6c5015b700 2024-03-06T07:39:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:04.821 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:04.821 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:04.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c3006b4d0 msgr2=0x7f6c3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c3006b4d0 0x7f6c3006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6c5015f9a0 tx=0x7f6c3c009040 comp rx=0 tx=0).stop 2024-03-06T07:39:04.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c5015b700 msgr2=0x7f6c501e5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:04.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c5015b700 0x7f6c501e5f80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f6c400072f0 tx=0x7f6c400077b0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 shutdown_connections 2024-03-06T07:39:04.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c500ce5e0 0x7f6c500b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f6c500ad860 0x7f6c500ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f6c3006b4d0 0x7f6c3006d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 --2- 172.21.15.114:0/1054540653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c5015b700 0x7f6c501e5f80 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:04.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 >> 172.21.15.114:0/1054540653 conn(0x7f6c500b1730 msgr2=0x7f6c500af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:04.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.821+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 shutdown_connections 2024-03-06T07:39:04.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:04.822+0000 7f6c57200640 1 -- 172.21.15.114:0/1054540653 wait complete. 2024-03-06T07:39:05.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:05.044 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:06.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:06.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.256+0000 7facd7000640 1 Processor -- start 2024-03-06T07:39:06.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.256+0000 7facd7000640 1 -- start start 2024-03-06T07:39:06.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7facd7000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7facd0144c00 0x7facd006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7facd7000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facd015f9d0 0x7facd015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7facd7000640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd006ee70 con 0x7facd00ce0a0 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7facd7000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd006cef0 con 0x7facd015f9d0 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7facd7000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd006d1d0 con 0x7facd0144c00 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7faccf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facd015f9d0 0x7facd015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7faccf400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facd015f9d0 0x7facd015fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38752/0 (socket says 172.21.15.114:38752) 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.258+0000 7faccf400640 1 -- 172.21.15.114:0/3503478893 learned_addr learned my addr 172.21.15.114:0/3503478893 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.259+0000 7faccfe00640 1 --2- 172.21.15.114:0/3503478893 >> v2:172.21.15.177:3300/0 conn(0x7facd0144c00 0x7facd006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.259+0000 7faccf400640 1 -- 172.21.15.114:0/3503478893 >> v2:172.21.15.177:3300/0 conn(0x7facd0144c00 msgr2=0x7facd006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.259+0000 7faccf400640 1 --2- 172.21.15.114:0/3503478893 >> v2:172.21.15.177:3300/0 conn(0x7facd0144c00 0x7facd006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.259+0000 7faccf400640 1 -- 172.21.15.114:0/3503478893 >> v1:172.21.15.177:6789/0 conn(0x7facd00ce0a0 legacy=0x7facd00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.259+0000 7faccf400640 1 -- 172.21.15.114:0/3503478893 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd006d4b0 con 0x7facd015f9d0 2024-03-06T07:39:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.259+0000 7faccfe00640 1 --2- 172.21.15.114:0/3503478893 >> v2:172.21.15.177:3300/0 conn(0x7facd0144c00 0x7facd006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:06.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.261+0000 7faccf400640 1 --2- 172.21.15.114:0/3503478893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd015fe60 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7facd00b25f0 tx=0x7facbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6443f978445033c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.261+0000 7faccea00640 1 -- 172.21.15.114:0/3503478893 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc035020 con 0x7facd015f9d0 2024-03-06T07:39:06.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.261+0000 7faccea00640 1 -- 172.21.15.114:0/3503478893 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc031870 con 0x7facd015f9d0 2024-03-06T07:39:06.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.261+0000 7faccea00640 1 -- 172.21.15.114:0/3503478893 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc031c40 con 0x7facd015f9d0 2024-03-06T07:39:06.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 -- 172.21.15.114:0/3503478893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 msgr2=0x7facd015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 --2- 172.21.15.114:0/3503478893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd015fe60 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7facd00b25f0 tx=0x7facbc02b910 comp rx=0 tx=0).stop 2024-03-06T07:39:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 -- 172.21.15.114:0/3503478893 shutdown_connections 2024-03-06T07:39:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 --2- 172.21.15.114:0/3503478893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd015fe60 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 --2- 172.21.15.114:0/3503478893 >> v2:172.21.15.177:3300/0 conn(0x7facd0144c00 0x7facd006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 -- 172.21.15.114:0/3503478893 >> 172.21.15.114:0/3503478893 conn(0x7facd00b1730 msgr2=0x7facd00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 -- 172.21.15.114:0/3503478893 shutdown_connections 2024-03-06T07:39:06.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.262+0000 7facd7000640 1 -- 172.21.15.114:0/3503478893 wait complete. 2024-03-06T07:39:06.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.264+0000 7facd7000640 1 Processor -- start 2024-03-06T07:39:06.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.264+0000 7facd7000640 1 -- start start 2024-03-06T07:39:06.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd7000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd00ce0a0 0x7facd015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd7000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd0144c00 0x7facd015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd7000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd0140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd7000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd006cb50 con 0x7facd015f9d0 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd7000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd0160730 con 0x7facd0144c00 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd7000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd006ccd0 con 0x7facd00ce0a0 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd0140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd0140370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38756/0 (socket says 172.21.15.114:38756) 2024-03-06T07:39:06.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 -- 172.21.15.114:0/3610992626 learned_addr learned my addr 172.21.15.114:0/3610992626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:06.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 -- 172.21.15.114:0/3610992626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd00ce0a0 msgr2=0x7facd015d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd00ce0a0 0x7facd015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 -- 172.21.15.114:0/3610992626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd0144c00 msgr2=0x7facd015b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd0144c00 0x7facd015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.265+0000 7facd5000640 1 -- 172.21.15.114:0/3610992626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd015bbb0 con 0x7facd015f9d0 2024-03-06T07:39:06.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.266+0000 7facd5000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd0140370 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7facc800b9a0 tx=0x7facc800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.266+0000 7facc7e00640 1 -- 172.21.15.114:0/3610992626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8019070 con 0x7facd015f9d0 2024-03-06T07:39:06.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.266+0000 7facc7e00640 1 -- 172.21.15.114:0/3610992626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc800cba0 con 0x7facd015f9d0 2024-03-06T07:39:06.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.266+0000 7facc7e00640 1 -- 172.21.15.114:0/3610992626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8014440 con 0x7facd015f9d0 2024-03-06T07:39:06.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.266+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd01f7e00 con 0x7facd015f9d0 2024-03-06T07:39:06.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.266+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd01f83c0 con 0x7facd015f9d0 2024-03-06T07:39:06.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.267+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd004f330 con 0x7facd015f9d0 2024-03-06T07:39:06.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.268+0000 7facc7e00640 1 -- 172.21.15.114:0/3610992626 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facc801c070 con 0x7facd015f9d0 2024-03-06T07:39:06.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.269+0000 7facc7e00640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7facb006b4d0 0x7facb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.269+0000 7facc7e00640 1 -- 172.21.15.114:0/3610992626 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7facc8089e60 con 0x7facd015f9d0 2024-03-06T07:39:06.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.269+0000 7faccfe00640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7facb006b4d0 0x7facb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.270+0000 7faccfe00640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7facb006b4d0 0x7facb006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7facc0006fd0 tx=0x7facc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.274+0000 7facc7e00640 1 -- 172.21.15.114:0/3610992626 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7facd015bbb0 con 0x7facd015f9d0 2024-03-06T07:39:06.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.535+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facd00b4b20 con 0x7facd015f9d0 2024-03-06T07:39:06.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.535+0000 7facc7e00640 1 -- 172.21.15.114:0/3610992626 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7facc8058c90 con 0x7facd015f9d0 2024-03-06T07:39:06.537 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:06.537 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:06.538 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7facb006b4d0 msgr2=0x7facb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7facb006b4d0 0x7facb006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7facc0006fd0 tx=0x7facc0008040 comp rx=0 tx=0).stop 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 msgr2=0x7facd0140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd0140370 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7facc800b9a0 tx=0x7facc800be60 comp rx=0 tx=0).stop 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 shutdown_connections 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7facb006b4d0 0x7facb006d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facd015f9d0 0x7facd0140370 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7facd0144c00 0x7facd015b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 --2- 172.21.15.114:0/3610992626 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7facd00ce0a0 0x7facd015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 >> 172.21.15.114:0/3610992626 conn(0x7facd00b1730 msgr2=0x7facd00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:06.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 shutdown_connections 2024-03-06T07:39:06.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:06.538+0000 7facd7000640 1 -- 172.21.15.114:0/3610992626 wait complete. 2024-03-06T07:39:06.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:06.761 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:07.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:07.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.988+0000 7f8a6f200640 1 Processor -- start 2024-03-06T07:39:07.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.988+0000 7f8a6f200640 1 -- start start 2024-03-06T07:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.990+0000 7f8a6f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 0x7f8a700ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.990+0000 7f8a6f200640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8a700aebf0 0x7f8a700cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.990+0000 7f8a6f200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7015c240 con 0x7f8a700cc800 2024-03-06T07:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.990+0000 7f8a6f200640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7015ce30 con 0x7f8a700aebf0 2024-03-06T07:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.990+0000 7f8a6f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7015c500 con 0x7f8a701561f0 2024-03-06T07:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.990+0000 7f8a6e800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8a700cc800 0x7f8a700cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42744/0 (socket says 172.21.15.114:42744) 2024-03-06T07:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.990+0000 7f8a6e800640 1 -- 172.21.15.114:0/573392958 learned_addr learned my addr 172.21.15.114:0/573392958 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:07.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6de00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 0x7f8a700ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6d400640 1 --2- 172.21.15.114:0/573392958 >> v2:172.21.15.177:3300/0 conn(0x7f8a700aebf0 0x7f8a700cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102194267 0 0) 0x7f8a7015c240 con 0x7f8a700cc800 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a54003520 con 0x7f8a700cc800 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2637219480 0 0) 0x7f8a54003520 con 0x7f8a700cc800 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 msgr2=0x7f8a700ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 0x7f8a700ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 >> v2:172.21.15.177:3300/0 conn(0x7f8a700aebf0 msgr2=0x7f8a700cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 --2- 172.21.15.114:0/573392958 >> v2:172.21.15.177:3300/0 conn(0x7f8a700aebf0 0x7f8a700cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7015c7e0 con 0x7f8a700cc800 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f8a68001670 con 0x7f8a700cc800 2024-03-06T07:39:07.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 >> v1:172.21.15.114:6789/0 conn(0x7f8a700cc800 legacy=0x7f8a700cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 0x7f8a54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.991+0000 7f8a6ca00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a54004d40 0x7f8a540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7015ce30 con 0x7f8a540039b0 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7015c500 con 0x7f8a54004d40 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5400b9c0 con 0x7f8a701561f0 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6de00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6e800640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a54004d40 0x7f8a540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6d400640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 0x7f8a54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6de00640 1 -- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 msgr2=0x7f8a54004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6de00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a701561f0 0x7f8a54004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6de00640 1 -- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a54004d40 msgr2=0x7f8a540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6de00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a54004d40 0x7f8a540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6de00640 1 -- 172.21.15.114:0/573392958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a54003520 con 0x7f8a540039b0 2024-03-06T07:39:07.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.992+0000 7f8a6e800640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a54004d40 0x7f8a540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:07.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.993+0000 7f8a6de00640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a54003dd0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009ac0 tx=0x7f8a54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b67c5d9e69e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:07.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.993+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60039070 con 0x7f8a540039b0 2024-03-06T07:39:07.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.993+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6002c6b0 con 0x7f8a540039b0 2024-03-06T07:39:07.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.993+0000 7f8a6ca00640 1 -- 172.21.15.114:0/573392958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60009690 con 0x7f8a540039b0 2024-03-06T07:39:07.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.994+0000 7f8a6f200640 1 -- 172.21.15.114:0/573392958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 msgr2=0x7f8a54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.994+0000 7f8a6f200640 1 --2- 172.21.15.114:0/573392958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a54003dd0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009ac0 tx=0x7f8a54005830 comp rx=0 tx=0).stop 2024-03-06T07:39:07.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.994+0000 7f8a6e800640 1 -- 172.21.15.114:0/573392958 reap_dead start 2024-03-06T07:39:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.994+0000 7f8a6f200640 1 -- 172.21.15.114:0/573392958 shutdown_connections 2024-03-06T07:39:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.994+0000 7f8a6f200640 1 -- 172.21.15.114:0/573392958 >> 172.21.15.114:0/573392958 conn(0x7f8a7006afc0 msgr2=0x7f8a7006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.994+0000 7f8a6f200640 1 -- 172.21.15.114:0/573392958 shutdown_connections 2024-03-06T07:39:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.994+0000 7f8a6f200640 1 -- 172.21.15.114:0/573392958 wait complete. 2024-03-06T07:39:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.996+0000 7f8a6f200640 1 Processor -- start 2024-03-06T07:39:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.996+0000 7f8a6f200640 1 -- start start 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a70155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a700cd610 0x7f8a700cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a700ad860 0x7f8a700adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a701565d0 con 0x7f8a540039b0 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700ccc90 con 0x7f8a700cd610 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a70156750 con 0x7f8a700ad860 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a70155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a70155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51046/0 (socket says 172.21.15.114:51046) 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.997+0000 7f8a6de00640 1 -- 172.21.15.114:0/4189920919 learned_addr learned my addr 172.21.15.114:0/4189920919 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a6de00640 1 -- 172.21.15.114:0/4189920919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a700ad860 msgr2=0x7f8a700adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a6de00640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a700ad860 0x7f8a700adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a6de00640 1 -- 172.21.15.114:0/4189920919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a700cd610 msgr2=0x7f8a700cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a6de00640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a700cd610 0x7f8a700cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a6de00640 1 -- 172.21.15.114:0/4189920919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a701593e0 con 0x7f8a540039b0 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a6de00640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a70155380 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f8a60037040 tx=0x7f8a60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a66000640 1 -- 172.21.15.114:0/4189920919 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60039050 con 0x7f8a540039b0 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a66000640 1 -- 172.21.15.114:0/4189920919 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a60045070 con 0x7f8a540039b0 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a701f3fc0 con 0x7f8a540039b0 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.998+0000 7f8a66000640 1 -- 172.21.15.114:0/4189920919 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60003a50 con 0x7f8a540039b0 2024-03-06T07:39:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:07.999+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a700ce460 con 0x7f8a540039b0 2024-03-06T07:39:08.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.000+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a700ae220 con 0x7f8a540039b0 2024-03-06T07:39:08.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.001+0000 7f8a66000640 1 -- 172.21.15.114:0/4189920919 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a60003bf0 con 0x7f8a540039b0 2024-03-06T07:39:08.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.001+0000 7f8a66000640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a4c06b4d0 0x7f8a4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.002+0000 7f8a66000640 1 -- 172.21.15.114:0/4189920919 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a600ab9e0 con 0x7f8a540039b0 2024-03-06T07:39:08.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.002+0000 7f8a6d400640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a4c06b4d0 0x7f8a4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.002+0000 7f8a6d400640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a4c06b4d0 0x7f8a4c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8a700cd580 tx=0x7f8a58009750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.007+0000 7f8a66000640 1 -- 172.21.15.114:0/4189920919 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a6007a840 con 0x7f8a540039b0 2024-03-06T07:39:08.169 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:08.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.242+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a7004f330 con 0x7f8a540039b0 2024-03-06T07:39:08.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.243+0000 7f8a66000640 1 -- 172.21.15.114:0/4189920919 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f8a60080030 con 0x7f8a540039b0 2024-03-06T07:39:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:08.245 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:08.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a4c06b4d0 msgr2=0x7f8a4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a4c06b4d0 0x7f8a4c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8a700cd580 tx=0x7f8a58009750 comp rx=0 tx=0).stop 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 msgr2=0x7f8a70155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a70155380 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f8a60037040 tx=0x7f8a60004730 comp rx=0 tx=0).stop 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 shutdown_connections 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f8a700ad860 0x7f8a700adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a700cd610 0x7f8a700cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f8a4c06b4d0 0x7f8a4c06d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.245+0000 7f8a6f200640 1 --2- 172.21.15.114:0/4189920919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a540039b0 0x7f8a70155380 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.246+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 >> 172.21.15.114:0/4189920919 conn(0x7f8a7006afc0 msgr2=0x7f8a700af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.246+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 shutdown_connections 2024-03-06T07:39:08.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:08.246+0000 7f8a6f200640 1 -- 172.21.15.114:0/4189920919 wait complete. 2024-03-06T07:39:08.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:08.470 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:09.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:09.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.688+0000 7fa19dc00640 1 Processor -- start 2024-03-06T07:39:09.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.688+0000 7fa19dc00640 1 -- start start 2024-03-06T07:39:09.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.690+0000 7fa19dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa19815e0d0 0x7fa1980d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.690+0000 7fa19dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815f470 0x7fa1980d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.690+0000 7fa19dc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1980cea60 con 0x7fa1980d0df0 2024-03-06T07:39:09.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.690+0000 7fa19dc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1980d1260 con 0x7fa19815e0d0 2024-03-06T07:39:09.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.690+0000 7fa19dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1980d2f60 con 0x7fa19815f470 2024-03-06T07:39:09.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa19815e0d0 0x7fa1980d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa19815e0d0 0x7fa1980d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51054/0 (socket says 172.21.15.114:51054) 2024-03-06T07:39:09.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196a00640 1 -- 172.21.15.114:0/721482075 learned_addr learned my addr 172.21.15.114:0/721482075 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:09.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196000640 1 --2- 172.21.15.114:0/721482075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815f470 0x7fa1980d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196a00640 1 -- 172.21.15.114:0/721482075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815f470 msgr2=0x7fa1980d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:09.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196a00640 1 --2- 172.21.15.114:0/721482075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815f470 0x7fa1980d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196a00640 1 -- 172.21.15.114:0/721482075 >> v1:172.21.15.177:6789/0 conn(0x7fa1980d0df0 legacy=0x7fa198160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:09.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196a00640 1 -- 172.21.15.114:0/721482075 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1980d2350 con 0x7fa19815e0d0 2024-03-06T07:39:09.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.691+0000 7fa196000640 1 --2- 172.21.15.114:0/721482075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815f470 0x7fa1980d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:09.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.692+0000 7fa196a00640 1 --2- 172.21.15.114:0/721482075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19815e0d0 0x7fa1980d1a90 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fa188009870 tx=0x7fa18802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c7ff725188d878 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:09.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.692+0000 7fa195600640 1 -- 172.21.15.114:0/721482075 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188034070 con 0x7fa19815e0d0 2024-03-06T07:39:09.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa195600640 1 -- 172.21.15.114:0/721482075 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa188030840 con 0x7fa19815e0d0 2024-03-06T07:39:09.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa195600640 1 -- 172.21.15.114:0/721482075 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188030c10 con 0x7fa19815e0d0 2024-03-06T07:39:09.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa19dc00640 1 -- 172.21.15.114:0/721482075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19815e0d0 msgr2=0x7fa1980d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa19dc00640 1 --2- 172.21.15.114:0/721482075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19815e0d0 0x7fa1980d1a90 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fa188009870 tx=0x7fa18802b980 comp rx=0 tx=0).stop 2024-03-06T07:39:09.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa19dc00640 1 -- 172.21.15.114:0/721482075 shutdown_connections 2024-03-06T07:39:09.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa19dc00640 1 --2- 172.21.15.114:0/721482075 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815f470 0x7fa1980d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa19dc00640 1 --2- 172.21.15.114:0/721482075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa19815e0d0 0x7fa1980d1a90 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.693+0000 7fa19dc00640 1 -- 172.21.15.114:0/721482075 >> 172.21.15.114:0/721482075 conn(0x7fa1980d3ec0 msgr2=0x7fa1980d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:09.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.694+0000 7fa19dc00640 1 -- 172.21.15.114:0/721482075 shutdown_connections 2024-03-06T07:39:09.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.694+0000 7fa19dc00640 1 -- 172.21.15.114:0/721482075 wait complete. 2024-03-06T07:39:09.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.696+0000 7fa19dc00640 1 Processor -- start 2024-03-06T07:39:09.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.696+0000 7fa19dc00640 1 -- start start 2024-03-06T07:39:09.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa19dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1980d0df0 0x7fa19815c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa19dc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815e0d0 0x7fa1980b9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa19dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19815f470 0x7fa1980cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa19dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1980d2bf0 con 0x7fa1980d0df0 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa19dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1980d2910 con 0x7fa19815f470 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa19dc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19815fb30 con 0x7fa19815e0d0 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa196a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1980d0df0 0x7fa19815c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa196a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1980d0df0 0x7fa19815c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51060/0 (socket says 172.21.15.114:51060) 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.698+0000 7fa196a00640 1 -- 172.21.15.114:0/2600127717 learned_addr learned my addr 172.21.15.114:0/2600127717 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa196000640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815e0d0 0x7fa1980b9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa196a00640 1 -- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815e0d0 msgr2=0x7fa1980b9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa196a00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815e0d0 0x7fa1980b9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa197400640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19815f470 0x7fa1980cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa196a00640 1 -- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19815f470 msgr2=0x7fa1980cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa196a00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19815f470 0x7fa1980cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa196a00640 1 -- 172.21.15.114:0/2600127717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1980b8060 con 0x7fa1980d0df0 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa197400640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19815f470 0x7fa1980cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.699+0000 7fa196a00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1980d0df0 0x7fa19815c6a0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa188000c00 tx=0x7fa1880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.700+0000 7fa18ea00640 1 -- 172.21.15.114:0/2600127717 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188034040 con 0x7fa1980d0df0 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.700+0000 7fa18ea00640 1 -- 172.21.15.114:0/2600127717 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa188039d90 con 0x7fa1980d0df0 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.700+0000 7fa18ea00640 1 -- 172.21.15.114:0/2600127717 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188043610 con 0x7fa1980d0df0 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.700+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1980b7aa0 con 0x7fa1980d0df0 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.700+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1981f7a80 con 0x7fa1980d0df0 2024-03-06T07:39:09.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.701+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19804f330 con 0x7fa1980d0df0 2024-03-06T07:39:09.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.702+0000 7fa18ea00640 1 -- 172.21.15.114:0/2600127717 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa18804a060 con 0x7fa1980d0df0 2024-03-06T07:39:09.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.703+0000 7fa18ea00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa17806b4d0 0x7fa17806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:09.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.704+0000 7fa18ea00640 1 -- 172.21.15.114:0/2600127717 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa1880ac4a0 con 0x7fa1980d0df0 2024-03-06T07:39:09.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.704+0000 7fa196000640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa17806b4d0 0x7fa17806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:09.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.704+0000 7fa196000640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa17806b4d0 0x7fa17806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa1980b4d20 tx=0x7fa184008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:09.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.710+0000 7fa18ea00640 1 -- 172.21.15.114:0/2600127717 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa18803e070 con 0x7fa1980d0df0 2024-03-06T07:39:09.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.943+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa19808d920 con 0x7fa1980d0df0 2024-03-06T07:39:09.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.944+0000 7fa18ea00640 1 -- 172.21.15.114:0/2600127717 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa18802d330 con 0x7fa1980d0df0 2024-03-06T07:39:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:09.946 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:09.946 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:09.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa17806b4d0 msgr2=0x7fa17806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa17806b4d0 0x7fa17806d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa1980b4d20 tx=0x7fa184008040 comp rx=0 tx=0).stop 2024-03-06T07:39:09.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1980d0df0 msgr2=0x7fa19815c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:09.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1980d0df0 0x7fa19815c6a0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fa188000c00 tx=0x7fa1880046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 shutdown_connections 2024-03-06T07:39:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa19815f470 0x7fa1980cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa19815e0d0 0x7fa1980b9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa17806b4d0 0x7fa17806d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 --2- 172.21.15.114:0/2600127717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1980d0df0 0x7fa19815c6a0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 >> 172.21.15.114:0/2600127717 conn(0x7fa1980d3ec0 msgr2=0x7fa1980b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.946+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 shutdown_connections 2024-03-06T07:39:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:09.947+0000 7fa19dc00640 1 -- 172.21.15.114:0/2600127717 wait complete. 2024-03-06T07:39:10.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:10.166 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:10.171 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:11.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.375+0000 7f08e8200640 1 Processor -- start 2024-03-06T07:39:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.375+0000 7f08e8200640 1 -- start start 2024-03-06T07:39:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.376+0000 7f08e8200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 0x7f08e00b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.376+0000 7f08e8200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e00cf340 con 0x7f08e00ce550 2024-03-06T07:39:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.376+0000 7f08e8200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e00cfe60 con 0x7f08e015e230 2024-03-06T07:39:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.376+0000 7f08e8200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e015e610 con 0x7f08e006edb0 2024-03-06T07:39:11.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.377+0000 7f08e6200640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f08e00ce550 0x7f08e00cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42758/0 (socket says 172.21.15.114:42758) 2024-03-06T07:39:11.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.377+0000 7f08e6200640 1 -- 172.21.15.114:0/574399193 learned_addr learned my addr 172.21.15.114:0/574399193 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.377+0000 7f08e4e00640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 0x7f08e00b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.377+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118963559 0 0) 0x7f08e00cf340 con 0x7f08e00ce550 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.377+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c8003520 con 0x7f08e00ce550 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.377+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252831897 0 0) 0x7f08c8003520 con 0x7f08e00ce550 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 msgr2=0x7f08e00b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 0x7f08e00b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 >> v1:172.21.15.177:6789/0 conn(0x7f08e015e230 legacy=0x7f08e00d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e006f390 con 0x7f08e00ce550 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f08d80019f0 con 0x7f08e00ce550 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 >> v1:172.21.15.114:6789/0 conn(0x7f08e00ce550 legacy=0x7f08e00cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08c80039e0 0x7f08c8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08c8004800 0x7f08c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 0x7f08c8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e015e610 con 0x7f08c80039e0 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c800baf0 con 0x7f08c8004800 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c800bcf0 con 0x7f08e006edb0 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08e5800640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08c80039e0 0x7f08c8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08e6200640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 0x7f08c8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.378+0000 7f08e4e00640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08c8004800 0x7f08c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.379+0000 7f08e5800640 1 -- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 msgr2=0x7f08c8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.379+0000 7f08e5800640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08e006edb0 0x7f08c8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.379+0000 7f08e5800640 1 -- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08c8004800 msgr2=0x7f08c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.379+0000 7f08e5800640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08c8004800 0x7f08c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.379+0000 7f08e5800640 1 -- 172.21.15.114:0/574399193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c8003520 con 0x7f08c80039e0 2024-03-06T07:39:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.380+0000 7f08e5800640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08c80039e0 0x7f08c8003e00 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f08d4001750 tx=0x7f08c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba6128ad981de90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.380+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4001370 con 0x7f08c80039e0 2024-03-06T07:39:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.380+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d40052d0 con 0x7f08c80039e0 2024-03-06T07:39:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.380+0000 7f08dfe00640 1 -- 172.21.15.114:0/574399193 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4003830 con 0x7f08c80039e0 2024-03-06T07:39:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.380+0000 7f08e8200640 1 -- 172.21.15.114:0/574399193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08c80039e0 msgr2=0x7f08c8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.380+0000 7f08e8200640 1 --2- 172.21.15.114:0/574399193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08c80039e0 0x7f08c8003e00 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f08d4001750 tx=0x7f08c8005830 comp rx=0 tx=0).stop 2024-03-06T07:39:11.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.380+0000 7f08e6200640 1 -- 172.21.15.114:0/574399193 reap_dead start 2024-03-06T07:39:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.381+0000 7f08e8200640 1 -- 172.21.15.114:0/574399193 shutdown_connections 2024-03-06T07:39:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.381+0000 7f08e8200640 1 -- 172.21.15.114:0/574399193 >> 172.21.15.114:0/574399193 conn(0x7f08e00b3800 msgr2=0x7f08e00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.381+0000 7f08e8200640 1 -- 172.21.15.114:0/574399193 shutdown_connections 2024-03-06T07:39:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.381+0000 7f08e8200640 1 -- 172.21.15.114:0/574399193 wait complete. 2024-03-06T07:39:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.382+0000 7f08e8200640 1 Processor -- start 2024-03-06T07:39:11.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.383+0000 7f08e8200640 1 -- start start 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.383+0000 7f08e8200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08c80039e0 0x7f08e01f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.383+0000 7f08e8200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e01f3fc0 0x7f08e0144c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.383+0000 7f08e8200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08e0145140 0x7f08e00c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e8200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00d06c0 con 0x7f08e01f3fc0 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e8200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00d03e0 con 0x7f08e0145140 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e8200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e00b4320 con 0x7f08c80039e0 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e01f3fc0 0x7f08e0144c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e01f3fc0 0x7f08e0144c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51070/0 (socket says 172.21.15.114:51070) 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 -- 172.21.15.114:0/1635738607 learned_addr learned my addr 172.21.15.114:0/1635738607 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e5800640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08c80039e0 0x7f08e01f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e6200640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08e0145140 0x7f08e00c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 -- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08c80039e0 msgr2=0x7f08e01f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08c80039e0 0x7f08e01f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 -- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08e0145140 msgr2=0x7f08e00c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08e0145140 0x7f08e00c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.384+0000 7f08e4e00640 1 -- 172.21.15.114:0/1635738607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e0159310 con 0x7f08e01f3fc0 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.385+0000 7f08e4e00640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e01f3fc0 0x7f08e0144c00 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f08cc011600 tx=0x7f08cc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.385+0000 7f08dd600640 1 -- 172.21.15.114:0/1635738607 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00f920 con 0x7f08e01f3fc0 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.385+0000 7f08dd600640 1 -- 172.21.15.114:0/1635738607 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc00fac0 con 0x7f08e01f3fc0 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.385+0000 7f08dd600640 1 -- 172.21.15.114:0/1635738607 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00fe70 con 0x7f08e01f3fc0 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.385+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e01595f0 con 0x7f08e01f3fc0 2024-03-06T07:39:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.385+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e0159bb0 con 0x7f08e01f3fc0 2024-03-06T07:39:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.386+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e004f330 con 0x7f08e01f3fc0 2024-03-06T07:39:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.387+0000 7f08dd600640 1 -- 172.21.15.114:0/1635738607 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08cc015460 con 0x7f08e01f3fc0 2024-03-06T07:39:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.388+0000 7f08dd600640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08c006b4d0 0x7f08c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.388+0000 7f08dd600640 1 -- 172.21.15.114:0/1635738607 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f08cc08ab40 con 0x7f08e01f3fc0 2024-03-06T07:39:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.388+0000 7f08e5800640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08c006b4d0 0x7f08c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.389+0000 7f08e5800640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08c006b4d0 0x7f08c006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f08d4033000 tx=0x7f08d4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:11.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.393+0000 7f08dd600640 1 -- 172.21.15.114:0/1635738607 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08cc019030 con 0x7f08e01f3fc0 2024-03-06T07:39:11.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.689+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e00aeae0 con 0x7f08e01f3fc0 2024-03-06T07:39:11.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.690+0000 7f08dd600640 1 -- 172.21.15.114:0/1635738607 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f08cc059970 con 0x7f08e01f3fc0 2024-03-06T07:39:11.693 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:11.693 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:11.693 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.693+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08c006b4d0 msgr2=0x7f08c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.693+0000 7f08e8200640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08c006b4d0 0x7f08c006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f08d4033000 tx=0x7f08d4009030 comp rx=0 tx=0).stop 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e01f3fc0 msgr2=0x7f08e0144c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e01f3fc0 0x7f08e0144c00 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f08cc011600 tx=0x7f08cc011ac0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 shutdown_connections 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08e0145140 0x7f08e00c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f08c006b4d0 0x7f08c006d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e01f3fc0 0x7f08e0144c00 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 --2- 172.21.15.114:0/1635738607 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f08c80039e0 0x7f08e01f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:11.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 >> 172.21.15.114:0/1635738607 conn(0x7f08e00b3800 msgr2=0x7f08e0156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:11.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 shutdown_connections 2024-03-06T07:39:11.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:11.694+0000 7f08e8200640 1 -- 172.21.15.114:0/1635738607 wait complete. 2024-03-06T07:39:11.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:11.924 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:12.173 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:12.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:13.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.139+0000 7f2247e00640 1 Processor -- start 2024-03-06T07:39:13.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.139+0000 7f2247e00640 1 -- start start 2024-03-06T07:39:13.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.141+0000 7f2247e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22480aedc0 0x7f22480cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.141+0000 7f2247e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f22480cc800 0x7f224815f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.141+0000 7f2247e00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224815c240 con 0x7f22481561f0 2024-03-06T07:39:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.141+0000 7f2247e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224815ce30 con 0x7f22480cc800 2024-03-06T07:39:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.141+0000 7f2247e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224815c500 con 0x7f22480aedc0 2024-03-06T07:39:13.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f223fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22480aedc0 0x7f22480cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f223fe00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22480aedc0 0x7f22480cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:44136/0 (socket says 172.21.15.114:44136) 2024-03-06T07:39:13.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f2246a00640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f22481561f0 0x7f22480ae6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.114:52576/0 (socket says 172.21.15.114:52576) 2024-03-06T07:39:13.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f223fe00640 1 -- 172.21.15.114:0/1229562573 learned_addr learned my addr 172.21.15.114:0/1229562573 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:13.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f2247400640 1 --2- 172.21.15.114:0/1229562573 >> v2:172.21.15.177:3300/0 conn(0x7f22480cc800 0x7f224815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f223fe00640 1 -- 172.21.15.114:0/1229562573 >> v2:172.21.15.177:3300/0 conn(0x7f22480cc800 msgr2=0x7f224815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:13.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f223fe00640 1 --2- 172.21.15.114:0/1229562573 >> v2:172.21.15.177:3300/0 conn(0x7f22480cc800 0x7f224815f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f223fe00640 1 -- 172.21.15.114:0/1229562573 >> v1:172.21.15.177:6789/0 conn(0x7f22481561f0 legacy=0x7f22480ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f223fe00640 1 -- 172.21.15.114:0/1229562573 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224815c7e0 con 0x7f22480aedc0 2024-03-06T07:39:13.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.142+0000 7f2247400640 1 --2- 172.21.15.114:0/1229562573 >> v2:172.21.15.177:3300/0 conn(0x7f22480cc800 0x7f224815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:39:13.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.144+0000 7f223fe00640 1 --2- 172.21.15.114:0/1229562573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22480aedc0 0x7f22480cc2c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2230009870 tx=0x7f223002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51e10049ce314dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.144+0000 7f2246000640 1 -- 172.21.15.114:0/1229562573 <== mon.2 v2:172.21.15.177:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230034070 con 0x7f22480aedc0 2024-03-06T07:39:13.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.144+0000 7f2246000640 1 -- 172.21.15.114:0/1229562573 <== mon.2 v2:172.21.15.177:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2230030840 con 0x7f22480aedc0 2024-03-06T07:39:13.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.144+0000 7f2246000640 1 -- 172.21.15.114:0/1229562573 <== mon.2 v2:172.21.15.177:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230030c10 con 0x7f22480aedc0 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 -- 172.21.15.114:0/1229562573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22480aedc0 msgr2=0x7f22480cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 --2- 172.21.15.114:0/1229562573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22480aedc0 0x7f22480cc2c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2230009870 tx=0x7f223002b980 comp rx=0 tx=0).stop 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 -- 172.21.15.114:0/1229562573 shutdown_connections 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 --2- 172.21.15.114:0/1229562573 >> v2:172.21.15.177:3300/0 conn(0x7f22480cc800 0x7f224815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 --2- 172.21.15.114:0/1229562573 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22480aedc0 0x7f22480cc2c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 -- 172.21.15.114:0/1229562573 >> 172.21.15.114:0/1229562573 conn(0x7f224806afc0 msgr2=0x7f224806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 -- 172.21.15.114:0/1229562573 shutdown_connections 2024-03-06T07:39:13.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.145+0000 7f2247e00640 1 -- 172.21.15.114:0/1229562573 wait complete. 2024-03-06T07:39:13.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.147+0000 7f2247e00640 1 Processor -- start 2024-03-06T07:39:13.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.147+0000 7f2247e00640 1 -- start start 2024-03-06T07:39:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.148+0000 7f2247e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22480aedc0 0x7f2248147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.148+0000 7f2247e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22480cc800 0x7f2248144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.148+0000 7f2247e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22481561f0 0x7f22481568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.148+0000 7f2247e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22481565d0 con 0x7f22480cc800 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.148+0000 7f2247e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22480ccc90 con 0x7f22480aedc0 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.148+0000 7f2247e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2248156750 con 0x7f22481561f0 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22480cc800 0x7f2248144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22480cc800 0x7f2248144a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51076/0 (socket says 172.21.15.114:51076) 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 -- 172.21.15.114:0/3477653111 learned_addr learned my addr 172.21.15.114:0/3477653111 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f2247400640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22481561f0 0x7f22481568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f2246a00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22480aedc0 0x7f2248147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 -- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22481561f0 msgr2=0x7f22481568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22481561f0 0x7f22481568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 -- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22480aedc0 msgr2=0x7f2248147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22480aedc0 0x7f2248147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 -- 172.21.15.114:0/3477653111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22481483a0 con 0x7f22480cc800 2024-03-06T07:39:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f2246a00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22480aedc0 0x7f2248147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.149+0000 7f223fe00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22480cc800 0x7f2248144a20 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f2230000c00 tx=0x7f2230004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.150+0000 7f223f400640 1 -- 172.21.15.114:0/3477653111 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230048070 con 0x7f22480cc800 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.150+0000 7f223f400640 1 -- 172.21.15.114:0/3477653111 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2230030640 con 0x7f22480cc800 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.150+0000 7f223f400640 1 -- 172.21.15.114:0/3477653111 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230034040 con 0x7f22480cc800 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.150+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2248145760 con 0x7f22480cc800 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.150+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2248146000 con 0x7f22480cc800 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.151+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224804f330 con 0x7f22480cc800 2024-03-06T07:39:13.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.152+0000 7f223f400640 1 -- 172.21.15.114:0/3477653111 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f223004b070 con 0x7f22480cc800 2024-03-06T07:39:13.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.153+0000 7f223f400640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f222406b4d0 0x7f222406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.153+0000 7f223f400640 1 -- 172.21.15.114:0/3477653111 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f22300ac150 con 0x7f22480cc800 2024-03-06T07:39:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.153+0000 7f2246a00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f222406b4d0 0x7f222406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.154+0000 7f2246a00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f222406b4d0 0x7f222406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f22480aec20 tx=0x7f2238009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.157+0000 7f223f400640 1 -- 172.21.15.114:0/3477653111 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22481483a0 con 0x7f22480cc800 2024-03-06T07:39:13.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.393+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22480b2b10 con 0x7f22480cc800 2024-03-06T07:39:13.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.394+0000 7f223f400640 1 -- 172.21.15.114:0/3477653111 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f223007aef0 con 0x7f22480cc800 2024-03-06T07:39:13.396 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:13.396 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:13.396 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f222406b4d0 msgr2=0x7f222406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f222406b4d0 0x7f222406d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f22480aec20 tx=0x7f2238009040 comp rx=0 tx=0).stop 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22480cc800 msgr2=0x7f2248144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22480cc800 0x7f2248144a20 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f2230000c00 tx=0x7f2230004730 comp rx=0 tx=0).stop 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 shutdown_connections 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f22481561f0 0x7f22481568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f222406b4d0 0x7f222406d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22480cc800 0x7f2248144a20 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 --2- 172.21.15.114:0/3477653111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22480aedc0 0x7f2248147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 >> 172.21.15.114:0/3477653111 conn(0x7f224806afc0 msgr2=0x7f2248159e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:13.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 shutdown_connections 2024-03-06T07:39:13.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:13.397+0000 7f2247e00640 1 -- 172.21.15.114:0/3477653111 wait complete. 2024-03-06T07:39:13.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:13.617 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:14.176 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:14.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:14.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.831+0000 7fd4d7400640 1 Processor -- start 2024-03-06T07:39:14.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.831+0000 7fd4d7400640 1 -- start start 2024-03-06T07:39:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.832+0000 7fd4d7400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4d015acb0 0x7fd4d00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.832+0000 7fd4d7400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.832+0000 7fd4d7400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d015b090 con 0x7fd4d0072340 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.832+0000 7fd4d7400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d00b2a30 con 0x7fd4d015acb0 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.832+0000 7fd4d7400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b4360 con 0x7fd4d00b4680 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4d4a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4d015acb0 0x7fd4d00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4d4a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd4d015acb0 0x7fd4d00b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51086/0 (socket says 172.21.15.114:51086) 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4d4a00640 1 -- 172.21.15.114:0/1860805902 learned_addr learned my addr 172.21.15.114:0/1860805902 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4cfe00640 1 --2- 172.21.15.114:0/1860805902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4d4a00640 1 -- 172.21.15.114:0/1860805902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 msgr2=0x7fd4d006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4d4a00640 1 --2- 172.21.15.114:0/1860805902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4d4a00640 1 -- 172.21.15.114:0/1860805902 >> v1:172.21.15.177:6789/0 conn(0x7fd4d0072340 legacy=0x7fd4d00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4d4a00640 1 -- 172.21.15.114:0/1860805902 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00b2cf0 con 0x7fd4d015acb0 2024-03-06T07:39:14.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.833+0000 7fd4cfe00640 1 --2- 172.21.15.114:0/1860805902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:14.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.834+0000 7fd4d4a00640 1 --2- 172.21.15.114:0/1860805902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d015acb0 0x7fd4d00b37d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4009870 tx=0x7fd4c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dc236837c48409 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:14.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4cf400640 1 -- 172.21.15.114:0/1860805902 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4035020 con 0x7fd4d015acb0 2024-03-06T07:39:14.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4cf400640 1 -- 172.21.15.114:0/1860805902 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c4031640 con 0x7fd4d015acb0 2024-03-06T07:39:14.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4cf400640 1 -- 172.21.15.114:0/1860805902 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c402d040 con 0x7fd4d015acb0 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4d7400640 1 -- 172.21.15.114:0/1860805902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d015acb0 msgr2=0x7fd4d00b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4d7400640 1 --2- 172.21.15.114:0/1860805902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d015acb0 0x7fd4d00b37d0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4009870 tx=0x7fd4c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4d7400640 1 -- 172.21.15.114:0/1860805902 shutdown_connections 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4d7400640 1 --2- 172.21.15.114:0/1860805902 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4d7400640 1 --2- 172.21.15.114:0/1860805902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d015acb0 0x7fd4d00b37d0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.835+0000 7fd4d7400640 1 -- 172.21.15.114:0/1860805902 >> 172.21.15.114:0/1860805902 conn(0x7fd4d00b1730 msgr2=0x7fd4d00b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.836+0000 7fd4d7400640 1 -- 172.21.15.114:0/1860805902 shutdown_connections 2024-03-06T07:39:14.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.836+0000 7fd4d7400640 1 -- 172.21.15.114:0/1860805902 wait complete. 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.837+0000 7fd4d7400640 1 Processor -- start 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.837+0000 7fd4d7400640 1 -- start start 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.838+0000 7fd4d7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0072340 0x7fd4d0148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.838+0000 7fd4d7400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.838+0000 7fd4d7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4d015acb0 0x7fd4d006d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.838+0000 7fd4d7400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b3d10 con 0x7fd4d0072340 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.838+0000 7fd4d7400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b32b0 con 0x7fd4d015acb0 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.838+0000 7fd4d7400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b4010 con 0x7fd4d00b4680 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0072340 0x7fd4d0148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0072340 0x7fd4d0148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51102/0 (socket says 172.21.15.114:51102) 2024-03-06T07:39:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 -- 172.21.15.114:0/3890104103 learned_addr learned my addr 172.21.15.114:0/3890104103 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 -- 172.21.15.114:0/3890104103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 msgr2=0x7fd4d006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 -- 172.21.15.114:0/3890104103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4d015acb0 msgr2=0x7fd4d006d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4d015acb0 0x7fd4d006d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 -- 172.21.15.114:0/3890104103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00aab40 con 0x7fd4d0072340 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.839+0000 7fd4d4a00640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0072340 0x7fd4d0148660 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4000c00 tx=0x7fd4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.840+0000 7fd4ccc00640 1 -- 172.21.15.114:0/3890104103 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4040070 con 0x7fd4d0072340 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.840+0000 7fd4ccc00640 1 -- 172.21.15.114:0/3890104103 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c4039c10 con 0x7fd4d0072340 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.840+0000 7fd4ccc00640 1 -- 172.21.15.114:0/3890104103 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4035040 con 0x7fd4d0072340 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.840+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d00aa2a0 con 0x7fd4d0072340 2024-03-06T07:39:14.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.840+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d00aa860 con 0x7fd4d0072340 2024-03-06T07:39:14.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.841+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d00cc5d0 con 0x7fd4d0072340 2024-03-06T07:39:14.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.843+0000 7fd4ccc00640 1 -- 172.21.15.114:0/3890104103 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4c4038020 con 0x7fd4d0072340 2024-03-06T07:39:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.843+0000 7fd4ccc00640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4b006b4d0 0x7fd4b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.843+0000 7fd4ccc00640 1 -- 172.21.15.114:0/3890104103 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd4c40abd60 con 0x7fd4d0072340 2024-03-06T07:39:14.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.844+0000 7fd4cfe00640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4b006b4d0 0x7fd4b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:14.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.844+0000 7fd4cfe00640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4b006b4d0 0x7fd4b006d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc006fd0 tx=0x7fd4bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:14.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:14.852+0000 7fd4ccc00640 1 -- 172.21.15.114:0/3890104103 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4d00cc5d0 con 0x7fd4d0072340 2024-03-06T07:39:15.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.084+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4d006ce60 con 0x7fd4d0072340 2024-03-06T07:39:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.085+0000 7fd4ccc00640 1 -- 172.21.15.114:0/3890104103 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd4d00aab40 con 0x7fd4d0072340 2024-03-06T07:39:15.088 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:15.088 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:15.088 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.087+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4b006b4d0 msgr2=0x7fd4b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.087+0000 7fd4d7400640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4b006b4d0 0x7fd4b006d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc006fd0 tx=0x7fd4bc008040 comp rx=0 tx=0).stop 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.087+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0072340 msgr2=0x7fd4d0148660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.087+0000 7fd4d7400640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0072340 0x7fd4d0148660 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4000c00 tx=0x7fd4c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 shutdown_connections 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd4d015acb0 0x7fd4d006d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd4d00b4680 0x7fd4d006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd4b006b4d0 0x7fd4b006d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 --2- 172.21.15.114:0/3890104103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4d0072340 0x7fd4d0148660 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 >> 172.21.15.114:0/3890104103 conn(0x7fd4d00b1730 msgr2=0x7fd4d00ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 shutdown_connections 2024-03-06T07:39:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:15.088+0000 7fd4d7400640 1 -- 172.21.15.114:0/3890104103 wait complete. 2024-03-06T07:39:15.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:15.309 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:16.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:16.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.518+0000 7fc16ae00640 1 Processor -- start 2024-03-06T07:39:16.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.518+0000 7fc16ae00640 1 -- start start 2024-03-06T07:39:16.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.520+0000 7fc16ae00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc16415f9d0 0x7fc16415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.520+0000 7fc16ae00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16406cef0 con 0x7fc164070b10 2024-03-06T07:39:16.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.520+0000 7fc16ae00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16406d1d0 con 0x7fc164144c00 2024-03-06T07:39:16.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.520+0000 7fc16ae00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16406d4b0 con 0x7fc16415f9d0 2024-03-06T07:39:16.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.520+0000 7fc168e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc164070b10 0x7fc164070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:42772/0 (socket says 172.21.15.114:42772) 2024-03-06T07:39:16.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.520+0000 7fc168e00640 1 -- 172.21.15.114:0/305573097 learned_addr learned my addr 172.21.15.114:0/305573097 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.520+0000 7fc15b400640 1 --2- 172.21.15.114:0/305573097 >> v2:172.21.15.177:3300/0 conn(0x7fc16415f9d0 0x7fc16415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635683773 0 0) 0x7fc16406cef0 con 0x7fc164070b10 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc148003520 con 0x7fc164070b10 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211619480 0 0) 0x7fc148003520 con 0x7fc164070b10 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 >> v2:172.21.15.177:3300/0 conn(0x7fc16415f9d0 msgr2=0x7fc16415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 --2- 172.21.15.114:0/305573097 >> v2:172.21.15.177:3300/0 conn(0x7fc16415f9d0 0x7fc16415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 >> v1:172.21.15.177:6789/0 conn(0x7fc164144c00 legacy=0x7fc16406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16406d790 con 0x7fc164070b10 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fc15c001670 con 0x7fc164070b10 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 >> v1:172.21.15.114:6789/0 conn(0x7fc164070b10 legacy=0x7fc164070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc148003a00 0x7fc148003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148004800 0x7fc148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc14800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16406d4b0 con 0x7fc1480051c0 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14800c0c0 con 0x7fc148004800 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.521+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14800c2c0 con 0x7fc148003a00 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc168e00640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc14800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc15b400640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148004800 0x7fc148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc163e00640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc148003a00 0x7fc148003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc168e00640 1 -- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc148003a00 msgr2=0x7fc148003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc168e00640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc148003a00 0x7fc148003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc168e00640 1 -- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148004800 msgr2=0x7fc148004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc168e00640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc148004800 0x7fc148004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.522+0000 7fc168e00640 1 -- 172.21.15.114:0/305573097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148003520 con 0x7fc1480051c0 2024-03-06T07:39:16.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.524+0000 7fc168e00640 1 -- 172.21.15.114:0/305573097 reap_dead start 2024-03-06T07:39:16.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.524+0000 7fc168e00640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc14800b9c0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c190 tx=0x7fc148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba347376aa550ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:16.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.524+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c007c60 con 0x7fc1480051c0 2024-03-06T07:39:16.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.524+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c030070 con 0x7fc1480051c0 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.524+0000 7fc163400640 1 -- 172.21.15.114:0/305573097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c005d50 con 0x7fc1480051c0 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.525+0000 7fc16ae00640 1 -- 172.21.15.114:0/305573097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 msgr2=0x7fc14800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.525+0000 7fc16ae00640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc14800b9c0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c190 tx=0x7fc148005830 comp rx=0 tx=0).stop 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.525+0000 7fc16ae00640 1 -- 172.21.15.114:0/305573097 shutdown_connections 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.525+0000 7fc16ae00640 1 --2- 172.21.15.114:0/305573097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc14800b9c0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.525+0000 7fc16ae00640 1 -- 172.21.15.114:0/305573097 >> 172.21.15.114:0/305573097 conn(0x7fc1640b1730 msgr2=0x7fc1640b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.525+0000 7fc16ae00640 1 -- 172.21.15.114:0/305573097 shutdown_connections 2024-03-06T07:39:16.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.525+0000 7fc16ae00640 1 -- 172.21.15.114:0/305573097 wait complete. 2024-03-06T07:39:16.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.527+0000 7fc16ae00640 1 Processor -- start 2024-03-06T07:39:16.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.527+0000 7fc16ae00640 1 -- start start 2024-03-06T07:39:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc16ae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc1640ae480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc16ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1640b4360 0x7fc1641534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc16ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1640ceb40 0x7fc1640ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc16ae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16406ccd0 con 0x7fc1480051c0 2024-03-06T07:39:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc16ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16406cb50 con 0x7fc1640ceb40 2024-03-06T07:39:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc16ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16406df90 con 0x7fc1640b4360 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc1640ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc1640ae480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51120/0 (socket says 172.21.15.114:51120) 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 -- 172.21.15.114:0/3987374451 learned_addr learned my addr 172.21.15.114:0/3987374451 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 -- 172.21.15.114:0/3987374451 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1640b4360 msgr2=0x7fc1641534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1640b4360 0x7fc1641534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 -- 172.21.15.114:0/3987374451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1640ceb40 msgr2=0x7fc1640ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc168e00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1640ceb40 0x7fc1640ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1640ceb40 0x7fc1640ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.528+0000 7fc163e00640 1 -- 172.21.15.114:0/3987374451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1640b4b40 con 0x7fc1480051c0 2024-03-06T07:39:16.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.529+0000 7fc163e00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc1640ae480 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fc15400e850 tx=0x7fc15400ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:16.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.529+0000 7fc160c00640 1 -- 172.21.15.114:0/3987374451 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154003b80 con 0x7fc1480051c0 2024-03-06T07:39:16.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.529+0000 7fc160c00640 1 -- 172.21.15.114:0/3987374451 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc154018070 con 0x7fc1480051c0 2024-03-06T07:39:16.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.529+0000 7fc160c00640 1 -- 172.21.15.114:0/3987374451 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1540050d0 con 0x7fc1480051c0 2024-03-06T07:39:16.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.529+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1640b4860 con 0x7fc1480051c0 2024-03-06T07:39:16.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.529+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1640b3930 con 0x7fc1480051c0 2024-03-06T07:39:16.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.531+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16404f330 con 0x7fc1480051c0 2024-03-06T07:39:16.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.531+0000 7fc160c00640 1 -- 172.21.15.114:0/3987374451 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc154005270 con 0x7fc1480051c0 2024-03-06T07:39:16.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.532+0000 7fc160c00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc14006b500 0x7fc14006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:16.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.532+0000 7fc160c00640 1 -- 172.21.15.114:0/3987374451 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc15408a1a0 con 0x7fc1480051c0 2024-03-06T07:39:16.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.532+0000 7fc15b400640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc14006b500 0x7fc14006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:16.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.533+0000 7fc15b400640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc14006b500 0x7fc14006d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc164153ce0 tx=0x7fc14c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:16.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.537+0000 7fc160c00640 1 -- 172.21.15.114:0/3987374451 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc154058f50 con 0x7fc1480051c0 2024-03-06T07:39:16.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.788+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc164072890 con 0x7fc1480051c0 2024-03-06T07:39:16.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.789+0000 7fc160c00640 1 -- 172.21.15.114:0/3987374451 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fc15401be40 con 0x7fc1480051c0 2024-03-06T07:39:16.792 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:16.792 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:16.793 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc14006b500 msgr2=0x7fc14006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc14006b500 0x7fc14006d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc164153ce0 tx=0x7fc14c008040 comp rx=0 tx=0).stop 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 msgr2=0x7fc1640ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc1640ae480 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fc15400e850 tx=0x7fc15400ed10 comp rx=0 tx=0).stop 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 shutdown_connections 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1640ceb40 0x7fc1640ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fc1640b4360 0x7fc1641534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fc14006b500 0x7fc14006d9c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 --2- 172.21.15.114:0/3987374451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1480051c0 0x7fc1640ae480 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:16.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 >> 172.21.15.114:0/3987374451 conn(0x7fc1640b1730 msgr2=0x7fc164143ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:16.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 shutdown_connections 2024-03-06T07:39:16.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:16.792+0000 7fc16ae00640 1 -- 172.21.15.114:0/3987374451 wait complete. 2024-03-06T07:39:17.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:17.017 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:17.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:17.874 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:18.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:18.179 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:18.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.222+0000 7f9e72e00640 1 Processor -- start 2024-03-06T07:39:18.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.223+0000 7f9e72e00640 1 -- start start 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e72e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f9e6c0cff20 0x7f9e6c0b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e72e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e6c0b7860 0x7f9e6c0d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e72e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c0b5e00 con 0x7f9e6c0b73e0 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e72e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c0b6a10 con 0x7f9e6c0b7860 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e72e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c0b60e0 con 0x7f9e6c0cff20 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e70e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9e6c0b7860 0x7f9e6c0d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e6b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9e6c0b73e0 0x7f9e6c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36298/0 (socket says 172.21.15.114:36298) 2024-03-06T07:39:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.224+0000 7f9e6b400640 1 -- 172.21.15.114:0/1355705071 learned_addr learned my addr 172.21.15.114:0/1355705071 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:18.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.225+0000 7f9e6be00640 1 --2- 172.21.15.114:0/1355705071 >> v2:172.21.15.177:3300/0 conn(0x7f9e6c0cff20 0x7f9e6c0b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.225+0000 7f9e70e00640 1 -- 172.21.15.114:0/1355705071 >> v2:172.21.15.177:3300/0 conn(0x7f9e6c0cff20 msgr2=0x7f9e6c0b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:18.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.225+0000 7f9e70e00640 1 --2- 172.21.15.114:0/1355705071 >> v2:172.21.15.177:3300/0 conn(0x7f9e6c0cff20 0x7f9e6c0b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.225+0000 7f9e70e00640 1 -- 172.21.15.114:0/1355705071 >> v1:172.21.15.114:6789/0 conn(0x7f9e6c0b73e0 legacy=0x7f9e6c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.225+0000 7f9e70e00640 1 -- 172.21.15.114:0/1355705071 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c0b63c0 con 0x7f9e6c0b7860 2024-03-06T07:39:18.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.226+0000 7f9e70e00640 1 --2- 172.21.15.114:0/1355705071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0b7860 0x7f9e6c0d0a10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c071f30 tx=0x7f9e6402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7abdc4d4991706 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.226+0000 7f9e6aa00640 1 -- 172.21.15.114:0/1355705071 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e64039070 con 0x7f9e6c0b7860 2024-03-06T07:39:18.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.226+0000 7f9e6aa00640 1 -- 172.21.15.114:0/1355705071 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6402cc80 con 0x7f9e6c0b7860 2024-03-06T07:39:18.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.226+0000 7f9e6aa00640 1 -- 172.21.15.114:0/1355705071 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e64034660 con 0x7f9e6c0b7860 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 -- 172.21.15.114:0/1355705071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0b7860 msgr2=0x7f9e6c0d0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 --2- 172.21.15.114:0/1355705071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0b7860 0x7f9e6c0d0a10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c071f30 tx=0x7f9e6402bed0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 -- 172.21.15.114:0/1355705071 shutdown_connections 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 --2- 172.21.15.114:0/1355705071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0b7860 0x7f9e6c0d0a10 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 --2- 172.21.15.114:0/1355705071 >> v2:172.21.15.177:3300/0 conn(0x7f9e6c0cff20 0x7f9e6c0b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 -- 172.21.15.114:0/1355705071 >> 172.21.15.114:0/1355705071 conn(0x7f9e6c0d1c90 msgr2=0x7f9e6c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 -- 172.21.15.114:0/1355705071 shutdown_connections 2024-03-06T07:39:18.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.227+0000 7f9e72e00640 1 -- 172.21.15.114:0/1355705071 wait complete. 2024-03-06T07:39:18.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.230+0000 7f9e72e00640 1 Processor -- start 2024-03-06T07:39:18.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.230+0000 7f9e72e00640 1 -- start start 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e72e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e6c0b73e0 0x7f9e6c15df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e72e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6c0b7860 0x7f9e6c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e72e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0cff20 0x7f9e6c157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e72e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c161a50 con 0x7f9e6c0cff20 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e72e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0b5af0 con 0x7f9e6c0b73e0 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e72e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0d0f50 con 0x7f9e6c0b7860 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0cff20 0x7f9e6c157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0cff20 0x7f9e6c157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57626/0 (socket says 172.21.15.114:57626) 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 -- 172.21.15.114:0/3094015985 learned_addr learned my addr 172.21.15.114:0/3094015985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e6be00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e6c0b73e0 0x7f9e6c15df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e6b400640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6c0b7860 0x7f9e6c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 -- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6c0b7860 msgr2=0x7f9e6c154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6c0b7860 0x7f9e6c154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 -- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e6c0b73e0 msgr2=0x7f9e6c15df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e6c0b73e0 0x7f9e6c15df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.232+0000 7f9e70e00640 1 -- 172.21.15.114:0/3094015985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c155750 con 0x7f9e6c0cff20 2024-03-06T07:39:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.233+0000 7f9e70e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0cff20 0x7f9e6c157f10 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009e80 tx=0x7f9e64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.233+0000 7f9e63e00640 1 -- 172.21.15.114:0/3094015985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e64044020 con 0x7f9e6c0cff20 2024-03-06T07:39:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.233+0000 7f9e63e00640 1 -- 172.21.15.114:0/3094015985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6403f470 con 0x7f9e6c0cff20 2024-03-06T07:39:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.233+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c15e670 con 0x7f9e6c0cff20 2024-03-06T07:39:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.233+0000 7f9e63e00640 1 -- 172.21.15.114:0/3094015985 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e64039040 con 0x7f9e6c0cff20 2024-03-06T07:39:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.233+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c15ec30 con 0x7f9e6c0cff20 2024-03-06T07:39:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.234+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c04f330 con 0x7f9e6c0cff20 2024-03-06T07:39:18.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.235+0000 7f9e63e00640 1 -- 172.21.15.114:0/3094015985 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e64047020 con 0x7f9e6c0cff20 2024-03-06T07:39:18.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.237+0000 7f9e63e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4c06b4d0 0x7f9e4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:18.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.237+0000 7f9e63e00640 1 -- 172.21.15.114:0/3094015985 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e640acdc0 con 0x7f9e6c0cff20 2024-03-06T07:39:18.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.237+0000 7f9e6be00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4c06b4d0 0x7f9e4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:18.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.238+0000 7f9e6be00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4c06b4d0 0x7f9e4c06d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c006fd0 tx=0x7f9e5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:18.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.241+0000 7f9e63e00640 1 -- 172.21.15.114:0/3094015985 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e6404e030 con 0x7f9e6c0cff20 2024-03-06T07:39:18.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.479+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e6c06c8c0 con 0x7f9e6c0cff20 2024-03-06T07:39:18.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.480+0000 7f9e63e00640 1 -- 172.21.15.114:0/3094015985 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f9e6c155750 con 0x7f9e6c0cff20 2024-03-06T07:39:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:18.483 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:18.483 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4c06b4d0 msgr2=0x7f9e4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4c06b4d0 0x7f9e4c06d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c006fd0 tx=0x7f9e5c008040 comp rx=0 tx=0).stop 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0cff20 msgr2=0x7f9e6c157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0cff20 0x7f9e6c157f10 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009e80 tx=0x7f9e64004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 shutdown_connections 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f9e4c06b4d0 0x7f9e4c06d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e6c0cff20 0x7f9e6c157f10 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f9e6c0b7860 0x7f9e6c154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 --2- 172.21.15.114:0/3094015985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e6c0b73e0 0x7f9e6c15df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:18.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 >> 172.21.15.114:0/3094015985 conn(0x7f9e6c0d1c90 msgr2=0x7f9e6c0c6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:18.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 shutdown_connections 2024-03-06T07:39:18.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:18.483+0000 7f9e72e00640 1 -- 172.21.15.114:0/3094015985 wait complete. 2024-03-06T07:39:18.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:18.705 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:19.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.915+0000 7f53b1000640 1 Processor -- start 2024-03-06T07:39:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.915+0000 7f53b1000640 1 -- start start 2024-03-06T07:39:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53b1000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f53ac144c00 0x7f53ac06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53b1000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 0x7f53ac0ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53b1000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ac06ee70 con 0x7f53ac15f9d0 2024-03-06T07:39:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53b1000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ac06cef0 con 0x7f53ac144c00 2024-03-06T07:39:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53b1000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac06d1d0 con 0x7f53ac15fe60 2024-03-06T07:39:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53a9400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f53ac15f9d0 0x7f53ac0ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36304/0 (socket says 172.21.15.114:36304) 2024-03-06T07:39:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53a9400640 1 -- 172.21.15.114:0/2120625502 learned_addr learned my addr 172.21.15.114:0/2120625502 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53aa800640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 0x7f53ac0ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53a9e00640 1 --2- 172.21.15.114:0/2120625502 >> v2:172.21.15.177:3300/0 conn(0x7f53ac144c00 0x7f53ac06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.917+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373542894 0 0) 0x7f5398001110 con 0x7f53ac15f9d0 2024-03-06T07:39:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5390003520 con 0x7f53ac15f9d0 2024-03-06T07:39:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3514024246 0 0) 0x7f5390003520 con 0x7f53ac15f9d0 2024-03-06T07:39:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 msgr2=0x7f53ac0ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 0x7f53ac0ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 >> v2:172.21.15.177:3300/0 conn(0x7f53ac144c00 msgr2=0x7f53ac06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 --2- 172.21.15.114:0/2120625502 >> v2:172.21.15.177:3300/0 conn(0x7f53ac144c00 0x7f53ac06e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac06d4b0 con 0x7f53ac15f9d0 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5398001840 con 0x7f53ac15f9d0 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 >> v1:172.21.15.114:6789/0 conn(0x7f53ac15f9d0 legacy=0x7f53ac0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53900039b0 0x7f5390003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 0x7f5390004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390004d40 0x7f53900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac06cef0 con 0x7f5390004d40 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac06d1d0 con 0x7f53900039b0 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.918+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539000b9c0 con 0x7f53ac15fe60 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53aa800640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390004d40 0x7f53900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53a9400640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 0x7f5390004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53a9e00640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53900039b0 0x7f5390003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53aa800640 1 -- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 msgr2=0x7f5390004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53aa800640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac15fe60 0x7f5390004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53aa800640 1 -- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53900039b0 msgr2=0x7f5390003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53aa800640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53900039b0 0x7f5390003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.919+0000 7f53aa800640 1 -- 172.21.15.114:0/2120625502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390003520 con 0x7f5390004d40 2024-03-06T07:39:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.920+0000 7f53aa800640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390004d40 0x7f53900051c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009fc0 tx=0x7f5390005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ca27c65ccde436 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.921+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4039070 con 0x7f5390004d40 2024-03-06T07:39:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.921+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a402cc20 con 0x7f5390004d40 2024-03-06T07:39:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.921+0000 7f53a8a00640 1 -- 172.21.15.114:0/2120625502 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4009c10 con 0x7f5390004d40 2024-03-06T07:39:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.921+0000 7f53b1000640 1 -- 172.21.15.114:0/2120625502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390004d40 msgr2=0x7f53900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.921+0000 7f53b1000640 1 --2- 172.21.15.114:0/2120625502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5390004d40 0x7f53900051c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009fc0 tx=0x7f5390005830 comp rx=0 tx=0).stop 2024-03-06T07:39:19.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.921+0000 7f53aa800640 1 -- 172.21.15.114:0/2120625502 reap_dead start 2024-03-06T07:39:19.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.922+0000 7f53b1000640 1 -- 172.21.15.114:0/2120625502 shutdown_connections 2024-03-06T07:39:19.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.922+0000 7f53b1000640 1 -- 172.21.15.114:0/2120625502 >> 172.21.15.114:0/2120625502 conn(0x7f53ac0b1730 msgr2=0x7f53ac0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:19.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.922+0000 7f53b1000640 1 -- 172.21.15.114:0/2120625502 shutdown_connections 2024-03-06T07:39:19.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.922+0000 7f53b1000640 1 -- 172.21.15.114:0/2120625502 wait complete. 2024-03-06T07:39:19.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.923+0000 7f53b1000640 1 Processor -- start 2024-03-06T07:39:19.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.924+0000 7f53b1000640 1 -- start start 2024-03-06T07:39:19.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5390004d40 0x7f53ac138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53b1000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac139440 0x7f53ac142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53b1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac138e80 0x7f53ac142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53b1000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac160730 con 0x7f53ac138e80 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53b1000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac160290 con 0x7f5390004d40 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53b1000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac06cb50 con 0x7f53ac139440 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53aa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac138e80 0x7f53ac142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53aa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac138e80 0x7f53ac142760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57656/0 (socket says 172.21.15.114:57656) 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53aa800640 1 -- 172.21.15.114:0/2539406805 learned_addr learned my addr 172.21.15.114:0/2539406805 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53a9400640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac139440 0x7f53ac142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.925+0000 7f53a9e00640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5390004d40 0x7f53ac138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53aa800640 1 -- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac139440 msgr2=0x7f53ac142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53aa800640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac139440 0x7f53ac142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53aa800640 1 -- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5390004d40 msgr2=0x7f53ac138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53aa800640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5390004d40 0x7f53ac138680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53aa800640 1 -- 172.21.15.114:0/2539406805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac1e6370 con 0x7f53ac138e80 2024-03-06T07:39:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53aa800640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac138e80 0x7f53ac142760 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f53a4002230 tx=0x7f53a4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53a2000640 1 -- 172.21.15.114:0/2539406805 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4039030 con 0x7f53ac138e80 2024-03-06T07:39:19.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53a2000640 1 -- 172.21.15.114:0/2539406805 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a4043070 con 0x7f53ac138e80 2024-03-06T07:39:19.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ac1e6650 con 0x7f53ac138e80 2024-03-06T07:39:19.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.926+0000 7f53a2000640 1 -- 172.21.15.114:0/2539406805 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a403d7f0 con 0x7f53ac138e80 2024-03-06T07:39:19.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.927+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac1e6c10 con 0x7f53ac138e80 2024-03-06T07:39:19.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.928+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac04f330 con 0x7f53ac138e80 2024-03-06T07:39:19.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.929+0000 7f53a2000640 1 -- 172.21.15.114:0/2539406805 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53a403d990 con 0x7f53ac138e80 2024-03-06T07:39:19.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.930+0000 7f53a2000640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f538c06b4d0 0x7f538c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.930+0000 7f53a2000640 1 -- 172.21.15.114:0/2539406805 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f53a40ac950 con 0x7f53ac138e80 2024-03-06T07:39:19.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.931+0000 7f53a9e00640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f538c06b4d0 0x7f538c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.931+0000 7f53a9e00640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f538c06b4d0 0x7f538c06d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f539c009b00 tx=0x7f539c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:19.939+0000 7f53a2000640 1 -- 172.21.15.114:0/2539406805 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53a407b790 con 0x7f53ac138e80 2024-03-06T07:39:20.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.177+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53ac1443f0 con 0x7f53ac138e80 2024-03-06T07:39:20.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.178+0000 7f53a2000640 1 -- 172.21.15.114:0/2539406805 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f53a407b090 con 0x7f53ac138e80 2024-03-06T07:39:20.181 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:20.181 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:20.182 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:20.182 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:20.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f538c06b4d0 msgr2=0x7f538c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f538c06b4d0 0x7f538c06d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f539c009b00 tx=0x7f539c008040 comp rx=0 tx=0).stop 2024-03-06T07:39:20.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac138e80 msgr2=0x7f53ac142760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac138e80 0x7f53ac142760 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f53a4002230 tx=0x7f53a4009750 comp rx=0 tx=0).stop 2024-03-06T07:39:20.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 shutdown_connections 2024-03-06T07:39:20.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f538c06b4d0 0x7f538c06d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53ac138e80 0x7f53ac142760 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f53ac139440 0x7f53ac142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 --2- 172.21.15.114:0/2539406805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5390004d40 0x7f53ac138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 >> 172.21.15.114:0/2539406805 conn(0x7f53ac0b1730 msgr2=0x7f53ac0ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:20.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.181+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 shutdown_connections 2024-03-06T07:39:20.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:20.182+0000 7f53b1000640 1 -- 172.21.15.114:0/2539406805 wait complete. 2024-03-06T07:39:20.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:20.403 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:21.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.615+0000 7fb2dcc00640 1 Processor -- start 2024-03-06T07:39:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.615+0000 7fb2dcc00640 1 -- start start 2024-03-06T07:39:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.617+0000 7fb2dcc00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb2d8144c00 0x7fb2d806e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.617+0000 7fb2dcc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d806ee70 con 0x7fb2d815f9d0 2024-03-06T07:39:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.618+0000 7fb2dcc00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d806cef0 con 0x7fb2d815fe60 2024-03-06T07:39:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.618+0000 7fb2dcc00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d806d1d0 con 0x7fb2d8144c00 2024-03-06T07:39:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.618+0000 7fb2d5400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb2d815f9d0 0x7fb2d80ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36310/0 (socket says 172.21.15.114:36310) 2024-03-06T07:39:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.618+0000 7fb2d5400640 1 -- 172.21.15.114:0/462142053 learned_addr learned my addr 172.21.15.114:0/462142053 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.618+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/462142053 >> v2:172.21.15.177:3300/0 conn(0x7fb2d8144c00 0x7fb2d806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.618+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779513695 0 0) 0x7fb2c4001110 con 0x7fb2d815f9d0 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.618+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2bc003520 con 0x7fb2d815f9d0 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3259063786 0 0) 0x7fb2bc003520 con 0x7fb2d815f9d0 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 >> v2:172.21.15.177:3300/0 conn(0x7fb2d8144c00 msgr2=0x7fb2d806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 --2- 172.21.15.114:0/462142053 >> v2:172.21.15.177:3300/0 conn(0x7fb2d8144c00 0x7fb2d806e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 >> v1:172.21.15.177:6789/0 conn(0x7fb2d815fe60 legacy=0x7fb2d80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d806d4b0 con 0x7fb2d815f9d0 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb2c4001840 con 0x7fb2d815f9d0 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 >> v1:172.21.15.114:6789/0 conn(0x7fb2d815f9d0 legacy=0x7fb2d80ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2bc0039b0 0x7fb2bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2bc004800 0x7fb2bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0051c0 0x7fb2bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d806d1d0 con 0x7fb2bc0039b0 2024-03-06T07:39:21.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc00c060 con 0x7fb2bc004800 2024-03-06T07:39:21.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc00c260 con 0x7fb2bc0051c0 2024-03-06T07:39:21.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.619+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2bc0039b0 0x7fb2bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d6800640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0051c0 0x7fb2bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d5400640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2bc004800 0x7fb2bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d5e00640 1 -- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0051c0 msgr2=0x7fb2bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0051c0 0x7fb2bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d5e00640 1 -- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2bc004800 msgr2=0x7fb2bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2bc004800 0x7fb2bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d5e00640 1 -- 172.21.15.114:0/462142053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc003520 con 0x7fb2bc0039b0 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d6800640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0051c0 0x7fb2bc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d5400640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2bc004800 0x7fb2bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.620+0000 7fb2d6800640 1 -- 172.21.15.114:0/462142053 reap_dead start 2024-03-06T07:39:21.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.621+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2bc0039b0 0x7fb2bc003dd0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009ac0 tx=0x7fb2bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a543e3720305cbaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:21.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8039070 con 0x7fb2bc0039b0 2024-03-06T07:39:21.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c802c6b0 con 0x7fb2bc0039b0 2024-03-06T07:39:21.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2d4a00640 1 -- 172.21.15.114:0/462142053 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8009690 con 0x7fb2bc0039b0 2024-03-06T07:39:21.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2dcc00640 1 -- 172.21.15.114:0/462142053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2bc0039b0 msgr2=0x7fb2bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2bc0039b0 0x7fb2bc003dd0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009ac0 tx=0x7fb2bc005830 comp rx=0 tx=0).stop 2024-03-06T07:39:21.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2dcc00640 1 -- 172.21.15.114:0/462142053 shutdown_connections 2024-03-06T07:39:21.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/462142053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2bc0039b0 0x7fb2bc003dd0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2dcc00640 1 -- 172.21.15.114:0/462142053 >> 172.21.15.114:0/462142053 conn(0x7fb2d80b1730 msgr2=0x7fb2d80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:21.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.622+0000 7fb2dcc00640 1 -- 172.21.15.114:0/462142053 shutdown_connections 2024-03-06T07:39:21.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.623+0000 7fb2dcc00640 1 -- 172.21.15.114:0/462142053 wait complete. 2024-03-06T07:39:21.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.625+0000 7fb2dcc00640 1 Processor -- start 2024-03-06T07:39:21.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.625+0000 7fb2dcc00640 1 -- start start 2024-03-06T07:39:21.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2dcc00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0039b0 0x7fb2d815b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2dcc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2d815f0f0 0x7fb2d815d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2d815f570 0x7fb2d8131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2dcc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d8160730 con 0x7fb2d815f0f0 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2dcc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d8160270 con 0x7fb2d815f570 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2dcc00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d806cb50 con 0x7fb2bc0039b0 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2d5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2d815f0f0 0x7fb2d815d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2d5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2d815f0f0 0x7fb2d815d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57686/0 (socket says 172.21.15.114:57686) 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.627+0000 7fb2d5400640 1 -- 172.21.15.114:0/2793631200 learned_addr learned my addr 172.21.15.114:0/2793631200 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0039b0 0x7fb2d815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d6800640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2d815f570 0x7fb2d8131e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d5400640 1 -- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0039b0 msgr2=0x7fb2d815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d5400640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0039b0 0x7fb2d815b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d5400640 1 -- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2d815f570 msgr2=0x7fb2d8131e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d5400640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2d815f570 0x7fb2d8131e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d5400640 1 -- 172.21.15.114:0/2793631200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d815e050 con 0x7fb2d815f0f0 2024-03-06T07:39:21.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d6800640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2d815f570 0x7fb2d8131e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:21.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.628+0000 7fb2d5400640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2d815f0f0 0x7fb2d815d590 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d830 tx=0x7fb2c400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:21.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.629+0000 7fb2ce000640 1 -- 172.21.15.114:0/2793631200 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4019070 con 0x7fb2d815f0f0 2024-03-06T07:39:21.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.629+0000 7fb2ce000640 1 -- 172.21.15.114:0/2793631200 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c40074e0 con 0x7fb2d815f0f0 2024-03-06T07:39:21.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.629+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d815e330 con 0x7fb2d815f0f0 2024-03-06T07:39:21.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.629+0000 7fb2ce000640 1 -- 172.21.15.114:0/2793631200 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40049e0 con 0x7fb2d815f0f0 2024-03-06T07:39:21.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.629+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d815e8f0 con 0x7fb2d815f0f0 2024-03-06T07:39:21.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.630+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d804f330 con 0x7fb2d815f0f0 2024-03-06T07:39:21.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.632+0000 7fb2ce000640 1 -- 172.21.15.114:0/2793631200 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2c401c070 con 0x7fb2d815f0f0 2024-03-06T07:39:21.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.633+0000 7fb2ce000640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb2b806b4d0 0x7fb2b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:21.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.633+0000 7fb2ce000640 1 -- 172.21.15.114:0/2793631200 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb2c4089ca0 con 0x7fb2d815f0f0 2024-03-06T07:39:21.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.633+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb2b806b4d0 0x7fb2b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:21.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.634+0000 7fb2d5e00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb2b806b4d0 0x7fb2b806d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8036000 tx=0x7fb2c8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.638+0000 7fb2ce000640 1 -- 172.21.15.114:0/2793631200 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2d806ee70 con 0x7fb2d815f0f0 2024-03-06T07:39:21.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.872+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d8072890 con 0x7fb2d815f0f0 2024-03-06T07:39:21.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.873+0000 7fb2ce000640 1 -- 172.21.15.114:0/2793631200 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fb2c40583f0 con 0x7fb2d815f0f0 2024-03-06T07:39:21.875 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:21.876 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:21.876 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.876+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb2b806b4d0 msgr2=0x7fb2b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.876+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb2b806b4d0 0x7fb2b806d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8036000 tx=0x7fb2c8030000 comp rx=0 tx=0).stop 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2d815f0f0 msgr2=0x7fb2d815d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2d815f0f0 0x7fb2d815d590 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d830 tx=0x7fb2c400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 shutdown_connections 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2d815f570 0x7fb2d8131e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb2b806b4d0 0x7fb2b806d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2d815f0f0 0x7fb2d815d590 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 --2- 172.21.15.114:0/2793631200 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb2bc0039b0 0x7fb2d815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 >> 172.21.15.114:0/2793631200 conn(0x7fb2d80b1730 msgr2=0x7fb2d80af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 shutdown_connections 2024-03-06T07:39:21.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:21.877+0000 7fb2dcc00640 1 -- 172.21.15.114:0/2793631200 wait complete. 2024-03-06T07:39:22.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:22.101 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:22.183 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:23.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:23.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.323+0000 7ff6f9800640 1 Processor -- start 2024-03-06T07:39:23.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.323+0000 7ff6f9800640 1 -- start start 2024-03-06T07:39:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6f9800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff6f4161580 0x7ff6f40d1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6f9800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff6f40d2a70 0x7ff6f40b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6f9800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f4161960 con 0x7ff6f40d1e90 2024-03-06T07:39:23.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6f9800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f40cf6f0 con 0x7ff6f40d2a70 2024-03-06T07:39:23.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6f9800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f40cf9d0 con 0x7ff6f4161580 2024-03-06T07:39:23.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6f8e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff6f40d2a70 0x7ff6f40b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6eb400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff6f40d1e90 0x7ff6f40d2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36320/0 (socket says 172.21.15.114:36320) 2024-03-06T07:39:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.325+0000 7ff6eb400640 1 -- 172.21.15.114:0/370390072 learned_addr learned my addr 172.21.15.114:0/370390072 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.326+0000 7ff6f8e00640 1 -- 172.21.15.114:0/370390072 >> v2:172.21.15.177:3300/0 conn(0x7ff6f4161580 msgr2=0x7ff6f40d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.326+0000 7ff6f8e00640 1 --2- 172.21.15.114:0/370390072 >> v2:172.21.15.177:3300/0 conn(0x7ff6f4161580 0x7ff6f40d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.326+0000 7ff6f8e00640 1 -- 172.21.15.114:0/370390072 >> v1:172.21.15.114:6789/0 conn(0x7ff6f40d1e90 legacy=0x7ff6f40d2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.326+0000 7ff6f8e00640 1 -- 172.21.15.114:0/370390072 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40cfcb0 con 0x7ff6f40d2a70 2024-03-06T07:39:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.326+0000 7ff6f3400640 1 -- 172.21.15.114:0/370390072 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621395230 0 0) 0x7ff6f4161960 con 0x7ff6f40d1e90 2024-03-06T07:39:23.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.327+0000 7ff6f8e00640 1 --2- 172.21.15.114:0/370390072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d2a70 0x7ff6f40b3ec0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4073740 tx=0x7ff6ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eced866e88772c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:23.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.327+0000 7ff6f3400640 1 -- 172.21.15.114:0/370390072 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec035020 con 0x7ff6f40d2a70 2024-03-06T07:39:23.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.327+0000 7ff6f3400640 1 -- 172.21.15.114:0/370390072 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec031830 con 0x7ff6f40d2a70 2024-03-06T07:39:23.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.327+0000 7ff6f3400640 1 -- 172.21.15.114:0/370390072 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec031c00 con 0x7ff6f40d2a70 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 -- 172.21.15.114:0/370390072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d2a70 msgr2=0x7ff6f40b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 --2- 172.21.15.114:0/370390072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d2a70 0x7ff6f40b3ec0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4073740 tx=0x7ff6ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 -- 172.21.15.114:0/370390072 shutdown_connections 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 --2- 172.21.15.114:0/370390072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d2a70 0x7ff6f40b3ec0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 --2- 172.21.15.114:0/370390072 >> v2:172.21.15.177:3300/0 conn(0x7ff6f4161580 0x7ff6f40d1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 -- 172.21.15.114:0/370390072 >> 172.21.15.114:0/370390072 conn(0x7ff6f40d3d80 msgr2=0x7ff6f40d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 -- 172.21.15.114:0/370390072 shutdown_connections 2024-03-06T07:39:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.328+0000 7ff6f9800640 1 -- 172.21.15.114:0/370390072 wait complete. 2024-03-06T07:39:23.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.330+0000 7ff6f9800640 1 Processor -- start 2024-03-06T07:39:23.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.330+0000 7ff6f9800640 1 -- start start 2024-03-06T07:39:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d1e90 0x7ff6f415e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f9800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6f40d2a70 0x7ff6f41f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f4161580 0x7ff6f40c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f9800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40cf3a0 con 0x7ff6f40d1e90 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40b39a0 con 0x7ff6f4161580 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f9800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40b4400 con 0x7ff6f40d2a70 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d1e90 0x7ff6f415e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d1e90 0x7ff6f415e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57706/0 (socket says 172.21.15.114:57706) 2024-03-06T07:39:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f3e00640 1 -- 172.21.15.114:0/1961457766 learned_addr learned my addr 172.21.15.114:0/1961457766 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.331+0000 7ff6f3e00640 1 -- 172.21.15.114:0/1961457766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6f40d2a70 msgr2=0x7ff6f41f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f3e00640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6f40d2a70 0x7ff6f41f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f3e00640 1 -- 172.21.15.114:0/1961457766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f4161580 msgr2=0x7ff6f40c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f3e00640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f4161580 0x7ff6f40c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f3e00640 1 -- 172.21.15.114:0/1961457766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f41f7350 con 0x7ff6f40d1e90 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f3e00640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d1e90 0x7ff6f415e230 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400ba80 tx=0x7ff6e400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f0c00640 1 -- 172.21.15.114:0/1961457766 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4019070 con 0x7ff6f40d1e90 2024-03-06T07:39:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f0c00640 1 -- 172.21.15.114:0/1961457766 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e400cc20 con 0x7ff6f40d1e90 2024-03-06T07:39:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f41f7630 con 0x7ff6f40d1e90 2024-03-06T07:39:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f0c00640 1 -- 172.21.15.114:0/1961457766 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4014440 con 0x7ff6f40d1e90 2024-03-06T07:39:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.332+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f41f7bf0 con 0x7ff6f40d1e90 2024-03-06T07:39:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.333+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f404f330 con 0x7ff6f40d1e90 2024-03-06T07:39:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.335+0000 7ff6f0c00640 1 -- 172.21.15.114:0/1961457766 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6e401c070 con 0x7ff6f40d1e90 2024-03-06T07:39:23.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.335+0000 7ff6f0c00640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff6d006b4d0 0x7ff6d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:23.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.336+0000 7ff6f0c00640 1 -- 172.21.15.114:0/1961457766 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff6e4089f80 con 0x7ff6f40d1e90 2024-03-06T07:39:23.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.336+0000 7ff6eb400640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff6d006b4d0 0x7ff6d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:23.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.336+0000 7ff6eb400640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff6d006b4d0 0x7ff6d006d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc007fd0 tx=0x7ff6dc007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:23.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.345+0000 7ff6f0c00640 1 -- 172.21.15.114:0/1961457766 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6e40586d0 con 0x7ff6f40d1e90 2024-03-06T07:39:23.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.579+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f40b6970 con 0x7ff6f40d1e90 2024-03-06T07:39:23.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.580+0000 7ff6f0c00640 1 -- 172.21.15.114:0/1961457766 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7ff6e405c770 con 0x7ff6f40d1e90 2024-03-06T07:39:23.583 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:23.583 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:23.583 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff6d006b4d0 msgr2=0x7ff6d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff6d006b4d0 0x7ff6d006d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc007fd0 tx=0x7ff6dc007920 comp rx=0 tx=0).stop 2024-03-06T07:39:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d1e90 msgr2=0x7ff6f415e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d1e90 0x7ff6f415e230 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400ba80 tx=0x7ff6e400bf40 comp rx=0 tx=0).stop 2024-03-06T07:39:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 shutdown_connections 2024-03-06T07:39:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6f4161580 0x7ff6f40c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7ff6f40d2a70 0x7ff6f41f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7ff6d006b4d0 0x7ff6d006d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 --2- 172.21.15.114:0/1961457766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f40d1e90 0x7ff6f415e230 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.582+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 >> 172.21.15.114:0/1961457766 conn(0x7ff6f40d3d80 msgr2=0x7ff6f41601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.583+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 shutdown_connections 2024-03-06T07:39:23.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:23.583+0000 7ff6f9800640 1 -- 172.21.15.114:0/1961457766 wait complete. 2024-03-06T07:39:23.804 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:23.804 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:24.186 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:24.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:25.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.028+0000 7faf9ae00640 1 Processor -- start 2024-03-06T07:39:25.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.028+0000 7faf9ae00640 1 -- start start 2024-03-06T07:39:25.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.029+0000 7faf9ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 0x7faf94162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.029+0000 7faf9ae00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faf940b3ac0 0x7faf9406b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.029+0000 7faf9ae00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf94162b60 con 0x7faf940af820 2024-03-06T07:39:25.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.029+0000 7faf9ae00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9406b0f0 con 0x7faf940b3ac0 2024-03-06T07:39:25.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.029+0000 7faf9ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf940b5620 con 0x7faf9415e160 2024-03-06T07:39:25.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf98e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faf940b3ac0 0x7faf9406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf8b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7faf940af820 0x7faf940b33b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36326/0 (socket says 172.21.15.114:36326) 2024-03-06T07:39:25.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf8b400640 1 -- 172.21.15.114:0/1687027645 learned_addr learned my addr 172.21.15.114:0/1687027645 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:25.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf93e00640 1 --2- 172.21.15.114:0/1687027645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 0x7faf94162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf98e00640 1 -- 172.21.15.114:0/1687027645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 msgr2=0x7faf94162620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:25.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf98e00640 1 --2- 172.21.15.114:0/1687027645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 0x7faf94162620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf98e00640 1 -- 172.21.15.114:0/1687027645 >> v1:172.21.15.114:6789/0 conn(0x7faf940af820 legacy=0x7faf940b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf98e00640 1 -- 172.21.15.114:0/1687027645 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9406f1f0 con 0x7faf940b3ac0 2024-03-06T07:39:25.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.030+0000 7faf93400640 1 -- 172.21.15.114:0/1687027645 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406019365 0 0) 0x7faf7c001110 con 0x7faf940af820 2024-03-06T07:39:25.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.032+0000 7faf98e00640 1 --2- 172.21.15.114:0/1687027645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940b3ac0 0x7faf9406b580 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0097a0 tx=0x7faf8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b1dc381d1e2065 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:25.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.032+0000 7faf93400640 1 -- 172.21.15.114:0/1687027645 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0315a0 con 0x7faf940b3ac0 2024-03-06T07:39:25.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.032+0000 7faf93400640 1 -- 172.21.15.114:0/1687027645 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c037070 con 0x7faf940b3ac0 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.032+0000 7faf93400640 1 -- 172.21.15.114:0/1687027645 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c031960 con 0x7faf940b3ac0 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.032+0000 7faf9ae00640 1 -- 172.21.15.114:0/1687027645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940b3ac0 msgr2=0x7faf9406b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.032+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1687027645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940b3ac0 0x7faf9406b580 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0097a0 tx=0x7faf8c02b840 comp rx=0 tx=0).stop 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.033+0000 7faf9ae00640 1 -- 172.21.15.114:0/1687027645 shutdown_connections 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.033+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1687027645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940b3ac0 0x7faf9406b580 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.033+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1687027645 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 0x7faf94162620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.033+0000 7faf9ae00640 1 -- 172.21.15.114:0/1687027645 >> 172.21.15.114:0/1687027645 conn(0x7faf940d0340 msgr2=0x7faf940d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.033+0000 7faf9ae00640 1 -- 172.21.15.114:0/1687027645 shutdown_connections 2024-03-06T07:39:25.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.033+0000 7faf9ae00640 1 -- 172.21.15.114:0/1687027645 wait complete. 2024-03-06T07:39:25.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.034+0000 7faf9ae00640 1 Processor -- start 2024-03-06T07:39:25.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.035+0000 7faf9ae00640 1 -- start start 2024-03-06T07:39:25.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf9ae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940af820 0x7faf94160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf9ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf940b3ac0 0x7faf940cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf9ae00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 0x7faf9415e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf9ae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9406ef10 con 0x7faf940af820 2024-03-06T07:39:25.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf9ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9406ec30 con 0x7faf940b3ac0 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf9ae00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9406bac0 con 0x7faf9415e160 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940af820 0x7faf94160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940af820 0x7faf94160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57722/0 (socket says 172.21.15.114:57722) 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 -- 172.21.15.114:0/1153059909 learned_addr learned my addr 172.21.15.114:0/1153059909 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 -- 172.21.15.114:0/1153059909 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 msgr2=0x7faf9415e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 0x7faf9415e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 -- 172.21.15.114:0/1153059909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf940b3ac0 msgr2=0x7faf940cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf940b3ac0 0x7faf940cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.036+0000 7faf93e00640 1 -- 172.21.15.114:0/1153059909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940cd580 con 0x7faf940af820 2024-03-06T07:39:25.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.037+0000 7faf93e00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940af820 0x7faf94160af0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7faf84007e40 tx=0x7faf8400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:25.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.037+0000 7faf90c00640 1 -- 172.21.15.114:0/1153059909 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84014620 con 0x7faf940af820 2024-03-06T07:39:25.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.037+0000 7faf90c00640 1 -- 172.21.15.114:0/1153059909 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf84019070 con 0x7faf940af820 2024-03-06T07:39:25.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.037+0000 7faf90c00640 1 -- 172.21.15.114:0/1153059909 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84014a00 con 0x7faf940af820 2024-03-06T07:39:25.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.037+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9415ebd0 con 0x7faf940af820 2024-03-06T07:39:25.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.037+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf941e7d10 con 0x7faf940af820 2024-03-06T07:39:25.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.039+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf94074ae0 con 0x7faf940af820 2024-03-06T07:39:25.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.039+0000 7faf90c00640 1 -- 172.21.15.114:0/1153059909 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf84014ba0 con 0x7faf940af820 2024-03-06T07:39:25.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.041+0000 7faf90c00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf7006b4d0 0x7faf7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:25.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.041+0000 7faf90c00640 1 -- 172.21.15.114:0/1153059909 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faf8408aaa0 con 0x7faf940af820 2024-03-06T07:39:25.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.041+0000 7faf8b400640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf7006b4d0 0x7faf7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:25.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.042+0000 7faf8b400640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf7006b4d0 0x7faf7006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7faf941e9770 tx=0x7faf7c001310 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:25.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.046+0000 7faf90c00640 1 -- 172.21.15.114:0/1153059909 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf9415ebd0 con 0x7faf940af820 2024-03-06T07:39:25.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.282+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf940d0f90 con 0x7faf940af820 2024-03-06T07:39:25.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.283+0000 7faf90c00640 1 -- 172.21.15.114:0/1153059909 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7faf840598d0 con 0x7faf940af820 2024-03-06T07:39:25.286 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:25.286 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:25.287 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:25.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf7006b4d0 msgr2=0x7faf7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf7006b4d0 0x7faf7006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7faf941e9770 tx=0x7faf7c001310 comp rx=0 tx=0).stop 2024-03-06T07:39:25.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940af820 msgr2=0x7faf94160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:25.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940af820 0x7faf94160af0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7faf84007e40 tx=0x7faf8400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 shutdown_connections 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7faf9415e160 0x7faf9415e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf940b3ac0 0x7faf940cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7faf7006b4d0 0x7faf7006d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 --2- 172.21.15.114:0/1153059909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf940af820 0x7faf94160af0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 >> 172.21.15.114:0/1153059909 conn(0x7faf940d0340 msgr2=0x7faf9415c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 shutdown_connections 2024-03-06T07:39:25.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:25.286+0000 7faf9ae00640 1 -- 172.21.15.114:0/1153059909 wait complete. 2024-03-06T07:39:25.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:25.517 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:26.188 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:26.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:26.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.768+0000 7fb272e00640 1 Processor -- start 2024-03-06T07:39:26.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.768+0000 7fb272e00640 1 -- start start 2024-03-06T07:39:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.770+0000 7fb272e00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb26c0d2360 0x7fb26c0d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.770+0000 7fb272e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 0x7fb26c0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.770+0000 7fb272e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c161960 con 0x7fb26c161580 2024-03-06T07:39:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.770+0000 7fb272e00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c0d2ce0 con 0x7fb26c0d2360 2024-03-06T07:39:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.770+0000 7fb272e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0cf6f0 con 0x7fb26c0b3ec0 2024-03-06T07:39:26.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.770+0000 7fb271a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb26c161580 0x7fb26c0d1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36328/0 (socket says 172.21.15.114:36328) 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.770+0000 7fb271a00640 1 -- 172.21.15.114:0/1518600520 learned_addr learned my addr 172.21.15.114:0/1518600520 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639511867 0 0) 0x7fb26c161960 con 0x7fb26c161580 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb272400640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 0x7fb26c0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb250003520 con 0x7fb26c161580 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb271000640 1 --2- 172.21.15.114:0/1518600520 >> v2:172.21.15.177:3300/0 conn(0x7fb26c0d2360 0x7fb26c0d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1589795673 0 0) 0x7fb250003520 con 0x7fb26c161580 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 msgr2=0x7fb26c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 0x7fb26c0b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 >> v2:172.21.15.177:3300/0 conn(0x7fb26c0d2360 msgr2=0x7fb26c0d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 --2- 172.21.15.114:0/1518600520 >> v2:172.21.15.177:3300/0 conn(0x7fb26c0d2360 0x7fb26c0d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0cf9d0 con 0x7fb26c161580 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7fb25c001670 con 0x7fb26c161580 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb272400640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 0x7fb26c0b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 >> v1:172.21.15.114:6789/0 conn(0x7fb26c161580 legacy=0x7fb26c0d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 0x7fb250004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0d2ce0 con 0x7fb250004d40 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0cf6f0 con 0x7fb2500039b0 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25000ba40 con 0x7fb26c0b3ec0 2024-03-06T07:39:26.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.771+0000 7fb272400640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb271000640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 0x7fb250004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb271a00640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb272400640 1 -- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 msgr2=0x7fb250004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb272400640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b3ec0 0x7fb250004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb272400640 1 -- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2500039b0 msgr2=0x7fb250003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb272400640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb272400640 1 -- 172.21.15.114:0/1518600520 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250003520 con 0x7fb250004d40 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.772+0000 7fb271a00640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2500039b0 0x7fb250003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:26.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.773+0000 7fb272400640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fb2640073e0 tx=0x7fb250005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef4df77c1256f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.773+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264039070 con 0x7fb250004d40 2024-03-06T07:39:26.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.773+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26402cd30 con 0x7fb250004d40 2024-03-06T07:39:26.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.773+0000 7fb26be00640 1 -- 172.21.15.114:0/1518600520 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264009c80 con 0x7fb250004d40 2024-03-06T07:39:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.774+0000 7fb272e00640 1 -- 172.21.15.114:0/1518600520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 msgr2=0x7fb2500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.774+0000 7fb272e00640 1 --2- 172.21.15.114:0/1518600520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb2500051c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fb2640073e0 tx=0x7fb250005830 comp rx=0 tx=0).stop 2024-03-06T07:39:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.774+0000 7fb272400640 1 -- 172.21.15.114:0/1518600520 reap_dead start 2024-03-06T07:39:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.774+0000 7fb272e00640 1 -- 172.21.15.114:0/1518600520 shutdown_connections 2024-03-06T07:39:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.774+0000 7fb272e00640 1 -- 172.21.15.114:0/1518600520 >> 172.21.15.114:0/1518600520 conn(0x7fb26c0d3d80 msgr2=0x7fb26c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.774+0000 7fb272e00640 1 -- 172.21.15.114:0/1518600520 shutdown_connections 2024-03-06T07:39:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.774+0000 7fb272e00640 1 -- 172.21.15.114:0/1518600520 wait complete. 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.776+0000 7fb272e00640 1 Processor -- start 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.776+0000 7fb272e00640 1 -- start start 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb26c0cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb26c1f3aa0 0x7fb26c0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272e00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b6970 0x7fb26c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0b39a0 con 0x7fb250004d40 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0b36c0 con 0x7fb26c1f3aa0 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272e00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0cf3a0 con 0x7fb26c0b6970 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb26c0cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb26c0cc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57748/0 (socket says 172.21.15.114:57748) 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 -- 172.21.15.114:0/810193647 learned_addr learned my addr 172.21.15.114:0/810193647 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272400640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b6970 0x7fb26c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271000640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb26c1f3aa0 0x7fb26c0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 -- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b6970 msgr2=0x7fb26c15f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b6970 0x7fb26c15f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 -- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb26c1f3aa0 msgr2=0x7fb26c0c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb26c1f3aa0 0x7fb26c0c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb271a00640 1 -- 172.21.15.114:0/810193647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c1f3f60 con 0x7fb250004d40 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.777+0000 7fb272400640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b6970 0x7fb26c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.778+0000 7fb271a00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb26c0cc010 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00d8b0 tx=0x7fb25c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.778+0000 7fb269600640 1 -- 172.21.15.114:0/810193647 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c019070 con 0x7fb250004d40 2024-03-06T07:39:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.778+0000 7fb269600640 1 -- 172.21.15.114:0/810193647 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c0074e0 con 0x7fb250004d40 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.778+0000 7fb269600640 1 -- 172.21.15.114:0/810193647 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c007690 con 0x7fb250004d40 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.778+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c1f4240 con 0x7fb250004d40 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.778+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c1f4800 con 0x7fb250004d40 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.780+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c1f4a00 con 0x7fb250004d40 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.780+0000 7fb269600640 1 -- 172.21.15.114:0/810193647 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb25c0049e0 con 0x7fb250004d40 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.781+0000 7fb269600640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb24c06b4d0 0x7fb24c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.781+0000 7fb269600640 1 -- 172.21.15.114:0/810193647 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb25c089e10 con 0x7fb250004d40 2024-03-06T07:39:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.781+0000 7fb271000640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb24c06b4d0 0x7fb24c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.782+0000 7fb271000640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb24c06b4d0 0x7fb24c06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0b2fb0 tx=0x7fb258008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:26.785+0000 7fb269600640 1 -- 172.21.15.114:0/810193647 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb25c08d030 con 0x7fb250004d40 2024-03-06T07:39:27.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.022+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb26c04f330 con 0x7fb250004d40 2024-03-06T07:39:27.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.023+0000 7fb269600640 1 -- 172.21.15.114:0/810193647 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fb25c058bb0 con 0x7fb250004d40 2024-03-06T07:39:27.026 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:27.026 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:27.027 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:27.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb24c06b4d0 msgr2=0x7fb24c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb24c06b4d0 0x7fb24c06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0b2fb0 tx=0x7fb258008040 comp rx=0 tx=0).stop 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 msgr2=0x7fb26c0cc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb26c0cc010 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00d8b0 tx=0x7fb25c00dd70 comp rx=0 tx=0).stop 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 shutdown_connections 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fb26c0b6970 0x7fb26c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb26c1f3aa0 0x7fb26c0c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fb24c06b4d0 0x7fb24c06d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 --2- 172.21.15.114:0/810193647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb250004d40 0x7fb26c0cc010 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.025+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 >> 172.21.15.114:0/810193647 conn(0x7fb26c0d3d80 msgr2=0x7fb26c06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:27.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.026+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 shutdown_connections 2024-03-06T07:39:27.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:27.026+0000 7fb272e00640 1 -- 172.21.15.114:0/810193647 wait complete. 2024-03-06T07:39:27.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:27.255 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:28.190 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:28.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:28.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.457+0000 7fd1bc400640 1 Processor -- start 2024-03-06T07:39:28.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.457+0000 7fd1bc400640 1 -- start start 2024-03-06T07:39:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1bc400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd1b4144c00 0x7fd1b406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1bc400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd1b415fe60 0x7fd1b40ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1bc400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b406ee70 con 0x7fd1b415f9d0 2024-03-06T07:39:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1bc400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b406cef0 con 0x7fd1b415fe60 2024-03-06T07:39:28.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1bc400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b406d1d0 con 0x7fd1b4144c00 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1b9000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fd1b415f9d0 0x7fd1b40ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41480/0 (socket says 172.21.15.114:41480) 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1b9000640 1 -- 172.21.15.114:0/3629378038 learned_addr learned my addr 172.21.15.114:0/3629378038 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1ba400640 1 --2- 172.21.15.114:0/3629378038 >> v2:172.21.15.114:3300/0 conn(0x7fd1b415fe60 0x7fd1b40ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.459+0000 7fd1b9a00640 1 --2- 172.21.15.114:0/3629378038 >> v2:172.21.15.177:3300/0 conn(0x7fd1b4144c00 0x7fd1b406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.460+0000 7fd1b3e00640 1 -- 172.21.15.114:0/3629378038 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281448666 0 0) 0x7fd1a0001110 con 0x7fd1b415f9d0 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.460+0000 7fd1b3e00640 1 -- 172.21.15.114:0/3629378038 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19c003520 con 0x7fd1b415f9d0 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.460+0000 7fd1ba400640 1 -- 172.21.15.114:0/3629378038 >> v2:172.21.15.177:3300/0 conn(0x7fd1b4144c00 msgr2=0x7fd1b406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.460+0000 7fd1ba400640 1 --2- 172.21.15.114:0/3629378038 >> v2:172.21.15.177:3300/0 conn(0x7fd1b4144c00 0x7fd1b406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.460+0000 7fd1ba400640 1 -- 172.21.15.114:0/3629378038 >> v1:172.21.15.114:6789/0 conn(0x7fd1b415f9d0 legacy=0x7fd1b40ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.460+0000 7fd1ba400640 1 -- 172.21.15.114:0/3629378038 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b406d4b0 con 0x7fd1b415fe60 2024-03-06T07:39:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.460+0000 7fd1b9a00640 1 --2- 172.21.15.114:0/3629378038 >> v2:172.21.15.177:3300/0 conn(0x7fd1b4144c00 0x7fd1b406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.461+0000 7fd1ba400640 1 --2- 172.21.15.114:0/3629378038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415fe60 0x7fd1b40ce7b0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40b25f0 tx=0x7fd1ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=977246cb41450037 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.461+0000 7fd1b3e00640 1 -- 172.21.15.114:0/3629378038 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac039070 con 0x7fd1b415fe60 2024-03-06T07:39:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.461+0000 7fd1b3e00640 1 -- 172.21.15.114:0/3629378038 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ac02cc80 con 0x7fd1b415fe60 2024-03-06T07:39:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.461+0000 7fd1b3e00640 1 -- 172.21.15.114:0/3629378038 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac034640 con 0x7fd1b415fe60 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 -- 172.21.15.114:0/3629378038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415fe60 msgr2=0x7fd1b40ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 --2- 172.21.15.114:0/3629378038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415fe60 0x7fd1b40ce7b0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40b25f0 tx=0x7fd1ac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 -- 172.21.15.114:0/3629378038 shutdown_connections 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 --2- 172.21.15.114:0/3629378038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415fe60 0x7fd1b40ce7b0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 --2- 172.21.15.114:0/3629378038 >> v2:172.21.15.177:3300/0 conn(0x7fd1b4144c00 0x7fd1b406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 -- 172.21.15.114:0/3629378038 >> 172.21.15.114:0/3629378038 conn(0x7fd1b40b1730 msgr2=0x7fd1b40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 -- 172.21.15.114:0/3629378038 shutdown_connections 2024-03-06T07:39:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.462+0000 7fd1bc400640 1 -- 172.21.15.114:0/3629378038 wait complete. 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.464+0000 7fd1bc400640 1 Processor -- start 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.464+0000 7fd1bc400640 1 -- start start 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1bc400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1b4144c00 0x7fd1b413bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1bc400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415f9d0 0x7fd1b4150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1bc400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1b415fe60 0x7fd1b415da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1bc400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b4160730 con 0x7fd1b415f9d0 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1bc400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b4160270 con 0x7fd1b415fe60 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1bc400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b406cb50 con 0x7fd1b4144c00 2024-03-06T07:39:28.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1b9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415f9d0 0x7fd1b4150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1b9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415f9d0 0x7fd1b4150a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56620/0 (socket says 172.21.15.114:56620) 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.465+0000 7fd1b9000640 1 -- 172.21.15.114:0/985499581 learned_addr learned my addr 172.21.15.114:0/985499581 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b9000640 1 -- 172.21.15.114:0/985499581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1b4144c00 msgr2=0x7fd1b413bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1ba400640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1b415fe60 0x7fd1b415da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b9000640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1b4144c00 0x7fd1b413bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b9000640 1 -- 172.21.15.114:0/985499581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1b415fe60 msgr2=0x7fd1b415da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b9000640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1b415fe60 0x7fd1b415da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b9000640 1 -- 172.21.15.114:0/985499581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b413c4e0 con 0x7fd1b415f9d0 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b9000640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415f9d0 0x7fd1b4150a30 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000c760 tx=0x7fd1a000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:28.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b1600640 1 -- 172.21.15.114:0/985499581 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0017070 con 0x7fd1b415f9d0 2024-03-06T07:39:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b1600640 1 -- 172.21.15.114:0/985499581 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a0005ad0 con 0x7fd1b415f9d0 2024-03-06T07:39:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1b1600640 1 -- 172.21.15.114:0/985499581 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0007e20 con 0x7fd1b415f9d0 2024-03-06T07:39:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.466+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b413c7c0 con 0x7fd1b415f9d0 2024-03-06T07:39:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.467+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b413cd80 con 0x7fd1b415f9d0 2024-03-06T07:39:28.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.468+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b404f330 con 0x7fd1b415f9d0 2024-03-06T07:39:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.469+0000 7fd1b1600640 1 -- 172.21.15.114:0/985499581 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1a001a070 con 0x7fd1b415f9d0 2024-03-06T07:39:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.469+0000 7fd1b1600640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd19406b500 0x7fd19406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.470+0000 7fd1b1600640 1 -- 172.21.15.114:0/985499581 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1a00895b0 con 0x7fd1b415f9d0 2024-03-06T07:39:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.470+0000 7fd1b9a00640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd19406b500 0x7fd19406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:28.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.470+0000 7fd1b9a00640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd19406b500 0x7fd19406d9c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8006fd0 tx=0x7fd1a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:28.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.474+0000 7fd1b1600640 1 -- 172.21.15.114:0/985499581 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1b406ee70 con 0x7fd1b415f9d0 2024-03-06T07:39:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.705+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1b40b4b20 con 0x7fd1b415f9d0 2024-03-06T07:39:28.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.706+0000 7fd1b1600640 1 -- 172.21.15.114:0/985499581 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd1a0087e20 con 0x7fd1b415f9d0 2024-03-06T07:39:28.709 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:28.710 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:28.710 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:28.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd19406b500 msgr2=0x7fd19406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd19406b500 0x7fd19406d9c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8006fd0 tx=0x7fd1a8008040 comp rx=0 tx=0).stop 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415f9d0 msgr2=0x7fd1b4150a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415f9d0 0x7fd1b4150a30 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000c760 tx=0x7fd1a000cc20 comp rx=0 tx=0).stop 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 shutdown_connections 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1b415fe60 0x7fd1b415da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd19406b500 0x7fd19406d9c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1b415f9d0 0x7fd1b4150a30 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 --2- 172.21.15.114:0/985499581 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1b4144c00 0x7fd1b413bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 >> 172.21.15.114:0/985499581 conn(0x7fd1b40b1730 msgr2=0x7fd1b40b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 shutdown_connections 2024-03-06T07:39:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:28.709+0000 7fd1bc400640 1 -- 172.21.15.114:0/985499581 wait complete. 2024-03-06T07:39:28.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:28.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:29.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.144+0000 7fa866000640 1 Processor -- start 2024-03-06T07:39:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.144+0000 7fa866000640 1 -- start start 2024-03-06T07:39:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.145+0000 7fa866000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa8600b4680 0x7fa86006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.145+0000 7fa866000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa860072340 0x7fa86006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.145+0000 7fa866000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86015b090 con 0x7fa86015acb0 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa866000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8600b2a30 con 0x7fa8600b4680 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa866000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8600b4360 con 0x7fa860072340 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85f400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa86015acb0 0x7fa8600b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41482/0 (socket says 172.21.15.114:41482) 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85f400640 1 -- 172.21.15.114:0/3900287538 learned_addr learned my addr 172.21.15.114:0/3900287538 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85ea00640 1 --2- 172.21.15.114:0/3900287538 >> v2:172.21.15.114:3300/0 conn(0x7fa8600b4680 0x7fa86006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85e000640 1 -- 172.21.15.114:0/3900287538 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169974691 0 0) 0x7fa86015b090 con 0x7fa86015acb0 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85fe00640 1 --2- 172.21.15.114:0/3900287538 >> v2:172.21.15.177:3300/0 conn(0x7fa860072340 0x7fa86006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85e000640 1 -- 172.21.15.114:0/3900287538 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa844003520 con 0x7fa86015acb0 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85ea00640 1 -- 172.21.15.114:0/3900287538 >> v2:172.21.15.177:3300/0 conn(0x7fa860072340 msgr2=0x7fa86006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85ea00640 1 --2- 172.21.15.114:0/3900287538 >> v2:172.21.15.177:3300/0 conn(0x7fa860072340 0x7fa86006ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85ea00640 1 -- 172.21.15.114:0/3900287538 >> v1:172.21.15.114:6789/0 conn(0x7fa86015acb0 legacy=0x7fa8600b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.146+0000 7fa85ea00640 1 -- 172.21.15.114:0/3900287538 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600b2cf0 con 0x7fa8600b4680 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.148+0000 7fa85ea00640 1 --2- 172.21.15.114:0/3900287538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8600b4680 0x7fa86006e4f0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa8600b4e60 tx=0x7fa85002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ad8abc8629e01c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.148+0000 7fa85e000640 1 -- 172.21.15.114:0/3900287538 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850035020 con 0x7fa8600b4680 2024-03-06T07:39:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.148+0000 7fa85e000640 1 -- 172.21.15.114:0/3900287538 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa850031830 con 0x7fa8600b4680 2024-03-06T07:39:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.148+0000 7fa85e000640 1 -- 172.21.15.114:0/3900287538 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850031be0 con 0x7fa8600b4680 2024-03-06T07:39:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.148+0000 7fa866000640 1 -- 172.21.15.114:0/3900287538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8600b4680 msgr2=0x7fa86006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.148+0000 7fa866000640 1 --2- 172.21.15.114:0/3900287538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8600b4680 0x7fa86006e4f0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa8600b4e60 tx=0x7fa85002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.149+0000 7fa866000640 1 -- 172.21.15.114:0/3900287538 shutdown_connections 2024-03-06T07:39:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.149+0000 7fa866000640 1 --2- 172.21.15.114:0/3900287538 >> v2:172.21.15.177:3300/0 conn(0x7fa860072340 0x7fa86006ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.149+0000 7fa866000640 1 --2- 172.21.15.114:0/3900287538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8600b4680 0x7fa86006e4f0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.149+0000 7fa866000640 1 -- 172.21.15.114:0/3900287538 >> 172.21.15.114:0/3900287538 conn(0x7fa8600b1730 msgr2=0x7fa8600b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.149+0000 7fa866000640 1 -- 172.21.15.114:0/3900287538 shutdown_connections 2024-03-06T07:39:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.149+0000 7fa866000640 1 -- 172.21.15.114:0/3900287538 wait complete. 2024-03-06T07:39:30.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.150+0000 7fa866000640 1 Processor -- start 2024-03-06T07:39:30.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.151+0000 7fa866000640 1 -- start start 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.151+0000 7fa866000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa860072340 0x7fa8601f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.151+0000 7fa866000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8600b4680 0x7fa860144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa866000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa86015acb0 0x7fa86006e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa866000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600b4010 con 0x7fa860072340 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa866000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600b32b0 con 0x7fa86015acb0 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa866000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600ccfb0 con 0x7fa8600b4680 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa860072340 0x7fa8601f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa860072340 0x7fa8601f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56642/0 (socket says 172.21.15.114:56642) 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 -- 172.21.15.114:0/3680953932 learned_addr learned my addr 172.21.15.114:0/3680953932 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 -- 172.21.15.114:0/3680953932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8600b4680 msgr2=0x7fa860144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8600b4680 0x7fa860144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 -- 172.21.15.114:0/3680953932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa86015acb0 msgr2=0x7fa86006e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa86015acb0 0x7fa86006e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 -- 172.21.15.114:0/3680953932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860148760 con 0x7fa860072340 2024-03-06T07:39:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.152+0000 7fa85f400640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa860072340 0x7fa8601f7cc0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c390 tx=0x7fa84c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.153+0000 7fa857400640 1 -- 172.21.15.114:0/3680953932 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c017070 con 0x7fa860072340 2024-03-06T07:39:30.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.153+0000 7fa857400640 1 -- 172.21.15.114:0/3680953932 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c005ce0 con 0x7fa860072340 2024-03-06T07:39:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.153+0000 7fa857400640 1 -- 172.21.15.114:0/3680953932 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c0072f0 con 0x7fa860072340 2024-03-06T07:39:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.153+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa860148480 con 0x7fa860072340 2024-03-06T07:39:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.153+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8601f74c0 con 0x7fa860072340 2024-03-06T07:39:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.154+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86004f330 con 0x7fa860072340 2024-03-06T07:39:30.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.155+0000 7fa857400640 1 -- 172.21.15.114:0/3680953932 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa84c01a070 con 0x7fa860072340 2024-03-06T07:39:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.156+0000 7fa857400640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa84006b4d0 0x7fa84006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.156+0000 7fa857400640 1 -- 172.21.15.114:0/3680953932 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa84c089650 con 0x7fa860072340 2024-03-06T07:39:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.156+0000 7fa85ea00640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa84006b4d0 0x7fa84006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.157+0000 7fa85ea00640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa84006b4d0 0x7fa84006d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa850036000 tx=0x7fa850030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.161+0000 7fa857400640 1 -- 172.21.15.114:0/3680953932 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa84c058480 con 0x7fa860072340 2024-03-06T07:39:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:30.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.396+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa86006ce60 con 0x7fa860072340 2024-03-06T07:39:30.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.397+0000 7fa857400640 1 -- 172.21.15.114:0/3680953932 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fa84c057d80 con 0x7fa860072340 2024-03-06T07:39:30.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:30.400 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa84006b4d0 msgr2=0x7fa84006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa84006b4d0 0x7fa84006d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa850036000 tx=0x7fa850030000 comp rx=0 tx=0).stop 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa860072340 msgr2=0x7fa8601f7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa860072340 0x7fa8601f7cc0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c390 tx=0x7fa84c00c850 comp rx=0 tx=0).stop 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 shutdown_connections 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa86015acb0 0x7fa86006e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fa8600b4680 0x7fa860144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fa84006b4d0 0x7fa84006d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 --2- 172.21.15.114:0/3680953932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa860072340 0x7fa8601f7cc0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 >> 172.21.15.114:0/3680953932 conn(0x7fa8600b1730 msgr2=0x7fa860071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 shutdown_connections 2024-03-06T07:39:30.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:30.400+0000 7fa866000640 1 -- 172.21.15.114:0/3680953932 wait complete. 2024-03-06T07:39:30.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:30.622 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:31.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:31.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.837+0000 7fd149a00640 1 Processor -- start 2024-03-06T07:39:31.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.837+0000 7fd149a00640 1 -- start start 2024-03-06T07:39:31.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.840+0000 7fd149a00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd144151660 0x7fd144071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.840+0000 7fd149a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1440cf430 0x7fd14406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.840+0000 7fd149a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd144071a60 con 0x7fd14406d920 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.840+0000 7fd149a00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1440b3ee0 con 0x7fd144151660 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.840+0000 7fd149a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1440b35b0 con 0x7fd1440cf430 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.840+0000 7fd13a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1440cf430 0x7fd14406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd13a000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1440cf430 0x7fd14406d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3301/0 says I am v2:172.21.15.114:48896/0 (socket says 172.21.15.114:48896) 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd13a000640 1 -- 172.21.15.114:0/1259435015 learned_addr learned my addr 172.21.15.114:0/1259435015 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd142a00640 1 --2- 172.21.15.114:0/1259435015 >> v2:172.21.15.177:3300/0 conn(0x7fd144151660 0x7fd144071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd142a00640 1 -- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1440cf430 msgr2=0x7fd14406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd142a00640 1 --2- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1440cf430 0x7fd14406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd142a00640 1 -- 172.21.15.114:0/1259435015 >> v1:172.21.15.177:6789/0 conn(0x7fd14406d920 legacy=0x7fd14415dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd142a00640 1 -- 172.21.15.114:0/1259435015 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1440b3890 con 0x7fd144151660 2024-03-06T07:39:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.841+0000 7fd13a000640 1 --2- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1440cf430 0x7fd14406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:39:31.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.843+0000 7fd142a00640 1 --2- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd144071520 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd134009d90 tx=0x7fd13402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c9330959485a35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.843+0000 7fd142000640 1 -- 172.21.15.114:0/1259435015 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134039070 con 0x7fd144151660 2024-03-06T07:39:31.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.843+0000 7fd142000640 1 -- 172.21.15.114:0/1259435015 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13402cb30 con 0x7fd144151660 2024-03-06T07:39:31.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.843+0000 7fd142000640 1 -- 172.21.15.114:0/1259435015 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134034440 con 0x7fd144151660 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 -- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 msgr2=0x7fd144071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 --2- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd144071520 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd134009d90 tx=0x7fd13402bd80 comp rx=0 tx=0).stop 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 -- 172.21.15.114:0/1259435015 shutdown_connections 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 --2- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd1440cf430 0x7fd14406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 --2- 172.21.15.114:0/1259435015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd144071520 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 -- 172.21.15.114:0/1259435015 >> 172.21.15.114:0/1259435015 conn(0x7fd1440b1730 msgr2=0x7fd1440b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 -- 172.21.15.114:0/1259435015 shutdown_connections 2024-03-06T07:39:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.844+0000 7fd149a00640 1 -- 172.21.15.114:0/1259435015 wait complete. 2024-03-06T07:39:31.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.847+0000 7fd149a00640 1 Processor -- start 2024-03-06T07:39:31.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.847+0000 7fd149a00640 1 -- start start 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd149a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd14406d920 0x7fd144152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd149a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1440cf430 0x7fd14414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd149a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd14414fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd149a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14415e700 con 0x7fd1440cf430 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd149a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd144071d30 con 0x7fd144151660 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd149a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14415e880 con 0x7fd14406d920 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd13a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1440cf430 0x7fd14414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd13a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1440cf430 0x7fd14414d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56644/0 (socket says 172.21.15.114:56644) 2024-03-06T07:39:31.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.848+0000 7fd13a000640 1 -- 172.21.15.114:0/3729595600 learned_addr learned my addr 172.21.15.114:0/3729595600 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd143400640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd14414fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd142a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd14406d920 0x7fd144152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13a000640 1 -- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd14406d920 msgr2=0x7fd144152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13a000640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd14406d920 0x7fd144152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13a000640 1 -- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 msgr2=0x7fd14414fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13a000640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd14414fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13a000640 1 -- 172.21.15.114:0/3729595600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144153270 con 0x7fd1440cf430 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd143400640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd14414fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd142a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd14406d920 0x7fd144152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13a000640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1440cf430 0x7fd14414d8b0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00e950 tx=0x7fd12c00ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13b400640 1 -- 172.21.15.114:0/3729595600 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c00b040 con 0x7fd1440cf430 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13b400640 1 -- 172.21.15.114:0/3729595600 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12c00cd00 con 0x7fd1440cf430 2024-03-06T07:39:31.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd13b400640 1 -- 172.21.15.114:0/3729595600 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c004230 con 0x7fd1440cf430 2024-03-06T07:39:31.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.849+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd144151c90 con 0x7fd1440cf430 2024-03-06T07:39:31.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.850+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd144152250 con 0x7fd1440cf430 2024-03-06T07:39:31.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.851+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14404f330 con 0x7fd1440cf430 2024-03-06T07:39:31.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.852+0000 7fd13b400640 1 -- 172.21.15.114:0/3729595600 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd12c003610 con 0x7fd1440cf430 2024-03-06T07:39:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.852+0000 7fd13b400640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd12006b4d0 0x7fd12006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.852+0000 7fd13b400640 1 -- 172.21.15.114:0/3729595600 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd12c089f00 con 0x7fd1440cf430 2024-03-06T07:39:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.853+0000 7fd142a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd12006b4d0 0x7fd12006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.853+0000 7fd142a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd12006b4d0 0x7fd12006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd134036000 tx=0x7fd134030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:31.857+0000 7fd13b400640 1 -- 172.21.15.114:0/3729595600 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd12c016020 con 0x7fd1440cf430 2024-03-06T07:39:32.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.112+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd14406e630 con 0x7fd1440cf430 2024-03-06T07:39:32.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.113+0000 7fd13b400640 1 -- 172.21.15.114:0/3729595600 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fd12c058d30 con 0x7fd1440cf430 2024-03-06T07:39:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:32.116 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:32.116 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd12006b4d0 msgr2=0x7fd12006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd12006b4d0 0x7fd12006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd134036000 tx=0x7fd134030000 comp rx=0 tx=0).stop 2024-03-06T07:39:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1440cf430 msgr2=0x7fd14414d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1440cf430 0x7fd14414d8b0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00e950 tx=0x7fd12c00ee10 comp rx=0 tx=0).stop 2024-03-06T07:39:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 shutdown_connections 2024-03-06T07:39:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd144151660 0x7fd14414fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fd12006b4d0 0x7fd12006d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1440cf430 0x7fd14414d8b0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 --2- 172.21.15.114:0/3729595600 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fd14406d920 0x7fd144152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 >> 172.21.15.114:0/3729595600 conn(0x7fd1440b1730 msgr2=0x7fd14406dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 shutdown_connections 2024-03-06T07:39:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:32.116+0000 7fd149a00640 1 -- 172.21.15.114:0/3729595600 wait complete. 2024-03-06T07:39:32.193 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:32.337 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:32.337 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:33.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:33.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.554+0000 7fdbd0600640 1 Processor -- start 2024-03-06T07:39:33.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.554+0000 7fdbd0600640 1 -- start start 2024-03-06T07:39:33.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbd0600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdbc815eff0 0x7fdbc8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbd0600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc806e4d0 0x7fdbc806e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbd0600640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc8071d30 con 0x7fdbc80af1c0 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbd0600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc8071260 con 0x7fdbc815eff0 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbd0600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc815f4a0 con 0x7fdbc806e4d0 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbcd200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdbc815eff0 0x7fdbc8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbcd200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdbc815eff0 0x7fdbc8070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56652/0 (socket says 172.21.15.114:56652) 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbcd200640 1 -- 172.21.15.114:0/2177272973 learned_addr learned my addr 172.21.15.114:0/2177272973 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.556+0000 7fdbce600640 1 --2- 172.21.15.114:0/2177272973 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc806e4d0 0x7fdbc806e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.557+0000 7fdbcd200640 1 -- 172.21.15.114:0/2177272973 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc806e4d0 msgr2=0x7fdbc806e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.557+0000 7fdbcd200640 1 --2- 172.21.15.114:0/2177272973 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc806e4d0 0x7fdbc806e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.557+0000 7fdbcd200640 1 -- 172.21.15.114:0/2177272973 >> v1:172.21.15.177:6789/0 conn(0x7fdbc80af1c0 legacy=0x7fdbc806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.557+0000 7fdbcd200640 1 -- 172.21.15.114:0/2177272973 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc8071490 con 0x7fdbc815eff0 2024-03-06T07:39:33.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.558+0000 7fdbcd200640 1 --2- 172.21.15.114:0/2177272973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc8070d20 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40098a0 tx=0x7fdbb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb6930cce08971a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.558+0000 7fdbc7e00640 1 -- 172.21.15.114:0/2177272973 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4035020 con 0x7fdbc815eff0 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.558+0000 7fdbc7e00640 1 -- 172.21.15.114:0/2177272973 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb4031640 con 0x7fdbc815eff0 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.558+0000 7fdbc7e00640 1 -- 172.21.15.114:0/2177272973 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb402d040 con 0x7fdbc815eff0 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 -- 172.21.15.114:0/2177272973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 msgr2=0x7fdbc8070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 --2- 172.21.15.114:0/2177272973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc8070d20 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40098a0 tx=0x7fdbb402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 -- 172.21.15.114:0/2177272973 shutdown_connections 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 --2- 172.21.15.114:0/2177272973 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc806e4d0 0x7fdbc806e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 --2- 172.21.15.114:0/2177272973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc8070d20 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 -- 172.21.15.114:0/2177272973 >> 172.21.15.114:0/2177272973 conn(0x7fdbc80b1730 msgr2=0x7fdbc80b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 -- 172.21.15.114:0/2177272973 shutdown_connections 2024-03-06T07:39:33.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.559+0000 7fdbd0600640 1 -- 172.21.15.114:0/2177272973 wait complete. 2024-03-06T07:39:33.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.561+0000 7fdbd0600640 1 Processor -- start 2024-03-06T07:39:33.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.561+0000 7fdbd0600640 1 -- start start 2024-03-06T07:39:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbd0600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc806e4d0 0x7fdbc814b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbd0600640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc80af1c0 0x7fdbc80cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbd0600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc80cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbd0600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc80b30b0 con 0x7fdbc815eff0 2024-03-06T07:39:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbd0600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8071a50 con 0x7fdbc806e4d0 2024-03-06T07:39:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbd0600640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc806d370 con 0x7fdbc80af1c0 2024-03-06T07:39:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbce600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc80cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbce600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc80cc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56656/0 (socket says 172.21.15.114:56656) 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbce600640 1 -- 172.21.15.114:0/1581774074 learned_addr learned my addr 172.21.15.114:0/1581774074 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.562+0000 7fdbcd200640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc80af1c0 0x7fdbc80cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbce600640 1 -- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc80af1c0 msgr2=0x7fdbc80cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbce600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc80af1c0 0x7fdbc80cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbce600640 1 -- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc806e4d0 msgr2=0x7fdbc814b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbcdc00640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc806e4d0 0x7fdbc814b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbce600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc806e4d0 0x7fdbc814b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbce600640 1 -- 172.21.15.114:0/1581774074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80cdd80 con 0x7fdbc815eff0 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbcdc00640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc806e4d0 0x7fdbc814b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbce600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc80cc970 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0007e60 tx=0x7fdbc000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbc5600640 1 -- 172.21.15.114:0/1581774074 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0014420 con 0x7fdbc815eff0 2024-03-06T07:39:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbc5600640 1 -- 172.21.15.114:0/1581774074 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc0019070 con 0x7fdbc815eff0 2024-03-06T07:39:33.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc81f3ed0 con 0x7fdbc815eff0 2024-03-06T07:39:33.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbc5600640 1 -- 172.21.15.114:0/1581774074 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0014820 con 0x7fdbc815eff0 2024-03-06T07:39:33.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.563+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc80cd280 con 0x7fdbc815eff0 2024-03-06T07:39:33.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.565+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc804f330 con 0x7fdbc815eff0 2024-03-06T07:39:33.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.566+0000 7fdbc5600640 1 -- 172.21.15.114:0/1581774074 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbc00149c0 con 0x7fdbc815eff0 2024-03-06T07:39:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.567+0000 7fdbc5600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.567+0000 7fdbc5600640 1 -- 172.21.15.114:0/1581774074 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdbc008ab40 con 0x7fdbc815eff0 2024-03-06T07:39:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.568+0000 7fdbcdc00640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:33.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.569+0000 7fdbcdc00640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdba806b4d0 0x7fdba806d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fdbc815eb20 tx=0x7fdbbc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:33.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.572+0000 7fdbc5600640 1 -- 172.21.15.114:0/1581774074 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbc00598e0 con 0x7fdbc815eff0 2024-03-06T07:39:33.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.806+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbc80cdf50 con 0x7fdbc815eff0 2024-03-06T07:39:33.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.807+0000 7fdbc5600640 1 -- 172.21.15.114:0/1581774074 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fdbc00591e0 con 0x7fdbc815eff0 2024-03-06T07:39:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:33.809 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:33.810 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:33.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdba806b4d0 msgr2=0x7fdba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdba806b4d0 0x7fdba806d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fdbc815eb20 tx=0x7fdbbc009040 comp rx=0 tx=0).stop 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 msgr2=0x7fdbc80cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc80cc970 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0007e60 tx=0x7fdbc000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 shutdown_connections 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fdba806b4d0 0x7fdba806d990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdbc815eff0 0x7fdbc80cc970 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fdbc80af1c0 0x7fdbc80cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 --2- 172.21.15.114:0/1581774074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc806e4d0 0x7fdbc814b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 >> 172.21.15.114:0/1581774074 conn(0x7fdbc80b1730 msgr2=0x7fdbc8160eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:33.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 shutdown_connections 2024-03-06T07:39:33.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:33.810+0000 7fdbd0600640 1 -- 172.21.15.114:0/1581774074 wait complete. 2024-03-06T07:39:34.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:34.031 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:34.195 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:35.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:35.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.244+0000 7f52a9a00640 1 Processor -- start 2024-03-06T07:39:35.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.244+0000 7f52a9a00640 1 -- start start 2024-03-06T07:39:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.245+0000 7f52a9a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a4144c00 0x7f52a406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a9a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a406ee70 con 0x7f52a415f9d0 2024-03-06T07:39:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a9a00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a406cef0 con 0x7f52a415fe30 2024-03-06T07:39:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a9a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a406d1d0 con 0x7f52a4144c00 2024-03-06T07:39:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a2000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f52a415f9d0 0x7f52a40ce0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41484/0 (socket says 172.21.15.114:41484) 2024-03-06T07:39:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a2000640 1 -- 172.21.15.114:0/792248128 learned_addr learned my addr 172.21.15.114:0/792248128 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:35.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a2a00640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a4144c00 0x7f52a406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400289820 0 0) 0x7f5290001110 con 0x7f52a415f9d0 2024-03-06T07:39:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.246+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5288003520 con 0x7f52a415f9d0 2024-03-06T07:39:35.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487492942 0 0) 0x7f5288003520 con 0x7f52a415f9d0 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a4144c00 msgr2=0x7f52a406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a4144c00 0x7f52a406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 >> v1:172.21.15.177:6789/0 conn(0x7f52a415fe30 legacy=0x7f52a40ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a406d4b0 con 0x7f52a415f9d0 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5290001840 con 0x7f52a415f9d0 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 >> v1:172.21.15.114:6789/0 conn(0x7f52a415f9d0 legacy=0x7f52a40ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52880039e0 0x7f5288003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5288004800 0x7f5288004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a4144c00 0x7f52880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a406d1d0 con 0x7f5288004800 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528800ba70 con 0x7f52880039e0 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528800bc70 con 0x7f52a4144c00 2024-03-06T07:39:35.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.247+0000 7f52a2000640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5288004800 0x7f5288004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.248+0000 7f52a2a00640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52880039e0 0x7f5288003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.248+0000 7f52a2000640 1 -- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a4144c00 msgr2=0x7f52880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:35.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.248+0000 7f52a2000640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a4144c00 0x7f52880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.248+0000 7f52a2000640 1 -- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52880039e0 msgr2=0x7f5288003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:35.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.248+0000 7f52a2000640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52880039e0 0x7f5288003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.248+0000 7f52a2000640 1 -- 172.21.15.114:0/792248128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5288003520 con 0x7f5288004800 2024-03-06T07:39:35.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.248+0000 7f52a2a00640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52880039e0 0x7f5288003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.249+0000 7f52a2000640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5288004800 0x7f5288004c80 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f529000c490 tx=0x7f5288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d21e6d704ed8127 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.249+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000c0e0 con 0x7f5288004800 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.249+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5290030070 con 0x7f5288004800 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.249+0000 7f52a1600640 1 -- 172.21.15.114:0/792248128 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52900058d0 con 0x7f5288004800 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.250+0000 7f52a9a00640 1 -- 172.21.15.114:0/792248128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5288004800 msgr2=0x7f5288004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.250+0000 7f52a9a00640 1 --2- 172.21.15.114:0/792248128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5288004800 0x7f5288004c80 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f529000c490 tx=0x7f5288005830 comp rx=0 tx=0).stop 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.250+0000 7f52a3400640 1 -- 172.21.15.114:0/792248128 reap_dead start 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.250+0000 7f52a9a00640 1 -- 172.21.15.114:0/792248128 shutdown_connections 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.250+0000 7f52a9a00640 1 -- 172.21.15.114:0/792248128 >> 172.21.15.114:0/792248128 conn(0x7f52a40b1730 msgr2=0x7f52a40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.250+0000 7f52a9a00640 1 -- 172.21.15.114:0/792248128 shutdown_connections 2024-03-06T07:39:35.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.250+0000 7f52a9a00640 1 -- 172.21.15.114:0/792248128 wait complete. 2024-03-06T07:39:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.252+0000 7f52a9a00640 1 Processor -- start 2024-03-06T07:39:35.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.252+0000 7f52a9a00640 1 -- start start 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5288004800 0x7f52a40b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a9a00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a40c49a0 0x7f52a40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a40c4e20 0x7f52a40c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a9a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4160730 con 0x7f52a40c4e20 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a9a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4160280 con 0x7f5288004800 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a9a00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40ceec0 con 0x7f52a40c49a0 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a40c4e20 0x7f52a40c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a40c4e20 0x7f52a40c5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56670/0 (socket says 172.21.15.114:56670) 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.253+0000 7f52a3400640 1 -- 172.21.15.114:0/1035559557 learned_addr learned my addr 172.21.15.114:0/1035559557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a2000640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a40c49a0 0x7f52a40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a3400640 1 -- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a40c49a0 msgr2=0x7f52a40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a3400640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a40c49a0 0x7f52a40c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a2a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5288004800 0x7f52a40b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a3400640 1 -- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5288004800 msgr2=0x7f52a40b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a3400640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5288004800 0x7f52a40b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a3400640 1 -- 172.21.15.114:0/1035559557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a415c3e0 con 0x7f52a40c4e20 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a3400640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a40c4e20 0x7f52a40c5880 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f52a406e500 tx=0x7f529c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f529aa00640 1 -- 172.21.15.114:0/1035559557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c005d40 con 0x7f52a40c4e20 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f529aa00640 1 -- 172.21.15.114:0/1035559557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c00a5e0 con 0x7f52a40c4e20 2024-03-06T07:39:35.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f529aa00640 1 -- 172.21.15.114:0/1035559557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c00a9d0 con 0x7f52a40c4e20 2024-03-06T07:39:35.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a415c6c0 con 0x7f52a40c4e20 2024-03-06T07:39:35.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.254+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a415cc80 con 0x7f52a40c4e20 2024-03-06T07:39:35.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.256+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a404f330 con 0x7f52a40c4e20 2024-03-06T07:39:35.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.257+0000 7f529aa00640 1 -- 172.21.15.114:0/1035559557 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f529c011070 con 0x7f52a40c4e20 2024-03-06T07:39:35.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.259+0000 7f529aa00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f528406b4d0 0x7f528406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:35.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.259+0000 7f529aa00640 1 -- 172.21.15.114:0/1035559557 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f529c089530 con 0x7f52a40c4e20 2024-03-06T07:39:35.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.259+0000 7f52a2a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f528406b4d0 0x7f528406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:35.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.260+0000 7f52a2a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f528406b4d0 0x7f528406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f52940099e0 tx=0x7f5294008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:35.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.264+0000 7f529aa00640 1 -- 172.21.15.114:0/1035559557 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f529c019060 con 0x7f52a40c4e20 2024-03-06T07:39:35.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.499+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52a4072890 con 0x7f52a40c4e20 2024-03-06T07:39:35.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.500+0000 7f529aa00640 1 -- 172.21.15.114:0/1035559557 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f529c058370 con 0x7f52a40c4e20 2024-03-06T07:39:35.502 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:35.502 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:35.503 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f528406b4d0 msgr2=0x7f528406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f528406b4d0 0x7f528406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f52940099e0 tx=0x7f5294008040 comp rx=0 tx=0).stop 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a40c4e20 msgr2=0x7f52a40c5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a40c4e20 0x7f52a40c5880 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f52a406e500 tx=0x7f529c00cb90 comp rx=0 tx=0).stop 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 shutdown_connections 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f528406b4d0 0x7f528406d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52a40c4e20 0x7f52a40c5880 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f52a40c49a0 0x7f52a40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 --2- 172.21.15.114:0/1035559557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5288004800 0x7f52a40b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:35.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 >> 172.21.15.114:0/1035559557 conn(0x7f52a40b1730 msgr2=0x7f52a4142490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:35.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 shutdown_connections 2024-03-06T07:39:35.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:35.503+0000 7f52a9a00640 1 -- 172.21.15.114:0/1035559557 wait complete. 2024-03-06T07:39:35.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:35.725 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:36.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:36.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.936+0000 7fda7b000640 1 Processor -- start 2024-03-06T07:39:36.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.936+0000 7fda7b000640 1 -- start start 2024-03-06T07:39:36.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda7b000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fda7415f9d0 0x7fda7415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda7b000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda7b000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7406cef0 con 0x7fda74144c00 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda7b000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7406d1d0 con 0x7fda7415f9d0 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda7b000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7406d4b0 con 0x7fda74070b10 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda73400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fda7415f9d0 0x7fda7415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda73e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fda74144c00 0x7fda7406e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41500/0 (socket says 172.21.15.114:41500) 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.938+0000 7fda73e00640 1 -- 172.21.15.114:0/775181568 learned_addr learned my addr 172.21.15.114:0/775181568 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.939+0000 7fda79000640 1 --2- 172.21.15.114:0/775181568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.939+0000 7fda73400640 1 -- 172.21.15.114:0/775181568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 msgr2=0x7fda74070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.939+0000 7fda73400640 1 --2- 172.21.15.114:0/775181568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.939+0000 7fda73400640 1 -- 172.21.15.114:0/775181568 >> v1:172.21.15.114:6789/0 conn(0x7fda74144c00 legacy=0x7fda7406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.939+0000 7fda72a00640 1 -- 172.21.15.114:0/775181568 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639492645 0 0) 0x7fda7406cef0 con 0x7fda74144c00 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.939+0000 7fda73400640 1 -- 172.21.15.114:0/775181568 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7406d790 con 0x7fda7415f9d0 2024-03-06T07:39:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.939+0000 7fda79000640 1 --2- 172.21.15.114:0/775181568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:36.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.940+0000 7fda73400640 1 --2- 172.21.15.114:0/775181568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda7415f9d0 0x7fda7415fe60 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fda600097a0 tx=0x7fda6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d513e97e4840a51f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.940+0000 7fda72a00640 1 -- 172.21.15.114:0/775181568 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60035020 con 0x7fda7415f9d0 2024-03-06T07:39:36.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.940+0000 7fda72a00640 1 -- 172.21.15.114:0/775181568 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda60031830 con 0x7fda7415f9d0 2024-03-06T07:39:36.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.940+0000 7fda72a00640 1 -- 172.21.15.114:0/775181568 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60031c00 con 0x7fda7415f9d0 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 -- 172.21.15.114:0/775181568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda7415f9d0 msgr2=0x7fda7415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 --2- 172.21.15.114:0/775181568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda7415f9d0 0x7fda7415fe60 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fda600097a0 tx=0x7fda6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 -- 172.21.15.114:0/775181568 shutdown_connections 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 --2- 172.21.15.114:0/775181568 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 --2- 172.21.15.114:0/775181568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda7415f9d0 0x7fda7415fe60 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 -- 172.21.15.114:0/775181568 >> 172.21.15.114:0/775181568 conn(0x7fda740b1730 msgr2=0x7fda740b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 -- 172.21.15.114:0/775181568 shutdown_connections 2024-03-06T07:39:36.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.941+0000 7fda7b000640 1 -- 172.21.15.114:0/775181568 wait complete. 2024-03-06T07:39:36.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.943+0000 7fda7b000640 1 Processor -- start 2024-03-06T07:39:36.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.943+0000 7fda7b000640 1 -- start start 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda7b000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda7b000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda74144c00 0x7fda7413acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda7b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda7415f9d0 0x7fda7413b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda7b000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7406ccd0 con 0x7fda74144c00 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda7b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7406cb50 con 0x7fda7415f9d0 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda7b000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7406df90 con 0x7fda74070b10 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda73400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda74144c00 0x7fda7413acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda73400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda74144c00 0x7fda7413acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56688/0 (socket says 172.21.15.114:56688) 2024-03-06T07:39:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.944+0000 7fda73400640 1 -- 172.21.15.114:0/3538716241 learned_addr learned my addr 172.21.15.114:0/3538716241 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:36.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda73400640 1 -- 172.21.15.114:0/3538716241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 msgr2=0x7fda74139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda73e00640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda73400640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74139d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda73400640 1 -- 172.21.15.114:0/3538716241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda7415f9d0 msgr2=0x7fda7413b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:36.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda73400640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda7415f9d0 0x7fda7413b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda73400640 1 -- 172.21.15.114:0/3538716241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7413a290 con 0x7fda74144c00 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda73400640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda74144c00 0x7fda7413acc0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fda6002bee0 tx=0x7fda600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda6be00640 1 -- 172.21.15.114:0/3538716241 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60041070 con 0x7fda74144c00 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda6be00640 1 -- 172.21.15.114:0/3538716241 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6003a950 con 0x7fda74144c00 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7413d120 con 0x7fda74144c00 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.945+0000 7fda6be00640 1 -- 172.21.15.114:0/3538716241 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60035040 con 0x7fda74144c00 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.946+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7413d6e0 con 0x7fda74144c00 2024-03-06T07:39:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.947+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7404f330 con 0x7fda74144c00 2024-03-06T07:39:36.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.948+0000 7fda6be00640 1 -- 172.21.15.114:0/3538716241 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda60038020 con 0x7fda74144c00 2024-03-06T07:39:36.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.949+0000 7fda6be00640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda5406b4d0 0x7fda5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:36.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.949+0000 7fda6be00640 1 -- 172.21.15.114:0/3538716241 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fda600abcc0 con 0x7fda74144c00 2024-03-06T07:39:36.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.950+0000 7fda73e00640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda5406b4d0 0x7fda5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:36.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.950+0000 7fda73e00640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda5406b4d0 0x7fda5406d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fda64007fd0 tx=0x7fda64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:36.954+0000 7fda6be00640 1 -- 172.21.15.114:0/3538716241 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda6007ab00 con 0x7fda74144c00 2024-03-06T07:39:37.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.189+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda74072890 con 0x7fda74144c00 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.189+0000 7fda6be00640 1 -- 172.21.15.114:0/3538716241 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7fda6007a400 con 0x7fda74144c00 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:37.191 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:37.192 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda5406b4d0 msgr2=0x7fda5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda5406b4d0 0x7fda5406d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fda64007fd0 tx=0x7fda64009040 comp rx=0 tx=0).stop 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda74144c00 msgr2=0x7fda7413acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda74144c00 0x7fda7413acc0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fda6002bee0 tx=0x7fda600046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 shutdown_connections 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda7415f9d0 0x7fda7413b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7fda5406b4d0 0x7fda5406d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda74144c00 0x7fda7413acc0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 --2- 172.21.15.114:0/3538716241 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7fda74070b10 0x7fda74139d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 >> 172.21.15.114:0/3538716241 conn(0x7fda740b1730 msgr2=0x7fda740afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 shutdown_connections 2024-03-06T07:39:37.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:37.192+0000 7fda7b000640 1 -- 172.21.15.114:0/3538716241 wait complete. 2024-03-06T07:39:37.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:37.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:38.200 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:38.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:38.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.625+0000 7f42e8400640 1 Processor -- start 2024-03-06T07:39:38.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.625+0000 7f42e8400640 1 -- start start 2024-03-06T07:39:38.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.626+0000 7f42e8400640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f42e00cf3e0 0x7f42e00ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.626+0000 7f42e8400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00cabf0 con 0x7f42e0072680 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.626+0000 7f42e8400640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00ce900 con 0x7f42e0144660 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.626+0000 7f42e8400640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e00cebe0 con 0x7f42e00cf3e0 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.626+0000 7f42e5000640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f42e0072680 0x7f42e00c9ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55780/0 (socket says 172.21.15.114:55780) 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.626+0000 7f42e5000640 1 -- 172.21.15.114:0/2207810830 learned_addr learned my addr 172.21.15.114:0/2207810830 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723134360 0 0) 0x7f42e00cabf0 con 0x7f42e0072680 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c4003520 con 0x7f42e0072680 2024-03-06T07:39:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42e6400640 1 --2- 172.21.15.114:0/2207810830 >> v2:172.21.15.177:3300/0 conn(0x7f42e00cf3e0 0x7f42e00ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1854117470 0 0) 0x7f42c4003520 con 0x7f42e0072680 2024-03-06T07:39:38.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 >> v2:172.21.15.177:3300/0 conn(0x7f42e00cf3e0 msgr2=0x7f42e00ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:38.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 --2- 172.21.15.114:0/2207810830 >> v2:172.21.15.177:3300/0 conn(0x7f42e00cf3e0 0x7f42e00ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 >> v1:172.21.15.177:6789/0 conn(0x7f42e0144660 legacy=0x7f42e0071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e00ceec0 con 0x7f42e0072680 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f42cc001670 con 0x7f42e0072680 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 >> v1:172.21.15.114:6789/0 conn(0x7f42e0072680 legacy=0x7f42e00c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.627+0000 7f42dfe00640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c4003a00 0x7f42c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42dfe00640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42c4004800 0x7f42c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42dfe00640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4005210 0x7f42c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00cebe0 con 0x7f42c4003a00 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c400c160 con 0x7f42c4005210 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c400c360 con 0x7f42c4004800 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5a00640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c4003a00 0x7f42c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e6400640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4005210 0x7f42c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5000640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42c4004800 0x7f42c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5a00640 1 -- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42c4004800 msgr2=0x7f42c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5a00640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42c4004800 0x7f42c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5a00640 1 -- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4005210 msgr2=0x7f42c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5a00640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4005210 0x7f42c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5a00640 1 -- 172.21.15.114:0/2207810830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c4003520 con 0x7f42c4003a00 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e6400640 1 -- 172.21.15.114:0/2207810830 reap_dead start 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.628+0000 7f42e5000640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42c4004800 0x7f42c4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.629+0000 7f42e5a00640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c4003a00 0x7f42c4003e20 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f42d0001750 tx=0x7f42c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962a53e4912c9320 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0001370 con 0x7f42c4003a00 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d00052d0 con 0x7f42c4003a00 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42dfe00640 1 -- 172.21.15.114:0/2207810830 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0003830 con 0x7f42c4003a00 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42e8400640 1 -- 172.21.15.114:0/2207810830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c4003a00 msgr2=0x7f42c4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42e8400640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c4003a00 0x7f42c4003e20 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f42d0001750 tx=0x7f42c4005830 comp rx=0 tx=0).stop 2024-03-06T07:39:38.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42e8400640 1 -- 172.21.15.114:0/2207810830 shutdown_connections 2024-03-06T07:39:38.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42e8400640 1 --2- 172.21.15.114:0/2207810830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c4003a00 0x7f42c4003e20 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42e8400640 1 -- 172.21.15.114:0/2207810830 >> 172.21.15.114:0/2207810830 conn(0x7f42e00d0b90 msgr2=0x7f42e00d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:38.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42e8400640 1 -- 172.21.15.114:0/2207810830 shutdown_connections 2024-03-06T07:39:38.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.630+0000 7f42e8400640 1 -- 172.21.15.114:0/2207810830 wait complete. 2024-03-06T07:39:38.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.633+0000 7f42e8400640 1 Processor -- start 2024-03-06T07:39:38.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.634+0000 7f42e8400640 1 -- start start 2024-03-06T07:39:38.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.634+0000 7f42e8400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4003a00 0x7f42e00b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.634+0000 7f42e8400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015e530 0x7f42e015d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.634+0000 7f42e8400640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e006f2e0 0x7f42e006f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.634+0000 7f42e8400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00720f0 con 0x7f42e015e530 2024-03-06T07:39:38.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.634+0000 7f42e8400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0071f70 con 0x7f42c4003a00 2024-03-06T07:39:38.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.634+0000 7f42e8400640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0072270 con 0x7f42e006f2e0 2024-03-06T07:39:38.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015e530 0x7f42e015d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015e530 0x7f42e015d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46392/0 (socket says 172.21.15.114:46392) 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 -- 172.21.15.114:0/2654666013 learned_addr learned my addr 172.21.15.114:0/2654666013 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5a00640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4003a00 0x7f42e00b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e6400640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e006f2e0 0x7f42e006f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 -- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e006f2e0 msgr2=0x7f42e006f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e006f2e0 0x7f42e006f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 -- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4003a00 msgr2=0x7f42e00b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4003a00 0x7f42e00b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 -- 172.21.15.114:0/2654666013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e0139b60 con 0x7f42e015e530 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.635+0000 7f42e5000640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015e530 0x7f42e015d650 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000ff0 tx=0x7f42cc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.636+0000 7f42dd600640 1 -- 172.21.15.114:0/2654666013 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc019070 con 0x7f42e015e530 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.636+0000 7f42dd600640 1 -- 172.21.15.114:0/2654666013 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc0072d0 con 0x7f42e015e530 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.636+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e015e990 con 0x7f42e015e530 2024-03-06T07:39:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.636+0000 7f42dd600640 1 -- 172.21.15.114:0/2654666013 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0047e0 con 0x7f42e015e530 2024-03-06T07:39:38.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.636+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e015ef10 con 0x7f42e015e530 2024-03-06T07:39:38.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.637+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e0067830 con 0x7f42e015e530 2024-03-06T07:39:38.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.639+0000 7f42dd600640 1 -- 172.21.15.114:0/2654666013 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42cc01c070 con 0x7f42e015e530 2024-03-06T07:39:38.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.640+0000 7f42dd600640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42c006b500 0x7f42c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:38.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.640+0000 7f42dd600640 1 -- 172.21.15.114:0/2654666013 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f42cc089af0 con 0x7f42e015e530 2024-03-06T07:39:38.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.640+0000 7f42e5a00640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42c006b500 0x7f42c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:38.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.641+0000 7f42e5a00640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42c006b500 0x7f42c006d9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f42d0033000 tx=0x7f42d0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:38.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.644+0000 7f42dd600640 1 -- 172.21.15.114:0/2654666013 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42e0067830 con 0x7f42e015e530 2024-03-06T07:39:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.879+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42e004f330 con 0x7f42e015e530 2024-03-06T07:39:38.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.880+0000 7f42dd600640 1 -- 172.21.15.114:0/2654666013 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f42cc058930 con 0x7f42e015e530 2024-03-06T07:39:38.882 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:38.882 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:38.883 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:38.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42c006b500 msgr2=0x7f42c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42c006b500 0x7f42c006d9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f42d0033000 tx=0x7f42d0009030 comp rx=0 tx=0).stop 2024-03-06T07:39:38.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015e530 msgr2=0x7f42e015d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:38.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015e530 0x7f42e015d650 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000ff0 tx=0x7f42cc00dc00 comp rx=0 tx=0).stop 2024-03-06T07:39:38.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 shutdown_connections 2024-03-06T07:39:38.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f42e006f2e0 0x7f42e006f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f42c006b500 0x7f42c006d9c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015e530 0x7f42e015d650 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 --2- 172.21.15.114:0/2654666013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4003a00 0x7f42e00b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:38.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 >> 172.21.15.114:0/2654666013 conn(0x7f42e00d0b90 msgr2=0x7f42e01424b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:38.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 shutdown_connections 2024-03-06T07:39:38.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:38.883+0000 7f42e8400640 1 -- 172.21.15.114:0/2654666013 wait complete. 2024-03-06T07:39:39.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:39.106 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:40.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:40.201 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:40.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.324+0000 7f5831000640 1 Processor -- start 2024-03-06T07:39:40.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.324+0000 7f5831000640 1 -- start start 2024-03-06T07:39:40.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.326+0000 7f5831000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f582c144c00 0x7f582c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.326+0000 7f5831000640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f582c15f9d0 0x7f582c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.326+0000 7f5831000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582c06ee70 con 0x7f582c0ce0a0 2024-03-06T07:39:40.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.326+0000 7f5831000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c06cef0 con 0x7f582c144c00 2024-03-06T07:39:40.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.326+0000 7f5831000640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c06d1d0 con 0x7f582c15f9d0 2024-03-06T07:39:40.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.326+0000 7f5829e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f582c144c00 0x7f582c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.326+0000 7f582a800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f582c0ce0a0 0x7f582c0ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55796/0 (socket says 172.21.15.114:55796) 2024-03-06T07:39:40.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.327+0000 7f582a800640 1 -- 172.21.15.114:0/3429514378 learned_addr learned my addr 172.21.15.114:0/3429514378 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:40.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.327+0000 7f5823e00640 1 --2- 172.21.15.114:0/3429514378 >> v2:172.21.15.177:3300/0 conn(0x7f582c15f9d0 0x7f582c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.327+0000 7f5829e00640 1 -- 172.21.15.114:0/3429514378 >> v2:172.21.15.177:3300/0 conn(0x7f582c15f9d0 msgr2=0x7f582c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:40.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.327+0000 7f5829e00640 1 --2- 172.21.15.114:0/3429514378 >> v2:172.21.15.177:3300/0 conn(0x7f582c15f9d0 0x7f582c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.327+0000 7f5829e00640 1 -- 172.21.15.114:0/3429514378 >> v1:172.21.15.114:6789/0 conn(0x7f582c0ce0a0 legacy=0x7f582c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.327+0000 7f5829e00640 1 -- 172.21.15.114:0/3429514378 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c06d4b0 con 0x7f582c144c00 2024-03-06T07:39:40.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.328+0000 7f5829e00640 1 --2- 172.21.15.114:0/3429514378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c06e930 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b26f0 tx=0x7f581c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77ba5d2af9c65ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.328+0000 7f5829400640 1 -- 172.21.15.114:0/3429514378 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c035020 con 0x7f582c144c00 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5829400640 1 -- 172.21.15.114:0/3429514378 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c031830 con 0x7f582c144c00 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5829400640 1 -- 172.21.15.114:0/3429514378 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c031c00 con 0x7f582c144c00 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 -- 172.21.15.114:0/3429514378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 msgr2=0x7f582c06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 --2- 172.21.15.114:0/3429514378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c06e930 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b26f0 tx=0x7f581c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 -- 172.21.15.114:0/3429514378 shutdown_connections 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 --2- 172.21.15.114:0/3429514378 >> v2:172.21.15.177:3300/0 conn(0x7f582c15f9d0 0x7f582c15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 --2- 172.21.15.114:0/3429514378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c06e930 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 -- 172.21.15.114:0/3429514378 >> 172.21.15.114:0/3429514378 conn(0x7f582c0b1730 msgr2=0x7f582c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:40.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 -- 172.21.15.114:0/3429514378 shutdown_connections 2024-03-06T07:39:40.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.329+0000 7f5831000640 1 -- 172.21.15.114:0/3429514378 wait complete. 2024-03-06T07:39:40.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.332+0000 7f5831000640 1 Processor -- start 2024-03-06T07:39:40.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.332+0000 7f5831000640 1 -- start start 2024-03-06T07:39:40.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5831000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582c0ce0a0 0x7f582c0b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5831000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c15a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5831000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f582c15f9d0 0x7f582c13c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5831000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c06cb50 con 0x7f582c144c00 2024-03-06T07:39:40.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5831000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c160730 con 0x7f582c15f9d0 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5831000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c06ccd0 con 0x7f582c0ce0a0 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5823e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5823e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46398/0 (socket says 172.21.15.114:46398) 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.334+0000 7f5823e00640 1 -- 172.21.15.114:0/59178297 learned_addr learned my addr 172.21.15.114:0/59178297 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5829e00640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582c0ce0a0 0x7f582c0b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5823e00640 1 -- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582c0ce0a0 msgr2=0x7f582c0b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5823e00640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582c0ce0a0 0x7f582c0b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f582a800640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f582c15f9d0 0x7f582c13c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5823e00640 1 -- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f582c15f9d0 msgr2=0x7f582c13c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5823e00640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f582c15f9d0 0x7f582c13c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5823e00640 1 -- 172.21.15.114:0/59178297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c13d260 con 0x7f582c144c00 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5829e00640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582c0ce0a0 0x7f582c0b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f582a800640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f582c15f9d0 0x7f582c13c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5823e00640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c15a210 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f581400d830 tx=0x7f581400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:40.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5822000640 1 -- 172.21.15.114:0/59178297 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581400b6c0 con 0x7f582c144c00 2024-03-06T07:39:40.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5822000640 1 -- 172.21.15.114:0/59178297 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5814019070 con 0x7f582c144c00 2024-03-06T07:39:40.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5822000640 1 -- 172.21.15.114:0/59178297 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581400ba80 con 0x7f582c144c00 2024-03-06T07:39:40.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.335+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c0b3790 con 0x7f582c144c00 2024-03-06T07:39:40.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.336+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c13d520 con 0x7f582c144c00 2024-03-06T07:39:40.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.337+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c0b3520 con 0x7f582c144c00 2024-03-06T07:39:40.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.339+0000 7f5822000640 1 -- 172.21.15.114:0/59178297 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f581401e460 con 0x7f582c144c00 2024-03-06T07:39:40.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.340+0000 7f5822000640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f580806b4d0 0x7f580806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:40.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.340+0000 7f5822000640 1 -- 172.21.15.114:0/59178297 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f581408a8f0 con 0x7f582c144c00 2024-03-06T07:39:40.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.340+0000 7f5829e00640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f580806b4d0 0x7f580806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:40.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.341+0000 7f5829e00640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f580806b4d0 0x7f580806d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f581c036000 tx=0x7f581c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:40.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.345+0000 7f5822000640 1 -- 172.21.15.114:0/59178297 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5814059730 con 0x7f582c144c00 2024-03-06T07:39:40.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.635+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f582c0cdc80 con 0x7f582c144c00 2024-03-06T07:39:40.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.635+0000 7f5822000640 1 -- 172.21.15.114:0/59178297 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3470 (secure 0 0 0) 0x7f582c0cdc80 con 0x7f582c144c00 2024-03-06T07:39:40.637 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:40.637 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:44.544180+0000","modified":"2024-03-06T07:38:14.634090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8331,"mds_1":8328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8328":{"gid":8328,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.114:6836/4077433793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":4077433793},{"type":"v1","addr":"172.21.15.114:6836","nonce":4077433793}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:44.929793+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8331":{"gid":8331,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6837/1465597546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":1465597546},{"type":"v1","addr":"172.21.15.114:6837","nonce":1465597546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8331,"qdb_cluster":[8331]},"id":9}]} 2024-03-06T07:39:40.638 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 105 2024-03-06T07:39:40.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f580806b4d0 msgr2=0x7f580806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f580806b4d0 0x7f580806d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f581c036000 tx=0x7f581c008030 comp rx=0 tx=0).stop 2024-03-06T07:39:40.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 msgr2=0x7f582c15a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:40.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c15a210 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f581400d830 tx=0x7f581400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 shutdown_connections 2024-03-06T07:39:40.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f582c15f9d0 0x7f582c13c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f580806b4d0 0x7f580806d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f582c144c00 0x7f582c15a210 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 --2- 172.21.15.114:0/59178297 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f582c0ce0a0 0x7f582c0b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:40.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 >> 172.21.15.114:0/59178297 conn(0x7f582c0b1730 msgr2=0x7f582c0b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:40.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 shutdown_connections 2024-03-06T07:39:40.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:40.638+0000 7f5831000640 1 -- 172.21.15.114:0/59178297 wait complete. 2024-03-06T07:39:40.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:44.544180+0000', 'modified': '2024-03-06T07:38:14.634090+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8331, 'mds_1': 8328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8328': {'gid': 8328, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.114:6836/4077433793', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 4077433793}, {'type': 'v1', 'addr': '172.21.15.114:6836', 'nonce': 4077433793}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:44.929793+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8331': {'gid': 8331, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6837/1465597546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6835', 'nonce': 1465597546}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1465597546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8331, 'qdb_cluster': [8331]} 2024-03-06T07:39:40.860 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8328:up:stopping 2024-03-06T07:39:41.862 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 105, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:35:44.544180+0000", "damaged": [], "data_pools": [ 19 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8328": { "addr": "172.21.15.114:6836/4077433793", "addrs": { "addrvec": [ { "addr": "172.21.15.114:6834", "nonce": 4077433793, "type": "v2" }, { "addr": "172.21.15.114:6836", "nonce": 4077433793, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8328, "incarnation": 98, "join_fscid": -1, "laggy_since": "2024-03-06T07:36:44.929793+0000", "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_8331": { "addr": "172.21.15.114:6837/1465597546", "addrs": { "addrvec": [ { "addr": "172.21.15.114:6835", "nonce": 1465597546, "type": "v2" }, { "addr": "172.21.15.114:6837", "nonce": 1465597546, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8331, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 18, "modified": "2024-03-06T07:38:14.634090+0000", "qdb_cluster": [ 8331 ], "qdb_leader": 8331, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8331, "mds_1": 8328 } } } ], "standbys": [] } 2024-03-06T07:39:41.863 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:39:41.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:39:41.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:39:42.101 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:39:42.102 INFO:tasks.cephfs.mount:Terminating background process 2024-03-06T07:39:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:39:42.203 INFO:teuthology.orchestra.run.smithi114.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:42.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:39:42.233 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:39:42.233 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:39:42.329 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-06T07:39:42.330 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:39:42.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:39:42.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.622+0000 7f5d82800640 1 Processor -- start 2024-03-06T07:39:42.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.622+0000 7f5d82800640 1 -- start start 2024-03-06T07:39:42.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.623+0000 7f5d82800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5d7c15f9d0 0x7f5d7c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.624+0000 7f5d82800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06cef0 con 0x7f5d7c070b10 2024-03-06T07:39:42.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.624+0000 7f5d82800640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06d1d0 con 0x7f5d7c144c00 2024-03-06T07:39:42.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.624+0000 7f5d82800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c06d4b0 con 0x7f5d7c15f9d0 2024-03-06T07:39:42.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.624+0000 7f5d7be00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5d7c070b10 0x7f5d7c070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55802/0 (socket says 172.21.15.114:55802) 2024-03-06T07:39:42.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.624+0000 7f5d7be00640 1 -- 172.21.15.114:0/1487485007 learned_addr learned my addr 172.21.15.114:0/1487485007 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.624+0000 7f5d7aa00640 1 --2- 172.21.15.114:0/1487485007 >> v2:172.21.15.177:3300/0 conn(0x7f5d7c15f9d0 0x7f5d7c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984245457 0 0) 0x7f5d7c06cef0 con 0x7f5d7c070b10 2024-03-06T07:39:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d60003520 con 0x7f5d7c070b10 2024-03-06T07:39:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130441346 0 0) 0x7f5d7c06d1d0 con 0x7f5d7c144c00 2024-03-06T07:39:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c06cef0 con 0x7f5d7c144c00 2024-03-06T07:39:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680860861 0 0) 0x7f5d60003520 con 0x7f5d7c070b10 2024-03-06T07:39:42.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 >> v2:172.21.15.177:3300/0 conn(0x7f5d7c15f9d0 msgr2=0x7f5d7c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 --2- 172.21.15.114:0/1487485007 >> v2:172.21.15.177:3300/0 conn(0x7f5d7c15f9d0 0x7f5d7c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 >> v1:172.21.15.177:6789/0 conn(0x7f5d7c144c00 legacy=0x7f5d7c06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c06d790 con 0x7f5d7c070b10 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f5d740019f0 con 0x7f5d7c070b10 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 >> v1:172.21.15.114:6789/0 conn(0x7f5d7c070b10 legacy=0x7f5d7c070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d60003b90 0x7f5d60003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d60004800 0x7f5d60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d600051c0 0x7f5d6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c06d4b0 con 0x7f5d60003b90 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6000c090 con 0x7f5d600051c0 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.625+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6000c290 con 0x7f5d60004800 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7b400640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d60003b90 0x7f5d60003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7be00640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d600051c0 0x7f5d6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7aa00640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d60004800 0x7f5d60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7b400640 1 -- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d60004800 msgr2=0x7f5d60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7b400640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d60004800 0x7f5d60004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7b400640 1 -- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d600051c0 msgr2=0x7f5d6000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7b400640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d600051c0 0x7f5d6000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7b400640 1 -- 172.21.15.114:0/1487485007 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d60003520 con 0x7f5d60003b90 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7be00640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d600051c0 0x7f5d6000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.626+0000 7f5d7be00640 1 -- 172.21.15.114:0/1487485007 reap_dead start 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.627+0000 7f5d7b400640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d60003b90 0x7f5d60003f90 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0018d0 tx=0x7f5d60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf4ae2a1ddce7d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.627+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c037070 con 0x7f5d60003b90 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.627+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c0052d0 con 0x7f5d60003b90 2024-03-06T07:39:42.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.627+0000 7f5d7a000640 1 -- 172.21.15.114:0/1487485007 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0014f0 con 0x7f5d60003b90 2024-03-06T07:39:42.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.628+0000 7f5d82800640 1 -- 172.21.15.114:0/1487485007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d60003b90 msgr2=0x7f5d60003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.628+0000 7f5d82800640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d60003b90 0x7f5d60003f90 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0018d0 tx=0x7f5d60005830 comp rx=0 tx=0).stop 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.628+0000 7f5d82800640 1 -- 172.21.15.114:0/1487485007 shutdown_connections 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.628+0000 7f5d82800640 1 --2- 172.21.15.114:0/1487485007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d60003b90 0x7f5d60003f90 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.628+0000 7f5d82800640 1 -- 172.21.15.114:0/1487485007 >> 172.21.15.114:0/1487485007 conn(0x7f5d7c0b1730 msgr2=0x7f5d7c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.628+0000 7f5d82800640 1 -- 172.21.15.114:0/1487485007 shutdown_connections 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.628+0000 7f5d82800640 1 -- 172.21.15.114:0/1487485007 wait complete. 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.631+0000 7f5d82800640 1 Processor -- start 2024-03-06T07:39:42.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.631+0000 7f5d82800640 1 -- start start 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d82800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d60003b90 0x7f5d7c13a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d82800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c13af40 0x7f5d7c13c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d82800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c1f0340 0x7f5d7c1f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d82800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c06ccd0 con 0x7f5d7c1f0340 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d82800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c06cb50 con 0x7f5d60003b90 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d82800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c06df90 con 0x7f5d7c13af40 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c1f0340 0x7f5d7c1f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c1f0340 0x7f5d7c1f07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46418/0 (socket says 172.21.15.114:46418) 2024-03-06T07:39:42.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 -- 172.21.15.114:0/994941228 learned_addr learned my addr 172.21.15.114:0/994941228 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7aa00640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c13af40 0x7f5d7c13c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 -- 172.21.15.114:0/994941228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c13af40 msgr2=0x7f5d7c13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c13af40 0x7f5d7c13c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 -- 172.21.15.114:0/994941228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d60003b90 msgr2=0x7f5d7c13a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d60003b90 0x7f5d7c13a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.633+0000 7f5d7be00640 1 -- 172.21.15.114:0/994941228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c13abc0 con 0x7f5d7c1f0340 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.634+0000 7f5d7be00640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c1f0340 0x7f5d7c1f07f0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c0737b0 tx=0x7f5d7400def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.634+0000 7f5d73400640 1 -- 172.21.15.114:0/994941228 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7400b800 con 0x7f5d7c1f0340 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.634+0000 7f5d73400640 1 -- 172.21.15.114:0/994941228 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7400b9a0 con 0x7f5d7c1f0340 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.634+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c13b3d0 con 0x7f5d7c1f0340 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.634+0000 7f5d73400640 1 -- 172.21.15.114:0/994941228 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7400bd90 con 0x7f5d7c1f0340 2024-03-06T07:39:42.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.634+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c13b990 con 0x7f5d7c1f0340 2024-03-06T07:39:42.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.635+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c04f330 con 0x7f5d7c1f0340 2024-03-06T07:39:42.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.636+0000 7f5d73400640 1 -- 172.21.15.114:0/994941228 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d7401b020 con 0x7f5d7c1f0340 2024-03-06T07:39:42.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.637+0000 7f5d73400640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5c06b4d0 0x7f5d5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:42.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.637+0000 7f5d73400640 1 -- 172.21.15.114:0/994941228 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5d7408a290 con 0x7f5d7c1f0340 2024-03-06T07:39:42.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.637+0000 7f5d7b400640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5c06b4d0 0x7f5d5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:42.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.638+0000 7f5d7b400640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5c06b4d0 0x7f5d5c06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c034000 tx=0x7f5d6c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:42.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.642+0000 7f5d73400640 1 -- 172.21.15.114:0/994941228 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d740590d0 con 0x7f5d7c1f0340 2024-03-06T07:39:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.840+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d7c0b4cb0 con 0x7f5d7c1f0340 2024-03-06T07:39:42.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.844+0000 7f5d73400640 1 -- 172.21.15.114:0/994941228 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16769 (secure 0 0 0) 0x7f5d740589d0 con 0x7f5d7c1f0340 2024-03-06T07:39:42.846 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-06T07:39:42.846 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":129,"fsid":"149c34a4-b335-4a4b-8a41-0fe21b0db808","created":"2024-03-06T07:27:30.028309+0000","modified":"2024-03-06T07:36:39.049384+0000","last_up_change":"2024-03-06T07:27:42.344389+0000","last_in_change":"2024-03-06T07:27:33.548233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:43.170577+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:42.299180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:43.529461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24fea251-8ceb-4e5a-8d7a-379c6e928d1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6801","nonce":3895229755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6803","nonce":3895229755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6807","nonce":3895229755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3895229755},{"type":"v1","addr":"172.21.15.114:6805","nonce":3895229755}]},"public_addr":"172.21.15.114:6801/3895229755","cluster_addr":"172.21.15.114:6803/3895229755","heartbeat_back_addr":"172.21.15.114:6807/3895229755","heartbeat_front_addr":"172.21.15.114:6805/3895229755","state":["exists","up"]},{"osd":1,"uuid":"8c89fbef-928a-4868-b4fc-da93fa4f0b9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6817","nonce":1955604845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6819","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6821","nonce":1955604845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6827","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6829","nonce":1955604845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6823","nonce":1955604845},{"type":"v1","addr":"172.21.15.114:6825","nonce":1955604845}]},"public_addr":"172.21.15.114:6817/1955604845","cluster_addr":"172.21.15.114:6821/1955604845","heartbeat_back_addr":"172.21.15.114:6829/1955604845","heartbeat_front_addr":"172.21.15.114:6825/1955604845","state":["exists","up"]},{"osd":2,"uuid":"4292fc64-f253-4fee-97b3-9440ad950e3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6809","nonce":1832993461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6811","nonce":1832993461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6815","nonce":1832993461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1832993461},{"type":"v1","addr":"172.21.15.114:6813","nonce":1832993461}]},"public_addr":"172.21.15.114:6809/1832993461","cluster_addr":"172.21.15.114:6811/1832993461","heartbeat_back_addr":"172.21.15.114:6815/1832993461","heartbeat_front_addr":"172.21.15.114:6813/1832993461","state":["exists","up"]},{"osd":3,"uuid":"56b8f5df-65df-4f34-80cd-d7dae655a279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6820","nonce":735520392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6824","nonce":735520392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6831","nonce":735520392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":735520392},{"type":"v1","addr":"172.21.15.114:6828","nonce":735520392}]},"public_addr":"172.21.15.114:6820/735520392","cluster_addr":"172.21.15.114:6824/735520392","heartbeat_back_addr":"172.21.15.114:6831/735520392","heartbeat_front_addr":"172.21.15.114:6828/735520392","state":["exists","up"]},{"osd":4,"uuid":"03440662-d24e-4004-bb4c-2edb870e3000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6801","nonce":4111454897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6803","nonce":4111454897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6807","nonce":4111454897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":4111454897},{"type":"v1","addr":"172.21.15.177:6805","nonce":4111454897}]},"public_addr":"172.21.15.177:6801/4111454897","cluster_addr":"172.21.15.177:6803/4111454897","heartbeat_back_addr":"172.21.15.177:6807/4111454897","heartbeat_front_addr":"172.21.15.177:6805/4111454897","state":["exists","up"]},{"osd":5,"uuid":"e8265969-0287-457e-968c-d7ec7047e491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6817","nonce":2170499622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6819","nonce":2170499622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6823","nonce":2170499622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":2170499622},{"type":"v1","addr":"172.21.15.177:6821","nonce":2170499622}]},"public_addr":"172.21.15.177:6817/2170499622","cluster_addr":"172.21.15.177:6819/2170499622","heartbeat_back_addr":"172.21.15.177:6823/2170499622","heartbeat_front_addr":"172.21.15.177:6821/2170499622","state":["exists","up"]},{"osd":6,"uuid":"4182701d-5859-4c42-bf7f-5b92a73e6ec4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6809","nonce":4041662010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6811","nonce":4041662010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6815","nonce":4041662010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":4041662010},{"type":"v1","addr":"172.21.15.177:6813","nonce":4041662010}]},"public_addr":"172.21.15.177:6809/4041662010","cluster_addr":"172.21.15.177:6811/4041662010","heartbeat_back_addr":"172.21.15.177:6815/4041662010","heartbeat_front_addr":"172.21.15.177:6813/4041662010","state":["exists","up"]},{"osd":7,"uuid":"13556954-d129-4900-ac0b-bc34c9368af3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6825","nonce":1593615297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6827","nonce":1593615297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6831","nonce":1593615297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1593615297},{"type":"v1","addr":"172.21.15.177:6829","nonce":1593615297}]},"public_addr":"172.21.15.177:6825/1593615297","cluster_addr":"172.21.15.177:6827/1593615297","heartbeat_back_addr":"172.21.15.177:6831/1593615297","heartbeat_front_addr":"172.21.15.177:6829/1593615297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.899639+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.920956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.915043+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.871950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.761439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.878250+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.891300+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:38.882993+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.22","mappings":[{"from":2,"to":0}]},{"pgid":"19.26","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:39:42.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5c06b4d0 msgr2=0x7f5d5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5c06b4d0 0x7f5d5c06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c034000 tx=0x7f5d6c009030 comp rx=0 tx=0).stop 2024-03-06T07:39:42.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c1f0340 msgr2=0x7f5d7c1f07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:42.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c1f0340 0x7f5d7c1f07f0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c0737b0 tx=0x7f5d7400def0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 shutdown_connections 2024-03-06T07:39:42.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f5d5c06b4d0 0x7f5d5c06d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d7c1f0340 0x7f5d7c1f07f0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f5d7c13af40 0x7f5d7c13c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 --2- 172.21.15.114:0/994941228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d60003b90 0x7f5d7c13a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:42.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.846+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 >> 172.21.15.114:0/994941228 conn(0x7f5d7c0b1730 msgr2=0x7f5d7c0ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:42.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.847+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 shutdown_connections 2024-03-06T07:39:42.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:42.847+0000 7f5d82800640 1 -- 172.21.15.114:0/994941228 wait complete. 2024-03-06T07:39:42.869 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:43.170577+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:35:42.299180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:35:43.529461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:39:42.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:39:43.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.061+0000 7f438f200640 1 Processor -- start 2024-03-06T07:39:43.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.061+0000 7f438f200640 1 -- start start 2024-03-06T07:39:43.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f438f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f439006e510 0x7f4390165030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f438f200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43900b3e70 con 0x7f4390072840 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f438f200640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43900cf1a0 con 0x7f439015d9f0 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f438f200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43900b4bf0 con 0x7f439006e510 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f4387e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4390072840 0x7f439006de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55816/0 (socket says 172.21.15.114:55816) 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f4387e00640 1 -- 172.21.15.114:0/703426585 learned_addr learned my addr 172.21.15.114:0/703426585 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f438e800640 1 --2- 172.21.15.114:0/703426585 >> v2:172.21.15.114:3300/0 conn(0x7f439006e510 0x7f4390165030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f438d400640 1 -- 172.21.15.114:0/703426585 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227985441 0 0) 0x7f43900b3e70 con 0x7f4390072840 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.063+0000 7f438d400640 1 -- 172.21.15.114:0/703426585 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4374003520 con 0x7f4390072840 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.064+0000 7f438e800640 1 -- 172.21.15.114:0/703426585 >> v1:172.21.15.177:6789/0 conn(0x7f439015d9f0 legacy=0x7f43900ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.064+0000 7f438e800640 1 -- 172.21.15.114:0/703426585 >> v1:172.21.15.114:6789/0 conn(0x7f4390072840 legacy=0x7f439006de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.064+0000 7f438e800640 1 -- 172.21.15.114:0/703426585 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43900b4110 con 0x7f439006e510 2024-03-06T07:39:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.065+0000 7f438e800640 1 --2- 172.21.15.114:0/703426585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f439006e510 0x7f4390165030 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4388009d90 tx=0x7f438802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e27446bef7b3b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.065+0000 7f438d400640 1 -- 172.21.15.114:0/703426585 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4388039070 con 0x7f439006e510 2024-03-06T07:39:43.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.065+0000 7f438d400640 1 -- 172.21.15.114:0/703426585 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438802ccf0 con 0x7f439006e510 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.065+0000 7f438d400640 1 -- 172.21.15.114:0/703426585 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4388034660 con 0x7f439006e510 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.065+0000 7f438f200640 1 -- 172.21.15.114:0/703426585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f439006e510 msgr2=0x7f4390165030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.065+0000 7f438f200640 1 --2- 172.21.15.114:0/703426585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f439006e510 0x7f4390165030 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4388009d90 tx=0x7f438802bf40 comp rx=0 tx=0).stop 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.066+0000 7f438f200640 1 -- 172.21.15.114:0/703426585 shutdown_connections 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.066+0000 7f438f200640 1 --2- 172.21.15.114:0/703426585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f439006e510 0x7f4390165030 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.066+0000 7f438f200640 1 -- 172.21.15.114:0/703426585 >> 172.21.15.114:0/703426585 conn(0x7f439006afc0 msgr2=0x7f439006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.066+0000 7f438f200640 1 -- 172.21.15.114:0/703426585 shutdown_connections 2024-03-06T07:39:43.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.066+0000 7f438f200640 1 -- 172.21.15.114:0/703426585 wait complete. 2024-03-06T07:39:43.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.067+0000 7f438f200640 1 Processor -- start 2024-03-06T07:39:43.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.068+0000 7f438f200640 1 -- start start 2024-03-06T07:39:43.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438f200640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f439006e510 0x7f4390152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390072840 0x7f43900b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f439015d9f0 0x7f439015c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390070970 con 0x7f4390072840 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439015ddd0 con 0x7f439015d9f0 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438f200640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4390070af0 con 0x7f439006e510 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390072840 0x7f43900b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390072840 0x7f43900b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46442/0 (socket says 172.21.15.114:46442) 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 -- 172.21.15.114:0/1811238625 learned_addr learned my addr 172.21.15.114:0/1811238625 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438e800640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f439015d9f0 0x7f439015c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438de00640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f439006e510 0x7f4390152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 -- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f439006e510 msgr2=0x7f4390152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f439006e510 0x7f4390152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 -- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f439015d9f0 msgr2=0x7f439015c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f439015d9f0 0x7f439015c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f4387e00640 1 -- 172.21.15.114:0/1811238625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43900b02f0 con 0x7f4390072840 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.069+0000 7f438e800640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f439015d9f0 0x7f439015c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.070+0000 7f4387e00640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390072840 0x7f43900b1d20 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f437800c500 tx=0x7f437800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.070+0000 7f4386000640 1 -- 172.21.15.114:0/1811238625 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378017070 con 0x7f4390072840 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.070+0000 7f4386000640 1 -- 172.21.15.114:0/1811238625 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4378005ce0 con 0x7f4390072840 2024-03-06T07:39:43.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.070+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43900b05d0 con 0x7f4390072840 2024-03-06T07:39:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.070+0000 7f4386000640 1 -- 172.21.15.114:0/1811238625 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378007330 con 0x7f4390072840 2024-03-06T07:39:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.070+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43900b0b90 con 0x7f4390072840 2024-03-06T07:39:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.071+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43900b5b40 con 0x7f4390072840 2024-03-06T07:39:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.072+0000 7f4386000640 1 -- 172.21.15.114:0/1811238625 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f437801a070 con 0x7f4390072840 2024-03-06T07:39:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.073+0000 7f4386000640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f436c06b4d0 0x7f436c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.073+0000 7f4386000640 1 -- 172.21.15.114:0/1811238625 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4378089670 con 0x7f4390072840 2024-03-06T07:39:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.073+0000 7f438de00640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f436c06b4d0 0x7f436c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.074+0000 7f438de00640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f436c06b4d0 0x7f436c06d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4380000f60 tx=0x7f4380009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.079+0000 7f4386000640 1 -- 172.21.15.114:0/1811238625 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43900b5b40 con 0x7f4390072840 2024-03-06T07:39:43.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.293+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f439004f330 con 0x7f4390072840 2024-03-06T07:39:43.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.294+0000 7f4386000640 1 -- 172.21.15.114:0/1811238625 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4378058600 con 0x7f4390072840 2024-03-06T07:39:43.296 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-03-06T07:39:43.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f436c06b4d0 msgr2=0x7f436c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f436c06b4d0 0x7f436c06d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4380000f60 tx=0x7f4380009040 comp rx=0 tx=0).stop 2024-03-06T07:39:43.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390072840 msgr2=0x7f43900b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390072840 0x7f43900b1d20 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f437800c500 tx=0x7f437800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 shutdown_connections 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f439015d9f0 0x7f439015c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f436c06b4d0 0x7f436c06d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390072840 0x7f43900b1d20 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 --2- 172.21.15.114:0/1811238625 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f439006e510 0x7f4390152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 >> 172.21.15.114:0/1811238625 conn(0x7f439006afc0 msgr2=0x7f4390156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 shutdown_connections 2024-03-06T07:39:43.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.298+0000 7f438f200640 1 -- 172.21.15.114:0/1811238625 wait complete. 2024-03-06T07:39:43.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:39:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.520+0000 7f1fc1800640 1 Processor -- start 2024-03-06T07:39:43.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.520+0000 7f1fc1800640 1 -- start start 2024-03-06T07:39:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fc1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc1618e0 0x7f1fbc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fc1800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1fbc0b2d80 0x7f1fbc0cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fc1800640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f1fbc0cf520 0x7f1fbc0d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fc1800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc0d0d20 con 0x7f1fbc0b2d80 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fc1800640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc0cf9b0 con 0x7f1fbc0cf520 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fc1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0b32f0 con 0x7f1fbc1618e0 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fb2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1fbc0b2d80 0x7f1fbc0cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fb2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1fbc0b2d80 0x7f1fbc0cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46456/0 (socket says 172.21.15.114:46456) 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fb2000640 1 -- 172.21.15.114:0/4136730924 learned_addr learned my addr 172.21.15.114:0/4136730924 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fbaa00640 1 --2- 172.21.15.114:0/4136730924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc1618e0 0x7f1fbc0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.522+0000 7f1fbb400640 1 --2- 172.21.15.114:0/4136730924 >> v2:172.21.15.177:3300/0 conn(0x7f1fbc0cf520 0x7f1fbc0d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.523+0000 7f1fb2000640 1 -- 172.21.15.114:0/4136730924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc1618e0 msgr2=0x7f1fbc0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.523+0000 7f1fb2000640 1 --2- 172.21.15.114:0/4136730924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc1618e0 0x7f1fbc0ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.523+0000 7f1fb2000640 1 -- 172.21.15.114:0/4136730924 >> v2:172.21.15.177:3300/0 conn(0x7f1fbc0cf520 msgr2=0x7f1fbc0d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.523+0000 7f1fb2000640 1 --2- 172.21.15.114:0/4136730924 >> v2:172.21.15.177:3300/0 conn(0x7f1fbc0cf520 0x7f1fbc0d1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.523+0000 7f1fb2000640 1 -- 172.21.15.114:0/4136730924 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc0d0f70 con 0x7f1fbc0b2d80 2024-03-06T07:39:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.524+0000 7f1fb2000640 1 --2- 172.21.15.114:0/4136730924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc0b2d80 0x7f1fbc0cefe0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009870 tx=0x7f1fa402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9acc5d7ed2ed6e0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fba000640 1 -- 172.21.15.114:0/4136730924 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40317c0 con 0x7f1fbc0b2d80 2024-03-06T07:39:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fba000640 1 -- 172.21.15.114:0/4136730924 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa4037070 con 0x7f1fbc0b2d80 2024-03-06T07:39:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fba000640 1 -- 172.21.15.114:0/4136730924 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4031b80 con 0x7f1fbc0b2d80 2024-03-06T07:39:43.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 -- 172.21.15.114:0/4136730924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc0b2d80 msgr2=0x7f1fbc0cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 --2- 172.21.15.114:0/4136730924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc0b2d80 0x7f1fbc0cefe0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009870 tx=0x7f1fa402b860 comp rx=0 tx=0).stop 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 -- 172.21.15.114:0/4136730924 shutdown_connections 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 --2- 172.21.15.114:0/4136730924 >> v2:172.21.15.177:3300/0 conn(0x7f1fbc0cf520 0x7f1fbc0d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 --2- 172.21.15.114:0/4136730924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc0b2d80 0x7f1fbc0cefe0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 --2- 172.21.15.114:0/4136730924 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc1618e0 0x7f1fbc0ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 -- 172.21.15.114:0/4136730924 >> 172.21.15.114:0/4136730924 conn(0x7f1fbc0d3e90 msgr2=0x7f1fbc0d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 -- 172.21.15.114:0/4136730924 shutdown_connections 2024-03-06T07:39:43.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.525+0000 7f1fc1800640 1 -- 172.21.15.114:0/4136730924 wait complete. 2024-03-06T07:39:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.527+0000 7f1fc1800640 1 Processor -- start 2024-03-06T07:39:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.527+0000 7f1fc1800640 1 -- start start 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fc1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1fbc0b2d80 0x7f1fbc15d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fc1800640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc0cf520 0x7f1fbc0c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fc1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc1618e0 0x7f1fbc1f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fc1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc161cc0 con 0x7f1fbc1618e0 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fc1800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0d1530 con 0x7f1fbc0b2d80 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fc1800640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0d22d0 con 0x7f1fbc0cf520 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fbb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc1618e0 0x7f1fbc1f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fbb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc1618e0 0x7f1fbc1f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46470/0 (socket says 172.21.15.114:46470) 2024-03-06T07:39:43.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.528+0000 7f1fbb400640 1 -- 172.21.15.114:0/356270118 learned_addr learned my addr 172.21.15.114:0/356270118 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fb2000640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc0cf520 0x7f1fbc0c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fbaa00640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1fbc0b2d80 0x7f1fbc15d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fbb400640 1 -- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc0cf520 msgr2=0x7f1fbc0c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fbb400640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc0cf520 0x7f1fbc0c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fbb400640 1 -- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1fbc0b2d80 msgr2=0x7f1fbc15d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fbb400640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1fbc0b2d80 0x7f1fbc15d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fbb400640 1 -- 172.21.15.114:0/356270118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc0c8060 con 0x7f1fbc1618e0 2024-03-06T07:39:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.529+0000 7f1fbb400640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc1618e0 0x7f1fbc1f3c90 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4007e60 tx=0x7f1fb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.530+0000 7f1fb3400640 1 -- 172.21.15.114:0/356270118 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4014620 con 0x7f1fbc1618e0 2024-03-06T07:39:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.530+0000 7f1fb3400640 1 -- 172.21.15.114:0/356270118 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4019070 con 0x7f1fbc1618e0 2024-03-06T07:39:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.530+0000 7f1fb3400640 1 -- 172.21.15.114:0/356270118 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4014a20 con 0x7f1fbc1618e0 2024-03-06T07:39:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.530+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc0c8340 con 0x7f1fbc1618e0 2024-03-06T07:39:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.530+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc0c8900 con 0x7f1fbc1618e0 2024-03-06T07:39:43.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.531+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc04f330 con 0x7f1fbc1618e0 2024-03-06T07:39:43.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.532+0000 7f1fb3400640 1 -- 172.21.15.114:0/356270118 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fb4014bc0 con 0x7f1fbc1618e0 2024-03-06T07:39:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.534+0000 7f1fb3400640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f9806b4d0 0x7f1f9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.534+0000 7f1fb3400640 1 -- 172.21.15.114:0/356270118 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1fb408abc0 con 0x7f1fbc1618e0 2024-03-06T07:39:43.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.534+0000 7f1fbaa00640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f9806b4d0 0x7f1f9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.535+0000 7f1fbaa00640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f9806b4d0 0x7f1f9806d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1fac006fd0 tx=0x7f1fac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.537+0000 7f1fb3400640 1 -- 172.21.15.114:0/356270118 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fb4059960 con 0x7f1fbc1618e0 2024-03-06T07:39:43.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.733+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1fbc0b26e0 con 0x7f1fbc1618e0 2024-03-06T07:39:43.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.734+0000 7f1fb3400640 1 -- 172.21.15.114:0/356270118 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1fb4059260 con 0x7f1fbc1618e0 2024-03-06T07:39:43.736 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-03-06T07:39:43.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f9806b4d0 msgr2=0x7f1f9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f9806b4d0 0x7f1f9806d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1fac006fd0 tx=0x7f1fac008040 comp rx=0 tx=0).stop 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc1618e0 msgr2=0x7f1fbc1f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc1618e0 0x7f1fbc1f3c90 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4007e60 tx=0x7f1fb400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 shutdown_connections 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f1f9806b4d0 0x7f1f9806d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fbc1618e0 0x7f1fbc1f3c90 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f1fbc0cf520 0x7f1fbc0c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 --2- 172.21.15.114:0/356270118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1fbc0b2d80 0x7f1fbc15d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 >> 172.21.15.114:0/356270118 conn(0x7f1fbc0d3e90 msgr2=0x7f1fbc0cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 shutdown_connections 2024-03-06T07:39:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.737+0000 7f1fc1800640 1 -- 172.21.15.114:0/356270118 wait complete. 2024-03-06T07:39:43.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:39:43.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.960+0000 7f87ed000640 1 Processor -- start 2024-03-06T07:39:43.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.960+0000 7f87ed000640 1 -- start start 2024-03-06T07:39:43.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87ed000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87e815c330 0x7f87e80d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87ed000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e80b43b0 0x7f87e815fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87ed000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e80ce1e0 con 0x7f87e81603e0 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87ed000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e8160830 con 0x7f87e815c330 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87ed000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80cedf0 con 0x7f87e80b43b0 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87e5e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f87e815c330 0x7f87e80d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87e6800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f87e81603e0 0x7f87e815c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55826/0 (socket says 172.21.15.114:55826) 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87e6800640 1 -- 172.21.15.114:0/5743271 learned_addr learned my addr 172.21.15.114:0/5743271 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.962+0000 7f87dd400640 1 --2- 172.21.15.114:0/5743271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e80b43b0 0x7f87e815fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.963+0000 7f87e5400640 1 -- 172.21.15.114:0/5743271 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017871796 0 0) 0x7f87e80ce1e0 con 0x7f87e81603e0 2024-03-06T07:39:43.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.963+0000 7f87e5e00640 1 -- 172.21.15.114:0/5743271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e80b43b0 msgr2=0x7f87e815fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:43.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.963+0000 7f87e5e00640 1 --2- 172.21.15.114:0/5743271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e80b43b0 0x7f87e815fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.963+0000 7f87e5e00640 1 -- 172.21.15.114:0/5743271 >> v1:172.21.15.114:6789/0 conn(0x7f87e81603e0 legacy=0x7f87e815c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.963+0000 7f87e5e00640 1 -- 172.21.15.114:0/5743271 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e80ce4c0 con 0x7f87e815c330 2024-03-06T07:39:43.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.964+0000 7f87e5e00640 1 --2- 172.21.15.114:0/5743271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e815c330 0x7f87e80d0570 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f87d80097a0 tx=0x7f87d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bea16bd1b4e4c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.964+0000 7f87e5400640 1 -- 172.21.15.114:0/5743271 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8035020 con 0x7f87e815c330 2024-03-06T07:39:43.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.964+0000 7f87e5400640 1 -- 172.21.15.114:0/5743271 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8031830 con 0x7f87e815c330 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.964+0000 7f87e5400640 1 -- 172.21.15.114:0/5743271 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8031c00 con 0x7f87e815c330 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 -- 172.21.15.114:0/5743271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e815c330 msgr2=0x7f87e80d0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 --2- 172.21.15.114:0/5743271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e815c330 0x7f87e80d0570 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f87d80097a0 tx=0x7f87d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 -- 172.21.15.114:0/5743271 shutdown_connections 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 --2- 172.21.15.114:0/5743271 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e80b43b0 0x7f87e815fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 --2- 172.21.15.114:0/5743271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e815c330 0x7f87e80d0570 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 -- 172.21.15.114:0/5743271 >> 172.21.15.114:0/5743271 conn(0x7f87e80b3790 msgr2=0x7f87e80b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 -- 172.21.15.114:0/5743271 shutdown_connections 2024-03-06T07:39:43.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.965+0000 7f87ed000640 1 -- 172.21.15.114:0/5743271 wait complete. 2024-03-06T07:39:43.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.967+0000 7f87ed000640 1 Processor -- start 2024-03-06T07:39:43.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.967+0000 7f87ed000640 1 -- start start 2024-03-06T07:39:43.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87ed000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87e80b43b0 0x7f87e8159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87ed000640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e815c330 0x7f87e80b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87ed000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e81603e0 0x7f87e80b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87ed000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80cd880 con 0x7f87e81603e0 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87ed000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80cea80 con 0x7f87e80b43b0 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87ed000640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80cf280 con 0x7f87e815c330 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e81603e0 0x7f87e80b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e81603e0 0x7f87e80b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46490/0 (socket says 172.21.15.114:46490) 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 -- 172.21.15.114:0/4210234558 learned_addr learned my addr 172.21.15.114:0/4210234558 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87dd400640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e815c330 0x7f87e80b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 -- 172.21.15.114:0/4210234558 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e815c330 msgr2=0x7f87e80b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e815c330 0x7f87e80b6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 -- 172.21.15.114:0/4210234558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87e80b43b0 msgr2=0x7f87e8159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87e80b43b0 0x7f87e8159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.968+0000 7f87e6800640 1 -- 172.21.15.114:0/4210234558 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e81e81f0 con 0x7f87e81603e0 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.969+0000 7f87e6800640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e81603e0 0x7f87e80b6a70 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f87e000c700 tx=0x7f87e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.969+0000 7f87dea00640 1 -- 172.21.15.114:0/4210234558 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0018070 con 0x7f87e81603e0 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.969+0000 7f87dea00640 1 -- 172.21.15.114:0/4210234558 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e000ce10 con 0x7f87e81603e0 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.969+0000 7f87dea00640 1 -- 172.21.15.114:0/4210234558 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e000a860 con 0x7f87e81603e0 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.969+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e81f7a80 con 0x7f87e81603e0 2024-03-06T07:39:43.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.969+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e81f8040 con 0x7f87e81603e0 2024-03-06T07:39:43.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.970+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e804f330 con 0x7f87e81603e0 2024-03-06T07:39:43.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.971+0000 7f87dea00640 1 -- 172.21.15.114:0/4210234558 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87e000aa00 con 0x7f87e81603e0 2024-03-06T07:39:43.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.972+0000 7f87dea00640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:43.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.972+0000 7f87dea00640 1 -- 172.21.15.114:0/4210234558 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f87e00897b0 con 0x7f87e81603e0 2024-03-06T07:39:43.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.973+0000 7f87e5e00640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:43.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.973+0000 7f87e5e00640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87c406b4d0 0x7f87c406d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f87d8036000 tx=0x7f87d8030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:43.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:43.977+0000 7f87dea00640 1 -- 172.21.15.114:0/4210234558 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87e81e81f0 con 0x7f87e81603e0 2024-03-06T07:39:44.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.269+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87e80af0d0 con 0x7f87e81603e0 2024-03-06T07:39:44.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.270+0000 7f87dea00640 1 -- 172.21.15.114:0/4210234558 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87e0058680 con 0x7f87e81603e0 2024-03-06T07:39:44.272 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-03-06T07:39:44.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.272+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87c406b4d0 msgr2=0x7f87c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87c406b4d0 0x7f87c406d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f87d8036000 tx=0x7f87d8030000 comp rx=0 tx=0).stop 2024-03-06T07:39:44.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e81603e0 msgr2=0x7f87e80b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e81603e0 0x7f87e80b6a70 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f87e000c700 tx=0x7f87e000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 shutdown_connections 2024-03-06T07:39:44.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87e81603e0 0x7f87e80b6a70 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f87e815c330 0x7f87e80b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 --2- 172.21.15.114:0/4210234558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87e80b43b0 0x7f87e8159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 >> 172.21.15.114:0/4210234558 conn(0x7f87e80b3790 msgr2=0x7f87e8070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:44.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 shutdown_connections 2024-03-06T07:39:44.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.273+0000 7f87ed000640 1 -- 172.21.15.114:0/4210234558 wait complete. 2024-03-06T07:39:44.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:39:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.494+0000 7f4a5ca00640 1 Processor -- start 2024-03-06T07:39:44.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.494+0000 7f4a5ca00640 1 -- start start 2024-03-06T07:39:44.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.495+0000 7f4a5ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 0x7f4a58166f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.495+0000 7f4a5ca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a58145b30 con 0x7f4a580c6460 2024-03-06T07:39:44.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.495+0000 7f4a5ca00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a58142ce0 con 0x7f4a580c4930 2024-03-06T07:39:44.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.495+0000 7f4a5ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580c68c0 con 0x7f4a58164b30 2024-03-06T07:39:44.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.496+0000 7f4a4fe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4a580c6460 0x7f4a580c5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55840/0 (socket says 172.21.15.114:55840) 2024-03-06T07:39:44.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.496+0000 7f4a4fe00640 1 -- 172.21.15.114:0/144121766 learned_addr learned my addr 172.21.15.114:0/144121766 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-06T07:39:44.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.496+0000 7f4a56800640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 0x7f4a58166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.496+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893166407 0 0) 0x7f4a40001110 con 0x7f4a580c6460 2024-03-06T07:39:44.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.496+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a3c003520 con 0x7f4a580c6460 2024-03-06T07:39:44.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4153706991 0 0) 0x7f4a3c003520 con 0x7f4a580c6460 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 msgr2=0x7f4a58166f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 0x7f4a58166f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 >> v1:172.21.15.177:6789/0 conn(0x7f4a580c4930 legacy=0x7f4a580c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580cbca0 con 0x7f4a580c6460 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 365149296 0 0) 0x7f4a40001840 con 0x7f4a580c6460 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 >> v1:172.21.15.114:6789/0 conn(0x7f4a580c6460 legacy=0x7f4a580c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c0039e0 0x7f4a3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 0x7f4a3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580c68c0 con 0x7f4a3c004d40 2024-03-06T07:39:44.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c00bc70 con 0x7f4a3c0039e0 2024-03-06T07:39:44.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c00be70 con 0x7f4a58164b30 2024-03-06T07:39:44.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.497+0000 7f4a56800640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a4fe00640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 0x7f4a3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a55e00640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c0039e0 0x7f4a3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a56800640 1 -- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 msgr2=0x7f4a3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:44.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a56800640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a58164b30 0x7f4a3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a56800640 1 -- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c0039e0 msgr2=0x7f4a3c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a56800640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c0039e0 0x7f4a3c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a56800640 1 -- 172.21.15.114:0/144121766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c003520 con 0x7f4a3c004d40 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.498+0000 7f4a55e00640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c0039e0 0x7f4a3c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.499+0000 7f4a56800640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a3c0051c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f4a58073330 tx=0x7f4a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8383b0026725815 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.499+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50039070 con 0x7f4a3c004d40 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.499+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5002cc70 con 0x7f4a3c004d40 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.499+0000 7f4a55400640 1 -- 172.21.15.114:0/144121766 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a50009be0 con 0x7f4a3c004d40 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.500+0000 7f4a5ca00640 1 -- 172.21.15.114:0/144121766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 msgr2=0x7f4a3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.500+0000 7f4a5ca00640 1 --2- 172.21.15.114:0/144121766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a3c0051c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f4a58073330 tx=0x7f4a3c005830 comp rx=0 tx=0).stop 2024-03-06T07:39:44.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.500+0000 7f4a56800640 1 -- 172.21.15.114:0/144121766 reap_dead start 2024-03-06T07:39:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.500+0000 7f4a5ca00640 1 -- 172.21.15.114:0/144121766 shutdown_connections 2024-03-06T07:39:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.500+0000 7f4a5ca00640 1 -- 172.21.15.114:0/144121766 >> 172.21.15.114:0/144121766 conn(0x7f4a580cc4f0 msgr2=0x7f4a580cc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.500+0000 7f4a5ca00640 1 -- 172.21.15.114:0/144121766 shutdown_connections 2024-03-06T07:39:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.500+0000 7f4a5ca00640 1 -- 172.21.15.114:0/144121766 wait complete. 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.502+0000 7f4a5ca00640 1 Processor -- start 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.502+0000 7f4a5ca00640 1 -- start start 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.503+0000 7f4a5ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a58141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.503+0000 7f4a5ca00640 1 --2- >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a580b4d30 0x7f4a580b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.503+0000 7f4a5ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a580b51b0 0x7f4a580b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.503+0000 7f4a5ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580ca3b0 con 0x7f4a3c004d40 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.503+0000 7f4a5ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580ca0d0 con 0x7f4a580b51b0 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.503+0000 7f4a5ca00640 1 -- --> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5806e600 con 0x7f4a580b4d30 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a58141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a58141dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46514/0 (socket says 172.21.15.114:46514) 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 -- 172.21.15.114:0/1884833440 learned_addr learned my addr 172.21.15.114:0/1884833440 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a4fe00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a580b4d30 0x7f4a580b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a56800640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a580b51b0 0x7f4a580b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 -- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a580b4d30 msgr2=0x7f4a580b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a580b4d30 0x7f4a580b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 -- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a580b51b0 msgr2=0x7f4a580b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a580b51b0 0x7f4a580b0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.504+0000 7f4a55e00640 1 -- 172.21.15.114:0/1884833440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a580b1800 con 0x7f4a3c004d40 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.505+0000 7f4a55e00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a58141dc0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f4a48000f60 tx=0x7f4a4800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.505+0000 7f4a4e000640 1 -- 172.21.15.114:0/1884833440 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48005480 con 0x7f4a3c004d40 2024-03-06T07:39:44.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.505+0000 7f4a4e000640 1 -- 172.21.15.114:0/1884833440 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a48017070 con 0x7f4a3c004d40 2024-03-06T07:39:44.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.505+0000 7f4a4e000640 1 -- 172.21.15.114:0/1884833440 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48003810 con 0x7f4a3c004d40 2024-03-06T07:39:44.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.505+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a580b1ae0 con 0x7f4a3c004d40 2024-03-06T07:39:44.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.505+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a580b14d0 con 0x7f4a3c004d40 2024-03-06T07:39:44.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.507+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5804f330 con 0x7f4a3c004d40 2024-03-06T07:39:44.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.507+0000 7f4a4e000640 1 -- 172.21.15.114:0/1884833440 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a480039b0 con 0x7f4a3c004d40 2024-03-06T07:39:44.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.508+0000 7f4a4e000640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a3406b500 0x7f4a3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:44.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.508+0000 7f4a4e000640 1 -- 172.21.15.114:0/1884833440 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a4808a070 con 0x7f4a3c004d40 2024-03-06T07:39:44.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.508+0000 7f4a4fe00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a3406b500 0x7f4a3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:44.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.509+0000 7f4a4fe00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a3406b500 0x7f4a3406d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4a40001050 tx=0x7f4a40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:44.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.512+0000 7f4a4e000640 1 -- 172.21.15.114:0/1884833440 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a48058ea0 con 0x7f4a3c004d40 2024-03-06T07:39:44.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.755+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4a5806e3e0 con 0x7f4a3c004d40 2024-03-06T07:39:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.756+0000 7f4a4e000640 1 -- 172.21.15.114:0/1884833440 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v129) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4a480587a0 con 0x7f4a3c004d40 2024-03-06T07:39:44.758 INFO:teuthology.orchestra.run.smithi114.stderr:listed 0 entries 2024-03-06T07:39:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.759+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a3406b500 msgr2=0x7f4a3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.759+0000 7f4a5ca00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a3406b500 0x7f4a3406d9c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4a40001050 tx=0x7f4a40009040 comp rx=0 tx=0).stop 2024-03-06T07:39:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.759+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 msgr2=0x7f4a58141dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.759+0000 7f4a5ca00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a58141dc0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f4a48000f60 tx=0x7f4a4800d9f0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 shutdown_connections 2024-03-06T07:39:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a580b51b0 0x7f4a580b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.177:3301/0,v1:172.21.15.177:6790/0] conn(0x7f4a580b4d30 0x7f4a580b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:6832/3967440666,v1:172.21.15.114:6833/3967440666] conn(0x7f4a3406b500 0x7f4a3406d9c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 --2- 172.21.15.114:0/1884833440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a3c004d40 0x7f4a58141dc0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 >> 172.21.15.114:0/1884833440 conn(0x7f4a580cc4f0 msgr2=0x7f4a5815eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 shutdown_connections 2024-03-06T07:39:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-06T07:39:44.760+0000 7f4a5ca00640 1 -- 172.21.15.114:0/1884833440 wait complete. 2024-03-06T07:39:44.787 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:39:44.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:39:44.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:39:48.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:48.145 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:18.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:18.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:48.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:48.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:18.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:18.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:48.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:48.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:18.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:18.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:49.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:49.202 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:19.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:19.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:49.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:49.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:19.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:19.976 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:44.793 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-03-06T07:44:44.795 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-06T07:44:50.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:50.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:20.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:20.319 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:50.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:50.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:20.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:20.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:50.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:50.805 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:20.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:20.951 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:51.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:51.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:21.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:21.238 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:51.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:51.415 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:21.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:21.561 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:51.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:51.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:21.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:21.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:51.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:52.002 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:22.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:22.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:52.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:52.305 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:22.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:22.444 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:52.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:52.582 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:22.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:22.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:52.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:52.901 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:23.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:23.035 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:53.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:53.167 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:23.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:23.314 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:53.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:53.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:23.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:23.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:53.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:53.958 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:24.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:24.097 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:54.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:54.244 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:24.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:24.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:54.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:54.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:24.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:24.746 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:54.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:54.919 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:25.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:25.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:55.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:55.209 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:25.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:25.355 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:55.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:55.500 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:25.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:25.655 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:55.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:55.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:26.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:26.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:56.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:56.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:26.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:26.292 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:56.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:56.442 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:26.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:26.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:56.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:56.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:26.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:26.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:57.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:57.009 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:27.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:27.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:57.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:57.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:27.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:27.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:57.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:57.853 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:27.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:28.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:58.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:58.450 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:28.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:28.589 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:58.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:58.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:28.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:28.954 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:59.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:59.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:29.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:29.530 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:59.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:59.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:29.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:29.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:00.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:00.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:30.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:30.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:00.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:00.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:30.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:30.781 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:00.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:00.919 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:31.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:31.065 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:01.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:01.209 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:31.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:31.346 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:01.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:01.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:31.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:31.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:01.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:01.827 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:32.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:32.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:02.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:02.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:32.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:32.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:02.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:02.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:32.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:32.888 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:03.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:03.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:33.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:33.324 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:03.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:03.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:33.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:33.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:03.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:03.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:34.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:34.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:04.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:04.401 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:34.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:34.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:04.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:04.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:34.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:34.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:05.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:05.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:35.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:35.372 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:05.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:05.509 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:35.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:35.722 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:05.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:05.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:36.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:36.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:06.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:06.277 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:36.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:36.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:06.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:06.681 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:36.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:36.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:07.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:07.018 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:37.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:37.221 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:07.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:07.615 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:37.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:37.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:08.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:08.026 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:38.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:38.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:08.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:08.620 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:38.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:38.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:08.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:08.977 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:39.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:39.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:09.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:09.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:39.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:39.623 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:09.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:09.819 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:40.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:40.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:10.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:10.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:30.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:30.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:30.487852+0000) 2024-03-06T08:39:31.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:31.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:31.488105+0000) 2024-03-06T08:39:32.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:32.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:32.488352+0000) 2024-03-06T08:39:33.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:33.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:33.488597+0000) 2024-03-06T08:39:34.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:34.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:34.488815+0000) 2024-03-06T08:39:35.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:35.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:35.489038+0000) 2024-03-06T08:39:36.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:36.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:36.489332+0000) 2024-03-06T08:39:37.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:37.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:37.489522+0000) 2024-03-06T08:39:38.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:38.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:38.489781+0000) 2024-03-06T08:39:39.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:39.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:39.490007+0000) 2024-03-06T08:39:40.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:40.465 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:40.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:40.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:40.490191+0000) 2024-03-06T08:39:41.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:41.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:41.490425+0000) 2024-03-06T08:39:42.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:42.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:42.490653+0000) 2024-03-06T08:39:43.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:43.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:43.490902+0000) 2024-03-06T08:39:44.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:44.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:44.491155+0000) 2024-03-06T08:39:45.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:45.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:45.491403+0000) 2024-03-06T08:39:46.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:46.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:46.491647+0000) 2024-03-06T08:39:47.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:47.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:47.491886+0000) 2024-03-06T08:39:48.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:48.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:48.492080+0000) 2024-03-06T08:39:49.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:49.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:49.492273+0000) 2024-03-06T08:39:50.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:50.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:50.492507+0000) 2024-03-06T08:39:51.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:51.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:51.492777+0000) 2024-03-06T08:39:52.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:52.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:52.493008+0000) 2024-03-06T08:39:53.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:53.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:53.493238+0000) 2024-03-06T08:39:54.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:54.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:54.493476+0000) 2024-03-06T08:39:55.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:55.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:55.493737+0000) 2024-03-06T08:39:56.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:56.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:56.493977+0000) 2024-03-06T08:39:57.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:57.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:57.494165+0000) 2024-03-06T08:39:58.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:58.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:58.494349+0000) 2024-03-06T08:39:59.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:39:59.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:59.494555+0000) 2024-03-06T08:40:00.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:00.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:00.494790+0000) 2024-03-06T08:40:01.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:01.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:01.495026+0000) 2024-03-06T08:40:02.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:02.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:02.495251+0000) 2024-03-06T08:40:03.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:03.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:03.495481+0000) 2024-03-06T08:40:04.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:04.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:04.495755+0000) 2024-03-06T08:40:05.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:05.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:05.495987+0000) 2024-03-06T08:40:06.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:06.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:06.496248+0000) 2024-03-06T08:40:07.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:07.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:07.496440+0000) 2024-03-06T08:40:08.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:08.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:08.496637+0000) 2024-03-06T08:40:09.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:09.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:09.496854+0000) 2024-03-06T08:40:10.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:10.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:10.497037+0000) 2024-03-06T08:40:10.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:10.602 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:11.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:11.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:11.497279+0000) 2024-03-06T08:40:12.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:12.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:12.497514+0000) 2024-03-06T08:40:13.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:13.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:13.497785+0000) 2024-03-06T08:40:14.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:14.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:14.497983+0000) 2024-03-06T08:40:15.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:15.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:15.498179+0000) 2024-03-06T08:40:16.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:16.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:16.498440+0000) 2024-03-06T08:40:17.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:17.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:17.498647+0000) 2024-03-06T08:40:18.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:18.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:18.498907+0000) 2024-03-06T08:40:19.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:19.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:19.499142+0000) 2024-03-06T08:40:20.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:20.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:20.499366+0000) 2024-03-06T08:40:21.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:21.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:21.499549+0000) 2024-03-06T08:40:22.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:22.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:22.499792+0000) 2024-03-06T08:40:23.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:23.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:23.500042+0000) 2024-03-06T08:40:24.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:24.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:24.500306+0000) 2024-03-06T08:40:25.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:25.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:25.500551+0000) 2024-03-06T08:40:26.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:26.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:26.500805+0000) 2024-03-06T08:40:27.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:27.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:27.500999+0000) 2024-03-06T08:40:28.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:28.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:28.501192+0000) 2024-03-06T08:40:29.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:29.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:29.501406+0000) 2024-03-06T08:40:30.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:30.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:30.501637+0000) 2024-03-06T08:40:31.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:31.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:31.501842+0000) 2024-03-06T08:40:32.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:32.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:32.502049+0000) 2024-03-06T08:40:33.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:33.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:33.502257+0000) 2024-03-06T08:40:34.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:34.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:34.502497+0000) 2024-03-06T08:40:35.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:35.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:35.502758+0000) 2024-03-06T08:40:36.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:36.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:36.503001+0000) 2024-03-06T08:40:37.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:37.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:37.503230+0000) 2024-03-06T08:40:38.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:38.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:38.503458+0000) 2024-03-06T08:40:39.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:39.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:39.503656+0000) 2024-03-06T08:40:40.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:40.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:40.503915+0000) 2024-03-06T08:40:40.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:40.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:41.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:41.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:41.504156+0000) 2024-03-06T08:40:42.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:42.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:42.504394+0000) 2024-03-06T08:40:43.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:43.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:43.504624+0000) 2024-03-06T08:40:44.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:44.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:44.504875+0000) 2024-03-06T08:40:45.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:45.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:45.505126+0000) 2024-03-06T08:40:46.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:46.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:46.505387+0000) 2024-03-06T08:40:47.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:47.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:47.505631+0000) 2024-03-06T08:40:48.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:48.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:48.505860+0000) 2024-03-06T08:40:49.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:49.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:49.506053+0000) 2024-03-06T08:40:50.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:50.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:50.506281+0000) 2024-03-06T08:40:51.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:51.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:51.506493+0000) 2024-03-06T08:40:52.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:52.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:52.506718+0000) 2024-03-06T08:40:53.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:53.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:53.506968+0000) 2024-03-06T08:40:54.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:54.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:54.507205+0000) 2024-03-06T08:40:55.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:55.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:55.507388+0000) 2024-03-06T08:40:56.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:56.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:56.507650+0000) 2024-03-06T08:40:57.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:57.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:57.507903+0000) 2024-03-06T08:40:58.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:58.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:58.508138+0000) 2024-03-06T08:40:59.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:40:59.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:59.508358+0000) 2024-03-06T08:41:00.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:00.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:00.508567+0000) 2024-03-06T08:41:01.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:01.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:01.508776+0000) 2024-03-06T08:41:02.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:02.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:02.509006+0000) 2024-03-06T08:41:03.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:03.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:03.509239+0000) 2024-03-06T08:41:04.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:04.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:04.509489+0000) 2024-03-06T08:41:05.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:05.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:05.509755+0000) 2024-03-06T08:41:06.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:06.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:06.509938+0000) 2024-03-06T08:41:07.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:07.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:07.510187+0000) 2024-03-06T08:41:08.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:08.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:08.510431+0000) 2024-03-06T08:41:09.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:09.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:09.510630+0000) 2024-03-06T08:41:10.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:10.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:10.510854+0000) 2024-03-06T08:41:11.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:11.027 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:11.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:11.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:11.511091+0000) 2024-03-06T08:41:12.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:12.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:12.511337+0000) 2024-03-06T08:41:13.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:13.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:13.511578+0000) 2024-03-06T08:41:14.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:14.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:14.511793+0000) 2024-03-06T08:41:15.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:15.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:15.512030+0000) 2024-03-06T08:41:16.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:16.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:16.512271+0000) 2024-03-06T08:41:17.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:17.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:17.512524+0000) 2024-03-06T08:41:18.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:18.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:18.512790+0000) 2024-03-06T08:41:19.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:19.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:19.513016+0000) 2024-03-06T08:41:20.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:20.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:20.513231+0000) 2024-03-06T08:41:21.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:21.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:21.513458+0000) 2024-03-06T08:41:22.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:22.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:22.513725+0000) 2024-03-06T08:41:23.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:23.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:23.514029+0000) 2024-03-06T08:41:24.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:24.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:24.514269+0000) 2024-03-06T08:41:25.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:25.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:25.514461+0000) 2024-03-06T08:41:26.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:26.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:26.514689+0000) 2024-03-06T08:41:27.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:27.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:27.514933+0000) 2024-03-06T08:41:28.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:28.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:28.515170+0000) 2024-03-06T08:41:29.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:29.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:29.515410+0000) 2024-03-06T08:41:30.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:30.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:30.515633+0000) 2024-03-06T08:41:31.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:31.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:31.515865+0000) 2024-03-06T08:41:32.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:32.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:32.516053+0000) 2024-03-06T08:41:33.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:33.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:33.516303+0000) 2024-03-06T08:41:34.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:34.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:34.516551+0000) 2024-03-06T08:41:35.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:35.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:35.516796+0000) 2024-03-06T08:41:36.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:36.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:36.517034+0000) 2024-03-06T08:41:37.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:37.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:37.517306+0000) 2024-03-06T08:41:38.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:38.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:38.517549+0000) 2024-03-06T08:41:39.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:39.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:39.517811+0000) 2024-03-06T08:41:40.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:40.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:40.518013+0000) 2024-03-06T08:41:41.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:41.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:41.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:41.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:41.518204+0000) 2024-03-06T08:41:42.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:42.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:42.518400+0000) 2024-03-06T08:41:43.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:43.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:43.518643+0000) 2024-03-06T08:41:44.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:44.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:44.518915+0000) 2024-03-06T08:41:45.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:45.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:45.519145+0000) 2024-03-06T08:41:46.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:46.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:46.519380+0000) 2024-03-06T08:41:47.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:47.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:47.519590+0000) 2024-03-06T08:41:48.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:48.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:48.519810+0000) 2024-03-06T08:41:49.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:49.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:49.520033+0000) 2024-03-06T08:41:50.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:50.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:50.520212+0000) 2024-03-06T08:41:51.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:51.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:51.520434+0000) 2024-03-06T08:41:52.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:52.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:52.520632+0000) 2024-03-06T08:41:53.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:53.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:53.520921+0000) 2024-03-06T08:41:54.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:54.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:54.521162+0000) 2024-03-06T08:41:55.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:55.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:55.521388+0000) 2024-03-06T08:41:56.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:56.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:56.521617+0000) 2024-03-06T08:41:57.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:57.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:57.521848+0000) 2024-03-06T08:41:58.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:58.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:58.522079+0000) 2024-03-06T08:41:59.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:41:59.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:59.522299+0000) 2024-03-06T08:42:00.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:00.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:00.522524+0000) 2024-03-06T08:42:01.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:01.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:01.522793+0000) 2024-03-06T08:42:02.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:02.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:02.523029+0000) 2024-03-06T08:42:03.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:03.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:03.523230+0000) 2024-03-06T08:42:04.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:04.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:04.523470+0000) 2024-03-06T08:42:05.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:05.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:05.523730+0000) 2024-03-06T08:42:06.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:06.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:06.523981+0000) 2024-03-06T08:42:07.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:07.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:07.524226+0000) 2024-03-06T08:42:08.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:08.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:08.524467+0000) 2024-03-06T08:42:09.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:09.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:09.524755+0000) 2024-03-06T08:42:10.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:10.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:10.524961+0000) 2024-03-06T08:42:11.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:11.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:11.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:11.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:11.525186+0000) 2024-03-06T08:42:12.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:12.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:12.525425+0000) 2024-03-06T08:42:13.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:13.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:13.525704+0000) 2024-03-06T08:42:14.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:14.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:14.525931+0000) 2024-03-06T08:42:15.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:15.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:15.526123+0000) 2024-03-06T08:42:16.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:16.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:16.526348+0000) 2024-03-06T08:42:17.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:17.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:17.526575+0000) 2024-03-06T08:42:18.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:18.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:18.526811+0000) 2024-03-06T08:42:19.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:19.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:19.527015+0000) 2024-03-06T08:42:20.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:20.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:20.527204+0000) 2024-03-06T08:42:21.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:21.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:21.527441+0000) 2024-03-06T08:42:22.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:22.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:22.527730+0000) 2024-03-06T08:42:23.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:23.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:23.527963+0000) 2024-03-06T08:42:24.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:24.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:24.528165+0000) 2024-03-06T08:42:25.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:25.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:25.528356+0000) 2024-03-06T08:42:26.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:26.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:26.528593+0000) 2024-03-06T08:42:27.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:27.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:27.528845+0000) 2024-03-06T08:42:28.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:28.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:28.529040+0000) 2024-03-06T08:42:29.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:29.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:29.529229+0000) 2024-03-06T08:42:30.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:30.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:30.529407+0000) 2024-03-06T08:42:31.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:31.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:31.529637+0000) 2024-03-06T08:42:32.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:32.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:32.529901+0000) 2024-03-06T08:42:33.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:33.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:33.530087+0000) 2024-03-06T08:42:34.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:34.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:34.530325+0000) 2024-03-06T08:42:35.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:35.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:35.530510+0000) 2024-03-06T08:42:36.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:36.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:36.530746+0000) 2024-03-06T08:42:37.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:37.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:37.530946+0000) 2024-03-06T08:42:38.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:38.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:38.531130+0000) 2024-03-06T08:42:39.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:39.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:39.531376+0000) 2024-03-06T08:42:40.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:40.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:40.531620+0000) 2024-03-06T08:42:41.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:41.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:41.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:41.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:41.531808+0000) 2024-03-06T08:42:42.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:42.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:42.531961+0000) 2024-03-06T08:42:43.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:43.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:43.532184+0000) 2024-03-06T08:42:44.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:44.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:44.532416+0000) 2024-03-06T08:42:45.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:45.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:45.532639+0000) 2024-03-06T08:42:46.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:46.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:46.532936+0000) 2024-03-06T08:42:47.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:47.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:47.533182+0000) 2024-03-06T08:42:48.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:48.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:48.533435+0000) 2024-03-06T08:42:49.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:49.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:49.533719+0000) 2024-03-06T08:42:50.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:50.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:50.533967+0000) 2024-03-06T08:42:51.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:51.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:51.534214+0000) 2024-03-06T08:42:52.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:52.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:52.534364+0000) 2024-03-06T08:42:53.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:53.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:53.534570+0000) 2024-03-06T08:42:54.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:54.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:54.534880+0000) 2024-03-06T08:42:55.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:55.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:55.535154+0000) 2024-03-06T08:42:56.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:56.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:56.535429+0000) 2024-03-06T08:42:57.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:57.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:57.535705+0000) 2024-03-06T08:42:58.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:58.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:58.535999+0000) 2024-03-06T08:42:59.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:42:59.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:59.536256+0000) 2024-03-06T08:43:00.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:00.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:00.536387+0000) 2024-03-06T08:43:01.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:01.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:01.536610+0000) 2024-03-06T08:43:02.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:02.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:02.536847+0000) 2024-03-06T08:43:03.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:03.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:03.537077+0000) 2024-03-06T08:43:04.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:04.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:04.537264+0000) 2024-03-06T08:43:05.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:05.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:05.537490+0000) 2024-03-06T08:43:06.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:06.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:06.537697+0000) 2024-03-06T08:43:07.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:07.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:07.537944+0000) 2024-03-06T08:43:08.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:08.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:08.538168+0000) 2024-03-06T08:43:09.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:09.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:09.538397+0000) 2024-03-06T08:43:10.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:10.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:10.538622+0000) 2024-03-06T08:43:11.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:11.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:11.538862+0000) 2024-03-06T08:43:11.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:11.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:12.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:12.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:12.539058+0000) 2024-03-06T08:43:13.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:13.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:13.539308+0000) 2024-03-06T08:43:14.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:14.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:14.539463+0000) 2024-03-06T08:43:15.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:15.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:15.539736+0000) 2024-03-06T08:43:16.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:16.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:16.539988+0000) 2024-03-06T08:43:17.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:17.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:17.540239+0000) 2024-03-06T08:43:18.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:18.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:18.540468+0000) 2024-03-06T08:43:19.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:19.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:19.540752+0000) 2024-03-06T08:43:20.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:20.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:20.540989+0000) 2024-03-06T08:43:21.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:21.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:21.541173+0000) 2024-03-06T08:43:22.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:22.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:22.541399+0000) 2024-03-06T08:43:23.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:23.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:23.541578+0000) 2024-03-06T08:43:24.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:24.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:24.541796+0000) 2024-03-06T08:43:25.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:25.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:25.542000+0000) 2024-03-06T08:43:26.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:26.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:26.542191+0000) 2024-03-06T08:43:27.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:27.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:27.542408+0000) 2024-03-06T08:43:28.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:28.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:28.542644+0000) 2024-03-06T08:43:29.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:29.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:29.542917+0000) 2024-03-06T08:43:30.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:30.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:30.543075+0000) 2024-03-06T08:43:31.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:31.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:31.543257+0000) 2024-03-06T08:43:32.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:32.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:32.543484+0000) 2024-03-06T08:43:33.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:33.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:33.543709+0000) 2024-03-06T08:43:34.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:34.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:34.543951+0000) 2024-03-06T08:43:35.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:35.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:35.544139+0000) 2024-03-06T08:43:36.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:36.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:36.544358+0000) 2024-03-06T08:43:37.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:37.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:37.544554+0000) 2024-03-06T08:43:38.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:38.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:38.544784+0000) 2024-03-06T08:43:39.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:39.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:39.545013+0000) 2024-03-06T08:43:40.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:40.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:40.545235+0000) 2024-03-06T08:43:41.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:41.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:41.545462+0000) 2024-03-06T08:43:42.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:42.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:42.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:42.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:42.545739+0000) 2024-03-06T08:43:43.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:43.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:43.545950+0000) 2024-03-06T08:43:44.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:44.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:44.546151+0000) 2024-03-06T08:43:45.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:45.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:45.546375+0000) 2024-03-06T08:43:46.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:46.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:46.546562+0000) 2024-03-06T08:43:47.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:47.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:47.546798+0000) 2024-03-06T08:43:48.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:48.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:48.546992+0000) 2024-03-06T08:43:49.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:49.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:49.547224+0000) 2024-03-06T08:43:50.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:50.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:50.547453+0000) 2024-03-06T08:43:51.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:51.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:51.547709+0000) 2024-03-06T08:43:52.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:52.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:52.548024+0000) 2024-03-06T08:43:53.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:53.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:53.548299+0000) 2024-03-06T08:43:54.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:54.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:54.548548+0000) 2024-03-06T08:43:55.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:55.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:55.548761+0000) 2024-03-06T08:43:56.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:56.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:56.548977+0000) 2024-03-06T08:43:57.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:57.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:57.549231+0000) 2024-03-06T08:43:58.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:58.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:58.549445+0000) 2024-03-06T08:43:59.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:43:59.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:59.549707+0000) 2024-03-06T08:44:00.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:00.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:00.549927+0000) 2024-03-06T08:44:01.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:01.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:01.550123+0000) 2024-03-06T08:44:02.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:02.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:02.550332+0000) 2024-03-06T08:44:03.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:03.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:03.550575+0000) 2024-03-06T08:44:04.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:04.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:04.550794+0000) 2024-03-06T08:44:05.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:05.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:05.551018+0000) 2024-03-06T08:44:06.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:06.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:06.551233+0000) 2024-03-06T08:44:07.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:07.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:07.551453+0000) 2024-03-06T08:44:08.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:08.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:08.551716+0000) 2024-03-06T08:44:09.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:09.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:09.551918+0000) 2024-03-06T08:44:10.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:10.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:10.552134+0000) 2024-03-06T08:44:11.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:11.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:11.552366+0000) 2024-03-06T08:44:12.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:12.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:12.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:12.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:12.552568+0000) 2024-03-06T08:44:13.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:13.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:13.552813+0000) 2024-03-06T08:44:14.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:14.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:14.553035+0000) 2024-03-06T08:44:15.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:15.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:15.553239+0000) 2024-03-06T08:44:16.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:16.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:16.553484+0000) 2024-03-06T08:44:17.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:17.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:17.553769+0000) 2024-03-06T08:44:18.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:18.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:18.553964+0000) 2024-03-06T08:44:19.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:19.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:19.554146+0000) 2024-03-06T08:44:20.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:20.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:20.554367+0000) 2024-03-06T08:44:21.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:21.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:21.554592+0000) 2024-03-06T08:44:22.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:22.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:22.554826+0000) 2024-03-06T08:44:23.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:23.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:23.555031+0000) 2024-03-06T08:44:24.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:24.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:24.555267+0000) 2024-03-06T08:44:25.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:25.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:25.555514+0000) 2024-03-06T08:44:26.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:26.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:26.555789+0000) 2024-03-06T08:44:27.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:27.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:27.555997+0000) 2024-03-06T08:44:28.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:28.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:28.556197+0000) 2024-03-06T08:44:29.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:29.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:29.556400+0000) 2024-03-06T08:44:30.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:30.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:30.556630+0000) 2024-03-06T08:44:31.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:31.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:31.556876+0000) 2024-03-06T08:44:32.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:32.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:32.557152+0000) 2024-03-06T08:44:33.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:33.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:33.557368+0000) 2024-03-06T08:44:34.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:34.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:34.557602+0000) 2024-03-06T08:44:35.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:35.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:35.557827+0000) 2024-03-06T08:44:36.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:36.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:36.558072+0000) 2024-03-06T08:44:37.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:37.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:37.558319+0000) 2024-03-06T08:44:38.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:38.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:38.558536+0000) 2024-03-06T08:44:39.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:39.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:39.558771+0000) 2024-03-06T08:44:40.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:40.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:40.558998+0000) 2024-03-06T08:44:41.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:41.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:41.559179+0000) 2024-03-06T08:44:42.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:42.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:42.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:42.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:42.559365+0000) 2024-03-06T08:44:43.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:43.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:43.559522+0000) 2024-03-06T08:44:44.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:44.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:44.559776+0000) 2024-03-06T08:44:45.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:45.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:45.560047+0000) 2024-03-06T08:44:46.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:46.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:46.560271+0000) 2024-03-06T08:44:47.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:47.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:47.560499+0000) 2024-03-06T08:44:48.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:48.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:48.560776+0000) 2024-03-06T08:44:49.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:49.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:49.560966+0000) 2024-03-06T08:44:50.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:50.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:50.561152+0000) 2024-03-06T08:44:51.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:51.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:51.561372+0000) 2024-03-06T08:44:52.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:52.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:52.561608+0000) 2024-03-06T08:44:53.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:53.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:53.561852+0000) 2024-03-06T08:44:54.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:54.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:54.562099+0000) 2024-03-06T08:44:55.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:55.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:55.562289+0000) 2024-03-06T08:44:56.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:56.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:56.562516+0000) 2024-03-06T08:44:57.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:57.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:57.562768+0000) 2024-03-06T08:44:58.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:58.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:58.563036+0000) 2024-03-06T08:44:59.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:44:59.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:59.563270+0000) 2024-03-06T08:45:00.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:00.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:00.563471+0000) 2024-03-06T08:45:01.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:01.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:01.563743+0000) 2024-03-06T08:45:02.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:02.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:02.563982+0000) 2024-03-06T08:45:03.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:03.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:03.564202+0000) 2024-03-06T08:45:04.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:04.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:04.564413+0000) 2024-03-06T08:45:05.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:05.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:05.564647+0000) 2024-03-06T08:45:06.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:06.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:06.564870+0000) 2024-03-06T08:45:07.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:07.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:07.565107+0000) 2024-03-06T08:45:08.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:08.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:08.565378+0000) 2024-03-06T08:45:09.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:09.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:09.565651+0000) 2024-03-06T08:45:10.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:10.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:10.565892+0000) 2024-03-06T08:45:11.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:11.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:11.566111+0000) 2024-03-06T08:45:12.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:12.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:12.566367+0000) 2024-03-06T08:45:12.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:12.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:13.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:13.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:13.566626+0000) 2024-03-06T08:45:14.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:14.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:14.566900+0000) 2024-03-06T08:45:15.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:15.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:15.567183+0000) 2024-03-06T08:45:16.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:16.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:16.567429+0000) 2024-03-06T08:45:17.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:17.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:17.567632+0000) 2024-03-06T08:45:18.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:18.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:18.567905+0000) 2024-03-06T08:45:19.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:19.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:19.568146+0000) 2024-03-06T08:45:20.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:20.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:20.568426+0000) 2024-03-06T08:45:21.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:21.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:21.568652+0000) 2024-03-06T08:45:22.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:22.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:22.568945+0000) 2024-03-06T08:45:23.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:23.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:23.569175+0000) 2024-03-06T08:45:24.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:24.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:24.569405+0000) 2024-03-06T08:45:25.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:25.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:25.569654+0000) 2024-03-06T08:45:26.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:26.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:26.569886+0000) 2024-03-06T08:45:27.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:27.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:27.570075+0000) 2024-03-06T08:45:28.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:28.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:28.570277+0000) 2024-03-06T08:45:29.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:29.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:29.570485+0000) 2024-03-06T08:45:30.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:30.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:30.570698+0000) 2024-03-06T08:45:31.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:31.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:31.570914+0000) 2024-03-06T08:45:32.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:32.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:32.571108+0000) 2024-03-06T08:45:33.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:33.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:33.571317+0000) 2024-03-06T08:45:34.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:34.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:34.571551+0000) 2024-03-06T08:45:35.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:35.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:35.571751+0000) 2024-03-06T08:45:36.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:36.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:36.571983+0000) 2024-03-06T08:45:37.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:37.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:37.572260+0000) 2024-03-06T08:45:38.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:38.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:38.572541+0000) 2024-03-06T08:45:39.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:39.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:39.572825+0000) 2024-03-06T08:45:40.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:40.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:40.573005+0000) 2024-03-06T08:45:41.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:41.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:41.573254+0000) 2024-03-06T08:45:42.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:42.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:42.573501+0000) 2024-03-06T08:45:43.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:43.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:43.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:43.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:43.573773+0000) 2024-03-06T08:45:44.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:44.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:44.573991+0000) 2024-03-06T08:45:45.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:45.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:45.574217+0000) 2024-03-06T08:45:46.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:46.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:46.574442+0000) 2024-03-06T08:45:47.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:47.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:47.574733+0000) 2024-03-06T08:45:48.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:48.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:48.574926+0000) 2024-03-06T08:45:49.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:49.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:49.575158+0000) 2024-03-06T08:45:50.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:50.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:50.575393+0000) 2024-03-06T08:45:51.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:51.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:51.575616+0000) 2024-03-06T08:45:52.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:52.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:52.575835+0000) 2024-03-06T08:45:53.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:53.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:53.576067+0000) 2024-03-06T08:45:54.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:54.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:54.576256+0000) 2024-03-06T08:45:55.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:55.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:55.576470+0000) 2024-03-06T08:45:56.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:56.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:56.576756+0000) 2024-03-06T08:45:57.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:57.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:57.577002+0000) 2024-03-06T08:45:58.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:58.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:58.577240+0000) 2024-03-06T08:45:59.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:45:59.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:59.577471+0000) 2024-03-06T08:46:00.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:00.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:00.577732+0000) 2024-03-06T08:46:01.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:01.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:01.577925+0000) 2024-03-06T08:46:02.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:02.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:02.578133+0000) 2024-03-06T08:46:03.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:03.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:03.578357+0000) 2024-03-06T08:46:04.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:04.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:04.578557+0000) 2024-03-06T08:46:05.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:05.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:05.578795+0000) 2024-03-06T08:46:06.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:06.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:06.579005+0000) 2024-03-06T08:46:07.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:07.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:07.579194+0000) 2024-03-06T08:46:08.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:08.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:08.579418+0000) 2024-03-06T08:46:09.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:09.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:09.579642+0000) 2024-03-06T08:46:10.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:10.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:10.579938+0000) 2024-03-06T08:46:11.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:11.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:11.580147+0000) 2024-03-06T08:46:12.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:12.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:12.580394+0000) 2024-03-06T08:46:13.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:13.438 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:13.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:13.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:13.580634+0000) 2024-03-06T08:46:14.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:14.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:14.580812+0000) 2024-03-06T08:46:15.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:15.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:15.580999+0000) 2024-03-06T08:46:16.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:16.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:16.581193+0000) 2024-03-06T08:46:17.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:17.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:17.581417+0000) 2024-03-06T08:46:18.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:18.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:18.581645+0000) 2024-03-06T08:46:19.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:19.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:19.581890+0000) 2024-03-06T08:46:20.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:20.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:20.582071+0000) 2024-03-06T08:46:21.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:21.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:21.582254+0000) 2024-03-06T08:46:22.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:22.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:22.582498+0000) 2024-03-06T08:46:23.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:23.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:23.582714+0000) 2024-03-06T08:46:24.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:24.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:24.582939+0000) 2024-03-06T08:46:25.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:25.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:25.583121+0000) 2024-03-06T08:46:26.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:26.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:26.583305+0000) 2024-03-06T08:46:27.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:27.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:27.583535+0000) 2024-03-06T08:46:28.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:28.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:28.583781+0000) 2024-03-06T08:46:29.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:29.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:29.583916+0000) 2024-03-06T08:46:30.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:30.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:30.584137+0000) 2024-03-06T08:46:31.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:31.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:31.584358+0000) 2024-03-06T08:46:32.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:32.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:32.584612+0000) 2024-03-06T08:46:33.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:33.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:33.584752+0000) 2024-03-06T08:46:34.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:34.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:34.584972+0000) 2024-03-06T08:46:35.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:35.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:35.585188+0000) 2024-03-06T08:46:36.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:36.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:36.585413+0000) 2024-03-06T08:46:37.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:37.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:37.585603+0000) 2024-03-06T08:46:38.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:38.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:38.585817+0000) 2024-03-06T08:46:39.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:39.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:39.586003+0000) 2024-03-06T08:46:40.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:40.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:40.586192+0000) 2024-03-06T08:46:41.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:41.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:41.586413+0000) 2024-03-06T08:46:42.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:42.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:42.586642+0000) 2024-03-06T08:46:43.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:43.584 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:43.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:43.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:43.586867+0000) 2024-03-06T08:46:44.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:44.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:44.587107+0000) 2024-03-06T08:46:45.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:45.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:45.587291+0000) 2024-03-06T08:46:46.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:46.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:46.587515+0000) 2024-03-06T08:46:47.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:47.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:47.587778+0000) 2024-03-06T08:46:48.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:48.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:48.587987+0000) 2024-03-06T08:46:49.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:49.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:49.588188+0000) 2024-03-06T08:46:50.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:50.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:50.588362+0000) 2024-03-06T08:46:51.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:51.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:51.588583+0000) 2024-03-06T08:46:52.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:52.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:52.588832+0000) 2024-03-06T08:46:53.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:53.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:53.589061+0000) 2024-03-06T08:46:54.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:54.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:54.589253+0000) 2024-03-06T08:46:55.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:55.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:55.589478+0000) 2024-03-06T08:46:56.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:56.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:56.589753+0000) 2024-03-06T08:46:57.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:57.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:57.589938+0000) 2024-03-06T08:46:58.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:58.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:58.590128+0000) 2024-03-06T08:46:59.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:46:59.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:59.590357+0000) 2024-03-06T08:47:00.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:00.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:00.590580+0000) 2024-03-06T08:47:01.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:01.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:01.590814+0000) 2024-03-06T08:47:02.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:02.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:02.591044+0000) 2024-03-06T08:47:03.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:03.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:03.591245+0000) 2024-03-06T08:47:04.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:04.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:04.591498+0000) 2024-03-06T08:47:05.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:05.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:05.591744+0000) 2024-03-06T08:47:06.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:06.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:06.591933+0000) 2024-03-06T08:47:07.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:07.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:07.592118+0000) 2024-03-06T08:47:08.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:08.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:08.592361+0000) 2024-03-06T08:47:09.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:09.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:09.592601+0000) 2024-03-06T08:47:10.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:10.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:10.592858+0000) 2024-03-06T08:47:11.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:11.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:11.593099+0000) 2024-03-06T08:47:12.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:12.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:12.593321+0000) 2024-03-06T08:47:13.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:13.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:13.593545+0000) 2024-03-06T08:47:13.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:13.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:14.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:14.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:14.593782+0000) 2024-03-06T08:47:15.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:15.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:15.594001+0000) 2024-03-06T08:47:16.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:16.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:16.594183+0000) 2024-03-06T08:47:17.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:17.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:17.594409+0000) 2024-03-06T08:47:18.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:18.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:18.594644+0000) 2024-03-06T08:47:19.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:19.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:19.594895+0000) 2024-03-06T08:47:20.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:20.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:20.595142+0000) 2024-03-06T08:47:21.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:21.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:21.595327+0000) 2024-03-06T08:47:22.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:22.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:22.595565+0000) 2024-03-06T08:47:23.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:23.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:23.595797+0000) 2024-03-06T08:47:24.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:24.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:24.596008+0000) 2024-03-06T08:47:25.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:25.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:25.596192+0000) 2024-03-06T08:47:26.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:26.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:26.596416+0000) 2024-03-06T08:47:27.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:27.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:27.596641+0000) 2024-03-06T08:47:28.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:28.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:28.596907+0000) 2024-03-06T08:47:29.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:29.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:29.597063+0000) 2024-03-06T08:47:30.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:30.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:30.597285+0000) 2024-03-06T08:47:31.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:31.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:31.597512+0000) 2024-03-06T08:47:32.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:32.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:32.597767+0000) 2024-03-06T08:47:33.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:33.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:33.597958+0000) 2024-03-06T08:47:34.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:34.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:34.598198+0000) 2024-03-06T08:47:35.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:35.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:35.598424+0000) 2024-03-06T08:47:36.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:36.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:36.598719+0000) 2024-03-06T08:47:37.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:37.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:37.598879+0000) 2024-03-06T08:47:38.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:38.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:38.599068+0000) 2024-03-06T08:47:39.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:39.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:39.599252+0000) 2024-03-06T08:47:40.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:40.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:40.599423+0000) 2024-03-06T08:47:41.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:41.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:41.599649+0000) 2024-03-06T08:47:42.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:42.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:42.599859+0000) 2024-03-06T08:47:43.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:43.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:43.600091+0000) 2024-03-06T08:47:43.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:44.000 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:44.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:44.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:44.600371+0000) 2024-03-06T08:47:45.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:45.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:45.600618+0000) 2024-03-06T08:47:46.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:46.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:46.600827+0000) 2024-03-06T08:47:47.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:47.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:47.601069+0000) 2024-03-06T08:47:48.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:48.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:48.601290+0000) 2024-03-06T08:47:49.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:49.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:49.601533+0000) 2024-03-06T08:47:50.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:50.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:50.601812+0000) 2024-03-06T08:47:51.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:51.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:51.602092+0000) 2024-03-06T08:47:52.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:52.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:52.602369+0000) 2024-03-06T08:47:53.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:53.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:53.602657+0000) 2024-03-06T08:47:54.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:54.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:54.602998+0000) 2024-03-06T08:47:55.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:55.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:55.603245+0000) 2024-03-06T08:47:56.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:56.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:56.603527+0000) 2024-03-06T08:47:57.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:57.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:57.603782+0000) 2024-03-06T08:47:58.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:58.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:58.604062+0000) 2024-03-06T08:47:59.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:47:59.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:59.604258+0000) 2024-03-06T08:48:00.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:00.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:00.604446+0000) 2024-03-06T08:48:01.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:01.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:01.604695+0000) 2024-03-06T08:48:02.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:02.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:02.604883+0000) 2024-03-06T08:48:03.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:03.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:03.605073+0000) 2024-03-06T08:48:04.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:04.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:04.605273+0000) 2024-03-06T08:48:05.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:05.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:05.605495+0000) 2024-03-06T08:48:06.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:06.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:06.605752+0000) 2024-03-06T08:48:07.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:07.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:07.605947+0000) 2024-03-06T08:48:08.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:08.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:08.606174+0000) 2024-03-06T08:48:09.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:09.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:09.606370+0000) 2024-03-06T08:48:10.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:10.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:10.606581+0000) 2024-03-06T08:48:11.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:11.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:11.606808+0000) 2024-03-06T08:48:12.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:12.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:12.607028+0000) 2024-03-06T08:48:13.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:13.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:13.607281+0000) 2024-03-06T08:48:14.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:14.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:14.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:14.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:14.607526+0000) 2024-03-06T08:48:15.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:15.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:15.607762+0000) 2024-03-06T08:48:16.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:16.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:16.607964+0000) 2024-03-06T08:48:17.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:17.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:17.608162+0000) 2024-03-06T08:48:18.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:18.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:18.608418+0000) 2024-03-06T08:48:19.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:19.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:19.608576+0000) 2024-03-06T08:48:20.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:20.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:20.608808+0000) 2024-03-06T08:48:21.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:21.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:21.609039+0000) 2024-03-06T08:48:22.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:22.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:22.609267+0000) 2024-03-06T08:48:23.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:23.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:23.609488+0000) 2024-03-06T08:48:24.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:24.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:24.609759+0000) 2024-03-06T08:48:25.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:25.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:25.610073+0000) 2024-03-06T08:48:26.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:26.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:26.610332+0000) 2024-03-06T08:48:27.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:27.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:27.610522+0000) 2024-03-06T08:48:28.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:28.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:28.610796+0000) 2024-03-06T08:48:29.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:29.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:29.610987+0000) 2024-03-06T08:48:30.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:30.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:30.611185+0000) 2024-03-06T08:48:31.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:31.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:31.611380+0000) 2024-03-06T08:48:32.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:32.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:32.611616+0000) 2024-03-06T08:48:33.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:33.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:33.611836+0000) 2024-03-06T08:48:34.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:34.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:34.612074+0000) 2024-03-06T08:48:35.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:35.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:35.612334+0000) 2024-03-06T08:48:36.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:36.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:36.612589+0000) 2024-03-06T08:48:37.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:37.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:37.612812+0000) 2024-03-06T08:48:38.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:38.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:38.613084+0000) 2024-03-06T08:48:39.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:39.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:39.613327+0000) 2024-03-06T08:48:40.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:40.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:40.613586+0000) 2024-03-06T08:48:41.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:41.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:41.613817+0000) 2024-03-06T08:48:42.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:42.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:42.614041+0000) 2024-03-06T08:48:43.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:43.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:43.614242+0000) 2024-03-06T08:48:44.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:44.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:44.614464+0000) 2024-03-06T08:48:44.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:44.625 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:45.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:45.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:45.614714+0000) 2024-03-06T08:48:46.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:46.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:46.614902+0000) 2024-03-06T08:48:47.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:47.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:47.615087+0000) 2024-03-06T08:48:48.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:48.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:48.615280+0000) 2024-03-06T08:48:49.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:49.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:49.615457+0000) 2024-03-06T08:48:50.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:50.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:50.615711+0000) 2024-03-06T08:48:51.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:51.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:51.615937+0000) 2024-03-06T08:48:52.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:52.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:52.616162+0000) 2024-03-06T08:48:53.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:53.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:53.616386+0000) 2024-03-06T08:48:54.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:54.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:54.616643+0000) 2024-03-06T08:48:55.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:55.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:55.616875+0000) 2024-03-06T08:48:56.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:56.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:56.617080+0000) 2024-03-06T08:48:57.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:57.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:57.617338+0000) 2024-03-06T08:48:58.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:58.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:58.617614+0000) 2024-03-06T08:48:59.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:48:59.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:59.617864+0000) 2024-03-06T08:49:00.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:00.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:00.618149+0000) 2024-03-06T08:49:01.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:01.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:01.618397+0000) 2024-03-06T08:49:02.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:02.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:02.618618+0000) 2024-03-06T08:49:03.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:03.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:03.618837+0000) 2024-03-06T08:49:04.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:04.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:04.619076+0000) 2024-03-06T08:49:05.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:05.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:05.619308+0000) 2024-03-06T08:49:06.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:06.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:06.619552+0000) 2024-03-06T08:49:07.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:07.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:07.619795+0000) 2024-03-06T08:49:08.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:08.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:08.620035+0000) 2024-03-06T08:49:09.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:09.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:09.620248+0000) 2024-03-06T08:49:10.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:10.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:10.620477+0000) 2024-03-06T08:49:11.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:11.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:11.620742+0000) 2024-03-06T08:49:12.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:12.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:12.620941+0000) 2024-03-06T08:49:13.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:13.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:13.621172+0000) 2024-03-06T08:49:14.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:14.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:14.621460+0000) 2024-03-06T08:49:14.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:14.774 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:15.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:15.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:15.621775+0000) 2024-03-06T08:49:16.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:16.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:16.622023+0000) 2024-03-06T08:49:17.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:17.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:17.622305+0000) 2024-03-06T08:49:18.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:18.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:18.622586+0000) 2024-03-06T08:49:19.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:19.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:19.622842+0000) 2024-03-06T08:49:20.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:20.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:20.623039+0000) 2024-03-06T08:49:21.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:21.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:21.623286+0000) 2024-03-06T08:49:22.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:22.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:22.623488+0000) 2024-03-06T08:49:23.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:23.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:23.623760+0000) 2024-03-06T08:49:24.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:24.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:24.623998+0000) 2024-03-06T08:49:25.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:25.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:25.624185+0000) 2024-03-06T08:49:26.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:26.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:26.624414+0000) 2024-03-06T08:49:27.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:27.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:27.624685+0000) 2024-03-06T08:49:28.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:28.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:28.624928+0000) 2024-03-06T08:49:29.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:29.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:29.625173+0000) 2024-03-06T08:49:30.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:30.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:30.625447+0000) 2024-03-06T08:49:31.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:31.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:31.625639+0000) 2024-03-06T08:49:32.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:32.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:32.625894+0000) 2024-03-06T08:49:33.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:33.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:33.626141+0000) 2024-03-06T08:49:34.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:34.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:34.626432+0000) 2024-03-06T08:49:35.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:35.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:35.626757+0000) 2024-03-06T08:49:36.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:36.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:36.626998+0000) 2024-03-06T08:49:37.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:37.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:37.627182+0000) 2024-03-06T08:49:38.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:38.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:38.627396+0000) 2024-03-06T08:49:39.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:39.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:39.627646+0000) 2024-03-06T08:49:40.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:40.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:40.627893+0000) 2024-03-06T08:49:41.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:41.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:41.628115+0000) 2024-03-06T08:49:42.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:42.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:42.628307+0000) 2024-03-06T08:49:43.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:43.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:43.628497+0000) 2024-03-06T08:49:44.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:44.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:44.628733+0000) 2024-03-06T08:49:44.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:44.968 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:45.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:45.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:45.628933+0000) 2024-03-06T08:49:46.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:46.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:46.629170+0000) 2024-03-06T08:49:47.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:47.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:47.629405+0000) 2024-03-06T08:49:48.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:48.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:48.629640+0000) 2024-03-06T08:49:49.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:49.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:49.629934+0000) 2024-03-06T08:49:50.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:50.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:50.630156+0000) 2024-03-06T08:49:51.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:51.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:51.630383+0000) 2024-03-06T08:49:52.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:52.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:52.630573+0000) 2024-03-06T08:49:53.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:53.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:53.630804+0000) 2024-03-06T08:49:54.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:54.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:54.631070+0000) 2024-03-06T08:49:55.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:55.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:55.631328+0000) 2024-03-06T08:49:56.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:56.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:56.631569+0000) 2024-03-06T08:49:57.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:57.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:57.631798+0000) 2024-03-06T08:49:58.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:58.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:58.631969+0000) 2024-03-06T08:49:59.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:49:59.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:59.632132+0000) 2024-03-06T08:50:00.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:00.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:00.632379+0000) 2024-03-06T08:50:01.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:01.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:01.632621+0000) 2024-03-06T08:50:02.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:02.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:02.632858+0000) 2024-03-06T08:50:03.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:03.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:03.633102+0000) 2024-03-06T08:50:04.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:04.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:04.633354+0000) 2024-03-06T08:50:05.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:05.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:05.633591+0000) 2024-03-06T08:50:06.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:06.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:06.633843+0000) 2024-03-06T08:50:07.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:07.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:07.634049+0000) 2024-03-06T08:50:08.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:08.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:08.634288+0000) 2024-03-06T08:50:09.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:09.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:09.634510+0000) 2024-03-06T08:50:10.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:10.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:10.634772+0000) 2024-03-06T08:50:11.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:11.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:11.634965+0000) 2024-03-06T08:50:12.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:12.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:12.635158+0000) 2024-03-06T08:50:13.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:13.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:13.635351+0000) 2024-03-06T08:50:14.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:14.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:14.635597+0000) 2024-03-06T08:50:15.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:15.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:15.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:15.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:15.635793+0000) 2024-03-06T08:50:16.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:16.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:16.635998+0000) 2024-03-06T08:50:17.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:17.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:17.636234+0000) 2024-03-06T08:50:18.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:18.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:18.636471+0000) 2024-03-06T08:50:19.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:19.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:19.636723+0000) 2024-03-06T08:50:20.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:20.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:20.636946+0000) 2024-03-06T08:50:21.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:21.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:21.637128+0000) 2024-03-06T08:50:22.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:22.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:22.637315+0000) 2024-03-06T08:50:23.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:23.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:23.637544+0000) 2024-03-06T08:50:24.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:24.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:24.637792+0000) 2024-03-06T08:50:25.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:25.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:25.637979+0000) 2024-03-06T08:50:26.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:26.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:26.638214+0000) 2024-03-06T08:50:27.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:27.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:27.638443+0000) 2024-03-06T08:50:28.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:28.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:28.638658+0000) 2024-03-06T08:50:29.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:29.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:29.638899+0000) 2024-03-06T08:50:30.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:30.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:30.639094+0000) 2024-03-06T08:50:31.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:31.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:31.639312+0000) 2024-03-06T08:50:32.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:32.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:32.639501+0000) 2024-03-06T08:50:33.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:33.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:33.639714+0000) 2024-03-06T08:50:34.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:34.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:34.639952+0000) 2024-03-06T08:50:35.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:35.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:35.640140+0000) 2024-03-06T08:50:36.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:36.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:36.640356+0000) 2024-03-06T08:50:37.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:37.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:37.640575+0000) 2024-03-06T08:50:38.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:38.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:38.640804+0000) 2024-03-06T08:50:39.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:39.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:39.640977+0000) 2024-03-06T08:50:40.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:40.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:40.641157+0000) 2024-03-06T08:50:41.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:41.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:41.641393+0000) 2024-03-06T08:50:42.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:42.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:42.641634+0000) 2024-03-06T08:50:43.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:43.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:43.641908+0000) 2024-03-06T08:50:44.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:44.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:44.642123+0000) 2024-03-06T08:50:45.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:45.543 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:45.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:45.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:45.642306+0000) 2024-03-06T08:50:46.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:46.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:46.642530+0000) 2024-03-06T08:50:47.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:47.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:47.642805+0000) 2024-03-06T08:50:48.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:48.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:48.643091+0000) 2024-03-06T08:50:49.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:49.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:49.643333+0000) 2024-03-06T08:50:50.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:50.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:50.643558+0000) 2024-03-06T08:50:51.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:51.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:51.643779+0000) 2024-03-06T08:50:52.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:52.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:52.644022+0000) 2024-03-06T08:50:53.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:53.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:53.644263+0000) 2024-03-06T08:50:54.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:54.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:54.644520+0000) 2024-03-06T08:50:55.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:55.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:55.644754+0000) 2024-03-06T08:50:56.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:56.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:56.644948+0000) 2024-03-06T08:50:57.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:57.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:57.645143+0000) 2024-03-06T08:50:58.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:58.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:58.645352+0000) 2024-03-06T08:50:59.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:50:59.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:59.645582+0000) 2024-03-06T08:51:00.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:00.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:00.645821+0000) 2024-03-06T08:51:01.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:01.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:01.646041+0000) 2024-03-06T08:51:02.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:02.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:02.646263+0000) 2024-03-06T08:51:03.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:03.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:03.646434+0000) 2024-03-06T08:51:04.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:04.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:04.646656+0000) 2024-03-06T08:51:05.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:05.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:05.646923+0000) 2024-03-06T08:51:06.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:06.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:06.647150+0000) 2024-03-06T08:51:07.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:07.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:07.647374+0000) 2024-03-06T08:51:08.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:08.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:08.647613+0000) 2024-03-06T08:51:09.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:09.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:09.647848+0000) 2024-03-06T08:51:10.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:10.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:10.648025+0000) 2024-03-06T08:51:11.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:11.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:11.648260+0000) 2024-03-06T08:51:12.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:12.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:12.648454+0000) 2024-03-06T08:51:13.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:13.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:13.648636+0000) 2024-03-06T08:51:14.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:14.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:14.648924+0000) 2024-03-06T08:51:15.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:15.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:15.649155+0000) 2024-03-06T08:51:15.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:15.774 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:16.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:16.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:16.649344+0000) 2024-03-06T08:51:17.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:17.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:17.649576+0000) 2024-03-06T08:51:18.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:18.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:18.649810+0000) 2024-03-06T08:51:19.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:19.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:19.649987+0000) 2024-03-06T08:51:20.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:20.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:20.650202+0000) 2024-03-06T08:51:21.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:21.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:21.650440+0000) 2024-03-06T08:51:22.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:22.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:22.650700+0000) 2024-03-06T08:51:23.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:23.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:23.650906+0000) 2024-03-06T08:51:24.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:24.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:24.651150+0000) 2024-03-06T08:51:25.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:25.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:25.651387+0000) 2024-03-06T08:51:26.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:26.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:26.651625+0000) 2024-03-06T08:51:27.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:27.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:27.651859+0000) 2024-03-06T08:51:28.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:28.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:28.652084+0000) 2024-03-06T08:51:29.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:29.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:29.652322+0000) 2024-03-06T08:51:30.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:30.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:30.652561+0000) 2024-03-06T08:51:31.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:31.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:31.652806+0000) 2024-03-06T08:51:32.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:32.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:32.653052+0000) 2024-03-06T08:51:33.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:33.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:33.653285+0000) 2024-03-06T08:51:34.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:34.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:34.653545+0000) 2024-03-06T08:51:35.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:35.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:35.653781+0000) 2024-03-06T08:51:36.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:36.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:36.654002+0000) 2024-03-06T08:51:37.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:37.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:37.654204+0000) 2024-03-06T08:51:38.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:38.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:38.654424+0000) 2024-03-06T08:51:39.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:39.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:39.654643+0000) 2024-03-06T08:51:40.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:40.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:40.654917+0000) 2024-03-06T08:51:41.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:41.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:41.655156+0000) 2024-03-06T08:51:42.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:42.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:42.655397+0000) 2024-03-06T08:51:43.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:43.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:43.655608+0000) 2024-03-06T08:51:44.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:44.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:44.655853+0000) 2024-03-06T08:51:45.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:45.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:45.656075+0000) 2024-03-06T08:51:45.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:45.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:46.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:46.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:46.656283+0000) 2024-03-06T08:51:47.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:47.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:47.656591+0000) 2024-03-06T08:51:48.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:48.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:48.656818+0000) 2024-03-06T08:51:49.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:49.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:49.657091+0000) 2024-03-06T08:51:50.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:50.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:50.657277+0000) 2024-03-06T08:51:51.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:51.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:51.657464+0000) 2024-03-06T08:51:52.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:52.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:52.657738+0000) 2024-03-06T08:51:53.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:53.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:53.657921+0000) 2024-03-06T08:51:54.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:54.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:54.658128+0000) 2024-03-06T08:51:55.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:55.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:55.658357+0000) 2024-03-06T08:51:56.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:56.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:56.658583+0000) 2024-03-06T08:51:57.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:57.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:57.658767+0000) 2024-03-06T08:51:58.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:58.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:58.659024+0000) 2024-03-06T08:51:59.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:51:59.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:59.659246+0000) 2024-03-06T08:52:00.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:00.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:00.659462+0000) 2024-03-06T08:52:01.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:01.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:01.659768+0000) 2024-03-06T08:52:02.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:02.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:02.660056+0000) 2024-03-06T08:52:03.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:03.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:03.660340+0000) 2024-03-06T08:52:04.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:04.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:04.660608+0000) 2024-03-06T08:52:05.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:05.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:05.660838+0000) 2024-03-06T08:52:06.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:06.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:06.661020+0000) 2024-03-06T08:52:07.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:07.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:07.661239+0000) 2024-03-06T08:52:08.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:08.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:08.661448+0000) 2024-03-06T08:52:09.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:09.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:09.661625+0000) 2024-03-06T08:52:10.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:10.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:10.661824+0000) 2024-03-06T08:52:11.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:11.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:11.662003+0000) 2024-03-06T08:52:12.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:12.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:12.662162+0000) 2024-03-06T08:52:13.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:13.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:13.662407+0000) 2024-03-06T08:52:14.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:14.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:14.662646+0000) 2024-03-06T08:52:15.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:15.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:15.662897+0000) 2024-03-06T08:52:16.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:16.275 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:16.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:16.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:16.663103+0000) 2024-03-06T08:52:17.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:17.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:17.663330+0000) 2024-03-06T08:52:18.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:18.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:18.663550+0000) 2024-03-06T08:52:19.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:19.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:19.663796+0000) 2024-03-06T08:52:20.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:20.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:20.663944+0000) 2024-03-06T08:52:21.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:21.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:21.664212+0000) 2024-03-06T08:52:22.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:22.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:22.664458+0000) 2024-03-06T08:52:23.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:23.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:23.664790+0000) 2024-03-06T08:52:24.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:24.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:24.665030+0000) 2024-03-06T08:52:25.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:25.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:25.665214+0000) 2024-03-06T08:52:26.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:26.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:26.665448+0000) 2024-03-06T08:52:27.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:27.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:27.665699+0000) 2024-03-06T08:52:28.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:28.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:28.665909+0000) 2024-03-06T08:52:29.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:29.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:29.666144+0000) 2024-03-06T08:52:30.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:30.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:30.666331+0000) 2024-03-06T08:52:31.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:31.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:31.666555+0000) 2024-03-06T08:52:32.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:32.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:32.666807+0000) 2024-03-06T08:52:33.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:33.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:33.667033+0000) 2024-03-06T08:52:34.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:34.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:34.667263+0000) 2024-03-06T08:52:35.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:35.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:35.667489+0000) 2024-03-06T08:52:36.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:36.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:36.667759+0000) 2024-03-06T08:52:37.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:37.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:37.667949+0000) 2024-03-06T08:52:38.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:38.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:38.668173+0000) 2024-03-06T08:52:39.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:39.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:39.668391+0000) 2024-03-06T08:52:40.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:40.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:40.668633+0000) 2024-03-06T08:52:41.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:41.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:41.668868+0000) 2024-03-06T08:52:42.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:42.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:42.669095+0000) 2024-03-06T08:52:43.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:43.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:43.669272+0000) 2024-03-06T08:52:44.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:44.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:44.669502+0000) 2024-03-06T08:52:45.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:45.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:45.669766+0000) 2024-03-06T08:52:46.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:46.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:46.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:46.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:46.670010+0000) 2024-03-06T08:52:47.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:47.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:47.670229+0000) 2024-03-06T08:52:48.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:48.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:48.670461+0000) 2024-03-06T08:52:49.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:49.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:49.670708+0000) 2024-03-06T08:52:50.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:50.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:50.670932+0000) 2024-03-06T08:52:51.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:51.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:51.671122+0000) 2024-03-06T08:52:52.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:52.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:52.671316+0000) 2024-03-06T08:52:53.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:53.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:53.671548+0000) 2024-03-06T08:52:54.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:54.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:54.671788+0000) 2024-03-06T08:52:55.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:55.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:55.671973+0000) 2024-03-06T08:52:56.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:56.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:56.672202+0000) 2024-03-06T08:52:57.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:57.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:57.672415+0000) 2024-03-06T08:52:58.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:58.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:58.672643+0000) 2024-03-06T08:52:59.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:52:59.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:59.672935+0000) 2024-03-06T08:53:00.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:00.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:00.673155+0000) 2024-03-06T08:53:01.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:01.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:01.673328+0000) 2024-03-06T08:53:02.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:02.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:02.673541+0000) 2024-03-06T08:53:03.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:03.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:03.673769+0000) 2024-03-06T08:53:04.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:04.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:04.674014+0000) 2024-03-06T08:53:05.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:05.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:05.674231+0000) 2024-03-06T08:53:06.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:06.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:06.674496+0000) 2024-03-06T08:53:07.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:07.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:07.674724+0000) 2024-03-06T08:53:08.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:08.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:08.674945+0000) 2024-03-06T08:53:09.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:09.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:09.675198+0000) 2024-03-06T08:53:10.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:10.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:10.675422+0000) 2024-03-06T08:53:11.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:11.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:11.675642+0000) 2024-03-06T08:53:12.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:12.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:12.675912+0000) 2024-03-06T08:53:13.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:13.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:13.676106+0000) 2024-03-06T08:53:14.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:14.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:14.676352+0000) 2024-03-06T08:53:15.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:15.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:15.676573+0000) 2024-03-06T08:53:16.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:16.576 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:16.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:16.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:16.676782+0000) 2024-03-06T08:53:17.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:17.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:17.677054+0000) 2024-03-06T08:53:18.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:18.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:18.677287+0000) 2024-03-06T08:53:19.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:19.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:19.677505+0000) 2024-03-06T08:53:20.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:20.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:20.677715+0000) 2024-03-06T08:53:21.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:21.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:21.677953+0000) 2024-03-06T08:53:22.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:22.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:22.678180+0000) 2024-03-06T08:53:23.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:23.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:23.678410+0000) 2024-03-06T08:53:24.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:24.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:24.678594+0000) 2024-03-06T08:53:25.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:25.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:25.678757+0000) 2024-03-06T08:53:26.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:26.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:26.678982+0000) 2024-03-06T08:53:27.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:27.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:27.679211+0000) 2024-03-06T08:53:28.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:28.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:28.679480+0000) 2024-03-06T08:53:29.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:29.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:29.679779+0000) 2024-03-06T08:53:30.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:30.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:30.680055+0000) 2024-03-06T08:53:31.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:31.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:31.680312+0000) 2024-03-06T08:53:32.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:32.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:32.680517+0000) 2024-03-06T08:53:33.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:33.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:33.680752+0000) 2024-03-06T08:53:34.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:34.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:34.681005+0000) 2024-03-06T08:53:35.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:35.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:35.681241+0000) 2024-03-06T08:53:36.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:36.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:36.681483+0000) 2024-03-06T08:53:37.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:37.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:37.681760+0000) 2024-03-06T08:53:38.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:38.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:38.682001+0000) 2024-03-06T08:53:39.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:39.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:39.682243+0000) 2024-03-06T08:53:40.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:40.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:40.682480+0000) 2024-03-06T08:53:41.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:41.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:41.682770+0000) 2024-03-06T08:53:42.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:42.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:42.683006+0000) 2024-03-06T08:53:43.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:43.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:43.683248+0000) 2024-03-06T08:53:44.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:44.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:44.683506+0000) 2024-03-06T08:53:45.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:45.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:45.683773+0000) 2024-03-06T08:53:46.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:46.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:46.684012+0000) 2024-03-06T08:53:46.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:46.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:47.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:47.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:47.684247+0000) 2024-03-06T08:53:48.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:48.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:48.684467+0000) 2024-03-06T08:53:49.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:49.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:49.684726+0000) 2024-03-06T08:53:50.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:50.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:50.684966+0000) 2024-03-06T08:53:51.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:51.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:51.685204+0000) 2024-03-06T08:53:52.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:52.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:52.685445+0000) 2024-03-06T08:53:53.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:53.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:53.685703+0000) 2024-03-06T08:53:54.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:54.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:54.685945+0000) 2024-03-06T08:53:55.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:55.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:55.686181+0000) 2024-03-06T08:53:56.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:56.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:56.686394+0000) 2024-03-06T08:53:57.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:57.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:57.686602+0000) 2024-03-06T08:53:58.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:58.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:58.686799+0000) 2024-03-06T08:53:59.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:53:59.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:59.686993+0000) 2024-03-06T08:54:00.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:00.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:00.687223+0000) 2024-03-06T08:54:01.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:01.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:01.687445+0000) 2024-03-06T08:54:02.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:02.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:02.687699+0000) 2024-03-06T08:54:03.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:03.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:03.687887+0000) 2024-03-06T08:54:04.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:04.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:04.688106+0000) 2024-03-06T08:54:05.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:05.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:05.688295+0000) 2024-03-06T08:54:06.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:06.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:06.688518+0000) 2024-03-06T08:54:07.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:07.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:07.688726+0000) 2024-03-06T08:54:08.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:08.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:08.688946+0000) 2024-03-06T08:54:09.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:09.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:09.689136+0000) 2024-03-06T08:54:10.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:10.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:10.689322+0000) 2024-03-06T08:54:11.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:11.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:11.689548+0000) 2024-03-06T08:54:12.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:12.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:12.689781+0000) 2024-03-06T08:54:13.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:13.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:13.689994+0000) 2024-03-06T08:54:14.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:14.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:14.690205+0000) 2024-03-06T08:54:15.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:15.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:15.690390+0000) 2024-03-06T08:54:16.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:16.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:16.690705+0000) 2024-03-06T08:54:16.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:16.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:17.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:17.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:17.690932+0000) 2024-03-06T08:54:18.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:18.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:18.691160+0000) 2024-03-06T08:54:19.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:19.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:19.691386+0000) 2024-03-06T08:54:20.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:20.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:20.691561+0000) 2024-03-06T08:54:21.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:21.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:21.691796+0000) 2024-03-06T08:54:22.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:22.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:22.692036+0000) 2024-03-06T08:54:23.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:23.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:23.692230+0000) 2024-03-06T08:54:24.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:24.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:24.692493+0000) 2024-03-06T08:54:25.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:25.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:25.692735+0000) 2024-03-06T08:54:26.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:26.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:26.692962+0000) 2024-03-06T08:54:27.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:27.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:27.693135+0000) 2024-03-06T08:54:28.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:28.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:28.693373+0000) 2024-03-06T08:54:29.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:29.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:29.693566+0000) 2024-03-06T08:54:30.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:30.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:30.693800+0000) 2024-03-06T08:54:31.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:31.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:31.694002+0000) 2024-03-06T08:54:32.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:32.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:32.694245+0000) 2024-03-06T08:54:33.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:33.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:33.694482+0000) 2024-03-06T08:54:34.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:34.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:34.694770+0000) 2024-03-06T08:54:35.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:35.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:35.694981+0000) 2024-03-06T08:54:36.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:36.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:36.695220+0000) 2024-03-06T08:54:37.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:37.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:37.695414+0000) 2024-03-06T08:54:38.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:38.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:38.695645+0000) 2024-03-06T08:54:39.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:39.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:39.695933+0000) 2024-03-06T08:54:40.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:40.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:40.696140+0000) 2024-03-06T08:54:41.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:41.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:41.696378+0000) 2024-03-06T08:54:42.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:42.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:42.696617+0000) 2024-03-06T08:54:43.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:43.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:43.696862+0000) 2024-03-06T08:54:44.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:44.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:44.697116+0000) 2024-03-06T08:54:45.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:45.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:45.697356+0000) 2024-03-06T08:54:46.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:46.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:46.697614+0000) 2024-03-06T08:54:47.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:47.214 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:47.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:47.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:47.697859+0000) 2024-03-06T08:54:48.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:48.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:48.698042+0000) 2024-03-06T08:54:49.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:49.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:49.698234+0000) 2024-03-06T08:54:50.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:50.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:50.698471+0000) 2024-03-06T08:54:51.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:51.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:51.698725+0000) 2024-03-06T08:54:52.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:52.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:52.698921+0000) 2024-03-06T08:54:53.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:53.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:53.699160+0000) 2024-03-06T08:54:54.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:54.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:54.699413+0000) 2024-03-06T08:54:55.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:55.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:55.699648+0000) 2024-03-06T08:54:56.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:56.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:56.699956+0000) 2024-03-06T08:54:57.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:57.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:57.700165+0000) 2024-03-06T08:54:58.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:58.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:58.700398+0000) 2024-03-06T08:54:59.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:54:59.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:59.700624+0000) 2024-03-06T08:55:00.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:00.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:00.700846+0000) 2024-03-06T08:55:01.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:01.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:01.701072+0000) 2024-03-06T08:55:02.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:02.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:02.701259+0000) 2024-03-06T08:55:03.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:03.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:03.701501+0000) 2024-03-06T08:55:04.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:04.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:04.701796+0000) 2024-03-06T08:55:05.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:05.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:05.702016+0000) 2024-03-06T08:55:06.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:06.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:06.702242+0000) 2024-03-06T08:55:07.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:07.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:07.702469+0000) 2024-03-06T08:55:08.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:08.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:08.702733+0000) 2024-03-06T08:55:09.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:09.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:09.702962+0000) 2024-03-06T08:55:10.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:10.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:10.703150+0000) 2024-03-06T08:55:11.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:11.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:11.703392+0000) 2024-03-06T08:55:12.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:12.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:12.703581+0000) 2024-03-06T08:55:13.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:13.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:13.703832+0000) 2024-03-06T08:55:14.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:14.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:14.704070+0000) 2024-03-06T08:55:15.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:15.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:15.704290+0000) 2024-03-06T08:55:16.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:16.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:16.704465+0000) 2024-03-06T08:55:17.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:17.367 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:17.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:17.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:17.704730+0000) 2024-03-06T08:55:18.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:18.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:18.704926+0000) 2024-03-06T08:55:19.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:19.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:19.705148+0000) 2024-03-06T08:55:20.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:20.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:20.705366+0000) 2024-03-06T08:55:21.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:21.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:21.705586+0000) 2024-03-06T08:55:22.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:22.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:22.705768+0000) 2024-03-06T08:55:23.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:23.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:23.706005+0000) 2024-03-06T08:55:24.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:24.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:24.706246+0000) 2024-03-06T08:55:25.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:25.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:25.706437+0000) 2024-03-06T08:55:26.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:26.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:26.706626+0000) 2024-03-06T08:55:27.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:27.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:27.706859+0000) 2024-03-06T08:55:28.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:28.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:28.707133+0000) 2024-03-06T08:55:29.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:29.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:29.707401+0000) 2024-03-06T08:55:30.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:30.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:30.707658+0000) 2024-03-06T08:55:31.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:31.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:31.707970+0000) 2024-03-06T08:55:32.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:32.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:32.708229+0000) 2024-03-06T08:55:33.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:33.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:33.708503+0000) 2024-03-06T08:55:34.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:34.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:34.708703+0000) 2024-03-06T08:55:35.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:35.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:35.708929+0000) 2024-03-06T08:55:36.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:36.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:36.709157+0000) 2024-03-06T08:55:37.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:37.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:37.709374+0000) 2024-03-06T08:55:38.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:38.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:38.709592+0000) 2024-03-06T08:55:39.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:39.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:39.709805+0000) 2024-03-06T08:55:40.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:40.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:40.710023+0000) 2024-03-06T08:55:41.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:41.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:41.710251+0000) 2024-03-06T08:55:42.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:42.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:42.710466+0000) 2024-03-06T08:55:43.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:43.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:43.710692+0000) 2024-03-06T08:55:44.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:44.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:44.710908+0000) 2024-03-06T08:55:45.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:45.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:45.711133+0000) 2024-03-06T08:55:46.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:46.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:46.711335+0000) 2024-03-06T08:55:47.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:47.569 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:47.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:47.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:47.711579+0000) 2024-03-06T08:55:48.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:48.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:48.711809+0000) 2024-03-06T08:55:49.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:49.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:49.711997+0000) 2024-03-06T08:55:50.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:50.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:50.712231+0000) 2024-03-06T08:55:51.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:51.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:51.712433+0000) 2024-03-06T08:55:52.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:52.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:52.712618+0000) 2024-03-06T08:55:53.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:53.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:53.712839+0000) 2024-03-06T08:55:54.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:54.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:54.713105+0000) 2024-03-06T08:55:55.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:55.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:55.713295+0000) 2024-03-06T08:55:56.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:56.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:56.713490+0000) 2024-03-06T08:55:57.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:57.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:57.713766+0000) 2024-03-06T08:55:58.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:58.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:58.714051+0000) 2024-03-06T08:55:59.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:55:59.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:59.714293+0000) 2024-03-06T08:56:00.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:00.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:00.714575+0000) 2024-03-06T08:56:01.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:01.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:01.714819+0000) 2024-03-06T08:56:02.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:02.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:02.714979+0000) 2024-03-06T08:56:03.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:03.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:03.715251+0000) 2024-03-06T08:56:04.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:04.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:04.715515+0000) 2024-03-06T08:56:05.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:05.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:05.715800+0000) 2024-03-06T08:56:06.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:06.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:06.716037+0000) 2024-03-06T08:56:07.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:07.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:07.716272+0000) 2024-03-06T08:56:08.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:08.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:08.716460+0000) 2024-03-06T08:56:09.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:09.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:09.716702+0000) 2024-03-06T08:56:10.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:10.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:10.716918+0000) 2024-03-06T08:56:11.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:11.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:11.717133+0000) 2024-03-06T08:56:12.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:12.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:12.717335+0000) 2024-03-06T08:56:13.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:13.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:13.717556+0000) 2024-03-06T08:56:14.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:14.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:14.717809+0000) 2024-03-06T08:56:15.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:15.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:15.718033+0000) 2024-03-06T08:56:16.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:16.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:16.718258+0000) 2024-03-06T08:56:17.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:17.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:17.718445+0000) 2024-03-06T08:56:17.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:17.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:18.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:18.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:18.718700+0000) 2024-03-06T08:56:19.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:19.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:19.718885+0000) 2024-03-06T08:56:20.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:20.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:20.719099+0000) 2024-03-06T08:56:21.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:21.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:21.719323+0000) 2024-03-06T08:56:22.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:22.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:22.719476+0000) 2024-03-06T08:56:23.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:23.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:23.719736+0000) 2024-03-06T08:56:24.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:24.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:24.719930+0000) 2024-03-06T08:56:25.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:25.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:25.720120+0000) 2024-03-06T08:56:26.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:26.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:26.720326+0000) 2024-03-06T08:56:27.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:27.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:27.720507+0000) 2024-03-06T08:56:28.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:28.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:28.720742+0000) 2024-03-06T08:56:29.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:29.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:29.720951+0000) 2024-03-06T08:56:30.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:30.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:30.721144+0000) 2024-03-06T08:56:31.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:31.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:31.721380+0000) 2024-03-06T08:56:32.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:32.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:32.721615+0000) 2024-03-06T08:56:33.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:33.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:33.721838+0000) 2024-03-06T08:56:34.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:34.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:34.722099+0000) 2024-03-06T08:56:35.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:35.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:35.722332+0000) 2024-03-06T08:56:36.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:36.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:36.722557+0000) 2024-03-06T08:56:37.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:37.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:37.722784+0000) 2024-03-06T08:56:38.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:38.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:38.723008+0000) 2024-03-06T08:56:39.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:39.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:39.723192+0000) 2024-03-06T08:56:40.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:40.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:40.723412+0000) 2024-03-06T08:56:41.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:41.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:41.723644+0000) 2024-03-06T08:56:42.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:42.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:42.723926+0000) 2024-03-06T08:56:43.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:43.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:43.724147+0000) 2024-03-06T08:56:44.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:44.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:44.724393+0000) 2024-03-06T08:56:45.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:45.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:45.724646+0000) 2024-03-06T08:56:46.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:46.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:46.724897+0000) 2024-03-06T08:56:47.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:47.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:47.725064+0000) 2024-03-06T08:56:48.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:48.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:48.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:48.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:48.725234+0000) 2024-03-06T08:56:49.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:49.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:49.725457+0000) 2024-03-06T08:56:50.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:50.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:50.725699+0000) 2024-03-06T08:56:51.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:51.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:51.725927+0000) 2024-03-06T08:56:52.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:52.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:52.726144+0000) 2024-03-06T08:56:53.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:53.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:53.726365+0000) 2024-03-06T08:56:54.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:54.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:54.726608+0000) 2024-03-06T08:56:55.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:55.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:55.726827+0000) 2024-03-06T08:56:56.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:56.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:56.727054+0000) 2024-03-06T08:56:57.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:57.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:57.727275+0000) 2024-03-06T08:56:58.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:58.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:58.727500+0000) 2024-03-06T08:56:59.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:56:59.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:59.727748+0000) 2024-03-06T08:57:00.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:00.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:00.727936+0000) 2024-03-06T08:57:01.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:01.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:01.728119+0000) 2024-03-06T08:57:02.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:02.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:02.728304+0000) 2024-03-06T08:57:03.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:03.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:03.728543+0000) 2024-03-06T08:57:04.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:04.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:04.728787+0000) 2024-03-06T08:57:05.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:05.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:05.728951+0000) 2024-03-06T08:57:06.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:06.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:06.729145+0000) 2024-03-06T08:57:07.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:07.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:07.729380+0000) 2024-03-06T08:57:08.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:08.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:08.729612+0000) 2024-03-06T08:57:09.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:09.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:09.729853+0000) 2024-03-06T08:57:10.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:10.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:10.730021+0000) 2024-03-06T08:57:11.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:11.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:11.730239+0000) 2024-03-06T08:57:12.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:12.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:12.730464+0000) 2024-03-06T08:57:13.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:13.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:13.730687+0000) 2024-03-06T08:57:14.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:14.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:14.730959+0000) 2024-03-06T08:57:15.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:15.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:15.731198+0000) 2024-03-06T08:57:16.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:16.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:16.731426+0000) 2024-03-06T08:57:17.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:17.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:17.731711+0000) 2024-03-06T08:57:18.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:18.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:18.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:18.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:18.731908+0000) 2024-03-06T08:57:19.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:19.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:19.732085+0000) 2024-03-06T08:57:20.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:20.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:20.732282+0000) 2024-03-06T08:57:21.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:21.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:21.732506+0000) 2024-03-06T08:57:22.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:22.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:22.732742+0000) 2024-03-06T08:57:23.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:23.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:23.732972+0000) 2024-03-06T08:57:24.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:24.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:24.733180+0000) 2024-03-06T08:57:25.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:25.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:25.733407+0000) 2024-03-06T08:57:26.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:26.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:26.733640+0000) 2024-03-06T08:57:27.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:27.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:27.733891+0000) 2024-03-06T08:57:28.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:28.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:28.734131+0000) 2024-03-06T08:57:29.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:29.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:29.734323+0000) 2024-03-06T08:57:30.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:30.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:30.734525+0000) 2024-03-06T08:57:31.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:31.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:31.734770+0000) 2024-03-06T08:57:32.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:32.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:32.734996+0000) 2024-03-06T08:57:33.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:33.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:33.735180+0000) 2024-03-06T08:57:34.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:34.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:34.735435+0000) 2024-03-06T08:57:35.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:35.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:35.735657+0000) 2024-03-06T08:57:36.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:36.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:36.735920+0000) 2024-03-06T08:57:37.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:37.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:37.736092+0000) 2024-03-06T08:57:38.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:38.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:38.736323+0000) 2024-03-06T08:57:39.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:39.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:39.736557+0000) 2024-03-06T08:57:40.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:40.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:40.736785+0000) 2024-03-06T08:57:41.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:41.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:41.737009+0000) 2024-03-06T08:57:42.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:42.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:42.737230+0000) 2024-03-06T08:57:43.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:43.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:43.737458+0000) 2024-03-06T08:57:44.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:44.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:44.737765+0000) 2024-03-06T08:57:45.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:45.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:45.737952+0000) 2024-03-06T08:57:46.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:46.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:46.738173+0000) 2024-03-06T08:57:47.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:47.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:47.738396+0000) 2024-03-06T08:57:48.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:48.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:48.738610+0000) 2024-03-06T08:57:48.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:48.822 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:49.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:49.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:49.738838+0000) 2024-03-06T08:57:50.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:50.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:50.739008+0000) 2024-03-06T08:57:51.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:51.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:51.739227+0000) 2024-03-06T08:57:52.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:52.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:52.739449+0000) 2024-03-06T08:57:53.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:53.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:53.739639+0000) 2024-03-06T08:57:54.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:54.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:54.739906+0000) 2024-03-06T08:57:55.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:55.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:55.740136+0000) 2024-03-06T08:57:56.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:56.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:56.740357+0000) 2024-03-06T08:57:57.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:57.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:57.740575+0000) 2024-03-06T08:57:58.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:58.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:58.740796+0000) 2024-03-06T08:57:59.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:57:59.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:59.741029+0000) 2024-03-06T08:58:00.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:00.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:00.741268+0000) 2024-03-06T08:58:01.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:01.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:01.741460+0000) 2024-03-06T08:58:02.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:02.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:02.741609+0000) 2024-03-06T08:58:03.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:03.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:03.741867+0000) 2024-03-06T08:58:04.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:04.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:04.742125+0000) 2024-03-06T08:58:05.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:05.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:05.742354+0000) 2024-03-06T08:58:06.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:06.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:06.742571+0000) 2024-03-06T08:58:07.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:07.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:07.742803+0000) 2024-03-06T08:58:08.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:08.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:08.743001+0000) 2024-03-06T08:58:09.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:09.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:09.743201+0000) 2024-03-06T08:58:10.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:10.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:10.743495+0000) 2024-03-06T08:58:11.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:11.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:11.743725+0000) 2024-03-06T08:58:12.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:12.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:12.743947+0000) 2024-03-06T08:58:13.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:13.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:13.744168+0000) 2024-03-06T08:58:14.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:14.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:14.744374+0000) 2024-03-06T08:58:15.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:15.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:15.744588+0000) 2024-03-06T08:58:16.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:16.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:16.744816+0000) 2024-03-06T08:58:17.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:17.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:17.745054+0000) 2024-03-06T08:58:18.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:18.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:18.745292+0000) 2024-03-06T08:58:19.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:19.178 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:19.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:19.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:19.745492+0000) 2024-03-06T08:58:20.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:20.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:20.745743+0000) 2024-03-06T08:58:21.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:21.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:21.745968+0000) 2024-03-06T08:58:22.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:22.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:22.746190+0000) 2024-03-06T08:58:23.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:23.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:23.746386+0000) 2024-03-06T08:58:24.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:24.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:24.746632+0000) 2024-03-06T08:58:25.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:25.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:25.746824+0000) 2024-03-06T08:58:26.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:26.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:26.747051+0000) 2024-03-06T08:58:27.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:27.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:27.747227+0000) 2024-03-06T08:58:28.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:28.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:28.747463+0000) 2024-03-06T08:58:29.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:29.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:29.747713+0000) 2024-03-06T08:58:30.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:30.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:30.747952+0000) 2024-03-06T08:58:31.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:31.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:31.748172+0000) 2024-03-06T08:58:32.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:32.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:32.748443+0000) 2024-03-06T08:58:33.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:33.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:33.748706+0000) 2024-03-06T08:58:34.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:34.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:34.748973+0000) 2024-03-06T08:58:35.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:35.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:35.749221+0000) 2024-03-06T08:58:36.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:36.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:36.749417+0000) 2024-03-06T08:58:37.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:37.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:37.749650+0000) 2024-03-06T08:58:38.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:38.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:38.749873+0000) 2024-03-06T08:58:39.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:39.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:39.750092+0000) 2024-03-06T08:58:40.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:40.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:40.750328+0000) 2024-03-06T08:58:41.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:41.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:41.750498+0000) 2024-03-06T08:58:42.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:42.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:42.750660+0000) 2024-03-06T08:58:43.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:43.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:43.750924+0000) 2024-03-06T08:58:44.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:44.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:44.751149+0000) 2024-03-06T08:58:45.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:45.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:45.751364+0000) 2024-03-06T08:58:46.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:46.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:46.751582+0000) 2024-03-06T08:58:47.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:47.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:47.751812+0000) 2024-03-06T08:58:48.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:48.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:48.752051+0000) 2024-03-06T08:58:49.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:49.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:49.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:49.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:49.752290+0000) 2024-03-06T08:58:50.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:50.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:50.752475+0000) 2024-03-06T08:58:51.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:51.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:51.752739+0000) 2024-03-06T08:58:52.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:52.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:52.752958+0000) 2024-03-06T08:58:53.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:53.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:53.753179+0000) 2024-03-06T08:58:54.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:54.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:54.753408+0000) 2024-03-06T08:58:55.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:55.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:55.753644+0000) 2024-03-06T08:58:56.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:56.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:56.753931+0000) 2024-03-06T08:58:57.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:57.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:57.754165+0000) 2024-03-06T08:58:58.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:58.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:58.754403+0000) 2024-03-06T08:58:59.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:58:59.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:59.754704+0000) 2024-03-06T08:59:00.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:00.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:00.754906+0000) 2024-03-06T08:59:01.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:01.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:01.755138+0000) 2024-03-06T08:59:02.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:02.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:02.755385+0000) 2024-03-06T08:59:03.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:03.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:03.755725+0000) 2024-03-06T08:59:04.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:04.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:04.756016+0000) 2024-03-06T08:59:05.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:05.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:05.756294+0000) 2024-03-06T08:59:06.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:06.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:06.756576+0000) 2024-03-06T08:59:07.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:07.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:07.756848+0000) 2024-03-06T08:59:08.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:08.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:08.757133+0000) 2024-03-06T08:59:09.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:09.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:09.757411+0000) 2024-03-06T08:59:10.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:10.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:10.757645+0000) 2024-03-06T08:59:11.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:11.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:11.757909+0000) 2024-03-06T08:59:12.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:12.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:12.758091+0000) 2024-03-06T08:59:13.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:13.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:13.758275+0000) 2024-03-06T08:59:14.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:14.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:14.758476+0000) 2024-03-06T08:59:15.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:15.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:15.758743+0000) 2024-03-06T08:59:16.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:16.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:16.758961+0000) 2024-03-06T08:59:17.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:17.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:17.759179+0000) 2024-03-06T08:59:18.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:18.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:18.759381+0000) 2024-03-06T08:59:19.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:19.532 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:19.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:19.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:19.759618+0000) 2024-03-06T08:59:20.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:20.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:20.759879+0000) 2024-03-06T08:59:21.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:21.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:21.760115+0000) 2024-03-06T08:59:22.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:22.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:22.760339+0000) 2024-03-06T08:59:23.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:23.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:23.760580+0000) 2024-03-06T08:59:24.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:24.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:24.760780+0000) 2024-03-06T08:59:25.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:25.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:25.760985+0000) 2024-03-06T08:59:26.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:26.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:26.761209+0000) 2024-03-06T08:59:27.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:27.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:27.761397+0000) 2024-03-06T08:59:28.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:28.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:28.761591+0000) 2024-03-06T08:59:29.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:29.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:29.761813+0000) 2024-03-06T08:59:30.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:30.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:30.762007+0000) 2024-03-06T08:59:31.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:31.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:31.762209+0000) 2024-03-06T08:59:32.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:32.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:32.762426+0000) 2024-03-06T08:59:33.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:33.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:33.762639+0000) 2024-03-06T08:59:34.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:34.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:34.762901+0000) 2024-03-06T08:59:35.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:35.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:35.763073+0000) 2024-03-06T08:59:36.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:36.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:36.763313+0000) 2024-03-06T08:59:37.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:37.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:37.763534+0000) 2024-03-06T08:59:38.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:38.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:38.763767+0000) 2024-03-06T08:59:39.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:39.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:39.763961+0000) 2024-03-06T08:59:40.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:40.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:40.764194+0000) 2024-03-06T08:59:41.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:41.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:41.764417+0000) 2024-03-06T08:59:42.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:42.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:42.764612+0000) 2024-03-06T08:59:43.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:43.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:43.764856+0000) 2024-03-06T08:59:44.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:44.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:44.765105+0000) 2024-03-06T08:59:45.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:45.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:45.765342+0000) 2024-03-06T08:59:46.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:46.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:46.765577+0000) 2024-03-06T08:59:47.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:47.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:47.765760+0000) 2024-03-06T08:59:48.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:48.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:48.765990+0000) 2024-03-06T08:59:49.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:49.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:49.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:49.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:49.766255+0000) 2024-03-06T08:59:50.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:50.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:50.766452+0000) 2024-03-06T08:59:51.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:51.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:51.766653+0000) 2024-03-06T08:59:52.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:52.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:52.766922+0000) 2024-03-06T08:59:53.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:53.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:53.767126+0000) 2024-03-06T08:59:54.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:54.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:54.767380+0000) 2024-03-06T08:59:55.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:55.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:55.767604+0000) 2024-03-06T08:59:56.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:56.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:56.767836+0000) 2024-03-06T08:59:57.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:57.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:57.768100+0000) 2024-03-06T08:59:58.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:58.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:58.768381+0000) 2024-03-06T08:59:59.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T08:59:59.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:59.768582+0000) 2024-03-06T09:00:00.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:00.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:00.768833+0000) 2024-03-06T09:00:01.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:01.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:01.769057+0000) 2024-03-06T09:00:02.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:02.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:02.769279+0000) 2024-03-06T09:00:03.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:03.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:03.769502+0000) 2024-03-06T09:00:04.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:04.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:04.769723+0000) 2024-03-06T09:00:05.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:05.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:05.769878+0000) 2024-03-06T09:00:06.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:06.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:06.770096+0000) 2024-03-06T09:00:07.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:07.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:07.770375+0000) 2024-03-06T09:00:08.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:08.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:08.770645+0000) 2024-03-06T09:00:09.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:09.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:09.770976+0000) 2024-03-06T09:00:10.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:10.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:10.771255+0000) 2024-03-06T09:00:11.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:11.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:11.771537+0000) 2024-03-06T09:00:12.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:12.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:12.771770+0000) 2024-03-06T09:00:13.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:13.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:13.771944+0000) 2024-03-06T09:00:14.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:14.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:14.772219+0000) 2024-03-06T09:00:15.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:15.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:15.772453+0000) 2024-03-06T09:00:16.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:16.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:16.772710+0000) 2024-03-06T09:00:17.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:17.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:17.772935+0000) 2024-03-06T09:00:18.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:18.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:18.773162+0000) 2024-03-06T09:00:19.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:19.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:19.773397+0000) 2024-03-06T09:00:19.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:19.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:20.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:20.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:20.773645+0000) 2024-03-06T09:00:21.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:21.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:21.773934+0000) 2024-03-06T09:00:22.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:22.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:22.774185+0000) 2024-03-06T09:00:23.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:23.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:23.774419+0000) 2024-03-06T09:00:24.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:24.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:24.774695+0000) 2024-03-06T09:00:25.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:25.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:25.774884+0000) 2024-03-06T09:00:26.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:26.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:26.775052+0000) 2024-03-06T09:00:27.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:27.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:27.775327+0000) 2024-03-06T09:00:28.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:28.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:28.775604+0000) 2024-03-06T09:00:29.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:29.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:29.775871+0000) 2024-03-06T09:00:30.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:30.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:30.776152+0000) 2024-03-06T09:00:31.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:31.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:31.776342+0000) 2024-03-06T09:00:32.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:32.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:32.776596+0000) 2024-03-06T09:00:33.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:33.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:33.776838+0000) 2024-03-06T09:00:34.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:34.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:34.777098+0000) 2024-03-06T09:00:35.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:35.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:35.777338+0000) 2024-03-06T09:00:36.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:36.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:36.777582+0000) 2024-03-06T09:00:37.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:37.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:37.777829+0000) 2024-03-06T09:00:38.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:38.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:38.778069+0000) 2024-03-06T09:00:39.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:39.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:39.778357+0000) 2024-03-06T09:00:40.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:40.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:40.778628+0000) 2024-03-06T09:00:41.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:41.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:41.778944+0000) 2024-03-06T09:00:42.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:42.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:42.779227+0000) 2024-03-06T09:00:43.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:43.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:43.779508+0000) 2024-03-06T09:00:44.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:44.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:44.779821+0000) 2024-03-06T09:00:45.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:45.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:45.780019+0000) 2024-03-06T09:00:46.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:46.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:46.780204+0000) 2024-03-06T09:00:47.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:47.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:47.780383+0000) 2024-03-06T09:00:48.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:48.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:48.780575+0000) 2024-03-06T09:00:49.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:49.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:49.780782+0000) 2024-03-06T09:00:50.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:50.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:50.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:50.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:50.780976+0000) 2024-03-06T09:00:51.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:51.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:51.781207+0000) 2024-03-06T09:00:52.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:52.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:52.781427+0000) 2024-03-06T09:00:53.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:53.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:53.781650+0000) 2024-03-06T09:00:54.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:54.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:54.781926+0000) 2024-03-06T09:00:55.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:55.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:55.782127+0000) 2024-03-06T09:00:56.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:56.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:56.782353+0000) 2024-03-06T09:00:57.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:57.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:57.782589+0000) 2024-03-06T09:00:58.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:58.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:58.782825+0000) 2024-03-06T09:00:59.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:00:59.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:59.783045+0000) 2024-03-06T09:01:00.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:00.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:00.783275+0000) 2024-03-06T09:01:01.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:01.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:01.783459+0000) 2024-03-06T09:01:02.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:02.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:02.783721+0000) 2024-03-06T09:01:03.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:03.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:03.783899+0000) 2024-03-06T09:01:04.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:04.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:04.784119+0000) 2024-03-06T09:01:05.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:05.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:05.784317+0000) 2024-03-06T09:01:06.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:06.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:06.784514+0000) 2024-03-06T09:01:07.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:07.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:07.784794+0000) 2024-03-06T09:01:08.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:08.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:08.785058+0000) 2024-03-06T09:01:09.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:09.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:09.785294+0000) 2024-03-06T09:01:10.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:10.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:10.785517+0000) 2024-03-06T09:01:11.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:11.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:11.785769+0000) 2024-03-06T09:01:12.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:12.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:12.785965+0000) 2024-03-06T09:01:13.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:13.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:13.786190+0000) 2024-03-06T09:01:14.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:14.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:14.786410+0000) 2024-03-06T09:01:15.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:15.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:15.786645+0000) 2024-03-06T09:01:16.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:16.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:16.786883+0000) 2024-03-06T09:01:17.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:17.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:17.787120+0000) 2024-03-06T09:01:18.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:18.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:18.787337+0000) 2024-03-06T09:01:19.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:19.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:19.787606+0000) 2024-03-06T09:01:20.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:20.341 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:20.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:20.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:20.787866+0000) 2024-03-06T09:01:21.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:21.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:21.788153+0000) 2024-03-06T09:01:22.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:22.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:22.788436+0000) 2024-03-06T09:01:23.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:23.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:23.788648+0000) 2024-03-06T09:01:24.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:24.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:24.788914+0000) 2024-03-06T09:01:25.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:25.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:25.789098+0000) 2024-03-06T09:01:26.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:26.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:26.789278+0000) 2024-03-06T09:01:27.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:27.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:27.789443+0000) 2024-03-06T09:01:28.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:28.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:28.789647+0000) 2024-03-06T09:01:29.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:29.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:29.789962+0000) 2024-03-06T09:01:30.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:30.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:30.790223+0000) 2024-03-06T09:01:31.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:31.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:31.790491+0000) 2024-03-06T09:01:32.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:32.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:32.790809+0000) 2024-03-06T09:01:33.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:33.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:33.791007+0000) 2024-03-06T09:01:34.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:34.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:34.791237+0000) 2024-03-06T09:01:35.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:35.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:35.791453+0000) 2024-03-06T09:01:36.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:36.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:36.791713+0000) 2024-03-06T09:01:37.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:37.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:37.791885+0000) 2024-03-06T09:01:38.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:38.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:38.792130+0000) 2024-03-06T09:01:39.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:39.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:39.792371+0000) 2024-03-06T09:01:40.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:40.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:40.792598+0000) 2024-03-06T09:01:41.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:41.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:41.792820+0000) 2024-03-06T09:01:42.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:42.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:42.793001+0000) 2024-03-06T09:01:43.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:43.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:43.793220+0000) 2024-03-06T09:01:44.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:44.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:44.793450+0000) 2024-03-06T09:01:45.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:45.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:45.793653+0000) 2024-03-06T09:01:46.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:46.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:46.793908+0000) 2024-03-06T09:01:47.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:47.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:47.794111+0000) 2024-03-06T09:01:48.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:48.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:48.794371+0000) 2024-03-06T09:01:49.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:49.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:49.794646+0000) 2024-03-06T09:01:50.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:50.479 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:50.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:50.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:50.794969+0000) 2024-03-06T09:01:51.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:51.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:51.795171+0000) 2024-03-06T09:01:52.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:52.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:52.795359+0000) 2024-03-06T09:01:53.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:53.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:53.795623+0000) 2024-03-06T09:01:54.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:54.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:54.795871+0000) 2024-03-06T09:01:55.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:55.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:55.796049+0000) 2024-03-06T09:01:56.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:56.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:56.796253+0000) 2024-03-06T09:01:57.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:57.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:57.796450+0000) 2024-03-06T09:01:58.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:58.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:58.796760+0000) 2024-03-06T09:01:59.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:01:59.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:59.797033+0000) 2024-03-06T09:02:00.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:00.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:00.797362+0000) 2024-03-06T09:02:01.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:01.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:01.797640+0000) 2024-03-06T09:02:02.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:02.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:02.797926+0000) 2024-03-06T09:02:03.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:03.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:03.798204+0000) 2024-03-06T09:02:04.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:04.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:04.798438+0000) 2024-03-06T09:02:05.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:05.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:05.798625+0000) 2024-03-06T09:02:06.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:06.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:06.798849+0000) 2024-03-06T09:02:07.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:07.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:07.799071+0000) 2024-03-06T09:02:08.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:08.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:08.799306+0000) 2024-03-06T09:02:09.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:09.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:09.799529+0000) 2024-03-06T09:02:10.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:10.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:10.799797+0000) 2024-03-06T09:02:11.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:11.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:11.800029+0000) 2024-03-06T09:02:12.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:12.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:12.800252+0000) 2024-03-06T09:02:13.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:13.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:13.800474+0000) 2024-03-06T09:02:14.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:14.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:14.800749+0000) 2024-03-06T09:02:15.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:15.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:15.800946+0000) 2024-03-06T09:02:16.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:16.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:16.801187+0000) 2024-03-06T09:02:17.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:17.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:17.801414+0000) 2024-03-06T09:02:18.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:18.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:18.801603+0000) 2024-03-06T09:02:19.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:19.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:19.801846+0000) 2024-03-06T09:02:20.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:20.674 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:20.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:20.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:20.802079+0000) 2024-03-06T09:02:21.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:21.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:21.802306+0000) 2024-03-06T09:02:22.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:22.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:22.802523+0000) 2024-03-06T09:02:23.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:23.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:23.802742+0000) 2024-03-06T09:02:24.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:24.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:24.802991+0000) 2024-03-06T09:02:25.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:25.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:25.803226+0000) 2024-03-06T09:02:26.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:26.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:26.803459+0000) 2024-03-06T09:02:27.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:27.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:27.803650+0000) 2024-03-06T09:02:28.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:28.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:28.803936+0000) 2024-03-06T09:02:29.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:29.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:29.804156+0000) 2024-03-06T09:02:30.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:30.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:30.804379+0000) 2024-03-06T09:02:31.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:31.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:31.804617+0000) 2024-03-06T09:02:32.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:32.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:32.804857+0000) 2024-03-06T09:02:33.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:33.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:33.805095+0000) 2024-03-06T09:02:34.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:34.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:34.805381+0000) 2024-03-06T09:02:35.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:35.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:35.805657+0000) 2024-03-06T09:02:36.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:36.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:36.805912+0000) 2024-03-06T09:02:37.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:37.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:37.806109+0000) 2024-03-06T09:02:38.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:38.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:38.806348+0000) 2024-03-06T09:02:39.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:39.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:39.806570+0000) 2024-03-06T09:02:40.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:40.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:40.806801+0000) 2024-03-06T09:02:41.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:41.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:41.807029+0000) 2024-03-06T09:02:42.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:42.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:42.807255+0000) 2024-03-06T09:02:43.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:43.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:43.807464+0000) 2024-03-06T09:02:44.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:44.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:44.807711+0000) 2024-03-06T09:02:45.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:45.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:45.807920+0000) 2024-03-06T09:02:46.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:46.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:46.808179+0000) 2024-03-06T09:02:47.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:47.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:47.808455+0000) 2024-03-06T09:02:48.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:48.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:48.808832+0000) 2024-03-06T09:02:49.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:49.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:49.809029+0000) 2024-03-06T09:02:50.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:50.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:50.809227+0000) 2024-03-06T09:02:50.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:50.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:51.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:51.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:51.809464+0000) 2024-03-06T09:02:52.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:52.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:52.809650+0000) 2024-03-06T09:02:53.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:53.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:53.809880+0000) 2024-03-06T09:02:54.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:54.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:54.810124+0000) 2024-03-06T09:02:55.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:55.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:55.810321+0000) 2024-03-06T09:02:56.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:56.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:56.810516+0000) 2024-03-06T09:02:57.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:57.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:57.810776+0000) 2024-03-06T09:02:58.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:58.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:58.810925+0000) 2024-03-06T09:02:59.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:02:59.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:59.811147+0000) 2024-03-06T09:03:00.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:00.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:00.811382+0000) 2024-03-06T09:03:01.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:01.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:01.811568+0000) 2024-03-06T09:03:02.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:02.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:02.811781+0000) 2024-03-06T09:03:03.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:03.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:03.812001+0000) 2024-03-06T09:03:04.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:04.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:04.812233+0000) 2024-03-06T09:03:05.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:05.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:05.812532+0000) 2024-03-06T09:03:06.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:06.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:06.812805+0000) 2024-03-06T09:03:07.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:07.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:07.813029+0000) 2024-03-06T09:03:08.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:08.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:08.813307+0000) 2024-03-06T09:03:09.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:09.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:09.813578+0000) 2024-03-06T09:03:10.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:10.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:10.813863+0000) 2024-03-06T09:03:11.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:11.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:11.814095+0000) 2024-03-06T09:03:12.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:12.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:12.814363+0000) 2024-03-06T09:03:13.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:13.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:13.814647+0000) 2024-03-06T09:03:14.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:14.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:14.814964+0000) 2024-03-06T09:03:15.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:15.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:15.815244+0000) 2024-03-06T09:03:16.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:16.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:16.815451+0000) 2024-03-06T09:03:17.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:17.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:17.815760+0000) 2024-03-06T09:03:18.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:18.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:18.816009+0000) 2024-03-06T09:03:19.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:19.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:19.816285+0000) 2024-03-06T09:03:20.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:20.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:20.816577+0000) 2024-03-06T09:03:21.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:21.082 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:21.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:21.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:21.816827+0000) 2024-03-06T09:03:22.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:22.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:22.817047+0000) 2024-03-06T09:03:23.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:23.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:23.817269+0000) 2024-03-06T09:03:24.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:24.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:24.817553+0000) 2024-03-06T09:03:25.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:25.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:25.817854+0000) 2024-03-06T09:03:26.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:26.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:26.818060+0000) 2024-03-06T09:03:27.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:27.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:27.818287+0000) 2024-03-06T09:03:28.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:28.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:28.818514+0000) 2024-03-06T09:03:29.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:29.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:29.818764+0000) 2024-03-06T09:03:30.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:30.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:30.819016+0000) 2024-03-06T09:03:31.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:31.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:31.819209+0000) 2024-03-06T09:03:32.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:32.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:32.819390+0000) 2024-03-06T09:03:33.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:33.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:33.819603+0000) 2024-03-06T09:03:34.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:34.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:34.819829+0000) 2024-03-06T09:03:35.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:35.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:35.820058+0000) 2024-03-06T09:03:36.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:36.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:36.820258+0000) 2024-03-06T09:03:37.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:37.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:37.820446+0000) 2024-03-06T09:03:38.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:38.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:38.820709+0000) 2024-03-06T09:03:39.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:39.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:39.820938+0000) 2024-03-06T09:03:40.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:40.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:40.821130+0000) 2024-03-06T09:03:41.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:41.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:41.821321+0000) 2024-03-06T09:03:42.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:42.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:42.821591+0000) 2024-03-06T09:03:43.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:43.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:43.821855+0000) 2024-03-06T09:03:44.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:44.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:44.822101+0000) 2024-03-06T09:03:45.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:45.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:45.822328+0000) 2024-03-06T09:03:46.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:46.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:46.822517+0000) 2024-03-06T09:03:47.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:47.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:47.822829+0000) 2024-03-06T09:03:48.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:48.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:48.823027+0000) 2024-03-06T09:03:49.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:49.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:49.823262+0000) 2024-03-06T09:03:50.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:50.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:50.823504+0000) 2024-03-06T09:03:51.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:51.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:51.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:51.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:51.823810+0000) 2024-03-06T09:03:52.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:52.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:52.824054+0000) 2024-03-06T09:03:53.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:53.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:53.824290+0000) 2024-03-06T09:03:54.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:54.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:54.824536+0000) 2024-03-06T09:03:55.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:55.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:55.824774+0000) 2024-03-06T09:03:56.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:56.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:56.824958+0000) 2024-03-06T09:03:57.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:57.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:57.825173+0000) 2024-03-06T09:03:58.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:58.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:58.825411+0000) 2024-03-06T09:03:59.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:03:59.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:59.825647+0000) 2024-03-06T09:04:00.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:00.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:00.825929+0000) 2024-03-06T09:04:01.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:01.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:01.826174+0000) 2024-03-06T09:04:02.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:02.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:02.826416+0000) 2024-03-06T09:04:03.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:03.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:03.826655+0000) 2024-03-06T09:04:04.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:04.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:04.826939+0000) 2024-03-06T09:04:05.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:05.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:05.827151+0000) 2024-03-06T09:04:06.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:06.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:06.827389+0000) 2024-03-06T09:04:07.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:07.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:07.827609+0000) 2024-03-06T09:04:08.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:08.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:08.827843+0000) 2024-03-06T09:04:09.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:09.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:09.828043+0000) 2024-03-06T09:04:10.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:10.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:10.828237+0000) 2024-03-06T09:04:11.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:11.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:11.828475+0000) 2024-03-06T09:04:12.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:12.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:12.828708+0000) 2024-03-06T09:04:13.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:13.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:13.828960+0000) 2024-03-06T09:04:14.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:14.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:14.829244+0000) 2024-03-06T09:04:15.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:15.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:15.829505+0000) 2024-03-06T09:04:16.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:16.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:16.829829+0000) 2024-03-06T09:04:17.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:17.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:17.830064+0000) 2024-03-06T09:04:18.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:18.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:18.830303+0000) 2024-03-06T09:04:19.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:19.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:19.830524+0000) 2024-03-06T09:04:20.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:20.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:20.830811+0000) 2024-03-06T09:04:21.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:21.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:21.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:21.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:21.830980+0000) 2024-03-06T09:04:22.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:22.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:22.831197+0000) 2024-03-06T09:04:23.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:23.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:23.831417+0000) 2024-03-06T09:04:24.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:24.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:24.831656+0000) 2024-03-06T09:04:25.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:25.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:25.831883+0000) 2024-03-06T09:04:26.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:26.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:26.832106+0000) 2024-03-06T09:04:27.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:27.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:27.832375+0000) 2024-03-06T09:04:28.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:28.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:28.832577+0000) 2024-03-06T09:04:29.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:29.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:29.832811+0000) 2024-03-06T09:04:30.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:30.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:30.833071+0000) 2024-03-06T09:04:31.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:31.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:31.833310+0000) 2024-03-06T09:04:32.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:32.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:32.833576+0000) 2024-03-06T09:04:33.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:33.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:33.833810+0000) 2024-03-06T09:04:34.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:34.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:34.834007+0000) 2024-03-06T09:04:35.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:35.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:35.834231+0000) 2024-03-06T09:04:36.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:36.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:36.834456+0000) 2024-03-06T09:04:37.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:37.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:37.834713+0000) 2024-03-06T09:04:38.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:38.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:38.834897+0000) 2024-03-06T09:04:39.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:39.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:39.835063+0000) 2024-03-06T09:04:40.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:40.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:40.835316+0000) 2024-03-06T09:04:41.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:41.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:41.835596+0000) 2024-03-06T09:04:42.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:42.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:42.835849+0000) 2024-03-06T09:04:43.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:43.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:43.836093+0000) 2024-03-06T09:04:44.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:44.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:44.836380+0000) 2024-03-06T09:04:45.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:45.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:45.836646+0000) 2024-03-06T09:04:46.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:46.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:46.836947+0000) 2024-03-06T09:04:47.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:47.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:47.837156+0000) 2024-03-06T09:04:48.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:48.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:48.837351+0000) 2024-03-06T09:04:49.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:49.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:49.837563+0000) 2024-03-06T09:04:50.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:50.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:50.837800+0000) 2024-03-06T09:04:51.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:51.739 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:51.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:51.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:51.837970+0000) 2024-03-06T09:04:52.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:52.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:52.838131+0000) 2024-03-06T09:04:53.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:53.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:53.838312+0000) 2024-03-06T09:04:54.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:54.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:54.838554+0000) 2024-03-06T09:04:55.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:55.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:55.838756+0000) 2024-03-06T09:04:56.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:56.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:56.839006+0000) 2024-03-06T09:04:57.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:57.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:57.839245+0000) 2024-03-06T09:04:58.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:58.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:58.839455+0000) 2024-03-06T09:04:59.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:04:59.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:59.839696+0000) 2024-03-06T09:05:00.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:00.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:00.839917+0000) 2024-03-06T09:05:01.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:01.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:01.840129+0000) 2024-03-06T09:05:02.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:02.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:02.840405+0000) 2024-03-06T09:05:03.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:03.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:03.840601+0000) 2024-03-06T09:05:04.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:04.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:04.840837+0000) 2024-03-06T09:05:05.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:05.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:05.841024+0000) 2024-03-06T09:05:06.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:06.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:06.841202+0000) 2024-03-06T09:05:07.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:07.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:07.841428+0000) 2024-03-06T09:05:08.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:08.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:08.841711+0000) 2024-03-06T09:05:09.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:09.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:09.841927+0000) 2024-03-06T09:05:10.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:10.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:10.842162+0000) 2024-03-06T09:05:11.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:11.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:11.842436+0000) 2024-03-06T09:05:12.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:12.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:12.842772+0000) 2024-03-06T09:05:13.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:13.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:13.842969+0000) 2024-03-06T09:05:14.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:14.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:14.843204+0000) 2024-03-06T09:05:15.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:15.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:15.843409+0000) 2024-03-06T09:05:16.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:16.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:16.843732+0000) 2024-03-06T09:05:17.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:17.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:17.843971+0000) 2024-03-06T09:05:18.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:18.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:18.844240+0000) 2024-03-06T09:05:19.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:19.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:19.844443+0000) 2024-03-06T09:05:20.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:20.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:20.844760+0000) 2024-03-06T09:05:21.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:21.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:21.844993+0000) 2024-03-06T09:05:21.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:21.949 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:22.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:22.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:22.845198+0000) 2024-03-06T09:05:23.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:23.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:23.845477+0000) 2024-03-06T09:05:24.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:24.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:24.845766+0000) 2024-03-06T09:05:25.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:25.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:25.845949+0000) 2024-03-06T09:05:26.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:26.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:26.846163+0000) 2024-03-06T09:05:27.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:27.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:27.846428+0000) 2024-03-06T09:05:28.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:28.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:28.846661+0000) 2024-03-06T09:05:29.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:29.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:29.846928+0000) 2024-03-06T09:05:30.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:30.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:30.847157+0000) 2024-03-06T09:05:31.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:31.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:31.847374+0000) 2024-03-06T09:05:32.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:32.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:32.847597+0000) 2024-03-06T09:05:33.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:33.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:33.847794+0000) 2024-03-06T09:05:34.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:34.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:34.847999+0000) 2024-03-06T09:05:35.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:35.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:35.848186+0000) 2024-03-06T09:05:36.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:36.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:36.848416+0000) 2024-03-06T09:05:37.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:37.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:37.848640+0000) 2024-03-06T09:05:38.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:38.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:38.848915+0000) 2024-03-06T09:05:39.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:39.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:39.849154+0000) 2024-03-06T09:05:40.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:40.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:40.849376+0000) 2024-03-06T09:05:41.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:41.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:41.849599+0000) 2024-03-06T09:05:42.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:42.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:42.849839+0000) 2024-03-06T09:05:43.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:43.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:43.850060+0000) 2024-03-06T09:05:44.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:44.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:44.850280+0000) 2024-03-06T09:05:45.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:45.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:45.850518+0000) 2024-03-06T09:05:46.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:46.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:46.850787+0000) 2024-03-06T09:05:47.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:47.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:47.850966+0000) 2024-03-06T09:05:48.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:48.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:48.851181+0000) 2024-03-06T09:05:49.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:49.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:49.851405+0000) 2024-03-06T09:05:50.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:50.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:50.851629+0000) 2024-03-06T09:05:51.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:51.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:51.851818+0000) 2024-03-06T09:05:52.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:52.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:52.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:52.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:52.852006+0000) 2024-03-06T09:05:53.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:53.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:53.852230+0000) 2024-03-06T09:05:54.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:54.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:54.852467+0000) 2024-03-06T09:05:55.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:55.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:55.852719+0000) 2024-03-06T09:05:56.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:56.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:56.852959+0000) 2024-03-06T09:05:57.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:57.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:57.853178+0000) 2024-03-06T09:05:58.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:58.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:58.853405+0000) 2024-03-06T09:05:59.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:05:59.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:59.853629+0000) 2024-03-06T09:06:00.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:00.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:00.853848+0000) 2024-03-06T09:06:01.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:01.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:01.854065+0000) 2024-03-06T09:06:02.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:02.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:02.854256+0000) 2024-03-06T09:06:03.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:03.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:03.854475+0000) 2024-03-06T09:06:04.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:04.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:04.854752+0000) 2024-03-06T09:06:05.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:05.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:05.854975+0000) 2024-03-06T09:06:06.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:06.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:06.855205+0000) 2024-03-06T09:06:07.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:07.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:07.855400+0000) 2024-03-06T09:06:08.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:08.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:08.855634+0000) 2024-03-06T09:06:09.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:09.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:09.855861+0000) 2024-03-06T09:06:10.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:10.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:10.856086+0000) 2024-03-06T09:06:11.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:11.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:11.856327+0000) 2024-03-06T09:06:12.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:12.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:12.856537+0000) 2024-03-06T09:06:13.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:13.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:13.856783+0000) 2024-03-06T09:06:14.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:14.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:14.857000+0000) 2024-03-06T09:06:15.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:15.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:15.857249+0000) 2024-03-06T09:06:16.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:16.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:16.857568+0000) 2024-03-06T09:06:17.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:17.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:17.857807+0000) 2024-03-06T09:06:18.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:18.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:18.857972+0000) 2024-03-06T09:06:19.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:19.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:19.858141+0000) 2024-03-06T09:06:20.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:20.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:20.858362+0000) 2024-03-06T09:06:21.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:21.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:21.858526+0000) 2024-03-06T09:06:22.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:22.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:22.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:22.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:22.858781+0000) 2024-03-06T09:06:23.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:23.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:23.859002+0000) 2024-03-06T09:06:24.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:24.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:24.859244+0000) 2024-03-06T09:06:25.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:25.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:25.859432+0000) 2024-03-06T09:06:26.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:26.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:26.859717+0000) 2024-03-06T09:06:27.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:27.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:27.859923+0000) 2024-03-06T09:06:28.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:28.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:28.860105+0000) 2024-03-06T09:06:29.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:29.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:29.860286+0000) 2024-03-06T09:06:30.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:30.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:30.860509+0000) 2024-03-06T09:06:31.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:31.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:31.860761+0000) 2024-03-06T09:06:32.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:32.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:32.860989+0000) 2024-03-06T09:06:33.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:33.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:33.861200+0000) 2024-03-06T09:06:34.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:34.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:34.861382+0000) 2024-03-06T09:06:35.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:35.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:35.861593+0000) 2024-03-06T09:06:36.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:36.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:36.861829+0000) 2024-03-06T09:06:37.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:37.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:37.862091+0000) 2024-03-06T09:06:38.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:38.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:38.862318+0000) 2024-03-06T09:06:39.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:39.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:39.862545+0000) 2024-03-06T09:06:40.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:40.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:40.862769+0000) 2024-03-06T09:06:41.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:41.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:41.862999+0000) 2024-03-06T09:06:42.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:42.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:42.863179+0000) 2024-03-06T09:06:43.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:43.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:43.863346+0000) 2024-03-06T09:06:44.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:44.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:44.863522+0000) 2024-03-06T09:06:45.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:45.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:45.863773+0000) 2024-03-06T09:06:46.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:46.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:46.864005+0000) 2024-03-06T09:06:47.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:47.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:47.864226+0000) 2024-03-06T09:06:48.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:48.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:48.864460+0000) 2024-03-06T09:06:49.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:49.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:49.864711+0000) 2024-03-06T09:06:50.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:50.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:50.864942+0000) 2024-03-06T09:06:51.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:51.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:51.865170+0000) 2024-03-06T09:06:52.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:52.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:52.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:52.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:52.865403+0000) 2024-03-06T09:06:53.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:53.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:53.865594+0000) 2024-03-06T09:06:54.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:54.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:54.865779+0000) 2024-03-06T09:06:55.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:55.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:55.865942+0000) 2024-03-06T09:06:56.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:56.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:56.866176+0000) 2024-03-06T09:06:57.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:57.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:57.866367+0000) 2024-03-06T09:06:58.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:58.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:58.866609+0000) 2024-03-06T09:06:59.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:06:59.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:59.866845+0000) 2024-03-06T09:07:00.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:00.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:00.867043+0000) 2024-03-06T09:07:01.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:01.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:01.867251+0000) 2024-03-06T09:07:02.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:02.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:02.867489+0000) 2024-03-06T09:07:03.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:03.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:03.867687+0000) 2024-03-06T09:07:04.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:04.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:04.867894+0000) 2024-03-06T09:07:05.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:05.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:05.868131+0000) 2024-03-06T09:07:06.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:06.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:06.868373+0000) 2024-03-06T09:07:07.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:07.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:07.868614+0000) 2024-03-06T09:07:08.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:08.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:08.868816+0000) 2024-03-06T09:07:09.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:09.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:09.869064+0000) 2024-03-06T09:07:10.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:10.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:10.869284+0000) 2024-03-06T09:07:11.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:11.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:11.869502+0000) 2024-03-06T09:07:12.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:12.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:12.869783+0000) 2024-03-06T09:07:13.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:13.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:13.869975+0000) 2024-03-06T09:07:14.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:14.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:14.870221+0000) 2024-03-06T09:07:15.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:15.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:15.870444+0000) 2024-03-06T09:07:16.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:16.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:16.870707+0000) 2024-03-06T09:07:17.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:17.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:17.870899+0000) 2024-03-06T09:07:18.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:18.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:18.871090+0000) 2024-03-06T09:07:19.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:19.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:19.871316+0000) 2024-03-06T09:07:20.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:20.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:20.871546+0000) 2024-03-06T09:07:21.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:21.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:21.871787+0000) 2024-03-06T09:07:22.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:22.649 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:22.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:22.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:22.871988+0000) 2024-03-06T09:07:23.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:23.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:23.872223+0000) 2024-03-06T09:07:24.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:24.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:24.872415+0000) 2024-03-06T09:07:25.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:25.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:25.872634+0000) 2024-03-06T09:07:26.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:26.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:26.872842+0000) 2024-03-06T09:07:27.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:27.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:27.873025+0000) 2024-03-06T09:07:28.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:28.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:28.873259+0000) 2024-03-06T09:07:29.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:29.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:29.873484+0000) 2024-03-06T09:07:30.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:30.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:30.873707+0000) 2024-03-06T09:07:31.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:31.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:31.873938+0000) 2024-03-06T09:07:32.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:32.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:32.874160+0000) 2024-03-06T09:07:33.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:33.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:33.874388+0000) 2024-03-06T09:07:34.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:34.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:34.874579+0000) 2024-03-06T09:07:35.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:35.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:35.874807+0000) 2024-03-06T09:07:36.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:36.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:36.875000+0000) 2024-03-06T09:07:37.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:37.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:37.875229+0000) 2024-03-06T09:07:38.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:38.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:38.875474+0000) 2024-03-06T09:07:39.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:39.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:39.875741+0000) 2024-03-06T09:07:40.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:40.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:40.875964+0000) 2024-03-06T09:07:41.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:41.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:41.876144+0000) 2024-03-06T09:07:42.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:42.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:42.876373+0000) 2024-03-06T09:07:43.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:43.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:43.876597+0000) 2024-03-06T09:07:44.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:44.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:44.876816+0000) 2024-03-06T09:07:45.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:45.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:45.877044+0000) 2024-03-06T09:07:46.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:46.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:46.877271+0000) 2024-03-06T09:07:47.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:47.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:47.877484+0000) 2024-03-06T09:07:48.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:48.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:48.877769+0000) 2024-03-06T09:07:49.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:49.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:49.877962+0000) 2024-03-06T09:07:50.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:50.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:50.878186+0000) 2024-03-06T09:07:51.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:51.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:51.878407+0000) 2024-03-06T09:07:52.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:52.793 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:52.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:52.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:52.878657+0000) 2024-03-06T09:07:53.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:53.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:53.878895+0000) 2024-03-06T09:07:54.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:54.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:54.879138+0000) 2024-03-06T09:07:55.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:55.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:55.879374+0000) 2024-03-06T09:07:56.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:56.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:56.879622+0000) 2024-03-06T09:07:57.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:57.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:57.879864+0000) 2024-03-06T09:07:58.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:58.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:58.880077+0000) 2024-03-06T09:07:59.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:07:59.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:59.880302+0000) 2024-03-06T09:08:00.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:00.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:00.880524+0000) 2024-03-06T09:08:01.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:01.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:01.880791+0000) 2024-03-06T09:08:02.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:02.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:02.881012+0000) 2024-03-06T09:08:03.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:03.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:03.881246+0000) 2024-03-06T09:08:04.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:04.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:04.881460+0000) 2024-03-06T09:08:05.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:05.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:05.881735+0000) 2024-03-06T09:08:06.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:06.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:06.881945+0000) 2024-03-06T09:08:07.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:07.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:07.882190+0000) 2024-03-06T09:08:08.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:08.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:08.882424+0000) 2024-03-06T09:08:09.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:09.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:09.882570+0000) 2024-03-06T09:08:10.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:10.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:10.882795+0000) 2024-03-06T09:08:11.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:11.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:11.882988+0000) 2024-03-06T09:08:12.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:12.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:12.883211+0000) 2024-03-06T09:08:13.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:13.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:13.883441+0000) 2024-03-06T09:08:14.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:14.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:14.883713+0000) 2024-03-06T09:08:15.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:15.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:15.883900+0000) 2024-03-06T09:08:16.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:16.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:16.884109+0000) 2024-03-06T09:08:17.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:17.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:17.884332+0000) 2024-03-06T09:08:18.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:18.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:18.884569+0000) 2024-03-06T09:08:19.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:19.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:19.884800+0000) 2024-03-06T09:08:20.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:20.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:20.885006+0000) 2024-03-06T09:08:21.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:21.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:21.885237+0000) 2024-03-06T09:08:22.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:22.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:22.885462+0000) 2024-03-06T09:08:22.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:23.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:23.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:23.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:23.885742+0000) 2024-03-06T09:08:24.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:24.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:24.885963+0000) 2024-03-06T09:08:25.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:25.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:25.886172+0000) 2024-03-06T09:08:26.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:26.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:26.886373+0000) 2024-03-06T09:08:27.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:27.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:27.886602+0000) 2024-03-06T09:08:28.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:28.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:28.886840+0000) 2024-03-06T09:08:29.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:29.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:29.887060+0000) 2024-03-06T09:08:30.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:30.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:30.887242+0000) 2024-03-06T09:08:31.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:31.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:31.887475+0000) 2024-03-06T09:08:32.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:32.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:32.887755+0000) 2024-03-06T09:08:33.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:33.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:33.887970+0000) 2024-03-06T09:08:34.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:34.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:34.888196+0000) 2024-03-06T09:08:35.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:35.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:35.888403+0000) 2024-03-06T09:08:36.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:36.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:36.888652+0000) 2024-03-06T09:08:37.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:37.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:37.889014+0000) 2024-03-06T09:08:38.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:38.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:38.889325+0000) 2024-03-06T09:08:39.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:39.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:39.889545+0000) 2024-03-06T09:08:40.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:40.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:40.889726+0000) 2024-03-06T09:08:41.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:41.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:41.889923+0000) 2024-03-06T09:08:42.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:42.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:42.890120+0000) 2024-03-06T09:08:43.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:43.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:43.890296+0000) 2024-03-06T09:08:44.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:44.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:44.890487+0000) 2024-03-06T09:08:45.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:45.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:45.890702+0000) 2024-03-06T09:08:46.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:46.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:46.890951+0000) 2024-03-06T09:08:47.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:47.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:47.891143+0000) 2024-03-06T09:08:48.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:48.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:48.891346+0000) 2024-03-06T09:08:49.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:49.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:49.891568+0000) 2024-03-06T09:08:50.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:50.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:50.891795+0000) 2024-03-06T09:08:51.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:51.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:51.892086+0000) 2024-03-06T09:08:52.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:52.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:52.892314+0000) 2024-03-06T09:08:53.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:53.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:53.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:53.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:53.892537+0000) 2024-03-06T09:08:54.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:54.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:54.892752+0000) 2024-03-06T09:08:55.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:55.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:55.892971+0000) 2024-03-06T09:08:56.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:56.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:56.893179+0000) 2024-03-06T09:08:57.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:57.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:57.893395+0000) 2024-03-06T09:08:58.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:58.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:58.893615+0000) 2024-03-06T09:08:59.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:08:59.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:59.893832+0000) 2024-03-06T09:09:00.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:00.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:00.894059+0000) 2024-03-06T09:09:01.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:01.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:01.894258+0000) 2024-03-06T09:09:02.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:02.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:02.894466+0000) 2024-03-06T09:09:03.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:03.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:03.894719+0000) 2024-03-06T09:09:04.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:04.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:04.894911+0000) 2024-03-06T09:09:05.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:05.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:05.895116+0000) 2024-03-06T09:09:06.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:06.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:06.895381+0000) 2024-03-06T09:09:07.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:07.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:07.895605+0000) 2024-03-06T09:09:08.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:08.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:08.895821+0000) 2024-03-06T09:09:09.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:09.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:09.896021+0000) 2024-03-06T09:09:10.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:10.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:10.896258+0000) 2024-03-06T09:09:11.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:11.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:11.896496+0000) 2024-03-06T09:09:12.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:12.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:12.896771+0000) 2024-03-06T09:09:13.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:13.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:13.896963+0000) 2024-03-06T09:09:14.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:14.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:14.897192+0000) 2024-03-06T09:09:15.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:15.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:15.897403+0000) 2024-03-06T09:09:16.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:16.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:16.897653+0000) 2024-03-06T09:09:17.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:17.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:17.897909+0000) 2024-03-06T09:09:18.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:18.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:18.898137+0000) 2024-03-06T09:09:19.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:19.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:19.898315+0000) 2024-03-06T09:09:20.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:20.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:20.898541+0000) 2024-03-06T09:09:21.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:21.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:21.898799+0000) 2024-03-06T09:09:22.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:22.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:22.898953+0000) 2024-03-06T09:09:23.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:23.473 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:23.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:23.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:23.899150+0000) 2024-03-06T09:09:24.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:24.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:24.899344+0000) 2024-03-06T09:09:25.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:25.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:25.899600+0000) 2024-03-06T09:09:26.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:26.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:26.899833+0000) 2024-03-06T09:09:27.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:27.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:27.900027+0000) 2024-03-06T09:09:28.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:28.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:28.900260+0000) 2024-03-06T09:09:29.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:29.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:29.900446+0000) 2024-03-06T09:09:30.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:30.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:30.900637+0000) 2024-03-06T09:09:31.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:31.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:31.900891+0000) 2024-03-06T09:09:32.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:32.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:32.901119+0000) 2024-03-06T09:09:33.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:33.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:33.901345+0000) 2024-03-06T09:09:34.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:34.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:34.901586+0000) 2024-03-06T09:09:35.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:35.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:35.901826+0000) 2024-03-06T09:09:36.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:36.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:36.901999+0000) 2024-03-06T09:09:37.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:37.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:37.902224+0000) 2024-03-06T09:09:38.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:38.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:38.902454+0000) 2024-03-06T09:09:39.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:39.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:39.902780+0000) 2024-03-06T09:09:40.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:40.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:40.902972+0000) 2024-03-06T09:09:41.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:41.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:41.903148+0000) 2024-03-06T09:09:42.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:42.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:42.903373+0000) 2024-03-06T09:09:43.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:43.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:43.903593+0000) 2024-03-06T09:09:44.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:44.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:44.903798+0000) 2024-03-06T09:09:45.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:45.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:45.904038+0000) 2024-03-06T09:09:46.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:46.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:46.904225+0000) 2024-03-06T09:09:47.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:47.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:47.904451+0000) 2024-03-06T09:09:48.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:48.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:48.904650+0000) 2024-03-06T09:09:49.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:49.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:49.904878+0000) 2024-03-06T09:09:50.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:50.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:50.905027+0000) 2024-03-06T09:09:51.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:51.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:51.905238+0000) 2024-03-06T09:09:52.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:52.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:52.905474+0000) 2024-03-06T09:09:53.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:53.612 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:53.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:53.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:53.905635+0000) 2024-03-06T09:09:54.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:54.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:54.905888+0000) 2024-03-06T09:09:55.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:55.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:55.906061+0000) 2024-03-06T09:09:56.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:56.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:56.906284+0000) 2024-03-06T09:09:57.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:57.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:57.906507+0000) 2024-03-06T09:09:58.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:58.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:58.906770+0000) 2024-03-06T09:09:59.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:09:59.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:59.906990+0000) 2024-03-06T09:10:00.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:00.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:00.907176+0000) 2024-03-06T09:10:01.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:01.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:01.907402+0000) 2024-03-06T09:10:02.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:02.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:02.907629+0000) 2024-03-06T09:10:03.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:03.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:03.907845+0000) 2024-03-06T09:10:04.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:04.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:04.908079+0000) 2024-03-06T09:10:05.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:05.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:05.908300+0000) 2024-03-06T09:10:06.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:06.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:06.908487+0000) 2024-03-06T09:10:07.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:07.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:07.908791+0000) 2024-03-06T09:10:08.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:08.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:08.909001+0000) 2024-03-06T09:10:09.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:09.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:09.909220+0000) 2024-03-06T09:10:10.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:10.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:10.909459+0000) 2024-03-06T09:10:11.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:11.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:11.909753+0000) 2024-03-06T09:10:12.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:12.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:12.909948+0000) 2024-03-06T09:10:13.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:13.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:13.910175+0000) 2024-03-06T09:10:14.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:14.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:14.910418+0000) 2024-03-06T09:10:15.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:15.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:15.910636+0000) 2024-03-06T09:10:16.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:16.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:16.910861+0000) 2024-03-06T09:10:17.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:17.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:17.911075+0000) 2024-03-06T09:10:18.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:18.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:18.911241+0000) 2024-03-06T09:10:19.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:19.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:19.911444+0000) 2024-03-06T09:10:20.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:20.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:20.911622+0000) 2024-03-06T09:10:21.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:21.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:21.911865+0000) 2024-03-06T09:10:22.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:22.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:22.912096+0000) 2024-03-06T09:10:23.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:23.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:23.912271+0000) 2024-03-06T09:10:24.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:24.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:24.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:24.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:24.912525+0000) 2024-03-06T09:10:25.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:25.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:25.912772+0000) 2024-03-06T09:10:26.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:26.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:26.912984+0000) 2024-03-06T09:10:27.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:27.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:27.913222+0000) 2024-03-06T09:10:28.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:28.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:28.913455+0000) 2024-03-06T09:10:29.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:29.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:29.913735+0000) 2024-03-06T09:10:30.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:30.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:30.913948+0000) 2024-03-06T09:10:31.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:31.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:31.914149+0000) 2024-03-06T09:10:32.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:32.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:32.914391+0000) 2024-03-06T09:10:33.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:33.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:33.914579+0000) 2024-03-06T09:10:34.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:34.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:34.914821+0000) 2024-03-06T09:10:35.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:35.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:35.915011+0000) 2024-03-06T09:10:36.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:36.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:36.915254+0000) 2024-03-06T09:10:37.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:37.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:37.915551+0000) 2024-03-06T09:10:38.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:38.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:38.915802+0000) 2024-03-06T09:10:39.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:39.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:39.915949+0000) 2024-03-06T09:10:40.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:40.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:40.916193+0000) 2024-03-06T09:10:41.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:41.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:41.916414+0000) 2024-03-06T09:10:42.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:42.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:42.916644+0000) 2024-03-06T09:10:43.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:43.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:43.916900+0000) 2024-03-06T09:10:44.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:44.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:44.917151+0000) 2024-03-06T09:10:45.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:45.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:45.917422+0000) 2024-03-06T09:10:46.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:46.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:46.917746+0000) 2024-03-06T09:10:47.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:47.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:47.918029+0000) 2024-03-06T09:10:48.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:48.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:48.918248+0000) 2024-03-06T09:10:49.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:49.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:49.918448+0000) 2024-03-06T09:10:50.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:50.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:50.918717+0000) 2024-03-06T09:10:51.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:51.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:51.918873+0000) 2024-03-06T09:10:52.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:52.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:52.919075+0000) 2024-03-06T09:10:53.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:53.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:53.919271+0000) 2024-03-06T09:10:54.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:54.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:54.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:54.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:54.919535+0000) 2024-03-06T09:10:55.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:55.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:55.919788+0000) 2024-03-06T09:10:56.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:56.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:56.920006+0000) 2024-03-06T09:10:57.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:57.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:57.920204+0000) 2024-03-06T09:10:58.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:58.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:58.920398+0000) 2024-03-06T09:10:59.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:10:59.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:59.920618+0000) 2024-03-06T09:11:00.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:00.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:00.920843+0000) 2024-03-06T09:11:01.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:01.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:01.921073+0000) 2024-03-06T09:11:02.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:02.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:02.921275+0000) 2024-03-06T09:11:03.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:03.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:03.921496+0000) 2024-03-06T09:11:04.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:04.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:04.921761+0000) 2024-03-06T09:11:05.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:05.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:05.921979+0000) 2024-03-06T09:11:06.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:06.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:06.922209+0000) 2024-03-06T09:11:07.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:07.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:07.922387+0000) 2024-03-06T09:11:08.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:08.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:08.922573+0000) 2024-03-06T09:11:09.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:09.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:09.922801+0000) 2024-03-06T09:11:10.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:10.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:10.923007+0000) 2024-03-06T09:11:11.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:11.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:11.923243+0000) 2024-03-06T09:11:12.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:12.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:12.923488+0000) 2024-03-06T09:11:13.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:13.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:13.923773+0000) 2024-03-06T09:11:14.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:14.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:14.924004+0000) 2024-03-06T09:11:15.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:15.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:15.924221+0000) 2024-03-06T09:11:16.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:16.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:16.924457+0000) 2024-03-06T09:11:17.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:17.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:17.924708+0000) 2024-03-06T09:11:18.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:18.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:18.924938+0000) 2024-03-06T09:11:19.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:19.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:19.925167+0000) 2024-03-06T09:11:20.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:20.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:20.925398+0000) 2024-03-06T09:11:21.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:21.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:21.925595+0000) 2024-03-06T09:11:22.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:22.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:22.925823+0000) 2024-03-06T09:11:23.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:23.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:23.926015+0000) 2024-03-06T09:11:24.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:24.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:24.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:24.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:24.926262+0000) 2024-03-06T09:11:25.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:25.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:25.926529+0000) 2024-03-06T09:11:26.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:26.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:26.926771+0000) 2024-03-06T09:11:27.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:27.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:27.926958+0000) 2024-03-06T09:11:28.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:28.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:28.927195+0000) 2024-03-06T09:11:29.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:29.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:29.927415+0000) 2024-03-06T09:11:30.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:30.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:30.927641+0000) 2024-03-06T09:11:31.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:31.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:31.927877+0000) 2024-03-06T09:11:32.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:32.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:32.928106+0000) 2024-03-06T09:11:33.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:33.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:33.928334+0000) 2024-03-06T09:11:34.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:34.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:34.928580+0000) 2024-03-06T09:11:35.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:35.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:35.928822+0000) 2024-03-06T09:11:36.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:36.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:36.929029+0000) 2024-03-06T09:11:37.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:37.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:37.929221+0000) 2024-03-06T09:11:38.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:38.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:38.929445+0000) 2024-03-06T09:11:39.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:39.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:39.929710+0000) 2024-03-06T09:11:40.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:40.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:40.929874+0000) 2024-03-06T09:11:41.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:41.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:41.930019+0000) 2024-03-06T09:11:42.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:42.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:42.930268+0000) 2024-03-06T09:11:43.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:43.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:43.930510+0000) 2024-03-06T09:11:44.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:44.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:44.930820+0000) 2024-03-06T09:11:45.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:45.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:45.931067+0000) 2024-03-06T09:11:46.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:46.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:46.931278+0000) 2024-03-06T09:11:47.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:47.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:47.931519+0000) 2024-03-06T09:11:48.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:48.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:48.931801+0000) 2024-03-06T09:11:49.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:49.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:49.932040+0000) 2024-03-06T09:11:50.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:50.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:50.932274+0000) 2024-03-06T09:11:51.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:51.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:51.932510+0000) 2024-03-06T09:11:52.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:52.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:52.932772+0000) 2024-03-06T09:11:53.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:53.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:53.932991+0000) 2024-03-06T09:11:54.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:54.743 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:54.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:54.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:54.933192+0000) 2024-03-06T09:11:55.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:55.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:55.933424+0000) 2024-03-06T09:11:56.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:56.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:56.933661+0000) 2024-03-06T09:11:57.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:57.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:57.933905+0000) 2024-03-06T09:11:58.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:58.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:58.934141+0000) 2024-03-06T09:11:59.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:11:59.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:59.934373+0000) 2024-03-06T09:12:00.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:00.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:00.934571+0000) 2024-03-06T09:12:01.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:01.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:01.934806+0000) 2024-03-06T09:12:02.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:02.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:02.935033+0000) 2024-03-06T09:12:03.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:03.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:03.935217+0000) 2024-03-06T09:12:04.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:04.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:04.935413+0000) 2024-03-06T09:12:05.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:05.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:05.935644+0000) 2024-03-06T09:12:06.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:06.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:06.935890+0000) 2024-03-06T09:12:07.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:07.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:07.936113+0000) 2024-03-06T09:12:08.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:08.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:08.936306+0000) 2024-03-06T09:12:09.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:09.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:09.936541+0000) 2024-03-06T09:12:10.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:10.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:10.936796+0000) 2024-03-06T09:12:11.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:11.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:11.936996+0000) 2024-03-06T09:12:12.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:12.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:12.937243+0000) 2024-03-06T09:12:13.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:13.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:13.937474+0000) 2024-03-06T09:12:14.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:14.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:14.937729+0000) 2024-03-06T09:12:15.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:15.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:15.937911+0000) 2024-03-06T09:12:16.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:16.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:16.938141+0000) 2024-03-06T09:12:17.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:17.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:17.938366+0000) 2024-03-06T09:12:18.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:18.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:18.938595+0000) 2024-03-06T09:12:19.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:19.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:19.938850+0000) 2024-03-06T09:12:20.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:20.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:20.939088+0000) 2024-03-06T09:12:21.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:21.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:21.939315+0000) 2024-03-06T09:12:22.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:22.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:22.939547+0000) 2024-03-06T09:12:23.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:23.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:23.939774+0000) 2024-03-06T09:12:24.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:24.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:24.939952+0000) 2024-03-06T09:12:24.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:24.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:25.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:25.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:25.940194+0000) 2024-03-06T09:12:26.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:26.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:26.940424+0000) 2024-03-06T09:12:27.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:27.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:27.940660+0000) 2024-03-06T09:12:28.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:28.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:28.940852+0000) 2024-03-06T09:12:29.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:29.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:29.941065+0000) 2024-03-06T09:12:30.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:30.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:30.941256+0000) 2024-03-06T09:12:31.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:31.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:31.941440+0000) 2024-03-06T09:12:32.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:32.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:32.941720+0000) 2024-03-06T09:12:33.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:33.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:33.941931+0000) 2024-03-06T09:12:34.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:34.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:34.942182+0000) 2024-03-06T09:12:35.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:35.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:35.942345+0000) 2024-03-06T09:12:36.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:36.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:36.942572+0000) 2024-03-06T09:12:37.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:37.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:37.942799+0000) 2024-03-06T09:12:38.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:38.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:38.943027+0000) 2024-03-06T09:12:39.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:39.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:39.943249+0000) 2024-03-06T09:12:40.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:40.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:40.943481+0000) 2024-03-06T09:12:41.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:41.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:41.943708+0000) 2024-03-06T09:12:42.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:42.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:42.943937+0000) 2024-03-06T09:12:43.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:43.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:43.944112+0000) 2024-03-06T09:12:44.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:44.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:44.944344+0000) 2024-03-06T09:12:45.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:45.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:45.944605+0000) 2024-03-06T09:12:46.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:46.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:46.944830+0000) 2024-03-06T09:12:47.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:47.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:47.944973+0000) 2024-03-06T09:12:48.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:48.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:48.945169+0000) 2024-03-06T09:12:49.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:49.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:49.945364+0000) 2024-03-06T09:12:50.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:50.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:50.945604+0000) 2024-03-06T09:12:51.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:51.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:51.945832+0000) 2024-03-06T09:12:52.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:52.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:52.946032+0000) 2024-03-06T09:12:53.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:53.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:53.946229+0000) 2024-03-06T09:12:54.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:54.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:54.946439+0000) 2024-03-06T09:12:55.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:55.178 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:55.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:55.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:55.946639+0000) 2024-03-06T09:12:56.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:56.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:56.946901+0000) 2024-03-06T09:12:57.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:57.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:57.947107+0000) 2024-03-06T09:12:58.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:58.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:58.947348+0000) 2024-03-06T09:12:59.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:12:59.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:59.947573+0000) 2024-03-06T09:13:00.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:00.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:00.947813+0000) 2024-03-06T09:13:01.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:01.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:01.948045+0000) 2024-03-06T09:13:02.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:02.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:02.948276+0000) 2024-03-06T09:13:03.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:03.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:03.948483+0000) 2024-03-06T09:13:04.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:04.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:04.948792+0000) 2024-03-06T09:13:05.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:05.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:05.949012+0000) 2024-03-06T09:13:06.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:06.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:06.949240+0000) 2024-03-06T09:13:07.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:07.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:07.949422+0000) 2024-03-06T09:13:08.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:08.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:08.949652+0000) 2024-03-06T09:13:09.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:09.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:09.949879+0000) 2024-03-06T09:13:10.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:10.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:10.950090+0000) 2024-03-06T09:13:11.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:11.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:11.950274+0000) 2024-03-06T09:13:12.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:12.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:12.950581+0000) 2024-03-06T09:13:13.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:13.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:13.950809+0000) 2024-03-06T09:13:14.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:14.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:14.951033+0000) 2024-03-06T09:13:15.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:15.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:15.951210+0000) 2024-03-06T09:13:16.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:16.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:16.951397+0000) 2024-03-06T09:13:17.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:17.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:17.951616+0000) 2024-03-06T09:13:18.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:18.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:18.951878+0000) 2024-03-06T09:13:19.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:19.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:19.952101+0000) 2024-03-06T09:13:20.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:20.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:20.952293+0000) 2024-03-06T09:13:21.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:21.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:21.952488+0000) 2024-03-06T09:13:22.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:22.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:22.952768+0000) 2024-03-06T09:13:23.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:23.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:23.953000+0000) 2024-03-06T09:13:24.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:24.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:24.953237+0000) 2024-03-06T09:13:25.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:25.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:25.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:25.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:25.953463+0000) 2024-03-06T09:13:26.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:26.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:26.953734+0000) 2024-03-06T09:13:27.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:27.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:27.953946+0000) 2024-03-06T09:13:28.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:28.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:28.954137+0000) 2024-03-06T09:13:29.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:29.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:29.954328+0000) 2024-03-06T09:13:30.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:30.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:30.954514+0000) 2024-03-06T09:13:31.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:31.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:31.954753+0000) 2024-03-06T09:13:32.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:32.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:32.955003+0000) 2024-03-06T09:13:33.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:33.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:33.955190+0000) 2024-03-06T09:13:34.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:34.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:34.955418+0000) 2024-03-06T09:13:35.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:35.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:35.955640+0000) 2024-03-06T09:13:36.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:36.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:36.955903+0000) 2024-03-06T09:13:37.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:37.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:37.956127+0000) 2024-03-06T09:13:38.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:38.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:38.956346+0000) 2024-03-06T09:13:39.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:39.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:39.956559+0000) 2024-03-06T09:13:40.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:40.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:40.956802+0000) 2024-03-06T09:13:41.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:41.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:41.957028+0000) 2024-03-06T09:13:42.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:42.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:42.957268+0000) 2024-03-06T09:13:43.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:43.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:43.957457+0000) 2024-03-06T09:13:44.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:44.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:44.957732+0000) 2024-03-06T09:13:46.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:45.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:45.957947+0000) 2024-03-06T09:13:46.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:46.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:46.958169+0000) 2024-03-06T09:13:47.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:47.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:47.958404+0000) 2024-03-06T09:13:48.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:48.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:48.958592+0000) 2024-03-06T09:13:49.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:49.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:49.958815+0000) 2024-03-06T09:13:50.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:50.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:50.959043+0000) 2024-03-06T09:13:51.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:51.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:51.959220+0000) 2024-03-06T09:13:52.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:52.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:52.959398+0000) 2024-03-06T09:13:53.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:53.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:53.959568+0000) 2024-03-06T09:13:54.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:54.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:54.959836+0000) 2024-03-06T09:13:55.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:55.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:55.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:55.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:55.960004+0000) 2024-03-06T09:13:56.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:56.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:56.960256+0000) 2024-03-06T09:13:57.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:57.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:57.960552+0000) 2024-03-06T09:13:58.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:58.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:58.960808+0000) 2024-03-06T09:13:59.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:13:59.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:59.961083+0000) 2024-03-06T09:14:00.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:00.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:00.961360+0000) 2024-03-06T09:14:01.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:01.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:01.961640+0000) 2024-03-06T09:14:02.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:02.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:02.961986+0000) 2024-03-06T09:14:03.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:03.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:03.962184+0000) 2024-03-06T09:14:04.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:04.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:04.962414+0000) 2024-03-06T09:14:05.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:05.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:05.962631+0000) 2024-03-06T09:14:06.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:06.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:06.962859+0000) 2024-03-06T09:14:07.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:07.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:07.963099+0000) 2024-03-06T09:14:08.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:08.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:08.963318+0000) 2024-03-06T09:14:09.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:09.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:09.963546+0000) 2024-03-06T09:14:10.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:10.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:10.963747+0000) 2024-03-06T09:14:11.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:11.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:11.964015+0000) 2024-03-06T09:14:12.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:12.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:12.964243+0000) 2024-03-06T09:14:13.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:13.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:13.964469+0000) 2024-03-06T09:14:14.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:14.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:14.964723+0000) 2024-03-06T09:14:15.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:15.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:15.964912+0000) 2024-03-06T09:14:16.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:16.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:16.965134+0000) 2024-03-06T09:14:17.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:17.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:17.965380+0000) 2024-03-06T09:14:18.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:18.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:18.965608+0000) 2024-03-06T09:14:19.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:19.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:19.965836+0000) 2024-03-06T09:14:20.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:20.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:20.966023+0000) 2024-03-06T09:14:21.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:21.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:21.966221+0000) 2024-03-06T09:14:22.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:22.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:22.966405+0000) 2024-03-06T09:14:23.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:23.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:23.966627+0000) 2024-03-06T09:14:24.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:24.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:24.966837+0000) 2024-03-06T09:14:25.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:25.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:25.967113+0000) 2024-03-06T09:14:26.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:26.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:26.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:26.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:26.967395+0000) 2024-03-06T09:14:27.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:27.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:27.967730+0000) 2024-03-06T09:14:28.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:28.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:28.967962+0000) 2024-03-06T09:14:29.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:29.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:29.968201+0000) 2024-03-06T09:14:30.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:30.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:30.968418+0000) 2024-03-06T09:14:31.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:31.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:31.968635+0000) 2024-03-06T09:14:32.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:32.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:32.968877+0000) 2024-03-06T09:14:33.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:33.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:33.969097+0000) 2024-03-06T09:14:34.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:34.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:34.969283+0000) 2024-03-06T09:14:35.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:35.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:35.969553+0000) 2024-03-06T09:14:36.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:36.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:36.969828+0000) 2024-03-06T09:14:37.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:37.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:37.970044+0000) 2024-03-06T09:14:38.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:38.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:38.970273+0000) 2024-03-06T09:14:39.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:39.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:39.970497+0000) 2024-03-06T09:14:40.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:40.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:40.970749+0000) 2024-03-06T09:14:41.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:41.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:41.970944+0000) 2024-03-06T09:14:42.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:42.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:42.971213+0000) 2024-03-06T09:14:43.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:43.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:43.971459+0000) 2024-03-06T09:14:44.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:44.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:44.971752+0000) 2024-03-06T09:14:45.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:45.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:45.971937+0000) 2024-03-06T09:14:46.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:46.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:46.972139+0000) 2024-03-06T09:14:47.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:47.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:47.972401+0000) 2024-03-06T09:14:48.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:48.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:48.972635+0000) 2024-03-06T09:14:49.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:49.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:49.972861+0000) 2024-03-06T09:14:50.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:50.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:50.973090+0000) 2024-03-06T09:14:51.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:51.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:51.973313+0000) 2024-03-06T09:14:52.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:52.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:52.973565+0000) 2024-03-06T09:14:53.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:53.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:53.973879+0000) 2024-03-06T09:14:54.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:54.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:54.974129+0000) 2024-03-06T09:14:55.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:55.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:55.974377+0000) 2024-03-06T09:14:56.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:56.346 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:56.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:56.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:56.974623+0000) 2024-03-06T09:14:57.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:57.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:57.974887+0000) 2024-03-06T09:14:58.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:58.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:58.975172+0000) 2024-03-06T09:14:59.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:14:59.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:59.975439+0000) 2024-03-06T09:15:00.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:00.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:00.975761+0000) 2024-03-06T09:15:01.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:01.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:01.976006+0000) 2024-03-06T09:15:02.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:02.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:02.976293+0000) 2024-03-06T09:15:03.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:03.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:03.976575+0000) 2024-03-06T09:15:04.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:04.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:04.976817+0000) 2024-03-06T09:15:05.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:05.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:05.977055+0000) 2024-03-06T09:15:06.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:06.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:06.977332+0000) 2024-03-06T09:15:07.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:07.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:07.977610+0000) 2024-03-06T09:15:08.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:08.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:08.977894+0000) 2024-03-06T09:15:09.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:09.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:09.978129+0000) 2024-03-06T09:15:10.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:10.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:10.978351+0000) 2024-03-06T09:15:11.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:11.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:11.978567+0000) 2024-03-06T09:15:12.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:12.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:12.978822+0000) 2024-03-06T09:15:13.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:13.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:13.979046+0000) 2024-03-06T09:15:14.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:14.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:14.979285+0000) 2024-03-06T09:15:15.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:15.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:15.979469+0000) 2024-03-06T09:15:16.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:16.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:16.979691+0000) 2024-03-06T09:15:17.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:17.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:17.979928+0000) 2024-03-06T09:15:18.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:18.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:18.980140+0000) 2024-03-06T09:15:19.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:19.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:19.980332+0000) 2024-03-06T09:15:20.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:20.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:20.980576+0000) 2024-03-06T09:15:21.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:21.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:21.980829+0000) 2024-03-06T09:15:22.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:22.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:22.981066+0000) 2024-03-06T09:15:23.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:23.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:23.981306+0000) 2024-03-06T09:15:24.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:24.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:24.981538+0000) 2024-03-06T09:15:25.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:25.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:25.981808+0000) 2024-03-06T09:15:26.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:26.552 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:26.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:26.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:26.982011+0000) 2024-03-06T09:15:27.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:27.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:27.982198+0000) 2024-03-06T09:15:28.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:28.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:28.982413+0000) 2024-03-06T09:15:29.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:29.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:29.982634+0000) 2024-03-06T09:15:30.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:30.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:30.982865+0000) 2024-03-06T09:15:31.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:31.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:31.983067+0000) 2024-03-06T09:15:32.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:32.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:32.983304+0000) 2024-03-06T09:15:33.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:33.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:33.983539+0000) 2024-03-06T09:15:34.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:34.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:34.983792+0000) 2024-03-06T09:15:35.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:35.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:35.983995+0000) 2024-03-06T09:15:36.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:36.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:36.984201+0000) 2024-03-06T09:15:37.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:37.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:37.984399+0000) 2024-03-06T09:15:38.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:38.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:38.984636+0000) 2024-03-06T09:15:39.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:39.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:39.984866+0000) 2024-03-06T09:15:40.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:40.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:40.985091+0000) 2024-03-06T09:15:41.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:41.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:41.985385+0000) 2024-03-06T09:15:42.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:42.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:42.985582+0000) 2024-03-06T09:15:43.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:43.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:43.985781+0000) 2024-03-06T09:15:44.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:44.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:44.986032+0000) 2024-03-06T09:15:45.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:45.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:45.986257+0000) 2024-03-06T09:15:46.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:46.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:46.986502+0000) 2024-03-06T09:15:47.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:47.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:47.986810+0000) 2024-03-06T09:15:48.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:48.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:48.987083+0000) 2024-03-06T09:15:49.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:49.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:49.987295+0000) 2024-03-06T09:15:50.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:50.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:50.987521+0000) 2024-03-06T09:15:51.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:51.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:51.987775+0000) 2024-03-06T09:15:52.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:52.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:52.988024+0000) 2024-03-06T09:15:53.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:53.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:53.988203+0000) 2024-03-06T09:15:54.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:54.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:54.988458+0000) 2024-03-06T09:15:55.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:55.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:55.988731+0000) 2024-03-06T09:15:56.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:56.762 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:56.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:56.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:56.989004+0000) 2024-03-06T09:15:57.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:57.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:57.989238+0000) 2024-03-06T09:15:58.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:58.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:58.989486+0000) 2024-03-06T09:15:59.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:15:59.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:59.989756+0000) 2024-03-06T09:16:00.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:00.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:00.989985+0000) 2024-03-06T09:16:01.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:01.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:01.990220+0000) 2024-03-06T09:16:02.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:02.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:02.990433+0000) 2024-03-06T09:16:03.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:03.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:03.990707+0000) 2024-03-06T09:16:04.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:04.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:04.990948+0000) 2024-03-06T09:16:05.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:05.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:05.991170+0000) 2024-03-06T09:16:06.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:06.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:06.991394+0000) 2024-03-06T09:16:07.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:07.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:07.991577+0000) 2024-03-06T09:16:08.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:08.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:08.991809+0000) 2024-03-06T09:16:09.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:09.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:09.992030+0000) 2024-03-06T09:16:10.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:10.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:10.992215+0000) 2024-03-06T09:16:11.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:11.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:11.992457+0000) 2024-03-06T09:16:12.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:12.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:12.992723+0000) 2024-03-06T09:16:13.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:13.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:13.992918+0000) 2024-03-06T09:16:14.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:14.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:14.993151+0000) 2024-03-06T09:16:15.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:15.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:15.993370+0000) 2024-03-06T09:16:16.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:16.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:16.993558+0000) 2024-03-06T09:16:17.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:17.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:17.993773+0000) 2024-03-06T09:16:18.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:18.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:18.993956+0000) 2024-03-06T09:16:19.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:19.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:19.994179+0000) 2024-03-06T09:16:20.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:20.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:20.994369+0000) 2024-03-06T09:16:21.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:21.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:21.994606+0000) 2024-03-06T09:16:22.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:22.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:22.994819+0000) 2024-03-06T09:16:23.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:23.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:23.995029+0000) 2024-03-06T09:16:24.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:24.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:24.995225+0000) 2024-03-06T09:16:25.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:25.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:25.995437+0000) 2024-03-06T09:16:26.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:26.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:26.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:26.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:26.995648+0000) 2024-03-06T09:16:27.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:27.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:27.995841+0000) 2024-03-06T09:16:28.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:28.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:28.996048+0000) 2024-03-06T09:16:29.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:29.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:29.996282+0000) 2024-03-06T09:16:30.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:30.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:30.996466+0000) 2024-03-06T09:16:31.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:31.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:31.996734+0000) 2024-03-06T09:16:32.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:32.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:32.996969+0000) 2024-03-06T09:16:33.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:33.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:33.997186+0000) 2024-03-06T09:16:34.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:34.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:34.997352+0000) 2024-03-06T09:16:35.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:35.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:35.997567+0000) 2024-03-06T09:16:36.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:36.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:36.997796+0000) 2024-03-06T09:16:37.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:37.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:37.997982+0000) 2024-03-06T09:16:38.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:38.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:38.998217+0000) 2024-03-06T09:16:39.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:39.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:39.998438+0000) 2024-03-06T09:16:40.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:40.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:40.998748+0000) 2024-03-06T09:16:41.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:41.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:41.998924+0000) 2024-03-06T09:16:42.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:42.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:42.999147+0000) 2024-03-06T09:16:43.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:43.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:43.999321+0000) 2024-03-06T09:16:44.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:44.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:44.999536+0000) 2024-03-06T09:16:45.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:45.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:45.999769+0000) 2024-03-06T09:16:46.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:46.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:46.999995+0000) 2024-03-06T09:16:47.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:47.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:48.000214+0000) 2024-03-06T09:16:48.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:48.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:49.000439+0000) 2024-03-06T09:16:49.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:49.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:50.000610+0000) 2024-03-06T09:16:50.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:51.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:51.000750+0000) 2024-03-06T09:16:51.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:52.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:52.000905+0000) 2024-03-06T09:16:52.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:53.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:53.001129+0000) 2024-03-06T09:16:54.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:54.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:54.001361+0000) 2024-03-06T09:16:55.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:55.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:55.001605+0000) 2024-03-06T09:16:56.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:56.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:56.001828+0000) 2024-03-06T09:16:57.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:57.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:57.002023+0000) 2024-03-06T09:16:57.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:57.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:58.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:58.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:58.002261+0000) 2024-03-06T09:16:59.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:16:59.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:59.002493+0000) 2024-03-06T09:17:00.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:00.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:00.002708+0000) 2024-03-06T09:17:01.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:01.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:01.002917+0000) 2024-03-06T09:17:02.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:02.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:02.003143+0000) 2024-03-06T09:17:03.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:03.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:03.003373+0000) 2024-03-06T09:17:04.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:04.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:04.003601+0000) 2024-03-06T09:17:05.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:05.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:05.003831+0000) 2024-03-06T09:17:06.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:06.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:06.004014+0000) 2024-03-06T09:17:07.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:07.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:07.004218+0000) 2024-03-06T09:17:08.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:08.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:08.004400+0000) 2024-03-06T09:17:09.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:09.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:09.004648+0000) 2024-03-06T09:17:10.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:10.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:10.004912+0000) 2024-03-06T09:17:11.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:11.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:11.005139+0000) 2024-03-06T09:17:12.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:12.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:12.005366+0000) 2024-03-06T09:17:13.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:13.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:13.005595+0000) 2024-03-06T09:17:14.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:14.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:14.005833+0000) 2024-03-06T09:17:15.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:15.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:15.006070+0000) 2024-03-06T09:17:16.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:16.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:16.006302+0000) 2024-03-06T09:17:17.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:17.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:17.006550+0000) 2024-03-06T09:17:18.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:18.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:18.006819+0000) 2024-03-06T09:17:19.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:19.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:19.007016+0000) 2024-03-06T09:17:20.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:20.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:20.007239+0000) 2024-03-06T09:17:21.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:21.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:21.007484+0000) 2024-03-06T09:17:22.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:22.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:22.007753+0000) 2024-03-06T09:17:23.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:23.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:23.007981+0000) 2024-03-06T09:17:24.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:24.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:24.008208+0000) 2024-03-06T09:17:25.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:25.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:25.008441+0000) 2024-03-06T09:17:26.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:26.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:26.008614+0000) 2024-03-06T09:17:27.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:27.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:27.008801+0000) 2024-03-06T09:17:27.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:27.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:28.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:28.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:28.008985+0000) 2024-03-06T09:17:29.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:29.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:29.009277+0000) 2024-03-06T09:17:30.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:30.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:30.009468+0000) 2024-03-06T09:17:31.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:31.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:31.009690+0000) 2024-03-06T09:17:32.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:32.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:32.009928+0000) 2024-03-06T09:17:33.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:33.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:33.010152+0000) 2024-03-06T09:17:34.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:34.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:34.010341+0000) 2024-03-06T09:17:35.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:35.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:35.010594+0000) 2024-03-06T09:17:36.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:36.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:36.010771+0000) 2024-03-06T09:17:37.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:37.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:37.011019+0000) 2024-03-06T09:17:38.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:38.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:38.011206+0000) 2024-03-06T09:17:39.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:39.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:39.011424+0000) 2024-03-06T09:17:40.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:40.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:40.011748+0000) 2024-03-06T09:17:41.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:41.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:41.011995+0000) 2024-03-06T09:17:42.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:42.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:42.012171+0000) 2024-03-06T09:17:43.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:43.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:43.012361+0000) 2024-03-06T09:17:44.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:44.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:44.012600+0000) 2024-03-06T09:17:45.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:45.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:45.012838+0000) 2024-03-06T09:17:46.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:46.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:46.013052+0000) 2024-03-06T09:17:47.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:47.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:47.013266+0000) 2024-03-06T09:17:48.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:48.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:48.013489+0000) 2024-03-06T09:17:49.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:49.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:49.013758+0000) 2024-03-06T09:17:50.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:50.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:50.013984+0000) 2024-03-06T09:17:51.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:51.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:51.014208+0000) 2024-03-06T09:17:52.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:52.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:52.014436+0000) 2024-03-06T09:17:53.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:53.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:53.014656+0000) 2024-03-06T09:17:54.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:54.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:54.014961+0000) 2024-03-06T09:17:55.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:55.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:55.015196+0000) 2024-03-06T09:17:56.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:56.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:56.015336+0000) 2024-03-06T09:17:57.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:57.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:57.015504+0000) 2024-03-06T09:17:57.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:57.530 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:58.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:58.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:58.015776+0000) 2024-03-06T09:17:59.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:17:59.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:59.016002+0000) 2024-03-06T09:18:00.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:00.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:00.016226+0000) 2024-03-06T09:18:01.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:01.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:01.016459+0000) 2024-03-06T09:18:02.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:02.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:02.016720+0000) 2024-03-06T09:18:03.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:03.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:03.017006+0000) 2024-03-06T09:18:04.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:04.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:04.017291+0000) 2024-03-06T09:18:05.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:05.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:05.017582+0000) 2024-03-06T09:18:06.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:06.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:06.017873+0000) 2024-03-06T09:18:07.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:07.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:07.018151+0000) 2024-03-06T09:18:08.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:08.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:08.018417+0000) 2024-03-06T09:18:09.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:09.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:09.018740+0000) 2024-03-06T09:18:10.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:10.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:10.018944+0000) 2024-03-06T09:18:11.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:11.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:11.019217+0000) 2024-03-06T09:18:12.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:12.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:12.019502+0000) 2024-03-06T09:18:13.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:13.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:13.019825+0000) 2024-03-06T09:18:14.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:14.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:14.020032+0000) 2024-03-06T09:18:15.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:15.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:15.020221+0000) 2024-03-06T09:18:16.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:16.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:16.020449+0000) 2024-03-06T09:18:17.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:17.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:17.020726+0000) 2024-03-06T09:18:18.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:18.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:18.020960+0000) 2024-03-06T09:18:19.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:19.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:19.021170+0000) 2024-03-06T09:18:20.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:20.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:20.021360+0000) 2024-03-06T09:18:21.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:21.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:21.021624+0000) 2024-03-06T09:18:22.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:22.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:22.021941+0000) 2024-03-06T09:18:23.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:23.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:23.022153+0000) 2024-03-06T09:18:24.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:24.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:24.022377+0000) 2024-03-06T09:18:25.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:25.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:25.022613+0000) 2024-03-06T09:18:26.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:26.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:26.022859+0000) 2024-03-06T09:18:27.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:27.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:27.023104+0000) 2024-03-06T09:18:27.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:27.708 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:28.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:28.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:28.023356+0000) 2024-03-06T09:18:29.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:29.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:29.023593+0000) 2024-03-06T09:18:30.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:30.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:30.023833+0000) 2024-03-06T09:18:31.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:31.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:31.024024+0000) 2024-03-06T09:18:32.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:32.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:32.024216+0000) 2024-03-06T09:18:33.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:33.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:33.024460+0000) 2024-03-06T09:18:34.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:34.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:34.024717+0000) 2024-03-06T09:18:35.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:35.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:35.025023+0000) 2024-03-06T09:18:36.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:36.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:36.025296+0000) 2024-03-06T09:18:37.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:37.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:37.025576+0000) 2024-03-06T09:18:38.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:38.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:38.025850+0000) 2024-03-06T09:18:39.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:39.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:39.026050+0000) 2024-03-06T09:18:40.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:40.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:40.026259+0000) 2024-03-06T09:18:41.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:41.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:41.026519+0000) 2024-03-06T09:18:42.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:42.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:42.026795+0000) 2024-03-06T09:18:43.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:43.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:43.026982+0000) 2024-03-06T09:18:44.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:44.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:44.027165+0000) 2024-03-06T09:18:45.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:45.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:45.027442+0000) 2024-03-06T09:18:46.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:46.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:46.027759+0000) 2024-03-06T09:18:47.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:47.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:47.027994+0000) 2024-03-06T09:18:48.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:48.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:48.028191+0000) 2024-03-06T09:18:49.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:49.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:49.028409+0000) 2024-03-06T09:18:50.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:50.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:50.028635+0000) 2024-03-06T09:18:51.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:51.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:51.028895+0000) 2024-03-06T09:18:52.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:52.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:52.029133+0000) 2024-03-06T09:18:53.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:53.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:53.029357+0000) 2024-03-06T09:18:54.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:54.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:54.029567+0000) 2024-03-06T09:18:55.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:55.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:55.029819+0000) 2024-03-06T09:18:56.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:56.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:56.030035+0000) 2024-03-06T09:18:57.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:57.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:57.030304+0000) 2024-03-06T09:18:57.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:57.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:58.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:58.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:58.030507+0000) 2024-03-06T09:18:59.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:18:59.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:59.030808+0000) 2024-03-06T09:19:00.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:00.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:00.031049+0000) 2024-03-06T09:19:01.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:01.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:01.031254+0000) 2024-03-06T09:19:02.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:02.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:02.031492+0000) 2024-03-06T09:19:03.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:03.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:03.031710+0000) 2024-03-06T09:19:04.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:04.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:04.031946+0000) 2024-03-06T09:19:05.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:05.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:05.032136+0000) 2024-03-06T09:19:06.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:06.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:06.032329+0000) 2024-03-06T09:19:07.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:07.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:07.032527+0000) 2024-03-06T09:19:08.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:08.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:08.032771+0000) 2024-03-06T09:19:09.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:09.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:09.032963+0000) 2024-03-06T09:19:10.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:10.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:10.033234+0000) 2024-03-06T09:19:11.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:11.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:11.033431+0000) 2024-03-06T09:19:12.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:12.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:12.033697+0000) 2024-03-06T09:19:13.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:13.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:13.033908+0000) 2024-03-06T09:19:14.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:14.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:14.034152+0000) 2024-03-06T09:19:15.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:15.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:15.034402+0000) 2024-03-06T09:19:16.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:16.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:16.034643+0000) 2024-03-06T09:19:17.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:17.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:17.034911+0000) 2024-03-06T09:19:18.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:18.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:18.035162+0000) 2024-03-06T09:19:19.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:19.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:19.035339+0000) 2024-03-06T09:19:20.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:20.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:20.035563+0000) 2024-03-06T09:19:21.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:21.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:21.035792+0000) 2024-03-06T09:19:22.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:22.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:22.035983+0000) 2024-03-06T09:19:23.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:23.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:23.036227+0000) 2024-03-06T09:19:24.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:24.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:24.036475+0000) 2024-03-06T09:19:25.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:25.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:25.036741+0000) 2024-03-06T09:19:26.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:26.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:26.036974+0000) 2024-03-06T09:19:27.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:27.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:27.037212+0000) 2024-03-06T09:19:28.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:28.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:28.037446+0000) 2024-03-06T09:19:28.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:28.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:29.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:29.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:29.037730+0000) 2024-03-06T09:19:30.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:30.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:30.037928+0000) 2024-03-06T09:19:31.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:31.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:31.038172+0000) 2024-03-06T09:19:32.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:32.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:32.038415+0000) 2024-03-06T09:19:33.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:33.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:33.038658+0000) 2024-03-06T09:19:34.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:34.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:34.038970+0000) 2024-03-06T09:19:35.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:35.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:35.039218+0000) 2024-03-06T09:19:36.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:36.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:36.039455+0000) 2024-03-06T09:19:37.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:37.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:37.039741+0000) 2024-03-06T09:19:38.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:38.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:38.039973+0000) 2024-03-06T09:19:39.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:39.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:39.040224+0000) 2024-03-06T09:19:40.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:40.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:40.040427+0000) 2024-03-06T09:19:41.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:41.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:41.040705+0000) 2024-03-06T09:19:42.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:42.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:42.040943+0000) 2024-03-06T09:19:43.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:43.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:43.041164+0000) 2024-03-06T09:19:44.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:44.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:44.041397+0000) 2024-03-06T09:19:45.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:45.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:45.041608+0000) 2024-03-06T09:19:46.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:46.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:46.041849+0000) 2024-03-06T09:19:47.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:47.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:47.042051+0000) 2024-03-06T09:19:48.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:48.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:48.042236+0000) 2024-03-06T09:19:49.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:49.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:49.042474+0000) 2024-03-06T09:19:50.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:50.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:50.042733+0000) 2024-03-06T09:19:51.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:51.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:51.042988+0000) 2024-03-06T09:19:52.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:52.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:52.043202+0000) 2024-03-06T09:19:53.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:53.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:53.043394+0000) 2024-03-06T09:19:54.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:54.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:54.043628+0000) 2024-03-06T09:19:55.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:55.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:55.043905+0000) 2024-03-06T09:19:56.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:56.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:56.044092+0000) 2024-03-06T09:19:57.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:57.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:57.044285+0000) 2024-03-06T09:19:58.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:58.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:58.044520+0000) 2024-03-06T09:19:58.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:58.354 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:59.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:19:59.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:59.044803+0000) 2024-03-06T09:20:00.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:00.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:00.045005+0000) 2024-03-06T09:20:01.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:01.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:01.045202+0000) 2024-03-06T09:20:02.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:02.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:02.045437+0000) 2024-03-06T09:20:03.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:03.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:03.045690+0000) 2024-03-06T09:20:04.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:04.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:04.045899+0000) 2024-03-06T09:20:05.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:05.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:05.046139+0000) 2024-03-06T09:20:06.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:06.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:06.046357+0000) 2024-03-06T09:20:07.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:07.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:07.046586+0000) 2024-03-06T09:20:08.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:08.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:08.046827+0000) 2024-03-06T09:20:09.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:09.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:09.047135+0000) 2024-03-06T09:20:10.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:10.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:10.047330+0000) 2024-03-06T09:20:11.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:11.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:11.047559+0000) 2024-03-06T09:20:12.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:12.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:12.047774+0000) 2024-03-06T09:20:13.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:13.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:13.048022+0000) 2024-03-06T09:20:14.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:14.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:14.048271+0000) 2024-03-06T09:20:15.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:15.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:15.048510+0000) 2024-03-06T09:20:16.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:16.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:16.048773+0000) 2024-03-06T09:20:17.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:17.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:17.049004+0000) 2024-03-06T09:20:18.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:18.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:18.049229+0000) 2024-03-06T09:20:19.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:19.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:19.049470+0000) 2024-03-06T09:20:20.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:20.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:20.049660+0000) 2024-03-06T09:20:21.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:21.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:21.049932+0000) 2024-03-06T09:20:22.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:22.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:22.050156+0000) 2024-03-06T09:20:23.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:23.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:23.050339+0000) 2024-03-06T09:20:24.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:24.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:24.050564+0000) 2024-03-06T09:20:25.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:25.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:25.050820+0000) 2024-03-06T09:20:26.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:26.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:26.051041+0000) 2024-03-06T09:20:27.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:27.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:27.051227+0000) 2024-03-06T09:20:28.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:28.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:28.051460+0000) 2024-03-06T09:20:28.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:28.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:29.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:29.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:29.051762+0000) 2024-03-06T09:20:30.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:30.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:30.051958+0000) 2024-03-06T09:20:31.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:31.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:31.052128+0000) 2024-03-06T09:20:32.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:32.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:32.052317+0000) 2024-03-06T09:20:33.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:33.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:33.052542+0000) 2024-03-06T09:20:34.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:34.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:34.052825+0000) 2024-03-06T09:20:35.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:35.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:35.053064+0000) 2024-03-06T09:20:36.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:36.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:36.053294+0000) 2024-03-06T09:20:37.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:37.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:37.053523+0000) 2024-03-06T09:20:38.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:38.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:38.053772+0000) 2024-03-06T09:20:39.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:39.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:39.054013+0000) 2024-03-06T09:20:40.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:40.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:40.054234+0000) 2024-03-06T09:20:41.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:41.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:41.054424+0000) 2024-03-06T09:20:42.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:42.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:42.054651+0000) 2024-03-06T09:20:43.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:43.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:43.054919+0000) 2024-03-06T09:20:44.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:44.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:44.055163+0000) 2024-03-06T09:20:45.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:45.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:45.055421+0000) 2024-03-06T09:20:46.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:46.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:46.055702+0000) 2024-03-06T09:20:47.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:47.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:47.055927+0000) 2024-03-06T09:20:48.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:48.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:48.056159+0000) 2024-03-06T09:20:49.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:49.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:49.056388+0000) 2024-03-06T09:20:50.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:50.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:50.056616+0000) 2024-03-06T09:20:51.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:51.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:51.056815+0000) 2024-03-06T09:20:52.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:52.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:52.057060+0000) 2024-03-06T09:20:53.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:53.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:53.057289+0000) 2024-03-06T09:20:54.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:54.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:54.057443+0000) 2024-03-06T09:20:55.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:55.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:55.057699+0000) 2024-03-06T09:20:56.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:56.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:56.057884+0000) 2024-03-06T09:20:57.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:57.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:57.058170+0000) 2024-03-06T09:20:58.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:58.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:58.058349+0000) 2024-03-06T09:20:58.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:58.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:59.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:20:59.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:59.058533+0000) 2024-03-06T09:21:00.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:00.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:00.058788+0000) 2024-03-06T09:21:01.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:01.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:01.059000+0000) 2024-03-06T09:21:02.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:02.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:02.059264+0000) 2024-03-06T09:21:03.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:03.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:03.059459+0000) 2024-03-06T09:21:04.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:04.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:04.059720+0000) 2024-03-06T09:21:05.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:05.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:05.059927+0000) 2024-03-06T09:21:06.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:06.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:06.060158+0000) 2024-03-06T09:21:07.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:07.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:07.060384+0000) 2024-03-06T09:21:08.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:08.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:08.060603+0000) 2024-03-06T09:21:09.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:09.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:09.060845+0000) 2024-03-06T09:21:10.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:10.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:10.061076+0000) 2024-03-06T09:21:11.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:11.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:11.061275+0000) 2024-03-06T09:21:12.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:12.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:12.061474+0000) 2024-03-06T09:21:13.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:13.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:13.061719+0000) 2024-03-06T09:21:14.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:14.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:14.061962+0000) 2024-03-06T09:21:15.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:15.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:15.062176+0000) 2024-03-06T09:21:16.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:16.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:16.062417+0000) 2024-03-06T09:21:17.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:17.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:17.062645+0000) 2024-03-06T09:21:18.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:18.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:18.062897+0000) 2024-03-06T09:21:19.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:19.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:19.063087+0000) 2024-03-06T09:21:20.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:20.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:20.063279+0000) 2024-03-06T09:21:21.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:21.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:21.063479+0000) 2024-03-06T09:21:22.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:22.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:22.063716+0000) 2024-03-06T09:21:23.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:23.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:23.063910+0000) 2024-03-06T09:21:24.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:24.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:24.064094+0000) 2024-03-06T09:21:25.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:25.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:25.064336+0000) 2024-03-06T09:21:26.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:26.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:26.064559+0000) 2024-03-06T09:21:27.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:27.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:27.064798+0000) 2024-03-06T09:21:28.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:28.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:28.065037+0000) 2024-03-06T09:21:29.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:29.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:29.065285+0000) 2024-03-06T09:21:29.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:29.122 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:30.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:30.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:30.065524+0000) 2024-03-06T09:21:31.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:31.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:31.065795+0000) 2024-03-06T09:21:32.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:32.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:32.066036+0000) 2024-03-06T09:21:33.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:33.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:33.066257+0000) 2024-03-06T09:21:34.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:34.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:34.066512+0000) 2024-03-06T09:21:35.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:35.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:35.066780+0000) 2024-03-06T09:21:36.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:36.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:36.066988+0000) 2024-03-06T09:21:37.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:37.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:37.067227+0000) 2024-03-06T09:21:38.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:38.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:38.067435+0000) 2024-03-06T09:21:39.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:39.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:39.067694+0000) 2024-03-06T09:21:40.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:40.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:40.067876+0000) 2024-03-06T09:21:41.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:41.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:41.068079+0000) 2024-03-06T09:21:42.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:42.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:42.068308+0000) 2024-03-06T09:21:43.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:43.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:43.068506+0000) 2024-03-06T09:21:44.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:44.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:44.068770+0000) 2024-03-06T09:21:45.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:45.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:45.069059+0000) 2024-03-06T09:21:46.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:46.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:46.069284+0000) 2024-03-06T09:21:47.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:47.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:47.069513+0000) 2024-03-06T09:21:48.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:48.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:48.069771+0000) 2024-03-06T09:21:49.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:49.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:49.069948+0000) 2024-03-06T09:21:50.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:50.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:50.070181+0000) 2024-03-06T09:21:51.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:51.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:51.070399+0000) 2024-03-06T09:21:52.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:52.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:52.070629+0000) 2024-03-06T09:21:53.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:53.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:53.070854+0000) 2024-03-06T09:21:54.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:54.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:54.071199+0000) 2024-03-06T09:21:55.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:55.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:55.071433+0000) 2024-03-06T09:21:56.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:56.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:56.071655+0000) 2024-03-06T09:21:57.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:57.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:57.071989+0000) 2024-03-06T09:21:58.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:58.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:58.072178+0000) 2024-03-06T09:21:59.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:21:59.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:59.072406+0000) 2024-03-06T09:21:59.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:59.330 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:00.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:00.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:00.072631+0000) 2024-03-06T09:22:01.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:01.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:01.072825+0000) 2024-03-06T09:22:02.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:02.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:02.073048+0000) 2024-03-06T09:22:03.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:03.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:03.073242+0000) 2024-03-06T09:22:04.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:04.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:04.073478+0000) 2024-03-06T09:22:05.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:05.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:05.073761+0000) 2024-03-06T09:22:06.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:06.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:06.074023+0000) 2024-03-06T09:22:07.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:07.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:07.074304+0000) 2024-03-06T09:22:08.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:08.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:08.074582+0000) 2024-03-06T09:22:09.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:09.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:09.074851+0000) 2024-03-06T09:22:10.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:10.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:10.075077+0000) 2024-03-06T09:22:11.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:11.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:11.075298+0000) 2024-03-06T09:22:12.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:12.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:12.075547+0000) 2024-03-06T09:22:13.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:13.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:13.075788+0000) 2024-03-06T09:22:14.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:14.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:14.076013+0000) 2024-03-06T09:22:15.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:15.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:15.076217+0000) 2024-03-06T09:22:16.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:16.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:16.076417+0000) 2024-03-06T09:22:17.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:17.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:17.076629+0000) 2024-03-06T09:22:18.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:18.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:18.076863+0000) 2024-03-06T09:22:19.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:19.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:19.077059+0000) 2024-03-06T09:22:20.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:20.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:20.077282+0000) 2024-03-06T09:22:21.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:21.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:21.077507+0000) 2024-03-06T09:22:22.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:22.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:22.077753+0000) 2024-03-06T09:22:23.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:23.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:23.077935+0000) 2024-03-06T09:22:24.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:24.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:24.078128+0000) 2024-03-06T09:22:25.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:25.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:25.078380+0000) 2024-03-06T09:22:26.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:26.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:26.078606+0000) 2024-03-06T09:22:27.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:27.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:27.078820+0000) 2024-03-06T09:22:28.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:28.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:28.079009+0000) 2024-03-06T09:22:29.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:29.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:29.079199+0000) 2024-03-06T09:22:29.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:29.559 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:30.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:30.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:30.079386+0000) 2024-03-06T09:22:31.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:31.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:31.079617+0000) 2024-03-06T09:22:32.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:32.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:32.079817+0000) 2024-03-06T09:22:33.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:33.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:33.080013+0000) 2024-03-06T09:22:34.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:34.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:34.080202+0000) 2024-03-06T09:22:35.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:35.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:35.080446+0000) 2024-03-06T09:22:36.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:36.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:36.080691+0000) 2024-03-06T09:22:37.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:37.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:37.080881+0000) 2024-03-06T09:22:38.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:38.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:38.081081+0000) 2024-03-06T09:22:39.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:39.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:39.081343+0000) 2024-03-06T09:22:40.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:40.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:40.081563+0000) 2024-03-06T09:22:41.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:41.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:41.081842+0000) 2024-03-06T09:22:42.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:42.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:42.082022+0000) 2024-03-06T09:22:43.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:43.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:43.082280+0000) 2024-03-06T09:22:44.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:44.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:44.082556+0000) 2024-03-06T09:22:45.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:45.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:45.082784+0000) 2024-03-06T09:22:46.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:46.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:46.083052+0000) 2024-03-06T09:22:47.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:47.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:47.083325+0000) 2024-03-06T09:22:48.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:48.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:48.083604+0000) 2024-03-06T09:22:49.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:49.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:49.083874+0000) 2024-03-06T09:22:50.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:50.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:50.084051+0000) 2024-03-06T09:22:51.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:51.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:51.084243+0000) 2024-03-06T09:22:52.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:52.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:52.084460+0000) 2024-03-06T09:22:53.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:53.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:53.084698+0000) 2024-03-06T09:22:54.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:54.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:54.084888+0000) 2024-03-06T09:22:55.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:55.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:55.085101+0000) 2024-03-06T09:22:56.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:56.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:56.085339+0000) 2024-03-06T09:22:57.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:57.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:57.085524+0000) 2024-03-06T09:22:58.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:58.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:58.085776+0000) 2024-03-06T09:22:59.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:22:59.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:59.086019+0000) 2024-03-06T09:22:59.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:59.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:00.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:00.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:00.086277+0000) 2024-03-06T09:23:01.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:01.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:01.086470+0000) 2024-03-06T09:23:02.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:02.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:02.086618+0000) 2024-03-06T09:23:03.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:03.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:03.086818+0000) 2024-03-06T09:23:04.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:04.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:04.087003+0000) 2024-03-06T09:23:05.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:05.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:05.087205+0000) 2024-03-06T09:23:06.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:06.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:06.087408+0000) 2024-03-06T09:23:07.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:07.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:07.087633+0000) 2024-03-06T09:23:08.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:08.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:08.087829+0000) 2024-03-06T09:23:09.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:09.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:09.088025+0000) 2024-03-06T09:23:10.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:10.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:10.088214+0000) 2024-03-06T09:23:11.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:11.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:11.088434+0000) 2024-03-06T09:23:12.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:12.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:12.088654+0000) 2024-03-06T09:23:13.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:13.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:13.088905+0000) 2024-03-06T09:23:14.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:14.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:14.089095+0000) 2024-03-06T09:23:15.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:15.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:15.089291+0000) 2024-03-06T09:23:16.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:16.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:16.089521+0000) 2024-03-06T09:23:17.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:17.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:17.089748+0000) 2024-03-06T09:23:18.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:18.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:18.089958+0000) 2024-03-06T09:23:19.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:19.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:19.090176+0000) 2024-03-06T09:23:20.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:20.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:20.090396+0000) 2024-03-06T09:23:21.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:21.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:21.090643+0000) 2024-03-06T09:23:22.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:22.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:22.090886+0000) 2024-03-06T09:23:23.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:23.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:23.091090+0000) 2024-03-06T09:23:24.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:24.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:24.091283+0000) 2024-03-06T09:23:25.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:25.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:25.091539+0000) 2024-03-06T09:23:26.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:26.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:26.091773+0000) 2024-03-06T09:23:27.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:27.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:27.091977+0000) 2024-03-06T09:23:28.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:28.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:28.092165+0000) 2024-03-06T09:23:29.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:29.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:29.092356+0000) 2024-03-06T09:23:29.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:29.935 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:30.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:30.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:30.092592+0000) 2024-03-06T09:23:31.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:31.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:31.092815+0000) 2024-03-06T09:23:32.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:32.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:32.093088+0000) 2024-03-06T09:23:33.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:33.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:33.093286+0000) 2024-03-06T09:23:34.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:34.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:34.093482+0000) 2024-03-06T09:23:35.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:35.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:35.093720+0000) 2024-03-06T09:23:36.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:36.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:36.093966+0000) 2024-03-06T09:23:37.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:37.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:37.094207+0000) 2024-03-06T09:23:38.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:38.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:38.094436+0000) 2024-03-06T09:23:39.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:39.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:39.094659+0000) 2024-03-06T09:23:40.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:40.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:40.094917+0000) 2024-03-06T09:23:41.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:41.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:41.095148+0000) 2024-03-06T09:23:42.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:42.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:42.095371+0000) 2024-03-06T09:23:43.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:43.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:43.095547+0000) 2024-03-06T09:23:44.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:44.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:44.095758+0000) 2024-03-06T09:23:45.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:45.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:45.095966+0000) 2024-03-06T09:23:46.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:46.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:46.096162+0000) 2024-03-06T09:23:47.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:47.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:47.096358+0000) 2024-03-06T09:23:48.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:48.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:48.096586+0000) 2024-03-06T09:23:49.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:49.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:49.096812+0000) 2024-03-06T09:23:50.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:50.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:50.097029+0000) 2024-03-06T09:23:51.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:51.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:51.097220+0000) 2024-03-06T09:23:52.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:52.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:52.097442+0000) 2024-03-06T09:23:53.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:53.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:53.097639+0000) 2024-03-06T09:23:54.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:54.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:54.097859+0000) 2024-03-06T09:23:55.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:55.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:55.098060+0000) 2024-03-06T09:23:56.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:56.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:56.098245+0000) 2024-03-06T09:23:57.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:57.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:57.098469+0000) 2024-03-06T09:23:58.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:58.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:58.098719+0000) 2024-03-06T09:23:59.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:23:59.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:59.098911+0000) 2024-03-06T09:24:00.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:00.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:00.099099+0000) 2024-03-06T09:24:00.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:00.149 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:01.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:01.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:01.099288+0000) 2024-03-06T09:24:02.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:02.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:02.099510+0000) 2024-03-06T09:24:03.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:03.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:03.099825+0000) 2024-03-06T09:24:04.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:04.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:04.100084+0000) 2024-03-06T09:24:05.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:05.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:05.100363+0000) 2024-03-06T09:24:06.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:06.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:06.100641+0000) 2024-03-06T09:24:07.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:07.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:07.100843+0000) 2024-03-06T09:24:08.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:08.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:08.100977+0000) 2024-03-06T09:24:09.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:09.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:09.101171+0000) 2024-03-06T09:24:10.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:10.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:10.101394+0000) 2024-03-06T09:24:11.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:11.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:11.101568+0000) 2024-03-06T09:24:12.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:12.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:12.101850+0000) 2024-03-06T09:24:13.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:13.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:13.102079+0000) 2024-03-06T09:24:14.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:14.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:14.102270+0000) 2024-03-06T09:24:15.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:15.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:15.102503+0000) 2024-03-06T09:24:16.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:16.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:16.102757+0000) 2024-03-06T09:24:17.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:17.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:17.102943+0000) 2024-03-06T09:24:18.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:18.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:18.103161+0000) 2024-03-06T09:24:19.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:19.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:19.103387+0000) 2024-03-06T09:24:20.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:20.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:20.103582+0000) 2024-03-06T09:24:21.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:21.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:21.103820+0000) 2024-03-06T09:24:22.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:22.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:22.104008+0000) 2024-03-06T09:24:23.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:23.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:23.104206+0000) 2024-03-06T09:24:24.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:24.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:24.104425+0000) 2024-03-06T09:24:25.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:25.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:25.104736+0000) 2024-03-06T09:24:26.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:26.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:26.104922+0000) 2024-03-06T09:24:27.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:27.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:27.105127+0000) 2024-03-06T09:24:28.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:28.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:28.105328+0000) 2024-03-06T09:24:29.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:29.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:29.105548+0000) 2024-03-06T09:24:30.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:30.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:30.105782+0000) 2024-03-06T09:24:30.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:30.361 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:31.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:31.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:31.105973+0000) 2024-03-06T09:24:32.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:32.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:32.106159+0000) 2024-03-06T09:24:33.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:33.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:33.106398+0000) 2024-03-06T09:24:34.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:34.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:34.106619+0000) 2024-03-06T09:24:35.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:35.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:35.106859+0000) 2024-03-06T09:24:36.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:36.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:36.107016+0000) 2024-03-06T09:24:37.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:37.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:37.107291+0000) 2024-03-06T09:24:38.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:38.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:38.107565+0000) 2024-03-06T09:24:39.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:39.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:39.107853+0000) 2024-03-06T09:24:40.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:40.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:40.108128+0000) 2024-03-06T09:24:41.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:41.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:41.108400+0000) 2024-03-06T09:24:42.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:42.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:42.108704+0000) 2024-03-06T09:24:43.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:43.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:43.108985+0000) 2024-03-06T09:24:44.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:44.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:44.109166+0000) 2024-03-06T09:24:45.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:45.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:45.109344+0000) 2024-03-06T09:24:46.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:46.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:46.109548+0000) 2024-03-06T09:24:47.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:47.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:47.109811+0000) 2024-03-06T09:24:48.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:48.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:48.109992+0000) 2024-03-06T09:24:49.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:49.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:49.110144+0000) 2024-03-06T09:24:50.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:50.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:50.110369+0000) 2024-03-06T09:24:51.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:51.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:51.110595+0000) 2024-03-06T09:24:52.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:52.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:52.110817+0000) 2024-03-06T09:24:53.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:53.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:53.111042+0000) 2024-03-06T09:24:54.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:54.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:54.111207+0000) 2024-03-06T09:24:55.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:55.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:55.111426+0000) 2024-03-06T09:24:56.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:56.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:56.111614+0000) 2024-03-06T09:24:57.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:57.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:57.111821+0000) 2024-03-06T09:24:58.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:58.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:58.112043+0000) 2024-03-06T09:24:59.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:24:59.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:59.112236+0000) 2024-03-06T09:25:00.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:00.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:00.112420+0000) 2024-03-06T09:25:00.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:00.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:01.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:01.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:01.112653+0000) 2024-03-06T09:25:02.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:02.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:02.112873+0000) 2024-03-06T09:25:03.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:03.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:03.113070+0000) 2024-03-06T09:25:04.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:04.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:04.113245+0000) 2024-03-06T09:25:05.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:05.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:05.113498+0000) 2024-03-06T09:25:06.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:06.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:06.113759+0000) 2024-03-06T09:25:07.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:07.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:07.113950+0000) 2024-03-06T09:25:08.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:08.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:08.114175+0000) 2024-03-06T09:25:09.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:09.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:09.114404+0000) 2024-03-06T09:25:10.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:10.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:10.114642+0000) 2024-03-06T09:25:11.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:11.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:11.114854+0000) 2024-03-06T09:25:12.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:12.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:12.115055+0000) 2024-03-06T09:25:13.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:13.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:13.115351+0000) 2024-03-06T09:25:14.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:14.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:14.115542+0000) 2024-03-06T09:25:15.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:15.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:15.115798+0000) 2024-03-06T09:25:16.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:16.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:16.116002+0000) 2024-03-06T09:25:17.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:17.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:17.116228+0000) 2024-03-06T09:25:18.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:18.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:18.116416+0000) 2024-03-06T09:25:19.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:19.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:19.116645+0000) 2024-03-06T09:25:20.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:20.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:20.116909+0000) 2024-03-06T09:25:21.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:21.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:21.117105+0000) 2024-03-06T09:25:22.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:22.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:22.117371+0000) 2024-03-06T09:25:23.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:23.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:23.117612+0000) 2024-03-06T09:25:24.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:24.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:24.117860+0000) 2024-03-06T09:25:25.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:25.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:25.118127+0000) 2024-03-06T09:25:26.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:26.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:26.118346+0000) 2024-03-06T09:25:27.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:27.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:27.118558+0000) 2024-03-06T09:25:28.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:28.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:28.118797+0000) 2024-03-06T09:25:29.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:29.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:29.119032+0000) 2024-03-06T09:25:30.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:30.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:30.119276+0000) 2024-03-06T09:25:30.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:30.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:31.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:31.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:31.119569+0000) 2024-03-06T09:25:32.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:32.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:32.119826+0000) 2024-03-06T09:25:33.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:33.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:33.120051+0000) 2024-03-06T09:25:34.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:34.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:34.120276+0000) 2024-03-06T09:25:35.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:35.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:35.120512+0000) 2024-03-06T09:25:36.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:36.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:36.120775+0000) 2024-03-06T09:25:37.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:37.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:37.121010+0000) 2024-03-06T09:25:38.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:38.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:38.121182+0000) 2024-03-06T09:25:39.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:39.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:39.121367+0000) 2024-03-06T09:25:40.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:40.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:40.121622+0000) 2024-03-06T09:25:41.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:41.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:41.121898+0000) 2024-03-06T09:25:42.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:42.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:42.122130+0000) 2024-03-06T09:25:43.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:43.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:43.122369+0000) 2024-03-06T09:25:44.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:44.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:44.122597+0000) 2024-03-06T09:25:45.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:45.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:45.122861+0000) 2024-03-06T09:25:46.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:46.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:46.123113+0000) 2024-03-06T09:25:47.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:47.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:47.123342+0000) 2024-03-06T09:25:48.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:48.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:48.123565+0000) 2024-03-06T09:25:49.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:49.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:49.123790+0000) 2024-03-06T09:25:50.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:50.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:50.123982+0000) 2024-03-06T09:25:51.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:51.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:51.124207+0000) 2024-03-06T09:25:52.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:52.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:52.124390+0000) 2024-03-06T09:25:53.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:53.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:53.124617+0000) 2024-03-06T09:25:54.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:54.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:54.124830+0000) 2024-03-06T09:25:55.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:55.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:55.125044+0000) 2024-03-06T09:25:56.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:56.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:56.125233+0000) 2024-03-06T09:25:57.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:57.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:57.125445+0000) 2024-03-06T09:25:58.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:58.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:58.125682+0000) 2024-03-06T09:25:59.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:25:59.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:59.125882+0000) 2024-03-06T09:26:00.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:00.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:00.126139+0000) 2024-03-06T09:26:00.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:00.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:01.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:01.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:01.126335+0000) 2024-03-06T09:26:02.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:02.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:02.126526+0000) 2024-03-06T09:26:03.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:03.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:03.126779+0000) 2024-03-06T09:26:04.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:04.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:04.127003+0000) 2024-03-06T09:26:05.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:05.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:05.127260+0000) 2024-03-06T09:26:06.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:06.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:06.127493+0000) 2024-03-06T09:26:07.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:07.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:07.127773+0000) 2024-03-06T09:26:08.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:08.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:08.127954+0000) 2024-03-06T09:26:09.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:09.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:09.128139+0000) 2024-03-06T09:26:10.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:10.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:10.128393+0000) 2024-03-06T09:26:11.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:11.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:11.128646+0000) 2024-03-06T09:26:12.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:12.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:12.128945+0000) 2024-03-06T09:26:13.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:13.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:13.129155+0000) 2024-03-06T09:26:14.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:14.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:14.129380+0000) 2024-03-06T09:26:15.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:15.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:15.129638+0000) 2024-03-06T09:26:16.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:16.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:16.129892+0000) 2024-03-06T09:26:17.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:17.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:17.130117+0000) 2024-03-06T09:26:18.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:18.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:18.130339+0000) 2024-03-06T09:26:19.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:19.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:19.130520+0000) 2024-03-06T09:26:20.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:20.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:20.130809+0000) 2024-03-06T09:26:21.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:21.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:21.130947+0000) 2024-03-06T09:26:22.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:22.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:22.131160+0000) 2024-03-06T09:26:23.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:23.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:23.131390+0000) 2024-03-06T09:26:24.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:24.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:24.131583+0000) 2024-03-06T09:26:25.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:25.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:25.131819+0000) 2024-03-06T09:26:26.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:26.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:26.131979+0000) 2024-03-06T09:26:27.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:27.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:27.132175+0000) 2024-03-06T09:26:28.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:28.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:28.132396+0000) 2024-03-06T09:26:29.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:29.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:29.132633+0000) 2024-03-06T09:26:30.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:30.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:30.132891+0000) 2024-03-06T09:26:31.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:31.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:31.133109+0000) 2024-03-06T09:26:31.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:31.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:32.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:32.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:32.133299+0000) 2024-03-06T09:26:33.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:33.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:33.133574+0000) 2024-03-06T09:26:34.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:34.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:34.133815+0000) 2024-03-06T09:26:35.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:35.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:35.134014+0000) 2024-03-06T09:26:36.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:36.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:36.134187+0000) 2024-03-06T09:26:37.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:37.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:37.134418+0000) 2024-03-06T09:26:38.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:38.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:38.134637+0000) 2024-03-06T09:26:39.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:39.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:39.134860+0000) 2024-03-06T09:26:40.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:40.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:40.135085+0000) 2024-03-06T09:26:41.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:41.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:41.135270+0000) 2024-03-06T09:26:42.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:42.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:42.135493+0000) 2024-03-06T09:26:43.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:43.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:43.135749+0000) 2024-03-06T09:26:44.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:44.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:44.135945+0000) 2024-03-06T09:26:45.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:45.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:45.136148+0000) 2024-03-06T09:26:46.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:46.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:46.136394+0000) 2024-03-06T09:26:47.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:47.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:47.136648+0000) 2024-03-06T09:26:48.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:48.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:48.136887+0000) 2024-03-06T09:26:49.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:49.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:49.137066+0000) 2024-03-06T09:26:50.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:50.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:50.137301+0000) 2024-03-06T09:26:51.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:51.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:51.137509+0000) 2024-03-06T09:26:52.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:52.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:52.137774+0000) 2024-03-06T09:26:53.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:53.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:53.137967+0000) 2024-03-06T09:26:54.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:54.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:54.138172+0000) 2024-03-06T09:26:55.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:55.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:55.138384+0000) 2024-03-06T09:26:56.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:56.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:56.138588+0000) 2024-03-06T09:26:57.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:57.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:57.138797+0000) 2024-03-06T09:26:58.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:58.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:58.139001+0000) 2024-03-06T09:26:59.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:26:59.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:59.139220+0000) 2024-03-06T09:27:00.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:00.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:00.139445+0000) 2024-03-06T09:27:01.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:01.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:01.139740+0000) 2024-03-06T09:27:01.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:01.446 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:02.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:02.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:02.139966+0000) 2024-03-06T09:27:03.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:03.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:03.140246+0000) 2024-03-06T09:27:04.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:04.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:04.140528+0000) 2024-03-06T09:27:05.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:05.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:05.140816+0000) 2024-03-06T09:27:06.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:06.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:06.141084+0000) 2024-03-06T09:27:07.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:07.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:07.141355+0000) 2024-03-06T09:27:08.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:08.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:08.141617+0000) 2024-03-06T09:27:09.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:09.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:09.141824+0000) 2024-03-06T09:27:10.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:10.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:10.142022+0000) 2024-03-06T09:27:11.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:11.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:11.142243+0000) 2024-03-06T09:27:12.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:12.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:12.142431+0000) 2024-03-06T09:27:13.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:13.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:13.142652+0000) 2024-03-06T09:27:14.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:14.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:14.142943+0000) 2024-03-06T09:27:15.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:15.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:15.143173+0000) 2024-03-06T09:27:16.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:16.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:16.143426+0000) 2024-03-06T09:27:17.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:17.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:17.143739+0000) 2024-03-06T09:27:18.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:18.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:18.143980+0000) 2024-03-06T09:27:19.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:19.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:19.144182+0000) 2024-03-06T09:27:20.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:20.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:20.144407+0000) 2024-03-06T09:27:21.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:21.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:21.144630+0000) 2024-03-06T09:27:22.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:22.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:22.144865+0000) 2024-03-06T09:27:23.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:23.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:23.145103+0000) 2024-03-06T09:27:24.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:24.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:24.145338+0000) 2024-03-06T09:27:25.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:25.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:25.145581+0000) 2024-03-06T09:27:26.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:26.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:26.145831+0000) 2024-03-06T09:27:27.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:27.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:27.146073+0000) 2024-03-06T09:27:28.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:28.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:28.146312+0000) 2024-03-06T09:27:29.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:29.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:29.146552+0000) 2024-03-06T09:27:30.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:30.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:30.146763+0000) 2024-03-06T09:27:31.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:31.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:31.146921+0000) 2024-03-06T09:27:31.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:31.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:32.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:32.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:32.147151+0000) 2024-03-06T09:27:33.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:33.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:33.147377+0000) 2024-03-06T09:27:34.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:34.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:34.147608+0000) 2024-03-06T09:27:35.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:35.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:35.147828+0000) 2024-03-06T09:27:36.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:36.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:36.148028+0000) 2024-03-06T09:27:37.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:37.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:37.148271+0000) 2024-03-06T09:27:38.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:38.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:38.148492+0000) 2024-03-06T09:27:39.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:39.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:39.148747+0000) 2024-03-06T09:27:40.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:40.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:40.148968+0000) 2024-03-06T09:27:41.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:41.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:41.149166+0000) 2024-03-06T09:27:42.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:42.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:42.149354+0000) 2024-03-06T09:27:43.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:43.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:43.149574+0000) 2024-03-06T09:27:44.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:44.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:44.150005+0000) 2024-03-06T09:27:45.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:45.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:45.150295+0000) 2024-03-06T09:27:46.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:46.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:46.150481+0000) 2024-03-06T09:27:47.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:47.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:47.150754+0000) 2024-03-06T09:27:48.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:48.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:48.150999+0000) 2024-03-06T09:27:49.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:49.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:49.151241+0000) 2024-03-06T09:27:50.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:50.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:50.151415+0000) 2024-03-06T09:27:51.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:51.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:51.151703+0000) 2024-03-06T09:27:52.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:52.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:52.151985+0000) 2024-03-06T09:27:53.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:53.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:53.152197+0000) 2024-03-06T09:27:54.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:54.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:54.152428+0000) 2024-03-06T09:27:55.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:55.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:55.152658+0000) 2024-03-06T09:27:56.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:56.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:56.152917+0000) 2024-03-06T09:27:57.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:57.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:57.153106+0000) 2024-03-06T09:27:58.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:58.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:58.153325+0000) 2024-03-06T09:27:59.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:27:59.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:59.153545+0000) 2024-03-06T09:28:00.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:00.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:00.153767+0000) 2024-03-06T09:28:01.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:01.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:01.153963+0000) 2024-03-06T09:28:01.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:01.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:02.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:02.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:02.154191+0000) 2024-03-06T09:28:03.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:03.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:03.154418+0000) 2024-03-06T09:28:04.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:04.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:04.154641+0000) 2024-03-06T09:28:05.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:05.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:05.154865+0000) 2024-03-06T09:28:06.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:06.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:06.155052+0000) 2024-03-06T09:28:07.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:07.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:07.155279+0000) 2024-03-06T09:28:08.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:08.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:08.155509+0000) 2024-03-06T09:28:09.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:09.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:09.155758+0000) 2024-03-06T09:28:10.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:10.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:10.155947+0000) 2024-03-06T09:28:11.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:11.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:11.156177+0000) 2024-03-06T09:28:12.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:12.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:12.156404+0000) 2024-03-06T09:28:13.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:13.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:13.156640+0000) 2024-03-06T09:28:14.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:14.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:14.156917+0000) 2024-03-06T09:28:15.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:15.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:15.157125+0000) 2024-03-06T09:28:16.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:16.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:16.157353+0000) 2024-03-06T09:28:17.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:17.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:17.157579+0000) 2024-03-06T09:28:18.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:18.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:18.157767+0000) 2024-03-06T09:28:19.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:19.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:19.157996+0000) 2024-03-06T09:28:20.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:20.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:20.158210+0000) 2024-03-06T09:28:21.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:21.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:21.158387+0000) 2024-03-06T09:28:22.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:22.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:22.158599+0000) 2024-03-06T09:28:23.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:23.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:23.158788+0000) 2024-03-06T09:28:24.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:24.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:24.158979+0000) 2024-03-06T09:28:25.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:25.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:25.159230+0000) 2024-03-06T09:28:26.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:26.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:26.159411+0000) 2024-03-06T09:28:27.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:27.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:27.159661+0000) 2024-03-06T09:28:28.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:28.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:28.159922+0000) 2024-03-06T09:28:29.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:29.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:29.160162+0000) 2024-03-06T09:28:30.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:30.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:30.160336+0000) 2024-03-06T09:28:31.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:31.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:31.160560+0000) 2024-03-06T09:28:32.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:32.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:32.160789+0000) 2024-03-06T09:28:32.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:32.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:33.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:33.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:33.160974+0000) 2024-03-06T09:28:34.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:34.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:34.161199+0000) 2024-03-06T09:28:35.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:35.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:35.161446+0000) 2024-03-06T09:28:36.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:36.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:36.161711+0000) 2024-03-06T09:28:37.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:37.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:37.161947+0000) 2024-03-06T09:28:38.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:38.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:38.162131+0000) 2024-03-06T09:28:39.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:39.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:39.162349+0000) 2024-03-06T09:28:40.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:40.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:40.162562+0000) 2024-03-06T09:28:41.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:41.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:41.162861+0000) 2024-03-06T09:28:42.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:42.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:42.163090+0000) 2024-03-06T09:28:43.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:43.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:43.163277+0000) 2024-03-06T09:28:44.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:44.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:44.163505+0000) 2024-03-06T09:28:45.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:45.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:45.163763+0000) 2024-03-06T09:28:46.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:46.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:46.164003+0000) 2024-03-06T09:28:47.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:47.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:47.164232+0000) 2024-03-06T09:28:48.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:48.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:48.164476+0000) 2024-03-06T09:28:49.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:49.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:49.164737+0000) 2024-03-06T09:28:50.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:50.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:50.164926+0000) 2024-03-06T09:28:51.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:51.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:51.165136+0000) 2024-03-06T09:28:52.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:52.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:52.165376+0000) 2024-03-06T09:28:53.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:53.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:53.165527+0000) 2024-03-06T09:28:54.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:54.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:54.165747+0000) 2024-03-06T09:28:55.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:55.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:55.165993+0000) 2024-03-06T09:28:56.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:56.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:56.166213+0000) 2024-03-06T09:28:57.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:57.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:57.166446+0000) 2024-03-06T09:28:58.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:58.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:58.166618+0000) 2024-03-06T09:28:59.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:28:59.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:59.166829+0000) 2024-03-06T09:29:00.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:00.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:00.167048+0000) 2024-03-06T09:29:01.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:01.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:01.167275+0000) 2024-03-06T09:29:02.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:02.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:02.167501+0000) 2024-03-06T09:29:02.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:02.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:03.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:03.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:03.167779+0000) 2024-03-06T09:29:04.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:04.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:04.168022+0000) 2024-03-06T09:29:05.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:05.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:05.168245+0000) 2024-03-06T09:29:06.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:06.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:06.168468+0000) 2024-03-06T09:29:07.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:07.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:07.168743+0000) 2024-03-06T09:29:08.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:08.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:08.168954+0000) 2024-03-06T09:29:09.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:09.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:09.169144+0000) 2024-03-06T09:29:10.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:10.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:10.169372+0000) 2024-03-06T09:29:11.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:11.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:11.169605+0000) 2024-03-06T09:29:12.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:12.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:12.169801+0000) 2024-03-06T09:29:13.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:13.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:13.170025+0000) 2024-03-06T09:29:14.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:14.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:14.170225+0000) 2024-03-06T09:29:15.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:15.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:15.170496+0000) 2024-03-06T09:29:16.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:16.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:16.170723+0000) 2024-03-06T09:29:17.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:17.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:17.170952+0000) 2024-03-06T09:29:18.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:18.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:18.171148+0000) 2024-03-06T09:29:19.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:19.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:19.171376+0000) 2024-03-06T09:29:20.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:20.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:20.171613+0000) 2024-03-06T09:29:21.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:21.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:21.171831+0000) 2024-03-06T09:29:22.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:22.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:22.172049+0000) 2024-03-06T09:29:23.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:23.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:23.172246+0000) 2024-03-06T09:29:24.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:24.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:24.172468+0000) 2024-03-06T09:29:25.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:25.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:25.172650+0000) 2024-03-06T09:29:26.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:26.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:26.172914+0000) 2024-03-06T09:29:27.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:27.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:27.173144+0000) 2024-03-06T09:29:28.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:28.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:28.173368+0000) 2024-03-06T09:29:29.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:29.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:29.173530+0000) 2024-03-06T09:29:30.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:30.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:30.173783+0000) 2024-03-06T09:29:31.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:31.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:31.174006+0000) 2024-03-06T09:29:32.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:32.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:32.174208+0000) 2024-03-06T09:29:32.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:32.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:33.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:33.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:33.174479+0000) 2024-03-06T09:29:34.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:34.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:34.174805+0000) 2024-03-06T09:29:35.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:35.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:35.175039+0000) 2024-03-06T09:29:36.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:36.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:36.175232+0000) 2024-03-06T09:29:37.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:37.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:37.175439+0000) 2024-03-06T09:29:38.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:38.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:38.175701+0000) 2024-03-06T09:29:39.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:39.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:39.175945+0000) 2024-03-06T09:29:40.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:40.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:40.176188+0000) 2024-03-06T09:29:41.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:41.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:41.176385+0000) 2024-03-06T09:29:42.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:42.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:42.176596+0000) 2024-03-06T09:29:43.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:43.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:43.176788+0000) 2024-03-06T09:29:44.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:44.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:44.176981+0000) 2024-03-06T09:29:45.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:45.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:45.177184+0000) 2024-03-06T09:29:46.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:46.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:46.177388+0000) 2024-03-06T09:29:47.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:47.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:47.177576+0000) 2024-03-06T09:29:48.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:48.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:48.177770+0000) 2024-03-06T09:29:49.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:49.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:49.177999+0000) 2024-03-06T09:29:50.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:50.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:50.178147+0000) 2024-03-06T09:29:51.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:51.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:51.178390+0000) 2024-03-06T09:29:52.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:52.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:52.178616+0000) 2024-03-06T09:29:53.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:53.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:53.178847+0000) 2024-03-06T09:29:54.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:54.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:54.179034+0000) 2024-03-06T09:29:55.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:55.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:55.179244+0000) 2024-03-06T09:29:56.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:56.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:56.179440+0000) 2024-03-06T09:29:57.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:57.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:57.179692+0000) 2024-03-06T09:29:58.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:58.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:58.179896+0000) 2024-03-06T09:29:59.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:29:59.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:59.180075+0000) 2024-03-06T09:30:00.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:00.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:00.180298+0000) 2024-03-06T09:30:01.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:01.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:01.180485+0000) 2024-03-06T09:30:02.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:02.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:02.180730+0000) 2024-03-06T09:30:02.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:02.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:03.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:03.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:03.180923+0000) 2024-03-06T09:30:04.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:04.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:04.181179+0000) 2024-03-06T09:30:05.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:05.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:05.181406+0000) 2024-03-06T09:30:06.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:06.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:06.181579+0000) 2024-03-06T09:30:07.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:07.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:07.181829+0000) 2024-03-06T09:30:08.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:08.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:08.182018+0000) 2024-03-06T09:30:09.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:09.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:09.182197+0000) 2024-03-06T09:30:10.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:10.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:10.182385+0000) 2024-03-06T09:30:11.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:11.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:11.182594+0000) 2024-03-06T09:30:12.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:12.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:12.182828+0000) 2024-03-06T09:30:13.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:13.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:13.183007+0000) 2024-03-06T09:30:14.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:14.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:14.183192+0000) 2024-03-06T09:30:15.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:15.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:15.183433+0000) 2024-03-06T09:30:16.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:16.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:16.183654+0000) 2024-03-06T09:30:17.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:17.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:17.183921+0000) 2024-03-06T09:30:18.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:18.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:18.184103+0000) 2024-03-06T09:30:19.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:19.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:19.184328+0000) 2024-03-06T09:30:20.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:20.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:20.184552+0000) 2024-03-06T09:30:21.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:21.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:21.184784+0000) 2024-03-06T09:30:22.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:22.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:22.185007+0000) 2024-03-06T09:30:23.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:23.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:23.185197+0000) 2024-03-06T09:30:24.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:24.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:24.185422+0000) 2024-03-06T09:30:25.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:25.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:25.185699+0000) 2024-03-06T09:30:26.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:26.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:26.185939+0000) 2024-03-06T09:30:27.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:27.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:27.186139+0000) 2024-03-06T09:30:28.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:28.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:28.186386+0000) 2024-03-06T09:30:29.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:29.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:29.186592+0000) 2024-03-06T09:30:30.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:30.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:30.186838+0000) 2024-03-06T09:30:31.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:31.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:31.187024+0000) 2024-03-06T09:30:32.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:32.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:32.187223+0000) 2024-03-06T09:30:33.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:33.181 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:33.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:33.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:33.187423+0000) 2024-03-06T09:30:34.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:34.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:34.187641+0000) 2024-03-06T09:30:35.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:35.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:35.187864+0000) 2024-03-06T09:30:36.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:36.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:36.188090+0000) 2024-03-06T09:30:37.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:37.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:37.188321+0000) 2024-03-06T09:30:38.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:38.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:38.188557+0000) 2024-03-06T09:30:39.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:39.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:39.188780+0000) 2024-03-06T09:30:40.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:40.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:40.188900+0000) 2024-03-06T09:30:41.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:41.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:41.189056+0000) 2024-03-06T09:30:42.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:42.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:42.189222+0000) 2024-03-06T09:30:43.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:43.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:43.189429+0000) 2024-03-06T09:30:44.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:44.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:44.189660+0000) 2024-03-06T09:30:45.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:45.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:45.189863+0000) 2024-03-06T09:30:46.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:46.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:46.190078+0000) 2024-03-06T09:30:47.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:47.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:47.190295+0000) 2024-03-06T09:30:48.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:48.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:48.190531+0000) 2024-03-06T09:30:49.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:49.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:49.190724+0000) 2024-03-06T09:30:50.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:50.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:50.190949+0000) 2024-03-06T09:30:51.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:51.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:51.191193+0000) 2024-03-06T09:30:52.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:52.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:52.191410+0000) 2024-03-06T09:30:53.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:53.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:53.191636+0000) 2024-03-06T09:30:54.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:54.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:54.191796+0000) 2024-03-06T09:30:55.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:55.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:55.192029+0000) 2024-03-06T09:30:56.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:56.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:56.192207+0000) 2024-03-06T09:30:57.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:57.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:57.192465+0000) 2024-03-06T09:30:58.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:58.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:58.192726+0000) 2024-03-06T09:30:59.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:30:59.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:59.192941+0000) 2024-03-06T09:31:00.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:00.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:00.193180+0000) 2024-03-06T09:31:01.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:01.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:01.193409+0000) 2024-03-06T09:31:02.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:02.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:02.193627+0000) 2024-03-06T09:31:03.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:03.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:03.193854+0000) 2024-03-06T09:31:03.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:03.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:04.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:04.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:04.194017+0000) 2024-03-06T09:31:05.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:05.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:05.194278+0000) 2024-03-06T09:31:06.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:06.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:06.194517+0000) 2024-03-06T09:31:07.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:07.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:07.194768+0000) 2024-03-06T09:31:08.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:08.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:08.194964+0000) 2024-03-06T09:31:09.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:09.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:09.195165+0000) 2024-03-06T09:31:10.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:10.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:10.195403+0000) 2024-03-06T09:31:11.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:11.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:11.195646+0000) 2024-03-06T09:31:12.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:12.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:12.195934+0000) 2024-03-06T09:31:13.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:13.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:13.196105+0000) 2024-03-06T09:31:14.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:14.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:14.196336+0000) 2024-03-06T09:31:15.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:15.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:15.196532+0000) 2024-03-06T09:31:16.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:16.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:16.196788+0000) 2024-03-06T09:31:17.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:17.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:17.196962+0000) 2024-03-06T09:31:18.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:18.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:18.197164+0000) 2024-03-06T09:31:19.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:19.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:19.197330+0000) 2024-03-06T09:31:20.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:20.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:20.197568+0000) 2024-03-06T09:31:21.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:21.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:21.197816+0000) 2024-03-06T09:31:22.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:22.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:22.198037+0000) 2024-03-06T09:31:23.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:23.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:23.198265+0000) 2024-03-06T09:31:24.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:24.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:24.198480+0000) 2024-03-06T09:31:25.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:25.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:25.198772+0000) 2024-03-06T09:31:26.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:26.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:26.198974+0000) 2024-03-06T09:31:27.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:27.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:27.199205+0000) 2024-03-06T09:31:28.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:28.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:28.199388+0000) 2024-03-06T09:31:29.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:29.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:29.199623+0000) 2024-03-06T09:31:30.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:30.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:30.199812+0000) 2024-03-06T09:31:31.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:31.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:31.200041+0000) 2024-03-06T09:31:32.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:32.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:32.200225+0000) 2024-03-06T09:31:33.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:33.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:33.200407+0000) 2024-03-06T09:31:33.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:33.569 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:34.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:34.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:34.200636+0000) 2024-03-06T09:31:35.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:35.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:35.200881+0000) 2024-03-06T09:31:36.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:36.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:36.201118+0000) 2024-03-06T09:31:37.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:37.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:37.201381+0000) 2024-03-06T09:31:38.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:38.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:38.201552+0000) 2024-03-06T09:31:39.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:39.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:39.201775+0000) 2024-03-06T09:31:40.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:40.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:40.201993+0000) 2024-03-06T09:31:41.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:41.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:41.202229+0000) 2024-03-06T09:31:42.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:42.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:42.202412+0000) 2024-03-06T09:31:43.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:43.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:43.202614+0000) 2024-03-06T09:31:44.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:44.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:44.202881+0000) 2024-03-06T09:31:45.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:45.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:45.203100+0000) 2024-03-06T09:31:46.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:46.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:46.203328+0000) 2024-03-06T09:31:47.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:47.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:47.203547+0000) 2024-03-06T09:31:48.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:48.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:48.203789+0000) 2024-03-06T09:31:49.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:49.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:49.203987+0000) 2024-03-06T09:31:50.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:50.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:50.204225+0000) 2024-03-06T09:31:51.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:51.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:51.204437+0000) 2024-03-06T09:31:52.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:52.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:52.204703+0000) 2024-03-06T09:31:53.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:53.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:53.204891+0000) 2024-03-06T09:31:54.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:54.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:54.205142+0000) 2024-03-06T09:31:55.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:55.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:55.205390+0000) 2024-03-06T09:31:56.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:56.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:56.205642+0000) 2024-03-06T09:31:57.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:57.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:57.205907+0000) 2024-03-06T09:31:58.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:58.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:58.206152+0000) 2024-03-06T09:31:59.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:31:59.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:59.206385+0000) 2024-03-06T09:32:00.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:00.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:00.206606+0000) 2024-03-06T09:32:01.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:01.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:01.206834+0000) 2024-03-06T09:32:02.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:02.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:02.207045+0000) 2024-03-06T09:32:03.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:03.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:03.207295+0000) 2024-03-06T09:32:03.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:03.769 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:04.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:04.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:04.207548+0000) 2024-03-06T09:32:05.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:05.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:05.207799+0000) 2024-03-06T09:32:06.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:06.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:06.208001+0000) 2024-03-06T09:32:07.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:07.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:07.208229+0000) 2024-03-06T09:32:08.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:08.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:08.208420+0000) 2024-03-06T09:32:09.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:09.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:09.208659+0000) 2024-03-06T09:32:10.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:10.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:10.208903+0000) 2024-03-06T09:32:11.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:11.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:11.209128+0000) 2024-03-06T09:32:12.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:12.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:12.209351+0000) 2024-03-06T09:32:13.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:13.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:13.209592+0000) 2024-03-06T09:32:14.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:14.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:14.209820+0000) 2024-03-06T09:32:15.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:15.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:15.209993+0000) 2024-03-06T09:32:16.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:16.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:16.210193+0000) 2024-03-06T09:32:17.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:17.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:17.210438+0000) 2024-03-06T09:32:18.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:18.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:18.210760+0000) 2024-03-06T09:32:19.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:19.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:19.210969+0000) 2024-03-06T09:32:20.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:20.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:20.211189+0000) 2024-03-06T09:32:21.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:21.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:21.211416+0000) 2024-03-06T09:32:22.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:22.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:22.211637+0000) 2024-03-06T09:32:23.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:23.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:23.211833+0000) 2024-03-06T09:32:24.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:24.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:24.212054+0000) 2024-03-06T09:32:25.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:25.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:25.212296+0000) 2024-03-06T09:32:26.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:26.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:26.212482+0000) 2024-03-06T09:32:27.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:27.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:27.212708+0000) 2024-03-06T09:32:28.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:28.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:28.212894+0000) 2024-03-06T09:32:29.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:29.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:29.213121+0000) 2024-03-06T09:32:30.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:30.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:30.213347+0000) 2024-03-06T09:32:31.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:31.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:31.213530+0000) 2024-03-06T09:32:32.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:32.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:32.213772+0000) 2024-03-06T09:32:33.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:33.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:33.213955+0000) 2024-03-06T09:32:34.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:34.011 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:34.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:34.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:34.214150+0000) 2024-03-06T09:32:35.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:35.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:35.214378+0000) 2024-03-06T09:32:36.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:36.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:36.214619+0000) 2024-03-06T09:32:37.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:37.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:37.214824+0000) 2024-03-06T09:32:38.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:38.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:38.214963+0000) 2024-03-06T09:32:39.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:39.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:39.215143+0000) 2024-03-06T09:32:40.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:40.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:40.215319+0000) 2024-03-06T09:32:41.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:41.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:41.215554+0000) 2024-03-06T09:32:42.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:42.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:42.215775+0000) 2024-03-06T09:32:43.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:43.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:43.215997+0000) 2024-03-06T09:32:44.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:44.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:44.216246+0000) 2024-03-06T09:32:45.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:45.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:45.216478+0000) 2024-03-06T09:32:46.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:46.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:46.216749+0000) 2024-03-06T09:32:47.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:47.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:47.216979+0000) 2024-03-06T09:32:48.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:48.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:48.217175+0000) 2024-03-06T09:32:49.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:49.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:49.217403+0000) 2024-03-06T09:32:50.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:50.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:50.217585+0000) 2024-03-06T09:32:51.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:51.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:51.217817+0000) 2024-03-06T09:32:52.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:52.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:52.218047+0000) 2024-03-06T09:32:53.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:53.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:53.218236+0000) 2024-03-06T09:32:54.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:54.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:54.218431+0000) 2024-03-06T09:32:55.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:55.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:55.218620+0000) 2024-03-06T09:32:56.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:56.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:56.218826+0000) 2024-03-06T09:32:57.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:57.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:57.219012+0000) 2024-03-06T09:32:58.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:58.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:58.219215+0000) 2024-03-06T09:32:59.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:32:59.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:59.219448+0000) 2024-03-06T09:33:00.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:00.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:00.219713+0000) 2024-03-06T09:33:01.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:01.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:01.219947+0000) 2024-03-06T09:33:02.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:02.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:02.220222+0000) 2024-03-06T09:33:03.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:03.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:03.220463+0000) 2024-03-06T09:33:04.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:04.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:04.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:04.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:04.220656+0000) 2024-03-06T09:33:05.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:05.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:05.220950+0000) 2024-03-06T09:33:06.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:06.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:06.221190+0000) 2024-03-06T09:33:07.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:07.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:07.221424+0000) 2024-03-06T09:33:08.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:08.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:08.221688+0000) 2024-03-06T09:33:09.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:09.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:09.221920+0000) 2024-03-06T09:33:10.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:10.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:10.222078+0000) 2024-03-06T09:33:11.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:11.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:11.222311+0000) 2024-03-06T09:33:12.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:12.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:12.222550+0000) 2024-03-06T09:33:13.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:13.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:13.222811+0000) 2024-03-06T09:33:14.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:14.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:14.223004+0000) 2024-03-06T09:33:15.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:15.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:15.223237+0000) 2024-03-06T09:33:16.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:16.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:16.223461+0000) 2024-03-06T09:33:17.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:17.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:17.223759+0000) 2024-03-06T09:33:18.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:18.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:18.224003+0000) 2024-03-06T09:33:19.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:19.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:19.224281+0000) 2024-03-06T09:33:20.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:20.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:20.224561+0000) 2024-03-06T09:33:21.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:21.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:21.224807+0000) 2024-03-06T09:33:22.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:22.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:22.225000+0000) 2024-03-06T09:33:23.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:23.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:23.225227+0000) 2024-03-06T09:33:24.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:24.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:24.225423+0000) 2024-03-06T09:33:25.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:25.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:25.225611+0000) 2024-03-06T09:33:26.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:26.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:26.225811+0000) 2024-03-06T09:33:27.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:27.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:27.225994+0000) 2024-03-06T09:33:28.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:28.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:28.226235+0000) 2024-03-06T09:33:29.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:29.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:29.226479+0000) 2024-03-06T09:33:30.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:30.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:30.226798+0000) 2024-03-06T09:33:31.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:31.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:31.227009+0000) 2024-03-06T09:33:32.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:32.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:32.227250+0000) 2024-03-06T09:33:33.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:33.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:33.227495+0000) 2024-03-06T09:33:34.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:34.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:34.227781+0000) 2024-03-06T09:33:34.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:34.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:35.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:35.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:35.228071+0000) 2024-03-06T09:33:36.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:36.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:36.228315+0000) 2024-03-06T09:33:37.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:37.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:37.228537+0000) 2024-03-06T09:33:38.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:38.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:38.228791+0000) 2024-03-06T09:33:39.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:39.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:39.228966+0000) 2024-03-06T09:33:40.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:40.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:40.229215+0000) 2024-03-06T09:33:41.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:41.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:41.229454+0000) 2024-03-06T09:33:42.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:42.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:42.229755+0000) 2024-03-06T09:33:43.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:43.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:43.229986+0000) 2024-03-06T09:33:44.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:44.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:44.230226+0000) 2024-03-06T09:33:45.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:45.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:45.230459+0000) 2024-03-06T09:33:46.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:46.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:46.230760+0000) 2024-03-06T09:33:47.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:47.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:47.230965+0000) 2024-03-06T09:33:48.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:48.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:48.231190+0000) 2024-03-06T09:33:49.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:49.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:49.231427+0000) 2024-03-06T09:33:50.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:50.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:50.231934+0000) 2024-03-06T09:33:51.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:51.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:51.232147+0000) 2024-03-06T09:33:52.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:52.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:52.232426+0000) 2024-03-06T09:33:53.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:53.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:53.232739+0000) 2024-03-06T09:33:54.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:54.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:54.232970+0000) 2024-03-06T09:33:55.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:55.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:55.233242+0000) 2024-03-06T09:33:56.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:56.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:56.233510+0000) 2024-03-06T09:33:57.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:57.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:57.233812+0000) 2024-03-06T09:33:58.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:58.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:58.234092+0000) 2024-03-06T09:33:59.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:33:59.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:59.234371+0000) 2024-03-06T09:34:00.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:00.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:00.234575+0000) 2024-03-06T09:34:01.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:01.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:01.234846+0000) 2024-03-06T09:34:02.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:02.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:02.235058+0000) 2024-03-06T09:34:03.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:03.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:03.235235+0000) 2024-03-06T09:34:04.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:04.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:04.235406+0000) 2024-03-06T09:34:04.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:04.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:05.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:05.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:05.235717+0000) 2024-03-06T09:34:06.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:06.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:06.235966+0000) 2024-03-06T09:34:07.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:07.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:07.236207+0000) 2024-03-06T09:34:08.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:08.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:08.236428+0000) 2024-03-06T09:34:09.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:09.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:09.236652+0000) 2024-03-06T09:34:10.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:10.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:10.236928+0000) 2024-03-06T09:34:11.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:11.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:11.237169+0000) 2024-03-06T09:34:12.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:12.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:12.237395+0000) 2024-03-06T09:34:13.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:13.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:13.237707+0000) 2024-03-06T09:34:14.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:14.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:14.237987+0000) 2024-03-06T09:34:15.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:15.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:15.238274+0000) 2024-03-06T09:34:16.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:16.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:16.238516+0000) 2024-03-06T09:34:17.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:17.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:17.238753+0000) 2024-03-06T09:34:18.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:18.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:18.239023+0000) 2024-03-06T09:34:19.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:19.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:19.239220+0000) 2024-03-06T09:34:20.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:20.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:20.239438+0000) 2024-03-06T09:34:21.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:21.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:21.239646+0000) 2024-03-06T09:34:22.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:22.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:22.239907+0000) 2024-03-06T09:34:23.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:23.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:23.240124+0000) 2024-03-06T09:34:24.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:24.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:24.240359+0000) 2024-03-06T09:34:25.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:25.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:25.240627+0000) 2024-03-06T09:34:26.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:26.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:26.240897+0000) 2024-03-06T09:34:27.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:27.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:27.241104+0000) 2024-03-06T09:34:28.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:28.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:28.241371+0000) 2024-03-06T09:34:29.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:29.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:29.241565+0000) 2024-03-06T09:34:30.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:30.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:30.241794+0000) 2024-03-06T09:34:31.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:31.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:31.242012+0000) 2024-03-06T09:34:32.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:32.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:32.242279+0000) 2024-03-06T09:34:33.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:33.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:33.242558+0000) 2024-03-06T09:34:34.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:34.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:34.242840+0000) 2024-03-06T09:34:34.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:34.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:35.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:35.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:35.243132+0000) 2024-03-06T09:34:36.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:36.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:36.243365+0000) 2024-03-06T09:34:37.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:37.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:37.243639+0000) 2024-03-06T09:34:38.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:38.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:38.243939+0000) 2024-03-06T09:34:39.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:39.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:39.244157+0000) 2024-03-06T09:34:40.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:40.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:40.244358+0000) 2024-03-06T09:34:41.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:41.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:41.244637+0000) 2024-03-06T09:34:42.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:42.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:42.244908+0000) 2024-03-06T09:34:43.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:43.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:43.245186+0000) 2024-03-06T09:34:44.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:44.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:44.245512+0000) 2024-03-06T09:34:45.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:45.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:45.245770+0000) 2024-03-06T09:34:46.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:46.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:46.245944+0000) 2024-03-06T09:34:47.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:47.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:47.246218+0000) 2024-03-06T09:34:48.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:48.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:48.246461+0000) 2024-03-06T09:34:49.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:49.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:49.246735+0000) 2024-03-06T09:34:50.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:50.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:50.246997+0000) 2024-03-06T09:34:51.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:51.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:51.247225+0000) 2024-03-06T09:34:52.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:52.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:52.247486+0000) 2024-03-06T09:34:53.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:53.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:53.247807+0000) 2024-03-06T09:34:54.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:54.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:54.248019+0000) 2024-03-06T09:34:55.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:55.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:55.248264+0000) 2024-03-06T09:34:56.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:56.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:56.248508+0000) 2024-03-06T09:34:57.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:57.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:57.248822+0000) 2024-03-06T09:34:58.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:58.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:58.249045+0000) 2024-03-06T09:34:59.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:34:59.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:59.249317+0000) 2024-03-06T09:35:00.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:00.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:00.249587+0000) 2024-03-06T09:35:01.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:01.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:01.249855+0000) 2024-03-06T09:35:02.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:02.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:02.250095+0000) 2024-03-06T09:35:03.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:03.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:03.250378+0000) 2024-03-06T09:35:04.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:04.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:04.250655+0000) 2024-03-06T09:35:05.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:05.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:05.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:05.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:05.250974+0000) 2024-03-06T09:35:06.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:06.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:06.251254+0000) 2024-03-06T09:35:07.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:07.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:07.251494+0000) 2024-03-06T09:35:08.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:08.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:08.251770+0000) 2024-03-06T09:35:09.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:09.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:09.251963+0000) 2024-03-06T09:35:10.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:10.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:10.252191+0000) 2024-03-06T09:35:11.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:11.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:11.252388+0000) 2024-03-06T09:35:12.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:12.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:12.252626+0000) 2024-03-06T09:35:13.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:13.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:13.252853+0000) 2024-03-06T09:35:14.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:14.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:14.253029+0000) 2024-03-06T09:35:15.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:15.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:15.253276+0000) 2024-03-06T09:35:16.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:16.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:16.253465+0000) 2024-03-06T09:35:17.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:17.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:17.253756+0000) 2024-03-06T09:35:18.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:18.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:18.254026+0000) 2024-03-06T09:35:19.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:19.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:19.254252+0000) 2024-03-06T09:35:20.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:20.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:20.254439+0000) 2024-03-06T09:35:21.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:21.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:21.254714+0000) 2024-03-06T09:35:22.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:22.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:22.254904+0000) 2024-03-06T09:35:23.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:23.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:23.255088+0000) 2024-03-06T09:35:24.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:24.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:24.255318+0000) 2024-03-06T09:35:25.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:25.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:25.255551+0000) 2024-03-06T09:35:26.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:26.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:26.255791+0000) 2024-03-06T09:35:27.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:27.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:27.256019+0000) 2024-03-06T09:35:28.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:28.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:28.256211+0000) 2024-03-06T09:35:29.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:29.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:29.256466+0000) 2024-03-06T09:35:30.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:30.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:30.256779+0000) 2024-03-06T09:35:31.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:31.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:31.257001+0000) 2024-03-06T09:35:32.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:32.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:32.257225+0000) 2024-03-06T09:35:33.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:33.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:33.257454+0000) 2024-03-06T09:35:34.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:34.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:34.257717+0000) 2024-03-06T09:35:35.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:35.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:35.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:35.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:35.257901+0000) 2024-03-06T09:35:36.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:36.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:36.258108+0000) 2024-03-06T09:35:37.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:37.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:37.258372+0000) 2024-03-06T09:35:38.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:38.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:38.258631+0000) 2024-03-06T09:35:39.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:39.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:39.258899+0000) 2024-03-06T09:35:40.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:40.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:40.259178+0000) 2024-03-06T09:35:41.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:41.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:41.259412+0000) 2024-03-06T09:35:42.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:42.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:42.259648+0000) 2024-03-06T09:35:43.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:43.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:43.259916+0000) 2024-03-06T09:35:44.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:44.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:44.260186+0000) 2024-03-06T09:35:45.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:45.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:45.260468+0000) 2024-03-06T09:35:46.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:46.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:46.260765+0000) 2024-03-06T09:35:47.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:47.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:47.260974+0000) 2024-03-06T09:35:48.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:48.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:48.261193+0000) 2024-03-06T09:35:49.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:49.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:49.261336+0000) 2024-03-06T09:35:50.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:50.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:50.261544+0000) 2024-03-06T09:35:51.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:51.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:51.261773+0000) 2024-03-06T09:35:52.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:52.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:52.261999+0000) 2024-03-06T09:35:53.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:53.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:53.262277+0000) 2024-03-06T09:35:54.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:54.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:54.262496+0000) 2024-03-06T09:35:55.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:55.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:55.262683+0000) 2024-03-06T09:35:56.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:56.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:56.262902+0000) 2024-03-06T09:35:57.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:57.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:57.263132+0000) 2024-03-06T09:35:58.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:58.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:58.263380+0000) 2024-03-06T09:35:59.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:35:59.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:59.263643+0000) 2024-03-06T09:36:00.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:00.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:00.263957+0000) 2024-03-06T09:36:01.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:01.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:01.264194+0000) 2024-03-06T09:36:02.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:02.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:02.264463+0000) 2024-03-06T09:36:03.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:03.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:03.264776+0000) 2024-03-06T09:36:04.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:04.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:04.265057+0000) 2024-03-06T09:36:05.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:05.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:05.265348+0000) 2024-03-06T09:36:05.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:05.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:06.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:06.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:06.265602+0000) 2024-03-06T09:36:07.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:07.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:07.265866+0000) 2024-03-06T09:36:08.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:08.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:08.266068+0000) 2024-03-06T09:36:09.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:09.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:09.266271+0000) 2024-03-06T09:36:10.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:10.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:10.266496+0000) 2024-03-06T09:36:11.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:11.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:11.266727+0000) 2024-03-06T09:36:12.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:12.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:12.266985+0000) 2024-03-06T09:36:13.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:13.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:13.267257+0000) 2024-03-06T09:36:14.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:14.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:14.267489+0000) 2024-03-06T09:36:15.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:15.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:15.267785+0000) 2024-03-06T09:36:16.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:16.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:16.267983+0000) 2024-03-06T09:36:17.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:17.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:17.268200+0000) 2024-03-06T09:36:18.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:18.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:18.268432+0000) 2024-03-06T09:36:19.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:19.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:19.268624+0000) 2024-03-06T09:36:20.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:20.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:20.268889+0000) 2024-03-06T09:36:21.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:21.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:21.269107+0000) 2024-03-06T09:36:22.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:22.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:22.269386+0000) 2024-03-06T09:36:23.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:23.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:23.269658+0000) 2024-03-06T09:36:24.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:24.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:24.269998+0000) 2024-03-06T09:36:25.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:25.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:25.270282+0000) 2024-03-06T09:36:26.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:26.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:26.270528+0000) 2024-03-06T09:36:27.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:27.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:27.270818+0000) 2024-03-06T09:36:28.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:28.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:28.271067+0000) 2024-03-06T09:36:29.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:29.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:29.271352+0000) 2024-03-06T09:36:30.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:30.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:30.271634+0000) 2024-03-06T09:36:31.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:31.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:31.271932+0000) 2024-03-06T09:36:32.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:32.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:32.272164+0000) 2024-03-06T09:36:33.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:33.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:33.272398+0000) 2024-03-06T09:36:34.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:34.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:34.272591+0000) 2024-03-06T09:36:35.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:35.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:35.272820+0000) 2024-03-06T09:36:35.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:35.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:36.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:36.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:36.273092+0000) 2024-03-06T09:36:37.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:37.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:37.273325+0000) 2024-03-06T09:36:38.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:38.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:38.273590+0000) 2024-03-06T09:36:39.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:39.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:39.273848+0000) 2024-03-06T09:36:40.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:40.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:40.274078+0000) 2024-03-06T09:36:41.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:41.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:41.274319+0000) 2024-03-06T09:36:42.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:42.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:42.274552+0000) 2024-03-06T09:36:43.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:43.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:43.274788+0000) 2024-03-06T09:36:44.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:44.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:44.275016+0000) 2024-03-06T09:36:45.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:45.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:45.275253+0000) 2024-03-06T09:36:46.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:46.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:46.275447+0000) 2024-03-06T09:36:47.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:47.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:47.275737+0000) 2024-03-06T09:36:48.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:48.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:48.276023+0000) 2024-03-06T09:36:49.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:49.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:49.276223+0000) 2024-03-06T09:36:50.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:50.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:50.276495+0000) 2024-03-06T09:36:51.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:51.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:51.276779+0000) 2024-03-06T09:36:52.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:52.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:52.277020+0000) 2024-03-06T09:36:53.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:53.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:53.277245+0000) 2024-03-06T09:36:54.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:54.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:54.277466+0000) 2024-03-06T09:36:55.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:55.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:55.277754+0000) 2024-03-06T09:36:56.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:56.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:56.277988+0000) 2024-03-06T09:36:57.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:57.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:57.278241+0000) 2024-03-06T09:36:58.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:58.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:58.278449+0000) 2024-03-06T09:36:59.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:36:59.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:59.278711+0000) 2024-03-06T09:37:00.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:00.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:00.278904+0000) 2024-03-06T09:37:01.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:01.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:01.279154+0000) 2024-03-06T09:37:02.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:02.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:02.279357+0000) 2024-03-06T09:37:03.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:03.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:03.279571+0000) 2024-03-06T09:37:04.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:04.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:04.279843+0000) 2024-03-06T09:37:05.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:05.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:05.280075+0000) 2024-03-06T09:37:05.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:05.923 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:06.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:06.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:06.280340+0000) 2024-03-06T09:37:07.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:07.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:07.280614+0000) 2024-03-06T09:37:08.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:08.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:08.280860+0000) 2024-03-06T09:37:09.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:09.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:09.281119+0000) 2024-03-06T09:37:10.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:10.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:10.281393+0000) 2024-03-06T09:37:11.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:11.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:11.281712+0000) 2024-03-06T09:37:12.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:12.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:12.281986+0000) 2024-03-06T09:37:13.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:13.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:13.282223+0000) 2024-03-06T09:37:14.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:14.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:14.282490+0000) 2024-03-06T09:37:15.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:15.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:15.282815+0000) 2024-03-06T09:37:16.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:16.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:16.283051+0000) 2024-03-06T09:37:17.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:17.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:17.283322+0000) 2024-03-06T09:37:18.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:18.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:18.283595+0000) 2024-03-06T09:37:19.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:19.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:19.283854+0000) 2024-03-06T09:37:20.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:20.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:20.284099+0000) 2024-03-06T09:37:21.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:21.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:21.284340+0000) 2024-03-06T09:37:22.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:22.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:22.284552+0000) 2024-03-06T09:37:23.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:23.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:23.284832+0000) 2024-03-06T09:37:24.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:24.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:24.285111+0000) 2024-03-06T09:37:25.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:25.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:25.285346+0000) 2024-03-06T09:37:26.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:26.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:26.285618+0000) 2024-03-06T09:37:27.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:27.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:27.285888+0000) 2024-03-06T09:37:28.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:28.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:28.286130+0000) 2024-03-06T09:37:29.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:29.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:29.286329+0000) 2024-03-06T09:37:30.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:30.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:30.286568+0000) 2024-03-06T09:37:31.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:31.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:31.286810+0000) 2024-03-06T09:37:32.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:32.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:32.287051+0000) 2024-03-06T09:37:33.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:33.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:33.287333+0000) 2024-03-06T09:37:34.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:34.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:34.287613+0000) 2024-03-06T09:37:35.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:35.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:35.287945+0000) 2024-03-06T09:37:36.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:36.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:36.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:36.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:36.288190+0000) 2024-03-06T09:37:37.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:37.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:37.288472+0000) 2024-03-06T09:37:38.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:38.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:38.288790+0000) 2024-03-06T09:37:39.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:39.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:39.289071+0000) 2024-03-06T09:37:40.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:40.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:40.289234+0000) 2024-03-06T09:37:41.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:41.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:41.289470+0000) 2024-03-06T09:37:42.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:42.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:42.289777+0000) 2024-03-06T09:37:43.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:43.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:43.289968+0000) 2024-03-06T09:37:44.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:44.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:44.290197+0000) 2024-03-06T09:37:45.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:45.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:45.290451+0000) 2024-03-06T09:37:46.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:46.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:46.290736+0000) 2024-03-06T09:37:47.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:47.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:47.290956+0000) 2024-03-06T09:37:48.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:48.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:48.291194+0000) 2024-03-06T09:37:49.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:49.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:49.291439+0000) 2024-03-06T09:37:50.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:50.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:50.291726+0000) 2024-03-06T09:37:51.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:51.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:51.291925+0000) 2024-03-06T09:37:52.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:52.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:52.292166+0000) 2024-03-06T09:37:53.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:53.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:53.292414+0000) 2024-03-06T09:37:54.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:54.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:54.292634+0000) 2024-03-06T09:37:55.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:55.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:55.292952+0000) 2024-03-06T09:37:56.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:56.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:56.293230+0000) 2024-03-06T09:37:57.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:57.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:57.293506+0000) 2024-03-06T09:37:58.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:58.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:58.293765+0000) 2024-03-06T09:37:59.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:37:59.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:59.294008+0000) 2024-03-06T09:38:00.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:00.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:00.294284+0000) 2024-03-06T09:38:01.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:01.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:01.294514+0000) 2024-03-06T09:38:02.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:02.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:02.294829+0000) 2024-03-06T09:38:03.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:03.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:03.295102+0000) 2024-03-06T09:38:04.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:04.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:04.295385+0000) 2024-03-06T09:38:05.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:05.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:05.295596+0000) 2024-03-06T09:38:06.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:06.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:06.295817+0000) 2024-03-06T09:38:06.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:06.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:07.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:07.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:07.296048+0000) 2024-03-06T09:38:08.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:08.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:08.296243+0000) 2024-03-06T09:38:09.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:09.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:09.296440+0000) 2024-03-06T09:38:10.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:10.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:10.296712+0000) 2024-03-06T09:38:11.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:11.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:11.296903+0000) 2024-03-06T09:38:12.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:12.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:12.297104+0000) 2024-03-06T09:38:13.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:13.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:13.297347+0000) 2024-03-06T09:38:14.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:14.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:14.297541+0000) 2024-03-06T09:38:15.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:15.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:15.297765+0000) 2024-03-06T09:38:16.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:16.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:16.297941+0000) 2024-03-06T09:38:17.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:17.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:17.298159+0000) 2024-03-06T09:38:18.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:18.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:18.298381+0000) 2024-03-06T09:38:19.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:19.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:19.298565+0000) 2024-03-06T09:38:20.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:20.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:20.298778+0000) 2024-03-06T09:38:21.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:21.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:21.299010+0000) 2024-03-06T09:38:22.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:22.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:22.299238+0000) 2024-03-06T09:38:23.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:23.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:23.299466+0000) 2024-03-06T09:38:24.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:24.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:24.299724+0000) 2024-03-06T09:38:25.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:25.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:25.299937+0000) 2024-03-06T09:38:26.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:26.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:26.300118+0000) 2024-03-06T09:38:27.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:27.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:27.300342+0000) 2024-03-06T09:38:28.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:28.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:28.300590+0000) 2024-03-06T09:38:29.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:29.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:29.300847+0000) 2024-03-06T09:38:30.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:30.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:30.301088+0000) 2024-03-06T09:38:31.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:31.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:31.301325+0000) 2024-03-06T09:38:32.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:32.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:32.301560+0000) 2024-03-06T09:38:33.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:33.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:33.301812+0000) 2024-03-06T09:38:34.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:34.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:34.301994+0000) 2024-03-06T09:38:35.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:35.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:35.302235+0000) 2024-03-06T09:38:36.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:36.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:36.302479+0000) 2024-03-06T09:38:36.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:36.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:37.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:37.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:37.302767+0000) 2024-03-06T09:38:38.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:38.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:38.302984+0000) 2024-03-06T09:38:39.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:39.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:39.303216+0000) 2024-03-06T09:38:40.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:40.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:40.303451+0000) 2024-03-06T09:38:41.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:41.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:41.303728+0000) 2024-03-06T09:38:42.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:42.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:42.303983+0000) 2024-03-06T09:38:43.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:43.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:43.304227+0000) 2024-03-06T09:38:44.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:44.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:44.304461+0000) 2024-03-06T09:38:45.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:45.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:45.304750+0000) 2024-03-06T09:38:46.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:46.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:46.304996+0000) 2024-03-06T09:38:47.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:47.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:47.305186+0000) 2024-03-06T09:38:48.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:48.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:48.305414+0000) 2024-03-06T09:38:49.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:49.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:49.305628+0000) 2024-03-06T09:38:50.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:50.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:50.305870+0000) 2024-03-06T09:38:51.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:51.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:51.306094+0000) 2024-03-06T09:38:52.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:52.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:52.306316+0000) 2024-03-06T09:38:53.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:53.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:53.306537+0000) 2024-03-06T09:38:54.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:54.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:54.306815+0000) 2024-03-06T09:38:55.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:55.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:55.307088+0000) 2024-03-06T09:38:56.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:56.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:56.307276+0000) 2024-03-06T09:38:57.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:57.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:57.307480+0000) 2024-03-06T09:38:58.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:58.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:58.307749+0000) 2024-03-06T09:38:59.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:38:59.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:59.307977+0000) 2024-03-06T09:39:00.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:00.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:00.308206+0000) 2024-03-06T09:39:01.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:01.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:01.308430+0000) 2024-03-06T09:39:02.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:02.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:02.308624+0000) 2024-03-06T09:39:03.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:03.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:03.308835+0000) 2024-03-06T09:39:04.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:04.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:04.309039+0000) 2024-03-06T09:39:05.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:05.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:05.309344+0000) 2024-03-06T09:39:06.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:06.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:06.309571+0000) 2024-03-06T09:39:06.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:06.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:07.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:07.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:07.309820+0000) 2024-03-06T09:39:08.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:08.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:08.310098+0000) 2024-03-06T09:39:09.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:09.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:09.310369+0000) 2024-03-06T09:39:10.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:10.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:10.310636+0000) 2024-03-06T09:39:11.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:11.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:11.310960+0000) 2024-03-06T09:39:12.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:12.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:12.311228+0000) 2024-03-06T09:39:13.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:13.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:13.311502+0000) 2024-03-06T09:39:14.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:14.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:14.311743+0000) 2024-03-06T09:39:15.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:15.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:15.311976+0000) 2024-03-06T09:39:16.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:16.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:16.312182+0000) 2024-03-06T09:39:17.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:17.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:17.312458+0000) 2024-03-06T09:39:18.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:18.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:18.312756+0000) 2024-03-06T09:39:19.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:19.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:19.313037+0000) 2024-03-06T09:39:20.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:20.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:20.313241+0000) 2024-03-06T09:39:21.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:21.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:21.313485+0000) 2024-03-06T09:39:22.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:22.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:22.313756+0000) 2024-03-06T09:39:23.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:23.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:23.314000+0000) 2024-03-06T09:39:24.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:24.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:24.314277+0000) 2024-03-06T09:39:25.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:25.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:25.314551+0000) 2024-03-06T09:39:26.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:26.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:26.314783+0000) 2024-03-06T09:39:27.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:27.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:27.314977+0000) 2024-03-06T09:39:28.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:28.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:28.315155+0000) 2024-03-06T09:39:29.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:29.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:29.315380+0000) 2024-03-06T09:39:30.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:30.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:30.315604+0000) 2024-03-06T09:39:31.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:31.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:31.315847+0000) 2024-03-06T09:39:32.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:32.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:32.316075+0000) 2024-03-06T09:39:33.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:33.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:33.316276+0000) 2024-03-06T09:39:34.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:34.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:34.316547+0000) 2024-03-06T09:39:35.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:35.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:35.316831+0000) 2024-03-06T09:39:36.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:36.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:36.317112+0000) 2024-03-06T09:39:36.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:36.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:37.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:37.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:37.317386+0000) 2024-03-06T09:39:38.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:38.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:38.317656+0000) 2024-03-06T09:39:39.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:39.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:39.317957+0000) 2024-03-06T09:39:40.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:40.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:40.318219+0000) 2024-03-06T09:39:41.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:41.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:41.318414+0000) 2024-03-06T09:39:42.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:42.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:42.318644+0000) 2024-03-06T09:39:43.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:43.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:43.318914+0000) 2024-03-06T09:39:44.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:44.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:44.319143+0000) 2024-03-06T09:39:45.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:45.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:45.319332+0000) 2024-03-06T09:39:46.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:46.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:46.319568+0000) 2024-03-06T09:39:47.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:47.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:47.319812+0000) 2024-03-06T09:39:48.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:48.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:48.320085+0000) 2024-03-06T09:39:49.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:49.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:49.320324+0000) 2024-03-06T09:39:50.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:50.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:50.320515+0000) 2024-03-06T09:39:51.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:51.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:51.320758+0000) 2024-03-06T09:39:52.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:52.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:52.320946+0000) 2024-03-06T09:39:53.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:53.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:53.321177+0000) 2024-03-06T09:39:54.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:54.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:54.321401+0000) 2024-03-06T09:39:55.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:55.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:55.321554+0000) 2024-03-06T09:39:56.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:56.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:56.321779+0000) 2024-03-06T09:39:57.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:57.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:57.321976+0000) 2024-03-06T09:39:58.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:58.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:58.322224+0000) 2024-03-06T09:39:59.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:39:59.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:59.322354+0000) 2024-03-06T09:40:00.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:00.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:00.322551+0000) 2024-03-06T09:40:01.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:01.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:01.322826+0000) 2024-03-06T09:40:02.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:02.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:02.323106+0000) 2024-03-06T09:40:03.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:03.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:03.323337+0000) 2024-03-06T09:40:04.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:04.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:04.323563+0000) 2024-03-06T09:40:05.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:05.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:05.323829+0000) 2024-03-06T09:40:06.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:06.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:06.324036+0000) 2024-03-06T09:40:07.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:07.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:07.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:07.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:07.324247+0000) 2024-03-06T09:40:08.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:08.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:08.324417+0000) 2024-03-06T09:40:09.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:09.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:09.324642+0000) 2024-03-06T09:40:10.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:10.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:10.324945+0000) 2024-03-06T09:40:11.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:11.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:11.325203+0000) 2024-03-06T09:40:12.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:12.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:12.325456+0000) 2024-03-06T09:40:13.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:13.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:13.325750+0000) 2024-03-06T09:40:14.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:14.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:14.325988+0000) 2024-03-06T09:40:15.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:15.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:15.326263+0000) 2024-03-06T09:40:16.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:16.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:16.326501+0000) 2024-03-06T09:40:17.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:17.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:17.326777+0000) 2024-03-06T09:40:18.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:18.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:18.327002+0000) 2024-03-06T09:40:19.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:19.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:19.327231+0000) 2024-03-06T09:40:20.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:20.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:20.327451+0000) 2024-03-06T09:40:21.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:21.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:21.327736+0000) 2024-03-06T09:40:22.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:22.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:22.327974+0000) 2024-03-06T09:40:23.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:23.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:23.328212+0000) 2024-03-06T09:40:24.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:24.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:24.328437+0000) 2024-03-06T09:40:25.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:25.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:25.328720+0000) 2024-03-06T09:40:26.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:26.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:26.328919+0000) 2024-03-06T09:40:27.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:27.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:27.329158+0000) 2024-03-06T09:40:28.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:28.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:28.329357+0000) 2024-03-06T09:40:29.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:29.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:29.329524+0000) 2024-03-06T09:40:30.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:30.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:30.329786+0000) 2024-03-06T09:40:31.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:31.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:31.330072+0000) 2024-03-06T09:40:32.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:32.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:32.330356+0000) 2024-03-06T09:40:33.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:33.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:33.330559+0000) 2024-03-06T09:40:34.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:34.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:34.330793+0000) 2024-03-06T09:40:35.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:35.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:35.331086+0000) 2024-03-06T09:40:36.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:36.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:36.331273+0000) 2024-03-06T09:40:37.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:37.209 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:37.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:37.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:37.331425+0000) 2024-03-06T09:40:38.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:38.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:38.331646+0000) 2024-03-06T09:40:39.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:39.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:39.331911+0000) 2024-03-06T09:40:40.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:40.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:40.332157+0000) 2024-03-06T09:40:41.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:41.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:41.332361+0000) 2024-03-06T09:40:42.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:42.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:42.332603+0000) 2024-03-06T09:40:43.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:43.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:43.332847+0000) 2024-03-06T09:40:44.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:44.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:44.333070+0000) 2024-03-06T09:40:45.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:45.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:45.333314+0000) 2024-03-06T09:40:46.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:46.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:46.333498+0000) 2024-03-06T09:40:47.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:47.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:47.333722+0000) 2024-03-06T09:40:48.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:48.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:48.333967+0000) 2024-03-06T09:40:49.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:49.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:49.334155+0000) 2024-03-06T09:40:50.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:50.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:50.334376+0000) 2024-03-06T09:40:51.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:51.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:51.334597+0000) 2024-03-06T09:40:52.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:52.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:52.334862+0000) 2024-03-06T09:40:53.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:53.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:53.335070+0000) 2024-03-06T09:40:54.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:54.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:54.335286+0000) 2024-03-06T09:40:55.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:55.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:55.335474+0000) 2024-03-06T09:40:56.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:56.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:56.335733+0000) 2024-03-06T09:40:57.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:57.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:57.335967+0000) 2024-03-06T09:40:58.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:58.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:58.336195+0000) 2024-03-06T09:40:59.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:40:59.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:59.336418+0000) 2024-03-06T09:41:00.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:00.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:00.336604+0000) 2024-03-06T09:41:01.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:01.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:01.336819+0000) 2024-03-06T09:41:02.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:02.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:02.337038+0000) 2024-03-06T09:41:03.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:03.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:03.337271+0000) 2024-03-06T09:41:04.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:04.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:04.337478+0000) 2024-03-06T09:41:05.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:05.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:05.337760+0000) 2024-03-06T09:41:06.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:06.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:06.337984+0000) 2024-03-06T09:41:07.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:07.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:07.338206+0000) 2024-03-06T09:41:07.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:07.432 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:08.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:08.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:08.338436+0000) 2024-03-06T09:41:09.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:09.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:09.338617+0000) 2024-03-06T09:41:10.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:10.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:10.338837+0000) 2024-03-06T09:41:11.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:11.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:11.339066+0000) 2024-03-06T09:41:12.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:12.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:12.339246+0000) 2024-03-06T09:41:13.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:13.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:13.339485+0000) 2024-03-06T09:41:14.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:14.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:14.339697+0000) 2024-03-06T09:41:15.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:15.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:15.339852+0000) 2024-03-06T09:41:16.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:16.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:16.340021+0000) 2024-03-06T09:41:17.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:17.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:17.340215+0000) 2024-03-06T09:41:18.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:18.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:18.340459+0000) 2024-03-06T09:41:19.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:19.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:19.340722+0000) 2024-03-06T09:41:20.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:20.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:20.340917+0000) 2024-03-06T09:41:21.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:21.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:21.341149+0000) 2024-03-06T09:41:22.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:22.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:22.341324+0000) 2024-03-06T09:41:23.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:23.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:23.341542+0000) 2024-03-06T09:41:24.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:24.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:24.341774+0000) 2024-03-06T09:41:25.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:25.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:25.341968+0000) 2024-03-06T09:41:26.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:26.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:26.342232+0000) 2024-03-06T09:41:27.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:27.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:27.342447+0000) 2024-03-06T09:41:28.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:28.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:28.342733+0000) 2024-03-06T09:41:29.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:29.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:29.342911+0000) 2024-03-06T09:41:30.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:30.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:30.343083+0000) 2024-03-06T09:41:31.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:31.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:31.343275+0000) 2024-03-06T09:41:32.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:32.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:32.343500+0000) 2024-03-06T09:41:33.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:33.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:33.343732+0000) 2024-03-06T09:41:34.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:34.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:34.344031+0000) 2024-03-06T09:41:35.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:35.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:35.344266+0000) 2024-03-06T09:41:36.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:36.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:36.344456+0000) 2024-03-06T09:41:37.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:37.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:37.344606+0000) 2024-03-06T09:41:37.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:37.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:38.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:38.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:38.344837+0000) 2024-03-06T09:41:39.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:39.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:39.345055+0000) 2024-03-06T09:41:40.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:40.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:40.345338+0000) 2024-03-06T09:41:41.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:41.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:41.345586+0000) 2024-03-06T09:41:42.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:42.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:42.345814+0000) 2024-03-06T09:41:43.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:43.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:43.346016+0000) 2024-03-06T09:41:44.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:44.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:44.346239+0000) 2024-03-06T09:41:45.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:45.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:45.346436+0000) 2024-03-06T09:41:46.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:46.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:46.346695+0000) 2024-03-06T09:41:47.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:47.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:47.346896+0000) 2024-03-06T09:41:48.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:48.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:48.347134+0000) 2024-03-06T09:41:49.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:49.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:49.347369+0000) 2024-03-06T09:41:50.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:50.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:50.347592+0000) 2024-03-06T09:41:51.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:51.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:51.347820+0000) 2024-03-06T09:41:52.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:52.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:52.348059+0000) 2024-03-06T09:41:53.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:53.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:53.348254+0000) 2024-03-06T09:41:54.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:54.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:54.348463+0000) 2024-03-06T09:41:55.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:55.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:55.348698+0000) 2024-03-06T09:41:56.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:56.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:56.348886+0000) 2024-03-06T09:41:57.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:57.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:57.349080+0000) 2024-03-06T09:41:58.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:58.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:58.349333+0000) 2024-03-06T09:41:59.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:41:59.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:59.349569+0000) 2024-03-06T09:42:00.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:00.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:00.349799+0000) 2024-03-06T09:42:01.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:01.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:01.350031+0000) 2024-03-06T09:42:02.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:02.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:02.350225+0000) 2024-03-06T09:42:03.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:03.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:03.350476+0000) 2024-03-06T09:42:04.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:04.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:04.350729+0000) 2024-03-06T09:42:05.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:05.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:05.350938+0000) 2024-03-06T09:42:06.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:06.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:06.351176+0000) 2024-03-06T09:42:07.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:07.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:07.351380+0000) 2024-03-06T09:42:07.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:07.749 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:08.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:08.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:08.351557+0000) 2024-03-06T09:42:09.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:09.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:09.351786+0000) 2024-03-06T09:42:10.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:10.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:10.351963+0000) 2024-03-06T09:42:11.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:11.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:11.352158+0000) 2024-03-06T09:42:12.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:12.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:12.352364+0000) 2024-03-06T09:42:13.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:13.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:13.352562+0000) 2024-03-06T09:42:14.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:14.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:14.352798+0000) 2024-03-06T09:42:15.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:15.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:15.353043+0000) 2024-03-06T09:42:16.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:16.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:16.353265+0000) 2024-03-06T09:42:17.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:17.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:17.353499+0000) 2024-03-06T09:42:18.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:18.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:18.353798+0000) 2024-03-06T09:42:19.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:19.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:19.354037+0000) 2024-03-06T09:42:20.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:20.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:20.354257+0000) 2024-03-06T09:42:21.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:21.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:21.354444+0000) 2024-03-06T09:42:22.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:22.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:22.354703+0000) 2024-03-06T09:42:23.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:23.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:23.354946+0000) 2024-03-06T09:42:24.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:24.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:24.355109+0000) 2024-03-06T09:42:25.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:25.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:25.355333+0000) 2024-03-06T09:42:26.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:26.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:26.355572+0000) 2024-03-06T09:42:27.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:27.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:27.355820+0000) 2024-03-06T09:42:28.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:28.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:28.356081+0000) 2024-03-06T09:42:29.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:29.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:29.356310+0000) 2024-03-06T09:42:30.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:30.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:30.356532+0000) 2024-03-06T09:42:31.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:31.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:31.356788+0000) 2024-03-06T09:42:32.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:32.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:32.357010+0000) 2024-03-06T09:42:33.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:33.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:33.357266+0000) 2024-03-06T09:42:34.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:34.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:34.357509+0000) 2024-03-06T09:42:35.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:35.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:35.357783+0000) 2024-03-06T09:42:36.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:36.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:36.358023+0000) 2024-03-06T09:42:37.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:37.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:37.358233+0000) 2024-03-06T09:42:37.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:37.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:38.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:38.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:38.358495+0000) 2024-03-06T09:42:39.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:39.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:39.358777+0000) 2024-03-06T09:42:40.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:40.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:40.359024+0000) 2024-03-06T09:42:41.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:41.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:41.359235+0000) 2024-03-06T09:42:42.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:42.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:42.359385+0000) 2024-03-06T09:42:43.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:43.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:43.359598+0000) 2024-03-06T09:42:44.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:44.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:44.359890+0000) 2024-03-06T09:42:45.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:45.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:45.360117+0000) 2024-03-06T09:42:46.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:46.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:46.360367+0000) 2024-03-06T09:42:47.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:47.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:47.360590+0000) 2024-03-06T09:42:48.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:48.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:48.360843+0000) 2024-03-06T09:42:49.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:49.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:49.361136+0000) 2024-03-06T09:42:50.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:50.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:50.361381+0000) 2024-03-06T09:42:51.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:51.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:51.361638+0000) 2024-03-06T09:42:52.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:52.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:52.361842+0000) 2024-03-06T09:42:53.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:53.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:53.362068+0000) 2024-03-06T09:42:54.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:54.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:54.362337+0000) 2024-03-06T09:42:55.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:55.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:55.362603+0000) 2024-03-06T09:42:56.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:56.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:56.362916+0000) 2024-03-06T09:42:57.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:57.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:57.363109+0000) 2024-03-06T09:42:58.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:58.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:58.363364+0000) 2024-03-06T09:42:59.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:42:59.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:59.363551+0000) 2024-03-06T09:43:00.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:00.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:00.363815+0000) 2024-03-06T09:43:01.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:01.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:01.364066+0000) 2024-03-06T09:43:02.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:02.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:02.364295+0000) 2024-03-06T09:43:03.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:03.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:03.364594+0000) 2024-03-06T09:43:04.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:04.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:04.364829+0000) 2024-03-06T09:43:05.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:05.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:05.365100+0000) 2024-03-06T09:43:06.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:06.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:06.365303+0000) 2024-03-06T09:43:07.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:07.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:07.365544+0000) 2024-03-06T09:43:08.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:08.101 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:08.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:08.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:08.365813+0000) 2024-03-06T09:43:09.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:09.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:09.366006+0000) 2024-03-06T09:43:10.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:10.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:10.366243+0000) 2024-03-06T09:43:11.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:11.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:11.366464+0000) 2024-03-06T09:43:12.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:12.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:12.366718+0000) 2024-03-06T09:43:13.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:13.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:13.366922+0000) 2024-03-06T09:43:14.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:14.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:14.367156+0000) 2024-03-06T09:43:15.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:15.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:15.367345+0000) 2024-03-06T09:43:16.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:16.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:16.367600+0000) 2024-03-06T09:43:17.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:17.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:17.367832+0000) 2024-03-06T09:43:18.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:18.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:18.368076+0000) 2024-03-06T09:43:19.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:19.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:19.368312+0000) 2024-03-06T09:43:20.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:20.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:20.368578+0000) 2024-03-06T09:43:21.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:21.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:21.368875+0000) 2024-03-06T09:43:22.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:22.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:22.369076+0000) 2024-03-06T09:43:23.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:23.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:23.369334+0000) 2024-03-06T09:43:24.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:24.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:24.369572+0000) 2024-03-06T09:43:25.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:25.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:25.369795+0000) 2024-03-06T09:43:26.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:26.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:26.370024+0000) 2024-03-06T09:43:27.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:27.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:27.370291+0000) 2024-03-06T09:43:28.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:28.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:28.370544+0000) 2024-03-06T09:43:29.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:29.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:29.370778+0000) 2024-03-06T09:43:30.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:30.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:30.370957+0000) 2024-03-06T09:43:31.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:31.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:31.371095+0000) 2024-03-06T09:43:32.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:32.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:32.371325+0000) 2024-03-06T09:43:33.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:33.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:33.371527+0000) 2024-03-06T09:43:34.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:34.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:34.371809+0000) 2024-03-06T09:43:35.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:35.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:35.372050+0000) 2024-03-06T09:43:36.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:36.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:36.372249+0000) 2024-03-06T09:43:37.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:37.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:37.372492+0000) 2024-03-06T09:43:38.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:38.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:38.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:38.372768+0000) 2024-03-06T09:43:38.384 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:39.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:39.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:39.372960+0000) 2024-03-06T09:43:40.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:40.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:40.373149+0000) 2024-03-06T09:43:41.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:41.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:41.373371+0000) 2024-03-06T09:43:42.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:42.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:42.373589+0000) 2024-03-06T09:43:43.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:43.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:43.373828+0000) 2024-03-06T09:43:44.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:44.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:44.374017+0000) 2024-03-06T09:43:45.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:45.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:45.374256+0000) 2024-03-06T09:43:46.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:46.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:46.374478+0000) 2024-03-06T09:43:47.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:47.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:47.374732+0000) 2024-03-06T09:43:48.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:48.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:48.374949+0000) 2024-03-06T09:43:49.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:49.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:49.375175+0000) 2024-03-06T09:43:50.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:50.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:50.375386+0000) 2024-03-06T09:43:51.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:51.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:51.375609+0000) 2024-03-06T09:43:52.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:52.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:52.375841+0000) 2024-03-06T09:43:53.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:53.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:53.375993+0000) 2024-03-06T09:43:54.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:54.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:54.376215+0000) 2024-03-06T09:43:55.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:55.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:55.376444+0000) 2024-03-06T09:43:56.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:56.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:56.376699+0000) 2024-03-06T09:43:57.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:57.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:57.376915+0000) 2024-03-06T09:43:58.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:58.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:58.377057+0000) 2024-03-06T09:43:59.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:43:59.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:59.377280+0000) 2024-03-06T09:44:00.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:00.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:00.377510+0000) 2024-03-06T09:44:01.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:01.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:01.377772+0000) 2024-03-06T09:44:02.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:02.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:02.377966+0000) 2024-03-06T09:44:03.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:03.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:03.378173+0000) 2024-03-06T09:44:04.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:04.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:04.378408+0000) 2024-03-06T09:44:05.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:05.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:05.378595+0000) 2024-03-06T09:44:06.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:06.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:06.378818+0000) 2024-03-06T09:44:07.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:07.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:07.379004+0000) 2024-03-06T09:44:08.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:08.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:08.379229+0000) 2024-03-06T09:44:08.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:08.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:09.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:09.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:09.379469+0000) 2024-03-06T09:44:10.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:10.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:10.379724+0000) 2024-03-06T09:44:11.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:11.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:11.379946+0000) 2024-03-06T09:44:12.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:12.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:12.380131+0000) 2024-03-06T09:44:13.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:13.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:13.380368+0000) 2024-03-06T09:44:14.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:14.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:14.380590+0000) 2024-03-06T09:44:15.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:15.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:15.380839+0000) 2024-03-06T09:44:16.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:16.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:16.381035+0000) 2024-03-06T09:44:17.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:17.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:17.381278+0000) 2024-03-06T09:44:18.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:18.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:18.381502+0000) 2024-03-06T09:44:19.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:19.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:19.381776+0000) 2024-03-06T09:44:20.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:20.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:20.381928+0000) 2024-03-06T09:44:21.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:21.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:21.382069+0000) 2024-03-06T09:44:22.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:22.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:22.382299+0000) 2024-03-06T09:44:23.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:23.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:23.382532+0000) 2024-03-06T09:44:24.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:24.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:24.382708+0000) 2024-03-06T09:44:25.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:25.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:25.382928+0000) 2024-03-06T09:44:26.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:26.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:26.383107+0000) 2024-03-06T09:44:27.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:27.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:27.383297+0000) 2024-03-06T09:44:28.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:28.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:28.383530+0000) 2024-03-06T09:44:29.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:29.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:29.383785+0000) 2024-03-06T09:44:30.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:30.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:30.383979+0000) 2024-03-06T09:44:31.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:31.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:31.384183+0000) 2024-03-06T09:44:32.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:32.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:32.384413+0000) 2024-03-06T09:44:33.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:33.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:33.384643+0000) 2024-03-06T09:44:34.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:34.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:34.384898+0000) 2024-03-06T09:44:35.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:35.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:35.385089+0000) 2024-03-06T09:44:36.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:36.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:36.385283+0000) 2024-03-06T09:44:37.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:37.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:37.385510+0000) 2024-03-06T09:44:38.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:38.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:38.385767+0000) 2024-03-06T09:44:38.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:38.818 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:39.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:39.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:39.385969+0000) 2024-03-06T09:44:40.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:40.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:40.386213+0000) 2024-03-06T09:44:41.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:41.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:41.386412+0000) 2024-03-06T09:44:42.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:42.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:42.386600+0000) 2024-03-06T09:44:43.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:43.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:43.386836+0000) 2024-03-06T09:44:44.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:44.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:44.387058+0000) 2024-03-06T09:44:45.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:45.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:45.387248+0000) 2024-03-06T09:44:46.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:46.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:46.387475+0000) 2024-03-06T09:44:47.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:47.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:47.387720+0000) 2024-03-06T09:44:48.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:48.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:48.387917+0000) 2024-03-06T09:44:49.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:49.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:49.388117+0000) 2024-03-06T09:44:50.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:50.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:50.388332+0000) 2024-03-06T09:44:51.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:51.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:51.388597+0000) 2024-03-06T09:44:52.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:52.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:52.388860+0000) 2024-03-06T09:44:53.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:53.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:53.389034+0000) 2024-03-06T09:44:54.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:54.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:54.389225+0000) 2024-03-06T09:44:55.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:55.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:55.389457+0000) 2024-03-06T09:44:56.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:56.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:56.389694+0000) 2024-03-06T09:44:57.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:57.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:57.389899+0000) 2024-03-06T09:44:58.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:58.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:58.390177+0000) 2024-03-06T09:44:59.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:44:59.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:59.390435+0000) 2024-03-06T09:45:00.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:00.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:00.390661+0000) 2024-03-06T09:45:01.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:01.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:01.390854+0000) 2024-03-06T09:45:02.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:02.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:02.391069+0000) 2024-03-06T09:45:03.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:03.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:03.391290+0000) 2024-03-06T09:45:04.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:04.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:04.391526+0000) 2024-03-06T09:45:05.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:05.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:05.391789+0000) 2024-03-06T09:45:06.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:06.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:06.392004+0000) 2024-03-06T09:45:07.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:07.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:07.392181+0000) 2024-03-06T09:45:08.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:08.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:08.392373+0000) 2024-03-06T09:45:09.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:09.040 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:09.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:09.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:09.392605+0000) 2024-03-06T09:45:10.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:10.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:10.392810+0000) 2024-03-06T09:45:11.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:11.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:11.393032+0000) 2024-03-06T09:45:12.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:12.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:12.393215+0000) 2024-03-06T09:45:13.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:13.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:13.393399+0000) 2024-03-06T09:45:14.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:14.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:14.393654+0000) 2024-03-06T09:45:15.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:15.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:15.393960+0000) 2024-03-06T09:45:16.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:16.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:16.394183+0000) 2024-03-06T09:45:17.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:17.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:17.394404+0000) 2024-03-06T09:45:18.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:18.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:18.394634+0000) 2024-03-06T09:45:19.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:19.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:19.394871+0000) 2024-03-06T09:45:20.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:20.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:20.395065+0000) 2024-03-06T09:45:21.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:21.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:21.395292+0000) 2024-03-06T09:45:22.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:22.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:22.395465+0000) 2024-03-06T09:45:23.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:23.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:23.395715+0000) 2024-03-06T09:45:24.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:24.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:24.395906+0000) 2024-03-06T09:45:25.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:25.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:25.396102+0000) 2024-03-06T09:45:26.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:26.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:26.396289+0000) 2024-03-06T09:45:27.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:27.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:27.396510+0000) 2024-03-06T09:45:28.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:28.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:28.396766+0000) 2024-03-06T09:45:29.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:29.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:29.396964+0000) 2024-03-06T09:45:30.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:30.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:30.397163+0000) 2024-03-06T09:45:31.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:31.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:31.397393+0000) 2024-03-06T09:45:32.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:32.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:32.397618+0000) 2024-03-06T09:45:33.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:33.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:33.397849+0000) 2024-03-06T09:45:34.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:34.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:34.398042+0000) 2024-03-06T09:45:35.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:35.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:35.398315+0000) 2024-03-06T09:45:36.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:36.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:36.398554+0000) 2024-03-06T09:45:37.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:37.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:37.398789+0000) 2024-03-06T09:45:38.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:38.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:38.399019+0000) 2024-03-06T09:45:39.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:39.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:39.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:39.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:39.399263+0000) 2024-03-06T09:45:40.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:40.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:40.399541+0000) 2024-03-06T09:45:41.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:41.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:41.399825+0000) 2024-03-06T09:45:42.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:42.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:42.400061+0000) 2024-03-06T09:45:43.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:43.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:43.400291+0000) 2024-03-06T09:45:44.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:44.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:44.400556+0000) 2024-03-06T09:45:45.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:45.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:45.400790+0000) 2024-03-06T09:45:46.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:46.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:46.401028+0000) 2024-03-06T09:45:47.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:47.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:47.401313+0000) 2024-03-06T09:45:48.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:48.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:48.401553+0000) 2024-03-06T09:45:49.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:49.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:49.401705+0000) 2024-03-06T09:45:50.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:50.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:50.401944+0000) 2024-03-06T09:45:51.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:51.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:51.402204+0000) 2024-03-06T09:45:52.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:52.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:52.402421+0000) 2024-03-06T09:45:53.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:53.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:53.402647+0000) 2024-03-06T09:45:54.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:54.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:54.402911+0000) 2024-03-06T09:45:55.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:55.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:55.403075+0000) 2024-03-06T09:45:56.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:56.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:56.403315+0000) 2024-03-06T09:45:57.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:57.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:57.403544+0000) 2024-03-06T09:45:58.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:58.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:58.403769+0000) 2024-03-06T09:45:59.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:45:59.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:59.403967+0000) 2024-03-06T09:46:00.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:00.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:00.404171+0000) 2024-03-06T09:46:01.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:01.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:01.404412+0000) 2024-03-06T09:46:02.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:02.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:02.404693+0000) 2024-03-06T09:46:03.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:03.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:03.404875+0000) 2024-03-06T09:46:04.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:04.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:04.405082+0000) 2024-03-06T09:46:05.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:05.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:05.405304+0000) 2024-03-06T09:46:06.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:06.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:06.405538+0000) 2024-03-06T09:46:07.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:07.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:07.405772+0000) 2024-03-06T09:46:08.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:08.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:08.406006+0000) 2024-03-06T09:46:09.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:09.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:09.406220+0000) 2024-03-06T09:46:09.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:09.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:10.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:10.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:10.406454+0000) 2024-03-06T09:46:11.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:11.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:11.406726+0000) 2024-03-06T09:46:12.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:12.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:12.406961+0000) 2024-03-06T09:46:13.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:13.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:13.407164+0000) 2024-03-06T09:46:14.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:14.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:14.407390+0000) 2024-03-06T09:46:15.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:15.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:15.407574+0000) 2024-03-06T09:46:16.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:16.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:16.407819+0000) 2024-03-06T09:46:17.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:17.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:17.408057+0000) 2024-03-06T09:46:18.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:18.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:18.408313+0000) 2024-03-06T09:46:19.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:19.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:19.408526+0000) 2024-03-06T09:46:20.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:20.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:20.408783+0000) 2024-03-06T09:46:21.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:21.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:21.409016+0000) 2024-03-06T09:46:22.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:22.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:22.409210+0000) 2024-03-06T09:46:23.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:23.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:23.409442+0000) 2024-03-06T09:46:24.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:24.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:24.409715+0000) 2024-03-06T09:46:25.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:25.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:25.409953+0000) 2024-03-06T09:46:26.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:26.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:26.410137+0000) 2024-03-06T09:46:27.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:27.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:27.410363+0000) 2024-03-06T09:46:28.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:28.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:28.410553+0000) 2024-03-06T09:46:29.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:29.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:29.410804+0000) 2024-03-06T09:46:30.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:30.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:30.410962+0000) 2024-03-06T09:46:31.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:31.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:31.411138+0000) 2024-03-06T09:46:32.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:32.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:32.411362+0000) 2024-03-06T09:46:33.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:33.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:33.411595+0000) 2024-03-06T09:46:34.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:34.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:34.411795+0000) 2024-03-06T09:46:35.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:35.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:35.412000+0000) 2024-03-06T09:46:36.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:36.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:36.412187+0000) 2024-03-06T09:46:37.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:37.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:37.412425+0000) 2024-03-06T09:46:38.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:38.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:38.412658+0000) 2024-03-06T09:46:39.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:39.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:39.412903+0000) 2024-03-06T09:46:39.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:39.587 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:40.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:40.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:40.413088+0000) 2024-03-06T09:46:41.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:41.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:41.413312+0000) 2024-03-06T09:46:42.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:42.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:42.413534+0000) 2024-03-06T09:46:43.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:43.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:43.413785+0000) 2024-03-06T09:46:44.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:44.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:44.414028+0000) 2024-03-06T09:46:45.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:45.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:45.414229+0000) 2024-03-06T09:46:46.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:46.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:46.414451+0000) 2024-03-06T09:46:47.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:47.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:47.414723+0000) 2024-03-06T09:46:48.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:48.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:48.414915+0000) 2024-03-06T09:46:49.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:49.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:49.415169+0000) 2024-03-06T09:46:50.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:50.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:50.415408+0000) 2024-03-06T09:46:51.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:51.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:51.415630+0000) 2024-03-06T09:46:52.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:52.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:52.415838+0000) 2024-03-06T09:46:53.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:53.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:53.416065+0000) 2024-03-06T09:46:54.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:54.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:54.416265+0000) 2024-03-06T09:46:55.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:55.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:55.416467+0000) 2024-03-06T09:46:56.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:56.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:56.416747+0000) 2024-03-06T09:46:57.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:57.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:57.416988+0000) 2024-03-06T09:46:58.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:58.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:58.417191+0000) 2024-03-06T09:46:59.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:46:59.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:59.417442+0000) 2024-03-06T09:47:00.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:00.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:00.417633+0000) 2024-03-06T09:47:01.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:01.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:01.417819+0000) 2024-03-06T09:47:02.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:02.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:02.418005+0000) 2024-03-06T09:47:03.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:03.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:03.418192+0000) 2024-03-06T09:47:04.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:04.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:04.418433+0000) 2024-03-06T09:47:05.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:05.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:05.418693+0000) 2024-03-06T09:47:06.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:06.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:06.418913+0000) 2024-03-06T09:47:07.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:07.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:07.419098+0000) 2024-03-06T09:47:08.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:08.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:08.419330+0000) 2024-03-06T09:47:09.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:09.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:09.419569+0000) 2024-03-06T09:47:09.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:09.780 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:10.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:10.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:10.419807+0000) 2024-03-06T09:47:11.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:11.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:11.420027+0000) 2024-03-06T09:47:12.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:12.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:12.420213+0000) 2024-03-06T09:47:13.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:13.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:13.420442+0000) 2024-03-06T09:47:14.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:14.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:14.420706+0000) 2024-03-06T09:47:15.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:15.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:15.420905+0000) 2024-03-06T09:47:16.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:16.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:16.421092+0000) 2024-03-06T09:47:17.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:17.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:17.421317+0000) 2024-03-06T09:47:18.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:18.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:18.421516+0000) 2024-03-06T09:47:19.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:19.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:19.421855+0000) 2024-03-06T09:47:20.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:20.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:20.422073+0000) 2024-03-06T09:47:21.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:21.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:21.422260+0000) 2024-03-06T09:47:22.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:22.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:22.422480+0000) 2024-03-06T09:47:23.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:23.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:23.422734+0000) 2024-03-06T09:47:24.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:24.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:24.422938+0000) 2024-03-06T09:47:25.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:25.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:25.423129+0000) 2024-03-06T09:47:26.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:26.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:26.423355+0000) 2024-03-06T09:47:27.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:27.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:27.423589+0000) 2024-03-06T09:47:28.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:28.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:28.423819+0000) 2024-03-06T09:47:29.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:29.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:29.424016+0000) 2024-03-06T09:47:30.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:30.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:30.424209+0000) 2024-03-06T09:47:31.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:31.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:31.424425+0000) 2024-03-06T09:47:32.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:32.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:32.424646+0000) 2024-03-06T09:47:33.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:33.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:33.424894+0000) 2024-03-06T09:47:34.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:34.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:34.425133+0000) 2024-03-06T09:47:35.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:35.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:35.425361+0000) 2024-03-06T09:47:36.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:36.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:36.425542+0000) 2024-03-06T09:47:37.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:37.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:37.425838+0000) 2024-03-06T09:47:38.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:38.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:38.426036+0000) 2024-03-06T09:47:39.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:39.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:39.426250+0000) 2024-03-06T09:47:39.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:39.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:40.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:40.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:40.426484+0000) 2024-03-06T09:47:41.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:41.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:41.426756+0000) 2024-03-06T09:47:42.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:42.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:42.426986+0000) 2024-03-06T09:47:43.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:43.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:43.427176+0000) 2024-03-06T09:47:44.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:44.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:44.427412+0000) 2024-03-06T09:47:45.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:45.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:45.427648+0000) 2024-03-06T09:47:46.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:46.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:46.427895+0000) 2024-03-06T09:47:47.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:47.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:47.428084+0000) 2024-03-06T09:47:48.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:48.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:48.428278+0000) 2024-03-06T09:47:49.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:49.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:49.428512+0000) 2024-03-06T09:47:50.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:50.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:50.428779+0000) 2024-03-06T09:47:51.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:51.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:51.429003+0000) 2024-03-06T09:47:52.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:52.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:52.429204+0000) 2024-03-06T09:47:53.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:53.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:53.429440+0000) 2024-03-06T09:47:54.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:54.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:54.429697+0000) 2024-03-06T09:47:55.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:55.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:55.429937+0000) 2024-03-06T09:47:56.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:56.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:56.430124+0000) 2024-03-06T09:47:57.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:57.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:57.430313+0000) 2024-03-06T09:47:58.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:58.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:58.430548+0000) 2024-03-06T09:47:59.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:47:59.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:59.430765+0000) 2024-03-06T09:48:00.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:00.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:00.430950+0000) 2024-03-06T09:48:01.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:01.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:01.431136+0000) 2024-03-06T09:48:02.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:02.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:02.431357+0000) 2024-03-06T09:48:03.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:03.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:03.431565+0000) 2024-03-06T09:48:04.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:04.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:04.431798+0000) 2024-03-06T09:48:05.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:05.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:05.432031+0000) 2024-03-06T09:48:06.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:06.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:06.432240+0000) 2024-03-06T09:48:07.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:07.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:07.432469+0000) 2024-03-06T09:48:08.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:08.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:08.432661+0000) 2024-03-06T09:48:09.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:09.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:09.432882+0000) 2024-03-06T09:48:10.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:10.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:10.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:10.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:10.433123+0000) 2024-03-06T09:48:11.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:11.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:11.433350+0000) 2024-03-06T09:48:12.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:12.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:12.433575+0000) 2024-03-06T09:48:13.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:13.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:13.433813+0000) 2024-03-06T09:48:14.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:14.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:14.433980+0000) 2024-03-06T09:48:15.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:15.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:15.434188+0000) 2024-03-06T09:48:16.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:16.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:16.434423+0000) 2024-03-06T09:48:17.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:17.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:17.434580+0000) 2024-03-06T09:48:18.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:18.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:18.434838+0000) 2024-03-06T09:48:19.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:19.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:19.435132+0000) 2024-03-06T09:48:20.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:20.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:20.435417+0000) 2024-03-06T09:48:21.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:21.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:21.435645+0000) 2024-03-06T09:48:22.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:22.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:22.435871+0000) 2024-03-06T09:48:23.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:23.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:23.436065+0000) 2024-03-06T09:48:24.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:24.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:24.436259+0000) 2024-03-06T09:48:25.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:25.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:25.436461+0000) 2024-03-06T09:48:26.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:26.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:26.436720+0000) 2024-03-06T09:48:27.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:27.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:27.436970+0000) 2024-03-06T09:48:28.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:28.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:28.437217+0000) 2024-03-06T09:48:29.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:29.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:29.437450+0000) 2024-03-06T09:48:30.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:30.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:30.437832+0000) 2024-03-06T09:48:31.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:31.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:31.438067+0000) 2024-03-06T09:48:32.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:32.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:32.438255+0000) 2024-03-06T09:48:33.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:33.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:33.438484+0000) 2024-03-06T09:48:34.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:34.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:34.438749+0000) 2024-03-06T09:48:35.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:35.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:35.438942+0000) 2024-03-06T09:48:36.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:36.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:36.439138+0000) 2024-03-06T09:48:37.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:37.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:37.439362+0000) 2024-03-06T09:48:38.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:38.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:38.439597+0000) 2024-03-06T09:48:39.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:39.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:39.439830+0000) 2024-03-06T09:48:40.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:40.314 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:40.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:40.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:40.440026+0000) 2024-03-06T09:48:41.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:41.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:41.440274+0000) 2024-03-06T09:48:42.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:42.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:42.440509+0000) 2024-03-06T09:48:43.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:43.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:43.440748+0000) 2024-03-06T09:48:44.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:44.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:44.440917+0000) 2024-03-06T09:48:45.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:45.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:45.441055+0000) 2024-03-06T09:48:46.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:46.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:46.441236+0000) 2024-03-06T09:48:47.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:47.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:47.441421+0000) 2024-03-06T09:48:48.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:48.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:48.441648+0000) 2024-03-06T09:48:49.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:49.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:49.441821+0000) 2024-03-06T09:48:50.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:50.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:50.442009+0000) 2024-03-06T09:48:51.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:51.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:51.442236+0000) 2024-03-06T09:48:52.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:52.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:52.442478+0000) 2024-03-06T09:48:53.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:53.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:53.442760+0000) 2024-03-06T09:48:54.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:54.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:54.443008+0000) 2024-03-06T09:48:55.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:55.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:55.443244+0000) 2024-03-06T09:48:56.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:56.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:56.443449+0000) 2024-03-06T09:48:57.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:57.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:57.443729+0000) 2024-03-06T09:48:58.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:58.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:58.443877+0000) 2024-03-06T09:48:59.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:48:59.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:59.444097+0000) 2024-03-06T09:49:00.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:00.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:00.444320+0000) 2024-03-06T09:49:01.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:01.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:01.444540+0000) 2024-03-06T09:49:02.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:02.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:02.444785+0000) 2024-03-06T09:49:03.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:03.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:03.445016+0000) 2024-03-06T09:49:04.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:04.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:04.445247+0000) 2024-03-06T09:49:05.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:05.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:05.445499+0000) 2024-03-06T09:49:06.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:06.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:06.445751+0000) 2024-03-06T09:49:07.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:07.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:07.445937+0000) 2024-03-06T09:49:08.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:08.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:08.446132+0000) 2024-03-06T09:49:09.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:09.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:09.446375+0000) 2024-03-06T09:49:10.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:10.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:10.446540+0000) 2024-03-06T09:49:10.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:10.539 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:11.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:11.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:11.446787+0000) 2024-03-06T09:49:12.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:12.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:12.447025+0000) 2024-03-06T09:49:13.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:13.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:13.447273+0000) 2024-03-06T09:49:14.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:14.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:14.447529+0000) 2024-03-06T09:49:15.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:15.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:15.447789+0000) 2024-03-06T09:49:16.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:16.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:16.447948+0000) 2024-03-06T09:49:17.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:17.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:17.448150+0000) 2024-03-06T09:49:18.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:18.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:18.448363+0000) 2024-03-06T09:49:19.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:19.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:19.448616+0000) 2024-03-06T09:49:20.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:20.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:20.448829+0000) 2024-03-06T09:49:21.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:21.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:21.449058+0000) 2024-03-06T09:49:22.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:22.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:22.449278+0000) 2024-03-06T09:49:23.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:23.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:23.449496+0000) 2024-03-06T09:49:24.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:24.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:24.449795+0000) 2024-03-06T09:49:25.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:25.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:25.449961+0000) 2024-03-06T09:49:26.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:26.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:26.450172+0000) 2024-03-06T09:49:27.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:27.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:27.450418+0000) 2024-03-06T09:49:28.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:28.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:28.450636+0000) 2024-03-06T09:49:29.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:29.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:29.450938+0000) 2024-03-06T09:49:30.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:30.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:30.451160+0000) 2024-03-06T09:49:31.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:31.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:31.451400+0000) 2024-03-06T09:49:32.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:32.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:32.451612+0000) 2024-03-06T09:49:33.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:33.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:33.451792+0000) 2024-03-06T09:49:34.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:34.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:34.451994+0000) 2024-03-06T09:49:35.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:35.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:35.452217+0000) 2024-03-06T09:49:36.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:36.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:36.452457+0000) 2024-03-06T09:49:37.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:37.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:37.452655+0000) 2024-03-06T09:49:38.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:38.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:38.452885+0000) 2024-03-06T09:49:39.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:39.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:39.453082+0000) 2024-03-06T09:49:40.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:40.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:40.453273+0000) 2024-03-06T09:49:40.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:40.678 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:41.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:41.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:41.453511+0000) 2024-03-06T09:49:42.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:42.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:42.453715+0000) 2024-03-06T09:49:43.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:43.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:43.453887+0000) 2024-03-06T09:49:44.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:44.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:44.454083+0000) 2024-03-06T09:49:45.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:45.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:45.454322+0000) 2024-03-06T09:49:46.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:46.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:46.454511+0000) 2024-03-06T09:49:47.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:47.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:47.454735+0000) 2024-03-06T09:49:48.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:48.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:48.454992+0000) 2024-03-06T09:49:49.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:49.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:49.455238+0000) 2024-03-06T09:49:50.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:50.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:50.455484+0000) 2024-03-06T09:49:51.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:51.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:51.455712+0000) 2024-03-06T09:49:52.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:52.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:52.455905+0000) 2024-03-06T09:49:53.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:53.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:53.456094+0000) 2024-03-06T09:49:54.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:54.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:54.456326+0000) 2024-03-06T09:49:55.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:55.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:55.456575+0000) 2024-03-06T09:49:56.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:56.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:56.456811+0000) 2024-03-06T09:49:57.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:57.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:57.457033+0000) 2024-03-06T09:49:58.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:58.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:58.457261+0000) 2024-03-06T09:49:59.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:49:59.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:59.457451+0000) 2024-03-06T09:50:00.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:00.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:00.457642+0000) 2024-03-06T09:50:01.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:01.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:01.457862+0000) 2024-03-06T09:50:02.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:02.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:02.458095+0000) 2024-03-06T09:50:03.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:03.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:03.458282+0000) 2024-03-06T09:50:04.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:04.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:04.458522+0000) 2024-03-06T09:50:05.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:05.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:05.458775+0000) 2024-03-06T09:50:06.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:06.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:06.459019+0000) 2024-03-06T09:50:07.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:07.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:07.459217+0000) 2024-03-06T09:50:08.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:08.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:08.459420+0000) 2024-03-06T09:50:09.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:09.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:09.459657+0000) 2024-03-06T09:50:10.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:10.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:10.459906+0000) 2024-03-06T09:50:10.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:10.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:11.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:11.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:11.460135+0000) 2024-03-06T09:50:12.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:12.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:12.460410+0000) 2024-03-06T09:50:13.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:13.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:13.460635+0000) 2024-03-06T09:50:14.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:14.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:14.460828+0000) 2024-03-06T09:50:15.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:15.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:15.461115+0000) 2024-03-06T09:50:16.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:16.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:16.461310+0000) 2024-03-06T09:50:17.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:17.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:17.461539+0000) 2024-03-06T09:50:18.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:18.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:18.461791+0000) 2024-03-06T09:50:19.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:19.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:19.462005+0000) 2024-03-06T09:50:20.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:20.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:20.462206+0000) 2024-03-06T09:50:21.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:21.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:21.462436+0000) 2024-03-06T09:50:22.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:22.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:22.462698+0000) 2024-03-06T09:50:23.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:23.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:23.462899+0000) 2024-03-06T09:50:24.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:24.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:24.463101+0000) 2024-03-06T09:50:25.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:25.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:25.463345+0000) 2024-03-06T09:50:26.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:26.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:26.463529+0000) 2024-03-06T09:50:27.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:27.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:27.463791+0000) 2024-03-06T09:50:28.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:28.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:28.463980+0000) 2024-03-06T09:50:29.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:29.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:29.464202+0000) 2024-03-06T09:50:30.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:30.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:30.464427+0000) 2024-03-06T09:50:31.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:31.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:31.464653+0000) 2024-03-06T09:50:32.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:32.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:32.464881+0000) 2024-03-06T09:50:33.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:33.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:33.465165+0000) 2024-03-06T09:50:34.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:34.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:34.465383+0000) 2024-03-06T09:50:35.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:35.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:35.465621+0000) 2024-03-06T09:50:36.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:36.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:36.465831+0000) 2024-03-06T09:50:37.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:37.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:37.466035+0000) 2024-03-06T09:50:38.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:38.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:38.466280+0000) 2024-03-06T09:50:39.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:39.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:39.466532+0000) 2024-03-06T09:50:40.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:40.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:40.466809+0000) 2024-03-06T09:50:40.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:41.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:41.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:41.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:41.467004+0000) 2024-03-06T09:50:42.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:42.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:42.467240+0000) 2024-03-06T09:50:43.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:43.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:43.467494+0000) 2024-03-06T09:50:44.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:44.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:44.467808+0000) 2024-03-06T09:50:45.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:45.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:45.468067+0000) 2024-03-06T09:50:46.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:46.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:46.468339+0000) 2024-03-06T09:50:47.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:47.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:47.468622+0000) 2024-03-06T09:50:48.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:48.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:48.468892+0000) 2024-03-06T09:50:49.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:49.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:49.469279+0000) 2024-03-06T09:50:50.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:50.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:50.469557+0000) 2024-03-06T09:50:51.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:51.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:51.469837+0000) 2024-03-06T09:50:52.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:52.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:52.470084+0000) 2024-03-06T09:50:53.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:53.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:53.470287+0000) 2024-03-06T09:50:54.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:54.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:54.470530+0000) 2024-03-06T09:50:55.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:55.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:55.470808+0000) 2024-03-06T09:50:56.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:56.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:56.471044+0000) 2024-03-06T09:50:57.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:57.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:57.471559+0000) 2024-03-06T09:50:58.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:58.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:58.471835+0000) 2024-03-06T09:50:59.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:50:59.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:59.472069+0000) 2024-03-06T09:51:00.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:00.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:00.472249+0000) 2024-03-06T09:51:01.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:01.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:01.472432+0000) 2024-03-06T09:51:02.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:02.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:02.472658+0000) 2024-03-06T09:51:03.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:03.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:03.472927+0000) 2024-03-06T09:51:04.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:04.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:04.473112+0000) 2024-03-06T09:51:05.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:05.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:05.473319+0000) 2024-03-06T09:51:06.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:06.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:06.473533+0000) 2024-03-06T09:51:07.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:07.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:07.473772+0000) 2024-03-06T09:51:08.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:08.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:08.474011+0000) 2024-03-06T09:51:09.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:09.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:09.474200+0000) 2024-03-06T09:51:10.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:10.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:10.474387+0000) 2024-03-06T09:51:11.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:11.156 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:11.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:11.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:11.474618+0000) 2024-03-06T09:51:12.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:12.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:12.474918+0000) 2024-03-06T09:51:13.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:13.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:13.475168+0000) 2024-03-06T09:51:14.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:14.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:14.475373+0000) 2024-03-06T09:51:15.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:15.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:15.475617+0000) 2024-03-06T09:51:16.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:16.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:16.475850+0000) 2024-03-06T09:51:17.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:17.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:17.476087+0000) 2024-03-06T09:51:18.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:18.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:18.476271+0000) 2024-03-06T09:51:19.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:19.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:19.476505+0000) 2024-03-06T09:51:20.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:20.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:20.476791+0000) 2024-03-06T09:51:21.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:21.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:21.477067+0000) 2024-03-06T09:51:22.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:22.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:22.477323+0000) 2024-03-06T09:51:23.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:23.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:23.477561+0000) 2024-03-06T09:51:24.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:24.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:24.477782+0000) 2024-03-06T09:51:25.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:25.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:25.478022+0000) 2024-03-06T09:51:26.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:26.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:26.478208+0000) 2024-03-06T09:51:27.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:27.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:27.478395+0000) 2024-03-06T09:51:28.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:28.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:28.478645+0000) 2024-03-06T09:51:29.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:29.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:29.478972+0000) 2024-03-06T09:51:30.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:30.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:30.479186+0000) 2024-03-06T09:51:31.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:31.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:31.479410+0000) 2024-03-06T09:51:32.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:32.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:32.479633+0000) 2024-03-06T09:51:33.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:33.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:33.479876+0000) 2024-03-06T09:51:34.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:34.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:34.480073+0000) 2024-03-06T09:51:35.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:35.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:35.480275+0000) 2024-03-06T09:51:36.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:36.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:36.480459+0000) 2024-03-06T09:51:37.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:37.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:37.480823+0000) 2024-03-06T09:51:38.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:38.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:38.481103+0000) 2024-03-06T09:51:39.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:39.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:39.481383+0000) 2024-03-06T09:51:40.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:40.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:40.481659+0000) 2024-03-06T09:51:41.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:41.348 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:41.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:41.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:41.481970+0000) 2024-03-06T09:51:42.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:42.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:42.482203+0000) 2024-03-06T09:51:43.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:43.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:43.482427+0000) 2024-03-06T09:51:44.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:44.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:44.482554+0000) 2024-03-06T09:51:45.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:45.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:45.482880+0000) 2024-03-06T09:51:46.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:46.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:46.483118+0000) 2024-03-06T09:51:47.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:47.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:47.483339+0000) 2024-03-06T09:51:48.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:48.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:48.483568+0000) 2024-03-06T09:51:49.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:49.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:49.483805+0000) 2024-03-06T09:51:50.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:50.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:50.484030+0000) 2024-03-06T09:51:51.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:51.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:51.484254+0000) 2024-03-06T09:51:52.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:52.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:52.484463+0000) 2024-03-06T09:51:53.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:53.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:53.484688+0000) 2024-03-06T09:51:54.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:54.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:54.484844+0000) 2024-03-06T09:51:55.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:55.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:55.485078+0000) 2024-03-06T09:51:56.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:56.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:56.485284+0000) 2024-03-06T09:51:57.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:57.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:57.485518+0000) 2024-03-06T09:51:58.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:58.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:58.485828+0000) 2024-03-06T09:51:59.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:51:59.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:59.486038+0000) 2024-03-06T09:52:00.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:00.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:00.486254+0000) 2024-03-06T09:52:01.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:01.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:01.486435+0000) 2024-03-06T09:52:02.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:02.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:02.486653+0000) 2024-03-06T09:52:03.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:03.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:03.486928+0000) 2024-03-06T09:52:04.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:04.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:04.487158+0000) 2024-03-06T09:52:05.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:05.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:05.487396+0000) 2024-03-06T09:52:06.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:06.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:06.487622+0000) 2024-03-06T09:52:07.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:07.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:07.487818+0000) 2024-03-06T09:52:08.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:08.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:08.488014+0000) 2024-03-06T09:52:09.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:09.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:09.488207+0000) 2024-03-06T09:52:10.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:10.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:10.488418+0000) 2024-03-06T09:52:11.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:11.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:11.488654+0000) 2024-03-06T09:52:11.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:11.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:12.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:12.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:12.488909+0000) 2024-03-06T09:52:13.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:13.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:13.489096+0000) 2024-03-06T09:52:14.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:14.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:14.489332+0000) 2024-03-06T09:52:15.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:15.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:15.489557+0000) 2024-03-06T09:52:16.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:16.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:16.489814+0000) 2024-03-06T09:52:17.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:17.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:17.490028+0000) 2024-03-06T09:52:18.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:18.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:18.490259+0000) 2024-03-06T09:52:19.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:19.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:19.490498+0000) 2024-03-06T09:52:20.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:20.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:20.490791+0000) 2024-03-06T09:52:21.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:21.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:21.491010+0000) 2024-03-06T09:52:22.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:22.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:22.491249+0000) 2024-03-06T09:52:23.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:23.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:23.491466+0000) 2024-03-06T09:52:24.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:24.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:24.491754+0000) 2024-03-06T09:52:25.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:25.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:25.491967+0000) 2024-03-06T09:52:26.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:26.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:26.492187+0000) 2024-03-06T09:52:27.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:27.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:27.492409+0000) 2024-03-06T09:52:28.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:28.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:28.492632+0000) 2024-03-06T09:52:29.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:29.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:29.492891+0000) 2024-03-06T09:52:30.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:30.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:30.493040+0000) 2024-03-06T09:52:31.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:31.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:31.493215+0000) 2024-03-06T09:52:32.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:32.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:32.493445+0000) 2024-03-06T09:52:33.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:33.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:33.493710+0000) 2024-03-06T09:52:34.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:34.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:34.493949+0000) 2024-03-06T09:52:35.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:35.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:35.494195+0000) 2024-03-06T09:52:36.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:36.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:36.494434+0000) 2024-03-06T09:52:37.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:37.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:37.494693+0000) 2024-03-06T09:52:38.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:38.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:38.494899+0000) 2024-03-06T09:52:39.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:39.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:39.495137+0000) 2024-03-06T09:52:40.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:40.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:40.495373+0000) 2024-03-06T09:52:41.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:41.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:41.495571+0000) 2024-03-06T09:52:41.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:41.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:42.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:42.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:42.495769+0000) 2024-03-06T09:52:43.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:43.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:43.495975+0000) 2024-03-06T09:52:44.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:44.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:44.496171+0000) 2024-03-06T09:52:45.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:45.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:45.496411+0000) 2024-03-06T09:52:46.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:46.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:46.496644+0000) 2024-03-06T09:52:47.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:47.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:47.496911+0000) 2024-03-06T09:52:48.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:48.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:48.497105+0000) 2024-03-06T09:52:49.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:49.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:49.497298+0000) 2024-03-06T09:52:50.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:50.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:50.497500+0000) 2024-03-06T09:52:51.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:51.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:51.497749+0000) 2024-03-06T09:52:52.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:52.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:52.497967+0000) 2024-03-06T09:52:53.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:53.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:53.498177+0000) 2024-03-06T09:52:54.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:54.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:54.498368+0000) 2024-03-06T09:52:55.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:55.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:55.498528+0000) 2024-03-06T09:52:56.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:56.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:56.498789+0000) 2024-03-06T09:52:57.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:57.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:57.498985+0000) 2024-03-06T09:52:58.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:58.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:58.499195+0000) 2024-03-06T09:52:59.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:52:59.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:59.499422+0000) 2024-03-06T09:53:00.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:00.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:00.499658+0000) 2024-03-06T09:53:01.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:01.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:01.499880+0000) 2024-03-06T09:53:02.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:02.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:02.500062+0000) 2024-03-06T09:53:03.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:03.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:03.500256+0000) 2024-03-06T09:53:04.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:04.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:04.500490+0000) 2024-03-06T09:53:05.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:05.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:05.500772+0000) 2024-03-06T09:53:06.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:06.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:06.500964+0000) 2024-03-06T09:53:07.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:07.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:07.501161+0000) 2024-03-06T09:53:08.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:08.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:08.501391+0000) 2024-03-06T09:53:09.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:09.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:09.501613+0000) 2024-03-06T09:53:10.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:10.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:10.501825+0000) 2024-03-06T09:53:11.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:11.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:11.502019+0000) 2024-03-06T09:53:11.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:11.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:12.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:12.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:12.502218+0000) 2024-03-06T09:53:13.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:13.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:13.502437+0000) 2024-03-06T09:53:14.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:14.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:14.502704+0000) 2024-03-06T09:53:15.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:15.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:15.502974+0000) 2024-03-06T09:53:16.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:16.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:16.503233+0000) 2024-03-06T09:53:17.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:17.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:17.503502+0000) 2024-03-06T09:53:18.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:18.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:18.503776+0000) 2024-03-06T09:53:19.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:19.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:19.503929+0000) 2024-03-06T09:53:20.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:20.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:20.504139+0000) 2024-03-06T09:53:21.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:21.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:21.504342+0000) 2024-03-06T09:53:22.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:22.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:22.504562+0000) 2024-03-06T09:53:23.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:23.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:23.504821+0000) 2024-03-06T09:53:24.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:24.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:24.505018+0000) 2024-03-06T09:53:25.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:25.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:25.505261+0000) 2024-03-06T09:53:26.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:26.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:26.505475+0000) 2024-03-06T09:53:27.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:27.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:27.505715+0000) 2024-03-06T09:53:28.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:28.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:28.505937+0000) 2024-03-06T09:53:29.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:29.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:29.506140+0000) 2024-03-06T09:53:30.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:30.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:30.506373+0000) 2024-03-06T09:53:31.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:31.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:31.506603+0000) 2024-03-06T09:53:32.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:32.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:32.506827+0000) 2024-03-06T09:53:33.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:33.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:33.507054+0000) 2024-03-06T09:53:34.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:34.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:34.507240+0000) 2024-03-06T09:53:35.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:35.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:35.507490+0000) 2024-03-06T09:53:36.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:36.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:36.507740+0000) 2024-03-06T09:53:37.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:37.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:37.507952+0000) 2024-03-06T09:53:38.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:38.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:38.508135+0000) 2024-03-06T09:53:39.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:39.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:39.508357+0000) 2024-03-06T09:53:40.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:40.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:40.508566+0000) 2024-03-06T09:53:41.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:41.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:41.508800+0000) 2024-03-06T09:53:41.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:41.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:42.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:42.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:42.508997+0000) 2024-03-06T09:53:43.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:43.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:43.509181+0000) 2024-03-06T09:53:44.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:44.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:44.509396+0000) 2024-03-06T09:53:45.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:45.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:45.509603+0000) 2024-03-06T09:53:46.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:46.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:46.509837+0000) 2024-03-06T09:53:47.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:47.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:47.510052+0000) 2024-03-06T09:53:48.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:48.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:48.510279+0000) 2024-03-06T09:53:49.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:49.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:49.510499+0000) 2024-03-06T09:53:50.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:50.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:50.510780+0000) 2024-03-06T09:53:51.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:51.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:51.511012+0000) 2024-03-06T09:53:52.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:52.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:52.511203+0000) 2024-03-06T09:53:53.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:53.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:53.511392+0000) 2024-03-06T09:53:54.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:54.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:54.511612+0000) 2024-03-06T09:53:55.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:55.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:55.511833+0000) 2024-03-06T09:53:56.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:56.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:56.512023+0000) 2024-03-06T09:53:57.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:57.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:57.512212+0000) 2024-03-06T09:53:58.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:58.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:58.512409+0000) 2024-03-06T09:53:59.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:53:59.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:59.512645+0000) 2024-03-06T09:54:00.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:00.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:00.512877+0000) 2024-03-06T09:54:01.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:01.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:01.513047+0000) 2024-03-06T09:54:02.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:02.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:02.513260+0000) 2024-03-06T09:54:03.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:03.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:03.513469+0000) 2024-03-06T09:54:04.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:04.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:04.513658+0000) 2024-03-06T09:54:05.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:05.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:05.513872+0000) 2024-03-06T09:54:06.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:06.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:06.514065+0000) 2024-03-06T09:54:07.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:07.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:07.514271+0000) 2024-03-06T09:54:08.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:08.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:08.514518+0000) 2024-03-06T09:54:09.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:09.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:09.514791+0000) 2024-03-06T09:54:10.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:10.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:10.515046+0000) 2024-03-06T09:54:11.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:11.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:11.515285+0000) 2024-03-06T09:54:12.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:12.205 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:12.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:12.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:12.515531+0000) 2024-03-06T09:54:13.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:13.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:13.515798+0000) 2024-03-06T09:54:14.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:14.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:14.516038+0000) 2024-03-06T09:54:15.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:15.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:15.516274+0000) 2024-03-06T09:54:16.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:16.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:16.516522+0000) 2024-03-06T09:54:17.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:17.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:17.516793+0000) 2024-03-06T09:54:18.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:18.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:18.516997+0000) 2024-03-06T09:54:19.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:19.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:19.517183+0000) 2024-03-06T09:54:20.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:20.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:20.517385+0000) 2024-03-06T09:54:21.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:21.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:21.517623+0000) 2024-03-06T09:54:22.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:22.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:22.517738+0000) 2024-03-06T09:54:23.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:23.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:23.517924+0000) 2024-03-06T09:54:24.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:24.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:24.518166+0000) 2024-03-06T09:54:25.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:25.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:25.518431+0000) 2024-03-06T09:54:26.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:26.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:26.518745+0000) 2024-03-06T09:54:27.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:27.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:27.518964+0000) 2024-03-06T09:54:28.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:28.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:28.519246+0000) 2024-03-06T09:54:29.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:29.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:29.519482+0000) 2024-03-06T09:54:30.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:30.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:30.519783+0000) 2024-03-06T09:54:31.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:31.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:31.519991+0000) 2024-03-06T09:54:32.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:32.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:32.520197+0000) 2024-03-06T09:54:33.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:33.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:33.520397+0000) 2024-03-06T09:54:34.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:34.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:34.520617+0000) 2024-03-06T09:54:35.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:35.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:35.520866+0000) 2024-03-06T09:54:36.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:36.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:36.521059+0000) 2024-03-06T09:54:37.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:37.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:37.521271+0000) 2024-03-06T09:54:38.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:38.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:38.521517+0000) 2024-03-06T09:54:39.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:39.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:39.521785+0000) 2024-03-06T09:54:40.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:40.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:40.522003+0000) 2024-03-06T09:54:41.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:41.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:41.522212+0000) 2024-03-06T09:54:42.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:42.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:42.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:42.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:42.522479+0000) 2024-03-06T09:54:43.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:43.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:43.522729+0000) 2024-03-06T09:54:44.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:44.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:44.522960+0000) 2024-03-06T09:54:45.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:45.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:45.523211+0000) 2024-03-06T09:54:46.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:46.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:46.523453+0000) 2024-03-06T09:54:47.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:47.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:47.523753+0000) 2024-03-06T09:54:48.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:48.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:48.524005+0000) 2024-03-06T09:54:49.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:49.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:49.524206+0000) 2024-03-06T09:54:50.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:50.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:50.524454+0000) 2024-03-06T09:54:51.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:51.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:51.524707+0000) 2024-03-06T09:54:52.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:52.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:52.524932+0000) 2024-03-06T09:54:53.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:53.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:53.525161+0000) 2024-03-06T09:54:54.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:54.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:54.525383+0000) 2024-03-06T09:54:55.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:55.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:55.525631+0000) 2024-03-06T09:54:56.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:56.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:56.525822+0000) 2024-03-06T09:54:57.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:57.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:57.525977+0000) 2024-03-06T09:54:58.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:58.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:58.526169+0000) 2024-03-06T09:54:59.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:54:59.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:59.526406+0000) 2024-03-06T09:55:00.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:00.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:00.526653+0000) 2024-03-06T09:55:01.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:01.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:01.526872+0000) 2024-03-06T09:55:02.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:02.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:02.527047+0000) 2024-03-06T09:55:03.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:03.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:03.527233+0000) 2024-03-06T09:55:04.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:04.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:04.527470+0000) 2024-03-06T09:55:05.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:05.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:05.527771+0000) 2024-03-06T09:55:06.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:06.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:06.527993+0000) 2024-03-06T09:55:07.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:07.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:07.528187+0000) 2024-03-06T09:55:08.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:08.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:08.528428+0000) 2024-03-06T09:55:09.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:09.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:09.528718+0000) 2024-03-06T09:55:10.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:10.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:10.528990+0000) 2024-03-06T09:55:11.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:11.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:11.529233+0000) 2024-03-06T09:55:12.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:12.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:12.529502+0000) 2024-03-06T09:55:12.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:12.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:13.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:13.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:13.529724+0000) 2024-03-06T09:55:14.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:14.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:14.529939+0000) 2024-03-06T09:55:15.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:15.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:15.530204+0000) 2024-03-06T09:55:16.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:16.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:16.530423+0000) 2024-03-06T09:55:17.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:17.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:17.530692+0000) 2024-03-06T09:55:18.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:18.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:18.530961+0000) 2024-03-06T09:55:19.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:19.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:19.531208+0000) 2024-03-06T09:55:20.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:20.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:20.531499+0000) 2024-03-06T09:55:21.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:21.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:21.531804+0000) 2024-03-06T09:55:22.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:22.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:22.532053+0000) 2024-03-06T09:55:23.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:23.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:23.532309+0000) 2024-03-06T09:55:24.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:24.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:24.532579+0000) 2024-03-06T09:55:25.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:25.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:25.532862+0000) 2024-03-06T09:55:26.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:26.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:26.533161+0000) 2024-03-06T09:55:27.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:27.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:27.533407+0000) 2024-03-06T09:55:28.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:28.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:28.533633+0000) 2024-03-06T09:55:29.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:29.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:29.533862+0000) 2024-03-06T09:55:30.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:30.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:30.534088+0000) 2024-03-06T09:55:31.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:31.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:31.534380+0000) 2024-03-06T09:55:32.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:32.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:32.534662+0000) 2024-03-06T09:55:33.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:33.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:33.534929+0000) 2024-03-06T09:55:34.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:34.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:34.535149+0000) 2024-03-06T09:55:35.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:35.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:35.535432+0000) 2024-03-06T09:55:36.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:36.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:36.535642+0000) 2024-03-06T09:55:37.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:37.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:37.535889+0000) 2024-03-06T09:55:38.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:38.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:38.536092+0000) 2024-03-06T09:55:39.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:39.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:39.536367+0000) 2024-03-06T09:55:40.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:40.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:40.536647+0000) 2024-03-06T09:55:41.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:41.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:41.536964+0000) 2024-03-06T09:55:42.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:42.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:42.537242+0000) 2024-03-06T09:55:42.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:42.888 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:43.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:43.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:43.537518+0000) 2024-03-06T09:55:44.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:44.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:44.537809+0000) 2024-03-06T09:55:45.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:45.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:45.538108+0000) 2024-03-06T09:55:46.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:46.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:46.538339+0000) 2024-03-06T09:55:47.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:47.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:47.538541+0000) 2024-03-06T09:55:48.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:48.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:48.538778+0000) 2024-03-06T09:55:49.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:49.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:49.538969+0000) 2024-03-06T09:55:50.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:50.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:50.539197+0000) 2024-03-06T09:55:51.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:51.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:51.539425+0000) 2024-03-06T09:55:52.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:52.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:52.539660+0000) 2024-03-06T09:55:53.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:53.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:53.539938+0000) 2024-03-06T09:55:54.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:54.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:54.540157+0000) 2024-03-06T09:55:55.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:55.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:55.540382+0000) 2024-03-06T09:55:56.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:56.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:56.540605+0000) 2024-03-06T09:55:57.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:57.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:57.540831+0000) 2024-03-06T09:55:58.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:58.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:58.540978+0000) 2024-03-06T09:55:59.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:55:59.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:59.541175+0000) 2024-03-06T09:56:00.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:00.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:00.541373+0000) 2024-03-06T09:56:01.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:01.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:01.541570+0000) 2024-03-06T09:56:02.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:02.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:02.541842+0000) 2024-03-06T09:56:03.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:03.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:03.542125+0000) 2024-03-06T09:56:04.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:04.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:04.542401+0000) 2024-03-06T09:56:05.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:05.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:05.542703+0000) 2024-03-06T09:56:06.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:06.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:06.542949+0000) 2024-03-06T09:56:07.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:07.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:07.543182+0000) 2024-03-06T09:56:08.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:08.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:08.543424+0000) 2024-03-06T09:56:09.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:09.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:09.543709+0000) 2024-03-06T09:56:10.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:10.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:10.543915+0000) 2024-03-06T09:56:11.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:11.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:11.544161+0000) 2024-03-06T09:56:12.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:12.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:12.544438+0000) 2024-03-06T09:56:13.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:13.032 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:13.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:13.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:13.544756+0000) 2024-03-06T09:56:14.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:14.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:14.545039+0000) 2024-03-06T09:56:15.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:15.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:15.545280+0000) 2024-03-06T09:56:16.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:16.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:16.545527+0000) 2024-03-06T09:56:17.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:17.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:17.545828+0000) 2024-03-06T09:56:18.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:18.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:18.546041+0000) 2024-03-06T09:56:19.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:19.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:19.546231+0000) 2024-03-06T09:56:20.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:20.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:20.546495+0000) 2024-03-06T09:56:21.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:21.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:21.546787+0000) 2024-03-06T09:56:22.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:22.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:22.547034+0000) 2024-03-06T09:56:23.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:23.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:23.547266+0000) 2024-03-06T09:56:24.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:24.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:24.547545+0000) 2024-03-06T09:56:25.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:25.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:25.547846+0000) 2024-03-06T09:56:26.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:26.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:26.548080+0000) 2024-03-06T09:56:27.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:27.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:27.548322+0000) 2024-03-06T09:56:28.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:28.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:28.548558+0000) 2024-03-06T09:56:29.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:29.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:29.548830+0000) 2024-03-06T09:56:30.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:30.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:30.549113+0000) 2024-03-06T09:56:31.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:31.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:31.549392+0000) 2024-03-06T09:56:32.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:32.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:32.549586+0000) 2024-03-06T09:56:33.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:33.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:33.549824+0000) 2024-03-06T09:56:34.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:34.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:34.550049+0000) 2024-03-06T09:56:35.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:35.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:35.550238+0000) 2024-03-06T09:56:36.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:36.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:36.550462+0000) 2024-03-06T09:56:37.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:37.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:37.550755+0000) 2024-03-06T09:56:38.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:38.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:38.551001+0000) 2024-03-06T09:56:39.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:39.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:39.551242+0000) 2024-03-06T09:56:40.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:40.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:40.551518+0000) 2024-03-06T09:56:41.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:41.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:41.551804+0000) 2024-03-06T09:56:42.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:42.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:42.552007+0000) 2024-03-06T09:56:43.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:43.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:43.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:43.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:43.552221+0000) 2024-03-06T09:56:44.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:44.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:44.552449+0000) 2024-03-06T09:56:45.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:45.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:45.552711+0000) 2024-03-06T09:56:46.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:46.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:46.552936+0000) 2024-03-06T09:56:47.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:47.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:47.553151+0000) 2024-03-06T09:56:48.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:48.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:48.553437+0000) 2024-03-06T09:56:49.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:49.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:49.553768+0000) 2024-03-06T09:56:50.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:50.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:50.554051+0000) 2024-03-06T09:56:51.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:51.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:51.554250+0000) 2024-03-06T09:56:52.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:52.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:52.554506+0000) 2024-03-06T09:56:53.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:53.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:53.554824+0000) 2024-03-06T09:56:54.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:54.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:54.555101+0000) 2024-03-06T09:56:55.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:55.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:55.555399+0000) 2024-03-06T09:56:56.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:56.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:56.555606+0000) 2024-03-06T09:56:57.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:57.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:57.555798+0000) 2024-03-06T09:56:58.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:58.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:58.556007+0000) 2024-03-06T09:56:59.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:56:59.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:59.556224+0000) 2024-03-06T09:57:00.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:00.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:00.556502+0000) 2024-03-06T09:57:01.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:01.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:01.556795+0000) 2024-03-06T09:57:02.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:02.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:02.556988+0000) 2024-03-06T09:57:03.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:03.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:03.557222+0000) 2024-03-06T09:57:04.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:04.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:04.557447+0000) 2024-03-06T09:57:05.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:05.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:05.557769+0000) 2024-03-06T09:57:06.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:06.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:06.558039+0000) 2024-03-06T09:57:07.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:07.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:07.558309+0000) 2024-03-06T09:57:08.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:08.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:08.558598+0000) 2024-03-06T09:57:09.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:09.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:09.558818+0000) 2024-03-06T09:57:10.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:10.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:10.558997+0000) 2024-03-06T09:57:11.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:11.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:11.559202+0000) 2024-03-06T09:57:12.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:12.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:12.559428+0000) 2024-03-06T09:57:13.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:13.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:13.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:13.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:13.559716+0000) 2024-03-06T09:57:14.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:14.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:14.559961+0000) 2024-03-06T09:57:15.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:15.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:15.560190+0000) 2024-03-06T09:57:16.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:16.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:16.560419+0000) 2024-03-06T09:57:17.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:17.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:17.560617+0000) 2024-03-06T09:57:18.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:18.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:18.560791+0000) 2024-03-06T09:57:19.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:19.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:19.560984+0000) 2024-03-06T09:57:20.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:20.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:20.561182+0000) 2024-03-06T09:57:21.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:21.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:21.561393+0000) 2024-03-06T09:57:22.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:22.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:22.561590+0000) 2024-03-06T09:57:23.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:23.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:23.561835+0000) 2024-03-06T09:57:24.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:24.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:24.562027+0000) 2024-03-06T09:57:25.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:25.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:25.562271+0000) 2024-03-06T09:57:26.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:26.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:26.562500+0000) 2024-03-06T09:57:27.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:27.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:27.562725+0000) 2024-03-06T09:57:28.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:28.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:28.562953+0000) 2024-03-06T09:57:29.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:29.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:29.563147+0000) 2024-03-06T09:57:30.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:30.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:30.563371+0000) 2024-03-06T09:57:31.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:31.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:31.563555+0000) 2024-03-06T09:57:32.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:32.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:32.563789+0000) 2024-03-06T09:57:33.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:33.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:33.564016+0000) 2024-03-06T09:57:34.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:34.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:34.564217+0000) 2024-03-06T09:57:35.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:35.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:35.564422+0000) 2024-03-06T09:57:36.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:36.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:36.564658+0000) 2024-03-06T09:57:37.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:37.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:37.564945+0000) 2024-03-06T09:57:38.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:38.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:38.565174+0000) 2024-03-06T09:57:39.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:39.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:39.565303+0000) 2024-03-06T09:57:40.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:40.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:40.565476+0000) 2024-03-06T09:57:41.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:41.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:41.565742+0000) 2024-03-06T09:57:42.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:42.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:42.565972+0000) 2024-03-06T09:57:43.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:43.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:43.566200+0000) 2024-03-06T09:57:43.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:43.573 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:44.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:44.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:44.566390+0000) 2024-03-06T09:57:45.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:45.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:45.566642+0000) 2024-03-06T09:57:46.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:46.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:46.566913+0000) 2024-03-06T09:57:47.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:47.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:47.567100+0000) 2024-03-06T09:57:48.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:48.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:48.567279+0000) 2024-03-06T09:57:49.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:49.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:49.567451+0000) 2024-03-06T09:57:50.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:50.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:50.567646+0000) 2024-03-06T09:57:51.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:51.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:51.567868+0000) 2024-03-06T09:57:52.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:52.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:52.568114+0000) 2024-03-06T09:57:53.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:53.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:53.568364+0000) 2024-03-06T09:57:54.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:54.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:54.568590+0000) 2024-03-06T09:57:55.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:55.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:55.568838+0000) 2024-03-06T09:57:56.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:56.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:56.569079+0000) 2024-03-06T09:57:57.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:57.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:57.569370+0000) 2024-03-06T09:57:58.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:58.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:58.569599+0000) 2024-03-06T09:57:59.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:57:59.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:59.569841+0000) 2024-03-06T09:58:00.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:00.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:00.570031+0000) 2024-03-06T09:58:01.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:01.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:01.570286+0000) 2024-03-06T09:58:02.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:02.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:02.570529+0000) 2024-03-06T09:58:03.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:03.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:03.570765+0000) 2024-03-06T09:58:04.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:04.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:04.571003+0000) 2024-03-06T09:58:05.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:05.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:05.571205+0000) 2024-03-06T09:58:06.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:06.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:06.571397+0000) 2024-03-06T09:58:07.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:07.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:07.571582+0000) 2024-03-06T09:58:08.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:08.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:08.571842+0000) 2024-03-06T09:58:09.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:09.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:09.572070+0000) 2024-03-06T09:58:10.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:10.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:10.572306+0000) 2024-03-06T09:58:11.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:11.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:11.572552+0000) 2024-03-06T09:58:12.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:12.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:12.572799+0000) 2024-03-06T09:58:13.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:13.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:13.573044+0000) 2024-03-06T09:58:13.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:13.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:14.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:14.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:14.573289+0000) 2024-03-06T09:58:15.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:15.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:15.573519+0000) 2024-03-06T09:58:16.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:16.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:16.573792+0000) 2024-03-06T09:58:17.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:17.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:17.573950+0000) 2024-03-06T09:58:18.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:18.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:18.574171+0000) 2024-03-06T09:58:19.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:19.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:19.574393+0000) 2024-03-06T09:58:20.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:20.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:20.574614+0000) 2024-03-06T09:58:21.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:21.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:21.574833+0000) 2024-03-06T09:58:22.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:22.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:22.575045+0000) 2024-03-06T09:58:23.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:23.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:23.575241+0000) 2024-03-06T09:58:24.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:24.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:24.575452+0000) 2024-03-06T09:58:25.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:25.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:25.575701+0000) 2024-03-06T09:58:26.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:26.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:26.575947+0000) 2024-03-06T09:58:27.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:27.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:27.576142+0000) 2024-03-06T09:58:28.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:28.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:28.576382+0000) 2024-03-06T09:58:29.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:29.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:29.576602+0000) 2024-03-06T09:58:30.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:30.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:30.576783+0000) 2024-03-06T09:58:31.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:31.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:31.576991+0000) 2024-03-06T09:58:32.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:32.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:32.577225+0000) 2024-03-06T09:58:33.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:33.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:33.577458+0000) 2024-03-06T09:58:34.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:34.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:34.577710+0000) 2024-03-06T09:58:35.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:35.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:35.577943+0000) 2024-03-06T09:58:36.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:36.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:36.578169+0000) 2024-03-06T09:58:37.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:37.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:37.578399+0000) 2024-03-06T09:58:38.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:38.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:38.578581+0000) 2024-03-06T09:58:39.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:39.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:39.578788+0000) 2024-03-06T09:58:40.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:40.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:40.578987+0000) 2024-03-06T09:58:41.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:41.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:41.579174+0000) 2024-03-06T09:58:42.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:42.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:42.579408+0000) 2024-03-06T09:58:43.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:43.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:43.579637+0000) 2024-03-06T09:58:43.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:43.961 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:44.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:44.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:44.579909+0000) 2024-03-06T09:58:45.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:45.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:45.580142+0000) 2024-03-06T09:58:46.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:46.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:46.580376+0000) 2024-03-06T09:58:47.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:47.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:47.580604+0000) 2024-03-06T09:58:48.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:48.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:48.580830+0000) 2024-03-06T09:58:49.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:49.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:49.581112+0000) 2024-03-06T09:58:50.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:50.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:50.581292+0000) 2024-03-06T09:58:51.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:51.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:51.581537+0000) 2024-03-06T09:58:52.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:52.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:52.581797+0000) 2024-03-06T09:58:53.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:53.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:53.582001+0000) 2024-03-06T09:58:54.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:54.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:54.582230+0000) 2024-03-06T09:58:55.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:55.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:55.582464+0000) 2024-03-06T09:58:56.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:56.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:56.582781+0000) 2024-03-06T09:58:57.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:57.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:57.583076+0000) 2024-03-06T09:58:58.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:58.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:58.583352+0000) 2024-03-06T09:58:59.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:58:59.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:59.583554+0000) 2024-03-06T09:59:00.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:00.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:00.583785+0000) 2024-03-06T09:59:01.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:01.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:01.584024+0000) 2024-03-06T09:59:02.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:02.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:02.584303+0000) 2024-03-06T09:59:03.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:03.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:03.584510+0000) 2024-03-06T09:59:04.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:04.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:04.584774+0000) 2024-03-06T09:59:05.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:05.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:05.584961+0000) 2024-03-06T09:59:06.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:06.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:06.585202+0000) 2024-03-06T09:59:07.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:07.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:07.585422+0000) 2024-03-06T09:59:08.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:08.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:08.585642+0000) 2024-03-06T09:59:09.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:09.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:09.585896+0000) 2024-03-06T09:59:10.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:10.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:10.586111+0000) 2024-03-06T09:59:11.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:11.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:11.586336+0000) 2024-03-06T09:59:12.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:12.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:12.586562+0000) 2024-03-06T09:59:13.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:13.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:13.586800+0000) 2024-03-06T09:59:14.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:14.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:14.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:14.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:14.587041+0000) 2024-03-06T09:59:15.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:15.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:15.587315+0000) 2024-03-06T09:59:16.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:16.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:16.587526+0000) 2024-03-06T09:59:17.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:17.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:17.587781+0000) 2024-03-06T09:59:18.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:18.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:18.588002+0000) 2024-03-06T09:59:19.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:19.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:19.588227+0000) 2024-03-06T09:59:20.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:20.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:20.588444+0000) 2024-03-06T09:59:21.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:21.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:21.588727+0000) 2024-03-06T09:59:22.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:22.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:22.588929+0000) 2024-03-06T09:59:23.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:23.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:23.589145+0000) 2024-03-06T09:59:24.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:24.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:24.589360+0000) 2024-03-06T09:59:25.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:25.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:25.589588+0000) 2024-03-06T09:59:26.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:26.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:26.589831+0000) 2024-03-06T09:59:27.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:27.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:27.590024+0000) 2024-03-06T09:59:28.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:28.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:28.590280+0000) 2024-03-06T09:59:29.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:29.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:29.590533+0000) 2024-03-06T09:59:30.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:30.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:30.590825+0000) 2024-03-06T09:59:31.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:31.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:31.591050+0000) 2024-03-06T09:59:32.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:32.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:32.591261+0000) 2024-03-06T09:59:33.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:33.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:33.591484+0000) 2024-03-06T09:59:34.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:34.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:34.591802+0000) 2024-03-06T09:59:35.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:35.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:35.592090+0000) 2024-03-06T09:59:36.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:36.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:36.592349+0000) 2024-03-06T09:59:37.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:37.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:37.592576+0000) 2024-03-06T09:59:38.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:38.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:38.592853+0000) 2024-03-06T09:59:39.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:39.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:39.593133+0000) 2024-03-06T09:59:40.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:40.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:40.593418+0000) 2024-03-06T09:59:41.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:41.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:41.593707+0000) 2024-03-06T09:59:42.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:42.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:42.593988+0000) 2024-03-06T09:59:43.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:43.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:43.594218+0000) 2024-03-06T09:59:44.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:44.288 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:44.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:44.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:44.594455+0000) 2024-03-06T09:59:45.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:45.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:45.594756+0000) 2024-03-06T09:59:46.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:46.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:46.595004+0000) 2024-03-06T09:59:47.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:47.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:47.595188+0000) 2024-03-06T09:59:48.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:48.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:48.595425+0000) 2024-03-06T09:59:49.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:49.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:49.595634+0000) 2024-03-06T09:59:50.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:50.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:50.595915+0000) 2024-03-06T09:59:51.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:51.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:51.596154+0000) 2024-03-06T09:59:52.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:52.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:52.596350+0000) 2024-03-06T09:59:53.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:53.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:53.596560+0000) 2024-03-06T09:59:54.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:54.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:54.596794+0000) 2024-03-06T09:59:55.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:55.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:55.597022+0000) 2024-03-06T09:59:56.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:56.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:56.597263+0000) 2024-03-06T09:59:57.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:57.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:57.597480+0000) 2024-03-06T09:59:58.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:58.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:58.597706+0000) 2024-03-06T09:59:59.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T09:59:59.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:59.597915+0000) 2024-03-06T10:00:00.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:00.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:00.598112+0000) 2024-03-06T10:00:01.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:01.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:01.598353+0000) 2024-03-06T10:00:02.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:02.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:02.598541+0000) 2024-03-06T10:00:03.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:03.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:03.598788+0000) 2024-03-06T10:00:04.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:04.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:04.599064+0000) 2024-03-06T10:00:05.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:05.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:05.599301+0000) 2024-03-06T10:00:06.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:06.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:06.599551+0000) 2024-03-06T10:00:07.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:07.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:07.599778+0000) 2024-03-06T10:00:08.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:08.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:08.600007+0000) 2024-03-06T10:00:09.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:09.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:09.600204+0000) 2024-03-06T10:00:10.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:10.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:10.600430+0000) 2024-03-06T10:00:11.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:11.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:11.600703+0000) 2024-03-06T10:00:12.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:12.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:12.600928+0000) 2024-03-06T10:00:13.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:13.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:13.601154+0000) 2024-03-06T10:00:14.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:14.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:14.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:14.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:14.601400+0000) 2024-03-06T10:00:15.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:15.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:15.601634+0000) 2024-03-06T10:00:16.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:16.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:16.601877+0000) 2024-03-06T10:00:17.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:17.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:17.602112+0000) 2024-03-06T10:00:18.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:18.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:18.602346+0000) 2024-03-06T10:00:19.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:19.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:19.602527+0000) 2024-03-06T10:00:20.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:20.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:20.602783+0000) 2024-03-06T10:00:21.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:21.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:21.603003+0000) 2024-03-06T10:00:22.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:22.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:22.603230+0000) 2024-03-06T10:00:23.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:23.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:23.603415+0000) 2024-03-06T10:00:24.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:24.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:24.603612+0000) 2024-03-06T10:00:25.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:25.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:25.603875+0000) 2024-03-06T10:00:26.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:26.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:26.604079+0000) 2024-03-06T10:00:27.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:27.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:27.604319+0000) 2024-03-06T10:00:28.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:28.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:28.604544+0000) 2024-03-06T10:00:29.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:29.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:29.604776+0000) 2024-03-06T10:00:30.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:30.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:30.605028+0000) 2024-03-06T10:00:31.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:31.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:31.605286+0000) 2024-03-06T10:00:32.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:32.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:32.605510+0000) 2024-03-06T10:00:33.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:33.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:33.605799+0000) 2024-03-06T10:00:34.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:34.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:34.606037+0000) 2024-03-06T10:00:35.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:35.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:35.606272+0000) 2024-03-06T10:00:36.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:36.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:36.606474+0000) 2024-03-06T10:00:37.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:37.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:37.606659+0000) 2024-03-06T10:00:38.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:38.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:38.606913+0000) 2024-03-06T10:00:39.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:39.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:39.607113+0000) 2024-03-06T10:00:40.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:40.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:40.607296+0000) 2024-03-06T10:00:41.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:41.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:41.607510+0000) 2024-03-06T10:00:42.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:42.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:42.607789+0000) 2024-03-06T10:00:43.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:43.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:43.608020+0000) 2024-03-06T10:00:44.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:44.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:44.608244+0000) 2024-03-06T10:00:44.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:44.674 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:45.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:45.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:45.608486+0000) 2024-03-06T10:00:46.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:46.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:46.608772+0000) 2024-03-06T10:00:47.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:47.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:47.609000+0000) 2024-03-06T10:00:48.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:48.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:48.609231+0000) 2024-03-06T10:00:49.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:49.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:49.609433+0000) 2024-03-06T10:00:50.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:50.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:50.609696+0000) 2024-03-06T10:00:51.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:51.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:51.609946+0000) 2024-03-06T10:00:52.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:52.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:52.610170+0000) 2024-03-06T10:00:53.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:53.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:53.610399+0000) 2024-03-06T10:00:54.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:54.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:54.610629+0000) 2024-03-06T10:00:55.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:55.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:55.610902+0000) 2024-03-06T10:00:56.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:56.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:56.611129+0000) 2024-03-06T10:00:57.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:57.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:57.611379+0000) 2024-03-06T10:00:58.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:58.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:58.611530+0000) 2024-03-06T10:00:59.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:00:59.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:59.611748+0000) 2024-03-06T10:01:00.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:00.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:00.611971+0000) 2024-03-06T10:01:01.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:01.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:01.612201+0000) 2024-03-06T10:01:02.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:02.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:02.612426+0000) 2024-03-06T10:01:03.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:03.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:03.612693+0000) 2024-03-06T10:01:04.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:04.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:04.612922+0000) 2024-03-06T10:01:05.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:05.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:05.613162+0000) 2024-03-06T10:01:06.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:06.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:06.613357+0000) 2024-03-06T10:01:07.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:07.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:07.613550+0000) 2024-03-06T10:01:08.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:08.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:08.613770+0000) 2024-03-06T10:01:09.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:09.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:09.613993+0000) 2024-03-06T10:01:10.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:10.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:10.614185+0000) 2024-03-06T10:01:11.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:11.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:11.614431+0000) 2024-03-06T10:01:12.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:12.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:12.614658+0000) 2024-03-06T10:01:13.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:13.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:13.614931+0000) 2024-03-06T10:01:14.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:14.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:14.615156+0000) 2024-03-06T10:01:14.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:14.805 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:15.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:15.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:15.615392+0000) 2024-03-06T10:01:16.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:16.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:16.615615+0000) 2024-03-06T10:01:17.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:17.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:17.615840+0000) 2024-03-06T10:01:18.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:18.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:18.616070+0000) 2024-03-06T10:01:19.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:19.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:19.616296+0000) 2024-03-06T10:01:20.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:20.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:20.616479+0000) 2024-03-06T10:01:21.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:21.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:21.616635+0000) 2024-03-06T10:01:22.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:22.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:22.616860+0000) 2024-03-06T10:01:23.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:23.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:23.617100+0000) 2024-03-06T10:01:24.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:24.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:24.617329+0000) 2024-03-06T10:01:25.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:25.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:25.617562+0000) 2024-03-06T10:01:26.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:26.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:26.617788+0000) 2024-03-06T10:01:27.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:27.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:27.618018+0000) 2024-03-06T10:01:28.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:28.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:28.618244+0000) 2024-03-06T10:01:29.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:29.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:29.618535+0000) 2024-03-06T10:01:30.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:30.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:30.618773+0000) 2024-03-06T10:01:31.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:31.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:31.618956+0000) 2024-03-06T10:01:32.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:32.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:32.619159+0000) 2024-03-06T10:01:33.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:33.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:33.619407+0000) 2024-03-06T10:01:34.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:34.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:34.619611+0000) 2024-03-06T10:01:35.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:35.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:35.619825+0000) 2024-03-06T10:01:36.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:36.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:36.620047+0000) 2024-03-06T10:01:37.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:37.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:37.620281+0000) 2024-03-06T10:01:38.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:38.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:38.620507+0000) 2024-03-06T10:01:39.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:39.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:39.620765+0000) 2024-03-06T10:01:40.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:40.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:40.620968+0000) 2024-03-06T10:01:41.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:41.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:41.621140+0000) 2024-03-06T10:01:42.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:42.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:42.621320+0000) 2024-03-06T10:01:43.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:43.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:43.621520+0000) 2024-03-06T10:01:44.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:44.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:44.621769+0000) 2024-03-06T10:01:45.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:45.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:45.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:45.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:45.622030+0000) 2024-03-06T10:01:46.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:46.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:46.622272+0000) 2024-03-06T10:01:47.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:47.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:47.622469+0000) 2024-03-06T10:01:48.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:48.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:48.622690+0000) 2024-03-06T10:01:49.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:49.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:49.622916+0000) 2024-03-06T10:01:50.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:50.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:50.623144+0000) 2024-03-06T10:01:51.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:51.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:51.623377+0000) 2024-03-06T10:01:52.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:52.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:52.623551+0000) 2024-03-06T10:01:53.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:53.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:53.623781+0000) 2024-03-06T10:01:54.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:54.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:54.624012+0000) 2024-03-06T10:01:55.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:55.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:55.624239+0000) 2024-03-06T10:01:56.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:56.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:56.624466+0000) 2024-03-06T10:01:57.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:57.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:57.624745+0000) 2024-03-06T10:01:58.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:58.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:58.624983+0000) 2024-03-06T10:01:59.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:01:59.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:59.625216+0000) 2024-03-06T10:02:00.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:00.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:00.625437+0000) 2024-03-06T10:02:01.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:01.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:01.625653+0000) 2024-03-06T10:02:02.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:02.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:02.625922+0000) 2024-03-06T10:02:03.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:03.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:03.626147+0000) 2024-03-06T10:02:04.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:04.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:04.626323+0000) 2024-03-06T10:02:05.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:05.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:05.626573+0000) 2024-03-06T10:02:06.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:06.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:06.626826+0000) 2024-03-06T10:02:07.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:07.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:07.627074+0000) 2024-03-06T10:02:08.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:08.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:08.627293+0000) 2024-03-06T10:02:09.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:09.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:09.627524+0000) 2024-03-06T10:02:10.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:10.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:10.627743+0000) 2024-03-06T10:02:11.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:11.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:11.627946+0000) 2024-03-06T10:02:12.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:12.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:12.628154+0000) 2024-03-06T10:02:13.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:13.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:13.628354+0000) 2024-03-06T10:02:14.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:14.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:14.628574+0000) 2024-03-06T10:02:15.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:15.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:15.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:15.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:15.628847+0000) 2024-03-06T10:02:16.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:16.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:16.629071+0000) 2024-03-06T10:02:17.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:17.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:17.629316+0000) 2024-03-06T10:02:18.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:18.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:18.629555+0000) 2024-03-06T10:02:19.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:19.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:19.629782+0000) 2024-03-06T10:02:20.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:20.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:20.629979+0000) 2024-03-06T10:02:21.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:21.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:21.630212+0000) 2024-03-06T10:02:22.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:22.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:22.630414+0000) 2024-03-06T10:02:23.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:23.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:23.630605+0000) 2024-03-06T10:02:24.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:24.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:24.630860+0000) 2024-03-06T10:02:25.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:25.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:25.631085+0000) 2024-03-06T10:02:26.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:26.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:26.631322+0000) 2024-03-06T10:02:27.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:27.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:27.631567+0000) 2024-03-06T10:02:28.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:28.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:28.631830+0000) 2024-03-06T10:02:29.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:29.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:29.632064+0000) 2024-03-06T10:02:30.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:30.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:30.632304+0000) 2024-03-06T10:02:31.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:31.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:31.632542+0000) 2024-03-06T10:02:32.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:32.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:32.632800+0000) 2024-03-06T10:02:33.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:33.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:33.633051+0000) 2024-03-06T10:02:34.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:34.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:34.633292+0000) 2024-03-06T10:02:35.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:35.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:35.633544+0000) 2024-03-06T10:02:36.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:36.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:36.633788+0000) 2024-03-06T10:02:37.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:37.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:37.634036+0000) 2024-03-06T10:02:38.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:38.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:38.634297+0000) 2024-03-06T10:02:39.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:39.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:39.634492+0000) 2024-03-06T10:02:40.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:40.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:40.634724+0000) 2024-03-06T10:02:41.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:41.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:41.634913+0000) 2024-03-06T10:02:42.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:42.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:42.635109+0000) 2024-03-06T10:02:43.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:43.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:43.635329+0000) 2024-03-06T10:02:44.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:44.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:44.635503+0000) 2024-03-06T10:02:45.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:45.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:45.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:45.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:45.635779+0000) 2024-03-06T10:02:46.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:46.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:46.636012+0000) 2024-03-06T10:02:47.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:47.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:47.636344+0000) 2024-03-06T10:02:48.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:48.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:48.636593+0000) 2024-03-06T10:02:49.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:49.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:49.636848+0000) 2024-03-06T10:02:50.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:50.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:50.637104+0000) 2024-03-06T10:02:51.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:51.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:51.637393+0000) 2024-03-06T10:02:52.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:52.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:52.637712+0000) 2024-03-06T10:02:53.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:53.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:53.637933+0000) 2024-03-06T10:02:54.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:54.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:54.638189+0000) 2024-03-06T10:02:55.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:55.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:55.638417+0000) 2024-03-06T10:02:56.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:56.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:56.638744+0000) 2024-03-06T10:02:57.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:57.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:57.638995+0000) 2024-03-06T10:02:58.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:58.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:58.639277+0000) 2024-03-06T10:02:59.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:02:59.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:59.639539+0000) 2024-03-06T10:03:00.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:00.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:00.639817+0000) 2024-03-06T10:03:01.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:01.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:01.640047+0000) 2024-03-06T10:03:02.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:02.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:02.640317+0000) 2024-03-06T10:03:03.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:03.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:03.640526+0000) 2024-03-06T10:03:04.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:04.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:04.640769+0000) 2024-03-06T10:03:05.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:05.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:05.641022+0000) 2024-03-06T10:03:06.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:06.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:06.641258+0000) 2024-03-06T10:03:07.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:07.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:07.641506+0000) 2024-03-06T10:03:08.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:08.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:08.641786+0000) 2024-03-06T10:03:09.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:09.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:09.642003+0000) 2024-03-06T10:03:10.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:10.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:10.642245+0000) 2024-03-06T10:03:11.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:11.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:11.642530+0000) 2024-03-06T10:03:12.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:12.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:12.642785+0000) 2024-03-06T10:03:13.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:13.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:13.643021+0000) 2024-03-06T10:03:14.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:14.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:14.643247+0000) 2024-03-06T10:03:15.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:15.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:15.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:15.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:15.643447+0000) 2024-03-06T10:03:16.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:16.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:16.643759+0000) 2024-03-06T10:03:17.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:17.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:17.643961+0000) 2024-03-06T10:03:18.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:18.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:18.644150+0000) 2024-03-06T10:03:19.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:19.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:19.644374+0000) 2024-03-06T10:03:20.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:20.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:20.644600+0000) 2024-03-06T10:03:21.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:21.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:21.644817+0000) 2024-03-06T10:03:22.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:22.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:22.645013+0000) 2024-03-06T10:03:23.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:23.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:23.645235+0000) 2024-03-06T10:03:24.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:24.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:24.645423+0000) 2024-03-06T10:03:25.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:25.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:25.645653+0000) 2024-03-06T10:03:26.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:26.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:26.645905+0000) 2024-03-06T10:03:27.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:27.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:27.646148+0000) 2024-03-06T10:03:28.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:28.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:28.646382+0000) 2024-03-06T10:03:29.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:29.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:29.646640+0000) 2024-03-06T10:03:30.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:30.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:30.646854+0000) 2024-03-06T10:03:31.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:31.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:31.647071+0000) 2024-03-06T10:03:32.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:32.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:32.647342+0000) 2024-03-06T10:03:33.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:33.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:33.647624+0000) 2024-03-06T10:03:34.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:34.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:34.647909+0000) 2024-03-06T10:03:35.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:35.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:35.648105+0000) 2024-03-06T10:03:36.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:36.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:36.648320+0000) 2024-03-06T10:03:37.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:37.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:37.648560+0000) 2024-03-06T10:03:38.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:38.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:38.648794+0000) 2024-03-06T10:03:39.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:39.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:39.649043+0000) 2024-03-06T10:03:40.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:40.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:40.649324+0000) 2024-03-06T10:03:41.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:41.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:41.649560+0000) 2024-03-06T10:03:42.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:42.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:42.649799+0000) 2024-03-06T10:03:43.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:43.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:43.650026+0000) 2024-03-06T10:03:44.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:44.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:44.650212+0000) 2024-03-06T10:03:45.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:45.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:45.650465+0000) 2024-03-06T10:03:45.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:45.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:46.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:46.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:46.650781+0000) 2024-03-06T10:03:47.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:47.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:47.651001+0000) 2024-03-06T10:03:48.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:48.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:48.651253+0000) 2024-03-06T10:03:49.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:49.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:49.651492+0000) 2024-03-06T10:03:50.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:50.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:50.651778+0000) 2024-03-06T10:03:51.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:51.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:51.651998+0000) 2024-03-06T10:03:52.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:52.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:52.652251+0000) 2024-03-06T10:03:53.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:53.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:53.652503+0000) 2024-03-06T10:03:54.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:54.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:54.652784+0000) 2024-03-06T10:03:55.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:55.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:55.653017+0000) 2024-03-06T10:03:56.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:56.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:56.653253+0000) 2024-03-06T10:03:57.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:57.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:57.653481+0000) 2024-03-06T10:03:58.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:58.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:58.653788+0000) 2024-03-06T10:03:59.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:03:59.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:59.653985+0000) 2024-03-06T10:04:00.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:00.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:00.654211+0000) 2024-03-06T10:04:01.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:01.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:01.654428+0000) 2024-03-06T10:04:02.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:02.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:02.654717+0000) 2024-03-06T10:04:03.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:03.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:03.654978+0000) 2024-03-06T10:04:04.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:04.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:04.655206+0000) 2024-03-06T10:04:05.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:05.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:05.655499+0000) 2024-03-06T10:04:06.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:06.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:06.655778+0000) 2024-03-06T10:04:07.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:07.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:07.655984+0000) 2024-03-06T10:04:08.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:08.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:08.656189+0000) 2024-03-06T10:04:09.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:09.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:09.656442+0000) 2024-03-06T10:04:10.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:10.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:10.656705+0000) 2024-03-06T10:04:11.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:11.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:11.656910+0000) 2024-03-06T10:04:12.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:12.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:12.657158+0000) 2024-03-06T10:04:13.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:13.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:13.657354+0000) 2024-03-06T10:04:14.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:14.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:14.657568+0000) 2024-03-06T10:04:15.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:15.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:15.657816+0000) 2024-03-06T10:04:15.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:15.973 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:16.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:16.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:16.658023+0000) 2024-03-06T10:04:17.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:17.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:17.658218+0000) 2024-03-06T10:04:18.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:18.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:18.658450+0000) 2024-03-06T10:04:19.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:19.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:19.658631+0000) 2024-03-06T10:04:20.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:20.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:20.658883+0000) 2024-03-06T10:04:21.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:21.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:21.659097+0000) 2024-03-06T10:04:22.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:22.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:22.659344+0000) 2024-03-06T10:04:23.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:23.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:23.659579+0000) 2024-03-06T10:04:24.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:24.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:24.659811+0000) 2024-03-06T10:04:25.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:25.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:25.660010+0000) 2024-03-06T10:04:26.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:26.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:26.660244+0000) 2024-03-06T10:04:27.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:27.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:27.660495+0000) 2024-03-06T10:04:28.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:28.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:28.660802+0000) 2024-03-06T10:04:29.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:29.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:29.661019+0000) 2024-03-06T10:04:30.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:30.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:30.661236+0000) 2024-03-06T10:04:31.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:31.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:31.661474+0000) 2024-03-06T10:04:32.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:32.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:32.661649+0000) 2024-03-06T10:04:33.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:33.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:33.661898+0000) 2024-03-06T10:04:34.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:34.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:34.662143+0000) 2024-03-06T10:04:35.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:35.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:35.662395+0000) 2024-03-06T10:04:36.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:36.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:36.662644+0000) 2024-03-06T10:04:37.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:37.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:37.662941+0000) 2024-03-06T10:04:38.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:38.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:38.663164+0000) 2024-03-06T10:04:39.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:39.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:39.663393+0000) 2024-03-06T10:04:40.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:40.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:40.663585+0000) 2024-03-06T10:04:41.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:41.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:41.663818+0000) 2024-03-06T10:04:42.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:42.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:42.664018+0000) 2024-03-06T10:04:43.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:43.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:43.664244+0000) 2024-03-06T10:04:44.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:44.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:44.664455+0000) 2024-03-06T10:04:45.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:45.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:45.664757+0000) 2024-03-06T10:04:46.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:46.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:46.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:46.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:46.664947+0000) 2024-03-06T10:04:47.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:47.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:47.665161+0000) 2024-03-06T10:04:48.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:48.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:48.665377+0000) 2024-03-06T10:04:49.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:49.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:49.665641+0000) 2024-03-06T10:04:50.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:50.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:50.665917+0000) 2024-03-06T10:04:51.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:51.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:51.666104+0000) 2024-03-06T10:04:52.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:52.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:52.666358+0000) 2024-03-06T10:04:53.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:53.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:53.666551+0000) 2024-03-06T10:04:54.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:54.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:54.666796+0000) 2024-03-06T10:04:55.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:55.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:55.667027+0000) 2024-03-06T10:04:56.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:56.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:56.667233+0000) 2024-03-06T10:04:57.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:57.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:57.667432+0000) 2024-03-06T10:04:58.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:58.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:58.667654+0000) 2024-03-06T10:04:59.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:04:59.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:59.667866+0000) 2024-03-06T10:05:00.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:00.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:00.668050+0000) 2024-03-06T10:05:01.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:01.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:01.668275+0000) 2024-03-06T10:05:02.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:02.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:02.668473+0000) 2024-03-06T10:05:03.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:03.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:03.668733+0000) 2024-03-06T10:05:04.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:04.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:04.668957+0000) 2024-03-06T10:05:05.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:05.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:05.669253+0000) 2024-03-06T10:05:06.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:06.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:06.669482+0000) 2024-03-06T10:05:07.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:07.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:07.669816+0000) 2024-03-06T10:05:08.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:08.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:08.670021+0000) 2024-03-06T10:05:09.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:09.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:09.670268+0000) 2024-03-06T10:05:10.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:10.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:10.670534+0000) 2024-03-06T10:05:11.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:11.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:11.670771+0000) 2024-03-06T10:05:12.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:12.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:12.670975+0000) 2024-03-06T10:05:13.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:13.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:13.671198+0000) 2024-03-06T10:05:14.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:14.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:14.671424+0000) 2024-03-06T10:05:15.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:15.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:15.671760+0000) 2024-03-06T10:05:16.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:16.294 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:16.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:16.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:16.671999+0000) 2024-03-06T10:05:17.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:17.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:17.672278+0000) 2024-03-06T10:05:18.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:18.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:18.672528+0000) 2024-03-06T10:05:19.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:19.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:19.672762+0000) 2024-03-06T10:05:20.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:20.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:20.672967+0000) 2024-03-06T10:05:21.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:21.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:21.673161+0000) 2024-03-06T10:05:22.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:22.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:22.673406+0000) 2024-03-06T10:05:23.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:23.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:23.673589+0000) 2024-03-06T10:05:24.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:24.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:24.673847+0000) 2024-03-06T10:05:25.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:25.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:25.674100+0000) 2024-03-06T10:05:26.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:26.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:26.674340+0000) 2024-03-06T10:05:27.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:27.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:27.674618+0000) 2024-03-06T10:05:28.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:28.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:28.674863+0000) 2024-03-06T10:05:29.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:29.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:29.675133+0000) 2024-03-06T10:05:30.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:30.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:30.675368+0000) 2024-03-06T10:05:31.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:31.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:31.675587+0000) 2024-03-06T10:05:32.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:32.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:32.675849+0000) 2024-03-06T10:05:33.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:33.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:33.676047+0000) 2024-03-06T10:05:34.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:34.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:34.676270+0000) 2024-03-06T10:05:35.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:35.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:35.676494+0000) 2024-03-06T10:05:36.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:36.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:36.676722+0000) 2024-03-06T10:05:37.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:37.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:37.676972+0000) 2024-03-06T10:05:38.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:38.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:38.677203+0000) 2024-03-06T10:05:39.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:39.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:39.677424+0000) 2024-03-06T10:05:40.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:40.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:40.677645+0000) 2024-03-06T10:05:41.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:41.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:41.677909+0000) 2024-03-06T10:05:42.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:42.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:42.678151+0000) 2024-03-06T10:05:43.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:43.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:43.678383+0000) 2024-03-06T10:05:44.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:44.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:44.678612+0000) 2024-03-06T10:05:45.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:45.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:45.678888+0000) 2024-03-06T10:05:46.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:46.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:46.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:46.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:46.679083+0000) 2024-03-06T10:05:47.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:47.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:47.679325+0000) 2024-03-06T10:05:48.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:48.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:48.679572+0000) 2024-03-06T10:05:49.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:49.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:49.679832+0000) 2024-03-06T10:05:50.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:50.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:50.680059+0000) 2024-03-06T10:05:51.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:51.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:51.680254+0000) 2024-03-06T10:05:52.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:52.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:52.680460+0000) 2024-03-06T10:05:53.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:53.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:53.680715+0000) 2024-03-06T10:05:54.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:54.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:54.680942+0000) 2024-03-06T10:05:55.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:55.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:55.681232+0000) 2024-03-06T10:05:56.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:56.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:56.681482+0000) 2024-03-06T10:05:57.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:57.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:57.681770+0000) 2024-03-06T10:05:58.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:58.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:58.681930+0000) 2024-03-06T10:05:59.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:05:59.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:59.682164+0000) 2024-03-06T10:06:00.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:00.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:00.682328+0000) 2024-03-06T10:06:01.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:01.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:01.682530+0000) 2024-03-06T10:06:02.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:02.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:02.682785+0000) 2024-03-06T10:06:03.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:03.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:03.682986+0000) 2024-03-06T10:06:04.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:04.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:04.683239+0000) 2024-03-06T10:06:05.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:05.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:05.683484+0000) 2024-03-06T10:06:06.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:06.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:06.683754+0000) 2024-03-06T10:06:07.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:07.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:07.683975+0000) 2024-03-06T10:06:08.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:08.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:08.684168+0000) 2024-03-06T10:06:09.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:09.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:09.684349+0000) 2024-03-06T10:06:10.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:10.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:10.684572+0000) 2024-03-06T10:06:11.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:11.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:11.684830+0000) 2024-03-06T10:06:12.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:12.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:12.685036+0000) 2024-03-06T10:06:13.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:13.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:13.685223+0000) 2024-03-06T10:06:14.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:14.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:14.685446+0000) 2024-03-06T10:06:15.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:15.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:15.685692+0000) 2024-03-06T10:06:16.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:16.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:16.685890+0000) 2024-03-06T10:06:16.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:16.691 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:17.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:17.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:17.686098+0000) 2024-03-06T10:06:18.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:18.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:18.686287+0000) 2024-03-06T10:06:19.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:19.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:19.686419+0000) 2024-03-06T10:06:20.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:20.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:20.686605+0000) 2024-03-06T10:06:21.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:21.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:21.686817+0000) 2024-03-06T10:06:22.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:22.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:22.687024+0000) 2024-03-06T10:06:23.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:23.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:23.687222+0000) 2024-03-06T10:06:24.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:24.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:24.687449+0000) 2024-03-06T10:06:25.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:25.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:25.687704+0000) 2024-03-06T10:06:26.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:26.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:26.687897+0000) 2024-03-06T10:06:27.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:27.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:27.688104+0000) 2024-03-06T10:06:28.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:28.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:28.688308+0000) 2024-03-06T10:06:29.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:29.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:29.688541+0000) 2024-03-06T10:06:30.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:30.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:30.688798+0000) 2024-03-06T10:06:31.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:31.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:31.688987+0000) 2024-03-06T10:06:32.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:32.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:32.689198+0000) 2024-03-06T10:06:33.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:33.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:33.689481+0000) 2024-03-06T10:06:34.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:34.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:34.689732+0000) 2024-03-06T10:06:35.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:35.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:35.689951+0000) 2024-03-06T10:06:36.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:36.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:36.690195+0000) 2024-03-06T10:06:37.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:37.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:37.690437+0000) 2024-03-06T10:06:38.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:38.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:38.690694+0000) 2024-03-06T10:06:39.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:39.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:39.690892+0000) 2024-03-06T10:06:40.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:40.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:40.691084+0000) 2024-03-06T10:06:41.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:41.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:41.691282+0000) 2024-03-06T10:06:42.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:42.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:42.691514+0000) 2024-03-06T10:06:43.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:43.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:43.691783+0000) 2024-03-06T10:06:44.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:44.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:44.691955+0000) 2024-03-06T10:06:45.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:45.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:45.692208+0000) 2024-03-06T10:06:46.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:46.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:46.692449+0000) 2024-03-06T10:06:46.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:46.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:47.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:47.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:47.692738+0000) 2024-03-06T10:06:48.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:48.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:48.692958+0000) 2024-03-06T10:06:49.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:49.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:49.693155+0000) 2024-03-06T10:06:50.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:50.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:50.693344+0000) 2024-03-06T10:06:51.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:51.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:51.693549+0000) 2024-03-06T10:06:52.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:52.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:52.693798+0000) 2024-03-06T10:06:53.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:53.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:53.693993+0000) 2024-03-06T10:06:54.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:54.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:54.694184+0000) 2024-03-06T10:06:55.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:55.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:55.694414+0000) 2024-03-06T10:06:56.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:56.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:56.694642+0000) 2024-03-06T10:06:57.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:57.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:57.694864+0000) 2024-03-06T10:06:58.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:58.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:58.695056+0000) 2024-03-06T10:06:59.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:06:59.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:59.695249+0000) 2024-03-06T10:07:00.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:00.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:00.695471+0000) 2024-03-06T10:07:01.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:01.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:01.695773+0000) 2024-03-06T10:07:02.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:02.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:02.696007+0000) 2024-03-06T10:07:03.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:03.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:03.696206+0000) 2024-03-06T10:07:04.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:04.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:04.696475+0000) 2024-03-06T10:07:05.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:05.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:05.696815+0000) 2024-03-06T10:07:06.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:06.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:06.697026+0000) 2024-03-06T10:07:07.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:07.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:07.697293+0000) 2024-03-06T10:07:08.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:08.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:08.697581+0000) 2024-03-06T10:07:09.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:09.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:09.697853+0000) 2024-03-06T10:07:10.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:10.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:10.698136+0000) 2024-03-06T10:07:11.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:11.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:11.698416+0000) 2024-03-06T10:07:12.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:12.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:12.698745+0000) 2024-03-06T10:07:13.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:13.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:13.699022+0000) 2024-03-06T10:07:14.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:14.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:14.699302+0000) 2024-03-06T10:07:15.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:15.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:15.699593+0000) 2024-03-06T10:07:16.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:16.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:16.699865+0000) 2024-03-06T10:07:17.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:17.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:17.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:17.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:17.700060+0000) 2024-03-06T10:07:18.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:18.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:18.700263+0000) 2024-03-06T10:07:19.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:19.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:19.700448+0000) 2024-03-06T10:07:20.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:20.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:20.700748+0000) 2024-03-06T10:07:21.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:21.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:21.700987+0000) 2024-03-06T10:07:22.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:22.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:22.701249+0000) 2024-03-06T10:07:23.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:23.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:23.701511+0000) 2024-03-06T10:07:24.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:24.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:24.701787+0000) 2024-03-06T10:07:25.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:25.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:25.702018+0000) 2024-03-06T10:07:26.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:26.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:26.702244+0000) 2024-03-06T10:07:27.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:27.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:27.702473+0000) 2024-03-06T10:07:28.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:28.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:28.702738+0000) 2024-03-06T10:07:29.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:29.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:29.702922+0000) 2024-03-06T10:07:30.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:30.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:30.703153+0000) 2024-03-06T10:07:31.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:31.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:31.703378+0000) 2024-03-06T10:07:32.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:32.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:32.703604+0000) 2024-03-06T10:07:33.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:33.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:33.703833+0000) 2024-03-06T10:07:34.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:34.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:34.704056+0000) 2024-03-06T10:07:35.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:35.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:35.704292+0000) 2024-03-06T10:07:36.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:36.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:36.704479+0000) 2024-03-06T10:07:37.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:37.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:37.704699+0000) 2024-03-06T10:07:38.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:38.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:38.704926+0000) 2024-03-06T10:07:39.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:39.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:39.705149+0000) 2024-03-06T10:07:40.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:40.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:40.705371+0000) 2024-03-06T10:07:41.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:41.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:41.705548+0000) 2024-03-06T10:07:42.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:42.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:42.705792+0000) 2024-03-06T10:07:43.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:43.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:43.705984+0000) 2024-03-06T10:07:44.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:44.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:44.706176+0000) 2024-03-06T10:07:45.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:45.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:45.706425+0000) 2024-03-06T10:07:46.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:46.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:46.706657+0000) 2024-03-06T10:07:47.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:47.187 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:47.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:47.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:47.706887+0000) 2024-03-06T10:07:48.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:48.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:48.707090+0000) 2024-03-06T10:07:49.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:49.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:49.707268+0000) 2024-03-06T10:07:50.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:50.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:50.707490+0000) 2024-03-06T10:07:51.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:51.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:51.707757+0000) 2024-03-06T10:07:52.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:52.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:52.707945+0000) 2024-03-06T10:07:53.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:53.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:53.708152+0000) 2024-03-06T10:07:54.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:54.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:54.708295+0000) 2024-03-06T10:07:55.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:55.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:55.708479+0000) 2024-03-06T10:07:56.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:56.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:56.708740+0000) 2024-03-06T10:07:57.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:57.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:57.708938+0000) 2024-03-06T10:07:58.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:58.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:58.709176+0000) 2024-03-06T10:07:59.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:07:59.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:59.709398+0000) 2024-03-06T10:08:00.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:00.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:00.709594+0000) 2024-03-06T10:08:01.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:01.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:01.709822+0000) 2024-03-06T10:08:02.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:02.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:02.710019+0000) 2024-03-06T10:08:03.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:03.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:03.710215+0000) 2024-03-06T10:08:04.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:04.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:04.710441+0000) 2024-03-06T10:08:05.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:05.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:05.710692+0000) 2024-03-06T10:08:06.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:06.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:06.710885+0000) 2024-03-06T10:08:07.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:07.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:07.711072+0000) 2024-03-06T10:08:08.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:08.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:08.711270+0000) 2024-03-06T10:08:09.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:09.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:09.711468+0000) 2024-03-06T10:08:10.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:10.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:10.711652+0000) 2024-03-06T10:08:11.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:11.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:11.711897+0000) 2024-03-06T10:08:12.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:12.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:12.712110+0000) 2024-03-06T10:08:13.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:13.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:13.712313+0000) 2024-03-06T10:08:14.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:14.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:14.712555+0000) 2024-03-06T10:08:15.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:15.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:15.712810+0000) 2024-03-06T10:08:16.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:16.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:16.713041+0000) 2024-03-06T10:08:17.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:17.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:17.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:17.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:17.713231+0000) 2024-03-06T10:08:18.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:18.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:18.713472+0000) 2024-03-06T10:08:19.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:19.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:19.713751+0000) 2024-03-06T10:08:20.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:20.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:20.713986+0000) 2024-03-06T10:08:21.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:21.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:21.714173+0000) 2024-03-06T10:08:22.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:22.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:22.714409+0000) 2024-03-06T10:08:23.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:23.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:23.714599+0000) 2024-03-06T10:08:24.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:24.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:24.714814+0000) 2024-03-06T10:08:25.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:25.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:25.715018+0000) 2024-03-06T10:08:26.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:26.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:26.715233+0000) 2024-03-06T10:08:27.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:27.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:27.715437+0000) 2024-03-06T10:08:28.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:28.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:28.715716+0000) 2024-03-06T10:08:29.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:29.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:29.715921+0000) 2024-03-06T10:08:30.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:30.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:30.716124+0000) 2024-03-06T10:08:31.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:31.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:31.716358+0000) 2024-03-06T10:08:32.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:32.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:32.716593+0000) 2024-03-06T10:08:33.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:33.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:33.716825+0000) 2024-03-06T10:08:34.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:34.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:34.717013+0000) 2024-03-06T10:08:35.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:35.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:35.717208+0000) 2024-03-06T10:08:36.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:36.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:36.717391+0000) 2024-03-06T10:08:37.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:37.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:37.717626+0000) 2024-03-06T10:08:38.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:38.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:38.717820+0000) 2024-03-06T10:08:39.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:39.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:39.718010+0000) 2024-03-06T10:08:40.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:40.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:40.718206+0000) 2024-03-06T10:08:41.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:41.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:41.718403+0000) 2024-03-06T10:08:42.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:42.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:42.718636+0000) 2024-03-06T10:08:43.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:43.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:43.718779+0000) 2024-03-06T10:08:44.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:44.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:44.718981+0000) 2024-03-06T10:08:45.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:45.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:45.719179+0000) 2024-03-06T10:08:46.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:46.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:46.719407+0000) 2024-03-06T10:08:47.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:47.587 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:47.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:47.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:47.719599+0000) 2024-03-06T10:08:48.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:48.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:48.719838+0000) 2024-03-06T10:08:49.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:49.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:49.720024+0000) 2024-03-06T10:08:50.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:50.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:50.720258+0000) 2024-03-06T10:08:51.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:51.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:51.720487+0000) 2024-03-06T10:08:52.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:52.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:52.720740+0000) 2024-03-06T10:08:53.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:53.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:53.720886+0000) 2024-03-06T10:08:54.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:54.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:54.721073+0000) 2024-03-06T10:08:55.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:55.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:55.721267+0000) 2024-03-06T10:08:56.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:56.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:56.721492+0000) 2024-03-06T10:08:57.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:57.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:57.721759+0000) 2024-03-06T10:08:58.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:58.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:58.721956+0000) 2024-03-06T10:08:59.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:08:59.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:59.722146+0000) 2024-03-06T10:09:00.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:00.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:00.722367+0000) 2024-03-06T10:09:01.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:01.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:01.722590+0000) 2024-03-06T10:09:02.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:02.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:02.722817+0000) 2024-03-06T10:09:03.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:03.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:03.723046+0000) 2024-03-06T10:09:04.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:04.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:04.723271+0000) 2024-03-06T10:09:05.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:05.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:05.723508+0000) 2024-03-06T10:09:06.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:06.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:06.723777+0000) 2024-03-06T10:09:07.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:07.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:07.724003+0000) 2024-03-06T10:09:08.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:08.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:08.724285+0000) 2024-03-06T10:09:09.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:09.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:09.724508+0000) 2024-03-06T10:09:10.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:10.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:10.724757+0000) 2024-03-06T10:09:11.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:11.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:11.724947+0000) 2024-03-06T10:09:12.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:12.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:12.725172+0000) 2024-03-06T10:09:13.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:13.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:13.725375+0000) 2024-03-06T10:09:14.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:14.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:14.725599+0000) 2024-03-06T10:09:15.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:15.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:15.725819+0000) 2024-03-06T10:09:16.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:16.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:16.726041+0000) 2024-03-06T10:09:17.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:17.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:17.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:17.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:17.726265+0000) 2024-03-06T10:09:18.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:18.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:18.726463+0000) 2024-03-06T10:09:19.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:19.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:19.726727+0000) 2024-03-06T10:09:20.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:20.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:20.726912+0000) 2024-03-06T10:09:21.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:21.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:21.727097+0000) 2024-03-06T10:09:22.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:22.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:22.727323+0000) 2024-03-06T10:09:23.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:23.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:23.727558+0000) 2024-03-06T10:09:24.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:24.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:24.727791+0000) 2024-03-06T10:09:25.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:25.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:25.727951+0000) 2024-03-06T10:09:26.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:26.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:26.728138+0000) 2024-03-06T10:09:27.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:27.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:27.728318+0000) 2024-03-06T10:09:28.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:28.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:28.728521+0000) 2024-03-06T10:09:29.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:29.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:29.728797+0000) 2024-03-06T10:09:30.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:30.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:30.728984+0000) 2024-03-06T10:09:31.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:31.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:31.729164+0000) 2024-03-06T10:09:32.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:32.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:32.729387+0000) 2024-03-06T10:09:33.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:33.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:33.729618+0000) 2024-03-06T10:09:34.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:34.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:34.729822+0000) 2024-03-06T10:09:35.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:35.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:35.730021+0000) 2024-03-06T10:09:36.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:36.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:36.730218+0000) 2024-03-06T10:09:37.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:37.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:37.730444+0000) 2024-03-06T10:09:38.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:38.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:38.730698+0000) 2024-03-06T10:09:39.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:39.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:39.730882+0000) 2024-03-06T10:09:40.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:40.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:40.731115+0000) 2024-03-06T10:09:41.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:41.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:41.731297+0000) 2024-03-06T10:09:42.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:42.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:42.731520+0000) 2024-03-06T10:09:43.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:43.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:43.731792+0000) 2024-03-06T10:09:44.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:44.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:44.731991+0000) 2024-03-06T10:09:45.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:45.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:45.732196+0000) 2024-03-06T10:09:46.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:46.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:46.732421+0000) 2024-03-06T10:09:47.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:47.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:47.732642+0000) 2024-03-06T10:09:47.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:47.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:48.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:48.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:48.732826+0000) 2024-03-06T10:09:49.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:49.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:49.733013+0000) 2024-03-06T10:09:50.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:50.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:50.733242+0000) 2024-03-06T10:09:51.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:51.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:51.733467+0000) 2024-03-06T10:09:52.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:52.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:52.733715+0000) 2024-03-06T10:09:53.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:53.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:53.733906+0000) 2024-03-06T10:09:54.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:54.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:54.734110+0000) 2024-03-06T10:09:55.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:55.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:55.734321+0000) 2024-03-06T10:09:56.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:56.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:56.734498+0000) 2024-03-06T10:09:57.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:57.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:57.734768+0000) 2024-03-06T10:09:58.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:58.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:58.735006+0000) 2024-03-06T10:09:59.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:09:59.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:59.735206+0000) 2024-03-06T10:10:00.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:00.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:00.735406+0000) 2024-03-06T10:10:01.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:01.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:01.735703+0000) 2024-03-06T10:10:02.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:02.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:02.735917+0000) 2024-03-06T10:10:03.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:03.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:03.736158+0000) 2024-03-06T10:10:04.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:04.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:04.736339+0000) 2024-03-06T10:10:05.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:05.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:05.736577+0000) 2024-03-06T10:10:06.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:06.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:06.736802+0000) 2024-03-06T10:10:07.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:07.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:07.737040+0000) 2024-03-06T10:10:08.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:08.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:08.737277+0000) 2024-03-06T10:10:09.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:09.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:09.737505+0000) 2024-03-06T10:10:10.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:10.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:10.737769+0000) 2024-03-06T10:10:11.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:11.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:11.737993+0000) 2024-03-06T10:10:12.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:12.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:12.738181+0000) 2024-03-06T10:10:13.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:13.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:13.738370+0000) 2024-03-06T10:10:14.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:14.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:14.738557+0000) 2024-03-06T10:10:15.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:15.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:15.738815+0000) 2024-03-06T10:10:16.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:16.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:16.739014+0000) 2024-03-06T10:10:17.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:17.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:17.739220+0000) 2024-03-06T10:10:18.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:18.075 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:18.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:18.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:18.739463+0000) 2024-03-06T10:10:19.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:19.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:19.739712+0000) 2024-03-06T10:10:20.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:20.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:20.739905+0000) 2024-03-06T10:10:21.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:21.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:21.740096+0000) 2024-03-06T10:10:22.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:22.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:22.740326+0000) 2024-03-06T10:10:23.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:23.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:23.740568+0000) 2024-03-06T10:10:24.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:24.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:24.740786+0000) 2024-03-06T10:10:25.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:25.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:25.741026+0000) 2024-03-06T10:10:26.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:26.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:26.741253+0000) 2024-03-06T10:10:27.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:27.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:27.741474+0000) 2024-03-06T10:10:28.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:28.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:28.741748+0000) 2024-03-06T10:10:29.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:29.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:29.741970+0000) 2024-03-06T10:10:30.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:30.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:30.742183+0000) 2024-03-06T10:10:31.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:31.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:31.742324+0000) 2024-03-06T10:10:32.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:32.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:32.742551+0000) 2024-03-06T10:10:33.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:33.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:33.742808+0000) 2024-03-06T10:10:34.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:34.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:34.742996+0000) 2024-03-06T10:10:35.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:35.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:35.743196+0000) 2024-03-06T10:10:36.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:36.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:36.743438+0000) 2024-03-06T10:10:37.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:37.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:37.743652+0000) 2024-03-06T10:10:38.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:38.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:38.743924+0000) 2024-03-06T10:10:39.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:39.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:39.744110+0000) 2024-03-06T10:10:40.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:40.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:40.744342+0000) 2024-03-06T10:10:41.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:41.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:41.744578+0000) 2024-03-06T10:10:42.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:42.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:42.744814+0000) 2024-03-06T10:10:43.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:43.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:43.745020+0000) 2024-03-06T10:10:44.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:44.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:44.745207+0000) 2024-03-06T10:10:45.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:45.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:45.745445+0000) 2024-03-06T10:10:46.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:46.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:46.745699+0000) 2024-03-06T10:10:47.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:47.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:47.745889+0000) 2024-03-06T10:10:48.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:48.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:48.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:48.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:48.746122+0000) 2024-03-06T10:10:49.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:49.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:49.746320+0000) 2024-03-06T10:10:50.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:50.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:50.746542+0000) 2024-03-06T10:10:51.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:51.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:51.746789+0000) 2024-03-06T10:10:52.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:52.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:52.746985+0000) 2024-03-06T10:10:53.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:53.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:53.747236+0000) 2024-03-06T10:10:54.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:54.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:54.747477+0000) 2024-03-06T10:10:55.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:55.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:55.747737+0000) 2024-03-06T10:10:56.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:56.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:56.747925+0000) 2024-03-06T10:10:57.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:57.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:57.748123+0000) 2024-03-06T10:10:58.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:58.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:58.748312+0000) 2024-03-06T10:10:59.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:10:59.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:59.748487+0000) 2024-03-06T10:11:00.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:00.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:00.748748+0000) 2024-03-06T10:11:01.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:01.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:01.748934+0000) 2024-03-06T10:11:02.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:02.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:02.749169+0000) 2024-03-06T10:11:03.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:03.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:03.749406+0000) 2024-03-06T10:11:04.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:04.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:04.749627+0000) 2024-03-06T10:11:05.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:05.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:05.749878+0000) 2024-03-06T10:11:06.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:06.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:06.750109+0000) 2024-03-06T10:11:07.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:07.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:07.750339+0000) 2024-03-06T10:11:08.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:08.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:08.750530+0000) 2024-03-06T10:11:09.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:09.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:09.750794+0000) 2024-03-06T10:11:10.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:10.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:10.751022+0000) 2024-03-06T10:11:11.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:11.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:11.751204+0000) 2024-03-06T10:11:12.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:12.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:12.751442+0000) 2024-03-06T10:11:13.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:13.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:13.751710+0000) 2024-03-06T10:11:14.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:14.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:14.751939+0000) 2024-03-06T10:11:15.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:15.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:15.752177+0000) 2024-03-06T10:11:16.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:16.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:16.752398+0000) 2024-03-06T10:11:17.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:17.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:17.752622+0000) 2024-03-06T10:11:18.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:18.449 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:18.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:18.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:18.752846+0000) 2024-03-06T10:11:19.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:19.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:19.753070+0000) 2024-03-06T10:11:20.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:20.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:20.753244+0000) 2024-03-06T10:11:21.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:21.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:21.753477+0000) 2024-03-06T10:11:22.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:22.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:22.753723+0000) 2024-03-06T10:11:23.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:23.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:23.753972+0000) 2024-03-06T10:11:24.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:24.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:24.754166+0000) 2024-03-06T10:11:25.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:25.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:25.754400+0000) 2024-03-06T10:11:26.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:26.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:26.754616+0000) 2024-03-06T10:11:27.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:27.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:27.754832+0000) 2024-03-06T10:11:28.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:28.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:28.755038+0000) 2024-03-06T10:11:29.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:29.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:29.755243+0000) 2024-03-06T10:11:30.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:30.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:30.755468+0000) 2024-03-06T10:11:31.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:31.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:31.755703+0000) 2024-03-06T10:11:32.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:32.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:32.755946+0000) 2024-03-06T10:11:33.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:33.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:33.756159+0000) 2024-03-06T10:11:34.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:34.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:34.756396+0000) 2024-03-06T10:11:35.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:35.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:35.756632+0000) 2024-03-06T10:11:36.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:36.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:36.756843+0000) 2024-03-06T10:11:37.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:37.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:37.757077+0000) 2024-03-06T10:11:38.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:38.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:38.757327+0000) 2024-03-06T10:11:39.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:39.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:39.757570+0000) 2024-03-06T10:11:40.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:40.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:40.757791+0000) 2024-03-06T10:11:41.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:41.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:41.758015+0000) 2024-03-06T10:11:42.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:42.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:42.758197+0000) 2024-03-06T10:11:43.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:43.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:43.758439+0000) 2024-03-06T10:11:44.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:44.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:44.758688+0000) 2024-03-06T10:11:45.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:45.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:45.758924+0000) 2024-03-06T10:11:46.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:46.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:46.759115+0000) 2024-03-06T10:11:47.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:47.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:47.759301+0000) 2024-03-06T10:11:48.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:48.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:48.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:48.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:48.759526+0000) 2024-03-06T10:11:49.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:49.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:49.759790+0000) 2024-03-06T10:11:50.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:50.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:50.759982+0000) 2024-03-06T10:11:51.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:51.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:51.760209+0000) 2024-03-06T10:11:52.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:52.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:52.760414+0000) 2024-03-06T10:11:53.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:53.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:53.760736+0000) 2024-03-06T10:11:54.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:54.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:54.760929+0000) 2024-03-06T10:11:55.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:55.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:55.761123+0000) 2024-03-06T10:11:56.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:56.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:56.761347+0000) 2024-03-06T10:11:57.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:57.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:57.761568+0000) 2024-03-06T10:11:58.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:58.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:58.761803+0000) 2024-03-06T10:11:59.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:11:59.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:59.761991+0000) 2024-03-06T10:12:00.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:00.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:00.762187+0000) 2024-03-06T10:12:01.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:01.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:01.762416+0000) 2024-03-06T10:12:02.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:02.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:02.762639+0000) 2024-03-06T10:12:03.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:03.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:03.762877+0000) 2024-03-06T10:12:04.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:04.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:04.763104+0000) 2024-03-06T10:12:05.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:05.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:05.763306+0000) 2024-03-06T10:12:06.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:06.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:06.763546+0000) 2024-03-06T10:12:07.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:07.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:07.763768+0000) 2024-03-06T10:12:08.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:08.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:08.764016+0000) 2024-03-06T10:12:09.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:09.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:09.764199+0000) 2024-03-06T10:12:10.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:10.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:10.764478+0000) 2024-03-06T10:12:11.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:11.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:11.764731+0000) 2024-03-06T10:12:12.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:12.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:12.764937+0000) 2024-03-06T10:12:13.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:13.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:13.765180+0000) 2024-03-06T10:12:14.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:14.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:14.765384+0000) 2024-03-06T10:12:15.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:15.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:15.765626+0000) 2024-03-06T10:12:16.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:16.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:16.765818+0000) 2024-03-06T10:12:17.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:17.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:17.766026+0000) 2024-03-06T10:12:18.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:18.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:18.766226+0000) 2024-03-06T10:12:18.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:18.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:19.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:19.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:19.766456+0000) 2024-03-06T10:12:20.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:20.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:20.766719+0000) 2024-03-06T10:12:21.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:21.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:21.766949+0000) 2024-03-06T10:12:22.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:22.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:22.767179+0000) 2024-03-06T10:12:23.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:23.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:23.767379+0000) 2024-03-06T10:12:24.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:24.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:24.767576+0000) 2024-03-06T10:12:25.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:25.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:25.767821+0000) 2024-03-06T10:12:26.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:26.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:26.768046+0000) 2024-03-06T10:12:27.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:27.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:27.768276+0000) 2024-03-06T10:12:28.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:28.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:28.768538+0000) 2024-03-06T10:12:29.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:29.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:29.768794+0000) 2024-03-06T10:12:30.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:30.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:30.768955+0000) 2024-03-06T10:12:31.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:31.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:31.769150+0000) 2024-03-06T10:12:32.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:32.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:32.769378+0000) 2024-03-06T10:12:33.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:33.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:33.769576+0000) 2024-03-06T10:12:34.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:34.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:34.769810+0000) 2024-03-06T10:12:35.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:35.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:35.770046+0000) 2024-03-06T10:12:36.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:36.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:36.770374+0000) 2024-03-06T10:12:37.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:37.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:37.770612+0000) 2024-03-06T10:12:38.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:38.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:38.770834+0000) 2024-03-06T10:12:39.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:39.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:39.771059+0000) 2024-03-06T10:12:40.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:40.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:40.771254+0000) 2024-03-06T10:12:41.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:41.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:41.771480+0000) 2024-03-06T10:12:42.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:42.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:42.771739+0000) 2024-03-06T10:12:43.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:43.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:43.771931+0000) 2024-03-06T10:12:44.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:44.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:44.772122+0000) 2024-03-06T10:12:45.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:45.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:45.772365+0000) 2024-03-06T10:12:46.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:46.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:46.772592+0000) 2024-03-06T10:12:47.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:47.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:47.772822+0000) 2024-03-06T10:12:48.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:48.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:48.773052+0000) 2024-03-06T10:12:49.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:49.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:49.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:49.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:49.773240+0000) 2024-03-06T10:12:50.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:50.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:50.773447+0000) 2024-03-06T10:12:51.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:51.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:51.773646+0000) 2024-03-06T10:12:52.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:52.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:52.773872+0000) 2024-03-06T10:12:53.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:53.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:53.774099+0000) 2024-03-06T10:12:54.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:54.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:54.774335+0000) 2024-03-06T10:12:55.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:55.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:55.774530+0000) 2024-03-06T10:12:56.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:56.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:56.774759+0000) 2024-03-06T10:12:57.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:57.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:57.774947+0000) 2024-03-06T10:12:58.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:58.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:58.775171+0000) 2024-03-06T10:12:59.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:12:59.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:59.775403+0000) 2024-03-06T10:13:00.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:00.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:00.775621+0000) 2024-03-06T10:13:01.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:01.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:01.775811+0000) 2024-03-06T10:13:02.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:02.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:02.776005+0000) 2024-03-06T10:13:03.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:03.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:03.776242+0000) 2024-03-06T10:13:04.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:04.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:04.776472+0000) 2024-03-06T10:13:05.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:05.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:05.776716+0000) 2024-03-06T10:13:06.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:06.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:06.776899+0000) 2024-03-06T10:13:07.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:07.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:07.777092+0000) 2024-03-06T10:13:08.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:08.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:08.777280+0000) 2024-03-06T10:13:09.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:09.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:09.777518+0000) 2024-03-06T10:13:10.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:10.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:10.777775+0000) 2024-03-06T10:13:11.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:11.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:11.777969+0000) 2024-03-06T10:13:12.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:12.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:12.778197+0000) 2024-03-06T10:13:13.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:13.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:13.778431+0000) 2024-03-06T10:13:14.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:14.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:14.778658+0000) 2024-03-06T10:13:15.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:15.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:15.778881+0000) 2024-03-06T10:13:16.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:16.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:16.779108+0000) 2024-03-06T10:13:17.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:17.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:17.779338+0000) 2024-03-06T10:13:18.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:18.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:18.779537+0000) 2024-03-06T10:13:19.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:19.275 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:19.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:19.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:19.779780+0000) 2024-03-06T10:13:20.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:20.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:20.779962+0000) 2024-03-06T10:13:21.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:21.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:21.780193+0000) 2024-03-06T10:13:22.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:22.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:22.780375+0000) 2024-03-06T10:13:23.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:23.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:23.780606+0000) 2024-03-06T10:13:24.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:24.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:24.780798+0000) 2024-03-06T10:13:25.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:25.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:25.780996+0000) 2024-03-06T10:13:26.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:26.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:26.781214+0000) 2024-03-06T10:13:27.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:27.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:27.781436+0000) 2024-03-06T10:13:28.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:28.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:28.781695+0000) 2024-03-06T10:13:29.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:29.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:29.781980+0000) 2024-03-06T10:13:30.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:30.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:30.782201+0000) 2024-03-06T10:13:31.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:31.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:31.782393+0000) 2024-03-06T10:13:32.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:32.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:32.782605+0000) 2024-03-06T10:13:33.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:33.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:33.782793+0000) 2024-03-06T10:13:34.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:34.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:34.783035+0000) 2024-03-06T10:13:35.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:35.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:35.783236+0000) 2024-03-06T10:13:36.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:36.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:36.783470+0000) 2024-03-06T10:13:37.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:37.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:37.783741+0000) 2024-03-06T10:13:38.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:38.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:38.783969+0000) 2024-03-06T10:13:39.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:39.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:39.784151+0000) 2024-03-06T10:13:40.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:40.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:40.784328+0000) 2024-03-06T10:13:41.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:41.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:41.784554+0000) 2024-03-06T10:13:42.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:42.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:42.784775+0000) 2024-03-06T10:13:43.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:43.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:43.784999+0000) 2024-03-06T10:13:44.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:44.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:44.785226+0000) 2024-03-06T10:13:45.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:45.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:45.785425+0000) 2024-03-06T10:13:46.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:46.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:46.785618+0000) 2024-03-06T10:13:47.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:47.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:47.785832+0000) 2024-03-06T10:13:48.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:48.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:48.786017+0000) 2024-03-06T10:13:49.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:49.499 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:49.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:49.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:49.786215+0000) 2024-03-06T10:13:50.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:50.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:50.786435+0000) 2024-03-06T10:13:51.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:51.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:51.786698+0000) 2024-03-06T10:13:52.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:52.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:52.786893+0000) 2024-03-06T10:13:53.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:53.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:53.787061+0000) 2024-03-06T10:13:54.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:54.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:54.787248+0000) 2024-03-06T10:13:55.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:55.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:55.787477+0000) 2024-03-06T10:13:56.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:56.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:56.787808+0000) 2024-03-06T10:13:57.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:57.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:57.788090+0000) 2024-03-06T10:13:58.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:58.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:58.788356+0000) 2024-03-06T10:13:59.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:13:59.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:59.788560+0000) 2024-03-06T10:14:00.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:00.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:00.788727+0000) 2024-03-06T10:14:01.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:01.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:01.788945+0000) 2024-03-06T10:14:02.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:02.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:02.789175+0000) 2024-03-06T10:14:03.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:03.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:03.789402+0000) 2024-03-06T10:14:04.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:04.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:04.789716+0000) 2024-03-06T10:14:05.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:05.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:05.790033+0000) 2024-03-06T10:14:06.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:06.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:06.790297+0000) 2024-03-06T10:14:07.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:07.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:07.790574+0000) 2024-03-06T10:14:08.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:08.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:08.790817+0000) 2024-03-06T10:14:09.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:09.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:09.791013+0000) 2024-03-06T10:14:10.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:10.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:10.791202+0000) 2024-03-06T10:14:11.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:11.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:11.791426+0000) 2024-03-06T10:14:12.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:12.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:12.791657+0000) 2024-03-06T10:14:13.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:13.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:13.791875+0000) 2024-03-06T10:14:14.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:14.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:14.792071+0000) 2024-03-06T10:14:15.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:15.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:15.792309+0000) 2024-03-06T10:14:16.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:16.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:16.792538+0000) 2024-03-06T10:14:17.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:17.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:17.792776+0000) 2024-03-06T10:14:18.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:18.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:18.792977+0000) 2024-03-06T10:14:19.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:19.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:19.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:19.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:19.793132+0000) 2024-03-06T10:14:20.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:20.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:20.793305+0000) 2024-03-06T10:14:21.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:21.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:21.793465+0000) 2024-03-06T10:14:22.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:22.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:22.793772+0000) 2024-03-06T10:14:23.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:23.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:23.793996+0000) 2024-03-06T10:14:24.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:24.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:24.794227+0000) 2024-03-06T10:14:25.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:25.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:25.794425+0000) 2024-03-06T10:14:26.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:26.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:26.794656+0000) 2024-03-06T10:14:27.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:27.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:27.794915+0000) 2024-03-06T10:14:28.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:28.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:28.795142+0000) 2024-03-06T10:14:29.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:29.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:29.795336+0000) 2024-03-06T10:14:30.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:30.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:30.795523+0000) 2024-03-06T10:14:31.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:31.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:31.795801+0000) 2024-03-06T10:14:32.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:32.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:32.796175+0000) 2024-03-06T10:14:33.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:33.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:33.796382+0000) 2024-03-06T10:14:34.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:34.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:34.796613+0000) 2024-03-06T10:14:35.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:35.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:35.796851+0000) 2024-03-06T10:14:36.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:36.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:36.797040+0000) 2024-03-06T10:14:37.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:37.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:37.797296+0000) 2024-03-06T10:14:38.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:38.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:38.797505+0000) 2024-03-06T10:14:39.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:39.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:39.797746+0000) 2024-03-06T10:14:40.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:40.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:40.797961+0000) 2024-03-06T10:14:41.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:41.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:41.798205+0000) 2024-03-06T10:14:42.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:42.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:42.798434+0000) 2024-03-06T10:14:43.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:43.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:43.798659+0000) 2024-03-06T10:14:44.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:44.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:44.798925+0000) 2024-03-06T10:14:45.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:45.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:45.799164+0000) 2024-03-06T10:14:46.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:46.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:46.799387+0000) 2024-03-06T10:14:47.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:47.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:47.799606+0000) 2024-03-06T10:14:48.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:48.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:48.799833+0000) 2024-03-06T10:14:49.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:49.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:49.800070+0000) 2024-03-06T10:14:49.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:49.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:50.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:50.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:50.800298+0000) 2024-03-06T10:14:51.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:51.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:51.800521+0000) 2024-03-06T10:14:52.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:52.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:52.800793+0000) 2024-03-06T10:14:53.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:53.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:53.801034+0000) 2024-03-06T10:14:54.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:54.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:54.801265+0000) 2024-03-06T10:14:55.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:55.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:55.801533+0000) 2024-03-06T10:14:56.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:56.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:56.801782+0000) 2024-03-06T10:14:57.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:57.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:57.802022+0000) 2024-03-06T10:14:58.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:58.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:58.802255+0000) 2024-03-06T10:14:59.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:14:59.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:59.802504+0000) 2024-03-06T10:15:00.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:00.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:00.802775+0000) 2024-03-06T10:15:01.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:01.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:01.803018+0000) 2024-03-06T10:15:02.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:02.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:02.803260+0000) 2024-03-06T10:15:03.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:03.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:03.803460+0000) 2024-03-06T10:15:04.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:04.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:04.803697+0000) 2024-03-06T10:15:05.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:05.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:05.803955+0000) 2024-03-06T10:15:06.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:06.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:06.804194+0000) 2024-03-06T10:15:07.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:07.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:07.804384+0000) 2024-03-06T10:15:08.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:08.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:08.804651+0000) 2024-03-06T10:15:09.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:09.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:09.805002+0000) 2024-03-06T10:15:10.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:10.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:10.805243+0000) 2024-03-06T10:15:11.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:11.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:11.805455+0000) 2024-03-06T10:15:12.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:12.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:12.805749+0000) 2024-03-06T10:15:13.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:13.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:13.805996+0000) 2024-03-06T10:15:14.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:14.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:14.806238+0000) 2024-03-06T10:15:15.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:15.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:15.806482+0000) 2024-03-06T10:15:16.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:16.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:16.806745+0000) 2024-03-06T10:15:17.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:17.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:17.806971+0000) 2024-03-06T10:15:18.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:18.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:18.807170+0000) 2024-03-06T10:15:19.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:19.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:19.807377+0000) 2024-03-06T10:15:20.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:20.146 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:20.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:20.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:20.807610+0000) 2024-03-06T10:15:21.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:21.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:21.807813+0000) 2024-03-06T10:15:22.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:22.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:22.808039+0000) 2024-03-06T10:15:23.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:23.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:23.808272+0000) 2024-03-06T10:15:24.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:24.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:24.808484+0000) 2024-03-06T10:15:25.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:25.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:25.808658+0000) 2024-03-06T10:15:26.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:26.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:26.808857+0000) 2024-03-06T10:15:27.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:27.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:27.809090+0000) 2024-03-06T10:15:28.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:28.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:28.809323+0000) 2024-03-06T10:15:29.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:29.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:29.809567+0000) 2024-03-06T10:15:30.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:30.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:30.809815+0000) 2024-03-06T10:15:31.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:31.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:31.810055+0000) 2024-03-06T10:15:32.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:32.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:32.810241+0000) 2024-03-06T10:15:33.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:33.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:33.810369+0000) 2024-03-06T10:15:34.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:34.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:34.810595+0000) 2024-03-06T10:15:35.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:35.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:35.810831+0000) 2024-03-06T10:15:36.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:36.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:36.811061+0000) 2024-03-06T10:15:37.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:37.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:37.811314+0000) 2024-03-06T10:15:38.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:38.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:38.811554+0000) 2024-03-06T10:15:39.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:39.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:39.811811+0000) 2024-03-06T10:15:40.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:40.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:40.812025+0000) 2024-03-06T10:15:41.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:41.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:41.812170+0000) 2024-03-06T10:15:42.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:42.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:42.812343+0000) 2024-03-06T10:15:43.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:43.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:43.812549+0000) 2024-03-06T10:15:44.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:44.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:44.812828+0000) 2024-03-06T10:15:45.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:45.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:45.813084+0000) 2024-03-06T10:15:46.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:46.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:46.813331+0000) 2024-03-06T10:15:47.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:47.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:47.813567+0000) 2024-03-06T10:15:48.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:48.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:48.813807+0000) 2024-03-06T10:15:49.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:49.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:49.813995+0000) 2024-03-06T10:15:50.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:50.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:50.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:50.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:50.814217+0000) 2024-03-06T10:15:51.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:51.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:51.814444+0000) 2024-03-06T10:15:52.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:52.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:52.814707+0000) 2024-03-06T10:15:53.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:53.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:53.814948+0000) 2024-03-06T10:15:54.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:54.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:54.815153+0000) 2024-03-06T10:15:55.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:55.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:55.815385+0000) 2024-03-06T10:15:56.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:56.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:56.815581+0000) 2024-03-06T10:15:57.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:57.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:57.815773+0000) 2024-03-06T10:15:58.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:58.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:58.816004+0000) 2024-03-06T10:15:59.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:15:59.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:59.816216+0000) 2024-03-06T10:16:00.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:00.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:00.816395+0000) 2024-03-06T10:16:01.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:01.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:01.816609+0000) 2024-03-06T10:16:02.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:02.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:02.816841+0000) 2024-03-06T10:16:03.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:03.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:03.817064+0000) 2024-03-06T10:16:04.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:04.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:04.817307+0000) 2024-03-06T10:16:05.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:05.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:05.817537+0000) 2024-03-06T10:16:06.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:06.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:06.817794+0000) 2024-03-06T10:16:07.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:07.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:07.817994+0000) 2024-03-06T10:16:08.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:08.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:08.818236+0000) 2024-03-06T10:16:09.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:09.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:09.818476+0000) 2024-03-06T10:16:10.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:10.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:10.818645+0000) 2024-03-06T10:16:11.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:11.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:11.818911+0000) 2024-03-06T10:16:12.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:12.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:12.819140+0000) 2024-03-06T10:16:13.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:13.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:13.819373+0000) 2024-03-06T10:16:14.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:14.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:14.819578+0000) 2024-03-06T10:16:15.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:15.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:15.819818+0000) 2024-03-06T10:16:16.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:16.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:16.819979+0000) 2024-03-06T10:16:17.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:17.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:17.820184+0000) 2024-03-06T10:16:18.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:18.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:18.820418+0000) 2024-03-06T10:16:19.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:19.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:19.820697+0000) 2024-03-06T10:16:20.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:20.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:20.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:20.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:20.820896+0000) 2024-03-06T10:16:21.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:21.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:21.821109+0000) 2024-03-06T10:16:22.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:22.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:22.821252+0000) 2024-03-06T10:16:23.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:23.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:23.821475+0000) 2024-03-06T10:16:24.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:24.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:24.821718+0000) 2024-03-06T10:16:25.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:25.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:25.821962+0000) 2024-03-06T10:16:26.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:26.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:26.822195+0000) 2024-03-06T10:16:27.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:27.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:27.822433+0000) 2024-03-06T10:16:28.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:28.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:28.822706+0000) 2024-03-06T10:16:29.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:29.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:29.822939+0000) 2024-03-06T10:16:30.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:30.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:30.823170+0000) 2024-03-06T10:16:31.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:31.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:31.823389+0000) 2024-03-06T10:16:32.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:32.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:32.823610+0000) 2024-03-06T10:16:33.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:33.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:33.823879+0000) 2024-03-06T10:16:34.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:34.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:34.824092+0000) 2024-03-06T10:16:35.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:35.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:35.824334+0000) 2024-03-06T10:16:36.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:36.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:36.824519+0000) 2024-03-06T10:16:37.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:37.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:37.824748+0000) 2024-03-06T10:16:38.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:38.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:38.824954+0000) 2024-03-06T10:16:39.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:39.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:39.825196+0000) 2024-03-06T10:16:40.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:40.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:40.825387+0000) 2024-03-06T10:16:41.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:41.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:41.825610+0000) 2024-03-06T10:16:42.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:42.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:42.825839+0000) 2024-03-06T10:16:43.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:43.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:43.826060+0000) 2024-03-06T10:16:44.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:44.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:44.826294+0000) 2024-03-06T10:16:45.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:45.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:45.826494+0000) 2024-03-06T10:16:46.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:46.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:46.826779+0000) 2024-03-06T10:16:47.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:47.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:47.826976+0000) 2024-03-06T10:16:48.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:48.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:48.827200+0000) 2024-03-06T10:16:49.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:49.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:49.827435+0000) 2024-03-06T10:16:50.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:50.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:50.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:50.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:50.827720+0000) 2024-03-06T10:16:51.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:51.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:51.827984+0000) 2024-03-06T10:16:52.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:52.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:52.828276+0000) 2024-03-06T10:16:53.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:53.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:53.828513+0000) 2024-03-06T10:16:54.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:54.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:54.828775+0000) 2024-03-06T10:16:55.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:55.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:55.828987+0000) 2024-03-06T10:16:56.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:56.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:56.829191+0000) 2024-03-06T10:16:57.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:57.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:57.829440+0000) 2024-03-06T10:16:58.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:58.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:58.829718+0000) 2024-03-06T10:16:59.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:16:59.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:59.829969+0000) 2024-03-06T10:17:00.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:00.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:00.830203+0000) 2024-03-06T10:17:01.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:01.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:01.830442+0000) 2024-03-06T10:17:02.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:02.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:02.830632+0000) 2024-03-06T10:17:03.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:03.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:03.830842+0000) 2024-03-06T10:17:04.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:04.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:04.831036+0000) 2024-03-06T10:17:05.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:05.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:05.831230+0000) 2024-03-06T10:17:06.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:06.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:06.831445+0000) 2024-03-06T10:17:07.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:07.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:07.831738+0000) 2024-03-06T10:17:08.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:08.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:08.831953+0000) 2024-03-06T10:17:09.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:09.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:09.832211+0000) 2024-03-06T10:17:10.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:10.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:10.832477+0000) 2024-03-06T10:17:11.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:11.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:11.832786+0000) 2024-03-06T10:17:12.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:12.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:12.832998+0000) 2024-03-06T10:17:13.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:13.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:13.833220+0000) 2024-03-06T10:17:14.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:14.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:14.833466+0000) 2024-03-06T10:17:15.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:15.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:15.833689+0000) 2024-03-06T10:17:16.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:16.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:16.833911+0000) 2024-03-06T10:17:17.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:17.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:17.834078+0000) 2024-03-06T10:17:18.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:18.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:18.834326+0000) 2024-03-06T10:17:19.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:19.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:19.834604+0000) 2024-03-06T10:17:20.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:20.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:20.834876+0000) 2024-03-06T10:17:20.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:20.951 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:21.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:21.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:21.835086+0000) 2024-03-06T10:17:22.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:22.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:22.835356+0000) 2024-03-06T10:17:23.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:23.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:23.835565+0000) 2024-03-06T10:17:24.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:24.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:24.835852+0000) 2024-03-06T10:17:25.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:25.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:25.836096+0000) 2024-03-06T10:17:26.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:26.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:26.836379+0000) 2024-03-06T10:17:27.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:27.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:27.836612+0000) 2024-03-06T10:17:28.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:28.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:28.836875+0000) 2024-03-06T10:17:29.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:29.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:29.837084+0000) 2024-03-06T10:17:30.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:30.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:30.837335+0000) 2024-03-06T10:17:31.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:31.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:31.837610+0000) 2024-03-06T10:17:32.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:32.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:32.837830+0000) 2024-03-06T10:17:33.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:33.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:33.838045+0000) 2024-03-06T10:17:34.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:34.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:34.838301+0000) 2024-03-06T10:17:35.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:35.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:35.838535+0000) 2024-03-06T10:17:36.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:36.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:36.838802+0000) 2024-03-06T10:17:37.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:37.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:37.838999+0000) 2024-03-06T10:17:38.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:38.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:38.839199+0000) 2024-03-06T10:17:39.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:39.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:39.839488+0000) 2024-03-06T10:17:40.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:40.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:40.839795+0000) 2024-03-06T10:17:41.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:41.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:41.840074+0000) 2024-03-06T10:17:42.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:42.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:42.840274+0000) 2024-03-06T10:17:43.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:43.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:43.840503+0000) 2024-03-06T10:17:44.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:44.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:44.840811+0000) 2024-03-06T10:17:45.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:45.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:45.841023+0000) 2024-03-06T10:17:46.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:46.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:46.841212+0000) 2024-03-06T10:17:47.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:47.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:47.841415+0000) 2024-03-06T10:17:48.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:48.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:48.841638+0000) 2024-03-06T10:17:49.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:49.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:49.841932+0000) 2024-03-06T10:17:50.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:50.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:50.842191+0000) 2024-03-06T10:17:51.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:51.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:51.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:51.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:51.842382+0000) 2024-03-06T10:17:52.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:52.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:52.842600+0000) 2024-03-06T10:17:53.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:53.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:53.842861+0000) 2024-03-06T10:17:54.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:54.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:54.843161+0000) 2024-03-06T10:17:55.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:55.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:55.843376+0000) 2024-03-06T10:17:56.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:56.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:56.843608+0000) 2024-03-06T10:17:57.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:57.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:57.843812+0000) 2024-03-06T10:17:58.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:58.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:58.844049+0000) 2024-03-06T10:17:59.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:17:59.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:59.844275+0000) 2024-03-06T10:18:00.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:00.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:00.844522+0000) 2024-03-06T10:18:01.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:01.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:01.844760+0000) 2024-03-06T10:18:02.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:02.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:02.844993+0000) 2024-03-06T10:18:03.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:03.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:03.845227+0000) 2024-03-06T10:18:04.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:04.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:04.845463+0000) 2024-03-06T10:18:05.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:05.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:05.845753+0000) 2024-03-06T10:18:06.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:06.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:06.845981+0000) 2024-03-06T10:18:07.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:07.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:07.846207+0000) 2024-03-06T10:18:08.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:08.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:08.846409+0000) 2024-03-06T10:18:09.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:09.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:09.846712+0000) 2024-03-06T10:18:10.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:10.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:10.846958+0000) 2024-03-06T10:18:11.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:11.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:11.847195+0000) 2024-03-06T10:18:12.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:12.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:12.847433+0000) 2024-03-06T10:18:13.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:13.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:13.847720+0000) 2024-03-06T10:18:14.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:14.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:14.847994+0000) 2024-03-06T10:18:15.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:15.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:15.848271+0000) 2024-03-06T10:18:16.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:16.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:16.848520+0000) 2024-03-06T10:18:17.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:17.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:17.848807+0000) 2024-03-06T10:18:18.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:18.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:18.848999+0000) 2024-03-06T10:18:19.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:19.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:19.849243+0000) 2024-03-06T10:18:20.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:20.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:20.849393+0000) 2024-03-06T10:18:21.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:21.244 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:21.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:21.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:21.849553+0000) 2024-03-06T10:18:22.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:22.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:22.849779+0000) 2024-03-06T10:18:23.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:23.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:23.849998+0000) 2024-03-06T10:18:24.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:24.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:24.850205+0000) 2024-03-06T10:18:25.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:25.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:25.850442+0000) 2024-03-06T10:18:26.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:26.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:26.850621+0000) 2024-03-06T10:18:27.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:27.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:27.850858+0000) 2024-03-06T10:18:28.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:28.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:28.851094+0000) 2024-03-06T10:18:29.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:29.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:29.851318+0000) 2024-03-06T10:18:30.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:30.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:30.851542+0000) 2024-03-06T10:18:31.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:31.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:31.851768+0000) 2024-03-06T10:18:32.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:32.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:32.851993+0000) 2024-03-06T10:18:33.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:33.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:33.852241+0000) 2024-03-06T10:18:34.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:34.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:34.852485+0000) 2024-03-06T10:18:35.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:35.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:35.852730+0000) 2024-03-06T10:18:36.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:36.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:36.852947+0000) 2024-03-06T10:18:37.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:37.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:37.853168+0000) 2024-03-06T10:18:38.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:38.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:38.853532+0000) 2024-03-06T10:18:39.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:39.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:39.853789+0000) 2024-03-06T10:18:40.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:40.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:40.854011+0000) 2024-03-06T10:18:41.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:41.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:41.854190+0000) 2024-03-06T10:18:42.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:42.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:42.854444+0000) 2024-03-06T10:18:43.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:43.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:43.854772+0000) 2024-03-06T10:18:44.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:44.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:44.855016+0000) 2024-03-06T10:18:45.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:45.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:45.855216+0000) 2024-03-06T10:18:46.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:46.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:46.855437+0000) 2024-03-06T10:18:47.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:47.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:47.855725+0000) 2024-03-06T10:18:48.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:48.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:48.855987+0000) 2024-03-06T10:18:49.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:49.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:49.856194+0000) 2024-03-06T10:18:50.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:50.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:50.856421+0000) 2024-03-06T10:18:51.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:51.449 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:51.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:51.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:51.856650+0000) 2024-03-06T10:18:52.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:52.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:52.856949+0000) 2024-03-06T10:18:53.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:53.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:53.857174+0000) 2024-03-06T10:18:54.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:54.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:54.857400+0000) 2024-03-06T10:18:55.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:55.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:55.857598+0000) 2024-03-06T10:18:56.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:56.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:56.857800+0000) 2024-03-06T10:18:57.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:57.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:57.857990+0000) 2024-03-06T10:18:58.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:58.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:58.858161+0000) 2024-03-06T10:18:59.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:18:59.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:59.858366+0000) 2024-03-06T10:19:00.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:00.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:00.858556+0000) 2024-03-06T10:19:01.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:01.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:01.858801+0000) 2024-03-06T10:19:02.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:02.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:02.858970+0000) 2024-03-06T10:19:03.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:03.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:03.859204+0000) 2024-03-06T10:19:04.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:04.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:04.859427+0000) 2024-03-06T10:19:05.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:05.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:05.859616+0000) 2024-03-06T10:19:06.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:06.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:06.859847+0000) 2024-03-06T10:19:07.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:07.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:07.860071+0000) 2024-03-06T10:19:08.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:08.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:08.860295+0000) 2024-03-06T10:19:09.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:09.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:09.860518+0000) 2024-03-06T10:19:10.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:10.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:10.860766+0000) 2024-03-06T10:19:11.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:11.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:11.860992+0000) 2024-03-06T10:19:12.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:12.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:12.861216+0000) 2024-03-06T10:19:13.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:13.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:13.861460+0000) 2024-03-06T10:19:14.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:14.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:14.861767+0000) 2024-03-06T10:19:15.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:15.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:15.861999+0000) 2024-03-06T10:19:16.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:16.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:16.862183+0000) 2024-03-06T10:19:17.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:17.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:17.862405+0000) 2024-03-06T10:19:18.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:18.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:18.862636+0000) 2024-03-06T10:19:19.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:19.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:19.862966+0000) 2024-03-06T10:19:20.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:20.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:20.863160+0000) 2024-03-06T10:19:21.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:21.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:21.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:21.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:21.863339+0000) 2024-03-06T10:19:22.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:22.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:22.863562+0000) 2024-03-06T10:19:23.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:23.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:23.863789+0000) 2024-03-06T10:19:24.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:24.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:24.864015+0000) 2024-03-06T10:19:25.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:25.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:25.864250+0000) 2024-03-06T10:19:26.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:26.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:26.864483+0000) 2024-03-06T10:19:27.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:27.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:27.864798+0000) 2024-03-06T10:19:28.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:28.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:28.865001+0000) 2024-03-06T10:19:29.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:29.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:29.865205+0000) 2024-03-06T10:19:30.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:30.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:30.865469+0000) 2024-03-06T10:19:31.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:31.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:31.865782+0000) 2024-03-06T10:19:32.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:32.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:32.866057+0000) 2024-03-06T10:19:33.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:33.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:33.866337+0000) 2024-03-06T10:19:34.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:34.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:34.866619+0000) 2024-03-06T10:19:35.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:35.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:35.866907+0000) 2024-03-06T10:19:36.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:36.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:36.867154+0000) 2024-03-06T10:19:37.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:37.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:37.867363+0000) 2024-03-06T10:19:38.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:38.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:38.867589+0000) 2024-03-06T10:19:39.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:39.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:39.867833+0000) 2024-03-06T10:19:40.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:40.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:40.868061+0000) 2024-03-06T10:19:41.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:41.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:41.868280+0000) 2024-03-06T10:19:42.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:42.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:42.868546+0000) 2024-03-06T10:19:43.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:43.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:43.868790+0000) 2024-03-06T10:19:44.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:44.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:44.869021+0000) 2024-03-06T10:19:45.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:45.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:45.869175+0000) 2024-03-06T10:19:46.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:46.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:46.869365+0000) 2024-03-06T10:19:47.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:47.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:47.869566+0000) 2024-03-06T10:19:48.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:48.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:48.869776+0000) 2024-03-06T10:19:49.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:49.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:49.870017+0000) 2024-03-06T10:19:50.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:50.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:50.870238+0000) 2024-03-06T10:19:51.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:51.871 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:51.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:51.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:51.870489+0000) 2024-03-06T10:19:52.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:52.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:52.870765+0000) 2024-03-06T10:19:53.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:53.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:53.871000+0000) 2024-03-06T10:19:54.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:54.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:54.871216+0000) 2024-03-06T10:19:55.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:55.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:55.871445+0000) 2024-03-06T10:19:56.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:56.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:56.871633+0000) 2024-03-06T10:19:57.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:57.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:57.871871+0000) 2024-03-06T10:19:58.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:58.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:58.872066+0000) 2024-03-06T10:19:59.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:19:59.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:59.872286+0000) 2024-03-06T10:20:00.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:00.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:00.872503+0000) 2024-03-06T10:20:01.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:01.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:01.872775+0000) 2024-03-06T10:20:02.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:02.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:02.872999+0000) 2024-03-06T10:20:03.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:03.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:03.873231+0000) 2024-03-06T10:20:04.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:04.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:04.873459+0000) 2024-03-06T10:20:05.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:05.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:05.873724+0000) 2024-03-06T10:20:06.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:06.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:06.873952+0000) 2024-03-06T10:20:07.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:07.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:07.874187+0000) 2024-03-06T10:20:08.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:08.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:08.874379+0000) 2024-03-06T10:20:09.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:09.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:09.874597+0000) 2024-03-06T10:20:10.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:10.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:10.874865+0000) 2024-03-06T10:20:11.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:11.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:11.875147+0000) 2024-03-06T10:20:12.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:12.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:12.875379+0000) 2024-03-06T10:20:13.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:13.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:13.875655+0000) 2024-03-06T10:20:14.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:14.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:14.875989+0000) 2024-03-06T10:20:15.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:15.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:15.876219+0000) 2024-03-06T10:20:16.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:16.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:16.876499+0000) 2024-03-06T10:20:17.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:17.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:17.876810+0000) 2024-03-06T10:20:18.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:18.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:18.877040+0000) 2024-03-06T10:20:19.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:19.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:19.877308+0000) 2024-03-06T10:20:20.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:20.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:20.877585+0000) 2024-03-06T10:20:21.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:21.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:21.877832+0000) 2024-03-06T10:20:22.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:22.076 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:22.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:22.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:22.878062+0000) 2024-03-06T10:20:23.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:23.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:23.878279+0000) 2024-03-06T10:20:24.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:24.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:24.878480+0000) 2024-03-06T10:20:25.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:25.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:25.878802+0000) 2024-03-06T10:20:26.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:26.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:26.879026+0000) 2024-03-06T10:20:27.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:27.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:27.879250+0000) 2024-03-06T10:20:28.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:28.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:28.879480+0000) 2024-03-06T10:20:29.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:29.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:29.879742+0000) 2024-03-06T10:20:30.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:30.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:30.879970+0000) 2024-03-06T10:20:31.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:31.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:31.880198+0000) 2024-03-06T10:20:32.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:32.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:32.880426+0000) 2024-03-06T10:20:33.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:33.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:33.880654+0000) 2024-03-06T10:20:34.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:34.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:34.880927+0000) 2024-03-06T10:20:35.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:35.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:35.881154+0000) 2024-03-06T10:20:36.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:36.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:36.881387+0000) 2024-03-06T10:20:37.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:37.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:37.881595+0000) 2024-03-06T10:20:38.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:38.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:38.881822+0000) 2024-03-06T10:20:39.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:39.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:39.882015+0000) 2024-03-06T10:20:40.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:40.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:40.882238+0000) 2024-03-06T10:20:41.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:41.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:41.882471+0000) 2024-03-06T10:20:42.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:42.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:42.882731+0000) 2024-03-06T10:20:43.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:43.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:43.882964+0000) 2024-03-06T10:20:44.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:44.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:44.883187+0000) 2024-03-06T10:20:45.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:45.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:45.883417+0000) 2024-03-06T10:20:46.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:46.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:46.883648+0000) 2024-03-06T10:20:47.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:47.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:47.883904+0000) 2024-03-06T10:20:48.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:48.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:48.884135+0000) 2024-03-06T10:20:49.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:49.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:49.884330+0000) 2024-03-06T10:20:50.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:50.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:50.884569+0000) 2024-03-06T10:20:51.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:51.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:51.884791+0000) 2024-03-06T10:20:52.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:52.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:52.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:52.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:52.885029+0000) 2024-03-06T10:20:53.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:53.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:53.885263+0000) 2024-03-06T10:20:54.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:54.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:54.885493+0000) 2024-03-06T10:20:55.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:55.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:55.885764+0000) 2024-03-06T10:20:56.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:56.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:56.885999+0000) 2024-03-06T10:20:57.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:57.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:57.886234+0000) 2024-03-06T10:20:58.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:58.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:58.886463+0000) 2024-03-06T10:20:59.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:20:59.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:59.886641+0000) 2024-03-06T10:21:00.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:00.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:00.886914+0000) 2024-03-06T10:21:01.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:01.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:01.887124+0000) 2024-03-06T10:21:02.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:02.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:02.887353+0000) 2024-03-06T10:21:03.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:03.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:03.887580+0000) 2024-03-06T10:21:04.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:04.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:04.887795+0000) 2024-03-06T10:21:05.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:05.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:05.888035+0000) 2024-03-06T10:21:06.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:06.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:06.888230+0000) 2024-03-06T10:21:07.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:07.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:07.888448+0000) 2024-03-06T10:21:08.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:08.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:08.888708+0000) 2024-03-06T10:21:09.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:09.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:09.888911+0000) 2024-03-06T10:21:10.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:10.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:10.889145+0000) 2024-03-06T10:21:11.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:11.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:11.889374+0000) 2024-03-06T10:21:12.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:12.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:12.889604+0000) 2024-03-06T10:21:13.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:13.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:13.889890+0000) 2024-03-06T10:21:14.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:14.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:14.890091+0000) 2024-03-06T10:21:15.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:15.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:15.890291+0000) 2024-03-06T10:21:16.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:16.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:16.890530+0000) 2024-03-06T10:21:17.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:17.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:17.890753+0000) 2024-03-06T10:21:18.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:18.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:18.890975+0000) 2024-03-06T10:21:19.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:19.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:19.891118+0000) 2024-03-06T10:21:20.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:20.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:20.891304+0000) 2024-03-06T10:21:21.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:21.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:21.891528+0000) 2024-03-06T10:21:22.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:22.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:22.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:22.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:22.891784+0000) 2024-03-06T10:21:23.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:23.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:23.892003+0000) 2024-03-06T10:21:24.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:24.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:24.892239+0000) 2024-03-06T10:21:25.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:25.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:25.892491+0000) 2024-03-06T10:21:26.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:26.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:26.892756+0000) 2024-03-06T10:21:27.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:27.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:27.893006+0000) 2024-03-06T10:21:28.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:28.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:28.893229+0000) 2024-03-06T10:21:29.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:29.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:29.893438+0000) 2024-03-06T10:21:30.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:30.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:30.893586+0000) 2024-03-06T10:21:31.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:31.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:31.893824+0000) 2024-03-06T10:21:32.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:32.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:32.893997+0000) 2024-03-06T10:21:33.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:33.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:33.894194+0000) 2024-03-06T10:21:34.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:34.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:34.894420+0000) 2024-03-06T10:21:35.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:35.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:35.894651+0000) 2024-03-06T10:21:36.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:36.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:36.894923+0000) 2024-03-06T10:21:37.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:37.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:37.895170+0000) 2024-03-06T10:21:38.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:38.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:38.895406+0000) 2024-03-06T10:21:39.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:39.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:39.895591+0000) 2024-03-06T10:21:40.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:40.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:40.895847+0000) 2024-03-06T10:21:41.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:41.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:41.896044+0000) 2024-03-06T10:21:42.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:42.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:42.896319+0000) 2024-03-06T10:21:43.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:43.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:43.896538+0000) 2024-03-06T10:21:44.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:44.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:44.896814+0000) 2024-03-06T10:21:45.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:45.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:45.897072+0000) 2024-03-06T10:21:46.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:46.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:46.897309+0000) 2024-03-06T10:21:47.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:47.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:47.897530+0000) 2024-03-06T10:21:48.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:48.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:48.897758+0000) 2024-03-06T10:21:49.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:49.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:49.897958+0000) 2024-03-06T10:21:50.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:50.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:50.898164+0000) 2024-03-06T10:21:51.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:51.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:51.898379+0000) 2024-03-06T10:21:52.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:52.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:52.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:52.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:52.898584+0000) 2024-03-06T10:21:53.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:53.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:53.898803+0000) 2024-03-06T10:21:54.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:54.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:54.899037+0000) 2024-03-06T10:21:55.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:55.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:55.899237+0000) 2024-03-06T10:21:56.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:56.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:56.899475+0000) 2024-03-06T10:21:57.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:57.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:57.899729+0000) 2024-03-06T10:21:58.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:58.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:58.899943+0000) 2024-03-06T10:21:59.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:21:59.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:59.900134+0000) 2024-03-06T10:22:00.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:00.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:00.900362+0000) 2024-03-06T10:22:01.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:01.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:01.900611+0000) 2024-03-06T10:22:02.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:02.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:02.900857+0000) 2024-03-06T10:22:03.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:03.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:03.901084+0000) 2024-03-06T10:22:04.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:04.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:04.901268+0000) 2024-03-06T10:22:05.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:05.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:05.901525+0000) 2024-03-06T10:22:06.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:06.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:06.901835+0000) 2024-03-06T10:22:07.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:07.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:07.902067+0000) 2024-03-06T10:22:08.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:08.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:08.902286+0000) 2024-03-06T10:22:09.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:09.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:09.902515+0000) 2024-03-06T10:22:10.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:10.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:10.902787+0000) 2024-03-06T10:22:11.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:11.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:11.902983+0000) 2024-03-06T10:22:12.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:12.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:12.903213+0000) 2024-03-06T10:22:13.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:13.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:13.903412+0000) 2024-03-06T10:22:14.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:14.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:14.903633+0000) 2024-03-06T10:22:15.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:15.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:15.903920+0000) 2024-03-06T10:22:16.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:16.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:16.904154+0000) 2024-03-06T10:22:17.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:17.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:17.904348+0000) 2024-03-06T10:22:18.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:18.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:18.904535+0000) 2024-03-06T10:22:19.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:19.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:19.904793+0000) 2024-03-06T10:22:20.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:20.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:20.904945+0000) 2024-03-06T10:22:21.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:21.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:21.905170+0000) 2024-03-06T10:22:22.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:22.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:22.905358+0000) 2024-03-06T10:22:22.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:22.953 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:23.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:23.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:23.905531+0000) 2024-03-06T10:22:24.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:24.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:24.905797+0000) 2024-03-06T10:22:25.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:25.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:25.906034+0000) 2024-03-06T10:22:26.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:26.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:26.906251+0000) 2024-03-06T10:22:27.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:27.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:27.906469+0000) 2024-03-06T10:22:28.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:28.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:28.906724+0000) 2024-03-06T10:22:29.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:29.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:29.906905+0000) 2024-03-06T10:22:30.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:30.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:30.907130+0000) 2024-03-06T10:22:31.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:31.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:31.907354+0000) 2024-03-06T10:22:32.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:32.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:32.907505+0000) 2024-03-06T10:22:33.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:33.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:33.907733+0000) 2024-03-06T10:22:34.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:34.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:34.907964+0000) 2024-03-06T10:22:35.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:35.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:35.908221+0000) 2024-03-06T10:22:36.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:36.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:36.908465+0000) 2024-03-06T10:22:37.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:37.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:37.908712+0000) 2024-03-06T10:22:38.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:38.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:38.908950+0000) 2024-03-06T10:22:39.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:39.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:39.909170+0000) 2024-03-06T10:22:40.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:40.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:40.909356+0000) 2024-03-06T10:22:41.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:41.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:41.909552+0000) 2024-03-06T10:22:42.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:42.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:42.909782+0000) 2024-03-06T10:22:43.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:43.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:43.910012+0000) 2024-03-06T10:22:44.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:44.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:44.910234+0000) 2024-03-06T10:22:45.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:45.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:45.910467+0000) 2024-03-06T10:22:46.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:46.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:46.910737+0000) 2024-03-06T10:22:47.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:47.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:47.910965+0000) 2024-03-06T10:22:48.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:48.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:48.911149+0000) 2024-03-06T10:22:49.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:49.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:49.911357+0000) 2024-03-06T10:22:50.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:50.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:50.911504+0000) 2024-03-06T10:22:51.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:51.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:51.911747+0000) 2024-03-06T10:22:52.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:52.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:52.911993+0000) 2024-03-06T10:22:53.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:53.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:53.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:53.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:53.912224+0000) 2024-03-06T10:22:54.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:54.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:54.912497+0000) 2024-03-06T10:22:55.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:55.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:55.912772+0000) 2024-03-06T10:22:56.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:56.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:56.913012+0000) 2024-03-06T10:22:57.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:57.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:57.913251+0000) 2024-03-06T10:22:58.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:58.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:58.913490+0000) 2024-03-06T10:22:59.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:22:59.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:59.913767+0000) 2024-03-06T10:23:00.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:00.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:00.913961+0000) 2024-03-06T10:23:01.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:01.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:01.914186+0000) 2024-03-06T10:23:02.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:02.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:02.914365+0000) 2024-03-06T10:23:03.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:03.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:03.914596+0000) 2024-03-06T10:23:04.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:04.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:04.914826+0000) 2024-03-06T10:23:05.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:05.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:05.915064+0000) 2024-03-06T10:23:06.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:06.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:06.915289+0000) 2024-03-06T10:23:07.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:07.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:07.915537+0000) 2024-03-06T10:23:08.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:08.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:08.915734+0000) 2024-03-06T10:23:09.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:09.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:09.915953+0000) 2024-03-06T10:23:10.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:10.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:10.916197+0000) 2024-03-06T10:23:11.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:11.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:11.916428+0000) 2024-03-06T10:23:12.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:12.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:12.916711+0000) 2024-03-06T10:23:13.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:13.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:13.916985+0000) 2024-03-06T10:23:14.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:14.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:14.917270+0000) 2024-03-06T10:23:15.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:15.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:15.917504+0000) 2024-03-06T10:23:16.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:16.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:16.917790+0000) 2024-03-06T10:23:17.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:17.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:17.918018+0000) 2024-03-06T10:23:18.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:18.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:18.918219+0000) 2024-03-06T10:23:19.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:19.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:19.918412+0000) 2024-03-06T10:23:20.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:20.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:20.918657+0000) 2024-03-06T10:23:21.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:21.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:21.918890+0000) 2024-03-06T10:23:22.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:22.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:22.919110+0000) 2024-03-06T10:23:23.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:23.287 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:23.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:23.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:23.919307+0000) 2024-03-06T10:23:24.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:24.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:24.919520+0000) 2024-03-06T10:23:25.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:25.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:25.919801+0000) 2024-03-06T10:23:26.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:26.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:26.920037+0000) 2024-03-06T10:23:27.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:27.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:27.920279+0000) 2024-03-06T10:23:28.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:28.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:28.920468+0000) 2024-03-06T10:23:29.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:29.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:29.920729+0000) 2024-03-06T10:23:30.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:30.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:30.920928+0000) 2024-03-06T10:23:31.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:31.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:31.921108+0000) 2024-03-06T10:23:32.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:32.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:32.921342+0000) 2024-03-06T10:23:33.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:33.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:33.921581+0000) 2024-03-06T10:23:34.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:34.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:34.921811+0000) 2024-03-06T10:23:35.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:35.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:35.922038+0000) 2024-03-06T10:23:36.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:36.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:36.922204+0000) 2024-03-06T10:23:37.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:37.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:37.922353+0000) 2024-03-06T10:23:38.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:38.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:38.922574+0000) 2024-03-06T10:23:39.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:39.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:39.922799+0000) 2024-03-06T10:23:40.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:40.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:40.923038+0000) 2024-03-06T10:23:41.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:41.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:41.923219+0000) 2024-03-06T10:23:42.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:42.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:42.923446+0000) 2024-03-06T10:23:43.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:43.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:43.923711+0000) 2024-03-06T10:23:44.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:44.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:44.923913+0000) 2024-03-06T10:23:45.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:45.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:45.924128+0000) 2024-03-06T10:23:46.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:46.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:46.924358+0000) 2024-03-06T10:23:47.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:47.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:47.924555+0000) 2024-03-06T10:23:48.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:48.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:48.924790+0000) 2024-03-06T10:23:49.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:49.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:49.925011+0000) 2024-03-06T10:23:50.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:50.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:50.925254+0000) 2024-03-06T10:23:51.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:51.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:51.925450+0000) 2024-03-06T10:23:52.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:52.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:52.925624+0000) 2024-03-06T10:23:53.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:53.483 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:53.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:53.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:53.925913+0000) 2024-03-06T10:23:54.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:54.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:54.926135+0000) 2024-03-06T10:23:55.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:55.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:55.926365+0000) 2024-03-06T10:23:56.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:56.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:56.926517+0000) 2024-03-06T10:23:57.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:57.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:57.926810+0000) 2024-03-06T10:23:58.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:58.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:58.927036+0000) 2024-03-06T10:23:59.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:23:59.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:59.927260+0000) 2024-03-06T10:24:00.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:00.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:00.927471+0000) 2024-03-06T10:24:01.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:01.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:01.927653+0000) 2024-03-06T10:24:02.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:02.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:02.927936+0000) 2024-03-06T10:24:03.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:03.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:03.928128+0000) 2024-03-06T10:24:04.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:04.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:04.928343+0000) 2024-03-06T10:24:05.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:05.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:05.928574+0000) 2024-03-06T10:24:06.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:06.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:06.928797+0000) 2024-03-06T10:24:07.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:07.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:07.929017+0000) 2024-03-06T10:24:08.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:08.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:08.929238+0000) 2024-03-06T10:24:09.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:09.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:09.929462+0000) 2024-03-06T10:24:10.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:10.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:10.929636+0000) 2024-03-06T10:24:11.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:11.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:11.929886+0000) 2024-03-06T10:24:12.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:12.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:12.930065+0000) 2024-03-06T10:24:13.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:13.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:13.930263+0000) 2024-03-06T10:24:14.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:14.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:14.930510+0000) 2024-03-06T10:24:15.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:15.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:15.930817+0000) 2024-03-06T10:24:16.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:16.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:16.931061+0000) 2024-03-06T10:24:17.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:17.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:17.931301+0000) 2024-03-06T10:24:18.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:18.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:18.931537+0000) 2024-03-06T10:24:19.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:19.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:19.931774+0000) 2024-03-06T10:24:20.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:20.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:20.931991+0000) 2024-03-06T10:24:21.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:21.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:21.932218+0000) 2024-03-06T10:24:22.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:22.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:22.932419+0000) 2024-03-06T10:24:23.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:23.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:23.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:23.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:23.932661+0000) 2024-03-06T10:24:24.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:24.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:24.932948+0000) 2024-03-06T10:24:25.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:25.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:25.933181+0000) 2024-03-06T10:24:26.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:26.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:26.933406+0000) 2024-03-06T10:24:27.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:27.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:27.933602+0000) 2024-03-06T10:24:28.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:28.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:28.933850+0000) 2024-03-06T10:24:29.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:29.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:29.934044+0000) 2024-03-06T10:24:30.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:30.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:30.934281+0000) 2024-03-06T10:24:31.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:31.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:31.934504+0000) 2024-03-06T10:24:32.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:32.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:32.934732+0000) 2024-03-06T10:24:33.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:33.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:33.934923+0000) 2024-03-06T10:24:34.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:34.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:34.935115+0000) 2024-03-06T10:24:35.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:35.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:35.935316+0000) 2024-03-06T10:24:36.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:36.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:36.935513+0000) 2024-03-06T10:24:37.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:37.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:37.935782+0000) 2024-03-06T10:24:38.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:38.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:38.936029+0000) 2024-03-06T10:24:39.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:39.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:39.936278+0000) 2024-03-06T10:24:40.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:40.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:40.936513+0000) 2024-03-06T10:24:41.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:41.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:41.936835+0000) 2024-03-06T10:24:42.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:42.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:42.937060+0000) 2024-03-06T10:24:43.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:43.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:43.937251+0000) 2024-03-06T10:24:44.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:44.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:44.937469+0000) 2024-03-06T10:24:45.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:45.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:45.937730+0000) 2024-03-06T10:24:46.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:46.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:46.937959+0000) 2024-03-06T10:24:47.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:47.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:47.938175+0000) 2024-03-06T10:24:48.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:48.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:48.938346+0000) 2024-03-06T10:24:49.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:49.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:49.938581+0000) 2024-03-06T10:24:50.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:50.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:50.938740+0000) 2024-03-06T10:24:51.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:51.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:51.938917+0000) 2024-03-06T10:24:52.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:52.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:52.939112+0000) 2024-03-06T10:24:53.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:53.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:53.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:53.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:53.939267+0000) 2024-03-06T10:24:54.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:54.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:54.939444+0000) 2024-03-06T10:24:55.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:55.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:55.939716+0000) 2024-03-06T10:24:56.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:56.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:56.939903+0000) 2024-03-06T10:24:57.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:57.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:57.940092+0000) 2024-03-06T10:24:58.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:58.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:58.940269+0000) 2024-03-06T10:24:59.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:24:59.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:59.940461+0000) 2024-03-06T10:25:00.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:00.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:00.940646+0000) 2024-03-06T10:25:01.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:01.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:01.940925+0000) 2024-03-06T10:25:02.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:02.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:02.941125+0000) 2024-03-06T10:25:03.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:03.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:03.941367+0000) 2024-03-06T10:25:04.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:04.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:04.941590+0000) 2024-03-06T10:25:05.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:05.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:05.941833+0000) 2024-03-06T10:25:06.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:06.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:06.942064+0000) 2024-03-06T10:25:07.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:07.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:07.942286+0000) 2024-03-06T10:25:08.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:08.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:08.942515+0000) 2024-03-06T10:25:09.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:09.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:09.942769+0000) 2024-03-06T10:25:10.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:10.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:10.942947+0000) 2024-03-06T10:25:11.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:11.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:11.943174+0000) 2024-03-06T10:25:12.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:12.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:12.943403+0000) 2024-03-06T10:25:13.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:13.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:13.943623+0000) 2024-03-06T10:25:14.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:14.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:14.943844+0000) 2024-03-06T10:25:15.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:15.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:15.944038+0000) 2024-03-06T10:25:16.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:16.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:16.944276+0000) 2024-03-06T10:25:17.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:17.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:17.944463+0000) 2024-03-06T10:25:18.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:18.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:18.944734+0000) 2024-03-06T10:25:19.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:19.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:19.944954+0000) 2024-03-06T10:25:20.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:20.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:20.945142+0000) 2024-03-06T10:25:21.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:21.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:21.945331+0000) 2024-03-06T10:25:22.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:22.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:22.945554+0000) 2024-03-06T10:25:23.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:23.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:23.945784+0000) 2024-03-06T10:25:24.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:24.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:24.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:24.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:24.946013+0000) 2024-03-06T10:25:25.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:25.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:25.946235+0000) 2024-03-06T10:25:26.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:26.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:26.946461+0000) 2024-03-06T10:25:27.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:27.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:27.946704+0000) 2024-03-06T10:25:28.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:28.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:28.946928+0000) 2024-03-06T10:25:29.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:29.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:29.947207+0000) 2024-03-06T10:25:30.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:30.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:30.947427+0000) 2024-03-06T10:25:31.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:31.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:31.947653+0000) 2024-03-06T10:25:32.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:32.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:32.947940+0000) 2024-03-06T10:25:33.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:33.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:33.948171+0000) 2024-03-06T10:25:34.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:34.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:34.948394+0000) 2024-03-06T10:25:35.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:35.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:35.948623+0000) 2024-03-06T10:25:36.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:36.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:36.948847+0000) 2024-03-06T10:25:37.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:37.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:37.949075+0000) 2024-03-06T10:25:38.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:38.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:38.949305+0000) 2024-03-06T10:25:39.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:39.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:39.949498+0000) 2024-03-06T10:25:40.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:40.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:40.949760+0000) 2024-03-06T10:25:41.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:41.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:41.949982+0000) 2024-03-06T10:25:42.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:42.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:42.950204+0000) 2024-03-06T10:25:43.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:43.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:43.950444+0000) 2024-03-06T10:25:44.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:44.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:44.950623+0000) 2024-03-06T10:25:45.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:45.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:45.950858+0000) 2024-03-06T10:25:46.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:46.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:46.951090+0000) 2024-03-06T10:25:47.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:47.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:47.951314+0000) 2024-03-06T10:25:48.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:48.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:48.951498+0000) 2024-03-06T10:25:49.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:49.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:49.951759+0000) 2024-03-06T10:25:50.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:50.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:50.951981+0000) 2024-03-06T10:25:51.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:51.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:51.952205+0000) 2024-03-06T10:25:52.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:52.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:52.952428+0000) 2024-03-06T10:25:53.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:53.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:53.952654+0000) 2024-03-06T10:25:54.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:54.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:54.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:54.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:54.952873+0000) 2024-03-06T10:25:55.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:55.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:55.953060+0000) 2024-03-06T10:25:56.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:56.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:56.953295+0000) 2024-03-06T10:25:57.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:57.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:57.953477+0000) 2024-03-06T10:25:58.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:58.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:58.953650+0000) 2024-03-06T10:25:59.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:25:59.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:59.953907+0000) 2024-03-06T10:26:00.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:00.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:00.954079+0000) 2024-03-06T10:26:01.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:01.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:01.954310+0000) 2024-03-06T10:26:02.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:02.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:02.954542+0000) 2024-03-06T10:26:03.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:03.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:03.954768+0000) 2024-03-06T10:26:04.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:04.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:04.954984+0000) 2024-03-06T10:26:05.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:05.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:05.955215+0000) 2024-03-06T10:26:06.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:06.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:06.955444+0000) 2024-03-06T10:26:07.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:07.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:07.955693+0000) 2024-03-06T10:26:08.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:08.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:08.955887+0000) 2024-03-06T10:26:09.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:09.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:09.956100+0000) 2024-03-06T10:26:10.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:10.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:10.956316+0000) 2024-03-06T10:26:11.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:11.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:11.956543+0000) 2024-03-06T10:26:12.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:12.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:12.956776+0000) 2024-03-06T10:26:13.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:13.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:13.957004+0000) 2024-03-06T10:26:14.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:14.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:14.957184+0000) 2024-03-06T10:26:15.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:15.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:15.957407+0000) 2024-03-06T10:26:16.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:16.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:16.957587+0000) 2024-03-06T10:26:17.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:17.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:17.957828+0000) 2024-03-06T10:26:18.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:18.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:18.958026+0000) 2024-03-06T10:26:19.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:19.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:19.958227+0000) 2024-03-06T10:26:20.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:20.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:20.958413+0000) 2024-03-06T10:26:21.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:21.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:21.958600+0000) 2024-03-06T10:26:22.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:22.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:22.958891+0000) 2024-03-06T10:26:23.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:23.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:23.959068+0000) 2024-03-06T10:26:24.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:24.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:24.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:24.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:24.959304+0000) 2024-03-06T10:26:25.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:25.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:25.959539+0000) 2024-03-06T10:26:26.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:26.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:26.959787+0000) 2024-03-06T10:26:27.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:27.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:27.960012+0000) 2024-03-06T10:26:28.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:28.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:28.960241+0000) 2024-03-06T10:26:29.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:29.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:29.960464+0000) 2024-03-06T10:26:30.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:30.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:30.960727+0000) 2024-03-06T10:26:31.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:31.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:31.960970+0000) 2024-03-06T10:26:32.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:32.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:32.961200+0000) 2024-03-06T10:26:33.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:33.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:33.961385+0000) 2024-03-06T10:26:34.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:34.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:34.961585+0000) 2024-03-06T10:26:35.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:35.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:35.961819+0000) 2024-03-06T10:26:36.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:36.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:36.962092+0000) 2024-03-06T10:26:37.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:37.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:37.962373+0000) 2024-03-06T10:26:38.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:38.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:38.962649+0000) 2024-03-06T10:26:39.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:39.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:39.962927+0000) 2024-03-06T10:26:40.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:40.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:40.963150+0000) 2024-03-06T10:26:41.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:41.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:41.963403+0000) 2024-03-06T10:26:42.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:42.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:42.963657+0000) 2024-03-06T10:26:43.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:43.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:43.963957+0000) 2024-03-06T10:26:44.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:44.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:44.964204+0000) 2024-03-06T10:26:45.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:45.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:45.964423+0000) 2024-03-06T10:26:46.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:46.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:46.964638+0000) 2024-03-06T10:26:47.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:47.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:47.964919+0000) 2024-03-06T10:26:48.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:48.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:48.965118+0000) 2024-03-06T10:26:49.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:49.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:49.965376+0000) 2024-03-06T10:26:50.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:50.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:50.965615+0000) 2024-03-06T10:26:51.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:51.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:51.965880+0000) 2024-03-06T10:26:52.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:52.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:52.966157+0000) 2024-03-06T10:26:53.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:53.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:53.966360+0000) 2024-03-06T10:26:54.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:54.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:54.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:54.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:54.966543+0000) 2024-03-06T10:26:55.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:55.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:55.966718+0000) 2024-03-06T10:26:56.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:56.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:56.966985+0000) 2024-03-06T10:26:57.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:57.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:57.967255+0000) 2024-03-06T10:26:58.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:58.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:58.967528+0000) 2024-03-06T10:26:59.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:26:59.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:59.967776+0000) 2024-03-06T10:27:00.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:00.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:00.967964+0000) 2024-03-06T10:27:01.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:01.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:01.968195+0000) 2024-03-06T10:27:02.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:02.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:02.968412+0000) 2024-03-06T10:27:03.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:03.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:03.968646+0000) 2024-03-06T10:27:04.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:04.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:04.968907+0000) 2024-03-06T10:27:05.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:05.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:05.969142+0000) 2024-03-06T10:27:06.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:06.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:06.969387+0000) 2024-03-06T10:27:07.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:07.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:07.969598+0000) 2024-03-06T10:27:08.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:08.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:08.969847+0000) 2024-03-06T10:27:09.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:09.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:09.970074+0000) 2024-03-06T10:27:10.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:10.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:10.970335+0000) 2024-03-06T10:27:11.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:11.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:11.970543+0000) 2024-03-06T10:27:12.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:12.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:12.970802+0000) 2024-03-06T10:27:13.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:13.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:13.971038+0000) 2024-03-06T10:27:14.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:14.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:14.971241+0000) 2024-03-06T10:27:15.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:15.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:15.971479+0000) 2024-03-06T10:27:16.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:16.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:16.971856+0000) 2024-03-06T10:27:17.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:17.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:17.972115+0000) 2024-03-06T10:27:18.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:18.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:18.972350+0000) 2024-03-06T10:27:19.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:19.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:19.972607+0000) 2024-03-06T10:27:20.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:20.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:20.972882+0000) 2024-03-06T10:27:21.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:21.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:21.973124+0000) 2024-03-06T10:27:22.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:22.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:22.973393+0000) 2024-03-06T10:27:23.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:23.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:23.973724+0000) 2024-03-06T10:27:24.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:24.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:24.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:24.974004+0000) 2024-03-06T10:27:24.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:25.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:25.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:25.974294+0000) 2024-03-06T10:27:26.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:26.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:26.974502+0000) 2024-03-06T10:27:27.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:27.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:27.974760+0000) 2024-03-06T10:27:28.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:28.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:28.974880+0000) 2024-03-06T10:27:29.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:29.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:29.975106+0000) 2024-03-06T10:27:30.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:30.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:30.975284+0000) 2024-03-06T10:27:31.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:31.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:31.975433+0000) 2024-03-06T10:27:32.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:32.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:32.975643+0000) 2024-03-06T10:27:33.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:33.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:33.975916+0000) 2024-03-06T10:27:34.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:34.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:34.976138+0000) 2024-03-06T10:27:35.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:35.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:35.976376+0000) 2024-03-06T10:27:36.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:36.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:36.976612+0000) 2024-03-06T10:27:37.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:37.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:37.976853+0000) 2024-03-06T10:27:38.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:38.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:38.977082+0000) 2024-03-06T10:27:39.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:39.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:39.977255+0000) 2024-03-06T10:27:40.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:40.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:40.977480+0000) 2024-03-06T10:27:41.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:41.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:41.977772+0000) 2024-03-06T10:27:42.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:42.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:42.978000+0000) 2024-03-06T10:27:43.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:43.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:43.978225+0000) 2024-03-06T10:27:44.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:44.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:44.978425+0000) 2024-03-06T10:27:45.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:45.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:45.978658+0000) 2024-03-06T10:27:46.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:46.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:46.978887+0000) 2024-03-06T10:27:47.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:47.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:47.979118+0000) 2024-03-06T10:27:48.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:48.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:48.979364+0000) 2024-03-06T10:27:49.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:49.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:49.979608+0000) 2024-03-06T10:27:50.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:50.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:50.979806+0000) 2024-03-06T10:27:51.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:51.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:51.980070+0000) 2024-03-06T10:27:52.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:52.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:52.980315+0000) 2024-03-06T10:27:53.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:53.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:53.980590+0000) 2024-03-06T10:27:54.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:54.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:54.980827+0000) 2024-03-06T10:27:55.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:55.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:55.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:55.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:55.981062+0000) 2024-03-06T10:27:56.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:56.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:56.981210+0000) 2024-03-06T10:27:57.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:57.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:57.981428+0000) 2024-03-06T10:27:58.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:58.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:58.981708+0000) 2024-03-06T10:27:59.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:27:59.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:59.981910+0000) 2024-03-06T10:28:00.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:00.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:00.982142+0000) 2024-03-06T10:28:01.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:01.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:01.982373+0000) 2024-03-06T10:28:02.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:02.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:02.982553+0000) 2024-03-06T10:28:03.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:03.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:03.982790+0000) 2024-03-06T10:28:04.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:04.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:04.982980+0000) 2024-03-06T10:28:05.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:05.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:05.983212+0000) 2024-03-06T10:28:06.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:06.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:06.983447+0000) 2024-03-06T10:28:07.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:07.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:07.983653+0000) 2024-03-06T10:28:08.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:08.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:08.983892+0000) 2024-03-06T10:28:09.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:09.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:09.984139+0000) 2024-03-06T10:28:10.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:10.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:10.984321+0000) 2024-03-06T10:28:11.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:11.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:11.984558+0000) 2024-03-06T10:28:12.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:12.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:12.984789+0000) 2024-03-06T10:28:13.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:13.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:13.985009+0000) 2024-03-06T10:28:14.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:14.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:14.985233+0000) 2024-03-06T10:28:15.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:15.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:15.985417+0000) 2024-03-06T10:28:16.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:16.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:16.985608+0000) 2024-03-06T10:28:17.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:17.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:17.985847+0000) 2024-03-06T10:28:18.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:18.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:18.986065+0000) 2024-03-06T10:28:19.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:19.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:19.986340+0000) 2024-03-06T10:28:20.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:20.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:20.986614+0000) 2024-03-06T10:28:21.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:21.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:21.986862+0000) 2024-03-06T10:28:22.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:22.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:22.987089+0000) 2024-03-06T10:28:23.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:23.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:23.987367+0000) 2024-03-06T10:28:24.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:24.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:24.987602+0000) 2024-03-06T10:28:25.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:25.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:25.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:25.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:25.987890+0000) 2024-03-06T10:28:26.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:26.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:26.988092+0000) 2024-03-06T10:28:27.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:27.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:27.988318+0000) 2024-03-06T10:28:28.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:28.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:28.988446+0000) 2024-03-06T10:28:29.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:29.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:29.988660+0000) 2024-03-06T10:28:30.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:30.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:30.988951+0000) 2024-03-06T10:28:31.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:31.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:31.989210+0000) 2024-03-06T10:28:32.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:32.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:32.989460+0000) 2024-03-06T10:28:33.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:33.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:33.989660+0000) 2024-03-06T10:28:34.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:34.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:34.989925+0000) 2024-03-06T10:28:35.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:35.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:35.990154+0000) 2024-03-06T10:28:36.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:36.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:36.990387+0000) 2024-03-06T10:28:37.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:37.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:37.990578+0000) 2024-03-06T10:28:38.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:38.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:38.990822+0000) 2024-03-06T10:28:39.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:39.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:39.991019+0000) 2024-03-06T10:28:40.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:40.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:40.991241+0000) 2024-03-06T10:28:41.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:41.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:41.991488+0000) 2024-03-06T10:28:42.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:42.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:42.991788+0000) 2024-03-06T10:28:43.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:43.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:43.992027+0000) 2024-03-06T10:28:44.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:44.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:44.992237+0000) 2024-03-06T10:28:45.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:45.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:45.992451+0000) 2024-03-06T10:28:46.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:46.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:46.992648+0000) 2024-03-06T10:28:47.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:47.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:47.992907+0000) 2024-03-06T10:28:48.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:48.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:48.993153+0000) 2024-03-06T10:28:49.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:49.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:49.993396+0000) 2024-03-06T10:28:50.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:50.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:50.993622+0000) 2024-03-06T10:28:51.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:51.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:51.993880+0000) 2024-03-06T10:28:52.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:52.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:52.994121+0000) 2024-03-06T10:28:53.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:53.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:53.994275+0000) 2024-03-06T10:28:54.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:54.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:54.994477+0000) 2024-03-06T10:28:55.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:55.549 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:55.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:55.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:55.994832+0000) 2024-03-06T10:28:56.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:56.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:56.995058+0000) 2024-03-06T10:28:57.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:57.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:57.995326+0000) 2024-03-06T10:28:58.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:58.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:58.995509+0000) 2024-03-06T10:28:59.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:28:59.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:59.995748+0000) 2024-03-06T10:29:00.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:00.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:00.995939+0000) 2024-03-06T10:29:01.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:01.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:01.996154+0000) 2024-03-06T10:29:02.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:02.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:02.996359+0000) 2024-03-06T10:29:03.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:03.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:03.996638+0000) 2024-03-06T10:29:04.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:04.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:04.996957+0000) 2024-03-06T10:29:05.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:05.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:05.997239+0000) 2024-03-06T10:29:06.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:06.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:06.997524+0000) 2024-03-06T10:29:07.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:07.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:07.997809+0000) 2024-03-06T10:29:08.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:08.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:08.998015+0000) 2024-03-06T10:29:09.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:09.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:09.998232+0000) 2024-03-06T10:29:10.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:10.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:10.998466+0000) 2024-03-06T10:29:11.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:11.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:11.998730+0000) 2024-03-06T10:29:12.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:12.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:12.998937+0000) 2024-03-06T10:29:13.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:13.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:13.999175+0000) 2024-03-06T10:29:14.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:14.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:14.999401+0000) 2024-03-06T10:29:15.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:15.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:15.999586+0000) 2024-03-06T10:29:16.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:16.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:16.999803+0000) 2024-03-06T10:29:17.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:17.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:18.000038+0000) 2024-03-06T10:29:18.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:18.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:19.000227+0000) 2024-03-06T10:29:20.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:19.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:20.000360+0000) 2024-03-06T10:29:21.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:20.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:21.000571+0000) 2024-03-06T10:29:22.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:22.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:22.000837+0000) 2024-03-06T10:29:23.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:23.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:23.001036+0000) 2024-03-06T10:29:24.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:24.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:24.001267+0000) 2024-03-06T10:29:25.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:25.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:25.001439+0000) 2024-03-06T10:29:25.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:25.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:26.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:26.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:26.001706+0000) 2024-03-06T10:29:27.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:27.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:27.001965+0000) 2024-03-06T10:29:28.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:28.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:28.002210+0000) 2024-03-06T10:29:29.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:29.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:29.002415+0000) 2024-03-06T10:29:30.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:30.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:30.002715+0000) 2024-03-06T10:29:31.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:31.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:31.002907+0000) 2024-03-06T10:29:32.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:32.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:32.003151+0000) 2024-03-06T10:29:33.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:33.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:33.003275+0000) 2024-03-06T10:29:34.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:34.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:34.003445+0000) 2024-03-06T10:29:35.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:35.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:35.003626+0000) 2024-03-06T10:29:36.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:36.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:36.003923+0000) 2024-03-06T10:29:37.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:37.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:37.004145+0000) 2024-03-06T10:29:38.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:38.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:38.004355+0000) 2024-03-06T10:29:39.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:39.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:39.004578+0000) 2024-03-06T10:29:40.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:40.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:40.004801+0000) 2024-03-06T10:29:41.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:41.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:41.005031+0000) 2024-03-06T10:29:42.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:42.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:42.005230+0000) 2024-03-06T10:29:43.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:43.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:43.005478+0000) 2024-03-06T10:29:44.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:44.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:44.005753+0000) 2024-03-06T10:29:45.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:45.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:45.005928+0000) 2024-03-06T10:29:46.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:46.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:46.006213+0000) 2024-03-06T10:29:47.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:47.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:47.006449+0000) 2024-03-06T10:29:48.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:48.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:48.006736+0000) 2024-03-06T10:29:49.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:49.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:49.006914+0000) 2024-03-06T10:29:50.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:50.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:50.007130+0000) 2024-03-06T10:29:51.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:51.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:51.007330+0000) 2024-03-06T10:29:52.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:52.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:52.007570+0000) 2024-03-06T10:29:53.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:53.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:53.007817+0000) 2024-03-06T10:29:54.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:54.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:54.008016+0000) 2024-03-06T10:29:55.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:55.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:55.008251+0000) 2024-03-06T10:29:55.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:55.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:56.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:56.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:56.008436+0000) 2024-03-06T10:29:57.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:57.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:57.008612+0000) 2024-03-06T10:29:58.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:58.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:58.008820+0000) 2024-03-06T10:29:59.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:29:59.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:59.009046+0000) 2024-03-06T10:30:00.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:00.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:00.009235+0000) 2024-03-06T10:30:01.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:01.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:01.009382+0000) 2024-03-06T10:30:02.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:02.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:02.009610+0000) 2024-03-06T10:30:03.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:03.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:03.009843+0000) 2024-03-06T10:30:04.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:04.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:04.010076+0000) 2024-03-06T10:30:05.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:05.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:05.010303+0000) 2024-03-06T10:30:06.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:06.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:06.010536+0000) 2024-03-06T10:30:07.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:07.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:07.010751+0000) 2024-03-06T10:30:08.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:08.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:08.010916+0000) 2024-03-06T10:30:09.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:09.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:09.011134+0000) 2024-03-06T10:30:10.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:10.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:10.011330+0000) 2024-03-06T10:30:11.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:11.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:11.011553+0000) 2024-03-06T10:30:12.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:12.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:12.011779+0000) 2024-03-06T10:30:13.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:13.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:13.011972+0000) 2024-03-06T10:30:14.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:14.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:14.012160+0000) 2024-03-06T10:30:15.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:15.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:15.012391+0000) 2024-03-06T10:30:16.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:16.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:16.012625+0000) 2024-03-06T10:30:17.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:17.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:17.012823+0000) 2024-03-06T10:30:18.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:18.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:18.013036+0000) 2024-03-06T10:30:19.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:19.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:19.013277+0000) 2024-03-06T10:30:20.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:20.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:20.013457+0000) 2024-03-06T10:30:21.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:21.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:21.013694+0000) 2024-03-06T10:30:22.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:22.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:22.013904+0000) 2024-03-06T10:30:23.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:23.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:23.014097+0000) 2024-03-06T10:30:24.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:24.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:24.014281+0000) 2024-03-06T10:30:25.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:25.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:25.014507+0000) 2024-03-06T10:30:26.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:26.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:26.014794+0000) 2024-03-06T10:30:26.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:26.163 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:27.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:27.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:27.015030+0000) 2024-03-06T10:30:28.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:28.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:28.015259+0000) 2024-03-06T10:30:29.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:29.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:29.015434+0000) 2024-03-06T10:30:30.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:30.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:30.015655+0000) 2024-03-06T10:30:31.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:31.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:31.015912+0000) 2024-03-06T10:30:32.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:32.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:32.016154+0000) 2024-03-06T10:30:33.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:33.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:33.016385+0000) 2024-03-06T10:30:34.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:34.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:34.016575+0000) 2024-03-06T10:30:35.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:35.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:35.016787+0000) 2024-03-06T10:30:36.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:36.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:36.016980+0000) 2024-03-06T10:30:37.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:37.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:37.017161+0000) 2024-03-06T10:30:38.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:38.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:38.017405+0000) 2024-03-06T10:30:39.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:39.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:39.017656+0000) 2024-03-06T10:30:40.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:40.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:40.017954+0000) 2024-03-06T10:30:41.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:41.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:41.018142+0000) 2024-03-06T10:30:42.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:42.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:42.018338+0000) 2024-03-06T10:30:43.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:43.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:43.018576+0000) 2024-03-06T10:30:44.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:44.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:44.018808+0000) 2024-03-06T10:30:45.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:45.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:45.019095+0000) 2024-03-06T10:30:46.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:46.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:46.019320+0000) 2024-03-06T10:30:47.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:47.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:47.019545+0000) 2024-03-06T10:30:48.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:48.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:48.019763+0000) 2024-03-06T10:30:49.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:49.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:49.020003+0000) 2024-03-06T10:30:50.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:50.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:50.020235+0000) 2024-03-06T10:30:51.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:51.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:51.020453+0000) 2024-03-06T10:30:52.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:52.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:52.020704+0000) 2024-03-06T10:30:53.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:53.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:53.020936+0000) 2024-03-06T10:30:54.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:54.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:54.021070+0000) 2024-03-06T10:30:55.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:55.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:55.021297+0000) 2024-03-06T10:30:56.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:56.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:56.021532+0000) 2024-03-06T10:30:56.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:56.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:57.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:57.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:57.021797+0000) 2024-03-06T10:30:58.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:58.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:58.022022+0000) 2024-03-06T10:30:59.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:30:59.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:59.022248+0000) 2024-03-06T10:31:00.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:00.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:00.022474+0000) 2024-03-06T10:31:01.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:01.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:01.022731+0000) 2024-03-06T10:31:02.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:02.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:02.022958+0000) 2024-03-06T10:31:03.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:03.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:03.023097+0000) 2024-03-06T10:31:04.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:04.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:04.023315+0000) 2024-03-06T10:31:05.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:05.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:05.023536+0000) 2024-03-06T10:31:06.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:06.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:06.023775+0000) 2024-03-06T10:31:07.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:07.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:07.023999+0000) 2024-03-06T10:31:08.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:08.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:08.024233+0000) 2024-03-06T10:31:09.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:09.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:09.024455+0000) 2024-03-06T10:31:10.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:10.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:10.024717+0000) 2024-03-06T10:31:11.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:11.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:11.024946+0000) 2024-03-06T10:31:12.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:12.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:12.025168+0000) 2024-03-06T10:31:13.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:13.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:13.025401+0000) 2024-03-06T10:31:14.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:14.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:14.025634+0000) 2024-03-06T10:31:15.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:15.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:15.025854+0000) 2024-03-06T10:31:16.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:16.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:16.026089+0000) 2024-03-06T10:31:17.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:17.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:17.026275+0000) 2024-03-06T10:31:18.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:18.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:18.026519+0000) 2024-03-06T10:31:19.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:19.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:19.026706+0000) 2024-03-06T10:31:20.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:20.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:20.026935+0000) 2024-03-06T10:31:21.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:21.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:21.027124+0000) 2024-03-06T10:31:22.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:22.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:22.027370+0000) 2024-03-06T10:31:23.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:23.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:23.027552+0000) 2024-03-06T10:31:24.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:24.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:24.027781+0000) 2024-03-06T10:31:25.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:25.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:25.028027+0000) 2024-03-06T10:31:26.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:26.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:26.028273+0000) 2024-03-06T10:31:26.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:26.506 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:27.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:27.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:27.028511+0000) 2024-03-06T10:31:28.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:28.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:28.028776+0000) 2024-03-06T10:31:29.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:29.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:29.029005+0000) 2024-03-06T10:31:30.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:30.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:30.029232+0000) 2024-03-06T10:31:31.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:31.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:31.029444+0000) 2024-03-06T10:31:32.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:32.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:32.029706+0000) 2024-03-06T10:31:33.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:33.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:33.030011+0000) 2024-03-06T10:31:34.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:34.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:34.030241+0000) 2024-03-06T10:31:35.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:35.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:35.030462+0000) 2024-03-06T10:31:36.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:36.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:36.030700+0000) 2024-03-06T10:31:37.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:37.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:37.030941+0000) 2024-03-06T10:31:38.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:38.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:38.031169+0000) 2024-03-06T10:31:39.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:39.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:39.031399+0000) 2024-03-06T10:31:40.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:40.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:40.031623+0000) 2024-03-06T10:31:41.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:41.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:41.031795+0000) 2024-03-06T10:31:42.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:42.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:42.031944+0000) 2024-03-06T10:31:43.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:43.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:43.032177+0000) 2024-03-06T10:31:44.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:44.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:44.032407+0000) 2024-03-06T10:31:45.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:45.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:45.032629+0000) 2024-03-06T10:31:46.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:46.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:46.032908+0000) 2024-03-06T10:31:47.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:47.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:47.033088+0000) 2024-03-06T10:31:48.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:48.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:48.033248+0000) 2024-03-06T10:31:49.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:49.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:49.033478+0000) 2024-03-06T10:31:50.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:50.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:50.033755+0000) 2024-03-06T10:31:51.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:51.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:51.033970+0000) 2024-03-06T10:31:52.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:52.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:52.034201+0000) 2024-03-06T10:31:53.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:53.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:53.034429+0000) 2024-03-06T10:31:54.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:54.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:54.034661+0000) 2024-03-06T10:31:55.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:55.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:55.034901+0000) 2024-03-06T10:31:56.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:56.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:56.035137+0000) 2024-03-06T10:31:56.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:56.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:57.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:57.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:57.035372+0000) 2024-03-06T10:31:58.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:58.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:58.035604+0000) 2024-03-06T10:31:59.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:31:59.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:59.035826+0000) 2024-03-06T10:32:00.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:00.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:00.036059+0000) 2024-03-06T10:32:01.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:01.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:01.036284+0000) 2024-03-06T10:32:02.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:02.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:02.036512+0000) 2024-03-06T10:32:03.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:03.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:03.036774+0000) 2024-03-06T10:32:04.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:04.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:04.036961+0000) 2024-03-06T10:32:05.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:05.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:05.037152+0000) 2024-03-06T10:32:06.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:06.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:06.037362+0000) 2024-03-06T10:32:07.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:07.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:07.037595+0000) 2024-03-06T10:32:08.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:08.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:08.037818+0000) 2024-03-06T10:32:09.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:09.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:09.038048+0000) 2024-03-06T10:32:10.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:10.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:10.038249+0000) 2024-03-06T10:32:11.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:11.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:11.038478+0000) 2024-03-06T10:32:12.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:12.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:12.038700+0000) 2024-03-06T10:32:13.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:13.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:13.038950+0000) 2024-03-06T10:32:14.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:14.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:14.039177+0000) 2024-03-06T10:32:15.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:15.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:15.039380+0000) 2024-03-06T10:32:16.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:16.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:16.039620+0000) 2024-03-06T10:32:17.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:17.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:17.039782+0000) 2024-03-06T10:32:18.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:18.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:18.039972+0000) 2024-03-06T10:32:19.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:19.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:19.040150+0000) 2024-03-06T10:32:20.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:20.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:20.040359+0000) 2024-03-06T10:32:21.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:21.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:21.040590+0000) 2024-03-06T10:32:22.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:22.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:22.040784+0000) 2024-03-06T10:32:23.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:23.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:23.041040+0000) 2024-03-06T10:32:24.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:24.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:24.041320+0000) 2024-03-06T10:32:25.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:25.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:25.041565+0000) 2024-03-06T10:32:26.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:26.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:26.041847+0000) 2024-03-06T10:32:26.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:26.874 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:27.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:27.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:27.042079+0000) 2024-03-06T10:32:28.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:28.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:28.042343+0000) 2024-03-06T10:32:29.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:29.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:29.042622+0000) 2024-03-06T10:32:30.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:30.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:30.042907+0000) 2024-03-06T10:32:31.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:31.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:31.043134+0000) 2024-03-06T10:32:32.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:32.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:32.043396+0000) 2024-03-06T10:32:33.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:33.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:33.043643+0000) 2024-03-06T10:32:34.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:34.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:34.043905+0000) 2024-03-06T10:32:35.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:35.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:35.044182+0000) 2024-03-06T10:32:36.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:36.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:36.044424+0000) 2024-03-06T10:32:37.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:37.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:37.044648+0000) 2024-03-06T10:32:38.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:38.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:38.044896+0000) 2024-03-06T10:32:39.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:39.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:39.045118+0000) 2024-03-06T10:32:40.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:40.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:40.045347+0000) 2024-03-06T10:32:41.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:41.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:41.045567+0000) 2024-03-06T10:32:42.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:42.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:42.045803+0000) 2024-03-06T10:32:43.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:43.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:43.045978+0000) 2024-03-06T10:32:44.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:44.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:44.046198+0000) 2024-03-06T10:32:45.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:45.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:45.046408+0000) 2024-03-06T10:32:46.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:46.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:46.046613+0000) 2024-03-06T10:32:47.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:47.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:47.046811+0000) 2024-03-06T10:32:48.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:48.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:48.047018+0000) 2024-03-06T10:32:49.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:49.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:49.047260+0000) 2024-03-06T10:32:50.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:50.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:50.047494+0000) 2024-03-06T10:32:51.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:51.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:51.047737+0000) 2024-03-06T10:32:52.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:52.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:52.047956+0000) 2024-03-06T10:32:53.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:53.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:53.048186+0000) 2024-03-06T10:32:54.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:54.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:54.048410+0000) 2024-03-06T10:32:55.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:55.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:55.048617+0000) 2024-03-06T10:32:56.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:56.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:56.048874+0000) 2024-03-06T10:32:57.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:57.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:57.049058+0000) 2024-03-06T10:32:57.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:57.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:58.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:58.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:58.049284+0000) 2024-03-06T10:32:59.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:32:59.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:59.049554+0000) 2024-03-06T10:33:00.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:00.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:00.049828+0000) 2024-03-06T10:33:01.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:01.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:01.050062+0000) 2024-03-06T10:33:02.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:02.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:02.050286+0000) 2024-03-06T10:33:03.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:03.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:03.050427+0000) 2024-03-06T10:33:04.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:04.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:04.050651+0000) 2024-03-06T10:33:05.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:05.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:05.050881+0000) 2024-03-06T10:33:06.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:06.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:06.051112+0000) 2024-03-06T10:33:07.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:07.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:07.051297+0000) 2024-03-06T10:33:08.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:08.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:08.051529+0000) 2024-03-06T10:33:09.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:09.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:09.051783+0000) 2024-03-06T10:33:10.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:10.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:10.051976+0000) 2024-03-06T10:33:11.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:11.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:11.052197+0000) 2024-03-06T10:33:12.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:12.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:12.052426+0000) 2024-03-06T10:33:13.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:13.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:13.052634+0000) 2024-03-06T10:33:14.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:14.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:14.052871+0000) 2024-03-06T10:33:15.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:15.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:15.053051+0000) 2024-03-06T10:33:16.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:16.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:16.053284+0000) 2024-03-06T10:33:17.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:17.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:17.053506+0000) 2024-03-06T10:33:18.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:18.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:18.053780+0000) 2024-03-06T10:33:19.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:19.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:19.054096+0000) 2024-03-06T10:33:20.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:20.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:20.054375+0000) 2024-03-06T10:33:21.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:21.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:21.054610+0000) 2024-03-06T10:33:22.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:22.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:22.054865+0000) 2024-03-06T10:33:23.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:23.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:23.055085+0000) 2024-03-06T10:33:24.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:24.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:24.055297+0000) 2024-03-06T10:33:25.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:25.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:25.055539+0000) 2024-03-06T10:33:26.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:26.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:26.055803+0000) 2024-03-06T10:33:27.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:27.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:27.056030+0000) 2024-03-06T10:33:27.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:27.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:28.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:28.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:28.056239+0000) 2024-03-06T10:33:29.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:29.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:29.056477+0000) 2024-03-06T10:33:30.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:30.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:30.056712+0000) 2024-03-06T10:33:31.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:31.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:31.056897+0000) 2024-03-06T10:33:32.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:32.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:32.057118+0000) 2024-03-06T10:33:33.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:33.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:33.057354+0000) 2024-03-06T10:33:34.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:34.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:34.057565+0000) 2024-03-06T10:33:35.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:35.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:35.057837+0000) 2024-03-06T10:33:36.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:36.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:36.058072+0000) 2024-03-06T10:33:37.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:37.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:37.058302+0000) 2024-03-06T10:33:38.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:38.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:38.058498+0000) 2024-03-06T10:33:39.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:39.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:39.058797+0000) 2024-03-06T10:33:40.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:40.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:40.058992+0000) 2024-03-06T10:33:41.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:41.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:41.059215+0000) 2024-03-06T10:33:42.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:42.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:42.059439+0000) 2024-03-06T10:33:43.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:43.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:43.059697+0000) 2024-03-06T10:33:44.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:44.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:44.059935+0000) 2024-03-06T10:33:45.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:45.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:45.060187+0000) 2024-03-06T10:33:46.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:46.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:46.060455+0000) 2024-03-06T10:33:47.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:47.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:47.060726+0000) 2024-03-06T10:33:48.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:48.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:48.061025+0000) 2024-03-06T10:33:49.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:49.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:49.061161+0000) 2024-03-06T10:33:50.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:50.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:50.061378+0000) 2024-03-06T10:33:51.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:51.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:51.061569+0000) 2024-03-06T10:33:52.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:52.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:52.061793+0000) 2024-03-06T10:33:53.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:53.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:53.061990+0000) 2024-03-06T10:33:54.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:54.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:54.062196+0000) 2024-03-06T10:33:55.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:55.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:55.062378+0000) 2024-03-06T10:33:56.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:56.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:56.062610+0000) 2024-03-06T10:33:57.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:57.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:57.062843+0000) 2024-03-06T10:33:57.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:57.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:58.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:58.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:58.063082+0000) 2024-03-06T10:33:59.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:33:59.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:59.063349+0000) 2024-03-06T10:34:00.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:00.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:00.063633+0000) 2024-03-06T10:34:01.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:01.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:01.063836+0000) 2024-03-06T10:34:02.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:02.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:02.064066+0000) 2024-03-06T10:34:03.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:03.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:03.064306+0000) 2024-03-06T10:34:04.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:04.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:04.064538+0000) 2024-03-06T10:34:05.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:05.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:05.064821+0000) 2024-03-06T10:34:06.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:06.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:06.065114+0000) 2024-03-06T10:34:07.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:07.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:07.065394+0000) 2024-03-06T10:34:08.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:08.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:08.065609+0000) 2024-03-06T10:34:09.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:09.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:09.065858+0000) 2024-03-06T10:34:10.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:10.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:10.066047+0000) 2024-03-06T10:34:11.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:11.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:11.066281+0000) 2024-03-06T10:34:12.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:12.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:12.066506+0000) 2024-03-06T10:34:13.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:13.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:13.066806+0000) 2024-03-06T10:34:14.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:14.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:14.067007+0000) 2024-03-06T10:34:15.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:15.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:15.067208+0000) 2024-03-06T10:34:16.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:16.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:16.067468+0000) 2024-03-06T10:34:17.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:17.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:17.067731+0000) 2024-03-06T10:34:18.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:18.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:18.067984+0000) 2024-03-06T10:34:19.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:19.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:19.068217+0000) 2024-03-06T10:34:20.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:20.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:20.068396+0000) 2024-03-06T10:34:21.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:21.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:21.068620+0000) 2024-03-06T10:34:22.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:22.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:22.068815+0000) 2024-03-06T10:34:23.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:23.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:23.069028+0000) 2024-03-06T10:34:24.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:24.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:24.069216+0000) 2024-03-06T10:34:25.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:25.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:25.069415+0000) 2024-03-06T10:34:26.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:26.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:26.069698+0000) 2024-03-06T10:34:27.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:27.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:27.069932+0000) 2024-03-06T10:34:27.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:27.691 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:28.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:28.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:28.070175+0000) 2024-03-06T10:34:29.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:29.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:29.070408+0000) 2024-03-06T10:34:30.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:30.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:30.070636+0000) 2024-03-06T10:34:31.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:31.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:31.070890+0000) 2024-03-06T10:34:32.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:32.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:32.071110+0000) 2024-03-06T10:34:33.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:33.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:33.071345+0000) 2024-03-06T10:34:34.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:34.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:34.071584+0000) 2024-03-06T10:34:35.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:35.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:35.071826+0000) 2024-03-06T10:34:36.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:36.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:36.072064+0000) 2024-03-06T10:34:37.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:37.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:37.072306+0000) 2024-03-06T10:34:38.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:38.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:38.072509+0000) 2024-03-06T10:34:39.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:39.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:39.072779+0000) 2024-03-06T10:34:40.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:40.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:40.073007+0000) 2024-03-06T10:34:41.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:41.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:41.073249+0000) 2024-03-06T10:34:42.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:42.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:42.073435+0000) 2024-03-06T10:34:43.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:43.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:43.073703+0000) 2024-03-06T10:34:44.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:44.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:44.073936+0000) 2024-03-06T10:34:45.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:45.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:45.074167+0000) 2024-03-06T10:34:46.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:46.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:46.074400+0000) 2024-03-06T10:34:47.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:47.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:47.074702+0000) 2024-03-06T10:34:48.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:48.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:48.074947+0000) 2024-03-06T10:34:49.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:49.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:49.075185+0000) 2024-03-06T10:34:50.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:50.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:50.075404+0000) 2024-03-06T10:34:51.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:51.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:51.075635+0000) 2024-03-06T10:34:52.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:52.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:52.075833+0000) 2024-03-06T10:34:53.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:53.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:53.076091+0000) 2024-03-06T10:34:54.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:54.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:54.076310+0000) 2024-03-06T10:34:55.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:55.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:55.076484+0000) 2024-03-06T10:34:56.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:56.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:56.076744+0000) 2024-03-06T10:34:57.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:57.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:57.076945+0000) 2024-03-06T10:34:57.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:57.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:58.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:58.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:58.077156+0000) 2024-03-06T10:34:59.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:34:59.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:59.077394+0000) 2024-03-06T10:35:00.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:00.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:00.077661+0000) 2024-03-06T10:35:01.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:01.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:01.078037+0000) 2024-03-06T10:35:02.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:02.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:02.078258+0000) 2024-03-06T10:35:03.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:03.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:03.078522+0000) 2024-03-06T10:35:04.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:04.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:04.078746+0000) 2024-03-06T10:35:05.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:05.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:05.078977+0000) 2024-03-06T10:35:06.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:06.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:06.079270+0000) 2024-03-06T10:35:07.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:07.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:07.079497+0000) 2024-03-06T10:35:08.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:08.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:08.079795+0000) 2024-03-06T10:35:09.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:09.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:09.079995+0000) 2024-03-06T10:35:10.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:10.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:10.080183+0000) 2024-03-06T10:35:11.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:11.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:11.080376+0000) 2024-03-06T10:35:12.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:12.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:12.080605+0000) 2024-03-06T10:35:13.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:13.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:13.080789+0000) 2024-03-06T10:35:14.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:14.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:14.081053+0000) 2024-03-06T10:35:15.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:15.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:15.081261+0000) 2024-03-06T10:35:16.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:16.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:16.081465+0000) 2024-03-06T10:35:17.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:17.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:17.081661+0000) 2024-03-06T10:35:18.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:18.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:18.081908+0000) 2024-03-06T10:35:19.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:19.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:19.082128+0000) 2024-03-06T10:35:20.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:20.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:20.082370+0000) 2024-03-06T10:35:21.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:21.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:21.082642+0000) 2024-03-06T10:35:22.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:22.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:22.082928+0000) 2024-03-06T10:35:23.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:23.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:23.083163+0000) 2024-03-06T10:35:24.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:24.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:24.083319+0000) 2024-03-06T10:35:25.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:25.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:25.083482+0000) 2024-03-06T10:35:26.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:26.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:26.083696+0000) 2024-03-06T10:35:27.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:27.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:27.083934+0000) 2024-03-06T10:35:28.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:28.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:28.084140+0000) 2024-03-06T10:35:28.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:28.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:29.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:29.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:29.084322+0000) 2024-03-06T10:35:30.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:30.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:30.084570+0000) 2024-03-06T10:35:31.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:31.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:31.084778+0000) 2024-03-06T10:35:32.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:32.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:32.085001+0000) 2024-03-06T10:35:33.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:33.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:33.085246+0000) 2024-03-06T10:35:34.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:34.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:34.085477+0000) 2024-03-06T10:35:35.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:35.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:35.085746+0000) 2024-03-06T10:35:36.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:36.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:36.085974+0000) 2024-03-06T10:35:37.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:37.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:37.086177+0000) 2024-03-06T10:35:38.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:38.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:38.086421+0000) 2024-03-06T10:35:39.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:39.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:39.086708+0000) 2024-03-06T10:35:40.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:40.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:40.086957+0000) 2024-03-06T10:35:41.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:41.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:41.087145+0000) 2024-03-06T10:35:42.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:42.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:42.087370+0000) 2024-03-06T10:35:43.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:43.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:43.087628+0000) 2024-03-06T10:35:44.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:44.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:44.087926+0000) 2024-03-06T10:35:45.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:45.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:45.088083+0000) 2024-03-06T10:35:46.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:46.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:46.088272+0000) 2024-03-06T10:35:47.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:47.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:47.088470+0000) 2024-03-06T10:35:48.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:48.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:48.088706+0000) 2024-03-06T10:35:49.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:49.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:49.088987+0000) 2024-03-06T10:35:50.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:50.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:50.089169+0000) 2024-03-06T10:35:51.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:51.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:51.089432+0000) 2024-03-06T10:35:52.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:52.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:52.089634+0000) 2024-03-06T10:35:53.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:53.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:53.089871+0000) 2024-03-06T10:35:54.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:54.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:54.090089+0000) 2024-03-06T10:35:55.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:55.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:55.090307+0000) 2024-03-06T10:35:56.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:56.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:56.090498+0000) 2024-03-06T10:35:57.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:57.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:57.090811+0000) 2024-03-06T10:35:58.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:58.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:58.091100+0000) 2024-03-06T10:35:58.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:58.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:59.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:35:59.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:59.091378+0000) 2024-03-06T10:36:00.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:00.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:00.091659+0000) 2024-03-06T10:36:01.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:01.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:01.091936+0000) 2024-03-06T10:36:02.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:02.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:02.092144+0000) 2024-03-06T10:36:03.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:03.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:03.092373+0000) 2024-03-06T10:36:04.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:04.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:04.092602+0000) 2024-03-06T10:36:05.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:05.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:05.092834+0000) 2024-03-06T10:36:06.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:06.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:06.093029+0000) 2024-03-06T10:36:07.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:07.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:07.093268+0000) 2024-03-06T10:36:08.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:08.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:08.093492+0000) 2024-03-06T10:36:09.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:09.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:09.093760+0000) 2024-03-06T10:36:10.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:10.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:10.093999+0000) 2024-03-06T10:36:11.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:11.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:11.094175+0000) 2024-03-06T10:36:12.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:12.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:12.094414+0000) 2024-03-06T10:36:13.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:13.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:13.094635+0000) 2024-03-06T10:36:14.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:14.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:14.094807+0000) 2024-03-06T10:36:15.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:15.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:15.094992+0000) 2024-03-06T10:36:16.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:16.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:16.095239+0000) 2024-03-06T10:36:17.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:17.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:17.095460+0000) 2024-03-06T10:36:18.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:18.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:18.095649+0000) 2024-03-06T10:36:19.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:19.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:19.095912+0000) 2024-03-06T10:36:20.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:20.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:20.096136+0000) 2024-03-06T10:36:21.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:21.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:21.096336+0000) 2024-03-06T10:36:22.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:22.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:22.096521+0000) 2024-03-06T10:36:23.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:23.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:23.096775+0000) 2024-03-06T10:36:24.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:24.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:24.097000+0000) 2024-03-06T10:36:25.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:25.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:25.097190+0000) 2024-03-06T10:36:26.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:26.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:26.097430+0000) 2024-03-06T10:36:27.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:27.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:27.097657+0000) 2024-03-06T10:36:28.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:28.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:28.097913+0000) 2024-03-06T10:36:28.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:28.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:29.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:29.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:29.098152+0000) 2024-03-06T10:36:30.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:30.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:30.098374+0000) 2024-03-06T10:36:31.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:31.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:31.098595+0000) 2024-03-06T10:36:32.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:32.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:32.098825+0000) 2024-03-06T10:36:33.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:33.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:33.099009+0000) 2024-03-06T10:36:34.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:34.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:34.099196+0000) 2024-03-06T10:36:35.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:35.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:35.099414+0000) 2024-03-06T10:36:36.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:36.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:36.099643+0000) 2024-03-06T10:36:37.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:37.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:37.099898+0000) 2024-03-06T10:36:38.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:38.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:38.100114+0000) 2024-03-06T10:36:39.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:39.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:39.100339+0000) 2024-03-06T10:36:40.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:40.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:40.100557+0000) 2024-03-06T10:36:41.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:41.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:41.100759+0000) 2024-03-06T10:36:42.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:42.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:42.100951+0000) 2024-03-06T10:36:43.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:43.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:43.101184+0000) 2024-03-06T10:36:44.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:44.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:44.101418+0000) 2024-03-06T10:36:45.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:45.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:45.101640+0000) 2024-03-06T10:36:46.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:46.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:46.101859+0000) 2024-03-06T10:36:47.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:47.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:47.102096+0000) 2024-03-06T10:36:48.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:48.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:48.102322+0000) 2024-03-06T10:36:49.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:49.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:49.102549+0000) 2024-03-06T10:36:50.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:50.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:50.102780+0000) 2024-03-06T10:36:51.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:51.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:51.103003+0000) 2024-03-06T10:36:52.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:52.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:52.103227+0000) 2024-03-06T10:36:53.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:53.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:53.103457+0000) 2024-03-06T10:36:54.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:54.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:54.103714+0000) 2024-03-06T10:36:55.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:55.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:55.103938+0000) 2024-03-06T10:36:56.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:56.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:56.104127+0000) 2024-03-06T10:36:57.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:57.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:57.104353+0000) 2024-03-06T10:36:58.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:58.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:58.104609+0000) 2024-03-06T10:36:58.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:58.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:59.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:36:59.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:59.104852+0000) 2024-03-06T10:37:00.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:00.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:00.105074+0000) 2024-03-06T10:37:01.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:01.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:01.105301+0000) 2024-03-06T10:37:02.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:02.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:02.105529+0000) 2024-03-06T10:37:03.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:03.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:03.105793+0000) 2024-03-06T10:37:04.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:04.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:04.105989+0000) 2024-03-06T10:37:05.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:05.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:05.106228+0000) 2024-03-06T10:37:06.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:06.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:06.106417+0000) 2024-03-06T10:37:07.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:07.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:07.106607+0000) 2024-03-06T10:37:08.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:08.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:08.106839+0000) 2024-03-06T10:37:09.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:09.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:09.107023+0000) 2024-03-06T10:37:10.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:10.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:10.107248+0000) 2024-03-06T10:37:11.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:11.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:11.107466+0000) 2024-03-06T10:37:12.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:12.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:12.107714+0000) 2024-03-06T10:37:13.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:13.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:13.107944+0000) 2024-03-06T10:37:14.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:14.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:14.108182+0000) 2024-03-06T10:37:15.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:15.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:15.108409+0000) 2024-03-06T10:37:16.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:16.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:16.108640+0000) 2024-03-06T10:37:17.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:17.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:17.108860+0000) 2024-03-06T10:37:18.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:18.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:18.109083+0000) 2024-03-06T10:37:19.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:19.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:19.109263+0000) 2024-03-06T10:37:20.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:20.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:20.109483+0000) 2024-03-06T10:37:21.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:21.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:21.109743+0000) 2024-03-06T10:37:22.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:22.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:22.110019+0000) 2024-03-06T10:37:23.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:23.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:23.110223+0000) 2024-03-06T10:37:24.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:24.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:24.110438+0000) 2024-03-06T10:37:25.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:25.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:25.110628+0000) 2024-03-06T10:37:26.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:26.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:26.110877+0000) 2024-03-06T10:37:27.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:27.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:27.111113+0000) 2024-03-06T10:37:28.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:28.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:28.111311+0000) 2024-03-06T10:37:28.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:28.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:29.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:29.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:29.111517+0000) 2024-03-06T10:37:30.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:30.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:30.111790+0000) 2024-03-06T10:37:31.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:31.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:31.112004+0000) 2024-03-06T10:37:32.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:32.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:32.112180+0000) 2024-03-06T10:37:33.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:33.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:33.112463+0000) 2024-03-06T10:37:34.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:34.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:34.112705+0000) 2024-03-06T10:37:35.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:35.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:35.112975+0000) 2024-03-06T10:37:36.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:36.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:36.113212+0000) 2024-03-06T10:37:37.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:37.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:37.113461+0000) 2024-03-06T10:37:38.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:38.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:38.113863+0000) 2024-03-06T10:37:39.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:39.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:39.114078+0000) 2024-03-06T10:37:40.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:40.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:40.114297+0000) 2024-03-06T10:37:41.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:41.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:41.114463+0000) 2024-03-06T10:37:42.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:42.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:42.114688+0000) 2024-03-06T10:37:43.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:43.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:43.114881+0000) 2024-03-06T10:37:44.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:44.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:44.115072+0000) 2024-03-06T10:37:45.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:45.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:45.115302+0000) 2024-03-06T10:37:46.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:46.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:46.115550+0000) 2024-03-06T10:37:47.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:47.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:47.115746+0000) 2024-03-06T10:37:48.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:48.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:48.115933+0000) 2024-03-06T10:37:49.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:49.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:49.116128+0000) 2024-03-06T10:37:50.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:50.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:50.116305+0000) 2024-03-06T10:37:51.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:51.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:51.116522+0000) 2024-03-06T10:37:52.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:52.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:52.116822+0000) 2024-03-06T10:37:53.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:53.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:53.117044+0000) 2024-03-06T10:37:54.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:54.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:54.117286+0000) 2024-03-06T10:37:55.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:55.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:55.117508+0000) 2024-03-06T10:37:56.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:56.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:56.117770+0000) 2024-03-06T10:37:57.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:57.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:57.117992+0000) 2024-03-06T10:37:58.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:58.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:58.118175+0000) 2024-03-06T10:37:59.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:59.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:59.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:37:59.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:59.118409+0000) 2024-03-06T10:38:00.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:00.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:00.118636+0000) 2024-03-06T10:38:01.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:01.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:01.118845+0000) 2024-03-06T10:38:02.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:02.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:02.119083+0000) 2024-03-06T10:38:03.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:03.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:03.119308+0000) 2024-03-06T10:38:04.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:04.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:04.119509+0000) 2024-03-06T10:38:05.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:05.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:05.119766+0000) 2024-03-06T10:38:06.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:06.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:06.119995+0000) 2024-03-06T10:38:07.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:07.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:07.120254+0000) 2024-03-06T10:38:08.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:08.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:08.120534+0000) 2024-03-06T10:38:09.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:09.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:09.120809+0000) 2024-03-06T10:38:10.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:10.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:10.121094+0000) 2024-03-06T10:38:11.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:11.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:11.121367+0000) 2024-03-06T10:38:12.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:12.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:12.121571+0000) 2024-03-06T10:38:13.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:13.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:13.121794+0000) 2024-03-06T10:38:14.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:14.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:14.122014+0000) 2024-03-06T10:38:15.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:15.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:15.122295+0000) 2024-03-06T10:38:16.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:16.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:16.122584+0000) 2024-03-06T10:38:17.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:17.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:17.122826+0000) 2024-03-06T10:38:18.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:18.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:18.123054+0000) 2024-03-06T10:38:19.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:19.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:19.123244+0000) 2024-03-06T10:38:20.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:20.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:20.123465+0000) 2024-03-06T10:38:21.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:21.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:21.123722+0000) 2024-03-06T10:38:22.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:22.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:22.123976+0000) 2024-03-06T10:38:23.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:23.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:23.124215+0000) 2024-03-06T10:38:24.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:24.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:24.124457+0000) 2024-03-06T10:38:25.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:25.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:25.124740+0000) 2024-03-06T10:38:26.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:26.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:26.124990+0000) 2024-03-06T10:38:27.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:27.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:27.125182+0000) 2024-03-06T10:38:28.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:28.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:28.125405+0000) 2024-03-06T10:38:29.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:29.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:29.125711+0000) 2024-03-06T10:38:29.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:29.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:30.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:30.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:30.125955+0000) 2024-03-06T10:38:31.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:31.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:31.126225+0000) 2024-03-06T10:38:32.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:32.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:32.126491+0000) 2024-03-06T10:38:33.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:33.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:33.126794+0000) 2024-03-06T10:38:34.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:34.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:34.127001+0000) 2024-03-06T10:38:35.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:35.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:35.127196+0000) 2024-03-06T10:38:36.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:36.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:36.127412+0000) 2024-03-06T10:38:37.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:37.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:37.127655+0000) 2024-03-06T10:38:38.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:38.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:38.127922+0000) 2024-03-06T10:38:39.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:39.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:39.128130+0000) 2024-03-06T10:38:40.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:40.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:40.128375+0000) 2024-03-06T10:38:41.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:41.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:41.128565+0000) 2024-03-06T10:38:42.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:42.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:42.128792+0000) 2024-03-06T10:38:43.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:43.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:43.128997+0000) 2024-03-06T10:38:44.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:44.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:44.129220+0000) 2024-03-06T10:38:45.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:45.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:45.129414+0000) 2024-03-06T10:38:46.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:46.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:46.129622+0000) 2024-03-06T10:38:47.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:47.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:47.129807+0000) 2024-03-06T10:38:48.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:48.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:48.129936+0000) 2024-03-06T10:38:49.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:49.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:49.130107+0000) 2024-03-06T10:38:50.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:50.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:50.130298+0000) 2024-03-06T10:38:51.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:51.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:51.130523+0000) 2024-03-06T10:38:52.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:52.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:52.130796+0000) 2024-03-06T10:38:53.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:53.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:53.131029+0000) 2024-03-06T10:38:54.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:54.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:54.131209+0000) 2024-03-06T10:38:55.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:55.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:55.131477+0000) 2024-03-06T10:38:56.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:56.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:56.131774+0000) 2024-03-06T10:38:57.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:57.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:57.131992+0000) 2024-03-06T10:38:58.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:58.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:58.132247+0000) 2024-03-06T10:38:59.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:38:59.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:59.132531+0000) 2024-03-06T10:38:59.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:59.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:00.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:00.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:00.132815+0000) 2024-03-06T10:39:01.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:01.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:01.133045+0000) 2024-03-06T10:39:02.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:02.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:02.133308+0000) 2024-03-06T10:39:03.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:03.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:03.133546+0000) 2024-03-06T10:39:04.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:04.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:04.133789+0000) 2024-03-06T10:39:05.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:05.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:05.134015+0000) 2024-03-06T10:39:06.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:06.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:06.134248+0000) 2024-03-06T10:39:07.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:07.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:07.134471+0000) 2024-03-06T10:39:08.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:08.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:08.134730+0000) 2024-03-06T10:39:09.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:09.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:09.134958+0000) 2024-03-06T10:39:10.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:10.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:10.135198+0000) 2024-03-06T10:39:11.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:11.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:11.135440+0000) 2024-03-06T10:39:12.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:12.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:12.135644+0000) 2024-03-06T10:39:13.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:13.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:13.135900+0000) 2024-03-06T10:39:14.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:14.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:14.136126+0000) 2024-03-06T10:39:15.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:15.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:15.136337+0000) 2024-03-06T10:39:16.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:16.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:16.136541+0000) 2024-03-06T10:39:17.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:17.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:17.136818+0000) 2024-03-06T10:39:18.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:18.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:18.137035+0000) 2024-03-06T10:39:19.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:19.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:19.137263+0000) 2024-03-06T10:39:20.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:20.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:20.137523+0000) 2024-03-06T10:39:21.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:21.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:21.137775+0000) 2024-03-06T10:39:22.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:22.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:22.138046+0000) 2024-03-06T10:39:23.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:23.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:23.138331+0000) 2024-03-06T10:39:24.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:24.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:24.138576+0000) 2024-03-06T10:39:25.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:25.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:25.138787+0000) 2024-03-06T10:39:26.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:26.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:26.139023+0000) 2024-03-06T10:39:27.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:27.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:27.139276+0000) 2024-03-06T10:39:28.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:28.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:28.139483+0000) 2024-03-06T10:39:29.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:29.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:29.139766+0000) 2024-03-06T10:39:29.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:29.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:30.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:30.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:30.140010+0000) 2024-03-06T10:39:31.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:31.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:31.140286+0000) 2024-03-06T10:39:32.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:32.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:32.140491+0000) 2024-03-06T10:39:33.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:33.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:33.140713+0000) 2024-03-06T10:39:34.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:34.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:34.140963+0000) 2024-03-06T10:39:35.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:35.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:35.141168+0000) 2024-03-06T10:39:36.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:36.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:36.141426+0000) 2024-03-06T10:39:37.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:37.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:37.141700+0000) 2024-03-06T10:39:38.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:38.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:38.141968+0000) 2024-03-06T10:39:39.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:39.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:39.142174+0000) 2024-03-06T10:39:40.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:40.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:40.142403+0000) 2024-03-06T10:39:41.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:41.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:41.142600+0000) 2024-03-06T10:39:42.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:42.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:42.142847+0000) 2024-03-06T10:39:43.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:43.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:43.143085+0000) 2024-03-06T10:39:44.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:44.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:44.143356+0000) 2024-03-06T10:39:45.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:45.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:45.143637+0000) 2024-03-06T10:39:46.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:46.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:46.143973+0000) 2024-03-06T10:39:47.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:47.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:47.144181+0000) 2024-03-06T10:39:48.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:48.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:48.144414+0000) 2024-03-06T10:39:49.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:49.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:49.144724+0000) 2024-03-06T10:39:50.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:50.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:50.144933+0000) 2024-03-06T10:39:51.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:51.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:51.145169+0000) 2024-03-06T10:39:52.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:52.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:52.145403+0000) 2024-03-06T10:39:53.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:53.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:53.145625+0000) 2024-03-06T10:39:54.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:54.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:54.145883+0000) 2024-03-06T10:39:55.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:55.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:55.146112+0000) 2024-03-06T10:39:56.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:56.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:56.146312+0000) 2024-03-06T10:39:57.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:57.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:57.146544+0000) 2024-03-06T10:39:58.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:58.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:58.146791+0000) 2024-03-06T10:39:59.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:39:59.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:59.147034+0000) 2024-03-06T10:39:59.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:59.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:00.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:00.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:00.147211+0000) 2024-03-06T10:40:01.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:01.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:01.147442+0000) 2024-03-06T10:40:02.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:02.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:02.147625+0000) 2024-03-06T10:40:03.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:03.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:03.147817+0000) 2024-03-06T10:40:04.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:04.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:04.147994+0000) 2024-03-06T10:40:05.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:05.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:05.148187+0000) 2024-03-06T10:40:06.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:06.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:06.148462+0000) 2024-03-06T10:40:07.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:07.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:07.148715+0000) 2024-03-06T10:40:08.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:08.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:08.148959+0000) 2024-03-06T10:40:09.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:09.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:09.149152+0000) 2024-03-06T10:40:10.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:10.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:10.149384+0000) 2024-03-06T10:40:11.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:11.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:11.149560+0000) 2024-03-06T10:40:12.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:12.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:12.149820+0000) 2024-03-06T10:40:13.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:13.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:13.150090+0000) 2024-03-06T10:40:14.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:14.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:14.150340+0000) 2024-03-06T10:40:15.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:15.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:15.150582+0000) 2024-03-06T10:40:16.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:16.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:16.150829+0000) 2024-03-06T10:40:17.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:17.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:17.151071+0000) 2024-03-06T10:40:18.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:18.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:18.151291+0000) 2024-03-06T10:40:19.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:19.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:19.151558+0000) 2024-03-06T10:40:20.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:20.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:20.151804+0000) 2024-03-06T10:40:21.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:21.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:21.151993+0000) 2024-03-06T10:40:22.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:22.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:22.152235+0000) 2024-03-06T10:40:23.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:23.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:23.152480+0000) 2024-03-06T10:40:24.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:24.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:24.152782+0000) 2024-03-06T10:40:25.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:25.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:25.152995+0000) 2024-03-06T10:40:26.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:26.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:26.153216+0000) 2024-03-06T10:40:27.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:27.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:27.153476+0000) 2024-03-06T10:40:28.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:28.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:28.153727+0000) 2024-03-06T10:40:29.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:29.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:29.153974+0000) 2024-03-06T10:40:30.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:30.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:30.154204+0000) 2024-03-06T10:40:30.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:30.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:31.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:31.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:31.154396+0000) 2024-03-06T10:40:32.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:32.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:32.154707+0000) 2024-03-06T10:40:33.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:33.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:33.154907+0000) 2024-03-06T10:40:34.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:34.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:34.155174+0000) 2024-03-06T10:40:35.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:35.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:35.155414+0000) 2024-03-06T10:40:36.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:36.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:36.155700+0000) 2024-03-06T10:40:37.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:37.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:37.155942+0000) 2024-03-06T10:40:38.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:38.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:38.156168+0000) 2024-03-06T10:40:39.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:39.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:39.156403+0000) 2024-03-06T10:40:40.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:40.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:40.156640+0000) 2024-03-06T10:40:41.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:41.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:41.156970+0000) 2024-03-06T10:40:42.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:42.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:42.157246+0000) 2024-03-06T10:40:43.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:43.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:43.157509+0000) 2024-03-06T10:40:44.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:44.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:44.157836+0000) 2024-03-06T10:40:45.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:45.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:45.158112+0000) 2024-03-06T10:40:46.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:46.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:46.158399+0000) 2024-03-06T10:40:47.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:47.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:47.158720+0000) 2024-03-06T10:40:48.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:48.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:48.158995+0000) 2024-03-06T10:40:49.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:49.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:49.159216+0000) 2024-03-06T10:40:50.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:50.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:50.159427+0000) 2024-03-06T10:40:51.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:51.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:51.159614+0000) 2024-03-06T10:40:52.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:52.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:52.159837+0000) 2024-03-06T10:40:53.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:53.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:53.160080+0000) 2024-03-06T10:40:54.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:54.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:54.160342+0000) 2024-03-06T10:40:55.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:55.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:55.160573+0000) 2024-03-06T10:40:56.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:56.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:56.160779+0000) 2024-03-06T10:40:57.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:57.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:57.160989+0000) 2024-03-06T10:40:58.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:58.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:58.161217+0000) 2024-03-06T10:40:59.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:40:59.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:59.161488+0000) 2024-03-06T10:41:00.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:00.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:00.161797+0000) 2024-03-06T10:41:00.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:00.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:01.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:01.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:01.161987+0000) 2024-03-06T10:41:02.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:02.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:02.162258+0000) 2024-03-06T10:41:03.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:03.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:03.162525+0000) 2024-03-06T10:41:04.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:04.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:04.162843+0000) 2024-03-06T10:41:05.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:05.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:05.163042+0000) 2024-03-06T10:41:06.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:06.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:06.163280+0000) 2024-03-06T10:41:07.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:07.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:07.163502+0000) 2024-03-06T10:41:08.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:08.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:08.163767+0000) 2024-03-06T10:41:09.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:09.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:09.164015+0000) 2024-03-06T10:41:10.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:10.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:10.164209+0000) 2024-03-06T10:41:11.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:11.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:11.164429+0000) 2024-03-06T10:41:12.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:12.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:12.164624+0000) 2024-03-06T10:41:13.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:13.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:13.164882+0000) 2024-03-06T10:41:14.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:14.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:14.165155+0000) 2024-03-06T10:41:15.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:15.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:15.165393+0000) 2024-03-06T10:41:16.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:16.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:16.165582+0000) 2024-03-06T10:41:17.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:17.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:17.165797+0000) 2024-03-06T10:41:18.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:18.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:18.165990+0000) 2024-03-06T10:41:19.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:19.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:19.166231+0000) 2024-03-06T10:41:20.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:20.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:20.166461+0000) 2024-03-06T10:41:21.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:21.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:21.166637+0000) 2024-03-06T10:41:22.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:22.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:22.166902+0000) 2024-03-06T10:41:23.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:23.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:23.167120+0000) 2024-03-06T10:41:24.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:24.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:24.167351+0000) 2024-03-06T10:41:25.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:25.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:25.167585+0000) 2024-03-06T10:41:26.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:26.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:26.167815+0000) 2024-03-06T10:41:27.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:27.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:27.168039+0000) 2024-03-06T10:41:28.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:28.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:28.168269+0000) 2024-03-06T10:41:29.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:29.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:29.168500+0000) 2024-03-06T10:41:30.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:30.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:30.168791+0000) 2024-03-06T10:41:30.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:30.532 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:31.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:31.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:31.169021+0000) 2024-03-06T10:41:32.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:32.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:32.169250+0000) 2024-03-06T10:41:33.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:33.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:33.169469+0000) 2024-03-06T10:41:34.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:34.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:34.169735+0000) 2024-03-06T10:41:35.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:35.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:35.169972+0000) 2024-03-06T10:41:36.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:36.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:36.170222+0000) 2024-03-06T10:41:37.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:37.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:37.170463+0000) 2024-03-06T10:41:38.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:38.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:38.170752+0000) 2024-03-06T10:41:39.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:39.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:39.171002+0000) 2024-03-06T10:41:40.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:40.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:40.171248+0000) 2024-03-06T10:41:41.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:41.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:41.171485+0000) 2024-03-06T10:41:42.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:42.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:42.171772+0000) 2024-03-06T10:41:43.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:43.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:43.171997+0000) 2024-03-06T10:41:44.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:44.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:44.172191+0000) 2024-03-06T10:41:45.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:45.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:45.172437+0000) 2024-03-06T10:41:46.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:46.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:46.172724+0000) 2024-03-06T10:41:47.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:47.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:47.172973+0000) 2024-03-06T10:41:48.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:48.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:48.173222+0000) 2024-03-06T10:41:49.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:49.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:49.173463+0000) 2024-03-06T10:41:50.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:50.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:50.173745+0000) 2024-03-06T10:41:51.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:51.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:51.174053+0000) 2024-03-06T10:41:52.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:52.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:52.174252+0000) 2024-03-06T10:41:53.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:53.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:53.174487+0000) 2024-03-06T10:41:54.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:54.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:54.174651+0000) 2024-03-06T10:41:55.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:55.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:55.174914+0000) 2024-03-06T10:41:56.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:56.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:56.175145+0000) 2024-03-06T10:41:57.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:57.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:57.175335+0000) 2024-03-06T10:41:58.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:58.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:58.175548+0000) 2024-03-06T10:41:59.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:41:59.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:59.175799+0000) 2024-03-06T10:42:00.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:00.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:00.176029+0000) 2024-03-06T10:42:00.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:00.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:01.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:01.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:01.176266+0000) 2024-03-06T10:42:02.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:02.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:02.176501+0000) 2024-03-06T10:42:03.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:03.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:03.176788+0000) 2024-03-06T10:42:04.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:04.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:04.176997+0000) 2024-03-06T10:42:05.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:05.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:05.177222+0000) 2024-03-06T10:42:06.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:06.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:06.177458+0000) 2024-03-06T10:42:07.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:07.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:07.177720+0000) 2024-03-06T10:42:08.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:08.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:08.177955+0000) 2024-03-06T10:42:09.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:09.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:09.178189+0000) 2024-03-06T10:42:10.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:10.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:10.178417+0000) 2024-03-06T10:42:11.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:11.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:11.178591+0000) 2024-03-06T10:42:12.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:12.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:12.178789+0000) 2024-03-06T10:42:13.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:13.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:13.178967+0000) 2024-03-06T10:42:14.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:14.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:14.179157+0000) 2024-03-06T10:42:15.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:15.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:15.179387+0000) 2024-03-06T10:42:16.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:16.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:16.179583+0000) 2024-03-06T10:42:17.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:17.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:17.179854+0000) 2024-03-06T10:42:18.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:18.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:18.180136+0000) 2024-03-06T10:42:19.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:19.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:19.180420+0000) 2024-03-06T10:42:20.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:20.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:20.180620+0000) 2024-03-06T10:42:21.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:21.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:21.180822+0000) 2024-03-06T10:42:22.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:22.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:22.181063+0000) 2024-03-06T10:42:23.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:23.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:23.181309+0000) 2024-03-06T10:42:24.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:24.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:24.181553+0000) 2024-03-06T10:42:25.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:25.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:25.181806+0000) 2024-03-06T10:42:26.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:26.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:26.182058+0000) 2024-03-06T10:42:27.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:27.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:27.182327+0000) 2024-03-06T10:42:28.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:28.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:28.182598+0000) 2024-03-06T10:42:29.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:29.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:29.182872+0000) 2024-03-06T10:42:30.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:30.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:30.183084+0000) 2024-03-06T10:42:30.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:30.894 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:31.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:31.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:31.183318+0000) 2024-03-06T10:42:32.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:32.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:32.183543+0000) 2024-03-06T10:42:33.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:33.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:33.183791+0000) 2024-03-06T10:42:34.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:34.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:34.183985+0000) 2024-03-06T10:42:35.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:35.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:35.184212+0000) 2024-03-06T10:42:36.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:36.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:36.184444+0000) 2024-03-06T10:42:37.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:37.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:37.184696+0000) 2024-03-06T10:42:38.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:38.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:38.184928+0000) 2024-03-06T10:42:39.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:39.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:39.185154+0000) 2024-03-06T10:42:40.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:40.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:40.185385+0000) 2024-03-06T10:42:41.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:41.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:41.185609+0000) 2024-03-06T10:42:42.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:42.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:42.185804+0000) 2024-03-06T10:42:43.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:43.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:43.186027+0000) 2024-03-06T10:42:44.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:44.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:44.186247+0000) 2024-03-06T10:42:45.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:45.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:45.186474+0000) 2024-03-06T10:42:46.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:46.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:46.186726+0000) 2024-03-06T10:42:47.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:47.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:47.186925+0000) 2024-03-06T10:42:48.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:48.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:48.187125+0000) 2024-03-06T10:42:49.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:49.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:49.187366+0000) 2024-03-06T10:42:50.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:50.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:50.187656+0000) 2024-03-06T10:42:51.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:51.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:51.187922+0000) 2024-03-06T10:42:52.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:52.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:52.188135+0000) 2024-03-06T10:42:53.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:53.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:53.188374+0000) 2024-03-06T10:42:54.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:54.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:54.188556+0000) 2024-03-06T10:42:55.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:55.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:55.188826+0000) 2024-03-06T10:42:56.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:56.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:56.189038+0000) 2024-03-06T10:42:57.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:57.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:57.189232+0000) 2024-03-06T10:42:58.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:58.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:58.189463+0000) 2024-03-06T10:42:59.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:42:59.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:59.189728+0000) 2024-03-06T10:43:00.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:00.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:00.189944+0000) 2024-03-06T10:43:01.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:01.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:01.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:01.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:01.190119+0000) 2024-03-06T10:43:02.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:02.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:02.190339+0000) 2024-03-06T10:43:03.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:03.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:03.190555+0000) 2024-03-06T10:43:04.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:04.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:04.190803+0000) 2024-03-06T10:43:05.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:05.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:05.191069+0000) 2024-03-06T10:43:06.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:06.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:06.191382+0000) 2024-03-06T10:43:07.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:07.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:07.191661+0000) 2024-03-06T10:43:08.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:08.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:08.191978+0000) 2024-03-06T10:43:09.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:09.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:09.192256+0000) 2024-03-06T10:43:10.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:10.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:10.192538+0000) 2024-03-06T10:43:11.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:11.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:11.192814+0000) 2024-03-06T10:43:12.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:12.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:12.193095+0000) 2024-03-06T10:43:13.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:13.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:13.193305+0000) 2024-03-06T10:43:14.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:14.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:14.193572+0000) 2024-03-06T10:43:15.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:15.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:15.193813+0000) 2024-03-06T10:43:16.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:16.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:16.194044+0000) 2024-03-06T10:43:17.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:17.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:17.194283+0000) 2024-03-06T10:43:18.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:18.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:18.194555+0000) 2024-03-06T10:43:19.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:19.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:19.194826+0000) 2024-03-06T10:43:20.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:20.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:20.195030+0000) 2024-03-06T10:43:21.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:21.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:21.195225+0000) 2024-03-06T10:43:22.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:22.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:22.195457+0000) 2024-03-06T10:43:23.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:23.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:23.195657+0000) 2024-03-06T10:43:24.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:24.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:24.195899+0000) 2024-03-06T10:43:25.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:25.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:25.196136+0000) 2024-03-06T10:43:26.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:26.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:26.196354+0000) 2024-03-06T10:43:27.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:27.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:27.196589+0000) 2024-03-06T10:43:28.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:28.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:28.196812+0000) 2024-03-06T10:43:29.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:29.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:29.197041+0000) 2024-03-06T10:43:30.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:30.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:30.197272+0000) 2024-03-06T10:43:31.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:31.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:31.197496+0000) 2024-03-06T10:43:31.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:31.301 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:32.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:32.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:32.197756+0000) 2024-03-06T10:43:33.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:33.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:33.198029+0000) 2024-03-06T10:43:34.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:34.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:34.198277+0000) 2024-03-06T10:43:35.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:35.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:35.198512+0000) 2024-03-06T10:43:36.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:36.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:36.198747+0000) 2024-03-06T10:43:37.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:37.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:37.198986+0000) 2024-03-06T10:43:38.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:38.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:38.199236+0000) 2024-03-06T10:43:39.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:39.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:39.199475+0000) 2024-03-06T10:43:40.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:40.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:40.199735+0000) 2024-03-06T10:43:41.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:41.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:41.200001+0000) 2024-03-06T10:43:42.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:42.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:42.200242+0000) 2024-03-06T10:43:43.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:43.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:43.200427+0000) 2024-03-06T10:43:44.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:44.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:44.200643+0000) 2024-03-06T10:43:45.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:45.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:45.200874+0000) 2024-03-06T10:43:46.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:46.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:46.201135+0000) 2024-03-06T10:43:47.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:47.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:47.201350+0000) 2024-03-06T10:43:48.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:48.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:48.201580+0000) 2024-03-06T10:43:49.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:49.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:49.201819+0000) 2024-03-06T10:43:50.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:50.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:50.202047+0000) 2024-03-06T10:43:51.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:51.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:51.202282+0000) 2024-03-06T10:43:52.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:52.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:52.202525+0000) 2024-03-06T10:43:53.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:53.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:53.202789+0000) 2024-03-06T10:43:54.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:54.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:54.202986+0000) 2024-03-06T10:43:55.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:55.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:55.203205+0000) 2024-03-06T10:43:56.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:56.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:56.203443+0000) 2024-03-06T10:43:57.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:57.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:57.203627+0000) 2024-03-06T10:43:58.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:58.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:58.203848+0000) 2024-03-06T10:43:59.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:43:59.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:59.204077+0000) 2024-03-06T10:44:00.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:00.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:00.204307+0000) 2024-03-06T10:44:01.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:01.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:01.204494+0000) 2024-03-06T10:44:01.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:01.514 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:02.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:02.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:02.204717+0000) 2024-03-06T10:44:03.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:03.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:03.204936+0000) 2024-03-06T10:44:04.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:04.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:04.205110+0000) 2024-03-06T10:44:05.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:05.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:05.205296+0000) 2024-03-06T10:44:06.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:06.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:06.205556+0000) 2024-03-06T10:44:07.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:07.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:07.205790+0000) 2024-03-06T10:44:08.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:08.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:08.206028+0000) 2024-03-06T10:44:09.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:09.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:09.206214+0000) 2024-03-06T10:44:10.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:10.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:10.206451+0000) 2024-03-06T10:44:11.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:11.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:11.206649+0000) 2024-03-06T10:44:12.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:12.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:12.206872+0000) 2024-03-06T10:44:13.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:13.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:13.207069+0000) 2024-03-06T10:44:14.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:14.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:14.207268+0000) 2024-03-06T10:44:15.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:15.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:15.207552+0000) 2024-03-06T10:44:16.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:16.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:16.207815+0000) 2024-03-06T10:44:17.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:17.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:17.208013+0000) 2024-03-06T10:44:18.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:18.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:18.208240+0000) 2024-03-06T10:44:19.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:19.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:19.208428+0000) 2024-03-06T10:44:20.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:20.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:20.208701+0000) 2024-03-06T10:44:21.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:21.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:21.208936+0000) 2024-03-06T10:44:22.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:22.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:22.209169+0000) 2024-03-06T10:44:23.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:23.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:23.209403+0000) 2024-03-06T10:44:24.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:24.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:24.209606+0000) 2024-03-06T10:44:25.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:25.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:25.209903+0000) 2024-03-06T10:44:26.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:26.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:26.210115+0000) 2024-03-06T10:44:27.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:27.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:27.210309+0000) 2024-03-06T10:44:28.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:28.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:28.210506+0000) 2024-03-06T10:44:29.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:29.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:29.210757+0000) 2024-03-06T10:44:30.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:30.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:30.211034+0000) 2024-03-06T10:44:31.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:31.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:31.211273+0000) 2024-03-06T10:44:31.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:31.691 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:32.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:32.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:32.211434+0000) 2024-03-06T10:44:33.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:33.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:33.211659+0000) 2024-03-06T10:44:34.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:34.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:34.211953+0000) 2024-03-06T10:44:35.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:35.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:35.212147+0000) 2024-03-06T10:44:36.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:36.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:36.212347+0000) 2024-03-06T10:44:37.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:37.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:37.212584+0000) 2024-03-06T10:44:38.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:38.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:38.212817+0000) 2024-03-06T10:44:39.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:39.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:39.213003+0000) 2024-03-06T10:44:40.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:40.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:40.213248+0000) 2024-03-06T10:44:41.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:41.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:41.213487+0000) 2024-03-06T10:44:42.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:42.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:42.213711+0000) 2024-03-06T10:44:43.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:43.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:43.213922+0000) 2024-03-06T10:44:44.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:44.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:44.214157+0000) 2024-03-06T10:44:45.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:45.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:45.214336+0000) 2024-03-06T10:44:46.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:46.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:46.214569+0000) 2024-03-06T10:44:47.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:47.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:47.214801+0000) 2024-03-06T10:44:48.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:48.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:48.215002+0000) 2024-03-06T10:44:49.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:49.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:49.215235+0000) 2024-03-06T10:44:50.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:50.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:50.215465+0000) 2024-03-06T10:44:51.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:51.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:51.215725+0000) 2024-03-06T10:44:52.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:52.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:52.215950+0000) 2024-03-06T10:44:53.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:53.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:53.216154+0000) 2024-03-06T10:44:54.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:54.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:54.216337+0000) 2024-03-06T10:44:55.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:55.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:55.216572+0000) 2024-03-06T10:44:56.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:56.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:56.216836+0000) 2024-03-06T10:44:57.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:57.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:57.217077+0000) 2024-03-06T10:44:58.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:58.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:58.217303+0000) 2024-03-06T10:44:59.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:44:59.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:59.217494+0000) 2024-03-06T10:45:00.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:00.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:00.217733+0000) 2024-03-06T10:45:01.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:01.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:01.217971+0000) 2024-03-06T10:45:01.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:01.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:02.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:02.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:02.218205+0000) 2024-03-06T10:45:03.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:03.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:03.218434+0000) 2024-03-06T10:45:04.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:04.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:04.218659+0000) 2024-03-06T10:45:05.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:05.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:05.218929+0000) 2024-03-06T10:45:06.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:06.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:06.219165+0000) 2024-03-06T10:45:07.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:07.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:07.219341+0000) 2024-03-06T10:45:08.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:08.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:08.219556+0000) 2024-03-06T10:45:09.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:09.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:09.219777+0000) 2024-03-06T10:45:10.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:10.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:10.220015+0000) 2024-03-06T10:45:11.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:11.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:11.220240+0000) 2024-03-06T10:45:12.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:12.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:12.220465+0000) 2024-03-06T10:45:13.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:13.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:13.220735+0000) 2024-03-06T10:45:14.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:14.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:14.220940+0000) 2024-03-06T10:45:15.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:15.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:15.221139+0000) 2024-03-06T10:45:16.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:16.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:16.221348+0000) 2024-03-06T10:45:17.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:17.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:17.221599+0000) 2024-03-06T10:45:18.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:18.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:18.221845+0000) 2024-03-06T10:45:19.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:19.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:19.222133+0000) 2024-03-06T10:45:20.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:20.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:20.222359+0000) 2024-03-06T10:45:21.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:21.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:21.222596+0000) 2024-03-06T10:45:22.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:22.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:22.222840+0000) 2024-03-06T10:45:23.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:23.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:23.223064+0000) 2024-03-06T10:45:24.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:24.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:24.223285+0000) 2024-03-06T10:45:25.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:25.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:25.223530+0000) 2024-03-06T10:45:26.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:26.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:26.223798+0000) 2024-03-06T10:45:27.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:27.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:27.224009+0000) 2024-03-06T10:45:28.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:28.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:28.224229+0000) 2024-03-06T10:45:29.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:29.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:29.224450+0000) 2024-03-06T10:45:30.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:30.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:30.224639+0000) 2024-03-06T10:45:31.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:31.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:31.224907+0000) 2024-03-06T10:45:32.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:32.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:32.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:32.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:32.225149+0000) 2024-03-06T10:45:33.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:33.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:33.225361+0000) 2024-03-06T10:45:34.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:34.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:34.225587+0000) 2024-03-06T10:45:35.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:35.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:35.225856+0000) 2024-03-06T10:45:36.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:36.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:36.226064+0000) 2024-03-06T10:45:37.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:37.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:37.226308+0000) 2024-03-06T10:45:38.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:38.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:38.226563+0000) 2024-03-06T10:45:39.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:39.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:39.226798+0000) 2024-03-06T10:45:40.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:40.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:40.227033+0000) 2024-03-06T10:45:41.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:41.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:41.227271+0000) 2024-03-06T10:45:42.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:42.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:42.227522+0000) 2024-03-06T10:45:43.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:43.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:43.227797+0000) 2024-03-06T10:45:44.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:44.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:44.228039+0000) 2024-03-06T10:45:45.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:45.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:45.228290+0000) 2024-03-06T10:45:46.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:46.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:46.228509+0000) 2024-03-06T10:45:47.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:47.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:47.228778+0000) 2024-03-06T10:45:48.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:48.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:48.228979+0000) 2024-03-06T10:45:49.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:49.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:49.229218+0000) 2024-03-06T10:45:50.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:50.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:50.229541+0000) 2024-03-06T10:45:51.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:51.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:51.229793+0000) 2024-03-06T10:45:52.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:52.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:52.229998+0000) 2024-03-06T10:45:53.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:53.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:53.230240+0000) 2024-03-06T10:45:54.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:54.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:54.230489+0000) 2024-03-06T10:45:55.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:55.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:55.230730+0000) 2024-03-06T10:45:56.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:56.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:56.230930+0000) 2024-03-06T10:45:57.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:57.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:57.231150+0000) 2024-03-06T10:45:58.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:58.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:58.231374+0000) 2024-03-06T10:45:59.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:45:59.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:59.231598+0000) 2024-03-06T10:46:00.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:00.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:00.231854+0000) 2024-03-06T10:46:01.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:01.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:01.232045+0000) 2024-03-06T10:46:02.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:02.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:02.232241+0000) 2024-03-06T10:46:02.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:02.255 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:03.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:03.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:03.232481+0000) 2024-03-06T10:46:04.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:04.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:04.232758+0000) 2024-03-06T10:46:05.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:05.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:05.232980+0000) 2024-03-06T10:46:06.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:06.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:06.233238+0000) 2024-03-06T10:46:07.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:07.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:07.233419+0000) 2024-03-06T10:46:08.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:08.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:08.233622+0000) 2024-03-06T10:46:09.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:09.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:09.233868+0000) 2024-03-06T10:46:10.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:10.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:10.234081+0000) 2024-03-06T10:46:11.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:11.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:11.234287+0000) 2024-03-06T10:46:12.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:12.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:12.234481+0000) 2024-03-06T10:46:13.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:13.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:13.234753+0000) 2024-03-06T10:46:14.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:14.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:14.234953+0000) 2024-03-06T10:46:15.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:15.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:15.235166+0000) 2024-03-06T10:46:16.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:16.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:16.235392+0000) 2024-03-06T10:46:17.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:17.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:17.235607+0000) 2024-03-06T10:46:18.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:18.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:18.235870+0000) 2024-03-06T10:46:19.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:19.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:19.236095+0000) 2024-03-06T10:46:20.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:20.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:20.236332+0000) 2024-03-06T10:46:21.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:21.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:21.236557+0000) 2024-03-06T10:46:22.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:22.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:22.236774+0000) 2024-03-06T10:46:23.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:23.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:23.236951+0000) 2024-03-06T10:46:24.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:24.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:24.237175+0000) 2024-03-06T10:46:25.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:25.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:25.237414+0000) 2024-03-06T10:46:26.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:26.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:26.237656+0000) 2024-03-06T10:46:27.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:27.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:27.237905+0000) 2024-03-06T10:46:28.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:28.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:28.238144+0000) 2024-03-06T10:46:29.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:29.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:29.238394+0000) 2024-03-06T10:46:30.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:30.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:30.238564+0000) 2024-03-06T10:46:31.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:31.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:31.238781+0000) 2024-03-06T10:46:32.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:32.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:32.238985+0000) 2024-03-06T10:46:32.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:32.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:33.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:33.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:33.239192+0000) 2024-03-06T10:46:34.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:34.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:34.239435+0000) 2024-03-06T10:46:35.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:35.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:35.239648+0000) 2024-03-06T10:46:36.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:36.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:36.239849+0000) 2024-03-06T10:46:37.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:37.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:37.240027+0000) 2024-03-06T10:46:38.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:38.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:38.240224+0000) 2024-03-06T10:46:39.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:39.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:39.240464+0000) 2024-03-06T10:46:40.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:40.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:40.240655+0000) 2024-03-06T10:46:41.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:41.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:41.240890+0000) 2024-03-06T10:46:42.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:42.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:42.241082+0000) 2024-03-06T10:46:43.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:43.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:43.241277+0000) 2024-03-06T10:46:44.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:44.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:44.241512+0000) 2024-03-06T10:46:45.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:45.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:45.241787+0000) 2024-03-06T10:46:46.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:46.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:46.242013+0000) 2024-03-06T10:46:47.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:47.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:47.242222+0000) 2024-03-06T10:46:48.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:48.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:48.242499+0000) 2024-03-06T10:46:49.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:49.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:49.242820+0000) 2024-03-06T10:46:50.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:50.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:50.243092+0000) 2024-03-06T10:46:51.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:51.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:51.243310+0000) 2024-03-06T10:46:52.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:52.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:52.243556+0000) 2024-03-06T10:46:53.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:53.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:53.243810+0000) 2024-03-06T10:46:54.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:54.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:54.244092+0000) 2024-03-06T10:46:55.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:55.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:55.244304+0000) 2024-03-06T10:46:56.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:56.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:56.244563+0000) 2024-03-06T10:46:57.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:57.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:57.244803+0000) 2024-03-06T10:46:58.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:58.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:58.244962+0000) 2024-03-06T10:46:59.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:46:59.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:59.245090+0000) 2024-03-06T10:47:00.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:00.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:00.245212+0000) 2024-03-06T10:47:01.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:01.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:01.245433+0000) 2024-03-06T10:47:02.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:02.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:02.245660+0000) 2024-03-06T10:47:02.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:02.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:03.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:03.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:03.245905+0000) 2024-03-06T10:47:04.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:04.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:04.246064+0000) 2024-03-06T10:47:05.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:05.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:05.246315+0000) 2024-03-06T10:47:06.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:06.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:06.246553+0000) 2024-03-06T10:47:07.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:07.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:07.246764+0000) 2024-03-06T10:47:08.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:08.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:08.246991+0000) 2024-03-06T10:47:09.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:09.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:09.247226+0000) 2024-03-06T10:47:10.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:10.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:10.247431+0000) 2024-03-06T10:47:11.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:11.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:11.247595+0000) 2024-03-06T10:47:12.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:12.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:12.247842+0000) 2024-03-06T10:47:13.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:13.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:13.248031+0000) 2024-03-06T10:47:14.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:14.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:14.248244+0000) 2024-03-06T10:47:15.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:15.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:15.248474+0000) 2024-03-06T10:47:16.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:16.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:16.248777+0000) 2024-03-06T10:47:17.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:17.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:17.248938+0000) 2024-03-06T10:47:18.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:18.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:18.249161+0000) 2024-03-06T10:47:19.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:19.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:19.249412+0000) 2024-03-06T10:47:20.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:20.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:20.249637+0000) 2024-03-06T10:47:21.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:21.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:21.249910+0000) 2024-03-06T10:47:22.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:22.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:22.250145+0000) 2024-03-06T10:47:23.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:23.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:23.250360+0000) 2024-03-06T10:47:24.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:24.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:24.250549+0000) 2024-03-06T10:47:25.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:25.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:25.250784+0000) 2024-03-06T10:47:26.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:26.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:26.251017+0000) 2024-03-06T10:47:27.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:27.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:27.251245+0000) 2024-03-06T10:47:28.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:28.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:28.251425+0000) 2024-03-06T10:47:29.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:29.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:29.251654+0000) 2024-03-06T10:47:30.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:30.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:30.251884+0000) 2024-03-06T10:47:31.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:31.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:31.252111+0000) 2024-03-06T10:47:32.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:32.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:32.252307+0000) 2024-03-06T10:47:32.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:32.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:33.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:33.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:33.252499+0000) 2024-03-06T10:47:34.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:34.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:34.252773+0000) 2024-03-06T10:47:35.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:35.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:35.253000+0000) 2024-03-06T10:47:36.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:36.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:36.253275+0000) 2024-03-06T10:47:37.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:37.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:37.253484+0000) 2024-03-06T10:47:38.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:38.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:38.253745+0000) 2024-03-06T10:47:39.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:39.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:39.253931+0000) 2024-03-06T10:47:40.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:40.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:40.254124+0000) 2024-03-06T10:47:41.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:41.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:41.254361+0000) 2024-03-06T10:47:42.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:42.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:42.254585+0000) 2024-03-06T10:47:43.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:43.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:43.254819+0000) 2024-03-06T10:47:44.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:44.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:44.255049+0000) 2024-03-06T10:47:45.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:45.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:45.255325+0000) 2024-03-06T10:47:46.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:46.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:46.255588+0000) 2024-03-06T10:47:47.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:47.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:47.255823+0000) 2024-03-06T10:47:48.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:48.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:48.256053+0000) 2024-03-06T10:47:49.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:49.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:49.256260+0000) 2024-03-06T10:47:50.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:50.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:50.256465+0000) 2024-03-06T10:47:51.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:51.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:51.256702+0000) 2024-03-06T10:47:52.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:52.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:52.256936+0000) 2024-03-06T10:47:53.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:53.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:53.257164+0000) 2024-03-06T10:47:54.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:54.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:54.257375+0000) 2024-03-06T10:47:55.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:55.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:55.257590+0000) 2024-03-06T10:47:56.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:56.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:56.257853+0000) 2024-03-06T10:47:57.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:57.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:57.258089+0000) 2024-03-06T10:47:58.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:58.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:58.258277+0000) 2024-03-06T10:47:59.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:47:59.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:59.258532+0000) 2024-03-06T10:48:00.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:00.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:00.258782+0000) 2024-03-06T10:48:01.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:01.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:01.259010+0000) 2024-03-06T10:48:02.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:02.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:02.259202+0000) 2024-03-06T10:48:03.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:03.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:03.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:03.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:03.259417+0000) 2024-03-06T10:48:04.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:04.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:04.259713+0000) 2024-03-06T10:48:05.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:05.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:05.259983+0000) 2024-03-06T10:48:06.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:06.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:06.260275+0000) 2024-03-06T10:48:07.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:07.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:07.260556+0000) 2024-03-06T10:48:08.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:08.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:08.260837+0000) 2024-03-06T10:48:09.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:09.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:09.261042+0000) 2024-03-06T10:48:10.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:10.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:10.261283+0000) 2024-03-06T10:48:11.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:11.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:11.261512+0000) 2024-03-06T10:48:12.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:12.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:12.261721+0000) 2024-03-06T10:48:13.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:13.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:13.261921+0000) 2024-03-06T10:48:14.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:14.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:14.262118+0000) 2024-03-06T10:48:15.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:15.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:15.262343+0000) 2024-03-06T10:48:16.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:16.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:16.262614+0000) 2024-03-06T10:48:17.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:17.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:17.262904+0000) 2024-03-06T10:48:18.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:18.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:18.263183+0000) 2024-03-06T10:48:19.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:19.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:19.263462+0000) 2024-03-06T10:48:20.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:20.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:20.263759+0000) 2024-03-06T10:48:21.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:21.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:21.263975+0000) 2024-03-06T10:48:22.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:22.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:22.264291+0000) 2024-03-06T10:48:23.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:23.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:23.264547+0000) 2024-03-06T10:48:24.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:24.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:24.264781+0000) 2024-03-06T10:48:25.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:25.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:25.264969+0000) 2024-03-06T10:48:26.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:26.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:26.265190+0000) 2024-03-06T10:48:27.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:27.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:27.265392+0000) 2024-03-06T10:48:28.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:28.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:28.265632+0000) 2024-03-06T10:48:29.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:29.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:29.265860+0000) 2024-03-06T10:48:30.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:30.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:30.266084+0000) 2024-03-06T10:48:31.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:31.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:31.266310+0000) 2024-03-06T10:48:32.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:32.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:32.266541+0000) 2024-03-06T10:48:33.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:33.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:33.266837+0000) 2024-03-06T10:48:33.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:33.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:34.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:34.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:34.267035+0000) 2024-03-06T10:48:35.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:35.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:35.267257+0000) 2024-03-06T10:48:36.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:36.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:36.267455+0000) 2024-03-06T10:48:37.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:37.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:37.267653+0000) 2024-03-06T10:48:38.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:38.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:38.267910+0000) 2024-03-06T10:48:39.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:39.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:39.268169+0000) 2024-03-06T10:48:40.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:40.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:40.268373+0000) 2024-03-06T10:48:41.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:41.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:41.268604+0000) 2024-03-06T10:48:42.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:42.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:42.268836+0000) 2024-03-06T10:48:43.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:43.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:43.269102+0000) 2024-03-06T10:48:44.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:44.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:44.269373+0000) 2024-03-06T10:48:45.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:45.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:45.269566+0000) 2024-03-06T10:48:46.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:46.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:46.269822+0000) 2024-03-06T10:48:47.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:47.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:47.270101+0000) 2024-03-06T10:48:48.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:48.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:48.270339+0000) 2024-03-06T10:48:49.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:49.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:49.270591+0000) 2024-03-06T10:48:50.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:50.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:50.270861+0000) 2024-03-06T10:48:51.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:51.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:51.271140+0000) 2024-03-06T10:48:52.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:52.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:52.271417+0000) 2024-03-06T10:48:53.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:53.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:53.271732+0000) 2024-03-06T10:48:54.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:54.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:54.272001+0000) 2024-03-06T10:48:55.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:55.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:55.272280+0000) 2024-03-06T10:48:56.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:56.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:56.272496+0000) 2024-03-06T10:48:57.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:57.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:57.272710+0000) 2024-03-06T10:48:58.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:58.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:58.272951+0000) 2024-03-06T10:48:59.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:48:59.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:59.273221+0000) 2024-03-06T10:49:00.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:00.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:00.273521+0000) 2024-03-06T10:49:01.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:01.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:01.273789+0000) 2024-03-06T10:49:02.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:02.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:02.274019+0000) 2024-03-06T10:49:03.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:03.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:03.274243+0000) 2024-03-06T10:49:03.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:03.422 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:04.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:04.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:04.274465+0000) 2024-03-06T10:49:05.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:05.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:05.274655+0000) 2024-03-06T10:49:06.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:06.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:06.274896+0000) 2024-03-06T10:49:07.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:07.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:07.275127+0000) 2024-03-06T10:49:08.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:08.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:08.275295+0000) 2024-03-06T10:49:09.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:09.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:09.275505+0000) 2024-03-06T10:49:10.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:10.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:10.275769+0000) 2024-03-06T10:49:11.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:11.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:11.275996+0000) 2024-03-06T10:49:12.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:12.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:12.276217+0000) 2024-03-06T10:49:13.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:13.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:13.276432+0000) 2024-03-06T10:49:14.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:14.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:14.276743+0000) 2024-03-06T10:49:15.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:15.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:15.277028+0000) 2024-03-06T10:49:16.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:16.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:16.277318+0000) 2024-03-06T10:49:17.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:17.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:17.277593+0000) 2024-03-06T10:49:18.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:18.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:18.277831+0000) 2024-03-06T10:49:19.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:19.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:19.278039+0000) 2024-03-06T10:49:20.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:20.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:20.278248+0000) 2024-03-06T10:49:21.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:21.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:21.278501+0000) 2024-03-06T10:49:22.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:22.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:22.278645+0000) 2024-03-06T10:49:23.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:23.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:23.278854+0000) 2024-03-06T10:49:24.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:24.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:24.279048+0000) 2024-03-06T10:49:25.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:25.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:25.279289+0000) 2024-03-06T10:49:26.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:26.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:26.279575+0000) 2024-03-06T10:49:27.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:27.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:27.279855+0000) 2024-03-06T10:49:28.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:28.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:28.280128+0000) 2024-03-06T10:49:29.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:29.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:29.280409+0000) 2024-03-06T10:49:30.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:30.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:30.280619+0000) 2024-03-06T10:49:31.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:31.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:31.280786+0000) 2024-03-06T10:49:32.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:32.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:32.281018+0000) 2024-03-06T10:49:33.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:33.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:33.281297+0000) 2024-03-06T10:49:33.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:33.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:34.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:34.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:34.281508+0000) 2024-03-06T10:49:35.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:35.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:35.281806+0000) 2024-03-06T10:49:36.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:36.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:36.281979+0000) 2024-03-06T10:49:37.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:37.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:37.282199+0000) 2024-03-06T10:49:38.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:38.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:38.282473+0000) 2024-03-06T10:49:39.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:39.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:39.282688+0000) 2024-03-06T10:49:40.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:40.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:40.282871+0000) 2024-03-06T10:49:41.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:41.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:41.283160+0000) 2024-03-06T10:49:42.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:42.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:42.283387+0000) 2024-03-06T10:49:43.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:43.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:43.283613+0000) 2024-03-06T10:49:44.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:44.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:44.283845+0000) 2024-03-06T10:49:45.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:45.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:45.284110+0000) 2024-03-06T10:49:46.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:46.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:46.284391+0000) 2024-03-06T10:49:47.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:47.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:47.284592+0000) 2024-03-06T10:49:48.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:48.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:48.284826+0000) 2024-03-06T10:49:49.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:49.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:49.285018+0000) 2024-03-06T10:49:50.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:50.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:50.285208+0000) 2024-03-06T10:49:51.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:51.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:51.285418+0000) 2024-03-06T10:49:52.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:52.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:52.285730+0000) 2024-03-06T10:49:53.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:53.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:53.286014+0000) 2024-03-06T10:49:54.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:54.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:54.286221+0000) 2024-03-06T10:49:55.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:55.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:55.286445+0000) 2024-03-06T10:49:56.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:56.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:56.286640+0000) 2024-03-06T10:49:57.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:57.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:57.286906+0000) 2024-03-06T10:49:58.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:58.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:58.287174+0000) 2024-03-06T10:49:59.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:49:59.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:59.287406+0000) 2024-03-06T10:50:00.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:00.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:00.287720+0000) 2024-03-06T10:50:01.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:01.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:01.288000+0000) 2024-03-06T10:50:02.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:02.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:02.288270+0000) 2024-03-06T10:50:03.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:03.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:03.288513+0000) 2024-03-06T10:50:03.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:03.831 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:04.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:04.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:04.288784+0000) 2024-03-06T10:50:05.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:05.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:05.289022+0000) 2024-03-06T10:50:06.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:06.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:06.289247+0000) 2024-03-06T10:50:07.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:07.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:07.289527+0000) 2024-03-06T10:50:08.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:08.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:08.289797+0000) 2024-03-06T10:50:09.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:09.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:09.290017+0000) 2024-03-06T10:50:10.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:10.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:10.290212+0000) 2024-03-06T10:50:11.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:11.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:11.290436+0000) 2024-03-06T10:50:12.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:12.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:12.290701+0000) 2024-03-06T10:50:13.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:13.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:13.290915+0000) 2024-03-06T10:50:14.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:14.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:14.291183+0000) 2024-03-06T10:50:15.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:15.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:15.291469+0000) 2024-03-06T10:50:16.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:16.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:16.291764+0000) 2024-03-06T10:50:17.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:17.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:17.291954+0000) 2024-03-06T10:50:18.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:18.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:18.292149+0000) 2024-03-06T10:50:19.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:19.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:19.292429+0000) 2024-03-06T10:50:20.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:20.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:20.292748+0000) 2024-03-06T10:50:21.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:21.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:21.292973+0000) 2024-03-06T10:50:22.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:22.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:22.293246+0000) 2024-03-06T10:50:23.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:23.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:23.293493+0000) 2024-03-06T10:50:24.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:24.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:24.293751+0000) 2024-03-06T10:50:25.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:25.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:25.293971+0000) 2024-03-06T10:50:26.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:26.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:26.294176+0000) 2024-03-06T10:50:27.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:27.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:27.294412+0000) 2024-03-06T10:50:28.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:28.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:28.294647+0000) 2024-03-06T10:50:29.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:29.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:29.294874+0000) 2024-03-06T10:50:30.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:30.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:30.295062+0000) 2024-03-06T10:50:31.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:31.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:31.295307+0000) 2024-03-06T10:50:32.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:32.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:32.295516+0000) 2024-03-06T10:50:33.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:33.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:33.295786+0000) 2024-03-06T10:50:34.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:34.060 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:34.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:34.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:34.296039+0000) 2024-03-06T10:50:35.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:35.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:35.296266+0000) 2024-03-06T10:50:36.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:36.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:36.296468+0000) 2024-03-06T10:50:37.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:37.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:37.296729+0000) 2024-03-06T10:50:38.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:38.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:38.296956+0000) 2024-03-06T10:50:39.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:39.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:39.297219+0000) 2024-03-06T10:50:40.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:40.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:40.297505+0000) 2024-03-06T10:50:41.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:41.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:41.297790+0000) 2024-03-06T10:50:42.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:42.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:42.297948+0000) 2024-03-06T10:50:43.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:43.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:43.298134+0000) 2024-03-06T10:50:44.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:44.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:44.298393+0000) 2024-03-06T10:50:45.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:45.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:45.298595+0000) 2024-03-06T10:50:46.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:46.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:46.298834+0000) 2024-03-06T10:50:47.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:47.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:47.299020+0000) 2024-03-06T10:50:48.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:48.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:48.299259+0000) 2024-03-06T10:50:49.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:49.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:49.299503+0000) 2024-03-06T10:50:50.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:50.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:50.299789+0000) 2024-03-06T10:50:51.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:51.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:51.300028+0000) 2024-03-06T10:50:52.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:52.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:52.300309+0000) 2024-03-06T10:50:53.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:53.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:53.300509+0000) 2024-03-06T10:50:54.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:54.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:54.300788+0000) 2024-03-06T10:50:55.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:55.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:55.301019+0000) 2024-03-06T10:50:56.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:56.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:56.301265+0000) 2024-03-06T10:50:57.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:57.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:57.301461+0000) 2024-03-06T10:50:58.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:58.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:58.301747+0000) 2024-03-06T10:50:59.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:50:59.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:59.301963+0000) 2024-03-06T10:51:00.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:00.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:00.302172+0000) 2024-03-06T10:51:01.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:01.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:01.302414+0000) 2024-03-06T10:51:02.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:02.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:02.302640+0000) 2024-03-06T10:51:03.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:03.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:03.302967+0000) 2024-03-06T10:51:04.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:04.188 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:04.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:04.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:04.303224+0000) 2024-03-06T10:51:05.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:05.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:05.303429+0000) 2024-03-06T10:51:06.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:06.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:06.303631+0000) 2024-03-06T10:51:07.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:07.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:07.303863+0000) 2024-03-06T10:51:08.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:08.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:08.304053+0000) 2024-03-06T10:51:09.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:09.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:09.304278+0000) 2024-03-06T10:51:10.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:10.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:10.304505+0000) 2024-03-06T10:51:11.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:11.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:11.304819+0000) 2024-03-06T10:51:12.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:12.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:12.305103+0000) 2024-03-06T10:51:13.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:13.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:13.305343+0000) 2024-03-06T10:51:14.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:14.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:14.305618+0000) 2024-03-06T10:51:15.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:15.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:15.305828+0000) 2024-03-06T10:51:16.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:16.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:16.306012+0000) 2024-03-06T10:51:17.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:17.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:17.306192+0000) 2024-03-06T10:51:18.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:18.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:18.306423+0000) 2024-03-06T10:51:19.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:19.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:19.306655+0000) 2024-03-06T10:51:20.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:20.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:20.306916+0000) 2024-03-06T10:51:21.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:21.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:21.307174+0000) 2024-03-06T10:51:22.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:22.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:22.307508+0000) 2024-03-06T10:51:23.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:23.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:23.307805+0000) 2024-03-06T10:51:24.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:24.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:24.308094+0000) 2024-03-06T10:51:25.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:25.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:25.308378+0000) 2024-03-06T10:51:26.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:26.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:26.308630+0000) 2024-03-06T10:51:27.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:27.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:27.308889+0000) 2024-03-06T10:51:28.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:28.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:28.309114+0000) 2024-03-06T10:51:29.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:29.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:29.309373+0000) 2024-03-06T10:51:30.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:30.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:30.309606+0000) 2024-03-06T10:51:31.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:31.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:31.309834+0000) 2024-03-06T10:51:32.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:32.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:32.310067+0000) 2024-03-06T10:51:33.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:33.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:33.310292+0000) 2024-03-06T10:51:34.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:34.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:34.310551+0000) 2024-03-06T10:51:34.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:34.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:35.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:35.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:35.310793+0000) 2024-03-06T10:51:36.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:36.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:36.311016+0000) 2024-03-06T10:51:37.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:37.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:37.311205+0000) 2024-03-06T10:51:38.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:38.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:38.311456+0000) 2024-03-06T10:51:39.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:39.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:39.311731+0000) 2024-03-06T10:51:40.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:40.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:40.311964+0000) 2024-03-06T10:51:41.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:41.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:41.312197+0000) 2024-03-06T10:51:42.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:42.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:42.312426+0000) 2024-03-06T10:51:43.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:43.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:43.312654+0000) 2024-03-06T10:51:44.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:44.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:44.312865+0000) 2024-03-06T10:51:45.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:45.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:45.313045+0000) 2024-03-06T10:51:46.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:46.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:46.313205+0000) 2024-03-06T10:51:47.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:47.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:47.313385+0000) 2024-03-06T10:51:48.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:48.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:48.313628+0000) 2024-03-06T10:51:49.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:49.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:49.313907+0000) 2024-03-06T10:51:50.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:50.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:50.314153+0000) 2024-03-06T10:51:51.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:51.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:51.314370+0000) 2024-03-06T10:51:52.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:52.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:52.314569+0000) 2024-03-06T10:51:53.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:53.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:53.314800+0000) 2024-03-06T10:51:54.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:54.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:54.315020+0000) 2024-03-06T10:51:55.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:55.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:55.315261+0000) 2024-03-06T10:51:56.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:56.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:56.315472+0000) 2024-03-06T10:51:57.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:57.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:57.315741+0000) 2024-03-06T10:51:58.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:58.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:58.315956+0000) 2024-03-06T10:51:59.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:51:59.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:59.316203+0000) 2024-03-06T10:52:00.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:00.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:00.316427+0000) 2024-03-06T10:52:01.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:01.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:01.316618+0000) 2024-03-06T10:52:02.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:02.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:02.316826+0000) 2024-03-06T10:52:03.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:03.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:03.317050+0000) 2024-03-06T10:52:04.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:04.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:04.317277+0000) 2024-03-06T10:52:04.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:04.630 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:05.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:05.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:05.317514+0000) 2024-03-06T10:52:06.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:06.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:06.317796+0000) 2024-03-06T10:52:07.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:07.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:07.318006+0000) 2024-03-06T10:52:08.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:08.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:08.318234+0000) 2024-03-06T10:52:09.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:09.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:09.318469+0000) 2024-03-06T10:52:10.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:10.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:10.318797+0000) 2024-03-06T10:52:11.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:11.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:11.319011+0000) 2024-03-06T10:52:12.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:12.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:12.319252+0000) 2024-03-06T10:52:13.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:13.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:13.319491+0000) 2024-03-06T10:52:14.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:14.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:14.319708+0000) 2024-03-06T10:52:15.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:15.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:15.319924+0000) 2024-03-06T10:52:16.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:16.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:16.320128+0000) 2024-03-06T10:52:17.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:17.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:17.320317+0000) 2024-03-06T10:52:18.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:18.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:18.320549+0000) 2024-03-06T10:52:19.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:19.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:19.320782+0000) 2024-03-06T10:52:20.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:20.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:20.320953+0000) 2024-03-06T10:52:21.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:21.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:21.321159+0000) 2024-03-06T10:52:22.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:22.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:22.321346+0000) 2024-03-06T10:52:23.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:23.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:23.321537+0000) 2024-03-06T10:52:24.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:24.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:24.321812+0000) 2024-03-06T10:52:25.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:25.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:25.321962+0000) 2024-03-06T10:52:26.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:26.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:26.322179+0000) 2024-03-06T10:52:27.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:27.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:27.322352+0000) 2024-03-06T10:52:28.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:28.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:28.322589+0000) 2024-03-06T10:52:29.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:29.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:29.322826+0000) 2024-03-06T10:52:30.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:30.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:30.323050+0000) 2024-03-06T10:52:31.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:31.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:31.323280+0000) 2024-03-06T10:52:32.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:32.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:32.323502+0000) 2024-03-06T10:52:33.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:33.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:33.323700+0000) 2024-03-06T10:52:34.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:34.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:34.323898+0000) 2024-03-06T10:52:34.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:34.852 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:35.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:35.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:35.324134+0000) 2024-03-06T10:52:36.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:36.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:36.324375+0000) 2024-03-06T10:52:37.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:37.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:37.324611+0000) 2024-03-06T10:52:38.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:38.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:38.324821+0000) 2024-03-06T10:52:39.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:39.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:39.325017+0000) 2024-03-06T10:52:40.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:40.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:40.325245+0000) 2024-03-06T10:52:41.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:41.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:41.325470+0000) 2024-03-06T10:52:42.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:42.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:42.325740+0000) 2024-03-06T10:52:43.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:43.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:43.325974+0000) 2024-03-06T10:52:44.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:44.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:44.326199+0000) 2024-03-06T10:52:45.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:45.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:45.326417+0000) 2024-03-06T10:52:46.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:46.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:46.326656+0000) 2024-03-06T10:52:47.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:47.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:47.326912+0000) 2024-03-06T10:52:48.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:48.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:48.327147+0000) 2024-03-06T10:52:49.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:49.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:49.327361+0000) 2024-03-06T10:52:50.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:50.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:50.327619+0000) 2024-03-06T10:52:51.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:51.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:51.327815+0000) 2024-03-06T10:52:52.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:52.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:52.328018+0000) 2024-03-06T10:52:53.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:53.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:53.328198+0000) 2024-03-06T10:52:54.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:54.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:54.328389+0000) 2024-03-06T10:52:55.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:55.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:55.328608+0000) 2024-03-06T10:52:56.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:56.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:56.328825+0000) 2024-03-06T10:52:57.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:57.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:57.329020+0000) 2024-03-06T10:52:58.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:58.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:58.329203+0000) 2024-03-06T10:52:59.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:52:59.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:59.329380+0000) 2024-03-06T10:53:00.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:00.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:00.329564+0000) 2024-03-06T10:53:01.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:01.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:01.329787+0000) 2024-03-06T10:53:02.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:02.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:02.330006+0000) 2024-03-06T10:53:03.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:03.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:03.330191+0000) 2024-03-06T10:53:04.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:04.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:04.330425+0000) 2024-03-06T10:53:05.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:05.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:05.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:05.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:05.330609+0000) 2024-03-06T10:53:06.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:06.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:06.330851+0000) 2024-03-06T10:53:07.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:07.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:07.331041+0000) 2024-03-06T10:53:08.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:08.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:08.331263+0000) 2024-03-06T10:53:09.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:09.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:09.331568+0000) 2024-03-06T10:53:10.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:10.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:10.331792+0000) 2024-03-06T10:53:11.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:11.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:11.332017+0000) 2024-03-06T10:53:12.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:12.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:12.332250+0000) 2024-03-06T10:53:13.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:13.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:13.332477+0000) 2024-03-06T10:53:14.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:14.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:14.332748+0000) 2024-03-06T10:53:15.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:15.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:15.332925+0000) 2024-03-06T10:53:16.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:16.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:16.333155+0000) 2024-03-06T10:53:17.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:17.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:17.333384+0000) 2024-03-06T10:53:18.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:18.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:18.333613+0000) 2024-03-06T10:53:19.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:19.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:19.333845+0000) 2024-03-06T10:53:20.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:20.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:20.334070+0000) 2024-03-06T10:53:21.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:21.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:21.334254+0000) 2024-03-06T10:53:22.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:22.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:22.334494+0000) 2024-03-06T10:53:23.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:23.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:23.334699+0000) 2024-03-06T10:53:24.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:24.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:24.334924+0000) 2024-03-06T10:53:25.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:25.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:25.335146+0000) 2024-03-06T10:53:26.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:26.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:26.335336+0000) 2024-03-06T10:53:27.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:27.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:27.335568+0000) 2024-03-06T10:53:28.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:28.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:28.335795+0000) 2024-03-06T10:53:29.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:29.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:29.335962+0000) 2024-03-06T10:53:30.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:30.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:30.336192+0000) 2024-03-06T10:53:31.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:31.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:31.336451+0000) 2024-03-06T10:53:32.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:32.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:32.336700+0000) 2024-03-06T10:53:33.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:33.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:33.336985+0000) 2024-03-06T10:53:34.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:34.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:34.337213+0000) 2024-03-06T10:53:35.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:35.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:35.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:35.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:35.337405+0000) 2024-03-06T10:53:36.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:36.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:36.337643+0000) 2024-03-06T10:53:37.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:37.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:37.337862+0000) 2024-03-06T10:53:38.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:38.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:38.338060+0000) 2024-03-06T10:53:39.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:39.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:39.338286+0000) 2024-03-06T10:53:40.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:40.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:40.338513+0000) 2024-03-06T10:53:41.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:41.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:41.338775+0000) 2024-03-06T10:53:42.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:42.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:42.339010+0000) 2024-03-06T10:53:43.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:43.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:43.339208+0000) 2024-03-06T10:53:44.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:44.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:44.339459+0000) 2024-03-06T10:53:45.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:45.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:45.339722+0000) 2024-03-06T10:53:46.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:46.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:46.339975+0000) 2024-03-06T10:53:47.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:47.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:47.340204+0000) 2024-03-06T10:53:48.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:48.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:48.340426+0000) 2024-03-06T10:53:49.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:49.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:49.340742+0000) 2024-03-06T10:53:50.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:50.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:50.341025+0000) 2024-03-06T10:53:51.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:51.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:51.341243+0000) 2024-03-06T10:53:52.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:52.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:52.341511+0000) 2024-03-06T10:53:53.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:53.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:53.341811+0000) 2024-03-06T10:53:54.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:54.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:54.342055+0000) 2024-03-06T10:53:55.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:55.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:55.342261+0000) 2024-03-06T10:53:56.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:56.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:56.342480+0000) 2024-03-06T10:53:57.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:57.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:57.342717+0000) 2024-03-06T10:53:58.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:58.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:58.342871+0000) 2024-03-06T10:53:59.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:53:59.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:59.343145+0000) 2024-03-06T10:54:00.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:00.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:00.343426+0000) 2024-03-06T10:54:01.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:01.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:01.343653+0000) 2024-03-06T10:54:02.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:02.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:02.343888+0000) 2024-03-06T10:54:03.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:03.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:03.344094+0000) 2024-03-06T10:54:04.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:04.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:04.344331+0000) 2024-03-06T10:54:05.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:05.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:05.344545+0000) 2024-03-06T10:54:05.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:05.440 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:06.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:06.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:06.344775+0000) 2024-03-06T10:54:07.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:07.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:07.345006+0000) 2024-03-06T10:54:08.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:08.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:08.345205+0000) 2024-03-06T10:54:09.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:09.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:09.345444+0000) 2024-03-06T10:54:10.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:10.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:10.345695+0000) 2024-03-06T10:54:11.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:11.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:11.345913+0000) 2024-03-06T10:54:12.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:12.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:12.346192+0000) 2024-03-06T10:54:13.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:13.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:13.346463+0000) 2024-03-06T10:54:14.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:14.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:14.346772+0000) 2024-03-06T10:54:15.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:15.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:15.347045+0000) 2024-03-06T10:54:16.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:16.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:16.347278+0000) 2024-03-06T10:54:17.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:17.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:17.347553+0000) 2024-03-06T10:54:18.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:18.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:18.347823+0000) 2024-03-06T10:54:19.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:19.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:19.348065+0000) 2024-03-06T10:54:20.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:20.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:20.348316+0000) 2024-03-06T10:54:21.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:21.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:21.348584+0000) 2024-03-06T10:54:22.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:22.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:22.348786+0000) 2024-03-06T10:54:23.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:23.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:23.348966+0000) 2024-03-06T10:54:24.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:24.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:24.349130+0000) 2024-03-06T10:54:25.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:25.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:25.349343+0000) 2024-03-06T10:54:26.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:26.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:26.349585+0000) 2024-03-06T10:54:27.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:27.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:27.349842+0000) 2024-03-06T10:54:28.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:28.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:28.350120+0000) 2024-03-06T10:54:29.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:29.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:29.350389+0000) 2024-03-06T10:54:30.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:30.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:30.350636+0000) 2024-03-06T10:54:31.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:31.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:31.350939+0000) 2024-03-06T10:54:32.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:32.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:32.351221+0000) 2024-03-06T10:54:33.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:33.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:33.351499+0000) 2024-03-06T10:54:34.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:34.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:34.351799+0000) 2024-03-06T10:54:35.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:35.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:35.352076+0000) 2024-03-06T10:54:35.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:35.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:36.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:36.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:36.352369+0000) 2024-03-06T10:54:37.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:37.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:37.352566+0000) 2024-03-06T10:54:38.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:38.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:38.352795+0000) 2024-03-06T10:54:39.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:39.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:39.352983+0000) 2024-03-06T10:54:40.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:40.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:40.353171+0000) 2024-03-06T10:54:41.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:41.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:41.353400+0000) 2024-03-06T10:54:42.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:42.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:42.353593+0000) 2024-03-06T10:54:43.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:43.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:43.353852+0000) 2024-03-06T10:54:44.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:44.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:44.354135+0000) 2024-03-06T10:54:45.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:45.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:45.354405+0000) 2024-03-06T10:54:46.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:46.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:46.354709+0000) 2024-03-06T10:54:47.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:47.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:47.354952+0000) 2024-03-06T10:54:48.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:48.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:48.355218+0000) 2024-03-06T10:54:49.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:49.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:49.355504+0000) 2024-03-06T10:54:50.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:50.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:50.355850+0000) 2024-03-06T10:54:51.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:51.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:51.356097+0000) 2024-03-06T10:54:52.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:52.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:52.356271+0000) 2024-03-06T10:54:53.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:53.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:53.356501+0000) 2024-03-06T10:54:54.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:54.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:54.356765+0000) 2024-03-06T10:54:55.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:55.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:55.356953+0000) 2024-03-06T10:54:56.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:56.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:56.357145+0000) 2024-03-06T10:54:57.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:57.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:57.357330+0000) 2024-03-06T10:54:58.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:58.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:58.357514+0000) 2024-03-06T10:54:59.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:54:59.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:59.357738+0000) 2024-03-06T10:55:00.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:00.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:00.357958+0000) 2024-03-06T10:55:01.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:01.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:01.358117+0000) 2024-03-06T10:55:02.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:02.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:02.358342+0000) 2024-03-06T10:55:03.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:03.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:03.358576+0000) 2024-03-06T10:55:04.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:04.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:04.358786+0000) 2024-03-06T10:55:05.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:05.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:05.359026+0000) 2024-03-06T10:55:05.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:05.857 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:06.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:06.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:06.359236+0000) 2024-03-06T10:55:07.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:07.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:07.359424+0000) 2024-03-06T10:55:08.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:08.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:08.359617+0000) 2024-03-06T10:55:09.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:09.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:09.359851+0000) 2024-03-06T10:55:10.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:10.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:10.360075+0000) 2024-03-06T10:55:11.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:11.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:11.360266+0000) 2024-03-06T10:55:12.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:12.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:12.360492+0000) 2024-03-06T10:55:13.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:13.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:13.360765+0000) 2024-03-06T10:55:14.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:14.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:14.360991+0000) 2024-03-06T10:55:15.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:15.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:15.361211+0000) 2024-03-06T10:55:16.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:16.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:16.361448+0000) 2024-03-06T10:55:17.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:17.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:17.361710+0000) 2024-03-06T10:55:18.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:18.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:18.361941+0000) 2024-03-06T10:55:19.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:19.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:19.362191+0000) 2024-03-06T10:55:20.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:20.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:20.362427+0000) 2024-03-06T10:55:21.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:21.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:21.362620+0000) 2024-03-06T10:55:22.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:22.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:22.362918+0000) 2024-03-06T10:55:23.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:23.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:23.363163+0000) 2024-03-06T10:55:24.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:24.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:24.363408+0000) 2024-03-06T10:55:25.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:25.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:25.363702+0000) 2024-03-06T10:55:26.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:26.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:26.363934+0000) 2024-03-06T10:55:27.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:27.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:27.364132+0000) 2024-03-06T10:55:28.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:28.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:28.364370+0000) 2024-03-06T10:55:29.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:29.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:29.364595+0000) 2024-03-06T10:55:30.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:30.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:30.364803+0000) 2024-03-06T10:55:31.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:31.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:31.365036+0000) 2024-03-06T10:55:32.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:32.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:32.365298+0000) 2024-03-06T10:55:33.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:33.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:33.365580+0000) 2024-03-06T10:55:34.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:34.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:34.365869+0000) 2024-03-06T10:55:35.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:35.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:35.366106+0000) 2024-03-06T10:55:36.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:36.077 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:36.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:36.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:36.366342+0000) 2024-03-06T10:55:37.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:37.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:37.366578+0000) 2024-03-06T10:55:38.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:38.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:38.366828+0000) 2024-03-06T10:55:39.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:39.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:39.367054+0000) 2024-03-06T10:55:40.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:40.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:40.367276+0000) 2024-03-06T10:55:41.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:41.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:41.367513+0000) 2024-03-06T10:55:42.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:42.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:42.367827+0000) 2024-03-06T10:55:43.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:43.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:43.368080+0000) 2024-03-06T10:55:44.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:44.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:44.368297+0000) 2024-03-06T10:55:45.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:45.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:45.368525+0000) 2024-03-06T10:55:46.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:46.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:46.368776+0000) 2024-03-06T10:55:47.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:47.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:47.368984+0000) 2024-03-06T10:55:48.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:48.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:48.369150+0000) 2024-03-06T10:55:49.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:49.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:49.369327+0000) 2024-03-06T10:55:50.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:50.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:50.369518+0000) 2024-03-06T10:55:51.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:51.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:51.369777+0000) 2024-03-06T10:55:52.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:52.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:52.370006+0000) 2024-03-06T10:55:53.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:53.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:53.370234+0000) 2024-03-06T10:55:54.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:54.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:54.370455+0000) 2024-03-06T10:55:55.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:55.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:55.370660+0000) 2024-03-06T10:55:56.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:56.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:56.370933+0000) 2024-03-06T10:55:57.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:57.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:57.371140+0000) 2024-03-06T10:55:58.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:58.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:58.371296+0000) 2024-03-06T10:55:59.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:55:59.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:59.371530+0000) 2024-03-06T10:56:00.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:00.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:00.371775+0000) 2024-03-06T10:56:01.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:01.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:01.372047+0000) 2024-03-06T10:56:02.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:02.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:02.372258+0000) 2024-03-06T10:56:03.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:03.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:03.372437+0000) 2024-03-06T10:56:04.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:04.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:04.372625+0000) 2024-03-06T10:56:05.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:05.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:05.372858+0000) 2024-03-06T10:56:06.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:06.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:06.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:06.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:06.373118+0000) 2024-03-06T10:56:07.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:07.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:07.373393+0000) 2024-03-06T10:56:08.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:08.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:08.373630+0000) 2024-03-06T10:56:09.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:09.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:09.373818+0000) 2024-03-06T10:56:10.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:10.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:10.374055+0000) 2024-03-06T10:56:11.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:11.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:11.374253+0000) 2024-03-06T10:56:12.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:12.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:12.374453+0000) 2024-03-06T10:56:13.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:13.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:13.374642+0000) 2024-03-06T10:56:14.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:14.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:14.374900+0000) 2024-03-06T10:56:15.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:15.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:15.375123+0000) 2024-03-06T10:56:16.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:16.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:16.375352+0000) 2024-03-06T10:56:17.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:17.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:17.375584+0000) 2024-03-06T10:56:18.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:18.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:18.375780+0000) 2024-03-06T10:56:19.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:19.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:19.376010+0000) 2024-03-06T10:56:20.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:20.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:20.376197+0000) 2024-03-06T10:56:21.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:21.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:21.376419+0000) 2024-03-06T10:56:22.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:22.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:22.376641+0000) 2024-03-06T10:56:23.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:23.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:23.376976+0000) 2024-03-06T10:56:24.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:24.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:24.377172+0000) 2024-03-06T10:56:25.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:25.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:25.377401+0000) 2024-03-06T10:56:26.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:26.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:26.377632+0000) 2024-03-06T10:56:27.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:27.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:27.377844+0000) 2024-03-06T10:56:28.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:28.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:28.378041+0000) 2024-03-06T10:56:29.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:29.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:29.378250+0000) 2024-03-06T10:56:30.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:30.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:30.378486+0000) 2024-03-06T10:56:31.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:31.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:31.378754+0000) 2024-03-06T10:56:32.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:32.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:32.379001+0000) 2024-03-06T10:56:33.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:33.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:33.379203+0000) 2024-03-06T10:56:34.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:34.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:34.379408+0000) 2024-03-06T10:56:35.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:35.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:35.379607+0000) 2024-03-06T10:56:36.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:36.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:36.379946+0000) 2024-03-06T10:56:36.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:36.460 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:37.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:37.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:37.380216+0000) 2024-03-06T10:56:38.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:38.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:38.380499+0000) 2024-03-06T10:56:39.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:39.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:39.380778+0000) 2024-03-06T10:56:40.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:40.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:40.381001+0000) 2024-03-06T10:56:41.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:41.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:41.381177+0000) 2024-03-06T10:56:42.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:42.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:42.381407+0000) 2024-03-06T10:56:43.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:43.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:43.381706+0000) 2024-03-06T10:56:44.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:44.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:44.381906+0000) 2024-03-06T10:56:45.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:45.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:45.382061+0000) 2024-03-06T10:56:46.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:46.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:46.382291+0000) 2024-03-06T10:56:47.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:47.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:47.382562+0000) 2024-03-06T10:56:48.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:48.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:48.382808+0000) 2024-03-06T10:56:49.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:49.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:49.383024+0000) 2024-03-06T10:56:50.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:50.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:50.383306+0000) 2024-03-06T10:56:51.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:51.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:51.383570+0000) 2024-03-06T10:56:52.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:52.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:52.383846+0000) 2024-03-06T10:56:53.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:53.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:53.384062+0000) 2024-03-06T10:56:54.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:54.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:54.384284+0000) 2024-03-06T10:56:55.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:55.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:55.384500+0000) 2024-03-06T10:56:56.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:56.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:56.384765+0000) 2024-03-06T10:56:57.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:57.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:57.384999+0000) 2024-03-06T10:56:58.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:58.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:58.385219+0000) 2024-03-06T10:56:59.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:56:59.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:59.385393+0000) 2024-03-06T10:57:00.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:00.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:00.385620+0000) 2024-03-06T10:57:01.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:01.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:01.385844+0000) 2024-03-06T10:57:02.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:02.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:02.386073+0000) 2024-03-06T10:57:03.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:03.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:03.386326+0000) 2024-03-06T10:57:04.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:04.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:04.386559+0000) 2024-03-06T10:57:05.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:05.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:05.386835+0000) 2024-03-06T10:57:06.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:06.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:06.387039+0000) 2024-03-06T10:57:06.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:06.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:07.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:07.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:07.387266+0000) 2024-03-06T10:57:08.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:08.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:08.387454+0000) 2024-03-06T10:57:09.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:09.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:09.387692+0000) 2024-03-06T10:57:10.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:10.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:10.387913+0000) 2024-03-06T10:57:11.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:11.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:11.388134+0000) 2024-03-06T10:57:12.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:12.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:12.388310+0000) 2024-03-06T10:57:13.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:13.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:13.388531+0000) 2024-03-06T10:57:14.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:14.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:14.388780+0000) 2024-03-06T10:57:15.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:15.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:15.389001+0000) 2024-03-06T10:57:16.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:16.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:16.389228+0000) 2024-03-06T10:57:17.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:17.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:17.389473+0000) 2024-03-06T10:57:18.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:18.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:18.389718+0000) 2024-03-06T10:57:19.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:19.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:19.389925+0000) 2024-03-06T10:57:20.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:20.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:20.390108+0000) 2024-03-06T10:57:21.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:21.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:21.390333+0000) 2024-03-06T10:57:22.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:22.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:22.390564+0000) 2024-03-06T10:57:23.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:23.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:23.390798+0000) 2024-03-06T10:57:24.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:24.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:24.390989+0000) 2024-03-06T10:57:25.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:25.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:25.391251+0000) 2024-03-06T10:57:26.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:26.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:26.391481+0000) 2024-03-06T10:57:27.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:27.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:27.391703+0000) 2024-03-06T10:57:28.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:28.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:28.391932+0000) 2024-03-06T10:57:29.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:29.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:29.392126+0000) 2024-03-06T10:57:30.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:30.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:30.392353+0000) 2024-03-06T10:57:31.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:31.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:31.392569+0000) 2024-03-06T10:57:32.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:32.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:32.392780+0000) 2024-03-06T10:57:33.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:33.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:33.393002+0000) 2024-03-06T10:57:34.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:34.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:34.393196+0000) 2024-03-06T10:57:35.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:35.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:35.393388+0000) 2024-03-06T10:57:36.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:36.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:36.393583+0000) 2024-03-06T10:57:36.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:36.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:37.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:37.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:37.393848+0000) 2024-03-06T10:57:38.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:38.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:38.394110+0000) 2024-03-06T10:57:39.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:39.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:39.394354+0000) 2024-03-06T10:57:40.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:40.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:40.394598+0000) 2024-03-06T10:57:41.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:41.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:41.394823+0000) 2024-03-06T10:57:42.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:42.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:42.394933+0000) 2024-03-06T10:57:43.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:43.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:43.395117+0000) 2024-03-06T10:57:44.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:44.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:44.395306+0000) 2024-03-06T10:57:45.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:45.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:45.395502+0000) 2024-03-06T10:57:46.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:46.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:46.395731+0000) 2024-03-06T10:57:47.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:47.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:47.395965+0000) 2024-03-06T10:57:48.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:48.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:48.396165+0000) 2024-03-06T10:57:49.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:49.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:49.396400+0000) 2024-03-06T10:57:50.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:50.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:50.396565+0000) 2024-03-06T10:57:51.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:51.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:51.396721+0000) 2024-03-06T10:57:52.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:52.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:52.396962+0000) 2024-03-06T10:57:53.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:53.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:53.397149+0000) 2024-03-06T10:57:54.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:54.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:54.397361+0000) 2024-03-06T10:57:55.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:55.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:55.397584+0000) 2024-03-06T10:57:56.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:56.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:56.397816+0000) 2024-03-06T10:57:57.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:57.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:57.398060+0000) 2024-03-06T10:57:58.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:58.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:58.398279+0000) 2024-03-06T10:57:59.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:57:59.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:59.398498+0000) 2024-03-06T10:58:00.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:00.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:00.398763+0000) 2024-03-06T10:58:01.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:01.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:01.398997+0000) 2024-03-06T10:58:02.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:02.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:02.399233+0000) 2024-03-06T10:58:03.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:03.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:03.399424+0000) 2024-03-06T10:58:04.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:04.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:04.399640+0000) 2024-03-06T10:58:05.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:05.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:05.399903+0000) 2024-03-06T10:58:06.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:06.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:06.400132+0000) 2024-03-06T10:58:07.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:07.083 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:07.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:07.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:07.400373+0000) 2024-03-06T10:58:08.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:08.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:08.400595+0000) 2024-03-06T10:58:09.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:09.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:09.400833+0000) 2024-03-06T10:58:10.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:10.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:10.401019+0000) 2024-03-06T10:58:11.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:11.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:11.401244+0000) 2024-03-06T10:58:12.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:12.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:12.401474+0000) 2024-03-06T10:58:13.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:13.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:13.401796+0000) 2024-03-06T10:58:14.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:14.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:14.401983+0000) 2024-03-06T10:58:15.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:15.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:15.402171+0000) 2024-03-06T10:58:16.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:16.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:16.402425+0000) 2024-03-06T10:58:17.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:17.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:17.402638+0000) 2024-03-06T10:58:18.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:18.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:18.402920+0000) 2024-03-06T10:58:19.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:19.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:19.403163+0000) 2024-03-06T10:58:20.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:20.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:20.403367+0000) 2024-03-06T10:58:21.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:21.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:21.403603+0000) 2024-03-06T10:58:22.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:22.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:22.403813+0000) 2024-03-06T10:58:23.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:23.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:23.404046+0000) 2024-03-06T10:58:24.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:24.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:24.404264+0000) 2024-03-06T10:58:25.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:25.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:25.404487+0000) 2024-03-06T10:58:26.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:26.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:26.404722+0000) 2024-03-06T10:58:27.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:27.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:27.404960+0000) 2024-03-06T10:58:28.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:28.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:28.405180+0000) 2024-03-06T10:58:29.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:29.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:29.405403+0000) 2024-03-06T10:58:30.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:30.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:30.405617+0000) 2024-03-06T10:58:31.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:31.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:31.405841+0000) 2024-03-06T10:58:32.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:32.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:32.406062+0000) 2024-03-06T10:58:33.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:33.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:33.406296+0000) 2024-03-06T10:58:34.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:34.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:34.406515+0000) 2024-03-06T10:58:35.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:35.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:35.406787+0000) 2024-03-06T10:58:36.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:36.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:36.407018+0000) 2024-03-06T10:58:37.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:37.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:37.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:37.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:37.407255+0000) 2024-03-06T10:58:38.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:38.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:38.407476+0000) 2024-03-06T10:58:39.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:39.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:39.407735+0000) 2024-03-06T10:58:40.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:40.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:40.407924+0000) 2024-03-06T10:58:41.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:41.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:41.408115+0000) 2024-03-06T10:58:42.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:42.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:42.408345+0000) 2024-03-06T10:58:43.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:43.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:43.408531+0000) 2024-03-06T10:58:44.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:44.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:44.408727+0000) 2024-03-06T10:58:45.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:45.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:45.408949+0000) 2024-03-06T10:58:46.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:46.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:46.409180+0000) 2024-03-06T10:58:47.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:47.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:47.409408+0000) 2024-03-06T10:58:48.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:48.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:48.409628+0000) 2024-03-06T10:58:49.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:49.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:49.409826+0000) 2024-03-06T10:58:50.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:50.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:50.410049+0000) 2024-03-06T10:58:51.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:51.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:51.410272+0000) 2024-03-06T10:58:52.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:52.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:52.410495+0000) 2024-03-06T10:58:53.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:53.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:53.410789+0000) 2024-03-06T10:58:54.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:54.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:54.411018+0000) 2024-03-06T10:58:55.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:55.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:55.411245+0000) 2024-03-06T10:58:56.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:56.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:56.411444+0000) 2024-03-06T10:58:57.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:57.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:57.411694+0000) 2024-03-06T10:58:58.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:58.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:58.411880+0000) 2024-03-06T10:58:59.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:58:59.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:59.412109+0000) 2024-03-06T10:59:00.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:00.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:00.412321+0000) 2024-03-06T10:59:01.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:01.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:01.412546+0000) 2024-03-06T10:59:02.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:02.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:02.412770+0000) 2024-03-06T10:59:03.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:03.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:03.412968+0000) 2024-03-06T10:59:04.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:04.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:04.413189+0000) 2024-03-06T10:59:05.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:05.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:05.413429+0000) 2024-03-06T10:59:06.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:06.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:06.413627+0000) 2024-03-06T10:59:07.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:07.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:07.413846+0000) 2024-03-06T10:59:07.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:07.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:08.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:08.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:08.414030+0000) 2024-03-06T10:59:09.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:09.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:09.414230+0000) 2024-03-06T10:59:10.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:10.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:10.414418+0000) 2024-03-06T10:59:11.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:11.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:11.414752+0000) 2024-03-06T10:59:12.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:12.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:12.414936+0000) 2024-03-06T10:59:13.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:13.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:13.415186+0000) 2024-03-06T10:59:14.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:14.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:14.415397+0000) 2024-03-06T10:59:15.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:15.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:15.415604+0000) 2024-03-06T10:59:16.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:16.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:16.415852+0000) 2024-03-06T10:59:17.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:17.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:17.416048+0000) 2024-03-06T10:59:18.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:18.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:18.416282+0000) 2024-03-06T10:59:19.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:19.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:19.416506+0000) 2024-03-06T10:59:20.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:20.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:20.416770+0000) 2024-03-06T10:59:21.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:21.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:21.416973+0000) 2024-03-06T10:59:22.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:22.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:22.417199+0000) 2024-03-06T10:59:23.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:23.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:23.417384+0000) 2024-03-06T10:59:24.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:24.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:24.417617+0000) 2024-03-06T10:59:25.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:25.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:25.417851+0000) 2024-03-06T10:59:26.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:26.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:26.418082+0000) 2024-03-06T10:59:27.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:27.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:27.418308+0000) 2024-03-06T10:59:28.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:28.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:28.418530+0000) 2024-03-06T10:59:29.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:29.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:29.418788+0000) 2024-03-06T10:59:30.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:30.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:30.419010+0000) 2024-03-06T10:59:31.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:31.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:31.419242+0000) 2024-03-06T10:59:32.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:32.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:32.419463+0000) 2024-03-06T10:59:33.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:33.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:33.419711+0000) 2024-03-06T10:59:34.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:34.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:34.419945+0000) 2024-03-06T10:59:35.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:35.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:35.420178+0000) 2024-03-06T10:59:36.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:36.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:36.420363+0000) 2024-03-06T10:59:37.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:37.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:37.420551+0000) 2024-03-06T10:59:37.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:37.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:38.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:38.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:38.420781+0000) 2024-03-06T10:59:39.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:39.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:39.420970+0000) 2024-03-06T10:59:40.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:40.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:40.421187+0000) 2024-03-06T10:59:41.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:41.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:41.421407+0000) 2024-03-06T10:59:42.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:42.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:42.421632+0000) 2024-03-06T10:59:43.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:43.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:43.421870+0000) 2024-03-06T10:59:44.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:44.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:44.422088+0000) 2024-03-06T10:59:45.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:45.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:45.422268+0000) 2024-03-06T10:59:46.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:46.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:46.422459+0000) 2024-03-06T10:59:47.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:47.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:47.422716+0000) 2024-03-06T10:59:48.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:48.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:48.422940+0000) 2024-03-06T10:59:49.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:49.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:49.423171+0000) 2024-03-06T10:59:50.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:50.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:50.423397+0000) 2024-03-06T10:59:51.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:51.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:51.423566+0000) 2024-03-06T10:59:52.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:52.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:52.423795+0000) 2024-03-06T10:59:53.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:53.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:53.424041+0000) 2024-03-06T10:59:54.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:54.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:54.424285+0000) 2024-03-06T10:59:55.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:55.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:55.424520+0000) 2024-03-06T10:59:56.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:56.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:56.424771+0000) 2024-03-06T10:59:57.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:57.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:57.425009+0000) 2024-03-06T10:59:58.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:58.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:58.425247+0000) 2024-03-06T10:59:59.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T10:59:59.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:59.425542+0000) 2024-03-06T11:00:00.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:00.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:00.425792+0000) 2024-03-06T11:00:01.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:01.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:01.425991+0000) 2024-03-06T11:00:02.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:02.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:02.426214+0000) 2024-03-06T11:00:03.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:03.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:03.426446+0000) 2024-03-06T11:00:04.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:04.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:04.426715+0000) 2024-03-06T11:00:05.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:05.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:05.426967+0000) 2024-03-06T11:00:06.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:06.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:06.427200+0000) 2024-03-06T11:00:07.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:07.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:07.427385+0000) 2024-03-06T11:00:07.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:07.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:08.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:08.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:08.427650+0000) 2024-03-06T11:00:09.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:09.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:09.428001+0000) 2024-03-06T11:00:10.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:10.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:10.428225+0000) 2024-03-06T11:00:11.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:11.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:11.428513+0000) 2024-03-06T11:00:12.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:12.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:12.428826+0000) 2024-03-06T11:00:13.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:13.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:13.429112+0000) 2024-03-06T11:00:14.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:14.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:14.429331+0000) 2024-03-06T11:00:15.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:15.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:15.429546+0000) 2024-03-06T11:00:16.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:16.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:16.429775+0000) 2024-03-06T11:00:17.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:17.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:17.430004+0000) 2024-03-06T11:00:18.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:18.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:18.430208+0000) 2024-03-06T11:00:19.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:19.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:19.430400+0000) 2024-03-06T11:00:20.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:20.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:20.430620+0000) 2024-03-06T11:00:21.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:21.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:21.430832+0000) 2024-03-06T11:00:22.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:22.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:22.431037+0000) 2024-03-06T11:00:23.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:23.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:23.431221+0000) 2024-03-06T11:00:24.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:24.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:24.431400+0000) 2024-03-06T11:00:25.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:25.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:25.431635+0000) 2024-03-06T11:00:26.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:26.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:26.431908+0000) 2024-03-06T11:00:27.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:27.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:27.432190+0000) 2024-03-06T11:00:28.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:28.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:28.432395+0000) 2024-03-06T11:00:29.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:29.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:29.432618+0000) 2024-03-06T11:00:30.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:30.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:30.432846+0000) 2024-03-06T11:00:31.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:31.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:31.433041+0000) 2024-03-06T11:00:32.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:32.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:32.433268+0000) 2024-03-06T11:00:33.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:33.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:33.433500+0000) 2024-03-06T11:00:34.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:34.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:34.433707+0000) 2024-03-06T11:00:35.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:35.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:35.433947+0000) 2024-03-06T11:00:36.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:36.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:36.434188+0000) 2024-03-06T11:00:37.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:37.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:37.434417+0000) 2024-03-06T11:00:38.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:38.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:38.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:38.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:38.434589+0000) 2024-03-06T11:00:39.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:39.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:39.434840+0000) 2024-03-06T11:00:40.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:40.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:40.435059+0000) 2024-03-06T11:00:41.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:41.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:41.435303+0000) 2024-03-06T11:00:42.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:42.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:42.435575+0000) 2024-03-06T11:00:43.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:43.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:43.435858+0000) 2024-03-06T11:00:44.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:44.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:44.436051+0000) 2024-03-06T11:00:45.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:45.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:45.436267+0000) 2024-03-06T11:00:46.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:46.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:46.436495+0000) 2024-03-06T11:00:47.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:47.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:47.436770+0000) 2024-03-06T11:00:48.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:48.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:48.437039+0000) 2024-03-06T11:00:49.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:49.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:49.437331+0000) 2024-03-06T11:00:50.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:50.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:50.437611+0000) 2024-03-06T11:00:51.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:51.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:51.437889+0000) 2024-03-06T11:00:52.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:52.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:52.438181+0000) 2024-03-06T11:00:53.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:53.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:53.438452+0000) 2024-03-06T11:00:54.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:54.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:54.438724+0000) 2024-03-06T11:00:55.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:55.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:55.438949+0000) 2024-03-06T11:00:56.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:56.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:56.439219+0000) 2024-03-06T11:00:57.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:57.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:57.439500+0000) 2024-03-06T11:00:58.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:58.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:58.439792+0000) 2024-03-06T11:00:59.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:00:59.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:59.440064+0000) 2024-03-06T11:01:00.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:00.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:00.440340+0000) 2024-03-06T11:01:01.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:01.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:01.440530+0000) 2024-03-06T11:01:02.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:02.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:02.440769+0000) 2024-03-06T11:01:03.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:03.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:03.441021+0000) 2024-03-06T11:01:04.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:04.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:04.441252+0000) 2024-03-06T11:01:05.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:05.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:05.441496+0000) 2024-03-06T11:01:06.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:06.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:06.441780+0000) 2024-03-06T11:01:07.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:07.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:07.442020+0000) 2024-03-06T11:01:08.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:08.320 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:08.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:08.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:08.442209+0000) 2024-03-06T11:01:09.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:09.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:09.442447+0000) 2024-03-06T11:01:10.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:10.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:10.442698+0000) 2024-03-06T11:01:11.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:11.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:11.442942+0000) 2024-03-06T11:01:12.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:12.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:12.443175+0000) 2024-03-06T11:01:13.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:13.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:13.443460+0000) 2024-03-06T11:01:14.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:14.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:14.443762+0000) 2024-03-06T11:01:15.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:15.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:15.444035+0000) 2024-03-06T11:01:16.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:16.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:16.444325+0000) 2024-03-06T11:01:17.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:17.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:17.444584+0000) 2024-03-06T11:01:18.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:18.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:18.444773+0000) 2024-03-06T11:01:19.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:19.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:19.444966+0000) 2024-03-06T11:01:20.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:20.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:20.445201+0000) 2024-03-06T11:01:21.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:21.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:21.445395+0000) 2024-03-06T11:01:22.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:22.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:22.445638+0000) 2024-03-06T11:01:23.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:23.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:23.445919+0000) 2024-03-06T11:01:24.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:24.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:24.446163+0000) 2024-03-06T11:01:25.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:25.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:25.446369+0000) 2024-03-06T11:01:26.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:26.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:26.446618+0000) 2024-03-06T11:01:27.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:27.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:27.446866+0000) 2024-03-06T11:01:28.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:28.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:28.447053+0000) 2024-03-06T11:01:29.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:29.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:29.447320+0000) 2024-03-06T11:01:30.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:30.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:30.447565+0000) 2024-03-06T11:01:31.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:31.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:31.447841+0000) 2024-03-06T11:01:32.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:32.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:32.448027+0000) 2024-03-06T11:01:33.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:33.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:33.448283+0000) 2024-03-06T11:01:34.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:34.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:34.448523+0000) 2024-03-06T11:01:35.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:35.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:35.448795+0000) 2024-03-06T11:01:36.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:36.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:36.449046+0000) 2024-03-06T11:01:37.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:37.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:37.449281+0000) 2024-03-06T11:01:38.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:38.455 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:38.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:38.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:38.449477+0000) 2024-03-06T11:01:39.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:39.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:39.449791+0000) 2024-03-06T11:01:40.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:40.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:40.450070+0000) 2024-03-06T11:01:41.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:41.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:41.450391+0000) 2024-03-06T11:01:42.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:42.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:42.450589+0000) 2024-03-06T11:01:43.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:43.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:43.450857+0000) 2024-03-06T11:01:44.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:44.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:44.451133+0000) 2024-03-06T11:01:45.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:45.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:45.451415+0000) 2024-03-06T11:01:46.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:46.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:46.451746+0000) 2024-03-06T11:01:47.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:47.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:47.452020+0000) 2024-03-06T11:01:48.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:48.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:48.452295+0000) 2024-03-06T11:01:49.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:49.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:49.452560+0000) 2024-03-06T11:01:50.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:50.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:50.452842+0000) 2024-03-06T11:01:51.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:51.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:51.453120+0000) 2024-03-06T11:01:52.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:52.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:52.453351+0000) 2024-03-06T11:01:53.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:53.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:53.453629+0000) 2024-03-06T11:01:54.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:54.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:54.453955+0000) 2024-03-06T11:01:55.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:55.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:55.454155+0000) 2024-03-06T11:01:56.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:56.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:56.454402+0000) 2024-03-06T11:01:57.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:57.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:57.454626+0000) 2024-03-06T11:01:58.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:58.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:58.454866+0000) 2024-03-06T11:01:59.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:01:59.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:59.455074+0000) 2024-03-06T11:02:00.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:00.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:00.455306+0000) 2024-03-06T11:02:01.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:01.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:01.455497+0000) 2024-03-06T11:02:02.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:02.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:02.455748+0000) 2024-03-06T11:02:03.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:03.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:03.455987+0000) 2024-03-06T11:02:04.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:04.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:04.456242+0000) 2024-03-06T11:02:05.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:05.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:05.456513+0000) 2024-03-06T11:02:06.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:06.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:06.456813+0000) 2024-03-06T11:02:07.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:07.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:07.457095+0000) 2024-03-06T11:02:08.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:08.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:08.457295+0000) 2024-03-06T11:02:08.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:08.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:09.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:09.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:09.457487+0000) 2024-03-06T11:02:10.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:10.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:10.457754+0000) 2024-03-06T11:02:11.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:11.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:11.458005+0000) 2024-03-06T11:02:12.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:12.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:12.458208+0000) 2024-03-06T11:02:13.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:13.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:13.458451+0000) 2024-03-06T11:02:14.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:14.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:14.458641+0000) 2024-03-06T11:02:15.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:15.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:15.458912+0000) 2024-03-06T11:02:16.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:16.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:16.459135+0000) 2024-03-06T11:02:17.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:17.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:17.459360+0000) 2024-03-06T11:02:18.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:18.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:18.459584+0000) 2024-03-06T11:02:19.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:19.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:19.459834+0000) 2024-03-06T11:02:20.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:20.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:20.460072+0000) 2024-03-06T11:02:21.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:21.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:21.460273+0000) 2024-03-06T11:02:22.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:22.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:22.460494+0000) 2024-03-06T11:02:23.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:23.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:23.460809+0000) 2024-03-06T11:02:24.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:24.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:24.461091+0000) 2024-03-06T11:02:25.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:25.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:25.461300+0000) 2024-03-06T11:02:26.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:26.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:26.461558+0000) 2024-03-06T11:02:27.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:27.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:27.461832+0000) 2024-03-06T11:02:28.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:28.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:28.462116+0000) 2024-03-06T11:02:29.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:29.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:29.462329+0000) 2024-03-06T11:02:30.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:30.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:30.462603+0000) 2024-03-06T11:02:31.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:31.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:31.462867+0000) 2024-03-06T11:02:32.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:32.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:32.463151+0000) 2024-03-06T11:02:33.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:33.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:33.463441+0000) 2024-03-06T11:02:34.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:34.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:34.463648+0000) 2024-03-06T11:02:35.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:35.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:35.463967+0000) 2024-03-06T11:02:36.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:36.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:36.464210+0000) 2024-03-06T11:02:37.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:37.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:37.464423+0000) 2024-03-06T11:02:38.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:38.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:38.464695+0000) 2024-03-06T11:02:38.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:38.881 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:39.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:39.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:39.464986+0000) 2024-03-06T11:02:40.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:40.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:40.465222+0000) 2024-03-06T11:02:41.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:41.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:41.465508+0000) 2024-03-06T11:02:42.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:42.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:42.465818+0000) 2024-03-06T11:02:43.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:43.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:43.466051+0000) 2024-03-06T11:02:44.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:44.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:44.466324+0000) 2024-03-06T11:02:45.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:45.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:45.466524+0000) 2024-03-06T11:02:46.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:46.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:46.466785+0000) 2024-03-06T11:02:47.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:47.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:47.467020+0000) 2024-03-06T11:02:48.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:48.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:48.467251+0000) 2024-03-06T11:02:49.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:49.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:49.467477+0000) 2024-03-06T11:02:50.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:50.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:50.467733+0000) 2024-03-06T11:02:51.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:51.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:51.467958+0000) 2024-03-06T11:02:52.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:52.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:52.468179+0000) 2024-03-06T11:02:53.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:53.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:53.468373+0000) 2024-03-06T11:02:54.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:54.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:54.468629+0000) 2024-03-06T11:02:55.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:55.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:55.468872+0000) 2024-03-06T11:02:56.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:56.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:56.469103+0000) 2024-03-06T11:02:57.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:57.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:57.469331+0000) 2024-03-06T11:02:58.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:58.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:58.469494+0000) 2024-03-06T11:02:59.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:02:59.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:59.469761+0000) 2024-03-06T11:03:00.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:00.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:00.469977+0000) 2024-03-06T11:03:01.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:01.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:01.470244+0000) 2024-03-06T11:03:02.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:02.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:02.470521+0000) 2024-03-06T11:03:03.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:03.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:03.470834+0000) 2024-03-06T11:03:04.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:04.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:04.471094+0000) 2024-03-06T11:03:05.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:05.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:05.471334+0000) 2024-03-06T11:03:06.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:06.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:06.471593+0000) 2024-03-06T11:03:07.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:07.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:07.471862+0000) 2024-03-06T11:03:08.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:08.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:08.472129+0000) 2024-03-06T11:03:09.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:09.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:09.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:09.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:09.472359+0000) 2024-03-06T11:03:10.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:10.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:10.472551+0000) 2024-03-06T11:03:11.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:11.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:11.472794+0000) 2024-03-06T11:03:12.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:12.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:12.473026+0000) 2024-03-06T11:03:13.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:13.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:13.473264+0000) 2024-03-06T11:03:14.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:14.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:14.473489+0000) 2024-03-06T11:03:15.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:15.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:15.473766+0000) 2024-03-06T11:03:16.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:16.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:16.473985+0000) 2024-03-06T11:03:17.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:17.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:17.474262+0000) 2024-03-06T11:03:18.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:18.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:18.474502+0000) 2024-03-06T11:03:19.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:19.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:19.474804+0000) 2024-03-06T11:03:20.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:20.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:20.475074+0000) 2024-03-06T11:03:21.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:21.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:21.475361+0000) 2024-03-06T11:03:22.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:22.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:22.475631+0000) 2024-03-06T11:03:23.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:23.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:23.475984+0000) 2024-03-06T11:03:24.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:24.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:24.476252+0000) 2024-03-06T11:03:25.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:25.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:25.476520+0000) 2024-03-06T11:03:26.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:26.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:26.476812+0000) 2024-03-06T11:03:27.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:27.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:27.477245+0000) 2024-03-06T11:03:28.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:28.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:28.477526+0000) 2024-03-06T11:03:29.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:29.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:29.477813+0000) 2024-03-06T11:03:30.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:30.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:30.478007+0000) 2024-03-06T11:03:31.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:31.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:31.478225+0000) 2024-03-06T11:03:32.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:32.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:32.478447+0000) 2024-03-06T11:03:33.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:33.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:33.478777+0000) 2024-03-06T11:03:34.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:34.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:34.478972+0000) 2024-03-06T11:03:35.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:35.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:35.479196+0000) 2024-03-06T11:03:36.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:36.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:36.479429+0000) 2024-03-06T11:03:37.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:37.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:37.479645+0000) 2024-03-06T11:03:38.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:38.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:38.479987+0000) 2024-03-06T11:03:39.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:39.280 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:39.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:39.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:39.480195+0000) 2024-03-06T11:03:40.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:40.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:40.480386+0000) 2024-03-06T11:03:41.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:41.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:41.480622+0000) 2024-03-06T11:03:42.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:42.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:42.480845+0000) 2024-03-06T11:03:43.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:43.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:43.481100+0000) 2024-03-06T11:03:44.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:44.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:44.481345+0000) 2024-03-06T11:03:45.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:45.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:45.481597+0000) 2024-03-06T11:03:46.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:46.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:46.481816+0000) 2024-03-06T11:03:47.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:47.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:47.482046+0000) 2024-03-06T11:03:48.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:48.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:48.482292+0000) 2024-03-06T11:03:49.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:49.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:49.482539+0000) 2024-03-06T11:03:50.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:50.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:50.482778+0000) 2024-03-06T11:03:51.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:51.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:51.482993+0000) 2024-03-06T11:03:52.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:52.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:52.483176+0000) 2024-03-06T11:03:53.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:53.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:53.483401+0000) 2024-03-06T11:03:54.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:54.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:54.483641+0000) 2024-03-06T11:03:55.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:55.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:55.483926+0000) 2024-03-06T11:03:56.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:56.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:56.484172+0000) 2024-03-06T11:03:57.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:57.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:57.484416+0000) 2024-03-06T11:03:58.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:58.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:58.484708+0000) 2024-03-06T11:03:59.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:03:59.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:59.484999+0000) 2024-03-06T11:04:00.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:00.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:00.485246+0000) 2024-03-06T11:04:01.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:01.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:01.485487+0000) 2024-03-06T11:04:02.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:02.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:02.485749+0000) 2024-03-06T11:04:03.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:03.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:03.485954+0000) 2024-03-06T11:04:04.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:04.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:04.486160+0000) 2024-03-06T11:04:05.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:05.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:05.486363+0000) 2024-03-06T11:04:06.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:06.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:06.486571+0000) 2024-03-06T11:04:07.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:07.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:07.486825+0000) 2024-03-06T11:04:08.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:08.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:08.487000+0000) 2024-03-06T11:04:09.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:09.494 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:09.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:09.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:09.487256+0000) 2024-03-06T11:04:10.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:10.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:10.487479+0000) 2024-03-06T11:04:11.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:11.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:11.487766+0000) 2024-03-06T11:04:12.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:12.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:12.488002+0000) 2024-03-06T11:04:13.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:13.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:13.488240+0000) 2024-03-06T11:04:14.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:14.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:14.488482+0000) 2024-03-06T11:04:15.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:15.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:15.488810+0000) 2024-03-06T11:04:16.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:16.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:16.489055+0000) 2024-03-06T11:04:17.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:17.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:17.489300+0000) 2024-03-06T11:04:18.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:18.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:18.489542+0000) 2024-03-06T11:04:19.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:19.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:19.489785+0000) 2024-03-06T11:04:20.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:20.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:20.489982+0000) 2024-03-06T11:04:21.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:21.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:21.490171+0000) 2024-03-06T11:04:22.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:22.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:22.490361+0000) 2024-03-06T11:04:23.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:23.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:23.490587+0000) 2024-03-06T11:04:24.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:24.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:24.490794+0000) 2024-03-06T11:04:25.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:25.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:25.490992+0000) 2024-03-06T11:04:26.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:26.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:26.491234+0000) 2024-03-06T11:04:27.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:27.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:27.491455+0000) 2024-03-06T11:04:28.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:28.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:28.491704+0000) 2024-03-06T11:04:29.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:29.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:29.491941+0000) 2024-03-06T11:04:30.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:30.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:30.492166+0000) 2024-03-06T11:04:31.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:31.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:31.492398+0000) 2024-03-06T11:04:32.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:32.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:32.492616+0000) 2024-03-06T11:04:33.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:33.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:33.492832+0000) 2024-03-06T11:04:34.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:34.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:34.493061+0000) 2024-03-06T11:04:35.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:35.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:35.493299+0000) 2024-03-06T11:04:36.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:36.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:36.493518+0000) 2024-03-06T11:04:37.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:37.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:37.493766+0000) 2024-03-06T11:04:38.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:38.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:38.493962+0000) 2024-03-06T11:04:39.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:39.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:39.494151+0000) 2024-03-06T11:04:39.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:39.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:40.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:40.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:40.494385+0000) 2024-03-06T11:04:41.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:41.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:41.494580+0000) 2024-03-06T11:04:42.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:42.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:42.494828+0000) 2024-03-06T11:04:43.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:43.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:43.495082+0000) 2024-03-06T11:04:44.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:44.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:44.495272+0000) 2024-03-06T11:04:45.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:45.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:45.495511+0000) 2024-03-06T11:04:46.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:46.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:46.495790+0000) 2024-03-06T11:04:47.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:47.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:47.496027+0000) 2024-03-06T11:04:48.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:48.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:48.496259+0000) 2024-03-06T11:04:49.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:49.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:49.496452+0000) 2024-03-06T11:04:50.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:50.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:50.496639+0000) 2024-03-06T11:04:51.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:51.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:51.496903+0000) 2024-03-06T11:04:52.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:52.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:52.497115+0000) 2024-03-06T11:04:53.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:53.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:53.497320+0000) 2024-03-06T11:04:54.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:54.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:54.497572+0000) 2024-03-06T11:04:55.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:55.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:55.497794+0000) 2024-03-06T11:04:56.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:56.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:56.498006+0000) 2024-03-06T11:04:57.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:57.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:57.498205+0000) 2024-03-06T11:04:58.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:58.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:58.498380+0000) 2024-03-06T11:04:59.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:04:59.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:59.498602+0000) 2024-03-06T11:05:00.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:00.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:00.498834+0000) 2024-03-06T11:05:01.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:01.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:01.499059+0000) 2024-03-06T11:05:02.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:02.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:02.499280+0000) 2024-03-06T11:05:03.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:03.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:03.499511+0000) 2024-03-06T11:05:04.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:04.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:04.499765+0000) 2024-03-06T11:05:05.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:05.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:05.499968+0000) 2024-03-06T11:05:06.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:06.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:06.500252+0000) 2024-03-06T11:05:07.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:07.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:07.500531+0000) 2024-03-06T11:05:08.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:08.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:08.500811+0000) 2024-03-06T11:05:09.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:09.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:09.501099+0000) 2024-03-06T11:05:09.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:09.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:10.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:10.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:10.501322+0000) 2024-03-06T11:05:11.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:11.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:11.501502+0000) 2024-03-06T11:05:12.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:12.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:12.501783+0000) 2024-03-06T11:05:13.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:13.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:13.502025+0000) 2024-03-06T11:05:14.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:14.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:14.502278+0000) 2024-03-06T11:05:15.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:15.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:15.502520+0000) 2024-03-06T11:05:16.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:16.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:16.502769+0000) 2024-03-06T11:05:17.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:17.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:17.502958+0000) 2024-03-06T11:05:18.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:18.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:18.503178+0000) 2024-03-06T11:05:19.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:19.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:19.503379+0000) 2024-03-06T11:05:20.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:20.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:20.503621+0000) 2024-03-06T11:05:21.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:21.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:21.503850+0000) 2024-03-06T11:05:22.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:22.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:22.504011+0000) 2024-03-06T11:05:23.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:23.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:23.504247+0000) 2024-03-06T11:05:24.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:24.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:24.504479+0000) 2024-03-06T11:05:25.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:25.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:25.504752+0000) 2024-03-06T11:05:26.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:26.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:26.504989+0000) 2024-03-06T11:05:27.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:27.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:27.505214+0000) 2024-03-06T11:05:28.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:28.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:28.505436+0000) 2024-03-06T11:05:29.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:29.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:29.505624+0000) 2024-03-06T11:05:30.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:30.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:30.505924+0000) 2024-03-06T11:05:31.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:31.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:31.506127+0000) 2024-03-06T11:05:32.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:32.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:32.506340+0000) 2024-03-06T11:05:33.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:33.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:33.506535+0000) 2024-03-06T11:05:34.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:34.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:34.506757+0000) 2024-03-06T11:05:35.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:35.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:35.507000+0000) 2024-03-06T11:05:36.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:36.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:36.507202+0000) 2024-03-06T11:05:37.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:37.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:37.507448+0000) 2024-03-06T11:05:38.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:38.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:38.507696+0000) 2024-03-06T11:05:39.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:39.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:39.507935+0000) 2024-03-06T11:05:40.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:40.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:40.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:40.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:40.508149+0000) 2024-03-06T11:05:41.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:41.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:41.508377+0000) 2024-03-06T11:05:42.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:42.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:42.508594+0000) 2024-03-06T11:05:43.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:43.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:43.508837+0000) 2024-03-06T11:05:44.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:44.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:44.509068+0000) 2024-03-06T11:05:45.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:45.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:45.509256+0000) 2024-03-06T11:05:46.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:46.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:46.509469+0000) 2024-03-06T11:05:47.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:47.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:47.509653+0000) 2024-03-06T11:05:48.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:48.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:48.509907+0000) 2024-03-06T11:05:49.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:49.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:49.510189+0000) 2024-03-06T11:05:50.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:50.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:50.510387+0000) 2024-03-06T11:05:51.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:51.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:51.510583+0000) 2024-03-06T11:05:52.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:52.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:52.510779+0000) 2024-03-06T11:05:53.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:53.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:53.510983+0000) 2024-03-06T11:05:54.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:54.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:54.511208+0000) 2024-03-06T11:05:55.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:55.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:55.511393+0000) 2024-03-06T11:05:56.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:56.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:56.511617+0000) 2024-03-06T11:05:57.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:57.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:57.511845+0000) 2024-03-06T11:05:58.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:58.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:58.512071+0000) 2024-03-06T11:05:59.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:05:59.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:59.512286+0000) 2024-03-06T11:06:00.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:00.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:00.512513+0000) 2024-03-06T11:06:01.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:01.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:01.512830+0000) 2024-03-06T11:06:02.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:02.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:02.513153+0000) 2024-03-06T11:06:03.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:03.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:03.513434+0000) 2024-03-06T11:06:04.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:04.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:04.513748+0000) 2024-03-06T11:06:05.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:05.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:05.514027+0000) 2024-03-06T11:06:06.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:06.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:06.514243+0000) 2024-03-06T11:06:07.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:07.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:07.514442+0000) 2024-03-06T11:06:08.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:08.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:08.514743+0000) 2024-03-06T11:06:09.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:09.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:09.515030+0000) 2024-03-06T11:06:10.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:10.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:10.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:10.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:10.515310+0000) 2024-03-06T11:06:11.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:11.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:11.515460+0000) 2024-03-06T11:06:12.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:12.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:12.515696+0000) 2024-03-06T11:06:13.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:13.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:13.515973+0000) 2024-03-06T11:06:14.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:14.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:14.516191+0000) 2024-03-06T11:06:15.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:15.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:15.516411+0000) 2024-03-06T11:06:16.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:16.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:16.516636+0000) 2024-03-06T11:06:17.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:17.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:17.516863+0000) 2024-03-06T11:06:18.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:18.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:18.517093+0000) 2024-03-06T11:06:19.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:19.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:19.517328+0000) 2024-03-06T11:06:20.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:20.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:20.517550+0000) 2024-03-06T11:06:21.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:21.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:21.517777+0000) 2024-03-06T11:06:22.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:22.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:22.517996+0000) 2024-03-06T11:06:23.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:23.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:23.518226+0000) 2024-03-06T11:06:24.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:24.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:24.518449+0000) 2024-03-06T11:06:25.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:25.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:25.518705+0000) 2024-03-06T11:06:26.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:26.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:26.518944+0000) 2024-03-06T11:06:27.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:27.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:27.519166+0000) 2024-03-06T11:06:28.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:28.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:28.519389+0000) 2024-03-06T11:06:29.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:29.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:29.519624+0000) 2024-03-06T11:06:30.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:30.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:30.519831+0000) 2024-03-06T11:06:31.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:31.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:31.520097+0000) 2024-03-06T11:06:32.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:32.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:32.520363+0000) 2024-03-06T11:06:33.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:33.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:33.520644+0000) 2024-03-06T11:06:34.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:34.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:34.520898+0000) 2024-03-06T11:06:35.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:35.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:35.521112+0000) 2024-03-06T11:06:36.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:36.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:36.521387+0000) 2024-03-06T11:06:37.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:37.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:37.521709+0000) 2024-03-06T11:06:38.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:38.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:38.521993+0000) 2024-03-06T11:06:39.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:39.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:39.522265+0000) 2024-03-06T11:06:40.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:40.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:40.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:40.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:40.522520+0000) 2024-03-06T11:06:41.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:41.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:41.522840+0000) 2024-03-06T11:06:42.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:42.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:42.523037+0000) 2024-03-06T11:06:43.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:43.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:43.523239+0000) 2024-03-06T11:06:44.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:44.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:44.523479+0000) 2024-03-06T11:06:45.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:45.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:45.523741+0000) 2024-03-06T11:06:46.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:46.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:46.523973+0000) 2024-03-06T11:06:47.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:47.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:47.524149+0000) 2024-03-06T11:06:48.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:48.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:48.524370+0000) 2024-03-06T11:06:49.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:49.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:49.524630+0000) 2024-03-06T11:06:50.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:50.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:50.524938+0000) 2024-03-06T11:06:51.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:51.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:51.525168+0000) 2024-03-06T11:06:52.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:52.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:52.525399+0000) 2024-03-06T11:06:53.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:53.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:53.525627+0000) 2024-03-06T11:06:54.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:54.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:54.525850+0000) 2024-03-06T11:06:55.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:55.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:55.526128+0000) 2024-03-06T11:06:56.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:56.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:56.526377+0000) 2024-03-06T11:06:57.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:57.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:57.526630+0000) 2024-03-06T11:06:58.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:58.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:58.526906+0000) 2024-03-06T11:06:59.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:06:59.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:59.527114+0000) 2024-03-06T11:07:00.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:00.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:00.527382+0000) 2024-03-06T11:07:01.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:01.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:01.527601+0000) 2024-03-06T11:07:02.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:02.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:02.527821+0000) 2024-03-06T11:07:03.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:03.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:03.528055+0000) 2024-03-06T11:07:04.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:04.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:04.528278+0000) 2024-03-06T11:07:05.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:05.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:05.528504+0000) 2024-03-06T11:07:06.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:06.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:06.528788+0000) 2024-03-06T11:07:07.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:07.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:07.529015+0000) 2024-03-06T11:07:08.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:08.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:08.529209+0000) 2024-03-06T11:07:09.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:09.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:09.529469+0000) 2024-03-06T11:07:10.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:10.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:10.529730+0000) 2024-03-06T11:07:10.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:10.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:11.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:11.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:11.529965+0000) 2024-03-06T11:07:12.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:12.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:12.530189+0000) 2024-03-06T11:07:13.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:13.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:13.530385+0000) 2024-03-06T11:07:14.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:14.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:14.530616+0000) 2024-03-06T11:07:15.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:15.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:15.530853+0000) 2024-03-06T11:07:16.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:16.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:16.531057+0000) 2024-03-06T11:07:17.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:17.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:17.531263+0000) 2024-03-06T11:07:18.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:18.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:18.531498+0000) 2024-03-06T11:07:19.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:19.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:19.531753+0000) 2024-03-06T11:07:20.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:20.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:20.531944+0000) 2024-03-06T11:07:21.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:21.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:21.532142+0000) 2024-03-06T11:07:22.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:22.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:22.532366+0000) 2024-03-06T11:07:23.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:23.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:23.532580+0000) 2024-03-06T11:07:24.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:24.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:24.532820+0000) 2024-03-06T11:07:25.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:25.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:25.533043+0000) 2024-03-06T11:07:26.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:26.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:26.533240+0000) 2024-03-06T11:07:27.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:27.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:27.533442+0000) 2024-03-06T11:07:28.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:28.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:28.533634+0000) 2024-03-06T11:07:29.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:29.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:29.533890+0000) 2024-03-06T11:07:30.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:30.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:30.534094+0000) 2024-03-06T11:07:31.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:31.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:31.534298+0000) 2024-03-06T11:07:32.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:32.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:32.534537+0000) 2024-03-06T11:07:33.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:33.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:33.534810+0000) 2024-03-06T11:07:34.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:34.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:34.535031+0000) 2024-03-06T11:07:35.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:35.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:35.535222+0000) 2024-03-06T11:07:36.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:36.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:36.535415+0000) 2024-03-06T11:07:37.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:37.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:37.535641+0000) 2024-03-06T11:07:38.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:38.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:38.535859+0000) 2024-03-06T11:07:39.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:39.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:39.536101+0000) 2024-03-06T11:07:40.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:40.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:40.536340+0000) 2024-03-06T11:07:40.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:40.773 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:41.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:41.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:41.536586+0000) 2024-03-06T11:07:42.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:42.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:42.536833+0000) 2024-03-06T11:07:43.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:43.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:43.537106+0000) 2024-03-06T11:07:44.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:44.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:44.537306+0000) 2024-03-06T11:07:45.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:45.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:45.537542+0000) 2024-03-06T11:07:46.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:46.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:46.537782+0000) 2024-03-06T11:07:47.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:47.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:47.538060+0000) 2024-03-06T11:07:48.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:48.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:48.538259+0000) 2024-03-06T11:07:49.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:49.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:49.538453+0000) 2024-03-06T11:07:50.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:50.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:50.538702+0000) 2024-03-06T11:07:51.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:51.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:51.538926+0000) 2024-03-06T11:07:52.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:52.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:52.539135+0000) 2024-03-06T11:07:53.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:53.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:53.539358+0000) 2024-03-06T11:07:54.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:54.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:54.539582+0000) 2024-03-06T11:07:55.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:55.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:55.539827+0000) 2024-03-06T11:07:56.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:56.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:56.540027+0000) 2024-03-06T11:07:57.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:57.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:57.540228+0000) 2024-03-06T11:07:58.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:58.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:58.540466+0000) 2024-03-06T11:07:59.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:07:59.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:59.540737+0000) 2024-03-06T11:08:00.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:00.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:00.540931+0000) 2024-03-06T11:08:01.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:01.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:01.541153+0000) 2024-03-06T11:08:02.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:02.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:02.541377+0000) 2024-03-06T11:08:03.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:03.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:03.541604+0000) 2024-03-06T11:08:04.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:04.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:04.541796+0000) 2024-03-06T11:08:05.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:05.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:05.541986+0000) 2024-03-06T11:08:06.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:06.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:06.542172+0000) 2024-03-06T11:08:07.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:07.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:07.542386+0000) 2024-03-06T11:08:08.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:08.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:08.542608+0000) 2024-03-06T11:08:09.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:09.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:09.542818+0000) 2024-03-06T11:08:10.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:10.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:10.543011+0000) 2024-03-06T11:08:10.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:10.987 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:11.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:11.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:11.543212+0000) 2024-03-06T11:08:12.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:12.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:12.543434+0000) 2024-03-06T11:08:13.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:13.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:13.543655+0000) 2024-03-06T11:08:14.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:14.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:14.543881+0000) 2024-03-06T11:08:15.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:15.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:15.544075+0000) 2024-03-06T11:08:16.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:16.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:16.544269+0000) 2024-03-06T11:08:17.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:17.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:17.544493+0000) 2024-03-06T11:08:18.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:18.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:18.544764+0000) 2024-03-06T11:08:19.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:19.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:19.544971+0000) 2024-03-06T11:08:20.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:20.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:20.545190+0000) 2024-03-06T11:08:21.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:21.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:21.545361+0000) 2024-03-06T11:08:22.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:22.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:22.545592+0000) 2024-03-06T11:08:23.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:23.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:23.545823+0000) 2024-03-06T11:08:24.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:24.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:24.546033+0000) 2024-03-06T11:08:25.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:25.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:25.546211+0000) 2024-03-06T11:08:26.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:26.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:26.546399+0000) 2024-03-06T11:08:27.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:27.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:27.546625+0000) 2024-03-06T11:08:28.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:28.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:28.546819+0000) 2024-03-06T11:08:29.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:29.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:29.547033+0000) 2024-03-06T11:08:30.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:30.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:30.547221+0000) 2024-03-06T11:08:31.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:31.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:31.547450+0000) 2024-03-06T11:08:32.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:32.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:32.547705+0000) 2024-03-06T11:08:33.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:33.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:33.547903+0000) 2024-03-06T11:08:34.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:34.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:34.548068+0000) 2024-03-06T11:08:35.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:35.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:35.548259+0000) 2024-03-06T11:08:36.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:36.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:36.548465+0000) 2024-03-06T11:08:37.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:37.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:37.548789+0000) 2024-03-06T11:08:38.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:38.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:38.549010+0000) 2024-03-06T11:08:39.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:39.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:39.549250+0000) 2024-03-06T11:08:40.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:40.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:40.549462+0000) 2024-03-06T11:08:41.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:41.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:41.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:41.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:41.549709+0000) 2024-03-06T11:08:42.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:42.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:42.549925+0000) 2024-03-06T11:08:43.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:43.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:43.550151+0000) 2024-03-06T11:08:44.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:44.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:44.550381+0000) 2024-03-06T11:08:45.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:45.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:45.550603+0000) 2024-03-06T11:08:46.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:46.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:46.550857+0000) 2024-03-06T11:08:47.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:47.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:47.551094+0000) 2024-03-06T11:08:48.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:48.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:48.551305+0000) 2024-03-06T11:08:49.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:49.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:49.551557+0000) 2024-03-06T11:08:50.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:50.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:50.551782+0000) 2024-03-06T11:08:51.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:51.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:51.552012+0000) 2024-03-06T11:08:52.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:52.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:52.552246+0000) 2024-03-06T11:08:53.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:53.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:53.552498+0000) 2024-03-06T11:08:54.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:54.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:54.552786+0000) 2024-03-06T11:08:55.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:55.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:55.552977+0000) 2024-03-06T11:08:56.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:56.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:56.553164+0000) 2024-03-06T11:08:57.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:57.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:57.553356+0000) 2024-03-06T11:08:58.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:58.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:58.553623+0000) 2024-03-06T11:08:59.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:08:59.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:59.553956+0000) 2024-03-06T11:09:00.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:00.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:00.554152+0000) 2024-03-06T11:09:01.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:01.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:01.554373+0000) 2024-03-06T11:09:02.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:02.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:02.554608+0000) 2024-03-06T11:09:03.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:03.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:03.554854+0000) 2024-03-06T11:09:04.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:04.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:04.555089+0000) 2024-03-06T11:09:05.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:05.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:05.555352+0000) 2024-03-06T11:09:06.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:06.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:06.555603+0000) 2024-03-06T11:09:07.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:07.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:07.555839+0000) 2024-03-06T11:09:08.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:08.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:08.556043+0000) 2024-03-06T11:09:09.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:09.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:09.556294+0000) 2024-03-06T11:09:10.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:10.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:10.556535+0000) 2024-03-06T11:09:11.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:11.401 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:11.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:11.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:11.556798+0000) 2024-03-06T11:09:12.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:12.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:12.557036+0000) 2024-03-06T11:09:13.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:13.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:13.557264+0000) 2024-03-06T11:09:14.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:14.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:14.557563+0000) 2024-03-06T11:09:15.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:15.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:15.557779+0000) 2024-03-06T11:09:16.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:16.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:16.557987+0000) 2024-03-06T11:09:17.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:17.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:17.558212+0000) 2024-03-06T11:09:18.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:18.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:18.558387+0000) 2024-03-06T11:09:19.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:19.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:19.558645+0000) 2024-03-06T11:09:20.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:20.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:20.558872+0000) 2024-03-06T11:09:21.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:21.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:21.559121+0000) 2024-03-06T11:09:22.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:22.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:22.559359+0000) 2024-03-06T11:09:23.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:23.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:23.559598+0000) 2024-03-06T11:09:24.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:24.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:24.559855+0000) 2024-03-06T11:09:25.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:25.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:25.560109+0000) 2024-03-06T11:09:26.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:26.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:26.560351+0000) 2024-03-06T11:09:27.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:27.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:27.560592+0000) 2024-03-06T11:09:28.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:28.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:28.560800+0000) 2024-03-06T11:09:29.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:29.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:29.560993+0000) 2024-03-06T11:09:30.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:30.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:30.561270+0000) 2024-03-06T11:09:31.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:31.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:31.561498+0000) 2024-03-06T11:09:32.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:32.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:32.561766+0000) 2024-03-06T11:09:33.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:33.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:33.561993+0000) 2024-03-06T11:09:34.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:34.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:34.562216+0000) 2024-03-06T11:09:35.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:35.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:35.562482+0000) 2024-03-06T11:09:36.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:36.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:36.562752+0000) 2024-03-06T11:09:37.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:37.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:37.562977+0000) 2024-03-06T11:09:38.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:38.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:38.563156+0000) 2024-03-06T11:09:39.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:39.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:39.563387+0000) 2024-03-06T11:09:40.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:40.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:40.563605+0000) 2024-03-06T11:09:41.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:41.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:41.563795+0000) 2024-03-06T11:09:41.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:41.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:42.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:42.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:42.564015+0000) 2024-03-06T11:09:43.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:43.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:43.564260+0000) 2024-03-06T11:09:44.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:44.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:44.564470+0000) 2024-03-06T11:09:45.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:45.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:45.564712+0000) 2024-03-06T11:09:46.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:46.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:46.564948+0000) 2024-03-06T11:09:47.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:47.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:47.565172+0000) 2024-03-06T11:09:48.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:48.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:48.565396+0000) 2024-03-06T11:09:49.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:49.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:49.565621+0000) 2024-03-06T11:09:50.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:50.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:50.565818+0000) 2024-03-06T11:09:51.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:51.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:51.566044+0000) 2024-03-06T11:09:52.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:52.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:52.566288+0000) 2024-03-06T11:09:53.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:53.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:53.566571+0000) 2024-03-06T11:09:54.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:54.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:54.566853+0000) 2024-03-06T11:09:55.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:55.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:55.567099+0000) 2024-03-06T11:09:56.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:56.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:56.567389+0000) 2024-03-06T11:09:57.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:57.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:57.567604+0000) 2024-03-06T11:09:58.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:58.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:58.567779+0000) 2024-03-06T11:09:59.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:09:59.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:59.567967+0000) 2024-03-06T11:10:00.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:00.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:00.568158+0000) 2024-03-06T11:10:01.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:01.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:01.568423+0000) 2024-03-06T11:10:02.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:02.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:02.568734+0000) 2024-03-06T11:10:03.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:03.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:03.568980+0000) 2024-03-06T11:10:04.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:04.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:04.569225+0000) 2024-03-06T11:10:05.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:05.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:05.569505+0000) 2024-03-06T11:10:06.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:06.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:06.569775+0000) 2024-03-06T11:10:07.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:07.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:07.569961+0000) 2024-03-06T11:10:08.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:08.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:08.570167+0000) 2024-03-06T11:10:09.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:09.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:09.570378+0000) 2024-03-06T11:10:10.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:10.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:10.570604+0000) 2024-03-06T11:10:11.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:11.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:11.570788+0000) 2024-03-06T11:10:11.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:11.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:12.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:12.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:12.570980+0000) 2024-03-06T11:10:13.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:13.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:13.571168+0000) 2024-03-06T11:10:14.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:14.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:14.571355+0000) 2024-03-06T11:10:15.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:15.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:15.571598+0000) 2024-03-06T11:10:16.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:16.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:16.571854+0000) 2024-03-06T11:10:17.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:17.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:17.572089+0000) 2024-03-06T11:10:18.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:18.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:18.572419+0000) 2024-03-06T11:10:19.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:19.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:19.572715+0000) 2024-03-06T11:10:20.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:20.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:20.572961+0000) 2024-03-06T11:10:21.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:21.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:21.573208+0000) 2024-03-06T11:10:22.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:22.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:22.573493+0000) 2024-03-06T11:10:23.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:23.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:23.573792+0000) 2024-03-06T11:10:24.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:24.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:24.574073+0000) 2024-03-06T11:10:25.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:25.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:25.574299+0000) 2024-03-06T11:10:26.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:26.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:26.574544+0000) 2024-03-06T11:10:27.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:27.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:27.574785+0000) 2024-03-06T11:10:28.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:28.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:28.574973+0000) 2024-03-06T11:10:29.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:29.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:29.575177+0000) 2024-03-06T11:10:30.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:30.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:30.575398+0000) 2024-03-06T11:10:31.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:31.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:31.575602+0000) 2024-03-06T11:10:32.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:32.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:32.575809+0000) 2024-03-06T11:10:33.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:33.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:33.576037+0000) 2024-03-06T11:10:34.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:34.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:34.576315+0000) 2024-03-06T11:10:35.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:35.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:35.576598+0000) 2024-03-06T11:10:36.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:36.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:36.576896+0000) 2024-03-06T11:10:37.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:37.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:37.577138+0000) 2024-03-06T11:10:38.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:38.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:38.577414+0000) 2024-03-06T11:10:39.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:39.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:39.577723+0000) 2024-03-06T11:10:40.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:40.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:40.577995+0000) 2024-03-06T11:10:41.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:41.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:41.578284+0000) 2024-03-06T11:10:42.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:42.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:42.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:42.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:42.578478+0000) 2024-03-06T11:10:43.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:43.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:43.578743+0000) 2024-03-06T11:10:44.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:44.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:44.578964+0000) 2024-03-06T11:10:45.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:45.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:45.579158+0000) 2024-03-06T11:10:46.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:46.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:46.579391+0000) 2024-03-06T11:10:47.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:47.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:47.579614+0000) 2024-03-06T11:10:48.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:48.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:48.579847+0000) 2024-03-06T11:10:49.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:49.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:49.580073+0000) 2024-03-06T11:10:50.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:50.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:50.580304+0000) 2024-03-06T11:10:51.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:51.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:51.580536+0000) 2024-03-06T11:10:52.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:52.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:52.580795+0000) 2024-03-06T11:10:53.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:53.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:53.581022+0000) 2024-03-06T11:10:54.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:54.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:54.581248+0000) 2024-03-06T11:10:55.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:55.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:55.581441+0000) 2024-03-06T11:10:56.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:56.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:56.581636+0000) 2024-03-06T11:10:57.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:57.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:57.581864+0000) 2024-03-06T11:10:58.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:58.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:58.582094+0000) 2024-03-06T11:10:59.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:10:59.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:59.582325+0000) 2024-03-06T11:11:00.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:00.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:00.582603+0000) 2024-03-06T11:11:01.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:01.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:01.582835+0000) 2024-03-06T11:11:02.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:02.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:02.583062+0000) 2024-03-06T11:11:03.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:03.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:03.583289+0000) 2024-03-06T11:11:04.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:04.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:04.583514+0000) 2024-03-06T11:11:05.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:05.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:05.583793+0000) 2024-03-06T11:11:06.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:06.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:06.584029+0000) 2024-03-06T11:11:07.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:07.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:07.584215+0000) 2024-03-06T11:11:08.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:08.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:08.584442+0000) 2024-03-06T11:11:09.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:09.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:09.584742+0000) 2024-03-06T11:11:10.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:10.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:10.584986+0000) 2024-03-06T11:11:11.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:11.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:11.585278+0000) 2024-03-06T11:11:12.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:12.247 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:12.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:12.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:12.585558+0000) 2024-03-06T11:11:13.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:13.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:13.585836+0000) 2024-03-06T11:11:14.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:14.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:14.586025+0000) 2024-03-06T11:11:15.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:15.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:15.586247+0000) 2024-03-06T11:11:16.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:16.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:16.586483+0000) 2024-03-06T11:11:17.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:17.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:17.586711+0000) 2024-03-06T11:11:18.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:18.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:18.586899+0000) 2024-03-06T11:11:19.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:19.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:19.587116+0000) 2024-03-06T11:11:20.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:20.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:20.587339+0000) 2024-03-06T11:11:21.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:21.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:21.587562+0000) 2024-03-06T11:11:22.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:22.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:22.587793+0000) 2024-03-06T11:11:23.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:23.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:23.587994+0000) 2024-03-06T11:11:24.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:24.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:24.588210+0000) 2024-03-06T11:11:25.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:25.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:25.588373+0000) 2024-03-06T11:11:26.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:26.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:26.588564+0000) 2024-03-06T11:11:27.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:27.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:27.588796+0000) 2024-03-06T11:11:28.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:28.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:28.589000+0000) 2024-03-06T11:11:29.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:29.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:29.589194+0000) 2024-03-06T11:11:30.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:30.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:30.589433+0000) 2024-03-06T11:11:31.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:31.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:31.589687+0000) 2024-03-06T11:11:32.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:32.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:32.589911+0000) 2024-03-06T11:11:33.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:33.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:33.590119+0000) 2024-03-06T11:11:34.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:34.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:34.590349+0000) 2024-03-06T11:11:35.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:35.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:35.590600+0000) 2024-03-06T11:11:36.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:36.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:36.590802+0000) 2024-03-06T11:11:37.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:37.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:37.590997+0000) 2024-03-06T11:11:38.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:38.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:38.591189+0000) 2024-03-06T11:11:39.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:39.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:39.591413+0000) 2024-03-06T11:11:40.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:40.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:40.591620+0000) 2024-03-06T11:11:41.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:41.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:41.591812+0000) 2024-03-06T11:11:42.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:42.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:42.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:42.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:42.592009+0000) 2024-03-06T11:11:43.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:43.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:43.592200+0000) 2024-03-06T11:11:44.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:44.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:44.592440+0000) 2024-03-06T11:11:45.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:45.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:45.592639+0000) 2024-03-06T11:11:46.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:46.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:46.592886+0000) 2024-03-06T11:11:47.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:47.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:47.593092+0000) 2024-03-06T11:11:48.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:48.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:48.593291+0000) 2024-03-06T11:11:49.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:49.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:49.593517+0000) 2024-03-06T11:11:50.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:50.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:50.593768+0000) 2024-03-06T11:11:51.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:51.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:51.593951+0000) 2024-03-06T11:11:52.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:52.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:52.594135+0000) 2024-03-06T11:11:53.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:53.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:53.594329+0000) 2024-03-06T11:11:54.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:54.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:54.594564+0000) 2024-03-06T11:11:55.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:55.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:55.594821+0000) 2024-03-06T11:11:56.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:56.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:56.595010+0000) 2024-03-06T11:11:57.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:57.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:57.595233+0000) 2024-03-06T11:11:58.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:58.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:58.595428+0000) 2024-03-06T11:11:59.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:11:59.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:59.595661+0000) 2024-03-06T11:12:00.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:00.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:00.595879+0000) 2024-03-06T11:12:01.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:01.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:01.596072+0000) 2024-03-06T11:12:02.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:02.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:02.596307+0000) 2024-03-06T11:12:03.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:03.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:03.596549+0000) 2024-03-06T11:12:04.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:04.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:04.596785+0000) 2024-03-06T11:12:05.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:05.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:05.597080+0000) 2024-03-06T11:12:06.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:06.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:06.597307+0000) 2024-03-06T11:12:07.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:07.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:07.597524+0000) 2024-03-06T11:12:08.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:08.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:08.597799+0000) 2024-03-06T11:12:09.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:09.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:09.598026+0000) 2024-03-06T11:12:10.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:10.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:10.598268+0000) 2024-03-06T11:12:11.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:11.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:11.598510+0000) 2024-03-06T11:12:12.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:12.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:12.598781+0000) 2024-03-06T11:12:12.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:12.671 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:13.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:13.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:13.599026+0000) 2024-03-06T11:12:14.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:14.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:14.599261+0000) 2024-03-06T11:12:15.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:15.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:15.599516+0000) 2024-03-06T11:12:16.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:16.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:16.599775+0000) 2024-03-06T11:12:17.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:17.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:17.600020+0000) 2024-03-06T11:12:18.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:18.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:18.600263+0000) 2024-03-06T11:12:19.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:19.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:19.600504+0000) 2024-03-06T11:12:20.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:20.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:20.600743+0000) 2024-03-06T11:12:21.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:21.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:21.600963+0000) 2024-03-06T11:12:22.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:22.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:22.601200+0000) 2024-03-06T11:12:23.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:23.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:23.601432+0000) 2024-03-06T11:12:24.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:24.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:24.601653+0000) 2024-03-06T11:12:25.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:25.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:25.601873+0000) 2024-03-06T11:12:26.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:26.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:26.602133+0000) 2024-03-06T11:12:27.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:27.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:27.602366+0000) 2024-03-06T11:12:28.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:28.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:28.602632+0000) 2024-03-06T11:12:29.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:29.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:29.602934+0000) 2024-03-06T11:12:30.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:30.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:30.603160+0000) 2024-03-06T11:12:31.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:31.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:31.603449+0000) 2024-03-06T11:12:32.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:32.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:32.603755+0000) 2024-03-06T11:12:33.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:33.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:33.604002+0000) 2024-03-06T11:12:34.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:34.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:34.604245+0000) 2024-03-06T11:12:35.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:35.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:35.604523+0000) 2024-03-06T11:12:36.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:36.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:36.604820+0000) 2024-03-06T11:12:37.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:37.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:37.605104+0000) 2024-03-06T11:12:38.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:38.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:38.605315+0000) 2024-03-06T11:12:39.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:39.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:39.605544+0000) 2024-03-06T11:12:40.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:40.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:40.605817+0000) 2024-03-06T11:12:41.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:41.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:41.606056+0000) 2024-03-06T11:12:42.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:42.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:42.606334+0000) 2024-03-06T11:12:42.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:42.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:43.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:43.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:43.606620+0000) 2024-03-06T11:12:44.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:44.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:44.606945+0000) 2024-03-06T11:12:45.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:45.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:45.607189+0000) 2024-03-06T11:12:46.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:46.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:46.607457+0000) 2024-03-06T11:12:47.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:47.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:47.607722+0000) 2024-03-06T11:12:48.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:48.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:48.607927+0000) 2024-03-06T11:12:49.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:49.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:49.608119+0000) 2024-03-06T11:12:50.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:50.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:50.608343+0000) 2024-03-06T11:12:51.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:51.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:51.608620+0000) 2024-03-06T11:12:52.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:52.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:52.608841+0000) 2024-03-06T11:12:53.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:53.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:53.609087+0000) 2024-03-06T11:12:54.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:54.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:54.609294+0000) 2024-03-06T11:12:55.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:55.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:55.609535+0000) 2024-03-06T11:12:56.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:56.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:56.609772+0000) 2024-03-06T11:12:57.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:57.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:57.609996+0000) 2024-03-06T11:12:58.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:58.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:58.610235+0000) 2024-03-06T11:12:59.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:12:59.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:59.610457+0000) 2024-03-06T11:13:00.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:00.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:00.610708+0000) 2024-03-06T11:13:01.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:01.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:01.610895+0000) 2024-03-06T11:13:02.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:02.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:02.611085+0000) 2024-03-06T11:13:03.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:03.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:03.611342+0000) 2024-03-06T11:13:04.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:04.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:04.611625+0000) 2024-03-06T11:13:05.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:05.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:05.611920+0000) 2024-03-06T11:13:06.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:06.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:06.612209+0000) 2024-03-06T11:13:07.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:07.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:07.612445+0000) 2024-03-06T11:13:08.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:08.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:08.612651+0000) 2024-03-06T11:13:09.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:09.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:09.612900+0000) 2024-03-06T11:13:10.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:10.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:10.613124+0000) 2024-03-06T11:13:11.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:11.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:11.613349+0000) 2024-03-06T11:13:12.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:12.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:12.613570+0000) 2024-03-06T11:13:13.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:13.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:13.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:13.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:13.613804+0000) 2024-03-06T11:13:14.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:14.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:14.614036+0000) 2024-03-06T11:13:15.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:15.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:15.614233+0000) 2024-03-06T11:13:16.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:16.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:16.614465+0000) 2024-03-06T11:13:17.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:17.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:17.614639+0000) 2024-03-06T11:13:18.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:18.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:18.614865+0000) 2024-03-06T11:13:19.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:19.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:19.615060+0000) 2024-03-06T11:13:20.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:20.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:20.615282+0000) 2024-03-06T11:13:21.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:21.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:21.615526+0000) 2024-03-06T11:13:22.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:22.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:22.615813+0000) 2024-03-06T11:13:23.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:23.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:23.616050+0000) 2024-03-06T11:13:24.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:24.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:24.616293+0000) 2024-03-06T11:13:25.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:25.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:25.616544+0000) 2024-03-06T11:13:26.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:26.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:26.616772+0000) 2024-03-06T11:13:27.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:27.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:27.616959+0000) 2024-03-06T11:13:28.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:28.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:28.617190+0000) 2024-03-06T11:13:29.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:29.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:29.617409+0000) 2024-03-06T11:13:30.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:30.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:30.617619+0000) 2024-03-06T11:13:31.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:31.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:31.617839+0000) 2024-03-06T11:13:32.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:32.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:32.617988+0000) 2024-03-06T11:13:33.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:33.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:33.618180+0000) 2024-03-06T11:13:34.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:34.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:34.618401+0000) 2024-03-06T11:13:35.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:35.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:35.618639+0000) 2024-03-06T11:13:36.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:36.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:36.618862+0000) 2024-03-06T11:13:37.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:37.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:37.619051+0000) 2024-03-06T11:13:38.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:38.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:38.619275+0000) 2024-03-06T11:13:39.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:39.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:39.619817+0000) 2024-03-06T11:13:40.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:40.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:40.620061+0000) 2024-03-06T11:13:41.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:41.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:41.620296+0000) 2024-03-06T11:13:42.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:42.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:42.620573+0000) 2024-03-06T11:13:43.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:43.159 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:43.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:43.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:43.620873+0000) 2024-03-06T11:13:44.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:44.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:44.621074+0000) 2024-03-06T11:13:45.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:45.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:45.621283+0000) 2024-03-06T11:13:46.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:46.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:46.621580+0000) 2024-03-06T11:13:47.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:47.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:47.621860+0000) 2024-03-06T11:13:48.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:48.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:48.622138+0000) 2024-03-06T11:13:49.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:49.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:49.622414+0000) 2024-03-06T11:13:50.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:50.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:50.622650+0000) 2024-03-06T11:13:51.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:51.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:51.622931+0000) 2024-03-06T11:13:52.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:52.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:52.623139+0000) 2024-03-06T11:13:53.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:53.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:53.623375+0000) 2024-03-06T11:13:54.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:54.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:54.623627+0000) 2024-03-06T11:13:55.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:55.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:55.623899+0000) 2024-03-06T11:13:56.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:56.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:56.624198+0000) 2024-03-06T11:13:57.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:57.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:57.624400+0000) 2024-03-06T11:13:58.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:58.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:58.624626+0000) 2024-03-06T11:13:59.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:13:59.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:59.624916+0000) 2024-03-06T11:14:00.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:00.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:00.625164+0000) 2024-03-06T11:14:01.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:01.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:01.625433+0000) 2024-03-06T11:14:02.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:02.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:02.625754+0000) 2024-03-06T11:14:03.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:03.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:03.625958+0000) 2024-03-06T11:14:04.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:04.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:04.626180+0000) 2024-03-06T11:14:05.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:05.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:05.626415+0000) 2024-03-06T11:14:06.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:06.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:06.626616+0000) 2024-03-06T11:14:07.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:07.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:07.626819+0000) 2024-03-06T11:14:08.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:08.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:08.627040+0000) 2024-03-06T11:14:09.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:09.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:09.627280+0000) 2024-03-06T11:14:10.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:10.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:10.627498+0000) 2024-03-06T11:14:11.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:11.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:11.627770+0000) 2024-03-06T11:14:12.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:12.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:12.628105+0000) 2024-03-06T11:14:13.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:13.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:13.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:13.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:13.628362+0000) 2024-03-06T11:14:14.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:14.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:14.628600+0000) 2024-03-06T11:14:15.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:15.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:15.628870+0000) 2024-03-06T11:14:16.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:16.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:16.629119+0000) 2024-03-06T11:14:17.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:17.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:17.629375+0000) 2024-03-06T11:14:18.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:18.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:18.629574+0000) 2024-03-06T11:14:19.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:19.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:19.629769+0000) 2024-03-06T11:14:20.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:20.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:20.629980+0000) 2024-03-06T11:14:21.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:21.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:21.630169+0000) 2024-03-06T11:14:22.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:22.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:22.630387+0000) 2024-03-06T11:14:23.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:23.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:23.630590+0000) 2024-03-06T11:14:24.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:24.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:24.630822+0000) 2024-03-06T11:14:25.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:25.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:25.631049+0000) 2024-03-06T11:14:26.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:26.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:26.631298+0000) 2024-03-06T11:14:27.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:27.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:27.631515+0000) 2024-03-06T11:14:28.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:28.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:28.631749+0000) 2024-03-06T11:14:29.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:29.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:29.631936+0000) 2024-03-06T11:14:30.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:30.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:30.632184+0000) 2024-03-06T11:14:31.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:31.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:31.632408+0000) 2024-03-06T11:14:32.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:32.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:32.632618+0000) 2024-03-06T11:14:33.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:33.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:33.632822+0000) 2024-03-06T11:14:34.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:34.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:34.633053+0000) 2024-03-06T11:14:35.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:35.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:35.633288+0000) 2024-03-06T11:14:36.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:36.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:36.633567+0000) 2024-03-06T11:14:37.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:37.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:37.633832+0000) 2024-03-06T11:14:38.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:38.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:38.634062+0000) 2024-03-06T11:14:39.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:39.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:39.634304+0000) 2024-03-06T11:14:40.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:40.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:40.634597+0000) 2024-03-06T11:14:41.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:41.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:41.634874+0000) 2024-03-06T11:14:42.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:42.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:42.635119+0000) 2024-03-06T11:14:43.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:43.519 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:43.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:43.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:43.635339+0000) 2024-03-06T11:14:44.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:44.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:44.635574+0000) 2024-03-06T11:14:45.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:45.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:45.635822+0000) 2024-03-06T11:14:46.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:46.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:46.636025+0000) 2024-03-06T11:14:47.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:47.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:47.636211+0000) 2024-03-06T11:14:48.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:48.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:48.636437+0000) 2024-03-06T11:14:49.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:49.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:49.636753+0000) 2024-03-06T11:14:50.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:50.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:50.637015+0000) 2024-03-06T11:14:51.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:51.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:51.637251+0000) 2024-03-06T11:14:52.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:52.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:52.637442+0000) 2024-03-06T11:14:53.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:53.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:53.637643+0000) 2024-03-06T11:14:54.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:54.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:54.637826+0000) 2024-03-06T11:14:55.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:55.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:55.638069+0000) 2024-03-06T11:14:56.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:56.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:56.638262+0000) 2024-03-06T11:14:57.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:57.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:57.638441+0000) 2024-03-06T11:14:58.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:58.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:58.638622+0000) 2024-03-06T11:14:59.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:14:59.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:59.638845+0000) 2024-03-06T11:15:00.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:00.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:00.639097+0000) 2024-03-06T11:15:01.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:01.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:01.639320+0000) 2024-03-06T11:15:02.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:02.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:02.639539+0000) 2024-03-06T11:15:03.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:03.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:03.639774+0000) 2024-03-06T11:15:04.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:04.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:04.640007+0000) 2024-03-06T11:15:05.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:05.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:05.640251+0000) 2024-03-06T11:15:06.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:06.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:06.640490+0000) 2024-03-06T11:15:07.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:07.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:07.640750+0000) 2024-03-06T11:15:08.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:08.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:08.640970+0000) 2024-03-06T11:15:09.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:09.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:09.641159+0000) 2024-03-06T11:15:10.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:10.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:10.641406+0000) 2024-03-06T11:15:11.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:11.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:11.641639+0000) 2024-03-06T11:15:12.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:12.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:12.641909+0000) 2024-03-06T11:15:13.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:13.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:13.642135+0000) 2024-03-06T11:15:13.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:13.716 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:14.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:14.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:14.642357+0000) 2024-03-06T11:15:15.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:15.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:15.642591+0000) 2024-03-06T11:15:16.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:16.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:16.642760+0000) 2024-03-06T11:15:17.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:17.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:17.642948+0000) 2024-03-06T11:15:18.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:18.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:18.643137+0000) 2024-03-06T11:15:19.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:19.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:19.643337+0000) 2024-03-06T11:15:20.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:20.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:20.643631+0000) 2024-03-06T11:15:21.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:21.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:21.643882+0000) 2024-03-06T11:15:22.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:22.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:22.644093+0000) 2024-03-06T11:15:23.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:23.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:23.644314+0000) 2024-03-06T11:15:24.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:24.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:24.644571+0000) 2024-03-06T11:15:25.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:25.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:25.644821+0000) 2024-03-06T11:15:26.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:26.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:26.645077+0000) 2024-03-06T11:15:27.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:27.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:27.645289+0000) 2024-03-06T11:15:28.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:28.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:28.645536+0000) 2024-03-06T11:15:29.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:29.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:29.645757+0000) 2024-03-06T11:15:30.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:30.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:30.645991+0000) 2024-03-06T11:15:31.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:31.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:31.646220+0000) 2024-03-06T11:15:32.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:32.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:32.646418+0000) 2024-03-06T11:15:33.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:33.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:33.646739+0000) 2024-03-06T11:15:34.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:34.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:34.646956+0000) 2024-03-06T11:15:35.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:35.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:35.647189+0000) 2024-03-06T11:15:36.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:36.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:36.647422+0000) 2024-03-06T11:15:37.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:37.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:37.647644+0000) 2024-03-06T11:15:38.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:38.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:38.647875+0000) 2024-03-06T11:15:39.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:39.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:39.648115+0000) 2024-03-06T11:15:40.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:40.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:40.648388+0000) 2024-03-06T11:15:41.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:41.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:41.648710+0000) 2024-03-06T11:15:42.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:42.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:42.648928+0000) 2024-03-06T11:15:43.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:43.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:43.649175+0000) 2024-03-06T11:15:43.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:43.942 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:44.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:44.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:44.649396+0000) 2024-03-06T11:15:45.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:45.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:45.649621+0000) 2024-03-06T11:15:46.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:46.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:46.649866+0000) 2024-03-06T11:15:47.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:47.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:47.650089+0000) 2024-03-06T11:15:48.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:48.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:48.650277+0000) 2024-03-06T11:15:49.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:49.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:49.650508+0000) 2024-03-06T11:15:50.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:50.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:50.650770+0000) 2024-03-06T11:15:51.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:51.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:51.651020+0000) 2024-03-06T11:15:52.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:52.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:52.651225+0000) 2024-03-06T11:15:53.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:53.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:53.651424+0000) 2024-03-06T11:15:54.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:54.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:54.651638+0000) 2024-03-06T11:15:55.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:55.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:55.651943+0000) 2024-03-06T11:15:56.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:56.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:56.652233+0000) 2024-03-06T11:15:57.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:57.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:57.652541+0000) 2024-03-06T11:15:58.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:58.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:58.652769+0000) 2024-03-06T11:15:59.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:15:59.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:59.652998+0000) 2024-03-06T11:16:00.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:00.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:00.653173+0000) 2024-03-06T11:16:01.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:01.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:01.653364+0000) 2024-03-06T11:16:02.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:02.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:02.653600+0000) 2024-03-06T11:16:03.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:03.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:03.653878+0000) 2024-03-06T11:16:04.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:04.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:04.654080+0000) 2024-03-06T11:16:05.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:05.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:05.654271+0000) 2024-03-06T11:16:06.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:06.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:06.654477+0000) 2024-03-06T11:16:07.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:07.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:07.654735+0000) 2024-03-06T11:16:08.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:08.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:08.654924+0000) 2024-03-06T11:16:09.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:09.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:09.655113+0000) 2024-03-06T11:16:10.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:10.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:10.655336+0000) 2024-03-06T11:16:11.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:11.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:11.655559+0000) 2024-03-06T11:16:12.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:12.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:12.655791+0000) 2024-03-06T11:16:13.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:13.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:13.656026+0000) 2024-03-06T11:16:14.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:14.082 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:14.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:14.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:14.656205+0000) 2024-03-06T11:16:15.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:15.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:15.656408+0000) 2024-03-06T11:16:16.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:16.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:16.656654+0000) 2024-03-06T11:16:17.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:17.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:17.656925+0000) 2024-03-06T11:16:18.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:18.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:18.657104+0000) 2024-03-06T11:16:19.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:19.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:19.657365+0000) 2024-03-06T11:16:20.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:20.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:20.657603+0000) 2024-03-06T11:16:21.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:21.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:21.657939+0000) 2024-03-06T11:16:22.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:22.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:22.658176+0000) 2024-03-06T11:16:23.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:23.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:23.658404+0000) 2024-03-06T11:16:24.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:24.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:24.658624+0000) 2024-03-06T11:16:25.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:25.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:25.658885+0000) 2024-03-06T11:16:26.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:26.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:26.659091+0000) 2024-03-06T11:16:27.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:27.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:27.659320+0000) 2024-03-06T11:16:28.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:28.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:28.659544+0000) 2024-03-06T11:16:29.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:29.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:29.659796+0000) 2024-03-06T11:16:30.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:30.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:30.660034+0000) 2024-03-06T11:16:31.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:31.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:31.660212+0000) 2024-03-06T11:16:32.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:32.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:32.660407+0000) 2024-03-06T11:16:33.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:33.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:33.660654+0000) 2024-03-06T11:16:34.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:34.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:34.660899+0000) 2024-03-06T11:16:35.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:35.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:35.661104+0000) 2024-03-06T11:16:36.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:36.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:36.661303+0000) 2024-03-06T11:16:37.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:37.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:37.661502+0000) 2024-03-06T11:16:38.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:38.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:38.661791+0000) 2024-03-06T11:16:39.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:39.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:39.661977+0000) 2024-03-06T11:16:40.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:40.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:40.662199+0000) 2024-03-06T11:16:41.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:41.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:41.662425+0000) 2024-03-06T11:16:42.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:42.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:42.662607+0000) 2024-03-06T11:16:43.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:43.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:43.662846+0000) 2024-03-06T11:16:44.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:44.318 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:44.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:44.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:44.663069+0000) 2024-03-06T11:16:45.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:45.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:45.663295+0000) 2024-03-06T11:16:46.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:46.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:46.663528+0000) 2024-03-06T11:16:47.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:47.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:47.663777+0000) 2024-03-06T11:16:48.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:48.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:48.664012+0000) 2024-03-06T11:16:49.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:49.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:49.664292+0000) 2024-03-06T11:16:50.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:50.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:50.664495+0000) 2024-03-06T11:16:51.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:51.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:51.664745+0000) 2024-03-06T11:16:52.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:52.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:52.664973+0000) 2024-03-06T11:16:53.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:53.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:53.665199+0000) 2024-03-06T11:16:54.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:54.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:54.665421+0000) 2024-03-06T11:16:55.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:55.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:55.665600+0000) 2024-03-06T11:16:56.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:56.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:56.665818+0000) 2024-03-06T11:16:57.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:57.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:57.666051+0000) 2024-03-06T11:16:58.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:58.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:58.666227+0000) 2024-03-06T11:16:59.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:16:59.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:59.666405+0000) 2024-03-06T11:17:00.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:00.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:00.666636+0000) 2024-03-06T11:17:01.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:01.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:01.666860+0000) 2024-03-06T11:17:02.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:02.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:02.667095+0000) 2024-03-06T11:17:03.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:03.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:03.667304+0000) 2024-03-06T11:17:04.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:04.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:04.667541+0000) 2024-03-06T11:17:05.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:05.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:05.667790+0000) 2024-03-06T11:17:06.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:06.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:06.668014+0000) 2024-03-06T11:17:07.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:07.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:07.668197+0000) 2024-03-06T11:17:08.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:08.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:08.668420+0000) 2024-03-06T11:17:09.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:09.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:09.668644+0000) 2024-03-06T11:17:10.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:10.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:10.668932+0000) 2024-03-06T11:17:11.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:11.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:11.669175+0000) 2024-03-06T11:17:12.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:12.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:12.669406+0000) 2024-03-06T11:17:13.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:13.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:13.669612+0000) 2024-03-06T11:17:14.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:14.538 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:14.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:14.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:14.669898+0000) 2024-03-06T11:17:15.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:15.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:15.670089+0000) 2024-03-06T11:17:16.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:16.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:16.670325+0000) 2024-03-06T11:17:17.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:17.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:17.670474+0000) 2024-03-06T11:17:18.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:18.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:18.670731+0000) 2024-03-06T11:17:19.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:19.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:19.670964+0000) 2024-03-06T11:17:20.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:20.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:20.671203+0000) 2024-03-06T11:17:21.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:21.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:21.671398+0000) 2024-03-06T11:17:22.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:22.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:22.671620+0000) 2024-03-06T11:17:23.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:23.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:23.671840+0000) 2024-03-06T11:17:24.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:24.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:24.672060+0000) 2024-03-06T11:17:25.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:25.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:25.672269+0000) 2024-03-06T11:17:26.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:26.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:26.672497+0000) 2024-03-06T11:17:27.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:27.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:27.672765+0000) 2024-03-06T11:17:28.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:28.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:28.672968+0000) 2024-03-06T11:17:29.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:29.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:29.673200+0000) 2024-03-06T11:17:30.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:30.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:30.673380+0000) 2024-03-06T11:17:31.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:31.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:31.673586+0000) 2024-03-06T11:17:32.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:32.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:32.673822+0000) 2024-03-06T11:17:33.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:33.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:33.674055+0000) 2024-03-06T11:17:34.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:34.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:34.674292+0000) 2024-03-06T11:17:35.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:35.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:35.674570+0000) 2024-03-06T11:17:36.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:36.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:36.674841+0000) 2024-03-06T11:17:37.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:37.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:37.675022+0000) 2024-03-06T11:17:38.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:38.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:38.675208+0000) 2024-03-06T11:17:39.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:39.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:39.675449+0000) 2024-03-06T11:17:40.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:40.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:40.675704+0000) 2024-03-06T11:17:41.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:41.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:41.675890+0000) 2024-03-06T11:17:42.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:42.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:42.676118+0000) 2024-03-06T11:17:43.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:43.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:43.676359+0000) 2024-03-06T11:17:44.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:44.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:44.676572+0000) 2024-03-06T11:17:44.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:44.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:45.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:45.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:45.676787+0000) 2024-03-06T11:17:46.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:46.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:46.677031+0000) 2024-03-06T11:17:47.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:47.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:47.677253+0000) 2024-03-06T11:17:48.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:48.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:48.677421+0000) 2024-03-06T11:17:49.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:49.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:49.677632+0000) 2024-03-06T11:17:50.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:50.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:50.677858+0000) 2024-03-06T11:17:51.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:51.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:51.678098+0000) 2024-03-06T11:17:52.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:52.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:52.678279+0000) 2024-03-06T11:17:53.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:53.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:53.678508+0000) 2024-03-06T11:17:54.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:54.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:54.678811+0000) 2024-03-06T11:17:55.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:55.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:55.679089+0000) 2024-03-06T11:17:56.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:56.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:56.679286+0000) 2024-03-06T11:17:57.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:57.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:57.679523+0000) 2024-03-06T11:17:58.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:58.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:58.679817+0000) 2024-03-06T11:17:59.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:17:59.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:59.680019+0000) 2024-03-06T11:18:00.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:00.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:00.680253+0000) 2024-03-06T11:18:01.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:01.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:01.680479+0000) 2024-03-06T11:18:02.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:02.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:02.680733+0000) 2024-03-06T11:18:03.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:03.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:03.680961+0000) 2024-03-06T11:18:04.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:04.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:04.681145+0000) 2024-03-06T11:18:05.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:05.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:05.681369+0000) 2024-03-06T11:18:06.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:06.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:06.681553+0000) 2024-03-06T11:18:07.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:07.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:07.681875+0000) 2024-03-06T11:18:08.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:08.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:08.682092+0000) 2024-03-06T11:18:09.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:09.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:09.682322+0000) 2024-03-06T11:18:10.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:10.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:10.682567+0000) 2024-03-06T11:18:11.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:11.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:11.682804+0000) 2024-03-06T11:18:12.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:12.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:12.683030+0000) 2024-03-06T11:18:13.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:13.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:13.683261+0000) 2024-03-06T11:18:14.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:14.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:14.683484+0000) 2024-03-06T11:18:14.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:14.956 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:15.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:15.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:15.683770+0000) 2024-03-06T11:18:16.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:16.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:16.684008+0000) 2024-03-06T11:18:17.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:17.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:17.684199+0000) 2024-03-06T11:18:18.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:18.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:18.684386+0000) 2024-03-06T11:18:19.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:19.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:19.684612+0000) 2024-03-06T11:18:20.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:20.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:20.684831+0000) 2024-03-06T11:18:21.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:21.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:21.685026+0000) 2024-03-06T11:18:22.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:22.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:22.685225+0000) 2024-03-06T11:18:23.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:23.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:23.685470+0000) 2024-03-06T11:18:24.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:24.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:24.685745+0000) 2024-03-06T11:18:25.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:25.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:25.685987+0000) 2024-03-06T11:18:26.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:26.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:26.686205+0000) 2024-03-06T11:18:27.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:27.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:27.686397+0000) 2024-03-06T11:18:28.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:28.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:28.686589+0000) 2024-03-06T11:18:29.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:29.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:29.686828+0000) 2024-03-06T11:18:30.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:30.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:30.687064+0000) 2024-03-06T11:18:31.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:31.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:31.687313+0000) 2024-03-06T11:18:32.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:32.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:32.687558+0000) 2024-03-06T11:18:33.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:33.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:33.687846+0000) 2024-03-06T11:18:34.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:34.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:34.688127+0000) 2024-03-06T11:18:35.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:35.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:35.688409+0000) 2024-03-06T11:18:36.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:36.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:36.688694+0000) 2024-03-06T11:18:37.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:37.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:37.688896+0000) 2024-03-06T11:18:38.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:38.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:38.689116+0000) 2024-03-06T11:18:39.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:39.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:39.689212+0000) 2024-03-06T11:18:40.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:40.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:40.689418+0000) 2024-03-06T11:18:41.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:41.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:41.689655+0000) 2024-03-06T11:18:42.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:42.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:42.689896+0000) 2024-03-06T11:18:43.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:43.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:43.690082+0000) 2024-03-06T11:18:44.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:44.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:44.690307+0000) 2024-03-06T11:18:45.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:45.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:45.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:45.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:45.690482+0000) 2024-03-06T11:18:46.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:46.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:46.690754+0000) 2024-03-06T11:18:47.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:47.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:47.690946+0000) 2024-03-06T11:18:48.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:48.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:48.691128+0000) 2024-03-06T11:18:49.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:49.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:49.691369+0000) 2024-03-06T11:18:50.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:50.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:50.691614+0000) 2024-03-06T11:18:51.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:51.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:51.691868+0000) 2024-03-06T11:18:52.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:52.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:52.692071+0000) 2024-03-06T11:18:53.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:53.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:53.692278+0000) 2024-03-06T11:18:54.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:54.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:54.692528+0000) 2024-03-06T11:18:55.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:55.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:55.692738+0000) 2024-03-06T11:18:56.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:56.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:56.692955+0000) 2024-03-06T11:18:57.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:57.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:57.693196+0000) 2024-03-06T11:18:58.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:58.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:58.693402+0000) 2024-03-06T11:18:59.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:18:59.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:59.693632+0000) 2024-03-06T11:19:00.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:00.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:00.693868+0000) 2024-03-06T11:19:01.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:01.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:01.694070+0000) 2024-03-06T11:19:02.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:02.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:02.694254+0000) 2024-03-06T11:19:03.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:03.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:03.694438+0000) 2024-03-06T11:19:04.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:04.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:04.694641+0000) 2024-03-06T11:19:05.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:05.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:05.694850+0000) 2024-03-06T11:19:06.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:06.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:06.695092+0000) 2024-03-06T11:19:07.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:07.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:07.695303+0000) 2024-03-06T11:19:08.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:08.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:08.695520+0000) 2024-03-06T11:19:09.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:09.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:09.695774+0000) 2024-03-06T11:19:10.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:10.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:10.695910+0000) 2024-03-06T11:19:11.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:11.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:11.696103+0000) 2024-03-06T11:19:12.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:12.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:12.696325+0000) 2024-03-06T11:19:13.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:13.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:13.696569+0000) 2024-03-06T11:19:14.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:14.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:14.696821+0000) 2024-03-06T11:19:15.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:15.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:15.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:15.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:15.697025+0000) 2024-03-06T11:19:16.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:16.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:16.697215+0000) 2024-03-06T11:19:17.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:17.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:17.697459+0000) 2024-03-06T11:19:18.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:18.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:18.697649+0000) 2024-03-06T11:19:19.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:19.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:19.697930+0000) 2024-03-06T11:19:20.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:20.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:20.698150+0000) 2024-03-06T11:19:21.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:21.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:21.698377+0000) 2024-03-06T11:19:22.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:22.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:22.698589+0000) 2024-03-06T11:19:23.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:23.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:23.698775+0000) 2024-03-06T11:19:24.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:24.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:24.698933+0000) 2024-03-06T11:19:25.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:25.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:25.699117+0000) 2024-03-06T11:19:26.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:26.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:26.699312+0000) 2024-03-06T11:19:27.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:27.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:27.699539+0000) 2024-03-06T11:19:28.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:28.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:28.699771+0000) 2024-03-06T11:19:29.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:29.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:29.700005+0000) 2024-03-06T11:19:30.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:30.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:30.700234+0000) 2024-03-06T11:19:31.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:31.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:31.700422+0000) 2024-03-06T11:19:32.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:32.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:32.700590+0000) 2024-03-06T11:19:33.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:33.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:33.700783+0000) 2024-03-06T11:19:34.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:34.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:34.700968+0000) 2024-03-06T11:19:35.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:35.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:35.701158+0000) 2024-03-06T11:19:36.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:36.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:36.701390+0000) 2024-03-06T11:19:37.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:37.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:37.701575+0000) 2024-03-06T11:19:38.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:38.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:38.701789+0000) 2024-03-06T11:19:39.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:39.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:39.701990+0000) 2024-03-06T11:19:40.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:40.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:40.702209+0000) 2024-03-06T11:19:41.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:41.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:41.702429+0000) 2024-03-06T11:19:42.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:42.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:42.702637+0000) 2024-03-06T11:19:43.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:43.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:43.702841+0000) 2024-03-06T11:19:44.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:44.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:44.703063+0000) 2024-03-06T11:19:45.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:45.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:45.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:45.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:45.703275+0000) 2024-03-06T11:19:46.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:46.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:46.703539+0000) 2024-03-06T11:19:47.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:47.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:47.703778+0000) 2024-03-06T11:19:48.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:48.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:48.703954+0000) 2024-03-06T11:19:49.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:49.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:49.704237+0000) 2024-03-06T11:19:50.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:50.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:50.704461+0000) 2024-03-06T11:19:51.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:51.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:51.704734+0000) 2024-03-06T11:19:52.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:52.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:52.704956+0000) 2024-03-06T11:19:53.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:53.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:53.705169+0000) 2024-03-06T11:19:54.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:54.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:54.705394+0000) 2024-03-06T11:19:55.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:55.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:55.705577+0000) 2024-03-06T11:19:56.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:56.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:56.705832+0000) 2024-03-06T11:19:57.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:57.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:57.706039+0000) 2024-03-06T11:19:58.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:58.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:58.706262+0000) 2024-03-06T11:19:59.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:19:59.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:59.706484+0000) 2024-03-06T11:20:00.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:00.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:00.706733+0000) 2024-03-06T11:20:01.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:01.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:01.706969+0000) 2024-03-06T11:20:02.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:02.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:02.707190+0000) 2024-03-06T11:20:03.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:03.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:03.707412+0000) 2024-03-06T11:20:04.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:04.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:04.707627+0000) 2024-03-06T11:20:05.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:05.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:05.707858+0000) 2024-03-06T11:20:06.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:06.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:06.708105+0000) 2024-03-06T11:20:07.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:07.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:07.708333+0000) 2024-03-06T11:20:08.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:08.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:08.708552+0000) 2024-03-06T11:20:09.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:09.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:09.708783+0000) 2024-03-06T11:20:10.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:10.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:10.709021+0000) 2024-03-06T11:20:11.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:11.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:11.709268+0000) 2024-03-06T11:20:12.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:12.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:12.709447+0000) 2024-03-06T11:20:13.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:13.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:13.709699+0000) 2024-03-06T11:20:14.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:14.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:14.709927+0000) 2024-03-06T11:20:15.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:15.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:15.710153+0000) 2024-03-06T11:20:15.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:15.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:16.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:16.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:16.710415+0000) 2024-03-06T11:20:17.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:17.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:17.710640+0000) 2024-03-06T11:20:18.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:18.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:18.710857+0000) 2024-03-06T11:20:19.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:19.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:19.711081+0000) 2024-03-06T11:20:20.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:20.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:20.711267+0000) 2024-03-06T11:20:21.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:21.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:21.711510+0000) 2024-03-06T11:20:22.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:22.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:22.711762+0000) 2024-03-06T11:20:23.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:23.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:23.711989+0000) 2024-03-06T11:20:24.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:24.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:24.712216+0000) 2024-03-06T11:20:25.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:25.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:25.712394+0000) 2024-03-06T11:20:26.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:26.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:26.712701+0000) 2024-03-06T11:20:27.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:27.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:27.712895+0000) 2024-03-06T11:20:28.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:28.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:28.713095+0000) 2024-03-06T11:20:29.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:29.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:29.713335+0000) 2024-03-06T11:20:30.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:30.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:30.713556+0000) 2024-03-06T11:20:31.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:31.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:31.713803+0000) 2024-03-06T11:20:32.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:32.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:32.713985+0000) 2024-03-06T11:20:33.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:33.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:33.714201+0000) 2024-03-06T11:20:34.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:34.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:34.714432+0000) 2024-03-06T11:20:35.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:35.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:35.714695+0000) 2024-03-06T11:20:36.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:36.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:36.714967+0000) 2024-03-06T11:20:37.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:37.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:37.715245+0000) 2024-03-06T11:20:38.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:38.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:38.715477+0000) 2024-03-06T11:20:39.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:39.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:39.715743+0000) 2024-03-06T11:20:40.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:40.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:40.715966+0000) 2024-03-06T11:20:41.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:41.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:41.716205+0000) 2024-03-06T11:20:42.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:42.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:42.716429+0000) 2024-03-06T11:20:43.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:43.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:43.716655+0000) 2024-03-06T11:20:44.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:44.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:44.716924+0000) 2024-03-06T11:20:45.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:45.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:45.717185+0000) 2024-03-06T11:20:46.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:46.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:46.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:46.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:46.717489+0000) 2024-03-06T11:20:47.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:47.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:47.717815+0000) 2024-03-06T11:20:48.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:48.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:48.718097+0000) 2024-03-06T11:20:49.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:49.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:49.718296+0000) 2024-03-06T11:20:50.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:50.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:50.718505+0000) 2024-03-06T11:20:51.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:51.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:51.718843+0000) 2024-03-06T11:20:52.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:52.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:52.719049+0000) 2024-03-06T11:20:53.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:53.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:53.719288+0000) 2024-03-06T11:20:54.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:54.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:54.719475+0000) 2024-03-06T11:20:55.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:55.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:55.719750+0000) 2024-03-06T11:20:56.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:56.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:56.720007+0000) 2024-03-06T11:20:57.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:57.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:57.720213+0000) 2024-03-06T11:20:58.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:58.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:58.720468+0000) 2024-03-06T11:20:59.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:20:59.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:59.720734+0000) 2024-03-06T11:21:00.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:00.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:00.720949+0000) 2024-03-06T11:21:01.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:01.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:01.721176+0000) 2024-03-06T11:21:02.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:02.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:02.721361+0000) 2024-03-06T11:21:03.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:03.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:03.721581+0000) 2024-03-06T11:21:04.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:04.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:04.721830+0000) 2024-03-06T11:21:05.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:05.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:05.722061+0000) 2024-03-06T11:21:06.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:06.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:06.722327+0000) 2024-03-06T11:21:07.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:07.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:07.722560+0000) 2024-03-06T11:21:08.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:08.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:08.722779+0000) 2024-03-06T11:21:09.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:09.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:09.722997+0000) 2024-03-06T11:21:10.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:10.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:10.723219+0000) 2024-03-06T11:21:11.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:11.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:11.723441+0000) 2024-03-06T11:21:12.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:12.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:12.723703+0000) 2024-03-06T11:21:13.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:13.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:13.723964+0000) 2024-03-06T11:21:14.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:14.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:14.724181+0000) 2024-03-06T11:21:15.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:15.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:15.724412+0000) 2024-03-06T11:21:16.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:16.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:16.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:16.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:16.724750+0000) 2024-03-06T11:21:17.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:17.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:17.724987+0000) 2024-03-06T11:21:18.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:18.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:18.725228+0000) 2024-03-06T11:21:19.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:19.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:19.725506+0000) 2024-03-06T11:21:20.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:20.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:20.725813+0000) 2024-03-06T11:21:21.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:21.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:21.726002+0000) 2024-03-06T11:21:22.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:22.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:22.726181+0000) 2024-03-06T11:21:23.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:23.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:23.726449+0000) 2024-03-06T11:21:24.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:24.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:24.726756+0000) 2024-03-06T11:21:25.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:25.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:25.727002+0000) 2024-03-06T11:21:26.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:26.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:26.727261+0000) 2024-03-06T11:21:27.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:27.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:27.727544+0000) 2024-03-06T11:21:28.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:28.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:28.727827+0000) 2024-03-06T11:21:29.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:29.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:29.728057+0000) 2024-03-06T11:21:30.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:30.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:30.728274+0000) 2024-03-06T11:21:31.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:31.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:31.728506+0000) 2024-03-06T11:21:32.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:32.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:32.728755+0000) 2024-03-06T11:21:33.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:33.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:33.728973+0000) 2024-03-06T11:21:34.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:34.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:34.729208+0000) 2024-03-06T11:21:35.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:35.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:35.729484+0000) 2024-03-06T11:21:36.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:36.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:36.729763+0000) 2024-03-06T11:21:37.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:37.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:37.730062+0000) 2024-03-06T11:21:38.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:38.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:38.730308+0000) 2024-03-06T11:21:39.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:39.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:39.730538+0000) 2024-03-06T11:21:40.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:40.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:40.730779+0000) 2024-03-06T11:21:41.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:41.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:41.731011+0000) 2024-03-06T11:21:42.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:42.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:42.731210+0000) 2024-03-06T11:21:43.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:43.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:43.731457+0000) 2024-03-06T11:21:44.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:44.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:44.731708+0000) 2024-03-06T11:21:45.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:45.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:45.731828+0000) 2024-03-06T11:21:46.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:46.436 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:46.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:46.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:46.732076+0000) 2024-03-06T11:21:47.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:47.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:47.732288+0000) 2024-03-06T11:21:48.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:48.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:48.732518+0000) 2024-03-06T11:21:49.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:49.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:49.732785+0000) 2024-03-06T11:21:50.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:50.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:50.733028+0000) 2024-03-06T11:21:51.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:51.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:51.733241+0000) 2024-03-06T11:21:52.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:52.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:52.733521+0000) 2024-03-06T11:21:53.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:53.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:53.733786+0000) 2024-03-06T11:21:54.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:54.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:54.734027+0000) 2024-03-06T11:21:55.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:55.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:55.734252+0000) 2024-03-06T11:21:56.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:56.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:56.734522+0000) 2024-03-06T11:21:57.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:57.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:57.734777+0000) 2024-03-06T11:21:58.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:58.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:58.734999+0000) 2024-03-06T11:21:59.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:21:59.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:59.735222+0000) 2024-03-06T11:22:00.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:00.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:00.735438+0000) 2024-03-06T11:22:01.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:01.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:01.735732+0000) 2024-03-06T11:22:02.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:02.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:02.736000+0000) 2024-03-06T11:22:03.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:03.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:03.736245+0000) 2024-03-06T11:22:04.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:04.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:04.736516+0000) 2024-03-06T11:22:05.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:05.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:05.736821+0000) 2024-03-06T11:22:06.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:06.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:06.736988+0000) 2024-03-06T11:22:07.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:07.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:07.737215+0000) 2024-03-06T11:22:08.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:08.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:08.737445+0000) 2024-03-06T11:22:09.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:09.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:09.737650+0000) 2024-03-06T11:22:10.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:10.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:10.737862+0000) 2024-03-06T11:22:11.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:11.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:11.738095+0000) 2024-03-06T11:22:12.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:12.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:12.738320+0000) 2024-03-06T11:22:13.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:13.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:13.738520+0000) 2024-03-06T11:22:14.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:14.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:14.738776+0000) 2024-03-06T11:22:15.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:15.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:15.738956+0000) 2024-03-06T11:22:16.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:16.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:16.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:16.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:16.739136+0000) 2024-03-06T11:22:17.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:17.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:17.739311+0000) 2024-03-06T11:22:18.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:18.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:18.739567+0000) 2024-03-06T11:22:19.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:19.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:19.739788+0000) 2024-03-06T11:22:20.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:20.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:20.740010+0000) 2024-03-06T11:22:21.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:21.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:21.740232+0000) 2024-03-06T11:22:22.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:22.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:22.740460+0000) 2024-03-06T11:22:23.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:23.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:23.740777+0000) 2024-03-06T11:22:24.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:24.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:24.741005+0000) 2024-03-06T11:22:25.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:25.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:25.741218+0000) 2024-03-06T11:22:26.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:26.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:26.741471+0000) 2024-03-06T11:22:27.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:27.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:27.741750+0000) 2024-03-06T11:22:28.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:28.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:28.741953+0000) 2024-03-06T11:22:29.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:29.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:29.742185+0000) 2024-03-06T11:22:30.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:30.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:30.742400+0000) 2024-03-06T11:22:31.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:31.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:31.742624+0000) 2024-03-06T11:22:32.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:32.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:32.742855+0000) 2024-03-06T11:22:33.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:33.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:33.743028+0000) 2024-03-06T11:22:34.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:34.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:34.743243+0000) 2024-03-06T11:22:35.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:35.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:35.743438+0000) 2024-03-06T11:22:36.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:36.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:36.743729+0000) 2024-03-06T11:22:37.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:37.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:37.743958+0000) 2024-03-06T11:22:38.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:38.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:38.744193+0000) 2024-03-06T11:22:39.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:39.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:39.744440+0000) 2024-03-06T11:22:40.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:40.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:40.744660+0000) 2024-03-06T11:22:41.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:41.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:41.744883+0000) 2024-03-06T11:22:42.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:42.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:42.745110+0000) 2024-03-06T11:22:43.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:43.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:43.745336+0000) 2024-03-06T11:22:44.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:44.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:44.745555+0000) 2024-03-06T11:22:45.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:45.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:45.745786+0000) 2024-03-06T11:22:46.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:46.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:46.746025+0000) 2024-03-06T11:22:46.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:46.819 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:47.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:47.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:47.746252+0000) 2024-03-06T11:22:48.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:48.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:48.746473+0000) 2024-03-06T11:22:49.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:49.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:49.746644+0000) 2024-03-06T11:22:50.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:50.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:50.746832+0000) 2024-03-06T11:22:51.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:51.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:51.747005+0000) 2024-03-06T11:22:52.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:52.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:52.747210+0000) 2024-03-06T11:22:53.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:53.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:53.747445+0000) 2024-03-06T11:22:54.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:54.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:54.747713+0000) 2024-03-06T11:22:55.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:55.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:55.747925+0000) 2024-03-06T11:22:56.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:56.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:56.748161+0000) 2024-03-06T11:22:57.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:57.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:57.748385+0000) 2024-03-06T11:22:58.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:58.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:58.748616+0000) 2024-03-06T11:22:59.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:22:59.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:59.748861+0000) 2024-03-06T11:23:00.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:00.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:00.749136+0000) 2024-03-06T11:23:01.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:01.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:01.749400+0000) 2024-03-06T11:23:02.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:02.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:02.749703+0000) 2024-03-06T11:23:03.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:03.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:03.749941+0000) 2024-03-06T11:23:04.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:04.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:04.750150+0000) 2024-03-06T11:23:05.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:05.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:05.750405+0000) 2024-03-06T11:23:06.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:06.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:06.750752+0000) 2024-03-06T11:23:07.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:07.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:07.750980+0000) 2024-03-06T11:23:08.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:08.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:08.751200+0000) 2024-03-06T11:23:09.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:09.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:09.751400+0000) 2024-03-06T11:23:10.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:10.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:10.751626+0000) 2024-03-06T11:23:11.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:11.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:11.751858+0000) 2024-03-06T11:23:12.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:12.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:12.752133+0000) 2024-03-06T11:23:13.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:13.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:13.752376+0000) 2024-03-06T11:23:14.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:14.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:14.752564+0000) 2024-03-06T11:23:15.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:15.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:15.752767+0000) 2024-03-06T11:23:16.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:16.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:16.752977+0000) 2024-03-06T11:23:17.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:17.050 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:17.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:17.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:17.753199+0000) 2024-03-06T11:23:18.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:18.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:18.753428+0000) 2024-03-06T11:23:19.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:19.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:19.753647+0000) 2024-03-06T11:23:20.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:20.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:20.753861+0000) 2024-03-06T11:23:21.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:21.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:21.754107+0000) 2024-03-06T11:23:22.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:22.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:22.754333+0000) 2024-03-06T11:23:23.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:23.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:23.754544+0000) 2024-03-06T11:23:24.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:24.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:24.754776+0000) 2024-03-06T11:23:25.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:25.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:25.754994+0000) 2024-03-06T11:23:26.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:26.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:26.755227+0000) 2024-03-06T11:23:27.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:27.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:27.755464+0000) 2024-03-06T11:23:28.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:28.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:28.755828+0000) 2024-03-06T11:23:29.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:29.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:29.756075+0000) 2024-03-06T11:23:30.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:30.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:30.756307+0000) 2024-03-06T11:23:31.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:31.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:31.756545+0000) 2024-03-06T11:23:32.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:32.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:32.756778+0000) 2024-03-06T11:23:33.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:33.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:33.757013+0000) 2024-03-06T11:23:34.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:34.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:34.757192+0000) 2024-03-06T11:23:35.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:35.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:35.757374+0000) 2024-03-06T11:23:36.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:36.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:36.757537+0000) 2024-03-06T11:23:37.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:37.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:37.757821+0000) 2024-03-06T11:23:38.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:38.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:38.758032+0000) 2024-03-06T11:23:39.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:39.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:39.758271+0000) 2024-03-06T11:23:40.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:40.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:40.758497+0000) 2024-03-06T11:23:41.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:41.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:41.758711+0000) 2024-03-06T11:23:42.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:42.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:42.758962+0000) 2024-03-06T11:23:43.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:43.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:43.759195+0000) 2024-03-06T11:23:44.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:44.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:44.759399+0000) 2024-03-06T11:23:45.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:45.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:45.759639+0000) 2024-03-06T11:23:46.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:46.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:46.759863+0000) 2024-03-06T11:23:47.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:47.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:47.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:47.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:47.760064+0000) 2024-03-06T11:23:48.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:48.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:48.760266+0000) 2024-03-06T11:23:49.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:49.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:49.760489+0000) 2024-03-06T11:23:50.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:50.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:50.760739+0000) 2024-03-06T11:23:51.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:51.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:51.760915+0000) 2024-03-06T11:23:52.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:52.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:52.761076+0000) 2024-03-06T11:23:53.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:53.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:53.761333+0000) 2024-03-06T11:23:54.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:54.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:54.761591+0000) 2024-03-06T11:23:55.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:55.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:55.761842+0000) 2024-03-06T11:23:56.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:56.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:56.762108+0000) 2024-03-06T11:23:57.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:57.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:57.762333+0000) 2024-03-06T11:23:58.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:58.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:58.762558+0000) 2024-03-06T11:23:59.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:23:59.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:59.762790+0000) 2024-03-06T11:24:00.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:00.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:00.762968+0000) 2024-03-06T11:24:01.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:01.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:01.763146+0000) 2024-03-06T11:24:02.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:02.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:02.763328+0000) 2024-03-06T11:24:03.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:03.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:03.763552+0000) 2024-03-06T11:24:04.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:04.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:04.763772+0000) 2024-03-06T11:24:05.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:05.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:05.764046+0000) 2024-03-06T11:24:06.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:06.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:06.764238+0000) 2024-03-06T11:24:07.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:07.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:07.764488+0000) 2024-03-06T11:24:08.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:08.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:08.764757+0000) 2024-03-06T11:24:09.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:09.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:09.765019+0000) 2024-03-06T11:24:10.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:10.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:10.765255+0000) 2024-03-06T11:24:11.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:11.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:11.765449+0000) 2024-03-06T11:24:12.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:12.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:12.765711+0000) 2024-03-06T11:24:13.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:13.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:13.765915+0000) 2024-03-06T11:24:14.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:14.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:14.766115+0000) 2024-03-06T11:24:15.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:15.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:15.766343+0000) 2024-03-06T11:24:16.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:16.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:16.766572+0000) 2024-03-06T11:24:17.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:17.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:17.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:17.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:17.766816+0000) 2024-03-06T11:24:18.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:18.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:18.767016+0000) 2024-03-06T11:24:19.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:19.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:19.767205+0000) 2024-03-06T11:24:20.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:20.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:20.767383+0000) 2024-03-06T11:24:21.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:21.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:21.767593+0000) 2024-03-06T11:24:22.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:22.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:22.767829+0000) 2024-03-06T11:24:23.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:23.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:23.768022+0000) 2024-03-06T11:24:24.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:24.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:24.768210+0000) 2024-03-06T11:24:25.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:25.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:25.768391+0000) 2024-03-06T11:24:26.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:26.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:26.768597+0000) 2024-03-06T11:24:27.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:27.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:27.768800+0000) 2024-03-06T11:24:28.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:28.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:28.769012+0000) 2024-03-06T11:24:29.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:29.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:29.769211+0000) 2024-03-06T11:24:30.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:30.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:30.769438+0000) 2024-03-06T11:24:31.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:31.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:31.769691+0000) 2024-03-06T11:24:32.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:32.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:32.769883+0000) 2024-03-06T11:24:33.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:33.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:33.770109+0000) 2024-03-06T11:24:34.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:34.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:34.770386+0000) 2024-03-06T11:24:35.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:35.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:35.770653+0000) 2024-03-06T11:24:36.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:36.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:36.771010+0000) 2024-03-06T11:24:37.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:37.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:37.771288+0000) 2024-03-06T11:24:38.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:38.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:38.771561+0000) 2024-03-06T11:24:39.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:39.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:39.771837+0000) 2024-03-06T11:24:40.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:40.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:40.772087+0000) 2024-03-06T11:24:41.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:41.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:41.772284+0000) 2024-03-06T11:24:42.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:42.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:42.772544+0000) 2024-03-06T11:24:43.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:43.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:43.772827+0000) 2024-03-06T11:24:44.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:44.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:44.773018+0000) 2024-03-06T11:24:45.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:45.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:45.773288+0000) 2024-03-06T11:24:46.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:46.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:46.773502+0000) 2024-03-06T11:24:47.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:47.661 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:47.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:47.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:47.773715+0000) 2024-03-06T11:24:48.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:48.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:48.773905+0000) 2024-03-06T11:24:49.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:49.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:49.774092+0000) 2024-03-06T11:24:50.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:50.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:50.774293+0000) 2024-03-06T11:24:51.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:51.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:51.774510+0000) 2024-03-06T11:24:52.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:52.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:52.774768+0000) 2024-03-06T11:24:53.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:53.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:53.774968+0000) 2024-03-06T11:24:54.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:54.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:54.775146+0000) 2024-03-06T11:24:55.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:55.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:55.775368+0000) 2024-03-06T11:24:56.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:56.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:56.775601+0000) 2024-03-06T11:24:57.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:57.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:57.775799+0000) 2024-03-06T11:24:58.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:58.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:58.776013+0000) 2024-03-06T11:24:59.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:24:59.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:59.776199+0000) 2024-03-06T11:25:00.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:00.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:00.776375+0000) 2024-03-06T11:25:01.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:01.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:01.776564+0000) 2024-03-06T11:25:02.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:02.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:02.776814+0000) 2024-03-06T11:25:03.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:03.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:03.777060+0000) 2024-03-06T11:25:04.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:04.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:04.777292+0000) 2024-03-06T11:25:05.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:05.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:05.777521+0000) 2024-03-06T11:25:06.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:06.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:06.777783+0000) 2024-03-06T11:25:07.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:07.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:07.777979+0000) 2024-03-06T11:25:08.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:08.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:08.778254+0000) 2024-03-06T11:25:09.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:09.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:09.778416+0000) 2024-03-06T11:25:10.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:10.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:10.778592+0000) 2024-03-06T11:25:11.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:11.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:11.778843+0000) 2024-03-06T11:25:12.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:12.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:12.779060+0000) 2024-03-06T11:25:13.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:13.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:13.779343+0000) 2024-03-06T11:25:14.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:14.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:14.779616+0000) 2024-03-06T11:25:15.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:15.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:15.779900+0000) 2024-03-06T11:25:16.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:16.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:16.780192+0000) 2024-03-06T11:25:17.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:17.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:17.780435+0000) 2024-03-06T11:25:17.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:17.882 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:18.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:18.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:18.780627+0000) 2024-03-06T11:25:19.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:19.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:19.780813+0000) 2024-03-06T11:25:20.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:20.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:20.781010+0000) 2024-03-06T11:25:21.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:21.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:21.781208+0000) 2024-03-06T11:25:22.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:22.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:22.781444+0000) 2024-03-06T11:25:23.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:23.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:23.781710+0000) 2024-03-06T11:25:24.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:24.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:24.781922+0000) 2024-03-06T11:25:25.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:25.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:25.782127+0000) 2024-03-06T11:25:26.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:26.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:26.782385+0000) 2024-03-06T11:25:27.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:27.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:27.782561+0000) 2024-03-06T11:25:28.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:28.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:28.782790+0000) 2024-03-06T11:25:29.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:29.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:29.782952+0000) 2024-03-06T11:25:30.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:30.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:30.783201+0000) 2024-03-06T11:25:31.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:31.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:31.783440+0000) 2024-03-06T11:25:32.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:32.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:32.783661+0000) 2024-03-06T11:25:33.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:33.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:33.783891+0000) 2024-03-06T11:25:34.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:34.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:34.784109+0000) 2024-03-06T11:25:35.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:35.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:35.784301+0000) 2024-03-06T11:25:36.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:36.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:36.784502+0000) 2024-03-06T11:25:37.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:37.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:37.784794+0000) 2024-03-06T11:25:38.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:38.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:38.784999+0000) 2024-03-06T11:25:39.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:39.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:39.785189+0000) 2024-03-06T11:25:40.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:40.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:40.785415+0000) 2024-03-06T11:25:41.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:41.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:41.785644+0000) 2024-03-06T11:25:42.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:42.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:42.785885+0000) 2024-03-06T11:25:43.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:43.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:43.786092+0000) 2024-03-06T11:25:44.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:44.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:44.786285+0000) 2024-03-06T11:25:45.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:45.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:45.786518+0000) 2024-03-06T11:25:46.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:46.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:46.786790+0000) 2024-03-06T11:25:47.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:47.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:47.786994+0000) 2024-03-06T11:25:48.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:48.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:48.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:48.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:48.787187+0000) 2024-03-06T11:25:49.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:49.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:49.787424+0000) 2024-03-06T11:25:50.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:50.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:50.787706+0000) 2024-03-06T11:25:51.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:51.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:51.787998+0000) 2024-03-06T11:25:52.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:52.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:52.788216+0000) 2024-03-06T11:25:53.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:53.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:53.788424+0000) 2024-03-06T11:25:54.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:54.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:54.788705+0000) 2024-03-06T11:25:55.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:55.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:55.788891+0000) 2024-03-06T11:25:56.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:56.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:56.789087+0000) 2024-03-06T11:25:57.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:57.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:57.789282+0000) 2024-03-06T11:25:58.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:58.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:58.789464+0000) 2024-03-06T11:25:59.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:25:59.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:59.789646+0000) 2024-03-06T11:26:00.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:00.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:00.789912+0000) 2024-03-06T11:26:01.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:01.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:01.790100+0000) 2024-03-06T11:26:02.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:02.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:02.790333+0000) 2024-03-06T11:26:03.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:03.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:03.790560+0000) 2024-03-06T11:26:04.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:04.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:04.790792+0000) 2024-03-06T11:26:05.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:05.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:05.790938+0000) 2024-03-06T11:26:06.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:06.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:06.791173+0000) 2024-03-06T11:26:07.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:07.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:07.791419+0000) 2024-03-06T11:26:08.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:08.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:08.791693+0000) 2024-03-06T11:26:09.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:09.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:09.791932+0000) 2024-03-06T11:26:10.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:10.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:10.792092+0000) 2024-03-06T11:26:11.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:11.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:11.792275+0000) 2024-03-06T11:26:12.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:12.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:12.792508+0000) 2024-03-06T11:26:13.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:13.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:13.792775+0000) 2024-03-06T11:26:14.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:14.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:14.792965+0000) 2024-03-06T11:26:15.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:15.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:15.793190+0000) 2024-03-06T11:26:16.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:16.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:16.793432+0000) 2024-03-06T11:26:17.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:17.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:17.793702+0000) 2024-03-06T11:26:18.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:18.303 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:18.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:18.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:18.793926+0000) 2024-03-06T11:26:19.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:19.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:19.794107+0000) 2024-03-06T11:26:20.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:20.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:20.794298+0000) 2024-03-06T11:26:21.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:21.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:21.794452+0000) 2024-03-06T11:26:22.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:22.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:22.794723+0000) 2024-03-06T11:26:23.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:23.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:23.794957+0000) 2024-03-06T11:26:24.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:24.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:24.795153+0000) 2024-03-06T11:26:25.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:25.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:25.795386+0000) 2024-03-06T11:26:26.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:26.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:26.795777+0000) 2024-03-06T11:26:27.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:27.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:27.796045+0000) 2024-03-06T11:26:28.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:28.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:28.796271+0000) 2024-03-06T11:26:29.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:29.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:29.796502+0000) 2024-03-06T11:26:30.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:30.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:30.796769+0000) 2024-03-06T11:26:31.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:31.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:31.796993+0000) 2024-03-06T11:26:32.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:32.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:32.797239+0000) 2024-03-06T11:26:33.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:33.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:33.797474+0000) 2024-03-06T11:26:34.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:34.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:34.797617+0000) 2024-03-06T11:26:35.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:35.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:35.797853+0000) 2024-03-06T11:26:36.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:36.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:36.798029+0000) 2024-03-06T11:26:37.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:37.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:37.798202+0000) 2024-03-06T11:26:38.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:38.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:38.798352+0000) 2024-03-06T11:26:39.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:39.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:39.798475+0000) 2024-03-06T11:26:40.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:40.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:40.798564+0000) 2024-03-06T11:26:41.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:41.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:41.798793+0000) 2024-03-06T11:26:42.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:42.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:42.799052+0000) 2024-03-06T11:26:43.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:43.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:43.799305+0000) 2024-03-06T11:26:44.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:44.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:44.799455+0000) 2024-03-06T11:26:45.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:45.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:45.799602+0000) 2024-03-06T11:26:46.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:46.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:46.799832+0000) 2024-03-06T11:26:47.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:47.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:47.800085+0000) 2024-03-06T11:26:48.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:48.512 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:48.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:48.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:48.800314+0000) 2024-03-06T11:26:49.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:49.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:49.800556+0000) 2024-03-06T11:26:50.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:50.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:50.800802+0000) 2024-03-06T11:26:51.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:51.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:51.801042+0000) 2024-03-06T11:26:52.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:52.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:52.801266+0000) 2024-03-06T11:26:53.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:53.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:53.801500+0000) 2024-03-06T11:26:54.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:54.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:54.801774+0000) 2024-03-06T11:26:55.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:55.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:55.802008+0000) 2024-03-06T11:26:56.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:56.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:56.802202+0000) 2024-03-06T11:26:57.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:57.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:57.802414+0000) 2024-03-06T11:26:58.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:58.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:58.802622+0000) 2024-03-06T11:26:59.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:26:59.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:59.802933+0000) 2024-03-06T11:27:00.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:00.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:00.803210+0000) 2024-03-06T11:27:01.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:01.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:01.803478+0000) 2024-03-06T11:27:02.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:02.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:02.803781+0000) 2024-03-06T11:27:03.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:03.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:03.803954+0000) 2024-03-06T11:27:04.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:04.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:04.804127+0000) 2024-03-06T11:27:05.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:05.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:05.804358+0000) 2024-03-06T11:27:06.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:06.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:06.804569+0000) 2024-03-06T11:27:07.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:07.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:07.804826+0000) 2024-03-06T11:27:08.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:08.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:08.805102+0000) 2024-03-06T11:27:09.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:09.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:09.805371+0000) 2024-03-06T11:27:10.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:10.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:10.805601+0000) 2024-03-06T11:27:11.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:11.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:11.805844+0000) 2024-03-06T11:27:12.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:12.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:12.806032+0000) 2024-03-06T11:27:13.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:13.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:13.806236+0000) 2024-03-06T11:27:14.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:14.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:14.806459+0000) 2024-03-06T11:27:15.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:15.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:15.806717+0000) 2024-03-06T11:27:16.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:16.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:16.806914+0000) 2024-03-06T11:27:17.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:17.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:17.807138+0000) 2024-03-06T11:27:18.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:18.643 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:18.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:18.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:18.807379+0000) 2024-03-06T11:27:19.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:19.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:19.807581+0000) 2024-03-06T11:27:20.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:20.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:20.807839+0000) 2024-03-06T11:27:21.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:21.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:21.808023+0000) 2024-03-06T11:27:22.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:22.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:22.808190+0000) 2024-03-06T11:27:23.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:23.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:23.808362+0000) 2024-03-06T11:27:24.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:24.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:24.808542+0000) 2024-03-06T11:27:25.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:25.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:25.808812+0000) 2024-03-06T11:27:26.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:26.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:26.809045+0000) 2024-03-06T11:27:27.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:27.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:27.809272+0000) 2024-03-06T11:27:28.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:28.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:28.809502+0000) 2024-03-06T11:27:29.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:29.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:29.809779+0000) 2024-03-06T11:27:30.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:30.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:30.810004+0000) 2024-03-06T11:27:31.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:31.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:31.810231+0000) 2024-03-06T11:27:32.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:32.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:32.810458+0000) 2024-03-06T11:27:33.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:33.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:33.810715+0000) 2024-03-06T11:27:34.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:34.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:34.810957+0000) 2024-03-06T11:27:35.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:35.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:35.811162+0000) 2024-03-06T11:27:36.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:36.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:36.811370+0000) 2024-03-06T11:27:37.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:37.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:37.811598+0000) 2024-03-06T11:27:38.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:38.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:38.811888+0000) 2024-03-06T11:27:39.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:39.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:39.812129+0000) 2024-03-06T11:27:40.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:40.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:40.812356+0000) 2024-03-06T11:27:41.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:41.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:41.812584+0000) 2024-03-06T11:27:42.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:42.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:42.812836+0000) 2024-03-06T11:27:43.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:43.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:43.813087+0000) 2024-03-06T11:27:44.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:44.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:44.813303+0000) 2024-03-06T11:27:45.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:45.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:45.813562+0000) 2024-03-06T11:27:46.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:46.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:46.813828+0000) 2024-03-06T11:27:47.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:47.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:47.814071+0000) 2024-03-06T11:27:48.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:48.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:48.814300+0000) 2024-03-06T11:27:48.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:48.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:49.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:49.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:49.814593+0000) 2024-03-06T11:27:50.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:50.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:50.814792+0000) 2024-03-06T11:27:51.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:51.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:51.815032+0000) 2024-03-06T11:27:52.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:52.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:52.815277+0000) 2024-03-06T11:27:53.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:53.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:53.815453+0000) 2024-03-06T11:27:54.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:54.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:54.815646+0000) 2024-03-06T11:27:55.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:55.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:55.815868+0000) 2024-03-06T11:27:56.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:56.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:56.816095+0000) 2024-03-06T11:27:57.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:57.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:57.816314+0000) 2024-03-06T11:27:58.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:58.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:58.816534+0000) 2024-03-06T11:27:59.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:27:59.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:59.816792+0000) 2024-03-06T11:28:00.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:00.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:00.817022+0000) 2024-03-06T11:28:01.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:01.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:01.817252+0000) 2024-03-06T11:28:02.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:02.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:02.817480+0000) 2024-03-06T11:28:03.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:03.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:03.817750+0000) 2024-03-06T11:28:04.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:04.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:04.817965+0000) 2024-03-06T11:28:05.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:05.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:05.818194+0000) 2024-03-06T11:28:06.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:06.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:06.818423+0000) 2024-03-06T11:28:07.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:07.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:07.818650+0000) 2024-03-06T11:28:08.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:08.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:08.818874+0000) 2024-03-06T11:28:09.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:09.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:09.819108+0000) 2024-03-06T11:28:10.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:10.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:10.819285+0000) 2024-03-06T11:28:11.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:11.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:11.819511+0000) 2024-03-06T11:28:12.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:12.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:12.819769+0000) 2024-03-06T11:28:13.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:13.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:13.820005+0000) 2024-03-06T11:28:14.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:14.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:14.820193+0000) 2024-03-06T11:28:15.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:15.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:15.820400+0000) 2024-03-06T11:28:16.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:16.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:16.820650+0000) 2024-03-06T11:28:17.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:17.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:17.820913+0000) 2024-03-06T11:28:18.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:18.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:18.821132+0000) 2024-03-06T11:28:19.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:19.065 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:19.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:19.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:19.821369+0000) 2024-03-06T11:28:20.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:20.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:20.821599+0000) 2024-03-06T11:28:21.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:21.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:21.821823+0000) 2024-03-06T11:28:22.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:22.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:22.821987+0000) 2024-03-06T11:28:23.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:23.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:23.822231+0000) 2024-03-06T11:28:24.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:24.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:24.822439+0000) 2024-03-06T11:28:25.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:25.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:25.822723+0000) 2024-03-06T11:28:26.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:26.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:26.823009+0000) 2024-03-06T11:28:27.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:27.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:27.823232+0000) 2024-03-06T11:28:28.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:28.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:28.823447+0000) 2024-03-06T11:28:29.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:29.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:29.823707+0000) 2024-03-06T11:28:30.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:30.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:30.823952+0000) 2024-03-06T11:28:31.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:31.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:31.824191+0000) 2024-03-06T11:28:32.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:32.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:32.824395+0000) 2024-03-06T11:28:33.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:33.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:33.824635+0000) 2024-03-06T11:28:34.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:34.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:34.824922+0000) 2024-03-06T11:28:35.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:35.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:35.825159+0000) 2024-03-06T11:28:36.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:36.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:36.825406+0000) 2024-03-06T11:28:37.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:37.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:37.825647+0000) 2024-03-06T11:28:38.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:38.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:38.825905+0000) 2024-03-06T11:28:39.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:39.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:39.826128+0000) 2024-03-06T11:28:40.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:40.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:40.826354+0000) 2024-03-06T11:28:41.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:41.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:41.826581+0000) 2024-03-06T11:28:42.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:42.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:42.826808+0000) 2024-03-06T11:28:43.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:43.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:43.827035+0000) 2024-03-06T11:28:44.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:44.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:44.827258+0000) 2024-03-06T11:28:45.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:45.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:45.827479+0000) 2024-03-06T11:28:46.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:46.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:46.827748+0000) 2024-03-06T11:28:47.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:47.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:47.827986+0000) 2024-03-06T11:28:48.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:48.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:48.828173+0000) 2024-03-06T11:28:49.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:49.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:49.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:49.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:49.828404+0000) 2024-03-06T11:28:50.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:50.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:50.828626+0000) 2024-03-06T11:28:51.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:51.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:51.828831+0000) 2024-03-06T11:28:52.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:52.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:52.829043+0000) 2024-03-06T11:28:53.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:53.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:53.829259+0000) 2024-03-06T11:28:54.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:54.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:54.829473+0000) 2024-03-06T11:28:55.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:55.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:55.829749+0000) 2024-03-06T11:28:56.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:56.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:56.829985+0000) 2024-03-06T11:28:57.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:57.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:57.830186+0000) 2024-03-06T11:28:58.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:58.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:58.830417+0000) 2024-03-06T11:28:59.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:28:59.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:59.830644+0000) 2024-03-06T11:29:00.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:00.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:00.830904+0000) 2024-03-06T11:29:01.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:01.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:01.831120+0000) 2024-03-06T11:29:02.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:02.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:02.831341+0000) 2024-03-06T11:29:03.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:03.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:03.831541+0000) 2024-03-06T11:29:04.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:04.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:04.831769+0000) 2024-03-06T11:29:05.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:05.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:05.831933+0000) 2024-03-06T11:29:06.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:06.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:06.832155+0000) 2024-03-06T11:29:07.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:07.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:07.832390+0000) 2024-03-06T11:29:08.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:08.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:08.832636+0000) 2024-03-06T11:29:09.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:09.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:09.832902+0000) 2024-03-06T11:29:10.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:10.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:10.833140+0000) 2024-03-06T11:29:11.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:11.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:11.833435+0000) 2024-03-06T11:29:12.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:12.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:12.833712+0000) 2024-03-06T11:29:13.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:13.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:13.833978+0000) 2024-03-06T11:29:14.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:14.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:14.834246+0000) 2024-03-06T11:29:15.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:15.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:15.834478+0000) 2024-03-06T11:29:16.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:16.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:16.834808+0000) 2024-03-06T11:29:17.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:17.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:17.835114+0000) 2024-03-06T11:29:18.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:18.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:18.835354+0000) 2024-03-06T11:29:19.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:19.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:19.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:19.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:19.835645+0000) 2024-03-06T11:29:20.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:20.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:20.835887+0000) 2024-03-06T11:29:21.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:21.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:21.836100+0000) 2024-03-06T11:29:22.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:22.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:22.836346+0000) 2024-03-06T11:29:23.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:23.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:23.836589+0000) 2024-03-06T11:29:24.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:24.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:24.836803+0000) 2024-03-06T11:29:25.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:25.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:25.836983+0000) 2024-03-06T11:29:26.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:26.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:26.837170+0000) 2024-03-06T11:29:27.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:27.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:27.837398+0000) 2024-03-06T11:29:28.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:28.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:28.837629+0000) 2024-03-06T11:29:29.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:29.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:29.837828+0000) 2024-03-06T11:29:30.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:30.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:30.838014+0000) 2024-03-06T11:29:31.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:31.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:31.838206+0000) 2024-03-06T11:29:32.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:32.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:32.838431+0000) 2024-03-06T11:29:33.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:33.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:33.838701+0000) 2024-03-06T11:29:34.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:34.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:34.838936+0000) 2024-03-06T11:29:35.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:35.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:35.839180+0000) 2024-03-06T11:29:36.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:36.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:36.839405+0000) 2024-03-06T11:29:37.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:37.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:37.839649+0000) 2024-03-06T11:29:38.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:38.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:38.839924+0000) 2024-03-06T11:29:39.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:39.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:39.840114+0000) 2024-03-06T11:29:40.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:40.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:40.840315+0000) 2024-03-06T11:29:41.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:41.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:41.840544+0000) 2024-03-06T11:29:42.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:42.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:42.840773+0000) 2024-03-06T11:29:43.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:43.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:43.840981+0000) 2024-03-06T11:29:44.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:44.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:44.841165+0000) 2024-03-06T11:29:45.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:45.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:45.841399+0000) 2024-03-06T11:29:46.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:46.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:46.841614+0000) 2024-03-06T11:29:47.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:47.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:47.841824+0000) 2024-03-06T11:29:48.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:48.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:48.842051+0000) 2024-03-06T11:29:49.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:49.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:49.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:49.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:49.842289+0000) 2024-03-06T11:29:50.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:50.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:50.842502+0000) 2024-03-06T11:29:51.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:51.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:51.842755+0000) 2024-03-06T11:29:52.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:52.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:52.842991+0000) 2024-03-06T11:29:53.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:53.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:53.843223+0000) 2024-03-06T11:29:54.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:54.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:54.843446+0000) 2024-03-06T11:29:55.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:55.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:55.843704+0000) 2024-03-06T11:29:56.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:56.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:56.843959+0000) 2024-03-06T11:29:57.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:57.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:57.844197+0000) 2024-03-06T11:29:58.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:58.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:58.844398+0000) 2024-03-06T11:29:59.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:29:59.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:59.844622+0000) 2024-03-06T11:30:00.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:00.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:00.844831+0000) 2024-03-06T11:30:01.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:01.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:01.845073+0000) 2024-03-06T11:30:02.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:02.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:02.845306+0000) 2024-03-06T11:30:03.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:03.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:03.845519+0000) 2024-03-06T11:30:04.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:04.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:04.845757+0000) 2024-03-06T11:30:05.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:05.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:05.845997+0000) 2024-03-06T11:30:06.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:06.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:06.846246+0000) 2024-03-06T11:30:07.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:07.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:07.846534+0000) 2024-03-06T11:30:08.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:08.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:08.846792+0000) 2024-03-06T11:30:09.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:09.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:09.847027+0000) 2024-03-06T11:30:10.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:10.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:10.847239+0000) 2024-03-06T11:30:11.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:11.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:11.847487+0000) 2024-03-06T11:30:12.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:12.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:12.847758+0000) 2024-03-06T11:30:13.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:13.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:13.847981+0000) 2024-03-06T11:30:14.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:14.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:14.848242+0000) 2024-03-06T11:30:15.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:15.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:15.848485+0000) 2024-03-06T11:30:16.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:16.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:16.848765+0000) 2024-03-06T11:30:17.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:17.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:17.848974+0000) 2024-03-06T11:30:18.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:18.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:18.849232+0000) 2024-03-06T11:30:19.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:19.706 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:19.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:19.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:19.849481+0000) 2024-03-06T11:30:20.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:20.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:20.849701+0000) 2024-03-06T11:30:21.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:21.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:21.849897+0000) 2024-03-06T11:30:22.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:22.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:22.850114+0000) 2024-03-06T11:30:23.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:23.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:23.850348+0000) 2024-03-06T11:30:24.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:24.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:24.850576+0000) 2024-03-06T11:30:25.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:25.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:25.850798+0000) 2024-03-06T11:30:26.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:26.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:26.851027+0000) 2024-03-06T11:30:27.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:27.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:27.851225+0000) 2024-03-06T11:30:28.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:28.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:28.851425+0000) 2024-03-06T11:30:29.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:29.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:29.851703+0000) 2024-03-06T11:30:30.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:30.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:30.851923+0000) 2024-03-06T11:30:31.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:31.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:31.852106+0000) 2024-03-06T11:30:32.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:32.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:32.852332+0000) 2024-03-06T11:30:33.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:33.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:33.852564+0000) 2024-03-06T11:30:34.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:34.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:34.852783+0000) 2024-03-06T11:30:35.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:35.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:35.852995+0000) 2024-03-06T11:30:36.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:36.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:36.853194+0000) 2024-03-06T11:30:37.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:37.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:37.853414+0000) 2024-03-06T11:30:38.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:38.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:38.853641+0000) 2024-03-06T11:30:39.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:39.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:39.853855+0000) 2024-03-06T11:30:40.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:40.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:40.854084+0000) 2024-03-06T11:30:41.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:41.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:41.854258+0000) 2024-03-06T11:30:42.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:42.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:42.854474+0000) 2024-03-06T11:30:43.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:43.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:43.854757+0000) 2024-03-06T11:30:44.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:44.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:44.854991+0000) 2024-03-06T11:30:45.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:45.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:45.855202+0000) 2024-03-06T11:30:46.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:46.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:46.855398+0000) 2024-03-06T11:30:47.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:47.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:47.855620+0000) 2024-03-06T11:30:48.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:48.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:48.855845+0000) 2024-03-06T11:30:49.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:49.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:49.856079+0000) 2024-03-06T11:30:49.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:49.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:50.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:50.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:50.856296+0000) 2024-03-06T11:30:51.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:51.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:51.856492+0000) 2024-03-06T11:30:52.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:52.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:52.856723+0000) 2024-03-06T11:30:53.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:53.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:53.856971+0000) 2024-03-06T11:30:54.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:54.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:54.857172+0000) 2024-03-06T11:30:55.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:55.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:55.857383+0000) 2024-03-06T11:30:56.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:56.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:56.857560+0000) 2024-03-06T11:30:57.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:57.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:57.857798+0000) 2024-03-06T11:30:58.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:58.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:58.858005+0000) 2024-03-06T11:30:59.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:30:59.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:59.858237+0000) 2024-03-06T11:31:00.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:00.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:00.858456+0000) 2024-03-06T11:31:01.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:01.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:01.858769+0000) 2024-03-06T11:31:02.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:02.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:02.858982+0000) 2024-03-06T11:31:03.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:03.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:03.859201+0000) 2024-03-06T11:31:04.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:04.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:04.859445+0000) 2024-03-06T11:31:05.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:05.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:05.859727+0000) 2024-03-06T11:31:06.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:06.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:06.859932+0000) 2024-03-06T11:31:07.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:07.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:07.860171+0000) 2024-03-06T11:31:08.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:08.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:08.860390+0000) 2024-03-06T11:31:09.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:09.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:09.860635+0000) 2024-03-06T11:31:10.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:10.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:10.860924+0000) 2024-03-06T11:31:11.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:11.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:11.861126+0000) 2024-03-06T11:31:12.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:12.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:12.861366+0000) 2024-03-06T11:31:13.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:13.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:13.861621+0000) 2024-03-06T11:31:14.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:14.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:14.861832+0000) 2024-03-06T11:31:15.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:15.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:15.862074+0000) 2024-03-06T11:31:16.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:16.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:16.862336+0000) 2024-03-06T11:31:17.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:17.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:17.862606+0000) 2024-03-06T11:31:18.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:18.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:18.862884+0000) 2024-03-06T11:31:19.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:19.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:19.863108+0000) 2024-03-06T11:31:20.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:20.126 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:20.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:20.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:20.863341+0000) 2024-03-06T11:31:21.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:21.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:21.863575+0000) 2024-03-06T11:31:22.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:22.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:22.863819+0000) 2024-03-06T11:31:23.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:23.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:23.864075+0000) 2024-03-06T11:31:24.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:24.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:24.864274+0000) 2024-03-06T11:31:25.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:25.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:25.864498+0000) 2024-03-06T11:31:26.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:26.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:26.864731+0000) 2024-03-06T11:31:27.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:27.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:27.864966+0000) 2024-03-06T11:31:28.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:28.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:28.865211+0000) 2024-03-06T11:31:29.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:29.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:29.865460+0000) 2024-03-06T11:31:30.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:30.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:30.865707+0000) 2024-03-06T11:31:31.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:31.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:31.865951+0000) 2024-03-06T11:31:32.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:32.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:32.866199+0000) 2024-03-06T11:31:33.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:33.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:33.866447+0000) 2024-03-06T11:31:34.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:34.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:34.866652+0000) 2024-03-06T11:31:35.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:35.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:35.866893+0000) 2024-03-06T11:31:36.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:36.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:36.867153+0000) 2024-03-06T11:31:37.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:37.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:37.867383+0000) 2024-03-06T11:31:38.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:38.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:38.867590+0000) 2024-03-06T11:31:39.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:39.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:39.867820+0000) 2024-03-06T11:31:40.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:40.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:40.868002+0000) 2024-03-06T11:31:41.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:41.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:41.868199+0000) 2024-03-06T11:31:42.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:42.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:42.868395+0000) 2024-03-06T11:31:43.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:43.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:43.868651+0000) 2024-03-06T11:31:44.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:44.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:44.868891+0000) 2024-03-06T11:31:45.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:45.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:45.869126+0000) 2024-03-06T11:31:46.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:46.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:46.869365+0000) 2024-03-06T11:31:47.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:47.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:47.869593+0000) 2024-03-06T11:31:48.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:48.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:48.869862+0000) 2024-03-06T11:31:49.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:49.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:49.870101+0000) 2024-03-06T11:31:50.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:50.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:50.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:50.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:50.870327+0000) 2024-03-06T11:31:51.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:51.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:51.870503+0000) 2024-03-06T11:31:52.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:52.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:52.870778+0000) 2024-03-06T11:31:53.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:53.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:53.871029+0000) 2024-03-06T11:31:54.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:54.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:54.871275+0000) 2024-03-06T11:31:55.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:55.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:55.871495+0000) 2024-03-06T11:31:56.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:56.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:56.871784+0000) 2024-03-06T11:31:57.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:57.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:57.871947+0000) 2024-03-06T11:31:58.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:58.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:58.872140+0000) 2024-03-06T11:31:59.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:31:59.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:59.872366+0000) 2024-03-06T11:32:00.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:00.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:00.872587+0000) 2024-03-06T11:32:01.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:01.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:01.872837+0000) 2024-03-06T11:32:02.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:02.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:02.873059+0000) 2024-03-06T11:32:03.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:03.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:03.873299+0000) 2024-03-06T11:32:04.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:04.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:04.873496+0000) 2024-03-06T11:32:05.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:05.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:05.873739+0000) 2024-03-06T11:32:06.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:06.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:06.873981+0000) 2024-03-06T11:32:07.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:07.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:07.874204+0000) 2024-03-06T11:32:08.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:08.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:08.874409+0000) 2024-03-06T11:32:09.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:09.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:09.874653+0000) 2024-03-06T11:32:10.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:10.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:10.874842+0000) 2024-03-06T11:32:11.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:11.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:11.875052+0000) 2024-03-06T11:32:12.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:12.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:12.875237+0000) 2024-03-06T11:32:13.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:13.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:13.875477+0000) 2024-03-06T11:32:14.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:14.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:14.875743+0000) 2024-03-06T11:32:15.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:15.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:15.875972+0000) 2024-03-06T11:32:16.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:16.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:16.876198+0000) 2024-03-06T11:32:17.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:17.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:17.876396+0000) 2024-03-06T11:32:18.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:18.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:18.876589+0000) 2024-03-06T11:32:19.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:19.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:19.876833+0000) 2024-03-06T11:32:20.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:20.495 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:20.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:20.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:20.877031+0000) 2024-03-06T11:32:21.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:21.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:21.877248+0000) 2024-03-06T11:32:22.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:22.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:22.877483+0000) 2024-03-06T11:32:23.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:23.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:23.877711+0000) 2024-03-06T11:32:24.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:24.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:24.877916+0000) 2024-03-06T11:32:25.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:25.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:25.878146+0000) 2024-03-06T11:32:26.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:26.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:26.878368+0000) 2024-03-06T11:32:27.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:27.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:27.878582+0000) 2024-03-06T11:32:28.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:28.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:28.878795+0000) 2024-03-06T11:32:29.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:29.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:29.879020+0000) 2024-03-06T11:32:30.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:30.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:30.879250+0000) 2024-03-06T11:32:31.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:31.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:31.879432+0000) 2024-03-06T11:32:32.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:32.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:32.879612+0000) 2024-03-06T11:32:33.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:33.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:33.879844+0000) 2024-03-06T11:32:34.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:34.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:34.880057+0000) 2024-03-06T11:32:35.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:35.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:35.880271+0000) 2024-03-06T11:32:36.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:36.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:36.880495+0000) 2024-03-06T11:32:37.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:37.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:37.880773+0000) 2024-03-06T11:32:38.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:38.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:38.881002+0000) 2024-03-06T11:32:39.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:39.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:39.881229+0000) 2024-03-06T11:32:40.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:40.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:40.881452+0000) 2024-03-06T11:32:41.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:41.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:41.881645+0000) 2024-03-06T11:32:42.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:42.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:42.881970+0000) 2024-03-06T11:32:43.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:43.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:43.882185+0000) 2024-03-06T11:32:44.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:44.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:44.882406+0000) 2024-03-06T11:32:45.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:45.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:45.882640+0000) 2024-03-06T11:32:46.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:46.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:46.882908+0000) 2024-03-06T11:32:47.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:47.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:47.883076+0000) 2024-03-06T11:32:48.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:48.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:48.883337+0000) 2024-03-06T11:32:49.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:49.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:49.883619+0000) 2024-03-06T11:32:50.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:50.706 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:50.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:50.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:50.883899+0000) 2024-03-06T11:32:51.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:51.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:51.884182+0000) 2024-03-06T11:32:52.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:52.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:52.884446+0000) 2024-03-06T11:32:53.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:53.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:53.884713+0000) 2024-03-06T11:32:54.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:54.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:54.884979+0000) 2024-03-06T11:32:55.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:55.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:55.885254+0000) 2024-03-06T11:32:56.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:56.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:56.885483+0000) 2024-03-06T11:32:57.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:57.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:57.885758+0000) 2024-03-06T11:32:58.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:58.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:58.885975+0000) 2024-03-06T11:32:59.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:32:59.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:59.886194+0000) 2024-03-06T11:33:00.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:00.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:00.886411+0000) 2024-03-06T11:33:01.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:01.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:01.886661+0000) 2024-03-06T11:33:02.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:02.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:02.886892+0000) 2024-03-06T11:33:03.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:03.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:03.887120+0000) 2024-03-06T11:33:04.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:04.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:04.887306+0000) 2024-03-06T11:33:05.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:05.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:05.887545+0000) 2024-03-06T11:33:06.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:06.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:06.887790+0000) 2024-03-06T11:33:07.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:07.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:07.888019+0000) 2024-03-06T11:33:08.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:08.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:08.888201+0000) 2024-03-06T11:33:09.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:09.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:09.888401+0000) 2024-03-06T11:33:10.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:10.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:10.888590+0000) 2024-03-06T11:33:11.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:11.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:11.888817+0000) 2024-03-06T11:33:12.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:12.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:12.889051+0000) 2024-03-06T11:33:13.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:13.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:13.889281+0000) 2024-03-06T11:33:14.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:14.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:14.889477+0000) 2024-03-06T11:33:15.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:15.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:15.889783+0000) 2024-03-06T11:33:16.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:16.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:16.890019+0000) 2024-03-06T11:33:17.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:17.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:17.890263+0000) 2024-03-06T11:33:18.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:18.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:18.890463+0000) 2024-03-06T11:33:19.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:19.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:19.890706+0000) 2024-03-06T11:33:20.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:20.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:20.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:20.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:20.890852+0000) 2024-03-06T11:33:21.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:21.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:21.891029+0000) 2024-03-06T11:33:22.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:22.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:22.891256+0000) 2024-03-06T11:33:23.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:23.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:23.891486+0000) 2024-03-06T11:33:24.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:24.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:24.891762+0000) 2024-03-06T11:33:25.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:25.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:25.891975+0000) 2024-03-06T11:33:26.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:26.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:26.892172+0000) 2024-03-06T11:33:27.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:27.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:27.892367+0000) 2024-03-06T11:33:28.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:28.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:28.892594+0000) 2024-03-06T11:33:29.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:29.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:29.892845+0000) 2024-03-06T11:33:30.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:30.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:30.893074+0000) 2024-03-06T11:33:31.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:31.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:31.893259+0000) 2024-03-06T11:33:32.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:32.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:32.893496+0000) 2024-03-06T11:33:33.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:33.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:33.893770+0000) 2024-03-06T11:33:34.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:34.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:34.894034+0000) 2024-03-06T11:33:35.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:35.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:35.894318+0000) 2024-03-06T11:33:36.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:36.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:36.894547+0000) 2024-03-06T11:33:37.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:37.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:37.894782+0000) 2024-03-06T11:33:38.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:38.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:38.894967+0000) 2024-03-06T11:33:39.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:39.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:39.895196+0000) 2024-03-06T11:33:40.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:40.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:40.895413+0000) 2024-03-06T11:33:41.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:41.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:41.895644+0000) 2024-03-06T11:33:42.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:42.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:42.895908+0000) 2024-03-06T11:33:43.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:43.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:43.896082+0000) 2024-03-06T11:33:44.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:44.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:44.896261+0000) 2024-03-06T11:33:45.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:45.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:45.896446+0000) 2024-03-06T11:33:46.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:46.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:46.896705+0000) 2024-03-06T11:33:47.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:47.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:47.896926+0000) 2024-03-06T11:33:48.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:48.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:48.897148+0000) 2024-03-06T11:33:49.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:49.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:49.897370+0000) 2024-03-06T11:33:50.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:50.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:50.897592+0000) 2024-03-06T11:33:51.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:51.038 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:51.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:51.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:51.897820+0000) 2024-03-06T11:33:52.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:52.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:52.898007+0000) 2024-03-06T11:33:53.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:53.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:53.898238+0000) 2024-03-06T11:33:54.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:54.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:54.898440+0000) 2024-03-06T11:33:55.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:55.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:55.898700+0000) 2024-03-06T11:33:56.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:56.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:56.898895+0000) 2024-03-06T11:33:57.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:57.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:57.899070+0000) 2024-03-06T11:33:58.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:58.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:58.899260+0000) 2024-03-06T11:33:59.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:33:59.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:59.899490+0000) 2024-03-06T11:34:00.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:00.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:00.899747+0000) 2024-03-06T11:34:01.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:01.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:01.899936+0000) 2024-03-06T11:34:02.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:02.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:02.900157+0000) 2024-03-06T11:34:03.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:03.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:03.900322+0000) 2024-03-06T11:34:04.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:04.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:04.900536+0000) 2024-03-06T11:34:05.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:05.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:05.900786+0000) 2024-03-06T11:34:06.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:06.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:06.901006+0000) 2024-03-06T11:34:07.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:07.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:07.901228+0000) 2024-03-06T11:34:08.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:08.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:08.901445+0000) 2024-03-06T11:34:09.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:09.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:09.901703+0000) 2024-03-06T11:34:10.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:10.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:10.901932+0000) 2024-03-06T11:34:11.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:11.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:11.902157+0000) 2024-03-06T11:34:12.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:12.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:12.902347+0000) 2024-03-06T11:34:13.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:13.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:13.902545+0000) 2024-03-06T11:34:14.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:14.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:14.902769+0000) 2024-03-06T11:34:15.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:15.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:15.903001+0000) 2024-03-06T11:34:16.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:16.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:16.903257+0000) 2024-03-06T11:34:17.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:17.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:17.903500+0000) 2024-03-06T11:34:18.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:18.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:18.903788+0000) 2024-03-06T11:34:19.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:19.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:19.904044+0000) 2024-03-06T11:34:20.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:20.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:20.904226+0000) 2024-03-06T11:34:21.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:21.169 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:21.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:21.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:21.904417+0000) 2024-03-06T11:34:22.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:22.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:22.904608+0000) 2024-03-06T11:34:23.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:23.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:23.904857+0000) 2024-03-06T11:34:24.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:24.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:24.905091+0000) 2024-03-06T11:34:25.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:25.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:25.905307+0000) 2024-03-06T11:34:26.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:26.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:26.905542+0000) 2024-03-06T11:34:27.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:27.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:27.905784+0000) 2024-03-06T11:34:28.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:28.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:28.906028+0000) 2024-03-06T11:34:29.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:29.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:29.906319+0000) 2024-03-06T11:34:30.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:30.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:30.906553+0000) 2024-03-06T11:34:31.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:31.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:31.906802+0000) 2024-03-06T11:34:32.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:32.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:32.907043+0000) 2024-03-06T11:34:33.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:33.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:33.907268+0000) 2024-03-06T11:34:34.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:34.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:34.907502+0000) 2024-03-06T11:34:35.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:35.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:35.907775+0000) 2024-03-06T11:34:36.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:36.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:36.908014+0000) 2024-03-06T11:34:37.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:37.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:37.908240+0000) 2024-03-06T11:34:38.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:38.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:38.908476+0000) 2024-03-06T11:34:39.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:39.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:39.908727+0000) 2024-03-06T11:34:40.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:40.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:40.908951+0000) 2024-03-06T11:34:41.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:41.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:41.909143+0000) 2024-03-06T11:34:42.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:42.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:42.909324+0000) 2024-03-06T11:34:43.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:43.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:43.909543+0000) 2024-03-06T11:34:44.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:44.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:44.909781+0000) 2024-03-06T11:34:45.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:45.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:45.910014+0000) 2024-03-06T11:34:46.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:46.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:46.910209+0000) 2024-03-06T11:34:47.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:47.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:47.910428+0000) 2024-03-06T11:34:48.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:48.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:48.910648+0000) 2024-03-06T11:34:49.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:49.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:49.910917+0000) 2024-03-06T11:34:50.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:50.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:50.911089+0000) 2024-03-06T11:34:51.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:51.391 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:51.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:51.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:51.911319+0000) 2024-03-06T11:34:52.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:52.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:52.911497+0000) 2024-03-06T11:34:53.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:53.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:53.911744+0000) 2024-03-06T11:34:54.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:54.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:54.911985+0000) 2024-03-06T11:34:55.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:55.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:55.912211+0000) 2024-03-06T11:34:56.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:56.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:56.912412+0000) 2024-03-06T11:34:57.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:57.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:57.912700+0000) 2024-03-06T11:34:58.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:58.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:58.912945+0000) 2024-03-06T11:34:59.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:34:59.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:59.913201+0000) 2024-03-06T11:35:00.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:00.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:00.913418+0000) 2024-03-06T11:35:01.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:01.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:01.913643+0000) 2024-03-06T11:35:02.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:02.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:02.913889+0000) 2024-03-06T11:35:03.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:03.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:03.914117+0000) 2024-03-06T11:35:04.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:04.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:04.914361+0000) 2024-03-06T11:35:05.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:05.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:05.914550+0000) 2024-03-06T11:35:06.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:06.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:06.914805+0000) 2024-03-06T11:35:07.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:07.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:07.914989+0000) 2024-03-06T11:35:08.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:08.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:08.915182+0000) 2024-03-06T11:35:09.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:09.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:09.915412+0000) 2024-03-06T11:35:10.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:10.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:10.915604+0000) 2024-03-06T11:35:11.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:11.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:11.915840+0000) 2024-03-06T11:35:12.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:12.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:12.916062+0000) 2024-03-06T11:35:13.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:13.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:13.916293+0000) 2024-03-06T11:35:14.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:14.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:14.916521+0000) 2024-03-06T11:35:15.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:15.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:15.916788+0000) 2024-03-06T11:35:16.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:16.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:16.917020+0000) 2024-03-06T11:35:17.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:17.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:17.917298+0000) 2024-03-06T11:35:18.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:18.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:18.917490+0000) 2024-03-06T11:35:19.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:19.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:19.917769+0000) 2024-03-06T11:35:20.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:20.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:20.917957+0000) 2024-03-06T11:35:21.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:21.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:21.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:21.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:21.918139+0000) 2024-03-06T11:35:22.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:22.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:22.918373+0000) 2024-03-06T11:35:23.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:23.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:23.918710+0000) 2024-03-06T11:35:24.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:24.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:24.918967+0000) 2024-03-06T11:35:25.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:25.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:25.919191+0000) 2024-03-06T11:35:26.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:26.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:26.919396+0000) 2024-03-06T11:35:27.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:27.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:27.919605+0000) 2024-03-06T11:35:28.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:28.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:28.919823+0000) 2024-03-06T11:35:29.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:29.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:29.920015+0000) 2024-03-06T11:35:30.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:30.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:30.920243+0000) 2024-03-06T11:35:31.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:31.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:31.920469+0000) 2024-03-06T11:35:32.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:32.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:32.920645+0000) 2024-03-06T11:35:33.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:33.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:33.920899+0000) 2024-03-06T11:35:34.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:34.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:34.921048+0000) 2024-03-06T11:35:35.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:35.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:35.921270+0000) 2024-03-06T11:35:36.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:36.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:36.921514+0000) 2024-03-06T11:35:37.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:37.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:37.921782+0000) 2024-03-06T11:35:38.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:38.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:38.921976+0000) 2024-03-06T11:35:39.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:39.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:39.922197+0000) 2024-03-06T11:35:40.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:40.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:40.922405+0000) 2024-03-06T11:35:41.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:41.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:41.922647+0000) 2024-03-06T11:35:42.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:42.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:42.922894+0000) 2024-03-06T11:35:43.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:43.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:43.923119+0000) 2024-03-06T11:35:44.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:44.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:44.923334+0000) 2024-03-06T11:35:45.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:45.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:45.923577+0000) 2024-03-06T11:35:46.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:46.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:46.923834+0000) 2024-03-06T11:35:47.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:47.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:47.924015+0000) 2024-03-06T11:35:48.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:48.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:48.924193+0000) 2024-03-06T11:35:49.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:49.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:49.924382+0000) 2024-03-06T11:35:50.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:50.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:50.924593+0000) 2024-03-06T11:35:51.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:51.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:51.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:51.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:51.924784+0000) 2024-03-06T11:35:52.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:52.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:52.924968+0000) 2024-03-06T11:35:53.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:53.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:53.925159+0000) 2024-03-06T11:35:54.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:54.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:54.925423+0000) 2024-03-06T11:35:55.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:55.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:55.925694+0000) 2024-03-06T11:35:56.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:56.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:56.925933+0000) 2024-03-06T11:35:57.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:57.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:57.926155+0000) 2024-03-06T11:35:58.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:58.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:58.926405+0000) 2024-03-06T11:35:59.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:35:59.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:59.926647+0000) 2024-03-06T11:36:00.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:00.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:00.926873+0000) 2024-03-06T11:36:01.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:01.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:01.927075+0000) 2024-03-06T11:36:02.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:02.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:02.927256+0000) 2024-03-06T11:36:03.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:03.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:03.927490+0000) 2024-03-06T11:36:04.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:04.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:04.927699+0000) 2024-03-06T11:36:05.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:05.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:05.927889+0000) 2024-03-06T11:36:06.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:06.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:06.928115+0000) 2024-03-06T11:36:07.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:07.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:07.928345+0000) 2024-03-06T11:36:08.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:08.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:08.928567+0000) 2024-03-06T11:36:09.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:09.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:09.928788+0000) 2024-03-06T11:36:10.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:10.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:10.929069+0000) 2024-03-06T11:36:11.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:11.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:11.929275+0000) 2024-03-06T11:36:12.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:12.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:12.929454+0000) 2024-03-06T11:36:13.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:13.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:13.929720+0000) 2024-03-06T11:36:14.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:14.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:14.929948+0000) 2024-03-06T11:36:15.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:15.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:15.930182+0000) 2024-03-06T11:36:16.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:16.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:16.930456+0000) 2024-03-06T11:36:17.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:17.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:17.930772+0000) 2024-03-06T11:36:18.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:18.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:18.930938+0000) 2024-03-06T11:36:19.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:19.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:19.931156+0000) 2024-03-06T11:36:20.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:20.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:20.931387+0000) 2024-03-06T11:36:21.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:21.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:21.931661+0000) 2024-03-06T11:36:22.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:22.022 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:22.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:22.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:22.931985+0000) 2024-03-06T11:36:23.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:23.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:23.932258+0000) 2024-03-06T11:36:24.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:24.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:24.932531+0000) 2024-03-06T11:36:25.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:25.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:25.932815+0000) 2024-03-06T11:36:26.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:26.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:26.933061+0000) 2024-03-06T11:36:27.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:27.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:27.933260+0000) 2024-03-06T11:36:28.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:28.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:28.933483+0000) 2024-03-06T11:36:29.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:29.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:29.933717+0000) 2024-03-06T11:36:30.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:30.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:30.933954+0000) 2024-03-06T11:36:31.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:31.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:31.934184+0000) 2024-03-06T11:36:32.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:32.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:32.934407+0000) 2024-03-06T11:36:33.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:33.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:33.934639+0000) 2024-03-06T11:36:34.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:34.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:34.934898+0000) 2024-03-06T11:36:35.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:35.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:35.935125+0000) 2024-03-06T11:36:36.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:36.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:36.935353+0000) 2024-03-06T11:36:37.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:37.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:37.935571+0000) 2024-03-06T11:36:38.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:38.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:38.935800+0000) 2024-03-06T11:36:39.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:39.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:39.936036+0000) 2024-03-06T11:36:40.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:40.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:40.936281+0000) 2024-03-06T11:36:41.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:41.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:41.936507+0000) 2024-03-06T11:36:42.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:42.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:42.936784+0000) 2024-03-06T11:36:43.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:43.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:43.936955+0000) 2024-03-06T11:36:44.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:44.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:44.937180+0000) 2024-03-06T11:36:45.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:45.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:45.937404+0000) 2024-03-06T11:36:46.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:46.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:46.937635+0000) 2024-03-06T11:36:47.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:47.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:47.937836+0000) 2024-03-06T11:36:48.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:48.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:48.938036+0000) 2024-03-06T11:36:49.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:49.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:49.938282+0000) 2024-03-06T11:36:50.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:50.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:50.938505+0000) 2024-03-06T11:36:51.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:51.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:51.938808+0000) 2024-03-06T11:36:52.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:52.173 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:52.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:52.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:52.939042+0000) 2024-03-06T11:36:53.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:53.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:53.939227+0000) 2024-03-06T11:36:54.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:54.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:54.939463+0000) 2024-03-06T11:36:55.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:55.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:55.939721+0000) 2024-03-06T11:36:56.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:56.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:56.939924+0000) 2024-03-06T11:36:57.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:57.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:57.940140+0000) 2024-03-06T11:36:58.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:58.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:58.940411+0000) 2024-03-06T11:36:59.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:36:59.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:59.940617+0000) 2024-03-06T11:37:00.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:00.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:00.940824+0000) 2024-03-06T11:37:01.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:01.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:01.941013+0000) 2024-03-06T11:37:02.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:02.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:02.941233+0000) 2024-03-06T11:37:03.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:03.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:03.941471+0000) 2024-03-06T11:37:04.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:04.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:04.941753+0000) 2024-03-06T11:37:05.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:05.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:05.941942+0000) 2024-03-06T11:37:06.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:06.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:06.942178+0000) 2024-03-06T11:37:07.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:07.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:07.942412+0000) 2024-03-06T11:37:08.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:08.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:08.942614+0000) 2024-03-06T11:37:09.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:09.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:09.942836+0000) 2024-03-06T11:37:10.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:10.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:10.943087+0000) 2024-03-06T11:37:11.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:11.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:11.943311+0000) 2024-03-06T11:37:12.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:12.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:12.943533+0000) 2024-03-06T11:37:13.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:13.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:13.943783+0000) 2024-03-06T11:37:14.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:14.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:14.944027+0000) 2024-03-06T11:37:15.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:15.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:15.944250+0000) 2024-03-06T11:37:16.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:16.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:16.944481+0000) 2024-03-06T11:37:17.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:17.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:17.944763+0000) 2024-03-06T11:37:18.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:18.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:18.944953+0000) 2024-03-06T11:37:19.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:19.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:19.945190+0000) 2024-03-06T11:37:20.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:20.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:20.945410+0000) 2024-03-06T11:37:21.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:21.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:21.945635+0000) 2024-03-06T11:37:22.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:22.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:22.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:22.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:22.945836+0000) 2024-03-06T11:37:23.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:23.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:23.946037+0000) 2024-03-06T11:37:24.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:24.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:24.946257+0000) 2024-03-06T11:37:25.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:25.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:25.946455+0000) 2024-03-06T11:37:26.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:26.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:26.946748+0000) 2024-03-06T11:37:27.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:27.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:27.946931+0000) 2024-03-06T11:37:28.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:28.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:28.947110+0000) 2024-03-06T11:37:29.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:29.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:29.947333+0000) 2024-03-06T11:37:30.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:30.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:30.947551+0000) 2024-03-06T11:37:31.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:31.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:31.947772+0000) 2024-03-06T11:37:32.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:32.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:32.947969+0000) 2024-03-06T11:37:33.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:33.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:33.948202+0000) 2024-03-06T11:37:34.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:34.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:34.948449+0000) 2024-03-06T11:37:35.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:35.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:35.948711+0000) 2024-03-06T11:37:36.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:36.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:36.948948+0000) 2024-03-06T11:37:37.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:37.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:37.949140+0000) 2024-03-06T11:37:38.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:38.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:38.949373+0000) 2024-03-06T11:37:39.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:39.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:39.949609+0000) 2024-03-06T11:37:40.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:40.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:40.949811+0000) 2024-03-06T11:37:41.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:41.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:41.950053+0000) 2024-03-06T11:37:42.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:42.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:42.950281+0000) 2024-03-06T11:37:43.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:43.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:43.950502+0000) 2024-03-06T11:37:44.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:44.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:44.950767+0000) 2024-03-06T11:37:45.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:45.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:45.950964+0000) 2024-03-06T11:37:46.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:46.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:46.951192+0000) 2024-03-06T11:37:47.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:47.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:47.951418+0000) 2024-03-06T11:37:48.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:48.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:48.951649+0000) 2024-03-06T11:37:49.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:49.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:49.951931+0000) 2024-03-06T11:37:50.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:50.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:50.952219+0000) 2024-03-06T11:37:51.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:51.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:51.952422+0000) 2024-03-06T11:37:52.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:52.615 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:52.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:52.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:52.952632+0000) 2024-03-06T11:37:53.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:53.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:53.952969+0000) 2024-03-06T11:37:54.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:54.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:54.953212+0000) 2024-03-06T11:37:55.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:55.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:55.953497+0000) 2024-03-06T11:37:56.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:56.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:56.953806+0000) 2024-03-06T11:37:57.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:57.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:57.954133+0000) 2024-03-06T11:37:58.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:58.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:58.954377+0000) 2024-03-06T11:37:59.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:37:59.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:59.954629+0000) 2024-03-06T11:38:00.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:00.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:00.954826+0000) 2024-03-06T11:38:01.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:01.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:01.955024+0000) 2024-03-06T11:38:02.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:02.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:02.955209+0000) 2024-03-06T11:38:03.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:03.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:03.955416+0000) 2024-03-06T11:38:04.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:04.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:04.955652+0000) 2024-03-06T11:38:05.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:05.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:05.955899+0000) 2024-03-06T11:38:06.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:06.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:06.956096+0000) 2024-03-06T11:38:07.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:07.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:07.956319+0000) 2024-03-06T11:38:08.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:08.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:08.956495+0000) 2024-03-06T11:38:09.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:09.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:09.956790+0000) 2024-03-06T11:38:10.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:10.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:10.956973+0000) 2024-03-06T11:38:11.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:11.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:11.957249+0000) 2024-03-06T11:38:12.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:12.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:12.957441+0000) 2024-03-06T11:38:13.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:13.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:13.957660+0000) 2024-03-06T11:38:14.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:14.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:14.957983+0000) 2024-03-06T11:38:15.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:15.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:15.958254+0000) 2024-03-06T11:38:16.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:16.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:16.958533+0000) 2024-03-06T11:38:17.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:17.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:17.958798+0000) 2024-03-06T11:38:18.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:18.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:18.959083+0000) 2024-03-06T11:38:19.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:19.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:19.959290+0000) 2024-03-06T11:38:20.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:20.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:20.959513+0000) 2024-03-06T11:38:21.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:21.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:21.959786+0000) 2024-03-06T11:38:22.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:22.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:22.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:22.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:22.960013+0000) 2024-03-06T11:38:23.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:23.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:23.960260+0000) 2024-03-06T11:38:24.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:24.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:24.960493+0000) 2024-03-06T11:38:25.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:25.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:25.960770+0000) 2024-03-06T11:38:26.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:26.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:26.960987+0000) 2024-03-06T11:38:27.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:27.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:27.961225+0000) 2024-03-06T11:38:28.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:28.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:28.961462+0000) 2024-03-06T11:38:29.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:29.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:29.961764+0000) 2024-03-06T11:38:30.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:30.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:30.962002+0000) 2024-03-06T11:38:31.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:31.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:31.962241+0000) 2024-03-06T11:38:32.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:32.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:32.962504+0000) 2024-03-06T11:38:33.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:33.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:33.962795+0000) 2024-03-06T11:38:34.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:34.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:34.963000+0000) 2024-03-06T11:38:35.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:35.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:35.963213+0000) 2024-03-06T11:38:36.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:36.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:36.963452+0000) 2024-03-06T11:38:37.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:37.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:37.963714+0000) 2024-03-06T11:38:38.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:38.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:38.963943+0000) 2024-03-06T11:38:39.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:39.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:39.964181+0000) 2024-03-06T11:38:40.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:40.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:40.964359+0000) 2024-03-06T11:38:41.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:41.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:41.964625+0000) 2024-03-06T11:38:42.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:42.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:42.964941+0000) 2024-03-06T11:38:43.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:43.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:43.965205+0000) 2024-03-06T11:38:44.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:44.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:44.965477+0000) 2024-03-06T11:38:45.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:45.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:45.965849+0000) 2024-03-06T11:38:46.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:46.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:46.966082+0000) 2024-03-06T11:38:47.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:47.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:47.966314+0000) 2024-03-06T11:38:48.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:48.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:48.966538+0000) 2024-03-06T11:38:49.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:49.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:49.966773+0000) 2024-03-06T11:38:50.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:50.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:50.966959+0000) 2024-03-06T11:38:51.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:51.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:51.967184+0000) 2024-03-06T11:38:52.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:52.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:52.967409+0000) 2024-03-06T11:38:53.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:53.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:53.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:53.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:53.967641+0000) 2024-03-06T11:38:54.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:54.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:54.967908+0000) 2024-03-06T11:38:55.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:55.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:55.968110+0000) 2024-03-06T11:38:56.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:56.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:56.968372+0000) 2024-03-06T11:38:57.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:57.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:57.968567+0000) 2024-03-06T11:38:58.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:58.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:58.968801+0000) 2024-03-06T11:38:59.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:38:59.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:59.969064+0000) 2024-03-06T11:39:00.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:00.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:00.969336+0000) 2024-03-06T11:39:01.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:01.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:01.969619+0000) 2024-03-06T11:39:02.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:02.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:02.969882+0000) 2024-03-06T11:39:03.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:03.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:03.970152+0000) 2024-03-06T11:39:04.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:04.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:04.970390+0000) 2024-03-06T11:39:05.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:05.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:05.970583+0000) 2024-03-06T11:39:06.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:06.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:06.970820+0000) 2024-03-06T11:39:07.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:07.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:07.971027+0000) 2024-03-06T11:39:08.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:08.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:08.971270+0000) 2024-03-06T11:39:09.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:09.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:09.971519+0000) 2024-03-06T11:39:10.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:10.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:10.971758+0000) 2024-03-06T11:39:11.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:11.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:11.971990+0000) 2024-03-06T11:39:12.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:12.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:12.972269+0000) 2024-03-06T11:39:13.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:13.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:13.972544+0000) 2024-03-06T11:39:14.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:14.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:14.972821+0000) 2024-03-06T11:39:15.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:15.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:15.973107+0000) 2024-03-06T11:39:16.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:16.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:16.973337+0000) 2024-03-06T11:39:17.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:17.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:17.973533+0000) 2024-03-06T11:39:18.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:18.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:18.973767+0000) 2024-03-06T11:39:19.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:19.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:19.973999+0000) 2024-03-06T11:39:20.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:20.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:20.974231+0000) 2024-03-06T11:39:21.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:21.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:21.974422+0000) 2024-03-06T11:39:22.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:22.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:22.974617+0000) 2024-03-06T11:39:23.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:23.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:23.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:23.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:23.974860+0000) 2024-03-06T11:39:24.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:24.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:24.975075+0000) 2024-03-06T11:39:25.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:25.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:25.975311+0000) 2024-03-06T11:39:26.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:26.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:26.975540+0000) 2024-03-06T11:39:27.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:27.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:27.975800+0000) 2024-03-06T11:39:28.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:28.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:28.976026+0000) 2024-03-06T11:39:29.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:29.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:29.976261+0000) 2024-03-06T11:39:30.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:30.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:30.976527+0000) 2024-03-06T11:39:31.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:31.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:31.976785+0000) 2024-03-06T11:39:32.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:32.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:32.977005+0000) 2024-03-06T11:39:33.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:33.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:33.977330+0000) 2024-03-06T11:39:34.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:34.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:34.977556+0000) 2024-03-06T11:39:35.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:35.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:35.977806+0000) 2024-03-06T11:39:36.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:36.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:36.978024+0000) 2024-03-06T11:39:37.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:37.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:37.978271+0000) 2024-03-06T11:39:38.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:38.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:38.978539+0000) 2024-03-06T11:39:39.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:39.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:39.978811+0000) 2024-03-06T11:39:40.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:40.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:40.979048+0000) 2024-03-06T11:39:41.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:41.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:41.979280+0000) 2024-03-06T11:39:42.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:42.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:42.979512+0000) 2024-03-06T11:39:43.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:43.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:43.979776+0000) 2024-03-06T11:39:44.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:44.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:44.979983+0000) 2024-03-06T11:39:45.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:45.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:45.980191+0000) 2024-03-06T11:39:46.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:46.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:46.980462+0000) 2024-03-06T11:39:47.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:47.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:47.980716+0000) 2024-03-06T11:39:48.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:48.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:48.980908+0000) 2024-03-06T11:39:49.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:49.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:49.981106+0000) 2024-03-06T11:39:50.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:50.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:50.981330+0000) 2024-03-06T11:39:51.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:51.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:51.981555+0000) 2024-03-06T11:39:52.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:52.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:52.981783+0000) 2024-03-06T11:39:53.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:53.420 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:53.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:53.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:53.981974+0000) 2024-03-06T11:39:54.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:54.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:54.982197+0000) 2024-03-06T11:39:55.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:55.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:55.982408+0000) 2024-03-06T11:39:56.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:56.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:56.982628+0000) 2024-03-06T11:39:57.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:57.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:57.982824+0000) 2024-03-06T11:39:58.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:58.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:58.983005+0000) 2024-03-06T11:39:59.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:39:59.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:59.983213+0000) 2024-03-06T11:40:00.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:00.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:00.983397+0000) 2024-03-06T11:40:01.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:01.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:01.983582+0000) 2024-03-06T11:40:02.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:02.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:02.983797+0000) 2024-03-06T11:40:03.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:03.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:03.984021+0000) 2024-03-06T11:40:04.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:04.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:04.984222+0000) 2024-03-06T11:40:05.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:05.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:05.984450+0000) 2024-03-06T11:40:06.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:06.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:06.984632+0000) 2024-03-06T11:40:07.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:07.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:07.984817+0000) 2024-03-06T11:40:08.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:08.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:08.985043+0000) 2024-03-06T11:40:09.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:09.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:09.985257+0000) 2024-03-06T11:40:10.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:10.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:10.985428+0000) 2024-03-06T11:40:11.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:11.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:11.985652+0000) 2024-03-06T11:40:12.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:12.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:12.985868+0000) 2024-03-06T11:40:13.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:13.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:13.986122+0000) 2024-03-06T11:40:14.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:14.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:14.986329+0000) 2024-03-06T11:40:15.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:15.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:15.986513+0000) 2024-03-06T11:40:16.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:16.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:16.986796+0000) 2024-03-06T11:40:17.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:17.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:17.987004+0000) 2024-03-06T11:40:18.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:18.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:18.987229+0000) 2024-03-06T11:40:19.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:19.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:19.987452+0000) 2024-03-06T11:40:20.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:20.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:20.987711+0000) 2024-03-06T11:40:21.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:21.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:21.987900+0000) 2024-03-06T11:40:22.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:22.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:22.988136+0000) 2024-03-06T11:40:23.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:23.618 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:23.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:23.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:23.988383+0000) 2024-03-06T11:40:24.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:24.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:24.988627+0000) 2024-03-06T11:40:25.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:25.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:25.988882+0000) 2024-03-06T11:40:26.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:26.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:26.989132+0000) 2024-03-06T11:40:27.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:27.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:27.989354+0000) 2024-03-06T11:40:28.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:28.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:28.989544+0000) 2024-03-06T11:40:29.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:29.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:29.989773+0000) 2024-03-06T11:40:30.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:30.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:30.989959+0000) 2024-03-06T11:40:31.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:31.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:31.990254+0000) 2024-03-06T11:40:32.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:32.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:32.990510+0000) 2024-03-06T11:40:33.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:33.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:33.990801+0000) 2024-03-06T11:40:34.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:34.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:34.990992+0000) 2024-03-06T11:40:35.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:35.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:35.991237+0000) 2024-03-06T11:40:36.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:36.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:36.991430+0000) 2024-03-06T11:40:37.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:37.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:37.991661+0000) 2024-03-06T11:40:38.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:38.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:38.991916+0000) 2024-03-06T11:40:39.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:39.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:39.992140+0000) 2024-03-06T11:40:40.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:40.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:40.992364+0000) 2024-03-06T11:40:41.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:41.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:41.992556+0000) 2024-03-06T11:40:42.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:42.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:42.992798+0000) 2024-03-06T11:40:43.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:43.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:43.992995+0000) 2024-03-06T11:40:44.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:44.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:44.993147+0000) 2024-03-06T11:40:45.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:45.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:45.993367+0000) 2024-03-06T11:40:46.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:46.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:46.993594+0000) 2024-03-06T11:40:47.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:47.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:47.993837+0000) 2024-03-06T11:40:48.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:48.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:48.994004+0000) 2024-03-06T11:40:49.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:49.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:49.994232+0000) 2024-03-06T11:40:50.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:50.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:50.994458+0000) 2024-03-06T11:40:51.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:51.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:51.994712+0000) 2024-03-06T11:40:52.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:52.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:52.994902+0000) 2024-03-06T11:40:53.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:53.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:53.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:53.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:53.995135+0000) 2024-03-06T11:40:54.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:54.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:54.995346+0000) 2024-03-06T11:40:55.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:55.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:55.995570+0000) 2024-03-06T11:40:56.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:56.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:56.995805+0000) 2024-03-06T11:40:57.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:57.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:57.996001+0000) 2024-03-06T11:40:58.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:58.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:58.996182+0000) 2024-03-06T11:40:59.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:40:59.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:59.996409+0000) 2024-03-06T11:41:00.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:00.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:00.996635+0000) 2024-03-06T11:41:01.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:01.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:01.996832+0000) 2024-03-06T11:41:02.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:02.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:02.997047+0000) 2024-03-06T11:41:03.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:03.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:03.997238+0000) 2024-03-06T11:41:04.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:04.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:04.997454+0000) 2024-03-06T11:41:05.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:05.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:05.997728+0000) 2024-03-06T11:41:06.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:06.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:06.997981+0000) 2024-03-06T11:41:07.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:07.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:07.998162+0000) 2024-03-06T11:41:08.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:08.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:08.998339+0000) 2024-03-06T11:41:09.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:09.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:09.998522+0000) 2024-03-06T11:41:10.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:10.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:10.998772+0000) 2024-03-06T11:41:11.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:11.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:11.999015+0000) 2024-03-06T11:41:12.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:12.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:12.999247+0000) 2024-03-06T11:41:13.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:13.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:13.999469+0000) 2024-03-06T11:41:14.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:14.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:14.999727+0000) 2024-03-06T11:41:15.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:15.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:15.999972+0000) 2024-03-06T11:41:16.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:16.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:17.000156+0000) 2024-03-06T11:41:18.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:17.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:18.000375+0000) 2024-03-06T11:41:19.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:18.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:19.000595+0000) 2024-03-06T11:41:20.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:20.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:20.000820+0000) 2024-03-06T11:41:21.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:21.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:21.001020+0000) 2024-03-06T11:41:22.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:22.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:22.001212+0000) 2024-03-06T11:41:23.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:23.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:23.001399+0000) 2024-03-06T11:41:24.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:24.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:24.001587+0000) 2024-03-06T11:41:24.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:24.028 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:25.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:25.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:25.001878+0000) 2024-03-06T11:41:26.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:26.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:26.002162+0000) 2024-03-06T11:41:27.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:27.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:27.002366+0000) 2024-03-06T11:41:28.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:28.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:28.002584+0000) 2024-03-06T11:41:29.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:29.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:29.002821+0000) 2024-03-06T11:41:30.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:30.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:30.003024+0000) 2024-03-06T11:41:31.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:31.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:31.003226+0000) 2024-03-06T11:41:32.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:32.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:32.003459+0000) 2024-03-06T11:41:33.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:33.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:33.003708+0000) 2024-03-06T11:41:34.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:34.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:34.003903+0000) 2024-03-06T11:41:35.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:35.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:35.004131+0000) 2024-03-06T11:41:36.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:36.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:36.004373+0000) 2024-03-06T11:41:37.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:37.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:37.004593+0000) 2024-03-06T11:41:38.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:38.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:38.004831+0000) 2024-03-06T11:41:39.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:39.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:39.005030+0000) 2024-03-06T11:41:40.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:40.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:40.005256+0000) 2024-03-06T11:41:41.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:41.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:41.005480+0000) 2024-03-06T11:41:42.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:42.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:42.005767+0000) 2024-03-06T11:41:43.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:43.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:43.005951+0000) 2024-03-06T11:41:44.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:44.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:44.006179+0000) 2024-03-06T11:41:45.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:45.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:45.006403+0000) 2024-03-06T11:41:46.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:46.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:46.006647+0000) 2024-03-06T11:41:47.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:47.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:47.006951+0000) 2024-03-06T11:41:48.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:48.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:48.007202+0000) 2024-03-06T11:41:49.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:49.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:49.007428+0000) 2024-03-06T11:41:50.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:50.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:50.007653+0000) 2024-03-06T11:41:51.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:51.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:51.007913+0000) 2024-03-06T11:41:52.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:52.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:52.008156+0000) 2024-03-06T11:41:53.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:53.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:53.008375+0000) 2024-03-06T11:41:54.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:54.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:54.008608+0000) 2024-03-06T11:41:54.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:54.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:55.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:55.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:55.008804+0000) 2024-03-06T11:41:56.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:56.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:56.009026+0000) 2024-03-06T11:41:57.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:57.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:57.009263+0000) 2024-03-06T11:41:58.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:58.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:58.009489+0000) 2024-03-06T11:41:59.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:41:59.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:59.009751+0000) 2024-03-06T11:42:00.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:00.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:00.009985+0000) 2024-03-06T11:42:01.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:01.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:01.010220+0000) 2024-03-06T11:42:02.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:02.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:02.010453+0000) 2024-03-06T11:42:03.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:03.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:03.010645+0000) 2024-03-06T11:42:04.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:04.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:04.010926+0000) 2024-03-06T11:42:05.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:05.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:05.011148+0000) 2024-03-06T11:42:06.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:06.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:06.011385+0000) 2024-03-06T11:42:07.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:07.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:07.011616+0000) 2024-03-06T11:42:08.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:08.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:08.011839+0000) 2024-03-06T11:42:09.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:09.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:09.012037+0000) 2024-03-06T11:42:10.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:10.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:10.012225+0000) 2024-03-06T11:42:11.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:11.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:11.012474+0000) 2024-03-06T11:42:12.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:12.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:12.012774+0000) 2024-03-06T11:42:13.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:13.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:13.012967+0000) 2024-03-06T11:42:14.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:14.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:14.013260+0000) 2024-03-06T11:42:15.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:15.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:15.013484+0000) 2024-03-06T11:42:16.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:16.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:16.013760+0000) 2024-03-06T11:42:17.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:17.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:17.013974+0000) 2024-03-06T11:42:18.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:18.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:18.014166+0000) 2024-03-06T11:42:19.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:19.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:19.014392+0000) 2024-03-06T11:42:20.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:20.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:20.014618+0000) 2024-03-06T11:42:21.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:21.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:21.014821+0000) 2024-03-06T11:42:22.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:22.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:22.015043+0000) 2024-03-06T11:42:23.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:23.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:23.015218+0000) 2024-03-06T11:42:24.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:24.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:24.015446+0000) 2024-03-06T11:42:24.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:24.444 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:25.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:25.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:25.015707+0000) 2024-03-06T11:42:26.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:26.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:26.015924+0000) 2024-03-06T11:42:27.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:27.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:27.016152+0000) 2024-03-06T11:42:28.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:28.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:28.016333+0000) 2024-03-06T11:42:29.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:29.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:29.016554+0000) 2024-03-06T11:42:30.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:30.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:30.016783+0000) 2024-03-06T11:42:31.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:31.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:31.017027+0000) 2024-03-06T11:42:32.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:32.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:32.017177+0000) 2024-03-06T11:42:33.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:33.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:33.017364+0000) 2024-03-06T11:42:34.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:34.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:34.017601+0000) 2024-03-06T11:42:35.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:35.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:35.017826+0000) 2024-03-06T11:42:36.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:36.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:36.018063+0000) 2024-03-06T11:42:37.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:37.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:37.018298+0000) 2024-03-06T11:42:38.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:38.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:38.018522+0000) 2024-03-06T11:42:39.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:39.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:39.018783+0000) 2024-03-06T11:42:40.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:40.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:40.018983+0000) 2024-03-06T11:42:41.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:41.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:41.019213+0000) 2024-03-06T11:42:42.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:42.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:42.019440+0000) 2024-03-06T11:42:43.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:43.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:43.019657+0000) 2024-03-06T11:42:44.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:44.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:44.019925+0000) 2024-03-06T11:42:45.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:45.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:45.020113+0000) 2024-03-06T11:42:46.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:46.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:46.020304+0000) 2024-03-06T11:42:47.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:47.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:47.020545+0000) 2024-03-06T11:42:48.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:48.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:48.020782+0000) 2024-03-06T11:42:49.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:49.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:49.021013+0000) 2024-03-06T11:42:50.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:50.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:50.021252+0000) 2024-03-06T11:42:51.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:51.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:51.021437+0000) 2024-03-06T11:42:52.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:52.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:52.021703+0000) 2024-03-06T11:42:53.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:53.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:53.021894+0000) 2024-03-06T11:42:54.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:54.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:54.022134+0000) 2024-03-06T11:42:54.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:54.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:55.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:55.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:55.022367+0000) 2024-03-06T11:42:56.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:56.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:56.022539+0000) 2024-03-06T11:42:57.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:57.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:57.022781+0000) 2024-03-06T11:42:58.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:58.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:58.023012+0000) 2024-03-06T11:42:59.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:42:59.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:59.023238+0000) 2024-03-06T11:43:00.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:00.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:00.023463+0000) 2024-03-06T11:43:01.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:01.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:01.023731+0000) 2024-03-06T11:43:02.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:02.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:02.024014+0000) 2024-03-06T11:43:03.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:03.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:03.024232+0000) 2024-03-06T11:43:04.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:04.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:04.024455+0000) 2024-03-06T11:43:05.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:05.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:05.024650+0000) 2024-03-06T11:43:06.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:06.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:06.024889+0000) 2024-03-06T11:43:07.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:07.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:07.025135+0000) 2024-03-06T11:43:08.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:08.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:08.025362+0000) 2024-03-06T11:43:09.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:09.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:09.025551+0000) 2024-03-06T11:43:10.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:10.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:10.025773+0000) 2024-03-06T11:43:11.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:11.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:11.025991+0000) 2024-03-06T11:43:12.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:12.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:12.026234+0000) 2024-03-06T11:43:13.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:13.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:13.026469+0000) 2024-03-06T11:43:14.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:14.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:14.026758+0000) 2024-03-06T11:43:15.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:15.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:15.026966+0000) 2024-03-06T11:43:16.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:16.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:16.027215+0000) 2024-03-06T11:43:17.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:17.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:17.027489+0000) 2024-03-06T11:43:18.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:18.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:18.027772+0000) 2024-03-06T11:43:19.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:19.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:19.027977+0000) 2024-03-06T11:43:20.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:20.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:20.028218+0000) 2024-03-06T11:43:21.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:21.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:21.028477+0000) 2024-03-06T11:43:22.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:22.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:22.028769+0000) 2024-03-06T11:43:23.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:23.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:23.028963+0000) 2024-03-06T11:43:24.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:24.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:24.029149+0000) 2024-03-06T11:43:24.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:24.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:25.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:25.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:25.029353+0000) 2024-03-06T11:43:26.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:26.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:26.029598+0000) 2024-03-06T11:43:27.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:27.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:27.029841+0000) 2024-03-06T11:43:28.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:28.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:28.030056+0000) 2024-03-06T11:43:29.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:29.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:29.030288+0000) 2024-03-06T11:43:30.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:30.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:30.030474+0000) 2024-03-06T11:43:31.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:31.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:31.030723+0000) 2024-03-06T11:43:32.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:32.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:32.030997+0000) 2024-03-06T11:43:33.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:33.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:33.031262+0000) 2024-03-06T11:43:34.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:34.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:34.031527+0000) 2024-03-06T11:43:35.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:35.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:35.031813+0000) 2024-03-06T11:43:36.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:36.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:36.032122+0000) 2024-03-06T11:43:37.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:37.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:37.032329+0000) 2024-03-06T11:43:38.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:38.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:38.032603+0000) 2024-03-06T11:43:39.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:39.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:39.032833+0000) 2024-03-06T11:43:40.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:40.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:40.033022+0000) 2024-03-06T11:43:41.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:41.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:41.033231+0000) 2024-03-06T11:43:42.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:42.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:42.033415+0000) 2024-03-06T11:43:43.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:43.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:43.033706+0000) 2024-03-06T11:43:44.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:44.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:44.033941+0000) 2024-03-06T11:43:45.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:45.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:45.034101+0000) 2024-03-06T11:43:46.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:46.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:46.034300+0000) 2024-03-06T11:43:47.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:47.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:47.034455+0000) 2024-03-06T11:43:48.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:48.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:48.034721+0000) 2024-03-06T11:43:49.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:49.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:49.034944+0000) 2024-03-06T11:43:50.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:50.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:50.035178+0000) 2024-03-06T11:43:51.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:51.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:51.035344+0000) 2024-03-06T11:43:52.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:52.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:52.035578+0000) 2024-03-06T11:43:53.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:53.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:53.035753+0000) 2024-03-06T11:43:54.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:54.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:54.035992+0000) 2024-03-06T11:43:55.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:55.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:55.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:55.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:55.036195+0000) 2024-03-06T11:43:56.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:56.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:56.036335+0000) 2024-03-06T11:43:57.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:57.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:57.036613+0000) 2024-03-06T11:43:58.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:58.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:58.036793+0000) 2024-03-06T11:43:59.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:43:59.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:59.037016+0000) 2024-03-06T11:44:00.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:00.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:00.037301+0000) 2024-03-06T11:44:01.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:01.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:01.037534+0000) 2024-03-06T11:44:02.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:02.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:02.037813+0000) 2024-03-06T11:44:03.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:03.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:03.038058+0000) 2024-03-06T11:44:04.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:04.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:04.038343+0000) 2024-03-06T11:44:05.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:05.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:05.038504+0000) 2024-03-06T11:44:06.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:06.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:06.038745+0000) 2024-03-06T11:44:07.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:07.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:07.038971+0000) 2024-03-06T11:44:08.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:08.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:08.039204+0000) 2024-03-06T11:44:09.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:09.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:09.039419+0000) 2024-03-06T11:44:10.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:10.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:10.039661+0000) 2024-03-06T11:44:11.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:11.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:11.039938+0000) 2024-03-06T11:44:12.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:12.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:12.040133+0000) 2024-03-06T11:44:13.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:13.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:13.040365+0000) 2024-03-06T11:44:14.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:14.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:14.040592+0000) 2024-03-06T11:44:15.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:15.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:15.040830+0000) 2024-03-06T11:44:16.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:16.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:16.041070+0000) 2024-03-06T11:44:17.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:17.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:17.041280+0000) 2024-03-06T11:44:18.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:18.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:18.041469+0000) 2024-03-06T11:44:19.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:19.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:19.041772+0000) 2024-03-06T11:44:20.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:20.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:20.041981+0000) 2024-03-06T11:44:21.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:21.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:21.042218+0000) 2024-03-06T11:44:22.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:22.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:22.042447+0000) 2024-03-06T11:44:23.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:23.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:23.042647+0000) 2024-03-06T11:44:24.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:24.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:24.042854+0000) 2024-03-06T11:44:25.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:25.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:25.043067+0000) 2024-03-06T11:44:25.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:25.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:26.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:26.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:26.043311+0000) 2024-03-06T11:44:27.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:27.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:27.043484+0000) 2024-03-06T11:44:28.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:28.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:28.043768+0000) 2024-03-06T11:44:29.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:29.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:29.044004+0000) 2024-03-06T11:44:30.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:30.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:30.044213+0000) 2024-03-06T11:44:31.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:31.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:31.044433+0000) 2024-03-06T11:44:32.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:32.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:32.044694+0000) 2024-03-06T11:44:33.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:33.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:33.044882+0000) 2024-03-06T11:44:34.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:34.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:34.045106+0000) 2024-03-06T11:44:35.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:35.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:35.045332+0000) 2024-03-06T11:44:36.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:36.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:36.045471+0000) 2024-03-06T11:44:37.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:37.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:37.045748+0000) 2024-03-06T11:44:38.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:38.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:38.045976+0000) 2024-03-06T11:44:39.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:39.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:39.046202+0000) 2024-03-06T11:44:40.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:40.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:40.046431+0000) 2024-03-06T11:44:41.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:41.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:41.046656+0000) 2024-03-06T11:44:42.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:42.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:42.046920+0000) 2024-03-06T11:44:43.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:43.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:43.047147+0000) 2024-03-06T11:44:44.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:44.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:44.047300+0000) 2024-03-06T11:44:45.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:45.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:45.047450+0000) 2024-03-06T11:44:46.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:46.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:46.047725+0000) 2024-03-06T11:44:47.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:47.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:47.047992+0000) 2024-03-06T11:44:48.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:48.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:48.048280+0000) 2024-03-06T11:44:49.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:49.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:49.048510+0000) 2024-03-06T11:44:50.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:50.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:50.048776+0000) 2024-03-06T11:44:51.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:51.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:51.048963+0000) 2024-03-06T11:44:52.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:52.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:52.049207+0000) 2024-03-06T11:44:53.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:53.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:53.049431+0000) 2024-03-06T11:44:54.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:54.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:54.049636+0000) 2024-03-06T11:44:55.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:55.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:55.049908+0000) 2024-03-06T11:44:55.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:55.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:56.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:56.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:56.050068+0000) 2024-03-06T11:44:57.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:57.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:57.050314+0000) 2024-03-06T11:44:58.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:58.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:58.050464+0000) 2024-03-06T11:44:59.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:44:59.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:59.050736+0000) 2024-03-06T11:45:00.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:00.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:00.050896+0000) 2024-03-06T11:45:01.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:01.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:01.051136+0000) 2024-03-06T11:45:02.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:02.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:02.051389+0000) 2024-03-06T11:45:03.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:03.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:03.051609+0000) 2024-03-06T11:45:04.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:04.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:04.051846+0000) 2024-03-06T11:45:05.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:05.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:05.052071+0000) 2024-03-06T11:45:06.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:06.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:06.052303+0000) 2024-03-06T11:45:07.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:07.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:07.052469+0000) 2024-03-06T11:45:08.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:08.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:08.052694+0000) 2024-03-06T11:45:09.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:09.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:09.052900+0000) 2024-03-06T11:45:10.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:10.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:10.053141+0000) 2024-03-06T11:45:11.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:11.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:11.053334+0000) 2024-03-06T11:45:12.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:12.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:12.053492+0000) 2024-03-06T11:45:13.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:13.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:13.053633+0000) 2024-03-06T11:45:14.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:14.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:14.053905+0000) 2024-03-06T11:45:15.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:15.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:15.054142+0000) 2024-03-06T11:45:16.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:16.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:16.054339+0000) 2024-03-06T11:45:17.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:17.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:17.054585+0000) 2024-03-06T11:45:18.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:18.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:18.054770+0000) 2024-03-06T11:45:19.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:19.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:19.054981+0000) 2024-03-06T11:45:20.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:20.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:20.055230+0000) 2024-03-06T11:45:21.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:21.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:21.055408+0000) 2024-03-06T11:45:22.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:22.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:22.055638+0000) 2024-03-06T11:45:23.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:23.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:23.055890+0000) 2024-03-06T11:45:24.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:24.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:24.056117+0000) 2024-03-06T11:45:25.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:25.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:25.056331+0000) 2024-03-06T11:45:25.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:25.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:26.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:26.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:26.056563+0000) 2024-03-06T11:45:27.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:27.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:27.056785+0000) 2024-03-06T11:45:28.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:28.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:28.057021+0000) 2024-03-06T11:45:29.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:29.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:29.057238+0000) 2024-03-06T11:45:30.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:30.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:30.057471+0000) 2024-03-06T11:45:31.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:31.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:31.057732+0000) 2024-03-06T11:45:32.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:32.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:32.057961+0000) 2024-03-06T11:45:33.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:33.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:33.058181+0000) 2024-03-06T11:45:34.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:34.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:34.058423+0000) 2024-03-06T11:45:35.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:35.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:35.058659+0000) 2024-03-06T11:45:36.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:36.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:36.058915+0000) 2024-03-06T11:45:37.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:37.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:37.059063+0000) 2024-03-06T11:45:38.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:38.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:38.059295+0000) 2024-03-06T11:45:39.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:39.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:39.059522+0000) 2024-03-06T11:45:40.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:40.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:40.059706+0000) 2024-03-06T11:45:41.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:41.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:41.059946+0000) 2024-03-06T11:45:42.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:42.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:42.060179+0000) 2024-03-06T11:45:43.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:43.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:43.060418+0000) 2024-03-06T11:45:44.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:44.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:44.060705+0000) 2024-03-06T11:45:45.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:45.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:45.060877+0000) 2024-03-06T11:45:46.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:46.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:46.061068+0000) 2024-03-06T11:45:47.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:47.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:47.061312+0000) 2024-03-06T11:45:48.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:48.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:48.061550+0000) 2024-03-06T11:45:49.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:49.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:49.061780+0000) 2024-03-06T11:45:50.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:50.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:50.061973+0000) 2024-03-06T11:45:51.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:51.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:51.062198+0000) 2024-03-06T11:45:52.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:52.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:52.062436+0000) 2024-03-06T11:45:53.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:53.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:53.062574+0000) 2024-03-06T11:45:54.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:54.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:54.062744+0000) 2024-03-06T11:45:55.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:55.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:55.062978+0000) 2024-03-06T11:45:55.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:55.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:56.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:56.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:56.063214+0000) 2024-03-06T11:45:57.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:57.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:57.063454+0000) 2024-03-06T11:45:58.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:58.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:58.063657+0000) 2024-03-06T11:45:59.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:45:59.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:59.063905+0000) 2024-03-06T11:46:00.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:00.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:00.064117+0000) 2024-03-06T11:46:01.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:01.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:01.064376+0000) 2024-03-06T11:46:02.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:02.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:02.064615+0000) 2024-03-06T11:46:03.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:03.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:03.064793+0000) 2024-03-06T11:46:04.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:04.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:04.065036+0000) 2024-03-06T11:46:05.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:05.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:05.065258+0000) 2024-03-06T11:46:06.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:06.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:06.065485+0000) 2024-03-06T11:46:07.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:07.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:07.065765+0000) 2024-03-06T11:46:08.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:08.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:08.065992+0000) 2024-03-06T11:46:09.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:09.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:09.066235+0000) 2024-03-06T11:46:10.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:10.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:10.066482+0000) 2024-03-06T11:46:11.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:11.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:11.066680+0000) 2024-03-06T11:46:12.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:12.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:12.066865+0000) 2024-03-06T11:46:13.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:13.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:13.067090+0000) 2024-03-06T11:46:14.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:14.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:14.067324+0000) 2024-03-06T11:46:15.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:15.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:15.067546+0000) 2024-03-06T11:46:16.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:16.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:16.067780+0000) 2024-03-06T11:46:17.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:17.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:17.068003+0000) 2024-03-06T11:46:18.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:18.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:18.068200+0000) 2024-03-06T11:46:19.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:19.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:19.068462+0000) 2024-03-06T11:46:20.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:20.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:20.068706+0000) 2024-03-06T11:46:21.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:21.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:21.068902+0000) 2024-03-06T11:46:22.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:22.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:22.069131+0000) 2024-03-06T11:46:23.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:23.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:23.069352+0000) 2024-03-06T11:46:24.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:24.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:24.069564+0000) 2024-03-06T11:46:25.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:25.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:25.069823+0000) 2024-03-06T11:46:26.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:26.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:26.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:26.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:26.069973+0000) 2024-03-06T11:46:27.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:27.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:27.070148+0000) 2024-03-06T11:46:28.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:28.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:28.070379+0000) 2024-03-06T11:46:29.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:29.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:29.070574+0000) 2024-03-06T11:46:30.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:30.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:30.070879+0000) 2024-03-06T11:46:31.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:31.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:31.071100+0000) 2024-03-06T11:46:32.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:32.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:32.071295+0000) 2024-03-06T11:46:33.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:33.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:33.071478+0000) 2024-03-06T11:46:34.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:34.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:34.071747+0000) 2024-03-06T11:46:35.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:35.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:35.071974+0000) 2024-03-06T11:46:36.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:36.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:36.072198+0000) 2024-03-06T11:46:37.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:37.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:37.072431+0000) 2024-03-06T11:46:38.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:38.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:38.072570+0000) 2024-03-06T11:46:39.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:39.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:39.072770+0000) 2024-03-06T11:46:40.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:40.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:40.072940+0000) 2024-03-06T11:46:41.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:41.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:41.073174+0000) 2024-03-06T11:46:42.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:42.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:42.073422+0000) 2024-03-06T11:46:43.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:43.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:43.073567+0000) 2024-03-06T11:46:44.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:44.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:44.073821+0000) 2024-03-06T11:46:45.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:45.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:45.074041+0000) 2024-03-06T11:46:46.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:46.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:46.074237+0000) 2024-03-06T11:46:47.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:47.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:47.074415+0000) 2024-03-06T11:46:48.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:48.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:48.074603+0000) 2024-03-06T11:46:49.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:49.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:49.074833+0000) 2024-03-06T11:46:50.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:50.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:50.075069+0000) 2024-03-06T11:46:51.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:51.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:51.075267+0000) 2024-03-06T11:46:52.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:52.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:52.075458+0000) 2024-03-06T11:46:53.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:53.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:53.075627+0000) 2024-03-06T11:46:54.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:54.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:54.075859+0000) 2024-03-06T11:46:55.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:55.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:55.076034+0000) 2024-03-06T11:46:56.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:56.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:56.076213+0000) 2024-03-06T11:46:56.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:56.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:57.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:57.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:57.076381+0000) 2024-03-06T11:46:58.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:58.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:58.076536+0000) 2024-03-06T11:46:59.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:46:59.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:59.076752+0000) 2024-03-06T11:47:00.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:00.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:00.076989+0000) 2024-03-06T11:47:01.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:01.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:01.077211+0000) 2024-03-06T11:47:02.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:02.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:02.077412+0000) 2024-03-06T11:47:03.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:03.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:03.077631+0000) 2024-03-06T11:47:04.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:04.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:04.077842+0000) 2024-03-06T11:47:05.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:05.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:05.078036+0000) 2024-03-06T11:47:06.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:06.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:06.078235+0000) 2024-03-06T11:47:07.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:07.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:07.078422+0000) 2024-03-06T11:47:08.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:08.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:08.078586+0000) 2024-03-06T11:47:09.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:09.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:09.078781+0000) 2024-03-06T11:47:10.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:10.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:10.078926+0000) 2024-03-06T11:47:11.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:11.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:11.079105+0000) 2024-03-06T11:47:12.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:12.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:12.079308+0000) 2024-03-06T11:47:13.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:13.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:13.079498+0000) 2024-03-06T11:47:14.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:14.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:14.079756+0000) 2024-03-06T11:47:15.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:15.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:15.079979+0000) 2024-03-06T11:47:16.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:16.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:16.080203+0000) 2024-03-06T11:47:17.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:17.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:17.080442+0000) 2024-03-06T11:47:18.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:18.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:18.080701+0000) 2024-03-06T11:47:19.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:19.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:19.080929+0000) 2024-03-06T11:47:20.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:20.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:20.081175+0000) 2024-03-06T11:47:21.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:21.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:21.081415+0000) 2024-03-06T11:47:22.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:22.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:22.081578+0000) 2024-03-06T11:47:23.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:23.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:23.081902+0000) 2024-03-06T11:47:24.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:24.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:24.082072+0000) 2024-03-06T11:47:25.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:25.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:25.082269+0000) 2024-03-06T11:47:26.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:26.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:26.082503+0000) 2024-03-06T11:47:26.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:26.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:27.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:27.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:27.082779+0000) 2024-03-06T11:47:28.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:28.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:28.082965+0000) 2024-03-06T11:47:29.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:29.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:29.083204+0000) 2024-03-06T11:47:30.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:30.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:30.083430+0000) 2024-03-06T11:47:31.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:31.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:31.083653+0000) 2024-03-06T11:47:32.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:32.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:32.083963+0000) 2024-03-06T11:47:33.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:33.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:33.084183+0000) 2024-03-06T11:47:34.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:34.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:34.084373+0000) 2024-03-06T11:47:35.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:35.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:35.084595+0000) 2024-03-06T11:47:36.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:36.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:36.084811+0000) 2024-03-06T11:47:37.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:37.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:37.085054+0000) 2024-03-06T11:47:38.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:38.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:38.085287+0000) 2024-03-06T11:47:39.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:39.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:39.085531+0000) 2024-03-06T11:47:40.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:40.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:40.085719+0000) 2024-03-06T11:47:41.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:41.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:41.085946+0000) 2024-03-06T11:47:42.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:42.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:42.086188+0000) 2024-03-06T11:47:43.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:43.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:43.086390+0000) 2024-03-06T11:47:44.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:44.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:44.086577+0000) 2024-03-06T11:47:45.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:45.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:45.086830+0000) 2024-03-06T11:47:46.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:46.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:46.087036+0000) 2024-03-06T11:47:47.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:47.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:47.087282+0000) 2024-03-06T11:47:48.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:48.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:48.087491+0000) 2024-03-06T11:47:49.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:49.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:49.087735+0000) 2024-03-06T11:47:50.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:50.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:50.087981+0000) 2024-03-06T11:47:51.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:51.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:51.088157+0000) 2024-03-06T11:47:52.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:52.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:52.088349+0000) 2024-03-06T11:47:53.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:53.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:53.088510+0000) 2024-03-06T11:47:54.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:54.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:54.088793+0000) 2024-03-06T11:47:55.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:55.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:55.089027+0000) 2024-03-06T11:47:56.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:56.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:56.089257+0000) 2024-03-06T11:47:56.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:56.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:57.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:57.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:57.089525+0000) 2024-03-06T11:47:58.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:58.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:58.089717+0000) 2024-03-06T11:47:59.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:47:59.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:59.089920+0000) 2024-03-06T11:48:00.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:00.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:00.090124+0000) 2024-03-06T11:48:01.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:01.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:01.090306+0000) 2024-03-06T11:48:02.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:02.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:02.090464+0000) 2024-03-06T11:48:03.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:03.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:03.090727+0000) 2024-03-06T11:48:04.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:04.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:04.090935+0000) 2024-03-06T11:48:05.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:05.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:05.091162+0000) 2024-03-06T11:48:06.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:06.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:06.091442+0000) 2024-03-06T11:48:07.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:07.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:07.091694+0000) 2024-03-06T11:48:08.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:08.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:08.091924+0000) 2024-03-06T11:48:09.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:09.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:09.092152+0000) 2024-03-06T11:48:10.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:10.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:10.092344+0000) 2024-03-06T11:48:11.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:11.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:11.092565+0000) 2024-03-06T11:48:12.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:12.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:12.092819+0000) 2024-03-06T11:48:13.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:13.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:13.092957+0000) 2024-03-06T11:48:14.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:14.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:14.093226+0000) 2024-03-06T11:48:15.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:15.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:15.093511+0000) 2024-03-06T11:48:16.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:16.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:16.093836+0000) 2024-03-06T11:48:17.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:17.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:17.094122+0000) 2024-03-06T11:48:18.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:18.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:18.094393+0000) 2024-03-06T11:48:19.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:19.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:19.094699+0000) 2024-03-06T11:48:20.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:20.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:20.094920+0000) 2024-03-06T11:48:21.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:21.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:21.095133+0000) 2024-03-06T11:48:22.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:22.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:22.095379+0000) 2024-03-06T11:48:23.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:23.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:23.095597+0000) 2024-03-06T11:48:24.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:24.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:24.095832+0000) 2024-03-06T11:48:25.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:25.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:25.096059+0000) 2024-03-06T11:48:26.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:26.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:26.096287+0000) 2024-03-06T11:48:26.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:26.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:27.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:27.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:27.096530+0000) 2024-03-06T11:48:28.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:28.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:28.096791+0000) 2024-03-06T11:48:29.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:29.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:29.096989+0000) 2024-03-06T11:48:30.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:30.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:30.097212+0000) 2024-03-06T11:48:31.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:31.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:31.097433+0000) 2024-03-06T11:48:32.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:32.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:32.097647+0000) 2024-03-06T11:48:33.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:33.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:33.097925+0000) 2024-03-06T11:48:34.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:34.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:34.098159+0000) 2024-03-06T11:48:35.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:35.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:35.098392+0000) 2024-03-06T11:48:36.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:36.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:36.098617+0000) 2024-03-06T11:48:37.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:37.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:37.098834+0000) 2024-03-06T11:48:38.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:38.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:38.099083+0000) 2024-03-06T11:48:39.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:39.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:39.099280+0000) 2024-03-06T11:48:40.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:40.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:40.099517+0000) 2024-03-06T11:48:41.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:41.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:41.099731+0000) 2024-03-06T11:48:42.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:42.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:42.099974+0000) 2024-03-06T11:48:43.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:43.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:43.100190+0000) 2024-03-06T11:48:44.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:44.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:44.100372+0000) 2024-03-06T11:48:45.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:45.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:45.100591+0000) 2024-03-06T11:48:46.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:46.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:46.100814+0000) 2024-03-06T11:48:47.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:47.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:47.101093+0000) 2024-03-06T11:48:48.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:48.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:48.101306+0000) 2024-03-06T11:48:49.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:49.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:49.101522+0000) 2024-03-06T11:48:50.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:50.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:50.101796+0000) 2024-03-06T11:48:51.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:51.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:51.101963+0000) 2024-03-06T11:48:52.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:52.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:52.102206+0000) 2024-03-06T11:48:53.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:53.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:53.102469+0000) 2024-03-06T11:48:54.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:54.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:54.102774+0000) 2024-03-06T11:48:55.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:55.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:55.102934+0000) 2024-03-06T11:48:56.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:56.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:56.103156+0000) 2024-03-06T11:48:56.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:56.995 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:57.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:57.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:57.103346+0000) 2024-03-06T11:48:58.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:58.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:58.103569+0000) 2024-03-06T11:48:59.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:48:59.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:59.103796+0000) 2024-03-06T11:49:00.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:00.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:00.103991+0000) 2024-03-06T11:49:01.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:01.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:01.104210+0000) 2024-03-06T11:49:02.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:02.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:02.104446+0000) 2024-03-06T11:49:03.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:03.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:03.104645+0000) 2024-03-06T11:49:04.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:04.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:04.104929+0000) 2024-03-06T11:49:05.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:05.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:05.105117+0000) 2024-03-06T11:49:06.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:06.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:06.105341+0000) 2024-03-06T11:49:07.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:07.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:07.105587+0000) 2024-03-06T11:49:08.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:08.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:08.105823+0000) 2024-03-06T11:49:09.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:09.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:09.106009+0000) 2024-03-06T11:49:10.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:10.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:10.106233+0000) 2024-03-06T11:49:11.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:11.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:11.106452+0000) 2024-03-06T11:49:12.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:12.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:12.106720+0000) 2024-03-06T11:49:13.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:13.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:13.106929+0000) 2024-03-06T11:49:14.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:14.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:14.107155+0000) 2024-03-06T11:49:15.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:15.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:15.107379+0000) 2024-03-06T11:49:16.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:16.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:16.107603+0000) 2024-03-06T11:49:17.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:17.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:17.107920+0000) 2024-03-06T11:49:18.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:18.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:18.108126+0000) 2024-03-06T11:49:19.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:19.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:19.108347+0000) 2024-03-06T11:49:20.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:20.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:20.108572+0000) 2024-03-06T11:49:21.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:21.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:21.108795+0000) 2024-03-06T11:49:22.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:22.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:22.109015+0000) 2024-03-06T11:49:23.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:23.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:23.109250+0000) 2024-03-06T11:49:24.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:24.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:24.109491+0000) 2024-03-06T11:49:25.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:25.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:25.109772+0000) 2024-03-06T11:49:26.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:26.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:26.109984+0000) 2024-03-06T11:49:27.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:27.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:27.110222+0000) 2024-03-06T11:49:27.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:27.194 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:28.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:28.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:28.110451+0000) 2024-03-06T11:49:29.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:29.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:29.110704+0000) 2024-03-06T11:49:30.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:30.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:30.110861+0000) 2024-03-06T11:49:31.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:31.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:31.111052+0000) 2024-03-06T11:49:32.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:32.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:32.111304+0000) 2024-03-06T11:49:33.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:33.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:33.111584+0000) 2024-03-06T11:49:34.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:34.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:34.111861+0000) 2024-03-06T11:49:35.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:35.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:35.112100+0000) 2024-03-06T11:49:36.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:36.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:36.112307+0000) 2024-03-06T11:49:37.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:37.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:37.112564+0000) 2024-03-06T11:49:38.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:38.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:38.112822+0000) 2024-03-06T11:49:39.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:39.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:39.112985+0000) 2024-03-06T11:49:40.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:40.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:40.113163+0000) 2024-03-06T11:49:41.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:41.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:41.113372+0000) 2024-03-06T11:49:42.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:42.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:42.113572+0000) 2024-03-06T11:49:43.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:43.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:43.113795+0000) 2024-03-06T11:49:44.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:44.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:44.113989+0000) 2024-03-06T11:49:45.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:45.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:45.114206+0000) 2024-03-06T11:49:46.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:46.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:46.114387+0000) 2024-03-06T11:49:47.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:47.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:47.114626+0000) 2024-03-06T11:49:48.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:48.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:48.114840+0000) 2024-03-06T11:49:49.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:49.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:49.115064+0000) 2024-03-06T11:49:50.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:50.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:50.115248+0000) 2024-03-06T11:49:51.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:51.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:51.115439+0000) 2024-03-06T11:49:52.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:52.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:52.115690+0000) 2024-03-06T11:49:53.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:53.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:53.115891+0000) 2024-03-06T11:49:54.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:54.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:54.116091+0000) 2024-03-06T11:49:55.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:55.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:55.116286+0000) 2024-03-06T11:49:56.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:56.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:56.116518+0000) 2024-03-06T11:49:57.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:57.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:57.116801+0000) 2024-03-06T11:49:57.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:57.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:58.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:58.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:58.117031+0000) 2024-03-06T11:49:59.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:49:59.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:59.117252+0000) 2024-03-06T11:50:00.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:00.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:00.117475+0000) 2024-03-06T11:50:01.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:01.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:01.117755+0000) 2024-03-06T11:50:02.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:02.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:02.117959+0000) 2024-03-06T11:50:03.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:03.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:03.118244+0000) 2024-03-06T11:50:04.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:04.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:04.118480+0000) 2024-03-06T11:50:05.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:05.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:05.118764+0000) 2024-03-06T11:50:06.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:06.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:06.118955+0000) 2024-03-06T11:50:07.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:07.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:07.119191+0000) 2024-03-06T11:50:08.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:08.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:08.119432+0000) 2024-03-06T11:50:09.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:09.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:09.119696+0000) 2024-03-06T11:50:10.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:10.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:10.119915+0000) 2024-03-06T11:50:11.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:11.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:11.120135+0000) 2024-03-06T11:50:12.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:12.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:12.120357+0000) 2024-03-06T11:50:13.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:13.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:13.120583+0000) 2024-03-06T11:50:14.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:14.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:14.120775+0000) 2024-03-06T11:50:15.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:15.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:15.120966+0000) 2024-03-06T11:50:16.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:16.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:16.121169+0000) 2024-03-06T11:50:17.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:17.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:17.121410+0000) 2024-03-06T11:50:18.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:18.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:18.121634+0000) 2024-03-06T11:50:19.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:19.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:19.121871+0000) 2024-03-06T11:50:20.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:20.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:20.122025+0000) 2024-03-06T11:50:21.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:21.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:21.122244+0000) 2024-03-06T11:50:22.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:22.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:22.122467+0000) 2024-03-06T11:50:23.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:23.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:23.122727+0000) 2024-03-06T11:50:24.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:24.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:24.122922+0000) 2024-03-06T11:50:25.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:25.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:25.123149+0000) 2024-03-06T11:50:26.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:26.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:26.123391+0000) 2024-03-06T11:50:27.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:27.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:27.123640+0000) 2024-03-06T11:50:27.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:27.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:28.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:28.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:28.123890+0000) 2024-03-06T11:50:29.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:29.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:29.124091+0000) 2024-03-06T11:50:30.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:30.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:30.124326+0000) 2024-03-06T11:50:31.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:31.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:31.124543+0000) 2024-03-06T11:50:32.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:32.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:32.124778+0000) 2024-03-06T11:50:33.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:33.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:33.125026+0000) 2024-03-06T11:50:34.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:34.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:34.125226+0000) 2024-03-06T11:50:35.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:35.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:35.125406+0000) 2024-03-06T11:50:36.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:36.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:36.125631+0000) 2024-03-06T11:50:37.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:37.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:37.125961+0000) 2024-03-06T11:50:38.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:38.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:38.126168+0000) 2024-03-06T11:50:39.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:39.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:39.126390+0000) 2024-03-06T11:50:40.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:40.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:40.126576+0000) 2024-03-06T11:50:41.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:41.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:41.126790+0000) 2024-03-06T11:50:42.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:42.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:42.126991+0000) 2024-03-06T11:50:43.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:43.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:43.127217+0000) 2024-03-06T11:50:44.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:44.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:44.127444+0000) 2024-03-06T11:50:45.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:45.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:45.127623+0000) 2024-03-06T11:50:46.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:46.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:46.127775+0000) 2024-03-06T11:50:47.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:47.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:47.128002+0000) 2024-03-06T11:50:48.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:48.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:48.128231+0000) 2024-03-06T11:50:49.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:49.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:49.128453+0000) 2024-03-06T11:50:50.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:50.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:50.128705+0000) 2024-03-06T11:50:51.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:51.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:51.128988+0000) 2024-03-06T11:50:52.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:52.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:52.129212+0000) 2024-03-06T11:50:53.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:53.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:53.129436+0000) 2024-03-06T11:50:54.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:54.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:54.129590+0000) 2024-03-06T11:50:55.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:55.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:55.129814+0000) 2024-03-06T11:50:56.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:56.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:56.129991+0000) 2024-03-06T11:50:57.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:57.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:57.130170+0000) 2024-03-06T11:50:57.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:57.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:58.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:58.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:58.130352+0000) 2024-03-06T11:50:59.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:50:59.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:59.130532+0000) 2024-03-06T11:51:00.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:00.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:00.130790+0000) 2024-03-06T11:51:01.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:01.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:01.130975+0000) 2024-03-06T11:51:02.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:02.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:02.131195+0000) 2024-03-06T11:51:03.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:03.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:03.131414+0000) 2024-03-06T11:51:04.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:04.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:04.131640+0000) 2024-03-06T11:51:05.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:05.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:05.131893+0000) 2024-03-06T11:51:06.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:06.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:06.132040+0000) 2024-03-06T11:51:07.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:07.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:07.132258+0000) 2024-03-06T11:51:08.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:08.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:08.132489+0000) 2024-03-06T11:51:09.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:09.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:09.132748+0000) 2024-03-06T11:51:10.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:10.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:10.132972+0000) 2024-03-06T11:51:11.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:11.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:11.133200+0000) 2024-03-06T11:51:12.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:12.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:12.133411+0000) 2024-03-06T11:51:13.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:13.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:13.133618+0000) 2024-03-06T11:51:14.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:14.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:14.133829+0000) 2024-03-06T11:51:15.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:15.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:15.134021+0000) 2024-03-06T11:51:16.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:16.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:16.134250+0000) 2024-03-06T11:51:17.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:17.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:17.134480+0000) 2024-03-06T11:51:18.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:18.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:18.134705+0000) 2024-03-06T11:51:19.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:19.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:19.134917+0000) 2024-03-06T11:51:20.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:20.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:20.135140+0000) 2024-03-06T11:51:21.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:21.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:21.135362+0000) 2024-03-06T11:51:22.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:22.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:22.135566+0000) 2024-03-06T11:51:23.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:23.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:23.135771+0000) 2024-03-06T11:51:24.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:24.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:24.135959+0000) 2024-03-06T11:51:25.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:25.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:25.136191+0000) 2024-03-06T11:51:26.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:26.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:26.136388+0000) 2024-03-06T11:51:27.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:27.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:27.136636+0000) 2024-03-06T11:51:27.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:27.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:28.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:28.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:28.136893+0000) 2024-03-06T11:51:29.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:29.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:29.137050+0000) 2024-03-06T11:51:30.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:30.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:30.137272+0000) 2024-03-06T11:51:31.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:31.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:31.137493+0000) 2024-03-06T11:51:32.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:32.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:32.137764+0000) 2024-03-06T11:51:33.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:33.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:33.137947+0000) 2024-03-06T11:51:34.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:34.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:34.138176+0000) 2024-03-06T11:51:35.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:35.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:35.138399+0000) 2024-03-06T11:51:36.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:36.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:36.138615+0000) 2024-03-06T11:51:37.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:37.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:37.138829+0000) 2024-03-06T11:51:38.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:38.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:38.139030+0000) 2024-03-06T11:51:39.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:39.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:39.139270+0000) 2024-03-06T11:51:40.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:40.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:40.139512+0000) 2024-03-06T11:51:41.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:41.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:41.139783+0000) 2024-03-06T11:51:42.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:42.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:42.139961+0000) 2024-03-06T11:51:43.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:43.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:43.140201+0000) 2024-03-06T11:51:44.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:44.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:44.140477+0000) 2024-03-06T11:51:45.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:45.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:45.140790+0000) 2024-03-06T11:51:46.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:46.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:46.141020+0000) 2024-03-06T11:51:47.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:47.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:47.141284+0000) 2024-03-06T11:51:48.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:48.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:48.141566+0000) 2024-03-06T11:51:49.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:49.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:49.141786+0000) 2024-03-06T11:51:50.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:50.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:50.142018+0000) 2024-03-06T11:51:51.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:51.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:51.142237+0000) 2024-03-06T11:51:52.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:52.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:52.142460+0000) 2024-03-06T11:51:53.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:53.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:53.142704+0000) 2024-03-06T11:51:54.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:54.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:54.142969+0000) 2024-03-06T11:51:55.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:55.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:55.143257+0000) 2024-03-06T11:51:56.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:56.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:56.143458+0000) 2024-03-06T11:51:57.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:57.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:57.143721+0000) 2024-03-06T11:51:58.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:58.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:58.143955+0000) 2024-03-06T11:51:58.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:58.180 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:59.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:51:59.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:59.144143+0000) 2024-03-06T11:52:00.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:00.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:00.144348+0000) 2024-03-06T11:52:01.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:01.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:01.144585+0000) 2024-03-06T11:52:02.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:02.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:02.144824+0000) 2024-03-06T11:52:03.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:03.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:03.145014+0000) 2024-03-06T11:52:04.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:04.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:04.145234+0000) 2024-03-06T11:52:05.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:05.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:05.145414+0000) 2024-03-06T11:52:06.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:06.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:06.145651+0000) 2024-03-06T11:52:07.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:07.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:07.145907+0000) 2024-03-06T11:52:08.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:08.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:08.146094+0000) 2024-03-06T11:52:09.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:09.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:09.146322+0000) 2024-03-06T11:52:10.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:10.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:10.146544+0000) 2024-03-06T11:52:11.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:11.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:11.146810+0000) 2024-03-06T11:52:12.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:12.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:12.147047+0000) 2024-03-06T11:52:13.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:13.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:13.147255+0000) 2024-03-06T11:52:14.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:14.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:14.147521+0000) 2024-03-06T11:52:15.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:15.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:15.147777+0000) 2024-03-06T11:52:16.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:16.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:16.147986+0000) 2024-03-06T11:52:17.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:17.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:17.148267+0000) 2024-03-06T11:52:18.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:18.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:18.148542+0000) 2024-03-06T11:52:19.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:19.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:19.148767+0000) 2024-03-06T11:52:20.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:20.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:20.148974+0000) 2024-03-06T11:52:21.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:21.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:21.149197+0000) 2024-03-06T11:52:22.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:22.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:22.149454+0000) 2024-03-06T11:52:23.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:23.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:23.149656+0000) 2024-03-06T11:52:24.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:24.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:24.149932+0000) 2024-03-06T11:52:25.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:25.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:25.150157+0000) 2024-03-06T11:52:26.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:26.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:26.150363+0000) 2024-03-06T11:52:27.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:27.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:27.150570+0000) 2024-03-06T11:52:28.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:28.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:28.150811+0000) 2024-03-06T11:52:28.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:28.331 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:29.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:29.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:29.151015+0000) 2024-03-06T11:52:30.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:30.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:30.151248+0000) 2024-03-06T11:52:31.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:31.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:31.151467+0000) 2024-03-06T11:52:32.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:32.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:32.151772+0000) 2024-03-06T11:52:33.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:33.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:33.152028+0000) 2024-03-06T11:52:34.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:34.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:34.152269+0000) 2024-03-06T11:52:35.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:35.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:35.152510+0000) 2024-03-06T11:52:36.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:36.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:36.152778+0000) 2024-03-06T11:52:37.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:37.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:37.153003+0000) 2024-03-06T11:52:38.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:38.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:38.153238+0000) 2024-03-06T11:52:39.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:39.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:39.153478+0000) 2024-03-06T11:52:40.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:40.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:40.153721+0000) 2024-03-06T11:52:41.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:41.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:41.153911+0000) 2024-03-06T11:52:42.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:42.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:42.154142+0000) 2024-03-06T11:52:43.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:43.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:43.154334+0000) 2024-03-06T11:52:44.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:44.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:44.154580+0000) 2024-03-06T11:52:45.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:45.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:45.154812+0000) 2024-03-06T11:52:46.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:46.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:46.154974+0000) 2024-03-06T11:52:47.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:47.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:47.155208+0000) 2024-03-06T11:52:48.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:48.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:48.155435+0000) 2024-03-06T11:52:49.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:49.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:49.155657+0000) 2024-03-06T11:52:50.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:50.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:50.155914+0000) 2024-03-06T11:52:51.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:51.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:51.156143+0000) 2024-03-06T11:52:52.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:52.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:52.156375+0000) 2024-03-06T11:52:53.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:53.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:53.156607+0000) 2024-03-06T11:52:54.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:54.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:54.156838+0000) 2024-03-06T11:52:55.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:55.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:55.157038+0000) 2024-03-06T11:52:56.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:56.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:56.157261+0000) 2024-03-06T11:52:57.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:57.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:57.157448+0000) 2024-03-06T11:52:58.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:58.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:58.157726+0000) 2024-03-06T11:52:58.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:58.559 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:59.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:52:59.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:59.157929+0000) 2024-03-06T11:53:00.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:00.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:00.158163+0000) 2024-03-06T11:53:01.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:01.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:01.158382+0000) 2024-03-06T11:53:02.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:02.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:02.158601+0000) 2024-03-06T11:53:03.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:03.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:03.158830+0000) 2024-03-06T11:53:04.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:04.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:04.159084+0000) 2024-03-06T11:53:05.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:05.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:05.159358+0000) 2024-03-06T11:53:06.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:06.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:06.159638+0000) 2024-03-06T11:53:07.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:07.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:07.159869+0000) 2024-03-06T11:53:08.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:08.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:08.160069+0000) 2024-03-06T11:53:09.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:09.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:09.160300+0000) 2024-03-06T11:53:10.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:10.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:10.160521+0000) 2024-03-06T11:53:11.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:11.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:11.160783+0000) 2024-03-06T11:53:12.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:12.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:12.161004+0000) 2024-03-06T11:53:13.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:13.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:13.161233+0000) 2024-03-06T11:53:14.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:14.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:14.161429+0000) 2024-03-06T11:53:15.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:15.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:15.161715+0000) 2024-03-06T11:53:16.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:16.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:16.161922+0000) 2024-03-06T11:53:17.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:17.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:17.162152+0000) 2024-03-06T11:53:18.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:18.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:18.162367+0000) 2024-03-06T11:53:19.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:19.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:19.162642+0000) 2024-03-06T11:53:20.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:20.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:20.162956+0000) 2024-03-06T11:53:21.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:21.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:21.163197+0000) 2024-03-06T11:53:22.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:22.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:22.163423+0000) 2024-03-06T11:53:23.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:23.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:23.163615+0000) 2024-03-06T11:53:24.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:24.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:24.163813+0000) 2024-03-06T11:53:25.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:25.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:25.164024+0000) 2024-03-06T11:53:26.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:26.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:26.164296+0000) 2024-03-06T11:53:27.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:27.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:27.164512+0000) 2024-03-06T11:53:28.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:28.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:28.164788+0000) 2024-03-06T11:53:28.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:28.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:29.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:29.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:29.164989+0000) 2024-03-06T11:53:30.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:30.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:30.165218+0000) 2024-03-06T11:53:31.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:31.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:31.165442+0000) 2024-03-06T11:53:32.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:32.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:32.165714+0000) 2024-03-06T11:53:33.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:33.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:33.165918+0000) 2024-03-06T11:53:34.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:34.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:34.166146+0000) 2024-03-06T11:53:35.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:35.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:35.166370+0000) 2024-03-06T11:53:36.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:36.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:36.166595+0000) 2024-03-06T11:53:37.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:37.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:37.166874+0000) 2024-03-06T11:53:38.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:38.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:38.167159+0000) 2024-03-06T11:53:39.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:39.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:39.167370+0000) 2024-03-06T11:53:40.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:40.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:40.167536+0000) 2024-03-06T11:53:41.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:41.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:41.167797+0000) 2024-03-06T11:53:42.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:42.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:42.167994+0000) 2024-03-06T11:53:43.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:43.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:43.168242+0000) 2024-03-06T11:53:44.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:44.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:44.168482+0000) 2024-03-06T11:53:45.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:45.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:45.168700+0000) 2024-03-06T11:53:46.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:46.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:46.168897+0000) 2024-03-06T11:53:47.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:47.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:47.169145+0000) 2024-03-06T11:53:48.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:48.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:48.169381+0000) 2024-03-06T11:53:49.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:49.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:49.169603+0000) 2024-03-06T11:53:50.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:50.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:50.169832+0000) 2024-03-06T11:53:51.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:51.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:51.170032+0000) 2024-03-06T11:53:52.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:52.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:52.170231+0000) 2024-03-06T11:53:53.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:53.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:53.170475+0000) 2024-03-06T11:53:54.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:54.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:54.170756+0000) 2024-03-06T11:53:55.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:55.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:55.170964+0000) 2024-03-06T11:53:56.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:56.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:56.171161+0000) 2024-03-06T11:53:57.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:57.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:57.171396+0000) 2024-03-06T11:53:58.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:58.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:58.171649+0000) 2024-03-06T11:53:58.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:58.987 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:59.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:53:59.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:59.171893+0000) 2024-03-06T11:54:00.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:00.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:00.172133+0000) 2024-03-06T11:54:01.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:01.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:01.172317+0000) 2024-03-06T11:54:02.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:02.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:02.172544+0000) 2024-03-06T11:54:03.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:03.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:03.172796+0000) 2024-03-06T11:54:04.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:04.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:04.173026+0000) 2024-03-06T11:54:05.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:05.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:05.173260+0000) 2024-03-06T11:54:06.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:06.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:06.173476+0000) 2024-03-06T11:54:07.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:07.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:07.173775+0000) 2024-03-06T11:54:08.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:08.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:08.173993+0000) 2024-03-06T11:54:09.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:09.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:09.174218+0000) 2024-03-06T11:54:10.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:10.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:10.174442+0000) 2024-03-06T11:54:11.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:11.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:11.174712+0000) 2024-03-06T11:54:12.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:12.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:12.174930+0000) 2024-03-06T11:54:13.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:13.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:13.175088+0000) 2024-03-06T11:54:14.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:14.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:14.175304+0000) 2024-03-06T11:54:15.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:15.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:15.175530+0000) 2024-03-06T11:54:16.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:16.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:16.175772+0000) 2024-03-06T11:54:17.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:17.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:17.176064+0000) 2024-03-06T11:54:18.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:18.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:18.176347+0000) 2024-03-06T11:54:19.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:19.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:19.176704+0000) 2024-03-06T11:54:20.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:20.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:20.176913+0000) 2024-03-06T11:54:21.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:21.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:21.177077+0000) 2024-03-06T11:54:22.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:22.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:22.177251+0000) 2024-03-06T11:54:23.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:23.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:23.177498+0000) 2024-03-06T11:54:24.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:24.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:24.177792+0000) 2024-03-06T11:54:25.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:25.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:25.178004+0000) 2024-03-06T11:54:26.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:26.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:26.178245+0000) 2024-03-06T11:54:27.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:27.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:27.178494+0000) 2024-03-06T11:54:28.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:28.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:28.178783+0000) 2024-03-06T11:54:29.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:29.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:29.179002+0000) 2024-03-06T11:54:29.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:29.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:30.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:30.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:30.179235+0000) 2024-03-06T11:54:31.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:31.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:31.179514+0000) 2024-03-06T11:54:32.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:32.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:32.179811+0000) 2024-03-06T11:54:33.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:33.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:33.180060+0000) 2024-03-06T11:54:34.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:34.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:34.180263+0000) 2024-03-06T11:54:35.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:35.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:35.180455+0000) 2024-03-06T11:54:36.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:36.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:36.180742+0000) 2024-03-06T11:54:37.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:37.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:37.180978+0000) 2024-03-06T11:54:38.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:38.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:38.181213+0000) 2024-03-06T11:54:39.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:39.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:39.181455+0000) 2024-03-06T11:54:40.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:40.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:40.181654+0000) 2024-03-06T11:54:41.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:41.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:41.181921+0000) 2024-03-06T11:54:42.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:42.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:42.182147+0000) 2024-03-06T11:54:43.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:43.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:43.182383+0000) 2024-03-06T11:54:44.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:44.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:44.182574+0000) 2024-03-06T11:54:45.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:45.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:45.182815+0000) 2024-03-06T11:54:46.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:46.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:46.183052+0000) 2024-03-06T11:54:47.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:47.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:47.183258+0000) 2024-03-06T11:54:48.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:48.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:48.183505+0000) 2024-03-06T11:54:49.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:49.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:49.183779+0000) 2024-03-06T11:54:50.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:50.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:50.183983+0000) 2024-03-06T11:54:51.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:51.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:51.184224+0000) 2024-03-06T11:54:52.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:52.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:52.184461+0000) 2024-03-06T11:54:53.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:53.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:53.184695+0000) 2024-03-06T11:54:54.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:54.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:54.184852+0000) 2024-03-06T11:54:55.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:55.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:55.185056+0000) 2024-03-06T11:54:56.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:56.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:56.185300+0000) 2024-03-06T11:54:57.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:57.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:57.185581+0000) 2024-03-06T11:54:58.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:58.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:58.185857+0000) 2024-03-06T11:54:59.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:54:59.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:59.186137+0000) 2024-03-06T11:54:59.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:59.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:00.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:00.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:00.186417+0000) 2024-03-06T11:55:01.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:01.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:01.186616+0000) 2024-03-06T11:55:02.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:02.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:02.186843+0000) 2024-03-06T11:55:03.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:03.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:03.187039+0000) 2024-03-06T11:55:04.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:04.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:04.187239+0000) 2024-03-06T11:55:05.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:05.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:05.187432+0000) 2024-03-06T11:55:06.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:06.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:06.187633+0000) 2024-03-06T11:55:07.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:07.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:07.187935+0000) 2024-03-06T11:55:08.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:08.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:08.188217+0000) 2024-03-06T11:55:09.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:09.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:09.188463+0000) 2024-03-06T11:55:10.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:10.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:10.188627+0000) 2024-03-06T11:55:11.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:11.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:11.188892+0000) 2024-03-06T11:55:12.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:12.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:12.189158+0000) 2024-03-06T11:55:13.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:13.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:13.189432+0000) 2024-03-06T11:55:14.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:14.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:14.189739+0000) 2024-03-06T11:55:15.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:15.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:15.189951+0000) 2024-03-06T11:55:16.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:16.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:16.190148+0000) 2024-03-06T11:55:17.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:17.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:17.190375+0000) 2024-03-06T11:55:18.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:18.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:18.190614+0000) 2024-03-06T11:55:19.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:19.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:19.190821+0000) 2024-03-06T11:55:20.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:20.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:20.191006+0000) 2024-03-06T11:55:21.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:21.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:21.191193+0000) 2024-03-06T11:55:22.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:22.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:22.191425+0000) 2024-03-06T11:55:23.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:23.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:23.191641+0000) 2024-03-06T11:55:24.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:24.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:24.191953+0000) 2024-03-06T11:55:25.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:25.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:25.192186+0000) 2024-03-06T11:55:26.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:26.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:26.192462+0000) 2024-03-06T11:55:27.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:27.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:27.192783+0000) 2024-03-06T11:55:28.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:28.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:28.192964+0000) 2024-03-06T11:55:29.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:29.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:29.193151+0000) 2024-03-06T11:55:29.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:29.632 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:30.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:30.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:30.193375+0000) 2024-03-06T11:55:31.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:31.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:31.193639+0000) 2024-03-06T11:55:32.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:32.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:32.193957+0000) 2024-03-06T11:55:33.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:33.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:33.194237+0000) 2024-03-06T11:55:34.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:34.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:34.194506+0000) 2024-03-06T11:55:35.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:35.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:35.194756+0000) 2024-03-06T11:55:36.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:36.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:36.194963+0000) 2024-03-06T11:55:37.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:37.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:37.195184+0000) 2024-03-06T11:55:38.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:38.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:38.195426+0000) 2024-03-06T11:55:39.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:39.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:39.195650+0000) 2024-03-06T11:55:40.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:40.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:40.195867+0000) 2024-03-06T11:55:41.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:41.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:41.196069+0000) 2024-03-06T11:55:42.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:42.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:42.196306+0000) 2024-03-06T11:55:43.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:43.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:43.196531+0000) 2024-03-06T11:55:44.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:44.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:44.196795+0000) 2024-03-06T11:55:45.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:45.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:45.197023+0000) 2024-03-06T11:55:46.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:46.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:46.197214+0000) 2024-03-06T11:55:47.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:47.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:47.197432+0000) 2024-03-06T11:55:48.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:48.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:48.197752+0000) 2024-03-06T11:55:49.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:49.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:49.197950+0000) 2024-03-06T11:55:50.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:50.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:50.198135+0000) 2024-03-06T11:55:51.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:51.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:51.198380+0000) 2024-03-06T11:55:52.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:52.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:52.198657+0000) 2024-03-06T11:55:53.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:53.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:53.198921+0000) 2024-03-06T11:55:54.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:54.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:54.199150+0000) 2024-03-06T11:55:55.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:55.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:55.199431+0000) 2024-03-06T11:55:56.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:56.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:56.199660+0000) 2024-03-06T11:55:57.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:57.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:57.199899+0000) 2024-03-06T11:55:58.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:58.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:58.200152+0000) 2024-03-06T11:55:59.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:55:59.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:59.200346+0000) 2024-03-06T11:55:59.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:59.854 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:00.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:00.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:00.200559+0000) 2024-03-06T11:56:01.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:01.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:01.200826+0000) 2024-03-06T11:56:02.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:02.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:02.201025+0000) 2024-03-06T11:56:03.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:03.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:03.201277+0000) 2024-03-06T11:56:04.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:04.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:04.201488+0000) 2024-03-06T11:56:05.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:05.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:05.201701+0000) 2024-03-06T11:56:06.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:06.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:06.201943+0000) 2024-03-06T11:56:07.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:07.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:07.202120+0000) 2024-03-06T11:56:08.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:08.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:08.202300+0000) 2024-03-06T11:56:09.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:09.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:09.202505+0000) 2024-03-06T11:56:10.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:10.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:10.202751+0000) 2024-03-06T11:56:11.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:11.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:11.202968+0000) 2024-03-06T11:56:12.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:12.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:12.203199+0000) 2024-03-06T11:56:13.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:13.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:13.203423+0000) 2024-03-06T11:56:14.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:14.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:14.203651+0000) 2024-03-06T11:56:15.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:15.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:15.203834+0000) 2024-03-06T11:56:16.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:16.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:16.204064+0000) 2024-03-06T11:56:17.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:17.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:17.204311+0000) 2024-03-06T11:56:18.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:18.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:18.204563+0000) 2024-03-06T11:56:19.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:19.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:19.204768+0000) 2024-03-06T11:56:20.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:20.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:20.205001+0000) 2024-03-06T11:56:21.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:21.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:21.205247+0000) 2024-03-06T11:56:22.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:22.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:22.205506+0000) 2024-03-06T11:56:23.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:23.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:23.205756+0000) 2024-03-06T11:56:24.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:24.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:24.205989+0000) 2024-03-06T11:56:25.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:25.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:25.206219+0000) 2024-03-06T11:56:26.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:26.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:26.206415+0000) 2024-03-06T11:56:27.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:27.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:27.206651+0000) 2024-03-06T11:56:28.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:28.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:28.206900+0000) 2024-03-06T11:56:29.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:29.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:29.207143+0000) 2024-03-06T11:56:30.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:30.073 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:30.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:30.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:30.207312+0000) 2024-03-06T11:56:31.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:31.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:31.207491+0000) 2024-03-06T11:56:32.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:32.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:32.207754+0000) 2024-03-06T11:56:33.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:33.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:33.207985+0000) 2024-03-06T11:56:34.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:34.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:34.208215+0000) 2024-03-06T11:56:35.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:35.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:35.208446+0000) 2024-03-06T11:56:36.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:36.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:36.208712+0000) 2024-03-06T11:56:37.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:37.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:37.208963+0000) 2024-03-06T11:56:38.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:38.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:38.209123+0000) 2024-03-06T11:56:39.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:39.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:39.209333+0000) 2024-03-06T11:56:40.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:40.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:40.209553+0000) 2024-03-06T11:56:41.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:41.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:41.209838+0000) 2024-03-06T11:56:42.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:42.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:42.210090+0000) 2024-03-06T11:56:43.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:43.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:43.210362+0000) 2024-03-06T11:56:44.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:44.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:44.210644+0000) 2024-03-06T11:56:45.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:45.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:45.210954+0000) 2024-03-06T11:56:46.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:46.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:46.211232+0000) 2024-03-06T11:56:47.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:47.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:47.211507+0000) 2024-03-06T11:56:48.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:48.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:48.211813+0000) 2024-03-06T11:56:49.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:49.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:49.212000+0000) 2024-03-06T11:56:50.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:50.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:50.212238+0000) 2024-03-06T11:56:51.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:51.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:51.212452+0000) 2024-03-06T11:56:52.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:52.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:52.212706+0000) 2024-03-06T11:56:53.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:53.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:53.212942+0000) 2024-03-06T11:56:54.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:54.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:54.213237+0000) 2024-03-06T11:56:55.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:55.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:55.213467+0000) 2024-03-06T11:56:56.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:56.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:56.213724+0000) 2024-03-06T11:56:57.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:57.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:57.213915+0000) 2024-03-06T11:56:58.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:58.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:58.214153+0000) 2024-03-06T11:56:59.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:56:59.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:59.214384+0000) 2024-03-06T11:57:00.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:00.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:00.214607+0000) 2024-03-06T11:57:00.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:00.294 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:01.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:01.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:01.214831+0000) 2024-03-06T11:57:02.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:02.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:02.215026+0000) 2024-03-06T11:57:03.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:03.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:03.215282+0000) 2024-03-06T11:57:04.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:04.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:04.215494+0000) 2024-03-06T11:57:05.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:05.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:05.215709+0000) 2024-03-06T11:57:06.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:06.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:06.215940+0000) 2024-03-06T11:57:07.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:07.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:07.216170+0000) 2024-03-06T11:57:08.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:08.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:08.216395+0000) 2024-03-06T11:57:09.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:09.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:09.216616+0000) 2024-03-06T11:57:10.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:10.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:10.216845+0000) 2024-03-06T11:57:11.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:11.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:11.217068+0000) 2024-03-06T11:57:12.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:12.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:12.217324+0000) 2024-03-06T11:57:13.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:13.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:13.217554+0000) 2024-03-06T11:57:14.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:14.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:14.217793+0000) 2024-03-06T11:57:15.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:15.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:15.217996+0000) 2024-03-06T11:57:16.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:16.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:16.218200+0000) 2024-03-06T11:57:17.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:17.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:17.218430+0000) 2024-03-06T11:57:18.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:18.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:18.218659+0000) 2024-03-06T11:57:19.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:19.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:19.218887+0000) 2024-03-06T11:57:20.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:20.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:20.219112+0000) 2024-03-06T11:57:21.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:21.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:21.219301+0000) 2024-03-06T11:57:22.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:22.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:22.219537+0000) 2024-03-06T11:57:23.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:23.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:23.219749+0000) 2024-03-06T11:57:24.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:24.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:24.219987+0000) 2024-03-06T11:57:25.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:25.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:25.220214+0000) 2024-03-06T11:57:26.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:26.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:26.220497+0000) 2024-03-06T11:57:27.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:27.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:27.220776+0000) 2024-03-06T11:57:28.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:28.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:28.221016+0000) 2024-03-06T11:57:29.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:29.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:29.221243+0000) 2024-03-06T11:57:30.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:30.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:30.221469+0000) 2024-03-06T11:57:30.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:30.499 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:31.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:31.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:31.221731+0000) 2024-03-06T11:57:32.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:32.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:32.221993+0000) 2024-03-06T11:57:33.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:33.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:33.222255+0000) 2024-03-06T11:57:34.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:34.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:34.222535+0000) 2024-03-06T11:57:35.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:35.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:35.222771+0000) 2024-03-06T11:57:36.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:36.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:36.222923+0000) 2024-03-06T11:57:37.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:37.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:37.223119+0000) 2024-03-06T11:57:38.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:38.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:38.223354+0000) 2024-03-06T11:57:39.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:39.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:39.223583+0000) 2024-03-06T11:57:40.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:40.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:40.223815+0000) 2024-03-06T11:57:41.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:41.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:41.224007+0000) 2024-03-06T11:57:42.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:42.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:42.224299+0000) 2024-03-06T11:57:43.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:43.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:43.224522+0000) 2024-03-06T11:57:44.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:44.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:44.224772+0000) 2024-03-06T11:57:45.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:45.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:45.224996+0000) 2024-03-06T11:57:46.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:46.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:46.225219+0000) 2024-03-06T11:57:47.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:47.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:47.225403+0000) 2024-03-06T11:57:48.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:48.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:48.225630+0000) 2024-03-06T11:57:49.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:49.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:49.225818+0000) 2024-03-06T11:57:50.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:50.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:50.226036+0000) 2024-03-06T11:57:51.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:51.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:51.226254+0000) 2024-03-06T11:57:52.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:52.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:52.226455+0000) 2024-03-06T11:57:53.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:53.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:53.226695+0000) 2024-03-06T11:57:54.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:54.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:54.226933+0000) 2024-03-06T11:57:55.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:55.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:55.227111+0000) 2024-03-06T11:57:56.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:56.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:56.227350+0000) 2024-03-06T11:57:57.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:57.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:57.227586+0000) 2024-03-06T11:57:58.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:58.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:58.227826+0000) 2024-03-06T11:57:59.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:57:59.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:59.228045+0000) 2024-03-06T11:58:00.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:00.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:00.228305+0000) 2024-03-06T11:58:00.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:00.703 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:01.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:01.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:01.228566+0000) 2024-03-06T11:58:02.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:02.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:02.228845+0000) 2024-03-06T11:58:03.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:03.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:03.229111+0000) 2024-03-06T11:58:04.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:04.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:04.229313+0000) 2024-03-06T11:58:05.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:05.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:05.229583+0000) 2024-03-06T11:58:06.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:06.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:06.229795+0000) 2024-03-06T11:58:07.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:07.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:07.230064+0000) 2024-03-06T11:58:08.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:08.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:08.230343+0000) 2024-03-06T11:58:09.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:09.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:09.230543+0000) 2024-03-06T11:58:10.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:10.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:10.230779+0000) 2024-03-06T11:58:11.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:11.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:11.230943+0000) 2024-03-06T11:58:12.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:12.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:12.231125+0000) 2024-03-06T11:58:13.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:13.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:13.231318+0000) 2024-03-06T11:58:14.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:14.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:14.231558+0000) 2024-03-06T11:58:15.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:15.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:15.231777+0000) 2024-03-06T11:58:16.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:16.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:16.232005+0000) 2024-03-06T11:58:17.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:17.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:17.232229+0000) 2024-03-06T11:58:18.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:18.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:18.232408+0000) 2024-03-06T11:58:19.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:19.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:19.232605+0000) 2024-03-06T11:58:20.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:20.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:20.232831+0000) 2024-03-06T11:58:21.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:21.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:21.233018+0000) 2024-03-06T11:58:22.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:22.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:22.233238+0000) 2024-03-06T11:58:23.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:23.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:23.233459+0000) 2024-03-06T11:58:24.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:24.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:24.233725+0000) 2024-03-06T11:58:25.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:25.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:25.233947+0000) 2024-03-06T11:58:26.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:26.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:26.234175+0000) 2024-03-06T11:58:27.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:27.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:27.234404+0000) 2024-03-06T11:58:28.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:28.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:28.234655+0000) 2024-03-06T11:58:29.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:29.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:29.234915+0000) 2024-03-06T11:58:30.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:30.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:30.235137+0000) 2024-03-06T11:58:30.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:30.924 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:31.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:31.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:31.235371+0000) 2024-03-06T11:58:32.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:32.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:32.235607+0000) 2024-03-06T11:58:33.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:33.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:33.235794+0000) 2024-03-06T11:58:34.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:34.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:34.236031+0000) 2024-03-06T11:58:35.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:35.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:35.236304+0000) 2024-03-06T11:58:36.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:36.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:36.236532+0000) 2024-03-06T11:58:37.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:37.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:37.236787+0000) 2024-03-06T11:58:38.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:38.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:38.237011+0000) 2024-03-06T11:58:39.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:39.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:39.237194+0000) 2024-03-06T11:58:40.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:40.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:40.237422+0000) 2024-03-06T11:58:41.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:41.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:41.237619+0000) 2024-03-06T11:58:42.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:42.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:42.237844+0000) 2024-03-06T11:58:43.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:43.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:43.238044+0000) 2024-03-06T11:58:44.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:44.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:44.238248+0000) 2024-03-06T11:58:45.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:45.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:45.238473+0000) 2024-03-06T11:58:46.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:46.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:46.238662+0000) 2024-03-06T11:58:47.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:47.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:47.238962+0000) 2024-03-06T11:58:48.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:48.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:48.239234+0000) 2024-03-06T11:58:49.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:49.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:49.239512+0000) 2024-03-06T11:58:50.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:50.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:50.239814+0000) 2024-03-06T11:58:51.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:51.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:51.240057+0000) 2024-03-06T11:58:52.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:52.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:52.240294+0000) 2024-03-06T11:58:53.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:53.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:53.240515+0000) 2024-03-06T11:58:54.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:54.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:54.240738+0000) 2024-03-06T11:58:55.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:55.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:55.240981+0000) 2024-03-06T11:58:56.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:56.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:56.241256+0000) 2024-03-06T11:58:57.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:57.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:57.241530+0000) 2024-03-06T11:58:58.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:58.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:58.241807+0000) 2024-03-06T11:58:59.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:58:59.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:59.242028+0000) 2024-03-06T11:59:00.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:00.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:00.242280+0000) 2024-03-06T11:59:01.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:01.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:01.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:01.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:01.242486+0000) 2024-03-06T11:59:02.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:02.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:02.242794+0000) 2024-03-06T11:59:03.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:03.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:03.243005+0000) 2024-03-06T11:59:04.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:04.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:04.243257+0000) 2024-03-06T11:59:05.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:05.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:05.243491+0000) 2024-03-06T11:59:06.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:06.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:06.243757+0000) 2024-03-06T11:59:07.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:07.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:07.243997+0000) 2024-03-06T11:59:08.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:08.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:08.244271+0000) 2024-03-06T11:59:09.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:09.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:09.244468+0000) 2024-03-06T11:59:10.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:10.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:10.244730+0000) 2024-03-06T11:59:11.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:11.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:11.244949+0000) 2024-03-06T11:59:12.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:12.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:12.245171+0000) 2024-03-06T11:59:13.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:13.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:13.245397+0000) 2024-03-06T11:59:14.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:14.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:14.245632+0000) 2024-03-06T11:59:15.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:15.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:15.245891+0000) 2024-03-06T11:59:16.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:16.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:16.246096+0000) 2024-03-06T11:59:17.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:17.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:17.246371+0000) 2024-03-06T11:59:18.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:18.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:18.246630+0000) 2024-03-06T11:59:19.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:19.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:19.246966+0000) 2024-03-06T11:59:20.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:20.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:20.247248+0000) 2024-03-06T11:59:21.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:21.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:21.247443+0000) 2024-03-06T11:59:22.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:22.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:22.247703+0000) 2024-03-06T11:59:23.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:23.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:23.248013+0000) 2024-03-06T11:59:24.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:24.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:24.248236+0000) 2024-03-06T11:59:25.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:25.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:25.248466+0000) 2024-03-06T11:59:26.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:26.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:26.248761+0000) 2024-03-06T11:59:27.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:27.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:27.249048+0000) 2024-03-06T11:59:28.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:28.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:28.249269+0000) 2024-03-06T11:59:29.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:29.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:29.249524+0000) 2024-03-06T11:59:30.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:30.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:30.249832+0000) 2024-03-06T11:59:31.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:31.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:31.250107+0000) 2024-03-06T11:59:31.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:31.366 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:32.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:32.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:32.250383+0000) 2024-03-06T11:59:33.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:33.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:33.250658+0000) 2024-03-06T11:59:34.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:34.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:34.250997+0000) 2024-03-06T11:59:35.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:35.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:35.251232+0000) 2024-03-06T11:59:36.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:36.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:36.251505+0000) 2024-03-06T11:59:37.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:37.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:37.251813+0000) 2024-03-06T11:59:38.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:38.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:38.252011+0000) 2024-03-06T11:59:39.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:39.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:39.252257+0000) 2024-03-06T11:59:40.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:40.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:40.252483+0000) 2024-03-06T11:59:41.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:41.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:41.252702+0000) 2024-03-06T11:59:42.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:42.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:42.252905+0000) 2024-03-06T11:59:43.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:43.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:43.253147+0000) 2024-03-06T11:59:44.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:44.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:44.253381+0000) 2024-03-06T11:59:45.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:45.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:45.253597+0000) 2024-03-06T11:59:46.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:46.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:46.253832+0000) 2024-03-06T11:59:47.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:47.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:47.254059+0000) 2024-03-06T11:59:48.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:48.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:48.254298+0000) 2024-03-06T11:59:49.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:49.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:49.254524+0000) 2024-03-06T11:59:50.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:50.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:50.254826+0000) 2024-03-06T11:59:51.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:51.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:51.255016+0000) 2024-03-06T11:59:52.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:52.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:52.255265+0000) 2024-03-06T11:59:53.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:53.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:53.255546+0000) 2024-03-06T11:59:54.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:54.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:54.255822+0000) 2024-03-06T11:59:55.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:55.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:55.256115+0000) 2024-03-06T11:59:56.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:56.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:56.256392+0000) 2024-03-06T11:59:57.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:57.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:57.256657+0000) 2024-03-06T11:59:58.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:58.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:58.256899+0000) 2024-03-06T11:59:59.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T11:59:59.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:59.257124+0000) 2024-03-06T12:00:00.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:00.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:00.257354+0000) 2024-03-06T12:00:01.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:01.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:01.257576+0000) 2024-03-06T12:00:01.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:01.595 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:02.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:02.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:02.257839+0000) 2024-03-06T12:00:03.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:03.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:03.258096+0000) 2024-03-06T12:00:04.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:04.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:04.258382+0000) 2024-03-06T12:00:05.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:05.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:05.258618+0000) 2024-03-06T12:00:06.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:06.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:06.258844+0000) 2024-03-06T12:00:07.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:07.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:07.259073+0000) 2024-03-06T12:00:08.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:08.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:08.259337+0000) 2024-03-06T12:00:09.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:09.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:09.259606+0000) 2024-03-06T12:00:10.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:10.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:10.259808+0000) 2024-03-06T12:00:11.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:11.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:11.260042+0000) 2024-03-06T12:00:12.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:12.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:12.260272+0000) 2024-03-06T12:00:13.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:13.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:13.260492+0000) 2024-03-06T12:00:14.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:14.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:14.260799+0000) 2024-03-06T12:00:15.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:15.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:15.261044+0000) 2024-03-06T12:00:16.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:16.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:16.261278+0000) 2024-03-06T12:00:17.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:17.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:17.261565+0000) 2024-03-06T12:00:18.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:18.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:18.261768+0000) 2024-03-06T12:00:19.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:19.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:19.262009+0000) 2024-03-06T12:00:20.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:20.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:20.262271+0000) 2024-03-06T12:00:21.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:21.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:21.262533+0000) 2024-03-06T12:00:22.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:22.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:22.262863+0000) 2024-03-06T12:00:23.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:23.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:23.263137+0000) 2024-03-06T12:00:24.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:24.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:24.263375+0000) 2024-03-06T12:00:25.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:25.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:25.263615+0000) 2024-03-06T12:00:26.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:26.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:26.263838+0000) 2024-03-06T12:00:27.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:27.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:27.264055+0000) 2024-03-06T12:00:28.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:28.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:28.264336+0000) 2024-03-06T12:00:29.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:29.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:29.264626+0000) 2024-03-06T12:00:30.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:30.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:30.264931+0000) 2024-03-06T12:00:31.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:31.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:31.265168+0000) 2024-03-06T12:00:31.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:31.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:32.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:32.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:32.265387+0000) 2024-03-06T12:00:33.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:33.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:33.265637+0000) 2024-03-06T12:00:34.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:34.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:34.265868+0000) 2024-03-06T12:00:35.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:35.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:35.266095+0000) 2024-03-06T12:00:36.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:36.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:36.266365+0000) 2024-03-06T12:00:37.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:37.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:37.266640+0000) 2024-03-06T12:00:38.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:38.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:38.266962+0000) 2024-03-06T12:00:39.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:39.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:39.267195+0000) 2024-03-06T12:00:40.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:40.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:40.267478+0000) 2024-03-06T12:00:41.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:41.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:41.267699+0000) 2024-03-06T12:00:42.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:42.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:42.267933+0000) 2024-03-06T12:00:43.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:43.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:43.268159+0000) 2024-03-06T12:00:44.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:44.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:44.268379+0000) 2024-03-06T12:00:45.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:45.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:45.268586+0000) 2024-03-06T12:00:46.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:46.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:46.268854+0000) 2024-03-06T12:00:47.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:47.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:47.269126+0000) 2024-03-06T12:00:48.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:48.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:48.269407+0000) 2024-03-06T12:00:49.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:49.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:49.269707+0000) 2024-03-06T12:00:50.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:50.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:50.269941+0000) 2024-03-06T12:00:51.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:51.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:51.270160+0000) 2024-03-06T12:00:52.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:52.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:52.270356+0000) 2024-03-06T12:00:53.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:53.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:53.270586+0000) 2024-03-06T12:00:54.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:54.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:54.270819+0000) 2024-03-06T12:00:55.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:55.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:55.271042+0000) 2024-03-06T12:00:56.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:56.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:56.271272+0000) 2024-03-06T12:00:57.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:57.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:57.271535+0000) 2024-03-06T12:00:58.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:58.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:58.271814+0000) 2024-03-06T12:00:59.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:00:59.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:59.272130+0000) 2024-03-06T12:01:00.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:00.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:00.272365+0000) 2024-03-06T12:01:01.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:01.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:01.272645+0000) 2024-03-06T12:01:02.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:02.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:02.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:02.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:02.272903+0000) 2024-03-06T12:01:03.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:03.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:03.273140+0000) 2024-03-06T12:01:04.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:04.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:04.273368+0000) 2024-03-06T12:01:05.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:05.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:05.273633+0000) 2024-03-06T12:01:06.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:06.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:06.273892+0000) 2024-03-06T12:01:07.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:07.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:07.274178+0000) 2024-03-06T12:01:08.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:08.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:08.274375+0000) 2024-03-06T12:01:09.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:09.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:09.274568+0000) 2024-03-06T12:01:10.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:10.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:10.274871+0000) 2024-03-06T12:01:11.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:11.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:11.275144+0000) 2024-03-06T12:01:12.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:12.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:12.275409+0000) 2024-03-06T12:01:13.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:13.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:13.275649+0000) 2024-03-06T12:01:14.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:14.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:14.275956+0000) 2024-03-06T12:01:15.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:15.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:15.276223+0000) 2024-03-06T12:01:16.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:16.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:16.276420+0000) 2024-03-06T12:01:17.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:17.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:17.276635+0000) 2024-03-06T12:01:18.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:18.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:18.276882+0000) 2024-03-06T12:01:19.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:19.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:19.277102+0000) 2024-03-06T12:01:20.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:20.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:20.277326+0000) 2024-03-06T12:01:21.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:21.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:21.277554+0000) 2024-03-06T12:01:22.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:22.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:22.277797+0000) 2024-03-06T12:01:23.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:23.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:23.278077+0000) 2024-03-06T12:01:24.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:24.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:24.278300+0000) 2024-03-06T12:01:25.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:25.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:25.278507+0000) 2024-03-06T12:01:26.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:26.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:26.278785+0000) 2024-03-06T12:01:27.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:27.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:27.279005+0000) 2024-03-06T12:01:28.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:28.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:28.279196+0000) 2024-03-06T12:01:29.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:29.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:29.279456+0000) 2024-03-06T12:01:30.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:30.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:30.279780+0000) 2024-03-06T12:01:31.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:31.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:31.280063+0000) 2024-03-06T12:01:32.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:32.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:32.280261+0000) 2024-03-06T12:01:32.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:32.288 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:33.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:33.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:33.280444+0000) 2024-03-06T12:01:34.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:34.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:34.280703+0000) 2024-03-06T12:01:35.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:35.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:35.280921+0000) 2024-03-06T12:01:36.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:36.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:36.281106+0000) 2024-03-06T12:01:37.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:37.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:37.281346+0000) 2024-03-06T12:01:38.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:38.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:38.281625+0000) 2024-03-06T12:01:39.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:39.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:39.281949+0000) 2024-03-06T12:01:40.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:40.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:40.282177+0000) 2024-03-06T12:01:41.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:41.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:41.282402+0000) 2024-03-06T12:01:42.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:42.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:42.282591+0000) 2024-03-06T12:01:43.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:43.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:43.282814+0000) 2024-03-06T12:01:44.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:44.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:44.283045+0000) 2024-03-06T12:01:45.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:45.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:45.283206+0000) 2024-03-06T12:01:46.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:46.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:46.283394+0000) 2024-03-06T12:01:47.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:47.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:47.283713+0000) 2024-03-06T12:01:48.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:48.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:48.283998+0000) 2024-03-06T12:01:49.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:49.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:49.284269+0000) 2024-03-06T12:01:50.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:50.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:50.284505+0000) 2024-03-06T12:01:51.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:51.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:51.284791+0000) 2024-03-06T12:01:52.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:52.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:52.285069+0000) 2024-03-06T12:01:53.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:53.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:53.285351+0000) 2024-03-06T12:01:54.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:54.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:54.285625+0000) 2024-03-06T12:01:55.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:55.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:55.285947+0000) 2024-03-06T12:01:56.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:56.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:56.286150+0000) 2024-03-06T12:01:57.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:57.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:57.286385+0000) 2024-03-06T12:01:58.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:58.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:58.286758+0000) 2024-03-06T12:01:59.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:01:59.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:59.287006+0000) 2024-03-06T12:02:00.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:00.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:00.287235+0000) 2024-03-06T12:02:01.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:01.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:01.287499+0000) 2024-03-06T12:02:02.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:02.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:02.287744+0000) 2024-03-06T12:02:02.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:02.430 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:03.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:03.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:03.287973+0000) 2024-03-06T12:02:04.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:04.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:04.288185+0000) 2024-03-06T12:02:05.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:05.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:05.288383+0000) 2024-03-06T12:02:06.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:06.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:06.288608+0000) 2024-03-06T12:02:07.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:07.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:07.288857+0000) 2024-03-06T12:02:08.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:08.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:08.289102+0000) 2024-03-06T12:02:09.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:09.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:09.289323+0000) 2024-03-06T12:02:10.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:10.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:10.289550+0000) 2024-03-06T12:02:11.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:11.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:11.289800+0000) 2024-03-06T12:02:12.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:12.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:12.290080+0000) 2024-03-06T12:02:13.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:13.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:13.290318+0000) 2024-03-06T12:02:14.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:14.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:14.290547+0000) 2024-03-06T12:02:15.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:15.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:15.290728+0000) 2024-03-06T12:02:16.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:16.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:16.290948+0000) 2024-03-06T12:02:17.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:17.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:17.291170+0000) 2024-03-06T12:02:18.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:18.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:18.291441+0000) 2024-03-06T12:02:19.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:19.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:19.291749+0000) 2024-03-06T12:02:20.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:20.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:20.292020+0000) 2024-03-06T12:02:21.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:21.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:21.292212+0000) 2024-03-06T12:02:22.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:22.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:22.292404+0000) 2024-03-06T12:02:23.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:23.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:23.292636+0000) 2024-03-06T12:02:24.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:24.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:24.292962+0000) 2024-03-06T12:02:25.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:25.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:25.293241+0000) 2024-03-06T12:02:26.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:26.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:26.293436+0000) 2024-03-06T12:02:27.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:27.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:27.293698+0000) 2024-03-06T12:02:28.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:28.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:28.293926+0000) 2024-03-06T12:02:29.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:29.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:29.294156+0000) 2024-03-06T12:02:30.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:30.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:30.294370+0000) 2024-03-06T12:02:31.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:31.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:31.294599+0000) 2024-03-06T12:02:32.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:32.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:32.294775+0000) 2024-03-06T12:02:32.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:32.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:33.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:33.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:33.294991+0000) 2024-03-06T12:02:34.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:34.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:34.295274+0000) 2024-03-06T12:02:35.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:35.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:35.295548+0000) 2024-03-06T12:02:36.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:36.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:36.295786+0000) 2024-03-06T12:02:37.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:37.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:37.296030+0000) 2024-03-06T12:02:38.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:38.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:38.296258+0000) 2024-03-06T12:02:39.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:39.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:39.296475+0000) 2024-03-06T12:02:40.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:40.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:40.296705+0000) 2024-03-06T12:02:41.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:41.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:41.296940+0000) 2024-03-06T12:02:42.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:42.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:42.297182+0000) 2024-03-06T12:02:43.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:43.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:43.297395+0000) 2024-03-06T12:02:44.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:44.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:44.297657+0000) 2024-03-06T12:02:45.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:45.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:45.297968+0000) 2024-03-06T12:02:46.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:46.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:46.298240+0000) 2024-03-06T12:02:47.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:47.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:47.298448+0000) 2024-03-06T12:02:48.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:48.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:48.298715+0000) 2024-03-06T12:02:49.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:49.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:49.298994+0000) 2024-03-06T12:02:50.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:50.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:50.299183+0000) 2024-03-06T12:02:51.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:51.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:51.299389+0000) 2024-03-06T12:02:52.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:52.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:52.299700+0000) 2024-03-06T12:02:53.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:53.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:53.299899+0000) 2024-03-06T12:02:54.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:54.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:54.300163+0000) 2024-03-06T12:02:55.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:55.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:55.300425+0000) 2024-03-06T12:02:56.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:56.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:56.300778+0000) 2024-03-06T12:02:57.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:57.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:57.301046+0000) 2024-03-06T12:02:58.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:58.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:58.301324+0000) 2024-03-06T12:02:59.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:02:59.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:59.301585+0000) 2024-03-06T12:03:00.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:00.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:00.301864+0000) 2024-03-06T12:03:01.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:01.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:01.302124+0000) 2024-03-06T12:03:02.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:02.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:02.302389+0000) 2024-03-06T12:03:02.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:02.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:03.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:03.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:03.302662+0000) 2024-03-06T12:03:04.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:04.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:04.303007+0000) 2024-03-06T12:03:05.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:05.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:05.303207+0000) 2024-03-06T12:03:06.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:06.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:06.303434+0000) 2024-03-06T12:03:07.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:07.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:07.303685+0000) 2024-03-06T12:03:08.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:08.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:08.303863+0000) 2024-03-06T12:03:09.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:09.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:09.304098+0000) 2024-03-06T12:03:10.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:10.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:10.304332+0000) 2024-03-06T12:03:11.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:11.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:11.304555+0000) 2024-03-06T12:03:12.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:12.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:12.304792+0000) 2024-03-06T12:03:13.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:13.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:13.305021+0000) 2024-03-06T12:03:14.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:14.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:14.305222+0000) 2024-03-06T12:03:15.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:15.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:15.305447+0000) 2024-03-06T12:03:16.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:16.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:16.305698+0000) 2024-03-06T12:03:17.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:17.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:17.305930+0000) 2024-03-06T12:03:18.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:18.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:18.306156+0000) 2024-03-06T12:03:19.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:19.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:19.306342+0000) 2024-03-06T12:03:20.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:20.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:20.306586+0000) 2024-03-06T12:03:21.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:21.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:21.306842+0000) 2024-03-06T12:03:22.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:22.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:22.307108+0000) 2024-03-06T12:03:23.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:23.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:23.307384+0000) 2024-03-06T12:03:24.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:24.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:24.307655+0000) 2024-03-06T12:03:25.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:25.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:25.308004+0000) 2024-03-06T12:03:26.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:26.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:26.308239+0000) 2024-03-06T12:03:27.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:27.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:27.308463+0000) 2024-03-06T12:03:28.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:28.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:28.308707+0000) 2024-03-06T12:03:29.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:29.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:29.308918+0000) 2024-03-06T12:03:30.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:30.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:30.309084+0000) 2024-03-06T12:03:31.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:31.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:31.309273+0000) 2024-03-06T12:03:32.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:32.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:32.309523+0000) 2024-03-06T12:03:33.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:33.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:33.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:33.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:33.309804+0000) 2024-03-06T12:03:34.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:34.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:34.310024+0000) 2024-03-06T12:03:35.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:35.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:35.310245+0000) 2024-03-06T12:03:36.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:36.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:36.310441+0000) 2024-03-06T12:03:37.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:37.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:37.310660+0000) 2024-03-06T12:03:38.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:38.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:38.310931+0000) 2024-03-06T12:03:39.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:39.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:39.311172+0000) 2024-03-06T12:03:40.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:40.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:40.311393+0000) 2024-03-06T12:03:41.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:41.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:41.311632+0000) 2024-03-06T12:03:42.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:42.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:42.311805+0000) 2024-03-06T12:03:43.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:43.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:43.312006+0000) 2024-03-06T12:03:44.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:44.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:44.312237+0000) 2024-03-06T12:03:45.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:45.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:45.312444+0000) 2024-03-06T12:03:46.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:46.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:46.312716+0000) 2024-03-06T12:03:47.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:47.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:47.312940+0000) 2024-03-06T12:03:48.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:48.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:48.313162+0000) 2024-03-06T12:03:49.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:49.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:49.313338+0000) 2024-03-06T12:03:50.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:50.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:50.313592+0000) 2024-03-06T12:03:51.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:51.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:51.313775+0000) 2024-03-06T12:03:52.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:52.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:52.313939+0000) 2024-03-06T12:03:53.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:53.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:53.314119+0000) 2024-03-06T12:03:54.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:54.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:54.314320+0000) 2024-03-06T12:03:55.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:55.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:55.314555+0000) 2024-03-06T12:03:56.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:56.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:56.314794+0000) 2024-03-06T12:03:57.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:57.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:57.315035+0000) 2024-03-06T12:03:58.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:58.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:58.315292+0000) 2024-03-06T12:03:59.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:03:59.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:59.315529+0000) 2024-03-06T12:04:00.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:00.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:00.315788+0000) 2024-03-06T12:04:01.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:01.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:01.315979+0000) 2024-03-06T12:04:02.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:02.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:02.316216+0000) 2024-03-06T12:04:03.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:03.272 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:03.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:03.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:03.316423+0000) 2024-03-06T12:04:04.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:04.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:04.316654+0000) 2024-03-06T12:04:05.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:05.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:05.316933+0000) 2024-03-06T12:04:06.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:06.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:06.317173+0000) 2024-03-06T12:04:07.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:07.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:07.317398+0000) 2024-03-06T12:04:08.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:08.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:08.317599+0000) 2024-03-06T12:04:09.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:09.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:09.317794+0000) 2024-03-06T12:04:10.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:10.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:10.318033+0000) 2024-03-06T12:04:11.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:11.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:11.318219+0000) 2024-03-06T12:04:12.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:12.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:12.318439+0000) 2024-03-06T12:04:13.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:13.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:13.318624+0000) 2024-03-06T12:04:14.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:14.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:14.318885+0000) 2024-03-06T12:04:15.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:15.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:15.319096+0000) 2024-03-06T12:04:16.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:16.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:16.319331+0000) 2024-03-06T12:04:17.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:17.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:17.319561+0000) 2024-03-06T12:04:18.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:18.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:18.319799+0000) 2024-03-06T12:04:19.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:19.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:19.320018+0000) 2024-03-06T12:04:20.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:20.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:20.320254+0000) 2024-03-06T12:04:21.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:21.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:21.320471+0000) 2024-03-06T12:04:22.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:22.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:22.320747+0000) 2024-03-06T12:04:23.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:23.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:23.320965+0000) 2024-03-06T12:04:24.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:24.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:24.321191+0000) 2024-03-06T12:04:25.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:25.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:25.321405+0000) 2024-03-06T12:04:26.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:26.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:26.321616+0000) 2024-03-06T12:04:27.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:27.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:27.321789+0000) 2024-03-06T12:04:28.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:28.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:28.322001+0000) 2024-03-06T12:04:29.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:29.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:29.322185+0000) 2024-03-06T12:04:30.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:30.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:30.322409+0000) 2024-03-06T12:04:31.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:31.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:31.322612+0000) 2024-03-06T12:04:32.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:32.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:32.322843+0000) 2024-03-06T12:04:33.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:33.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:33.323068+0000) 2024-03-06T12:04:33.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:33.409 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:34.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:34.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:34.323308+0000) 2024-03-06T12:04:35.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:35.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:35.323556+0000) 2024-03-06T12:04:36.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:36.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:36.323786+0000) 2024-03-06T12:04:37.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:37.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:37.324018+0000) 2024-03-06T12:04:38.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:38.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:38.324339+0000) 2024-03-06T12:04:39.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:39.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:39.324576+0000) 2024-03-06T12:04:40.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:40.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:40.324807+0000) 2024-03-06T12:04:41.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:41.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:41.325031+0000) 2024-03-06T12:04:42.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:42.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:42.325260+0000) 2024-03-06T12:04:43.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:43.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:43.325482+0000) 2024-03-06T12:04:44.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:44.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:44.325746+0000) 2024-03-06T12:04:45.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:45.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:45.325999+0000) 2024-03-06T12:04:46.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:46.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:46.326185+0000) 2024-03-06T12:04:47.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:47.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:47.326405+0000) 2024-03-06T12:04:48.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:48.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:48.326637+0000) 2024-03-06T12:04:49.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:49.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:49.326848+0000) 2024-03-06T12:04:50.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:50.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:50.327089+0000) 2024-03-06T12:04:51.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:51.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:51.327305+0000) 2024-03-06T12:04:52.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:52.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:52.327529+0000) 2024-03-06T12:04:53.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:53.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:53.327796+0000) 2024-03-06T12:04:54.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:54.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:54.327981+0000) 2024-03-06T12:04:55.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:55.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:55.328178+0000) 2024-03-06T12:04:56.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:56.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:56.328411+0000) 2024-03-06T12:04:57.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:57.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:57.328596+0000) 2024-03-06T12:04:58.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:58.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:58.328848+0000) 2024-03-06T12:04:59.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:04:59.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:59.329093+0000) 2024-03-06T12:05:00.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:00.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:00.329380+0000) 2024-03-06T12:05:01.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:01.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:01.329646+0000) 2024-03-06T12:05:02.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:02.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:02.329885+0000) 2024-03-06T12:05:03.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:03.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:03.330079+0000) 2024-03-06T12:05:03.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:03.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:04.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:04.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:04.330293+0000) 2024-03-06T12:05:05.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:05.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:05.330544+0000) 2024-03-06T12:05:06.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:06.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:06.330790+0000) 2024-03-06T12:05:07.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:07.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:07.331015+0000) 2024-03-06T12:05:08.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:08.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:08.331204+0000) 2024-03-06T12:05:09.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:09.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:09.331416+0000) 2024-03-06T12:05:10.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:10.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:10.331739+0000) 2024-03-06T12:05:11.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:11.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:11.331934+0000) 2024-03-06T12:05:12.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:12.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:12.332168+0000) 2024-03-06T12:05:13.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:13.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:13.332386+0000) 2024-03-06T12:05:14.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:14.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:14.332615+0000) 2024-03-06T12:05:15.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:15.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:15.332874+0000) 2024-03-06T12:05:16.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:16.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:16.333117+0000) 2024-03-06T12:05:17.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:17.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:17.333382+0000) 2024-03-06T12:05:18.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:18.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:18.333660+0000) 2024-03-06T12:05:19.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:19.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:19.333974+0000) 2024-03-06T12:05:20.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:20.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:20.334271+0000) 2024-03-06T12:05:21.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:21.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:21.334529+0000) 2024-03-06T12:05:22.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:22.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:22.334812+0000) 2024-03-06T12:05:23.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:23.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:23.335093+0000) 2024-03-06T12:05:24.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:24.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:24.335373+0000) 2024-03-06T12:05:25.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:25.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:25.335711+0000) 2024-03-06T12:05:26.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:26.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:26.335981+0000) 2024-03-06T12:05:27.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:27.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:27.336225+0000) 2024-03-06T12:05:28.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:28.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:28.336497+0000) 2024-03-06T12:05:29.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:29.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:29.336751+0000) 2024-03-06T12:05:30.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:30.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:30.337006+0000) 2024-03-06T12:05:31.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:31.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:31.337250+0000) 2024-03-06T12:05:32.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:32.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:32.337495+0000) 2024-03-06T12:05:33.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:33.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:33.337776+0000) 2024-03-06T12:05:33.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:33.744 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:34.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:34.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:34.338007+0000) 2024-03-06T12:05:35.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:35.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:35.338263+0000) 2024-03-06T12:05:36.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:36.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:36.338504+0000) 2024-03-06T12:05:37.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:37.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:37.338794+0000) 2024-03-06T12:05:38.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:38.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:38.339025+0000) 2024-03-06T12:05:39.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:39.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:39.339256+0000) 2024-03-06T12:05:40.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:40.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:40.339441+0000) 2024-03-06T12:05:41.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:41.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:41.339617+0000) 2024-03-06T12:05:42.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:42.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:42.339830+0000) 2024-03-06T12:05:43.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:43.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:43.340019+0000) 2024-03-06T12:05:44.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:44.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:44.340246+0000) 2024-03-06T12:05:45.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:45.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:45.340454+0000) 2024-03-06T12:05:46.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:46.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:46.340652+0000) 2024-03-06T12:05:47.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:47.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:47.340981+0000) 2024-03-06T12:05:48.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:48.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:48.341208+0000) 2024-03-06T12:05:49.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:49.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:49.341444+0000) 2024-03-06T12:05:50.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:50.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:50.341649+0000) 2024-03-06T12:05:51.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:51.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:51.341954+0000) 2024-03-06T12:05:52.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:52.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:52.342199+0000) 2024-03-06T12:05:53.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:53.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:53.342481+0000) 2024-03-06T12:05:54.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:54.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:54.342755+0000) 2024-03-06T12:05:55.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:55.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:55.342960+0000) 2024-03-06T12:05:56.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:56.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:56.343153+0000) 2024-03-06T12:05:57.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:57.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:57.343343+0000) 2024-03-06T12:05:58.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:58.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:58.343570+0000) 2024-03-06T12:05:59.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:05:59.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:59.343792+0000) 2024-03-06T12:06:00.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:00.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:00.344000+0000) 2024-03-06T12:06:01.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:01.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:01.344229+0000) 2024-03-06T12:06:02.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:02.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:02.344453+0000) 2024-03-06T12:06:03.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:03.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:03.344731+0000) 2024-03-06T12:06:03.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:03.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:04.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:04.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:04.344975+0000) 2024-03-06T12:06:05.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:05.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:05.345181+0000) 2024-03-06T12:06:06.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:06.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:06.345368+0000) 2024-03-06T12:06:07.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:07.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:07.345597+0000) 2024-03-06T12:06:08.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:08.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:08.345875+0000) 2024-03-06T12:06:09.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:09.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:09.346112+0000) 2024-03-06T12:06:10.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:10.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:10.346320+0000) 2024-03-06T12:06:11.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:11.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:11.346509+0000) 2024-03-06T12:06:12.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:12.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:12.346782+0000) 2024-03-06T12:06:13.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:13.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:13.346973+0000) 2024-03-06T12:06:14.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:14.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:14.347211+0000) 2024-03-06T12:06:15.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:15.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:15.347452+0000) 2024-03-06T12:06:16.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:16.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:16.347712+0000) 2024-03-06T12:06:17.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:17.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:17.347950+0000) 2024-03-06T12:06:18.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:18.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:18.348190+0000) 2024-03-06T12:06:19.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:19.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:19.348368+0000) 2024-03-06T12:06:20.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:20.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:20.348588+0000) 2024-03-06T12:06:21.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:21.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:21.348828+0000) 2024-03-06T12:06:22.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:22.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:22.349061+0000) 2024-03-06T12:06:23.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:23.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:23.349289+0000) 2024-03-06T12:06:24.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:24.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:24.349494+0000) 2024-03-06T12:06:25.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:25.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:25.349718+0000) 2024-03-06T12:06:26.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:26.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:26.349943+0000) 2024-03-06T12:06:27.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:27.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:27.350190+0000) 2024-03-06T12:06:28.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:28.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:28.350419+0000) 2024-03-06T12:06:29.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:29.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:29.350628+0000) 2024-03-06T12:06:30.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:30.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:30.350888+0000) 2024-03-06T12:06:31.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:31.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:31.351107+0000) 2024-03-06T12:06:32.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:32.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:32.351285+0000) 2024-03-06T12:06:33.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:33.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:33.351475+0000) 2024-03-06T12:06:34.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:34.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:34.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:34.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:34.351704+0000) 2024-03-06T12:06:35.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:35.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:35.351895+0000) 2024-03-06T12:06:36.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:36.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:36.352114+0000) 2024-03-06T12:06:37.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:37.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:37.352350+0000) 2024-03-06T12:06:38.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:38.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:38.352583+0000) 2024-03-06T12:06:39.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:39.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:39.352814+0000) 2024-03-06T12:06:40.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:40.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:40.353029+0000) 2024-03-06T12:06:41.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:41.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:41.353220+0000) 2024-03-06T12:06:42.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:42.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:42.353447+0000) 2024-03-06T12:06:43.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:43.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:43.353646+0000) 2024-03-06T12:06:44.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:44.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:44.353911+0000) 2024-03-06T12:06:45.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:45.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:45.354110+0000) 2024-03-06T12:06:46.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:46.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:46.354304+0000) 2024-03-06T12:06:47.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:47.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:47.354533+0000) 2024-03-06T12:06:48.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:48.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:48.354765+0000) 2024-03-06T12:06:49.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:49.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:49.354964+0000) 2024-03-06T12:06:50.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:50.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:50.355168+0000) 2024-03-06T12:06:51.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:51.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:51.355357+0000) 2024-03-06T12:06:52.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:52.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:52.355576+0000) 2024-03-06T12:06:53.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:53.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:53.355802+0000) 2024-03-06T12:06:54.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:54.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:54.355991+0000) 2024-03-06T12:06:55.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:55.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:55.356182+0000) 2024-03-06T12:06:56.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:56.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:56.356412+0000) 2024-03-06T12:06:57.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:57.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:57.356648+0000) 2024-03-06T12:06:58.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:58.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:58.356878+0000) 2024-03-06T12:06:59.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:06:59.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:59.357088+0000) 2024-03-06T12:07:00.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:00.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:00.357335+0000) 2024-03-06T12:07:01.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:01.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:01.357616+0000) 2024-03-06T12:07:02.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:02.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:02.357827+0000) 2024-03-06T12:07:03.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:03.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:03.358017+0000) 2024-03-06T12:07:04.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:04.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:04.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:04.358207+0000) 2024-03-06T12:07:04.369 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:05.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:05.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:05.358403+0000) 2024-03-06T12:07:06.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:06.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:06.358624+0000) 2024-03-06T12:07:07.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:07.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:07.358879+0000) 2024-03-06T12:07:08.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:08.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:08.359163+0000) 2024-03-06T12:07:09.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:09.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:09.359398+0000) 2024-03-06T12:07:10.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:10.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:10.359720+0000) 2024-03-06T12:07:11.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:11.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:11.359953+0000) 2024-03-06T12:07:12.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:12.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:12.360165+0000) 2024-03-06T12:07:13.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:13.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:13.360446+0000) 2024-03-06T12:07:14.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:14.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:14.360774+0000) 2024-03-06T12:07:15.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:15.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:15.360938+0000) 2024-03-06T12:07:16.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:16.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:16.361127+0000) 2024-03-06T12:07:17.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:17.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:17.361323+0000) 2024-03-06T12:07:18.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:18.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:18.361575+0000) 2024-03-06T12:07:19.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:19.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:19.361807+0000) 2024-03-06T12:07:20.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:20.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:20.362018+0000) 2024-03-06T12:07:21.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:21.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:21.362261+0000) 2024-03-06T12:07:22.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:22.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:22.362531+0000) 2024-03-06T12:07:23.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:23.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:23.362763+0000) 2024-03-06T12:07:24.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:24.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:24.362981+0000) 2024-03-06T12:07:25.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:25.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:25.363251+0000) 2024-03-06T12:07:26.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:26.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:26.363533+0000) 2024-03-06T12:07:27.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:27.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:27.363784+0000) 2024-03-06T12:07:28.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:28.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:28.363999+0000) 2024-03-06T12:07:29.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:29.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:29.364196+0000) 2024-03-06T12:07:30.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:30.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:30.364386+0000) 2024-03-06T12:07:31.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:31.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:31.364616+0000) 2024-03-06T12:07:32.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:32.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:32.364792+0000) 2024-03-06T12:07:33.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:33.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:33.365035+0000) 2024-03-06T12:07:34.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:34.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:34.365233+0000) 2024-03-06T12:07:34.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:34.509 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:35.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:35.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:35.365431+0000) 2024-03-06T12:07:36.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:36.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:36.365627+0000) 2024-03-06T12:07:37.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:37.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:37.365827+0000) 2024-03-06T12:07:38.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:38.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:38.366026+0000) 2024-03-06T12:07:39.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:39.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:39.366222+0000) 2024-03-06T12:07:40.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:40.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:40.366423+0000) 2024-03-06T12:07:41.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:41.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:41.366660+0000) 2024-03-06T12:07:42.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:42.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:42.366924+0000) 2024-03-06T12:07:43.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:43.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:43.367130+0000) 2024-03-06T12:07:44.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:44.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:44.367377+0000) 2024-03-06T12:07:45.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:45.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:45.367601+0000) 2024-03-06T12:07:46.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:46.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:46.367831+0000) 2024-03-06T12:07:47.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:47.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:47.368027+0000) 2024-03-06T12:07:48.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:48.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:48.368262+0000) 2024-03-06T12:07:49.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:49.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:49.368479+0000) 2024-03-06T12:07:50.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:50.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:50.368693+0000) 2024-03-06T12:07:51.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:51.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:51.368888+0000) 2024-03-06T12:07:52.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:52.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:52.369095+0000) 2024-03-06T12:07:53.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:53.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:53.369287+0000) 2024-03-06T12:07:54.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:54.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:54.369525+0000) 2024-03-06T12:07:55.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:55.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:55.369770+0000) 2024-03-06T12:07:56.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:56.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:56.370011+0000) 2024-03-06T12:07:57.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:57.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:57.370221+0000) 2024-03-06T12:07:58.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:58.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:58.370451+0000) 2024-03-06T12:07:59.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:07:59.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:59.370698+0000) 2024-03-06T12:08:00.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:00.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:00.370935+0000) 2024-03-06T12:08:01.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:01.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:01.371127+0000) 2024-03-06T12:08:02.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:02.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:02.371319+0000) 2024-03-06T12:08:03.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:03.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:03.371510+0000) 2024-03-06T12:08:04.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:04.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:04.371766+0000) 2024-03-06T12:08:04.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:04.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:05.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:05.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:05.371965+0000) 2024-03-06T12:08:06.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:06.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:06.372152+0000) 2024-03-06T12:08:07.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:07.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:07.372385+0000) 2024-03-06T12:08:08.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:08.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:08.372619+0000) 2024-03-06T12:08:09.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:09.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:09.372828+0000) 2024-03-06T12:08:10.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:10.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:10.373059+0000) 2024-03-06T12:08:11.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:11.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:11.373256+0000) 2024-03-06T12:08:12.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:12.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:12.373488+0000) 2024-03-06T12:08:13.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:13.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:13.373751+0000) 2024-03-06T12:08:14.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:14.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:14.373955+0000) 2024-03-06T12:08:15.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:15.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:15.374145+0000) 2024-03-06T12:08:16.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:16.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:16.374393+0000) 2024-03-06T12:08:17.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:17.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:17.374647+0000) 2024-03-06T12:08:18.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:18.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:18.374897+0000) 2024-03-06T12:08:19.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:19.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:19.375103+0000) 2024-03-06T12:08:20.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:20.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:20.375364+0000) 2024-03-06T12:08:21.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:21.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:21.375610+0000) 2024-03-06T12:08:22.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:22.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:22.375789+0000) 2024-03-06T12:08:23.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:23.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:23.376003+0000) 2024-03-06T12:08:24.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:24.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:24.376252+0000) 2024-03-06T12:08:25.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:25.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:25.376498+0000) 2024-03-06T12:08:26.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:26.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:26.376751+0000) 2024-03-06T12:08:27.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:27.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:27.376953+0000) 2024-03-06T12:08:28.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:28.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:28.377173+0000) 2024-03-06T12:08:29.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:29.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:29.377422+0000) 2024-03-06T12:08:30.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:30.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:30.377699+0000) 2024-03-06T12:08:31.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:31.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:31.377910+0000) 2024-03-06T12:08:32.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:32.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:32.378117+0000) 2024-03-06T12:08:33.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:33.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:33.378304+0000) 2024-03-06T12:08:34.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:34.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:34.378534+0000) 2024-03-06T12:08:34.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:34.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:35.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:35.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:35.378813+0000) 2024-03-06T12:08:36.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:36.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:36.379045+0000) 2024-03-06T12:08:37.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:37.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:37.379283+0000) 2024-03-06T12:08:38.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:38.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:38.379528+0000) 2024-03-06T12:08:39.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:39.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:39.379842+0000) 2024-03-06T12:08:40.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:40.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:40.380074+0000) 2024-03-06T12:08:41.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:41.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:41.380296+0000) 2024-03-06T12:08:42.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:42.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:42.380522+0000) 2024-03-06T12:08:43.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:43.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:43.380791+0000) 2024-03-06T12:08:44.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:44.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:44.380995+0000) 2024-03-06T12:08:45.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:45.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:45.381224+0000) 2024-03-06T12:08:46.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:46.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:46.381449+0000) 2024-03-06T12:08:47.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:47.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:47.381713+0000) 2024-03-06T12:08:48.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:48.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:48.381921+0000) 2024-03-06T12:08:49.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:49.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:49.382153+0000) 2024-03-06T12:08:50.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:50.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:50.382393+0000) 2024-03-06T12:08:51.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:51.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:51.382620+0000) 2024-03-06T12:08:52.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:52.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:52.382863+0000) 2024-03-06T12:08:53.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:53.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:53.383111+0000) 2024-03-06T12:08:54.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:54.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:54.383406+0000) 2024-03-06T12:08:55.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:55.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:55.383635+0000) 2024-03-06T12:08:56.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:56.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:56.383870+0000) 2024-03-06T12:08:57.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:57.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:57.384121+0000) 2024-03-06T12:08:58.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:58.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:58.384367+0000) 2024-03-06T12:08:59.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:08:59.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:59.384610+0000) 2024-03-06T12:09:00.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:00.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:00.384856+0000) 2024-03-06T12:09:01.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:01.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:01.385103+0000) 2024-03-06T12:09:02.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:02.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:02.385347+0000) 2024-03-06T12:09:03.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:03.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:03.385585+0000) 2024-03-06T12:09:04.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:04.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:04.385874+0000) 2024-03-06T12:09:05.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:05.126 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:05.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:05.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:05.386118+0000) 2024-03-06T12:09:06.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:06.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:06.386397+0000) 2024-03-06T12:09:07.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:07.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:07.386626+0000) 2024-03-06T12:09:08.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:08.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:08.386927+0000) 2024-03-06T12:09:09.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:09.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:09.387167+0000) 2024-03-06T12:09:10.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:10.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:10.387378+0000) 2024-03-06T12:09:11.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:11.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:11.387621+0000) 2024-03-06T12:09:12.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:12.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:12.387867+0000) 2024-03-06T12:09:13.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:13.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:13.388120+0000) 2024-03-06T12:09:14.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:14.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:14.388372+0000) 2024-03-06T12:09:15.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:15.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:15.388625+0000) 2024-03-06T12:09:16.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:16.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:16.388932+0000) 2024-03-06T12:09:17.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:17.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:17.389253+0000) 2024-03-06T12:09:18.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:18.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:18.389488+0000) 2024-03-06T12:09:19.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:19.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:19.389771+0000) 2024-03-06T12:09:20.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:20.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:20.390023+0000) 2024-03-06T12:09:21.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:21.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:21.390255+0000) 2024-03-06T12:09:22.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:22.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:22.390506+0000) 2024-03-06T12:09:23.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:23.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:23.390798+0000) 2024-03-06T12:09:24.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:24.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:24.391044+0000) 2024-03-06T12:09:25.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:25.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:25.391287+0000) 2024-03-06T12:09:26.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:26.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:26.391535+0000) 2024-03-06T12:09:27.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:27.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:27.391780+0000) 2024-03-06T12:09:28.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:28.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:28.392007+0000) 2024-03-06T12:09:29.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:29.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:29.392260+0000) 2024-03-06T12:09:30.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:30.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:30.392485+0000) 2024-03-06T12:09:31.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:31.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:31.392719+0000) 2024-03-06T12:09:32.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:32.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:32.392947+0000) 2024-03-06T12:09:33.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:33.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:33.393145+0000) 2024-03-06T12:09:34.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:34.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:34.393374+0000) 2024-03-06T12:09:35.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:35.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:35.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:35.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:35.393568+0000) 2024-03-06T12:09:36.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:36.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:36.393766+0000) 2024-03-06T12:09:37.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:37.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:37.393979+0000) 2024-03-06T12:09:38.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:38.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:38.394182+0000) 2024-03-06T12:09:39.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:39.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:39.394375+0000) 2024-03-06T12:09:40.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:40.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:40.394561+0000) 2024-03-06T12:09:41.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:41.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:41.394778+0000) 2024-03-06T12:09:42.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:42.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:42.395020+0000) 2024-03-06T12:09:43.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:43.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:43.395212+0000) 2024-03-06T12:09:44.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:44.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:44.395440+0000) 2024-03-06T12:09:45.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:45.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:45.395640+0000) 2024-03-06T12:09:46.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:46.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:46.395889+0000) 2024-03-06T12:09:47.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:47.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:47.396156+0000) 2024-03-06T12:09:48.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:48.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:48.396350+0000) 2024-03-06T12:09:49.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:49.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:49.396565+0000) 2024-03-06T12:09:50.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:50.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:50.396803+0000) 2024-03-06T12:09:51.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:51.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:51.397030+0000) 2024-03-06T12:09:52.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:52.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:52.397230+0000) 2024-03-06T12:09:53.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:53.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:53.397419+0000) 2024-03-06T12:09:54.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:54.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:54.397658+0000) 2024-03-06T12:09:55.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:55.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:55.397952+0000) 2024-03-06T12:09:56.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:56.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:56.398172+0000) 2024-03-06T12:09:57.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:57.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:57.398417+0000) 2024-03-06T12:09:58.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:58.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:58.398654+0000) 2024-03-06T12:09:59.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:09:59.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:59.398905+0000) 2024-03-06T12:10:00.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:00.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:00.399109+0000) 2024-03-06T12:10:01.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:01.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:01.399345+0000) 2024-03-06T12:10:02.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:02.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:02.399577+0000) 2024-03-06T12:10:03.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:03.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:03.399819+0000) 2024-03-06T12:10:04.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:04.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:04.400052+0000) 2024-03-06T12:10:05.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:05.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:05.400280+0000) 2024-03-06T12:10:05.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:05.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:06.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:06.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:06.400509+0000) 2024-03-06T12:10:07.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:07.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:07.400779+0000) 2024-03-06T12:10:08.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:08.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:08.401015+0000) 2024-03-06T12:10:09.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:09.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:09.401221+0000) 2024-03-06T12:10:10.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:10.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:10.401418+0000) 2024-03-06T12:10:11.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:11.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:11.401618+0000) 2024-03-06T12:10:12.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:12.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:12.401852+0000) 2024-03-06T12:10:13.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:13.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:13.402093+0000) 2024-03-06T12:10:14.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:14.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:14.402334+0000) 2024-03-06T12:10:15.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:15.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:15.402522+0000) 2024-03-06T12:10:16.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:16.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:16.402792+0000) 2024-03-06T12:10:17.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:17.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:17.402993+0000) 2024-03-06T12:10:18.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:18.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:18.403192+0000) 2024-03-06T12:10:19.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:19.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:19.403383+0000) 2024-03-06T12:10:20.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:20.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:20.403617+0000) 2024-03-06T12:10:21.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:21.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:21.403846+0000) 2024-03-06T12:10:22.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:22.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:22.404071+0000) 2024-03-06T12:10:23.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:23.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:23.404296+0000) 2024-03-06T12:10:24.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:24.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:24.404528+0000) 2024-03-06T12:10:25.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:25.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:25.404797+0000) 2024-03-06T12:10:26.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:26.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:26.405034+0000) 2024-03-06T12:10:27.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:27.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:27.405234+0000) 2024-03-06T12:10:28.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:28.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:28.405475+0000) 2024-03-06T12:10:29.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:29.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:29.405686+0000) 2024-03-06T12:10:30.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:30.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:30.405979+0000) 2024-03-06T12:10:31.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:31.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:31.406209+0000) 2024-03-06T12:10:32.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:32.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:32.406406+0000) 2024-03-06T12:10:33.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:33.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:33.406646+0000) 2024-03-06T12:10:34.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:34.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:34.406858+0000) 2024-03-06T12:10:35.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:35.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:35.407050+0000) 2024-03-06T12:10:35.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:35.777 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:36.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:36.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:36.407233+0000) 2024-03-06T12:10:37.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:37.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:37.407424+0000) 2024-03-06T12:10:38.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:38.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:38.407617+0000) 2024-03-06T12:10:39.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:39.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:39.407782+0000) 2024-03-06T12:10:40.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:40.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:40.408031+0000) 2024-03-06T12:10:41.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:41.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:41.408263+0000) 2024-03-06T12:10:42.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:42.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:42.408468+0000) 2024-03-06T12:10:43.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:43.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:43.408720+0000) 2024-03-06T12:10:44.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:44.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:44.408906+0000) 2024-03-06T12:10:45.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:45.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:45.409092+0000) 2024-03-06T12:10:46.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:46.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:46.409278+0000) 2024-03-06T12:10:47.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:47.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:47.409505+0000) 2024-03-06T12:10:48.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:48.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:48.409764+0000) 2024-03-06T12:10:49.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:49.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:49.409953+0000) 2024-03-06T12:10:50.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:50.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:50.410152+0000) 2024-03-06T12:10:51.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:51.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:51.410381+0000) 2024-03-06T12:10:52.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:52.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:52.410608+0000) 2024-03-06T12:10:53.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:53.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:53.410841+0000) 2024-03-06T12:10:54.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:54.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:54.411032+0000) 2024-03-06T12:10:55.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:55.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:55.411218+0000) 2024-03-06T12:10:56.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:56.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:56.411402+0000) 2024-03-06T12:10:57.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:57.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:57.411630+0000) 2024-03-06T12:10:58.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:58.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:58.411820+0000) 2024-03-06T12:10:59.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:10:59.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:59.412017+0000) 2024-03-06T12:11:00.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:00.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:00.412205+0000) 2024-03-06T12:11:01.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:01.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:01.412427+0000) 2024-03-06T12:11:02.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:02.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:02.412617+0000) 2024-03-06T12:11:03.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:03.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:03.412828+0000) 2024-03-06T12:11:04.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:04.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:04.413007+0000) 2024-03-06T12:11:05.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:05.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:05.413188+0000) 2024-03-06T12:11:05.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:06.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:06.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:06.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:06.413342+0000) 2024-03-06T12:11:07.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:07.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:07.413593+0000) 2024-03-06T12:11:08.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:08.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:08.413843+0000) 2024-03-06T12:11:09.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:09.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:09.414079+0000) 2024-03-06T12:11:10.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:10.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:10.414282+0000) 2024-03-06T12:11:11.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:11.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:11.414532+0000) 2024-03-06T12:11:12.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:12.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:12.414783+0000) 2024-03-06T12:11:13.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:13.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:13.415022+0000) 2024-03-06T12:11:14.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:14.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:14.415216+0000) 2024-03-06T12:11:15.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:15.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:15.415436+0000) 2024-03-06T12:11:16.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:16.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:16.415702+0000) 2024-03-06T12:11:17.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:17.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:17.415904+0000) 2024-03-06T12:11:18.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:18.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:18.416135+0000) 2024-03-06T12:11:19.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:19.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:19.416322+0000) 2024-03-06T12:11:20.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:20.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:20.416559+0000) 2024-03-06T12:11:21.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:21.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:21.416795+0000) 2024-03-06T12:11:22.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:22.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:22.417025+0000) 2024-03-06T12:11:23.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:23.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:23.417206+0000) 2024-03-06T12:11:24.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:24.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:24.417438+0000) 2024-03-06T12:11:25.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:25.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:25.417660+0000) 2024-03-06T12:11:26.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:26.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:26.417877+0000) 2024-03-06T12:11:27.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:27.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:27.418082+0000) 2024-03-06T12:11:28.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:28.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:28.418332+0000) 2024-03-06T12:11:29.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:29.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:29.418573+0000) 2024-03-06T12:11:30.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:30.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:30.418820+0000) 2024-03-06T12:11:31.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:31.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:31.419069+0000) 2024-03-06T12:11:32.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:32.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:32.419300+0000) 2024-03-06T12:11:33.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:33.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:33.419497+0000) 2024-03-06T12:11:34.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:34.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:34.419711+0000) 2024-03-06T12:11:35.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:35.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:35.419936+0000) 2024-03-06T12:11:36.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:36.227 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:36.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:36.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:36.420134+0000) 2024-03-06T12:11:37.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:37.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:37.420327+0000) 2024-03-06T12:11:38.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:38.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:38.420575+0000) 2024-03-06T12:11:39.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:39.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:39.420850+0000) 2024-03-06T12:11:40.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:40.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:40.421087+0000) 2024-03-06T12:11:41.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:41.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:41.421329+0000) 2024-03-06T12:11:42.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:42.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:42.421540+0000) 2024-03-06T12:11:43.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:43.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:43.421784+0000) 2024-03-06T12:11:44.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:44.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:44.422013+0000) 2024-03-06T12:11:45.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:45.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:45.422206+0000) 2024-03-06T12:11:46.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:46.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:46.422385+0000) 2024-03-06T12:11:47.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:47.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:47.422632+0000) 2024-03-06T12:11:48.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:48.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:48.422852+0000) 2024-03-06T12:11:49.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:49.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:49.423052+0000) 2024-03-06T12:11:50.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:50.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:50.423266+0000) 2024-03-06T12:11:51.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:51.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:51.423506+0000) 2024-03-06T12:11:52.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:52.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:52.423727+0000) 2024-03-06T12:11:53.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:53.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:53.423926+0000) 2024-03-06T12:11:54.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:54.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:54.424121+0000) 2024-03-06T12:11:55.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:55.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:55.424326+0000) 2024-03-06T12:11:56.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:56.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:56.424580+0000) 2024-03-06T12:11:57.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:57.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:57.424812+0000) 2024-03-06T12:11:58.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:58.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:58.425009+0000) 2024-03-06T12:11:59.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:11:59.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:59.425195+0000) 2024-03-06T12:12:00.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:00.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:00.425429+0000) 2024-03-06T12:12:01.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:01.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:01.425648+0000) 2024-03-06T12:12:02.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:02.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:02.425883+0000) 2024-03-06T12:12:03.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:03.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:03.426082+0000) 2024-03-06T12:12:04.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:04.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:04.426315+0000) 2024-03-06T12:12:05.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:05.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:05.426530+0000) 2024-03-06T12:12:06.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:06.371 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:06.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:06.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:06.426737+0000) 2024-03-06T12:12:07.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:07.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:07.426941+0000) 2024-03-06T12:12:08.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:08.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:08.427173+0000) 2024-03-06T12:12:09.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:09.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:09.427396+0000) 2024-03-06T12:12:10.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:10.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:10.427624+0000) 2024-03-06T12:12:11.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:11.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:11.427859+0000) 2024-03-06T12:12:12.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:12.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:12.428091+0000) 2024-03-06T12:12:13.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:13.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:13.428310+0000) 2024-03-06T12:12:14.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:14.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:14.428537+0000) 2024-03-06T12:12:15.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:15.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:15.428749+0000) 2024-03-06T12:12:16.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:16.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:16.428957+0000) 2024-03-06T12:12:17.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:17.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:17.429162+0000) 2024-03-06T12:12:18.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:18.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:18.429355+0000) 2024-03-06T12:12:19.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:19.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:19.429550+0000) 2024-03-06T12:12:20.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:20.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:20.429811+0000) 2024-03-06T12:12:21.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:21.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:21.430023+0000) 2024-03-06T12:12:22.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:22.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:22.430216+0000) 2024-03-06T12:12:23.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:23.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:23.430443+0000) 2024-03-06T12:12:24.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:24.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:24.430639+0000) 2024-03-06T12:12:25.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:25.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:25.430867+0000) 2024-03-06T12:12:26.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:26.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:26.431068+0000) 2024-03-06T12:12:27.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:27.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:27.431319+0000) 2024-03-06T12:12:28.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:28.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:28.431513+0000) 2024-03-06T12:12:29.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:29.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:29.431780+0000) 2024-03-06T12:12:30.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:30.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:30.431981+0000) 2024-03-06T12:12:31.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:31.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:31.432181+0000) 2024-03-06T12:12:32.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:32.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:32.432415+0000) 2024-03-06T12:12:33.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:33.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:33.432640+0000) 2024-03-06T12:12:34.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:34.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:34.432897+0000) 2024-03-06T12:12:35.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:35.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:35.433072+0000) 2024-03-06T12:12:36.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:36.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:36.433265+0000) 2024-03-06T12:12:36.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:36.584 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:37.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:37.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:37.433505+0000) 2024-03-06T12:12:38.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:38.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:38.433779+0000) 2024-03-06T12:12:39.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:39.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:39.433972+0000) 2024-03-06T12:12:40.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:40.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:40.434168+0000) 2024-03-06T12:12:41.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:41.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:41.434395+0000) 2024-03-06T12:12:42.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:42.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:42.434626+0000) 2024-03-06T12:12:43.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:43.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:43.434850+0000) 2024-03-06T12:12:44.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:44.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:44.435047+0000) 2024-03-06T12:12:45.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:45.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:45.435231+0000) 2024-03-06T12:12:46.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:46.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:46.435429+0000) 2024-03-06T12:12:47.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:47.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:47.435726+0000) 2024-03-06T12:12:48.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:48.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:48.435975+0000) 2024-03-06T12:12:49.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:49.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:49.436151+0000) 2024-03-06T12:12:50.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:50.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:50.436374+0000) 2024-03-06T12:12:51.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:51.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:51.436601+0000) 2024-03-06T12:12:52.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:52.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:52.436793+0000) 2024-03-06T12:12:53.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:53.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:53.436999+0000) 2024-03-06T12:12:54.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:54.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:54.437205+0000) 2024-03-06T12:12:55.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:55.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:55.437434+0000) 2024-03-06T12:12:56.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:56.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:56.437615+0000) 2024-03-06T12:12:57.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:57.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:57.437886+0000) 2024-03-06T12:12:58.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:58.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:58.438133+0000) 2024-03-06T12:12:59.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:12:59.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:59.438356+0000) 2024-03-06T12:13:00.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:00.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:00.438546+0000) 2024-03-06T12:13:01.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:01.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:01.438769+0000) 2024-03-06T12:13:02.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:02.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:02.439010+0000) 2024-03-06T12:13:03.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:03.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:03.439239+0000) 2024-03-06T12:13:04.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:04.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:04.439473+0000) 2024-03-06T12:13:05.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:05.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:05.439657+0000) 2024-03-06T12:13:06.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:06.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:06.439911+0000) 2024-03-06T12:13:06.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:06.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:07.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:07.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:07.440143+0000) 2024-03-06T12:13:08.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:08.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:08.440390+0000) 2024-03-06T12:13:09.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:09.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:09.440573+0000) 2024-03-06T12:13:10.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:10.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:10.440799+0000) 2024-03-06T12:13:11.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:11.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:11.441023+0000) 2024-03-06T12:13:12.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:12.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:12.441261+0000) 2024-03-06T12:13:13.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:13.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:13.441465+0000) 2024-03-06T12:13:14.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:14.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:14.441739+0000) 2024-03-06T12:13:15.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:15.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:15.442013+0000) 2024-03-06T12:13:16.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:16.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:16.442237+0000) 2024-03-06T12:13:17.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:17.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:17.442468+0000) 2024-03-06T12:13:18.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:18.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:18.442737+0000) 2024-03-06T12:13:19.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:19.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:19.442955+0000) 2024-03-06T12:13:20.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:20.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:20.443181+0000) 2024-03-06T12:13:21.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:21.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:21.443364+0000) 2024-03-06T12:13:22.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:22.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:22.443594+0000) 2024-03-06T12:13:23.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:23.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:23.443817+0000) 2024-03-06T12:13:24.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:24.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:24.444039+0000) 2024-03-06T12:13:25.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:25.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:25.444225+0000) 2024-03-06T12:13:26.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:26.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:26.444460+0000) 2024-03-06T12:13:27.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:27.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:27.444736+0000) 2024-03-06T12:13:28.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:28.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:28.444980+0000) 2024-03-06T12:13:29.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:29.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:29.445204+0000) 2024-03-06T12:13:30.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:30.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:30.445428+0000) 2024-03-06T12:13:31.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:31.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:31.445613+0000) 2024-03-06T12:13:32.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:32.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:32.445839+0000) 2024-03-06T12:13:33.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:33.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:33.446068+0000) 2024-03-06T12:13:34.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:34.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:34.446296+0000) 2024-03-06T12:13:35.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:35.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:35.446515+0000) 2024-03-06T12:13:36.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:36.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:36.446779+0000) 2024-03-06T12:13:36.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:37.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:37.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:37.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:37.447024+0000) 2024-03-06T12:13:38.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:38.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:38.447221+0000) 2024-03-06T12:13:39.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:39.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:39.447414+0000) 2024-03-06T12:13:40.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:40.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:40.447650+0000) 2024-03-06T12:13:41.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:41.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:41.447896+0000) 2024-03-06T12:13:42.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:42.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:42.448082+0000) 2024-03-06T12:13:43.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:43.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:43.448309+0000) 2024-03-06T12:13:44.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:44.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:44.448540+0000) 2024-03-06T12:13:45.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:45.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:45.448768+0000) 2024-03-06T12:13:46.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:46.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:46.448988+0000) 2024-03-06T12:13:47.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:47.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:47.449228+0000) 2024-03-06T12:13:48.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:48.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:48.449424+0000) 2024-03-06T12:13:49.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:49.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:49.449652+0000) 2024-03-06T12:13:50.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:50.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:50.449840+0000) 2024-03-06T12:13:51.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:51.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:51.450027+0000) 2024-03-06T12:13:52.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:52.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:52.450232+0000) 2024-03-06T12:13:53.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:53.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:53.450475+0000) 2024-03-06T12:13:54.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:54.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:54.450730+0000) 2024-03-06T12:13:55.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:55.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:55.450963+0000) 2024-03-06T12:13:56.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:56.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:56.451184+0000) 2024-03-06T12:13:57.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:57.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:57.451359+0000) 2024-03-06T12:13:58.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:58.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:58.451599+0000) 2024-03-06T12:13:59.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:13:59.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:59.451814+0000) 2024-03-06T12:14:00.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:00.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:00.452043+0000) 2024-03-06T12:14:01.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:01.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:01.452264+0000) 2024-03-06T12:14:02.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:02.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:02.452451+0000) 2024-03-06T12:14:03.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:03.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:03.452770+0000) 2024-03-06T12:14:04.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:04.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:04.452925+0000) 2024-03-06T12:14:05.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:05.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:05.453100+0000) 2024-03-06T12:14:06.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:06.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:06.453337+0000) 2024-03-06T12:14:07.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:07.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:07.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:07.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:07.453612+0000) 2024-03-06T12:14:08.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:08.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:08.453877+0000) 2024-03-06T12:14:09.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:09.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:09.454099+0000) 2024-03-06T12:14:10.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:10.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:10.454326+0000) 2024-03-06T12:14:11.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:11.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:11.454556+0000) 2024-03-06T12:14:12.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:12.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:12.454807+0000) 2024-03-06T12:14:13.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:13.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:13.455052+0000) 2024-03-06T12:14:14.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:14.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:14.455246+0000) 2024-03-06T12:14:15.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:15.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:15.455480+0000) 2024-03-06T12:14:16.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:16.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:16.455738+0000) 2024-03-06T12:14:17.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:17.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:17.455974+0000) 2024-03-06T12:14:18.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:18.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:18.456165+0000) 2024-03-06T12:14:19.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:19.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:19.456361+0000) 2024-03-06T12:14:20.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:20.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:20.456579+0000) 2024-03-06T12:14:21.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:21.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:21.456817+0000) 2024-03-06T12:14:22.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:22.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:22.457050+0000) 2024-03-06T12:14:23.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:23.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:23.457283+0000) 2024-03-06T12:14:24.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:24.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:24.457501+0000) 2024-03-06T12:14:25.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:25.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:25.457755+0000) 2024-03-06T12:14:26.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:26.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:26.457925+0000) 2024-03-06T12:14:27.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:27.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:27.458162+0000) 2024-03-06T12:14:28.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:28.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:28.458389+0000) 2024-03-06T12:14:29.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:29.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:29.458606+0000) 2024-03-06T12:14:30.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:30.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:30.458838+0000) 2024-03-06T12:14:31.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:31.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:31.459054+0000) 2024-03-06T12:14:32.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:32.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:32.459242+0000) 2024-03-06T12:14:33.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:33.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:33.459434+0000) 2024-03-06T12:14:34.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:34.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:34.459696+0000) 2024-03-06T12:14:35.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:35.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:35.459935+0000) 2024-03-06T12:14:36.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:36.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:36.460126+0000) 2024-03-06T12:14:37.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:37.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:37.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:37.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:37.460299+0000) 2024-03-06T12:14:38.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:38.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:38.460522+0000) 2024-03-06T12:14:39.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:39.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:39.460791+0000) 2024-03-06T12:14:40.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:40.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:40.460993+0000) 2024-03-06T12:14:41.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:41.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:41.461185+0000) 2024-03-06T12:14:42.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:42.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:42.461375+0000) 2024-03-06T12:14:43.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:43.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:43.461612+0000) 2024-03-06T12:14:44.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:44.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:44.461817+0000) 2024-03-06T12:14:45.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:45.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:45.462020+0000) 2024-03-06T12:14:46.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:46.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:46.462192+0000) 2024-03-06T12:14:47.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:47.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:47.462393+0000) 2024-03-06T12:14:48.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:48.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:48.462616+0000) 2024-03-06T12:14:49.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:49.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:49.462827+0000) 2024-03-06T12:14:50.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:50.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:50.463025+0000) 2024-03-06T12:14:51.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:51.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:51.463215+0000) 2024-03-06T12:14:52.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:52.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:52.463409+0000) 2024-03-06T12:14:53.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:53.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:53.463636+0000) 2024-03-06T12:14:54.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:54.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:54.463899+0000) 2024-03-06T12:14:55.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:55.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:55.464123+0000) 2024-03-06T12:14:56.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:56.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:56.464397+0000) 2024-03-06T12:14:57.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:57.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:57.464641+0000) 2024-03-06T12:14:58.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:58.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:58.464880+0000) 2024-03-06T12:14:59.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:14:59.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:59.465105+0000) 2024-03-06T12:15:00.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:00.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:00.465288+0000) 2024-03-06T12:15:01.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:01.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:01.465453+0000) 2024-03-06T12:15:02.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:02.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:02.465648+0000) 2024-03-06T12:15:03.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:03.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:03.465865+0000) 2024-03-06T12:15:04.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:04.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:04.466113+0000) 2024-03-06T12:15:05.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:05.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:05.466303+0000) 2024-03-06T12:15:06.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:06.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:06.466528+0000) 2024-03-06T12:15:07.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:07.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:07.466801+0000) 2024-03-06T12:15:07.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:07.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:08.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:08.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:08.466954+0000) 2024-03-06T12:15:09.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:09.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:09.467138+0000) 2024-03-06T12:15:10.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:10.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:10.467338+0000) 2024-03-06T12:15:11.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:11.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:11.467573+0000) 2024-03-06T12:15:12.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:12.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:12.467813+0000) 2024-03-06T12:15:13.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:13.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:13.468043+0000) 2024-03-06T12:15:14.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:14.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:14.468263+0000) 2024-03-06T12:15:15.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:15.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:15.468487+0000) 2024-03-06T12:15:16.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:16.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:16.468762+0000) 2024-03-06T12:15:17.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:17.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:17.468970+0000) 2024-03-06T12:15:18.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:18.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:18.469158+0000) 2024-03-06T12:15:19.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:19.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:19.469351+0000) 2024-03-06T12:15:20.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:20.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:20.469574+0000) 2024-03-06T12:15:21.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:21.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:21.469807+0000) 2024-03-06T12:15:22.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:22.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:22.470006+0000) 2024-03-06T12:15:23.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:23.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:23.470239+0000) 2024-03-06T12:15:24.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:24.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:24.470471+0000) 2024-03-06T12:15:25.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:25.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:25.470709+0000) 2024-03-06T12:15:26.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:26.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:26.470944+0000) 2024-03-06T12:15:27.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:27.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:27.471186+0000) 2024-03-06T12:15:28.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:28.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:28.471420+0000) 2024-03-06T12:15:29.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:29.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:29.471646+0000) 2024-03-06T12:15:30.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:30.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:30.471916+0000) 2024-03-06T12:15:31.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:31.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:31.472135+0000) 2024-03-06T12:15:32.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:32.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:32.472360+0000) 2024-03-06T12:15:33.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:33.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:33.472553+0000) 2024-03-06T12:15:34.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:34.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:34.472823+0000) 2024-03-06T12:15:35.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:35.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:35.473021+0000) 2024-03-06T12:15:36.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:36.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:36.473230+0000) 2024-03-06T12:15:37.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:37.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:37.473455+0000) 2024-03-06T12:15:37.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:37.803 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:38.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:38.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:38.473719+0000) 2024-03-06T12:15:39.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:39.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:39.473952+0000) 2024-03-06T12:15:40.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:40.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:40.474188+0000) 2024-03-06T12:15:41.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:41.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:41.474378+0000) 2024-03-06T12:15:42.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:42.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:42.474603+0000) 2024-03-06T12:15:43.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:43.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:43.474814+0000) 2024-03-06T12:15:44.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:44.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:44.475009+0000) 2024-03-06T12:15:45.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:45.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:45.475221+0000) 2024-03-06T12:15:46.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:46.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:46.475454+0000) 2024-03-06T12:15:47.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:47.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:47.475763+0000) 2024-03-06T12:15:48.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:48.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:48.475992+0000) 2024-03-06T12:15:49.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:49.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:49.476226+0000) 2024-03-06T12:15:50.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:50.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:50.476531+0000) 2024-03-06T12:15:51.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:51.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:51.476811+0000) 2024-03-06T12:15:52.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:52.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:52.477088+0000) 2024-03-06T12:15:53.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:53.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:53.477312+0000) 2024-03-06T12:15:54.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:54.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:54.477585+0000) 2024-03-06T12:15:55.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:55.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:55.477872+0000) 2024-03-06T12:15:56.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:56.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:56.478101+0000) 2024-03-06T12:15:57.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:57.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:57.478295+0000) 2024-03-06T12:15:58.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:58.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:58.478500+0000) 2024-03-06T12:15:59.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:15:59.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:59.478766+0000) 2024-03-06T12:16:00.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:00.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:00.478993+0000) 2024-03-06T12:16:01.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:01.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:01.479214+0000) 2024-03-06T12:16:02.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:02.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:02.479453+0000) 2024-03-06T12:16:03.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:03.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:03.479706+0000) 2024-03-06T12:16:04.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:04.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:04.479990+0000) 2024-03-06T12:16:05.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:05.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:05.480219+0000) 2024-03-06T12:16:06.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:06.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:06.480497+0000) 2024-03-06T12:16:07.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:07.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:07.480797+0000) 2024-03-06T12:16:08.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:08.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:08.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:08.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:08.481043+0000) 2024-03-06T12:16:09.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:09.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:09.481267+0000) 2024-03-06T12:16:10.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:10.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:10.481451+0000) 2024-03-06T12:16:11.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:11.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:11.481647+0000) 2024-03-06T12:16:12.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:12.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:12.481934+0000) 2024-03-06T12:16:13.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:13.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:13.482124+0000) 2024-03-06T12:16:14.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:14.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:14.482322+0000) 2024-03-06T12:16:15.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:15.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:15.482554+0000) 2024-03-06T12:16:16.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:16.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:16.482792+0000) 2024-03-06T12:16:17.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:17.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:17.483001+0000) 2024-03-06T12:16:18.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:18.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:18.483196+0000) 2024-03-06T12:16:19.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:19.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:19.483407+0000) 2024-03-06T12:16:20.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:20.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:20.483635+0000) 2024-03-06T12:16:21.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:21.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:21.483815+0000) 2024-03-06T12:16:22.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:22.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:22.484018+0000) 2024-03-06T12:16:23.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:23.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:23.484242+0000) 2024-03-06T12:16:24.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:24.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:24.484522+0000) 2024-03-06T12:16:25.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:25.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:25.484830+0000) 2024-03-06T12:16:26.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:26.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:26.485106+0000) 2024-03-06T12:16:27.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:27.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:27.485359+0000) 2024-03-06T12:16:28.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:28.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:28.485581+0000) 2024-03-06T12:16:29.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:29.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:29.485783+0000) 2024-03-06T12:16:30.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:30.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:30.486032+0000) 2024-03-06T12:16:31.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:31.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:31.486305+0000) 2024-03-06T12:16:32.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:32.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:32.486585+0000) 2024-03-06T12:16:33.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:33.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:33.486803+0000) 2024-03-06T12:16:34.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:34.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:34.487043+0000) 2024-03-06T12:16:35.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:35.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:35.487264+0000) 2024-03-06T12:16:36.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:36.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:36.487524+0000) 2024-03-06T12:16:37.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:37.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:37.487811+0000) 2024-03-06T12:16:38.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:38.226 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:38.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:38.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:38.487987+0000) 2024-03-06T12:16:39.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:39.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:39.488168+0000) 2024-03-06T12:16:40.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:40.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:40.488393+0000) 2024-03-06T12:16:41.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:41.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:41.488580+0000) 2024-03-06T12:16:42.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:42.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:42.488825+0000) 2024-03-06T12:16:43.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:43.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:43.489088+0000) 2024-03-06T12:16:44.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:44.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:44.489362+0000) 2024-03-06T12:16:45.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:45.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:45.489630+0000) 2024-03-06T12:16:46.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:46.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:46.489909+0000) 2024-03-06T12:16:47.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:47.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:47.490181+0000) 2024-03-06T12:16:48.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:48.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:48.490460+0000) 2024-03-06T12:16:49.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:49.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:49.490768+0000) 2024-03-06T12:16:50.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:50.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:50.490942+0000) 2024-03-06T12:16:51.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:51.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:51.491119+0000) 2024-03-06T12:16:52.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:52.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:52.491309+0000) 2024-03-06T12:16:53.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:53.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:53.491511+0000) 2024-03-06T12:16:54.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:54.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:54.491782+0000) 2024-03-06T12:16:55.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:55.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:55.492008+0000) 2024-03-06T12:16:56.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:56.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:56.492196+0000) 2024-03-06T12:16:57.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:57.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:57.492436+0000) 2024-03-06T12:16:58.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:58.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:58.492704+0000) 2024-03-06T12:16:59.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:16:59.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:59.492910+0000) 2024-03-06T12:17:00.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:00.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:00.493148+0000) 2024-03-06T12:17:01.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:01.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:01.493372+0000) 2024-03-06T12:17:02.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:02.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:02.493566+0000) 2024-03-06T12:17:03.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:03.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:03.493827+0000) 2024-03-06T12:17:04.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:04.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:04.494014+0000) 2024-03-06T12:17:05.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:05.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:05.494188+0000) 2024-03-06T12:17:06.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:06.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:06.494415+0000) 2024-03-06T12:17:07.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:07.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:07.494686+0000) 2024-03-06T12:17:08.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:08.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:08.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:08.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:08.494839+0000) 2024-03-06T12:17:09.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:09.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:09.495107+0000) 2024-03-06T12:17:10.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:10.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:10.495376+0000) 2024-03-06T12:17:11.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:11.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:11.495653+0000) 2024-03-06T12:17:12.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:12.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:12.495866+0000) 2024-03-06T12:17:13.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:13.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:13.496069+0000) 2024-03-06T12:17:14.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:14.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:14.496282+0000) 2024-03-06T12:17:15.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:15.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:15.496495+0000) 2024-03-06T12:17:16.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:16.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:16.496646+0000) 2024-03-06T12:17:17.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:17.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:17.496832+0000) 2024-03-06T12:17:18.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:18.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:18.497230+0000) 2024-03-06T12:17:19.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:19.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:19.497411+0000) 2024-03-06T12:17:20.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:20.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:20.497691+0000) 2024-03-06T12:17:21.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:21.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:21.497889+0000) 2024-03-06T12:17:22.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:22.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:22.498099+0000) 2024-03-06T12:17:23.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:23.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:23.498335+0000) 2024-03-06T12:17:24.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:24.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:24.498588+0000) 2024-03-06T12:17:25.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:25.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:25.498788+0000) 2024-03-06T12:17:26.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:26.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:26.499027+0000) 2024-03-06T12:17:27.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:27.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:27.499280+0000) 2024-03-06T12:17:28.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:28.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:28.499509+0000) 2024-03-06T12:17:29.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:29.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:29.499790+0000) 2024-03-06T12:17:30.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:30.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:30.499988+0000) 2024-03-06T12:17:31.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:31.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:31.500274+0000) 2024-03-06T12:17:32.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:32.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:32.500522+0000) 2024-03-06T12:17:33.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:33.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:33.500796+0000) 2024-03-06T12:17:34.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:34.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:34.501018+0000) 2024-03-06T12:17:35.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:35.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:35.501245+0000) 2024-03-06T12:17:36.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:36.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:36.501475+0000) 2024-03-06T12:17:37.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:37.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:37.501776+0000) 2024-03-06T12:17:38.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:38.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:38.501973+0000) 2024-03-06T12:17:38.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:38.599 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:39.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:39.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:39.502198+0000) 2024-03-06T12:17:40.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:40.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:40.502441+0000) 2024-03-06T12:17:41.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:41.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:41.502698+0000) 2024-03-06T12:17:42.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:42.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:42.502918+0000) 2024-03-06T12:17:43.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:43.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:43.503089+0000) 2024-03-06T12:17:44.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:44.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:44.503313+0000) 2024-03-06T12:17:45.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:45.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:45.503558+0000) 2024-03-06T12:17:46.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:46.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:46.503792+0000) 2024-03-06T12:17:47.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:47.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:47.504015+0000) 2024-03-06T12:17:48.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:48.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:48.504214+0000) 2024-03-06T12:17:49.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:49.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:49.504441+0000) 2024-03-06T12:17:50.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:50.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:50.504642+0000) 2024-03-06T12:17:51.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:51.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:51.504909+0000) 2024-03-06T12:17:52.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:52.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:52.505161+0000) 2024-03-06T12:17:53.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:53.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:53.505397+0000) 2024-03-06T12:17:54.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:54.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:54.505626+0000) 2024-03-06T12:17:55.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:55.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:55.505819+0000) 2024-03-06T12:17:56.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:56.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:56.506041+0000) 2024-03-06T12:17:57.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:57.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:57.506260+0000) 2024-03-06T12:17:58.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:58.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:58.506461+0000) 2024-03-06T12:17:59.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:17:59.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:59.506726+0000) 2024-03-06T12:18:00.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:00.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:00.506955+0000) 2024-03-06T12:18:01.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:01.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:01.507150+0000) 2024-03-06T12:18:02.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:02.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:02.507329+0000) 2024-03-06T12:18:03.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:03.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:03.507575+0000) 2024-03-06T12:18:04.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:04.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:04.507806+0000) 2024-03-06T12:18:05.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:05.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:05.508005+0000) 2024-03-06T12:18:06.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:06.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:06.508272+0000) 2024-03-06T12:18:07.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:07.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:07.508568+0000) 2024-03-06T12:18:08.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:08.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:08.508857+0000) 2024-03-06T12:18:08.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:08.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:09.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:09.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:09.509105+0000) 2024-03-06T12:18:10.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:10.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:10.509283+0000) 2024-03-06T12:18:11.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:11.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:11.509517+0000) 2024-03-06T12:18:12.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:12.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:12.509816+0000) 2024-03-06T12:18:13.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:13.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:13.510013+0000) 2024-03-06T12:18:14.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:14.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:14.510221+0000) 2024-03-06T12:18:15.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:15.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:15.510442+0000) 2024-03-06T12:18:16.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:16.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:16.510700+0000) 2024-03-06T12:18:17.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:17.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:17.510954+0000) 2024-03-06T12:18:18.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:18.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:18.511102+0000) 2024-03-06T12:18:19.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:19.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:19.511326+0000) 2024-03-06T12:18:20.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:20.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:20.511561+0000) 2024-03-06T12:18:21.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:21.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:21.511791+0000) 2024-03-06T12:18:22.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:22.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:22.512033+0000) 2024-03-06T12:18:23.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:23.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:23.512237+0000) 2024-03-06T12:18:24.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:24.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:24.512479+0000) 2024-03-06T12:18:25.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:25.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:25.512753+0000) 2024-03-06T12:18:26.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:26.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:26.512942+0000) 2024-03-06T12:18:27.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:27.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:27.513118+0000) 2024-03-06T12:18:28.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:28.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:28.513361+0000) 2024-03-06T12:18:29.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:29.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:29.513579+0000) 2024-03-06T12:18:30.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:30.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:30.513803+0000) 2024-03-06T12:18:31.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:31.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:31.513999+0000) 2024-03-06T12:18:32.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:32.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:32.514228+0000) 2024-03-06T12:18:33.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:33.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:33.514468+0000) 2024-03-06T12:18:34.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:34.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:34.514758+0000) 2024-03-06T12:18:35.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:35.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:35.514955+0000) 2024-03-06T12:18:36.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:36.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:36.515179+0000) 2024-03-06T12:18:37.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:37.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:37.515413+0000) 2024-03-06T12:18:38.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:38.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:38.515653+0000) 2024-03-06T12:18:39.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:39.018 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:39.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:39.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:39.515894+0000) 2024-03-06T12:18:40.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:40.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:40.516097+0000) 2024-03-06T12:18:41.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:41.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:41.516252+0000) 2024-03-06T12:18:42.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:42.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:42.516501+0000) 2024-03-06T12:18:43.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:43.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:43.516781+0000) 2024-03-06T12:18:44.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:44.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:44.516978+0000) 2024-03-06T12:18:45.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:45.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:45.517197+0000) 2024-03-06T12:18:46.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:46.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:46.517417+0000) 2024-03-06T12:18:47.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:47.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:47.517639+0000) 2024-03-06T12:18:48.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:48.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:48.517921+0000) 2024-03-06T12:18:49.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:49.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:49.518111+0000) 2024-03-06T12:18:50.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:50.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:50.518314+0000) 2024-03-06T12:18:51.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:51.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:51.518538+0000) 2024-03-06T12:18:52.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:52.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:52.518749+0000) 2024-03-06T12:18:53.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:53.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:53.518966+0000) 2024-03-06T12:18:54.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:54.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:54.519207+0000) 2024-03-06T12:18:55.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:55.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:55.519433+0000) 2024-03-06T12:18:56.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:56.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:56.519605+0000) 2024-03-06T12:18:57.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:57.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:57.519842+0000) 2024-03-06T12:18:58.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:58.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:58.520078+0000) 2024-03-06T12:18:59.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:18:59.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:59.520276+0000) 2024-03-06T12:19:00.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:00.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:00.520515+0000) 2024-03-06T12:19:01.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:01.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:01.520778+0000) 2024-03-06T12:19:02.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:02.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:02.521005+0000) 2024-03-06T12:19:03.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:03.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:03.521201+0000) 2024-03-06T12:19:04.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:04.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:04.521435+0000) 2024-03-06T12:19:05.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:05.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:05.521662+0000) 2024-03-06T12:19:06.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:06.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:06.521914+0000) 2024-03-06T12:19:07.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:07.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:07.522120+0000) 2024-03-06T12:19:08.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:08.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:08.522314+0000) 2024-03-06T12:19:09.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:09.225 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:09.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:09.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:09.522548+0000) 2024-03-06T12:19:10.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:10.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:10.522802+0000) 2024-03-06T12:19:11.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:11.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:11.523015+0000) 2024-03-06T12:19:12.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:12.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:12.523226+0000) 2024-03-06T12:19:13.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:13.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:13.523439+0000) 2024-03-06T12:19:14.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:14.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:14.523713+0000) 2024-03-06T12:19:15.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:15.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:15.523915+0000) 2024-03-06T12:19:16.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:16.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:16.524107+0000) 2024-03-06T12:19:17.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:17.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:17.524270+0000) 2024-03-06T12:19:18.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:18.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:18.524602+0000) 2024-03-06T12:19:19.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:19.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:19.524819+0000) 2024-03-06T12:19:20.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:20.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:20.525011+0000) 2024-03-06T12:19:21.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:21.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:21.525200+0000) 2024-03-06T12:19:22.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:22.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:22.525426+0000) 2024-03-06T12:19:23.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:23.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:23.525635+0000) 2024-03-06T12:19:24.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:24.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:24.525897+0000) 2024-03-06T12:19:25.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:25.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:25.526119+0000) 2024-03-06T12:19:26.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:26.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:26.526260+0000) 2024-03-06T12:19:27.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:27.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:27.526493+0000) 2024-03-06T12:19:28.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:28.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:28.526712+0000) 2024-03-06T12:19:29.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:29.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:29.526915+0000) 2024-03-06T12:19:30.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:30.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:30.527117+0000) 2024-03-06T12:19:31.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:31.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:31.527309+0000) 2024-03-06T12:19:32.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:32.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:32.527534+0000) 2024-03-06T12:19:33.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:33.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:33.527796+0000) 2024-03-06T12:19:34.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:34.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:34.527992+0000) 2024-03-06T12:19:35.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:35.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:35.528217+0000) 2024-03-06T12:19:36.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:36.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:36.528453+0000) 2024-03-06T12:19:37.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:37.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:37.528619+0000) 2024-03-06T12:19:38.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:38.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:38.528868+0000) 2024-03-06T12:19:39.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:39.413 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:39.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:39.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:39.529077+0000) 2024-03-06T12:19:40.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:40.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:40.529293+0000) 2024-03-06T12:19:41.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:41.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:41.529517+0000) 2024-03-06T12:19:42.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:42.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:42.529790+0000) 2024-03-06T12:19:43.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:43.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:43.529989+0000) 2024-03-06T12:19:44.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:44.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:44.530235+0000) 2024-03-06T12:19:45.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:45.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:45.530495+0000) 2024-03-06T12:19:46.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:46.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:46.530646+0000) 2024-03-06T12:19:47.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:47.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:47.530878+0000) 2024-03-06T12:19:48.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:48.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:48.531074+0000) 2024-03-06T12:19:49.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:49.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:49.531259+0000) 2024-03-06T12:19:50.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:50.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:50.531487+0000) 2024-03-06T12:19:51.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:51.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:51.531758+0000) 2024-03-06T12:19:52.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:52.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:52.531953+0000) 2024-03-06T12:19:53.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:53.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:53.532149+0000) 2024-03-06T12:19:54.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:54.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:54.532353+0000) 2024-03-06T12:19:55.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:55.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:55.532590+0000) 2024-03-06T12:19:56.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:56.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:56.532815+0000) 2024-03-06T12:19:57.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:57.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:57.533007+0000) 2024-03-06T12:19:58.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:58.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:58.533206+0000) 2024-03-06T12:19:59.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:19:59.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:59.533427+0000) 2024-03-06T12:20:00.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:00.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:00.533632+0000) 2024-03-06T12:20:01.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:01.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:01.533869+0000) 2024-03-06T12:20:02.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:02.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:02.534063+0000) 2024-03-06T12:20:03.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:03.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:03.534256+0000) 2024-03-06T12:20:04.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:04.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:04.534464+0000) 2024-03-06T12:20:05.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:05.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:05.534637+0000) 2024-03-06T12:20:06.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:06.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:06.534927+0000) 2024-03-06T12:20:07.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:07.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:07.535163+0000) 2024-03-06T12:20:08.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:08.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:08.535405+0000) 2024-03-06T12:20:09.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:09.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:09.535632+0000) 2024-03-06T12:20:09.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:09.549 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:10.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:10.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:10.535885+0000) 2024-03-06T12:20:11.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:11.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:11.536060+0000) 2024-03-06T12:20:12.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:12.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:12.536261+0000) 2024-03-06T12:20:13.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:13.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:13.536426+0000) 2024-03-06T12:20:14.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:14.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:14.536699+0000) 2024-03-06T12:20:15.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:15.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:15.536880+0000) 2024-03-06T12:20:16.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:16.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:16.537084+0000) 2024-03-06T12:20:17.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:17.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:17.537282+0000) 2024-03-06T12:20:18.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:18.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:18.537481+0000) 2024-03-06T12:20:19.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:19.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:19.537744+0000) 2024-03-06T12:20:20.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:20.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:20.537965+0000) 2024-03-06T12:20:21.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:21.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:21.538192+0000) 2024-03-06T12:20:22.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:22.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:22.538420+0000) 2024-03-06T12:20:23.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:23.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:23.538612+0000) 2024-03-06T12:20:24.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:24.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:24.538820+0000) 2024-03-06T12:20:25.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:25.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:25.539008+0000) 2024-03-06T12:20:26.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:26.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:26.539197+0000) 2024-03-06T12:20:27.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:27.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:27.539402+0000) 2024-03-06T12:20:28.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:28.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:28.539638+0000) 2024-03-06T12:20:29.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:29.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:29.539814+0000) 2024-03-06T12:20:30.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:30.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:30.539995+0000) 2024-03-06T12:20:31.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:31.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:31.540191+0000) 2024-03-06T12:20:32.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:32.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:32.540421+0000) 2024-03-06T12:20:33.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:33.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:33.540656+0000) 2024-03-06T12:20:34.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:34.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:34.540917+0000) 2024-03-06T12:20:35.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:35.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:35.541108+0000) 2024-03-06T12:20:36.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:36.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:36.541337+0000) 2024-03-06T12:20:37.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:37.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:37.541615+0000) 2024-03-06T12:20:38.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:38.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:38.541862+0000) 2024-03-06T12:20:39.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:39.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:39.542068+0000) 2024-03-06T12:20:39.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:39.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:40.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:40.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:40.542286+0000) 2024-03-06T12:20:41.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:41.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:41.542548+0000) 2024-03-06T12:20:42.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:42.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:42.542840+0000) 2024-03-06T12:20:43.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:43.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:43.543090+0000) 2024-03-06T12:20:44.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:44.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:44.543258+0000) 2024-03-06T12:20:45.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:45.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:45.543490+0000) 2024-03-06T12:20:46.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:46.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:46.543770+0000) 2024-03-06T12:20:47.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:47.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:47.543998+0000) 2024-03-06T12:20:48.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:48.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:48.544246+0000) 2024-03-06T12:20:49.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:49.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:49.544483+0000) 2024-03-06T12:20:50.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:50.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:50.544761+0000) 2024-03-06T12:20:51.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:51.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:51.544947+0000) 2024-03-06T12:20:52.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:52.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:52.545111+0000) 2024-03-06T12:20:53.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:53.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:53.545349+0000) 2024-03-06T12:20:54.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:54.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:54.545574+0000) 2024-03-06T12:20:55.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:55.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:55.545773+0000) 2024-03-06T12:20:56.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:56.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:56.545999+0000) 2024-03-06T12:20:57.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:57.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:57.546230+0000) 2024-03-06T12:20:58.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:58.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:58.546472+0000) 2024-03-06T12:20:59.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:20:59.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:59.546794+0000) 2024-03-06T12:21:00.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:00.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:00.546987+0000) 2024-03-06T12:21:01.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:01.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:01.547138+0000) 2024-03-06T12:21:02.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:02.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:02.547371+0000) 2024-03-06T12:21:03.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:03.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:03.547600+0000) 2024-03-06T12:21:04.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:04.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:04.547829+0000) 2024-03-06T12:21:05.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:05.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:05.548017+0000) 2024-03-06T12:21:06.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:06.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:06.548217+0000) 2024-03-06T12:21:07.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:07.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:07.548434+0000) 2024-03-06T12:21:08.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:08.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:08.548609+0000) 2024-03-06T12:21:09.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:09.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:09.548819+0000) 2024-03-06T12:21:09.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:09.941 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:10.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:10.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:10.549010+0000) 2024-03-06T12:21:11.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:11.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:11.549195+0000) 2024-03-06T12:21:12.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:12.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:12.549426+0000) 2024-03-06T12:21:13.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:13.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:13.549656+0000) 2024-03-06T12:21:14.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:14.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:14.549875+0000) 2024-03-06T12:21:15.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:15.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:15.550066+0000) 2024-03-06T12:21:16.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:16.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:16.550251+0000) 2024-03-06T12:21:17.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:17.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:17.550488+0000) 2024-03-06T12:21:18.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:18.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:18.550768+0000) 2024-03-06T12:21:19.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:19.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:19.550958+0000) 2024-03-06T12:21:20.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:20.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:20.551161+0000) 2024-03-06T12:21:21.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:21.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:21.551383+0000) 2024-03-06T12:21:22.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:22.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:22.551522+0000) 2024-03-06T12:21:23.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:23.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:23.551799+0000) 2024-03-06T12:21:24.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:24.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:24.551988+0000) 2024-03-06T12:21:25.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:25.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:25.552143+0000) 2024-03-06T12:21:26.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:26.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:26.552386+0000) 2024-03-06T12:21:27.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:27.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:27.552631+0000) 2024-03-06T12:21:28.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:28.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:28.552811+0000) 2024-03-06T12:21:29.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:29.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:29.553019+0000) 2024-03-06T12:21:30.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:30.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:30.553172+0000) 2024-03-06T12:21:31.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:31.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:31.553406+0000) 2024-03-06T12:21:32.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:32.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:32.553631+0000) 2024-03-06T12:21:33.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:33.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:33.553847+0000) 2024-03-06T12:21:34.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:34.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:34.554070+0000) 2024-03-06T12:21:35.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:35.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:35.554306+0000) 2024-03-06T12:21:36.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:36.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:36.554530+0000) 2024-03-06T12:21:37.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:37.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:37.554708+0000) 2024-03-06T12:21:38.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:38.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:38.554950+0000) 2024-03-06T12:21:39.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:39.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:39.555181+0000) 2024-03-06T12:21:40.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:40.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:40.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:40.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:40.555329+0000) 2024-03-06T12:21:41.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:41.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:41.555474+0000) 2024-03-06T12:21:42.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:42.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:42.555746+0000) 2024-03-06T12:21:43.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:43.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:43.555958+0000) 2024-03-06T12:21:44.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:44.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:44.556166+0000) 2024-03-06T12:21:45.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:45.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:45.556392+0000) 2024-03-06T12:21:46.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:46.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:46.556613+0000) 2024-03-06T12:21:47.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:47.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:47.556902+0000) 2024-03-06T12:21:48.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:48.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:48.557118+0000) 2024-03-06T12:21:49.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:49.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:49.557354+0000) 2024-03-06T12:21:50.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:50.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:50.557592+0000) 2024-03-06T12:21:51.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:51.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:51.557789+0000) 2024-03-06T12:21:52.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:52.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:52.557989+0000) 2024-03-06T12:21:53.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:53.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:53.558190+0000) 2024-03-06T12:21:54.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:54.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:54.558419+0000) 2024-03-06T12:21:55.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:55.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:55.558643+0000) 2024-03-06T12:21:56.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:56.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:56.558859+0000) 2024-03-06T12:21:57.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:57.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:57.559066+0000) 2024-03-06T12:21:58.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:58.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:58.559263+0000) 2024-03-06T12:21:59.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:21:59.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:59.559490+0000) 2024-03-06T12:22:00.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:00.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:00.559757+0000) 2024-03-06T12:22:01.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:01.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:01.559941+0000) 2024-03-06T12:22:02.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:02.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:02.560135+0000) 2024-03-06T12:22:03.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:03.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:03.560353+0000) 2024-03-06T12:22:04.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:04.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:04.560600+0000) 2024-03-06T12:22:05.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:05.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:05.560804+0000) 2024-03-06T12:22:06.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:06.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:06.561030+0000) 2024-03-06T12:22:07.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:07.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:07.561227+0000) 2024-03-06T12:22:08.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:08.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:08.561477+0000) 2024-03-06T12:22:09.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:09.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:09.561698+0000) 2024-03-06T12:22:10.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:10.498 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:10.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:10.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:10.561906+0000) 2024-03-06T12:22:11.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:11.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:11.562093+0000) 2024-03-06T12:22:12.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:12.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:12.562299+0000) 2024-03-06T12:22:13.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:13.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:13.562500+0000) 2024-03-06T12:22:14.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:14.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:14.562776+0000) 2024-03-06T12:22:15.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:15.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:15.562956+0000) 2024-03-06T12:22:16.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:16.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:16.563179+0000) 2024-03-06T12:22:17.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:17.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:17.563420+0000) 2024-03-06T12:22:18.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:18.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:18.563704+0000) 2024-03-06T12:22:19.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:19.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:19.563955+0000) 2024-03-06T12:22:20.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:20.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:20.564196+0000) 2024-03-06T12:22:21.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:21.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:21.564353+0000) 2024-03-06T12:22:22.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:22.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:22.564573+0000) 2024-03-06T12:22:23.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:23.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:23.564817+0000) 2024-03-06T12:22:24.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:24.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:24.565016+0000) 2024-03-06T12:22:25.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:25.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:25.565166+0000) 2024-03-06T12:22:26.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:26.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:26.565395+0000) 2024-03-06T12:22:27.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:27.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:27.565555+0000) 2024-03-06T12:22:28.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:28.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:28.565820+0000) 2024-03-06T12:22:29.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:29.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:29.566012+0000) 2024-03-06T12:22:30.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:30.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:30.566248+0000) 2024-03-06T12:22:31.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:31.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:31.566388+0000) 2024-03-06T12:22:32.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:32.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:32.566629+0000) 2024-03-06T12:22:33.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:33.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:33.566858+0000) 2024-03-06T12:22:34.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:34.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:34.567063+0000) 2024-03-06T12:22:35.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:35.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:35.567215+0000) 2024-03-06T12:22:36.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:36.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:36.567445+0000) 2024-03-06T12:22:37.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:37.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:37.567582+0000) 2024-03-06T12:22:38.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:38.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:38.567869+0000) 2024-03-06T12:22:39.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:39.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:39.568028+0000) 2024-03-06T12:22:40.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:40.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:40.568245+0000) 2024-03-06T12:22:40.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:40.704 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:41.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:41.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:41.568438+0000) 2024-03-06T12:22:42.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:42.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:42.568705+0000) 2024-03-06T12:22:43.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:43.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:43.568904+0000) 2024-03-06T12:22:44.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:44.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:44.569097+0000) 2024-03-06T12:22:45.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:45.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:45.569319+0000) 2024-03-06T12:22:46.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:46.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:46.569599+0000) 2024-03-06T12:22:47.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:47.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:47.569842+0000) 2024-03-06T12:22:48.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:48.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:48.570056+0000) 2024-03-06T12:22:49.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:49.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:49.570269+0000) 2024-03-06T12:22:50.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:50.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:50.570504+0000) 2024-03-06T12:22:51.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:51.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:51.570760+0000) 2024-03-06T12:22:52.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:52.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:52.570961+0000) 2024-03-06T12:22:53.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:53.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:53.571161+0000) 2024-03-06T12:22:54.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:54.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:54.571393+0000) 2024-03-06T12:22:55.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:55.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:55.571617+0000) 2024-03-06T12:22:56.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:56.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:56.571791+0000) 2024-03-06T12:22:57.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:57.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:57.571994+0000) 2024-03-06T12:22:58.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:58.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:58.572200+0000) 2024-03-06T12:22:59.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:22:59.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:59.572414+0000) 2024-03-06T12:23:00.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:00.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:00.572644+0000) 2024-03-06T12:23:01.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:01.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:01.572861+0000) 2024-03-06T12:23:02.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:02.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:02.573092+0000) 2024-03-06T12:23:03.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:03.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:03.573280+0000) 2024-03-06T12:23:04.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:04.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:04.573485+0000) 2024-03-06T12:23:05.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:05.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:05.573747+0000) 2024-03-06T12:23:06.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:06.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:06.573938+0000) 2024-03-06T12:23:07.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:07.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:07.574176+0000) 2024-03-06T12:23:08.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:08.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:08.574412+0000) 2024-03-06T12:23:09.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:09.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:09.574637+0000) 2024-03-06T12:23:10.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:10.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:10.574845+0000) 2024-03-06T12:23:10.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:10.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:11.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:11.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:11.575042+0000) 2024-03-06T12:23:12.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:12.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:12.575191+0000) 2024-03-06T12:23:13.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:13.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:13.575440+0000) 2024-03-06T12:23:14.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:14.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:14.575702+0000) 2024-03-06T12:23:15.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:15.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:15.575875+0000) 2024-03-06T12:23:16.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:16.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:16.576078+0000) 2024-03-06T12:23:17.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:17.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:17.576281+0000) 2024-03-06T12:23:18.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:18.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:18.576516+0000) 2024-03-06T12:23:19.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:19.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:19.576789+0000) 2024-03-06T12:23:20.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:20.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:20.576966+0000) 2024-03-06T12:23:21.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:21.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:21.577195+0000) 2024-03-06T12:23:22.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:22.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:22.577441+0000) 2024-03-06T12:23:23.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:23.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:23.577734+0000) 2024-03-06T12:23:24.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:24.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:24.577941+0000) 2024-03-06T12:23:25.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:25.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:25.578143+0000) 2024-03-06T12:23:26.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:26.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:26.578372+0000) 2024-03-06T12:23:27.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:27.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:27.578605+0000) 2024-03-06T12:23:28.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:28.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:28.578863+0000) 2024-03-06T12:23:29.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:29.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:29.579074+0000) 2024-03-06T12:23:30.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:30.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:30.579309+0000) 2024-03-06T12:23:31.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:31.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:31.579535+0000) 2024-03-06T12:23:32.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:32.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:32.579716+0000) 2024-03-06T12:23:33.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:33.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:33.579929+0000) 2024-03-06T12:23:34.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:34.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:34.580176+0000) 2024-03-06T12:23:35.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:35.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:35.580328+0000) 2024-03-06T12:23:36.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:36.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:36.580567+0000) 2024-03-06T12:23:37.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:37.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:37.580819+0000) 2024-03-06T12:23:38.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:38.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:38.581033+0000) 2024-03-06T12:23:39.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:39.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:39.581265+0000) 2024-03-06T12:23:40.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:40.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:40.581493+0000) 2024-03-06T12:23:41.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:41.128 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:41.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:41.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:41.581697+0000) 2024-03-06T12:23:42.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:42.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:42.581901+0000) 2024-03-06T12:23:43.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:43.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:43.582130+0000) 2024-03-06T12:23:44.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:44.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:44.582387+0000) 2024-03-06T12:23:45.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:45.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:45.582574+0000) 2024-03-06T12:23:46.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:46.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:46.582779+0000) 2024-03-06T12:23:47.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:47.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:47.583023+0000) 2024-03-06T12:23:48.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:48.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:48.583213+0000) 2024-03-06T12:23:49.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:49.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:49.583411+0000) 2024-03-06T12:23:50.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:50.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:50.583649+0000) 2024-03-06T12:23:51.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:51.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:51.583871+0000) 2024-03-06T12:23:52.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:52.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:52.584062+0000) 2024-03-06T12:23:53.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:53.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:53.584249+0000) 2024-03-06T12:23:54.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:54.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:54.584478+0000) 2024-03-06T12:23:55.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:55.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:55.584733+0000) 2024-03-06T12:23:56.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:56.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:56.584911+0000) 2024-03-06T12:23:57.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:57.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:57.585104+0000) 2024-03-06T12:23:58.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:58.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:58.585337+0000) 2024-03-06T12:23:59.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:23:59.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:59.585531+0000) 2024-03-06T12:24:00.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:00.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:00.585776+0000) 2024-03-06T12:24:01.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:01.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:01.585962+0000) 2024-03-06T12:24:02.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:02.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:02.586210+0000) 2024-03-06T12:24:03.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:03.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:03.586449+0000) 2024-03-06T12:24:04.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:04.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:04.586633+0000) 2024-03-06T12:24:05.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:05.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:05.586860+0000) 2024-03-06T12:24:06.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:06.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:06.587004+0000) 2024-03-06T12:24:07.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:07.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:07.587205+0000) 2024-03-06T12:24:08.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:08.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:08.587379+0000) 2024-03-06T12:24:09.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:09.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:09.587616+0000) 2024-03-06T12:24:10.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:10.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:10.587840+0000) 2024-03-06T12:24:11.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:11.358 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:11.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:11.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:11.588048+0000) 2024-03-06T12:24:12.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:12.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:12.588279+0000) 2024-03-06T12:24:13.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:13.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:13.588535+0000) 2024-03-06T12:24:14.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:14.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:14.588817+0000) 2024-03-06T12:24:15.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:15.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:15.588949+0000) 2024-03-06T12:24:16.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:16.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:16.589155+0000) 2024-03-06T12:24:17.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:17.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:17.589362+0000) 2024-03-06T12:24:18.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:18.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:18.589604+0000) 2024-03-06T12:24:19.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:19.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:19.589809+0000) 2024-03-06T12:24:20.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:20.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:20.589995+0000) 2024-03-06T12:24:21.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:21.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:21.590217+0000) 2024-03-06T12:24:22.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:22.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:22.590541+0000) 2024-03-06T12:24:23.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:23.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:23.590836+0000) 2024-03-06T12:24:24.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:24.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:24.591081+0000) 2024-03-06T12:24:25.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:25.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:25.591335+0000) 2024-03-06T12:24:26.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:26.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:26.591581+0000) 2024-03-06T12:24:27.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:27.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:27.591792+0000) 2024-03-06T12:24:28.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:28.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:28.591989+0000) 2024-03-06T12:24:29.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:29.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:29.592168+0000) 2024-03-06T12:24:30.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:30.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:30.592393+0000) 2024-03-06T12:24:31.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:31.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:31.592608+0000) 2024-03-06T12:24:32.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:32.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:32.592819+0000) 2024-03-06T12:24:33.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:33.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:33.593013+0000) 2024-03-06T12:24:34.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:34.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:34.593156+0000) 2024-03-06T12:24:35.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:35.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:35.593388+0000) 2024-03-06T12:24:36.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:36.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:36.593600+0000) 2024-03-06T12:24:37.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:37.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:37.593851+0000) 2024-03-06T12:24:38.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:38.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:38.594079+0000) 2024-03-06T12:24:39.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:39.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:39.594287+0000) 2024-03-06T12:24:40.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:40.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:40.594552+0000) 2024-03-06T12:24:41.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:41.579 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:41.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:41.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:41.594831+0000) 2024-03-06T12:24:42.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:42.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:42.595066+0000) 2024-03-06T12:24:43.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:43.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:43.595307+0000) 2024-03-06T12:24:44.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:44.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:44.595536+0000) 2024-03-06T12:24:45.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:45.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:45.595778+0000) 2024-03-06T12:24:46.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:46.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:46.595990+0000) 2024-03-06T12:24:47.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:47.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:47.596226+0000) 2024-03-06T12:24:48.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:48.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:48.596481+0000) 2024-03-06T12:24:49.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:49.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:49.596654+0000) 2024-03-06T12:24:50.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:50.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:50.596898+0000) 2024-03-06T12:24:51.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:51.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:51.597123+0000) 2024-03-06T12:24:52.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:52.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:52.597375+0000) 2024-03-06T12:24:53.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:53.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:53.597655+0000) 2024-03-06T12:24:54.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:54.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:54.597962+0000) 2024-03-06T12:24:55.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:55.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:55.598183+0000) 2024-03-06T12:24:56.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:56.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:56.598468+0000) 2024-03-06T12:24:57.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:57.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:57.598799+0000) 2024-03-06T12:24:58.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:58.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:58.599048+0000) 2024-03-06T12:24:59.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:24:59.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:59.599281+0000) 2024-03-06T12:25:00.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:00.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:00.599557+0000) 2024-03-06T12:25:01.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:01.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:01.599861+0000) 2024-03-06T12:25:02.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:02.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:02.600071+0000) 2024-03-06T12:25:03.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:03.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:03.600261+0000) 2024-03-06T12:25:04.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:04.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:04.600488+0000) 2024-03-06T12:25:05.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:05.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:05.600702+0000) 2024-03-06T12:25:06.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:06.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:06.600897+0000) 2024-03-06T12:25:07.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:07.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:07.601092+0000) 2024-03-06T12:25:08.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:08.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:08.601325+0000) 2024-03-06T12:25:09.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:09.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:09.601553+0000) 2024-03-06T12:25:10.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:10.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:10.601792+0000) 2024-03-06T12:25:11.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:11.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:11.602002+0000) 2024-03-06T12:25:11.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:11.715 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:12.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:12.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:12.602209+0000) 2024-03-06T12:25:13.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:13.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:13.602438+0000) 2024-03-06T12:25:14.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:14.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:14.602697+0000) 2024-03-06T12:25:15.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:15.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:15.602882+0000) 2024-03-06T12:25:16.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:16.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:16.603068+0000) 2024-03-06T12:25:17.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:17.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:17.603268+0000) 2024-03-06T12:25:18.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:18.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:18.603517+0000) 2024-03-06T12:25:19.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:19.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:19.603770+0000) 2024-03-06T12:25:20.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:20.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:20.604057+0000) 2024-03-06T12:25:21.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:21.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:21.604296+0000) 2024-03-06T12:25:22.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:22.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:22.604525+0000) 2024-03-06T12:25:23.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:23.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:23.604725+0000) 2024-03-06T12:25:24.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:24.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:24.604920+0000) 2024-03-06T12:25:25.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:25.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:25.605108+0000) 2024-03-06T12:25:26.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:26.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:26.605332+0000) 2024-03-06T12:25:27.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:27.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:27.605566+0000) 2024-03-06T12:25:28.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:28.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:28.605808+0000) 2024-03-06T12:25:29.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:29.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:29.606023+0000) 2024-03-06T12:25:30.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:30.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:30.606221+0000) 2024-03-06T12:25:31.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:31.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:31.606446+0000) 2024-03-06T12:25:32.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:32.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:32.606695+0000) 2024-03-06T12:25:33.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:33.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:33.606890+0000) 2024-03-06T12:25:34.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:34.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:34.607098+0000) 2024-03-06T12:25:35.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:35.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:35.607299+0000) 2024-03-06T12:25:36.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:36.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:36.607531+0000) 2024-03-06T12:25:37.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:37.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:37.607785+0000) 2024-03-06T12:25:38.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:38.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:38.607987+0000) 2024-03-06T12:25:39.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:39.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:39.608178+0000) 2024-03-06T12:25:40.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:40.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:40.608411+0000) 2024-03-06T12:25:41.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:41.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:41.608629+0000) 2024-03-06T12:25:41.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:41.927 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:42.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:42.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:42.608795+0000) 2024-03-06T12:25:43.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:43.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:43.608968+0000) 2024-03-06T12:25:44.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:44.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:44.609212+0000) 2024-03-06T12:25:45.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:45.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:45.609442+0000) 2024-03-06T12:25:46.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:46.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:46.609706+0000) 2024-03-06T12:25:47.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:47.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:47.609943+0000) 2024-03-06T12:25:48.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:48.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:48.610175+0000) 2024-03-06T12:25:49.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:49.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:49.610405+0000) 2024-03-06T12:25:50.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:50.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:50.610639+0000) 2024-03-06T12:25:51.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:51.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:51.610907+0000) 2024-03-06T12:25:52.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:52.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:52.611140+0000) 2024-03-06T12:25:53.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:53.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:53.611307+0000) 2024-03-06T12:25:54.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:54.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:54.611513+0000) 2024-03-06T12:25:55.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:55.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:55.611744+0000) 2024-03-06T12:25:56.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:56.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:56.611949+0000) 2024-03-06T12:25:57.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:57.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:57.612199+0000) 2024-03-06T12:25:58.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:58.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:58.612399+0000) 2024-03-06T12:25:59.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:25:59.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:59.612563+0000) 2024-03-06T12:26:00.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:00.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:00.612812+0000) 2024-03-06T12:26:01.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:01.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:01.613003+0000) 2024-03-06T12:26:02.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:02.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:02.613235+0000) 2024-03-06T12:26:03.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:03.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:03.613471+0000) 2024-03-06T12:26:04.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:04.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:04.613695+0000) 2024-03-06T12:26:05.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:05.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:05.613900+0000) 2024-03-06T12:26:06.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:06.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:06.614095+0000) 2024-03-06T12:26:07.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:07.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:07.614340+0000) 2024-03-06T12:26:08.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:08.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:08.614578+0000) 2024-03-06T12:26:09.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:09.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:09.614815+0000) 2024-03-06T12:26:10.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:10.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:10.615020+0000) 2024-03-06T12:26:11.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:11.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:11.615238+0000) 2024-03-06T12:26:12.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:12.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:12.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:12.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:12.615478+0000) 2024-03-06T12:26:13.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:13.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:13.615757+0000) 2024-03-06T12:26:14.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:14.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:14.616015+0000) 2024-03-06T12:26:15.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:15.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:15.616242+0000) 2024-03-06T12:26:16.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:16.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:16.616424+0000) 2024-03-06T12:26:17.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:17.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:17.616689+0000) 2024-03-06T12:26:18.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:18.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:18.616881+0000) 2024-03-06T12:26:19.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:19.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:19.617088+0000) 2024-03-06T12:26:20.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:20.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:20.617306+0000) 2024-03-06T12:26:21.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:21.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:21.617539+0000) 2024-03-06T12:26:22.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:22.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:22.617781+0000) 2024-03-06T12:26:23.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:23.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:23.618027+0000) 2024-03-06T12:26:24.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:24.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:24.618244+0000) 2024-03-06T12:26:25.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:25.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:25.618497+0000) 2024-03-06T12:26:26.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:26.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:26.618738+0000) 2024-03-06T12:26:27.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:27.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:27.618980+0000) 2024-03-06T12:26:28.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:28.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:28.619223+0000) 2024-03-06T12:26:29.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:29.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:29.619449+0000) 2024-03-06T12:26:30.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:30.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:30.619741+0000) 2024-03-06T12:26:31.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:31.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:31.619985+0000) 2024-03-06T12:26:32.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:32.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:32.620224+0000) 2024-03-06T12:26:33.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:33.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:33.620473+0000) 2024-03-06T12:26:34.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:34.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:34.620762+0000) 2024-03-06T12:26:35.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:35.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:35.620974+0000) 2024-03-06T12:26:36.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:36.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:36.621185+0000) 2024-03-06T12:26:37.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:37.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:37.621436+0000) 2024-03-06T12:26:38.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:38.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:38.621699+0000) 2024-03-06T12:26:39.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:39.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:39.621921+0000) 2024-03-06T12:26:40.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:40.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:40.622151+0000) 2024-03-06T12:26:41.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:41.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:41.622381+0000) 2024-03-06T12:26:42.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:42.346 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:42.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:42.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:42.622570+0000) 2024-03-06T12:26:43.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:43.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:43.622806+0000) 2024-03-06T12:26:44.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:44.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:44.623033+0000) 2024-03-06T12:26:45.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:45.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:45.623255+0000) 2024-03-06T12:26:46.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:46.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:46.623480+0000) 2024-03-06T12:26:47.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:47.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:47.623763+0000) 2024-03-06T12:26:48.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:48.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:48.623966+0000) 2024-03-06T12:26:49.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:49.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:49.624191+0000) 2024-03-06T12:26:50.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:50.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:50.624444+0000) 2024-03-06T12:26:51.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:51.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:51.624647+0000) 2024-03-06T12:26:52.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:52.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:52.624890+0000) 2024-03-06T12:26:53.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:53.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:53.625139+0000) 2024-03-06T12:26:54.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:54.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:54.625369+0000) 2024-03-06T12:26:55.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:55.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:55.625587+0000) 2024-03-06T12:26:56.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:56.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:56.625817+0000) 2024-03-06T12:26:57.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:57.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:57.626020+0000) 2024-03-06T12:26:58.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:58.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:58.626310+0000) 2024-03-06T12:26:59.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:26:59.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:59.626556+0000) 2024-03-06T12:27:00.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:00.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:00.626784+0000) 2024-03-06T12:27:01.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:01.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:01.626975+0000) 2024-03-06T12:27:02.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:02.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:02.627241+0000) 2024-03-06T12:27:03.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:03.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:03.627455+0000) 2024-03-06T12:27:04.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:04.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:04.627727+0000) 2024-03-06T12:27:05.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:05.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:05.627886+0000) 2024-03-06T12:27:06.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:06.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:06.628071+0000) 2024-03-06T12:27:07.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:07.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:07.628316+0000) 2024-03-06T12:27:08.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:08.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:08.628557+0000) 2024-03-06T12:27:09.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:09.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:09.628808+0000) 2024-03-06T12:27:10.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:10.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:10.628995+0000) 2024-03-06T12:27:11.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:11.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:11.629177+0000) 2024-03-06T12:27:12.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:12.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:12.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:12.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:12.629421+0000) 2024-03-06T12:27:13.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:13.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:13.629709+0000) 2024-03-06T12:27:14.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:14.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:14.629906+0000) 2024-03-06T12:27:15.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:15.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:15.630095+0000) 2024-03-06T12:27:16.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:16.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:16.630310+0000) 2024-03-06T12:27:17.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:17.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:17.630559+0000) 2024-03-06T12:27:18.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:18.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:18.630809+0000) 2024-03-06T12:27:19.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:19.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:19.631026+0000) 2024-03-06T12:27:20.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:20.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:20.631275+0000) 2024-03-06T12:27:21.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:21.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:21.631510+0000) 2024-03-06T12:27:22.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:22.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:22.631775+0000) 2024-03-06T12:27:23.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:23.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:23.631978+0000) 2024-03-06T12:27:24.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:24.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:24.632210+0000) 2024-03-06T12:27:25.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:25.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:25.632440+0000) 2024-03-06T12:27:26.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:26.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:26.632618+0000) 2024-03-06T12:27:27.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:27.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:27.632858+0000) 2024-03-06T12:27:28.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:28.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:28.633094+0000) 2024-03-06T12:27:29.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:29.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:29.633309+0000) 2024-03-06T12:27:30.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:30.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:30.633557+0000) 2024-03-06T12:27:31.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:31.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:31.633788+0000) 2024-03-06T12:27:32.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:32.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:32.633980+0000) 2024-03-06T12:27:33.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:33.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:33.634214+0000) 2024-03-06T12:27:34.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:34.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:34.634455+0000) 2024-03-06T12:27:35.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:35.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:35.634720+0000) 2024-03-06T12:27:36.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:36.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:36.634919+0000) 2024-03-06T12:27:37.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:37.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:37.635151+0000) 2024-03-06T12:27:38.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:38.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:38.635347+0000) 2024-03-06T12:27:39.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:39.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:39.635585+0000) 2024-03-06T12:27:40.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:40.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:40.635835+0000) 2024-03-06T12:27:41.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:41.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:41.636019+0000) 2024-03-06T12:27:42.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:42.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:42.636255+0000) 2024-03-06T12:27:42.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:42.814 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:43.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:43.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:43.636476+0000) 2024-03-06T12:27:44.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:44.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:44.636804+0000) 2024-03-06T12:27:45.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:45.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:45.637055+0000) 2024-03-06T12:27:46.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:46.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:46.637321+0000) 2024-03-06T12:27:47.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:47.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:47.637572+0000) 2024-03-06T12:27:48.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:48.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:48.637791+0000) 2024-03-06T12:27:49.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:49.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:49.637970+0000) 2024-03-06T12:27:50.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:50.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:50.638257+0000) 2024-03-06T12:27:51.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:51.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:51.638487+0000) 2024-03-06T12:27:52.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:52.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:52.638775+0000) 2024-03-06T12:27:53.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:53.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:53.639017+0000) 2024-03-06T12:27:54.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:54.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:54.639268+0000) 2024-03-06T12:27:55.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:55.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:55.639453+0000) 2024-03-06T12:27:56.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:56.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:56.639695+0000) 2024-03-06T12:27:57.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:57.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:57.639935+0000) 2024-03-06T12:27:58.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:58.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:58.640164+0000) 2024-03-06T12:27:59.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:27:59.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:59.640399+0000) 2024-03-06T12:28:00.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:00.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:00.640628+0000) 2024-03-06T12:28:01.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:01.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:01.640858+0000) 2024-03-06T12:28:02.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:02.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:02.641045+0000) 2024-03-06T12:28:03.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:03.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:03.641272+0000) 2024-03-06T12:28:04.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:04.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:04.641465+0000) 2024-03-06T12:28:05.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:05.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:05.641650+0000) 2024-03-06T12:28:06.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:06.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:06.641926+0000) 2024-03-06T12:28:07.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:07.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:07.642152+0000) 2024-03-06T12:28:08.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:08.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:08.642367+0000) 2024-03-06T12:28:09.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:09.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:09.642600+0000) 2024-03-06T12:28:10.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:10.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:10.642847+0000) 2024-03-06T12:28:11.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:11.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:11.643067+0000) 2024-03-06T12:28:12.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:12.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:12.643292+0000) 2024-03-06T12:28:13.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:13.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:13.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:13.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:13.643520+0000) 2024-03-06T12:28:14.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:14.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:14.643796+0000) 2024-03-06T12:28:15.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:15.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:15.643996+0000) 2024-03-06T12:28:16.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:16.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:16.644217+0000) 2024-03-06T12:28:17.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:17.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:17.644449+0000) 2024-03-06T12:28:18.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:18.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:18.644715+0000) 2024-03-06T12:28:19.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:19.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:19.644965+0000) 2024-03-06T12:28:20.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:20.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:20.645152+0000) 2024-03-06T12:28:21.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:21.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:21.645367+0000) 2024-03-06T12:28:22.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:22.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:22.645588+0000) 2024-03-06T12:28:23.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:23.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:23.645825+0000) 2024-03-06T12:28:24.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:24.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:24.646067+0000) 2024-03-06T12:28:25.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:25.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:25.646252+0000) 2024-03-06T12:28:26.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:26.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:26.646482+0000) 2024-03-06T12:28:27.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:27.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:27.646712+0000) 2024-03-06T12:28:28.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:28.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:28.646906+0000) 2024-03-06T12:28:29.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:29.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:29.647111+0000) 2024-03-06T12:28:30.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:30.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:30.647337+0000) 2024-03-06T12:28:31.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:31.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:31.647549+0000) 2024-03-06T12:28:32.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:32.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:32.647773+0000) 2024-03-06T12:28:33.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:33.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:33.648004+0000) 2024-03-06T12:28:34.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:34.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:34.648235+0000) 2024-03-06T12:28:35.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:35.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:35.648411+0000) 2024-03-06T12:28:36.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:36.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:36.648629+0000) 2024-03-06T12:28:37.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:37.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:37.648893+0000) 2024-03-06T12:28:38.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:38.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:38.649351+0000) 2024-03-06T12:28:39.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:39.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:39.649620+0000) 2024-03-06T12:28:40.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:40.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:40.649852+0000) 2024-03-06T12:28:41.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:41.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:41.650025+0000) 2024-03-06T12:28:42.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:42.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:42.650251+0000) 2024-03-06T12:28:43.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:43.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:43.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:43.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:43.650497+0000) 2024-03-06T12:28:44.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:44.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:44.650633+0000) 2024-03-06T12:28:45.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:45.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:45.650843+0000) 2024-03-06T12:28:46.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:46.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:46.651002+0000) 2024-03-06T12:28:47.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:47.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:47.651230+0000) 2024-03-06T12:28:48.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:48.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:48.651533+0000) 2024-03-06T12:28:49.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:49.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:49.651807+0000) 2024-03-06T12:28:50.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:50.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:50.652009+0000) 2024-03-06T12:28:51.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:51.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:51.652232+0000) 2024-03-06T12:28:52.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:52.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:52.652459+0000) 2024-03-06T12:28:53.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:53.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:53.652641+0000) 2024-03-06T12:28:54.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:54.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:54.652916+0000) 2024-03-06T12:28:55.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:55.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:55.653103+0000) 2024-03-06T12:28:56.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:56.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:56.653325+0000) 2024-03-06T12:28:57.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:57.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:57.653573+0000) 2024-03-06T12:28:58.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:58.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:58.653812+0000) 2024-03-06T12:28:59.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:28:59.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:59.654061+0000) 2024-03-06T12:29:00.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:00.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:00.654257+0000) 2024-03-06T12:29:01.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:01.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:01.654457+0000) 2024-03-06T12:29:02.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:02.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:02.654734+0000) 2024-03-06T12:29:03.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:03.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:03.654970+0000) 2024-03-06T12:29:04.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:04.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:04.655151+0000) 2024-03-06T12:29:05.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:05.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:05.655377+0000) 2024-03-06T12:29:06.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:06.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:06.655605+0000) 2024-03-06T12:29:07.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:07.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:07.655833+0000) 2024-03-06T12:29:08.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:08.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:08.656060+0000) 2024-03-06T12:29:09.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:09.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:09.656296+0000) 2024-03-06T12:29:10.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:10.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:10.656524+0000) 2024-03-06T12:29:11.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:11.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:11.656778+0000) 2024-03-06T12:29:12.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:12.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:12.657015+0000) 2024-03-06T12:29:13.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:13.473 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:13.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:13.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:13.657260+0000) 2024-03-06T12:29:14.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:14.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:14.657457+0000) 2024-03-06T12:29:15.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:15.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:15.657648+0000) 2024-03-06T12:29:16.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:16.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:16.657938+0000) 2024-03-06T12:29:17.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:17.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:17.658190+0000) 2024-03-06T12:29:18.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:18.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:18.658442+0000) 2024-03-06T12:29:19.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:19.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:19.658726+0000) 2024-03-06T12:29:20.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:20.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:20.658976+0000) 2024-03-06T12:29:21.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:21.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:21.659164+0000) 2024-03-06T12:29:22.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:22.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:22.659302+0000) 2024-03-06T12:29:23.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:23.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:23.659526+0000) 2024-03-06T12:29:24.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:24.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:24.659759+0000) 2024-03-06T12:29:25.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:25.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:25.659982+0000) 2024-03-06T12:29:26.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:26.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:26.660217+0000) 2024-03-06T12:29:27.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:27.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:27.660464+0000) 2024-03-06T12:29:28.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:28.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:28.660752+0000) 2024-03-06T12:29:29.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:29.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:29.661009+0000) 2024-03-06T12:29:30.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:30.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:30.661231+0000) 2024-03-06T12:29:31.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:31.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:31.661463+0000) 2024-03-06T12:29:32.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:32.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:32.661727+0000) 2024-03-06T12:29:33.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:33.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:33.661959+0000) 2024-03-06T12:29:34.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:34.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:34.662198+0000) 2024-03-06T12:29:35.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:35.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:35.662441+0000) 2024-03-06T12:29:36.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:36.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:36.662814+0000) 2024-03-06T12:29:37.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:37.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:37.663051+0000) 2024-03-06T12:29:38.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:38.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:38.663288+0000) 2024-03-06T12:29:39.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:39.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:39.663497+0000) 2024-03-06T12:29:40.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:40.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:40.663773+0000) 2024-03-06T12:29:41.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:41.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:41.664003+0000) 2024-03-06T12:29:42.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:42.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:42.664253+0000) 2024-03-06T12:29:43.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:43.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:43.664504+0000) 2024-03-06T12:29:43.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:43.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:44.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:44.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:44.664756+0000) 2024-03-06T12:29:45.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:45.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:45.664985+0000) 2024-03-06T12:29:46.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:46.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:46.665203+0000) 2024-03-06T12:29:47.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:47.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:47.665440+0000) 2024-03-06T12:29:48.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:48.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:48.665699+0000) 2024-03-06T12:29:49.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:49.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:49.665914+0000) 2024-03-06T12:29:50.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:50.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:50.666189+0000) 2024-03-06T12:29:51.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:51.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:51.666445+0000) 2024-03-06T12:29:52.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:52.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:52.666651+0000) 2024-03-06T12:29:53.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:53.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:53.666901+0000) 2024-03-06T12:29:54.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:54.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:54.667096+0000) 2024-03-06T12:29:55.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:55.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:55.667342+0000) 2024-03-06T12:29:56.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:56.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:56.667543+0000) 2024-03-06T12:29:57.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:57.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:57.667778+0000) 2024-03-06T12:29:58.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:58.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:58.667962+0000) 2024-03-06T12:29:59.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:29:59.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:59.668161+0000) 2024-03-06T12:30:00.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:00.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:00.668350+0000) 2024-03-06T12:30:01.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:01.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:01.668561+0000) 2024-03-06T12:30:02.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:02.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:02.668812+0000) 2024-03-06T12:30:03.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:03.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:03.669003+0000) 2024-03-06T12:30:04.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:04.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:04.669242+0000) 2024-03-06T12:30:05.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:05.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:05.669484+0000) 2024-03-06T12:30:06.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:06.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:06.669762+0000) 2024-03-06T12:30:07.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:07.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:07.670021+0000) 2024-03-06T12:30:08.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:08.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:08.670267+0000) 2024-03-06T12:30:09.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:09.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:09.670450+0000) 2024-03-06T12:30:10.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:10.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:10.670741+0000) 2024-03-06T12:30:11.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:11.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:11.670978+0000) 2024-03-06T12:30:12.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:12.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:12.671218+0000) 2024-03-06T12:30:13.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:13.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:13.671445+0000) 2024-03-06T12:30:13.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:13.925 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:14.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:14.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:14.671709+0000) 2024-03-06T12:30:15.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:15.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:15.671915+0000) 2024-03-06T12:30:16.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:16.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:16.672097+0000) 2024-03-06T12:30:17.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:17.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:17.672333+0000) 2024-03-06T12:30:18.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:18.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:18.672519+0000) 2024-03-06T12:30:19.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:19.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:19.672804+0000) 2024-03-06T12:30:20.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:20.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:20.673057+0000) 2024-03-06T12:30:21.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:21.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:21.673304+0000) 2024-03-06T12:30:22.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:22.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:22.673512+0000) 2024-03-06T12:30:23.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:23.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:23.673799+0000) 2024-03-06T12:30:24.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:24.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:24.674038+0000) 2024-03-06T12:30:25.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:25.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:25.674277+0000) 2024-03-06T12:30:26.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:26.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:26.674526+0000) 2024-03-06T12:30:27.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:27.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:27.674793+0000) 2024-03-06T12:30:28.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:28.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:28.675023+0000) 2024-03-06T12:30:29.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:29.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:29.675257+0000) 2024-03-06T12:30:30.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:30.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:30.675506+0000) 2024-03-06T12:30:31.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:31.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:31.675728+0000) 2024-03-06T12:30:32.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:32.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:32.675878+0000) 2024-03-06T12:30:33.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:33.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:33.676028+0000) 2024-03-06T12:30:34.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:34.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:34.676157+0000) 2024-03-06T12:30:35.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:35.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:35.676381+0000) 2024-03-06T12:30:36.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:36.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:36.676598+0000) 2024-03-06T12:30:37.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:37.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:37.676829+0000) 2024-03-06T12:30:38.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:38.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:38.677073+0000) 2024-03-06T12:30:39.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:39.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:39.677316+0000) 2024-03-06T12:30:40.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:40.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:40.677568+0000) 2024-03-06T12:30:41.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:41.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:41.677822+0000) 2024-03-06T12:30:42.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:42.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:42.678053+0000) 2024-03-06T12:30:43.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:43.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:43.678284+0000) 2024-03-06T12:30:44.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:44.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:44.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:44.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:44.678496+0000) 2024-03-06T12:30:45.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:45.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:45.678768+0000) 2024-03-06T12:30:46.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:46.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:46.678992+0000) 2024-03-06T12:30:47.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:47.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:47.679221+0000) 2024-03-06T12:30:48.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:48.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:48.679450+0000) 2024-03-06T12:30:49.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:49.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:49.679721+0000) 2024-03-06T12:30:50.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:50.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:50.679965+0000) 2024-03-06T12:30:51.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:51.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:51.680196+0000) 2024-03-06T12:30:52.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:52.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:52.680375+0000) 2024-03-06T12:30:53.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:53.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:53.680611+0000) 2024-03-06T12:30:54.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:54.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:54.680829+0000) 2024-03-06T12:30:55.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:55.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:55.681025+0000) 2024-03-06T12:30:56.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:56.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:56.681262+0000) 2024-03-06T12:30:57.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:57.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:57.681500+0000) 2024-03-06T12:30:58.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:58.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:58.681767+0000) 2024-03-06T12:30:59.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:30:59.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:59.681907+0000) 2024-03-06T12:31:00.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:00.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:00.682133+0000) 2024-03-06T12:31:01.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:01.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:01.682318+0000) 2024-03-06T12:31:02.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:02.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:02.682522+0000) 2024-03-06T12:31:03.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:03.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:03.682825+0000) 2024-03-06T12:31:04.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:04.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:04.683051+0000) 2024-03-06T12:31:05.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:05.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:05.683192+0000) 2024-03-06T12:31:06.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:06.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:06.683358+0000) 2024-03-06T12:31:07.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:07.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:07.683538+0000) 2024-03-06T12:31:08.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:08.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:08.683772+0000) 2024-03-06T12:31:09.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:09.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:09.684004+0000) 2024-03-06T12:31:10.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:10.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:10.684186+0000) 2024-03-06T12:31:11.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:11.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:11.684408+0000) 2024-03-06T12:31:12.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:12.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:12.684654+0000) 2024-03-06T12:31:13.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:13.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:13.684930+0000) 2024-03-06T12:31:14.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:14.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:14.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:14.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:14.685118+0000) 2024-03-06T12:31:15.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:15.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:15.685376+0000) 2024-03-06T12:31:16.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:16.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:16.685624+0000) 2024-03-06T12:31:17.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:17.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:17.685790+0000) 2024-03-06T12:31:18.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:18.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:18.685997+0000) 2024-03-06T12:31:19.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:19.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:19.686215+0000) 2024-03-06T12:31:20.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:20.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:20.686401+0000) 2024-03-06T12:31:21.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:21.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:21.686625+0000) 2024-03-06T12:31:22.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:22.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:22.686834+0000) 2024-03-06T12:31:23.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:23.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:23.687023+0000) 2024-03-06T12:31:24.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:24.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:24.687254+0000) 2024-03-06T12:31:25.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:25.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:25.687503+0000) 2024-03-06T12:31:26.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:26.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:26.687791+0000) 2024-03-06T12:31:27.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:27.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:27.687990+0000) 2024-03-06T12:31:28.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:28.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:28.688182+0000) 2024-03-06T12:31:29.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:29.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:29.688423+0000) 2024-03-06T12:31:30.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:30.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:30.688635+0000) 2024-03-06T12:31:31.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:31.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:31.688907+0000) 2024-03-06T12:31:32.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:32.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:32.689135+0000) 2024-03-06T12:31:33.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:33.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:33.689333+0000) 2024-03-06T12:31:34.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:34.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:34.689574+0000) 2024-03-06T12:31:35.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:35.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:35.689791+0000) 2024-03-06T12:31:36.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:36.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:36.690036+0000) 2024-03-06T12:31:37.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:37.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:37.690267+0000) 2024-03-06T12:31:38.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:38.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:38.690521+0000) 2024-03-06T12:31:39.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:39.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:39.690833+0000) 2024-03-06T12:31:40.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:40.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:40.691082+0000) 2024-03-06T12:31:41.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:41.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:41.691323+0000) 2024-03-06T12:31:42.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:42.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:42.691551+0000) 2024-03-06T12:31:43.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:43.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:43.691797+0000) 2024-03-06T12:31:44.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:44.551 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:44.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:44.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:44.692057+0000) 2024-03-06T12:31:45.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:45.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:45.692259+0000) 2024-03-06T12:31:46.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:46.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:46.692495+0000) 2024-03-06T12:31:47.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:47.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:47.692788+0000) 2024-03-06T12:31:48.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:48.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:48.693028+0000) 2024-03-06T12:31:49.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:49.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:49.693270+0000) 2024-03-06T12:31:50.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:50.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:50.693513+0000) 2024-03-06T12:31:51.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:51.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:51.693789+0000) 2024-03-06T12:31:52.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:52.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:52.694029+0000) 2024-03-06T12:31:53.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:53.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:53.694227+0000) 2024-03-06T12:31:54.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:54.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:54.694465+0000) 2024-03-06T12:31:55.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:55.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:55.694732+0000) 2024-03-06T12:31:56.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:56.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:56.694960+0000) 2024-03-06T12:31:57.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:57.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:57.695198+0000) 2024-03-06T12:31:58.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:58.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:58.695423+0000) 2024-03-06T12:31:59.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:31:59.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:59.695604+0000) 2024-03-06T12:32:00.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:00.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:00.695824+0000) 2024-03-06T12:32:01.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:01.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:01.696063+0000) 2024-03-06T12:32:02.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:02.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:02.696219+0000) 2024-03-06T12:32:03.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:03.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:03.696442+0000) 2024-03-06T12:32:04.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:04.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:04.696688+0000) 2024-03-06T12:32:05.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:05.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:05.696926+0000) 2024-03-06T12:32:06.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:06.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:06.697115+0000) 2024-03-06T12:32:07.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:07.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:07.697359+0000) 2024-03-06T12:32:08.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:08.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:08.697588+0000) 2024-03-06T12:32:09.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:09.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:09.697820+0000) 2024-03-06T12:32:10.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:10.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:10.698052+0000) 2024-03-06T12:32:11.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:11.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:11.698338+0000) 2024-03-06T12:32:12.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:12.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:12.698526+0000) 2024-03-06T12:32:13.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:13.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:13.698804+0000) 2024-03-06T12:32:14.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:14.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:14.699045+0000) 2024-03-06T12:32:14.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:14.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:15.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:15.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:15.699259+0000) 2024-03-06T12:32:16.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:16.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:16.699487+0000) 2024-03-06T12:32:17.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:17.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:17.699710+0000) 2024-03-06T12:32:18.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:18.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:18.699921+0000) 2024-03-06T12:32:19.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:19.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:19.700153+0000) 2024-03-06T12:32:20.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:20.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:20.700363+0000) 2024-03-06T12:32:21.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:21.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:21.700577+0000) 2024-03-06T12:32:22.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:22.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:22.700803+0000) 2024-03-06T12:32:23.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:23.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:23.701046+0000) 2024-03-06T12:32:24.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:24.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:24.701275+0000) 2024-03-06T12:32:25.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:25.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:25.701461+0000) 2024-03-06T12:32:26.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:26.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:26.701653+0000) 2024-03-06T12:32:27.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:27.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:27.701941+0000) 2024-03-06T12:32:28.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:28.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:28.702153+0000) 2024-03-06T12:32:29.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:29.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:29.702391+0000) 2024-03-06T12:32:30.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:30.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:30.702633+0000) 2024-03-06T12:32:31.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:31.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:31.702866+0000) 2024-03-06T12:32:32.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:32.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:32.703099+0000) 2024-03-06T12:32:33.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:33.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:33.703321+0000) 2024-03-06T12:32:34.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:34.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:34.703493+0000) 2024-03-06T12:32:35.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:35.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:35.703719+0000) 2024-03-06T12:32:36.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:36.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:36.703962+0000) 2024-03-06T12:32:37.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:37.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:37.704159+0000) 2024-03-06T12:32:38.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:38.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:38.704383+0000) 2024-03-06T12:32:39.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:39.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:39.704607+0000) 2024-03-06T12:32:40.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:40.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:40.704848+0000) 2024-03-06T12:32:41.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:41.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:41.705069+0000) 2024-03-06T12:32:42.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:42.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:42.705303+0000) 2024-03-06T12:32:43.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:43.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:43.705535+0000) 2024-03-06T12:32:44.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:44.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:44.705791+0000) 2024-03-06T12:32:44.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:44.977 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:45.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:45.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:45.706022+0000) 2024-03-06T12:32:46.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:46.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:46.706201+0000) 2024-03-06T12:32:47.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:47.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:47.706383+0000) 2024-03-06T12:32:48.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:48.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:48.706580+0000) 2024-03-06T12:32:49.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:49.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:49.706820+0000) 2024-03-06T12:32:50.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:50.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:50.707007+0000) 2024-03-06T12:32:51.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:51.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:51.707185+0000) 2024-03-06T12:32:52.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:52.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:52.707402+0000) 2024-03-06T12:32:53.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:53.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:53.707627+0000) 2024-03-06T12:32:54.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:54.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:54.707854+0000) 2024-03-06T12:32:55.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:55.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:55.708093+0000) 2024-03-06T12:32:56.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:56.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:56.708312+0000) 2024-03-06T12:32:57.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:57.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:57.708511+0000) 2024-03-06T12:32:58.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:58.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:58.708773+0000) 2024-03-06T12:32:59.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:32:59.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:59.708990+0000) 2024-03-06T12:33:00.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:00.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:00.709226+0000) 2024-03-06T12:33:01.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:01.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:01.709446+0000) 2024-03-06T12:33:02.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:02.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:02.709706+0000) 2024-03-06T12:33:03.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:03.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:03.709948+0000) 2024-03-06T12:33:04.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:04.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:04.710221+0000) 2024-03-06T12:33:05.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:05.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:05.710457+0000) 2024-03-06T12:33:06.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:06.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:06.710708+0000) 2024-03-06T12:33:07.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:07.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:07.710912+0000) 2024-03-06T12:33:08.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:08.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:08.711120+0000) 2024-03-06T12:33:09.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:09.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:09.711299+0000) 2024-03-06T12:33:10.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:10.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:10.711553+0000) 2024-03-06T12:33:11.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:11.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:11.711788+0000) 2024-03-06T12:33:12.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:12.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:12.711983+0000) 2024-03-06T12:33:13.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:13.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:13.712229+0000) 2024-03-06T12:33:14.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:14.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:14.712460+0000) 2024-03-06T12:33:15.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:15.182 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:15.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:15.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:15.712752+0000) 2024-03-06T12:33:16.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:16.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:16.712981+0000) 2024-03-06T12:33:17.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:17.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:17.713175+0000) 2024-03-06T12:33:18.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:18.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:18.713414+0000) 2024-03-06T12:33:19.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:19.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:19.713655+0000) 2024-03-06T12:33:20.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:20.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:20.713912+0000) 2024-03-06T12:33:21.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:21.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:21.714097+0000) 2024-03-06T12:33:22.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:22.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:22.714341+0000) 2024-03-06T12:33:23.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:23.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:23.714542+0000) 2024-03-06T12:33:24.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:24.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:24.714767+0000) 2024-03-06T12:33:25.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:25.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:25.715010+0000) 2024-03-06T12:33:26.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:26.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:26.715197+0000) 2024-03-06T12:33:27.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:27.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:27.715441+0000) 2024-03-06T12:33:28.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:28.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:28.715625+0000) 2024-03-06T12:33:29.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:29.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:29.715917+0000) 2024-03-06T12:33:30.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:30.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:30.716126+0000) 2024-03-06T12:33:31.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:31.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:31.716369+0000) 2024-03-06T12:33:32.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:32.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:32.716604+0000) 2024-03-06T12:33:33.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:33.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:33.716849+0000) 2024-03-06T12:33:34.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:34.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:34.717091+0000) 2024-03-06T12:33:35.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:35.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:35.717330+0000) 2024-03-06T12:33:36.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:36.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:36.717543+0000) 2024-03-06T12:33:37.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:37.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:37.717799+0000) 2024-03-06T12:33:38.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:38.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:38.718029+0000) 2024-03-06T12:33:39.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:39.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:39.718255+0000) 2024-03-06T12:33:40.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:40.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:40.718490+0000) 2024-03-06T12:33:41.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:41.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:41.718745+0000) 2024-03-06T12:33:42.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:42.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:42.718968+0000) 2024-03-06T12:33:43.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:43.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:43.719194+0000) 2024-03-06T12:33:44.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:44.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:44.719424+0000) 2024-03-06T12:33:45.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:45.382 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:45.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:45.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:45.719661+0000) 2024-03-06T12:33:46.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:46.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:46.719920+0000) 2024-03-06T12:33:47.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:47.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:47.720125+0000) 2024-03-06T12:33:48.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:48.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:48.720373+0000) 2024-03-06T12:33:49.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:49.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:49.720610+0000) 2024-03-06T12:33:50.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:50.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:50.720864+0000) 2024-03-06T12:33:51.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:51.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:51.721111+0000) 2024-03-06T12:33:52.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:52.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:52.721348+0000) 2024-03-06T12:33:53.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:53.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:53.721538+0000) 2024-03-06T12:33:54.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:54.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:54.721807+0000) 2024-03-06T12:33:55.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:55.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:55.722012+0000) 2024-03-06T12:33:56.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:56.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:56.722195+0000) 2024-03-06T12:33:57.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:57.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:57.722445+0000) 2024-03-06T12:33:58.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:58.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:58.722729+0000) 2024-03-06T12:33:59.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:33:59.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:59.722923+0000) 2024-03-06T12:34:00.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:00.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:00.723170+0000) 2024-03-06T12:34:01.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:01.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:01.723411+0000) 2024-03-06T12:34:02.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:02.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:02.723651+0000) 2024-03-06T12:34:03.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:03.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:03.723942+0000) 2024-03-06T12:34:04.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:04.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:04.724179+0000) 2024-03-06T12:34:05.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:05.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:05.724433+0000) 2024-03-06T12:34:06.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:06.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:06.724723+0000) 2024-03-06T12:34:07.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:07.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:07.724964+0000) 2024-03-06T12:34:08.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:08.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:08.725193+0000) 2024-03-06T12:34:09.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:09.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:09.725418+0000) 2024-03-06T12:34:10.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:10.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:10.725620+0000) 2024-03-06T12:34:11.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:11.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:11.725870+0000) 2024-03-06T12:34:12.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:12.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:12.726117+0000) 2024-03-06T12:34:13.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:13.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:13.726359+0000) 2024-03-06T12:34:14.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:14.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:14.726580+0000) 2024-03-06T12:34:15.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:15.579 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:15.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:15.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:15.726776+0000) 2024-03-06T12:34:16.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:16.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:16.726949+0000) 2024-03-06T12:34:17.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:17.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:17.727134+0000) 2024-03-06T12:34:18.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:18.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:18.727331+0000) 2024-03-06T12:34:19.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:19.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:19.727567+0000) 2024-03-06T12:34:20.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:20.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:20.727800+0000) 2024-03-06T12:34:21.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:21.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:21.728024+0000) 2024-03-06T12:34:22.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:22.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:22.728244+0000) 2024-03-06T12:34:23.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:23.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:23.728480+0000) 2024-03-06T12:34:24.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:24.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:24.728758+0000) 2024-03-06T12:34:25.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:25.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:25.728947+0000) 2024-03-06T12:34:26.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:26.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:26.729138+0000) 2024-03-06T12:34:27.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:27.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:27.729326+0000) 2024-03-06T12:34:28.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:28.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:28.729557+0000) 2024-03-06T12:34:29.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:29.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:29.729771+0000) 2024-03-06T12:34:30.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:30.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:30.730009+0000) 2024-03-06T12:34:31.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:31.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:31.730238+0000) 2024-03-06T12:34:32.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:32.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:32.730432+0000) 2024-03-06T12:34:33.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:33.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:33.730706+0000) 2024-03-06T12:34:34.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:34.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:34.730934+0000) 2024-03-06T12:34:35.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:35.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:35.731178+0000) 2024-03-06T12:34:36.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:36.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:36.731401+0000) 2024-03-06T12:34:37.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:37.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:37.731632+0000) 2024-03-06T12:34:38.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:38.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:38.731869+0000) 2024-03-06T12:34:39.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:39.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:39.732095+0000) 2024-03-06T12:34:40.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:40.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:40.732335+0000) 2024-03-06T12:34:41.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:41.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:41.732524+0000) 2024-03-06T12:34:42.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:42.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:42.732758+0000) 2024-03-06T12:34:43.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:43.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:43.732917+0000) 2024-03-06T12:34:44.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:44.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:44.733137+0000) 2024-03-06T12:34:45.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:45.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:45.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:45.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:45.733313+0000) 2024-03-06T12:34:46.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:46.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:46.733535+0000) 2024-03-06T12:34:47.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:47.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:47.733775+0000) 2024-03-06T12:34:48.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:48.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:48.734025+0000) 2024-03-06T12:34:49.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:49.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:49.734242+0000) 2024-03-06T12:34:50.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:50.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:50.734492+0000) 2024-03-06T12:34:51.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:51.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:51.734814+0000) 2024-03-06T12:34:52.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:52.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:52.734964+0000) 2024-03-06T12:34:53.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:53.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:53.735177+0000) 2024-03-06T12:34:54.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:54.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:54.735376+0000) 2024-03-06T12:34:55.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:55.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:55.735623+0000) 2024-03-06T12:34:56.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:56.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:56.735837+0000) 2024-03-06T12:34:57.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:57.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:57.736044+0000) 2024-03-06T12:34:58.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:58.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:58.736296+0000) 2024-03-06T12:34:59.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:34:59.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:59.736513+0000) 2024-03-06T12:35:00.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:00.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:00.736815+0000) 2024-03-06T12:35:01.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:01.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:01.737042+0000) 2024-03-06T12:35:02.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:02.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:02.737266+0000) 2024-03-06T12:35:03.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:03.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:03.737502+0000) 2024-03-06T12:35:04.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:04.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:04.737772+0000) 2024-03-06T12:35:05.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:05.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:05.738007+0000) 2024-03-06T12:35:06.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:06.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:06.738287+0000) 2024-03-06T12:35:07.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:07.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:07.738564+0000) 2024-03-06T12:35:08.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:08.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:08.738805+0000) 2024-03-06T12:35:09.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:09.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:09.739070+0000) 2024-03-06T12:35:10.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:10.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:10.739354+0000) 2024-03-06T12:35:11.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:11.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:11.739636+0000) 2024-03-06T12:35:12.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:12.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:12.739984+0000) 2024-03-06T12:35:13.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:13.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:13.740189+0000) 2024-03-06T12:35:14.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:14.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:14.740375+0000) 2024-03-06T12:35:15.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:15.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:15.740605+0000) 2024-03-06T12:35:15.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:15.850 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:16.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:16.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:16.740836+0000) 2024-03-06T12:35:17.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:17.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:17.741080+0000) 2024-03-06T12:35:18.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:18.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:18.741360+0000) 2024-03-06T12:35:19.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:19.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:19.741630+0000) 2024-03-06T12:35:20.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:20.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:20.741911+0000) 2024-03-06T12:35:21.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:21.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:21.742131+0000) 2024-03-06T12:35:22.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:22.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:22.742347+0000) 2024-03-06T12:35:23.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:23.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:23.742557+0000) 2024-03-06T12:35:24.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:24.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:24.742785+0000) 2024-03-06T12:35:25.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:25.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:25.743015+0000) 2024-03-06T12:35:26.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:26.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:26.743206+0000) 2024-03-06T12:35:27.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:27.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:27.743398+0000) 2024-03-06T12:35:28.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:28.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:28.743602+0000) 2024-03-06T12:35:29.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:29.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:29.743806+0000) 2024-03-06T12:35:30.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:30.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:30.743985+0000) 2024-03-06T12:35:31.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:31.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:31.744253+0000) 2024-03-06T12:35:32.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:32.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:32.744498+0000) 2024-03-06T12:35:33.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:33.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:33.744802+0000) 2024-03-06T12:35:34.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:34.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:34.745075+0000) 2024-03-06T12:35:35.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:35.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:35.745357+0000) 2024-03-06T12:35:36.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:36.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:36.745627+0000) 2024-03-06T12:35:37.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:37.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:37.745968+0000) 2024-03-06T12:35:38.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:38.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:38.746248+0000) 2024-03-06T12:35:39.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:39.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:39.746498+0000) 2024-03-06T12:35:40.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:40.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:40.746774+0000) 2024-03-06T12:35:41.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:41.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:41.747010+0000) 2024-03-06T12:35:42.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:42.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:42.747240+0000) 2024-03-06T12:35:43.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:43.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:43.747466+0000) 2024-03-06T12:35:44.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:44.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:44.747746+0000) 2024-03-06T12:35:45.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:45.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:45.747987+0000) 2024-03-06T12:35:46.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:46.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:46.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:46.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:46.748205+0000) 2024-03-06T12:35:47.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:47.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:47.748433+0000) 2024-03-06T12:35:48.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:48.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:48.748635+0000) 2024-03-06T12:35:49.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:49.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:49.748864+0000) 2024-03-06T12:35:50.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:50.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:50.749118+0000) 2024-03-06T12:35:51.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:51.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:51.749353+0000) 2024-03-06T12:35:52.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:52.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:52.749485+0000) 2024-03-06T12:35:53.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:53.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:53.749738+0000) 2024-03-06T12:35:54.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:54.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:54.750031+0000) 2024-03-06T12:35:55.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:55.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:55.750313+0000) 2024-03-06T12:35:56.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:56.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:56.750601+0000) 2024-03-06T12:35:57.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:57.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:57.750959+0000) 2024-03-06T12:35:58.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:58.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:58.751189+0000) 2024-03-06T12:35:59.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:35:59.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:59.751366+0000) 2024-03-06T12:36:00.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:00.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:00.751591+0000) 2024-03-06T12:36:01.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:01.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:01.751806+0000) 2024-03-06T12:36:02.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:02.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:02.752046+0000) 2024-03-06T12:36:03.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:03.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:03.752276+0000) 2024-03-06T12:36:04.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:04.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:04.752495+0000) 2024-03-06T12:36:05.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:05.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:05.752767+0000) 2024-03-06T12:36:06.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:06.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:06.752991+0000) 2024-03-06T12:36:07.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:07.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:07.753189+0000) 2024-03-06T12:36:08.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:08.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:08.753431+0000) 2024-03-06T12:36:09.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:09.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:09.753654+0000) 2024-03-06T12:36:10.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:10.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:10.753914+0000) 2024-03-06T12:36:11.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:11.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:11.754189+0000) 2024-03-06T12:36:12.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:12.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:12.754473+0000) 2024-03-06T12:36:13.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:13.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:13.754786+0000) 2024-03-06T12:36:14.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:14.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:14.755066+0000) 2024-03-06T12:36:15.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:15.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:15.755275+0000) 2024-03-06T12:36:16.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:16.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:16.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:16.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:16.755518+0000) 2024-03-06T12:36:17.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:17.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:17.755753+0000) 2024-03-06T12:36:18.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:18.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:18.755988+0000) 2024-03-06T12:36:19.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:19.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:19.756215+0000) 2024-03-06T12:36:20.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:20.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:20.756440+0000) 2024-03-06T12:36:21.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:21.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:21.756654+0000) 2024-03-06T12:36:22.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:22.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:22.756894+0000) 2024-03-06T12:36:23.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:23.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:23.757085+0000) 2024-03-06T12:36:24.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:24.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:24.757274+0000) 2024-03-06T12:36:25.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:25.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:25.757450+0000) 2024-03-06T12:36:26.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:26.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:26.757641+0000) 2024-03-06T12:36:27.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:27.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:27.757884+0000) 2024-03-06T12:36:28.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:28.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:28.758071+0000) 2024-03-06T12:36:29.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:29.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:29.758284+0000) 2024-03-06T12:36:30.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:30.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:30.758521+0000) 2024-03-06T12:36:31.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:31.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:31.758785+0000) 2024-03-06T12:36:32.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:32.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:32.759008+0000) 2024-03-06T12:36:33.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:33.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:33.759237+0000) 2024-03-06T12:36:34.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:34.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:34.759446+0000) 2024-03-06T12:36:35.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:35.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:35.759766+0000) 2024-03-06T12:36:36.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:36.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:36.760030+0000) 2024-03-06T12:36:37.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:37.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:37.760259+0000) 2024-03-06T12:36:38.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:38.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:38.760499+0000) 2024-03-06T12:36:39.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:39.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:39.760702+0000) 2024-03-06T12:36:40.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:40.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:40.760941+0000) 2024-03-06T12:36:41.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:41.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:41.761169+0000) 2024-03-06T12:36:42.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:42.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:42.761402+0000) 2024-03-06T12:36:43.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:43.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:43.761706+0000) 2024-03-06T12:36:44.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:44.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:44.761942+0000) 2024-03-06T12:36:45.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:45.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:45.762120+0000) 2024-03-06T12:36:46.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:46.554 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:46.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:46.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:46.762322+0000) 2024-03-06T12:36:47.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:47.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:47.762604+0000) 2024-03-06T12:36:48.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:48.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:48.762847+0000) 2024-03-06T12:36:49.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:49.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:49.763052+0000) 2024-03-06T12:36:50.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:50.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:50.763234+0000) 2024-03-06T12:36:51.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:51.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:51.763415+0000) 2024-03-06T12:36:52.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:52.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:52.763638+0000) 2024-03-06T12:36:53.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:53.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:53.763918+0000) 2024-03-06T12:36:54.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:54.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:54.764144+0000) 2024-03-06T12:36:55.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:55.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:55.764368+0000) 2024-03-06T12:36:56.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:56.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:56.764587+0000) 2024-03-06T12:36:57.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:57.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:57.764824+0000) 2024-03-06T12:36:58.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:58.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:58.765056+0000) 2024-03-06T12:36:59.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:36:59.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:59.765247+0000) 2024-03-06T12:37:00.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:00.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:00.765482+0000) 2024-03-06T12:37:01.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:01.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:01.765720+0000) 2024-03-06T12:37:02.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:02.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:02.766003+0000) 2024-03-06T12:37:03.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:03.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:03.766271+0000) 2024-03-06T12:37:04.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:04.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:04.766552+0000) 2024-03-06T12:37:05.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:05.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:05.766824+0000) 2024-03-06T12:37:06.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:06.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:06.767107+0000) 2024-03-06T12:37:07.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:07.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:07.767369+0000) 2024-03-06T12:37:08.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:08.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:08.767549+0000) 2024-03-06T12:37:09.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:09.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:09.767784+0000) 2024-03-06T12:37:10.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:10.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:10.767991+0000) 2024-03-06T12:37:11.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:11.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:11.768210+0000) 2024-03-06T12:37:12.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:12.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:12.768340+0000) 2024-03-06T12:37:13.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:13.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:13.768552+0000) 2024-03-06T12:37:14.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:14.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:14.768812+0000) 2024-03-06T12:37:15.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:15.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:15.769081+0000) 2024-03-06T12:37:16.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:16.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:16.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:16.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:16.769237+0000) 2024-03-06T12:37:17.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:17.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:17.769470+0000) 2024-03-06T12:37:18.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:18.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:18.769732+0000) 2024-03-06T12:37:19.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:19.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:19.769959+0000) 2024-03-06T12:37:20.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:20.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:20.770249+0000) 2024-03-06T12:37:21.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:21.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:21.770472+0000) 2024-03-06T12:37:22.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:22.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:22.770731+0000) 2024-03-06T12:37:23.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:23.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:23.770968+0000) 2024-03-06T12:37:24.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:24.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:24.771199+0000) 2024-03-06T12:37:25.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:25.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:25.771426+0000) 2024-03-06T12:37:26.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:26.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:26.771655+0000) 2024-03-06T12:37:27.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:27.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:27.771940+0000) 2024-03-06T12:37:28.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:28.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:28.772207+0000) 2024-03-06T12:37:29.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:29.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:29.772493+0000) 2024-03-06T12:37:30.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:30.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:30.772731+0000) 2024-03-06T12:37:31.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:31.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:31.772990+0000) 2024-03-06T12:37:32.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:32.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:32.773229+0000) 2024-03-06T12:37:33.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:33.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:33.773501+0000) 2024-03-06T12:37:34.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:34.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:34.773819+0000) 2024-03-06T12:37:35.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:35.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:35.774024+0000) 2024-03-06T12:37:36.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:36.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:36.774248+0000) 2024-03-06T12:37:37.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:37.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:37.774483+0000) 2024-03-06T12:37:38.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:38.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:38.774778+0000) 2024-03-06T12:37:39.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:39.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:39.775005+0000) 2024-03-06T12:37:40.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:40.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:40.775221+0000) 2024-03-06T12:37:41.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:41.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:41.775504+0000) 2024-03-06T12:37:42.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:42.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:42.775804+0000) 2024-03-06T12:37:43.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:43.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:43.776082+0000) 2024-03-06T12:37:44.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:44.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:44.776354+0000) 2024-03-06T12:37:45.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:45.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:45.776636+0000) 2024-03-06T12:37:46.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:46.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:46.776959+0000) 2024-03-06T12:37:46.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:46.894 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:47.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:47.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:47.777240+0000) 2024-03-06T12:37:48.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:48.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:48.777478+0000) 2024-03-06T12:37:49.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:49.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:49.777698+0000) 2024-03-06T12:37:50.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:50.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:50.777927+0000) 2024-03-06T12:37:51.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:51.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:51.778060+0000) 2024-03-06T12:37:52.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:52.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:52.778286+0000) 2024-03-06T12:37:53.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:53.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:53.778495+0000) 2024-03-06T12:37:54.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:54.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:54.778770+0000) 2024-03-06T12:37:55.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:55.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:55.778912+0000) 2024-03-06T12:37:56.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:56.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:56.779146+0000) 2024-03-06T12:37:57.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:57.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:57.779404+0000) 2024-03-06T12:37:58.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:58.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:58.779659+0000) 2024-03-06T12:37:59.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:37:59.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:59.779946+0000) 2024-03-06T12:38:00.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:00.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:00.780189+0000) 2024-03-06T12:38:01.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:01.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:01.780428+0000) 2024-03-06T12:38:02.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:02.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:02.780703+0000) 2024-03-06T12:38:03.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:03.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:03.780927+0000) 2024-03-06T12:38:04.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:04.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:04.781199+0000) 2024-03-06T12:38:05.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:05.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:05.781480+0000) 2024-03-06T12:38:06.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:06.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:06.781799+0000) 2024-03-06T12:38:07.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:07.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:07.782094+0000) 2024-03-06T12:38:08.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:08.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:08.782366+0000) 2024-03-06T12:38:09.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:09.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:09.782580+0000) 2024-03-06T12:38:10.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:10.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:10.782830+0000) 2024-03-06T12:38:11.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:11.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:11.783072+0000) 2024-03-06T12:38:12.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:12.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:12.783303+0000) 2024-03-06T12:38:13.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:13.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:13.783552+0000) 2024-03-06T12:38:14.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:14.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:14.783890+0000) 2024-03-06T12:38:15.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:15.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:15.784098+0000) 2024-03-06T12:38:16.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:16.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:16.784347+0000) 2024-03-06T12:38:17.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:17.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:17.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:17.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:17.784590+0000) 2024-03-06T12:38:18.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:18.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:18.784828+0000) 2024-03-06T12:38:19.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:19.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:19.785042+0000) 2024-03-06T12:38:20.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:20.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:20.785252+0000) 2024-03-06T12:38:21.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:21.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:21.785479+0000) 2024-03-06T12:38:22.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:22.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:22.785741+0000) 2024-03-06T12:38:23.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:23.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:23.786021+0000) 2024-03-06T12:38:24.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:24.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:24.786230+0000) 2024-03-06T12:38:25.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:25.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:25.786460+0000) 2024-03-06T12:38:26.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:26.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:26.786733+0000) 2024-03-06T12:38:27.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:27.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:27.786924+0000) 2024-03-06T12:38:28.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:28.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:28.787124+0000) 2024-03-06T12:38:29.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:29.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:29.787353+0000) 2024-03-06T12:38:30.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:30.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:30.787577+0000) 2024-03-06T12:38:31.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:31.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:31.787789+0000) 2024-03-06T12:38:32.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:32.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:32.788030+0000) 2024-03-06T12:38:33.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:33.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:33.788215+0000) 2024-03-06T12:38:34.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:34.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:34.788455+0000) 2024-03-06T12:38:35.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:35.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:35.788737+0000) 2024-03-06T12:38:36.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:36.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:36.789016+0000) 2024-03-06T12:38:37.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:37.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:37.789271+0000) 2024-03-06T12:38:38.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:38.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:38.789516+0000) 2024-03-06T12:38:39.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:39.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:39.789796+0000) 2024-03-06T12:38:40.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:40.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:40.790014+0000) 2024-03-06T12:38:41.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:41.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:41.790262+0000) 2024-03-06T12:38:42.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:42.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:42.790492+0000) 2024-03-06T12:38:43.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:43.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:43.790768+0000) 2024-03-06T12:38:44.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:44.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:44.790996+0000) 2024-03-06T12:38:45.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:45.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:45.791231+0000) 2024-03-06T12:38:46.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:46.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:46.791489+0000) 2024-03-06T12:38:47.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:47.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:47.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:47.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:47.791771+0000) 2024-03-06T12:38:48.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:48.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:48.792006+0000) 2024-03-06T12:38:49.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:49.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:49.792193+0000) 2024-03-06T12:38:50.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:50.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:50.792429+0000) 2024-03-06T12:38:51.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:51.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:51.792688+0000) 2024-03-06T12:38:52.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:52.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:52.792906+0000) 2024-03-06T12:38:53.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:53.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:53.793157+0000) 2024-03-06T12:38:54.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:54.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:54.793354+0000) 2024-03-06T12:38:55.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:55.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:55.793595+0000) 2024-03-06T12:38:56.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:56.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:56.793848+0000) 2024-03-06T12:38:57.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:57.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:57.794107+0000) 2024-03-06T12:38:58.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:58.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:58.794352+0000) 2024-03-06T12:38:59.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:38:59.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:59.794593+0000) 2024-03-06T12:39:00.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:00.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:00.794829+0000) 2024-03-06T12:39:01.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:01.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:01.795005+0000) 2024-03-06T12:39:02.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:02.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:02.795235+0000) 2024-03-06T12:39:03.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:03.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:03.795419+0000) 2024-03-06T12:39:04.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:04.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:04.795637+0000) 2024-03-06T12:39:05.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:05.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:05.795834+0000) 2024-03-06T12:39:06.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:06.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:06.796078+0000) 2024-03-06T12:39:07.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:07.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:07.796371+0000) 2024-03-06T12:39:08.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:08.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:08.796602+0000) 2024-03-06T12:39:09.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:09.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:09.796824+0000) 2024-03-06T12:39:10.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:10.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:10.797060+0000) 2024-03-06T12:39:11.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:11.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:11.797301+0000) 2024-03-06T12:39:12.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:12.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:12.797495+0000) 2024-03-06T12:39:13.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:13.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:13.797739+0000) 2024-03-06T12:39:14.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:14.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:14.797949+0000) 2024-03-06T12:39:15.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:15.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:15.798183+0000) 2024-03-06T12:39:16.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:16.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:16.798434+0000) 2024-03-06T12:39:17.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:17.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:17.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:17.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:17.798741+0000) 2024-03-06T12:39:18.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:18.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:18.798996+0000) 2024-03-06T12:39:19.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:19.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:19.799230+0000) 2024-03-06T12:39:20.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:20.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:20.799444+0000) 2024-03-06T12:39:21.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:21.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:21.799642+0000) 2024-03-06T12:39:22.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:22.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:22.799875+0000) 2024-03-06T12:39:23.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:23.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:23.800086+0000) 2024-03-06T12:39:24.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:24.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:24.800322+0000) 2024-03-06T12:39:25.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:25.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:25.800587+0000) 2024-03-06T12:39:26.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:26.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:26.800879+0000) 2024-03-06T12:39:27.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:27.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:27.801139+0000) 2024-03-06T12:39:28.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:28.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:28.801359+0000) 2024-03-06T12:39:29.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:29.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:29.801584+0000) 2024-03-06T12:39:30.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:30.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:30.801830+0000) 2024-03-06T12:39:31.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:31.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:31.802064+0000) 2024-03-06T12:39:32.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:32.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:32.802289+0000) 2024-03-06T12:39:33.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:33.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:33.802550+0000) 2024-03-06T12:39:34.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:34.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:34.802830+0000) 2024-03-06T12:39:35.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:35.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:35.803108+0000) 2024-03-06T12:39:36.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:36.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:36.803379+0000) 2024-03-06T12:39:37.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:37.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:37.803707+0000) 2024-03-06T12:39:38.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:38.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:38.803998+0000) 2024-03-06T12:39:39.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:39.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:39.804288+0000) 2024-03-06T12:39:40.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:40.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:40.804563+0000) 2024-03-06T12:39:41.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:41.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:41.804845+0000) 2024-03-06T12:39:42.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:42.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:42.805122+0000) 2024-03-06T12:39:43.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:43.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:43.805398+0000) 2024-03-06T12:39:44.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:44.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:44.805701+0000) 2024-03-06T12:39:45.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:45.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:45.805905+0000) 2024-03-06T12:39:46.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:46.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:46.806154+0000) 2024-03-06T12:39:47.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:47.752 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:47.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:47.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:47.806347+0000) 2024-03-06T12:39:48.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:48.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:48.806580+0000) 2024-03-06T12:39:49.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:49.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:49.806790+0000) 2024-03-06T12:39:50.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:50.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:50.807049+0000) 2024-03-06T12:39:51.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:51.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:51.807340+0000) 2024-03-06T12:39:52.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:52.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:52.807625+0000) 2024-03-06T12:39:53.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:53.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:53.807865+0000) 2024-03-06T12:39:54.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:54.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:54.808047+0000) 2024-03-06T12:39:55.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:55.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:55.808307+0000) 2024-03-06T12:39:56.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:56.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:56.808560+0000) 2024-03-06T12:39:57.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:57.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:57.808808+0000) 2024-03-06T12:39:58.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:58.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:58.808998+0000) 2024-03-06T12:39:59.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:39:59.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:59.809219+0000) 2024-03-06T12:40:00.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:00.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:00.809452+0000) 2024-03-06T12:40:01.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:01.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:01.809760+0000) 2024-03-06T12:40:02.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:02.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:02.809939+0000) 2024-03-06T12:40:03.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:03.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:03.810173+0000) 2024-03-06T12:40:04.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:04.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:04.810398+0000) 2024-03-06T12:40:05.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:05.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:05.810619+0000) 2024-03-06T12:40:06.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:06.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:06.810863+0000) 2024-03-06T12:40:07.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:07.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:07.811053+0000) 2024-03-06T12:40:08.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:08.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:08.811281+0000) 2024-03-06T12:40:09.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:09.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:09.811457+0000) 2024-03-06T12:40:10.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:10.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:10.811715+0000) 2024-03-06T12:40:11.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:11.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:11.811969+0000) 2024-03-06T12:40:12.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:12.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:12.812171+0000) 2024-03-06T12:40:13.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:13.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:13.812371+0000) 2024-03-06T12:40:14.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:14.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:14.812593+0000) 2024-03-06T12:40:15.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:15.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:15.812832+0000) 2024-03-06T12:40:16.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:16.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:16.813094+0000) 2024-03-06T12:40:17.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:17.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:17.813318+0000) 2024-03-06T12:40:17.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:17.991 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:18.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:18.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:18.813518+0000) 2024-03-06T12:40:19.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:19.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:19.813788+0000) 2024-03-06T12:40:20.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:20.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:20.814037+0000) 2024-03-06T12:40:21.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:21.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:21.814254+0000) 2024-03-06T12:40:22.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:22.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:22.814489+0000) 2024-03-06T12:40:23.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:23.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:23.814771+0000) 2024-03-06T12:40:24.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:24.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:24.814996+0000) 2024-03-06T12:40:25.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:25.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:25.815234+0000) 2024-03-06T12:40:26.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:26.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:26.815482+0000) 2024-03-06T12:40:27.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:27.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:27.815778+0000) 2024-03-06T12:40:28.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:28.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:28.816021+0000) 2024-03-06T12:40:29.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:29.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:29.816267+0000) 2024-03-06T12:40:30.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:30.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:30.816516+0000) 2024-03-06T12:40:31.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:31.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:31.816818+0000) 2024-03-06T12:40:32.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:32.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:32.817074+0000) 2024-03-06T12:40:33.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:33.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:33.817336+0000) 2024-03-06T12:40:34.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:34.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:34.817598+0000) 2024-03-06T12:40:35.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:35.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:35.817796+0000) 2024-03-06T12:40:36.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:36.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:36.818019+0000) 2024-03-06T12:40:37.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:37.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:37.818210+0000) 2024-03-06T12:40:38.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:38.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:38.818371+0000) 2024-03-06T12:40:39.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:39.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:39.818603+0000) 2024-03-06T12:40:40.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:40.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:40.818843+0000) 2024-03-06T12:40:41.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:41.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:41.819053+0000) 2024-03-06T12:40:42.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:42.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:42.819290+0000) 2024-03-06T12:40:43.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:43.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:43.819532+0000) 2024-03-06T12:40:44.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:44.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:44.819784+0000) 2024-03-06T12:40:45.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:45.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:45.820067+0000) 2024-03-06T12:40:46.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:46.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:46.820267+0000) 2024-03-06T12:40:47.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:47.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:47.820511+0000) 2024-03-06T12:40:48.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:48.211 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:48.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:48.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:48.820829+0000) 2024-03-06T12:40:49.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:49.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:49.821068+0000) 2024-03-06T12:40:50.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:50.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:50.821273+0000) 2024-03-06T12:40:51.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:51.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:51.821545+0000) 2024-03-06T12:40:52.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:52.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:52.821837+0000) 2024-03-06T12:40:53.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:53.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:53.822064+0000) 2024-03-06T12:40:54.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:54.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:54.822252+0000) 2024-03-06T12:40:55.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:55.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:55.822440+0000) 2024-03-06T12:40:56.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:56.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:56.822602+0000) 2024-03-06T12:40:57.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:57.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:57.822824+0000) 2024-03-06T12:40:58.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:58.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:58.823026+0000) 2024-03-06T12:40:59.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:40:59.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:59.823177+0000) 2024-03-06T12:41:00.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:00.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:00.823365+0000) 2024-03-06T12:41:01.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:01.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:01.823581+0000) 2024-03-06T12:41:02.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:02.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:02.823806+0000) 2024-03-06T12:41:03.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:03.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:03.823967+0000) 2024-03-06T12:41:04.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:04.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:04.824179+0000) 2024-03-06T12:41:05.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:05.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:05.824429+0000) 2024-03-06T12:41:06.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:06.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:06.824641+0000) 2024-03-06T12:41:07.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:07.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:07.824912+0000) 2024-03-06T12:41:08.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:08.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:08.825116+0000) 2024-03-06T12:41:09.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:09.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:09.825308+0000) 2024-03-06T12:41:10.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:10.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:10.825508+0000) 2024-03-06T12:41:11.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:11.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:11.825749+0000) 2024-03-06T12:41:12.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:12.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:12.825949+0000) 2024-03-06T12:41:13.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:13.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:13.826152+0000) 2024-03-06T12:41:14.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:14.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:14.826345+0000) 2024-03-06T12:41:15.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:15.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:15.826512+0000) 2024-03-06T12:41:16.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:16.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:16.826773+0000) 2024-03-06T12:41:17.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:17.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:17.826964+0000) 2024-03-06T12:41:18.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:18.373 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:18.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:18.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:18.827186+0000) 2024-03-06T12:41:19.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:19.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:19.827369+0000) 2024-03-06T12:41:20.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:20.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:20.827564+0000) 2024-03-06T12:41:21.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:21.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:21.827808+0000) 2024-03-06T12:41:22.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:22.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:22.828020+0000) 2024-03-06T12:41:23.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:23.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:23.828208+0000) 2024-03-06T12:41:24.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:24.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:24.828391+0000) 2024-03-06T12:41:25.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:25.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:25.828567+0000) 2024-03-06T12:41:26.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:26.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:26.828801+0000) 2024-03-06T12:41:27.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:27.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:27.829005+0000) 2024-03-06T12:41:28.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:28.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:28.829189+0000) 2024-03-06T12:41:29.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:29.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:29.829422+0000) 2024-03-06T12:41:30.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:30.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:30.829611+0000) 2024-03-06T12:41:31.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:31.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:31.829825+0000) 2024-03-06T12:41:32.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:32.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:32.830037+0000) 2024-03-06T12:41:33.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:33.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:33.830223+0000) 2024-03-06T12:41:34.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:34.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:34.830443+0000) 2024-03-06T12:41:35.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:35.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:35.830656+0000) 2024-03-06T12:41:36.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:36.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:36.830951+0000) 2024-03-06T12:41:37.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:37.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:37.831149+0000) 2024-03-06T12:41:38.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:38.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:38.831385+0000) 2024-03-06T12:41:39.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:39.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:39.831616+0000) 2024-03-06T12:41:40.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:40.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:40.831833+0000) 2024-03-06T12:41:41.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:41.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:41.832060+0000) 2024-03-06T12:41:42.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:42.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:42.832245+0000) 2024-03-06T12:41:43.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:43.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:43.832491+0000) 2024-03-06T12:41:44.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:44.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:44.832787+0000) 2024-03-06T12:41:45.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:45.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:45.833039+0000) 2024-03-06T12:41:46.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:46.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:46.833250+0000) 2024-03-06T12:41:47.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:47.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:47.833509+0000) 2024-03-06T12:41:48.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:48.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:48.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:48.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:48.833750+0000) 2024-03-06T12:41:49.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:49.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:49.834008+0000) 2024-03-06T12:41:50.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:50.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:50.834258+0000) 2024-03-06T12:41:51.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:51.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:51.834466+0000) 2024-03-06T12:41:52.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:52.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:52.834699+0000) 2024-03-06T12:41:53.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:53.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:53.834945+0000) 2024-03-06T12:41:54.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:54.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:54.835171+0000) 2024-03-06T12:41:55.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:55.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:55.835399+0000) 2024-03-06T12:41:56.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:56.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:56.835624+0000) 2024-03-06T12:41:57.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:57.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:57.835832+0000) 2024-03-06T12:41:58.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:58.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:58.836043+0000) 2024-03-06T12:41:59.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:41:59.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:59.836263+0000) 2024-03-06T12:42:00.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:00.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:00.836501+0000) 2024-03-06T12:42:01.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:01.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:01.836713+0000) 2024-03-06T12:42:02.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:02.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:02.836955+0000) 2024-03-06T12:42:03.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:03.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:03.837184+0000) 2024-03-06T12:42:04.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:04.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:04.837426+0000) 2024-03-06T12:42:05.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:05.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:05.837620+0000) 2024-03-06T12:42:06.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:06.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:06.837847+0000) 2024-03-06T12:42:07.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:07.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:07.838094+0000) 2024-03-06T12:42:08.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:08.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:08.838341+0000) 2024-03-06T12:42:09.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:09.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:09.838571+0000) 2024-03-06T12:42:10.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:10.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:10.838812+0000) 2024-03-06T12:42:11.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:11.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:11.839058+0000) 2024-03-06T12:42:12.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:12.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:12.839253+0000) 2024-03-06T12:42:13.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:13.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:13.839487+0000) 2024-03-06T12:42:14.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:14.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:14.839759+0000) 2024-03-06T12:42:15.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:15.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:15.839989+0000) 2024-03-06T12:42:16.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:16.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:16.840213+0000) 2024-03-06T12:42:17.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:17.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:17.840452+0000) 2024-03-06T12:42:18.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:18.766 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:18.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:18.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:18.840625+0000) 2024-03-06T12:42:19.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:19.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:19.840787+0000) 2024-03-06T12:42:20.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:20.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:20.840979+0000) 2024-03-06T12:42:21.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:21.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:21.841207+0000) 2024-03-06T12:42:22.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:22.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:22.841438+0000) 2024-03-06T12:42:23.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:23.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:23.841633+0000) 2024-03-06T12:42:24.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:24.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:24.841920+0000) 2024-03-06T12:42:25.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:25.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:25.842122+0000) 2024-03-06T12:42:26.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:26.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:26.842311+0000) 2024-03-06T12:42:27.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:27.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:27.842569+0000) 2024-03-06T12:42:28.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:28.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:28.842842+0000) 2024-03-06T12:42:29.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:29.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:29.843077+0000) 2024-03-06T12:42:30.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:30.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:30.843317+0000) 2024-03-06T12:42:31.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:31.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:31.843548+0000) 2024-03-06T12:42:32.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:32.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:32.843792+0000) 2024-03-06T12:42:33.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:33.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:33.843995+0000) 2024-03-06T12:42:34.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:34.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:34.844226+0000) 2024-03-06T12:42:35.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:35.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:35.844495+0000) 2024-03-06T12:42:36.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:36.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:36.844769+0000) 2024-03-06T12:42:37.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:37.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:37.844965+0000) 2024-03-06T12:42:38.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:38.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:38.845193+0000) 2024-03-06T12:42:39.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:39.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:39.845380+0000) 2024-03-06T12:42:40.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:40.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:40.845567+0000) 2024-03-06T12:42:41.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:41.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:41.845789+0000) 2024-03-06T12:42:42.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:42.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:42.846013+0000) 2024-03-06T12:42:43.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:43.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:43.846288+0000) 2024-03-06T12:42:44.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:44.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:44.846496+0000) 2024-03-06T12:42:45.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:45.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:45.846785+0000) 2024-03-06T12:42:46.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:46.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:46.847014+0000) 2024-03-06T12:42:47.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:47.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:47.847304+0000) 2024-03-06T12:42:48.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:48.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:48.847531+0000) 2024-03-06T12:42:48.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:48.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:49.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:49.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:49.847786+0000) 2024-03-06T12:42:50.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:50.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:50.847999+0000) 2024-03-06T12:42:51.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:51.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:51.848238+0000) 2024-03-06T12:42:52.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:52.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:52.848506+0000) 2024-03-06T12:42:53.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:53.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:53.848817+0000) 2024-03-06T12:42:54.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:54.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:54.849065+0000) 2024-03-06T12:42:55.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:55.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:55.849262+0000) 2024-03-06T12:42:56.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:56.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:56.849532+0000) 2024-03-06T12:42:57.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:57.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:57.849783+0000) 2024-03-06T12:42:58.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:58.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:58.849968+0000) 2024-03-06T12:42:59.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:42:59.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:59.850226+0000) 2024-03-06T12:43:00.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:00.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:00.850506+0000) 2024-03-06T12:43:01.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:01.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:01.850745+0000) 2024-03-06T12:43:02.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:02.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:02.850976+0000) 2024-03-06T12:43:03.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:03.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:03.851203+0000) 2024-03-06T12:43:04.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:04.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:04.851383+0000) 2024-03-06T12:43:05.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:05.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:05.851630+0000) 2024-03-06T12:43:06.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:06.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:06.851883+0000) 2024-03-06T12:43:07.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:07.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:07.852095+0000) 2024-03-06T12:43:08.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:08.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:08.852303+0000) 2024-03-06T12:43:09.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:09.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:09.852528+0000) 2024-03-06T12:43:10.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:10.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:10.852875+0000) 2024-03-06T12:43:11.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:11.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:11.853121+0000) 2024-03-06T12:43:12.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:12.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:12.853311+0000) 2024-03-06T12:43:13.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:13.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:13.853562+0000) 2024-03-06T12:43:14.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:14.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:14.853817+0000) 2024-03-06T12:43:15.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:15.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:15.854018+0000) 2024-03-06T12:43:16.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:16.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:16.854185+0000) 2024-03-06T12:43:17.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:17.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:17.854376+0000) 2024-03-06T12:43:18.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:18.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:18.854601+0000) 2024-03-06T12:43:19.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:19.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:19.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:19.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:19.854829+0000) 2024-03-06T12:43:20.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:20.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:20.855013+0000) 2024-03-06T12:43:21.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:21.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:21.855147+0000) 2024-03-06T12:43:22.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:22.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:22.855366+0000) 2024-03-06T12:43:23.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:23.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:23.855610+0000) 2024-03-06T12:43:24.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:24.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:24.855818+0000) 2024-03-06T12:43:25.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:25.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:25.856025+0000) 2024-03-06T12:43:26.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:26.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:26.856201+0000) 2024-03-06T12:43:27.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:27.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:27.856419+0000) 2024-03-06T12:43:28.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:28.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:28.856702+0000) 2024-03-06T12:43:29.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:29.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:29.856937+0000) 2024-03-06T12:43:30.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:30.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:30.857120+0000) 2024-03-06T12:43:31.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:31.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:31.857340+0000) 2024-03-06T12:43:32.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:32.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:32.857576+0000) 2024-03-06T12:43:33.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:33.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:33.857811+0000) 2024-03-06T12:43:34.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:34.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:34.858003+0000) 2024-03-06T12:43:35.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:35.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:35.858276+0000) 2024-03-06T12:43:36.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:36.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:36.858542+0000) 2024-03-06T12:43:37.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:37.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:37.858837+0000) 2024-03-06T12:43:38.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:38.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:38.859089+0000) 2024-03-06T12:43:39.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:39.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:39.859363+0000) 2024-03-06T12:43:40.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:40.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:40.859641+0000) 2024-03-06T12:43:41.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:41.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:41.859923+0000) 2024-03-06T12:43:42.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:42.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:42.860174+0000) 2024-03-06T12:43:43.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:43.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:43.860383+0000) 2024-03-06T12:43:44.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:44.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:44.860568+0000) 2024-03-06T12:43:45.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:45.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:45.860798+0000) 2024-03-06T12:43:46.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:46.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:46.861036+0000) 2024-03-06T12:43:47.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:47.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:47.861274+0000) 2024-03-06T12:43:48.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:48.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:48.861488+0000) 2024-03-06T12:43:49.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:49.308 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:49.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:49.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:49.861736+0000) 2024-03-06T12:43:50.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:50.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:50.861952+0000) 2024-03-06T12:43:51.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:51.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:51.862195+0000) 2024-03-06T12:43:52.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:52.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:52.862434+0000) 2024-03-06T12:43:53.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:53.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:53.862623+0000) 2024-03-06T12:43:54.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:54.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:54.862809+0000) 2024-03-06T12:43:55.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:55.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:55.863014+0000) 2024-03-06T12:43:56.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:56.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:56.863235+0000) 2024-03-06T12:43:57.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:57.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:57.863470+0000) 2024-03-06T12:43:58.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:58.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:58.863709+0000) 2024-03-06T12:43:59.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:43:59.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:59.863925+0000) 2024-03-06T12:44:00.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:00.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:00.864162+0000) 2024-03-06T12:44:01.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:01.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:01.864354+0000) 2024-03-06T12:44:02.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:02.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:02.864598+0000) 2024-03-06T12:44:03.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:03.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:03.864817+0000) 2024-03-06T12:44:04.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:04.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:04.865039+0000) 2024-03-06T12:44:05.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:05.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:05.865206+0000) 2024-03-06T12:44:06.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:06.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:06.865473+0000) 2024-03-06T12:44:07.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:07.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:07.865776+0000) 2024-03-06T12:44:08.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:08.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:08.865963+0000) 2024-03-06T12:44:09.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:09.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:09.866142+0000) 2024-03-06T12:44:10.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:10.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:10.866374+0000) 2024-03-06T12:44:11.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:11.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:11.866595+0000) 2024-03-06T12:44:12.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:12.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:12.866837+0000) 2024-03-06T12:44:13.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:13.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:13.867018+0000) 2024-03-06T12:44:14.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:14.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:14.867208+0000) 2024-03-06T12:44:15.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:15.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:15.867407+0000) 2024-03-06T12:44:16.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:16.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:16.867633+0000) 2024-03-06T12:44:17.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:17.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:17.867885+0000) 2024-03-06T12:44:18.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:18.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:18.868094+0000) 2024-03-06T12:44:19.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:19.517 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:19.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:19.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:19.868330+0000) 2024-03-06T12:44:20.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:20.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:20.868517+0000) 2024-03-06T12:44:21.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:21.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:21.868783+0000) 2024-03-06T12:44:22.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:22.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:22.868984+0000) 2024-03-06T12:44:23.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:23.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:23.869167+0000) 2024-03-06T12:44:24.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:24.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:24.869373+0000) 2024-03-06T12:44:25.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:25.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:25.869601+0000) 2024-03-06T12:44:26.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:26.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:26.869813+0000) 2024-03-06T12:44:27.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:27.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:27.870052+0000) 2024-03-06T12:44:28.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:28.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:28.870263+0000) 2024-03-06T12:44:29.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:29.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:29.870443+0000) 2024-03-06T12:44:30.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:30.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:30.870661+0000) 2024-03-06T12:44:31.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:31.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:31.870903+0000) 2024-03-06T12:44:32.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:32.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:32.871157+0000) 2024-03-06T12:44:33.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:33.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:33.871357+0000) 2024-03-06T12:44:34.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:34.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:34.871558+0000) 2024-03-06T12:44:35.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:35.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:35.871802+0000) 2024-03-06T12:44:36.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:36.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:36.872026+0000) 2024-03-06T12:44:37.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:37.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:37.872274+0000) 2024-03-06T12:44:38.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:38.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:38.872514+0000) 2024-03-06T12:44:39.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:39.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:39.872723+0000) 2024-03-06T12:44:40.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:40.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:40.872958+0000) 2024-03-06T12:44:41.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:41.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:41.873190+0000) 2024-03-06T12:44:42.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:42.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:42.873409+0000) 2024-03-06T12:44:43.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:43.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:43.873646+0000) 2024-03-06T12:44:44.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:44.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:44.873910+0000) 2024-03-06T12:44:45.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:45.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:45.874146+0000) 2024-03-06T12:44:46.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:46.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:46.874411+0000) 2024-03-06T12:44:47.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:47.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:47.874762+0000) 2024-03-06T12:44:48.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:48.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:48.874973+0000) 2024-03-06T12:44:49.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:49.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:49.875208+0000) 2024-03-06T12:44:49.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:49.954 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:50.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:50.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:50.875399+0000) 2024-03-06T12:44:51.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:51.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:51.875599+0000) 2024-03-06T12:44:52.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:52.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:52.875850+0000) 2024-03-06T12:44:53.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:53.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:53.876084+0000) 2024-03-06T12:44:54.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:54.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:54.876323+0000) 2024-03-06T12:44:55.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:55.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:55.876549+0000) 2024-03-06T12:44:56.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:56.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:56.876800+0000) 2024-03-06T12:44:57.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:57.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:57.877019+0000) 2024-03-06T12:44:58.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:58.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:58.877266+0000) 2024-03-06T12:44:59.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:44:59.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:59.877490+0000) 2024-03-06T12:45:00.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:00.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:00.877765+0000) 2024-03-06T12:45:01.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:01.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:01.877952+0000) 2024-03-06T12:45:02.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:02.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:02.878214+0000) 2024-03-06T12:45:03.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:03.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:03.878456+0000) 2024-03-06T12:45:04.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:04.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:04.878736+0000) 2024-03-06T12:45:05.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:05.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:05.878980+0000) 2024-03-06T12:45:06.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:06.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:06.879221+0000) 2024-03-06T12:45:07.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:07.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:07.879442+0000) 2024-03-06T12:45:08.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:08.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:08.879720+0000) 2024-03-06T12:45:09.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:09.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:09.879921+0000) 2024-03-06T12:45:10.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:10.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:10.880147+0000) 2024-03-06T12:45:11.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:11.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:11.880337+0000) 2024-03-06T12:45:12.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:12.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:12.880591+0000) 2024-03-06T12:45:13.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:13.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:13.880821+0000) 2024-03-06T12:45:14.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:14.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:14.881026+0000) 2024-03-06T12:45:15.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:15.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:15.881240+0000) 2024-03-06T12:45:16.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:16.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:16.881480+0000) 2024-03-06T12:45:17.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:17.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:17.881799+0000) 2024-03-06T12:45:18.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:18.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:18.882016+0000) 2024-03-06T12:45:19.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:19.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:19.882236+0000) 2024-03-06T12:45:20.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:20.098 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:20.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:20.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:20.882454+0000) 2024-03-06T12:45:21.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:21.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:21.882741+0000) 2024-03-06T12:45:22.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:22.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:22.882949+0000) 2024-03-06T12:45:23.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:23.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:23.883146+0000) 2024-03-06T12:45:24.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:24.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:24.883381+0000) 2024-03-06T12:45:25.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:25.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:25.883860+0000) 2024-03-06T12:45:26.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:26.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:26.884074+0000) 2024-03-06T12:45:27.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:27.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:27.884321+0000) 2024-03-06T12:45:28.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:28.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:28.884522+0000) 2024-03-06T12:45:29.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:29.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:29.884819+0000) 2024-03-06T12:45:30.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:30.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:30.885061+0000) 2024-03-06T12:45:31.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:31.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:31.885333+0000) 2024-03-06T12:45:32.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:32.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:32.885556+0000) 2024-03-06T12:45:33.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:33.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:33.885786+0000) 2024-03-06T12:45:34.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:34.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:34.886005+0000) 2024-03-06T12:45:35.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:35.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:35.886218+0000) 2024-03-06T12:45:36.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:36.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:36.886421+0000) 2024-03-06T12:45:37.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:37.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:37.886643+0000) 2024-03-06T12:45:38.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:38.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:38.886910+0000) 2024-03-06T12:45:39.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:39.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:39.887134+0000) 2024-03-06T12:45:40.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:40.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:40.887315+0000) 2024-03-06T12:45:41.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:41.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:41.887546+0000) 2024-03-06T12:45:42.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:42.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:42.887794+0000) 2024-03-06T12:45:43.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:43.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:43.888030+0000) 2024-03-06T12:45:44.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:44.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:44.888239+0000) 2024-03-06T12:45:45.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:45.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:45.888482+0000) 2024-03-06T12:45:46.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:46.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:46.888660+0000) 2024-03-06T12:45:47.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:47.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:47.889042+0000) 2024-03-06T12:45:48.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:48.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:48.889313+0000) 2024-03-06T12:45:49.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:49.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:49.889615+0000) 2024-03-06T12:45:50.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:50.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:50.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:50.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:50.889875+0000) 2024-03-06T12:45:51.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:51.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:51.890081+0000) 2024-03-06T12:45:52.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:52.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:52.890350+0000) 2024-03-06T12:45:53.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:53.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:53.890580+0000) 2024-03-06T12:45:54.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:54.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:54.890850+0000) 2024-03-06T12:45:55.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:55.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:55.891094+0000) 2024-03-06T12:45:56.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:56.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:56.891353+0000) 2024-03-06T12:45:57.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:57.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:57.891653+0000) 2024-03-06T12:45:58.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:58.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:58.891947+0000) 2024-03-06T12:45:59.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:45:59.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:59.892233+0000) 2024-03-06T12:46:00.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:00.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:00.892451+0000) 2024-03-06T12:46:01.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:01.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:01.892650+0000) 2024-03-06T12:46:02.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:02.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:02.892929+0000) 2024-03-06T12:46:03.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:03.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:03.893151+0000) 2024-03-06T12:46:04.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:04.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:04.893419+0000) 2024-03-06T12:46:05.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:05.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:05.893652+0000) 2024-03-06T12:46:06.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:06.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:06.893962+0000) 2024-03-06T12:46:07.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:07.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:07.894218+0000) 2024-03-06T12:46:08.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:08.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:08.894427+0000) 2024-03-06T12:46:09.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:09.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:09.894703+0000) 2024-03-06T12:46:10.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:10.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:10.894935+0000) 2024-03-06T12:46:11.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:11.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:11.895180+0000) 2024-03-06T12:46:12.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:12.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:12.895428+0000) 2024-03-06T12:46:13.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:13.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:13.895658+0000) 2024-03-06T12:46:14.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:14.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:14.895921+0000) 2024-03-06T12:46:15.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:15.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:15.896139+0000) 2024-03-06T12:46:16.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:16.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:16.896339+0000) 2024-03-06T12:46:17.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:17.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:17.896596+0000) 2024-03-06T12:46:18.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:18.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:18.896821+0000) 2024-03-06T12:46:19.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:19.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:19.897051+0000) 2024-03-06T12:46:20.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:20.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:20.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:20.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:20.897252+0000) 2024-03-06T12:46:21.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:21.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:21.897459+0000) 2024-03-06T12:46:22.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:22.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:22.897768+0000) 2024-03-06T12:46:23.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:23.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:23.898001+0000) 2024-03-06T12:46:24.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:24.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:24.898226+0000) 2024-03-06T12:46:25.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:25.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:25.898463+0000) 2024-03-06T12:46:26.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:26.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:26.898722+0000) 2024-03-06T12:46:27.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:27.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:27.898954+0000) 2024-03-06T12:46:28.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:28.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:28.899179+0000) 2024-03-06T12:46:29.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:29.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:29.899405+0000) 2024-03-06T12:46:30.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:30.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:30.899602+0000) 2024-03-06T12:46:31.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:31.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:31.899846+0000) 2024-03-06T12:46:32.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:32.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:32.900072+0000) 2024-03-06T12:46:33.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:33.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:33.900322+0000) 2024-03-06T12:46:34.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:34.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:34.900561+0000) 2024-03-06T12:46:35.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:35.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:35.900890+0000) 2024-03-06T12:46:36.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:36.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:36.901123+0000) 2024-03-06T12:46:37.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:37.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:37.901331+0000) 2024-03-06T12:46:38.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:38.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:38.901521+0000) 2024-03-06T12:46:39.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:39.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:39.901731+0000) 2024-03-06T12:46:40.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:40.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:40.901951+0000) 2024-03-06T12:46:41.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:41.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:41.902131+0000) 2024-03-06T12:46:42.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:42.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:42.902303+0000) 2024-03-06T12:46:43.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:43.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:43.902535+0000) 2024-03-06T12:46:44.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:44.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:44.902778+0000) 2024-03-06T12:46:45.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:45.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:45.902981+0000) 2024-03-06T12:46:46.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:46.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:46.903166+0000) 2024-03-06T12:46:47.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:47.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:47.903397+0000) 2024-03-06T12:46:48.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:48.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:48.903626+0000) 2024-03-06T12:46:49.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:49.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:49.903874+0000) 2024-03-06T12:46:50.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:50.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:50.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:50.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:50.904081+0000) 2024-03-06T12:46:51.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:51.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:51.904313+0000) 2024-03-06T12:46:52.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:52.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:52.904541+0000) 2024-03-06T12:46:53.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:53.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:53.904768+0000) 2024-03-06T12:46:54.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:54.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:54.904996+0000) 2024-03-06T12:46:55.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:55.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:55.905287+0000) 2024-03-06T12:46:56.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:56.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:56.905569+0000) 2024-03-06T12:46:57.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:57.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:57.905817+0000) 2024-03-06T12:46:58.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:58.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:58.906038+0000) 2024-03-06T12:46:59.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:46:59.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:59.906266+0000) 2024-03-06T12:47:00.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:00.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:00.906479+0000) 2024-03-06T12:47:01.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:01.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:01.906734+0000) 2024-03-06T12:47:02.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:02.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:02.906990+0000) 2024-03-06T12:47:03.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:03.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:03.907220+0000) 2024-03-06T12:47:04.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:04.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:04.907456+0000) 2024-03-06T12:47:05.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:05.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:05.907728+0000) 2024-03-06T12:47:06.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:06.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:06.907957+0000) 2024-03-06T12:47:07.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:07.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:07.908197+0000) 2024-03-06T12:47:08.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:08.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:08.908427+0000) 2024-03-06T12:47:09.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:09.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:09.908630+0000) 2024-03-06T12:47:10.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:10.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:10.908918+0000) 2024-03-06T12:47:11.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:11.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:11.909150+0000) 2024-03-06T12:47:12.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:12.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:12.909440+0000) 2024-03-06T12:47:13.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:13.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:13.909704+0000) 2024-03-06T12:47:14.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:14.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:14.909946+0000) 2024-03-06T12:47:15.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:15.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:15.910221+0000) 2024-03-06T12:47:16.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:16.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:16.910487+0000) 2024-03-06T12:47:17.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:17.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:17.910809+0000) 2024-03-06T12:47:18.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:18.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:18.911058+0000) 2024-03-06T12:47:19.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:19.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:19.911242+0000) 2024-03-06T12:47:20.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:20.857 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:20.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:20.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:20.911425+0000) 2024-03-06T12:47:21.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:21.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:21.911701+0000) 2024-03-06T12:47:22.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:22.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:22.911952+0000) 2024-03-06T12:47:23.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:23.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:23.912302+0000) 2024-03-06T12:47:24.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:24.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:24.912580+0000) 2024-03-06T12:47:25.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:25.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:25.912860+0000) 2024-03-06T12:47:26.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:26.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:26.913103+0000) 2024-03-06T12:47:27.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:27.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:27.913391+0000) 2024-03-06T12:47:28.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:28.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:28.913708+0000) 2024-03-06T12:47:29.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:29.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:29.913917+0000) 2024-03-06T12:47:30.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:30.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:30.914148+0000) 2024-03-06T12:47:31.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:31.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:31.914389+0000) 2024-03-06T12:47:32.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:32.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:32.914608+0000) 2024-03-06T12:47:33.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:33.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:33.914864+0000) 2024-03-06T12:47:34.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:34.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:34.915103+0000) 2024-03-06T12:47:35.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:35.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:35.915366+0000) 2024-03-06T12:47:36.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:36.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:36.915598+0000) 2024-03-06T12:47:37.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:37.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:37.915849+0000) 2024-03-06T12:47:38.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:38.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:38.916082+0000) 2024-03-06T12:47:39.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:39.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:39.916265+0000) 2024-03-06T12:47:40.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:40.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:40.916530+0000) 2024-03-06T12:47:41.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:41.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:41.916766+0000) 2024-03-06T12:47:42.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:42.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:42.916962+0000) 2024-03-06T12:47:43.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:43.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:43.917191+0000) 2024-03-06T12:47:44.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:44.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:44.917419+0000) 2024-03-06T12:47:45.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:45.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:45.917647+0000) 2024-03-06T12:47:46.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:46.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:46.917917+0000) 2024-03-06T12:47:47.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:47.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:47.918158+0000) 2024-03-06T12:47:48.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:48.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:48.918404+0000) 2024-03-06T12:47:49.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:49.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:49.918649+0000) 2024-03-06T12:47:50.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:50.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:50.918956+0000) 2024-03-06T12:47:51.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:51.012 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:51.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:51.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:51.919162+0000) 2024-03-06T12:47:52.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:52.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:52.919408+0000) 2024-03-06T12:47:53.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:53.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:53.919635+0000) 2024-03-06T12:47:54.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:54.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:54.919888+0000) 2024-03-06T12:47:55.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:55.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:55.920046+0000) 2024-03-06T12:47:56.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:56.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:56.920244+0000) 2024-03-06T12:47:57.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:57.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:57.920444+0000) 2024-03-06T12:47:58.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:58.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:58.920638+0000) 2024-03-06T12:47:59.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:47:59.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:59.920874+0000) 2024-03-06T12:48:00.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:00.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:00.921069+0000) 2024-03-06T12:48:01.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:01.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:01.921278+0000) 2024-03-06T12:48:02.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:02.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:02.921519+0000) 2024-03-06T12:48:03.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:03.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:03.921810+0000) 2024-03-06T12:48:04.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:04.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:04.922005+0000) 2024-03-06T12:48:05.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:05.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:05.922248+0000) 2024-03-06T12:48:06.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:06.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:06.922498+0000) 2024-03-06T12:48:07.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:07.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:07.922790+0000) 2024-03-06T12:48:08.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:08.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:08.922980+0000) 2024-03-06T12:48:09.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:09.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:09.923223+0000) 2024-03-06T12:48:10.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:10.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:10.923460+0000) 2024-03-06T12:48:11.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:11.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:11.923731+0000) 2024-03-06T12:48:12.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:12.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:12.923970+0000) 2024-03-06T12:48:13.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:13.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:13.924183+0000) 2024-03-06T12:48:14.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:14.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:14.924379+0000) 2024-03-06T12:48:15.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:15.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:15.924560+0000) 2024-03-06T12:48:16.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:16.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:16.924855+0000) 2024-03-06T12:48:17.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:17.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:17.925087+0000) 2024-03-06T12:48:18.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:18.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:18.925288+0000) 2024-03-06T12:48:19.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:19.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:19.925481+0000) 2024-03-06T12:48:20.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:20.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:20.925762+0000) 2024-03-06T12:48:21.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:21.227 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:21.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:21.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:21.925962+0000) 2024-03-06T12:48:22.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:22.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:22.926208+0000) 2024-03-06T12:48:23.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:23.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:23.926442+0000) 2024-03-06T12:48:24.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:24.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:24.926658+0000) 2024-03-06T12:48:25.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:25.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:25.926910+0000) 2024-03-06T12:48:26.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:26.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:26.927097+0000) 2024-03-06T12:48:27.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:27.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:27.927335+0000) 2024-03-06T12:48:28.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:28.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:28.927577+0000) 2024-03-06T12:48:29.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:29.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:29.927828+0000) 2024-03-06T12:48:30.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:30.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:30.928015+0000) 2024-03-06T12:48:31.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:31.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:31.928197+0000) 2024-03-06T12:48:32.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:32.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:32.928399+0000) 2024-03-06T12:48:33.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:33.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:33.928607+0000) 2024-03-06T12:48:34.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:34.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:34.928834+0000) 2024-03-06T12:48:35.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:35.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:35.928983+0000) 2024-03-06T12:48:36.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:36.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:36.929183+0000) 2024-03-06T12:48:37.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:37.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:37.929431+0000) 2024-03-06T12:48:38.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:38.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:38.929694+0000) 2024-03-06T12:48:39.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:39.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:39.929880+0000) 2024-03-06T12:48:40.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:40.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:40.930096+0000) 2024-03-06T12:48:41.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:41.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:41.930345+0000) 2024-03-06T12:48:42.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:42.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:42.930588+0000) 2024-03-06T12:48:43.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:43.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:43.930816+0000) 2024-03-06T12:48:44.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:44.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:44.931002+0000) 2024-03-06T12:48:45.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:45.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:45.931231+0000) 2024-03-06T12:48:46.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:46.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:46.931456+0000) 2024-03-06T12:48:47.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:47.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:47.931656+0000) 2024-03-06T12:48:48.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:48.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:48.931852+0000) 2024-03-06T12:48:49.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:49.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:49.932079+0000) 2024-03-06T12:48:50.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:50.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:50.932311+0000) 2024-03-06T12:48:51.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:51.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:51.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:51.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:51.932522+0000) 2024-03-06T12:48:52.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:52.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:52.932773+0000) 2024-03-06T12:48:53.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:53.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:53.932996+0000) 2024-03-06T12:48:54.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:54.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:54.933241+0000) 2024-03-06T12:48:55.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:55.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:55.933492+0000) 2024-03-06T12:48:56.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:56.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:56.933763+0000) 2024-03-06T12:48:57.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:57.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:57.933968+0000) 2024-03-06T12:48:58.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:58.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:58.934165+0000) 2024-03-06T12:48:59.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:48:59.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:59.934418+0000) 2024-03-06T12:49:00.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:00.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:00.934649+0000) 2024-03-06T12:49:01.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:01.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:01.934880+0000) 2024-03-06T12:49:02.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:02.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:02.935110+0000) 2024-03-06T12:49:03.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:03.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:03.935301+0000) 2024-03-06T12:49:04.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:04.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:04.935579+0000) 2024-03-06T12:49:05.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:05.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:05.935819+0000) 2024-03-06T12:49:06.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:06.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:06.936007+0000) 2024-03-06T12:49:07.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:07.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:07.936262+0000) 2024-03-06T12:49:08.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:08.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:08.936510+0000) 2024-03-06T12:49:09.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:09.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:09.936808+0000) 2024-03-06T12:49:10.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:10.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:10.937002+0000) 2024-03-06T12:49:11.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:11.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:11.937225+0000) 2024-03-06T12:49:12.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:12.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:12.937407+0000) 2024-03-06T12:49:13.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:13.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:13.937640+0000) 2024-03-06T12:49:14.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:14.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:14.937905+0000) 2024-03-06T12:49:15.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:15.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:15.938157+0000) 2024-03-06T12:49:16.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:16.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:16.938401+0000) 2024-03-06T12:49:17.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:17.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:17.938643+0000) 2024-03-06T12:49:18.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:18.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:18.938899+0000) 2024-03-06T12:49:19.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:19.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:19.939102+0000) 2024-03-06T12:49:20.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:20.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:20.939306+0000) 2024-03-06T12:49:21.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:21.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:21.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:21.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:21.939543+0000) 2024-03-06T12:49:22.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:22.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:22.939769+0000) 2024-03-06T12:49:23.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:23.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:23.939926+0000) 2024-03-06T12:49:24.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:24.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:24.940150+0000) 2024-03-06T12:49:25.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:25.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:25.940384+0000) 2024-03-06T12:49:26.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:26.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:26.940625+0000) 2024-03-06T12:49:27.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:27.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:27.940841+0000) 2024-03-06T12:49:28.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:28.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:28.941054+0000) 2024-03-06T12:49:29.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:29.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:29.941296+0000) 2024-03-06T12:49:30.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:30.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:30.941540+0000) 2024-03-06T12:49:31.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:31.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:31.941787+0000) 2024-03-06T12:49:32.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:32.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:32.941989+0000) 2024-03-06T12:49:33.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:33.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:33.942217+0000) 2024-03-06T12:49:34.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:34.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:34.942441+0000) 2024-03-06T12:49:35.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:35.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:35.942719+0000) 2024-03-06T12:49:36.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:36.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:36.942932+0000) 2024-03-06T12:49:37.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:37.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:37.943107+0000) 2024-03-06T12:49:38.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:38.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:38.943331+0000) 2024-03-06T12:49:39.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:39.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:39.943566+0000) 2024-03-06T12:49:40.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:40.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:40.943796+0000) 2024-03-06T12:49:41.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:41.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:41.944023+0000) 2024-03-06T12:49:42.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:42.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:42.944222+0000) 2024-03-06T12:49:43.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:43.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:43.944442+0000) 2024-03-06T12:49:44.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:44.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:44.944656+0000) 2024-03-06T12:49:45.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:45.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:45.944949+0000) 2024-03-06T12:49:46.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:46.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:46.945178+0000) 2024-03-06T12:49:47.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:47.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:47.945422+0000) 2024-03-06T12:49:48.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:48.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:48.945658+0000) 2024-03-06T12:49:49.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:49.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:49.945895+0000) 2024-03-06T12:49:50.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:50.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:50.946093+0000) 2024-03-06T12:49:51.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:51.812 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:51.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:51.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:51.946347+0000) 2024-03-06T12:49:52.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:52.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:52.946635+0000) 2024-03-06T12:49:53.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:53.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:53.946832+0000) 2024-03-06T12:49:54.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:54.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:54.947067+0000) 2024-03-06T12:49:55.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:55.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:55.947249+0000) 2024-03-06T12:49:56.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:56.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:56.947483+0000) 2024-03-06T12:49:57.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:57.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:57.947759+0000) 2024-03-06T12:49:58.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:58.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:58.947994+0000) 2024-03-06T12:49:59.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:49:59.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:59.948192+0000) 2024-03-06T12:50:00.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:00.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:00.948426+0000) 2024-03-06T12:50:01.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:01.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:01.948643+0000) 2024-03-06T12:50:02.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:02.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:02.948883+0000) 2024-03-06T12:50:03.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:03.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:03.949138+0000) 2024-03-06T12:50:04.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:04.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:04.949358+0000) 2024-03-06T12:50:05.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:05.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:05.949587+0000) 2024-03-06T12:50:06.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:06.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:06.949814+0000) 2024-03-06T12:50:07.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:07.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:07.950028+0000) 2024-03-06T12:50:08.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:08.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:08.950227+0000) 2024-03-06T12:50:09.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:09.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:09.950467+0000) 2024-03-06T12:50:10.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:10.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:10.950693+0000) 2024-03-06T12:50:11.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:11.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:11.950903+0000) 2024-03-06T12:50:12.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:12.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:12.951123+0000) 2024-03-06T12:50:13.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:13.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:13.951330+0000) 2024-03-06T12:50:14.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:14.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:14.951572+0000) 2024-03-06T12:50:15.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:15.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:15.951799+0000) 2024-03-06T12:50:16.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:16.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:16.952007+0000) 2024-03-06T12:50:17.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:17.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:17.952205+0000) 2024-03-06T12:50:18.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:18.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:18.952447+0000) 2024-03-06T12:50:19.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:19.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:19.952646+0000) 2024-03-06T12:50:20.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:20.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:20.952842+0000) 2024-03-06T12:50:21.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:21.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:21.953033+0000) 2024-03-06T12:50:21.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:21.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:22.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:22.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:22.953265+0000) 2024-03-06T12:50:23.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:23.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:23.953471+0000) 2024-03-06T12:50:24.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:24.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:24.953752+0000) 2024-03-06T12:50:25.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:25.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:25.953956+0000) 2024-03-06T12:50:26.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:26.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:26.954149+0000) 2024-03-06T12:50:27.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:27.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:27.954354+0000) 2024-03-06T12:50:28.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:28.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:28.954650+0000) 2024-03-06T12:50:29.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:29.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:29.954922+0000) 2024-03-06T12:50:30.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:30.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:30.955130+0000) 2024-03-06T12:50:31.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:31.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:31.955394+0000) 2024-03-06T12:50:32.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:32.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:32.955593+0000) 2024-03-06T12:50:33.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:33.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:33.955832+0000) 2024-03-06T12:50:34.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:34.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:34.956060+0000) 2024-03-06T12:50:35.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:35.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:35.956287+0000) 2024-03-06T12:50:36.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:36.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:36.956511+0000) 2024-03-06T12:50:37.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:37.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:37.956786+0000) 2024-03-06T12:50:38.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:38.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:38.957026+0000) 2024-03-06T12:50:39.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:39.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:39.957261+0000) 2024-03-06T12:50:40.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:40.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:40.957488+0000) 2024-03-06T12:50:41.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:41.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:41.957785+0000) 2024-03-06T12:50:42.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:42.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:42.958037+0000) 2024-03-06T12:50:43.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:43.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:43.958271+0000) 2024-03-06T12:50:44.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:44.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:44.958515+0000) 2024-03-06T12:50:45.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:45.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:45.958785+0000) 2024-03-06T12:50:46.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:46.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:46.958973+0000) 2024-03-06T12:50:47.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:47.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:47.959208+0000) 2024-03-06T12:50:48.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:48.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:48.959404+0000) 2024-03-06T12:50:49.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:49.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:49.959594+0000) 2024-03-06T12:50:50.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:50.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:50.959777+0000) 2024-03-06T12:50:51.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:51.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:51.959971+0000) 2024-03-06T12:50:52.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:52.117 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:52.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:52.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:52.960141+0000) 2024-03-06T12:50:53.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:53.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:53.960388+0000) 2024-03-06T12:50:54.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:54.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:54.960623+0000) 2024-03-06T12:50:55.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:55.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:55.960839+0000) 2024-03-06T12:50:56.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:56.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:56.961048+0000) 2024-03-06T12:50:57.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:57.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:57.961279+0000) 2024-03-06T12:50:58.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:58.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:58.961543+0000) 2024-03-06T12:50:59.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:50:59.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:59.961781+0000) 2024-03-06T12:51:00.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:00.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:00.961989+0000) 2024-03-06T12:51:01.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:01.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:01.962230+0000) 2024-03-06T12:51:02.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:02.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:02.962473+0000) 2024-03-06T12:51:03.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:03.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:03.962781+0000) 2024-03-06T12:51:04.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:04.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:04.962991+0000) 2024-03-06T12:51:05.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:05.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:05.963238+0000) 2024-03-06T12:51:06.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:06.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:06.963481+0000) 2024-03-06T12:51:07.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:07.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:07.963792+0000) 2024-03-06T12:51:08.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:08.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:08.964001+0000) 2024-03-06T12:51:09.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:09.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:09.964219+0000) 2024-03-06T12:51:10.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:10.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:10.964365+0000) 2024-03-06T12:51:11.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:11.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:11.964591+0000) 2024-03-06T12:51:12.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:12.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:12.964796+0000) 2024-03-06T12:51:13.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:13.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:13.965037+0000) 2024-03-06T12:51:14.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:14.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:14.965293+0000) 2024-03-06T12:51:15.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:15.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:15.965540+0000) 2024-03-06T12:51:16.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:16.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:16.965782+0000) 2024-03-06T12:51:17.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:17.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:17.966012+0000) 2024-03-06T12:51:18.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:18.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:18.966269+0000) 2024-03-06T12:51:19.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:19.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:19.966501+0000) 2024-03-06T12:51:20.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:20.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:20.966775+0000) 2024-03-06T12:51:21.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:21.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:21.966962+0000) 2024-03-06T12:51:22.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:22.357 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:22.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:22.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:22.967201+0000) 2024-03-06T12:51:23.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:23.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:23.967447+0000) 2024-03-06T12:51:24.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:24.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:24.967704+0000) 2024-03-06T12:51:25.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:25.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:25.967952+0000) 2024-03-06T12:51:26.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:26.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:26.968161+0000) 2024-03-06T12:51:27.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:27.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:27.968435+0000) 2024-03-06T12:51:28.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:28.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:28.968704+0000) 2024-03-06T12:51:29.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:29.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:29.968920+0000) 2024-03-06T12:51:30.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:30.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:30.969151+0000) 2024-03-06T12:51:31.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:31.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:31.969381+0000) 2024-03-06T12:51:32.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:32.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:32.969612+0000) 2024-03-06T12:51:33.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:33.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:33.969846+0000) 2024-03-06T12:51:34.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:34.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:34.970096+0000) 2024-03-06T12:51:35.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:35.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:35.970331+0000) 2024-03-06T12:51:36.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:36.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:36.970548+0000) 2024-03-06T12:51:37.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:37.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:37.970834+0000) 2024-03-06T12:51:38.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:38.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:38.971062+0000) 2024-03-06T12:51:39.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:39.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:39.971255+0000) 2024-03-06T12:51:40.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:40.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:40.971488+0000) 2024-03-06T12:51:41.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:41.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:41.971763+0000) 2024-03-06T12:51:42.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:42.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:42.971990+0000) 2024-03-06T12:51:43.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:43.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:43.972237+0000) 2024-03-06T12:51:44.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:44.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:44.972432+0000) 2024-03-06T12:51:45.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:45.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:45.972650+0000) 2024-03-06T12:51:46.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:46.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:46.972920+0000) 2024-03-06T12:51:47.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:47.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:47.973161+0000) 2024-03-06T12:51:48.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:48.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:48.973414+0000) 2024-03-06T12:51:49.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:49.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:49.973643+0000) 2024-03-06T12:51:50.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:50.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:50.973921+0000) 2024-03-06T12:51:51.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:51.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:51.974116+0000) 2024-03-06T12:51:52.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:52.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:52.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:52.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:52.974311+0000) 2024-03-06T12:51:53.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:53.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:53.974621+0000) 2024-03-06T12:51:54.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:54.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:54.974903+0000) 2024-03-06T12:51:55.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:55.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:55.975082+0000) 2024-03-06T12:51:56.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:56.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:56.975305+0000) 2024-03-06T12:51:57.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:57.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:57.975494+0000) 2024-03-06T12:51:58.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:58.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:58.975769+0000) 2024-03-06T12:51:59.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:51:59.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:59.975994+0000) 2024-03-06T12:52:00.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:00.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:00.976221+0000) 2024-03-06T12:52:01.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:01.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:01.976446+0000) 2024-03-06T12:52:02.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:02.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:02.976720+0000) 2024-03-06T12:52:03.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:03.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:03.976945+0000) 2024-03-06T12:52:04.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:04.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:04.977169+0000) 2024-03-06T12:52:05.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:05.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:05.977406+0000) 2024-03-06T12:52:06.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:06.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:06.977632+0000) 2024-03-06T12:52:07.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:07.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:07.977890+0000) 2024-03-06T12:52:08.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:08.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:08.978134+0000) 2024-03-06T12:52:09.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:09.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:09.978365+0000) 2024-03-06T12:52:10.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:10.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:10.978643+0000) 2024-03-06T12:52:11.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:11.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:11.978963+0000) 2024-03-06T12:52:12.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:12.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:12.979248+0000) 2024-03-06T12:52:13.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:13.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:13.979530+0000) 2024-03-06T12:52:14.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:14.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:14.979799+0000) 2024-03-06T12:52:15.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:15.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:15.980043+0000) 2024-03-06T12:52:16.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:16.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:16.980266+0000) 2024-03-06T12:52:17.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:17.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:17.980504+0000) 2024-03-06T12:52:18.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:18.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:18.980800+0000) 2024-03-06T12:52:19.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:19.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:19.981038+0000) 2024-03-06T12:52:20.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:20.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:20.981261+0000) 2024-03-06T12:52:21.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:21.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:21.981410+0000) 2024-03-06T12:52:22.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:22.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:22.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:22.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:22.981646+0000) 2024-03-06T12:52:23.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:23.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:23.981868+0000) 2024-03-06T12:52:24.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:24.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:24.982121+0000) 2024-03-06T12:52:25.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:25.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:25.982351+0000) 2024-03-06T12:52:26.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:26.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:26.982593+0000) 2024-03-06T12:52:27.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:27.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:27.982831+0000) 2024-03-06T12:52:28.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:28.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:28.983056+0000) 2024-03-06T12:52:29.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:29.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:29.983298+0000) 2024-03-06T12:52:30.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:30.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:30.983540+0000) 2024-03-06T12:52:31.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:31.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:31.983789+0000) 2024-03-06T12:52:32.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:32.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:32.984027+0000) 2024-03-06T12:52:33.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:33.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:33.984182+0000) 2024-03-06T12:52:34.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:34.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:34.984399+0000) 2024-03-06T12:52:35.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:35.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:35.984628+0000) 2024-03-06T12:52:36.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:36.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:36.984826+0000) 2024-03-06T12:52:37.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:37.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:37.985078+0000) 2024-03-06T12:52:38.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:38.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:38.985318+0000) 2024-03-06T12:52:39.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:39.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:39.985536+0000) 2024-03-06T12:52:40.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:40.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:40.985725+0000) 2024-03-06T12:52:41.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:41.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:41.985888+0000) 2024-03-06T12:52:42.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:42.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:42.986074+0000) 2024-03-06T12:52:43.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:43.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:43.986327+0000) 2024-03-06T12:52:44.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:44.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:44.986562+0000) 2024-03-06T12:52:45.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:45.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:45.986793+0000) 2024-03-06T12:52:46.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:46.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:46.987037+0000) 2024-03-06T12:52:47.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:47.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:47.987290+0000) 2024-03-06T12:52:48.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:48.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:48.987451+0000) 2024-03-06T12:52:49.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:49.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:49.987718+0000) 2024-03-06T12:52:50.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:50.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:50.987986+0000) 2024-03-06T12:52:51.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:51.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:51.988272+0000) 2024-03-06T12:52:52.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:52.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:52.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:52.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:52.988487+0000) 2024-03-06T12:52:53.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:53.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:53.988652+0000) 2024-03-06T12:52:54.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:54.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:54.988879+0000) 2024-03-06T12:52:55.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:55.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:55.989106+0000) 2024-03-06T12:52:56.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:56.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:56.989334+0000) 2024-03-06T12:52:57.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:57.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:57.989507+0000) 2024-03-06T12:52:58.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:58.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:58.989769+0000) 2024-03-06T12:52:59.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:52:59.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:59.989965+0000) 2024-03-06T12:53:00.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:00.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:00.990152+0000) 2024-03-06T12:53:01.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:01.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:01.990382+0000) 2024-03-06T12:53:02.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:02.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:02.990605+0000) 2024-03-06T12:53:03.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:03.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:03.990841+0000) 2024-03-06T12:53:04.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:04.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:04.991075+0000) 2024-03-06T12:53:05.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:05.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:05.991277+0000) 2024-03-06T12:53:06.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:06.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:06.991470+0000) 2024-03-06T12:53:07.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:07.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:07.991803+0000) 2024-03-06T12:53:08.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:08.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:08.992034+0000) 2024-03-06T12:53:09.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:09.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:09.992214+0000) 2024-03-06T12:53:10.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:10.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:10.992433+0000) 2024-03-06T12:53:11.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:11.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:11.992656+0000) 2024-03-06T12:53:12.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:12.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:12.992889+0000) 2024-03-06T12:53:13.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:13.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:13.993074+0000) 2024-03-06T12:53:14.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:14.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:14.993288+0000) 2024-03-06T12:53:15.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:15.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:15.993555+0000) 2024-03-06T12:53:16.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:16.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:16.993789+0000) 2024-03-06T12:53:17.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:17.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:17.994021+0000) 2024-03-06T12:53:18.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:18.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:18.994254+0000) 2024-03-06T12:53:19.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:19.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:19.994488+0000) 2024-03-06T12:53:20.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:20.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:20.994756+0000) 2024-03-06T12:53:21.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:21.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:21.994947+0000) 2024-03-06T12:53:22.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:22.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:22.995170+0000) 2024-03-06T12:53:23.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:23.155 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:23.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:23.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:23.995366+0000) 2024-03-06T12:53:24.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:24.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:24.995596+0000) 2024-03-06T12:53:25.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:25.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:25.995828+0000) 2024-03-06T12:53:26.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:26.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:26.996024+0000) 2024-03-06T12:53:27.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:27.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:27.996232+0000) 2024-03-06T12:53:28.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:28.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:28.996482+0000) 2024-03-06T12:53:29.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:29.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:29.996751+0000) 2024-03-06T12:53:30.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:30.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:30.996973+0000) 2024-03-06T12:53:31.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:31.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:31.997165+0000) 2024-03-06T12:53:32.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:32.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:32.997342+0000) 2024-03-06T12:53:33.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:33.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:33.997568+0000) 2024-03-06T12:53:34.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:34.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:34.997803+0000) 2024-03-06T12:53:35.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:35.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:35.998003+0000) 2024-03-06T12:53:36.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:36.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:36.998246+0000) 2024-03-06T12:53:37.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:37.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:37.998434+0000) 2024-03-06T12:53:38.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:38.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:38.998609+0000) 2024-03-06T12:53:39.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:39.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:39.998822+0000) 2024-03-06T12:53:40.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:40.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:40.999017+0000) 2024-03-06T12:53:41.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:41.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:41.999277+0000) 2024-03-06T12:53:42.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:42.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:42.999555+0000) 2024-03-06T12:53:43.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:43.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:43.999783+0000) 2024-03-06T12:53:44.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:44.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:45.000024+0000) 2024-03-06T12:53:45.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:45.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:46.000256+0000) 2024-03-06T12:53:46.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:46.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:47.000482+0000) 2024-03-06T12:53:47.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:48.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:48.000770+0000) 2024-03-06T12:53:48.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:49.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:49.000955+0000) 2024-03-06T12:53:49.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:50.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:50.001160+0000) 2024-03-06T12:53:50.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:51.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:51.001403+0000) 2024-03-06T12:53:51.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:52.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:52.001601+0000) 2024-03-06T12:53:52.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:53.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:53.001825+0000) 2024-03-06T12:53:53.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:53.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:53.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:54.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:54.002022+0000) 2024-03-06T12:53:54.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:55.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:55.002234+0000) 2024-03-06T12:53:55.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:56.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:56.002469+0000) 2024-03-06T12:53:56.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:57.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:57.002650+0000) 2024-03-06T12:53:57.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:58.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:58.002962+0000) 2024-03-06T12:53:58.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:53:59.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:59.003194+0000) 2024-03-06T12:53:59.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:00.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:00.003428+0000) 2024-03-06T12:54:00.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:01.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:01.003734+0000) 2024-03-06T12:54:01.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:02.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:02.003995+0000) 2024-03-06T12:54:02.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:03.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:03.004179+0000) 2024-03-06T12:54:03.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:04.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:04.004478+0000) 2024-03-06T12:54:04.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:05.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:05.004765+0000) 2024-03-06T12:54:05.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:06.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:06.005005+0000) 2024-03-06T12:54:06.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:07.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:07.005242+0000) 2024-03-06T12:54:07.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:08.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:08.005493+0000) 2024-03-06T12:54:08.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:09.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:09.005742+0000) 2024-03-06T12:54:09.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:10.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:10.005994+0000) 2024-03-06T12:54:10.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:11.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:11.006183+0000) 2024-03-06T12:54:11.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:12.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:12.006417+0000) 2024-03-06T12:54:12.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:13.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:13.006598+0000) 2024-03-06T12:54:13.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:14.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:14.006841+0000) 2024-03-06T12:54:14.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:15.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:15.007088+0000) 2024-03-06T12:54:15.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:16.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:16.007338+0000) 2024-03-06T12:54:16.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:17.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:17.007621+0000) 2024-03-06T12:54:17.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:18.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:18.007904+0000) 2024-03-06T12:54:18.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:19.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:19.008108+0000) 2024-03-06T12:54:19.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:20.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:20.008367+0000) 2024-03-06T12:54:20.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:21.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:21.008603+0000) 2024-03-06T12:54:21.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:22.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:22.008837+0000) 2024-03-06T12:54:22.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:23.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:23.009078+0000) 2024-03-06T12:54:23.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:23.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:23.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:24.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:24.009330+0000) 2024-03-06T12:54:24.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:25.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:25.009579+0000) 2024-03-06T12:54:25.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:26.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:26.009820+0000) 2024-03-06T12:54:26.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:27.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:27.010060+0000) 2024-03-06T12:54:27.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:28.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:28.010295+0000) 2024-03-06T12:54:29.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:29.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:29.010517+0000) 2024-03-06T12:54:30.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:30.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:30.010822+0000) 2024-03-06T12:54:31.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:31.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:31.011056+0000) 2024-03-06T12:54:32.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:32.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:32.011292+0000) 2024-03-06T12:54:33.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:33.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:33.011506+0000) 2024-03-06T12:54:34.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:34.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:34.011787+0000) 2024-03-06T12:54:35.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:35.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:35.012023+0000) 2024-03-06T12:54:36.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:36.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:36.012260+0000) 2024-03-06T12:54:37.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:37.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:37.012500+0000) 2024-03-06T12:54:38.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:38.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:38.012802+0000) 2024-03-06T12:54:39.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:39.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:39.013042+0000) 2024-03-06T12:54:40.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:40.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:40.013228+0000) 2024-03-06T12:54:41.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:41.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:41.013417+0000) 2024-03-06T12:54:42.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:42.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:42.013706+0000) 2024-03-06T12:54:43.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:43.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:43.013907+0000) 2024-03-06T12:54:44.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:44.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:44.014143+0000) 2024-03-06T12:54:45.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:45.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:45.014375+0000) 2024-03-06T12:54:46.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:46.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:46.014574+0000) 2024-03-06T12:54:47.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:47.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:47.014798+0000) 2024-03-06T12:54:48.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:48.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:48.015052+0000) 2024-03-06T12:54:49.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:49.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:49.015297+0000) 2024-03-06T12:54:50.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:50.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:50.015517+0000) 2024-03-06T12:54:51.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:51.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:51.015789+0000) 2024-03-06T12:54:52.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:52.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:52.016033+0000) 2024-03-06T12:54:53.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:53.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:53.016225+0000) 2024-03-06T12:54:53.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:53.760 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:54.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:54.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:54.016424+0000) 2024-03-06T12:54:55.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:55.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:55.016700+0000) 2024-03-06T12:54:56.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:56.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:56.016929+0000) 2024-03-06T12:54:57.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:57.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:57.017169+0000) 2024-03-06T12:54:58.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:58.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:58.017424+0000) 2024-03-06T12:54:59.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:54:59.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:59.017621+0000) 2024-03-06T12:55:00.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:00.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:00.017845+0000) 2024-03-06T12:55:01.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:01.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:01.017976+0000) 2024-03-06T12:55:02.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:02.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:02.018191+0000) 2024-03-06T12:55:03.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:03.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:03.018422+0000) 2024-03-06T12:55:04.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:04.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:04.018636+0000) 2024-03-06T12:55:05.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:05.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:05.018883+0000) 2024-03-06T12:55:06.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:06.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:06.019118+0000) 2024-03-06T12:55:07.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:07.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:07.019348+0000) 2024-03-06T12:55:08.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:08.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:08.019551+0000) 2024-03-06T12:55:09.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:09.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:09.019792+0000) 2024-03-06T12:55:10.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:10.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:10.020020+0000) 2024-03-06T12:55:11.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:11.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:11.020237+0000) 2024-03-06T12:55:12.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:12.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:12.020430+0000) 2024-03-06T12:55:13.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:13.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:13.020714+0000) 2024-03-06T12:55:14.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:14.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:14.020993+0000) 2024-03-06T12:55:15.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:15.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:15.021241+0000) 2024-03-06T12:55:16.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:16.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:16.021423+0000) 2024-03-06T12:55:17.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:17.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:17.021707+0000) 2024-03-06T12:55:18.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:18.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:18.021986+0000) 2024-03-06T12:55:19.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:19.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:19.022235+0000) 2024-03-06T12:55:20.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:20.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:20.022476+0000) 2024-03-06T12:55:21.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:21.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:21.022761+0000) 2024-03-06T12:55:22.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:22.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:22.022923+0000) 2024-03-06T12:55:23.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:23.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:23.023155+0000) 2024-03-06T12:55:23.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:23.959 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:24.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:24.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:24.023321+0000) 2024-03-06T12:55:25.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:25.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:25.023509+0000) 2024-03-06T12:55:26.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:26.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:26.023767+0000) 2024-03-06T12:55:27.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:27.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:27.023983+0000) 2024-03-06T12:55:28.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:28.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:28.024199+0000) 2024-03-06T12:55:29.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:29.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:29.024470+0000) 2024-03-06T12:55:30.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:30.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:30.024747+0000) 2024-03-06T12:55:31.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:31.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:31.024941+0000) 2024-03-06T12:55:32.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:32.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:32.025125+0000) 2024-03-06T12:55:33.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:33.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:33.025314+0000) 2024-03-06T12:55:34.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:34.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:34.025539+0000) 2024-03-06T12:55:35.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:35.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:35.025774+0000) 2024-03-06T12:55:36.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:36.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:36.025968+0000) 2024-03-06T12:55:37.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:37.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:37.026170+0000) 2024-03-06T12:55:38.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:38.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:38.026442+0000) 2024-03-06T12:55:39.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:39.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:39.026653+0000) 2024-03-06T12:55:40.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:40.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:40.026887+0000) 2024-03-06T12:55:41.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:41.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:41.027077+0000) 2024-03-06T12:55:42.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:42.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:42.027265+0000) 2024-03-06T12:55:43.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:43.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:43.027490+0000) 2024-03-06T12:55:44.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:44.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:44.027746+0000) 2024-03-06T12:55:45.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:45.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:45.028074+0000) 2024-03-06T12:55:46.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:46.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:46.028288+0000) 2024-03-06T12:55:47.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:47.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:47.028517+0000) 2024-03-06T12:55:48.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:48.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:48.028766+0000) 2024-03-06T12:55:49.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:49.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:49.028995+0000) 2024-03-06T12:55:50.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:50.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:50.029190+0000) 2024-03-06T12:55:51.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:51.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:51.029386+0000) 2024-03-06T12:55:52.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:52.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:52.029620+0000) 2024-03-06T12:55:53.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:53.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:53.029926+0000) 2024-03-06T12:55:54.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:54.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:54.030160+0000) 2024-03-06T12:55:54.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:54.116 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:55.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:55.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:55.030436+0000) 2024-03-06T12:55:56.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:56.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:56.030736+0000) 2024-03-06T12:55:57.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:57.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:57.030964+0000) 2024-03-06T12:55:58.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:58.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:58.031215+0000) 2024-03-06T12:55:59.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:55:59.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:59.031488+0000) 2024-03-06T12:56:00.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:00.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:00.031812+0000) 2024-03-06T12:56:01.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:01.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:01.032066+0000) 2024-03-06T12:56:02.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:02.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:02.032311+0000) 2024-03-06T12:56:03.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:03.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:03.032588+0000) 2024-03-06T12:56:04.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:04.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:04.032845+0000) 2024-03-06T12:56:05.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:05.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:05.033086+0000) 2024-03-06T12:56:06.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:06.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:06.033369+0000) 2024-03-06T12:56:07.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:07.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:07.033648+0000) 2024-03-06T12:56:08.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:08.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:08.033935+0000) 2024-03-06T12:56:09.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:09.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:09.034113+0000) 2024-03-06T12:56:10.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:10.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:10.034341+0000) 2024-03-06T12:56:11.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:11.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:11.034547+0000) 2024-03-06T12:56:12.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:12.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:12.034834+0000) 2024-03-06T12:56:13.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:13.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:13.035063+0000) 2024-03-06T12:56:14.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:14.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:14.035259+0000) 2024-03-06T12:56:15.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:15.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:15.035505+0000) 2024-03-06T12:56:16.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:16.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:16.035769+0000) 2024-03-06T12:56:17.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:17.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:17.035962+0000) 2024-03-06T12:56:18.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:18.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:18.036177+0000) 2024-03-06T12:56:19.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:19.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:19.036407+0000) 2024-03-06T12:56:20.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:20.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:20.036646+0000) 2024-03-06T12:56:21.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:21.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:21.036904+0000) 2024-03-06T12:56:22.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:22.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:22.037088+0000) 2024-03-06T12:56:23.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:23.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:23.037317+0000) 2024-03-06T12:56:24.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:24.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:24.037547+0000) 2024-03-06T12:56:24.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:24.326 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:25.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:25.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:25.037776+0000) 2024-03-06T12:56:26.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:26.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:26.037991+0000) 2024-03-06T12:56:27.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:27.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:27.038199+0000) 2024-03-06T12:56:28.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:28.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:28.038388+0000) 2024-03-06T12:56:29.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:29.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:29.038620+0000) 2024-03-06T12:56:30.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:30.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:30.038830+0000) 2024-03-06T12:56:31.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:31.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:31.039053+0000) 2024-03-06T12:56:32.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:32.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:32.039277+0000) 2024-03-06T12:56:33.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:33.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:33.039559+0000) 2024-03-06T12:56:34.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:34.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:34.039808+0000) 2024-03-06T12:56:35.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:35.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:35.040046+0000) 2024-03-06T12:56:36.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:36.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:36.040254+0000) 2024-03-06T12:56:37.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:37.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:37.040489+0000) 2024-03-06T12:56:38.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:38.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:38.040646+0000) 2024-03-06T12:56:39.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:39.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:39.040890+0000) 2024-03-06T12:56:40.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:40.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:40.041129+0000) 2024-03-06T12:56:41.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:41.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:41.041344+0000) 2024-03-06T12:56:42.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:42.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:42.041535+0000) 2024-03-06T12:56:43.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:43.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:43.041769+0000) 2024-03-06T12:56:44.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:44.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:44.041998+0000) 2024-03-06T12:56:45.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:45.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:45.042226+0000) 2024-03-06T12:56:46.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:46.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:46.042457+0000) 2024-03-06T12:56:47.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:47.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:47.042729+0000) 2024-03-06T12:56:48.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:48.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:48.042936+0000) 2024-03-06T12:56:49.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:49.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:49.043133+0000) 2024-03-06T12:56:50.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:50.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:50.043335+0000) 2024-03-06T12:56:51.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:51.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:51.043534+0000) 2024-03-06T12:56:52.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:52.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:52.043770+0000) 2024-03-06T12:56:53.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:53.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:53.043965+0000) 2024-03-06T12:56:54.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:54.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:54.044171+0000) 2024-03-06T12:56:54.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:54.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:55.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:55.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:55.044442+0000) 2024-03-06T12:56:56.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:56.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:56.044705+0000) 2024-03-06T12:56:57.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:57.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:57.044957+0000) 2024-03-06T12:56:58.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:58.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:58.045208+0000) 2024-03-06T12:56:59.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:56:59.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:59.045393+0000) 2024-03-06T12:57:00.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:00.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:00.045602+0000) 2024-03-06T12:57:01.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:01.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:01.045832+0000) 2024-03-06T12:57:02.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:02.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:02.046026+0000) 2024-03-06T12:57:03.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:03.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:03.046219+0000) 2024-03-06T12:57:04.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:04.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:04.046423+0000) 2024-03-06T12:57:05.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:05.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:05.046689+0000) 2024-03-06T12:57:06.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:06.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:06.046897+0000) 2024-03-06T12:57:07.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:07.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:07.047104+0000) 2024-03-06T12:57:08.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:08.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:08.047323+0000) 2024-03-06T12:57:09.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:09.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:09.047523+0000) 2024-03-06T12:57:10.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:10.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:10.047825+0000) 2024-03-06T12:57:11.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:11.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:11.048058+0000) 2024-03-06T12:57:12.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:12.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:12.048241+0000) 2024-03-06T12:57:13.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:13.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:13.048436+0000) 2024-03-06T12:57:14.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:14.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:14.048625+0000) 2024-03-06T12:57:15.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:15.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:15.048836+0000) 2024-03-06T12:57:16.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:16.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:16.049025+0000) 2024-03-06T12:57:17.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:17.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:17.049209+0000) 2024-03-06T12:57:18.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:18.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:18.049443+0000) 2024-03-06T12:57:19.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:19.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:19.049620+0000) 2024-03-06T12:57:20.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:20.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:20.049843+0000) 2024-03-06T12:57:21.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:21.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:21.050052+0000) 2024-03-06T12:57:22.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:22.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:22.050236+0000) 2024-03-06T12:57:23.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:23.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:23.050463+0000) 2024-03-06T12:57:24.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:24.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:24.050724+0000) 2024-03-06T12:57:24.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:24.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:25.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:25.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:25.050942+0000) 2024-03-06T12:57:26.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:26.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:26.051185+0000) 2024-03-06T12:57:27.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:27.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:27.051462+0000) 2024-03-06T12:57:28.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:28.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:28.051730+0000) 2024-03-06T12:57:29.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:29.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:29.051967+0000) 2024-03-06T12:57:30.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:30.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:30.052161+0000) 2024-03-06T12:57:31.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:31.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:31.052387+0000) 2024-03-06T12:57:32.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:32.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:32.052540+0000) 2024-03-06T12:57:33.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:33.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:33.052774+0000) 2024-03-06T12:57:34.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:34.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:34.052957+0000) 2024-03-06T12:57:35.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:35.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:35.053195+0000) 2024-03-06T12:57:36.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:36.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:36.053428+0000) 2024-03-06T12:57:37.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:37.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:37.053696+0000) 2024-03-06T12:57:38.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:38.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:38.053899+0000) 2024-03-06T12:57:39.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:39.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:39.054093+0000) 2024-03-06T12:57:40.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:40.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:40.054298+0000) 2024-03-06T12:57:41.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:41.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:41.054498+0000) 2024-03-06T12:57:42.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:42.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:42.054757+0000) 2024-03-06T12:57:43.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:43.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:43.054988+0000) 2024-03-06T12:57:44.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:44.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:44.055220+0000) 2024-03-06T12:57:45.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:45.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:45.055463+0000) 2024-03-06T12:57:46.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:46.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:46.055647+0000) 2024-03-06T12:57:47.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:47.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:47.055878+0000) 2024-03-06T12:57:48.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:48.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:48.056080+0000) 2024-03-06T12:57:49.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:49.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:49.056266+0000) 2024-03-06T12:57:50.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:50.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:50.056498+0000) 2024-03-06T12:57:51.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:51.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:51.056768+0000) 2024-03-06T12:57:52.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:52.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:52.056958+0000) 2024-03-06T12:57:53.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:53.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:53.057156+0000) 2024-03-06T12:57:54.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:54.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:54.057352+0000) 2024-03-06T12:57:54.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:54.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:55.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:55.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:55.057604+0000) 2024-03-06T12:57:56.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:56.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:56.057824+0000) 2024-03-06T12:57:57.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:57.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:57.058014+0000) 2024-03-06T12:57:58.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:58.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:58.058216+0000) 2024-03-06T12:57:59.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:57:59.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:59.058411+0000) 2024-03-06T12:58:00.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:00.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:00.058614+0000) 2024-03-06T12:58:01.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:01.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:01.058840+0000) 2024-03-06T12:58:02.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:02.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:02.059132+0000) 2024-03-06T12:58:03.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:03.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:03.059356+0000) 2024-03-06T12:58:04.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:04.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:04.059566+0000) 2024-03-06T12:58:05.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:05.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:05.059738+0000) 2024-03-06T12:58:06.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:06.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:06.059961+0000) 2024-03-06T12:58:07.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:07.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:07.060161+0000) 2024-03-06T12:58:08.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:08.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:08.060364+0000) 2024-03-06T12:58:09.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:09.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:09.060575+0000) 2024-03-06T12:58:10.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:10.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:10.060838+0000) 2024-03-06T12:58:11.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:11.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:11.061030+0000) 2024-03-06T12:58:12.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:12.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:12.061228+0000) 2024-03-06T12:58:13.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:13.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:13.061413+0000) 2024-03-06T12:58:14.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:14.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:14.061599+0000) 2024-03-06T12:58:15.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:15.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:15.061889+0000) 2024-03-06T12:58:16.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:16.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:16.062136+0000) 2024-03-06T12:58:17.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:17.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:17.062317+0000) 2024-03-06T12:58:18.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:18.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:18.062555+0000) 2024-03-06T12:58:19.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:19.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:19.062782+0000) 2024-03-06T12:58:20.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:20.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:20.062985+0000) 2024-03-06T12:58:21.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:21.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:21.063184+0000) 2024-03-06T12:58:22.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:22.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:22.063410+0000) 2024-03-06T12:58:23.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:23.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:23.063632+0000) 2024-03-06T12:58:24.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:24.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:24.063866+0000) 2024-03-06T12:58:25.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:25.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:25.064066+0000) 2024-03-06T12:58:25.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:25.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:26.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:26.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:26.064263+0000) 2024-03-06T12:58:27.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:27.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:27.064453+0000) 2024-03-06T12:58:28.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:28.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:28.064738+0000) 2024-03-06T12:58:29.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:29.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:29.064934+0000) 2024-03-06T12:58:30.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:30.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:30.065132+0000) 2024-03-06T12:58:31.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:31.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:31.065359+0000) 2024-03-06T12:58:32.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:32.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:32.065605+0000) 2024-03-06T12:58:33.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:33.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:33.065834+0000) 2024-03-06T12:58:34.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:34.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:34.066036+0000) 2024-03-06T12:58:35.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:35.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:35.066238+0000) 2024-03-06T12:58:36.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:36.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:36.066472+0000) 2024-03-06T12:58:37.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:37.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:37.066726+0000) 2024-03-06T12:58:38.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:38.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:38.066925+0000) 2024-03-06T12:58:39.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:39.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:39.067155+0000) 2024-03-06T12:58:40.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:40.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:40.067419+0000) 2024-03-06T12:58:41.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:41.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:41.067703+0000) 2024-03-06T12:58:42.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:42.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:42.067903+0000) 2024-03-06T12:58:43.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:43.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:43.068105+0000) 2024-03-06T12:58:44.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:44.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:44.068343+0000) 2024-03-06T12:58:45.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:45.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:45.068551+0000) 2024-03-06T12:58:46.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:46.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:46.068826+0000) 2024-03-06T12:58:47.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:47.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:47.069018+0000) 2024-03-06T12:58:48.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:48.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:48.069216+0000) 2024-03-06T12:58:49.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:49.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:49.069453+0000) 2024-03-06T12:58:50.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:50.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:50.069643+0000) 2024-03-06T12:58:51.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:51.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:51.069929+0000) 2024-03-06T12:58:52.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:52.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:52.070202+0000) 2024-03-06T12:58:53.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:53.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:53.070482+0000) 2024-03-06T12:58:54.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:54.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:54.070736+0000) 2024-03-06T12:58:55.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:55.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:55.070975+0000) 2024-03-06T12:58:55.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:55.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:56.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:56.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:56.071173+0000) 2024-03-06T12:58:57.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:57.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:57.071407+0000) 2024-03-06T12:58:58.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:58.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:58.071654+0000) 2024-03-06T12:58:59.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:58:59.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:59.071864+0000) 2024-03-06T12:59:00.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:00.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:00.072059+0000) 2024-03-06T12:59:01.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:01.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:01.072297+0000) 2024-03-06T12:59:02.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:02.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:02.072549+0000) 2024-03-06T12:59:03.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:03.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:03.072854+0000) 2024-03-06T12:59:04.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:04.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:04.073068+0000) 2024-03-06T12:59:05.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:05.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:05.073287+0000) 2024-03-06T12:59:06.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:06.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:06.073543+0000) 2024-03-06T12:59:07.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:07.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:07.073808+0000) 2024-03-06T12:59:08.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:08.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:08.074034+0000) 2024-03-06T12:59:09.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:09.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:09.074277+0000) 2024-03-06T12:59:10.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:10.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:10.074525+0000) 2024-03-06T12:59:11.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:11.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:11.074766+0000) 2024-03-06T12:59:12.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:12.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:12.074993+0000) 2024-03-06T12:59:13.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:13.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:13.075200+0000) 2024-03-06T12:59:14.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:14.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:14.075399+0000) 2024-03-06T12:59:15.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:15.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:15.075630+0000) 2024-03-06T12:59:16.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:16.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:16.075834+0000) 2024-03-06T12:59:17.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:17.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:17.076021+0000) 2024-03-06T12:59:18.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:18.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:18.076221+0000) 2024-03-06T12:59:19.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:19.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:19.076455+0000) 2024-03-06T12:59:20.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:20.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:20.076738+0000) 2024-03-06T12:59:21.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:21.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:21.076952+0000) 2024-03-06T12:59:22.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:22.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:22.077173+0000) 2024-03-06T12:59:23.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:23.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:23.077347+0000) 2024-03-06T12:59:24.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:24.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:24.077535+0000) 2024-03-06T12:59:25.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:25.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:25.077794+0000) 2024-03-06T12:59:25.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:25.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:26.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:26.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:26.078034+0000) 2024-03-06T12:59:27.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:27.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:27.078234+0000) 2024-03-06T12:59:28.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:28.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:28.078508+0000) 2024-03-06T12:59:29.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:29.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:29.078755+0000) 2024-03-06T12:59:30.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:30.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:30.078964+0000) 2024-03-06T12:59:31.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:31.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:31.079166+0000) 2024-03-06T12:59:32.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:32.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:32.079396+0000) 2024-03-06T12:59:33.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:33.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:33.079619+0000) 2024-03-06T12:59:34.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:34.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:34.079822+0000) 2024-03-06T12:59:35.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:35.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:35.080053+0000) 2024-03-06T12:59:36.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:36.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:36.080255+0000) 2024-03-06T12:59:37.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:37.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:37.080452+0000) 2024-03-06T12:59:38.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:38.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:38.080659+0000) 2024-03-06T12:59:39.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:39.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:39.080930+0000) 2024-03-06T12:59:40.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:40.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:40.081122+0000) 2024-03-06T12:59:41.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:41.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:41.081353+0000) 2024-03-06T12:59:42.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:42.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:42.081598+0000) 2024-03-06T12:59:43.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:43.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:43.081824+0000) 2024-03-06T12:59:44.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:44.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:44.082056+0000) 2024-03-06T12:59:45.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:45.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:45.082286+0000) 2024-03-06T12:59:46.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:46.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:46.082519+0000) 2024-03-06T12:59:47.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:47.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:47.082755+0000) 2024-03-06T12:59:48.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:48.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:48.082962+0000) 2024-03-06T12:59:49.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:49.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:49.083152+0000) 2024-03-06T12:59:50.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:50.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:50.083368+0000) 2024-03-06T12:59:51.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:51.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:51.083594+0000) 2024-03-06T12:59:52.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:52.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:52.083796+0000) 2024-03-06T12:59:53.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:53.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:53.084000+0000) 2024-03-06T12:59:54.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:54.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:54.084197+0000) 2024-03-06T12:59:55.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:55.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:55.084425+0000) 2024-03-06T12:59:55.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:55.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:56.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:56.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:56.084767+0000) 2024-03-06T12:59:57.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:57.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:57.085005+0000) 2024-03-06T12:59:58.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:58.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:58.085230+0000) 2024-03-06T12:59:59.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T12:59:59.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:59.085448+0000) 2024-03-06T13:00:00.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:00.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:00.085689+0000) 2024-03-06T13:00:01.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:01.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:01.085903+0000) 2024-03-06T13:00:02.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:02.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:02.086106+0000) 2024-03-06T13:00:03.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:03.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:03.086307+0000) 2024-03-06T13:00:04.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:04.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:04.086540+0000) 2024-03-06T13:00:05.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:05.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:05.086809+0000) 2024-03-06T13:00:06.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:06.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:06.087078+0000) 2024-03-06T13:00:07.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:07.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:07.087310+0000) 2024-03-06T13:00:08.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:08.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:08.087548+0000) 2024-03-06T13:00:09.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:09.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:09.087839+0000) 2024-03-06T13:00:10.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:10.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:10.088123+0000) 2024-03-06T13:00:11.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:11.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:11.088371+0000) 2024-03-06T13:00:12.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:12.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:12.088607+0000) 2024-03-06T13:00:13.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:13.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:13.088784+0000) 2024-03-06T13:00:14.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:14.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:14.089024+0000) 2024-03-06T13:00:15.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:15.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:15.089252+0000) 2024-03-06T13:00:16.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:16.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:16.089538+0000) 2024-03-06T13:00:17.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:17.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:17.089776+0000) 2024-03-06T13:00:18.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:18.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:18.090041+0000) 2024-03-06T13:00:19.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:19.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:19.090265+0000) 2024-03-06T13:00:20.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:20.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:20.090488+0000) 2024-03-06T13:00:21.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:21.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:21.090769+0000) 2024-03-06T13:00:22.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:22.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:22.090989+0000) 2024-03-06T13:00:23.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:23.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:23.091251+0000) 2024-03-06T13:00:24.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:24.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:24.091532+0000) 2024-03-06T13:00:25.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:25.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:25.091785+0000) 2024-03-06T13:00:25.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:26.004 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:26.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:26.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:26.091975+0000) 2024-03-06T13:00:27.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:27.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:27.092150+0000) 2024-03-06T13:00:28.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:28.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:28.092339+0000) 2024-03-06T13:00:29.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:29.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:29.092576+0000) 2024-03-06T13:00:30.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:30.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:30.092784+0000) 2024-03-06T13:00:31.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:31.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:31.092996+0000) 2024-03-06T13:00:32.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:32.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:32.093200+0000) 2024-03-06T13:00:33.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:33.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:33.093397+0000) 2024-03-06T13:00:34.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:34.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:34.093605+0000) 2024-03-06T13:00:35.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:35.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:35.093778+0000) 2024-03-06T13:00:36.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:36.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:36.093982+0000) 2024-03-06T13:00:37.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:37.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:37.094180+0000) 2024-03-06T13:00:38.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:38.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:38.094388+0000) 2024-03-06T13:00:39.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:39.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:39.094595+0000) 2024-03-06T13:00:40.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:40.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:40.094784+0000) 2024-03-06T13:00:41.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:41.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:41.094972+0000) 2024-03-06T13:00:42.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:42.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:42.095178+0000) 2024-03-06T13:00:43.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:43.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:43.095387+0000) 2024-03-06T13:00:44.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:44.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:44.095649+0000) 2024-03-06T13:00:45.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:45.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:45.095920+0000) 2024-03-06T13:00:46.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:46.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:46.096155+0000) 2024-03-06T13:00:47.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:47.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:47.096403+0000) 2024-03-06T13:00:48.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:48.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:48.096594+0000) 2024-03-06T13:00:49.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:49.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:49.096784+0000) 2024-03-06T13:00:50.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:50.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:50.096971+0000) 2024-03-06T13:00:51.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:51.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:51.097150+0000) 2024-03-06T13:00:52.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:52.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:52.097395+0000) 2024-03-06T13:00:53.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:53.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:53.097591+0000) 2024-03-06T13:00:54.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:54.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:54.097830+0000) 2024-03-06T13:00:55.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:55.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:55.098052+0000) 2024-03-06T13:00:56.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:56.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:56.098264+0000) 2024-03-06T13:00:56.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:56.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:57.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:57.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:57.098470+0000) 2024-03-06T13:00:58.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:58.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:58.098738+0000) 2024-03-06T13:00:59.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:00:59.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:59.098935+0000) 2024-03-06T13:01:00.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:00.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:00.099132+0000) 2024-03-06T13:01:01.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:01.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:01.099310+0000) 2024-03-06T13:01:02.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:02.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:02.099540+0000) 2024-03-06T13:01:03.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:03.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:03.099792+0000) 2024-03-06T13:01:04.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:04.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:04.100025+0000) 2024-03-06T13:01:05.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:05.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:05.100258+0000) 2024-03-06T13:01:06.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:06.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:06.100511+0000) 2024-03-06T13:01:07.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:07.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:07.100761+0000) 2024-03-06T13:01:08.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:08.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:08.101005+0000) 2024-03-06T13:01:09.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:09.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:09.101195+0000) 2024-03-06T13:01:10.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:10.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:10.101419+0000) 2024-03-06T13:01:11.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:11.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:11.101617+0000) 2024-03-06T13:01:12.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:12.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:12.101869+0000) 2024-03-06T13:01:13.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:13.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:13.102090+0000) 2024-03-06T13:01:14.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:14.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:14.102325+0000) 2024-03-06T13:01:15.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:15.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:15.102568+0000) 2024-03-06T13:01:16.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:16.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:16.102782+0000) 2024-03-06T13:01:17.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:17.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:17.103018+0000) 2024-03-06T13:01:18.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:18.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:18.103309+0000) 2024-03-06T13:01:19.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:19.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:19.103520+0000) 2024-03-06T13:01:20.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:20.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:20.103784+0000) 2024-03-06T13:01:21.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:21.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:21.104005+0000) 2024-03-06T13:01:22.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:22.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:22.104257+0000) 2024-03-06T13:01:23.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:23.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:23.104499+0000) 2024-03-06T13:01:24.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:24.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:24.104763+0000) 2024-03-06T13:01:25.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:25.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:25.105011+0000) 2024-03-06T13:01:26.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:26.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:26.105253+0000) 2024-03-06T13:01:26.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:26.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:27.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:27.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:27.105469+0000) 2024-03-06T13:01:28.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:28.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:28.105776+0000) 2024-03-06T13:01:29.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:29.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:29.105982+0000) 2024-03-06T13:01:30.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:30.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:30.106168+0000) 2024-03-06T13:01:31.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:31.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:31.106365+0000) 2024-03-06T13:01:32.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:32.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:32.106562+0000) 2024-03-06T13:01:33.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:33.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:33.106783+0000) 2024-03-06T13:01:34.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:34.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:34.107013+0000) 2024-03-06T13:01:35.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:35.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:35.107193+0000) 2024-03-06T13:01:36.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:36.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:36.107373+0000) 2024-03-06T13:01:37.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:37.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:37.107595+0000) 2024-03-06T13:01:38.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:38.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:38.107859+0000) 2024-03-06T13:01:39.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:39.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:39.108095+0000) 2024-03-06T13:01:40.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:40.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:40.108306+0000) 2024-03-06T13:01:41.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:41.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:41.108556+0000) 2024-03-06T13:01:42.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:42.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:42.108842+0000) 2024-03-06T13:01:43.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:43.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:43.109068+0000) 2024-03-06T13:01:44.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:44.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:44.109352+0000) 2024-03-06T13:01:45.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:45.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:45.109576+0000) 2024-03-06T13:01:46.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:46.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:46.109857+0000) 2024-03-06T13:01:47.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:47.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:47.110064+0000) 2024-03-06T13:01:48.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:48.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:48.110301+0000) 2024-03-06T13:01:49.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:49.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:49.110500+0000) 2024-03-06T13:01:50.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:50.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:50.110715+0000) 2024-03-06T13:01:51.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:51.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:51.110921+0000) 2024-03-06T13:01:52.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:52.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:52.111128+0000) 2024-03-06T13:01:53.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:53.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:53.111346+0000) 2024-03-06T13:01:54.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:54.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:54.111594+0000) 2024-03-06T13:01:55.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:55.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:55.111780+0000) 2024-03-06T13:01:56.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:56.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:56.111980+0000) 2024-03-06T13:01:56.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:56.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:57.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:57.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:57.112169+0000) 2024-03-06T13:01:58.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:58.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:58.112407+0000) 2024-03-06T13:01:59.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:01:59.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:59.112594+0000) 2024-03-06T13:02:00.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:00.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:00.112835+0000) 2024-03-06T13:02:01.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:01.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:01.113021+0000) 2024-03-06T13:02:02.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:02.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:02.113218+0000) 2024-03-06T13:02:03.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:03.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:03.113442+0000) 2024-03-06T13:02:04.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:04.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:04.113647+0000) 2024-03-06T13:02:05.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:05.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:05.113921+0000) 2024-03-06T13:02:06.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:06.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:06.114155+0000) 2024-03-06T13:02:07.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:07.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:07.114378+0000) 2024-03-06T13:02:08.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:08.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:08.114594+0000) 2024-03-06T13:02:09.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:09.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:09.114797+0000) 2024-03-06T13:02:10.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:10.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:10.115038+0000) 2024-03-06T13:02:11.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:11.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:11.115223+0000) 2024-03-06T13:02:12.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:12.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:12.115452+0000) 2024-03-06T13:02:13.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:13.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:13.115697+0000) 2024-03-06T13:02:14.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:14.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:14.115892+0000) 2024-03-06T13:02:15.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:15.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:15.116082+0000) 2024-03-06T13:02:16.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:16.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:16.116291+0000) 2024-03-06T13:02:17.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:17.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:17.116485+0000) 2024-03-06T13:02:18.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:18.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:18.116760+0000) 2024-03-06T13:02:19.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:19.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:19.117041+0000) 2024-03-06T13:02:20.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:20.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:20.117286+0000) 2024-03-06T13:02:21.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:21.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:21.117523+0000) 2024-03-06T13:02:22.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:22.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:22.117818+0000) 2024-03-06T13:02:23.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:23.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:23.118041+0000) 2024-03-06T13:02:24.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:24.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:24.118278+0000) 2024-03-06T13:02:25.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:25.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:25.118511+0000) 2024-03-06T13:02:26.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:26.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:26.118775+0000) 2024-03-06T13:02:26.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:26.697 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:27.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:27.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:27.118973+0000) 2024-03-06T13:02:28.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:28.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:28.119226+0000) 2024-03-06T13:02:29.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:29.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:29.119426+0000) 2024-03-06T13:02:30.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:30.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:30.119631+0000) 2024-03-06T13:02:31.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:31.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:31.119917+0000) 2024-03-06T13:02:32.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:32.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:32.120165+0000) 2024-03-06T13:02:33.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:33.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:33.120361+0000) 2024-03-06T13:02:34.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:34.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:34.120560+0000) 2024-03-06T13:02:35.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:35.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:35.120799+0000) 2024-03-06T13:02:36.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:36.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:36.121055+0000) 2024-03-06T13:02:37.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:37.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:37.121240+0000) 2024-03-06T13:02:38.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:38.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:38.121470+0000) 2024-03-06T13:02:39.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:39.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:39.121715+0000) 2024-03-06T13:02:40.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:40.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:40.121927+0000) 2024-03-06T13:02:41.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:41.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:41.122167+0000) 2024-03-06T13:02:42.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:42.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:42.122422+0000) 2024-03-06T13:02:43.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:43.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:43.122701+0000) 2024-03-06T13:02:44.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:44.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:44.122954+0000) 2024-03-06T13:02:45.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:45.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:45.123154+0000) 2024-03-06T13:02:46.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:46.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:46.123400+0000) 2024-03-06T13:02:47.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:47.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:47.123597+0000) 2024-03-06T13:02:48.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:48.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:48.123858+0000) 2024-03-06T13:02:49.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:49.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:49.124100+0000) 2024-03-06T13:02:50.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:50.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:50.124376+0000) 2024-03-06T13:02:51.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:51.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:51.124619+0000) 2024-03-06T13:02:52.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:52.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:52.124861+0000) 2024-03-06T13:02:53.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:53.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:53.125046+0000) 2024-03-06T13:02:54.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:54.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:54.125281+0000) 2024-03-06T13:02:55.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:55.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:55.125501+0000) 2024-03-06T13:02:56.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:56.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:56.125783+0000) 2024-03-06T13:02:56.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:56.900 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:57.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:57.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:57.126070+0000) 2024-03-06T13:02:58.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:58.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:58.126317+0000) 2024-03-06T13:02:59.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:02:59.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:59.126558+0000) 2024-03-06T13:03:00.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:00.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:00.126768+0000) 2024-03-06T13:03:01.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:01.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:01.127004+0000) 2024-03-06T13:03:02.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:02.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:02.127246+0000) 2024-03-06T13:03:03.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:03.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:03.127433+0000) 2024-03-06T13:03:04.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:04.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:04.127698+0000) 2024-03-06T13:03:05.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:05.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:05.127929+0000) 2024-03-06T13:03:06.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:06.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:06.128168+0000) 2024-03-06T13:03:07.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:07.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:07.128357+0000) 2024-03-06T13:03:08.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:08.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:08.128600+0000) 2024-03-06T13:03:09.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:09.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:09.128831+0000) 2024-03-06T13:03:10.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:10.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:10.129057+0000) 2024-03-06T13:03:11.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:11.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:11.129290+0000) 2024-03-06T13:03:12.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:12.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:12.129524+0000) 2024-03-06T13:03:13.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:13.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:13.129845+0000) 2024-03-06T13:03:14.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:14.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:14.130037+0000) 2024-03-06T13:03:15.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:15.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:15.130261+0000) 2024-03-06T13:03:16.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:16.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:16.130447+0000) 2024-03-06T13:03:17.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:17.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:17.130692+0000) 2024-03-06T13:03:18.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:18.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:18.130892+0000) 2024-03-06T13:03:19.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:19.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:19.131126+0000) 2024-03-06T13:03:20.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:20.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:20.131356+0000) 2024-03-06T13:03:21.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:21.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:21.131538+0000) 2024-03-06T13:03:22.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:22.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:22.131816+0000) 2024-03-06T13:03:23.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:23.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:23.132047+0000) 2024-03-06T13:03:24.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:24.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:24.132300+0000) 2024-03-06T13:03:25.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:25.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:25.132562+0000) 2024-03-06T13:03:26.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:26.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:26.132819+0000) 2024-03-06T13:03:27.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:27.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:27.133074+0000) 2024-03-06T13:03:27.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:27.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:28.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:28.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:28.133375+0000) 2024-03-06T13:03:29.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:29.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:29.133614+0000) 2024-03-06T13:03:30.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:30.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:30.133780+0000) 2024-03-06T13:03:31.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:31.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:31.133985+0000) 2024-03-06T13:03:32.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:32.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:32.134200+0000) 2024-03-06T13:03:33.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:33.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:33.134397+0000) 2024-03-06T13:03:34.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:34.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:34.134632+0000) 2024-03-06T13:03:35.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:35.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:35.134864+0000) 2024-03-06T13:03:36.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:36.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:36.135071+0000) 2024-03-06T13:03:37.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:37.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:37.135246+0000) 2024-03-06T13:03:38.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:38.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:38.135485+0000) 2024-03-06T13:03:39.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:39.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:39.135737+0000) 2024-03-06T13:03:40.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:40.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:40.135974+0000) 2024-03-06T13:03:41.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:41.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:41.136199+0000) 2024-03-06T13:03:42.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:42.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:42.136382+0000) 2024-03-06T13:03:43.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:43.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:43.136601+0000) 2024-03-06T13:03:44.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:44.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:44.136826+0000) 2024-03-06T13:03:45.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:45.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:45.137055+0000) 2024-03-06T13:03:46.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:46.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:46.137286+0000) 2024-03-06T13:03:47.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:47.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:47.137507+0000) 2024-03-06T13:03:48.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:48.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:48.137769+0000) 2024-03-06T13:03:49.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:49.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:49.137992+0000) 2024-03-06T13:03:50.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:50.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:50.138224+0000) 2024-03-06T13:03:51.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:51.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:51.138452+0000) 2024-03-06T13:03:52.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:52.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:52.138771+0000) 2024-03-06T13:03:53.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:53.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:53.138972+0000) 2024-03-06T13:03:54.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:54.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:54.139191+0000) 2024-03-06T13:03:55.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:55.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:55.139411+0000) 2024-03-06T13:03:56.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:56.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:56.139637+0000) 2024-03-06T13:03:57.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:57.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:57.139816+0000) 2024-03-06T13:03:57.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:57.285 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:58.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:58.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:58.140025+0000) 2024-03-06T13:03:59.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:03:59.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:59.140205+0000) 2024-03-06T13:04:00.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:00.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:00.140396+0000) 2024-03-06T13:04:01.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:01.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:01.140660+0000) 2024-03-06T13:04:02.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:02.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:02.140937+0000) 2024-03-06T13:04:03.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:03.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:03.141156+0000) 2024-03-06T13:04:04.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:04.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:04.141439+0000) 2024-03-06T13:04:05.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:05.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:05.141639+0000) 2024-03-06T13:04:06.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:06.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:06.141908+0000) 2024-03-06T13:04:07.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:07.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:07.142151+0000) 2024-03-06T13:04:08.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:08.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:08.142351+0000) 2024-03-06T13:04:09.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:09.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:09.142586+0000) 2024-03-06T13:04:10.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:10.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:10.142811+0000) 2024-03-06T13:04:11.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:11.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:11.143030+0000) 2024-03-06T13:04:12.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:12.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:12.143255+0000) 2024-03-06T13:04:13.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:13.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:13.143469+0000) 2024-03-06T13:04:14.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:14.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:14.143732+0000) 2024-03-06T13:04:15.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:15.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:15.143956+0000) 2024-03-06T13:04:16.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:16.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:16.144183+0000) 2024-03-06T13:04:17.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:17.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:17.144412+0000) 2024-03-06T13:04:18.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:18.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:18.144605+0000) 2024-03-06T13:04:19.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:19.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:19.144794+0000) 2024-03-06T13:04:20.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:20.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:20.144994+0000) 2024-03-06T13:04:21.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:21.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:21.145178+0000) 2024-03-06T13:04:22.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:22.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:22.145404+0000) 2024-03-06T13:04:23.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:23.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:23.145629+0000) 2024-03-06T13:04:24.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:24.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:24.145863+0000) 2024-03-06T13:04:25.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:25.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:25.146054+0000) 2024-03-06T13:04:26.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:26.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:26.146253+0000) 2024-03-06T13:04:27.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:27.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:27.146476+0000) 2024-03-06T13:04:27.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:27.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:28.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:28.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:28.146761+0000) 2024-03-06T13:04:29.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:29.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:29.146949+0000) 2024-03-06T13:04:30.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:30.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:30.147152+0000) 2024-03-06T13:04:31.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:31.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:31.147400+0000) 2024-03-06T13:04:32.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:32.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:32.147642+0000) 2024-03-06T13:04:33.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:33.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:33.147891+0000) 2024-03-06T13:04:34.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:34.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:34.148085+0000) 2024-03-06T13:04:35.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:35.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:35.148266+0000) 2024-03-06T13:04:36.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:36.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:36.148512+0000) 2024-03-06T13:04:37.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:37.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:37.148763+0000) 2024-03-06T13:04:38.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:38.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:38.148963+0000) 2024-03-06T13:04:39.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:39.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:39.149151+0000) 2024-03-06T13:04:40.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:40.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:40.149340+0000) 2024-03-06T13:04:41.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:41.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:41.149600+0000) 2024-03-06T13:04:42.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:42.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:42.149806+0000) 2024-03-06T13:04:43.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:43.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:43.149990+0000) 2024-03-06T13:04:44.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:44.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:44.150216+0000) 2024-03-06T13:04:45.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:45.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:45.150397+0000) 2024-03-06T13:04:46.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:46.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:46.150625+0000) 2024-03-06T13:04:47.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:47.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:47.150849+0000) 2024-03-06T13:04:48.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:48.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:48.151071+0000) 2024-03-06T13:04:49.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:49.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:49.151291+0000) 2024-03-06T13:04:50.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:50.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:50.151526+0000) 2024-03-06T13:04:51.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:51.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:51.151781+0000) 2024-03-06T13:04:52.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:52.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:52.152016+0000) 2024-03-06T13:04:53.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:53.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:53.152244+0000) 2024-03-06T13:04:54.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:54.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:54.152473+0000) 2024-03-06T13:04:55.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:55.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:55.152777+0000) 2024-03-06T13:04:56.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:56.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:56.153008+0000) 2024-03-06T13:04:57.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:57.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:57.153181+0000) 2024-03-06T13:04:57.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:57.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:58.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:58.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:58.153380+0000) 2024-03-06T13:04:59.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:04:59.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:59.153603+0000) 2024-03-06T13:05:00.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:00.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:00.153822+0000) 2024-03-06T13:05:01.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:01.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:01.154009+0000) 2024-03-06T13:05:02.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:02.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:02.154206+0000) 2024-03-06T13:05:03.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:03.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:03.154427+0000) 2024-03-06T13:05:04.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:04.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:04.154632+0000) 2024-03-06T13:05:05.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:05.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:05.154855+0000) 2024-03-06T13:05:06.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:06.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:06.155010+0000) 2024-03-06T13:05:07.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:07.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:07.155197+0000) 2024-03-06T13:05:08.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:08.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:08.155439+0000) 2024-03-06T13:05:09.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:09.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:09.155679+0000) 2024-03-06T13:05:10.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:10.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:10.155874+0000) 2024-03-06T13:05:11.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:11.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:11.156062+0000) 2024-03-06T13:05:12.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:12.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:12.156251+0000) 2024-03-06T13:05:13.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:13.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:13.156473+0000) 2024-03-06T13:05:14.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:14.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:14.156730+0000) 2024-03-06T13:05:15.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:15.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:15.156915+0000) 2024-03-06T13:05:16.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:16.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:16.157114+0000) 2024-03-06T13:05:17.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:17.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:17.157343+0000) 2024-03-06T13:05:18.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:18.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:18.157575+0000) 2024-03-06T13:05:19.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:19.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:19.157793+0000) 2024-03-06T13:05:20.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:20.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:20.157997+0000) 2024-03-06T13:05:21.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:21.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:21.158237+0000) 2024-03-06T13:05:22.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:22.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:22.158425+0000) 2024-03-06T13:05:23.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:23.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:23.158613+0000) 2024-03-06T13:05:24.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:24.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:24.158855+0000) 2024-03-06T13:05:25.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:25.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:25.159057+0000) 2024-03-06T13:05:26.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:26.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:26.159280+0000) 2024-03-06T13:05:27.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:27.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:27.159549+0000) 2024-03-06T13:05:27.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:27.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:28.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:28.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:28.159797+0000) 2024-03-06T13:05:29.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:29.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:29.160004+0000) 2024-03-06T13:05:30.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:30.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:30.160211+0000) 2024-03-06T13:05:31.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:31.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:31.160389+0000) 2024-03-06T13:05:32.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:32.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:32.160584+0000) 2024-03-06T13:05:33.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:33.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:33.160819+0000) 2024-03-06T13:05:34.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:34.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:34.161032+0000) 2024-03-06T13:05:35.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:35.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:35.161271+0000) 2024-03-06T13:05:36.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:36.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:36.161502+0000) 2024-03-06T13:05:37.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:37.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:37.161791+0000) 2024-03-06T13:05:38.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:38.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:38.161994+0000) 2024-03-06T13:05:39.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:39.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:39.162183+0000) 2024-03-06T13:05:40.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:40.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:40.162367+0000) 2024-03-06T13:05:41.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:41.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:41.162592+0000) 2024-03-06T13:05:42.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:42.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:42.162842+0000) 2024-03-06T13:05:43.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:43.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:43.163149+0000) 2024-03-06T13:05:44.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:44.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:44.163434+0000) 2024-03-06T13:05:45.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:45.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:45.163739+0000) 2024-03-06T13:05:46.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:46.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:46.163905+0000) 2024-03-06T13:05:47.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:47.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:47.164090+0000) 2024-03-06T13:05:48.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:48.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:48.164325+0000) 2024-03-06T13:05:49.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:49.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:49.164557+0000) 2024-03-06T13:05:50.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:50.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:50.164785+0000) 2024-03-06T13:05:51.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:51.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:51.164979+0000) 2024-03-06T13:05:52.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:52.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:52.165175+0000) 2024-03-06T13:05:53.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:53.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:53.165399+0000) 2024-03-06T13:05:54.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:54.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:54.165647+0000) 2024-03-06T13:05:55.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:55.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:55.165912+0000) 2024-03-06T13:05:56.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:56.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:56.166097+0000) 2024-03-06T13:05:57.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:57.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:57.166266+0000) 2024-03-06T13:05:58.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:58.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:58.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:58.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:58.166474+0000) 2024-03-06T13:05:59.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:05:59.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:59.166728+0000) 2024-03-06T13:06:00.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:00.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:00.166960+0000) 2024-03-06T13:06:01.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:01.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:01.167135+0000) 2024-03-06T13:06:02.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:02.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:02.167410+0000) 2024-03-06T13:06:03.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:03.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:03.167644+0000) 2024-03-06T13:06:04.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:04.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:04.167876+0000) 2024-03-06T13:06:05.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:05.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:05.168101+0000) 2024-03-06T13:06:06.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:06.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:06.168330+0000) 2024-03-06T13:06:07.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:07.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:07.168565+0000) 2024-03-06T13:06:08.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:08.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:08.168799+0000) 2024-03-06T13:06:09.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:09.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:09.169024+0000) 2024-03-06T13:06:10.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:10.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:10.169295+0000) 2024-03-06T13:06:11.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:11.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:11.169532+0000) 2024-03-06T13:06:12.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:12.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:12.169789+0000) 2024-03-06T13:06:13.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:13.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:13.170032+0000) 2024-03-06T13:06:14.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:14.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:14.170286+0000) 2024-03-06T13:06:15.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:15.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:15.170526+0000) 2024-03-06T13:06:16.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:16.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:16.170803+0000) 2024-03-06T13:06:17.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:17.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:17.171041+0000) 2024-03-06T13:06:18.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:18.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:18.171285+0000) 2024-03-06T13:06:19.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:19.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:19.171521+0000) 2024-03-06T13:06:20.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:20.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:20.171790+0000) 2024-03-06T13:06:21.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:21.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:21.171928+0000) 2024-03-06T13:06:22.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:22.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:22.172115+0000) 2024-03-06T13:06:23.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:23.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:23.172299+0000) 2024-03-06T13:06:24.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:24.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:24.172530+0000) 2024-03-06T13:06:25.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:25.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:25.172772+0000) 2024-03-06T13:06:26.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:26.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:26.172969+0000) 2024-03-06T13:06:27.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:27.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:27.173153+0000) 2024-03-06T13:06:28.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:28.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:28.173387+0000) 2024-03-06T13:06:28.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:28.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:29.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:29.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:29.173610+0000) 2024-03-06T13:06:30.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:30.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:30.173811+0000) 2024-03-06T13:06:31.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:31.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:31.173987+0000) 2024-03-06T13:06:32.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:32.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:32.174182+0000) 2024-03-06T13:06:33.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:33.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:33.174359+0000) 2024-03-06T13:06:34.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:34.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:34.174593+0000) 2024-03-06T13:06:35.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:35.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:35.174811+0000) 2024-03-06T13:06:36.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:36.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:36.175064+0000) 2024-03-06T13:06:37.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:37.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:37.175253+0000) 2024-03-06T13:06:38.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:38.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:38.175454+0000) 2024-03-06T13:06:39.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:39.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:39.175705+0000) 2024-03-06T13:06:40.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:40.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:40.175899+0000) 2024-03-06T13:06:41.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:41.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:41.176097+0000) 2024-03-06T13:06:42.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:42.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:42.176312+0000) 2024-03-06T13:06:43.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:43.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:43.176548+0000) 2024-03-06T13:06:44.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:44.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:44.176784+0000) 2024-03-06T13:06:45.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:45.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:45.176929+0000) 2024-03-06T13:06:46.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:46.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:46.177123+0000) 2024-03-06T13:06:47.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:47.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:47.177310+0000) 2024-03-06T13:06:48.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:48.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:48.177553+0000) 2024-03-06T13:06:49.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:49.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:49.177789+0000) 2024-03-06T13:06:50.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:50.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:50.177980+0000) 2024-03-06T13:06:51.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:51.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:51.178172+0000) 2024-03-06T13:06:52.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:52.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:52.178410+0000) 2024-03-06T13:06:53.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:53.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:53.178583+0000) 2024-03-06T13:06:54.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:54.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:54.178823+0000) 2024-03-06T13:06:55.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:55.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:55.178994+0000) 2024-03-06T13:06:56.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:56.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:56.179178+0000) 2024-03-06T13:06:57.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:57.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:57.179405+0000) 2024-03-06T13:06:58.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:58.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:58.179600+0000) 2024-03-06T13:06:58.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:58.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:59.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:06:59.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:59.179824+0000) 2024-03-06T13:07:00.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:00.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:00.180007+0000) 2024-03-06T13:07:01.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:01.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:01.180192+0000) 2024-03-06T13:07:02.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:02.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:02.180386+0000) 2024-03-06T13:07:03.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:03.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:03.180612+0000) 2024-03-06T13:07:04.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:04.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:04.180836+0000) 2024-03-06T13:07:05.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:05.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:05.181023+0000) 2024-03-06T13:07:06.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:06.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:06.181222+0000) 2024-03-06T13:07:07.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:07.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:07.181454+0000) 2024-03-06T13:07:08.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:08.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:08.181732+0000) 2024-03-06T13:07:09.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:09.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:09.181918+0000) 2024-03-06T13:07:10.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:10.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:10.182105+0000) 2024-03-06T13:07:11.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:11.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:11.182292+0000) 2024-03-06T13:07:12.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:12.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:12.182521+0000) 2024-03-06T13:07:13.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:13.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:13.182741+0000) 2024-03-06T13:07:14.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:14.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:14.182992+0000) 2024-03-06T13:07:15.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:15.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:15.183214+0000) 2024-03-06T13:07:16.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:16.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:16.183442+0000) 2024-03-06T13:07:17.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:17.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:17.183630+0000) 2024-03-06T13:07:18.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:18.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:18.183848+0000) 2024-03-06T13:07:19.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:19.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:19.184055+0000) 2024-03-06T13:07:20.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:20.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:20.184295+0000) 2024-03-06T13:07:21.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:21.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:21.184528+0000) 2024-03-06T13:07:22.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:22.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:22.184816+0000) 2024-03-06T13:07:23.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:23.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:23.185066+0000) 2024-03-06T13:07:24.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:24.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:24.185315+0000) 2024-03-06T13:07:25.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:25.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:25.185541+0000) 2024-03-06T13:07:26.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:26.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:26.185773+0000) 2024-03-06T13:07:27.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:27.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:27.186002+0000) 2024-03-06T13:07:28.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:28.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:28.186208+0000) 2024-03-06T13:07:28.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:28.677 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:29.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:29.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:29.186431+0000) 2024-03-06T13:07:30.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:30.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:30.186637+0000) 2024-03-06T13:07:31.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:31.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:31.186902+0000) 2024-03-06T13:07:32.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:32.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:32.187140+0000) 2024-03-06T13:07:33.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:33.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:33.187358+0000) 2024-03-06T13:07:34.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:34.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:34.187583+0000) 2024-03-06T13:07:35.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:35.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:35.187827+0000) 2024-03-06T13:07:36.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:36.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:36.188060+0000) 2024-03-06T13:07:37.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:37.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:37.188249+0000) 2024-03-06T13:07:38.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:38.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:38.188463+0000) 2024-03-06T13:07:39.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:39.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:39.188649+0000) 2024-03-06T13:07:40.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:40.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:40.188927+0000) 2024-03-06T13:07:41.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:41.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:41.189133+0000) 2024-03-06T13:07:42.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:42.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:42.189389+0000) 2024-03-06T13:07:43.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:43.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:43.189626+0000) 2024-03-06T13:07:44.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:44.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:44.189905+0000) 2024-03-06T13:07:45.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:45.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:45.190110+0000) 2024-03-06T13:07:46.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:46.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:46.190340+0000) 2024-03-06T13:07:47.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:47.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:47.190561+0000) 2024-03-06T13:07:48.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:48.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:48.190796+0000) 2024-03-06T13:07:49.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:49.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:49.191017+0000) 2024-03-06T13:07:50.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:50.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:50.191274+0000) 2024-03-06T13:07:51.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:51.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:51.191468+0000) 2024-03-06T13:07:52.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:52.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:52.191739+0000) 2024-03-06T13:07:53.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:53.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:53.191938+0000) 2024-03-06T13:07:54.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:54.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:54.192130+0000) 2024-03-06T13:07:55.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:55.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:55.192354+0000) 2024-03-06T13:07:56.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:56.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:56.192583+0000) 2024-03-06T13:07:57.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:57.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:57.192817+0000) 2024-03-06T13:07:58.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:58.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:58.193012+0000) 2024-03-06T13:07:58.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:58.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:59.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:07:59.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:59.193191+0000) 2024-03-06T13:08:00.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:00.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:00.193424+0000) 2024-03-06T13:08:01.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:01.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:01.193653+0000) 2024-03-06T13:08:02.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:02.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:02.193868+0000) 2024-03-06T13:08:03.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:03.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:03.194055+0000) 2024-03-06T13:08:04.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:04.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:04.194297+0000) 2024-03-06T13:08:05.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:05.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:05.194515+0000) 2024-03-06T13:08:06.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:06.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:06.194760+0000) 2024-03-06T13:08:07.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:07.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:07.195045+0000) 2024-03-06T13:08:08.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:08.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:08.195353+0000) 2024-03-06T13:08:09.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:09.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:09.195599+0000) 2024-03-06T13:08:10.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:10.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:10.195871+0000) 2024-03-06T13:08:11.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:11.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:11.196078+0000) 2024-03-06T13:08:12.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:12.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:12.196306+0000) 2024-03-06T13:08:13.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:13.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:13.196522+0000) 2024-03-06T13:08:14.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:14.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:14.196784+0000) 2024-03-06T13:08:15.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:15.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:15.197003+0000) 2024-03-06T13:08:16.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:16.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:16.197207+0000) 2024-03-06T13:08:17.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:17.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:17.197449+0000) 2024-03-06T13:08:18.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:18.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:18.197723+0000) 2024-03-06T13:08:19.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:19.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:19.197989+0000) 2024-03-06T13:08:20.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:20.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:20.198270+0000) 2024-03-06T13:08:21.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:21.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:21.198548+0000) 2024-03-06T13:08:22.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:22.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:22.198814+0000) 2024-03-06T13:08:23.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:23.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:23.199028+0000) 2024-03-06T13:08:24.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:24.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:24.199207+0000) 2024-03-06T13:08:25.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:25.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:25.199427+0000) 2024-03-06T13:08:26.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:26.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:26.199622+0000) 2024-03-06T13:08:27.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:27.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:27.199840+0000) 2024-03-06T13:08:28.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:28.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:28.200066+0000) 2024-03-06T13:08:29.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:29.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:29.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:29.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:29.200285+0000) 2024-03-06T13:08:30.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:30.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:30.200513+0000) 2024-03-06T13:08:31.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:31.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:31.200761+0000) 2024-03-06T13:08:32.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:32.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:32.200956+0000) 2024-03-06T13:08:33.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:33.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:33.201183+0000) 2024-03-06T13:08:34.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:34.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:34.201425+0000) 2024-03-06T13:08:35.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:35.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:35.201716+0000) 2024-03-06T13:08:36.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:36.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:36.201988+0000) 2024-03-06T13:08:37.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:37.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:37.202233+0000) 2024-03-06T13:08:38.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:38.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:38.202470+0000) 2024-03-06T13:08:39.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:39.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:39.202747+0000) 2024-03-06T13:08:40.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:40.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:40.203042+0000) 2024-03-06T13:08:41.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:41.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:41.203267+0000) 2024-03-06T13:08:42.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:42.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:42.203517+0000) 2024-03-06T13:08:43.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:43.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:43.203812+0000) 2024-03-06T13:08:44.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:44.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:44.204087+0000) 2024-03-06T13:08:45.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:45.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:45.204352+0000) 2024-03-06T13:08:46.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:46.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:46.204585+0000) 2024-03-06T13:08:47.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:47.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:47.204857+0000) 2024-03-06T13:08:48.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:48.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:48.205086+0000) 2024-03-06T13:08:49.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:49.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:49.205309+0000) 2024-03-06T13:08:50.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:50.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:50.205550+0000) 2024-03-06T13:08:51.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:51.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:51.205798+0000) 2024-03-06T13:08:52.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:52.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:52.206012+0000) 2024-03-06T13:08:53.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:53.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:53.206214+0000) 2024-03-06T13:08:54.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:54.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:54.206463+0000) 2024-03-06T13:08:55.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:55.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:55.206731+0000) 2024-03-06T13:08:56.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:56.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:56.206939+0000) 2024-03-06T13:08:57.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:57.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:57.207141+0000) 2024-03-06T13:08:58.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:58.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:58.207389+0000) 2024-03-06T13:08:59.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:08:59.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:59.207568+0000) 2024-03-06T13:08:59.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:59.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:00.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:00.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:00.207812+0000) 2024-03-06T13:09:01.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:01.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:01.208037+0000) 2024-03-06T13:09:02.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:02.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:02.208236+0000) 2024-03-06T13:09:03.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:03.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:03.208460+0000) 2024-03-06T13:09:04.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:04.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:04.208725+0000) 2024-03-06T13:09:05.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:05.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:05.209006+0000) 2024-03-06T13:09:06.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:06.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:06.209242+0000) 2024-03-06T13:09:07.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:07.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:07.209476+0000) 2024-03-06T13:09:08.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:08.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:08.209766+0000) 2024-03-06T13:09:09.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:09.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:09.209990+0000) 2024-03-06T13:09:10.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:10.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:10.210196+0000) 2024-03-06T13:09:11.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:11.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:11.210447+0000) 2024-03-06T13:09:12.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:12.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:12.210750+0000) 2024-03-06T13:09:13.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:13.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:13.210960+0000) 2024-03-06T13:09:14.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:14.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:14.211216+0000) 2024-03-06T13:09:15.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:15.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:15.211374+0000) 2024-03-06T13:09:16.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:16.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:16.211601+0000) 2024-03-06T13:09:17.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:17.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:17.211828+0000) 2024-03-06T13:09:18.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:18.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:18.212089+0000) 2024-03-06T13:09:19.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:19.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:19.212365+0000) 2024-03-06T13:09:20.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:20.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:20.212622+0000) 2024-03-06T13:09:21.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:21.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:21.212920+0000) 2024-03-06T13:09:22.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:22.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:22.213180+0000) 2024-03-06T13:09:23.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:23.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:23.213388+0000) 2024-03-06T13:09:24.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:24.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:24.213656+0000) 2024-03-06T13:09:25.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:25.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:25.214003+0000) 2024-03-06T13:09:26.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:26.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:26.214292+0000) 2024-03-06T13:09:27.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:27.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:27.214521+0000) 2024-03-06T13:09:28.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:28.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:28.214821+0000) 2024-03-06T13:09:29.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:29.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:29.215059+0000) 2024-03-06T13:09:29.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:29.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:30.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:30.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:30.215339+0000) 2024-03-06T13:09:31.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:31.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:31.215509+0000) 2024-03-06T13:09:32.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:32.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:32.215744+0000) 2024-03-06T13:09:33.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:33.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:33.215900+0000) 2024-03-06T13:09:34.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:34.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:34.216092+0000) 2024-03-06T13:09:35.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:35.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:35.216265+0000) 2024-03-06T13:09:36.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:36.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:36.216461+0000) 2024-03-06T13:09:37.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:37.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:37.216705+0000) 2024-03-06T13:09:38.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:38.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:38.216881+0000) 2024-03-06T13:09:39.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:39.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:39.217075+0000) 2024-03-06T13:09:40.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:40.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:40.217274+0000) 2024-03-06T13:09:41.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:41.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:41.217496+0000) 2024-03-06T13:09:42.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:42.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:42.217836+0000) 2024-03-06T13:09:43.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:43.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:43.218076+0000) 2024-03-06T13:09:44.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:44.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:44.218366+0000) 2024-03-06T13:09:45.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:45.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:45.218541+0000) 2024-03-06T13:09:46.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:46.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:46.218801+0000) 2024-03-06T13:09:47.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:47.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:47.219018+0000) 2024-03-06T13:09:48.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:48.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:48.219292+0000) 2024-03-06T13:09:49.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:49.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:49.219543+0000) 2024-03-06T13:09:50.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:50.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:50.219812+0000) 2024-03-06T13:09:51.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:51.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:51.220010+0000) 2024-03-06T13:09:52.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:52.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:52.220257+0000) 2024-03-06T13:09:53.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:53.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:53.220465+0000) 2024-03-06T13:09:54.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:54.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:54.220722+0000) 2024-03-06T13:09:55.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:55.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:55.220901+0000) 2024-03-06T13:09:56.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:56.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:56.221098+0000) 2024-03-06T13:09:57.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:57.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:57.221289+0000) 2024-03-06T13:09:58.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:58.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:58.221498+0000) 2024-03-06T13:09:59.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:09:59.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:59.221692+0000) 2024-03-06T13:09:59.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:59.662 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:00.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:00.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:00.221898+0000) 2024-03-06T13:10:01.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:01.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:01.222122+0000) 2024-03-06T13:10:02.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:02.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:02.222371+0000) 2024-03-06T13:10:03.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:03.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:03.222586+0000) 2024-03-06T13:10:04.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:04.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:04.222779+0000) 2024-03-06T13:10:05.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:05.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:05.222970+0000) 2024-03-06T13:10:06.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:06.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:06.223160+0000) 2024-03-06T13:10:07.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:07.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:07.223385+0000) 2024-03-06T13:10:08.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:08.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:08.223619+0000) 2024-03-06T13:10:09.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:09.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:09.223863+0000) 2024-03-06T13:10:10.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:10.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:10.224095+0000) 2024-03-06T13:10:11.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:11.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:11.224296+0000) 2024-03-06T13:10:12.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:12.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:12.224496+0000) 2024-03-06T13:10:13.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:13.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:13.224731+0000) 2024-03-06T13:10:14.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:14.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:14.224926+0000) 2024-03-06T13:10:15.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:15.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:15.225114+0000) 2024-03-06T13:10:16.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:16.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:16.225348+0000) 2024-03-06T13:10:17.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:17.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:17.225584+0000) 2024-03-06T13:10:18.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:18.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:18.225826+0000) 2024-03-06T13:10:19.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:19.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:19.226010+0000) 2024-03-06T13:10:20.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:20.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:20.226194+0000) 2024-03-06T13:10:21.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:21.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:21.226381+0000) 2024-03-06T13:10:22.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:22.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:22.226611+0000) 2024-03-06T13:10:23.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:23.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:23.226828+0000) 2024-03-06T13:10:24.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:24.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:24.227023+0000) 2024-03-06T13:10:25.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:25.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:25.227215+0000) 2024-03-06T13:10:26.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:26.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:26.227412+0000) 2024-03-06T13:10:27.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:27.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:27.227627+0000) 2024-03-06T13:10:28.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:28.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:28.227838+0000) 2024-03-06T13:10:29.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:29.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:29.228055+0000) 2024-03-06T13:10:29.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:29.882 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:30.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:30.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:30.228246+0000) 2024-03-06T13:10:31.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:31.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:31.228420+0000) 2024-03-06T13:10:32.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:32.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:32.228734+0000) 2024-03-06T13:10:33.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:33.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:33.228949+0000) 2024-03-06T13:10:34.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:34.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:34.229142+0000) 2024-03-06T13:10:35.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:35.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:35.229325+0000) 2024-03-06T13:10:36.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:36.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:36.229604+0000) 2024-03-06T13:10:37.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:37.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:37.229795+0000) 2024-03-06T13:10:38.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:38.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:38.230038+0000) 2024-03-06T13:10:39.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:39.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:39.230319+0000) 2024-03-06T13:10:40.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:40.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:40.230602+0000) 2024-03-06T13:10:41.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:41.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:41.230866+0000) 2024-03-06T13:10:42.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:42.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:42.231121+0000) 2024-03-06T13:10:43.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:43.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:43.231361+0000) 2024-03-06T13:10:44.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:44.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:44.231641+0000) 2024-03-06T13:10:45.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:45.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:45.231915+0000) 2024-03-06T13:10:46.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:46.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:46.232131+0000) 2024-03-06T13:10:47.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:47.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:47.232423+0000) 2024-03-06T13:10:48.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:48.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:48.232749+0000) 2024-03-06T13:10:49.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:49.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:49.232946+0000) 2024-03-06T13:10:50.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:50.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:50.233141+0000) 2024-03-06T13:10:51.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:51.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:51.233317+0000) 2024-03-06T13:10:52.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:52.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:52.233550+0000) 2024-03-06T13:10:53.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:53.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:53.233772+0000) 2024-03-06T13:10:54.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:54.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:54.233971+0000) 2024-03-06T13:10:55.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:55.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:55.234169+0000) 2024-03-06T13:10:56.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:56.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:56.234372+0000) 2024-03-06T13:10:57.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:57.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:57.234611+0000) 2024-03-06T13:10:58.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:58.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:58.234842+0000) 2024-03-06T13:10:59.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:10:59.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:59.235069+0000) 2024-03-06T13:11:00.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:00.086 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:00.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:00.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:00.235259+0000) 2024-03-06T13:11:01.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:01.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:01.235454+0000) 2024-03-06T13:11:02.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:02.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:02.235717+0000) 2024-03-06T13:11:03.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:03.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:03.235948+0000) 2024-03-06T13:11:04.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:04.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:04.236194+0000) 2024-03-06T13:11:05.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:05.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:05.236457+0000) 2024-03-06T13:11:06.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:06.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:06.236729+0000) 2024-03-06T13:11:07.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:07.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:07.236966+0000) 2024-03-06T13:11:08.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:08.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:08.237207+0000) 2024-03-06T13:11:09.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:09.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:09.237393+0000) 2024-03-06T13:11:10.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:10.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:10.237617+0000) 2024-03-06T13:11:11.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:11.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:11.237822+0000) 2024-03-06T13:11:12.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:12.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:12.238014+0000) 2024-03-06T13:11:13.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:13.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:13.238212+0000) 2024-03-06T13:11:14.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:14.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:14.238406+0000) 2024-03-06T13:11:15.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:15.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:15.238637+0000) 2024-03-06T13:11:16.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:16.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:16.238867+0000) 2024-03-06T13:11:17.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:17.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:17.239051+0000) 2024-03-06T13:11:18.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:18.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:18.239264+0000) 2024-03-06T13:11:19.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:19.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:19.239474+0000) 2024-03-06T13:11:20.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:20.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:20.239695+0000) 2024-03-06T13:11:21.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:21.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:21.239924+0000) 2024-03-06T13:11:22.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:22.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:22.240114+0000) 2024-03-06T13:11:23.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:23.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:23.240309+0000) 2024-03-06T13:11:24.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:24.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:24.240544+0000) 2024-03-06T13:11:25.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:25.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:25.240762+0000) 2024-03-06T13:11:26.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:26.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:26.240993+0000) 2024-03-06T13:11:27.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:27.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:27.241217+0000) 2024-03-06T13:11:28.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:28.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:28.241460+0000) 2024-03-06T13:11:29.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:29.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:29.241702+0000) 2024-03-06T13:11:30.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:30.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:30.241934+0000) 2024-03-06T13:11:30.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:30.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:31.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:31.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:31.242156+0000) 2024-03-06T13:11:32.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:32.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:32.242381+0000) 2024-03-06T13:11:33.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:33.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:33.242602+0000) 2024-03-06T13:11:34.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:34.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:34.242839+0000) 2024-03-06T13:11:35.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:35.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:35.243073+0000) 2024-03-06T13:11:36.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:36.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:36.243300+0000) 2024-03-06T13:11:37.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:37.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:37.243520+0000) 2024-03-06T13:11:38.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:38.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:38.243785+0000) 2024-03-06T13:11:39.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:39.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:39.243920+0000) 2024-03-06T13:11:40.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:40.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:40.244122+0000) 2024-03-06T13:11:41.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:41.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:41.244315+0000) 2024-03-06T13:11:42.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:42.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:42.244539+0000) 2024-03-06T13:11:43.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:43.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:43.244763+0000) 2024-03-06T13:11:44.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:44.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:44.244962+0000) 2024-03-06T13:11:45.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:45.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:45.245154+0000) 2024-03-06T13:11:46.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:46.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:46.245389+0000) 2024-03-06T13:11:47.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:47.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:47.245603+0000) 2024-03-06T13:11:48.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:48.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:48.245812+0000) 2024-03-06T13:11:49.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:49.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:49.245995+0000) 2024-03-06T13:11:50.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:50.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:50.246184+0000) 2024-03-06T13:11:51.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:51.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:51.246403+0000) 2024-03-06T13:11:52.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:52.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:52.246632+0000) 2024-03-06T13:11:53.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:53.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:53.246832+0000) 2024-03-06T13:11:54.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:54.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:54.247030+0000) 2024-03-06T13:11:55.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:55.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:55.247288+0000) 2024-03-06T13:11:56.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:56.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:56.247539+0000) 2024-03-06T13:11:57.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:57.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:57.247761+0000) 2024-03-06T13:11:58.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:58.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:58.247978+0000) 2024-03-06T13:11:59.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:11:59.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:59.248263+0000) 2024-03-06T13:12:00.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:00.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:00.248494+0000) 2024-03-06T13:12:00.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:00.505 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:01.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:01.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:01.248800+0000) 2024-03-06T13:12:02.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:02.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:02.249026+0000) 2024-03-06T13:12:03.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:03.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:03.249268+0000) 2024-03-06T13:12:04.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:04.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:04.249507+0000) 2024-03-06T13:12:05.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:05.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:05.249799+0000) 2024-03-06T13:12:06.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:06.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:06.250083+0000) 2024-03-06T13:12:07.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:07.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:07.250324+0000) 2024-03-06T13:12:08.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:08.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:08.250541+0000) 2024-03-06T13:12:09.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:09.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:09.250752+0000) 2024-03-06T13:12:10.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:10.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:10.250951+0000) 2024-03-06T13:12:11.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:11.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:11.251140+0000) 2024-03-06T13:12:12.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:12.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:12.251330+0000) 2024-03-06T13:12:13.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:13.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:13.251534+0000) 2024-03-06T13:12:14.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:14.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:14.251772+0000) 2024-03-06T13:12:15.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:15.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:15.251957+0000) 2024-03-06T13:12:16.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:16.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:16.252152+0000) 2024-03-06T13:12:17.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:17.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:17.252343+0000) 2024-03-06T13:12:18.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:18.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:18.252552+0000) 2024-03-06T13:12:19.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:19.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:19.252799+0000) 2024-03-06T13:12:20.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:20.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:20.252994+0000) 2024-03-06T13:12:21.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:21.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:21.253180+0000) 2024-03-06T13:12:22.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:22.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:22.253415+0000) 2024-03-06T13:12:23.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:23.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:23.253729+0000) 2024-03-06T13:12:24.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:24.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:24.253909+0000) 2024-03-06T13:12:25.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:25.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:25.254095+0000) 2024-03-06T13:12:26.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:26.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:26.254298+0000) 2024-03-06T13:12:27.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:27.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:27.254516+0000) 2024-03-06T13:12:28.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:28.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:28.254762+0000) 2024-03-06T13:12:29.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:29.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:29.254962+0000) 2024-03-06T13:12:30.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:30.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:30.255163+0000) 2024-03-06T13:12:30.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:30.725 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:31.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:31.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:31.255363+0000) 2024-03-06T13:12:32.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:32.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:32.255589+0000) 2024-03-06T13:12:33.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:33.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:33.255816+0000) 2024-03-06T13:12:34.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:34.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:34.256016+0000) 2024-03-06T13:12:35.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:35.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:35.256201+0000) 2024-03-06T13:12:36.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:36.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:36.256434+0000) 2024-03-06T13:12:37.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:37.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:37.256691+0000) 2024-03-06T13:12:38.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:38.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:38.256935+0000) 2024-03-06T13:12:39.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:39.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:39.257154+0000) 2024-03-06T13:12:40.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:40.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:40.257394+0000) 2024-03-06T13:12:41.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:41.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:41.257627+0000) 2024-03-06T13:12:42.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:42.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:42.257795+0000) 2024-03-06T13:12:43.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:43.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:43.257985+0000) 2024-03-06T13:12:44.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:44.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:44.258191+0000) 2024-03-06T13:12:45.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:45.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:45.258378+0000) 2024-03-06T13:12:46.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:46.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:46.258615+0000) 2024-03-06T13:12:47.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:47.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:47.258842+0000) 2024-03-06T13:12:48.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:48.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:48.259083+0000) 2024-03-06T13:12:49.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:49.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:49.259243+0000) 2024-03-06T13:12:50.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:50.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:50.259475+0000) 2024-03-06T13:12:51.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:51.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:51.259731+0000) 2024-03-06T13:12:52.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:52.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:52.259922+0000) 2024-03-06T13:12:53.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:53.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:53.260112+0000) 2024-03-06T13:12:54.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:54.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:54.260302+0000) 2024-03-06T13:12:55.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:55.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:55.260519+0000) 2024-03-06T13:12:56.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:56.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:56.260754+0000) 2024-03-06T13:12:57.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:57.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:57.260953+0000) 2024-03-06T13:12:58.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:58.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:58.261162+0000) 2024-03-06T13:12:59.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:12:59.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:59.261348+0000) 2024-03-06T13:13:00.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:00.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:00.261576+0000) 2024-03-06T13:13:00.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:00.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:01.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:01.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:01.261781+0000) 2024-03-06T13:13:02.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:02.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:02.261984+0000) 2024-03-06T13:13:03.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:03.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:03.262169+0000) 2024-03-06T13:13:04.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:04.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:04.262371+0000) 2024-03-06T13:13:05.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:05.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:05.262579+0000) 2024-03-06T13:13:06.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:06.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:06.262821+0000) 2024-03-06T13:13:07.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:07.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:07.263064+0000) 2024-03-06T13:13:08.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:08.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:08.263334+0000) 2024-03-06T13:13:09.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:09.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:09.263572+0000) 2024-03-06T13:13:10.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:10.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:10.263790+0000) 2024-03-06T13:13:11.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:11.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:11.263969+0000) 2024-03-06T13:13:12.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:12.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:12.264189+0000) 2024-03-06T13:13:13.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:13.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:13.264466+0000) 2024-03-06T13:13:14.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:14.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:14.264778+0000) 2024-03-06T13:13:15.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:15.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:15.265043+0000) 2024-03-06T13:13:16.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:16.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:16.265318+0000) 2024-03-06T13:13:17.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:17.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:17.265515+0000) 2024-03-06T13:13:18.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:18.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:18.265784+0000) 2024-03-06T13:13:19.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:19.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:19.266006+0000) 2024-03-06T13:13:20.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:20.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:20.266251+0000) 2024-03-06T13:13:21.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:21.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:21.266493+0000) 2024-03-06T13:13:22.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:22.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:22.266734+0000) 2024-03-06T13:13:23.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:23.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:23.266919+0000) 2024-03-06T13:13:24.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:24.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:24.267111+0000) 2024-03-06T13:13:25.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:25.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:25.267294+0000) 2024-03-06T13:13:26.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:26.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:26.267530+0000) 2024-03-06T13:13:27.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:27.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:27.267780+0000) 2024-03-06T13:13:28.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:28.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:28.267984+0000) 2024-03-06T13:13:29.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:29.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:29.268207+0000) 2024-03-06T13:13:30.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:30.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:30.268448+0000) 2024-03-06T13:13:31.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:31.167 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:31.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:31.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:31.268715+0000) 2024-03-06T13:13:32.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:32.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:32.268912+0000) 2024-03-06T13:13:33.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:33.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:33.269100+0000) 2024-03-06T13:13:34.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:34.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:34.269285+0000) 2024-03-06T13:13:35.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:35.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:35.269465+0000) 2024-03-06T13:13:36.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:36.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:36.269737+0000) 2024-03-06T13:13:37.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:37.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:37.269930+0000) 2024-03-06T13:13:38.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:38.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:38.270168+0000) 2024-03-06T13:13:39.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:39.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:39.270386+0000) 2024-03-06T13:13:40.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:40.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:40.270620+0000) 2024-03-06T13:13:41.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:41.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:41.270832+0000) 2024-03-06T13:13:42.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:42.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:42.271057+0000) 2024-03-06T13:13:43.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:43.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:43.271284+0000) 2024-03-06T13:13:44.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:44.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:44.271510+0000) 2024-03-06T13:13:45.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:45.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:45.271784+0000) 2024-03-06T13:13:46.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:46.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:46.271959+0000) 2024-03-06T13:13:47.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:47.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:47.272161+0000) 2024-03-06T13:13:48.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:48.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:48.272413+0000) 2024-03-06T13:13:49.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:49.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:49.272635+0000) 2024-03-06T13:13:50.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:50.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:50.272821+0000) 2024-03-06T13:13:51.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:51.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:51.273005+0000) 2024-03-06T13:13:52.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:52.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:52.273196+0000) 2024-03-06T13:13:53.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:53.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:53.273390+0000) 2024-03-06T13:13:54.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:54.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:54.273624+0000) 2024-03-06T13:13:55.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:55.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:55.273813+0000) 2024-03-06T13:13:56.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:56.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:56.274007+0000) 2024-03-06T13:13:57.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:57.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:57.274212+0000) 2024-03-06T13:13:58.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:58.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:58.274416+0000) 2024-03-06T13:13:59.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:13:59.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:59.274642+0000) 2024-03-06T13:14:00.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:00.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:00.274904+0000) 2024-03-06T13:14:01.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:01.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:01.275098+0000) 2024-03-06T13:14:01.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:01.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:02.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:02.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:02.275301+0000) 2024-03-06T13:14:03.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:03.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:03.275525+0000) 2024-03-06T13:14:04.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:04.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:04.275795+0000) 2024-03-06T13:14:05.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:05.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:05.275973+0000) 2024-03-06T13:14:06.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:06.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:06.276166+0000) 2024-03-06T13:14:07.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:07.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:07.276343+0000) 2024-03-06T13:14:08.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:08.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:08.276558+0000) 2024-03-06T13:14:09.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:09.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:09.276814+0000) 2024-03-06T13:14:10.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:10.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:10.277012+0000) 2024-03-06T13:14:11.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:11.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:11.277234+0000) 2024-03-06T13:14:12.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:12.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:12.277460+0000) 2024-03-06T13:14:13.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:13.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:13.277645+0000) 2024-03-06T13:14:14.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:14.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:14.277891+0000) 2024-03-06T13:14:15.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:15.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:15.278136+0000) 2024-03-06T13:14:16.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:16.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:16.278364+0000) 2024-03-06T13:14:17.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:17.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:17.278600+0000) 2024-03-06T13:14:18.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:18.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:18.278849+0000) 2024-03-06T13:14:19.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:19.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:19.279072+0000) 2024-03-06T13:14:20.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:20.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:20.279307+0000) 2024-03-06T13:14:21.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:21.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:21.279523+0000) 2024-03-06T13:14:22.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:22.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:22.279794+0000) 2024-03-06T13:14:23.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:23.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:23.280070+0000) 2024-03-06T13:14:24.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:24.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:24.280361+0000) 2024-03-06T13:14:25.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:25.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:25.280635+0000) 2024-03-06T13:14:26.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:26.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:26.280974+0000) 2024-03-06T13:14:27.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:27.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:27.281166+0000) 2024-03-06T13:14:28.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:28.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:28.281412+0000) 2024-03-06T13:14:29.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:29.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:29.281643+0000) 2024-03-06T13:14:30.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:30.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:30.281909+0000) 2024-03-06T13:14:31.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:31.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:31.282174+0000) 2024-03-06T13:14:31.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:31.584 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:32.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:32.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:32.282428+0000) 2024-03-06T13:14:33.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:33.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:33.282736+0000) 2024-03-06T13:14:34.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:34.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:34.283030+0000) 2024-03-06T13:14:35.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:35.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:35.283267+0000) 2024-03-06T13:14:36.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:36.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:36.283543+0000) 2024-03-06T13:14:37.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:37.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:37.283780+0000) 2024-03-06T13:14:38.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:38.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:38.284026+0000) 2024-03-06T13:14:39.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:39.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:39.284291+0000) 2024-03-06T13:14:40.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:40.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:40.284527+0000) 2024-03-06T13:14:41.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:41.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:41.284781+0000) 2024-03-06T13:14:42.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:42.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:42.284989+0000) 2024-03-06T13:14:43.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:43.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:43.285222+0000) 2024-03-06T13:14:44.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:44.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:44.285449+0000) 2024-03-06T13:14:45.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:45.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:45.285695+0000) 2024-03-06T13:14:46.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:46.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:46.285890+0000) 2024-03-06T13:14:47.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:47.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:47.286079+0000) 2024-03-06T13:14:48.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:48.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:48.286326+0000) 2024-03-06T13:14:49.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:49.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:49.286551+0000) 2024-03-06T13:14:50.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:50.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:50.286791+0000) 2024-03-06T13:14:51.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:51.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:51.287013+0000) 2024-03-06T13:14:52.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:52.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:52.287204+0000) 2024-03-06T13:14:53.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:53.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:53.287439+0000) 2024-03-06T13:14:54.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:54.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:54.287694+0000) 2024-03-06T13:14:55.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:55.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:55.287862+0000) 2024-03-06T13:14:56.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:56.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:56.288060+0000) 2024-03-06T13:14:57.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:57.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:57.288341+0000) 2024-03-06T13:14:58.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:58.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:58.288605+0000) 2024-03-06T13:14:59.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:14:59.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:59.288829+0000) 2024-03-06T13:15:00.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:00.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:00.289059+0000) 2024-03-06T13:15:01.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:01.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:01.289320+0000) 2024-03-06T13:15:01.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:01.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:02.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:02.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:02.289521+0000) 2024-03-06T13:15:03.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:03.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:03.289821+0000) 2024-03-06T13:15:04.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:04.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:04.290109+0000) 2024-03-06T13:15:05.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:05.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:05.290349+0000) 2024-03-06T13:15:06.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:06.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:06.290588+0000) 2024-03-06T13:15:07.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:07.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:07.290851+0000) 2024-03-06T13:15:08.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:08.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:08.291171+0000) 2024-03-06T13:15:09.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:09.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:09.291447+0000) 2024-03-06T13:15:10.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:10.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:10.291755+0000) 2024-03-06T13:15:11.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:11.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:11.292039+0000) 2024-03-06T13:15:12.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:12.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:12.292274+0000) 2024-03-06T13:15:13.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:13.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:13.292493+0000) 2024-03-06T13:15:14.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:14.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:14.292773+0000) 2024-03-06T13:15:15.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:15.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:15.292951+0000) 2024-03-06T13:15:16.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:16.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:16.293142+0000) 2024-03-06T13:15:17.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:17.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:17.293360+0000) 2024-03-06T13:15:18.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:18.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:18.293561+0000) 2024-03-06T13:15:19.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:19.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:19.293803+0000) 2024-03-06T13:15:20.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:20.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:20.294037+0000) 2024-03-06T13:15:21.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:21.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:21.294250+0000) 2024-03-06T13:15:22.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:22.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:22.294489+0000) 2024-03-06T13:15:23.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:23.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:23.294752+0000) 2024-03-06T13:15:24.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:24.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:24.295030+0000) 2024-03-06T13:15:25.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:25.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:25.295221+0000) 2024-03-06T13:15:26.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:26.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:26.295463+0000) 2024-03-06T13:15:27.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:27.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:27.295648+0000) 2024-03-06T13:15:28.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:28.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:28.295951+0000) 2024-03-06T13:15:29.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:29.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:29.296152+0000) 2024-03-06T13:15:30.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:30.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:30.296396+0000) 2024-03-06T13:15:31.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:31.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:31.296638+0000) 2024-03-06T13:15:32.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:32.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:32.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:32.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:32.296959+0000) 2024-03-06T13:15:33.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:33.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:33.297176+0000) 2024-03-06T13:15:34.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:34.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:34.297425+0000) 2024-03-06T13:15:35.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:35.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:35.297704+0000) 2024-03-06T13:15:36.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:36.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:36.297952+0000) 2024-03-06T13:15:37.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:37.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:37.298158+0000) 2024-03-06T13:15:38.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:38.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:38.298374+0000) 2024-03-06T13:15:39.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:39.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:39.298600+0000) 2024-03-06T13:15:40.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:40.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:40.298850+0000) 2024-03-06T13:15:41.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:41.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:41.299069+0000) 2024-03-06T13:15:42.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:42.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:42.299288+0000) 2024-03-06T13:15:43.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:43.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:43.299522+0000) 2024-03-06T13:15:44.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:44.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:44.299803+0000) 2024-03-06T13:15:45.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:45.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:45.300044+0000) 2024-03-06T13:15:46.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:46.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:46.300240+0000) 2024-03-06T13:15:47.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:47.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:47.300437+0000) 2024-03-06T13:15:48.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:48.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:48.300730+0000) 2024-03-06T13:15:49.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:49.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:49.300923+0000) 2024-03-06T13:15:50.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:50.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:50.301149+0000) 2024-03-06T13:15:51.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:51.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:51.301375+0000) 2024-03-06T13:15:52.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:52.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:52.301594+0000) 2024-03-06T13:15:53.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:53.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:53.301835+0000) 2024-03-06T13:15:54.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:54.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:54.302072+0000) 2024-03-06T13:15:55.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:55.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:55.302256+0000) 2024-03-06T13:15:56.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:56.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:56.302497+0000) 2024-03-06T13:15:57.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:57.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:57.302721+0000) 2024-03-06T13:15:58.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:58.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:58.302928+0000) 2024-03-06T13:15:59.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:15:59.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:59.303156+0000) 2024-03-06T13:16:00.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:00.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:00.303385+0000) 2024-03-06T13:16:01.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:01.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:01.303652+0000) 2024-03-06T13:16:02.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:02.206 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:02.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:02.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:02.303894+0000) 2024-03-06T13:16:03.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:03.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:03.304129+0000) 2024-03-06T13:16:04.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:04.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:04.304335+0000) 2024-03-06T13:16:05.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:05.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:05.304538+0000) 2024-03-06T13:16:06.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:06.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:06.304793+0000) 2024-03-06T13:16:07.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:07.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:07.305011+0000) 2024-03-06T13:16:08.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:08.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:08.305216+0000) 2024-03-06T13:16:09.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:09.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:09.305445+0000) 2024-03-06T13:16:10.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:10.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:10.305705+0000) 2024-03-06T13:16:11.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:11.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:11.305942+0000) 2024-03-06T13:16:12.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:12.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:12.306227+0000) 2024-03-06T13:16:13.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:13.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:13.306509+0000) 2024-03-06T13:16:14.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:14.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:14.306836+0000) 2024-03-06T13:16:15.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:15.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:15.307077+0000) 2024-03-06T13:16:16.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:16.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:16.307304+0000) 2024-03-06T13:16:17.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:17.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:17.307535+0000) 2024-03-06T13:16:18.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:18.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:18.307784+0000) 2024-03-06T13:16:19.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:19.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:19.308012+0000) 2024-03-06T13:16:20.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:20.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:20.308247+0000) 2024-03-06T13:16:21.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:21.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:21.308475+0000) 2024-03-06T13:16:22.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:22.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:22.308744+0000) 2024-03-06T13:16:23.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:23.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:23.308976+0000) 2024-03-06T13:16:24.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:24.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:24.309209+0000) 2024-03-06T13:16:25.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:25.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:25.309477+0000) 2024-03-06T13:16:26.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:26.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:26.309803+0000) 2024-03-06T13:16:27.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:27.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:27.309966+0000) 2024-03-06T13:16:28.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:28.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:28.310227+0000) 2024-03-06T13:16:29.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:29.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:29.310467+0000) 2024-03-06T13:16:30.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:30.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:30.310752+0000) 2024-03-06T13:16:31.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:31.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:31.310996+0000) 2024-03-06T13:16:32.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:32.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:32.311245+0000) 2024-03-06T13:16:32.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:32.416 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:33.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:33.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:33.311544+0000) 2024-03-06T13:16:34.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:34.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:34.311784+0000) 2024-03-06T13:16:35.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:35.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:35.311962+0000) 2024-03-06T13:16:36.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:36.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:36.312194+0000) 2024-03-06T13:16:37.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:37.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:37.312411+0000) 2024-03-06T13:16:38.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:38.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:38.312641+0000) 2024-03-06T13:16:39.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:39.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:39.312978+0000) 2024-03-06T13:16:40.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:40.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:40.313268+0000) 2024-03-06T13:16:41.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:41.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:41.313550+0000) 2024-03-06T13:16:42.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:42.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:42.313830+0000) 2024-03-06T13:16:43.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:43.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:43.314112+0000) 2024-03-06T13:16:44.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:44.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:44.314389+0000) 2024-03-06T13:16:45.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:45.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:45.314651+0000) 2024-03-06T13:16:46.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:46.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:46.314945+0000) 2024-03-06T13:16:47.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:47.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:47.315200+0000) 2024-03-06T13:16:48.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:48.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:48.315512+0000) 2024-03-06T13:16:49.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:49.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:49.315819+0000) 2024-03-06T13:16:50.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:50.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:50.316071+0000) 2024-03-06T13:16:51.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:51.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:51.316273+0000) 2024-03-06T13:16:52.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:52.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:52.316513+0000) 2024-03-06T13:16:53.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:53.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:53.316826+0000) 2024-03-06T13:16:54.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:54.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:54.317058+0000) 2024-03-06T13:16:55.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:55.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:55.317286+0000) 2024-03-06T13:16:56.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:56.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:56.317521+0000) 2024-03-06T13:16:57.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:57.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:57.317779+0000) 2024-03-06T13:16:58.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:58.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:58.318020+0000) 2024-03-06T13:16:59.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:16:59.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:59.318226+0000) 2024-03-06T13:17:00.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:00.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:00.318430+0000) 2024-03-06T13:17:01.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:01.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:01.318655+0000) 2024-03-06T13:17:02.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:02.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:02.318923+0000) 2024-03-06T13:17:02.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:02.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:03.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:03.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:03.319140+0000) 2024-03-06T13:17:04.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:04.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:04.319342+0000) 2024-03-06T13:17:05.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:05.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:05.319578+0000) 2024-03-06T13:17:06.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:06.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:06.319823+0000) 2024-03-06T13:17:07.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:07.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:07.320042+0000) 2024-03-06T13:17:08.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:08.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:08.320278+0000) 2024-03-06T13:17:09.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:09.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:09.320499+0000) 2024-03-06T13:17:10.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:10.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:10.320778+0000) 2024-03-06T13:17:11.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:11.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:11.321003+0000) 2024-03-06T13:17:12.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:12.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:12.321230+0000) 2024-03-06T13:17:13.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:13.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:13.321453+0000) 2024-03-06T13:17:14.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:14.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:14.321724+0000) 2024-03-06T13:17:15.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:15.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:15.321952+0000) 2024-03-06T13:17:16.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:16.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:16.322182+0000) 2024-03-06T13:17:17.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:17.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:17.322403+0000) 2024-03-06T13:17:18.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:18.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:18.322638+0000) 2024-03-06T13:17:19.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:19.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:19.322902+0000) 2024-03-06T13:17:20.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:20.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:20.323129+0000) 2024-03-06T13:17:21.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:21.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:21.323359+0000) 2024-03-06T13:17:22.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:22.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:22.323582+0000) 2024-03-06T13:17:23.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:23.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:23.323831+0000) 2024-03-06T13:17:24.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:24.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:24.324063+0000) 2024-03-06T13:17:25.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:25.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:25.324300+0000) 2024-03-06T13:17:26.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:26.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:26.324483+0000) 2024-03-06T13:17:27.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:27.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:27.324794+0000) 2024-03-06T13:17:28.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:28.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:28.325031+0000) 2024-03-06T13:17:29.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:29.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:29.325257+0000) 2024-03-06T13:17:30.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:30.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:30.325501+0000) 2024-03-06T13:17:31.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:31.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:31.325723+0000) 2024-03-06T13:17:32.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:32.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:32.325942+0000) 2024-03-06T13:17:32.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:32.811 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:33.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:33.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:33.326182+0000) 2024-03-06T13:17:34.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:34.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:34.326388+0000) 2024-03-06T13:17:35.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:35.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:35.326622+0000) 2024-03-06T13:17:36.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:36.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:36.326864+0000) 2024-03-06T13:17:37.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:37.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:37.327099+0000) 2024-03-06T13:17:38.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:38.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:38.327290+0000) 2024-03-06T13:17:39.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:39.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:39.327535+0000) 2024-03-06T13:17:40.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:40.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:40.327814+0000) 2024-03-06T13:17:41.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:41.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:41.328014+0000) 2024-03-06T13:17:42.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:42.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:42.328256+0000) 2024-03-06T13:17:43.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:43.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:43.328495+0000) 2024-03-06T13:17:44.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:44.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:44.328700+0000) 2024-03-06T13:17:45.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:45.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:45.328927+0000) 2024-03-06T13:17:46.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:46.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:46.329162+0000) 2024-03-06T13:17:47.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:47.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:47.329384+0000) 2024-03-06T13:17:48.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:48.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:48.329610+0000) 2024-03-06T13:17:49.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:49.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:49.329837+0000) 2024-03-06T13:17:50.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:50.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:50.330064+0000) 2024-03-06T13:17:51.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:51.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:51.330286+0000) 2024-03-06T13:17:52.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:52.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:52.330522+0000) 2024-03-06T13:17:53.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:53.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:53.330755+0000) 2024-03-06T13:17:54.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:54.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:54.331006+0000) 2024-03-06T13:17:55.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:55.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:55.331234+0000) 2024-03-06T13:17:56.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:56.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:56.331434+0000) 2024-03-06T13:17:57.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:57.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:57.331700+0000) 2024-03-06T13:17:58.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:58.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:58.331891+0000) 2024-03-06T13:17:59.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:17:59.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:59.332090+0000) 2024-03-06T13:18:00.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:00.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:00.332266+0000) 2024-03-06T13:18:01.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:01.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:01.332446+0000) 2024-03-06T13:18:02.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:02.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:02.332661+0000) 2024-03-06T13:18:03.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:03.045 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:03.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:03.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:03.332917+0000) 2024-03-06T13:18:04.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:04.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:04.333154+0000) 2024-03-06T13:18:05.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:05.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:05.333383+0000) 2024-03-06T13:18:06.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:06.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:06.333613+0000) 2024-03-06T13:18:07.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:07.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:07.333814+0000) 2024-03-06T13:18:08.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:08.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:08.334002+0000) 2024-03-06T13:18:09.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:09.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:09.334261+0000) 2024-03-06T13:18:10.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:10.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:10.334498+0000) 2024-03-06T13:18:11.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:11.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:11.334765+0000) 2024-03-06T13:18:12.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:12.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:12.334989+0000) 2024-03-06T13:18:13.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:13.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:13.335228+0000) 2024-03-06T13:18:14.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:14.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:14.335462+0000) 2024-03-06T13:18:15.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:15.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:15.335793+0000) 2024-03-06T13:18:16.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:16.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:16.336030+0000) 2024-03-06T13:18:17.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:17.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:17.336253+0000) 2024-03-06T13:18:18.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:18.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:18.336494+0000) 2024-03-06T13:18:19.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:19.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:19.336714+0000) 2024-03-06T13:18:20.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:20.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:20.336915+0000) 2024-03-06T13:18:21.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:21.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:21.337108+0000) 2024-03-06T13:18:22.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:22.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:22.337365+0000) 2024-03-06T13:18:23.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:23.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:23.337591+0000) 2024-03-06T13:18:24.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:24.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:24.337839+0000) 2024-03-06T13:18:25.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:25.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:25.338061+0000) 2024-03-06T13:18:26.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:26.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:26.338289+0000) 2024-03-06T13:18:27.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:27.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:27.338508+0000) 2024-03-06T13:18:28.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:28.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:28.338729+0000) 2024-03-06T13:18:29.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:29.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:29.338917+0000) 2024-03-06T13:18:30.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:30.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:30.339178+0000) 2024-03-06T13:18:31.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:31.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:31.339395+0000) 2024-03-06T13:18:32.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:32.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:32.339703+0000) 2024-03-06T13:18:33.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:33.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:33.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:33.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:33.339835+0000) 2024-03-06T13:18:34.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:34.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:34.340076+0000) 2024-03-06T13:18:35.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:35.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:35.340316+0000) 2024-03-06T13:18:36.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:36.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:36.340513+0000) 2024-03-06T13:18:37.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:37.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:37.340768+0000) 2024-03-06T13:18:38.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:38.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:38.341026+0000) 2024-03-06T13:18:39.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:39.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:39.341210+0000) 2024-03-06T13:18:40.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:40.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:40.341392+0000) 2024-03-06T13:18:41.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:41.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:41.341622+0000) 2024-03-06T13:18:42.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:42.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:42.341853+0000) 2024-03-06T13:18:43.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:43.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:43.342098+0000) 2024-03-06T13:18:44.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:44.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:44.342344+0000) 2024-03-06T13:18:45.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:45.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:45.342604+0000) 2024-03-06T13:18:46.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:46.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:46.342884+0000) 2024-03-06T13:18:47.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:47.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:47.343087+0000) 2024-03-06T13:18:48.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:48.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:48.343341+0000) 2024-03-06T13:18:49.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:49.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:49.343582+0000) 2024-03-06T13:18:50.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:50.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:50.343844+0000) 2024-03-06T13:18:51.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:51.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:51.344092+0000) 2024-03-06T13:18:52.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:52.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:52.344337+0000) 2024-03-06T13:18:53.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:53.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:53.344575+0000) 2024-03-06T13:18:54.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:54.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:54.344828+0000) 2024-03-06T13:18:55.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:55.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:55.345064+0000) 2024-03-06T13:18:56.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:56.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:56.345314+0000) 2024-03-06T13:18:57.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:57.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:57.345565+0000) 2024-03-06T13:18:58.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:58.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:58.345818+0000) 2024-03-06T13:18:59.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:18:59.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:59.346058+0000) 2024-03-06T13:19:00.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:00.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:00.346312+0000) 2024-03-06T13:19:01.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:01.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:01.346513+0000) 2024-03-06T13:19:02.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:02.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:02.346775+0000) 2024-03-06T13:19:03.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:03.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:03.347049+0000) 2024-03-06T13:19:03.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:03.390 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:04.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:04.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:04.347273+0000) 2024-03-06T13:19:05.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:05.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:05.347484+0000) 2024-03-06T13:19:06.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:06.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:06.347766+0000) 2024-03-06T13:19:07.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:07.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:07.347993+0000) 2024-03-06T13:19:08.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:08.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:08.348199+0000) 2024-03-06T13:19:09.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:09.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:09.348461+0000) 2024-03-06T13:19:10.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:10.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:10.348761+0000) 2024-03-06T13:19:11.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:11.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:11.349015+0000) 2024-03-06T13:19:12.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:12.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:12.349256+0000) 2024-03-06T13:19:13.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:13.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:13.349424+0000) 2024-03-06T13:19:14.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:14.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:14.349619+0000) 2024-03-06T13:19:15.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:15.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:15.349839+0000) 2024-03-06T13:19:16.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:16.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:16.350123+0000) 2024-03-06T13:19:17.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:17.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:17.350405+0000) 2024-03-06T13:19:18.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:18.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:18.350749+0000) 2024-03-06T13:19:19.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:19.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:19.350974+0000) 2024-03-06T13:19:20.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:20.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:20.351244+0000) 2024-03-06T13:19:21.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:21.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:21.351489+0000) 2024-03-06T13:19:22.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:22.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:22.351806+0000) 2024-03-06T13:19:23.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:23.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:23.352088+0000) 2024-03-06T13:19:24.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:24.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:24.352366+0000) 2024-03-06T13:19:25.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:25.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:25.352629+0000) 2024-03-06T13:19:26.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:26.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:26.352962+0000) 2024-03-06T13:19:27.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:27.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:27.353199+0000) 2024-03-06T13:19:28.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:28.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:28.353490+0000) 2024-03-06T13:19:29.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:29.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:29.353740+0000) 2024-03-06T13:19:30.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:30.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:30.353991+0000) 2024-03-06T13:19:31.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:31.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:31.354234+0000) 2024-03-06T13:19:32.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:32.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:32.354487+0000) 2024-03-06T13:19:33.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:33.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:33.354771+0000) 2024-03-06T13:19:33.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:33.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:34.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:34.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:34.355012+0000) 2024-03-06T13:19:35.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:35.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:35.355238+0000) 2024-03-06T13:19:36.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:36.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:36.355502+0000) 2024-03-06T13:19:37.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:37.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:37.355738+0000) 2024-03-06T13:19:38.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:38.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:38.355979+0000) 2024-03-06T13:19:39.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:39.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:39.356245+0000) 2024-03-06T13:19:40.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:40.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:40.356527+0000) 2024-03-06T13:19:41.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:41.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:41.356764+0000) 2024-03-06T13:19:42.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:42.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:42.356990+0000) 2024-03-06T13:19:43.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:43.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:43.357227+0000) 2024-03-06T13:19:44.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:44.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:44.357457+0000) 2024-03-06T13:19:45.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:45.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:45.357708+0000) 2024-03-06T13:19:46.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:46.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:46.357908+0000) 2024-03-06T13:19:47.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:47.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:47.358099+0000) 2024-03-06T13:19:48.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:48.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:48.358329+0000) 2024-03-06T13:19:49.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:49.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:49.358556+0000) 2024-03-06T13:19:50.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:50.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:50.358828+0000) 2024-03-06T13:19:51.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:51.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:51.358986+0000) 2024-03-06T13:19:52.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:52.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:52.359213+0000) 2024-03-06T13:19:53.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:53.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:53.359442+0000) 2024-03-06T13:19:54.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:54.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:54.359628+0000) 2024-03-06T13:19:55.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:55.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:55.359890+0000) 2024-03-06T13:19:56.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:56.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:56.360153+0000) 2024-03-06T13:19:57.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:57.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:57.360424+0000) 2024-03-06T13:19:58.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:58.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:58.360698+0000) 2024-03-06T13:19:59.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:19:59.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:59.360973+0000) 2024-03-06T13:20:00.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:00.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:00.361263+0000) 2024-03-06T13:20:01.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:01.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:01.361477+0000) 2024-03-06T13:20:02.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:02.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:02.361751+0000) 2024-03-06T13:20:03.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:03.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:03.361938+0000) 2024-03-06T13:20:03.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:03.744 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:04.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:04.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:04.362165+0000) 2024-03-06T13:20:05.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:05.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:05.362340+0000) 2024-03-06T13:20:06.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:06.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:06.362561+0000) 2024-03-06T13:20:07.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:07.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:07.362839+0000) 2024-03-06T13:20:08.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:08.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:08.363092+0000) 2024-03-06T13:20:09.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:09.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:09.363341+0000) 2024-03-06T13:20:10.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:10.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:10.363531+0000) 2024-03-06T13:20:11.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:11.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:11.363799+0000) 2024-03-06T13:20:12.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:12.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:12.364036+0000) 2024-03-06T13:20:13.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:13.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:13.364263+0000) 2024-03-06T13:20:14.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:14.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:14.364502+0000) 2024-03-06T13:20:15.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:15.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:15.364768+0000) 2024-03-06T13:20:16.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:16.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:16.365006+0000) 2024-03-06T13:20:17.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:17.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:17.365186+0000) 2024-03-06T13:20:18.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:18.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:18.365427+0000) 2024-03-06T13:20:19.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:19.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:19.365661+0000) 2024-03-06T13:20:20.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:20.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:20.365933+0000) 2024-03-06T13:20:21.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:21.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:21.366158+0000) 2024-03-06T13:20:22.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:22.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:22.366356+0000) 2024-03-06T13:20:23.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:23.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:23.366572+0000) 2024-03-06T13:20:24.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:24.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:24.366805+0000) 2024-03-06T13:20:25.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:25.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:25.366999+0000) 2024-03-06T13:20:26.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:26.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:26.367227+0000) 2024-03-06T13:20:27.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:27.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:27.367448+0000) 2024-03-06T13:20:28.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:28.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:28.367771+0000) 2024-03-06T13:20:29.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:29.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:29.368028+0000) 2024-03-06T13:20:30.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:30.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:30.368230+0000) 2024-03-06T13:20:31.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:31.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:31.368449+0000) 2024-03-06T13:20:32.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:32.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:32.368717+0000) 2024-03-06T13:20:33.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:33.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:33.368900+0000) 2024-03-06T13:20:33.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:33.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:34.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:34.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:34.369098+0000) 2024-03-06T13:20:35.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:35.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:35.369288+0000) 2024-03-06T13:20:36.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:36.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:36.369523+0000) 2024-03-06T13:20:37.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:37.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:37.369783+0000) 2024-03-06T13:20:38.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:38.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:38.370026+0000) 2024-03-06T13:20:39.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:39.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:39.370264+0000) 2024-03-06T13:20:40.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:40.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:40.370497+0000) 2024-03-06T13:20:41.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:41.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:41.370769+0000) 2024-03-06T13:20:42.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:42.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:42.371052+0000) 2024-03-06T13:20:43.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:43.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:43.371298+0000) 2024-03-06T13:20:44.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:44.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:44.371591+0000) 2024-03-06T13:20:45.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:45.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:45.371783+0000) 2024-03-06T13:20:46.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:46.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:46.372013+0000) 2024-03-06T13:20:47.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:47.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:47.372234+0000) 2024-03-06T13:20:48.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:48.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:48.372474+0000) 2024-03-06T13:20:49.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:49.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:49.372787+0000) 2024-03-06T13:20:50.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:50.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:50.373074+0000) 2024-03-06T13:20:51.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:51.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:51.373356+0000) 2024-03-06T13:20:52.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:52.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:52.373636+0000) 2024-03-06T13:20:53.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:53.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:53.373912+0000) 2024-03-06T13:20:54.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:54.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:54.374166+0000) 2024-03-06T13:20:55.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:55.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:55.374445+0000) 2024-03-06T13:20:56.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:56.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:56.374765+0000) 2024-03-06T13:20:57.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:57.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:57.374966+0000) 2024-03-06T13:20:58.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:58.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:58.375171+0000) 2024-03-06T13:20:59.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:20:59.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:59.375408+0000) 2024-03-06T13:21:00.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:00.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:00.375699+0000) 2024-03-06T13:21:01.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:01.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:01.375985+0000) 2024-03-06T13:21:02.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:02.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:02.376268+0000) 2024-03-06T13:21:03.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:03.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:03.376506+0000) 2024-03-06T13:21:04.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:04.172 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:04.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:04.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:04.376807+0000) 2024-03-06T13:21:05.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:05.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:05.376978+0000) 2024-03-06T13:21:06.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:06.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:06.377209+0000) 2024-03-06T13:21:07.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:07.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:07.377388+0000) 2024-03-06T13:21:08.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:08.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:08.377627+0000) 2024-03-06T13:21:09.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:09.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:09.377873+0000) 2024-03-06T13:21:10.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:10.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:10.378086+0000) 2024-03-06T13:21:11.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:11.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:11.378330+0000) 2024-03-06T13:21:12.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:12.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:12.378547+0000) 2024-03-06T13:21:13.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:13.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:13.378802+0000) 2024-03-06T13:21:14.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:14.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:14.379006+0000) 2024-03-06T13:21:15.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:15.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:15.379252+0000) 2024-03-06T13:21:16.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:16.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:16.379462+0000) 2024-03-06T13:21:17.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:17.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:17.379715+0000) 2024-03-06T13:21:18.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:18.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:18.379933+0000) 2024-03-06T13:21:19.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:19.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:19.380195+0000) 2024-03-06T13:21:20.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:20.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:20.380477+0000) 2024-03-06T13:21:21.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:21.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:21.380758+0000) 2024-03-06T13:21:22.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:22.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:22.380971+0000) 2024-03-06T13:21:23.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:23.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:23.381206+0000) 2024-03-06T13:21:24.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:24.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:24.381441+0000) 2024-03-06T13:21:25.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:25.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:25.381696+0000) 2024-03-06T13:21:26.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:26.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:26.381904+0000) 2024-03-06T13:21:27.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:27.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:27.382090+0000) 2024-03-06T13:21:28.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:28.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:28.382313+0000) 2024-03-06T13:21:29.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:29.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:29.382597+0000) 2024-03-06T13:21:30.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:30.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:30.382808+0000) 2024-03-06T13:21:31.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:31.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:31.382999+0000) 2024-03-06T13:21:32.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:32.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:32.383202+0000) 2024-03-06T13:21:33.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:33.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:33.383434+0000) 2024-03-06T13:21:34.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:34.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:34.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:34.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:34.383601+0000) 2024-03-06T13:21:35.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:35.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:35.383861+0000) 2024-03-06T13:21:36.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:36.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:36.384148+0000) 2024-03-06T13:21:37.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:37.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:37.384429+0000) 2024-03-06T13:21:38.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:38.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:38.384780+0000) 2024-03-06T13:21:39.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:39.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:39.385073+0000) 2024-03-06T13:21:40.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:40.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:40.385345+0000) 2024-03-06T13:21:41.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:41.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:41.385545+0000) 2024-03-06T13:21:42.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:42.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:42.385779+0000) 2024-03-06T13:21:43.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:43.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:43.386003+0000) 2024-03-06T13:21:44.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:44.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:44.386205+0000) 2024-03-06T13:21:45.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:45.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:45.386392+0000) 2024-03-06T13:21:46.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:46.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:46.386620+0000) 2024-03-06T13:21:47.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:47.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:47.386843+0000) 2024-03-06T13:21:48.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:48.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:48.387088+0000) 2024-03-06T13:21:49.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:49.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:49.387302+0000) 2024-03-06T13:21:50.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:50.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:50.387550+0000) 2024-03-06T13:21:51.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:51.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:51.387805+0000) 2024-03-06T13:21:52.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:52.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:52.388042+0000) 2024-03-06T13:21:53.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:53.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:53.388247+0000) 2024-03-06T13:21:54.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:54.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:54.388462+0000) 2024-03-06T13:21:55.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:55.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:55.388727+0000) 2024-03-06T13:21:56.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:56.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:56.388941+0000) 2024-03-06T13:21:57.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:57.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:57.389143+0000) 2024-03-06T13:21:58.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:58.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:58.389366+0000) 2024-03-06T13:21:59.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:21:59.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:59.389610+0000) 2024-03-06T13:22:00.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:00.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:00.389831+0000) 2024-03-06T13:22:01.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:01.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:01.390009+0000) 2024-03-06T13:22:02.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:02.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:02.390280+0000) 2024-03-06T13:22:03.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:03.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:03.390535+0000) 2024-03-06T13:22:04.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:04.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:04.390796+0000) 2024-03-06T13:22:04.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:04.491 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:05.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:05.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:05.391048+0000) 2024-03-06T13:22:06.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:06.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:06.391295+0000) 2024-03-06T13:22:07.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:07.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:07.391532+0000) 2024-03-06T13:22:08.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:08.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:08.391795+0000) 2024-03-06T13:22:09.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:09.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:09.392049+0000) 2024-03-06T13:22:10.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:10.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:10.392294+0000) 2024-03-06T13:22:11.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:11.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:11.392530+0000) 2024-03-06T13:22:12.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:12.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:12.392772+0000) 2024-03-06T13:22:13.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:13.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:13.393011+0000) 2024-03-06T13:22:14.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:14.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:14.393245+0000) 2024-03-06T13:22:15.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:15.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:15.393494+0000) 2024-03-06T13:22:16.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:16.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:16.393780+0000) 2024-03-06T13:22:17.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:17.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:17.394018+0000) 2024-03-06T13:22:18.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:18.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:18.394271+0000) 2024-03-06T13:22:19.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:19.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:19.394523+0000) 2024-03-06T13:22:20.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:20.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:20.394807+0000) 2024-03-06T13:22:21.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:21.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:21.395013+0000) 2024-03-06T13:22:22.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:22.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:22.395244+0000) 2024-03-06T13:22:23.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:23.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:23.395486+0000) 2024-03-06T13:22:24.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:24.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:24.395722+0000) 2024-03-06T13:22:25.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:25.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:25.396017+0000) 2024-03-06T13:22:26.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:26.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:26.396229+0000) 2024-03-06T13:22:27.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:27.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:27.396461+0000) 2024-03-06T13:22:28.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:28.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:28.396743+0000) 2024-03-06T13:22:29.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:29.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:29.397003+0000) 2024-03-06T13:22:30.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:30.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:30.397225+0000) 2024-03-06T13:22:31.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:31.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:31.397459+0000) 2024-03-06T13:22:32.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:32.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:32.397723+0000) 2024-03-06T13:22:33.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:33.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:33.397940+0000) 2024-03-06T13:22:34.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:34.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:34.398140+0000) 2024-03-06T13:22:34.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:34.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:35.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:35.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:35.398324+0000) 2024-03-06T13:22:36.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:36.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:36.398514+0000) 2024-03-06T13:22:37.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:37.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:37.398744+0000) 2024-03-06T13:22:38.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:38.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:38.398994+0000) 2024-03-06T13:22:39.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:39.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:39.399134+0000) 2024-03-06T13:22:40.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:40.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:40.399370+0000) 2024-03-06T13:22:41.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:41.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:41.399597+0000) 2024-03-06T13:22:42.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:42.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:42.399829+0000) 2024-03-06T13:22:43.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:43.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:43.400017+0000) 2024-03-06T13:22:44.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:44.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:44.400242+0000) 2024-03-06T13:22:45.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:45.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:45.400463+0000) 2024-03-06T13:22:46.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:46.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:46.400699+0000) 2024-03-06T13:22:47.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:47.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:47.400945+0000) 2024-03-06T13:22:48.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:48.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:48.401179+0000) 2024-03-06T13:22:49.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:49.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:49.401356+0000) 2024-03-06T13:22:50.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:50.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:50.401557+0000) 2024-03-06T13:22:51.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:51.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:51.401791+0000) 2024-03-06T13:22:52.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:52.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:52.402017+0000) 2024-03-06T13:22:53.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:53.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:53.402237+0000) 2024-03-06T13:22:54.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:54.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:54.402465+0000) 2024-03-06T13:22:55.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:55.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:55.402731+0000) 2024-03-06T13:22:56.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:56.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:56.402936+0000) 2024-03-06T13:22:57.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:57.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:57.403176+0000) 2024-03-06T13:22:58.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:58.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:58.403412+0000) 2024-03-06T13:22:59.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:22:59.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:59.403647+0000) 2024-03-06T13:23:00.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:00.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:00.403912+0000) 2024-03-06T13:23:01.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:01.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:01.404141+0000) 2024-03-06T13:23:02.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:02.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:02.404374+0000) 2024-03-06T13:23:03.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:03.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:03.404602+0000) 2024-03-06T13:23:04.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:04.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:04.404826+0000) 2024-03-06T13:23:04.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:04.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:05.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:05.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:05.405014+0000) 2024-03-06T13:23:06.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:06.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:06.405209+0000) 2024-03-06T13:23:07.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:07.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:07.405432+0000) 2024-03-06T13:23:08.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:08.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:08.405697+0000) 2024-03-06T13:23:09.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:09.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:09.405915+0000) 2024-03-06T13:23:10.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:10.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:10.406165+0000) 2024-03-06T13:23:11.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:11.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:11.406447+0000) 2024-03-06T13:23:12.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:12.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:12.406655+0000) 2024-03-06T13:23:13.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:13.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:13.406992+0000) 2024-03-06T13:23:14.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:14.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:14.407234+0000) 2024-03-06T13:23:15.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:15.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:15.407513+0000) 2024-03-06T13:23:16.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:16.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:16.407821+0000) 2024-03-06T13:23:17.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:17.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:17.408038+0000) 2024-03-06T13:23:18.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:18.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:18.408305+0000) 2024-03-06T13:23:19.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:19.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:19.408536+0000) 2024-03-06T13:23:20.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:20.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:20.408817+0000) 2024-03-06T13:23:21.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:21.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:21.409030+0000) 2024-03-06T13:23:22.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:22.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:22.409232+0000) 2024-03-06T13:23:23.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:23.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:23.409446+0000) 2024-03-06T13:23:24.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:24.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:24.409600+0000) 2024-03-06T13:23:25.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:25.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:25.409806+0000) 2024-03-06T13:23:26.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:26.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:26.409985+0000) 2024-03-06T13:23:27.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:27.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:27.410212+0000) 2024-03-06T13:23:28.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:28.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:28.410476+0000) 2024-03-06T13:23:29.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:29.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:29.410757+0000) 2024-03-06T13:23:30.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:30.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:30.411008+0000) 2024-03-06T13:23:31.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:31.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:31.411251+0000) 2024-03-06T13:23:32.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:32.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:32.411485+0000) 2024-03-06T13:23:33.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:33.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:33.411759+0000) 2024-03-06T13:23:34.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:34.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:34.411964+0000) 2024-03-06T13:23:35.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:35.093 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:35.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:35.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:35.412158+0000) 2024-03-06T13:23:36.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:36.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:36.412413+0000) 2024-03-06T13:23:37.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:37.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:37.412703+0000) 2024-03-06T13:23:38.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:38.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:38.412974+0000) 2024-03-06T13:23:39.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:39.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:39.413214+0000) 2024-03-06T13:23:40.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:40.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:40.413465+0000) 2024-03-06T13:23:41.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:41.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:41.413729+0000) 2024-03-06T13:23:42.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:42.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:42.414004+0000) 2024-03-06T13:23:43.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:43.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:43.414245+0000) 2024-03-06T13:23:44.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:44.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:44.414430+0000) 2024-03-06T13:23:45.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:45.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:45.414613+0000) 2024-03-06T13:23:46.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:46.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:46.414865+0000) 2024-03-06T13:23:47.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:47.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:47.415100+0000) 2024-03-06T13:23:48.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:48.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:48.415362+0000) 2024-03-06T13:23:49.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:49.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:49.415608+0000) 2024-03-06T13:23:50.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:50.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:50.415855+0000) 2024-03-06T13:23:51.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:51.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:51.416091+0000) 2024-03-06T13:23:52.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:52.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:52.416332+0000) 2024-03-06T13:23:53.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:53.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:53.416558+0000) 2024-03-06T13:23:54.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:54.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:54.416791+0000) 2024-03-06T13:23:55.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:55.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:55.416982+0000) 2024-03-06T13:23:56.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:56.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:56.417221+0000) 2024-03-06T13:23:57.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:57.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:57.417441+0000) 2024-03-06T13:23:58.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:58.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:58.417738+0000) 2024-03-06T13:23:59.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:23:59.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:59.417971+0000) 2024-03-06T13:24:00.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:00.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:00.418159+0000) 2024-03-06T13:24:01.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:01.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:01.418395+0000) 2024-03-06T13:24:02.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:02.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:02.418626+0000) 2024-03-06T13:24:03.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:03.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:03.418826+0000) 2024-03-06T13:24:04.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:04.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:04.419056+0000) 2024-03-06T13:24:05.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:05.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:05.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:05.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:05.419271+0000) 2024-03-06T13:24:06.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:06.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:06.419462+0000) 2024-03-06T13:24:07.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:07.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:07.419775+0000) 2024-03-06T13:24:08.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:08.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:08.419996+0000) 2024-03-06T13:24:09.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:09.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:09.420195+0000) 2024-03-06T13:24:10.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:10.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:10.420429+0000) 2024-03-06T13:24:11.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:11.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:11.420742+0000) 2024-03-06T13:24:12.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:12.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:12.421021+0000) 2024-03-06T13:24:13.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:13.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:13.421219+0000) 2024-03-06T13:24:14.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:14.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:14.421476+0000) 2024-03-06T13:24:15.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:15.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:15.421761+0000) 2024-03-06T13:24:16.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:16.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:16.421977+0000) 2024-03-06T13:24:17.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:17.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:17.422221+0000) 2024-03-06T13:24:18.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:18.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:18.422439+0000) 2024-03-06T13:24:19.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:19.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:19.422703+0000) 2024-03-06T13:24:20.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:20.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:20.422894+0000) 2024-03-06T13:24:21.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:21.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:21.423101+0000) 2024-03-06T13:24:22.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:22.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:22.423342+0000) 2024-03-06T13:24:23.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:23.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:23.423569+0000) 2024-03-06T13:24:24.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:24.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:24.423801+0000) 2024-03-06T13:24:25.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:25.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:25.424024+0000) 2024-03-06T13:24:26.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:26.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:26.424266+0000) 2024-03-06T13:24:27.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:27.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:27.424491+0000) 2024-03-06T13:24:28.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:28.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:28.424769+0000) 2024-03-06T13:24:29.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:29.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:29.424994+0000) 2024-03-06T13:24:30.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:30.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:30.425223+0000) 2024-03-06T13:24:31.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:31.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:31.425437+0000) 2024-03-06T13:24:32.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:32.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:32.425613+0000) 2024-03-06T13:24:33.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:33.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:33.425817+0000) 2024-03-06T13:24:34.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:34.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:34.426053+0000) 2024-03-06T13:24:35.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:35.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:35.426283+0000) 2024-03-06T13:24:35.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:35.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:36.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:36.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:36.426485+0000) 2024-03-06T13:24:37.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:37.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:37.426760+0000) 2024-03-06T13:24:38.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:38.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:38.426949+0000) 2024-03-06T13:24:39.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:39.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:39.427133+0000) 2024-03-06T13:24:40.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:40.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:40.427372+0000) 2024-03-06T13:24:41.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:41.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:41.427597+0000) 2024-03-06T13:24:42.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:42.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:42.427789+0000) 2024-03-06T13:24:43.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:43.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:43.427994+0000) 2024-03-06T13:24:44.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:44.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:44.428232+0000) 2024-03-06T13:24:45.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:45.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:45.428459+0000) 2024-03-06T13:24:46.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:46.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:46.428717+0000) 2024-03-06T13:24:47.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:47.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:47.428950+0000) 2024-03-06T13:24:48.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:48.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:48.429146+0000) 2024-03-06T13:24:49.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:49.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:49.429381+0000) 2024-03-06T13:24:50.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:50.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:50.429617+0000) 2024-03-06T13:24:51.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:51.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:51.429848+0000) 2024-03-06T13:24:52.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:52.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:52.430066+0000) 2024-03-06T13:24:53.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:53.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:53.430288+0000) 2024-03-06T13:24:54.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:54.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:54.430521+0000) 2024-03-06T13:24:55.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:55.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:55.430883+0000) 2024-03-06T13:24:56.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:56.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:56.431141+0000) 2024-03-06T13:24:57.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:57.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:57.431341+0000) 2024-03-06T13:24:58.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:58.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:58.431582+0000) 2024-03-06T13:24:59.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:24:59.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:59.431810+0000) 2024-03-06T13:25:00.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:00.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:00.432039+0000) 2024-03-06T13:25:01.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:01.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:01.432265+0000) 2024-03-06T13:25:02.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:02.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:02.432800+0000) 2024-03-06T13:25:03.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:03.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:03.432977+0000) 2024-03-06T13:25:04.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:04.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:04.433216+0000) 2024-03-06T13:25:05.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:05.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:05.433452+0000) 2024-03-06T13:25:05.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:05.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:06.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:06.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:06.433725+0000) 2024-03-06T13:25:07.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:07.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:07.433892+0000) 2024-03-06T13:25:08.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:08.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:08.434109+0000) 2024-03-06T13:25:09.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:09.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:09.434255+0000) 2024-03-06T13:25:10.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:10.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:10.434498+0000) 2024-03-06T13:25:11.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:11.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:11.434781+0000) 2024-03-06T13:25:12.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:12.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:12.434982+0000) 2024-03-06T13:25:13.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:13.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:13.435226+0000) 2024-03-06T13:25:14.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:14.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:14.435376+0000) 2024-03-06T13:25:15.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:15.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:15.435535+0000) 2024-03-06T13:25:16.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:16.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:16.435780+0000) 2024-03-06T13:25:17.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:17.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:17.435989+0000) 2024-03-06T13:25:18.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:18.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:18.436110+0000) 2024-03-06T13:25:19.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:19.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:19.436306+0000) 2024-03-06T13:25:20.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:20.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:20.436523+0000) 2024-03-06T13:25:21.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:21.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:21.436769+0000) 2024-03-06T13:25:22.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:22.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:22.436979+0000) 2024-03-06T13:25:23.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:23.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:23.437173+0000) 2024-03-06T13:25:24.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:24.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:24.437362+0000) 2024-03-06T13:25:25.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:25.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:25.437607+0000) 2024-03-06T13:25:26.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:26.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:26.437836+0000) 2024-03-06T13:25:27.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:27.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:27.438067+0000) 2024-03-06T13:25:28.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:28.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:28.438309+0000) 2024-03-06T13:25:29.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:29.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:29.438508+0000) 2024-03-06T13:25:30.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:30.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:30.438792+0000) 2024-03-06T13:25:31.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:31.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:31.439016+0000) 2024-03-06T13:25:32.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:32.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:32.439240+0000) 2024-03-06T13:25:33.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:33.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:33.439467+0000) 2024-03-06T13:25:34.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:34.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:34.439733+0000) 2024-03-06T13:25:35.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:35.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:35.439935+0000) 2024-03-06T13:25:35.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:35.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:36.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:36.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:36.440166+0000) 2024-03-06T13:25:37.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:37.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:37.440396+0000) 2024-03-06T13:25:38.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:38.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:38.440639+0000) 2024-03-06T13:25:39.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:39.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:39.440896+0000) 2024-03-06T13:25:40.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:40.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:40.441139+0000) 2024-03-06T13:25:41.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:41.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:41.441296+0000) 2024-03-06T13:25:42.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:42.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:42.441534+0000) 2024-03-06T13:25:43.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:43.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:43.441821+0000) 2024-03-06T13:25:44.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:44.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:44.441984+0000) 2024-03-06T13:25:45.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:45.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:45.442154+0000) 2024-03-06T13:25:46.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:46.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:46.442346+0000) 2024-03-06T13:25:47.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:47.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:47.442572+0000) 2024-03-06T13:25:48.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:48.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:48.442816+0000) 2024-03-06T13:25:49.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:49.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:49.442935+0000) 2024-03-06T13:25:50.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:50.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:50.443128+0000) 2024-03-06T13:25:51.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:51.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:51.443363+0000) 2024-03-06T13:25:52.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:52.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:52.443546+0000) 2024-03-06T13:25:53.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:53.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:53.443773+0000) 2024-03-06T13:25:54.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:54.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:54.444012+0000) 2024-03-06T13:25:55.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:55.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:55.444213+0000) 2024-03-06T13:25:56.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:56.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:56.444445+0000) 2024-03-06T13:25:57.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:57.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:57.444639+0000) 2024-03-06T13:25:58.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:58.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:58.444863+0000) 2024-03-06T13:25:59.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:25:59.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:59.445042+0000) 2024-03-06T13:26:00.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:00.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:00.445274+0000) 2024-03-06T13:26:01.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:01.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:01.445499+0000) 2024-03-06T13:26:02.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:02.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:02.445756+0000) 2024-03-06T13:26:03.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:03.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:03.446004+0000) 2024-03-06T13:26:04.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:04.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:04.446211+0000) 2024-03-06T13:26:05.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:05.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:05.446462+0000) 2024-03-06T13:26:06.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:06.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:06.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:06.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:06.446646+0000) 2024-03-06T13:26:07.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:07.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:07.446923+0000) 2024-03-06T13:26:08.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:08.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:08.447166+0000) 2024-03-06T13:26:09.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:09.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:09.447348+0000) 2024-03-06T13:26:10.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:10.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:10.447584+0000) 2024-03-06T13:26:11.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:11.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:11.447821+0000) 2024-03-06T13:26:12.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:12.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:12.448046+0000) 2024-03-06T13:26:13.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:13.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:13.448269+0000) 2024-03-06T13:26:14.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:14.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:14.448502+0000) 2024-03-06T13:26:15.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:15.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:15.448783+0000) 2024-03-06T13:26:16.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:16.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:16.449017+0000) 2024-03-06T13:26:17.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:17.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:17.449244+0000) 2024-03-06T13:26:18.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:18.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:18.449484+0000) 2024-03-06T13:26:19.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:19.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:19.449703+0000) 2024-03-06T13:26:20.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:20.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:20.449952+0000) 2024-03-06T13:26:21.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:21.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:21.450148+0000) 2024-03-06T13:26:22.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:22.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:22.450379+0000) 2024-03-06T13:26:23.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:23.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:23.450532+0000) 2024-03-06T13:26:24.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:24.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:24.450774+0000) 2024-03-06T13:26:25.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:25.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:25.450988+0000) 2024-03-06T13:26:26.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:26.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:26.451206+0000) 2024-03-06T13:26:27.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:27.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:27.451485+0000) 2024-03-06T13:26:28.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:28.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:28.451783+0000) 2024-03-06T13:26:29.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:29.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:29.452002+0000) 2024-03-06T13:26:30.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:30.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:30.452290+0000) 2024-03-06T13:26:31.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:31.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:31.452525+0000) 2024-03-06T13:26:32.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:32.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:32.452824+0000) 2024-03-06T13:26:33.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:33.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:33.453101+0000) 2024-03-06T13:26:34.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:34.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:34.453386+0000) 2024-03-06T13:26:35.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:35.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:35.453713+0000) 2024-03-06T13:26:36.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:36.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:36.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:36.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:36.454051+0000) 2024-03-06T13:26:37.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:37.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:37.454334+0000) 2024-03-06T13:26:38.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:38.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:38.454627+0000) 2024-03-06T13:26:39.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:39.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:39.454887+0000) 2024-03-06T13:26:40.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:40.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:40.455167+0000) 2024-03-06T13:26:41.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:41.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:41.455451+0000) 2024-03-06T13:26:42.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:42.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:42.455617+0000) 2024-03-06T13:26:43.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:43.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:43.455854+0000) 2024-03-06T13:26:44.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:44.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:44.456090+0000) 2024-03-06T13:26:45.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:45.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:45.456330+0000) 2024-03-06T13:26:46.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:46.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:46.456568+0000) 2024-03-06T13:26:47.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:47.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:47.456828+0000) 2024-03-06T13:26:48.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:48.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:48.457035+0000) 2024-03-06T13:26:49.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:49.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:49.457193+0000) 2024-03-06T13:26:50.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:50.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:50.457435+0000) 2024-03-06T13:26:51.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:51.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:51.457694+0000) 2024-03-06T13:26:52.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:52.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:52.457890+0000) 2024-03-06T13:26:53.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:53.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:53.458041+0000) 2024-03-06T13:26:54.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:54.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:54.458245+0000) 2024-03-06T13:26:55.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:55.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:55.458418+0000) 2024-03-06T13:26:56.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:56.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:56.458630+0000) 2024-03-06T13:26:57.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:57.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:57.458917+0000) 2024-03-06T13:26:58.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:58.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:58.459121+0000) 2024-03-06T13:26:59.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:26:59.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:59.459368+0000) 2024-03-06T13:27:00.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:00.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:00.459611+0000) 2024-03-06T13:27:01.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:01.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:01.459844+0000) 2024-03-06T13:27:02.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:02.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:02.460054+0000) 2024-03-06T13:27:03.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:03.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:03.460306+0000) 2024-03-06T13:27:04.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:04.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:04.460534+0000) 2024-03-06T13:27:05.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:05.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:05.460801+0000) 2024-03-06T13:27:06.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:06.441 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:06.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:06.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:06.461026+0000) 2024-03-06T13:27:07.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:07.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:07.461220+0000) 2024-03-06T13:27:08.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:08.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:08.461460+0000) 2024-03-06T13:27:09.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:09.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:09.461656+0000) 2024-03-06T13:27:10.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:10.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:10.461930+0000) 2024-03-06T13:27:11.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:11.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:11.462169+0000) 2024-03-06T13:27:12.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:12.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:12.462371+0000) 2024-03-06T13:27:13.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:13.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:13.462611+0000) 2024-03-06T13:27:14.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:14.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:14.462846+0000) 2024-03-06T13:27:15.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:15.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:15.462986+0000) 2024-03-06T13:27:16.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:16.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:16.463222+0000) 2024-03-06T13:27:17.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:17.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:17.463441+0000) 2024-03-06T13:27:18.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:18.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:18.463711+0000) 2024-03-06T13:27:19.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:19.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:19.463944+0000) 2024-03-06T13:27:20.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:20.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:20.464177+0000) 2024-03-06T13:27:21.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:21.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:21.464406+0000) 2024-03-06T13:27:22.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:22.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:22.464635+0000) 2024-03-06T13:27:23.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:23.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:23.464859+0000) 2024-03-06T13:27:24.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:24.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:24.465113+0000) 2024-03-06T13:27:25.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:25.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:25.465350+0000) 2024-03-06T13:27:26.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:26.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:26.465543+0000) 2024-03-06T13:27:27.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:27.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:27.465765+0000) 2024-03-06T13:27:28.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:28.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:28.465968+0000) 2024-03-06T13:27:29.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:29.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:29.466196+0000) 2024-03-06T13:27:30.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:30.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:30.466441+0000) 2024-03-06T13:27:31.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:31.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:31.466637+0000) 2024-03-06T13:27:32.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:32.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:32.466920+0000) 2024-03-06T13:27:33.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:33.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:33.467130+0000) 2024-03-06T13:27:34.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:34.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:34.467358+0000) 2024-03-06T13:27:35.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:35.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:35.467538+0000) 2024-03-06T13:27:36.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:36.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:36.467806+0000) 2024-03-06T13:27:36.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:36.578 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:37.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:37.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:37.468041+0000) 2024-03-06T13:27:38.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:38.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:38.468284+0000) 2024-03-06T13:27:39.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:39.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:39.468512+0000) 2024-03-06T13:27:40.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:40.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:40.468783+0000) 2024-03-06T13:27:41.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:41.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:41.469006+0000) 2024-03-06T13:27:42.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:42.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:42.469229+0000) 2024-03-06T13:27:43.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:43.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:43.469455+0000) 2024-03-06T13:27:44.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:44.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:44.469709+0000) 2024-03-06T13:27:45.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:45.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:45.469912+0000) 2024-03-06T13:27:46.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:46.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:46.470150+0000) 2024-03-06T13:27:47.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:47.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:47.470406+0000) 2024-03-06T13:27:48.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:48.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:48.470646+0000) 2024-03-06T13:27:49.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:49.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:49.470879+0000) 2024-03-06T13:27:50.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:50.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:50.471125+0000) 2024-03-06T13:27:51.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:51.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:51.471350+0000) 2024-03-06T13:27:52.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:52.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:52.471570+0000) 2024-03-06T13:27:53.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:53.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:53.471799+0000) 2024-03-06T13:27:54.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:54.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:54.472034+0000) 2024-03-06T13:27:55.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:55.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:55.472272+0000) 2024-03-06T13:27:56.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:56.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:56.472501+0000) 2024-03-06T13:27:57.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:57.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:57.472768+0000) 2024-03-06T13:27:58.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:58.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:58.473002+0000) 2024-03-06T13:27:59.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:27:59.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:59.473226+0000) 2024-03-06T13:28:00.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:00.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:00.473461+0000) 2024-03-06T13:28:01.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:01.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:01.473723+0000) 2024-03-06T13:28:02.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:02.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:02.473952+0000) 2024-03-06T13:28:03.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:03.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:03.474221+0000) 2024-03-06T13:28:04.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:04.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:04.474482+0000) 2024-03-06T13:28:05.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:05.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:05.474794+0000) 2024-03-06T13:28:06.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:06.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:06.474982+0000) 2024-03-06T13:28:06.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:06.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:07.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:07.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:07.475227+0000) 2024-03-06T13:28:08.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:08.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:08.475469+0000) 2024-03-06T13:28:09.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:09.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:09.475736+0000) 2024-03-06T13:28:10.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:10.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:10.475893+0000) 2024-03-06T13:28:11.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:11.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:11.476169+0000) 2024-03-06T13:28:12.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:12.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:12.476339+0000) 2024-03-06T13:28:13.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:13.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:13.476500+0000) 2024-03-06T13:28:14.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:14.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:14.476699+0000) 2024-03-06T13:28:15.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:15.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:15.476943+0000) 2024-03-06T13:28:16.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:16.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:16.477185+0000) 2024-03-06T13:28:17.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:17.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:17.477449+0000) 2024-03-06T13:28:18.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:18.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:18.477701+0000) 2024-03-06T13:28:19.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:19.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:19.477949+0000) 2024-03-06T13:28:20.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:20.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:20.478182+0000) 2024-03-06T13:28:21.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:21.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:21.478415+0000) 2024-03-06T13:28:22.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:22.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:22.478619+0000) 2024-03-06T13:28:23.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:23.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:23.478872+0000) 2024-03-06T13:28:24.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:24.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:24.479087+0000) 2024-03-06T13:28:25.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:25.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:25.479248+0000) 2024-03-06T13:28:26.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:26.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:26.479490+0000) 2024-03-06T13:28:27.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:27.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:27.479640+0000) 2024-03-06T13:28:28.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:28.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:28.479949+0000) 2024-03-06T13:28:29.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:29.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:29.480190+0000) 2024-03-06T13:28:30.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:30.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:30.480418+0000) 2024-03-06T13:28:31.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:31.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:31.480593+0000) 2024-03-06T13:28:32.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:32.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:32.480832+0000) 2024-03-06T13:28:33.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:33.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:33.481006+0000) 2024-03-06T13:28:34.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:34.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:34.481251+0000) 2024-03-06T13:28:35.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:35.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:35.481473+0000) 2024-03-06T13:28:36.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:36.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:36.481623+0000) 2024-03-06T13:28:37.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:37.026 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:37.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:37.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:37.481874+0000) 2024-03-06T13:28:38.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:38.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:38.482113+0000) 2024-03-06T13:28:39.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:39.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:39.482300+0000) 2024-03-06T13:28:40.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:40.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:40.482478+0000) 2024-03-06T13:28:41.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:41.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:41.482748+0000) 2024-03-06T13:28:42.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:42.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:42.482971+0000) 2024-03-06T13:28:43.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:43.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:43.483192+0000) 2024-03-06T13:28:44.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:44.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:44.483385+0000) 2024-03-06T13:28:45.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:45.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:45.483626+0000) 2024-03-06T13:28:46.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:46.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:46.483855+0000) 2024-03-06T13:28:47.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:47.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:47.484079+0000) 2024-03-06T13:28:48.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:48.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:48.484243+0000) 2024-03-06T13:28:49.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:49.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:49.484436+0000) 2024-03-06T13:28:50.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:50.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:50.484696+0000) 2024-03-06T13:28:51.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:51.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:51.484923+0000) 2024-03-06T13:28:52.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:52.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:52.485144+0000) 2024-03-06T13:28:53.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:53.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:53.485369+0000) 2024-03-06T13:28:54.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:54.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:54.485589+0000) 2024-03-06T13:28:55.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:55.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:55.485818+0000) 2024-03-06T13:28:56.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:56.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:56.486044+0000) 2024-03-06T13:28:57.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:57.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:57.486279+0000) 2024-03-06T13:28:58.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:58.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:58.486478+0000) 2024-03-06T13:28:59.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:28:59.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:59.486712+0000) 2024-03-06T13:29:00.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:00.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:00.486895+0000) 2024-03-06T13:29:01.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:01.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:01.487130+0000) 2024-03-06T13:29:02.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:02.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:02.487359+0000) 2024-03-06T13:29:03.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:03.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:03.487582+0000) 2024-03-06T13:29:04.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:04.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:04.487786+0000) 2024-03-06T13:29:05.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:05.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:05.488010+0000) 2024-03-06T13:29:06.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:06.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:06.488239+0000) 2024-03-06T13:29:07.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:07.244 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:07.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:07.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:07.488482+0000) 2024-03-06T13:29:08.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:08.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:08.488789+0000) 2024-03-06T13:29:09.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:09.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:09.489021+0000) 2024-03-06T13:29:10.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:10.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:10.489241+0000) 2024-03-06T13:29:11.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:11.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:11.489470+0000) 2024-03-06T13:29:12.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:12.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:12.489622+0000) 2024-03-06T13:29:13.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:13.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:13.489831+0000) 2024-03-06T13:29:14.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:14.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:14.490047+0000) 2024-03-06T13:29:15.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:15.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:15.490241+0000) 2024-03-06T13:29:16.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:16.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:16.490418+0000) 2024-03-06T13:29:17.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:17.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:17.490655+0000) 2024-03-06T13:29:18.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:18.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:18.490934+0000) 2024-03-06T13:29:19.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:19.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:19.491161+0000) 2024-03-06T13:29:20.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:20.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:20.491385+0000) 2024-03-06T13:29:21.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:21.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:21.491617+0000) 2024-03-06T13:29:22.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:22.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:22.491792+0000) 2024-03-06T13:29:23.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:23.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:23.492017+0000) 2024-03-06T13:29:24.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:24.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:24.492239+0000) 2024-03-06T13:29:25.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:25.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:25.492457+0000) 2024-03-06T13:29:26.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:26.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:26.492610+0000) 2024-03-06T13:29:27.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:27.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:27.492855+0000) 2024-03-06T13:29:28.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:28.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:28.493094+0000) 2024-03-06T13:29:29.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:29.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:29.493293+0000) 2024-03-06T13:29:30.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:30.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:30.493444+0000) 2024-03-06T13:29:31.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:31.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:31.493726+0000) 2024-03-06T13:29:32.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:32.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:32.493984+0000) 2024-03-06T13:29:33.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:33.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:33.494216+0000) 2024-03-06T13:29:34.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:34.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:34.494377+0000) 2024-03-06T13:29:35.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:35.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:35.494620+0000) 2024-03-06T13:29:36.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:36.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:36.494861+0000) 2024-03-06T13:29:37.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:37.473 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:37.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:37.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:37.495078+0000) 2024-03-06T13:29:38.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:38.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:38.495246+0000) 2024-03-06T13:29:39.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:39.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:39.495478+0000) 2024-03-06T13:29:40.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:40.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:40.495741+0000) 2024-03-06T13:29:41.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:41.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:41.495977+0000) 2024-03-06T13:29:42.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:42.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:42.496207+0000) 2024-03-06T13:29:43.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:43.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:43.496438+0000) 2024-03-06T13:29:44.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:44.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:44.496585+0000) 2024-03-06T13:29:45.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:45.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:45.496824+0000) 2024-03-06T13:29:46.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:46.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:46.497055+0000) 2024-03-06T13:29:47.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:47.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:47.497284+0000) 2024-03-06T13:29:48.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:48.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:48.497555+0000) 2024-03-06T13:29:49.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:49.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:49.497796+0000) 2024-03-06T13:29:50.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:50.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:50.498017+0000) 2024-03-06T13:29:51.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:51.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:51.498216+0000) 2024-03-06T13:29:52.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:52.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:52.498454+0000) 2024-03-06T13:29:53.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:53.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:53.498786+0000) 2024-03-06T13:29:54.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:54.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:54.499032+0000) 2024-03-06T13:29:55.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:55.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:55.499269+0000) 2024-03-06T13:29:56.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:56.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:56.499519+0000) 2024-03-06T13:29:57.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:57.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:57.499793+0000) 2024-03-06T13:29:58.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:58.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:58.500034+0000) 2024-03-06T13:29:59.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:29:59.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:59.500277+0000) 2024-03-06T13:30:00.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:00.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:00.500509+0000) 2024-03-06T13:30:01.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:01.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:01.500784+0000) 2024-03-06T13:30:02.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:02.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:02.501035+0000) 2024-03-06T13:30:03.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:03.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:03.501254+0000) 2024-03-06T13:30:04.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:04.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:04.501469+0000) 2024-03-06T13:30:05.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:05.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:05.501731+0000) 2024-03-06T13:30:06.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:06.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:06.501967+0000) 2024-03-06T13:30:07.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:07.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:07.502078+0000) 2024-03-06T13:30:07.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:07.613 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:08.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:08.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:08.502322+0000) 2024-03-06T13:30:09.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:09.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:09.502550+0000) 2024-03-06T13:30:10.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:10.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:10.502786+0000) 2024-03-06T13:30:11.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:11.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:11.503016+0000) 2024-03-06T13:30:12.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:12.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:12.503199+0000) 2024-03-06T13:30:13.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:13.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:13.503442+0000) 2024-03-06T13:30:14.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:14.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:14.503650+0000) 2024-03-06T13:30:15.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:15.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:15.503951+0000) 2024-03-06T13:30:16.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:16.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:16.504195+0000) 2024-03-06T13:30:17.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:17.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:17.504350+0000) 2024-03-06T13:30:18.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:18.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:18.504594+0000) 2024-03-06T13:30:19.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:19.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:19.504811+0000) 2024-03-06T13:30:20.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:20.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:20.505045+0000) 2024-03-06T13:30:21.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:21.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:21.505274+0000) 2024-03-06T13:30:22.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:22.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:22.505495+0000) 2024-03-06T13:30:23.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:23.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:23.505788+0000) 2024-03-06T13:30:24.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:24.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:24.506007+0000) 2024-03-06T13:30:25.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:25.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:25.506240+0000) 2024-03-06T13:30:26.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:26.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:26.506445+0000) 2024-03-06T13:30:27.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:27.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:27.506650+0000) 2024-03-06T13:30:28.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:28.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:28.506943+0000) 2024-03-06T13:30:29.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:29.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:29.507164+0000) 2024-03-06T13:30:30.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:30.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:30.507387+0000) 2024-03-06T13:30:31.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:31.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:31.507619+0000) 2024-03-06T13:30:32.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:32.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:32.507839+0000) 2024-03-06T13:30:33.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:33.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:33.507979+0000) 2024-03-06T13:30:34.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:34.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:34.508216+0000) 2024-03-06T13:30:35.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:35.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:35.508454+0000) 2024-03-06T13:30:36.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:36.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:36.508739+0000) 2024-03-06T13:30:37.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:37.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:37.508974+0000) 2024-03-06T13:30:37.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:37.831 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:38.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:38.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:38.509218+0000) 2024-03-06T13:30:39.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:39.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:39.509415+0000) 2024-03-06T13:30:40.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:40.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:40.509605+0000) 2024-03-06T13:30:41.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:41.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:41.509845+0000) 2024-03-06T13:30:42.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:42.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:42.510040+0000) 2024-03-06T13:30:43.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:43.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:43.510282+0000) 2024-03-06T13:30:44.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:44.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:44.510511+0000) 2024-03-06T13:30:45.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:45.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:45.510628+0000) 2024-03-06T13:30:46.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:46.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:46.510857+0000) 2024-03-06T13:30:47.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:47.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:47.511092+0000) 2024-03-06T13:30:48.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:48.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:48.511346+0000) 2024-03-06T13:30:49.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:49.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:49.511575+0000) 2024-03-06T13:30:50.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:50.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:50.511779+0000) 2024-03-06T13:30:51.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:51.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:51.512023+0000) 2024-03-06T13:30:52.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:52.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:52.512149+0000) 2024-03-06T13:30:53.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:53.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:53.512370+0000) 2024-03-06T13:30:54.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:54.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:54.512564+0000) 2024-03-06T13:30:55.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:55.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:55.512820+0000) 2024-03-06T13:30:56.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:56.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:56.513053+0000) 2024-03-06T13:30:57.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:57.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:57.513285+0000) 2024-03-06T13:30:58.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:58.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:58.513501+0000) 2024-03-06T13:30:59.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:30:59.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:59.513778+0000) 2024-03-06T13:31:00.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:00.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:00.514005+0000) 2024-03-06T13:31:01.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:01.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:01.514238+0000) 2024-03-06T13:31:02.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:02.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:02.514460+0000) 2024-03-06T13:31:03.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:03.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:03.514726+0000) 2024-03-06T13:31:04.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:04.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:04.514882+0000) 2024-03-06T13:31:05.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:05.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:05.515121+0000) 2024-03-06T13:31:06.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:06.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:06.515312+0000) 2024-03-06T13:31:07.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:07.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:07.515499+0000) 2024-03-06T13:31:08.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:08.050 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:08.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:08.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:08.515800+0000) 2024-03-06T13:31:09.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:09.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:09.516028+0000) 2024-03-06T13:31:10.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:10.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:10.516249+0000) 2024-03-06T13:31:11.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:11.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:11.516484+0000) 2024-03-06T13:31:12.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:12.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:12.516635+0000) 2024-03-06T13:31:13.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:13.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:13.516887+0000) 2024-03-06T13:31:14.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:14.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:14.517090+0000) 2024-03-06T13:31:15.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:15.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:15.517312+0000) 2024-03-06T13:31:16.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:16.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:16.517552+0000) 2024-03-06T13:31:17.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:17.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:17.517776+0000) 2024-03-06T13:31:18.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:18.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:18.518009+0000) 2024-03-06T13:31:19.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:19.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:19.518240+0000) 2024-03-06T13:31:20.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:20.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:20.518458+0000) 2024-03-06T13:31:21.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:21.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:21.518620+0000) 2024-03-06T13:31:22.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:22.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:22.518870+0000) 2024-03-06T13:31:23.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:23.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:23.519097+0000) 2024-03-06T13:31:24.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:24.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:24.519338+0000) 2024-03-06T13:31:25.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:25.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:25.519587+0000) 2024-03-06T13:31:26.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:26.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:26.519836+0000) 2024-03-06T13:31:27.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:27.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:27.520077+0000) 2024-03-06T13:31:28.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:28.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:28.520336+0000) 2024-03-06T13:31:29.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:29.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:29.520503+0000) 2024-03-06T13:31:30.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:30.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:30.520737+0000) 2024-03-06T13:31:31.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:31.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:31.520913+0000) 2024-03-06T13:31:32.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:32.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:32.521144+0000) 2024-03-06T13:31:33.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:33.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:33.521367+0000) 2024-03-06T13:31:34.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:34.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:34.521611+0000) 2024-03-06T13:31:35.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:35.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:35.521889+0000) 2024-03-06T13:31:36.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:36.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:36.522126+0000) 2024-03-06T13:31:37.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:37.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:37.522324+0000) 2024-03-06T13:31:38.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:38.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:38.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:38.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:38.522502+0000) 2024-03-06T13:31:39.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:39.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:39.522660+0000) 2024-03-06T13:31:40.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:40.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:40.522881+0000) 2024-03-06T13:31:41.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:41.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:41.523135+0000) 2024-03-06T13:31:42.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:42.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:42.523359+0000) 2024-03-06T13:31:43.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:43.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:43.523594+0000) 2024-03-06T13:31:44.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:44.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:44.523835+0000) 2024-03-06T13:31:45.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:45.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:45.524067+0000) 2024-03-06T13:31:46.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:46.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:46.524307+0000) 2024-03-06T13:31:47.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:47.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:47.524539+0000) 2024-03-06T13:31:48.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:48.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:48.524777+0000) 2024-03-06T13:31:49.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:49.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:49.525004+0000) 2024-03-06T13:31:50.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:50.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:50.525226+0000) 2024-03-06T13:31:51.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:51.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:51.525461+0000) 2024-03-06T13:31:52.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:52.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:52.525707+0000) 2024-03-06T13:31:53.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:53.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:53.525940+0000) 2024-03-06T13:31:54.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:54.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:54.526171+0000) 2024-03-06T13:31:55.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:55.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:55.526398+0000) 2024-03-06T13:31:56.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:56.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:56.526629+0000) 2024-03-06T13:31:57.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:57.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:57.526858+0000) 2024-03-06T13:31:58.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:58.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:58.527101+0000) 2024-03-06T13:31:59.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:31:59.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:59.527329+0000) 2024-03-06T13:32:00.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:00.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:00.527550+0000) 2024-03-06T13:32:01.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:01.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:01.527796+0000) 2024-03-06T13:32:02.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:02.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:02.528029+0000) 2024-03-06T13:32:03.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:03.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:03.528224+0000) 2024-03-06T13:32:04.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:04.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:04.528433+0000) 2024-03-06T13:32:05.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:05.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:05.528646+0000) 2024-03-06T13:32:06.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:06.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:06.528889+0000) 2024-03-06T13:32:07.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:07.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:07.529075+0000) 2024-03-06T13:32:08.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:08.410 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:08.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:08.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:08.529289+0000) 2024-03-06T13:32:09.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:09.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:09.529526+0000) 2024-03-06T13:32:10.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:10.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:10.529703+0000) 2024-03-06T13:32:11.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:11.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:11.529949+0000) 2024-03-06T13:32:12.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:12.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:12.530139+0000) 2024-03-06T13:32:13.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:13.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:13.530374+0000) 2024-03-06T13:32:14.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:14.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:14.530599+0000) 2024-03-06T13:32:15.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:15.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:15.530827+0000) 2024-03-06T13:32:16.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:16.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:16.531063+0000) 2024-03-06T13:32:17.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:17.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:17.531287+0000) 2024-03-06T13:32:18.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:18.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:18.531447+0000) 2024-03-06T13:32:19.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:19.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:19.531720+0000) 2024-03-06T13:32:20.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:20.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:20.531867+0000) 2024-03-06T13:32:21.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:21.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:21.532008+0000) 2024-03-06T13:32:22.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:22.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:22.532179+0000) 2024-03-06T13:32:23.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:23.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:23.532471+0000) 2024-03-06T13:32:24.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:24.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:24.532731+0000) 2024-03-06T13:32:25.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:25.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:25.532891+0000) 2024-03-06T13:32:26.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:26.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:26.533075+0000) 2024-03-06T13:32:27.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:27.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:27.533310+0000) 2024-03-06T13:32:28.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:28.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:28.533540+0000) 2024-03-06T13:32:29.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:29.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:29.533767+0000) 2024-03-06T13:32:30.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:30.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:30.533948+0000) 2024-03-06T13:32:31.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:31.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:31.534201+0000) 2024-03-06T13:32:32.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:32.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:32.534436+0000) 2024-03-06T13:32:33.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:33.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:33.534604+0000) 2024-03-06T13:32:34.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:34.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:34.534839+0000) 2024-03-06T13:32:35.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:35.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:35.535063+0000) 2024-03-06T13:32:36.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:36.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:36.535293+0000) 2024-03-06T13:32:37.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:37.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:37.535523+0000) 2024-03-06T13:32:38.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:38.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:38.535803+0000) 2024-03-06T13:32:38.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:38.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:39.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:39.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:39.536033+0000) 2024-03-06T13:32:40.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:40.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:40.536263+0000) 2024-03-06T13:32:41.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:41.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:41.536500+0000) 2024-03-06T13:32:42.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:42.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:42.536767+0000) 2024-03-06T13:32:43.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:43.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:43.536959+0000) 2024-03-06T13:32:44.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:44.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:44.537166+0000) 2024-03-06T13:32:45.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:45.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:45.537336+0000) 2024-03-06T13:32:46.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:46.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:46.537537+0000) 2024-03-06T13:32:47.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:47.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:47.537800+0000) 2024-03-06T13:32:48.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:48.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:48.538031+0000) 2024-03-06T13:32:49.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:49.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:49.538265+0000) 2024-03-06T13:32:50.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:50.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:50.538447+0000) 2024-03-06T13:32:51.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:51.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:51.538698+0000) 2024-03-06T13:32:52.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:52.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:52.538906+0000) 2024-03-06T13:32:53.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:53.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:53.539148+0000) 2024-03-06T13:32:54.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:54.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:54.539370+0000) 2024-03-06T13:32:55.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:55.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:55.539584+0000) 2024-03-06T13:32:56.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:56.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:56.539827+0000) 2024-03-06T13:32:57.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:57.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:57.540018+0000) 2024-03-06T13:32:58.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:58.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:58.540242+0000) 2024-03-06T13:32:59.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:32:59.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:59.540479+0000) 2024-03-06T13:33:00.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:00.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:00.540659+0000) 2024-03-06T13:33:01.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:01.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:01.540928+0000) 2024-03-06T13:33:02.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:02.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:02.541085+0000) 2024-03-06T13:33:03.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:03.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:03.541336+0000) 2024-03-06T13:33:04.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:04.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:04.541534+0000) 2024-03-06T13:33:05.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:05.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:05.541795+0000) 2024-03-06T13:33:06.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:06.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:06.541993+0000) 2024-03-06T13:33:07.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:07.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:07.542213+0000) 2024-03-06T13:33:08.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:08.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:08.542450+0000) 2024-03-06T13:33:08.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:08.745 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:09.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:09.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:09.542699+0000) 2024-03-06T13:33:10.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:10.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:10.542930+0000) 2024-03-06T13:33:11.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:11.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:11.543085+0000) 2024-03-06T13:33:12.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:12.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:12.543328+0000) 2024-03-06T13:33:13.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:13.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:13.543558+0000) 2024-03-06T13:33:14.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:14.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:14.543783+0000) 2024-03-06T13:33:15.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:15.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:15.544003+0000) 2024-03-06T13:33:16.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:16.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:16.544230+0000) 2024-03-06T13:33:17.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:17.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:17.544377+0000) 2024-03-06T13:33:18.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:18.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:18.544549+0000) 2024-03-06T13:33:19.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:19.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:19.544782+0000) 2024-03-06T13:33:20.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:20.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:20.545006+0000) 2024-03-06T13:33:21.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:21.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:21.545244+0000) 2024-03-06T13:33:22.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:22.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:22.545473+0000) 2024-03-06T13:33:23.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:23.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:23.545733+0000) 2024-03-06T13:33:24.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:24.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:24.545970+0000) 2024-03-06T13:33:25.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:25.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:25.546131+0000) 2024-03-06T13:33:26.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:26.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:26.546303+0000) 2024-03-06T13:33:27.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:27.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:27.546542+0000) 2024-03-06T13:33:28.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:28.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:28.546777+0000) 2024-03-06T13:33:29.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:29.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:29.547008+0000) 2024-03-06T13:33:30.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:30.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:30.547213+0000) 2024-03-06T13:33:31.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:31.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:31.547442+0000) 2024-03-06T13:33:32.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:32.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:32.547595+0000) 2024-03-06T13:33:33.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:33.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:33.547788+0000) 2024-03-06T13:33:34.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:34.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:34.547994+0000) 2024-03-06T13:33:35.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:35.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:35.548196+0000) 2024-03-06T13:33:36.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:36.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:36.548414+0000) 2024-03-06T13:33:37.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:37.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:37.548655+0000) 2024-03-06T13:33:38.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:38.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:38.548937+0000) 2024-03-06T13:33:38.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:38.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:39.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:39.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:39.549170+0000) 2024-03-06T13:33:40.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:40.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:40.549276+0000) 2024-03-06T13:33:41.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:41.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:41.549397+0000) 2024-03-06T13:33:42.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:42.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:42.549631+0000) 2024-03-06T13:33:43.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:43.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:43.549790+0000) 2024-03-06T13:33:44.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:44.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:44.550047+0000) 2024-03-06T13:33:45.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:45.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:45.550283+0000) 2024-03-06T13:33:46.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:46.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:46.550521+0000) 2024-03-06T13:33:47.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:47.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:47.550780+0000) 2024-03-06T13:33:48.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:48.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:48.550995+0000) 2024-03-06T13:33:49.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:49.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:49.551171+0000) 2024-03-06T13:33:50.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:50.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:50.551409+0000) 2024-03-06T13:33:51.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:51.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:51.551642+0000) 2024-03-06T13:33:52.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:52.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:52.551895+0000) 2024-03-06T13:33:53.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:53.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:53.552144+0000) 2024-03-06T13:33:54.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:54.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:54.552333+0000) 2024-03-06T13:33:55.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:55.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:55.552560+0000) 2024-03-06T13:33:56.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:56.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:56.552793+0000) 2024-03-06T13:33:57.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:57.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:57.553017+0000) 2024-03-06T13:33:58.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:58.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:58.553248+0000) 2024-03-06T13:33:59.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:33:59.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:59.553478+0000) 2024-03-06T13:34:00.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:00.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:00.553731+0000) 2024-03-06T13:34:01.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:01.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:01.553942+0000) 2024-03-06T13:34:02.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:02.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:02.554166+0000) 2024-03-06T13:34:03.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:03.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:03.554351+0000) 2024-03-06T13:34:04.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:04.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:04.554638+0000) 2024-03-06T13:34:05.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:05.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:05.554901+0000) 2024-03-06T13:34:06.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:06.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:06.555068+0000) 2024-03-06T13:34:07.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:07.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:07.555235+0000) 2024-03-06T13:34:08.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:08.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:08.555448+0000) 2024-03-06T13:34:09.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:09.179 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:09.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:09.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:09.555734+0000) 2024-03-06T13:34:10.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:10.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:10.555975+0000) 2024-03-06T13:34:11.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:11.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:11.556150+0000) 2024-03-06T13:34:12.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:12.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:12.556389+0000) 2024-03-06T13:34:13.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:13.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:13.556613+0000) 2024-03-06T13:34:14.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:14.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:14.556830+0000) 2024-03-06T13:34:15.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:15.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:15.557064+0000) 2024-03-06T13:34:16.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:16.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:16.557286+0000) 2024-03-06T13:34:17.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:17.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:17.557486+0000) 2024-03-06T13:34:18.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:18.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:18.557781+0000) 2024-03-06T13:34:19.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:19.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:19.557983+0000) 2024-03-06T13:34:20.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:20.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:20.558211+0000) 2024-03-06T13:34:21.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:21.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:21.558374+0000) 2024-03-06T13:34:22.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:22.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:22.558601+0000) 2024-03-06T13:34:23.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:23.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:23.558779+0000) 2024-03-06T13:34:24.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:24.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:24.559009+0000) 2024-03-06T13:34:25.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:25.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:25.559244+0000) 2024-03-06T13:34:26.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:26.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:26.559474+0000) 2024-03-06T13:34:27.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:27.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:27.559695+0000) 2024-03-06T13:34:28.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:28.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:28.559907+0000) 2024-03-06T13:34:29.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:29.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:29.560078+0000) 2024-03-06T13:34:30.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:30.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:30.560253+0000) 2024-03-06T13:34:31.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:31.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:31.560493+0000) 2024-03-06T13:34:32.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:32.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:32.560775+0000) 2024-03-06T13:34:33.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:33.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:33.560980+0000) 2024-03-06T13:34:34.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:34.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:34.561143+0000) 2024-03-06T13:34:35.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:35.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:35.561411+0000) 2024-03-06T13:34:36.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:36.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:36.561641+0000) 2024-03-06T13:34:37.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:37.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:37.561880+0000) 2024-03-06T13:34:38.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:38.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:38.562112+0000) 2024-03-06T13:34:39.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:39.392 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:39.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:39.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:39.562279+0000) 2024-03-06T13:34:40.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:40.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:40.562458+0000) 2024-03-06T13:34:41.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:41.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:41.562699+0000) 2024-03-06T13:34:42.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:42.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:42.562964+0000) 2024-03-06T13:34:43.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:43.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:43.563208+0000) 2024-03-06T13:34:44.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:44.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:44.563390+0000) 2024-03-06T13:34:45.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:45.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:45.563584+0000) 2024-03-06T13:34:46.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:46.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:46.563819+0000) 2024-03-06T13:34:47.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:47.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:47.564015+0000) 2024-03-06T13:34:48.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:48.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:48.564227+0000) 2024-03-06T13:34:49.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:49.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:49.564466+0000) 2024-03-06T13:34:50.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:50.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:50.564756+0000) 2024-03-06T13:34:51.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:51.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:51.564989+0000) 2024-03-06T13:34:52.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:52.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:52.565221+0000) 2024-03-06T13:34:53.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:53.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:53.565464+0000) 2024-03-06T13:34:54.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:54.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:54.565776+0000) 2024-03-06T13:34:55.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:55.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:55.566064+0000) 2024-03-06T13:34:56.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:56.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:56.566263+0000) 2024-03-06T13:34:57.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:57.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:57.566508+0000) 2024-03-06T13:34:58.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:58.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:58.566785+0000) 2024-03-06T13:34:59.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:34:59.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:59.566971+0000) 2024-03-06T13:35:00.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:00.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:00.567201+0000) 2024-03-06T13:35:01.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:01.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:01.567402+0000) 2024-03-06T13:35:02.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:02.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:02.567635+0000) 2024-03-06T13:35:03.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:03.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:03.567792+0000) 2024-03-06T13:35:04.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:04.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:04.567998+0000) 2024-03-06T13:35:05.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:05.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:05.568193+0000) 2024-03-06T13:35:06.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:06.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:06.568420+0000) 2024-03-06T13:35:07.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:07.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:07.568622+0000) 2024-03-06T13:35:08.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:08.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:08.568877+0000) 2024-03-06T13:35:09.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:09.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:09.569048+0000) 2024-03-06T13:35:09.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:09.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:10.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:10.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:10.569272+0000) 2024-03-06T13:35:11.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:11.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:11.569470+0000) 2024-03-06T13:35:12.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:12.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:12.569756+0000) 2024-03-06T13:35:13.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:13.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:13.569957+0000) 2024-03-06T13:35:14.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:14.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:14.570142+0000) 2024-03-06T13:35:15.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:15.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:15.570317+0000) 2024-03-06T13:35:16.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:16.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:16.570534+0000) 2024-03-06T13:35:17.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:17.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:17.570756+0000) 2024-03-06T13:35:18.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:18.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:18.570989+0000) 2024-03-06T13:35:19.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:19.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:19.571170+0000) 2024-03-06T13:35:20.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:20.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:20.571393+0000) 2024-03-06T13:35:21.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:21.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:21.571576+0000) 2024-03-06T13:35:22.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:22.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:22.571778+0000) 2024-03-06T13:35:23.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:23.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:23.572022+0000) 2024-03-06T13:35:24.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:24.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:24.572248+0000) 2024-03-06T13:35:25.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:25.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:25.572461+0000) 2024-03-06T13:35:26.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:26.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:26.572715+0000) 2024-03-06T13:35:27.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:27.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:27.572869+0000) 2024-03-06T13:35:28.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:28.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:28.573065+0000) 2024-03-06T13:35:29.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:29.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:29.573228+0000) 2024-03-06T13:35:30.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:30.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:30.573451+0000) 2024-03-06T13:35:31.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:31.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:31.573621+0000) 2024-03-06T13:35:32.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:32.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:32.573788+0000) 2024-03-06T13:35:33.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:33.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:33.573972+0000) 2024-03-06T13:35:34.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:34.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:34.574269+0000) 2024-03-06T13:35:35.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:35.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:35.574452+0000) 2024-03-06T13:35:36.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:36.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:36.574699+0000) 2024-03-06T13:35:37.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:37.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:37.574900+0000) 2024-03-06T13:35:38.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:38.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:38.575163+0000) 2024-03-06T13:35:39.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:39.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:39.575359+0000) 2024-03-06T13:35:39.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:39.762 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:40.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:40.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:40.575600+0000) 2024-03-06T13:35:41.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:41.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:41.575819+0000) 2024-03-06T13:35:42.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:42.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:42.576044+0000) 2024-03-06T13:35:43.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:43.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:43.576268+0000) 2024-03-06T13:35:44.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:44.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:44.576503+0000) 2024-03-06T13:35:45.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:45.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:45.576769+0000) 2024-03-06T13:35:46.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:46.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:46.576986+0000) 2024-03-06T13:35:47.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:47.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:47.577231+0000) 2024-03-06T13:35:48.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:48.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:48.577539+0000) 2024-03-06T13:35:49.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:49.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:49.577751+0000) 2024-03-06T13:35:50.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:50.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:50.577947+0000) 2024-03-06T13:35:51.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:51.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:51.578156+0000) 2024-03-06T13:35:52.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:52.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:52.578350+0000) 2024-03-06T13:35:53.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:53.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:53.578567+0000) 2024-03-06T13:35:54.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:54.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:54.578863+0000) 2024-03-06T13:35:55.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:55.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:55.579079+0000) 2024-03-06T13:35:56.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:56.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:56.579333+0000) 2024-03-06T13:35:57.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:57.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:57.579567+0000) 2024-03-06T13:35:58.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:58.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:58.579788+0000) 2024-03-06T13:35:59.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:35:59.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:59.580046+0000) 2024-03-06T13:36:00.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:00.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:00.580293+0000) 2024-03-06T13:36:01.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:01.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:01.580512+0000) 2024-03-06T13:36:02.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:02.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:02.580794+0000) 2024-03-06T13:36:03.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:03.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:03.580993+0000) 2024-03-06T13:36:04.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:04.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:04.581230+0000) 2024-03-06T13:36:05.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:05.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:05.581421+0000) 2024-03-06T13:36:06.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:06.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:06.581615+0000) 2024-03-06T13:36:07.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:07.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:07.581799+0000) 2024-03-06T13:36:08.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:08.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:08.582012+0000) 2024-03-06T13:36:09.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:09.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:09.582153+0000) 2024-03-06T13:36:09.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:09.979 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:10.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:10.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:10.582390+0000) 2024-03-06T13:36:11.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:11.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:11.582594+0000) 2024-03-06T13:36:12.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:12.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:12.582846+0000) 2024-03-06T13:36:13.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:13.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:13.583019+0000) 2024-03-06T13:36:14.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:14.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:14.583250+0000) 2024-03-06T13:36:15.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:15.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:15.583471+0000) 2024-03-06T13:36:16.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:16.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:16.583718+0000) 2024-03-06T13:36:17.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:17.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:17.583903+0000) 2024-03-06T13:36:18.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:18.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:18.584087+0000) 2024-03-06T13:36:19.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:19.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:19.584253+0000) 2024-03-06T13:36:20.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:20.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:20.584487+0000) 2024-03-06T13:36:21.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:21.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:21.584746+0000) 2024-03-06T13:36:22.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:22.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:22.584954+0000) 2024-03-06T13:36:23.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:23.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:23.585155+0000) 2024-03-06T13:36:24.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:24.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:24.585337+0000) 2024-03-06T13:36:25.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:25.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:25.585544+0000) 2024-03-06T13:36:26.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:26.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:26.585765+0000) 2024-03-06T13:36:27.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:27.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:27.586025+0000) 2024-03-06T13:36:28.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:28.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:28.586254+0000) 2024-03-06T13:36:29.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:29.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:29.586462+0000) 2024-03-06T13:36:30.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:30.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:30.586711+0000) 2024-03-06T13:36:31.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:31.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:31.586877+0000) 2024-03-06T13:36:32.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:32.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:32.587144+0000) 2024-03-06T13:36:33.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:33.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:33.587438+0000) 2024-03-06T13:36:34.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:34.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:34.587639+0000) 2024-03-06T13:36:35.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:35.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:35.587811+0000) 2024-03-06T13:36:36.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:36.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:36.587999+0000) 2024-03-06T13:36:37.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:37.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:37.588228+0000) 2024-03-06T13:36:38.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:38.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:38.588496+0000) 2024-03-06T13:36:39.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:39.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:39.588774+0000) 2024-03-06T13:36:40.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:40.191 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:40.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:40.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:40.588983+0000) 2024-03-06T13:36:41.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:41.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:41.589205+0000) 2024-03-06T13:36:42.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:42.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:42.589426+0000) 2024-03-06T13:36:43.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:43.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:43.589704+0000) 2024-03-06T13:36:44.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:44.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:44.589830+0000) 2024-03-06T13:36:45.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:45.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:45.589999+0000) 2024-03-06T13:36:46.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:46.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:46.590185+0000) 2024-03-06T13:36:47.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:47.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:47.590377+0000) 2024-03-06T13:36:48.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:48.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:48.590633+0000) 2024-03-06T13:36:49.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:49.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:49.590864+0000) 2024-03-06T13:36:50.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:50.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:50.591108+0000) 2024-03-06T13:36:51.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:51.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:51.591297+0000) 2024-03-06T13:36:52.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:52.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:52.591543+0000) 2024-03-06T13:36:53.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:53.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:53.591770+0000) 2024-03-06T13:36:54.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:54.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:54.591974+0000) 2024-03-06T13:36:55.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:55.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:55.592167+0000) 2024-03-06T13:36:56.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:56.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:56.592339+0000) 2024-03-06T13:36:57.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:57.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:57.592568+0000) 2024-03-06T13:36:58.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:58.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:58.592838+0000) 2024-03-06T13:36:59.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:36:59.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:59.593044+0000) 2024-03-06T13:37:00.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:00.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:00.593277+0000) 2024-03-06T13:37:01.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:01.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:01.593508+0000) 2024-03-06T13:37:02.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:02.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:02.593760+0000) 2024-03-06T13:37:03.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:03.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:03.593966+0000) 2024-03-06T13:37:04.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:04.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:04.594150+0000) 2024-03-06T13:37:05.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:05.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:05.594383+0000) 2024-03-06T13:37:06.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:06.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:06.594549+0000) 2024-03-06T13:37:07.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:07.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:07.594788+0000) 2024-03-06T13:37:08.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:08.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:08.595028+0000) 2024-03-06T13:37:09.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:09.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:09.595249+0000) 2024-03-06T13:37:10.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:10.414 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:10.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:10.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:10.595457+0000) 2024-03-06T13:37:11.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:11.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:11.595700+0000) 2024-03-06T13:37:12.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:12.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:12.595926+0000) 2024-03-06T13:37:13.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:13.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:13.596135+0000) 2024-03-06T13:37:14.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:14.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:14.596375+0000) 2024-03-06T13:37:15.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:15.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:15.596601+0000) 2024-03-06T13:37:16.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:16.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:16.596797+0000) 2024-03-06T13:37:17.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:17.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:17.597048+0000) 2024-03-06T13:37:18.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:18.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:18.597290+0000) 2024-03-06T13:37:19.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:19.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:19.597512+0000) 2024-03-06T13:37:20.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:20.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:20.597721+0000) 2024-03-06T13:37:21.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:21.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:21.597910+0000) 2024-03-06T13:37:22.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:22.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:22.598094+0000) 2024-03-06T13:37:23.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:23.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:23.598326+0000) 2024-03-06T13:37:24.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:24.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:24.598537+0000) 2024-03-06T13:37:25.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:25.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:25.598774+0000) 2024-03-06T13:37:26.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:26.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:26.598988+0000) 2024-03-06T13:37:27.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:27.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:27.599238+0000) 2024-03-06T13:37:28.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:28.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:28.599437+0000) 2024-03-06T13:37:29.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:29.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:29.599660+0000) 2024-03-06T13:37:30.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:30.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:30.600001+0000) 2024-03-06T13:37:31.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:31.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:31.600233+0000) 2024-03-06T13:37:32.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:32.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:32.600450+0000) 2024-03-06T13:37:33.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:33.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:33.600770+0000) 2024-03-06T13:37:34.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:34.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:34.601003+0000) 2024-03-06T13:37:35.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:35.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:35.601277+0000) 2024-03-06T13:37:36.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:36.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:36.601554+0000) 2024-03-06T13:37:37.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:37.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:37.601862+0000) 2024-03-06T13:37:38.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:38.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:38.602116+0000) 2024-03-06T13:37:39.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:39.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:39.602358+0000) 2024-03-06T13:37:40.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:40.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:40.602636+0000) 2024-03-06T13:37:40.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:40.630 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:41.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:41.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:41.602947+0000) 2024-03-06T13:37:42.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:42.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:42.603235+0000) 2024-03-06T13:37:43.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:43.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:43.603511+0000) 2024-03-06T13:37:44.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:44.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:44.603808+0000) 2024-03-06T13:37:45.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:45.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:45.604047+0000) 2024-03-06T13:37:46.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:46.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:46.604253+0000) 2024-03-06T13:37:47.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:47.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:47.604449+0000) 2024-03-06T13:37:48.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:48.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:48.604716+0000) 2024-03-06T13:37:49.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:49.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:49.604939+0000) 2024-03-06T13:37:50.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:50.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:50.605175+0000) 2024-03-06T13:37:51.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:51.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:51.605411+0000) 2024-03-06T13:37:52.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:52.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:52.605726+0000) 2024-03-06T13:37:53.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:53.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:53.605956+0000) 2024-03-06T13:37:54.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:54.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:54.606153+0000) 2024-03-06T13:37:55.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:55.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:55.606346+0000) 2024-03-06T13:37:56.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:56.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:56.606540+0000) 2024-03-06T13:37:57.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:57.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:57.606791+0000) 2024-03-06T13:37:58.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:58.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:58.607039+0000) 2024-03-06T13:37:59.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:37:59.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:59.607226+0000) 2024-03-06T13:38:00.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:00.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:00.607443+0000) 2024-03-06T13:38:01.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:01.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:01.607645+0000) 2024-03-06T13:38:02.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:02.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:02.607855+0000) 2024-03-06T13:38:03.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:03.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:03.608047+0000) 2024-03-06T13:38:04.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:04.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:04.608235+0000) 2024-03-06T13:38:05.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:05.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:05.608459+0000) 2024-03-06T13:38:06.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:06.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:06.608730+0000) 2024-03-06T13:38:07.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:07.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:07.608967+0000) 2024-03-06T13:38:08.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:08.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:08.609186+0000) 2024-03-06T13:38:09.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:09.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:09.609426+0000) 2024-03-06T13:38:10.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:10.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:10.609704+0000) 2024-03-06T13:38:10.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:10.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:11.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:11.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:11.609952+0000) 2024-03-06T13:38:12.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:12.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:12.610201+0000) 2024-03-06T13:38:13.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:13.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:13.610482+0000) 2024-03-06T13:38:14.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:14.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:14.610721+0000) 2024-03-06T13:38:15.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:15.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:15.610997+0000) 2024-03-06T13:38:16.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:16.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:16.611239+0000) 2024-03-06T13:38:17.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:17.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:17.611523+0000) 2024-03-06T13:38:18.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:18.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:18.611790+0000) 2024-03-06T13:38:19.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:19.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:19.612009+0000) 2024-03-06T13:38:20.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:20.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:20.612234+0000) 2024-03-06T13:38:21.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:21.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:21.612455+0000) 2024-03-06T13:38:22.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:22.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:22.612712+0000) 2024-03-06T13:38:23.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:23.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:23.612931+0000) 2024-03-06T13:38:24.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:24.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:24.613213+0000) 2024-03-06T13:38:25.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:25.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:25.613490+0000) 2024-03-06T13:38:26.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:26.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:26.613806+0000) 2024-03-06T13:38:27.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:27.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:27.614050+0000) 2024-03-06T13:38:28.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:28.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:28.614296+0000) 2024-03-06T13:38:29.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:29.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:29.614571+0000) 2024-03-06T13:38:30.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:30.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:30.614856+0000) 2024-03-06T13:38:31.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:31.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:31.615060+0000) 2024-03-06T13:38:32.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:32.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:32.615272+0000) 2024-03-06T13:38:33.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:33.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:33.615513+0000) 2024-03-06T13:38:34.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:34.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:34.615791+0000) 2024-03-06T13:38:35.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:35.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:35.615979+0000) 2024-03-06T13:38:36.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:36.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:36.616170+0000) 2024-03-06T13:38:37.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:37.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:37.616389+0000) 2024-03-06T13:38:38.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:38.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:38.616594+0000) 2024-03-06T13:38:39.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:39.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:39.616845+0000) 2024-03-06T13:38:40.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:40.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:40.617127+0000) 2024-03-06T13:38:41.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:41.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:41.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:41.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:41.617369+0000) 2024-03-06T13:38:42.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:42.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:42.617641+0000) 2024-03-06T13:38:43.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:43.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:43.617921+0000) 2024-03-06T13:38:44.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:44.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:44.618112+0000) 2024-03-06T13:38:45.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:45.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:45.618330+0000) 2024-03-06T13:38:46.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:46.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:46.618492+0000) 2024-03-06T13:38:47.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:47.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:47.618784+0000) 2024-03-06T13:38:48.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:48.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:48.619061+0000) 2024-03-06T13:38:49.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:49.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:49.619217+0000) 2024-03-06T13:38:50.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:50.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:50.619380+0000) 2024-03-06T13:38:51.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:51.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:51.619579+0000) 2024-03-06T13:38:52.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:52.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:52.619795+0000) 2024-03-06T13:38:53.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:53.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:53.619980+0000) 2024-03-06T13:38:54.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:54.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:54.620197+0000) 2024-03-06T13:38:55.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:55.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:55.620379+0000) 2024-03-06T13:38:56.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:56.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:56.620608+0000) 2024-03-06T13:38:57.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:57.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:57.620773+0000) 2024-03-06T13:38:58.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:58.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:58.621010+0000) 2024-03-06T13:38:59.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:38:59.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:59.621231+0000) 2024-03-06T13:39:00.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:00.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:00.621408+0000) 2024-03-06T13:39:01.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:01.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:01.621580+0000) 2024-03-06T13:39:02.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:02.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:02.621816+0000) 2024-03-06T13:39:03.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:03.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:03.622058+0000) 2024-03-06T13:39:04.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:04.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:04.622286+0000) 2024-03-06T13:39:05.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:05.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:05.622495+0000) 2024-03-06T13:39:06.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:06.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:06.622736+0000) 2024-03-06T13:39:07.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:07.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:07.622973+0000) 2024-03-06T13:39:08.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:08.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:08.623230+0000) 2024-03-06T13:39:09.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:09.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:09.623505+0000) 2024-03-06T13:39:10.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:10.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:10.623808+0000) 2024-03-06T13:39:11.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:11.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:11.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:11.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:11.624050+0000) 2024-03-06T13:39:12.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:12.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:12.624255+0000) 2024-03-06T13:39:13.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:13.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:13.624520+0000) 2024-03-06T13:39:14.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:14.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:14.624740+0000) 2024-03-06T13:39:15.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:15.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:15.624982+0000) 2024-03-06T13:39:16.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:16.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:16.625221+0000) 2024-03-06T13:39:17.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:17.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:17.625460+0000) 2024-03-06T13:39:18.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:18.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:18.625781+0000) 2024-03-06T13:39:19.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:19.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:19.626025+0000) 2024-03-06T13:39:20.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:20.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:20.626256+0000) 2024-03-06T13:39:21.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:21.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:21.626464+0000) 2024-03-06T13:39:22.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:22.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:22.626621+0000) 2024-03-06T13:39:23.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:23.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:23.626838+0000) 2024-03-06T13:39:24.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:24.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:24.627032+0000) 2024-03-06T13:39:25.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:25.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:25.627221+0000) 2024-03-06T13:39:26.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:26.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:26.627405+0000) 2024-03-06T13:39:27.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:27.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:27.627631+0000) 2024-03-06T13:39:28.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:28.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:28.627897+0000) 2024-03-06T13:39:29.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:29.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:29.628121+0000) 2024-03-06T13:39:30.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:30.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:30.628346+0000) 2024-03-06T13:39:31.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:31.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:31.628566+0000) 2024-03-06T13:39:32.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:32.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:32.628791+0000) 2024-03-06T13:39:33.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:33.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:33.629010+0000) 2024-03-06T13:39:34.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:34.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:34.629231+0000) 2024-03-06T13:39:35.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:35.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:35.629459+0000) 2024-03-06T13:39:36.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:36.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:36.629653+0000) 2024-03-06T13:39:37.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:37.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:37.629878+0000) 2024-03-06T13:39:38.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:38.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:38.630116+0000) 2024-03-06T13:39:39.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:39.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:39.630343+0000) 2024-03-06T13:39:40.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:40.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:40.630575+0000) 2024-03-06T13:39:41.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:41.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:41.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:41.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:41.630869+0000) 2024-03-06T13:39:42.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:42.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:42.631148+0000) 2024-03-06T13:39:43.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:43.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:43.631415+0000) 2024-03-06T13:39:44.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:44.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:44.631716+0000) 2024-03-06T13:39:45.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:45.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:45.631956+0000) 2024-03-06T13:39:46.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:46.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:46.632151+0000) 2024-03-06T13:39:47.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:47.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:47.632421+0000) 2024-03-06T13:39:48.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:48.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:48.632741+0000) 2024-03-06T13:39:49.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:49.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:49.632970+0000) 2024-03-06T13:39:50.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:50.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:50.633195+0000) 2024-03-06T13:39:51.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:51.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:51.633435+0000) 2024-03-06T13:39:52.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:52.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:52.633743+0000) 2024-03-06T13:39:53.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:53.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:53.634022+0000) 2024-03-06T13:39:54.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:54.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:54.634220+0000) 2024-03-06T13:39:55.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:55.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:55.634443+0000) 2024-03-06T13:39:56.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:56.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:56.634768+0000) 2024-03-06T13:39:57.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:57.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:57.634992+0000) 2024-03-06T13:39:58.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:58.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:58.635170+0000) 2024-03-06T13:39:59.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:39:59.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:59.635392+0000) 2024-03-06T13:40:00.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:00.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:00.635619+0000) 2024-03-06T13:40:01.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:01.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:01.635814+0000) 2024-03-06T13:40:02.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:02.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:02.636044+0000) 2024-03-06T13:40:03.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:03.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:03.636286+0000) 2024-03-06T13:40:04.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:04.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:04.636566+0000) 2024-03-06T13:40:05.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:05.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:05.636846+0000) 2024-03-06T13:40:06.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:06.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:06.637088+0000) 2024-03-06T13:40:07.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:07.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:07.637334+0000) 2024-03-06T13:40:08.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:08.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:08.637637+0000) 2024-03-06T13:40:09.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:09.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:09.637915+0000) 2024-03-06T13:40:10.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:10.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:10.638119+0000) 2024-03-06T13:40:11.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:11.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:11.638348+0000) 2024-03-06T13:40:11.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:11.657 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:12.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:12.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:12.638581+0000) 2024-03-06T13:40:13.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:13.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:13.638818+0000) 2024-03-06T13:40:14.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:14.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:14.639034+0000) 2024-03-06T13:40:15.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:15.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:15.639266+0000) 2024-03-06T13:40:16.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:16.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:16.639544+0000) 2024-03-06T13:40:17.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:17.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:17.639824+0000) 2024-03-06T13:40:18.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:18.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:18.640076+0000) 2024-03-06T13:40:19.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:19.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:19.640305+0000) 2024-03-06T13:40:20.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:20.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:20.640586+0000) 2024-03-06T13:40:21.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:21.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:21.640863+0000) 2024-03-06T13:40:22.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:22.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:22.641094+0000) 2024-03-06T13:40:23.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:23.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:23.641338+0000) 2024-03-06T13:40:24.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:24.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:24.641600+0000) 2024-03-06T13:40:25.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:25.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:25.641827+0000) 2024-03-06T13:40:26.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:26.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:26.641974+0000) 2024-03-06T13:40:27.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:27.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:27.642160+0000) 2024-03-06T13:40:28.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:28.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:28.642400+0000) 2024-03-06T13:40:29.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:29.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:29.642620+0000) 2024-03-06T13:40:30.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:30.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:30.642843+0000) 2024-03-06T13:40:31.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:31.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:31.643028+0000) 2024-03-06T13:40:32.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:32.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:32.643215+0000) 2024-03-06T13:40:33.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:33.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:33.643446+0000) 2024-03-06T13:40:34.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:34.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:34.643696+0000) 2024-03-06T13:40:35.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:35.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:35.643922+0000) 2024-03-06T13:40:36.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:36.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:36.644149+0000) 2024-03-06T13:40:37.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:37.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:37.644371+0000) 2024-03-06T13:40:38.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:38.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:38.644588+0000) 2024-03-06T13:40:39.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:39.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:39.644843+0000) 2024-03-06T13:40:40.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:40.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:40.645086+0000) 2024-03-06T13:40:41.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:41.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:41.645282+0000) 2024-03-06T13:40:41.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:41.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:42.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:42.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:42.645514+0000) 2024-03-06T13:40:43.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:43.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:43.645767+0000) 2024-03-06T13:40:44.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:44.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:44.646040+0000) 2024-03-06T13:40:45.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:45.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:45.646226+0000) 2024-03-06T13:40:46.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:46.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:46.646448+0000) 2024-03-06T13:40:47.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:47.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:47.646699+0000) 2024-03-06T13:40:48.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:48.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:48.647005+0000) 2024-03-06T13:40:49.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:49.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:49.647204+0000) 2024-03-06T13:40:50.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:50.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:50.647427+0000) 2024-03-06T13:40:51.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:51.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:51.647722+0000) 2024-03-06T13:40:52.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:52.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:52.648001+0000) 2024-03-06T13:40:53.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:53.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:53.648195+0000) 2024-03-06T13:40:54.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:54.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:54.648427+0000) 2024-03-06T13:40:55.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:55.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:55.648602+0000) 2024-03-06T13:40:56.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:56.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:56.648825+0000) 2024-03-06T13:40:57.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:57.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:57.649049+0000) 2024-03-06T13:40:58.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:58.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:58.649299+0000) 2024-03-06T13:40:59.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:40:59.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:59.649526+0000) 2024-03-06T13:41:00.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:00.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:00.649754+0000) 2024-03-06T13:41:01.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:01.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:01.649979+0000) 2024-03-06T13:41:02.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:02.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:02.650202+0000) 2024-03-06T13:41:03.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:03.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:03.650430+0000) 2024-03-06T13:41:04.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:04.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:04.650652+0000) 2024-03-06T13:41:05.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:05.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:05.650909+0000) 2024-03-06T13:41:06.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:06.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:06.651108+0000) 2024-03-06T13:41:07.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:07.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:07.651305+0000) 2024-03-06T13:41:08.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:08.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:08.651529+0000) 2024-03-06T13:41:09.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:09.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:09.651762+0000) 2024-03-06T13:41:10.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:10.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:10.651991+0000) 2024-03-06T13:41:11.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:11.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:11.652217+0000) 2024-03-06T13:41:12.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:12.079 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:12.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:12.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:12.652448+0000) 2024-03-06T13:41:13.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:13.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:13.652698+0000) 2024-03-06T13:41:14.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:14.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:14.652933+0000) 2024-03-06T13:41:15.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:15.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:15.653151+0000) 2024-03-06T13:41:16.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:16.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:16.653378+0000) 2024-03-06T13:41:17.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:17.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:17.653604+0000) 2024-03-06T13:41:18.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:18.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:18.653847+0000) 2024-03-06T13:41:19.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:19.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:19.654078+0000) 2024-03-06T13:41:20.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:20.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:20.654299+0000) 2024-03-06T13:41:21.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:21.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:21.654496+0000) 2024-03-06T13:41:22.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:22.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:22.654727+0000) 2024-03-06T13:41:23.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:23.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:23.654921+0000) 2024-03-06T13:41:24.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:24.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:24.655156+0000) 2024-03-06T13:41:25.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:25.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:25.655380+0000) 2024-03-06T13:41:26.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:26.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:26.655567+0000) 2024-03-06T13:41:27.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:27.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:27.655798+0000) 2024-03-06T13:41:28.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:28.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:28.656002+0000) 2024-03-06T13:41:29.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:29.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:29.656216+0000) 2024-03-06T13:41:30.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:30.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:30.656438+0000) 2024-03-06T13:41:31.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:31.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:31.656659+0000) 2024-03-06T13:41:32.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:32.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:32.656924+0000) 2024-03-06T13:41:33.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:33.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:33.657147+0000) 2024-03-06T13:41:34.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:34.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:34.657380+0000) 2024-03-06T13:41:35.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:35.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:35.657601+0000) 2024-03-06T13:41:36.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:36.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:36.657817+0000) 2024-03-06T13:41:37.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:37.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:37.658028+0000) 2024-03-06T13:41:38.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:38.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:38.658219+0000) 2024-03-06T13:41:39.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:39.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:39.658441+0000) 2024-03-06T13:41:40.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:40.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:40.658700+0000) 2024-03-06T13:41:41.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:41.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:41.658932+0000) 2024-03-06T13:41:42.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:42.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:42.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:42.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:42.659159+0000) 2024-03-06T13:41:43.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:43.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:43.659387+0000) 2024-03-06T13:41:44.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:44.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:44.659583+0000) 2024-03-06T13:41:45.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:45.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:45.659762+0000) 2024-03-06T13:41:46.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:46.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:46.659936+0000) 2024-03-06T13:41:47.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:47.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:47.660176+0000) 2024-03-06T13:41:48.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:48.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:48.660459+0000) 2024-03-06T13:41:49.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:49.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:49.660760+0000) 2024-03-06T13:41:50.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:50.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:50.660990+0000) 2024-03-06T13:41:51.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:51.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:51.661176+0000) 2024-03-06T13:41:52.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:52.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:52.661438+0000) 2024-03-06T13:41:53.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:53.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:53.661745+0000) 2024-03-06T13:41:54.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:54.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:54.661973+0000) 2024-03-06T13:41:55.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:55.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:55.662159+0000) 2024-03-06T13:41:56.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:56.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:56.662386+0000) 2024-03-06T13:41:57.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:57.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:57.662613+0000) 2024-03-06T13:41:58.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:58.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:58.662849+0000) 2024-03-06T13:41:59.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:41:59.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:59.663041+0000) 2024-03-06T13:42:00.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:00.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:00.663230+0000) 2024-03-06T13:42:01.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:01.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:01.663460+0000) 2024-03-06T13:42:02.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:02.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:02.663719+0000) 2024-03-06T13:42:03.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:03.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:03.663915+0000) 2024-03-06T13:42:04.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:04.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:04.664108+0000) 2024-03-06T13:42:05.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:05.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:05.664333+0000) 2024-03-06T13:42:06.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:06.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:06.664545+0000) 2024-03-06T13:42:07.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:07.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:07.664771+0000) 2024-03-06T13:42:08.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:08.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:08.665001+0000) 2024-03-06T13:42:09.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:09.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:09.665198+0000) 2024-03-06T13:42:10.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:10.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:10.665429+0000) 2024-03-06T13:42:11.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:11.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:11.665660+0000) 2024-03-06T13:42:12.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:12.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:12.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:12.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:12.665905+0000) 2024-03-06T13:42:13.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:13.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:13.666106+0000) 2024-03-06T13:42:14.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:14.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:14.666332+0000) 2024-03-06T13:42:15.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:15.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:15.666564+0000) 2024-03-06T13:42:16.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:16.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:16.666797+0000) 2024-03-06T13:42:17.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:17.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:17.667009+0000) 2024-03-06T13:42:18.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:18.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:18.667215+0000) 2024-03-06T13:42:19.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:19.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:19.667417+0000) 2024-03-06T13:42:20.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:20.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:20.667610+0000) 2024-03-06T13:42:21.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:21.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:21.667847+0000) 2024-03-06T13:42:22.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:22.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:22.668001+0000) 2024-03-06T13:42:23.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:23.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:23.668193+0000) 2024-03-06T13:42:24.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:24.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:24.668417+0000) 2024-03-06T13:42:25.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:25.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:25.668734+0000) 2024-03-06T13:42:26.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:26.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:26.668926+0000) 2024-03-06T13:42:27.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:27.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:27.669152+0000) 2024-03-06T13:42:28.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:28.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:28.669388+0000) 2024-03-06T13:42:29.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:29.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:29.669620+0000) 2024-03-06T13:42:30.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:30.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:30.669846+0000) 2024-03-06T13:42:31.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:31.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:31.670037+0000) 2024-03-06T13:42:32.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:32.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:32.670223+0000) 2024-03-06T13:42:33.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:33.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:33.670425+0000) 2024-03-06T13:42:34.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:34.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:34.670657+0000) 2024-03-06T13:42:35.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:35.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:35.670884+0000) 2024-03-06T13:42:36.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:36.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:36.671084+0000) 2024-03-06T13:42:37.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:37.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:37.671285+0000) 2024-03-06T13:42:38.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:38.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:38.671469+0000) 2024-03-06T13:42:39.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:39.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:39.671728+0000) 2024-03-06T13:42:40.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:40.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:40.671913+0000) 2024-03-06T13:42:41.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:41.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:41.672122+0000) 2024-03-06T13:42:42.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:42.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:42.672278+0000) 2024-03-06T13:42:42.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:42.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:43.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:43.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:43.672458+0000) 2024-03-06T13:42:44.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:44.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:44.672711+0000) 2024-03-06T13:42:45.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:45.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:45.672900+0000) 2024-03-06T13:42:46.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:46.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:46.673088+0000) 2024-03-06T13:42:47.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:47.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:47.673219+0000) 2024-03-06T13:42:48.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:48.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:48.673417+0000) 2024-03-06T13:42:49.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:49.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:49.673641+0000) 2024-03-06T13:42:50.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:50.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:50.673857+0000) 2024-03-06T13:42:51.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:51.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:51.674034+0000) 2024-03-06T13:42:52.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:52.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:52.674210+0000) 2024-03-06T13:42:53.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:53.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:53.674394+0000) 2024-03-06T13:42:54.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:54.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:54.674624+0000) 2024-03-06T13:42:55.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:55.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:55.674796+0000) 2024-03-06T13:42:56.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:56.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:56.675021+0000) 2024-03-06T13:42:57.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:57.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:57.675204+0000) 2024-03-06T13:42:58.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:58.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:58.675400+0000) 2024-03-06T13:42:59.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:42:59.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:59.675570+0000) 2024-03-06T13:43:00.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:00.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:00.675815+0000) 2024-03-06T13:43:01.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:01.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:01.675999+0000) 2024-03-06T13:43:02.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:02.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:02.676176+0000) 2024-03-06T13:43:03.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:03.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:03.676373+0000) 2024-03-06T13:43:04.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:04.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:04.676600+0000) 2024-03-06T13:43:05.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:05.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:05.676904+0000) 2024-03-06T13:43:06.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:06.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:06.677093+0000) 2024-03-06T13:43:07.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:07.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:07.677317+0000) 2024-03-06T13:43:08.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:08.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:08.677560+0000) 2024-03-06T13:43:09.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:09.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:09.677790+0000) 2024-03-06T13:43:10.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:10.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:10.677977+0000) 2024-03-06T13:43:11.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:11.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:11.678177+0000) 2024-03-06T13:43:12.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:12.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:12.678407+0000) 2024-03-06T13:43:12.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:12.899 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:13.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:13.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:13.678705+0000) 2024-03-06T13:43:14.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:14.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:14.678912+0000) 2024-03-06T13:43:15.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:15.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:15.679097+0000) 2024-03-06T13:43:16.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:16.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:16.679325+0000) 2024-03-06T13:43:17.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:17.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:17.679548+0000) 2024-03-06T13:43:18.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:18.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:18.679796+0000) 2024-03-06T13:43:19.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:19.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:19.679986+0000) 2024-03-06T13:43:20.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:20.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:20.680172+0000) 2024-03-06T13:43:21.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:21.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:21.680394+0000) 2024-03-06T13:43:22.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:22.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:22.680617+0000) 2024-03-06T13:43:23.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:23.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:23.680839+0000) 2024-03-06T13:43:24.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:24.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:24.681031+0000) 2024-03-06T13:43:25.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:25.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:25.681216+0000) 2024-03-06T13:43:26.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:26.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:26.681434+0000) 2024-03-06T13:43:27.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:27.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:27.681693+0000) 2024-03-06T13:43:28.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:28.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:28.681977+0000) 2024-03-06T13:43:29.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:29.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:29.682218+0000) 2024-03-06T13:43:30.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:30.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:30.682429+0000) 2024-03-06T13:43:31.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:31.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:31.682615+0000) 2024-03-06T13:43:32.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:32.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:32.682822+0000) 2024-03-06T13:43:33.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:33.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:33.683029+0000) 2024-03-06T13:43:34.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:34.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:34.683223+0000) 2024-03-06T13:43:35.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:35.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:35.683429+0000) 2024-03-06T13:43:36.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:36.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:36.683639+0000) 2024-03-06T13:43:37.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:37.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:37.683873+0000) 2024-03-06T13:43:38.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:38.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:38.684104+0000) 2024-03-06T13:43:39.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:39.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:39.684331+0000) 2024-03-06T13:43:40.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:40.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:40.684567+0000) 2024-03-06T13:43:41.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:41.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:41.684808+0000) 2024-03-06T13:43:42.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:42.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:42.684998+0000) 2024-03-06T13:43:43.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:43.102 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:43.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:43.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:43.685207+0000) 2024-03-06T13:43:44.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:44.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:44.685395+0000) 2024-03-06T13:43:45.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:45.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:45.685619+0000) 2024-03-06T13:43:46.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:46.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:46.685839+0000) 2024-03-06T13:43:47.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:47.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:47.685975+0000) 2024-03-06T13:43:48.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:48.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:48.686177+0000) 2024-03-06T13:43:49.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:49.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:49.686412+0000) 2024-03-06T13:43:50.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:50.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:50.686601+0000) 2024-03-06T13:43:51.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:51.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:51.686826+0000) 2024-03-06T13:43:52.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:52.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:52.687053+0000) 2024-03-06T13:43:53.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:53.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:53.687253+0000) 2024-03-06T13:43:54.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:54.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:54.687457+0000) 2024-03-06T13:43:55.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:55.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:55.687630+0000) 2024-03-06T13:43:56.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:56.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:56.687893+0000) 2024-03-06T13:43:57.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:57.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:57.688115+0000) 2024-03-06T13:43:58.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:58.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:58.688334+0000) 2024-03-06T13:43:59.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:43:59.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:59.688553+0000) 2024-03-06T13:44:00.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:00.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:00.688773+0000) 2024-03-06T13:44:01.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:01.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:01.688922+0000) 2024-03-06T13:44:02.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:02.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:02.689160+0000) 2024-03-06T13:44:03.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:03.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:03.689413+0000) 2024-03-06T13:44:04.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:04.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:04.689644+0000) 2024-03-06T13:44:05.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:05.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:05.689893+0000) 2024-03-06T13:44:06.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:06.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:06.690121+0000) 2024-03-06T13:44:07.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:07.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:07.690392+0000) 2024-03-06T13:44:08.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:08.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:08.690634+0000) 2024-03-06T13:44:09.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:09.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:09.690867+0000) 2024-03-06T13:44:10.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:10.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:10.691055+0000) 2024-03-06T13:44:11.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:11.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:11.691193+0000) 2024-03-06T13:44:12.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:12.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:12.691402+0000) 2024-03-06T13:44:13.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:13.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:13.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:13.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:13.691634+0000) 2024-03-06T13:44:14.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:14.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:14.691858+0000) 2024-03-06T13:44:15.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:15.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:15.692074+0000) 2024-03-06T13:44:16.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:16.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:16.692254+0000) 2024-03-06T13:44:17.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:17.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:17.692479+0000) 2024-03-06T13:44:18.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:18.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:18.692759+0000) 2024-03-06T13:44:19.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:19.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:19.692984+0000) 2024-03-06T13:44:20.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:20.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:20.693212+0000) 2024-03-06T13:44:21.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:21.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:21.693436+0000) 2024-03-06T13:44:22.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:22.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:22.693657+0000) 2024-03-06T13:44:23.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:23.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:23.693895+0000) 2024-03-06T13:44:24.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:24.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:24.694135+0000) 2024-03-06T13:44:25.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:25.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:25.694361+0000) 2024-03-06T13:44:26.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:26.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:26.694593+0000) 2024-03-06T13:44:27.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:27.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:27.694820+0000) 2024-03-06T13:44:28.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:28.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:28.695024+0000) 2024-03-06T13:44:29.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:29.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:29.695222+0000) 2024-03-06T13:44:30.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:30.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:30.695420+0000) 2024-03-06T13:44:31.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:31.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:31.695654+0000) 2024-03-06T13:44:32.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:32.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:32.695881+0000) 2024-03-06T13:44:33.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:33.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:33.696090+0000) 2024-03-06T13:44:34.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:34.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:34.696280+0000) 2024-03-06T13:44:35.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:35.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:35.696516+0000) 2024-03-06T13:44:36.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:36.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:36.696784+0000) 2024-03-06T13:44:37.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:37.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:37.696987+0000) 2024-03-06T13:44:38.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:38.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:38.697192+0000) 2024-03-06T13:44:39.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:39.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:39.697418+0000) 2024-03-06T13:44:40.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:40.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:40.697645+0000) 2024-03-06T13:44:41.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:41.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:41.697876+0000) 2024-03-06T13:44:42.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:42.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:42.698070+0000) 2024-03-06T13:44:43.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:43.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:43.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:43.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:43.698255+0000) 2024-03-06T13:44:44.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:44.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:44.698447+0000) 2024-03-06T13:44:45.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:45.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:45.698697+0000) 2024-03-06T13:44:46.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:46.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:46.698921+0000) 2024-03-06T13:44:47.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:47.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:47.699139+0000) 2024-03-06T13:44:48.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:48.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:48.699379+0000) 2024-03-06T13:44:49.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:49.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:49.699606+0000) 2024-03-06T13:44:50.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:50.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:50.699841+0000) 2024-03-06T13:44:51.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:51.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:51.700070+0000) 2024-03-06T13:44:52.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:52.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:52.700289+0000) 2024-03-06T13:44:53.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:53.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:53.700538+0000) 2024-03-06T13:44:54.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:54.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:54.700788+0000) 2024-03-06T13:44:55.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:55.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:55.701061+0000) 2024-03-06T13:44:56.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:56.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:56.701290+0000) 2024-03-06T13:44:57.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:57.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:57.701468+0000) 2024-03-06T13:44:58.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:58.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:58.701779+0000) 2024-03-06T13:44:59.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:44:59.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:59.702021+0000) 2024-03-06T13:45:00.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:00.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:00.702261+0000) 2024-03-06T13:45:01.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:01.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:01.702482+0000) 2024-03-06T13:45:02.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:02.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:02.702760+0000) 2024-03-06T13:45:03.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:03.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:03.703000+0000) 2024-03-06T13:45:04.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:04.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:04.703231+0000) 2024-03-06T13:45:05.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:05.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:05.703462+0000) 2024-03-06T13:45:06.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:06.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:06.703759+0000) 2024-03-06T13:45:07.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:07.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:07.703940+0000) 2024-03-06T13:45:08.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:08.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:08.704143+0000) 2024-03-06T13:45:09.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:09.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:09.704352+0000) 2024-03-06T13:45:10.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:10.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:10.704562+0000) 2024-03-06T13:45:11.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:11.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:11.704813+0000) 2024-03-06T13:45:12.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:12.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:12.705021+0000) 2024-03-06T13:45:13.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:13.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:13.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:13.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:13.705238+0000) 2024-03-06T13:45:14.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:14.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:14.705462+0000) 2024-03-06T13:45:15.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:15.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:15.705745+0000) 2024-03-06T13:45:16.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:16.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:16.705951+0000) 2024-03-06T13:45:17.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:17.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:17.706156+0000) 2024-03-06T13:45:18.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:18.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:18.706360+0000) 2024-03-06T13:45:19.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:19.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:19.706596+0000) 2024-03-06T13:45:20.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:20.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:20.706814+0000) 2024-03-06T13:45:21.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:21.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:21.707003+0000) 2024-03-06T13:45:22.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:22.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:22.707196+0000) 2024-03-06T13:45:23.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:23.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:23.707432+0000) 2024-03-06T13:45:24.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:24.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:24.707693+0000) 2024-03-06T13:45:25.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:25.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:25.707884+0000) 2024-03-06T13:45:26.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:26.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:26.708089+0000) 2024-03-06T13:45:27.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:27.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:27.708334+0000) 2024-03-06T13:45:28.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:28.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:28.708588+0000) 2024-03-06T13:45:29.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:29.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:29.708818+0000) 2024-03-06T13:45:30.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:30.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:30.708983+0000) 2024-03-06T13:45:31.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:31.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:31.709210+0000) 2024-03-06T13:45:32.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:32.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:32.709435+0000) 2024-03-06T13:45:33.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:33.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:33.709617+0000) 2024-03-06T13:45:34.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:34.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:34.709854+0000) 2024-03-06T13:45:35.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:35.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:35.710043+0000) 2024-03-06T13:45:36.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:36.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:36.710270+0000) 2024-03-06T13:45:37.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:37.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:37.710476+0000) 2024-03-06T13:45:38.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:38.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:38.710785+0000) 2024-03-06T13:45:39.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:39.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:39.711013+0000) 2024-03-06T13:45:40.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:40.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:40.711206+0000) 2024-03-06T13:45:41.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:41.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:41.711351+0000) 2024-03-06T13:45:42.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:42.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:42.711575+0000) 2024-03-06T13:45:43.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:43.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:43.711859+0000) 2024-03-06T13:45:43.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:43.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:44.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:44.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:44.712063+0000) 2024-03-06T13:45:45.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:45.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:45.712292+0000) 2024-03-06T13:45:46.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:46.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:46.712516+0000) 2024-03-06T13:45:47.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:47.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:47.712771+0000) 2024-03-06T13:45:48.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:48.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:48.713012+0000) 2024-03-06T13:45:49.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:49.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:49.713235+0000) 2024-03-06T13:45:50.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:50.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:50.713469+0000) 2024-03-06T13:45:51.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:51.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:51.713692+0000) 2024-03-06T13:45:52.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:52.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:52.713919+0000) 2024-03-06T13:45:53.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:53.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:53.714114+0000) 2024-03-06T13:45:54.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:54.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:54.714305+0000) 2024-03-06T13:45:55.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:55.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:55.714505+0000) 2024-03-06T13:45:56.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:56.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:56.714789+0000) 2024-03-06T13:45:57.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:57.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:57.715000+0000) 2024-03-06T13:45:58.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:58.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:58.715213+0000) 2024-03-06T13:45:59.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:45:59.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:59.715413+0000) 2024-03-06T13:46:00.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:00.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:00.715606+0000) 2024-03-06T13:46:01.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:01.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:01.715844+0000) 2024-03-06T13:46:02.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:02.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:02.716046+0000) 2024-03-06T13:46:03.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:03.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:03.716247+0000) 2024-03-06T13:46:04.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:04.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:04.716462+0000) 2024-03-06T13:46:05.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:05.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:05.716695+0000) 2024-03-06T13:46:06.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:06.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:06.716923+0000) 2024-03-06T13:46:07.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:07.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:07.717105+0000) 2024-03-06T13:46:08.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:08.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:08.717315+0000) 2024-03-06T13:46:09.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:09.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:09.717518+0000) 2024-03-06T13:46:10.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:10.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:10.717735+0000) 2024-03-06T13:46:11.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:11.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:11.717958+0000) 2024-03-06T13:46:12.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:12.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:12.718186+0000) 2024-03-06T13:46:13.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:13.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:13.718408+0000) 2024-03-06T13:46:14.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:14.017 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:14.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:14.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:14.718639+0000) 2024-03-06T13:46:15.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:15.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:15.718867+0000) 2024-03-06T13:46:16.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:16.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:16.719069+0000) 2024-03-06T13:46:17.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:17.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:17.719303+0000) 2024-03-06T13:46:18.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:18.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:18.719499+0000) 2024-03-06T13:46:19.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:19.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:19.719744+0000) 2024-03-06T13:46:20.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:20.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:20.719940+0000) 2024-03-06T13:46:21.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:21.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:21.720170+0000) 2024-03-06T13:46:22.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:22.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:22.720324+0000) 2024-03-06T13:46:23.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:23.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:23.720537+0000) 2024-03-06T13:46:24.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:24.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:24.720805+0000) 2024-03-06T13:46:25.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:25.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:25.721036+0000) 2024-03-06T13:46:26.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:26.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:26.721276+0000) 2024-03-06T13:46:27.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:27.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:27.721471+0000) 2024-03-06T13:46:28.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:28.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:28.721761+0000) 2024-03-06T13:46:29.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:29.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:29.721991+0000) 2024-03-06T13:46:30.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:30.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:30.722217+0000) 2024-03-06T13:46:31.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:31.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:31.722445+0000) 2024-03-06T13:46:32.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:32.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:32.722700+0000) 2024-03-06T13:46:33.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:33.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:33.722950+0000) 2024-03-06T13:46:34.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:34.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:34.723145+0000) 2024-03-06T13:46:35.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:35.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:35.723369+0000) 2024-03-06T13:46:36.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:36.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:36.723648+0000) 2024-03-06T13:46:37.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:37.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:37.723862+0000) 2024-03-06T13:46:38.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:38.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:38.724100+0000) 2024-03-06T13:46:39.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:39.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:39.724336+0000) 2024-03-06T13:46:40.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:40.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:40.724529+0000) 2024-03-06T13:46:41.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:41.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:41.724756+0000) 2024-03-06T13:46:42.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:42.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:42.725000+0000) 2024-03-06T13:46:43.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:43.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:43.725177+0000) 2024-03-06T13:46:44.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:44.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:44.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:44.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:44.725401+0000) 2024-03-06T13:46:45.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:45.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:45.725618+0000) 2024-03-06T13:46:46.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:46.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:46.725887+0000) 2024-03-06T13:46:47.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:47.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:47.726102+0000) 2024-03-06T13:46:48.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:48.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:48.726409+0000) 2024-03-06T13:46:49.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:49.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:49.726718+0000) 2024-03-06T13:46:50.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:50.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:50.727005+0000) 2024-03-06T13:46:51.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:51.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:51.727276+0000) 2024-03-06T13:46:52.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:52.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:52.727516+0000) 2024-03-06T13:46:53.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:53.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:53.727815+0000) 2024-03-06T13:46:54.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:54.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:54.728087+0000) 2024-03-06T13:46:55.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:55.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:55.728365+0000) 2024-03-06T13:46:56.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:56.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:56.728644+0000) 2024-03-06T13:46:57.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:57.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:57.728897+0000) 2024-03-06T13:46:58.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:58.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:58.729097+0000) 2024-03-06T13:46:59.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:46:59.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:59.729320+0000) 2024-03-06T13:47:00.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:00.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:00.729548+0000) 2024-03-06T13:47:01.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:01.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:01.729766+0000) 2024-03-06T13:47:02.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:02.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:02.729982+0000) 2024-03-06T13:47:03.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:03.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:03.730203+0000) 2024-03-06T13:47:04.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:04.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:04.730430+0000) 2024-03-06T13:47:05.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:05.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:05.730653+0000) 2024-03-06T13:47:06.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:06.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:06.730857+0000) 2024-03-06T13:47:07.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:07.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:07.731099+0000) 2024-03-06T13:47:08.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:08.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:08.731381+0000) 2024-03-06T13:47:09.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:09.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:09.731629+0000) 2024-03-06T13:47:10.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:10.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:10.731917+0000) 2024-03-06T13:47:11.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:11.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:11.732102+0000) 2024-03-06T13:47:12.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:12.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:12.732342+0000) 2024-03-06T13:47:13.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:13.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:13.732582+0000) 2024-03-06T13:47:14.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:14.367 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:14.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:14.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:14.732783+0000) 2024-03-06T13:47:15.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:15.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:15.733002+0000) 2024-03-06T13:47:16.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:16.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:16.733242+0000) 2024-03-06T13:47:17.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:17.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:17.733484+0000) 2024-03-06T13:47:18.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:18.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:18.733738+0000) 2024-03-06T13:47:19.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:19.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:19.733982+0000) 2024-03-06T13:47:20.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:20.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:20.734200+0000) 2024-03-06T13:47:21.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:21.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:21.734419+0000) 2024-03-06T13:47:22.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:22.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:22.734637+0000) 2024-03-06T13:47:23.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:23.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:23.734851+0000) 2024-03-06T13:47:24.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:24.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:24.735145+0000) 2024-03-06T13:47:25.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:25.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:25.735388+0000) 2024-03-06T13:47:26.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:26.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:26.735623+0000) 2024-03-06T13:47:27.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:27.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:27.735844+0000) 2024-03-06T13:47:28.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:28.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:28.736079+0000) 2024-03-06T13:47:29.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:29.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:29.736265+0000) 2024-03-06T13:47:30.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:30.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:30.736454+0000) 2024-03-06T13:47:31.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:31.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:31.736702+0000) 2024-03-06T13:47:32.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:32.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:32.736931+0000) 2024-03-06T13:47:33.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:33.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:33.737154+0000) 2024-03-06T13:47:34.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:34.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:34.737382+0000) 2024-03-06T13:47:35.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:35.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:35.737570+0000) 2024-03-06T13:47:36.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:36.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:36.737740+0000) 2024-03-06T13:47:37.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:37.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:37.737951+0000) 2024-03-06T13:47:38.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:38.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:38.738186+0000) 2024-03-06T13:47:39.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:39.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:39.738380+0000) 2024-03-06T13:47:40.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:40.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:40.738571+0000) 2024-03-06T13:47:41.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:41.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:41.738770+0000) 2024-03-06T13:47:42.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:42.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:42.738985+0000) 2024-03-06T13:47:43.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:43.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:43.739178+0000) 2024-03-06T13:47:44.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:44.572 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:44.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:44.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:44.739362+0000) 2024-03-06T13:47:45.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:45.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:45.739590+0000) 2024-03-06T13:47:46.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:46.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:46.739817+0000) 2024-03-06T13:47:47.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:47.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:47.740038+0000) 2024-03-06T13:47:48.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:48.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:48.740278+0000) 2024-03-06T13:47:49.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:49.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:49.740504+0000) 2024-03-06T13:47:50.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:50.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:50.740761+0000) 2024-03-06T13:47:51.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:51.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:51.740985+0000) 2024-03-06T13:47:52.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:52.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:52.741173+0000) 2024-03-06T13:47:53.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:53.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:53.741404+0000) 2024-03-06T13:47:54.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:54.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:54.741635+0000) 2024-03-06T13:47:55.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:55.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:55.741860+0000) 2024-03-06T13:47:56.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:56.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:56.742088+0000) 2024-03-06T13:47:57.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:57.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:57.742310+0000) 2024-03-06T13:47:58.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:58.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:58.742541+0000) 2024-03-06T13:47:59.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:47:59.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:59.742776+0000) 2024-03-06T13:48:00.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:00.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:00.743002+0000) 2024-03-06T13:48:01.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:01.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:01.743181+0000) 2024-03-06T13:48:02.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:02.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:02.743364+0000) 2024-03-06T13:48:03.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:03.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:03.743543+0000) 2024-03-06T13:48:04.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:04.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:04.743740+0000) 2024-03-06T13:48:05.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:05.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:05.743912+0000) 2024-03-06T13:48:06.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:06.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:06.744127+0000) 2024-03-06T13:48:07.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:07.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:07.744346+0000) 2024-03-06T13:48:08.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:08.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:08.744580+0000) 2024-03-06T13:48:09.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:09.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:09.744824+0000) 2024-03-06T13:48:10.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:10.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:10.745488+0000) 2024-03-06T13:48:11.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:11.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:11.745721+0000) 2024-03-06T13:48:12.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:12.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:12.745961+0000) 2024-03-06T13:48:13.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:13.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:13.746153+0000) 2024-03-06T13:48:14.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:14.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:14.746246+0000) 2024-03-06T13:48:14.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:14.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:15.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:15.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:15.746372+0000) 2024-03-06T13:48:16.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:16.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:16.746498+0000) 2024-03-06T13:48:17.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:17.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:17.746648+0000) 2024-03-06T13:48:18.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:18.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:18.746847+0000) 2024-03-06T13:48:19.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:19.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:19.747095+0000) 2024-03-06T13:48:20.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:20.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:20.747273+0000) 2024-03-06T13:48:21.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:21.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:21.747495+0000) 2024-03-06T13:48:22.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:22.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:22.747752+0000) 2024-03-06T13:48:23.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:23.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:23.747977+0000) 2024-03-06T13:48:24.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:24.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:24.748126+0000) 2024-03-06T13:48:25.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:25.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:25.748319+0000) 2024-03-06T13:48:26.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:26.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:26.748553+0000) 2024-03-06T13:48:27.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:27.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:27.748790+0000) 2024-03-06T13:48:28.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:28.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:28.748958+0000) 2024-03-06T13:48:29.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:29.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:29.749168+0000) 2024-03-06T13:48:30.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:30.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:30.749380+0000) 2024-03-06T13:48:31.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:31.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:31.749600+0000) 2024-03-06T13:48:32.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:32.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:32.749821+0000) 2024-03-06T13:48:33.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:33.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:33.750052+0000) 2024-03-06T13:48:34.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:34.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:34.750278+0000) 2024-03-06T13:48:35.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:35.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:35.750401+0000) 2024-03-06T13:48:36.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:36.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:36.750634+0000) 2024-03-06T13:48:37.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:37.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:37.750811+0000) 2024-03-06T13:48:38.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:38.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:38.751073+0000) 2024-03-06T13:48:39.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:39.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:39.751228+0000) 2024-03-06T13:48:40.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:40.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:40.751415+0000) 2024-03-06T13:48:41.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:41.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:41.751654+0000) 2024-03-06T13:48:42.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:42.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:42.751915+0000) 2024-03-06T13:48:43.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:43.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:43.752141+0000) 2024-03-06T13:48:44.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:44.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:44.752390+0000) 2024-03-06T13:48:44.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:44.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:45.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:45.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:45.752644+0000) 2024-03-06T13:48:46.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:46.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:46.752919+0000) 2024-03-06T13:48:47.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:47.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:47.753143+0000) 2024-03-06T13:48:48.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:48.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:48.753379+0000) 2024-03-06T13:48:49.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:49.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:49.753617+0000) 2024-03-06T13:48:50.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:50.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:50.753848+0000) 2024-03-06T13:48:51.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:51.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:51.754023+0000) 2024-03-06T13:48:52.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:52.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:52.754252+0000) 2024-03-06T13:48:53.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:53.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:53.754475+0000) 2024-03-06T13:48:54.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:54.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:54.754735+0000) 2024-03-06T13:48:55.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:55.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:55.754957+0000) 2024-03-06T13:48:56.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:56.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:56.755143+0000) 2024-03-06T13:48:57.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:57.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:57.755373+0000) 2024-03-06T13:48:58.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:58.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:58.755616+0000) 2024-03-06T13:48:59.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:48:59.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:59.755840+0000) 2024-03-06T13:49:00.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:00.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:00.755997+0000) 2024-03-06T13:49:01.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:01.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:01.756216+0000) 2024-03-06T13:49:02.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:02.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:02.756349+0000) 2024-03-06T13:49:03.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:03.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:03.756558+0000) 2024-03-06T13:49:04.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:04.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:04.756799+0000) 2024-03-06T13:49:05.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:05.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:05.757083+0000) 2024-03-06T13:49:06.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:06.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:06.757277+0000) 2024-03-06T13:49:07.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:07.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:07.757505+0000) 2024-03-06T13:49:08.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:08.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:08.757764+0000) 2024-03-06T13:49:09.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:09.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:09.758042+0000) 2024-03-06T13:49:10.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:10.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:10.758308+0000) 2024-03-06T13:49:11.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:11.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:11.758579+0000) 2024-03-06T13:49:12.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:12.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:12.758861+0000) 2024-03-06T13:49:13.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:13.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:13.759074+0000) 2024-03-06T13:49:14.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:14.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:14.759329+0000) 2024-03-06T13:49:15.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:15.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:15.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:15.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:15.759506+0000) 2024-03-06T13:49:16.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:16.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:16.759781+0000) 2024-03-06T13:49:17.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:17.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:17.760008+0000) 2024-03-06T13:49:18.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:18.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:18.760244+0000) 2024-03-06T13:49:19.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:19.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:19.760492+0000) 2024-03-06T13:49:20.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:20.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:20.760722+0000) 2024-03-06T13:49:21.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:21.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:21.760968+0000) 2024-03-06T13:49:22.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:22.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:22.761195+0000) 2024-03-06T13:49:23.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:23.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:23.761429+0000) 2024-03-06T13:49:24.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:24.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:24.761703+0000) 2024-03-06T13:49:25.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:25.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:25.761928+0000) 2024-03-06T13:49:26.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:26.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:26.762113+0000) 2024-03-06T13:49:27.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:27.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:27.762353+0000) 2024-03-06T13:49:28.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:28.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:28.762556+0000) 2024-03-06T13:49:29.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:29.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:29.762823+0000) 2024-03-06T13:49:30.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:30.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:30.763050+0000) 2024-03-06T13:49:31.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:31.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:31.763278+0000) 2024-03-06T13:49:32.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:32.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:32.763463+0000) 2024-03-06T13:49:33.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:33.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:33.763650+0000) 2024-03-06T13:49:34.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:34.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:34.763884+0000) 2024-03-06T13:49:35.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:35.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:35.764095+0000) 2024-03-06T13:49:36.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:36.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:36.764309+0000) 2024-03-06T13:49:37.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:37.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:37.764548+0000) 2024-03-06T13:49:38.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:38.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:38.764792+0000) 2024-03-06T13:49:39.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:39.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:39.765027+0000) 2024-03-06T13:49:40.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:40.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:40.765259+0000) 2024-03-06T13:49:41.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:41.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:41.765483+0000) 2024-03-06T13:49:42.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:42.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:42.765707+0000) 2024-03-06T13:49:43.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:43.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:43.765961+0000) 2024-03-06T13:49:44.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:44.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:44.766197+0000) 2024-03-06T13:49:45.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:45.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:45.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:45.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:45.766443+0000) 2024-03-06T13:49:46.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:46.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:46.766767+0000) 2024-03-06T13:49:47.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:47.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:47.767034+0000) 2024-03-06T13:49:48.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:48.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:48.767316+0000) 2024-03-06T13:49:49.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:49.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:49.767588+0000) 2024-03-06T13:49:50.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:50.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:50.767819+0000) 2024-03-06T13:49:51.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:51.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:51.768024+0000) 2024-03-06T13:49:52.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:52.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:52.768249+0000) 2024-03-06T13:49:53.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:53.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:53.768532+0000) 2024-03-06T13:49:54.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:54.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:54.768764+0000) 2024-03-06T13:49:55.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:55.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:55.769000+0000) 2024-03-06T13:49:56.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:56.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:56.769223+0000) 2024-03-06T13:49:57.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:57.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:57.769438+0000) 2024-03-06T13:49:58.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:58.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:58.769759+0000) 2024-03-06T13:49:59.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:49:59.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:59.770035+0000) 2024-03-06T13:50:00.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:00.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:00.770315+0000) 2024-03-06T13:50:01.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:01.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:01.770587+0000) 2024-03-06T13:50:02.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:02.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:02.770822+0000) 2024-03-06T13:50:03.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:03.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:03.770999+0000) 2024-03-06T13:50:04.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:04.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:04.771201+0000) 2024-03-06T13:50:05.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:05.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:05.771469+0000) 2024-03-06T13:50:06.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:06.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:06.771751+0000) 2024-03-06T13:50:07.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:07.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:07.771980+0000) 2024-03-06T13:50:08.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:08.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:08.772222+0000) 2024-03-06T13:50:09.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:09.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:09.772438+0000) 2024-03-06T13:50:10.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:10.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:10.772751+0000) 2024-03-06T13:50:11.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:11.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:11.772993+0000) 2024-03-06T13:50:12.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:12.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:12.773192+0000) 2024-03-06T13:50:13.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:13.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:13.773375+0000) 2024-03-06T13:50:14.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:14.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:14.773603+0000) 2024-03-06T13:50:15.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:15.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:15.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:15.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:15.773837+0000) 2024-03-06T13:50:16.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:16.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:16.774079+0000) 2024-03-06T13:50:17.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:17.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:17.774320+0000) 2024-03-06T13:50:18.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:18.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:18.774571+0000) 2024-03-06T13:50:19.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:19.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:19.774814+0000) 2024-03-06T13:50:20.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:20.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:20.775041+0000) 2024-03-06T13:50:21.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:21.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:21.775272+0000) 2024-03-06T13:50:22.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:22.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:22.775492+0000) 2024-03-06T13:50:23.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:23.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:23.775754+0000) 2024-03-06T13:50:24.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:24.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:24.775961+0000) 2024-03-06T13:50:25.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:25.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:25.776193+0000) 2024-03-06T13:50:26.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:26.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:26.776423+0000) 2024-03-06T13:50:27.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:27.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:27.776659+0000) 2024-03-06T13:50:28.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:28.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:28.776976+0000) 2024-03-06T13:50:29.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:29.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:29.777235+0000) 2024-03-06T13:50:30.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:30.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:30.777433+0000) 2024-03-06T13:50:31.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:31.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:31.777657+0000) 2024-03-06T13:50:32.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:32.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:32.777876+0000) 2024-03-06T13:50:33.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:33.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:33.778120+0000) 2024-03-06T13:50:34.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:34.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:34.778390+0000) 2024-03-06T13:50:35.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:35.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:35.778661+0000) 2024-03-06T13:50:36.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:36.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:36.778957+0000) 2024-03-06T13:50:37.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:37.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:37.779103+0000) 2024-03-06T13:50:38.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:38.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:38.779323+0000) 2024-03-06T13:50:39.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:39.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:39.779483+0000) 2024-03-06T13:50:40.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:40.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:40.779772+0000) 2024-03-06T13:50:41.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:41.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:41.779982+0000) 2024-03-06T13:50:42.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:42.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:42.780257+0000) 2024-03-06T13:50:43.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:43.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:43.780530+0000) 2024-03-06T13:50:44.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:44.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:44.780800+0000) 2024-03-06T13:50:45.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:45.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:45.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:45.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:45.781010+0000) 2024-03-06T13:50:46.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:46.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:46.781223+0000) 2024-03-06T13:50:47.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:47.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:47.781499+0000) 2024-03-06T13:50:48.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:48.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:48.781771+0000) 2024-03-06T13:50:49.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:49.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:49.782002+0000) 2024-03-06T13:50:50.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:50.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:50.782234+0000) 2024-03-06T13:50:51.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:51.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:51.782455+0000) 2024-03-06T13:50:52.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:52.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:52.782723+0000) 2024-03-06T13:50:53.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:53.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:53.782927+0000) 2024-03-06T13:50:54.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:54.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:54.783176+0000) 2024-03-06T13:50:55.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:55.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:55.783408+0000) 2024-03-06T13:50:56.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:56.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:56.783636+0000) 2024-03-06T13:50:57.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:57.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:57.783793+0000) 2024-03-06T13:50:58.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:58.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:58.784007+0000) 2024-03-06T13:50:59.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:50:59.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:59.784228+0000) 2024-03-06T13:51:00.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:00.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:00.784453+0000) 2024-03-06T13:51:01.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:01.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:01.784712+0000) 2024-03-06T13:51:02.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:02.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:02.784917+0000) 2024-03-06T13:51:03.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:03.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:03.785154+0000) 2024-03-06T13:51:04.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:04.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:04.785436+0000) 2024-03-06T13:51:05.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:05.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:05.785713+0000) 2024-03-06T13:51:06.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:06.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:06.785989+0000) 2024-03-06T13:51:07.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:07.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:07.786250+0000) 2024-03-06T13:51:08.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:08.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:08.786470+0000) 2024-03-06T13:51:09.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:09.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:09.786659+0000) 2024-03-06T13:51:10.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:10.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:10.786940+0000) 2024-03-06T13:51:11.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:11.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:11.787153+0000) 2024-03-06T13:51:12.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:12.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:12.787330+0000) 2024-03-06T13:51:13.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:13.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:13.787559+0000) 2024-03-06T13:51:14.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:14.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:14.787793+0000) 2024-03-06T13:51:15.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:15.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:15.788018+0000) 2024-03-06T13:51:15.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:15.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:16.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:16.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:16.788207+0000) 2024-03-06T13:51:17.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:17.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:17.788483+0000) 2024-03-06T13:51:18.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:18.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:18.788798+0000) 2024-03-06T13:51:19.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:19.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:19.789071+0000) 2024-03-06T13:51:20.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:20.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:20.789306+0000) 2024-03-06T13:51:21.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:21.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:21.789532+0000) 2024-03-06T13:51:22.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:22.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:22.789811+0000) 2024-03-06T13:51:23.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:23.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:23.790050+0000) 2024-03-06T13:51:24.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:24.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:24.790308+0000) 2024-03-06T13:51:25.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:25.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:25.790581+0000) 2024-03-06T13:51:26.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:26.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:26.790844+0000) 2024-03-06T13:51:27.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:27.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:27.791138+0000) 2024-03-06T13:51:28.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:28.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:28.791359+0000) 2024-03-06T13:51:29.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:29.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:29.791581+0000) 2024-03-06T13:51:30.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:30.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:30.791850+0000) 2024-03-06T13:51:31.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:31.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:31.792043+0000) 2024-03-06T13:51:32.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:32.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:32.792223+0000) 2024-03-06T13:51:33.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:33.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:33.792485+0000) 2024-03-06T13:51:34.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:34.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:34.792750+0000) 2024-03-06T13:51:35.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:35.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:35.793018+0000) 2024-03-06T13:51:36.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:36.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:36.793239+0000) 2024-03-06T13:51:37.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:37.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:37.793458+0000) 2024-03-06T13:51:38.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:38.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:38.793728+0000) 2024-03-06T13:51:39.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:39.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:39.793950+0000) 2024-03-06T13:51:40.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:40.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:40.794186+0000) 2024-03-06T13:51:41.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:41.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:41.794401+0000) 2024-03-06T13:51:42.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:42.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:42.794577+0000) 2024-03-06T13:51:43.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:43.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:43.794765+0000) 2024-03-06T13:51:44.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:44.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:44.795000+0000) 2024-03-06T13:51:45.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:45.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:45.795230+0000) 2024-03-06T13:51:46.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:46.077 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:46.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:46.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:46.795467+0000) 2024-03-06T13:51:47.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:47.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:47.795727+0000) 2024-03-06T13:51:48.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:48.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:48.795964+0000) 2024-03-06T13:51:49.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:49.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:49.796185+0000) 2024-03-06T13:51:50.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:50.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:50.796366+0000) 2024-03-06T13:51:51.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:51.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:51.796600+0000) 2024-03-06T13:51:52.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:52.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:52.796840+0000) 2024-03-06T13:51:53.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:53.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:53.797092+0000) 2024-03-06T13:51:54.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:54.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:54.797375+0000) 2024-03-06T13:51:55.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:55.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:55.797655+0000) 2024-03-06T13:51:56.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:56.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:56.797912+0000) 2024-03-06T13:51:57.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:57.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:57.798132+0000) 2024-03-06T13:51:58.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:58.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:58.798275+0000) 2024-03-06T13:51:59.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:51:59.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:59.798483+0000) 2024-03-06T13:52:00.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:00.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:00.798755+0000) 2024-03-06T13:52:01.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:01.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:01.798950+0000) 2024-03-06T13:52:02.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:02.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:02.799175+0000) 2024-03-06T13:52:03.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:03.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:03.799368+0000) 2024-03-06T13:52:04.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:04.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:04.799602+0000) 2024-03-06T13:52:05.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:05.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:05.799801+0000) 2024-03-06T13:52:06.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:06.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:06.800017+0000) 2024-03-06T13:52:07.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:07.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:07.800145+0000) 2024-03-06T13:52:08.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:08.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:08.800367+0000) 2024-03-06T13:52:09.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:09.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:09.800580+0000) 2024-03-06T13:52:10.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:10.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:10.800811+0000) 2024-03-06T13:52:11.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:11.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:11.801032+0000) 2024-03-06T13:52:12.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:12.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:12.801256+0000) 2024-03-06T13:52:13.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:13.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:13.801476+0000) 2024-03-06T13:52:14.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:14.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:14.801729+0000) 2024-03-06T13:52:15.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:15.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:15.801951+0000) 2024-03-06T13:52:16.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:16.274 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:16.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:16.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:16.802075+0000) 2024-03-06T13:52:17.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:17.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:17.802290+0000) 2024-03-06T13:52:18.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:18.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:18.802526+0000) 2024-03-06T13:52:19.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:19.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:19.802766+0000) 2024-03-06T13:52:20.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:20.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:20.803038+0000) 2024-03-06T13:52:21.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:21.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:21.803275+0000) 2024-03-06T13:52:22.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:22.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:22.803499+0000) 2024-03-06T13:52:23.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:23.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:23.803811+0000) 2024-03-06T13:52:24.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:24.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:24.804035+0000) 2024-03-06T13:52:25.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:25.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:25.804256+0000) 2024-03-06T13:52:26.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:26.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:26.804476+0000) 2024-03-06T13:52:27.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:27.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:27.804696+0000) 2024-03-06T13:52:28.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:28.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:28.804910+0000) 2024-03-06T13:52:29.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:29.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:29.805156+0000) 2024-03-06T13:52:30.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:30.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:30.805415+0000) 2024-03-06T13:52:31.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:31.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:31.805635+0000) 2024-03-06T13:52:32.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:32.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:32.805858+0000) 2024-03-06T13:52:33.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:33.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:33.806085+0000) 2024-03-06T13:52:34.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:34.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:34.806277+0000) 2024-03-06T13:52:35.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:35.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:35.806464+0000) 2024-03-06T13:52:36.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:36.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:36.806706+0000) 2024-03-06T13:52:37.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:37.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:37.806903+0000) 2024-03-06T13:52:38.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:38.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:38.807108+0000) 2024-03-06T13:52:39.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:39.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:39.807301+0000) 2024-03-06T13:52:40.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:40.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:40.807528+0000) 2024-03-06T13:52:41.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:41.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:41.807780+0000) 2024-03-06T13:52:42.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:42.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:42.808052+0000) 2024-03-06T13:52:43.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:43.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:43.808330+0000) 2024-03-06T13:52:44.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:44.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:44.808543+0000) 2024-03-06T13:52:45.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:45.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:45.808771+0000) 2024-03-06T13:52:46.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:46.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:46.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:46.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:46.808995+0000) 2024-03-06T13:52:47.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:47.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:47.809221+0000) 2024-03-06T13:52:48.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:48.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:48.809459+0000) 2024-03-06T13:52:49.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:49.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:49.809734+0000) 2024-03-06T13:52:50.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:50.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:50.809948+0000) 2024-03-06T13:52:51.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:51.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:51.810172+0000) 2024-03-06T13:52:52.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:52.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:52.810389+0000) 2024-03-06T13:52:53.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:53.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:53.810711+0000) 2024-03-06T13:52:54.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:54.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:54.810991+0000) 2024-03-06T13:52:55.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:55.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:55.811201+0000) 2024-03-06T13:52:56.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:56.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:56.811478+0000) 2024-03-06T13:52:57.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:57.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:57.811775+0000) 2024-03-06T13:52:58.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:58.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:58.812000+0000) 2024-03-06T13:52:59.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:52:59.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:59.812228+0000) 2024-03-06T13:53:00.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:00.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:00.812424+0000) 2024-03-06T13:53:01.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:01.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:01.812619+0000) 2024-03-06T13:53:02.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:02.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:02.812802+0000) 2024-03-06T13:53:03.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:03.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:03.812952+0000) 2024-03-06T13:53:04.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:04.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:04.813157+0000) 2024-03-06T13:53:05.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:05.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:05.813348+0000) 2024-03-06T13:53:06.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:06.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:06.813570+0000) 2024-03-06T13:53:07.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:07.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:07.813805+0000) 2024-03-06T13:53:08.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:08.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:08.813998+0000) 2024-03-06T13:53:09.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:09.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:09.814212+0000) 2024-03-06T13:53:10.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:10.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:10.814443+0000) 2024-03-06T13:53:11.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:11.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:11.814701+0000) 2024-03-06T13:53:12.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:12.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:12.814937+0000) 2024-03-06T13:53:13.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:13.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:13.815165+0000) 2024-03-06T13:53:14.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:14.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:14.815342+0000) 2024-03-06T13:53:15.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:15.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:15.815572+0000) 2024-03-06T13:53:16.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:16.618 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:16.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:16.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:16.815838+0000) 2024-03-06T13:53:17.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:17.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:17.816027+0000) 2024-03-06T13:53:18.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:18.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:18.816271+0000) 2024-03-06T13:53:19.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:19.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:19.816494+0000) 2024-03-06T13:53:20.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:20.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:20.816759+0000) 2024-03-06T13:53:21.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:21.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:21.816984+0000) 2024-03-06T13:53:22.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:22.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:22.817208+0000) 2024-03-06T13:53:23.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:23.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:23.817418+0000) 2024-03-06T13:53:24.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:24.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:24.817635+0000) 2024-03-06T13:53:25.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:25.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:25.817854+0000) 2024-03-06T13:53:26.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:26.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:26.818084+0000) 2024-03-06T13:53:27.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:27.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:27.818360+0000) 2024-03-06T13:53:28.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:28.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:28.818652+0000) 2024-03-06T13:53:29.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:29.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:29.818959+0000) 2024-03-06T13:53:30.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:30.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:30.819094+0000) 2024-03-06T13:53:31.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:31.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:31.819268+0000) 2024-03-06T13:53:32.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:32.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:32.819515+0000) 2024-03-06T13:53:33.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:33.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:33.819769+0000) 2024-03-06T13:53:34.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:34.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:34.820009+0000) 2024-03-06T13:53:35.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:35.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:35.820239+0000) 2024-03-06T13:53:36.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:36.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:36.820436+0000) 2024-03-06T13:53:37.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:37.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:37.820706+0000) 2024-03-06T13:53:38.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:38.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:38.820976+0000) 2024-03-06T13:53:39.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:39.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:39.821225+0000) 2024-03-06T13:53:40.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:40.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:40.821424+0000) 2024-03-06T13:53:41.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:41.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:41.821711+0000) 2024-03-06T13:53:42.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:42.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:42.821957+0000) 2024-03-06T13:53:43.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:43.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:43.822168+0000) 2024-03-06T13:53:44.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:44.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:44.822391+0000) 2024-03-06T13:53:45.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:45.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:45.822619+0000) 2024-03-06T13:53:46.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:46.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:46.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:46.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:46.822780+0000) 2024-03-06T13:53:47.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:47.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:47.822994+0000) 2024-03-06T13:53:48.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:48.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:48.823234+0000) 2024-03-06T13:53:49.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:49.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:49.823461+0000) 2024-03-06T13:53:50.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:50.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:50.823721+0000) 2024-03-06T13:53:51.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:51.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:51.823948+0000) 2024-03-06T13:53:52.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:52.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:52.824162+0000) 2024-03-06T13:53:53.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:53.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:53.824389+0000) 2024-03-06T13:53:54.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:54.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:54.824575+0000) 2024-03-06T13:53:55.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:55.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:55.824816+0000) 2024-03-06T13:53:56.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:56.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:56.825031+0000) 2024-03-06T13:53:57.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:57.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:57.825225+0000) 2024-03-06T13:53:58.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:58.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:58.825443+0000) 2024-03-06T13:53:59.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:53:59.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:59.825652+0000) 2024-03-06T13:54:00.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:00.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:00.825911+0000) 2024-03-06T13:54:01.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:01.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:01.826147+0000) 2024-03-06T13:54:02.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:02.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:02.826380+0000) 2024-03-06T13:54:03.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:03.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:03.826605+0000) 2024-03-06T13:54:04.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:04.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:04.826886+0000) 2024-03-06T13:54:05.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:05.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:05.827114+0000) 2024-03-06T13:54:06.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:06.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:06.827394+0000) 2024-03-06T13:54:07.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:07.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:07.827637+0000) 2024-03-06T13:54:08.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:08.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:08.827903+0000) 2024-03-06T13:54:09.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:09.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:09.828129+0000) 2024-03-06T13:54:10.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:10.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:10.828346+0000) 2024-03-06T13:54:11.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:11.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:11.828623+0000) 2024-03-06T13:54:12.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:12.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:12.828852+0000) 2024-03-06T13:54:13.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:13.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:13.829081+0000) 2024-03-06T13:54:14.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:14.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:14.829291+0000) 2024-03-06T13:54:15.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:15.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:15.829494+0000) 2024-03-06T13:54:16.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:16.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:16.829749+0000) 2024-03-06T13:54:16.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:16.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:17.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:17.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:17.829979+0000) 2024-03-06T13:54:18.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:18.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:18.830162+0000) 2024-03-06T13:54:19.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:19.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:19.830385+0000) 2024-03-06T13:54:20.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:20.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:20.830629+0000) 2024-03-06T13:54:21.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:21.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:21.830818+0000) 2024-03-06T13:54:22.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:22.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:22.831021+0000) 2024-03-06T13:54:23.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:23.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:23.831264+0000) 2024-03-06T13:54:24.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:24.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:24.831458+0000) 2024-03-06T13:54:25.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:25.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:25.831694+0000) 2024-03-06T13:54:26.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:26.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:26.831888+0000) 2024-03-06T13:54:27.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:27.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:27.832110+0000) 2024-03-06T13:54:28.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:28.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:28.832345+0000) 2024-03-06T13:54:29.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:29.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:29.832521+0000) 2024-03-06T13:54:30.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:30.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:30.832766+0000) 2024-03-06T13:54:31.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:31.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:31.833001+0000) 2024-03-06T13:54:32.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:32.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:32.833160+0000) 2024-03-06T13:54:33.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:33.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:33.833382+0000) 2024-03-06T13:54:34.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:34.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:34.833587+0000) 2024-03-06T13:54:35.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:35.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:35.833795+0000) 2024-03-06T13:54:36.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:36.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:36.834063+0000) 2024-03-06T13:54:37.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:37.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:37.834263+0000) 2024-03-06T13:54:38.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:38.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:38.834459+0000) 2024-03-06T13:54:39.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:39.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:39.834752+0000) 2024-03-06T13:54:40.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:40.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:40.834958+0000) 2024-03-06T13:54:41.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:41.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:41.835181+0000) 2024-03-06T13:54:42.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:42.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:42.835378+0000) 2024-03-06T13:54:43.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:43.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:43.835610+0000) 2024-03-06T13:54:44.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:44.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:44.835838+0000) 2024-03-06T13:54:45.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:45.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:45.836020+0000) 2024-03-06T13:54:46.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:46.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:46.836219+0000) 2024-03-06T13:54:47.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:47.194 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:47.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:47.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:47.836454+0000) 2024-03-06T13:54:48.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:48.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:48.836716+0000) 2024-03-06T13:54:49.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:49.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:49.836983+0000) 2024-03-06T13:54:50.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:50.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:50.837274+0000) 2024-03-06T13:54:51.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:51.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:51.837515+0000) 2024-03-06T13:54:52.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:52.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:52.837766+0000) 2024-03-06T13:54:53.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:53.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:53.837994+0000) 2024-03-06T13:54:54.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:54.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:54.838229+0000) 2024-03-06T13:54:55.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:55.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:55.838473+0000) 2024-03-06T13:54:56.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:56.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:56.838745+0000) 2024-03-06T13:54:57.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:57.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:57.838944+0000) 2024-03-06T13:54:58.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:58.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:58.839165+0000) 2024-03-06T13:54:59.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:54:59.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:59.839349+0000) 2024-03-06T13:55:00.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:00.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:00.839591+0000) 2024-03-06T13:55:01.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:01.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:01.839817+0000) 2024-03-06T13:55:02.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:02.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:02.840040+0000) 2024-03-06T13:55:03.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:03.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:03.840244+0000) 2024-03-06T13:55:04.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:04.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:04.840482+0000) 2024-03-06T13:55:05.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:05.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:05.840794+0000) 2024-03-06T13:55:06.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:06.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:06.841070+0000) 2024-03-06T13:55:07.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:07.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:07.841269+0000) 2024-03-06T13:55:08.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:08.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:08.841524+0000) 2024-03-06T13:55:09.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:09.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:09.841762+0000) 2024-03-06T13:55:10.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:10.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:10.842006+0000) 2024-03-06T13:55:11.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:11.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:11.842280+0000) 2024-03-06T13:55:12.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:12.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:12.842557+0000) 2024-03-06T13:55:13.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:13.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:13.842806+0000) 2024-03-06T13:55:14.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:14.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:14.843072+0000) 2024-03-06T13:55:15.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:15.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:15.843274+0000) 2024-03-06T13:55:16.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:16.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:16.843494+0000) 2024-03-06T13:55:17.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:17.402 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:17.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:17.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:17.843716+0000) 2024-03-06T13:55:18.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:18.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:18.843977+0000) 2024-03-06T13:55:19.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:19.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:19.844193+0000) 2024-03-06T13:55:20.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:20.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:20.844374+0000) 2024-03-06T13:55:21.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:21.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:21.844596+0000) 2024-03-06T13:55:22.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:22.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:22.844847+0000) 2024-03-06T13:55:23.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:23.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:23.845088+0000) 2024-03-06T13:55:24.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:24.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:24.845329+0000) 2024-03-06T13:55:25.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:25.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:25.845539+0000) 2024-03-06T13:55:26.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:26.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:26.845785+0000) 2024-03-06T13:55:27.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:27.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:27.845988+0000) 2024-03-06T13:55:28.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:28.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:28.846245+0000) 2024-03-06T13:55:29.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:29.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:29.846481+0000) 2024-03-06T13:55:30.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:30.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:30.846741+0000) 2024-03-06T13:55:31.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:31.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:31.846876+0000) 2024-03-06T13:55:32.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:32.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:32.847042+0000) 2024-03-06T13:55:33.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:33.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:33.847268+0000) 2024-03-06T13:55:34.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:34.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:34.847491+0000) 2024-03-06T13:55:35.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:35.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:35.847774+0000) 2024-03-06T13:55:36.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:36.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:36.847967+0000) 2024-03-06T13:55:37.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:37.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:37.848151+0000) 2024-03-06T13:55:38.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:38.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:38.848398+0000) 2024-03-06T13:55:39.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:39.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:39.848620+0000) 2024-03-06T13:55:40.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:40.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:40.848857+0000) 2024-03-06T13:55:41.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:41.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:41.849076+0000) 2024-03-06T13:55:42.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:42.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:42.849294+0000) 2024-03-06T13:55:43.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:43.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:43.849519+0000) 2024-03-06T13:55:44.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:44.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:44.849817+0000) 2024-03-06T13:55:45.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:45.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:45.850018+0000) 2024-03-06T13:55:46.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:46.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:46.850213+0000) 2024-03-06T13:55:47.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:47.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:47.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:47.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:47.850366+0000) 2024-03-06T13:55:48.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:48.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:48.850552+0000) 2024-03-06T13:55:49.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:49.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:49.850778+0000) 2024-03-06T13:55:50.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:50.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:50.850958+0000) 2024-03-06T13:55:51.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:51.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:51.851188+0000) 2024-03-06T13:55:52.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:52.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:52.851340+0000) 2024-03-06T13:55:53.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:53.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:53.851551+0000) 2024-03-06T13:55:54.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:54.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:54.851783+0000) 2024-03-06T13:55:55.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:55.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:55.852009+0000) 2024-03-06T13:55:56.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:56.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:56.852246+0000) 2024-03-06T13:55:57.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:57.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:57.852470+0000) 2024-03-06T13:55:58.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:58.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:58.852753+0000) 2024-03-06T13:55:59.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:55:59.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:59.852900+0000) 2024-03-06T13:56:00.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:00.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:00.853027+0000) 2024-03-06T13:56:01.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:01.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:01.853202+0000) 2024-03-06T13:56:02.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:02.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:02.853412+0000) 2024-03-06T13:56:03.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:03.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:03.853578+0000) 2024-03-06T13:56:04.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:04.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:04.853781+0000) 2024-03-06T13:56:05.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:05.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:05.853977+0000) 2024-03-06T13:56:06.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:06.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:06.854211+0000) 2024-03-06T13:56:07.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:07.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:07.854436+0000) 2024-03-06T13:56:08.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:08.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:08.854705+0000) 2024-03-06T13:56:09.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:09.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:09.854892+0000) 2024-03-06T13:56:10.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:10.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:10.855119+0000) 2024-03-06T13:56:11.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:11.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:11.855346+0000) 2024-03-06T13:56:12.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:12.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:12.855545+0000) 2024-03-06T13:56:13.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:13.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:13.855776+0000) 2024-03-06T13:56:14.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:14.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:14.855914+0000) 2024-03-06T13:56:15.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:15.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:15.856060+0000) 2024-03-06T13:56:16.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:16.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:16.856280+0000) 2024-03-06T13:56:17.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:17.860 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:17.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:17.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:17.856430+0000) 2024-03-06T13:56:18.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:18.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:18.856697+0000) 2024-03-06T13:56:19.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:19.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:19.856926+0000) 2024-03-06T13:56:20.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:20.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:20.857150+0000) 2024-03-06T13:56:21.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:21.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:21.857368+0000) 2024-03-06T13:56:22.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:22.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:22.857561+0000) 2024-03-06T13:56:23.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:23.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:23.857802+0000) 2024-03-06T13:56:24.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:24.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:24.858020+0000) 2024-03-06T13:56:25.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:25.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:25.858228+0000) 2024-03-06T13:56:26.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:26.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:26.858423+0000) 2024-03-06T13:56:27.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:27.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:27.858614+0000) 2024-03-06T13:56:28.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:28.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:28.858814+0000) 2024-03-06T13:56:29.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:29.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:29.859028+0000) 2024-03-06T13:56:30.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:30.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:30.859255+0000) 2024-03-06T13:56:31.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:31.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:31.859473+0000) 2024-03-06T13:56:32.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:32.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:32.859745+0000) 2024-03-06T13:56:33.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:33.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:33.860025+0000) 2024-03-06T13:56:34.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:34.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:34.860258+0000) 2024-03-06T13:56:35.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:35.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:35.860448+0000) 2024-03-06T13:56:36.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:36.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:36.860696+0000) 2024-03-06T13:56:37.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:37.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:37.860921+0000) 2024-03-06T13:56:38.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:38.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:38.861082+0000) 2024-03-06T13:56:39.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:39.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:39.861260+0000) 2024-03-06T13:56:40.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:40.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:40.861509+0000) 2024-03-06T13:56:41.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:41.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:41.861764+0000) 2024-03-06T13:56:42.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:42.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:42.862002+0000) 2024-03-06T13:56:43.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:43.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:43.862229+0000) 2024-03-06T13:56:44.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:44.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:44.862468+0000) 2024-03-06T13:56:45.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:45.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:45.862761+0000) 2024-03-06T13:56:46.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:46.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:46.863017+0000) 2024-03-06T13:56:47.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:47.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:47.863162+0000) 2024-03-06T13:56:47.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:48.000 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:48.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:48.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:48.863393+0000) 2024-03-06T13:56:49.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:49.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:49.863571+0000) 2024-03-06T13:56:50.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:50.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:50.863811+0000) 2024-03-06T13:56:51.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:51.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:51.864039+0000) 2024-03-06T13:56:52.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:52.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:52.864268+0000) 2024-03-06T13:56:53.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:53.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:53.864447+0000) 2024-03-06T13:56:54.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:54.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:54.864605+0000) 2024-03-06T13:56:55.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:55.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:55.864851+0000) 2024-03-06T13:56:56.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:56.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:56.865037+0000) 2024-03-06T13:56:57.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:57.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:57.865279+0000) 2024-03-06T13:56:58.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:58.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:58.865485+0000) 2024-03-06T13:56:59.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:56:59.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:59.865762+0000) 2024-03-06T13:57:00.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:00.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:00.866008+0000) 2024-03-06T13:57:01.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:01.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:01.866237+0000) 2024-03-06T13:57:02.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:02.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:02.866438+0000) 2024-03-06T13:57:03.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:03.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:03.866710+0000) 2024-03-06T13:57:04.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:04.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:04.866896+0000) 2024-03-06T13:57:05.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:05.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:05.867100+0000) 2024-03-06T13:57:06.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:06.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:06.867330+0000) 2024-03-06T13:57:07.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:07.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:07.867535+0000) 2024-03-06T13:57:08.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:08.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:08.867794+0000) 2024-03-06T13:57:09.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:09.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:09.868023+0000) 2024-03-06T13:57:10.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:10.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:10.868217+0000) 2024-03-06T13:57:11.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:11.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:11.868373+0000) 2024-03-06T13:57:12.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:12.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:12.868543+0000) 2024-03-06T13:57:13.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:13.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:13.868797+0000) 2024-03-06T13:57:14.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:14.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:14.869019+0000) 2024-03-06T13:57:15.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:15.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:15.869253+0000) 2024-03-06T13:57:16.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:16.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:16.869473+0000) 2024-03-06T13:57:17.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:17.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:17.869749+0000) 2024-03-06T13:57:18.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:18.204 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:18.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:18.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:18.869984+0000) 2024-03-06T13:57:19.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:19.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:19.870167+0000) 2024-03-06T13:57:20.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:20.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:20.870360+0000) 2024-03-06T13:57:21.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:21.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:21.870548+0000) 2024-03-06T13:57:22.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:22.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:22.870801+0000) 2024-03-06T13:57:23.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:23.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:23.871036+0000) 2024-03-06T13:57:24.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:24.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:24.871251+0000) 2024-03-06T13:57:25.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:25.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:25.871470+0000) 2024-03-06T13:57:26.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:26.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:26.871720+0000) 2024-03-06T13:57:27.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:27.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:27.872002+0000) 2024-03-06T13:57:28.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:28.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:28.872238+0000) 2024-03-06T13:57:29.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:29.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:29.872412+0000) 2024-03-06T13:57:30.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:30.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:30.872642+0000) 2024-03-06T13:57:31.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:31.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:31.872910+0000) 2024-03-06T13:57:32.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:32.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:32.873055+0000) 2024-03-06T13:57:33.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:33.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:33.873239+0000) 2024-03-06T13:57:34.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:34.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:34.873425+0000) 2024-03-06T13:57:35.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:35.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:35.873646+0000) 2024-03-06T13:57:36.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:36.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:36.873968+0000) 2024-03-06T13:57:37.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:37.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:37.874181+0000) 2024-03-06T13:57:38.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:38.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:38.874425+0000) 2024-03-06T13:57:39.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:39.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:39.874622+0000) 2024-03-06T13:57:40.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:40.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:40.874856+0000) 2024-03-06T13:57:41.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:41.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:41.875137+0000) 2024-03-06T13:57:42.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:42.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:42.875355+0000) 2024-03-06T13:57:43.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:43.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:43.875594+0000) 2024-03-06T13:57:44.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:44.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:44.875793+0000) 2024-03-06T13:57:45.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:45.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:45.875989+0000) 2024-03-06T13:57:46.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:46.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:46.876221+0000) 2024-03-06T13:57:47.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:47.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:47.876449+0000) 2024-03-06T13:57:48.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:48.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:48.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:48.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:48.876733+0000) 2024-03-06T13:57:49.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:49.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:49.876958+0000) 2024-03-06T13:57:50.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:50.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:50.877183+0000) 2024-03-06T13:57:51.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:51.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:51.877416+0000) 2024-03-06T13:57:52.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:52.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:52.877642+0000) 2024-03-06T13:57:53.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:53.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:53.877867+0000) 2024-03-06T13:57:54.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:54.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:54.878097+0000) 2024-03-06T13:57:55.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:55.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:55.878316+0000) 2024-03-06T13:57:56.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:56.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:56.878531+0000) 2024-03-06T13:57:57.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:57.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:57.878801+0000) 2024-03-06T13:57:58.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:58.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:58.879116+0000) 2024-03-06T13:57:59.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:57:59.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:59.879376+0000) 2024-03-06T13:58:00.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:00.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:00.879651+0000) 2024-03-06T13:58:01.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:01.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:01.879913+0000) 2024-03-06T13:58:02.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:02.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:02.880081+0000) 2024-03-06T13:58:03.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:03.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:03.880253+0000) 2024-03-06T13:58:04.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:04.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:04.880480+0000) 2024-03-06T13:58:05.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:05.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:05.880661+0000) 2024-03-06T13:58:06.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:06.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:06.880929+0000) 2024-03-06T13:58:07.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:07.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:07.881171+0000) 2024-03-06T13:58:08.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:08.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:08.881361+0000) 2024-03-06T13:58:09.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:09.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:09.881573+0000) 2024-03-06T13:58:10.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:10.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:10.881830+0000) 2024-03-06T13:58:11.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:11.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:11.882108+0000) 2024-03-06T13:58:12.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:12.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:12.882326+0000) 2024-03-06T13:58:13.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:13.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:13.882565+0000) 2024-03-06T13:58:14.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:14.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:14.882843+0000) 2024-03-06T13:58:15.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:15.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:15.883127+0000) 2024-03-06T13:58:16.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:16.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:16.883310+0000) 2024-03-06T13:58:17.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:17.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:17.883533+0000) 2024-03-06T13:58:18.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:18.580 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:18.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:18.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:18.883838+0000) 2024-03-06T13:58:19.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:19.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:19.884079+0000) 2024-03-06T13:58:20.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:20.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:20.884334+0000) 2024-03-06T13:58:21.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:21.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:21.884548+0000) 2024-03-06T13:58:22.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:22.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:22.884772+0000) 2024-03-06T13:58:23.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:23.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:23.885003+0000) 2024-03-06T13:58:24.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:24.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:24.885191+0000) 2024-03-06T13:58:25.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:25.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:25.885418+0000) 2024-03-06T13:58:26.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:26.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:26.885638+0000) 2024-03-06T13:58:27.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:27.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:27.885892+0000) 2024-03-06T13:58:28.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:28.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:28.886079+0000) 2024-03-06T13:58:29.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:29.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:29.886299+0000) 2024-03-06T13:58:30.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:30.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:30.886474+0000) 2024-03-06T13:58:31.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:31.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:31.886719+0000) 2024-03-06T13:58:32.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:32.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:32.886945+0000) 2024-03-06T13:58:33.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:33.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:33.887178+0000) 2024-03-06T13:58:34.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:34.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:34.887408+0000) 2024-03-06T13:58:35.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:35.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:35.887633+0000) 2024-03-06T13:58:36.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:36.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:36.887863+0000) 2024-03-06T13:58:37.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:37.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:37.888029+0000) 2024-03-06T13:58:38.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:38.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:38.888259+0000) 2024-03-06T13:58:39.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:39.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:39.888474+0000) 2024-03-06T13:58:40.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:40.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:40.888713+0000) 2024-03-06T13:58:41.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:41.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:41.888961+0000) 2024-03-06T13:58:42.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:42.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:42.889184+0000) 2024-03-06T13:58:43.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:43.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:43.889362+0000) 2024-03-06T13:58:44.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:44.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:44.889588+0000) 2024-03-06T13:58:45.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:45.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:45.889823+0000) 2024-03-06T13:58:46.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:46.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:46.890040+0000) 2024-03-06T13:58:47.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:47.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:47.890265+0000) 2024-03-06T13:58:48.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:48.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:48.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:48.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:48.890453+0000) 2024-03-06T13:58:49.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:49.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:49.890699+0000) 2024-03-06T13:58:50.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:50.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:50.890926+0000) 2024-03-06T13:58:51.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:51.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:51.891119+0000) 2024-03-06T13:58:52.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:52.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:52.891361+0000) 2024-03-06T13:58:53.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:53.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:53.891584+0000) 2024-03-06T13:58:54.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:54.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:54.891819+0000) 2024-03-06T13:58:55.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:55.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:55.892036+0000) 2024-03-06T13:58:56.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:56.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:56.892288+0000) 2024-03-06T13:58:57.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:57.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:57.892498+0000) 2024-03-06T13:58:58.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:58.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:58.892821+0000) 2024-03-06T13:58:59.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:58:59.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:59.893092+0000) 2024-03-06T13:59:00.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:00.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:00.893361+0000) 2024-03-06T13:59:01.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:01.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:01.893592+0000) 2024-03-06T13:59:02.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:02.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:02.893847+0000) 2024-03-06T13:59:03.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:03.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:03.894168+0000) 2024-03-06T13:59:04.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:04.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:04.894443+0000) 2024-03-06T13:59:05.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:05.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:05.894715+0000) 2024-03-06T13:59:06.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:06.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:06.894936+0000) 2024-03-06T13:59:07.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:07.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:07.895158+0000) 2024-03-06T13:59:08.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:08.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:08.895395+0000) 2024-03-06T13:59:09.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:09.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:09.895616+0000) 2024-03-06T13:59:10.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:10.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:10.895831+0000) 2024-03-06T13:59:11.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:11.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:11.896057+0000) 2024-03-06T13:59:12.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:12.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:12.896248+0000) 2024-03-06T13:59:13.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:13.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:13.896456+0000) 2024-03-06T13:59:14.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:14.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:14.896728+0000) 2024-03-06T13:59:15.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:15.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:15.896917+0000) 2024-03-06T13:59:16.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:16.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:16.897114+0000) 2024-03-06T13:59:17.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:17.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:17.897315+0000) 2024-03-06T13:59:18.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:18.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:18.897517+0000) 2024-03-06T13:59:18.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:18.948 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:19.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:19.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:19.897778+0000) 2024-03-06T13:59:20.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:20.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:20.897953+0000) 2024-03-06T13:59:21.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:21.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:21.898140+0000) 2024-03-06T13:59:22.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:22.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:22.898372+0000) 2024-03-06T13:59:23.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:23.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:23.898546+0000) 2024-03-06T13:59:24.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:24.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:24.898702+0000) 2024-03-06T13:59:25.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:25.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:25.898927+0000) 2024-03-06T13:59:26.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:26.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:26.899150+0000) 2024-03-06T13:59:27.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:27.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:27.899362+0000) 2024-03-06T13:59:28.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:28.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:28.899591+0000) 2024-03-06T13:59:29.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:29.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:29.899807+0000) 2024-03-06T13:59:30.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:30.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:30.900037+0000) 2024-03-06T13:59:31.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:31.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:31.900255+0000) 2024-03-06T13:59:32.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:32.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:32.900438+0000) 2024-03-06T13:59:33.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:33.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:33.900655+0000) 2024-03-06T13:59:34.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:34.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:34.900912+0000) 2024-03-06T13:59:35.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:35.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:35.901150+0000) 2024-03-06T13:59:36.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:36.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:36.901386+0000) 2024-03-06T13:59:37.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:37.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:37.901620+0000) 2024-03-06T13:59:38.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:38.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:38.901862+0000) 2024-03-06T13:59:39.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:39.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:39.902081+0000) 2024-03-06T13:59:40.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:40.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:40.902271+0000) 2024-03-06T13:59:41.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:41.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:41.902483+0000) 2024-03-06T13:59:42.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:42.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:42.902753+0000) 2024-03-06T13:59:43.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:43.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:43.902982+0000) 2024-03-06T13:59:44.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:44.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:44.903158+0000) 2024-03-06T13:59:45.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:45.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:45.903380+0000) 2024-03-06T13:59:46.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:46.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:46.903610+0000) 2024-03-06T13:59:47.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:47.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:47.903791+0000) 2024-03-06T13:59:48.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:48.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:48.903999+0000) 2024-03-06T13:59:49.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:49.160 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:49.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:49.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:49.904193+0000) 2024-03-06T13:59:50.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:50.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:50.904431+0000) 2024-03-06T13:59:51.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:51.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:51.904628+0000) 2024-03-06T13:59:52.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:52.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:52.904836+0000) 2024-03-06T13:59:53.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:53.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:53.905027+0000) 2024-03-06T13:59:54.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:54.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:54.905262+0000) 2024-03-06T13:59:55.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:55.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:55.905486+0000) 2024-03-06T13:59:56.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:56.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:56.905824+0000) 2024-03-06T13:59:57.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:57.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:57.906042+0000) 2024-03-06T13:59:58.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:58.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:58.906323+0000) 2024-03-06T13:59:59.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T13:59:59.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:59.906600+0000) 2024-03-06T14:00:00.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:00.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:00.906820+0000) 2024-03-06T14:00:01.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:01.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:01.907063+0000) 2024-03-06T14:00:02.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:02.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:02.907306+0000) 2024-03-06T14:00:03.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:03.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:03.907549+0000) 2024-03-06T14:00:04.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:04.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:04.907789+0000) 2024-03-06T14:00:05.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:05.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:05.908003+0000) 2024-03-06T14:00:06.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:06.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:06.908240+0000) 2024-03-06T14:00:07.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:07.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:07.908434+0000) 2024-03-06T14:00:08.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:08.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:08.908699+0000) 2024-03-06T14:00:09.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:09.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:09.908949+0000) 2024-03-06T14:00:10.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:10.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:10.909180+0000) 2024-03-06T14:00:11.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:11.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:11.909396+0000) 2024-03-06T14:00:12.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:12.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:12.909636+0000) 2024-03-06T14:00:13.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:13.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:13.909877+0000) 2024-03-06T14:00:14.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:14.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:14.910067+0000) 2024-03-06T14:00:15.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:15.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:15.910284+0000) 2024-03-06T14:00:16.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:16.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:16.910548+0000) 2024-03-06T14:00:17.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:17.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:17.910818+0000) 2024-03-06T14:00:18.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:18.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:18.911073+0000) 2024-03-06T14:00:19.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:19.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:19.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:19.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:19.911259+0000) 2024-03-06T14:00:20.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:20.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:20.911487+0000) 2024-03-06T14:00:21.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:21.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:21.911747+0000) 2024-03-06T14:00:22.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:22.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:22.911991+0000) 2024-03-06T14:00:23.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:23.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:23.912231+0000) 2024-03-06T14:00:24.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:24.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:24.912510+0000) 2024-03-06T14:00:25.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:25.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:25.912778+0000) 2024-03-06T14:00:26.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:26.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:26.913062+0000) 2024-03-06T14:00:27.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:27.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:27.913339+0000) 2024-03-06T14:00:28.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:28.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:28.913621+0000) 2024-03-06T14:00:29.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:29.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:29.913898+0000) 2024-03-06T14:00:30.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:30.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:30.914163+0000) 2024-03-06T14:00:31.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:31.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:31.914368+0000) 2024-03-06T14:00:32.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:32.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:32.914593+0000) 2024-03-06T14:00:33.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:33.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:33.914811+0000) 2024-03-06T14:00:34.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:34.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:34.915036+0000) 2024-03-06T14:00:35.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:35.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:35.915254+0000) 2024-03-06T14:00:36.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:36.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:36.915534+0000) 2024-03-06T14:00:37.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:37.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:37.915812+0000) 2024-03-06T14:00:38.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:38.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:38.916023+0000) 2024-03-06T14:00:39.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:39.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:39.916252+0000) 2024-03-06T14:00:40.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:40.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:40.916475+0000) 2024-03-06T14:00:41.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:41.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:41.916745+0000) 2024-03-06T14:00:42.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:42.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:42.916976+0000) 2024-03-06T14:00:43.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:43.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:43.917196+0000) 2024-03-06T14:00:44.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:44.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:44.917467+0000) 2024-03-06T14:00:45.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:45.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:45.917721+0000) 2024-03-06T14:00:46.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:46.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:46.917958+0000) 2024-03-06T14:00:47.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:47.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:47.918183+0000) 2024-03-06T14:00:48.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:48.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:48.918448+0000) 2024-03-06T14:00:49.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:49.713 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:49.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:49.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:49.918646+0000) 2024-03-06T14:00:50.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:50.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:50.918903+0000) 2024-03-06T14:00:51.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:51.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:51.919137+0000) 2024-03-06T14:00:52.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:52.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:52.919335+0000) 2024-03-06T14:00:53.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:53.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:53.919537+0000) 2024-03-06T14:00:54.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:54.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:54.919760+0000) 2024-03-06T14:00:55.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:55.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:55.919980+0000) 2024-03-06T14:00:56.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:56.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:56.920204+0000) 2024-03-06T14:00:57.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:57.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:57.920450+0000) 2024-03-06T14:00:58.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:58.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:58.920701+0000) 2024-03-06T14:00:59.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:00:59.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:59.920928+0000) 2024-03-06T14:01:00.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:00.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:00.921154+0000) 2024-03-06T14:01:01.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:01.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:01.921376+0000) 2024-03-06T14:01:02.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:02.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:02.921598+0000) 2024-03-06T14:01:03.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:03.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:03.921835+0000) 2024-03-06T14:01:04.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:04.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:04.922013+0000) 2024-03-06T14:01:05.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:05.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:05.922251+0000) 2024-03-06T14:01:06.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:06.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:06.922408+0000) 2024-03-06T14:01:07.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:07.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:07.922630+0000) 2024-03-06T14:01:08.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:08.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:08.922905+0000) 2024-03-06T14:01:09.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:09.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:09.923155+0000) 2024-03-06T14:01:10.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:10.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:10.923379+0000) 2024-03-06T14:01:11.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:11.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:11.923580+0000) 2024-03-06T14:01:12.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:12.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:12.923838+0000) 2024-03-06T14:01:13.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:13.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:13.924068+0000) 2024-03-06T14:01:14.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:14.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:14.924293+0000) 2024-03-06T14:01:15.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:15.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:15.924515+0000) 2024-03-06T14:01:16.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:16.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:16.924720+0000) 2024-03-06T14:01:17.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:17.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:17.924943+0000) 2024-03-06T14:01:18.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:18.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:18.925178+0000) 2024-03-06T14:01:19.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:19.918 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:19.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:19.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:19.925380+0000) 2024-03-06T14:01:20.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:20.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:20.925601+0000) 2024-03-06T14:01:21.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:21.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:21.925818+0000) 2024-03-06T14:01:22.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:22.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:22.926046+0000) 2024-03-06T14:01:23.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:23.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:23.926276+0000) 2024-03-06T14:01:24.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:24.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:24.926524+0000) 2024-03-06T14:01:25.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:25.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:25.926792+0000) 2024-03-06T14:01:26.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:26.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:26.927038+0000) 2024-03-06T14:01:27.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:27.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:27.927284+0000) 2024-03-06T14:01:28.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:28.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:28.927538+0000) 2024-03-06T14:01:29.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:29.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:29.927778+0000) 2024-03-06T14:01:30.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:30.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:30.927990+0000) 2024-03-06T14:01:31.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:31.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:31.928182+0000) 2024-03-06T14:01:32.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:32.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:32.928364+0000) 2024-03-06T14:01:33.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:33.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:33.928504+0000) 2024-03-06T14:01:34.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:34.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:34.928694+0000) 2024-03-06T14:01:35.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:35.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:35.928913+0000) 2024-03-06T14:01:36.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:36.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:36.929133+0000) 2024-03-06T14:01:37.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:37.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:37.929409+0000) 2024-03-06T14:01:38.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:38.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:38.929608+0000) 2024-03-06T14:01:39.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:39.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:39.929812+0000) 2024-03-06T14:01:40.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:40.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:40.930016+0000) 2024-03-06T14:01:41.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:41.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:41.930242+0000) 2024-03-06T14:01:42.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:42.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:42.930412+0000) 2024-03-06T14:01:43.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:43.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:43.930631+0000) 2024-03-06T14:01:44.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:44.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:44.930863+0000) 2024-03-06T14:01:45.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:45.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:45.931047+0000) 2024-03-06T14:01:46.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:46.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:46.931269+0000) 2024-03-06T14:01:47.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:47.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:47.931430+0000) 2024-03-06T14:01:48.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:48.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:48.931696+0000) 2024-03-06T14:01:49.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:49.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:49.931893+0000) 2024-03-06T14:01:50.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:50.042 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:50.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:50.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:50.932088+0000) 2024-03-06T14:01:51.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:51.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:51.932324+0000) 2024-03-06T14:01:52.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:52.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:52.932560+0000) 2024-03-06T14:01:53.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:53.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:53.932789+0000) 2024-03-06T14:01:54.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:54.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:54.932982+0000) 2024-03-06T14:01:55.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:55.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:55.933174+0000) 2024-03-06T14:01:56.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:56.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:56.933402+0000) 2024-03-06T14:01:57.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:57.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:57.933610+0000) 2024-03-06T14:01:58.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:58.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:58.933848+0000) 2024-03-06T14:01:59.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:01:59.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:59.934074+0000) 2024-03-06T14:02:00.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:00.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:00.934297+0000) 2024-03-06T14:02:01.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:01.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:01.934504+0000) 2024-03-06T14:02:02.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:02.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:02.934755+0000) 2024-03-06T14:02:03.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:03.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:03.934963+0000) 2024-03-06T14:02:04.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:04.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:04.935191+0000) 2024-03-06T14:02:05.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:05.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:05.935380+0000) 2024-03-06T14:02:06.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:06.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:06.935570+0000) 2024-03-06T14:02:07.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:07.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:07.935819+0000) 2024-03-06T14:02:08.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:08.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:08.936069+0000) 2024-03-06T14:02:09.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:09.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:09.936315+0000) 2024-03-06T14:02:10.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:10.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:10.936517+0000) 2024-03-06T14:02:11.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:11.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:11.936747+0000) 2024-03-06T14:02:12.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:12.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:12.937007+0000) 2024-03-06T14:02:13.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:13.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:13.937200+0000) 2024-03-06T14:02:14.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:14.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:14.937413+0000) 2024-03-06T14:02:15.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:15.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:15.937632+0000) 2024-03-06T14:02:16.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:16.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:16.937872+0000) 2024-03-06T14:02:17.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:17.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:17.938097+0000) 2024-03-06T14:02:18.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:18.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:18.938347+0000) 2024-03-06T14:02:19.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:19.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:19.938572+0000) 2024-03-06T14:02:20.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:20.237 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:20.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:20.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:20.938810+0000) 2024-03-06T14:02:21.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:21.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:21.938953+0000) 2024-03-06T14:02:22.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:22.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:22.939172+0000) 2024-03-06T14:02:23.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:23.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:23.939413+0000) 2024-03-06T14:02:24.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:24.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:24.939649+0000) 2024-03-06T14:02:25.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:25.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:25.939974+0000) 2024-03-06T14:02:26.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:26.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:26.940172+0000) 2024-03-06T14:02:27.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:27.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:27.940444+0000) 2024-03-06T14:02:28.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:28.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:28.940747+0000) 2024-03-06T14:02:29.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:29.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:29.940985+0000) 2024-03-06T14:02:30.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:30.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:30.941230+0000) 2024-03-06T14:02:31.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:31.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:31.941475+0000) 2024-03-06T14:02:32.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:32.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:32.941710+0000) 2024-03-06T14:02:33.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:33.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:33.941938+0000) 2024-03-06T14:02:34.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:34.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:34.942175+0000) 2024-03-06T14:02:35.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:35.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:35.942399+0000) 2024-03-06T14:02:36.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:36.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:36.942580+0000) 2024-03-06T14:02:37.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:37.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:37.942827+0000) 2024-03-06T14:02:38.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:38.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:38.943004+0000) 2024-03-06T14:02:39.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:39.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:39.943214+0000) 2024-03-06T14:02:40.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:40.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:40.943376+0000) 2024-03-06T14:02:41.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:41.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:41.943606+0000) 2024-03-06T14:02:42.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:42.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:42.943830+0000) 2024-03-06T14:02:43.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:43.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:43.944044+0000) 2024-03-06T14:02:44.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:44.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:44.944276+0000) 2024-03-06T14:02:45.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:45.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:45.944465+0000) 2024-03-06T14:02:46.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:46.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:46.944745+0000) 2024-03-06T14:02:47.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:47.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:47.944957+0000) 2024-03-06T14:02:48.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:48.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:48.945125+0000) 2024-03-06T14:02:49.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:49.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:49.945343+0000) 2024-03-06T14:02:50.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:50.470 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:50.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:50.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:50.945567+0000) 2024-03-06T14:02:51.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:51.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:51.945806+0000) 2024-03-06T14:02:52.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:52.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:52.946026+0000) 2024-03-06T14:02:53.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:53.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:53.946260+0000) 2024-03-06T14:02:54.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:54.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:54.946468+0000) 2024-03-06T14:02:55.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:55.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:55.946657+0000) 2024-03-06T14:02:56.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:56.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:56.946893+0000) 2024-03-06T14:02:57.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:57.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:57.947097+0000) 2024-03-06T14:02:58.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:58.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:58.947333+0000) 2024-03-06T14:02:59.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:02:59.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:59.947505+0000) 2024-03-06T14:03:00.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:00.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:00.947765+0000) 2024-03-06T14:03:01.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:01.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:01.947995+0000) 2024-03-06T14:03:02.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:02.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:02.948219+0000) 2024-03-06T14:03:03.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:03.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:03.948441+0000) 2024-03-06T14:03:04.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:04.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:04.948661+0000) 2024-03-06T14:03:05.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:05.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:05.948930+0000) 2024-03-06T14:03:06.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:06.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:06.949071+0000) 2024-03-06T14:03:07.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:07.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:07.949289+0000) 2024-03-06T14:03:08.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:08.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:08.949489+0000) 2024-03-06T14:03:09.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:09.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:09.949778+0000) 2024-03-06T14:03:10.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:10.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:10.949984+0000) 2024-03-06T14:03:11.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:11.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:11.950216+0000) 2024-03-06T14:03:12.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:12.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:12.950448+0000) 2024-03-06T14:03:13.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:13.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:13.950708+0000) 2024-03-06T14:03:14.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:14.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:14.950923+0000) 2024-03-06T14:03:15.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:15.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:15.951144+0000) 2024-03-06T14:03:16.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:16.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:16.951366+0000) 2024-03-06T14:03:17.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:17.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:17.951549+0000) 2024-03-06T14:03:18.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:18.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:18.951787+0000) 2024-03-06T14:03:19.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:19.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:19.951979+0000) 2024-03-06T14:03:20.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:20.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:20.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:20.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:20.952200+0000) 2024-03-06T14:03:21.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:21.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:21.952421+0000) 2024-03-06T14:03:22.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:22.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:22.952647+0000) 2024-03-06T14:03:23.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:23.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:23.952922+0000) 2024-03-06T14:03:24.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:24.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:24.953171+0000) 2024-03-06T14:03:25.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:25.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:25.953409+0000) 2024-03-06T14:03:26.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:26.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:26.953642+0000) 2024-03-06T14:03:27.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:27.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:27.953898+0000) 2024-03-06T14:03:28.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:28.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:28.954100+0000) 2024-03-06T14:03:29.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:29.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:29.954339+0000) 2024-03-06T14:03:30.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:30.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:30.954539+0000) 2024-03-06T14:03:31.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:31.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:31.954720+0000) 2024-03-06T14:03:32.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:32.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:32.954914+0000) 2024-03-06T14:03:33.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:33.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:33.955132+0000) 2024-03-06T14:03:34.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:34.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:34.955336+0000) 2024-03-06T14:03:35.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:35.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:35.955574+0000) 2024-03-06T14:03:36.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:36.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:36.955826+0000) 2024-03-06T14:03:37.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:37.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:37.956049+0000) 2024-03-06T14:03:38.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:38.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:38.956296+0000) 2024-03-06T14:03:39.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:39.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:39.956485+0000) 2024-03-06T14:03:40.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:40.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:40.956655+0000) 2024-03-06T14:03:41.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:41.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:41.956897+0000) 2024-03-06T14:03:42.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:42.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:42.957123+0000) 2024-03-06T14:03:43.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:43.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:43.957297+0000) 2024-03-06T14:03:44.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:44.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:44.957538+0000) 2024-03-06T14:03:45.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:45.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:45.957755+0000) 2024-03-06T14:03:46.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:46.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:46.957979+0000) 2024-03-06T14:03:47.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:47.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:47.958165+0000) 2024-03-06T14:03:48.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:48.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:48.958427+0000) 2024-03-06T14:03:49.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:49.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:49.958654+0000) 2024-03-06T14:03:50.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:50.886 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:50.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:50.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:50.958896+0000) 2024-03-06T14:03:51.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:51.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:51.959111+0000) 2024-03-06T14:03:52.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:52.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:52.959340+0000) 2024-03-06T14:03:53.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:53.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:53.959572+0000) 2024-03-06T14:03:54.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:54.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:54.959812+0000) 2024-03-06T14:03:55.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:55.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:55.960047+0000) 2024-03-06T14:03:56.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:56.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:56.960278+0000) 2024-03-06T14:03:57.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:57.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:57.960502+0000) 2024-03-06T14:03:58.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:58.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:58.960769+0000) 2024-03-06T14:03:59.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:03:59.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:59.960988+0000) 2024-03-06T14:04:00.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:00.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:00.961179+0000) 2024-03-06T14:04:01.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:01.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:01.961378+0000) 2024-03-06T14:04:02.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:02.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:02.961626+0000) 2024-03-06T14:04:03.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:03.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:03.961872+0000) 2024-03-06T14:04:04.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:04.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:04.962119+0000) 2024-03-06T14:04:05.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:05.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:05.962319+0000) 2024-03-06T14:04:06.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:06.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:06.962566+0000) 2024-03-06T14:04:07.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:07.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:07.962861+0000) 2024-03-06T14:04:08.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:08.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:08.963111+0000) 2024-03-06T14:04:09.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:09.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:09.963348+0000) 2024-03-06T14:04:10.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:10.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:10.963591+0000) 2024-03-06T14:04:11.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:11.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:11.963825+0000) 2024-03-06T14:04:12.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:12.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:12.964067+0000) 2024-03-06T14:04:13.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:13.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:13.964269+0000) 2024-03-06T14:04:14.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:14.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:14.964491+0000) 2024-03-06T14:04:15.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:15.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:15.964776+0000) 2024-03-06T14:04:16.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:16.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:16.965001+0000) 2024-03-06T14:04:17.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:17.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:17.965247+0000) 2024-03-06T14:04:18.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:18.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:18.965496+0000) 2024-03-06T14:04:19.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:19.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:19.965709+0000) 2024-03-06T14:04:20.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:20.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:20.965955+0000) 2024-03-06T14:04:21.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:21.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:21.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:21.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:21.966150+0000) 2024-03-06T14:04:22.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:22.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:22.966346+0000) 2024-03-06T14:04:23.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:23.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:23.966530+0000) 2024-03-06T14:04:24.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:24.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:24.966782+0000) 2024-03-06T14:04:25.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:25.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:25.967022+0000) 2024-03-06T14:04:26.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:26.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:26.967253+0000) 2024-03-06T14:04:27.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:27.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:27.967492+0000) 2024-03-06T14:04:28.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:28.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:28.967788+0000) 2024-03-06T14:04:29.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:29.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:29.967947+0000) 2024-03-06T14:04:30.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:30.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:30.968171+0000) 2024-03-06T14:04:31.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:31.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:31.968393+0000) 2024-03-06T14:04:32.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:32.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:32.968604+0000) 2024-03-06T14:04:33.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:33.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:33.968845+0000) 2024-03-06T14:04:34.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:34.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:34.969069+0000) 2024-03-06T14:04:35.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:35.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:35.969295+0000) 2024-03-06T14:04:36.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:36.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:36.969518+0000) 2024-03-06T14:04:37.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:37.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:37.969795+0000) 2024-03-06T14:04:38.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:38.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:38.970031+0000) 2024-03-06T14:04:39.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:39.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:39.970253+0000) 2024-03-06T14:04:40.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:40.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:40.970467+0000) 2024-03-06T14:04:41.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:41.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:41.970776+0000) 2024-03-06T14:04:42.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:42.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:42.971044+0000) 2024-03-06T14:04:43.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:43.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:43.971313+0000) 2024-03-06T14:04:44.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:44.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:44.971525+0000) 2024-03-06T14:04:45.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:45.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:45.971800+0000) 2024-03-06T14:04:46.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:46.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:46.972048+0000) 2024-03-06T14:04:47.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:47.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:47.972285+0000) 2024-03-06T14:04:48.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:48.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:48.972508+0000) 2024-03-06T14:04:49.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:49.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:49.972783+0000) 2024-03-06T14:04:50.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:50.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:50.973006+0000) 2024-03-06T14:04:51.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:51.226 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:51.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:51.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:51.973230+0000) 2024-03-06T14:04:52.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:52.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:52.973472+0000) 2024-03-06T14:04:53.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:53.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:53.973754+0000) 2024-03-06T14:04:54.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:54.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:54.973965+0000) 2024-03-06T14:04:55.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:55.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:55.974253+0000) 2024-03-06T14:04:56.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:56.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:56.974488+0000) 2024-03-06T14:04:57.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:57.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:57.974780+0000) 2024-03-06T14:04:58.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:58.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:58.975039+0000) 2024-03-06T14:04:59.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:04:59.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:59.975276+0000) 2024-03-06T14:05:00.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:00.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:00.975496+0000) 2024-03-06T14:05:01.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:01.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:01.975744+0000) 2024-03-06T14:05:02.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:02.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:02.975969+0000) 2024-03-06T14:05:03.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:03.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:03.976164+0000) 2024-03-06T14:05:04.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:04.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:04.976388+0000) 2024-03-06T14:05:05.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:05.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:05.976618+0000) 2024-03-06T14:05:06.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:06.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:06.976887+0000) 2024-03-06T14:05:07.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:07.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:07.977074+0000) 2024-03-06T14:05:08.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:08.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:08.977279+0000) 2024-03-06T14:05:09.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:09.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:09.977483+0000) 2024-03-06T14:05:10.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:10.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:10.977755+0000) 2024-03-06T14:05:11.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:11.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:11.978023+0000) 2024-03-06T14:05:12.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:12.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:12.978292+0000) 2024-03-06T14:05:13.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:13.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:13.978495+0000) 2024-03-06T14:05:14.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:14.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:14.978774+0000) 2024-03-06T14:05:15.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:15.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:15.978975+0000) 2024-03-06T14:05:16.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:16.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:16.979209+0000) 2024-03-06T14:05:17.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:17.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:17.979443+0000) 2024-03-06T14:05:18.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:18.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:18.979636+0000) 2024-03-06T14:05:19.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:19.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:19.979871+0000) 2024-03-06T14:05:20.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:20.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:20.980046+0000) 2024-03-06T14:05:21.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:21.464 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:21.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:21.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:21.980267+0000) 2024-03-06T14:05:22.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:22.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:22.980482+0000) 2024-03-06T14:05:23.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:23.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:23.980746+0000) 2024-03-06T14:05:24.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:24.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:24.980975+0000) 2024-03-06T14:05:25.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:25.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:25.981211+0000) 2024-03-06T14:05:26.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:26.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:26.981422+0000) 2024-03-06T14:05:27.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:27.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:27.981645+0000) 2024-03-06T14:05:28.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:28.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:28.981926+0000) 2024-03-06T14:05:29.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:29.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:29.982157+0000) 2024-03-06T14:05:30.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:30.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:30.982335+0000) 2024-03-06T14:05:31.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:31.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:31.982565+0000) 2024-03-06T14:05:32.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:32.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:32.982799+0000) 2024-03-06T14:05:33.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:33.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:33.983028+0000) 2024-03-06T14:05:34.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:34.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:34.983253+0000) 2024-03-06T14:05:35.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:35.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:35.983464+0000) 2024-03-06T14:05:36.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:36.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:36.983704+0000) 2024-03-06T14:05:37.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:37.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:37.983891+0000) 2024-03-06T14:05:38.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:38.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:38.984071+0000) 2024-03-06T14:05:39.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:39.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:39.984253+0000) 2024-03-06T14:05:40.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:40.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:40.984489+0000) 2024-03-06T14:05:41.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:41.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:41.984767+0000) 2024-03-06T14:05:42.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:42.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:42.984996+0000) 2024-03-06T14:05:43.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:43.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:43.985296+0000) 2024-03-06T14:05:44.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:44.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:44.985553+0000) 2024-03-06T14:05:45.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:45.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:45.985808+0000) 2024-03-06T14:05:46.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:46.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:46.986052+0000) 2024-03-06T14:05:47.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:47.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:47.986291+0000) 2024-03-06T14:05:48.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:48.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:48.986532+0000) 2024-03-06T14:05:49.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:49.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:49.986776+0000) 2024-03-06T14:05:50.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:50.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:50.987016+0000) 2024-03-06T14:05:51.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:51.686 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:51.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:51.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:51.987218+0000) 2024-03-06T14:05:52.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:52.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:52.987449+0000) 2024-03-06T14:05:53.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:53.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:53.987645+0000) 2024-03-06T14:05:54.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:54.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:54.987898+0000) 2024-03-06T14:05:55.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:55.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:55.988144+0000) 2024-03-06T14:05:56.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:56.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:56.988366+0000) 2024-03-06T14:05:57.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:57.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:57.988595+0000) 2024-03-06T14:05:58.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:58.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:58.988866+0000) 2024-03-06T14:05:59.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:05:59.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:59.989172+0000) 2024-03-06T14:06:00.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:00.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:00.989446+0000) 2024-03-06T14:06:01.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:01.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:01.989650+0000) 2024-03-06T14:06:02.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:02.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:02.989907+0000) 2024-03-06T14:06:03.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:03.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:03.990122+0000) 2024-03-06T14:06:04.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:04.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:04.990351+0000) 2024-03-06T14:06:05.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:05.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:05.990573+0000) 2024-03-06T14:06:06.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:06.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:06.990801+0000) 2024-03-06T14:06:07.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:07.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:07.991047+0000) 2024-03-06T14:06:08.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:08.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:08.991250+0000) 2024-03-06T14:06:09.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:09.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:09.991501+0000) 2024-03-06T14:06:10.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:10.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:10.991778+0000) 2024-03-06T14:06:11.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:11.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:11.991989+0000) 2024-03-06T14:06:12.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:12.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:12.992190+0000) 2024-03-06T14:06:13.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:13.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:13.992413+0000) 2024-03-06T14:06:14.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:14.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:14.992645+0000) 2024-03-06T14:06:15.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:15.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:15.992931+0000) 2024-03-06T14:06:16.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:16.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:16.993153+0000) 2024-03-06T14:06:17.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:17.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:17.993372+0000) 2024-03-06T14:06:18.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:18.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:18.993592+0000) 2024-03-06T14:06:19.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:19.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:19.993801+0000) 2024-03-06T14:06:20.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:20.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:20.994037+0000) 2024-03-06T14:06:21.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:21.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:21.994261+0000) 2024-03-06T14:06:22.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:22.032 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:22.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:22.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:22.994511+0000) 2024-03-06T14:06:23.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:23.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:23.994788+0000) 2024-03-06T14:06:24.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:24.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:24.994992+0000) 2024-03-06T14:06:25.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:25.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:25.995156+0000) 2024-03-06T14:06:26.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:26.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:26.995386+0000) 2024-03-06T14:06:27.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:27.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:27.995634+0000) 2024-03-06T14:06:28.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:28.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:28.995893+0000) 2024-03-06T14:06:29.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:29.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:29.996104+0000) 2024-03-06T14:06:30.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:30.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:30.996339+0000) 2024-03-06T14:06:31.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:31.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:31.996568+0000) 2024-03-06T14:06:32.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:32.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:32.996811+0000) 2024-03-06T14:06:33.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:33.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:33.997021+0000) 2024-03-06T14:06:34.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:34.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:34.997242+0000) 2024-03-06T14:06:35.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:35.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:35.997433+0000) 2024-03-06T14:06:36.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:36.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:36.997719+0000) 2024-03-06T14:06:37.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:37.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:37.997940+0000) 2024-03-06T14:06:38.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:38.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:38.998190+0000) 2024-03-06T14:06:39.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:39.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:39.998421+0000) 2024-03-06T14:06:40.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:40.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:40.998708+0000) 2024-03-06T14:06:41.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:41.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:41.998908+0000) 2024-03-06T14:06:42.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:42.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:42.999158+0000) 2024-03-06T14:06:43.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:43.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:43.999367+0000) 2024-03-06T14:06:44.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:44.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:44.999556+0000) 2024-03-06T14:06:45.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:45.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:45.999771+0000) 2024-03-06T14:06:46.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:46.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:46.999988+0000) 2024-03-06T14:06:47.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:47.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:48.000222+0000) 2024-03-06T14:06:48.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:48.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:49.000415+0000) 2024-03-06T14:06:49.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:49.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:50.000640+0000) 2024-03-06T14:06:50.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:51.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:51.000904+0000) 2024-03-06T14:06:51.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:52.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:52.001088+0000) 2024-03-06T14:06:52.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:52.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:52.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:53.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:53.001317+0000) 2024-03-06T14:06:53.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:54.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:54.001506+0000) 2024-03-06T14:06:55.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:55.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:55.001780+0000) 2024-03-06T14:06:56.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:56.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:56.001976+0000) 2024-03-06T14:06:57.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:57.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:57.002193+0000) 2024-03-06T14:06:58.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:58.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:58.002422+0000) 2024-03-06T14:06:59.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:06:59.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:59.002627+0000) 2024-03-06T14:07:00.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:00.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:00.002838+0000) 2024-03-06T14:07:01.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:01.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:01.003050+0000) 2024-03-06T14:07:02.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:02.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:02.003272+0000) 2024-03-06T14:07:03.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:03.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:03.003494+0000) 2024-03-06T14:07:04.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:04.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:04.003749+0000) 2024-03-06T14:07:05.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:05.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:05.003977+0000) 2024-03-06T14:07:06.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:06.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:06.004207+0000) 2024-03-06T14:07:07.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:07.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:07.004434+0000) 2024-03-06T14:07:08.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:08.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:08.004620+0000) 2024-03-06T14:07:09.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:09.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:09.004822+0000) 2024-03-06T14:07:10.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:10.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:10.005035+0000) 2024-03-06T14:07:11.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:11.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:11.005256+0000) 2024-03-06T14:07:12.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:12.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:12.005480+0000) 2024-03-06T14:07:13.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:13.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:13.005699+0000) 2024-03-06T14:07:14.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:14.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:14.005892+0000) 2024-03-06T14:07:15.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:15.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:15.006098+0000) 2024-03-06T14:07:16.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:16.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:16.006371+0000) 2024-03-06T14:07:17.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:17.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:17.006620+0000) 2024-03-06T14:07:18.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:18.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:18.006855+0000) 2024-03-06T14:07:19.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:19.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:19.007046+0000) 2024-03-06T14:07:20.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:20.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:20.007267+0000) 2024-03-06T14:07:21.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:21.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:21.007489+0000) 2024-03-06T14:07:22.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:22.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:22.007763+0000) 2024-03-06T14:07:22.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:22.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:23.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:23.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:23.007968+0000) 2024-03-06T14:07:24.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:24.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:24.008180+0000) 2024-03-06T14:07:25.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:25.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:25.008458+0000) 2024-03-06T14:07:26.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:26.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:26.008705+0000) 2024-03-06T14:07:27.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:27.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:27.008883+0000) 2024-03-06T14:07:28.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:28.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:28.009123+0000) 2024-03-06T14:07:29.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:29.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:29.009374+0000) 2024-03-06T14:07:30.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:30.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:30.009577+0000) 2024-03-06T14:07:31.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:31.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:31.009809+0000) 2024-03-06T14:07:32.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:32.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:32.010048+0000) 2024-03-06T14:07:33.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:33.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:33.010282+0000) 2024-03-06T14:07:34.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:34.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:34.010559+0000) 2024-03-06T14:07:35.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:35.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:35.010780+0000) 2024-03-06T14:07:36.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:36.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:36.010988+0000) 2024-03-06T14:07:37.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:37.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:37.011209+0000) 2024-03-06T14:07:38.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:38.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:38.011393+0000) 2024-03-06T14:07:39.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:39.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:39.011647+0000) 2024-03-06T14:07:40.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:40.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:40.011877+0000) 2024-03-06T14:07:41.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:41.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:41.012100+0000) 2024-03-06T14:07:42.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:42.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:42.012318+0000) 2024-03-06T14:07:43.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:43.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:43.012532+0000) 2024-03-06T14:07:44.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:44.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:44.012807+0000) 2024-03-06T14:07:45.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:45.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:45.013024+0000) 2024-03-06T14:07:46.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:46.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:46.013249+0000) 2024-03-06T14:07:47.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:47.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:47.013504+0000) 2024-03-06T14:07:48.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:48.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:48.013775+0000) 2024-03-06T14:07:49.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:49.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:49.014017+0000) 2024-03-06T14:07:50.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:50.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:50.014256+0000) 2024-03-06T14:07:51.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:51.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:51.014529+0000) 2024-03-06T14:07:52.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:52.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:52.014807+0000) 2024-03-06T14:07:52.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:52.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:53.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:53.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:53.015067+0000) 2024-03-06T14:07:54.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:54.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:54.015335+0000) 2024-03-06T14:07:55.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:55.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:55.015604+0000) 2024-03-06T14:07:56.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:56.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:56.015881+0000) 2024-03-06T14:07:57.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:57.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:57.016095+0000) 2024-03-06T14:07:58.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:58.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:58.016317+0000) 2024-03-06T14:07:59.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:07:59.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:59.016562+0000) 2024-03-06T14:08:00.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:00.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:00.016786+0000) 2024-03-06T14:08:01.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:01.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:01.016993+0000) 2024-03-06T14:08:02.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:02.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:02.017219+0000) 2024-03-06T14:08:03.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:03.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:03.017413+0000) 2024-03-06T14:08:04.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:04.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:04.017646+0000) 2024-03-06T14:08:05.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:05.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:05.017886+0000) 2024-03-06T14:08:06.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:06.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:06.018085+0000) 2024-03-06T14:08:07.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:07.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:07.018309+0000) 2024-03-06T14:08:08.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:08.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:08.018551+0000) 2024-03-06T14:08:09.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:09.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:09.018795+0000) 2024-03-06T14:08:10.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:10.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:10.019020+0000) 2024-03-06T14:08:11.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:11.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:11.019240+0000) 2024-03-06T14:08:12.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:12.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:12.019458+0000) 2024-03-06T14:08:13.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:13.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:13.019778+0000) 2024-03-06T14:08:14.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:14.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:14.020005+0000) 2024-03-06T14:08:15.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:15.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:15.020139+0000) 2024-03-06T14:08:16.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:16.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:16.020353+0000) 2024-03-06T14:08:17.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:17.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:17.020573+0000) 2024-03-06T14:08:18.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:18.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:18.020805+0000) 2024-03-06T14:08:19.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:19.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:19.020994+0000) 2024-03-06T14:08:20.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:20.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:20.021219+0000) 2024-03-06T14:08:21.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:21.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:21.021441+0000) 2024-03-06T14:08:22.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:22.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:22.021632+0000) 2024-03-06T14:08:22.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:22.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:23.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:23.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:23.021870+0000) 2024-03-06T14:08:24.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:24.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:24.022097+0000) 2024-03-06T14:08:25.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:25.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:25.022324+0000) 2024-03-06T14:08:26.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:26.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:26.022546+0000) 2024-03-06T14:08:27.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:27.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:27.022772+0000) 2024-03-06T14:08:28.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:28.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:28.022988+0000) 2024-03-06T14:08:29.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:29.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:29.023185+0000) 2024-03-06T14:08:30.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:30.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:30.023366+0000) 2024-03-06T14:08:31.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:31.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:31.023596+0000) 2024-03-06T14:08:32.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:32.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:32.023838+0000) 2024-03-06T14:08:33.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:33.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:33.024067+0000) 2024-03-06T14:08:34.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:34.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:34.024269+0000) 2024-03-06T14:08:35.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:35.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:35.024504+0000) 2024-03-06T14:08:36.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:36.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:36.024773+0000) 2024-03-06T14:08:37.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:37.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:37.024998+0000) 2024-03-06T14:08:38.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:38.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:38.025228+0000) 2024-03-06T14:08:39.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:39.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:39.025471+0000) 2024-03-06T14:08:40.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:40.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:40.025756+0000) 2024-03-06T14:08:41.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:41.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:41.025983+0000) 2024-03-06T14:08:42.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:42.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:42.026125+0000) 2024-03-06T14:08:43.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:43.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:43.026338+0000) 2024-03-06T14:08:44.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:44.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:44.026565+0000) 2024-03-06T14:08:45.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:45.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:45.026789+0000) 2024-03-06T14:08:46.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:46.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:46.027007+0000) 2024-03-06T14:08:47.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:47.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:47.027229+0000) 2024-03-06T14:08:48.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:48.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:48.027445+0000) 2024-03-06T14:08:49.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:49.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:49.027706+0000) 2024-03-06T14:08:50.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:50.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:50.027925+0000) 2024-03-06T14:08:51.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:51.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:51.028110+0000) 2024-03-06T14:08:52.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:52.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:52.028320+0000) 2024-03-06T14:08:53.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:53.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:53.028543+0000) 2024-03-06T14:08:53.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:53.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:54.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:54.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:54.028779+0000) 2024-03-06T14:08:55.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:55.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:55.028971+0000) 2024-03-06T14:08:56.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:56.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:56.029156+0000) 2024-03-06T14:08:57.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:57.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:57.029394+0000) 2024-03-06T14:08:58.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:58.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:58.029633+0000) 2024-03-06T14:08:59.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:08:59.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:59.029924+0000) 2024-03-06T14:09:00.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:00.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:00.030168+0000) 2024-03-06T14:09:01.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:01.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:01.030413+0000) 2024-03-06T14:09:02.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:02.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:02.030604+0000) 2024-03-06T14:09:03.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:03.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:03.030814+0000) 2024-03-06T14:09:04.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:04.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:04.031054+0000) 2024-03-06T14:09:05.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:05.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:05.031276+0000) 2024-03-06T14:09:06.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:06.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:06.031559+0000) 2024-03-06T14:09:07.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:07.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:07.031787+0000) 2024-03-06T14:09:08.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:08.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:08.031984+0000) 2024-03-06T14:09:09.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:09.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:09.032195+0000) 2024-03-06T14:09:10.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:10.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:10.032385+0000) 2024-03-06T14:09:11.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:11.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:11.032650+0000) 2024-03-06T14:09:12.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:12.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:12.032884+0000) 2024-03-06T14:09:13.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:13.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:13.033120+0000) 2024-03-06T14:09:14.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:14.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:14.033346+0000) 2024-03-06T14:09:15.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:15.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:15.033567+0000) 2024-03-06T14:09:16.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:16.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:16.033784+0000) 2024-03-06T14:09:17.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:17.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:17.033977+0000) 2024-03-06T14:09:18.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:18.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:18.034155+0000) 2024-03-06T14:09:19.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:19.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:19.034381+0000) 2024-03-06T14:09:20.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:20.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:20.034600+0000) 2024-03-06T14:09:21.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:21.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:21.034840+0000) 2024-03-06T14:09:22.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:22.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:22.035021+0000) 2024-03-06T14:09:23.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:23.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:23.035213+0000) 2024-03-06T14:09:23.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:23.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:24.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:24.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:24.035460+0000) 2024-03-06T14:09:25.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:25.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:25.035768+0000) 2024-03-06T14:09:26.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:26.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:26.036045+0000) 2024-03-06T14:09:27.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:27.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:27.036255+0000) 2024-03-06T14:09:28.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:28.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:28.036496+0000) 2024-03-06T14:09:29.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:29.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:29.036783+0000) 2024-03-06T14:09:30.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:30.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:30.037017+0000) 2024-03-06T14:09:31.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:31.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:31.037245+0000) 2024-03-06T14:09:32.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:32.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:32.037424+0000) 2024-03-06T14:09:33.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:33.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:33.037697+0000) 2024-03-06T14:09:34.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:34.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:34.037869+0000) 2024-03-06T14:09:35.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:35.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:35.038003+0000) 2024-03-06T14:09:36.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:36.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:36.038182+0000) 2024-03-06T14:09:37.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:37.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:37.038365+0000) 2024-03-06T14:09:38.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:38.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:38.038648+0000) 2024-03-06T14:09:39.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:39.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:39.038977+0000) 2024-03-06T14:09:40.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:40.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:40.039246+0000) 2024-03-06T14:09:41.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:41.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:41.039516+0000) 2024-03-06T14:09:42.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:42.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:42.039832+0000) 2024-03-06T14:09:43.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:43.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:43.040117+0000) 2024-03-06T14:09:44.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:44.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:44.040391+0000) 2024-03-06T14:09:45.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:45.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:45.040599+0000) 2024-03-06T14:09:46.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:46.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:46.040874+0000) 2024-03-06T14:09:47.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:47.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:47.041122+0000) 2024-03-06T14:09:48.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:48.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:48.041354+0000) 2024-03-06T14:09:49.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:49.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:49.041650+0000) 2024-03-06T14:09:50.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:50.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:50.041971+0000) 2024-03-06T14:09:51.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:51.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:51.042246+0000) 2024-03-06T14:09:52.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:52.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:52.042443+0000) 2024-03-06T14:09:53.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:53.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:53.042648+0000) 2024-03-06T14:09:53.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:53.490 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:54.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:54.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:54.042947+0000) 2024-03-06T14:09:55.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:55.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:55.043131+0000) 2024-03-06T14:09:56.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:56.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:56.043352+0000) 2024-03-06T14:09:57.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:57.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:57.043544+0000) 2024-03-06T14:09:58.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:58.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:58.043761+0000) 2024-03-06T14:09:59.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:09:59.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:59.043992+0000) 2024-03-06T14:10:00.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:00.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:00.044230+0000) 2024-03-06T14:10:01.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:01.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:01.044458+0000) 2024-03-06T14:10:02.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:02.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:02.044706+0000) 2024-03-06T14:10:03.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:03.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:03.044944+0000) 2024-03-06T14:10:04.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:04.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:04.045124+0000) 2024-03-06T14:10:05.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:05.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:05.045313+0000) 2024-03-06T14:10:06.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:06.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:06.045514+0000) 2024-03-06T14:10:07.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:07.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:07.045787+0000) 2024-03-06T14:10:08.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:08.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:08.046002+0000) 2024-03-06T14:10:09.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:09.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:09.046200+0000) 2024-03-06T14:10:10.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:10.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:10.046389+0000) 2024-03-06T14:10:11.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:11.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:11.046593+0000) 2024-03-06T14:10:12.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:12.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:12.046824+0000) 2024-03-06T14:10:13.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:13.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:13.047055+0000) 2024-03-06T14:10:14.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:14.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:14.047280+0000) 2024-03-06T14:10:15.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:15.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:15.047543+0000) 2024-03-06T14:10:16.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:16.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:16.047795+0000) 2024-03-06T14:10:17.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:17.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:17.047992+0000) 2024-03-06T14:10:18.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:18.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:18.048203+0000) 2024-03-06T14:10:19.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:19.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:19.048443+0000) 2024-03-06T14:10:20.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:20.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:20.048721+0000) 2024-03-06T14:10:21.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:21.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:21.048989+0000) 2024-03-06T14:10:22.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:22.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:22.049254+0000) 2024-03-06T14:10:23.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:23.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:23.049509+0000) 2024-03-06T14:10:23.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:23.706 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:24.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:24.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:24.049793+0000) 2024-03-06T14:10:25.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:25.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:25.050066+0000) 2024-03-06T14:10:26.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:26.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:26.050339+0000) 2024-03-06T14:10:27.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:27.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:27.050621+0000) 2024-03-06T14:10:28.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:28.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:28.050897+0000) 2024-03-06T14:10:29.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:29.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:29.051150+0000) 2024-03-06T14:10:30.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:30.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:30.051384+0000) 2024-03-06T14:10:31.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:31.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:31.051656+0000) 2024-03-06T14:10:32.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:32.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:32.051964+0000) 2024-03-06T14:10:33.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:33.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:33.052226+0000) 2024-03-06T14:10:34.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:34.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:34.052426+0000) 2024-03-06T14:10:35.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:35.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:35.052648+0000) 2024-03-06T14:10:36.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:36.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:36.052927+0000) 2024-03-06T14:10:37.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:37.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:37.053104+0000) 2024-03-06T14:10:38.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:38.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:38.053321+0000) 2024-03-06T14:10:39.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:39.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:39.053549+0000) 2024-03-06T14:10:40.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:40.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:40.053772+0000) 2024-03-06T14:10:41.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:41.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:41.053993+0000) 2024-03-06T14:10:42.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:42.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:42.054216+0000) 2024-03-06T14:10:43.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:43.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:43.054441+0000) 2024-03-06T14:10:44.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:44.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:44.054610+0000) 2024-03-06T14:10:45.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:45.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:45.054833+0000) 2024-03-06T14:10:46.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:46.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:46.055019+0000) 2024-03-06T14:10:47.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:47.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:47.055199+0000) 2024-03-06T14:10:48.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:48.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:48.055477+0000) 2024-03-06T14:10:49.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:49.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:49.055769+0000) 2024-03-06T14:10:50.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:50.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:50.055990+0000) 2024-03-06T14:10:51.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:51.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:51.056271+0000) 2024-03-06T14:10:52.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:52.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:52.056548+0000) 2024-03-06T14:10:53.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:53.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:53.056818+0000) 2024-03-06T14:10:53.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:53.927 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:54.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:54.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:54.057086+0000) 2024-03-06T14:10:55.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:55.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:55.057361+0000) 2024-03-06T14:10:56.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:56.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:56.057639+0000) 2024-03-06T14:10:57.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:57.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:57.057956+0000) 2024-03-06T14:10:58.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:58.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:58.058213+0000) 2024-03-06T14:10:59.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:10:59.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:59.058454+0000) 2024-03-06T14:11:00.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:00.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:00.058636+0000) 2024-03-06T14:11:01.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:01.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:01.058869+0000) 2024-03-06T14:11:02.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:02.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:02.059057+0000) 2024-03-06T14:11:03.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:03.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:03.059277+0000) 2024-03-06T14:11:04.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:04.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:04.059509+0000) 2024-03-06T14:11:05.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:05.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:05.059767+0000) 2024-03-06T14:11:06.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:06.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:06.059994+0000) 2024-03-06T14:11:07.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:07.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:07.060215+0000) 2024-03-06T14:11:08.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:08.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:08.060431+0000) 2024-03-06T14:11:09.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:09.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:09.060700+0000) 2024-03-06T14:11:10.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:10.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:10.060898+0000) 2024-03-06T14:11:11.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:11.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:11.061195+0000) 2024-03-06T14:11:12.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:12.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:12.061484+0000) 2024-03-06T14:11:13.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:13.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:13.061783+0000) 2024-03-06T14:11:14.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:14.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:14.062065+0000) 2024-03-06T14:11:15.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:15.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:15.062303+0000) 2024-03-06T14:11:16.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:16.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:16.062537+0000) 2024-03-06T14:11:17.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:17.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:17.062770+0000) 2024-03-06T14:11:18.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:18.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:18.063047+0000) 2024-03-06T14:11:19.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:19.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:19.063294+0000) 2024-03-06T14:11:20.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:20.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:20.063561+0000) 2024-03-06T14:11:21.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:21.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:21.063848+0000) 2024-03-06T14:11:22.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:22.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:22.064124+0000) 2024-03-06T14:11:23.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:23.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:23.064397+0000) 2024-03-06T14:11:24.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:24.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:24.064659+0000) 2024-03-06T14:11:24.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:24.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:25.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:25.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:25.064900+0000) 2024-03-06T14:11:26.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:26.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:26.065121+0000) 2024-03-06T14:11:27.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:27.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:27.065344+0000) 2024-03-06T14:11:28.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:28.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:28.065566+0000) 2024-03-06T14:11:29.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:29.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:29.065782+0000) 2024-03-06T14:11:30.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:30.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:30.066018+0000) 2024-03-06T14:11:31.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:31.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:31.066251+0000) 2024-03-06T14:11:32.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:32.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:32.066477+0000) 2024-03-06T14:11:33.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:33.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:33.066698+0000) 2024-03-06T14:11:34.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:34.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:34.066930+0000) 2024-03-06T14:11:35.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:35.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:35.067151+0000) 2024-03-06T14:11:36.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:36.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:36.067429+0000) 2024-03-06T14:11:37.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:37.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:37.067651+0000) 2024-03-06T14:11:38.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:38.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:38.067981+0000) 2024-03-06T14:11:39.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:39.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:39.068262+0000) 2024-03-06T14:11:40.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:40.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:40.068454+0000) 2024-03-06T14:11:41.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:41.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:41.068708+0000) 2024-03-06T14:11:42.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:42.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:42.068923+0000) 2024-03-06T14:11:43.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:43.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:43.069137+0000) 2024-03-06T14:11:44.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:44.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:44.069350+0000) 2024-03-06T14:11:45.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:45.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:45.069523+0000) 2024-03-06T14:11:46.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:46.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:46.069732+0000) 2024-03-06T14:11:47.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:47.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:47.069909+0000) 2024-03-06T14:11:48.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:48.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:48.070132+0000) 2024-03-06T14:11:49.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:49.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:49.070362+0000) 2024-03-06T14:11:50.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:50.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:50.070578+0000) 2024-03-06T14:11:51.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:51.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:51.070797+0000) 2024-03-06T14:11:52.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:52.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:52.071025+0000) 2024-03-06T14:11:53.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:53.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:53.071216+0000) 2024-03-06T14:11:54.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:54.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:54.071453+0000) 2024-03-06T14:11:54.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:54.346 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:55.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:55.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:55.071645+0000) 2024-03-06T14:11:56.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:56.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:56.071852+0000) 2024-03-06T14:11:57.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:57.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:57.072041+0000) 2024-03-06T14:11:58.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:58.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:58.072294+0000) 2024-03-06T14:11:59.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:11:59.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:59.072526+0000) 2024-03-06T14:12:00.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:00.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:00.072783+0000) 2024-03-06T14:12:01.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:01.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:01.073003+0000) 2024-03-06T14:12:02.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:02.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:02.073233+0000) 2024-03-06T14:12:03.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:03.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:03.073464+0000) 2024-03-06T14:12:04.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:04.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:04.073734+0000) 2024-03-06T14:12:05.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:05.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:05.073938+0000) 2024-03-06T14:12:06.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:06.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:06.074159+0000) 2024-03-06T14:12:07.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:07.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:07.074337+0000) 2024-03-06T14:12:08.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:08.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:08.074568+0000) 2024-03-06T14:12:09.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:09.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:09.074816+0000) 2024-03-06T14:12:10.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:10.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:10.075030+0000) 2024-03-06T14:12:11.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:11.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:11.075269+0000) 2024-03-06T14:12:12.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:12.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:12.075491+0000) 2024-03-06T14:12:13.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:13.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:13.075762+0000) 2024-03-06T14:12:14.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:14.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:14.075995+0000) 2024-03-06T14:12:15.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:15.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:15.076221+0000) 2024-03-06T14:12:16.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:16.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:16.076477+0000) 2024-03-06T14:12:17.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:17.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:17.076782+0000) 2024-03-06T14:12:18.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:18.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:18.076985+0000) 2024-03-06T14:12:19.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:19.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:19.077269+0000) 2024-03-06T14:12:20.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:20.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:20.077485+0000) 2024-03-06T14:12:21.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:21.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:21.077699+0000) 2024-03-06T14:12:22.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:22.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:22.077927+0000) 2024-03-06T14:12:23.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:23.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:23.078150+0000) 2024-03-06T14:12:24.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:24.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:24.078443+0000) 2024-03-06T14:12:24.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:24.558 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:25.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:25.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:25.078754+0000) 2024-03-06T14:12:26.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:26.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:26.078974+0000) 2024-03-06T14:12:27.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:27.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:27.079254+0000) 2024-03-06T14:12:28.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:28.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:28.079532+0000) 2024-03-06T14:12:29.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:29.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:29.079837+0000) 2024-03-06T14:12:30.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:30.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:30.080059+0000) 2024-03-06T14:12:31.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:31.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:31.080262+0000) 2024-03-06T14:12:32.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:32.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:32.080501+0000) 2024-03-06T14:12:33.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:33.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:33.080776+0000) 2024-03-06T14:12:34.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:34.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:34.080997+0000) 2024-03-06T14:12:35.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:35.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:35.081204+0000) 2024-03-06T14:12:36.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:36.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:36.081434+0000) 2024-03-06T14:12:37.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:37.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:37.081703+0000) 2024-03-06T14:12:38.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:38.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:38.081897+0000) 2024-03-06T14:12:39.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:39.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:39.082096+0000) 2024-03-06T14:12:40.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:40.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:40.082316+0000) 2024-03-06T14:12:41.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:41.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:41.082619+0000) 2024-03-06T14:12:42.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:42.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:42.082964+0000) 2024-03-06T14:12:43.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:43.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:43.083246+0000) 2024-03-06T14:12:44.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:44.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:44.083478+0000) 2024-03-06T14:12:45.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:45.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:45.083808+0000) 2024-03-06T14:12:46.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:46.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:46.084085+0000) 2024-03-06T14:12:47.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:47.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:47.084319+0000) 2024-03-06T14:12:48.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:48.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:48.084604+0000) 2024-03-06T14:12:49.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:49.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:49.084889+0000) 2024-03-06T14:12:50.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:50.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:50.085112+0000) 2024-03-06T14:12:51.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:51.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:51.085318+0000) 2024-03-06T14:12:52.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:52.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:52.085593+0000) 2024-03-06T14:12:53.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:53.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:53.085829+0000) 2024-03-06T14:12:54.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:54.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:54.086025+0000) 2024-03-06T14:12:54.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:54.795 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:55.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:55.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:55.086257+0000) 2024-03-06T14:12:56.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:56.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:56.086491+0000) 2024-03-06T14:12:57.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:57.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:57.086748+0000) 2024-03-06T14:12:58.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:58.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:58.086987+0000) 2024-03-06T14:12:59.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:12:59.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:59.087226+0000) 2024-03-06T14:13:00.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:00.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:00.087454+0000) 2024-03-06T14:13:01.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:01.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:01.087706+0000) 2024-03-06T14:13:02.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:02.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:02.087931+0000) 2024-03-06T14:13:03.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:03.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:03.088146+0000) 2024-03-06T14:13:04.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:04.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:04.088378+0000) 2024-03-06T14:13:05.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:05.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:05.088607+0000) 2024-03-06T14:13:06.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:06.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:06.088838+0000) 2024-03-06T14:13:07.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:07.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:07.089071+0000) 2024-03-06T14:13:08.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:08.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:08.089298+0000) 2024-03-06T14:13:09.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:09.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:09.089561+0000) 2024-03-06T14:13:10.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:10.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:10.089815+0000) 2024-03-06T14:13:11.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:11.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:11.090005+0000) 2024-03-06T14:13:12.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:12.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:12.090185+0000) 2024-03-06T14:13:13.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:13.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:13.090407+0000) 2024-03-06T14:13:14.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:14.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:14.090633+0000) 2024-03-06T14:13:15.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:15.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:15.090865+0000) 2024-03-06T14:13:16.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:16.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:16.091077+0000) 2024-03-06T14:13:17.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:17.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:17.091361+0000) 2024-03-06T14:13:18.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:18.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:18.091598+0000) 2024-03-06T14:13:19.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:19.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:19.091855+0000) 2024-03-06T14:13:20.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:20.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:20.092096+0000) 2024-03-06T14:13:21.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:21.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:21.092338+0000) 2024-03-06T14:13:22.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:22.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:22.092568+0000) 2024-03-06T14:13:23.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:23.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:23.092804+0000) 2024-03-06T14:13:24.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:24.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:24.093045+0000) 2024-03-06T14:13:24.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:24.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:25.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:25.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:25.093227+0000) 2024-03-06T14:13:26.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:26.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:26.093411+0000) 2024-03-06T14:13:27.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:27.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:27.093632+0000) 2024-03-06T14:13:28.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:28.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:28.093865+0000) 2024-03-06T14:13:29.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:29.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:29.094111+0000) 2024-03-06T14:13:30.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:30.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:30.094300+0000) 2024-03-06T14:13:31.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:31.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:31.094536+0000) 2024-03-06T14:13:32.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:32.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:32.094788+0000) 2024-03-06T14:13:33.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:33.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:33.095014+0000) 2024-03-06T14:13:34.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:34.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:34.095242+0000) 2024-03-06T14:13:35.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:35.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:35.095462+0000) 2024-03-06T14:13:36.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:36.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:36.095720+0000) 2024-03-06T14:13:37.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:37.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:37.095924+0000) 2024-03-06T14:13:38.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:38.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:38.096165+0000) 2024-03-06T14:13:39.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:39.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:39.096377+0000) 2024-03-06T14:13:40.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:40.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:40.096624+0000) 2024-03-06T14:13:41.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:41.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:41.096850+0000) 2024-03-06T14:13:42.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:42.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:42.097075+0000) 2024-03-06T14:13:43.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:43.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:43.097265+0000) 2024-03-06T14:13:44.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:44.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:44.097491+0000) 2024-03-06T14:13:45.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:45.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:45.097760+0000) 2024-03-06T14:13:46.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:46.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:46.097950+0000) 2024-03-06T14:13:47.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:47.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:47.098166+0000) 2024-03-06T14:13:48.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:48.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:48.098378+0000) 2024-03-06T14:13:49.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:49.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:49.098595+0000) 2024-03-06T14:13:50.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:50.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:50.098830+0000) 2024-03-06T14:13:51.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:51.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:51.099045+0000) 2024-03-06T14:13:52.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:52.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:52.099227+0000) 2024-03-06T14:13:53.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:53.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:53.099455+0000) 2024-03-06T14:13:54.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:54.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:54.099657+0000) 2024-03-06T14:13:55.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:55.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:55.099898+0000) 2024-03-06T14:13:55.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:55.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:56.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:56.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:56.100130+0000) 2024-03-06T14:13:57.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:57.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:57.100350+0000) 2024-03-06T14:13:58.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:58.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:58.100530+0000) 2024-03-06T14:13:59.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:13:59.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:59.100780+0000) 2024-03-06T14:14:00.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:00.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:00.100983+0000) 2024-03-06T14:14:01.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:01.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:01.101214+0000) 2024-03-06T14:14:02.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:02.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:02.101405+0000) 2024-03-06T14:14:03.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:03.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:03.101586+0000) 2024-03-06T14:14:04.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:04.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:04.101797+0000) 2024-03-06T14:14:05.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:05.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:05.102076+0000) 2024-03-06T14:14:06.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:06.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:06.102264+0000) 2024-03-06T14:14:07.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:07.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:07.102445+0000) 2024-03-06T14:14:08.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:08.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:08.102710+0000) 2024-03-06T14:14:09.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:09.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:09.102953+0000) 2024-03-06T14:14:10.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:10.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:10.103138+0000) 2024-03-06T14:14:11.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:11.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:11.103375+0000) 2024-03-06T14:14:12.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:12.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:12.103645+0000) 2024-03-06T14:14:13.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:13.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:13.103917+0000) 2024-03-06T14:14:14.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:14.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:14.104114+0000) 2024-03-06T14:14:15.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:15.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:15.104314+0000) 2024-03-06T14:14:16.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:16.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:16.104511+0000) 2024-03-06T14:14:17.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:17.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:17.104747+0000) 2024-03-06T14:14:18.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:18.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:18.104933+0000) 2024-03-06T14:14:19.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:19.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:19.105174+0000) 2024-03-06T14:14:20.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:20.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:20.105398+0000) 2024-03-06T14:14:21.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:21.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:21.105617+0000) 2024-03-06T14:14:22.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:22.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:22.105819+0000) 2024-03-06T14:14:23.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:23.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:23.106047+0000) 2024-03-06T14:14:24.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:24.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:24.106280+0000) 2024-03-06T14:14:25.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:25.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:25.106527+0000) 2024-03-06T14:14:25.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:25.341 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:26.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:26.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:26.106796+0000) 2024-03-06T14:14:27.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:27.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:27.107021+0000) 2024-03-06T14:14:28.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:28.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:28.107207+0000) 2024-03-06T14:14:29.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:29.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:29.107446+0000) 2024-03-06T14:14:30.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:30.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:30.107749+0000) 2024-03-06T14:14:31.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:31.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:31.107942+0000) 2024-03-06T14:14:32.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:32.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:32.108179+0000) 2024-03-06T14:14:33.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:33.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:33.108420+0000) 2024-03-06T14:14:34.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:34.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:34.108695+0000) 2024-03-06T14:14:35.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:35.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:35.108946+0000) 2024-03-06T14:14:36.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:36.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:36.109179+0000) 2024-03-06T14:14:37.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:37.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:37.109389+0000) 2024-03-06T14:14:38.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:38.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:38.109609+0000) 2024-03-06T14:14:39.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:39.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:39.109857+0000) 2024-03-06T14:14:40.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:40.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:40.110057+0000) 2024-03-06T14:14:41.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:41.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:41.110256+0000) 2024-03-06T14:14:42.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:42.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:42.110481+0000) 2024-03-06T14:14:43.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:43.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:43.110699+0000) 2024-03-06T14:14:44.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:44.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:44.110944+0000) 2024-03-06T14:14:45.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:45.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:45.111177+0000) 2024-03-06T14:14:46.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:46.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:46.111373+0000) 2024-03-06T14:14:47.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:47.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:47.111616+0000) 2024-03-06T14:14:48.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:48.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:48.111852+0000) 2024-03-06T14:14:49.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:49.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:49.112078+0000) 2024-03-06T14:14:50.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:50.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:50.112286+0000) 2024-03-06T14:14:51.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:51.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:51.112499+0000) 2024-03-06T14:14:52.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:52.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:52.112782+0000) 2024-03-06T14:14:53.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:53.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:53.113055+0000) 2024-03-06T14:14:54.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:54.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:54.113260+0000) 2024-03-06T14:14:55.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:55.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:55.113492+0000) 2024-03-06T14:14:55.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:55.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:56.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:56.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:56.113704+0000) 2024-03-06T14:14:57.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:57.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:57.113893+0000) 2024-03-06T14:14:58.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:58.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:58.114115+0000) 2024-03-06T14:14:59.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:14:59.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:59.114350+0000) 2024-03-06T14:15:00.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:00.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:00.114576+0000) 2024-03-06T14:15:01.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:01.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:01.114807+0000) 2024-03-06T14:15:02.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:02.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:02.115047+0000) 2024-03-06T14:15:03.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:03.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:03.115266+0000) 2024-03-06T14:15:04.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:04.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:04.115487+0000) 2024-03-06T14:15:05.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:05.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:05.115751+0000) 2024-03-06T14:15:06.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:06.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:06.115950+0000) 2024-03-06T14:15:07.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:07.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:07.116169+0000) 2024-03-06T14:15:08.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:08.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:08.116384+0000) 2024-03-06T14:15:09.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:09.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:09.116619+0000) 2024-03-06T14:15:10.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:10.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:10.116849+0000) 2024-03-06T14:15:11.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:11.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:11.117080+0000) 2024-03-06T14:15:12.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:12.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:12.117297+0000) 2024-03-06T14:15:13.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:13.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:13.117518+0000) 2024-03-06T14:15:14.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:14.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:14.117780+0000) 2024-03-06T14:15:15.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:15.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:15.118018+0000) 2024-03-06T14:15:16.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:16.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:16.118298+0000) 2024-03-06T14:15:17.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:17.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:17.118571+0000) 2024-03-06T14:15:18.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:18.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:18.118846+0000) 2024-03-06T14:15:19.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:19.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:19.119139+0000) 2024-03-06T14:15:20.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:20.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:20.119349+0000) 2024-03-06T14:15:21.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:21.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:21.119588+0000) 2024-03-06T14:15:22.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:22.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:22.119791+0000) 2024-03-06T14:15:23.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:23.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:23.120030+0000) 2024-03-06T14:15:24.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:24.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:24.120216+0000) 2024-03-06T14:15:25.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:25.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:25.120422+0000) 2024-03-06T14:15:25.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:25.617 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:26.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:26.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:26.120625+0000) 2024-03-06T14:15:27.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:27.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:27.120847+0000) 2024-03-06T14:15:28.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:28.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:28.121078+0000) 2024-03-06T14:15:29.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:29.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:29.121303+0000) 2024-03-06T14:15:30.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:30.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:30.121513+0000) 2024-03-06T14:15:31.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:31.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:31.121780+0000) 2024-03-06T14:15:32.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:32.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:32.121967+0000) 2024-03-06T14:15:33.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:33.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:33.122158+0000) 2024-03-06T14:15:34.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:34.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:34.122392+0000) 2024-03-06T14:15:35.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:35.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:35.122627+0000) 2024-03-06T14:15:36.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:36.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:36.122857+0000) 2024-03-06T14:15:37.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:37.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:37.123037+0000) 2024-03-06T14:15:38.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:38.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:38.123211+0000) 2024-03-06T14:15:39.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:39.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:39.123432+0000) 2024-03-06T14:15:40.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:40.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:40.123725+0000) 2024-03-06T14:15:41.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:41.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:41.123974+0000) 2024-03-06T14:15:42.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:42.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:42.124194+0000) 2024-03-06T14:15:43.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:43.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:43.124418+0000) 2024-03-06T14:15:44.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:44.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:44.124641+0000) 2024-03-06T14:15:45.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:45.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:45.124888+0000) 2024-03-06T14:15:46.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:46.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:46.125158+0000) 2024-03-06T14:15:47.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:47.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:47.125361+0000) 2024-03-06T14:15:48.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:48.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:48.125582+0000) 2024-03-06T14:15:49.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:49.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:49.125804+0000) 2024-03-06T14:15:50.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:50.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:50.126017+0000) 2024-03-06T14:15:51.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:51.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:51.126236+0000) 2024-03-06T14:15:52.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:52.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:52.126458+0000) 2024-03-06T14:15:53.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:53.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:53.126696+0000) 2024-03-06T14:15:54.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:54.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:54.126921+0000) 2024-03-06T14:15:55.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:55.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:55.127148+0000) 2024-03-06T14:15:55.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:55.763 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:56.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:56.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:56.127349+0000) 2024-03-06T14:15:57.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:57.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:57.127592+0000) 2024-03-06T14:15:58.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:58.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:58.127835+0000) 2024-03-06T14:15:59.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:15:59.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:59.128069+0000) 2024-03-06T14:16:00.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:00.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:00.128296+0000) 2024-03-06T14:16:01.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:01.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:01.128506+0000) 2024-03-06T14:16:02.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:02.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:02.128782+0000) 2024-03-06T14:16:03.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:03.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:03.128963+0000) 2024-03-06T14:16:04.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:04.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:04.129176+0000) 2024-03-06T14:16:05.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:05.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:05.129402+0000) 2024-03-06T14:16:06.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:06.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:06.129624+0000) 2024-03-06T14:16:07.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:07.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:07.129849+0000) 2024-03-06T14:16:08.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:08.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:08.130049+0000) 2024-03-06T14:16:09.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:09.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:09.130280+0000) 2024-03-06T14:16:10.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:10.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:10.130468+0000) 2024-03-06T14:16:11.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:11.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:11.130725+0000) 2024-03-06T14:16:12.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:12.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:12.130947+0000) 2024-03-06T14:16:13.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:13.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:13.131166+0000) 2024-03-06T14:16:14.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:14.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:14.131397+0000) 2024-03-06T14:16:15.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:15.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:15.131641+0000) 2024-03-06T14:16:16.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:16.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:16.131929+0000) 2024-03-06T14:16:17.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:17.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:17.132144+0000) 2024-03-06T14:16:18.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:18.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:18.132391+0000) 2024-03-06T14:16:19.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:19.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:19.132614+0000) 2024-03-06T14:16:20.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:20.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:20.132833+0000) 2024-03-06T14:16:21.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:21.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:21.133077+0000) 2024-03-06T14:16:22.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:22.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:22.133289+0000) 2024-03-06T14:16:23.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:23.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:23.133565+0000) 2024-03-06T14:16:24.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:24.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:24.133812+0000) 2024-03-06T14:16:25.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:25.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:25.133982+0000) 2024-03-06T14:16:25.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:25.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:26.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:26.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:26.134174+0000) 2024-03-06T14:16:27.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:27.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:27.134354+0000) 2024-03-06T14:16:28.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:28.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:28.134593+0000) 2024-03-06T14:16:29.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:29.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:29.134850+0000) 2024-03-06T14:16:30.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:30.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:30.135093+0000) 2024-03-06T14:16:31.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:31.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:31.135288+0000) 2024-03-06T14:16:32.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:32.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:32.135503+0000) 2024-03-06T14:16:33.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:33.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:33.135784+0000) 2024-03-06T14:16:34.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:34.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:34.136033+0000) 2024-03-06T14:16:35.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:35.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:35.136289+0000) 2024-03-06T14:16:36.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:36.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:36.136567+0000) 2024-03-06T14:16:37.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:37.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:37.136806+0000) 2024-03-06T14:16:38.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:38.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:38.137023+0000) 2024-03-06T14:16:39.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:39.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:39.137254+0000) 2024-03-06T14:16:40.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:40.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:40.137513+0000) 2024-03-06T14:16:41.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:41.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:41.137819+0000) 2024-03-06T14:16:42.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:42.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:42.138066+0000) 2024-03-06T14:16:43.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:43.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:43.138302+0000) 2024-03-06T14:16:44.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:44.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:44.138499+0000) 2024-03-06T14:16:45.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:45.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:45.138802+0000) 2024-03-06T14:16:46.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:46.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:46.138995+0000) 2024-03-06T14:16:47.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:47.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:47.139222+0000) 2024-03-06T14:16:48.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:48.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:48.139395+0000) 2024-03-06T14:16:49.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:49.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:49.139608+0000) 2024-03-06T14:16:50.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:50.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:50.139852+0000) 2024-03-06T14:16:51.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:51.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:51.140074+0000) 2024-03-06T14:16:52.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:52.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:52.140256+0000) 2024-03-06T14:16:53.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:53.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:53.140473+0000) 2024-03-06T14:16:54.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:54.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:54.140782+0000) 2024-03-06T14:16:55.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:55.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:55.141050+0000) 2024-03-06T14:16:56.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:56.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:56.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:56.141316+0000) 2024-03-06T14:16:56.147 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:57.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:57.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:57.141595+0000) 2024-03-06T14:16:58.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:58.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:58.141861+0000) 2024-03-06T14:16:59.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:16:59.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:59.142075+0000) 2024-03-06T14:17:00.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:00.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:00.142314+0000) 2024-03-06T14:17:01.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:01.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:01.142540+0000) 2024-03-06T14:17:02.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:02.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:02.142759+0000) 2024-03-06T14:17:03.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:03.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:03.142986+0000) 2024-03-06T14:17:04.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:04.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:04.143199+0000) 2024-03-06T14:17:05.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:05.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:05.143422+0000) 2024-03-06T14:17:06.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:06.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:06.143648+0000) 2024-03-06T14:17:07.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:07.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:07.143909+0000) 2024-03-06T14:17:08.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:08.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:08.144095+0000) 2024-03-06T14:17:09.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:09.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:09.144298+0000) 2024-03-06T14:17:10.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:10.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:10.144525+0000) 2024-03-06T14:17:11.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:11.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:11.144816+0000) 2024-03-06T14:17:12.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:12.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:12.145087+0000) 2024-03-06T14:17:13.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:13.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:13.145345+0000) 2024-03-06T14:17:14.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:14.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:14.145619+0000) 2024-03-06T14:17:15.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:15.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:15.145825+0000) 2024-03-06T14:17:16.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:16.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:16.146039+0000) 2024-03-06T14:17:17.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:17.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:17.146260+0000) 2024-03-06T14:17:18.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:18.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:18.146479+0000) 2024-03-06T14:17:19.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:19.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:19.146758+0000) 2024-03-06T14:17:20.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:20.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:20.146986+0000) 2024-03-06T14:17:21.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:21.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:21.147212+0000) 2024-03-06T14:17:22.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:22.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:22.147362+0000) 2024-03-06T14:17:23.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:23.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:23.147588+0000) 2024-03-06T14:17:24.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:24.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:24.147868+0000) 2024-03-06T14:17:25.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:25.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:25.148109+0000) 2024-03-06T14:17:26.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:26.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:26.148395+0000) 2024-03-06T14:17:26.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:26.368 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:27.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:27.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:27.148601+0000) 2024-03-06T14:17:28.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:28.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:28.148883+0000) 2024-03-06T14:17:29.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:29.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:29.149095+0000) 2024-03-06T14:17:30.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:30.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:30.149297+0000) 2024-03-06T14:17:31.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:31.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:31.149511+0000) 2024-03-06T14:17:32.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:32.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:32.149728+0000) 2024-03-06T14:17:33.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:33.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:33.149960+0000) 2024-03-06T14:17:34.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:34.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:34.150221+0000) 2024-03-06T14:17:35.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:35.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:35.150495+0000) 2024-03-06T14:17:36.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:36.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:36.150735+0000) 2024-03-06T14:17:37.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:37.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:37.150979+0000) 2024-03-06T14:17:38.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:38.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:38.151184+0000) 2024-03-06T14:17:39.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:39.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:39.151395+0000) 2024-03-06T14:17:40.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:40.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:40.151638+0000) 2024-03-06T14:17:41.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:41.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:41.151935+0000) 2024-03-06T14:17:42.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:42.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:42.152175+0000) 2024-03-06T14:17:43.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:43.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:43.152396+0000) 2024-03-06T14:17:44.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:44.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:44.152554+0000) 2024-03-06T14:17:45.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:45.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:45.152819+0000) 2024-03-06T14:17:46.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:46.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:46.153033+0000) 2024-03-06T14:17:47.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:47.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:47.153312+0000) 2024-03-06T14:17:48.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:48.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:48.153512+0000) 2024-03-06T14:17:49.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:49.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:49.153785+0000) 2024-03-06T14:17:50.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:50.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:50.154017+0000) 2024-03-06T14:17:51.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:51.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:51.154260+0000) 2024-03-06T14:17:52.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:52.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:52.154493+0000) 2024-03-06T14:17:53.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:53.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:53.154695+0000) 2024-03-06T14:17:54.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:54.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:54.154881+0000) 2024-03-06T14:17:55.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:55.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:55.155102+0000) 2024-03-06T14:17:56.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:56.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:56.155279+0000) 2024-03-06T14:17:56.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:56.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:57.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:57.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:57.155470+0000) 2024-03-06T14:17:58.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:58.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:58.155698+0000) 2024-03-06T14:17:59.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:17:59.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:59.155918+0000) 2024-03-06T14:18:00.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:00.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:00.156148+0000) 2024-03-06T14:18:01.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:01.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:01.156360+0000) 2024-03-06T14:18:02.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:02.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:02.156548+0000) 2024-03-06T14:18:03.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:03.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:03.156777+0000) 2024-03-06T14:18:04.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:04.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:04.157005+0000) 2024-03-06T14:18:05.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:05.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:05.157250+0000) 2024-03-06T14:18:06.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:06.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:06.157474+0000) 2024-03-06T14:18:07.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:07.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:07.157735+0000) 2024-03-06T14:18:08.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:08.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:08.157961+0000) 2024-03-06T14:18:09.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:09.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:09.158195+0000) 2024-03-06T14:18:10.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:10.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:10.158427+0000) 2024-03-06T14:18:11.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:11.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:11.158652+0000) 2024-03-06T14:18:12.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:12.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:12.158919+0000) 2024-03-06T14:18:13.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:13.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:13.159161+0000) 2024-03-06T14:18:14.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:14.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:14.159357+0000) 2024-03-06T14:18:15.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:15.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:15.159602+0000) 2024-03-06T14:18:16.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:16.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:16.159883+0000) 2024-03-06T14:18:17.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:17.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:17.160112+0000) 2024-03-06T14:18:18.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:18.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:18.160328+0000) 2024-03-06T14:18:19.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:19.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:19.160593+0000) 2024-03-06T14:18:20.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:20.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:20.160866+0000) 2024-03-06T14:18:21.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:21.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:21.161135+0000) 2024-03-06T14:18:22.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:22.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:22.161343+0000) 2024-03-06T14:18:23.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:23.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:23.161529+0000) 2024-03-06T14:18:24.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:24.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:24.161794+0000) 2024-03-06T14:18:25.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:25.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:25.162024+0000) 2024-03-06T14:18:26.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:26.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:26.162256+0000) 2024-03-06T14:18:26.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:26.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:27.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:27.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:27.162490+0000) 2024-03-06T14:18:28.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:28.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:28.162753+0000) 2024-03-06T14:18:29.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:29.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:29.162997+0000) 2024-03-06T14:18:30.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:30.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:30.163203+0000) 2024-03-06T14:18:31.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:31.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:31.163452+0000) 2024-03-06T14:18:32.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:32.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:32.163700+0000) 2024-03-06T14:18:33.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:33.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:33.163896+0000) 2024-03-06T14:18:34.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:34.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:34.164092+0000) 2024-03-06T14:18:35.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:35.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:35.164294+0000) 2024-03-06T14:18:36.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:36.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:36.164516+0000) 2024-03-06T14:18:37.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:37.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:37.164781+0000) 2024-03-06T14:18:38.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:38.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:38.164974+0000) 2024-03-06T14:18:39.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:39.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:39.165230+0000) 2024-03-06T14:18:40.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:40.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:40.165422+0000) 2024-03-06T14:18:41.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:41.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:41.165644+0000) 2024-03-06T14:18:42.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:42.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:42.165901+0000) 2024-03-06T14:18:43.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:43.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:43.166122+0000) 2024-03-06T14:18:44.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:44.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:44.166330+0000) 2024-03-06T14:18:45.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:45.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:45.166566+0000) 2024-03-06T14:18:46.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:46.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:46.166793+0000) 2024-03-06T14:18:47.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:47.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:47.166971+0000) 2024-03-06T14:18:48.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:48.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:48.167194+0000) 2024-03-06T14:18:49.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:49.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:49.167430+0000) 2024-03-06T14:18:50.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:50.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:50.167602+0000) 2024-03-06T14:18:51.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:51.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:51.167800+0000) 2024-03-06T14:18:52.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:52.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:52.168009+0000) 2024-03-06T14:18:53.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:53.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:53.168234+0000) 2024-03-06T14:18:54.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:54.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:54.168472+0000) 2024-03-06T14:18:55.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:55.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:55.168698+0000) 2024-03-06T14:18:56.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:56.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:56.168921+0000) 2024-03-06T14:18:56.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:56.970 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:57.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:57.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:57.169150+0000) 2024-03-06T14:18:58.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:58.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:58.169343+0000) 2024-03-06T14:18:59.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:18:59.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:59.169591+0000) 2024-03-06T14:19:00.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:00.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:00.169830+0000) 2024-03-06T14:19:01.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:01.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:01.170046+0000) 2024-03-06T14:19:02.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:02.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:02.170258+0000) 2024-03-06T14:19:03.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:03.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:03.170479+0000) 2024-03-06T14:19:04.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:04.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:04.170753+0000) 2024-03-06T14:19:05.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:05.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:05.170995+0000) 2024-03-06T14:19:06.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:06.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:06.171242+0000) 2024-03-06T14:19:07.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:07.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:07.171489+0000) 2024-03-06T14:19:08.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:08.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:08.171755+0000) 2024-03-06T14:19:09.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:09.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:09.171989+0000) 2024-03-06T14:19:10.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:10.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:10.172219+0000) 2024-03-06T14:19:11.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:11.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:11.172448+0000) 2024-03-06T14:19:12.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:12.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:12.172702+0000) 2024-03-06T14:19:13.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:13.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:13.172879+0000) 2024-03-06T14:19:14.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:14.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:14.173123+0000) 2024-03-06T14:19:15.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:15.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:15.173364+0000) 2024-03-06T14:19:16.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:16.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:16.173608+0000) 2024-03-06T14:19:17.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:17.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:17.173855+0000) 2024-03-06T14:19:18.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:18.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:18.174040+0000) 2024-03-06T14:19:19.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:19.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:19.174274+0000) 2024-03-06T14:19:20.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:20.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:20.174506+0000) 2024-03-06T14:19:21.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:21.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:21.174754+0000) 2024-03-06T14:19:22.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:22.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:22.174980+0000) 2024-03-06T14:19:23.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:23.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:23.175209+0000) 2024-03-06T14:19:24.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:24.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:24.175436+0000) 2024-03-06T14:19:25.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:25.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:25.175701+0000) 2024-03-06T14:19:26.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:26.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:26.175929+0000) 2024-03-06T14:19:27.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:27.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:27.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:27.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:27.176179+0000) 2024-03-06T14:19:28.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:28.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:28.176437+0000) 2024-03-06T14:19:29.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:29.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:29.176725+0000) 2024-03-06T14:19:30.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:30.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:30.176943+0000) 2024-03-06T14:19:31.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:31.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:31.177178+0000) 2024-03-06T14:19:32.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:32.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:32.177402+0000) 2024-03-06T14:19:33.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:33.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:33.177636+0000) 2024-03-06T14:19:34.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:34.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:34.177922+0000) 2024-03-06T14:19:35.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:35.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:35.178186+0000) 2024-03-06T14:19:36.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:36.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:36.178433+0000) 2024-03-06T14:19:37.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:37.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:37.178710+0000) 2024-03-06T14:19:38.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:38.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:38.178923+0000) 2024-03-06T14:19:39.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:39.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:39.179129+0000) 2024-03-06T14:19:40.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:40.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:40.179380+0000) 2024-03-06T14:19:41.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:41.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:41.179600+0000) 2024-03-06T14:19:42.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:42.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:42.179841+0000) 2024-03-06T14:19:43.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:43.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:43.180034+0000) 2024-03-06T14:19:44.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:44.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:44.180252+0000) 2024-03-06T14:19:45.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:45.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:45.180478+0000) 2024-03-06T14:19:46.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:46.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:46.180745+0000) 2024-03-06T14:19:47.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:47.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:47.180935+0000) 2024-03-06T14:19:48.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:48.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:48.181177+0000) 2024-03-06T14:19:49.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:49.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:49.181434+0000) 2024-03-06T14:19:50.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:50.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:50.181661+0000) 2024-03-06T14:19:51.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:51.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:51.181918+0000) 2024-03-06T14:19:52.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:52.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:52.182160+0000) 2024-03-06T14:19:53.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:53.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:53.182398+0000) 2024-03-06T14:19:54.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:54.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:54.182649+0000) 2024-03-06T14:19:55.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:55.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:55.182917+0000) 2024-03-06T14:19:56.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:56.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:56.183111+0000) 2024-03-06T14:19:57.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:57.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:57.183401+0000) 2024-03-06T14:19:57.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:57.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:58.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:58.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:58.183587+0000) 2024-03-06T14:19:59.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:19:59.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:59.183827+0000) 2024-03-06T14:20:00.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:00.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:00.184022+0000) 2024-03-06T14:20:01.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:01.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:01.184251+0000) 2024-03-06T14:20:02.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:02.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:02.184477+0000) 2024-03-06T14:20:03.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:03.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:03.184695+0000) 2024-03-06T14:20:04.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:04.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:04.184901+0000) 2024-03-06T14:20:05.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:05.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:05.185126+0000) 2024-03-06T14:20:06.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:06.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:06.185346+0000) 2024-03-06T14:20:07.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:07.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:07.185569+0000) 2024-03-06T14:20:08.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:08.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:08.185819+0000) 2024-03-06T14:20:09.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:09.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:09.186056+0000) 2024-03-06T14:20:10.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:10.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:10.186262+0000) 2024-03-06T14:20:11.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:11.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:11.186511+0000) 2024-03-06T14:20:12.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:12.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:12.186745+0000) 2024-03-06T14:20:13.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:13.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:13.186985+0000) 2024-03-06T14:20:14.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:14.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:14.187199+0000) 2024-03-06T14:20:15.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:15.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:15.187428+0000) 2024-03-06T14:20:16.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:16.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:16.187618+0000) 2024-03-06T14:20:17.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:17.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:17.187829+0000) 2024-03-06T14:20:18.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:18.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:18.188070+0000) 2024-03-06T14:20:19.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:19.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:19.188295+0000) 2024-03-06T14:20:20.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:20.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:20.188523+0000) 2024-03-06T14:20:21.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:21.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:21.188783+0000) 2024-03-06T14:20:22.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:22.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:22.189004+0000) 2024-03-06T14:20:23.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:23.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:23.189236+0000) 2024-03-06T14:20:24.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:24.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:24.189474+0000) 2024-03-06T14:20:25.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:25.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:25.189758+0000) 2024-03-06T14:20:26.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:26.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:26.190014+0000) 2024-03-06T14:20:27.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:27.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:27.190214+0000) 2024-03-06T14:20:27.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:27.474 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:28.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:28.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:28.190441+0000) 2024-03-06T14:20:29.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:29.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:29.190715+0000) 2024-03-06T14:20:30.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:30.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:30.190938+0000) 2024-03-06T14:20:31.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:31.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:31.191173+0000) 2024-03-06T14:20:32.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:32.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:32.191428+0000) 2024-03-06T14:20:33.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:33.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:33.191628+0000) 2024-03-06T14:20:34.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:34.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:34.191841+0000) 2024-03-06T14:20:35.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:35.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:35.192012+0000) 2024-03-06T14:20:36.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:36.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:36.192245+0000) 2024-03-06T14:20:37.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:37.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:37.192458+0000) 2024-03-06T14:20:38.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:38.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:38.192711+0000) 2024-03-06T14:20:39.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:39.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:39.192932+0000) 2024-03-06T14:20:40.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:40.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:40.193170+0000) 2024-03-06T14:20:41.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:41.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:41.193361+0000) 2024-03-06T14:20:42.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:42.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:42.193579+0000) 2024-03-06T14:20:43.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:43.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:43.193791+0000) 2024-03-06T14:20:44.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:44.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:44.193996+0000) 2024-03-06T14:20:45.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:45.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:45.194281+0000) 2024-03-06T14:20:46.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:46.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:46.194483+0000) 2024-03-06T14:20:47.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:47.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:47.194762+0000) 2024-03-06T14:20:48.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:48.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:48.194986+0000) 2024-03-06T14:20:49.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:49.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:49.195223+0000) 2024-03-06T14:20:50.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:50.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:50.195423+0000) 2024-03-06T14:20:51.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:51.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:51.195708+0000) 2024-03-06T14:20:52.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:52.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:52.195904+0000) 2024-03-06T14:20:53.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:53.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:53.196141+0000) 2024-03-06T14:20:54.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:54.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:54.196408+0000) 2024-03-06T14:20:55.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:55.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:55.196604+0000) 2024-03-06T14:20:56.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:56.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:56.196826+0000) 2024-03-06T14:20:57.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:57.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:57.197043+0000) 2024-03-06T14:20:57.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:57.676 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:58.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:58.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:58.197272+0000) 2024-03-06T14:20:59.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:20:59.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:59.197469+0000) 2024-03-06T14:21:00.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:00.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:00.197653+0000) 2024-03-06T14:21:01.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:01.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:01.197931+0000) 2024-03-06T14:21:02.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:02.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:02.198193+0000) 2024-03-06T14:21:03.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:03.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:03.198397+0000) 2024-03-06T14:21:04.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:04.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:04.198624+0000) 2024-03-06T14:21:05.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:05.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:05.198850+0000) 2024-03-06T14:21:06.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:06.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:06.199039+0000) 2024-03-06T14:21:07.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:07.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:07.199263+0000) 2024-03-06T14:21:08.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:08.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:08.199490+0000) 2024-03-06T14:21:09.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:09.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:09.199753+0000) 2024-03-06T14:21:10.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:10.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:10.200030+0000) 2024-03-06T14:21:11.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:11.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:11.200310+0000) 2024-03-06T14:21:12.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:12.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:12.200548+0000) 2024-03-06T14:21:13.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:13.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:13.200780+0000) 2024-03-06T14:21:14.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:14.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:14.201006+0000) 2024-03-06T14:21:15.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:15.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:15.201195+0000) 2024-03-06T14:21:16.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:16.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:16.201425+0000) 2024-03-06T14:21:17.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:17.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:17.201654+0000) 2024-03-06T14:21:18.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:18.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:18.201919+0000) 2024-03-06T14:21:19.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:19.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:19.202152+0000) 2024-03-06T14:21:20.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:20.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:20.202336+0000) 2024-03-06T14:21:21.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:21.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:21.202539+0000) 2024-03-06T14:21:22.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:22.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:22.202778+0000) 2024-03-06T14:21:23.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:23.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:23.203013+0000) 2024-03-06T14:21:24.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:24.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:24.203242+0000) 2024-03-06T14:21:25.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:25.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:25.203466+0000) 2024-03-06T14:21:26.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:26.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:26.203647+0000) 2024-03-06T14:21:27.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:27.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:27.203912+0000) 2024-03-06T14:21:27.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:27.844 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:28.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:28.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:28.204147+0000) 2024-03-06T14:21:29.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:29.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:29.204335+0000) 2024-03-06T14:21:30.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:30.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:30.204554+0000) 2024-03-06T14:21:31.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:31.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:31.204777+0000) 2024-03-06T14:21:32.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:32.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:32.204998+0000) 2024-03-06T14:21:33.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:33.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:33.205282+0000) 2024-03-06T14:21:34.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:34.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:34.205470+0000) 2024-03-06T14:21:35.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:35.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:35.205692+0000) 2024-03-06T14:21:36.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:36.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:36.205929+0000) 2024-03-06T14:21:37.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:37.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:37.206154+0000) 2024-03-06T14:21:38.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:38.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:38.206377+0000) 2024-03-06T14:21:39.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:39.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:39.206618+0000) 2024-03-06T14:21:40.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:40.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:40.206837+0000) 2024-03-06T14:21:41.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:41.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:41.207067+0000) 2024-03-06T14:21:42.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:42.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:42.207293+0000) 2024-03-06T14:21:43.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:43.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:43.207510+0000) 2024-03-06T14:21:44.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:44.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:44.207719+0000) 2024-03-06T14:21:45.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:45.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:45.207902+0000) 2024-03-06T14:21:46.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:46.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:46.208104+0000) 2024-03-06T14:21:47.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:47.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:47.208325+0000) 2024-03-06T14:21:48.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:48.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:48.208546+0000) 2024-03-06T14:21:49.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:49.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:49.208789+0000) 2024-03-06T14:21:50.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:50.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:50.208946+0000) 2024-03-06T14:21:51.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:51.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:51.209169+0000) 2024-03-06T14:21:52.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:52.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:52.209392+0000) 2024-03-06T14:21:53.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:53.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:53.209616+0000) 2024-03-06T14:21:54.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:54.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:54.209848+0000) 2024-03-06T14:21:55.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:55.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:55.210081+0000) 2024-03-06T14:21:56.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:56.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:56.210331+0000) 2024-03-06T14:21:57.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:57.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:57.210576+0000) 2024-03-06T14:21:58.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:58.043 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:58.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:58.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:58.210837+0000) 2024-03-06T14:21:59.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:21:59.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:59.211079+0000) 2024-03-06T14:22:00.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:00.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:00.211267+0000) 2024-03-06T14:22:01.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:01.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:01.211511+0000) 2024-03-06T14:22:02.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:02.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:02.211786+0000) 2024-03-06T14:22:03.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:03.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:03.212014+0000) 2024-03-06T14:22:04.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:04.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:04.212207+0000) 2024-03-06T14:22:05.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:05.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:05.212447+0000) 2024-03-06T14:22:06.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:06.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:06.212692+0000) 2024-03-06T14:22:07.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:07.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:07.212913+0000) 2024-03-06T14:22:08.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:08.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:08.213166+0000) 2024-03-06T14:22:09.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:09.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:09.213410+0000) 2024-03-06T14:22:10.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:10.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:10.213638+0000) 2024-03-06T14:22:11.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:11.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:11.213912+0000) 2024-03-06T14:22:12.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:12.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:12.214091+0000) 2024-03-06T14:22:13.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:13.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:13.214316+0000) 2024-03-06T14:22:14.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:14.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:14.214504+0000) 2024-03-06T14:22:15.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:15.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:15.214724+0000) 2024-03-06T14:22:16.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:16.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:16.214965+0000) 2024-03-06T14:22:17.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:17.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:17.215152+0000) 2024-03-06T14:22:18.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:18.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:18.215375+0000) 2024-03-06T14:22:19.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:19.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:19.215610+0000) 2024-03-06T14:22:20.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:20.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:20.215805+0000) 2024-03-06T14:22:21.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:21.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:21.216050+0000) 2024-03-06T14:22:22.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:22.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:22.216279+0000) 2024-03-06T14:22:23.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:23.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:23.216512+0000) 2024-03-06T14:22:24.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:24.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:24.216742+0000) 2024-03-06T14:22:25.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:25.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:25.216982+0000) 2024-03-06T14:22:26.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:26.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:26.217210+0000) 2024-03-06T14:22:27.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:27.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:27.217436+0000) 2024-03-06T14:22:28.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:28.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:28.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:28.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:28.217654+0000) 2024-03-06T14:22:29.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:29.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:29.217924+0000) 2024-03-06T14:22:30.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:30.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:30.218145+0000) 2024-03-06T14:22:31.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:31.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:31.218387+0000) 2024-03-06T14:22:32.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:32.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:32.218610+0000) 2024-03-06T14:22:33.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:33.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:33.218890+0000) 2024-03-06T14:22:34.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:34.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:34.219158+0000) 2024-03-06T14:22:35.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:35.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:35.219436+0000) 2024-03-06T14:22:36.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:36.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:36.219713+0000) 2024-03-06T14:22:37.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:37.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:37.219996+0000) 2024-03-06T14:22:38.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:38.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:38.220262+0000) 2024-03-06T14:22:39.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:39.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:39.220544+0000) 2024-03-06T14:22:40.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:40.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:40.220793+0000) 2024-03-06T14:22:41.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:41.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:41.220986+0000) 2024-03-06T14:22:42.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:42.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:42.221221+0000) 2024-03-06T14:22:43.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:43.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:43.221402+0000) 2024-03-06T14:22:44.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:44.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:44.221597+0000) 2024-03-06T14:22:45.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:45.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:45.221786+0000) 2024-03-06T14:22:46.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:46.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:46.221979+0000) 2024-03-06T14:22:47.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:47.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:47.222197+0000) 2024-03-06T14:22:48.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:48.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:48.222421+0000) 2024-03-06T14:22:49.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:49.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:49.222656+0000) 2024-03-06T14:22:50.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:50.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:50.222904+0000) 2024-03-06T14:22:51.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:51.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:51.223101+0000) 2024-03-06T14:22:52.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:52.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:52.223288+0000) 2024-03-06T14:22:53.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:53.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:53.223522+0000) 2024-03-06T14:22:54.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:54.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:54.223730+0000) 2024-03-06T14:22:55.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:55.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:55.223943+0000) 2024-03-06T14:22:56.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:56.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:56.224172+0000) 2024-03-06T14:22:57.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:57.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:57.224354+0000) 2024-03-06T14:22:58.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:58.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:58.224569+0000) 2024-03-06T14:22:58.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:58.354 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:59.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:22:59.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:59.224834+0000) 2024-03-06T14:23:00.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:00.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:00.225105+0000) 2024-03-06T14:23:01.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:01.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:01.225311+0000) 2024-03-06T14:23:02.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:02.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:02.225579+0000) 2024-03-06T14:23:03.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:03.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:03.225855+0000) 2024-03-06T14:23:04.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:04.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:04.226124+0000) 2024-03-06T14:23:05.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:05.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:05.226369+0000) 2024-03-06T14:23:06.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:06.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:06.226783+0000) 2024-03-06T14:23:07.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:07.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:07.226993+0000) 2024-03-06T14:23:08.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:08.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:08.227156+0000) 2024-03-06T14:23:09.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:09.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:09.227391+0000) 2024-03-06T14:23:10.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:10.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:10.227620+0000) 2024-03-06T14:23:11.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:11.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:11.227859+0000) 2024-03-06T14:23:12.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:12.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:12.228118+0000) 2024-03-06T14:23:13.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:13.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:13.228361+0000) 2024-03-06T14:23:14.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:14.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:14.228648+0000) 2024-03-06T14:23:15.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:15.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:15.228878+0000) 2024-03-06T14:23:16.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:16.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:16.229063+0000) 2024-03-06T14:23:17.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:17.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:17.229237+0000) 2024-03-06T14:23:18.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:18.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:18.229455+0000) 2024-03-06T14:23:19.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:19.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:19.229724+0000) 2024-03-06T14:23:20.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:20.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:20.229952+0000) 2024-03-06T14:23:21.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:21.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:21.230148+0000) 2024-03-06T14:23:22.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:22.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:22.230551+0000) 2024-03-06T14:23:23.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:23.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:23.231039+0000) 2024-03-06T14:23:24.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:24.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:24.231264+0000) 2024-03-06T14:23:25.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:25.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:25.231506+0000) 2024-03-06T14:23:26.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:26.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:26.231721+0000) 2024-03-06T14:23:27.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:27.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:27.231970+0000) 2024-03-06T14:23:28.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:28.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:28.232200+0000) 2024-03-06T14:23:28.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:28.595 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:29.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:29.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:29.232419+0000) 2024-03-06T14:23:30.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:30.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:30.232605+0000) 2024-03-06T14:23:31.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:31.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:31.232827+0000) 2024-03-06T14:23:32.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:32.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:32.233014+0000) 2024-03-06T14:23:33.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:33.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:33.233215+0000) 2024-03-06T14:23:34.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:34.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:34.233435+0000) 2024-03-06T14:23:35.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:35.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:35.233662+0000) 2024-03-06T14:23:36.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:36.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:36.233931+0000) 2024-03-06T14:23:37.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:37.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:37.234118+0000) 2024-03-06T14:23:38.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:38.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:38.234347+0000) 2024-03-06T14:23:39.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:39.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:39.234578+0000) 2024-03-06T14:23:40.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:40.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:40.234858+0000) 2024-03-06T14:23:41.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:41.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:41.235088+0000) 2024-03-06T14:23:42.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:42.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:42.235279+0000) 2024-03-06T14:23:43.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:43.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:43.235474+0000) 2024-03-06T14:23:44.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:44.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:44.235738+0000) 2024-03-06T14:23:45.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:45.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:45.235967+0000) 2024-03-06T14:23:46.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:46.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:46.236151+0000) 2024-03-06T14:23:47.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:47.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:47.236370+0000) 2024-03-06T14:23:48.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:48.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:48.236592+0000) 2024-03-06T14:23:49.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:49.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:49.236840+0000) 2024-03-06T14:23:50.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:50.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:50.237067+0000) 2024-03-06T14:23:51.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:51.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:51.237303+0000) 2024-03-06T14:23:52.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:52.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:52.237529+0000) 2024-03-06T14:23:53.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:53.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:53.237781+0000) 2024-03-06T14:23:54.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:54.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:54.238020+0000) 2024-03-06T14:23:55.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:55.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:55.238318+0000) 2024-03-06T14:23:56.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:56.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:56.238511+0000) 2024-03-06T14:23:57.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:57.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:57.238778+0000) 2024-03-06T14:23:58.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:58.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:58.239005+0000) 2024-03-06T14:23:58.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:58.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:59.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:23:59.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:59.239238+0000) 2024-03-06T14:24:00.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:00.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:00.239437+0000) 2024-03-06T14:24:01.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:01.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:01.239644+0000) 2024-03-06T14:24:02.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:02.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:02.239898+0000) 2024-03-06T14:24:03.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:03.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:03.240101+0000) 2024-03-06T14:24:04.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:04.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:04.240342+0000) 2024-03-06T14:24:05.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:05.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:05.240578+0000) 2024-03-06T14:24:06.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:06.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:06.240805+0000) 2024-03-06T14:24:07.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:07.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:07.241048+0000) 2024-03-06T14:24:08.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:08.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:08.241341+0000) 2024-03-06T14:24:09.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:09.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:09.241598+0000) 2024-03-06T14:24:10.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:10.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:10.241863+0000) 2024-03-06T14:24:11.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:11.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:11.242099+0000) 2024-03-06T14:24:12.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:12.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:12.242355+0000) 2024-03-06T14:24:13.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:13.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:13.242615+0000) 2024-03-06T14:24:14.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:14.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:14.242833+0000) 2024-03-06T14:24:15.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:15.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:15.243016+0000) 2024-03-06T14:24:16.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:16.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:16.243246+0000) 2024-03-06T14:24:17.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:17.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:17.243470+0000) 2024-03-06T14:24:18.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:18.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:18.243723+0000) 2024-03-06T14:24:19.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:19.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:19.243962+0000) 2024-03-06T14:24:20.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:20.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:20.244182+0000) 2024-03-06T14:24:21.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:21.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:21.244425+0000) 2024-03-06T14:24:22.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:22.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:22.244631+0000) 2024-03-06T14:24:23.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:23.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:23.244906+0000) 2024-03-06T14:24:24.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:24.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:24.245098+0000) 2024-03-06T14:24:25.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:25.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:25.245326+0000) 2024-03-06T14:24:26.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:26.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:26.245571+0000) 2024-03-06T14:24:27.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:27.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:27.245803+0000) 2024-03-06T14:24:28.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:28.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:28.245987+0000) 2024-03-06T14:24:29.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:29.028 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:29.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:29.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:29.246175+0000) 2024-03-06T14:24:30.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:30.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:30.246383+0000) 2024-03-06T14:24:31.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:31.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:31.246615+0000) 2024-03-06T14:24:32.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:32.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:32.246820+0000) 2024-03-06T14:24:33.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:33.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:33.247020+0000) 2024-03-06T14:24:34.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:34.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:34.247212+0000) 2024-03-06T14:24:35.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:35.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:35.247443+0000) 2024-03-06T14:24:36.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:36.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:36.247635+0000) 2024-03-06T14:24:37.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:37.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:37.247864+0000) 2024-03-06T14:24:38.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:38.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:38.248087+0000) 2024-03-06T14:24:39.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:39.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:39.248327+0000) 2024-03-06T14:24:40.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:40.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:40.248559+0000) 2024-03-06T14:24:41.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:41.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:41.248796+0000) 2024-03-06T14:24:42.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:42.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:42.248978+0000) 2024-03-06T14:24:43.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:43.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:43.249204+0000) 2024-03-06T14:24:44.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:44.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:44.249433+0000) 2024-03-06T14:24:45.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:45.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:45.249659+0000) 2024-03-06T14:24:46.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:46.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:46.249948+0000) 2024-03-06T14:24:47.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:47.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:47.250172+0000) 2024-03-06T14:24:48.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:48.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:48.250361+0000) 2024-03-06T14:24:49.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:49.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:49.250551+0000) 2024-03-06T14:24:50.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:50.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:50.250811+0000) 2024-03-06T14:24:51.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:51.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:51.251013+0000) 2024-03-06T14:24:52.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:52.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:52.251242+0000) 2024-03-06T14:24:53.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:53.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:53.251429+0000) 2024-03-06T14:24:54.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:54.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:54.251656+0000) 2024-03-06T14:24:55.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:55.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:55.251940+0000) 2024-03-06T14:24:56.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:56.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:56.252260+0000) 2024-03-06T14:24:57.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:57.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:57.252462+0000) 2024-03-06T14:24:58.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:58.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:58.252694+0000) 2024-03-06T14:24:59.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:59.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:59.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:24:59.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:59.252904+0000) 2024-03-06T14:25:00.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:00.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:00.253132+0000) 2024-03-06T14:25:01.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:01.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:01.253325+0000) 2024-03-06T14:25:02.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:02.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:02.253563+0000) 2024-03-06T14:25:03.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:03.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:03.253788+0000) 2024-03-06T14:25:04.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:04.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:04.253967+0000) 2024-03-06T14:25:05.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:05.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:05.254189+0000) 2024-03-06T14:25:06.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:06.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:06.254331+0000) 2024-03-06T14:25:07.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:07.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:07.254559+0000) 2024-03-06T14:25:08.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:08.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:08.254789+0000) 2024-03-06T14:25:09.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:09.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:09.255053+0000) 2024-03-06T14:25:10.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:10.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:10.255252+0000) 2024-03-06T14:25:11.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:11.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:11.255479+0000) 2024-03-06T14:25:12.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:12.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:12.255758+0000) 2024-03-06T14:25:13.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:13.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:13.255950+0000) 2024-03-06T14:25:14.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:14.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:14.256195+0000) 2024-03-06T14:25:15.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:15.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:15.256374+0000) 2024-03-06T14:25:16.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:16.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:16.256598+0000) 2024-03-06T14:25:17.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:17.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:17.256821+0000) 2024-03-06T14:25:18.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:18.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:18.257047+0000) 2024-03-06T14:25:19.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:19.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:19.257291+0000) 2024-03-06T14:25:20.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:20.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:20.257482+0000) 2024-03-06T14:25:21.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:21.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:21.257701+0000) 2024-03-06T14:25:22.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:22.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:22.257940+0000) 2024-03-06T14:25:23.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:23.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:23.258164+0000) 2024-03-06T14:25:24.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:24.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:24.258388+0000) 2024-03-06T14:25:25.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:25.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:25.258619+0000) 2024-03-06T14:25:26.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:26.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:26.258802+0000) 2024-03-06T14:25:27.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:27.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:27.259010+0000) 2024-03-06T14:25:28.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:28.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:28.259214+0000) 2024-03-06T14:25:29.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:29.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:29.259462+0000) 2024-03-06T14:25:29.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:29.478 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:30.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:30.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:30.259788+0000) 2024-03-06T14:25:31.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:31.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:31.259992+0000) 2024-03-06T14:25:32.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:32.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:32.260214+0000) 2024-03-06T14:25:33.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:33.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:33.260441+0000) 2024-03-06T14:25:34.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:34.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:34.260693+0000) 2024-03-06T14:25:35.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:35.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:35.260929+0000) 2024-03-06T14:25:36.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:36.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:36.261167+0000) 2024-03-06T14:25:37.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:37.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:37.261363+0000) 2024-03-06T14:25:38.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:38.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:38.261562+0000) 2024-03-06T14:25:39.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:39.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:39.261785+0000) 2024-03-06T14:25:40.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:40.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:40.262041+0000) 2024-03-06T14:25:41.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:41.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:41.262330+0000) 2024-03-06T14:25:42.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:42.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:42.262614+0000) 2024-03-06T14:25:43.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:43.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:43.262865+0000) 2024-03-06T14:25:44.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:44.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:44.263165+0000) 2024-03-06T14:25:45.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:45.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:45.263454+0000) 2024-03-06T14:25:46.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:46.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:46.263787+0000) 2024-03-06T14:25:47.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:47.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:47.264072+0000) 2024-03-06T14:25:48.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:48.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:48.264275+0000) 2024-03-06T14:25:49.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:49.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:49.264503+0000) 2024-03-06T14:25:50.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:50.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:50.264767+0000) 2024-03-06T14:25:51.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:51.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:51.264996+0000) 2024-03-06T14:25:52.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:52.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:52.265231+0000) 2024-03-06T14:25:53.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:53.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:53.265425+0000) 2024-03-06T14:25:54.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:54.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:54.265688+0000) 2024-03-06T14:25:55.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:55.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:55.265906+0000) 2024-03-06T14:25:56.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:56.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:56.266184+0000) 2024-03-06T14:25:57.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:57.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:57.266459+0000) 2024-03-06T14:25:58.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:58.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:58.266692+0000) 2024-03-06T14:25:59.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:25:59.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:59.266941+0000) 2024-03-06T14:25:59.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:59.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:00.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:00.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:00.267213+0000) 2024-03-06T14:26:01.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:01.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:01.267530+0000) 2024-03-06T14:26:02.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:02.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:02.267781+0000) 2024-03-06T14:26:03.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:03.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:03.267988+0000) 2024-03-06T14:26:04.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:04.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:04.268216+0000) 2024-03-06T14:26:05.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:05.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:05.268444+0000) 2024-03-06T14:26:06.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:06.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:06.268705+0000) 2024-03-06T14:26:07.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:07.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:07.268892+0000) 2024-03-06T14:26:08.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:08.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:08.269095+0000) 2024-03-06T14:26:09.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:09.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:09.269322+0000) 2024-03-06T14:26:10.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:10.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:10.269516+0000) 2024-03-06T14:26:11.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:11.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:11.269791+0000) 2024-03-06T14:26:12.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:12.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:12.269980+0000) 2024-03-06T14:26:13.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:13.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:13.270198+0000) 2024-03-06T14:26:14.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:14.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:14.270426+0000) 2024-03-06T14:26:15.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:15.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:15.270742+0000) 2024-03-06T14:26:16.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:16.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:16.271011+0000) 2024-03-06T14:26:17.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:17.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:17.271287+0000) 2024-03-06T14:26:18.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:18.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:18.271539+0000) 2024-03-06T14:26:19.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:19.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:19.271799+0000) 2024-03-06T14:26:20.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:20.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:20.272034+0000) 2024-03-06T14:26:21.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:21.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:21.272271+0000) 2024-03-06T14:26:22.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:22.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:22.272508+0000) 2024-03-06T14:26:23.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:23.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:23.272790+0000) 2024-03-06T14:26:24.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:24.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:24.272988+0000) 2024-03-06T14:26:25.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:25.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:25.273230+0000) 2024-03-06T14:26:26.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:26.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:26.273450+0000) 2024-03-06T14:26:27.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:27.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:27.273717+0000) 2024-03-06T14:26:28.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:28.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:28.273957+0000) 2024-03-06T14:26:29.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:29.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:29.274218+0000) 2024-03-06T14:26:29.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:29.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:30.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:30.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:30.274447+0000) 2024-03-06T14:26:31.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:31.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:31.274699+0000) 2024-03-06T14:26:32.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:32.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:32.274893+0000) 2024-03-06T14:26:33.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:33.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:33.275051+0000) 2024-03-06T14:26:34.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:34.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:34.275240+0000) 2024-03-06T14:26:35.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:35.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:35.275431+0000) 2024-03-06T14:26:36.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:36.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:36.275637+0000) 2024-03-06T14:26:37.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:37.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:37.275951+0000) 2024-03-06T14:26:38.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:38.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:38.276150+0000) 2024-03-06T14:26:39.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:39.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:39.276393+0000) 2024-03-06T14:26:40.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:40.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:40.276634+0000) 2024-03-06T14:26:41.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:41.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:41.276885+0000) 2024-03-06T14:26:42.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:42.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:42.277097+0000) 2024-03-06T14:26:43.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:43.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:43.277326+0000) 2024-03-06T14:26:44.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:44.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:44.277553+0000) 2024-03-06T14:26:45.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:45.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:45.277804+0000) 2024-03-06T14:26:46.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:46.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:46.278000+0000) 2024-03-06T14:26:47.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:47.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:47.278267+0000) 2024-03-06T14:26:48.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:48.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:48.278517+0000) 2024-03-06T14:26:49.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:49.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:49.278798+0000) 2024-03-06T14:26:50.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:50.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:50.278979+0000) 2024-03-06T14:26:51.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:51.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:51.279198+0000) 2024-03-06T14:26:52.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:52.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:52.279424+0000) 2024-03-06T14:26:53.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:53.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:53.279709+0000) 2024-03-06T14:26:54.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:54.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:54.279943+0000) 2024-03-06T14:26:55.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:55.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:55.280127+0000) 2024-03-06T14:26:56.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:56.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:56.280372+0000) 2024-03-06T14:26:57.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:57.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:57.280613+0000) 2024-03-06T14:26:58.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:58.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:58.280820+0000) 2024-03-06T14:26:59.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:26:59.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:59.281031+0000) 2024-03-06T14:27:00.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:00.082 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:00.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:00.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:00.281254+0000) 2024-03-06T14:27:01.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:01.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:01.281483+0000) 2024-03-06T14:27:02.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:02.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:02.281752+0000) 2024-03-06T14:27:03.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:03.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:03.281947+0000) 2024-03-06T14:27:04.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:04.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:04.282134+0000) 2024-03-06T14:27:05.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:05.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:05.282327+0000) 2024-03-06T14:27:06.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:06.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:06.282519+0000) 2024-03-06T14:27:07.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:07.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:07.282773+0000) 2024-03-06T14:27:08.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:08.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:08.283015+0000) 2024-03-06T14:27:09.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:09.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:09.283274+0000) 2024-03-06T14:27:10.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:10.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:10.283519+0000) 2024-03-06T14:27:11.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:11.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:11.283799+0000) 2024-03-06T14:27:12.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:12.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:12.283946+0000) 2024-03-06T14:27:13.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:13.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:13.284169+0000) 2024-03-06T14:27:14.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:14.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:14.284408+0000) 2024-03-06T14:27:15.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:15.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:15.284635+0000) 2024-03-06T14:27:16.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:16.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:16.284830+0000) 2024-03-06T14:27:17.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:17.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:17.285020+0000) 2024-03-06T14:27:18.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:18.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:18.285258+0000) 2024-03-06T14:27:19.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:19.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:19.285511+0000) 2024-03-06T14:27:20.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:20.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:20.285787+0000) 2024-03-06T14:27:21.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:21.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:21.286024+0000) 2024-03-06T14:27:22.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:22.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:22.286231+0000) 2024-03-06T14:27:23.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:23.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:23.286467+0000) 2024-03-06T14:27:24.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:24.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:24.286649+0000) 2024-03-06T14:27:25.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:25.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:25.286917+0000) 2024-03-06T14:27:26.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:26.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:26.287139+0000) 2024-03-06T14:27:27.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:27.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:27.287347+0000) 2024-03-06T14:27:28.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:28.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:28.287543+0000) 2024-03-06T14:27:29.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:29.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:29.287778+0000) 2024-03-06T14:27:30.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:30.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:30.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:30.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:30.287964+0000) 2024-03-06T14:27:31.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:31.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:31.288195+0000) 2024-03-06T14:27:32.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:32.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:32.288423+0000) 2024-03-06T14:27:33.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:33.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:33.288755+0000) 2024-03-06T14:27:34.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:34.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:34.289035+0000) 2024-03-06T14:27:35.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:35.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:35.289302+0000) 2024-03-06T14:27:36.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:36.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:36.289504+0000) 2024-03-06T14:27:37.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:37.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:37.289753+0000) 2024-03-06T14:27:38.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:38.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:38.289955+0000) 2024-03-06T14:27:39.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:39.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:39.290185+0000) 2024-03-06T14:27:40.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:40.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:40.290401+0000) 2024-03-06T14:27:41.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:41.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:41.290625+0000) 2024-03-06T14:27:42.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:42.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:42.290833+0000) 2024-03-06T14:27:43.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:43.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:43.291033+0000) 2024-03-06T14:27:44.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:44.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:44.291258+0000) 2024-03-06T14:27:45.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:45.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:45.291479+0000) 2024-03-06T14:27:46.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:46.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:46.291704+0000) 2024-03-06T14:27:47.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:47.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:47.291951+0000) 2024-03-06T14:27:48.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:48.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:48.292248+0000) 2024-03-06T14:27:49.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:49.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:49.292554+0000) 2024-03-06T14:27:50.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:50.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:50.292790+0000) 2024-03-06T14:27:51.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:51.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:51.292997+0000) 2024-03-06T14:27:52.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:52.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:52.293268+0000) 2024-03-06T14:27:53.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:53.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:53.293537+0000) 2024-03-06T14:27:54.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:54.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:54.293828+0000) 2024-03-06T14:27:55.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:55.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:55.294047+0000) 2024-03-06T14:27:56.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:56.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:56.294354+0000) 2024-03-06T14:27:57.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:57.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:57.294594+0000) 2024-03-06T14:27:58.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:58.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:58.294824+0000) 2024-03-06T14:27:59.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:27:59.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:59.295029+0000) 2024-03-06T14:28:00.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:00.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:00.295228+0000) 2024-03-06T14:28:00.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:00.473 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:01.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:01.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:01.295478+0000) 2024-03-06T14:28:02.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:02.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:02.295718+0000) 2024-03-06T14:28:03.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:03.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:03.295907+0000) 2024-03-06T14:28:04.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:04.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:04.296150+0000) 2024-03-06T14:28:05.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:05.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:05.296378+0000) 2024-03-06T14:28:06.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:06.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:06.296617+0000) 2024-03-06T14:28:07.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:07.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:07.296876+0000) 2024-03-06T14:28:08.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:08.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:08.297127+0000) 2024-03-06T14:28:09.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:09.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:09.297407+0000) 2024-03-06T14:28:10.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:10.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:10.297656+0000) 2024-03-06T14:28:11.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:11.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:11.297847+0000) 2024-03-06T14:28:12.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:12.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:12.298091+0000) 2024-03-06T14:28:13.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:13.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:13.298379+0000) 2024-03-06T14:28:14.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:14.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:14.298612+0000) 2024-03-06T14:28:15.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:15.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:15.298840+0000) 2024-03-06T14:28:16.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:16.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:16.299070+0000) 2024-03-06T14:28:17.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:17.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:17.299315+0000) 2024-03-06T14:28:18.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:18.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:18.299578+0000) 2024-03-06T14:28:19.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:19.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:19.299793+0000) 2024-03-06T14:28:20.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:20.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:20.300023+0000) 2024-03-06T14:28:21.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:21.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:21.300262+0000) 2024-03-06T14:28:22.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:22.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:22.300496+0000) 2024-03-06T14:28:23.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:23.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:23.300778+0000) 2024-03-06T14:28:24.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:24.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:24.301016+0000) 2024-03-06T14:28:25.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:25.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:25.301239+0000) 2024-03-06T14:28:26.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:26.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:26.301484+0000) 2024-03-06T14:28:27.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:27.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:27.301724+0000) 2024-03-06T14:28:28.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:28.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:28.301973+0000) 2024-03-06T14:28:29.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:29.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:29.302230+0000) 2024-03-06T14:28:30.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:30.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:30.302467+0000) 2024-03-06T14:28:30.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:30.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:31.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:31.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:31.302687+0000) 2024-03-06T14:28:32.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:32.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:32.302902+0000) 2024-03-06T14:28:33.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:33.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:33.303135+0000) 2024-03-06T14:28:34.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:34.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:34.303369+0000) 2024-03-06T14:28:35.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:35.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:35.303609+0000) 2024-03-06T14:28:36.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:36.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:36.303854+0000) 2024-03-06T14:28:37.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:37.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:37.304088+0000) 2024-03-06T14:28:38.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:38.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:38.304312+0000) 2024-03-06T14:28:39.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:39.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:39.304503+0000) 2024-03-06T14:28:40.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:40.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:40.304738+0000) 2024-03-06T14:28:41.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:41.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:41.304993+0000) 2024-03-06T14:28:42.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:42.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:42.305199+0000) 2024-03-06T14:28:43.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:43.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:43.305425+0000) 2024-03-06T14:28:44.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:44.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:44.305644+0000) 2024-03-06T14:28:45.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:45.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:45.305916+0000) 2024-03-06T14:28:46.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:46.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:46.306117+0000) 2024-03-06T14:28:47.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:47.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:47.306322+0000) 2024-03-06T14:28:48.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:48.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:48.306547+0000) 2024-03-06T14:28:49.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:49.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:49.306803+0000) 2024-03-06T14:28:50.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:50.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:50.307023+0000) 2024-03-06T14:28:51.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:51.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:51.307306+0000) 2024-03-06T14:28:52.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:52.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:52.307591+0000) 2024-03-06T14:28:53.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:53.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:53.307842+0000) 2024-03-06T14:28:54.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:54.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:54.308090+0000) 2024-03-06T14:28:55.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:55.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:55.308280+0000) 2024-03-06T14:28:56.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:56.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:56.308511+0000) 2024-03-06T14:28:57.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:57.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:57.308764+0000) 2024-03-06T14:28:58.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:58.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:58.309036+0000) 2024-03-06T14:28:59.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:28:59.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:59.309348+0000) 2024-03-06T14:29:00.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:00.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:00.309616+0000) 2024-03-06T14:29:00.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:00.864 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:01.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:01.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:01.309845+0000) 2024-03-06T14:29:02.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:02.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:02.310124+0000) 2024-03-06T14:29:03.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:03.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:03.310339+0000) 2024-03-06T14:29:04.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:04.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:04.310541+0000) 2024-03-06T14:29:05.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:05.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:05.310788+0000) 2024-03-06T14:29:06.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:06.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:06.311121+0000) 2024-03-06T14:29:07.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:07.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:07.311412+0000) 2024-03-06T14:29:08.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:08.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:08.311750+0000) 2024-03-06T14:29:09.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:09.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:09.312048+0000) 2024-03-06T14:29:10.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:10.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:10.312317+0000) 2024-03-06T14:29:11.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:11.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:11.312525+0000) 2024-03-06T14:29:12.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:12.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:12.312801+0000) 2024-03-06T14:29:13.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:13.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:13.313036+0000) 2024-03-06T14:29:14.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:14.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:14.313246+0000) 2024-03-06T14:29:15.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:15.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:15.313477+0000) 2024-03-06T14:29:16.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:16.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:16.313738+0000) 2024-03-06T14:29:17.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:17.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:17.313981+0000) 2024-03-06T14:29:18.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:18.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:18.314173+0000) 2024-03-06T14:29:19.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:19.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:19.314418+0000) 2024-03-06T14:29:20.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:20.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:20.314742+0000) 2024-03-06T14:29:21.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:21.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:21.315034+0000) 2024-03-06T14:29:22.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:22.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:22.315306+0000) 2024-03-06T14:29:23.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:23.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:23.315536+0000) 2024-03-06T14:29:24.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:24.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:24.315777+0000) 2024-03-06T14:29:25.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:25.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:25.316020+0000) 2024-03-06T14:29:26.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:26.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:26.316230+0000) 2024-03-06T14:29:27.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:27.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:27.316476+0000) 2024-03-06T14:29:28.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:28.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:28.316754+0000) 2024-03-06T14:29:29.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:29.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:29.316980+0000) 2024-03-06T14:29:30.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:30.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:30.317222+0000) 2024-03-06T14:29:31.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:31.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:31.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:31.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:31.317489+0000) 2024-03-06T14:29:32.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:32.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:32.317815+0000) 2024-03-06T14:29:33.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:33.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:33.318095+0000) 2024-03-06T14:29:34.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:34.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:34.318338+0000) 2024-03-06T14:29:35.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:35.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:35.318537+0000) 2024-03-06T14:29:36.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:36.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:36.318790+0000) 2024-03-06T14:29:37.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:37.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:37.319038+0000) 2024-03-06T14:29:38.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:38.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:38.319237+0000) 2024-03-06T14:29:39.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:39.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:39.319482+0000) 2024-03-06T14:29:40.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:40.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:40.319701+0000) 2024-03-06T14:29:41.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:41.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:41.319918+0000) 2024-03-06T14:29:42.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:42.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:42.320171+0000) 2024-03-06T14:29:43.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:43.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:43.320375+0000) 2024-03-06T14:29:44.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:44.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:44.320615+0000) 2024-03-06T14:29:45.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:45.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:45.320844+0000) 2024-03-06T14:29:46.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:46.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:46.321042+0000) 2024-03-06T14:29:47.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:47.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:47.321280+0000) 2024-03-06T14:29:48.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:48.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:48.321552+0000) 2024-03-06T14:29:49.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:49.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:49.321861+0000) 2024-03-06T14:29:50.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:50.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:50.322093+0000) 2024-03-06T14:29:51.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:51.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:51.322332+0000) 2024-03-06T14:29:52.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:52.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:52.322618+0000) 2024-03-06T14:29:53.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:53.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:53.322907+0000) 2024-03-06T14:29:54.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:54.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:54.323190+0000) 2024-03-06T14:29:55.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:55.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:55.323393+0000) 2024-03-06T14:29:56.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:56.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:56.323585+0000) 2024-03-06T14:29:57.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:57.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:57.323834+0000) 2024-03-06T14:29:58.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:58.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:58.324116+0000) 2024-03-06T14:29:59.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:29:59.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:59.324337+0000) 2024-03-06T14:30:00.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:00.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:00.324571+0000) 2024-03-06T14:30:01.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:01.250 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:01.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:01.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:01.324733+0000) 2024-03-06T14:30:02.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:02.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:02.324992+0000) 2024-03-06T14:30:03.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:03.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:03.325236+0000) 2024-03-06T14:30:04.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:04.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:04.325465+0000) 2024-03-06T14:30:05.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:05.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:05.325690+0000) 2024-03-06T14:30:06.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:06.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:06.325896+0000) 2024-03-06T14:30:07.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:07.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:07.326133+0000) 2024-03-06T14:30:08.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:08.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:08.326362+0000) 2024-03-06T14:30:09.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:09.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:09.326606+0000) 2024-03-06T14:30:10.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:10.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:10.326837+0000) 2024-03-06T14:30:11.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:11.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:11.327065+0000) 2024-03-06T14:30:12.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:12.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:12.327272+0000) 2024-03-06T14:30:13.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:13.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:13.327515+0000) 2024-03-06T14:30:14.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:14.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:14.327739+0000) 2024-03-06T14:30:15.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:15.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:15.327967+0000) 2024-03-06T14:30:16.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:16.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:16.328195+0000) 2024-03-06T14:30:17.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:17.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:17.328438+0000) 2024-03-06T14:30:18.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:18.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:18.328728+0000) 2024-03-06T14:30:19.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:19.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:19.328976+0000) 2024-03-06T14:30:20.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:20.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:20.329197+0000) 2024-03-06T14:30:21.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:21.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:21.329404+0000) 2024-03-06T14:30:22.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:22.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:22.329645+0000) 2024-03-06T14:30:23.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:23.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:23.329881+0000) 2024-03-06T14:30:24.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:24.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:24.330072+0000) 2024-03-06T14:30:25.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:25.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:25.330267+0000) 2024-03-06T14:30:26.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:26.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:26.330492+0000) 2024-03-06T14:30:27.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:27.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:27.330738+0000) 2024-03-06T14:30:28.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:28.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:28.330983+0000) 2024-03-06T14:30:29.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:29.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:29.331219+0000) 2024-03-06T14:30:30.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:30.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:30.331441+0000) 2024-03-06T14:30:31.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:31.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:31.331635+0000) 2024-03-06T14:30:31.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:31.403 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:32.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:32.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:32.331892+0000) 2024-03-06T14:30:33.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:33.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:33.332143+0000) 2024-03-06T14:30:34.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:34.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:34.332343+0000) 2024-03-06T14:30:35.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:35.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:35.332592+0000) 2024-03-06T14:30:36.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:36.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:36.332867+0000) 2024-03-06T14:30:37.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:37.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:37.333142+0000) 2024-03-06T14:30:38.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:38.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:38.333425+0000) 2024-03-06T14:30:39.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:39.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:39.333765+0000) 2024-03-06T14:30:40.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:40.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:40.333979+0000) 2024-03-06T14:30:41.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:41.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:41.334250+0000) 2024-03-06T14:30:42.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:42.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:42.334528+0000) 2024-03-06T14:30:43.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:43.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:43.334839+0000) 2024-03-06T14:30:44.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:44.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:44.335121+0000) 2024-03-06T14:30:45.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:45.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:45.335399+0000) 2024-03-06T14:30:46.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:46.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:46.335614+0000) 2024-03-06T14:30:47.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:47.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:47.335868+0000) 2024-03-06T14:30:48.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:48.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:48.336205+0000) 2024-03-06T14:30:49.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:49.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:49.336502+0000) 2024-03-06T14:30:50.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:50.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:50.336818+0000) 2024-03-06T14:30:51.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:51.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:51.337055+0000) 2024-03-06T14:30:52.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:52.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:52.337339+0000) 2024-03-06T14:30:53.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:53.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:53.337617+0000) 2024-03-06T14:30:54.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:54.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:54.337881+0000) 2024-03-06T14:30:55.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:55.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:55.338092+0000) 2024-03-06T14:30:56.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:56.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:56.338300+0000) 2024-03-06T14:30:57.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:57.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:57.338543+0000) 2024-03-06T14:30:58.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:58.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:58.338781+0000) 2024-03-06T14:30:59.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:30:59.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:59.338987+0000) 2024-03-06T14:31:00.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:00.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:00.339214+0000) 2024-03-06T14:31:01.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:01.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:01.339432+0000) 2024-03-06T14:31:01.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:01.540 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:02.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:02.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:02.339760+0000) 2024-03-06T14:31:03.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:03.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:03.339968+0000) 2024-03-06T14:31:04.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:04.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:04.340205+0000) 2024-03-06T14:31:05.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:05.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:05.340431+0000) 2024-03-06T14:31:06.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:06.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:06.340714+0000) 2024-03-06T14:31:07.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:07.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:07.340967+0000) 2024-03-06T14:31:08.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:08.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:08.341196+0000) 2024-03-06T14:31:09.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:09.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:09.341428+0000) 2024-03-06T14:31:10.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:10.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:10.341618+0000) 2024-03-06T14:31:11.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:11.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:11.341833+0000) 2024-03-06T14:31:12.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:12.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:12.342060+0000) 2024-03-06T14:31:13.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:13.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:13.342306+0000) 2024-03-06T14:31:14.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:14.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:14.342532+0000) 2024-03-06T14:31:15.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:15.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:15.342787+0000) 2024-03-06T14:31:16.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:16.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:16.343012+0000) 2024-03-06T14:31:17.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:17.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:17.343278+0000) 2024-03-06T14:31:18.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:18.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:18.343573+0000) 2024-03-06T14:31:19.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:19.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:19.343793+0000) 2024-03-06T14:31:20.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:20.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:20.343985+0000) 2024-03-06T14:31:21.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:21.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:21.344230+0000) 2024-03-06T14:31:22.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:22.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:22.344463+0000) 2024-03-06T14:31:23.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:23.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:23.344734+0000) 2024-03-06T14:31:24.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:24.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:24.344958+0000) 2024-03-06T14:31:25.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:25.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:25.345190+0000) 2024-03-06T14:31:26.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:26.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:26.345412+0000) 2024-03-06T14:31:27.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:27.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:27.345604+0000) 2024-03-06T14:31:28.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:28.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:28.345860+0000) 2024-03-06T14:31:29.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:29.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:29.346155+0000) 2024-03-06T14:31:30.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:30.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:30.346424+0000) 2024-03-06T14:31:31.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:31.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:31.346624+0000) 2024-03-06T14:31:31.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:31.678 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:32.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:32.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:32.346861+0000) 2024-03-06T14:31:33.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:33.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:33.347098+0000) 2024-03-06T14:31:34.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:34.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:34.347348+0000) 2024-03-06T14:31:35.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:35.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:35.347623+0000) 2024-03-06T14:31:36.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:36.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:36.347955+0000) 2024-03-06T14:31:37.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:37.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:37.348186+0000) 2024-03-06T14:31:38.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:38.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:38.348379+0000) 2024-03-06T14:31:39.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:39.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:39.348709+0000) 2024-03-06T14:31:40.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:40.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:40.348977+0000) 2024-03-06T14:31:41.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:41.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:41.349204+0000) 2024-03-06T14:31:42.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:42.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:42.349432+0000) 2024-03-06T14:31:43.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:43.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:43.349693+0000) 2024-03-06T14:31:44.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:44.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:44.349887+0000) 2024-03-06T14:31:45.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:45.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:45.350116+0000) 2024-03-06T14:31:46.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:46.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:46.350302+0000) 2024-03-06T14:31:47.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:47.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:47.350538+0000) 2024-03-06T14:31:48.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:48.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:48.350804+0000) 2024-03-06T14:31:49.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:49.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:49.351037+0000) 2024-03-06T14:31:50.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:50.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:50.351258+0000) 2024-03-06T14:31:51.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:51.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:51.351445+0000) 2024-03-06T14:31:52.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:52.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:52.351653+0000) 2024-03-06T14:31:53.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:53.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:53.351949+0000) 2024-03-06T14:31:54.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:54.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:54.352187+0000) 2024-03-06T14:31:55.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:55.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:55.352354+0000) 2024-03-06T14:31:56.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:56.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:56.352583+0000) 2024-03-06T14:31:57.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:57.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:57.352820+0000) 2024-03-06T14:31:58.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:58.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:58.353015+0000) 2024-03-06T14:31:59.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:31:59.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:59.353255+0000) 2024-03-06T14:32:00.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:00.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:00.353469+0000) 2024-03-06T14:32:01.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:01.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:01.353773+0000) 2024-03-06T14:32:01.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:01.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:02.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:02.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:02.354012+0000) 2024-03-06T14:32:03.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:03.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:03.354242+0000) 2024-03-06T14:32:04.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:04.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:04.354464+0000) 2024-03-06T14:32:05.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:05.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:05.354727+0000) 2024-03-06T14:32:06.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:06.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:06.354951+0000) 2024-03-06T14:32:07.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:07.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:07.355176+0000) 2024-03-06T14:32:08.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:08.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:08.355407+0000) 2024-03-06T14:32:09.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:09.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:09.355590+0000) 2024-03-06T14:32:10.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:10.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:10.355775+0000) 2024-03-06T14:32:11.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:11.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:11.356016+0000) 2024-03-06T14:32:12.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:12.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:12.356211+0000) 2024-03-06T14:32:13.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:13.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:13.356432+0000) 2024-03-06T14:32:14.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:14.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:14.356648+0000) 2024-03-06T14:32:15.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:15.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:15.356899+0000) 2024-03-06T14:32:16.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:16.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:16.357118+0000) 2024-03-06T14:32:17.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:17.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:17.357358+0000) 2024-03-06T14:32:18.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:18.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:18.357592+0000) 2024-03-06T14:32:19.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:19.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:19.357834+0000) 2024-03-06T14:32:20.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:20.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:20.358055+0000) 2024-03-06T14:32:21.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:21.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:21.358286+0000) 2024-03-06T14:32:22.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:22.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:22.358506+0000) 2024-03-06T14:32:23.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:23.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:23.358776+0000) 2024-03-06T14:32:24.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:24.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:24.359032+0000) 2024-03-06T14:32:25.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:25.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:25.359246+0000) 2024-03-06T14:32:26.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:26.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:26.359445+0000) 2024-03-06T14:32:27.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:27.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:27.359707+0000) 2024-03-06T14:32:28.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:28.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:28.359952+0000) 2024-03-06T14:32:29.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:29.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:29.360193+0000) 2024-03-06T14:32:30.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:30.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:30.360414+0000) 2024-03-06T14:32:31.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:31.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:31.360634+0000) 2024-03-06T14:32:31.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:31.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:32.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:32.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:32.360882+0000) 2024-03-06T14:32:33.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:33.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:33.361120+0000) 2024-03-06T14:32:34.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:34.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:34.361348+0000) 2024-03-06T14:32:35.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:35.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:35.361573+0000) 2024-03-06T14:32:36.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:36.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:36.361807+0000) 2024-03-06T14:32:37.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:37.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:37.362028+0000) 2024-03-06T14:32:38.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:38.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:38.362263+0000) 2024-03-06T14:32:39.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:39.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:39.362499+0000) 2024-03-06T14:32:40.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:40.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:40.362647+0000) 2024-03-06T14:32:41.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:41.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:41.362853+0000) 2024-03-06T14:32:42.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:42.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:42.363080+0000) 2024-03-06T14:32:43.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:43.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:43.363311+0000) 2024-03-06T14:32:44.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:44.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:44.363535+0000) 2024-03-06T14:32:45.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:45.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:45.363801+0000) 2024-03-06T14:32:46.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:46.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:46.364028+0000) 2024-03-06T14:32:47.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:47.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:47.364255+0000) 2024-03-06T14:32:48.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:48.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:48.364493+0000) 2024-03-06T14:32:49.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:49.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:49.364758+0000) 2024-03-06T14:32:50.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:50.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:50.364998+0000) 2024-03-06T14:32:51.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:51.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:51.365183+0000) 2024-03-06T14:32:52.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:52.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:52.365404+0000) 2024-03-06T14:32:53.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:53.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:53.365637+0000) 2024-03-06T14:32:54.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:54.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:54.365857+0000) 2024-03-06T14:32:55.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:55.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:55.366077+0000) 2024-03-06T14:32:56.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:56.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:56.366268+0000) 2024-03-06T14:32:57.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:57.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:57.366504+0000) 2024-03-06T14:32:58.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:58.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:58.366792+0000) 2024-03-06T14:32:59.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:32:59.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:59.367030+0000) 2024-03-06T14:33:00.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:00.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:00.367215+0000) 2024-03-06T14:33:01.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:01.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:01.367458+0000) 2024-03-06T14:33:02.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:02.202 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:02.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:02.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:02.367735+0000) 2024-03-06T14:33:03.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:03.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:03.367979+0000) 2024-03-06T14:33:04.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:04.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:04.368207+0000) 2024-03-06T14:33:05.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:05.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:05.368392+0000) 2024-03-06T14:33:06.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:06.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:06.368630+0000) 2024-03-06T14:33:07.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:07.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:07.368861+0000) 2024-03-06T14:33:08.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:08.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:08.369083+0000) 2024-03-06T14:33:09.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:09.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:09.369280+0000) 2024-03-06T14:33:10.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:10.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:10.369499+0000) 2024-03-06T14:33:11.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:11.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:11.369766+0000) 2024-03-06T14:33:12.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:12.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:12.370011+0000) 2024-03-06T14:33:13.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:13.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:13.370246+0000) 2024-03-06T14:33:14.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:14.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:14.370442+0000) 2024-03-06T14:33:15.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:15.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:15.370655+0000) 2024-03-06T14:33:16.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:16.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:16.370892+0000) 2024-03-06T14:33:17.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:17.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:17.371182+0000) 2024-03-06T14:33:18.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:18.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:18.371426+0000) 2024-03-06T14:33:19.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:19.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:19.371700+0000) 2024-03-06T14:33:20.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:20.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:20.371925+0000) 2024-03-06T14:33:21.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:21.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:21.372155+0000) 2024-03-06T14:33:22.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:22.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:22.372348+0000) 2024-03-06T14:33:23.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:23.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:23.372596+0000) 2024-03-06T14:33:24.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:24.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:24.372833+0000) 2024-03-06T14:33:25.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:25.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:25.373056+0000) 2024-03-06T14:33:26.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:26.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:26.373276+0000) 2024-03-06T14:33:27.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:27.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:27.373501+0000) 2024-03-06T14:33:28.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:28.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:28.373786+0000) 2024-03-06T14:33:29.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:29.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:29.374034+0000) 2024-03-06T14:33:30.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:30.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:30.374208+0000) 2024-03-06T14:33:31.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:31.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:31.374433+0000) 2024-03-06T14:33:32.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:32.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:32.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:32.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:32.374578+0000) 2024-03-06T14:33:33.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:33.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:33.374814+0000) 2024-03-06T14:33:34.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:34.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:34.375040+0000) 2024-03-06T14:33:35.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:35.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:35.375277+0000) 2024-03-06T14:33:36.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:36.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:36.375520+0000) 2024-03-06T14:33:37.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:37.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:37.375785+0000) 2024-03-06T14:33:38.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:38.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:38.376032+0000) 2024-03-06T14:33:39.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:39.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:39.376228+0000) 2024-03-06T14:33:40.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:40.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:40.376449+0000) 2024-03-06T14:33:41.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:41.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:41.376706+0000) 2024-03-06T14:33:42.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:42.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:42.376942+0000) 2024-03-06T14:33:43.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:43.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:43.377135+0000) 2024-03-06T14:33:44.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:44.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:44.377365+0000) 2024-03-06T14:33:45.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:45.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:45.377595+0000) 2024-03-06T14:33:46.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:46.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:46.377782+0000) 2024-03-06T14:33:47.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:47.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:47.378012+0000) 2024-03-06T14:33:48.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:48.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:48.378226+0000) 2024-03-06T14:33:49.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:49.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:49.378482+0000) 2024-03-06T14:33:50.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:50.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:50.378753+0000) 2024-03-06T14:33:51.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:51.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:51.378986+0000) 2024-03-06T14:33:52.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:52.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:52.379214+0000) 2024-03-06T14:33:53.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:53.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:53.379415+0000) 2024-03-06T14:33:54.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:54.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:54.379608+0000) 2024-03-06T14:33:55.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:55.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:55.379842+0000) 2024-03-06T14:33:56.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:56.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:56.380077+0000) 2024-03-06T14:33:57.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:57.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:57.380294+0000) 2024-03-06T14:33:58.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:58.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:58.380531+0000) 2024-03-06T14:33:59.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:33:59.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:59.380787+0000) 2024-03-06T14:34:00.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:00.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:00.380966+0000) 2024-03-06T14:34:01.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:01.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:01.381194+0000) 2024-03-06T14:34:02.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:02.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:02.381427+0000) 2024-03-06T14:34:02.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:02.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:03.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:03.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:03.381616+0000) 2024-03-06T14:34:04.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:04.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:04.381870+0000) 2024-03-06T14:34:05.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:05.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:05.382122+0000) 2024-03-06T14:34:06.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:06.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:06.382346+0000) 2024-03-06T14:34:07.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:07.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:07.382553+0000) 2024-03-06T14:34:08.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:08.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:08.382797+0000) 2024-03-06T14:34:09.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:09.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:09.383042+0000) 2024-03-06T14:34:10.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:10.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:10.383272+0000) 2024-03-06T14:34:11.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:11.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:11.383463+0000) 2024-03-06T14:34:12.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:12.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:12.383693+0000) 2024-03-06T14:34:13.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:13.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:13.383912+0000) 2024-03-06T14:34:14.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:14.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:14.384152+0000) 2024-03-06T14:34:15.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:15.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:15.384354+0000) 2024-03-06T14:34:16.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:16.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:16.384586+0000) 2024-03-06T14:34:17.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:17.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:17.384812+0000) 2024-03-06T14:34:18.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:18.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:18.385003+0000) 2024-03-06T14:34:19.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:19.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:19.385241+0000) 2024-03-06T14:34:20.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:20.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:20.385476+0000) 2024-03-06T14:34:21.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:21.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:21.385765+0000) 2024-03-06T14:34:22.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:22.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:22.385964+0000) 2024-03-06T14:34:23.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:23.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:23.386174+0000) 2024-03-06T14:34:24.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:24.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:24.386409+0000) 2024-03-06T14:34:25.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:25.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:25.386641+0000) 2024-03-06T14:34:26.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:26.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:26.386867+0000) 2024-03-06T14:34:27.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:27.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:27.387097+0000) 2024-03-06T14:34:28.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:28.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:28.387320+0000) 2024-03-06T14:34:29.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:29.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:29.387551+0000) 2024-03-06T14:34:30.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:30.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:30.387771+0000) 2024-03-06T14:34:31.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:31.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:31.388014+0000) 2024-03-06T14:34:32.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:32.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:32.388180+0000) 2024-03-06T14:34:32.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:32.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:33.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:33.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:33.388386+0000) 2024-03-06T14:34:34.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:34.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:34.388628+0000) 2024-03-06T14:34:35.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:35.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:35.388868+0000) 2024-03-06T14:34:36.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:36.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:36.389095+0000) 2024-03-06T14:34:37.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:37.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:37.389316+0000) 2024-03-06T14:34:38.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:38.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:38.389557+0000) 2024-03-06T14:34:39.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:39.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:39.389821+0000) 2024-03-06T14:34:40.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:40.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:40.390001+0000) 2024-03-06T14:34:41.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:41.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:41.390204+0000) 2024-03-06T14:34:42.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:42.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:42.390436+0000) 2024-03-06T14:34:43.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:43.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:43.390641+0000) 2024-03-06T14:34:44.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:44.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:44.390884+0000) 2024-03-06T14:34:45.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:45.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:45.391128+0000) 2024-03-06T14:34:46.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:46.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:46.391322+0000) 2024-03-06T14:34:47.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:47.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:47.391563+0000) 2024-03-06T14:34:48.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:48.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:48.391803+0000) 2024-03-06T14:34:49.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:49.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:49.392001+0000) 2024-03-06T14:34:50.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:50.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:50.392231+0000) 2024-03-06T14:34:51.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:51.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:51.392427+0000) 2024-03-06T14:34:52.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:52.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:52.392655+0000) 2024-03-06T14:34:53.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:53.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:53.392991+0000) 2024-03-06T14:34:54.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:54.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:54.393220+0000) 2024-03-06T14:34:55.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:55.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:55.393442+0000) 2024-03-06T14:34:56.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:56.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:56.393712+0000) 2024-03-06T14:34:57.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:57.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:57.393885+0000) 2024-03-06T14:34:58.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:58.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:58.394136+0000) 2024-03-06T14:34:59.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:34:59.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:59.394380+0000) 2024-03-06T14:35:00.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:00.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:00.394599+0000) 2024-03-06T14:35:01.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:01.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:01.394840+0000) 2024-03-06T14:35:02.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:02.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:02.395064+0000) 2024-03-06T14:35:02.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:02.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:03.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:03.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:03.395264+0000) 2024-03-06T14:35:04.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:04.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:04.395514+0000) 2024-03-06T14:35:05.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:05.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:05.395794+0000) 2024-03-06T14:35:06.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:06.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:06.396022+0000) 2024-03-06T14:35:07.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:07.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:07.396254+0000) 2024-03-06T14:35:08.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:08.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:08.396487+0000) 2024-03-06T14:35:09.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:09.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:09.396781+0000) 2024-03-06T14:35:10.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:10.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:10.397030+0000) 2024-03-06T14:35:11.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:11.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:11.397266+0000) 2024-03-06T14:35:12.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:12.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:12.397502+0000) 2024-03-06T14:35:13.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:13.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:13.397734+0000) 2024-03-06T14:35:14.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:14.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:14.397974+0000) 2024-03-06T14:35:15.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:15.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:15.398181+0000) 2024-03-06T14:35:16.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:16.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:16.398408+0000) 2024-03-06T14:35:17.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:17.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:17.398647+0000) 2024-03-06T14:35:18.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:18.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:18.398920+0000) 2024-03-06T14:35:19.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:19.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:19.399137+0000) 2024-03-06T14:35:20.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:20.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:20.399366+0000) 2024-03-06T14:35:21.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:21.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:21.399603+0000) 2024-03-06T14:35:22.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:22.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:22.399843+0000) 2024-03-06T14:35:23.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:23.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:23.400085+0000) 2024-03-06T14:35:24.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:24.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:24.400310+0000) 2024-03-06T14:35:25.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:25.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:25.400536+0000) 2024-03-06T14:35:26.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:26.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:26.400775+0000) 2024-03-06T14:35:27.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:27.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:27.401020+0000) 2024-03-06T14:35:28.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:28.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:28.401261+0000) 2024-03-06T14:35:29.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:29.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:29.401508+0000) 2024-03-06T14:35:30.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:30.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:30.401782+0000) 2024-03-06T14:35:31.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:31.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:31.401974+0000) 2024-03-06T14:35:32.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:32.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:32.402204+0000) 2024-03-06T14:35:33.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:33.170 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:33.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:33.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:33.402457+0000) 2024-03-06T14:35:34.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:34.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:34.402724+0000) 2024-03-06T14:35:35.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:35.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:35.402958+0000) 2024-03-06T14:35:36.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:36.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:36.403149+0000) 2024-03-06T14:35:37.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:37.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:37.403377+0000) 2024-03-06T14:35:38.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:38.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:38.403571+0000) 2024-03-06T14:35:39.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:39.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:39.403781+0000) 2024-03-06T14:35:40.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:40.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:40.404012+0000) 2024-03-06T14:35:41.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:41.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:41.404249+0000) 2024-03-06T14:35:42.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:42.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:42.404471+0000) 2024-03-06T14:35:43.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:43.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:43.404751+0000) 2024-03-06T14:35:44.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:44.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:44.404991+0000) 2024-03-06T14:35:45.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:45.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:45.405174+0000) 2024-03-06T14:35:46.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:46.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:46.405458+0000) 2024-03-06T14:35:47.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:47.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:47.405641+0000) 2024-03-06T14:35:48.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:48.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:48.405928+0000) 2024-03-06T14:35:49.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:49.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:49.406174+0000) 2024-03-06T14:35:50.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:50.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:50.406361+0000) 2024-03-06T14:35:51.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:51.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:51.406601+0000) 2024-03-06T14:35:52.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:52.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:52.406831+0000) 2024-03-06T14:35:53.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:53.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:53.407024+0000) 2024-03-06T14:35:54.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:54.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:54.407264+0000) 2024-03-06T14:35:55.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:55.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:55.407456+0000) 2024-03-06T14:35:56.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:56.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:56.407635+0000) 2024-03-06T14:35:57.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:57.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:57.407909+0000) 2024-03-06T14:35:58.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:58.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:58.408106+0000) 2024-03-06T14:35:59.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:35:59.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:59.408301+0000) 2024-03-06T14:36:00.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:00.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:00.408519+0000) 2024-03-06T14:36:01.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:01.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:01.408776+0000) 2024-03-06T14:36:02.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:02.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:02.409014+0000) 2024-03-06T14:36:03.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:03.392 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:03.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:03.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:03.409240+0000) 2024-03-06T14:36:04.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:04.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:04.409440+0000) 2024-03-06T14:36:05.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:05.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:05.409582+0000) 2024-03-06T14:36:06.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:06.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:06.409841+0000) 2024-03-06T14:36:07.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:07.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:07.410080+0000) 2024-03-06T14:36:08.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:08.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:08.410338+0000) 2024-03-06T14:36:09.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:09.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:09.410604+0000) 2024-03-06T14:36:10.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:10.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:10.410836+0000) 2024-03-06T14:36:11.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:11.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:11.411040+0000) 2024-03-06T14:36:12.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:12.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:12.411233+0000) 2024-03-06T14:36:13.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:13.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:13.411471+0000) 2024-03-06T14:36:14.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:14.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:14.411713+0000) 2024-03-06T14:36:15.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:15.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:15.411956+0000) 2024-03-06T14:36:16.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:16.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:16.412165+0000) 2024-03-06T14:36:17.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:17.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:17.412408+0000) 2024-03-06T14:36:18.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:18.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:18.412603+0000) 2024-03-06T14:36:19.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:19.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:19.412873+0000) 2024-03-06T14:36:20.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:20.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:20.413113+0000) 2024-03-06T14:36:21.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:21.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:21.413313+0000) 2024-03-06T14:36:22.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:22.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:22.413533+0000) 2024-03-06T14:36:23.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:23.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:23.413780+0000) 2024-03-06T14:36:24.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:24.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:24.414035+0000) 2024-03-06T14:36:25.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:25.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:25.414279+0000) 2024-03-06T14:36:26.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:26.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:26.414474+0000) 2024-03-06T14:36:27.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:27.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:27.414734+0000) 2024-03-06T14:36:28.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:28.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:28.414967+0000) 2024-03-06T14:36:29.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:29.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:29.415204+0000) 2024-03-06T14:36:30.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:30.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:30.415420+0000) 2024-03-06T14:36:31.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:31.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:31.415578+0000) 2024-03-06T14:36:32.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:32.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:32.415796+0000) 2024-03-06T14:36:33.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:33.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:33.416047+0000) 2024-03-06T14:36:33.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:33.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:34.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:34.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:34.416310+0000) 2024-03-06T14:36:35.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:35.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:35.416554+0000) 2024-03-06T14:36:36.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:36.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:36.416782+0000) 2024-03-06T14:36:37.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:37.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:37.416987+0000) 2024-03-06T14:36:38.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:38.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:38.417216+0000) 2024-03-06T14:36:39.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:39.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:39.417465+0000) 2024-03-06T14:36:40.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:40.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:40.417739+0000) 2024-03-06T14:36:41.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:41.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:41.417976+0000) 2024-03-06T14:36:42.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:42.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:42.418205+0000) 2024-03-06T14:36:43.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:43.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:43.418455+0000) 2024-03-06T14:36:44.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:44.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:44.418707+0000) 2024-03-06T14:36:45.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:45.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:45.418950+0000) 2024-03-06T14:36:46.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:46.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:46.419154+0000) 2024-03-06T14:36:47.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:47.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:47.419386+0000) 2024-03-06T14:36:48.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:48.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:48.419621+0000) 2024-03-06T14:36:49.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:49.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:49.419841+0000) 2024-03-06T14:36:50.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:50.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:50.420038+0000) 2024-03-06T14:36:51.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:51.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:51.420225+0000) 2024-03-06T14:36:52.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:52.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:52.420450+0000) 2024-03-06T14:36:53.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:53.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:53.420699+0000) 2024-03-06T14:36:54.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:54.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:54.420932+0000) 2024-03-06T14:36:55.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:55.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:55.421194+0000) 2024-03-06T14:36:56.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:56.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:56.421425+0000) 2024-03-06T14:36:57.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:57.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:57.421652+0000) 2024-03-06T14:36:58.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:58.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:58.421937+0000) 2024-03-06T14:36:59.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:36:59.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:59.422160+0000) 2024-03-06T14:37:00.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:00.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:00.422346+0000) 2024-03-06T14:37:01.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:01.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:01.422586+0000) 2024-03-06T14:37:02.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:02.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:02.422824+0000) 2024-03-06T14:37:03.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:03.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:03.423041+0000) 2024-03-06T14:37:03.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:03.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:04.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:04.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:04.423285+0000) 2024-03-06T14:37:05.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:05.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:05.423505+0000) 2024-03-06T14:37:06.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:06.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:06.423787+0000) 2024-03-06T14:37:07.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:07.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:07.424016+0000) 2024-03-06T14:37:08.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:08.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:08.424251+0000) 2024-03-06T14:37:09.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:09.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:09.424491+0000) 2024-03-06T14:37:10.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:10.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:10.424762+0000) 2024-03-06T14:37:11.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:11.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:11.424985+0000) 2024-03-06T14:37:12.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:12.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:12.425182+0000) 2024-03-06T14:37:13.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:13.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:13.425460+0000) 2024-03-06T14:37:14.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:14.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:14.425791+0000) 2024-03-06T14:37:15.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:15.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:15.426043+0000) 2024-03-06T14:37:16.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:16.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:16.426324+0000) 2024-03-06T14:37:17.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:17.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:17.426605+0000) 2024-03-06T14:37:18.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:18.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:18.426858+0000) 2024-03-06T14:37:19.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:19.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:19.427131+0000) 2024-03-06T14:37:20.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:20.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:20.427406+0000) 2024-03-06T14:37:21.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:21.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:21.427732+0000) 2024-03-06T14:37:22.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:22.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:22.428016+0000) 2024-03-06T14:37:23.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:23.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:23.428306+0000) 2024-03-06T14:37:24.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:24.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:24.428594+0000) 2024-03-06T14:37:25.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:25.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:25.428800+0000) 2024-03-06T14:37:26.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:26.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:26.429029+0000) 2024-03-06T14:37:27.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:27.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:27.429255+0000) 2024-03-06T14:37:28.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:28.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:28.429578+0000) 2024-03-06T14:37:29.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:29.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:29.429790+0000) 2024-03-06T14:37:30.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:30.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:30.430027+0000) 2024-03-06T14:37:31.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:31.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:31.430251+0000) 2024-03-06T14:37:32.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:32.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:32.430485+0000) 2024-03-06T14:37:33.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:33.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:33.430760+0000) 2024-03-06T14:37:34.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:34.055 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:34.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:34.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:34.430944+0000) 2024-03-06T14:37:35.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:35.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:35.431146+0000) 2024-03-06T14:37:36.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:36.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:36.431387+0000) 2024-03-06T14:37:37.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:37.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:37.431644+0000) 2024-03-06T14:37:38.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:38.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:38.431975+0000) 2024-03-06T14:37:39.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:39.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:39.432274+0000) 2024-03-06T14:37:40.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:40.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:40.432544+0000) 2024-03-06T14:37:41.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:41.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:41.432832+0000) 2024-03-06T14:37:42.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:42.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:42.433107+0000) 2024-03-06T14:37:43.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:43.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:43.433393+0000) 2024-03-06T14:37:44.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:44.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:44.433722+0000) 2024-03-06T14:37:45.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:45.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:45.434004+0000) 2024-03-06T14:37:46.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:46.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:46.434285+0000) 2024-03-06T14:37:47.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:47.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:47.434567+0000) 2024-03-06T14:37:48.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:48.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:48.434855+0000) 2024-03-06T14:37:49.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:49.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:49.435147+0000) 2024-03-06T14:37:50.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:50.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:50.435338+0000) 2024-03-06T14:37:51.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:51.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:51.435569+0000) 2024-03-06T14:37:52.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:52.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:52.435794+0000) 2024-03-06T14:37:53.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:53.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:53.436029+0000) 2024-03-06T14:37:54.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:54.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:54.436231+0000) 2024-03-06T14:37:55.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:55.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:55.436455+0000) 2024-03-06T14:37:56.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:56.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:56.436713+0000) 2024-03-06T14:37:57.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:57.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:57.436941+0000) 2024-03-06T14:37:58.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:58.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:58.437171+0000) 2024-03-06T14:37:59.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:37:59.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:59.437415+0000) 2024-03-06T14:38:00.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:00.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:00.437647+0000) 2024-03-06T14:38:01.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:01.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:01.437916+0000) 2024-03-06T14:38:02.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:02.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:02.438148+0000) 2024-03-06T14:38:03.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:03.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:03.438342+0000) 2024-03-06T14:38:04.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:04.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:04.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:04.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:04.438557+0000) 2024-03-06T14:38:05.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:05.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:05.438803+0000) 2024-03-06T14:38:06.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:06.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:06.439029+0000) 2024-03-06T14:38:07.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:07.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:07.439206+0000) 2024-03-06T14:38:08.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:08.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:08.439431+0000) 2024-03-06T14:38:09.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:09.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:09.439709+0000) 2024-03-06T14:38:10.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:10.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:10.439952+0000) 2024-03-06T14:38:11.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:11.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:11.440140+0000) 2024-03-06T14:38:12.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:12.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:12.440371+0000) 2024-03-06T14:38:13.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:13.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:13.440602+0000) 2024-03-06T14:38:14.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:14.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:14.440850+0000) 2024-03-06T14:38:15.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:15.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:15.441092+0000) 2024-03-06T14:38:16.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:16.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:16.441366+0000) 2024-03-06T14:38:17.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:17.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:17.441649+0000) 2024-03-06T14:38:18.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:18.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:18.441978+0000) 2024-03-06T14:38:19.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:19.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:19.442251+0000) 2024-03-06T14:38:20.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:20.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:20.442495+0000) 2024-03-06T14:38:21.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:21.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:21.442771+0000) 2024-03-06T14:38:22.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:22.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:22.443018+0000) 2024-03-06T14:38:23.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:23.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:23.443299+0000) 2024-03-06T14:38:24.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:24.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:24.443587+0000) 2024-03-06T14:38:25.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:25.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:25.443795+0000) 2024-03-06T14:38:26.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:26.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:26.443998+0000) 2024-03-06T14:38:27.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:27.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:27.444248+0000) 2024-03-06T14:38:28.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:28.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:28.444482+0000) 2024-03-06T14:38:29.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:29.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:29.444761+0000) 2024-03-06T14:38:30.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:30.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:30.444979+0000) 2024-03-06T14:38:31.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:31.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:31.445238+0000) 2024-03-06T14:38:32.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:32.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:32.445445+0000) 2024-03-06T14:38:33.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:33.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:33.445771+0000) 2024-03-06T14:38:34.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:34.421 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:34.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:34.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:34.445986+0000) 2024-03-06T14:38:35.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:35.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:35.446178+0000) 2024-03-06T14:38:36.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:36.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:36.446399+0000) 2024-03-06T14:38:37.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:37.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:37.446594+0000) 2024-03-06T14:38:38.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:38.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:38.446812+0000) 2024-03-06T14:38:39.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:39.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:39.447073+0000) 2024-03-06T14:38:40.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:40.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:40.447260+0000) 2024-03-06T14:38:41.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:41.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:41.447489+0000) 2024-03-06T14:38:42.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:42.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:42.447750+0000) 2024-03-06T14:38:43.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:43.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:43.447915+0000) 2024-03-06T14:38:44.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:44.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:44.448158+0000) 2024-03-06T14:38:45.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:45.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:45.448396+0000) 2024-03-06T14:38:46.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:46.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:46.448631+0000) 2024-03-06T14:38:47.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:47.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:47.448912+0000) 2024-03-06T14:38:48.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:48.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:48.449133+0000) 2024-03-06T14:38:49.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:49.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:49.449368+0000) 2024-03-06T14:38:50.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:50.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:50.449620+0000) 2024-03-06T14:38:51.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:51.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:51.449864+0000) 2024-03-06T14:38:52.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:52.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:52.450101+0000) 2024-03-06T14:38:53.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:53.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:53.450355+0000) 2024-03-06T14:38:54.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:54.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:54.450608+0000) 2024-03-06T14:38:55.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:55.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:55.450849+0000) 2024-03-06T14:38:56.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:56.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:56.451070+0000) 2024-03-06T14:38:57.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:57.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:57.451315+0000) 2024-03-06T14:38:58.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:58.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:58.451508+0000) 2024-03-06T14:38:59.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:38:59.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:59.451759+0000) 2024-03-06T14:39:00.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:00.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:00.451986+0000) 2024-03-06T14:39:01.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:01.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:01.452221+0000) 2024-03-06T14:39:02.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:02.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:02.452437+0000) 2024-03-06T14:39:03.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:03.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:03.452634+0000) 2024-03-06T14:39:04.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:04.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:04.452923+0000) 2024-03-06T14:39:04.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:04.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:05.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:05.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:05.453120+0000) 2024-03-06T14:39:06.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:06.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:06.453309+0000) 2024-03-06T14:39:07.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:07.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:07.453533+0000) 2024-03-06T14:39:08.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:08.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:08.453810+0000) 2024-03-06T14:39:09.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:09.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:09.454074+0000) 2024-03-06T14:39:10.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:10.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:10.454307+0000) 2024-03-06T14:39:11.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:11.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:11.454532+0000) 2024-03-06T14:39:12.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:12.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:12.454775+0000) 2024-03-06T14:39:13.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:13.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:13.454988+0000) 2024-03-06T14:39:14.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:14.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:14.455209+0000) 2024-03-06T14:39:15.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:15.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:15.455439+0000) 2024-03-06T14:39:16.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:16.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:16.455701+0000) 2024-03-06T14:39:17.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:17.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:17.455929+0000) 2024-03-06T14:39:18.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:18.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:18.456159+0000) 2024-03-06T14:39:19.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:19.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:19.456461+0000) 2024-03-06T14:39:20.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:20.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:20.456708+0000) 2024-03-06T14:39:21.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:21.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:21.456933+0000) 2024-03-06T14:39:22.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:22.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:22.457197+0000) 2024-03-06T14:39:23.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:23.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:23.457480+0000) 2024-03-06T14:39:24.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:24.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:24.457822+0000) 2024-03-06T14:39:25.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:25.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:25.458095+0000) 2024-03-06T14:39:26.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:26.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:26.458332+0000) 2024-03-06T14:39:27.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:27.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:27.458553+0000) 2024-03-06T14:39:28.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:28.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:28.458819+0000) 2024-03-06T14:39:29.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:29.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:29.459086+0000) 2024-03-06T14:39:30.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:30.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:30.459368+0000) 2024-03-06T14:39:31.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:31.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:31.459649+0000) 2024-03-06T14:39:32.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:32.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:32.459981+0000) 2024-03-06T14:39:33.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:33.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:33.460267+0000) 2024-03-06T14:39:34.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:34.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:34.460555+0000) 2024-03-06T14:39:34.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:34.864 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:35.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:35.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:35.460839+0000) 2024-03-06T14:39:36.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:36.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:36.461110+0000) 2024-03-06T14:39:37.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:37.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:37.461365+0000) 2024-03-06T14:39:38.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:38.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:38.461647+0000) 2024-03-06T14:39:39.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:39.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:39.461975+0000) 2024-03-06T14:39:40.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:40.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:40.462162+0000) 2024-03-06T14:39:41.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:41.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:41.462380+0000) 2024-03-06T14:39:42.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:42.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:42.462565+0000) 2024-03-06T14:39:43.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:43.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:43.462844+0000) 2024-03-06T14:39:44.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:44.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:44.463126+0000) 2024-03-06T14:39:45.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:45.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:45.463412+0000) 2024-03-06T14:39:46.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:46.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:46.463723+0000) 2024-03-06T14:39:47.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:47.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:47.463987+0000) 2024-03-06T14:39:48.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:48.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:48.464225+0000) 2024-03-06T14:39:49.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:49.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:49.464486+0000) 2024-03-06T14:39:50.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:50.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:50.464708+0000) 2024-03-06T14:39:51.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:51.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:51.464948+0000) 2024-03-06T14:39:52.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:52.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:52.465164+0000) 2024-03-06T14:39:53.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:53.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:53.465358+0000) 2024-03-06T14:39:54.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:54.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:54.465603+0000) 2024-03-06T14:39:55.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:55.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:55.465837+0000) 2024-03-06T14:39:56.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:56.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:56.466072+0000) 2024-03-06T14:39:57.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:57.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:57.466362+0000) 2024-03-06T14:39:58.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:58.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:58.466588+0000) 2024-03-06T14:39:59.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:39:59.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:59.466835+0000) 2024-03-06T14:40:00.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:00.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:00.467027+0000) 2024-03-06T14:40:01.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:01.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:01.467265+0000) 2024-03-06T14:40:02.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:02.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:02.467499+0000) 2024-03-06T14:40:03.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:03.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:03.467776+0000) 2024-03-06T14:40:04.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:04.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:04.468056+0000) 2024-03-06T14:40:05.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:05.060 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:05.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:05.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:05.468333+0000) 2024-03-06T14:40:06.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:06.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:06.468615+0000) 2024-03-06T14:40:07.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:07.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:07.468839+0000) 2024-03-06T14:40:08.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:08.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:08.469040+0000) 2024-03-06T14:40:09.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:09.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:09.469297+0000) 2024-03-06T14:40:10.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:10.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:10.469533+0000) 2024-03-06T14:40:11.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:11.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:11.469787+0000) 2024-03-06T14:40:12.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:12.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:12.469999+0000) 2024-03-06T14:40:13.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:13.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:13.470231+0000) 2024-03-06T14:40:14.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:14.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:14.470470+0000) 2024-03-06T14:40:15.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:15.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:15.470717+0000) 2024-03-06T14:40:16.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:16.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:16.470941+0000) 2024-03-06T14:40:17.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:17.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:17.471177+0000) 2024-03-06T14:40:18.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:18.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:18.471401+0000) 2024-03-06T14:40:19.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:19.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:19.471698+0000) 2024-03-06T14:40:20.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:20.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:20.471933+0000) 2024-03-06T14:40:21.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:21.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:21.472213+0000) 2024-03-06T14:40:22.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:22.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:22.472490+0000) 2024-03-06T14:40:23.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:23.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:23.472817+0000) 2024-03-06T14:40:24.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:24.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:24.473033+0000) 2024-03-06T14:40:25.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:25.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:25.473221+0000) 2024-03-06T14:40:26.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:26.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:26.473456+0000) 2024-03-06T14:40:27.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:27.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:27.473721+0000) 2024-03-06T14:40:28.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:28.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:28.473944+0000) 2024-03-06T14:40:29.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:29.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:29.474267+0000) 2024-03-06T14:40:30.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:30.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:30.474547+0000) 2024-03-06T14:40:31.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:31.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:31.474833+0000) 2024-03-06T14:40:32.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:32.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:32.475100+0000) 2024-03-06T14:40:33.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:33.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:33.475355+0000) 2024-03-06T14:40:34.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:34.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:34.475642+0000) 2024-03-06T14:40:35.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:35.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:35.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:35.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:35.475994+0000) 2024-03-06T14:40:36.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:36.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:36.476285+0000) 2024-03-06T14:40:37.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:37.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:37.476497+0000) 2024-03-06T14:40:38.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:38.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:38.476777+0000) 2024-03-06T14:40:39.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:39.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:39.477063+0000) 2024-03-06T14:40:40.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:40.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:40.477340+0000) 2024-03-06T14:40:41.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:41.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:41.477574+0000) 2024-03-06T14:40:42.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:42.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:42.477869+0000) 2024-03-06T14:40:43.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:43.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:43.478160+0000) 2024-03-06T14:40:44.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:44.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:44.478447+0000) 2024-03-06T14:40:45.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:45.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:45.478818+0000) 2024-03-06T14:40:46.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:46.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:46.479094+0000) 2024-03-06T14:40:47.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:47.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:47.479302+0000) 2024-03-06T14:40:48.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:48.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:48.479551+0000) 2024-03-06T14:40:49.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:49.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:49.479798+0000) 2024-03-06T14:40:50.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:50.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:50.479989+0000) 2024-03-06T14:40:51.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:51.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:51.480189+0000) 2024-03-06T14:40:52.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:52.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:52.480406+0000) 2024-03-06T14:40:53.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:53.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:53.480712+0000) 2024-03-06T14:40:54.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:54.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:54.480930+0000) 2024-03-06T14:40:55.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:55.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:55.481194+0000) 2024-03-06T14:40:56.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:56.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:56.481434+0000) 2024-03-06T14:40:57.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:57.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:57.481613+0000) 2024-03-06T14:40:58.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:58.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:58.481832+0000) 2024-03-06T14:40:59.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:40:59.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:59.482057+0000) 2024-03-06T14:41:00.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:00.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:00.482312+0000) 2024-03-06T14:41:01.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:01.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:01.482526+0000) 2024-03-06T14:41:02.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:02.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:02.482726+0000) 2024-03-06T14:41:03.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:03.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:03.482967+0000) 2024-03-06T14:41:04.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:04.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:04.483221+0000) 2024-03-06T14:41:05.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:05.444 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:05.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:05.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:05.483379+0000) 2024-03-06T14:41:06.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:06.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:06.483573+0000) 2024-03-06T14:41:07.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:07.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:07.483764+0000) 2024-03-06T14:41:08.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:08.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:08.483991+0000) 2024-03-06T14:41:09.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:09.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:09.484208+0000) 2024-03-06T14:41:10.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:10.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:10.484412+0000) 2024-03-06T14:41:11.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:11.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:11.484628+0000) 2024-03-06T14:41:12.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:12.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:12.484839+0000) 2024-03-06T14:41:13.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:13.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:13.485042+0000) 2024-03-06T14:41:14.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:14.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:14.485237+0000) 2024-03-06T14:41:15.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:15.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:15.485423+0000) 2024-03-06T14:41:16.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:16.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:16.485603+0000) 2024-03-06T14:41:17.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:17.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:17.485789+0000) 2024-03-06T14:41:18.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:18.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:18.485973+0000) 2024-03-06T14:41:19.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:19.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:19.486211+0000) 2024-03-06T14:41:20.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:20.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:20.486415+0000) 2024-03-06T14:41:21.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:21.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:21.486606+0000) 2024-03-06T14:41:22.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:22.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:22.486929+0000) 2024-03-06T14:41:23.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:23.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:23.487124+0000) 2024-03-06T14:41:24.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:24.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:24.487299+0000) 2024-03-06T14:41:25.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:25.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:25.487484+0000) 2024-03-06T14:41:26.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:26.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:26.487759+0000) 2024-03-06T14:41:27.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:27.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:27.487952+0000) 2024-03-06T14:41:28.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:28.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:28.488169+0000) 2024-03-06T14:41:29.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:29.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:29.488370+0000) 2024-03-06T14:41:30.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:30.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:30.488568+0000) 2024-03-06T14:41:31.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:31.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:31.488835+0000) 2024-03-06T14:41:32.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:32.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:32.489037+0000) 2024-03-06T14:41:33.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:33.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:33.489298+0000) 2024-03-06T14:41:34.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:34.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:34.489493+0000) 2024-03-06T14:41:35.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:35.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:35.489784+0000) 2024-03-06T14:41:35.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:35.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:36.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:36.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:36.489980+0000) 2024-03-06T14:41:37.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:37.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:37.490176+0000) 2024-03-06T14:41:38.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:38.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:38.490411+0000) 2024-03-06T14:41:39.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:39.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:39.490653+0000) 2024-03-06T14:41:40.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:40.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:40.490899+0000) 2024-03-06T14:41:41.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:41.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:41.491085+0000) 2024-03-06T14:41:42.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:42.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:42.491282+0000) 2024-03-06T14:41:43.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:43.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:43.491484+0000) 2024-03-06T14:41:44.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:44.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:44.491745+0000) 2024-03-06T14:41:45.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:45.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:45.491957+0000) 2024-03-06T14:41:46.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:46.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:46.492191+0000) 2024-03-06T14:41:47.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:47.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:47.492405+0000) 2024-03-06T14:41:48.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:48.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:48.492635+0000) 2024-03-06T14:41:49.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:49.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:49.492922+0000) 2024-03-06T14:41:50.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:50.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:50.493150+0000) 2024-03-06T14:41:51.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:51.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:51.493380+0000) 2024-03-06T14:41:52.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:52.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:52.493556+0000) 2024-03-06T14:41:53.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:53.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:53.493798+0000) 2024-03-06T14:41:54.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:54.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:54.494008+0000) 2024-03-06T14:41:55.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:55.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:55.494233+0000) 2024-03-06T14:41:56.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:56.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:56.494461+0000) 2024-03-06T14:41:57.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:57.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:57.494731+0000) 2024-03-06T14:41:58.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:58.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:58.494932+0000) 2024-03-06T14:41:59.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:41:59.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:59.495179+0000) 2024-03-06T14:42:00.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:00.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:00.495409+0000) 2024-03-06T14:42:01.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:01.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:01.495645+0000) 2024-03-06T14:42:02.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:02.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:02.495891+0000) 2024-03-06T14:42:03.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:03.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:03.496092+0000) 2024-03-06T14:42:04.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:04.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:04.496289+0000) 2024-03-06T14:42:05.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:05.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:05.496537+0000) 2024-03-06T14:42:05.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:05.892 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:06.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:06.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:06.496794+0000) 2024-03-06T14:42:07.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:07.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:07.497039+0000) 2024-03-06T14:42:08.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:08.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:08.497251+0000) 2024-03-06T14:42:09.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:09.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:09.497507+0000) 2024-03-06T14:42:10.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:10.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:10.497798+0000) 2024-03-06T14:42:11.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:11.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:11.498004+0000) 2024-03-06T14:42:12.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:12.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:12.498200+0000) 2024-03-06T14:42:13.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:13.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:13.498439+0000) 2024-03-06T14:42:14.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:14.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:14.498635+0000) 2024-03-06T14:42:15.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:15.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:15.498899+0000) 2024-03-06T14:42:16.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:16.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:16.499124+0000) 2024-03-06T14:42:17.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:17.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:17.499355+0000) 2024-03-06T14:42:18.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:18.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:18.499578+0000) 2024-03-06T14:42:19.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:19.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:19.499796+0000) 2024-03-06T14:42:20.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:20.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:20.500035+0000) 2024-03-06T14:42:21.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:21.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:21.500269+0000) 2024-03-06T14:42:22.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:22.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:22.500507+0000) 2024-03-06T14:42:23.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:23.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:23.500744+0000) 2024-03-06T14:42:24.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:24.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:24.500983+0000) 2024-03-06T14:42:25.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:25.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:25.501214+0000) 2024-03-06T14:42:26.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:26.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:26.501458+0000) 2024-03-06T14:42:27.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:27.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:27.501719+0000) 2024-03-06T14:42:28.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:28.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:28.501905+0000) 2024-03-06T14:42:29.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:29.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:29.502148+0000) 2024-03-06T14:42:30.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:30.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:30.502352+0000) 2024-03-06T14:42:31.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:31.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:31.502596+0000) 2024-03-06T14:42:32.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:32.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:32.502828+0000) 2024-03-06T14:42:33.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:33.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:33.503006+0000) 2024-03-06T14:42:34.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:34.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:34.503242+0000) 2024-03-06T14:42:35.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:35.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:35.503477+0000) 2024-03-06T14:42:36.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:36.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:36.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:36.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:36.503762+0000) 2024-03-06T14:42:37.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:37.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:37.503932+0000) 2024-03-06T14:42:38.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:38.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:38.504163+0000) 2024-03-06T14:42:39.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:39.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:39.504399+0000) 2024-03-06T14:42:40.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:40.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:40.504629+0000) 2024-03-06T14:42:41.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:41.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:41.504870+0000) 2024-03-06T14:42:42.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:42.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:42.505158+0000) 2024-03-06T14:42:43.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:43.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:43.505370+0000) 2024-03-06T14:42:44.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:44.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:44.505560+0000) 2024-03-06T14:42:45.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:45.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:45.505813+0000) 2024-03-06T14:42:46.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:46.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:46.506056+0000) 2024-03-06T14:42:47.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:47.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:47.506308+0000) 2024-03-06T14:42:48.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:48.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:48.506544+0000) 2024-03-06T14:42:49.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:49.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:49.506787+0000) 2024-03-06T14:42:50.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:50.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:50.506976+0000) 2024-03-06T14:42:51.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:51.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:51.507209+0000) 2024-03-06T14:42:52.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:52.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:52.507428+0000) 2024-03-06T14:42:53.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:53.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:53.507654+0000) 2024-03-06T14:42:54.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:54.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:54.507978+0000) 2024-03-06T14:42:55.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:55.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:55.508271+0000) 2024-03-06T14:42:56.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:56.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:56.508560+0000) 2024-03-06T14:42:57.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:57.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:57.508846+0000) 2024-03-06T14:42:58.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:58.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:58.509067+0000) 2024-03-06T14:42:59.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:42:59.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:59.509329+0000) 2024-03-06T14:43:00.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:00.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:00.509561+0000) 2024-03-06T14:43:01.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:01.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:01.509779+0000) 2024-03-06T14:43:02.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:02.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:02.510006+0000) 2024-03-06T14:43:03.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:03.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:03.510238+0000) 2024-03-06T14:43:04.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:04.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:04.510477+0000) 2024-03-06T14:43:05.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:05.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:05.510763+0000) 2024-03-06T14:43:06.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:06.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:06.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:06.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:06.511000+0000) 2024-03-06T14:43:07.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:07.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:07.511190+0000) 2024-03-06T14:43:08.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:08.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:08.511385+0000) 2024-03-06T14:43:09.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:09.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:09.511631+0000) 2024-03-06T14:43:10.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:10.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:10.511867+0000) 2024-03-06T14:43:11.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:11.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:11.512157+0000) 2024-03-06T14:43:12.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:12.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:12.512422+0000) 2024-03-06T14:43:13.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:13.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:13.512753+0000) 2024-03-06T14:43:14.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:14.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:14.513084+0000) 2024-03-06T14:43:15.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:15.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:15.513375+0000) 2024-03-06T14:43:16.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:16.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:16.513651+0000) 2024-03-06T14:43:17.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:17.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:17.513991+0000) 2024-03-06T14:43:18.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:18.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:18.514202+0000) 2024-03-06T14:43:19.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:19.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:19.514457+0000) 2024-03-06T14:43:20.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:20.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:20.514697+0000) 2024-03-06T14:43:21.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:21.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:21.514942+0000) 2024-03-06T14:43:22.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:22.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:22.515175+0000) 2024-03-06T14:43:23.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:23.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:23.515413+0000) 2024-03-06T14:43:24.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:24.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:24.515736+0000) 2024-03-06T14:43:25.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:25.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:25.515989+0000) 2024-03-06T14:43:26.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:26.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:26.516206+0000) 2024-03-06T14:43:27.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:27.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:27.516426+0000) 2024-03-06T14:43:28.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:28.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:28.516702+0000) 2024-03-06T14:43:29.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:29.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:29.516904+0000) 2024-03-06T14:43:30.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:30.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:30.517155+0000) 2024-03-06T14:43:31.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:31.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:31.517372+0000) 2024-03-06T14:43:32.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:32.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:32.517621+0000) 2024-03-06T14:43:33.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:33.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:33.517827+0000) 2024-03-06T14:43:34.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:34.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:34.518045+0000) 2024-03-06T14:43:35.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:35.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:35.518277+0000) 2024-03-06T14:43:36.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:36.475 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:36.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:36.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:36.518437+0000) 2024-03-06T14:43:37.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:37.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:37.518737+0000) 2024-03-06T14:43:38.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:38.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:38.519041+0000) 2024-03-06T14:43:39.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:39.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:39.519357+0000) 2024-03-06T14:43:40.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:40.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:40.519646+0000) 2024-03-06T14:43:41.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:41.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:41.519986+0000) 2024-03-06T14:43:42.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:42.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:42.520268+0000) 2024-03-06T14:43:43.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:43.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:43.520563+0000) 2024-03-06T14:43:44.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:44.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:44.520827+0000) 2024-03-06T14:43:45.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:45.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:45.521059+0000) 2024-03-06T14:43:46.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:46.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:46.521292+0000) 2024-03-06T14:43:47.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:47.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:47.521517+0000) 2024-03-06T14:43:48.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:48.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:48.521829+0000) 2024-03-06T14:43:49.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:49.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:49.522117+0000) 2024-03-06T14:43:50.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:50.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:50.522402+0000) 2024-03-06T14:43:51.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:51.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:51.522646+0000) 2024-03-06T14:43:52.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:52.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:52.522888+0000) 2024-03-06T14:43:53.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:53.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:53.523120+0000) 2024-03-06T14:43:54.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:54.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:54.523355+0000) 2024-03-06T14:43:55.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:55.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:55.523626+0000) 2024-03-06T14:43:56.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:56.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:56.523892+0000) 2024-03-06T14:43:57.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:57.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:57.524119+0000) 2024-03-06T14:43:58.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:58.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:58.524342+0000) 2024-03-06T14:43:59.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:43:59.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:59.524625+0000) 2024-03-06T14:44:00.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:00.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:00.524912+0000) 2024-03-06T14:44:01.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:01.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:01.525192+0000) 2024-03-06T14:44:02.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:02.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:02.525466+0000) 2024-03-06T14:44:03.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:03.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:03.525788+0000) 2024-03-06T14:44:04.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:04.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:04.526031+0000) 2024-03-06T14:44:05.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:05.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:05.526310+0000) 2024-03-06T14:44:06.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:06.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:06.526590+0000) 2024-03-06T14:44:06.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:06.686 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:07.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:07.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:07.526859+0000) 2024-03-06T14:44:08.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:08.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:08.527116+0000) 2024-03-06T14:44:09.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:09.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:09.527410+0000) 2024-03-06T14:44:10.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:10.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:10.527724+0000) 2024-03-06T14:44:11.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:11.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:11.527972+0000) 2024-03-06T14:44:12.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:12.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:12.528253+0000) 2024-03-06T14:44:13.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:13.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:13.528502+0000) 2024-03-06T14:44:14.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:14.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:14.528774+0000) 2024-03-06T14:44:15.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:15.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:15.528978+0000) 2024-03-06T14:44:16.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:16.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:16.529237+0000) 2024-03-06T14:44:17.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:17.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:17.529529+0000) 2024-03-06T14:44:18.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:18.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:18.529775+0000) 2024-03-06T14:44:19.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:19.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:19.529986+0000) 2024-03-06T14:44:20.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:20.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:20.530210+0000) 2024-03-06T14:44:21.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:21.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:21.530442+0000) 2024-03-06T14:44:22.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:22.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:22.530704+0000) 2024-03-06T14:44:23.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:23.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:23.530958+0000) 2024-03-06T14:44:24.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:24.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:24.531243+0000) 2024-03-06T14:44:25.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:25.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:25.531518+0000) 2024-03-06T14:44:26.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:26.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:26.531743+0000) 2024-03-06T14:44:27.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:27.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:27.531974+0000) 2024-03-06T14:44:28.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:28.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:28.532208+0000) 2024-03-06T14:44:29.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:29.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:29.532450+0000) 2024-03-06T14:44:30.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:30.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:30.532647+0000) 2024-03-06T14:44:31.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:31.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:31.532894+0000) 2024-03-06T14:44:32.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:32.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:32.533136+0000) 2024-03-06T14:44:33.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:33.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:33.533399+0000) 2024-03-06T14:44:34.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:34.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:34.533622+0000) 2024-03-06T14:44:35.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:35.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:35.533854+0000) 2024-03-06T14:44:36.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:36.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:36.534036+0000) 2024-03-06T14:44:36.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:36.823 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:37.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:37.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:37.534240+0000) 2024-03-06T14:44:38.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:38.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:38.534483+0000) 2024-03-06T14:44:39.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:39.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:39.534759+0000) 2024-03-06T14:44:40.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:40.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:40.535001+0000) 2024-03-06T14:44:41.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:41.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:41.535236+0000) 2024-03-06T14:44:42.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:42.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:42.535471+0000) 2024-03-06T14:44:43.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:43.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:43.535732+0000) 2024-03-06T14:44:44.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:44.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:44.535916+0000) 2024-03-06T14:44:45.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:45.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:45.536189+0000) 2024-03-06T14:44:46.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:46.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:46.536381+0000) 2024-03-06T14:44:47.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:47.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:47.536576+0000) 2024-03-06T14:44:48.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:48.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:48.536805+0000) 2024-03-06T14:44:49.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:49.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:49.537055+0000) 2024-03-06T14:44:50.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:50.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:50.537256+0000) 2024-03-06T14:44:51.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:51.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:51.537499+0000) 2024-03-06T14:44:52.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:52.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:52.537700+0000) 2024-03-06T14:44:53.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:53.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:53.537943+0000) 2024-03-06T14:44:54.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:54.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:54.538178+0000) 2024-03-06T14:44:55.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:55.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:55.538410+0000) 2024-03-06T14:44:56.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:56.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:56.538648+0000) 2024-03-06T14:44:57.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:57.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:57.538904+0000) 2024-03-06T14:44:58.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:58.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:58.539155+0000) 2024-03-06T14:44:59.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:44:59.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:59.539399+0000) 2024-03-06T14:45:00.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:00.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:00.539627+0000) 2024-03-06T14:45:01.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:01.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:01.539886+0000) 2024-03-06T14:45:02.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:02.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:02.540115+0000) 2024-03-06T14:45:03.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:03.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:03.540351+0000) 2024-03-06T14:45:04.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:04.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:04.540596+0000) 2024-03-06T14:45:05.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:05.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:05.540840+0000) 2024-03-06T14:45:06.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:06.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:06.541024+0000) 2024-03-06T14:45:07.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:07.016 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:07.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:07.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:07.541249+0000) 2024-03-06T14:45:08.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:08.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:08.541462+0000) 2024-03-06T14:45:09.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:09.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:09.541763+0000) 2024-03-06T14:45:10.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:10.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:10.541983+0000) 2024-03-06T14:45:11.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:11.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:11.542270+0000) 2024-03-06T14:45:12.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:12.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:12.542547+0000) 2024-03-06T14:45:13.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:13.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:13.542831+0000) 2024-03-06T14:45:14.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:14.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:14.543066+0000) 2024-03-06T14:45:15.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:15.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:15.543314+0000) 2024-03-06T14:45:16.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:16.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:16.543572+0000) 2024-03-06T14:45:17.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:17.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:17.543853+0000) 2024-03-06T14:45:18.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:18.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:18.544135+0000) 2024-03-06T14:45:19.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:19.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:19.544445+0000) 2024-03-06T14:45:20.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:20.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:20.544775+0000) 2024-03-06T14:45:21.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:21.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:21.545008+0000) 2024-03-06T14:45:22.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:22.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:22.545207+0000) 2024-03-06T14:45:23.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:23.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:23.545397+0000) 2024-03-06T14:45:24.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:24.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:24.545598+0000) 2024-03-06T14:45:25.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:25.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:25.545868+0000) 2024-03-06T14:45:26.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:26.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:26.546104+0000) 2024-03-06T14:45:27.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:27.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:27.546306+0000) 2024-03-06T14:45:28.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:28.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:28.546556+0000) 2024-03-06T14:45:29.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:29.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:29.546817+0000) 2024-03-06T14:45:30.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:30.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:30.547022+0000) 2024-03-06T14:45:31.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:31.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:31.547252+0000) 2024-03-06T14:45:32.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:32.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:32.547466+0000) 2024-03-06T14:45:33.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:33.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:33.547733+0000) 2024-03-06T14:45:34.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:34.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:34.547970+0000) 2024-03-06T14:45:35.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:35.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:35.548168+0000) 2024-03-06T14:45:36.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:36.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:36.548406+0000) 2024-03-06T14:45:37.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:37.188 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:37.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:37.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:37.548638+0000) 2024-03-06T14:45:38.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:38.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:38.548921+0000) 2024-03-06T14:45:39.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:39.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:39.549166+0000) 2024-03-06T14:45:40.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:40.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:40.549402+0000) 2024-03-06T14:45:41.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:41.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:41.549622+0000) 2024-03-06T14:45:42.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:42.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:42.549840+0000) 2024-03-06T14:45:43.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:43.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:43.550090+0000) 2024-03-06T14:45:44.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:44.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:44.550325+0000) 2024-03-06T14:45:45.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:45.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:45.550515+0000) 2024-03-06T14:45:46.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:46.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:46.550785+0000) 2024-03-06T14:45:47.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:47.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:47.550978+0000) 2024-03-06T14:45:48.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:48.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:48.551218+0000) 2024-03-06T14:45:49.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:49.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:49.551426+0000) 2024-03-06T14:45:50.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:50.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:50.551624+0000) 2024-03-06T14:45:51.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:51.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:51.551866+0000) 2024-03-06T14:45:52.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:52.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:52.552100+0000) 2024-03-06T14:45:53.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:53.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:53.552333+0000) 2024-03-06T14:45:54.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:54.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:54.552522+0000) 2024-03-06T14:45:55.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:55.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:55.552799+0000) 2024-03-06T14:45:56.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:56.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:56.552997+0000) 2024-03-06T14:45:57.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:57.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:57.553238+0000) 2024-03-06T14:45:58.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:58.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:58.553469+0000) 2024-03-06T14:45:59.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:45:59.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:59.553743+0000) 2024-03-06T14:46:00.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:00.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:00.553914+0000) 2024-03-06T14:46:01.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:01.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:01.554151+0000) 2024-03-06T14:46:02.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:02.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:02.554390+0000) 2024-03-06T14:46:03.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:03.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:03.554576+0000) 2024-03-06T14:46:04.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:04.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:04.554779+0000) 2024-03-06T14:46:05.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:05.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:05.555020+0000) 2024-03-06T14:46:06.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:06.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:06.555252+0000) 2024-03-06T14:46:07.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:07.402 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:07.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:07.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:07.555492+0000) 2024-03-06T14:46:08.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:08.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:08.555757+0000) 2024-03-06T14:46:09.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:09.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:09.555978+0000) 2024-03-06T14:46:10.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:10.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:10.556209+0000) 2024-03-06T14:46:11.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:11.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:11.556387+0000) 2024-03-06T14:46:12.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:12.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:12.556602+0000) 2024-03-06T14:46:13.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:13.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:13.556851+0000) 2024-03-06T14:46:14.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:14.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:14.557081+0000) 2024-03-06T14:46:15.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:15.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:15.557322+0000) 2024-03-06T14:46:16.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:16.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:16.557554+0000) 2024-03-06T14:46:17.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:17.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:17.557786+0000) 2024-03-06T14:46:18.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:18.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:18.558020+0000) 2024-03-06T14:46:19.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:19.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:19.558250+0000) 2024-03-06T14:46:20.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:20.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:20.558475+0000) 2024-03-06T14:46:21.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:21.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:21.558651+0000) 2024-03-06T14:46:22.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:22.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:22.558871+0000) 2024-03-06T14:46:23.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:23.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:23.559097+0000) 2024-03-06T14:46:24.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:24.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:24.559322+0000) 2024-03-06T14:46:25.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:25.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:25.559558+0000) 2024-03-06T14:46:26.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:26.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:26.559789+0000) 2024-03-06T14:46:27.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:27.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:27.560022+0000) 2024-03-06T14:46:28.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:28.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:28.560209+0000) 2024-03-06T14:46:29.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:29.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:29.560415+0000) 2024-03-06T14:46:30.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:30.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:30.560622+0000) 2024-03-06T14:46:31.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:31.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:31.560788+0000) 2024-03-06T14:46:32.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:32.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:32.560998+0000) 2024-03-06T14:46:33.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:33.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:33.561229+0000) 2024-03-06T14:46:34.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:34.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:34.561425+0000) 2024-03-06T14:46:35.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:35.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:35.561642+0000) 2024-03-06T14:46:36.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:36.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:36.561885+0000) 2024-03-06T14:46:37.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:37.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:37.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:37.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:37.562088+0000) 2024-03-06T14:46:38.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:38.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:38.562312+0000) 2024-03-06T14:46:39.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:39.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:39.562521+0000) 2024-03-06T14:46:40.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:40.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:40.562814+0000) 2024-03-06T14:46:41.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:41.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:41.563023+0000) 2024-03-06T14:46:42.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:42.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:42.563227+0000) 2024-03-06T14:46:43.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:43.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:43.563453+0000) 2024-03-06T14:46:44.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:44.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:44.563772+0000) 2024-03-06T14:46:45.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:45.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:45.564028+0000) 2024-03-06T14:46:46.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:46.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:46.564265+0000) 2024-03-06T14:46:47.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:47.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:47.564445+0000) 2024-03-06T14:46:48.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:48.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:48.564707+0000) 2024-03-06T14:46:49.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:49.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:49.564947+0000) 2024-03-06T14:46:50.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:50.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:50.565190+0000) 2024-03-06T14:46:51.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:51.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:51.565423+0000) 2024-03-06T14:46:52.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:52.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:52.565638+0000) 2024-03-06T14:46:53.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:53.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:53.565906+0000) 2024-03-06T14:46:54.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:54.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:54.566139+0000) 2024-03-06T14:46:55.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:55.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:55.566342+0000) 2024-03-06T14:46:56.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:56.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:56.566594+0000) 2024-03-06T14:46:57.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:57.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:57.566791+0000) 2024-03-06T14:46:58.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:58.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:58.567027+0000) 2024-03-06T14:46:59.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:46:59.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:59.567293+0000) 2024-03-06T14:47:00.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:00.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:00.567561+0000) 2024-03-06T14:47:01.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:01.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:01.567816+0000) 2024-03-06T14:47:02.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:02.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:02.568054+0000) 2024-03-06T14:47:03.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:03.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:03.568279+0000) 2024-03-06T14:47:04.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:04.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:04.568526+0000) 2024-03-06T14:47:05.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:05.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:05.568823+0000) 2024-03-06T14:47:06.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:06.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:06.569038+0000) 2024-03-06T14:47:07.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:07.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:07.569279+0000) 2024-03-06T14:47:07.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:07.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:08.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:08.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:08.569518+0000) 2024-03-06T14:47:09.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:09.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:09.569794+0000) 2024-03-06T14:47:10.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:10.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:10.570007+0000) 2024-03-06T14:47:11.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:11.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:11.570275+0000) 2024-03-06T14:47:12.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:12.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:12.570520+0000) 2024-03-06T14:47:13.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:13.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:13.570817+0000) 2024-03-06T14:47:14.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:14.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:14.571048+0000) 2024-03-06T14:47:15.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:15.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:15.571294+0000) 2024-03-06T14:47:16.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:16.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:16.571484+0000) 2024-03-06T14:47:17.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:17.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:17.571753+0000) 2024-03-06T14:47:18.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:18.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:18.571981+0000) 2024-03-06T14:47:19.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:19.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:19.572236+0000) 2024-03-06T14:47:20.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:20.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:20.572520+0000) 2024-03-06T14:47:21.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:21.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:21.572845+0000) 2024-03-06T14:47:22.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:22.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:22.573096+0000) 2024-03-06T14:47:23.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:23.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:23.573379+0000) 2024-03-06T14:47:24.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:24.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:24.573644+0000) 2024-03-06T14:47:25.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:25.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:25.573972+0000) 2024-03-06T14:47:26.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:26.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:26.574207+0000) 2024-03-06T14:47:27.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:27.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:27.574451+0000) 2024-03-06T14:47:28.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:28.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:28.574744+0000) 2024-03-06T14:47:29.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:29.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:29.575006+0000) 2024-03-06T14:47:30.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:30.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:30.575251+0000) 2024-03-06T14:47:31.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:31.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:31.575481+0000) 2024-03-06T14:47:32.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:32.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:32.575709+0000) 2024-03-06T14:47:33.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:33.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:33.575923+0000) 2024-03-06T14:47:34.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:34.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:34.576178+0000) 2024-03-06T14:47:35.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:35.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:35.576461+0000) 2024-03-06T14:47:36.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:36.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:36.576714+0000) 2024-03-06T14:47:37.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:37.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:37.576929+0000) 2024-03-06T14:47:37.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:37.940 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:38.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:38.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:38.577168+0000) 2024-03-06T14:47:39.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:39.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:39.577461+0000) 2024-03-06T14:47:40.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:40.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:40.577782+0000) 2024-03-06T14:47:41.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:41.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:41.578060+0000) 2024-03-06T14:47:42.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:42.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:42.578342+0000) 2024-03-06T14:47:43.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:43.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:43.578630+0000) 2024-03-06T14:47:44.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:44.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:44.578960+0000) 2024-03-06T14:47:45.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:45.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:45.579253+0000) 2024-03-06T14:47:46.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:46.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:46.579506+0000) 2024-03-06T14:47:47.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:47.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:47.579731+0000) 2024-03-06T14:47:48.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:48.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:48.579933+0000) 2024-03-06T14:47:49.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:49.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:49.580161+0000) 2024-03-06T14:47:50.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:50.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:50.580371+0000) 2024-03-06T14:47:51.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:51.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:51.580621+0000) 2024-03-06T14:47:52.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:52.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:52.580853+0000) 2024-03-06T14:47:53.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:53.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:53.581084+0000) 2024-03-06T14:47:54.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:54.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:54.581317+0000) 2024-03-06T14:47:55.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:55.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:55.581591+0000) 2024-03-06T14:47:56.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:56.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:56.581875+0000) 2024-03-06T14:47:57.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:57.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:57.582159+0000) 2024-03-06T14:47:58.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:58.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:58.582435+0000) 2024-03-06T14:47:59.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:47:59.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:59.582758+0000) 2024-03-06T14:48:00.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:00.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:00.582973+0000) 2024-03-06T14:48:01.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:01.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:01.583180+0000) 2024-03-06T14:48:02.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:02.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:02.583418+0000) 2024-03-06T14:48:03.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:03.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:03.583608+0000) 2024-03-06T14:48:04.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:04.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:04.583825+0000) 2024-03-06T14:48:05.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:05.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:05.584031+0000) 2024-03-06T14:48:06.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:06.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:06.584235+0000) 2024-03-06T14:48:07.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:07.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:07.584427+0000) 2024-03-06T14:48:08.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:08.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:08.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:08.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:08.584631+0000) 2024-03-06T14:48:09.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:09.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:09.584908+0000) 2024-03-06T14:48:10.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:10.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:10.585144+0000) 2024-03-06T14:48:11.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:11.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:11.585374+0000) 2024-03-06T14:48:12.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:12.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:12.585550+0000) 2024-03-06T14:48:13.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:13.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:13.585827+0000) 2024-03-06T14:48:14.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:14.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:14.586102+0000) 2024-03-06T14:48:15.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:15.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:15.586308+0000) 2024-03-06T14:48:16.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:16.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:16.586578+0000) 2024-03-06T14:48:17.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:17.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:17.586876+0000) 2024-03-06T14:48:18.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:18.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:18.587134+0000) 2024-03-06T14:48:19.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:19.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:19.587428+0000) 2024-03-06T14:48:20.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:20.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:20.587760+0000) 2024-03-06T14:48:21.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:21.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:21.588016+0000) 2024-03-06T14:48:22.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:22.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:22.588212+0000) 2024-03-06T14:48:23.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:23.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:23.588460+0000) 2024-03-06T14:48:24.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:24.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:24.588729+0000) 2024-03-06T14:48:25.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:25.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:25.588962+0000) 2024-03-06T14:48:26.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:26.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:26.589194+0000) 2024-03-06T14:48:27.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:27.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:27.589374+0000) 2024-03-06T14:48:28.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:28.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:28.589560+0000) 2024-03-06T14:48:29.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:29.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:29.589838+0000) 2024-03-06T14:48:30.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:30.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:30.590126+0000) 2024-03-06T14:48:31.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:31.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:31.590353+0000) 2024-03-06T14:48:32.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:32.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:32.590632+0000) 2024-03-06T14:48:33.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:33.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:33.590887+0000) 2024-03-06T14:48:34.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:34.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:34.591155+0000) 2024-03-06T14:48:35.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:35.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:35.591439+0000) 2024-03-06T14:48:36.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:36.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:36.591660+0000) 2024-03-06T14:48:37.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:37.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:37.592009+0000) 2024-03-06T14:48:38.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:38.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:38.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:38.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:38.592201+0000) 2024-03-06T14:48:39.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:39.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:39.592481+0000) 2024-03-06T14:48:40.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:40.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:40.592784+0000) 2024-03-06T14:48:41.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:41.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:41.593009+0000) 2024-03-06T14:48:42.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:42.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:42.593192+0000) 2024-03-06T14:48:43.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:43.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:43.593414+0000) 2024-03-06T14:48:44.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:44.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:44.593651+0000) 2024-03-06T14:48:45.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:45.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:45.593958+0000) 2024-03-06T14:48:46.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:46.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:46.594240+0000) 2024-03-06T14:48:47.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:47.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:47.594443+0000) 2024-03-06T14:48:48.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:48.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:48.594720+0000) 2024-03-06T14:48:49.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:49.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:49.594978+0000) 2024-03-06T14:48:50.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:50.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:50.595228+0000) 2024-03-06T14:48:51.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:51.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:51.595482+0000) 2024-03-06T14:48:52.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:52.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:52.595775+0000) 2024-03-06T14:48:53.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:53.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:53.595991+0000) 2024-03-06T14:48:54.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:54.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:54.596229+0000) 2024-03-06T14:48:55.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:55.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:55.596477+0000) 2024-03-06T14:48:56.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:56.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:56.596798+0000) 2024-03-06T14:48:57.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:57.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:57.597004+0000) 2024-03-06T14:48:58.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:58.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:58.597231+0000) 2024-03-06T14:48:59.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:48:59.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:59.597460+0000) 2024-03-06T14:49:00.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:00.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:00.597723+0000) 2024-03-06T14:49:01.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:01.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:01.597960+0000) 2024-03-06T14:49:02.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:02.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:02.598184+0000) 2024-03-06T14:49:03.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:03.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:03.598411+0000) 2024-03-06T14:49:04.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:04.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:04.598590+0000) 2024-03-06T14:49:05.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:05.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:05.598830+0000) 2024-03-06T14:49:06.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:06.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:06.599037+0000) 2024-03-06T14:49:07.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:07.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:07.599260+0000) 2024-03-06T14:49:08.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:08.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:08.599526+0000) 2024-03-06T14:49:08.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:08.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:09.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:09.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:09.599814+0000) 2024-03-06T14:49:10.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:10.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:10.600001+0000) 2024-03-06T14:49:11.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:11.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:11.600255+0000) 2024-03-06T14:49:12.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:12.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:12.600506+0000) 2024-03-06T14:49:13.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:13.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:13.600728+0000) 2024-03-06T14:49:14.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:14.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:14.600922+0000) 2024-03-06T14:49:15.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:15.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:15.601093+0000) 2024-03-06T14:49:16.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:16.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:16.601314+0000) 2024-03-06T14:49:17.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:17.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:17.601545+0000) 2024-03-06T14:49:18.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:18.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:18.601799+0000) 2024-03-06T14:49:19.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:19.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:19.602051+0000) 2024-03-06T14:49:20.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:20.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:20.602288+0000) 2024-03-06T14:49:21.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:21.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:21.602506+0000) 2024-03-06T14:49:22.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:22.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:22.602736+0000) 2024-03-06T14:49:23.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:23.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:23.602937+0000) 2024-03-06T14:49:24.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:24.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:24.603152+0000) 2024-03-06T14:49:25.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:25.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:25.603378+0000) 2024-03-06T14:49:26.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:26.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:26.603606+0000) 2024-03-06T14:49:27.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:27.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:27.603837+0000) 2024-03-06T14:49:28.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:28.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:28.604082+0000) 2024-03-06T14:49:29.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:29.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:29.604342+0000) 2024-03-06T14:49:30.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:30.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:30.604587+0000) 2024-03-06T14:49:31.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:31.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:31.604835+0000) 2024-03-06T14:49:32.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:32.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:32.605036+0000) 2024-03-06T14:49:33.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:33.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:33.605286+0000) 2024-03-06T14:49:34.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:34.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:34.605522+0000) 2024-03-06T14:49:35.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:35.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:35.605800+0000) 2024-03-06T14:49:36.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:36.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:36.606010+0000) 2024-03-06T14:49:37.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:37.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:37.606242+0000) 2024-03-06T14:49:38.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:38.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:38.606480+0000) 2024-03-06T14:49:38.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:38.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:39.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:39.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:39.606724+0000) 2024-03-06T14:49:40.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:40.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:40.606948+0000) 2024-03-06T14:49:41.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:41.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:41.607182+0000) 2024-03-06T14:49:42.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:42.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:42.607393+0000) 2024-03-06T14:49:43.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:43.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:43.607703+0000) 2024-03-06T14:49:44.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:44.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:44.607972+0000) 2024-03-06T14:49:45.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:45.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:45.608258+0000) 2024-03-06T14:49:46.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:46.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:46.608465+0000) 2024-03-06T14:49:47.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:47.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:47.608769+0000) 2024-03-06T14:49:48.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:48.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:48.608981+0000) 2024-03-06T14:49:49.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:49.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:49.609233+0000) 2024-03-06T14:49:50.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:50.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:50.609462+0000) 2024-03-06T14:49:51.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:51.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:51.609776+0000) 2024-03-06T14:49:52.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:52.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:52.609991+0000) 2024-03-06T14:49:53.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:53.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:53.610178+0000) 2024-03-06T14:49:54.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:54.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:54.610479+0000) 2024-03-06T14:49:55.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:55.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:55.610756+0000) 2024-03-06T14:49:56.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:56.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:56.611027+0000) 2024-03-06T14:49:57.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:57.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:57.611302+0000) 2024-03-06T14:49:58.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:58.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:58.611579+0000) 2024-03-06T14:49:59.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:49:59.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:59.611864+0000) 2024-03-06T14:50:00.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:00.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:00.612063+0000) 2024-03-06T14:50:01.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:01.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:01.612305+0000) 2024-03-06T14:50:02.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:02.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:02.612571+0000) 2024-03-06T14:50:03.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:03.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:03.612772+0000) 2024-03-06T14:50:04.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:04.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:04.612993+0000) 2024-03-06T14:50:05.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:05.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:05.613174+0000) 2024-03-06T14:50:06.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:06.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:06.613420+0000) 2024-03-06T14:50:07.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:07.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:07.613726+0000) 2024-03-06T14:50:08.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:08.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:08.613934+0000) 2024-03-06T14:50:09.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:09.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:09.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:09.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:09.614185+0000) 2024-03-06T14:50:10.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:10.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:10.614416+0000) 2024-03-06T14:50:11.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:11.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:11.614648+0000) 2024-03-06T14:50:12.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:12.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:12.614887+0000) 2024-03-06T14:50:13.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:13.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:13.615097+0000) 2024-03-06T14:50:14.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:14.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:14.615338+0000) 2024-03-06T14:50:15.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:15.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:15.615601+0000) 2024-03-06T14:50:16.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:16.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:16.615835+0000) 2024-03-06T14:50:17.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:17.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:17.616052+0000) 2024-03-06T14:50:18.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:18.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:18.616300+0000) 2024-03-06T14:50:19.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:19.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:19.616557+0000) 2024-03-06T14:50:20.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:20.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:20.616794+0000) 2024-03-06T14:50:21.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:21.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:21.617002+0000) 2024-03-06T14:50:22.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:22.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:22.617207+0000) 2024-03-06T14:50:23.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:23.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:23.617456+0000) 2024-03-06T14:50:24.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:24.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:24.617754+0000) 2024-03-06T14:50:25.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:25.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:25.618004+0000) 2024-03-06T14:50:26.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:26.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:26.618258+0000) 2024-03-06T14:50:27.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:27.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:27.618502+0000) 2024-03-06T14:50:28.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:28.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:28.618797+0000) 2024-03-06T14:50:29.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:29.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:29.619012+0000) 2024-03-06T14:50:30.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:30.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:30.619237+0000) 2024-03-06T14:50:31.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:31.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:31.619481+0000) 2024-03-06T14:50:32.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:32.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:32.619775+0000) 2024-03-06T14:50:33.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:33.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:33.619969+0000) 2024-03-06T14:50:34.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:34.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:34.620201+0000) 2024-03-06T14:50:35.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:35.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:35.620432+0000) 2024-03-06T14:50:36.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:36.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:36.620661+0000) 2024-03-06T14:50:37.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:37.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:37.620928+0000) 2024-03-06T14:50:38.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:38.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:38.621182+0000) 2024-03-06T14:50:39.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:39.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:39.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:39.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:39.621478+0000) 2024-03-06T14:50:40.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:40.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:40.621714+0000) 2024-03-06T14:50:41.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:41.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:41.621955+0000) 2024-03-06T14:50:42.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:42.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:42.622238+0000) 2024-03-06T14:50:43.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:43.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:43.622469+0000) 2024-03-06T14:50:44.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:44.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:44.622656+0000) 2024-03-06T14:50:45.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:45.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:45.622926+0000) 2024-03-06T14:50:46.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:46.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:46.623116+0000) 2024-03-06T14:50:47.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:47.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:47.623321+0000) 2024-03-06T14:50:48.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:48.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:48.623543+0000) 2024-03-06T14:50:49.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:49.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:49.623803+0000) 2024-03-06T14:50:50.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:50.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:50.623999+0000) 2024-03-06T14:50:51.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:51.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:51.624236+0000) 2024-03-06T14:50:52.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:52.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:52.624474+0000) 2024-03-06T14:50:53.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:53.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:53.624763+0000) 2024-03-06T14:50:54.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:54.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:54.624962+0000) 2024-03-06T14:50:55.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:55.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:55.625190+0000) 2024-03-06T14:50:56.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:56.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:56.625425+0000) 2024-03-06T14:50:57.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:57.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:57.625702+0000) 2024-03-06T14:50:58.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:58.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:58.625904+0000) 2024-03-06T14:50:59.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:50:59.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:59.626116+0000) 2024-03-06T14:51:00.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:00.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:00.626308+0000) 2024-03-06T14:51:01.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:01.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:01.626558+0000) 2024-03-06T14:51:02.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:02.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:02.626769+0000) 2024-03-06T14:51:03.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:03.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:03.626933+0000) 2024-03-06T14:51:04.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:04.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:04.627112+0000) 2024-03-06T14:51:05.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:05.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:05.627349+0000) 2024-03-06T14:51:06.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:06.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:06.627627+0000) 2024-03-06T14:51:07.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:07.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:07.627917+0000) 2024-03-06T14:51:08.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:08.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:08.628197+0000) 2024-03-06T14:51:09.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:09.471 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:09.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:09.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:09.628495+0000) 2024-03-06T14:51:10.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:10.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:10.628725+0000) 2024-03-06T14:51:11.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:11.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:11.629007+0000) 2024-03-06T14:51:12.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:12.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:12.629263+0000) 2024-03-06T14:51:13.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:13.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:13.629568+0000) 2024-03-06T14:51:14.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:14.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:14.629868+0000) 2024-03-06T14:51:15.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:15.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:15.630156+0000) 2024-03-06T14:51:16.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:16.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:16.630408+0000) 2024-03-06T14:51:17.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:17.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:17.630594+0000) 2024-03-06T14:51:18.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:18.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:18.630801+0000) 2024-03-06T14:51:19.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:19.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:19.631041+0000) 2024-03-06T14:51:20.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:20.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:20.631266+0000) 2024-03-06T14:51:21.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:21.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:21.631501+0000) 2024-03-06T14:51:22.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:22.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:22.631801+0000) 2024-03-06T14:51:23.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:23.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:23.632017+0000) 2024-03-06T14:51:24.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:24.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:24.632235+0000) 2024-03-06T14:51:25.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:25.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:25.632468+0000) 2024-03-06T14:51:26.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:26.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:26.632768+0000) 2024-03-06T14:51:27.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:27.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:27.633000+0000) 2024-03-06T14:51:28.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:28.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:28.633246+0000) 2024-03-06T14:51:29.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:29.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:29.633505+0000) 2024-03-06T14:51:30.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:30.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:30.633788+0000) 2024-03-06T14:51:31.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:31.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:31.634015+0000) 2024-03-06T14:51:32.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:32.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:32.634254+0000) 2024-03-06T14:51:33.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:33.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:33.634505+0000) 2024-03-06T14:51:34.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:34.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:34.634778+0000) 2024-03-06T14:51:35.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:35.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:35.634971+0000) 2024-03-06T14:51:36.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:36.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:36.635299+0000) 2024-03-06T14:51:37.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:37.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:37.635536+0000) 2024-03-06T14:51:38.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:38.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:38.635805+0000) 2024-03-06T14:51:39.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:39.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:39.636120+0000) 2024-03-06T14:51:39.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:39.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:40.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:40.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:40.636406+0000) 2024-03-06T14:51:41.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:41.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:41.636626+0000) 2024-03-06T14:51:42.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:42.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:42.636883+0000) 2024-03-06T14:51:43.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:43.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:43.637122+0000) 2024-03-06T14:51:44.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:44.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:44.637347+0000) 2024-03-06T14:51:45.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:45.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:45.637540+0000) 2024-03-06T14:51:46.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:46.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:46.637806+0000) 2024-03-06T14:51:47.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:47.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:47.638009+0000) 2024-03-06T14:51:48.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:48.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:48.638238+0000) 2024-03-06T14:51:49.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:49.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:49.638470+0000) 2024-03-06T14:51:50.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:50.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:50.638768+0000) 2024-03-06T14:51:51.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:51.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:51.639059+0000) 2024-03-06T14:51:52.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:52.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:52.639288+0000) 2024-03-06T14:51:53.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:53.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:53.639497+0000) 2024-03-06T14:51:54.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:54.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:54.639780+0000) 2024-03-06T14:51:55.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:55.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:55.639996+0000) 2024-03-06T14:51:56.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:56.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:56.640254+0000) 2024-03-06T14:51:57.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:57.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:57.640498+0000) 2024-03-06T14:51:58.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:58.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:58.640730+0000) 2024-03-06T14:51:59.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:51:59.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:59.641007+0000) 2024-03-06T14:52:00.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:00.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:00.641277+0000) 2024-03-06T14:52:01.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:01.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:01.641565+0000) 2024-03-06T14:52:02.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:02.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:02.641842+0000) 2024-03-06T14:52:03.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:03.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:03.642116+0000) 2024-03-06T14:52:04.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:04.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:04.642337+0000) 2024-03-06T14:52:05.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:05.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:05.642552+0000) 2024-03-06T14:52:06.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:06.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:06.642805+0000) 2024-03-06T14:52:07.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:07.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:07.643012+0000) 2024-03-06T14:52:08.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:08.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:08.643235+0000) 2024-03-06T14:52:09.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:09.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:09.643406+0000) 2024-03-06T14:52:09.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:09.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:10.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:10.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:10.643656+0000) 2024-03-06T14:52:11.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:11.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:11.643940+0000) 2024-03-06T14:52:12.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:12.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:12.644154+0000) 2024-03-06T14:52:13.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:13.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:13.644402+0000) 2024-03-06T14:52:14.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:14.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:14.644636+0000) 2024-03-06T14:52:15.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:15.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:15.644895+0000) 2024-03-06T14:52:16.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:16.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:16.645141+0000) 2024-03-06T14:52:17.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:17.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:17.645337+0000) 2024-03-06T14:52:18.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:18.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:18.645567+0000) 2024-03-06T14:52:19.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:19.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:19.645799+0000) 2024-03-06T14:52:20.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:20.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:20.645960+0000) 2024-03-06T14:52:21.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:21.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:21.646175+0000) 2024-03-06T14:52:22.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:22.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:22.646366+0000) 2024-03-06T14:52:23.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:23.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:23.646614+0000) 2024-03-06T14:52:24.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:24.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:24.646920+0000) 2024-03-06T14:52:25.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:25.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:25.647118+0000) 2024-03-06T14:52:26.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:26.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:26.647379+0000) 2024-03-06T14:52:27.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:27.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:27.647621+0000) 2024-03-06T14:52:28.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:28.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:28.647883+0000) 2024-03-06T14:52:29.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:29.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:29.648128+0000) 2024-03-06T14:52:30.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:30.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:30.648407+0000) 2024-03-06T14:52:31.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:31.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:31.648719+0000) 2024-03-06T14:52:32.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:32.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:32.648979+0000) 2024-03-06T14:52:33.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:33.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:33.649257+0000) 2024-03-06T14:52:34.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:34.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:34.649461+0000) 2024-03-06T14:52:35.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:35.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:35.649752+0000) 2024-03-06T14:52:36.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:36.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:36.649979+0000) 2024-03-06T14:52:37.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:37.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:37.650229+0000) 2024-03-06T14:52:38.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:38.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:38.650473+0000) 2024-03-06T14:52:39.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:39.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:39.650762+0000) 2024-03-06T14:52:40.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:40.115 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:40.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:40.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:40.650986+0000) 2024-03-06T14:52:41.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:41.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:41.651209+0000) 2024-03-06T14:52:42.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:42.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:42.651449+0000) 2024-03-06T14:52:43.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:43.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:43.651752+0000) 2024-03-06T14:52:44.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:44.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:44.651985+0000) 2024-03-06T14:52:45.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:45.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:45.652190+0000) 2024-03-06T14:52:46.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:46.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:46.652448+0000) 2024-03-06T14:52:47.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:47.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:47.652733+0000) 2024-03-06T14:52:48.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:48.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:48.652985+0000) 2024-03-06T14:52:49.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:49.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:49.653213+0000) 2024-03-06T14:52:50.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:50.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:50.653406+0000) 2024-03-06T14:52:51.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:51.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:51.653640+0000) 2024-03-06T14:52:52.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:52.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:52.653889+0000) 2024-03-06T14:52:53.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:53.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:53.654133+0000) 2024-03-06T14:52:54.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:54.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:54.654331+0000) 2024-03-06T14:52:55.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:55.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:55.654542+0000) 2024-03-06T14:52:56.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:56.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:56.654824+0000) 2024-03-06T14:52:57.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:57.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:57.655079+0000) 2024-03-06T14:52:58.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:58.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:58.655335+0000) 2024-03-06T14:52:59.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:52:59.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:59.655559+0000) 2024-03-06T14:53:00.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:00.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:00.655822+0000) 2024-03-06T14:53:01.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:01.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:01.656042+0000) 2024-03-06T14:53:02.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:02.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:02.656218+0000) 2024-03-06T14:53:03.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:03.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:03.656401+0000) 2024-03-06T14:53:04.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:04.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:04.656606+0000) 2024-03-06T14:53:05.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:05.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:05.656788+0000) 2024-03-06T14:53:06.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:06.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:06.656931+0000) 2024-03-06T14:53:07.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:07.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:07.657115+0000) 2024-03-06T14:53:08.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:08.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:08.657317+0000) 2024-03-06T14:53:09.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:09.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:09.657516+0000) 2024-03-06T14:53:10.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:10.310 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:10.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:10.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:10.657778+0000) 2024-03-06T14:53:11.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:11.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:11.658038+0000) 2024-03-06T14:53:12.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:12.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:12.658221+0000) 2024-03-06T14:53:13.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:13.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:13.658404+0000) 2024-03-06T14:53:14.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:14.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:14.658595+0000) 2024-03-06T14:53:15.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:15.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:15.658832+0000) 2024-03-06T14:53:16.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:16.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:16.659028+0000) 2024-03-06T14:53:17.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:17.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:17.659266+0000) 2024-03-06T14:53:18.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:18.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:18.659520+0000) 2024-03-06T14:53:19.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:19.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:19.659792+0000) 2024-03-06T14:53:20.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:20.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:20.659949+0000) 2024-03-06T14:53:21.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:21.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:21.660183+0000) 2024-03-06T14:53:22.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:22.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:22.660385+0000) 2024-03-06T14:53:23.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:23.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:23.660613+0000) 2024-03-06T14:53:24.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:24.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:24.660839+0000) 2024-03-06T14:53:25.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:25.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:25.661024+0000) 2024-03-06T14:53:26.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:26.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:26.661258+0000) 2024-03-06T14:53:27.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:27.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:27.661466+0000) 2024-03-06T14:53:28.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:28.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:28.661737+0000) 2024-03-06T14:53:29.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:29.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:29.661977+0000) 2024-03-06T14:53:30.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:30.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:30.662165+0000) 2024-03-06T14:53:31.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:31.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:31.662446+0000) 2024-03-06T14:53:32.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:32.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:32.662765+0000) 2024-03-06T14:53:33.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:33.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:33.663044+0000) 2024-03-06T14:53:34.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:34.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:34.663243+0000) 2024-03-06T14:53:35.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:35.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:35.663438+0000) 2024-03-06T14:53:36.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:36.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:36.663745+0000) 2024-03-06T14:53:37.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:37.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:37.663987+0000) 2024-03-06T14:53:38.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:38.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:38.664236+0000) 2024-03-06T14:53:39.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:39.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:39.664478+0000) 2024-03-06T14:53:40.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:40.632 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:40.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:40.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:40.664640+0000) 2024-03-06T14:53:41.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:41.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:41.664829+0000) 2024-03-06T14:53:42.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:42.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:42.665053+0000) 2024-03-06T14:53:43.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:43.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:43.665315+0000) 2024-03-06T14:53:44.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:44.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:44.665562+0000) 2024-03-06T14:53:45.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:45.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:45.665809+0000) 2024-03-06T14:53:46.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:46.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:46.666051+0000) 2024-03-06T14:53:47.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:47.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:47.666299+0000) 2024-03-06T14:53:48.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:48.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:48.666554+0000) 2024-03-06T14:53:49.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:49.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:49.666818+0000) 2024-03-06T14:53:50.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:50.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:50.667024+0000) 2024-03-06T14:53:51.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:51.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:51.667242+0000) 2024-03-06T14:53:52.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:52.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:52.667487+0000) 2024-03-06T14:53:53.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:53.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:53.667803+0000) 2024-03-06T14:53:54.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:54.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:54.668032+0000) 2024-03-06T14:53:55.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:55.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:55.668284+0000) 2024-03-06T14:53:56.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:56.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:56.668533+0000) 2024-03-06T14:53:57.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:57.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:57.668792+0000) 2024-03-06T14:53:58.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:58.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:58.668985+0000) 2024-03-06T14:53:59.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:53:59.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:59.669212+0000) 2024-03-06T14:54:00.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:00.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:00.669410+0000) 2024-03-06T14:54:01.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:01.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:01.669709+0000) 2024-03-06T14:54:02.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:02.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:02.669992+0000) 2024-03-06T14:54:03.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:03.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:03.670281+0000) 2024-03-06T14:54:04.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:04.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:04.670560+0000) 2024-03-06T14:54:05.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:05.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:05.670857+0000) 2024-03-06T14:54:06.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:06.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:06.671124+0000) 2024-03-06T14:54:07.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:07.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:07.671401+0000) 2024-03-06T14:54:08.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:08.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:08.671628+0000) 2024-03-06T14:54:09.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:09.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:09.671884+0000) 2024-03-06T14:54:10.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:10.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:10.672083+0000) 2024-03-06T14:54:10.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:10.852 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:11.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:11.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:11.672313+0000) 2024-03-06T14:54:12.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:12.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:12.672544+0000) 2024-03-06T14:54:13.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:13.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:13.672779+0000) 2024-03-06T14:54:14.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:14.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:14.672992+0000) 2024-03-06T14:54:15.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:15.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:15.673241+0000) 2024-03-06T14:54:16.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:16.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:16.673468+0000) 2024-03-06T14:54:17.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:17.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:17.673694+0000) 2024-03-06T14:54:18.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:18.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:18.673931+0000) 2024-03-06T14:54:19.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:19.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:19.674181+0000) 2024-03-06T14:54:20.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:20.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:20.674416+0000) 2024-03-06T14:54:21.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:21.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:21.674655+0000) 2024-03-06T14:54:22.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:22.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:22.674983+0000) 2024-03-06T14:54:23.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:23.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:23.675256+0000) 2024-03-06T14:54:24.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:24.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:24.675461+0000) 2024-03-06T14:54:25.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:25.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:25.675745+0000) 2024-03-06T14:54:26.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:26.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:26.675906+0000) 2024-03-06T14:54:27.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:27.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:27.676144+0000) 2024-03-06T14:54:28.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:28.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:28.676387+0000) 2024-03-06T14:54:29.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:29.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:29.676642+0000) 2024-03-06T14:54:30.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:30.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:30.676933+0000) 2024-03-06T14:54:31.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:31.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:31.677185+0000) 2024-03-06T14:54:32.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:32.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:32.677433+0000) 2024-03-06T14:54:33.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:33.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:33.677635+0000) 2024-03-06T14:54:34.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:34.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:34.677886+0000) 2024-03-06T14:54:35.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:35.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:35.678102+0000) 2024-03-06T14:54:36.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:36.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:36.678351+0000) 2024-03-06T14:54:37.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:37.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:37.678651+0000) 2024-03-06T14:54:38.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:38.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:38.679004+0000) 2024-03-06T14:54:39.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:39.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:39.679298+0000) 2024-03-06T14:54:40.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:40.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:40.679570+0000) 2024-03-06T14:54:41.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:41.049 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:41.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:41.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:41.679853+0000) 2024-03-06T14:54:42.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:42.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:42.680104+0000) 2024-03-06T14:54:43.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:43.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:43.680393+0000) 2024-03-06T14:54:44.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:44.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:44.680715+0000) 2024-03-06T14:54:45.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:45.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:45.680961+0000) 2024-03-06T14:54:46.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:46.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:46.681171+0000) 2024-03-06T14:54:47.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:47.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:47.681405+0000) 2024-03-06T14:54:48.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:48.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:48.681642+0000) 2024-03-06T14:54:49.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:49.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:49.681924+0000) 2024-03-06T14:54:50.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:50.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:50.682158+0000) 2024-03-06T14:54:51.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:51.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:51.682394+0000) 2024-03-06T14:54:52.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:52.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:52.682624+0000) 2024-03-06T14:54:53.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:53.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:53.682908+0000) 2024-03-06T14:54:54.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:54.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:54.683131+0000) 2024-03-06T14:54:55.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:55.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:55.683364+0000) 2024-03-06T14:54:56.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:56.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:56.683599+0000) 2024-03-06T14:54:57.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:57.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:57.683847+0000) 2024-03-06T14:54:58.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:58.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:58.684062+0000) 2024-03-06T14:54:59.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:54:59.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:59.684322+0000) 2024-03-06T14:55:00.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:00.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:00.684534+0000) 2024-03-06T14:55:01.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:01.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:01.684820+0000) 2024-03-06T14:55:02.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:02.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:02.685064+0000) 2024-03-06T14:55:03.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:03.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:03.685339+0000) 2024-03-06T14:55:04.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:04.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:04.685533+0000) 2024-03-06T14:55:05.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:05.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:05.685815+0000) 2024-03-06T14:55:06.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:06.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:06.686050+0000) 2024-03-06T14:55:07.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:07.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:07.686282+0000) 2024-03-06T14:55:08.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:08.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:08.686511+0000) 2024-03-06T14:55:09.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:09.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:09.686752+0000) 2024-03-06T14:55:10.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:10.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:10.686946+0000) 2024-03-06T14:55:11.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:11.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:11.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:11.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:11.687144+0000) 2024-03-06T14:55:12.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:12.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:12.687418+0000) 2024-03-06T14:55:13.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:13.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:13.687626+0000) 2024-03-06T14:55:14.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:14.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:14.687812+0000) 2024-03-06T14:55:15.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:15.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:15.688017+0000) 2024-03-06T14:55:16.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:16.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:16.688242+0000) 2024-03-06T14:55:17.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:17.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:17.688514+0000) 2024-03-06T14:55:18.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:18.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:18.688775+0000) 2024-03-06T14:55:19.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:19.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:19.689010+0000) 2024-03-06T14:55:20.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:20.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:20.689237+0000) 2024-03-06T14:55:21.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:21.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:21.689467+0000) 2024-03-06T14:55:22.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:22.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:22.689744+0000) 2024-03-06T14:55:23.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:23.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:23.689978+0000) 2024-03-06T14:55:24.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:24.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:24.690204+0000) 2024-03-06T14:55:25.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:25.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:25.690446+0000) 2024-03-06T14:55:26.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:26.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:26.690720+0000) 2024-03-06T14:55:27.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:27.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:27.690971+0000) 2024-03-06T14:55:28.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:28.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:28.691201+0000) 2024-03-06T14:55:29.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:29.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:29.691416+0000) 2024-03-06T14:55:30.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:30.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:30.691697+0000) 2024-03-06T14:55:31.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:31.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:31.691903+0000) 2024-03-06T14:55:32.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:32.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:32.692182+0000) 2024-03-06T14:55:33.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:33.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:33.692466+0000) 2024-03-06T14:55:34.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:34.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:34.692764+0000) 2024-03-06T14:55:35.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:35.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:35.693046+0000) 2024-03-06T14:55:36.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:36.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:36.693324+0000) 2024-03-06T14:55:37.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:37.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:37.693610+0000) 2024-03-06T14:55:38.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:38.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:38.693854+0000) 2024-03-06T14:55:39.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:39.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:39.694051+0000) 2024-03-06T14:55:40.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:40.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:40.694272+0000) 2024-03-06T14:55:41.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:41.424 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:41.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:41.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:41.694479+0000) 2024-03-06T14:55:42.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:42.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:42.694785+0000) 2024-03-06T14:55:43.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:43.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:43.694966+0000) 2024-03-06T14:55:44.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:44.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:44.695146+0000) 2024-03-06T14:55:45.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:45.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:45.695420+0000) 2024-03-06T14:55:46.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:46.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:46.695808+0000) 2024-03-06T14:55:47.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:47.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:47.696005+0000) 2024-03-06T14:55:48.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:48.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:48.696237+0000) 2024-03-06T14:55:49.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:49.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:49.696476+0000) 2024-03-06T14:55:50.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:50.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:50.696702+0000) 2024-03-06T14:55:51.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:51.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:51.696903+0000) 2024-03-06T14:55:52.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:52.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:52.697177+0000) 2024-03-06T14:55:53.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:53.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:53.697464+0000) 2024-03-06T14:55:54.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:54.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:54.697787+0000) 2024-03-06T14:55:55.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:55.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:55.698068+0000) 2024-03-06T14:55:56.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:56.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:56.698349+0000) 2024-03-06T14:55:57.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:57.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:57.698633+0000) 2024-03-06T14:55:58.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:58.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:58.698896+0000) 2024-03-06T14:55:59.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:55:59.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:59.699140+0000) 2024-03-06T14:56:00.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:00.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:00.699355+0000) 2024-03-06T14:56:01.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:01.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:01.699569+0000) 2024-03-06T14:56:02.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:02.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:02.699840+0000) 2024-03-06T14:56:03.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:03.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:03.700097+0000) 2024-03-06T14:56:04.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:04.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:04.700375+0000) 2024-03-06T14:56:05.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:05.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:05.700657+0000) 2024-03-06T14:56:06.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:06.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:06.700990+0000) 2024-03-06T14:56:07.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:07.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:07.701266+0000) 2024-03-06T14:56:08.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:08.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:08.701540+0000) 2024-03-06T14:56:09.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:09.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:09.701831+0000) 2024-03-06T14:56:10.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:10.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:10.702050+0000) 2024-03-06T14:56:11.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:11.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:11.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:11.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:11.702228+0000) 2024-03-06T14:56:12.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:12.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:12.702498+0000) 2024-03-06T14:56:13.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:13.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:13.702819+0000) 2024-03-06T14:56:14.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:14.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:14.703056+0000) 2024-03-06T14:56:15.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:15.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:15.703262+0000) 2024-03-06T14:56:16.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:16.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:16.703487+0000) 2024-03-06T14:56:17.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:17.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:17.703715+0000) 2024-03-06T14:56:18.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:18.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:18.703959+0000) 2024-03-06T14:56:19.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:19.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:19.704163+0000) 2024-03-06T14:56:20.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:20.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:20.704393+0000) 2024-03-06T14:56:21.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:21.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:21.704621+0000) 2024-03-06T14:56:22.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:22.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:22.704831+0000) 2024-03-06T14:56:23.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:23.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:23.705025+0000) 2024-03-06T14:56:24.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:24.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:24.705240+0000) 2024-03-06T14:56:25.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:25.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:25.705411+0000) 2024-03-06T14:56:26.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:26.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:26.705595+0000) 2024-03-06T14:56:27.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:27.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:27.705837+0000) 2024-03-06T14:56:28.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:28.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:28.706029+0000) 2024-03-06T14:56:29.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:29.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:29.706273+0000) 2024-03-06T14:56:30.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:30.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:30.706499+0000) 2024-03-06T14:56:31.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:31.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:31.706782+0000) 2024-03-06T14:56:32.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:32.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:32.706948+0000) 2024-03-06T14:56:33.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:33.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:33.707165+0000) 2024-03-06T14:56:34.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:34.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:34.707357+0000) 2024-03-06T14:56:35.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:35.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:35.707582+0000) 2024-03-06T14:56:36.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:36.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:36.707811+0000) 2024-03-06T14:56:37.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:37.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:37.707974+0000) 2024-03-06T14:56:38.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:38.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:38.708208+0000) 2024-03-06T14:56:39.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:39.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:39.708438+0000) 2024-03-06T14:56:40.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:40.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:40.708698+0000) 2024-03-06T14:56:41.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:41.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:41.708904+0000) 2024-03-06T14:56:41.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:41.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:42.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:42.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:42.709137+0000) 2024-03-06T14:56:43.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:43.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:43.709375+0000) 2024-03-06T14:56:44.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:44.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:44.709613+0000) 2024-03-06T14:56:45.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:45.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:45.709826+0000) 2024-03-06T14:56:46.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:46.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:46.710022+0000) 2024-03-06T14:56:47.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:47.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:47.710253+0000) 2024-03-06T14:56:48.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:48.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:48.710483+0000) 2024-03-06T14:56:49.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:49.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:49.710722+0000) 2024-03-06T14:56:50.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:50.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:50.710931+0000) 2024-03-06T14:56:51.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:51.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:51.711092+0000) 2024-03-06T14:56:52.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:52.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:52.711326+0000) 2024-03-06T14:56:53.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:53.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:53.711558+0000) 2024-03-06T14:56:54.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:54.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:54.711786+0000) 2024-03-06T14:56:55.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:55.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:55.712024+0000) 2024-03-06T14:56:56.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:56.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:56.712251+0000) 2024-03-06T14:56:57.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:57.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:57.712486+0000) 2024-03-06T14:56:58.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:58.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:58.712734+0000) 2024-03-06T14:56:59.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:56:59.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:59.712983+0000) 2024-03-06T14:57:00.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:00.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:00.713181+0000) 2024-03-06T14:57:01.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:01.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:01.713397+0000) 2024-03-06T14:57:02.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:02.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:02.713645+0000) 2024-03-06T14:57:03.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:03.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:03.713904+0000) 2024-03-06T14:57:04.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:04.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:04.714100+0000) 2024-03-06T14:57:05.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:05.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:05.714345+0000) 2024-03-06T14:57:06.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:06.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:06.714600+0000) 2024-03-06T14:57:07.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:07.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:07.714853+0000) 2024-03-06T14:57:08.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:08.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:08.715107+0000) 2024-03-06T14:57:09.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:09.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:09.715338+0000) 2024-03-06T14:57:10.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:10.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:10.715572+0000) 2024-03-06T14:57:11.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:11.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:11.715804+0000) 2024-03-06T14:57:12.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:12.095 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:12.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:12.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:12.716002+0000) 2024-03-06T14:57:13.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:13.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:13.716235+0000) 2024-03-06T14:57:14.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:14.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:14.716470+0000) 2024-03-06T14:57:15.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:15.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:15.716740+0000) 2024-03-06T14:57:16.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:16.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:16.716934+0000) 2024-03-06T14:57:17.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:17.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:17.717142+0000) 2024-03-06T14:57:18.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:18.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:18.717370+0000) 2024-03-06T14:57:19.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:19.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:19.717606+0000) 2024-03-06T14:57:20.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:20.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:20.717851+0000) 2024-03-06T14:57:21.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:21.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:21.718052+0000) 2024-03-06T14:57:22.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:22.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:22.718342+0000) 2024-03-06T14:57:23.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:23.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:23.718575+0000) 2024-03-06T14:57:24.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:24.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:24.718777+0000) 2024-03-06T14:57:25.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:25.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:25.718972+0000) 2024-03-06T14:57:26.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:26.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:26.719208+0000) 2024-03-06T14:57:27.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:27.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:27.719465+0000) 2024-03-06T14:57:28.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:28.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:28.719756+0000) 2024-03-06T14:57:29.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:29.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:29.720049+0000) 2024-03-06T14:57:30.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:30.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:30.720295+0000) 2024-03-06T14:57:31.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:31.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:31.720489+0000) 2024-03-06T14:57:32.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:32.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:32.720794+0000) 2024-03-06T14:57:33.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:33.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:33.721007+0000) 2024-03-06T14:57:34.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:34.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:34.721232+0000) 2024-03-06T14:57:35.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:35.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:35.721453+0000) 2024-03-06T14:57:36.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:36.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:36.721717+0000) 2024-03-06T14:57:37.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:37.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:37.721926+0000) 2024-03-06T14:57:38.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:38.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:38.722160+0000) 2024-03-06T14:57:39.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:39.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:39.722394+0000) 2024-03-06T14:57:40.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:40.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:40.722575+0000) 2024-03-06T14:57:41.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:41.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:41.722809+0000) 2024-03-06T14:57:42.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:42.309 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:42.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:42.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:42.723052+0000) 2024-03-06T14:57:43.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:43.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:43.723239+0000) 2024-03-06T14:57:44.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:44.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:44.723452+0000) 2024-03-06T14:57:45.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:45.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:45.723656+0000) 2024-03-06T14:57:46.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:46.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:46.723881+0000) 2024-03-06T14:57:47.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:47.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:47.724047+0000) 2024-03-06T14:57:48.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:48.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:48.724237+0000) 2024-03-06T14:57:49.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:49.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:49.724456+0000) 2024-03-06T14:57:50.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:50.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:50.724732+0000) 2024-03-06T14:57:51.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:51.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:51.724929+0000) 2024-03-06T14:57:52.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:52.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:52.725166+0000) 2024-03-06T14:57:53.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:53.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:53.725400+0000) 2024-03-06T14:57:54.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:54.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:54.725714+0000) 2024-03-06T14:57:55.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:55.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:55.725931+0000) 2024-03-06T14:57:56.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:56.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:56.726156+0000) 2024-03-06T14:57:57.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:57.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:57.726397+0000) 2024-03-06T14:57:58.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:58.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:58.726604+0000) 2024-03-06T14:57:59.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:57:59.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:59.726844+0000) 2024-03-06T14:58:00.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:00.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:00.727040+0000) 2024-03-06T14:58:01.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:01.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:01.727233+0000) 2024-03-06T14:58:02.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:02.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:02.727483+0000) 2024-03-06T14:58:03.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:03.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:03.727752+0000) 2024-03-06T14:58:04.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:04.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:04.727929+0000) 2024-03-06T14:58:05.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:05.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:05.728135+0000) 2024-03-06T14:58:06.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:06.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:06.728390+0000) 2024-03-06T14:58:07.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:07.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:07.728694+0000) 2024-03-06T14:58:08.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:08.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:08.728860+0000) 2024-03-06T14:58:09.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:09.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:09.729102+0000) 2024-03-06T14:58:10.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:10.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:10.729333+0000) 2024-03-06T14:58:11.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:11.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:11.729561+0000) 2024-03-06T14:58:12.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:12.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:12.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:12.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:12.729798+0000) 2024-03-06T14:58:13.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:13.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:13.730021+0000) 2024-03-06T14:58:14.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:14.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:14.730242+0000) 2024-03-06T14:58:15.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:15.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:15.730466+0000) 2024-03-06T14:58:16.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:16.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:16.730782+0000) 2024-03-06T14:58:17.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:17.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:17.731025+0000) 2024-03-06T14:58:18.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:18.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:18.731259+0000) 2024-03-06T14:58:19.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:19.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:19.731508+0000) 2024-03-06T14:58:20.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:20.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:20.731792+0000) 2024-03-06T14:58:21.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:21.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:21.732034+0000) 2024-03-06T14:58:22.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:22.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:22.732295+0000) 2024-03-06T14:58:23.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:23.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:23.732542+0000) 2024-03-06T14:58:24.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:24.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:24.732782+0000) 2024-03-06T14:58:25.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:25.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:25.732990+0000) 2024-03-06T14:58:26.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:26.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:26.733237+0000) 2024-03-06T14:58:27.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:27.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:27.733486+0000) 2024-03-06T14:58:28.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:28.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:28.733763+0000) 2024-03-06T14:58:29.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:29.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:29.734024+0000) 2024-03-06T14:58:30.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:30.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:30.734270+0000) 2024-03-06T14:58:31.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:31.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:31.734507+0000) 2024-03-06T14:58:32.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:32.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:32.734797+0000) 2024-03-06T14:58:33.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:33.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:33.735044+0000) 2024-03-06T14:58:34.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:34.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:34.735261+0000) 2024-03-06T14:58:35.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:35.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:35.735504+0000) 2024-03-06T14:58:36.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:36.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:36.735771+0000) 2024-03-06T14:58:37.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:37.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:37.736010+0000) 2024-03-06T14:58:38.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:38.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:38.736207+0000) 2024-03-06T14:58:39.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:39.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:39.736465+0000) 2024-03-06T14:58:40.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:40.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:40.736643+0000) 2024-03-06T14:58:41.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:41.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:41.736908+0000) 2024-03-06T14:58:42.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:42.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:42.737111+0000) 2024-03-06T14:58:42.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:42.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:43.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:43.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:43.737335+0000) 2024-03-06T14:58:44.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:44.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:44.737521+0000) 2024-03-06T14:58:45.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:45.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:45.737790+0000) 2024-03-06T14:58:46.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:46.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:46.738021+0000) 2024-03-06T14:58:47.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:47.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:47.738256+0000) 2024-03-06T14:58:48.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:48.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:48.738458+0000) 2024-03-06T14:58:49.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:49.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:49.738638+0000) 2024-03-06T14:58:50.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:50.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:50.738835+0000) 2024-03-06T14:58:51.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:51.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:51.739078+0000) 2024-03-06T14:58:52.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:52.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:52.739311+0000) 2024-03-06T14:58:53.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:53.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:53.739556+0000) 2024-03-06T14:58:54.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:54.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:54.739790+0000) 2024-03-06T14:58:55.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:55.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:55.739990+0000) 2024-03-06T14:58:56.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:56.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:56.740224+0000) 2024-03-06T14:58:57.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:57.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:57.740459+0000) 2024-03-06T14:58:58.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:58.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:58.740648+0000) 2024-03-06T14:58:59.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:58:59.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:59.740962+0000) 2024-03-06T14:59:00.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:00.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:00.741212+0000) 2024-03-06T14:59:01.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:01.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:01.741454+0000) 2024-03-06T14:59:02.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:02.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:02.741659+0000) 2024-03-06T14:59:03.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:03.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:03.741906+0000) 2024-03-06T14:59:04.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:04.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:04.742178+0000) 2024-03-06T14:59:05.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:05.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:05.742410+0000) 2024-03-06T14:59:06.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:06.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:06.742589+0000) 2024-03-06T14:59:07.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:07.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:07.742841+0000) 2024-03-06T14:59:08.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:08.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:08.743091+0000) 2024-03-06T14:59:09.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:09.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:09.743308+0000) 2024-03-06T14:59:10.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:10.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:10.743506+0000) 2024-03-06T14:59:11.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:11.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:11.743788+0000) 2024-03-06T14:59:12.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:12.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:12.744024+0000) 2024-03-06T14:59:12.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:12.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:13.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:13.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:13.744282+0000) 2024-03-06T14:59:14.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:14.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:14.744533+0000) 2024-03-06T14:59:15.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:15.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:15.744779+0000) 2024-03-06T14:59:16.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:16.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:16.744995+0000) 2024-03-06T14:59:17.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:17.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:17.745246+0000) 2024-03-06T14:59:18.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:18.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:18.745454+0000) 2024-03-06T14:59:19.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:19.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:19.745755+0000) 2024-03-06T14:59:20.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:20.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:20.745981+0000) 2024-03-06T14:59:21.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:21.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:21.746242+0000) 2024-03-06T14:59:22.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:22.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:22.746491+0000) 2024-03-06T14:59:23.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:23.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:23.746728+0000) 2024-03-06T14:59:24.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:24.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:24.746979+0000) 2024-03-06T14:59:25.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:25.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:25.747234+0000) 2024-03-06T14:59:26.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:26.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:26.747431+0000) 2024-03-06T14:59:27.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:27.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:27.747727+0000) 2024-03-06T14:59:28.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:28.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:28.747976+0000) 2024-03-06T14:59:29.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:29.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:29.748234+0000) 2024-03-06T14:59:30.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:30.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:30.748481+0000) 2024-03-06T14:59:31.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:31.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:31.748716+0000) 2024-03-06T14:59:32.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:32.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:32.748909+0000) 2024-03-06T14:59:33.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:33.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:33.749114+0000) 2024-03-06T14:59:34.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:34.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:34.749315+0000) 2024-03-06T14:59:35.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:35.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:35.749560+0000) 2024-03-06T14:59:36.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:36.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:36.749804+0000) 2024-03-06T14:59:37.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:37.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:37.750016+0000) 2024-03-06T14:59:38.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:38.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:38.750254+0000) 2024-03-06T14:59:39.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:39.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:39.750499+0000) 2024-03-06T14:59:40.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:40.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:40.750769+0000) 2024-03-06T14:59:41.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:41.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:41.751015+0000) 2024-03-06T14:59:42.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:42.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:42.751265+0000) 2024-03-06T14:59:43.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:43.209 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:43.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:43.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:43.751515+0000) 2024-03-06T14:59:44.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:44.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:44.751796+0000) 2024-03-06T14:59:45.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:45.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:45.752005+0000) 2024-03-06T14:59:46.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:46.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:46.752241+0000) 2024-03-06T14:59:47.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:47.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:47.752474+0000) 2024-03-06T14:59:48.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:48.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:48.752756+0000) 2024-03-06T14:59:49.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:49.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:49.752965+0000) 2024-03-06T14:59:50.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:50.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:50.753195+0000) 2024-03-06T14:59:51.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:51.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:51.753393+0000) 2024-03-06T14:59:52.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:52.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:52.753635+0000) 2024-03-06T14:59:53.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:53.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:53.753875+0000) 2024-03-06T14:59:54.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:54.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:54.754104+0000) 2024-03-06T14:59:55.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:55.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:55.754334+0000) 2024-03-06T14:59:56.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:56.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:56.754551+0000) 2024-03-06T14:59:57.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:57.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:57.754840+0000) 2024-03-06T14:59:58.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:58.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:58.755071+0000) 2024-03-06T14:59:59.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T14:59:59.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:59.755309+0000) 2024-03-06T15:00:00.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:00.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:00.755534+0000) 2024-03-06T15:00:01.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:01.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:01.755783+0000) 2024-03-06T15:00:02.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:02.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:02.755979+0000) 2024-03-06T15:00:03.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:03.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:03.756214+0000) 2024-03-06T15:00:04.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:04.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:04.756454+0000) 2024-03-06T15:00:05.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:05.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:05.756731+0000) 2024-03-06T15:00:06.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:06.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:06.756983+0000) 2024-03-06T15:00:07.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:07.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:07.757231+0000) 2024-03-06T15:00:08.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:08.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:08.757471+0000) 2024-03-06T15:00:09.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:09.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:09.757756+0000) 2024-03-06T15:00:10.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:10.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:10.757987+0000) 2024-03-06T15:00:11.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:11.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:11.758227+0000) 2024-03-06T15:00:12.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:12.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:12.758471+0000) 2024-03-06T15:00:13.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:13.404 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:13.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:13.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:13.758740+0000) 2024-03-06T15:00:14.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:14.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:14.758924+0000) 2024-03-06T15:00:15.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:15.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:15.759161+0000) 2024-03-06T15:00:16.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:16.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:16.759393+0000) 2024-03-06T15:00:17.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:17.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:17.759593+0000) 2024-03-06T15:00:18.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:18.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:18.759830+0000) 2024-03-06T15:00:19.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:19.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:19.760047+0000) 2024-03-06T15:00:20.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:20.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:20.760261+0000) 2024-03-06T15:00:21.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:21.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:21.760489+0000) 2024-03-06T15:00:22.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:22.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:22.760776+0000) 2024-03-06T15:00:23.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:23.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:23.760983+0000) 2024-03-06T15:00:24.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:24.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:24.761207+0000) 2024-03-06T15:00:25.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:25.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:25.761438+0000) 2024-03-06T15:00:26.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:26.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:26.761728+0000) 2024-03-06T15:00:27.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:27.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:27.761919+0000) 2024-03-06T15:00:28.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:28.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:28.762149+0000) 2024-03-06T15:00:29.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:29.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:29.762388+0000) 2024-03-06T15:00:30.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:30.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:30.762626+0000) 2024-03-06T15:00:31.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:31.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:31.762823+0000) 2024-03-06T15:00:32.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:32.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:32.763072+0000) 2024-03-06T15:00:33.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:33.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:33.763306+0000) 2024-03-06T15:00:34.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:34.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:34.763544+0000) 2024-03-06T15:00:35.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:35.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:35.763783+0000) 2024-03-06T15:00:36.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:36.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:36.764010+0000) 2024-03-06T15:00:37.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:37.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:37.764197+0000) 2024-03-06T15:00:38.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:38.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:38.764430+0000) 2024-03-06T15:00:39.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:39.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:39.764696+0000) 2024-03-06T15:00:40.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:40.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:40.764923+0000) 2024-03-06T15:00:41.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:41.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:41.765158+0000) 2024-03-06T15:00:42.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:42.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:42.765385+0000) 2024-03-06T15:00:43.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:43.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:43.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:43.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:43.765571+0000) 2024-03-06T15:00:44.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:44.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:44.765764+0000) 2024-03-06T15:00:45.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:45.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:45.766021+0000) 2024-03-06T15:00:46.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:46.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:46.766248+0000) 2024-03-06T15:00:47.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:47.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:47.766473+0000) 2024-03-06T15:00:48.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:48.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:48.766738+0000) 2024-03-06T15:00:49.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:49.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:49.766943+0000) 2024-03-06T15:00:50.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:50.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:50.767141+0000) 2024-03-06T15:00:51.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:51.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:51.767344+0000) 2024-03-06T15:00:52.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:52.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:52.767584+0000) 2024-03-06T15:00:53.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:53.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:53.767823+0000) 2024-03-06T15:00:54.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:54.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:54.768000+0000) 2024-03-06T15:00:55.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:55.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:55.768229+0000) 2024-03-06T15:00:56.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:56.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:56.768432+0000) 2024-03-06T15:00:57.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:57.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:57.768706+0000) 2024-03-06T15:00:58.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:58.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:58.768941+0000) 2024-03-06T15:00:59.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:00:59.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:59.769178+0000) 2024-03-06T15:01:00.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:00.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:00.769398+0000) 2024-03-06T15:01:01.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:01.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:01.769630+0000) 2024-03-06T15:01:02.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:02.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:02.769859+0000) 2024-03-06T15:01:03.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:03.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:03.770081+0000) 2024-03-06T15:01:04.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:04.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:04.770302+0000) 2024-03-06T15:01:05.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:05.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:05.770524+0000) 2024-03-06T15:01:06.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:06.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:06.770768+0000) 2024-03-06T15:01:07.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:07.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:07.770971+0000) 2024-03-06T15:01:08.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:08.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:08.771222+0000) 2024-03-06T15:01:09.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:09.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:09.771478+0000) 2024-03-06T15:01:10.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:10.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:10.771705+0000) 2024-03-06T15:01:11.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:11.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:11.771930+0000) 2024-03-06T15:01:12.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:12.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:12.772159+0000) 2024-03-06T15:01:13.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:13.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:13.772397+0000) 2024-03-06T15:01:13.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:13.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:14.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:14.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:14.772593+0000) 2024-03-06T15:01:15.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:15.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:15.772822+0000) 2024-03-06T15:01:16.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:16.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:16.773046+0000) 2024-03-06T15:01:17.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:17.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:17.773229+0000) 2024-03-06T15:01:18.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:18.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:18.773498+0000) 2024-03-06T15:01:19.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:19.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:19.773779+0000) 2024-03-06T15:01:20.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:20.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:20.774001+0000) 2024-03-06T15:01:21.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:21.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:21.774238+0000) 2024-03-06T15:01:22.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:22.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:22.774465+0000) 2024-03-06T15:01:23.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:23.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:23.774694+0000) 2024-03-06T15:01:24.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:24.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:24.774882+0000) 2024-03-06T15:01:25.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:25.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:25.775058+0000) 2024-03-06T15:01:26.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:26.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:26.775295+0000) 2024-03-06T15:01:27.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:27.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:27.775522+0000) 2024-03-06T15:01:28.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:28.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:28.775787+0000) 2024-03-06T15:01:29.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:29.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:29.776031+0000) 2024-03-06T15:01:30.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:30.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:30.776268+0000) 2024-03-06T15:01:31.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:31.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:31.776499+0000) 2024-03-06T15:01:32.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:32.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:32.776759+0000) 2024-03-06T15:01:33.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:33.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:33.777054+0000) 2024-03-06T15:01:34.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:34.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:34.777281+0000) 2024-03-06T15:01:35.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:35.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:35.777514+0000) 2024-03-06T15:01:36.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:36.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:36.777778+0000) 2024-03-06T15:01:37.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:37.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:37.777972+0000) 2024-03-06T15:01:38.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:38.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:38.778197+0000) 2024-03-06T15:01:39.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:39.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:39.778370+0000) 2024-03-06T15:01:40.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:40.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:40.778596+0000) 2024-03-06T15:01:41.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:41.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:41.778853+0000) 2024-03-06T15:01:42.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:42.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:42.779098+0000) 2024-03-06T15:01:43.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:43.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:43.779342+0000) 2024-03-06T15:01:44.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:44.116 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:44.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:44.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:44.779545+0000) 2024-03-06T15:01:45.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:45.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:45.779782+0000) 2024-03-06T15:01:46.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:46.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:46.779989+0000) 2024-03-06T15:01:47.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:47.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:47.780221+0000) 2024-03-06T15:01:48.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:48.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:48.780453+0000) 2024-03-06T15:01:49.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:49.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:49.780645+0000) 2024-03-06T15:01:50.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:50.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:50.780859+0000) 2024-03-06T15:01:51.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:51.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:51.781070+0000) 2024-03-06T15:01:52.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:52.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:52.781274+0000) 2024-03-06T15:01:53.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:53.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:53.781519+0000) 2024-03-06T15:01:54.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:54.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:54.781788+0000) 2024-03-06T15:01:55.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:55.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:55.781969+0000) 2024-03-06T15:01:56.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:56.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:56.782221+0000) 2024-03-06T15:01:57.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:57.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:57.782435+0000) 2024-03-06T15:01:58.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:58.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:58.782659+0000) 2024-03-06T15:01:59.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:01:59.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:59.782912+0000) 2024-03-06T15:02:00.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:00.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:00.783156+0000) 2024-03-06T15:02:01.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:01.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:01.783383+0000) 2024-03-06T15:02:02.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:02.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:02.783570+0000) 2024-03-06T15:02:03.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:03.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:03.783800+0000) 2024-03-06T15:02:04.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:04.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:04.784022+0000) 2024-03-06T15:02:05.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:05.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:05.784225+0000) 2024-03-06T15:02:06.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:06.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:06.784465+0000) 2024-03-06T15:02:07.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:07.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:07.784744+0000) 2024-03-06T15:02:08.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:08.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:08.784970+0000) 2024-03-06T15:02:09.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:09.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:09.785234+0000) 2024-03-06T15:02:10.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:10.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:10.785485+0000) 2024-03-06T15:02:11.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:11.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:11.785765+0000) 2024-03-06T15:02:12.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:12.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:12.786003+0000) 2024-03-06T15:02:13.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:13.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:13.786256+0000) 2024-03-06T15:02:14.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:14.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:14.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:14.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:14.786492+0000) 2024-03-06T15:02:15.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:15.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:15.786807+0000) 2024-03-06T15:02:16.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:16.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:16.787022+0000) 2024-03-06T15:02:17.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:17.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:17.787288+0000) 2024-03-06T15:02:18.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:18.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:18.787570+0000) 2024-03-06T15:02:19.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:19.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:19.787868+0000) 2024-03-06T15:02:20.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:20.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:20.788132+0000) 2024-03-06T15:02:21.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:21.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:21.788403+0000) 2024-03-06T15:02:22.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:22.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:22.788730+0000) 2024-03-06T15:02:23.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:23.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:23.788975+0000) 2024-03-06T15:02:24.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:24.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:24.789245+0000) 2024-03-06T15:02:25.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:25.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:25.789512+0000) 2024-03-06T15:02:26.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:26.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:26.789891+0000) 2024-03-06T15:02:27.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:27.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:27.790146+0000) 2024-03-06T15:02:28.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:28.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:28.790351+0000) 2024-03-06T15:02:29.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:29.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:29.790586+0000) 2024-03-06T15:02:30.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:30.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:30.790820+0000) 2024-03-06T15:02:31.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:31.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:31.791003+0000) 2024-03-06T15:02:32.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:32.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:32.791209+0000) 2024-03-06T15:02:33.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:33.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:33.791459+0000) 2024-03-06T15:02:34.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:34.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:34.791723+0000) 2024-03-06T15:02:35.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:35.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:35.791962+0000) 2024-03-06T15:02:36.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:36.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:36.792181+0000) 2024-03-06T15:02:37.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:37.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:37.792417+0000) 2024-03-06T15:02:38.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:38.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:38.792637+0000) 2024-03-06T15:02:39.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:39.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:39.792886+0000) 2024-03-06T15:02:40.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:40.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:40.793106+0000) 2024-03-06T15:02:41.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:41.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:41.793289+0000) 2024-03-06T15:02:42.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:42.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:42.793525+0000) 2024-03-06T15:02:43.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:43.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:43.793793+0000) 2024-03-06T15:02:44.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:44.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:44.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:44.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:44.793997+0000) 2024-03-06T15:02:45.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:45.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:45.794190+0000) 2024-03-06T15:02:46.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:46.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:46.794432+0000) 2024-03-06T15:02:47.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:47.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:47.794695+0000) 2024-03-06T15:02:48.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:48.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:48.794932+0000) 2024-03-06T15:02:49.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:49.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:49.795165+0000) 2024-03-06T15:02:50.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:50.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:50.795371+0000) 2024-03-06T15:02:51.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:51.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:51.795618+0000) 2024-03-06T15:02:52.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:52.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:52.795858+0000) 2024-03-06T15:02:53.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:53.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:53.796075+0000) 2024-03-06T15:02:54.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:54.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:54.796318+0000) 2024-03-06T15:02:55.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:55.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:55.796503+0000) 2024-03-06T15:02:56.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:56.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:56.796773+0000) 2024-03-06T15:02:57.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:57.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:57.797001+0000) 2024-03-06T15:02:58.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:58.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:58.797288+0000) 2024-03-06T15:02:59.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:02:59.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:59.797582+0000) 2024-03-06T15:03:00.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:00.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:00.797865+0000) 2024-03-06T15:03:01.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:01.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:01.798145+0000) 2024-03-06T15:03:02.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:02.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:02.798411+0000) 2024-03-06T15:03:03.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:03.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:03.798744+0000) 2024-03-06T15:03:04.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:04.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:04.798980+0000) 2024-03-06T15:03:05.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:05.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:05.799211+0000) 2024-03-06T15:03:06.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:06.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:06.799391+0000) 2024-03-06T15:03:07.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:07.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:07.799637+0000) 2024-03-06T15:03:08.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:08.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:08.799893+0000) 2024-03-06T15:03:09.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:09.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:09.800102+0000) 2024-03-06T15:03:10.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:10.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:10.800323+0000) 2024-03-06T15:03:11.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:11.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:11.800552+0000) 2024-03-06T15:03:12.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:12.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:12.800790+0000) 2024-03-06T15:03:13.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:13.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:13.801039+0000) 2024-03-06T15:03:14.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:14.762 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:14.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:14.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:14.801228+0000) 2024-03-06T15:03:15.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:15.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:15.801458+0000) 2024-03-06T15:03:16.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:16.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:16.801642+0000) 2024-03-06T15:03:17.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:17.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:17.801883+0000) 2024-03-06T15:03:18.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:18.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:18.802083+0000) 2024-03-06T15:03:19.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:19.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:19.802280+0000) 2024-03-06T15:03:20.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:20.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:20.802514+0000) 2024-03-06T15:03:21.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:21.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:21.802826+0000) 2024-03-06T15:03:22.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:22.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:22.803014+0000) 2024-03-06T15:03:23.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:23.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:23.803199+0000) 2024-03-06T15:03:24.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:24.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:24.803378+0000) 2024-03-06T15:03:25.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:25.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:25.803571+0000) 2024-03-06T15:03:26.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:26.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:26.803801+0000) 2024-03-06T15:03:27.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:27.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:27.804014+0000) 2024-03-06T15:03:28.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:28.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:28.804255+0000) 2024-03-06T15:03:29.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:29.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:29.804500+0000) 2024-03-06T15:03:30.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:30.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:30.804737+0000) 2024-03-06T15:03:31.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:31.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:31.804900+0000) 2024-03-06T15:03:32.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:32.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:32.805123+0000) 2024-03-06T15:03:33.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:33.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:33.805358+0000) 2024-03-06T15:03:34.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:34.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:34.805585+0000) 2024-03-06T15:03:35.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:35.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:35.805819+0000) 2024-03-06T15:03:36.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:36.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:36.806019+0000) 2024-03-06T15:03:37.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:37.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:37.806243+0000) 2024-03-06T15:03:38.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:38.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:38.806460+0000) 2024-03-06T15:03:39.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:39.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:39.806652+0000) 2024-03-06T15:03:40.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:40.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:40.806935+0000) 2024-03-06T15:03:41.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:41.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:41.807177+0000) 2024-03-06T15:03:42.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:42.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:42.807399+0000) 2024-03-06T15:03:43.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:43.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:43.807650+0000) 2024-03-06T15:03:44.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:44.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:44.807946+0000) 2024-03-06T15:03:44.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:44.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:45.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:45.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:45.808201+0000) 2024-03-06T15:03:46.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:46.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:46.808462+0000) 2024-03-06T15:03:47.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:47.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:47.808760+0000) 2024-03-06T15:03:48.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:48.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:48.808960+0000) 2024-03-06T15:03:49.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:49.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:49.809174+0000) 2024-03-06T15:03:50.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:50.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:50.809417+0000) 2024-03-06T15:03:51.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:51.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:51.809654+0000) 2024-03-06T15:03:52.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:52.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:52.809921+0000) 2024-03-06T15:03:53.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:53.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:53.810168+0000) 2024-03-06T15:03:54.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:54.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:54.810364+0000) 2024-03-06T15:03:55.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:55.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:55.810582+0000) 2024-03-06T15:03:56.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:56.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:56.810797+0000) 2024-03-06T15:03:57.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:57.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:57.810992+0000) 2024-03-06T15:03:58.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:58.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:58.811207+0000) 2024-03-06T15:03:59.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:03:59.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:59.811461+0000) 2024-03-06T15:04:00.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:00.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:00.811645+0000) 2024-03-06T15:04:01.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:01.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:01.811878+0000) 2024-03-06T15:04:02.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:02.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:02.812075+0000) 2024-03-06T15:04:03.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:03.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:03.812303+0000) 2024-03-06T15:04:04.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:04.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:04.812540+0000) 2024-03-06T15:04:05.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:05.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:05.812786+0000) 2024-03-06T15:04:06.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:06.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:06.813021+0000) 2024-03-06T15:04:07.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:07.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:07.813247+0000) 2024-03-06T15:04:08.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:08.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:08.813548+0000) 2024-03-06T15:04:09.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:09.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:09.813805+0000) 2024-03-06T15:04:10.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:10.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:10.813995+0000) 2024-03-06T15:04:11.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:11.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:11.814227+0000) 2024-03-06T15:04:12.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:12.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:12.814450+0000) 2024-03-06T15:04:13.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:13.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:13.814720+0000) 2024-03-06T15:04:14.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:14.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:14.814948+0000) 2024-03-06T15:04:15.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:15.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:15.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:15.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:15.815201+0000) 2024-03-06T15:04:16.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:16.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:16.815421+0000) 2024-03-06T15:04:17.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:17.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:17.815625+0000) 2024-03-06T15:04:18.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:18.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:18.815845+0000) 2024-03-06T15:04:19.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:19.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:19.816107+0000) 2024-03-06T15:04:20.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:20.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:20.816299+0000) 2024-03-06T15:04:21.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:21.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:21.816582+0000) 2024-03-06T15:04:22.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:22.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:22.816795+0000) 2024-03-06T15:04:23.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:23.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:23.817039+0000) 2024-03-06T15:04:24.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:24.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:24.817227+0000) 2024-03-06T15:04:25.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:25.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:25.817478+0000) 2024-03-06T15:04:26.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:26.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:26.817708+0000) 2024-03-06T15:04:27.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:27.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:27.817947+0000) 2024-03-06T15:04:28.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:28.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:28.818212+0000) 2024-03-06T15:04:29.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:29.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:29.818482+0000) 2024-03-06T15:04:30.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:30.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:30.818788+0000) 2024-03-06T15:04:31.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:31.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:31.819018+0000) 2024-03-06T15:04:32.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:32.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:32.819260+0000) 2024-03-06T15:04:33.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:33.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:33.819511+0000) 2024-03-06T15:04:34.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:34.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:34.819795+0000) 2024-03-06T15:04:35.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:35.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:35.819992+0000) 2024-03-06T15:04:36.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:36.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:36.820226+0000) 2024-03-06T15:04:37.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:37.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:37.820446+0000) 2024-03-06T15:04:38.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:38.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:38.820768+0000) 2024-03-06T15:04:39.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:39.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:39.821068+0000) 2024-03-06T15:04:40.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:40.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:40.821284+0000) 2024-03-06T15:04:41.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:41.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:41.821570+0000) 2024-03-06T15:04:42.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:42.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:42.821864+0000) 2024-03-06T15:04:43.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:43.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:43.822157+0000) 2024-03-06T15:04:44.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:44.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:44.822434+0000) 2024-03-06T15:04:45.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:45.381 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:45.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:45.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:45.822767+0000) 2024-03-06T15:04:46.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:46.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:46.823000+0000) 2024-03-06T15:04:47.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:47.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:47.823281+0000) 2024-03-06T15:04:48.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:48.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:48.823555+0000) 2024-03-06T15:04:49.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:49.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:49.823859+0000) 2024-03-06T15:04:50.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:50.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:50.824132+0000) 2024-03-06T15:04:51.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:51.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:51.824421+0000) 2024-03-06T15:04:52.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:52.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:52.824623+0000) 2024-03-06T15:04:53.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:53.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:53.824856+0000) 2024-03-06T15:04:54.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:54.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:54.825093+0000) 2024-03-06T15:04:55.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:55.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:55.825364+0000) 2024-03-06T15:04:56.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:56.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:56.825754+0000) 2024-03-06T15:04:57.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:57.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:57.826035+0000) 2024-03-06T15:04:58.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:58.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:58.826326+0000) 2024-03-06T15:04:59.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:04:59.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:59.826627+0000) 2024-03-06T15:05:00.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:00.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:00.826885+0000) 2024-03-06T15:05:01.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:01.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:01.827160+0000) 2024-03-06T15:05:02.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:02.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:02.827442+0000) 2024-03-06T15:05:03.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:03.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:03.827786+0000) 2024-03-06T15:05:04.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:04.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:04.828006+0000) 2024-03-06T15:05:05.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:05.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:05.828294+0000) 2024-03-06T15:05:06.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:06.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:06.828569+0000) 2024-03-06T15:05:07.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:07.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:07.828861+0000) 2024-03-06T15:05:08.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:08.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:08.829098+0000) 2024-03-06T15:05:09.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:09.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:09.829370+0000) 2024-03-06T15:05:10.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:10.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:10.829609+0000) 2024-03-06T15:05:11.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:11.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:11.829852+0000) 2024-03-06T15:05:12.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:12.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:12.830055+0000) 2024-03-06T15:05:13.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:13.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:13.830310+0000) 2024-03-06T15:05:14.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:14.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:14.830532+0000) 2024-03-06T15:05:15.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:15.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:15.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:15.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:15.830808+0000) 2024-03-06T15:05:16.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:16.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:16.831057+0000) 2024-03-06T15:05:17.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:17.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:17.831274+0000) 2024-03-06T15:05:18.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:18.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:18.831506+0000) 2024-03-06T15:05:19.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:19.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:19.831782+0000) 2024-03-06T15:05:20.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:20.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:20.831982+0000) 2024-03-06T15:05:21.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:21.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:21.832220+0000) 2024-03-06T15:05:22.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:22.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:22.832451+0000) 2024-03-06T15:05:23.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:23.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:23.832697+0000) 2024-03-06T15:05:24.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:24.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:24.832929+0000) 2024-03-06T15:05:25.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:25.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:25.833115+0000) 2024-03-06T15:05:26.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:26.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:26.833337+0000) 2024-03-06T15:05:27.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:27.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:27.833557+0000) 2024-03-06T15:05:28.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:28.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:28.833768+0000) 2024-03-06T15:05:29.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:29.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:29.834009+0000) 2024-03-06T15:05:30.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:30.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:30.834215+0000) 2024-03-06T15:05:31.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:31.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:31.834430+0000) 2024-03-06T15:05:32.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:32.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:32.834605+0000) 2024-03-06T15:05:33.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:33.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:33.834831+0000) 2024-03-06T15:05:34.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:34.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:34.835067+0000) 2024-03-06T15:05:35.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:35.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:35.835294+0000) 2024-03-06T15:05:36.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:36.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:36.835473+0000) 2024-03-06T15:05:37.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:37.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:37.835653+0000) 2024-03-06T15:05:38.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:38.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:38.835927+0000) 2024-03-06T15:05:39.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:39.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:39.836152+0000) 2024-03-06T15:05:40.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:40.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:40.836369+0000) 2024-03-06T15:05:41.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:41.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:41.836600+0000) 2024-03-06T15:05:42.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:42.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:42.836852+0000) 2024-03-06T15:05:43.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:43.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:43.837075+0000) 2024-03-06T15:05:44.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:44.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:44.837335+0000) 2024-03-06T15:05:45.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:45.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:45.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:45.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:45.837512+0000) 2024-03-06T15:05:46.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:46.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:46.837777+0000) 2024-03-06T15:05:47.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:47.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:47.838012+0000) 2024-03-06T15:05:48.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:48.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:48.838193+0000) 2024-03-06T15:05:49.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:49.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:49.838411+0000) 2024-03-06T15:05:50.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:50.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:50.838641+0000) 2024-03-06T15:05:51.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:51.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:51.838898+0000) 2024-03-06T15:05:52.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:52.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:52.839117+0000) 2024-03-06T15:05:53.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:53.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:53.839350+0000) 2024-03-06T15:05:54.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:54.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:54.839558+0000) 2024-03-06T15:05:55.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:55.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:55.839774+0000) 2024-03-06T15:05:56.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:56.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:56.840003+0000) 2024-03-06T15:05:57.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:57.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:57.840184+0000) 2024-03-06T15:05:58.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:58.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:58.840370+0000) 2024-03-06T15:05:59.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:05:59.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:59.840557+0000) 2024-03-06T15:06:00.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:00.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:00.840793+0000) 2024-03-06T15:06:01.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:01.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:01.841032+0000) 2024-03-06T15:06:02.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:02.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:02.841257+0000) 2024-03-06T15:06:03.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:03.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:03.841442+0000) 2024-03-06T15:06:04.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:04.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:04.841645+0000) 2024-03-06T15:06:05.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:05.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:05.841919+0000) 2024-03-06T15:06:06.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:06.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:06.842086+0000) 2024-03-06T15:06:07.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:07.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:07.842313+0000) 2024-03-06T15:06:08.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:08.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:08.842495+0000) 2024-03-06T15:06:09.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:09.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:09.842728+0000) 2024-03-06T15:06:10.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:10.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:10.842958+0000) 2024-03-06T15:06:11.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:11.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:11.843183+0000) 2024-03-06T15:06:12.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:12.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:12.843370+0000) 2024-03-06T15:06:13.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:13.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:13.843610+0000) 2024-03-06T15:06:14.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:14.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:14.843839+0000) 2024-03-06T15:06:15.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:15.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:15.844041+0000) 2024-03-06T15:06:15.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:15.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:16.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:16.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:16.844288+0000) 2024-03-06T15:06:17.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:17.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:17.844516+0000) 2024-03-06T15:06:18.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:18.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:18.844800+0000) 2024-03-06T15:06:19.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:19.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:19.844990+0000) 2024-03-06T15:06:20.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:20.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:20.845165+0000) 2024-03-06T15:06:21.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:21.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:21.845399+0000) 2024-03-06T15:06:22.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:22.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:22.845613+0000) 2024-03-06T15:06:23.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:23.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:23.845814+0000) 2024-03-06T15:06:24.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:24.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:24.846013+0000) 2024-03-06T15:06:25.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:25.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:25.846236+0000) 2024-03-06T15:06:26.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:26.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:26.846463+0000) 2024-03-06T15:06:27.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:27.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:27.846636+0000) 2024-03-06T15:06:28.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:28.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:28.846861+0000) 2024-03-06T15:06:29.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:29.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:29.847099+0000) 2024-03-06T15:06:30.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:30.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:30.847282+0000) 2024-03-06T15:06:31.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:31.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:31.847516+0000) 2024-03-06T15:06:32.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:32.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:32.847807+0000) 2024-03-06T15:06:33.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:33.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:33.848044+0000) 2024-03-06T15:06:34.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:34.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:34.848274+0000) 2024-03-06T15:06:35.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:35.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:35.848494+0000) 2024-03-06T15:06:36.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:36.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:36.848747+0000) 2024-03-06T15:06:37.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:37.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:37.849041+0000) 2024-03-06T15:06:38.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:38.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:38.849275+0000) 2024-03-06T15:06:39.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:39.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:39.849510+0000) 2024-03-06T15:06:40.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:40.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:40.849747+0000) 2024-03-06T15:06:41.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:41.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:41.849964+0000) 2024-03-06T15:06:42.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:42.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:42.850246+0000) 2024-03-06T15:06:43.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:43.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:43.850512+0000) 2024-03-06T15:06:44.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:44.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:44.850823+0000) 2024-03-06T15:06:45.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:45.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:45.851016+0000) 2024-03-06T15:06:46.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:46.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:46.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:46.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:46.851243+0000) 2024-03-06T15:06:47.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:47.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:47.851466+0000) 2024-03-06T15:06:48.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:48.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:48.851729+0000) 2024-03-06T15:06:49.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:49.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:49.851961+0000) 2024-03-06T15:06:50.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:50.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:50.852150+0000) 2024-03-06T15:06:51.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:51.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:51.852380+0000) 2024-03-06T15:06:52.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:52.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:52.852598+0000) 2024-03-06T15:06:53.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:53.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:53.852845+0000) 2024-03-06T15:06:54.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:54.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:54.853072+0000) 2024-03-06T15:06:55.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:55.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:55.853252+0000) 2024-03-06T15:06:56.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:56.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:56.853469+0000) 2024-03-06T15:06:57.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:57.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:57.853731+0000) 2024-03-06T15:06:58.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:58.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:58.853960+0000) 2024-03-06T15:06:59.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:06:59.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:59.854196+0000) 2024-03-06T15:07:00.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:00.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:00.854418+0000) 2024-03-06T15:07:01.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:01.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:01.854594+0000) 2024-03-06T15:07:02.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:02.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:02.854820+0000) 2024-03-06T15:07:03.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:03.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:03.855042+0000) 2024-03-06T15:07:04.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:04.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:04.855239+0000) 2024-03-06T15:07:05.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:05.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:05.855464+0000) 2024-03-06T15:07:06.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:06.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:06.855751+0000) 2024-03-06T15:07:07.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:07.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:07.855983+0000) 2024-03-06T15:07:08.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:08.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:08.856231+0000) 2024-03-06T15:07:09.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:09.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:09.856511+0000) 2024-03-06T15:07:10.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:10.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:10.856796+0000) 2024-03-06T15:07:11.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:11.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:11.857066+0000) 2024-03-06T15:07:12.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:12.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:12.857323+0000) 2024-03-06T15:07:13.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:13.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:13.857526+0000) 2024-03-06T15:07:14.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:14.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:14.857801+0000) 2024-03-06T15:07:15.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:15.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:15.858012+0000) 2024-03-06T15:07:16.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:16.441 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:16.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:16.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:16.858158+0000) 2024-03-06T15:07:17.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:17.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:17.858337+0000) 2024-03-06T15:07:18.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:18.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:18.858579+0000) 2024-03-06T15:07:19.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:19.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:19.858852+0000) 2024-03-06T15:07:20.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:20.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:20.859082+0000) 2024-03-06T15:07:21.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:21.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:21.859320+0000) 2024-03-06T15:07:22.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:22.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:22.859554+0000) 2024-03-06T15:07:23.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:23.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:23.859758+0000) 2024-03-06T15:07:24.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:24.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:24.859950+0000) 2024-03-06T15:07:25.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:25.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:25.860240+0000) 2024-03-06T15:07:26.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:26.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:26.860471+0000) 2024-03-06T15:07:27.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:27.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:27.860748+0000) 2024-03-06T15:07:28.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:28.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:28.860954+0000) 2024-03-06T15:07:29.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:29.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:29.861195+0000) 2024-03-06T15:07:30.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:30.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:30.861426+0000) 2024-03-06T15:07:31.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:31.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:31.861584+0000) 2024-03-06T15:07:32.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:32.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:32.861787+0000) 2024-03-06T15:07:33.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:33.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:33.862048+0000) 2024-03-06T15:07:34.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:34.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:34.862259+0000) 2024-03-06T15:07:35.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:35.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:35.862534+0000) 2024-03-06T15:07:36.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:36.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:36.862769+0000) 2024-03-06T15:07:37.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:37.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:37.863125+0000) 2024-03-06T15:07:38.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:38.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:38.863376+0000) 2024-03-06T15:07:39.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:39.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:39.863613+0000) 2024-03-06T15:07:40.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:40.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:40.863835+0000) 2024-03-06T15:07:41.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:41.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:41.864056+0000) 2024-03-06T15:07:42.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:42.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:42.864287+0000) 2024-03-06T15:07:43.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:43.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:43.864508+0000) 2024-03-06T15:07:44.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:44.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:44.864787+0000) 2024-03-06T15:07:45.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:45.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:45.865009+0000) 2024-03-06T15:07:46.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:46.650 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:46.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:46.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:46.865261+0000) 2024-03-06T15:07:47.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:47.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:47.865493+0000) 2024-03-06T15:07:48.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:48.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:48.865726+0000) 2024-03-06T15:07:49.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:49.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:49.865970+0000) 2024-03-06T15:07:50.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:50.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:50.866189+0000) 2024-03-06T15:07:51.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:51.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:51.866421+0000) 2024-03-06T15:07:52.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:52.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:52.866648+0000) 2024-03-06T15:07:53.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:53.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:53.866946+0000) 2024-03-06T15:07:54.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:54.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:54.867195+0000) 2024-03-06T15:07:55.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:55.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:55.867414+0000) 2024-03-06T15:07:56.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:56.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:56.867630+0000) 2024-03-06T15:07:57.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:57.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:57.867831+0000) 2024-03-06T15:07:58.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:58.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:58.868029+0000) 2024-03-06T15:07:59.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:07:59.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:59.868262+0000) 2024-03-06T15:08:00.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:00.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:00.868483+0000) 2024-03-06T15:08:01.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:01.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:01.868743+0000) 2024-03-06T15:08:02.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:02.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:02.868985+0000) 2024-03-06T15:08:03.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:03.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:03.869217+0000) 2024-03-06T15:08:04.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:04.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:04.869456+0000) 2024-03-06T15:08:05.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:05.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:05.869719+0000) 2024-03-06T15:08:06.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:06.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:06.869898+0000) 2024-03-06T15:08:07.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:07.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:07.870120+0000) 2024-03-06T15:08:08.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:08.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:08.870356+0000) 2024-03-06T15:08:09.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:09.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:09.870606+0000) 2024-03-06T15:08:10.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:10.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:10.870786+0000) 2024-03-06T15:08:11.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:11.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:11.870984+0000) 2024-03-06T15:08:12.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:12.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:12.871239+0000) 2024-03-06T15:08:13.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:13.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:13.871548+0000) 2024-03-06T15:08:14.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:14.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:14.871831+0000) 2024-03-06T15:08:15.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:15.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:15.872106+0000) 2024-03-06T15:08:16.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:16.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:16.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:16.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:16.872294+0000) 2024-03-06T15:08:17.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:17.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:17.872477+0000) 2024-03-06T15:08:18.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:18.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:18.872757+0000) 2024-03-06T15:08:19.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:19.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:19.873015+0000) 2024-03-06T15:08:20.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:20.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:20.873232+0000) 2024-03-06T15:08:21.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:21.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:21.873455+0000) 2024-03-06T15:08:22.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:22.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:22.873743+0000) 2024-03-06T15:08:23.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:23.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:23.873987+0000) 2024-03-06T15:08:24.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:24.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:24.874171+0000) 2024-03-06T15:08:25.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:25.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:25.874348+0000) 2024-03-06T15:08:26.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:26.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:26.874539+0000) 2024-03-06T15:08:27.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:27.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:27.874809+0000) 2024-03-06T15:08:28.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:28.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:28.875035+0000) 2024-03-06T15:08:29.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:29.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:29.875275+0000) 2024-03-06T15:08:30.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:30.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:30.875524+0000) 2024-03-06T15:08:31.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:31.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:31.875819+0000) 2024-03-06T15:08:32.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:32.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:32.876081+0000) 2024-03-06T15:08:33.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:33.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:33.876365+0000) 2024-03-06T15:08:34.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:34.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:34.876652+0000) 2024-03-06T15:08:35.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:35.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:35.876897+0000) 2024-03-06T15:08:36.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:36.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:36.877086+0000) 2024-03-06T15:08:37.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:37.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:37.877315+0000) 2024-03-06T15:08:38.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:38.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:38.877498+0000) 2024-03-06T15:08:39.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:39.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:39.877757+0000) 2024-03-06T15:08:40.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:40.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:40.877970+0000) 2024-03-06T15:08:41.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:41.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:41.878244+0000) 2024-03-06T15:08:42.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:42.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:42.878524+0000) 2024-03-06T15:08:43.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:43.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:43.878757+0000) 2024-03-06T15:08:44.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:44.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:44.879008+0000) 2024-03-06T15:08:45.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:45.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:45.879214+0000) 2024-03-06T15:08:46.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:46.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:46.879447+0000) 2024-03-06T15:08:47.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:47.009 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:47.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:47.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:47.879721+0000) 2024-03-06T15:08:48.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:48.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:48.879943+0000) 2024-03-06T15:08:49.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:49.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:49.880165+0000) 2024-03-06T15:08:50.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:50.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:50.880406+0000) 2024-03-06T15:08:51.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:51.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:51.880642+0000) 2024-03-06T15:08:52.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:52.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:52.880929+0000) 2024-03-06T15:08:53.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:53.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:53.881163+0000) 2024-03-06T15:08:54.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:54.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:54.881388+0000) 2024-03-06T15:08:55.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:55.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:55.881608+0000) 2024-03-06T15:08:56.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:56.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:56.881825+0000) 2024-03-06T15:08:57.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:57.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:57.882046+0000) 2024-03-06T15:08:58.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:58.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:58.882274+0000) 2024-03-06T15:08:59.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:08:59.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:59.882540+0000) 2024-03-06T15:09:00.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:00.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:00.882777+0000) 2024-03-06T15:09:01.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:01.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:01.883007+0000) 2024-03-06T15:09:02.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:02.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:02.883225+0000) 2024-03-06T15:09:03.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:03.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:03.883437+0000) 2024-03-06T15:09:04.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:04.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:04.883780+0000) 2024-03-06T15:09:05.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:05.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:05.884034+0000) 2024-03-06T15:09:06.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:06.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:06.884278+0000) 2024-03-06T15:09:07.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:07.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:07.884500+0000) 2024-03-06T15:09:08.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:08.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:08.884765+0000) 2024-03-06T15:09:09.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:09.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:09.884993+0000) 2024-03-06T15:09:10.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:10.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:10.885214+0000) 2024-03-06T15:09:11.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:11.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:11.885432+0000) 2024-03-06T15:09:12.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:12.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:12.885708+0000) 2024-03-06T15:09:13.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:13.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:13.885954+0000) 2024-03-06T15:09:14.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:14.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:14.886227+0000) 2024-03-06T15:09:15.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:15.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:15.886431+0000) 2024-03-06T15:09:16.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:16.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:16.886592+0000) 2024-03-06T15:09:17.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:17.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:17.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:17.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:17.886827+0000) 2024-03-06T15:09:18.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:18.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:18.887078+0000) 2024-03-06T15:09:19.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:19.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:19.887370+0000) 2024-03-06T15:09:20.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:20.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:20.887629+0000) 2024-03-06T15:09:21.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:21.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:21.887955+0000) 2024-03-06T15:09:22.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:22.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:22.888235+0000) 2024-03-06T15:09:23.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:23.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:23.888520+0000) 2024-03-06T15:09:24.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:24.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:24.888831+0000) 2024-03-06T15:09:25.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:25.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:25.889119+0000) 2024-03-06T15:09:26.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:26.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:26.889375+0000) 2024-03-06T15:09:27.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:27.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:27.889632+0000) 2024-03-06T15:09:28.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:28.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:28.889941+0000) 2024-03-06T15:09:29.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:29.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:29.890227+0000) 2024-03-06T15:09:30.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:30.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:30.890434+0000) 2024-03-06T15:09:31.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:31.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:31.890698+0000) 2024-03-06T15:09:32.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:32.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:32.890927+0000) 2024-03-06T15:09:33.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:33.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:33.891194+0000) 2024-03-06T15:09:34.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:34.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:34.891478+0000) 2024-03-06T15:09:35.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:35.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:35.891750+0000) 2024-03-06T15:09:36.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:36.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:36.891957+0000) 2024-03-06T15:09:37.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:37.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:37.892187+0000) 2024-03-06T15:09:38.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:38.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:38.892397+0000) 2024-03-06T15:09:39.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:39.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:39.892647+0000) 2024-03-06T15:09:40.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:40.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:40.892893+0000) 2024-03-06T15:09:41.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:41.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:41.893118+0000) 2024-03-06T15:09:42.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:42.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:42.893318+0000) 2024-03-06T15:09:43.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:43.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:43.893554+0000) 2024-03-06T15:09:44.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:44.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:44.893797+0000) 2024-03-06T15:09:45.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:45.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:45.894025+0000) 2024-03-06T15:09:46.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:46.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:46.894259+0000) 2024-03-06T15:09:47.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:47.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:47.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:47.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:47.894503+0000) 2024-03-06T15:09:48.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:48.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:48.894721+0000) 2024-03-06T15:09:49.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:49.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:49.894919+0000) 2024-03-06T15:09:50.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:50.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:50.895132+0000) 2024-03-06T15:09:51.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:51.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:51.895357+0000) 2024-03-06T15:09:52.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:52.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:52.895580+0000) 2024-03-06T15:09:53.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:53.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:53.895780+0000) 2024-03-06T15:09:54.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:54.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:54.896091+0000) 2024-03-06T15:09:55.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:55.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:55.896370+0000) 2024-03-06T15:09:56.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:56.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:56.896639+0000) 2024-03-06T15:09:57.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:57.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:57.896887+0000) 2024-03-06T15:09:58.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:58.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:58.897106+0000) 2024-03-06T15:09:59.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:09:59.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:59.897302+0000) 2024-03-06T15:10:00.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:00.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:00.897542+0000) 2024-03-06T15:10:01.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:01.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:01.897767+0000) 2024-03-06T15:10:02.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:02.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:02.898008+0000) 2024-03-06T15:10:03.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:03.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:03.898240+0000) 2024-03-06T15:10:04.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:04.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:04.898477+0000) 2024-03-06T15:10:05.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:05.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:05.898753+0000) 2024-03-06T15:10:06.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:06.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:06.898997+0000) 2024-03-06T15:10:07.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:07.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:07.899223+0000) 2024-03-06T15:10:08.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:08.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:08.899418+0000) 2024-03-06T15:10:09.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:09.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:09.899696+0000) 2024-03-06T15:10:10.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:10.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:10.899934+0000) 2024-03-06T15:10:11.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:11.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:11.900117+0000) 2024-03-06T15:10:12.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:12.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:12.900349+0000) 2024-03-06T15:10:13.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:13.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:13.900562+0000) 2024-03-06T15:10:14.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:14.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:14.900824+0000) 2024-03-06T15:10:15.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:15.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:15.901007+0000) 2024-03-06T15:10:16.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:16.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:16.901212+0000) 2024-03-06T15:10:17.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:17.656 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:17.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:17.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:17.901425+0000) 2024-03-06T15:10:18.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:18.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:18.901612+0000) 2024-03-06T15:10:19.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:19.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:19.901894+0000) 2024-03-06T15:10:20.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:20.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:20.902135+0000) 2024-03-06T15:10:21.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:21.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:21.902374+0000) 2024-03-06T15:10:22.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:22.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:22.902563+0000) 2024-03-06T15:10:23.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:23.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:23.902803+0000) 2024-03-06T15:10:24.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:24.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:24.903029+0000) 2024-03-06T15:10:25.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:25.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:25.903240+0000) 2024-03-06T15:10:26.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:26.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:26.903483+0000) 2024-03-06T15:10:27.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:27.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:27.903762+0000) 2024-03-06T15:10:28.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:28.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:28.903997+0000) 2024-03-06T15:10:29.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:29.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:29.904195+0000) 2024-03-06T15:10:30.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:30.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:30.904420+0000) 2024-03-06T15:10:31.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:31.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:31.904651+0000) 2024-03-06T15:10:32.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:32.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:32.904876+0000) 2024-03-06T15:10:33.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:33.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:33.905120+0000) 2024-03-06T15:10:34.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:34.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:34.905364+0000) 2024-03-06T15:10:35.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:35.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:35.905546+0000) 2024-03-06T15:10:36.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:36.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:36.905817+0000) 2024-03-06T15:10:37.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:37.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:37.906087+0000) 2024-03-06T15:10:38.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:38.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:38.906372+0000) 2024-03-06T15:10:39.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:39.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:39.906583+0000) 2024-03-06T15:10:40.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:40.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:40.906827+0000) 2024-03-06T15:10:41.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:41.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:41.907014+0000) 2024-03-06T15:10:42.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:42.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:42.907295+0000) 2024-03-06T15:10:43.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:43.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:43.907561+0000) 2024-03-06T15:10:44.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:44.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:44.907845+0000) 2024-03-06T15:10:45.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:45.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:45.908089+0000) 2024-03-06T15:10:46.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:46.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:46.908371+0000) 2024-03-06T15:10:47.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:47.851 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:47.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:47.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:47.908538+0000) 2024-03-06T15:10:48.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:48.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:48.908800+0000) 2024-03-06T15:10:49.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:49.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:49.909041+0000) 2024-03-06T15:10:50.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:50.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:50.909280+0000) 2024-03-06T15:10:51.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:51.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:51.909508+0000) 2024-03-06T15:10:52.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:52.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:52.909777+0000) 2024-03-06T15:10:53.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:53.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:53.909990+0000) 2024-03-06T15:10:54.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:54.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:54.910190+0000) 2024-03-06T15:10:55.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:55.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:55.910462+0000) 2024-03-06T15:10:56.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:56.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:56.910698+0000) 2024-03-06T15:10:57.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:57.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:57.910925+0000) 2024-03-06T15:10:58.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:58.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:58.911160+0000) 2024-03-06T15:10:59.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:10:59.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:59.911398+0000) 2024-03-06T15:11:00.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:00.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:00.911592+0000) 2024-03-06T15:11:01.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:01.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:01.911836+0000) 2024-03-06T15:11:02.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:02.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:02.912015+0000) 2024-03-06T15:11:03.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:03.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:03.912227+0000) 2024-03-06T15:11:04.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:04.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:04.912494+0000) 2024-03-06T15:11:05.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:05.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:05.912716+0000) 2024-03-06T15:11:06.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:06.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:06.912891+0000) 2024-03-06T15:11:07.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:07.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:07.913073+0000) 2024-03-06T15:11:08.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:08.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:08.913287+0000) 2024-03-06T15:11:09.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:09.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:09.913564+0000) 2024-03-06T15:11:10.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:10.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:10.913761+0000) 2024-03-06T15:11:11.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:11.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:11.914014+0000) 2024-03-06T15:11:12.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:12.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:12.914290+0000) 2024-03-06T15:11:13.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:13.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:13.914454+0000) 2024-03-06T15:11:14.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:14.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:14.914719+0000) 2024-03-06T15:11:15.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:15.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:15.914915+0000) 2024-03-06T15:11:16.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:16.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:16.915147+0000) 2024-03-06T15:11:17.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:17.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:17.915345+0000) 2024-03-06T15:11:17.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:17.982 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:18.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:18.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:18.915539+0000) 2024-03-06T15:11:19.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:19.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:19.915804+0000) 2024-03-06T15:11:20.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:20.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:20.916046+0000) 2024-03-06T15:11:21.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:21.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:21.916323+0000) 2024-03-06T15:11:22.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:22.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:22.916599+0000) 2024-03-06T15:11:23.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:23.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:23.916875+0000) 2024-03-06T15:11:24.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:24.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:24.917082+0000) 2024-03-06T15:11:25.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:25.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:25.917311+0000) 2024-03-06T15:11:26.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:26.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:26.917539+0000) 2024-03-06T15:11:27.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:27.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:27.917769+0000) 2024-03-06T15:11:28.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:28.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:28.917954+0000) 2024-03-06T15:11:29.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:29.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:29.918126+0000) 2024-03-06T15:11:30.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:30.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:30.918346+0000) 2024-03-06T15:11:31.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:31.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:31.918560+0000) 2024-03-06T15:11:32.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:32.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:32.918768+0000) 2024-03-06T15:11:33.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:33.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:33.919003+0000) 2024-03-06T15:11:34.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:34.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:34.919231+0000) 2024-03-06T15:11:35.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:35.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:35.919452+0000) 2024-03-06T15:11:36.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:36.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:36.919765+0000) 2024-03-06T15:11:37.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:37.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:37.919998+0000) 2024-03-06T15:11:38.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:38.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:38.920238+0000) 2024-03-06T15:11:39.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:39.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:39.920469+0000) 2024-03-06T15:11:40.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:40.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:40.920652+0000) 2024-03-06T15:11:41.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:41.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:41.920905+0000) 2024-03-06T15:11:42.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:42.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:42.921182+0000) 2024-03-06T15:11:43.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:43.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:43.921446+0000) 2024-03-06T15:11:44.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:44.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:44.921768+0000) 2024-03-06T15:11:45.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:45.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:45.922046+0000) 2024-03-06T15:11:46.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:46.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:46.922212+0000) 2024-03-06T15:11:47.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:47.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:47.922405+0000) 2024-03-06T15:11:48.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:48.185 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:48.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:48.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:48.922596+0000) 2024-03-06T15:11:49.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:49.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:49.922836+0000) 2024-03-06T15:11:50.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:50.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:50.922992+0000) 2024-03-06T15:11:51.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:51.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:51.923204+0000) 2024-03-06T15:11:52.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:52.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:52.923431+0000) 2024-03-06T15:11:53.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:53.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:53.923652+0000) 2024-03-06T15:11:54.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:54.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:54.923914+0000) 2024-03-06T15:11:55.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:55.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:55.924148+0000) 2024-03-06T15:11:56.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:56.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:56.924390+0000) 2024-03-06T15:11:57.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:57.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:57.924578+0000) 2024-03-06T15:11:58.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:58.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:58.924828+0000) 2024-03-06T15:11:59.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:11:59.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:59.925072+0000) 2024-03-06T15:12:00.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:00.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:00.925297+0000) 2024-03-06T15:12:01.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:01.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:01.925530+0000) 2024-03-06T15:12:02.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:02.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:02.925789+0000) 2024-03-06T15:12:03.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:03.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:03.926005+0000) 2024-03-06T15:12:04.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:04.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:04.926226+0000) 2024-03-06T15:12:05.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:05.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:05.926455+0000) 2024-03-06T15:12:06.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:06.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:06.926720+0000) 2024-03-06T15:12:07.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:07.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:07.926933+0000) 2024-03-06T15:12:08.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:08.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:08.927100+0000) 2024-03-06T15:12:09.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:09.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:09.927288+0000) 2024-03-06T15:12:10.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:10.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:10.927531+0000) 2024-03-06T15:12:11.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:11.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:11.927750+0000) 2024-03-06T15:12:12.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:12.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:12.927993+0000) 2024-03-06T15:12:13.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:13.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:13.928201+0000) 2024-03-06T15:12:14.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:14.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:14.928384+0000) 2024-03-06T15:12:15.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:15.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:15.928593+0000) 2024-03-06T15:12:16.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:16.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:16.928854+0000) 2024-03-06T15:12:17.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:17.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:17.929149+0000) 2024-03-06T15:12:18.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:18.407 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:18.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:18.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:18.929368+0000) 2024-03-06T15:12:19.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:19.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:19.929599+0000) 2024-03-06T15:12:20.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:20.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:20.929831+0000) 2024-03-06T15:12:21.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:21.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:21.930039+0000) 2024-03-06T15:12:22.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:22.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:22.930218+0000) 2024-03-06T15:12:23.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:23.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:23.930431+0000) 2024-03-06T15:12:24.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:24.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:24.930707+0000) 2024-03-06T15:12:25.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:25.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:25.930892+0000) 2024-03-06T15:12:26.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:26.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:26.931094+0000) 2024-03-06T15:12:27.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:27.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:27.931330+0000) 2024-03-06T15:12:28.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:28.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:28.931554+0000) 2024-03-06T15:12:29.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:29.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:29.931809+0000) 2024-03-06T15:12:30.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:30.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:30.932054+0000) 2024-03-06T15:12:31.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:31.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:31.932289+0000) 2024-03-06T15:12:32.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:32.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:32.932526+0000) 2024-03-06T15:12:33.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:33.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:33.932784+0000) 2024-03-06T15:12:34.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:34.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:34.933012+0000) 2024-03-06T15:12:35.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:35.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:35.933201+0000) 2024-03-06T15:12:36.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:36.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:36.933429+0000) 2024-03-06T15:12:37.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:37.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:37.933627+0000) 2024-03-06T15:12:38.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:38.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:38.933847+0000) 2024-03-06T15:12:39.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:39.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:39.934055+0000) 2024-03-06T15:12:40.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:40.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:40.934289+0000) 2024-03-06T15:12:41.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:41.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:41.934506+0000) 2024-03-06T15:12:42.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:42.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:42.934803+0000) 2024-03-06T15:12:43.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:43.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:43.935011+0000) 2024-03-06T15:12:44.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:44.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:44.935244+0000) 2024-03-06T15:12:45.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:45.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:45.935435+0000) 2024-03-06T15:12:46.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:46.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:46.935707+0000) 2024-03-06T15:12:47.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:47.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:47.935910+0000) 2024-03-06T15:12:48.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:48.536 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:48.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:48.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:48.936151+0000) 2024-03-06T15:12:49.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:49.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:49.936390+0000) 2024-03-06T15:12:50.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:50.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:50.936645+0000) 2024-03-06T15:12:51.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:51.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:51.936902+0000) 2024-03-06T15:12:52.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:52.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:52.937148+0000) 2024-03-06T15:12:53.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:53.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:53.937356+0000) 2024-03-06T15:12:54.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:54.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:54.937632+0000) 2024-03-06T15:12:55.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:55.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:55.937951+0000) 2024-03-06T15:12:56.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:56.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:56.938222+0000) 2024-03-06T15:12:57.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:57.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:57.938500+0000) 2024-03-06T15:12:58.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:58.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:58.938781+0000) 2024-03-06T15:12:59.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:12:59.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:59.939051+0000) 2024-03-06T15:13:00.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:00.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:00.939491+0000) 2024-03-06T15:13:01.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:01.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:01.939746+0000) 2024-03-06T15:13:02.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:02.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:02.939938+0000) 2024-03-06T15:13:03.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:03.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:03.940122+0000) 2024-03-06T15:13:04.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:04.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:04.940347+0000) 2024-03-06T15:13:05.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:05.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:05.940568+0000) 2024-03-06T15:13:06.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:06.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:06.940827+0000) 2024-03-06T15:13:07.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:07.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:07.941109+0000) 2024-03-06T15:13:08.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:08.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:08.941356+0000) 2024-03-06T15:13:09.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:09.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:09.941647+0000) 2024-03-06T15:13:10.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:10.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:10.941898+0000) 2024-03-06T15:13:11.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:11.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:11.942082+0000) 2024-03-06T15:13:12.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:12.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:12.942316+0000) 2024-03-06T15:13:13.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:13.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:13.942538+0000) 2024-03-06T15:13:14.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:14.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:14.942803+0000) 2024-03-06T15:13:15.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:15.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:15.943027+0000) 2024-03-06T15:13:16.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:16.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:16.943258+0000) 2024-03-06T15:13:17.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:17.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:17.943532+0000) 2024-03-06T15:13:18.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:18.748 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:18.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:18.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:18.943797+0000) 2024-03-06T15:13:19.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:19.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:19.944034+0000) 2024-03-06T15:13:20.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:20.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:20.944262+0000) 2024-03-06T15:13:21.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:21.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:21.944462+0000) 2024-03-06T15:13:22.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:22.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:22.944708+0000) 2024-03-06T15:13:23.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:23.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:23.944949+0000) 2024-03-06T15:13:24.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:24.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:24.945145+0000) 2024-03-06T15:13:25.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:25.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:25.945365+0000) 2024-03-06T15:13:26.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:26.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:26.945590+0000) 2024-03-06T15:13:27.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:27.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:27.945807+0000) 2024-03-06T15:13:28.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:28.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:28.945994+0000) 2024-03-06T15:13:29.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:29.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:29.946251+0000) 2024-03-06T15:13:30.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:30.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:30.946480+0000) 2024-03-06T15:13:31.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:31.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:31.946662+0000) 2024-03-06T15:13:32.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:32.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:32.946892+0000) 2024-03-06T15:13:33.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:33.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:33.947142+0000) 2024-03-06T15:13:34.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:34.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:34.947432+0000) 2024-03-06T15:13:35.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:35.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:35.947707+0000) 2024-03-06T15:13:36.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:36.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:36.947953+0000) 2024-03-06T15:13:37.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:37.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:37.948145+0000) 2024-03-06T15:13:38.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:38.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:38.948365+0000) 2024-03-06T15:13:39.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:39.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:39.948631+0000) 2024-03-06T15:13:40.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:40.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:40.948952+0000) 2024-03-06T15:13:41.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:41.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:41.949168+0000) 2024-03-06T15:13:42.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:42.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:42.949354+0000) 2024-03-06T15:13:43.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:43.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:43.949585+0000) 2024-03-06T15:13:44.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:44.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:44.949848+0000) 2024-03-06T15:13:45.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:45.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:45.950050+0000) 2024-03-06T15:13:46.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:46.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:46.950320+0000) 2024-03-06T15:13:47.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:47.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:47.950585+0000) 2024-03-06T15:13:48.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:48.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:48.950872+0000) 2024-03-06T15:13:48.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:48.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:49.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:49.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:49.951055+0000) 2024-03-06T15:13:50.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:50.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:50.951243+0000) 2024-03-06T15:13:51.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:51.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:51.951478+0000) 2024-03-06T15:13:52.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:52.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:52.951767+0000) 2024-03-06T15:13:53.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:53.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:53.951916+0000) 2024-03-06T15:13:54.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:54.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:54.952123+0000) 2024-03-06T15:13:55.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:55.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:55.952356+0000) 2024-03-06T15:13:56.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:56.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:56.952599+0000) 2024-03-06T15:13:57.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:57.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:57.952854+0000) 2024-03-06T15:13:58.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:58.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:58.953132+0000) 2024-03-06T15:13:59.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:13:59.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:59.953352+0000) 2024-03-06T15:14:00.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:00.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:00.953548+0000) 2024-03-06T15:14:01.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:01.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:01.953779+0000) 2024-03-06T15:14:02.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:02.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:02.954005+0000) 2024-03-06T15:14:03.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:03.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:03.954176+0000) 2024-03-06T15:14:04.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:04.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:04.954367+0000) 2024-03-06T15:14:05.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:05.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:05.954572+0000) 2024-03-06T15:14:06.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:06.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:06.954840+0000) 2024-03-06T15:14:07.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:07.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:07.955024+0000) 2024-03-06T15:14:08.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:08.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:08.955217+0000) 2024-03-06T15:14:09.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:09.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:09.955460+0000) 2024-03-06T15:14:10.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:10.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:10.955741+0000) 2024-03-06T15:14:11.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:11.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:11.955923+0000) 2024-03-06T15:14:12.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:12.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:12.956112+0000) 2024-03-06T15:14:13.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:13.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:13.956295+0000) 2024-03-06T15:14:14.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:14.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:14.956479+0000) 2024-03-06T15:14:15.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:15.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:15.956653+0000) 2024-03-06T15:14:16.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:16.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:16.956906+0000) 2024-03-06T15:14:17.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:17.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:17.957085+0000) 2024-03-06T15:14:18.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:18.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:18.957275+0000) 2024-03-06T15:14:19.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:19.106 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:19.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:19.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:19.957469+0000) 2024-03-06T15:14:20.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:20.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:20.957725+0000) 2024-03-06T15:14:21.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:21.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:21.957896+0000) 2024-03-06T15:14:22.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:22.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:22.958084+0000) 2024-03-06T15:14:23.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:23.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:23.958275+0000) 2024-03-06T15:14:24.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:24.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:24.958497+0000) 2024-03-06T15:14:25.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:25.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:25.958704+0000) 2024-03-06T15:14:26.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:26.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:26.958909+0000) 2024-03-06T15:14:27.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:27.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:27.959097+0000) 2024-03-06T15:14:28.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:28.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:28.959288+0000) 2024-03-06T15:14:29.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:29.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:29.959517+0000) 2024-03-06T15:14:30.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:30.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:30.959761+0000) 2024-03-06T15:14:31.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:31.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:31.959951+0000) 2024-03-06T15:14:32.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:32.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:32.960138+0000) 2024-03-06T15:14:33.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:33.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:33.960356+0000) 2024-03-06T15:14:34.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:34.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:34.960590+0000) 2024-03-06T15:14:35.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:35.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:35.960785+0000) 2024-03-06T15:14:36.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:36.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:36.960999+0000) 2024-03-06T15:14:37.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:37.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:37.961232+0000) 2024-03-06T15:14:38.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:38.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:38.961449+0000) 2024-03-06T15:14:39.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:39.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:39.961719+0000) 2024-03-06T15:14:40.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:40.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:40.962005+0000) 2024-03-06T15:14:41.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:41.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:41.962199+0000) 2024-03-06T15:14:42.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:42.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:42.962422+0000) 2024-03-06T15:14:43.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:43.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:43.962705+0000) 2024-03-06T15:14:44.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:44.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:44.962949+0000) 2024-03-06T15:14:45.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:45.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:45.963197+0000) 2024-03-06T15:14:46.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:46.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:46.963474+0000) 2024-03-06T15:14:47.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:47.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:47.963771+0000) 2024-03-06T15:14:48.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:48.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:48.963973+0000) 2024-03-06T15:14:49.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:49.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:49.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:49.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:49.964247+0000) 2024-03-06T15:14:50.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:50.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:50.964457+0000) 2024-03-06T15:14:51.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:51.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:51.964738+0000) 2024-03-06T15:14:52.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:52.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:52.964950+0000) 2024-03-06T15:14:53.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:53.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:53.965264+0000) 2024-03-06T15:14:54.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:54.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:54.965496+0000) 2024-03-06T15:14:55.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:55.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:55.965803+0000) 2024-03-06T15:14:56.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:56.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:56.966042+0000) 2024-03-06T15:14:57.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:57.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:57.966303+0000) 2024-03-06T15:14:58.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:58.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:58.966553+0000) 2024-03-06T15:14:59.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:14:59.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:59.966831+0000) 2024-03-06T15:15:00.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:00.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:00.967076+0000) 2024-03-06T15:15:01.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:01.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:01.967336+0000) 2024-03-06T15:15:02.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:02.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:02.967513+0000) 2024-03-06T15:15:03.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:03.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:03.967750+0000) 2024-03-06T15:15:04.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:04.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:04.967934+0000) 2024-03-06T15:15:05.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:05.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:05.968134+0000) 2024-03-06T15:15:06.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:06.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:06.968374+0000) 2024-03-06T15:15:07.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:07.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:07.968600+0000) 2024-03-06T15:15:08.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:08.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:08.968814+0000) 2024-03-06T15:15:09.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:09.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:09.969053+0000) 2024-03-06T15:15:10.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:10.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:10.969336+0000) 2024-03-06T15:15:11.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:11.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:11.969600+0000) 2024-03-06T15:15:12.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:12.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:12.969861+0000) 2024-03-06T15:15:13.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:13.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:13.970121+0000) 2024-03-06T15:15:14.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:14.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:14.970391+0000) 2024-03-06T15:15:15.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:15.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:15.970593+0000) 2024-03-06T15:15:16.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:16.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:16.970827+0000) 2024-03-06T15:15:17.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:17.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:17.971010+0000) 2024-03-06T15:15:18.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:18.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:18.971293+0000) 2024-03-06T15:15:19.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:19.527 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:19.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:19.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:19.971587+0000) 2024-03-06T15:15:20.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:20.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:20.971870+0000) 2024-03-06T15:15:21.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:21.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:21.972095+0000) 2024-03-06T15:15:22.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:22.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:22.972260+0000) 2024-03-06T15:15:23.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:23.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:23.972484+0000) 2024-03-06T15:15:24.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:24.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:24.972758+0000) 2024-03-06T15:15:25.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:25.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:25.972965+0000) 2024-03-06T15:15:26.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:26.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:26.973168+0000) 2024-03-06T15:15:27.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:27.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:27.973384+0000) 2024-03-06T15:15:28.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:28.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:28.973603+0000) 2024-03-06T15:15:29.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:29.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:29.973835+0000) 2024-03-06T15:15:30.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:30.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:30.974051+0000) 2024-03-06T15:15:31.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:31.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:31.974281+0000) 2024-03-06T15:15:32.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:32.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:32.974509+0000) 2024-03-06T15:15:33.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:33.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:33.974748+0000) 2024-03-06T15:15:34.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:34.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:34.974970+0000) 2024-03-06T15:15:35.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:35.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:35.975205+0000) 2024-03-06T15:15:36.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:36.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:36.975439+0000) 2024-03-06T15:15:37.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:37.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:37.975697+0000) 2024-03-06T15:15:38.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:38.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:38.975928+0000) 2024-03-06T15:15:39.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:39.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:39.976105+0000) 2024-03-06T15:15:40.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:40.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:40.976328+0000) 2024-03-06T15:15:41.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:41.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:41.976510+0000) 2024-03-06T15:15:42.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:42.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:42.976743+0000) 2024-03-06T15:15:43.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:43.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:43.976940+0000) 2024-03-06T15:15:44.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:44.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:44.977130+0000) 2024-03-06T15:15:45.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:45.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:45.977351+0000) 2024-03-06T15:15:46.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:46.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:46.977581+0000) 2024-03-06T15:15:47.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:47.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:47.977811+0000) 2024-03-06T15:15:48.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:48.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:48.978003+0000) 2024-03-06T15:15:49.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:49.749 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:49.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:49.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:49.978243+0000) 2024-03-06T15:15:50.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:50.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:50.978446+0000) 2024-03-06T15:15:51.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:51.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:51.978706+0000) 2024-03-06T15:15:52.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:52.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:52.978899+0000) 2024-03-06T15:15:53.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:53.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:53.979084+0000) 2024-03-06T15:15:54.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:54.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:54.979281+0000) 2024-03-06T15:15:55.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:55.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:55.979524+0000) 2024-03-06T15:15:56.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:56.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:56.979777+0000) 2024-03-06T15:15:57.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:57.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:57.979952+0000) 2024-03-06T15:15:58.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:58.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:58.980161+0000) 2024-03-06T15:15:59.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:15:59.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:59.980414+0000) 2024-03-06T15:16:00.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:00.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:00.980696+0000) 2024-03-06T15:16:01.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:01.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:01.980944+0000) 2024-03-06T15:16:02.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:02.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:02.981199+0000) 2024-03-06T15:16:03.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:03.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:03.981434+0000) 2024-03-06T15:16:04.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:04.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:04.981705+0000) 2024-03-06T15:16:05.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:05.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:05.981993+0000) 2024-03-06T15:16:06.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:06.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:06.982202+0000) 2024-03-06T15:16:07.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:07.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:07.982429+0000) 2024-03-06T15:16:08.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:08.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:08.982713+0000) 2024-03-06T15:16:09.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:09.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:09.982966+0000) 2024-03-06T15:16:10.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:10.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:10.983182+0000) 2024-03-06T15:16:11.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:11.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:11.983373+0000) 2024-03-06T15:16:12.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:12.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:12.983643+0000) 2024-03-06T15:16:13.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:13.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:13.983938+0000) 2024-03-06T15:16:14.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:14.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:14.984131+0000) 2024-03-06T15:16:15.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:15.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:15.984370+0000) 2024-03-06T15:16:16.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:16.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:16.984615+0000) 2024-03-06T15:16:17.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:17.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:17.984823+0000) 2024-03-06T15:16:18.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:18.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:18.985076+0000) 2024-03-06T15:16:19.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:19.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:19.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:19.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:19.985346+0000) 2024-03-06T15:16:20.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:20.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:20.985600+0000) 2024-03-06T15:16:21.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:21.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:21.985820+0000) 2024-03-06T15:16:22.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:22.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:22.986035+0000) 2024-03-06T15:16:23.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:23.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:23.986226+0000) 2024-03-06T15:16:24.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:24.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:24.986451+0000) 2024-03-06T15:16:25.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:25.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:25.986715+0000) 2024-03-06T15:16:26.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:26.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:26.986915+0000) 2024-03-06T15:16:27.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:27.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:27.987097+0000) 2024-03-06T15:16:28.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:28.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:28.987293+0000) 2024-03-06T15:16:29.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:29.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:29.987533+0000) 2024-03-06T15:16:30.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:30.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:30.987796+0000) 2024-03-06T15:16:31.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:31.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:31.988002+0000) 2024-03-06T15:16:32.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:32.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:32.988192+0000) 2024-03-06T15:16:33.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:33.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:33.988380+0000) 2024-03-06T15:16:34.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:34.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:34.988696+0000) 2024-03-06T15:16:35.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:35.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:35.988890+0000) 2024-03-06T15:16:36.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:36.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:36.989127+0000) 2024-03-06T15:16:37.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:37.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:37.989352+0000) 2024-03-06T15:16:38.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:38.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:38.989562+0000) 2024-03-06T15:16:39.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:39.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:39.989817+0000) 2024-03-06T15:16:40.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:40.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:40.990056+0000) 2024-03-06T15:16:41.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:41.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:41.990271+0000) 2024-03-06T15:16:42.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:42.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:42.990514+0000) 2024-03-06T15:16:43.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:43.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:43.990777+0000) 2024-03-06T15:16:44.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:44.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:44.990980+0000) 2024-03-06T15:16:45.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:45.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:45.991193+0000) 2024-03-06T15:16:46.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:46.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:46.991464+0000) 2024-03-06T15:16:47.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:47.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:47.991704+0000) 2024-03-06T15:16:48.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:48.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:48.991928+0000) 2024-03-06T15:16:49.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:49.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:49.992194+0000) 2024-03-06T15:16:50.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:50.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:50.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:50.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:50.992431+0000) 2024-03-06T15:16:51.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:51.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:51.992654+0000) 2024-03-06T15:16:52.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:52.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:52.992879+0000) 2024-03-06T15:16:53.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:53.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:53.993101+0000) 2024-03-06T15:16:54.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:54.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:54.993282+0000) 2024-03-06T15:16:55.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:55.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:55.993502+0000) 2024-03-06T15:16:56.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:56.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:56.993782+0000) 2024-03-06T15:16:57.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:57.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:57.994002+0000) 2024-03-06T15:16:58.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:58.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:58.994232+0000) 2024-03-06T15:16:59.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:16:59.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:59.994464+0000) 2024-03-06T15:17:00.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:00.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:00.994654+0000) 2024-03-06T15:17:01.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:01.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:01.994919+0000) 2024-03-06T15:17:02.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:02.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:02.995142+0000) 2024-03-06T15:17:03.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:03.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:03.995322+0000) 2024-03-06T15:17:04.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:04.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:04.995545+0000) 2024-03-06T15:17:05.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:05.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:05.995768+0000) 2024-03-06T15:17:07.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:06.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:06.995999+0000) 2024-03-06T15:17:08.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:07.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:07.996181+0000) 2024-03-06T15:17:09.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:08.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:08.996344+0000) 2024-03-06T15:17:10.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:09.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:09.996563+0000) 2024-03-06T15:17:11.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:10.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:10.996785+0000) 2024-03-06T15:17:12.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:11.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:11.997018+0000) 2024-03-06T15:17:13.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:12.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:12.997208+0000) 2024-03-06T15:17:14.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:13.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:13.997391+0000) 2024-03-06T15:17:15.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:14.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:14.997602+0000) 2024-03-06T15:17:16.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:15.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:15.997833+0000) 2024-03-06T15:17:17.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:16.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:16.998064+0000) 2024-03-06T15:17:18.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:17.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:17.998247+0000) 2024-03-06T15:17:19.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:18.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:18.998470+0000) 2024-03-06T15:17:20.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:19.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:19.998728+0000) 2024-03-06T15:17:20.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:20.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:21.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:20.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:20.998965+0000) 2024-03-06T15:17:22.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:21.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:21.999149+0000) 2024-03-06T15:17:23.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:22.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:22.999405+0000) 2024-03-06T15:17:24.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:23.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:23.999576+0000) 2024-03-06T15:17:25.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:24.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:24.999813+0000) 2024-03-06T15:17:26.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:25.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:26.000060+0000) 2024-03-06T15:17:27.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:26.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:27.000304+0000) 2024-03-06T15:17:28.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:27.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:28.000499+0000) 2024-03-06T15:17:29.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:29.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:29.000790+0000) 2024-03-06T15:17:30.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:30.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:30.001041+0000) 2024-03-06T15:17:31.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:30.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:31.001276+0000) 2024-03-06T15:17:32.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:31.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:32.001466+0000) 2024-03-06T15:17:33.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:33.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:33.001762+0000) 2024-03-06T15:17:34.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:34.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:34.001978+0000) 2024-03-06T15:17:35.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:35.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:35.002218+0000) 2024-03-06T15:17:36.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:36.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:36.002420+0000) 2024-03-06T15:17:37.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:37.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:37.002644+0000) 2024-03-06T15:17:38.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:38.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:38.002861+0000) 2024-03-06T15:17:39.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:39.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:39.003095+0000) 2024-03-06T15:17:40.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:40.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:40.003318+0000) 2024-03-06T15:17:41.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:41.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:41.003555+0000) 2024-03-06T15:17:42.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:42.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:42.003791+0000) 2024-03-06T15:17:43.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:43.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:43.004015+0000) 2024-03-06T15:17:44.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:44.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:44.004207+0000) 2024-03-06T15:17:45.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:45.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:45.004404+0000) 2024-03-06T15:17:46.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:46.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:46.004631+0000) 2024-03-06T15:17:47.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:47.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:47.004879+0000) 2024-03-06T15:17:48.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:48.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:48.005086+0000) 2024-03-06T15:17:49.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:49.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:49.005264+0000) 2024-03-06T15:17:50.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:50.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:50.005512+0000) 2024-03-06T15:17:50.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:50.660 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:51.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:51.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:51.005802+0000) 2024-03-06T15:17:52.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:52.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:52.006043+0000) 2024-03-06T15:17:53.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:53.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:53.006243+0000) 2024-03-06T15:17:54.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:54.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:54.006446+0000) 2024-03-06T15:17:55.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:55.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:55.006744+0000) 2024-03-06T15:17:56.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:56.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:56.006926+0000) 2024-03-06T15:17:57.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:57.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:57.007158+0000) 2024-03-06T15:17:58.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:58.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:58.007307+0000) 2024-03-06T15:17:59.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:17:59.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:59.007553+0000) 2024-03-06T15:18:00.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:00.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:00.007800+0000) 2024-03-06T15:18:01.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:01.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:01.008054+0000) 2024-03-06T15:18:02.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:02.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:02.008283+0000) 2024-03-06T15:18:03.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:03.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:03.008511+0000) 2024-03-06T15:18:04.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:04.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:04.008768+0000) 2024-03-06T15:18:05.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:05.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:05.009011+0000) 2024-03-06T15:18:06.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:06.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:06.009230+0000) 2024-03-06T15:18:07.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:07.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:07.009459+0000) 2024-03-06T15:18:08.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:08.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:08.009714+0000) 2024-03-06T15:18:09.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:09.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:09.009935+0000) 2024-03-06T15:18:10.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:10.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:10.010093+0000) 2024-03-06T15:18:11.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:11.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:11.010285+0000) 2024-03-06T15:18:12.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:12.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:12.010514+0000) 2024-03-06T15:18:13.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:13.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:13.010785+0000) 2024-03-06T15:18:14.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:14.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:14.011044+0000) 2024-03-06T15:18:15.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:15.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:15.011323+0000) 2024-03-06T15:18:16.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:16.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:16.011496+0000) 2024-03-06T15:18:17.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:17.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:17.011819+0000) 2024-03-06T15:18:18.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:18.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:18.012036+0000) 2024-03-06T15:18:19.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:19.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:19.012269+0000) 2024-03-06T15:18:20.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:20.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:20.012505+0000) 2024-03-06T15:18:20.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:20.879 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:21.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:21.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:21.012760+0000) 2024-03-06T15:18:22.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:22.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:22.012986+0000) 2024-03-06T15:18:23.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:23.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:23.013196+0000) 2024-03-06T15:18:24.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:24.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:24.013418+0000) 2024-03-06T15:18:25.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:25.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:25.013606+0000) 2024-03-06T15:18:26.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:26.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:26.013849+0000) 2024-03-06T15:18:27.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:27.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:27.014105+0000) 2024-03-06T15:18:28.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:28.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:28.014365+0000) 2024-03-06T15:18:29.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:29.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:29.014604+0000) 2024-03-06T15:18:30.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:30.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:30.014828+0000) 2024-03-06T15:18:31.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:31.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:31.015016+0000) 2024-03-06T15:18:32.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:32.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:32.015248+0000) 2024-03-06T15:18:33.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:33.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:33.015476+0000) 2024-03-06T15:18:34.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:34.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:34.015771+0000) 2024-03-06T15:18:35.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:35.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:35.016043+0000) 2024-03-06T15:18:36.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:36.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:36.016310+0000) 2024-03-06T15:18:37.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:37.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:37.016595+0000) 2024-03-06T15:18:38.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:38.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:38.016817+0000) 2024-03-06T15:18:39.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:39.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:39.017010+0000) 2024-03-06T15:18:40.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:40.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:40.017189+0000) 2024-03-06T15:18:41.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:41.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:41.017427+0000) 2024-03-06T15:18:42.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:42.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:42.017643+0000) 2024-03-06T15:18:43.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:43.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:43.017933+0000) 2024-03-06T15:18:44.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:44.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:44.018136+0000) 2024-03-06T15:18:45.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:45.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:45.018346+0000) 2024-03-06T15:18:46.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:46.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:46.018570+0000) 2024-03-06T15:18:47.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:47.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:47.018854+0000) 2024-03-06T15:18:48.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:48.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:48.019055+0000) 2024-03-06T15:18:49.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:49.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:49.019277+0000) 2024-03-06T15:18:50.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:50.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:50.019497+0000) 2024-03-06T15:18:51.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:51.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:51.019774+0000) 2024-03-06T15:18:51.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:51.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:52.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:52.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:52.019994+0000) 2024-03-06T15:18:53.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:53.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:53.020230+0000) 2024-03-06T15:18:54.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:54.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:54.020493+0000) 2024-03-06T15:18:55.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:55.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:55.020800+0000) 2024-03-06T15:18:56.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:56.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:56.021031+0000) 2024-03-06T15:18:57.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:57.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:57.021321+0000) 2024-03-06T15:18:58.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:58.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:58.021521+0000) 2024-03-06T15:18:59.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:18:59.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:59.021781+0000) 2024-03-06T15:19:00.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:00.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:00.022010+0000) 2024-03-06T15:19:01.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:01.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:01.022199+0000) 2024-03-06T15:19:02.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:02.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:02.022436+0000) 2024-03-06T15:19:03.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:03.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:03.022660+0000) 2024-03-06T15:19:04.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:04.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:04.022934+0000) 2024-03-06T15:19:05.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:05.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:05.023204+0000) 2024-03-06T15:19:06.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:06.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:06.023484+0000) 2024-03-06T15:19:07.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:07.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:07.023761+0000) 2024-03-06T15:19:08.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:08.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:08.023999+0000) 2024-03-06T15:19:09.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:09.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:09.024228+0000) 2024-03-06T15:19:10.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:10.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:10.024422+0000) 2024-03-06T15:19:11.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:11.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:11.024639+0000) 2024-03-06T15:19:12.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:12.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:12.024970+0000) 2024-03-06T15:19:13.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:13.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:13.025251+0000) 2024-03-06T15:19:14.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:14.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:14.025549+0000) 2024-03-06T15:19:15.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:15.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:15.025857+0000) 2024-03-06T15:19:16.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:16.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:16.026069+0000) 2024-03-06T15:19:17.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:17.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:17.026300+0000) 2024-03-06T15:19:18.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:18.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:18.026522+0000) 2024-03-06T15:19:19.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:19.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:19.026789+0000) 2024-03-06T15:19:20.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:20.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:20.027049+0000) 2024-03-06T15:19:21.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:21.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:21.027317+0000) 2024-03-06T15:19:21.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:21.304 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:22.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:22.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:22.027516+0000) 2024-03-06T15:19:23.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:23.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:23.027813+0000) 2024-03-06T15:19:24.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:24.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:24.028012+0000) 2024-03-06T15:19:25.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:25.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:25.028228+0000) 2024-03-06T15:19:26.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:26.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:26.028508+0000) 2024-03-06T15:19:27.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:27.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:27.028815+0000) 2024-03-06T15:19:28.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:28.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:28.029029+0000) 2024-03-06T15:19:29.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:29.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:29.029309+0000) 2024-03-06T15:19:30.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:30.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:30.029607+0000) 2024-03-06T15:19:31.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:31.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:31.029879+0000) 2024-03-06T15:19:32.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:32.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:32.030080+0000) 2024-03-06T15:19:33.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:33.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:33.030284+0000) 2024-03-06T15:19:34.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:34.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:34.030513+0000) 2024-03-06T15:19:35.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:35.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:35.030723+0000) 2024-03-06T15:19:36.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:36.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:36.030932+0000) 2024-03-06T15:19:37.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:37.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:37.031176+0000) 2024-03-06T15:19:38.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:38.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:38.031367+0000) 2024-03-06T15:19:39.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:39.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:39.031548+0000) 2024-03-06T15:19:40.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:40.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:40.031777+0000) 2024-03-06T15:19:41.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:41.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:41.031992+0000) 2024-03-06T15:19:42.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:42.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:42.032256+0000) 2024-03-06T15:19:43.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:43.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:43.032533+0000) 2024-03-06T15:19:44.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:44.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:44.032817+0000) 2024-03-06T15:19:45.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:45.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:45.033047+0000) 2024-03-06T15:19:46.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:46.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:46.033326+0000) 2024-03-06T15:19:47.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:47.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:47.033604+0000) 2024-03-06T15:19:48.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:48.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:48.033780+0000) 2024-03-06T15:19:49.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:49.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:49.033982+0000) 2024-03-06T15:19:50.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:50.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:50.034184+0000) 2024-03-06T15:19:51.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:51.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:51.034472+0000) 2024-03-06T15:19:51.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:51.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:52.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:52.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:52.034785+0000) 2024-03-06T15:19:53.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:53.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:53.034971+0000) 2024-03-06T15:19:54.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:54.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:54.035145+0000) 2024-03-06T15:19:55.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:55.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:55.035415+0000) 2024-03-06T15:19:56.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:56.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:56.035715+0000) 2024-03-06T15:19:57.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:57.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:57.035994+0000) 2024-03-06T15:19:58.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:58.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:58.036268+0000) 2024-03-06T15:19:59.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:19:59.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:59.036488+0000) 2024-03-06T15:20:00.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:00.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:00.036739+0000) 2024-03-06T15:20:01.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:01.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:01.036952+0000) 2024-03-06T15:20:02.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:02.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:02.037216+0000) 2024-03-06T15:20:03.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:03.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:03.037431+0000) 2024-03-06T15:20:04.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:04.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:04.037638+0000) 2024-03-06T15:20:05.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:05.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:05.037898+0000) 2024-03-06T15:20:06.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:06.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:06.038139+0000) 2024-03-06T15:20:07.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:07.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:07.038359+0000) 2024-03-06T15:20:08.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:08.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:08.038578+0000) 2024-03-06T15:20:09.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:09.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:09.038815+0000) 2024-03-06T15:20:10.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:10.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:10.039060+0000) 2024-03-06T15:20:11.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:11.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:11.039303+0000) 2024-03-06T15:20:12.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:12.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:12.039582+0000) 2024-03-06T15:20:13.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:13.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:13.039874+0000) 2024-03-06T15:20:14.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:14.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:14.040076+0000) 2024-03-06T15:20:15.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:15.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:15.040302+0000) 2024-03-06T15:20:16.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:16.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:16.040566+0000) 2024-03-06T15:20:17.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:17.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:17.040854+0000) 2024-03-06T15:20:18.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:18.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:18.041136+0000) 2024-03-06T15:20:19.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:19.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:19.041408+0000) 2024-03-06T15:20:20.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:20.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:20.041747+0000) 2024-03-06T15:20:21.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:21.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:21.042036+0000) 2024-03-06T15:20:21.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:21.740 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:22.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:22.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:22.042319+0000) 2024-03-06T15:20:23.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:23.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:23.042606+0000) 2024-03-06T15:20:24.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:24.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:24.042808+0000) 2024-03-06T15:20:25.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:25.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:25.043087+0000) 2024-03-06T15:20:26.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:26.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:26.043276+0000) 2024-03-06T15:20:27.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:27.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:27.043482+0000) 2024-03-06T15:20:28.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:28.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:28.043760+0000) 2024-03-06T15:20:29.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:29.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:29.043975+0000) 2024-03-06T15:20:30.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:30.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:30.044261+0000) 2024-03-06T15:20:31.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:31.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:31.044465+0000) 2024-03-06T15:20:32.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:32.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:32.044660+0000) 2024-03-06T15:20:33.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:33.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:33.044937+0000) 2024-03-06T15:20:34.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:34.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:34.045155+0000) 2024-03-06T15:20:35.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:35.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:35.045425+0000) 2024-03-06T15:20:36.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:36.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:36.045744+0000) 2024-03-06T15:20:37.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:37.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:37.046002+0000) 2024-03-06T15:20:38.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:38.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:38.046200+0000) 2024-03-06T15:20:39.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:39.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:39.046419+0000) 2024-03-06T15:20:40.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:40.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:40.046753+0000) 2024-03-06T15:20:41.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:41.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:41.046988+0000) 2024-03-06T15:20:42.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:42.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:42.047276+0000) 2024-03-06T15:20:43.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:43.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:43.047502+0000) 2024-03-06T15:20:44.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:44.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:44.047761+0000) 2024-03-06T15:20:45.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:45.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:45.047986+0000) 2024-03-06T15:20:46.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:46.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:46.048306+0000) 2024-03-06T15:20:47.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:47.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:47.048520+0000) 2024-03-06T15:20:48.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:48.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:48.048779+0000) 2024-03-06T15:20:49.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:49.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:49.049004+0000) 2024-03-06T15:20:50.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:50.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:50.049296+0000) 2024-03-06T15:20:51.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:51.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:51.049498+0000) 2024-03-06T15:20:51.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:51.867 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:52.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:52.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:52.049757+0000) 2024-03-06T15:20:53.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:53.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:53.049954+0000) 2024-03-06T15:20:54.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:54.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:54.050180+0000) 2024-03-06T15:20:55.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:55.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:55.050405+0000) 2024-03-06T15:20:56.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:56.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:56.050627+0000) 2024-03-06T15:20:57.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:57.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:57.050866+0000) 2024-03-06T15:20:58.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:58.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:58.051094+0000) 2024-03-06T15:20:59.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:20:59.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:59.051324+0000) 2024-03-06T15:21:00.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:00.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:00.051556+0000) 2024-03-06T15:21:01.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:01.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:01.051789+0000) 2024-03-06T15:21:02.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:02.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:02.052031+0000) 2024-03-06T15:21:03.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:03.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:03.052262+0000) 2024-03-06T15:21:04.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:04.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:04.052488+0000) 2024-03-06T15:21:05.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:05.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:05.052798+0000) 2024-03-06T15:21:06.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:06.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:06.053081+0000) 2024-03-06T15:21:07.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:07.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:07.053361+0000) 2024-03-06T15:21:08.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:08.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:08.053650+0000) 2024-03-06T15:21:09.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:09.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:09.053911+0000) 2024-03-06T15:21:10.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:10.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:10.054146+0000) 2024-03-06T15:21:11.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:11.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:11.054351+0000) 2024-03-06T15:21:12.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:12.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:12.054595+0000) 2024-03-06T15:21:13.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:13.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:13.054857+0000) 2024-03-06T15:21:14.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:14.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:14.055103+0000) 2024-03-06T15:21:15.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:15.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:15.055310+0000) 2024-03-06T15:21:16.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:16.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:16.055509+0000) 2024-03-06T15:21:17.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:17.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:17.055761+0000) 2024-03-06T15:21:18.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:18.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:18.056000+0000) 2024-03-06T15:21:19.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:19.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:19.056237+0000) 2024-03-06T15:21:20.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:20.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:20.056533+0000) 2024-03-06T15:21:21.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:21.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:21.056813+0000) 2024-03-06T15:21:22.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:22.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:22.057093+0000) 2024-03-06T15:21:22.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:22.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:23.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:23.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:23.057363+0000) 2024-03-06T15:21:24.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:24.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:24.057650+0000) 2024-03-06T15:21:25.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:25.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:25.057891+0000) 2024-03-06T15:21:26.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:26.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:26.058121+0000) 2024-03-06T15:21:27.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:27.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:27.058372+0000) 2024-03-06T15:21:28.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:28.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:28.058605+0000) 2024-03-06T15:21:29.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:29.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:29.058816+0000) 2024-03-06T15:21:30.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:30.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:30.059059+0000) 2024-03-06T15:21:31.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:31.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:31.059282+0000) 2024-03-06T15:21:32.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:32.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:32.059513+0000) 2024-03-06T15:21:33.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:33.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:33.059781+0000) 2024-03-06T15:21:34.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:34.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:34.060063+0000) 2024-03-06T15:21:35.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:35.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:35.060341+0000) 2024-03-06T15:21:36.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:36.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:36.060606+0000) 2024-03-06T15:21:37.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:37.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:37.060889+0000) 2024-03-06T15:21:38.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:38.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:38.061171+0000) 2024-03-06T15:21:39.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:39.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:39.061453+0000) 2024-03-06T15:21:40.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:40.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:40.061783+0000) 2024-03-06T15:21:41.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:41.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:41.062048+0000) 2024-03-06T15:21:42.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:42.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:42.062343+0000) 2024-03-06T15:21:43.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:43.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:43.062586+0000) 2024-03-06T15:21:44.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:44.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:44.062855+0000) 2024-03-06T15:21:45.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:45.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:45.063142+0000) 2024-03-06T15:21:46.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:46.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:46.063410+0000) 2024-03-06T15:21:47.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:47.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:47.063624+0000) 2024-03-06T15:21:48.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:48.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:48.063851+0000) 2024-03-06T15:21:49.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:49.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:49.064096+0000) 2024-03-06T15:21:50.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:50.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:50.064336+0000) 2024-03-06T15:21:51.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:51.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:51.064622+0000) 2024-03-06T15:21:52.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:52.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:52.064940+0000) 2024-03-06T15:21:52.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:52.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:53.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:53.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:53.065147+0000) 2024-03-06T15:21:54.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:54.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:54.065419+0000) 2024-03-06T15:21:55.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:55.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:55.065721+0000) 2024-03-06T15:21:56.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:56.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:56.066000+0000) 2024-03-06T15:21:57.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:57.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:57.066250+0000) 2024-03-06T15:21:58.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:58.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:58.066546+0000) 2024-03-06T15:21:59.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:21:59.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:59.066828+0000) 2024-03-06T15:22:00.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:00.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:00.067045+0000) 2024-03-06T15:22:01.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:01.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:01.067288+0000) 2024-03-06T15:22:02.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:02.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:02.067491+0000) 2024-03-06T15:22:03.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:03.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:03.067738+0000) 2024-03-06T15:22:04.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:04.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:04.067964+0000) 2024-03-06T15:22:05.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:05.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:05.068164+0000) 2024-03-06T15:22:06.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:06.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:06.068389+0000) 2024-03-06T15:22:07.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:07.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:07.068624+0000) 2024-03-06T15:22:08.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:08.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:08.068847+0000) 2024-03-06T15:22:09.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:09.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:09.069038+0000) 2024-03-06T15:22:10.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:10.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:10.069225+0000) 2024-03-06T15:22:11.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:11.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:11.069416+0000) 2024-03-06T15:22:12.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:12.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:12.069730+0000) 2024-03-06T15:22:13.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:13.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:13.070020+0000) 2024-03-06T15:22:14.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:14.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:14.070303+0000) 2024-03-06T15:22:15.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:15.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:15.070581+0000) 2024-03-06T15:22:16.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:16.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:16.070855+0000) 2024-03-06T15:22:17.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:17.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:17.071144+0000) 2024-03-06T15:22:18.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:18.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:18.071422+0000) 2024-03-06T15:22:19.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:19.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:19.071715+0000) 2024-03-06T15:22:20.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:20.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:20.072008+0000) 2024-03-06T15:22:21.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:21.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:21.072293+0000) 2024-03-06T15:22:22.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:22.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:22.072539+0000) 2024-03-06T15:22:22.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:22.462 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:23.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:23.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:23.072808+0000) 2024-03-06T15:22:24.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:24.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:24.073010+0000) 2024-03-06T15:22:25.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:25.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:25.073238+0000) 2024-03-06T15:22:26.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:26.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:26.073455+0000) 2024-03-06T15:22:27.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:27.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:27.073773+0000) 2024-03-06T15:22:28.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:28.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:28.074052+0000) 2024-03-06T15:22:29.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:29.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:29.074333+0000) 2024-03-06T15:22:30.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:30.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:30.074550+0000) 2024-03-06T15:22:31.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:31.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:31.074770+0000) 2024-03-06T15:22:32.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:32.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:32.074958+0000) 2024-03-06T15:22:33.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:33.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:33.075187+0000) 2024-03-06T15:22:34.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:34.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:34.075378+0000) 2024-03-06T15:22:35.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:35.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:35.075581+0000) 2024-03-06T15:22:36.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:36.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:36.075832+0000) 2024-03-06T15:22:37.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:37.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:37.076037+0000) 2024-03-06T15:22:38.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:38.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:38.076279+0000) 2024-03-06T15:22:39.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:39.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:39.076459+0000) 2024-03-06T15:22:40.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:40.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:40.076717+0000) 2024-03-06T15:22:41.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:41.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:41.076938+0000) 2024-03-06T15:22:42.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:42.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:42.077165+0000) 2024-03-06T15:22:43.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:43.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:43.077396+0000) 2024-03-06T15:22:44.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:44.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:44.077592+0000) 2024-03-06T15:22:45.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:45.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:45.077811+0000) 2024-03-06T15:22:46.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:46.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:46.078019+0000) 2024-03-06T15:22:47.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:47.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:47.078252+0000) 2024-03-06T15:22:48.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:48.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:48.078475+0000) 2024-03-06T15:22:49.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:49.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:49.078696+0000) 2024-03-06T15:22:50.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:50.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:50.078936+0000) 2024-03-06T15:22:51.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:51.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:51.079112+0000) 2024-03-06T15:22:52.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:52.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:52.079340+0000) 2024-03-06T15:22:52.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:52.634 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:53.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:53.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:53.079570+0000) 2024-03-06T15:22:54.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:54.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:54.079802+0000) 2024-03-06T15:22:55.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:55.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:55.080037+0000) 2024-03-06T15:22:56.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:56.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:56.080230+0000) 2024-03-06T15:22:57.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:57.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:57.080453+0000) 2024-03-06T15:22:58.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:58.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:58.080692+0000) 2024-03-06T15:22:59.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:22:59.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:59.080892+0000) 2024-03-06T15:23:00.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:00.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:00.081128+0000) 2024-03-06T15:23:01.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:01.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:01.081349+0000) 2024-03-06T15:23:02.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:02.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:02.081574+0000) 2024-03-06T15:23:03.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:03.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:03.081805+0000) 2024-03-06T15:23:04.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:04.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:04.081995+0000) 2024-03-06T15:23:05.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:05.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:05.082187+0000) 2024-03-06T15:23:06.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:06.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:06.082424+0000) 2024-03-06T15:23:07.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:07.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:07.082654+0000) 2024-03-06T15:23:08.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:08.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:08.082917+0000) 2024-03-06T15:23:09.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:09.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:09.083099+0000) 2024-03-06T15:23:10.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:10.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:10.083300+0000) 2024-03-06T15:23:11.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:11.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:11.083480+0000) 2024-03-06T15:23:12.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:12.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:12.083699+0000) 2024-03-06T15:23:13.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:13.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:13.083898+0000) 2024-03-06T15:23:14.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:14.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:14.084082+0000) 2024-03-06T15:23:15.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:15.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:15.084323+0000) 2024-03-06T15:23:16.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:16.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:16.084546+0000) 2024-03-06T15:23:17.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:17.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:17.084792+0000) 2024-03-06T15:23:18.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:18.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:18.084988+0000) 2024-03-06T15:23:19.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:19.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:19.085214+0000) 2024-03-06T15:23:20.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:20.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:20.085446+0000) 2024-03-06T15:23:21.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:21.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:21.085696+0000) 2024-03-06T15:23:22.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:22.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:22.085890+0000) 2024-03-06T15:23:22.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:22.854 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:23.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:23.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:23.086069+0000) 2024-03-06T15:23:24.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:24.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:24.086236+0000) 2024-03-06T15:23:25.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:25.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:25.086468+0000) 2024-03-06T15:23:26.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:26.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:26.086735+0000) 2024-03-06T15:23:27.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:27.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:27.086980+0000) 2024-03-06T15:23:28.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:28.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:28.087220+0000) 2024-03-06T15:23:29.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:29.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:29.087457+0000) 2024-03-06T15:23:30.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:30.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:30.087941+0000) 2024-03-06T15:23:31.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:31.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:31.088176+0000) 2024-03-06T15:23:32.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:32.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:32.088416+0000) 2024-03-06T15:23:33.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:33.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:33.088700+0000) 2024-03-06T15:23:34.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:34.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:34.088927+0000) 2024-03-06T15:23:35.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:35.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:35.089170+0000) 2024-03-06T15:23:36.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:36.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:36.089409+0000) 2024-03-06T15:23:37.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:37.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:37.089656+0000) 2024-03-06T15:23:38.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:38.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:38.089950+0000) 2024-03-06T15:23:39.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:39.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:39.090167+0000) 2024-03-06T15:23:40.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:40.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:40.090392+0000) 2024-03-06T15:23:41.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:41.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:41.090571+0000) 2024-03-06T15:23:42.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:42.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:42.090809+0000) 2024-03-06T15:23:43.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:43.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:43.091051+0000) 2024-03-06T15:23:44.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:44.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:44.091240+0000) 2024-03-06T15:23:45.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:45.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:45.091464+0000) 2024-03-06T15:23:46.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:46.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:46.091730+0000) 2024-03-06T15:23:47.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:47.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:47.091963+0000) 2024-03-06T15:23:48.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:48.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:48.092136+0000) 2024-03-06T15:23:49.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:49.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:49.092362+0000) 2024-03-06T15:23:50.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:50.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:50.092595+0000) 2024-03-06T15:23:51.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:51.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:51.092827+0000) 2024-03-06T15:23:52.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:52.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:52.093020+0000) 2024-03-06T15:23:53.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:53.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:53.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:53.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:53.093207+0000) 2024-03-06T15:23:54.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:54.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:54.093425+0000) 2024-03-06T15:23:55.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:55.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:55.093660+0000) 2024-03-06T15:23:56.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:56.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:56.093923+0000) 2024-03-06T15:23:57.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:57.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:57.094107+0000) 2024-03-06T15:23:58.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:58.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:58.094347+0000) 2024-03-06T15:23:59.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:23:59.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:59.094588+0000) 2024-03-06T15:24:00.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:00.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:00.094839+0000) 2024-03-06T15:24:01.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:01.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:01.095077+0000) 2024-03-06T15:24:02.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:02.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:02.095316+0000) 2024-03-06T15:24:03.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:03.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:03.095609+0000) 2024-03-06T15:24:04.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:04.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:04.095845+0000) 2024-03-06T15:24:05.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:05.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:05.096081+0000) 2024-03-06T15:24:06.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:06.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:06.096317+0000) 2024-03-06T15:24:07.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:07.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:07.096564+0000) 2024-03-06T15:24:08.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:08.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:08.096804+0000) 2024-03-06T15:24:09.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:09.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:09.097017+0000) 2024-03-06T15:24:10.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:10.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:10.097260+0000) 2024-03-06T15:24:11.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:11.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:11.097463+0000) 2024-03-06T15:24:12.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:12.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:12.097715+0000) 2024-03-06T15:24:13.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:13.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:13.097905+0000) 2024-03-06T15:24:14.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:14.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:14.098168+0000) 2024-03-06T15:24:15.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:15.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:15.098433+0000) 2024-03-06T15:24:16.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:16.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:16.098704+0000) 2024-03-06T15:24:17.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:17.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:17.098962+0000) 2024-03-06T15:24:18.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:18.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:18.099217+0000) 2024-03-06T15:24:19.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:19.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:19.099465+0000) 2024-03-06T15:24:20.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:20.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:20.099764+0000) 2024-03-06T15:24:21.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:21.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:21.099983+0000) 2024-03-06T15:24:22.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:22.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:22.100223+0000) 2024-03-06T15:24:23.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:23.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:23.100441+0000) 2024-03-06T15:24:23.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:23.272 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:24.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:24.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:24.100755+0000) 2024-03-06T15:24:25.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:25.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:25.101006+0000) 2024-03-06T15:24:26.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:26.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:26.101207+0000) 2024-03-06T15:24:27.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:27.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:27.101355+0000) 2024-03-06T15:24:28.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:28.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:28.101582+0000) 2024-03-06T15:24:29.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:29.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:29.101822+0000) 2024-03-06T15:24:30.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:30.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:30.102011+0000) 2024-03-06T15:24:31.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:31.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:31.102215+0000) 2024-03-06T15:24:32.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:32.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:32.102438+0000) 2024-03-06T15:24:33.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:33.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:33.102634+0000) 2024-03-06T15:24:34.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:34.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:34.102829+0000) 2024-03-06T15:24:35.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:35.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:35.103060+0000) 2024-03-06T15:24:36.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:36.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:36.103240+0000) 2024-03-06T15:24:37.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:37.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:37.103480+0000) 2024-03-06T15:24:38.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:38.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:38.103722+0000) 2024-03-06T15:24:39.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:39.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:39.103975+0000) 2024-03-06T15:24:40.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:40.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:40.104207+0000) 2024-03-06T15:24:41.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:41.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:41.104404+0000) 2024-03-06T15:24:42.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:42.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:42.104626+0000) 2024-03-06T15:24:43.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:43.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:43.104861+0000) 2024-03-06T15:24:44.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:44.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:44.105088+0000) 2024-03-06T15:24:45.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:45.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:45.105312+0000) 2024-03-06T15:24:46.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:46.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:46.105503+0000) 2024-03-06T15:24:47.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:47.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:47.105730+0000) 2024-03-06T15:24:48.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:48.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:48.105945+0000) 2024-03-06T15:24:49.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:49.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:49.106124+0000) 2024-03-06T15:24:50.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:50.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:50.106325+0000) 2024-03-06T15:24:51.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:51.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:51.106526+0000) 2024-03-06T15:24:52.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:52.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:52.106782+0000) 2024-03-06T15:24:53.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:53.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:53.107020+0000) 2024-03-06T15:24:53.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:53.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:54.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:54.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:54.107245+0000) 2024-03-06T15:24:55.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:55.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:55.107471+0000) 2024-03-06T15:24:56.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:56.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:56.107728+0000) 2024-03-06T15:24:57.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:57.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:57.107994+0000) 2024-03-06T15:24:58.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:58.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:58.108227+0000) 2024-03-06T15:24:59.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:24:59.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:59.108454+0000) 2024-03-06T15:25:00.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:00.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:00.108714+0000) 2024-03-06T15:25:01.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:01.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:01.108924+0000) 2024-03-06T15:25:02.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:02.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:02.109129+0000) 2024-03-06T15:25:03.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:03.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:03.109332+0000) 2024-03-06T15:25:04.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:04.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:04.109558+0000) 2024-03-06T15:25:05.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:05.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:05.109789+0000) 2024-03-06T15:25:06.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:06.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:06.110010+0000) 2024-03-06T15:25:07.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:07.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:07.110202+0000) 2024-03-06T15:25:08.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:08.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:08.110439+0000) 2024-03-06T15:25:09.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:09.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:09.110623+0000) 2024-03-06T15:25:10.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:10.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:10.110860+0000) 2024-03-06T15:25:11.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:11.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:11.111096+0000) 2024-03-06T15:25:12.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:12.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:12.111312+0000) 2024-03-06T15:25:13.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:13.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:13.111548+0000) 2024-03-06T15:25:14.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:14.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:14.111768+0000) 2024-03-06T15:25:15.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:15.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:15.112003+0000) 2024-03-06T15:25:16.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:16.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:16.112173+0000) 2024-03-06T15:25:17.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:17.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:17.112395+0000) 2024-03-06T15:25:18.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:18.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:18.112621+0000) 2024-03-06T15:25:19.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:19.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:19.112814+0000) 2024-03-06T15:25:20.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:20.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:20.113058+0000) 2024-03-06T15:25:21.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:21.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:21.113255+0000) 2024-03-06T15:25:22.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:22.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:22.113474+0000) 2024-03-06T15:25:23.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:23.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:23.113759+0000) 2024-03-06T15:25:23.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:23.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:24.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:24.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:24.113943+0000) 2024-03-06T15:25:25.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:25.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:25.114163+0000) 2024-03-06T15:25:26.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:26.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:26.114349+0000) 2024-03-06T15:25:27.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:27.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:27.114600+0000) 2024-03-06T15:25:28.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:28.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:28.114846+0000) 2024-03-06T15:25:29.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:29.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:29.115095+0000) 2024-03-06T15:25:30.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:30.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:30.115342+0000) 2024-03-06T15:25:31.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:31.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:31.115535+0000) 2024-03-06T15:25:32.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:32.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:32.115793+0000) 2024-03-06T15:25:33.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:33.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:33.116027+0000) 2024-03-06T15:25:34.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:34.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:34.116218+0000) 2024-03-06T15:25:35.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:35.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:35.116417+0000) 2024-03-06T15:25:36.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:36.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:36.116608+0000) 2024-03-06T15:25:37.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:37.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:37.116790+0000) 2024-03-06T15:25:38.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:38.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:38.116977+0000) 2024-03-06T15:25:39.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:39.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:39.117215+0000) 2024-03-06T15:25:40.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:40.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:40.117385+0000) 2024-03-06T15:25:41.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:41.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:41.117620+0000) 2024-03-06T15:25:42.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:42.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:42.117850+0000) 2024-03-06T15:25:43.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:43.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:43.118082+0000) 2024-03-06T15:25:44.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:44.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:44.118311+0000) 2024-03-06T15:25:45.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:45.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:45.118592+0000) 2024-03-06T15:25:46.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:46.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:46.118826+0000) 2024-03-06T15:25:47.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:47.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:47.119016+0000) 2024-03-06T15:25:48.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:48.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:48.119249+0000) 2024-03-06T15:25:49.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:49.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:49.119478+0000) 2024-03-06T15:25:50.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:50.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:50.119765+0000) 2024-03-06T15:25:51.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:51.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:51.119958+0000) 2024-03-06T15:25:52.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:52.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:52.120193+0000) 2024-03-06T15:25:53.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:53.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:53.120473+0000) 2024-03-06T15:25:53.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:53.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:54.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:54.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:54.120744+0000) 2024-03-06T15:25:55.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:55.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:55.120973+0000) 2024-03-06T15:25:56.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:56.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:56.121193+0000) 2024-03-06T15:25:57.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:57.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:57.121384+0000) 2024-03-06T15:25:58.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:58.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:58.121618+0000) 2024-03-06T15:25:59.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:25:59.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:59.121834+0000) 2024-03-06T15:26:00.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:00.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:00.122095+0000) 2024-03-06T15:26:01.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:01.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:01.122321+0000) 2024-03-06T15:26:02.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:02.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:02.122572+0000) 2024-03-06T15:26:03.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:03.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:03.122806+0000) 2024-03-06T15:26:04.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:04.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:04.122962+0000) 2024-03-06T15:26:05.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:05.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:05.123185+0000) 2024-03-06T15:26:06.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:06.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:06.123406+0000) 2024-03-06T15:26:07.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:07.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:07.123586+0000) 2024-03-06T15:26:08.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:08.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:08.123829+0000) 2024-03-06T15:26:09.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:09.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:09.124059+0000) 2024-03-06T15:26:10.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:10.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:10.124291+0000) 2024-03-06T15:26:11.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:11.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:11.124470+0000) 2024-03-06T15:26:12.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:12.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:12.124650+0000) 2024-03-06T15:26:13.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:13.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:13.124926+0000) 2024-03-06T15:26:14.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:14.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:14.125118+0000) 2024-03-06T15:26:15.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:15.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:15.125370+0000) 2024-03-06T15:26:16.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:16.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:16.125609+0000) 2024-03-06T15:26:17.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:17.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:17.125817+0000) 2024-03-06T15:26:18.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:18.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:18.126037+0000) 2024-03-06T15:26:19.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:19.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:19.126286+0000) 2024-03-06T15:26:20.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:20.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:20.126465+0000) 2024-03-06T15:26:21.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:21.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:21.126742+0000) 2024-03-06T15:26:22.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:22.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:22.126991+0000) 2024-03-06T15:26:23.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:23.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:23.127247+0000) 2024-03-06T15:26:23.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:23.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:24.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:24.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:24.127488+0000) 2024-03-06T15:26:25.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:25.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:25.127776+0000) 2024-03-06T15:26:26.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:26.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:26.127995+0000) 2024-03-06T15:26:27.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:27.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:27.128224+0000) 2024-03-06T15:26:28.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:28.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:28.128466+0000) 2024-03-06T15:26:29.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:29.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:29.128753+0000) 2024-03-06T15:26:30.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:30.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:30.128993+0000) 2024-03-06T15:26:31.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:31.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:31.129202+0000) 2024-03-06T15:26:32.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:32.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:32.129371+0000) 2024-03-06T15:26:33.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:33.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:33.129611+0000) 2024-03-06T15:26:34.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:34.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:34.129797+0000) 2024-03-06T15:26:35.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:35.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:35.130035+0000) 2024-03-06T15:26:36.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:36.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:36.130253+0000) 2024-03-06T15:26:37.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:37.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:37.130438+0000) 2024-03-06T15:26:38.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:38.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:38.130775+0000) 2024-03-06T15:26:39.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:39.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:39.131004+0000) 2024-03-06T15:26:40.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:40.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:40.131258+0000) 2024-03-06T15:26:41.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:41.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:41.131494+0000) 2024-03-06T15:26:42.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:42.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:42.131773+0000) 2024-03-06T15:26:43.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:43.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:43.131989+0000) 2024-03-06T15:26:44.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:44.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:44.132212+0000) 2024-03-06T15:26:45.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:45.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:45.132443+0000) 2024-03-06T15:26:46.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:46.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:46.132699+0000) 2024-03-06T15:26:47.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:47.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:47.132925+0000) 2024-03-06T15:26:48.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:48.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:48.133169+0000) 2024-03-06T15:26:49.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:49.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:49.133350+0000) 2024-03-06T15:26:50.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:50.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:50.133580+0000) 2024-03-06T15:26:51.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:51.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:51.133805+0000) 2024-03-06T15:26:52.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:52.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:52.134038+0000) 2024-03-06T15:26:53.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:53.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:53.134249+0000) 2024-03-06T15:26:53.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:53.999 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:54.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:54.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:54.134476+0000) 2024-03-06T15:26:55.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:55.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:55.134736+0000) 2024-03-06T15:26:56.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:56.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:56.134914+0000) 2024-03-06T15:26:57.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:57.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:57.135149+0000) 2024-03-06T15:26:58.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:58.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:58.135403+0000) 2024-03-06T15:26:59.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:26:59.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:59.135640+0000) 2024-03-06T15:27:00.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:00.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:00.135892+0000) 2024-03-06T15:27:01.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:01.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:01.136079+0000) 2024-03-06T15:27:02.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:02.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:02.136310+0000) 2024-03-06T15:27:03.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:03.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:03.136553+0000) 2024-03-06T15:27:04.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:04.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:04.136778+0000) 2024-03-06T15:27:05.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:05.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:05.136962+0000) 2024-03-06T15:27:06.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:06.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:06.137142+0000) 2024-03-06T15:27:07.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:07.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:07.137366+0000) 2024-03-06T15:27:08.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:08.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:08.137584+0000) 2024-03-06T15:27:09.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:09.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:09.137810+0000) 2024-03-06T15:27:10.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:10.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:10.137995+0000) 2024-03-06T15:27:11.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:11.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:11.138208+0000) 2024-03-06T15:27:12.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:12.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:12.138423+0000) 2024-03-06T15:27:13.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:13.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:13.138696+0000) 2024-03-06T15:27:14.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:14.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:14.138922+0000) 2024-03-06T15:27:15.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:15.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:15.139113+0000) 2024-03-06T15:27:16.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:16.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:16.139332+0000) 2024-03-06T15:27:17.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:17.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:17.139570+0000) 2024-03-06T15:27:18.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:18.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:18.139819+0000) 2024-03-06T15:27:19.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:19.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:19.140063+0000) 2024-03-06T15:27:20.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:20.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:20.140307+0000) 2024-03-06T15:27:21.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:21.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:21.140515+0000) 2024-03-06T15:27:22.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:22.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:22.140770+0000) 2024-03-06T15:27:23.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:23.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:23.140973+0000) 2024-03-06T15:27:24.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:24.136 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:24.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:24.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:24.141176+0000) 2024-03-06T15:27:25.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:25.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:25.141382+0000) 2024-03-06T15:27:26.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:26.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:26.141649+0000) 2024-03-06T15:27:27.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:27.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:27.141894+0000) 2024-03-06T15:27:28.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:28.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:28.142047+0000) 2024-03-06T15:27:29.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:29.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:29.142271+0000) 2024-03-06T15:27:30.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:30.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:30.142463+0000) 2024-03-06T15:27:31.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:31.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:31.142650+0000) 2024-03-06T15:27:32.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:32.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:32.142918+0000) 2024-03-06T15:27:33.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:33.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:33.143108+0000) 2024-03-06T15:27:34.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:34.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:34.143334+0000) 2024-03-06T15:27:35.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:35.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:35.143568+0000) 2024-03-06T15:27:36.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:36.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:36.143783+0000) 2024-03-06T15:27:37.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:37.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:37.143988+0000) 2024-03-06T15:27:38.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:38.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:38.144217+0000) 2024-03-06T15:27:39.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:39.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:39.144450+0000) 2024-03-06T15:27:40.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:40.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:40.144723+0000) 2024-03-06T15:27:41.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:41.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:41.144956+0000) 2024-03-06T15:27:42.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:42.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:42.145176+0000) 2024-03-06T15:27:43.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:43.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:43.145409+0000) 2024-03-06T15:27:44.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:44.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:44.145614+0000) 2024-03-06T15:27:45.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:45.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:45.145839+0000) 2024-03-06T15:27:46.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:46.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:46.146062+0000) 2024-03-06T15:27:47.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:47.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:47.146340+0000) 2024-03-06T15:27:48.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:48.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:48.146594+0000) 2024-03-06T15:27:49.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:49.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:49.146834+0000) 2024-03-06T15:27:50.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:50.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:50.147066+0000) 2024-03-06T15:27:51.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:51.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:51.147281+0000) 2024-03-06T15:27:52.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:52.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:52.147504+0000) 2024-03-06T15:27:53.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:53.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:53.147779+0000) 2024-03-06T15:27:54.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:54.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:54.147971+0000) 2024-03-06T15:27:54.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:54.278 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:55.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:55.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:55.148155+0000) 2024-03-06T15:27:56.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:56.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:56.148382+0000) 2024-03-06T15:27:57.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:57.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:57.148612+0000) 2024-03-06T15:27:58.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:58.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:58.148826+0000) 2024-03-06T15:27:59.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:27:59.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:59.149049+0000) 2024-03-06T15:28:00.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:00.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:00.149276+0000) 2024-03-06T15:28:01.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:01.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:01.149500+0000) 2024-03-06T15:28:02.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:02.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:02.149733+0000) 2024-03-06T15:28:03.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:03.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:03.149991+0000) 2024-03-06T15:28:04.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:04.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:04.150213+0000) 2024-03-06T15:28:05.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:05.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:05.150434+0000) 2024-03-06T15:28:06.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:06.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:06.150656+0000) 2024-03-06T15:28:07.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:07.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:07.150922+0000) 2024-03-06T15:28:08.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:08.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:08.151105+0000) 2024-03-06T15:28:09.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:09.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:09.151338+0000) 2024-03-06T15:28:10.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:10.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:10.151582+0000) 2024-03-06T15:28:11.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:11.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:11.151805+0000) 2024-03-06T15:28:12.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:12.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:12.152005+0000) 2024-03-06T15:28:13.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:13.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:13.152255+0000) 2024-03-06T15:28:14.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:14.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:14.152502+0000) 2024-03-06T15:28:15.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:15.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:15.152784+0000) 2024-03-06T15:28:16.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:16.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:16.153018+0000) 2024-03-06T15:28:17.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:17.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:17.153242+0000) 2024-03-06T15:28:18.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:18.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:18.153465+0000) 2024-03-06T15:28:19.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:19.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:19.153650+0000) 2024-03-06T15:28:20.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:20.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:20.153888+0000) 2024-03-06T15:28:21.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:21.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:21.154112+0000) 2024-03-06T15:28:22.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:22.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:22.154334+0000) 2024-03-06T15:28:23.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:23.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:23.154548+0000) 2024-03-06T15:28:24.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:24.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:24.154772+0000) 2024-03-06T15:28:24.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:24.490 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:25.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:25.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:25.154976+0000) 2024-03-06T15:28:26.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:26.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:26.155202+0000) 2024-03-06T15:28:27.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:27.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:27.155427+0000) 2024-03-06T15:28:28.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:28.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:28.155651+0000) 2024-03-06T15:28:29.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:29.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:29.155912+0000) 2024-03-06T15:28:30.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:30.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:30.156144+0000) 2024-03-06T15:28:31.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:31.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:31.156377+0000) 2024-03-06T15:28:32.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:32.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:32.156575+0000) 2024-03-06T15:28:33.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:33.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:33.156816+0000) 2024-03-06T15:28:34.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:34.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:34.157021+0000) 2024-03-06T15:28:35.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:35.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:35.157270+0000) 2024-03-06T15:28:36.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:36.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:36.157470+0000) 2024-03-06T15:28:37.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:37.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:37.157741+0000) 2024-03-06T15:28:38.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:38.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:38.157926+0000) 2024-03-06T15:28:39.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:39.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:39.158118+0000) 2024-03-06T15:28:40.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:40.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:40.158297+0000) 2024-03-06T15:28:41.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:41.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:41.158487+0000) 2024-03-06T15:28:42.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:42.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:42.158790+0000) 2024-03-06T15:28:43.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:43.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:43.159004+0000) 2024-03-06T15:28:44.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:44.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:44.159235+0000) 2024-03-06T15:28:45.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:45.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:45.159457+0000) 2024-03-06T15:28:46.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:46.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:46.159778+0000) 2024-03-06T15:28:47.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:47.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:47.159999+0000) 2024-03-06T15:28:48.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:48.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:48.160205+0000) 2024-03-06T15:28:49.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:49.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:49.160403+0000) 2024-03-06T15:28:50.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:50.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:50.160586+0000) 2024-03-06T15:28:51.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:51.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:51.160784+0000) 2024-03-06T15:28:52.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:52.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:52.161020+0000) 2024-03-06T15:28:53.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:53.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:53.161251+0000) 2024-03-06T15:28:54.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:54.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:54.161473+0000) 2024-03-06T15:28:54.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:54.699 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:55.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:55.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:55.161698+0000) 2024-03-06T15:28:56.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:56.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:56.161921+0000) 2024-03-06T15:28:57.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:57.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:57.162142+0000) 2024-03-06T15:28:58.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:58.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:58.162360+0000) 2024-03-06T15:28:59.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:28:59.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:59.162635+0000) 2024-03-06T15:29:00.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:00.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:00.162916+0000) 2024-03-06T15:29:01.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:01.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:01.163122+0000) 2024-03-06T15:29:02.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:02.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:02.163310+0000) 2024-03-06T15:29:03.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:03.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:03.163505+0000) 2024-03-06T15:29:04.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:04.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:04.163770+0000) 2024-03-06T15:29:05.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:05.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:05.164018+0000) 2024-03-06T15:29:06.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:06.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:06.164254+0000) 2024-03-06T15:29:07.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:07.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:07.164504+0000) 2024-03-06T15:29:08.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:08.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:08.164793+0000) 2024-03-06T15:29:09.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:09.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:09.164999+0000) 2024-03-06T15:29:10.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:10.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:10.165200+0000) 2024-03-06T15:29:11.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:11.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:11.165438+0000) 2024-03-06T15:29:12.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:12.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:12.165631+0000) 2024-03-06T15:29:13.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:13.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:13.165844+0000) 2024-03-06T15:29:14.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:14.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:14.166089+0000) 2024-03-06T15:29:15.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:15.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:15.166316+0000) 2024-03-06T15:29:16.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:16.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:16.166530+0000) 2024-03-06T15:29:17.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:17.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:17.166793+0000) 2024-03-06T15:29:18.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:18.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:18.167015+0000) 2024-03-06T15:29:19.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:19.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:19.167239+0000) 2024-03-06T15:29:20.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:20.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:20.167465+0000) 2024-03-06T15:29:21.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:21.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:21.167691+0000) 2024-03-06T15:29:22.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:22.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:22.167929+0000) 2024-03-06T15:29:23.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:23.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:23.168119+0000) 2024-03-06T15:29:24.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:24.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:24.168307+0000) 2024-03-06T15:29:24.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:24.931 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:25.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:25.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:25.168545+0000) 2024-03-06T15:29:26.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:26.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:26.168780+0000) 2024-03-06T15:29:27.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:27.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:27.168967+0000) 2024-03-06T15:29:28.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:28.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:28.169185+0000) 2024-03-06T15:29:29.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:29.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:29.169415+0000) 2024-03-06T15:29:30.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:30.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:30.169643+0000) 2024-03-06T15:29:31.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:31.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:31.169920+0000) 2024-03-06T15:29:32.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:32.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:32.170089+0000) 2024-03-06T15:29:33.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:33.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:33.170314+0000) 2024-03-06T15:29:34.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:34.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:34.170501+0000) 2024-03-06T15:29:35.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:35.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:35.170733+0000) 2024-03-06T15:29:36.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:36.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:36.170938+0000) 2024-03-06T15:29:37.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:37.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:37.171167+0000) 2024-03-06T15:29:38.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:38.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:38.171388+0000) 2024-03-06T15:29:39.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:39.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:39.171617+0000) 2024-03-06T15:29:40.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:40.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:40.171826+0000) 2024-03-06T15:29:41.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:41.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:41.172049+0000) 2024-03-06T15:29:42.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:42.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:42.172266+0000) 2024-03-06T15:29:43.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:43.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:43.172482+0000) 2024-03-06T15:29:44.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:44.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:44.172762+0000) 2024-03-06T15:29:45.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:45.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:45.172951+0000) 2024-03-06T15:29:46.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:46.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:46.173168+0000) 2024-03-06T15:29:47.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:47.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:47.173398+0000) 2024-03-06T15:29:48.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:48.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:48.173616+0000) 2024-03-06T15:29:49.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:49.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:49.173839+0000) 2024-03-06T15:29:50.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:50.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:50.174050+0000) 2024-03-06T15:29:51.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:51.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:51.174265+0000) 2024-03-06T15:29:52.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:52.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:52.174491+0000) 2024-03-06T15:29:53.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:53.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:53.174764+0000) 2024-03-06T15:29:54.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:54.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:54.174987+0000) 2024-03-06T15:29:55.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:55.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:55.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:55.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:55.175164+0000) 2024-03-06T15:29:56.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:56.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:56.175387+0000) 2024-03-06T15:29:57.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:57.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:57.175623+0000) 2024-03-06T15:29:58.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:58.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:58.175894+0000) 2024-03-06T15:29:59.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:29:59.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:59.176136+0000) 2024-03-06T15:30:00.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:00.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:00.176384+0000) 2024-03-06T15:30:01.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:01.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:01.176621+0000) 2024-03-06T15:30:02.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:02.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:02.176838+0000) 2024-03-06T15:30:03.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:03.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:03.177069+0000) 2024-03-06T15:30:04.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:04.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:04.177295+0000) 2024-03-06T15:30:05.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:05.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:05.177534+0000) 2024-03-06T15:30:06.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:06.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:06.177765+0000) 2024-03-06T15:30:07.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:07.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:07.177983+0000) 2024-03-06T15:30:08.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:08.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:08.178186+0000) 2024-03-06T15:30:09.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:09.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:09.178397+0000) 2024-03-06T15:30:10.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:10.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:10.178643+0000) 2024-03-06T15:30:11.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:11.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:11.178885+0000) 2024-03-06T15:30:12.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:12.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:12.179089+0000) 2024-03-06T15:30:13.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:13.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:13.179349+0000) 2024-03-06T15:30:14.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:14.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:14.179603+0000) 2024-03-06T15:30:15.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:15.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:15.179857+0000) 2024-03-06T15:30:16.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:16.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:16.180077+0000) 2024-03-06T15:30:17.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:17.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:17.180316+0000) 2024-03-06T15:30:18.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:18.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:18.180499+0000) 2024-03-06T15:30:19.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:19.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:19.180795+0000) 2024-03-06T15:30:20.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:20.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:20.181011+0000) 2024-03-06T15:30:21.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:21.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:21.181224+0000) 2024-03-06T15:30:22.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:22.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:22.181466+0000) 2024-03-06T15:30:23.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:23.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:23.181742+0000) 2024-03-06T15:30:24.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:24.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:24.181965+0000) 2024-03-06T15:30:25.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:25.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:25.182194+0000) 2024-03-06T15:30:25.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:25.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:26.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:26.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:26.182422+0000) 2024-03-06T15:30:27.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:27.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:27.182639+0000) 2024-03-06T15:30:28.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:28.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:28.182861+0000) 2024-03-06T15:30:29.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:29.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:29.183042+0000) 2024-03-06T15:30:30.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:30.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:30.183251+0000) 2024-03-06T15:30:31.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:31.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:31.183494+0000) 2024-03-06T15:30:32.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:32.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:32.183768+0000) 2024-03-06T15:30:33.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:33.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:33.184010+0000) 2024-03-06T15:30:34.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:34.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:34.184253+0000) 2024-03-06T15:30:35.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:35.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:35.184485+0000) 2024-03-06T15:30:36.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:36.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:36.184757+0000) 2024-03-06T15:30:37.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:37.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:37.184980+0000) 2024-03-06T15:30:38.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:38.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:38.185208+0000) 2024-03-06T15:30:39.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:39.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:39.185447+0000) 2024-03-06T15:30:40.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:40.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:40.185708+0000) 2024-03-06T15:30:41.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:41.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:41.185933+0000) 2024-03-06T15:30:42.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:42.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:42.186170+0000) 2024-03-06T15:30:43.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:43.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:43.186363+0000) 2024-03-06T15:30:44.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:44.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:44.186597+0000) 2024-03-06T15:30:45.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:45.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:45.186829+0000) 2024-03-06T15:30:46.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:46.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:46.187020+0000) 2024-03-06T15:30:47.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:47.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:47.187216+0000) 2024-03-06T15:30:48.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:48.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:48.187404+0000) 2024-03-06T15:30:49.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:49.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:49.187642+0000) 2024-03-06T15:30:50.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:50.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:50.187882+0000) 2024-03-06T15:30:51.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:51.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:51.188114+0000) 2024-03-06T15:30:52.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:52.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:52.188333+0000) 2024-03-06T15:30:53.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:53.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:53.188569+0000) 2024-03-06T15:30:54.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:54.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:54.188807+0000) 2024-03-06T15:30:55.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:55.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:55.189054+0000) 2024-03-06T15:30:55.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:55.503 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:56.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:56.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:56.189283+0000) 2024-03-06T15:30:57.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:57.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:57.189510+0000) 2024-03-06T15:30:58.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:58.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:58.189772+0000) 2024-03-06T15:30:59.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:30:59.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:59.190012+0000) 2024-03-06T15:31:00.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:00.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:00.190242+0000) 2024-03-06T15:31:01.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:01.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:01.190474+0000) 2024-03-06T15:31:02.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:02.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:02.190747+0000) 2024-03-06T15:31:03.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:03.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:03.190988+0000) 2024-03-06T15:31:04.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:04.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:04.191225+0000) 2024-03-06T15:31:05.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:05.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:05.191472+0000) 2024-03-06T15:31:06.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:06.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:06.191635+0000) 2024-03-06T15:31:07.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:07.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:07.191837+0000) 2024-03-06T15:31:08.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:08.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:08.192028+0000) 2024-03-06T15:31:09.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:09.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:09.192228+0000) 2024-03-06T15:31:10.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:10.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:10.192459+0000) 2024-03-06T15:31:11.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:11.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:11.192635+0000) 2024-03-06T15:31:12.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:12.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:12.192831+0000) 2024-03-06T15:31:13.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:13.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:13.193013+0000) 2024-03-06T15:31:14.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:14.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:14.193232+0000) 2024-03-06T15:31:15.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:15.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:15.193461+0000) 2024-03-06T15:31:16.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:16.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:16.193637+0000) 2024-03-06T15:31:17.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:17.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:17.193829+0000) 2024-03-06T15:31:18.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:18.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:18.194051+0000) 2024-03-06T15:31:19.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:19.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:19.194285+0000) 2024-03-06T15:31:20.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:20.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:20.194515+0000) 2024-03-06T15:31:21.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:21.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:21.194773+0000) 2024-03-06T15:31:22.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:22.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:22.194946+0000) 2024-03-06T15:31:23.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:23.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:23.195137+0000) 2024-03-06T15:31:24.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:24.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:24.195358+0000) 2024-03-06T15:31:25.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:25.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:25.195590+0000) 2024-03-06T15:31:25.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:25.683 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:26.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:26.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:26.195822+0000) 2024-03-06T15:31:27.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:27.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:27.196045+0000) 2024-03-06T15:31:28.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:28.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:28.196270+0000) 2024-03-06T15:31:29.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:29.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:29.196507+0000) 2024-03-06T15:31:30.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:30.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:30.196782+0000) 2024-03-06T15:31:31.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:31.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:31.196981+0000) 2024-03-06T15:31:32.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:32.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:32.197154+0000) 2024-03-06T15:31:33.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:33.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:33.197385+0000) 2024-03-06T15:31:34.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:34.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:34.197626+0000) 2024-03-06T15:31:35.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:35.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:35.197883+0000) 2024-03-06T15:31:36.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:36.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:36.198125+0000) 2024-03-06T15:31:37.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:37.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:37.198426+0000) 2024-03-06T15:31:38.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:38.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:38.198634+0000) 2024-03-06T15:31:39.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:39.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:39.198903+0000) 2024-03-06T15:31:40.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:40.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:40.199094+0000) 2024-03-06T15:31:41.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:41.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:41.199324+0000) 2024-03-06T15:31:42.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:42.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:42.199549+0000) 2024-03-06T15:31:43.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:43.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:43.199782+0000) 2024-03-06T15:31:44.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:44.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:44.199992+0000) 2024-03-06T15:31:45.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:45.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:45.200220+0000) 2024-03-06T15:31:46.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:46.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:46.200444+0000) 2024-03-06T15:31:47.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:47.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:47.200715+0000) 2024-03-06T15:31:48.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:48.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:48.200909+0000) 2024-03-06T15:31:49.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:49.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:49.201151+0000) 2024-03-06T15:31:50.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:50.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:50.201386+0000) 2024-03-06T15:31:51.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:51.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:51.201623+0000) 2024-03-06T15:31:52.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:52.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:52.201822+0000) 2024-03-06T15:31:53.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:53.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:53.202055+0000) 2024-03-06T15:31:54.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:54.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:54.202289+0000) 2024-03-06T15:31:55.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:55.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:55.202519+0000) 2024-03-06T15:31:55.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:55.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:56.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:56.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:56.202781+0000) 2024-03-06T15:31:57.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:57.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:57.202984+0000) 2024-03-06T15:31:58.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:58.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:58.203167+0000) 2024-03-06T15:31:59.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:31:59.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:59.203387+0000) 2024-03-06T15:32:00.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:00.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:00.203629+0000) 2024-03-06T15:32:01.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:01.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:01.203835+0000) 2024-03-06T15:32:02.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:02.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:02.204059+0000) 2024-03-06T15:32:03.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:03.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:03.204296+0000) 2024-03-06T15:32:04.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:04.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:04.204525+0000) 2024-03-06T15:32:05.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:05.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:05.204746+0000) 2024-03-06T15:32:06.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:06.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:06.204931+0000) 2024-03-06T15:32:07.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:07.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:07.205166+0000) 2024-03-06T15:32:08.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:08.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:08.205391+0000) 2024-03-06T15:32:09.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:09.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:09.205714+0000) 2024-03-06T15:32:10.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:10.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:10.206009+0000) 2024-03-06T15:32:11.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:11.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:11.206218+0000) 2024-03-06T15:32:12.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:12.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:12.206501+0000) 2024-03-06T15:32:13.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:13.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:13.206822+0000) 2024-03-06T15:32:14.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:14.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:14.207063+0000) 2024-03-06T15:32:15.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:15.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:15.207347+0000) 2024-03-06T15:32:16.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:16.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:16.207623+0000) 2024-03-06T15:32:17.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:17.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:17.207941+0000) 2024-03-06T15:32:18.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:18.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:18.208173+0000) 2024-03-06T15:32:19.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:19.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:19.208467+0000) 2024-03-06T15:32:20.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:20.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:20.208792+0000) 2024-03-06T15:32:21.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:21.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:21.209002+0000) 2024-03-06T15:32:22.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:22.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:22.209186+0000) 2024-03-06T15:32:23.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:23.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:23.209407+0000) 2024-03-06T15:32:24.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:24.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:24.209647+0000) 2024-03-06T15:32:25.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:25.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:25.210002+0000) 2024-03-06T15:32:26.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:26.044 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:26.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:26.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:26.210258+0000) 2024-03-06T15:32:27.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:27.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:27.210467+0000) 2024-03-06T15:32:28.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:28.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:28.210749+0000) 2024-03-06T15:32:29.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:29.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:29.210972+0000) 2024-03-06T15:32:30.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:30.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:30.211170+0000) 2024-03-06T15:32:31.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:31.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:31.211343+0000) 2024-03-06T15:32:32.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:32.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:32.211564+0000) 2024-03-06T15:32:33.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:33.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:33.211792+0000) 2024-03-06T15:32:34.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:34.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:34.211990+0000) 2024-03-06T15:32:35.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:35.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:35.212214+0000) 2024-03-06T15:32:36.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:36.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:36.212434+0000) 2024-03-06T15:32:37.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:37.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:37.212643+0000) 2024-03-06T15:32:38.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:38.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:38.212928+0000) 2024-03-06T15:32:39.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:39.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:39.213215+0000) 2024-03-06T15:32:40.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:40.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:40.213399+0000) 2024-03-06T15:32:41.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:41.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:41.213585+0000) 2024-03-06T15:32:42.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:42.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:42.213781+0000) 2024-03-06T15:32:43.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:43.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:43.213984+0000) 2024-03-06T15:32:44.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:44.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:44.214195+0000) 2024-03-06T15:32:45.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:45.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:45.214430+0000) 2024-03-06T15:32:46.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:46.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:46.214640+0000) 2024-03-06T15:32:47.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:47.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:47.214971+0000) 2024-03-06T15:32:48.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:48.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:48.215208+0000) 2024-03-06T15:32:49.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:49.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:49.215458+0000) 2024-03-06T15:32:50.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:50.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:50.215732+0000) 2024-03-06T15:32:51.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:51.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:51.215938+0000) 2024-03-06T15:32:52.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:52.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:52.216175+0000) 2024-03-06T15:32:53.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:53.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:53.216405+0000) 2024-03-06T15:32:54.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:54.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:54.216605+0000) 2024-03-06T15:32:55.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:55.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:55.216830+0000) 2024-03-06T15:32:56.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:56.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:56.217049+0000) 2024-03-06T15:32:56.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:56.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:57.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:57.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:57.217280+0000) 2024-03-06T15:32:58.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:58.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:58.217467+0000) 2024-03-06T15:32:59.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:32:59.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:59.217726+0000) 2024-03-06T15:33:00.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:00.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:00.217943+0000) 2024-03-06T15:33:01.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:01.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:01.218131+0000) 2024-03-06T15:33:02.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:02.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:02.218321+0000) 2024-03-06T15:33:03.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:03.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:03.218563+0000) 2024-03-06T15:33:04.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:04.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:04.218899+0000) 2024-03-06T15:33:05.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:05.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:05.219182+0000) 2024-03-06T15:33:06.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:06.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:06.219379+0000) 2024-03-06T15:33:07.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:07.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:07.219636+0000) 2024-03-06T15:33:08.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:08.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:08.219942+0000) 2024-03-06T15:33:09.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:09.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:09.220189+0000) 2024-03-06T15:33:10.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:10.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:10.220422+0000) 2024-03-06T15:33:11.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:11.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:11.220746+0000) 2024-03-06T15:33:12.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:12.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:12.220974+0000) 2024-03-06T15:33:13.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:13.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:13.221238+0000) 2024-03-06T15:33:14.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:14.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:14.221456+0000) 2024-03-06T15:33:15.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:15.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:15.221766+0000) 2024-03-06T15:33:16.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:16.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:16.221952+0000) 2024-03-06T15:33:17.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:17.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:17.222227+0000) 2024-03-06T15:33:18.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:18.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:18.222544+0000) 2024-03-06T15:33:19.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:19.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:19.222833+0000) 2024-03-06T15:33:20.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:20.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:20.223080+0000) 2024-03-06T15:33:21.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:21.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:21.223365+0000) 2024-03-06T15:33:22.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:22.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:22.223573+0000) 2024-03-06T15:33:23.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:23.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:23.223851+0000) 2024-03-06T15:33:24.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:24.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:24.224071+0000) 2024-03-06T15:33:25.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:25.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:25.224320+0000) 2024-03-06T15:33:26.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:26.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:26.224562+0000) 2024-03-06T15:33:26.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:26.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:27.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:27.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:27.224810+0000) 2024-03-06T15:33:28.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:28.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:28.225015+0000) 2024-03-06T15:33:29.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:29.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:29.225273+0000) 2024-03-06T15:33:30.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:30.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:30.225525+0000) 2024-03-06T15:33:31.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:31.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:31.225795+0000) 2024-03-06T15:33:32.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:32.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:32.225979+0000) 2024-03-06T15:33:33.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:33.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:33.226220+0000) 2024-03-06T15:33:34.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:34.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:34.226405+0000) 2024-03-06T15:33:35.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:35.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:35.226622+0000) 2024-03-06T15:33:36.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:36.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:36.226853+0000) 2024-03-06T15:33:37.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:37.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:37.227078+0000) 2024-03-06T15:33:38.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:38.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:38.227264+0000) 2024-03-06T15:33:39.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:39.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:39.227513+0000) 2024-03-06T15:33:40.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:40.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:40.227761+0000) 2024-03-06T15:33:41.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:41.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:41.227975+0000) 2024-03-06T15:33:42.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:42.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:42.228179+0000) 2024-03-06T15:33:43.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:43.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:43.228404+0000) 2024-03-06T15:33:44.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:44.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:44.228602+0000) 2024-03-06T15:33:45.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:45.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:45.228825+0000) 2024-03-06T15:33:46.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:46.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:46.229027+0000) 2024-03-06T15:33:47.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:47.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:47.229205+0000) 2024-03-06T15:33:48.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:48.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:48.229394+0000) 2024-03-06T15:33:49.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:49.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:49.229634+0000) 2024-03-06T15:33:50.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:50.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:50.229890+0000) 2024-03-06T15:33:51.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:51.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:51.230090+0000) 2024-03-06T15:33:52.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:52.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:52.230304+0000) 2024-03-06T15:33:53.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:53.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:53.230540+0000) 2024-03-06T15:33:54.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:54.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:54.230782+0000) 2024-03-06T15:33:55.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:55.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:55.230979+0000) 2024-03-06T15:33:56.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:56.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:56.231162+0000) 2024-03-06T15:33:56.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:56.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:57.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:57.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:57.231346+0000) 2024-03-06T15:33:58.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:58.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:58.231592+0000) 2024-03-06T15:33:59.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:33:59.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:59.231819+0000) 2024-03-06T15:34:00.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:00.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:00.232013+0000) 2024-03-06T15:34:01.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:01.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:01.232202+0000) 2024-03-06T15:34:02.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:02.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:02.232425+0000) 2024-03-06T15:34:03.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:03.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:03.232703+0000) 2024-03-06T15:34:04.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:04.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:04.232900+0000) 2024-03-06T15:34:05.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:05.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:05.233125+0000) 2024-03-06T15:34:06.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:06.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:06.233417+0000) 2024-03-06T15:34:07.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:07.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:07.233593+0000) 2024-03-06T15:34:08.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:08.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:08.233811+0000) 2024-03-06T15:34:09.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:09.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:09.234042+0000) 2024-03-06T15:34:10.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:10.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:10.234227+0000) 2024-03-06T15:34:11.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:11.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:11.234460+0000) 2024-03-06T15:34:12.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:12.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:12.234634+0000) 2024-03-06T15:34:13.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:13.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:13.234841+0000) 2024-03-06T15:34:14.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:14.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:14.235037+0000) 2024-03-06T15:34:15.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:15.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:15.235228+0000) 2024-03-06T15:34:16.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:16.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:16.235414+0000) 2024-03-06T15:34:17.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:17.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:17.235656+0000) 2024-03-06T15:34:18.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:18.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:18.235936+0000) 2024-03-06T15:34:19.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:19.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:19.236167+0000) 2024-03-06T15:34:20.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:20.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:20.236396+0000) 2024-03-06T15:34:21.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:21.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:21.236626+0000) 2024-03-06T15:34:22.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:22.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:22.236827+0000) 2024-03-06T15:34:23.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:23.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:23.237021+0000) 2024-03-06T15:34:24.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:24.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:24.237252+0000) 2024-03-06T15:34:25.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:25.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:25.237473+0000) 2024-03-06T15:34:26.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:26.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:26.237743+0000) 2024-03-06T15:34:26.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:26.645 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:27.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:27.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:27.237943+0000) 2024-03-06T15:34:28.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:28.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:28.238161+0000) 2024-03-06T15:34:29.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:29.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:29.238389+0000) 2024-03-06T15:34:30.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:30.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:30.238619+0000) 2024-03-06T15:34:31.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:31.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:31.238839+0000) 2024-03-06T15:34:32.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:32.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:32.239022+0000) 2024-03-06T15:34:33.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:33.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:33.239206+0000) 2024-03-06T15:34:34.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:34.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:34.239432+0000) 2024-03-06T15:34:35.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:35.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:35.239660+0000) 2024-03-06T15:34:36.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:36.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:36.239918+0000) 2024-03-06T15:34:37.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:37.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:37.240166+0000) 2024-03-06T15:34:38.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:38.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:38.240430+0000) 2024-03-06T15:34:39.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:39.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:39.240761+0000) 2024-03-06T15:34:40.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:40.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:40.241055+0000) 2024-03-06T15:34:41.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:41.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:41.241333+0000) 2024-03-06T15:34:42.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:42.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:42.241534+0000) 2024-03-06T15:34:43.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:43.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:43.241799+0000) 2024-03-06T15:34:44.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:44.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:44.242025+0000) 2024-03-06T15:34:45.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:45.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:45.242311+0000) 2024-03-06T15:34:46.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:46.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:46.242580+0000) 2024-03-06T15:34:47.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:47.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:47.242855+0000) 2024-03-06T15:34:48.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:48.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:48.243115+0000) 2024-03-06T15:34:49.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:49.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:49.243398+0000) 2024-03-06T15:34:50.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:50.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:50.243748+0000) 2024-03-06T15:34:51.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:51.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:51.243959+0000) 2024-03-06T15:34:52.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:52.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:52.244234+0000) 2024-03-06T15:34:53.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:53.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:53.244517+0000) 2024-03-06T15:34:54.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:54.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:54.244830+0000) 2024-03-06T15:34:55.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:55.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:55.245059+0000) 2024-03-06T15:34:56.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:56.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:56.245268+0000) 2024-03-06T15:34:56.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:56.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:57.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:57.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:57.245498+0000) 2024-03-06T15:34:58.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:58.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:58.245799+0000) 2024-03-06T15:34:59.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:34:59.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:59.245977+0000) 2024-03-06T15:35:00.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:00.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:00.246211+0000) 2024-03-06T15:35:01.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:01.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:01.246438+0000) 2024-03-06T15:35:02.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:02.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:02.246660+0000) 2024-03-06T15:35:03.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:03.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:03.246935+0000) 2024-03-06T15:35:04.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:04.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:04.247168+0000) 2024-03-06T15:35:05.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:05.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:05.247449+0000) 2024-03-06T15:35:06.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:06.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:06.247771+0000) 2024-03-06T15:35:07.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:07.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:07.248041+0000) 2024-03-06T15:35:08.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:08.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:08.248288+0000) 2024-03-06T15:35:09.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:09.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:09.248556+0000) 2024-03-06T15:35:10.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:10.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:10.248859+0000) 2024-03-06T15:35:11.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:11.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:11.249091+0000) 2024-03-06T15:35:12.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:12.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:12.249379+0000) 2024-03-06T15:35:13.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:13.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:13.249609+0000) 2024-03-06T15:35:14.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:14.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:14.249826+0000) 2024-03-06T15:35:15.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:15.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:15.250059+0000) 2024-03-06T15:35:16.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:16.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:16.250276+0000) 2024-03-06T15:35:17.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:17.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:17.250502+0000) 2024-03-06T15:35:18.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:18.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:18.250765+0000) 2024-03-06T15:35:19.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:19.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:19.250919+0000) 2024-03-06T15:35:20.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:20.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:20.251106+0000) 2024-03-06T15:35:21.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:21.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:21.251348+0000) 2024-03-06T15:35:22.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:22.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:22.251566+0000) 2024-03-06T15:35:23.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:23.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:23.251775+0000) 2024-03-06T15:35:24.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:24.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:24.251960+0000) 2024-03-06T15:35:25.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:25.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:25.252157+0000) 2024-03-06T15:35:26.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:26.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:26.252387+0000) 2024-03-06T15:35:26.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:26.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:27.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:27.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:27.252616+0000) 2024-03-06T15:35:28.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:28.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:28.252820+0000) 2024-03-06T15:35:29.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:29.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:29.253017+0000) 2024-03-06T15:35:30.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:30.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:30.253268+0000) 2024-03-06T15:35:31.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:31.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:31.253514+0000) 2024-03-06T15:35:32.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:32.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:32.253793+0000) 2024-03-06T15:35:33.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:33.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:33.254019+0000) 2024-03-06T15:35:34.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:34.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:34.254232+0000) 2024-03-06T15:35:35.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:35.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:35.254472+0000) 2024-03-06T15:35:36.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:36.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:36.254757+0000) 2024-03-06T15:35:37.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:37.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:37.254951+0000) 2024-03-06T15:35:38.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:38.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:38.255177+0000) 2024-03-06T15:35:39.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:39.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:39.255406+0000) 2024-03-06T15:35:40.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:40.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:40.255647+0000) 2024-03-06T15:35:41.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:41.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:41.255939+0000) 2024-03-06T15:35:42.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:42.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:42.256127+0000) 2024-03-06T15:35:43.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:43.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:43.256359+0000) 2024-03-06T15:35:44.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:44.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:44.256586+0000) 2024-03-06T15:35:45.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:45.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:45.256824+0000) 2024-03-06T15:35:46.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:46.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:46.257030+0000) 2024-03-06T15:35:47.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:47.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:47.257315+0000) 2024-03-06T15:35:48.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:48.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:48.257517+0000) 2024-03-06T15:35:49.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:49.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:49.257803+0000) 2024-03-06T15:35:50.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:50.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:50.258040+0000) 2024-03-06T15:35:51.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:51.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:51.258323+0000) 2024-03-06T15:35:52.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:52.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:52.258590+0000) 2024-03-06T15:35:53.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:53.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:53.258864+0000) 2024-03-06T15:35:54.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:54.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:54.259150+0000) 2024-03-06T15:35:55.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:55.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:55.259418+0000) 2024-03-06T15:35:56.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:56.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:56.259721+0000) 2024-03-06T15:35:57.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:57.040 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:57.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:57.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:57.260018+0000) 2024-03-06T15:35:58.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:58.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:58.260293+0000) 2024-03-06T15:35:59.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:35:59.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:59.260576+0000) 2024-03-06T15:36:00.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:00.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:00.260868+0000) 2024-03-06T15:36:01.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:01.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:01.261148+0000) 2024-03-06T15:36:02.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:02.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:02.261427+0000) 2024-03-06T15:36:03.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:03.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:03.261751+0000) 2024-03-06T15:36:04.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:04.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:04.262003+0000) 2024-03-06T15:36:05.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:05.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:05.262236+0000) 2024-03-06T15:36:06.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:06.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:06.262504+0000) 2024-03-06T15:36:07.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:07.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:07.262817+0000) 2024-03-06T15:36:08.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:08.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:08.263100+0000) 2024-03-06T15:36:09.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:09.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:09.263533+0000) 2024-03-06T15:36:10.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:10.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:10.263794+0000) 2024-03-06T15:36:11.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:11.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:11.263992+0000) 2024-03-06T15:36:12.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:12.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:12.264162+0000) 2024-03-06T15:36:13.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:13.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:13.264406+0000) 2024-03-06T15:36:14.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:14.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:14.264631+0000) 2024-03-06T15:36:15.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:15.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:15.264868+0000) 2024-03-06T15:36:16.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:16.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:16.265100+0000) 2024-03-06T15:36:17.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:17.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:17.265323+0000) 2024-03-06T15:36:18.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:18.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:18.265557+0000) 2024-03-06T15:36:19.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:19.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:19.265793+0000) 2024-03-06T15:36:20.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:20.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:20.266030+0000) 2024-03-06T15:36:21.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:21.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:21.266227+0000) 2024-03-06T15:36:22.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:22.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:22.266458+0000) 2024-03-06T15:36:23.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:23.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:23.266616+0000) 2024-03-06T15:36:24.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:24.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:24.266870+0000) 2024-03-06T15:36:25.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:25.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:25.267101+0000) 2024-03-06T15:36:26.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:26.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:26.267319+0000) 2024-03-06T15:36:27.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:27.171 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:27.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:27.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:27.267443+0000) 2024-03-06T15:36:28.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:28.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:28.267696+0000) 2024-03-06T15:36:29.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:29.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:29.267890+0000) 2024-03-06T15:36:30.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:30.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:30.268174+0000) 2024-03-06T15:36:31.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:31.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:31.268424+0000) 2024-03-06T15:36:32.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:32.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:32.268644+0000) 2024-03-06T15:36:33.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:33.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:33.268914+0000) 2024-03-06T15:36:34.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:34.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:34.269148+0000) 2024-03-06T15:36:35.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:35.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:35.269394+0000) 2024-03-06T15:36:36.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:36.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:36.269622+0000) 2024-03-06T15:36:37.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:37.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:37.269853+0000) 2024-03-06T15:36:38.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:38.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:38.270041+0000) 2024-03-06T15:36:39.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:39.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:39.270254+0000) 2024-03-06T15:36:40.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:40.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:40.270505+0000) 2024-03-06T15:36:41.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:41.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:41.270770+0000) 2024-03-06T15:36:42.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:42.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:42.270965+0000) 2024-03-06T15:36:43.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:43.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:43.271203+0000) 2024-03-06T15:36:44.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:44.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:44.271385+0000) 2024-03-06T15:36:45.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:45.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:45.271558+0000) 2024-03-06T15:36:46.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:46.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:46.271801+0000) 2024-03-06T15:36:47.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:47.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:47.272048+0000) 2024-03-06T15:36:48.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:48.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:48.272240+0000) 2024-03-06T15:36:49.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:49.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:49.272470+0000) 2024-03-06T15:36:50.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:50.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:50.272688+0000) 2024-03-06T15:36:51.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:51.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:51.272887+0000) 2024-03-06T15:36:52.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:52.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:52.273111+0000) 2024-03-06T15:36:53.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:53.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:53.273336+0000) 2024-03-06T15:36:54.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:54.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:54.273583+0000) 2024-03-06T15:36:55.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:55.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:55.273785+0000) 2024-03-06T15:36:56.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:56.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:56.274015+0000) 2024-03-06T15:36:57.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:57.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:57.274258+0000) 2024-03-06T15:36:57.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:57.310 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:58.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:58.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:58.274486+0000) 2024-03-06T15:36:59.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:36:59.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:59.274628+0000) 2024-03-06T15:37:00.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:00.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:00.274866+0000) 2024-03-06T15:37:01.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:01.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:01.275061+0000) 2024-03-06T15:37:02.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:02.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:02.275308+0000) 2024-03-06T15:37:03.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:03.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:03.275511+0000) 2024-03-06T15:37:04.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:04.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:04.275801+0000) 2024-03-06T15:37:05.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:05.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:05.276030+0000) 2024-03-06T15:37:06.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:06.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:06.276237+0000) 2024-03-06T15:37:07.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:07.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:07.276467+0000) 2024-03-06T15:37:08.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:08.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:08.276705+0000) 2024-03-06T15:37:09.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:09.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:09.276947+0000) 2024-03-06T15:37:10.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:10.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:10.277181+0000) 2024-03-06T15:37:11.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:11.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:11.277405+0000) 2024-03-06T15:37:12.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:12.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:12.277620+0000) 2024-03-06T15:37:13.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:13.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:13.277847+0000) 2024-03-06T15:37:14.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:14.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:14.278068+0000) 2024-03-06T15:37:15.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:15.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:15.278305+0000) 2024-03-06T15:37:16.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:16.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:16.278464+0000) 2024-03-06T15:37:17.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:17.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:17.278754+0000) 2024-03-06T15:37:18.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:18.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:18.279003+0000) 2024-03-06T15:37:19.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:19.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:19.279196+0000) 2024-03-06T15:37:20.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:20.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:20.279405+0000) 2024-03-06T15:37:21.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:21.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:21.279559+0000) 2024-03-06T15:37:22.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:22.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:22.279803+0000) 2024-03-06T15:37:23.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:23.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:23.280096+0000) 2024-03-06T15:37:24.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:24.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:24.280321+0000) 2024-03-06T15:37:25.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:25.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:25.280560+0000) 2024-03-06T15:37:26.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:26.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:26.280780+0000) 2024-03-06T15:37:27.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:27.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:27.281017+0000) 2024-03-06T15:37:27.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:27.469 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:28.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:28.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:28.281254+0000) 2024-03-06T15:37:29.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:29.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:29.281486+0000) 2024-03-06T15:37:30.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:30.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:30.281776+0000) 2024-03-06T15:37:31.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:31.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:31.282007+0000) 2024-03-06T15:37:32.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:32.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:32.282160+0000) 2024-03-06T15:37:33.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:33.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:33.282394+0000) 2024-03-06T15:37:34.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:34.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:34.282623+0000) 2024-03-06T15:37:35.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:35.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:35.282868+0000) 2024-03-06T15:37:36.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:36.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:36.283101+0000) 2024-03-06T15:37:37.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:37.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:37.283292+0000) 2024-03-06T15:37:38.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:38.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:38.283504+0000) 2024-03-06T15:37:39.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:39.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:39.283786+0000) 2024-03-06T15:37:40.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:40.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:40.284037+0000) 2024-03-06T15:37:41.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:41.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:41.284270+0000) 2024-03-06T15:37:42.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:42.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:42.284493+0000) 2024-03-06T15:37:43.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:43.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:43.284759+0000) 2024-03-06T15:37:44.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:44.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:44.284979+0000) 2024-03-06T15:37:45.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:45.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:45.285180+0000) 2024-03-06T15:37:46.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:46.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:46.285366+0000) 2024-03-06T15:37:47.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:47.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:47.285601+0000) 2024-03-06T15:37:48.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:48.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:48.285825+0000) 2024-03-06T15:37:49.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:49.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:49.286100+0000) 2024-03-06T15:37:50.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:50.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:50.286341+0000) 2024-03-06T15:37:51.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:51.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:51.286534+0000) 2024-03-06T15:37:52.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:52.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:52.286775+0000) 2024-03-06T15:37:53.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:53.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:53.287003+0000) 2024-03-06T15:37:54.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:54.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:54.287250+0000) 2024-03-06T15:37:55.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:55.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:55.287467+0000) 2024-03-06T15:37:56.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:56.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:56.287749+0000) 2024-03-06T15:37:57.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:57.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:57.287977+0000) 2024-03-06T15:37:57.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:57.630 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:58.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:58.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:58.288203+0000) 2024-03-06T15:37:59.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:37:59.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:59.288422+0000) 2024-03-06T15:38:00.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:00.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:00.288723+0000) 2024-03-06T15:38:01.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:01.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:01.288989+0000) 2024-03-06T15:38:02.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:02.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:02.289229+0000) 2024-03-06T15:38:03.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:03.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:03.289441+0000) 2024-03-06T15:38:04.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:04.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:04.289556+0000) 2024-03-06T15:38:05.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:05.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:05.289760+0000) 2024-03-06T15:38:06.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:06.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:06.289981+0000) 2024-03-06T15:38:07.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:07.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:07.290193+0000) 2024-03-06T15:38:08.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:08.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:08.290415+0000) 2024-03-06T15:38:09.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:09.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:09.290650+0000) 2024-03-06T15:38:10.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:10.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:10.290940+0000) 2024-03-06T15:38:11.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:11.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:11.291175+0000) 2024-03-06T15:38:12.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:12.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:12.291329+0000) 2024-03-06T15:38:13.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:13.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:13.291612+0000) 2024-03-06T15:38:14.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:14.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:14.291836+0000) 2024-03-06T15:38:15.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:15.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:15.292090+0000) 2024-03-06T15:38:16.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:16.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:16.292311+0000) 2024-03-06T15:38:17.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:17.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:17.292588+0000) 2024-03-06T15:38:18.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:18.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:18.292788+0000) 2024-03-06T15:38:19.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:19.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:19.292994+0000) 2024-03-06T15:38:20.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:20.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:20.293220+0000) 2024-03-06T15:38:21.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:21.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:21.293459+0000) 2024-03-06T15:38:22.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:22.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:22.293733+0000) 2024-03-06T15:38:23.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:23.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:23.293960+0000) 2024-03-06T15:38:24.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:24.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:24.294171+0000) 2024-03-06T15:38:25.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:25.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:25.294429+0000) 2024-03-06T15:38:26.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:26.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:26.294656+0000) 2024-03-06T15:38:27.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:27.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:27.294923+0000) 2024-03-06T15:38:27.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:27.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:28.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:28.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:28.295148+0000) 2024-03-06T15:38:29.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:29.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:29.295310+0000) 2024-03-06T15:38:30.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:30.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:30.295617+0000) 2024-03-06T15:38:31.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:31.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:31.295903+0000) 2024-03-06T15:38:32.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:32.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:32.296142+0000) 2024-03-06T15:38:33.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:33.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:33.296330+0000) 2024-03-06T15:38:34.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:34.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:34.296520+0000) 2024-03-06T15:38:35.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:35.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:35.296786+0000) 2024-03-06T15:38:36.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:36.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:36.297021+0000) 2024-03-06T15:38:37.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:37.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:37.297242+0000) 2024-03-06T15:38:38.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:38.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:38.297460+0000) 2024-03-06T15:38:39.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:39.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:39.297718+0000) 2024-03-06T15:38:40.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:40.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:40.297967+0000) 2024-03-06T15:38:41.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:41.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:41.298200+0000) 2024-03-06T15:38:42.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:42.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:42.298425+0000) 2024-03-06T15:38:43.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:43.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:43.298699+0000) 2024-03-06T15:38:44.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:44.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:44.298894+0000) 2024-03-06T15:38:45.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:45.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:45.299125+0000) 2024-03-06T15:38:46.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:46.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:46.299335+0000) 2024-03-06T15:38:47.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:47.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:47.299556+0000) 2024-03-06T15:38:48.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:48.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:48.299776+0000) 2024-03-06T15:38:49.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:49.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:49.299992+0000) 2024-03-06T15:38:50.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:50.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:50.300195+0000) 2024-03-06T15:38:51.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:51.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:51.300455+0000) 2024-03-06T15:38:52.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:52.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:52.300613+0000) 2024-03-06T15:38:53.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:53.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:53.300849+0000) 2024-03-06T15:38:54.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:54.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:54.301082+0000) 2024-03-06T15:38:55.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:55.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:55.301234+0000) 2024-03-06T15:38:56.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:56.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:56.301440+0000) 2024-03-06T15:38:57.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:57.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:57.301699+0000) 2024-03-06T15:38:57.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:57.982 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:58.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:58.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:58.301916+0000) 2024-03-06T15:38:59.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:38:59.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:59.302161+0000) 2024-03-06T15:39:00.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:00.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:00.302397+0000) 2024-03-06T15:39:01.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:01.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:01.302557+0000) 2024-03-06T15:39:02.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:02.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:02.302799+0000) 2024-03-06T15:39:03.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:03.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:03.303040+0000) 2024-03-06T15:39:04.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:04.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:04.303244+0000) 2024-03-06T15:39:05.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:05.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:05.303515+0000) 2024-03-06T15:39:06.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:06.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:06.303763+0000) 2024-03-06T15:39:07.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:07.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:07.303964+0000) 2024-03-06T15:39:08.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:08.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:08.304188+0000) 2024-03-06T15:39:09.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:09.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:09.304416+0000) 2024-03-06T15:39:10.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:10.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:10.304590+0000) 2024-03-06T15:39:11.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:11.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:11.304829+0000) 2024-03-06T15:39:12.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:12.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:12.305053+0000) 2024-03-06T15:39:13.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:13.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:13.305276+0000) 2024-03-06T15:39:14.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:14.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:14.305498+0000) 2024-03-06T15:39:15.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:15.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:15.305783+0000) 2024-03-06T15:39:16.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:16.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:16.305979+0000) 2024-03-06T15:39:17.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:17.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:17.306202+0000) 2024-03-06T15:39:18.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:18.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:18.306415+0000) 2024-03-06T15:39:19.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:19.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:19.306641+0000) 2024-03-06T15:39:20.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:20.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:20.306912+0000) 2024-03-06T15:39:21.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:21.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:21.307143+0000) 2024-03-06T15:39:22.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:22.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:22.307338+0000) 2024-03-06T15:39:23.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:23.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:23.307575+0000) 2024-03-06T15:39:24.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:24.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:24.307798+0000) 2024-03-06T15:39:25.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:25.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:25.308015+0000) 2024-03-06T15:39:26.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:26.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:26.308251+0000) 2024-03-06T15:39:27.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:27.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:27.308410+0000) 2024-03-06T15:39:28.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:28.146 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:28.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:28.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:28.308649+0000) 2024-03-06T15:39:29.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:29.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:29.308925+0000) 2024-03-06T15:39:30.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:30.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:30.309167+0000) 2024-03-06T15:39:31.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:31.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:31.309405+0000) 2024-03-06T15:39:32.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:32.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:32.309628+0000) 2024-03-06T15:39:33.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:33.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:33.309788+0000) 2024-03-06T15:39:34.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:34.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:34.310037+0000) 2024-03-06T15:39:35.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:35.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:35.310274+0000) 2024-03-06T15:39:36.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:36.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:36.310457+0000) 2024-03-06T15:39:37.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:37.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:37.310720+0000) 2024-03-06T15:39:38.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:38.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:38.310944+0000) 2024-03-06T15:39:39.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:39.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:39.311099+0000) 2024-03-06T15:39:40.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:40.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:40.311360+0000) 2024-03-06T15:39:41.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:41.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:41.311580+0000) 2024-03-06T15:39:42.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:42.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:42.311790+0000) 2024-03-06T15:39:43.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:43.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:43.312041+0000) 2024-03-06T15:39:44.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:44.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:44.312219+0000) 2024-03-06T15:39:45.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:45.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:45.312456+0000) 2024-03-06T15:39:46.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:46.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:46.312701+0000) 2024-03-06T15:39:47.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:47.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:47.312927+0000) 2024-03-06T15:39:48.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:48.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:48.313148+0000) 2024-03-06T15:39:49.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:49.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:49.313342+0000) 2024-03-06T15:39:50.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:50.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:50.313580+0000) 2024-03-06T15:39:51.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:51.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:51.313822+0000) 2024-03-06T15:39:52.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:52.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:52.314039+0000) 2024-03-06T15:39:53.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:53.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:53.314250+0000) 2024-03-06T15:39:54.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:54.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:54.314476+0000) 2024-03-06T15:39:55.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:55.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:55.314633+0000) 2024-03-06T15:39:56.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:56.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:56.314900+0000) 2024-03-06T15:39:57.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:57.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:57.315090+0000) 2024-03-06T15:39:58.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:58.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:58.315372+0000) 2024-03-06T15:39:58.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:58.361 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:59.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:39:59.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:59.315523+0000) 2024-03-06T15:40:00.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:00.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:00.315769+0000) 2024-03-06T15:40:01.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:01.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:01.315998+0000) 2024-03-06T15:40:02.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:02.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:02.316150+0000) 2024-03-06T15:40:03.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:03.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:03.316383+0000) 2024-03-06T15:40:04.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:04.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:04.316569+0000) 2024-03-06T15:40:05.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:05.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:05.316813+0000) 2024-03-06T15:40:06.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:06.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:06.317043+0000) 2024-03-06T15:40:07.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:07.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:07.317231+0000) 2024-03-06T15:40:08.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:08.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:08.317435+0000) 2024-03-06T15:40:09.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:09.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:09.317635+0000) 2024-03-06T15:40:10.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:10.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:10.317865+0000) 2024-03-06T15:40:11.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:11.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:11.318056+0000) 2024-03-06T15:40:12.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:12.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:12.318292+0000) 2024-03-06T15:40:13.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:13.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:13.318606+0000) 2024-03-06T15:40:14.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:14.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:14.318909+0000) 2024-03-06T15:40:15.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:15.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:15.319143+0000) 2024-03-06T15:40:16.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:16.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:16.319369+0000) 2024-03-06T15:40:17.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:17.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:17.319549+0000) 2024-03-06T15:40:18.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:18.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:18.319776+0000) 2024-03-06T15:40:19.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:19.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:19.319977+0000) 2024-03-06T15:40:20.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:20.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:20.320167+0000) 2024-03-06T15:40:21.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:21.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:21.320388+0000) 2024-03-06T15:40:22.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:22.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:22.320560+0000) 2024-03-06T15:40:23.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:23.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:23.320815+0000) 2024-03-06T15:40:24.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:24.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:24.321071+0000) 2024-03-06T15:40:25.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:25.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:25.321308+0000) 2024-03-06T15:40:26.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:26.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:26.321584+0000) 2024-03-06T15:40:27.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:27.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:27.321842+0000) 2024-03-06T15:40:28.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:28.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:28.322091+0000) 2024-03-06T15:40:28.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:28.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:29.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:29.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:29.322320+0000) 2024-03-06T15:40:30.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:30.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:30.322537+0000) 2024-03-06T15:40:31.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:31.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:31.322819+0000) 2024-03-06T15:40:32.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:32.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:32.323005+0000) 2024-03-06T15:40:33.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:33.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:33.323229+0000) 2024-03-06T15:40:34.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:34.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:34.323516+0000) 2024-03-06T15:40:35.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:35.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:35.323825+0000) 2024-03-06T15:40:36.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:36.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:36.324011+0000) 2024-03-06T15:40:37.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:37.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:37.324209+0000) 2024-03-06T15:40:38.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:38.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:38.324449+0000) 2024-03-06T15:40:39.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:39.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:39.324635+0000) 2024-03-06T15:40:40.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:40.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:40.324892+0000) 2024-03-06T15:40:41.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:41.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:41.325163+0000) 2024-03-06T15:40:42.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:42.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:42.325438+0000) 2024-03-06T15:40:43.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:43.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:43.325715+0000) 2024-03-06T15:40:44.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:44.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:44.325917+0000) 2024-03-06T15:40:45.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:45.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:45.326093+0000) 2024-03-06T15:40:46.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:46.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:46.326421+0000) 2024-03-06T15:40:47.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:47.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:47.326746+0000) 2024-03-06T15:40:48.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:48.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:48.326967+0000) 2024-03-06T15:40:49.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:49.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:49.327244+0000) 2024-03-06T15:40:50.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:50.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:50.327444+0000) 2024-03-06T15:40:51.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:51.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:51.327696+0000) 2024-03-06T15:40:52.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:52.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:52.327915+0000) 2024-03-06T15:40:53.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:53.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:53.328096+0000) 2024-03-06T15:40:54.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:54.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:54.328278+0000) 2024-03-06T15:40:55.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:55.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:55.328505+0000) 2024-03-06T15:40:56.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:56.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:56.328772+0000) 2024-03-06T15:40:57.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:57.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:57.328999+0000) 2024-03-06T15:40:58.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:58.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:58.329179+0000) 2024-03-06T15:40:58.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:58.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:59.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:40:59.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:59.329392+0000) 2024-03-06T15:41:00.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:00.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:00.329623+0000) 2024-03-06T15:41:01.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:01.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:01.329849+0000) 2024-03-06T15:41:02.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:02.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:02.330032+0000) 2024-03-06T15:41:03.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:03.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:03.330255+0000) 2024-03-06T15:41:04.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:04.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:04.330478+0000) 2024-03-06T15:41:05.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:05.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:05.330741+0000) 2024-03-06T15:41:06.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:06.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:06.330973+0000) 2024-03-06T15:41:07.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:07.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:07.331176+0000) 2024-03-06T15:41:08.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:08.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:08.331422+0000) 2024-03-06T15:41:09.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:09.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:09.331642+0000) 2024-03-06T15:41:10.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:10.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:10.331892+0000) 2024-03-06T15:41:11.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:11.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:11.332118+0000) 2024-03-06T15:41:12.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:12.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:12.332334+0000) 2024-03-06T15:41:13.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:13.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:13.332564+0000) 2024-03-06T15:41:14.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:14.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:14.332787+0000) 2024-03-06T15:41:15.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:15.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:15.333016+0000) 2024-03-06T15:41:16.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:16.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:16.333235+0000) 2024-03-06T15:41:17.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:17.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:17.333407+0000) 2024-03-06T15:41:18.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:18.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:18.333631+0000) 2024-03-06T15:41:19.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:19.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:19.333908+0000) 2024-03-06T15:41:20.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:20.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:20.334135+0000) 2024-03-06T15:41:21.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:21.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:21.334373+0000) 2024-03-06T15:41:22.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:22.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:22.334594+0000) 2024-03-06T15:41:23.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:23.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:23.334783+0000) 2024-03-06T15:41:24.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:24.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:24.334956+0000) 2024-03-06T15:41:25.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:25.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:25.335143+0000) 2024-03-06T15:41:26.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:26.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:26.335366+0000) 2024-03-06T15:41:27.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:27.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:27.335588+0000) 2024-03-06T15:41:28.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:28.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:28.335844+0000) 2024-03-06T15:41:29.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:29.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:29.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:29.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:29.336064+0000) 2024-03-06T15:41:30.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:30.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:30.336300+0000) 2024-03-06T15:41:31.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:31.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:31.336534+0000) 2024-03-06T15:41:32.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:32.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:32.336791+0000) 2024-03-06T15:41:33.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:33.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:33.336978+0000) 2024-03-06T15:41:34.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:34.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:34.337254+0000) 2024-03-06T15:41:35.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:35.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:35.337473+0000) 2024-03-06T15:41:36.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:36.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:36.337745+0000) 2024-03-06T15:41:37.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:37.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:37.337937+0000) 2024-03-06T15:41:38.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:38.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:38.338136+0000) 2024-03-06T15:41:39.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:39.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:39.338379+0000) 2024-03-06T15:41:40.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:40.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:40.338609+0000) 2024-03-06T15:41:41.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:41.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:41.338789+0000) 2024-03-06T15:41:42.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:42.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:42.338992+0000) 2024-03-06T15:41:43.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:43.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:43.339182+0000) 2024-03-06T15:41:44.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:44.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:44.339387+0000) 2024-03-06T15:41:45.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:45.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:45.339614+0000) 2024-03-06T15:41:46.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:46.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:46.339817+0000) 2024-03-06T15:41:47.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:47.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:47.340040+0000) 2024-03-06T15:41:48.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:48.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:48.340218+0000) 2024-03-06T15:41:49.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:49.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:49.340463+0000) 2024-03-06T15:41:50.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:50.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:50.340778+0000) 2024-03-06T15:41:51.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:51.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:51.341029+0000) 2024-03-06T15:41:52.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:52.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:52.341227+0000) 2024-03-06T15:41:53.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:53.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:53.341445+0000) 2024-03-06T15:41:54.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:54.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:54.341712+0000) 2024-03-06T15:41:55.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:55.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:55.341952+0000) 2024-03-06T15:41:56.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:56.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:56.342151+0000) 2024-03-06T15:41:57.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:57.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:57.342371+0000) 2024-03-06T15:41:58.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:58.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:58.342592+0000) 2024-03-06T15:41:59.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:59.243 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:59.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:41:59.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:59.342827+0000) 2024-03-06T15:42:00.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:00.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:00.343063+0000) 2024-03-06T15:42:01.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:01.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:01.343298+0000) 2024-03-06T15:42:02.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:02.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:02.343515+0000) 2024-03-06T15:42:03.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:03.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:03.343749+0000) 2024-03-06T15:42:04.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:04.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:04.343936+0000) 2024-03-06T15:42:05.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:05.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:05.344114+0000) 2024-03-06T15:42:06.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:06.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:06.344367+0000) 2024-03-06T15:42:07.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:07.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:07.344606+0000) 2024-03-06T15:42:08.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:08.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:08.344842+0000) 2024-03-06T15:42:09.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:09.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:09.345083+0000) 2024-03-06T15:42:10.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:10.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:10.345330+0000) 2024-03-06T15:42:11.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:11.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:11.345575+0000) 2024-03-06T15:42:12.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:12.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:12.345810+0000) 2024-03-06T15:42:13.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:13.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:13.345999+0000) 2024-03-06T15:42:14.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:14.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:14.346253+0000) 2024-03-06T15:42:15.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:15.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:15.346443+0000) 2024-03-06T15:42:16.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:16.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:16.346638+0000) 2024-03-06T15:42:17.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:17.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:17.346909+0000) 2024-03-06T15:42:18.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:18.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:18.347220+0000) 2024-03-06T15:42:19.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:19.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:19.347425+0000) 2024-03-06T15:42:20.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:20.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:20.347707+0000) 2024-03-06T15:42:21.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:21.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:21.347958+0000) 2024-03-06T15:42:22.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:22.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:22.348148+0000) 2024-03-06T15:42:23.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:23.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:23.348378+0000) 2024-03-06T15:42:24.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:24.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:24.348616+0000) 2024-03-06T15:42:25.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:25.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:25.348864+0000) 2024-03-06T15:42:26.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:26.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:26.349113+0000) 2024-03-06T15:42:27.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:27.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:27.349350+0000) 2024-03-06T15:42:28.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:28.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:28.349545+0000) 2024-03-06T15:42:29.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:29.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:29.349779+0000) 2024-03-06T15:42:29.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:29.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:30.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:30.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:30.349976+0000) 2024-03-06T15:42:31.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:31.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:31.350214+0000) 2024-03-06T15:42:32.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:32.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:32.350434+0000) 2024-03-06T15:42:33.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:33.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:33.350710+0000) 2024-03-06T15:42:34.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:34.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:34.350901+0000) 2024-03-06T15:42:35.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:35.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:35.351087+0000) 2024-03-06T15:42:36.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:36.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:36.351316+0000) 2024-03-06T15:42:37.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:37.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:37.351531+0000) 2024-03-06T15:42:38.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:38.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:38.351774+0000) 2024-03-06T15:42:39.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:39.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:39.352021+0000) 2024-03-06T15:42:40.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:40.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:40.352259+0000) 2024-03-06T15:42:41.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:41.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:41.352491+0000) 2024-03-06T15:42:42.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:42.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:42.352739+0000) 2024-03-06T15:42:43.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:43.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:43.352948+0000) 2024-03-06T15:42:44.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:44.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:44.353127+0000) 2024-03-06T15:42:45.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:45.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:45.353348+0000) 2024-03-06T15:42:46.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:46.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:46.353577+0000) 2024-03-06T15:42:47.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:47.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:47.353799+0000) 2024-03-06T15:42:48.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:48.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:48.353980+0000) 2024-03-06T15:42:49.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:49.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:49.354209+0000) 2024-03-06T15:42:50.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:50.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:50.354392+0000) 2024-03-06T15:42:51.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:51.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:51.354581+0000) 2024-03-06T15:42:52.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:52.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:52.354800+0000) 2024-03-06T15:42:53.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:53.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:53.355034+0000) 2024-03-06T15:42:54.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:54.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:54.355250+0000) 2024-03-06T15:42:55.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:55.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:55.355483+0000) 2024-03-06T15:42:56.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:56.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:56.355701+0000) 2024-03-06T15:42:57.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:57.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:57.355889+0000) 2024-03-06T15:42:58.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:58.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:58.356081+0000) 2024-03-06T15:42:59.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:42:59.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:59.356277+0000) 2024-03-06T15:42:59.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:59.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:00.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:00.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:00.356524+0000) 2024-03-06T15:43:01.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:01.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:01.356761+0000) 2024-03-06T15:43:02.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:02.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:02.356982+0000) 2024-03-06T15:43:03.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:03.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:03.357189+0000) 2024-03-06T15:43:04.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:04.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:04.357425+0000) 2024-03-06T15:43:05.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:05.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:05.357658+0000) 2024-03-06T15:43:06.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:06.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:06.357929+0000) 2024-03-06T15:43:07.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:07.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:07.358173+0000) 2024-03-06T15:43:08.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:08.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:08.358369+0000) 2024-03-06T15:43:09.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:09.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:09.358611+0000) 2024-03-06T15:43:10.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:10.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:10.358912+0000) 2024-03-06T15:43:11.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:11.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:11.359139+0000) 2024-03-06T15:43:12.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:12.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:12.359324+0000) 2024-03-06T15:43:13.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:13.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:13.359559+0000) 2024-03-06T15:43:14.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:14.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:14.359787+0000) 2024-03-06T15:43:15.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:15.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:15.360069+0000) 2024-03-06T15:43:16.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:16.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:16.360302+0000) 2024-03-06T15:43:17.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:17.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:17.360514+0000) 2024-03-06T15:43:18.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:18.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:18.360799+0000) 2024-03-06T15:43:19.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:19.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:19.360984+0000) 2024-03-06T15:43:20.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:20.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:20.361213+0000) 2024-03-06T15:43:21.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:21.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:21.361453+0000) 2024-03-06T15:43:22.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:22.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:22.361648+0000) 2024-03-06T15:43:23.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:23.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:23.361904+0000) 2024-03-06T15:43:24.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:24.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:24.362096+0000) 2024-03-06T15:43:25.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:25.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:25.362316+0000) 2024-03-06T15:43:26.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:26.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:26.362504+0000) 2024-03-06T15:43:27.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:27.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:27.362783+0000) 2024-03-06T15:43:28.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:28.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:28.363047+0000) 2024-03-06T15:43:29.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:29.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:29.363302+0000) 2024-03-06T15:43:29.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:29.888 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:30.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:30.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:30.363532+0000) 2024-03-06T15:43:31.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:31.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:31.363809+0000) 2024-03-06T15:43:32.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:32.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:32.363948+0000) 2024-03-06T15:43:33.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:33.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:33.364175+0000) 2024-03-06T15:43:34.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:34.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:34.364413+0000) 2024-03-06T15:43:35.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:35.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:35.364628+0000) 2024-03-06T15:43:36.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:36.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:36.364873+0000) 2024-03-06T15:43:37.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:37.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:37.365113+0000) 2024-03-06T15:43:38.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:38.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:38.365301+0000) 2024-03-06T15:43:39.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:39.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:39.365527+0000) 2024-03-06T15:43:40.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:40.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:40.365785+0000) 2024-03-06T15:43:41.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:41.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:41.366018+0000) 2024-03-06T15:43:42.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:42.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:42.366242+0000) 2024-03-06T15:43:43.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:43.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:43.366473+0000) 2024-03-06T15:43:44.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:44.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:44.366734+0000) 2024-03-06T15:43:45.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:45.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:45.366931+0000) 2024-03-06T15:43:46.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:46.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:46.367181+0000) 2024-03-06T15:43:47.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:47.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:47.367405+0000) 2024-03-06T15:43:48.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:48.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:48.367640+0000) 2024-03-06T15:43:49.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:49.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:49.367882+0000) 2024-03-06T15:43:50.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:50.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:50.368079+0000) 2024-03-06T15:43:51.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:51.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:51.368319+0000) 2024-03-06T15:43:52.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:52.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:52.368500+0000) 2024-03-06T15:43:53.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:53.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:53.368800+0000) 2024-03-06T15:43:54.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:54.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:54.369039+0000) 2024-03-06T15:43:55.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:55.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:55.369233+0000) 2024-03-06T15:43:56.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:56.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:56.369477+0000) 2024-03-06T15:43:57.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:57.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:57.369759+0000) 2024-03-06T15:43:58.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:58.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:58.369912+0000) 2024-03-06T15:43:59.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:43:59.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:59.370129+0000) 2024-03-06T15:44:00.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:00.059 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:00.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:00.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:00.370384+0000) 2024-03-06T15:44:01.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:01.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:01.370624+0000) 2024-03-06T15:44:02.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:02.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:02.370829+0000) 2024-03-06T15:44:03.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:03.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:03.371073+0000) 2024-03-06T15:44:04.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:04.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:04.371295+0000) 2024-03-06T15:44:05.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:05.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:05.371520+0000) 2024-03-06T15:44:06.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:06.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:06.371803+0000) 2024-03-06T15:44:07.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:07.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:07.371983+0000) 2024-03-06T15:44:08.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:08.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:08.372178+0000) 2024-03-06T15:44:09.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:09.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:09.372377+0000) 2024-03-06T15:44:10.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:10.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:10.372615+0000) 2024-03-06T15:44:11.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:11.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:11.372835+0000) 2024-03-06T15:44:12.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:12.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:12.373009+0000) 2024-03-06T15:44:13.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:13.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:13.373195+0000) 2024-03-06T15:44:14.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:14.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:14.373420+0000) 2024-03-06T15:44:15.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:15.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:15.373642+0000) 2024-03-06T15:44:16.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:16.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:16.373893+0000) 2024-03-06T15:44:17.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:17.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:17.374084+0000) 2024-03-06T15:44:18.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:18.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:18.374266+0000) 2024-03-06T15:44:19.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:19.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:19.374509+0000) 2024-03-06T15:44:20.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:20.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:20.374818+0000) 2024-03-06T15:44:21.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:21.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:21.375044+0000) 2024-03-06T15:44:22.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:22.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:22.375248+0000) 2024-03-06T15:44:23.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:23.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:23.375492+0000) 2024-03-06T15:44:24.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:24.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:24.375776+0000) 2024-03-06T15:44:25.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:25.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:25.375982+0000) 2024-03-06T15:44:26.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:26.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:26.376221+0000) 2024-03-06T15:44:27.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:27.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:27.376415+0000) 2024-03-06T15:44:28.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:28.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:28.376643+0000) 2024-03-06T15:44:29.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:29.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:29.376899+0000) 2024-03-06T15:44:30.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:30.263 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:30.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:30.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:30.377082+0000) 2024-03-06T15:44:31.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:31.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:31.377344+0000) 2024-03-06T15:44:32.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:32.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:32.377628+0000) 2024-03-06T15:44:33.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:33.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:33.377940+0000) 2024-03-06T15:44:34.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:34.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:34.378177+0000) 2024-03-06T15:44:35.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:35.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:35.378410+0000) 2024-03-06T15:44:36.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:36.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:36.378649+0000) 2024-03-06T15:44:37.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:37.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:37.378907+0000) 2024-03-06T15:44:38.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:38.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:38.379082+0000) 2024-03-06T15:44:39.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:39.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:39.379278+0000) 2024-03-06T15:44:40.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:40.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:40.379525+0000) 2024-03-06T15:44:41.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:41.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:41.379814+0000) 2024-03-06T15:44:42.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:42.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:42.380045+0000) 2024-03-06T15:44:43.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:43.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:43.380219+0000) 2024-03-06T15:44:44.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:44.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:44.380436+0000) 2024-03-06T15:44:45.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:45.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:45.380659+0000) 2024-03-06T15:44:46.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:46.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:46.380902+0000) 2024-03-06T15:44:47.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:47.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:47.381091+0000) 2024-03-06T15:44:48.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:48.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:48.381269+0000) 2024-03-06T15:44:49.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:49.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:49.381493+0000) 2024-03-06T15:44:50.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:50.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:50.381703+0000) 2024-03-06T15:44:51.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:51.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:51.381946+0000) 2024-03-06T15:44:52.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:52.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:52.382130+0000) 2024-03-06T15:44:53.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:53.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:53.382376+0000) 2024-03-06T15:44:54.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:54.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:54.382624+0000) 2024-03-06T15:44:55.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:55.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:55.382845+0000) 2024-03-06T15:44:56.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:56.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:56.383000+0000) 2024-03-06T15:44:57.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:57.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:57.383272+0000) 2024-03-06T15:44:58.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:58.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:58.383510+0000) 2024-03-06T15:44:59.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:44:59.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:59.383825+0000) 2024-03-06T15:45:00.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:00.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:00.384119+0000) 2024-03-06T15:45:00.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:00.483 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:01.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:01.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:01.384325+0000) 2024-03-06T15:45:02.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:02.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:02.384557+0000) 2024-03-06T15:45:03.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:03.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:03.384785+0000) 2024-03-06T15:45:04.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:04.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:04.384975+0000) 2024-03-06T15:45:05.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:05.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:05.385213+0000) 2024-03-06T15:45:06.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:06.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:06.385445+0000) 2024-03-06T15:45:07.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:07.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:07.385710+0000) 2024-03-06T15:45:08.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:08.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:08.385905+0000) 2024-03-06T15:45:09.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:09.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:09.386111+0000) 2024-03-06T15:45:10.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:10.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:10.386356+0000) 2024-03-06T15:45:11.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:11.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:11.386590+0000) 2024-03-06T15:45:12.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:12.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:12.386827+0000) 2024-03-06T15:45:13.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:13.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:13.387042+0000) 2024-03-06T15:45:14.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:14.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:14.387229+0000) 2024-03-06T15:45:15.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:15.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:15.387459+0000) 2024-03-06T15:45:16.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:16.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:16.387721+0000) 2024-03-06T15:45:17.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:17.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:17.387901+0000) 2024-03-06T15:45:18.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:18.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:18.388098+0000) 2024-03-06T15:45:19.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:19.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:19.388317+0000) 2024-03-06T15:45:20.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:20.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:20.388546+0000) 2024-03-06T15:45:21.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:21.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:21.388778+0000) 2024-03-06T15:45:22.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:22.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:22.388966+0000) 2024-03-06T15:45:23.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:23.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:23.389157+0000) 2024-03-06T15:45:24.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:24.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:24.389381+0000) 2024-03-06T15:45:25.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:25.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:25.389601+0000) 2024-03-06T15:45:26.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:26.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:26.389836+0000) 2024-03-06T15:45:27.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:27.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:27.390031+0000) 2024-03-06T15:45:28.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:28.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:28.390220+0000) 2024-03-06T15:45:29.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:29.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:29.390451+0000) 2024-03-06T15:45:30.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:30.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:30.390740+0000) 2024-03-06T15:45:30.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:30.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:31.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:31.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:31.390930+0000) 2024-03-06T15:45:32.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:32.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:32.391141+0000) 2024-03-06T15:45:33.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:33.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:33.391362+0000) 2024-03-06T15:45:34.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:34.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:34.391587+0000) 2024-03-06T15:45:35.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:35.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:35.391820+0000) 2024-03-06T15:45:36.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:36.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:36.392021+0000) 2024-03-06T15:45:37.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:37.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:37.392248+0000) 2024-03-06T15:45:38.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:38.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:38.392471+0000) 2024-03-06T15:45:39.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:39.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:39.392754+0000) 2024-03-06T15:45:40.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:40.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:40.392958+0000) 2024-03-06T15:45:41.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:41.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:41.393146+0000) 2024-03-06T15:45:42.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:42.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:42.393330+0000) 2024-03-06T15:45:43.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:43.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:43.393526+0000) 2024-03-06T15:45:44.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:44.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:44.393754+0000) 2024-03-06T15:45:45.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:45.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:45.394042+0000) 2024-03-06T15:45:46.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:46.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:46.394255+0000) 2024-03-06T15:45:47.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:47.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:47.394426+0000) 2024-03-06T15:45:48.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:48.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:48.394619+0000) 2024-03-06T15:45:49.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:49.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:49.394813+0000) 2024-03-06T15:45:50.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:50.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:50.395002+0000) 2024-03-06T15:45:51.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:51.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:51.395201+0000) 2024-03-06T15:45:52.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:52.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:52.395421+0000) 2024-03-06T15:45:53.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:53.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:53.395650+0000) 2024-03-06T15:45:54.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:54.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:54.395877+0000) 2024-03-06T15:45:55.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:55.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:55.396102+0000) 2024-03-06T15:45:56.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:56.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:56.396327+0000) 2024-03-06T15:45:57.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:57.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:57.396497+0000) 2024-03-06T15:45:58.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:58.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:58.396777+0000) 2024-03-06T15:45:59.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:45:59.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:59.396970+0000) 2024-03-06T15:46:00.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:00.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:00.397214+0000) 2024-03-06T15:46:00.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:00.874 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:01.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:01.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:01.397460+0000) 2024-03-06T15:46:02.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:02.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:02.397657+0000) 2024-03-06T15:46:03.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:03.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:03.397908+0000) 2024-03-06T15:46:04.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:04.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:04.398100+0000) 2024-03-06T15:46:05.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:05.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:05.398277+0000) 2024-03-06T15:46:06.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:06.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:06.398526+0000) 2024-03-06T15:46:07.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:07.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:07.398785+0000) 2024-03-06T15:46:08.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:08.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:08.399032+0000) 2024-03-06T15:46:09.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:09.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:09.399277+0000) 2024-03-06T15:46:10.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:10.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:10.399527+0000) 2024-03-06T15:46:11.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:11.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:11.399796+0000) 2024-03-06T15:46:12.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:12.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:12.399994+0000) 2024-03-06T15:46:13.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:13.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:13.400232+0000) 2024-03-06T15:46:14.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:14.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:14.400452+0000) 2024-03-06T15:46:15.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:15.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:15.400715+0000) 2024-03-06T15:46:16.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:16.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:16.400926+0000) 2024-03-06T15:46:17.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:17.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:17.401169+0000) 2024-03-06T15:46:18.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:18.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:18.401377+0000) 2024-03-06T15:46:19.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:19.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:19.401591+0000) 2024-03-06T15:46:20.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:20.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:20.401810+0000) 2024-03-06T15:46:21.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:21.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:21.402018+0000) 2024-03-06T15:46:22.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:22.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:22.402208+0000) 2024-03-06T15:46:23.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:23.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:23.402386+0000) 2024-03-06T15:46:24.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:24.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:24.402605+0000) 2024-03-06T15:46:25.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:25.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:25.402776+0000) 2024-03-06T15:46:26.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:26.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:26.402972+0000) 2024-03-06T15:46:27.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:27.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:27.403199+0000) 2024-03-06T15:46:28.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:28.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:28.403383+0000) 2024-03-06T15:46:29.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:29.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:29.403612+0000) 2024-03-06T15:46:30.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:30.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:30.403886+0000) 2024-03-06T15:46:31.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:31.094 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:31.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:31.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:31.404105+0000) 2024-03-06T15:46:32.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:32.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:32.404344+0000) 2024-03-06T15:46:33.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:33.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:33.404597+0000) 2024-03-06T15:46:34.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:34.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:34.404833+0000) 2024-03-06T15:46:35.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:35.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:35.405052+0000) 2024-03-06T15:46:36.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:36.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:36.405240+0000) 2024-03-06T15:46:37.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:37.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:37.405422+0000) 2024-03-06T15:46:38.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:38.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:38.405718+0000) 2024-03-06T15:46:39.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:39.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:39.405961+0000) 2024-03-06T15:46:40.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:40.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:40.406193+0000) 2024-03-06T15:46:41.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:41.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:41.406377+0000) 2024-03-06T15:46:42.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:42.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:42.406551+0000) 2024-03-06T15:46:43.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:43.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:43.406782+0000) 2024-03-06T15:46:44.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:44.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:44.407004+0000) 2024-03-06T15:46:45.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:45.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:45.407203+0000) 2024-03-06T15:46:46.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:46.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:46.407368+0000) 2024-03-06T15:46:47.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:47.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:47.407624+0000) 2024-03-06T15:46:48.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:48.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:48.407871+0000) 2024-03-06T15:46:49.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:49.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:49.408074+0000) 2024-03-06T15:46:50.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:50.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:50.408323+0000) 2024-03-06T15:46:51.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:51.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:51.408568+0000) 2024-03-06T15:46:52.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:52.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:52.408806+0000) 2024-03-06T15:46:53.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:53.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:53.409020+0000) 2024-03-06T15:46:54.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:54.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:54.409262+0000) 2024-03-06T15:46:55.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:55.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:55.409460+0000) 2024-03-06T15:46:56.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:56.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:56.409748+0000) 2024-03-06T15:46:57.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:57.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:57.409956+0000) 2024-03-06T15:46:58.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:58.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:58.410199+0000) 2024-03-06T15:46:59.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:46:59.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:59.410403+0000) 2024-03-06T15:47:00.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:00.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:00.410632+0000) 2024-03-06T15:47:01.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:01.306 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:01.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:01.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:01.410780+0000) 2024-03-06T15:47:02.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:02.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:02.410933+0000) 2024-03-06T15:47:03.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:03.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:03.411149+0000) 2024-03-06T15:47:04.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:04.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:04.411363+0000) 2024-03-06T15:47:05.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:05.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:05.411591+0000) 2024-03-06T15:47:06.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:06.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:06.411812+0000) 2024-03-06T15:47:07.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:07.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:07.411999+0000) 2024-03-06T15:47:08.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:08.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:08.412228+0000) 2024-03-06T15:47:09.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:09.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:09.412453+0000) 2024-03-06T15:47:10.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:10.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:10.412716+0000) 2024-03-06T15:47:11.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:11.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:11.412942+0000) 2024-03-06T15:47:12.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:12.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:12.413117+0000) 2024-03-06T15:47:13.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:13.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:13.413288+0000) 2024-03-06T15:47:14.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:14.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:14.413512+0000) 2024-03-06T15:47:15.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:15.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:15.413774+0000) 2024-03-06T15:47:16.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:16.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:16.413966+0000) 2024-03-06T15:47:17.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:17.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:17.414189+0000) 2024-03-06T15:47:18.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:18.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:18.414419+0000) 2024-03-06T15:47:19.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:19.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:19.414659+0000) 2024-03-06T15:47:20.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:20.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:20.414927+0000) 2024-03-06T15:47:21.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:21.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:21.415168+0000) 2024-03-06T15:47:22.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:22.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:22.415404+0000) 2024-03-06T15:47:23.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:23.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:23.415650+0000) 2024-03-06T15:47:24.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:24.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:24.415894+0000) 2024-03-06T15:47:25.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:25.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:25.416042+0000) 2024-03-06T15:47:26.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:26.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:26.416303+0000) 2024-03-06T15:47:27.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:27.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:27.416531+0000) 2024-03-06T15:47:28.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:28.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:28.416822+0000) 2024-03-06T15:47:29.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:29.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:29.416982+0000) 2024-03-06T15:47:30.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:30.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:30.417157+0000) 2024-03-06T15:47:31.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:31.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:31.417342+0000) 2024-03-06T15:47:31.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:31.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:32.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:32.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:32.417568+0000) 2024-03-06T15:47:33.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:33.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:33.417799+0000) 2024-03-06T15:47:34.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:34.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:34.417934+0000) 2024-03-06T15:47:35.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:35.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:35.418165+0000) 2024-03-06T15:47:36.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:36.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:36.418396+0000) 2024-03-06T15:47:37.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:37.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:37.418622+0000) 2024-03-06T15:47:38.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:38.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:38.418832+0000) 2024-03-06T15:47:39.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:39.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:39.419067+0000) 2024-03-06T15:47:40.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:40.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:40.419299+0000) 2024-03-06T15:47:41.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:41.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:41.419531+0000) 2024-03-06T15:47:42.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:42.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:42.419772+0000) 2024-03-06T15:47:43.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:43.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:43.419990+0000) 2024-03-06T15:47:44.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:44.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:44.420217+0000) 2024-03-06T15:47:45.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:45.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:45.420449+0000) 2024-03-06T15:47:46.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:46.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:46.420629+0000) 2024-03-06T15:47:47.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:47.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:47.420874+0000) 2024-03-06T15:47:48.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:48.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:48.421162+0000) 2024-03-06T15:47:49.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:49.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:49.421446+0000) 2024-03-06T15:47:50.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:50.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:50.421710+0000) 2024-03-06T15:47:51.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:51.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:51.421952+0000) 2024-03-06T15:47:52.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:52.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:52.422180+0000) 2024-03-06T15:47:53.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:53.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:53.422424+0000) 2024-03-06T15:47:54.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:54.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:54.422609+0000) 2024-03-06T15:47:55.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:55.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:55.422837+0000) 2024-03-06T15:47:56.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:56.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:56.423032+0000) 2024-03-06T15:47:57.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:57.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:57.423254+0000) 2024-03-06T15:47:58.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:58.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:58.423479+0000) 2024-03-06T15:47:59.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:47:59.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:59.423744+0000) 2024-03-06T15:48:00.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:00.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:00.423978+0000) 2024-03-06T15:48:01.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:01.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:01.424202+0000) 2024-03-06T15:48:01.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:01.664 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:02.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:02.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:02.424424+0000) 2024-03-06T15:48:03.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:03.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:03.424653+0000) 2024-03-06T15:48:04.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:04.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:04.424914+0000) 2024-03-06T15:48:05.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:05.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:05.425147+0000) 2024-03-06T15:48:06.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:06.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:06.425370+0000) 2024-03-06T15:48:07.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:07.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:07.425595+0000) 2024-03-06T15:48:08.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:08.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:08.425815+0000) 2024-03-06T15:48:09.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:09.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:09.426047+0000) 2024-03-06T15:48:10.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:10.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:10.426277+0000) 2024-03-06T15:48:11.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:11.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:11.426505+0000) 2024-03-06T15:48:12.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:12.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:12.426801+0000) 2024-03-06T15:48:13.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:13.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:13.427047+0000) 2024-03-06T15:48:14.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:14.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:14.427365+0000) 2024-03-06T15:48:15.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:15.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:15.427649+0000) 2024-03-06T15:48:16.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:16.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:16.427984+0000) 2024-03-06T15:48:17.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:17.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:17.428258+0000) 2024-03-06T15:48:18.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:18.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:18.428497+0000) 2024-03-06T15:48:19.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:19.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:19.428718+0000) 2024-03-06T15:48:20.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:20.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:20.428954+0000) 2024-03-06T15:48:21.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:21.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:21.429145+0000) 2024-03-06T15:48:22.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:22.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:22.429382+0000) 2024-03-06T15:48:23.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:23.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:23.429611+0000) 2024-03-06T15:48:24.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:24.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:24.429820+0000) 2024-03-06T15:48:25.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:25.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:25.430011+0000) 2024-03-06T15:48:26.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:26.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:26.430236+0000) 2024-03-06T15:48:27.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:27.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:27.430457+0000) 2024-03-06T15:48:28.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:28.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:28.430757+0000) 2024-03-06T15:48:29.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:29.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:29.430996+0000) 2024-03-06T15:48:30.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:30.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:30.431225+0000) 2024-03-06T15:48:31.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:31.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:31.431400+0000) 2024-03-06T15:48:31.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:31.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:32.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:32.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:32.431625+0000) 2024-03-06T15:48:33.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:33.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:33.431852+0000) 2024-03-06T15:48:34.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:34.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:34.432062+0000) 2024-03-06T15:48:35.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:35.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:35.432291+0000) 2024-03-06T15:48:36.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:36.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:36.432508+0000) 2024-03-06T15:48:37.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:37.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:37.432777+0000) 2024-03-06T15:48:38.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:38.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:38.432974+0000) 2024-03-06T15:48:39.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:39.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:39.433199+0000) 2024-03-06T15:48:40.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:40.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:40.433427+0000) 2024-03-06T15:48:41.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:41.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:41.433654+0000) 2024-03-06T15:48:42.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:42.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:42.433906+0000) 2024-03-06T15:48:43.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:43.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:43.434152+0000) 2024-03-06T15:48:44.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:44.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:44.434331+0000) 2024-03-06T15:48:45.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:45.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:45.434555+0000) 2024-03-06T15:48:46.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:46.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:46.434786+0000) 2024-03-06T15:48:47.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:47.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:47.434990+0000) 2024-03-06T15:48:48.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:48.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:48.435236+0000) 2024-03-06T15:48:49.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:49.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:49.435466+0000) 2024-03-06T15:48:50.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:50.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:50.435780+0000) 2024-03-06T15:48:51.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:51.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:51.435987+0000) 2024-03-06T15:48:52.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:52.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:52.436217+0000) 2024-03-06T15:48:53.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:53.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:53.436416+0000) 2024-03-06T15:48:54.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:54.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:54.436750+0000) 2024-03-06T15:48:55.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:55.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:55.436982+0000) 2024-03-06T15:48:56.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:56.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:56.437208+0000) 2024-03-06T15:48:57.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:57.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:57.437427+0000) 2024-03-06T15:48:58.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:58.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:58.437730+0000) 2024-03-06T15:48:59.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:48:59.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:59.438007+0000) 2024-03-06T15:49:00.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:00.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:00.438212+0000) 2024-03-06T15:49:01.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:01.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:01.438437+0000) 2024-03-06T15:49:02.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:02.092 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:02.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:02.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:02.438658+0000) 2024-03-06T15:49:03.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:03.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:03.438890+0000) 2024-03-06T15:49:04.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:04.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:04.439094+0000) 2024-03-06T15:49:05.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:05.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:05.439321+0000) 2024-03-06T15:49:06.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:06.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:06.439542+0000) 2024-03-06T15:49:07.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:07.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:07.439836+0000) 2024-03-06T15:49:08.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:08.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:08.440054+0000) 2024-03-06T15:49:09.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:09.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:09.440283+0000) 2024-03-06T15:49:10.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:10.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:10.440504+0000) 2024-03-06T15:49:11.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:11.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:11.440791+0000) 2024-03-06T15:49:12.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:12.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:12.441020+0000) 2024-03-06T15:49:13.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:13.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:13.441209+0000) 2024-03-06T15:49:14.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:14.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:14.441432+0000) 2024-03-06T15:49:15.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:15.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:15.441699+0000) 2024-03-06T15:49:16.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:16.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:16.441895+0000) 2024-03-06T15:49:17.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:17.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:17.442098+0000) 2024-03-06T15:49:18.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:18.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:18.442324+0000) 2024-03-06T15:49:19.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:19.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:19.442547+0000) 2024-03-06T15:49:20.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:20.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:20.442782+0000) 2024-03-06T15:49:21.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:21.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:21.442983+0000) 2024-03-06T15:49:22.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:22.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:22.443209+0000) 2024-03-06T15:49:23.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:23.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:23.443442+0000) 2024-03-06T15:49:24.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:24.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:24.443657+0000) 2024-03-06T15:49:25.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:25.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:25.443927+0000) 2024-03-06T15:49:26.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:26.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:26.444110+0000) 2024-03-06T15:49:27.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:27.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:27.444285+0000) 2024-03-06T15:49:28.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:28.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:28.444525+0000) 2024-03-06T15:49:29.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:29.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:29.444792+0000) 2024-03-06T15:49:30.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:30.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:30.444968+0000) 2024-03-06T15:49:31.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:31.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:31.445192+0000) 2024-03-06T15:49:32.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:32.234 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:32.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:32.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:32.445358+0000) 2024-03-06T15:49:33.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:33.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:33.445529+0000) 2024-03-06T15:49:34.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:34.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:34.445752+0000) 2024-03-06T15:49:35.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:35.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:35.445990+0000) 2024-03-06T15:49:36.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:36.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:36.446216+0000) 2024-03-06T15:49:37.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:37.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:37.446382+0000) 2024-03-06T15:49:38.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:38.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:38.446611+0000) 2024-03-06T15:49:39.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:39.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:39.446848+0000) 2024-03-06T15:49:40.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:40.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:40.447039+0000) 2024-03-06T15:49:41.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:41.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:41.447277+0000) 2024-03-06T15:49:42.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:42.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:42.447514+0000) 2024-03-06T15:49:43.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:43.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:43.447774+0000) 2024-03-06T15:49:44.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:44.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:44.447987+0000) 2024-03-06T15:49:45.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:45.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:45.448173+0000) 2024-03-06T15:49:46.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:46.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:46.448368+0000) 2024-03-06T15:49:47.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:47.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:47.448612+0000) 2024-03-06T15:49:48.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:48.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:48.448838+0000) 2024-03-06T15:49:49.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:49.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:49.449050+0000) 2024-03-06T15:49:50.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:50.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:50.449234+0000) 2024-03-06T15:49:51.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:51.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:51.449451+0000) 2024-03-06T15:49:52.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:52.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:52.449720+0000) 2024-03-06T15:49:53.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:53.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:53.449926+0000) 2024-03-06T15:49:54.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:54.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:54.450117+0000) 2024-03-06T15:49:55.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:55.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:55.450349+0000) 2024-03-06T15:49:56.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:56.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:56.450568+0000) 2024-03-06T15:49:57.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:57.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:57.450813+0000) 2024-03-06T15:49:58.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:58.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:58.451013+0000) 2024-03-06T15:49:59.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:49:59.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:59.451229+0000) 2024-03-06T15:50:00.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:00.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:00.451447+0000) 2024-03-06T15:50:01.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:01.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:01.451708+0000) 2024-03-06T15:50:02.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:02.371 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:02.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:02.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:02.451870+0000) 2024-03-06T15:50:03.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:03.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:03.452043+0000) 2024-03-06T15:50:04.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:04.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:04.452242+0000) 2024-03-06T15:50:05.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:05.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:05.452450+0000) 2024-03-06T15:50:06.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:06.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:06.452707+0000) 2024-03-06T15:50:07.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:07.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:07.452918+0000) 2024-03-06T15:50:08.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:08.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:08.453117+0000) 2024-03-06T15:50:09.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:09.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:09.453320+0000) 2024-03-06T15:50:10.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:10.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:10.453536+0000) 2024-03-06T15:50:11.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:11.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:11.453784+0000) 2024-03-06T15:50:12.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:12.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:12.453981+0000) 2024-03-06T15:50:13.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:13.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:13.454191+0000) 2024-03-06T15:50:14.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:14.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:14.454426+0000) 2024-03-06T15:50:15.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:15.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:15.454650+0000) 2024-03-06T15:50:16.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:16.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:16.454890+0000) 2024-03-06T15:50:17.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:17.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:17.455098+0000) 2024-03-06T15:50:18.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:18.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:18.455348+0000) 2024-03-06T15:50:19.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:19.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:19.455538+0000) 2024-03-06T15:50:20.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:20.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:20.455768+0000) 2024-03-06T15:50:21.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:21.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:21.455998+0000) 2024-03-06T15:50:22.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:22.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:22.456226+0000) 2024-03-06T15:50:23.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:23.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:23.456463+0000) 2024-03-06T15:50:24.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:24.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:24.456710+0000) 2024-03-06T15:50:25.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:25.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:25.456943+0000) 2024-03-06T15:50:26.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:26.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:26.457176+0000) 2024-03-06T15:50:27.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:27.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:27.457360+0000) 2024-03-06T15:50:28.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:28.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:28.457592+0000) 2024-03-06T15:50:29.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:29.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:29.457814+0000) 2024-03-06T15:50:30.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:30.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:30.457998+0000) 2024-03-06T15:50:31.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:31.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:31.458163+0000) 2024-03-06T15:50:32.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:32.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:32.458355+0000) 2024-03-06T15:50:32.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:32.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:33.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:33.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:33.458588+0000) 2024-03-06T15:50:34.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:34.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:34.458810+0000) 2024-03-06T15:50:35.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:35.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:35.459018+0000) 2024-03-06T15:50:36.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:36.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:36.459262+0000) 2024-03-06T15:50:37.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:37.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:37.459500+0000) 2024-03-06T15:50:38.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:38.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:38.459784+0000) 2024-03-06T15:50:39.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:39.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:39.460027+0000) 2024-03-06T15:50:40.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:40.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:40.460263+0000) 2024-03-06T15:50:41.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:41.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:41.460459+0000) 2024-03-06T15:50:42.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:42.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:42.460715+0000) 2024-03-06T15:50:43.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:43.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:43.461021+0000) 2024-03-06T15:50:44.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:44.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:44.461222+0000) 2024-03-06T15:50:45.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:45.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:45.461461+0000) 2024-03-06T15:50:46.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:46.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:46.461638+0000) 2024-03-06T15:50:47.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:47.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:47.461829+0000) 2024-03-06T15:50:48.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:48.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:48.462014+0000) 2024-03-06T15:50:49.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:49.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:49.462256+0000) 2024-03-06T15:50:50.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:50.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:50.462502+0000) 2024-03-06T15:50:51.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:51.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:51.462767+0000) 2024-03-06T15:50:52.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:52.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:52.463004+0000) 2024-03-06T15:50:53.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:53.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:53.463207+0000) 2024-03-06T15:50:54.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:54.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:54.463425+0000) 2024-03-06T15:50:55.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:55.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:55.463611+0000) 2024-03-06T15:50:56.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:56.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:56.463817+0000) 2024-03-06T15:50:57.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:57.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:57.464004+0000) 2024-03-06T15:50:58.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:58.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:58.464227+0000) 2024-03-06T15:50:59.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:50:59.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:59.464445+0000) 2024-03-06T15:51:00.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:00.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:00.464716+0000) 2024-03-06T15:51:01.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:01.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:01.464920+0000) 2024-03-06T15:51:02.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:02.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:02.465160+0000) 2024-03-06T15:51:02.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:02.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:03.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:03.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:03.465392+0000) 2024-03-06T15:51:04.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:04.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:04.465620+0000) 2024-03-06T15:51:05.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:05.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:05.465819+0000) 2024-03-06T15:51:06.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:06.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:06.466018+0000) 2024-03-06T15:51:07.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:07.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:07.466258+0000) 2024-03-06T15:51:08.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:08.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:08.466491+0000) 2024-03-06T15:51:09.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:09.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:09.466762+0000) 2024-03-06T15:51:10.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:10.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:10.466997+0000) 2024-03-06T15:51:11.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:11.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:11.467218+0000) 2024-03-06T15:51:12.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:12.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:12.467409+0000) 2024-03-06T15:51:13.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:13.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:13.467651+0000) 2024-03-06T15:51:14.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:14.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:14.467987+0000) 2024-03-06T15:51:15.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:15.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:15.468203+0000) 2024-03-06T15:51:16.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:16.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:16.468420+0000) 2024-03-06T15:51:17.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:17.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:17.468634+0000) 2024-03-06T15:51:18.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:18.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:18.468916+0000) 2024-03-06T15:51:19.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:19.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:19.469141+0000) 2024-03-06T15:51:20.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:20.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:20.469393+0000) 2024-03-06T15:51:21.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:21.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:21.469641+0000) 2024-03-06T15:51:22.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:22.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:22.470017+0000) 2024-03-06T15:51:23.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:23.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:23.470275+0000) 2024-03-06T15:51:24.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:24.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:24.470494+0000) 2024-03-06T15:51:25.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:25.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:25.470724+0000) 2024-03-06T15:51:26.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:26.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:26.470932+0000) 2024-03-06T15:51:27.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:27.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:27.471137+0000) 2024-03-06T15:51:28.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:28.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:28.471375+0000) 2024-03-06T15:51:29.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:29.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:29.471604+0000) 2024-03-06T15:51:30.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:30.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:30.471824+0000) 2024-03-06T15:51:31.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:31.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:31.472031+0000) 2024-03-06T15:51:32.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:32.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:32.472241+0000) 2024-03-06T15:51:32.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:32.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:33.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:33.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:33.472486+0000) 2024-03-06T15:51:34.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:34.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:34.472750+0000) 2024-03-06T15:51:35.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:35.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:35.472984+0000) 2024-03-06T15:51:36.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:36.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:36.473169+0000) 2024-03-06T15:51:37.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:37.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:37.473447+0000) 2024-03-06T15:51:38.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:38.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:38.473650+0000) 2024-03-06T15:51:39.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:39.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:39.473888+0000) 2024-03-06T15:51:40.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:40.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:40.474124+0000) 2024-03-06T15:51:41.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:41.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:41.474311+0000) 2024-03-06T15:51:42.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:42.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:42.474539+0000) 2024-03-06T15:51:43.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:43.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:43.474794+0000) 2024-03-06T15:51:44.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:44.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:44.474987+0000) 2024-03-06T15:51:45.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:45.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:45.475177+0000) 2024-03-06T15:51:46.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:46.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:46.475463+0000) 2024-03-06T15:51:47.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:47.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:47.475723+0000) 2024-03-06T15:51:48.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:48.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:48.475974+0000) 2024-03-06T15:51:49.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:49.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:49.476202+0000) 2024-03-06T15:51:50.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:50.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:50.476438+0000) 2024-03-06T15:51:51.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:51.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:51.476699+0000) 2024-03-06T15:51:52.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:52.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:52.476907+0000) 2024-03-06T15:51:53.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:53.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:53.477153+0000) 2024-03-06T15:51:54.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:54.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:54.477623+0000) 2024-03-06T15:51:55.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:55.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:55.477864+0000) 2024-03-06T15:51:56.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:56.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:56.478060+0000) 2024-03-06T15:51:57.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:57.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:57.478257+0000) 2024-03-06T15:51:58.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:58.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:58.478487+0000) 2024-03-06T15:51:59.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:51:59.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:59.478741+0000) 2024-03-06T15:52:00.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:00.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:00.478936+0000) 2024-03-06T15:52:01.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:01.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:01.479147+0000) 2024-03-06T15:52:02.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:02.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:02.479495+0000) 2024-03-06T15:52:02.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:02.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:03.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:03.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:03.479782+0000) 2024-03-06T15:52:04.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:04.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:04.479970+0000) 2024-03-06T15:52:05.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:05.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:05.480191+0000) 2024-03-06T15:52:06.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:06.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:06.480410+0000) 2024-03-06T15:52:07.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:07.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:07.480630+0000) 2024-03-06T15:52:08.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:08.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:08.480864+0000) 2024-03-06T15:52:09.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:09.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:09.481055+0000) 2024-03-06T15:52:10.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:10.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:10.481346+0000) 2024-03-06T15:52:11.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:11.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:11.481531+0000) 2024-03-06T15:52:12.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:12.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:12.481781+0000) 2024-03-06T15:52:13.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:13.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:13.481970+0000) 2024-03-06T15:52:14.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:14.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:14.482249+0000) 2024-03-06T15:52:15.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:15.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:15.482535+0000) 2024-03-06T15:52:16.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:16.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:16.482812+0000) 2024-03-06T15:52:17.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:17.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:17.483057+0000) 2024-03-06T15:52:18.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:18.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:18.483256+0000) 2024-03-06T15:52:19.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:19.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:19.483492+0000) 2024-03-06T15:52:20.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:20.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:20.483640+0000) 2024-03-06T15:52:21.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:21.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:21.483854+0000) 2024-03-06T15:52:22.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:22.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:22.484072+0000) 2024-03-06T15:52:23.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:23.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:23.484309+0000) 2024-03-06T15:52:24.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:24.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:24.484483+0000) 2024-03-06T15:52:25.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:25.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:25.484818+0000) 2024-03-06T15:52:26.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:26.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:26.485036+0000) 2024-03-06T15:52:27.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:27.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:27.485273+0000) 2024-03-06T15:52:28.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:28.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:28.485496+0000) 2024-03-06T15:52:29.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:29.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:29.485773+0000) 2024-03-06T15:52:30.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:30.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:30.485987+0000) 2024-03-06T15:52:31.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:31.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:31.486212+0000) 2024-03-06T15:52:32.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:32.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:32.486407+0000) 2024-03-06T15:52:33.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:33.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:33.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:33.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:33.486647+0000) 2024-03-06T15:52:34.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:34.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:34.486898+0000) 2024-03-06T15:52:35.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:35.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:35.487139+0000) 2024-03-06T15:52:36.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:36.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:36.487346+0000) 2024-03-06T15:52:37.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:37.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:37.487530+0000) 2024-03-06T15:52:38.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:38.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:38.487832+0000) 2024-03-06T15:52:39.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:39.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:39.488081+0000) 2024-03-06T15:52:40.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:40.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:40.488349+0000) 2024-03-06T15:52:41.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:41.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:41.488591+0000) 2024-03-06T15:52:42.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:42.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:42.488804+0000) 2024-03-06T15:52:43.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:43.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:43.488998+0000) 2024-03-06T15:52:44.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:44.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:44.489235+0000) 2024-03-06T15:52:45.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:45.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:45.489434+0000) 2024-03-06T15:52:46.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:46.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:46.489657+0000) 2024-03-06T15:52:47.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:47.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:47.489880+0000) 2024-03-06T15:52:48.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:48.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:48.490102+0000) 2024-03-06T15:52:49.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:49.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:49.490330+0000) 2024-03-06T15:52:50.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:50.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:50.490566+0000) 2024-03-06T15:52:51.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:51.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:51.490786+0000) 2024-03-06T15:52:52.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:52.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:52.490983+0000) 2024-03-06T15:52:53.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:53.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:53.491212+0000) 2024-03-06T15:52:54.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:54.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:54.491411+0000) 2024-03-06T15:52:55.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:55.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:55.491649+0000) 2024-03-06T15:52:56.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:56.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:56.491918+0000) 2024-03-06T15:52:57.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:57.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:57.492141+0000) 2024-03-06T15:52:58.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:58.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:58.492379+0000) 2024-03-06T15:52:59.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:52:59.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:59.492596+0000) 2024-03-06T15:53:00.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:00.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:00.492750+0000) 2024-03-06T15:53:01.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:01.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:01.492942+0000) 2024-03-06T15:53:02.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:02.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:02.493164+0000) 2024-03-06T15:53:03.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:03.197 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:03.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:03.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:03.493409+0000) 2024-03-06T15:53:04.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:04.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:04.493632+0000) 2024-03-06T15:53:05.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:05.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:05.493841+0000) 2024-03-06T15:53:06.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:06.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:06.494018+0000) 2024-03-06T15:53:07.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:07.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:07.494237+0000) 2024-03-06T15:53:08.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:08.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:08.494416+0000) 2024-03-06T15:53:09.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:09.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:09.494656+0000) 2024-03-06T15:53:10.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:10.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:10.494957+0000) 2024-03-06T15:53:11.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:11.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:11.495202+0000) 2024-03-06T15:53:12.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:12.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:12.495431+0000) 2024-03-06T15:53:13.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:13.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:13.495724+0000) 2024-03-06T15:53:14.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:14.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:14.495977+0000) 2024-03-06T15:53:15.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:15.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:15.496227+0000) 2024-03-06T15:53:16.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:16.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:16.496474+0000) 2024-03-06T15:53:17.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:17.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:17.496759+0000) 2024-03-06T15:53:18.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:18.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:18.497053+0000) 2024-03-06T15:53:19.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:19.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:19.497288+0000) 2024-03-06T15:53:20.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:20.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:20.497479+0000) 2024-03-06T15:53:21.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:21.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:21.497701+0000) 2024-03-06T15:53:22.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:22.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:22.497901+0000) 2024-03-06T15:53:23.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:23.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:23.498067+0000) 2024-03-06T15:53:24.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:24.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:24.498299+0000) 2024-03-06T15:53:25.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:25.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:25.498529+0000) 2024-03-06T15:53:26.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:26.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:26.498827+0000) 2024-03-06T15:53:27.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:27.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:27.499039+0000) 2024-03-06T15:53:28.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:28.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:28.499283+0000) 2024-03-06T15:53:29.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:29.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:29.499524+0000) 2024-03-06T15:53:30.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:30.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:30.499780+0000) 2024-03-06T15:53:31.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:31.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:31.499996+0000) 2024-03-06T15:53:32.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:32.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:32.500231+0000) 2024-03-06T15:53:33.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:33.364 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:33.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:33.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:33.500459+0000) 2024-03-06T15:53:34.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:34.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:34.500759+0000) 2024-03-06T15:53:35.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:35.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:35.500988+0000) 2024-03-06T15:53:36.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:36.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:36.501177+0000) 2024-03-06T15:53:37.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:37.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:37.501368+0000) 2024-03-06T15:53:38.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:38.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:38.501571+0000) 2024-03-06T15:53:39.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:39.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:39.501760+0000) 2024-03-06T15:53:40.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:40.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:40.501989+0000) 2024-03-06T15:53:41.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:41.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:41.502183+0000) 2024-03-06T15:53:42.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:42.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:42.502411+0000) 2024-03-06T15:53:43.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:43.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:43.502556+0000) 2024-03-06T15:53:44.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:44.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:44.502771+0000) 2024-03-06T15:53:45.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:45.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:45.502994+0000) 2024-03-06T15:53:46.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:46.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:46.503226+0000) 2024-03-06T15:53:47.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:47.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:47.503452+0000) 2024-03-06T15:53:48.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:48.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:48.503643+0000) 2024-03-06T15:53:49.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:49.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:49.503888+0000) 2024-03-06T15:53:50.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:50.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:50.504084+0000) 2024-03-06T15:53:51.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:51.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:51.504273+0000) 2024-03-06T15:53:52.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:52.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:52.504516+0000) 2024-03-06T15:53:53.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:53.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:53.504787+0000) 2024-03-06T15:53:54.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:54.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:54.504982+0000) 2024-03-06T15:53:55.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:55.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:55.505168+0000) 2024-03-06T15:53:56.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:56.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:56.505338+0000) 2024-03-06T15:53:57.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:57.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:57.505574+0000) 2024-03-06T15:53:58.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:58.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:58.505828+0000) 2024-03-06T15:53:59.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:53:59.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:59.506078+0000) 2024-03-06T15:54:00.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:00.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:00.506330+0000) 2024-03-06T15:54:01.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:01.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:01.506556+0000) 2024-03-06T15:54:02.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:02.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:02.506785+0000) 2024-03-06T15:54:03.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:03.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:03.507021+0000) 2024-03-06T15:54:03.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:03.528 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:04.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:04.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:04.507249+0000) 2024-03-06T15:54:05.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:05.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:05.507474+0000) 2024-03-06T15:54:06.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:06.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:06.507784+0000) 2024-03-06T15:54:07.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:07.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:07.508009+0000) 2024-03-06T15:54:08.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:08.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:08.508231+0000) 2024-03-06T15:54:09.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:09.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:09.508473+0000) 2024-03-06T15:54:10.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:10.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:10.508690+0000) 2024-03-06T15:54:11.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:11.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:11.508929+0000) 2024-03-06T15:54:12.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:12.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:12.509058+0000) 2024-03-06T15:54:13.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:13.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:13.509255+0000) 2024-03-06T15:54:14.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:14.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:14.509432+0000) 2024-03-06T15:54:15.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:15.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:15.509693+0000) 2024-03-06T15:54:16.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:16.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:16.509887+0000) 2024-03-06T15:54:17.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:17.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:17.510096+0000) 2024-03-06T15:54:18.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:18.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:18.510344+0000) 2024-03-06T15:54:19.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:19.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:19.510583+0000) 2024-03-06T15:54:20.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:20.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:20.510830+0000) 2024-03-06T15:54:21.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:21.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:21.511064+0000) 2024-03-06T15:54:22.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:22.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:22.511309+0000) 2024-03-06T15:54:23.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:23.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:23.511565+0000) 2024-03-06T15:54:24.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:24.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:24.511855+0000) 2024-03-06T15:54:25.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:25.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:25.512066+0000) 2024-03-06T15:54:26.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:26.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:26.512312+0000) 2024-03-06T15:54:27.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:27.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:27.512550+0000) 2024-03-06T15:54:28.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:28.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:28.512796+0000) 2024-03-06T15:54:29.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:29.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:29.513010+0000) 2024-03-06T15:54:30.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:30.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:30.513259+0000) 2024-03-06T15:54:31.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:31.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:31.513501+0000) 2024-03-06T15:54:32.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:32.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:32.513785+0000) 2024-03-06T15:54:33.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:33.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:33.514046+0000) 2024-03-06T15:54:33.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:33.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:34.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:34.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:34.514256+0000) 2024-03-06T15:54:35.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:35.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:35.514497+0000) 2024-03-06T15:54:36.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:36.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:36.514780+0000) 2024-03-06T15:54:37.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:37.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:37.514989+0000) 2024-03-06T15:54:38.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:38.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:38.515227+0000) 2024-03-06T15:54:39.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:39.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:39.515457+0000) 2024-03-06T15:54:40.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:40.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:40.515721+0000) 2024-03-06T15:54:41.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:41.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:41.515887+0000) 2024-03-06T15:54:42.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:42.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:42.516087+0000) 2024-03-06T15:54:43.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:43.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:43.516369+0000) 2024-03-06T15:54:44.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:44.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:44.516625+0000) 2024-03-06T15:54:45.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:45.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:45.516865+0000) 2024-03-06T15:54:46.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:46.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:46.517072+0000) 2024-03-06T15:54:47.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:47.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:47.517261+0000) 2024-03-06T15:54:48.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:48.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:48.517472+0000) 2024-03-06T15:54:49.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:49.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:49.517739+0000) 2024-03-06T15:54:50.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:50.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:50.517976+0000) 2024-03-06T15:54:51.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:51.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:51.518222+0000) 2024-03-06T15:54:52.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:52.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:52.518410+0000) 2024-03-06T15:54:53.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:53.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:53.518607+0000) 2024-03-06T15:54:54.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:54.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:54.518874+0000) 2024-03-06T15:54:55.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:55.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:55.519072+0000) 2024-03-06T15:54:56.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:56.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:56.519309+0000) 2024-03-06T15:54:57.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:57.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:57.519545+0000) 2024-03-06T15:54:58.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:58.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:58.519831+0000) 2024-03-06T15:54:59.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:54:59.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:59.520069+0000) 2024-03-06T15:55:00.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:00.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:00.520272+0000) 2024-03-06T15:55:01.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:01.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:01.520453+0000) 2024-03-06T15:55:02.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:02.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:02.520706+0000) 2024-03-06T15:55:03.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:03.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:03.520947+0000) 2024-03-06T15:55:03.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:03.802 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:04.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:04.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:04.521171+0000) 2024-03-06T15:55:05.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:05.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:05.521378+0000) 2024-03-06T15:55:06.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:06.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:06.521567+0000) 2024-03-06T15:55:07.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:07.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:07.521776+0000) 2024-03-06T15:55:08.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:08.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:08.522048+0000) 2024-03-06T15:55:09.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:09.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:09.522224+0000) 2024-03-06T15:55:10.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:10.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:10.522349+0000) 2024-03-06T15:55:11.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:11.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:11.522488+0000) 2024-03-06T15:55:12.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:12.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:12.522746+0000) 2024-03-06T15:55:13.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:13.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:13.522951+0000) 2024-03-06T15:55:14.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:14.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:14.523200+0000) 2024-03-06T15:55:15.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:15.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:15.523437+0000) 2024-03-06T15:55:16.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:16.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:16.523659+0000) 2024-03-06T15:55:17.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:17.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:17.523925+0000) 2024-03-06T15:55:18.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:18.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:18.524071+0000) 2024-03-06T15:55:19.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:19.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:19.524208+0000) 2024-03-06T15:55:20.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:20.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:20.524436+0000) 2024-03-06T15:55:21.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:21.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:21.524572+0000) 2024-03-06T15:55:22.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:22.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:22.524824+0000) 2024-03-06T15:55:23.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:23.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:23.525037+0000) 2024-03-06T15:55:24.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:24.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:24.525210+0000) 2024-03-06T15:55:25.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:25.524+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:25.525439+0000) 2024-03-06T15:55:26.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:26.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:26.525660+0000) 2024-03-06T15:55:27.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:27.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:27.525897+0000) 2024-03-06T15:55:28.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:28.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:28.526109+0000) 2024-03-06T15:55:29.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:29.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:29.526341+0000) 2024-03-06T15:55:30.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:30.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:30.526536+0000) 2024-03-06T15:55:31.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:31.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:31.526823+0000) 2024-03-06T15:55:32.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:32.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:32.527012+0000) 2024-03-06T15:55:33.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:33.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:33.527251+0000) 2024-03-06T15:55:34.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:34.011 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:34.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:34.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:34.527483+0000) 2024-03-06T15:55:35.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:35.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:35.527788+0000) 2024-03-06T15:55:36.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:36.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:36.527966+0000) 2024-03-06T15:55:37.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:37.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:37.528181+0000) 2024-03-06T15:55:38.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:38.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:38.528392+0000) 2024-03-06T15:55:39.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:39.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:39.528635+0000) 2024-03-06T15:55:40.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:40.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:40.528879+0000) 2024-03-06T15:55:41.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:41.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:41.529105+0000) 2024-03-06T15:55:42.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:42.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:42.529347+0000) 2024-03-06T15:55:43.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:43.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:43.529587+0000) 2024-03-06T15:55:44.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:44.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:44.529819+0000) 2024-03-06T15:55:45.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:45.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:45.530042+0000) 2024-03-06T15:55:46.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:46.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:46.530242+0000) 2024-03-06T15:55:47.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:47.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:47.530526+0000) 2024-03-06T15:55:48.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:48.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:48.530792+0000) 2024-03-06T15:55:49.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:49.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:49.531003+0000) 2024-03-06T15:55:50.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:50.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:50.531193+0000) 2024-03-06T15:55:51.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:51.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:51.531397+0000) 2024-03-06T15:55:52.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:52.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:52.531614+0000) 2024-03-06T15:55:53.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:53.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:53.531874+0000) 2024-03-06T15:55:54.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:54.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:54.532059+0000) 2024-03-06T15:55:55.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:55.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:55.532295+0000) 2024-03-06T15:55:56.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:56.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:56.532527+0000) 2024-03-06T15:55:57.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:57.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:57.532789+0000) 2024-03-06T15:55:58.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:58.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:58.533027+0000) 2024-03-06T15:55:59.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:55:59.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:59.533250+0000) 2024-03-06T15:56:00.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:00.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:00.533454+0000) 2024-03-06T15:56:01.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:01.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:01.533775+0000) 2024-03-06T15:56:02.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:02.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:02.534029+0000) 2024-03-06T15:56:03.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:03.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:03.534278+0000) 2024-03-06T15:56:04.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:04.208 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:04.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:04.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:04.534538+0000) 2024-03-06T15:56:05.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:05.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:05.534786+0000) 2024-03-06T15:56:06.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:06.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:06.535030+0000) 2024-03-06T15:56:07.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:07.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:07.535269+0000) 2024-03-06T15:56:08.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:08.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:08.535486+0000) 2024-03-06T15:56:09.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:09.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:09.535765+0000) 2024-03-06T15:56:10.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:10.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:10.536052+0000) 2024-03-06T15:56:11.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:11.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:11.536293+0000) 2024-03-06T15:56:12.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:12.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:12.536499+0000) 2024-03-06T15:56:13.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:13.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:13.536802+0000) 2024-03-06T15:56:14.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:14.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:14.537024+0000) 2024-03-06T15:56:15.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:15.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:15.537255+0000) 2024-03-06T15:56:16.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:16.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:16.537464+0000) 2024-03-06T15:56:17.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:17.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:17.537725+0000) 2024-03-06T15:56:18.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:18.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:18.537949+0000) 2024-03-06T15:56:19.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:19.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:19.538191+0000) 2024-03-06T15:56:20.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:20.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:20.538419+0000) 2024-03-06T15:56:21.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:21.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:21.538645+0000) 2024-03-06T15:56:22.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:22.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:22.538869+0000) 2024-03-06T15:56:23.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:23.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:23.539090+0000) 2024-03-06T15:56:24.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:24.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:24.539330+0000) 2024-03-06T15:56:25.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:25.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:25.539568+0000) 2024-03-06T15:56:26.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:26.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:26.539826+0000) 2024-03-06T15:56:27.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:27.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:27.540029+0000) 2024-03-06T15:56:28.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:28.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:28.540285+0000) 2024-03-06T15:56:29.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:29.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:29.540499+0000) 2024-03-06T15:56:30.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:30.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:30.540811+0000) 2024-03-06T15:56:31.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:31.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:31.541059+0000) 2024-03-06T15:56:32.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:32.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:32.541268+0000) 2024-03-06T15:56:33.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:33.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:33.541510+0000) 2024-03-06T15:56:34.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:34.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:34.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:34.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:34.541803+0000) 2024-03-06T15:56:35.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:35.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:35.542094+0000) 2024-03-06T15:56:36.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:36.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:36.542317+0000) 2024-03-06T15:56:37.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:37.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:37.542533+0000) 2024-03-06T15:56:38.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:38.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:38.542825+0000) 2024-03-06T15:56:39.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:39.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:39.543027+0000) 2024-03-06T15:56:40.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:40.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:40.543234+0000) 2024-03-06T15:56:41.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:41.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:41.543475+0000) 2024-03-06T15:56:42.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:42.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:42.543752+0000) 2024-03-06T15:56:43.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:43.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:43.543940+0000) 2024-03-06T15:56:44.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:44.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:44.544166+0000) 2024-03-06T15:56:45.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:45.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:45.544396+0000) 2024-03-06T15:56:46.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:46.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:46.544616+0000) 2024-03-06T15:56:47.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:47.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:47.544846+0000) 2024-03-06T15:56:48.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:48.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:48.545089+0000) 2024-03-06T15:56:49.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:49.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:49.545319+0000) 2024-03-06T15:56:50.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:50.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:50.545558+0000) 2024-03-06T15:56:51.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:51.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:51.545781+0000) 2024-03-06T15:56:52.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:52.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:52.546013+0000) 2024-03-06T15:56:53.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:53.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:53.546249+0000) 2024-03-06T15:56:54.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:54.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:54.546476+0000) 2024-03-06T15:56:55.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:55.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:55.546744+0000) 2024-03-06T15:56:56.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:56.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:56.546891+0000) 2024-03-06T15:56:57.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:57.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:57.547082+0000) 2024-03-06T15:56:58.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:58.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:58.547318+0000) 2024-03-06T15:56:59.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:56:59.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:59.547511+0000) 2024-03-06T15:57:00.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:00.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:00.547761+0000) 2024-03-06T15:57:01.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:01.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:01.547997+0000) 2024-03-06T15:57:02.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:02.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:02.548244+0000) 2024-03-06T15:57:03.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:03.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:03.548489+0000) 2024-03-06T15:57:04.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:04.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:04.548736+0000) 2024-03-06T15:57:04.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:04.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:05.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:05.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:05.548934+0000) 2024-03-06T15:57:06.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:06.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:06.549158+0000) 2024-03-06T15:57:07.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:07.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:07.549351+0000) 2024-03-06T15:57:08.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:08.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:08.549597+0000) 2024-03-06T15:57:09.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:09.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:09.549790+0000) 2024-03-06T15:57:10.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:10.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:10.549986+0000) 2024-03-06T15:57:11.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:11.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:11.550207+0000) 2024-03-06T15:57:12.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:12.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:12.550481+0000) 2024-03-06T15:57:13.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:13.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:13.550740+0000) 2024-03-06T15:57:14.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:14.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:14.550977+0000) 2024-03-06T15:57:15.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:15.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:15.551169+0000) 2024-03-06T15:57:16.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:16.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:16.551382+0000) 2024-03-06T15:57:17.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:17.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:17.551578+0000) 2024-03-06T15:57:18.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:18.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:18.551830+0000) 2024-03-06T15:57:19.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:19.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:19.552003+0000) 2024-03-06T15:57:20.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:20.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:20.552251+0000) 2024-03-06T15:57:21.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:21.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:21.552468+0000) 2024-03-06T15:57:22.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:22.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:22.552703+0000) 2024-03-06T15:57:23.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:23.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:23.552971+0000) 2024-03-06T15:57:24.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:24.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:24.553176+0000) 2024-03-06T15:57:25.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:25.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:25.553388+0000) 2024-03-06T15:57:26.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:26.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:26.553623+0000) 2024-03-06T15:57:27.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:27.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:27.553911+0000) 2024-03-06T15:57:28.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:28.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:28.554160+0000) 2024-03-06T15:57:29.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:29.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:29.554410+0000) 2024-03-06T15:57:30.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:30.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:30.554606+0000) 2024-03-06T15:57:31.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:31.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:31.554830+0000) 2024-03-06T15:57:32.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:32.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:32.555019+0000) 2024-03-06T15:57:33.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:33.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:33.555309+0000) 2024-03-06T15:57:34.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:34.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:34.555576+0000) 2024-03-06T15:57:34.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:34.731 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:35.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:35.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:35.555820+0000) 2024-03-06T15:57:36.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:36.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:36.556069+0000) 2024-03-06T15:57:37.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:37.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:37.556346+0000) 2024-03-06T15:57:38.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:38.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:38.556615+0000) 2024-03-06T15:57:39.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:39.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:39.556870+0000) 2024-03-06T15:57:40.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:40.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:40.557087+0000) 2024-03-06T15:57:41.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:41.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:41.557286+0000) 2024-03-06T15:57:42.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:42.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:42.557509+0000) 2024-03-06T15:57:43.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:43.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:43.557793+0000) 2024-03-06T15:57:44.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:44.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:44.558015+0000) 2024-03-06T15:57:45.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:45.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:45.558197+0000) 2024-03-06T15:57:46.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:46.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:46.558425+0000) 2024-03-06T15:57:47.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:47.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:47.558594+0000) 2024-03-06T15:57:48.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:48.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:48.558858+0000) 2024-03-06T15:57:49.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:49.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:49.559102+0000) 2024-03-06T15:57:50.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:50.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:50.559315+0000) 2024-03-06T15:57:51.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:51.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:51.559543+0000) 2024-03-06T15:57:52.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:52.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:52.559762+0000) 2024-03-06T15:57:53.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:53.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:53.559963+0000) 2024-03-06T15:57:54.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:54.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:54.560193+0000) 2024-03-06T15:57:55.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:55.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:55.560413+0000) 2024-03-06T15:57:56.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:56.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:56.560650+0000) 2024-03-06T15:57:57.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:57.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:57.560930+0000) 2024-03-06T15:57:58.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:58.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:58.561158+0000) 2024-03-06T15:57:59.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:57:59.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:59.561332+0000) 2024-03-06T15:58:00.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:00.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:00.561485+0000) 2024-03-06T15:58:01.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:01.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:01.561761+0000) 2024-03-06T15:58:02.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:02.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:02.561914+0000) 2024-03-06T15:58:03.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:03.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:03.562143+0000) 2024-03-06T15:58:04.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:04.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:04.562389+0000) 2024-03-06T15:58:04.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:04.946 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:05.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:05.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:05.562608+0000) 2024-03-06T15:58:06.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:06.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:06.562849+0000) 2024-03-06T15:58:07.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:07.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:07.563082+0000) 2024-03-06T15:58:08.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:08.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:08.563353+0000) 2024-03-06T15:58:09.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:09.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:09.563573+0000) 2024-03-06T15:58:10.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:10.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:10.563789+0000) 2024-03-06T15:58:11.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:11.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:11.563961+0000) 2024-03-06T15:58:12.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:12.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:12.564110+0000) 2024-03-06T15:58:13.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:13.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:13.564288+0000) 2024-03-06T15:58:14.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:14.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:14.564465+0000) 2024-03-06T15:58:15.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:15.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:15.564718+0000) 2024-03-06T15:58:16.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:16.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:16.564929+0000) 2024-03-06T15:58:17.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:17.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:17.565275+0000) 2024-03-06T15:58:18.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:18.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:18.565550+0000) 2024-03-06T15:58:19.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:19.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:19.565824+0000) 2024-03-06T15:58:20.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:20.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:20.566120+0000) 2024-03-06T15:58:21.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:21.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:21.566298+0000) 2024-03-06T15:58:22.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:22.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:22.566489+0000) 2024-03-06T15:58:23.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:23.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:23.566755+0000) 2024-03-06T15:58:24.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:24.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:24.566942+0000) 2024-03-06T15:58:25.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:25.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:25.567202+0000) 2024-03-06T15:58:26.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:26.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:26.567429+0000) 2024-03-06T15:58:27.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:27.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:27.567757+0000) 2024-03-06T15:58:28.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:28.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:28.567949+0000) 2024-03-06T15:58:29.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:29.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:29.568147+0000) 2024-03-06T15:58:30.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:30.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:30.568394+0000) 2024-03-06T15:58:31.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:31.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:31.568599+0000) 2024-03-06T15:58:32.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:32.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:32.568837+0000) 2024-03-06T15:58:33.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:33.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:33.569094+0000) 2024-03-06T15:58:34.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:34.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:34.569267+0000) 2024-03-06T15:58:35.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:35.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:35.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:35.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:35.569430+0000) 2024-03-06T15:58:36.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:36.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:36.569608+0000) 2024-03-06T15:58:37.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:37.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:37.569846+0000) 2024-03-06T15:58:38.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:38.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:38.570045+0000) 2024-03-06T15:58:39.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:39.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:39.570269+0000) 2024-03-06T15:58:40.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:40.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:40.570476+0000) 2024-03-06T15:58:41.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:41.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:41.571277+0000) 2024-03-06T15:58:42.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:42.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:42.571519+0000) 2024-03-06T15:58:43.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:43.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:43.571784+0000) 2024-03-06T15:58:44.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:44.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:44.572012+0000) 2024-03-06T15:58:45.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:45.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:45.572238+0000) 2024-03-06T15:58:46.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:46.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:46.572413+0000) 2024-03-06T15:58:47.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:47.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:47.572644+0000) 2024-03-06T15:58:48.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:48.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:48.572978+0000) 2024-03-06T15:58:49.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:49.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:49.573210+0000) 2024-03-06T15:58:50.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:50.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:50.573480+0000) 2024-03-06T15:58:51.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:51.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:51.573697+0000) 2024-03-06T15:58:52.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:52.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:52.573907+0000) 2024-03-06T15:58:53.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:53.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:53.574089+0000) 2024-03-06T15:58:54.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:54.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:54.574312+0000) 2024-03-06T15:58:55.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:55.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:55.574493+0000) 2024-03-06T15:58:56.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:56.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:56.574767+0000) 2024-03-06T15:58:57.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:57.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:57.574933+0000) 2024-03-06T15:58:58.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:58.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:58.575130+0000) 2024-03-06T15:58:59.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:58:59.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:59.575371+0000) 2024-03-06T15:59:00.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:00.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:00.575599+0000) 2024-03-06T15:59:01.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:01.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:01.575816+0000) 2024-03-06T15:59:02.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:02.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:02.575995+0000) 2024-03-06T15:59:03.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:03.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:03.576213+0000) 2024-03-06T15:59:04.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:04.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:04.576492+0000) 2024-03-06T15:59:05.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:05.384 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:05.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:05.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:05.576783+0000) 2024-03-06T15:59:06.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:06.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:06.577044+0000) 2024-03-06T15:59:07.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:07.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:07.577314+0000) 2024-03-06T15:59:08.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:08.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:08.577594+0000) 2024-03-06T15:59:09.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:09.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:09.577879+0000) 2024-03-06T15:59:10.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:10.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:10.578130+0000) 2024-03-06T15:59:11.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:11.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:11.578410+0000) 2024-03-06T15:59:12.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:12.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:12.578711+0000) 2024-03-06T15:59:13.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:13.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:13.578935+0000) 2024-03-06T15:59:14.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:14.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:14.579199+0000) 2024-03-06T15:59:15.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:15.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:15.579470+0000) 2024-03-06T15:59:16.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:16.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:16.579783+0000) 2024-03-06T15:59:17.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:17.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:17.580059+0000) 2024-03-06T15:59:18.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:18.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:18.580323+0000) 2024-03-06T15:59:19.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:19.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:19.580597+0000) 2024-03-06T15:59:20.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:20.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:20.580846+0000) 2024-03-06T15:59:21.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:21.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:21.581054+0000) 2024-03-06T15:59:22.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:22.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:22.581242+0000) 2024-03-06T15:59:23.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:23.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:23.581488+0000) 2024-03-06T15:59:24.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:24.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:24.581758+0000) 2024-03-06T15:59:25.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:25.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:25.582018+0000) 2024-03-06T15:59:26.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:26.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:26.582247+0000) 2024-03-06T15:59:27.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:27.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:27.582469+0000) 2024-03-06T15:59:28.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:28.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:28.582753+0000) 2024-03-06T15:59:29.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:29.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:29.582975+0000) 2024-03-06T15:59:30.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:30.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:30.583210+0000) 2024-03-06T15:59:31.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:31.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:31.583414+0000) 2024-03-06T15:59:32.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:32.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:32.583637+0000) 2024-03-06T15:59:33.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:33.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:33.583869+0000) 2024-03-06T15:59:34.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:34.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:34.584095+0000) 2024-03-06T15:59:35.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:35.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:35.584277+0000) 2024-03-06T15:59:35.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:35.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:36.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:36.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:36.584499+0000) 2024-03-06T15:59:37.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:37.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:37.584683+0000) 2024-03-06T15:59:38.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:38.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:38.584901+0000) 2024-03-06T15:59:39.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:39.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:39.585133+0000) 2024-03-06T15:59:40.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:40.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:40.585362+0000) 2024-03-06T15:59:41.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:41.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:41.585585+0000) 2024-03-06T15:59:42.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:42.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:42.585783+0000) 2024-03-06T15:59:43.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:43.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:43.586019+0000) 2024-03-06T15:59:44.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:44.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:44.586248+0000) 2024-03-06T15:59:45.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:45.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:45.586443+0000) 2024-03-06T15:59:46.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:46.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:46.586648+0000) 2024-03-06T15:59:47.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:47.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:47.586919+0000) 2024-03-06T15:59:48.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:48.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:48.587117+0000) 2024-03-06T15:59:49.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:49.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:49.587352+0000) 2024-03-06T15:59:50.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:50.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:50.587570+0000) 2024-03-06T15:59:51.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:51.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:51.587854+0000) 2024-03-06T15:59:52.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:52.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:52.588107+0000) 2024-03-06T15:59:53.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:53.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:53.588380+0000) 2024-03-06T15:59:54.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:54.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:54.588643+0000) 2024-03-06T15:59:55.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:55.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:55.588900+0000) 2024-03-06T15:59:56.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:56.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:56.589161+0000) 2024-03-06T15:59:57.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:57.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:57.589389+0000) 2024-03-06T15:59:58.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:58.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:58.589762+0000) 2024-03-06T15:59:59.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T15:59:59.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:59.589973+0000) 2024-03-06T16:00:00.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:00.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:00.590180+0000) 2024-03-06T16:00:01.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:01.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:01.590414+0000) 2024-03-06T16:00:02.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:02.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:02.590630+0000) 2024-03-06T16:00:03.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:03.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:03.590865+0000) 2024-03-06T16:00:04.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:04.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:04.591108+0000) 2024-03-06T16:00:05.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:05.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:05.591334+0000) 2024-03-06T16:00:05.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:05.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:06.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:06.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:06.591551+0000) 2024-03-06T16:00:07.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:07.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:07.591786+0000) 2024-03-06T16:00:08.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:08.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:08.591981+0000) 2024-03-06T16:00:09.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:09.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:09.592246+0000) 2024-03-06T16:00:10.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:10.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:10.592535+0000) 2024-03-06T16:00:11.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:11.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:11.592740+0000) 2024-03-06T16:00:12.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:12.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:12.592920+0000) 2024-03-06T16:00:13.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:13.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:13.593158+0000) 2024-03-06T16:00:14.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:14.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:14.593400+0000) 2024-03-06T16:00:15.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:15.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:15.593635+0000) 2024-03-06T16:00:16.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:16.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:16.593918+0000) 2024-03-06T16:00:17.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:17.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:17.594127+0000) 2024-03-06T16:00:18.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:18.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:18.594376+0000) 2024-03-06T16:00:19.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:19.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:19.594613+0000) 2024-03-06T16:00:20.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:20.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:20.594844+0000) 2024-03-06T16:00:21.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:21.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:21.595066+0000) 2024-03-06T16:00:22.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:22.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:22.595305+0000) 2024-03-06T16:00:23.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:23.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:23.595548+0000) 2024-03-06T16:00:24.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:24.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:24.595790+0000) 2024-03-06T16:00:25.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:25.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:25.596025+0000) 2024-03-06T16:00:26.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:26.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:26.596246+0000) 2024-03-06T16:00:27.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:27.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:27.596491+0000) 2024-03-06T16:00:28.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:28.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:28.596769+0000) 2024-03-06T16:00:29.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:29.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:29.597003+0000) 2024-03-06T16:00:30.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:30.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:30.597235+0000) 2024-03-06T16:00:31.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:31.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:31.597458+0000) 2024-03-06T16:00:32.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:32.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:32.597710+0000) 2024-03-06T16:00:33.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:33.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:33.597922+0000) 2024-03-06T16:00:34.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:34.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:34.598125+0000) 2024-03-06T16:00:35.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:35.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:35.598357+0000) 2024-03-06T16:00:36.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:36.026 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:36.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:36.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:36.598551+0000) 2024-03-06T16:00:37.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:37.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:37.598773+0000) 2024-03-06T16:00:38.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:38.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:38.599016+0000) 2024-03-06T16:00:39.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:39.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:39.599242+0000) 2024-03-06T16:00:40.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:40.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:40.599500+0000) 2024-03-06T16:00:41.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:41.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:41.599770+0000) 2024-03-06T16:00:42.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:42.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:42.599959+0000) 2024-03-06T16:00:43.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:43.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:43.600188+0000) 2024-03-06T16:00:44.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:44.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:44.600383+0000) 2024-03-06T16:00:45.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:45.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:45.600610+0000) 2024-03-06T16:00:46.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:46.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:46.600884+0000) 2024-03-06T16:00:47.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:47.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:47.601104+0000) 2024-03-06T16:00:48.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:48.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:48.601286+0000) 2024-03-06T16:00:49.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:49.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:49.601465+0000) 2024-03-06T16:00:50.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:50.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:50.601652+0000) 2024-03-06T16:00:51.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:51.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:51.601910+0000) 2024-03-06T16:00:52.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:52.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:52.602111+0000) 2024-03-06T16:00:53.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:53.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:53.602312+0000) 2024-03-06T16:00:54.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:54.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:54.602546+0000) 2024-03-06T16:00:55.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:55.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:55.602774+0000) 2024-03-06T16:00:56.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:56.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:56.602949+0000) 2024-03-06T16:00:57.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:57.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:57.603149+0000) 2024-03-06T16:00:58.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:58.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:58.603403+0000) 2024-03-06T16:00:59.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:00:59.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:59.603611+0000) 2024-03-06T16:01:00.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:00.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:00.603818+0000) 2024-03-06T16:01:01.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:01.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:01.604064+0000) 2024-03-06T16:01:02.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:02.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:02.604280+0000) 2024-03-06T16:01:03.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:03.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:03.604473+0000) 2024-03-06T16:01:04.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:04.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:04.604757+0000) 2024-03-06T16:01:05.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:05.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:05.604991+0000) 2024-03-06T16:01:06.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:06.230 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:06.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:06.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:06.605170+0000) 2024-03-06T16:01:07.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:07.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:07.605409+0000) 2024-03-06T16:01:08.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:08.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:08.605654+0000) 2024-03-06T16:01:09.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:09.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:09.605945+0000) 2024-03-06T16:01:10.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:10.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:10.606192+0000) 2024-03-06T16:01:11.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:11.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:11.606475+0000) 2024-03-06T16:01:12.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:12.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:12.606793+0000) 2024-03-06T16:01:13.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:13.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:13.607077+0000) 2024-03-06T16:01:14.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:14.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:14.607334+0000) 2024-03-06T16:01:15.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:15.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:15.607516+0000) 2024-03-06T16:01:16.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:16.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:16.607772+0000) 2024-03-06T16:01:17.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:17.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:17.607991+0000) 2024-03-06T16:01:18.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:18.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:18.608213+0000) 2024-03-06T16:01:19.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:19.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:19.608408+0000) 2024-03-06T16:01:20.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:20.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:20.608651+0000) 2024-03-06T16:01:21.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:21.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:21.608875+0000) 2024-03-06T16:01:22.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:22.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:22.609097+0000) 2024-03-06T16:01:23.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:23.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:23.609322+0000) 2024-03-06T16:01:24.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:24.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:24.609563+0000) 2024-03-06T16:01:25.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:25.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:25.609795+0000) 2024-03-06T16:01:26.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:26.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:26.609994+0000) 2024-03-06T16:01:27.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:27.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:27.610211+0000) 2024-03-06T16:01:28.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:28.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:28.610447+0000) 2024-03-06T16:01:29.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:29.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:29.610722+0000) 2024-03-06T16:01:30.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:30.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:30.610920+0000) 2024-03-06T16:01:31.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:31.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:31.611157+0000) 2024-03-06T16:01:32.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:32.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:32.611388+0000) 2024-03-06T16:01:33.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:33.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:33.611616+0000) 2024-03-06T16:01:34.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:34.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:34.611915+0000) 2024-03-06T16:01:35.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:35.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:35.612146+0000) 2024-03-06T16:01:36.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:36.422 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:36.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:36.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:36.612388+0000) 2024-03-06T16:01:37.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:37.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:37.612577+0000) 2024-03-06T16:01:38.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:38.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:38.612822+0000) 2024-03-06T16:01:39.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:39.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:39.613055+0000) 2024-03-06T16:01:40.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:40.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:40.613268+0000) 2024-03-06T16:01:41.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:41.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:41.613549+0000) 2024-03-06T16:01:42.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:42.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:42.613791+0000) 2024-03-06T16:01:43.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:43.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:43.614069+0000) 2024-03-06T16:01:44.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:44.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:44.614316+0000) 2024-03-06T16:01:45.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:45.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:45.614489+0000) 2024-03-06T16:01:46.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:46.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:46.614706+0000) 2024-03-06T16:01:47.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:47.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:47.614939+0000) 2024-03-06T16:01:48.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:48.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:48.615119+0000) 2024-03-06T16:01:49.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:49.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:49.615313+0000) 2024-03-06T16:01:50.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:50.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:50.615562+0000) 2024-03-06T16:01:51.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:51.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:51.615828+0000) 2024-03-06T16:01:52.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:52.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:52.616061+0000) 2024-03-06T16:01:53.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:53.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:53.616344+0000) 2024-03-06T16:01:54.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:54.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:54.616558+0000) 2024-03-06T16:01:55.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:55.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:55.616808+0000) 2024-03-06T16:01:56.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:56.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:56.617023+0000) 2024-03-06T16:01:57.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:57.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:57.617288+0000) 2024-03-06T16:01:58.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:58.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:58.617519+0000) 2024-03-06T16:01:59.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:01:59.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:59.617794+0000) 2024-03-06T16:02:00.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:00.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:00.618040+0000) 2024-03-06T16:02:01.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:01.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:01.618241+0000) 2024-03-06T16:02:02.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:02.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:02.618423+0000) 2024-03-06T16:02:03.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:03.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:03.618642+0000) 2024-03-06T16:02:04.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:04.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:04.618910+0000) 2024-03-06T16:02:05.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:05.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:05.619129+0000) 2024-03-06T16:02:06.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:06.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:06.619355+0000) 2024-03-06T16:02:06.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:06.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:07.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:07.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:07.619574+0000) 2024-03-06T16:02:08.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:08.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:08.619801+0000) 2024-03-06T16:02:09.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:09.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:09.619999+0000) 2024-03-06T16:02:10.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:10.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:10.620253+0000) 2024-03-06T16:02:11.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:11.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:11.620481+0000) 2024-03-06T16:02:12.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:12.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:12.620726+0000) 2024-03-06T16:02:13.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:13.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:13.620929+0000) 2024-03-06T16:02:14.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:14.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:14.621144+0000) 2024-03-06T16:02:15.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:15.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:15.621388+0000) 2024-03-06T16:02:16.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:16.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:16.621614+0000) 2024-03-06T16:02:17.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:17.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:17.621854+0000) 2024-03-06T16:02:18.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:18.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:18.622093+0000) 2024-03-06T16:02:19.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:19.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:19.622320+0000) 2024-03-06T16:02:20.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:20.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:20.622542+0000) 2024-03-06T16:02:21.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:21.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:21.622763+0000) 2024-03-06T16:02:22.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:22.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:22.622992+0000) 2024-03-06T16:02:23.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:23.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:23.623215+0000) 2024-03-06T16:02:24.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:24.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:24.623453+0000) 2024-03-06T16:02:25.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:25.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:25.623638+0000) 2024-03-06T16:02:26.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:26.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:26.623886+0000) 2024-03-06T16:02:27.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:27.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:27.624152+0000) 2024-03-06T16:02:28.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:28.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:28.624374+0000) 2024-03-06T16:02:29.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:29.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:29.624581+0000) 2024-03-06T16:02:30.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:30.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:30.624852+0000) 2024-03-06T16:02:31.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:31.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:31.625043+0000) 2024-03-06T16:02:32.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:32.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:32.625232+0000) 2024-03-06T16:02:33.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:33.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:33.625448+0000) 2024-03-06T16:02:34.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:34.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:34.625730+0000) 2024-03-06T16:02:35.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:35.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:35.625943+0000) 2024-03-06T16:02:36.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:36.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:36.626179+0000) 2024-03-06T16:02:36.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:36.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:37.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:37.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:37.626358+0000) 2024-03-06T16:02:38.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:38.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:38.626563+0000) 2024-03-06T16:02:39.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:39.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:39.626850+0000) 2024-03-06T16:02:40.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:40.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:40.627052+0000) 2024-03-06T16:02:41.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:41.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:41.627295+0000) 2024-03-06T16:02:42.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:42.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:42.627500+0000) 2024-03-06T16:02:43.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:43.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:43.627731+0000) 2024-03-06T16:02:44.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:44.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:44.627950+0000) 2024-03-06T16:02:45.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:45.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:45.628229+0000) 2024-03-06T16:02:46.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:46.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:46.628434+0000) 2024-03-06T16:02:47.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:47.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:47.628654+0000) 2024-03-06T16:02:48.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:48.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:48.628942+0000) 2024-03-06T16:02:49.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:49.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:49.629191+0000) 2024-03-06T16:02:50.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:50.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:50.629371+0000) 2024-03-06T16:02:51.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:51.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:51.629599+0000) 2024-03-06T16:02:52.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:52.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:52.629835+0000) 2024-03-06T16:02:53.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:53.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:53.630065+0000) 2024-03-06T16:02:54.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:54.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:54.630293+0000) 2024-03-06T16:02:55.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:55.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:55.630523+0000) 2024-03-06T16:02:56.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:56.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:56.630736+0000) 2024-03-06T16:02:57.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:57.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:57.630924+0000) 2024-03-06T16:02:58.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:58.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:58.631108+0000) 2024-03-06T16:02:59.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:02:59.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:59.631353+0000) 2024-03-06T16:03:00.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:00.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:00.631567+0000) 2024-03-06T16:03:01.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:01.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:01.631803+0000) 2024-03-06T16:03:02.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:02.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:02.632025+0000) 2024-03-06T16:03:03.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:03.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:03.632258+0000) 2024-03-06T16:03:04.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:04.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:04.632489+0000) 2024-03-06T16:03:05.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:05.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:05.632767+0000) 2024-03-06T16:03:06.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:06.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:06.632997+0000) 2024-03-06T16:03:07.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:07.011 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:07.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:07.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:07.633224+0000) 2024-03-06T16:03:08.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:08.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:08.633448+0000) 2024-03-06T16:03:09.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:09.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:09.633632+0000) 2024-03-06T16:03:10.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:10.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:10.633883+0000) 2024-03-06T16:03:11.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:11.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:11.634095+0000) 2024-03-06T16:03:12.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:12.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:12.634314+0000) 2024-03-06T16:03:13.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:13.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:13.634583+0000) 2024-03-06T16:03:14.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:14.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:14.634885+0000) 2024-03-06T16:03:15.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:15.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:15.635189+0000) 2024-03-06T16:03:16.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:16.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:16.635444+0000) 2024-03-06T16:03:17.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:17.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:17.635729+0000) 2024-03-06T16:03:18.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:18.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:18.635931+0000) 2024-03-06T16:03:19.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:19.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:19.636122+0000) 2024-03-06T16:03:20.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:20.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:20.636369+0000) 2024-03-06T16:03:21.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:21.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:21.636585+0000) 2024-03-06T16:03:22.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:22.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:22.636837+0000) 2024-03-06T16:03:23.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:23.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:23.637031+0000) 2024-03-06T16:03:24.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:24.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:24.637263+0000) 2024-03-06T16:03:25.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:25.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:25.637493+0000) 2024-03-06T16:03:26.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:26.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:26.637694+0000) 2024-03-06T16:03:27.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:27.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:27.637885+0000) 2024-03-06T16:03:28.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:28.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:28.638106+0000) 2024-03-06T16:03:29.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:29.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:29.638337+0000) 2024-03-06T16:03:30.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:30.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:30.638539+0000) 2024-03-06T16:03:31.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:31.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:31.638794+0000) 2024-03-06T16:03:32.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:32.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:32.639019+0000) 2024-03-06T16:03:33.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:33.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:33.639261+0000) 2024-03-06T16:03:34.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:34.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:34.639501+0000) 2024-03-06T16:03:35.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:35.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:35.639790+0000) 2024-03-06T16:03:36.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:36.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:36.640026+0000) 2024-03-06T16:03:37.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:37.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:37.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:37.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:37.640232+0000) 2024-03-06T16:03:38.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:38.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:38.640478+0000) 2024-03-06T16:03:39.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:39.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:39.640760+0000) 2024-03-06T16:03:40.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:40.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:40.640965+0000) 2024-03-06T16:03:41.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:41.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:41.641187+0000) 2024-03-06T16:03:42.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:42.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:42.641401+0000) 2024-03-06T16:03:43.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:43.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:43.641625+0000) 2024-03-06T16:03:44.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:44.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:44.641823+0000) 2024-03-06T16:03:45.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:45.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:45.642027+0000) 2024-03-06T16:03:46.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:46.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:46.642209+0000) 2024-03-06T16:03:47.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:47.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:47.642433+0000) 2024-03-06T16:03:48.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:48.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:48.642616+0000) 2024-03-06T16:03:49.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:49.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:49.642870+0000) 2024-03-06T16:03:50.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:50.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:50.643126+0000) 2024-03-06T16:03:51.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:51.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:51.643408+0000) 2024-03-06T16:03:52.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:52.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:52.643643+0000) 2024-03-06T16:03:53.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:53.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:53.643918+0000) 2024-03-06T16:03:54.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:54.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:54.644161+0000) 2024-03-06T16:03:55.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:55.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:55.644438+0000) 2024-03-06T16:03:56.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:56.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:56.644635+0000) 2024-03-06T16:03:57.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:57.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:57.644938+0000) 2024-03-06T16:03:58.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:58.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:58.645179+0000) 2024-03-06T16:03:59.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:03:59.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:59.645403+0000) 2024-03-06T16:04:00.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:00.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:00.645593+0000) 2024-03-06T16:04:01.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:01.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:01.645813+0000) 2024-03-06T16:04:02.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:02.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:02.646068+0000) 2024-03-06T16:04:03.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:03.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:03.646387+0000) 2024-03-06T16:04:04.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:04.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:04.646574+0000) 2024-03-06T16:04:05.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:05.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:05.646807+0000) 2024-03-06T16:04:06.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:06.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:06.646964+0000) 2024-03-06T16:04:07.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:07.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:07.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:07.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:07.647216+0000) 2024-03-06T16:04:08.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:08.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:08.647461+0000) 2024-03-06T16:04:09.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:09.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:09.647728+0000) 2024-03-06T16:04:10.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:10.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:10.647988+0000) 2024-03-06T16:04:11.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:11.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:11.648161+0000) 2024-03-06T16:04:12.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:12.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:12.648402+0000) 2024-03-06T16:04:13.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:13.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:13.648645+0000) 2024-03-06T16:04:14.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:14.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:14.648933+0000) 2024-03-06T16:04:15.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:15.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:15.649175+0000) 2024-03-06T16:04:16.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:16.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:16.649421+0000) 2024-03-06T16:04:17.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:17.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:17.649661+0000) 2024-03-06T16:04:18.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:18.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:18.649947+0000) 2024-03-06T16:04:19.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:19.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:19.650193+0000) 2024-03-06T16:04:20.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:20.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:20.650444+0000) 2024-03-06T16:04:21.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:21.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:21.650639+0000) 2024-03-06T16:04:22.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:22.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:22.650881+0000) 2024-03-06T16:04:23.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:23.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:23.651117+0000) 2024-03-06T16:04:24.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:24.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:24.651353+0000) 2024-03-06T16:04:25.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:25.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:25.651603+0000) 2024-03-06T16:04:26.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:26.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:26.651844+0000) 2024-03-06T16:04:27.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:27.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:27.652021+0000) 2024-03-06T16:04:28.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:28.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:28.652191+0000) 2024-03-06T16:04:29.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:29.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:29.652394+0000) 2024-03-06T16:04:30.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:30.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:30.652613+0000) 2024-03-06T16:04:31.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:31.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:31.652820+0000) 2024-03-06T16:04:32.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:32.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:32.653034+0000) 2024-03-06T16:04:33.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:33.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:33.653286+0000) 2024-03-06T16:04:34.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:34.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:34.653455+0000) 2024-03-06T16:04:35.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:35.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:35.653714+0000) 2024-03-06T16:04:36.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:36.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:36.653977+0000) 2024-03-06T16:04:37.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:37.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:37.654198+0000) 2024-03-06T16:04:37.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:37.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:38.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:38.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:38.654442+0000) 2024-03-06T16:04:39.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:39.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:39.654759+0000) 2024-03-06T16:04:40.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:40.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:40.655067+0000) 2024-03-06T16:04:41.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:41.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:41.655350+0000) 2024-03-06T16:04:42.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:42.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:42.655588+0000) 2024-03-06T16:04:43.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:43.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:43.655802+0000) 2024-03-06T16:04:44.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:44.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:44.656027+0000) 2024-03-06T16:04:45.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:45.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:45.656413+0000) 2024-03-06T16:04:46.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:46.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:46.656618+0000) 2024-03-06T16:04:47.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:47.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:47.656835+0000) 2024-03-06T16:04:48.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:48.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:48.657073+0000) 2024-03-06T16:04:49.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:49.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:49.657316+0000) 2024-03-06T16:04:50.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:50.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:50.657525+0000) 2024-03-06T16:04:51.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:51.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:51.657797+0000) 2024-03-06T16:04:52.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:52.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:52.658061+0000) 2024-03-06T16:04:53.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:53.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:53.658312+0000) 2024-03-06T16:04:54.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:54.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:54.658481+0000) 2024-03-06T16:04:55.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:55.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:55.658735+0000) 2024-03-06T16:04:56.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:56.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:56.658952+0000) 2024-03-06T16:04:57.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:57.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:57.659228+0000) 2024-03-06T16:04:58.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:58.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:58.659469+0000) 2024-03-06T16:04:59.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:04:59.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:59.659720+0000) 2024-03-06T16:05:00.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:00.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:00.659957+0000) 2024-03-06T16:05:01.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:01.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:01.660146+0000) 2024-03-06T16:05:02.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:02.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:02.660365+0000) 2024-03-06T16:05:03.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:03.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:03.660591+0000) 2024-03-06T16:05:04.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:04.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:04.660784+0000) 2024-03-06T16:05:05.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:05.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:05.661029+0000) 2024-03-06T16:05:06.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:06.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:06.661207+0000) 2024-03-06T16:05:07.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:07.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:07.661433+0000) 2024-03-06T16:05:07.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:07.892 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:08.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:08.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:08.661659+0000) 2024-03-06T16:05:09.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:09.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:09.661930+0000) 2024-03-06T16:05:10.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:10.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:10.662130+0000) 2024-03-06T16:05:11.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:11.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:11.662346+0000) 2024-03-06T16:05:12.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:12.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:12.662559+0000) 2024-03-06T16:05:13.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:13.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:13.662768+0000) 2024-03-06T16:05:14.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:14.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:14.662992+0000) 2024-03-06T16:05:15.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:15.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:15.663234+0000) 2024-03-06T16:05:16.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:16.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:16.663454+0000) 2024-03-06T16:05:17.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:17.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:17.663655+0000) 2024-03-06T16:05:18.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:18.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:18.663920+0000) 2024-03-06T16:05:19.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:19.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:19.664103+0000) 2024-03-06T16:05:20.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:20.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:20.664346+0000) 2024-03-06T16:05:21.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:21.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:21.664517+0000) 2024-03-06T16:05:22.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:22.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:22.664737+0000) 2024-03-06T16:05:23.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:23.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:23.664966+0000) 2024-03-06T16:05:24.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:24.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:24.665141+0000) 2024-03-06T16:05:25.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:25.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:25.665388+0000) 2024-03-06T16:05:26.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:26.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:26.665630+0000) 2024-03-06T16:05:27.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:27.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:27.665857+0000) 2024-03-06T16:05:28.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:28.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:28.666085+0000) 2024-03-06T16:05:29.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:29.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:29.666314+0000) 2024-03-06T16:05:30.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:30.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:30.666560+0000) 2024-03-06T16:05:31.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:31.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:31.666782+0000) 2024-03-06T16:05:32.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:32.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:32.666995+0000) 2024-03-06T16:05:33.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:33.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:33.667227+0000) 2024-03-06T16:05:34.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:34.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:34.667412+0000) 2024-03-06T16:05:35.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:35.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:35.667658+0000) 2024-03-06T16:05:36.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:36.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:36.667910+0000) 2024-03-06T16:05:37.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:37.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:37.668105+0000) 2024-03-06T16:05:38.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:38.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:38.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:38.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:38.668338+0000) 2024-03-06T16:05:39.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:39.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:39.668569+0000) 2024-03-06T16:05:40.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:40.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:40.668812+0000) 2024-03-06T16:05:41.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:41.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:41.669044+0000) 2024-03-06T16:05:42.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:42.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:42.669257+0000) 2024-03-06T16:05:43.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:43.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:43.669478+0000) 2024-03-06T16:05:44.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:44.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:44.669731+0000) 2024-03-06T16:05:45.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:45.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:45.670024+0000) 2024-03-06T16:05:46.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:46.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:46.670252+0000) 2024-03-06T16:05:47.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:47.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:47.670426+0000) 2024-03-06T16:05:48.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:48.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:48.670592+0000) 2024-03-06T16:05:49.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:49.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:49.670810+0000) 2024-03-06T16:05:50.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:50.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:50.671056+0000) 2024-03-06T16:05:51.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:51.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:51.671276+0000) 2024-03-06T16:05:52.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:52.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:52.671494+0000) 2024-03-06T16:05:53.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:53.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:53.671767+0000) 2024-03-06T16:05:54.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:54.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:54.671993+0000) 2024-03-06T16:05:55.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:55.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:55.672224+0000) 2024-03-06T16:05:56.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:56.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:56.672443+0000) 2024-03-06T16:05:57.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:57.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:57.672699+0000) 2024-03-06T16:05:58.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:58.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:58.672940+0000) 2024-03-06T16:05:59.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:05:59.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:59.673168+0000) 2024-03-06T16:06:00.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:00.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:00.673413+0000) 2024-03-06T16:06:01.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:01.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:01.673645+0000) 2024-03-06T16:06:02.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:02.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:02.673914+0000) 2024-03-06T16:06:03.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:03.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:03.674096+0000) 2024-03-06T16:06:04.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:04.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:04.674323+0000) 2024-03-06T16:06:05.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:05.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:05.674610+0000) 2024-03-06T16:06:06.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:06.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:06.674834+0000) 2024-03-06T16:06:07.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:07.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:07.675065+0000) 2024-03-06T16:06:08.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:08.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:08.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:08.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:08.675298+0000) 2024-03-06T16:06:09.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:09.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:09.675506+0000) 2024-03-06T16:06:10.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:10.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:10.675771+0000) 2024-03-06T16:06:11.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:11.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:11.676001+0000) 2024-03-06T16:06:12.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:12.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:12.676219+0000) 2024-03-06T16:06:13.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:13.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:13.676441+0000) 2024-03-06T16:06:14.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:14.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:14.676694+0000) 2024-03-06T16:06:15.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:15.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:15.676897+0000) 2024-03-06T16:06:16.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:16.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:16.677120+0000) 2024-03-06T16:06:17.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:17.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:17.677345+0000) 2024-03-06T16:06:18.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:18.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:18.677539+0000) 2024-03-06T16:06:19.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:19.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:19.677792+0000) 2024-03-06T16:06:20.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:20.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:20.678009+0000) 2024-03-06T16:06:21.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:21.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:21.678248+0000) 2024-03-06T16:06:22.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:22.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:22.678465+0000) 2024-03-06T16:06:23.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:23.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:23.678746+0000) 2024-03-06T16:06:24.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:24.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:24.678984+0000) 2024-03-06T16:06:25.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:25.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:25.679243+0000) 2024-03-06T16:06:26.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:26.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:26.679492+0000) 2024-03-06T16:06:27.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:27.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:27.679781+0000) 2024-03-06T16:06:28.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:28.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:28.680012+0000) 2024-03-06T16:06:29.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:29.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:29.680190+0000) 2024-03-06T16:06:30.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:30.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:30.680393+0000) 2024-03-06T16:06:31.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:31.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:31.680617+0000) 2024-03-06T16:06:32.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:32.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:32.680821+0000) 2024-03-06T16:06:33.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:33.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:33.681106+0000) 2024-03-06T16:06:34.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:34.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:34.681321+0000) 2024-03-06T16:06:35.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:35.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:35.681568+0000) 2024-03-06T16:06:36.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:36.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:36.681794+0000) 2024-03-06T16:06:37.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:37.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:37.681954+0000) 2024-03-06T16:06:38.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:38.419 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:38.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:38.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:38.682141+0000) 2024-03-06T16:06:39.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:39.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:39.682364+0000) 2024-03-06T16:06:40.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:40.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:40.682585+0000) 2024-03-06T16:06:41.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:41.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:41.682839+0000) 2024-03-06T16:06:42.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:42.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:42.683053+0000) 2024-03-06T16:06:43.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:43.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:43.683253+0000) 2024-03-06T16:06:44.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:44.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:44.683435+0000) 2024-03-06T16:06:45.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:45.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:45.683710+0000) 2024-03-06T16:06:46.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:46.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:46.683941+0000) 2024-03-06T16:06:47.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:47.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:47.684149+0000) 2024-03-06T16:06:48.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:48.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:48.684366+0000) 2024-03-06T16:06:49.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:49.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:49.684596+0000) 2024-03-06T16:06:50.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:50.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:50.684842+0000) 2024-03-06T16:06:51.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:51.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:51.685043+0000) 2024-03-06T16:06:52.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:52.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:52.685268+0000) 2024-03-06T16:06:53.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:53.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:53.685453+0000) 2024-03-06T16:06:54.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:54.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:54.685636+0000) 2024-03-06T16:06:55.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:55.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:55.685918+0000) 2024-03-06T16:06:56.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:56.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:56.686104+0000) 2024-03-06T16:06:57.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:57.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:57.686284+0000) 2024-03-06T16:06:58.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:58.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:58.686461+0000) 2024-03-06T16:06:59.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:06:59.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:59.686621+0000) 2024-03-06T16:07:00.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:00.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:00.686806+0000) 2024-03-06T16:07:01.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:01.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:01.687032+0000) 2024-03-06T16:07:02.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:02.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:02.687190+0000) 2024-03-06T16:07:03.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:03.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:03.687444+0000) 2024-03-06T16:07:04.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:04.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:04.687740+0000) 2024-03-06T16:07:05.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:05.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:05.688035+0000) 2024-03-06T16:07:06.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:06.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:06.688306+0000) 2024-03-06T16:07:07.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:07.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:07.688496+0000) 2024-03-06T16:07:08.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:08.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:08.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:08.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:08.688687+0000) 2024-03-06T16:07:09.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:09.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:09.688912+0000) 2024-03-06T16:07:10.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:10.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:10.689165+0000) 2024-03-06T16:07:11.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:11.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:11.689368+0000) 2024-03-06T16:07:12.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:12.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:12.689599+0000) 2024-03-06T16:07:13.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:13.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:13.689850+0000) 2024-03-06T16:07:14.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:14.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:14.690103+0000) 2024-03-06T16:07:15.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:15.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:15.690334+0000) 2024-03-06T16:07:16.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:16.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:16.690556+0000) 2024-03-06T16:07:17.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:17.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:17.690784+0000) 2024-03-06T16:07:18.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:18.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:18.691010+0000) 2024-03-06T16:07:19.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:19.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:19.691235+0000) 2024-03-06T16:07:20.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:20.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:20.691474+0000) 2024-03-06T16:07:21.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:21.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:21.691755+0000) 2024-03-06T16:07:22.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:22.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:22.691984+0000) 2024-03-06T16:07:23.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:23.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:23.692169+0000) 2024-03-06T16:07:24.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:24.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:24.692388+0000) 2024-03-06T16:07:25.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:25.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:25.692620+0000) 2024-03-06T16:07:26.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:26.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:26.692902+0000) 2024-03-06T16:07:27.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:27.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:27.693126+0000) 2024-03-06T16:07:28.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:28.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:28.693312+0000) 2024-03-06T16:07:29.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:29.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:29.693538+0000) 2024-03-06T16:07:30.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:30.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:30.693732+0000) 2024-03-06T16:07:31.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:31.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:31.693926+0000) 2024-03-06T16:07:32.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:32.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:32.694143+0000) 2024-03-06T16:07:33.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:33.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:33.694280+0000) 2024-03-06T16:07:34.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:34.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:34.694460+0000) 2024-03-06T16:07:35.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:35.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:35.694654+0000) 2024-03-06T16:07:36.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:36.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:36.694885+0000) 2024-03-06T16:07:37.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:37.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:37.695113+0000) 2024-03-06T16:07:38.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:38.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:38.695301+0000) 2024-03-06T16:07:38.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:38.874 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:39.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:39.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:39.695551+0000) 2024-03-06T16:07:40.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:40.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:40.695797+0000) 2024-03-06T16:07:41.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:41.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:41.696023+0000) 2024-03-06T16:07:42.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:42.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:42.696161+0000) 2024-03-06T16:07:43.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:43.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:43.696292+0000) 2024-03-06T16:07:44.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:44.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:44.696488+0000) 2024-03-06T16:07:45.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:45.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:45.696755+0000) 2024-03-06T16:07:46.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:46.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:46.696941+0000) 2024-03-06T16:07:47.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:47.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:47.697215+0000) 2024-03-06T16:07:48.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:48.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:48.697494+0000) 2024-03-06T16:07:49.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:49.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:49.697804+0000) 2024-03-06T16:07:50.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:50.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:50.698117+0000) 2024-03-06T16:07:51.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:51.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:51.698367+0000) 2024-03-06T16:07:52.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:52.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:52.698578+0000) 2024-03-06T16:07:53.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:53.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:53.698800+0000) 2024-03-06T16:07:54.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:54.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:54.699026+0000) 2024-03-06T16:07:55.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:55.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:55.699217+0000) 2024-03-06T16:07:56.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:56.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:56.699449+0000) 2024-03-06T16:07:57.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:57.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:57.699696+0000) 2024-03-06T16:07:58.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:58.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:58.699899+0000) 2024-03-06T16:07:59.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:07:59.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:59.700132+0000) 2024-03-06T16:08:00.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:00.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:00.700406+0000) 2024-03-06T16:08:01.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:01.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:01.700744+0000) 2024-03-06T16:08:02.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:02.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:02.701028+0000) 2024-03-06T16:08:03.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:03.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:03.701231+0000) 2024-03-06T16:08:04.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:04.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:04.701429+0000) 2024-03-06T16:08:05.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:05.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:05.701705+0000) 2024-03-06T16:08:06.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:06.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:06.701935+0000) 2024-03-06T16:08:07.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:07.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:07.702162+0000) 2024-03-06T16:08:08.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:08.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:08.702366+0000) 2024-03-06T16:08:09.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:09.059 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:09.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:09.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:09.702617+0000) 2024-03-06T16:08:10.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:10.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:10.702810+0000) 2024-03-06T16:08:11.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:11.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:11.703024+0000) 2024-03-06T16:08:12.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:12.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:12.703196+0000) 2024-03-06T16:08:13.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:13.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:13.703382+0000) 2024-03-06T16:08:14.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:14.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:14.703697+0000) 2024-03-06T16:08:15.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:15.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:15.703907+0000) 2024-03-06T16:08:16.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:16.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:16.704151+0000) 2024-03-06T16:08:17.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:17.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:17.704377+0000) 2024-03-06T16:08:18.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:18.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:18.704608+0000) 2024-03-06T16:08:19.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:19.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:19.704833+0000) 2024-03-06T16:08:20.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:20.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:20.705068+0000) 2024-03-06T16:08:21.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:21.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:21.705253+0000) 2024-03-06T16:08:22.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:22.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:22.705453+0000) 2024-03-06T16:08:23.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:23.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:23.705714+0000) 2024-03-06T16:08:24.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:24.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:24.705909+0000) 2024-03-06T16:08:25.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:25.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:25.706112+0000) 2024-03-06T16:08:26.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:26.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:26.706332+0000) 2024-03-06T16:08:27.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:27.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:27.706545+0000) 2024-03-06T16:08:28.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:28.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:28.706776+0000) 2024-03-06T16:08:29.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:29.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:29.707004+0000) 2024-03-06T16:08:30.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:30.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:30.707253+0000) 2024-03-06T16:08:31.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:31.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:31.707481+0000) 2024-03-06T16:08:32.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:32.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:32.707733+0000) 2024-03-06T16:08:33.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:33.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:33.707953+0000) 2024-03-06T16:08:34.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:34.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:34.708160+0000) 2024-03-06T16:08:35.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:35.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:35.708395+0000) 2024-03-06T16:08:36.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:36.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:36.708624+0000) 2024-03-06T16:08:37.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:37.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:37.708842+0000) 2024-03-06T16:08:38.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:38.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:38.709013+0000) 2024-03-06T16:08:39.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:39.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:39.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:39.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:39.709192+0000) 2024-03-06T16:08:40.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:40.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:40.709395+0000) 2024-03-06T16:08:41.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:41.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:41.709645+0000) 2024-03-06T16:08:42.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:42.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:42.709911+0000) 2024-03-06T16:08:43.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:43.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:43.710166+0000) 2024-03-06T16:08:44.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:44.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:44.710398+0000) 2024-03-06T16:08:45.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:45.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:45.710640+0000) 2024-03-06T16:08:46.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:46.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:46.710927+0000) 2024-03-06T16:08:47.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:47.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:47.711158+0000) 2024-03-06T16:08:48.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:48.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:48.711400+0000) 2024-03-06T16:08:49.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:49.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:49.711648+0000) 2024-03-06T16:08:50.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:50.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:50.711978+0000) 2024-03-06T16:08:51.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:51.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:51.712224+0000) 2024-03-06T16:08:52.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:52.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:52.712462+0000) 2024-03-06T16:08:53.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:53.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:53.712721+0000) 2024-03-06T16:08:54.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:54.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:54.712918+0000) 2024-03-06T16:08:55.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:55.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:55.713165+0000) 2024-03-06T16:08:56.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:56.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:56.713406+0000) 2024-03-06T16:08:57.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:57.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:57.713648+0000) 2024-03-06T16:08:58.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:58.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:58.713908+0000) 2024-03-06T16:08:59.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:08:59.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:59.714094+0000) 2024-03-06T16:09:00.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:00.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:00.714338+0000) 2024-03-06T16:09:01.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:01.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:01.714567+0000) 2024-03-06T16:09:02.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:02.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:02.714791+0000) 2024-03-06T16:09:03.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:03.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:03.714983+0000) 2024-03-06T16:09:04.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:04.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:04.715204+0000) 2024-03-06T16:09:05.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:05.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:05.715439+0000) 2024-03-06T16:09:06.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:06.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:06.715609+0000) 2024-03-06T16:09:07.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:07.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:07.715820+0000) 2024-03-06T16:09:08.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:08.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:08.716057+0000) 2024-03-06T16:09:09.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:09.392 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:09.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:09.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:09.716285+0000) 2024-03-06T16:09:10.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:10.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:10.716534+0000) 2024-03-06T16:09:11.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:11.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:11.716804+0000) 2024-03-06T16:09:12.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:12.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:12.716973+0000) 2024-03-06T16:09:13.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:13.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:13.717186+0000) 2024-03-06T16:09:14.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:14.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:14.717428+0000) 2024-03-06T16:09:15.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:15.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:15.717661+0000) 2024-03-06T16:09:16.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:16.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:16.717890+0000) 2024-03-06T16:09:17.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:17.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:17.718114+0000) 2024-03-06T16:09:18.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:18.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:18.718318+0000) 2024-03-06T16:09:19.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:19.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:19.718568+0000) 2024-03-06T16:09:20.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:20.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:20.718807+0000) 2024-03-06T16:09:21.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:21.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:21.719040+0000) 2024-03-06T16:09:22.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:22.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:22.719222+0000) 2024-03-06T16:09:23.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:23.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:23.719455+0000) 2024-03-06T16:09:24.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:24.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:24.719702+0000) 2024-03-06T16:09:25.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:25.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:25.719938+0000) 2024-03-06T16:09:26.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:26.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:26.720120+0000) 2024-03-06T16:09:27.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:27.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:27.720350+0000) 2024-03-06T16:09:28.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:28.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:28.720566+0000) 2024-03-06T16:09:29.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:29.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:29.720791+0000) 2024-03-06T16:09:30.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:30.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:30.720989+0000) 2024-03-06T16:09:31.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:31.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:31.721177+0000) 2024-03-06T16:09:32.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:32.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:32.721397+0000) 2024-03-06T16:09:33.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:33.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:33.721628+0000) 2024-03-06T16:09:34.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:34.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:34.721866+0000) 2024-03-06T16:09:35.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:35.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:35.722101+0000) 2024-03-06T16:09:36.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:36.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:36.722333+0000) 2024-03-06T16:09:37.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:37.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:37.722553+0000) 2024-03-06T16:09:38.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:38.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:38.722774+0000) 2024-03-06T16:09:39.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:39.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:39.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:39.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:39.723021+0000) 2024-03-06T16:09:40.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:40.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:40.723253+0000) 2024-03-06T16:09:41.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:41.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:41.723478+0000) 2024-03-06T16:09:42.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:42.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:42.723660+0000) 2024-03-06T16:09:43.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:43.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:43.723922+0000) 2024-03-06T16:09:44.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:44.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:44.724178+0000) 2024-03-06T16:09:45.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:45.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:45.724430+0000) 2024-03-06T16:09:46.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:46.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:46.724691+0000) 2024-03-06T16:09:47.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:47.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:47.724920+0000) 2024-03-06T16:09:48.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:48.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:48.725186+0000) 2024-03-06T16:09:49.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:49.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:49.725472+0000) 2024-03-06T16:09:50.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:50.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:50.725762+0000) 2024-03-06T16:09:51.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:51.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:51.725991+0000) 2024-03-06T16:09:52.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:52.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:52.726164+0000) 2024-03-06T16:09:53.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:53.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:53.726309+0000) 2024-03-06T16:09:54.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:54.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:54.726501+0000) 2024-03-06T16:09:55.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:55.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:55.726827+0000) 2024-03-06T16:09:56.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:56.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:56.727051+0000) 2024-03-06T16:09:57.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:57.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:57.727269+0000) 2024-03-06T16:09:58.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:58.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:58.727493+0000) 2024-03-06T16:09:59.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:09:59.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:59.727757+0000) 2024-03-06T16:10:00.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:00.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:00.728000+0000) 2024-03-06T16:10:01.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:01.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:01.728256+0000) 2024-03-06T16:10:02.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:02.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:02.728498+0000) 2024-03-06T16:10:03.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:03.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:03.728771+0000) 2024-03-06T16:10:04.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:04.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:04.728954+0000) 2024-03-06T16:10:05.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:05.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:05.729156+0000) 2024-03-06T16:10:06.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:06.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:06.729350+0000) 2024-03-06T16:10:07.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:07.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:07.729589+0000) 2024-03-06T16:10:08.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:08.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:08.729847+0000) 2024-03-06T16:10:09.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:09.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:09.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:09.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:09.730027+0000) 2024-03-06T16:10:10.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:10.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:10.730257+0000) 2024-03-06T16:10:11.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:11.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:11.730510+0000) 2024-03-06T16:10:12.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:12.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:12.730777+0000) 2024-03-06T16:10:13.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:13.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:13.731024+0000) 2024-03-06T16:10:14.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:14.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:14.731263+0000) 2024-03-06T16:10:15.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:15.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:15.731465+0000) 2024-03-06T16:10:16.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:16.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:16.731754+0000) 2024-03-06T16:10:17.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:17.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:17.731983+0000) 2024-03-06T16:10:18.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:18.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:18.732208+0000) 2024-03-06T16:10:19.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:19.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:19.732399+0000) 2024-03-06T16:10:20.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:20.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:20.732571+0000) 2024-03-06T16:10:21.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:21.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:21.732807+0000) 2024-03-06T16:10:22.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:22.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:22.732936+0000) 2024-03-06T16:10:23.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:23.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:23.733161+0000) 2024-03-06T16:10:24.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:24.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:24.733389+0000) 2024-03-06T16:10:25.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:25.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:25.733623+0000) 2024-03-06T16:10:26.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:26.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:26.733821+0000) 2024-03-06T16:10:27.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:27.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:27.734056+0000) 2024-03-06T16:10:28.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:28.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:28.734273+0000) 2024-03-06T16:10:29.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:29.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:29.734531+0000) 2024-03-06T16:10:30.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:30.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:30.734799+0000) 2024-03-06T16:10:31.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:31.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:31.735068+0000) 2024-03-06T16:10:32.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:32.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:32.735331+0000) 2024-03-06T16:10:33.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:33.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:33.735540+0000) 2024-03-06T16:10:34.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:34.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:34.735784+0000) 2024-03-06T16:10:35.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:35.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:35.736020+0000) 2024-03-06T16:10:36.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:36.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:36.736250+0000) 2024-03-06T16:10:37.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:37.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:37.736477+0000) 2024-03-06T16:10:38.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:38.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:38.736738+0000) 2024-03-06T16:10:39.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:39.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:39.736950+0000) 2024-03-06T16:10:39.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:39.846 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:40.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:40.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:40.737187+0000) 2024-03-06T16:10:41.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:41.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:41.737448+0000) 2024-03-06T16:10:42.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:42.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:42.737714+0000) 2024-03-06T16:10:43.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:43.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:43.738041+0000) 2024-03-06T16:10:44.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:44.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:44.738304+0000) 2024-03-06T16:10:45.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:45.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:45.738578+0000) 2024-03-06T16:10:46.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:46.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:46.738803+0000) 2024-03-06T16:10:47.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:47.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:47.739009+0000) 2024-03-06T16:10:48.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:48.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:48.739253+0000) 2024-03-06T16:10:49.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:49.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:49.739457+0000) 2024-03-06T16:10:50.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:50.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:50.739734+0000) 2024-03-06T16:10:51.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:51.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:51.739956+0000) 2024-03-06T16:10:52.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:52.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:52.740187+0000) 2024-03-06T16:10:53.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:53.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:53.740453+0000) 2024-03-06T16:10:54.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:54.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:54.740651+0000) 2024-03-06T16:10:55.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:55.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:55.740879+0000) 2024-03-06T16:10:56.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:56.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:56.741144+0000) 2024-03-06T16:10:57.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:57.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:57.741378+0000) 2024-03-06T16:10:58.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:58.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:58.741643+0000) 2024-03-06T16:10:59.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:10:59.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:59.741950+0000) 2024-03-06T16:11:00.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:00.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:00.742220+0000) 2024-03-06T16:11:01.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:01.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:01.742497+0000) 2024-03-06T16:11:02.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:02.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:02.742809+0000) 2024-03-06T16:11:03.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:03.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:03.743088+0000) 2024-03-06T16:11:04.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:04.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:04.743354+0000) 2024-03-06T16:11:05.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:05.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:05.743629+0000) 2024-03-06T16:11:06.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:06.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:06.743940+0000) 2024-03-06T16:11:07.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:07.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:07.744217+0000) 2024-03-06T16:11:08.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:08.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:08.744487+0000) 2024-03-06T16:11:09.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:09.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:09.744822+0000) 2024-03-06T16:11:10.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:10.034 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:10.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:10.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:10.745056+0000) 2024-03-06T16:11:11.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:11.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:11.745331+0000) 2024-03-06T16:11:12.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:12.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:12.745603+0000) 2024-03-06T16:11:13.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:13.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:13.745804+0000) 2024-03-06T16:11:14.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:14.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:14.746030+0000) 2024-03-06T16:11:15.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:15.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:15.746217+0000) 2024-03-06T16:11:16.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:16.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:16.746438+0000) 2024-03-06T16:11:17.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:17.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:17.746708+0000) 2024-03-06T16:11:18.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:18.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:18.746934+0000) 2024-03-06T16:11:19.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:19.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:19.747197+0000) 2024-03-06T16:11:20.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:20.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:20.747503+0000) 2024-03-06T16:11:21.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:21.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:21.747761+0000) 2024-03-06T16:11:22.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:22.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:22.748024+0000) 2024-03-06T16:11:23.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:23.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:23.748290+0000) 2024-03-06T16:11:24.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:24.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:24.748511+0000) 2024-03-06T16:11:25.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:25.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:25.748781+0000) 2024-03-06T16:11:26.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:26.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:26.749041+0000) 2024-03-06T16:11:27.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:27.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:27.749271+0000) 2024-03-06T16:11:28.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:28.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:28.749492+0000) 2024-03-06T16:11:29.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:29.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:29.749789+0000) 2024-03-06T16:11:30.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:30.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:30.750080+0000) 2024-03-06T16:11:31.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:31.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:31.750370+0000) 2024-03-06T16:11:32.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:32.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:32.750604+0000) 2024-03-06T16:11:33.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:33.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:33.750854+0000) 2024-03-06T16:11:34.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:34.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:34.751081+0000) 2024-03-06T16:11:35.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:35.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:35.751264+0000) 2024-03-06T16:11:36.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:36.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:36.751494+0000) 2024-03-06T16:11:37.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:37.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:37.751853+0000) 2024-03-06T16:11:38.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:38.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:38.752093+0000) 2024-03-06T16:11:39.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:39.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:39.752377+0000) 2024-03-06T16:11:40.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:40.254 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:40.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:40.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:40.752631+0000) 2024-03-06T16:11:41.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:41.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:41.752875+0000) 2024-03-06T16:11:42.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:42.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:42.753100+0000) 2024-03-06T16:11:43.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:43.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:43.753327+0000) 2024-03-06T16:11:44.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:44.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:44.753543+0000) 2024-03-06T16:11:45.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:45.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:45.753787+0000) 2024-03-06T16:11:46.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:46.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:46.753990+0000) 2024-03-06T16:11:47.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:47.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:47.754183+0000) 2024-03-06T16:11:48.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:48.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:48.754409+0000) 2024-03-06T16:11:49.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:49.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:49.754730+0000) 2024-03-06T16:11:50.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:50.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:50.755017+0000) 2024-03-06T16:11:51.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:51.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:51.755233+0000) 2024-03-06T16:11:52.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:52.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:52.755459+0000) 2024-03-06T16:11:53.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:53.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:53.755697+0000) 2024-03-06T16:11:54.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:54.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:54.755942+0000) 2024-03-06T16:11:55.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:55.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:55.756164+0000) 2024-03-06T16:11:56.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:56.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:56.756439+0000) 2024-03-06T16:11:57.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:57.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:57.756743+0000) 2024-03-06T16:11:58.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:58.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:58.757016+0000) 2024-03-06T16:11:59.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:11:59.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:59.757250+0000) 2024-03-06T16:12:00.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:00.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:00.757477+0000) 2024-03-06T16:12:01.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:01.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:01.757754+0000) 2024-03-06T16:12:02.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:02.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:02.757983+0000) 2024-03-06T16:12:03.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:03.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:03.758168+0000) 2024-03-06T16:12:04.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:04.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:04.758381+0000) 2024-03-06T16:12:05.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:05.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:05.758560+0000) 2024-03-06T16:12:06.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:06.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:06.758802+0000) 2024-03-06T16:12:07.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:07.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:07.758996+0000) 2024-03-06T16:12:08.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:08.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:08.759242+0000) 2024-03-06T16:12:09.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:09.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:09.759445+0000) 2024-03-06T16:12:10.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:10.466 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:10.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:10.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:10.759716+0000) 2024-03-06T16:12:11.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:11.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:11.760012+0000) 2024-03-06T16:12:12.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:12.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:12.760207+0000) 2024-03-06T16:12:13.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:13.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:13.760423+0000) 2024-03-06T16:12:14.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:14.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:14.760642+0000) 2024-03-06T16:12:15.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:15.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:15.760869+0000) 2024-03-06T16:12:16.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:16.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:16.761107+0000) 2024-03-06T16:12:17.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:17.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:17.761282+0000) 2024-03-06T16:12:18.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:18.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:18.761509+0000) 2024-03-06T16:12:19.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:19.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:19.761769+0000) 2024-03-06T16:12:20.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:20.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:20.761999+0000) 2024-03-06T16:12:21.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:21.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:21.762204+0000) 2024-03-06T16:12:22.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:22.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:22.762316+0000) 2024-03-06T16:12:23.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:23.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:23.762539+0000) 2024-03-06T16:12:24.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:24.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:24.762763+0000) 2024-03-06T16:12:25.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:25.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:25.763008+0000) 2024-03-06T16:12:26.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:26.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:26.763222+0000) 2024-03-06T16:12:27.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:27.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:27.763399+0000) 2024-03-06T16:12:28.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:28.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:28.763609+0000) 2024-03-06T16:12:29.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:29.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:29.763817+0000) 2024-03-06T16:12:30.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:30.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:30.764060+0000) 2024-03-06T16:12:31.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:31.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:31.764223+0000) 2024-03-06T16:12:32.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:32.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:32.764477+0000) 2024-03-06T16:12:33.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:33.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:33.764754+0000) 2024-03-06T16:12:34.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:34.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:34.764976+0000) 2024-03-06T16:12:35.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:35.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:35.765184+0000) 2024-03-06T16:12:36.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:36.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:36.765428+0000) 2024-03-06T16:12:37.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:37.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:37.765653+0000) 2024-03-06T16:12:38.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:38.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:38.765913+0000) 2024-03-06T16:12:39.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:39.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:39.766136+0000) 2024-03-06T16:12:40.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:40.688 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:40.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:40.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:40.766299+0000) 2024-03-06T16:12:41.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:41.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:41.766513+0000) 2024-03-06T16:12:42.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:42.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:42.766726+0000) 2024-03-06T16:12:43.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:43.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:43.766921+0000) 2024-03-06T16:12:44.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:44.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:44.767114+0000) 2024-03-06T16:12:45.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:45.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:45.767341+0000) 2024-03-06T16:12:46.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:46.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:46.767626+0000) 2024-03-06T16:12:47.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:47.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:47.767895+0000) 2024-03-06T16:12:48.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:48.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:48.768140+0000) 2024-03-06T16:12:49.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:49.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:49.768336+0000) 2024-03-06T16:12:50.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:50.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:50.768577+0000) 2024-03-06T16:12:51.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:51.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:51.768825+0000) 2024-03-06T16:12:52.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:52.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:52.769041+0000) 2024-03-06T16:12:53.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:53.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:53.769270+0000) 2024-03-06T16:12:54.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:54.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:54.769494+0000) 2024-03-06T16:12:55.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:55.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:55.769743+0000) 2024-03-06T16:12:56.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:56.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:56.769936+0000) 2024-03-06T16:12:57.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:57.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:57.770138+0000) 2024-03-06T16:12:58.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:58.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:58.770323+0000) 2024-03-06T16:12:59.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:12:59.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:59.770502+0000) 2024-03-06T16:13:00.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:00.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:00.770806+0000) 2024-03-06T16:13:01.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:01.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:01.771097+0000) 2024-03-06T16:13:02.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:02.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:02.771334+0000) 2024-03-06T16:13:03.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:03.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:03.771604+0000) 2024-03-06T16:13:04.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:04.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:04.771828+0000) 2024-03-06T16:13:05.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:05.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:05.772026+0000) 2024-03-06T16:13:06.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:06.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:06.772214+0000) 2024-03-06T16:13:07.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:07.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:07.772395+0000) 2024-03-06T16:13:08.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:08.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:08.772555+0000) 2024-03-06T16:13:09.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:09.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:09.772719+0000) 2024-03-06T16:13:10.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:10.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:10.772855+0000) 2024-03-06T16:13:10.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:10.874 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:11.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:11.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:11.772982+0000) 2024-03-06T16:13:12.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:12.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:12.773196+0000) 2024-03-06T16:13:13.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:13.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:13.773435+0000) 2024-03-06T16:13:14.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:14.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:14.773618+0000) 2024-03-06T16:13:15.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:15.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:15.773843+0000) 2024-03-06T16:13:16.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:16.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:16.774023+0000) 2024-03-06T16:13:17.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:17.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:17.774243+0000) 2024-03-06T16:13:18.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:18.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:18.774469+0000) 2024-03-06T16:13:19.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:19.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:19.774736+0000) 2024-03-06T16:13:20.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:20.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:20.774962+0000) 2024-03-06T16:13:21.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:21.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:21.775152+0000) 2024-03-06T16:13:22.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:22.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:22.775309+0000) 2024-03-06T16:13:23.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:23.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:23.775492+0000) 2024-03-06T16:13:24.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:24.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:24.775753+0000) 2024-03-06T16:13:25.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:25.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:25.775949+0000) 2024-03-06T16:13:26.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:26.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:26.776137+0000) 2024-03-06T16:13:27.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:27.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:27.776366+0000) 2024-03-06T16:13:28.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:28.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:28.776539+0000) 2024-03-06T16:13:29.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:29.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:29.776770+0000) 2024-03-06T16:13:30.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:30.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:30.776957+0000) 2024-03-06T16:13:31.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:31.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:31.777203+0000) 2024-03-06T16:13:32.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:32.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:32.777426+0000) 2024-03-06T16:13:33.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:33.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:33.777650+0000) 2024-03-06T16:13:34.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:34.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:34.777932+0000) 2024-03-06T16:13:35.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:35.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:35.778112+0000) 2024-03-06T16:13:36.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:36.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:36.778350+0000) 2024-03-06T16:13:37.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:37.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:37.778543+0000) 2024-03-06T16:13:38.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:38.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:38.778783+0000) 2024-03-06T16:13:39.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:39.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:39.779008+0000) 2024-03-06T16:13:40.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:40.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:40.779241+0000) 2024-03-06T16:13:41.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:41.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:41.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:41.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:41.779479+0000) 2024-03-06T16:13:42.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:42.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:42.779743+0000) 2024-03-06T16:13:43.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:43.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:43.779972+0000) 2024-03-06T16:13:44.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:44.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:44.780162+0000) 2024-03-06T16:13:45.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:45.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:45.780394+0000) 2024-03-06T16:13:46.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:46.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:46.780547+0000) 2024-03-06T16:13:47.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:47.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:47.780773+0000) 2024-03-06T16:13:48.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:48.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:48.780966+0000) 2024-03-06T16:13:49.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:49.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:49.781195+0000) 2024-03-06T16:13:50.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:50.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:50.781395+0000) 2024-03-06T16:13:51.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:51.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:51.781697+0000) 2024-03-06T16:13:52.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:52.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:52.781914+0000) 2024-03-06T16:13:53.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:53.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:53.782100+0000) 2024-03-06T16:13:54.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:54.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:54.782320+0000) 2024-03-06T16:13:55.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:55.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:55.782537+0000) 2024-03-06T16:13:56.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:56.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:56.782796+0000) 2024-03-06T16:13:57.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:57.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:57.783015+0000) 2024-03-06T16:13:58.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:58.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:58.783205+0000) 2024-03-06T16:13:59.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:13:59.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:59.783431+0000) 2024-03-06T16:14:00.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:00.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:00.783655+0000) 2024-03-06T16:14:01.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:01.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:01.783880+0000) 2024-03-06T16:14:02.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:02.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:02.784069+0000) 2024-03-06T16:14:03.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:03.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:03.784259+0000) 2024-03-06T16:14:04.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:04.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:04.784432+0000) 2024-03-06T16:14:05.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:05.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:05.784658+0000) 2024-03-06T16:14:06.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:06.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:06.785001+0000) 2024-03-06T16:14:07.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:07.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:07.785164+0000) 2024-03-06T16:14:08.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:08.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:08.785393+0000) 2024-03-06T16:14:09.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:09.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:09.785618+0000) 2024-03-06T16:14:10.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:10.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:10.785856+0000) 2024-03-06T16:14:11.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:11.159 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:11.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:11.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:11.786118+0000) 2024-03-06T16:14:12.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:12.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:12.786360+0000) 2024-03-06T16:14:13.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:13.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:13.786603+0000) 2024-03-06T16:14:14.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:14.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:14.786841+0000) 2024-03-06T16:14:15.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:15.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:15.787075+0000) 2024-03-06T16:14:16.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:16.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:16.787291+0000) 2024-03-06T16:14:17.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:17.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:17.787544+0000) 2024-03-06T16:14:18.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:18.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:18.787779+0000) 2024-03-06T16:14:19.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:19.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:19.787970+0000) 2024-03-06T16:14:20.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:20.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:20.788199+0000) 2024-03-06T16:14:21.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:21.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:21.788401+0000) 2024-03-06T16:14:22.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:22.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:22.788609+0000) 2024-03-06T16:14:23.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:23.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:23.788779+0000) 2024-03-06T16:14:24.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:24.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:24.788993+0000) 2024-03-06T16:14:25.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:25.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:25.789210+0000) 2024-03-06T16:14:26.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:26.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:26.789442+0000) 2024-03-06T16:14:27.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:27.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:27.789687+0000) 2024-03-06T16:14:28.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:28.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:28.789906+0000) 2024-03-06T16:14:29.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:29.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:29.790127+0000) 2024-03-06T16:14:30.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:30.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:30.790323+0000) 2024-03-06T16:14:31.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:31.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:31.790565+0000) 2024-03-06T16:14:32.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:32.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:32.790781+0000) 2024-03-06T16:14:33.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:33.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:33.791005+0000) 2024-03-06T16:14:34.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:34.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:34.791187+0000) 2024-03-06T16:14:35.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:35.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:35.791419+0000) 2024-03-06T16:14:36.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:36.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:36.791653+0000) 2024-03-06T16:14:37.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:37.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:37.791918+0000) 2024-03-06T16:14:38.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:38.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:38.792145+0000) 2024-03-06T16:14:39.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:39.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:39.792352+0000) 2024-03-06T16:14:40.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:40.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:40.792611+0000) 2024-03-06T16:14:41.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:41.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:41.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:41.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:41.792852+0000) 2024-03-06T16:14:42.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:42.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:42.793073+0000) 2024-03-06T16:14:43.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:43.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:43.793302+0000) 2024-03-06T16:14:44.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:44.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:44.793479+0000) 2024-03-06T16:14:45.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:45.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:45.793784+0000) 2024-03-06T16:14:46.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:46.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:46.793976+0000) 2024-03-06T16:14:47.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:47.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:47.794164+0000) 2024-03-06T16:14:48.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:48.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:48.794365+0000) 2024-03-06T16:14:49.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:49.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:49.794515+0000) 2024-03-06T16:14:50.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:50.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:50.794744+0000) 2024-03-06T16:14:51.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:51.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:51.794980+0000) 2024-03-06T16:14:52.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:52.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:52.795170+0000) 2024-03-06T16:14:53.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:53.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:53.795406+0000) 2024-03-06T16:14:54.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:54.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:54.795658+0000) 2024-03-06T16:14:55.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:55.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:55.795912+0000) 2024-03-06T16:14:56.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:56.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:56.796069+0000) 2024-03-06T16:14:57.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:57.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:57.796288+0000) 2024-03-06T16:14:58.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:58.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:58.796475+0000) 2024-03-06T16:14:59.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:14:59.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:59.796700+0000) 2024-03-06T16:15:00.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:00.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:00.796953+0000) 2024-03-06T16:15:01.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:01.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:01.797178+0000) 2024-03-06T16:15:02.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:02.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:02.797330+0000) 2024-03-06T16:15:03.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:03.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:03.797556+0000) 2024-03-06T16:15:04.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:04.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:04.797780+0000) 2024-03-06T16:15:05.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:05.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:05.798005+0000) 2024-03-06T16:15:06.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:06.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:06.798235+0000) 2024-03-06T16:15:07.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:07.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:07.798418+0000) 2024-03-06T16:15:08.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:08.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:08.798605+0000) 2024-03-06T16:15:09.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:09.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:09.798861+0000) 2024-03-06T16:15:10.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:10.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:10.799109+0000) 2024-03-06T16:15:11.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:11.524 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:11.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:11.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:11.799304+0000) 2024-03-06T16:15:12.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:12.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:12.799528+0000) 2024-03-06T16:15:13.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:13.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:13.799793+0000) 2024-03-06T16:15:14.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:14.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:14.799974+0000) 2024-03-06T16:15:15.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:15.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:15.800204+0000) 2024-03-06T16:15:16.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:16.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:16.800426+0000) 2024-03-06T16:15:17.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:17.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:17.800648+0000) 2024-03-06T16:15:18.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:18.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:18.800914+0000) 2024-03-06T16:15:19.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:19.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:19.801079+0000) 2024-03-06T16:15:20.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:20.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:20.801284+0000) 2024-03-06T16:15:21.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:21.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:21.801477+0000) 2024-03-06T16:15:22.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:22.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:22.801753+0000) 2024-03-06T16:15:23.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:23.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:23.801989+0000) 2024-03-06T16:15:24.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:24.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:24.802171+0000) 2024-03-06T16:15:25.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:25.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:25.802346+0000) 2024-03-06T16:15:26.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:26.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:26.802574+0000) 2024-03-06T16:15:27.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:27.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:27.802794+0000) 2024-03-06T16:15:28.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:28.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:28.803022+0000) 2024-03-06T16:15:29.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:29.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:29.803208+0000) 2024-03-06T16:15:30.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:30.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:30.803425+0000) 2024-03-06T16:15:31.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:31.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:31.803650+0000) 2024-03-06T16:15:32.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:32.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:32.803887+0000) 2024-03-06T16:15:33.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:33.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:33.804083+0000) 2024-03-06T16:15:34.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:34.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:34.804276+0000) 2024-03-06T16:15:35.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:35.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:35.804505+0000) 2024-03-06T16:15:36.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:36.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:36.804769+0000) 2024-03-06T16:15:37.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:37.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:37.804997+0000) 2024-03-06T16:15:38.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:38.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:38.805135+0000) 2024-03-06T16:15:39.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:39.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:39.805311+0000) 2024-03-06T16:15:40.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:40.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:40.805523+0000) 2024-03-06T16:15:41.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:41.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:41.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:41.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:41.805811+0000) 2024-03-06T16:15:42.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:42.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:42.806038+0000) 2024-03-06T16:15:43.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:43.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:43.806219+0000) 2024-03-06T16:15:44.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:44.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:44.806439+0000) 2024-03-06T16:15:45.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:45.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:45.806696+0000) 2024-03-06T16:15:46.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:46.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:46.806918+0000) 2024-03-06T16:15:47.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:47.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:47.807191+0000) 2024-03-06T16:15:48.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:48.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:48.807412+0000) 2024-03-06T16:15:49.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:49.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:49.807641+0000) 2024-03-06T16:15:50.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:50.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:50.807937+0000) 2024-03-06T16:15:51.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:51.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:51.808122+0000) 2024-03-06T16:15:52.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:52.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:52.808339+0000) 2024-03-06T16:15:53.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:53.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:53.808572+0000) 2024-03-06T16:15:54.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:54.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:54.808802+0000) 2024-03-06T16:15:55.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:55.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:55.808994+0000) 2024-03-06T16:15:56.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:56.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:56.809212+0000) 2024-03-06T16:15:57.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:57.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:57.809397+0000) 2024-03-06T16:15:58.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:58.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:58.809584+0000) 2024-03-06T16:15:59.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:15:59.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:59.809813+0000) 2024-03-06T16:16:00.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:00.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:00.810034+0000) 2024-03-06T16:16:01.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:01.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:01.810261+0000) 2024-03-06T16:16:02.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:02.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:02.810484+0000) 2024-03-06T16:16:03.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:03.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:03.810696+0000) 2024-03-06T16:16:04.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:04.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:04.810916+0000) 2024-03-06T16:16:05.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:05.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:05.811098+0000) 2024-03-06T16:16:06.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:06.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:06.811322+0000) 2024-03-06T16:16:07.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:07.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:07.811550+0000) 2024-03-06T16:16:08.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:08.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:08.811717+0000) 2024-03-06T16:16:09.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:09.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:09.811944+0000) 2024-03-06T16:16:10.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:10.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:10.812181+0000) 2024-03-06T16:16:11.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:11.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:11.812409+0000) 2024-03-06T16:16:11.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:11.959 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:12.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:12.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:12.812635+0000) 2024-03-06T16:16:13.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:13.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:13.812863+0000) 2024-03-06T16:16:14.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:14.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:14.813051+0000) 2024-03-06T16:16:15.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:15.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:15.813272+0000) 2024-03-06T16:16:16.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:16.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:16.813500+0000) 2024-03-06T16:16:17.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:17.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:17.813769+0000) 2024-03-06T16:16:18.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:18.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:18.813996+0000) 2024-03-06T16:16:19.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:19.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:19.814180+0000) 2024-03-06T16:16:20.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:20.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:20.814414+0000) 2024-03-06T16:16:21.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:21.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:21.814636+0000) 2024-03-06T16:16:22.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:22.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:22.814859+0000) 2024-03-06T16:16:23.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:23.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:23.815085+0000) 2024-03-06T16:16:24.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:24.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:24.815303+0000) 2024-03-06T16:16:25.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:25.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:25.815528+0000) 2024-03-06T16:16:26.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:26.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:26.815790+0000) 2024-03-06T16:16:27.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:27.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:27.816018+0000) 2024-03-06T16:16:28.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:28.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:28.816255+0000) 2024-03-06T16:16:29.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:29.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:29.816482+0000) 2024-03-06T16:16:30.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:30.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:30.816720+0000) 2024-03-06T16:16:31.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:31.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:31.816952+0000) 2024-03-06T16:16:32.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:32.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:32.817180+0000) 2024-03-06T16:16:33.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:33.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:33.817406+0000) 2024-03-06T16:16:34.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:34.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:34.817602+0000) 2024-03-06T16:16:35.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:35.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:35.817827+0000) 2024-03-06T16:16:36.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:36.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:36.818052+0000) 2024-03-06T16:16:37.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:37.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:37.818280+0000) 2024-03-06T16:16:38.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:38.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:38.818504+0000) 2024-03-06T16:16:39.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:39.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:39.818827+0000) 2024-03-06T16:16:40.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:40.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:40.819179+0000) 2024-03-06T16:16:41.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:41.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:41.819453+0000) 2024-03-06T16:16:42.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:42.164 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:42.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:42.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:42.819775+0000) 2024-03-06T16:16:43.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:43.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:43.820046+0000) 2024-03-06T16:16:44.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:44.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:44.820321+0000) 2024-03-06T16:16:45.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:45.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:45.820589+0000) 2024-03-06T16:16:46.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:46.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:46.820858+0000) 2024-03-06T16:16:47.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:47.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:47.821103+0000) 2024-03-06T16:16:48.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:48.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:48.821388+0000) 2024-03-06T16:16:49.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:49.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:49.821716+0000) 2024-03-06T16:16:50.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:50.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:50.822026+0000) 2024-03-06T16:16:51.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:51.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:51.822242+0000) 2024-03-06T16:16:52.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:52.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:52.822452+0000) 2024-03-06T16:16:53.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:53.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:53.822769+0000) 2024-03-06T16:16:54.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:54.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:54.823049+0000) 2024-03-06T16:16:55.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:55.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:55.823199+0000) 2024-03-06T16:16:56.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:56.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:56.823462+0000) 2024-03-06T16:16:57.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:57.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:57.823697+0000) 2024-03-06T16:16:58.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:58.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:58.823924+0000) 2024-03-06T16:16:59.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:16:59.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:59.824153+0000) 2024-03-06T16:17:00.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:00.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:00.824390+0000) 2024-03-06T16:17:01.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:01.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:01.824618+0000) 2024-03-06T16:17:02.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:02.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:02.824853+0000) 2024-03-06T16:17:03.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:03.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:03.825087+0000) 2024-03-06T16:17:04.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:04.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:04.825293+0000) 2024-03-06T16:17:05.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:05.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:05.825501+0000) 2024-03-06T16:17:06.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:06.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:06.825798+0000) 2024-03-06T16:17:07.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:07.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:07.826061+0000) 2024-03-06T16:17:08.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:08.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:08.826340+0000) 2024-03-06T16:17:09.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:09.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:09.826610+0000) 2024-03-06T16:17:10.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:10.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:10.826900+0000) 2024-03-06T16:17:11.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:11.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:11.827180+0000) 2024-03-06T16:17:12.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:12.359 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:12.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:12.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:12.827453+0000) 2024-03-06T16:17:13.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:13.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:13.827773+0000) 2024-03-06T16:17:14.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:14.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:14.827986+0000) 2024-03-06T16:17:15.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:15.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:15.828190+0000) 2024-03-06T16:17:16.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:16.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:16.828404+0000) 2024-03-06T16:17:17.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:17.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:17.828633+0000) 2024-03-06T16:17:18.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:18.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:18.828872+0000) 2024-03-06T16:17:19.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:19.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:19.829105+0000) 2024-03-06T16:17:20.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:20.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:20.829338+0000) 2024-03-06T16:17:21.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:21.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:21.829557+0000) 2024-03-06T16:17:22.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:22.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:22.829791+0000) 2024-03-06T16:17:23.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:23.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:23.830082+0000) 2024-03-06T16:17:24.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:24.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:24.830279+0000) 2024-03-06T16:17:25.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:25.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:25.830484+0000) 2024-03-06T16:17:26.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:26.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:26.830765+0000) 2024-03-06T16:17:27.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:27.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:27.830979+0000) 2024-03-06T16:17:28.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:28.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:28.831261+0000) 2024-03-06T16:17:29.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:29.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:29.831466+0000) 2024-03-06T16:17:30.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:30.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:30.831762+0000) 2024-03-06T16:17:31.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:31.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:31.831916+0000) 2024-03-06T16:17:32.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:32.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:32.832174+0000) 2024-03-06T16:17:33.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:33.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:33.832397+0000) 2024-03-06T16:17:34.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:34.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:34.832562+0000) 2024-03-06T16:17:35.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:35.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:35.832783+0000) 2024-03-06T16:17:36.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:36.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:36.833015+0000) 2024-03-06T16:17:37.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:37.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:37.833266+0000) 2024-03-06T16:17:38.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:38.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:38.833472+0000) 2024-03-06T16:17:39.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:39.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:39.833760+0000) 2024-03-06T16:17:40.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:40.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:40.833994+0000) 2024-03-06T16:17:41.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:41.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:41.834245+0000) 2024-03-06T16:17:42.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:42.555 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:42.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:42.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:42.834532+0000) 2024-03-06T16:17:43.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:43.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:43.834817+0000) 2024-03-06T16:17:44.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:44.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:44.835095+0000) 2024-03-06T16:17:45.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:45.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:45.835365+0000) 2024-03-06T16:17:46.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:46.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:46.835646+0000) 2024-03-06T16:17:47.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:47.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:47.835961+0000) 2024-03-06T16:17:48.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:48.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:48.836158+0000) 2024-03-06T16:17:49.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:49.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:49.836322+0000) 2024-03-06T16:17:50.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:50.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:50.836517+0000) 2024-03-06T16:17:51.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:51.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:51.836786+0000) 2024-03-06T16:17:52.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:52.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:52.837021+0000) 2024-03-06T16:17:53.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:53.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:53.837221+0000) 2024-03-06T16:17:54.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:54.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:54.837404+0000) 2024-03-06T16:17:55.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:55.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:55.837599+0000) 2024-03-06T16:17:56.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:56.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:56.837796+0000) 2024-03-06T16:17:57.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:57.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:57.838014+0000) 2024-03-06T16:17:58.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:58.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:58.838285+0000) 2024-03-06T16:17:59.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:17:59.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:59.838530+0000) 2024-03-06T16:18:00.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:00.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:00.838774+0000) 2024-03-06T16:18:01.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:01.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:01.838993+0000) 2024-03-06T16:18:02.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:02.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:02.839236+0000) 2024-03-06T16:18:03.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:03.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:03.839468+0000) 2024-03-06T16:18:04.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:04.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:04.839748+0000) 2024-03-06T16:18:05.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:05.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:05.839977+0000) 2024-03-06T16:18:06.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:06.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:06.840206+0000) 2024-03-06T16:18:07.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:07.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:07.840455+0000) 2024-03-06T16:18:08.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:08.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:08.840714+0000) 2024-03-06T16:18:09.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:09.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:09.840951+0000) 2024-03-06T16:18:10.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:10.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:10.841193+0000) 2024-03-06T16:18:11.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:11.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:11.841479+0000) 2024-03-06T16:18:12.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:12.683 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:12.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:12.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:12.841773+0000) 2024-03-06T16:18:13.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:13.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:13.841982+0000) 2024-03-06T16:18:14.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:14.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:14.842168+0000) 2024-03-06T16:18:15.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:15.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:15.842382+0000) 2024-03-06T16:18:16.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:16.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:16.842540+0000) 2024-03-06T16:18:17.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:17.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:17.842737+0000) 2024-03-06T16:18:18.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:18.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:18.842950+0000) 2024-03-06T16:18:19.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:19.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:19.843124+0000) 2024-03-06T16:18:20.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:20.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:20.843385+0000) 2024-03-06T16:18:21.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:21.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:21.843718+0000) 2024-03-06T16:18:22.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:22.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:22.844012+0000) 2024-03-06T16:18:23.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:23.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:23.844278+0000) 2024-03-06T16:18:24.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:24.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:24.844422+0000) 2024-03-06T16:18:25.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:25.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:25.844632+0000) 2024-03-06T16:18:26.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:26.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:26.844878+0000) 2024-03-06T16:18:27.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:27.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:27.845058+0000) 2024-03-06T16:18:28.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:28.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:28.845290+0000) 2024-03-06T16:18:29.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:29.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:29.845569+0000) 2024-03-06T16:18:30.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:30.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:30.845856+0000) 2024-03-06T16:18:31.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:31.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:31.846093+0000) 2024-03-06T16:18:32.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:32.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:32.846313+0000) 2024-03-06T16:18:33.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:33.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:33.846551+0000) 2024-03-06T16:18:34.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:34.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:34.846765+0000) 2024-03-06T16:18:35.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:35.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:35.846905+0000) 2024-03-06T16:18:36.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:36.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:36.847115+0000) 2024-03-06T16:18:37.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:37.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:37.847350+0000) 2024-03-06T16:18:38.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:38.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:38.847535+0000) 2024-03-06T16:18:39.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:39.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:39.847766+0000) 2024-03-06T16:18:40.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:40.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:40.848003+0000) 2024-03-06T16:18:41.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:41.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:41.848212+0000) 2024-03-06T16:18:42.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:42.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:42.848441+0000) 2024-03-06T16:18:42.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:42.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:43.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:43.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:43.848636+0000) 2024-03-06T16:18:44.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:44.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:44.848889+0000) 2024-03-06T16:18:45.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:45.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:45.849088+0000) 2024-03-06T16:18:46.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:46.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:46.849307+0000) 2024-03-06T16:18:47.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:47.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:47.849501+0000) 2024-03-06T16:18:48.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:48.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:48.849762+0000) 2024-03-06T16:18:49.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:49.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:49.849943+0000) 2024-03-06T16:18:50.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:50.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:50.850174+0000) 2024-03-06T16:18:51.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:51.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:51.850379+0000) 2024-03-06T16:18:52.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:52.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:52.850610+0000) 2024-03-06T16:18:53.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:53.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:53.850855+0000) 2024-03-06T16:18:54.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:54.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:54.851080+0000) 2024-03-06T16:18:55.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:55.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:55.851305+0000) 2024-03-06T16:18:56.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:56.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:56.851516+0000) 2024-03-06T16:18:57.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:57.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:57.851756+0000) 2024-03-06T16:18:58.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:58.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:58.851960+0000) 2024-03-06T16:18:59.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:18:59.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:59.852162+0000) 2024-03-06T16:19:00.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:00.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:00.852395+0000) 2024-03-06T16:19:01.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:01.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:01.852625+0000) 2024-03-06T16:19:02.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:02.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:02.852855+0000) 2024-03-06T16:19:03.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:03.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:03.853083+0000) 2024-03-06T16:19:04.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:04.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:04.853302+0000) 2024-03-06T16:19:05.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:05.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:05.853583+0000) 2024-03-06T16:19:06.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:06.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:06.853804+0000) 2024-03-06T16:19:07.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:07.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:07.854033+0000) 2024-03-06T16:19:08.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:08.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:08.854252+0000) 2024-03-06T16:19:09.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:09.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:09.854475+0000) 2024-03-06T16:19:10.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:10.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:10.854715+0000) 2024-03-06T16:19:11.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:11.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:11.854904+0000) 2024-03-06T16:19:12.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:12.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:12.855101+0000) 2024-03-06T16:19:13.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:13.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:13.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:13.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:13.855293+0000) 2024-03-06T16:19:14.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:14.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:14.855517+0000) 2024-03-06T16:19:15.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:15.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:15.855780+0000) 2024-03-06T16:19:16.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:16.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:16.855999+0000) 2024-03-06T16:19:17.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:17.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:17.856232+0000) 2024-03-06T16:19:18.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:18.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:18.856464+0000) 2024-03-06T16:19:19.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:19.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:19.856736+0000) 2024-03-06T16:19:20.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:20.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:20.856968+0000) 2024-03-06T16:19:21.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:21.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:21.857139+0000) 2024-03-06T16:19:22.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:22.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:22.857358+0000) 2024-03-06T16:19:23.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:23.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:23.857517+0000) 2024-03-06T16:19:24.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:24.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:24.857812+0000) 2024-03-06T16:19:25.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:25.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:25.858016+0000) 2024-03-06T16:19:26.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:26.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:26.858252+0000) 2024-03-06T16:19:27.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:27.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:27.858433+0000) 2024-03-06T16:19:28.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:28.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:28.858659+0000) 2024-03-06T16:19:29.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:29.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:29.858935+0000) 2024-03-06T16:19:30.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:30.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:30.859168+0000) 2024-03-06T16:19:31.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:31.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:31.859394+0000) 2024-03-06T16:19:32.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:32.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:32.859617+0000) 2024-03-06T16:19:33.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:33.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:33.859850+0000) 2024-03-06T16:19:34.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:34.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:34.860082+0000) 2024-03-06T16:19:35.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:35.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:35.860290+0000) 2024-03-06T16:19:36.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:36.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:36.860500+0000) 2024-03-06T16:19:37.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:37.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:37.860737+0000) 2024-03-06T16:19:38.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:38.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:38.860900+0000) 2024-03-06T16:19:39.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:39.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:39.861120+0000) 2024-03-06T16:19:40.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:40.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:40.861345+0000) 2024-03-06T16:19:41.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:41.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:41.861565+0000) 2024-03-06T16:19:42.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:42.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:42.861803+0000) 2024-03-06T16:19:43.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:43.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:43.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:43.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:43.862032+0000) 2024-03-06T16:19:44.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:44.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:44.862253+0000) 2024-03-06T16:19:45.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:45.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:45.862481+0000) 2024-03-06T16:19:46.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:46.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:46.862755+0000) 2024-03-06T16:19:47.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:47.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:47.862995+0000) 2024-03-06T16:19:48.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:48.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:48.863226+0000) 2024-03-06T16:19:49.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:49.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:49.863452+0000) 2024-03-06T16:19:50.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:50.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:50.863717+0000) 2024-03-06T16:19:51.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:51.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:51.863938+0000) 2024-03-06T16:19:52.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:52.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:52.864167+0000) 2024-03-06T16:19:53.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:53.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:53.864445+0000) 2024-03-06T16:19:54.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:54.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:54.864704+0000) 2024-03-06T16:19:55.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:55.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:55.864927+0000) 2024-03-06T16:19:56.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:56.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:56.865149+0000) 2024-03-06T16:19:57.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:57.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:57.865374+0000) 2024-03-06T16:19:58.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:58.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:58.865603+0000) 2024-03-06T16:19:59.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:19:59.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:59.865812+0000) 2024-03-06T16:20:00.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:00.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:00.866046+0000) 2024-03-06T16:20:01.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:01.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:01.866246+0000) 2024-03-06T16:20:02.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:02.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:02.866470+0000) 2024-03-06T16:20:03.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:03.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:03.866647+0000) 2024-03-06T16:20:04.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:04.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:04.866939+0000) 2024-03-06T16:20:05.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:05.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:05.867163+0000) 2024-03-06T16:20:06.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:06.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:06.867387+0000) 2024-03-06T16:20:07.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:07.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:07.867659+0000) 2024-03-06T16:20:08.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:08.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:08.867951+0000) 2024-03-06T16:20:09.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:09.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:09.868162+0000) 2024-03-06T16:20:10.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:10.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:10.868412+0000) 2024-03-06T16:20:11.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:11.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:11.868635+0000) 2024-03-06T16:20:12.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:12.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:12.868873+0000) 2024-03-06T16:20:13.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:13.401 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:13.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:13.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:13.869066+0000) 2024-03-06T16:20:14.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:14.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:14.869304+0000) 2024-03-06T16:20:15.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:15.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:15.869540+0000) 2024-03-06T16:20:16.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:16.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:16.869791+0000) 2024-03-06T16:20:17.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:17.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:17.870027+0000) 2024-03-06T16:20:18.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:18.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:18.870262+0000) 2024-03-06T16:20:19.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:19.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:19.870497+0000) 2024-03-06T16:20:20.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:20.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:20.870741+0000) 2024-03-06T16:20:21.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:21.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:21.870992+0000) 2024-03-06T16:20:22.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:22.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:22.871232+0000) 2024-03-06T16:20:23.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:23.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:23.871473+0000) 2024-03-06T16:20:24.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:24.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:24.871755+0000) 2024-03-06T16:20:25.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:25.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:25.871991+0000) 2024-03-06T16:20:26.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:26.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:26.872177+0000) 2024-03-06T16:20:27.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:27.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:27.872399+0000) 2024-03-06T16:20:28.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:28.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:28.872629+0000) 2024-03-06T16:20:29.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:29.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:29.872856+0000) 2024-03-06T16:20:30.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:30.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:30.873072+0000) 2024-03-06T16:20:31.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:31.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:31.873282+0000) 2024-03-06T16:20:32.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:32.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:32.873506+0000) 2024-03-06T16:20:33.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:33.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:33.873735+0000) 2024-03-06T16:20:34.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:34.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:34.873934+0000) 2024-03-06T16:20:35.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:35.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:35.874176+0000) 2024-03-06T16:20:36.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:36.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:36.874416+0000) 2024-03-06T16:20:37.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:37.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:37.874656+0000) 2024-03-06T16:20:38.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:38.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:38.874942+0000) 2024-03-06T16:20:39.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:39.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:39.875162+0000) 2024-03-06T16:20:40.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:40.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:40.875390+0000) 2024-03-06T16:20:41.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:41.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:41.875627+0000) 2024-03-06T16:20:42.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:42.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:42.875831+0000) 2024-03-06T16:20:43.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:43.533 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:43.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:43.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:43.876025+0000) 2024-03-06T16:20:44.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:44.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:44.876251+0000) 2024-03-06T16:20:45.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:45.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:45.876482+0000) 2024-03-06T16:20:46.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:46.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:46.876786+0000) 2024-03-06T16:20:47.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:47.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:47.877032+0000) 2024-03-06T16:20:48.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:48.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:48.877263+0000) 2024-03-06T16:20:49.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:49.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:49.877490+0000) 2024-03-06T16:20:50.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:50.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:50.877772+0000) 2024-03-06T16:20:51.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:51.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:51.877997+0000) 2024-03-06T16:20:52.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:52.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:52.878187+0000) 2024-03-06T16:20:53.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:53.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:53.878427+0000) 2024-03-06T16:20:54.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:54.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:54.878624+0000) 2024-03-06T16:20:55.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:55.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:55.878856+0000) 2024-03-06T16:20:56.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:56.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:56.879084+0000) 2024-03-06T16:20:57.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:57.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:57.879282+0000) 2024-03-06T16:20:58.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:58.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:58.879516+0000) 2024-03-06T16:20:59.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:20:59.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:59.879734+0000) 2024-03-06T16:21:00.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:00.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:00.879976+0000) 2024-03-06T16:21:01.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:01.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:01.880196+0000) 2024-03-06T16:21:02.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:02.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:02.880420+0000) 2024-03-06T16:21:03.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:03.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:03.880653+0000) 2024-03-06T16:21:04.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:04.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:04.880871+0000) 2024-03-06T16:21:05.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:05.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:05.881084+0000) 2024-03-06T16:21:06.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:06.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:06.881265+0000) 2024-03-06T16:21:07.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:07.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:07.881482+0000) 2024-03-06T16:21:08.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:08.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:08.881711+0000) 2024-03-06T16:21:09.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:09.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:09.881901+0000) 2024-03-06T16:21:10.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:10.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:10.882106+0000) 2024-03-06T16:21:11.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:11.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:11.882339+0000) 2024-03-06T16:21:12.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:12.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:12.882558+0000) 2024-03-06T16:21:13.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:13.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:13.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:13.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:13.882728+0000) 2024-03-06T16:21:14.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:14.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:14.882902+0000) 2024-03-06T16:21:15.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:15.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:15.883148+0000) 2024-03-06T16:21:16.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:16.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:16.883345+0000) 2024-03-06T16:21:17.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:17.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:17.883582+0000) 2024-03-06T16:21:18.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:18.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:18.883815+0000) 2024-03-06T16:21:19.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:19.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:19.883991+0000) 2024-03-06T16:21:20.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:20.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:20.884233+0000) 2024-03-06T16:21:21.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:21.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:21.884449+0000) 2024-03-06T16:21:22.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:22.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:22.884714+0000) 2024-03-06T16:21:23.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:23.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:23.884941+0000) 2024-03-06T16:21:24.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:24.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:24.885132+0000) 2024-03-06T16:21:25.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:25.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:25.885364+0000) 2024-03-06T16:21:26.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:26.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:26.885555+0000) 2024-03-06T16:21:27.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:27.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:27.885760+0000) 2024-03-06T16:21:28.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:28.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:28.885952+0000) 2024-03-06T16:21:29.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:29.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:29.886195+0000) 2024-03-06T16:21:30.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:30.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:30.886451+0000) 2024-03-06T16:21:31.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:31.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:31.886577+0000) 2024-03-06T16:21:32.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:32.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:32.886793+0000) 2024-03-06T16:21:33.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:33.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:33.887000+0000) 2024-03-06T16:21:34.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:34.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:34.887288+0000) 2024-03-06T16:21:35.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:35.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:35.887478+0000) 2024-03-06T16:21:36.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:36.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:36.887754+0000) 2024-03-06T16:21:37.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:37.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:37.887992+0000) 2024-03-06T16:21:38.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:38.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:38.888184+0000) 2024-03-06T16:21:39.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:39.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:39.888414+0000) 2024-03-06T16:21:40.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:40.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:40.888642+0000) 2024-03-06T16:21:41.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:41.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:41.888931+0000) 2024-03-06T16:21:42.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:42.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:42.889168+0000) 2024-03-06T16:21:43.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:43.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:43.889399+0000) 2024-03-06T16:21:43.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:43.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:44.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:44.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:44.889620+0000) 2024-03-06T16:21:45.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:45.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:45.889817+0000) 2024-03-06T16:21:46.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:46.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:46.890042+0000) 2024-03-06T16:21:47.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:47.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:47.890263+0000) 2024-03-06T16:21:48.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:48.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:48.890491+0000) 2024-03-06T16:21:49.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:49.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:49.890699+0000) 2024-03-06T16:21:50.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:50.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:50.890953+0000) 2024-03-06T16:21:51.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:51.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:51.891184+0000) 2024-03-06T16:21:52.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:52.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:52.891404+0000) 2024-03-06T16:21:53.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:53.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:53.891638+0000) 2024-03-06T16:21:54.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:54.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:54.891900+0000) 2024-03-06T16:21:55.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:55.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:55.892057+0000) 2024-03-06T16:21:56.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:56.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:56.892286+0000) 2024-03-06T16:21:57.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:57.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:57.892531+0000) 2024-03-06T16:21:58.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:58.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:58.892783+0000) 2024-03-06T16:21:59.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:21:59.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:59.893027+0000) 2024-03-06T16:22:00.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:00.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:00.893277+0000) 2024-03-06T16:22:01.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:01.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:01.893520+0000) 2024-03-06T16:22:02.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:02.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:02.893789+0000) 2024-03-06T16:22:03.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:03.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:03.894038+0000) 2024-03-06T16:22:04.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:04.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:04.894238+0000) 2024-03-06T16:22:05.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:05.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:05.894435+0000) 2024-03-06T16:22:06.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:06.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:06.894706+0000) 2024-03-06T16:22:07.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:07.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:07.894943+0000) 2024-03-06T16:22:08.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:08.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:08.895120+0000) 2024-03-06T16:22:09.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:09.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:09.895356+0000) 2024-03-06T16:22:10.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:10.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:10.895550+0000) 2024-03-06T16:22:11.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:11.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:11.895770+0000) 2024-03-06T16:22:12.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:12.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:12.895955+0000) 2024-03-06T16:22:13.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:13.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:13.896139+0000) 2024-03-06T16:22:14.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:14.147 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:14.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:14.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:14.896373+0000) 2024-03-06T16:22:15.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:15.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:15.896599+0000) 2024-03-06T16:22:16.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:16.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:16.896786+0000) 2024-03-06T16:22:17.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:17.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:17.897025+0000) 2024-03-06T16:22:18.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:18.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:18.897248+0000) 2024-03-06T16:22:19.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:19.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:19.897467+0000) 2024-03-06T16:22:20.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:20.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:20.897737+0000) 2024-03-06T16:22:21.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:21.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:21.897923+0000) 2024-03-06T16:22:22.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:22.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:22.898147+0000) 2024-03-06T16:22:23.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:23.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:23.898370+0000) 2024-03-06T16:22:24.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:24.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:24.898554+0000) 2024-03-06T16:22:25.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:25.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:25.898791+0000) 2024-03-06T16:22:26.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:26.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:26.899017+0000) 2024-03-06T16:22:27.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:27.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:27.899292+0000) 2024-03-06T16:22:28.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:28.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:28.899458+0000) 2024-03-06T16:22:29.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:29.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:29.899633+0000) 2024-03-06T16:22:30.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:30.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:30.899919+0000) 2024-03-06T16:22:31.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:31.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:31.900142+0000) 2024-03-06T16:22:32.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:32.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:32.900367+0000) 2024-03-06T16:22:33.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:33.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:33.900601+0000) 2024-03-06T16:22:34.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:34.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:34.900837+0000) 2024-03-06T16:22:35.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:35.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:35.901060+0000) 2024-03-06T16:22:36.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:36.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:36.901209+0000) 2024-03-06T16:22:37.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:37.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:37.901394+0000) 2024-03-06T16:22:38.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:38.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:38.901628+0000) 2024-03-06T16:22:39.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:39.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:39.901854+0000) 2024-03-06T16:22:40.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:40.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:40.902089+0000) 2024-03-06T16:22:41.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:41.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:41.902317+0000) 2024-03-06T16:22:42.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:42.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:42.902499+0000) 2024-03-06T16:22:43.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:43.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:43.902728+0000) 2024-03-06T16:22:44.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:44.359 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:44.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:44.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:44.902914+0000) 2024-03-06T16:22:45.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:45.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:45.903139+0000) 2024-03-06T16:22:46.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:46.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:46.903382+0000) 2024-03-06T16:22:47.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:47.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:47.903575+0000) 2024-03-06T16:22:48.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:48.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:48.903805+0000) 2024-03-06T16:22:49.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:49.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:49.904012+0000) 2024-03-06T16:22:50.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:50.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:50.904256+0000) 2024-03-06T16:22:51.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:51.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:51.904488+0000) 2024-03-06T16:22:52.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:52.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:52.904737+0000) 2024-03-06T16:22:53.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:53.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:53.904970+0000) 2024-03-06T16:22:54.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:54.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:54.905164+0000) 2024-03-06T16:22:55.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:55.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:55.905357+0000) 2024-03-06T16:22:56.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:56.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:56.905600+0000) 2024-03-06T16:22:57.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:57.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:57.905830+0000) 2024-03-06T16:22:58.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:58.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:58.906066+0000) 2024-03-06T16:22:59.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:22:59.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:59.906289+0000) 2024-03-06T16:23:00.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:00.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:00.906508+0000) 2024-03-06T16:23:01.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:01.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:01.906773+0000) 2024-03-06T16:23:02.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:02.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:02.907007+0000) 2024-03-06T16:23:03.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:03.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:03.907248+0000) 2024-03-06T16:23:04.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:04.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:04.907481+0000) 2024-03-06T16:23:05.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:05.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:05.907711+0000) 2024-03-06T16:23:06.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:06.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:06.907925+0000) 2024-03-06T16:23:07.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:07.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:07.908122+0000) 2024-03-06T16:23:08.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:08.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:08.908360+0000) 2024-03-06T16:23:09.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:09.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:09.908574+0000) 2024-03-06T16:23:10.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:10.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:10.908801+0000) 2024-03-06T16:23:11.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:11.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:11.909018+0000) 2024-03-06T16:23:12.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:12.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:12.909243+0000) 2024-03-06T16:23:13.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:13.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:13.909497+0000) 2024-03-06T16:23:14.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:14.547 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:14.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:14.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:14.909766+0000) 2024-03-06T16:23:15.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:15.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:15.910045+0000) 2024-03-06T16:23:16.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:16.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:16.910282+0000) 2024-03-06T16:23:17.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:17.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:17.910492+0000) 2024-03-06T16:23:18.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:18.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:18.910784+0000) 2024-03-06T16:23:19.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:19.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:19.911022+0000) 2024-03-06T16:23:20.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:20.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:20.911261+0000) 2024-03-06T16:23:21.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:21.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:21.911526+0000) 2024-03-06T16:23:22.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:22.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:22.911778+0000) 2024-03-06T16:23:23.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:23.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:23.911987+0000) 2024-03-06T16:23:24.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:24.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:24.912227+0000) 2024-03-06T16:23:25.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:25.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:25.912468+0000) 2024-03-06T16:23:26.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:26.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:26.912708+0000) 2024-03-06T16:23:27.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:27.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:27.912912+0000) 2024-03-06T16:23:28.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:28.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:28.913157+0000) 2024-03-06T16:23:29.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:29.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:29.913362+0000) 2024-03-06T16:23:30.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:30.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:30.913551+0000) 2024-03-06T16:23:31.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:31.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:31.913787+0000) 2024-03-06T16:23:32.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:32.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:32.913940+0000) 2024-03-06T16:23:33.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:33.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:33.914127+0000) 2024-03-06T16:23:34.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:34.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:34.914324+0000) 2024-03-06T16:23:35.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:35.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:35.914561+0000) 2024-03-06T16:23:36.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:36.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:36.914779+0000) 2024-03-06T16:23:37.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:37.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:37.915011+0000) 2024-03-06T16:23:38.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:38.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:38.915202+0000) 2024-03-06T16:23:39.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:39.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:39.915432+0000) 2024-03-06T16:23:40.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:40.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:40.915616+0000) 2024-03-06T16:23:41.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:41.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:41.915850+0000) 2024-03-06T16:23:42.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:42.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:42.916088+0000) 2024-03-06T16:23:43.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:43.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:43.916296+0000) 2024-03-06T16:23:44.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:44.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:44.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:44.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:44.916551+0000) 2024-03-06T16:23:45.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:45.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:45.916781+0000) 2024-03-06T16:23:46.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:46.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:46.916993+0000) 2024-03-06T16:23:47.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:47.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:47.917230+0000) 2024-03-06T16:23:48.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:48.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:48.917373+0000) 2024-03-06T16:23:49.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:49.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:49.917531+0000) 2024-03-06T16:23:50.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:50.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:50.917724+0000) 2024-03-06T16:23:51.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:51.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:51.917944+0000) 2024-03-06T16:23:52.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:52.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:52.918186+0000) 2024-03-06T16:23:53.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:53.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:53.918475+0000) 2024-03-06T16:23:54.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:54.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:54.918801+0000) 2024-03-06T16:23:55.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:55.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:55.918971+0000) 2024-03-06T16:23:56.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:56.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:56.919162+0000) 2024-03-06T16:23:57.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:57.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:57.919381+0000) 2024-03-06T16:23:58.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:58.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:58.919621+0000) 2024-03-06T16:23:59.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:23:59.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:59.919847+0000) 2024-03-06T16:24:00.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:00.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:00.920080+0000) 2024-03-06T16:24:01.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:01.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:01.920268+0000) 2024-03-06T16:24:02.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:02.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:02.920445+0000) 2024-03-06T16:24:03.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:03.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:03.920658+0000) 2024-03-06T16:24:04.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:04.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:04.920920+0000) 2024-03-06T16:24:05.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:05.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:05.921107+0000) 2024-03-06T16:24:06.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:06.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:06.921341+0000) 2024-03-06T16:24:07.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:07.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:07.921548+0000) 2024-03-06T16:24:08.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:08.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:08.921801+0000) 2024-03-06T16:24:09.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:09.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:09.922055+0000) 2024-03-06T16:24:10.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:10.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:10.922252+0000) 2024-03-06T16:24:11.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:11.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:11.922425+0000) 2024-03-06T16:24:12.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:12.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:12.922631+0000) 2024-03-06T16:24:13.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:13.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:13.922884+0000) 2024-03-06T16:24:14.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:14.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:14.923078+0000) 2024-03-06T16:24:14.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:14.965 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:15.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:15.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:15.923319+0000) 2024-03-06T16:24:16.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:16.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:16.923529+0000) 2024-03-06T16:24:17.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:17.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:17.923788+0000) 2024-03-06T16:24:18.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:18.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:18.924010+0000) 2024-03-06T16:24:19.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:19.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:19.924258+0000) 2024-03-06T16:24:20.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:20.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:20.924490+0000) 2024-03-06T16:24:21.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:21.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:21.924733+0000) 2024-03-06T16:24:22.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:22.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:22.924992+0000) 2024-03-06T16:24:23.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:23.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:23.925272+0000) 2024-03-06T16:24:24.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:24.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:24.925554+0000) 2024-03-06T16:24:25.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:25.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:25.925824+0000) 2024-03-06T16:24:26.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:26.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:26.926098+0000) 2024-03-06T16:24:27.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:27.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:27.926312+0000) 2024-03-06T16:24:28.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:28.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:28.926562+0000) 2024-03-06T16:24:29.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:29.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:29.926830+0000) 2024-03-06T16:24:30.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:30.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:30.927143+0000) 2024-03-06T16:24:31.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:31.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:31.927422+0000) 2024-03-06T16:24:32.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:32.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:32.927638+0000) 2024-03-06T16:24:33.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:33.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:33.927911+0000) 2024-03-06T16:24:34.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:34.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:34.928128+0000) 2024-03-06T16:24:35.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:35.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:35.928372+0000) 2024-03-06T16:24:36.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:36.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:36.928592+0000) 2024-03-06T16:24:37.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:37.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:37.928829+0000) 2024-03-06T16:24:38.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:38.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:38.929001+0000) 2024-03-06T16:24:39.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:39.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:39.929186+0000) 2024-03-06T16:24:40.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:40.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:40.929422+0000) 2024-03-06T16:24:41.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:41.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:41.929654+0000) 2024-03-06T16:24:42.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:42.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:42.929877+0000) 2024-03-06T16:24:43.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:43.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:43.930088+0000) 2024-03-06T16:24:44.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:44.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:44.930306+0000) 2024-03-06T16:24:45.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:45.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:45.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:45.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:45.930551+0000) 2024-03-06T16:24:46.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:46.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:46.930814+0000) 2024-03-06T16:24:47.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:47.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:47.931054+0000) 2024-03-06T16:24:48.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:48.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:48.931340+0000) 2024-03-06T16:24:49.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:49.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:49.931553+0000) 2024-03-06T16:24:50.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:50.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:50.931755+0000) 2024-03-06T16:24:51.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:51.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:51.931947+0000) 2024-03-06T16:24:52.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:52.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:52.932167+0000) 2024-03-06T16:24:53.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:53.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:53.932397+0000) 2024-03-06T16:24:54.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:54.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:54.932580+0000) 2024-03-06T16:24:55.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:55.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:55.932790+0000) 2024-03-06T16:24:56.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:56.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:56.933028+0000) 2024-03-06T16:24:57.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:57.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:57.933307+0000) 2024-03-06T16:24:58.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:58.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:58.933537+0000) 2024-03-06T16:24:59.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:24:59.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:59.933777+0000) 2024-03-06T16:25:00.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:00.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:00.934005+0000) 2024-03-06T16:25:01.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:01.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:01.934247+0000) 2024-03-06T16:25:02.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:02.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:02.934470+0000) 2024-03-06T16:25:03.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:03.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:03.934744+0000) 2024-03-06T16:25:04.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:04.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:04.934970+0000) 2024-03-06T16:25:05.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:05.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:05.935199+0000) 2024-03-06T16:25:06.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:06.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:06.935428+0000) 2024-03-06T16:25:07.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:07.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:07.935625+0000) 2024-03-06T16:25:08.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:08.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:08.935922+0000) 2024-03-06T16:25:09.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:09.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:09.936166+0000) 2024-03-06T16:25:10.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:10.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:10.936417+0000) 2024-03-06T16:25:11.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:11.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:11.936643+0000) 2024-03-06T16:25:12.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:12.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:12.936943+0000) 2024-03-06T16:25:13.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:13.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:13.937159+0000) 2024-03-06T16:25:14.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:14.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:14.937352+0000) 2024-03-06T16:25:15.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:15.399 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:15.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:15.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:15.937579+0000) 2024-03-06T16:25:16.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:16.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:16.937800+0000) 2024-03-06T16:25:17.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:17.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:17.938024+0000) 2024-03-06T16:25:18.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:18.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:18.938267+0000) 2024-03-06T16:25:19.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:19.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:19.938465+0000) 2024-03-06T16:25:20.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:20.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:20.938691+0000) 2024-03-06T16:25:21.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:21.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:21.938929+0000) 2024-03-06T16:25:22.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:22.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:22.939152+0000) 2024-03-06T16:25:23.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:23.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:23.939370+0000) 2024-03-06T16:25:24.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:24.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:24.939562+0000) 2024-03-06T16:25:25.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:25.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:25.939794+0000) 2024-03-06T16:25:26.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:26.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:26.940028+0000) 2024-03-06T16:25:27.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:27.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:27.940220+0000) 2024-03-06T16:25:28.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:28.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:28.940468+0000) 2024-03-06T16:25:29.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:29.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:29.940742+0000) 2024-03-06T16:25:30.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:30.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:30.940979+0000) 2024-03-06T16:25:31.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:31.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:31.941205+0000) 2024-03-06T16:25:32.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:32.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:32.941427+0000) 2024-03-06T16:25:33.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:33.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:33.941620+0000) 2024-03-06T16:25:34.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:34.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:34.941862+0000) 2024-03-06T16:25:35.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:35.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:35.942078+0000) 2024-03-06T16:25:36.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:36.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:36.942290+0000) 2024-03-06T16:25:37.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:37.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:37.942523+0000) 2024-03-06T16:25:38.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:38.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:38.942794+0000) 2024-03-06T16:25:39.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:39.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:39.943007+0000) 2024-03-06T16:25:40.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:40.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:40.943245+0000) 2024-03-06T16:25:41.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:41.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:41.943467+0000) 2024-03-06T16:25:42.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:42.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:42.943610+0000) 2024-03-06T16:25:43.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:43.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:43.943838+0000) 2024-03-06T16:25:44.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:44.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:44.944067+0000) 2024-03-06T16:25:45.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:45.604 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:45.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:45.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:45.944301+0000) 2024-03-06T16:25:46.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:46.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:46.944538+0000) 2024-03-06T16:25:47.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:47.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:47.944781+0000) 2024-03-06T16:25:48.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:48.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:48.944987+0000) 2024-03-06T16:25:49.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:49.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:49.945192+0000) 2024-03-06T16:25:50.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:50.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:50.945410+0000) 2024-03-06T16:25:51.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:51.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:51.945604+0000) 2024-03-06T16:25:52.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:52.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:52.945847+0000) 2024-03-06T16:25:53.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:53.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:53.946077+0000) 2024-03-06T16:25:54.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:54.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:54.946284+0000) 2024-03-06T16:25:55.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:55.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:55.946468+0000) 2024-03-06T16:25:56.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:56.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:56.946739+0000) 2024-03-06T16:25:57.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:57.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:57.946958+0000) 2024-03-06T16:25:58.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:58.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:58.947167+0000) 2024-03-06T16:25:59.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:25:59.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:59.947375+0000) 2024-03-06T16:26:00.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:00.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:00.947633+0000) 2024-03-06T16:26:01.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:01.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:01.947928+0000) 2024-03-06T16:26:02.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:02.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:02.948172+0000) 2024-03-06T16:26:03.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:03.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:03.948424+0000) 2024-03-06T16:26:04.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:04.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:04.948654+0000) 2024-03-06T16:26:05.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:05.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:05.948917+0000) 2024-03-06T16:26:06.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:06.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:06.949088+0000) 2024-03-06T16:26:07.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:07.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:07.949320+0000) 2024-03-06T16:26:08.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:08.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:08.949526+0000) 2024-03-06T16:26:09.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:09.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:09.949780+0000) 2024-03-06T16:26:10.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:10.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:10.950012+0000) 2024-03-06T16:26:11.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:11.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:11.950239+0000) 2024-03-06T16:26:12.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:12.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:12.950455+0000) 2024-03-06T16:26:13.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:13.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:13.950717+0000) 2024-03-06T16:26:14.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:14.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:14.950945+0000) 2024-03-06T16:26:15.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:15.768 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:15.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:15.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:15.951181+0000) 2024-03-06T16:26:16.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:16.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:16.951405+0000) 2024-03-06T16:26:17.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:17.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:17.951594+0000) 2024-03-06T16:26:18.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:18.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:18.951830+0000) 2024-03-06T16:26:19.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:19.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:19.952059+0000) 2024-03-06T16:26:20.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:20.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:20.952296+0000) 2024-03-06T16:26:21.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:21.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:21.952523+0000) 2024-03-06T16:26:22.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:22.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:22.952794+0000) 2024-03-06T16:26:23.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:23.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:23.953024+0000) 2024-03-06T16:26:24.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:24.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:24.953251+0000) 2024-03-06T16:26:25.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:25.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:25.953482+0000) 2024-03-06T16:26:26.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:26.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:26.953769+0000) 2024-03-06T16:26:27.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:27.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:27.954008+0000) 2024-03-06T16:26:28.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:28.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:28.954245+0000) 2024-03-06T16:26:29.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:29.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:29.954383+0000) 2024-03-06T16:26:30.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:30.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:30.954548+0000) 2024-03-06T16:26:31.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:31.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:31.954800+0000) 2024-03-06T16:26:32.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:32.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:32.954998+0000) 2024-03-06T16:26:33.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:33.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:33.955232+0000) 2024-03-06T16:26:34.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:34.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:34.955413+0000) 2024-03-06T16:26:35.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:35.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:35.955591+0000) 2024-03-06T16:26:36.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:36.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:36.955764+0000) 2024-03-06T16:26:37.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:37.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:37.955993+0000) 2024-03-06T16:26:38.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:38.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:38.956206+0000) 2024-03-06T16:26:39.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:39.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:39.956438+0000) 2024-03-06T16:26:40.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:40.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:40.956737+0000) 2024-03-06T16:26:41.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:41.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:41.956935+0000) 2024-03-06T16:26:42.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:42.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:42.957156+0000) 2024-03-06T16:26:43.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:43.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:43.957363+0000) 2024-03-06T16:26:44.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:44.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:44.957606+0000) 2024-03-06T16:26:45.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:45.897 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:45.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:45.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:45.957824+0000) 2024-03-06T16:26:46.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:46.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:46.958022+0000) 2024-03-06T16:26:47.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:47.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:47.958246+0000) 2024-03-06T16:26:48.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:48.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:48.958465+0000) 2024-03-06T16:26:49.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:49.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:49.958647+0000) 2024-03-06T16:26:50.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:50.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:50.958930+0000) 2024-03-06T16:26:51.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:51.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:51.959149+0000) 2024-03-06T16:26:52.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:52.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:52.959358+0000) 2024-03-06T16:26:53.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:53.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:53.959635+0000) 2024-03-06T16:26:54.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:54.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:54.959941+0000) 2024-03-06T16:26:55.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:55.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:55.960166+0000) 2024-03-06T16:26:56.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:56.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:56.960430+0000) 2024-03-06T16:26:57.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:57.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:57.960630+0000) 2024-03-06T16:26:58.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:58.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:58.960909+0000) 2024-03-06T16:26:59.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:26:59.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:59.961161+0000) 2024-03-06T16:27:00.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:00.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:00.961447+0000) 2024-03-06T16:27:01.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:01.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:01.961772+0000) 2024-03-06T16:27:02.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:02.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:02.962049+0000) 2024-03-06T16:27:03.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:03.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:03.962316+0000) 2024-03-06T16:27:04.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:04.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:04.962579+0000) 2024-03-06T16:27:05.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:05.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:05.962860+0000) 2024-03-06T16:27:06.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:06.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:06.963138+0000) 2024-03-06T16:27:07.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:07.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:07.963375+0000) 2024-03-06T16:27:08.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:08.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:08.963648+0000) 2024-03-06T16:27:09.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:09.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:09.963967+0000) 2024-03-06T16:27:10.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:10.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:10.964221+0000) 2024-03-06T16:27:11.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:11.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:11.964484+0000) 2024-03-06T16:27:12.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:12.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:12.964770+0000) 2024-03-06T16:27:13.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:13.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:13.965005+0000) 2024-03-06T16:27:14.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:14.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:14.965214+0000) 2024-03-06T16:27:15.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:15.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:15.965390+0000) 2024-03-06T16:27:16.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:16.062 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:16.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:16.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:16.965603+0000) 2024-03-06T16:27:17.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:17.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:17.965786+0000) 2024-03-06T16:27:18.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:18.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:18.965978+0000) 2024-03-06T16:27:19.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:19.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:19.966198+0000) 2024-03-06T16:27:20.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:20.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:20.966419+0000) 2024-03-06T16:27:21.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:21.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:21.966604+0000) 2024-03-06T16:27:22.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:22.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:22.966821+0000) 2024-03-06T16:27:23.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:23.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:23.967049+0000) 2024-03-06T16:27:24.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:24.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:24.967252+0000) 2024-03-06T16:27:25.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:25.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:25.967492+0000) 2024-03-06T16:27:26.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:26.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:26.967795+0000) 2024-03-06T16:27:27.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:27.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:27.968018+0000) 2024-03-06T16:27:28.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:28.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:28.968261+0000) 2024-03-06T16:27:29.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:29.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:29.968543+0000) 2024-03-06T16:27:30.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:30.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:30.968831+0000) 2024-03-06T16:27:31.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:31.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:31.969032+0000) 2024-03-06T16:27:32.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:32.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:32.969274+0000) 2024-03-06T16:27:33.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:33.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:33.969514+0000) 2024-03-06T16:27:34.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:34.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:34.969787+0000) 2024-03-06T16:27:35.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:35.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:35.970006+0000) 2024-03-06T16:27:36.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:36.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:36.970241+0000) 2024-03-06T16:27:37.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:37.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:37.970462+0000) 2024-03-06T16:27:38.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:38.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:38.970786+0000) 2024-03-06T16:27:39.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:39.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:39.971063+0000) 2024-03-06T16:27:40.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:40.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:40.971350+0000) 2024-03-06T16:27:41.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:41.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:41.971553+0000) 2024-03-06T16:27:42.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:42.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:42.971806+0000) 2024-03-06T16:27:43.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:43.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:43.972023+0000) 2024-03-06T16:27:44.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:44.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:44.972288+0000) 2024-03-06T16:27:45.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:45.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:45.972567+0000) 2024-03-06T16:27:46.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:46.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:46.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:46.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:46.972799+0000) 2024-03-06T16:27:47.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:47.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:47.973003+0000) 2024-03-06T16:27:48.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:48.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:48.973207+0000) 2024-03-06T16:27:49.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:49.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:49.973435+0000) 2024-03-06T16:27:50.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:50.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:50.973718+0000) 2024-03-06T16:27:51.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:51.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:51.973913+0000) 2024-03-06T16:27:52.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:52.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:52.974044+0000) 2024-03-06T16:27:53.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:53.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:53.974238+0000) 2024-03-06T16:27:54.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:54.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:54.974473+0000) 2024-03-06T16:27:55.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:55.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:55.974730+0000) 2024-03-06T16:27:56.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:56.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:56.974923+0000) 2024-03-06T16:27:57.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:57.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:57.975154+0000) 2024-03-06T16:27:58.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:58.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:58.975394+0000) 2024-03-06T16:27:59.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:27:59.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:59.975611+0000) 2024-03-06T16:28:00.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:00.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:00.975855+0000) 2024-03-06T16:28:01.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:01.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:01.976080+0000) 2024-03-06T16:28:02.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:02.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:02.976309+0000) 2024-03-06T16:28:03.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:03.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:03.976541+0000) 2024-03-06T16:28:04.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:04.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:04.976791+0000) 2024-03-06T16:28:05.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:05.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:05.977012+0000) 2024-03-06T16:28:06.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:06.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:06.977205+0000) 2024-03-06T16:28:07.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:07.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:07.977445+0000) 2024-03-06T16:28:08.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:08.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:08.977651+0000) 2024-03-06T16:28:09.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:09.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:09.977899+0000) 2024-03-06T16:28:10.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:10.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:10.978102+0000) 2024-03-06T16:28:11.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:11.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:11.978333+0000) 2024-03-06T16:28:12.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:12.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:12.978529+0000) 2024-03-06T16:28:13.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:13.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:13.978771+0000) 2024-03-06T16:28:14.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:14.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:14.978964+0000) 2024-03-06T16:28:15.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:15.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:15.979174+0000) 2024-03-06T16:28:16.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:16.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:16.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:16.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:16.979411+0000) 2024-03-06T16:28:17.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:17.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:17.979642+0000) 2024-03-06T16:28:18.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:18.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:18.979906+0000) 2024-03-06T16:28:19.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:19.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:19.980142+0000) 2024-03-06T16:28:20.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:20.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:20.980281+0000) 2024-03-06T16:28:21.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:21.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:21.980492+0000) 2024-03-06T16:28:22.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:22.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:22.980764+0000) 2024-03-06T16:28:23.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:23.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:23.980997+0000) 2024-03-06T16:28:24.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:24.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:24.981190+0000) 2024-03-06T16:28:25.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:25.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:25.981396+0000) 2024-03-06T16:28:26.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:26.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:26.981626+0000) 2024-03-06T16:28:27.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:27.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:27.981811+0000) 2024-03-06T16:28:28.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:28.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:28.982006+0000) 2024-03-06T16:28:29.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:29.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:29.982238+0000) 2024-03-06T16:28:30.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:30.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:30.982434+0000) 2024-03-06T16:28:31.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:31.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:31.982625+0000) 2024-03-06T16:28:32.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:32.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:32.982881+0000) 2024-03-06T16:28:33.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:33.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:33.983074+0000) 2024-03-06T16:28:34.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:34.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:34.983301+0000) 2024-03-06T16:28:35.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:35.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:35.983519+0000) 2024-03-06T16:28:36.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:36.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:36.983743+0000) 2024-03-06T16:28:37.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:37.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:37.983955+0000) 2024-03-06T16:28:38.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:38.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:38.984191+0000) 2024-03-06T16:28:39.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:39.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:39.984425+0000) 2024-03-06T16:28:40.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:40.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:40.984661+0000) 2024-03-06T16:28:41.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:41.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:41.984948+0000) 2024-03-06T16:28:42.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:42.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:42.985189+0000) 2024-03-06T16:28:43.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:43.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:43.985433+0000) 2024-03-06T16:28:44.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:44.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:44.985646+0000) 2024-03-06T16:28:45.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:45.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:45.985867+0000) 2024-03-06T16:28:46.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:46.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:46.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:46.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:46.986113+0000) 2024-03-06T16:28:47.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:47.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:47.986310+0000) 2024-03-06T16:28:48.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:48.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:48.986497+0000) 2024-03-06T16:28:49.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:49.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:49.986751+0000) 2024-03-06T16:28:50.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:50.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:50.986949+0000) 2024-03-06T16:28:51.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:51.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:51.987173+0000) 2024-03-06T16:28:52.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:52.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:52.987321+0000) 2024-03-06T16:28:53.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:53.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:53.987517+0000) 2024-03-06T16:28:54.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:54.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:54.987792+0000) 2024-03-06T16:28:55.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:55.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:55.987998+0000) 2024-03-06T16:28:56.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:56.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:56.988229+0000) 2024-03-06T16:28:57.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:57.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:57.988453+0000) 2024-03-06T16:28:58.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:58.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:58.988645+0000) 2024-03-06T16:28:59.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:28:59.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:59.988926+0000) 2024-03-06T16:29:00.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:00.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:00.989178+0000) 2024-03-06T16:29:01.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:01.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:01.989385+0000) 2024-03-06T16:29:02.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:02.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:02.989568+0000) 2024-03-06T16:29:03.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:03.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:03.989799+0000) 2024-03-06T16:29:04.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:04.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:04.990030+0000) 2024-03-06T16:29:05.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:05.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:05.990267+0000) 2024-03-06T16:29:06.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:06.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:06.990442+0000) 2024-03-06T16:29:07.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:07.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:07.990749+0000) 2024-03-06T16:29:08.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:08.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:08.990910+0000) 2024-03-06T16:29:09.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:09.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:09.991129+0000) 2024-03-06T16:29:10.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:10.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:10.991321+0000) 2024-03-06T16:29:11.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:11.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:11.991519+0000) 2024-03-06T16:29:12.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:12.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:12.991785+0000) 2024-03-06T16:29:13.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:13.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:13.992012+0000) 2024-03-06T16:29:14.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:14.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:14.992251+0000) 2024-03-06T16:29:15.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:15.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:15.992482+0000) 2024-03-06T16:29:16.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:16.781 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:16.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:16.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:16.992753+0000) 2024-03-06T16:29:17.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:17.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:17.992989+0000) 2024-03-06T16:29:18.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:18.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:18.993215+0000) 2024-03-06T16:29:19.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:19.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:19.993449+0000) 2024-03-06T16:29:20.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:20.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:20.993728+0000) 2024-03-06T16:29:21.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:21.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:21.993964+0000) 2024-03-06T16:29:22.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:22.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:22.994153+0000) 2024-03-06T16:29:23.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:23.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:23.994400+0000) 2024-03-06T16:29:24.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:24.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:24.994583+0000) 2024-03-06T16:29:25.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:25.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:25.994830+0000) 2024-03-06T16:29:26.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:26.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:26.995043+0000) 2024-03-06T16:29:27.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:27.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:27.995240+0000) 2024-03-06T16:29:28.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:28.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:28.995473+0000) 2024-03-06T16:29:29.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:29.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:29.995714+0000) 2024-03-06T16:29:30.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:30.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:30.995973+0000) 2024-03-06T16:29:31.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:31.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:31.996195+0000) 2024-03-06T16:29:32.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:32.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:32.996378+0000) 2024-03-06T16:29:33.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:33.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:33.996618+0000) 2024-03-06T16:29:34.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:34.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:34.996881+0000) 2024-03-06T16:29:35.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:35.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:35.997104+0000) 2024-03-06T16:29:36.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:36.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:36.997346+0000) 2024-03-06T16:29:37.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:37.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:37.997565+0000) 2024-03-06T16:29:38.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:38.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:38.997799+0000) 2024-03-06T16:29:39.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:39.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:39.998040+0000) 2024-03-06T16:29:40.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:40.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:40.998336+0000) 2024-03-06T16:29:41.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:41.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:41.998572+0000) 2024-03-06T16:29:42.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:42.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:42.998776+0000) 2024-03-06T16:29:43.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:43.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:43.998996+0000) 2024-03-06T16:29:44.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:44.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:44.999232+0000) 2024-03-06T16:29:45.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:45.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:45.999464+0000) 2024-03-06T16:29:46.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:46.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:46.999730+0000) 2024-03-06T16:29:47.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:47.201 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:47.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:47.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:47.999926+0000) 2024-03-06T16:29:48.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:48.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:49.000132+0000) 2024-03-06T16:29:49.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:49.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:50.000406+0000) 2024-03-06T16:29:50.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:50.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:51.000623+0000) 2024-03-06T16:29:51.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:52.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:52.000859+0000) 2024-03-06T16:29:52.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:53.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:53.001083+0000) 2024-03-06T16:29:54.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:54.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:54.001353+0000) 2024-03-06T16:29:55.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:55.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:55.001639+0000) 2024-03-06T16:29:56.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:56.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:56.002006+0000) 2024-03-06T16:29:57.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:57.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:57.002269+0000) 2024-03-06T16:29:58.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:58.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:58.002543+0000) 2024-03-06T16:29:59.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:29:59.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:59.002823+0000) 2024-03-06T16:30:00.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:00.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:00.003065+0000) 2024-03-06T16:30:01.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:01.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:01.003305+0000) 2024-03-06T16:30:02.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:02.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:02.003579+0000) 2024-03-06T16:30:03.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:03.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:03.003791+0000) 2024-03-06T16:30:04.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:04.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:04.004004+0000) 2024-03-06T16:30:05.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:05.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:05.004230+0000) 2024-03-06T16:30:06.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:06.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:06.004473+0000) 2024-03-06T16:30:07.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:07.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:07.004789+0000) 2024-03-06T16:30:08.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:08.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:08.004943+0000) 2024-03-06T16:30:09.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:09.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:09.005172+0000) 2024-03-06T16:30:10.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:10.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:10.005409+0000) 2024-03-06T16:30:11.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:11.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:11.005648+0000) 2024-03-06T16:30:12.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:12.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:12.005871+0000) 2024-03-06T16:30:13.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:13.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:13.006071+0000) 2024-03-06T16:30:14.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:14.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:14.006293+0000) 2024-03-06T16:30:15.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:15.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:15.006535+0000) 2024-03-06T16:30:16.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:16.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:16.006774+0000) 2024-03-06T16:30:17.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:17.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:17.006959+0000) 2024-03-06T16:30:17.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:17.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:18.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:18.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:18.007189+0000) 2024-03-06T16:30:19.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:19.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:19.007413+0000) 2024-03-06T16:30:20.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:20.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:20.007724+0000) 2024-03-06T16:30:21.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:21.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:21.008047+0000) 2024-03-06T16:30:22.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:22.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:22.008246+0000) 2024-03-06T16:30:23.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:23.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:23.008426+0000) 2024-03-06T16:30:24.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:24.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:24.008661+0000) 2024-03-06T16:30:25.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:25.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:25.008965+0000) 2024-03-06T16:30:26.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:26.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:26.009205+0000) 2024-03-06T16:30:27.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:27.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:27.009417+0000) 2024-03-06T16:30:28.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:28.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:28.009624+0000) 2024-03-06T16:30:29.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:29.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:29.009947+0000) 2024-03-06T16:30:30.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:30.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:30.010196+0000) 2024-03-06T16:30:31.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:31.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:31.010475+0000) 2024-03-06T16:30:32.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:32.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:32.010786+0000) 2024-03-06T16:30:33.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:33.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:33.010990+0000) 2024-03-06T16:30:34.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:34.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:34.011200+0000) 2024-03-06T16:30:35.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:35.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:35.011468+0000) 2024-03-06T16:30:36.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:36.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:36.011793+0000) 2024-03-06T16:30:37.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:37.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:37.012033+0000) 2024-03-06T16:30:38.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:38.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:38.012212+0000) 2024-03-06T16:30:39.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:39.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:39.012441+0000) 2024-03-06T16:30:40.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:40.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:40.012659+0000) 2024-03-06T16:30:41.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:41.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:41.012908+0000) 2024-03-06T16:30:42.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:42.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:42.013099+0000) 2024-03-06T16:30:43.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:43.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:43.013299+0000) 2024-03-06T16:30:44.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:44.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:44.013570+0000) 2024-03-06T16:30:45.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:45.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:45.013800+0000) 2024-03-06T16:30:46.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:46.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:46.014011+0000) 2024-03-06T16:30:47.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:47.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:47.014232+0000) 2024-03-06T16:30:47.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:47.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:48.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:48.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:48.014407+0000) 2024-03-06T16:30:49.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:49.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:49.014643+0000) 2024-03-06T16:30:50.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:50.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:50.014886+0000) 2024-03-06T16:30:51.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:51.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:51.015134+0000) 2024-03-06T16:30:52.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:52.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:52.015383+0000) 2024-03-06T16:30:53.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:53.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:53.015620+0000) 2024-03-06T16:30:54.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:54.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:54.015849+0000) 2024-03-06T16:30:55.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:55.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:55.016068+0000) 2024-03-06T16:30:56.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:56.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:56.016309+0000) 2024-03-06T16:30:57.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:57.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:57.016533+0000) 2024-03-06T16:30:58.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:58.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:58.016797+0000) 2024-03-06T16:30:59.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:30:59.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:59.017003+0000) 2024-03-06T16:31:00.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:00.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:00.017214+0000) 2024-03-06T16:31:01.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:01.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:01.017476+0000) 2024-03-06T16:31:02.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:02.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:02.017780+0000) 2024-03-06T16:31:03.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:03.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:03.018056+0000) 2024-03-06T16:31:04.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:04.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:04.018260+0000) 2024-03-06T16:31:05.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:05.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:05.018508+0000) 2024-03-06T16:31:06.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:06.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:06.018775+0000) 2024-03-06T16:31:07.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:07.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:07.018966+0000) 2024-03-06T16:31:08.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:08.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:08.019185+0000) 2024-03-06T16:31:09.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:09.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:09.019380+0000) 2024-03-06T16:31:10.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:10.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:10.019620+0000) 2024-03-06T16:31:11.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:11.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:11.019837+0000) 2024-03-06T16:31:12.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:12.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:12.020029+0000) 2024-03-06T16:31:13.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:13.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:13.020248+0000) 2024-03-06T16:31:14.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:14.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:14.020471+0000) 2024-03-06T16:31:15.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:15.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:15.020698+0000) 2024-03-06T16:31:16.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:16.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:16.020908+0000) 2024-03-06T16:31:17.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:17.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:17.021178+0000) 2024-03-06T16:31:17.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:17.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:18.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:18.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:18.021462+0000) 2024-03-06T16:31:19.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:19.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:19.021689+0000) 2024-03-06T16:31:20.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:20.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:20.021929+0000) 2024-03-06T16:31:21.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:21.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:21.022211+0000) 2024-03-06T16:31:22.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:22.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:22.022488+0000) 2024-03-06T16:31:23.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:23.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:23.022803+0000) 2024-03-06T16:31:24.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:24.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:24.023078+0000) 2024-03-06T16:31:25.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:25.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:25.023342+0000) 2024-03-06T16:31:26.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:26.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:26.023628+0000) 2024-03-06T16:31:27.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:27.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:27.023903+0000) 2024-03-06T16:31:28.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:28.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:28.024142+0000) 2024-03-06T16:31:29.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:29.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:29.024378+0000) 2024-03-06T16:31:30.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:30.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:30.024619+0000) 2024-03-06T16:31:31.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:31.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:31.024886+0000) 2024-03-06T16:31:32.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:32.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:32.025062+0000) 2024-03-06T16:31:33.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:33.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:33.025304+0000) 2024-03-06T16:31:34.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:34.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:34.025542+0000) 2024-03-06T16:31:35.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:35.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:35.025781+0000) 2024-03-06T16:31:36.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:36.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:36.026011+0000) 2024-03-06T16:31:37.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:37.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:37.026232+0000) 2024-03-06T16:31:38.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:38.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:38.026509+0000) 2024-03-06T16:31:39.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:39.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:39.026801+0000) 2024-03-06T16:31:40.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:40.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:40.027027+0000) 2024-03-06T16:31:41.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:41.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:41.027217+0000) 2024-03-06T16:31:42.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:42.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:42.027444+0000) 2024-03-06T16:31:43.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:43.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:43.027700+0000) 2024-03-06T16:31:44.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:44.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:44.027926+0000) 2024-03-06T16:31:45.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:45.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:45.028115+0000) 2024-03-06T16:31:46.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:46.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:46.028299+0000) 2024-03-06T16:31:47.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:47.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:47.028536+0000) 2024-03-06T16:31:47.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:47.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:48.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:48.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:48.028817+0000) 2024-03-06T16:31:49.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:49.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:49.029038+0000) 2024-03-06T16:31:50.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:50.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:50.029259+0000) 2024-03-06T16:31:51.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:51.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:51.029491+0000) 2024-03-06T16:31:52.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:52.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:52.029697+0000) 2024-03-06T16:31:53.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:53.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:53.029919+0000) 2024-03-06T16:31:54.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:54.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:54.030151+0000) 2024-03-06T16:31:55.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:55.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:55.030391+0000) 2024-03-06T16:31:56.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:56.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:56.030625+0000) 2024-03-06T16:31:57.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:57.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:57.030882+0000) 2024-03-06T16:31:58.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:58.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:58.031133+0000) 2024-03-06T16:31:59.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:31:59.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:59.031322+0000) 2024-03-06T16:32:00.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:00.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:00.031545+0000) 2024-03-06T16:32:01.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:01.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:01.031791+0000) 2024-03-06T16:32:02.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:02.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:02.032017+0000) 2024-03-06T16:32:03.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:03.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:03.032239+0000) 2024-03-06T16:32:04.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:04.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:04.032465+0000) 2024-03-06T16:32:05.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:05.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:05.032752+0000) 2024-03-06T16:32:06.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:06.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:06.032949+0000) 2024-03-06T16:32:07.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:07.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:07.033174+0000) 2024-03-06T16:32:08.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:08.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:08.033391+0000) 2024-03-06T16:32:09.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:09.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:09.033611+0000) 2024-03-06T16:32:10.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:10.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:10.033825+0000) 2024-03-06T16:32:11.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:11.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:11.034027+0000) 2024-03-06T16:32:12.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:12.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:12.034213+0000) 2024-03-06T16:32:13.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:13.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:13.034444+0000) 2024-03-06T16:32:14.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:14.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:14.034709+0000) 2024-03-06T16:32:15.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:15.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:15.034928+0000) 2024-03-06T16:32:16.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:16.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:16.035118+0000) 2024-03-06T16:32:17.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:17.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:17.035398+0000) 2024-03-06T16:32:17.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:17.984 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:18.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:18.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:18.035604+0000) 2024-03-06T16:32:19.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:19.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:19.035828+0000) 2024-03-06T16:32:20.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:20.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:20.036017+0000) 2024-03-06T16:32:21.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:21.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:21.036246+0000) 2024-03-06T16:32:22.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:22.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:22.036454+0000) 2024-03-06T16:32:23.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:23.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:23.036661+0000) 2024-03-06T16:32:24.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:24.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:24.036929+0000) 2024-03-06T16:32:25.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:25.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:25.037163+0000) 2024-03-06T16:32:26.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:26.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:26.037452+0000) 2024-03-06T16:32:27.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:27.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:27.037660+0000) 2024-03-06T16:32:28.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:28.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:28.037856+0000) 2024-03-06T16:32:29.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:29.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:29.038068+0000) 2024-03-06T16:32:30.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:30.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:30.038268+0000) 2024-03-06T16:32:31.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:31.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:31.038472+0000) 2024-03-06T16:32:32.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:32.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:32.038700+0000) 2024-03-06T16:32:33.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:33.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:33.038977+0000) 2024-03-06T16:32:34.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:34.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:34.039168+0000) 2024-03-06T16:32:35.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:35.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:35.039359+0000) 2024-03-06T16:32:36.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:36.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:36.039598+0000) 2024-03-06T16:32:37.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:37.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:37.039844+0000) 2024-03-06T16:32:38.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:38.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:38.040120+0000) 2024-03-06T16:32:39.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:39.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:39.040387+0000) 2024-03-06T16:32:40.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:40.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:40.040622+0000) 2024-03-06T16:32:41.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:41.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:41.040831+0000) 2024-03-06T16:32:42.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:42.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:42.041045+0000) 2024-03-06T16:32:43.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:43.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:43.041283+0000) 2024-03-06T16:32:44.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:44.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:44.041504+0000) 2024-03-06T16:32:45.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:45.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:45.041785+0000) 2024-03-06T16:32:46.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:46.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:46.041979+0000) 2024-03-06T16:32:47.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:47.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:47.042206+0000) 2024-03-06T16:32:48.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:48.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:48.042461+0000) 2024-03-06T16:32:48.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:48.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:49.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:49.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:49.042765+0000) 2024-03-06T16:32:50.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:50.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:50.042982+0000) 2024-03-06T16:32:51.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:51.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:51.043220+0000) 2024-03-06T16:32:52.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:52.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:52.043449+0000) 2024-03-06T16:32:53.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:53.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:53.043706+0000) 2024-03-06T16:32:54.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:54.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:54.043930+0000) 2024-03-06T16:32:55.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:55.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:55.044202+0000) 2024-03-06T16:32:56.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:56.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:56.044468+0000) 2024-03-06T16:32:57.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:57.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:57.044768+0000) 2024-03-06T16:32:58.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:58.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:58.044974+0000) 2024-03-06T16:32:59.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:32:59.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:59.045158+0000) 2024-03-06T16:33:00.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:00.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:00.045378+0000) 2024-03-06T16:33:01.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:01.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:01.045599+0000) 2024-03-06T16:33:02.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:02.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:02.045859+0000) 2024-03-06T16:33:03.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:03.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:03.046107+0000) 2024-03-06T16:33:04.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:04.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:04.046376+0000) 2024-03-06T16:33:05.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:05.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:05.046645+0000) 2024-03-06T16:33:06.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:06.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:06.046938+0000) 2024-03-06T16:33:07.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:07.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:07.047169+0000) 2024-03-06T16:33:08.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:08.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:08.047363+0000) 2024-03-06T16:33:09.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:09.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:09.047604+0000) 2024-03-06T16:33:10.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:10.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:10.047813+0000) 2024-03-06T16:33:11.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:11.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:11.048049+0000) 2024-03-06T16:33:12.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:12.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:12.048326+0000) 2024-03-06T16:33:13.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:13.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:13.048606+0000) 2024-03-06T16:33:14.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:14.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:14.048929+0000) 2024-03-06T16:33:15.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:15.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:15.049150+0000) 2024-03-06T16:33:16.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:16.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:16.049371+0000) 2024-03-06T16:33:17.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:17.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:17.049610+0000) 2024-03-06T16:33:18.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:18.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:18.049819+0000) 2024-03-06T16:33:18.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:18.328 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:19.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:19.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:19.050021+0000) 2024-03-06T16:33:20.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:20.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:20.050289+0000) 2024-03-06T16:33:21.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:21.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:21.050502+0000) 2024-03-06T16:33:22.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:22.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:22.050765+0000) 2024-03-06T16:33:23.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:23.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:23.051050+0000) 2024-03-06T16:33:24.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:24.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:24.051328+0000) 2024-03-06T16:33:25.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:25.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:25.051613+0000) 2024-03-06T16:33:26.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:26.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:26.051908+0000) 2024-03-06T16:33:27.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:27.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:27.052159+0000) 2024-03-06T16:33:28.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:28.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:28.052422+0000) 2024-03-06T16:33:29.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:29.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:29.052702+0000) 2024-03-06T16:33:30.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:30.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:30.052901+0000) 2024-03-06T16:33:31.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:31.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:31.053102+0000) 2024-03-06T16:33:32.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:32.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:32.053329+0000) 2024-03-06T16:33:33.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:33.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:33.053574+0000) 2024-03-06T16:33:34.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:34.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:34.053841+0000) 2024-03-06T16:33:35.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:35.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:35.054088+0000) 2024-03-06T16:33:36.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:36.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:36.054363+0000) 2024-03-06T16:33:37.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:37.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:37.054626+0000) 2024-03-06T16:33:38.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:38.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:38.054881+0000) 2024-03-06T16:33:39.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:39.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:39.055118+0000) 2024-03-06T16:33:40.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:40.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:40.055393+0000) 2024-03-06T16:33:41.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:41.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:41.055605+0000) 2024-03-06T16:33:42.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:42.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:42.055822+0000) 2024-03-06T16:33:43.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:43.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:43.056013+0000) 2024-03-06T16:33:44.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:44.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:44.056208+0000) 2024-03-06T16:33:45.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:45.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:45.056465+0000) 2024-03-06T16:33:46.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:46.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:46.056799+0000) 2024-03-06T16:33:47.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:47.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:47.057028+0000) 2024-03-06T16:33:48.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:48.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:48.057245+0000) 2024-03-06T16:33:48.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:48.464 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:49.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:49.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:49.057463+0000) 2024-03-06T16:33:50.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:50.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:50.057718+0000) 2024-03-06T16:33:51.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:51.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:51.057958+0000) 2024-03-06T16:33:52.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:52.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:52.058181+0000) 2024-03-06T16:33:53.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:53.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:53.058404+0000) 2024-03-06T16:33:54.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:54.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:54.058626+0000) 2024-03-06T16:33:55.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:55.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:55.058818+0000) 2024-03-06T16:33:56.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:56.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:56.059050+0000) 2024-03-06T16:33:57.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:57.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:57.059342+0000) 2024-03-06T16:33:58.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:58.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:58.059608+0000) 2024-03-06T16:33:59.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:33:59.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:59.059855+0000) 2024-03-06T16:34:00.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:00.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:00.060095+0000) 2024-03-06T16:34:01.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:01.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:01.060380+0000) 2024-03-06T16:34:02.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:02.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:02.060657+0000) 2024-03-06T16:34:03.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:03.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:03.060935+0000) 2024-03-06T16:34:04.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:04.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:04.061206+0000) 2024-03-06T16:34:05.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:05.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:05.061480+0000) 2024-03-06T16:34:06.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:06.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:06.061764+0000) 2024-03-06T16:34:07.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:07.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:07.062053+0000) 2024-03-06T16:34:08.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:08.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:08.062330+0000) 2024-03-06T16:34:09.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:09.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:09.062606+0000) 2024-03-06T16:34:10.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:10.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:10.062854+0000) 2024-03-06T16:34:11.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:11.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:11.063078+0000) 2024-03-06T16:34:12.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:12.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:12.063313+0000) 2024-03-06T16:34:13.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:13.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:13.063580+0000) 2024-03-06T16:34:14.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:14.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:14.063861+0000) 2024-03-06T16:34:15.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:15.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:15.064100+0000) 2024-03-06T16:34:16.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:16.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:16.064378+0000) 2024-03-06T16:34:17.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:17.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:17.064659+0000) 2024-03-06T16:34:18.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:18.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:18.064919+0000) 2024-03-06T16:34:18.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:18.695 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:19.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:19.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:19.065157+0000) 2024-03-06T16:34:20.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:20.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:20.065382+0000) 2024-03-06T16:34:21.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:21.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:21.065618+0000) 2024-03-06T16:34:22.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:22.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:22.065829+0000) 2024-03-06T16:34:23.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:23.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:23.066024+0000) 2024-03-06T16:34:24.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:24.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:24.066254+0000) 2024-03-06T16:34:25.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:25.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:25.066537+0000) 2024-03-06T16:34:26.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:26.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:26.066805+0000) 2024-03-06T16:34:27.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:27.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:27.066984+0000) 2024-03-06T16:34:28.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:28.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:28.067204+0000) 2024-03-06T16:34:29.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:29.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:29.067398+0000) 2024-03-06T16:34:30.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:30.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:30.067635+0000) 2024-03-06T16:34:31.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:31.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:31.067870+0000) 2024-03-06T16:34:32.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:32.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:32.068070+0000) 2024-03-06T16:34:33.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:33.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:33.068296+0000) 2024-03-06T16:34:34.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:34.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:34.068515+0000) 2024-03-06T16:34:35.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:35.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:35.068798+0000) 2024-03-06T16:34:36.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:36.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:36.069023+0000) 2024-03-06T16:34:37.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:37.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:37.069277+0000) 2024-03-06T16:34:38.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:38.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:38.069552+0000) 2024-03-06T16:34:39.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:39.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:39.069813+0000) 2024-03-06T16:34:40.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:40.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:40.069989+0000) 2024-03-06T16:34:41.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:41.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:41.070221+0000) 2024-03-06T16:34:42.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:42.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:42.070497+0000) 2024-03-06T16:34:43.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:43.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:43.070808+0000) 2024-03-06T16:34:44.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:44.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:44.070980+0000) 2024-03-06T16:34:45.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:45.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:45.071207+0000) 2024-03-06T16:34:46.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:46.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:46.071454+0000) 2024-03-06T16:34:47.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:47.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:47.071762+0000) 2024-03-06T16:34:48.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:48.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:48.072007+0000) 2024-03-06T16:34:48.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:48.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:49.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:49.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:49.072260+0000) 2024-03-06T16:34:50.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:50.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:50.072505+0000) 2024-03-06T16:34:51.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:51.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:51.072788+0000) 2024-03-06T16:34:52.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:52.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:52.073017+0000) 2024-03-06T16:34:53.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:53.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:53.073231+0000) 2024-03-06T16:34:54.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:54.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:54.073419+0000) 2024-03-06T16:34:55.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:55.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:55.073738+0000) 2024-03-06T16:34:56.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:56.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:56.073929+0000) 2024-03-06T16:34:57.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:57.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:57.074169+0000) 2024-03-06T16:34:58.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:58.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:58.074391+0000) 2024-03-06T16:34:59.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:34:59.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:59.074627+0000) 2024-03-06T16:35:00.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:00.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:00.074853+0000) 2024-03-06T16:35:01.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:01.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:01.075057+0000) 2024-03-06T16:35:02.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:02.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:02.075282+0000) 2024-03-06T16:35:03.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:03.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:03.075499+0000) 2024-03-06T16:35:04.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:04.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:04.075762+0000) 2024-03-06T16:35:05.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:05.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:05.076004+0000) 2024-03-06T16:35:06.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:06.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:06.076233+0000) 2024-03-06T16:35:07.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:07.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:07.076460+0000) 2024-03-06T16:35:08.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:08.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:08.076712+0000) 2024-03-06T16:35:09.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:09.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:09.076953+0000) 2024-03-06T16:35:10.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:10.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:10.077181+0000) 2024-03-06T16:35:11.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:11.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:11.077429+0000) 2024-03-06T16:35:12.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:12.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:12.077656+0000) 2024-03-06T16:35:13.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:13.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:13.077922+0000) 2024-03-06T16:35:14.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:14.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:14.078153+0000) 2024-03-06T16:35:15.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:15.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:15.078353+0000) 2024-03-06T16:35:16.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:16.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:16.078603+0000) 2024-03-06T16:35:17.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:17.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:17.078838+0000) 2024-03-06T16:35:18.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:18.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:18.079054+0000) 2024-03-06T16:35:19.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:19.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:19.079275+0000) 2024-03-06T16:35:19.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:19.094 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:20.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:20.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:20.079508+0000) 2024-03-06T16:35:21.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:21.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:21.079792+0000) 2024-03-06T16:35:22.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:22.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:22.079990+0000) 2024-03-06T16:35:23.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:23.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:23.080234+0000) 2024-03-06T16:35:24.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:24.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:24.080447+0000) 2024-03-06T16:35:25.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:25.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:25.080711+0000) 2024-03-06T16:35:26.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:26.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:26.080926+0000) 2024-03-06T16:35:27.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:27.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:27.081191+0000) 2024-03-06T16:35:28.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:28.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:28.081428+0000) 2024-03-06T16:35:29.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:29.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:29.081654+0000) 2024-03-06T16:35:30.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:30.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:30.081950+0000) 2024-03-06T16:35:31.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:31.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:31.082213+0000) 2024-03-06T16:35:32.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:32.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:32.082415+0000) 2024-03-06T16:35:33.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:33.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:33.082596+0000) 2024-03-06T16:35:34.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:34.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:34.082782+0000) 2024-03-06T16:35:35.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:35.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:35.082980+0000) 2024-03-06T16:35:36.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:36.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:36.083228+0000) 2024-03-06T16:35:37.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:37.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:37.083488+0000) 2024-03-06T16:35:38.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:38.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:38.083758+0000) 2024-03-06T16:35:39.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:39.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:39.083943+0000) 2024-03-06T16:35:40.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:40.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:40.084170+0000) 2024-03-06T16:35:41.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:41.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:41.084443+0000) 2024-03-06T16:35:42.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:42.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:42.084644+0000) 2024-03-06T16:35:43.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:43.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:43.084909+0000) 2024-03-06T16:35:44.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:44.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:44.085144+0000) 2024-03-06T16:35:45.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:45.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:45.085326+0000) 2024-03-06T16:35:46.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:46.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:46.085534+0000) 2024-03-06T16:35:47.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:47.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:47.085806+0000) 2024-03-06T16:35:48.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:48.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:48.086001+0000) 2024-03-06T16:35:49.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:49.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:49.086225+0000) 2024-03-06T16:35:49.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:49.290 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:50.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:50.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:50.086459+0000) 2024-03-06T16:35:51.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:51.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:51.086762+0000) 2024-03-06T16:35:52.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:52.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:52.086957+0000) 2024-03-06T16:35:53.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:53.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:53.087184+0000) 2024-03-06T16:35:54.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:54.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:54.087415+0000) 2024-03-06T16:35:55.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:55.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:55.087643+0000) 2024-03-06T16:35:56.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:56.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:56.087885+0000) 2024-03-06T16:35:57.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:57.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:57.088115+0000) 2024-03-06T16:35:58.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:58.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:58.088382+0000) 2024-03-06T16:35:59.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:35:59.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:59.088658+0000) 2024-03-06T16:36:00.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:00.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:00.088927+0000) 2024-03-06T16:36:01.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:01.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:01.089151+0000) 2024-03-06T16:36:02.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:02.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:02.089373+0000) 2024-03-06T16:36:03.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:03.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:03.089555+0000) 2024-03-06T16:36:04.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:04.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:04.089786+0000) 2024-03-06T16:36:05.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:05.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:05.090001+0000) 2024-03-06T16:36:06.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:06.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:06.090217+0000) 2024-03-06T16:36:07.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:07.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:07.090381+0000) 2024-03-06T16:36:08.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:08.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:08.090601+0000) 2024-03-06T16:36:09.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:09.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:09.090846+0000) 2024-03-06T16:36:10.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:10.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:10.091075+0000) 2024-03-06T16:36:11.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:11.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:11.091312+0000) 2024-03-06T16:36:12.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:12.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:12.091542+0000) 2024-03-06T16:36:13.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:13.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:13.091760+0000) 2024-03-06T16:36:14.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:14.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:14.091955+0000) 2024-03-06T16:36:15.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:15.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:15.092188+0000) 2024-03-06T16:36:16.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:16.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:16.092391+0000) 2024-03-06T16:36:17.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:17.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:17.092626+0000) 2024-03-06T16:36:18.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:18.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:18.092844+0000) 2024-03-06T16:36:19.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:19.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:19.093013+0000) 2024-03-06T16:36:19.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:19.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:20.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:20.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:20.093212+0000) 2024-03-06T16:36:21.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:21.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:21.093444+0000) 2024-03-06T16:36:22.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:22.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:22.093697+0000) 2024-03-06T16:36:23.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:23.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:23.093918+0000) 2024-03-06T16:36:24.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:24.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:24.094143+0000) 2024-03-06T16:36:25.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:25.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:25.094354+0000) 2024-03-06T16:36:26.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:26.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:26.094575+0000) 2024-03-06T16:36:27.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:27.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:27.094796+0000) 2024-03-06T16:36:28.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:28.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:28.094942+0000) 2024-03-06T16:36:29.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:29.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:29.095172+0000) 2024-03-06T16:36:30.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:30.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:30.095413+0000) 2024-03-06T16:36:31.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:31.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:31.095701+0000) 2024-03-06T16:36:32.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:32.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:32.095931+0000) 2024-03-06T16:36:33.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:33.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:33.096173+0000) 2024-03-06T16:36:34.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:34.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:34.096411+0000) 2024-03-06T16:36:35.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:35.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:35.096609+0000) 2024-03-06T16:36:36.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:36.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:36.096876+0000) 2024-03-06T16:36:37.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:37.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:37.097073+0000) 2024-03-06T16:36:38.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:38.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:38.097254+0000) 2024-03-06T16:36:39.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:39.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:39.097481+0000) 2024-03-06T16:36:40.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:40.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:40.097752+0000) 2024-03-06T16:36:41.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:41.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:41.097993+0000) 2024-03-06T16:36:42.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:42.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:42.098189+0000) 2024-03-06T16:36:43.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:43.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:43.098425+0000) 2024-03-06T16:36:44.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:44.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:44.098697+0000) 2024-03-06T16:36:45.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:45.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:45.098902+0000) 2024-03-06T16:36:46.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:46.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:46.099122+0000) 2024-03-06T16:36:47.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:47.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:47.099360+0000) 2024-03-06T16:36:48.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:48.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:48.099621+0000) 2024-03-06T16:36:49.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:49.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:49.099830+0000) 2024-03-06T16:36:49.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:49.726 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:50.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:50.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:50.100029+0000) 2024-03-06T16:36:51.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:51.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:51.100274+0000) 2024-03-06T16:36:52.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:52.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:52.100502+0000) 2024-03-06T16:36:53.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:53.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:53.100791+0000) 2024-03-06T16:36:54.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:54.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:54.100974+0000) 2024-03-06T16:36:55.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:55.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:55.101194+0000) 2024-03-06T16:36:56.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:56.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:56.101443+0000) 2024-03-06T16:36:57.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:57.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:57.101717+0000) 2024-03-06T16:36:58.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:58.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:58.101937+0000) 2024-03-06T16:36:59.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:36:59.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:59.102174+0000) 2024-03-06T16:37:00.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:00.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:00.102403+0000) 2024-03-06T16:37:01.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:01.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:01.102646+0000) 2024-03-06T16:37:02.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:02.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:02.102869+0000) 2024-03-06T16:37:03.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:03.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:03.103051+0000) 2024-03-06T16:37:04.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:04.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:04.103286+0000) 2024-03-06T16:37:05.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:05.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:05.103522+0000) 2024-03-06T16:37:06.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:06.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:06.103786+0000) 2024-03-06T16:37:07.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:07.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:07.103968+0000) 2024-03-06T16:37:08.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:08.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:08.104171+0000) 2024-03-06T16:37:09.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:09.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:09.104382+0000) 2024-03-06T16:37:10.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:10.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:10.104615+0000) 2024-03-06T16:37:11.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:11.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:11.104820+0000) 2024-03-06T16:37:12.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:12.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:12.105063+0000) 2024-03-06T16:37:13.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:13.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:13.105293+0000) 2024-03-06T16:37:14.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:14.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:14.105532+0000) 2024-03-06T16:37:15.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:15.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:15.105785+0000) 2024-03-06T16:37:16.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:16.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:16.106045+0000) 2024-03-06T16:37:17.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:17.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:17.106323+0000) 2024-03-06T16:37:18.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:18.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:18.106596+0000) 2024-03-06T16:37:19.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:19.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:19.106862+0000) 2024-03-06T16:37:19.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:19.909 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:20.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:20.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:20.107050+0000) 2024-03-06T16:37:21.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:21.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:21.107311+0000) 2024-03-06T16:37:22.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:22.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:22.107590+0000) 2024-03-06T16:37:23.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:23.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:23.107865+0000) 2024-03-06T16:37:24.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:24.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:24.108161+0000) 2024-03-06T16:37:25.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:25.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:25.108403+0000) 2024-03-06T16:37:26.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:26.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:26.108652+0000) 2024-03-06T16:37:27.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:27.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:27.108967+0000) 2024-03-06T16:37:28.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:28.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:28.109237+0000) 2024-03-06T16:37:29.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:29.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:29.109462+0000) 2024-03-06T16:37:30.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:30.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:30.109772+0000) 2024-03-06T16:37:31.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:31.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:31.109963+0000) 2024-03-06T16:37:32.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:32.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:32.110194+0000) 2024-03-06T16:37:33.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:33.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:33.110413+0000) 2024-03-06T16:37:34.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:34.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:34.110641+0000) 2024-03-06T16:37:35.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:35.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:35.110909+0000) 2024-03-06T16:37:36.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:36.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:36.111137+0000) 2024-03-06T16:37:37.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:37.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:37.111366+0000) 2024-03-06T16:37:38.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:38.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:38.111592+0000) 2024-03-06T16:37:39.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:39.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:39.111781+0000) 2024-03-06T16:37:40.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:40.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:40.112017+0000) 2024-03-06T16:37:41.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:41.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:41.112249+0000) 2024-03-06T16:37:42.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:42.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:42.112453+0000) 2024-03-06T16:37:43.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:43.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:43.112710+0000) 2024-03-06T16:37:44.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:44.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:44.112936+0000) 2024-03-06T16:37:45.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:45.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:45.113134+0000) 2024-03-06T16:37:46.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:46.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:46.113337+0000) 2024-03-06T16:37:47.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:47.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:47.113563+0000) 2024-03-06T16:37:48.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:48.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:48.113794+0000) 2024-03-06T16:37:49.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:49.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:49.114002+0000) 2024-03-06T16:37:50.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:50.120 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:50.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:50.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:50.114280+0000) 2024-03-06T16:37:51.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:51.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:51.114565+0000) 2024-03-06T16:37:52.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:52.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:52.114805+0000) 2024-03-06T16:37:53.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:53.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:53.115023+0000) 2024-03-06T16:37:54.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:54.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:54.115247+0000) 2024-03-06T16:37:55.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:55.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:55.115470+0000) 2024-03-06T16:37:56.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:56.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:56.115782+0000) 2024-03-06T16:37:57.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:57.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:57.115976+0000) 2024-03-06T16:37:58.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:58.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:58.116197+0000) 2024-03-06T16:37:59.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:37:59.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:59.116422+0000) 2024-03-06T16:38:00.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:00.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:00.116655+0000) 2024-03-06T16:38:01.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:01.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:01.116909+0000) 2024-03-06T16:38:02.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:02.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:02.117099+0000) 2024-03-06T16:38:03.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:03.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:03.117290+0000) 2024-03-06T16:38:04.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:04.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:04.117485+0000) 2024-03-06T16:38:05.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:05.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:05.117708+0000) 2024-03-06T16:38:06.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:06.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:06.117885+0000) 2024-03-06T16:38:07.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:07.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:07.118093+0000) 2024-03-06T16:38:08.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:08.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:08.118312+0000) 2024-03-06T16:38:09.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:09.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:09.118541+0000) 2024-03-06T16:38:10.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:10.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:10.118772+0000) 2024-03-06T16:38:11.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:11.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:11.118993+0000) 2024-03-06T16:38:12.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:12.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:12.119239+0000) 2024-03-06T16:38:13.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:13.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:13.119479+0000) 2024-03-06T16:38:14.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:14.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:14.119761+0000) 2024-03-06T16:38:15.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:15.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:15.119985+0000) 2024-03-06T16:38:16.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:16.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:16.120208+0000) 2024-03-06T16:38:17.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:17.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:17.120451+0000) 2024-03-06T16:38:18.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:18.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:18.120649+0000) 2024-03-06T16:38:19.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:19.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:19.120888+0000) 2024-03-06T16:38:20.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:20.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:20.121119+0000) 2024-03-06T16:38:20.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:20.353 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:21.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:21.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:21.121322+0000) 2024-03-06T16:38:22.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:22.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:22.121553+0000) 2024-03-06T16:38:23.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:23.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:23.121763+0000) 2024-03-06T16:38:24.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:24.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:24.121994+0000) 2024-03-06T16:38:25.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:25.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:25.122218+0000) 2024-03-06T16:38:26.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:26.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:26.122451+0000) 2024-03-06T16:38:27.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:27.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:27.122712+0000) 2024-03-06T16:38:28.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:28.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:28.122847+0000) 2024-03-06T16:38:29.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:29.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:29.123024+0000) 2024-03-06T16:38:30.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:30.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:30.123216+0000) 2024-03-06T16:38:31.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:31.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:31.123408+0000) 2024-03-06T16:38:32.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:32.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:32.123616+0000) 2024-03-06T16:38:33.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:33.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:33.123814+0000) 2024-03-06T16:38:34.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:34.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:34.124035+0000) 2024-03-06T16:38:35.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:35.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:35.124217+0000) 2024-03-06T16:38:36.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:36.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:36.124440+0000) 2024-03-06T16:38:37.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:37.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:37.124692+0000) 2024-03-06T16:38:38.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:38.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:38.124918+0000) 2024-03-06T16:38:39.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:39.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:39.125142+0000) 2024-03-06T16:38:40.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:40.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:40.125367+0000) 2024-03-06T16:38:41.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:41.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:41.125567+0000) 2024-03-06T16:38:42.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:42.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:42.125838+0000) 2024-03-06T16:38:43.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:43.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:43.125994+0000) 2024-03-06T16:38:44.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:44.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:44.126190+0000) 2024-03-06T16:38:45.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:45.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:45.126416+0000) 2024-03-06T16:38:46.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:46.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:46.126644+0000) 2024-03-06T16:38:47.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:47.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:47.126892+0000) 2024-03-06T16:38:48.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:48.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:48.127116+0000) 2024-03-06T16:38:49.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:49.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:49.127289+0000) 2024-03-06T16:38:50.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:50.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:50.127520+0000) 2024-03-06T16:38:50.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:50.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:51.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:51.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:51.127834+0000) 2024-03-06T16:38:52.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:52.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:52.128068+0000) 2024-03-06T16:38:53.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:53.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:53.128332+0000) 2024-03-06T16:38:54.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:54.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:54.128605+0000) 2024-03-06T16:38:55.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:55.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:55.128843+0000) 2024-03-06T16:38:56.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:56.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:56.129079+0000) 2024-03-06T16:38:57.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:57.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:57.129318+0000) 2024-03-06T16:38:58.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:58.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:58.129553+0000) 2024-03-06T16:38:59.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:38:59.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:59.129790+0000) 2024-03-06T16:39:00.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:00.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:00.130020+0000) 2024-03-06T16:39:01.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:01.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:01.130247+0000) 2024-03-06T16:39:02.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:02.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:02.130460+0000) 2024-03-06T16:39:03.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:03.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:03.130726+0000) 2024-03-06T16:39:04.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:04.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:04.130967+0000) 2024-03-06T16:39:05.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:05.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:05.131159+0000) 2024-03-06T16:39:06.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:06.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:06.131436+0000) 2024-03-06T16:39:07.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:07.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:07.131681+0000) 2024-03-06T16:39:08.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:08.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:08.131902+0000) 2024-03-06T16:39:09.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:09.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:09.132087+0000) 2024-03-06T16:39:10.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:10.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:10.132284+0000) 2024-03-06T16:39:11.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:11.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:11.132534+0000) 2024-03-06T16:39:12.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:12.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:12.132817+0000) 2024-03-06T16:39:13.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:13.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:13.133000+0000) 2024-03-06T16:39:14.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:14.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:14.133234+0000) 2024-03-06T16:39:15.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:15.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:15.133498+0000) 2024-03-06T16:39:16.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:16.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:16.133813+0000) 2024-03-06T16:39:17.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:17.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:17.134044+0000) 2024-03-06T16:39:18.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:18.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:18.134306+0000) 2024-03-06T16:39:19.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:19.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:19.134586+0000) 2024-03-06T16:39:20.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:20.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:20.134844+0000) 2024-03-06T16:39:20.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:20.777 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:21.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:21.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:21.135098+0000) 2024-03-06T16:39:22.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:22.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:22.135373+0000) 2024-03-06T16:39:23.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:23.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:23.135658+0000) 2024-03-06T16:39:24.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:24.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:24.135900+0000) 2024-03-06T16:39:25.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:25.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:25.136116+0000) 2024-03-06T16:39:26.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:26.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:26.136340+0000) 2024-03-06T16:39:27.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:27.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:27.136548+0000) 2024-03-06T16:39:28.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:28.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:28.136776+0000) 2024-03-06T16:39:29.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:29.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:29.136975+0000) 2024-03-06T16:39:30.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:30.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:30.137228+0000) 2024-03-06T16:39:31.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:31.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:31.137506+0000) 2024-03-06T16:39:32.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:32.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:32.137813+0000) 2024-03-06T16:39:33.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:33.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:33.138018+0000) 2024-03-06T16:39:34.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:34.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:34.138249+0000) 2024-03-06T16:39:35.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:35.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:35.138466+0000) 2024-03-06T16:39:36.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:36.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:36.138730+0000) 2024-03-06T16:39:37.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:37.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:37.138959+0000) 2024-03-06T16:39:38.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:38.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:38.139191+0000) 2024-03-06T16:39:39.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:39.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:39.139419+0000) 2024-03-06T16:39:40.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:40.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:40.139650+0000) 2024-03-06T16:39:41.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:41.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:41.139888+0000) 2024-03-06T16:39:42.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:42.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:42.140079+0000) 2024-03-06T16:39:43.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:43.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:43.140301+0000) 2024-03-06T16:39:44.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:44.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:44.140477+0000) 2024-03-06T16:39:45.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:45.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:45.140757+0000) 2024-03-06T16:39:46.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:46.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:46.140989+0000) 2024-03-06T16:39:47.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:47.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:47.141262+0000) 2024-03-06T16:39:48.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:48.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:48.141543+0000) 2024-03-06T16:39:49.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:49.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:49.141819+0000) 2024-03-06T16:39:50.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:50.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:50.142052+0000) 2024-03-06T16:39:50.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:50.924 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:51.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:51.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:51.142351+0000) 2024-03-06T16:39:52.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:52.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:52.142559+0000) 2024-03-06T16:39:53.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:53.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:53.142816+0000) 2024-03-06T16:39:54.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:54.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:54.143075+0000) 2024-03-06T16:39:55.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:55.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:55.143270+0000) 2024-03-06T16:39:56.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:56.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:56.143464+0000) 2024-03-06T16:39:57.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:57.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:57.143756+0000) 2024-03-06T16:39:58.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:58.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:58.143947+0000) 2024-03-06T16:39:59.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:39:59.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:59.144172+0000) 2024-03-06T16:40:00.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:00.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:00.144403+0000) 2024-03-06T16:40:01.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:01.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:01.144638+0000) 2024-03-06T16:40:02.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:02.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:02.144930+0000) 2024-03-06T16:40:03.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:03.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:03.145212+0000) 2024-03-06T16:40:04.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:04.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:04.145477+0000) 2024-03-06T16:40:05.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:05.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:05.145780+0000) 2024-03-06T16:40:06.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:06.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:06.145993+0000) 2024-03-06T16:40:07.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:07.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:07.146225+0000) 2024-03-06T16:40:08.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:08.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:08.146405+0000) 2024-03-06T16:40:09.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:09.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:09.146580+0000) 2024-03-06T16:40:10.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:10.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:10.146819+0000) 2024-03-06T16:40:11.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:11.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:11.147059+0000) 2024-03-06T16:40:12.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:12.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:12.147282+0000) 2024-03-06T16:40:13.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:13.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:13.147483+0000) 2024-03-06T16:40:14.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:14.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:14.147764+0000) 2024-03-06T16:40:15.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:15.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:15.147990+0000) 2024-03-06T16:40:16.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:16.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:16.148216+0000) 2024-03-06T16:40:17.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:17.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:17.148443+0000) 2024-03-06T16:40:18.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:18.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:18.148627+0000) 2024-03-06T16:40:19.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:19.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:19.148819+0000) 2024-03-06T16:40:20.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:20.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:20.149046+0000) 2024-03-06T16:40:21.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:21.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:21.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:21.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:21.149233+0000) 2024-03-06T16:40:22.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:22.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:22.149459+0000) 2024-03-06T16:40:23.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:23.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:23.149703+0000) 2024-03-06T16:40:24.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:24.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:24.149916+0000) 2024-03-06T16:40:25.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:25.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:25.150190+0000) 2024-03-06T16:40:26.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:26.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:26.150395+0000) 2024-03-06T16:40:27.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:27.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:27.150645+0000) 2024-03-06T16:40:28.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:28.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:28.150869+0000) 2024-03-06T16:40:29.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:29.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:29.151060+0000) 2024-03-06T16:40:30.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:30.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:30.151239+0000) 2024-03-06T16:40:31.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:31.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:31.151469+0000) 2024-03-06T16:40:32.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:32.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:32.151738+0000) 2024-03-06T16:40:33.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:33.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:33.151961+0000) 2024-03-06T16:40:34.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:34.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:34.152174+0000) 2024-03-06T16:40:35.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:35.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:35.152371+0000) 2024-03-06T16:40:36.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:36.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:36.152624+0000) 2024-03-06T16:40:37.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:37.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:37.152891+0000) 2024-03-06T16:40:38.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:38.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:38.153123+0000) 2024-03-06T16:40:39.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:39.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:39.153331+0000) 2024-03-06T16:40:40.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:40.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:40.153552+0000) 2024-03-06T16:40:41.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:41.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:41.153800+0000) 2024-03-06T16:40:42.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:42.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:42.154049+0000) 2024-03-06T16:40:43.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:43.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:43.154285+0000) 2024-03-06T16:40:44.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:44.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:44.154510+0000) 2024-03-06T16:40:45.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:45.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:45.154763+0000) 2024-03-06T16:40:46.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:46.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:46.154953+0000) 2024-03-06T16:40:47.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:47.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:47.155248+0000) 2024-03-06T16:40:48.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:48.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:48.155481+0000) 2024-03-06T16:40:49.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:49.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:49.155751+0000) 2024-03-06T16:40:50.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:50.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:50.155948+0000) 2024-03-06T16:40:51.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:51.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:51.156180+0000) 2024-03-06T16:40:51.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:51.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:52.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:52.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:52.156417+0000) 2024-03-06T16:40:53.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:53.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:53.156641+0000) 2024-03-06T16:40:54.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:54.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:54.156871+0000) 2024-03-06T16:40:55.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:55.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:55.157056+0000) 2024-03-06T16:40:56.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:56.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:56.157275+0000) 2024-03-06T16:40:57.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:57.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:57.157504+0000) 2024-03-06T16:40:58.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:58.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:58.157802+0000) 2024-03-06T16:40:59.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:40:59.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:59.157993+0000) 2024-03-06T16:41:00.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:00.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:00.158181+0000) 2024-03-06T16:41:01.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:01.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:01.158367+0000) 2024-03-06T16:41:02.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:02.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:02.158598+0000) 2024-03-06T16:41:03.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:03.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:03.158792+0000) 2024-03-06T16:41:04.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:04.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:04.158983+0000) 2024-03-06T16:41:05.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:05.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:05.159243+0000) 2024-03-06T16:41:06.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:06.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:06.159506+0000) 2024-03-06T16:41:07.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:07.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:07.159807+0000) 2024-03-06T16:41:08.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:08.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:08.160022+0000) 2024-03-06T16:41:09.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:09.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:09.160243+0000) 2024-03-06T16:41:10.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:10.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:10.160469+0000) 2024-03-06T16:41:11.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:11.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:11.160644+0000) 2024-03-06T16:41:12.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:12.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:12.160886+0000) 2024-03-06T16:41:13.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:13.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:13.161106+0000) 2024-03-06T16:41:14.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:14.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:14.161312+0000) 2024-03-06T16:41:15.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:15.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:15.161538+0000) 2024-03-06T16:41:16.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:16.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:16.161794+0000) 2024-03-06T16:41:17.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:17.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:17.162029+0000) 2024-03-06T16:41:18.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:18.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:18.162256+0000) 2024-03-06T16:41:19.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:19.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:19.162492+0000) 2024-03-06T16:41:20.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:20.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:20.162708+0000) 2024-03-06T16:41:21.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:21.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:21.162945+0000) 2024-03-06T16:41:21.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:21.577 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:22.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:22.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:22.163134+0000) 2024-03-06T16:41:23.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:23.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:23.163362+0000) 2024-03-06T16:41:24.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:24.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:24.163596+0000) 2024-03-06T16:41:25.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:25.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:25.163837+0000) 2024-03-06T16:41:26.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:26.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:26.164062+0000) 2024-03-06T16:41:27.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:27.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:27.164308+0000) 2024-03-06T16:41:28.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:28.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:28.164470+0000) 2024-03-06T16:41:29.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:29.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:29.164747+0000) 2024-03-06T16:41:30.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:30.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:30.164946+0000) 2024-03-06T16:41:31.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:31.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:31.165190+0000) 2024-03-06T16:41:32.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:32.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:32.165464+0000) 2024-03-06T16:41:33.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:33.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:33.165633+0000) 2024-03-06T16:41:34.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:34.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:34.165855+0000) 2024-03-06T16:41:35.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:35.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:35.166131+0000) 2024-03-06T16:41:36.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:36.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:36.166367+0000) 2024-03-06T16:41:37.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:37.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:37.166586+0000) 2024-03-06T16:41:38.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:38.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:38.166826+0000) 2024-03-06T16:41:39.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:39.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:39.167053+0000) 2024-03-06T16:41:40.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:40.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:40.167235+0000) 2024-03-06T16:41:41.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:41.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:41.167468+0000) 2024-03-06T16:41:42.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:42.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:42.167754+0000) 2024-03-06T16:41:43.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:43.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:43.167985+0000) 2024-03-06T16:41:44.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:44.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:44.168123+0000) 2024-03-06T16:41:45.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:45.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:45.168291+0000) 2024-03-06T16:41:46.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:46.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:46.168526+0000) 2024-03-06T16:41:47.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:47.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:47.168785+0000) 2024-03-06T16:41:48.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:48.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:48.168978+0000) 2024-03-06T16:41:49.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:49.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:49.169198+0000) 2024-03-06T16:41:50.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:50.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:50.169419+0000) 2024-03-06T16:41:51.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:51.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:51.169655+0000) 2024-03-06T16:41:51.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:51.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:52.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:52.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:52.169905+0000) 2024-03-06T16:41:53.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:53.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:53.170129+0000) 2024-03-06T16:41:54.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:54.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:54.170351+0000) 2024-03-06T16:41:55.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:55.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:55.170575+0000) 2024-03-06T16:41:56.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:56.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:56.170911+0000) 2024-03-06T16:41:57.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:57.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:57.171151+0000) 2024-03-06T16:41:58.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:58.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:58.171335+0000) 2024-03-06T16:41:59.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:41:59.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:59.171570+0000) 2024-03-06T16:42:00.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:00.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:00.171795+0000) 2024-03-06T16:42:01.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:01.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:01.171969+0000) 2024-03-06T16:42:02.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:02.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:02.172149+0000) 2024-03-06T16:42:03.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:03.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:03.172377+0000) 2024-03-06T16:42:04.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:04.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:04.172559+0000) 2024-03-06T16:42:05.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:05.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:05.172797+0000) 2024-03-06T16:42:06.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:06.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:06.173020+0000) 2024-03-06T16:42:07.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:07.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:07.173225+0000) 2024-03-06T16:42:08.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:08.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:08.173465+0000) 2024-03-06T16:42:09.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:09.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:09.173660+0000) 2024-03-06T16:42:10.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:10.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:10.173898+0000) 2024-03-06T16:42:11.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:11.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:11.174040+0000) 2024-03-06T16:42:12.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:12.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:12.174265+0000) 2024-03-06T16:42:13.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:13.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:13.174501+0000) 2024-03-06T16:42:14.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:14.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:14.174774+0000) 2024-03-06T16:42:15.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:15.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:15.174925+0000) 2024-03-06T16:42:16.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:16.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:16.175110+0000) 2024-03-06T16:42:17.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:17.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:17.175283+0000) 2024-03-06T16:42:18.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:18.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:18.175530+0000) 2024-03-06T16:42:19.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:19.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:19.175783+0000) 2024-03-06T16:42:20.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:20.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:20.175955+0000) 2024-03-06T16:42:21.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:21.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:21.176190+0000) 2024-03-06T16:42:21.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:22.003 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:22.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:22.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:22.176363+0000) 2024-03-06T16:42:23.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:23.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:23.176548+0000) 2024-03-06T16:42:24.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:24.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:24.176792+0000) 2024-03-06T16:42:25.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:25.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:25.176976+0000) 2024-03-06T16:42:26.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:26.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:26.177128+0000) 2024-03-06T16:42:27.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:27.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:27.177351+0000) 2024-03-06T16:42:28.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:28.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:28.177590+0000) 2024-03-06T16:42:29.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:29.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:29.177829+0000) 2024-03-06T16:42:30.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:30.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:30.178056+0000) 2024-03-06T16:42:31.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:31.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:31.178291+0000) 2024-03-06T16:42:32.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:32.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:32.178521+0000) 2024-03-06T16:42:33.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:33.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:33.178829+0000) 2024-03-06T16:42:34.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:34.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:34.178992+0000) 2024-03-06T16:42:35.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:35.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:35.179178+0000) 2024-03-06T16:42:36.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:36.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:36.179372+0000) 2024-03-06T16:42:37.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:37.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:37.179596+0000) 2024-03-06T16:42:38.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:38.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:38.179816+0000) 2024-03-06T16:42:39.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:39.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:39.180011+0000) 2024-03-06T16:42:40.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:40.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:40.180229+0000) 2024-03-06T16:42:41.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:41.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:41.180455+0000) 2024-03-06T16:42:42.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:42.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:42.180720+0000) 2024-03-06T16:42:43.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:43.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:43.180916+0000) 2024-03-06T16:42:44.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:44.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:44.181155+0000) 2024-03-06T16:42:45.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:45.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:45.181374+0000) 2024-03-06T16:42:46.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:46.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:46.181597+0000) 2024-03-06T16:42:47.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:47.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:47.181829+0000) 2024-03-06T16:42:48.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:48.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:48.182024+0000) 2024-03-06T16:42:49.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:49.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:49.182234+0000) 2024-03-06T16:42:50.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:50.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:50.182459+0000) 2024-03-06T16:42:51.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:51.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:51.182737+0000) 2024-03-06T16:42:52.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:52.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:52.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:52.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:52.182959+0000) 2024-03-06T16:42:53.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:53.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:53.183141+0000) 2024-03-06T16:42:54.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:54.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:54.183362+0000) 2024-03-06T16:42:55.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:55.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:55.183563+0000) 2024-03-06T16:42:56.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:56.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:56.183794+0000) 2024-03-06T16:42:57.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:57.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:57.184036+0000) 2024-03-06T16:42:58.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:58.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:58.184264+0000) 2024-03-06T16:42:59.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:42:59.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:59.184494+0000) 2024-03-06T16:43:00.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:00.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:00.184754+0000) 2024-03-06T16:43:01.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:01.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:01.184980+0000) 2024-03-06T16:43:02.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:02.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:02.185204+0000) 2024-03-06T16:43:03.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:03.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:03.185428+0000) 2024-03-06T16:43:04.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:04.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:04.185651+0000) 2024-03-06T16:43:05.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:05.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:05.185917+0000) 2024-03-06T16:43:06.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:06.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:06.186108+0000) 2024-03-06T16:43:07.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:07.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:07.186300+0000) 2024-03-06T16:43:08.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:08.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:08.186527+0000) 2024-03-06T16:43:09.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:09.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:09.186793+0000) 2024-03-06T16:43:10.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:10.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:10.187015+0000) 2024-03-06T16:43:11.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:11.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:11.187204+0000) 2024-03-06T16:43:12.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:12.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:12.187425+0000) 2024-03-06T16:43:13.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:13.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:13.187617+0000) 2024-03-06T16:43:14.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:14.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:14.187834+0000) 2024-03-06T16:43:15.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:15.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:15.188028+0000) 2024-03-06T16:43:16.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:16.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:16.188226+0000) 2024-03-06T16:43:17.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:17.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:17.188456+0000) 2024-03-06T16:43:18.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:18.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:18.188716+0000) 2024-03-06T16:43:19.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:19.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:19.188947+0000) 2024-03-06T16:43:20.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:20.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:20.189167+0000) 2024-03-06T16:43:21.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:21.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:21.189365+0000) 2024-03-06T16:43:22.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:22.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:22.189558+0000) 2024-03-06T16:43:22.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:22.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:23.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:23.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:23.189760+0000) 2024-03-06T16:43:24.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:24.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:24.189978+0000) 2024-03-06T16:43:25.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:25.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:25.190173+0000) 2024-03-06T16:43:26.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:26.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:26.190404+0000) 2024-03-06T16:43:27.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:27.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:27.190632+0000) 2024-03-06T16:43:28.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:28.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:28.190848+0000) 2024-03-06T16:43:29.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:29.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:29.191065+0000) 2024-03-06T16:43:30.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:30.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:30.191268+0000) 2024-03-06T16:43:31.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:31.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:31.191475+0000) 2024-03-06T16:43:32.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:32.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:32.191754+0000) 2024-03-06T16:43:33.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:33.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:33.192000+0000) 2024-03-06T16:43:34.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:34.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:34.192224+0000) 2024-03-06T16:43:35.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:35.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:35.192407+0000) 2024-03-06T16:43:36.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:36.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:36.192634+0000) 2024-03-06T16:43:37.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:37.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:37.192863+0000) 2024-03-06T16:43:38.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:38.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:38.193093+0000) 2024-03-06T16:43:39.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:39.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:39.193326+0000) 2024-03-06T16:43:40.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:40.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:40.193549+0000) 2024-03-06T16:43:41.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:41.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:41.193782+0000) 2024-03-06T16:43:42.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:42.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:42.193988+0000) 2024-03-06T16:43:43.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:43.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:43.194225+0000) 2024-03-06T16:43:44.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:44.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:44.194379+0000) 2024-03-06T16:43:45.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:45.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:45.194588+0000) 2024-03-06T16:43:46.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:46.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:46.194791+0000) 2024-03-06T16:43:47.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:47.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:47.194995+0000) 2024-03-06T16:43:48.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:48.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:48.195168+0000) 2024-03-06T16:43:49.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:49.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:49.195340+0000) 2024-03-06T16:43:50.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:50.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:50.195533+0000) 2024-03-06T16:43:51.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:51.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:51.195798+0000) 2024-03-06T16:43:52.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:52.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:52.195992+0000) 2024-03-06T16:43:52.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:52.569 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:53.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:53.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:53.196215+0000) 2024-03-06T16:43:54.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:54.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:54.196453+0000) 2024-03-06T16:43:55.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:55.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:55.196716+0000) 2024-03-06T16:43:56.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:56.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:56.196907+0000) 2024-03-06T16:43:57.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:57.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:57.197139+0000) 2024-03-06T16:43:58.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:58.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:58.197403+0000) 2024-03-06T16:43:59.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:43:59.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:59.197596+0000) 2024-03-06T16:44:00.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:00.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:00.197800+0000) 2024-03-06T16:44:01.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:01.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:01.198046+0000) 2024-03-06T16:44:02.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:02.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:02.198232+0000) 2024-03-06T16:44:03.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:03.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:03.198479+0000) 2024-03-06T16:44:04.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:04.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:04.198847+0000) 2024-03-06T16:44:05.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:05.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:05.199066+0000) 2024-03-06T16:44:06.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:06.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:06.199258+0000) 2024-03-06T16:44:07.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:07.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:07.199477+0000) 2024-03-06T16:44:08.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:08.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:08.199661+0000) 2024-03-06T16:44:09.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:09.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:09.199925+0000) 2024-03-06T16:44:10.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:10.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:10.200157+0000) 2024-03-06T16:44:11.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:11.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:11.200396+0000) 2024-03-06T16:44:12.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:12.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:12.200621+0000) 2024-03-06T16:44:13.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:13.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:13.200903+0000) 2024-03-06T16:44:14.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:14.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:14.201146+0000) 2024-03-06T16:44:15.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:15.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:15.201347+0000) 2024-03-06T16:44:16.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:16.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:16.201586+0000) 2024-03-06T16:44:17.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:17.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:17.201852+0000) 2024-03-06T16:44:18.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:18.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:18.202081+0000) 2024-03-06T16:44:19.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:19.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:19.202267+0000) 2024-03-06T16:44:20.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:20.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:20.202506+0000) 2024-03-06T16:44:21.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:21.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:21.202787+0000) 2024-03-06T16:44:22.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:22.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:22.203023+0000) 2024-03-06T16:44:22.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:22.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:23.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:23.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:23.203223+0000) 2024-03-06T16:44:24.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:24.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:24.203464+0000) 2024-03-06T16:44:25.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:25.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:25.203702+0000) 2024-03-06T16:44:26.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:26.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:26.203897+0000) 2024-03-06T16:44:27.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:27.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:27.204083+0000) 2024-03-06T16:44:28.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:28.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:28.204294+0000) 2024-03-06T16:44:29.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:29.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:29.204479+0000) 2024-03-06T16:44:30.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:30.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:30.204781+0000) 2024-03-06T16:44:31.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:31.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:31.205002+0000) 2024-03-06T16:44:32.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:32.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:32.205245+0000) 2024-03-06T16:44:33.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:33.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:33.205452+0000) 2024-03-06T16:44:34.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:34.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:34.205734+0000) 2024-03-06T16:44:35.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:35.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:35.205941+0000) 2024-03-06T16:44:36.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:36.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:36.206183+0000) 2024-03-06T16:44:37.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:37.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:37.206420+0000) 2024-03-06T16:44:38.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:38.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:38.206702+0000) 2024-03-06T16:44:39.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:39.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:39.206911+0000) 2024-03-06T16:44:40.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:40.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:40.207147+0000) 2024-03-06T16:44:41.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:41.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:41.207402+0000) 2024-03-06T16:44:42.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:42.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:42.207651+0000) 2024-03-06T16:44:43.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:43.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:43.207908+0000) 2024-03-06T16:44:44.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:44.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:44.208141+0000) 2024-03-06T16:44:45.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:45.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:45.208342+0000) 2024-03-06T16:44:46.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:46.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:46.208582+0000) 2024-03-06T16:44:47.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:47.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:47.208821+0000) 2024-03-06T16:44:48.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:48.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:48.209015+0000) 2024-03-06T16:44:49.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:49.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:49.209246+0000) 2024-03-06T16:44:50.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:50.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:50.209441+0000) 2024-03-06T16:44:51.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:51.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:51.209735+0000) 2024-03-06T16:44:52.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:52.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:52.209917+0000) 2024-03-06T16:44:52.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:52.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:53.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:53.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:53.210113+0000) 2024-03-06T16:44:54.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:54.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:54.210303+0000) 2024-03-06T16:44:55.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:55.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:55.210500+0000) 2024-03-06T16:44:56.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:56.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:56.210772+0000) 2024-03-06T16:44:57.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:57.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:57.210970+0000) 2024-03-06T16:44:58.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:58.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:58.211261+0000) 2024-03-06T16:44:59.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:44:59.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:59.211525+0000) 2024-03-06T16:45:00.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:00.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:00.211775+0000) 2024-03-06T16:45:01.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:01.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:01.211989+0000) 2024-03-06T16:45:02.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:02.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:02.212222+0000) 2024-03-06T16:45:03.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:03.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:03.212466+0000) 2024-03-06T16:45:04.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:04.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:04.212761+0000) 2024-03-06T16:45:05.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:05.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:05.212978+0000) 2024-03-06T16:45:06.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:06.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:06.213174+0000) 2024-03-06T16:45:07.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:07.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:07.213402+0000) 2024-03-06T16:45:08.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:08.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:08.213710+0000) 2024-03-06T16:45:09.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:09.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:09.213996+0000) 2024-03-06T16:45:10.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:10.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:10.214275+0000) 2024-03-06T16:45:11.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:11.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:11.214484+0000) 2024-03-06T16:45:12.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:12.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:12.214778+0000) 2024-03-06T16:45:13.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:13.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:13.214990+0000) 2024-03-06T16:45:14.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:14.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:14.215241+0000) 2024-03-06T16:45:15.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:15.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:15.215470+0000) 2024-03-06T16:45:16.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:16.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:16.215732+0000) 2024-03-06T16:45:17.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:17.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:17.216007+0000) 2024-03-06T16:45:18.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:18.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:18.216277+0000) 2024-03-06T16:45:19.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:19.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:19.216549+0000) 2024-03-06T16:45:20.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:20.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:20.216832+0000) 2024-03-06T16:45:21.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:21.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:21.217123+0000) 2024-03-06T16:45:22.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:22.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:22.217389+0000) 2024-03-06T16:45:23.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:23.214 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:23.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:23.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:23.217599+0000) 2024-03-06T16:45:24.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:24.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:24.217821+0000) 2024-03-06T16:45:25.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:25.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:25.218013+0000) 2024-03-06T16:45:26.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:26.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:26.218239+0000) 2024-03-06T16:45:27.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:27.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:27.218467+0000) 2024-03-06T16:45:28.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:28.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:28.218735+0000) 2024-03-06T16:45:29.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:29.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:29.218936+0000) 2024-03-06T16:45:30.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:30.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:30.219162+0000) 2024-03-06T16:45:31.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:31.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:31.219397+0000) 2024-03-06T16:45:32.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:32.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:32.219703+0000) 2024-03-06T16:45:33.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:33.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:33.219924+0000) 2024-03-06T16:45:34.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:34.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:34.220145+0000) 2024-03-06T16:45:35.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:35.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:35.220367+0000) 2024-03-06T16:45:36.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:36.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:36.220594+0000) 2024-03-06T16:45:37.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:37.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:37.220821+0000) 2024-03-06T16:45:38.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:38.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:38.221024+0000) 2024-03-06T16:45:39.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:39.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:39.221209+0000) 2024-03-06T16:45:40.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:40.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:40.221434+0000) 2024-03-06T16:45:41.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:41.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:41.221699+0000) 2024-03-06T16:45:42.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:42.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:42.221907+0000) 2024-03-06T16:45:43.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:43.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:43.222139+0000) 2024-03-06T16:45:44.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:44.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:44.222340+0000) 2024-03-06T16:45:45.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:45.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:45.222577+0000) 2024-03-06T16:45:46.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:46.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:46.222863+0000) 2024-03-06T16:45:47.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:47.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:47.223051+0000) 2024-03-06T16:45:48.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:48.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:48.223282+0000) 2024-03-06T16:45:49.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:49.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:49.223509+0000) 2024-03-06T16:45:50.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:50.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:50.223722+0000) 2024-03-06T16:45:51.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:51.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:51.223939+0000) 2024-03-06T16:45:52.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:52.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:52.224147+0000) 2024-03-06T16:45:53.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:53.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:53.224386+0000) 2024-03-06T16:45:53.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:53.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:54.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:54.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:54.224608+0000) 2024-03-06T16:45:55.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:55.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:55.224818+0000) 2024-03-06T16:45:56.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:56.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:56.225047+0000) 2024-03-06T16:45:57.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:57.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:57.225273+0000) 2024-03-06T16:45:58.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:58.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:58.225467+0000) 2024-03-06T16:45:59.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:45:59.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:59.225724+0000) 2024-03-06T16:46:00.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:00.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:00.225930+0000) 2024-03-06T16:46:01.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:01.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:01.226176+0000) 2024-03-06T16:46:02.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:02.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:02.226444+0000) 2024-03-06T16:46:03.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:03.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:03.226717+0000) 2024-03-06T16:46:04.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:04.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:04.226956+0000) 2024-03-06T16:46:05.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:05.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:05.227217+0000) 2024-03-06T16:46:06.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:06.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:06.227442+0000) 2024-03-06T16:46:07.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:07.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:07.227642+0000) 2024-03-06T16:46:08.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:08.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:08.227908+0000) 2024-03-06T16:46:09.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:09.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:09.228144+0000) 2024-03-06T16:46:10.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:10.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:10.228338+0000) 2024-03-06T16:46:11.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:11.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:11.228624+0000) 2024-03-06T16:46:12.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:12.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:12.228937+0000) 2024-03-06T16:46:13.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:13.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:13.229215+0000) 2024-03-06T16:46:14.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:14.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:14.229452+0000) 2024-03-06T16:46:15.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:15.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:15.229705+0000) 2024-03-06T16:46:16.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:16.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:16.229949+0000) 2024-03-06T16:46:17.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:17.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:17.230177+0000) 2024-03-06T16:46:18.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:18.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:18.230406+0000) 2024-03-06T16:46:19.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:19.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:19.230721+0000) 2024-03-06T16:46:20.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:20.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:20.230964+0000) 2024-03-06T16:46:21.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:21.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:21.231208+0000) 2024-03-06T16:46:22.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:22.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:22.231469+0000) 2024-03-06T16:46:23.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:23.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:23.231708+0000) 2024-03-06T16:46:23.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:23.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:24.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:24.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:24.231964+0000) 2024-03-06T16:46:25.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:25.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:25.232159+0000) 2024-03-06T16:46:26.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:26.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:26.232366+0000) 2024-03-06T16:46:27.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:27.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:27.232565+0000) 2024-03-06T16:46:28.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:28.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:28.232856+0000) 2024-03-06T16:46:29.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:29.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:29.233075+0000) 2024-03-06T16:46:30.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:30.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:30.233291+0000) 2024-03-06T16:46:31.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:31.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:31.233524+0000) 2024-03-06T16:46:32.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:32.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:32.233781+0000) 2024-03-06T16:46:33.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:33.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:33.234044+0000) 2024-03-06T16:46:34.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:34.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:34.234356+0000) 2024-03-06T16:46:35.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:35.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:35.234621+0000) 2024-03-06T16:46:36.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:36.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:36.234936+0000) 2024-03-06T16:46:37.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:37.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:37.235179+0000) 2024-03-06T16:46:38.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:38.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:38.235463+0000) 2024-03-06T16:46:39.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:39.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:39.235791+0000) 2024-03-06T16:46:40.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:40.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:40.236032+0000) 2024-03-06T16:46:41.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:41.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:41.236230+0000) 2024-03-06T16:46:42.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:42.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:42.236467+0000) 2024-03-06T16:46:43.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:43.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:43.236735+0000) 2024-03-06T16:46:44.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:44.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:44.236967+0000) 2024-03-06T16:46:45.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:45.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:45.237255+0000) 2024-03-06T16:46:46.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:46.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:46.237460+0000) 2024-03-06T16:46:47.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:47.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:47.237758+0000) 2024-03-06T16:46:48.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:48.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:48.237982+0000) 2024-03-06T16:46:49.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:49.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:49.238231+0000) 2024-03-06T16:46:50.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:50.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:50.238471+0000) 2024-03-06T16:46:51.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:51.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:51.238799+0000) 2024-03-06T16:46:52.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:52.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:52.239048+0000) 2024-03-06T16:46:53.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:53.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:53.239297+0000) 2024-03-06T16:46:53.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:53.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:54.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:54.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:54.239539+0000) 2024-03-06T16:46:55.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:55.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:55.239783+0000) 2024-03-06T16:46:56.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:56.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:56.239986+0000) 2024-03-06T16:46:57.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:57.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:57.240223+0000) 2024-03-06T16:46:58.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:58.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:58.240474+0000) 2024-03-06T16:46:59.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:46:59.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:59.240753+0000) 2024-03-06T16:47:00.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:00.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:00.240925+0000) 2024-03-06T16:47:01.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:01.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:01.241184+0000) 2024-03-06T16:47:02.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:02.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:02.241431+0000) 2024-03-06T16:47:03.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:03.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:03.241761+0000) 2024-03-06T16:47:04.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:04.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:04.242044+0000) 2024-03-06T16:47:05.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:05.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:05.242328+0000) 2024-03-06T16:47:06.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:06.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:06.242579+0000) 2024-03-06T16:47:07.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:07.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:07.242777+0000) 2024-03-06T16:47:08.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:08.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:08.243045+0000) 2024-03-06T16:47:09.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:09.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:09.243329+0000) 2024-03-06T16:47:10.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:10.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:10.243594+0000) 2024-03-06T16:47:11.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:11.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:11.243820+0000) 2024-03-06T16:47:12.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:12.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:12.244041+0000) 2024-03-06T16:47:13.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:13.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:13.244271+0000) 2024-03-06T16:47:14.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:14.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:14.244548+0000) 2024-03-06T16:47:15.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:15.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:15.244801+0000) 2024-03-06T16:47:16.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:16.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:16.245071+0000) 2024-03-06T16:47:17.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:17.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:17.245353+0000) 2024-03-06T16:47:18.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:18.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:18.245578+0000) 2024-03-06T16:47:19.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:19.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:19.245850+0000) 2024-03-06T16:47:20.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:20.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:20.246089+0000) 2024-03-06T16:47:21.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:21.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:21.246320+0000) 2024-03-06T16:47:22.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:22.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:22.246517+0000) 2024-03-06T16:47:23.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:23.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:23.246750+0000) 2024-03-06T16:47:23.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:24.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:24.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:24.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:24.246938+0000) 2024-03-06T16:47:25.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:25.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:25.247120+0000) 2024-03-06T16:47:26.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:26.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:26.247340+0000) 2024-03-06T16:47:27.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:27.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:27.247600+0000) 2024-03-06T16:47:28.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:28.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:28.247826+0000) 2024-03-06T16:47:29.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:29.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:29.248028+0000) 2024-03-06T16:47:30.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:30.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:30.248244+0000) 2024-03-06T16:47:31.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:31.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:31.248492+0000) 2024-03-06T16:47:32.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:32.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:32.248758+0000) 2024-03-06T16:47:33.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:33.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:33.248958+0000) 2024-03-06T16:47:34.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:34.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:34.249183+0000) 2024-03-06T16:47:35.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:35.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:35.249411+0000) 2024-03-06T16:47:36.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:36.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:36.249628+0000) 2024-03-06T16:47:37.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:37.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:37.249860+0000) 2024-03-06T16:47:38.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:38.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:38.250118+0000) 2024-03-06T16:47:39.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:39.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:39.250383+0000) 2024-03-06T16:47:40.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:40.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:40.250571+0000) 2024-03-06T16:47:41.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:41.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:41.250805+0000) 2024-03-06T16:47:42.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:42.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:42.250981+0000) 2024-03-06T16:47:43.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:43.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:43.251195+0000) 2024-03-06T16:47:44.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:44.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:44.251439+0000) 2024-03-06T16:47:45.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:45.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:45.251717+0000) 2024-03-06T16:47:46.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:46.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:46.251925+0000) 2024-03-06T16:47:47.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:47.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:47.252170+0000) 2024-03-06T16:47:48.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:48.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:48.252380+0000) 2024-03-06T16:47:49.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:49.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:49.252588+0000) 2024-03-06T16:47:50.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:50.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:50.252826+0000) 2024-03-06T16:47:51.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:51.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:51.253029+0000) 2024-03-06T16:47:52.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:52.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:52.253254+0000) 2024-03-06T16:47:53.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:53.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:53.253436+0000) 2024-03-06T16:47:54.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:54.143 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:54.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:54.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:54.253739+0000) 2024-03-06T16:47:55.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:55.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:55.253948+0000) 2024-03-06T16:47:56.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:56.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:56.254171+0000) 2024-03-06T16:47:57.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:57.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:57.254401+0000) 2024-03-06T16:47:58.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:58.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:58.254641+0000) 2024-03-06T16:47:59.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:47:59.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:59.254889+0000) 2024-03-06T16:48:00.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:00.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:00.255091+0000) 2024-03-06T16:48:01.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:01.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:01.255278+0000) 2024-03-06T16:48:02.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:02.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:02.255507+0000) 2024-03-06T16:48:03.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:03.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:03.255754+0000) 2024-03-06T16:48:04.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:04.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:04.255982+0000) 2024-03-06T16:48:05.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:05.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:05.256219+0000) 2024-03-06T16:48:06.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:06.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:06.256444+0000) 2024-03-06T16:48:07.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:07.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:07.256723+0000) 2024-03-06T16:48:08.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:08.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:08.256952+0000) 2024-03-06T16:48:09.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:09.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:09.257200+0000) 2024-03-06T16:48:10.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:10.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:10.257384+0000) 2024-03-06T16:48:11.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:11.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:11.257627+0000) 2024-03-06T16:48:12.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:12.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:12.257858+0000) 2024-03-06T16:48:13.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:13.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:13.258095+0000) 2024-03-06T16:48:14.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:14.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:14.258329+0000) 2024-03-06T16:48:15.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:15.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:15.258567+0000) 2024-03-06T16:48:16.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:16.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:16.258810+0000) 2024-03-06T16:48:17.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:17.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:17.259001+0000) 2024-03-06T16:48:18.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:18.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:18.259236+0000) 2024-03-06T16:48:19.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:19.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:19.259455+0000) 2024-03-06T16:48:20.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:20.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:20.259718+0000) 2024-03-06T16:48:21.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:21.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:21.259927+0000) 2024-03-06T16:48:22.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:22.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:22.260152+0000) 2024-03-06T16:48:23.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:23.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:23.260392+0000) 2024-03-06T16:48:24.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:24.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:24.260627+0000) 2024-03-06T16:48:24.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:24.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:25.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:25.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:25.260840+0000) 2024-03-06T16:48:26.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:26.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:26.261065+0000) 2024-03-06T16:48:27.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:27.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:27.261287+0000) 2024-03-06T16:48:28.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:28.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:28.261513+0000) 2024-03-06T16:48:29.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:29.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:29.261776+0000) 2024-03-06T16:48:30.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:30.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:30.262002+0000) 2024-03-06T16:48:31.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:31.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:31.262210+0000) 2024-03-06T16:48:32.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:32.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:32.262446+0000) 2024-03-06T16:48:33.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:33.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:33.262645+0000) 2024-03-06T16:48:34.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:34.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:34.262884+0000) 2024-03-06T16:48:35.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:35.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:35.263104+0000) 2024-03-06T16:48:36.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:36.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:36.263320+0000) 2024-03-06T16:48:37.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:37.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:37.263510+0000) 2024-03-06T16:48:38.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:38.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:38.263814+0000) 2024-03-06T16:48:39.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:39.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:39.264030+0000) 2024-03-06T16:48:40.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:40.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:40.264261+0000) 2024-03-06T16:48:41.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:41.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:41.264513+0000) 2024-03-06T16:48:42.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:42.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:42.264816+0000) 2024-03-06T16:48:43.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:43.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:43.265045+0000) 2024-03-06T16:48:44.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:44.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:44.265295+0000) 2024-03-06T16:48:45.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:45.264+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:45.265493+0000) 2024-03-06T16:48:46.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:46.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:46.265726+0000) 2024-03-06T16:48:47.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:47.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:47.265918+0000) 2024-03-06T16:48:48.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:48.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:48.266108+0000) 2024-03-06T16:48:49.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:49.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:49.266304+0000) 2024-03-06T16:48:50.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:50.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:50.266536+0000) 2024-03-06T16:48:51.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:51.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:51.266775+0000) 2024-03-06T16:48:52.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:52.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:52.266999+0000) 2024-03-06T16:48:53.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:53.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:53.267269+0000) 2024-03-06T16:48:54.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:54.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:54.267486+0000) 2024-03-06T16:48:54.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:54.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:55.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:55.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:55.267719+0000) 2024-03-06T16:48:56.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:56.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:56.267949+0000) 2024-03-06T16:48:57.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:57.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:57.268173+0000) 2024-03-06T16:48:58.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:58.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:58.268404+0000) 2024-03-06T16:48:59.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:48:59.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:59.268627+0000) 2024-03-06T16:49:00.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:00.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:00.268804+0000) 2024-03-06T16:49:01.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:01.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:01.269038+0000) 2024-03-06T16:49:02.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:02.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:02.269254+0000) 2024-03-06T16:49:03.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:03.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:03.269484+0000) 2024-03-06T16:49:04.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:04.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:04.269750+0000) 2024-03-06T16:49:05.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:05.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:05.269971+0000) 2024-03-06T16:49:06.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:06.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:06.270175+0000) 2024-03-06T16:49:07.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:07.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:07.270412+0000) 2024-03-06T16:49:08.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:08.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:08.270612+0000) 2024-03-06T16:49:09.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:09.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:09.270814+0000) 2024-03-06T16:49:10.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:10.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:10.271017+0000) 2024-03-06T16:49:11.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:11.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:11.271240+0000) 2024-03-06T16:49:12.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:12.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:12.271461+0000) 2024-03-06T16:49:13.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:13.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:13.271728+0000) 2024-03-06T16:49:14.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:14.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:14.271939+0000) 2024-03-06T16:49:15.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:15.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:15.272165+0000) 2024-03-06T16:49:16.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:16.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:16.272370+0000) 2024-03-06T16:49:17.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:17.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:17.272744+0000) 2024-03-06T16:49:18.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:18.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:18.272971+0000) 2024-03-06T16:49:19.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:19.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:19.273166+0000) 2024-03-06T16:49:20.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:20.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:20.273389+0000) 2024-03-06T16:49:21.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:21.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:21.273556+0000) 2024-03-06T16:49:22.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:22.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:22.273797+0000) 2024-03-06T16:49:23.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:23.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:23.273998+0000) 2024-03-06T16:49:24.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:24.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:24.274226+0000) 2024-03-06T16:49:24.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:24.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:25.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:25.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:25.274419+0000) 2024-03-06T16:49:26.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:26.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:26.274652+0000) 2024-03-06T16:49:27.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:27.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:27.274889+0000) 2024-03-06T16:49:28.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:28.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:28.275087+0000) 2024-03-06T16:49:29.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:29.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:29.275316+0000) 2024-03-06T16:49:30.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:30.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:30.275547+0000) 2024-03-06T16:49:31.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:31.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:31.275735+0000) 2024-03-06T16:49:32.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:32.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:32.275970+0000) 2024-03-06T16:49:33.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:33.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:33.276162+0000) 2024-03-06T16:49:34.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:34.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:34.276387+0000) 2024-03-06T16:49:35.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:35.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:35.276609+0000) 2024-03-06T16:49:36.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:36.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:36.276816+0000) 2024-03-06T16:49:37.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:37.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:37.277005+0000) 2024-03-06T16:49:38.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:38.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:38.277196+0000) 2024-03-06T16:49:39.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:39.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:39.277427+0000) 2024-03-06T16:49:40.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:40.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:40.277660+0000) 2024-03-06T16:49:41.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:41.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:41.277850+0000) 2024-03-06T16:49:42.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:42.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:42.278019+0000) 2024-03-06T16:49:43.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:43.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:43.278194+0000) 2024-03-06T16:49:44.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:44.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:44.278399+0000) 2024-03-06T16:49:45.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:45.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:45.278618+0000) 2024-03-06T16:49:46.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:46.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:46.278785+0000) 2024-03-06T16:49:47.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:47.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:47.278937+0000) 2024-03-06T16:49:48.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:48.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:48.279167+0000) 2024-03-06T16:49:49.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:49.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:49.279390+0000) 2024-03-06T16:49:50.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:50.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:50.279588+0000) 2024-03-06T16:49:51.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:51.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:51.279827+0000) 2024-03-06T16:49:52.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:52.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:52.280026+0000) 2024-03-06T16:49:53.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:53.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:53.280275+0000) 2024-03-06T16:49:54.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:54.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:54.280550+0000) 2024-03-06T16:49:54.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:54.921 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:55.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:55.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:55.280815+0000) 2024-03-06T16:49:56.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:56.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:56.281100+0000) 2024-03-06T16:49:57.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:57.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:57.281333+0000) 2024-03-06T16:49:58.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:58.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:58.281511+0000) 2024-03-06T16:49:59.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:49:59.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:59.281805+0000) 2024-03-06T16:50:00.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:00.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:00.282079+0000) 2024-03-06T16:50:01.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:01.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:01.282362+0000) 2024-03-06T16:50:02.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:02.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:02.282630+0000) 2024-03-06T16:50:03.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:03.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:03.282885+0000) 2024-03-06T16:50:04.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:04.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:04.283083+0000) 2024-03-06T16:50:05.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:05.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:05.283328+0000) 2024-03-06T16:50:06.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:06.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:06.283621+0000) 2024-03-06T16:50:07.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:07.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:07.283889+0000) 2024-03-06T16:50:08.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:08.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:08.284115+0000) 2024-03-06T16:50:09.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:09.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:09.284386+0000) 2024-03-06T16:50:10.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:10.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:10.284553+0000) 2024-03-06T16:50:11.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:11.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:11.284787+0000) 2024-03-06T16:50:12.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:12.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:12.284994+0000) 2024-03-06T16:50:13.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:13.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:13.285226+0000) 2024-03-06T16:50:14.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:14.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:14.285449+0000) 2024-03-06T16:50:15.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:15.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:15.285645+0000) 2024-03-06T16:50:16.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:16.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:16.285874+0000) 2024-03-06T16:50:17.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:17.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:17.286054+0000) 2024-03-06T16:50:18.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:18.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:18.286284+0000) 2024-03-06T16:50:19.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:19.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:19.286539+0000) 2024-03-06T16:50:20.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:20.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:20.286830+0000) 2024-03-06T16:50:21.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:21.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:21.287070+0000) 2024-03-06T16:50:22.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:22.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:22.287332+0000) 2024-03-06T16:50:23.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:23.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:23.287573+0000) 2024-03-06T16:50:24.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:24.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:24.287845+0000) 2024-03-06T16:50:25.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:25.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:25.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:25.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:25.288088+0000) 2024-03-06T16:50:26.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:26.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:26.288359+0000) 2024-03-06T16:50:27.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:27.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:27.288633+0000) 2024-03-06T16:50:28.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:28.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:28.288907+0000) 2024-03-06T16:50:29.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:29.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:29.289187+0000) 2024-03-06T16:50:30.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:30.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:30.289465+0000) 2024-03-06T16:50:31.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:31.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:31.289787+0000) 2024-03-06T16:50:32.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:32.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:32.289943+0000) 2024-03-06T16:50:33.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:33.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:33.290146+0000) 2024-03-06T16:50:34.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:34.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:34.290376+0000) 2024-03-06T16:50:35.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:35.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:35.290602+0000) 2024-03-06T16:50:36.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:36.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:36.290819+0000) 2024-03-06T16:50:37.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:37.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:37.291050+0000) 2024-03-06T16:50:38.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:38.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:38.291236+0000) 2024-03-06T16:50:39.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:39.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:39.291460+0000) 2024-03-06T16:50:40.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:40.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:40.291699+0000) 2024-03-06T16:50:41.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:41.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:41.291905+0000) 2024-03-06T16:50:42.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:42.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:42.292098+0000) 2024-03-06T16:50:43.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:43.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:43.292294+0000) 2024-03-06T16:50:44.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:44.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:44.292523+0000) 2024-03-06T16:50:45.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:45.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:45.292798+0000) 2024-03-06T16:50:46.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:46.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:46.292993+0000) 2024-03-06T16:50:47.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:47.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:47.293189+0000) 2024-03-06T16:50:48.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:48.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:48.293352+0000) 2024-03-06T16:50:49.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:49.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:49.293606+0000) 2024-03-06T16:50:50.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:50.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:50.293827+0000) 2024-03-06T16:50:51.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:51.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:51.294036+0000) 2024-03-06T16:50:52.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:52.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:52.294298+0000) 2024-03-06T16:50:53.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:53.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:53.294579+0000) 2024-03-06T16:50:54.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:54.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:54.294808+0000) 2024-03-06T16:50:55.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:55.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:55.295002+0000) 2024-03-06T16:50:55.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:55.368 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:56.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:56.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:56.295155+0000) 2024-03-06T16:50:57.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:57.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:57.295384+0000) 2024-03-06T16:50:58.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:58.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:58.295567+0000) 2024-03-06T16:50:59.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:50:59.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:59.295809+0000) 2024-03-06T16:51:00.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:00.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:00.296002+0000) 2024-03-06T16:51:01.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:01.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:01.296276+0000) 2024-03-06T16:51:02.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:02.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:02.296554+0000) 2024-03-06T16:51:03.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:03.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:03.296810+0000) 2024-03-06T16:51:04.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:04.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:04.297074+0000) 2024-03-06T16:51:05.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:05.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:05.297348+0000) 2024-03-06T16:51:06.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:06.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:06.297613+0000) 2024-03-06T16:51:07.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:07.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:07.297900+0000) 2024-03-06T16:51:08.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:08.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:08.298173+0000) 2024-03-06T16:51:09.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:09.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:09.298455+0000) 2024-03-06T16:51:10.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:10.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:10.298764+0000) 2024-03-06T16:51:11.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:11.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:11.298946+0000) 2024-03-06T16:51:12.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:12.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:12.299146+0000) 2024-03-06T16:51:13.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:13.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:13.299316+0000) 2024-03-06T16:51:14.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:14.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:14.299553+0000) 2024-03-06T16:51:15.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:15.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:15.299797+0000) 2024-03-06T16:51:16.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:16.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:16.299986+0000) 2024-03-06T16:51:17.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:17.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:17.300170+0000) 2024-03-06T16:51:18.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:18.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:18.300394+0000) 2024-03-06T16:51:19.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:19.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:19.300640+0000) 2024-03-06T16:51:20.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:20.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:20.300855+0000) 2024-03-06T16:51:21.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:21.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:21.301060+0000) 2024-03-06T16:51:22.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:22.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:22.301293+0000) 2024-03-06T16:51:23.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:23.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:23.301525+0000) 2024-03-06T16:51:24.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:24.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:24.301806+0000) 2024-03-06T16:51:25.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:25.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:25.301993+0000) 2024-03-06T16:51:25.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:25.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:26.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:26.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:26.302283+0000) 2024-03-06T16:51:27.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:27.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:27.302487+0000) 2024-03-06T16:51:28.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:28.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:28.302752+0000) 2024-03-06T16:51:29.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:29.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:29.302970+0000) 2024-03-06T16:51:30.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:30.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:30.303170+0000) 2024-03-06T16:51:31.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:31.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:31.303384+0000) 2024-03-06T16:51:32.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:32.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:32.303576+0000) 2024-03-06T16:51:33.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:33.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:33.303819+0000) 2024-03-06T16:51:34.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:34.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:34.304014+0000) 2024-03-06T16:51:35.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:35.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:35.304207+0000) 2024-03-06T16:51:36.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:36.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:36.304437+0000) 2024-03-06T16:51:37.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:37.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:37.304690+0000) 2024-03-06T16:51:38.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:38.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:38.304974+0000) 2024-03-06T16:51:39.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:39.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:39.305167+0000) 2024-03-06T16:51:40.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:40.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:40.305391+0000) 2024-03-06T16:51:41.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:41.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:41.305620+0000) 2024-03-06T16:51:42.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:42.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:42.305838+0000) 2024-03-06T16:51:43.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:43.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:43.306029+0000) 2024-03-06T16:51:44.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:44.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:44.306217+0000) 2024-03-06T16:51:45.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:45.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:45.306446+0000) 2024-03-06T16:51:46.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:46.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:46.306631+0000) 2024-03-06T16:51:47.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:47.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:47.306859+0000) 2024-03-06T16:51:48.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:48.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:48.307049+0000) 2024-03-06T16:51:49.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:49.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:49.307236+0000) 2024-03-06T16:51:50.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:50.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:50.307428+0000) 2024-03-06T16:51:51.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:51.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:51.307648+0000) 2024-03-06T16:51:52.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:52.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:52.307844+0000) 2024-03-06T16:51:53.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:53.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:53.308066+0000) 2024-03-06T16:51:54.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:54.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:54.308299+0000) 2024-03-06T16:51:55.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:55.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:55.308526+0000) 2024-03-06T16:51:55.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:55.772 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:56.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:56.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:56.308774+0000) 2024-03-06T16:51:57.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:57.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:57.308998+0000) 2024-03-06T16:51:58.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:58.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:58.309166+0000) 2024-03-06T16:51:59.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:51:59.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:59.309319+0000) 2024-03-06T16:52:00.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:00.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:00.309539+0000) 2024-03-06T16:52:01.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:01.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:01.309806+0000) 2024-03-06T16:52:02.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:02.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:02.310015+0000) 2024-03-06T16:52:03.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:03.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:03.310202+0000) 2024-03-06T16:52:04.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:04.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:04.310413+0000) 2024-03-06T16:52:05.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:05.309+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:05.310649+0000) 2024-03-06T16:52:06.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:06.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:06.310906+0000) 2024-03-06T16:52:07.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:07.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:07.311144+0000) 2024-03-06T16:52:08.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:08.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:08.311337+0000) 2024-03-06T16:52:09.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:09.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:09.311568+0000) 2024-03-06T16:52:10.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:10.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:10.311802+0000) 2024-03-06T16:52:11.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:11.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:11.312048+0000) 2024-03-06T16:52:12.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:12.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:12.312274+0000) 2024-03-06T16:52:13.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:13.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:13.312511+0000) 2024-03-06T16:52:14.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:14.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:14.312789+0000) 2024-03-06T16:52:15.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:15.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:15.313037+0000) 2024-03-06T16:52:16.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:16.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:16.313275+0000) 2024-03-06T16:52:17.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:17.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:17.313513+0000) 2024-03-06T16:52:18.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:18.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:18.313712+0000) 2024-03-06T16:52:19.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:19.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:19.313954+0000) 2024-03-06T16:52:20.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:20.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:20.314183+0000) 2024-03-06T16:52:21.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:21.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:21.314374+0000) 2024-03-06T16:52:22.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:22.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:22.314568+0000) 2024-03-06T16:52:23.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:23.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:23.314814+0000) 2024-03-06T16:52:24.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:24.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:24.315003+0000) 2024-03-06T16:52:25.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:25.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:25.315251+0000) 2024-03-06T16:52:25.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:25.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:26.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:26.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:26.315576+0000) 2024-03-06T16:52:27.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:27.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:27.315858+0000) 2024-03-06T16:52:28.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:28.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:28.316060+0000) 2024-03-06T16:52:29.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:29.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:29.316292+0000) 2024-03-06T16:52:30.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:30.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:30.316517+0000) 2024-03-06T16:52:31.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:31.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:31.316780+0000) 2024-03-06T16:52:32.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:32.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:32.316925+0000) 2024-03-06T16:52:33.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:33.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:33.317128+0000) 2024-03-06T16:52:34.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:34.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:34.317375+0000) 2024-03-06T16:52:35.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:35.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:35.317583+0000) 2024-03-06T16:52:36.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:36.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:36.317856+0000) 2024-03-06T16:52:37.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:37.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:37.318101+0000) 2024-03-06T16:52:38.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:38.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:38.318381+0000) 2024-03-06T16:52:39.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:39.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:39.318603+0000) 2024-03-06T16:52:40.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:40.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:40.318814+0000) 2024-03-06T16:52:41.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:41.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:41.319016+0000) 2024-03-06T16:52:42.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:42.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:42.319246+0000) 2024-03-06T16:52:43.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:43.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:43.319433+0000) 2024-03-06T16:52:44.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:44.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:44.319629+0000) 2024-03-06T16:52:45.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:45.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:45.319925+0000) 2024-03-06T16:52:46.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:46.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:46.320164+0000) 2024-03-06T16:52:47.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:47.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:47.320404+0000) 2024-03-06T16:52:48.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:48.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:48.320646+0000) 2024-03-06T16:52:49.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:49.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:49.320906+0000) 2024-03-06T16:52:50.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:50.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:50.321122+0000) 2024-03-06T16:52:51.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:51.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:51.321370+0000) 2024-03-06T16:52:52.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:52.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:52.321563+0000) 2024-03-06T16:52:53.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:53.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:53.321816+0000) 2024-03-06T16:52:54.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:54.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:54.321995+0000) 2024-03-06T16:52:55.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:55.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:55.322161+0000) 2024-03-06T16:52:56.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:56.212 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:56.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:56.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:56.322316+0000) 2024-03-06T16:52:57.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:57.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:57.322555+0000) 2024-03-06T16:52:58.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:58.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:58.322794+0000) 2024-03-06T16:52:59.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:52:59.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:59.323013+0000) 2024-03-06T16:53:00.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:00.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:00.323203+0000) 2024-03-06T16:53:01.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:01.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:01.323444+0000) 2024-03-06T16:53:02.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:02.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:02.323704+0000) 2024-03-06T16:53:03.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:03.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:03.323903+0000) 2024-03-06T16:53:04.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:04.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:04.324101+0000) 2024-03-06T16:53:05.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:05.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:05.324327+0000) 2024-03-06T16:53:06.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:06.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:06.324550+0000) 2024-03-06T16:53:07.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:07.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:07.324797+0000) 2024-03-06T16:53:08.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:08.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:08.324982+0000) 2024-03-06T16:53:09.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:09.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:09.325177+0000) 2024-03-06T16:53:10.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:10.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:10.325368+0000) 2024-03-06T16:53:11.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:11.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:11.325566+0000) 2024-03-06T16:53:12.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:12.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:12.325805+0000) 2024-03-06T16:53:13.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:13.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:13.325994+0000) 2024-03-06T16:53:14.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:14.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:14.326234+0000) 2024-03-06T16:53:15.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:15.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:15.326463+0000) 2024-03-06T16:53:16.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:16.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:16.326729+0000) 2024-03-06T16:53:17.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:17.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:17.326915+0000) 2024-03-06T16:53:18.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:18.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:18.327106+0000) 2024-03-06T16:53:19.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:19.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:19.327299+0000) 2024-03-06T16:53:20.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:20.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:20.327538+0000) 2024-03-06T16:53:21.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:21.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:21.327788+0000) 2024-03-06T16:53:22.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:22.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:22.328002+0000) 2024-03-06T16:53:23.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:23.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:23.328217+0000) 2024-03-06T16:53:24.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:24.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:24.328433+0000) 2024-03-06T16:53:25.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:25.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:25.328692+0000) 2024-03-06T16:53:26.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:26.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:26.328889+0000) 2024-03-06T16:53:26.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:26.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:27.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:27.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:27.329041+0000) 2024-03-06T16:53:28.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:28.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:28.329265+0000) 2024-03-06T16:53:29.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:29.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:29.329496+0000) 2024-03-06T16:53:30.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:30.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:30.329748+0000) 2024-03-06T16:53:31.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:31.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:31.330003+0000) 2024-03-06T16:53:32.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:32.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:32.330234+0000) 2024-03-06T16:53:33.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:33.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:33.330476+0000) 2024-03-06T16:53:34.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:34.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:34.330756+0000) 2024-03-06T16:53:35.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:35.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:35.331025+0000) 2024-03-06T16:53:36.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:36.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:36.331262+0000) 2024-03-06T16:53:37.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:37.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:37.331543+0000) 2024-03-06T16:53:38.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:38.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:38.331776+0000) 2024-03-06T16:53:39.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:39.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:39.332003+0000) 2024-03-06T16:53:40.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:40.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:40.332198+0000) 2024-03-06T16:53:41.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:41.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:41.332479+0000) 2024-03-06T16:53:42.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:42.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:42.332760+0000) 2024-03-06T16:53:43.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:43.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:43.332962+0000) 2024-03-06T16:53:44.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:44.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:44.333199+0000) 2024-03-06T16:53:45.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:45.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:45.333429+0000) 2024-03-06T16:53:46.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:46.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:46.333627+0000) 2024-03-06T16:53:47.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:47.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:47.333905+0000) 2024-03-06T16:53:48.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:48.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:48.334098+0000) 2024-03-06T16:53:49.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:49.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:49.334302+0000) 2024-03-06T16:53:50.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:50.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:50.334478+0000) 2024-03-06T16:53:51.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:51.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:51.334761+0000) 2024-03-06T16:53:52.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:52.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:52.335010+0000) 2024-03-06T16:53:53.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:53.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:53.335203+0000) 2024-03-06T16:53:54.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:54.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:54.335452+0000) 2024-03-06T16:53:55.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:55.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:55.335729+0000) 2024-03-06T16:53:56.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:56.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:56.335946+0000) 2024-03-06T16:53:56.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:56.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:57.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:57.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:57.336161+0000) 2024-03-06T16:53:58.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:58.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:58.336406+0000) 2024-03-06T16:53:59.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:53:59.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:59.336654+0000) 2024-03-06T16:54:00.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:00.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:00.336924+0000) 2024-03-06T16:54:01.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:01.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:01.337145+0000) 2024-03-06T16:54:02.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:02.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:02.337387+0000) 2024-03-06T16:54:03.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:03.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:03.337585+0000) 2024-03-06T16:54:04.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:04.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:04.337836+0000) 2024-03-06T16:54:05.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:05.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:05.338033+0000) 2024-03-06T16:54:06.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:06.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:06.338225+0000) 2024-03-06T16:54:07.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:07.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:07.338514+0000) 2024-03-06T16:54:08.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:08.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:08.338767+0000) 2024-03-06T16:54:09.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:09.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:09.338963+0000) 2024-03-06T16:54:10.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:10.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:10.339174+0000) 2024-03-06T16:54:11.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:11.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:11.339425+0000) 2024-03-06T16:54:12.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:12.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:12.339701+0000) 2024-03-06T16:54:13.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:13.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:13.339902+0000) 2024-03-06T16:54:14.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:14.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:14.340155+0000) 2024-03-06T16:54:15.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:15.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:15.340404+0000) 2024-03-06T16:54:16.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:16.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:16.340635+0000) 2024-03-06T16:54:17.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:17.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:17.340875+0000) 2024-03-06T16:54:18.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:18.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:18.341066+0000) 2024-03-06T16:54:19.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:19.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:19.341262+0000) 2024-03-06T16:54:20.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:20.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:20.341508+0000) 2024-03-06T16:54:21.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:21.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:21.341780+0000) 2024-03-06T16:54:22.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:22.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:22.341980+0000) 2024-03-06T16:54:23.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:23.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:23.342182+0000) 2024-03-06T16:54:24.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:24.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:24.342430+0000) 2024-03-06T16:54:25.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:25.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:25.342706+0000) 2024-03-06T16:54:26.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:26.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:26.342921+0000) 2024-03-06T16:54:26.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:26.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:27.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:27.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:27.343113+0000) 2024-03-06T16:54:28.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:28.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:28.343303+0000) 2024-03-06T16:54:29.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:29.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:29.343544+0000) 2024-03-06T16:54:30.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:30.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:30.343791+0000) 2024-03-06T16:54:31.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:31.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:31.344016+0000) 2024-03-06T16:54:32.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:32.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:32.344222+0000) 2024-03-06T16:54:33.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:33.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:33.344476+0000) 2024-03-06T16:54:34.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:34.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:34.344746+0000) 2024-03-06T16:54:35.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:35.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:35.344936+0000) 2024-03-06T16:54:36.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:36.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:36.345123+0000) 2024-03-06T16:54:37.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:37.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:37.345338+0000) 2024-03-06T16:54:38.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:38.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:38.345581+0000) 2024-03-06T16:54:39.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:39.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:39.345799+0000) 2024-03-06T16:54:40.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:40.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:40.346020+0000) 2024-03-06T16:54:41.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:41.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:41.346262+0000) 2024-03-06T16:54:42.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:42.344+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:42.346453+0000) 2024-03-06T16:54:43.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:43.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:43.346746+0000) 2024-03-06T16:54:44.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:44.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:44.346991+0000) 2024-03-06T16:54:45.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:45.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:45.347233+0000) 2024-03-06T16:54:46.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:46.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:46.347470+0000) 2024-03-06T16:54:47.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:47.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:47.347742+0000) 2024-03-06T16:54:48.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:48.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:48.348003+0000) 2024-03-06T16:54:49.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:49.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:49.348210+0000) 2024-03-06T16:54:50.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:50.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:50.348461+0000) 2024-03-06T16:54:51.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:51.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:51.348731+0000) 2024-03-06T16:54:52.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:52.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:52.348933+0000) 2024-03-06T16:54:53.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:53.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:53.349123+0000) 2024-03-06T16:54:54.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:54.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:54.349353+0000) 2024-03-06T16:54:55.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:55.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:55.349644+0000) 2024-03-06T16:54:56.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:56.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:56.349872+0000) 2024-03-06T16:54:56.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:56.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:57.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:57.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:57.350078+0000) 2024-03-06T16:54:58.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:58.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:58.350274+0000) 2024-03-06T16:54:59.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:54:59.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:59.350517+0000) 2024-03-06T16:55:00.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:00.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:00.350776+0000) 2024-03-06T16:55:01.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:01.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:01.350972+0000) 2024-03-06T16:55:02.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:02.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:02.351196+0000) 2024-03-06T16:55:03.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:03.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:03.351420+0000) 2024-03-06T16:55:04.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:04.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:04.351655+0000) 2024-03-06T16:55:05.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:05.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:05.351911+0000) 2024-03-06T16:55:06.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:06.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:06.352103+0000) 2024-03-06T16:55:07.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:07.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:07.352261+0000) 2024-03-06T16:55:08.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:08.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:08.352489+0000) 2024-03-06T16:55:09.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:09.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:09.352765+0000) 2024-03-06T16:55:10.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:10.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:10.352967+0000) 2024-03-06T16:55:11.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:11.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:11.353208+0000) 2024-03-06T16:55:12.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:12.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:12.353436+0000) 2024-03-06T16:55:13.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:13.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:13.353659+0000) 2024-03-06T16:55:14.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:14.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:14.353890+0000) 2024-03-06T16:55:15.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:15.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:15.354083+0000) 2024-03-06T16:55:16.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:16.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:16.354313+0000) 2024-03-06T16:55:17.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:17.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:17.354528+0000) 2024-03-06T16:55:18.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:18.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:18.354808+0000) 2024-03-06T16:55:19.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:19.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:19.355030+0000) 2024-03-06T16:55:20.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:20.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:20.355229+0000) 2024-03-06T16:55:21.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:21.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:21.355437+0000) 2024-03-06T16:55:22.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:22.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:22.355634+0000) 2024-03-06T16:55:23.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:23.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:23.355928+0000) 2024-03-06T16:55:24.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:24.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:24.356181+0000) 2024-03-06T16:55:25.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:25.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:25.356393+0000) 2024-03-06T16:55:26.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:26.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:26.356640+0000) 2024-03-06T16:55:27.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:27.018 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:27.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:27.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:27.356941+0000) 2024-03-06T16:55:28.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:28.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:28.357186+0000) 2024-03-06T16:55:29.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:29.355+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:29.357468+0000) 2024-03-06T16:55:30.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:30.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:30.357748+0000) 2024-03-06T16:55:31.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:31.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:31.357949+0000) 2024-03-06T16:55:32.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:32.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:32.358138+0000) 2024-03-06T16:55:33.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:33.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:33.358357+0000) 2024-03-06T16:55:34.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:34.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:34.358640+0000) 2024-03-06T16:55:35.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:35.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:35.358902+0000) 2024-03-06T16:55:36.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:36.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:36.359142+0000) 2024-03-06T16:55:37.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:37.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:37.359371+0000) 2024-03-06T16:55:38.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:38.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:38.359648+0000) 2024-03-06T16:55:39.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:39.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:39.359976+0000) 2024-03-06T16:55:40.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:40.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:40.360196+0000) 2024-03-06T16:55:41.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:41.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:41.360485+0000) 2024-03-06T16:55:42.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:42.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:42.360798+0000) 2024-03-06T16:55:43.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:43.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:43.361036+0000) 2024-03-06T16:55:44.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:44.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:44.361287+0000) 2024-03-06T16:55:45.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:45.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:45.361528+0000) 2024-03-06T16:55:46.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:46.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:46.361650+0000) 2024-03-06T16:55:47.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:47.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:47.361715+0000) 2024-03-06T16:55:48.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:48.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:48.361985+0000) 2024-03-06T16:55:49.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:49.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:49.362203+0000) 2024-03-06T16:55:50.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:50.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:50.362427+0000) 2024-03-06T16:55:51.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:51.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:51.362691+0000) 2024-03-06T16:55:52.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:52.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:52.362884+0000) 2024-03-06T16:55:53.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:53.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:53.363072+0000) 2024-03-06T16:55:54.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:54.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:54.363300+0000) 2024-03-06T16:55:55.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:55.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:55.363491+0000) 2024-03-06T16:55:56.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:56.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:56.363792+0000) 2024-03-06T16:55:57.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:57.153 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:57.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:57.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:57.364050+0000) 2024-03-06T16:55:58.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:58.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:58.364250+0000) 2024-03-06T16:55:59.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:55:59.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:59.364480+0000) 2024-03-06T16:56:00.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:00.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:00.364746+0000) 2024-03-06T16:56:01.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:01.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:01.365019+0000) 2024-03-06T16:56:02.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:02.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:02.365265+0000) 2024-03-06T16:56:03.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:03.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:03.365510+0000) 2024-03-06T16:56:04.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:04.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:04.365746+0000) 2024-03-06T16:56:05.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:05.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:05.365949+0000) 2024-03-06T16:56:06.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:06.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:06.366151+0000) 2024-03-06T16:56:07.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:07.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:07.366347+0000) 2024-03-06T16:56:08.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:08.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:08.366624+0000) 2024-03-06T16:56:09.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:09.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:09.366933+0000) 2024-03-06T16:56:10.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:10.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:10.367211+0000) 2024-03-06T16:56:11.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:11.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:11.367516+0000) 2024-03-06T16:56:12.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:12.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:12.367818+0000) 2024-03-06T16:56:13.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:13.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:13.368044+0000) 2024-03-06T16:56:14.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:14.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:14.368246+0000) 2024-03-06T16:56:15.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:15.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:15.368445+0000) 2024-03-06T16:56:16.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:16.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:16.368645+0000) 2024-03-06T16:56:17.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:17.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:17.368840+0000) 2024-03-06T16:56:18.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:18.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:18.369073+0000) 2024-03-06T16:56:19.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:19.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:19.369353+0000) 2024-03-06T16:56:20.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:20.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:20.369625+0000) 2024-03-06T16:56:21.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:21.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:21.369899+0000) 2024-03-06T16:56:22.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:22.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:22.370142+0000) 2024-03-06T16:56:23.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:23.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:23.370368+0000) 2024-03-06T16:56:24.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:24.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:24.370603+0000) 2024-03-06T16:56:25.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:25.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:25.370807+0000) 2024-03-06T16:56:26.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:26.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:26.371014+0000) 2024-03-06T16:56:27.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:27.367 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:27.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:27.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:27.371219+0000) 2024-03-06T16:56:28.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:28.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:28.371458+0000) 2024-03-06T16:56:29.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:29.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:29.371709+0000) 2024-03-06T16:56:30.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:30.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:30.371964+0000) 2024-03-06T16:56:31.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:31.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:31.372223+0000) 2024-03-06T16:56:32.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:32.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:32.372504+0000) 2024-03-06T16:56:33.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:33.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:33.372812+0000) 2024-03-06T16:56:34.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:34.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:34.373061+0000) 2024-03-06T16:56:35.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:35.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:35.373256+0000) 2024-03-06T16:56:36.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:36.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:36.373527+0000) 2024-03-06T16:56:37.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:37.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:37.373798+0000) 2024-03-06T16:56:38.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:38.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:38.374016+0000) 2024-03-06T16:56:39.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:39.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:39.374243+0000) 2024-03-06T16:56:40.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:40.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:40.374468+0000) 2024-03-06T16:56:41.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:41.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:41.374741+0000) 2024-03-06T16:56:42.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:42.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:42.374983+0000) 2024-03-06T16:56:43.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:43.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:43.375213+0000) 2024-03-06T16:56:44.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:44.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:44.375448+0000) 2024-03-06T16:56:45.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:45.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:45.375771+0000) 2024-03-06T16:56:46.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:46.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:46.376020+0000) 2024-03-06T16:56:47.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:47.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:47.376271+0000) 2024-03-06T16:56:48.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:48.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:48.376551+0000) 2024-03-06T16:56:49.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:49.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:49.376854+0000) 2024-03-06T16:56:50.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:50.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:50.377107+0000) 2024-03-06T16:56:51.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:51.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:51.377402+0000) 2024-03-06T16:56:52.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:52.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:52.377720+0000) 2024-03-06T16:56:53.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:53.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:53.378008+0000) 2024-03-06T16:56:54.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:54.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:54.378221+0000) 2024-03-06T16:56:55.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:55.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:55.378455+0000) 2024-03-06T16:56:56.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:56.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:56.378699+0000) 2024-03-06T16:56:57.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:57.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:57.378903+0000) 2024-03-06T16:56:57.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:57.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:58.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:58.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:58.379101+0000) 2024-03-06T16:56:59.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:56:59.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:59.379290+0000) 2024-03-06T16:57:00.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:00.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:00.379504+0000) 2024-03-06T16:57:01.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:01.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:01.379779+0000) 2024-03-06T16:57:02.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:02.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:02.380028+0000) 2024-03-06T16:57:03.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:03.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:03.380268+0000) 2024-03-06T16:57:04.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:04.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:04.380557+0000) 2024-03-06T16:57:05.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:05.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:05.380850+0000) 2024-03-06T16:57:06.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:06.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:06.381060+0000) 2024-03-06T16:57:07.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:07.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:07.381271+0000) 2024-03-06T16:57:08.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:08.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:08.381505+0000) 2024-03-06T16:57:09.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:09.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:09.381781+0000) 2024-03-06T16:57:10.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:10.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:10.381972+0000) 2024-03-06T16:57:11.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:11.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:11.382180+0000) 2024-03-06T16:57:12.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:12.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:12.382402+0000) 2024-03-06T16:57:13.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:13.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:13.382700+0000) 2024-03-06T16:57:14.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:14.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:14.382956+0000) 2024-03-06T16:57:15.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:15.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:15.383205+0000) 2024-03-06T16:57:16.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:16.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:16.383472+0000) 2024-03-06T16:57:17.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:17.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:17.383775+0000) 2024-03-06T16:57:18.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:18.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:18.384059+0000) 2024-03-06T16:57:19.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:19.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:19.384265+0000) 2024-03-06T16:57:20.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:20.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:20.384459+0000) 2024-03-06T16:57:21.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:21.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:21.384657+0000) 2024-03-06T16:57:22.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:22.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:22.384948+0000) 2024-03-06T16:57:23.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:23.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:23.385201+0000) 2024-03-06T16:57:24.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:24.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:24.385451+0000) 2024-03-06T16:57:25.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:25.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:25.385707+0000) 2024-03-06T16:57:26.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:26.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:26.385953+0000) 2024-03-06T16:57:27.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:27.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:27.386159+0000) 2024-03-06T16:57:27.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:27.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:28.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:28.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:28.386391+0000) 2024-03-06T16:57:29.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:29.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:29.386626+0000) 2024-03-06T16:57:30.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:30.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:30.386889+0000) 2024-03-06T16:57:31.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:31.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:31.387159+0000) 2024-03-06T16:57:32.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:32.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:32.387414+0000) 2024-03-06T16:57:33.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:33.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:33.387646+0000) 2024-03-06T16:57:34.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:34.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:34.387930+0000) 2024-03-06T16:57:35.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:35.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:35.388184+0000) 2024-03-06T16:57:36.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:36.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:36.388408+0000) 2024-03-06T16:57:37.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:37.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:37.388720+0000) 2024-03-06T16:57:38.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:38.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:38.388976+0000) 2024-03-06T16:57:39.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:39.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:39.389231+0000) 2024-03-06T16:57:40.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:40.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:40.389451+0000) 2024-03-06T16:57:41.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:41.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:41.389651+0000) 2024-03-06T16:57:42.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:42.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:42.389891+0000) 2024-03-06T16:57:43.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:43.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:43.390116+0000) 2024-03-06T16:57:44.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:44.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:44.390340+0000) 2024-03-06T16:57:45.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:45.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:45.390575+0000) 2024-03-06T16:57:46.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:46.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:46.390811+0000) 2024-03-06T16:57:47.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:47.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:47.391047+0000) 2024-03-06T16:57:48.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:48.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:48.391283+0000) 2024-03-06T16:57:49.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:49.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:49.391514+0000) 2024-03-06T16:57:50.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:50.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:50.391830+0000) 2024-03-06T16:57:51.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:51.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:51.392113+0000) 2024-03-06T16:57:52.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:52.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:52.392388+0000) 2024-03-06T16:57:53.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:53.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:53.392704+0000) 2024-03-06T16:57:54.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:54.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:54.392953+0000) 2024-03-06T16:57:55.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:55.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:55.393206+0000) 2024-03-06T16:57:56.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:56.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:56.393487+0000) 2024-03-06T16:57:57.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:57.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:57.393810+0000) 2024-03-06T16:57:57.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:57.989 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:58.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:58.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:58.394028+0000) 2024-03-06T16:57:59.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:57:59.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:59.394272+0000) 2024-03-06T16:58:00.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:00.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:00.394559+0000) 2024-03-06T16:58:01.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:01.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:01.394878+0000) 2024-03-06T16:58:02.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:02.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:02.395116+0000) 2024-03-06T16:58:03.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:03.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:03.395323+0000) 2024-03-06T16:58:04.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:04.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:04.395568+0000) 2024-03-06T16:58:05.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:05.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:05.395827+0000) 2024-03-06T16:58:06.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:06.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:06.396075+0000) 2024-03-06T16:58:07.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:07.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:07.396329+0000) 2024-03-06T16:58:08.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:08.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:08.396568+0000) 2024-03-06T16:58:09.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:09.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:09.396846+0000) 2024-03-06T16:58:10.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:10.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:10.397140+0000) 2024-03-06T16:58:11.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:11.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:11.397404+0000) 2024-03-06T16:58:12.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:12.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:12.397600+0000) 2024-03-06T16:58:13.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:13.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:13.397826+0000) 2024-03-06T16:58:14.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:14.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:14.398064+0000) 2024-03-06T16:58:15.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:15.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:15.398287+0000) 2024-03-06T16:58:16.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:16.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:16.398523+0000) 2024-03-06T16:58:17.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:17.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:17.398803+0000) 2024-03-06T16:58:18.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:18.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:18.399051+0000) 2024-03-06T16:58:19.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:19.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:19.399284+0000) 2024-03-06T16:58:20.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:20.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:20.399517+0000) 2024-03-06T16:58:21.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:21.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:21.399809+0000) 2024-03-06T16:58:22.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:22.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:22.400012+0000) 2024-03-06T16:58:23.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:23.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:23.400211+0000) 2024-03-06T16:58:24.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:24.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:24.400438+0000) 2024-03-06T16:58:25.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:25.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:25.400693+0000) 2024-03-06T16:58:26.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:26.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:26.400898+0000) 2024-03-06T16:58:27.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:27.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:27.401093+0000) 2024-03-06T16:58:28.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:28.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:28.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:28.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:28.401297+0000) 2024-03-06T16:58:29.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:29.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:29.401526+0000) 2024-03-06T16:58:30.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:30.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:30.401789+0000) 2024-03-06T16:58:31.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:31.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:31.401990+0000) 2024-03-06T16:58:32.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:32.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:32.402184+0000) 2024-03-06T16:58:33.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:33.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:33.402419+0000) 2024-03-06T16:58:34.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:34.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:34.402649+0000) 2024-03-06T16:58:35.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:35.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:35.402871+0000) 2024-03-06T16:58:36.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:36.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:36.403110+0000) 2024-03-06T16:58:37.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:37.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:37.403348+0000) 2024-03-06T16:58:38.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:38.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:38.403534+0000) 2024-03-06T16:58:39.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:39.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:39.403777+0000) 2024-03-06T16:58:40.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:40.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:40.403981+0000) 2024-03-06T16:58:41.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:41.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:41.404176+0000) 2024-03-06T16:58:42.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:42.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:42.404352+0000) 2024-03-06T16:58:43.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:43.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:43.404586+0000) 2024-03-06T16:58:44.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:44.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:44.404832+0000) 2024-03-06T16:58:45.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:45.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:45.405055+0000) 2024-03-06T16:58:46.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:46.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:46.405263+0000) 2024-03-06T16:58:47.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:47.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:47.405487+0000) 2024-03-06T16:58:48.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:48.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:48.405744+0000) 2024-03-06T16:58:49.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:49.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:49.405990+0000) 2024-03-06T16:58:50.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:50.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:50.406201+0000) 2024-03-06T16:58:51.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:51.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:51.406486+0000) 2024-03-06T16:58:52.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:52.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:52.406709+0000) 2024-03-06T16:58:53.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:53.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:53.406965+0000) 2024-03-06T16:58:54.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:54.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:54.407176+0000) 2024-03-06T16:58:55.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:55.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:55.407392+0000) 2024-03-06T16:58:56.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:56.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:56.407619+0000) 2024-03-06T16:58:57.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:57.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:57.407827+0000) 2024-03-06T16:58:58.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:58.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:58.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:58.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:58.408015+0000) 2024-03-06T16:58:59.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:58:59.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:59.408220+0000) 2024-03-06T16:59:00.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:00.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:00.408440+0000) 2024-03-06T16:59:01.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:01.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:01.408731+0000) 2024-03-06T16:59:02.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:02.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:02.408966+0000) 2024-03-06T16:59:03.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:03.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:03.409239+0000) 2024-03-06T16:59:04.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:04.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:04.409440+0000) 2024-03-06T16:59:05.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:05.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:05.409696+0000) 2024-03-06T16:59:06.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:06.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:06.409991+0000) 2024-03-06T16:59:07.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:07.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:07.410208+0000) 2024-03-06T16:59:08.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:08.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:08.410403+0000) 2024-03-06T16:59:09.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:09.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:09.410588+0000) 2024-03-06T16:59:10.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:10.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:10.410785+0000) 2024-03-06T16:59:11.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:11.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:11.411037+0000) 2024-03-06T16:59:12.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:12.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:12.411221+0000) 2024-03-06T16:59:13.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:13.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:13.411458+0000) 2024-03-06T16:59:14.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:14.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:14.411733+0000) 2024-03-06T16:59:15.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:15.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:15.411938+0000) 2024-03-06T16:59:16.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:16.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:16.412112+0000) 2024-03-06T16:59:17.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:17.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:17.412314+0000) 2024-03-06T16:59:18.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:18.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:18.412537+0000) 2024-03-06T16:59:19.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:19.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:19.412792+0000) 2024-03-06T16:59:20.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:20.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:20.413001+0000) 2024-03-06T16:59:21.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:21.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:21.413208+0000) 2024-03-06T16:59:22.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:22.411+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:22.413440+0000) 2024-03-06T16:59:23.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:23.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:23.413691+0000) 2024-03-06T16:59:24.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:24.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:24.413887+0000) 2024-03-06T16:59:25.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:25.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:25.414104+0000) 2024-03-06T16:59:26.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:26.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:26.414335+0000) 2024-03-06T16:59:27.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:27.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:27.414565+0000) 2024-03-06T16:59:28.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:28.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:28.414809+0000) 2024-03-06T16:59:28.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:28.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:29.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:29.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:29.415017+0000) 2024-03-06T16:59:30.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:30.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:30.415220+0000) 2024-03-06T16:59:31.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:31.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:31.415475+0000) 2024-03-06T16:59:32.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:32.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:32.415750+0000) 2024-03-06T16:59:33.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:33.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:33.415944+0000) 2024-03-06T16:59:34.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:34.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:34.416129+0000) 2024-03-06T16:59:35.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:35.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:35.416366+0000) 2024-03-06T16:59:36.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:36.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:36.416603+0000) 2024-03-06T16:59:37.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:37.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:37.416811+0000) 2024-03-06T16:59:38.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:38.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:38.417047+0000) 2024-03-06T16:59:39.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:39.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:39.417280+0000) 2024-03-06T16:59:40.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:40.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:40.417515+0000) 2024-03-06T16:59:41.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:41.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:41.417791+0000) 2024-03-06T16:59:42.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:42.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:42.418048+0000) 2024-03-06T16:59:43.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:43.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:43.418306+0000) 2024-03-06T16:59:44.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:44.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:44.418542+0000) 2024-03-06T16:59:45.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:45.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:45.418774+0000) 2024-03-06T16:59:46.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:46.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:46.418968+0000) 2024-03-06T16:59:47.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:47.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:47.419159+0000) 2024-03-06T16:59:48.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:48.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:48.419388+0000) 2024-03-06T16:59:49.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:49.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:49.419610+0000) 2024-03-06T16:59:50.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:50.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:50.419837+0000) 2024-03-06T16:59:51.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:51.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:51.420048+0000) 2024-03-06T16:59:52.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:52.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:52.420286+0000) 2024-03-06T16:59:53.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:53.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:53.420513+0000) 2024-03-06T16:59:54.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:54.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:54.420727+0000) 2024-03-06T16:59:55.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:55.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:55.420928+0000) 2024-03-06T16:59:56.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:56.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:56.421123+0000) 2024-03-06T16:59:57.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:57.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:57.421342+0000) 2024-03-06T16:59:58.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:58.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:58.421572+0000) 2024-03-06T16:59:58.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:58.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:59.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T16:59:59.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:59.421820+0000) 2024-03-06T17:00:00.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:00.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:00.422022+0000) 2024-03-06T17:00:01.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:01.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:01.422266+0000) 2024-03-06T17:00:02.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:02.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:02.422492+0000) 2024-03-06T17:00:03.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:03.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:03.422698+0000) 2024-03-06T17:00:04.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:04.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:04.422906+0000) 2024-03-06T17:00:05.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:05.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:05.423106+0000) 2024-03-06T17:00:06.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:06.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:06.423335+0000) 2024-03-06T17:00:07.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:07.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:07.423563+0000) 2024-03-06T17:00:08.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:08.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:08.423792+0000) 2024-03-06T17:00:09.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:09.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:09.423979+0000) 2024-03-06T17:00:10.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:10.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:10.424233+0000) 2024-03-06T17:00:11.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:11.422+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:11.424453+0000) 2024-03-06T17:00:12.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:12.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:12.424745+0000) 2024-03-06T17:00:13.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:13.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:13.424933+0000) 2024-03-06T17:00:14.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:14.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:14.425149+0000) 2024-03-06T17:00:15.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:15.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:15.425394+0000) 2024-03-06T17:00:16.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:16.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:16.425584+0000) 2024-03-06T17:00:17.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:17.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:17.425819+0000) 2024-03-06T17:00:18.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:18.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:18.426048+0000) 2024-03-06T17:00:19.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:19.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:19.426293+0000) 2024-03-06T17:00:20.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:20.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:20.426568+0000) 2024-03-06T17:00:21.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:21.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:21.426844+0000) 2024-03-06T17:00:22.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:22.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:22.427072+0000) 2024-03-06T17:00:23.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:23.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:23.427301+0000) 2024-03-06T17:00:24.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:24.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:24.427542+0000) 2024-03-06T17:00:25.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:25.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:25.427784+0000) 2024-03-06T17:00:26.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:26.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:26.428023+0000) 2024-03-06T17:00:27.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:27.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:27.428217+0000) 2024-03-06T17:00:28.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:28.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:28.428449+0000) 2024-03-06T17:00:28.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:28.815 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:29.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:29.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:29.428701+0000) 2024-03-06T17:00:30.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:30.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:30.428935+0000) 2024-03-06T17:00:31.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:31.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:31.429145+0000) 2024-03-06T17:00:32.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:32.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:32.429388+0000) 2024-03-06T17:00:33.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:33.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:33.429595+0000) 2024-03-06T17:00:34.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:34.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:34.429866+0000) 2024-03-06T17:00:35.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:35.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:35.430092+0000) 2024-03-06T17:00:36.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:36.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:36.430323+0000) 2024-03-06T17:00:37.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:37.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:37.430518+0000) 2024-03-06T17:00:38.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:38.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:38.430813+0000) 2024-03-06T17:00:39.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:39.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:39.431049+0000) 2024-03-06T17:00:40.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:40.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:40.431282+0000) 2024-03-06T17:00:41.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:41.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:41.431511+0000) 2024-03-06T17:00:42.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:42.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:42.431786+0000) 2024-03-06T17:00:43.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:43.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:43.431985+0000) 2024-03-06T17:00:44.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:44.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:44.432163+0000) 2024-03-06T17:00:45.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:45.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:45.432425+0000) 2024-03-06T17:00:46.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:46.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:46.432754+0000) 2024-03-06T17:00:47.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:47.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:47.433002+0000) 2024-03-06T17:00:48.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:48.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:48.433191+0000) 2024-03-06T17:00:49.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:49.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:49.433424+0000) 2024-03-06T17:00:50.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:50.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:50.433652+0000) 2024-03-06T17:00:51.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:51.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:51.433891+0000) 2024-03-06T17:00:52.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:52.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:52.434087+0000) 2024-03-06T17:00:53.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:53.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:53.434313+0000) 2024-03-06T17:00:54.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:54.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:54.434541+0000) 2024-03-06T17:00:55.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:55.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:55.434820+0000) 2024-03-06T17:00:56.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:56.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:56.435024+0000) 2024-03-06T17:00:57.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:57.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:57.435218+0000) 2024-03-06T17:00:58.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:58.433+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:58.435467+0000) 2024-03-06T17:00:58.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:58.955 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:59.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:00:59.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:59.435754+0000) 2024-03-06T17:01:00.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:00.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:00.435992+0000) 2024-03-06T17:01:01.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:01.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:01.436147+0000) 2024-03-06T17:01:02.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:02.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:02.436364+0000) 2024-03-06T17:01:03.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:03.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:03.436565+0000) 2024-03-06T17:01:04.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:04.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:04.436805+0000) 2024-03-06T17:01:05.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:05.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:05.436991+0000) 2024-03-06T17:01:06.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:06.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:06.437220+0000) 2024-03-06T17:01:07.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:07.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:07.437484+0000) 2024-03-06T17:01:08.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:08.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:08.437813+0000) 2024-03-06T17:01:09.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:09.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:09.438091+0000) 2024-03-06T17:01:10.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:10.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:10.438341+0000) 2024-03-06T17:01:11.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:11.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:11.438640+0000) 2024-03-06T17:01:12.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:12.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:12.438960+0000) 2024-03-06T17:01:13.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:13.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:13.439162+0000) 2024-03-06T17:01:14.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:14.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:14.439378+0000) 2024-03-06T17:01:15.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:15.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:15.439620+0000) 2024-03-06T17:01:16.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:16.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:16.439826+0000) 2024-03-06T17:01:17.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:17.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:17.440048+0000) 2024-03-06T17:01:18.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:18.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:18.440285+0000) 2024-03-06T17:01:19.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:19.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:19.440508+0000) 2024-03-06T17:01:20.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:20.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:20.440693+0000) 2024-03-06T17:01:21.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:21.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:21.440929+0000) 2024-03-06T17:01:22.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:22.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:22.441159+0000) 2024-03-06T17:01:23.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:23.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:23.441372+0000) 2024-03-06T17:01:24.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:24.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:24.441591+0000) 2024-03-06T17:01:25.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:25.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:25.441781+0000) 2024-03-06T17:01:26.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:26.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:26.442026+0000) 2024-03-06T17:01:27.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:27.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:27.442222+0000) 2024-03-06T17:01:28.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:28.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:28.442411+0000) 2024-03-06T17:01:29.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:29.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:29.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:29.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:29.442661+0000) 2024-03-06T17:01:30.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:30.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:30.442924+0000) 2024-03-06T17:01:31.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:31.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:31.443141+0000) 2024-03-06T17:01:32.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:32.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:32.443374+0000) 2024-03-06T17:01:33.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:33.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:33.443600+0000) 2024-03-06T17:01:34.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:34.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:34.443820+0000) 2024-03-06T17:01:35.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:35.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:35.444091+0000) 2024-03-06T17:01:36.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:36.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:36.444336+0000) 2024-03-06T17:01:37.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:37.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:37.444520+0000) 2024-03-06T17:01:38.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:38.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:38.444795+0000) 2024-03-06T17:01:39.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:39.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:39.444988+0000) 2024-03-06T17:01:40.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:40.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:40.445211+0000) 2024-03-06T17:01:41.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:41.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:41.445423+0000) 2024-03-06T17:01:42.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:42.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:42.445638+0000) 2024-03-06T17:01:43.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:43.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:43.445926+0000) 2024-03-06T17:01:44.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:44.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:44.446153+0000) 2024-03-06T17:01:45.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:45.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:45.446431+0000) 2024-03-06T17:01:46.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:46.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:46.446755+0000) 2024-03-06T17:01:47.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:47.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:47.447002+0000) 2024-03-06T17:01:48.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:48.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:48.447230+0000) 2024-03-06T17:01:49.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:49.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:49.447424+0000) 2024-03-06T17:01:50.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:50.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:50.447609+0000) 2024-03-06T17:01:51.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:51.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:51.447797+0000) 2024-03-06T17:01:52.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:52.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:52.447989+0000) 2024-03-06T17:01:53.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:53.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:53.448190+0000) 2024-03-06T17:01:54.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:54.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:54.448410+0000) 2024-03-06T17:01:55.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:55.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:55.448639+0000) 2024-03-06T17:01:56.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:56.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:56.448878+0000) 2024-03-06T17:01:57.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:57.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:57.449108+0000) 2024-03-06T17:01:58.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:58.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:58.449353+0000) 2024-03-06T17:01:59.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:59.317 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:59.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:01:59.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:59.449542+0000) 2024-03-06T17:02:00.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:00.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:00.449847+0000) 2024-03-06T17:02:01.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:01.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:01.450065+0000) 2024-03-06T17:02:02.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:02.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:02.450302+0000) 2024-03-06T17:02:03.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:03.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:03.450532+0000) 2024-03-06T17:02:04.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:04.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:04.450799+0000) 2024-03-06T17:02:05.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:05.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:05.450992+0000) 2024-03-06T17:02:06.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:06.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:06.451221+0000) 2024-03-06T17:02:07.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:07.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:07.451447+0000) 2024-03-06T17:02:08.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:08.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:08.451720+0000) 2024-03-06T17:02:09.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:09.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:09.451947+0000) 2024-03-06T17:02:10.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:10.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:10.452191+0000) 2024-03-06T17:02:11.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:11.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:11.452389+0000) 2024-03-06T17:02:12.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:12.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:12.452604+0000) 2024-03-06T17:02:13.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:13.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:13.452838+0000) 2024-03-06T17:02:14.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:14.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:14.453006+0000) 2024-03-06T17:02:15.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:15.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:15.453233+0000) 2024-03-06T17:02:16.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:16.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:16.453467+0000) 2024-03-06T17:02:17.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:17.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:17.453741+0000) 2024-03-06T17:02:18.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:18.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:18.453943+0000) 2024-03-06T17:02:19.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:19.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:19.454169+0000) 2024-03-06T17:02:20.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:20.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:20.454392+0000) 2024-03-06T17:02:21.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:21.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:21.454626+0000) 2024-03-06T17:02:22.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:22.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:22.454833+0000) 2024-03-06T17:02:23.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:23.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:23.455065+0000) 2024-03-06T17:02:24.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:24.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:24.455315+0000) 2024-03-06T17:02:25.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:25.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:25.455583+0000) 2024-03-06T17:02:26.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:26.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:26.455835+0000) 2024-03-06T17:02:27.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:27.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:27.456032+0000) 2024-03-06T17:02:28.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:28.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:28.456240+0000) 2024-03-06T17:02:29.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:29.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:29.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:29.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:29.456452+0000) 2024-03-06T17:02:30.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:30.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:30.456703+0000) 2024-03-06T17:02:31.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:31.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:31.456960+0000) 2024-03-06T17:02:32.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:32.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:32.457188+0000) 2024-03-06T17:02:33.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:33.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:33.457422+0000) 2024-03-06T17:02:34.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:34.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:34.457703+0000) 2024-03-06T17:02:35.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:35.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:35.457919+0000) 2024-03-06T17:02:36.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:36.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:36.458131+0000) 2024-03-06T17:02:37.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:37.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:37.458356+0000) 2024-03-06T17:02:38.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:38.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:38.458558+0000) 2024-03-06T17:02:39.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:39.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:39.458791+0000) 2024-03-06T17:02:40.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:40.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:40.458994+0000) 2024-03-06T17:02:41.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:41.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:41.459183+0000) 2024-03-06T17:02:42.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:42.457+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:42.459388+0000) 2024-03-06T17:02:43.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:43.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:43.459652+0000) 2024-03-06T17:02:44.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:44.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:44.459915+0000) 2024-03-06T17:02:45.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:45.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:45.460103+0000) 2024-03-06T17:02:46.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:46.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:46.460397+0000) 2024-03-06T17:02:47.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:47.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:47.460594+0000) 2024-03-06T17:02:48.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:48.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:48.460825+0000) 2024-03-06T17:02:49.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:49.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:49.461020+0000) 2024-03-06T17:02:50.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:50.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:50.461204+0000) 2024-03-06T17:02:51.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:51.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:51.461474+0000) 2024-03-06T17:02:52.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:52.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:52.461746+0000) 2024-03-06T17:02:53.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:53.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:53.462028+0000) 2024-03-06T17:02:54.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:54.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:54.462306+0000) 2024-03-06T17:02:55.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:55.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:55.462502+0000) 2024-03-06T17:02:56.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:56.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:56.462769+0000) 2024-03-06T17:02:57.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:57.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:57.462989+0000) 2024-03-06T17:02:58.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:58.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:58.463220+0000) 2024-03-06T17:02:59.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:02:59.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:59.463448+0000) 2024-03-06T17:02:59.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:59.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:00.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:00.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:00.463710+0000) 2024-03-06T17:03:01.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:01.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:01.463866+0000) 2024-03-06T17:03:02.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:02.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:02.464052+0000) 2024-03-06T17:03:03.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:03.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:03.464284+0000) 2024-03-06T17:03:04.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:04.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:04.464502+0000) 2024-03-06T17:03:05.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:05.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:05.464774+0000) 2024-03-06T17:03:06.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:06.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:06.465012+0000) 2024-03-06T17:03:07.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:07.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:07.465213+0000) 2024-03-06T17:03:08.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:08.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:08.465453+0000) 2024-03-06T17:03:09.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:09.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:09.465621+0000) 2024-03-06T17:03:10.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:10.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:10.465832+0000) 2024-03-06T17:03:11.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:11.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:11.465997+0000) 2024-03-06T17:03:12.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:12.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:12.466173+0000) 2024-03-06T17:03:13.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:13.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:13.466422+0000) 2024-03-06T17:03:14.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:14.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:14.466729+0000) 2024-03-06T17:03:15.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:15.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:15.466966+0000) 2024-03-06T17:03:16.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:16.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:16.467296+0000) 2024-03-06T17:03:17.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:17.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:17.467558+0000) 2024-03-06T17:03:18.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:18.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:18.467800+0000) 2024-03-06T17:03:19.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:19.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:19.468008+0000) 2024-03-06T17:03:20.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:20.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:20.468255+0000) 2024-03-06T17:03:21.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:21.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:21.468516+0000) 2024-03-06T17:03:22.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:22.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:22.468816+0000) 2024-03-06T17:03:23.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:23.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:23.469044+0000) 2024-03-06T17:03:24.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:24.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:24.469272+0000) 2024-03-06T17:03:25.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:25.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:25.469452+0000) 2024-03-06T17:03:26.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:26.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:26.469774+0000) 2024-03-06T17:03:27.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:27.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:27.470017+0000) 2024-03-06T17:03:28.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:28.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:28.470286+0000) 2024-03-06T17:03:29.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:29.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:29.470525+0000) 2024-03-06T17:03:29.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:29.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:30.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:30.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:30.470772+0000) 2024-03-06T17:03:31.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:31.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:31.471039+0000) 2024-03-06T17:03:32.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:32.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:32.471273+0000) 2024-03-06T17:03:33.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:33.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:33.471542+0000) 2024-03-06T17:03:34.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:34.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:34.471814+0000) 2024-03-06T17:03:35.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:35.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:35.472058+0000) 2024-03-06T17:03:36.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:36.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:36.472299+0000) 2024-03-06T17:03:37.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:37.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:37.472525+0000) 2024-03-06T17:03:38.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:38.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:38.472808+0000) 2024-03-06T17:03:39.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:39.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:39.473016+0000) 2024-03-06T17:03:40.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:40.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:40.473255+0000) 2024-03-06T17:03:41.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:41.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:41.473516+0000) 2024-03-06T17:03:42.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:42.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:42.473775+0000) 2024-03-06T17:03:43.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:43.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:43.473997+0000) 2024-03-06T17:03:44.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:44.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:44.474247+0000) 2024-03-06T17:03:45.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:45.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:45.474461+0000) 2024-03-06T17:03:46.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:46.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:46.474785+0000) 2024-03-06T17:03:47.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:47.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:47.474992+0000) 2024-03-06T17:03:48.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:48.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:48.475229+0000) 2024-03-06T17:03:49.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:49.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:49.475456+0000) 2024-03-06T17:03:50.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:50.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:50.475615+0000) 2024-03-06T17:03:51.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:51.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:51.475834+0000) 2024-03-06T17:03:52.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:52.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:52.476028+0000) 2024-03-06T17:03:53.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:53.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:53.476261+0000) 2024-03-06T17:03:54.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:54.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:54.476495+0000) 2024-03-06T17:03:55.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:55.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:55.476698+0000) 2024-03-06T17:03:56.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:56.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:56.476943+0000) 2024-03-06T17:03:57.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:57.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:57.477163+0000) 2024-03-06T17:03:58.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:58.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:58.477389+0000) 2024-03-06T17:03:59.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:03:59.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:59.477617+0000) 2024-03-06T17:04:00.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:00.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:00.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:00.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:00.477825+0000) 2024-03-06T17:04:01.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:01.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:01.478045+0000) 2024-03-06T17:04:02.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:02.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:02.478229+0000) 2024-03-06T17:04:03.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:03.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:03.478414+0000) 2024-03-06T17:04:04.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:04.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:04.478733+0000) 2024-03-06T17:04:05.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:05.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:05.478961+0000) 2024-03-06T17:04:06.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:06.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:06.479149+0000) 2024-03-06T17:04:07.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:07.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:07.479382+0000) 2024-03-06T17:04:08.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:08.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:08.479611+0000) 2024-03-06T17:04:09.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:09.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:09.479820+0000) 2024-03-06T17:04:10.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:10.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:10.480015+0000) 2024-03-06T17:04:11.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:11.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:11.480262+0000) 2024-03-06T17:04:12.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:12.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:12.480490+0000) 2024-03-06T17:04:13.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:13.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:13.480773+0000) 2024-03-06T17:04:14.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:14.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:14.481002+0000) 2024-03-06T17:04:15.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:15.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:15.481223+0000) 2024-03-06T17:04:16.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:16.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:16.481461+0000) 2024-03-06T17:04:17.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:17.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:17.481710+0000) 2024-03-06T17:04:18.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:18.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:18.481942+0000) 2024-03-06T17:04:19.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:19.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:19.482170+0000) 2024-03-06T17:04:20.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:20.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:20.482412+0000) 2024-03-06T17:04:21.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:21.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:21.482706+0000) 2024-03-06T17:04:22.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:22.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:22.482913+0000) 2024-03-06T17:04:23.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:23.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:23.483113+0000) 2024-03-06T17:04:24.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:24.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:24.483345+0000) 2024-03-06T17:04:25.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:25.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:25.483573+0000) 2024-03-06T17:04:26.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:26.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:26.483797+0000) 2024-03-06T17:04:27.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:27.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:27.483988+0000) 2024-03-06T17:04:28.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:28.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:28.484183+0000) 2024-03-06T17:04:29.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:29.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:29.484428+0000) 2024-03-06T17:04:30.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:30.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:30.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:30.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:30.484645+0000) 2024-03-06T17:04:31.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:31.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:31.484837+0000) 2024-03-06T17:04:32.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:32.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:32.485062+0000) 2024-03-06T17:04:33.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:33.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:33.485309+0000) 2024-03-06T17:04:34.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:34.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:34.485525+0000) 2024-03-06T17:04:35.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:35.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:35.485726+0000) 2024-03-06T17:04:36.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:36.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:36.485945+0000) 2024-03-06T17:04:37.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:37.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:37.486170+0000) 2024-03-06T17:04:38.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:38.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:38.486374+0000) 2024-03-06T17:04:39.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:39.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:39.486613+0000) 2024-03-06T17:04:40.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:40.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:40.486815+0000) 2024-03-06T17:04:41.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:41.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:41.487020+0000) 2024-03-06T17:04:42.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:42.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:42.487220+0000) 2024-03-06T17:04:43.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:43.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:43.487457+0000) 2024-03-06T17:04:44.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:44.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:44.487731+0000) 2024-03-06T17:04:45.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:45.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:45.487952+0000) 2024-03-06T17:04:46.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:46.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:46.488168+0000) 2024-03-06T17:04:47.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:47.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:47.488363+0000) 2024-03-06T17:04:48.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:48.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:48.488552+0000) 2024-03-06T17:04:49.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:49.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:49.488781+0000) 2024-03-06T17:04:50.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:50.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:50.488980+0000) 2024-03-06T17:04:51.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:51.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:51.489230+0000) 2024-03-06T17:04:52.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:52.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:52.489461+0000) 2024-03-06T17:04:53.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:53.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:53.489740+0000) 2024-03-06T17:04:54.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:54.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:54.489965+0000) 2024-03-06T17:04:55.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:55.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:55.490194+0000) 2024-03-06T17:04:56.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:56.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:56.490386+0000) 2024-03-06T17:04:57.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:57.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:57.490574+0000) 2024-03-06T17:04:58.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:58.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:58.490845+0000) 2024-03-06T17:04:59.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:04:59.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:59.491041+0000) 2024-03-06T17:05:00.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:00.422 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:00.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:00.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:00.491314+0000) 2024-03-06T17:05:01.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:01.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:01.491627+0000) 2024-03-06T17:05:02.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:02.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:02.491953+0000) 2024-03-06T17:05:03.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:03.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:03.492227+0000) 2024-03-06T17:05:04.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:04.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:04.492509+0000) 2024-03-06T17:05:05.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:05.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:05.492818+0000) 2024-03-06T17:05:06.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:06.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:06.493099+0000) 2024-03-06T17:05:07.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:07.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:07.493370+0000) 2024-03-06T17:05:08.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:08.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:08.493586+0000) 2024-03-06T17:05:09.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:09.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:09.493841+0000) 2024-03-06T17:05:10.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:10.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:10.494061+0000) 2024-03-06T17:05:11.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:11.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:11.494326+0000) 2024-03-06T17:05:12.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:12.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:12.494551+0000) 2024-03-06T17:05:13.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:13.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:13.494787+0000) 2024-03-06T17:05:14.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:14.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:14.495012+0000) 2024-03-06T17:05:15.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:15.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:15.495278+0000) 2024-03-06T17:05:16.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:16.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:16.495557+0000) 2024-03-06T17:05:17.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:17.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:17.495799+0000) 2024-03-06T17:05:18.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:18.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:18.496069+0000) 2024-03-06T17:05:19.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:19.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:19.496279+0000) 2024-03-06T17:05:20.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:20.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:20.496479+0000) 2024-03-06T17:05:21.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:21.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:21.496726+0000) 2024-03-06T17:05:22.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:22.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:22.496944+0000) 2024-03-06T17:05:23.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:23.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:23.497196+0000) 2024-03-06T17:05:24.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:24.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:24.497439+0000) 2024-03-06T17:05:25.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:25.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:25.497643+0000) 2024-03-06T17:05:26.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:26.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:26.497907+0000) 2024-03-06T17:05:27.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:27.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:27.498094+0000) 2024-03-06T17:05:28.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:28.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:28.498331+0000) 2024-03-06T17:05:29.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:29.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:29.498523+0000) 2024-03-06T17:05:30.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:30.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:30.498786+0000) 2024-03-06T17:05:30.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:30.633 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:31.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:31.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:31.499033+0000) 2024-03-06T17:05:32.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:32.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:32.499266+0000) 2024-03-06T17:05:33.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:33.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:33.499497+0000) 2024-03-06T17:05:34.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:34.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:34.499775+0000) 2024-03-06T17:05:35.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:35.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:35.500006+0000) 2024-03-06T17:05:36.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:36.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:36.500237+0000) 2024-03-06T17:05:37.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:37.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:37.500427+0000) 2024-03-06T17:05:38.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:38.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:38.500600+0000) 2024-03-06T17:05:39.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:39.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:39.500798+0000) 2024-03-06T17:05:40.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:40.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:40.500991+0000) 2024-03-06T17:05:41.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:41.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:41.501235+0000) 2024-03-06T17:05:42.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:42.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:42.501464+0000) 2024-03-06T17:05:43.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:43.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:43.501704+0000) 2024-03-06T17:05:44.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:44.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:44.501917+0000) 2024-03-06T17:05:45.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:45.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:45.502111+0000) 2024-03-06T17:05:46.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:46.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:46.502405+0000) 2024-03-06T17:05:47.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:47.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:47.502587+0000) 2024-03-06T17:05:48.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:48.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:48.502810+0000) 2024-03-06T17:05:49.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:49.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:49.503038+0000) 2024-03-06T17:05:50.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:50.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:50.503268+0000) 2024-03-06T17:05:51.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:51.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:51.503526+0000) 2024-03-06T17:05:52.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:52.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:52.503768+0000) 2024-03-06T17:05:53.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:53.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:53.503970+0000) 2024-03-06T17:05:54.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:54.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:54.504191+0000) 2024-03-06T17:05:55.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:55.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:55.504410+0000) 2024-03-06T17:05:56.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:56.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:56.504605+0000) 2024-03-06T17:05:57.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:57.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:57.504839+0000) 2024-03-06T17:05:58.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:58.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:58.505070+0000) 2024-03-06T17:05:59.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:05:59.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:59.505305+0000) 2024-03-06T17:06:00.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:00.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:00.505531+0000) 2024-03-06T17:06:00.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:00.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:01.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:01.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:01.505808+0000) 2024-03-06T17:06:02.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:02.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:02.506046+0000) 2024-03-06T17:06:03.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:03.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:03.506279+0000) 2024-03-06T17:06:04.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:04.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:04.506508+0000) 2024-03-06T17:06:05.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:05.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:05.506779+0000) 2024-03-06T17:06:06.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:06.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:06.507015+0000) 2024-03-06T17:06:07.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:07.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:07.507218+0000) 2024-03-06T17:06:08.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:08.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:08.507450+0000) 2024-03-06T17:06:09.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:09.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:09.507631+0000) 2024-03-06T17:06:10.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:10.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:10.507860+0000) 2024-03-06T17:06:11.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:11.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:11.508054+0000) 2024-03-06T17:06:12.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:12.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:12.508288+0000) 2024-03-06T17:06:13.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:13.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:13.508523+0000) 2024-03-06T17:06:14.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:14.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:14.508800+0000) 2024-03-06T17:06:15.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:15.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:15.509021+0000) 2024-03-06T17:06:16.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:16.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:16.509209+0000) 2024-03-06T17:06:17.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:17.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:17.509455+0000) 2024-03-06T17:06:18.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:18.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:18.509659+0000) 2024-03-06T17:06:19.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:19.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:19.509967+0000) 2024-03-06T17:06:20.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:20.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:20.510179+0000) 2024-03-06T17:06:21.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:21.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:21.510448+0000) 2024-03-06T17:06:22.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:22.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:22.510728+0000) 2024-03-06T17:06:23.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:23.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:23.510970+0000) 2024-03-06T17:06:24.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:24.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:24.511215+0000) 2024-03-06T17:06:25.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:25.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:25.511452+0000) 2024-03-06T17:06:26.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:26.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:26.511753+0000) 2024-03-06T17:06:27.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:27.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:27.511958+0000) 2024-03-06T17:06:28.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:28.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:28.512148+0000) 2024-03-06T17:06:29.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:29.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:29.512377+0000) 2024-03-06T17:06:30.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:30.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:30.512602+0000) 2024-03-06T17:06:31.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:31.076 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:31.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:31.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:31.512814+0000) 2024-03-06T17:06:32.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:32.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:32.513041+0000) 2024-03-06T17:06:33.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:33.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:33.513271+0000) 2024-03-06T17:06:34.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:34.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:34.513517+0000) 2024-03-06T17:06:35.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:35.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:35.513787+0000) 2024-03-06T17:06:36.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:36.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:36.514034+0000) 2024-03-06T17:06:37.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:37.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:37.514260+0000) 2024-03-06T17:06:38.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:38.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:38.514456+0000) 2024-03-06T17:06:39.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:39.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:39.514699+0000) 2024-03-06T17:06:40.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:40.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:40.514885+0000) 2024-03-06T17:06:41.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:41.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:41.515126+0000) 2024-03-06T17:06:42.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:42.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:42.515354+0000) 2024-03-06T17:06:43.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:43.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:43.515586+0000) 2024-03-06T17:06:44.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:44.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:44.515800+0000) 2024-03-06T17:06:45.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:45.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:45.515980+0000) 2024-03-06T17:06:46.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:46.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:46.516159+0000) 2024-03-06T17:06:47.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:47.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:47.516395+0000) 2024-03-06T17:06:48.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:48.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:48.516623+0000) 2024-03-06T17:06:49.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:49.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:49.516857+0000) 2024-03-06T17:06:50.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:50.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:50.517088+0000) 2024-03-06T17:06:51.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:51.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:51.517323+0000) 2024-03-06T17:06:52.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:52.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:52.517564+0000) 2024-03-06T17:06:53.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:53.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:53.517793+0000) 2024-03-06T17:06:54.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:54.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:54.518020+0000) 2024-03-06T17:06:55.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:55.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:55.518243+0000) 2024-03-06T17:06:56.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:56.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:56.518428+0000) 2024-03-06T17:06:57.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:57.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:57.518647+0000) 2024-03-06T17:06:58.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:58.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:58.518937+0000) 2024-03-06T17:06:59.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:06:59.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:59.519168+0000) 2024-03-06T17:07:00.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:00.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:00.519348+0000) 2024-03-06T17:07:01.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:01.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:01.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:01.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:01.519536+0000) 2024-03-06T17:07:02.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:02.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:02.519773+0000) 2024-03-06T17:07:03.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:03.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:03.520021+0000) 2024-03-06T17:07:04.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:04.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:04.520251+0000) 2024-03-06T17:07:05.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:05.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:05.520476+0000) 2024-03-06T17:07:06.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:06.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:06.520747+0000) 2024-03-06T17:07:07.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:07.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:07.520979+0000) 2024-03-06T17:07:08.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:08.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:08.521159+0000) 2024-03-06T17:07:09.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:09.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:09.521381+0000) 2024-03-06T17:07:10.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:10.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:10.521602+0000) 2024-03-06T17:07:11.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:11.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:11.521816+0000) 2024-03-06T17:07:12.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:12.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:12.522055+0000) 2024-03-06T17:07:13.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:13.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:13.522286+0000) 2024-03-06T17:07:14.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:14.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:14.522512+0000) 2024-03-06T17:07:15.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:15.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:15.522805+0000) 2024-03-06T17:07:16.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:16.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:16.523038+0000) 2024-03-06T17:07:17.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:17.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:17.523262+0000) 2024-03-06T17:07:18.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:18.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:18.523451+0000) 2024-03-06T17:07:19.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:19.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:19.523624+0000) 2024-03-06T17:07:20.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:20.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:20.523861+0000) 2024-03-06T17:07:21.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:21.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:21.524096+0000) 2024-03-06T17:07:22.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:22.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:22.524342+0000) 2024-03-06T17:07:23.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:23.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:23.524573+0000) 2024-03-06T17:07:24.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:24.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:24.524795+0000) 2024-03-06T17:07:25.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:25.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:25.525016+0000) 2024-03-06T17:07:26.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:26.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:26.525241+0000) 2024-03-06T17:07:27.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:27.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:27.525513+0000) 2024-03-06T17:07:28.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:28.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:28.525796+0000) 2024-03-06T17:07:29.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:29.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:29.525976+0000) 2024-03-06T17:07:30.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:30.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:30.526197+0000) 2024-03-06T17:07:31.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:31.517 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:31.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:31.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:31.526341+0000) 2024-03-06T17:07:32.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:32.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:32.526560+0000) 2024-03-06T17:07:33.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:33.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:33.526783+0000) 2024-03-06T17:07:34.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:34.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:34.527008+0000) 2024-03-06T17:07:35.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:35.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:35.527233+0000) 2024-03-06T17:07:36.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:36.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:36.527456+0000) 2024-03-06T17:07:37.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:37.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:37.527714+0000) 2024-03-06T17:07:38.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:38.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:38.527956+0000) 2024-03-06T17:07:39.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:39.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:39.528189+0000) 2024-03-06T17:07:40.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:40.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:40.528414+0000) 2024-03-06T17:07:41.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:41.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:41.528603+0000) 2024-03-06T17:07:42.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:42.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:42.528836+0000) 2024-03-06T17:07:43.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:43.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:43.529065+0000) 2024-03-06T17:07:44.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:44.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:44.529292+0000) 2024-03-06T17:07:45.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:45.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:45.529509+0000) 2024-03-06T17:07:46.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:46.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:46.529783+0000) 2024-03-06T17:07:47.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:47.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:47.530015+0000) 2024-03-06T17:07:48.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:48.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:48.530242+0000) 2024-03-06T17:07:49.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:49.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:49.530470+0000) 2024-03-06T17:07:50.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:50.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:50.530649+0000) 2024-03-06T17:07:51.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:51.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:51.530848+0000) 2024-03-06T17:07:52.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:52.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:52.531081+0000) 2024-03-06T17:07:53.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:53.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:53.531311+0000) 2024-03-06T17:07:54.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:54.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:54.531542+0000) 2024-03-06T17:07:55.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:55.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:55.531800+0000) 2024-03-06T17:07:56.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:56.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:56.531998+0000) 2024-03-06T17:07:57.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:57.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:57.532245+0000) 2024-03-06T17:07:58.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:58.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:58.532436+0000) 2024-03-06T17:07:59.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:07:59.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:59.532699+0000) 2024-03-06T17:08:00.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:00.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:00.532919+0000) 2024-03-06T17:08:01.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:01.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:01.533160+0000) 2024-03-06T17:08:01.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:01.738 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:02.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:02.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:02.533392+0000) 2024-03-06T17:08:03.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:03.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:03.533625+0000) 2024-03-06T17:08:04.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:04.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:04.533859+0000) 2024-03-06T17:08:05.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:05.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:05.534082+0000) 2024-03-06T17:08:06.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:06.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:06.534270+0000) 2024-03-06T17:08:07.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:07.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:07.534522+0000) 2024-03-06T17:08:08.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:08.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:08.534805+0000) 2024-03-06T17:08:09.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:09.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:09.535030+0000) 2024-03-06T17:08:10.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:10.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:10.535263+0000) 2024-03-06T17:08:11.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:11.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:11.535504+0000) 2024-03-06T17:08:12.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:12.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:12.535781+0000) 2024-03-06T17:08:13.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:13.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:13.535986+0000) 2024-03-06T17:08:14.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:14.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:14.536204+0000) 2024-03-06T17:08:15.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:15.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:15.536472+0000) 2024-03-06T17:08:16.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:16.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:16.536647+0000) 2024-03-06T17:08:17.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:17.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:17.536932+0000) 2024-03-06T17:08:18.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:18.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:18.537170+0000) 2024-03-06T17:08:19.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:19.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:19.537362+0000) 2024-03-06T17:08:20.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:20.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:20.537577+0000) 2024-03-06T17:08:21.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:21.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:21.537830+0000) 2024-03-06T17:08:22.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:22.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:22.538066+0000) 2024-03-06T17:08:23.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:23.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:23.538292+0000) 2024-03-06T17:08:24.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:24.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:24.538524+0000) 2024-03-06T17:08:25.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:25.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:25.538805+0000) 2024-03-06T17:08:26.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:26.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:26.538990+0000) 2024-03-06T17:08:27.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:27.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:27.539185+0000) 2024-03-06T17:08:28.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:28.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:28.539380+0000) 2024-03-06T17:08:29.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:29.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:29.539600+0000) 2024-03-06T17:08:30.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:30.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:30.539826+0000) 2024-03-06T17:08:31.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:31.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:31.540077+0000) 2024-03-06T17:08:31.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:31.876 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:32.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:32.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:32.540326+0000) 2024-03-06T17:08:33.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:33.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:33.540566+0000) 2024-03-06T17:08:34.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:34.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:34.540801+0000) 2024-03-06T17:08:35.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:35.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:35.541039+0000) 2024-03-06T17:08:36.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:36.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:36.541181+0000) 2024-03-06T17:08:37.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:37.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:37.541368+0000) 2024-03-06T17:08:38.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:38.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:38.541611+0000) 2024-03-06T17:08:39.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:39.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:39.541838+0000) 2024-03-06T17:08:40.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:40.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:40.542058+0000) 2024-03-06T17:08:41.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:41.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:41.542290+0000) 2024-03-06T17:08:42.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:42.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:42.542534+0000) 2024-03-06T17:08:43.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:43.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:43.542769+0000) 2024-03-06T17:08:44.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:44.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:44.542965+0000) 2024-03-06T17:08:45.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:45.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:45.543192+0000) 2024-03-06T17:08:46.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:46.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:46.543404+0000) 2024-03-06T17:08:47.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:47.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:47.543641+0000) 2024-03-06T17:08:48.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:48.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:48.543933+0000) 2024-03-06T17:08:49.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:49.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:49.544169+0000) 2024-03-06T17:08:50.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:50.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:50.544400+0000) 2024-03-06T17:08:51.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:51.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:51.544636+0000) 2024-03-06T17:08:52.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:52.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:52.544938+0000) 2024-03-06T17:08:53.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:53.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:53.545183+0000) 2024-03-06T17:08:54.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:54.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:54.545343+0000) 2024-03-06T17:08:55.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:55.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:55.545562+0000) 2024-03-06T17:08:56.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:56.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:56.545799+0000) 2024-03-06T17:08:57.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:57.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:57.546051+0000) 2024-03-06T17:08:58.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:58.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:58.546268+0000) 2024-03-06T17:08:59.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:08:59.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:59.546433+0000) 2024-03-06T17:09:00.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:00.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:00.546659+0000) 2024-03-06T17:09:01.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:01.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:01.546960+0000) 2024-03-06T17:09:02.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:02.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:02.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:02.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:02.547193+0000) 2024-03-06T17:09:03.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:03.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:03.547426+0000) 2024-03-06T17:09:04.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:04.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:04.547720+0000) 2024-03-06T17:09:05.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:05.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:05.547961+0000) 2024-03-06T17:09:06.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:06.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:06.548121+0000) 2024-03-06T17:09:07.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:07.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:07.548341+0000) 2024-03-06T17:09:08.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:08.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:08.548625+0000) 2024-03-06T17:09:09.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:09.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:09.548782+0000) 2024-03-06T17:09:10.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:10.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:10.548983+0000) 2024-03-06T17:09:11.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:11.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:11.549217+0000) 2024-03-06T17:09:12.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:12.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:12.549455+0000) 2024-03-06T17:09:13.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:13.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:13.549713+0000) 2024-03-06T17:09:14.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:14.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:14.549940+0000) 2024-03-06T17:09:15.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:15.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:15.550167+0000) 2024-03-06T17:09:16.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:16.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:16.550354+0000) 2024-03-06T17:09:17.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:17.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:17.550591+0000) 2024-03-06T17:09:18.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:18.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:18.550829+0000) 2024-03-06T17:09:19.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:19.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:19.551029+0000) 2024-03-06T17:09:20.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:20.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:20.551263+0000) 2024-03-06T17:09:21.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:21.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:21.551465+0000) 2024-03-06T17:09:22.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:22.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:22.551635+0000) 2024-03-06T17:09:23.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:23.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:23.551824+0000) 2024-03-06T17:09:24.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:24.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:24.552052+0000) 2024-03-06T17:09:25.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:25.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:25.552274+0000) 2024-03-06T17:09:26.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:26.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:26.552497+0000) 2024-03-06T17:09:27.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:27.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:27.552778+0000) 2024-03-06T17:09:28.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:28.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:28.552972+0000) 2024-03-06T17:09:29.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:29.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:29.553203+0000) 2024-03-06T17:09:30.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:30.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:30.553430+0000) 2024-03-06T17:09:31.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:31.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:31.553619+0000) 2024-03-06T17:09:32.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:32.250 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:32.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:32.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:32.553882+0000) 2024-03-06T17:09:33.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:33.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:33.554076+0000) 2024-03-06T17:09:34.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:34.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:34.554308+0000) 2024-03-06T17:09:35.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:35.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:35.554530+0000) 2024-03-06T17:09:36.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:36.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:36.554750+0000) 2024-03-06T17:09:37.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:37.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:37.554956+0000) 2024-03-06T17:09:38.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:38.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:38.555148+0000) 2024-03-06T17:09:39.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:39.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:39.555380+0000) 2024-03-06T17:09:40.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:40.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:40.555569+0000) 2024-03-06T17:09:41.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:41.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:41.555825+0000) 2024-03-06T17:09:42.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:42.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:42.556082+0000) 2024-03-06T17:09:43.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:43.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:43.556315+0000) 2024-03-06T17:09:44.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:44.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:44.556451+0000) 2024-03-06T17:09:45.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:45.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:45.556705+0000) 2024-03-06T17:09:46.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:46.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:46.556939+0000) 2024-03-06T17:09:47.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:47.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:47.557137+0000) 2024-03-06T17:09:48.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:48.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:48.557350+0000) 2024-03-06T17:09:49.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:49.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:49.557539+0000) 2024-03-06T17:09:50.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:50.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:50.557769+0000) 2024-03-06T17:09:51.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:51.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:51.557973+0000) 2024-03-06T17:09:52.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:52.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:52.558197+0000) 2024-03-06T17:09:53.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:53.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:53.558434+0000) 2024-03-06T17:09:54.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:54.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:54.558696+0000) 2024-03-06T17:09:55.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:55.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:55.558895+0000) 2024-03-06T17:09:56.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:56.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:56.559130+0000) 2024-03-06T17:09:57.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:57.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:57.559370+0000) 2024-03-06T17:09:58.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:58.558+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:58.559622+0000) 2024-03-06T17:09:59.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:09:59.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:59.559865+0000) 2024-03-06T17:10:00.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:00.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:00.560058+0000) 2024-03-06T17:10:01.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:01.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:01.560272+0000) 2024-03-06T17:10:02.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:02.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:02.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:02.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:02.560483+0000) 2024-03-06T17:10:03.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:03.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:03.560699+0000) 2024-03-06T17:10:04.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:04.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:04.560859+0000) 2024-03-06T17:10:05.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:05.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:05.561032+0000) 2024-03-06T17:10:06.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:06.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:06.561233+0000) 2024-03-06T17:10:07.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:07.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:07.561439+0000) 2024-03-06T17:10:08.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:08.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:08.561723+0000) 2024-03-06T17:10:09.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:09.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:09.561947+0000) 2024-03-06T17:10:10.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:10.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:10.562178+0000) 2024-03-06T17:10:11.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:11.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:11.562420+0000) 2024-03-06T17:10:12.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:12.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:12.562710+0000) 2024-03-06T17:10:13.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:13.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:13.563009+0000) 2024-03-06T17:10:14.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:14.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:14.563217+0000) 2024-03-06T17:10:15.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:15.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:15.563494+0000) 2024-03-06T17:10:16.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:16.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:16.563773+0000) 2024-03-06T17:10:17.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:17.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:17.564016+0000) 2024-03-06T17:10:18.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:18.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:18.564229+0000) 2024-03-06T17:10:19.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:19.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:19.564412+0000) 2024-03-06T17:10:20.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:20.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:20.564600+0000) 2024-03-06T17:10:21.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:21.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:21.564811+0000) 2024-03-06T17:10:22.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:22.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:22.565039+0000) 2024-03-06T17:10:23.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:23.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:23.565234+0000) 2024-03-06T17:10:24.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:24.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:24.565487+0000) 2024-03-06T17:10:25.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:25.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:25.565750+0000) 2024-03-06T17:10:26.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:26.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:26.565956+0000) 2024-03-06T17:10:27.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:27.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:27.566197+0000) 2024-03-06T17:10:28.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:28.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:28.566433+0000) 2024-03-06T17:10:29.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:29.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:29.566659+0000) 2024-03-06T17:10:30.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:30.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:30.566933+0000) 2024-03-06T17:10:31.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:31.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:31.567130+0000) 2024-03-06T17:10:32.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:32.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:32.567383+0000) 2024-03-06T17:10:32.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:32.640 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:33.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:33.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:33.567638+0000) 2024-03-06T17:10:34.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:34.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:34.567926+0000) 2024-03-06T17:10:35.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:35.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:35.568130+0000) 2024-03-06T17:10:36.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:36.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:36.568314+0000) 2024-03-06T17:10:37.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:37.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:37.568464+0000) 2024-03-06T17:10:38.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:38.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:38.568731+0000) 2024-03-06T17:10:39.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:39.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:39.568923+0000) 2024-03-06T17:10:40.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:40.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:40.569173+0000) 2024-03-06T17:10:41.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:41.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:41.569421+0000) 2024-03-06T17:10:42.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:42.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:42.569654+0000) 2024-03-06T17:10:43.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:43.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:43.569893+0000) 2024-03-06T17:10:44.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:44.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:44.570161+0000) 2024-03-06T17:10:45.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:45.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:45.570342+0000) 2024-03-06T17:10:46.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:46.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:46.570583+0000) 2024-03-06T17:10:47.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:47.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:47.570826+0000) 2024-03-06T17:10:48.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:48.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:48.571069+0000) 2024-03-06T17:10:49.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:49.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:49.571294+0000) 2024-03-06T17:10:50.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:50.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:50.571525+0000) 2024-03-06T17:10:51.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:51.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:51.571741+0000) 2024-03-06T17:10:52.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:52.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:52.571947+0000) 2024-03-06T17:10:53.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:53.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:53.572138+0000) 2024-03-06T17:10:54.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:54.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:54.572347+0000) 2024-03-06T17:10:55.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:55.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:55.572608+0000) 2024-03-06T17:10:56.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:56.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:56.572845+0000) 2024-03-06T17:10:57.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:57.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:57.573112+0000) 2024-03-06T17:10:58.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:58.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:58.573316+0000) 2024-03-06T17:10:59.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:10:59.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:59.573552+0000) 2024-03-06T17:11:00.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:00.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:00.573824+0000) 2024-03-06T17:11:01.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:01.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:01.574116+0000) 2024-03-06T17:11:02.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:02.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:02.574396+0000) 2024-03-06T17:11:02.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:02.854 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:03.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:03.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:03.574716+0000) 2024-03-06T17:11:04.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:04.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:04.575000+0000) 2024-03-06T17:11:05.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:05.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:05.575285+0000) 2024-03-06T17:11:06.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:06.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:06.575562+0000) 2024-03-06T17:11:07.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:07.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:07.575846+0000) 2024-03-06T17:11:08.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:08.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:08.576098+0000) 2024-03-06T17:11:09.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:09.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:09.576383+0000) 2024-03-06T17:11:10.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:10.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:10.576613+0000) 2024-03-06T17:11:11.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:11.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:11.576950+0000) 2024-03-06T17:11:12.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:12.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:12.577227+0000) 2024-03-06T17:11:13.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:13.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:13.577511+0000) 2024-03-06T17:11:14.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:14.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:14.577816+0000) 2024-03-06T17:11:15.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:15.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:15.578091+0000) 2024-03-06T17:11:16.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:16.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:16.578359+0000) 2024-03-06T17:11:17.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:17.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:17.578572+0000) 2024-03-06T17:11:18.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:18.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:18.578799+0000) 2024-03-06T17:11:19.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:19.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:19.579023+0000) 2024-03-06T17:11:20.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:20.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:20.579242+0000) 2024-03-06T17:11:21.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:21.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:21.579453+0000) 2024-03-06T17:11:22.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:22.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:22.579739+0000) 2024-03-06T17:11:23.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:23.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:23.579936+0000) 2024-03-06T17:11:24.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:24.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:24.580111+0000) 2024-03-06T17:11:25.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:25.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:25.580325+0000) 2024-03-06T17:11:26.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:26.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:26.580546+0000) 2024-03-06T17:11:27.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:27.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:27.580794+0000) 2024-03-06T17:11:28.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:28.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:28.580989+0000) 2024-03-06T17:11:29.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:29.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:29.581175+0000) 2024-03-06T17:11:30.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:30.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:30.581403+0000) 2024-03-06T17:11:31.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:31.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:31.581646+0000) 2024-03-06T17:11:32.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:32.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:32.581879+0000) 2024-03-06T17:11:33.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:33.084 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:33.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:33.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:33.582114+0000) 2024-03-06T17:11:34.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:34.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:34.582333+0000) 2024-03-06T17:11:35.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:35.581+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:35.582557+0000) 2024-03-06T17:11:36.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:36.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:36.582788+0000) 2024-03-06T17:11:37.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:37.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:37.583021+0000) 2024-03-06T17:11:38.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:38.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:38.583307+0000) 2024-03-06T17:11:39.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:39.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:39.583494+0000) 2024-03-06T17:11:40.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:40.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:40.583774+0000) 2024-03-06T17:11:41.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:41.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:41.584010+0000) 2024-03-06T17:11:42.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:42.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:42.584231+0000) 2024-03-06T17:11:43.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:43.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:43.584471+0000) 2024-03-06T17:11:44.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:44.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:44.584644+0000) 2024-03-06T17:11:45.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:45.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:45.584898+0000) 2024-03-06T17:11:46.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:46.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:46.585074+0000) 2024-03-06T17:11:47.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:47.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:47.585272+0000) 2024-03-06T17:11:48.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:48.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:48.585499+0000) 2024-03-06T17:11:49.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:49.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:49.585761+0000) 2024-03-06T17:11:50.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:50.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:50.585987+0000) 2024-03-06T17:11:51.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:51.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:51.586231+0000) 2024-03-06T17:11:52.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:52.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:52.586460+0000) 2024-03-06T17:11:53.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:53.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:53.586727+0000) 2024-03-06T17:11:54.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:54.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:54.586955+0000) 2024-03-06T17:11:55.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:55.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:55.587174+0000) 2024-03-06T17:11:56.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:56.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:56.587332+0000) 2024-03-06T17:11:57.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:57.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:57.587563+0000) 2024-03-06T17:11:58.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:58.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:58.587786+0000) 2024-03-06T17:11:59.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:11:59.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:59.588009+0000) 2024-03-06T17:12:00.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:00.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:00.588197+0000) 2024-03-06T17:12:01.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:01.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:01.588411+0000) 2024-03-06T17:12:02.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:02.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:02.588643+0000) 2024-03-06T17:12:03.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:03.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:03.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:03.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:03.588914+0000) 2024-03-06T17:12:04.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:04.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:04.589160+0000) 2024-03-06T17:12:05.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:05.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:05.589355+0000) 2024-03-06T17:12:06.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:06.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:06.589592+0000) 2024-03-06T17:12:07.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:07.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:07.589837+0000) 2024-03-06T17:12:08.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:08.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:08.590067+0000) 2024-03-06T17:12:09.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:09.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:09.590252+0000) 2024-03-06T17:12:10.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:10.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:10.590482+0000) 2024-03-06T17:12:11.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:11.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:11.590770+0000) 2024-03-06T17:12:12.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:12.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:12.591046+0000) 2024-03-06T17:12:13.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:13.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:13.591335+0000) 2024-03-06T17:12:14.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:14.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:14.591615+0000) 2024-03-06T17:12:15.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:15.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:15.591893+0000) 2024-03-06T17:12:16.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:16.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:16.592096+0000) 2024-03-06T17:12:17.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:17.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:17.592333+0000) 2024-03-06T17:12:18.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:18.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:18.592557+0000) 2024-03-06T17:12:19.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:19.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:19.592795+0000) 2024-03-06T17:12:20.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:20.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:20.593016+0000) 2024-03-06T17:12:21.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:21.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:21.593249+0000) 2024-03-06T17:12:22.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:22.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:22.593474+0000) 2024-03-06T17:12:23.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:23.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:23.593700+0000) 2024-03-06T17:12:24.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:24.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:24.593881+0000) 2024-03-06T17:12:25.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:25.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:25.594071+0000) 2024-03-06T17:12:26.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:26.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:26.594314+0000) 2024-03-06T17:12:27.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:27.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:27.594534+0000) 2024-03-06T17:12:28.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:28.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:28.594792+0000) 2024-03-06T17:12:29.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:29.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:29.594988+0000) 2024-03-06T17:12:30.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:30.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:30.595185+0000) 2024-03-06T17:12:31.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:31.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:31.595444+0000) 2024-03-06T17:12:32.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:32.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:32.595641+0000) 2024-03-06T17:12:33.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:33.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:33.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:33.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:33.595847+0000) 2024-03-06T17:12:34.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:34.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:34.596063+0000) 2024-03-06T17:12:35.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:35.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:35.596245+0000) 2024-03-06T17:12:36.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:36.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:36.596480+0000) 2024-03-06T17:12:37.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:37.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:37.596733+0000) 2024-03-06T17:12:38.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:38.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:38.596955+0000) 2024-03-06T17:12:39.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:39.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:39.597184+0000) 2024-03-06T17:12:40.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:40.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:40.597362+0000) 2024-03-06T17:12:41.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:41.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:41.597597+0000) 2024-03-06T17:12:42.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:42.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:42.597822+0000) 2024-03-06T17:12:43.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:43.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:43.598054+0000) 2024-03-06T17:12:44.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:44.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:44.598239+0000) 2024-03-06T17:12:45.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:45.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:45.598470+0000) 2024-03-06T17:12:46.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:46.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:46.598727+0000) 2024-03-06T17:12:47.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:47.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:47.598947+0000) 2024-03-06T17:12:48.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:48.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:48.599108+0000) 2024-03-06T17:12:49.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:49.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:49.599335+0000) 2024-03-06T17:12:50.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:50.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:50.599494+0000) 2024-03-06T17:12:51.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:51.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:51.599774+0000) 2024-03-06T17:12:52.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:52.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:52.600001+0000) 2024-03-06T17:12:53.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:53.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:53.600247+0000) 2024-03-06T17:12:54.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:54.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:54.600509+0000) 2024-03-06T17:12:55.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:55.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:55.600781+0000) 2024-03-06T17:12:56.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:56.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:56.600988+0000) 2024-03-06T17:12:57.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:57.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:57.601181+0000) 2024-03-06T17:12:58.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:58.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:58.601406+0000) 2024-03-06T17:12:59.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:12:59.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:59.601638+0000) 2024-03-06T17:13:00.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:00.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:00.601905+0000) 2024-03-06T17:13:01.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:01.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:01.602136+0000) 2024-03-06T17:13:02.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:02.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:02.602348+0000) 2024-03-06T17:13:03.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:03.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:03.602583+0000) 2024-03-06T17:13:03.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:03.729 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:04.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:04.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:04.602773+0000) 2024-03-06T17:13:05.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:05.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:05.603043+0000) 2024-03-06T17:13:06.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:06.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:06.603321+0000) 2024-03-06T17:13:07.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:07.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:07.603566+0000) 2024-03-06T17:13:08.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:08.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:08.603753+0000) 2024-03-06T17:13:09.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:09.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:09.603977+0000) 2024-03-06T17:13:10.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:10.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:10.604203+0000) 2024-03-06T17:13:11.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:11.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:11.604447+0000) 2024-03-06T17:13:12.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:12.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:12.604718+0000) 2024-03-06T17:13:13.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:13.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:13.604976+0000) 2024-03-06T17:13:14.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:14.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:14.605226+0000) 2024-03-06T17:13:15.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:15.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:15.605399+0000) 2024-03-06T17:13:16.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:16.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:16.605711+0000) 2024-03-06T17:13:17.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:17.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:17.605972+0000) 2024-03-06T17:13:18.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:18.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:18.606186+0000) 2024-03-06T17:13:19.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:19.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:19.606375+0000) 2024-03-06T17:13:20.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:20.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:20.606602+0000) 2024-03-06T17:13:21.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:21.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:21.606842+0000) 2024-03-06T17:13:22.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:22.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:22.607083+0000) 2024-03-06T17:13:23.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:23.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:23.607290+0000) 2024-03-06T17:13:24.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:24.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:24.607529+0000) 2024-03-06T17:13:25.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:25.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:25.607781+0000) 2024-03-06T17:13:26.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:26.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:26.607991+0000) 2024-03-06T17:13:27.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:27.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:27.608225+0000) 2024-03-06T17:13:28.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:28.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:28.608435+0000) 2024-03-06T17:13:29.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:29.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:29.608631+0000) 2024-03-06T17:13:30.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:30.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:30.608820+0000) 2024-03-06T17:13:31.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:31.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:31.609023+0000) 2024-03-06T17:13:32.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:32.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:32.609254+0000) 2024-03-06T17:13:33.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:33.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:33.609489+0000) 2024-03-06T17:13:33.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:33.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:34.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:34.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:34.609756+0000) 2024-03-06T17:13:35.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:35.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:35.609951+0000) 2024-03-06T17:13:36.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:36.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:36.610188+0000) 2024-03-06T17:13:37.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:37.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:37.610363+0000) 2024-03-06T17:13:38.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:38.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:38.610580+0000) 2024-03-06T17:13:39.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:39.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:39.610804+0000) 2024-03-06T17:13:40.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:40.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:40.611049+0000) 2024-03-06T17:13:41.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:41.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:41.611355+0000) 2024-03-06T17:13:42.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:42.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:42.611592+0000) 2024-03-06T17:13:43.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:43.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:43.611783+0000) 2024-03-06T17:13:44.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:44.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:44.612053+0000) 2024-03-06T17:13:45.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:45.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:45.612328+0000) 2024-03-06T17:13:46.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:46.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:46.612524+0000) 2024-03-06T17:13:47.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:47.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:47.612775+0000) 2024-03-06T17:13:48.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:48.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:48.613017+0000) 2024-03-06T17:13:49.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:49.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:49.613212+0000) 2024-03-06T17:13:50.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:50.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:50.613444+0000) 2024-03-06T17:13:51.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:51.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:51.613722+0000) 2024-03-06T17:13:52.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:52.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:52.613960+0000) 2024-03-06T17:13:53.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:53.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:53.614203+0000) 2024-03-06T17:13:54.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:54.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:54.614432+0000) 2024-03-06T17:13:55.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:55.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:55.614655+0000) 2024-03-06T17:13:56.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:56.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:56.614927+0000) 2024-03-06T17:13:57.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:57.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:57.615169+0000) 2024-03-06T17:13:58.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:58.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:58.615367+0000) 2024-03-06T17:13:59.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:13:59.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:59.615592+0000) 2024-03-06T17:14:00.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:00.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:00.615764+0000) 2024-03-06T17:14:01.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:01.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:01.615979+0000) 2024-03-06T17:14:02.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:02.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:02.616112+0000) 2024-03-06T17:14:03.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:03.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:03.616298+0000) 2024-03-06T17:14:04.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:04.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:04.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:04.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:04.616568+0000) 2024-03-06T17:14:05.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:05.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:05.616835+0000) 2024-03-06T17:14:06.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:06.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:06.617030+0000) 2024-03-06T17:14:07.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:07.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:07.617310+0000) 2024-03-06T17:14:08.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:08.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:08.617554+0000) 2024-03-06T17:14:09.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:09.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:09.617802+0000) 2024-03-06T17:14:10.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:10.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:10.618043+0000) 2024-03-06T17:14:11.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:11.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:11.618289+0000) 2024-03-06T17:14:12.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:12.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:12.618522+0000) 2024-03-06T17:14:13.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:13.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:13.618792+0000) 2024-03-06T17:14:14.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:14.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:14.619018+0000) 2024-03-06T17:14:15.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:15.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:15.619259+0000) 2024-03-06T17:14:16.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:16.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:16.619498+0000) 2024-03-06T17:14:17.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:17.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:17.619767+0000) 2024-03-06T17:14:18.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:18.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:18.619960+0000) 2024-03-06T17:14:19.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:19.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:19.620194+0000) 2024-03-06T17:14:20.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:20.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:20.620417+0000) 2024-03-06T17:14:21.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:21.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:21.620658+0000) 2024-03-06T17:14:22.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:22.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:22.620944+0000) 2024-03-06T17:14:23.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:23.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:23.621198+0000) 2024-03-06T17:14:24.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:24.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:24.621420+0000) 2024-03-06T17:14:25.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:25.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:25.621640+0000) 2024-03-06T17:14:26.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:26.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:26.621853+0000) 2024-03-06T17:14:27.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:27.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:27.622080+0000) 2024-03-06T17:14:28.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:28.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:28.622329+0000) 2024-03-06T17:14:29.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:29.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:29.622613+0000) 2024-03-06T17:14:30.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:30.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:30.622822+0000) 2024-03-06T17:14:31.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:31.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:31.623023+0000) 2024-03-06T17:14:32.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:32.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:32.623201+0000) 2024-03-06T17:14:33.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:33.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:33.623404+0000) 2024-03-06T17:14:34.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:34.287 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:34.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:34.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:34.623610+0000) 2024-03-06T17:14:35.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:35.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:35.623829+0000) 2024-03-06T17:14:36.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:36.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:36.624054+0000) 2024-03-06T17:14:37.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:37.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:37.624278+0000) 2024-03-06T17:14:38.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:38.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:38.624508+0000) 2024-03-06T17:14:39.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:39.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:39.624777+0000) 2024-03-06T17:14:40.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:40.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:40.625020+0000) 2024-03-06T17:14:41.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:41.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:41.625237+0000) 2024-03-06T17:14:42.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:42.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:42.625458+0000) 2024-03-06T17:14:43.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:43.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:43.625727+0000) 2024-03-06T17:14:44.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:44.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:44.625917+0000) 2024-03-06T17:14:45.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:45.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:45.626096+0000) 2024-03-06T17:14:46.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:46.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:46.626316+0000) 2024-03-06T17:14:47.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:47.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:47.626548+0000) 2024-03-06T17:14:48.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:48.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:48.626776+0000) 2024-03-06T17:14:49.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:49.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:49.626973+0000) 2024-03-06T17:14:50.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:50.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:50.627201+0000) 2024-03-06T17:14:51.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:51.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:51.627436+0000) 2024-03-06T17:14:52.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:52.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:52.627616+0000) 2024-03-06T17:14:53.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:53.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:53.627872+0000) 2024-03-06T17:14:54.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:54.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:54.628098+0000) 2024-03-06T17:14:55.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:55.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:55.628376+0000) 2024-03-06T17:14:56.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:56.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:56.628601+0000) 2024-03-06T17:14:57.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:57.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:57.628856+0000) 2024-03-06T17:14:58.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:58.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:58.629013+0000) 2024-03-06T17:14:59.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:14:59.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:59.629203+0000) 2024-03-06T17:15:00.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:00.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:00.629399+0000) 2024-03-06T17:15:01.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:01.628+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:01.629638+0000) 2024-03-06T17:15:02.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:02.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:02.629900+0000) 2024-03-06T17:15:03.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:03.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:03.630176+0000) 2024-03-06T17:15:04.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:04.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:04.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:04.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:04.630413+0000) 2024-03-06T17:15:05.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:05.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:05.630710+0000) 2024-03-06T17:15:06.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:06.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:06.630986+0000) 2024-03-06T17:15:07.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:07.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:07.631273+0000) 2024-03-06T17:15:08.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:08.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:08.631551+0000) 2024-03-06T17:15:09.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:09.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:09.631833+0000) 2024-03-06T17:15:10.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:10.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:10.632099+0000) 2024-03-06T17:15:11.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:11.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:11.632333+0000) 2024-03-06T17:15:12.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:12.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:12.632550+0000) 2024-03-06T17:15:13.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:13.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:13.632801+0000) 2024-03-06T17:15:14.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:14.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:14.633028+0000) 2024-03-06T17:15:15.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:15.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:15.633293+0000) 2024-03-06T17:15:16.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:16.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:16.633496+0000) 2024-03-06T17:15:17.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:17.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:17.633767+0000) 2024-03-06T17:15:18.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:18.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:18.633940+0000) 2024-03-06T17:15:19.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:19.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:19.634157+0000) 2024-03-06T17:15:20.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:20.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:20.634380+0000) 2024-03-06T17:15:21.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:21.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:21.634609+0000) 2024-03-06T17:15:22.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:22.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:22.634858+0000) 2024-03-06T17:15:23.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:23.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:23.635117+0000) 2024-03-06T17:15:24.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:24.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:24.635342+0000) 2024-03-06T17:15:25.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:25.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:25.635562+0000) 2024-03-06T17:15:26.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:26.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:26.635791+0000) 2024-03-06T17:15:27.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:27.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:27.636018+0000) 2024-03-06T17:15:28.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:28.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:28.636218+0000) 2024-03-06T17:15:29.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:29.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:29.636486+0000) 2024-03-06T17:15:30.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:30.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:30.636787+0000) 2024-03-06T17:15:31.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:31.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:31.637036+0000) 2024-03-06T17:15:32.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:32.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:32.637230+0000) 2024-03-06T17:15:33.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:33.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:33.637480+0000) 2024-03-06T17:15:34.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:34.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:34.637753+0000) 2024-03-06T17:15:34.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:34.684 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:35.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:35.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:35.637982+0000) 2024-03-06T17:15:36.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:36.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:36.638202+0000) 2024-03-06T17:15:37.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:37.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:37.638426+0000) 2024-03-06T17:15:38.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:38.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:38.638655+0000) 2024-03-06T17:15:39.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:39.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:39.638927+0000) 2024-03-06T17:15:40.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:40.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:40.639151+0000) 2024-03-06T17:15:41.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:41.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:41.639392+0000) 2024-03-06T17:15:42.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:42.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:42.639625+0000) 2024-03-06T17:15:43.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:43.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:43.639900+0000) 2024-03-06T17:15:44.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:44.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:44.640135+0000) 2024-03-06T17:15:45.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:45.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:45.640325+0000) 2024-03-06T17:15:46.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:46.639+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:46.640582+0000) 2024-03-06T17:15:47.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:47.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:47.640864+0000) 2024-03-06T17:15:48.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:48.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:48.641149+0000) 2024-03-06T17:15:49.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:49.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:49.641433+0000) 2024-03-06T17:15:50.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:50.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:50.641652+0000) 2024-03-06T17:15:51.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:51.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:51.641907+0000) 2024-03-06T17:15:52.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:52.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:52.642150+0000) 2024-03-06T17:15:53.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:53.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:53.642377+0000) 2024-03-06T17:15:54.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:54.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:54.642617+0000) 2024-03-06T17:15:55.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:55.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:55.642794+0000) 2024-03-06T17:15:56.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:56.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:56.642995+0000) 2024-03-06T17:15:57.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:57.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:57.643205+0000) 2024-03-06T17:15:58.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:58.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:58.643417+0000) 2024-03-06T17:15:59.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:15:59.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:59.643744+0000) 2024-03-06T17:16:00.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:00.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:00.644031+0000) 2024-03-06T17:16:01.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:01.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:01.644341+0000) 2024-03-06T17:16:02.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:02.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:02.644558+0000) 2024-03-06T17:16:03.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:03.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:03.644838+0000) 2024-03-06T17:16:04.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:04.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:04.645101+0000) 2024-03-06T17:16:04.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:04.814 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:05.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:05.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:05.645365+0000) 2024-03-06T17:16:06.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:06.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:06.645641+0000) 2024-03-06T17:16:07.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:07.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:07.645915+0000) 2024-03-06T17:16:08.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:08.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:08.646201+0000) 2024-03-06T17:16:09.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:09.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:09.646457+0000) 2024-03-06T17:16:10.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:10.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:10.646659+0000) 2024-03-06T17:16:11.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:11.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:11.646894+0000) 2024-03-06T17:16:12.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:12.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:12.647083+0000) 2024-03-06T17:16:13.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:13.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:13.647289+0000) 2024-03-06T17:16:14.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:14.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:14.647508+0000) 2024-03-06T17:16:15.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:15.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:15.647745+0000) 2024-03-06T17:16:16.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:16.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:16.647942+0000) 2024-03-06T17:16:17.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:17.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:17.648135+0000) 2024-03-06T17:16:18.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:18.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:18.648329+0000) 2024-03-06T17:16:19.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:19.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:19.648566+0000) 2024-03-06T17:16:20.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:20.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:20.648816+0000) 2024-03-06T17:16:21.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:21.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:21.649014+0000) 2024-03-06T17:16:22.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:22.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:22.649234+0000) 2024-03-06T17:16:23.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:23.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:23.649465+0000) 2024-03-06T17:16:24.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:24.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:24.649724+0000) 2024-03-06T17:16:25.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:25.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:25.649913+0000) 2024-03-06T17:16:26.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:26.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:26.650102+0000) 2024-03-06T17:16:27.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:27.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:27.650285+0000) 2024-03-06T17:16:28.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:28.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:28.650525+0000) 2024-03-06T17:16:29.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:29.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:29.650792+0000) 2024-03-06T17:16:30.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:30.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:30.650979+0000) 2024-03-06T17:16:31.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:31.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:31.651179+0000) 2024-03-06T17:16:32.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:32.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:32.651401+0000) 2024-03-06T17:16:33.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:33.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:33.651660+0000) 2024-03-06T17:16:34.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:34.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:34.651941+0000) 2024-03-06T17:16:35.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:35.060 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:35.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:35.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:35.652164+0000) 2024-03-06T17:16:36.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:36.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:36.652492+0000) 2024-03-06T17:16:37.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:37.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:37.652800+0000) 2024-03-06T17:16:38.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:38.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:38.653080+0000) 2024-03-06T17:16:39.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:39.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:39.653350+0000) 2024-03-06T17:16:40.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:40.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:40.653538+0000) 2024-03-06T17:16:41.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:41.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:41.653824+0000) 2024-03-06T17:16:42.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:42.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:42.654082+0000) 2024-03-06T17:16:43.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:43.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:43.654370+0000) 2024-03-06T17:16:44.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:44.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:44.654643+0000) 2024-03-06T17:16:45.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:45.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:45.654951+0000) 2024-03-06T17:16:46.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:46.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:46.655215+0000) 2024-03-06T17:16:47.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:47.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:47.655491+0000) 2024-03-06T17:16:48.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:48.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:48.655804+0000) 2024-03-06T17:16:49.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:49.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:49.656022+0000) 2024-03-06T17:16:50.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:50.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:50.656193+0000) 2024-03-06T17:16:51.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:51.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:51.656460+0000) 2024-03-06T17:16:52.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:52.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:52.656645+0000) 2024-03-06T17:16:53.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:53.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:53.656865+0000) 2024-03-06T17:16:54.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:54.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:54.657060+0000) 2024-03-06T17:16:55.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:55.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:55.657258+0000) 2024-03-06T17:16:56.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:56.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:56.657484+0000) 2024-03-06T17:16:57.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:57.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:57.657742+0000) 2024-03-06T17:16:58.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:58.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:58.657933+0000) 2024-03-06T17:16:59.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:16:59.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:59.658163+0000) 2024-03-06T17:17:00.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:00.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:00.658371+0000) 2024-03-06T17:17:01.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:01.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:01.658554+0000) 2024-03-06T17:17:02.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:02.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:02.658790+0000) 2024-03-06T17:17:03.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:03.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:03.658991+0000) 2024-03-06T17:17:04.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:04.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:04.659176+0000) 2024-03-06T17:17:05.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:05.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:05.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:05.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:05.659375+0000) 2024-03-06T17:17:06.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:06.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:06.659600+0000) 2024-03-06T17:17:07.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:07.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:07.659774+0000) 2024-03-06T17:17:08.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:08.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:08.659973+0000) 2024-03-06T17:17:09.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:09.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:09.660165+0000) 2024-03-06T17:17:10.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:10.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:10.660424+0000) 2024-03-06T17:17:11.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:11.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:11.660697+0000) 2024-03-06T17:17:12.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:12.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:12.660893+0000) 2024-03-06T17:17:13.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:13.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:13.661094+0000) 2024-03-06T17:17:14.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:14.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:14.661288+0000) 2024-03-06T17:17:15.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:15.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:15.661513+0000) 2024-03-06T17:17:16.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:16.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:16.661772+0000) 2024-03-06T17:17:17.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:17.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:17.661967+0000) 2024-03-06T17:17:18.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:18.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:18.662208+0000) 2024-03-06T17:17:19.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:19.661+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:19.662447+0000) 2024-03-06T17:17:20.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:20.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:20.662734+0000) 2024-03-06T17:17:21.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:21.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:21.662974+0000) 2024-03-06T17:17:22.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:22.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:22.663195+0000) 2024-03-06T17:17:23.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:23.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:23.663430+0000) 2024-03-06T17:17:24.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:24.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:24.663658+0000) 2024-03-06T17:17:25.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:25.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:25.663872+0000) 2024-03-06T17:17:26.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:26.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:26.664107+0000) 2024-03-06T17:17:27.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:27.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:27.664329+0000) 2024-03-06T17:17:28.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:28.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:28.664556+0000) 2024-03-06T17:17:29.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:29.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:29.664785+0000) 2024-03-06T17:17:30.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:30.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:30.665011+0000) 2024-03-06T17:17:31.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:31.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:31.665304+0000) 2024-03-06T17:17:32.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:32.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:32.665566+0000) 2024-03-06T17:17:33.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:33.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:33.665865+0000) 2024-03-06T17:17:34.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:34.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:34.666066+0000) 2024-03-06T17:17:35.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:35.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:35.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:35.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:35.666310+0000) 2024-03-06T17:17:36.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:36.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:36.666537+0000) 2024-03-06T17:17:37.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:37.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:37.666760+0000) 2024-03-06T17:17:38.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:38.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:38.666991+0000) 2024-03-06T17:17:39.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:39.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:39.667225+0000) 2024-03-06T17:17:40.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:40.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:40.667451+0000) 2024-03-06T17:17:41.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:41.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:41.667650+0000) 2024-03-06T17:17:42.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:42.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:42.667898+0000) 2024-03-06T17:17:43.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:43.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:43.668140+0000) 2024-03-06T17:17:44.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:44.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:44.668404+0000) 2024-03-06T17:17:45.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:45.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:45.668609+0000) 2024-03-06T17:17:46.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:46.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:46.668835+0000) 2024-03-06T17:17:47.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:47.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:47.669020+0000) 2024-03-06T17:17:48.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:48.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:48.669252+0000) 2024-03-06T17:17:49.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:49.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:49.669481+0000) 2024-03-06T17:17:50.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:50.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:50.669738+0000) 2024-03-06T17:17:51.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:51.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:51.669985+0000) 2024-03-06T17:17:52.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:52.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:52.670207+0000) 2024-03-06T17:17:53.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:53.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:53.670432+0000) 2024-03-06T17:17:54.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:54.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:54.670652+0000) 2024-03-06T17:17:55.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:55.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:55.670907+0000) 2024-03-06T17:17:56.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:56.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:56.671095+0000) 2024-03-06T17:17:57.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:57.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:57.671325+0000) 2024-03-06T17:17:58.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:58.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:58.671540+0000) 2024-03-06T17:17:59.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:17:59.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:59.671767+0000) 2024-03-06T17:18:00.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:00.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:00.671990+0000) 2024-03-06T17:18:01.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:01.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:01.672224+0000) 2024-03-06T17:18:02.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:02.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:02.672452+0000) 2024-03-06T17:18:03.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:03.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:03.672723+0000) 2024-03-06T17:18:04.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:04.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:04.672911+0000) 2024-03-06T17:18:05.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:05.613 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:05.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:05.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:05.673098+0000) 2024-03-06T17:18:06.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:06.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:06.673320+0000) 2024-03-06T17:18:07.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:07.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:07.673598+0000) 2024-03-06T17:18:08.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:08.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:08.673854+0000) 2024-03-06T17:18:09.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:09.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:09.674098+0000) 2024-03-06T17:18:10.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:10.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:10.674376+0000) 2024-03-06T17:18:11.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:11.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:11.674703+0000) 2024-03-06T17:18:12.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:12.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:12.674926+0000) 2024-03-06T17:18:13.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:13.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:13.675181+0000) 2024-03-06T17:18:14.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:14.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:14.675451+0000) 2024-03-06T17:18:15.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:15.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:15.675764+0000) 2024-03-06T17:18:16.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:16.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:16.676046+0000) 2024-03-06T17:18:17.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:17.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:17.676287+0000) 2024-03-06T17:18:18.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:18.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:18.676580+0000) 2024-03-06T17:18:19.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:19.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:19.676831+0000) 2024-03-06T17:18:20.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:20.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:20.677058+0000) 2024-03-06T17:18:21.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:21.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:21.677307+0000) 2024-03-06T17:18:22.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:22.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:22.677499+0000) 2024-03-06T17:18:23.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:23.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:23.677773+0000) 2024-03-06T17:18:24.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:24.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:24.677951+0000) 2024-03-06T17:18:25.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:25.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:25.678223+0000) 2024-03-06T17:18:26.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:26.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:26.678501+0000) 2024-03-06T17:18:27.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:27.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:27.678803+0000) 2024-03-06T17:18:28.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:28.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:28.679089+0000) 2024-03-06T17:18:29.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:29.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:29.679358+0000) 2024-03-06T17:18:30.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:30.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:30.679586+0000) 2024-03-06T17:18:31.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:31.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:31.679830+0000) 2024-03-06T17:18:32.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:32.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:32.680017+0000) 2024-03-06T17:18:33.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:33.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:33.680218+0000) 2024-03-06T17:18:34.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:34.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:34.680441+0000) 2024-03-06T17:18:35.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:35.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:35.680692+0000) 2024-03-06T17:18:35.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:35.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:36.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:36.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:36.680884+0000) 2024-03-06T17:18:37.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:37.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:37.681118+0000) 2024-03-06T17:18:38.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:38.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:38.681300+0000) 2024-03-06T17:18:39.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:39.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:39.681515+0000) 2024-03-06T17:18:40.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:40.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:40.681815+0000) 2024-03-06T17:18:41.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:41.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:41.682105+0000) 2024-03-06T17:18:42.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:42.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:42.682385+0000) 2024-03-06T17:18:43.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:43.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:43.682589+0000) 2024-03-06T17:18:44.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:44.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:44.682790+0000) 2024-03-06T17:18:45.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:45.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:45.682992+0000) 2024-03-06T17:18:46.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:46.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:46.683177+0000) 2024-03-06T17:18:47.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:47.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:47.683395+0000) 2024-03-06T17:18:48.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:48.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:48.683595+0000) 2024-03-06T17:18:49.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:49.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:49.683840+0000) 2024-03-06T17:18:50.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:50.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:50.684031+0000) 2024-03-06T17:18:51.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:51.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:51.684276+0000) 2024-03-06T17:18:52.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:52.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:52.684467+0000) 2024-03-06T17:18:53.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:53.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:53.684753+0000) 2024-03-06T17:18:54.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:54.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:54.684938+0000) 2024-03-06T17:18:55.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:55.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:55.685124+0000) 2024-03-06T17:18:56.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:56.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:56.685308+0000) 2024-03-06T17:18:57.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:57.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:57.685532+0000) 2024-03-06T17:18:58.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:58.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:58.685799+0000) 2024-03-06T17:18:59.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:18:59.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:59.685992+0000) 2024-03-06T17:19:00.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:00.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:00.686187+0000) 2024-03-06T17:19:01.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:01.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:01.686415+0000) 2024-03-06T17:19:02.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:02.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:02.686632+0000) 2024-03-06T17:19:03.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:03.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:03.686828+0000) 2024-03-06T17:19:04.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:04.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:04.687020+0000) 2024-03-06T17:19:05.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:05.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:05.687212+0000) 2024-03-06T17:19:05.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:06.006 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:06.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:06.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:06.687446+0000) 2024-03-06T17:19:07.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:07.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:07.687690+0000) 2024-03-06T17:19:08.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:08.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:08.687874+0000) 2024-03-06T17:19:09.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:09.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:09.688064+0000) 2024-03-06T17:19:10.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:10.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:10.688244+0000) 2024-03-06T17:19:11.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:11.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:11.688455+0000) 2024-03-06T17:19:12.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:12.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:12.688717+0000) 2024-03-06T17:19:13.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:13.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:13.688962+0000) 2024-03-06T17:19:14.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:14.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:14.689196+0000) 2024-03-06T17:19:15.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:15.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:15.689403+0000) 2024-03-06T17:19:16.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:16.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:16.689591+0000) 2024-03-06T17:19:17.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:17.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:17.689829+0000) 2024-03-06T17:19:18.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:18.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:18.690017+0000) 2024-03-06T17:19:19.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:19.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:19.690225+0000) 2024-03-06T17:19:20.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:20.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:20.690455+0000) 2024-03-06T17:19:21.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:21.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:21.690658+0000) 2024-03-06T17:19:22.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:22.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:22.690879+0000) 2024-03-06T17:19:23.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:23.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:23.691124+0000) 2024-03-06T17:19:24.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:24.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:24.691341+0000) 2024-03-06T17:19:25.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:25.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:25.691571+0000) 2024-03-06T17:19:26.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:26.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:26.691815+0000) 2024-03-06T17:19:27.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:27.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:27.692003+0000) 2024-03-06T17:19:28.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:28.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:28.692188+0000) 2024-03-06T17:19:29.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:29.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:29.692419+0000) 2024-03-06T17:19:30.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:30.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:30.692640+0000) 2024-03-06T17:19:31.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:31.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:31.692865+0000) 2024-03-06T17:19:32.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:32.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:32.693052+0000) 2024-03-06T17:19:33.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:33.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:33.693242+0000) 2024-03-06T17:19:34.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:34.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:34.693463+0000) 2024-03-06T17:19:35.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:35.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:35.693643+0000) 2024-03-06T17:19:36.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:36.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:36.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:36.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:36.693839+0000) 2024-03-06T17:19:37.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:37.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:37.694026+0000) 2024-03-06T17:19:38.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:38.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:38.694206+0000) 2024-03-06T17:19:39.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:39.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:39.694450+0000) 2024-03-06T17:19:40.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:40.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:40.694700+0000) 2024-03-06T17:19:41.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:41.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:41.694924+0000) 2024-03-06T17:19:42.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:42.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:42.695138+0000) 2024-03-06T17:19:43.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:43.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:43.695363+0000) 2024-03-06T17:19:44.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:44.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:44.695603+0000) 2024-03-06T17:19:45.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:45.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:45.695842+0000) 2024-03-06T17:19:46.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:46.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:46.696052+0000) 2024-03-06T17:19:47.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:47.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:47.696200+0000) 2024-03-06T17:19:48.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:48.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:48.696434+0000) 2024-03-06T17:19:49.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:49.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:49.696693+0000) 2024-03-06T17:19:50.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:50.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:50.696883+0000) 2024-03-06T17:19:51.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:51.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:51.697062+0000) 2024-03-06T17:19:52.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:52.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:52.697281+0000) 2024-03-06T17:19:53.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:53.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:53.697480+0000) 2024-03-06T17:19:54.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:54.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:54.697807+0000) 2024-03-06T17:19:55.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:55.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:55.698036+0000) 2024-03-06T17:19:56.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:56.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:56.698259+0000) 2024-03-06T17:19:57.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:57.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:57.698482+0000) 2024-03-06T17:19:58.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:58.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:58.698726+0000) 2024-03-06T17:19:59.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:19:59.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:59.698971+0000) 2024-03-06T17:20:00.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:00.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:00.699184+0000) 2024-03-06T17:20:01.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:01.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:01.699425+0000) 2024-03-06T17:20:02.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:02.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:02.699647+0000) 2024-03-06T17:20:03.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:03.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:03.699857+0000) 2024-03-06T17:20:04.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:04.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:04.700053+0000) 2024-03-06T17:20:05.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:05.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:05.700247+0000) 2024-03-06T17:20:06.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:06.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:06.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:06.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:06.700466+0000) 2024-03-06T17:20:07.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:07.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:07.700736+0000) 2024-03-06T17:20:08.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:08.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:08.700922+0000) 2024-03-06T17:20:09.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:09.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:09.701130+0000) 2024-03-06T17:20:10.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:10.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:10.701365+0000) 2024-03-06T17:20:11.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:11.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:11.701606+0000) 2024-03-06T17:20:12.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:12.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:12.701834+0000) 2024-03-06T17:20:13.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:13.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:13.702019+0000) 2024-03-06T17:20:14.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:14.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:14.702202+0000) 2024-03-06T17:20:15.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:15.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:15.702427+0000) 2024-03-06T17:20:16.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:16.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:16.702643+0000) 2024-03-06T17:20:17.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:17.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:17.702836+0000) 2024-03-06T17:20:18.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:18.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:18.703025+0000) 2024-03-06T17:20:19.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:19.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:19.703256+0000) 2024-03-06T17:20:20.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:20.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:20.703478+0000) 2024-03-06T17:20:21.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:21.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:21.703749+0000) 2024-03-06T17:20:22.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:22.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:22.703949+0000) 2024-03-06T17:20:23.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:23.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:23.704159+0000) 2024-03-06T17:20:24.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:24.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:24.704367+0000) 2024-03-06T17:20:25.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:25.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:25.704536+0000) 2024-03-06T17:20:26.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:26.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:26.704773+0000) 2024-03-06T17:20:27.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:27.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:27.704969+0000) 2024-03-06T17:20:28.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:28.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:28.705186+0000) 2024-03-06T17:20:29.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:29.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:29.705427+0000) 2024-03-06T17:20:30.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:30.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:30.705612+0000) 2024-03-06T17:20:31.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:31.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:31.705843+0000) 2024-03-06T17:20:32.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:32.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:32.705992+0000) 2024-03-06T17:20:33.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:33.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:33.706182+0000) 2024-03-06T17:20:34.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:34.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:34.706407+0000) 2024-03-06T17:20:35.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:35.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:35.706617+0000) 2024-03-06T17:20:36.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:36.635 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:36.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:36.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:36.706823+0000) 2024-03-06T17:20:37.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:37.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:37.707010+0000) 2024-03-06T17:20:38.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:38.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:38.707201+0000) 2024-03-06T17:20:39.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:39.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:39.707405+0000) 2024-03-06T17:20:40.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:40.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:40.707616+0000) 2024-03-06T17:20:41.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:41.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:41.707899+0000) 2024-03-06T17:20:42.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:42.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:42.708121+0000) 2024-03-06T17:20:43.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:43.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:43.708346+0000) 2024-03-06T17:20:44.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:44.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:44.708566+0000) 2024-03-06T17:20:45.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:45.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:45.708786+0000) 2024-03-06T17:20:46.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:46.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:46.709007+0000) 2024-03-06T17:20:47.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:47.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:47.709249+0000) 2024-03-06T17:20:48.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:48.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:48.709480+0000) 2024-03-06T17:20:49.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:49.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:49.709757+0000) 2024-03-06T17:20:50.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:50.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:50.709978+0000) 2024-03-06T17:20:51.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:51.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:51.710210+0000) 2024-03-06T17:20:52.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:52.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:52.710429+0000) 2024-03-06T17:20:53.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:53.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:53.710650+0000) 2024-03-06T17:20:54.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:54.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:54.710915+0000) 2024-03-06T17:20:55.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:55.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:55.711135+0000) 2024-03-06T17:20:56.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:56.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:56.711357+0000) 2024-03-06T17:20:57.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:57.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:57.711577+0000) 2024-03-06T17:20:58.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:58.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:58.711807+0000) 2024-03-06T17:20:59.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:20:59.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:59.712041+0000) 2024-03-06T17:21:00.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:00.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:00.712214+0000) 2024-03-06T17:21:01.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:01.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:01.712448+0000) 2024-03-06T17:21:02.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:02.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:02.712696+0000) 2024-03-06T17:21:03.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:03.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:03.712946+0000) 2024-03-06T17:21:04.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:04.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:04.713140+0000) 2024-03-06T17:21:05.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:05.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:05.713371+0000) 2024-03-06T17:21:06.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:06.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:06.713591+0000) 2024-03-06T17:21:06.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:06.857 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:07.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:07.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:07.713825+0000) 2024-03-06T17:21:08.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:08.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:08.714016+0000) 2024-03-06T17:21:09.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:09.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:09.714227+0000) 2024-03-06T17:21:10.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:10.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:10.714459+0000) 2024-03-06T17:21:11.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:11.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:11.714716+0000) 2024-03-06T17:21:12.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:12.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:12.714901+0000) 2024-03-06T17:21:13.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:13.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:13.715129+0000) 2024-03-06T17:21:14.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:14.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:14.715303+0000) 2024-03-06T17:21:15.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:15.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:15.715571+0000) 2024-03-06T17:21:16.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:16.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:16.715859+0000) 2024-03-06T17:21:17.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:17.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:17.716117+0000) 2024-03-06T17:21:18.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:18.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:18.716307+0000) 2024-03-06T17:21:19.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:19.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:19.716551+0000) 2024-03-06T17:21:20.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:20.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:20.716816+0000) 2024-03-06T17:21:21.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:21.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:21.717083+0000) 2024-03-06T17:21:22.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:22.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:22.717262+0000) 2024-03-06T17:21:23.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:23.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:23.717455+0000) 2024-03-06T17:21:24.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:24.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:24.717717+0000) 2024-03-06T17:21:25.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:25.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:25.717951+0000) 2024-03-06T17:21:26.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:26.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:26.718175+0000) 2024-03-06T17:21:27.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:27.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:27.718407+0000) 2024-03-06T17:21:28.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:28.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:28.718604+0000) 2024-03-06T17:21:29.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:29.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:29.718863+0000) 2024-03-06T17:21:30.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:30.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:30.719211+0000) 2024-03-06T17:21:31.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:31.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:31.719493+0000) 2024-03-06T17:21:32.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:32.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:32.719795+0000) 2024-03-06T17:21:33.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:33.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:33.720073+0000) 2024-03-06T17:21:34.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:34.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:34.720350+0000) 2024-03-06T17:21:35.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:35.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:35.720639+0000) 2024-03-06T17:21:36.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:36.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:36.720937+0000) 2024-03-06T17:21:37.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:37.086 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:37.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:37.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:37.721222+0000) 2024-03-06T17:21:38.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:38.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:38.721425+0000) 2024-03-06T17:21:39.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:39.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:39.721635+0000) 2024-03-06T17:21:40.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:40.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:40.721905+0000) 2024-03-06T17:21:41.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:41.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:41.722137+0000) 2024-03-06T17:21:42.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:42.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:42.722359+0000) 2024-03-06T17:21:43.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:43.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:43.722631+0000) 2024-03-06T17:21:44.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:44.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:44.722932+0000) 2024-03-06T17:21:45.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:45.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:45.723162+0000) 2024-03-06T17:21:46.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:46.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:46.723371+0000) 2024-03-06T17:21:47.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:47.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:47.723603+0000) 2024-03-06T17:21:48.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:48.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:48.723795+0000) 2024-03-06T17:21:49.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:49.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:49.724058+0000) 2024-03-06T17:21:50.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:50.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:50.724259+0000) 2024-03-06T17:21:51.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:51.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:51.724482+0000) 2024-03-06T17:21:52.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:52.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:52.724761+0000) 2024-03-06T17:21:53.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:53.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:53.724949+0000) 2024-03-06T17:21:54.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:54.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:54.725196+0000) 2024-03-06T17:21:55.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:55.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:55.725432+0000) 2024-03-06T17:21:56.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:56.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:56.725652+0000) 2024-03-06T17:21:57.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:57.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:57.725908+0000) 2024-03-06T17:21:58.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:58.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:58.726136+0000) 2024-03-06T17:21:59.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:21:59.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:59.726375+0000) 2024-03-06T17:22:00.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:00.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:00.726557+0000) 2024-03-06T17:22:01.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:01.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:01.726784+0000) 2024-03-06T17:22:02.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:02.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:02.726985+0000) 2024-03-06T17:22:03.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:03.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:03.727227+0000) 2024-03-06T17:22:04.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:04.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:04.727451+0000) 2024-03-06T17:22:05.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:05.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:05.727695+0000) 2024-03-06T17:22:06.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:06.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:06.727915+0000) 2024-03-06T17:22:07.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:07.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:07.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:07.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:07.728154+0000) 2024-03-06T17:22:08.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:08.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:08.728314+0000) 2024-03-06T17:22:09.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:09.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:09.728565+0000) 2024-03-06T17:22:10.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:10.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:10.728781+0000) 2024-03-06T17:22:11.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:11.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:11.728974+0000) 2024-03-06T17:22:12.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:12.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:12.729182+0000) 2024-03-06T17:22:13.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:13.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:13.729420+0000) 2024-03-06T17:22:14.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:14.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:14.729693+0000) 2024-03-06T17:22:15.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:15.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:15.729958+0000) 2024-03-06T17:22:16.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:16.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:16.730156+0000) 2024-03-06T17:22:17.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:17.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:17.730377+0000) 2024-03-06T17:22:18.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:18.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:18.730572+0000) 2024-03-06T17:22:19.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:19.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:19.730822+0000) 2024-03-06T17:22:20.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:20.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:20.731008+0000) 2024-03-06T17:22:21.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:21.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:21.731194+0000) 2024-03-06T17:22:22.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:22.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:22.731403+0000) 2024-03-06T17:22:23.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:23.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:23.731733+0000) 2024-03-06T17:22:24.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:24.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:24.731889+0000) 2024-03-06T17:22:25.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:25.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:25.732081+0000) 2024-03-06T17:22:26.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:26.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:26.732290+0000) 2024-03-06T17:22:27.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:27.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:27.732491+0000) 2024-03-06T17:22:28.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:28.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:28.732744+0000) 2024-03-06T17:22:29.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:29.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:29.732997+0000) 2024-03-06T17:22:30.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:30.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:30.733276+0000) 2024-03-06T17:22:31.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:31.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:31.733588+0000) 2024-03-06T17:22:32.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:32.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:32.733868+0000) 2024-03-06T17:22:33.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:33.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:33.734065+0000) 2024-03-06T17:22:34.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:34.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:34.734290+0000) 2024-03-06T17:22:35.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:35.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:35.734530+0000) 2024-03-06T17:22:36.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:36.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:36.734785+0000) 2024-03-06T17:22:37.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:37.445 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:37.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:37.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:37.735027+0000) 2024-03-06T17:22:38.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:38.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:38.735255+0000) 2024-03-06T17:22:39.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:39.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:39.735505+0000) 2024-03-06T17:22:40.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:40.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:40.735766+0000) 2024-03-06T17:22:41.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:41.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:41.735991+0000) 2024-03-06T17:22:42.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:42.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:42.736184+0000) 2024-03-06T17:22:43.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:43.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:43.736363+0000) 2024-03-06T17:22:44.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:44.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:44.736541+0000) 2024-03-06T17:22:45.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:45.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:45.736749+0000) 2024-03-06T17:22:46.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:46.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:46.736996+0000) 2024-03-06T17:22:47.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:47.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:47.737219+0000) 2024-03-06T17:22:48.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:48.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:48.737380+0000) 2024-03-06T17:22:49.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:49.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:49.737633+0000) 2024-03-06T17:22:50.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:50.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:50.737782+0000) 2024-03-06T17:22:51.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:51.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:51.738014+0000) 2024-03-06T17:22:52.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:52.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:52.738205+0000) 2024-03-06T17:22:53.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:53.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:53.738420+0000) 2024-03-06T17:22:54.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:54.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:54.738650+0000) 2024-03-06T17:22:55.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:55.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:55.738893+0000) 2024-03-06T17:22:56.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:56.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:56.739118+0000) 2024-03-06T17:22:57.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:57.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:57.739352+0000) 2024-03-06T17:22:58.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:58.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:58.739485+0000) 2024-03-06T17:22:59.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:22:59.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:59.739770+0000) 2024-03-06T17:23:00.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:00.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:00.739968+0000) 2024-03-06T17:23:01.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:01.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:01.740209+0000) 2024-03-06T17:23:02.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:02.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:02.740433+0000) 2024-03-06T17:23:03.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:03.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:03.740622+0000) 2024-03-06T17:23:04.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:04.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:04.740790+0000) 2024-03-06T17:23:05.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:05.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:05.741038+0000) 2024-03-06T17:23:06.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:06.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:06.741213+0000) 2024-03-06T17:23:07.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:07.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:07.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:07.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:07.741376+0000) 2024-03-06T17:23:08.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:08.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:08.741523+0000) 2024-03-06T17:23:09.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:09.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:09.741750+0000) 2024-03-06T17:23:10.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:10.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:10.741939+0000) 2024-03-06T17:23:11.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:11.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:11.742151+0000) 2024-03-06T17:23:12.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:12.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:12.742390+0000) 2024-03-06T17:23:13.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:13.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:13.742604+0000) 2024-03-06T17:23:14.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:14.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:14.742831+0000) 2024-03-06T17:23:15.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:15.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:15.743058+0000) 2024-03-06T17:23:16.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:16.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:16.743272+0000) 2024-03-06T17:23:17.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:17.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:17.743502+0000) 2024-03-06T17:23:18.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:18.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:18.743763+0000) 2024-03-06T17:23:19.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:19.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:19.743984+0000) 2024-03-06T17:23:20.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:20.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:20.744226+0000) 2024-03-06T17:23:21.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:21.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:21.744420+0000) 2024-03-06T17:23:22.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:22.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:22.744617+0000) 2024-03-06T17:23:23.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:23.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:23.744828+0000) 2024-03-06T17:23:24.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:24.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:24.745058+0000) 2024-03-06T17:23:25.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:25.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:25.745220+0000) 2024-03-06T17:23:26.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:26.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:26.745449+0000) 2024-03-06T17:23:27.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:27.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:27.745709+0000) 2024-03-06T17:23:28.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:28.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:28.745947+0000) 2024-03-06T17:23:29.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:29.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:29.746103+0000) 2024-03-06T17:23:30.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:30.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:30.746343+0000) 2024-03-06T17:23:31.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:31.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:31.746541+0000) 2024-03-06T17:23:32.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:32.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:32.746786+0000) 2024-03-06T17:23:33.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:33.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:33.747019+0000) 2024-03-06T17:23:34.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:34.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:34.747270+0000) 2024-03-06T17:23:35.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:35.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:35.747528+0000) 2024-03-06T17:23:36.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:36.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:36.747770+0000) 2024-03-06T17:23:37.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:37.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:37.747991+0000) 2024-03-06T17:23:37.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:37.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:38.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:38.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:38.748196+0000) 2024-03-06T17:23:39.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:39.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:39.748427+0000) 2024-03-06T17:23:40.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:40.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:40.748651+0000) 2024-03-06T17:23:41.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:41.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:41.748824+0000) 2024-03-06T17:23:42.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:42.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:42.749059+0000) 2024-03-06T17:23:43.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:43.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:43.749306+0000) 2024-03-06T17:23:44.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:44.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:44.749541+0000) 2024-03-06T17:23:45.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:45.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:45.749806+0000) 2024-03-06T17:23:46.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:46.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:46.750038+0000) 2024-03-06T17:23:47.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:47.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:47.750258+0000) 2024-03-06T17:23:48.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:48.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:48.750483+0000) 2024-03-06T17:23:49.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:49.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:49.750751+0000) 2024-03-06T17:23:50.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:50.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:50.750937+0000) 2024-03-06T17:23:51.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:51.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:51.751175+0000) 2024-03-06T17:23:52.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:52.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:52.751394+0000) 2024-03-06T17:23:53.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:53.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:53.751619+0000) 2024-03-06T17:23:54.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:54.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:54.751863+0000) 2024-03-06T17:23:55.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:55.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:55.752093+0000) 2024-03-06T17:23:56.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:56.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:56.752294+0000) 2024-03-06T17:23:57.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:57.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:57.752465+0000) 2024-03-06T17:23:58.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:58.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:58.752621+0000) 2024-03-06T17:23:59.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:23:59.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:59.752796+0000) 2024-03-06T17:24:00.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:00.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:00.753027+0000) 2024-03-06T17:24:01.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:01.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:01.753190+0000) 2024-03-06T17:24:02.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:02.752+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:02.753427+0000) 2024-03-06T17:24:03.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:03.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:03.753657+0000) 2024-03-06T17:24:04.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:04.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:04.753982+0000) 2024-03-06T17:24:05.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:05.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:05.754210+0000) 2024-03-06T17:24:06.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:06.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:06.754434+0000) 2024-03-06T17:24:07.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:07.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:07.754661+0000) 2024-03-06T17:24:08.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:08.087 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:08.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:08.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:08.754937+0000) 2024-03-06T17:24:09.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:09.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:09.755179+0000) 2024-03-06T17:24:10.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:10.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:10.755408+0000) 2024-03-06T17:24:11.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:11.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:11.755636+0000) 2024-03-06T17:24:12.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:12.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:12.755884+0000) 2024-03-06T17:24:13.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:13.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:13.756107+0000) 2024-03-06T17:24:14.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:14.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:14.756328+0000) 2024-03-06T17:24:15.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:15.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:15.756550+0000) 2024-03-06T17:24:16.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:16.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:16.756724+0000) 2024-03-06T17:24:17.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:17.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:17.756885+0000) 2024-03-06T17:24:18.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:18.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:18.757121+0000) 2024-03-06T17:24:19.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:19.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:19.757274+0000) 2024-03-06T17:24:20.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:20.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:20.757510+0000) 2024-03-06T17:24:21.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:21.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:21.757762+0000) 2024-03-06T17:24:22.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:22.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:22.757981+0000) 2024-03-06T17:24:23.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:23.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:23.758169+0000) 2024-03-06T17:24:24.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:24.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:24.758359+0000) 2024-03-06T17:24:25.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:25.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:25.758525+0000) 2024-03-06T17:24:26.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:26.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:26.758788+0000) 2024-03-06T17:24:27.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:27.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:27.758991+0000) 2024-03-06T17:24:28.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:28.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:28.759213+0000) 2024-03-06T17:24:29.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:29.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:29.759441+0000) 2024-03-06T17:24:30.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:30.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:30.759694+0000) 2024-03-06T17:24:31.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:31.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:31.759940+0000) 2024-03-06T17:24:32.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:32.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:32.760116+0000) 2024-03-06T17:24:33.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:33.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:33.760345+0000) 2024-03-06T17:24:34.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:34.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:34.760574+0000) 2024-03-06T17:24:35.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:35.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:35.760768+0000) 2024-03-06T17:24:36.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:36.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:36.761007+0000) 2024-03-06T17:24:37.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:37.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:37.761204+0000) 2024-03-06T17:24:38.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:38.299 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:38.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:38.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:38.761387+0000) 2024-03-06T17:24:39.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:39.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:39.761615+0000) 2024-03-06T17:24:40.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:40.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:40.761847+0000) 2024-03-06T17:24:41.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:41.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:41.762079+0000) 2024-03-06T17:24:42.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:42.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:42.762310+0000) 2024-03-06T17:24:43.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:43.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:43.762464+0000) 2024-03-06T17:24:44.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:44.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:44.762733+0000) 2024-03-06T17:24:45.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:45.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:45.762965+0000) 2024-03-06T17:24:46.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:46.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:46.763189+0000) 2024-03-06T17:24:47.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:47.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:47.763424+0000) 2024-03-06T17:24:48.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:48.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:48.763617+0000) 2024-03-06T17:24:49.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:49.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:49.763840+0000) 2024-03-06T17:24:50.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:50.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:50.764012+0000) 2024-03-06T17:24:51.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:51.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:51.764244+0000) 2024-03-06T17:24:52.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:52.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:52.764496+0000) 2024-03-06T17:24:53.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:53.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:53.764656+0000) 2024-03-06T17:24:54.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:54.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:54.764942+0000) 2024-03-06T17:24:55.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:55.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:55.765102+0000) 2024-03-06T17:24:56.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:56.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:56.765312+0000) 2024-03-06T17:24:57.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:57.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:57.765550+0000) 2024-03-06T17:24:58.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:58.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:58.765834+0000) 2024-03-06T17:24:59.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:24:59.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:59.766071+0000) 2024-03-06T17:25:00.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:00.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:00.766219+0000) 2024-03-06T17:25:01.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:01.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:01.766461+0000) 2024-03-06T17:25:02.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:02.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:02.766712+0000) 2024-03-06T17:25:03.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:03.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:03.766951+0000) 2024-03-06T17:25:04.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:04.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:04.767133+0000) 2024-03-06T17:25:05.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:05.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:05.767376+0000) 2024-03-06T17:25:06.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:06.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:06.767605+0000) 2024-03-06T17:25:07.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:07.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:07.767783+0000) 2024-03-06T17:25:08.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:08.432 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:08.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:08.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:08.767997+0000) 2024-03-06T17:25:09.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:09.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:09.768196+0000) 2024-03-06T17:25:10.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:10.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:10.768432+0000) 2024-03-06T17:25:11.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:11.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:11.768692+0000) 2024-03-06T17:25:12.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:12.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:12.768957+0000) 2024-03-06T17:25:13.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:13.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:13.769124+0000) 2024-03-06T17:25:14.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:14.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:14.769367+0000) 2024-03-06T17:25:15.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:15.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:15.769597+0000) 2024-03-06T17:25:16.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:16.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:16.769839+0000) 2024-03-06T17:25:17.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:17.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:17.770051+0000) 2024-03-06T17:25:18.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:18.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:18.770272+0000) 2024-03-06T17:25:19.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:19.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:19.770506+0000) 2024-03-06T17:25:20.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:20.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:20.770753+0000) 2024-03-06T17:25:21.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:21.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:21.771005+0000) 2024-03-06T17:25:22.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:22.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:22.771222+0000) 2024-03-06T17:25:23.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:23.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:23.771449+0000) 2024-03-06T17:25:24.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:24.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:24.771641+0000) 2024-03-06T17:25:25.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:25.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:25.771897+0000) 2024-03-06T17:25:26.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:26.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:26.772099+0000) 2024-03-06T17:25:27.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:27.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:27.772300+0000) 2024-03-06T17:25:28.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:28.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:28.772540+0000) 2024-03-06T17:25:29.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:29.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:29.772808+0000) 2024-03-06T17:25:30.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:30.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:30.772998+0000) 2024-03-06T17:25:31.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:31.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:31.773238+0000) 2024-03-06T17:25:32.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:32.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:32.773455+0000) 2024-03-06T17:25:33.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:33.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:33.773693+0000) 2024-03-06T17:25:34.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:34.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:34.773893+0000) 2024-03-06T17:25:35.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:35.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:35.774133+0000) 2024-03-06T17:25:36.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:36.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:36.774355+0000) 2024-03-06T17:25:37.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:37.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:37.774538+0000) 2024-03-06T17:25:38.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:38.636 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:38.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:38.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:38.774700+0000) 2024-03-06T17:25:39.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:39.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:39.774934+0000) 2024-03-06T17:25:40.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:40.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:40.775123+0000) 2024-03-06T17:25:41.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:41.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:41.775357+0000) 2024-03-06T17:25:42.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:42.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:42.775495+0000) 2024-03-06T17:25:43.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:43.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:43.775780+0000) 2024-03-06T17:25:44.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:44.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:44.775964+0000) 2024-03-06T17:25:45.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:45.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:45.776177+0000) 2024-03-06T17:25:46.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:46.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:46.776503+0000) 2024-03-06T17:25:47.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:47.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:47.776717+0000) 2024-03-06T17:25:48.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:48.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:48.776990+0000) 2024-03-06T17:25:49.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:49.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:49.777263+0000) 2024-03-06T17:25:50.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:50.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:50.777538+0000) 2024-03-06T17:25:51.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:51.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:51.777833+0000) 2024-03-06T17:25:52.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:52.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:52.778067+0000) 2024-03-06T17:25:53.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:53.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:53.778347+0000) 2024-03-06T17:25:54.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:54.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:54.778575+0000) 2024-03-06T17:25:55.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:55.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:55.778830+0000) 2024-03-06T17:25:56.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:56.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:56.779053+0000) 2024-03-06T17:25:57.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:57.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:57.779264+0000) 2024-03-06T17:25:58.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:58.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:58.779494+0000) 2024-03-06T17:25:59.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:25:59.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:59.779800+0000) 2024-03-06T17:26:00.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:00.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:00.780031+0000) 2024-03-06T17:26:01.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:01.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:01.780303+0000) 2024-03-06T17:26:02.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:02.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:02.780521+0000) 2024-03-06T17:26:03.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:03.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:03.780837+0000) 2024-03-06T17:26:04.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:04.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:04.781098+0000) 2024-03-06T17:26:05.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:05.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:05.781330+0000) 2024-03-06T17:26:06.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:06.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:06.781561+0000) 2024-03-06T17:26:07.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:07.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:07.781799+0000) 2024-03-06T17:26:08.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:08.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:08.782061+0000) 2024-03-06T17:26:08.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:08.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:09.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:09.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:09.782313+0000) 2024-03-06T17:26:10.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:10.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:10.782494+0000) 2024-03-06T17:26:11.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:11.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:11.782767+0000) 2024-03-06T17:26:12.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:12.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:12.782996+0000) 2024-03-06T17:26:13.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:13.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:13.783178+0000) 2024-03-06T17:26:14.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:14.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:14.783361+0000) 2024-03-06T17:26:15.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:15.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:15.783606+0000) 2024-03-06T17:26:16.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:16.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:16.783787+0000) 2024-03-06T17:26:17.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:17.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:17.783968+0000) 2024-03-06T17:26:18.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:18.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:18.784163+0000) 2024-03-06T17:26:19.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:19.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:19.784400+0000) 2024-03-06T17:26:20.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:20.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:20.784631+0000) 2024-03-06T17:26:21.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:21.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:21.784905+0000) 2024-03-06T17:26:22.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:22.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:22.785099+0000) 2024-03-06T17:26:23.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:23.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:23.785348+0000) 2024-03-06T17:26:24.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:24.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:24.785573+0000) 2024-03-06T17:26:25.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:25.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:25.785838+0000) 2024-03-06T17:26:26.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:26.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:26.786027+0000) 2024-03-06T17:26:27.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:27.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:27.786211+0000) 2024-03-06T17:26:28.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:28.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:28.786438+0000) 2024-03-06T17:26:29.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:29.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:29.786694+0000) 2024-03-06T17:26:30.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:30.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:30.786894+0000) 2024-03-06T17:26:31.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:31.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:31.787093+0000) 2024-03-06T17:26:32.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:32.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:32.787308+0000) 2024-03-06T17:26:33.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:33.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:33.787498+0000) 2024-03-06T17:26:34.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:34.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:34.787816+0000) 2024-03-06T17:26:35.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:35.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:35.788053+0000) 2024-03-06T17:26:36.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:36.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:36.788241+0000) 2024-03-06T17:26:37.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:37.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:37.788421+0000) 2024-03-06T17:26:38.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:38.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:38.788654+0000) 2024-03-06T17:26:39.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:39.029 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:39.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:39.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:39.788946+0000) 2024-03-06T17:26:40.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:40.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:40.789134+0000) 2024-03-06T17:26:41.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:41.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:41.789328+0000) 2024-03-06T17:26:42.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:42.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:42.789503+0000) 2024-03-06T17:26:43.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:43.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:43.789767+0000) 2024-03-06T17:26:44.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:44.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:44.789967+0000) 2024-03-06T17:26:45.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:45.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:45.790195+0000) 2024-03-06T17:26:46.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:46.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:46.790417+0000) 2024-03-06T17:26:47.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:47.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:47.790713+0000) 2024-03-06T17:26:48.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:48.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:48.790952+0000) 2024-03-06T17:26:49.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:49.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:49.791139+0000) 2024-03-06T17:26:50.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:50.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:50.791364+0000) 2024-03-06T17:26:51.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:51.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:51.791602+0000) 2024-03-06T17:26:52.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:52.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:52.791815+0000) 2024-03-06T17:26:53.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:53.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:53.792016+0000) 2024-03-06T17:26:54.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:54.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:54.792751+0000) 2024-03-06T17:26:55.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:55.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:55.793001+0000) 2024-03-06T17:26:56.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:56.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:56.793225+0000) 2024-03-06T17:26:57.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:57.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:57.793450+0000) 2024-03-06T17:26:58.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:58.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:58.793737+0000) 2024-03-06T17:26:59.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:26:59.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:59.793964+0000) 2024-03-06T17:27:00.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:00.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:00.794203+0000) 2024-03-06T17:27:01.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:01.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:01.794454+0000) 2024-03-06T17:27:02.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:02.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:02.794722+0000) 2024-03-06T17:27:03.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:03.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:03.794951+0000) 2024-03-06T17:27:04.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:04.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:04.795140+0000) 2024-03-06T17:27:05.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:05.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:05.795403+0000) 2024-03-06T17:27:06.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:06.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:06.795592+0000) 2024-03-06T17:27:07.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:07.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:07.795842+0000) 2024-03-06T17:27:08.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:08.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:08.796079+0000) 2024-03-06T17:27:09.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:09.232 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:09.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:09.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:09.796297+0000) 2024-03-06T17:27:10.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:10.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:10.796516+0000) 2024-03-06T17:27:11.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:11.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:11.796801+0000) 2024-03-06T17:27:12.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:12.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:12.797019+0000) 2024-03-06T17:27:13.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:13.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:13.797249+0000) 2024-03-06T17:27:14.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:14.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:14.797431+0000) 2024-03-06T17:27:15.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:15.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:15.797634+0000) 2024-03-06T17:27:16.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:16.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:16.797854+0000) 2024-03-06T17:27:17.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:17.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:17.798073+0000) 2024-03-06T17:27:18.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:18.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:18.798295+0000) 2024-03-06T17:27:19.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:19.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:19.798519+0000) 2024-03-06T17:27:20.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:20.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:20.798782+0000) 2024-03-06T17:27:21.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:21.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:21.799148+0000) 2024-03-06T17:27:22.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:22.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:22.799333+0000) 2024-03-06T17:27:23.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:23.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:23.799529+0000) 2024-03-06T17:27:24.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:24.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:24.799790+0000) 2024-03-06T17:27:25.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:25.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:25.799999+0000) 2024-03-06T17:27:26.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:26.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:26.800188+0000) 2024-03-06T17:27:27.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:27.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:27.800636+0000) 2024-03-06T17:27:28.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:28.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:28.800876+0000) 2024-03-06T17:27:29.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:29.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:29.801082+0000) 2024-03-06T17:27:30.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:30.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:30.801292+0000) 2024-03-06T17:27:31.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:31.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:31.801500+0000) 2024-03-06T17:27:32.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:32.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:32.801897+0000) 2024-03-06T17:27:33.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:33.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:33.802105+0000) 2024-03-06T17:27:34.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:34.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:34.802333+0000) 2024-03-06T17:27:35.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:35.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:35.802577+0000) 2024-03-06T17:27:36.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:36.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:36.802775+0000) 2024-03-06T17:27:37.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:37.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:37.802999+0000) 2024-03-06T17:27:38.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:38.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:38.803430+0000) 2024-03-06T17:27:39.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:39.424 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:39.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:39.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:39.803733+0000) 2024-03-06T17:27:40.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:40.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:40.803944+0000) 2024-03-06T17:27:41.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:41.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:41.804146+0000) 2024-03-06T17:27:42.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:42.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:42.804397+0000) 2024-03-06T17:27:43.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:43.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:43.804642+0000) 2024-03-06T17:27:44.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:44.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:44.804848+0000) 2024-03-06T17:27:45.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:45.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:45.805109+0000) 2024-03-06T17:27:46.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:46.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:46.805316+0000) 2024-03-06T17:27:47.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:47.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:47.805528+0000) 2024-03-06T17:27:48.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:48.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:48.805796+0000) 2024-03-06T17:27:49.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:49.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:49.806023+0000) 2024-03-06T17:27:50.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:50.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:50.806242+0000) 2024-03-06T17:27:51.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:51.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:51.806483+0000) 2024-03-06T17:27:52.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:52.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:52.806758+0000) 2024-03-06T17:27:53.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:53.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:53.806921+0000) 2024-03-06T17:27:54.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:54.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:54.807091+0000) 2024-03-06T17:27:55.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:55.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:55.807286+0000) 2024-03-06T17:27:56.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:56.805+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:56.807511+0000) 2024-03-06T17:27:57.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:57.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:57.807756+0000) 2024-03-06T17:27:58.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:58.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:58.807949+0000) 2024-03-06T17:27:59.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:27:59.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:59.808155+0000) 2024-03-06T17:28:00.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:00.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:00.808379+0000) 2024-03-06T17:28:01.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:01.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:01.808574+0000) 2024-03-06T17:28:02.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:02.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:02.808768+0000) 2024-03-06T17:28:03.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:03.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:03.809009+0000) 2024-03-06T17:28:04.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:04.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:04.809202+0000) 2024-03-06T17:28:05.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:05.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:05.809425+0000) 2024-03-06T17:28:06.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:06.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:06.809644+0000) 2024-03-06T17:28:07.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:07.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:07.809860+0000) 2024-03-06T17:28:08.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:08.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:08.810053+0000) 2024-03-06T17:28:09.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:09.630 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:09.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:09.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:09.810266+0000) 2024-03-06T17:28:10.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:10.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:10.810449+0000) 2024-03-06T17:28:11.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:11.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:11.810719+0000) 2024-03-06T17:28:12.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:12.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:12.810889+0000) 2024-03-06T17:28:13.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:13.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:13.811119+0000) 2024-03-06T17:28:14.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:14.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:14.811296+0000) 2024-03-06T17:28:15.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:15.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:15.811584+0000) 2024-03-06T17:28:16.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:16.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:16.811780+0000) 2024-03-06T17:28:17.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:17.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:17.811946+0000) 2024-03-06T17:28:18.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:18.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:18.812127+0000) 2024-03-06T17:28:19.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:19.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:19.812383+0000) 2024-03-06T17:28:20.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:20.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:20.812702+0000) 2024-03-06T17:28:21.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:21.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:21.812942+0000) 2024-03-06T17:28:22.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:22.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:22.813122+0000) 2024-03-06T17:28:23.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:23.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:23.813352+0000) 2024-03-06T17:28:24.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:24.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:24.813574+0000) 2024-03-06T17:28:25.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:25.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:25.813830+0000) 2024-03-06T17:28:26.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:26.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:26.814072+0000) 2024-03-06T17:28:27.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:27.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:27.814230+0000) 2024-03-06T17:28:28.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:28.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:28.814404+0000) 2024-03-06T17:28:29.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:29.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:29.814628+0000) 2024-03-06T17:28:30.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:30.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:30.814812+0000) 2024-03-06T17:28:31.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:31.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:31.815007+0000) 2024-03-06T17:28:32.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:32.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:32.815226+0000) 2024-03-06T17:28:33.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:33.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:33.815449+0000) 2024-03-06T17:28:34.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:34.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:34.815688+0000) 2024-03-06T17:28:35.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:35.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:35.815891+0000) 2024-03-06T17:28:36.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:36.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:36.816084+0000) 2024-03-06T17:28:37.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:37.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:37.816327+0000) 2024-03-06T17:28:38.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:38.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:38.816520+0000) 2024-03-06T17:28:39.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:39.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:39.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:39.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:39.816761+0000) 2024-03-06T17:28:40.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:40.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:40.816935+0000) 2024-03-06T17:28:41.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:41.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:41.817126+0000) 2024-03-06T17:28:42.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:42.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:42.817333+0000) 2024-03-06T17:28:43.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:43.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:43.817614+0000) 2024-03-06T17:28:44.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:44.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:44.817865+0000) 2024-03-06T17:28:45.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:45.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:45.818110+0000) 2024-03-06T17:28:46.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:46.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:46.818394+0000) 2024-03-06T17:28:47.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:47.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:47.818689+0000) 2024-03-06T17:28:48.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:48.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:48.818968+0000) 2024-03-06T17:28:49.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:49.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:49.819251+0000) 2024-03-06T17:28:50.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:50.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:50.819502+0000) 2024-03-06T17:28:51.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:51.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:51.819786+0000) 2024-03-06T17:28:52.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:52.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:52.819999+0000) 2024-03-06T17:28:53.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:53.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:53.820230+0000) 2024-03-06T17:28:54.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:54.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:54.820432+0000) 2024-03-06T17:28:55.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:55.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:55.820756+0000) 2024-03-06T17:28:56.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:56.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:56.820999+0000) 2024-03-06T17:28:57.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:57.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:57.821238+0000) 2024-03-06T17:28:58.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:58.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:58.821455+0000) 2024-03-06T17:28:59.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:28:59.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:59.821719+0000) 2024-03-06T17:29:00.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:00.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:00.821914+0000) 2024-03-06T17:29:01.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:01.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:01.822140+0000) 2024-03-06T17:29:02.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:02.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:02.822326+0000) 2024-03-06T17:29:03.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:03.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:03.822578+0000) 2024-03-06T17:29:04.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:04.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:04.822766+0000) 2024-03-06T17:29:05.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:05.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:05.822952+0000) 2024-03-06T17:29:06.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:06.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:06.823191+0000) 2024-03-06T17:29:07.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:07.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:07.823454+0000) 2024-03-06T17:29:08.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:08.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:08.823759+0000) 2024-03-06T17:29:09.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:09.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:09.823969+0000) 2024-03-06T17:29:10.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:10.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:10.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:10.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:10.824183+0000) 2024-03-06T17:29:11.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:11.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:11.824417+0000) 2024-03-06T17:29:12.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:12.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:12.824644+0000) 2024-03-06T17:29:13.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:13.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:13.824818+0000) 2024-03-06T17:29:14.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:14.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:14.825006+0000) 2024-03-06T17:29:15.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:15.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:15.825203+0000) 2024-03-06T17:29:16.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:16.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:16.825435+0000) 2024-03-06T17:29:17.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:17.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:17.825648+0000) 2024-03-06T17:29:18.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:18.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:18.825962+0000) 2024-03-06T17:29:19.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:19.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:19.826210+0000) 2024-03-06T17:29:20.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:20.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:20.826428+0000) 2024-03-06T17:29:21.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:21.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:21.826634+0000) 2024-03-06T17:29:22.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:22.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:22.826816+0000) 2024-03-06T17:29:23.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:23.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:23.827006+0000) 2024-03-06T17:29:24.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:24.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:24.827234+0000) 2024-03-06T17:29:25.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:25.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:25.827519+0000) 2024-03-06T17:29:26.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:26.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:26.827812+0000) 2024-03-06T17:29:27.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:27.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:27.828055+0000) 2024-03-06T17:29:28.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:28.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:28.828247+0000) 2024-03-06T17:29:29.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:29.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:29.828472+0000) 2024-03-06T17:29:30.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:30.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:30.828722+0000) 2024-03-06T17:29:31.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:31.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:31.828945+0000) 2024-03-06T17:29:32.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:32.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:32.829170+0000) 2024-03-06T17:29:33.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:33.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:33.829399+0000) 2024-03-06T17:29:34.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:34.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:34.829620+0000) 2024-03-06T17:29:35.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:35.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:35.829849+0000) 2024-03-06T17:29:36.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:36.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:36.830136+0000) 2024-03-06T17:29:37.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:37.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:37.830334+0000) 2024-03-06T17:29:38.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:38.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:38.830544+0000) 2024-03-06T17:29:39.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:39.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:39.830796+0000) 2024-03-06T17:29:40.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:40.237 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:40.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:40.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:40.831023+0000) 2024-03-06T17:29:41.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:41.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:41.831212+0000) 2024-03-06T17:29:42.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:42.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:42.831408+0000) 2024-03-06T17:29:43.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:43.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:43.831650+0000) 2024-03-06T17:29:44.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:44.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:44.831891+0000) 2024-03-06T17:29:45.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:45.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:45.832140+0000) 2024-03-06T17:29:46.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:46.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:46.832337+0000) 2024-03-06T17:29:47.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:47.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:47.832553+0000) 2024-03-06T17:29:48.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:48.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:48.832773+0000) 2024-03-06T17:29:49.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:49.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:49.832993+0000) 2024-03-06T17:29:50.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:50.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:50.833212+0000) 2024-03-06T17:29:51.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:51.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:51.833457+0000) 2024-03-06T17:29:52.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:52.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:52.833705+0000) 2024-03-06T17:29:53.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:53.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:53.833903+0000) 2024-03-06T17:29:54.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:54.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:54.834105+0000) 2024-03-06T17:29:55.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:55.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:55.834278+0000) 2024-03-06T17:29:56.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:56.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:56.834553+0000) 2024-03-06T17:29:57.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:57.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:57.834836+0000) 2024-03-06T17:29:58.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:58.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:58.835046+0000) 2024-03-06T17:29:59.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:29:59.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:59.835265+0000) 2024-03-06T17:30:00.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:00.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:00.835488+0000) 2024-03-06T17:30:01.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:01.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:01.835709+0000) 2024-03-06T17:30:02.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:02.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:02.835946+0000) 2024-03-06T17:30:03.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:03.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:03.836128+0000) 2024-03-06T17:30:04.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:04.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:04.836352+0000) 2024-03-06T17:30:05.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:05.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:05.836560+0000) 2024-03-06T17:30:06.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:06.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:06.836780+0000) 2024-03-06T17:30:07.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:07.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:07.837009+0000) 2024-03-06T17:30:08.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:08.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:08.837250+0000) 2024-03-06T17:30:09.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:09.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:09.837500+0000) 2024-03-06T17:30:10.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:10.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:10.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:10.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:10.837735+0000) 2024-03-06T17:30:11.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:11.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:11.837936+0000) 2024-03-06T17:30:12.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:12.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:12.838116+0000) 2024-03-06T17:30:13.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:13.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:13.838349+0000) 2024-03-06T17:30:14.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:14.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:14.838575+0000) 2024-03-06T17:30:15.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:15.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:15.838805+0000) 2024-03-06T17:30:16.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:16.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:16.838981+0000) 2024-03-06T17:30:17.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:17.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:17.839159+0000) 2024-03-06T17:30:18.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:18.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:18.839386+0000) 2024-03-06T17:30:19.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:19.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:19.839584+0000) 2024-03-06T17:30:20.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:20.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:20.839832+0000) 2024-03-06T17:30:21.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:21.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:21.840072+0000) 2024-03-06T17:30:22.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:22.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:22.840269+0000) 2024-03-06T17:30:23.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:23.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:23.840512+0000) 2024-03-06T17:30:24.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:24.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:24.840772+0000) 2024-03-06T17:30:25.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:25.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:25.841034+0000) 2024-03-06T17:30:26.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:26.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:26.841279+0000) 2024-03-06T17:30:27.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:27.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:27.841501+0000) 2024-03-06T17:30:28.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:28.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:28.841764+0000) 2024-03-06T17:30:29.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:29.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:29.841999+0000) 2024-03-06T17:30:30.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:30.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:30.842217+0000) 2024-03-06T17:30:31.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:31.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:31.842426+0000) 2024-03-06T17:30:32.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:32.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:32.842804+0000) 2024-03-06T17:30:33.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:33.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:33.843067+0000) 2024-03-06T17:30:34.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:34.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:34.843287+0000) 2024-03-06T17:30:35.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:35.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:35.843479+0000) 2024-03-06T17:30:36.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:36.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:36.843753+0000) 2024-03-06T17:30:37.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:37.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:37.843966+0000) 2024-03-06T17:30:38.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:38.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:38.844188+0000) 2024-03-06T17:30:39.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:39.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:39.844403+0000) 2024-03-06T17:30:40.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:40.696 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:40.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:40.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:40.844622+0000) 2024-03-06T17:30:41.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:41.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:41.844923+0000) 2024-03-06T17:30:42.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:42.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:42.845176+0000) 2024-03-06T17:30:43.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:43.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:43.845388+0000) 2024-03-06T17:30:44.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:44.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:44.845704+0000) 2024-03-06T17:30:45.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:45.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:45.845954+0000) 2024-03-06T17:30:46.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:46.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:46.846184+0000) 2024-03-06T17:30:47.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:47.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:47.846416+0000) 2024-03-06T17:30:48.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:48.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:48.846656+0000) 2024-03-06T17:30:49.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:49.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:49.847054+0000) 2024-03-06T17:30:50.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:50.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:50.847305+0000) 2024-03-06T17:30:51.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:51.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:51.847542+0000) 2024-03-06T17:30:52.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:52.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:52.847770+0000) 2024-03-06T17:30:53.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:53.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:53.847956+0000) 2024-03-06T17:30:54.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:54.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:54.848184+0000) 2024-03-06T17:30:55.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:55.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:55.848409+0000) 2024-03-06T17:30:56.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:56.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:56.848572+0000) 2024-03-06T17:30:57.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:57.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:57.848976+0000) 2024-03-06T17:30:58.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:58.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:58.849210+0000) 2024-03-06T17:30:59.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:30:59.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:59.849442+0000) 2024-03-06T17:31:00.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:00.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:00.849706+0000) 2024-03-06T17:31:01.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:01.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:01.849918+0000) 2024-03-06T17:31:02.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:02.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:02.850086+0000) 2024-03-06T17:31:03.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:03.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:03.850295+0000) 2024-03-06T17:31:04.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:04.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:04.850511+0000) 2024-03-06T17:31:05.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:05.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:05.850741+0000) 2024-03-06T17:31:06.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:06.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:06.850942+0000) 2024-03-06T17:31:07.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:07.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:07.851239+0000) 2024-03-06T17:31:08.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:08.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:08.851464+0000) 2024-03-06T17:31:09.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:09.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:09.851727+0000) 2024-03-06T17:31:10.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:10.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:10.851914+0000) 2024-03-06T17:31:10.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:10.896 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:11.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:11.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:11.852141+0000) 2024-03-06T17:31:12.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:12.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:12.852334+0000) 2024-03-06T17:31:13.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:13.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:13.852587+0000) 2024-03-06T17:31:14.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:14.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:14.852821+0000) 2024-03-06T17:31:15.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:15.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:15.853035+0000) 2024-03-06T17:31:16.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:16.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:16.853272+0000) 2024-03-06T17:31:17.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:17.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:17.853476+0000) 2024-03-06T17:31:18.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:18.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:18.853740+0000) 2024-03-06T17:31:19.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:19.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:19.853937+0000) 2024-03-06T17:31:20.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:20.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:20.854102+0000) 2024-03-06T17:31:21.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:21.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:21.854295+0000) 2024-03-06T17:31:22.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:22.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:22.854483+0000) 2024-03-06T17:31:23.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:23.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:23.854705+0000) 2024-03-06T17:31:24.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:24.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:24.854914+0000) 2024-03-06T17:31:25.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:25.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:25.855116+0000) 2024-03-06T17:31:26.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:26.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:26.855311+0000) 2024-03-06T17:31:27.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:27.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:27.855553+0000) 2024-03-06T17:31:28.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:28.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:28.855788+0000) 2024-03-06T17:31:29.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:29.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:29.855989+0000) 2024-03-06T17:31:30.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:30.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:30.856182+0000) 2024-03-06T17:31:31.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:31.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:31.856420+0000) 2024-03-06T17:31:32.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:32.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:32.856655+0000) 2024-03-06T17:31:33.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:33.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:33.856867+0000) 2024-03-06T17:31:34.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:34.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:34.857060+0000) 2024-03-06T17:31:35.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:35.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:35.857253+0000) 2024-03-06T17:31:36.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:36.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:36.857477+0000) 2024-03-06T17:31:37.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:37.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:37.857719+0000) 2024-03-06T17:31:38.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:38.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:38.857964+0000) 2024-03-06T17:31:39.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:39.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:39.858157+0000) 2024-03-06T17:31:40.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:40.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:40.858385+0000) 2024-03-06T17:31:41.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:41.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:41.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:41.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:41.858627+0000) 2024-03-06T17:31:42.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:42.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:42.858878+0000) 2024-03-06T17:31:43.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:43.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:43.859073+0000) 2024-03-06T17:31:44.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:44.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:44.859266+0000) 2024-03-06T17:31:45.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:45.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:45.859437+0000) 2024-03-06T17:31:46.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:46.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:46.859700+0000) 2024-03-06T17:31:47.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:47.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:47.859921+0000) 2024-03-06T17:31:48.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:48.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:48.860142+0000) 2024-03-06T17:31:49.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:49.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:49.860364+0000) 2024-03-06T17:31:50.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:50.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:50.860566+0000) 2024-03-06T17:31:51.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:51.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:51.860808+0000) 2024-03-06T17:31:52.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:52.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:52.861033+0000) 2024-03-06T17:31:53.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:53.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:53.861217+0000) 2024-03-06T17:31:54.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:54.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:54.861408+0000) 2024-03-06T17:31:55.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:55.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:55.861631+0000) 2024-03-06T17:31:56.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:56.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:56.861822+0000) 2024-03-06T17:31:57.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:57.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:57.862046+0000) 2024-03-06T17:31:58.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:58.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:58.862238+0000) 2024-03-06T17:31:59.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:31:59.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:59.862468+0000) 2024-03-06T17:32:00.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:00.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:00.862707+0000) 2024-03-06T17:32:01.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:01.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:01.862970+0000) 2024-03-06T17:32:02.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:02.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:02.863163+0000) 2024-03-06T17:32:03.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:03.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:03.863356+0000) 2024-03-06T17:32:04.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:04.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:04.863578+0000) 2024-03-06T17:32:05.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:05.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:05.863829+0000) 2024-03-06T17:32:06.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:06.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:06.864029+0000) 2024-03-06T17:32:07.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:07.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:07.864269+0000) 2024-03-06T17:32:08.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:08.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:08.864465+0000) 2024-03-06T17:32:09.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:09.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:09.864718+0000) 2024-03-06T17:32:10.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:10.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:10.864957+0000) 2024-03-06T17:32:11.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:11.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:11.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:11.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:11.865172+0000) 2024-03-06T17:32:12.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:12.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:12.865407+0000) 2024-03-06T17:32:13.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:13.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:13.865631+0000) 2024-03-06T17:32:14.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:14.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:14.865832+0000) 2024-03-06T17:32:15.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:15.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:15.866037+0000) 2024-03-06T17:32:16.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:16.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:16.866275+0000) 2024-03-06T17:32:17.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:17.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:17.866487+0000) 2024-03-06T17:32:18.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:18.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:18.866750+0000) 2024-03-06T17:32:19.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:19.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:19.866902+0000) 2024-03-06T17:32:20.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:20.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:20.867089+0000) 2024-03-06T17:32:21.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:21.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:21.867313+0000) 2024-03-06T17:32:22.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:22.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:22.867534+0000) 2024-03-06T17:32:23.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:23.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:23.867782+0000) 2024-03-06T17:32:24.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:24.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:24.868024+0000) 2024-03-06T17:32:25.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:25.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:25.868218+0000) 2024-03-06T17:32:26.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:26.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:26.868516+0000) 2024-03-06T17:32:27.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:27.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:27.868774+0000) 2024-03-06T17:32:28.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:28.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:28.868980+0000) 2024-03-06T17:32:29.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:29.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:29.869181+0000) 2024-03-06T17:32:30.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:30.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:30.869400+0000) 2024-03-06T17:32:31.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:31.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:31.869656+0000) 2024-03-06T17:32:32.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:32.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:32.869870+0000) 2024-03-06T17:32:33.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:33.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:33.870073+0000) 2024-03-06T17:32:34.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:34.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:34.870249+0000) 2024-03-06T17:32:35.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:35.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:35.870467+0000) 2024-03-06T17:32:36.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:36.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:36.870760+0000) 2024-03-06T17:32:37.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:37.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:37.870990+0000) 2024-03-06T17:32:38.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:38.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:38.871214+0000) 2024-03-06T17:32:39.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:39.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:39.871443+0000) 2024-03-06T17:32:40.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:40.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:40.871700+0000) 2024-03-06T17:32:41.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:41.512 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:41.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:41.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:41.871921+0000) 2024-03-06T17:32:42.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:42.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:42.872142+0000) 2024-03-06T17:32:43.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:43.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:43.872381+0000) 2024-03-06T17:32:44.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:44.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:44.872569+0000) 2024-03-06T17:32:45.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:45.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:45.872783+0000) 2024-03-06T17:32:46.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:46.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:46.872970+0000) 2024-03-06T17:32:47.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:47.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:47.873183+0000) 2024-03-06T17:32:48.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:48.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:48.873403+0000) 2024-03-06T17:32:49.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:49.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:49.873585+0000) 2024-03-06T17:32:50.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:50.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:50.873824+0000) 2024-03-06T17:32:51.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:51.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:51.874073+0000) 2024-03-06T17:32:52.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:52.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:52.874295+0000) 2024-03-06T17:32:53.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:53.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:53.874520+0000) 2024-03-06T17:32:54.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:54.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:54.874792+0000) 2024-03-06T17:32:55.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:55.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:55.875003+0000) 2024-03-06T17:32:56.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:56.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:56.875239+0000) 2024-03-06T17:32:57.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:57.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:57.875411+0000) 2024-03-06T17:32:58.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:58.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:58.875637+0000) 2024-03-06T17:32:59.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:32:59.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:59.875828+0000) 2024-03-06T17:33:00.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:00.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:00.876013+0000) 2024-03-06T17:33:01.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:01.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:01.876266+0000) 2024-03-06T17:33:02.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:02.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:02.876470+0000) 2024-03-06T17:33:03.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:03.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:03.876727+0000) 2024-03-06T17:33:04.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:04.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:04.876936+0000) 2024-03-06T17:33:05.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:05.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:05.877131+0000) 2024-03-06T17:33:06.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:06.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:06.877348+0000) 2024-03-06T17:33:07.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:07.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:07.877546+0000) 2024-03-06T17:33:08.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:08.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:08.877814+0000) 2024-03-06T17:33:09.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:09.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:09.877993+0000) 2024-03-06T17:33:10.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:10.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:10.878177+0000) 2024-03-06T17:33:11.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:11.741 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:11.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:11.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:11.878390+0000) 2024-03-06T17:33:12.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:12.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:12.878608+0000) 2024-03-06T17:33:13.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:13.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:13.878827+0000) 2024-03-06T17:33:14.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:14.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:14.879081+0000) 2024-03-06T17:33:15.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:15.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:15.879312+0000) 2024-03-06T17:33:16.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:16.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:16.879525+0000) 2024-03-06T17:33:17.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:17.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:17.879781+0000) 2024-03-06T17:33:18.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:18.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:18.879971+0000) 2024-03-06T17:33:19.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:19.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:19.880216+0000) 2024-03-06T17:33:20.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:20.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:20.880441+0000) 2024-03-06T17:33:21.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:21.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:21.880734+0000) 2024-03-06T17:33:22.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:22.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:22.880915+0000) 2024-03-06T17:33:23.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:23.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:23.881127+0000) 2024-03-06T17:33:24.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:24.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:24.881363+0000) 2024-03-06T17:33:25.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:25.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:25.881604+0000) 2024-03-06T17:33:26.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:26.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:26.881860+0000) 2024-03-06T17:33:27.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:27.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:27.882104+0000) 2024-03-06T17:33:28.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:28.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:28.882347+0000) 2024-03-06T17:33:29.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:29.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:29.882591+0000) 2024-03-06T17:33:30.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:30.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:30.882822+0000) 2024-03-06T17:33:31.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:31.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:31.883083+0000) 2024-03-06T17:33:32.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:32.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:32.883327+0000) 2024-03-06T17:33:33.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:33.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:33.883521+0000) 2024-03-06T17:33:34.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:34.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:34.883783+0000) 2024-03-06T17:33:35.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:35.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:35.884028+0000) 2024-03-06T17:33:36.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:36.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:36.884266+0000) 2024-03-06T17:33:37.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:37.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:37.884503+0000) 2024-03-06T17:33:38.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:38.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:38.884782+0000) 2024-03-06T17:33:39.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:39.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:39.884995+0000) 2024-03-06T17:33:40.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:40.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:40.885203+0000) 2024-03-06T17:33:41.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:41.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:41.885449+0000) 2024-03-06T17:33:41.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:41.961 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:42.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:42.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:42.885735+0000) 2024-03-06T17:33:43.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:43.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:43.885977+0000) 2024-03-06T17:33:44.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:44.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:44.886205+0000) 2024-03-06T17:33:45.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:45.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:45.886430+0000) 2024-03-06T17:33:46.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:46.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:46.886700+0000) 2024-03-06T17:33:47.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:47.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:47.886870+0000) 2024-03-06T17:33:48.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:48.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:48.887062+0000) 2024-03-06T17:33:49.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:49.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:49.887298+0000) 2024-03-06T17:33:50.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:50.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:50.887521+0000) 2024-03-06T17:33:51.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:51.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:51.887769+0000) 2024-03-06T17:33:52.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:52.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:52.887955+0000) 2024-03-06T17:33:53.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:53.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:53.888148+0000) 2024-03-06T17:33:54.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:54.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:54.888372+0000) 2024-03-06T17:33:55.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:55.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:55.888597+0000) 2024-03-06T17:33:56.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:56.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:56.888839+0000) 2024-03-06T17:33:57.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:57.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:57.889014+0000) 2024-03-06T17:33:58.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:58.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:58.889212+0000) 2024-03-06T17:33:59.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:33:59.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:59.889429+0000) 2024-03-06T17:34:00.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:00.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:00.889710+0000) 2024-03-06T17:34:01.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:01.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:01.889958+0000) 2024-03-06T17:34:02.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:02.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:02.890160+0000) 2024-03-06T17:34:03.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:03.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:03.890382+0000) 2024-03-06T17:34:04.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:04.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:04.890587+0000) 2024-03-06T17:34:05.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:05.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:05.890820+0000) 2024-03-06T17:34:06.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:06.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:06.891039+0000) 2024-03-06T17:34:07.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:07.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:07.891275+0000) 2024-03-06T17:34:08.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:08.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:08.891486+0000) 2024-03-06T17:34:09.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:09.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:09.891755+0000) 2024-03-06T17:34:10.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:10.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:10.891977+0000) 2024-03-06T17:34:11.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:11.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:11.892222+0000) 2024-03-06T17:34:12.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:12.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:12.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:12.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:12.892451+0000) 2024-03-06T17:34:13.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:13.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:13.892631+0000) 2024-03-06T17:34:14.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:14.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:14.892838+0000) 2024-03-06T17:34:15.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:15.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:15.893029+0000) 2024-03-06T17:34:16.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:16.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:16.893232+0000) 2024-03-06T17:34:17.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:17.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:17.893434+0000) 2024-03-06T17:34:18.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:18.892+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:18.893595+0000) 2024-03-06T17:34:19.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:19.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:19.893826+0000) 2024-03-06T17:34:20.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:20.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:20.894044+0000) 2024-03-06T17:34:21.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:21.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:21.894287+0000) 2024-03-06T17:34:22.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:22.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:22.894505+0000) 2024-03-06T17:34:23.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:23.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:23.894774+0000) 2024-03-06T17:34:24.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:24.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:24.894966+0000) 2024-03-06T17:34:25.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:25.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:25.895192+0000) 2024-03-06T17:34:26.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:26.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:26.895404+0000) 2024-03-06T17:34:27.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:27.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:27.895627+0000) 2024-03-06T17:34:28.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:28.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:28.895833+0000) 2024-03-06T17:34:29.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:29.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:29.896055+0000) 2024-03-06T17:34:30.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:30.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:30.896222+0000) 2024-03-06T17:34:31.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:31.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:31.896463+0000) 2024-03-06T17:34:32.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:32.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:32.896641+0000) 2024-03-06T17:34:33.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:33.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:33.896920+0000) 2024-03-06T17:34:34.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:34.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:34.897177+0000) 2024-03-06T17:34:35.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:35.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:35.897430+0000) 2024-03-06T17:34:36.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:36.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:36.897629+0000) 2024-03-06T17:34:37.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:37.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:37.897905+0000) 2024-03-06T17:34:38.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:38.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:38.898111+0000) 2024-03-06T17:34:39.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:39.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:39.898354+0000) 2024-03-06T17:34:40.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:40.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:40.898591+0000) 2024-03-06T17:34:41.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:41.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:41.898882+0000) 2024-03-06T17:34:42.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:42.370 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:42.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:42.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:42.899087+0000) 2024-03-06T17:34:43.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:43.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:43.899299+0000) 2024-03-06T17:34:44.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:44.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:44.899485+0000) 2024-03-06T17:34:45.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:45.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:45.899756+0000) 2024-03-06T17:34:46.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:46.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:46.899991+0000) 2024-03-06T17:34:47.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:47.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:47.900221+0000) 2024-03-06T17:34:48.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:48.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:48.900489+0000) 2024-03-06T17:34:49.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:49.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:49.900794+0000) 2024-03-06T17:34:50.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:50.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:50.901030+0000) 2024-03-06T17:34:51.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:51.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:51.901307+0000) 2024-03-06T17:34:52.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:52.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:52.901588+0000) 2024-03-06T17:34:53.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:53.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:53.901806+0000) 2024-03-06T17:34:54.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:54.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:54.902011+0000) 2024-03-06T17:34:55.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:55.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:55.902267+0000) 2024-03-06T17:34:56.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:56.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:56.902545+0000) 2024-03-06T17:34:57.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:57.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:57.902836+0000) 2024-03-06T17:34:58.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:58.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:58.903001+0000) 2024-03-06T17:34:59.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:34:59.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:59.903228+0000) 2024-03-06T17:35:00.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:00.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:00.903512+0000) 2024-03-06T17:35:01.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:01.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:01.903823+0000) 2024-03-06T17:35:02.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:02.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:02.904070+0000) 2024-03-06T17:35:03.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:03.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:03.904316+0000) 2024-03-06T17:35:04.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:04.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:04.904595+0000) 2024-03-06T17:35:05.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:05.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:05.904813+0000) 2024-03-06T17:35:06.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:06.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:06.905026+0000) 2024-03-06T17:35:07.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:07.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:07.905236+0000) 2024-03-06T17:35:08.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:08.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:08.905491+0000) 2024-03-06T17:35:09.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:09.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:09.905798+0000) 2024-03-06T17:35:10.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:10.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:10.905997+0000) 2024-03-06T17:35:11.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:11.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:11.906200+0000) 2024-03-06T17:35:12.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:12.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:12.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:12.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:12.906424+0000) 2024-03-06T17:35:13.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:13.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:13.906621+0000) 2024-03-06T17:35:14.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:14.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:14.906869+0000) 2024-03-06T17:35:15.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:15.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:15.907112+0000) 2024-03-06T17:35:16.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:16.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:16.907379+0000) 2024-03-06T17:35:17.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:17.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:17.907658+0000) 2024-03-06T17:35:18.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:18.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:18.907888+0000) 2024-03-06T17:35:19.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:19.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:19.908114+0000) 2024-03-06T17:35:20.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:20.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:20.908332+0000) 2024-03-06T17:35:21.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:21.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:21.908575+0000) 2024-03-06T17:35:22.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:22.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:22.908806+0000) 2024-03-06T17:35:23.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:23.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:23.909030+0000) 2024-03-06T17:35:24.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:24.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:24.909226+0000) 2024-03-06T17:35:25.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:25.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:25.909427+0000) 2024-03-06T17:35:26.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:26.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:26.909704+0000) 2024-03-06T17:35:27.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:27.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:27.909906+0000) 2024-03-06T17:35:28.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:28.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:28.910138+0000) 2024-03-06T17:35:29.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:29.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:29.910321+0000) 2024-03-06T17:35:30.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:30.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:30.910502+0000) 2024-03-06T17:35:31.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:31.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:31.910780+0000) 2024-03-06T17:35:32.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:32.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:32.910975+0000) 2024-03-06T17:35:33.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:33.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:33.911176+0000) 2024-03-06T17:35:34.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:34.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:34.911395+0000) 2024-03-06T17:35:35.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:35.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:35.911622+0000) 2024-03-06T17:35:36.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:36.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:36.911817+0000) 2024-03-06T17:35:37.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:37.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:37.912006+0000) 2024-03-06T17:35:38.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:38.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:38.912206+0000) 2024-03-06T17:35:39.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:39.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:39.912389+0000) 2024-03-06T17:35:40.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:40.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:40.912609+0000) 2024-03-06T17:35:41.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:41.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:41.912816+0000) 2024-03-06T17:35:42.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:42.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:42.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:42.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:42.913012+0000) 2024-03-06T17:35:43.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:43.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:43.913291+0000) 2024-03-06T17:35:44.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:44.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:44.913561+0000) 2024-03-06T17:35:45.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:45.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:45.913848+0000) 2024-03-06T17:35:46.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:46.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:46.914069+0000) 2024-03-06T17:35:47.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:47.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:47.914251+0000) 2024-03-06T17:35:48.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:48.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:48.914480+0000) 2024-03-06T17:35:49.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:49.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:49.914745+0000) 2024-03-06T17:35:50.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:50.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:50.915022+0000) 2024-03-06T17:35:51.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:51.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:51.915292+0000) 2024-03-06T17:35:52.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:52.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:52.915473+0000) 2024-03-06T17:35:53.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:53.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:53.915657+0000) 2024-03-06T17:35:54.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:54.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:54.915880+0000) 2024-03-06T17:35:55.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:55.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:55.916063+0000) 2024-03-06T17:35:56.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:56.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:56.916253+0000) 2024-03-06T17:35:57.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:57.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:57.916472+0000) 2024-03-06T17:35:58.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:58.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:58.916776+0000) 2024-03-06T17:35:59.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:35:59.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:59.917015+0000) 2024-03-06T17:36:00.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:00.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:00.917215+0000) 2024-03-06T17:36:01.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:01.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:01.917449+0000) 2024-03-06T17:36:02.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:02.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:02.917697+0000) 2024-03-06T17:36:03.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:03.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:03.917899+0000) 2024-03-06T17:36:04.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:04.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:04.918092+0000) 2024-03-06T17:36:05.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:05.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:05.918270+0000) 2024-03-06T17:36:06.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:06.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:06.918493+0000) 2024-03-06T17:36:07.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:07.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:07.918757+0000) 2024-03-06T17:36:08.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:08.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:08.918978+0000) 2024-03-06T17:36:09.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:09.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:09.919198+0000) 2024-03-06T17:36:10.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:10.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:10.919417+0000) 2024-03-06T17:36:11.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:11.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:11.919651+0000) 2024-03-06T17:36:12.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:12.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:12.919859+0000) 2024-03-06T17:36:12.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:12.970 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:13.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:13.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:13.920041+0000) 2024-03-06T17:36:14.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:14.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:14.920234+0000) 2024-03-06T17:36:15.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:15.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:15.920464+0000) 2024-03-06T17:36:16.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:16.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:16.920719+0000) 2024-03-06T17:36:17.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:17.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:17.920913+0000) 2024-03-06T17:36:18.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:18.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:18.921113+0000) 2024-03-06T17:36:19.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:19.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:19.921366+0000) 2024-03-06T17:36:20.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:20.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:20.921648+0000) 2024-03-06T17:36:21.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:21.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:21.921904+0000) 2024-03-06T17:36:22.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:22.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:22.922127+0000) 2024-03-06T17:36:23.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:23.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:23.922322+0000) 2024-03-06T17:36:24.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:24.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:24.922529+0000) 2024-03-06T17:36:25.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:25.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:25.922815+0000) 2024-03-06T17:36:26.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:26.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:26.923074+0000) 2024-03-06T17:36:27.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:27.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:27.923365+0000) 2024-03-06T17:36:28.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:28.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:28.923649+0000) 2024-03-06T17:36:29.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:29.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:29.923915+0000) 2024-03-06T17:36:30.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:30.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:30.924359+0000) 2024-03-06T17:36:31.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:31.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:31.924624+0000) 2024-03-06T17:36:32.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:32.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:32.924810+0000) 2024-03-06T17:36:33.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:33.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:33.925014+0000) 2024-03-06T17:36:34.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:34.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:34.925242+0000) 2024-03-06T17:36:35.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:35.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:35.925470+0000) 2024-03-06T17:36:36.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:36.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:36.925776+0000) 2024-03-06T17:36:37.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:37.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:37.925973+0000) 2024-03-06T17:36:38.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:38.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:38.926230+0000) 2024-03-06T17:36:39.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:39.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:39.926498+0000) 2024-03-06T17:36:40.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:40.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:40.926795+0000) 2024-03-06T17:36:41.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:41.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:41.927058+0000) 2024-03-06T17:36:42.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:42.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:42.927282+0000) 2024-03-06T17:36:43.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:43.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:43.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:43.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:43.927516+0000) 2024-03-06T17:36:44.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:44.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:44.927830+0000) 2024-03-06T17:36:45.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:45.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:45.928103+0000) 2024-03-06T17:36:46.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:46.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:46.928339+0000) 2024-03-06T17:36:47.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:47.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:47.928561+0000) 2024-03-06T17:36:48.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:48.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:48.928777+0000) 2024-03-06T17:36:49.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:49.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:49.929005+0000) 2024-03-06T17:36:50.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:50.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:50.929224+0000) 2024-03-06T17:36:51.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:51.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:51.929479+0000) 2024-03-06T17:36:52.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:52.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:52.929784+0000) 2024-03-06T17:36:53.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:53.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:53.930064+0000) 2024-03-06T17:36:54.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:54.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:54.930340+0000) 2024-03-06T17:36:55.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:55.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:55.930621+0000) 2024-03-06T17:36:56.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:56.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:56.930901+0000) 2024-03-06T17:36:57.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:57.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:57.931141+0000) 2024-03-06T17:36:58.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:58.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:58.931371+0000) 2024-03-06T17:36:59.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:36:59.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:59.931590+0000) 2024-03-06T17:37:00.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:00.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:00.931821+0000) 2024-03-06T17:37:01.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:01.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:01.932061+0000) 2024-03-06T17:37:02.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:02.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:02.932292+0000) 2024-03-06T17:37:03.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:03.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:03.932494+0000) 2024-03-06T17:37:04.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:04.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:04.932808+0000) 2024-03-06T17:37:05.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:05.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:05.933087+0000) 2024-03-06T17:37:06.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:06.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:06.933379+0000) 2024-03-06T17:37:07.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:07.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:07.933583+0000) 2024-03-06T17:37:08.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:08.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:08.933827+0000) 2024-03-06T17:37:09.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:09.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:09.934058+0000) 2024-03-06T17:37:10.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:10.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:10.934276+0000) 2024-03-06T17:37:11.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:11.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:11.934502+0000) 2024-03-06T17:37:12.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:12.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:12.934753+0000) 2024-03-06T17:37:13.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:13.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:13.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:13.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:13.934980+0000) 2024-03-06T17:37:14.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:14.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:14.935204+0000) 2024-03-06T17:37:15.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:15.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:15.935352+0000) 2024-03-06T17:37:16.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:16.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:16.935571+0000) 2024-03-06T17:37:17.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:17.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:17.935812+0000) 2024-03-06T17:37:18.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:18.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:18.936031+0000) 2024-03-06T17:37:19.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:19.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:19.936250+0000) 2024-03-06T17:37:20.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:20.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:20.936439+0000) 2024-03-06T17:37:21.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:21.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:21.936703+0000) 2024-03-06T17:37:22.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:22.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:22.936903+0000) 2024-03-06T17:37:23.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:23.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:23.937180+0000) 2024-03-06T17:37:24.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:24.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:24.937470+0000) 2024-03-06T17:37:25.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:25.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:25.937695+0000) 2024-03-06T17:37:26.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:26.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:26.937972+0000) 2024-03-06T17:37:27.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:27.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:27.938241+0000) 2024-03-06T17:37:28.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:28.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:28.938507+0000) 2024-03-06T17:37:29.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:29.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:29.938829+0000) 2024-03-06T17:37:30.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:30.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:30.939061+0000) 2024-03-06T17:37:31.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:31.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:31.939304+0000) 2024-03-06T17:37:32.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:32.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:32.939532+0000) 2024-03-06T17:37:33.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:33.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:33.939804+0000) 2024-03-06T17:37:34.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:34.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:34.940039+0000) 2024-03-06T17:37:35.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:35.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:35.940255+0000) 2024-03-06T17:37:36.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:36.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:36.940440+0000) 2024-03-06T17:37:37.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:37.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:37.940699+0000) 2024-03-06T17:37:38.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:38.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:38.940905+0000) 2024-03-06T17:37:39.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:39.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:39.941073+0000) 2024-03-06T17:37:40.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:40.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:40.941301+0000) 2024-03-06T17:37:41.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:41.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:41.941491+0000) 2024-03-06T17:37:42.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:42.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:42.941774+0000) 2024-03-06T17:37:43.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:43.439 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:43.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:43.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:43.941964+0000) 2024-03-06T17:37:44.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:44.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:44.942188+0000) 2024-03-06T17:37:45.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:45.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:45.942415+0000) 2024-03-06T17:37:46.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:46.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:46.942736+0000) 2024-03-06T17:37:47.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:47.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:47.943021+0000) 2024-03-06T17:37:48.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:48.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:48.943218+0000) 2024-03-06T17:37:49.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:49.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:49.943405+0000) 2024-03-06T17:37:50.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:50.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:50.943635+0000) 2024-03-06T17:37:51.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:51.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:51.943890+0000) 2024-03-06T17:37:52.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:52.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:52.944101+0000) 2024-03-06T17:37:53.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:53.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:53.944374+0000) 2024-03-06T17:37:54.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:54.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:54.944614+0000) 2024-03-06T17:37:55.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:55.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:55.944840+0000) 2024-03-06T17:37:56.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:56.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:56.945064+0000) 2024-03-06T17:37:57.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:57.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:57.945241+0000) 2024-03-06T17:37:58.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:58.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:58.945462+0000) 2024-03-06T17:37:59.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:37:59.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:59.945720+0000) 2024-03-06T17:38:00.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:00.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:00.945946+0000) 2024-03-06T17:38:01.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:01.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:01.946144+0000) 2024-03-06T17:38:02.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:02.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:02.946375+0000) 2024-03-06T17:38:03.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:03.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:03.946605+0000) 2024-03-06T17:38:04.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:04.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:04.946822+0000) 2024-03-06T17:38:05.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:05.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:05.947028+0000) 2024-03-06T17:38:06.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:06.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:06.947251+0000) 2024-03-06T17:38:07.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:07.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:07.947456+0000) 2024-03-06T17:38:08.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:08.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:08.947704+0000) 2024-03-06T17:38:09.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:09.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:09.947910+0000) 2024-03-06T17:38:10.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:10.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:10.948121+0000) 2024-03-06T17:38:11.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:11.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:11.948371+0000) 2024-03-06T17:38:12.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:12.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:12.948572+0000) 2024-03-06T17:38:13.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:13.592 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:13.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:13.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:13.948830+0000) 2024-03-06T17:38:14.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:14.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:14.949047+0000) 2024-03-06T17:38:15.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:15.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:15.949227+0000) 2024-03-06T17:38:16.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:16.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:16.949411+0000) 2024-03-06T17:38:17.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:17.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:17.949590+0000) 2024-03-06T17:38:18.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:18.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:18.949841+0000) 2024-03-06T17:38:19.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:19.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:19.950082+0000) 2024-03-06T17:38:20.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:20.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:20.950308+0000) 2024-03-06T17:38:21.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:21.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:21.950495+0000) 2024-03-06T17:38:22.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:22.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:22.950762+0000) 2024-03-06T17:38:23.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:23.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:23.951013+0000) 2024-03-06T17:38:24.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:24.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:24.951240+0000) 2024-03-06T17:38:25.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:25.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:25.951492+0000) 2024-03-06T17:38:26.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:26.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:26.951714+0000) 2024-03-06T17:38:27.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:27.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:27.951938+0000) 2024-03-06T17:38:28.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:28.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:28.952173+0000) 2024-03-06T17:38:29.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:29.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:29.952408+0000) 2024-03-06T17:38:30.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:30.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:30.952635+0000) 2024-03-06T17:38:31.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:31.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:31.952915+0000) 2024-03-06T17:38:32.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:32.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:32.953144+0000) 2024-03-06T17:38:33.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:33.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:33.953383+0000) 2024-03-06T17:38:34.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:34.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:34.953630+0000) 2024-03-06T17:38:35.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:35.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:35.953914+0000) 2024-03-06T17:38:36.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:36.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:36.954104+0000) 2024-03-06T17:38:37.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:37.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:37.954344+0000) 2024-03-06T17:38:38.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:38.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:38.954561+0000) 2024-03-06T17:38:39.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:39.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:39.954813+0000) 2024-03-06T17:38:40.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:40.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:40.955032+0000) 2024-03-06T17:38:41.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:41.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:41.955258+0000) 2024-03-06T17:38:42.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:42.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:42.955492+0000) 2024-03-06T17:38:43.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:43.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:43.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:43.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:43.955780+0000) 2024-03-06T17:38:44.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:44.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:44.955978+0000) 2024-03-06T17:38:45.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:45.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:45.956163+0000) 2024-03-06T17:38:46.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:46.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:46.956339+0000) 2024-03-06T17:38:47.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:47.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:47.956542+0000) 2024-03-06T17:38:48.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:48.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:48.956768+0000) 2024-03-06T17:38:49.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:49.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:49.956975+0000) 2024-03-06T17:38:50.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:50.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:50.957204+0000) 2024-03-06T17:38:51.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:51.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:51.957435+0000) 2024-03-06T17:38:52.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:52.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:52.957652+0000) 2024-03-06T17:38:53.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:53.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:53.957878+0000) 2024-03-06T17:38:54.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:54.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:54.958100+0000) 2024-03-06T17:38:55.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:55.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:55.958281+0000) 2024-03-06T17:38:56.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:56.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:56.958466+0000) 2024-03-06T17:38:57.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:57.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:57.958699+0000) 2024-03-06T17:38:58.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:58.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:58.958901+0000) 2024-03-06T17:38:59.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:38:59.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:59.959105+0000) 2024-03-06T17:39:00.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:00.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:00.959299+0000) 2024-03-06T17:39:01.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:01.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:01.959509+0000) 2024-03-06T17:39:02.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:02.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:02.959766+0000) 2024-03-06T17:39:03.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:03.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:03.959969+0000) 2024-03-06T17:39:04.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:04.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:04.960207+0000) 2024-03-06T17:39:05.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:05.959+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:05.960439+0000) 2024-03-06T17:39:06.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:06.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:06.960768+0000) 2024-03-06T17:39:07.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:07.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:07.960959+0000) 2024-03-06T17:39:08.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:08.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:08.961182+0000) 2024-03-06T17:39:09.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:09.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:09.961377+0000) 2024-03-06T17:39:10.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:10.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:10.961608+0000) 2024-03-06T17:39:11.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:11.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:11.961851+0000) 2024-03-06T17:39:12.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:12.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:12.962076+0000) 2024-03-06T17:39:13.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:13.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:13.962305+0000) 2024-03-06T17:39:14.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:14.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:14.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:14.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:14.962547+0000) 2024-03-06T17:39:15.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:15.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:15.962774+0000) 2024-03-06T17:39:16.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:16.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:16.962999+0000) 2024-03-06T17:39:17.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:17.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:17.963215+0000) 2024-03-06T17:39:18.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:18.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:18.963410+0000) 2024-03-06T17:39:19.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:19.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:19.963656+0000) 2024-03-06T17:39:20.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:20.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:20.963912+0000) 2024-03-06T17:39:21.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:21.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:21.964148+0000) 2024-03-06T17:39:22.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:22.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:22.964340+0000) 2024-03-06T17:39:23.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:23.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:23.964539+0000) 2024-03-06T17:39:24.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:24.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:24.964764+0000) 2024-03-06T17:39:25.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:25.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:25.964981+0000) 2024-03-06T17:39:26.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:26.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:26.965201+0000) 2024-03-06T17:39:27.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:27.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:27.965419+0000) 2024-03-06T17:39:28.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:28.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:28.965641+0000) 2024-03-06T17:39:29.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:29.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:29.965905+0000) 2024-03-06T17:39:30.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:30.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:30.966125+0000) 2024-03-06T17:39:31.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:31.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:31.966359+0000) 2024-03-06T17:39:32.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:32.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:32.966538+0000) 2024-03-06T17:39:33.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:33.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:33.966806+0000) 2024-03-06T17:39:34.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:34.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:34.967032+0000) 2024-03-06T17:39:35.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:35.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:35.967258+0000) 2024-03-06T17:39:36.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:36.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:36.967434+0000) 2024-03-06T17:39:37.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:37.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:37.967661+0000) 2024-03-06T17:39:38.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:38.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:38.967920+0000) 2024-03-06T17:39:39.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:39.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:39.968137+0000) 2024-03-06T17:39:40.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:40.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:40.968308+0000) 2024-03-06T17:39:41.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:41.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:41.968539+0000) 2024-03-06T17:39:42.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:42.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:42.968792+0000) 2024-03-06T17:39:43.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:43.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:43.969019+0000) 2024-03-06T17:39:44.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:44.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:44.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:44.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:44.969243+0000) 2024-03-06T17:39:45.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:45.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:45.969483+0000) 2024-03-06T17:39:46.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:46.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:46.969765+0000) 2024-03-06T17:39:47.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:47.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:47.969993+0000) 2024-03-06T17:39:48.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:48.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:48.970179+0000) 2024-03-06T17:39:49.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:49.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:49.970377+0000) 2024-03-06T17:39:50.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:50.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:50.970510+0000) 2024-03-06T17:39:51.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:51.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:51.970797+0000) 2024-03-06T17:39:52.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:52.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:52.971006+0000) 2024-03-06T17:39:53.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:53.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:53.971250+0000) 2024-03-06T17:39:54.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:54.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:54.971536+0000) 2024-03-06T17:39:55.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:55.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:55.971795+0000) 2024-03-06T17:39:56.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:56.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:56.972017+0000) 2024-03-06T17:39:57.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:57.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:57.972205+0000) 2024-03-06T17:39:58.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:58.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:58.972402+0000) 2024-03-06T17:39:59.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:39:59.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:59.972587+0000) 2024-03-06T17:40:00.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:00.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:00.972944+0000) 2024-03-06T17:40:01.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:01.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:01.973190+0000) 2024-03-06T17:40:02.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:02.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:02.973431+0000) 2024-03-06T17:40:03.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:03.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:03.973719+0000) 2024-03-06T17:40:04.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:04.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:04.973923+0000) 2024-03-06T17:40:05.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:05.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:05.974112+0000) 2024-03-06T17:40:06.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:06.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:06.974345+0000) 2024-03-06T17:40:07.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:07.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:07.974577+0000) 2024-03-06T17:40:08.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:08.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:08.974828+0000) 2024-03-06T17:40:09.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:09.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:09.975062+0000) 2024-03-06T17:40:10.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:10.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:10.975295+0000) 2024-03-06T17:40:11.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:11.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:11.975528+0000) 2024-03-06T17:40:12.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:12.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:12.975798+0000) 2024-03-06T17:40:13.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:13.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:13.976034+0000) 2024-03-06T17:40:14.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:14.459 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:14.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:14.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:14.976256+0000) 2024-03-06T17:40:15.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:15.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:15.976442+0000) 2024-03-06T17:40:16.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:16.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:16.976647+0000) 2024-03-06T17:40:17.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:17.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:17.976909+0000) 2024-03-06T17:40:18.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:18.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:18.977153+0000) 2024-03-06T17:40:19.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:19.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:19.977372+0000) 2024-03-06T17:40:20.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:20.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:20.977604+0000) 2024-03-06T17:40:21.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:21.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:21.977899+0000) 2024-03-06T17:40:22.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:22.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:22.978126+0000) 2024-03-06T17:40:23.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:23.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:23.978326+0000) 2024-03-06T17:40:24.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:24.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:24.978544+0000) 2024-03-06T17:40:25.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:25.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:25.978779+0000) 2024-03-06T17:40:26.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:26.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:26.978965+0000) 2024-03-06T17:40:27.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:27.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:27.979204+0000) 2024-03-06T17:40:28.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:28.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:28.979428+0000) 2024-03-06T17:40:29.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:29.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:29.979598+0000) 2024-03-06T17:40:30.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:30.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:30.979823+0000) 2024-03-06T17:40:31.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:31.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:31.980013+0000) 2024-03-06T17:40:32.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:32.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:32.980187+0000) 2024-03-06T17:40:33.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:33.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:33.980414+0000) 2024-03-06T17:40:34.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:34.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:34.980632+0000) 2024-03-06T17:40:35.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:35.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:35.980856+0000) 2024-03-06T17:40:36.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:36.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:36.981048+0000) 2024-03-06T17:40:37.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:37.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:37.981262+0000) 2024-03-06T17:40:38.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:38.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:38.981487+0000) 2024-03-06T17:40:39.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:39.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:39.981619+0000) 2024-03-06T17:40:40.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:40.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:40.981839+0000) 2024-03-06T17:40:41.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:41.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:41.982080+0000) 2024-03-06T17:40:42.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:42.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:42.982266+0000) 2024-03-06T17:40:43.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:43.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:43.982489+0000) 2024-03-06T17:40:44.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:44.623 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:44.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:44.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:44.982771+0000) 2024-03-06T17:40:45.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:45.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:45.983011+0000) 2024-03-06T17:40:46.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:46.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:46.983230+0000) 2024-03-06T17:40:47.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:47.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:47.983502+0000) 2024-03-06T17:40:48.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:48.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:48.983770+0000) 2024-03-06T17:40:49.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:49.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:49.983980+0000) 2024-03-06T17:40:50.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:50.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:50.984167+0000) 2024-03-06T17:40:51.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:51.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:51.984349+0000) 2024-03-06T17:40:52.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:52.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:52.984575+0000) 2024-03-06T17:40:53.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:53.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:53.984808+0000) 2024-03-06T17:40:54.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:54.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:54.985021+0000) 2024-03-06T17:40:55.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:55.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:55.985244+0000) 2024-03-06T17:40:56.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:56.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:56.985484+0000) 2024-03-06T17:40:57.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:57.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:57.985703+0000) 2024-03-06T17:40:58.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:58.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:58.985954+0000) 2024-03-06T17:40:59.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:40:59.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:59.986161+0000) 2024-03-06T17:41:00.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:00.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:00.986383+0000) 2024-03-06T17:41:01.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:01.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:01.986596+0000) 2024-03-06T17:41:02.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:02.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:02.986811+0000) 2024-03-06T17:41:03.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:03.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:03.987022+0000) 2024-03-06T17:41:04.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:04.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:04.987240+0000) 2024-03-06T17:41:05.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:05.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:05.987457+0000) 2024-03-06T17:41:06.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:06.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:06.987653+0000) 2024-03-06T17:41:07.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:07.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:07.987926+0000) 2024-03-06T17:41:08.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:08.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:08.988118+0000) 2024-03-06T17:41:09.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:09.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:09.988355+0000) 2024-03-06T17:41:10.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:10.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:10.988579+0000) 2024-03-06T17:41:11.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:11.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:11.988799+0000) 2024-03-06T17:41:12.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:12.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:12.989010+0000) 2024-03-06T17:41:13.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:13.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:13.989234+0000) 2024-03-06T17:41:14.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:14.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:14.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:14.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:14.989390+0000) 2024-03-06T17:41:15.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:15.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:15.989620+0000) 2024-03-06T17:41:16.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:16.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:16.989796+0000) 2024-03-06T17:41:17.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:17.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:17.990031+0000) 2024-03-06T17:41:18.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:18.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:18.990221+0000) 2024-03-06T17:41:19.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:19.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:19.990449+0000) 2024-03-06T17:41:20.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:20.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:20.990704+0000) 2024-03-06T17:41:21.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:21.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:21.990953+0000) 2024-03-06T17:41:22.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:22.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:22.991135+0000) 2024-03-06T17:41:23.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:23.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:23.991374+0000) 2024-03-06T17:41:24.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:24.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:24.991570+0000) 2024-03-06T17:41:25.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:25.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:25.991819+0000) 2024-03-06T17:41:26.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:26.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:26.992023+0000) 2024-03-06T17:41:27.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:27.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:27.992244+0000) 2024-03-06T17:41:28.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:28.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:28.992470+0000) 2024-03-06T17:41:29.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:29.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:29.992741+0000) 2024-03-06T17:41:30.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:30.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:30.992939+0000) 2024-03-06T17:41:31.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:31.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:31.993155+0000) 2024-03-06T17:41:32.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:32.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:32.993392+0000) 2024-03-06T17:41:33.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:33.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:33.993550+0000) 2024-03-06T17:41:34.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:34.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:34.993768+0000) 2024-03-06T17:41:35.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:35.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:35.993999+0000) 2024-03-06T17:41:36.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:36.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:36.994240+0000) 2024-03-06T17:41:37.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:37.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:37.994466+0000) 2024-03-06T17:41:38.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:38.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:38.994661+0000) 2024-03-06T17:41:39.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:39.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:39.994932+0000) 2024-03-06T17:41:41.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:40.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:40.995107+0000) 2024-03-06T17:41:41.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:41.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:41.995329+0000) 2024-03-06T17:41:42.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:42.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:42.995547+0000) 2024-03-06T17:41:43.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:43.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:43.995775+0000) 2024-03-06T17:41:44.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:44.997 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:44.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:44.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:44.996024+0000) 2024-03-06T17:41:45.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:45.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:45.996270+0000) 2024-03-06T17:41:46.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:46.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:46.996456+0000) 2024-03-06T17:41:47.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:47.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:47.996707+0000) 2024-03-06T17:41:48.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:48.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:48.996931+0000) 2024-03-06T17:41:49.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:49.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:49.997121+0000) 2024-03-06T17:41:50.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:50.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:50.997345+0000) 2024-03-06T17:41:52.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:51.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:51.997578+0000) 2024-03-06T17:41:52.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:52.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:52.997804+0000) 2024-03-06T17:41:53.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:53.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:53.998055+0000) 2024-03-06T17:41:55.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:54.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:54.998299+0000) 2024-03-06T17:41:56.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:55.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:55.998517+0000) 2024-03-06T17:41:57.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:56.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:56.998777+0000) 2024-03-06T17:41:58.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:57.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:57.998971+0000) 2024-03-06T17:41:59.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:58.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:58.999197+0000) 2024-03-06T17:42:00.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:41:59.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:59.999424+0000) 2024-03-06T17:42:01.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:00.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:00.999622+0000) 2024-03-06T17:42:02.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:01.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:01.999923+0000) 2024-03-06T17:42:03.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:02.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:03.000101+0000) 2024-03-06T17:42:04.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:03.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:04.000350+0000) 2024-03-06T17:42:05.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:04.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:05.000587+0000) 2024-03-06T17:42:06.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:05.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:06.000843+0000) 2024-03-06T17:42:07.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:07.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:07.001066+0000) 2024-03-06T17:42:08.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:08.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:08.001292+0000) 2024-03-06T17:42:09.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:09.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:09.001510+0000) 2024-03-06T17:42:10.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:10.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:10.001772+0000) 2024-03-06T17:42:11.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:11.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:11.001939+0000) 2024-03-06T17:42:12.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:12.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:12.002185+0000) 2024-03-06T17:42:13.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:13.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:13.002435+0000) 2024-03-06T17:42:14.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:14.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:14.002708+0000) 2024-03-06T17:42:15.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:15.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:15.002954+0000) 2024-03-06T17:42:15.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:15.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:16.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:16.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:16.003207+0000) 2024-03-06T17:42:17.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:17.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:17.003448+0000) 2024-03-06T17:42:18.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:18.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:18.003711+0000) 2024-03-06T17:42:19.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:19.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:19.003965+0000) 2024-03-06T17:42:20.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:20.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:20.004141+0000) 2024-03-06T17:42:21.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:21.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:21.004317+0000) 2024-03-06T17:42:22.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:22.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:22.004548+0000) 2024-03-06T17:42:23.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:23.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:23.004785+0000) 2024-03-06T17:42:24.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:24.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:24.005080+0000) 2024-03-06T17:42:25.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:25.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:25.005307+0000) 2024-03-06T17:42:26.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:26.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:26.005542+0000) 2024-03-06T17:42:27.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:27.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:27.005779+0000) 2024-03-06T17:42:28.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:28.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:28.006011+0000) 2024-03-06T17:42:29.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:29.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:29.006242+0000) 2024-03-06T17:42:30.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:30.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:30.006427+0000) 2024-03-06T17:42:31.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:31.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:31.006619+0000) 2024-03-06T17:42:32.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:32.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:32.006864+0000) 2024-03-06T17:42:33.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:33.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:33.007078+0000) 2024-03-06T17:42:34.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:34.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:34.007311+0000) 2024-03-06T17:42:35.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:35.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:35.007544+0000) 2024-03-06T17:42:36.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:36.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:36.007768+0000) 2024-03-06T17:42:37.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:37.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:37.007983+0000) 2024-03-06T17:42:38.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:38.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:38.008195+0000) 2024-03-06T17:42:39.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:39.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:39.008430+0000) 2024-03-06T17:42:40.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:40.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:40.008634+0000) 2024-03-06T17:42:41.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:41.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:41.008859+0000) 2024-03-06T17:42:42.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:42.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:42.009116+0000) 2024-03-06T17:42:43.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:43.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:43.009333+0000) 2024-03-06T17:42:44.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:44.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:44.009575+0000) 2024-03-06T17:42:45.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:45.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:45.009834+0000) 2024-03-06T17:42:45.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:45.339 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:46.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:46.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:46.010076+0000) 2024-03-06T17:42:47.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:47.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:47.010326+0000) 2024-03-06T17:42:48.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:48.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:48.010537+0000) 2024-03-06T17:42:49.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:49.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:49.010774+0000) 2024-03-06T17:42:50.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:50.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:50.010976+0000) 2024-03-06T17:42:51.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:51.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:51.011222+0000) 2024-03-06T17:42:52.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:52.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:52.011450+0000) 2024-03-06T17:42:53.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:53.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:53.011719+0000) 2024-03-06T17:42:54.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:54.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:54.011927+0000) 2024-03-06T17:42:55.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:55.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:55.012117+0000) 2024-03-06T17:42:56.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:56.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:56.012346+0000) 2024-03-06T17:42:57.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:57.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:57.012562+0000) 2024-03-06T17:42:58.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:58.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:58.012789+0000) 2024-03-06T17:42:59.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:42:59.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:59.013001+0000) 2024-03-06T17:43:00.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:00.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:00.013229+0000) 2024-03-06T17:43:01.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:01.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:01.013446+0000) 2024-03-06T17:43:02.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:02.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:02.013616+0000) 2024-03-06T17:43:03.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:03.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:03.013832+0000) 2024-03-06T17:43:04.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:04.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:04.014073+0000) 2024-03-06T17:43:05.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:05.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:05.014298+0000) 2024-03-06T17:43:06.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:06.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:06.014531+0000) 2024-03-06T17:43:07.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:07.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:07.014803+0000) 2024-03-06T17:43:08.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:08.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:08.015060+0000) 2024-03-06T17:43:09.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:09.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:09.015342+0000) 2024-03-06T17:43:10.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:10.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:10.015556+0000) 2024-03-06T17:43:11.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:11.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:11.015809+0000) 2024-03-06T17:43:12.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:12.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:12.016002+0000) 2024-03-06T17:43:13.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:13.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:13.016230+0000) 2024-03-06T17:43:14.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:14.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:14.016467+0000) 2024-03-06T17:43:15.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:15.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:15.016647+0000) 2024-03-06T17:43:15.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:15.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:16.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:16.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:16.016920+0000) 2024-03-06T17:43:17.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:17.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:17.017202+0000) 2024-03-06T17:43:18.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:18.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:18.017428+0000) 2024-03-06T17:43:19.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:19.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:19.017649+0000) 2024-03-06T17:43:20.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:20.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:20.017884+0000) 2024-03-06T17:43:21.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:21.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:21.018112+0000) 2024-03-06T17:43:22.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:22.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:22.018350+0000) 2024-03-06T17:43:23.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:23.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:23.018575+0000) 2024-03-06T17:43:24.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:24.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:24.018816+0000) 2024-03-06T17:43:25.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:25.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:25.019041+0000) 2024-03-06T17:43:26.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:26.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:26.019263+0000) 2024-03-06T17:43:27.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:27.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:27.019476+0000) 2024-03-06T17:43:28.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:28.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:28.019723+0000) 2024-03-06T17:43:29.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:29.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:29.019956+0000) 2024-03-06T17:43:30.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:30.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:30.020184+0000) 2024-03-06T17:43:31.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:31.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:31.020383+0000) 2024-03-06T17:43:32.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:32.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:32.020636+0000) 2024-03-06T17:43:33.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:33.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:33.020872+0000) 2024-03-06T17:43:34.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:34.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:34.021126+0000) 2024-03-06T17:43:35.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:35.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:35.021331+0000) 2024-03-06T17:43:36.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:36.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:36.021549+0000) 2024-03-06T17:43:37.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:37.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:37.021786+0000) 2024-03-06T17:43:38.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:38.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:38.022010+0000) 2024-03-06T17:43:39.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:39.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:39.022212+0000) 2024-03-06T17:43:40.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:40.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:40.022451+0000) 2024-03-06T17:43:41.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:41.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:41.022704+0000) 2024-03-06T17:43:42.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:42.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:42.022894+0000) 2024-03-06T17:43:43.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:43.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:43.023072+0000) 2024-03-06T17:43:44.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:44.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:44.023320+0000) 2024-03-06T17:43:45.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:45.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:45.023503+0000) 2024-03-06T17:43:45.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:45.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:46.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:46.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:46.023781+0000) 2024-03-06T17:43:47.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:47.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:47.024026+0000) 2024-03-06T17:43:48.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:48.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:48.024264+0000) 2024-03-06T17:43:49.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:49.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:49.024525+0000) 2024-03-06T17:43:50.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:50.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:50.024752+0000) 2024-03-06T17:43:51.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:51.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:51.024949+0000) 2024-03-06T17:43:52.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:52.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:52.025147+0000) 2024-03-06T17:43:53.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:53.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:53.025317+0000) 2024-03-06T17:43:54.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:54.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:54.025556+0000) 2024-03-06T17:43:55.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:55.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:55.025789+0000) 2024-03-06T17:43:56.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:56.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:56.026011+0000) 2024-03-06T17:43:57.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:57.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:57.026238+0000) 2024-03-06T17:43:58.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:58.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:58.026589+0000) 2024-03-06T17:43:59.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:43:59.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:59.026869+0000) 2024-03-06T17:44:00.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:00.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:00.027111+0000) 2024-03-06T17:44:01.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:01.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:01.027369+0000) 2024-03-06T17:44:02.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:02.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:02.027583+0000) 2024-03-06T17:44:03.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:03.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:03.027826+0000) 2024-03-06T17:44:04.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:04.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:04.028115+0000) 2024-03-06T17:44:05.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:05.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:05.028593+0000) 2024-03-06T17:44:06.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:06.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:06.028798+0000) 2024-03-06T17:44:07.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:07.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:07.028974+0000) 2024-03-06T17:44:08.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:08.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:08.029196+0000) 2024-03-06T17:44:09.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:09.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:09.029438+0000) 2024-03-06T17:44:10.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:10.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:10.029660+0000) 2024-03-06T17:44:11.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:11.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:11.029977+0000) 2024-03-06T17:44:12.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:12.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:12.030231+0000) 2024-03-06T17:44:13.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:13.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:13.030409+0000) 2024-03-06T17:44:14.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:14.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:14.030597+0000) 2024-03-06T17:44:15.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:15.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:15.030863+0000) 2024-03-06T17:44:15.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:15.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:16.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:16.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:16.031154+0000) 2024-03-06T17:44:17.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:17.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:17.031389+0000) 2024-03-06T17:44:18.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:18.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:18.031648+0000) 2024-03-06T17:44:19.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:19.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:19.031980+0000) 2024-03-06T17:44:20.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:20.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:20.032181+0000) 2024-03-06T17:44:21.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:21.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:21.032456+0000) 2024-03-06T17:44:22.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:22.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:22.032780+0000) 2024-03-06T17:44:23.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:23.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:23.033042+0000) 2024-03-06T17:44:24.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:24.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:24.033314+0000) 2024-03-06T17:44:25.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:25.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:25.033507+0000) 2024-03-06T17:44:26.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:26.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:26.033738+0000) 2024-03-06T17:44:27.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:27.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:27.033993+0000) 2024-03-06T17:44:28.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:28.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:28.034181+0000) 2024-03-06T17:44:29.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:29.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:29.034366+0000) 2024-03-06T17:44:30.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:30.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:30.034598+0000) 2024-03-06T17:44:31.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:31.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:31.034837+0000) 2024-03-06T17:44:32.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:32.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:32.035059+0000) 2024-03-06T17:44:33.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:33.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:33.035258+0000) 2024-03-06T17:44:34.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:34.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:34.035480+0000) 2024-03-06T17:44:35.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:35.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:35.035792+0000) 2024-03-06T17:44:36.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:36.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:36.035984+0000) 2024-03-06T17:44:37.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:37.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:37.036195+0000) 2024-03-06T17:44:38.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:38.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:38.036459+0000) 2024-03-06T17:44:39.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:39.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:39.036785+0000) 2024-03-06T17:44:40.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:40.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:40.037021+0000) 2024-03-06T17:44:41.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:41.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:41.037241+0000) 2024-03-06T17:44:42.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:42.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:42.037509+0000) 2024-03-06T17:44:43.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:43.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:43.037739+0000) 2024-03-06T17:44:44.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:44.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:44.037984+0000) 2024-03-06T17:44:45.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:45.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:45.038212+0000) 2024-03-06T17:44:46.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:46.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:46.038438+0000) 2024-03-06T17:44:46.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:46.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:47.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:47.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:47.038660+0000) 2024-03-06T17:44:48.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:48.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:48.038925+0000) 2024-03-06T17:44:49.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:49.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:49.039182+0000) 2024-03-06T17:44:50.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:50.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:50.039388+0000) 2024-03-06T17:44:51.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:51.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:51.039591+0000) 2024-03-06T17:44:52.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:52.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:52.039833+0000) 2024-03-06T17:44:53.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:53.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:53.040059+0000) 2024-03-06T17:44:54.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:54.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:54.040284+0000) 2024-03-06T17:44:55.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:55.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:55.040527+0000) 2024-03-06T17:44:56.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:56.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:56.040783+0000) 2024-03-06T17:44:57.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:57.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:57.040978+0000) 2024-03-06T17:44:58.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:58.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:58.041171+0000) 2024-03-06T17:44:59.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:44:59.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:59.041417+0000) 2024-03-06T17:45:00.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:00.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:00.041714+0000) 2024-03-06T17:45:01.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:01.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:01.041944+0000) 2024-03-06T17:45:02.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:02.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:02.042178+0000) 2024-03-06T17:45:03.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:03.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:03.042394+0000) 2024-03-06T17:45:04.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:04.040+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:04.042612+0000) 2024-03-06T17:45:05.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:05.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:05.042899+0000) 2024-03-06T17:45:06.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:06.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:06.043127+0000) 2024-03-06T17:45:07.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:07.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:07.043371+0000) 2024-03-06T17:45:08.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:08.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:08.043603+0000) 2024-03-06T17:45:09.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:09.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:09.043839+0000) 2024-03-06T17:45:10.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:10.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:10.044065+0000) 2024-03-06T17:45:11.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:11.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:11.044277+0000) 2024-03-06T17:45:12.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:12.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:12.044524+0000) 2024-03-06T17:45:13.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:13.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:13.044650+0000) 2024-03-06T17:45:14.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:14.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:14.044898+0000) 2024-03-06T17:45:15.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:15.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:15.045129+0000) 2024-03-06T17:45:16.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:16.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:16.045341+0000) 2024-03-06T17:45:16.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:16.228 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:17.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:17.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:17.045509+0000) 2024-03-06T17:45:18.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:18.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:18.045775+0000) 2024-03-06T17:45:19.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:19.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:19.046009+0000) 2024-03-06T17:45:20.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:20.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:20.046245+0000) 2024-03-06T17:45:21.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:21.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:21.046510+0000) 2024-03-06T17:45:22.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:22.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:22.046808+0000) 2024-03-06T17:45:23.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:23.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:23.047003+0000) 2024-03-06T17:45:24.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:24.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:24.047239+0000) 2024-03-06T17:45:25.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:25.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:25.047428+0000) 2024-03-06T17:45:26.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:26.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:26.047649+0000) 2024-03-06T17:45:27.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:27.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:27.047878+0000) 2024-03-06T17:45:28.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:28.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:28.048057+0000) 2024-03-06T17:45:29.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:29.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:29.048267+0000) 2024-03-06T17:45:30.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:30.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:30.048555+0000) 2024-03-06T17:45:31.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:31.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:31.048825+0000) 2024-03-06T17:45:32.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:32.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:32.049062+0000) 2024-03-06T17:45:33.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:33.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:33.049339+0000) 2024-03-06T17:45:34.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:34.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:34.049576+0000) 2024-03-06T17:45:35.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:35.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:35.049847+0000) 2024-03-06T17:45:36.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:36.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:36.050111+0000) 2024-03-06T17:45:37.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:37.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:37.050387+0000) 2024-03-06T17:45:38.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:38.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:38.050620+0000) 2024-03-06T17:45:39.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:39.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:39.050939+0000) 2024-03-06T17:45:40.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:40.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:40.051220+0000) 2024-03-06T17:45:41.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:41.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:41.051493+0000) 2024-03-06T17:45:42.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:42.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:42.051811+0000) 2024-03-06T17:45:43.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:43.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:43.052080+0000) 2024-03-06T17:45:44.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:44.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:44.052328+0000) 2024-03-06T17:45:45.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:45.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:45.052505+0000) 2024-03-06T17:45:46.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:46.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:46.052798+0000) 2024-03-06T17:45:46.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:46.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:47.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:47.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:47.053133+0000) 2024-03-06T17:45:48.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:48.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:48.053297+0000) 2024-03-06T17:45:49.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:49.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:49.053469+0000) 2024-03-06T17:45:50.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:50.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:50.053717+0000) 2024-03-06T17:45:51.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:51.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:51.053877+0000) 2024-03-06T17:45:52.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:52.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:52.054108+0000) 2024-03-06T17:45:53.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:53.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:53.054328+0000) 2024-03-06T17:45:54.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:54.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:54.054563+0000) 2024-03-06T17:45:55.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:55.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:55.054808+0000) 2024-03-06T17:45:56.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:56.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:56.054998+0000) 2024-03-06T17:45:57.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:57.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:57.055202+0000) 2024-03-06T17:45:58.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:58.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:58.055400+0000) 2024-03-06T17:45:59.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:45:59.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:59.055637+0000) 2024-03-06T17:46:00.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:00.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:00.055902+0000) 2024-03-06T17:46:01.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:01.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:01.056122+0000) 2024-03-06T17:46:02.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:02.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:02.056368+0000) 2024-03-06T17:46:03.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:03.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:03.056606+0000) 2024-03-06T17:46:04.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:04.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:04.056831+0000) 2024-03-06T17:46:05.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:05.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:05.057053+0000) 2024-03-06T17:46:06.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:06.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:06.057279+0000) 2024-03-06T17:46:07.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:07.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:07.057464+0000) 2024-03-06T17:46:08.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:08.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:08.057606+0000) 2024-03-06T17:46:09.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:09.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:09.057797+0000) 2024-03-06T17:46:10.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:10.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:10.058034+0000) 2024-03-06T17:46:11.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:11.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:11.058260+0000) 2024-03-06T17:46:12.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:12.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:12.058495+0000) 2024-03-06T17:46:13.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:13.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:13.058753+0000) 2024-03-06T17:46:14.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:14.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:14.058961+0000) 2024-03-06T17:46:15.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:15.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:15.059199+0000) 2024-03-06T17:46:16.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:16.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:16.059379+0000) 2024-03-06T17:46:16.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:16.575 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:17.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:17.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:17.059606+0000) 2024-03-06T17:46:18.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:18.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:18.059818+0000) 2024-03-06T17:46:19.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:19.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:19.060056+0000) 2024-03-06T17:46:20.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:20.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:20.060285+0000) 2024-03-06T17:46:21.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:21.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:21.060508+0000) 2024-03-06T17:46:22.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:22.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:22.060779+0000) 2024-03-06T17:46:23.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:23.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:23.060999+0000) 2024-03-06T17:46:24.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:24.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:24.061236+0000) 2024-03-06T17:46:25.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:25.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:25.061460+0000) 2024-03-06T17:46:26.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:26.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:26.061646+0000) 2024-03-06T17:46:27.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:27.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:27.061875+0000) 2024-03-06T17:46:28.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:28.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:28.062055+0000) 2024-03-06T17:46:29.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:29.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:29.062286+0000) 2024-03-06T17:46:30.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:30.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:30.062466+0000) 2024-03-06T17:46:31.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:31.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:31.062659+0000) 2024-03-06T17:46:32.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:32.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:32.062916+0000) 2024-03-06T17:46:33.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:33.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:33.063146+0000) 2024-03-06T17:46:34.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:34.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:34.063386+0000) 2024-03-06T17:46:35.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:35.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:35.063717+0000) 2024-03-06T17:46:36.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:36.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:36.063949+0000) 2024-03-06T17:46:37.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:37.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:37.064114+0000) 2024-03-06T17:46:38.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:38.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:38.064333+0000) 2024-03-06T17:46:39.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:39.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:39.064562+0000) 2024-03-06T17:46:40.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:40.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:40.064787+0000) 2024-03-06T17:46:41.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:41.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:41.064975+0000) 2024-03-06T17:46:42.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:42.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:42.065168+0000) 2024-03-06T17:46:43.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:43.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:43.065392+0000) 2024-03-06T17:46:44.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:44.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:44.065585+0000) 2024-03-06T17:46:45.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:45.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:45.065826+0000) 2024-03-06T17:46:46.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:46.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:46.066046+0000) 2024-03-06T17:46:46.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:46.708 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:47.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:47.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:47.066243+0000) 2024-03-06T17:46:48.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:48.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:48.066468+0000) 2024-03-06T17:46:49.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:49.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:49.066701+0000) 2024-03-06T17:46:50.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:50.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:50.066920+0000) 2024-03-06T17:46:51.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:51.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:51.067149+0000) 2024-03-06T17:46:52.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:52.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:52.067395+0000) 2024-03-06T17:46:53.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:53.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:53.067614+0000) 2024-03-06T17:46:54.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:54.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:54.067853+0000) 2024-03-06T17:46:55.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:55.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:55.068133+0000) 2024-03-06T17:46:56.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:56.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:56.068406+0000) 2024-03-06T17:46:57.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:57.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:57.068718+0000) 2024-03-06T17:46:58.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:58.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:58.068949+0000) 2024-03-06T17:46:59.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:46:59.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:59.069235+0000) 2024-03-06T17:47:00.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:00.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:00.069442+0000) 2024-03-06T17:47:01.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:01.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:01.069702+0000) 2024-03-06T17:47:02.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:02.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:02.069937+0000) 2024-03-06T17:47:03.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:03.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:03.070164+0000) 2024-03-06T17:47:04.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:04.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:04.070395+0000) 2024-03-06T17:47:05.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:05.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:05.070616+0000) 2024-03-06T17:47:06.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:06.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:06.070804+0000) 2024-03-06T17:47:07.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:07.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:07.071065+0000) 2024-03-06T17:47:08.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:08.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:08.071271+0000) 2024-03-06T17:47:09.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:09.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:09.071507+0000) 2024-03-06T17:47:10.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:10.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:10.071792+0000) 2024-03-06T17:47:11.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:11.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:11.072009+0000) 2024-03-06T17:47:12.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:12.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:12.072241+0000) 2024-03-06T17:47:13.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:13.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:13.072462+0000) 2024-03-06T17:47:14.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:14.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:14.072718+0000) 2024-03-06T17:47:15.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:15.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:15.072969+0000) 2024-03-06T17:47:16.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:16.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:16.073194+0000) 2024-03-06T17:47:16.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:16.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:17.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:17.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:17.073380+0000) 2024-03-06T17:47:18.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:18.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:18.073597+0000) 2024-03-06T17:47:19.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:19.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:19.073830+0000) 2024-03-06T17:47:20.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:20.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:20.074059+0000) 2024-03-06T17:47:21.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:21.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:21.074275+0000) 2024-03-06T17:47:22.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:22.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:22.074565+0000) 2024-03-06T17:47:23.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:23.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:23.074824+0000) 2024-03-06T17:47:24.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:24.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:24.075020+0000) 2024-03-06T17:47:25.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:25.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:25.075229+0000) 2024-03-06T17:47:26.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:26.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:26.075414+0000) 2024-03-06T17:47:27.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:27.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:27.075654+0000) 2024-03-06T17:47:28.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:28.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:28.075962+0000) 2024-03-06T17:47:29.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:29.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:29.076187+0000) 2024-03-06T17:47:30.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:30.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:30.076398+0000) 2024-03-06T17:47:31.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:31.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:31.076702+0000) 2024-03-06T17:47:32.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:32.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:32.076919+0000) 2024-03-06T17:47:33.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:33.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:33.077100+0000) 2024-03-06T17:47:34.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:34.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:34.077334+0000) 2024-03-06T17:47:35.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:35.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:35.077538+0000) 2024-03-06T17:47:36.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:36.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:36.077791+0000) 2024-03-06T17:47:37.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:37.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:37.078009+0000) 2024-03-06T17:47:38.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:38.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:38.078264+0000) 2024-03-06T17:47:39.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:39.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:39.078484+0000) 2024-03-06T17:47:40.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:40.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:40.078772+0000) 2024-03-06T17:47:41.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:41.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:41.079011+0000) 2024-03-06T17:47:42.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:42.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:42.079264+0000) 2024-03-06T17:47:43.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:43.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:43.079489+0000) 2024-03-06T17:47:44.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:44.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:44.079721+0000) 2024-03-06T17:47:45.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:45.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:45.079952+0000) 2024-03-06T17:47:46.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:46.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:46.080156+0000) 2024-03-06T17:47:47.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:47.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:47.080380+0000) 2024-03-06T17:47:47.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:47.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:48.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:48.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:48.080542+0000) 2024-03-06T17:47:49.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:49.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:49.080776+0000) 2024-03-06T17:47:50.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:50.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:50.081003+0000) 2024-03-06T17:47:51.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:51.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:51.081230+0000) 2024-03-06T17:47:52.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:52.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:52.081436+0000) 2024-03-06T17:47:53.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:53.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:53.081702+0000) 2024-03-06T17:47:54.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:54.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:54.081927+0000) 2024-03-06T17:47:55.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:55.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:55.082157+0000) 2024-03-06T17:47:56.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:56.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:56.082381+0000) 2024-03-06T17:47:57.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:57.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:57.082559+0000) 2024-03-06T17:47:58.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:58.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:58.082777+0000) 2024-03-06T17:47:59.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:47:59.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:59.082974+0000) 2024-03-06T17:48:00.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:00.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:00.083202+0000) 2024-03-06T17:48:01.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:01.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:01.083419+0000) 2024-03-06T17:48:02.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:02.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:02.083652+0000) 2024-03-06T17:48:03.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:03.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:03.083911+0000) 2024-03-06T17:48:04.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:04.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:04.084137+0000) 2024-03-06T17:48:05.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:05.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:05.084285+0000) 2024-03-06T17:48:06.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:06.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:06.084429+0000) 2024-03-06T17:48:07.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:07.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:07.084647+0000) 2024-03-06T17:48:08.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:08.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:08.084885+0000) 2024-03-06T17:48:09.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:09.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:09.085080+0000) 2024-03-06T17:48:10.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:10.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:10.085326+0000) 2024-03-06T17:48:11.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:11.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:11.085550+0000) 2024-03-06T17:48:12.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:12.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:12.085780+0000) 2024-03-06T17:48:13.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:13.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:13.086003+0000) 2024-03-06T17:48:14.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:14.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:14.086202+0000) 2024-03-06T17:48:15.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:15.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:15.086430+0000) 2024-03-06T17:48:16.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:16.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:16.086750+0000) 2024-03-06T17:48:17.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:17.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:17.086946+0000) 2024-03-06T17:48:17.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:17.415 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:18.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:18.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:18.087126+0000) 2024-03-06T17:48:19.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:19.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:19.087357+0000) 2024-03-06T17:48:20.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:20.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:20.087557+0000) 2024-03-06T17:48:21.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:21.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:21.087797+0000) 2024-03-06T17:48:22.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:22.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:22.088044+0000) 2024-03-06T17:48:23.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:23.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:23.088213+0000) 2024-03-06T17:48:24.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:24.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:24.088438+0000) 2024-03-06T17:48:25.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:25.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:25.088705+0000) 2024-03-06T17:48:26.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:26.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:26.088905+0000) 2024-03-06T17:48:27.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:27.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:27.089158+0000) 2024-03-06T17:48:28.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:28.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:28.089354+0000) 2024-03-06T17:48:29.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:29.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:29.089557+0000) 2024-03-06T17:48:30.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:30.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:30.089784+0000) 2024-03-06T17:48:31.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:31.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:31.090041+0000) 2024-03-06T17:48:32.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:32.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:32.090333+0000) 2024-03-06T17:48:33.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:33.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:33.090577+0000) 2024-03-06T17:48:34.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:34.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:34.090788+0000) 2024-03-06T17:48:35.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:35.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:35.091009+0000) 2024-03-06T17:48:36.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:36.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:36.091229+0000) 2024-03-06T17:48:37.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:37.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:37.091457+0000) 2024-03-06T17:48:38.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:38.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:38.091709+0000) 2024-03-06T17:48:39.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:39.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:39.091906+0000) 2024-03-06T17:48:40.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:40.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:40.092096+0000) 2024-03-06T17:48:41.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:41.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:41.092313+0000) 2024-03-06T17:48:42.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:42.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:42.092554+0000) 2024-03-06T17:48:43.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:43.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:43.092765+0000) 2024-03-06T17:48:44.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:44.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:44.092959+0000) 2024-03-06T17:48:45.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:45.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:45.093143+0000) 2024-03-06T17:48:46.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:46.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:46.093378+0000) 2024-03-06T17:48:47.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:47.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:47.093574+0000) 2024-03-06T17:48:47.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:47.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:48.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:48.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:48.093811+0000) 2024-03-06T17:48:49.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:49.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:49.094039+0000) 2024-03-06T17:48:50.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:50.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:50.094279+0000) 2024-03-06T17:48:51.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:51.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:51.094503+0000) 2024-03-06T17:48:52.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:52.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:52.094775+0000) 2024-03-06T17:48:53.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:53.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:53.095000+0000) 2024-03-06T17:48:54.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:54.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:54.095230+0000) 2024-03-06T17:48:55.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:55.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:55.095456+0000) 2024-03-06T17:48:56.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:56.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:56.095710+0000) 2024-03-06T17:48:57.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:57.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:57.095946+0000) 2024-03-06T17:48:58.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:58.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:58.096176+0000) 2024-03-06T17:48:59.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:48:59.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:59.096414+0000) 2024-03-06T17:49:00.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:00.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:00.096706+0000) 2024-03-06T17:49:01.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:01.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:01.096946+0000) 2024-03-06T17:49:02.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:02.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:02.097192+0000) 2024-03-06T17:49:03.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:03.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:03.097415+0000) 2024-03-06T17:49:04.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:04.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:04.097737+0000) 2024-03-06T17:49:05.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:05.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:05.098002+0000) 2024-03-06T17:49:06.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:06.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:06.098219+0000) 2024-03-06T17:49:07.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:07.096+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:07.098456+0000) 2024-03-06T17:49:08.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:08.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:08.098706+0000) 2024-03-06T17:49:09.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:09.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:09.098950+0000) 2024-03-06T17:49:10.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:10.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:10.099205+0000) 2024-03-06T17:49:11.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:11.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:11.099445+0000) 2024-03-06T17:49:12.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:12.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:12.099701+0000) 2024-03-06T17:49:13.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:13.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:13.099912+0000) 2024-03-06T17:49:14.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:14.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:14.100167+0000) 2024-03-06T17:49:15.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:15.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:15.100388+0000) 2024-03-06T17:49:16.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:16.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:16.100627+0000) 2024-03-06T17:49:17.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:17.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:17.100826+0000) 2024-03-06T17:49:17.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:17.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:18.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:18.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:18.101064+0000) 2024-03-06T17:49:19.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:19.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:19.101297+0000) 2024-03-06T17:49:20.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:20.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:20.101560+0000) 2024-03-06T17:49:21.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:21.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:21.101820+0000) 2024-03-06T17:49:22.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:22.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:22.102088+0000) 2024-03-06T17:49:23.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:23.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:23.102361+0000) 2024-03-06T17:49:24.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:24.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:24.102549+0000) 2024-03-06T17:49:25.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:25.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:25.102782+0000) 2024-03-06T17:49:26.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:26.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:26.103038+0000) 2024-03-06T17:49:27.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:27.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:27.103322+0000) 2024-03-06T17:49:28.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:28.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:28.103601+0000) 2024-03-06T17:49:29.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:29.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:29.103874+0000) 2024-03-06T17:49:30.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:30.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:30.104086+0000) 2024-03-06T17:49:31.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:31.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:31.104326+0000) 2024-03-06T17:49:32.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:32.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:32.104540+0000) 2024-03-06T17:49:33.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:33.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:33.104785+0000) 2024-03-06T17:49:34.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:34.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:34.105062+0000) 2024-03-06T17:49:35.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:35.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:35.105340+0000) 2024-03-06T17:49:36.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:36.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:36.105619+0000) 2024-03-06T17:49:37.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:37.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:37.105903+0000) 2024-03-06T17:49:38.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:38.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:38.106171+0000) 2024-03-06T17:49:39.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:39.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:39.106451+0000) 2024-03-06T17:49:40.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:40.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:40.106775+0000) 2024-03-06T17:49:41.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:41.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:41.107040+0000) 2024-03-06T17:49:42.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:42.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:42.107322+0000) 2024-03-06T17:49:43.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:43.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:43.107586+0000) 2024-03-06T17:49:44.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:44.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:44.107835+0000) 2024-03-06T17:49:45.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:45.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:45.108086+0000) 2024-03-06T17:49:46.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:46.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:46.108290+0000) 2024-03-06T17:49:47.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:47.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:47.108532+0000) 2024-03-06T17:49:47.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:47.949 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:48.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:48.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:48.108782+0000) 2024-03-06T17:49:49.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:49.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:49.109024+0000) 2024-03-06T17:49:50.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:50.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:50.109243+0000) 2024-03-06T17:49:51.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:51.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:51.109464+0000) 2024-03-06T17:49:52.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:52.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:52.109741+0000) 2024-03-06T17:49:53.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:53.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:53.109983+0000) 2024-03-06T17:49:54.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:54.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:54.110226+0000) 2024-03-06T17:49:55.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:55.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:55.110496+0000) 2024-03-06T17:49:56.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:56.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:56.110782+0000) 2024-03-06T17:49:57.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:57.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:57.111086+0000) 2024-03-06T17:49:58.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:58.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:58.111315+0000) 2024-03-06T17:49:59.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:49:59.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:59.111555+0000) 2024-03-06T17:50:00.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:00.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:00.111814+0000) 2024-03-06T17:50:01.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:01.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:01.112049+0000) 2024-03-06T17:50:02.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:02.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:02.112305+0000) 2024-03-06T17:50:03.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:03.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:03.112559+0000) 2024-03-06T17:50:04.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:04.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:04.112832+0000) 2024-03-06T17:50:05.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:05.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:05.113112+0000) 2024-03-06T17:50:06.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:06.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:06.113390+0000) 2024-03-06T17:50:07.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:07.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:07.113703+0000) 2024-03-06T17:50:08.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:08.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:08.113899+0000) 2024-03-06T17:50:09.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:09.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:09.114031+0000) 2024-03-06T17:50:10.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:10.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:10.114224+0000) 2024-03-06T17:50:11.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:11.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:11.114484+0000) 2024-03-06T17:50:12.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:12.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:12.114778+0000) 2024-03-06T17:50:13.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:13.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:13.115034+0000) 2024-03-06T17:50:14.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:14.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:14.115276+0000) 2024-03-06T17:50:15.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:15.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:15.115565+0000) 2024-03-06T17:50:16.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:16.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:16.115848+0000) 2024-03-06T17:50:17.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:17.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:17.116130+0000) 2024-03-06T17:50:18.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:18.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:18.116373+0000) 2024-03-06T17:50:18.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:18.171 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:19.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:19.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:19.116567+0000) 2024-03-06T17:50:20.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:20.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:20.116814+0000) 2024-03-06T17:50:21.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:21.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:21.117034+0000) 2024-03-06T17:50:22.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:22.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:22.117261+0000) 2024-03-06T17:50:23.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:23.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:23.117485+0000) 2024-03-06T17:50:24.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:24.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:24.117701+0000) 2024-03-06T17:50:25.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:25.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:25.117954+0000) 2024-03-06T17:50:26.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:26.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:26.118182+0000) 2024-03-06T17:50:27.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:27.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:27.118367+0000) 2024-03-06T17:50:28.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:28.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:28.118589+0000) 2024-03-06T17:50:29.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:29.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:29.118810+0000) 2024-03-06T17:50:30.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:30.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:30.119040+0000) 2024-03-06T17:50:31.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:31.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:31.119232+0000) 2024-03-06T17:50:32.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:32.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:32.119479+0000) 2024-03-06T17:50:33.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:33.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:33.119694+0000) 2024-03-06T17:50:34.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:34.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:34.119928+0000) 2024-03-06T17:50:35.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:35.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:35.120161+0000) 2024-03-06T17:50:36.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:36.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:36.120385+0000) 2024-03-06T17:50:37.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:37.118+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:37.120611+0000) 2024-03-06T17:50:38.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:38.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:38.120831+0000) 2024-03-06T17:50:39.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:39.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:39.121052+0000) 2024-03-06T17:50:40.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:40.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:40.121293+0000) 2024-03-06T17:50:41.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:41.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:41.121521+0000) 2024-03-06T17:50:42.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:42.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:42.121737+0000) 2024-03-06T17:50:43.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:43.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:43.121955+0000) 2024-03-06T17:50:44.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:44.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:44.122149+0000) 2024-03-06T17:50:45.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:45.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:45.122432+0000) 2024-03-06T17:50:46.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:46.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:46.122585+0000) 2024-03-06T17:50:47.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:47.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:47.122814+0000) 2024-03-06T17:50:48.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:48.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:48.123023+0000) 2024-03-06T17:50:48.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:48.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:49.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:49.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:49.123202+0000) 2024-03-06T17:50:50.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:50.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:50.123441+0000) 2024-03-06T17:50:51.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:51.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:51.123690+0000) 2024-03-06T17:50:52.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:52.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:52.123943+0000) 2024-03-06T17:50:53.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:53.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:53.124184+0000) 2024-03-06T17:50:54.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:54.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:54.124434+0000) 2024-03-06T17:50:55.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:55.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:55.124696+0000) 2024-03-06T17:50:56.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:56.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:56.124944+0000) 2024-03-06T17:50:57.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:57.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:57.125191+0000) 2024-03-06T17:50:58.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:58.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:58.125432+0000) 2024-03-06T17:50:59.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:50:59.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:59.125693+0000) 2024-03-06T17:51:00.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:00.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:00.125954+0000) 2024-03-06T17:51:01.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:01.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:01.126225+0000) 2024-03-06T17:51:02.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:02.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:02.126472+0000) 2024-03-06T17:51:03.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:03.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:03.126657+0000) 2024-03-06T17:51:04.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:04.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:04.126918+0000) 2024-03-06T17:51:05.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:05.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:05.127199+0000) 2024-03-06T17:51:06.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:06.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:06.127493+0000) 2024-03-06T17:51:07.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:07.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:07.127804+0000) 2024-03-06T17:51:08.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:08.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:08.128064+0000) 2024-03-06T17:51:09.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:09.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:09.128340+0000) 2024-03-06T17:51:10.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:10.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:10.128609+0000) 2024-03-06T17:51:11.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:11.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:11.128880+0000) 2024-03-06T17:51:12.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:12.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:12.129142+0000) 2024-03-06T17:51:13.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:13.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:13.129363+0000) 2024-03-06T17:51:14.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:14.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:14.129898+0000) 2024-03-06T17:51:15.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:15.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:15.130106+0000) 2024-03-06T17:51:16.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:16.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:16.130333+0000) 2024-03-06T17:51:17.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:17.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:17.130547+0000) 2024-03-06T17:51:18.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:18.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:18.130798+0000) 2024-03-06T17:51:18.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:18.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:19.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:19.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:19.131031+0000) 2024-03-06T17:51:20.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:20.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:20.131225+0000) 2024-03-06T17:51:21.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:21.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:21.131655+0000) 2024-03-06T17:51:22.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:22.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:22.132041+0000) 2024-03-06T17:51:23.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:23.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:23.132306+0000) 2024-03-06T17:51:24.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:24.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:24.132483+0000) 2024-03-06T17:51:25.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:25.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:25.132773+0000) 2024-03-06T17:51:26.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:26.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:26.133004+0000) 2024-03-06T17:51:27.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:27.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:27.133162+0000) 2024-03-06T17:51:28.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:28.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:28.133405+0000) 2024-03-06T17:51:29.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:29.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:29.133623+0000) 2024-03-06T17:51:30.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:30.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:30.133787+0000) 2024-03-06T17:51:31.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:31.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:31.134034+0000) 2024-03-06T17:51:32.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:32.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:32.134287+0000) 2024-03-06T17:51:33.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:33.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:33.134500+0000) 2024-03-06T17:51:34.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:34.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:34.134738+0000) 2024-03-06T17:51:35.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:35.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:35.134955+0000) 2024-03-06T17:51:36.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:36.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:36.135185+0000) 2024-03-06T17:51:37.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:37.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:37.135409+0000) 2024-03-06T17:51:38.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:38.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:38.135638+0000) 2024-03-06T17:51:39.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:39.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:39.135974+0000) 2024-03-06T17:51:40.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:40.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:40.136214+0000) 2024-03-06T17:51:41.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:41.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:41.136439+0000) 2024-03-06T17:51:42.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:42.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:42.136589+0000) 2024-03-06T17:51:43.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:43.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:43.136801+0000) 2024-03-06T17:51:44.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:44.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:44.137051+0000) 2024-03-06T17:51:45.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:45.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:45.137291+0000) 2024-03-06T17:51:46.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:46.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:46.137537+0000) 2024-03-06T17:51:47.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:47.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:47.137800+0000) 2024-03-06T17:51:48.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:48.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:48.137993+0000) 2024-03-06T17:51:48.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:48.750 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:49.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:49.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:49.138186+0000) 2024-03-06T17:51:50.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:50.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:50.138378+0000) 2024-03-06T17:51:51.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:51.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:51.138597+0000) 2024-03-06T17:51:52.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:52.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:52.138818+0000) 2024-03-06T17:51:53.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:53.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:53.139004+0000) 2024-03-06T17:51:54.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:54.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:54.139201+0000) 2024-03-06T17:51:55.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:55.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:55.139346+0000) 2024-03-06T17:51:56.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:56.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:56.139575+0000) 2024-03-06T17:51:57.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:57.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:57.139803+0000) 2024-03-06T17:51:58.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:58.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:58.140021+0000) 2024-03-06T17:51:59.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:51:59.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:59.140245+0000) 2024-03-06T17:52:00.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:00.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:00.140461+0000) 2024-03-06T17:52:01.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:01.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:01.140750+0000) 2024-03-06T17:52:02.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:02.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:02.140987+0000) 2024-03-06T17:52:03.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:03.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:03.141220+0000) 2024-03-06T17:52:04.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:04.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:04.141403+0000) 2024-03-06T17:52:05.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:05.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:05.141631+0000) 2024-03-06T17:52:06.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:06.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:06.141865+0000) 2024-03-06T17:52:07.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:07.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:07.142090+0000) 2024-03-06T17:52:08.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:08.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:08.142310+0000) 2024-03-06T17:52:09.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:09.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:09.142495+0000) 2024-03-06T17:52:10.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:10.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:10.142774+0000) 2024-03-06T17:52:11.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:11.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:11.142953+0000) 2024-03-06T17:52:12.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:12.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:12.143147+0000) 2024-03-06T17:52:13.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:13.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:13.143367+0000) 2024-03-06T17:52:14.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:14.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:14.143594+0000) 2024-03-06T17:52:15.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:15.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:15.143802+0000) 2024-03-06T17:52:16.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:16.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:16.144010+0000) 2024-03-06T17:52:17.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:17.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:17.144422+0000) 2024-03-06T17:52:18.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:18.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:18.144646+0000) 2024-03-06T17:52:18.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:18.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:19.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:19.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:19.144897+0000) 2024-03-06T17:52:20.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:20.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:20.145130+0000) 2024-03-06T17:52:21.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:21.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:21.145356+0000) 2024-03-06T17:52:22.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:22.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:22.145533+0000) 2024-03-06T17:52:23.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:23.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:23.145763+0000) 2024-03-06T17:52:24.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:24.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:24.145992+0000) 2024-03-06T17:52:25.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:25.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:25.146174+0000) 2024-03-06T17:52:26.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:26.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:26.146362+0000) 2024-03-06T17:52:27.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:27.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:27.146648+0000) 2024-03-06T17:52:28.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:28.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:28.146910+0000) 2024-03-06T17:52:29.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:29.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:29.147097+0000) 2024-03-06T17:52:30.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:30.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:30.147321+0000) 2024-03-06T17:52:31.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:31.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:31.147660+0000) 2024-03-06T17:52:32.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:32.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:32.147959+0000) 2024-03-06T17:52:33.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:33.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:33.148143+0000) 2024-03-06T17:52:34.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:34.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:34.148392+0000) 2024-03-06T17:52:35.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:35.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:35.148627+0000) 2024-03-06T17:52:36.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:36.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:36.148849+0000) 2024-03-06T17:52:37.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:37.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:37.149035+0000) 2024-03-06T17:52:38.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:38.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:38.149229+0000) 2024-03-06T17:52:39.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:39.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:39.149436+0000) 2024-03-06T17:52:40.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:40.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:40.149634+0000) 2024-03-06T17:52:41.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:41.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:41.149898+0000) 2024-03-06T17:52:42.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:42.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:42.150108+0000) 2024-03-06T17:52:43.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:43.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:43.150330+0000) 2024-03-06T17:52:44.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:44.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:44.150601+0000) 2024-03-06T17:52:45.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:45.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:45.150881+0000) 2024-03-06T17:52:46.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:46.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:46.151116+0000) 2024-03-06T17:52:47.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:47.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:47.151333+0000) 2024-03-06T17:52:48.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:48.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:48.151511+0000) 2024-03-06T17:52:49.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:49.124 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:49.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:49.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:49.151702+0000) 2024-03-06T17:52:50.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:50.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:50.151935+0000) 2024-03-06T17:52:51.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:51.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:51.152152+0000) 2024-03-06T17:52:52.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:52.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:52.152387+0000) 2024-03-06T17:52:53.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:53.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:53.152611+0000) 2024-03-06T17:52:54.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:54.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:54.152874+0000) 2024-03-06T17:52:55.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:55.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:55.153082+0000) 2024-03-06T17:52:56.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:56.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:56.153311+0000) 2024-03-06T17:52:57.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:57.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:57.153499+0000) 2024-03-06T17:52:58.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:58.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:58.153708+0000) 2024-03-06T17:52:59.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:52:59.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:59.153944+0000) 2024-03-06T17:53:00.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:00.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:00.154177+0000) 2024-03-06T17:53:01.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:01.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:01.154402+0000) 2024-03-06T17:53:02.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:02.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:02.154634+0000) 2024-03-06T17:53:03.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:03.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:03.154866+0000) 2024-03-06T17:53:04.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:04.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:04.155102+0000) 2024-03-06T17:53:05.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:05.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:05.155296+0000) 2024-03-06T17:53:06.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:06.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:06.155538+0000) 2024-03-06T17:53:07.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:07.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:07.155804+0000) 2024-03-06T17:53:08.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:08.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:08.155984+0000) 2024-03-06T17:53:09.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:09.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:09.156211+0000) 2024-03-06T17:53:10.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:10.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:10.156411+0000) 2024-03-06T17:53:11.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:11.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:11.156645+0000) 2024-03-06T17:53:12.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:12.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:12.156876+0000) 2024-03-06T17:53:13.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:13.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:13.157051+0000) 2024-03-06T17:53:14.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:14.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:14.157235+0000) 2024-03-06T17:53:15.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:15.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:15.157527+0000) 2024-03-06T17:53:16.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:16.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:16.157787+0000) 2024-03-06T17:53:17.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:17.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:17.158023+0000) 2024-03-06T17:53:18.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:18.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:18.158200+0000) 2024-03-06T17:53:19.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:19.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:19.158379+0000) 2024-03-06T17:53:19.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:19.336 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:20.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:20.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:20.158617+0000) 2024-03-06T17:53:21.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:21.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:21.158845+0000) 2024-03-06T17:53:22.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:22.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:22.159076+0000) 2024-03-06T17:53:23.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:23.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:23.159297+0000) 2024-03-06T17:53:24.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:24.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:24.159481+0000) 2024-03-06T17:53:25.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:25.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:25.159745+0000) 2024-03-06T17:53:26.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:26.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:26.159977+0000) 2024-03-06T17:53:27.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:27.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:27.160158+0000) 2024-03-06T17:53:28.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:28.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:28.160334+0000) 2024-03-06T17:53:29.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:29.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:29.160531+0000) 2024-03-06T17:53:30.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:30.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:30.160793+0000) 2024-03-06T17:53:31.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:31.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:31.161024+0000) 2024-03-06T17:53:32.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:32.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:32.161258+0000) 2024-03-06T17:53:33.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:33.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:33.161489+0000) 2024-03-06T17:53:34.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:34.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:34.161754+0000) 2024-03-06T17:53:35.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:35.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:35.161996+0000) 2024-03-06T17:53:36.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:36.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:36.162206+0000) 2024-03-06T17:53:37.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:37.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:37.162386+0000) 2024-03-06T17:53:38.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:38.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:38.162601+0000) 2024-03-06T17:53:39.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:39.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:39.162839+0000) 2024-03-06T17:53:40.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:40.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:40.163060+0000) 2024-03-06T17:53:41.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:41.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:41.163300+0000) 2024-03-06T17:53:42.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:42.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:42.163527+0000) 2024-03-06T17:53:43.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:43.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:43.163759+0000) 2024-03-06T17:53:44.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:44.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:44.163963+0000) 2024-03-06T17:53:45.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:45.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:45.164145+0000) 2024-03-06T17:53:46.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:46.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:46.164347+0000) 2024-03-06T17:53:47.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:47.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:47.164528+0000) 2024-03-06T17:53:48.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:48.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:48.164743+0000) 2024-03-06T17:53:49.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:49.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:49.164974+0000) 2024-03-06T17:53:49.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:49.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:50.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:50.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:50.165215+0000) 2024-03-06T17:53:51.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:51.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:51.165448+0000) 2024-03-06T17:53:52.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:52.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:52.165719+0000) 2024-03-06T17:53:53.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:53.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:53.165891+0000) 2024-03-06T17:53:54.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:54.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:54.166116+0000) 2024-03-06T17:53:55.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:55.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:55.166380+0000) 2024-03-06T17:53:56.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:56.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:56.166655+0000) 2024-03-06T17:53:57.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:57.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:57.166965+0000) 2024-03-06T17:53:58.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:58.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:58.167169+0000) 2024-03-06T17:53:59.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:53:59.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:59.167350+0000) 2024-03-06T17:54:00.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:00.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:00.167529+0000) 2024-03-06T17:54:01.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:01.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:01.167750+0000) 2024-03-06T17:54:02.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:02.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:02.167984+0000) 2024-03-06T17:54:03.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:03.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:03.168186+0000) 2024-03-06T17:54:04.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:04.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:04.168377+0000) 2024-03-06T17:54:05.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:05.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:05.168596+0000) 2024-03-06T17:54:06.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:06.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:06.168829+0000) 2024-03-06T17:54:07.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:07.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:07.169061+0000) 2024-03-06T17:54:08.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:08.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:08.169333+0000) 2024-03-06T17:54:09.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:09.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:09.169551+0000) 2024-03-06T17:54:10.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:10.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:10.169777+0000) 2024-03-06T17:54:11.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:11.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:11.169954+0000) 2024-03-06T17:54:12.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:12.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:12.170184+0000) 2024-03-06T17:54:13.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:13.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:13.170366+0000) 2024-03-06T17:54:14.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:14.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:14.170542+0000) 2024-03-06T17:54:15.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:15.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:15.170766+0000) 2024-03-06T17:54:16.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:16.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:16.170999+0000) 2024-03-06T17:54:17.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:17.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:17.171210+0000) 2024-03-06T17:54:18.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:18.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:18.171452+0000) 2024-03-06T17:54:19.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:19.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:19.171708+0000) 2024-03-06T17:54:19.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:19.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:20.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:20.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:20.171955+0000) 2024-03-06T17:54:21.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:21.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:21.172196+0000) 2024-03-06T17:54:22.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:22.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:22.172424+0000) 2024-03-06T17:54:23.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:23.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:23.172651+0000) 2024-03-06T17:54:24.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:24.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:24.172951+0000) 2024-03-06T17:54:25.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:25.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:25.173188+0000) 2024-03-06T17:54:26.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:26.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:26.173415+0000) 2024-03-06T17:54:27.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:27.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:27.173629+0000) 2024-03-06T17:54:28.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:28.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:28.173883+0000) 2024-03-06T17:54:29.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:29.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:29.174119+0000) 2024-03-06T17:54:30.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:30.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:30.174296+0000) 2024-03-06T17:54:31.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:31.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:31.174530+0000) 2024-03-06T17:54:32.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:32.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:32.174809+0000) 2024-03-06T17:54:33.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:33.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:33.175033+0000) 2024-03-06T17:54:34.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:34.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:34.175244+0000) 2024-03-06T17:54:35.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:35.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:35.175492+0000) 2024-03-06T17:54:36.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:36.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:36.175784+0000) 2024-03-06T17:54:37.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:37.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:37.175968+0000) 2024-03-06T17:54:38.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:38.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:38.176162+0000) 2024-03-06T17:54:39.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:39.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:39.176388+0000) 2024-03-06T17:54:40.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:40.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:40.176572+0000) 2024-03-06T17:54:41.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:41.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:41.176836+0000) 2024-03-06T17:54:42.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:42.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:42.177103+0000) 2024-03-06T17:54:43.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:43.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:43.177315+0000) 2024-03-06T17:54:44.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:44.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:44.177508+0000) 2024-03-06T17:54:45.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:45.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:45.177764+0000) 2024-03-06T17:54:46.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:46.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:46.177968+0000) 2024-03-06T17:54:47.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:47.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:47.178163+0000) 2024-03-06T17:54:48.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:48.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:48.178356+0000) 2024-03-06T17:54:49.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:49.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:49.178528+0000) 2024-03-06T17:54:49.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:49.958 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:50.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:50.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:50.178783+0000) 2024-03-06T17:54:51.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:51.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:51.179046+0000) 2024-03-06T17:54:52.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:52.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:52.179301+0000) 2024-03-06T17:54:53.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:53.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:53.179504+0000) 2024-03-06T17:54:54.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:54.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:54.179730+0000) 2024-03-06T17:54:55.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:55.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:55.179951+0000) 2024-03-06T17:54:56.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:56.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:56.180221+0000) 2024-03-06T17:54:57.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:57.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:57.180452+0000) 2024-03-06T17:54:58.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:58.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:58.180702+0000) 2024-03-06T17:54:59.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:54:59.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:59.180927+0000) 2024-03-06T17:55:00.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:00.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:00.181121+0000) 2024-03-06T17:55:01.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:01.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:01.181316+0000) 2024-03-06T17:55:02.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:02.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:02.181525+0000) 2024-03-06T17:55:03.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:03.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:03.181772+0000) 2024-03-06T17:55:04.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:04.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:04.182016+0000) 2024-03-06T17:55:05.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:05.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:05.182250+0000) 2024-03-06T17:55:06.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:06.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:06.182477+0000) 2024-03-06T17:55:07.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:07.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:07.182704+0000) 2024-03-06T17:55:08.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:08.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:08.182944+0000) 2024-03-06T17:55:09.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:09.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:09.183181+0000) 2024-03-06T17:55:10.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:10.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:10.183381+0000) 2024-03-06T17:55:11.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:11.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:11.183617+0000) 2024-03-06T17:55:12.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:12.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:12.183895+0000) 2024-03-06T17:55:13.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:13.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:13.184112+0000) 2024-03-06T17:55:14.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:14.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:14.184291+0000) 2024-03-06T17:55:15.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:15.183+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:15.184509+0000) 2024-03-06T17:55:16.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:16.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:16.184829+0000) 2024-03-06T17:55:17.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:17.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:17.185022+0000) 2024-03-06T17:55:18.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:18.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:18.185242+0000) 2024-03-06T17:55:19.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:19.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:19.185482+0000) 2024-03-06T17:55:20.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:20.086 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:20.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:20.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:20.185780+0000) 2024-03-06T17:55:21.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:21.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:21.186022+0000) 2024-03-06T17:55:22.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:22.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:22.186338+0000) 2024-03-06T17:55:23.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:23.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:23.186611+0000) 2024-03-06T17:55:24.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:24.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:24.186875+0000) 2024-03-06T17:55:25.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:25.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:25.187084+0000) 2024-03-06T17:55:26.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:26.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:26.187329+0000) 2024-03-06T17:55:27.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:27.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:27.187544+0000) 2024-03-06T17:55:28.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:28.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:28.187780+0000) 2024-03-06T17:55:29.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:29.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:29.188015+0000) 2024-03-06T17:55:30.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:30.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:30.188207+0000) 2024-03-06T17:55:31.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:31.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:31.188398+0000) 2024-03-06T17:55:32.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:32.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:32.188593+0000) 2024-03-06T17:55:33.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:33.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:33.188858+0000) 2024-03-06T17:55:34.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:34.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:34.189135+0000) 2024-03-06T17:55:35.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:35.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:35.189330+0000) 2024-03-06T17:55:36.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:36.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:36.189551+0000) 2024-03-06T17:55:37.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:37.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:37.189777+0000) 2024-03-06T17:55:38.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:38.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:38.189995+0000) 2024-03-06T17:55:39.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:39.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:39.190189+0000) 2024-03-06T17:55:40.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:40.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:40.190423+0000) 2024-03-06T17:55:41.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:41.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:41.190711+0000) 2024-03-06T17:55:42.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:42.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:42.190976+0000) 2024-03-06T17:55:43.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:43.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:43.191212+0000) 2024-03-06T17:55:44.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:44.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:44.191515+0000) 2024-03-06T17:55:45.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:45.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:45.191728+0000) 2024-03-06T17:55:46.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:46.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:46.191974+0000) 2024-03-06T17:55:47.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:47.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:47.192202+0000) 2024-03-06T17:55:48.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:48.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:48.192397+0000) 2024-03-06T17:55:49.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:49.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:49.192624+0000) 2024-03-06T17:55:50.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:50.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:50.192823+0000) 2024-03-06T17:55:50.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:50.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:51.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:51.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:51.193078+0000) 2024-03-06T17:55:52.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:52.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:52.193329+0000) 2024-03-06T17:55:53.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:53.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:53.193556+0000) 2024-03-06T17:55:54.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:54.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:54.193806+0000) 2024-03-06T17:55:55.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:55.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:55.194050+0000) 2024-03-06T17:55:56.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:56.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:56.194254+0000) 2024-03-06T17:55:57.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:57.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:57.194472+0000) 2024-03-06T17:55:58.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:58.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:58.194784+0000) 2024-03-06T17:55:59.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:55:59.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:59.194968+0000) 2024-03-06T17:56:00.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:00.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:00.195153+0000) 2024-03-06T17:56:01.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:01.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:01.195410+0000) 2024-03-06T17:56:02.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:02.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:02.195619+0000) 2024-03-06T17:56:03.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:03.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:03.195854+0000) 2024-03-06T17:56:04.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:04.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:04.196083+0000) 2024-03-06T17:56:05.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:05.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:05.196313+0000) 2024-03-06T17:56:06.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:06.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:06.196555+0000) 2024-03-06T17:56:07.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:07.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:07.196796+0000) 2024-03-06T17:56:08.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:08.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:08.197078+0000) 2024-03-06T17:56:09.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:09.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:09.197267+0000) 2024-03-06T17:56:10.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:10.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:10.197488+0000) 2024-03-06T17:56:11.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:11.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:11.197732+0000) 2024-03-06T17:56:12.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:12.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:12.197974+0000) 2024-03-06T17:56:13.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:13.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:13.198159+0000) 2024-03-06T17:56:14.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:14.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:14.198357+0000) 2024-03-06T17:56:15.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:15.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:15.198558+0000) 2024-03-06T17:56:16.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:16.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:16.198794+0000) 2024-03-06T17:56:17.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:17.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:17.199021+0000) 2024-03-06T17:56:18.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:18.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:18.199201+0000) 2024-03-06T17:56:19.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:19.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:19.199420+0000) 2024-03-06T17:56:20.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:20.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:20.199715+0000) 2024-03-06T17:56:20.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:20.337 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:21.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:21.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:21.199919+0000) 2024-03-06T17:56:22.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:22.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:22.200142+0000) 2024-03-06T17:56:23.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:23.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:23.200304+0000) 2024-03-06T17:56:24.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:24.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:24.200534+0000) 2024-03-06T17:56:25.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:25.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:25.200771+0000) 2024-03-06T17:56:26.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:26.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:26.201010+0000) 2024-03-06T17:56:27.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:27.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:27.201213+0000) 2024-03-06T17:56:28.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:28.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:28.201442+0000) 2024-03-06T17:56:29.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:29.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:29.201698+0000) 2024-03-06T17:56:30.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:30.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:30.201910+0000) 2024-03-06T17:56:31.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:31.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:31.202127+0000) 2024-03-06T17:56:32.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:32.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:32.202417+0000) 2024-03-06T17:56:33.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:33.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:33.202608+0000) 2024-03-06T17:56:34.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:34.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:34.202834+0000) 2024-03-06T17:56:35.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:35.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:35.203022+0000) 2024-03-06T17:56:36.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:36.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:36.203218+0000) 2024-03-06T17:56:37.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:37.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:37.203496+0000) 2024-03-06T17:56:38.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:38.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:38.203754+0000) 2024-03-06T17:56:39.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:39.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:39.203943+0000) 2024-03-06T17:56:40.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:40.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:40.204179+0000) 2024-03-06T17:56:41.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:41.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:41.204405+0000) 2024-03-06T17:56:42.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:42.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:42.204639+0000) 2024-03-06T17:56:43.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:43.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:43.204909+0000) 2024-03-06T17:56:44.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:44.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:44.205139+0000) 2024-03-06T17:56:45.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:45.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:45.205358+0000) 2024-03-06T17:56:46.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:46.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:46.205588+0000) 2024-03-06T17:56:47.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:47.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:47.205854+0000) 2024-03-06T17:56:48.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:48.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:48.206134+0000) 2024-03-06T17:56:49.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:49.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:49.206408+0000) 2024-03-06T17:56:50.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:50.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:50.206650+0000) 2024-03-06T17:56:50.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:50.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:51.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:51.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:51.206966+0000) 2024-03-06T17:56:52.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:52.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:52.207247+0000) 2024-03-06T17:56:53.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:53.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:53.207481+0000) 2024-03-06T17:56:54.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:54.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:54.207747+0000) 2024-03-06T17:56:55.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:55.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:55.207969+0000) 2024-03-06T17:56:56.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:56.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:56.208196+0000) 2024-03-06T17:56:57.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:57.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:57.208380+0000) 2024-03-06T17:56:58.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:58.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:58.208613+0000) 2024-03-06T17:56:59.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:56:59.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:59.208908+0000) 2024-03-06T17:57:00.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:00.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:00.209155+0000) 2024-03-06T17:57:01.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:01.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:01.209371+0000) 2024-03-06T17:57:02.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:02.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:02.209623+0000) 2024-03-06T17:57:03.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:03.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:03.209877+0000) 2024-03-06T17:57:04.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:04.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:04.210093+0000) 2024-03-06T17:57:05.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:05.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:05.210300+0000) 2024-03-06T17:57:06.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:06.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:06.210550+0000) 2024-03-06T17:57:07.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:07.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:07.210781+0000) 2024-03-06T17:57:08.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:08.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:08.210997+0000) 2024-03-06T17:57:09.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:09.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:09.211226+0000) 2024-03-06T17:57:10.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:10.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:10.211446+0000) 2024-03-06T17:57:11.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:11.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:11.211639+0000) 2024-03-06T17:57:12.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:12.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:12.211873+0000) 2024-03-06T17:57:13.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:13.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:13.212057+0000) 2024-03-06T17:57:14.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:14.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:14.212281+0000) 2024-03-06T17:57:15.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:15.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:15.212506+0000) 2024-03-06T17:57:16.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:16.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:16.212748+0000) 2024-03-06T17:57:17.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:17.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:17.212981+0000) 2024-03-06T17:57:18.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:18.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:18.213172+0000) 2024-03-06T17:57:19.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:19.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:19.213413+0000) 2024-03-06T17:57:20.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:20.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:20.213648+0000) 2024-03-06T17:57:20.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:20.665 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:21.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:21.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:21.213875+0000) 2024-03-06T17:57:22.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:22.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:22.214129+0000) 2024-03-06T17:57:23.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:23.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:23.214350+0000) 2024-03-06T17:57:24.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:24.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:24.214580+0000) 2024-03-06T17:57:25.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:25.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:25.214863+0000) 2024-03-06T17:57:26.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:26.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:26.215122+0000) 2024-03-06T17:57:27.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:27.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:27.215311+0000) 2024-03-06T17:57:28.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:28.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:28.215531+0000) 2024-03-06T17:57:29.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:29.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:29.215782+0000) 2024-03-06T17:57:30.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:30.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:30.216010+0000) 2024-03-06T17:57:31.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:31.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:31.216213+0000) 2024-03-06T17:57:32.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:32.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:32.216468+0000) 2024-03-06T17:57:33.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:33.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:33.216727+0000) 2024-03-06T17:57:34.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:34.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:34.216969+0000) 2024-03-06T17:57:35.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:35.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:35.217228+0000) 2024-03-06T17:57:36.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:36.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:36.217516+0000) 2024-03-06T17:57:37.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:37.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:37.217777+0000) 2024-03-06T17:57:38.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:38.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:38.217970+0000) 2024-03-06T17:57:39.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:39.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:39.218145+0000) 2024-03-06T17:57:40.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:40.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:40.218333+0000) 2024-03-06T17:57:41.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:41.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:41.218525+0000) 2024-03-06T17:57:42.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:42.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:42.218744+0000) 2024-03-06T17:57:43.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:43.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:43.218954+0000) 2024-03-06T17:57:44.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:44.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:44.219231+0000) 2024-03-06T17:57:45.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:45.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:45.219461+0000) 2024-03-06T17:57:46.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:46.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:46.219718+0000) 2024-03-06T17:57:47.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:47.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:47.219927+0000) 2024-03-06T17:57:48.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:48.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:48.220120+0000) 2024-03-06T17:57:49.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:49.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:49.220316+0000) 2024-03-06T17:57:50.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:50.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:50.220522+0000) 2024-03-06T17:57:50.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:50.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:51.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:51.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:51.220803+0000) 2024-03-06T17:57:52.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:52.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:52.221008+0000) 2024-03-06T17:57:53.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:53.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:53.221187+0000) 2024-03-06T17:57:54.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:54.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:54.221425+0000) 2024-03-06T17:57:55.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:55.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:55.221645+0000) 2024-03-06T17:57:56.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:56.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:56.221876+0000) 2024-03-06T17:57:57.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:57.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:57.222065+0000) 2024-03-06T17:57:58.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:58.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:58.222263+0000) 2024-03-06T17:57:59.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:57:59.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:59.222440+0000) 2024-03-06T17:58:00.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:00.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:00.222690+0000) 2024-03-06T17:58:01.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:01.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:01.222879+0000) 2024-03-06T17:58:02.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:02.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:02.223076+0000) 2024-03-06T17:58:03.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:03.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:03.223264+0000) 2024-03-06T17:58:04.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:04.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:04.223508+0000) 2024-03-06T17:58:05.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:05.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:05.223770+0000) 2024-03-06T17:58:06.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:06.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:06.223973+0000) 2024-03-06T17:58:07.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:07.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:07.224157+0000) 2024-03-06T17:58:08.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:08.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:08.224378+0000) 2024-03-06T17:58:09.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:09.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:09.224600+0000) 2024-03-06T17:58:10.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:10.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:10.224869+0000) 2024-03-06T17:58:11.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:11.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:11.225142+0000) 2024-03-06T17:58:12.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:12.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:12.225377+0000) 2024-03-06T17:58:13.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:13.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:13.225660+0000) 2024-03-06T17:58:14.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:14.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:14.225977+0000) 2024-03-06T17:58:15.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:15.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:15.226210+0000) 2024-03-06T17:58:16.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:16.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:16.226426+0000) 2024-03-06T17:58:17.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:17.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:17.226659+0000) 2024-03-06T17:58:18.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:18.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:18.226907+0000) 2024-03-06T17:58:19.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:19.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:19.227192+0000) 2024-03-06T17:58:20.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:20.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:20.227424+0000) 2024-03-06T17:58:20.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:20.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:21.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:21.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:21.227699+0000) 2024-03-06T17:58:22.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:22.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:22.227951+0000) 2024-03-06T17:58:23.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:23.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:23.228172+0000) 2024-03-06T17:58:24.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:24.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:24.228400+0000) 2024-03-06T17:58:25.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:25.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:25.228623+0000) 2024-03-06T17:58:26.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:26.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:26.228856+0000) 2024-03-06T17:58:27.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:27.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:27.229085+0000) 2024-03-06T17:58:28.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:28.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:28.229264+0000) 2024-03-06T17:58:29.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:29.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:29.229453+0000) 2024-03-06T17:58:30.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:30.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:30.229720+0000) 2024-03-06T17:58:31.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:31.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:31.229903+0000) 2024-03-06T17:58:32.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:32.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:32.230098+0000) 2024-03-06T17:58:33.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:33.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:33.230257+0000) 2024-03-06T17:58:34.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:34.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:34.230489+0000) 2024-03-06T17:58:35.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:35.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:35.230743+0000) 2024-03-06T17:58:36.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:36.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:36.230905+0000) 2024-03-06T17:58:37.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:37.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:37.231094+0000) 2024-03-06T17:58:38.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:38.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:38.231285+0000) 2024-03-06T17:58:39.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:39.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:39.231508+0000) 2024-03-06T17:58:40.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:40.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:40.231770+0000) 2024-03-06T17:58:41.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:41.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:41.231964+0000) 2024-03-06T17:58:42.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:42.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:42.232168+0000) 2024-03-06T17:58:43.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:43.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:43.232353+0000) 2024-03-06T17:58:44.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:44.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:44.232582+0000) 2024-03-06T17:58:45.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:45.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:45.232780+0000) 2024-03-06T17:58:46.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:46.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:46.232971+0000) 2024-03-06T17:58:47.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:47.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:47.233185+0000) 2024-03-06T17:58:48.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:48.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:48.233425+0000) 2024-03-06T17:58:49.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:49.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:49.233709+0000) 2024-03-06T17:58:50.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:50.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:50.233946+0000) 2024-03-06T17:58:51.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:51.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:51.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:51.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:51.234192+0000) 2024-03-06T17:58:52.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:52.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:52.234398+0000) 2024-03-06T17:58:53.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:53.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:53.234723+0000) 2024-03-06T17:58:54.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:54.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:54.235016+0000) 2024-03-06T17:58:55.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:55.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:55.235305+0000) 2024-03-06T17:58:56.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:56.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:56.235523+0000) 2024-03-06T17:58:57.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:57.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:57.235782+0000) 2024-03-06T17:58:58.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:58.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:58.236004+0000) 2024-03-06T17:58:59.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:58:59.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:59.236197+0000) 2024-03-06T17:59:00.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:00.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:00.236384+0000) 2024-03-06T17:59:01.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:01.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:01.236657+0000) 2024-03-06T17:59:02.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:02.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:02.236943+0000) 2024-03-06T17:59:03.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:03.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:03.237112+0000) 2024-03-06T17:59:04.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:04.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:04.237303+0000) 2024-03-06T17:59:05.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:05.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:05.237499+0000) 2024-03-06T17:59:06.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:06.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:06.237778+0000) 2024-03-06T17:59:07.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:07.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:07.238018+0000) 2024-03-06T17:59:08.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:08.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:08.238213+0000) 2024-03-06T17:59:09.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:09.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:09.238434+0000) 2024-03-06T17:59:10.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:10.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:10.238659+0000) 2024-03-06T17:59:11.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:11.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:11.238877+0000) 2024-03-06T17:59:12.240 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:12.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:12.239110+0000) 2024-03-06T17:59:13.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:13.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:13.239308+0000) 2024-03-06T17:59:14.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:14.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:14.239521+0000) 2024-03-06T17:59:15.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:15.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:15.239780+0000) 2024-03-06T17:59:16.241 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:16.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:16.239970+0000) 2024-03-06T17:59:17.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:17.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:17.240161+0000) 2024-03-06T17:59:18.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:18.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:18.240355+0000) 2024-03-06T17:59:19.242 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:19.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:19.240532+0000) 2024-03-06T17:59:20.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:20.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:20.240807+0000) 2024-03-06T17:59:21.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:21.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:21.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:21.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:21.241038+0000) 2024-03-06T17:59:22.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:22.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:22.241257+0000) 2024-03-06T17:59:23.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:23.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:23.241451+0000) 2024-03-06T17:59:24.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:24.239+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:24.241652+0000) 2024-03-06T17:59:25.243 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:25.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:25.241951+0000) 2024-03-06T17:59:26.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:26.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:26.242157+0000) 2024-03-06T17:59:27.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:27.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:27.242373+0000) 2024-03-06T17:59:28.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:28.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:28.242599+0000) 2024-03-06T17:59:29.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:29.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:29.242813+0000) 2024-03-06T17:59:30.244 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:30.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:30.243003+0000) 2024-03-06T17:59:31.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:31.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:31.243194+0000) 2024-03-06T17:59:32.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:32.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:32.243431+0000) 2024-03-06T17:59:33.245 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:33.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:33.243660+0000) 2024-03-06T17:59:34.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:34.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:34.243911+0000) 2024-03-06T17:59:35.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:35.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:35.244124+0000) 2024-03-06T17:59:36.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:36.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:36.244351+0000) 2024-03-06T17:59:37.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:37.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:37.244575+0000) 2024-03-06T17:59:38.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:38.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:38.244808+0000) 2024-03-06T17:59:39.246 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:39.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:39.245032+0000) 2024-03-06T17:59:40.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:40.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:40.245213+0000) 2024-03-06T17:59:41.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:41.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:41.245440+0000) 2024-03-06T17:59:42.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:42.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:42.245655+0000) 2024-03-06T17:59:43.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:43.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:43.245880+0000) 2024-03-06T17:59:44.247 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:44.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:44.246082+0000) 2024-03-06T17:59:45.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:45.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:45.246293+0000) 2024-03-06T17:59:46.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:46.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:46.246489+0000) 2024-03-06T17:59:47.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:47.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:47.246757+0000) 2024-03-06T17:59:48.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:48.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:48.246983+0000) 2024-03-06T17:59:49.248 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:49.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:49.247206+0000) 2024-03-06T17:59:50.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:50.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:50.247373+0000) 2024-03-06T17:59:51.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:51.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:51.247597+0000) 2024-03-06T17:59:51.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:51.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:52.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:52.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:52.247828+0000) 2024-03-06T17:59:53.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:53.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:53.248012+0000) 2024-03-06T17:59:54.249 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:54.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:54.248241+0000) 2024-03-06T17:59:55.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:55.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:55.248503+0000) 2024-03-06T17:59:56.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:56.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:56.248756+0000) 2024-03-06T17:59:57.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:57.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:57.248951+0000) 2024-03-06T17:59:58.250 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:58.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:58.249136+0000) 2024-03-06T17:59:59.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T17:59:59.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:59.249359+0000) 2024-03-06T18:00:00.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:00.248+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:00.249589+0000) 2024-03-06T18:00:01.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:01.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:01.249803+0000) 2024-03-06T18:00:02.251 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:02.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:02.250043+0000) 2024-03-06T18:00:03.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:03.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:03.250231+0000) 2024-03-06T18:00:04.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:04.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:04.250419+0000) 2024-03-06T18:00:05.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:05.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:05.250655+0000) 2024-03-06T18:00:06.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:06.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:06.250916+0000) 2024-03-06T18:00:07.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:07.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:07.251069+0000) 2024-03-06T18:00:08.252 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:08.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:08.251257+0000) 2024-03-06T18:00:09.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:09.249+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:09.251474+0000) 2024-03-06T18:00:10.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:10.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:10.251738+0000) 2024-03-06T18:00:11.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:11.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:11.251930+0000) 2024-03-06T18:00:12.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:12.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:12.252126+0000) 2024-03-06T18:00:13.253 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:13.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:13.252323+0000) 2024-03-06T18:00:14.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:14.250+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:14.252554+0000) 2024-03-06T18:00:15.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:15.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:15.252775+0000) 2024-03-06T18:00:16.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:16.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:16.253005+0000) 2024-03-06T18:00:17.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:17.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:17.253192+0000) 2024-03-06T18:00:18.254 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:18.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:18.253419+0000) 2024-03-06T18:00:19.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:19.251+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:19.253603+0000) 2024-03-06T18:00:20.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:20.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:20.253841+0000) 2024-03-06T18:00:21.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:21.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:21.254063+0000) 2024-03-06T18:00:21.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:21.582 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:22.255 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:22.252+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:22.254268+0000) 2024-03-06T18:00:23.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:23.253+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:23.254502+0000) 2024-03-06T18:00:24.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:24.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:24.254744+0000) 2024-03-06T18:00:25.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:25.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:25.254945+0000) 2024-03-06T18:00:26.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:26.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:26.255179+0000) 2024-03-06T18:00:27.256 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:27.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:27.255416+0000) 2024-03-06T18:00:28.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:28.254+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:28.255651+0000) 2024-03-06T18:00:29.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:29.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:29.255864+0000) 2024-03-06T18:00:30.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:30.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:30.256040+0000) 2024-03-06T18:00:31.257 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:31.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:31.256262+0000) 2024-03-06T18:00:32.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:32.255+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:32.256508+0000) 2024-03-06T18:00:33.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:33.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:33.256770+0000) 2024-03-06T18:00:34.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:34.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:34.256977+0000) 2024-03-06T18:00:35.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:35.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:35.257181+0000) 2024-03-06T18:00:36.258 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:36.256+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:36.257423+0000) 2024-03-06T18:00:37.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:37.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:37.257655+0000) 2024-03-06T18:00:38.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:38.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:38.257890+0000) 2024-03-06T18:00:39.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:39.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:39.258075+0000) 2024-03-06T18:00:40.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:40.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:40.258265+0000) 2024-03-06T18:00:41.259 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:41.257+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:41.258485+0000) 2024-03-06T18:00:42.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:42.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:42.258783+0000) 2024-03-06T18:00:43.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:43.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:43.259012+0000) 2024-03-06T18:00:44.260 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:44.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:44.259294+0000) 2024-03-06T18:00:45.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:45.258+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:45.259455+0000) 2024-03-06T18:00:46.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:46.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:46.259718+0000) 2024-03-06T18:00:47.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:47.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:47.259922+0000) 2024-03-06T18:00:48.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:48.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:48.260163+0000) 2024-03-06T18:00:49.261 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:49.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:49.260403+0000) 2024-03-06T18:00:50.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:50.259+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:50.260645+0000) 2024-03-06T18:00:51.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:51.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:51.260923+0000) 2024-03-06T18:00:51.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:51.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:52.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:52.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:52.261163+0000) 2024-03-06T18:00:53.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:53.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:53.261412+0000) 2024-03-06T18:00:54.262 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:54.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:54.261609+0000) 2024-03-06T18:00:55.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:55.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:55.261882+0000) 2024-03-06T18:00:56.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:56.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:56.262128+0000) 2024-03-06T18:00:57.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:57.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:57.262368+0000) 2024-03-06T18:00:58.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:58.260+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:58.262547+0000) 2024-03-06T18:00:59.263 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:00:59.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:59.262775+0000) 2024-03-06T18:01:00.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:00.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:00.263033+0000) 2024-03-06T18:01:01.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:01.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:01.263313+0000) 2024-03-06T18:01:02.264 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:02.261+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:02.263598+0000) 2024-03-06T18:01:03.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:03.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:03.263832+0000) 2024-03-06T18:01:04.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:04.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:04.264086+0000) 2024-03-06T18:01:05.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:05.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:05.264333+0000) 2024-03-06T18:01:06.265 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:06.262+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:06.264591+0000) 2024-03-06T18:01:07.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:07.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:07.264879+0000) 2024-03-06T18:01:08.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:08.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:08.265092+0000) 2024-03-06T18:01:09.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:09.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:09.265298+0000) 2024-03-06T18:01:10.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:10.263+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:10.265504+0000) 2024-03-06T18:01:11.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:11.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:11.265712+0000) 2024-03-06T18:01:12.266 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:12.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:12.265898+0000) 2024-03-06T18:01:13.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:13.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:13.266087+0000) 2024-03-06T18:01:14.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:14.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:14.266278+0000) 2024-03-06T18:01:15.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:15.265+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:15.266468+0000) 2024-03-06T18:01:16.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:16.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:16.266654+0000) 2024-03-06T18:01:17.267 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:17.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:17.266889+0000) 2024-03-06T18:01:18.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:18.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:18.267068+0000) 2024-03-06T18:01:19.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:19.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:19.267251+0000) 2024-03-06T18:01:20.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:20.266+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:20.267466+0000) 2024-03-06T18:01:21.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:21.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:21.267711+0000) 2024-03-06T18:01:21.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:21.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:22.268 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:22.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:22.267953+0000) 2024-03-06T18:01:23.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:23.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:23.268169+0000) 2024-03-06T18:01:24.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:24.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:24.268369+0000) 2024-03-06T18:01:25.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:25.267+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:25.268566+0000) 2024-03-06T18:01:26.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:26.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:26.268789+0000) 2024-03-06T18:01:27.269 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:27.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:27.269037+0000) 2024-03-06T18:01:28.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:28.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:28.269246+0000) 2024-03-06T18:01:29.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:29.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:29.269434+0000) 2024-03-06T18:01:30.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:30.268+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:30.269625+0000) 2024-03-06T18:01:31.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:31.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:31.269819+0000) 2024-03-06T18:01:32.270 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:32.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:32.270043+0000) 2024-03-06T18:01:33.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:33.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:33.270243+0000) 2024-03-06T18:01:34.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:34.269+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:34.270462+0000) 2024-03-06T18:01:35.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:35.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:35.270717+0000) 2024-03-06T18:01:36.271 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:36.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:36.270960+0000) 2024-03-06T18:01:37.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:37.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:37.271167+0000) 2024-03-06T18:01:38.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:38.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:38.271346+0000) 2024-03-06T18:01:39.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:39.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:39.271567+0000) 2024-03-06T18:01:40.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:40.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:40.271819+0000) 2024-03-06T18:01:41.272 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:41.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:41.272054+0000) 2024-03-06T18:01:42.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:42.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:42.272300+0000) 2024-03-06T18:01:43.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:43.270+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:43.272481+0000) 2024-03-06T18:01:44.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:44.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:44.272745+0000) 2024-03-06T18:01:45.273 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:45.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:45.272932+0000) 2024-03-06T18:01:46.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:46.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:46.273156+0000) 2024-03-06T18:01:47.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:47.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:47.273440+0000) 2024-03-06T18:01:48.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:48.271+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:48.273632+0000) 2024-03-06T18:01:49.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:49.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:49.273854+0000) 2024-03-06T18:01:50.274 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:50.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:50.274053+0000) 2024-03-06T18:01:51.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:51.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:51.274251+0000) 2024-03-06T18:01:52.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:52.114 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:52.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:52.272+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:52.274460+0000) 2024-03-06T18:01:53.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:53.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:53.274764+0000) 2024-03-06T18:01:54.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:54.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:54.274941+0000) 2024-03-06T18:01:55.275 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:55.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:55.275130+0000) 2024-03-06T18:01:56.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:56.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:56.275311+0000) 2024-03-06T18:01:57.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:57.273+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:57.275545+0000) 2024-03-06T18:01:58.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:58.274+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:58.275765+0000) 2024-03-06T18:01:59.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:01:59.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:59.275955+0000) 2024-03-06T18:02:00.276 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:00.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:00.276133+0000) 2024-03-06T18:02:01.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:01.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:01.276356+0000) 2024-03-06T18:02:02.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:02.275+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:02.276590+0000) 2024-03-06T18:02:03.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:03.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:03.276850+0000) 2024-03-06T18:02:04.277 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:04.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:04.277133+0000) 2024-03-06T18:02:05.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:05.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:05.277367+0000) 2024-03-06T18:02:06.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:06.276+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:06.277647+0000) 2024-03-06T18:02:07.278 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:07.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:07.277983+0000) 2024-03-06T18:02:08.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:08.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:08.278251+0000) 2024-03-06T18:02:09.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:09.277+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:09.278534+0000) 2024-03-06T18:02:10.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:10.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:10.278815+0000) 2024-03-06T18:02:11.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:11.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:11.279095+0000) 2024-03-06T18:02:12.279 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:12.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:12.279344+0000) 2024-03-06T18:02:13.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:13.278+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:13.279621+0000) 2024-03-06T18:02:14.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:14.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:14.279843+0000) 2024-03-06T18:02:15.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:15.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:15.280023+0000) 2024-03-06T18:02:16.280 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:16.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:16.280239+0000) 2024-03-06T18:02:17.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:17.279+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:17.280464+0000) 2024-03-06T18:02:18.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:18.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:18.280711+0000) 2024-03-06T18:02:19.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:19.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:19.280970+0000) 2024-03-06T18:02:20.281 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:20.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:20.281188+0000) 2024-03-06T18:02:21.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:21.280+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:21.281396+0000) 2024-03-06T18:02:22.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:22.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:22.281720+0000) 2024-03-06T18:02:22.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:22.303 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:23.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:23.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:23.281911+0000) 2024-03-06T18:02:24.282 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:24.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:24.282162+0000) 2024-03-06T18:02:25.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:25.281+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:25.282408+0000) 2024-03-06T18:02:26.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:26.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:26.282708+0000) 2024-03-06T18:02:27.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:27.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:27.283010+0000) 2024-03-06T18:02:28.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:28.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:28.283238+0000) 2024-03-06T18:02:29.283 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:29.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:29.283508+0000) 2024-03-06T18:02:30.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:30.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:30.283808+0000) 2024-03-06T18:02:31.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:31.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:31.284086+0000) 2024-03-06T18:02:32.284 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:32.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:32.284406+0000) 2024-03-06T18:02:33.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:33.282+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:33.284631+0000) 2024-03-06T18:02:34.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:34.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:34.284861+0000) 2024-03-06T18:02:35.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:35.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:35.285083+0000) 2024-03-06T18:02:36.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:36.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:36.285307+0000) 2024-03-06T18:02:37.285 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:37.283+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:37.285510+0000) 2024-03-06T18:02:38.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:38.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:38.285753+0000) 2024-03-06T18:02:39.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:39.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:39.285987+0000) 2024-03-06T18:02:40.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:40.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:40.286217+0000) 2024-03-06T18:02:41.286 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:41.284+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:41.286433+0000) 2024-03-06T18:02:42.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:42.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:42.286754+0000) 2024-03-06T18:02:43.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:43.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:43.287003+0000) 2024-03-06T18:02:44.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:44.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:44.287248+0000) 2024-03-06T18:02:45.287 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:45.285+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:45.287474+0000) 2024-03-06T18:02:46.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:46.286+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:46.287770+0000) 2024-03-06T18:02:47.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:47.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:47.288014+0000) 2024-03-06T18:02:48.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:48.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:48.288248+0000) 2024-03-06T18:02:49.288 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:49.287+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:49.288485+0000) 2024-03-06T18:02:50.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:50.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:50.288739+0000) 2024-03-06T18:02:51.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:51.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:51.288968+0000) 2024-03-06T18:02:52.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:52.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:52.289176+0000) 2024-03-06T18:02:52.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:52.442 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:53.289 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:53.288+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:53.289488+0000) 2024-03-06T18:02:54.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:54.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:54.289807+0000) 2024-03-06T18:02:55.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:55.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:55.290106+0000) 2024-03-06T18:02:56.290 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:56.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:56.290350+0000) 2024-03-06T18:02:57.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:57.289+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:57.290621+0000) 2024-03-06T18:02:58.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:58.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:58.290904+0000) 2024-03-06T18:02:59.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:02:59.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:59.291174+0000) 2024-03-06T18:03:00.291 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:00.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:00.291366+0000) 2024-03-06T18:03:01.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:01.290+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:01.291628+0000) 2024-03-06T18:03:02.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:02.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:02.291957+0000) 2024-03-06T18:03:03.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:03.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:03.292242+0000) 2024-03-06T18:03:04.292 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:04.291+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:04.292508+0000) 2024-03-06T18:03:05.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:05.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:05.292799+0000) 2024-03-06T18:03:06.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:06.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:06.293042+0000) 2024-03-06T18:03:07.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:07.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:07.293317+0000) 2024-03-06T18:03:08.293 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:08.292+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:08.293600+0000) 2024-03-06T18:03:09.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:09.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:09.293813+0000) 2024-03-06T18:03:10.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:10.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:10.294003+0000) 2024-03-06T18:03:11.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:11.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:11.294266+0000) 2024-03-06T18:03:12.294 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:12.293+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:12.294508+0000) 2024-03-06T18:03:13.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:13.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:13.294767+0000) 2024-03-06T18:03:14.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:14.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:14.294999+0000) 2024-03-06T18:03:15.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:15.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:15.295184+0000) 2024-03-06T18:03:16.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:16.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:16.295406+0000) 2024-03-06T18:03:17.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:17.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:17.295640+0000) 2024-03-06T18:03:18.295 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:18.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:18.295861+0000) 2024-03-06T18:03:19.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:19.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:19.296043+0000) 2024-03-06T18:03:20.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:20.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:20.296231+0000) 2024-03-06T18:03:21.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:21.294+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:21.296451+0000) 2024-03-06T18:03:22.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:22.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:22.296721+0000) 2024-03-06T18:03:22.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:22.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:23.296 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:23.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:23.296927+0000) 2024-03-06T18:03:24.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:24.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:24.297137+0000) 2024-03-06T18:03:25.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:25.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:25.297371+0000) 2024-03-06T18:03:26.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:26.295+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:26.297598+0000) 2024-03-06T18:03:27.297 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:27.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:27.297825+0000) 2024-03-06T18:03:28.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:28.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:28.298020+0000) 2024-03-06T18:03:29.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:29.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:29.298209+0000) 2024-03-06T18:03:30.298 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:30.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:30.298430+0000) 2024-03-06T18:03:31.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:31.296+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:31.298651+0000) 2024-03-06T18:03:32.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:32.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:32.298927+0000) 2024-03-06T18:03:33.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:33.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:33.299118+0000) 2024-03-06T18:03:34.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:34.297+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:34.299304+0000) 2024-03-06T18:03:35.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:35.298+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:35.299520+0000) 2024-03-06T18:03:36.299 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:36.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:36.299765+0000) 2024-03-06T18:03:37.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:37.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:37.299955+0000) 2024-03-06T18:03:38.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:38.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:38.300145+0000) 2024-03-06T18:03:39.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:39.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:39.300369+0000) 2024-03-06T18:03:40.300 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:40.299+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:40.300595+0000) 2024-03-06T18:03:41.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:41.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:41.300850+0000) 2024-03-06T18:03:42.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:42.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:42.301085+0000) 2024-03-06T18:03:43.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:43.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:43.301281+0000) 2024-03-06T18:03:44.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:44.300+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:44.301519+0000) 2024-03-06T18:03:45.301 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:45.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:45.301824+0000) 2024-03-06T18:03:46.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:46.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:46.302051+0000) 2024-03-06T18:03:47.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:47.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:47.302332+0000) 2024-03-06T18:03:48.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:48.301+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:48.302603+0000) 2024-03-06T18:03:49.302 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:49.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:49.302880+0000) 2024-03-06T18:03:50.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:50.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:50.303131+0000) 2024-03-06T18:03:51.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:51.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:51.303320+0000) 2024-03-06T18:03:52.303 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:52.302+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:52.303593+0000) 2024-03-06T18:03:52.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:52.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:53.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:53.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:53.303831+0000) 2024-03-06T18:03:54.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:54.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:54.304014+0000) 2024-03-06T18:03:55.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:55.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:55.304200+0000) 2024-03-06T18:03:56.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:56.303+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:56.304433+0000) 2024-03-06T18:03:57.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:57.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:57.304701+0000) 2024-03-06T18:03:58.304 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:58.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:58.304893+0000) 2024-03-06T18:03:59.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:03:59.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:59.305064+0000) 2024-03-06T18:04:00.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:00.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:00.305294+0000) 2024-03-06T18:04:01.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:01.304+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:01.305556+0000) 2024-03-06T18:04:02.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:02.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:02.305792+0000) 2024-03-06T18:04:03.305 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:03.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:03.305989+0000) 2024-03-06T18:04:04.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:04.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:04.306177+0000) 2024-03-06T18:04:05.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:05.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:05.306451+0000) 2024-03-06T18:04:06.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:06.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:06.306689+0000) 2024-03-06T18:04:07.306 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:07.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:07.306949+0000) 2024-03-06T18:04:08.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:08.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:08.307178+0000) 2024-03-06T18:04:09.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:09.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:09.307386+0000) 2024-03-06T18:04:10.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:10.305+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:10.307572+0000) 2024-03-06T18:04:11.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:11.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:11.307797+0000) 2024-03-06T18:04:12.307 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:12.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:12.308032+0000) 2024-03-06T18:04:13.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:13.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:13.308256+0000) 2024-03-06T18:04:14.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:14.306+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:14.308456+0000) 2024-03-06T18:04:15.308 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:15.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:15.308713+0000) 2024-03-06T18:04:16.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:16.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:16.309155+0000) 2024-03-06T18:04:17.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:17.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:17.309397+0000) 2024-03-06T18:04:18.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:18.307+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:18.309631+0000) 2024-03-06T18:04:19.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:19.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:19.309811+0000) 2024-03-06T18:04:20.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:20.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:20.310013+0000) 2024-03-06T18:04:21.309 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:21.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:21.310201+0000) 2024-03-06T18:04:22.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:22.308+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:22.310477+0000) 2024-03-06T18:04:22.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:22.963 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:23.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:23.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:23.310777+0000) 2024-03-06T18:04:24.310 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:24.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:24.311022+0000) 2024-03-06T18:04:25.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:25.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:25.311259+0000) 2024-03-06T18:04:26.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:26.310+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:26.311509+0000) 2024-03-06T18:04:27.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:27.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:27.311820+0000) 2024-03-06T18:04:28.311 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:28.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:28.312064+0000) 2024-03-06T18:04:29.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:29.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:29.312299+0000) 2024-03-06T18:04:30.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:30.311+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:30.312585+0000) 2024-03-06T18:04:31.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:31.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:31.312842+0000) 2024-03-06T18:04:32.312 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:32.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:32.313110+0000) 2024-03-06T18:04:33.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:33.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:33.313303+0000) 2024-03-06T18:04:34.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:34.312+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:34.313581+0000) 2024-03-06T18:04:35.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:35.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:35.313849+0000) 2024-03-06T18:04:36.313 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:36.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:36.314094+0000) 2024-03-06T18:04:37.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:37.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:37.314365+0000) 2024-03-06T18:04:38.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:38.313+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:38.314612+0000) 2024-03-06T18:04:39.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:39.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:39.314814+0000) 2024-03-06T18:04:40.314 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:40.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:40.315006+0000) 2024-03-06T18:04:41.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:41.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:41.315261+0000) 2024-03-06T18:04:42.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:42.314+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:42.315527+0000) 2024-03-06T18:04:43.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:43.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:43.315737+0000) 2024-03-06T18:04:44.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:44.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:44.315944+0000) 2024-03-06T18:04:45.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:45.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:45.316164+0000) 2024-03-06T18:04:46.315 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:46.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:46.316390+0000) 2024-03-06T18:04:47.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:47.315+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:47.316579+0000) 2024-03-06T18:04:48.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:48.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:48.316790+0000) 2024-03-06T18:04:49.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:49.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:49.316968+0000) 2024-03-06T18:04:50.316 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:50.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:50.317178+0000) 2024-03-06T18:04:51.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:51.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:51.317458+0000) 2024-03-06T18:04:52.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:52.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:52.317773+0000) 2024-03-06T18:04:53.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:53.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:53.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:53.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:53.318082+0000) 2024-03-06T18:04:54.317 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:54.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:54.318326+0000) 2024-03-06T18:04:55.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:55.316+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:55.318546+0000) 2024-03-06T18:04:56.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:56.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:56.318777+0000) 2024-03-06T18:04:57.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:57.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:57.318966+0000) 2024-03-06T18:04:58.318 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:58.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:58.319186+0000) 2024-03-06T18:04:59.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:04:59.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:59.319407+0000) 2024-03-06T18:05:00.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:00.317+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:00.319636+0000) 2024-03-06T18:05:01.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:01.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:01.319916+0000) 2024-03-06T18:05:02.319 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:02.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:02.320201+0000) 2024-03-06T18:05:03.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:03.318+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:03.320490+0000) 2024-03-06T18:05:04.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:04.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:04.320818+0000) 2024-03-06T18:05:05.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:05.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:05.321021+0000) 2024-03-06T18:05:06.320 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:06.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:06.321245+0000) 2024-03-06T18:05:07.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:07.319+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:07.321469+0000) 2024-03-06T18:05:08.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:08.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:08.321773+0000) 2024-03-06T18:05:09.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:09.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:09.322015+0000) 2024-03-06T18:05:10.321 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:10.320+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:10.322289+0000) 2024-03-06T18:05:11.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:11.321+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:11.322552+0000) 2024-03-06T18:05:12.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:12.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:12.322840+0000) 2024-03-06T18:05:13.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:13.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:13.323078+0000) 2024-03-06T18:05:14.322 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:14.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:14.323358+0000) 2024-03-06T18:05:15.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:15.322+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:15.323626+0000) 2024-03-06T18:05:16.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:16.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:16.323906+0000) 2024-03-06T18:05:17.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:17.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:17.324183+0000) 2024-03-06T18:05:18.323 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:18.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:18.324421+0000) 2024-03-06T18:05:19.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:19.323+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:19.324640+0000) 2024-03-06T18:05:20.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:20.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:20.324885+0000) 2024-03-06T18:05:21.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:21.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:21.325115+0000) 2024-03-06T18:05:22.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:22.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:22.325346+0000) 2024-03-06T18:05:23.324 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:23.324+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:23.325577+0000) 2024-03-06T18:05:23.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:23.403 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:24.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:24.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:24.325779+0000) 2024-03-06T18:05:25.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:25.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:25.325976+0000) 2024-03-06T18:05:26.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:26.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:26.326174+0000) 2024-03-06T18:05:27.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:27.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:27.326317+0000) 2024-03-06T18:05:28.325 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:28.325+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:28.326532+0000) 2024-03-06T18:05:29.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:29.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:29.326783+0000) 2024-03-06T18:05:30.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:30.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:30.327016+0000) 2024-03-06T18:05:31.326 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:31.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:31.327294+0000) 2024-03-06T18:05:32.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:32.326+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:32.327582+0000) 2024-03-06T18:05:33.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:33.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:33.327837+0000) 2024-03-06T18:05:34.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:34.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:34.328035+0000) 2024-03-06T18:05:35.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:35.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:35.328246+0000) 2024-03-06T18:05:36.327 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:36.327+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:36.328473+0000) 2024-03-06T18:05:37.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:37.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:37.328750+0000) 2024-03-06T18:05:38.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:38.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:38.328949+0000) 2024-03-06T18:05:39.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:39.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:39.329197+0000) 2024-03-06T18:05:40.328 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:40.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:40.329452+0000) 2024-03-06T18:05:41.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:41.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:41.329724+0000) 2024-03-06T18:05:42.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:42.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:42.330012+0000) 2024-03-06T18:05:43.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:43.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:43.330249+0000) 2024-03-06T18:05:44.329 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:44.328+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:44.330482+0000) 2024-03-06T18:05:45.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:45.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:45.330780+0000) 2024-03-06T18:05:46.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:46.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:46.331059+0000) 2024-03-06T18:05:47.330 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:47.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:47.331381+0000) 2024-03-06T18:05:48.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:48.329+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:48.331634+0000) 2024-03-06T18:05:49.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:49.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:49.331893+0000) 2024-03-06T18:05:50.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:50.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:50.332160+0000) 2024-03-06T18:05:51.331 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:51.330+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:51.332439+0000) 2024-03-06T18:05:52.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:52.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:52.332714+0000) 2024-03-06T18:05:53.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:53.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:53.332932+0000) 2024-03-06T18:05:53.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:53.624 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:54.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:54.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:54.333162+0000) 2024-03-06T18:05:55.332 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:55.331+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:55.333386+0000) 2024-03-06T18:05:56.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:56.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:56.333651+0000) 2024-03-06T18:05:57.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:57.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:57.333934+0000) 2024-03-06T18:05:58.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:58.332+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:58.334182+0000) 2024-03-06T18:05:59.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:05:59.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:59.334402+0000) 2024-03-06T18:06:00.333 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:00.333+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:00.334640+0000) 2024-03-06T18:06:01.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:01.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:01.334930+0000) 2024-03-06T18:06:02.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:02.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:02.335236+0000) 2024-03-06T18:06:03.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:03.334+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:03.335486+0000) 2024-03-06T18:06:04.334 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:04.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:04.335763+0000) 2024-03-06T18:06:05.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:05.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:05.335982+0000) 2024-03-06T18:06:06.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:06.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:06.336262+0000) 2024-03-06T18:06:07.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:07.335+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:07.336498+0000) 2024-03-06T18:06:08.335 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:08.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:08.336729+0000) 2024-03-06T18:06:09.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:09.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:09.336951+0000) 2024-03-06T18:06:10.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:10.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:10.337140+0000) 2024-03-06T18:06:11.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:11.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:11.337337+0000) 2024-03-06T18:06:12.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:12.336+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:12.337586+0000) 2024-03-06T18:06:13.336 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:13.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:13.337839+0000) 2024-03-06T18:06:14.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:14.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:14.338082+0000) 2024-03-06T18:06:15.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:15.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:15.338325+0000) 2024-03-06T18:06:16.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:16.337+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:16.338572+0000) 2024-03-06T18:06:17.337 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:17.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:17.338807+0000) 2024-03-06T18:06:18.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:18.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:18.339014+0000) 2024-03-06T18:06:19.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:19.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:19.339247+0000) 2024-03-06T18:06:20.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:20.338+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:20.339476+0000) 2024-03-06T18:06:21.338 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:21.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:21.339716+0000) 2024-03-06T18:06:22.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:22.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:22.339953+0000) 2024-03-06T18:06:23.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:23.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:23.340152+0000) 2024-03-06T18:06:23.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:23.831 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:24.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:24.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:24.340388+0000) 2024-03-06T18:06:25.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:25.339+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:25.340609+0000) 2024-03-06T18:06:26.339 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:26.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:26.340836+0000) 2024-03-06T18:06:27.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:27.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:27.341072+0000) 2024-03-06T18:06:28.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:28.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:28.341311+0000) 2024-03-06T18:06:29.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:29.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:29.341539+0000) 2024-03-06T18:06:30.340 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:30.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:30.341799+0000) 2024-03-06T18:06:31.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:31.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:31.342031+0000) 2024-03-06T18:06:32.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:32.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:32.342263+0000) 2024-03-06T18:06:33.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:33.340+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:33.342506+0000) 2024-03-06T18:06:34.341 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:34.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:34.342776+0000) 2024-03-06T18:06:35.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:35.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:35.343024+0000) 2024-03-06T18:06:36.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:36.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:36.343207+0000) 2024-03-06T18:06:37.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:37.341+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:37.343438+0000) 2024-03-06T18:06:38.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:38.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:38.343706+0000) 2024-03-06T18:06:39.342 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:39.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:39.343928+0000) 2024-03-06T18:06:40.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:40.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:40.344158+0000) 2024-03-06T18:06:41.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:41.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:41.344378+0000) 2024-03-06T18:06:42.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:42.342+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:42.344580+0000) 2024-03-06T18:06:43.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:43.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:43.344842+0000) 2024-03-06T18:06:44.343 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:44.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:44.345079+0000) 2024-03-06T18:06:45.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:45.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:45.345305+0000) 2024-03-06T18:06:46.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:46.343+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:46.345546+0000) 2024-03-06T18:06:47.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:47.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:47.345811+0000) 2024-03-06T18:06:48.344 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:48.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:48.346012+0000) 2024-03-06T18:06:49.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:49.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:49.346232+0000) 2024-03-06T18:06:50.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:50.345+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:50.346457+0000) 2024-03-06T18:06:51.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:51.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:51.346717+0000) 2024-03-06T18:06:52.345 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:52.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:52.346917+0000) 2024-03-06T18:06:53.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:53.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:53.347106+0000) 2024-03-06T18:06:53.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:53.974 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:54.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:54.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:54.347344+0000) 2024-03-06T18:06:55.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:55.346+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:55.347568+0000) 2024-03-06T18:06:56.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:56.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:56.347789+0000) 2024-03-06T18:06:57.346 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:57.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:57.347979+0000) 2024-03-06T18:06:58.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:58.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:58.348221+0000) 2024-03-06T18:06:59.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:06:59.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:59.348416+0000) 2024-03-06T18:07:00.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:00.347+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:00.348596+0000) 2024-03-06T18:07:01.347 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:01.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:01.348818+0000) 2024-03-06T18:07:02.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:02.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:02.349040+0000) 2024-03-06T18:07:03.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:03.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:03.349329+0000) 2024-03-06T18:07:04.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:04.348+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:04.349541+0000) 2024-03-06T18:07:05.348 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:05.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:05.349805+0000) 2024-03-06T18:07:06.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:06.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:06.350093+0000) 2024-03-06T18:07:07.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:07.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:07.350327+0000) 2024-03-06T18:07:08.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:08.349+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:08.350566+0000) 2024-03-06T18:07:09.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:09.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:09.350786+0000) 2024-03-06T18:07:10.349 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:10.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:10.351001+0000) 2024-03-06T18:07:11.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:11.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:11.351214+0000) 2024-03-06T18:07:12.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:12.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:12.351406+0000) 2024-03-06T18:07:13.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:13.350+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:13.351643+0000) 2024-03-06T18:07:14.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:14.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:14.351889+0000) 2024-03-06T18:07:15.350 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:15.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:15.352087+0000) 2024-03-06T18:07:16.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:16.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:16.352271+0000) 2024-03-06T18:07:17.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:17.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:17.352515+0000) 2024-03-06T18:07:18.351 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:18.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:18.352789+0000) 2024-03-06T18:07:19.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:19.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:19.353023+0000) 2024-03-06T18:07:20.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:20.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:20.353228+0000) 2024-03-06T18:07:21.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:21.351+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:21.353475+0000) 2024-03-06T18:07:22.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:22.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:22.353772+0000) 2024-03-06T18:07:23.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:23.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:23.354010+0000) 2024-03-06T18:07:24.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:24.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:24.352 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:24.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:24.354204+0000) 2024-03-06T18:07:25.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:25.352+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:25.354455+0000) 2024-03-06T18:07:26.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:26.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:26.354654+0000) 2024-03-06T18:07:27.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:27.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:27.354929+0000) 2024-03-06T18:07:28.353 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:28.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:28.355238+0000) 2024-03-06T18:07:29.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:29.353+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:29.355481+0000) 2024-03-06T18:07:30.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:30.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:30.355722+0000) 2024-03-06T18:07:31.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:31.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:31.355961+0000) 2024-03-06T18:07:32.354 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:32.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:32.356153+0000) 2024-03-06T18:07:33.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:33.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:33.356409+0000) 2024-03-06T18:07:34.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:34.354+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:34.356594+0000) 2024-03-06T18:07:35.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:35.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:35.356829+0000) 2024-03-06T18:07:36.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:36.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:36.357026+0000) 2024-03-06T18:07:37.355 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:37.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:37.357255+0000) 2024-03-06T18:07:38.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:38.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:38.357450+0000) 2024-03-06T18:07:39.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:39.356+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:39.357651+0000) 2024-03-06T18:07:40.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:40.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:40.357876+0000) 2024-03-06T18:07:41.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:41.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:41.358076+0000) 2024-03-06T18:07:42.356 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:42.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:42.358282+0000) 2024-03-06T18:07:43.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:43.357+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:43.358522+0000) 2024-03-06T18:07:44.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:44.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:44.358787+0000) 2024-03-06T18:07:45.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:45.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:45.359003+0000) 2024-03-06T18:07:46.357 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:46.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:46.359303+0000) 2024-03-06T18:07:47.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:47.358+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:47.359583+0000) 2024-03-06T18:07:48.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:48.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:48.359864+0000) 2024-03-06T18:07:49.358 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:49.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:49.360108+0000) 2024-03-06T18:07:50.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:50.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:50.360319+0000) 2024-03-06T18:07:51.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:51.359+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:51.360566+0000) 2024-03-06T18:07:52.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:52.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:52.360811+0000) 2024-03-06T18:07:53.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:53.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:53.361070+0000) 2024-03-06T18:07:54.359 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:54.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:54.361323+0000) 2024-03-06T18:07:54.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:54.398 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:55.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:55.360+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:55.361561+0000) 2024-03-06T18:07:56.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:56.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:56.361803+0000) 2024-03-06T18:07:57.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:57.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:57.362030+0000) 2024-03-06T18:07:58.360 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:58.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:58.362279+0000) 2024-03-06T18:07:59.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:07:59.361+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:59.362500+0000) 2024-03-06T18:08:00.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:00.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:00.362773+0000) 2024-03-06T18:08:01.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:01.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:01.363011+0000) 2024-03-06T18:08:02.361 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:02.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:02.363269+0000) 2024-03-06T18:08:03.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:03.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:03.363516+0000) 2024-03-06T18:08:04.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:04.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:04.363787+0000) 2024-03-06T18:08:05.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:05.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:05.363974+0000) 2024-03-06T18:08:06.362 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:06.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:06.364218+0000) 2024-03-06T18:08:07.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:07.362+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:07.364429+0000) 2024-03-06T18:08:08.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:08.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:08.364706+0000) 2024-03-06T18:08:09.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:09.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:09.364904+0000) 2024-03-06T18:08:10.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:10.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:10.365106+0000) 2024-03-06T18:08:11.363 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:11.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:11.365299+0000) 2024-03-06T18:08:12.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:12.363+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:12.365495+0000) 2024-03-06T18:08:13.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:13.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:13.365771+0000) 2024-03-06T18:08:14.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:14.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:14.365977+0000) 2024-03-06T18:08:15.364 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:15.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:15.366204+0000) 2024-03-06T18:08:16.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:16.364+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:16.366502+0000) 2024-03-06T18:08:17.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:17.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:17.366761+0000) 2024-03-06T18:08:18.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:18.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:18.366964+0000) 2024-03-06T18:08:19.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:19.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:19.367201+0000) 2024-03-06T18:08:20.365 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:20.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:20.367427+0000) 2024-03-06T18:08:21.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:21.365+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:21.367646+0000) 2024-03-06T18:08:22.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:22.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:22.367956+0000) 2024-03-06T18:08:23.366 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:23.366+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:23.368192+0000) 2024-03-06T18:08:24.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:24.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:24.368413+0000) 2024-03-06T18:08:24.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:24.612 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:25.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:25.367+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:25.368639+0000) 2024-03-06T18:08:26.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:26.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:26.368884+0000) 2024-03-06T18:08:27.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:27.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:27.369059+0000) 2024-03-06T18:08:28.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:28.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:28.369292+0000) 2024-03-06T18:08:29.367 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:29.368+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:29.369513+0000) 2024-03-06T18:08:30.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:30.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:30.369739+0000) 2024-03-06T18:08:31.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:31.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:31.369966+0000) 2024-03-06T18:08:32.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:32.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:32.370198+0000) 2024-03-06T18:08:33.368 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:33.369+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:33.370433+0000) 2024-03-06T18:08:34.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:34.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:34.370720+0000) 2024-03-06T18:08:35.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:35.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:35.370953+0000) 2024-03-06T18:08:36.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:36.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:36.371135+0000) 2024-03-06T18:08:37.369 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:37.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:37.371351+0000) 2024-03-06T18:08:38.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:38.370+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:38.371593+0000) 2024-03-06T18:08:39.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:39.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:39.371838+0000) 2024-03-06T18:08:40.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:40.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:40.372018+0000) 2024-03-06T18:08:41.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:41.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:41.372243+0000) 2024-03-06T18:08:42.370 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:42.371+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:42.372479+0000) 2024-03-06T18:08:43.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:43.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:43.372786+0000) 2024-03-06T18:08:44.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:44.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:44.372986+0000) 2024-03-06T18:08:45.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:45.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:45.373210+0000) 2024-03-06T18:08:46.371 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:46.372+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:46.373448+0000) 2024-03-06T18:08:47.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:47.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:47.373717+0000) 2024-03-06T18:08:48.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:48.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:48.373914+0000) 2024-03-06T18:08:49.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:49.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:49.374135+0000) 2024-03-06T18:08:50.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:50.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:50.374359+0000) 2024-03-06T18:08:51.372 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:51.373+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:51.374575+0000) 2024-03-06T18:08:52.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:52.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:52.374807+0000) 2024-03-06T18:08:53.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:53.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:53.375040+0000) 2024-03-06T18:08:54.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:54.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:54.375263+0000) 2024-03-06T18:08:54.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:54.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:55.373 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:55.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:55.375506+0000) 2024-03-06T18:08:56.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:56.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:56.375791+0000) 2024-03-06T18:08:57.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:57.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:57.375993+0000) 2024-03-06T18:08:58.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:58.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:58.376201+0000) 2024-03-06T18:08:59.374 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:08:59.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:59.376394+0000) 2024-03-06T18:09:00.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:00.374+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:00.376617+0000) 2024-03-06T18:09:01.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:01.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:01.376839+0000) 2024-03-06T18:09:02.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:02.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:02.377072+0000) 2024-03-06T18:09:03.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:03.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:03.377261+0000) 2024-03-06T18:09:04.375 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:04.375+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:04.377562+0000) 2024-03-06T18:09:05.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:05.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:05.377799+0000) 2024-03-06T18:09:06.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:06.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:06.378022+0000) 2024-03-06T18:09:07.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:07.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:07.378250+0000) 2024-03-06T18:09:08.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:08.376+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:08.378487+0000) 2024-03-06T18:09:09.376 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:09.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:09.378749+0000) 2024-03-06T18:09:10.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:10.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:10.378985+0000) 2024-03-06T18:09:11.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:11.377+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:11.379204+0000) 2024-03-06T18:09:12.377 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:12.378+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:12.379438+0000) 2024-03-06T18:09:13.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:13.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:13.379709+0000) 2024-03-06T18:09:14.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:14.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:14.379948+0000) 2024-03-06T18:09:15.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:15.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:15.380181+0000) 2024-03-06T18:09:16.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:16.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:16.380398+0000) 2024-03-06T18:09:17.378 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:17.379+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:17.380570+0000) 2024-03-06T18:09:18.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:18.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:18.380822+0000) 2024-03-06T18:09:19.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:19.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:19.381036+0000) 2024-03-06T18:09:20.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:20.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:20.381246+0000) 2024-03-06T18:09:21.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:21.380+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:21.381433+0000) 2024-03-06T18:09:22.379 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:22.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:22.381723+0000) 2024-03-06T18:09:23.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:23.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:23.381931+0000) 2024-03-06T18:09:24.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:24.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:24.382159+0000) 2024-03-06T18:09:24.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:24.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:25.380 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:25.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:25.382397+0000) 2024-03-06T18:09:26.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:26.381+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:26.382624+0000) 2024-03-06T18:09:27.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:27.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:27.382811+0000) 2024-03-06T18:09:28.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:28.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:28.383012+0000) 2024-03-06T18:09:29.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:29.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:29.383255+0000) 2024-03-06T18:09:30.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:30.382+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:30.383482+0000) 2024-03-06T18:09:31.381 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:31.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:31.383768+0000) 2024-03-06T18:09:32.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:32.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:32.384005+0000) 2024-03-06T18:09:33.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:33.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:33.384234+0000) 2024-03-06T18:09:34.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:34.383+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:34.384460+0000) 2024-03-06T18:09:35.382 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:35.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:35.384705+0000) 2024-03-06T18:09:36.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:36.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:36.384909+0000) 2024-03-06T18:09:37.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:37.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:37.385079+0000) 2024-03-06T18:09:38.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:38.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:38.385286+0000) 2024-03-06T18:09:39.383 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:39.384+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:39.385520+0000) 2024-03-06T18:09:40.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:40.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:40.385783+0000) 2024-03-06T18:09:41.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:41.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:41.386000+0000) 2024-03-06T18:09:42.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:42.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:42.386246+0000) 2024-03-06T18:09:43.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:43.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:43.386479+0000) 2024-03-06T18:09:44.384 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:44.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:44.386742+0000) 2024-03-06T18:09:45.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:45.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:45.386960+0000) 2024-03-06T18:09:46.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:46.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:46.387141+0000) 2024-03-06T18:09:47.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:47.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:47.387360+0000) 2024-03-06T18:09:48.385 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:48.385+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:48.387543+0000) 2024-03-06T18:09:49.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:49.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:49.387770+0000) 2024-03-06T18:09:50.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:50.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:50.388001+0000) 2024-03-06T18:09:51.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:51.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:51.388182+0000) 2024-03-06T18:09:52.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:52.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:52.388407+0000) 2024-03-06T18:09:53.386 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:53.386+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:53.388635+0000) 2024-03-06T18:09:54.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:54.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:54.388913+0000) 2024-03-06T18:09:55.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:55.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:55.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:55.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:55.389159+0000) 2024-03-06T18:09:56.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:56.387+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:56.389405+0000) 2024-03-06T18:09:57.387 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:57.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:57.389725+0000) 2024-03-06T18:09:58.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:58.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:58.389972+0000) 2024-03-06T18:09:59.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:09:59.388+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:59.390210+0000) 2024-03-06T18:10:00.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:00.389+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:00.390433+0000) 2024-03-06T18:10:01.388 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:01.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:01.390732+0000) 2024-03-06T18:10:02.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:02.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:02.390981+0000) 2024-03-06T18:10:03.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:03.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:03.391210+0000) 2024-03-06T18:10:04.389 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:04.390+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:04.391438+0000) 2024-03-06T18:10:05.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:05.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:05.391733+0000) 2024-03-06T18:10:06.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:06.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:06.391974+0000) 2024-03-06T18:10:07.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:07.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:07.392209+0000) 2024-03-06T18:10:08.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:08.391+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:08.392448+0000) 2024-03-06T18:10:09.390 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:09.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:09.392760+0000) 2024-03-06T18:10:10.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:10.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:10.392997+0000) 2024-03-06T18:10:11.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:11.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:11.393255+0000) 2024-03-06T18:10:12.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:12.392+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:12.393548+0000) 2024-03-06T18:10:13.391 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:13.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:13.393798+0000) 2024-03-06T18:10:14.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:14.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:14.394022+0000) 2024-03-06T18:10:15.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:15.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:15.394230+0000) 2024-03-06T18:10:16.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:16.393+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:16.394501+0000) 2024-03-06T18:10:17.392 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:17.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:17.394800+0000) 2024-03-06T18:10:18.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:18.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:18.395017+0000) 2024-03-06T18:10:19.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:19.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:19.395209+0000) 2024-03-06T18:10:20.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:20.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:20.395376+0000) 2024-03-06T18:10:21.393 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:21.394+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:21.395594+0000) 2024-03-06T18:10:22.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:22.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:22.395838+0000) 2024-03-06T18:10:23.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:23.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:23.396054+0000) 2024-03-06T18:10:24.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:24.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:24.396316+0000) 2024-03-06T18:10:25.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:25.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:25.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:25.395+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:25.396551+0000) 2024-03-06T18:10:26.394 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:26.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:26.396841+0000) 2024-03-06T18:10:27.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:27.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:27.397070+0000) 2024-03-06T18:10:28.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:28.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:28.397350+0000) 2024-03-06T18:10:29.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:29.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:29.397555+0000) 2024-03-06T18:10:30.395 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:30.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:30.397800+0000) 2024-03-06T18:10:31.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:31.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:31.398020+0000) 2024-03-06T18:10:32.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:32.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:32.398258+0000) 2024-03-06T18:10:33.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:33.396+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:33.398477+0000) 2024-03-06T18:10:34.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:34.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:34.398739+0000) 2024-03-06T18:10:35.396 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:35.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:35.398997+0000) 2024-03-06T18:10:36.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:36.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:36.399228+0000) 2024-03-06T18:10:37.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:37.397+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:37.399432+0000) 2024-03-06T18:10:38.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:38.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:38.399710+0000) 2024-03-06T18:10:39.397 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:39.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:39.399910+0000) 2024-03-06T18:10:40.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:40.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:40.400100+0000) 2024-03-06T18:10:41.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:41.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:41.400288+0000) 2024-03-06T18:10:42.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:42.398+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:42.400532+0000) 2024-03-06T18:10:43.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:43.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:43.400804+0000) 2024-03-06T18:10:44.398 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:44.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:44.400997+0000) 2024-03-06T18:10:45.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:45.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:45.401217+0000) 2024-03-06T18:10:46.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:46.399+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:46.401439+0000) 2024-03-06T18:10:47.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:47.400+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:47.401694+0000) 2024-03-06T18:10:48.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:48.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:48.401845+0000) 2024-03-06T18:10:49.399 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:49.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:49.402044+0000) 2024-03-06T18:10:50.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:50.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:50.402231+0000) 2024-03-06T18:10:51.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:51.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:51.402407+0000) 2024-03-06T18:10:52.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:52.401+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:52.402642+0000) 2024-03-06T18:10:53.400 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:53.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:53.402902+0000) 2024-03-06T18:10:54.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:54.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:54.403136+0000) 2024-03-06T18:10:55.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:55.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:55.403323+0000) 2024-03-06T18:10:55.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:55.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:56.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:56.402+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:56.403559+0000) 2024-03-06T18:10:57.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:57.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:57.403774+0000) 2024-03-06T18:10:58.401 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:58.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:58.403942+0000) 2024-03-06T18:10:59.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:10:59.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:59.404137+0000) 2024-03-06T18:11:00.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:00.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:00.404315+0000) 2024-03-06T18:11:01.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:01.403+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:01.404529+0000) 2024-03-06T18:11:02.402 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:02.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:02.404763+0000) 2024-03-06T18:11:03.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:03.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:03.404935+0000) 2024-03-06T18:11:04.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:04.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:04.405162+0000) 2024-03-06T18:11:05.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:05.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:05.405392+0000) 2024-03-06T18:11:06.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:06.404+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:06.405616+0000) 2024-03-06T18:11:07.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:07.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:07.405826+0000) 2024-03-06T18:11:08.403 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:08.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:08.406024+0000) 2024-03-06T18:11:09.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:09.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:09.406254+0000) 2024-03-06T18:11:10.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:10.405+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:10.406445+0000) 2024-03-06T18:11:11.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:11.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:11.406696+0000) 2024-03-06T18:11:12.404 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:12.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:12.406935+0000) 2024-03-06T18:11:13.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:13.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:13.407178+0000) 2024-03-06T18:11:14.405 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:14.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:14.407402+0000) 2024-03-06T18:11:15.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:15.406+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:15.407627+0000) 2024-03-06T18:11:16.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:16.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:16.407902+0000) 2024-03-06T18:11:17.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:17.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:17.408161+0000) 2024-03-06T18:11:18.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:18.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:18.408403+0000) 2024-03-06T18:11:19.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:19.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:19.408660+0000) 2024-03-06T18:11:20.406 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:20.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:20.408970+0000) 2024-03-06T18:11:21.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:21.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:21.409180+0000) 2024-03-06T18:11:22.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:22.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:22.409419+0000) 2024-03-06T18:11:23.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:23.407+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:23.409594+0000) 2024-03-06T18:11:24.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:24.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:24.409835+0000) 2024-03-06T18:11:25.407 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:25.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:25.410033+0000) 2024-03-06T18:11:25.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:25.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:26.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:26.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:26.410267+0000) 2024-03-06T18:11:27.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:27.408+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:27.410530+0000) 2024-03-06T18:11:28.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:28.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:28.410776+0000) 2024-03-06T18:11:29.408 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:29.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:29.411013+0000) 2024-03-06T18:11:30.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:30.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:30.411286+0000) 2024-03-06T18:11:31.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:31.409+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:31.411523+0000) 2024-03-06T18:11:32.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:32.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:32.411802+0000) 2024-03-06T18:11:33.409 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:33.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:33.412056+0000) 2024-03-06T18:11:34.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:34.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:34.412284+0000) 2024-03-06T18:11:35.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:35.410+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:35.412504+0000) 2024-03-06T18:11:36.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:36.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:36.412778+0000) 2024-03-06T18:11:37.410 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:37.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:37.412966+0000) 2024-03-06T18:11:38.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:38.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:38.413221+0000) 2024-03-06T18:11:39.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:39.412+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:39.413497+0000) 2024-03-06T18:11:40.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:40.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:40.413738+0000) 2024-03-06T18:11:41.411 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:41.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:41.413962+0000) 2024-03-06T18:11:42.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:42.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:42.414200+0000) 2024-03-06T18:11:43.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:43.413+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:43.414438+0000) 2024-03-06T18:11:44.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:44.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:44.414706+0000) 2024-03-06T18:11:45.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:45.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:45.414931+0000) 2024-03-06T18:11:46.412 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:46.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:46.415173+0000) 2024-03-06T18:11:47.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:47.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:47.415395+0000) 2024-03-06T18:11:48.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:48.414+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:48.415627+0000) 2024-03-06T18:11:49.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:49.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:49.415823+0000) 2024-03-06T18:11:50.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:50.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:50.416014+0000) 2024-03-06T18:11:51.413 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:51.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:51.416193+0000) 2024-03-06T18:11:52.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:52.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:52.416428+0000) 2024-03-06T18:11:53.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:53.415+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:53.416648+0000) 2024-03-06T18:11:54.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:54.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:54.416875+0000) 2024-03-06T18:11:55.414 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:55.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:55.417117+0000) 2024-03-06T18:11:55.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:55.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:56.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:56.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:56.417317+0000) 2024-03-06T18:11:57.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:57.416+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:57.417538+0000) 2024-03-06T18:11:58.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:58.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:58.417788+0000) 2024-03-06T18:11:59.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:11:59.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:59.417989+0000) 2024-03-06T18:12:00.415 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:00.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:00.418219+0000) 2024-03-06T18:12:01.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:01.417+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:01.418440+0000) 2024-03-06T18:12:02.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:02.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:02.418717+0000) 2024-03-06T18:12:03.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:03.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:03.418921+0000) 2024-03-06T18:12:04.416 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:04.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:04.419158+0000) 2024-03-06T18:12:05.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:05.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:05.419376+0000) 2024-03-06T18:12:06.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:06.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:06.419603+0000) 2024-03-06T18:12:07.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:07.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:07.419815+0000) 2024-03-06T18:12:08.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:08.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:08.420012+0000) 2024-03-06T18:12:09.417 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:09.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:09.420250+0000) 2024-03-06T18:12:10.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:10.418+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:10.420481+0000) 2024-03-06T18:12:11.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:11.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:11.420748+0000) 2024-03-06T18:12:12.418 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:12.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:12.420989+0000) 2024-03-06T18:12:13.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:13.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:13.421194+0000) 2024-03-06T18:12:14.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:14.419+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:14.421443+0000) 2024-03-06T18:12:15.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:15.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:15.421661+0000) 2024-03-06T18:12:16.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:16.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:16.421908+0000) 2024-03-06T18:12:17.419 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:17.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:17.422102+0000) 2024-03-06T18:12:18.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:18.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:18.422339+0000) 2024-03-06T18:12:19.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:19.420+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:19.422579+0000) 2024-03-06T18:12:20.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:20.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:20.422807+0000) 2024-03-06T18:12:21.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:21.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:21.423013+0000) 2024-03-06T18:12:22.420 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:22.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:22.423198+0000) 2024-03-06T18:12:23.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:23.421+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:23.423421+0000) 2024-03-06T18:12:24.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:24.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:24.423704+0000) 2024-03-06T18:12:25.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:25.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:25.423911+0000) 2024-03-06T18:12:26.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:26.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:26.421 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:26.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:26.424145+0000) 2024-03-06T18:12:27.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:27.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:27.424378+0000) 2024-03-06T18:12:28.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:28.423+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:28.424605+0000) 2024-03-06T18:12:29.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:29.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:29.424834+0000) 2024-03-06T18:12:30.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:30.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:30.425065+0000) 2024-03-06T18:12:31.422 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:31.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:31.425285+0000) 2024-03-06T18:12:32.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:32.424+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:32.425474+0000) 2024-03-06T18:12:33.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:33.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:33.425765+0000) 2024-03-06T18:12:34.423 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:34.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:34.426051+0000) 2024-03-06T18:12:35.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:35.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:35.426285+0000) 2024-03-06T18:12:36.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:36.425+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:36.426543+0000) 2024-03-06T18:12:37.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:37.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:37.426777+0000) 2024-03-06T18:12:38.424 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:38.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:38.427044+0000) 2024-03-06T18:12:39.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:39.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:39.427297+0000) 2024-03-06T18:12:40.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:40.426+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:40.427518+0000) 2024-03-06T18:12:41.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:41.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:41.427812+0000) 2024-03-06T18:12:42.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:42.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:42.428084+0000) 2024-03-06T18:12:43.425 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:43.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:43.428316+0000) 2024-03-06T18:12:44.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:44.427+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:44.428572+0000) 2024-03-06T18:12:45.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:45.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:45.428821+0000) 2024-03-06T18:12:46.426 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:46.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:46.429032+0000) 2024-03-06T18:12:47.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:47.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:47.429267+0000) 2024-03-06T18:12:48.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:48.428+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:48.429467+0000) 2024-03-06T18:12:49.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:49.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:49.429726+0000) 2024-03-06T18:12:50.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:50.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:50.429890+0000) 2024-03-06T18:12:51.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:51.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:51.430087+0000) 2024-03-06T18:12:52.427 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:52.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:52.430304+0000) 2024-03-06T18:12:53.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:53.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:53.430534+0000) 2024-03-06T18:12:54.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:54.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:54.430828+0000) 2024-03-06T18:12:55.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:55.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:55.431019+0000) 2024-03-06T18:12:56.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:56.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:56.428 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:56.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:56.431260+0000) 2024-03-06T18:12:57.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:57.429+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:57.431485+0000) 2024-03-06T18:12:58.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:58.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:58.431786+0000) 2024-03-06T18:12:59.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:12:59.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:59.431987+0000) 2024-03-06T18:13:00.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:00.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:00.432181+0000) 2024-03-06T18:13:01.429 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:01.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:01.432415+0000) 2024-03-06T18:13:02.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:02.430+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:02.432641+0000) 2024-03-06T18:13:03.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:03.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:03.432846+0000) 2024-03-06T18:13:04.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:04.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:04.433050+0000) 2024-03-06T18:13:05.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:05.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:05.433230+0000) 2024-03-06T18:13:06.430 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:06.431+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:06.433459+0000) 2024-03-06T18:13:07.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:07.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:07.433657+0000) 2024-03-06T18:13:08.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:08.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:08.433969+0000) 2024-03-06T18:13:09.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:09.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:09.434183+0000) 2024-03-06T18:13:10.431 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:10.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:10.434414+0000) 2024-03-06T18:13:11.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:11.432+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:11.434639+0000) 2024-03-06T18:13:12.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:12.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:12.434876+0000) 2024-03-06T18:13:13.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:13.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:13.435072+0000) 2024-03-06T18:13:14.432 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:14.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:14.435295+0000) 2024-03-06T18:13:15.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:15.434+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:15.435551+0000) 2024-03-06T18:13:16.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:16.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:16.435767+0000) 2024-03-06T18:13:17.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:17.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:17.435998+0000) 2024-03-06T18:13:18.433 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:18.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:18.436222+0000) 2024-03-06T18:13:19.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:19.435+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:19.436490+0000) 2024-03-06T18:13:20.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:20.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:20.436757+0000) 2024-03-06T18:13:21.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:21.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:21.436965+0000) 2024-03-06T18:13:22.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:22.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:22.437218+0000) 2024-03-06T18:13:23.434 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:23.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:23.437396+0000) 2024-03-06T18:13:24.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:24.436+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:24.437621+0000) 2024-03-06T18:13:25.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:25.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:25.437857+0000) 2024-03-06T18:13:26.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:26.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:26.438085+0000) 2024-03-06T18:13:26.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:26.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:27.435 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:27.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:27.438317+0000) 2024-03-06T18:13:28.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:28.437+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:28.438540+0000) 2024-03-06T18:13:29.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:29.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:29.438802+0000) 2024-03-06T18:13:30.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:30.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:30.439073+0000) 2024-03-06T18:13:31.436 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:31.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:31.439356+0000) 2024-03-06T18:13:32.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:32.438+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:32.439649+0000) 2024-03-06T18:13:33.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:33.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:33.439940+0000) 2024-03-06T18:13:34.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:34.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:34.440124+0000) 2024-03-06T18:13:35.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:35.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:35.440358+0000) 2024-03-06T18:13:36.437 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:36.439+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:36.440591+0000) 2024-03-06T18:13:37.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:37.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:37.440807+0000) 2024-03-06T18:13:38.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:38.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:38.440989+0000) 2024-03-06T18:13:39.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:39.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:39.441199+0000) 2024-03-06T18:13:40.438 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:40.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:40.441477+0000) 2024-03-06T18:13:41.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:41.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:41.441800+0000) 2024-03-06T18:13:42.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:42.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:42.442091+0000) 2024-03-06T18:13:43.439 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:43.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:43.442324+0000) 2024-03-06T18:13:44.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:44.440+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:44.442598+0000) 2024-03-06T18:13:45.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:45.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:45.442880+0000) 2024-03-06T18:13:46.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:46.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:46.443163+0000) 2024-03-06T18:13:47.440 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:47.441+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:47.443470+0000) 2024-03-06T18:13:48.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:48.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:48.443705+0000) 2024-03-06T18:13:49.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:49.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:49.443930+0000) 2024-03-06T18:13:50.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:50.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:50.444203+0000) 2024-03-06T18:13:51.441 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:51.442+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:51.444479+0000) 2024-03-06T18:13:52.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:52.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:52.444819+0000) 2024-03-06T18:13:53.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:53.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:53.445095+0000) 2024-03-06T18:13:54.442 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:54.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:54.445370+0000) 2024-03-06T18:13:55.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:55.443+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:55.445636+0000) 2024-03-06T18:13:56.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:56.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:56.445918+0000) 2024-03-06T18:13:56.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:56.689 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:57.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:57.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:57.446186+0000) 2024-03-06T18:13:58.443 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:58.444+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:58.446417+0000) 2024-03-06T18:13:59.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:13:59.445+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:59.446736+0000) 2024-03-06T18:14:00.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:00.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:00.446990+0000) 2024-03-06T18:14:01.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:01.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:01.447220+0000) 2024-03-06T18:14:02.444 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:02.446+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:02.447500+0000) 2024-03-06T18:14:03.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:03.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:03.447802+0000) 2024-03-06T18:14:04.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:04.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:04.448007+0000) 2024-03-06T18:14:05.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:05.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:05.448190+0000) 2024-03-06T18:14:06.445 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:06.447+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:06.448418+0000) 2024-03-06T18:14:07.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:07.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:07.448709+0000) 2024-03-06T18:14:08.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:08.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:08.448984+0000) 2024-03-06T18:14:09.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:09.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:09.449195+0000) 2024-03-06T18:14:10.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:10.448+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:10.449428+0000) 2024-03-06T18:14:11.446 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:11.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:11.449706+0000) 2024-03-06T18:14:12.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:12.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:12.449947+0000) 2024-03-06T18:14:13.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:13.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:13.450163+0000) 2024-03-06T18:14:14.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:14.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:14.450367+0000) 2024-03-06T18:14:15.447 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:15.449+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:15.450532+0000) 2024-03-06T18:14:16.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:16.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:16.450786+0000) 2024-03-06T18:14:17.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:17.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:17.451024+0000) 2024-03-06T18:14:18.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:18.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:18.451264+0000) 2024-03-06T18:14:19.448 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:19.450+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:19.451530+0000) 2024-03-06T18:14:20.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:20.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:20.451821+0000) 2024-03-06T18:14:21.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:21.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:21.452062+0000) 2024-03-06T18:14:22.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:22.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:22.452313+0000) 2024-03-06T18:14:23.449 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:23.451+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:23.452539+0000) 2024-03-06T18:14:24.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:24.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:24.452811+0000) 2024-03-06T18:14:25.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:25.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:25.453037+0000) 2024-03-06T18:14:26.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:26.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:26.453306+0000) 2024-03-06T18:14:26.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:26.911 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:27.450 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:27.452+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:27.453517+0000) 2024-03-06T18:14:28.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:28.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:28.453807+0000) 2024-03-06T18:14:29.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:29.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:29.454044+0000) 2024-03-06T18:14:30.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:30.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:30.454279+0000) 2024-03-06T18:14:31.451 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:31.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:31.454523+0000) 2024-03-06T18:14:32.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:32.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:32.454794+0000) 2024-03-06T18:14:33.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:33.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:33.454986+0000) 2024-03-06T18:14:34.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:34.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:34.455233+0000) 2024-03-06T18:14:35.452 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:35.453+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:35.455455+0000) 2024-03-06T18:14:36.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:36.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:36.455743+0000) 2024-03-06T18:14:37.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:37.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:37.455938+0000) 2024-03-06T18:14:38.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:38.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:38.456167+0000) 2024-03-06T18:14:39.453 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:39.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:39.456393+0000) 2024-03-06T18:14:40.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:40.454+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:40.456632+0000) 2024-03-06T18:14:41.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:41.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:41.456822+0000) 2024-03-06T18:14:42.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:42.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:42.457072+0000) 2024-03-06T18:14:43.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:43.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:43.457303+0000) 2024-03-06T18:14:44.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:44.455+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:44.457526+0000) 2024-03-06T18:14:45.454 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:45.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:45.457804+0000) 2024-03-06T18:14:46.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:46.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:46.458013+0000) 2024-03-06T18:14:47.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:47.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:47.458238+0000) 2024-03-06T18:14:48.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:48.456+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:48.458468+0000) 2024-03-06T18:14:49.455 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:49.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:49.458739+0000) 2024-03-06T18:14:50.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:50.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:50.458992+0000) 2024-03-06T18:14:51.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:51.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:51.459247+0000) 2024-03-06T18:14:52.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:52.458+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:52.459474+0000) 2024-03-06T18:14:53.456 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:53.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:53.459723+0000) 2024-03-06T18:14:54.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:54.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:54.459952+0000) 2024-03-06T18:14:55.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:55.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:55.460175+0000) 2024-03-06T18:14:56.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:56.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:56.460420+0000) 2024-03-06T18:14:57.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:57.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:57.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:57.459+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:57.460624+0000) 2024-03-06T18:14:58.457 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:58.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:58.460885+0000) 2024-03-06T18:14:59.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:14:59.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:59.461072+0000) 2024-03-06T18:15:00.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:00.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:00.461310+0000) 2024-03-06T18:15:01.458 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:01.460+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:01.461570+0000) 2024-03-06T18:15:02.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:02.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:02.461829+0000) 2024-03-06T18:15:03.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:03.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:03.462016+0000) 2024-03-06T18:15:04.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:04.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:04.462252+0000) 2024-03-06T18:15:05.459 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:05.461+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:05.462518+0000) 2024-03-06T18:15:06.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:06.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:06.462846+0000) 2024-03-06T18:15:07.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:07.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:07.463090+0000) 2024-03-06T18:15:08.460 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:08.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:08.463333+0000) 2024-03-06T18:15:09.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:09.462+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:09.463567+0000) 2024-03-06T18:15:10.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:10.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:10.463838+0000) 2024-03-06T18:15:11.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:11.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:11.464047+0000) 2024-03-06T18:15:12.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:12.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:12.464271+0000) 2024-03-06T18:15:13.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:13.463+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:13.464467+0000) 2024-03-06T18:15:14.461 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:14.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:14.464756+0000) 2024-03-06T18:15:15.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:15.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:15.464965+0000) 2024-03-06T18:15:16.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:16.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:16.465174+0000) 2024-03-06T18:15:17.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:17.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:17.465413+0000) 2024-03-06T18:15:18.462 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:18.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:18.465601+0000) 2024-03-06T18:15:19.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:19.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:19.465817+0000) 2024-03-06T18:15:20.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:20.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:20.466017+0000) 2024-03-06T18:15:21.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:21.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:21.466166+0000) 2024-03-06T18:15:22.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:22.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:22.466354+0000) 2024-03-06T18:15:23.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:23.464+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:23.466551+0000) 2024-03-06T18:15:24.463 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:24.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:24.466806+0000) 2024-03-06T18:15:25.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:25.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:25.467002+0000) 2024-03-06T18:15:26.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:26.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:26.467192+0000) 2024-03-06T18:15:27.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:27.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:27.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:27.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:27.467387+0000) 2024-03-06T18:15:28.464 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:28.465+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:28.467622+0000) 2024-03-06T18:15:29.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:29.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:29.467818+0000) 2024-03-06T18:15:30.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:30.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:30.468044+0000) 2024-03-06T18:15:31.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:31.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:31.468287+0000) 2024-03-06T18:15:32.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:32.466+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:32.468468+0000) 2024-03-06T18:15:33.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:33.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:33.468737+0000) 2024-03-06T18:15:34.465 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:34.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:34.468929+0000) 2024-03-06T18:15:35.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:35.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:35.469164+0000) 2024-03-06T18:15:36.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:36.467+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:36.469342+0000) 2024-03-06T18:15:37.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:37.468+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:37.469571+0000) 2024-03-06T18:15:38.466 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:38.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:38.469807+0000) 2024-03-06T18:15:39.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:39.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:39.470006+0000) 2024-03-06T18:15:40.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:40.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:40.470225+0000) 2024-03-06T18:15:41.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:41.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:41.470403+0000) 2024-03-06T18:15:42.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:42.469+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:42.470602+0000) 2024-03-06T18:15:43.467 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:43.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:43.470799+0000) 2024-03-06T18:15:44.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:44.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:44.471087+0000) 2024-03-06T18:15:45.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:45.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:45.471305+0000) 2024-03-06T18:15:46.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:46.470+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:46.471534+0000) 2024-03-06T18:15:47.468 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:47.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:47.471781+0000) 2024-03-06T18:15:48.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:48.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:48.472009+0000) 2024-03-06T18:15:49.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:49.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:49.472219+0000) 2024-03-06T18:15:50.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:50.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:50.472410+0000) 2024-03-06T18:15:51.469 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:51.471+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:51.472594+0000) 2024-03-06T18:15:52.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:52.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:52.472852+0000) 2024-03-06T18:15:53.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:53.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:53.473051+0000) 2024-03-06T18:15:54.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:54.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:54.473251+0000) 2024-03-06T18:15:55.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:55.472+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:55.473530+0000) 2024-03-06T18:15:56.470 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:56.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:56.473806+0000) 2024-03-06T18:15:57.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:57.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:57.474067+0000) 2024-03-06T18:15:57.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:57.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:58.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:58.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:58.474319+0000) 2024-03-06T18:15:59.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:15:59.473+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:59.474559+0000) 2024-03-06T18:16:00.471 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:00.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:00.474806+0000) 2024-03-06T18:16:01.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:01.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:01.475032+0000) 2024-03-06T18:16:02.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:02.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:02.475280+0000) 2024-03-06T18:16:03.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:03.474+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:03.475508+0000) 2024-03-06T18:16:04.472 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:04.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:04.475798+0000) 2024-03-06T18:16:05.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:05.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:05.476043+0000) 2024-03-06T18:16:06.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:06.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:06.476261+0000) 2024-03-06T18:16:07.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:07.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:07.476414+0000) 2024-03-06T18:16:08.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:08.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:08.476643+0000) 2024-03-06T18:16:09.473 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:09.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:09.476890+0000) 2024-03-06T18:16:10.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:10.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:10.477090+0000) 2024-03-06T18:16:11.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:11.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:11.477317+0000) 2024-03-06T18:16:12.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:12.475+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:12.477507+0000) 2024-03-06T18:16:13.474 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:13.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:13.477770+0000) 2024-03-06T18:16:14.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:14.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:14.478005+0000) 2024-03-06T18:16:15.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:15.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:15.478244+0000) 2024-03-06T18:16:16.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:16.476+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:16.478480+0000) 2024-03-06T18:16:17.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:17.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:17.478740+0000) 2024-03-06T18:16:18.475 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:18.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:18.478979+0000) 2024-03-06T18:16:19.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:19.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:19.479212+0000) 2024-03-06T18:16:20.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:20.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:20.479410+0000) 2024-03-06T18:16:21.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:21.477+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:21.479602+0000) 2024-03-06T18:16:22.476 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:22.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:22.479850+0000) 2024-03-06T18:16:23.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:23.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:23.480069+0000) 2024-03-06T18:16:24.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:24.478+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:24.480288+0000) 2024-03-06T18:16:25.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:25.479+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:25.480517+0000) 2024-03-06T18:16:26.477 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:26.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:26.480807+0000) 2024-03-06T18:16:27.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:27.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:27.481000+0000) 2024-03-06T18:16:27.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:27.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:28.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:28.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:28.481230+0000) 2024-03-06T18:16:29.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:29.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:29.481449+0000) 2024-03-06T18:16:30.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:30.480+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:30.481639+0000) 2024-03-06T18:16:31.478 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:31.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:31.481857+0000) 2024-03-06T18:16:32.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:32.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:32.482054+0000) 2024-03-06T18:16:33.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:33.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:33.482244+0000) 2024-03-06T18:16:34.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:34.481+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:34.482489+0000) 2024-03-06T18:16:35.479 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:35.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:35.482709+0000) 2024-03-06T18:16:36.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:36.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:36.482953+0000) 2024-03-06T18:16:37.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:37.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:37.483243+0000) 2024-03-06T18:16:38.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:38.482+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:38.483484+0000) 2024-03-06T18:16:39.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:39.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:39.483743+0000) 2024-03-06T18:16:40.480 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:40.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:40.483973+0000) 2024-03-06T18:16:41.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:41.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:41.484204+0000) 2024-03-06T18:16:42.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:42.483+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:42.484452+0000) 2024-03-06T18:16:43.481 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:43.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:43.484728+0000) 2024-03-06T18:16:44.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:44.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:44.484941+0000) 2024-03-06T18:16:45.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:45.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:45.485140+0000) 2024-03-06T18:16:46.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:46.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:46.485380+0000) 2024-03-06T18:16:47.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:47.484+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:47.485552+0000) 2024-03-06T18:16:48.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:48.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:48.485785+0000) 2024-03-06T18:16:49.482 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:49.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:49.485991+0000) 2024-03-06T18:16:50.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:50.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:50.486223+0000) 2024-03-06T18:16:51.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:51.485+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:51.486449+0000) 2024-03-06T18:16:52.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:52.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:52.486714+0000) 2024-03-06T18:16:53.483 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:53.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:53.486898+0000) 2024-03-06T18:16:54.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:54.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:54.487120+0000) 2024-03-06T18:16:55.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:55.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:55.487355+0000) 2024-03-06T18:16:56.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:56.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:56.487593+0000) 2024-03-06T18:16:57.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:57.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:57.487812+0000) 2024-03-06T18:16:57.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:57.989 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:58.484 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:58.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:58.488011+0000) 2024-03-06T18:16:59.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:16:59.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:59.488233+0000) 2024-03-06T18:17:00.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:00.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:00.488419+0000) 2024-03-06T18:17:01.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:01.486+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:01.488636+0000) 2024-03-06T18:17:02.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:02.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:02.488882+0000) 2024-03-06T18:17:03.485 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:03.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:03.489082+0000) 2024-03-06T18:17:04.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:04.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:04.489310+0000) 2024-03-06T18:17:05.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:05.487+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:05.489539+0000) 2024-03-06T18:17:06.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:06.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:06.489792+0000) 2024-03-06T18:17:07.486 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:07.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:07.490001+0000) 2024-03-06T18:17:08.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:08.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:08.490230+0000) 2024-03-06T18:17:09.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:09.488+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:09.490453+0000) 2024-03-06T18:17:10.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:10.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:10.490649+0000) 2024-03-06T18:17:11.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:11.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:11.490936+0000) 2024-03-06T18:17:12.487 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:12.489+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:12.491185+0000) 2024-03-06T18:17:13.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:13.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:13.491420+0000) 2024-03-06T18:17:14.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:14.490+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:14.491639+0000) 2024-03-06T18:17:15.488 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:15.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:15.491917+0000) 2024-03-06T18:17:16.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:16.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:16.492145+0000) 2024-03-06T18:17:17.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:17.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:17.492334+0000) 2024-03-06T18:17:18.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:18.491+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:18.492565+0000) 2024-03-06T18:17:19.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:19.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:19.492781+0000) 2024-03-06T18:17:20.489 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:20.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:20.493018+0000) 2024-03-06T18:17:21.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:21.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:21.493242+0000) 2024-03-06T18:17:22.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:22.492+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:22.493475+0000) 2024-03-06T18:17:23.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:23.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:23.493700+0000) 2024-03-06T18:17:24.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:24.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:24.493942+0000) 2024-03-06T18:17:25.490 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:25.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:25.494229+0000) 2024-03-06T18:17:26.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:26.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:26.494448+0000) 2024-03-06T18:17:27.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:27.493+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:27.494649+0000) 2024-03-06T18:17:28.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:28.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:28.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:28.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:28.494847+0000) 2024-03-06T18:17:29.491 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:29.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:29.495080+0000) 2024-03-06T18:17:30.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:30.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:30.495313+0000) 2024-03-06T18:17:31.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:31.494+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:31.495540+0000) 2024-03-06T18:17:32.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:32.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:32.495783+0000) 2024-03-06T18:17:33.492 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:33.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:33.495976+0000) 2024-03-06T18:17:34.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:34.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:34.496198+0000) 2024-03-06T18:17:35.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:35.495+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:35.496432+0000) 2024-03-06T18:17:36.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:36.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:36.496657+0000) 2024-03-06T18:17:37.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:37.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:37.496921+0000) 2024-03-06T18:17:38.493 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:38.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:38.497134+0000) 2024-03-06T18:17:39.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:39.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:39.497330+0000) 2024-03-06T18:17:40.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:40.496+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:40.497571+0000) 2024-03-06T18:17:41.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:41.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:41.497830+0000) 2024-03-06T18:17:42.494 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:42.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:42.498078+0000) 2024-03-06T18:17:43.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:43.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:43.498353+0000) 2024-03-06T18:17:44.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:44.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:44.498566+0000) 2024-03-06T18:17:45.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:45.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:45.498838+0000) 2024-03-06T18:17:46.495 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:46.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:46.499118+0000) 2024-03-06T18:17:47.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:47.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:47.499321+0000) 2024-03-06T18:17:48.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:48.497+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:48.499515+0000) 2024-03-06T18:17:49.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:49.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:49.499798+0000) 2024-03-06T18:17:50.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:50.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:50.499998+0000) 2024-03-06T18:17:51.496 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:51.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:51.500224+0000) 2024-03-06T18:17:52.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:52.498+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:52.500453+0000) 2024-03-06T18:17:53.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:53.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:53.500765+0000) 2024-03-06T18:17:54.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:54.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:54.500968+0000) 2024-03-06T18:17:55.497 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:55.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:55.501167+0000) 2024-03-06T18:17:56.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:56.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:56.501398+0000) 2024-03-06T18:17:57.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:57.499+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:57.501624+0000) 2024-03-06T18:17:58.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:58.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:58.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:58.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:58.501829+0000) 2024-03-06T18:17:59.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:17:59.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:59.502049+0000) 2024-03-06T18:18:00.498 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:00.500+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:00.502294+0000) 2024-03-06T18:18:01.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:01.501+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:01.502525+0000) 2024-03-06T18:18:02.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:02.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:02.502801+0000) 2024-03-06T18:18:03.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:03.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:03.503026+0000) 2024-03-06T18:18:04.499 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:04.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:04.503263+0000) 2024-03-06T18:18:05.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:05.502+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:05.503505+0000) 2024-03-06T18:18:06.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:06.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:06.503765+0000) 2024-03-06T18:18:07.500 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:07.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:07.503989+0000) 2024-03-06T18:18:08.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:08.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:08.504214+0000) 2024-03-06T18:18:09.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:09.503+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:09.504433+0000) 2024-03-06T18:18:10.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:10.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:10.504660+0000) 2024-03-06T18:18:11.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:11.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:11.504925+0000) 2024-03-06T18:18:12.501 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:12.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:12.505158+0000) 2024-03-06T18:18:13.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:13.504+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:13.505427+0000) 2024-03-06T18:18:14.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:14.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:14.505740+0000) 2024-03-06T18:18:15.502 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:15.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:15.506031+0000) 2024-03-06T18:18:16.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:16.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:16.506310+0000) 2024-03-06T18:18:17.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:17.505+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:17.506587+0000) 2024-03-06T18:18:18.503 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:18.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:18.506790+0000) 2024-03-06T18:18:19.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:19.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:19.507070+0000) 2024-03-06T18:18:20.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:20.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:20.507361+0000) 2024-03-06T18:18:21.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:21.506+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:21.507565+0000) 2024-03-06T18:18:22.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:22.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:22.507813+0000) 2024-03-06T18:18:23.504 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:23.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:23.508014+0000) 2024-03-06T18:18:24.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:24.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:24.508251+0000) 2024-03-06T18:18:25.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:25.507+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:25.508527+0000) 2024-03-06T18:18:26.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:26.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:26.508804+0000) 2024-03-06T18:18:27.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:27.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:27.509036+0000) 2024-03-06T18:18:28.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:28.489 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:28.505 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:28.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:28.509234+0000) 2024-03-06T18:18:29.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:29.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:29.509493+0000) 2024-03-06T18:18:30.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:30.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:30.509817+0000) 2024-03-06T18:18:31.506 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:31.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:31.510061+0000) 2024-03-06T18:18:32.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:32.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:32.510339+0000) 2024-03-06T18:18:33.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:33.508+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:33.510621+0000) 2024-03-06T18:18:34.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:34.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:34.510870+0000) 2024-03-06T18:18:35.507 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:35.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:35.511131+0000) 2024-03-06T18:18:36.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:36.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:36.511377+0000) 2024-03-06T18:18:37.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:37.509+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:37.511629+0000) 2024-03-06T18:18:38.508 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:38.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:38.511899+0000) 2024-03-06T18:18:39.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:39.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:39.512164+0000) 2024-03-06T18:18:40.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:40.510+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:40.512450+0000) 2024-03-06T18:18:41.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:41.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:41.512750+0000) 2024-03-06T18:18:42.509 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:42.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:42.513020+0000) 2024-03-06T18:18:43.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:43.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:43.513297+0000) 2024-03-06T18:18:44.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:44.511+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:44.513574+0000) 2024-03-06T18:18:45.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:45.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:45.513807+0000) 2024-03-06T18:18:46.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:46.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:46.514016+0000) 2024-03-06T18:18:47.510 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:47.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:47.514195+0000) 2024-03-06T18:18:48.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:48.512+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:48.514394+0000) 2024-03-06T18:18:49.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:49.513+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:49.514583+0000) 2024-03-06T18:18:50.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:50.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:50.514880+0000) 2024-03-06T18:18:51.511 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:51.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:51.515109+0000) 2024-03-06T18:18:52.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:52.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:52.515349+0000) 2024-03-06T18:18:53.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:53.514+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:53.515571+0000) 2024-03-06T18:18:54.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:54.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:54.515799+0000) 2024-03-06T18:18:55.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:55.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:55.516035+0000) 2024-03-06T18:18:56.512 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:56.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:56.516263+0000) 2024-03-06T18:18:57.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:57.515+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:57.516482+0000) 2024-03-06T18:18:58.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:58.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:58.516697+0000) 2024-03-06T18:18:58.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:58.623 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:59.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:18:59.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:59.516958+0000) 2024-03-06T18:19:00.513 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:00.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:00.517201+0000) 2024-03-06T18:19:01.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:01.516+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:01.517430+0000) 2024-03-06T18:19:02.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:02.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:02.517661+0000) 2024-03-06T18:19:03.514 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:03.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:03.517955+0000) 2024-03-06T18:19:04.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:04.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:04.518167+0000) 2024-03-06T18:19:05.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:05.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:05.518419+0000) 2024-03-06T18:19:06.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:06.517+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:06.518619+0000) 2024-03-06T18:19:07.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:07.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:07.518914+0000) 2024-03-06T18:19:08.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:08.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:08.519115+0000) 2024-03-06T18:19:09.515 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:09.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:09.519341+0000) 2024-03-06T18:19:10.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:10.518+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:10.519565+0000) 2024-03-06T18:19:11.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:11.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:11.519776+0000) 2024-03-06T18:19:12.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:12.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:12.519972+0000) 2024-03-06T18:19:13.516 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:13.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:13.520209+0000) 2024-03-06T18:19:14.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:14.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:14.520464+0000) 2024-03-06T18:19:15.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:15.519+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:15.520634+0000) 2024-03-06T18:19:16.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:16.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:16.520854+0000) 2024-03-06T18:19:17.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:17.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:17.521042+0000) 2024-03-06T18:19:18.517 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:18.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:18.521239+0000) 2024-03-06T18:19:19.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:19.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:19.521453+0000) 2024-03-06T18:19:20.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:20.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:20.521701+0000) 2024-03-06T18:19:21.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:21.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:21.521952+0000) 2024-03-06T18:19:22.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:22.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:22.522181+0000) 2024-03-06T18:19:23.518 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:23.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:23.522370+0000) 2024-03-06T18:19:24.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:24.520+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:24.522601+0000) 2024-03-06T18:19:25.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:25.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:25.522833+0000) 2024-03-06T18:19:26.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:26.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:26.523026+0000) 2024-03-06T18:19:27.519 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:27.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:27.523215+0000) 2024-03-06T18:19:28.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:28.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:28.523405+0000) 2024-03-06T18:19:28.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:28.765 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:29.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:29.521+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:29.523640+0000) 2024-03-06T18:19:30.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:30.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:30.523903+0000) 2024-03-06T18:19:31.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:31.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:31.524103+0000) 2024-03-06T18:19:32.520 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:32.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:32.524335+0000) 2024-03-06T18:19:33.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:33.522+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:33.524604+0000) 2024-03-06T18:19:34.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:34.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:34.524878+0000) 2024-03-06T18:19:35.521 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:35.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:35.525132+0000) 2024-03-06T18:19:36.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:36.523+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:36.525403+0000) 2024-03-06T18:19:37.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:37.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:37.525717+0000) 2024-03-06T18:19:38.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:38.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:38.525935+0000) 2024-03-06T18:19:39.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:39.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:39.526161+0000) 2024-03-06T18:19:40.522 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:40.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:40.526350+0000) 2024-03-06T18:19:41.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:41.525+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:41.526590+0000) 2024-03-06T18:19:42.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:42.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:42.526838+0000) 2024-03-06T18:19:43.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:43.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:43.527049+0000) 2024-03-06T18:19:44.523 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:44.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:44.527282+0000) 2024-03-06T18:19:45.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:45.526+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:45.527523+0000) 2024-03-06T18:19:46.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:46.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:46.527797+0000) 2024-03-06T18:19:47.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:47.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:47.528019+0000) 2024-03-06T18:19:48.524 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:48.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:48.528250+0000) 2024-03-06T18:19:49.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:49.527+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:49.528497+0000) 2024-03-06T18:19:50.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:50.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:50.528771+0000) 2024-03-06T18:19:51.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:51.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:51.529015+0000) 2024-03-06T18:19:52.525 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:52.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:52.529271+0000) 2024-03-06T18:19:53.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:53.528+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:53.529492+0000) 2024-03-06T18:19:54.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:54.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:54.529771+0000) 2024-03-06T18:19:55.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:55.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:55.530083+0000) 2024-03-06T18:19:56.526 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:56.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:56.530315+0000) 2024-03-06T18:19:57.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:57.529+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:57.530548+0000) 2024-03-06T18:19:58.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:58.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:58.530783+0000) 2024-03-06T18:19:58.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:58.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:59.527 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:19:59.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:59.531036+0000) 2024-03-06T18:20:00.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:00.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:00.531287+0000) 2024-03-06T18:20:01.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:01.530+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:01.531514+0000) 2024-03-06T18:20:02.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:02.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:02.531771+0000) 2024-03-06T18:20:03.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:03.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:03.531971+0000) 2024-03-06T18:20:04.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:04.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:04.532157+0000) 2024-03-06T18:20:05.528 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:05.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:05.532391+0000) 2024-03-06T18:20:06.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:06.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:06.532624+0000) 2024-03-06T18:20:07.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:07.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:07.532840+0000) 2024-03-06T18:20:08.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:08.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:08.533086+0000) 2024-03-06T18:20:09.529 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:09.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:09.533288+0000) 2024-03-06T18:20:10.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:10.531+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:10.533533+0000) 2024-03-06T18:20:11.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:11.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:11.533797+0000) 2024-03-06T18:20:12.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:12.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:12.534026+0000) 2024-03-06T18:20:13.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:13.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:13.534252+0000) 2024-03-06T18:20:14.530 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:14.532+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:14.534474+0000) 2024-03-06T18:20:15.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:15.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:15.534761+0000) 2024-03-06T18:20:16.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:16.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:16.534952+0000) 2024-03-06T18:20:17.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:17.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:17.535132+0000) 2024-03-06T18:20:18.531 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:18.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:18.535350+0000) 2024-03-06T18:20:19.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:19.533+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:19.535580+0000) 2024-03-06T18:20:20.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:20.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:20.535818+0000) 2024-03-06T18:20:21.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:21.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:21.536005+0000) 2024-03-06T18:20:22.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:22.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:22.536213+0000) 2024-03-06T18:20:23.532 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:23.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:23.536388+0000) 2024-03-06T18:20:24.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:24.534+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:24.536626+0000) 2024-03-06T18:20:25.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:25.535+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:25.536916+0000) 2024-03-06T18:20:26.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:26.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:26.537158+0000) 2024-03-06T18:20:27.533 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:27.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:27.537402+0000) 2024-03-06T18:20:28.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:28.536+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:28.537645+0000) 2024-03-06T18:20:29.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:29.187 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:29.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:29.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:29.537864+0000) 2024-03-06T18:20:30.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:30.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:30.538060+0000) 2024-03-06T18:20:31.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:31.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:31.538302+0000) 2024-03-06T18:20:32.534 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:32.537+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:32.538551+0000) 2024-03-06T18:20:33.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:33.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:33.538798+0000) 2024-03-06T18:20:34.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:34.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:34.539045+0000) 2024-03-06T18:20:35.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:35.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:35.539283+0000) 2024-03-06T18:20:36.535 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:36.538+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:36.539472+0000) 2024-03-06T18:20:37.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:37.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:37.539751+0000) 2024-03-06T18:20:38.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:38.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:38.539973+0000) 2024-03-06T18:20:39.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:39.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:39.540200+0000) 2024-03-06T18:20:40.536 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:40.539+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:40.540436+0000) 2024-03-06T18:20:41.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:41.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:41.540701+0000) 2024-03-06T18:20:42.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:42.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:42.540941+0000) 2024-03-06T18:20:43.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:43.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:43.541161+0000) 2024-03-06T18:20:44.537 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:44.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:44.541386+0000) 2024-03-06T18:20:45.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:45.540+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:45.541566+0000) 2024-03-06T18:20:46.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:46.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:46.541811+0000) 2024-03-06T18:20:47.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:47.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:47.541998+0000) 2024-03-06T18:20:48.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:48.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:48.542294+0000) 2024-03-06T18:20:49.538 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:49.541+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:49.542506+0000) 2024-03-06T18:20:50.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:50.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:50.542774+0000) 2024-03-06T18:20:51.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:51.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:51.542971+0000) 2024-03-06T18:20:52.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:52.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:52.543178+0000) 2024-03-06T18:20:53.539 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:53.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:53.543383+0000) 2024-03-06T18:20:54.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:54.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:54.543578+0000) 2024-03-06T18:20:55.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:55.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:55.543816+0000) 2024-03-06T18:20:56.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:56.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:56.544061+0000) 2024-03-06T18:20:57.540 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:57.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:57.544262+0000) 2024-03-06T18:20:58.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:58.542+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:58.544496+0000) 2024-03-06T18:20:59.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:59.390 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:59.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:20:59.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:59.544783+0000) 2024-03-06T18:21:00.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:00.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:00.545031+0000) 2024-03-06T18:21:01.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:01.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:01.545274+0000) 2024-03-06T18:21:02.541 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:02.543+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:02.545519+0000) 2024-03-06T18:21:03.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:03.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:03.545810+0000) 2024-03-06T18:21:04.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:04.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:04.546050+0000) 2024-03-06T18:21:05.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:05.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:05.546271+0000) 2024-03-06T18:21:06.542 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:06.544+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:06.546503+0000) 2024-03-06T18:21:07.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:07.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:07.546787+0000) 2024-03-06T18:21:08.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:08.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:08.546993+0000) 2024-03-06T18:21:09.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:09.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:09.547240+0000) 2024-03-06T18:21:10.543 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:10.545+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:10.547474+0000) 2024-03-06T18:21:11.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:11.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:11.547698+0000) 2024-03-06T18:21:12.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:12.546+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:12.547887+0000) 2024-03-06T18:21:13.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:13.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:13.548088+0000) 2024-03-06T18:21:14.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:14.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:14.548281+0000) 2024-03-06T18:21:15.544 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:15.547+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:15.548517+0000) 2024-03-06T18:21:16.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:16.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:16.548800+0000) 2024-03-06T18:21:17.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:17.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:17.548995+0000) 2024-03-06T18:21:18.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:18.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:18.549202+0000) 2024-03-06T18:21:19.545 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:19.548+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:19.549439+0000) 2024-03-06T18:21:20.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:20.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:20.549705+0000) 2024-03-06T18:21:21.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:21.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:21.549907+0000) 2024-03-06T18:21:22.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:22.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:22.550156+0000) 2024-03-06T18:21:23.546 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:23.549+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:23.550413+0000) 2024-03-06T18:21:24.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:24.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:24.550654+0000) 2024-03-06T18:21:25.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:25.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:25.550915+0000) 2024-03-06T18:21:26.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:26.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:26.551144+0000) 2024-03-06T18:21:27.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:27.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:27.551370+0000) 2024-03-06T18:21:28.547 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:28.550+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:28.551592+0000) 2024-03-06T18:21:29.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:29.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:29.551830+0000) 2024-03-06T18:21:29.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:29.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:30.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:30.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:30.552088+0000) 2024-03-06T18:21:31.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:31.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:31.552333+0000) 2024-03-06T18:21:32.548 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:32.551+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:32.552582+0000) 2024-03-06T18:21:33.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:33.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:33.552829+0000) 2024-03-06T18:21:34.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:34.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:34.553000+0000) 2024-03-06T18:21:35.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:35.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:35.553190+0000) 2024-03-06T18:21:36.549 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:36.552+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:36.553487+0000) 2024-03-06T18:21:37.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:37.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:37.553723+0000) 2024-03-06T18:21:38.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:38.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:38.553990+0000) 2024-03-06T18:21:39.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:39.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:39.554218+0000) 2024-03-06T18:21:40.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:40.553+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:40.554444+0000) 2024-03-06T18:21:41.550 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:41.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:41.554733+0000) 2024-03-06T18:21:42.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:42.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:42.554965+0000) 2024-03-06T18:21:43.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:43.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:43.555147+0000) 2024-03-06T18:21:44.551 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:44.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:44.555372+0000) 2024-03-06T18:21:45.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:45.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:45.555601+0000) 2024-03-06T18:21:46.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:46.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:46.555851+0000) 2024-03-06T18:21:47.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:47.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:47.556041+0000) 2024-03-06T18:21:48.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:48.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:48.556271+0000) 2024-03-06T18:21:49.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:49.554+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:49.556484+0000) 2024-03-06T18:21:50.552 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:50.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:50.556748+0000) 2024-03-06T18:21:51.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:51.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:51.556982+0000) 2024-03-06T18:21:52.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:52.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:52.557166+0000) 2024-03-06T18:21:53.553 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:53.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:53.557393+0000) 2024-03-06T18:21:54.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:54.555+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:54.557592+0000) 2024-03-06T18:21:55.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:55.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:55.557843+0000) 2024-03-06T18:21:56.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:56.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:56.558125+0000) 2024-03-06T18:21:57.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:57.556+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:57.558404+0000) 2024-03-06T18:21:58.554 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:58.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:58.558720+0000) 2024-03-06T18:21:59.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:21:59.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:59.558942+0000) 2024-03-06T18:21:59.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:59.832 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:00.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:00.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:00.559172+0000) 2024-03-06T18:22:01.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:01.557+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:01.559443+0000) 2024-03-06T18:22:02.555 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:02.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:02.559753+0000) 2024-03-06T18:22:03.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:03.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:03.559943+0000) 2024-03-06T18:22:04.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:04.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:04.560167+0000) 2024-03-06T18:22:05.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:05.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:05.560344+0000) 2024-03-06T18:22:06.556 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:06.559+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:06.560571+0000) 2024-03-06T18:22:07.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:07.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:07.560790+0000) 2024-03-06T18:22:08.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:08.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:08.561021+0000) 2024-03-06T18:22:09.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:09.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:09.561242+0000) 2024-03-06T18:22:10.557 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:10.560+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:10.561472+0000) 2024-03-06T18:22:11.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:11.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:11.561715+0000) 2024-03-06T18:22:12.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:12.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:12.561953+0000) 2024-03-06T18:22:13.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:13.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:13.562193+0000) 2024-03-06T18:22:14.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:14.561+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:14.562437+0000) 2024-03-06T18:22:15.558 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:15.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:15.562762+0000) 2024-03-06T18:22:16.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:16.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:16.562949+0000) 2024-03-06T18:22:17.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:17.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:17.563167+0000) 2024-03-06T18:22:18.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:18.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:18.563399+0000) 2024-03-06T18:22:19.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:19.562+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:19.563625+0000) 2024-03-06T18:22:20.559 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:20.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:20.563830+0000) 2024-03-06T18:22:21.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:21.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:21.564033+0000) 2024-03-06T18:22:22.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:22.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:22.564250+0000) 2024-03-06T18:22:23.560 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:23.563+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:23.564478+0000) 2024-03-06T18:22:24.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:24.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:24.564808+0000) 2024-03-06T18:22:25.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:25.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:25.565033+0000) 2024-03-06T18:22:26.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:26.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:26.565306+0000) 2024-03-06T18:22:27.561 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:27.564+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:27.565588+0000) 2024-03-06T18:22:28.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:28.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:28.565827+0000) 2024-03-06T18:22:29.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:29.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:29.566094+0000) 2024-03-06T18:22:29.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:29.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:30.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:30.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:30.566378+0000) 2024-03-06T18:22:31.562 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:31.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:31.566726+0000) 2024-03-06T18:22:32.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:32.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:32.566941+0000) 2024-03-06T18:22:33.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:33.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:33.567210+0000) 2024-03-06T18:22:34.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:34.565+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:34.567451+0000) 2024-03-06T18:22:35.563 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:35.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:35.567736+0000) 2024-03-06T18:22:36.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:36.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:36.567947+0000) 2024-03-06T18:22:37.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:37.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:37.568225+0000) 2024-03-06T18:22:38.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:38.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:38.568428+0000) 2024-03-06T18:22:39.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:39.566+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:39.568647+0000) 2024-03-06T18:22:40.564 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:40.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:40.568871+0000) 2024-03-06T18:22:41.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:41.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:41.569100+0000) 2024-03-06T18:22:42.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:42.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:42.569332+0000) 2024-03-06T18:22:43.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:43.567+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:43.569560+0000) 2024-03-06T18:22:44.565 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:44.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:44.569787+0000) 2024-03-06T18:22:45.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:45.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:45.569986+0000) 2024-03-06T18:22:46.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:46.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:46.570228+0000) 2024-03-06T18:22:47.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:47.568+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:47.570457+0000) 2024-03-06T18:22:48.566 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:48.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:48.570748+0000) 2024-03-06T18:22:49.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:49.569+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:49.570948+0000) 2024-03-06T18:22:50.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:50.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:50.571155+0000) 2024-03-06T18:22:51.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:51.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:51.571404+0000) 2024-03-06T18:22:52.567 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:52.570+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:52.571621+0000) 2024-03-06T18:22:53.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:53.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:53.571847+0000) 2024-03-06T18:22:54.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:54.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:54.572076+0000) 2024-03-06T18:22:55.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:55.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:55.572299+0000) 2024-03-06T18:22:56.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:56.571+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:56.572533+0000) 2024-03-06T18:22:57.568 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:57.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:57.572792+0000) 2024-03-06T18:22:58.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:58.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:58.573062+0000) 2024-03-06T18:22:59.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:22:59.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:59.573282+0000) 2024-03-06T18:23:00.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:00.200 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:00.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:00.572+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:00.573518+0000) 2024-03-06T18:23:01.569 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:01.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:01.573801+0000) 2024-03-06T18:23:02.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:02.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:02.574031+0000) 2024-03-06T18:23:03.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:03.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:03.574301+0000) 2024-03-06T18:23:04.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:04.573+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:04.574587+0000) 2024-03-06T18:23:05.570 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:05.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:05.574800+0000) 2024-03-06T18:23:06.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:06.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:06.575000+0000) 2024-03-06T18:23:07.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:07.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:07.575236+0000) 2024-03-06T18:23:08.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:08.574+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:08.575469+0000) 2024-03-06T18:23:09.571 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:09.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:09.575743+0000) 2024-03-06T18:23:10.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:10.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:10.575988+0000) 2024-03-06T18:23:11.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:11.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:11.576228+0000) 2024-03-06T18:23:12.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:12.575+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:12.576510+0000) 2024-03-06T18:23:13.572 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:13.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:13.576825+0000) 2024-03-06T18:23:14.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:14.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:14.577061+0000) 2024-03-06T18:23:15.573 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:15.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:15.577326+0000) 2024-03-06T18:23:16.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:16.576+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:16.577618+0000) 2024-03-06T18:23:17.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:17.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:17.577943+0000) 2024-03-06T18:23:18.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:18.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:18.578167+0000) 2024-03-06T18:23:19.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:19.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:19.578434+0000) 2024-03-06T18:23:20.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:20.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:20.578648+0000) 2024-03-06T18:23:21.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:21.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:21.578830+0000) 2024-03-06T18:23:22.574 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:22.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:22.579030+0000) 2024-03-06T18:23:23.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:23.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:23.579216+0000) 2024-03-06T18:23:24.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:24.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:24.579398+0000) 2024-03-06T18:23:25.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:25.577+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:25.579617+0000) 2024-03-06T18:23:26.575 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:26.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:26.579849+0000) 2024-03-06T18:23:27.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:27.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:27.580062+0000) 2024-03-06T18:23:28.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:28.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:28.580292+0000) 2024-03-06T18:23:29.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:29.578+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:29.580469+0000) 2024-03-06T18:23:30.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:30.420 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:30.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:30.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:30.580692+0000) 2024-03-06T18:23:31.576 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:31.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:31.580930+0000) 2024-03-06T18:23:32.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:32.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:32.581162+0000) 2024-03-06T18:23:33.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:33.579+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:33.581431+0000) 2024-03-06T18:23:34.577 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:34.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:34.581714+0000) 2024-03-06T18:23:35.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:35.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:35.581940+0000) 2024-03-06T18:23:36.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:36.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:36.582222+0000) 2024-03-06T18:23:37.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:37.580+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:37.582484+0000) 2024-03-06T18:23:38.578 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:38.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:38.582796+0000) 2024-03-06T18:23:39.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:39.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:39.583029+0000) 2024-03-06T18:23:40.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:40.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:40.583300+0000) 2024-03-06T18:23:41.579 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:41.582+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:41.583574+0000) 2024-03-06T18:23:42.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:42.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:42.583806+0000) 2024-03-06T18:23:43.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:43.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:43.584034+0000) 2024-03-06T18:23:44.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:44.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:44.584298+0000) 2024-03-06T18:23:45.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:45.583+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:45.584562+0000) 2024-03-06T18:23:46.580 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:46.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:46.584781+0000) 2024-03-06T18:23:47.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:47.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:47.584972+0000) 2024-03-06T18:23:48.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:48.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:48.585204+0000) 2024-03-06T18:23:49.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:49.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:49.585425+0000) 2024-03-06T18:23:50.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:50.584+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:50.585646+0000) 2024-03-06T18:23:51.581 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:51.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:51.585896+0000) 2024-03-06T18:23:52.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:52.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:52.586160+0000) 2024-03-06T18:23:53.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:53.585+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:53.586423+0000) 2024-03-06T18:23:54.582 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:54.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:54.586732+0000) 2024-03-06T18:23:55.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:55.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:55.586973+0000) 2024-03-06T18:23:56.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:56.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:56.587250+0000) 2024-03-06T18:23:57.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:57.586+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:57.587512+0000) 2024-03-06T18:23:58.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:58.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:58.587815+0000) 2024-03-06T18:23:59.583 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:23:59.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:59.588042+0000) 2024-03-06T18:24:00.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:00.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:00.588310+0000) 2024-03-06T18:24:00.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:00.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:01.584 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:01.587+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:01.588583+0000) 2024-03-06T18:24:02.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:02.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:02.588874+0000) 2024-03-06T18:24:03.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:03.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:03.589116+0000) 2024-03-06T18:24:04.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:04.588+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:04.589379+0000) 2024-03-06T18:24:05.585 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:05.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:05.589755+0000) 2024-03-06T18:24:06.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:06.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:06.590001+0000) 2024-03-06T18:24:07.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:07.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:07.590263+0000) 2024-03-06T18:24:08.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:08.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:08.590462+0000) 2024-03-06T18:24:09.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:09.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:09.590710+0000) 2024-03-06T18:24:10.586 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:10.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:10.590961+0000) 2024-03-06T18:24:11.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:11.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:11.591224+0000) 2024-03-06T18:24:12.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:12.589+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:12.591473+0000) 2024-03-06T18:24:13.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:13.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:13.591751+0000) 2024-03-06T18:24:14.587 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:14.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:14.591998+0000) 2024-03-06T18:24:15.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:15.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:15.592233+0000) 2024-03-06T18:24:16.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:16.590+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:16.592519+0000) 2024-03-06T18:24:17.588 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:17.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:17.592815+0000) 2024-03-06T18:24:18.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:18.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:18.593057+0000) 2024-03-06T18:24:19.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:19.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:19.593337+0000) 2024-03-06T18:24:20.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:20.591+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:20.593601+0000) 2024-03-06T18:24:21.589 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:21.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:21.593866+0000) 2024-03-06T18:24:22.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:22.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:22.594087+0000) 2024-03-06T18:24:23.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:23.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:23.594279+0000) 2024-03-06T18:24:24.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:24.592+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:24.594503+0000) 2024-03-06T18:24:25.590 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:25.593+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:25.594771+0000) 2024-03-06T18:24:26.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:26.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:26.594999+0000) 2024-03-06T18:24:27.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:27.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:27.595259+0000) 2024-03-06T18:24:28.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:28.594+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:28.595466+0000) 2024-03-06T18:24:29.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:29.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:29.595762+0000) 2024-03-06T18:24:30.591 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:30.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:30.595981+0000) 2024-03-06T18:24:30.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:30.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:31.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:31.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:31.596268+0000) 2024-03-06T18:24:32.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:32.595+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:32.596571+0000) 2024-03-06T18:24:33.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:33.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:33.596813+0000) 2024-03-06T18:24:34.592 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:34.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:34.597031+0000) 2024-03-06T18:24:35.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:35.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:35.597267+0000) 2024-03-06T18:24:36.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:36.596+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:36.597524+0000) 2024-03-06T18:24:37.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:37.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:37.597797+0000) 2024-03-06T18:24:38.593 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:38.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:38.598043+0000) 2024-03-06T18:24:39.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:39.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:39.598282+0000) 2024-03-06T18:24:40.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:40.597+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:40.598521+0000) 2024-03-06T18:24:41.594 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:41.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:41.598816+0000) 2024-03-06T18:24:42.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:42.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:42.599068+0000) 2024-03-06T18:24:43.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:43.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:43.599242+0000) 2024-03-06T18:24:44.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:44.598+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:44.599474+0000) 2024-03-06T18:24:45.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:45.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:45.599728+0000) 2024-03-06T18:24:46.595 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:46.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:46.599965+0000) 2024-03-06T18:24:47.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:47.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:47.600153+0000) 2024-03-06T18:24:48.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:48.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:48.600384+0000) 2024-03-06T18:24:49.596 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:49.599+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:49.600613+0000) 2024-03-06T18:24:50.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:50.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:50.600852+0000) 2024-03-06T18:24:51.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:51.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:51.601045+0000) 2024-03-06T18:24:52.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:52.600+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:52.601336+0000) 2024-03-06T18:24:53.597 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:53.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:53.601652+0000) 2024-03-06T18:24:54.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:54.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:54.601977+0000) 2024-03-06T18:24:55.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:55.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:55.602238+0000) 2024-03-06T18:24:56.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:56.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:56.602516+0000) 2024-03-06T18:24:57.598 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:57.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:57.602779+0000) 2024-03-06T18:24:58.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:58.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:58.602983+0000) 2024-03-06T18:24:59.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:24:59.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:59.603163+0000) 2024-03-06T18:25:00.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:00.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:00.603353+0000) 2024-03-06T18:25:01.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:01.023 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:01.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:01.601+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:01.603601+0000) 2024-03-06T18:25:02.599 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:02.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:02.603831+0000) 2024-03-06T18:25:03.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:03.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:03.604069+0000) 2024-03-06T18:25:04.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:04.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:04.604311+0000) 2024-03-06T18:25:05.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:05.602+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:05.604495+0000) 2024-03-06T18:25:06.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:06.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:06.604743+0000) 2024-03-06T18:25:07.600 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:07.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:07.604972+0000) 2024-03-06T18:25:08.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:08.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:08.605195+0000) 2024-03-06T18:25:09.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:09.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:09.605419+0000) 2024-03-06T18:25:10.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:10.603+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:10.605615+0000) 2024-03-06T18:25:11.601 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:11.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:11.605874+0000) 2024-03-06T18:25:12.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:12.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:12.606122+0000) 2024-03-06T18:25:13.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:13.604+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:13.606322+0000) 2024-03-06T18:25:14.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:14.605+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:14.606541+0000) 2024-03-06T18:25:15.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:15.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:15.606787+0000) 2024-03-06T18:25:16.602 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:16.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:16.607004+0000) 2024-03-06T18:25:17.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:17.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:17.607236+0000) 2024-03-06T18:25:18.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:18.606+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:18.607463+0000) 2024-03-06T18:25:19.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:19.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:19.607732+0000) 2024-03-06T18:25:20.603 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:20.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:20.607930+0000) 2024-03-06T18:25:21.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:21.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:21.608186+0000) 2024-03-06T18:25:22.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:22.607+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:22.608467+0000) 2024-03-06T18:25:23.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:23.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:23.608742+0000) 2024-03-06T18:25:24.604 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:24.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:24.608965+0000) 2024-03-06T18:25:25.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:25.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:25.609218+0000) 2024-03-06T18:25:26.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:26.608+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:26.609506+0000) 2024-03-06T18:25:27.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:27.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:27.609780+0000) 2024-03-06T18:25:28.605 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:28.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:28.609968+0000) 2024-03-06T18:25:29.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:29.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:29.610207+0000) 2024-03-06T18:25:30.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:30.609+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:30.610450+0000) 2024-03-06T18:25:31.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:31.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:31.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:31.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:31.610720+0000) 2024-03-06T18:25:32.606 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:32.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:32.610952+0000) 2024-03-06T18:25:33.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:33.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:33.611301+0000) 2024-03-06T18:25:34.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:34.610+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:34.611535+0000) 2024-03-06T18:25:35.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:35.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:35.611811+0000) 2024-03-06T18:25:36.607 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:36.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:36.612015+0000) 2024-03-06T18:25:37.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:37.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:37.612243+0000) 2024-03-06T18:25:38.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:38.611+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:38.612474+0000) 2024-03-06T18:25:39.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:39.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:39.612738+0000) 2024-03-06T18:25:40.608 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:40.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:40.612955+0000) 2024-03-06T18:25:41.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:41.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:41.613181+0000) 2024-03-06T18:25:42.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:42.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:42.613425+0000) 2024-03-06T18:25:43.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:43.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:43.613655+0000) 2024-03-06T18:25:44.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:44.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:44.613917+0000) 2024-03-06T18:25:45.609 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:45.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:45.614112+0000) 2024-03-06T18:25:46.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:46.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:46.614357+0000) 2024-03-06T18:25:47.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:47.612+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:47.614644+0000) 2024-03-06T18:25:48.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:48.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:48.614909+0000) 2024-03-06T18:25:49.610 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:49.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:49.615100+0000) 2024-03-06T18:25:50.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:50.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:50.615325+0000) 2024-03-06T18:25:51.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:51.613+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:51.615554+0000) 2024-03-06T18:25:52.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:52.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:52.615792+0000) 2024-03-06T18:25:53.611 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:53.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:53.616015+0000) 2024-03-06T18:25:54.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:54.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:54.616188+0000) 2024-03-06T18:25:55.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:55.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:55.616415+0000) 2024-03-06T18:25:56.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:56.614+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:56.616648+0000) 2024-03-06T18:25:57.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:57.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:57.616904+0000) 2024-03-06T18:25:58.612 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:58.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:58.617132+0000) 2024-03-06T18:25:59.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:25:59.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:59.617348+0000) 2024-03-06T18:26:00.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:00.615+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:00.617621+0000) 2024-03-06T18:26:01.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:01.426 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:01.613 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:01.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:01.617915+0000) 2024-03-06T18:26:02.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:02.616+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:02.618228+0000) 2024-03-06T18:26:03.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:03.617+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:03.618502+0000) 2024-03-06T18:26:04.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:04.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:04.618805+0000) 2024-03-06T18:26:05.614 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:05.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:05.619029+0000) 2024-03-06T18:26:06.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:06.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:06.619218+0000) 2024-03-06T18:26:07.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:07.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:07.619441+0000) 2024-03-06T18:26:08.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:08.618+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:08.619643+0000) 2024-03-06T18:26:09.615 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:09.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:09.619874+0000) 2024-03-06T18:26:10.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:10.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:10.620106+0000) 2024-03-06T18:26:11.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:11.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:11.620343+0000) 2024-03-06T18:26:12.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:12.619+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:12.620601+0000) 2024-03-06T18:26:13.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:13.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:13.620835+0000) 2024-03-06T18:26:14.616 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:14.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:14.621076+0000) 2024-03-06T18:26:15.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:15.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:15.621322+0000) 2024-03-06T18:26:16.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:16.620+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:16.621573+0000) 2024-03-06T18:26:17.617 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:17.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:17.621829+0000) 2024-03-06T18:26:18.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:18.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:18.622079+0000) 2024-03-06T18:26:19.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:19.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:19.622354+0000) 2024-03-06T18:26:20.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:20.621+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:20.622639+0000) 2024-03-06T18:26:21.618 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:21.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:21.622930+0000) 2024-03-06T18:26:22.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:22.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:22.623198+0000) 2024-03-06T18:26:23.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:23.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:23.623408+0000) 2024-03-06T18:26:24.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:24.622+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:24.623642+0000) 2024-03-06T18:26:25.619 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:25.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:25.623907+0000) 2024-03-06T18:26:26.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:26.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:26.624142+0000) 2024-03-06T18:26:27.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:27.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:27.624375+0000) 2024-03-06T18:26:28.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:28.623+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:28.624556+0000) 2024-03-06T18:26:29.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:29.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:29.624787+0000) 2024-03-06T18:26:30.620 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:30.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:30.625002+0000) 2024-03-06T18:26:31.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:31.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:31.625192+0000) 2024-03-06T18:26:31.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:31.656 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:32.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:32.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:32.625446+0000) 2024-03-06T18:26:33.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:33.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:33.625703+0000) 2024-03-06T18:26:34.621 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:34.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:34.625914+0000) 2024-03-06T18:26:35.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:35.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:35.626196+0000) 2024-03-06T18:26:36.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:36.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:36.626387+0000) 2024-03-06T18:26:37.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:37.624+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:37.626614+0000) 2024-03-06T18:26:38.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:38.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:38.626852+0000) 2024-03-06T18:26:39.622 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:39.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:39.627080+0000) 2024-03-06T18:26:40.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:40.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:40.627310+0000) 2024-03-06T18:26:41.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:41.625+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:41.627540+0000) 2024-03-06T18:26:42.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:42.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:42.627781+0000) 2024-03-06T18:26:43.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:43.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:43.628011+0000) 2024-03-06T18:26:44.623 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:44.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:44.628194+0000) 2024-03-06T18:26:45.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:45.626+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:45.628462+0000) 2024-03-06T18:26:46.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:46.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:46.628659+0000) 2024-03-06T18:26:47.624 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:47.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:47.628932+0000) 2024-03-06T18:26:48.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:48.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:48.629158+0000) 2024-03-06T18:26:49.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:49.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:49.629379+0000) 2024-03-06T18:26:50.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:50.627+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:50.629575+0000) 2024-03-06T18:26:51.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:51.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:51.629815+0000) 2024-03-06T18:26:52.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:52.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:52.630015+0000) 2024-03-06T18:26:53.625 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:53.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:53.630210+0000) 2024-03-06T18:26:54.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:54.629+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:54.630436+0000) 2024-03-06T18:26:55.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:55.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:55.630656+0000) 2024-03-06T18:26:56.626 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:56.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:56.630927+0000) 2024-03-06T18:26:57.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:57.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:57.631142+0000) 2024-03-06T18:26:58.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:58.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:58.631319+0000) 2024-03-06T18:26:59.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:26:59.630+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:59.631542+0000) 2024-03-06T18:27:00.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:00.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:00.631756+0000) 2024-03-06T18:27:01.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:01.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:01.631984+0000) 2024-03-06T18:27:01.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:01.884 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:02.627 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:02.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:02.632183+0000) 2024-03-06T18:27:03.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:03.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:03.632377+0000) 2024-03-06T18:27:04.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:04.631+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:04.632571+0000) 2024-03-06T18:27:05.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:05.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:05.632778+0000) 2024-03-06T18:27:06.628 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:06.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:06.633004+0000) 2024-03-06T18:27:07.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:07.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:07.633227+0000) 2024-03-06T18:27:08.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:08.632+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:08.633447+0000) 2024-03-06T18:27:09.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:09.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:09.633697+0000) 2024-03-06T18:27:10.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:10.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:10.633913+0000) 2024-03-06T18:27:11.629 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:11.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:11.634131+0000) 2024-03-06T18:27:12.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:12.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:12.634409+0000) 2024-03-06T18:27:13.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:13.633+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:13.634637+0000) 2024-03-06T18:27:14.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:14.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:14.634864+0000) 2024-03-06T18:27:15.630 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:15.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:15.635090+0000) 2024-03-06T18:27:16.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:16.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:16.635285+0000) 2024-03-06T18:27:17.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:17.634+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:17.635477+0000) 2024-03-06T18:27:18.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:18.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:18.635724+0000) 2024-03-06T18:27:19.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:19.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:19.635947+0000) 2024-03-06T18:27:20.631 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:20.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:20.636173+0000) 2024-03-06T18:27:21.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:21.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:21.636400+0000) 2024-03-06T18:27:22.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:22.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:22.636633+0000) 2024-03-06T18:27:23.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:23.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:23.636840+0000) 2024-03-06T18:27:24.632 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:24.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:24.637042+0000) 2024-03-06T18:27:25.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:25.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:25.637265+0000) 2024-03-06T18:27:26.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:26.635+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:26.637490+0000) 2024-03-06T18:27:27.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:27.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:27.637730+0000) 2024-03-06T18:27:28.633 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:28.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:28.638050+0000) 2024-03-06T18:27:29.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:29.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:29.638330+0000) 2024-03-06T18:27:30.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:30.636+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:30.638619+0000) 2024-03-06T18:27:31.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:31.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:31.638836+0000) 2024-03-06T18:27:32.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:32.020 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:32.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:32.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:32.639000+0000) 2024-03-06T18:27:33.634 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:33.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:33.639211+0000) 2024-03-06T18:27:34.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:34.637+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:34.639451+0000) 2024-03-06T18:27:35.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:35.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:35.639698+0000) 2024-03-06T18:27:36.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:36.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:36.639947+0000) 2024-03-06T18:27:37.635 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:37.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:37.640198+0000) 2024-03-06T18:27:38.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:38.638+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:38.640444+0000) 2024-03-06T18:27:39.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:39.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:39.640699+0000) 2024-03-06T18:27:40.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:40.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:40.640933+0000) 2024-03-06T18:27:41.636 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:41.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:41.641168+0000) 2024-03-06T18:27:42.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:42.640+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:42.641429+0000) 2024-03-06T18:27:43.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:43.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:43.641705+0000) 2024-03-06T18:27:44.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:44.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:44.641887+0000) 2024-03-06T18:27:45.637 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:45.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:45.642106+0000) 2024-03-06T18:27:46.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:46.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:46.642292+0000) 2024-03-06T18:27:47.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:47.641+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:47.642492+0000) 2024-03-06T18:27:48.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:48.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:48.642765+0000) 2024-03-06T18:27:49.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:49.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:49.643010+0000) 2024-03-06T18:27:50.638 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:50.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:50.643223+0000) 2024-03-06T18:27:51.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:51.642+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:51.643458+0000) 2024-03-06T18:27:52.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:52.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:52.643719+0000) 2024-03-06T18:27:53.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:53.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:53.643939+0000) 2024-03-06T18:27:54.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:54.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:54.644114+0000) 2024-03-06T18:27:55.639 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:55.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:55.644336+0000) 2024-03-06T18:27:56.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:56.643+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:56.644563+0000) 2024-03-06T18:27:57.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:57.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:57.644792+0000) 2024-03-06T18:27:58.640 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:58.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:58.645023+0000) 2024-03-06T18:27:59.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:27:59.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:59.645218+0000) 2024-03-06T18:28:00.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:00.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:00.645444+0000) 2024-03-06T18:28:01.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:01.644+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:01.645622+0000) 2024-03-06T18:28:02.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:02.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:02.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:02.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:02.645851+0000) 2024-03-06T18:28:03.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:03.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:03.646037+0000) 2024-03-06T18:28:04.641 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:04.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:04.646220+0000) 2024-03-06T18:28:05.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:05.645+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:05.646445+0000) 2024-03-06T18:28:06.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:06.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:06.646697+0000) 2024-03-06T18:28:07.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:07.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:07.646897+0000) 2024-03-06T18:28:08.642 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:08.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:08.647093+0000) 2024-03-06T18:28:09.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:09.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:09.647311+0000) 2024-03-06T18:28:10.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:10.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:10.647480+0000) 2024-03-06T18:28:11.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:11.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:11.647749+0000) 2024-03-06T18:28:12.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:12.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:12.647988+0000) 2024-03-06T18:28:13.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:13.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:13.648180+0000) 2024-03-06T18:28:14.643 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:14.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:14.648368+0000) 2024-03-06T18:28:15.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:15.646+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:15.648546+0000) 2024-03-06T18:28:16.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:16.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:16.648849+0000) 2024-03-06T18:28:17.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:17.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:17.649032+0000) 2024-03-06T18:28:18.644 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:18.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:18.649221+0000) 2024-03-06T18:28:19.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:19.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:19.649395+0000) 2024-03-06T18:28:20.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:20.647+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:20.649633+0000) 2024-03-06T18:28:21.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:21.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:21.649833+0000) 2024-03-06T18:28:22.645 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:22.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:22.650108+0000) 2024-03-06T18:28:23.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:23.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:23.650331+0000) 2024-03-06T18:28:24.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:24.648+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:24.650559+0000) 2024-03-06T18:28:25.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:25.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:25.650780+0000) 2024-03-06T18:28:26.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:26.649+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:26.650977+0000) 2024-03-06T18:28:27.646 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:27.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:27.651209+0000) 2024-03-06T18:28:28.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:28.650+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:28.651431+0000) 2024-03-06T18:28:29.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:29.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:29.651657+0000) 2024-03-06T18:28:30.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:30.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:30.651920+0000) 2024-03-06T18:28:31.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:31.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:31.652140+0000) 2024-03-06T18:28:32.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:32.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:32.647 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:32.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:32.652357+0000) 2024-03-06T18:28:33.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:33.651+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:33.652595+0000) 2024-03-06T18:28:34.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:34.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:34.652832+0000) 2024-03-06T18:28:35.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:35.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:35.653018+0000) 2024-03-06T18:28:36.648 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:36.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:36.653242+0000) 2024-03-06T18:28:37.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:37.652+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:37.653479+0000) 2024-03-06T18:28:38.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:38.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:38.653743+0000) 2024-03-06T18:28:39.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:39.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:39.653958+0000) 2024-03-06T18:28:40.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:40.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:40.654185+0000) 2024-03-06T18:28:41.649 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:41.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:41.654409+0000) 2024-03-06T18:28:42.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:42.653+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:42.654647+0000) 2024-03-06T18:28:43.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:43.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:43.654917+0000) 2024-03-06T18:28:44.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:44.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:44.655105+0000) 2024-03-06T18:28:45.650 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:45.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:45.655281+0000) 2024-03-06T18:28:46.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:46.654+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:46.655513+0000) 2024-03-06T18:28:47.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:47.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:47.655736+0000) 2024-03-06T18:28:48.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:48.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:48.655961+0000) 2024-03-06T18:28:49.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:49.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:49.656190+0000) 2024-03-06T18:28:50.651 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:50.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:50.656413+0000) 2024-03-06T18:28:51.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:51.655+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:51.656590+0000) 2024-03-06T18:28:52.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:52.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:52.656857+0000) 2024-03-06T18:28:53.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:53.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:53.657087+0000) 2024-03-06T18:28:54.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:54.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:54.657256+0000) 2024-03-06T18:28:55.652 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:55.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:55.657399+0000) 2024-03-06T18:28:56.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:56.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:56.657594+0000) 2024-03-06T18:28:57.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:57.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:57.657847+0000) 2024-03-06T18:28:58.653 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:58.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:58.658082+0000) 2024-03-06T18:28:59.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:28:59.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:59.658300+0000) 2024-03-06T18:29:00.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:00.656+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:00.658500+0000) 2024-03-06T18:29:01.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:01.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:01.658751+0000) 2024-03-06T18:29:02.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:02.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:02.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:02.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:02.659034+0000) 2024-03-06T18:29:03.654 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:03.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:03.659315+0000) 2024-03-06T18:29:04.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:04.657+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:04.659591+0000) 2024-03-06T18:29:05.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:05.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:05.659793+0000) 2024-03-06T18:29:06.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:06.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:06.660007+0000) 2024-03-06T18:29:07.655 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:07.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:07.660252+0000) 2024-03-06T18:29:08.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:08.658+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:08.660512+0000) 2024-03-06T18:29:09.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:09.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:09.660823+0000) 2024-03-06T18:29:10.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:10.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:10.661103+0000) 2024-03-06T18:29:11.656 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:11.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:11.661341+0000) 2024-03-06T18:29:12.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:12.659+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:12.661630+0000) 2024-03-06T18:29:13.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:13.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:13.661931+0000) 2024-03-06T18:29:14.657 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:14.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:14.662215+0000) 2024-03-06T18:29:15.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:15.660+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:15.662483+0000) 2024-03-06T18:29:16.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:16.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:16.662800+0000) 2024-03-06T18:29:17.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:17.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:17.663075+0000) 2024-03-06T18:29:18.658 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:18.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:18.663322+0000) 2024-03-06T18:29:19.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:19.662+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:19.663581+0000) 2024-03-06T18:29:20.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:20.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:20.663855+0000) 2024-03-06T18:29:21.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:21.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:21.664137+0000) 2024-03-06T18:29:22.659 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:22.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:22.664366+0000) 2024-03-06T18:29:23.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:23.663+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:23.664561+0000) 2024-03-06T18:29:24.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:24.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:24.664780+0000) 2024-03-06T18:29:25.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:25.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:25.665002+0000) 2024-03-06T18:29:26.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:26.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:26.665223+0000) 2024-03-06T18:29:27.660 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:27.664+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:27.665491+0000) 2024-03-06T18:29:28.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:28.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:28.665807+0000) 2024-03-06T18:29:29.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:29.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:29.666006+0000) 2024-03-06T18:29:30.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:30.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:30.666241+0000) 2024-03-06T18:29:31.661 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:31.665+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:31.666438+0000) 2024-03-06T18:29:32.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:32.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:32.666655+0000) 2024-03-06T18:29:32.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:32.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:33.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:33.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:33.666885+0000) 2024-03-06T18:29:34.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:34.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:34.667111+0000) 2024-03-06T18:29:35.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:35.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:35.667280+0000) 2024-03-06T18:29:36.662 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:36.666+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:36.667509+0000) 2024-03-06T18:29:37.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:37.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:37.667768+0000) 2024-03-06T18:29:38.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:38.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:38.667993+0000) 2024-03-06T18:29:39.663 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:39.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:39.668218+0000) 2024-03-06T18:29:40.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:40.667+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:40.668438+0000) 2024-03-06T18:29:41.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:41.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:41.668654+0000) 2024-03-06T18:29:42.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:42.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:42.668887+0000) 2024-03-06T18:29:43.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:43.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:43.669106+0000) 2024-03-06T18:29:44.664 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:44.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:44.669328+0000) 2024-03-06T18:29:45.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:45.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:45.669552+0000) 2024-03-06T18:29:46.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:46.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:46.669785+0000) 2024-03-06T18:29:47.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:47.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:47.670005+0000) 2024-03-06T18:29:48.665 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:48.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:48.670248+0000) 2024-03-06T18:29:49.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:49.668+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:49.670473+0000) 2024-03-06T18:29:50.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:50.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:50.670735+0000) 2024-03-06T18:29:51.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:51.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:51.670963+0000) 2024-03-06T18:29:52.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:52.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:52.671206+0000) 2024-03-06T18:29:53.666 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:53.669+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:53.671440+0000) 2024-03-06T18:29:54.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:54.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:54.671705+0000) 2024-03-06T18:29:55.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:55.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:55.671948+0000) 2024-03-06T18:29:56.667 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:56.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:56.672191+0000) 2024-03-06T18:29:57.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:57.670+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:57.672428+0000) 2024-03-06T18:29:58.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:58.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:58.672713+0000) 2024-03-06T18:29:59.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:29:59.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:59.672954+0000) 2024-03-06T18:30:00.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:00.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:00.673171+0000) 2024-03-06T18:30:01.668 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:01.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:01.673399+0000) 2024-03-06T18:30:02.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:02.671+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:02.673646+0000) 2024-03-06T18:30:03.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:03.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:03.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:03.672+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:03.673886+0000) 2024-03-06T18:30:04.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:04.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:04.674081+0000) 2024-03-06T18:30:05.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:05.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:05.674301+0000) 2024-03-06T18:30:06.669 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:06.673+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:06.674529+0000) 2024-03-06T18:30:07.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:07.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:07.674808+0000) 2024-03-06T18:30:08.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:08.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:08.674997+0000) 2024-03-06T18:30:09.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:09.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:09.675192+0000) 2024-03-06T18:30:10.670 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:10.674+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:10.675434+0000) 2024-03-06T18:30:11.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:11.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:11.675748+0000) 2024-03-06T18:30:12.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:12.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:12.676055+0000) 2024-03-06T18:30:13.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:13.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:13.676322+0000) 2024-03-06T18:30:14.671 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:14.675+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:14.676516+0000) 2024-03-06T18:30:15.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:15.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:15.676784+0000) 2024-03-06T18:30:16.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:16.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:16.676993+0000) 2024-03-06T18:30:17.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:17.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:17.677225+0000) 2024-03-06T18:30:18.672 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:18.676+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:18.677477+0000) 2024-03-06T18:30:19.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:19.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:19.677743+0000) 2024-03-06T18:30:20.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:20.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:20.677957+0000) 2024-03-06T18:30:21.673 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:21.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:21.678219+0000) 2024-03-06T18:30:22.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:22.677+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:22.678523+0000) 2024-03-06T18:30:23.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:23.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:23.678810+0000) 2024-03-06T18:30:24.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:24.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:24.679090+0000) 2024-03-06T18:30:25.674 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:25.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:25.679284+0000) 2024-03-06T18:30:26.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:26.678+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:26.679520+0000) 2024-03-06T18:30:27.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:27.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:27.679799+0000) 2024-03-06T18:30:28.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:28.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:28.680029+0000) 2024-03-06T18:30:29.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:29.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:29.680248+0000) 2024-03-06T18:30:30.675 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:30.679+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:30.680472+0000) 2024-03-06T18:30:31.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:31.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:31.680755+0000) 2024-03-06T18:30:32.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:32.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:32.681058+0000) 2024-03-06T18:30:33.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:33.219 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:33.676 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:33.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:33.681338+0000) 2024-03-06T18:30:34.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:34.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:34.681623+0000) 2024-03-06T18:30:35.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:35.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:35.681924+0000) 2024-03-06T18:30:36.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:36.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:36.682208+0000) 2024-03-06T18:30:37.677 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:37.680+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:37.682504+0000) 2024-03-06T18:30:38.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:38.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:38.682833+0000) 2024-03-06T18:30:39.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:39.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:39.683083+0000) 2024-03-06T18:30:40.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:40.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:40.683317+0000) 2024-03-06T18:30:41.678 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:41.681+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:41.683537+0000) 2024-03-06T18:30:42.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:42.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:42.683771+0000) 2024-03-06T18:30:43.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:43.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:43.684001+0000) 2024-03-06T18:30:44.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:44.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:44.684248+0000) 2024-03-06T18:30:45.679 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:45.682+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:45.684501+0000) 2024-03-06T18:30:46.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:46.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:46.684764+0000) 2024-03-06T18:30:47.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:47.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:47.684987+0000) 2024-03-06T18:30:48.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:48.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:48.685269+0000) 2024-03-06T18:30:49.680 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:49.683+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:49.685540+0000) 2024-03-06T18:30:50.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:50.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:50.685821+0000) 2024-03-06T18:30:51.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:51.684+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:51.686051+0000) 2024-03-06T18:30:52.681 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:52.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:52.686292+0000) 2024-03-06T18:30:53.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:53.685+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:53.686536+0000) 2024-03-06T18:30:54.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:54.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:54.686787+0000) 2024-03-06T18:30:55.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:55.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:55.686976+0000) 2024-03-06T18:30:56.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:56.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:56.687180+0000) 2024-03-06T18:30:57.682 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:57.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:57.687407+0000) 2024-03-06T18:30:58.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:58.686+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:58.687632+0000) 2024-03-06T18:30:59.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:30:59.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:59.687871+0000) 2024-03-06T18:31:00.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:00.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:00.688107+0000) 2024-03-06T18:31:01.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:01.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:01.688351+0000) 2024-03-06T18:31:02.683 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:02.687+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:02.688612+0000) 2024-03-06T18:31:03.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:03.408 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:03.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:03.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:03.688849+0000) 2024-03-06T18:31:04.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:04.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:04.689045+0000) 2024-03-06T18:31:05.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:05.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:05.689238+0000) 2024-03-06T18:31:06.684 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:06.688+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:06.689425+0000) 2024-03-06T18:31:07.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:07.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:07.689713+0000) 2024-03-06T18:31:08.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:08.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:08.689912+0000) 2024-03-06T18:31:09.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:09.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:09.690108+0000) 2024-03-06T18:31:10.685 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:10.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:10.690341+0000) 2024-03-06T18:31:11.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:11.689+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:11.690555+0000) 2024-03-06T18:31:12.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:12.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:12.690789+0000) 2024-03-06T18:31:13.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:13.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:13.691001+0000) 2024-03-06T18:31:14.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:14.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:14.691241+0000) 2024-03-06T18:31:15.686 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:15.690+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:15.691461+0000) 2024-03-06T18:31:16.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:16.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:16.691718+0000) 2024-03-06T18:31:17.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:17.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:17.691952+0000) 2024-03-06T18:31:18.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:18.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:18.692175+0000) 2024-03-06T18:31:19.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:19.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:19.692391+0000) 2024-03-06T18:31:20.687 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:20.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:20.692604+0000) 2024-03-06T18:31:21.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:21.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:21.692820+0000) 2024-03-06T18:31:22.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:22.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:22.693052+0000) 2024-03-06T18:31:23.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:23.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:23.693231+0000) 2024-03-06T18:31:24.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:24.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:24.693460+0000) 2024-03-06T18:31:25.688 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:25.691+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:25.693635+0000) 2024-03-06T18:31:26.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:26.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:26.693937+0000) 2024-03-06T18:31:27.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:27.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:27.694124+0000) 2024-03-06T18:31:28.689 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:28.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:28.694324+0000) 2024-03-06T18:31:29.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:29.692+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:29.694554+0000) 2024-03-06T18:31:30.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:30.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:30.694779+0000) 2024-03-06T18:31:31.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:31.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:31.695005+0000) 2024-03-06T18:31:32.690 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:32.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:32.695279+0000) 2024-03-06T18:31:33.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:33.603 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:33.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:33.693+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:33.695535+0000) 2024-03-06T18:31:34.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:34.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:34.695763+0000) 2024-03-06T18:31:35.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:35.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:35.695992+0000) 2024-03-06T18:31:36.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:36.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:36.696241+0000) 2024-03-06T18:31:37.691 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:37.694+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:37.696497+0000) 2024-03-06T18:31:38.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:38.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:38.696795+0000) 2024-03-06T18:31:39.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:39.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:39.697034+0000) 2024-03-06T18:31:40.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:40.695+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:40.697259+0000) 2024-03-06T18:31:41.692 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:41.696+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:41.697451+0000) 2024-03-06T18:31:42.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:42.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:42.697740+0000) 2024-03-06T18:31:43.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:43.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:43.697935+0000) 2024-03-06T18:31:44.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:44.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:44.698141+0000) 2024-03-06T18:31:45.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:45.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:45.698372+0000) 2024-03-06T18:31:46.693 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:46.697+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:46.698555+0000) 2024-03-06T18:31:47.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:47.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:47.698830+0000) 2024-03-06T18:31:48.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:48.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:48.699080+0000) 2024-03-06T18:31:49.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:49.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:49.699323+0000) 2024-03-06T18:31:50.694 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:50.698+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:50.699526+0000) 2024-03-06T18:31:51.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:51.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:51.699770+0000) 2024-03-06T18:31:52.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:52.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:52.699997+0000) 2024-03-06T18:31:53.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:53.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:53.700279+0000) 2024-03-06T18:31:54.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:54.699+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:54.700555+0000) 2024-03-06T18:31:55.695 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:55.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:55.700793+0000) 2024-03-06T18:31:56.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:56.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:56.701022+0000) 2024-03-06T18:31:57.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:57.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:57.701262+0000) 2024-03-06T18:31:58.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:58.700+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:58.701476+0000) 2024-03-06T18:31:59.696 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:31:59.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:59.701778+0000) 2024-03-06T18:32:00.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:00.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:00.702045+0000) 2024-03-06T18:32:01.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:01.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:01.702323+0000) 2024-03-06T18:32:02.697 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:02.701+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:02.702613+0000) 2024-03-06T18:32:03.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:03.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:03.702838+0000) 2024-03-06T18:32:03.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:03.786 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:04.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:04.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:04.703066+0000) 2024-03-06T18:32:05.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:05.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:05.703292+0000) 2024-03-06T18:32:06.698 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:06.702+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:06.703522+0000) 2024-03-06T18:32:07.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:07.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:07.703830+0000) 2024-03-06T18:32:08.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:08.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:08.704103+0000) 2024-03-06T18:32:09.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:09.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:09.704384+0000) 2024-03-06T18:32:10.699 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:10.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:10.704614+0000) 2024-03-06T18:32:11.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:11.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:11.704829+0000) 2024-03-06T18:32:12.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:12.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:12.705059+0000) 2024-03-06T18:32:13.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:13.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:13.705326+0000) 2024-03-06T18:32:14.700 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:14.703+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:14.705605+0000) 2024-03-06T18:32:15.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:15.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:15.705865+0000) 2024-03-06T18:32:16.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:16.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:16.706152+0000) 2024-03-06T18:32:17.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:17.704+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:17.706419+0000) 2024-03-06T18:32:18.701 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:18.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:18.706738+0000) 2024-03-06T18:32:19.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:19.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:19.707016+0000) 2024-03-06T18:32:20.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:20.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:20.707225+0000) 2024-03-06T18:32:21.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:21.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:21.707415+0000) 2024-03-06T18:32:22.702 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:22.705+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:22.707640+0000) 2024-03-06T18:32:23.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:23.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:23.707952+0000) 2024-03-06T18:32:24.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:24.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:24.708233+0000) 2024-03-06T18:32:25.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:25.706+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:25.708497+0000) 2024-03-06T18:32:26.703 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:26.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:26.708740+0000) 2024-03-06T18:32:27.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:27.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:27.709016+0000) 2024-03-06T18:32:28.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:28.707+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:28.709240+0000) 2024-03-06T18:32:29.704 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:29.708+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:29.709470+0000) 2024-03-06T18:32:30.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:30.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:30.709770+0000) 2024-03-06T18:32:31.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:31.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:31.710038+0000) 2024-03-06T18:32:32.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:32.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:32.710308+0000) 2024-03-06T18:32:33.705 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:33.709+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:33.710589+0000) 2024-03-06T18:32:33.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:33.996 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:34.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:34.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:34.710870+0000) 2024-03-06T18:32:35.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:35.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:35.711144+0000) 2024-03-06T18:32:36.706 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:36.710+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:36.711424+0000) 2024-03-06T18:32:37.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:37.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:37.711754+0000) 2024-03-06T18:32:38.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:38.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:38.712043+0000) 2024-03-06T18:32:39.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:39.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:39.712320+0000) 2024-03-06T18:32:40.707 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:40.711+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:40.712596+0000) 2024-03-06T18:32:41.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:41.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:41.712878+0000) 2024-03-06T18:32:42.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:42.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:42.713169+0000) 2024-03-06T18:32:43.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:43.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:43.713373+0000) 2024-03-06T18:32:44.708 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:44.712+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:44.713610+0000) 2024-03-06T18:32:45.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:45.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:45.713849+0000) 2024-03-06T18:32:46.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:46.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:46.714090+0000) 2024-03-06T18:32:47.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:47.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:47.714327+0000) 2024-03-06T18:32:48.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:48.713+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:48.714541+0000) 2024-03-06T18:32:49.709 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:49.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:49.714820+0000) 2024-03-06T18:32:50.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:50.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:50.715032+0000) 2024-03-06T18:32:51.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:51.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:51.715257+0000) 2024-03-06T18:32:52.710 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:52.714+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:52.715489+0000) 2024-03-06T18:32:53.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:53.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:53.715788+0000) 2024-03-06T18:32:54.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:54.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:54.716067+0000) 2024-03-06T18:32:55.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:55.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:55.716307+0000) 2024-03-06T18:32:56.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:56.715+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:56.716558+0000) 2024-03-06T18:32:57.711 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:57.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:57.716788+0000) 2024-03-06T18:32:58.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:58.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:58.717001+0000) 2024-03-06T18:32:59.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:32:59.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:59.717231+0000) 2024-03-06T18:33:00.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:00.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:00.717459+0000) 2024-03-06T18:33:01.712 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:01.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:01.717655+0000) 2024-03-06T18:33:02.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:02.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:02.717962+0000) 2024-03-06T18:33:03.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:03.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:03.718189+0000) 2024-03-06T18:33:04.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:04.175 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:04.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:04.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:04.718427+0000) 2024-03-06T18:33:05.713 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:05.716+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:05.718641+0000) 2024-03-06T18:33:06.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:06.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:06.718874+0000) 2024-03-06T18:33:07.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:07.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:07.719101+0000) 2024-03-06T18:33:08.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:08.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:08.719334+0000) 2024-03-06T18:33:09.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:09.717+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:09.719529+0000) 2024-03-06T18:33:10.714 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:10.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:10.719811+0000) 2024-03-06T18:33:11.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:11.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:11.720091+0000) 2024-03-06T18:33:12.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:12.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:12.720307+0000) 2024-03-06T18:33:13.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:13.718+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:13.720546+0000) 2024-03-06T18:33:14.715 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:14.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:14.720782+0000) 2024-03-06T18:33:15.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:15.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:15.721057+0000) 2024-03-06T18:33:16.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:16.719+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:16.721281+0000) 2024-03-06T18:33:17.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:17.720+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:17.721514+0000) 2024-03-06T18:33:18.716 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:18.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:18.721742+0000) 2024-03-06T18:33:19.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:19.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:19.722013+0000) 2024-03-06T18:33:20.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:20.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:20.722280+0000) 2024-03-06T18:33:21.717 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:21.721+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:21.722563+0000) 2024-03-06T18:33:22.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:22.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:22.722856+0000) 2024-03-06T18:33:23.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:23.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:23.723061+0000) 2024-03-06T18:33:24.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:24.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:24.723254+0000) 2024-03-06T18:33:25.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:25.722+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:25.723452+0000) 2024-03-06T18:33:26.718 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:26.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:26.723713+0000) 2024-03-06T18:33:27.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:27.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:27.723904+0000) 2024-03-06T18:33:28.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:28.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:28.724095+0000) 2024-03-06T18:33:29.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:29.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:29.724317+0000) 2024-03-06T18:33:30.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:30.723+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:30.724538+0000) 2024-03-06T18:33:31.719 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:31.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:31.724818+0000) 2024-03-06T18:33:32.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:32.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:32.725018+0000) 2024-03-06T18:33:33.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:33.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:33.725251+0000) 2024-03-06T18:33:34.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:34.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:34.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:34.724+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:34.725487+0000) 2024-03-06T18:33:35.720 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:35.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:35.725752+0000) 2024-03-06T18:33:36.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:36.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:36.725949+0000) 2024-03-06T18:33:37.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:37.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:37.726136+0000) 2024-03-06T18:33:38.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:38.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:38.726327+0000) 2024-03-06T18:33:39.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:39.725+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:39.726564+0000) 2024-03-06T18:33:40.721 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:40.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:40.726789+0000) 2024-03-06T18:33:41.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:41.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:41.726981+0000) 2024-03-06T18:33:42.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:42.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:42.727232+0000) 2024-03-06T18:33:43.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:43.726+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:43.727465+0000) 2024-03-06T18:33:44.722 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:44.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:44.727720+0000) 2024-03-06T18:33:45.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:45.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:45.727909+0000) 2024-03-06T18:33:46.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:46.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:46.728112+0000) 2024-03-06T18:33:47.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:47.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:47.728341+0000) 2024-03-06T18:33:48.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:48.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:48.728567+0000) 2024-03-06T18:33:49.723 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:49.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:49.728796+0000) 2024-03-06T18:33:50.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:50.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:50.729009+0000) 2024-03-06T18:33:51.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:51.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:51.729215+0000) 2024-03-06T18:33:52.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:52.727+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:52.729464+0000) 2024-03-06T18:33:53.724 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:53.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:53.729714+0000) 2024-03-06T18:33:54.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:54.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:54.729905+0000) 2024-03-06T18:33:55.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:55.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:55.730091+0000) 2024-03-06T18:33:56.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:56.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:56.730295+0000) 2024-03-06T18:33:57.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:57.728+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:57.730531+0000) 2024-03-06T18:33:58.725 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:58.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:58.730810+0000) 2024-03-06T18:33:59.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:33:59.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:59.731038+0000) 2024-03-06T18:34:00.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:00.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:00.731265+0000) 2024-03-06T18:34:01.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:01.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:01.731435+0000) 2024-03-06T18:34:02.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:02.729+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:02.731603+0000) 2024-03-06T18:34:03.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:03.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:03.731803+0000) 2024-03-06T18:34:04.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:04.601 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:04.726 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:04.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:04.732052+0000) 2024-03-06T18:34:05.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:05.730+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:05.732286+0000) 2024-03-06T18:34:06.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:06.731+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:06.732509+0000) 2024-03-06T18:34:07.727 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:07.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:07.732750+0000) 2024-03-06T18:34:08.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:08.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:08.733032+0000) 2024-03-06T18:34:09.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:09.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:09.733224+0000) 2024-03-06T18:34:10.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:10.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:10.733405+0000) 2024-03-06T18:34:11.728 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:11.732+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:11.733636+0000) 2024-03-06T18:34:12.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:12.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:12.733869+0000) 2024-03-06T18:34:13.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:13.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:13.734058+0000) 2024-03-06T18:34:14.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:14.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:14.734245+0000) 2024-03-06T18:34:15.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:15.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:15.734430+0000) 2024-03-06T18:34:16.729 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:16.733+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:16.734614+0000) 2024-03-06T18:34:17.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:17.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:17.734838+0000) 2024-03-06T18:34:18.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:18.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:18.735029+0000) 2024-03-06T18:34:19.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:19.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:19.735234+0000) 2024-03-06T18:34:20.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:20.734+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:20.735422+0000) 2024-03-06T18:34:21.730 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:21.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:21.735657+0000) 2024-03-06T18:34:22.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:22.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:22.735861+0000) 2024-03-06T18:34:23.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:23.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:23.736100+0000) 2024-03-06T18:34:24.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:24.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:24.736327+0000) 2024-03-06T18:34:25.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:25.735+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:25.736560+0000) 2024-03-06T18:34:26.731 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:26.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:26.736781+0000) 2024-03-06T18:34:27.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:27.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:27.737003+0000) 2024-03-06T18:34:28.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:28.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:28.737173+0000) 2024-03-06T18:34:29.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:29.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:29.737333+0000) 2024-03-06T18:34:30.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:30.736+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:30.737541+0000) 2024-03-06T18:34:31.732 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:31.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:31.737807+0000) 2024-03-06T18:34:32.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:32.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:32.738043+0000) 2024-03-06T18:34:33.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:33.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:33.738239+0000) 2024-03-06T18:34:34.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:34.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:34.738425+0000) 2024-03-06T18:34:34.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:34.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:35.733 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:35.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:35.738581+0000) 2024-03-06T18:34:36.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:36.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:36.738834+0000) 2024-03-06T18:34:37.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:37.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:37.738984+0000) 2024-03-06T18:34:38.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:38.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:38.739187+0000) 2024-03-06T18:34:39.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:39.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:39.739372+0000) 2024-03-06T18:34:40.734 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:40.737+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:40.739613+0000) 2024-03-06T18:34:41.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:41.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:41.739774+0000) 2024-03-06T18:34:42.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:42.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:42.739937+0000) 2024-03-06T18:34:43.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:43.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:43.740151+0000) 2024-03-06T18:34:44.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:44.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:44.740393+0000) 2024-03-06T18:34:45.735 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:45.738+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:45.740624+0000) 2024-03-06T18:34:46.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:46.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:46.740827+0000) 2024-03-06T18:34:47.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:47.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:47.741047+0000) 2024-03-06T18:34:48.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:48.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:48.741284+0000) 2024-03-06T18:34:49.736 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:49.739+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:49.741513+0000) 2024-03-06T18:34:50.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:50.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:50.741782+0000) 2024-03-06T18:34:51.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:51.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:51.741938+0000) 2024-03-06T18:34:52.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:52.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:52.742163+0000) 2024-03-06T18:34:53.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:53.740+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:53.742410+0000) 2024-03-06T18:34:54.737 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:54.741+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:54.742637+0000) 2024-03-06T18:34:55.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:55.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:55.742896+0000) 2024-03-06T18:34:56.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:56.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:56.743176+0000) 2024-03-06T18:34:57.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:57.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:57.743342+0000) 2024-03-06T18:34:58.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:58.742+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:58.743513+0000) 2024-03-06T18:34:59.738 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:34:59.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:59.743782+0000) 2024-03-06T18:35:00.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:00.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:00.743985+0000) 2024-03-06T18:35:01.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:01.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:01.744163+0000) 2024-03-06T18:35:02.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:02.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:02.744393+0000) 2024-03-06T18:35:03.739 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:03.743+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:03.744626+0000) 2024-03-06T18:35:04.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:04.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:04.744872+0000) 2024-03-06T18:35:05.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:05.008 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:05.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:05.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:05.745073+0000) 2024-03-06T18:35:06.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:06.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:06.745267+0000) 2024-03-06T18:35:07.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:07.744+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:07.745520+0000) 2024-03-06T18:35:08.740 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:08.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:08.745756+0000) 2024-03-06T18:35:09.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:09.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:09.745979+0000) 2024-03-06T18:35:10.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:10.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:10.746163+0000) 2024-03-06T18:35:11.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:11.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:11.746381+0000) 2024-03-06T18:35:12.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:12.745+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:12.746614+0000) 2024-03-06T18:35:13.741 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:13.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:13.746818+0000) 2024-03-06T18:35:14.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:14.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:14.747029+0000) 2024-03-06T18:35:15.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:15.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:15.747222+0000) 2024-03-06T18:35:16.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:16.746+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:16.747497+0000) 2024-03-06T18:35:17.742 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:17.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:17.747779+0000) 2024-03-06T18:35:18.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:18.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:18.748052+0000) 2024-03-06T18:35:19.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:19.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:19.748255+0000) 2024-03-06T18:35:20.743 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:20.747+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:20.748481+0000) 2024-03-06T18:35:21.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:21.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:21.748752+0000) 2024-03-06T18:35:22.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:22.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:22.748958+0000) 2024-03-06T18:35:23.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:23.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:23.749166+0000) 2024-03-06T18:35:24.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:24.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:24.749398+0000) 2024-03-06T18:35:25.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:25.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:25.749582+0000) 2024-03-06T18:35:26.744 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:26.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:26.749815+0000) 2024-03-06T18:35:27.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:27.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:27.749996+0000) 2024-03-06T18:35:28.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:28.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:28.750195+0000) 2024-03-06T18:35:29.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:29.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:29.750413+0000) 2024-03-06T18:35:30.745 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:30.748+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:30.750637+0000) 2024-03-06T18:35:31.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:31.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:31.750883+0000) 2024-03-06T18:35:32.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:32.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:32.751092+0000) 2024-03-06T18:35:33.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:33.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:33.751305+0000) 2024-03-06T18:35:34.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:34.749+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:34.751570+0000) 2024-03-06T18:35:35.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:35.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:35.746 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:35.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:35.751849+0000) 2024-03-06T18:35:36.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:36.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:36.752126+0000) 2024-03-06T18:35:37.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:37.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:37.752356+0000) 2024-03-06T18:35:38.747 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:38.750+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:38.752638+0000) 2024-03-06T18:35:39.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:39.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:39.752961+0000) 2024-03-06T18:35:40.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:40.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:40.753236+0000) 2024-03-06T18:35:41.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:41.751+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:41.753513+0000) 2024-03-06T18:35:42.748 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:42.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:42.753832+0000) 2024-03-06T18:35:43.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:43.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:43.754117+0000) 2024-03-06T18:35:44.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:44.753+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:44.754444+0000) 2024-03-06T18:35:45.749 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:45.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:45.754764+0000) 2024-03-06T18:35:46.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:46.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:46.754954+0000) 2024-03-06T18:35:47.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:47.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:47.755220+0000) 2024-03-06T18:35:48.750 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:48.754+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:48.755512+0000) 2024-03-06T18:35:49.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:49.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:49.755747+0000) 2024-03-06T18:35:50.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:50.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:50.755971+0000) 2024-03-06T18:35:51.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:51.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:51.756150+0000) 2024-03-06T18:35:52.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:52.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:52.756353+0000) 2024-03-06T18:35:53.751 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:53.755+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:53.756588+0000) 2024-03-06T18:35:54.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:54.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:54.756812+0000) 2024-03-06T18:35:55.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:55.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:55.757000+0000) 2024-03-06T18:35:56.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:56.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:56.757185+0000) 2024-03-06T18:35:57.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:57.756+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:57.757403+0000) 2024-03-06T18:35:58.752 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:58.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:58.757655+0000) 2024-03-06T18:35:59.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:35:59.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:59.757891+0000) 2024-03-06T18:36:00.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:00.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:00.758120+0000) 2024-03-06T18:36:01.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:01.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:01.758343+0000) 2024-03-06T18:36:02.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:02.757+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:02.758575+0000) 2024-03-06T18:36:03.753 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:03.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:03.758782+0000) 2024-03-06T18:36:04.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:04.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:04.759005+0000) 2024-03-06T18:36:05.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:05.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:05.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:05.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:05.759220+0000) 2024-03-06T18:36:06.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:06.758+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:06.759463+0000) 2024-03-06T18:36:07.754 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:07.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:07.759719+0000) 2024-03-06T18:36:08.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:08.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:08.759925+0000) 2024-03-06T18:36:09.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:09.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:09.760112+0000) 2024-03-06T18:36:10.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:10.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:10.760298+0000) 2024-03-06T18:36:11.755 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:11.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:11.760514+0000) 2024-03-06T18:36:12.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:12.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:12.760768+0000) 2024-03-06T18:36:13.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:13.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:13.760967+0000) 2024-03-06T18:36:14.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:14.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:14.761166+0000) 2024-03-06T18:36:15.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:15.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:15.761416+0000) 2024-03-06T18:36:16.756 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:16.759+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:16.761648+0000) 2024-03-06T18:36:17.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:17.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:17.761858+0000) 2024-03-06T18:36:18.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:18.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:18.762055+0000) 2024-03-06T18:36:19.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:19.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:19.762247+0000) 2024-03-06T18:36:20.757 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:20.760+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:20.762480+0000) 2024-03-06T18:36:21.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:21.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:21.762735+0000) 2024-03-06T18:36:22.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:22.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:22.762904+0000) 2024-03-06T18:36:23.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:23.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:23.763144+0000) 2024-03-06T18:36:24.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:24.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:24.763371+0000) 2024-03-06T18:36:25.758 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:25.761+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:25.763546+0000) 2024-03-06T18:36:26.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:26.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:26.763779+0000) 2024-03-06T18:36:27.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:27.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:27.764004+0000) 2024-03-06T18:36:28.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:28.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:28.764237+0000) 2024-03-06T18:36:29.759 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:29.762+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:29.764460+0000) 2024-03-06T18:36:30.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:30.763+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:30.764715+0000) 2024-03-06T18:36:31.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:31.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:31.764905+0000) 2024-03-06T18:36:32.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:32.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:32.765095+0000) 2024-03-06T18:36:33.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:33.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:33.765325+0000) 2024-03-06T18:36:34.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:34.764+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:34.765550+0000) 2024-03-06T18:36:35.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:35.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:35.760 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:35.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:35.765759+0000) 2024-03-06T18:36:36.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:36.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:36.765932+0000) 2024-03-06T18:36:37.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:37.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:37.766200+0000) 2024-03-06T18:36:38.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:38.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:38.766416+0000) 2024-03-06T18:36:39.761 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:39.765+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:39.766599+0000) 2024-03-06T18:36:40.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:40.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:40.766818+0000) 2024-03-06T18:36:41.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:41.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:41.767003+0000) 2024-03-06T18:36:42.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:42.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:42.767211+0000) 2024-03-06T18:36:43.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:43.766+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:43.767446+0000) 2024-03-06T18:36:44.762 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:44.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:44.767703+0000) 2024-03-06T18:36:45.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:45.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:45.767859+0000) 2024-03-06T18:36:46.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:46.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:46.768086+0000) 2024-03-06T18:36:47.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:47.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:47.768267+0000) 2024-03-06T18:36:48.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:48.767+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:48.768514+0000) 2024-03-06T18:36:49.763 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:49.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:49.768774+0000) 2024-03-06T18:36:50.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:50.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:50.768979+0000) 2024-03-06T18:36:51.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:51.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:51.769176+0000) 2024-03-06T18:36:52.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:52.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:52.769378+0000) 2024-03-06T18:36:53.764 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:53.768+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:53.769624+0000) 2024-03-06T18:36:54.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:54.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:54.769814+0000) 2024-03-06T18:36:55.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:55.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:55.769995+0000) 2024-03-06T18:36:56.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:56.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:56.770189+0000) 2024-03-06T18:36:57.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:57.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:57.770410+0000) 2024-03-06T18:36:58.765 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:58.769+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:58.770635+0000) 2024-03-06T18:36:59.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:36:59.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:59.770870+0000) 2024-03-06T18:37:00.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:00.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:00.771055+0000) 2024-03-06T18:37:01.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:01.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:01.771238+0000) 2024-03-06T18:37:02.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:02.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:02.771426+0000) 2024-03-06T18:37:03.766 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:03.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:03.771654+0000) 2024-03-06T18:37:04.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:04.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:04.771886+0000) 2024-03-06T18:37:05.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:05.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:05.772115+0000) 2024-03-06T18:37:05.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:05.859 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:06.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:06.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:06.772341+0000) 2024-03-06T18:37:07.767 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:07.770+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:07.772565+0000) 2024-03-06T18:37:08.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:08.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:08.772816+0000) 2024-03-06T18:37:09.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:09.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:09.773011+0000) 2024-03-06T18:37:10.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:10.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:10.773194+0000) 2024-03-06T18:37:11.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:11.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:11.773420+0000) 2024-03-06T18:37:12.768 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:12.771+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:12.773607+0000) 2024-03-06T18:37:13.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:13.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:13.773846+0000) 2024-03-06T18:37:14.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:14.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:14.774003+0000) 2024-03-06T18:37:15.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:15.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:15.774201+0000) 2024-03-06T18:37:16.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:16.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:16.774426+0000) 2024-03-06T18:37:17.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:17.772+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:17.774574+0000) 2024-03-06T18:37:18.769 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:18.773+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:18.774797+0000) 2024-03-06T18:37:19.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:19.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:19.775022+0000) 2024-03-06T18:37:20.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:20.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:20.775255+0000) 2024-03-06T18:37:21.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:21.774+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:21.775490+0000) 2024-03-06T18:37:22.770 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:22.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:22.775772+0000) 2024-03-06T18:37:23.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:23.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:23.775973+0000) 2024-03-06T18:37:24.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:24.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:24.776202+0000) 2024-03-06T18:37:25.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:25.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:25.776335+0000) 2024-03-06T18:37:26.771 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:26.775+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:26.776555+0000) 2024-03-06T18:37:27.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:27.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:27.776781+0000) 2024-03-06T18:37:28.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:28.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:28.776985+0000) 2024-03-06T18:37:29.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:29.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:29.777226+0000) 2024-03-06T18:37:30.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:30.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:30.777406+0000) 2024-03-06T18:37:31.772 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:31.776+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:31.777640+0000) 2024-03-06T18:37:32.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:32.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:32.777832+0000) 2024-03-06T18:37:33.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:33.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:33.777997+0000) 2024-03-06T18:37:34.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:34.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:34.778163+0000) 2024-03-06T18:37:35.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:35.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:35.778327+0000) 2024-03-06T18:37:36.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:36.092 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:36.773 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:36.777+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:36.778558+0000) 2024-03-06T18:37:37.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:37.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:37.778799+0000) 2024-03-06T18:37:38.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:38.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:38.779001+0000) 2024-03-06T18:37:39.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:39.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:39.779189+0000) 2024-03-06T18:37:40.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:40.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:40.779408+0000) 2024-03-06T18:37:41.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:41.778+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:41.779595+0000) 2024-03-06T18:37:42.774 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:42.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:42.779836+0000) 2024-03-06T18:37:43.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:43.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:43.780031+0000) 2024-03-06T18:37:44.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:44.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:44.780228+0000) 2024-03-06T18:37:45.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:45.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:45.780410+0000) 2024-03-06T18:37:46.775 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:46.779+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:46.780636+0000) 2024-03-06T18:37:47.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:47.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:47.780892+0000) 2024-03-06T18:37:48.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:48.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:48.781123+0000) 2024-03-06T18:37:49.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:49.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:49.781309+0000) 2024-03-06T18:37:50.776 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:50.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:50.781503+0000) 2024-03-06T18:37:51.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:51.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:51.781764+0000) 2024-03-06T18:37:52.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:52.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:52.781964+0000) 2024-03-06T18:37:53.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:53.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:53.782162+0000) 2024-03-06T18:37:54.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:54.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:54.782401+0000) 2024-03-06T18:37:55.777 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:55.780+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:55.782632+0000) 2024-03-06T18:37:56.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:56.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:56.782830+0000) 2024-03-06T18:37:57.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:57.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:57.783052+0000) 2024-03-06T18:37:58.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:58.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:58.783288+0000) 2024-03-06T18:37:59.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:37:59.781+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:59.783515+0000) 2024-03-06T18:38:00.778 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:00.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:00.783765+0000) 2024-03-06T18:38:01.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:01.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:01.783995+0000) 2024-03-06T18:38:02.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:02.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:02.784237+0000) 2024-03-06T18:38:03.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:03.782+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:03.784483+0000) 2024-03-06T18:38:04.779 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:04.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:04.784765+0000) 2024-03-06T18:38:05.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:05.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:05.784982+0000) 2024-03-06T18:38:06.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:06.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:06.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:06.783+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:06.785174+0000) 2024-03-06T18:38:07.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:07.784+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:07.785416+0000) 2024-03-06T18:38:08.780 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:08.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:08.785654+0000) 2024-03-06T18:38:09.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:09.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:09.785894+0000) 2024-03-06T18:38:10.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:10.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:10.786124+0000) 2024-03-06T18:38:11.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:11.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:11.786268+0000) 2024-03-06T18:38:12.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:12.785+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:12.786444+0000) 2024-03-06T18:38:13.781 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:13.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:13.786748+0000) 2024-03-06T18:38:14.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:14.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:14.786939+0000) 2024-03-06T18:38:15.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:15.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:15.787130+0000) 2024-03-06T18:38:16.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:16.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:16.787327+0000) 2024-03-06T18:38:17.782 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:17.786+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:17.787559+0000) 2024-03-06T18:38:18.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:18.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:18.787805+0000) 2024-03-06T18:38:19.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:19.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:19.788001+0000) 2024-03-06T18:38:20.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:20.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:20.788189+0000) 2024-03-06T18:38:21.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:21.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:21.788377+0000) 2024-03-06T18:38:22.783 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:22.787+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:22.788534+0000) 2024-03-06T18:38:23.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:23.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:23.788786+0000) 2024-03-06T18:38:24.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:24.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:24.789042+0000) 2024-03-06T18:38:25.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:25.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:25.789281+0000) 2024-03-06T18:38:26.784 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:26.788+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:26.789507+0000) 2024-03-06T18:38:27.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:27.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:27.789776+0000) 2024-03-06T18:38:28.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:28.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:28.790012+0000) 2024-03-06T18:38:29.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:29.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:29.790240+0000) 2024-03-06T18:38:30.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:30.789+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:30.790462+0000) 2024-03-06T18:38:31.785 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:31.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:31.790718+0000) 2024-03-06T18:38:32.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:32.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:32.791001+0000) 2024-03-06T18:38:33.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:33.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:33.791246+0000) 2024-03-06T18:38:34.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:34.790+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:34.791491+0000) 2024-03-06T18:38:35.786 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:35.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:35.791797+0000) 2024-03-06T18:38:36.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:36.438 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:36.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:36.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:36.792010+0000) 2024-03-06T18:38:37.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:37.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:37.792234+0000) 2024-03-06T18:38:38.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:38.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:38.792416+0000) 2024-03-06T18:38:39.787 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:39.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:39.792570+0000) 2024-03-06T18:38:40.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:40.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:40.792808+0000) 2024-03-06T18:38:41.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:41.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:41.793015+0000) 2024-03-06T18:38:42.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:42.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:42.793275+0000) 2024-03-06T18:38:43.788 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:43.791+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:43.793519+0000) 2024-03-06T18:38:44.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:44.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:44.793761+0000) 2024-03-06T18:38:45.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:45.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:45.793983+0000) 2024-03-06T18:38:46.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:46.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:46.794177+0000) 2024-03-06T18:38:47.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:47.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:47.794385+0000) 2024-03-06T18:38:48.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:48.792+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:48.794625+0000) 2024-03-06T18:38:49.789 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:49.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:49.794819+0000) 2024-03-06T18:38:50.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:50.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:50.795016+0000) 2024-03-06T18:38:51.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:51.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:51.795205+0000) 2024-03-06T18:38:52.790 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:52.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:52.795404+0000) 2024-03-06T18:38:53.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:53.793+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:53.795643+0000) 2024-03-06T18:38:54.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:54.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:54.795872+0000) 2024-03-06T18:38:55.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:55.794+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:55.796058+0000) 2024-03-06T18:38:56.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:56.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:56.796265+0000) 2024-03-06T18:38:57.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:57.795+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:57.796487+0000) 2024-03-06T18:38:58.791 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:58.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:58.796759+0000) 2024-03-06T18:38:59.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:38:59.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:59.796958+0000) 2024-03-06T18:39:00.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:00.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:00.797146+0000) 2024-03-06T18:39:01.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:01.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:01.797337+0000) 2024-03-06T18:39:02.792 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:02.796+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:02.797590+0000) 2024-03-06T18:39:03.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:03.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:03.797830+0000) 2024-03-06T18:39:04.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:04.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:04.798037+0000) 2024-03-06T18:39:05.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:05.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:05.798223+0000) 2024-03-06T18:39:06.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:06.631 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:06.793 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:06.797+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:06.798381+0000) 2024-03-06T18:39:07.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:07.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:07.798734+0000) 2024-03-06T18:39:08.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:08.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:08.798929+0000) 2024-03-06T18:39:09.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:09.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:09.799132+0000) 2024-03-06T18:39:10.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:10.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:10.799354+0000) 2024-03-06T18:39:11.794 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:11.798+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:11.799577+0000) 2024-03-06T18:39:12.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:12.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:12.799835+0000) 2024-03-06T18:39:13.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:13.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:13.800055+0000) 2024-03-06T18:39:14.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:14.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:14.800251+0000) 2024-03-06T18:39:15.795 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:15.799+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:15.800483+0000) 2024-03-06T18:39:16.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:16.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:16.800704+0000) 2024-03-06T18:39:17.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:17.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:17.800908+0000) 2024-03-06T18:39:18.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:18.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:18.801120+0000) 2024-03-06T18:39:19.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:19.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:19.801364+0000) 2024-03-06T18:39:20.796 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:20.800+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:20.801587+0000) 2024-03-06T18:39:21.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:21.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:21.801831+0000) 2024-03-06T18:39:22.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:22.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:22.802030+0000) 2024-03-06T18:39:23.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:23.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:23.802243+0000) 2024-03-06T18:39:24.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:24.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:24.802478+0000) 2024-03-06T18:39:25.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:25.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:25.802624+0000) 2024-03-06T18:39:26.797 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:26.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:26.802766+0000) 2024-03-06T18:39:27.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:27.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:27.802983+0000) 2024-03-06T18:39:28.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:28.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:28.803254+0000) 2024-03-06T18:39:29.798 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:29.801+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:29.803465+0000) 2024-03-06T18:39:30.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:30.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:30.803730+0000) 2024-03-06T18:39:31.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:31.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:31.803918+0000) 2024-03-06T18:39:32.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:32.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:32.804117+0000) 2024-03-06T18:39:33.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:33.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:33.804344+0000) 2024-03-06T18:39:34.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:34.802+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:34.804573+0000) 2024-03-06T18:39:35.799 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:35.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:35.804812+0000) 2024-03-06T18:39:36.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:36.803 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:36.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:36.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:36.804917+0000) 2024-03-06T18:39:37.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:37.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:37.805142+0000) 2024-03-06T18:39:38.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:38.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:38.805379+0000) 2024-03-06T18:39:39.800 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:39.803+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:39.805613+0000) 2024-03-06T18:39:40.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:40.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:40.805845+0000) 2024-03-06T18:39:41.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:41.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:41.806001+0000) 2024-03-06T18:39:42.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:42.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:42.806234+0000) 2024-03-06T18:39:43.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:43.804+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:43.806454+0000) 2024-03-06T18:39:44.801 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:44.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:44.806710+0000) 2024-03-06T18:39:45.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:45.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:45.806906+0000) 2024-03-06T18:39:46.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:46.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:46.807096+0000) 2024-03-06T18:39:47.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:47.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:47.807286+0000) 2024-03-06T18:39:48.802 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:48.806+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:48.807509+0000) 2024-03-06T18:39:49.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:49.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:49.807762+0000) 2024-03-06T18:39:50.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:50.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:50.807960+0000) 2024-03-06T18:39:51.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:51.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:51.808158+0000) 2024-03-06T18:39:52.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:52.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:52.808378+0000) 2024-03-06T18:39:53.803 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:53.807+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:53.808611+0000) 2024-03-06T18:39:54.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:54.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:54.808833+0000) 2024-03-06T18:39:55.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:55.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:55.809092+0000) 2024-03-06T18:39:56.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:56.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:56.809278+0000) 2024-03-06T18:39:57.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:57.808+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:57.809502+0000) 2024-03-06T18:39:58.804 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:58.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:58.809722+0000) 2024-03-06T18:39:59.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:39:59.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:59.809930+0000) 2024-03-06T18:40:00.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:00.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:00.810133+0000) 2024-03-06T18:40:01.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:01.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:01.810351+0000) 2024-03-06T18:40:02.805 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:02.809+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:02.810594+0000) 2024-03-06T18:40:03.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:03.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:03.810823+0000) 2024-03-06T18:40:04.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:04.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:04.811048+0000) 2024-03-06T18:40:05.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:05.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:05.811263+0000) 2024-03-06T18:40:06.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:06.810+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:06.811447+0000) 2024-03-06T18:40:06.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:06.942 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:07.806 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:07.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:07.811715+0000) 2024-03-06T18:40:08.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:08.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:08.811926+0000) 2024-03-06T18:40:09.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:09.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:09.812141+0000) 2024-03-06T18:40:10.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:10.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:10.812383+0000) 2024-03-06T18:40:11.807 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:11.811+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:11.812633+0000) 2024-03-06T18:40:12.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:12.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:12.812890+0000) 2024-03-06T18:40:13.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:13.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:13.813095+0000) 2024-03-06T18:40:14.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:14.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:14.813338+0000) 2024-03-06T18:40:15.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:15.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:15.813575+0000) 2024-03-06T18:40:16.808 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:16.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:16.813806+0000) 2024-03-06T18:40:17.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:17.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:17.814014+0000) 2024-03-06T18:40:18.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:18.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:18.814255+0000) 2024-03-06T18:40:19.809 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:19.812+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:19.814503+0000) 2024-03-06T18:40:20.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:20.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:20.814762+0000) 2024-03-06T18:40:21.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:21.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:21.814950+0000) 2024-03-06T18:40:22.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:22.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:22.815150+0000) 2024-03-06T18:40:23.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:23.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:23.815385+0000) 2024-03-06T18:40:24.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:24.813+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:24.815620+0000) 2024-03-06T18:40:25.810 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:25.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:25.815826+0000) 2024-03-06T18:40:26.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:26.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:26.816028+0000) 2024-03-06T18:40:27.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:27.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:27.816218+0000) 2024-03-06T18:40:28.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:28.814+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:28.816443+0000) 2024-03-06T18:40:29.811 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:29.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:29.816695+0000) 2024-03-06T18:40:30.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:30.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:30.816896+0000) 2024-03-06T18:40:31.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:31.815+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:31.817126+0000) 2024-03-06T18:40:32.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:32.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:32.817376+0000) 2024-03-06T18:40:33.812 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:33.816+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:33.817602+0000) 2024-03-06T18:40:34.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:34.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:34.817832+0000) 2024-03-06T18:40:35.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:35.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:35.818024+0000) 2024-03-06T18:40:36.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:36.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:36.818219+0000) 2024-03-06T18:40:37.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:37.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:37.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:37.817+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:37.818462+0000) 2024-03-06T18:40:38.813 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:38.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:38.818724+0000) 2024-03-06T18:40:39.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:39.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:39.818936+0000) 2024-03-06T18:40:40.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:40.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:40.819139+0000) 2024-03-06T18:40:41.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:41.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:41.819368+0000) 2024-03-06T18:40:42.814 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:42.818+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:42.819608+0000) 2024-03-06T18:40:43.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:43.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:43.819895+0000) 2024-03-06T18:40:44.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:44.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:44.820120+0000) 2024-03-06T18:40:45.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:45.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:45.820345+0000) 2024-03-06T18:40:46.815 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:46.819+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:46.820570+0000) 2024-03-06T18:40:47.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:47.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:47.820799+0000) 2024-03-06T18:40:48.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:48.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:48.821055+0000) 2024-03-06T18:40:49.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:49.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:49.821240+0000) 2024-03-06T18:40:50.816 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:50.820+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:50.821471+0000) 2024-03-06T18:40:51.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:51.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:51.821659+0000) 2024-03-06T18:40:52.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:52.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:52.821859+0000) 2024-03-06T18:40:53.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:53.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:53.822013+0000) 2024-03-06T18:40:54.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:54.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:54.822238+0000) 2024-03-06T18:40:55.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:55.821+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:55.822476+0000) 2024-03-06T18:40:56.817 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:56.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:56.822739+0000) 2024-03-06T18:40:57.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:57.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:57.822929+0000) 2024-03-06T18:40:58.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:58.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:58.823161+0000) 2024-03-06T18:40:59.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:40:59.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:59.823390+0000) 2024-03-06T18:41:00.818 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:00.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:00.823619+0000) 2024-03-06T18:41:01.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:01.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:01.823840+0000) 2024-03-06T18:41:02.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:02.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:02.824033+0000) 2024-03-06T18:41:03.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:03.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:03.824265+0000) 2024-03-06T18:41:04.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:04.822+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:04.824493+0000) 2024-03-06T18:41:05.819 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:05.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:05.824752+0000) 2024-03-06T18:41:06.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:06.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:06.824947+0000) 2024-03-06T18:41:07.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:07.301 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:07.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:07.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:07.825141+0000) 2024-03-06T18:41:08.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:08.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:08.825336+0000) 2024-03-06T18:41:09.820 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:09.823+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:09.825579+0000) 2024-03-06T18:41:10.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:10.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:10.825812+0000) 2024-03-06T18:41:11.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:11.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:11.826045+0000) 2024-03-06T18:41:12.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:12.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:12.826245+0000) 2024-03-06T18:41:13.821 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:13.824+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:13.826484+0000) 2024-03-06T18:41:14.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:14.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:14.826707+0000) 2024-03-06T18:41:15.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:15.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:15.826953+0000) 2024-03-06T18:41:16.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:16.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:16.827193+0000) 2024-03-06T18:41:17.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:17.825+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:17.827423+0000) 2024-03-06T18:41:18.822 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:18.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:18.827701+0000) 2024-03-06T18:41:19.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:19.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:19.827947+0000) 2024-03-06T18:41:20.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:20.826+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:20.828167+0000) 2024-03-06T18:41:21.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:21.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:21.828396+0000) 2024-03-06T18:41:22.823 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:22.827+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:22.828631+0000) 2024-03-06T18:41:23.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:23.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:23.828905+0000) 2024-03-06T18:41:24.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:24.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:24.829139+0000) 2024-03-06T18:41:25.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:25.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:25.829367+0000) 2024-03-06T18:41:26.824 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:26.828+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:26.829598+0000) 2024-03-06T18:41:27.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:27.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:27.829825+0000) 2024-03-06T18:41:28.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:28.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:28.830037+0000) 2024-03-06T18:41:29.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:29.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:29.830268+0000) 2024-03-06T18:41:30.825 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:30.829+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:30.830508+0000) 2024-03-06T18:41:31.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:31.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:31.830807+0000) 2024-03-06T18:41:32.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:32.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:32.831036+0000) 2024-03-06T18:41:33.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:33.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:33.831282+0000) 2024-03-06T18:41:34.826 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:34.830+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:34.831570+0000) 2024-03-06T18:41:35.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:35.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:35.831723+0000) 2024-03-06T18:41:36.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:36.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:36.831991+0000) 2024-03-06T18:41:37.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:37.496 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:37.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:37.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:37.832215+0000) 2024-03-06T18:41:38.827 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:38.831+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:38.832484+0000) 2024-03-06T18:41:39.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:39.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:39.832821+0000) 2024-03-06T18:41:40.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:40.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:40.833016+0000) 2024-03-06T18:41:41.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:41.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:41.833250+0000) 2024-03-06T18:41:42.828 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:42.832+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:42.833475+0000) 2024-03-06T18:41:43.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:43.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:43.833790+0000) 2024-03-06T18:41:44.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:44.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:44.834048+0000) 2024-03-06T18:41:45.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:45.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:45.834295+0000) 2024-03-06T18:41:46.829 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:46.833+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:46.834576+0000) 2024-03-06T18:41:47.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:47.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:47.834846+0000) 2024-03-06T18:41:48.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:48.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:48.835092+0000) 2024-03-06T18:41:49.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:49.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:49.835340+0000) 2024-03-06T18:41:50.830 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:50.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:50.835620+0000) 2024-03-06T18:41:51.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:51.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:51.835810+0000) 2024-03-06T18:41:52.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:52.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:52.836049+0000) 2024-03-06T18:41:53.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:53.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:53.836232+0000) 2024-03-06T18:41:54.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:54.834+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:54.836522+0000) 2024-03-06T18:41:55.831 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:55.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:55.836807+0000) 2024-03-06T18:41:56.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:56.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:56.837053+0000) 2024-03-06T18:41:57.832 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:57.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:57.837297+0000) 2024-03-06T18:41:58.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:58.835+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:58.837583+0000) 2024-03-06T18:41:59.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:41:59.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:59.837879+0000) 2024-03-06T18:42:00.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:00.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:00.838127+0000) 2024-03-06T18:42:01.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:01.836+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:01.838406+0000) 2024-03-06T18:42:02.833 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:02.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:02.838751+0000) 2024-03-06T18:42:03.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:03.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:03.839109+0000) 2024-03-06T18:42:04.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:04.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:04.839369+0000) 2024-03-06T18:42:05.834 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:05.837+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:05.839602+0000) 2024-03-06T18:42:06.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:06.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:06.839826+0000) 2024-03-06T18:42:07.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:07.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:07.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:07.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:07.840066+0000) 2024-03-06T18:42:08.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:08.838+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:08.840267+0000) 2024-03-06T18:42:09.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:09.839+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:09.840475+0000) 2024-03-06T18:42:10.835 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:10.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:10.840748+0000) 2024-03-06T18:42:11.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:11.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:11.840938+0000) 2024-03-06T18:42:12.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:12.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:12.841136+0000) 2024-03-06T18:42:13.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:13.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:13.841344+0000) 2024-03-06T18:42:14.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:14.840+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:14.841631+0000) 2024-03-06T18:42:15.836 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:15.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:15.841806+0000) 2024-03-06T18:42:16.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:16.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:16.842030+0000) 2024-03-06T18:42:17.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:17.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:17.842227+0000) 2024-03-06T18:42:18.837 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:18.841+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:18.842471+0000) 2024-03-06T18:42:19.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:19.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:19.842721+0000) 2024-03-06T18:42:20.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:20.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:20.842903+0000) 2024-03-06T18:42:21.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:21.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:21.843090+0000) 2024-03-06T18:42:22.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:22.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:22.843293+0000) 2024-03-06T18:42:23.838 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:23.842+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:23.843514+0000) 2024-03-06T18:42:24.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:24.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:24.843829+0000) 2024-03-06T18:42:25.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:25.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:25.844009+0000) 2024-03-06T18:42:26.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:26.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:26.844278+0000) 2024-03-06T18:42:27.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:27.843+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:27.844535+0000) 2024-03-06T18:42:28.839 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:28.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:28.844801+0000) 2024-03-06T18:42:29.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:29.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:29.845039+0000) 2024-03-06T18:42:30.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:30.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:30.845249+0000) 2024-03-06T18:42:31.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:31.844+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:31.845525+0000) 2024-03-06T18:42:32.840 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:32.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:32.845803+0000) 2024-03-06T18:42:33.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:33.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:33.846080+0000) 2024-03-06T18:42:34.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:34.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:34.846336+0000) 2024-03-06T18:42:35.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:35.845+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:35.846520+0000) 2024-03-06T18:42:36.841 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:36.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:36.846786+0000) 2024-03-06T18:42:37.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:37.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:37.846935+0000) 2024-03-06T18:42:37.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:37.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:38.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:38.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:38.847127+0000) 2024-03-06T18:42:39.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:39.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:39.847326+0000) 2024-03-06T18:42:40.842 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:40.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:40.847565+0000) 2024-03-06T18:42:41.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:41.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:41.847805+0000) 2024-03-06T18:42:42.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:42.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:42.848104+0000) 2024-03-06T18:42:43.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:43.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:43.848330+0000) 2024-03-06T18:42:44.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:44.846+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:44.848510+0000) 2024-03-06T18:42:45.843 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:45.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:45.848729+0000) 2024-03-06T18:42:46.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:46.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:46.848922+0000) 2024-03-06T18:42:47.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:47.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:47.849106+0000) 2024-03-06T18:42:48.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:48.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:48.849295+0000) 2024-03-06T18:42:49.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:49.847+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:49.849525+0000) 2024-03-06T18:42:50.844 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:50.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:50.849768+0000) 2024-03-06T18:42:51.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:51.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:51.849971+0000) 2024-03-06T18:42:52.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:52.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:52.850171+0000) 2024-03-06T18:42:53.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:53.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:53.850397+0000) 2024-03-06T18:42:54.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:54.848+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:54.850573+0000) 2024-03-06T18:42:55.845 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:55.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:55.850787+0000) 2024-03-06T18:42:56.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:56.849+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:56.851009+0000) 2024-03-06T18:42:57.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:57.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:57.851200+0000) 2024-03-06T18:42:58.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:58.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:58.851424+0000) 2024-03-06T18:42:59.846 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:42:59.850+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:59.851630+0000) 2024-03-06T18:43:00.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:00.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:00.851941+0000) 2024-03-06T18:43:01.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:01.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:01.852170+0000) 2024-03-06T18:43:02.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:02.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:02.852408+0000) 2024-03-06T18:43:03.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:03.851+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:03.852628+0000) 2024-03-06T18:43:04.847 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:04.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:04.852831+0000) 2024-03-06T18:43:05.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:05.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:05.853059+0000) 2024-03-06T18:43:06.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:06.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:06.853280+0000) 2024-03-06T18:43:07.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:07.852+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:07.853468+0000) 2024-03-06T18:43:08.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:08.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:08.848 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:08.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:08.853762+0000) 2024-03-06T18:43:09.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:09.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:09.853978+0000) 2024-03-06T18:43:10.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:10.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:10.854187+0000) 2024-03-06T18:43:11.849 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:11.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:11.854381+0000) 2024-03-06T18:43:12.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:12.853+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:12.854618+0000) 2024-03-06T18:43:13.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:13.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:13.854837+0000) 2024-03-06T18:43:14.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:14.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:14.855069+0000) 2024-03-06T18:43:15.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:15.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:15.855299+0000) 2024-03-06T18:43:16.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:16.854+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:16.855491+0000) 2024-03-06T18:43:17.850 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:17.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:17.855694+0000) 2024-03-06T18:43:18.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:18.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:18.855902+0000) 2024-03-06T18:43:19.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:19.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:19.856122+0000) 2024-03-06T18:43:20.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:20.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:20.856352+0000) 2024-03-06T18:43:21.851 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:21.855+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:21.856534+0000) 2024-03-06T18:43:22.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:22.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:22.856802+0000) 2024-03-06T18:43:23.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:23.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:23.856994+0000) 2024-03-06T18:43:24.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:24.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:24.857202+0000) 2024-03-06T18:43:25.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:25.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:25.857417+0000) 2024-03-06T18:43:26.852 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:26.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:26.857597+0000) 2024-03-06T18:43:27.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:27.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:27.857831+0000) 2024-03-06T18:43:28.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:28.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:28.858024+0000) 2024-03-06T18:43:29.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:29.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:29.858225+0000) 2024-03-06T18:43:30.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:30.856+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:30.858432+0000) 2024-03-06T18:43:31.853 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:31.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:31.858693+0000) 2024-03-06T18:43:32.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:32.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:32.858875+0000) 2024-03-06T18:43:33.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:33.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:33.859063+0000) 2024-03-06T18:43:34.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:34.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:34.859264+0000) 2024-03-06T18:43:35.854 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:35.857+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:35.859490+0000) 2024-03-06T18:43:36.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:36.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:36.859773+0000) 2024-03-06T18:43:37.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:37.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:37.859961+0000) 2024-03-06T18:43:38.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:38.265 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:38.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:38.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:38.860159+0000) 2024-03-06T18:43:39.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:39.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:39.860395+0000) 2024-03-06T18:43:40.855 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:40.858+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:40.860598+0000) 2024-03-06T18:43:41.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:41.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:41.860807+0000) 2024-03-06T18:43:42.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:42.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:42.861022+0000) 2024-03-06T18:43:43.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:43.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:43.861220+0000) 2024-03-06T18:43:44.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:44.859+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:44.861455+0000) 2024-03-06T18:43:45.856 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:45.860+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:45.861715+0000) 2024-03-06T18:43:46.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:46.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:46.861907+0000) 2024-03-06T18:43:47.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:47.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:47.862123+0000) 2024-03-06T18:43:48.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:48.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:48.862403+0000) 2024-03-06T18:43:49.857 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:49.861+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:49.862604+0000) 2024-03-06T18:43:50.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:50.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:50.862787+0000) 2024-03-06T18:43:51.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:51.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:51.862975+0000) 2024-03-06T18:43:52.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:52.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:52.863168+0000) 2024-03-06T18:43:53.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:53.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:53.863355+0000) 2024-03-06T18:43:54.858 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:54.862+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:54.863589+0000) 2024-03-06T18:43:55.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:55.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:55.863785+0000) 2024-03-06T18:43:56.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:56.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:56.863961+0000) 2024-03-06T18:43:57.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:57.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:57.864142+0000) 2024-03-06T18:43:58.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:58.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:58.864357+0000) 2024-03-06T18:43:59.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:43:59.863+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:59.864585+0000) 2024-03-06T18:44:00.859 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:00.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:00.864821+0000) 2024-03-06T18:44:01.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:01.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:01.865033+0000) 2024-03-06T18:44:02.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:02.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:02.865228+0000) 2024-03-06T18:44:03.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:03.864+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:03.865452+0000) 2024-03-06T18:44:04.860 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:04.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:04.865695+0000) 2024-03-06T18:44:05.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:05.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:05.865941+0000) 2024-03-06T18:44:06.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:06.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:06.866143+0000) 2024-03-06T18:44:07.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:07.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:07.866375+0000) 2024-03-06T18:44:08.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:08.402 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:08.861 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:08.865+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:08.866567+0000) 2024-03-06T18:44:09.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:09.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:09.866836+0000) 2024-03-06T18:44:10.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:10.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:10.867026+0000) 2024-03-06T18:44:11.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:11.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:11.867221+0000) 2024-03-06T18:44:12.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:12.866+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:12.867408+0000) 2024-03-06T18:44:13.862 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:13.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:13.867719+0000) 2024-03-06T18:44:14.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:14.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:14.867974+0000) 2024-03-06T18:44:15.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:15.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:15.868244+0000) 2024-03-06T18:44:16.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:16.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:16.868446+0000) 2024-03-06T18:44:17.863 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:17.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:17.868699+0000) 2024-03-06T18:44:18.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:18.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:18.868943+0000) 2024-03-06T18:44:19.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:19.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:19.869138+0000) 2024-03-06T18:44:20.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:20.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:20.869377+0000) 2024-03-06T18:44:21.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:21.867+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:21.869572+0000) 2024-03-06T18:44:22.864 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:22.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:22.869779+0000) 2024-03-06T18:44:23.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:23.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:23.869988+0000) 2024-03-06T18:44:24.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:24.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:24.870192+0000) 2024-03-06T18:44:25.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:25.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:25.870418+0000) 2024-03-06T18:44:26.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:26.868+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:26.870643+0000) 2024-03-06T18:44:27.865 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:27.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:27.870875+0000) 2024-03-06T18:44:28.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:28.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:28.871067+0000) 2024-03-06T18:44:29.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:29.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:29.871263+0000) 2024-03-06T18:44:30.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:30.869+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:30.871482+0000) 2024-03-06T18:44:31.866 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:31.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:31.871745+0000) 2024-03-06T18:44:32.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:32.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:32.871951+0000) 2024-03-06T18:44:33.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:33.870+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:33.872138+0000) 2024-03-06T18:44:34.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:34.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:34.872345+0000) 2024-03-06T18:44:35.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:35.871+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:35.872582+0000) 2024-03-06T18:44:36.867 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:36.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:36.872827+0000) 2024-03-06T18:44:37.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:37.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:37.873028+0000) 2024-03-06T18:44:38.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:38.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:38.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:38.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:38.873240+0000) 2024-03-06T18:44:39.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:39.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:39.873425+0000) 2024-03-06T18:44:40.868 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:40.872+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:40.873647+0000) 2024-03-06T18:44:41.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:41.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:41.873949+0000) 2024-03-06T18:44:42.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:42.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:42.874183+0000) 2024-03-06T18:44:43.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:43.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:43.874375+0000) 2024-03-06T18:44:44.869 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:44.873+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:44.874582+0000) 2024-03-06T18:44:45.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:45.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:45.874733+0000) 2024-03-06T18:44:46.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:46.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:46.874961+0000) 2024-03-06T18:44:47.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:47.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:47.875143+0000) 2024-03-06T18:44:48.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:48.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:48.875375+0000) 2024-03-06T18:44:49.870 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:49.874+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:49.875632+0000) 2024-03-06T18:44:50.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:50.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:50.875844+0000) 2024-03-06T18:44:51.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:51.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:51.876000+0000) 2024-03-06T18:44:52.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:52.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:52.876200+0000) 2024-03-06T18:44:53.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:53.875+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:53.876428+0000) 2024-03-06T18:44:54.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:54.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:54.876661+0000) 2024-03-06T18:44:55.871 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:55.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:55.876883+0000) 2024-03-06T18:44:56.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:56.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:56.877098+0000) 2024-03-06T18:44:57.872 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:57.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:57.877290+0000) 2024-03-06T18:44:58.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:58.876+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:58.877514+0000) 2024-03-06T18:44:59.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:44:59.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:59.877843+0000) 2024-03-06T18:45:00.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:00.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:00.878080+0000) 2024-03-06T18:45:01.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:01.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:01.878314+0000) 2024-03-06T18:45:02.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:02.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:02.878564+0000) 2024-03-06T18:45:03.873 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:03.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:03.878805+0000) 2024-03-06T18:45:04.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:04.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:04.879016+0000) 2024-03-06T18:45:05.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:05.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:05.879211+0000) 2024-03-06T18:45:06.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:06.877+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:06.879439+0000) 2024-03-06T18:45:07.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:07.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:07.879701+0000) 2024-03-06T18:45:08.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:08.741 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:08.874 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:08.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:08.879913+0000) 2024-03-06T18:45:09.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:09.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:09.880112+0000) 2024-03-06T18:45:10.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:10.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:10.880320+0000) 2024-03-06T18:45:11.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:11.878+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:11.880563+0000) 2024-03-06T18:45:12.875 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:12.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:12.880836+0000) 2024-03-06T18:45:13.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:13.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:13.881029+0000) 2024-03-06T18:45:14.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:14.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:14.881212+0000) 2024-03-06T18:45:15.876 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:15.879+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:15.881446+0000) 2024-03-06T18:45:16.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:16.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:16.881715+0000) 2024-03-06T18:45:17.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:17.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:17.881914+0000) 2024-03-06T18:45:18.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:18.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:18.882109+0000) 2024-03-06T18:45:19.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:19.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:19.882299+0000) 2024-03-06T18:45:20.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:20.880+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:20.882504+0000) 2024-03-06T18:45:21.877 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:21.881+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:21.882788+0000) 2024-03-06T18:45:22.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:22.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:22.882995+0000) 2024-03-06T18:45:23.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:23.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:23.883227+0000) 2024-03-06T18:45:24.878 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:24.882+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:24.883453+0000) 2024-03-06T18:45:25.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:25.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:25.883716+0000) 2024-03-06T18:45:26.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:26.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:26.883911+0000) 2024-03-06T18:45:27.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:27.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:27.884103+0000) 2024-03-06T18:45:28.879 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:28.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:28.884290+0000) 2024-03-06T18:45:29.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:29.883+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:29.884529+0000) 2024-03-06T18:45:30.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:30.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:30.884797+0000) 2024-03-06T18:45:31.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:31.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:31.884992+0000) 2024-03-06T18:45:32.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:32.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:32.885186+0000) 2024-03-06T18:45:33.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:33.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:33.885380+0000) 2024-03-06T18:45:34.880 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:34.884+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:34.885623+0000) 2024-03-06T18:45:35.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:35.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:35.885917+0000) 2024-03-06T18:45:36.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:36.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:36.886168+0000) 2024-03-06T18:45:37.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:37.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:37.886416+0000) 2024-03-06T18:45:38.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:38.880 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:38.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:38.885+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:38.886627+0000) 2024-03-06T18:45:39.881 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:39.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:39.886862+0000) 2024-03-06T18:45:40.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:40.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:40.887062+0000) 2024-03-06T18:45:41.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:41.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:41.887266+0000) 2024-03-06T18:45:42.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:42.886+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:42.887467+0000) 2024-03-06T18:45:43.882 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:43.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:43.887754+0000) 2024-03-06T18:45:44.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:44.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:44.887951+0000) 2024-03-06T18:45:45.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:45.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:45.888145+0000) 2024-03-06T18:45:46.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:46.887+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:46.888372+0000) 2024-03-06T18:45:47.883 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:47.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:47.888723+0000) 2024-03-06T18:45:48.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:48.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:48.888922+0000) 2024-03-06T18:45:49.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:49.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:49.889102+0000) 2024-03-06T18:45:50.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:50.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:50.889287+0000) 2024-03-06T18:45:51.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:51.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:51.889511+0000) 2024-03-06T18:45:52.884 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:52.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:52.889790+0000) 2024-03-06T18:45:53.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:53.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:53.889983+0000) 2024-03-06T18:45:54.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:54.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:54.890211+0000) 2024-03-06T18:45:55.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:55.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:55.890430+0000) 2024-03-06T18:45:56.885 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:56.888+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:56.890649+0000) 2024-03-06T18:45:57.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:57.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:57.890859+0000) 2024-03-06T18:45:58.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:58.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:58.891004+0000) 2024-03-06T18:45:59.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:45:59.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:59.891194+0000) 2024-03-06T18:46:00.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:00.889+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:00.891436+0000) 2024-03-06T18:46:01.886 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:01.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:01.891696+0000) 2024-03-06T18:46:02.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:02.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:02.891926+0000) 2024-03-06T18:46:03.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:03.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:03.892143+0000) 2024-03-06T18:46:04.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:04.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:04.892347+0000) 2024-03-06T18:46:05.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:05.890+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:05.892545+0000) 2024-03-06T18:46:06.887 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:06.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:06.892787+0000) 2024-03-06T18:46:07.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:07.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:07.892971+0000) 2024-03-06T18:46:08.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:08.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:08.893159+0000) 2024-03-06T18:46:09.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:09.096 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:09.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:09.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:09.893339+0000) 2024-03-06T18:46:10.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:10.891+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:10.893552+0000) 2024-03-06T18:46:11.888 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:11.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:11.893785+0000) 2024-03-06T18:46:12.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:12.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:12.893981+0000) 2024-03-06T18:46:13.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:13.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:13.894168+0000) 2024-03-06T18:46:14.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:14.893+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:14.894402+0000) 2024-03-06T18:46:15.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:15.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:15.894652+0000) 2024-03-06T18:46:16.889 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:16.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:16.894843+0000) 2024-03-06T18:46:17.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:17.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:17.895059+0000) 2024-03-06T18:46:18.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:18.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:18.895286+0000) 2024-03-06T18:46:19.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:19.894+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:19.895518+0000) 2024-03-06T18:46:20.890 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:20.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:20.895763+0000) 2024-03-06T18:46:21.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:21.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:21.895951+0000) 2024-03-06T18:46:22.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:22.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:22.896193+0000) 2024-03-06T18:46:23.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:23.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:23.896397+0000) 2024-03-06T18:46:24.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:24.895+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:24.896624+0000) 2024-03-06T18:46:25.891 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:25.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:25.896835+0000) 2024-03-06T18:46:26.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:26.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:26.897063+0000) 2024-03-06T18:46:27.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:27.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:27.897281+0000) 2024-03-06T18:46:28.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:28.896+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:28.897517+0000) 2024-03-06T18:46:29.892 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:29.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:29.897796+0000) 2024-03-06T18:46:30.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:30.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:30.898044+0000) 2024-03-06T18:46:31.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:31.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:31.898263+0000) 2024-03-06T18:46:32.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:32.897+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:32.898515+0000) 2024-03-06T18:46:33.893 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:33.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:33.898765+0000) 2024-03-06T18:46:34.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:34.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:34.898951+0000) 2024-03-06T18:46:35.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:35.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:35.899207+0000) 2024-03-06T18:46:36.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:36.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:36.899438+0000) 2024-03-06T18:46:37.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:37.898+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:37.899630+0000) 2024-03-06T18:46:38.894 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:38.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:38.899848+0000) 2024-03-06T18:46:39.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:39.322 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:39.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:39.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:39.900035+0000) 2024-03-06T18:46:40.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:40.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:40.900235+0000) 2024-03-06T18:46:41.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:41.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:41.900462+0000) 2024-03-06T18:46:42.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:42.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:42.900724+0000) 2024-03-06T18:46:43.895 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:43.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:43.900954+0000) 2024-03-06T18:46:44.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:44.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:44.901146+0000) 2024-03-06T18:46:45.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:45.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:45.901387+0000) 2024-03-06T18:46:46.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:46.899+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:46.901620+0000) 2024-03-06T18:46:47.896 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:47.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:47.901822+0000) 2024-03-06T18:46:48.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:48.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:48.902049+0000) 2024-03-06T18:46:49.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:49.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:49.902272+0000) 2024-03-06T18:46:50.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:50.900+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:50.902504+0000) 2024-03-06T18:46:51.897 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:51.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:51.902744+0000) 2024-03-06T18:46:52.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:52.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:52.902992+0000) 2024-03-06T18:46:53.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:53.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:53.903189+0000) 2024-03-06T18:46:54.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:54.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:54.903422+0000) 2024-03-06T18:46:55.898 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:55.901+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:55.903618+0000) 2024-03-06T18:46:56.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:56.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:56.903806+0000) 2024-03-06T18:46:57.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:57.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:57.904051+0000) 2024-03-06T18:46:58.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:58.902+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:58.904291+0000) 2024-03-06T18:46:59.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:46:59.903+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:59.904485+0000) 2024-03-06T18:47:00.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:00.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:00.904715+0000) 2024-03-06T18:47:01.899 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:01.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:01.904966+0000) 2024-03-06T18:47:02.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:02.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:02.905245+0000) 2024-03-06T18:47:03.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:03.904+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:03.905491+0000) 2024-03-06T18:47:04.900 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:04.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:04.905758+0000) 2024-03-06T18:47:05.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:05.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:05.906018+0000) 2024-03-06T18:47:06.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:06.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:06.906236+0000) 2024-03-06T18:47:07.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:07.905+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:07.906484+0000) 2024-03-06T18:47:08.901 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:08.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:08.906794+0000) 2024-03-06T18:47:09.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:09.467 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:09.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:09.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:09.907053+0000) 2024-03-06T18:47:10.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:10.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:10.907300+0000) 2024-03-06T18:47:11.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:11.906+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:11.907540+0000) 2024-03-06T18:47:12.902 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:12.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:12.907801+0000) 2024-03-06T18:47:13.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:13.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:13.908010+0000) 2024-03-06T18:47:14.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:14.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:14.908266+0000) 2024-03-06T18:47:15.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:15.907+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:15.908471+0000) 2024-03-06T18:47:16.903 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:16.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:16.908723+0000) 2024-03-06T18:47:17.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:17.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:17.908967+0000) 2024-03-06T18:47:18.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:18.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:18.909186+0000) 2024-03-06T18:47:19.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:19.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:19.909413+0000) 2024-03-06T18:47:20.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:20.908+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:20.909609+0000) 2024-03-06T18:47:21.904 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:21.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:21.909812+0000) 2024-03-06T18:47:22.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:22.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:22.910044+0000) 2024-03-06T18:47:23.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:23.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:23.910265+0000) 2024-03-06T18:47:24.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:24.909+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:24.910493+0000) 2024-03-06T18:47:25.905 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:25.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:25.910777+0000) 2024-03-06T18:47:26.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:26.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:26.911003+0000) 2024-03-06T18:47:27.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:27.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:27.911182+0000) 2024-03-06T18:47:28.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:28.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:28.911458+0000) 2024-03-06T18:47:29.906 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:29.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:29.911718+0000) 2024-03-06T18:47:30.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:30.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:30.911959+0000) 2024-03-06T18:47:31.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:31.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:31.912189+0000) 2024-03-06T18:47:32.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:32.910+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:32.912426+0000) 2024-03-06T18:47:33.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:33.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:33.912655+0000) 2024-03-06T18:47:34.907 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:34.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:34.912886+0000) 2024-03-06T18:47:35.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:35.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:35.913087+0000) 2024-03-06T18:47:36.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:36.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:36.913311+0000) 2024-03-06T18:47:37.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:37.911+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:37.913533+0000) 2024-03-06T18:47:38.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:38.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:38.913779+0000) 2024-03-06T18:47:39.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:39.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:39.908 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:39.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:39.914040+0000) 2024-03-06T18:47:40.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:40.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:40.914286+0000) 2024-03-06T18:47:41.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:41.912+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:41.914520+0000) 2024-03-06T18:47:42.909 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:42.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:42.914786+0000) 2024-03-06T18:47:43.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:43.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:43.915013+0000) 2024-03-06T18:47:44.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:44.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:44.915242+0000) 2024-03-06T18:47:45.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:45.913+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:45.915484+0000) 2024-03-06T18:47:46.910 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:46.914+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:46.915764+0000) 2024-03-06T18:47:47.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:47.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:47.915994+0000) 2024-03-06T18:47:48.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:48.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:48.916198+0000) 2024-03-06T18:47:49.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:49.915+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:49.916424+0000) 2024-03-06T18:47:50.911 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:50.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:50.916694+0000) 2024-03-06T18:47:51.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:51.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:51.916926+0000) 2024-03-06T18:47:52.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:52.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:52.917156+0000) 2024-03-06T18:47:53.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:53.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:53.917380+0000) 2024-03-06T18:47:54.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:54.916+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:54.917603+0000) 2024-03-06T18:47:55.912 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:55.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:55.917857+0000) 2024-03-06T18:47:56.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:56.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:56.918093+0000) 2024-03-06T18:47:57.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:57.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:57.918280+0000) 2024-03-06T18:47:58.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:58.917+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:58.918513+0000) 2024-03-06T18:47:59.913 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:47:59.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:59.918768+0000) 2024-03-06T18:48:00.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:00.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:00.919025+0000) 2024-03-06T18:48:01.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:01.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:01.919239+0000) 2024-03-06T18:48:02.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:02.918+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:02.919475+0000) 2024-03-06T18:48:03.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:03.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:03.919751+0000) 2024-03-06T18:48:04.914 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:04.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:04.919986+0000) 2024-03-06T18:48:05.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:05.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:05.920196+0000) 2024-03-06T18:48:06.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:06.919+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:06.920441+0000) 2024-03-06T18:48:07.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:07.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:07.920698+0000) 2024-03-06T18:48:08.915 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:08.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:08.920937+0000) 2024-03-06T18:48:09.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:09.910 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:09.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:09.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:09.921085+0000) 2024-03-06T18:48:10.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:10.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:10.921260+0000) 2024-03-06T18:48:11.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:11.920+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:11.921480+0000) 2024-03-06T18:48:12.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:12.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:12.921661+0000) 2024-03-06T18:48:13.916 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:13.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:13.921927+0000) 2024-03-06T18:48:14.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:14.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:14.922112+0000) 2024-03-06T18:48:15.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:15.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:15.922318+0000) 2024-03-06T18:48:16.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:16.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:16.922617+0000) 2024-03-06T18:48:17.917 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:17.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:17.922839+0000) 2024-03-06T18:48:18.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:18.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:18.923047+0000) 2024-03-06T18:48:19.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:19.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:19.923285+0000) 2024-03-06T18:48:20.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:20.921+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:20.923515+0000) 2024-03-06T18:48:21.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:21.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:21.923779+0000) 2024-03-06T18:48:22.918 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:22.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:22.923972+0000) 2024-03-06T18:48:23.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:23.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:23.924206+0000) 2024-03-06T18:48:24.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:24.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:24.924437+0000) 2024-03-06T18:48:25.919 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:25.922+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:25.924639+0000) 2024-03-06T18:48:26.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:26.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:26.924924+0000) 2024-03-06T18:48:27.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:27.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:27.925151+0000) 2024-03-06T18:48:28.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:28.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:28.925374+0000) 2024-03-06T18:48:29.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:29.923+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:29.925591+0000) 2024-03-06T18:48:30.920 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:30.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:30.925827+0000) 2024-03-06T18:48:31.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:31.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:31.926056+0000) 2024-03-06T18:48:32.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:32.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:32.926251+0000) 2024-03-06T18:48:33.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:33.924+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:33.926445+0000) 2024-03-06T18:48:34.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:34.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:34.926695+0000) 2024-03-06T18:48:35.921 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:35.925+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:35.926915+0000) 2024-03-06T18:48:36.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:36.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:36.927148+0000) 2024-03-06T18:48:37.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:37.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:37.927335+0000) 2024-03-06T18:48:38.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:38.926+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:38.927566+0000) 2024-03-06T18:48:39.922 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:39.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:39.927797+0000) 2024-03-06T18:48:40.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:40.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:40.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:40.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:40.928046+0000) 2024-03-06T18:48:41.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:41.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:41.928242+0000) 2024-03-06T18:48:42.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:42.927+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:42.928485+0000) 2024-03-06T18:48:43.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:43.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:43.928656+0000) 2024-03-06T18:48:44.923 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:44.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:44.928826+0000) 2024-03-06T18:48:45.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:45.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:45.929074+0000) 2024-03-06T18:48:46.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:46.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:46.929351+0000) 2024-03-06T18:48:47.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:47.928+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:47.929556+0000) 2024-03-06T18:48:48.924 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:48.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:48.929806+0000) 2024-03-06T18:48:49.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:49.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:49.930064+0000) 2024-03-06T18:48:50.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:50.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:50.930269+0000) 2024-03-06T18:48:51.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:51.929+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:51.930504+0000) 2024-03-06T18:48:52.925 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:52.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:52.930782+0000) 2024-03-06T18:48:53.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:53.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:53.931013+0000) 2024-03-06T18:48:54.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:54.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:54.931296+0000) 2024-03-06T18:48:55.926 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:55.930+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:55.931567+0000) 2024-03-06T18:48:56.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:56.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:56.931843+0000) 2024-03-06T18:48:57.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:57.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:57.932080+0000) 2024-03-06T18:48:58.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:58.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:58.932322+0000) 2024-03-06T18:48:59.927 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:48:59.931+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:59.932524+0000) 2024-03-06T18:49:00.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:00.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:00.932786+0000) 2024-03-06T18:49:01.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:01.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:01.933034+0000) 2024-03-06T18:49:02.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:02.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:02.933290+0000) 2024-03-06T18:49:03.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:03.932+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:03.933540+0000) 2024-03-06T18:49:04.928 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:04.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:04.933835+0000) 2024-03-06T18:49:05.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:05.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:05.934092+0000) 2024-03-06T18:49:06.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:06.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:06.934338+0000) 2024-03-06T18:49:07.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:07.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:07.934564+0000) 2024-03-06T18:49:08.929 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:08.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:08.934848+0000) 2024-03-06T18:49:09.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:09.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:09.935125+0000) 2024-03-06T18:49:10.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:10.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:10.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:10.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:10.935312+0000) 2024-03-06T18:49:11.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:11.933+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:11.935597+0000) 2024-03-06T18:49:12.930 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:12.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:12.935882+0000) 2024-03-06T18:49:13.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:13.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:13.936160+0000) 2024-03-06T18:49:14.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:14.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:14.936368+0000) 2024-03-06T18:49:15.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:15.934+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:15.936617+0000) 2024-03-06T18:49:16.931 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:16.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:16.936797+0000) 2024-03-06T18:49:17.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:17.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:17.937035+0000) 2024-03-06T18:49:18.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:18.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:18.937271+0000) 2024-03-06T18:49:19.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:19.935+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:19.937500+0000) 2024-03-06T18:49:20.932 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:20.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:20.937809+0000) 2024-03-06T18:49:21.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:21.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:21.938004+0000) 2024-03-06T18:49:22.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:22.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:22.938210+0000) 2024-03-06T18:49:23.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:23.936+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:23.938455+0000) 2024-03-06T18:49:24.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:24.937+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:24.938648+0000) 2024-03-06T18:49:25.933 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:25.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:25.938870+0000) 2024-03-06T18:49:26.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:26.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:26.939109+0000) 2024-03-06T18:49:27.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:27.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:27.939305+0000) 2024-03-06T18:49:28.934 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:28.938+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:28.939551+0000) 2024-03-06T18:49:29.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:29.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:29.939776+0000) 2024-03-06T18:49:30.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:30.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:30.939988+0000) 2024-03-06T18:49:31.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:31.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:31.940170+0000) 2024-03-06T18:49:32.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:32.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:32.940385+0000) 2024-03-06T18:49:33.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:33.939+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:33.940621+0000) 2024-03-06T18:49:34.935 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:34.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:34.940865+0000) 2024-03-06T18:49:35.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:35.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:35.941093+0000) 2024-03-06T18:49:36.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:36.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:36.941358+0000) 2024-03-06T18:49:37.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:37.940+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:37.941562+0000) 2024-03-06T18:49:38.936 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:38.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:38.941807+0000) 2024-03-06T18:49:39.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:39.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:39.942024+0000) 2024-03-06T18:49:40.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:40.483 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:40.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:40.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:40.942273+0000) 2024-03-06T18:49:41.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:41.941+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:41.942496+0000) 2024-03-06T18:49:42.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:42.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:42.942770+0000) 2024-03-06T18:49:43.937 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:43.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:43.942934+0000) 2024-03-06T18:49:44.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:44.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:44.943118+0000) 2024-03-06T18:49:45.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:45.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:45.943302+0000) 2024-03-06T18:49:46.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:46.942+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:46.943511+0000) 2024-03-06T18:49:47.938 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:47.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:47.943729+0000) 2024-03-06T18:49:48.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:48.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:48.943926+0000) 2024-03-06T18:49:49.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:49.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:49.944099+0000) 2024-03-06T18:49:50.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:50.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:50.944277+0000) 2024-03-06T18:49:51.939 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:51.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:51.944508+0000) 2024-03-06T18:49:52.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:52.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:52.944780+0000) 2024-03-06T18:49:53.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:53.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:53.944973+0000) 2024-03-06T18:49:54.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:54.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:54.945160+0000) 2024-03-06T18:49:55.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:55.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:55.945337+0000) 2024-03-06T18:49:56.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:56.943+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:56.945565+0000) 2024-03-06T18:49:57.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:57.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:57.945846+0000) 2024-03-06T18:49:58.940 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:58.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:58.946035+0000) 2024-03-06T18:49:59.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:49:59.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:59.946216+0000) 2024-03-06T18:50:00.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:00.944+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:00.946443+0000) 2024-03-06T18:50:01.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:01.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:01.946699+0000) 2024-03-06T18:50:02.941 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:02.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:02.946956+0000) 2024-03-06T18:50:03.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:03.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:03.947184+0000) 2024-03-06T18:50:04.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:04.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:04.947410+0000) 2024-03-06T18:50:05.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:05.945+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:05.947635+0000) 2024-03-06T18:50:06.942 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:06.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:06.947862+0000) 2024-03-06T18:50:07.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:07.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:07.948036+0000) 2024-03-06T18:50:08.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:08.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:08.948264+0000) 2024-03-06T18:50:09.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:09.946+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:09.948487+0000) 2024-03-06T18:50:10.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:10.680 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:10.943 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:10.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:10.948763+0000) 2024-03-06T18:50:11.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:11.947+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:11.948991+0000) 2024-03-06T18:50:12.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:12.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:12.949229+0000) 2024-03-06T18:50:13.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:13.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:13.949421+0000) 2024-03-06T18:50:14.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:14.948+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:14.949647+0000) 2024-03-06T18:50:15.944 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:15.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:15.949919+0000) 2024-03-06T18:50:16.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:16.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:16.950155+0000) 2024-03-06T18:50:17.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:17.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:17.950376+0000) 2024-03-06T18:50:18.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:18.949+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:18.950564+0000) 2024-03-06T18:50:19.945 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:19.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:19.950803+0000) 2024-03-06T18:50:20.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:20.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:20.951031+0000) 2024-03-06T18:50:21.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:21.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:21.951218+0000) 2024-03-06T18:50:22.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:22.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:22.951418+0000) 2024-03-06T18:50:23.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:23.950+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:23.951613+0000) 2024-03-06T18:50:24.946 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:24.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:24.951853+0000) 2024-03-06T18:50:25.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:25.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:25.952110+0000) 2024-03-06T18:50:26.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:26.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:26.952337+0000) 2024-03-06T18:50:27.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:27.951+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:27.952574+0000) 2024-03-06T18:50:28.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:28.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:28.952807+0000) 2024-03-06T18:50:29.947 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:29.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:29.953009+0000) 2024-03-06T18:50:30.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:30.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:30.953202+0000) 2024-03-06T18:50:31.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:31.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:31.953402+0000) 2024-03-06T18:50:32.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:32.952+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:32.953595+0000) 2024-03-06T18:50:33.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:33.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:33.953840+0000) 2024-03-06T18:50:34.948 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:34.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:34.954057+0000) 2024-03-06T18:50:35.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:35.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:35.954299+0000) 2024-03-06T18:50:36.949 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:36.953+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:36.954499+0000) 2024-03-06T18:50:37.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:37.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:37.954786+0000) 2024-03-06T18:50:38.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:38.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:38.955029+0000) 2024-03-06T18:50:39.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:39.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:39.955296+0000) 2024-03-06T18:50:40.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:40.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:40.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:40.954+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:40.955481+0000) 2024-03-06T18:50:41.950 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:41.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:41.955774+0000) 2024-03-06T18:50:42.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:42.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:42.956049+0000) 2024-03-06T18:50:43.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:43.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:43.956321+0000) 2024-03-06T18:50:44.951 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:44.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:44.956516+0000) 2024-03-06T18:50:45.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:45.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:45.956869+0000) 2024-03-06T18:50:46.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:46.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:46.957137+0000) 2024-03-06T18:50:47.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:47.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:47.957417+0000) 2024-03-06T18:50:48.952 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:48.955+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:48.957641+0000) 2024-03-06T18:50:49.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:49.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:49.957897+0000) 2024-03-06T18:50:50.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:50.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:50.958089+0000) 2024-03-06T18:50:51.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:51.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:51.958357+0000) 2024-03-06T18:50:52.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:52.956+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:52.958645+0000) 2024-03-06T18:50:53.953 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:53.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:53.958897+0000) 2024-03-06T18:50:54.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:54.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:54.959143+0000) 2024-03-06T18:50:55.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:55.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:55.959357+0000) 2024-03-06T18:50:56.954 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:56.957+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:56.959593+0000) 2024-03-06T18:50:57.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:57.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:57.959849+0000) 2024-03-06T18:50:58.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:58.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:58.960055+0000) 2024-03-06T18:50:59.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:50:59.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:59.960327+0000) 2024-03-06T18:51:00.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:00.958+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:00.960567+0000) 2024-03-06T18:51:01.955 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:01.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:01.960814+0000) 2024-03-06T18:51:02.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:02.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:02.961020+0000) 2024-03-06T18:51:03.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:03.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:03.961244+0000) 2024-03-06T18:51:04.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:04.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:04.961423+0000) 2024-03-06T18:51:05.956 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:05.960+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:05.961619+0000) 2024-03-06T18:51:06.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:06.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:06.961906+0000) 2024-03-06T18:51:07.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:07.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:07.962097+0000) 2024-03-06T18:51:08.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:08.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:08.962330+0000) 2024-03-06T18:51:09.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:09.961+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:09.962558+0000) 2024-03-06T18:51:10.957 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:10.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:10.962814+0000) 2024-03-06T18:51:11.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:11.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:11.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:11.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:11.963013+0000) 2024-03-06T18:51:12.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:12.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:12.963244+0000) 2024-03-06T18:51:13.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:13.962+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:13.963473+0000) 2024-03-06T18:51:14.958 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:14.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:14.963738+0000) 2024-03-06T18:51:15.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:15.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:15.963929+0000) 2024-03-06T18:51:16.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:16.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:16.964102+0000) 2024-03-06T18:51:17.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:17.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:17.964312+0000) 2024-03-06T18:51:18.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:18.963+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:18.964561+0000) 2024-03-06T18:51:19.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:19.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:19.964806+0000) 2024-03-06T18:51:20.959 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:20.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:20.965009+0000) 2024-03-06T18:51:21.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:21.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:21.965233+0000) 2024-03-06T18:51:22.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:22.964+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:22.965447+0000) 2024-03-06T18:51:23.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:23.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:23.965688+0000) 2024-03-06T18:51:24.960 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:24.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:24.965915+0000) 2024-03-06T18:51:25.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:25.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:25.966141+0000) 2024-03-06T18:51:26.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:26.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:26.966342+0000) 2024-03-06T18:51:27.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:27.965+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:27.966579+0000) 2024-03-06T18:51:28.961 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:28.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:28.966819+0000) 2024-03-06T18:51:29.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:29.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:29.967044+0000) 2024-03-06T18:51:30.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:30.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:30.967271+0000) 2024-03-06T18:51:31.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:31.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:31.967473+0000) 2024-03-06T18:51:32.962 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:32.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:32.967717+0000) 2024-03-06T18:51:33.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:33.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:33.968025+0000) 2024-03-06T18:51:34.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:34.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:34.968200+0000) 2024-03-06T18:51:35.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:35.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:35.968421+0000) 2024-03-06T18:51:36.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:36.966+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:36.968609+0000) 2024-03-06T18:51:37.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:37.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:37.968831+0000) 2024-03-06T18:51:38.963 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:38.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:38.969027+0000) 2024-03-06T18:51:39.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:39.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:39.969259+0000) 2024-03-06T18:51:40.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:40.967+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:40.969462+0000) 2024-03-06T18:51:41.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:41.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:41.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:41.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:41.969732+0000) 2024-03-06T18:51:42.964 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:42.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:42.969969+0000) 2024-03-06T18:51:43.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:43.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:43.970149+0000) 2024-03-06T18:51:44.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:44.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:44.970379+0000) 2024-03-06T18:51:45.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:45.968+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:45.970598+0000) 2024-03-06T18:51:46.965 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:46.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:46.970850+0000) 2024-03-06T18:51:47.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:47.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:47.971099+0000) 2024-03-06T18:51:48.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:48.969+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:48.971319+0000) 2024-03-06T18:51:49.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:49.970+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:49.971576+0000) 2024-03-06T18:51:50.966 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:50.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:50.971820+0000) 2024-03-06T18:51:51.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:51.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:51.972077+0000) 2024-03-06T18:51:52.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:52.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:52.972324+0000) 2024-03-06T18:51:53.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:53.971+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:53.972544+0000) 2024-03-06T18:51:54.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:54.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:54.972835+0000) 2024-03-06T18:51:55.967 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:55.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:55.973049+0000) 2024-03-06T18:51:56.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:56.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:56.973274+0000) 2024-03-06T18:51:57.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:57.972+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:57.973524+0000) 2024-03-06T18:51:58.968 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:58.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:58.973793+0000) 2024-03-06T18:51:59.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:51:59.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:59.974022+0000) 2024-03-06T18:52:00.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:00.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:00.974250+0000) 2024-03-06T18:52:01.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:01.973+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:01.974498+0000) 2024-03-06T18:52:02.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:02.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:02.974766+0000) 2024-03-06T18:52:03.969 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:03.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:03.974999+0000) 2024-03-06T18:52:04.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:04.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:04.975176+0000) 2024-03-06T18:52:05.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:05.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:05.975399+0000) 2024-03-06T18:52:06.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:06.974+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:06.975625+0000) 2024-03-06T18:52:07.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:07.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:07.975782+0000) 2024-03-06T18:52:08.970 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:08.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:08.976023+0000) 2024-03-06T18:52:09.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:09.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:09.976281+0000) 2024-03-06T18:52:10.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:10.975+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:10.976510+0000) 2024-03-06T18:52:11.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:11.583 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:11.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:11.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:11.976818+0000) 2024-03-06T18:52:12.971 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:12.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:12.977032+0000) 2024-03-06T18:52:13.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:13.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:13.977254+0000) 2024-03-06T18:52:14.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:14.976+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:14.977473+0000) 2024-03-06T18:52:15.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:15.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:15.977659+0000) 2024-03-06T18:52:16.972 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:16.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:16.977891+0000) 2024-03-06T18:52:17.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:17.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:17.978084+0000) 2024-03-06T18:52:18.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:18.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:18.978309+0000) 2024-03-06T18:52:19.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:19.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:19.978474+0000) 2024-03-06T18:52:20.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:20.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:20.978700+0000) 2024-03-06T18:52:21.973 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:21.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:21.978953+0000) 2024-03-06T18:52:22.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:22.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:22.979194+0000) 2024-03-06T18:52:23.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:23.977+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:23.979425+0000) 2024-03-06T18:52:24.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:24.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:24.979700+0000) 2024-03-06T18:52:25.974 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:25.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:25.979916+0000) 2024-03-06T18:52:26.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:26.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:26.980197+0000) 2024-03-06T18:52:27.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:27.978+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:27.980468+0000) 2024-03-06T18:52:28.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:28.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:28.980768+0000) 2024-03-06T18:52:29.975 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:29.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:29.981050+0000) 2024-03-06T18:52:30.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:30.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:30.981313+0000) 2024-03-06T18:52:31.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:31.979+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:31.981594+0000) 2024-03-06T18:52:32.976 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:32.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:32.981859+0000) 2024-03-06T18:52:33.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:33.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:33.982105+0000) 2024-03-06T18:52:34.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:34.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:34.982313+0000) 2024-03-06T18:52:35.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:35.980+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:35.982551+0000) 2024-03-06T18:52:36.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:36.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:36.982788+0000) 2024-03-06T18:52:37.977 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:37.981+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:37.982986+0000) 2024-03-06T18:52:38.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:38.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:38.983210+0000) 2024-03-06T18:52:39.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:39.982+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:39.983437+0000) 2024-03-06T18:52:40.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:40.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:40.983700+0000) 2024-03-06T18:52:41.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:41.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:41.978 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:41.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:41.983946+0000) 2024-03-06T18:52:42.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:42.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:42.984261+0000) 2024-03-06T18:52:43.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:43.983+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:43.984489+0000) 2024-03-06T18:52:44.979 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:44.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:44.984743+0000) 2024-03-06T18:52:45.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:45.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:45.984989+0000) 2024-03-06T18:52:46.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:46.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:46.985149+0000) 2024-03-06T18:52:47.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:47.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:47.985386+0000) 2024-03-06T18:52:48.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:48.984+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:48.985593+0000) 2024-03-06T18:52:49.980 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:49.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:49.985830+0000) 2024-03-06T18:52:50.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:50.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:50.986111+0000) 2024-03-06T18:52:51.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:51.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:51.986358+0000) 2024-03-06T18:52:52.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:52.985+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:52.986548+0000) 2024-03-06T18:52:53.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:53.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:53.986791+0000) 2024-03-06T18:52:54.981 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:54.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:54.986982+0000) 2024-03-06T18:52:55.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:55.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:55.987205+0000) 2024-03-06T18:52:56.982 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:56.986+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:56.987436+0000) 2024-03-06T18:52:57.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:57.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:57.987661+0000) 2024-03-06T18:52:58.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:58.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:58.987939+0000) 2024-03-06T18:52:59.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:52:59.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:59.988138+0000) 2024-03-06T18:53:00.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:00.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:00.988362+0000) 2024-03-06T18:53:01.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:01.987+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:01.988599+0000) 2024-03-06T18:53:02.983 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:02.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:02.988835+0000) 2024-03-06T18:53:03.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:03.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:03.989027+0000) 2024-03-06T18:53:04.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:04.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:04.989256+0000) 2024-03-06T18:53:05.984 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:05.988+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:05.989439+0000) 2024-03-06T18:53:06.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:06.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:06.989661+0000) 2024-03-06T18:53:07.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:07.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:07.990187+0000) 2024-03-06T18:53:08.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:08.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:08.990418+0000) 2024-03-06T18:53:09.985 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:09.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:09.990641+0000) 2024-03-06T18:53:10.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:10.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:10.990900+0000) 2024-03-06T18:53:11.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:11.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:11.991135+0000) 2024-03-06T18:53:11.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:12.005 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:12.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:12.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:12.991383+0000) 2024-03-06T18:53:13.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:13.989+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:13.991593+0000) 2024-03-06T18:53:14.986 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:14.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:14.991838+0000) 2024-03-06T18:53:15.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:15.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:15.992099+0000) 2024-03-06T18:53:16.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:16.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:16.992327+0000) 2024-03-06T18:53:17.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:17.990+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:17.992563+0000) 2024-03-06T18:53:18.987 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:18.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:18.992798+0000) 2024-03-06T18:53:19.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:19.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:19.993025+0000) 2024-03-06T18:53:20.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:20.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:20.993251+0000) 2024-03-06T18:53:21.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:21.991+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:21.993461+0000) 2024-03-06T18:53:22.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:22.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:22.993744+0000) 2024-03-06T18:53:23.988 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:23.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:23.993932+0000) 2024-03-06T18:53:24.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:24.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:24.994177+0000) 2024-03-06T18:53:25.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:25.992+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:25.994400+0000) 2024-03-06T18:53:26.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:26.993+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:26.994639+0000) 2024-03-06T18:53:27.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:27.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:27.994866+0000) 2024-03-06T18:53:28.989 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:28.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:28.995070+0000) 2024-03-06T18:53:29.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:29.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:29.995319+0000) 2024-03-06T18:53:30.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:30.994+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:30.995526+0000) 2024-03-06T18:53:31.990 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:31.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:31.996014+0000) 2024-03-06T18:53:32.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:32.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:32.996257+0000) 2024-03-06T18:53:33.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:33.995+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:33.996455+0000) 2024-03-06T18:53:34.991 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:34.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:34.996725+0000) 2024-03-06T18:53:35.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:35.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:35.996949+0000) 2024-03-06T18:53:36.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:36.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:36.997210+0000) 2024-03-06T18:53:37.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:37.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:37.997458+0000) 2024-03-06T18:53:38.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:38.996+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:38.997596+0000) 2024-03-06T18:53:39.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:39.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:39.997819+0000) 2024-03-06T18:53:40.992 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:40.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:40.998073+0000) 2024-03-06T18:53:41.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:41.997+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:41.998360+0000) 2024-03-06T18:53:42.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:42.208 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:42.993 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:42.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:42.998654+0000) 2024-03-06T18:53:43.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:43.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:43.998988+0000) 2024-03-06T18:53:44.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:44.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:44.999218+0000) 2024-03-06T18:53:45.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:45.998+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:45.999490+0000) 2024-03-06T18:53:46.994 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:46.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:46.999800+0000) 2024-03-06T18:53:47.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:47.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:48.000074+0000) 2024-03-06T18:53:48.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:48.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:49.000292+0000) 2024-03-06T18:53:49.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:49.999+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:50.000485+0000) 2024-03-06T18:53:50.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:51.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:51.000747+0000) 2024-03-06T18:53:51.995 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:52.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:52.000982+0000) 2024-03-06T18:53:52.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:53.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:53.001273+0000) 2024-03-06T18:53:53.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:54.000+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:54.001551+0000) 2024-03-06T18:53:54.996 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:55.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:55.001810+0000) 2024-03-06T18:53:55.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:56.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:56.002094+0000) 2024-03-06T18:53:56.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:57.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:57.002372+0000) 2024-03-06T18:53:57.997 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:58.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:58.002583+0000) 2024-03-06T18:53:59.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:53:59.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:59.002827+0000) 2024-03-06T18:53:59.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:00.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:00.003104+0000) 2024-03-06T18:54:00.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:01.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:01.003337+0000) 2024-03-06T18:54:01.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:02.001+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:02.003610+0000) 2024-03-06T18:54:02.998 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:03.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:03.003853+0000) 2024-03-06T18:54:03.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:04.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:04.004054+0000) 2024-03-06T18:54:04.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:05.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:05.004244+0000) 2024-03-06T18:54:05.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:06.002+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:06.004485+0000) 2024-03-06T18:54:06.999 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:07.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:07.004800+0000) 2024-03-06T18:54:08.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:08.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:08.005057+0000) 2024-03-06T18:54:09.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:09.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:09.005375+0000) 2024-03-06T18:54:10.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:10.003+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:10.005521+0000) 2024-03-06T18:54:11.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:11.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:11.005757+0000) 2024-03-06T18:54:12.000 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:12.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:12.006001+0000) 2024-03-06T18:54:12.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:12.429 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:13.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:13.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:13.006289+0000) 2024-03-06T18:54:14.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:14.004+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:14.006534+0000) 2024-03-06T18:54:15.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:15.005+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:15.006779+0000) 2024-03-06T18:54:16.001 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:16.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:16.006964+0000) 2024-03-06T18:54:17.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:17.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:17.007211+0000) 2024-03-06T18:54:18.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:18.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:18.007390+0000) 2024-03-06T18:54:19.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:19.006+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:19.007613+0000) 2024-03-06T18:54:20.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:20.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:20.007853+0000) 2024-03-06T18:54:21.002 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:21.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:21.008051+0000) 2024-03-06T18:54:22.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:22.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:22.008291+0000) 2024-03-06T18:54:23.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:23.007+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:23.008537+0000) 2024-03-06T18:54:24.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:24.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:24.008787+0000) 2024-03-06T18:54:25.003 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:25.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:25.009037+0000) 2024-03-06T18:54:26.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:26.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:26.009263+0000) 2024-03-06T18:54:27.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:27.008+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:27.009505+0000) 2024-03-06T18:54:28.004 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:28.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:28.009755+0000) 2024-03-06T18:54:29.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:29.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:29.009910+0000) 2024-03-06T18:54:30.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:30.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:30.010132+0000) 2024-03-06T18:54:31.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:31.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:31.010351+0000) 2024-03-06T18:54:32.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:32.009+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:32.010583+0000) 2024-03-06T18:54:33.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:33.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:33.010829+0000) 2024-03-06T18:54:34.005 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:34.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:34.011047+0000) 2024-03-06T18:54:35.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:35.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:35.011329+0000) 2024-03-06T18:54:36.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:36.010+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:36.011564+0000) 2024-03-06T18:54:37.006 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:37.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:37.011844+0000) 2024-03-06T18:54:38.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:38.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:38.012124+0000) 2024-03-06T18:54:39.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:39.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:39.012331+0000) 2024-03-06T18:54:40.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:40.011+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:40.012562+0000) 2024-03-06T18:54:41.007 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:41.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:41.012819+0000) 2024-03-06T18:54:42.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:42.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:42.013051+0000) 2024-03-06T18:54:42.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:42.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:43.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:43.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:43.013349+0000) 2024-03-06T18:54:44.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:44.012+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:44.013627+0000) 2024-03-06T18:54:45.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:45.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:45.013904+0000) 2024-03-06T18:54:46.008 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:46.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:46.014164+0000) 2024-03-06T18:54:47.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:47.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:47.014363+0000) 2024-03-06T18:54:48.009 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:48.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:48.014625+0000) 2024-03-06T18:54:49.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:49.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:49.014960+0000) 2024-03-06T18:54:50.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:50.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:50.015217+0000) 2024-03-06T18:54:51.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:51.013+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:51.015422+0000) 2024-03-06T18:54:52.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:52.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:52.015660+0000) 2024-03-06T18:54:53.010 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:53.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:53.016013+0000) 2024-03-06T18:54:54.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:54.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:54.016288+0000) 2024-03-06T18:54:55.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:55.014+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:55.016560+0000) 2024-03-06T18:54:56.011 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:56.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:56.016824+0000) 2024-03-06T18:54:57.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:57.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:57.017120+0000) 2024-03-06T18:54:58.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:58.015+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:58.017388+0000) 2024-03-06T18:54:59.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:54:59.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:59.017661+0000) 2024-03-06T18:55:00.012 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:00.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:00.017941+0000) 2024-03-06T18:55:01.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:01.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:01.018216+0000) 2024-03-06T18:55:02.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:02.016+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:02.018437+0000) 2024-03-06T18:55:03.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:03.017+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:03.018724+0000) 2024-03-06T18:55:04.013 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:04.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:04.018954+0000) 2024-03-06T18:55:05.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:05.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:05.019199+0000) 2024-03-06T18:55:06.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:06.018+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:06.019443+0000) 2024-03-06T18:55:07.014 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:07.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:07.019687+0000) 2024-03-06T18:55:08.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:08.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:08.019916+0000) 2024-03-06T18:55:09.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:09.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:09.020139+0000) 2024-03-06T18:55:10.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:10.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:10.020369+0000) 2024-03-06T18:55:11.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:11.019+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:11.020602+0000) 2024-03-06T18:55:12.015 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:12.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:12.020866+0000) 2024-03-06T18:55:12.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:12.865 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:13.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:13.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:13.021142+0000) 2024-03-06T18:55:14.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:14.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:14.021386+0000) 2024-03-06T18:55:15.016 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:15.020+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:15.021588+0000) 2024-03-06T18:55:16.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:16.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:16.021833+0000) 2024-03-06T18:55:17.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:17.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:17.022086+0000) 2024-03-06T18:55:18.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:18.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:18.022330+0000) 2024-03-06T18:55:19.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:19.021+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:19.022575+0000) 2024-03-06T18:55:20.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:20.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:20.022828+0000) 2024-03-06T18:55:21.017 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:21.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:21.023068+0000) 2024-03-06T18:55:22.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:22.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:22.023315+0000) 2024-03-06T18:55:23.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:23.022+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:23.023516+0000) 2024-03-06T18:55:24.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:24.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:24.023768+0000) 2024-03-06T18:55:25.018 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:25.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:25.023996+0000) 2024-03-06T18:55:26.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:26.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:26.024225+0000) 2024-03-06T18:55:27.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:27.023+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:27.024455+0000) 2024-03-06T18:55:28.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:28.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:28.024720+0000) 2024-03-06T18:55:29.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:29.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:29.024952+0000) 2024-03-06T18:55:30.019 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:30.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:30.025126+0000) 2024-03-06T18:55:31.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:31.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:31.025366+0000) 2024-03-06T18:55:32.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:32.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:32.025586+0000) 2024-03-06T18:55:33.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:33.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:33.025809+0000) 2024-03-06T18:55:34.020 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:34.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:34.025990+0000) 2024-03-06T18:55:35.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:35.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:35.026164+0000) 2024-03-06T18:55:36.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:36.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:36.026384+0000) 2024-03-06T18:55:37.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:37.024+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:37.026617+0000) 2024-03-06T18:55:38.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:38.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:38.026839+0000) 2024-03-06T18:55:39.021 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:39.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:39.027018+0000) 2024-03-06T18:55:40.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:40.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:40.027244+0000) 2024-03-06T18:55:41.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:41.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:41.027464+0000) 2024-03-06T18:55:42.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:42.025+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:42.027608+0000) 2024-03-06T18:55:43.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:43.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:43.027848+0000) 2024-03-06T18:55:43.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:43.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:44.022 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:44.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:44.028078+0000) 2024-03-06T18:55:45.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:45.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:45.028357+0000) 2024-03-06T18:55:46.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:46.026+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:46.028531+0000) 2024-03-06T18:55:47.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:47.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:47.028813+0000) 2024-03-06T18:55:48.023 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:48.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:48.029029+0000) 2024-03-06T18:55:49.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:49.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:49.029257+0000) 2024-03-06T18:55:50.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:50.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:50.029425+0000) 2024-03-06T18:55:51.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:51.027+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:51.029633+0000) 2024-03-06T18:55:52.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:52.028+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:52.029857+0000) 2024-03-06T18:55:53.024 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:53.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:53.030099+0000) 2024-03-06T18:55:54.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:54.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:54.030324+0000) 2024-03-06T18:55:55.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:55.029+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:55.030558+0000) 2024-03-06T18:55:56.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:56.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:56.030801+0000) 2024-03-06T18:55:57.025 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:57.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:57.031030+0000) 2024-03-06T18:55:58.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:58.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:58.031255+0000) 2024-03-06T18:55:59.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:55:59.030+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:59.031483+0000) 2024-03-06T18:56:00.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:00.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:00.031742+0000) 2024-03-06T18:56:01.026 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:01.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:01.031970+0000) 2024-03-06T18:56:02.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:02.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:02.032204+0000) 2024-03-06T18:56:03.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:03.031+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:03.032443+0000) 2024-03-06T18:56:04.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:04.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:04.032700+0000) 2024-03-06T18:56:05.027 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:05.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:05.032924+0000) 2024-03-06T18:56:06.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:06.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:06.033151+0000) 2024-03-06T18:56:07.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:07.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:07.033341+0000) 2024-03-06T18:56:08.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:08.032+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:08.033557+0000) 2024-03-06T18:56:09.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:09.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:09.033786+0000) 2024-03-06T18:56:10.028 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:10.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:10.034006+0000) 2024-03-06T18:56:11.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:11.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:11.034237+0000) 2024-03-06T18:56:12.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:12.033+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:12.034523+0000) 2024-03-06T18:56:13.029 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:13.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:13.034816+0000) 2024-03-06T18:56:13.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:14.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:14.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:14.035059+0000) 2024-03-06T18:56:15.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:15.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:15.035269+0000) 2024-03-06T18:56:16.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:16.034+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:16.035473+0000) 2024-03-06T18:56:17.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:17.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:17.035749+0000) 2024-03-06T18:56:18.030 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:18.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:18.035976+0000) 2024-03-06T18:56:19.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:19.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:19.036261+0000) 2024-03-06T18:56:20.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:20.035+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:20.036540+0000) 2024-03-06T18:56:21.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:21.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:21.036816+0000) 2024-03-06T18:56:22.031 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:22.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:22.037059+0000) 2024-03-06T18:56:23.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:23.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:23.037330+0000) 2024-03-06T18:56:24.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:24.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:24.037586+0000) 2024-03-06T18:56:25.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:25.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:25.037837+0000) 2024-03-06T18:56:26.032 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:26.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:26.038047+0000) 2024-03-06T18:56:27.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:27.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:27.038294+0000) 2024-03-06T18:56:28.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:28.036+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:28.038547+0000) 2024-03-06T18:56:29.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:29.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:29.038791+0000) 2024-03-06T18:56:30.033 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:30.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:30.039012+0000) 2024-03-06T18:56:31.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:31.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:31.039238+0000) 2024-03-06T18:56:32.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:32.037+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:32.039466+0000) 2024-03-06T18:56:33.034 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:33.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:33.039770+0000) 2024-03-06T18:56:34.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:34.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:34.040012+0000) 2024-03-06T18:56:35.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:35.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:35.040261+0000) 2024-03-06T18:56:36.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:36.038+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:36.040503+0000) 2024-03-06T18:56:37.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:37.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:37.040787+0000) 2024-03-06T18:56:38.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:38.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:38.040965+0000) 2024-03-06T18:56:39.035 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:39.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:39.041176+0000) 2024-03-06T18:56:40.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:40.039+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:40.041436+0000) 2024-03-06T18:56:41.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:41.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:41.041754+0000) 2024-03-06T18:56:42.036 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:42.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:42.041991+0000) 2024-03-06T18:56:43.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:43.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:43.042257+0000) 2024-03-06T18:56:43.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:43.492 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:44.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:44.041+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:44.042495+0000) 2024-03-06T18:56:45.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:45.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:45.042717+0000) 2024-03-06T18:56:46.037 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:46.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:46.042968+0000) 2024-03-06T18:56:47.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:47.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:47.043210+0000) 2024-03-06T18:56:48.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:48.042+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:48.043451+0000) 2024-03-06T18:56:49.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:49.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:49.043736+0000) 2024-03-06T18:56:50.038 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:50.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:50.043984+0000) 2024-03-06T18:56:51.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:51.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:51.044238+0000) 2024-03-06T18:56:52.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:52.043+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:52.044483+0000) 2024-03-06T18:56:53.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:53.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:53.044777+0000) 2024-03-06T18:56:54.039 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:54.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:54.045010+0000) 2024-03-06T18:56:55.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:55.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:55.045237+0000) 2024-03-06T18:56:56.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:56.044+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:56.045470+0000) 2024-03-06T18:56:57.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:57.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:57.045754+0000) 2024-03-06T18:56:58.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:58.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:58.045957+0000) 2024-03-06T18:56:59.040 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:56:59.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:59.046156+0000) 2024-03-06T18:57:00.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:00.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:00.046396+0000) 2024-03-06T18:57:01.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:01.045+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:01.046634+0000) 2024-03-06T18:57:02.041 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:02.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:02.046897+0000) 2024-03-06T18:57:03.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:03.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:03.047149+0000) 2024-03-06T18:57:04.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:04.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:04.047383+0000) 2024-03-06T18:57:05.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:05.046+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:05.047614+0000) 2024-03-06T18:57:06.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:06.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:06.047830+0000) 2024-03-06T18:57:07.042 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:07.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:07.048031+0000) 2024-03-06T18:57:08.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:08.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:08.048282+0000) 2024-03-06T18:57:09.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:09.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:09.048465+0000) 2024-03-06T18:57:10.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:10.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:10.048705+0000) 2024-03-06T18:57:11.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:11.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:11.048944+0000) 2024-03-06T18:57:12.043 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:12.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:12.049175+0000) 2024-03-06T18:57:13.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:13.047+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:13.049433+0000) 2024-03-06T18:57:13.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:13.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:14.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:14.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:14.049722+0000) 2024-03-06T18:57:15.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:15.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:15.049988+0000) 2024-03-06T18:57:16.044 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:16.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:16.050183+0000) 2024-03-06T18:57:17.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:17.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:17.050427+0000) 2024-03-06T18:57:18.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:18.048+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:18.050623+0000) 2024-03-06T18:57:19.045 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:19.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:19.050863+0000) 2024-03-06T18:57:20.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:20.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:20.051057+0000) 2024-03-06T18:57:21.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:21.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:21.051289+0000) 2024-03-06T18:57:22.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:22.049+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:22.051524+0000) 2024-03-06T18:57:23.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:23.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:23.051791+0000) 2024-03-06T18:57:24.046 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:24.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:24.052001+0000) 2024-03-06T18:57:25.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:25.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:25.052217+0000) 2024-03-06T18:57:26.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:26.050+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:26.052490+0000) 2024-03-06T18:57:27.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:27.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:27.052763+0000) 2024-03-06T18:57:28.047 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:28.051+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:28.053004+0000) 2024-03-06T18:57:29.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:29.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:29.053242+0000) 2024-03-06T18:57:30.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:30.052+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:30.053471+0000) 2024-03-06T18:57:31.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:31.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:31.053752+0000) 2024-03-06T18:57:32.048 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:32.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:32.053980+0000) 2024-03-06T18:57:33.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:33.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:33.054178+0000) 2024-03-06T18:57:34.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:34.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:34.054378+0000) 2024-03-06T18:57:35.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:35.053+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:35.054579+0000) 2024-03-06T18:57:36.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:36.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:36.054813+0000) 2024-03-06T18:57:37.049 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:37.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:37.055054+0000) 2024-03-06T18:57:38.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:38.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:38.055287+0000) 2024-03-06T18:57:39.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:39.054+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:39.055521+0000) 2024-03-06T18:57:40.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:40.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:40.055726+0000) 2024-03-06T18:57:41.050 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:41.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:41.055952+0000) 2024-03-06T18:57:42.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:42.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:42.056147+0000) 2024-03-06T18:57:43.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:43.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:43.056302+0000) 2024-03-06T18:57:43.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:43.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:44.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:44.055+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:44.056530+0000) 2024-03-06T18:57:45.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:45.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:45.056790+0000) 2024-03-06T18:57:46.051 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:46.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:46.057013+0000) 2024-03-06T18:57:47.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:47.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:47.057209+0000) 2024-03-06T18:57:48.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:48.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:48.057391+0000) 2024-03-06T18:57:49.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:49.056+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:49.057618+0000) 2024-03-06T18:57:50.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:50.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:50.057848+0000) 2024-03-06T18:57:51.052 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:51.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:51.058069+0000) 2024-03-06T18:57:52.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:52.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:52.058291+0000) 2024-03-06T18:57:53.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:53.057+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:53.058531+0000) 2024-03-06T18:57:54.053 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:54.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:54.058819+0000) 2024-03-06T18:57:55.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:55.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:55.059046+0000) 2024-03-06T18:57:56.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:56.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:56.059269+0000) 2024-03-06T18:57:57.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:57.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:57.059498+0000) 2024-03-06T18:57:58.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:58.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:58.059760+0000) 2024-03-06T18:57:59.054 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:57:59.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:59.059992+0000) 2024-03-06T18:58:00.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:00.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:00.060213+0000) 2024-03-06T18:58:01.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:01.058+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:01.060443+0000) 2024-03-06T18:58:02.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:02.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:02.060700+0000) 2024-03-06T18:58:03.055 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:03.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:03.060901+0000) 2024-03-06T18:58:04.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:04.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:04.061133+0000) 2024-03-06T18:58:05.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:05.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:05.061361+0000) 2024-03-06T18:58:06.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:06.059+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:06.061581+0000) 2024-03-06T18:58:07.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:07.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:07.061807+0000) 2024-03-06T18:58:08.056 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:08.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:08.062038+0000) 2024-03-06T18:58:09.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:09.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:09.062269+0000) 2024-03-06T18:58:10.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:10.060+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:10.062490+0000) 2024-03-06T18:58:11.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:11.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:11.062695+0000) 2024-03-06T18:58:12.057 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:12.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:12.062880+0000) 2024-03-06T18:58:13.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:13.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:13.063118+0000) 2024-03-06T18:58:14.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:14.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:14.063387+0000) 2024-03-06T18:58:14.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:14.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:15.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:15.061+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:15.063611+0000) 2024-03-06T18:58:16.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:16.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:16.063822+0000) 2024-03-06T18:58:17.058 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:17.062+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:17.064018+0000) 2024-03-06T18:58:18.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:18.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:18.064248+0000) 2024-03-06T18:58:19.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:19.063+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:19.064453+0000) 2024-03-06T18:58:20.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:20.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:20.064725+0000) 2024-03-06T18:58:21.059 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:21.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:21.064933+0000) 2024-03-06T18:58:22.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:22.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:22.065114+0000) 2024-03-06T18:58:23.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:23.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:23.065356+0000) 2024-03-06T18:58:24.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:24.064+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:24.065584+0000) 2024-03-06T18:58:25.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:25.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:25.065819+0000) 2024-03-06T18:58:26.060 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:26.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:26.066049+0000) 2024-03-06T18:58:27.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:27.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:27.066243+0000) 2024-03-06T18:58:28.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:28.065+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:28.066482+0000) 2024-03-06T18:58:29.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:29.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:29.066743+0000) 2024-03-06T18:58:30.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:30.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:30.066954+0000) 2024-03-06T18:58:31.061 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:31.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:31.067175+0000) 2024-03-06T18:58:32.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:32.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:32.067402+0000) 2024-03-06T18:58:33.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:33.066+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:33.067648+0000) 2024-03-06T18:58:34.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:34.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:34.067907+0000) 2024-03-06T18:58:35.062 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:35.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:35.068148+0000) 2024-03-06T18:58:36.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:36.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:36.068383+0000) 2024-03-06T18:58:37.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:37.067+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:37.068556+0000) 2024-03-06T18:58:38.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:38.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:38.068729+0000) 2024-03-06T18:58:39.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:39.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:39.068873+0000) 2024-03-06T18:58:40.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:40.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:40.069041+0000) 2024-03-06T18:58:41.063 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:41.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:41.069271+0000) 2024-03-06T18:58:42.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:42.068+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:42.069496+0000) 2024-03-06T18:58:43.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:43.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:43.069779+0000) 2024-03-06T18:58:44.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:44.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:44.069987+0000) 2024-03-06T18:58:44.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:44.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:45.064 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:45.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:45.070176+0000) 2024-03-06T18:58:46.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:46.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:46.070383+0000) 2024-03-06T18:58:47.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:47.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:47.070624+0000) 2024-03-06T18:58:48.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:48.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:48.070868+0000) 2024-03-06T18:58:49.065 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:49.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:49.071106+0000) 2024-03-06T18:58:50.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:50.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:50.071329+0000) 2024-03-06T18:58:51.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:51.069+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:51.071521+0000) 2024-03-06T18:58:52.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:52.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:52.071789+0000) 2024-03-06T18:58:53.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:53.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:53.071987+0000) 2024-03-06T18:58:54.066 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:54.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:54.072168+0000) 2024-03-06T18:58:55.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:55.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:55.072390+0000) 2024-03-06T18:58:56.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:56.070+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:56.072640+0000) 2024-03-06T18:58:57.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:57.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:57.072933+0000) 2024-03-06T18:58:58.067 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:58.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:58.073177+0000) 2024-03-06T18:58:59.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:58:59.071+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:59.073421+0000) 2024-03-06T18:59:00.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:00.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:00.073702+0000) 2024-03-06T18:59:01.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:01.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:01.073906+0000) 2024-03-06T18:59:02.068 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:02.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:02.074134+0000) 2024-03-06T18:59:03.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:03.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:03.074374+0000) 2024-03-06T18:59:04.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:04.072+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:04.074603+0000) 2024-03-06T18:59:05.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:05.073+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:05.074798+0000) 2024-03-06T18:59:06.069 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:06.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:06.074991+0000) 2024-03-06T18:59:07.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:07.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:07.075188+0000) 2024-03-06T18:59:08.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:08.074+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:08.075450+0000) 2024-03-06T18:59:09.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:09.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:09.075729+0000) 2024-03-06T18:59:10.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:10.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:10.075934+0000) 2024-03-06T18:59:11.070 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:11.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:11.076153+0000) 2024-03-06T18:59:12.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:12.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:12.076346+0000) 2024-03-06T18:59:13.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:13.075+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:13.076631+0000) 2024-03-06T18:59:14.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:14.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:14.076881+0000) 2024-03-06T18:59:14.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:14.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:15.071 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:15.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:15.077072+0000) 2024-03-06T18:59:16.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:16.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:16.077281+0000) 2024-03-06T18:59:17.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:17.076+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:17.077518+0000) 2024-03-06T18:59:18.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:18.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:18.077814+0000) 2024-03-06T18:59:19.072 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:19.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:19.078004+0000) 2024-03-06T18:59:20.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:20.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:20.078226+0000) 2024-03-06T18:59:21.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:21.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:21.078451+0000) 2024-03-06T18:59:22.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:22.077+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:22.078639+0000) 2024-03-06T18:59:23.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:23.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:23.078935+0000) 2024-03-06T18:59:24.073 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:24.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:24.079175+0000) 2024-03-06T18:59:25.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:25.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:25.079402+0000) 2024-03-06T18:59:26.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:26.078+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:26.079626+0000) 2024-03-06T18:59:27.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:27.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:27.079848+0000) 2024-03-06T18:59:28.074 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:28.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:28.080032+0000) 2024-03-06T18:59:29.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:29.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:29.080275+0000) 2024-03-06T18:59:30.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:30.079+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:30.080507+0000) 2024-03-06T18:59:31.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:31.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:31.080730+0000) 2024-03-06T18:59:32.075 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:32.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:32.080970+0000) 2024-03-06T18:59:33.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:33.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:33.081216+0000) 2024-03-06T18:59:34.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:34.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:34.081453+0000) 2024-03-06T18:59:35.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:35.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:35.081657+0000) 2024-03-06T18:59:36.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:36.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:36.081902+0000) 2024-03-06T18:59:37.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:37.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:37.082147+0000) 2024-03-06T18:59:38.076 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:38.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:38.082371+0000) 2024-03-06T18:59:39.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:39.080+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:39.082563+0000) 2024-03-06T18:59:40.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:40.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:40.082785+0000) 2024-03-06T18:59:41.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:41.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:41.083000+0000) 2024-03-06T18:59:42.077 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:42.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:42.083201+0000) 2024-03-06T18:59:43.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:43.081+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:43.083456+0000) 2024-03-06T18:59:44.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:44.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:44.083731+0000) 2024-03-06T18:59:44.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:44.619 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:45.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:45.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:45.083976+0000) 2024-03-06T18:59:46.078 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:46.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:46.084215+0000) 2024-03-06T18:59:47.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:47.082+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:47.084443+0000) 2024-03-06T18:59:48.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:48.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:48.084714+0000) 2024-03-06T18:59:49.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:49.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:49.084949+0000) 2024-03-06T18:59:50.079 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:50.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:50.085130+0000) 2024-03-06T18:59:51.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:51.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:51.085316+0000) 2024-03-06T18:59:52.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:52.083+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:52.085538+0000) 2024-03-06T18:59:53.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:53.084+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:53.085793+0000) 2024-03-06T18:59:54.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:54.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:54.085944+0000) 2024-03-06T18:59:55.080 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:55.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:55.086174+0000) 2024-03-06T18:59:56.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:56.085+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:56.086478+0000) 2024-03-06T18:59:57.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:57.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:57.086750+0000) 2024-03-06T18:59:58.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:58.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:58.087009+0000) 2024-03-06T18:59:59.081 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T18:59:59.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:59.087250+0000) 2024-03-06T19:00:00.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:00.086+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:00.087471+0000) 2024-03-06T19:00:01.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:01.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:01.087762+0000) 2024-03-06T19:00:02.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:02.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:02.087984+0000) 2024-03-06T19:00:03.082 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:03.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:03.088234+0000) 2024-03-06T19:00:04.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:04.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:04.088460+0000) 2024-03-06T19:00:05.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:05.087+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:05.088638+0000) 2024-03-06T19:00:06.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:06.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:06.088866+0000) 2024-03-06T19:00:07.083 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:07.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:07.089046+0000) 2024-03-06T19:00:08.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:08.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:08.089284+0000) 2024-03-06T19:00:09.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:09.088+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:09.089511+0000) 2024-03-06T19:00:10.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:10.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:10.089777+0000) 2024-03-06T19:00:11.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:11.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:11.089978+0000) 2024-03-06T19:00:12.084 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:12.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:12.090203+0000) 2024-03-06T19:00:13.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:13.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:13.090415+0000) 2024-03-06T19:00:14.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:14.089+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:14.090590+0000) 2024-03-06T19:00:14.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:14.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:15.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:15.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:15.090843+0000) 2024-03-06T19:00:16.085 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:16.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:16.091102+0000) 2024-03-06T19:00:17.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:17.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:17.091348+0000) 2024-03-06T19:00:18.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:18.090+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:18.091576+0000) 2024-03-06T19:00:19.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:19.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:19.091818+0000) 2024-03-06T19:00:20.086 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:20.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:20.092083+0000) 2024-03-06T19:00:21.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:21.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:21.092362+0000) 2024-03-06T19:00:22.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:22.091+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:22.092635+0000) 2024-03-06T19:00:23.087 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:23.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:23.092965+0000) 2024-03-06T19:00:24.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:24.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:24.093241+0000) 2024-03-06T19:00:25.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:25.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:25.093517+0000) 2024-03-06T19:00:26.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:26.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:26.093822+0000) 2024-03-06T19:00:27.088 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:27.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:27.094090+0000) 2024-03-06T19:00:28.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:28.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:28.094324+0000) 2024-03-06T19:00:29.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:29.092+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:29.094523+0000) 2024-03-06T19:00:30.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:30.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:30.094820+0000) 2024-03-06T19:00:31.089 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:31.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:31.095097+0000) 2024-03-06T19:00:32.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:32.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:32.095324+0000) 2024-03-06T19:00:33.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:33.093+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:33.095562+0000) 2024-03-06T19:00:34.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:34.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:34.095789+0000) 2024-03-06T19:00:35.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:35.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:35.096046+0000) 2024-03-06T19:00:36.090 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:36.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:36.096282+0000) 2024-03-06T19:00:37.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:37.094+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:37.096507+0000) 2024-03-06T19:00:38.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:38.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:38.096784+0000) 2024-03-06T19:00:39.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:39.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:39.096988+0000) 2024-03-06T19:00:40.091 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:40.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:40.097224+0000) 2024-03-06T19:00:41.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:41.095+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:41.097457+0000) 2024-03-06T19:00:42.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:42.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:42.097705+0000) 2024-03-06T19:00:43.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:43.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:43.097955+0000) 2024-03-06T19:00:44.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:44.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:44.098154+0000) 2024-03-06T19:00:45.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:45.060 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:45.092 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:45.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:45.098311+0000) 2024-03-06T19:00:46.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:46.097+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:46.098490+0000) 2024-03-06T19:00:47.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:47.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:47.098761+0000) 2024-03-06T19:00:48.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:48.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:48.098994+0000) 2024-03-06T19:00:49.093 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:49.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:49.099295+0000) 2024-03-06T19:00:50.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:50.098+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:50.099471+0000) 2024-03-06T19:00:51.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:51.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:51.099733+0000) 2024-03-06T19:00:52.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:52.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:52.099909+0000) 2024-03-06T19:00:53.094 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:53.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:53.100149+0000) 2024-03-06T19:00:54.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:54.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:54.100385+0000) 2024-03-06T19:00:55.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:55.099+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:55.100610+0000) 2024-03-06T19:00:56.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:56.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:56.100840+0000) 2024-03-06T19:00:57.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:57.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:57.101019+0000) 2024-03-06T19:00:58.095 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:58.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:58.101209+0000) 2024-03-06T19:00:59.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:00:59.100+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:59.101434+0000) 2024-03-06T19:01:00.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:00.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:00.101653+0000) 2024-03-06T19:01:01.096 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:01.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:01.101881+0000) 2024-03-06T19:01:02.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:02.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:02.102075+0000) 2024-03-06T19:01:03.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:03.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:03.102273+0000) 2024-03-06T19:01:04.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:04.101+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:04.102514+0000) 2024-03-06T19:01:05.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:05.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:05.102781+0000) 2024-03-06T19:01:06.097 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:06.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:06.103010+0000) 2024-03-06T19:01:07.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:07.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:07.103233+0000) 2024-03-06T19:01:08.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:08.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:08.103413+0000) 2024-03-06T19:01:09.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:09.102+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:09.103642+0000) 2024-03-06T19:01:10.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:10.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:10.103907+0000) 2024-03-06T19:01:11.098 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:11.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:11.104129+0000) 2024-03-06T19:01:12.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:12.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:12.104350+0000) 2024-03-06T19:01:13.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:13.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:13.104631+0000) 2024-03-06T19:01:14.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:14.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:14.104881+0000) 2024-03-06T19:01:15.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:15.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:15.105087+0000) 2024-03-06T19:01:15.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:15.273 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:16.099 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:16.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:16.105280+0000) 2024-03-06T19:01:17.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:17.103+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:17.105476+0000) 2024-03-06T19:01:18.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:18.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:18.105743+0000) 2024-03-06T19:01:19.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:19.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:19.105934+0000) 2024-03-06T19:01:20.100 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:20.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:20.106113+0000) 2024-03-06T19:01:21.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:21.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:21.106302+0000) 2024-03-06T19:01:22.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:22.104+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:22.106496+0000) 2024-03-06T19:01:23.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:23.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:23.106775+0000) 2024-03-06T19:01:24.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:24.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:24.106997+0000) 2024-03-06T19:01:25.101 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:25.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:25.107218+0000) 2024-03-06T19:01:26.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:26.105+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:26.107458+0000) 2024-03-06T19:01:27.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:27.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:27.107715+0000) 2024-03-06T19:01:28.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:28.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:28.107956+0000) 2024-03-06T19:01:29.102 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:29.106+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:29.108193+0000) 2024-03-06T19:01:30.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:30.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:30.108407+0000) 2024-03-06T19:01:31.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:31.107+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:31.108624+0000) 2024-03-06T19:01:32.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:32.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:32.108861+0000) 2024-03-06T19:01:33.103 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:33.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:33.109134+0000) 2024-03-06T19:01:34.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:34.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:34.109383+0000) 2024-03-06T19:01:35.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:35.108+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:35.109602+0000) 2024-03-06T19:01:36.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:36.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:36.109839+0000) 2024-03-06T19:01:37.104 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:37.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:37.110121+0000) 2024-03-06T19:01:38.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:38.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:38.110350+0000) 2024-03-06T19:01:39.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:39.109+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:39.110580+0000) 2024-03-06T19:01:40.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:40.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:40.110827+0000) 2024-03-06T19:01:41.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:41.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:41.111058+0000) 2024-03-06T19:01:42.105 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:42.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:42.111304+0000) 2024-03-06T19:01:43.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:43.110+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:43.111536+0000) 2024-03-06T19:01:44.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:44.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:44.111814+0000) 2024-03-06T19:01:45.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:45.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:45.112069+0000) 2024-03-06T19:01:45.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:45.484 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:46.106 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:46.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:46.112318+0000) 2024-03-06T19:01:47.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:47.111+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:47.112506+0000) 2024-03-06T19:01:48.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:48.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:48.112788+0000) 2024-03-06T19:01:49.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:49.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:49.113021+0000) 2024-03-06T19:01:50.107 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:50.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:50.113215+0000) 2024-03-06T19:01:51.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:51.112+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:51.113449+0000) 2024-03-06T19:01:52.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:52.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:52.113700+0000) 2024-03-06T19:01:53.108 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:53.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:53.113897+0000) 2024-03-06T19:01:54.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:54.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:54.114116+0000) 2024-03-06T19:01:55.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:55.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:55.114297+0000) 2024-03-06T19:01:56.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:56.113+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:56.114518+0000) 2024-03-06T19:01:57.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:57.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:57.114740+0000) 2024-03-06T19:01:58.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:58.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:58.114983+0000) 2024-03-06T19:01:59.109 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:01:59.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:59.115210+0000) 2024-03-06T19:02:00.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:00.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:00.115434+0000) 2024-03-06T19:02:01.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:01.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:01.115661+0000) 2024-03-06T19:02:02.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:02.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:02.115911+0000) 2024-03-06T19:02:03.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:03.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:03.116149+0000) 2024-03-06T19:02:04.110 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:04.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:04.116392+0000) 2024-03-06T19:02:05.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:05.114+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:05.116577+0000) 2024-03-06T19:02:06.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:06.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:06.116793+0000) 2024-03-06T19:02:07.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:07.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:07.117023+0000) 2024-03-06T19:02:08.111 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:08.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:08.117201+0000) 2024-03-06T19:02:09.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:09.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:09.117395+0000) 2024-03-06T19:02:10.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:10.115+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:10.117643+0000) 2024-03-06T19:02:11.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:11.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:11.117944+0000) 2024-03-06T19:02:12.112 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:12.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:12.118192+0000) 2024-03-06T19:02:13.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:13.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:13.118429+0000) 2024-03-06T19:02:14.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:14.116+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:14.118627+0000) 2024-03-06T19:02:15.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:15.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:15.118822+0000) 2024-03-06T19:02:15.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:15.672 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:16.113 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:16.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:16.119020+0000) 2024-03-06T19:02:17.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:17.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:17.119249+0000) 2024-03-06T19:02:18.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:18.117+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:18.119518+0000) 2024-03-06T19:02:19.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:19.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:19.119745+0000) 2024-03-06T19:02:20.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:20.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:20.119937+0000) 2024-03-06T19:02:21.114 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:21.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:21.120126+0000) 2024-03-06T19:02:22.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:22.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:22.120315+0000) 2024-03-06T19:02:23.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:23.119+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:23.120584+0000) 2024-03-06T19:02:24.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:24.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:24.120785+0000) 2024-03-06T19:02:25.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:25.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:25.121036+0000) 2024-03-06T19:02:26.115 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:26.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:26.121260+0000) 2024-03-06T19:02:27.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:27.120+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:27.121465+0000) 2024-03-06T19:02:28.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:28.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:28.121715+0000) 2024-03-06T19:02:29.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:29.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:29.121914+0000) 2024-03-06T19:02:30.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:30.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:30.122101+0000) 2024-03-06T19:02:31.116 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:31.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:31.122289+0000) 2024-03-06T19:02:32.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:32.121+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:32.122507+0000) 2024-03-06T19:02:33.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:33.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:33.122738+0000) 2024-03-06T19:02:34.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:34.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:34.122989+0000) 2024-03-06T19:02:35.117 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:35.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:35.123217+0000) 2024-03-06T19:02:36.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:36.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:36.123443+0000) 2024-03-06T19:02:37.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:37.122+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:37.123626+0000) 2024-03-06T19:02:38.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:38.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:38.123825+0000) 2024-03-06T19:02:39.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:39.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:39.124047+0000) 2024-03-06T19:02:40.118 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:40.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:40.124275+0000) 2024-03-06T19:02:41.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:41.123+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:41.124484+0000) 2024-03-06T19:02:42.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:42.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:42.124722+0000) 2024-03-06T19:02:43.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:43.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:43.124912+0000) 2024-03-06T19:02:44.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:44.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:44.125117+0000) 2024-03-06T19:02:45.119 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:45.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:45.125363+0000) 2024-03-06T19:02:45.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:45.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:46.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:46.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:46.125613+0000) 2024-03-06T19:02:47.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:47.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:47.125817+0000) 2024-03-06T19:02:48.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:48.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:48.126046+0000) 2024-03-06T19:02:49.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:49.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:49.126239+0000) 2024-03-06T19:02:50.120 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:50.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:50.126427+0000) 2024-03-06T19:02:51.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:51.124+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:51.126619+0000) 2024-03-06T19:02:52.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:52.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:52.126812+0000) 2024-03-06T19:02:53.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:53.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:53.127050+0000) 2024-03-06T19:02:54.121 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:54.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:54.127290+0000) 2024-03-06T19:02:55.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:55.125+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:55.127463+0000) 2024-03-06T19:02:56.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:56.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:56.127712+0000) 2024-03-06T19:02:57.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:57.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:57.127945+0000) 2024-03-06T19:02:58.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:58.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:58.128148+0000) 2024-03-06T19:02:59.122 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:02:59.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:59.128394+0000) 2024-03-06T19:03:00.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:00.126+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:00.128591+0000) 2024-03-06T19:03:01.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:01.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:01.128814+0000) 2024-03-06T19:03:02.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:02.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:02.129015+0000) 2024-03-06T19:03:03.123 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:03.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:03.129226+0000) 2024-03-06T19:03:04.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:04.127+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:04.129439+0000) 2024-03-06T19:03:05.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:05.128+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:05.129658+0000) 2024-03-06T19:03:06.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:06.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:06.129914+0000) 2024-03-06T19:03:07.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:07.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:07.130097+0000) 2024-03-06T19:03:08.124 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:08.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:08.130287+0000) 2024-03-06T19:03:09.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:09.129+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:09.130470+0000) 2024-03-06T19:03:10.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:10.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:10.130715+0000) 2024-03-06T19:03:11.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:11.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:11.130903+0000) 2024-03-06T19:03:12.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:12.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:12.131090+0000) 2024-03-06T19:03:13.125 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:13.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:13.131337+0000) 2024-03-06T19:03:14.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:14.130+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:14.131546+0000) 2024-03-06T19:03:15.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:15.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:15.131771+0000) 2024-03-06T19:03:16.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:16.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:16.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:16.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:16.131976+0000) 2024-03-06T19:03:17.126 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:17.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:17.132193+0000) 2024-03-06T19:03:18.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:18.131+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:18.132471+0000) 2024-03-06T19:03:19.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:19.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:19.132742+0000) 2024-03-06T19:03:20.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:20.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:20.132972+0000) 2024-03-06T19:03:21.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:21.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:21.133189+0000) 2024-03-06T19:03:22.127 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:22.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:22.133376+0000) 2024-03-06T19:03:23.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:23.132+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:23.133572+0000) 2024-03-06T19:03:24.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:24.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:24.133785+0000) 2024-03-06T19:03:25.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:25.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:25.133983+0000) 2024-03-06T19:03:26.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:26.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:26.134205+0000) 2024-03-06T19:03:27.128 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:27.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:27.134384+0000) 2024-03-06T19:03:28.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:28.133+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:28.134604+0000) 2024-03-06T19:03:29.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:29.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:29.134836+0000) 2024-03-06T19:03:30.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:30.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:30.135025+0000) 2024-03-06T19:03:31.129 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:31.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:31.135215+0000) 2024-03-06T19:03:32.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:32.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:32.135451+0000) 2024-03-06T19:03:33.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:33.134+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:33.135627+0000) 2024-03-06T19:03:34.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:34.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:34.135818+0000) 2024-03-06T19:03:35.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:35.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:35.136005+0000) 2024-03-06T19:03:36.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:36.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:36.136210+0000) 2024-03-06T19:03:37.130 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:37.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:37.136443+0000) 2024-03-06T19:03:38.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:38.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:38.136691+0000) 2024-03-06T19:03:39.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:39.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:39.136929+0000) 2024-03-06T19:03:40.131 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:40.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:40.137144+0000) 2024-03-06T19:03:41.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:41.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:41.137344+0000) 2024-03-06T19:03:42.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:42.135+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:42.137568+0000) 2024-03-06T19:03:43.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:43.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:43.137768+0000) 2024-03-06T19:03:44.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:44.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:44.138006+0000) 2024-03-06T19:03:45.132 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:45.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:45.138197+0000) 2024-03-06T19:03:46.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:46.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:46.138388+0000) 2024-03-06T19:03:46.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:46.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:47.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:47.136+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:47.138613+0000) 2024-03-06T19:03:48.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:48.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:48.138815+0000) 2024-03-06T19:03:49.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:49.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:49.139012+0000) 2024-03-06T19:03:50.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:50.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:50.139201+0000) 2024-03-06T19:03:51.133 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:51.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:51.139421+0000) 2024-03-06T19:03:52.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:52.137+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:52.139645+0000) 2024-03-06T19:03:53.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:53.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:53.139885+0000) 2024-03-06T19:03:54.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:54.138+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:54.140103+0000) 2024-03-06T19:03:55.134 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:55.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:55.140334+0000) 2024-03-06T19:03:56.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:56.139+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:56.140535+0000) 2024-03-06T19:03:57.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:57.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:57.140802+0000) 2024-03-06T19:03:58.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:58.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:58.140995+0000) 2024-03-06T19:03:59.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:03:59.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:59.141199+0000) 2024-03-06T19:04:00.135 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:00.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:00.141427+0000) 2024-03-06T19:04:01.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:01.140+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:01.141649+0000) 2024-03-06T19:04:02.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:02.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:02.141828+0000) 2024-03-06T19:04:03.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:03.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:03.142033+0000) 2024-03-06T19:04:04.136 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:04.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:04.142274+0000) 2024-03-06T19:04:05.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:05.141+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:05.142527+0000) 2024-03-06T19:04:06.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:06.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:06.142780+0000) 2024-03-06T19:04:07.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:07.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:07.143014+0000) 2024-03-06T19:04:08.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:08.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:08.143205+0000) 2024-03-06T19:04:09.137 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:09.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:09.143398+0000) 2024-03-06T19:04:10.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:10.142+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:10.143623+0000) 2024-03-06T19:04:11.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:11.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:11.143824+0000) 2024-03-06T19:04:12.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:12.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:12.144023+0000) 2024-03-06T19:04:13.138 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:13.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:13.144241+0000) 2024-03-06T19:04:14.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:14.143+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:14.144474+0000) 2024-03-06T19:04:15.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:15.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:15.144721+0000) 2024-03-06T19:04:16.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:16.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:16.144899+0000) 2024-03-06T19:04:16.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:16.526 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:17.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:17.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:17.145100+0000) 2024-03-06T19:04:18.139 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:18.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:18.145339+0000) 2024-03-06T19:04:19.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:19.144+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:19.145568+0000) 2024-03-06T19:04:20.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:20.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:20.145810+0000) 2024-03-06T19:04:21.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:21.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:21.146042+0000) 2024-03-06T19:04:22.140 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:22.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:22.146249+0000) 2024-03-06T19:04:23.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:23.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:23.146536+0000) 2024-03-06T19:04:24.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:24.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:24.146798+0000) 2024-03-06T19:04:25.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:25.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:25.146989+0000) 2024-03-06T19:04:26.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:26.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:26.147175+0000) 2024-03-06T19:04:27.141 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:27.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:27.147359+0000) 2024-03-06T19:04:28.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:28.145+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:28.147591+0000) 2024-03-06T19:04:29.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:29.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:29.147817+0000) 2024-03-06T19:04:30.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:30.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:30.147998+0000) 2024-03-06T19:04:31.142 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:31.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:31.148221+0000) 2024-03-06T19:04:32.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:32.146+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:32.148449+0000) 2024-03-06T19:04:33.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:33.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:33.148723+0000) 2024-03-06T19:04:34.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:34.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:34.148928+0000) 2024-03-06T19:04:35.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:35.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:35.149117+0000) 2024-03-06T19:04:36.143 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:36.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:36.149343+0000) 2024-03-06T19:04:37.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:37.147+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:37.149573+0000) 2024-03-06T19:04:38.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:38.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:38.149822+0000) 2024-03-06T19:04:39.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:39.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:39.150075+0000) 2024-03-06T19:04:40.144 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:40.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:40.150287+0000) 2024-03-06T19:04:41.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:41.148+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:41.150530+0000) 2024-03-06T19:04:42.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:42.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:42.150780+0000) 2024-03-06T19:04:43.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:43.149+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:43.150974+0000) 2024-03-06T19:04:44.145 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:44.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:44.151214+0000) 2024-03-06T19:04:45.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:45.150+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:45.151439+0000) 2024-03-06T19:04:46.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:46.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:46.151679+0000) 2024-03-06T19:04:46.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:46.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:47.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:47.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:47.151884+0000) 2024-03-06T19:04:48.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:48.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:48.152071+0000) 2024-03-06T19:04:49.146 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:49.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:49.152276+0000) 2024-03-06T19:04:50.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:50.151+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:50.152501+0000) 2024-03-06T19:04:51.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:51.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:51.152717+0000) 2024-03-06T19:04:52.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:52.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:52.152920+0000) 2024-03-06T19:04:53.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:53.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:53.153111+0000) 2024-03-06T19:04:54.147 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:54.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:54.153367+0000) 2024-03-06T19:04:55.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:55.152+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:55.153592+0000) 2024-03-06T19:04:56.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:56.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:56.153816+0000) 2024-03-06T19:04:57.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:57.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:57.154007+0000) 2024-03-06T19:04:58.148 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:58.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:58.154217+0000) 2024-03-06T19:04:59.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:04:59.153+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:59.154502+0000) 2024-03-06T19:05:00.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:00.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:00.154727+0000) 2024-03-06T19:05:01.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:01.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:01.154955+0000) 2024-03-06T19:05:02.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:02.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:02.155143+0000) 2024-03-06T19:05:03.149 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:03.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:03.155375+0000) 2024-03-06T19:05:04.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:04.154+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:04.155620+0000) 2024-03-06T19:05:05.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:05.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:05.155791+0000) 2024-03-06T19:05:06.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:06.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:06.156024+0000) 2024-03-06T19:05:07.150 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:07.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:07.156259+0000) 2024-03-06T19:05:08.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:08.155+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:08.156491+0000) 2024-03-06T19:05:09.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:09.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:09.156776+0000) 2024-03-06T19:05:10.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:10.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:10.157006+0000) 2024-03-06T19:05:11.151 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:11.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:11.157215+0000) 2024-03-06T19:05:12.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:12.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:12.157440+0000) 2024-03-06T19:05:13.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:13.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:13.157699+0000) 2024-03-06T19:05:14.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:14.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:14.157904+0000) 2024-03-06T19:05:15.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:15.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:15.158092+0000) 2024-03-06T19:05:16.152 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:16.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:16.158337+0000) 2024-03-06T19:05:16.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:16.949 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:17.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:17.156+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:17.158570+0000) 2024-03-06T19:05:18.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:18.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:18.158836+0000) 2024-03-06T19:05:19.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:19.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:19.159081+0000) 2024-03-06T19:05:20.153 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:20.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:20.159336+0000) 2024-03-06T19:05:21.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:21.157+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:21.159559+0000) 2024-03-06T19:05:22.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:22.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:22.159856+0000) 2024-03-06T19:05:23.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:23.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:23.160078+0000) 2024-03-06T19:05:24.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:24.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:24.160271+0000) 2024-03-06T19:05:25.154 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:25.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:25.160456+0000) 2024-03-06T19:05:26.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:26.158+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:26.160651+0000) 2024-03-06T19:05:27.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:27.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:27.160877+0000) 2024-03-06T19:05:28.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:28.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:28.161059+0000) 2024-03-06T19:05:29.155 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:29.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:29.161290+0000) 2024-03-06T19:05:30.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:30.159+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:30.161523+0000) 2024-03-06T19:05:31.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:31.160+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:31.161786+0000) 2024-03-06T19:05:32.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:32.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:32.162051+0000) 2024-03-06T19:05:33.156 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:33.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:33.162343+0000) 2024-03-06T19:05:34.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:34.161+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:34.162617+0000) 2024-03-06T19:05:35.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:35.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:35.162933+0000) 2024-03-06T19:05:36.157 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:36.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:36.163213+0000) 2024-03-06T19:05:37.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:37.162+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:37.163436+0000) 2024-03-06T19:05:38.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:38.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:38.163766+0000) 2024-03-06T19:05:39.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:39.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:39.164031+0000) 2024-03-06T19:05:40.158 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:40.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:40.164236+0000) 2024-03-06T19:05:41.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:41.163+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:41.164499+0000) 2024-03-06T19:05:42.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:42.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:42.164793+0000) 2024-03-06T19:05:43.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:43.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:43.165004+0000) 2024-03-06T19:05:44.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:44.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:44.165241+0000) 2024-03-06T19:05:45.159 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:45.164+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:45.165467+0000) 2024-03-06T19:05:46.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:46.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:46.165733+0000) 2024-03-06T19:05:47.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:47.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:47.165956+0000) 2024-03-06T19:05:47.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:47.173 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:48.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:48.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:48.166228+0000) 2024-03-06T19:05:49.160 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:49.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:49.166472+0000) 2024-03-06T19:05:50.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:50.165+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:50.166635+0000) 2024-03-06T19:05:51.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:51.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:51.166847+0000) 2024-03-06T19:05:52.161 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:52.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:52.167076+0000) 2024-03-06T19:05:53.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:53.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:53.167319+0000) 2024-03-06T19:05:54.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:54.166+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:54.167550+0000) 2024-03-06T19:05:55.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:55.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:55.167780+0000) 2024-03-06T19:05:56.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:56.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:56.167982+0000) 2024-03-06T19:05:57.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:57.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:57.168217+0000) 2024-03-06T19:05:58.162 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:58.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:58.168397+0000) 2024-03-06T19:05:59.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:05:59.167+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:59.168624+0000) 2024-03-06T19:06:00.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:00.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:00.168821+0000) 2024-03-06T19:06:01.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:01.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:01.169045+0000) 2024-03-06T19:06:02.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:02.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:02.169236+0000) 2024-03-06T19:06:03.163 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:03.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:03.169441+0000) 2024-03-06T19:06:04.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:04.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:04.169707+0000) 2024-03-06T19:06:05.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:05.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:05.169910+0000) 2024-03-06T19:06:06.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:06.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:06.170144+0000) 2024-03-06T19:06:07.164 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:07.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:07.170379+0000) 2024-03-06T19:06:08.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:08.168+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:08.170605+0000) 2024-03-06T19:06:09.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:09.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:09.170849+0000) 2024-03-06T19:06:10.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:10.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:10.171063+0000) 2024-03-06T19:06:11.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:11.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:11.171317+0000) 2024-03-06T19:06:12.165 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:12.169+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:12.171508+0000) 2024-03-06T19:06:13.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:13.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:13.171768+0000) 2024-03-06T19:06:14.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:14.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:14.172003+0000) 2024-03-06T19:06:15.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:15.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:15.172284+0000) 2024-03-06T19:06:16.166 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:16.170+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:16.172479+0000) 2024-03-06T19:06:17.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:17.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:17.172749+0000) 2024-03-06T19:06:17.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:17.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:18.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:18.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:18.172955+0000) 2024-03-06T19:06:19.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:19.171+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:19.173136+0000) 2024-03-06T19:06:20.167 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:20.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:20.173366+0000) 2024-03-06T19:06:21.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:21.172+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:21.173585+0000) 2024-03-06T19:06:22.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:22.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:22.173820+0000) 2024-03-06T19:06:23.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:23.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:23.174047+0000) 2024-03-06T19:06:24.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:24.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:24.174243+0000) 2024-03-06T19:06:25.168 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:25.173+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:25.174472+0000) 2024-03-06T19:06:26.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:26.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:26.174727+0000) 2024-03-06T19:06:27.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:27.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:27.174929+0000) 2024-03-06T19:06:28.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:28.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:28.175174+0000) 2024-03-06T19:06:29.169 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:29.174+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:29.175413+0000) 2024-03-06T19:06:30.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:30.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:30.175702+0000) 2024-03-06T19:06:31.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:31.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:31.175907+0000) 2024-03-06T19:06:32.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:32.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:32.176115+0000) 2024-03-06T19:06:33.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:33.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:33.176343+0000) 2024-03-06T19:06:34.170 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:34.175+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:34.176533+0000) 2024-03-06T19:06:35.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:35.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:35.176756+0000) 2024-03-06T19:06:36.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:36.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:36.177031+0000) 2024-03-06T19:06:37.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:37.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:37.177256+0000) 2024-03-06T19:06:38.171 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:38.176+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:38.177480+0000) 2024-03-06T19:06:39.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:39.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:39.177758+0000) 2024-03-06T19:06:40.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:40.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:40.177991+0000) 2024-03-06T19:06:41.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:41.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:41.178228+0000) 2024-03-06T19:06:42.172 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:42.177+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:42.178460+0000) 2024-03-06T19:06:43.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:43.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:43.178723+0000) 2024-03-06T19:06:44.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:44.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:44.178959+0000) 2024-03-06T19:06:45.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:45.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:45.179178+0000) 2024-03-06T19:06:46.173 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:46.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:46.179408+0000) 2024-03-06T19:06:47.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:47.178+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:47.179632+0000) 2024-03-06T19:06:47.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:47.569 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:48.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:48.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:48.179868+0000) 2024-03-06T19:06:49.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:49.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:49.180024+0000) 2024-03-06T19:06:50.174 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:50.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:50.180311+0000) 2024-03-06T19:06:51.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:51.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:51.180508+0000) 2024-03-06T19:06:52.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:52.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:52.180765+0000) 2024-03-06T19:06:53.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:53.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:53.180997+0000) 2024-03-06T19:06:54.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:54.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:54.181221+0000) 2024-03-06T19:06:55.175 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:55.179+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:55.181448+0000) 2024-03-06T19:06:56.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:56.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:56.181661+0000) 2024-03-06T19:06:57.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:57.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:57.181940+0000) 2024-03-06T19:06:58.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:58.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:58.182178+0000) 2024-03-06T19:06:59.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:06:59.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:59.182368+0000) 2024-03-06T19:07:00.176 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:00.180+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:00.182605+0000) 2024-03-06T19:07:01.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:01.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:01.182831+0000) 2024-03-06T19:07:02.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:02.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:02.183034+0000) 2024-03-06T19:07:03.177 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:03.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:03.183264+0000) 2024-03-06T19:07:04.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:04.181+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:04.183444+0000) 2024-03-06T19:07:05.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:05.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:05.183695+0000) 2024-03-06T19:07:06.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:06.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:06.183884+0000) 2024-03-06T19:07:07.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:07.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:07.184145+0000) 2024-03-06T19:07:08.178 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:08.182+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:08.184409+0000) 2024-03-06T19:07:09.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:09.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:09.184691+0000) 2024-03-06T19:07:10.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:10.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:10.184904+0000) 2024-03-06T19:07:11.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:11.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:11.185126+0000) 2024-03-06T19:07:12.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:12.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:12.185337+0000) 2024-03-06T19:07:13.179 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:13.184+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:13.185561+0000) 2024-03-06T19:07:14.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:14.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:14.185794+0000) 2024-03-06T19:07:15.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:15.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:15.186031+0000) 2024-03-06T19:07:16.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:16.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:16.186260+0000) 2024-03-06T19:07:17.180 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:17.185+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:17.186478+0000) 2024-03-06T19:07:17.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:17.706 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:18.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:18.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:18.186719+0000) 2024-03-06T19:07:19.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:19.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:19.186945+0000) 2024-03-06T19:07:20.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:20.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:20.187198+0000) 2024-03-06T19:07:21.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:21.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:21.187442+0000) 2024-03-06T19:07:22.181 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:22.186+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:22.187646+0000) 2024-03-06T19:07:23.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:23.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:23.187887+0000) 2024-03-06T19:07:24.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:24.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:24.188082+0000) 2024-03-06T19:07:25.182 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:25.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:25.188315+0000) 2024-03-06T19:07:26.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:26.187+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:26.188590+0000) 2024-03-06T19:07:27.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:27.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:27.188857+0000) 2024-03-06T19:07:28.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:28.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:28.189089+0000) 2024-03-06T19:07:29.183 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:29.188+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:29.189383+0000) 2024-03-06T19:07:30.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:30.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:30.189654+0000) 2024-03-06T19:07:31.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:31.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:31.189925+0000) 2024-03-06T19:07:32.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:32.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:32.190136+0000) 2024-03-06T19:07:33.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:33.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:33.190376+0000) 2024-03-06T19:07:34.184 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:34.189+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:34.190639+0000) 2024-03-06T19:07:35.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:35.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:35.190929+0000) 2024-03-06T19:07:36.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:36.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:36.191152+0000) 2024-03-06T19:07:37.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:37.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:37.191383+0000) 2024-03-06T19:07:38.185 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:38.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:38.191586+0000) 2024-03-06T19:07:39.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:39.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:39.191790+0000) 2024-03-06T19:07:40.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:40.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:40.192009+0000) 2024-03-06T19:07:41.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:41.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:41.192255+0000) 2024-03-06T19:07:42.186 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:42.190+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:42.192533+0000) 2024-03-06T19:07:43.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:43.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:43.192817+0000) 2024-03-06T19:07:44.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:44.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:44.193048+0000) 2024-03-06T19:07:45.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:45.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:45.193290+0000) 2024-03-06T19:07:46.187 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:46.191+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:46.193569+0000) 2024-03-06T19:07:47.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:47.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:47.193856+0000) 2024-03-06T19:07:47.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:47.904 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:48.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:48.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:48.194162+0000) 2024-03-06T19:07:49.188 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:49.192+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:49.194445+0000) 2024-03-06T19:07:50.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:50.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:50.194719+0000) 2024-03-06T19:07:51.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:51.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:51.194908+0000) 2024-03-06T19:07:52.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:52.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:52.195097+0000) 2024-03-06T19:07:53.189 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:53.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:53.195318+0000) 2024-03-06T19:07:54.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:54.193+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:54.195560+0000) 2024-03-06T19:07:55.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:55.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:55.195840+0000) 2024-03-06T19:07:56.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:56.194+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:56.196099+0000) 2024-03-06T19:07:57.190 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:57.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:57.196341+0000) 2024-03-06T19:07:58.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:58.195+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:58.196619+0000) 2024-03-06T19:07:59.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:07:59.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:59.196861+0000) 2024-03-06T19:08:00.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:00.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:00.197106+0000) 2024-03-06T19:08:01.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:01.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:01.197295+0000) 2024-03-06T19:08:02.191 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:02.196+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:02.197531+0000) 2024-03-06T19:08:03.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:03.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:03.197798+0000) 2024-03-06T19:08:04.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:04.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:04.198075+0000) 2024-03-06T19:08:05.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:05.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:05.198304+0000) 2024-03-06T19:08:06.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:06.197+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:06.198572+0000) 2024-03-06T19:08:07.192 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:07.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:07.198850+0000) 2024-03-06T19:08:08.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:08.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:08.199131+0000) 2024-03-06T19:08:09.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:09.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:09.199363+0000) 2024-03-06T19:08:10.193 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:10.198+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:10.199636+0000) 2024-03-06T19:08:11.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:11.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:11.199883+0000) 2024-03-06T19:08:12.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:12.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:12.200111+0000) 2024-03-06T19:08:13.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:13.199+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:13.200401+0000) 2024-03-06T19:08:14.194 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:14.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:14.200708+0000) 2024-03-06T19:08:15.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:15.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:15.200978+0000) 2024-03-06T19:08:16.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:16.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:16.201218+0000) 2024-03-06T19:08:17.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:17.200+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:17.201540+0000) 2024-03-06T19:08:18.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:18.074 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:18.195 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:18.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:18.201831+0000) 2024-03-06T19:08:19.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:19.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:19.202117+0000) 2024-03-06T19:08:20.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:20.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:20.202333+0000) 2024-03-06T19:08:21.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:21.201+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:21.202572+0000) 2024-03-06T19:08:22.196 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:22.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:22.202819+0000) 2024-03-06T19:08:23.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:23.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:23.203006+0000) 2024-03-06T19:08:24.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:24.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:24.203222+0000) 2024-03-06T19:08:25.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:25.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:25.203464+0000) 2024-03-06T19:08:26.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:26.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:26.203687+0000) 2024-03-06T19:08:27.197 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:27.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:27.203924+0000) 2024-03-06T19:08:28.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:28.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:28.204160+0000) 2024-03-06T19:08:29.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:29.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:29.204378+0000) 2024-03-06T19:08:30.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:30.202+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:30.204608+0000) 2024-03-06T19:08:31.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:31.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:31.204846+0000) 2024-03-06T19:08:32.198 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:32.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:32.205069+0000) 2024-03-06T19:08:33.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:33.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:33.205304+0000) 2024-03-06T19:08:34.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:34.203+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:34.205533+0000) 2024-03-06T19:08:35.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:35.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:35.205780+0000) 2024-03-06T19:08:36.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:36.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:36.206005+0000) 2024-03-06T19:08:37.199 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:37.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:37.206185+0000) 2024-03-06T19:08:38.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:38.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:38.206420+0000) 2024-03-06T19:08:39.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:39.204+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:39.206649+0000) 2024-03-06T19:08:40.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:40.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:40.206912+0000) 2024-03-06T19:08:41.200 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:41.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:41.207107+0000) 2024-03-06T19:08:42.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:42.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:42.207297+0000) 2024-03-06T19:08:43.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:43.205+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:43.207534+0000) 2024-03-06T19:08:44.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:44.206+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:44.207786+0000) 2024-03-06T19:08:45.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:45.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:45.207978+0000) 2024-03-06T19:08:46.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:46.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:46.208169+0000) 2024-03-06T19:08:47.201 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:47.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:47.208389+0000) 2024-03-06T19:08:48.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:48.207+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:48.208611+0000) 2024-03-06T19:08:48.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:48.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:49.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:49.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:49.208824+0000) 2024-03-06T19:08:50.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:50.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:50.209028+0000) 2024-03-06T19:08:51.202 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:51.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:51.209237+0000) 2024-03-06T19:08:52.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:52.208+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:52.209459+0000) 2024-03-06T19:08:53.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:53.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:53.209742+0000) 2024-03-06T19:08:54.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:54.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:54.209944+0000) 2024-03-06T19:08:55.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:55.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:55.210145+0000) 2024-03-06T19:08:56.203 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:56.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:56.210369+0000) 2024-03-06T19:08:57.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:57.209+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:57.210595+0000) 2024-03-06T19:08:58.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:58.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:58.210830+0000) 2024-03-06T19:08:59.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:08:59.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:59.211044+0000) 2024-03-06T19:09:00.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:00.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:00.211243+0000) 2024-03-06T19:09:01.204 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:01.210+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:01.211506+0000) 2024-03-06T19:09:02.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:02.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:02.211758+0000) 2024-03-06T19:09:03.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:03.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:03.211959+0000) 2024-03-06T19:09:04.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:04.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:04.212177+0000) 2024-03-06T19:09:05.205 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:05.211+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:05.212466+0000) 2024-03-06T19:09:06.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:06.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:06.212736+0000) 2024-03-06T19:09:07.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:07.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:07.212925+0000) 2024-03-06T19:09:08.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:08.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:08.213116+0000) 2024-03-06T19:09:09.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:09.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:09.213301+0000) 2024-03-06T19:09:10.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:10.212+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:10.213512+0000) 2024-03-06T19:09:11.206 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:11.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:11.213772+0000) 2024-03-06T19:09:12.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:12.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:12.213980+0000) 2024-03-06T19:09:13.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:13.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:13.214181+0000) 2024-03-06T19:09:14.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:14.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:14.214410+0000) 2024-03-06T19:09:15.207 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:15.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:15.214636+0000) 2024-03-06T19:09:16.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:16.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:16.214884+0000) 2024-03-06T19:09:17.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:17.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:17.215071+0000) 2024-03-06T19:09:18.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:18.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:18.215292+0000) 2024-03-06T19:09:18.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:18.406 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:19.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:19.213+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:19.215516+0000) 2024-03-06T19:09:20.208 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:20.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:20.215790+0000) 2024-03-06T19:09:21.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:21.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:21.216022+0000) 2024-03-06T19:09:22.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:22.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:22.216245+0000) 2024-03-06T19:09:23.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:23.214+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:23.216486+0000) 2024-03-06T19:09:24.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:24.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:24.216660+0000) 2024-03-06T19:09:25.209 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:25.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:25.216900+0000) 2024-03-06T19:09:26.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:26.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:26.217049+0000) 2024-03-06T19:09:27.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:27.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:27.217230+0000) 2024-03-06T19:09:28.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:28.215+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:28.217450+0000) 2024-03-06T19:09:29.210 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:29.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:29.217696+0000) 2024-03-06T19:09:30.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:30.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:30.217899+0000) 2024-03-06T19:09:31.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:31.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:31.218111+0000) 2024-03-06T19:09:32.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:32.216+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:32.218344+0000) 2024-03-06T19:09:33.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:33.217+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:33.218574+0000) 2024-03-06T19:09:34.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:34.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:34.218820+0000) 2024-03-06T19:09:35.211 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:35.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:35.219055+0000) 2024-03-06T19:09:36.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:36.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:36.219247+0000) 2024-03-06T19:09:37.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:37.218+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:37.219476+0000) 2024-03-06T19:09:38.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:38.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:38.219692+0000) 2024-03-06T19:09:39.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:39.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:39.219886+0000) 2024-03-06T19:09:40.212 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:40.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:40.220121+0000) 2024-03-06T19:09:41.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:41.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:41.220356+0000) 2024-03-06T19:09:42.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:42.219+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:42.220548+0000) 2024-03-06T19:09:43.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:43.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:43.220785+0000) 2024-03-06T19:09:44.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:44.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:44.221010+0000) 2024-03-06T19:09:45.213 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:45.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:45.221210+0000) 2024-03-06T19:09:46.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:46.220+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:46.221438+0000) 2024-03-06T19:09:47.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:47.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:47.221688+0000) 2024-03-06T19:09:48.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:48.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:48.221886+0000) 2024-03-06T19:09:48.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:48.600 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:49.214 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:49.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:49.222084+0000) 2024-03-06T19:09:50.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:50.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:50.222273+0000) 2024-03-06T19:09:51.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:51.221+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:51.222470+0000) 2024-03-06T19:09:52.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:52.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:52.222744+0000) 2024-03-06T19:09:53.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:53.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:53.222978+0000) 2024-03-06T19:09:54.215 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:54.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:54.223207+0000) 2024-03-06T19:09:55.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:55.222+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:55.223453+0000) 2024-03-06T19:09:56.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:56.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:56.223659+0000) 2024-03-06T19:09:57.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:57.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:57.223904+0000) 2024-03-06T19:09:58.216 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:58.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:58.224182+0000) 2024-03-06T19:09:59.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:09:59.223+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:59.224418+0000) 2024-03-06T19:10:00.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:00.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:00.224704+0000) 2024-03-06T19:10:01.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:01.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:01.224950+0000) 2024-03-06T19:10:02.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:02.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:02.225142+0000) 2024-03-06T19:10:03.217 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:03.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:03.225391+0000) 2024-03-06T19:10:04.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:04.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:04.225629+0000) 2024-03-06T19:10:05.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:05.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:05.225860+0000) 2024-03-06T19:10:06.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:06.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:06.226030+0000) 2024-03-06T19:10:07.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:07.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:07.226204+0000) 2024-03-06T19:10:08.218 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:08.224+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:08.226429+0000) 2024-03-06T19:10:09.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:09.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:09.226696+0000) 2024-03-06T19:10:10.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:10.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:10.226847+0000) 2024-03-06T19:10:11.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:11.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:11.227039+0000) 2024-03-06T19:10:12.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:12.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:12.227272+0000) 2024-03-06T19:10:13.219 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:13.225+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:13.227514+0000) 2024-03-06T19:10:14.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:14.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:14.227743+0000) 2024-03-06T19:10:15.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:15.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:15.227954+0000) 2024-03-06T19:10:16.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:16.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:16.228131+0000) 2024-03-06T19:10:17.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:17.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:17.228321+0000) 2024-03-06T19:10:18.220 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:18.226+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:18.228546+0000) 2024-03-06T19:10:18.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:18.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:19.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:19.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:19.228783+0000) 2024-03-06T19:10:20.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:20.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:20.228985+0000) 2024-03-06T19:10:21.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:21.227+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:21.229206+0000) 2024-03-06T19:10:22.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:22.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:22.229393+0000) 2024-03-06T19:10:23.221 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:23.228+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:23.229604+0000) 2024-03-06T19:10:24.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:24.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:24.229833+0000) 2024-03-06T19:10:25.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:25.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:25.230042+0000) 2024-03-06T19:10:26.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:26.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:26.230284+0000) 2024-03-06T19:10:27.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:27.229+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:27.230481+0000) 2024-03-06T19:10:28.222 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:28.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:28.230754+0000) 2024-03-06T19:10:29.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:29.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:29.230998+0000) 2024-03-06T19:10:30.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:30.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:30.231268+0000) 2024-03-06T19:10:31.223 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:31.230+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:31.231546+0000) 2024-03-06T19:10:32.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:32.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:32.231843+0000) 2024-03-06T19:10:33.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:33.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:33.232096+0000) 2024-03-06T19:10:34.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:34.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:34.232359+0000) 2024-03-06T19:10:35.224 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:35.231+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:35.232630+0000) 2024-03-06T19:10:36.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:36.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:36.232917+0000) 2024-03-06T19:10:37.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:37.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:37.233154+0000) 2024-03-06T19:10:38.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:38.232+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:38.233428+0000) 2024-03-06T19:10:39.225 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:39.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:39.233739+0000) 2024-03-06T19:10:40.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:40.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:40.233987+0000) 2024-03-06T19:10:41.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:41.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:41.234220+0000) 2024-03-06T19:10:42.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:42.233+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:42.234504+0000) 2024-03-06T19:10:43.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:43.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:43.234744+0000) 2024-03-06T19:10:44.226 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:44.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:44.234940+0000) 2024-03-06T19:10:45.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:45.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:45.235183+0000) 2024-03-06T19:10:46.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:46.234+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:46.235456+0000) 2024-03-06T19:10:47.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:47.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:47.235707+0000) 2024-03-06T19:10:48.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:48.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:48.235896+0000) 2024-03-06T19:10:48.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:48.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:49.227 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:49.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:49.236088+0000) 2024-03-06T19:10:50.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:50.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:50.236271+0000) 2024-03-06T19:10:51.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:51.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:51.236496+0000) 2024-03-06T19:10:52.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:52.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:52.236765+0000) 2024-03-06T19:10:53.228 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:53.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:53.236957+0000) 2024-03-06T19:10:54.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:54.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:54.237149+0000) 2024-03-06T19:10:55.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:55.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:55.237345+0000) 2024-03-06T19:10:56.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:56.235+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:56.237582+0000) 2024-03-06T19:10:57.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:57.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:57.237837+0000) 2024-03-06T19:10:58.229 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:58.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:58.238063+0000) 2024-03-06T19:10:59.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:10:59.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:59.238312+0000) 2024-03-06T19:11:00.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:00.236+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:00.238599+0000) 2024-03-06T19:11:01.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:01.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:01.238857+0000) 2024-03-06T19:11:02.230 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:02.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:02.239094+0000) 2024-03-06T19:11:03.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:03.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:03.239291+0000) 2024-03-06T19:11:04.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:04.237+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:04.239513+0000) 2024-03-06T19:11:05.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:05.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:05.239738+0000) 2024-03-06T19:11:06.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:06.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:06.239944+0000) 2024-03-06T19:11:07.231 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:07.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:07.240173+0000) 2024-03-06T19:11:08.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:08.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:08.240357+0000) 2024-03-06T19:11:09.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:09.238+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:09.240595+0000) 2024-03-06T19:11:10.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:10.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:10.240858+0000) 2024-03-06T19:11:11.232 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:11.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:11.241089+0000) 2024-03-06T19:11:12.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:12.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:12.241314+0000) 2024-03-06T19:11:13.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:13.240+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:13.241557+0000) 2024-03-06T19:11:14.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:14.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:14.241799+0000) 2024-03-06T19:11:15.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:15.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:15.242028+0000) 2024-03-06T19:11:16.233 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:16.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:16.242240+0000) 2024-03-06T19:11:17.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:17.241+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:17.242465+0000) 2024-03-06T19:11:18.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:18.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:18.242718+0000) 2024-03-06T19:11:19.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:19.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:19.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:19.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:19.242886+0000) 2024-03-06T19:11:20.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:20.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:20.243083+0000) 2024-03-06T19:11:21.234 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:21.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:21.243321+0000) 2024-03-06T19:11:22.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:22.242+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:22.243537+0000) 2024-03-06T19:11:23.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:23.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:23.243790+0000) 2024-03-06T19:11:24.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:24.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:24.243998+0000) 2024-03-06T19:11:25.235 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:25.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:25.244238+0000) 2024-03-06T19:11:26.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:26.243+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:26.244478+0000) 2024-03-06T19:11:27.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:27.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:27.244779+0000) 2024-03-06T19:11:28.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:28.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:28.245057+0000) 2024-03-06T19:11:29.236 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:29.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:29.245297+0000) 2024-03-06T19:11:30.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:30.244+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:30.245540+0000) 2024-03-06T19:11:31.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:31.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:31.245779+0000) 2024-03-06T19:11:32.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:32.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:32.245980+0000) 2024-03-06T19:11:33.237 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:33.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:33.246174+0000) 2024-03-06T19:11:34.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:34.245+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:34.246442+0000) 2024-03-06T19:11:35.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:35.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:35.246645+0000) 2024-03-06T19:11:36.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:36.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:36.246950+0000) 2024-03-06T19:11:37.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:37.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:37.247195+0000) 2024-03-06T19:11:38.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:38.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:38.247404+0000) 2024-03-06T19:11:39.238 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:39.246+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:39.247644+0000) 2024-03-06T19:11:40.239 INFO:tasks.ceph.mds.a.smithi114.stderr:2024-03-06T19:11:40.247+0000 7f73b0800640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:40.247920+0000) 2024-03-06T19:11:41.171 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-06T19:11:41.171 DEBUG:teuthology.task.console_log:Killing console logger for smithi114 2024-03-06T19:11:41.172 DEBUG:teuthology.task.console_log:Killing console logger for smithi177 2024-03-06T19:11:41.172 DEBUG:teuthology.exit:Finished running handlers